2024-05-28T05:50:43.678 INFO:root:teuthology version: 0.0.1.dev291+ge7ae758 2024-05-28T05:50:43.678 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/rishabh-2024-05-28_05:40:25-fs:volumes-rishabh-clone-stats-testing-default-smithi/7729374 --description fs:volumes/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/volumes/{overrides test/clone-progress}} --owner scheduled_rishabh@teuthology --verbose --name rishabh-2024-05-28_05:40:25-fs:volumes-rishabh-clone-stats-testing-default-smithi -- /home/teuthworker/archive/rishabh-2024-05-28_05:40:25-fs:volumes-rishabh-clone-stats-testing-default-smithi/7729374/orig.config.yaml 2024-05-28T05:50:43.727 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-28T05:50:43.812 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/rishabh-2024-05-28_05:40:25-fs:volumes-rishabh-clone-stats-testing-default-smithi/7729374 branch: rishabh-clone-stats description: fs:volumes/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/volumes/{overrides test/clone-progress}} email: null first_in_suite: false job_id: '7729374' kernel: flavor: default kdb: 1 sha1: b9459a26b0888785d05c47deb35abc9d0354ab93 last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: rishabh-2024-05-28_05:40:25-fs:volumes-rishabh-clone-stats-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: rishabh-clone-stats ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug finisher: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - OSD full dropping all updates - OSD near full - pausewr flag - failsafe engaged, dropping updates - failsafe disengaged, no longer dropping - is full \(reached quota - POOL_FULL - POOL_BACKFILLFULL sha1: 9ed025fbf4406fb54980a0647506b15895b6d863 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 9ed025fbf4406fb54980a0647506b15895b6d863 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: mgr-vol-clone-stats sha1: 9ed025fbf4406fb54980a0647506b15895b6d863 owner: scheduled_rishabh@teuthology priority: 50 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 8528 sha1: 9ed025fbf4406fb54980a0647506b15895b6d863 sleep_before_teardown: 0 suite: fs:volumes suite_branch: mgr-vol-clone-stats suite_path: /home/teuthworker/src/github.com_rishabh-d-dave_ceph_9ed025fbf4406fb54980a0647506b15895b6d863/qa suite_relpath: qa suite_repo: https://github.com/rishabh-d-dave/ceph suite_sha1: 9ed025fbf4406fb54980a0647506b15895b6d863 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi096.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL4jJA/0Z9YK9hfc6842L/ak9ogtcxjH3nLInKMwUJjkE2BSbM6O+geRv29I7bIO27Bmr6wNs2kEMVNHKHiW//k= smithi123.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPlSxAwcnXg7ZNeI06pVJq57T5CWYy1BuOlwhUhGt6s9B0g4MLXL9KbpN/TiOZ+dk3kD7+DwK5iYwTQ9FclVPP4= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - sequential: - mgrmodules - ceph-fuse: null - cephfs_test_runner: fail_on_skip: false modules: - tasks.cephfs.test_volumes.TestCloneProgressReporter teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: e7ae7580c417d6ab27fda98fecadca5d9f85d63e timestamp: 2024-05-28_05:40:25 tube: smithi user: rishabh verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2698841 2024-05-28T05:50:43.813 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_rishabh-d-dave_ceph_9ed025fbf4406fb54980a0647506b15895b6d863/qa; will attempt to use it 2024-05-28T05:50:43.813 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_rishabh-d-dave_ceph_9ed025fbf4406fb54980a0647506b15895b6d863/qa/tasks 2024-05-28T05:50:43.814 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-05-28T05:50:43.815 INFO:teuthology.task.internal:Checking packages... 2024-05-28T05:50:43.840 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '9ed025fbf4406fb54980a0647506b15895b6d863' 2024-05-28T05:50:43.840 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-05-28T05:50:43.840 INFO:teuthology.packaging:ref: None 2024-05-28T05:50:43.840 INFO:teuthology.packaging:tag: None 2024-05-28T05:50:43.840 INFO:teuthology.packaging:branch: rishabh-clone-stats 2024-05-28T05:50:43.840 INFO:teuthology.packaging:sha1: 9ed025fbf4406fb54980a0647506b15895b6d863 2024-05-28T05:50:43.841 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=rishabh-clone-stats 2024-05-28T05:50:44.067 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3154.g9ed025fb 2024-05-28T05:50:44.069 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-05-28T05:50:44.075 INFO:teuthology.task.internal:no buildpackages task found 2024-05-28T05:50:44.075 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-05-28T05:50:44.098 INFO:teuthology.task.internal:Saving configuration 2024-05-28T05:50:44.115 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-05-28T05:50:44.122 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-28T05:50:44.149 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi096.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-05-28_05:40:25-fs:volumes-rishabh-clone-stats-testing-default-smithi/7729374', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-28 05:43:51.257171', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL4jJA/0Z9YK9hfc6842L/ak9ogtcxjH3nLInKMwUJjkE2BSbM6O+geRv29I7bIO27Bmr6wNs2kEMVNHKHiW//k='} 2024-05-28T05:50:44.173 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi123.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-05-28_05:40:25-fs:volumes-rishabh-clone-stats-testing-default-smithi/7729374', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-28 05:43:51.255357', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPlSxAwcnXg7ZNeI06pVJq57T5CWYy1BuOlwhUhGt6s9B0g4MLXL9KbpN/TiOZ+dk3kD7+DwK5iYwTQ9FclVPP4='} 2024-05-28T05:50:44.174 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-05-28T05:50:44.180 INFO:teuthology.task.internal:roles: ubuntu@smithi096.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2024-05-28T05:50:44.180 INFO:teuthology.task.internal:roles: ubuntu@smithi123.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2024-05-28T05:50:44.180 INFO:teuthology.run_tasks:Running task console_log... 2024-05-28T05:50:44.278 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f1423c03be0>, signals=[15]) 2024-05-28T05:50:44.278 INFO:teuthology.run_tasks:Running task internal.connect... 2024-05-28T05:50:44.383 INFO:teuthology.task.internal:Opening connections... 2024-05-28T05:50:44.384 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi096.front.sepia.ceph.com 2024-05-28T05:50:44.385 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi096.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-28T05:50:44.476 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi123.front.sepia.ceph.com 2024-05-28T05:50:44.477 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-28T05:50:44.551 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-05-28T05:50:44.557 DEBUG:teuthology.orchestra.run.smithi096:> uname -m 2024-05-28T05:50:44.575 INFO:teuthology.orchestra.run.smithi096.stdout:x86_64 2024-05-28T05:50:44.575 DEBUG:teuthology.orchestra.run.smithi096:> cat /etc/os-release 2024-05-28T05:50:44.632 INFO:teuthology.orchestra.run.smithi096.stdout:NAME="CentOS Stream" 2024-05-28T05:50:44.632 INFO:teuthology.orchestra.run.smithi096.stdout:VERSION="9" 2024-05-28T05:50:44.632 INFO:teuthology.orchestra.run.smithi096.stdout:ID="centos" 2024-05-28T05:50:44.633 INFO:teuthology.orchestra.run.smithi096.stdout:ID_LIKE="rhel fedora" 2024-05-28T05:50:44.633 INFO:teuthology.orchestra.run.smithi096.stdout:VERSION_ID="9" 2024-05-28T05:50:44.633 INFO:teuthology.orchestra.run.smithi096.stdout:PLATFORM_ID="platform:el9" 2024-05-28T05:50:44.633 INFO:teuthology.orchestra.run.smithi096.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-28T05:50:44.633 INFO:teuthology.orchestra.run.smithi096.stdout:ANSI_COLOR="0;31" 2024-05-28T05:50:44.633 INFO:teuthology.orchestra.run.smithi096.stdout:LOGO="fedora-logo-icon" 2024-05-28T05:50:44.633 INFO:teuthology.orchestra.run.smithi096.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-28T05:50:44.633 INFO:teuthology.orchestra.run.smithi096.stdout:HOME_URL="https://centos.org/" 2024-05-28T05:50:44.633 INFO:teuthology.orchestra.run.smithi096.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-28T05:50:44.633 INFO:teuthology.orchestra.run.smithi096.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-28T05:50:44.633 INFO:teuthology.orchestra.run.smithi096.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-28T05:50:44.634 INFO:teuthology.lock.ops:Updating smithi096.front.sepia.ceph.com on lock server 2024-05-28T05:50:44.652 DEBUG:teuthology.orchestra.run.smithi123:> uname -m 2024-05-28T05:50:44.669 INFO:teuthology.orchestra.run.smithi123.stdout:x86_64 2024-05-28T05:50:44.670 DEBUG:teuthology.orchestra.run.smithi123:> cat /etc/os-release 2024-05-28T05:50:44.727 INFO:teuthology.orchestra.run.smithi123.stdout:NAME="CentOS Stream" 2024-05-28T05:50:44.727 INFO:teuthology.orchestra.run.smithi123.stdout:VERSION="9" 2024-05-28T05:50:44.727 INFO:teuthology.orchestra.run.smithi123.stdout:ID="centos" 2024-05-28T05:50:44.727 INFO:teuthology.orchestra.run.smithi123.stdout:ID_LIKE="rhel fedora" 2024-05-28T05:50:44.727 INFO:teuthology.orchestra.run.smithi123.stdout:VERSION_ID="9" 2024-05-28T05:50:44.727 INFO:teuthology.orchestra.run.smithi123.stdout:PLATFORM_ID="platform:el9" 2024-05-28T05:50:44.727 INFO:teuthology.orchestra.run.smithi123.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-28T05:50:44.728 INFO:teuthology.orchestra.run.smithi123.stdout:ANSI_COLOR="0;31" 2024-05-28T05:50:44.728 INFO:teuthology.orchestra.run.smithi123.stdout:LOGO="fedora-logo-icon" 2024-05-28T05:50:44.728 INFO:teuthology.orchestra.run.smithi123.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-28T05:50:44.728 INFO:teuthology.orchestra.run.smithi123.stdout:HOME_URL="https://centos.org/" 2024-05-28T05:50:44.728 INFO:teuthology.orchestra.run.smithi123.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-28T05:50:44.728 INFO:teuthology.orchestra.run.smithi123.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-28T05:50:44.728 INFO:teuthology.orchestra.run.smithi123.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-28T05:50:44.728 INFO:teuthology.lock.ops:Updating smithi123.front.sepia.ceph.com on lock server 2024-05-28T05:50:44.769 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-05-28T05:50:44.778 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-05-28T05:50:44.852 INFO:teuthology.task.internal:Checking for old test directory... 2024-05-28T05:50:44.853 DEBUG:teuthology.orchestra.run.smithi096:> test '!' -e /home/ubuntu/cephtest 2024-05-28T05:50:44.855 DEBUG:teuthology.orchestra.run.smithi123:> test '!' -e /home/ubuntu/cephtest 2024-05-28T05:50:44.872 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-05-28T05:50:44.956 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-05-28T05:50:44.956 DEBUG:teuthology.orchestra.run.smithi096:> test -z $(ls -A /var/lib/ceph) 2024-05-28T05:50:44.959 DEBUG:teuthology.orchestra.run.smithi123:> test -z $(ls -A /var/lib/ceph) 2024-05-28T05:50:45.059 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-05-28T05:50:45.114 INFO:teuthology.run_tasks:Running task kernel... 2024-05-28T05:50:45.128 INFO:teuthology.task.kernel:normalize config orig: {'flavor': 'default', 'kdb': 1, 'sha1': 'b9459a26b0888785d05c47deb35abc9d0354ab93'} 2024-05-28T05:50:45.128 INFO:teuthology.task.kernel:config {'mon.a': {'flavor': 'default', 'kdb': 1, 'sha1': 'b9459a26b0888785d05c47deb35abc9d0354ab93'}, 'mon.b': {'flavor': 'default', 'kdb': 1, 'sha1': 'b9459a26b0888785d05c47deb35abc9d0354ab93'}}, timeout 300 2024-05-28T05:50:45.128 DEBUG:teuthology.orchestra.run.smithi096:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-28T05:50:45.129 DEBUG:teuthology.orchestra.run.smithi123:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-28T05:50:45.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-28T05:50:45.146 DEBUG:teuthology.task.kernel:sha1 for mon.b is b9459a26b0888785d05c47deb35abc9d0354ab93 2024-05-28T05:50:45.146 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "b9459a26b0888785d05c47deb35abc9d0354ab93"... 2024-05-28T05:50:45.146 DEBUG:teuthology.orchestra.run.smithi123:> uname -r 2024-05-28T05:50:45.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-28T05:50:45.147 DEBUG:teuthology.task.kernel:sha1 for mon.a is b9459a26b0888785d05c47deb35abc9d0354ab93 2024-05-28T05:50:45.148 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "b9459a26b0888785d05c47deb35abc9d0354ab93"... 2024-05-28T05:50:45.148 DEBUG:teuthology.orchestra.run.smithi096:> uname -r 2024-05-28T05:50:45.204 INFO:teuthology.orchestra.run.smithi123.stdout:5.14.0-437.el9.x86_64 2024-05-28T05:50:45.204 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-437.el9.x86_64 vs b9459a26b0888785d05c47deb35abc9d0354ab93 2024-05-28T05:50:45.204 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-05-28T05:50:45.204 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b9459a26b0888785d05c47deb35abc9d0354ab93 2024-05-28T05:50:45.208 INFO:teuthology.orchestra.run.smithi096.stdout:5.14.0-437.el9.x86_64 2024-05-28T05:50:45.208 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-437.el9.x86_64 vs b9459a26b0888785d05c47deb35abc9d0354ab93 2024-05-28T05:50:45.208 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-05-28T05:50:45.208 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b9459a26b0888785d05c47deb35abc9d0354ab93 2024-05-28T05:50:45.350 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum upgrade -y linux-firmware 2024-05-28T05:50:45.429 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum upgrade -y linux-firmware 2024-05-28T05:50:47.468 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.1 MB 00:00 2024-05-28T05:50:48.953 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - BaseOS 4.3 MB/s | 8.1 MB 00:01 2024-05-28T05:50:50.693 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - AppStream 28 MB/s | 19 MB 00:00 2024-05-28T05:50:51.552 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 9 - AppStream 6.3 MB/s | 19 MB 00:03 2024-05-28T05:50:55.068 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - CRB 8.5 MB/s | 6.2 MB 00:00 2024-05-28T05:50:56.394 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - Extras packages 91 kB/s | 17 kB 00:00 2024-05-28T05:50:57.086 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 9 - CRB 3.3 MB/s | 6.2 MB 00:01 2024-05-28T05:50:57.953 INFO:teuthology.orchestra.run.smithi096.stdout:Extra Packages for Enterprise Linux 15 MB/s | 21 MB 00:01 2024-05-28T05:50:58.670 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 9 - Extras packages 39 kB/s | 17 kB 00:00 2024-05-28T05:51:01.846 INFO:teuthology.orchestra.run.smithi123.stdout:Extra Packages for Enterprise Linux 6.9 MB/s | 21 MB 00:03 2024-05-28T05:51:03.094 INFO:teuthology.orchestra.run.smithi096.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-05-28T05:51:05.157 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-05-28T05:51:05.157 INFO:teuthology.orchestra.run.smithi096.stdout:Nothing to do. 2024-05-28T05:51:05.158 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-05-28T05:51:05.287 INFO:teuthology.task.kernel:Downloading kernel b9459a26b0888785d05c47deb35abc9d0354ab93 on mon.a... 2024-05-28T05:51:05.288 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b9459a26b0888785d05c47deb35abc9d0354ab93 2024-05-28T05:51:05.516 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/b9459a26b0888785d05c47deb35abc9d0354ab93/centos/9/flavors/default/x86_64/ 2024-05-28T05:51:05.516 DEBUG:teuthology.orchestra.run.smithi096:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.9.0_gb9459a26b088-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/b9459a26b0888785d05c47deb35abc9d0354ab93/centos/9/flavors/default/x86_64/ --input-file=- 2024-05-28T05:51:05.519 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi096.front.sepia.ceph.com to complete... 2024-05-28T05:51:07.004 INFO:teuthology.orchestra.run.smithi123.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-05-28T05:51:08.995 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-05-28T05:51:08.996 INFO:teuthology.orchestra.run.smithi123.stdout:Nothing to do. 2024-05-28T05:51:08.996 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-05-28T05:51:09.104 INFO:teuthology.task.kernel:Downloading kernel b9459a26b0888785d05c47deb35abc9d0354ab93 on mon.b... 2024-05-28T05:51:09.105 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b9459a26b0888785d05c47deb35abc9d0354ab93 2024-05-28T05:51:09.330 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/b9459a26b0888785d05c47deb35abc9d0354ab93/centos/9/flavors/default/x86_64/ 2024-05-28T05:51:09.330 DEBUG:teuthology.orchestra.run.smithi123:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.9.0_gb9459a26b088-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/b9459a26b0888785d05c47deb35abc9d0354ab93/centos/9/flavors/default/x86_64/ --input-file=- 2024-05-28T05:51:09.333 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi123.front.sepia.ceph.com to complete... 2024-05-28T05:51:23.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28 05:51:23 URL:https://3.chacra.ceph.com/r/kernel/testing/b9459a26b0888785d05c47deb35abc9d0354ab93/centos/9/flavors/default/x86_64/kernel-6.9.0_gb9459a26b088-1.x86_64.rpm [356863406/356863406] -> "/tmp/kernel.x86_64.rpm" [1] 2024-05-28T05:51:23.627 INFO:teuthology.orchestra.run.smithi096.stderr:FINISHED --2024-05-28 05:51:23-- 2024-05-28T05:51:23.627 INFO:teuthology.orchestra.run.smithi096.stderr:Total wall clock time: 18s 2024-05-28T05:51:23.627 INFO:teuthology.orchestra.run.smithi096.stderr:Downloaded: 1 files, 340M in 18s (19.3 MB/s) 2024-05-28T05:51:23.629 INFO:teuthology.task.kernel:Installing kernel b9459a26b0888785d05c47deb35abc9d0354ab93 on mon.a... 2024-05-28T05:51:23.630 DEBUG:teuthology.orchestra.run.smithi096:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-05-28T05:51:25.714 INFO:teuthology.orchestra.run.smithi096.stdout:Verifying... ######################################## 2024-05-28T05:51:25.893 INFO:teuthology.orchestra.run.smithi096.stdout:Preparing... ######################################## 2024-05-28T05:51:25.998 INFO:teuthology.orchestra.run.smithi096.stdout:Updating / installing... 2024-05-28T05:51:31.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28 05:51:31 URL:https://3.chacra.ceph.com/r/kernel/testing/b9459a26b0888785d05c47deb35abc9d0354ab93/centos/9/flavors/default/x86_64/kernel-6.9.0_gb9459a26b088-1.x86_64.rpm [356863406/356863406] -> "/tmp/kernel.x86_64.rpm" [1] 2024-05-28T05:51:31.352 INFO:teuthology.orchestra.run.smithi123.stderr:FINISHED --2024-05-28 05:51:31-- 2024-05-28T05:51:31.352 INFO:teuthology.orchestra.run.smithi123.stderr:Total wall clock time: 22s 2024-05-28T05:51:31.352 INFO:teuthology.orchestra.run.smithi123.stderr:Downloaded: 1 files, 340M in 21s (16.0 MB/s) 2024-05-28T05:51:31.354 INFO:teuthology.task.kernel:Installing kernel b9459a26b0888785d05c47deb35abc9d0354ab93 on mon.b... 2024-05-28T05:51:31.355 DEBUG:teuthology.orchestra.run.smithi123:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-05-28T05:51:33.474 INFO:teuthology.orchestra.run.smithi123.stdout:Verifying... ######################################## 2024-05-28T05:51:33.675 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing... ######################################## 2024-05-28T05:51:33.784 INFO:teuthology.orchestra.run.smithi123.stdout:Updating / installing... 2024-05-28T05:51:33.984 INFO:teuthology.orchestra.run.smithi096.stdout:kernel-6.9.0_gb9459a26b088-1 ######################################## 2024-05-28T05:51:41.675 INFO:teuthology.orchestra.run.smithi123.stdout:kernel-6.9.0_gb9459a26b088-1 ######################################## 2024-05-28T05:52:13.567 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi096.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-05-28T05:52:13.568 DEBUG:teuthology.orchestra.run.smithi096:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-05-28T05:52:13.692 INFO:teuthology.orchestra.run.smithi096.stdout:/boot/System.map-6.9.0-gb9459a26b088 2024-05-28T05:52:13.692 INFO:teuthology.orchestra.run.smithi096.stdout:/boot/config-6.9.0-gb9459a26b088 2024-05-28T05:52:13.692 INFO:teuthology.orchestra.run.smithi096.stdout:/boot/vmlinuz-6.9.0-gb9459a26b088 2024-05-28T05:52:13.692 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/System.map 2024-05-28T05:52:13.692 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/config 2024-05-28T05:52:13.692 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel 2024-05-28T05:52:13.692 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch 2024-05-28T05:52:13.692 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86 2024-05-28T05:52:13.692 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto 2024-05-28T05:52:13.692 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-05-28T05:52:13.693 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-05-28T05:52:13.693 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-05-28T05:52:13.693 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-05-28T05:52:13.693 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-05-28T05:52:13.693 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-05-28T05:52:13.693 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-05-28T05:52:13.693 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/crc32c-intel.ko 2024-05-28T05:52:13.693 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-05-28T05:52:13.693 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-05-28T05:52:13.693 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-05-28T05:52:13.693 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-05-28T05:52:13.693 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/serpent-avx2.ko 2024-05-28T05:52:13.694 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-05-28T05:52:13.694 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-05-28T05:52:13.694 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-05-28T05:52:13.694 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-05-28T05:52:13.694 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-05-28T05:52:13.694 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-05-28T05:52:13.694 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-05-28T05:52:13.694 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kernel 2024-05-28T05:52:13.694 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kernel/cpu 2024-05-28T05:52:13.694 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kernel/cpu/mce 2024-05-28T05:52:13.694 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-05-28T05:52:13.694 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kvm 2024-05-28T05:52:13.695 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kvm/kvm-amd.ko 2024-05-28T05:52:13.695 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kvm/kvm-intel.ko 2024-05-28T05:52:13.695 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kvm/kvm.ko 2024-05-28T05:52:13.695 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto 2024-05-28T05:52:13.695 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/ansi_cprng.ko 2024-05-28T05:52:13.695 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/anubis.ko 2024-05-28T05:52:13.695 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/arc4.ko 2024-05-28T05:52:13.695 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/async_tx 2024-05-28T05:52:13.695 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/async_tx/async_memcpy.ko 2024-05-28T05:52:13.695 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/async_tx/async_pq.ko 2024-05-28T05:52:13.695 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/async_tx/async_raid6_recov.ko 2024-05-28T05:52:13.695 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/async_tx/async_tx.ko 2024-05-28T05:52:13.695 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/async_tx/async_xor.ko 2024-05-28T05:52:13.696 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/async_tx/raid6test.ko 2024-05-28T05:52:13.696 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/authenc.ko 2024-05-28T05:52:13.696 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/authencesn.ko 2024-05-28T05:52:13.696 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/blake2b_generic.ko 2024-05-28T05:52:13.696 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/blowfish_common.ko 2024-05-28T05:52:13.696 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/blowfish_generic.ko 2024-05-28T05:52:13.696 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/camellia_generic.ko 2024-05-28T05:52:13.696 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/cast5_generic.ko 2024-05-28T05:52:13.696 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/cast6_generic.ko 2024-05-28T05:52:13.696 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/cast_common.ko 2024-05-28T05:52:13.696 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/ccm.ko 2024-05-28T05:52:13.696 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/cmac.ko 2024-05-28T05:52:13.696 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/crc32_generic.ko 2024-05-28T05:52:13.697 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/crypto_engine.ko 2024-05-28T05:52:13.697 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/crypto_user.ko 2024-05-28T05:52:13.697 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/cts.ko 2024-05-28T05:52:13.697 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/des_generic.ko 2024-05-28T05:52:13.697 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/dh_generic.ko 2024-05-28T05:52:13.697 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/echainiv.ko 2024-05-28T05:52:13.697 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/essiv.ko 2024-05-28T05:52:13.697 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/fcrypt.ko 2024-05-28T05:52:13.697 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/khazad.ko 2024-05-28T05:52:13.697 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/lz4.ko 2024-05-28T05:52:13.697 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/lz4hc.ko 2024-05-28T05:52:13.697 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/md4.ko 2024-05-28T05:52:13.697 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/michael_mic.ko 2024-05-28T05:52:13.698 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/pcbc.ko 2024-05-28T05:52:13.698 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/pcrypt.ko 2024-05-28T05:52:13.698 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/rmd160.ko 2024-05-28T05:52:13.698 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/seed.ko 2024-05-28T05:52:13.698 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/serpent_generic.ko 2024-05-28T05:52:13.698 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/tcrypt.ko 2024-05-28T05:52:13.698 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/tea.ko 2024-05-28T05:52:13.698 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/twofish_common.ko 2024-05-28T05:52:13.698 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/twofish_generic.ko 2024-05-28T05:52:13.698 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/vmac.ko 2024-05-28T05:52:13.698 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/wp512.ko 2024-05-28T05:52:13.698 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/xcbc.ko 2024-05-28T05:52:13.699 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/xor.ko 2024-05-28T05:52:13.699 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/xxhash_generic.ko 2024-05-28T05:52:13.699 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers 2024-05-28T05:52:13.699 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi 2024-05-28T05:52:13.699 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/acpi_ipmi.ko 2024-05-28T05:52:13.699 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/acpi_pad.ko 2024-05-28T05:52:13.699 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/ec_sys.ko 2024-05-28T05:52:13.699 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/platform_profile.ko 2024-05-28T05:52:13.699 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/sbs.ko 2024-05-28T05:52:13.699 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/sbshc.ko 2024-05-28T05:52:13.699 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/video.ko 2024-05-28T05:52:13.699 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata 2024-05-28T05:52:13.700 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/acard-ahci.ko 2024-05-28T05:52:13.700 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/ahci_platform.ko 2024-05-28T05:52:13.700 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/ata_generic.ko 2024-05-28T05:52:13.700 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/libahci_platform.ko 2024-05-28T05:52:13.700 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_acpi.ko 2024-05-28T05:52:13.700 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_ali.ko 2024-05-28T05:52:13.700 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_amd.ko 2024-05-28T05:52:13.700 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_artop.ko 2024-05-28T05:52:13.700 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_atiixp.ko 2024-05-28T05:52:13.700 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_atp867x.ko 2024-05-28T05:52:13.700 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_cmd640.ko 2024-05-28T05:52:13.700 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_cmd64x.ko 2024-05-28T05:52:13.700 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_cypress.ko 2024-05-28T05:52:13.700 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_efar.ko 2024-05-28T05:52:13.701 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_hpt366.ko 2024-05-28T05:52:13.701 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_hpt37x.ko 2024-05-28T05:52:13.701 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_hpt3x2n.ko 2024-05-28T05:52:13.701 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_hpt3x3.ko 2024-05-28T05:52:13.701 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_it8213.ko 2024-05-28T05:52:13.701 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_it821x.ko 2024-05-28T05:52:13.701 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_jmicron.ko 2024-05-28T05:52:13.701 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_marvell.ko 2024-05-28T05:52:13.701 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_mpiix.ko 2024-05-28T05:52:13.701 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_netcell.ko 2024-05-28T05:52:13.701 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_ninja32.ko 2024-05-28T05:52:13.701 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_ns87410.ko 2024-05-28T05:52:13.701 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_ns87415.ko 2024-05-28T05:52:13.701 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_oldpiix.ko 2024-05-28T05:52:13.701 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_opti.ko 2024-05-28T05:52:13.702 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_optidma.ko 2024-05-28T05:52:13.702 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_pcmcia.ko 2024-05-28T05:52:13.702 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_pdc2027x.ko 2024-05-28T05:52:13.702 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-05-28T05:52:13.702 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_piccolo.ko 2024-05-28T05:52:13.702 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_rdc.ko 2024-05-28T05:52:13.702 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_sch.ko 2024-05-28T05:52:13.702 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_serverworks.ko 2024-05-28T05:52:13.702 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_sil680.ko 2024-05-28T05:52:13.702 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_sis.ko 2024-05-28T05:52:13.702 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_sl82c105.ko 2024-05-28T05:52:13.702 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_triflex.ko 2024-05-28T05:52:13.702 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_via.ko 2024-05-28T05:52:13.703 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pdc_adma.ko 2024-05-28T05:52:13.703 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_inic162x.ko 2024-05-28T05:52:13.703 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_mv.ko 2024-05-28T05:52:13.703 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_nv.ko 2024-05-28T05:52:13.703 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_promise.ko 2024-05-28T05:52:13.703 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_qstor.ko 2024-05-28T05:52:13.703 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_sil.ko 2024-05-28T05:52:13.703 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_sil24.ko 2024-05-28T05:52:13.703 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_sis.ko 2024-05-28T05:52:13.703 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_svw.ko 2024-05-28T05:52:13.703 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_sx4.ko 2024-05-28T05:52:13.703 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_uli.ko 2024-05-28T05:52:13.703 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_via.ko 2024-05-28T05:52:13.703 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_vsc.ko 2024-05-28T05:52:13.704 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/auxdisplay 2024-05-28T05:52:13.704 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/auxdisplay/ks0108.ko 2024-05-28T05:52:13.704 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/base 2024-05-28T05:52:13.704 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/base/regmap 2024-05-28T05:52:13.704 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/base/regmap/regmap-i2c.ko 2024-05-28T05:52:13.704 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/base/regmap/regmap-mmio.ko 2024-05-28T05:52:13.704 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/bcma 2024-05-28T05:52:13.704 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/bcma/bcma.ko 2024-05-28T05:52:13.704 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block 2024-05-28T05:52:13.704 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/aoe 2024-05-28T05:52:13.704 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/aoe/aoe.ko 2024-05-28T05:52:13.704 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/brd.ko 2024-05-28T05:52:13.704 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/drbd 2024-05-28T05:52:13.705 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/drbd/drbd.ko 2024-05-28T05:52:13.705 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/floppy.ko 2024-05-28T05:52:13.705 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/loop.ko 2024-05-28T05:52:13.705 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/mtip32xx 2024-05-28T05:52:13.705 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-05-28T05:52:13.705 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/nbd.ko 2024-05-28T05:52:13.705 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/null_blk 2024-05-28T05:52:13.705 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/null_blk/null_blk.ko 2024-05-28T05:52:13.705 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/pktcdvd.ko 2024-05-28T05:52:13.705 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/rbd.ko 2024-05-28T05:52:13.705 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/virtio_blk.ko 2024-05-28T05:52:13.705 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/xen-blkback 2024-05-28T05:52:13.705 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-05-28T05:52:13.705 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/xen-blkfront.ko 2024-05-28T05:52:13.706 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/zram 2024-05-28T05:52:13.706 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/zram/zram.ko 2024-05-28T05:52:13.706 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char 2024-05-28T05:52:13.706 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/hangcheck-timer.ko 2024-05-28T05:52:13.706 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/hw_random 2024-05-28T05:52:13.706 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/hw_random/amd-rng.ko 2024-05-28T05:52:13.706 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/hw_random/intel-rng.ko 2024-05-28T05:52:13.706 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-05-28T05:52:13.706 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/hw_random/via-rng.ko 2024-05-28T05:52:13.706 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/hw_random/virtio-rng.ko 2024-05-28T05:52:13.706 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ipmi 2024-05-28T05:52:13.706 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-05-28T05:52:13.706 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-05-28T05:52:13.707 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-05-28T05:52:13.707 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ipmi/ipmi_si.ko 2024-05-28T05:52:13.707 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-05-28T05:52:13.707 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-05-28T05:52:13.707 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/lp.ko 2024-05-28T05:52:13.707 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/mwave 2024-05-28T05:52:13.707 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/mwave/mwave.ko 2024-05-28T05:52:13.707 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ppdev.ko 2024-05-28T05:52:13.707 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tlclk.ko 2024-05-28T05:52:13.707 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm 2024-05-28T05:52:13.707 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm/tpm.ko 2024-05-28T05:52:13.707 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm/tpm_atmel.ko 2024-05-28T05:52:13.708 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm/tpm_infineon.ko 2024-05-28T05:52:13.708 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm/tpm_nsc.ko 2024-05-28T05:52:13.708 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm/tpm_tis.ko 2024-05-28T05:52:13.708 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-05-28T05:52:13.708 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/uv_mmtimer.ko 2024-05-28T05:52:13.708 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/virtio_console.ko 2024-05-28T05:52:13.708 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/cpufreq 2024-05-28T05:52:13.708 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-05-28T05:52:13.708 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-05-28T05:52:13.708 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/cpufreq/p4-clockmod.ko 2024-05-28T05:52:13.708 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-05-28T05:52:13.708 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/cpufreq/powernow-k8.ko 2024-05-28T05:52:13.708 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/cpufreq/speedstep-lib.ko 2024-05-28T05:52:13.709 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto 2024-05-28T05:52:13.709 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/ccp 2024-05-28T05:52:13.709 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-05-28T05:52:13.709 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/ccp/ccp.ko 2024-05-28T05:52:13.709 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/intel 2024-05-28T05:52:13.709 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/intel/qat 2024-05-28T05:52:13.709 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/intel/qat/qat_common 2024-05-28T05:52:13.709 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-05-28T05:52:13.709 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-05-28T05:52:13.709 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-05-28T05:52:13.709 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/padlock-aes.ko 2024-05-28T05:52:13.709 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/padlock-sha.ko 2024-05-28T05:52:13.709 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/virtio 2024-05-28T05:52:13.710 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-05-28T05:52:13.710 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/dca 2024-05-28T05:52:13.710 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/dca/dca.ko 2024-05-28T05:52:13.710 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/devfreq 2024-05-28T05:52:13.710 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-05-28T05:52:13.710 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/dma 2024-05-28T05:52:13.710 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/dma/dw 2024-05-28T05:52:13.710 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/dma/dw/dw_dmac.ko 2024-05-28T05:52:13.710 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/dma/ioat 2024-05-28T05:52:13.710 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/dma/ioat/ioatdma.ko 2024-05-28T05:52:13.710 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac 2024-05-28T05:52:13.710 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/amd64_edac.ko 2024-05-28T05:52:13.710 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/e752x_edac.ko 2024-05-28T05:52:13.710 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/edac_mce_amd.ko 2024-05-28T05:52:13.711 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/i3000_edac.ko 2024-05-28T05:52:13.711 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/i3200_edac.ko 2024-05-28T05:52:13.711 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/i5100_edac.ko 2024-05-28T05:52:13.711 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/i5400_edac.ko 2024-05-28T05:52:13.711 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/i7300_edac.ko 2024-05-28T05:52:13.711 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/i7core_edac.ko 2024-05-28T05:52:13.711 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/i82975x_edac.ko 2024-05-28T05:52:13.711 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/ie31200_edac.ko 2024-05-28T05:52:13.711 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/sb_edac.ko 2024-05-28T05:52:13.711 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/x38_edac.ko 2024-05-28T05:52:13.711 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/firmware 2024-05-28T05:52:13.711 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/firmware/edd.ko 2024-05-28T05:52:13.711 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/firmware/iscsi_ibft.ko 2024-05-28T05:52:13.712 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio 2024-05-28T05:52:13.712 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-ich.ko 2024-05-28T05:52:13.712 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-viperboard.ko 2024-05-28T05:52:13.712 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid 2024-05-28T05:52:13.712 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-appleir.ko 2024-05-28T05:52:13.712 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-aureal.ko 2024-05-28T05:52:13.712 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-axff.ko 2024-05-28T05:52:13.712 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-dr.ko 2024-05-28T05:52:13.712 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-elecom.ko 2024-05-28T05:52:13.712 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-elo.ko 2024-05-28T05:52:13.712 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-emsff.ko 2024-05-28T05:52:13.712 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-gaff.ko 2024-05-28T05:52:13.712 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-gt683r.ko 2024-05-28T05:52:13.712 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-gyration.ko 2024-05-28T05:52:13.712 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-holtek-kbd.ko 2024-05-28T05:52:13.713 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-holtek-mouse.ko 2024-05-28T05:52:13.713 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-holtekff.ko 2024-05-28T05:52:13.713 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-hyperv.ko 2024-05-28T05:52:13.713 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-icade.ko 2024-05-28T05:52:13.713 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-keytouch.ko 2024-05-28T05:52:13.713 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-kye.ko 2024-05-28T05:52:13.713 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-lcpower.ko 2024-05-28T05:52:13.713 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-led.ko 2024-05-28T05:52:13.713 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-lenovo.ko 2024-05-28T05:52:13.713 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-logitech-dj.ko 2024-05-28T05:52:13.713 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-05-28T05:52:13.713 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-multitouch.ko 2024-05-28T05:52:13.713 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-ortek.ko 2024-05-28T05:52:13.714 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-penmount.ko 2024-05-28T05:52:13.714 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-petalynx.ko 2024-05-28T05:52:13.714 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-picolcd.ko 2024-05-28T05:52:13.714 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-pl.ko 2024-05-28T05:52:13.714 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-plantronics.ko 2024-05-28T05:52:13.714 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-primax.ko 2024-05-28T05:52:13.714 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-rmi.ko 2024-05-28T05:52:13.714 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-arvo.ko 2024-05-28T05:52:13.714 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-common.ko 2024-05-28T05:52:13.714 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-isku.ko 2024-05-28T05:52:13.714 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-kone.ko 2024-05-28T05:52:13.714 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-05-28T05:52:13.714 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-konepure.ko 2024-05-28T05:52:13.715 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-05-28T05:52:13.715 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-lua.ko 2024-05-28T05:52:13.715 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-pyra.ko 2024-05-28T05:52:13.715 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-ryos.ko 2024-05-28T05:52:13.715 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-savu.ko 2024-05-28T05:52:13.715 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat.ko 2024-05-28T05:52:13.715 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-saitek.ko 2024-05-28T05:52:13.715 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-samsung.ko 2024-05-28T05:52:13.715 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-sensor-hub.ko 2024-05-28T05:52:13.715 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-sjoy.ko 2024-05-28T05:52:13.715 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-sony.ko 2024-05-28T05:52:13.715 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-speedlink.ko 2024-05-28T05:52:13.715 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-steelseries.ko 2024-05-28T05:52:13.715 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-sunplus.ko 2024-05-28T05:52:13.716 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-thrustmaster.ko 2024-05-28T05:52:13.716 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-tivo.ko 2024-05-28T05:52:13.716 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-tmff.ko 2024-05-28T05:52:13.716 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-topseed.ko 2024-05-28T05:52:13.716 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-twinhan.ko 2024-05-28T05:52:13.716 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-uclogic.ko 2024-05-28T05:52:13.716 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-waltop.ko 2024-05-28T05:52:13.716 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-wiimote.ko 2024-05-28T05:52:13.716 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-xinmo.ko 2024-05-28T05:52:13.716 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-zpff.ko 2024-05-28T05:52:13.716 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-zydacron.ko 2024-05-28T05:52:13.716 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/uhid.ko 2024-05-28T05:52:13.716 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/wacom.ko 2024-05-28T05:52:13.716 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hv 2024-05-28T05:52:13.717 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hv/hv_balloon.ko 2024-05-28T05:52:13.717 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hv/hv_utils.ko 2024-05-28T05:52:13.717 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hv/hv_vmbus.ko 2024-05-28T05:52:13.717 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon 2024-05-28T05:52:13.717 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/abituguru.ko 2024-05-28T05:52:13.717 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/abituguru3.ko 2024-05-28T05:52:13.717 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/acpi_power_meter.ko 2024-05-28T05:52:13.717 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ad7414.ko 2024-05-28T05:52:13.717 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ad7418.ko 2024-05-28T05:52:13.717 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adc128d818.ko 2024-05-28T05:52:13.717 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adm1025.ko 2024-05-28T05:52:13.717 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adm1026.ko 2024-05-28T05:52:13.717 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adm1029.ko 2024-05-28T05:52:13.717 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adm1031.ko 2024-05-28T05:52:13.718 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adm9240.ko 2024-05-28T05:52:13.718 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ads7828.ko 2024-05-28T05:52:13.718 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adt7410.ko 2024-05-28T05:52:13.718 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adt7411.ko 2024-05-28T05:52:13.718 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adt7462.ko 2024-05-28T05:52:13.718 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adt7470.ko 2024-05-28T05:52:13.718 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adt7475.ko 2024-05-28T05:52:13.718 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adt7x10.ko 2024-05-28T05:52:13.718 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/amc6821.ko 2024-05-28T05:52:13.718 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/applesmc.ko 2024-05-28T05:52:13.718 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/asb100.ko 2024-05-28T05:52:13.718 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/asc7621.ko 2024-05-28T05:52:13.718 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/asus_atk0110.ko 2024-05-28T05:52:13.719 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/atxp1.ko 2024-05-28T05:52:13.719 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/coretemp.ko 2024-05-28T05:52:13.719 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-05-28T05:52:13.719 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/dme1737.ko 2024-05-28T05:52:13.719 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ds1621.ko 2024-05-28T05:52:13.719 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ds620.ko 2024-05-28T05:52:13.719 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/emc1403.ko 2024-05-28T05:52:13.719 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/emc6w201.ko 2024-05-28T05:52:13.719 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/f71805f.ko 2024-05-28T05:52:13.719 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/f71882fg.ko 2024-05-28T05:52:13.719 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/f75375s.ko 2024-05-28T05:52:13.719 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/fam15h_power.ko 2024-05-28T05:52:13.719 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/fschmd.ko 2024-05-28T05:52:13.719 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/g760a.ko 2024-05-28T05:52:13.720 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/g762.ko 2024-05-28T05:52:13.720 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/gl518sm.ko 2024-05-28T05:52:13.720 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/gl520sm.ko 2024-05-28T05:52:13.720 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/hwmon-vid.ko 2024-05-28T05:52:13.720 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/i5500_temp.ko 2024-05-28T05:52:13.720 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/i5k_amb.ko 2024-05-28T05:52:13.720 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ibmaem.ko 2024-05-28T05:52:13.720 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ibmpex.ko 2024-05-28T05:52:13.720 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ina209.ko 2024-05-28T05:52:13.720 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ina2xx.ko 2024-05-28T05:52:13.720 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/it87.ko 2024-05-28T05:52:13.720 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/k10temp.ko 2024-05-28T05:52:13.721 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/k8temp.ko 2024-05-28T05:52:13.721 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lineage-pem.ko 2024-05-28T05:52:13.721 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm63.ko 2024-05-28T05:52:13.721 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm73.ko 2024-05-28T05:52:13.721 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm75.ko 2024-05-28T05:52:13.721 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm77.ko 2024-05-28T05:52:13.721 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm78.ko 2024-05-28T05:52:13.721 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm80.ko 2024-05-28T05:52:13.721 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm83.ko 2024-05-28T05:52:13.721 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm85.ko 2024-05-28T05:52:13.721 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm87.ko 2024-05-28T05:52:13.721 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm90.ko 2024-05-28T05:52:13.721 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm92.ko 2024-05-28T05:52:13.722 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm93.ko 2024-05-28T05:52:13.722 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm95234.ko 2024-05-28T05:52:13.722 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm95241.ko 2024-05-28T05:52:13.722 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm95245.ko 2024-05-28T05:52:13.722 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ltc2945.ko 2024-05-28T05:52:13.722 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ltc4151.ko 2024-05-28T05:52:13.722 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ltc4215.ko 2024-05-28T05:52:13.722 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ltc4222.ko 2024-05-28T05:52:13.722 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ltc4245.ko 2024-05-28T05:52:13.722 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ltc4260.ko 2024-05-28T05:52:13.722 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ltc4261.ko 2024-05-28T05:52:13.722 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/max16065.ko 2024-05-28T05:52:13.722 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/max1619.ko 2024-05-28T05:52:13.723 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/max1668.ko 2024-05-28T05:52:13.723 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/max197.ko 2024-05-28T05:52:13.723 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/max6639.ko 2024-05-28T05:52:13.723 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/max6650.ko 2024-05-28T05:52:13.723 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/max6697.ko 2024-05-28T05:52:13.723 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/mcp3021.ko 2024-05-28T05:52:13.723 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/nct6683.ko 2024-05-28T05:52:13.723 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/nct6775-core.ko 2024-05-28T05:52:13.723 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/nct6775.ko 2024-05-28T05:52:13.723 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/nct7802.ko 2024-05-28T05:52:13.723 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pc87360.ko 2024-05-28T05:52:13.723 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pc87427.ko 2024-05-28T05:52:13.723 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pcf8591.ko 2024-05-28T05:52:13.723 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus 2024-05-28T05:52:13.723 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-05-28T05:52:13.724 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-05-28T05:52:13.724 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-05-28T05:52:13.724 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/max16064.ko 2024-05-28T05:52:13.724 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/max34440.ko 2024-05-28T05:52:13.724 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/max8688.ko 2024-05-28T05:52:13.724 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-05-28T05:52:13.724 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-05-28T05:52:13.724 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-05-28T05:52:13.724 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-05-28T05:52:13.724 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-05-28T05:52:13.724 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-05-28T05:52:13.724 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/powr1220.ko 2024-05-28T05:52:13.724 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/sch5627.ko 2024-05-28T05:52:13.725 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/sch5636.ko 2024-05-28T05:52:13.725 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/sch56xx-common.ko 2024-05-28T05:52:13.725 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/sht15.ko 2024-05-28T05:52:13.725 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/sht21.ko 2024-05-28T05:52:13.725 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/shtc1.ko 2024-05-28T05:52:13.725 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/sis5595.ko 2024-05-28T05:52:13.725 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/smsc47b397.ko 2024-05-28T05:52:13.725 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/smsc47m1.ko 2024-05-28T05:52:13.725 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/smsc47m192.ko 2024-05-28T05:52:13.725 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/thmc50.ko 2024-05-28T05:52:13.725 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/tmp102.ko 2024-05-28T05:52:13.725 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/tmp103.ko 2024-05-28T05:52:13.725 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/tmp401.ko 2024-05-28T05:52:13.725 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/tmp421.ko 2024-05-28T05:52:13.726 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/via-cputemp.ko 2024-05-28T05:52:13.726 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/via686a.ko 2024-05-28T05:52:13.726 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/vt1211.ko 2024-05-28T05:52:13.726 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/vt8231.ko 2024-05-28T05:52:13.726 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83627ehf.ko 2024-05-28T05:52:13.726 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83627hf.ko 2024-05-28T05:52:13.726 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83781d.ko 2024-05-28T05:52:13.726 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83791d.ko 2024-05-28T05:52:13.726 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83792d.ko 2024-05-28T05:52:13.726 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83793.ko 2024-05-28T05:52:13.726 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83795.ko 2024-05-28T05:52:13.726 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83l785ts.ko 2024-05-28T05:52:13.726 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83l786ng.ko 2024-05-28T05:52:13.727 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c 2024-05-28T05:52:13.727 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/algos 2024-05-28T05:52:13.727 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-05-28T05:52:13.727 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-05-28T05:52:13.727 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses 2024-05-28T05:52:13.727 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-05-28T05:52:13.727 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-05-28T05:52:13.727 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-05-28T05:52:13.727 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-05-28T05:52:13.727 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-05-28T05:52:13.727 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-05-28T05:52:13.727 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-05-28T05:52:13.727 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-05-28T05:52:13.727 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-i801.ko 2024-05-28T05:52:13.728 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-isch.ko 2024-05-28T05:52:13.728 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-05-28T05:52:13.728 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-05-28T05:52:13.728 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-05-28T05:52:13.728 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-parport.ko 2024-05-28T05:52:13.728 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-05-28T05:52:13.728 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-05-28T05:52:13.728 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-05-28T05:52:13.729 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-05-28T05:52:13.729 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-05-28T05:52:13.729 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-05-28T05:52:13.729 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-via.ko 2024-05-28T05:52:13.729 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-05-28T05:52:13.729 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-05-28T05:52:13.729 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/i2c-dev.ko 2024-05-28T05:52:13.730 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/i2c-mux.ko 2024-05-28T05:52:13.730 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-05-28T05:52:13.730 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/i2c-smbus.ko 2024-05-28T05:52:13.730 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/i2c-stub.ko 2024-05-28T05:52:13.730 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband 2024-05-28T05:52:13.730 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core 2024-05-28T05:52:13.730 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/ib_cm.ko 2024-05-28T05:52:13.730 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/ib_core.ko 2024-05-28T05:52:13.730 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/ib_umad.ko 2024-05-28T05:52:13.730 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-05-28T05:52:13.730 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/iw_cm.ko 2024-05-28T05:52:13.730 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/rdma_cm.ko 2024-05-28T05:52:13.731 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-05-28T05:52:13.731 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw 2024-05-28T05:52:13.731 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/cxgb4 2024-05-28T05:52:13.731 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-05-28T05:52:13.731 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/mlx4 2024-05-28T05:52:13.731 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-05-28T05:52:13.731 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/mlx5 2024-05-28T05:52:13.731 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-05-28T05:52:13.731 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/mthca 2024-05-28T05:52:13.731 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-05-28T05:52:13.731 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp 2024-05-28T05:52:13.731 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/ipoib 2024-05-28T05:52:13.731 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-05-28T05:52:13.732 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/iser 2024-05-28T05:52:13.732 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-05-28T05:52:13.732 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/isert 2024-05-28T05:52:13.732 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-05-28T05:52:13.732 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/srp 2024-05-28T05:52:13.732 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-05-28T05:52:13.732 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/srpt 2024-05-28T05:52:13.732 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-05-28T05:52:13.732 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input 2024-05-28T05:52:13.732 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/gameport 2024-05-28T05:52:13.732 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-05-28T05:52:13.732 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/gameport/fm801-gp.ko 2024-05-28T05:52:13.732 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/gameport/gameport.ko 2024-05-28T05:52:13.732 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/joydev.ko 2024-05-28T05:52:13.733 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard 2024-05-28T05:52:13.733 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard/gpio_keys.ko 2024-05-28T05:52:13.733 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse 2024-05-28T05:52:13.733 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/appletouch.ko 2024-05-28T05:52:13.733 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/bcm5974.ko 2024-05-28T05:52:13.733 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/cyapatp.ko 2024-05-28T05:52:13.733 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/elan_i2c.ko 2024-05-28T05:52:13.733 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/sermouse.ko 2024-05-28T05:52:13.733 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-05-28T05:52:13.733 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/synaptics_usb.ko 2024-05-28T05:52:13.733 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/vsxxxaa.ko 2024-05-28T05:52:13.733 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/rmi4 2024-05-28T05:52:13.733 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/rmi4/rmi_core.ko 2024-05-28T05:52:13.734 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/serio 2024-05-28T05:52:13.734 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/serio/altera_ps2.ko 2024-05-28T05:52:13.734 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/serio/arc_ps2.ko 2024-05-28T05:52:13.734 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-05-28T05:52:13.734 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/serio/serio_raw.ko 2024-05-28T05:52:13.734 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/sparse-keymap.ko 2024-05-28T05:52:13.734 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds 2024-05-28T05:52:13.734 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-blinkm.ko 2024-05-28T05:52:13.734 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-lm3530.ko 2024-05-28T05:52:13.734 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-lp3944.ko 2024-05-28T05:52:13.734 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-ss4200.ko 2024-05-28T05:52:13.734 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger 2024-05-28T05:52:13.734 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-05-28T05:52:13.735 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-05-28T05:52:13.735 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-05-28T05:52:13.735 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-05-28T05:52:13.735 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-05-28T05:52:13.735 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-05-28T05:52:13.735 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-05-28T05:52:13.735 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-05-28T05:52:13.735 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-05-28T05:52:13.735 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md 2024-05-28T05:52:13.735 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/bcache 2024-05-28T05:52:13.735 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/bcache/bcache.ko 2024-05-28T05:52:13.735 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-bio-prison.ko 2024-05-28T05:52:13.735 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-cache-smq.ko 2024-05-28T05:52:13.735 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-cache.ko 2024-05-28T05:52:13.736 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-crypt.ko 2024-05-28T05:52:13.736 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-delay.ko 2024-05-28T05:52:13.736 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-flakey.ko 2024-05-28T05:52:13.736 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-log-userspace.ko 2024-05-28T05:52:13.736 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-multipath.ko 2024-05-28T05:52:13.736 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-queue-length.ko 2024-05-28T05:52:13.736 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-raid.ko 2024-05-28T05:52:13.736 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-round-robin.ko 2024-05-28T05:52:13.736 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-service-time.ko 2024-05-28T05:52:13.736 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-switch.ko 2024-05-28T05:52:13.736 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-thin-pool.ko 2024-05-28T05:52:13.736 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-verity.ko 2024-05-28T05:52:13.736 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/persistent-data 2024-05-28T05:52:13.736 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-05-28T05:52:13.737 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/raid0.ko 2024-05-28T05:52:13.737 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/raid1.ko 2024-05-28T05:52:13.737 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/raid10.ko 2024-05-28T05:52:13.737 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/raid456.ko 2024-05-28T05:52:13.737 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message 2024-05-28T05:52:13.737 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion 2024-05-28T05:52:13.737 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/mptbase.ko 2024-05-28T05:52:13.737 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/mptctl.ko 2024-05-28T05:52:13.737 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/mptfc.ko 2024-05-28T05:52:13.737 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/mptlan.ko 2024-05-28T05:52:13.737 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/mptsas.ko 2024-05-28T05:52:13.737 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/mptscsih.ko 2024-05-28T05:52:13.737 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/mptspi.ko 2024-05-28T05:52:13.738 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd 2024-05-28T05:52:13.738 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/lpc_ich.ko 2024-05-28T05:52:13.738 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/lpc_sch.ko 2024-05-28T05:52:13.738 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/mfd-core.ko 2024-05-28T05:52:13.738 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/sm501.ko 2024-05-28T05:52:13.738 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/viperboard.ko 2024-05-28T05:52:13.738 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/vx855.ko 2024-05-28T05:52:13.738 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/wl1273-core.ko 2024-05-28T05:52:13.738 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc 2024-05-28T05:52:13.738 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/altera-stapl 2024-05-28T05:52:13.738 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-05-28T05:52:13.738 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/apds9802als.ko 2024-05-28T05:52:13.738 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/apds990x.ko 2024-05-28T05:52:13.739 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/bh1770glc.ko 2024-05-28T05:52:13.739 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/cb710 2024-05-28T05:52:13.739 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/cb710/cb710.ko 2024-05-28T05:52:13.739 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/eeprom 2024-05-28T05:52:13.739 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/eeprom/at24.ko 2024-05-28T05:52:13.739 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-05-28T05:52:13.739 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/eeprom/max6875.ko 2024-05-28T05:52:13.739 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/enclosure.ko 2024-05-28T05:52:13.739 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/hpilo.ko 2024-05-28T05:52:13.739 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/isl29003.ko 2024-05-28T05:52:13.739 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/isl29020.ko 2024-05-28T05:52:13.739 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/lis3lv02d 2024-05-28T05:52:13.739 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-05-28T05:52:13.739 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-05-28T05:52:13.740 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/mei 2024-05-28T05:52:13.740 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/mei/mei-me.ko 2024-05-28T05:52:13.740 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/mei/mei-txe.ko 2024-05-28T05:52:13.740 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/mei/mei.ko 2024-05-28T05:52:13.740 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/sgi-gru 2024-05-28T05:52:13.740 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/sgi-gru/gru.ko 2024-05-28T05:52:13.740 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/sgi-xp 2024-05-28T05:52:13.740 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/sgi-xp/xp.ko 2024-05-28T05:52:13.740 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/sgi-xp/xpc.ko 2024-05-28T05:52:13.740 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-05-28T05:52:13.740 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/tifm_7xx1.ko 2024-05-28T05:52:13.740 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/tifm_core.ko 2024-05-28T05:52:13.740 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/tsl2550.ko 2024-05-28T05:52:13.740 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/vmw_balloon.ko 2024-05-28T05:52:13.741 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/vmw_vmci 2024-05-28T05:52:13.741 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-05-28T05:52:13.741 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net 2024-05-28T05:52:13.741 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/bonding 2024-05-28T05:52:13.741 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/bonding/bonding.ko 2024-05-28T05:52:13.741 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dsa 2024-05-28T05:52:13.741 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dsa/b53 2024-05-28T05:52:13.741 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dsa/b53/b53_common.ko 2024-05-28T05:52:13.741 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dsa/bcm-sf2.ko 2024-05-28T05:52:13.741 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dsa/mv88e6060.ko 2024-05-28T05:52:13.741 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dsa/mv88e6xxx 2024-05-28T05:52:13.741 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-05-28T05:52:13.741 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dummy.ko 2024-05-28T05:52:13.741 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/eql.ko 2024-05-28T05:52:13.741 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet 2024-05-28T05:52:13.742 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/3com 2024-05-28T05:52:13.742 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-05-28T05:52:13.742 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-05-28T05:52:13.742 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-05-28T05:52:13.742 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-05-28T05:52:13.742 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/8390 2024-05-28T05:52:13.742 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/8390/8390.ko 2024-05-28T05:52:13.742 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-05-28T05:52:13.742 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-05-28T05:52:13.742 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-05-28T05:52:13.742 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/adaptec 2024-05-28T05:52:13.742 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-05-28T05:52:13.743 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/agere 2024-05-28T05:52:13.743 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/agere/et131x.ko 2024-05-28T05:52:13.743 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/alteon 2024-05-28T05:52:13.743 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-05-28T05:52:13.743 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/altera 2024-05-28T05:52:13.743 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-05-28T05:52:13.743 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/amd 2024-05-28T05:52:13.743 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-05-28T05:52:13.743 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-05-28T05:52:13.743 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-05-28T05:52:13.743 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros 2024-05-28T05:52:13.743 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/alx 2024-05-28T05:52:13.743 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-05-28T05:52:13.743 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/atl1c 2024-05-28T05:52:13.744 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-05-28T05:52:13.744 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/atl1e 2024-05-28T05:52:13.744 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-05-28T05:52:13.744 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/atlx 2024-05-28T05:52:13.744 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-05-28T05:52:13.744 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-05-28T05:52:13.744 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom 2024-05-28T05:52:13.744 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-05-28T05:52:13.744 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-05-28T05:52:13.744 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-05-28T05:52:13.744 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-05-28T05:52:13.744 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-05-28T05:52:13.744 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/genet 2024-05-28T05:52:13.744 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-05-28T05:52:13.744 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-05-28T05:52:13.745 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/brocade 2024-05-28T05:52:13.745 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/brocade/bna 2024-05-28T05:52:13.745 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-05-28T05:52:13.745 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio 2024-05-28T05:52:13.745 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb 2024-05-28T05:52:13.745 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-05-28T05:52:13.745 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-05-28T05:52:13.745 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-05-28T05:52:13.745 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-05-28T05:52:13.745 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-05-28T05:52:13.745 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-05-28T05:52:13.745 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-05-28T05:52:13.745 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-05-28T05:52:13.745 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-05-28T05:52:13.745 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/cisco 2024-05-28T05:52:13.746 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/cisco/enic 2024-05-28T05:52:13.746 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-05-28T05:52:13.746 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec 2024-05-28T05:52:13.746 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/tulip 2024-05-28T05:52:13.746 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-05-28T05:52:13.746 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-05-28T05:52:13.746 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-05-28T05:52:13.746 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-05-28T05:52:13.746 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-05-28T05:52:13.746 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-05-28T05:52:13.746 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dlink 2024-05-28T05:52:13.746 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-05-28T05:52:13.746 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-05-28T05:52:13.747 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dnet.ko 2024-05-28T05:52:13.747 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/emulex 2024-05-28T05:52:13.747 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/emulex/benet 2024-05-28T05:52:13.747 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-05-28T05:52:13.747 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/ethoc.ko 2024-05-28T05:52:13.747 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/fealnx.ko 2024-05-28T05:52:13.747 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel 2024-05-28T05:52:13.747 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/e100.ko 2024-05-28T05:52:13.747 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/e1000 2024-05-28T05:52:13.747 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-05-28T05:52:13.747 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/e1000e 2024-05-28T05:52:13.747 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-05-28T05:52:13.747 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/fm10k 2024-05-28T05:52:13.747 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-05-28T05:52:13.748 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/i40e 2024-05-28T05:52:13.748 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-05-28T05:52:13.748 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/iavf 2024-05-28T05:52:13.748 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-05-28T05:52:13.748 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/igb 2024-05-28T05:52:13.748 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-05-28T05:52:13.748 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/igbvf 2024-05-28T05:52:13.748 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-05-28T05:52:13.748 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/ixgbe 2024-05-28T05:52:13.748 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-05-28T05:52:13.748 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/ixgbevf 2024-05-28T05:52:13.748 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-05-28T05:52:13.749 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/jme.ko 2024-05-28T05:52:13.749 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/marvell 2024-05-28T05:52:13.749 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-05-28T05:52:13.749 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/marvell/skge.ko 2024-05-28T05:52:13.749 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-05-28T05:52:13.749 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/mellanox 2024-05-28T05:52:13.749 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/mellanox/mlx4 2024-05-28T05:52:13.749 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-05-28T05:52:13.749 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-05-28T05:52:13.749 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/mellanox/mlx5 2024-05-28T05:52:13.749 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-05-28T05:52:13.749 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-05-28T05:52:13.749 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/micrel 2024-05-28T05:52:13.749 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-05-28T05:52:13.750 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/myricom 2024-05-28T05:52:13.750 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/myricom/myri10ge 2024-05-28T05:52:13.750 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-05-28T05:52:13.750 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/natsemi 2024-05-28T05:52:13.750 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-05-28T05:52:13.750 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-05-28T05:52:13.750 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/neterion 2024-05-28T05:52:13.750 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-05-28T05:52:13.750 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/nvidia 2024-05-28T05:52:13.750 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-05-28T05:52:13.750 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/packetengines 2024-05-28T05:52:13.750 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-05-28T05:52:13.750 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-05-28T05:52:13.751 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/qlogic 2024-05-28T05:52:13.751 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/qlogic/netxen 2024-05-28T05:52:13.751 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-05-28T05:52:13.751 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-05-28T05:52:13.751 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-05-28T05:52:13.751 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-05-28T05:52:13.751 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/rdc 2024-05-28T05:52:13.751 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-05-28T05:52:13.751 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/realtek 2024-05-28T05:52:13.751 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-05-28T05:52:13.751 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-05-28T05:52:13.751 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/realtek/atp.ko 2024-05-28T05:52:13.751 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-05-28T05:52:13.752 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/rocker 2024-05-28T05:52:13.752 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-05-28T05:52:13.752 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sfc 2024-05-28T05:52:13.752 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-05-28T05:52:13.752 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/silan 2024-05-28T05:52:13.752 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-05-28T05:52:13.752 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sis 2024-05-28T05:52:13.752 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sis/sis190.ko 2024-05-28T05:52:13.752 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sis/sis900.ko 2024-05-28T05:52:13.752 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/smsc 2024-05-28T05:52:13.752 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-05-28T05:52:13.752 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-05-28T05:52:13.752 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-05-28T05:52:13.752 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-05-28T05:52:13.753 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/stmicro 2024-05-28T05:52:13.753 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/stmicro/stmmac 2024-05-28T05:52:13.753 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-05-28T05:52:13.753 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-05-28T05:52:13.753 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sun 2024-05-28T05:52:13.753 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sun/cassini.ko 2024-05-28T05:52:13.753 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sun/niu.ko 2024-05-28T05:52:13.753 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sun/sungem.ko 2024-05-28T05:52:13.753 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-05-28T05:52:13.753 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/tehuti 2024-05-28T05:52:13.753 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-05-28T05:52:13.753 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/ti 2024-05-28T05:52:13.753 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/ti/tlan.ko 2024-05-28T05:52:13.753 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/via 2024-05-28T05:52:13.753 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-05-28T05:52:13.754 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-05-28T05:52:13.754 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/wiznet 2024-05-28T05:52:13.754 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-05-28T05:52:13.754 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-05-28T05:52:13.754 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/xircom 2024-05-28T05:52:13.754 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-05-28T05:52:13.754 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/geneve.ko 2024-05-28T05:52:13.754 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/hyperv 2024-05-28T05:52:13.754 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-05-28T05:52:13.754 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ieee802154 2024-05-28T05:52:13.754 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ieee802154/fakelb.ko 2024-05-28T05:52:13.754 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ifb.ko 2024-05-28T05:52:13.754 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ipvlan 2024-05-28T05:52:13.754 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ipvlan/ipvlan.ko 2024-05-28T05:52:13.755 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/macvlan.ko 2024-05-28T05:52:13.755 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/macvtap.ko 2024-05-28T05:52:13.755 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/mdio 2024-05-28T05:52:13.755 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/mdio.ko 2024-05-28T05:52:13.755 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-05-28T05:52:13.755 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-05-28T05:52:13.755 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/mdio/mdio-regmap.ko 2024-05-28T05:52:13.755 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/mii.ko 2024-05-28T05:52:13.755 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/net_failover.ko 2024-05-28T05:52:13.755 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/netconsole.ko 2024-05-28T05:52:13.755 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/nlmon.ko 2024-05-28T05:52:13.755 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/pcs 2024-05-28T05:52:13.755 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/pcs/pcs-lynx.ko 2024-05-28T05:52:13.756 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-05-28T05:52:13.756 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy 2024-05-28T05:52:13.756 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/amd.ko 2024-05-28T05:52:13.756 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-05-28T05:52:13.756 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-05-28T05:52:13.756 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/bcm7xxx.ko 2024-05-28T05:52:13.756 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/bcm87xx.ko 2024-05-28T05:52:13.756 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/broadcom.ko 2024-05-28T05:52:13.756 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/cicada.ko 2024-05-28T05:52:13.756 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/davicom.ko 2024-05-28T05:52:13.756 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/dp83640.ko 2024-05-28T05:52:13.756 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/et1011c.ko 2024-05-28T05:52:13.756 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/icplus.ko 2024-05-28T05:52:13.756 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/lxt.ko 2024-05-28T05:52:13.757 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/marvell.ko 2024-05-28T05:52:13.757 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/micrel.ko 2024-05-28T05:52:13.757 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/national.ko 2024-05-28T05:52:13.757 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/phylink.ko 2024-05-28T05:52:13.757 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/qsemi.ko 2024-05-28T05:52:13.757 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/realtek.ko 2024-05-28T05:52:13.757 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/smsc.ko 2024-05-28T05:52:13.757 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/ste10Xp.ko 2024-05-28T05:52:13.757 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/vitesse.ko 2024-05-28T05:52:13.757 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp 2024-05-28T05:52:13.757 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/bsd_comp.ko 2024-05-28T05:52:13.757 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/ppp_async.ko 2024-05-28T05:52:13.757 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/ppp_deflate.ko 2024-05-28T05:52:13.757 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/ppp_generic.ko 2024-05-28T05:52:13.758 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/ppp_mppe.ko 2024-05-28T05:52:13.758 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/ppp_synctty.ko 2024-05-28T05:52:13.758 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/pppoe.ko 2024-05-28T05:52:13.758 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/pppox.ko 2024-05-28T05:52:13.758 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/pptp.ko 2024-05-28T05:52:13.758 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/slip 2024-05-28T05:52:13.758 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/slip/slhc.ko 2024-05-28T05:52:13.758 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/slip/slip.ko 2024-05-28T05:52:13.758 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/sungem_phy.ko 2024-05-28T05:52:13.758 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/tap.ko 2024-05-28T05:52:13.758 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/team 2024-05-28T05:52:13.758 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/team/team.ko 2024-05-28T05:52:13.759 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/team/team_mode_activebackup.ko 2024-05-28T05:52:13.759 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/team/team_mode_broadcast.ko 2024-05-28T05:52:13.759 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-05-28T05:52:13.759 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/team/team_mode_random.ko 2024-05-28T05:52:13.759 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-05-28T05:52:13.759 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/tun.ko 2024-05-28T05:52:13.759 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/veth.ko 2024-05-28T05:52:13.759 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/virtio_net.ko 2024-05-28T05:52:13.759 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/vmxnet3 2024-05-28T05:52:13.759 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-05-28T05:52:13.759 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/vxlan 2024-05-28T05:52:13.759 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/vxlan/vxlan.ko 2024-05-28T05:52:13.759 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/xen-netback 2024-05-28T05:52:13.760 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/xen-netback/xen-netback.ko 2024-05-28T05:52:13.760 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/xen-netfront.ko 2024-05-28T05:52:13.760 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ntb 2024-05-28T05:52:13.760 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ntb/ntb.ko 2024-05-28T05:52:13.760 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/nvme 2024-05-28T05:52:13.760 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/nvme/host 2024-05-28T05:52:13.760 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/nvme/host/nvme-core.ko 2024-05-28T05:52:13.760 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/nvme/host/nvme.ko 2024-05-28T05:52:13.760 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/parport 2024-05-28T05:52:13.760 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/parport/parport.ko 2024-05-28T05:52:13.760 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/parport/parport_cs.ko 2024-05-28T05:52:13.760 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/parport/parport_pc.ko 2024-05-28T05:52:13.760 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/parport/parport_serial.ko 2024-05-28T05:52:13.760 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pci 2024-05-28T05:52:13.761 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pci/hotplug 2024-05-28T05:52:13.761 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-05-28T05:52:13.761 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pci/pcie 2024-05-28T05:52:13.761 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pci/pcie/aer_inject.ko 2024-05-28T05:52:13.761 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pci/xen-pcifront.ko 2024-05-28T05:52:13.761 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pcmcia 2024-05-28T05:52:13.761 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pcmcia/i82092.ko 2024-05-28T05:52:13.761 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pcmcia/pd6729.ko 2024-05-28T05:52:13.761 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pcmcia/yenta_socket.ko 2024-05-28T05:52:13.761 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pinctrl 2024-05-28T05:52:13.761 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pinctrl/intel 2024-05-28T05:52:13.761 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-05-28T05:52:13.762 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform 2024-05-28T05:52:13.762 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86 2024-05-28T05:52:13.762 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/acer-wmi.ko 2024-05-28T05:52:13.762 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/acerhdf.ko 2024-05-28T05:52:13.762 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/apple-gmux.ko 2024-05-28T05:52:13.762 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/asus-laptop.ko 2024-05-28T05:52:13.762 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-05-28T05:52:13.762 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/asus-wmi.ko 2024-05-28T05:52:13.762 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/classmate-laptop.ko 2024-05-28T05:52:13.762 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-05-28T05:52:13.762 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-05-28T05:52:13.762 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-05-28T05:52:13.763 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-05-28T05:52:13.763 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/hdaps.ko 2024-05-28T05:52:13.763 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/intel 2024-05-28T05:52:13.763 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-05-28T05:52:13.763 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/intel_ips.ko 2024-05-28T05:52:13.763 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/msi-wmi.ko 2024-05-28T05:52:13.763 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/mxm-wmi.ko 2024-05-28T05:52:13.763 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-05-28T05:52:13.763 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/samsung-laptop.ko 2024-05-28T05:52:13.763 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/samsung-q10.ko 2024-05-28T05:52:13.763 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/topstar-laptop.ko 2024-05-28T05:52:13.763 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-05-28T05:52:13.763 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/toshiba_haps.ko 2024-05-28T05:52:13.763 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/wmi-bmof.ko 2024-05-28T05:52:13.764 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/wmi.ko 2024-05-28T05:52:13.764 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/powercap 2024-05-28T05:52:13.764 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/powercap/intel_rapl_common.ko 2024-05-28T05:52:13.764 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/powercap/intel_rapl_msr.ko 2024-05-28T05:52:13.764 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pps 2024-05-28T05:52:13.764 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pps/clients 2024-05-28T05:52:13.764 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pps/clients/pps-gpio.ko 2024-05-28T05:52:13.764 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pps/clients/pps-ldisc.ko 2024-05-28T05:52:13.764 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pps/clients/pps_parport.ko 2024-05-28T05:52:13.764 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pps/pps_core.ko 2024-05-28T05:52:13.764 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ptp 2024-05-28T05:52:13.764 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ptp/ptp.ko 2024-05-28T05:52:13.764 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ptp/ptp_kvm.ko 2024-05-28T05:52:13.765 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ras 2024-05-28T05:52:13.765 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ras/amd 2024-05-28T05:52:13.765 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ras/amd/atl 2024-05-28T05:52:13.765 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ras/amd/atl/amd_atl.ko 2024-05-28T05:52:13.765 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ras/amd/fmpm.ko 2024-05-28T05:52:13.765 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc 2024-05-28T05:52:13.765 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-bq32k.ko 2024-05-28T05:52:13.765 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds1286.ko 2024-05-28T05:52:13.765 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds1307.ko 2024-05-28T05:52:13.765 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds1374.ko 2024-05-28T05:52:13.765 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds1511.ko 2024-05-28T05:52:13.765 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds1553.ko 2024-05-28T05:52:13.765 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds1672.ko 2024-05-28T05:52:13.765 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds1742.ko 2024-05-28T05:52:13.766 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds2404.ko 2024-05-28T05:52:13.766 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds3232.ko 2024-05-28T05:52:13.766 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-em3027.ko 2024-05-28T05:52:13.766 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-fm3130.ko 2024-05-28T05:52:13.766 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-isl12022.ko 2024-05-28T05:52:13.766 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-isl1208.ko 2024-05-28T05:52:13.766 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-m41t80.ko 2024-05-28T05:52:13.766 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-m48t35.ko 2024-05-28T05:52:13.766 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-m48t59.ko 2024-05-28T05:52:13.766 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-max6900.ko 2024-05-28T05:52:13.766 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-msm6242.ko 2024-05-28T05:52:13.766 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-pcf2127.ko 2024-05-28T05:52:13.766 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-pcf85063.ko 2024-05-28T05:52:13.767 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-pcf8523.ko 2024-05-28T05:52:13.767 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-pcf8563.ko 2024-05-28T05:52:13.767 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-pcf8583.ko 2024-05-28T05:52:13.767 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-rp5c01.ko 2024-05-28T05:52:13.767 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-rs5c372.ko 2024-05-28T05:52:13.767 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-05-28T05:52:13.767 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-rx8025.ko 2024-05-28T05:52:13.767 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-rx8581.ko 2024-05-28T05:52:13.767 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-05-28T05:52:13.767 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-x1205.ko 2024-05-28T05:52:13.767 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi 2024-05-28T05:52:13.767 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/3w-9xxx.ko 2024-05-28T05:52:13.767 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/3w-sas.ko 2024-05-28T05:52:13.767 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/3w-xxxx.ko 2024-05-28T05:52:13.768 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/BusLogic.ko 2024-05-28T05:52:13.768 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/a100u2w.ko 2024-05-28T05:52:13.768 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/aacraid 2024-05-28T05:52:13.768 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/aacraid/aacraid.ko 2024-05-28T05:52:13.768 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/advansys.ko 2024-05-28T05:52:13.768 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/aic7xxx 2024-05-28T05:52:13.768 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-05-28T05:52:13.768 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-05-28T05:52:13.768 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/am53c974.ko 2024-05-28T05:52:13.768 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/arcmsr 2024-05-28T05:52:13.768 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-05-28T05:52:13.768 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/atp870u.ko 2024-05-28T05:52:13.768 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/be2iscsi 2024-05-28T05:52:13.769 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-05-28T05:52:13.769 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/bfa 2024-05-28T05:52:13.769 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/bfa/bfa.ko 2024-05-28T05:52:13.769 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/bnx2fc 2024-05-28T05:52:13.769 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-05-28T05:52:13.769 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/bnx2i 2024-05-28T05:52:13.769 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-05-28T05:52:13.769 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/ch.ko 2024-05-28T05:52:13.769 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/csiostor 2024-05-28T05:52:13.769 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/csiostor/csiostor.ko 2024-05-28T05:52:13.769 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/cxgbi 2024-05-28T05:52:13.769 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/cxgbi/cxgb3i 2024-05-28T05:52:13.769 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-05-28T05:52:13.769 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/cxgbi/cxgb4i 2024-05-28T05:52:13.770 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-05-28T05:52:13.770 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-05-28T05:52:13.770 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/dc395x.ko 2024-05-28T05:52:13.770 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/device_handler 2024-05-28T05:52:13.770 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-05-28T05:52:13.770 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-05-28T05:52:13.770 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-05-28T05:52:13.770 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-05-28T05:52:13.770 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/dmx3191d.ko 2024-05-28T05:52:13.770 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/esas2r 2024-05-28T05:52:13.770 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/esas2r/esas2r.ko 2024-05-28T05:52:13.770 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/esp_scsi.ko 2024-05-28T05:52:13.770 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/fcoe 2024-05-28T05:52:13.771 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/fcoe/fcoe.ko 2024-05-28T05:52:13.771 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-05-28T05:52:13.771 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/fnic 2024-05-28T05:52:13.771 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/fnic/fnic.ko 2024-05-28T05:52:13.771 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/hpsa.ko 2024-05-28T05:52:13.771 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/hptiop.ko 2024-05-28T05:52:13.771 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/hv_storvsc.ko 2024-05-28T05:52:13.771 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/initio.ko 2024-05-28T05:52:13.771 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/ipr.ko 2024-05-28T05:52:13.771 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/ips.ko 2024-05-28T05:52:13.771 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/isci 2024-05-28T05:52:13.771 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/isci/isci.ko 2024-05-28T05:52:13.771 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-05-28T05:52:13.772 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/iscsi_tcp.ko 2024-05-28T05:52:13.772 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/libfc 2024-05-28T05:52:13.772 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/libfc/libfc.ko 2024-05-28T05:52:13.772 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/libiscsi.ko 2024-05-28T05:52:13.772 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/libiscsi_tcp.ko 2024-05-28T05:52:13.772 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/libsas 2024-05-28T05:52:13.772 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/libsas/libsas.ko 2024-05-28T05:52:13.772 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/lpfc 2024-05-28T05:52:13.772 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/lpfc/lpfc.ko 2024-05-28T05:52:13.772 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/megaraid 2024-05-28T05:52:13.772 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/megaraid.ko 2024-05-28T05:52:13.772 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-05-28T05:52:13.772 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-05-28T05:52:13.772 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-05-28T05:52:13.772 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/mpt3sas 2024-05-28T05:52:13.773 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-05-28T05:52:13.773 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/mvsas 2024-05-28T05:52:13.773 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/mvsas/mvsas.ko 2024-05-28T05:52:13.773 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/mvumi.ko 2024-05-28T05:52:13.773 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/pm8001 2024-05-28T05:52:13.773 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-05-28T05:52:13.773 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/pmcraid.ko 2024-05-28T05:52:13.773 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/qla1280.ko 2024-05-28T05:52:13.773 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/qla2xxx 2024-05-28T05:52:13.773 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-05-28T05:52:13.773 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-05-28T05:52:13.773 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/qla4xxx 2024-05-28T05:52:13.773 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-05-28T05:52:13.773 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/raid_class.ko 2024-05-28T05:52:13.774 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/scsi_debug.ko 2024-05-28T05:52:13.774 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/scsi_transport_fc.ko 2024-05-28T05:52:13.774 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-05-28T05:52:13.774 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/scsi_transport_sas.ko 2024-05-28T05:52:13.774 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/scsi_transport_spi.ko 2024-05-28T05:52:13.774 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/scsi_transport_srp.ko 2024-05-28T05:52:13.774 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/ses.ko 2024-05-28T05:52:13.774 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/st.ko 2024-05-28T05:52:13.774 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/stex.ko 2024-05-28T05:52:13.774 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/sym53c8xx_2 2024-05-28T05:52:13.774 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-05-28T05:52:13.774 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/virtio_scsi.ko 2024-05-28T05:52:13.774 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/vmw_pvscsi.ko 2024-05-28T05:52:13.775 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/wd719x.ko 2024-05-28T05:52:13.775 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ssb 2024-05-28T05:52:13.775 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ssb/ssb.ko 2024-05-28T05:52:13.775 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target 2024-05-28T05:52:13.775 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/iscsi 2024-05-28T05:52:13.775 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-05-28T05:52:13.775 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/loopback 2024-05-28T05:52:13.775 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/loopback/tcm_loop.ko 2024-05-28T05:52:13.775 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/target_core_file.ko 2024-05-28T05:52:13.775 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/target_core_iblock.ko 2024-05-28T05:52:13.775 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/target_core_mod.ko 2024-05-28T05:52:13.775 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/target_core_pscsi.ko 2024-05-28T05:52:13.775 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/target_core_user.ko 2024-05-28T05:52:13.776 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/tcm_fc 2024-05-28T05:52:13.776 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-05-28T05:52:13.776 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal 2024-05-28T05:52:13.776 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel 2024-05-28T05:52:13.776 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal 2024-05-28T05:52:13.776 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-05-28T05:52:13.776 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-05-28T05:52:13.776 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-05-28T05:52:13.776 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-05-28T05:52:13.776 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-05-28T05:52:13.776 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-05-28T05:52:13.776 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-05-28T05:52:13.776 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-05-28T05:52:13.777 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-05-28T05:52:13.777 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-05-28T05:52:13.777 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-05-28T05:52:13.777 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-05-28T05:52:13.777 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-05-28T05:52:13.777 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-05-28T05:52:13.777 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-05-28T05:52:13.777 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-05-28T05:52:13.777 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-05-28T05:52:13.777 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-05-28T05:52:13.777 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty 2024-05-28T05:52:13.777 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/ipwireless 2024-05-28T05:52:13.777 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-05-28T05:52:13.777 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/n_gsm.ko 2024-05-28T05:52:13.778 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/n_hdlc.ko 2024-05-28T05:52:13.778 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/nozomi.ko 2024-05-28T05:52:13.778 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial 2024-05-28T05:52:13.778 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial/8250 2024-05-28T05:52:13.778 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-05-28T05:52:13.778 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial/arc_uart.ko 2024-05-28T05:52:13.778 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial/jsm 2024-05-28T05:52:13.778 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial/jsm/jsm.ko 2024-05-28T05:52:13.778 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ufs 2024-05-28T05:52:13.778 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ufs/core 2024-05-28T05:52:13.778 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ufs/core/ufshcd-core.ko 2024-05-28T05:52:13.778 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ufs/host 2024-05-28T05:52:13.779 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-05-28T05:52:13.779 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/uio 2024-05-28T05:52:13.779 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/uio/uio.ko 2024-05-28T05:52:13.779 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/uio/uio_aec.ko 2024-05-28T05:52:13.779 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/uio/uio_cif.ko 2024-05-28T05:52:13.779 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/uio/uio_pci_generic.ko 2024-05-28T05:52:13.779 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/uio/uio_sercos3.ko 2024-05-28T05:52:13.779 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb 2024-05-28T05:52:13.779 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/class 2024-05-28T05:52:13.779 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/class/cdc-acm.ko 2024-05-28T05:52:13.779 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/class/cdc-wdm.ko 2024-05-28T05:52:13.779 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/class/usblp.ko 2024-05-28T05:52:13.779 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/class/usbtmc.ko 2024-05-28T05:52:13.779 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/host 2024-05-28T05:52:13.780 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/host/sl811-hcd.ko 2024-05-28T05:52:13.780 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/image 2024-05-28T05:52:13.780 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/image/mdc800.ko 2024-05-28T05:52:13.780 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/image/microtek.ko 2024-05-28T05:52:13.780 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc 2024-05-28T05:52:13.780 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/adutux.ko 2024-05-28T05:52:13.780 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/appledisplay.ko 2024-05-28T05:52:13.780 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/emi26.ko 2024-05-28T05:52:13.780 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/emi62.ko 2024-05-28T05:52:13.780 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/ezusb.ko 2024-05-28T05:52:13.780 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/idmouse.ko 2024-05-28T05:52:13.780 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/iowarrior.ko 2024-05-28T05:52:13.780 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/isight_firmware.ko 2024-05-28T05:52:13.780 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/ldusb.ko 2024-05-28T05:52:13.781 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/legousbtower.ko 2024-05-28T05:52:13.781 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/sisusbvga 2024-05-28T05:52:13.781 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-05-28T05:52:13.781 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/trancevibrator.ko 2024-05-28T05:52:13.781 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/usb3503.ko 2024-05-28T05:52:13.781 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/usblcd.ko 2024-05-28T05:52:13.781 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/usbsevseg.ko 2024-05-28T05:52:13.781 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/uss720.ko 2024-05-28T05:52:13.781 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/yurex.ko 2024-05-28T05:52:13.781 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/phy 2024-05-28T05:52:13.781 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/phy/phy-generic.ko 2024-05-28T05:52:13.781 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial 2024-05-28T05:52:13.781 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/aircable.ko 2024-05-28T05:52:13.782 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ark3116.ko 2024-05-28T05:52:13.782 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/belkin_sa.ko 2024-05-28T05:52:13.782 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ch341.ko 2024-05-28T05:52:13.782 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/cp210x.ko 2024-05-28T05:52:13.782 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/cyberjack.ko 2024-05-28T05:52:13.782 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/cypress_m8.ko 2024-05-28T05:52:13.782 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/digi_acceleport.ko 2024-05-28T05:52:13.782 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/empeg.ko 2024-05-28T05:52:13.782 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ftdi_sio.ko 2024-05-28T05:52:13.782 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/garmin_gps.ko 2024-05-28T05:52:13.782 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/io_edgeport.ko 2024-05-28T05:52:13.782 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/io_ti.ko 2024-05-28T05:52:13.782 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ipaq.ko 2024-05-28T05:52:13.783 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ipw.ko 2024-05-28T05:52:13.783 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ir-usb.ko 2024-05-28T05:52:13.783 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-05-28T05:52:13.783 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/keyspan.ko 2024-05-28T05:52:13.783 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/keyspan_pda.ko 2024-05-28T05:52:13.783 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/kl5kusb105.ko 2024-05-28T05:52:13.783 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/kobil_sct.ko 2024-05-28T05:52:13.783 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/mct_u232.ko 2024-05-28T05:52:13.783 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/mos7720.ko 2024-05-28T05:52:13.783 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/mos7840.ko 2024-05-28T05:52:13.783 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/navman.ko 2024-05-28T05:52:13.783 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/omninet.ko 2024-05-28T05:52:13.783 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/opticon.ko 2024-05-28T05:52:13.783 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/option.ko 2024-05-28T05:52:13.783 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/oti6858.ko 2024-05-28T05:52:13.783 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/pl2303.ko 2024-05-28T05:52:13.784 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/qcaux.ko 2024-05-28T05:52:13.784 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/qcserial.ko 2024-05-28T05:52:13.784 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/quatech2.ko 2024-05-28T05:52:13.784 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/safe_serial.ko 2024-05-28T05:52:13.784 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/sierra.ko 2024-05-28T05:52:13.784 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/spcp8x5.ko 2024-05-28T05:52:13.784 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ssu100.ko 2024-05-28T05:52:13.784 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/symbolserial.ko 2024-05-28T05:52:13.784 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-05-28T05:52:13.784 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-05-28T05:52:13.784 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/usb_debug.ko 2024-05-28T05:52:13.784 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/usb_wwan.ko 2024-05-28T05:52:13.784 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/visor.ko 2024-05-28T05:52:13.784 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/whiteheat.ko 2024-05-28T05:52:13.784 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/xsens_mt.ko 2024-05-28T05:52:13.784 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage 2024-05-28T05:52:13.785 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/uas.ko 2024-05-28T05:52:13.785 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-alauda.ko 2024-05-28T05:52:13.785 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-cypress.ko 2024-05-28T05:52:13.785 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-datafab.ko 2024-05-28T05:52:13.785 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-05-28T05:52:13.785 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-freecom.ko 2024-05-28T05:52:13.785 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-isd200.ko 2024-05-28T05:52:13.785 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-05-28T05:52:13.785 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-karma.ko 2024-05-28T05:52:13.785 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-onetouch.ko 2024-05-28T05:52:13.785 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-realtek.ko 2024-05-28T05:52:13.785 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-sddr09.ko 2024-05-28T05:52:13.785 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-sddr55.ko 2024-05-28T05:52:13.785 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-usbat.ko 2024-05-28T05:52:13.786 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/usb-storage.ko 2024-05-28T05:52:13.786 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/usbip 2024-05-28T05:52:13.786 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/usbip/usbip-core.ko 2024-05-28T05:52:13.786 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/usbip/usbip-host.ko 2024-05-28T05:52:13.786 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-05-28T05:52:13.786 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vfio 2024-05-28T05:52:13.786 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vfio/pci 2024-05-28T05:52:13.786 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-05-28T05:52:13.786 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vfio/pci/vfio-pci.ko 2024-05-28T05:52:13.786 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vfio/vfio.ko 2024-05-28T05:52:13.786 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-05-28T05:52:13.786 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vhost 2024-05-28T05:52:13.787 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vhost/vhost.ko 2024-05-28T05:52:13.787 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vhost/vhost_iotlb.ko 2024-05-28T05:52:13.787 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vhost/vhost_net.ko 2024-05-28T05:52:13.787 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vhost/vhost_scsi.ko 2024-05-28T05:52:13.787 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video 2024-05-28T05:52:13.787 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight 2024-05-28T05:52:13.787 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/apple_bl.ko 2024-05-28T05:52:13.787 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/lcd.ko 2024-05-28T05:52:13.787 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/platform_lcd.ko 2024-05-28T05:52:13.787 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/virtio 2024-05-28T05:52:13.787 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/virtio/virtio.ko 2024-05-28T05:52:13.787 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/virtio/virtio_balloon.ko 2024-05-28T05:52:13.787 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/virtio/virtio_mmio.ko 2024-05-28T05:52:13.788 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/virtio/virtio_pci.ko 2024-05-28T05:52:13.788 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-05-28T05:52:13.788 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-05-28T05:52:13.788 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/virtio/virtio_ring.ko 2024-05-28T05:52:13.788 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1 2024-05-28T05:52:13.788 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/masters 2024-05-28T05:52:13.788 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/masters/ds2482.ko 2024-05-28T05:52:13.788 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/masters/ds2490.ko 2024-05-28T05:52:13.788 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves 2024-05-28T05:52:13.788 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-05-28T05:52:13.788 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-05-28T05:52:13.788 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-05-28T05:52:13.789 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-05-28T05:52:13.789 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-05-28T05:52:13.789 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-05-28T05:52:13.789 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-05-28T05:52:13.789 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-05-28T05:52:13.789 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-05-28T05:52:13.789 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_smem.ko 2024-05-28T05:52:13.789 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_therm.ko 2024-05-28T05:52:13.789 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/wire.ko 2024-05-28T05:52:13.789 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog 2024-05-28T05:52:13.789 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/alim1535_wdt.ko 2024-05-28T05:52:13.789 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/alim7101_wdt.ko 2024-05-28T05:52:13.789 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/f71808e_wdt.ko 2024-05-28T05:52:13.790 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/hpwdt.ko 2024-05-28T05:52:13.790 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/i6300esb.ko 2024-05-28T05:52:13.790 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-05-28T05:52:13.790 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/iTCO_wdt.ko 2024-05-28T05:52:13.790 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/ib700wdt.ko 2024-05-28T05:52:13.790 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/ibmasr.ko 2024-05-28T05:52:13.790 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-05-28T05:52:13.790 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/it8712f_wdt.ko 2024-05-28T05:52:13.790 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/it87_wdt.ko 2024-05-28T05:52:13.790 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/machzwd.ko 2024-05-28T05:52:13.790 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/nv_tco.ko 2024-05-28T05:52:13.790 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/pcwd_pci.ko 2024-05-28T05:52:13.790 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/pcwd_usb.ko 2024-05-28T05:52:13.790 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-05-28T05:52:13.791 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/sch311x_wdt.ko 2024-05-28T05:52:13.791 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/softdog.ko 2024-05-28T05:52:13.791 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/sp5100_tco.ko 2024-05-28T05:52:13.791 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/via_wdt.ko 2024-05-28T05:52:13.791 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-05-28T05:52:13.791 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/w83877f_wdt.ko 2024-05-28T05:52:13.791 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/w83977f_wdt.ko 2024-05-28T05:52:13.791 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/wdt_pci.ko 2024-05-28T05:52:13.791 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/xen_wdt.ko 2024-05-28T05:52:13.791 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen 2024-05-28T05:52:13.791 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xen-acpi-processor.ko 2024-05-28T05:52:13.791 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xen-evtchn.ko 2024-05-28T05:52:13.791 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xen-gntalloc.ko 2024-05-28T05:52:13.791 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xen-gntdev.ko 2024-05-28T05:52:13.792 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xen-pciback 2024-05-28T05:52:13.792 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-05-28T05:52:13.792 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xen-privcmd.ko 2024-05-28T05:52:13.792 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xenfs 2024-05-28T05:52:13.792 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xenfs/xenfs.ko 2024-05-28T05:52:13.792 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs 2024-05-28T05:52:13.792 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/9p 2024-05-28T05:52:13.792 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/9p/9p.ko 2024-05-28T05:52:13.792 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/affs 2024-05-28T05:52:13.792 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/affs/affs.ko 2024-05-28T05:52:13.792 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/befs 2024-05-28T05:52:13.792 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/befs/befs.ko 2024-05-28T05:52:13.792 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/binfmt_misc.ko 2024-05-28T05:52:13.792 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/btrfs 2024-05-28T05:52:13.792 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/btrfs/btrfs.ko 2024-05-28T05:52:13.793 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/cachefiles 2024-05-28T05:52:13.793 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/cachefiles/cachefiles.ko 2024-05-28T05:52:13.793 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ceph 2024-05-28T05:52:13.793 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ceph/ceph.ko 2024-05-28T05:52:13.793 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/coda 2024-05-28T05:52:13.793 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/coda/coda.ko 2024-05-28T05:52:13.793 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/cramfs 2024-05-28T05:52:13.793 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/cramfs/cramfs.ko 2024-05-28T05:52:13.793 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/dlm 2024-05-28T05:52:13.793 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/dlm/dlm.ko 2024-05-28T05:52:13.793 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ecryptfs 2024-05-28T05:52:13.793 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ecryptfs/ecryptfs.ko 2024-05-28T05:52:13.793 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/f2fs 2024-05-28T05:52:13.793 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/f2fs/f2fs.ko 2024-05-28T05:52:13.793 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/fat 2024-05-28T05:52:13.794 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/fat/fat.ko 2024-05-28T05:52:13.794 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/fat/msdos.ko 2024-05-28T05:52:13.794 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/fat/vfat.ko 2024-05-28T05:52:13.794 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/fuse 2024-05-28T05:52:13.794 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/fuse/cuse.ko 2024-05-28T05:52:13.794 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/fuse/fuse.ko 2024-05-28T05:52:13.794 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/gfs2 2024-05-28T05:52:13.794 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/gfs2/gfs2.ko 2024-05-28T05:52:13.794 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/hfs 2024-05-28T05:52:13.794 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/hfs/hfs.ko 2024-05-28T05:52:13.794 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/hfsplus 2024-05-28T05:52:13.794 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/hfsplus/hfsplus.ko 2024-05-28T05:52:13.794 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/isofs 2024-05-28T05:52:13.794 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/isofs/isofs.ko 2024-05-28T05:52:13.795 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/jfs 2024-05-28T05:52:13.795 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/jfs/jfs.ko 2024-05-28T05:52:13.795 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/lockd 2024-05-28T05:52:13.795 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/lockd/lockd.ko 2024-05-28T05:52:13.795 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/minix 2024-05-28T05:52:13.795 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/minix/minix.ko 2024-05-28T05:52:13.795 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/netfs 2024-05-28T05:52:13.795 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/netfs/netfs.ko 2024-05-28T05:52:13.795 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs 2024-05-28T05:52:13.795 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/blocklayout 2024-05-28T05:52:13.795 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-05-28T05:52:13.795 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/filelayout 2024-05-28T05:52:13.795 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-05-28T05:52:13.795 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/flexfilelayout 2024-05-28T05:52:13.795 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-05-28T05:52:13.795 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/nfs.ko 2024-05-28T05:52:13.796 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/nfsv3.ko 2024-05-28T05:52:13.796 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/nfsv4.ko 2024-05-28T05:52:13.796 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs_common 2024-05-28T05:52:13.796 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs_common/grace.ko 2024-05-28T05:52:13.796 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs_common/nfs_acl.ko 2024-05-28T05:52:13.796 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfsd 2024-05-28T05:52:13.796 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfsd/nfsd.ko 2024-05-28T05:52:13.796 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nilfs2 2024-05-28T05:52:13.796 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nilfs2/nilfs2.ko 2024-05-28T05:52:13.796 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls 2024-05-28T05:52:13.796 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-celtic.ko 2024-05-28T05:52:13.796 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-centeuro.ko 2024-05-28T05:52:13.796 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-croatian.ko 2024-05-28T05:52:13.796 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-cyrillic.ko 2024-05-28T05:52:13.796 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-gaelic.ko 2024-05-28T05:52:13.797 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-greek.ko 2024-05-28T05:52:13.797 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-iceland.ko 2024-05-28T05:52:13.797 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-inuit.ko 2024-05-28T05:52:13.797 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-roman.ko 2024-05-28T05:52:13.797 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-romanian.ko 2024-05-28T05:52:13.797 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-turkish.ko 2024-05-28T05:52:13.797 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp1250.ko 2024-05-28T05:52:13.797 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp1251.ko 2024-05-28T05:52:13.797 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp1255.ko 2024-05-28T05:52:13.797 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp737.ko 2024-05-28T05:52:13.797 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp775.ko 2024-05-28T05:52:13.797 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp850.ko 2024-05-28T05:52:13.797 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp852.ko 2024-05-28T05:52:13.797 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp855.ko 2024-05-28T05:52:13.797 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp857.ko 2024-05-28T05:52:13.797 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp860.ko 2024-05-28T05:52:13.798 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp861.ko 2024-05-28T05:52:13.798 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp862.ko 2024-05-28T05:52:13.798 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp863.ko 2024-05-28T05:52:13.798 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp864.ko 2024-05-28T05:52:13.798 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp865.ko 2024-05-28T05:52:13.798 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp866.ko 2024-05-28T05:52:13.798 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp869.ko 2024-05-28T05:52:13.798 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp874.ko 2024-05-28T05:52:13.798 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp932.ko 2024-05-28T05:52:13.798 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp936.ko 2024-05-28T05:52:13.798 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp949.ko 2024-05-28T05:52:13.798 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp950.ko 2024-05-28T05:52:13.798 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_euc-jp.ko 2024-05-28T05:52:13.799 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-1.ko 2024-05-28T05:52:13.799 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-13.ko 2024-05-28T05:52:13.799 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-14.ko 2024-05-28T05:52:13.799 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-15.ko 2024-05-28T05:52:13.799 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-2.ko 2024-05-28T05:52:13.799 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-3.ko 2024-05-28T05:52:13.799 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-4.ko 2024-05-28T05:52:13.799 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-5.ko 2024-05-28T05:52:13.799 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-6.ko 2024-05-28T05:52:13.799 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-7.ko 2024-05-28T05:52:13.799 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-9.ko 2024-05-28T05:52:13.799 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_koi8-r.ko 2024-05-28T05:52:13.799 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_koi8-ru.ko 2024-05-28T05:52:13.800 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_koi8-u.ko 2024-05-28T05:52:13.800 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_ucs2_utils.ko 2024-05-28T05:52:13.800 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_utf8.ko 2024-05-28T05:52:13.800 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2 2024-05-28T05:52:13.800 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/cluster 2024-05-28T05:52:13.800 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-05-28T05:52:13.800 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/dlm 2024-05-28T05:52:13.800 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-05-28T05:52:13.800 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/dlmfs 2024-05-28T05:52:13.800 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-05-28T05:52:13.800 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/ocfs2.ko 2024-05-28T05:52:13.800 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-05-28T05:52:13.800 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-05-28T05:52:13.801 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-05-28T05:52:13.801 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/overlayfs 2024-05-28T05:52:13.801 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/overlayfs/overlay.ko 2024-05-28T05:52:13.801 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/pstore 2024-05-28T05:52:13.801 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/pstore/ramoops.ko 2024-05-28T05:52:13.801 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/reiserfs 2024-05-28T05:52:13.801 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/reiserfs/reiserfs.ko 2024-05-28T05:52:13.801 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/romfs 2024-05-28T05:52:13.801 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/romfs/romfs.ko 2024-05-28T05:52:13.801 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/smb 2024-05-28T05:52:13.801 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/smb/client 2024-05-28T05:52:13.801 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/smb/client/cifs.ko 2024-05-28T05:52:13.801 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/smb/common 2024-05-28T05:52:13.802 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/smb/common/cifs_arc4.ko 2024-05-28T05:52:13.802 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/smb/common/cifs_md4.ko 2024-05-28T05:52:13.802 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/squashfs 2024-05-28T05:52:13.802 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/squashfs/squashfs.ko 2024-05-28T05:52:13.802 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/sysv 2024-05-28T05:52:13.802 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/sysv/sysv.ko 2024-05-28T05:52:13.802 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/udf 2024-05-28T05:52:13.802 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/udf/udf.ko 2024-05-28T05:52:13.802 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ufs 2024-05-28T05:52:13.802 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ufs/ufs.ko 2024-05-28T05:52:13.802 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/xfs 2024-05-28T05:52:13.802 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/xfs/xfs.ko 2024-05-28T05:52:13.802 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/kernel 2024-05-28T05:52:13.803 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/kernel/rcu 2024-05-28T05:52:13.803 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/kernel/rcu/rcutorture.ko 2024-05-28T05:52:13.803 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/kernel/torture.ko 2024-05-28T05:52:13.803 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/kernel/trace 2024-05-28T05:52:13.803 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-05-28T05:52:13.803 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib 2024-05-28T05:52:13.803 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/asn1_encoder.ko 2024-05-28T05:52:13.803 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/crc-itu-t.ko 2024-05-28T05:52:13.803 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/crc8.ko 2024-05-28T05:52:13.803 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/crypto 2024-05-28T05:52:13.803 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/crypto/libarc4.ko 2024-05-28T05:52:13.803 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/crypto/libdes.ko 2024-05-28T05:52:13.804 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/libcrc32c.ko 2024-05-28T05:52:13.804 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/lru_cache.ko 2024-05-28T05:52:13.804 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/lz4 2024-05-28T05:52:13.804 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/lz4/lz4_compress.ko 2024-05-28T05:52:13.804 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/lz4/lz4hc_compress.ko 2024-05-28T05:52:13.804 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/math 2024-05-28T05:52:13.804 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/math/cordic.ko 2024-05-28T05:52:13.804 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/raid6 2024-05-28T05:52:13.804 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/raid6/raid6_pq.ko 2024-05-28T05:52:13.804 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/reed_solomon 2024-05-28T05:52:13.804 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/reed_solomon/reed_solomon.ko 2024-05-28T05:52:13.804 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/ts_bm.ko 2024-05-28T05:52:13.804 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/ts_fsm.ko 2024-05-28T05:52:13.805 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/ts_kmp.ko 2024-05-28T05:52:13.805 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/zstd 2024-05-28T05:52:13.805 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/zstd/zstd_compress.ko 2024-05-28T05:52:13.805 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/mm 2024-05-28T05:52:13.805 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/mm/hwpoison-inject.ko 2024-05-28T05:52:13.805 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net 2024-05-28T05:52:13.805 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan 2024-05-28T05:52:13.805 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/6lowpan.ko 2024-05-28T05:52:13.805 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/nhc_dest.ko 2024-05-28T05:52:13.805 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/nhc_fragment.ko 2024-05-28T05:52:13.805 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/nhc_hop.ko 2024-05-28T05:52:13.805 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/nhc_ipv6.ko 2024-05-28T05:52:13.805 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/nhc_mobility.ko 2024-05-28T05:52:13.806 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/nhc_routing.ko 2024-05-28T05:52:13.806 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/nhc_udp.ko 2024-05-28T05:52:13.806 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/802 2024-05-28T05:52:13.806 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/802/garp.ko 2024-05-28T05:52:13.806 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/802/mrp.ko 2024-05-28T05:52:13.806 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/802/p8022.ko 2024-05-28T05:52:13.806 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/802/psnap.ko 2024-05-28T05:52:13.806 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/802/stp.ko 2024-05-28T05:52:13.806 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/8021q 2024-05-28T05:52:13.806 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/8021q/8021q.ko 2024-05-28T05:52:13.806 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/9p 2024-05-28T05:52:13.806 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/9p/9pnet.ko 2024-05-28T05:52:13.806 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/9p/9pnet_fd.ko 2024-05-28T05:52:13.806 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/9p/9pnet_rdma.ko 2024-05-28T05:52:13.807 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/9p/9pnet_virtio.ko 2024-05-28T05:52:13.807 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/batman-adv 2024-05-28T05:52:13.807 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/batman-adv/batman-adv.ko 2024-05-28T05:52:13.807 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge 2024-05-28T05:52:13.807 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/br_netfilter.ko 2024-05-28T05:52:13.807 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/bridge.ko 2024-05-28T05:52:13.807 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter 2024-05-28T05:52:13.807 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-05-28T05:52:13.807 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_among.ko 2024-05-28T05:52:13.807 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_arp.ko 2024-05-28T05:52:13.807 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-05-28T05:52:13.807 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-05-28T05:52:13.807 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_ip.ko 2024-05-28T05:52:13.807 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-05-28T05:52:13.808 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_limit.ko 2024-05-28T05:52:13.808 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_log.ko 2024-05-28T05:52:13.808 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_mark.ko 2024-05-28T05:52:13.808 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-05-28T05:52:13.808 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-05-28T05:52:13.808 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-05-28T05:52:13.808 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-05-28T05:52:13.808 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_snat.ko 2024-05-28T05:52:13.808 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_stp.ko 2024-05-28T05:52:13.808 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-05-28T05:52:13.808 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-05-28T05:52:13.808 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-05-28T05:52:13.808 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-05-28T05:52:13.808 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebtables.ko 2024-05-28T05:52:13.809 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-05-28T05:52:13.809 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-05-28T05:52:13.809 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-05-28T05:52:13.809 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ceph 2024-05-28T05:52:13.809 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ceph/libceph.ko 2024-05-28T05:52:13.809 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/core 2024-05-28T05:52:13.809 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/core/failover.ko 2024-05-28T05:52:13.809 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/core/pktgen.ko 2024-05-28T05:52:13.809 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dccp 2024-05-28T05:52:13.809 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dccp/dccp.ko 2024-05-28T05:52:13.809 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dccp/dccp_diag.ko 2024-05-28T05:52:13.809 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dccp/dccp_ipv4.ko 2024-05-28T05:52:13.809 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dccp/dccp_ipv6.ko 2024-05-28T05:52:13.809 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dns_resolver 2024-05-28T05:52:13.809 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dns_resolver/dns_resolver.ko 2024-05-28T05:52:13.810 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dsa 2024-05-28T05:52:13.810 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dsa/dsa_core.ko 2024-05-28T05:52:13.810 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dsa/tag_brcm.ko 2024-05-28T05:52:13.810 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dsa/tag_dsa.ko 2024-05-28T05:52:13.810 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dsa/tag_none.ko 2024-05-28T05:52:13.810 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dsa/tag_trailer.ko 2024-05-28T05:52:13.810 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ieee802154 2024-05-28T05:52:13.810 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ieee802154/6lowpan 2024-05-28T05:52:13.810 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-05-28T05:52:13.810 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ieee802154/ieee802154.ko 2024-05-28T05:52:13.810 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ieee802154/ieee802154_socket.ko 2024-05-28T05:52:13.810 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4 2024-05-28T05:52:13.810 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/ah4.ko 2024-05-28T05:52:13.811 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/esp4.ko 2024-05-28T05:52:13.811 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/fou.ko 2024-05-28T05:52:13.811 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/gre.ko 2024-05-28T05:52:13.811 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/inet_diag.ko 2024-05-28T05:52:13.811 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/ip_gre.ko 2024-05-28T05:52:13.811 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/ip_tunnel.ko 2024-05-28T05:52:13.811 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/ip_vti.ko 2024-05-28T05:52:13.811 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/ipcomp.ko 2024-05-28T05:52:13.811 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/ipip.ko 2024-05-28T05:52:13.811 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter 2024-05-28T05:52:13.811 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/arp_tables.ko 2024-05-28T05:52:13.811 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-05-28T05:52:13.811 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-05-28T05:52:13.811 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-05-28T05:52:13.811 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-05-28T05:52:13.812 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-05-28T05:52:13.812 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-05-28T05:52:13.812 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-05-28T05:52:13.812 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-05-28T05:52:13.812 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-05-28T05:52:13.812 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/iptable_security.ko 2024-05-28T05:52:13.812 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-05-28T05:52:13.812 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-05-28T05:52:13.812 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-05-28T05:52:13.812 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-05-28T05:52:13.812 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-05-28T05:52:13.812 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-05-28T05:52:13.812 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-05-28T05:52:13.813 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-05-28T05:52:13.813 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-05-28T05:52:13.813 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-05-28T05:52:13.813 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_bic.ko 2024-05-28T05:52:13.813 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_dctcp.ko 2024-05-28T05:52:13.813 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_diag.ko 2024-05-28T05:52:13.813 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_highspeed.ko 2024-05-28T05:52:13.813 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_htcp.ko 2024-05-28T05:52:13.813 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_hybla.ko 2024-05-28T05:52:13.813 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_illinois.ko 2024-05-28T05:52:13.813 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_lp.ko 2024-05-28T05:52:13.813 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_scalable.ko 2024-05-28T05:52:13.813 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_vegas.ko 2024-05-28T05:52:13.814 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_veno.ko 2024-05-28T05:52:13.814 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_westwood.ko 2024-05-28T05:52:13.814 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_yeah.ko 2024-05-28T05:52:13.814 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tunnel4.ko 2024-05-28T05:52:13.814 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/udp_diag.ko 2024-05-28T05:52:13.814 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/udp_tunnel.ko 2024-05-28T05:52:13.814 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/xfrm4_tunnel.ko 2024-05-28T05:52:13.814 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6 2024-05-28T05:52:13.814 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/ah6.ko 2024-05-28T05:52:13.814 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/esp6.ko 2024-05-28T05:52:13.814 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/fou6.ko 2024-05-28T05:52:13.814 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/ip6_tunnel.ko 2024-05-28T05:52:13.814 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-05-28T05:52:13.815 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/ip6_vti.ko 2024-05-28T05:52:13.815 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/ipcomp6.ko 2024-05-28T05:52:13.815 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter 2024-05-28T05:52:13.815 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-05-28T05:52:13.815 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-05-28T05:52:13.815 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-05-28T05:52:13.815 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-05-28T05:52:13.815 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-05-28T05:52:13.815 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-05-28T05:52:13.815 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-05-28T05:52:13.815 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-05-28T05:52:13.815 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-05-28T05:52:13.816 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-05-28T05:52:13.816 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-05-28T05:52:13.816 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-05-28T05:52:13.816 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-05-28T05:52:13.816 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-05-28T05:52:13.816 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-05-28T05:52:13.816 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-05-28T05:52:13.816 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-05-28T05:52:13.816 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-05-28T05:52:13.816 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-05-28T05:52:13.816 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-05-28T05:52:13.816 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-05-28T05:52:13.816 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-05-28T05:52:13.816 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-05-28T05:52:13.817 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-05-28T05:52:13.817 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/sit.ko 2024-05-28T05:52:13.817 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/tunnel6.ko 2024-05-28T05:52:13.817 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/xfrm6_tunnel.ko 2024-05-28T05:52:13.817 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/key 2024-05-28T05:52:13.817 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/key/af_key.ko 2024-05-28T05:52:13.817 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp 2024-05-28T05:52:13.817 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/l2tp_core.ko 2024-05-28T05:52:13.817 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/l2tp_debugfs.ko 2024-05-28T05:52:13.817 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/l2tp_eth.ko 2024-05-28T05:52:13.817 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/l2tp_ip.ko 2024-05-28T05:52:13.817 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/l2tp_ip6.ko 2024-05-28T05:52:13.817 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/l2tp_netlink.ko 2024-05-28T05:52:13.817 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/l2tp_ppp.ko 2024-05-28T05:52:13.817 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/llc 2024-05-28T05:52:13.818 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/llc/llc.ko 2024-05-28T05:52:13.818 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/mac802154 2024-05-28T05:52:13.818 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/mac802154/mac802154.ko 2024-05-28T05:52:13.818 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/mpls 2024-05-28T05:52:13.818 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/mpls/mpls_gso.ko 2024-05-28T05:52:13.818 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter 2024-05-28T05:52:13.818 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset 2024-05-28T05:52:13.818 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set.ko 2024-05-28T05:52:13.818 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-05-28T05:52:13.818 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-05-28T05:52:13.818 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-05-28T05:52:13.818 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-05-28T05:52:13.818 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-05-28T05:52:13.818 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-05-28T05:52:13.819 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-05-28T05:52:13.819 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-05-28T05:52:13.819 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-05-28T05:52:13.819 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-05-28T05:52:13.819 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-05-28T05:52:13.819 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-05-28T05:52:13.819 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-05-28T05:52:13.819 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-05-28T05:52:13.819 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-05-28T05:52:13.819 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs 2024-05-28T05:52:13.819 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs.ko 2024-05-28T05:52:13.819 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-05-28T05:52:13.819 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-05-28T05:52:13.820 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-05-28T05:52:13.820 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-05-28T05:52:13.820 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-05-28T05:52:13.820 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-05-28T05:52:13.820 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-05-28T05:52:13.820 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-05-28T05:52:13.820 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-05-28T05:52:13.820 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-05-28T05:52:13.820 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-05-28T05:52:13.820 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-05-28T05:52:13.820 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-05-28T05:52:13.820 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conncount.ko 2024-05-28T05:52:13.820 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack.ko 2024-05-28T05:52:13.820 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-05-28T05:52:13.821 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-05-28T05:52:13.821 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-05-28T05:52:13.821 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_h323.ko 2024-05-28T05:52:13.821 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_irc.ko 2024-05-28T05:52:13.821 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-05-28T05:52:13.821 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-05-28T05:52:13.821 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-05-28T05:52:13.821 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_sane.ko 2024-05-28T05:52:13.821 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_sip.ko 2024-05-28T05:52:13.821 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-05-28T05:52:13.821 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-05-28T05:52:13.821 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_dup_netdev.ko 2024-05-28T05:52:13.821 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_flow_table.ko 2024-05-28T05:52:13.821 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_flow_table_inet.ko 2024-05-28T05:52:13.821 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_log_syslog.ko 2024-05-28T05:52:13.822 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_nat.ko 2024-05-28T05:52:13.822 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_nat_amanda.ko 2024-05-28T05:52:13.822 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_nat_ftp.ko 2024-05-28T05:52:13.822 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_nat_irc.ko 2024-05-28T05:52:13.822 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_nat_sip.ko 2024-05-28T05:52:13.822 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_nat_tftp.ko 2024-05-28T05:52:13.822 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_synproxy_core.ko 2024-05-28T05:52:13.822 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_tables.ko 2024-05-28T05:52:13.822 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nfnetlink.ko 2024-05-28T05:52:13.822 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nfnetlink_acct.ko 2024-05-28T05:52:13.822 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nfnetlink_log.ko 2024-05-28T05:52:13.822 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nfnetlink_osf.ko 2024-05-28T05:52:13.822 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nfnetlink_queue.ko 2024-05-28T05:52:13.823 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_chain_nat.ko 2024-05-28T05:52:13.823 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_compat.ko 2024-05-28T05:52:13.823 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_ct.ko 2024-05-28T05:52:13.823 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_dup_netdev.ko 2024-05-28T05:52:13.823 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_fib.ko 2024-05-28T05:52:13.823 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_fwd_netdev.ko 2024-05-28T05:52:13.823 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_hash.ko 2024-05-28T05:52:13.823 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_limit.ko 2024-05-28T05:52:13.823 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_log.ko 2024-05-28T05:52:13.823 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_masq.ko 2024-05-28T05:52:13.823 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_nat.ko 2024-05-28T05:52:13.823 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_numgen.ko 2024-05-28T05:52:13.823 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_queue.ko 2024-05-28T05:52:13.823 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_quota.ko 2024-05-28T05:52:13.823 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_redir.ko 2024-05-28T05:52:13.823 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_reject.ko 2024-05-28T05:52:13.824 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_reject_inet.ko 2024-05-28T05:52:13.824 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_socket.ko 2024-05-28T05:52:13.824 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_tunnel.ko 2024-05-28T05:52:13.824 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_AUDIT.ko 2024-05-28T05:52:13.824 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_CHECKSUM.ko 2024-05-28T05:52:13.824 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_CLASSIFY.ko 2024-05-28T05:52:13.824 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-05-28T05:52:13.824 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_CT.ko 2024-05-28T05:52:13.824 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_DSCP.ko 2024-05-28T05:52:13.824 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_HL.ko 2024-05-28T05:52:13.824 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_HMARK.ko 2024-05-28T05:52:13.824 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_IDLETIMER.ko 2024-05-28T05:52:13.824 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_LED.ko 2024-05-28T05:52:13.824 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_LOG.ko 2024-05-28T05:52:13.824 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_MASQUERADE.ko 2024-05-28T05:52:13.824 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_NETMAP.ko 2024-05-28T05:52:13.825 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_NFLOG.ko 2024-05-28T05:52:13.825 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_NFQUEUE.ko 2024-05-28T05:52:13.825 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_RATEEST.ko 2024-05-28T05:52:13.825 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_REDIRECT.ko 2024-05-28T05:52:13.825 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_SECMARK.ko 2024-05-28T05:52:13.825 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_TCPMSS.ko 2024-05-28T05:52:13.825 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-05-28T05:52:13.825 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_TEE.ko 2024-05-28T05:52:13.825 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_TPROXY.ko 2024-05-28T05:52:13.825 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_TRACE.ko 2024-05-28T05:52:13.825 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_addrtype.ko 2024-05-28T05:52:13.825 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_bpf.ko 2024-05-28T05:52:13.825 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_cgroup.ko 2024-05-28T05:52:13.826 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_cluster.ko 2024-05-28T05:52:13.826 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_comment.ko 2024-05-28T05:52:13.826 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_connbytes.ko 2024-05-28T05:52:13.826 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_connlabel.ko 2024-05-28T05:52:13.826 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_connlimit.ko 2024-05-28T05:52:13.826 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_connmark.ko 2024-05-28T05:52:13.826 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_conntrack.ko 2024-05-28T05:52:13.826 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_cpu.ko 2024-05-28T05:52:13.826 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_dccp.ko 2024-05-28T05:52:13.826 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_devgroup.ko 2024-05-28T05:52:13.826 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_dscp.ko 2024-05-28T05:52:13.826 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_ecn.ko 2024-05-28T05:52:13.826 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_esp.ko 2024-05-28T05:52:13.826 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_hashlimit.ko 2024-05-28T05:52:13.827 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_helper.ko 2024-05-28T05:52:13.827 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_hl.ko 2024-05-28T05:52:13.827 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_ipcomp.ko 2024-05-28T05:52:13.827 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_iprange.ko 2024-05-28T05:52:13.827 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_ipvs.ko 2024-05-28T05:52:13.827 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_l2tp.ko 2024-05-28T05:52:13.827 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_length.ko 2024-05-28T05:52:13.827 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_limit.ko 2024-05-28T05:52:13.827 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_mac.ko 2024-05-28T05:52:13.827 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_mark.ko 2024-05-28T05:52:13.827 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_multiport.ko 2024-05-28T05:52:13.827 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_nat.ko 2024-05-28T05:52:13.827 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_nfacct.ko 2024-05-28T05:52:13.827 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_osf.ko 2024-05-28T05:52:13.827 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_owner.ko 2024-05-28T05:52:13.828 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_physdev.ko 2024-05-28T05:52:13.828 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_pkttype.ko 2024-05-28T05:52:13.828 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_policy.ko 2024-05-28T05:52:13.828 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_quota.ko 2024-05-28T05:52:13.828 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_rateest.ko 2024-05-28T05:52:13.828 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_realm.ko 2024-05-28T05:52:13.828 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_recent.ko 2024-05-28T05:52:13.828 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_sctp.ko 2024-05-28T05:52:13.828 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_set.ko 2024-05-28T05:52:13.828 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_socket.ko 2024-05-28T05:52:13.828 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_state.ko 2024-05-28T05:52:13.828 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_statistic.ko 2024-05-28T05:52:13.828 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_string.ko 2024-05-28T05:52:13.828 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_tcpmss.ko 2024-05-28T05:52:13.828 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_time.ko 2024-05-28T05:52:13.829 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_u32.ko 2024-05-28T05:52:13.829 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netlink 2024-05-28T05:52:13.829 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netlink/netlink_diag.ko 2024-05-28T05:52:13.829 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/nsh 2024-05-28T05:52:13.829 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/nsh/nsh.ko 2024-05-28T05:52:13.829 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/openvswitch 2024-05-28T05:52:13.829 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/openvswitch/openvswitch.ko 2024-05-28T05:52:13.829 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/openvswitch/vport-geneve.ko 2024-05-28T05:52:13.829 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/openvswitch/vport-gre.ko 2024-05-28T05:52:13.829 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/openvswitch/vport-vxlan.ko 2024-05-28T05:52:13.829 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/packet 2024-05-28T05:52:13.829 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/packet/af_packet_diag.ko 2024-05-28T05:52:13.829 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/rds 2024-05-28T05:52:13.829 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/rds/rds.ko 2024-05-28T05:52:13.830 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/rds/rds_rdma.ko 2024-05-28T05:52:13.830 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/rds/rds_tcp.ko 2024-05-28T05:52:13.830 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched 2024-05-28T05:52:13.830 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_csum.ko 2024-05-28T05:52:13.830 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_gact.ko 2024-05-28T05:52:13.830 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_mirred.ko 2024-05-28T05:52:13.830 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_nat.ko 2024-05-28T05:52:13.830 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_pedit.ko 2024-05-28T05:52:13.830 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_police.ko 2024-05-28T05:52:13.830 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_simple.ko 2024-05-28T05:52:13.830 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_skbedit.ko 2024-05-28T05:52:13.830 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_vlan.ko 2024-05-28T05:52:13.830 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/cls_basic.ko 2024-05-28T05:52:13.830 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/cls_bpf.ko 2024-05-28T05:52:13.831 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/cls_flow.ko 2024-05-28T05:52:13.831 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/cls_fw.ko 2024-05-28T05:52:13.831 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/cls_route.ko 2024-05-28T05:52:13.831 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/cls_u32.ko 2024-05-28T05:52:13.831 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/em_cmp.ko 2024-05-28T05:52:13.831 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/em_ipset.ko 2024-05-28T05:52:13.831 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/em_meta.ko 2024-05-28T05:52:13.831 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/em_nbyte.ko 2024-05-28T05:52:13.831 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/em_text.ko 2024-05-28T05:52:13.831 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/em_u32.ko 2024-05-28T05:52:13.831 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_choke.ko 2024-05-28T05:52:13.831 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_codel.ko 2024-05-28T05:52:13.831 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_drr.ko 2024-05-28T05:52:13.832 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_fq.ko 2024-05-28T05:52:13.832 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_gred.ko 2024-05-28T05:52:13.832 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_hfsc.ko 2024-05-28T05:52:13.832 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_hhf.ko 2024-05-28T05:52:13.832 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_htb.ko 2024-05-28T05:52:13.832 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_ingress.ko 2024-05-28T05:52:13.832 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_mqprio.ko 2024-05-28T05:52:13.832 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_mqprio_lib.ko 2024-05-28T05:52:13.832 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_multiq.ko 2024-05-28T05:52:13.832 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_netem.ko 2024-05-28T05:52:13.832 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_pie.ko 2024-05-28T05:52:13.832 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_plug.ko 2024-05-28T05:52:13.833 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_prio.ko 2024-05-28T05:52:13.833 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_qfq.ko 2024-05-28T05:52:13.833 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_red.ko 2024-05-28T05:52:13.833 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_sfb.ko 2024-05-28T05:52:13.833 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_sfq.ko 2024-05-28T05:52:13.833 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_tbf.ko 2024-05-28T05:52:13.833 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_teql.ko 2024-05-28T05:52:13.833 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sctp 2024-05-28T05:52:13.833 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sctp/sctp.ko 2024-05-28T05:52:13.833 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sctp/sctp_diag.ko 2024-05-28T05:52:13.833 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sunrpc 2024-05-28T05:52:13.833 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sunrpc/auth_gss 2024-05-28T05:52:13.834 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-05-28T05:52:13.834 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-05-28T05:52:13.834 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sunrpc/sunrpc.ko 2024-05-28T05:52:13.834 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sunrpc/xprtrdma 2024-05-28T05:52:13.834 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-05-28T05:52:13.834 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/tipc 2024-05-28T05:52:13.834 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/tipc/tipc.ko 2024-05-28T05:52:13.834 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/tipc/tipc_diag.ko 2024-05-28T05:52:13.834 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/unix 2024-05-28T05:52:13.834 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/unix/unix_diag.ko 2024-05-28T05:52:13.834 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/vmw_vsock 2024-05-28T05:52:13.834 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-05-28T05:52:13.835 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-05-28T05:52:13.835 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/vmw_vsock/vsock.ko 2024-05-28T05:52:13.835 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/vmw_vsock/vsock_diag.ko 2024-05-28T05:52:13.835 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/vmw_vsock/vsock_loopback.ko 2024-05-28T05:52:13.835 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/xfrm 2024-05-28T05:52:13.835 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/xfrm/xfrm_ipcomp.ko 2024-05-28T05:52:13.835 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/security 2024-05-28T05:52:13.835 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/security/keys 2024-05-28T05:52:13.835 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/security/keys/encrypted-keys 2024-05-28T05:52:13.835 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-05-28T05:52:13.835 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/security/keys/trusted-keys 2024-05-28T05:52:13.835 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/security/keys/trusted-keys/trusted.ko 2024-05-28T05:52:13.835 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.alias 2024-05-28T05:52:13.836 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.alias.bin 2024-05-28T05:52:13.836 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.builtin 2024-05-28T05:52:13.836 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.builtin.alias.bin 2024-05-28T05:52:13.836 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.builtin.bin 2024-05-28T05:52:13.836 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.builtin.modinfo 2024-05-28T05:52:13.836 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.dep 2024-05-28T05:52:13.836 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.dep.bin 2024-05-28T05:52:13.836 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.devname 2024-05-28T05:52:13.836 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.order 2024-05-28T05:52:13.836 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.softdep 2024-05-28T05:52:13.836 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.symbols 2024-05-28T05:52:13.836 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.symbols.bin 2024-05-28T05:52:13.836 INFO:teuthology.orchestra.run.smithi096.stdout:/lib/modules/6.9.0-gb9459a26b088/vmlinuz 2024-05-28T05:52:13.841 DEBUG:teuthology.task.kernel:get_image_version: 6.9.0-gb9459a26b088 2024-05-28T05:52:13.841 DEBUG:teuthology.orchestra.run.smithi096:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-05-28T05:52:13.863 INFO:teuthology.orchestra.run.smithi096.stdout:postinstall scriptlet (using /bin/sh): 2024-05-28T05:52:13.863 INFO:teuthology.orchestra.run.smithi096.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-05-28T05:52:13.863 INFO:teuthology.orchestra.run.smithi096.stdout: /usr/bin/kernel-install add 6.9.0-gb9459a26b088 /lib/modules/6.9.0-gb9459a26b088/vmlinuz 2024-05-28T05:52:13.863 INFO:teuthology.orchestra.run.smithi096.stdout:fi 2024-05-28T05:52:13.863 INFO:teuthology.orchestra.run.smithi096.stdout:for file in vmlinuz System.map config; do 2024-05-28T05:52:13.863 INFO:teuthology.orchestra.run.smithi096.stdout: if ! cmp --silent "/lib/modules/6.9.0-gb9459a26b088/${file}" "/boot/${file}-6.9.0-gb9459a26b088"; then 2024-05-28T05:52:13.863 INFO:teuthology.orchestra.run.smithi096.stdout: cp "/lib/modules/6.9.0-gb9459a26b088/${file}" "/boot/${file}-6.9.0-gb9459a26b088" 2024-05-28T05:52:13.863 INFO:teuthology.orchestra.run.smithi096.stdout: fi 2024-05-28T05:52:13.864 INFO:teuthology.orchestra.run.smithi096.stdout:done 2024-05-28T05:52:13.864 INFO:teuthology.orchestra.run.smithi096.stdout:if [ -d "/lib/modules/6.9.0-gb9459a26b088/dtb" ] && \ 2024-05-28T05:52:13.864 INFO:teuthology.orchestra.run.smithi096.stdout: ! diff -rq "/lib/modules/6.9.0-gb9459a26b088/dtb" "/boot/dtb-6.9.0-gb9459a26b088" >/dev/null 2>&1; then 2024-05-28T05:52:13.864 INFO:teuthology.orchestra.run.smithi096.stdout: rm -rf "/boot/dtb-6.9.0-gb9459a26b088" 2024-05-28T05:52:13.864 INFO:teuthology.orchestra.run.smithi096.stdout: cp -r "/lib/modules/6.9.0-gb9459a26b088/dtb" "/boot/dtb-6.9.0-gb9459a26b088" 2024-05-28T05:52:13.864 INFO:teuthology.orchestra.run.smithi096.stdout:fi 2024-05-28T05:52:13.864 INFO:teuthology.orchestra.run.smithi096.stdout:if [ ! -e "/lib/modules/6.9.0-gb9459a26b088/modules.dep" ]; then 2024-05-28T05:52:13.864 INFO:teuthology.orchestra.run.smithi096.stdout: /usr/sbin/depmod 6.9.0-gb9459a26b088 2024-05-28T05:52:13.864 INFO:teuthology.orchestra.run.smithi096.stdout:fi 2024-05-28T05:52:13.864 INFO:teuthology.orchestra.run.smithi096.stdout:preuninstall scriptlet (using /bin/sh): 2024-05-28T05:52:13.864 INFO:teuthology.orchestra.run.smithi096.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-05-28T05:52:13.865 INFO:teuthology.orchestra.run.smithi096.stdout:kernel-install remove 6.9.0-gb9459a26b088 2024-05-28T05:52:13.865 INFO:teuthology.orchestra.run.smithi096.stdout:fi 2024-05-28T05:52:13.865 DEBUG:teuthology.orchestra.run.smithi096:> sudo rpm -qi grub2-tools 2024-05-28T05:52:13.946 INFO:teuthology.orchestra.run.smithi096.stdout:Name : grub2-tools 2024-05-28T05:52:13.946 INFO:teuthology.orchestra.run.smithi096.stdout:Epoch : 1 2024-05-28T05:52:13.947 INFO:teuthology.orchestra.run.smithi096.stdout:Version : 2.06 2024-05-28T05:52:13.947 INFO:teuthology.orchestra.run.smithi096.stdout:Release : 68.el9 2024-05-28T05:52:13.947 INFO:teuthology.orchestra.run.smithi096.stdout:Architecture: x86_64 2024-05-28T05:52:13.947 INFO:teuthology.orchestra.run.smithi096.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-05-28T05:52:13.947 INFO:teuthology.orchestra.run.smithi096.stdout:Group : Unspecified 2024-05-28T05:52:13.947 INFO:teuthology.orchestra.run.smithi096.stdout:Size : 8264254 2024-05-28T05:52:13.947 INFO:teuthology.orchestra.run.smithi096.stdout:License : GPLv3+ 2024-05-28T05:52:13.947 INFO:teuthology.orchestra.run.smithi096.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-05-28T05:52:13.947 INFO:teuthology.orchestra.run.smithi096.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-05-28T05:52:13.947 INFO:teuthology.orchestra.run.smithi096.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-05-28T05:52:13.947 INFO:teuthology.orchestra.run.smithi096.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-05-28T05:52:13.947 INFO:teuthology.orchestra.run.smithi096.stdout:Packager : builder@centos.org 2024-05-28T05:52:13.947 INFO:teuthology.orchestra.run.smithi096.stdout:Vendor : CentOS 2024-05-28T05:52:13.948 INFO:teuthology.orchestra.run.smithi096.stdout:URL : http://www.gnu.org/software/grub/ 2024-05-28T05:52:13.948 INFO:teuthology.orchestra.run.smithi096.stdout:Summary : Support tools for GRUB. 2024-05-28T05:52:13.948 INFO:teuthology.orchestra.run.smithi096.stdout:Description : 2024-05-28T05:52:13.948 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T05:52:13.948 INFO:teuthology.orchestra.run.smithi096.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-05-28T05:52:13.948 INFO:teuthology.orchestra.run.smithi096.stdout:customizable bootloader with modular architecture. It supports a rich 2024-05-28T05:52:13.948 INFO:teuthology.orchestra.run.smithi096.stdout:variety of kernel formats, file systems, computer architectures and 2024-05-28T05:52:13.948 INFO:teuthology.orchestra.run.smithi096.stdout:hardware devices. 2024-05-28T05:52:13.948 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T05:52:13.948 INFO:teuthology.orchestra.run.smithi096.stdout:This subpackage provides tools for support of all platforms. 2024-05-28T05:52:13.949 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-05-28T05:52:13.949 INFO:teuthology.task.kernel:Updating grub on smithi096 to boot 6.9.0-gb9459a26b088 2024-05-28T05:52:13.949 DEBUG:teuthology.orchestra.run.smithi096:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-05-28T05:52:14.488 INFO:teuthology.orchestra.run.smithi096.stderr:Generating grub configuration file ... 2024-05-28T05:52:15.850 INFO:teuthology.orchestra.run.smithi096.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-05-28T05:52:15.877 INFO:teuthology.orchestra.run.smithi096.stderr:done 2024-05-28T05:52:15.879 DEBUG:teuthology.orchestra.run.smithi096:> mktemp 2024-05-28T05:52:15.910 INFO:teuthology.orchestra.run.smithi096.stdout:/tmp/tmp.WIawS0ilu2 2024-05-28T05:52:15.910 DEBUG:teuthology.orchestra.run.smithi096:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.WIawS0ilu2 2024-05-28T05:52:15.983 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod 0666 /tmp/tmp.WIawS0ilu2 2024-05-28T05:52:16.115 DEBUG:teuthology.orchestra.remote:smithi096:/tmp/tmp.WIawS0ilu2 is 6KB 2024-05-28T05:52:16.162 DEBUG:teuthology.orchestra.run.smithi096:> rm -fr /tmp/tmp.WIawS0ilu2 2024-05-28T05:52:16.177 DEBUG:teuthology.orchestra.run.smithi096:> sudo /bin/ls /boot/loader/entries || true 2024-05-28T05:52:16.246 INFO:teuthology.orchestra.run.smithi096.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-05-28T05:52:16.246 INFO:teuthology.orchestra.run.smithi096.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-05-28T05:52:16.246 INFO:teuthology.orchestra.run.smithi096.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-05-28T05:52:16.246 INFO:teuthology.orchestra.run.smithi096.stdout:7fc8828e06d0482694bcf4a447d13cb6-6.9.0-gb9459a26b088.conf 2024-05-28T05:52:16.247 DEBUG:teuthology.orchestra.run.smithi096:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-6.9.0-gb9459a26b088 2024-05-28T05:52:16.399 DEBUG:teuthology.orchestra.run.smithi096:> sudo shutdown -r now 2024-05-28T05:52:20.803 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi123.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-05-28T05:52:20.804 DEBUG:teuthology.orchestra.run.smithi123:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-05-28T05:52:20.876 INFO:teuthology.orchestra.run.smithi123.stdout:/boot/System.map-6.9.0-gb9459a26b088 2024-05-28T05:52:20.876 INFO:teuthology.orchestra.run.smithi123.stdout:/boot/config-6.9.0-gb9459a26b088 2024-05-28T05:52:20.876 INFO:teuthology.orchestra.run.smithi123.stdout:/boot/vmlinuz-6.9.0-gb9459a26b088 2024-05-28T05:52:20.876 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/System.map 2024-05-28T05:52:20.877 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/config 2024-05-28T05:52:20.877 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel 2024-05-28T05:52:20.877 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch 2024-05-28T05:52:20.877 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86 2024-05-28T05:52:20.877 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto 2024-05-28T05:52:20.877 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-05-28T05:52:20.877 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-05-28T05:52:20.877 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-05-28T05:52:20.877 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-05-28T05:52:20.877 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-05-28T05:52:20.877 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-05-28T05:52:20.877 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-05-28T05:52:20.877 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/crc32c-intel.ko 2024-05-28T05:52:20.878 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-05-28T05:52:20.878 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-05-28T05:52:20.878 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-05-28T05:52:20.878 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-05-28T05:52:20.878 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/serpent-avx2.ko 2024-05-28T05:52:20.878 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-05-28T05:52:20.878 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-05-28T05:52:20.878 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-05-28T05:52:20.878 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-05-28T05:52:20.878 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-05-28T05:52:20.878 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-05-28T05:52:20.878 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-05-28T05:52:20.879 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kernel 2024-05-28T05:52:20.879 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kernel/cpu 2024-05-28T05:52:20.879 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kernel/cpu/mce 2024-05-28T05:52:20.879 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-05-28T05:52:20.879 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kvm 2024-05-28T05:52:20.879 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kvm/kvm-amd.ko 2024-05-28T05:52:20.879 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kvm/kvm-intel.ko 2024-05-28T05:52:20.879 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kvm/kvm.ko 2024-05-28T05:52:20.879 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto 2024-05-28T05:52:20.879 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/ansi_cprng.ko 2024-05-28T05:52:20.879 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/anubis.ko 2024-05-28T05:52:20.879 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/arc4.ko 2024-05-28T05:52:20.879 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/async_tx 2024-05-28T05:52:20.879 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/async_tx/async_memcpy.ko 2024-05-28T05:52:20.880 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/async_tx/async_pq.ko 2024-05-28T05:52:20.880 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/async_tx/async_raid6_recov.ko 2024-05-28T05:52:20.880 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/async_tx/async_tx.ko 2024-05-28T05:52:20.880 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/async_tx/async_xor.ko 2024-05-28T05:52:20.880 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/async_tx/raid6test.ko 2024-05-28T05:52:20.880 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/authenc.ko 2024-05-28T05:52:20.880 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/authencesn.ko 2024-05-28T05:52:20.880 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/blake2b_generic.ko 2024-05-28T05:52:20.880 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/blowfish_common.ko 2024-05-28T05:52:20.880 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/blowfish_generic.ko 2024-05-28T05:52:20.880 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/camellia_generic.ko 2024-05-28T05:52:20.880 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/cast5_generic.ko 2024-05-28T05:52:20.880 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/cast6_generic.ko 2024-05-28T05:52:20.881 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/cast_common.ko 2024-05-28T05:52:20.881 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/ccm.ko 2024-05-28T05:52:20.881 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/cmac.ko 2024-05-28T05:52:20.881 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/crc32_generic.ko 2024-05-28T05:52:20.881 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/crypto_engine.ko 2024-05-28T05:52:20.881 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/crypto_user.ko 2024-05-28T05:52:20.881 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/cts.ko 2024-05-28T05:52:20.881 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/des_generic.ko 2024-05-28T05:52:20.881 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/dh_generic.ko 2024-05-28T05:52:20.881 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/echainiv.ko 2024-05-28T05:52:20.881 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/essiv.ko 2024-05-28T05:52:20.881 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/fcrypt.ko 2024-05-28T05:52:20.881 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/khazad.ko 2024-05-28T05:52:20.881 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/lz4.ko 2024-05-28T05:52:20.882 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/lz4hc.ko 2024-05-28T05:52:20.882 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/md4.ko 2024-05-28T05:52:20.882 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/michael_mic.ko 2024-05-28T05:52:20.882 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/pcbc.ko 2024-05-28T05:52:20.882 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/pcrypt.ko 2024-05-28T05:52:20.882 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/rmd160.ko 2024-05-28T05:52:20.882 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/seed.ko 2024-05-28T05:52:20.882 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/serpent_generic.ko 2024-05-28T05:52:20.882 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/tcrypt.ko 2024-05-28T05:52:20.882 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/tea.ko 2024-05-28T05:52:20.882 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/twofish_common.ko 2024-05-28T05:52:20.882 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/twofish_generic.ko 2024-05-28T05:52:20.882 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/vmac.ko 2024-05-28T05:52:20.883 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/wp512.ko 2024-05-28T05:52:20.883 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/xcbc.ko 2024-05-28T05:52:20.883 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/xor.ko 2024-05-28T05:52:20.883 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/xxhash_generic.ko 2024-05-28T05:52:20.883 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers 2024-05-28T05:52:20.883 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi 2024-05-28T05:52:20.883 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/acpi_ipmi.ko 2024-05-28T05:52:20.883 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/acpi_pad.ko 2024-05-28T05:52:20.883 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/ec_sys.ko 2024-05-28T05:52:20.883 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/platform_profile.ko 2024-05-28T05:52:20.883 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/sbs.ko 2024-05-28T05:52:20.883 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/sbshc.ko 2024-05-28T05:52:20.883 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/video.ko 2024-05-28T05:52:20.884 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata 2024-05-28T05:52:20.884 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/acard-ahci.ko 2024-05-28T05:52:20.884 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/ahci_platform.ko 2024-05-28T05:52:20.884 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/ata_generic.ko 2024-05-28T05:52:20.884 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/libahci_platform.ko 2024-05-28T05:52:20.884 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_acpi.ko 2024-05-28T05:52:20.884 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_ali.ko 2024-05-28T05:52:20.884 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_amd.ko 2024-05-28T05:52:20.884 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_artop.ko 2024-05-28T05:52:20.884 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_atiixp.ko 2024-05-28T05:52:20.884 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_atp867x.ko 2024-05-28T05:52:20.884 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_cmd640.ko 2024-05-28T05:52:20.884 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_cmd64x.ko 2024-05-28T05:52:20.885 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_cypress.ko 2024-05-28T05:52:20.885 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_efar.ko 2024-05-28T05:52:20.885 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_hpt366.ko 2024-05-28T05:52:20.885 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_hpt37x.ko 2024-05-28T05:52:20.885 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_hpt3x2n.ko 2024-05-28T05:52:20.885 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_hpt3x3.ko 2024-05-28T05:52:20.885 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_it8213.ko 2024-05-28T05:52:20.885 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_it821x.ko 2024-05-28T05:52:20.885 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_jmicron.ko 2024-05-28T05:52:20.885 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_marvell.ko 2024-05-28T05:52:20.885 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_mpiix.ko 2024-05-28T05:52:20.885 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_netcell.ko 2024-05-28T05:52:20.885 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_ninja32.ko 2024-05-28T05:52:20.885 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_ns87410.ko 2024-05-28T05:52:20.886 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_ns87415.ko 2024-05-28T05:52:20.886 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_oldpiix.ko 2024-05-28T05:52:20.886 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_opti.ko 2024-05-28T05:52:20.886 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_optidma.ko 2024-05-28T05:52:20.886 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_pcmcia.ko 2024-05-28T05:52:20.886 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_pdc2027x.ko 2024-05-28T05:52:20.886 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-05-28T05:52:20.886 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_piccolo.ko 2024-05-28T05:52:20.886 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_rdc.ko 2024-05-28T05:52:20.886 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_sch.ko 2024-05-28T05:52:20.886 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_serverworks.ko 2024-05-28T05:52:20.886 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_sil680.ko 2024-05-28T05:52:20.886 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_sis.ko 2024-05-28T05:52:20.887 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_sl82c105.ko 2024-05-28T05:52:20.887 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_triflex.ko 2024-05-28T05:52:20.887 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_via.ko 2024-05-28T05:52:20.887 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pdc_adma.ko 2024-05-28T05:52:20.887 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_inic162x.ko 2024-05-28T05:52:20.887 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_mv.ko 2024-05-28T05:52:20.887 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_nv.ko 2024-05-28T05:52:20.887 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_promise.ko 2024-05-28T05:52:20.887 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_qstor.ko 2024-05-28T05:52:20.887 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_sil.ko 2024-05-28T05:52:20.887 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_sil24.ko 2024-05-28T05:52:20.887 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_sis.ko 2024-05-28T05:52:20.887 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_svw.ko 2024-05-28T05:52:20.887 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_sx4.ko 2024-05-28T05:52:20.888 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_uli.ko 2024-05-28T05:52:20.888 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_via.ko 2024-05-28T05:52:20.888 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_vsc.ko 2024-05-28T05:52:20.888 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/auxdisplay 2024-05-28T05:52:20.888 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/auxdisplay/ks0108.ko 2024-05-28T05:52:20.888 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/base 2024-05-28T05:52:20.888 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/base/regmap 2024-05-28T05:52:20.889 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/base/regmap/regmap-i2c.ko 2024-05-28T05:52:20.889 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/base/regmap/regmap-mmio.ko 2024-05-28T05:52:20.889 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/bcma 2024-05-28T05:52:20.889 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/bcma/bcma.ko 2024-05-28T05:52:20.889 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block 2024-05-28T05:52:20.889 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/aoe 2024-05-28T05:52:20.889 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/aoe/aoe.ko 2024-05-28T05:52:20.889 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/brd.ko 2024-05-28T05:52:20.889 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/drbd 2024-05-28T05:52:20.889 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/drbd/drbd.ko 2024-05-28T05:52:20.890 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/floppy.ko 2024-05-28T05:52:20.890 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/loop.ko 2024-05-28T05:52:20.890 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/mtip32xx 2024-05-28T05:52:20.890 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-05-28T05:52:20.890 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/nbd.ko 2024-05-28T05:52:20.890 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/null_blk 2024-05-28T05:52:20.890 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/null_blk/null_blk.ko 2024-05-28T05:52:20.890 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/pktcdvd.ko 2024-05-28T05:52:20.890 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/rbd.ko 2024-05-28T05:52:20.890 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/virtio_blk.ko 2024-05-28T05:52:20.890 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/xen-blkback 2024-05-28T05:52:20.890 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-05-28T05:52:20.890 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/xen-blkfront.ko 2024-05-28T05:52:20.891 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/zram 2024-05-28T05:52:20.891 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/zram/zram.ko 2024-05-28T05:52:20.891 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char 2024-05-28T05:52:20.891 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/hangcheck-timer.ko 2024-05-28T05:52:20.891 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/hw_random 2024-05-28T05:52:20.891 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/hw_random/amd-rng.ko 2024-05-28T05:52:20.891 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/hw_random/intel-rng.ko 2024-05-28T05:52:20.891 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-05-28T05:52:20.891 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/hw_random/via-rng.ko 2024-05-28T05:52:20.891 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/hw_random/virtio-rng.ko 2024-05-28T05:52:20.891 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ipmi 2024-05-28T05:52:20.891 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-05-28T05:52:20.891 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-05-28T05:52:20.892 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-05-28T05:52:20.892 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ipmi/ipmi_si.ko 2024-05-28T05:52:20.892 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-05-28T05:52:20.892 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-05-28T05:52:20.892 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/lp.ko 2024-05-28T05:52:20.892 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/mwave 2024-05-28T05:52:20.892 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/mwave/mwave.ko 2024-05-28T05:52:20.892 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ppdev.ko 2024-05-28T05:52:20.892 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tlclk.ko 2024-05-28T05:52:20.892 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm 2024-05-28T05:52:20.892 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm/tpm.ko 2024-05-28T05:52:20.892 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm/tpm_atmel.ko 2024-05-28T05:52:20.892 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm/tpm_infineon.ko 2024-05-28T05:52:20.892 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm/tpm_nsc.ko 2024-05-28T05:52:20.892 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm/tpm_tis.ko 2024-05-28T05:52:20.893 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-05-28T05:52:20.893 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/uv_mmtimer.ko 2024-05-28T05:52:20.893 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/virtio_console.ko 2024-05-28T05:52:20.893 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/cpufreq 2024-05-28T05:52:20.893 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-05-28T05:52:20.893 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-05-28T05:52:20.893 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/cpufreq/p4-clockmod.ko 2024-05-28T05:52:20.893 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-05-28T05:52:20.893 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/cpufreq/powernow-k8.ko 2024-05-28T05:52:20.893 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/cpufreq/speedstep-lib.ko 2024-05-28T05:52:20.893 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto 2024-05-28T05:52:20.893 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/ccp 2024-05-28T05:52:20.893 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-05-28T05:52:20.893 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/ccp/ccp.ko 2024-05-28T05:52:20.894 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/intel 2024-05-28T05:52:20.894 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/intel/qat 2024-05-28T05:52:20.894 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/intel/qat/qat_common 2024-05-28T05:52:20.894 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-05-28T05:52:20.894 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-05-28T05:52:20.894 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-05-28T05:52:20.894 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/padlock-aes.ko 2024-05-28T05:52:20.894 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/padlock-sha.ko 2024-05-28T05:52:20.894 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/virtio 2024-05-28T05:52:20.894 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-05-28T05:52:20.894 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/dca 2024-05-28T05:52:20.894 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/dca/dca.ko 2024-05-28T05:52:20.894 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/devfreq 2024-05-28T05:52:20.894 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-05-28T05:52:20.895 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/dma 2024-05-28T05:52:20.895 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/dma/dw 2024-05-28T05:52:20.895 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/dma/dw/dw_dmac.ko 2024-05-28T05:52:20.895 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/dma/ioat 2024-05-28T05:52:20.895 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/dma/ioat/ioatdma.ko 2024-05-28T05:52:20.895 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac 2024-05-28T05:52:20.895 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/amd64_edac.ko 2024-05-28T05:52:20.895 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/e752x_edac.ko 2024-05-28T05:52:20.895 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/edac_mce_amd.ko 2024-05-28T05:52:20.895 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/i3000_edac.ko 2024-05-28T05:52:20.895 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/i3200_edac.ko 2024-05-28T05:52:20.895 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/i5100_edac.ko 2024-05-28T05:52:20.895 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/i5400_edac.ko 2024-05-28T05:52:20.895 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/i7300_edac.ko 2024-05-28T05:52:20.895 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/i7core_edac.ko 2024-05-28T05:52:20.896 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/i82975x_edac.ko 2024-05-28T05:52:20.896 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/ie31200_edac.ko 2024-05-28T05:52:20.896 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/sb_edac.ko 2024-05-28T05:52:20.896 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/x38_edac.ko 2024-05-28T05:52:20.896 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/firmware 2024-05-28T05:52:20.896 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/firmware/edd.ko 2024-05-28T05:52:20.896 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/firmware/iscsi_ibft.ko 2024-05-28T05:52:20.896 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio 2024-05-28T05:52:20.896 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-ich.ko 2024-05-28T05:52:20.896 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-viperboard.ko 2024-05-28T05:52:20.896 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid 2024-05-28T05:52:20.896 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-appleir.ko 2024-05-28T05:52:20.896 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-aureal.ko 2024-05-28T05:52:20.897 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-axff.ko 2024-05-28T05:52:20.897 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-dr.ko 2024-05-28T05:52:20.897 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-elecom.ko 2024-05-28T05:52:20.897 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-elo.ko 2024-05-28T05:52:20.897 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-emsff.ko 2024-05-28T05:52:20.897 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-gaff.ko 2024-05-28T05:52:20.897 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-gt683r.ko 2024-05-28T05:52:20.897 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-gyration.ko 2024-05-28T05:52:20.897 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-holtek-kbd.ko 2024-05-28T05:52:20.897 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-holtek-mouse.ko 2024-05-28T05:52:20.897 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-holtekff.ko 2024-05-28T05:52:20.897 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-hyperv.ko 2024-05-28T05:52:20.897 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-icade.ko 2024-05-28T05:52:20.897 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-keytouch.ko 2024-05-28T05:52:20.898 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-kye.ko 2024-05-28T05:52:20.898 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-lcpower.ko 2024-05-28T05:52:20.898 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-led.ko 2024-05-28T05:52:20.898 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-lenovo.ko 2024-05-28T05:52:20.898 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-logitech-dj.ko 2024-05-28T05:52:20.898 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-05-28T05:52:20.898 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-multitouch.ko 2024-05-28T05:52:20.898 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-ortek.ko 2024-05-28T05:52:20.898 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-penmount.ko 2024-05-28T05:52:20.898 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-petalynx.ko 2024-05-28T05:52:20.898 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-picolcd.ko 2024-05-28T05:52:20.898 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-pl.ko 2024-05-28T05:52:20.898 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-plantronics.ko 2024-05-28T05:52:20.899 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-primax.ko 2024-05-28T05:52:20.899 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-rmi.ko 2024-05-28T05:52:20.899 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-arvo.ko 2024-05-28T05:52:20.899 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-common.ko 2024-05-28T05:52:20.899 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-isku.ko 2024-05-28T05:52:20.899 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-kone.ko 2024-05-28T05:52:20.899 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-05-28T05:52:20.899 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-konepure.ko 2024-05-28T05:52:20.899 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-05-28T05:52:20.899 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-lua.ko 2024-05-28T05:52:20.899 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-pyra.ko 2024-05-28T05:52:20.899 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-ryos.ko 2024-05-28T05:52:20.899 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-savu.ko 2024-05-28T05:52:20.899 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat.ko 2024-05-28T05:52:20.900 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-saitek.ko 2024-05-28T05:52:20.900 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-samsung.ko 2024-05-28T05:52:20.900 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-sensor-hub.ko 2024-05-28T05:52:20.900 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-sjoy.ko 2024-05-28T05:52:20.900 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-sony.ko 2024-05-28T05:52:20.900 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-speedlink.ko 2024-05-28T05:52:20.900 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-steelseries.ko 2024-05-28T05:52:20.900 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-sunplus.ko 2024-05-28T05:52:20.900 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-thrustmaster.ko 2024-05-28T05:52:20.900 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-tivo.ko 2024-05-28T05:52:20.900 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-tmff.ko 2024-05-28T05:52:20.900 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-topseed.ko 2024-05-28T05:52:20.900 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-twinhan.ko 2024-05-28T05:52:20.900 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-uclogic.ko 2024-05-28T05:52:20.901 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-waltop.ko 2024-05-28T05:52:20.901 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-wiimote.ko 2024-05-28T05:52:20.901 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-xinmo.ko 2024-05-28T05:52:20.901 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-zpff.ko 2024-05-28T05:52:20.901 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-zydacron.ko 2024-05-28T05:52:20.901 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/uhid.ko 2024-05-28T05:52:20.901 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/wacom.ko 2024-05-28T05:52:20.901 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hv 2024-05-28T05:52:20.901 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hv/hv_balloon.ko 2024-05-28T05:52:20.901 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hv/hv_utils.ko 2024-05-28T05:52:20.901 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hv/hv_vmbus.ko 2024-05-28T05:52:20.901 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon 2024-05-28T05:52:20.901 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/abituguru.ko 2024-05-28T05:52:20.901 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/abituguru3.ko 2024-05-28T05:52:20.902 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/acpi_power_meter.ko 2024-05-28T05:52:20.902 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ad7414.ko 2024-05-28T05:52:20.902 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ad7418.ko 2024-05-28T05:52:20.902 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adc128d818.ko 2024-05-28T05:52:20.902 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adm1025.ko 2024-05-28T05:52:20.902 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adm1026.ko 2024-05-28T05:52:20.902 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adm1029.ko 2024-05-28T05:52:20.902 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adm1031.ko 2024-05-28T05:52:20.902 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adm9240.ko 2024-05-28T05:52:20.902 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ads7828.ko 2024-05-28T05:52:20.902 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adt7410.ko 2024-05-28T05:52:20.902 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adt7411.ko 2024-05-28T05:52:20.902 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adt7462.ko 2024-05-28T05:52:20.902 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adt7470.ko 2024-05-28T05:52:20.903 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adt7475.ko 2024-05-28T05:52:20.903 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adt7x10.ko 2024-05-28T05:52:20.903 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/amc6821.ko 2024-05-28T05:52:20.903 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/applesmc.ko 2024-05-28T05:52:20.903 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/asb100.ko 2024-05-28T05:52:20.903 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/asc7621.ko 2024-05-28T05:52:20.903 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/asus_atk0110.ko 2024-05-28T05:52:20.903 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/atxp1.ko 2024-05-28T05:52:20.903 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/coretemp.ko 2024-05-28T05:52:20.903 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-05-28T05:52:20.903 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/dme1737.ko 2024-05-28T05:52:20.903 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ds1621.ko 2024-05-28T05:52:20.903 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ds620.ko 2024-05-28T05:52:20.903 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/emc1403.ko 2024-05-28T05:52:20.904 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/emc6w201.ko 2024-05-28T05:52:20.904 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/f71805f.ko 2024-05-28T05:52:20.904 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/f71882fg.ko 2024-05-28T05:52:20.904 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/f75375s.ko 2024-05-28T05:52:20.904 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/fam15h_power.ko 2024-05-28T05:52:20.904 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/fschmd.ko 2024-05-28T05:52:20.904 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/g760a.ko 2024-05-28T05:52:20.904 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/g762.ko 2024-05-28T05:52:20.904 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/gl518sm.ko 2024-05-28T05:52:20.904 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/gl520sm.ko 2024-05-28T05:52:20.904 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/hwmon-vid.ko 2024-05-28T05:52:20.904 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/i5500_temp.ko 2024-05-28T05:52:20.904 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/i5k_amb.ko 2024-05-28T05:52:20.904 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ibmaem.ko 2024-05-28T05:52:20.905 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ibmpex.ko 2024-05-28T05:52:20.905 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ina209.ko 2024-05-28T05:52:20.905 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ina2xx.ko 2024-05-28T05:52:20.905 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/it87.ko 2024-05-28T05:52:20.905 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/k10temp.ko 2024-05-28T05:52:20.905 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/k8temp.ko 2024-05-28T05:52:20.905 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lineage-pem.ko 2024-05-28T05:52:20.905 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm63.ko 2024-05-28T05:52:20.905 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm73.ko 2024-05-28T05:52:20.905 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm75.ko 2024-05-28T05:52:20.905 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm77.ko 2024-05-28T05:52:20.905 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm78.ko 2024-05-28T05:52:20.905 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm80.ko 2024-05-28T05:52:20.906 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm83.ko 2024-05-28T05:52:20.906 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm85.ko 2024-05-28T05:52:20.906 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm87.ko 2024-05-28T05:52:20.906 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm90.ko 2024-05-28T05:52:20.906 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm92.ko 2024-05-28T05:52:20.906 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm93.ko 2024-05-28T05:52:20.906 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm95234.ko 2024-05-28T05:52:20.906 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm95241.ko 2024-05-28T05:52:20.906 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm95245.ko 2024-05-28T05:52:20.906 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ltc2945.ko 2024-05-28T05:52:20.906 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ltc4151.ko 2024-05-28T05:52:20.906 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ltc4215.ko 2024-05-28T05:52:20.906 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ltc4222.ko 2024-05-28T05:52:20.907 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ltc4245.ko 2024-05-28T05:52:20.907 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ltc4260.ko 2024-05-28T05:52:20.907 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ltc4261.ko 2024-05-28T05:52:20.907 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/max16065.ko 2024-05-28T05:52:20.907 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/max1619.ko 2024-05-28T05:52:20.907 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/max1668.ko 2024-05-28T05:52:20.907 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/max197.ko 2024-05-28T05:52:20.907 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/max6639.ko 2024-05-28T05:52:20.907 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/max6650.ko 2024-05-28T05:52:20.907 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/max6697.ko 2024-05-28T05:52:20.907 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/mcp3021.ko 2024-05-28T05:52:20.907 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/nct6683.ko 2024-05-28T05:52:20.907 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/nct6775-core.ko 2024-05-28T05:52:20.907 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/nct6775.ko 2024-05-28T05:52:20.908 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/nct7802.ko 2024-05-28T05:52:20.908 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pc87360.ko 2024-05-28T05:52:20.908 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pc87427.ko 2024-05-28T05:52:20.908 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pcf8591.ko 2024-05-28T05:52:20.908 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus 2024-05-28T05:52:20.908 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-05-28T05:52:20.908 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-05-28T05:52:20.908 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-05-28T05:52:20.908 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/max16064.ko 2024-05-28T05:52:20.908 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/max34440.ko 2024-05-28T05:52:20.908 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/max8688.ko 2024-05-28T05:52:20.908 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-05-28T05:52:20.908 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-05-28T05:52:20.908 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-05-28T05:52:20.909 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-05-28T05:52:20.909 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-05-28T05:52:20.909 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-05-28T05:52:20.909 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/powr1220.ko 2024-05-28T05:52:20.909 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/sch5627.ko 2024-05-28T05:52:20.909 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/sch5636.ko 2024-05-28T05:52:20.909 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/sch56xx-common.ko 2024-05-28T05:52:20.909 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/sht15.ko 2024-05-28T05:52:20.909 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/sht21.ko 2024-05-28T05:52:20.909 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/shtc1.ko 2024-05-28T05:52:20.909 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/sis5595.ko 2024-05-28T05:52:20.909 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/smsc47b397.ko 2024-05-28T05:52:20.909 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/smsc47m1.ko 2024-05-28T05:52:20.909 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/smsc47m192.ko 2024-05-28T05:52:20.910 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/thmc50.ko 2024-05-28T05:52:20.910 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/tmp102.ko 2024-05-28T05:52:20.910 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/tmp103.ko 2024-05-28T05:52:20.910 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/tmp401.ko 2024-05-28T05:52:20.910 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/tmp421.ko 2024-05-28T05:52:20.910 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/via-cputemp.ko 2024-05-28T05:52:20.910 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/via686a.ko 2024-05-28T05:52:20.910 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/vt1211.ko 2024-05-28T05:52:20.910 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/vt8231.ko 2024-05-28T05:52:20.910 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83627ehf.ko 2024-05-28T05:52:20.910 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83627hf.ko 2024-05-28T05:52:20.910 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83781d.ko 2024-05-28T05:52:20.910 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83791d.ko 2024-05-28T05:52:20.910 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83792d.ko 2024-05-28T05:52:20.911 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83793.ko 2024-05-28T05:52:20.911 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83795.ko 2024-05-28T05:52:20.911 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83l785ts.ko 2024-05-28T05:52:20.911 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83l786ng.ko 2024-05-28T05:52:20.911 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c 2024-05-28T05:52:20.911 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/algos 2024-05-28T05:52:20.911 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-05-28T05:52:20.911 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-05-28T05:52:20.911 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses 2024-05-28T05:52:20.911 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-05-28T05:52:20.911 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-05-28T05:52:20.911 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-05-28T05:52:20.911 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-05-28T05:52:20.911 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-05-28T05:52:20.912 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-05-28T05:52:20.912 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-05-28T05:52:20.912 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-05-28T05:52:20.912 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-i801.ko 2024-05-28T05:52:20.912 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-isch.ko 2024-05-28T05:52:20.912 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-05-28T05:52:20.912 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-05-28T05:52:20.912 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-05-28T05:52:20.912 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-parport.ko 2024-05-28T05:52:20.912 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-05-28T05:52:20.912 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-05-28T05:52:20.912 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-05-28T05:52:20.912 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-05-28T05:52:20.912 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-05-28T05:52:20.912 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-05-28T05:52:20.913 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-via.ko 2024-05-28T05:52:20.913 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-05-28T05:52:20.913 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-05-28T05:52:20.913 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/i2c-dev.ko 2024-05-28T05:52:20.913 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/i2c-mux.ko 2024-05-28T05:52:20.913 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-05-28T05:52:20.913 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/i2c-smbus.ko 2024-05-28T05:52:20.913 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/i2c-stub.ko 2024-05-28T05:52:20.913 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband 2024-05-28T05:52:20.913 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core 2024-05-28T05:52:20.913 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/ib_cm.ko 2024-05-28T05:52:20.913 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/ib_core.ko 2024-05-28T05:52:20.913 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/ib_umad.ko 2024-05-28T05:52:20.913 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-05-28T05:52:20.914 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/iw_cm.ko 2024-05-28T05:52:20.914 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/rdma_cm.ko 2024-05-28T05:52:20.914 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-05-28T05:52:20.914 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw 2024-05-28T05:52:20.914 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/cxgb4 2024-05-28T05:52:20.914 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-05-28T05:52:20.914 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/mlx4 2024-05-28T05:52:20.914 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-05-28T05:52:20.914 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/mlx5 2024-05-28T05:52:20.914 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-05-28T05:52:20.914 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/mthca 2024-05-28T05:52:20.914 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-05-28T05:52:20.914 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp 2024-05-28T05:52:20.915 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/ipoib 2024-05-28T05:52:20.915 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-05-28T05:52:20.915 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/iser 2024-05-28T05:52:20.915 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-05-28T05:52:20.915 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/isert 2024-05-28T05:52:20.915 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-05-28T05:52:20.915 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/srp 2024-05-28T05:52:20.915 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-05-28T05:52:20.915 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/srpt 2024-05-28T05:52:20.915 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-05-28T05:52:20.915 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input 2024-05-28T05:52:20.915 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/gameport 2024-05-28T05:52:20.915 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-05-28T05:52:20.915 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/gameport/fm801-gp.ko 2024-05-28T05:52:20.916 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/gameport/gameport.ko 2024-05-28T05:52:20.916 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/joydev.ko 2024-05-28T05:52:20.916 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard 2024-05-28T05:52:20.916 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard/gpio_keys.ko 2024-05-28T05:52:20.916 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse 2024-05-28T05:52:20.916 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/appletouch.ko 2024-05-28T05:52:20.916 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/bcm5974.ko 2024-05-28T05:52:20.916 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/cyapatp.ko 2024-05-28T05:52:20.916 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/elan_i2c.ko 2024-05-28T05:52:20.916 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/sermouse.ko 2024-05-28T05:52:20.916 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-05-28T05:52:20.916 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/synaptics_usb.ko 2024-05-28T05:52:20.916 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/vsxxxaa.ko 2024-05-28T05:52:20.916 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/rmi4 2024-05-28T05:52:20.916 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/rmi4/rmi_core.ko 2024-05-28T05:52:20.917 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/serio 2024-05-28T05:52:20.917 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/serio/altera_ps2.ko 2024-05-28T05:52:20.917 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/serio/arc_ps2.ko 2024-05-28T05:52:20.917 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-05-28T05:52:20.917 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/serio/serio_raw.ko 2024-05-28T05:52:20.917 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/sparse-keymap.ko 2024-05-28T05:52:20.917 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds 2024-05-28T05:52:20.917 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-blinkm.ko 2024-05-28T05:52:20.917 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-lm3530.ko 2024-05-28T05:52:20.917 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-lp3944.ko 2024-05-28T05:52:20.917 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-ss4200.ko 2024-05-28T05:52:20.917 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger 2024-05-28T05:52:20.917 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-05-28T05:52:20.917 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-05-28T05:52:20.918 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-05-28T05:52:20.918 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-05-28T05:52:20.918 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-05-28T05:52:20.918 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-05-28T05:52:20.918 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-05-28T05:52:20.918 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-05-28T05:52:20.918 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-05-28T05:52:20.918 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md 2024-05-28T05:52:20.918 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/bcache 2024-05-28T05:52:20.918 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/bcache/bcache.ko 2024-05-28T05:52:20.918 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-bio-prison.ko 2024-05-28T05:52:20.918 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-cache-smq.ko 2024-05-28T05:52:20.918 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-cache.ko 2024-05-28T05:52:20.918 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-crypt.ko 2024-05-28T05:52:20.919 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-delay.ko 2024-05-28T05:52:20.919 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-flakey.ko 2024-05-28T05:52:20.919 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-log-userspace.ko 2024-05-28T05:52:20.919 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-multipath.ko 2024-05-28T05:52:20.919 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-queue-length.ko 2024-05-28T05:52:20.919 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-raid.ko 2024-05-28T05:52:20.919 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-round-robin.ko 2024-05-28T05:52:20.919 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-service-time.ko 2024-05-28T05:52:20.919 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-switch.ko 2024-05-28T05:52:20.919 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-thin-pool.ko 2024-05-28T05:52:20.919 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-verity.ko 2024-05-28T05:52:20.919 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/persistent-data 2024-05-28T05:52:20.919 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-05-28T05:52:20.920 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/raid0.ko 2024-05-28T05:52:20.920 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/raid1.ko 2024-05-28T05:52:20.920 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/raid10.ko 2024-05-28T05:52:20.920 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/raid456.ko 2024-05-28T05:52:20.920 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message 2024-05-28T05:52:20.920 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion 2024-05-28T05:52:20.920 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/mptbase.ko 2024-05-28T05:52:20.920 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/mptctl.ko 2024-05-28T05:52:20.920 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/mptfc.ko 2024-05-28T05:52:20.920 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/mptlan.ko 2024-05-28T05:52:20.920 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/mptsas.ko 2024-05-28T05:52:20.920 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/mptscsih.ko 2024-05-28T05:52:20.920 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/mptspi.ko 2024-05-28T05:52:20.920 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd 2024-05-28T05:52:20.921 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/lpc_ich.ko 2024-05-28T05:52:20.921 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/lpc_sch.ko 2024-05-28T05:52:20.921 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/mfd-core.ko 2024-05-28T05:52:20.921 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/sm501.ko 2024-05-28T05:52:20.921 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/viperboard.ko 2024-05-28T05:52:20.921 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/vx855.ko 2024-05-28T05:52:20.921 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/wl1273-core.ko 2024-05-28T05:52:20.921 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc 2024-05-28T05:52:20.921 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/altera-stapl 2024-05-28T05:52:20.921 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-05-28T05:52:20.921 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/apds9802als.ko 2024-05-28T05:52:20.921 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/apds990x.ko 2024-05-28T05:52:20.921 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/bh1770glc.ko 2024-05-28T05:52:20.921 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/cb710 2024-05-28T05:52:20.922 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/cb710/cb710.ko 2024-05-28T05:52:20.922 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/eeprom 2024-05-28T05:52:20.922 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/eeprom/at24.ko 2024-05-28T05:52:20.922 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-05-28T05:52:20.922 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/eeprom/max6875.ko 2024-05-28T05:52:20.922 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/enclosure.ko 2024-05-28T05:52:20.922 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/hpilo.ko 2024-05-28T05:52:20.922 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/isl29003.ko 2024-05-28T05:52:20.922 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/isl29020.ko 2024-05-28T05:52:20.922 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/lis3lv02d 2024-05-28T05:52:20.922 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-05-28T05:52:20.922 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-05-28T05:52:20.922 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/mei 2024-05-28T05:52:20.923 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/mei/mei-me.ko 2024-05-28T05:52:20.923 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/mei/mei-txe.ko 2024-05-28T05:52:20.923 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/mei/mei.ko 2024-05-28T05:52:20.923 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/sgi-gru 2024-05-28T05:52:20.923 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/sgi-gru/gru.ko 2024-05-28T05:52:20.923 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/sgi-xp 2024-05-28T05:52:20.923 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/sgi-xp/xp.ko 2024-05-28T05:52:20.923 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/sgi-xp/xpc.ko 2024-05-28T05:52:20.923 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-05-28T05:52:20.923 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/tifm_7xx1.ko 2024-05-28T05:52:20.923 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/tifm_core.ko 2024-05-28T05:52:20.923 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/tsl2550.ko 2024-05-28T05:52:20.923 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/vmw_balloon.ko 2024-05-28T05:52:20.923 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/vmw_vmci 2024-05-28T05:52:20.923 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-05-28T05:52:20.924 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net 2024-05-28T05:52:20.924 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/bonding 2024-05-28T05:52:20.924 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/bonding/bonding.ko 2024-05-28T05:52:20.924 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dsa 2024-05-28T05:52:20.924 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dsa/b53 2024-05-28T05:52:20.924 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dsa/b53/b53_common.ko 2024-05-28T05:52:20.924 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dsa/bcm-sf2.ko 2024-05-28T05:52:20.924 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dsa/mv88e6060.ko 2024-05-28T05:52:20.924 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dsa/mv88e6xxx 2024-05-28T05:52:20.924 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-05-28T05:52:20.924 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dummy.ko 2024-05-28T05:52:20.924 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/eql.ko 2024-05-28T05:52:20.924 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet 2024-05-28T05:52:20.924 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/3com 2024-05-28T05:52:20.925 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-05-28T05:52:20.925 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-05-28T05:52:20.925 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-05-28T05:52:20.925 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-05-28T05:52:20.925 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/8390 2024-05-28T05:52:20.925 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/8390/8390.ko 2024-05-28T05:52:20.925 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-05-28T05:52:20.925 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-05-28T05:52:20.925 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-05-28T05:52:20.925 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/adaptec 2024-05-28T05:52:20.925 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-05-28T05:52:20.925 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/agere 2024-05-28T05:52:20.925 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/agere/et131x.ko 2024-05-28T05:52:20.925 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/alteon 2024-05-28T05:52:20.926 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-05-28T05:52:20.926 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/altera 2024-05-28T05:52:20.926 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-05-28T05:52:20.926 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/amd 2024-05-28T05:52:20.926 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-05-28T05:52:20.926 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-05-28T05:52:20.926 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-05-28T05:52:20.926 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros 2024-05-28T05:52:20.926 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/alx 2024-05-28T05:52:20.926 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-05-28T05:52:20.926 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/atl1c 2024-05-28T05:52:20.926 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-05-28T05:52:20.926 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/atl1e 2024-05-28T05:52:20.927 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-05-28T05:52:20.927 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/atlx 2024-05-28T05:52:20.927 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-05-28T05:52:20.927 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-05-28T05:52:20.927 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom 2024-05-28T05:52:20.927 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-05-28T05:52:20.927 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-05-28T05:52:20.927 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-05-28T05:52:20.927 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-05-28T05:52:20.927 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-05-28T05:52:20.927 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/genet 2024-05-28T05:52:20.928 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-05-28T05:52:20.928 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-05-28T05:52:20.928 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/brocade 2024-05-28T05:52:20.928 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/brocade/bna 2024-05-28T05:52:20.928 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-05-28T05:52:20.928 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio 2024-05-28T05:52:20.928 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb 2024-05-28T05:52:20.928 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-05-28T05:52:20.928 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-05-28T05:52:20.928 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-05-28T05:52:20.928 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-05-28T05:52:20.928 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-05-28T05:52:20.928 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-05-28T05:52:20.929 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-05-28T05:52:20.929 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-05-28T05:52:20.929 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-05-28T05:52:20.929 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/cisco 2024-05-28T05:52:20.929 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/cisco/enic 2024-05-28T05:52:20.929 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-05-28T05:52:20.929 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec 2024-05-28T05:52:20.929 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/tulip 2024-05-28T05:52:20.929 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-05-28T05:52:20.929 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-05-28T05:52:20.929 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-05-28T05:52:20.929 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-05-28T05:52:20.929 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-05-28T05:52:20.930 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-05-28T05:52:20.930 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dlink 2024-05-28T05:52:20.930 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-05-28T05:52:20.930 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-05-28T05:52:20.930 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dnet.ko 2024-05-28T05:52:20.930 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/emulex 2024-05-28T05:52:20.930 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/emulex/benet 2024-05-28T05:52:20.930 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-05-28T05:52:20.930 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/ethoc.ko 2024-05-28T05:52:20.930 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/fealnx.ko 2024-05-28T05:52:20.930 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel 2024-05-28T05:52:20.930 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/e100.ko 2024-05-28T05:52:20.930 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/e1000 2024-05-28T05:52:20.930 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-05-28T05:52:20.931 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/e1000e 2024-05-28T05:52:20.931 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-05-28T05:52:20.931 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/fm10k 2024-05-28T05:52:20.931 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-05-28T05:52:20.931 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/i40e 2024-05-28T05:52:20.931 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-05-28T05:52:20.931 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/iavf 2024-05-28T05:52:20.931 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-05-28T05:52:20.931 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/igb 2024-05-28T05:52:20.931 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-05-28T05:52:20.931 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/igbvf 2024-05-28T05:52:20.931 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-05-28T05:52:20.931 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/ixgbe 2024-05-28T05:52:20.932 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-05-28T05:52:20.932 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/ixgbevf 2024-05-28T05:52:20.932 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-05-28T05:52:20.932 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/jme.ko 2024-05-28T05:52:20.932 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/marvell 2024-05-28T05:52:20.932 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-05-28T05:52:20.932 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/marvell/skge.ko 2024-05-28T05:52:20.932 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-05-28T05:52:20.932 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/mellanox 2024-05-28T05:52:20.932 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/mellanox/mlx4 2024-05-28T05:52:20.932 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-05-28T05:52:20.932 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-05-28T05:52:20.932 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/mellanox/mlx5 2024-05-28T05:52:20.933 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-05-28T05:52:20.933 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-05-28T05:52:20.933 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/micrel 2024-05-28T05:52:20.933 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-05-28T05:52:20.933 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/myricom 2024-05-28T05:52:20.933 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/myricom/myri10ge 2024-05-28T05:52:20.933 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-05-28T05:52:20.933 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/natsemi 2024-05-28T05:52:20.933 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-05-28T05:52:20.933 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-05-28T05:52:20.933 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/neterion 2024-05-28T05:52:20.933 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-05-28T05:52:20.933 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/nvidia 2024-05-28T05:52:20.934 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-05-28T05:52:20.934 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/packetengines 2024-05-28T05:52:20.934 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-05-28T05:52:20.934 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-05-28T05:52:20.934 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/qlogic 2024-05-28T05:52:20.934 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/qlogic/netxen 2024-05-28T05:52:20.934 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-05-28T05:52:20.934 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-05-28T05:52:20.934 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-05-28T05:52:20.934 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-05-28T05:52:20.934 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/rdc 2024-05-28T05:52:20.935 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-05-28T05:52:20.935 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/realtek 2024-05-28T05:52:20.935 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-05-28T05:52:20.935 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-05-28T05:52:20.935 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/realtek/atp.ko 2024-05-28T05:52:20.935 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-05-28T05:52:20.935 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/rocker 2024-05-28T05:52:20.935 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-05-28T05:52:20.935 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sfc 2024-05-28T05:52:20.935 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-05-28T05:52:20.935 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/silan 2024-05-28T05:52:20.935 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-05-28T05:52:20.935 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sis 2024-05-28T05:52:20.935 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sis/sis190.ko 2024-05-28T05:52:20.936 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sis/sis900.ko 2024-05-28T05:52:20.936 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/smsc 2024-05-28T05:52:20.936 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-05-28T05:52:20.936 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-05-28T05:52:20.936 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-05-28T05:52:20.936 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-05-28T05:52:20.936 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/stmicro 2024-05-28T05:52:20.936 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/stmicro/stmmac 2024-05-28T05:52:20.936 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-05-28T05:52:20.936 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-05-28T05:52:20.936 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sun 2024-05-28T05:52:20.936 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sun/cassini.ko 2024-05-28T05:52:20.936 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sun/niu.ko 2024-05-28T05:52:20.936 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sun/sungem.ko 2024-05-28T05:52:20.937 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-05-28T05:52:20.937 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/tehuti 2024-05-28T05:52:20.937 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-05-28T05:52:20.937 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/ti 2024-05-28T05:52:20.937 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/ti/tlan.ko 2024-05-28T05:52:20.937 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/via 2024-05-28T05:52:20.937 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-05-28T05:52:20.937 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-05-28T05:52:20.937 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/wiznet 2024-05-28T05:52:20.937 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-05-28T05:52:20.937 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-05-28T05:52:20.937 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/xircom 2024-05-28T05:52:20.937 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-05-28T05:52:20.937 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/geneve.ko 2024-05-28T05:52:20.938 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/hyperv 2024-05-28T05:52:20.938 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-05-28T05:52:20.938 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ieee802154 2024-05-28T05:52:20.938 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ieee802154/fakelb.ko 2024-05-28T05:52:20.938 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ifb.ko 2024-05-28T05:52:20.938 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ipvlan 2024-05-28T05:52:20.938 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ipvlan/ipvlan.ko 2024-05-28T05:52:20.938 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/macvlan.ko 2024-05-28T05:52:20.938 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/macvtap.ko 2024-05-28T05:52:20.938 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/mdio 2024-05-28T05:52:20.938 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/mdio.ko 2024-05-28T05:52:20.938 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-05-28T05:52:20.938 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-05-28T05:52:20.939 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/mdio/mdio-regmap.ko 2024-05-28T05:52:20.939 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/mii.ko 2024-05-28T05:52:20.939 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/net_failover.ko 2024-05-28T05:52:20.939 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/netconsole.ko 2024-05-28T05:52:20.939 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/nlmon.ko 2024-05-28T05:52:20.939 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/pcs 2024-05-28T05:52:20.939 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/pcs/pcs-lynx.ko 2024-05-28T05:52:20.939 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-05-28T05:52:20.939 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy 2024-05-28T05:52:20.939 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/amd.ko 2024-05-28T05:52:20.939 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-05-28T05:52:20.939 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-05-28T05:52:20.939 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/bcm7xxx.ko 2024-05-28T05:52:20.940 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/bcm87xx.ko 2024-05-28T05:52:20.940 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/broadcom.ko 2024-05-28T05:52:20.940 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/cicada.ko 2024-05-28T05:52:20.940 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/davicom.ko 2024-05-28T05:52:20.940 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/dp83640.ko 2024-05-28T05:52:20.940 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/et1011c.ko 2024-05-28T05:52:20.940 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/icplus.ko 2024-05-28T05:52:20.940 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/lxt.ko 2024-05-28T05:52:20.940 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/marvell.ko 2024-05-28T05:52:20.940 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/micrel.ko 2024-05-28T05:52:20.940 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/national.ko 2024-05-28T05:52:20.940 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/phylink.ko 2024-05-28T05:52:20.940 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/qsemi.ko 2024-05-28T05:52:20.941 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/realtek.ko 2024-05-28T05:52:20.941 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/smsc.ko 2024-05-28T05:52:20.941 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/ste10Xp.ko 2024-05-28T05:52:20.941 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/vitesse.ko 2024-05-28T05:52:20.941 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp 2024-05-28T05:52:20.941 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/bsd_comp.ko 2024-05-28T05:52:20.941 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/ppp_async.ko 2024-05-28T05:52:20.941 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/ppp_deflate.ko 2024-05-28T05:52:20.941 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/ppp_generic.ko 2024-05-28T05:52:20.941 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/ppp_mppe.ko 2024-05-28T05:52:20.941 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/ppp_synctty.ko 2024-05-28T05:52:20.941 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/pppoe.ko 2024-05-28T05:52:20.941 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/pppox.ko 2024-05-28T05:52:20.942 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/pptp.ko 2024-05-28T05:52:20.942 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/slip 2024-05-28T05:52:20.942 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/slip/slhc.ko 2024-05-28T05:52:20.942 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/slip/slip.ko 2024-05-28T05:52:20.942 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/sungem_phy.ko 2024-05-28T05:52:20.942 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/tap.ko 2024-05-28T05:52:20.942 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/team 2024-05-28T05:52:20.942 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/team/team.ko 2024-05-28T05:52:20.942 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/team/team_mode_activebackup.ko 2024-05-28T05:52:20.942 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/team/team_mode_broadcast.ko 2024-05-28T05:52:20.942 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-05-28T05:52:20.942 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/team/team_mode_random.ko 2024-05-28T05:52:20.942 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-05-28T05:52:20.942 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/tun.ko 2024-05-28T05:52:20.943 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/veth.ko 2024-05-28T05:52:20.943 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/virtio_net.ko 2024-05-28T05:52:20.943 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/vmxnet3 2024-05-28T05:52:20.943 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-05-28T05:52:20.943 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/vxlan 2024-05-28T05:52:20.943 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/vxlan/vxlan.ko 2024-05-28T05:52:20.943 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/xen-netback 2024-05-28T05:52:20.943 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/xen-netback/xen-netback.ko 2024-05-28T05:52:20.943 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/xen-netfront.ko 2024-05-28T05:52:20.943 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ntb 2024-05-28T05:52:20.943 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ntb/ntb.ko 2024-05-28T05:52:20.943 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/nvme 2024-05-28T05:52:20.943 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/nvme/host 2024-05-28T05:52:20.943 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/nvme/host/nvme-core.ko 2024-05-28T05:52:20.944 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/nvme/host/nvme.ko 2024-05-28T05:52:20.944 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/parport 2024-05-28T05:52:20.944 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/parport/parport.ko 2024-05-28T05:52:20.944 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/parport/parport_cs.ko 2024-05-28T05:52:20.944 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/parport/parport_pc.ko 2024-05-28T05:52:20.944 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/parport/parport_serial.ko 2024-05-28T05:52:20.944 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pci 2024-05-28T05:52:20.944 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pci/hotplug 2024-05-28T05:52:20.944 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-05-28T05:52:20.944 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pci/pcie 2024-05-28T05:52:20.944 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pci/pcie/aer_inject.ko 2024-05-28T05:52:20.944 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pci/xen-pcifront.ko 2024-05-28T05:52:20.944 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pcmcia 2024-05-28T05:52:20.945 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pcmcia/i82092.ko 2024-05-28T05:52:20.945 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pcmcia/pd6729.ko 2024-05-28T05:52:20.945 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pcmcia/yenta_socket.ko 2024-05-28T05:52:20.945 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pinctrl 2024-05-28T05:52:20.945 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pinctrl/intel 2024-05-28T05:52:20.945 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-05-28T05:52:20.945 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform 2024-05-28T05:52:20.945 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86 2024-05-28T05:52:20.945 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/acer-wmi.ko 2024-05-28T05:52:20.945 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/acerhdf.ko 2024-05-28T05:52:20.945 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/apple-gmux.ko 2024-05-28T05:52:20.945 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/asus-laptop.ko 2024-05-28T05:52:20.945 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-05-28T05:52:20.945 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/asus-wmi.ko 2024-05-28T05:52:20.946 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/classmate-laptop.ko 2024-05-28T05:52:20.946 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-05-28T05:52:20.946 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-05-28T05:52:20.946 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-05-28T05:52:20.946 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-05-28T05:52:20.946 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/hdaps.ko 2024-05-28T05:52:20.946 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/intel 2024-05-28T05:52:20.946 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-05-28T05:52:20.946 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/intel_ips.ko 2024-05-28T05:52:20.946 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/msi-wmi.ko 2024-05-28T05:52:20.946 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/mxm-wmi.ko 2024-05-28T05:52:20.946 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-05-28T05:52:20.946 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/samsung-laptop.ko 2024-05-28T05:52:20.946 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/samsung-q10.ko 2024-05-28T05:52:20.947 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/topstar-laptop.ko 2024-05-28T05:52:20.947 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-05-28T05:52:20.947 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/toshiba_haps.ko 2024-05-28T05:52:20.947 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/wmi-bmof.ko 2024-05-28T05:52:20.947 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/wmi.ko 2024-05-28T05:52:20.947 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/powercap 2024-05-28T05:52:20.947 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/powercap/intel_rapl_common.ko 2024-05-28T05:52:20.947 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/powercap/intel_rapl_msr.ko 2024-05-28T05:52:20.947 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pps 2024-05-28T05:52:20.947 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pps/clients 2024-05-28T05:52:20.947 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pps/clients/pps-gpio.ko 2024-05-28T05:52:20.947 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pps/clients/pps-ldisc.ko 2024-05-28T05:52:20.947 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pps/clients/pps_parport.ko 2024-05-28T05:52:20.947 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pps/pps_core.ko 2024-05-28T05:52:20.947 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ptp 2024-05-28T05:52:20.948 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ptp/ptp.ko 2024-05-28T05:52:20.948 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ptp/ptp_kvm.ko 2024-05-28T05:52:20.948 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ras 2024-05-28T05:52:20.948 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ras/amd 2024-05-28T05:52:20.948 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ras/amd/atl 2024-05-28T05:52:20.948 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ras/amd/atl/amd_atl.ko 2024-05-28T05:52:20.948 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ras/amd/fmpm.ko 2024-05-28T05:52:20.948 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc 2024-05-28T05:52:20.948 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-bq32k.ko 2024-05-28T05:52:20.948 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds1286.ko 2024-05-28T05:52:20.948 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds1307.ko 2024-05-28T05:52:20.948 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds1374.ko 2024-05-28T05:52:20.948 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds1511.ko 2024-05-28T05:52:20.948 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds1553.ko 2024-05-28T05:52:20.949 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds1672.ko 2024-05-28T05:52:20.949 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds1742.ko 2024-05-28T05:52:20.949 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds2404.ko 2024-05-28T05:52:20.949 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds3232.ko 2024-05-28T05:52:20.949 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-em3027.ko 2024-05-28T05:52:20.949 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-fm3130.ko 2024-05-28T05:52:20.949 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-isl12022.ko 2024-05-28T05:52:20.949 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-isl1208.ko 2024-05-28T05:52:20.949 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-m41t80.ko 2024-05-28T05:52:20.949 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-m48t35.ko 2024-05-28T05:52:20.949 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-m48t59.ko 2024-05-28T05:52:20.949 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-max6900.ko 2024-05-28T05:52:20.949 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-msm6242.ko 2024-05-28T05:52:20.949 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-pcf2127.ko 2024-05-28T05:52:20.950 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-pcf85063.ko 2024-05-28T05:52:20.950 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-pcf8523.ko 2024-05-28T05:52:20.950 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-pcf8563.ko 2024-05-28T05:52:20.950 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-pcf8583.ko 2024-05-28T05:52:20.950 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-rp5c01.ko 2024-05-28T05:52:20.950 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-rs5c372.ko 2024-05-28T05:52:20.950 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-05-28T05:52:20.950 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-rx8025.ko 2024-05-28T05:52:20.950 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-rx8581.ko 2024-05-28T05:52:20.950 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-05-28T05:52:20.950 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-x1205.ko 2024-05-28T05:52:20.950 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi 2024-05-28T05:52:20.951 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/3w-9xxx.ko 2024-05-28T05:52:20.951 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/3w-sas.ko 2024-05-28T05:52:20.951 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/3w-xxxx.ko 2024-05-28T05:52:20.951 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/BusLogic.ko 2024-05-28T05:52:20.951 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/a100u2w.ko 2024-05-28T05:52:20.951 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/aacraid 2024-05-28T05:52:20.951 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/aacraid/aacraid.ko 2024-05-28T05:52:20.951 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/advansys.ko 2024-05-28T05:52:20.951 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/aic7xxx 2024-05-28T05:52:20.951 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-05-28T05:52:20.951 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-05-28T05:52:20.951 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/am53c974.ko 2024-05-28T05:52:20.951 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/arcmsr 2024-05-28T05:52:20.952 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-05-28T05:52:20.952 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/atp870u.ko 2024-05-28T05:52:20.952 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/be2iscsi 2024-05-28T05:52:20.952 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-05-28T05:52:20.952 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/bfa 2024-05-28T05:52:20.952 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/bfa/bfa.ko 2024-05-28T05:52:20.952 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/bnx2fc 2024-05-28T05:52:20.952 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-05-28T05:52:20.952 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/bnx2i 2024-05-28T05:52:20.952 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-05-28T05:52:20.952 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/ch.ko 2024-05-28T05:52:20.952 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/csiostor 2024-05-28T05:52:20.952 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/csiostor/csiostor.ko 2024-05-28T05:52:20.953 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/cxgbi 2024-05-28T05:52:20.953 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/cxgbi/cxgb3i 2024-05-28T05:52:20.953 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-05-28T05:52:20.953 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/cxgbi/cxgb4i 2024-05-28T05:52:20.953 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-05-28T05:52:20.953 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-05-28T05:52:20.953 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/dc395x.ko 2024-05-28T05:52:20.953 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/device_handler 2024-05-28T05:52:20.953 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-05-28T05:52:20.953 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-05-28T05:52:20.953 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-05-28T05:52:20.953 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-05-28T05:52:20.954 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/dmx3191d.ko 2024-05-28T05:52:20.954 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/esas2r 2024-05-28T05:52:20.954 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/esas2r/esas2r.ko 2024-05-28T05:52:20.954 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/esp_scsi.ko 2024-05-28T05:52:20.954 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/fcoe 2024-05-28T05:52:20.954 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/fcoe/fcoe.ko 2024-05-28T05:52:20.954 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-05-28T05:52:20.954 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/fnic 2024-05-28T05:52:20.954 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/fnic/fnic.ko 2024-05-28T05:52:20.954 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/hpsa.ko 2024-05-28T05:52:20.954 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/hptiop.ko 2024-05-28T05:52:20.954 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/hv_storvsc.ko 2024-05-28T05:52:20.954 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/initio.ko 2024-05-28T05:52:20.955 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/ipr.ko 2024-05-28T05:52:20.955 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/ips.ko 2024-05-28T05:52:20.955 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/isci 2024-05-28T05:52:20.955 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/isci/isci.ko 2024-05-28T05:52:20.955 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-05-28T05:52:20.955 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/iscsi_tcp.ko 2024-05-28T05:52:20.955 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/libfc 2024-05-28T05:52:20.955 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/libfc/libfc.ko 2024-05-28T05:52:20.955 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/libiscsi.ko 2024-05-28T05:52:20.955 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/libiscsi_tcp.ko 2024-05-28T05:52:20.955 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/libsas 2024-05-28T05:52:20.955 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/libsas/libsas.ko 2024-05-28T05:52:20.955 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/lpfc 2024-05-28T05:52:20.956 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/lpfc/lpfc.ko 2024-05-28T05:52:20.956 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/megaraid 2024-05-28T05:52:20.956 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/megaraid.ko 2024-05-28T05:52:20.956 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-05-28T05:52:20.956 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-05-28T05:52:20.956 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-05-28T05:52:20.956 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/mpt3sas 2024-05-28T05:52:20.956 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-05-28T05:52:20.956 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/mvsas 2024-05-28T05:52:20.956 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/mvsas/mvsas.ko 2024-05-28T05:52:20.956 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/mvumi.ko 2024-05-28T05:52:20.956 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/pm8001 2024-05-28T05:52:20.956 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-05-28T05:52:20.957 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/pmcraid.ko 2024-05-28T05:52:20.957 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/qla1280.ko 2024-05-28T05:52:20.957 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/qla2xxx 2024-05-28T05:52:20.957 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-05-28T05:52:20.957 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-05-28T05:52:20.957 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/qla4xxx 2024-05-28T05:52:20.957 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-05-28T05:52:20.957 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/raid_class.ko 2024-05-28T05:52:20.957 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/scsi_debug.ko 2024-05-28T05:52:20.957 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/scsi_transport_fc.ko 2024-05-28T05:52:20.957 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-05-28T05:52:20.957 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/scsi_transport_sas.ko 2024-05-28T05:52:20.957 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/scsi_transport_spi.ko 2024-05-28T05:52:20.958 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/scsi_transport_srp.ko 2024-05-28T05:52:20.958 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/ses.ko 2024-05-28T05:52:20.958 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/st.ko 2024-05-28T05:52:20.958 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/stex.ko 2024-05-28T05:52:20.958 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/sym53c8xx_2 2024-05-28T05:52:20.958 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-05-28T05:52:20.958 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/virtio_scsi.ko 2024-05-28T05:52:20.958 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/vmw_pvscsi.ko 2024-05-28T05:52:20.958 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/wd719x.ko 2024-05-28T05:52:20.958 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ssb 2024-05-28T05:52:20.958 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ssb/ssb.ko 2024-05-28T05:52:20.958 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target 2024-05-28T05:52:20.959 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/iscsi 2024-05-28T05:52:20.959 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-05-28T05:52:20.959 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/loopback 2024-05-28T05:52:20.959 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/loopback/tcm_loop.ko 2024-05-28T05:52:20.959 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/target_core_file.ko 2024-05-28T05:52:20.959 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/target_core_iblock.ko 2024-05-28T05:52:20.959 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/target_core_mod.ko 2024-05-28T05:52:20.959 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/target_core_pscsi.ko 2024-05-28T05:52:20.959 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/target_core_user.ko 2024-05-28T05:52:20.959 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/tcm_fc 2024-05-28T05:52:20.959 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-05-28T05:52:20.959 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal 2024-05-28T05:52:20.960 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel 2024-05-28T05:52:20.960 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal 2024-05-28T05:52:20.960 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-05-28T05:52:20.960 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-05-28T05:52:20.960 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-05-28T05:52:20.960 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-05-28T05:52:20.960 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-05-28T05:52:20.960 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-05-28T05:52:20.960 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-05-28T05:52:20.960 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-05-28T05:52:20.960 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-05-28T05:52:20.960 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-05-28T05:52:20.960 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-05-28T05:52:20.960 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-05-28T05:52:20.961 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-05-28T05:52:20.961 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-05-28T05:52:20.961 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-05-28T05:52:20.961 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-05-28T05:52:20.961 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-05-28T05:52:20.961 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-05-28T05:52:20.961 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty 2024-05-28T05:52:20.961 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/ipwireless 2024-05-28T05:52:20.961 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-05-28T05:52:20.961 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/n_gsm.ko 2024-05-28T05:52:20.961 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/n_hdlc.ko 2024-05-28T05:52:20.961 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/nozomi.ko 2024-05-28T05:52:20.961 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial 2024-05-28T05:52:20.962 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial/8250 2024-05-28T05:52:20.962 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-05-28T05:52:20.962 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial/arc_uart.ko 2024-05-28T05:52:20.962 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial/jsm 2024-05-28T05:52:20.962 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial/jsm/jsm.ko 2024-05-28T05:52:20.962 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ufs 2024-05-28T05:52:20.962 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ufs/core 2024-05-28T05:52:20.962 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ufs/core/ufshcd-core.ko 2024-05-28T05:52:20.962 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ufs/host 2024-05-28T05:52:20.962 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-05-28T05:52:20.962 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/uio 2024-05-28T05:52:20.962 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/uio/uio.ko 2024-05-28T05:52:20.962 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/uio/uio_aec.ko 2024-05-28T05:52:20.962 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/uio/uio_cif.ko 2024-05-28T05:52:20.963 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/uio/uio_pci_generic.ko 2024-05-28T05:52:20.963 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/uio/uio_sercos3.ko 2024-05-28T05:52:20.963 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb 2024-05-28T05:52:20.963 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/class 2024-05-28T05:52:20.963 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/class/cdc-acm.ko 2024-05-28T05:52:20.963 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/class/cdc-wdm.ko 2024-05-28T05:52:20.963 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/class/usblp.ko 2024-05-28T05:52:20.963 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/class/usbtmc.ko 2024-05-28T05:52:20.963 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/host 2024-05-28T05:52:20.963 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/host/sl811-hcd.ko 2024-05-28T05:52:20.963 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/image 2024-05-28T05:52:20.963 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/image/mdc800.ko 2024-05-28T05:52:20.963 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/image/microtek.ko 2024-05-28T05:52:20.963 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc 2024-05-28T05:52:20.963 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/adutux.ko 2024-05-28T05:52:20.964 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/appledisplay.ko 2024-05-28T05:52:20.964 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/emi26.ko 2024-05-28T05:52:20.964 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/emi62.ko 2024-05-28T05:52:20.964 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/ezusb.ko 2024-05-28T05:52:20.964 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/idmouse.ko 2024-05-28T05:52:20.964 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/iowarrior.ko 2024-05-28T05:52:20.964 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/isight_firmware.ko 2024-05-28T05:52:20.964 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/ldusb.ko 2024-05-28T05:52:20.964 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/legousbtower.ko 2024-05-28T05:52:20.964 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/sisusbvga 2024-05-28T05:52:20.964 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-05-28T05:52:20.964 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/trancevibrator.ko 2024-05-28T05:52:20.964 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/usb3503.ko 2024-05-28T05:52:20.964 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/usblcd.ko 2024-05-28T05:52:20.965 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/usbsevseg.ko 2024-05-28T05:52:20.965 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/uss720.ko 2024-05-28T05:52:20.965 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/yurex.ko 2024-05-28T05:52:20.965 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/phy 2024-05-28T05:52:20.965 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/phy/phy-generic.ko 2024-05-28T05:52:20.965 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial 2024-05-28T05:52:20.965 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/aircable.ko 2024-05-28T05:52:20.965 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ark3116.ko 2024-05-28T05:52:20.965 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/belkin_sa.ko 2024-05-28T05:52:20.965 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ch341.ko 2024-05-28T05:52:20.965 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/cp210x.ko 2024-05-28T05:52:20.965 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/cyberjack.ko 2024-05-28T05:52:20.965 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/cypress_m8.ko 2024-05-28T05:52:20.965 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/digi_acceleport.ko 2024-05-28T05:52:20.965 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/empeg.ko 2024-05-28T05:52:20.966 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ftdi_sio.ko 2024-05-28T05:52:20.966 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/garmin_gps.ko 2024-05-28T05:52:20.966 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/io_edgeport.ko 2024-05-28T05:52:20.966 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/io_ti.ko 2024-05-28T05:52:20.966 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ipaq.ko 2024-05-28T05:52:20.966 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ipw.ko 2024-05-28T05:52:20.966 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ir-usb.ko 2024-05-28T05:52:20.966 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-05-28T05:52:20.966 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/keyspan.ko 2024-05-28T05:52:20.966 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/keyspan_pda.ko 2024-05-28T05:52:20.966 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/kl5kusb105.ko 2024-05-28T05:52:20.966 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/kobil_sct.ko 2024-05-28T05:52:20.966 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/mct_u232.ko 2024-05-28T05:52:20.966 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/mos7720.ko 2024-05-28T05:52:20.967 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/mos7840.ko 2024-05-28T05:52:20.967 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/navman.ko 2024-05-28T05:52:20.967 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/omninet.ko 2024-05-28T05:52:20.967 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/opticon.ko 2024-05-28T05:52:20.967 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/option.ko 2024-05-28T05:52:20.967 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/oti6858.ko 2024-05-28T05:52:20.967 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/pl2303.ko 2024-05-28T05:52:20.967 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/qcaux.ko 2024-05-28T05:52:20.967 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/qcserial.ko 2024-05-28T05:52:20.967 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/quatech2.ko 2024-05-28T05:52:20.967 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/safe_serial.ko 2024-05-28T05:52:20.967 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/sierra.ko 2024-05-28T05:52:20.967 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/spcp8x5.ko 2024-05-28T05:52:20.967 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ssu100.ko 2024-05-28T05:52:20.968 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/symbolserial.ko 2024-05-28T05:52:20.968 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-05-28T05:52:20.968 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-05-28T05:52:20.968 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/usb_debug.ko 2024-05-28T05:52:20.968 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/usb_wwan.ko 2024-05-28T05:52:20.968 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/visor.ko 2024-05-28T05:52:20.968 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/whiteheat.ko 2024-05-28T05:52:20.968 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/xsens_mt.ko 2024-05-28T05:52:20.968 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage 2024-05-28T05:52:20.968 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/uas.ko 2024-05-28T05:52:20.968 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-alauda.ko 2024-05-28T05:52:20.968 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-cypress.ko 2024-05-28T05:52:20.968 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-datafab.ko 2024-05-28T05:52:20.968 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-05-28T05:52:20.968 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-freecom.ko 2024-05-28T05:52:20.969 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-isd200.ko 2024-05-28T05:52:20.969 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-05-28T05:52:20.969 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-karma.ko 2024-05-28T05:52:20.969 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-onetouch.ko 2024-05-28T05:52:20.969 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-realtek.ko 2024-05-28T05:52:20.969 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-sddr09.ko 2024-05-28T05:52:20.969 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-sddr55.ko 2024-05-28T05:52:20.969 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-usbat.ko 2024-05-28T05:52:20.969 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/usb-storage.ko 2024-05-28T05:52:20.969 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/usbip 2024-05-28T05:52:20.969 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/usbip/usbip-core.ko 2024-05-28T05:52:20.969 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/usbip/usbip-host.ko 2024-05-28T05:52:20.969 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-05-28T05:52:20.970 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vfio 2024-05-28T05:52:20.970 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vfio/pci 2024-05-28T05:52:20.970 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-05-28T05:52:20.970 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vfio/pci/vfio-pci.ko 2024-05-28T05:52:20.970 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vfio/vfio.ko 2024-05-28T05:52:20.970 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-05-28T05:52:20.970 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vhost 2024-05-28T05:52:20.970 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vhost/vhost.ko 2024-05-28T05:52:20.970 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vhost/vhost_iotlb.ko 2024-05-28T05:52:20.970 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vhost/vhost_net.ko 2024-05-28T05:52:20.970 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vhost/vhost_scsi.ko 2024-05-28T05:52:20.970 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video 2024-05-28T05:52:20.970 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight 2024-05-28T05:52:20.971 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/apple_bl.ko 2024-05-28T05:52:20.971 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/lcd.ko 2024-05-28T05:52:20.971 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/platform_lcd.ko 2024-05-28T05:52:20.971 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/virtio 2024-05-28T05:52:20.971 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/virtio/virtio.ko 2024-05-28T05:52:20.971 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/virtio/virtio_balloon.ko 2024-05-28T05:52:20.971 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/virtio/virtio_mmio.ko 2024-05-28T05:52:20.971 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/virtio/virtio_pci.ko 2024-05-28T05:52:20.971 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-05-28T05:52:20.971 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-05-28T05:52:20.971 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/virtio/virtio_ring.ko 2024-05-28T05:52:20.971 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1 2024-05-28T05:52:20.971 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/masters 2024-05-28T05:52:20.971 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/masters/ds2482.ko 2024-05-28T05:52:20.972 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/masters/ds2490.ko 2024-05-28T05:52:20.972 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves 2024-05-28T05:52:20.972 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-05-28T05:52:20.972 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-05-28T05:52:20.972 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-05-28T05:52:20.972 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-05-28T05:52:20.972 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-05-28T05:52:20.972 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-05-28T05:52:20.972 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-05-28T05:52:20.972 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-05-28T05:52:20.972 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-05-28T05:52:20.972 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_smem.ko 2024-05-28T05:52:20.972 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_therm.ko 2024-05-28T05:52:20.972 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/wire.ko 2024-05-28T05:52:20.972 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog 2024-05-28T05:52:20.972 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/alim1535_wdt.ko 2024-05-28T05:52:20.973 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/alim7101_wdt.ko 2024-05-28T05:52:20.973 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/f71808e_wdt.ko 2024-05-28T05:52:20.973 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/hpwdt.ko 2024-05-28T05:52:20.973 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/i6300esb.ko 2024-05-28T05:52:20.973 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-05-28T05:52:20.973 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/iTCO_wdt.ko 2024-05-28T05:52:20.973 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/ib700wdt.ko 2024-05-28T05:52:20.973 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/ibmasr.ko 2024-05-28T05:52:20.973 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-05-28T05:52:20.973 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/it8712f_wdt.ko 2024-05-28T05:52:20.973 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/it87_wdt.ko 2024-05-28T05:52:20.973 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/machzwd.ko 2024-05-28T05:52:20.973 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/nv_tco.ko 2024-05-28T05:52:20.973 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/pcwd_pci.ko 2024-05-28T05:52:20.973 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/pcwd_usb.ko 2024-05-28T05:52:20.974 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-05-28T05:52:20.974 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/sch311x_wdt.ko 2024-05-28T05:52:20.974 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/softdog.ko 2024-05-28T05:52:20.974 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/sp5100_tco.ko 2024-05-28T05:52:20.974 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/via_wdt.ko 2024-05-28T05:52:20.974 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-05-28T05:52:20.974 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/w83877f_wdt.ko 2024-05-28T05:52:20.974 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/w83977f_wdt.ko 2024-05-28T05:52:20.974 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/wdt_pci.ko 2024-05-28T05:52:20.974 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/xen_wdt.ko 2024-05-28T05:52:20.974 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen 2024-05-28T05:52:20.975 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xen-acpi-processor.ko 2024-05-28T05:52:20.975 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xen-evtchn.ko 2024-05-28T05:52:20.975 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xen-gntalloc.ko 2024-05-28T05:52:20.975 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xen-gntdev.ko 2024-05-28T05:52:20.975 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xen-pciback 2024-05-28T05:52:20.975 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-05-28T05:52:20.975 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xen-privcmd.ko 2024-05-28T05:52:20.975 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xenfs 2024-05-28T05:52:20.975 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xenfs/xenfs.ko 2024-05-28T05:52:20.975 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs 2024-05-28T05:52:20.975 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/9p 2024-05-28T05:52:20.975 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/9p/9p.ko 2024-05-28T05:52:20.975 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/affs 2024-05-28T05:52:20.975 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/affs/affs.ko 2024-05-28T05:52:20.975 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/befs 2024-05-28T05:52:20.976 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/befs/befs.ko 2024-05-28T05:52:20.976 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/binfmt_misc.ko 2024-05-28T05:52:20.976 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/btrfs 2024-05-28T05:52:20.976 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/btrfs/btrfs.ko 2024-05-28T05:52:20.976 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/cachefiles 2024-05-28T05:52:20.976 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/cachefiles/cachefiles.ko 2024-05-28T05:52:20.976 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ceph 2024-05-28T05:52:20.976 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ceph/ceph.ko 2024-05-28T05:52:20.976 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/coda 2024-05-28T05:52:20.976 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/coda/coda.ko 2024-05-28T05:52:20.976 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/cramfs 2024-05-28T05:52:20.976 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/cramfs/cramfs.ko 2024-05-28T05:52:20.976 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/dlm 2024-05-28T05:52:20.976 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/dlm/dlm.ko 2024-05-28T05:52:20.977 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ecryptfs 2024-05-28T05:52:20.977 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ecryptfs/ecryptfs.ko 2024-05-28T05:52:20.977 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/f2fs 2024-05-28T05:52:20.977 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/f2fs/f2fs.ko 2024-05-28T05:52:20.977 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/fat 2024-05-28T05:52:20.977 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/fat/fat.ko 2024-05-28T05:52:20.977 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/fat/msdos.ko 2024-05-28T05:52:20.977 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/fat/vfat.ko 2024-05-28T05:52:20.977 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/fuse 2024-05-28T05:52:20.977 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/fuse/cuse.ko 2024-05-28T05:52:20.977 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/fuse/fuse.ko 2024-05-28T05:52:20.977 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/gfs2 2024-05-28T05:52:20.977 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/gfs2/gfs2.ko 2024-05-28T05:52:20.977 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/hfs 2024-05-28T05:52:20.978 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/hfs/hfs.ko 2024-05-28T05:52:20.978 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/hfsplus 2024-05-28T05:52:20.978 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/hfsplus/hfsplus.ko 2024-05-28T05:52:20.978 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/isofs 2024-05-28T05:52:20.978 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/isofs/isofs.ko 2024-05-28T05:52:20.978 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/jfs 2024-05-28T05:52:20.978 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/jfs/jfs.ko 2024-05-28T05:52:20.978 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/lockd 2024-05-28T05:52:20.978 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/lockd/lockd.ko 2024-05-28T05:52:20.978 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/minix 2024-05-28T05:52:20.978 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/minix/minix.ko 2024-05-28T05:52:20.978 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/netfs 2024-05-28T05:52:20.978 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/netfs/netfs.ko 2024-05-28T05:52:20.978 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs 2024-05-28T05:52:20.979 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/blocklayout 2024-05-28T05:52:20.979 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-05-28T05:52:20.979 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/filelayout 2024-05-28T05:52:20.979 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-05-28T05:52:20.979 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/flexfilelayout 2024-05-28T05:52:20.979 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-05-28T05:52:20.979 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/nfs.ko 2024-05-28T05:52:20.979 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/nfsv3.ko 2024-05-28T05:52:20.979 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/nfsv4.ko 2024-05-28T05:52:20.979 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs_common 2024-05-28T05:52:20.979 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs_common/grace.ko 2024-05-28T05:52:20.979 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs_common/nfs_acl.ko 2024-05-28T05:52:20.979 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfsd 2024-05-28T05:52:20.980 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfsd/nfsd.ko 2024-05-28T05:52:20.980 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nilfs2 2024-05-28T05:52:20.980 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nilfs2/nilfs2.ko 2024-05-28T05:52:20.980 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls 2024-05-28T05:52:20.980 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-celtic.ko 2024-05-28T05:52:20.980 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-centeuro.ko 2024-05-28T05:52:20.980 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-croatian.ko 2024-05-28T05:52:20.980 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-cyrillic.ko 2024-05-28T05:52:20.980 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-gaelic.ko 2024-05-28T05:52:20.980 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-greek.ko 2024-05-28T05:52:20.980 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-iceland.ko 2024-05-28T05:52:20.980 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-inuit.ko 2024-05-28T05:52:20.980 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-roman.ko 2024-05-28T05:52:20.980 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-romanian.ko 2024-05-28T05:52:20.980 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-turkish.ko 2024-05-28T05:52:20.981 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp1250.ko 2024-05-28T05:52:20.981 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp1251.ko 2024-05-28T05:52:20.981 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp1255.ko 2024-05-28T05:52:20.981 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp737.ko 2024-05-28T05:52:20.981 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp775.ko 2024-05-28T05:52:20.981 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp850.ko 2024-05-28T05:52:20.981 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp852.ko 2024-05-28T05:52:20.981 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp855.ko 2024-05-28T05:52:20.981 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp857.ko 2024-05-28T05:52:20.981 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp860.ko 2024-05-28T05:52:20.981 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp861.ko 2024-05-28T05:52:20.981 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp862.ko 2024-05-28T05:52:20.981 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp863.ko 2024-05-28T05:52:20.981 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp864.ko 2024-05-28T05:52:20.981 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp865.ko 2024-05-28T05:52:20.981 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp866.ko 2024-05-28T05:52:20.982 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp869.ko 2024-05-28T05:52:20.982 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp874.ko 2024-05-28T05:52:20.982 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp932.ko 2024-05-28T05:52:20.982 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp936.ko 2024-05-28T05:52:20.982 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp949.ko 2024-05-28T05:52:20.982 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp950.ko 2024-05-28T05:52:20.982 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_euc-jp.ko 2024-05-28T05:52:20.982 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-1.ko 2024-05-28T05:52:20.982 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-13.ko 2024-05-28T05:52:20.982 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-14.ko 2024-05-28T05:52:20.982 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-15.ko 2024-05-28T05:52:20.982 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-2.ko 2024-05-28T05:52:20.982 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-3.ko 2024-05-28T05:52:20.983 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-4.ko 2024-05-28T05:52:20.983 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-5.ko 2024-05-28T05:52:20.983 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-6.ko 2024-05-28T05:52:20.983 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-7.ko 2024-05-28T05:52:20.983 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-9.ko 2024-05-28T05:52:20.983 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_koi8-r.ko 2024-05-28T05:52:20.983 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_koi8-ru.ko 2024-05-28T05:52:20.983 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_koi8-u.ko 2024-05-28T05:52:20.983 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_ucs2_utils.ko 2024-05-28T05:52:20.983 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_utf8.ko 2024-05-28T05:52:20.983 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2 2024-05-28T05:52:20.983 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/cluster 2024-05-28T05:52:20.983 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-05-28T05:52:20.983 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/dlm 2024-05-28T05:52:20.983 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-05-28T05:52:20.984 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/dlmfs 2024-05-28T05:52:20.984 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-05-28T05:52:20.984 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/ocfs2.ko 2024-05-28T05:52:20.984 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-05-28T05:52:20.984 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-05-28T05:52:20.984 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-05-28T05:52:20.984 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/overlayfs 2024-05-28T05:52:20.984 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/overlayfs/overlay.ko 2024-05-28T05:52:20.984 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/pstore 2024-05-28T05:52:20.984 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/pstore/ramoops.ko 2024-05-28T05:52:20.984 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/reiserfs 2024-05-28T05:52:20.984 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/reiserfs/reiserfs.ko 2024-05-28T05:52:20.985 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/romfs 2024-05-28T05:52:20.985 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/romfs/romfs.ko 2024-05-28T05:52:20.985 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/smb 2024-05-28T05:52:20.985 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/smb/client 2024-05-28T05:52:20.985 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/smb/client/cifs.ko 2024-05-28T05:52:20.985 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/smb/common 2024-05-28T05:52:20.985 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/smb/common/cifs_arc4.ko 2024-05-28T05:52:20.985 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/smb/common/cifs_md4.ko 2024-05-28T05:52:20.985 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/squashfs 2024-05-28T05:52:20.985 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/squashfs/squashfs.ko 2024-05-28T05:52:20.985 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/sysv 2024-05-28T05:52:20.985 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/sysv/sysv.ko 2024-05-28T05:52:20.985 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/udf 2024-05-28T05:52:20.986 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/udf/udf.ko 2024-05-28T05:52:20.986 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ufs 2024-05-28T05:52:20.986 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ufs/ufs.ko 2024-05-28T05:52:20.986 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/xfs 2024-05-28T05:52:20.986 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/xfs/xfs.ko 2024-05-28T05:52:20.986 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/kernel 2024-05-28T05:52:20.986 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/kernel/rcu 2024-05-28T05:52:20.986 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/kernel/rcu/rcutorture.ko 2024-05-28T05:52:20.986 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/kernel/torture.ko 2024-05-28T05:52:20.986 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/kernel/trace 2024-05-28T05:52:20.986 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-05-28T05:52:20.986 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib 2024-05-28T05:52:20.987 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/asn1_encoder.ko 2024-05-28T05:52:20.987 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/crc-itu-t.ko 2024-05-28T05:52:20.987 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/crc8.ko 2024-05-28T05:52:20.987 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/crypto 2024-05-28T05:52:20.987 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/crypto/libarc4.ko 2024-05-28T05:52:20.987 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/crypto/libdes.ko 2024-05-28T05:52:20.987 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/libcrc32c.ko 2024-05-28T05:52:20.987 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/lru_cache.ko 2024-05-28T05:52:20.987 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/lz4 2024-05-28T05:52:20.987 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/lz4/lz4_compress.ko 2024-05-28T05:52:20.987 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/lz4/lz4hc_compress.ko 2024-05-28T05:52:20.987 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/math 2024-05-28T05:52:20.987 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/math/cordic.ko 2024-05-28T05:52:20.988 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/raid6 2024-05-28T05:52:20.988 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/raid6/raid6_pq.ko 2024-05-28T05:52:20.988 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/reed_solomon 2024-05-28T05:52:20.988 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/reed_solomon/reed_solomon.ko 2024-05-28T05:52:20.988 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/ts_bm.ko 2024-05-28T05:52:20.988 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/ts_fsm.ko 2024-05-28T05:52:20.988 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/ts_kmp.ko 2024-05-28T05:52:20.988 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/zstd 2024-05-28T05:52:20.988 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/zstd/zstd_compress.ko 2024-05-28T05:52:20.988 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/mm 2024-05-28T05:52:20.988 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/mm/hwpoison-inject.ko 2024-05-28T05:52:20.988 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net 2024-05-28T05:52:20.988 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan 2024-05-28T05:52:20.988 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/6lowpan.ko 2024-05-28T05:52:20.988 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/nhc_dest.ko 2024-05-28T05:52:20.989 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/nhc_fragment.ko 2024-05-28T05:52:20.989 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/nhc_hop.ko 2024-05-28T05:52:20.989 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/nhc_ipv6.ko 2024-05-28T05:52:20.989 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/nhc_mobility.ko 2024-05-28T05:52:20.989 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/nhc_routing.ko 2024-05-28T05:52:20.989 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/nhc_udp.ko 2024-05-28T05:52:20.989 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/802 2024-05-28T05:52:20.989 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/802/garp.ko 2024-05-28T05:52:20.989 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/802/mrp.ko 2024-05-28T05:52:20.989 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/802/p8022.ko 2024-05-28T05:52:20.989 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/802/psnap.ko 2024-05-28T05:52:20.989 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/802/stp.ko 2024-05-28T05:52:20.989 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/8021q 2024-05-28T05:52:20.990 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/8021q/8021q.ko 2024-05-28T05:52:20.990 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/9p 2024-05-28T05:52:20.990 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/9p/9pnet.ko 2024-05-28T05:52:20.990 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/9p/9pnet_fd.ko 2024-05-28T05:52:20.990 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/9p/9pnet_rdma.ko 2024-05-28T05:52:20.990 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/9p/9pnet_virtio.ko 2024-05-28T05:52:20.990 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/batman-adv 2024-05-28T05:52:20.990 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/batman-adv/batman-adv.ko 2024-05-28T05:52:20.990 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge 2024-05-28T05:52:20.990 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/br_netfilter.ko 2024-05-28T05:52:20.990 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/bridge.ko 2024-05-28T05:52:20.990 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter 2024-05-28T05:52:20.990 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-05-28T05:52:20.991 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_among.ko 2024-05-28T05:52:20.991 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_arp.ko 2024-05-28T05:52:20.991 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-05-28T05:52:20.991 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-05-28T05:52:20.991 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_ip.ko 2024-05-28T05:52:20.991 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-05-28T05:52:20.991 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_limit.ko 2024-05-28T05:52:20.991 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_log.ko 2024-05-28T05:52:20.991 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_mark.ko 2024-05-28T05:52:20.991 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-05-28T05:52:20.991 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-05-28T05:52:20.991 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-05-28T05:52:20.991 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-05-28T05:52:20.991 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_snat.ko 2024-05-28T05:52:20.991 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_stp.ko 2024-05-28T05:52:20.992 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-05-28T05:52:20.992 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-05-28T05:52:20.992 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-05-28T05:52:20.992 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-05-28T05:52:20.992 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebtables.ko 2024-05-28T05:52:20.992 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-05-28T05:52:20.992 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-05-28T05:52:20.992 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-05-28T05:52:20.992 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ceph 2024-05-28T05:52:20.992 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ceph/libceph.ko 2024-05-28T05:52:20.992 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/core 2024-05-28T05:52:20.992 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/core/failover.ko 2024-05-28T05:52:20.992 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/core/pktgen.ko 2024-05-28T05:52:20.992 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dccp 2024-05-28T05:52:20.993 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dccp/dccp.ko 2024-05-28T05:52:20.993 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dccp/dccp_diag.ko 2024-05-28T05:52:20.993 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dccp/dccp_ipv4.ko 2024-05-28T05:52:20.993 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dccp/dccp_ipv6.ko 2024-05-28T05:52:20.993 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dns_resolver 2024-05-28T05:52:20.993 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dns_resolver/dns_resolver.ko 2024-05-28T05:52:20.993 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dsa 2024-05-28T05:52:20.993 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dsa/dsa_core.ko 2024-05-28T05:52:20.993 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dsa/tag_brcm.ko 2024-05-28T05:52:20.993 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dsa/tag_dsa.ko 2024-05-28T05:52:20.993 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dsa/tag_none.ko 2024-05-28T05:52:20.993 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dsa/tag_trailer.ko 2024-05-28T05:52:20.993 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ieee802154 2024-05-28T05:52:20.993 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ieee802154/6lowpan 2024-05-28T05:52:20.993 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-05-28T05:52:20.993 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ieee802154/ieee802154.ko 2024-05-28T05:52:20.994 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ieee802154/ieee802154_socket.ko 2024-05-28T05:52:20.994 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4 2024-05-28T05:52:20.994 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/ah4.ko 2024-05-28T05:52:20.994 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/esp4.ko 2024-05-28T05:52:20.994 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/fou.ko 2024-05-28T05:52:20.994 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/gre.ko 2024-05-28T05:52:20.994 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/inet_diag.ko 2024-05-28T05:52:20.994 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/ip_gre.ko 2024-05-28T05:52:20.994 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/ip_tunnel.ko 2024-05-28T05:52:20.994 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/ip_vti.ko 2024-05-28T05:52:20.994 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/ipcomp.ko 2024-05-28T05:52:20.994 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/ipip.ko 2024-05-28T05:52:20.994 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter 2024-05-28T05:52:20.995 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/arp_tables.ko 2024-05-28T05:52:20.995 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-05-28T05:52:20.995 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-05-28T05:52:20.995 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-05-28T05:52:20.995 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-05-28T05:52:20.995 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-05-28T05:52:20.995 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-05-28T05:52:20.995 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-05-28T05:52:20.995 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-05-28T05:52:20.995 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-05-28T05:52:20.995 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/iptable_security.ko 2024-05-28T05:52:20.995 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-05-28T05:52:20.995 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-05-28T05:52:20.995 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-05-28T05:52:20.996 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-05-28T05:52:20.996 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-05-28T05:52:20.996 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-05-28T05:52:20.996 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-05-28T05:52:20.996 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-05-28T05:52:20.996 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-05-28T05:52:20.996 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-05-28T05:52:20.996 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_bic.ko 2024-05-28T05:52:20.996 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_dctcp.ko 2024-05-28T05:52:20.996 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_diag.ko 2024-05-28T05:52:20.996 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_highspeed.ko 2024-05-28T05:52:20.996 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_htcp.ko 2024-05-28T05:52:20.996 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_hybla.ko 2024-05-28T05:52:20.996 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_illinois.ko 2024-05-28T05:52:20.996 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_lp.ko 2024-05-28T05:52:20.997 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_scalable.ko 2024-05-28T05:52:20.997 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_vegas.ko 2024-05-28T05:52:20.997 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_veno.ko 2024-05-28T05:52:20.997 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_westwood.ko 2024-05-28T05:52:20.997 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_yeah.ko 2024-05-28T05:52:20.997 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tunnel4.ko 2024-05-28T05:52:20.997 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/udp_diag.ko 2024-05-28T05:52:20.997 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/udp_tunnel.ko 2024-05-28T05:52:20.997 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/xfrm4_tunnel.ko 2024-05-28T05:52:20.997 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6 2024-05-28T05:52:20.997 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/ah6.ko 2024-05-28T05:52:20.997 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/esp6.ko 2024-05-28T05:52:20.997 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/fou6.ko 2024-05-28T05:52:20.997 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/ip6_tunnel.ko 2024-05-28T05:52:20.997 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-05-28T05:52:20.998 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/ip6_vti.ko 2024-05-28T05:52:20.998 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/ipcomp6.ko 2024-05-28T05:52:20.998 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter 2024-05-28T05:52:20.998 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-05-28T05:52:20.998 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-05-28T05:52:20.998 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-05-28T05:52:20.998 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-05-28T05:52:20.998 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-05-28T05:52:20.998 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-05-28T05:52:20.998 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-05-28T05:52:20.998 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-05-28T05:52:20.998 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-05-28T05:52:20.998 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-05-28T05:52:20.998 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-05-28T05:52:20.998 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-05-28T05:52:20.999 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-05-28T05:52:20.999 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-05-28T05:52:20.999 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-05-28T05:52:20.999 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-05-28T05:52:20.999 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-05-28T05:52:20.999 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-05-28T05:52:20.999 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-05-28T05:52:20.999 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-05-28T05:52:20.999 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-05-28T05:52:20.999 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-05-28T05:52:20.999 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-05-28T05:52:20.999 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-05-28T05:52:20.999 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/sit.ko 2024-05-28T05:52:20.999 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/tunnel6.ko 2024-05-28T05:52:20.999 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/xfrm6_tunnel.ko 2024-05-28T05:52:21.000 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/key 2024-05-28T05:52:21.000 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/key/af_key.ko 2024-05-28T05:52:21.000 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp 2024-05-28T05:52:21.000 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/l2tp_core.ko 2024-05-28T05:52:21.000 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/l2tp_debugfs.ko 2024-05-28T05:52:21.000 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/l2tp_eth.ko 2024-05-28T05:52:21.000 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/l2tp_ip.ko 2024-05-28T05:52:21.000 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/l2tp_ip6.ko 2024-05-28T05:52:21.000 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/l2tp_netlink.ko 2024-05-28T05:52:21.000 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/l2tp_ppp.ko 2024-05-28T05:52:21.000 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/llc 2024-05-28T05:52:21.000 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/llc/llc.ko 2024-05-28T05:52:21.000 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/mac802154 2024-05-28T05:52:21.000 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/mac802154/mac802154.ko 2024-05-28T05:52:21.000 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/mpls 2024-05-28T05:52:21.001 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/mpls/mpls_gso.ko 2024-05-28T05:52:21.001 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter 2024-05-28T05:52:21.001 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset 2024-05-28T05:52:21.001 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set.ko 2024-05-28T05:52:21.001 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-05-28T05:52:21.001 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-05-28T05:52:21.001 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-05-28T05:52:21.001 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-05-28T05:52:21.001 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-05-28T05:52:21.001 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-05-28T05:52:21.001 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-05-28T05:52:21.001 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-05-28T05:52:21.001 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-05-28T05:52:21.001 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-05-28T05:52:21.001 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-05-28T05:52:21.002 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-05-28T05:52:21.002 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-05-28T05:52:21.002 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-05-28T05:52:21.002 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-05-28T05:52:21.002 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs 2024-05-28T05:52:21.002 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs.ko 2024-05-28T05:52:21.002 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-05-28T05:52:21.002 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-05-28T05:52:21.002 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-05-28T05:52:21.002 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-05-28T05:52:21.002 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-05-28T05:52:21.002 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-05-28T05:52:21.003 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-05-28T05:52:21.003 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-05-28T05:52:21.003 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-05-28T05:52:21.003 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-05-28T05:52:21.003 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-05-28T05:52:21.003 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-05-28T05:52:21.003 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-05-28T05:52:21.003 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conncount.ko 2024-05-28T05:52:21.003 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack.ko 2024-05-28T05:52:21.003 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-05-28T05:52:21.003 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-05-28T05:52:21.003 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-05-28T05:52:21.003 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_h323.ko 2024-05-28T05:52:21.003 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_irc.ko 2024-05-28T05:52:21.004 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-05-28T05:52:21.004 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-05-28T05:52:21.004 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-05-28T05:52:21.004 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_sane.ko 2024-05-28T05:52:21.004 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_sip.ko 2024-05-28T05:52:21.004 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-05-28T05:52:21.004 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-05-28T05:52:21.004 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_dup_netdev.ko 2024-05-28T05:52:21.004 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_flow_table.ko 2024-05-28T05:52:21.004 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_flow_table_inet.ko 2024-05-28T05:52:21.004 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_log_syslog.ko 2024-05-28T05:52:21.004 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_nat.ko 2024-05-28T05:52:21.004 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_nat_amanda.ko 2024-05-28T05:52:21.004 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_nat_ftp.ko 2024-05-28T05:52:21.004 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_nat_irc.ko 2024-05-28T05:52:21.005 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_nat_sip.ko 2024-05-28T05:52:21.005 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_nat_tftp.ko 2024-05-28T05:52:21.005 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_synproxy_core.ko 2024-05-28T05:52:21.005 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_tables.ko 2024-05-28T05:52:21.005 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nfnetlink.ko 2024-05-28T05:52:21.005 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nfnetlink_acct.ko 2024-05-28T05:52:21.005 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nfnetlink_log.ko 2024-05-28T05:52:21.005 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nfnetlink_osf.ko 2024-05-28T05:52:21.005 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nfnetlink_queue.ko 2024-05-28T05:52:21.005 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_chain_nat.ko 2024-05-28T05:52:21.005 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_compat.ko 2024-05-28T05:52:21.005 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_ct.ko 2024-05-28T05:52:21.005 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_dup_netdev.ko 2024-05-28T05:52:21.005 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_fib.ko 2024-05-28T05:52:21.006 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_fwd_netdev.ko 2024-05-28T05:52:21.006 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_hash.ko 2024-05-28T05:52:21.006 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_limit.ko 2024-05-28T05:52:21.006 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_log.ko 2024-05-28T05:52:21.006 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_masq.ko 2024-05-28T05:52:21.006 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_nat.ko 2024-05-28T05:52:21.006 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_numgen.ko 2024-05-28T05:52:21.006 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_queue.ko 2024-05-28T05:52:21.006 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_quota.ko 2024-05-28T05:52:21.006 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_redir.ko 2024-05-28T05:52:21.006 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_reject.ko 2024-05-28T05:52:21.006 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_reject_inet.ko 2024-05-28T05:52:21.006 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_socket.ko 2024-05-28T05:52:21.007 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_tunnel.ko 2024-05-28T05:52:21.007 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_AUDIT.ko 2024-05-28T05:52:21.007 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_CHECKSUM.ko 2024-05-28T05:52:21.007 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_CLASSIFY.ko 2024-05-28T05:52:21.007 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-05-28T05:52:21.007 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_CT.ko 2024-05-28T05:52:21.007 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_DSCP.ko 2024-05-28T05:52:21.007 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_HL.ko 2024-05-28T05:52:21.007 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_HMARK.ko 2024-05-28T05:52:21.007 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_IDLETIMER.ko 2024-05-28T05:52:21.007 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_LED.ko 2024-05-28T05:52:21.007 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_LOG.ko 2024-05-28T05:52:21.007 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_MASQUERADE.ko 2024-05-28T05:52:21.007 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_NETMAP.ko 2024-05-28T05:52:21.008 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_NFLOG.ko 2024-05-28T05:52:21.008 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_NFQUEUE.ko 2024-05-28T05:52:21.008 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_RATEEST.ko 2024-05-28T05:52:21.008 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_REDIRECT.ko 2024-05-28T05:52:21.008 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_SECMARK.ko 2024-05-28T05:52:21.008 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_TCPMSS.ko 2024-05-28T05:52:21.008 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-05-28T05:52:21.008 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_TEE.ko 2024-05-28T05:52:21.008 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_TPROXY.ko 2024-05-28T05:52:21.008 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_TRACE.ko 2024-05-28T05:52:21.008 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_addrtype.ko 2024-05-28T05:52:21.008 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_bpf.ko 2024-05-28T05:52:21.008 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_cgroup.ko 2024-05-28T05:52:21.008 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_cluster.ko 2024-05-28T05:52:21.009 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_comment.ko 2024-05-28T05:52:21.009 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_connbytes.ko 2024-05-28T05:52:21.009 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_connlabel.ko 2024-05-28T05:52:21.009 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_connlimit.ko 2024-05-28T05:52:21.009 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_connmark.ko 2024-05-28T05:52:21.009 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_conntrack.ko 2024-05-28T05:52:21.009 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_cpu.ko 2024-05-28T05:52:21.009 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_dccp.ko 2024-05-28T05:52:21.009 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_devgroup.ko 2024-05-28T05:52:21.009 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_dscp.ko 2024-05-28T05:52:21.009 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_ecn.ko 2024-05-28T05:52:21.009 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_esp.ko 2024-05-28T05:52:21.009 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_hashlimit.ko 2024-05-28T05:52:21.009 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_helper.ko 2024-05-28T05:52:21.009 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_hl.ko 2024-05-28T05:52:21.010 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_ipcomp.ko 2024-05-28T05:52:21.010 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_iprange.ko 2024-05-28T05:52:21.010 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_ipvs.ko 2024-05-28T05:52:21.010 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_l2tp.ko 2024-05-28T05:52:21.010 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_length.ko 2024-05-28T05:52:21.010 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_limit.ko 2024-05-28T05:52:21.010 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_mac.ko 2024-05-28T05:52:21.010 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_mark.ko 2024-05-28T05:52:21.010 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_multiport.ko 2024-05-28T05:52:21.010 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_nat.ko 2024-05-28T05:52:21.010 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_nfacct.ko 2024-05-28T05:52:21.010 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_osf.ko 2024-05-28T05:52:21.010 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_owner.ko 2024-05-28T05:52:21.011 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_physdev.ko 2024-05-28T05:52:21.011 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_pkttype.ko 2024-05-28T05:52:21.011 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_policy.ko 2024-05-28T05:52:21.011 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_quota.ko 2024-05-28T05:52:21.011 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_rateest.ko 2024-05-28T05:52:21.011 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_realm.ko 2024-05-28T05:52:21.011 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_recent.ko 2024-05-28T05:52:21.011 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_sctp.ko 2024-05-28T05:52:21.011 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_set.ko 2024-05-28T05:52:21.011 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_socket.ko 2024-05-28T05:52:21.011 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_state.ko 2024-05-28T05:52:21.011 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_statistic.ko 2024-05-28T05:52:21.011 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_string.ko 2024-05-28T05:52:21.011 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_tcpmss.ko 2024-05-28T05:52:21.011 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_time.ko 2024-05-28T05:52:21.012 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_u32.ko 2024-05-28T05:52:21.012 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netlink 2024-05-28T05:52:21.012 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netlink/netlink_diag.ko 2024-05-28T05:52:21.012 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/nsh 2024-05-28T05:52:21.012 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/nsh/nsh.ko 2024-05-28T05:52:21.012 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/openvswitch 2024-05-28T05:52:21.012 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/openvswitch/openvswitch.ko 2024-05-28T05:52:21.012 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/openvswitch/vport-geneve.ko 2024-05-28T05:52:21.012 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/openvswitch/vport-gre.ko 2024-05-28T05:52:21.012 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/openvswitch/vport-vxlan.ko 2024-05-28T05:52:21.012 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/packet 2024-05-28T05:52:21.012 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/packet/af_packet_diag.ko 2024-05-28T05:52:21.012 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/rds 2024-05-28T05:52:21.012 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/rds/rds.ko 2024-05-28T05:52:21.012 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/rds/rds_rdma.ko 2024-05-28T05:52:21.012 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/rds/rds_tcp.ko 2024-05-28T05:52:21.013 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched 2024-05-28T05:52:21.013 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_csum.ko 2024-05-28T05:52:21.013 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_gact.ko 2024-05-28T05:52:21.013 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_mirred.ko 2024-05-28T05:52:21.013 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_nat.ko 2024-05-28T05:52:21.013 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_pedit.ko 2024-05-28T05:52:21.013 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_police.ko 2024-05-28T05:52:21.013 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_simple.ko 2024-05-28T05:52:21.013 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_skbedit.ko 2024-05-28T05:52:21.013 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_vlan.ko 2024-05-28T05:52:21.013 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/cls_basic.ko 2024-05-28T05:52:21.013 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/cls_bpf.ko 2024-05-28T05:52:21.013 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/cls_flow.ko 2024-05-28T05:52:21.013 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/cls_fw.ko 2024-05-28T05:52:21.014 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/cls_route.ko 2024-05-28T05:52:21.014 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/cls_u32.ko 2024-05-28T05:52:21.014 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/em_cmp.ko 2024-05-28T05:52:21.014 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/em_ipset.ko 2024-05-28T05:52:21.014 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/em_meta.ko 2024-05-28T05:52:21.014 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/em_nbyte.ko 2024-05-28T05:52:21.014 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/em_text.ko 2024-05-28T05:52:21.014 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/em_u32.ko 2024-05-28T05:52:21.014 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_choke.ko 2024-05-28T05:52:21.014 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_codel.ko 2024-05-28T05:52:21.014 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_drr.ko 2024-05-28T05:52:21.014 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_fq.ko 2024-05-28T05:52:21.014 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_gred.ko 2024-05-28T05:52:21.014 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_hfsc.ko 2024-05-28T05:52:21.015 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_hhf.ko 2024-05-28T05:52:21.015 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_htb.ko 2024-05-28T05:52:21.015 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_ingress.ko 2024-05-28T05:52:21.015 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_mqprio.ko 2024-05-28T05:52:21.015 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_mqprio_lib.ko 2024-05-28T05:52:21.015 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_multiq.ko 2024-05-28T05:52:21.015 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_netem.ko 2024-05-28T05:52:21.015 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_pie.ko 2024-05-28T05:52:21.015 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_plug.ko 2024-05-28T05:52:21.015 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_prio.ko 2024-05-28T05:52:21.015 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_qfq.ko 2024-05-28T05:52:21.015 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_red.ko 2024-05-28T05:52:21.015 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_sfb.ko 2024-05-28T05:52:21.015 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_sfq.ko 2024-05-28T05:52:21.016 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_tbf.ko 2024-05-28T05:52:21.016 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_teql.ko 2024-05-28T05:52:21.016 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sctp 2024-05-28T05:52:21.016 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sctp/sctp.ko 2024-05-28T05:52:21.016 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sctp/sctp_diag.ko 2024-05-28T05:52:21.016 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sunrpc 2024-05-28T05:52:21.016 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sunrpc/auth_gss 2024-05-28T05:52:21.016 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-05-28T05:52:21.016 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-05-28T05:52:21.016 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sunrpc/sunrpc.ko 2024-05-28T05:52:21.016 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sunrpc/xprtrdma 2024-05-28T05:52:21.016 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-05-28T05:52:21.016 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/tipc 2024-05-28T05:52:21.016 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/tipc/tipc.ko 2024-05-28T05:52:21.017 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/tipc/tipc_diag.ko 2024-05-28T05:52:21.017 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/unix 2024-05-28T05:52:21.017 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/unix/unix_diag.ko 2024-05-28T05:52:21.017 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/vmw_vsock 2024-05-28T05:52:21.017 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-05-28T05:52:21.017 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-05-28T05:52:21.017 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/vmw_vsock/vsock.ko 2024-05-28T05:52:21.017 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/vmw_vsock/vsock_diag.ko 2024-05-28T05:52:21.017 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/vmw_vsock/vsock_loopback.ko 2024-05-28T05:52:21.017 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/xfrm 2024-05-28T05:52:21.017 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/xfrm/xfrm_ipcomp.ko 2024-05-28T05:52:21.017 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/security 2024-05-28T05:52:21.017 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/security/keys 2024-05-28T05:52:21.017 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/security/keys/encrypted-keys 2024-05-28T05:52:21.018 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-05-28T05:52:21.018 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/security/keys/trusted-keys 2024-05-28T05:52:21.018 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/security/keys/trusted-keys/trusted.ko 2024-05-28T05:52:21.018 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.alias 2024-05-28T05:52:21.018 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.alias.bin 2024-05-28T05:52:21.018 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.builtin 2024-05-28T05:52:21.018 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.builtin.alias.bin 2024-05-28T05:52:21.018 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.builtin.bin 2024-05-28T05:52:21.018 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.builtin.modinfo 2024-05-28T05:52:21.018 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.dep 2024-05-28T05:52:21.018 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.dep.bin 2024-05-28T05:52:21.018 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.devname 2024-05-28T05:52:21.018 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.order 2024-05-28T05:52:21.018 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.softdep 2024-05-28T05:52:21.019 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.symbols 2024-05-28T05:52:21.019 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.symbols.bin 2024-05-28T05:52:21.019 INFO:teuthology.orchestra.run.smithi123.stdout:/lib/modules/6.9.0-gb9459a26b088/vmlinuz 2024-05-28T05:52:21.022 DEBUG:teuthology.task.kernel:get_image_version: 6.9.0-gb9459a26b088 2024-05-28T05:52:21.022 DEBUG:teuthology.orchestra.run.smithi123:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-05-28T05:52:21.045 INFO:teuthology.orchestra.run.smithi123.stdout:postinstall scriptlet (using /bin/sh): 2024-05-28T05:52:21.045 INFO:teuthology.orchestra.run.smithi123.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-05-28T05:52:21.046 INFO:teuthology.orchestra.run.smithi123.stdout: /usr/bin/kernel-install add 6.9.0-gb9459a26b088 /lib/modules/6.9.0-gb9459a26b088/vmlinuz 2024-05-28T05:52:21.046 INFO:teuthology.orchestra.run.smithi123.stdout:fi 2024-05-28T05:52:21.046 INFO:teuthology.orchestra.run.smithi123.stdout:for file in vmlinuz System.map config; do 2024-05-28T05:52:21.046 INFO:teuthology.orchestra.run.smithi123.stdout: if ! cmp --silent "/lib/modules/6.9.0-gb9459a26b088/${file}" "/boot/${file}-6.9.0-gb9459a26b088"; then 2024-05-28T05:52:21.046 INFO:teuthology.orchestra.run.smithi123.stdout: cp "/lib/modules/6.9.0-gb9459a26b088/${file}" "/boot/${file}-6.9.0-gb9459a26b088" 2024-05-28T05:52:21.046 INFO:teuthology.orchestra.run.smithi123.stdout: fi 2024-05-28T05:52:21.046 INFO:teuthology.orchestra.run.smithi123.stdout:done 2024-05-28T05:52:21.046 INFO:teuthology.orchestra.run.smithi123.stdout:if [ -d "/lib/modules/6.9.0-gb9459a26b088/dtb" ] && \ 2024-05-28T05:52:21.046 INFO:teuthology.orchestra.run.smithi123.stdout: ! diff -rq "/lib/modules/6.9.0-gb9459a26b088/dtb" "/boot/dtb-6.9.0-gb9459a26b088" >/dev/null 2>&1; then 2024-05-28T05:52:21.046 INFO:teuthology.orchestra.run.smithi123.stdout: rm -rf "/boot/dtb-6.9.0-gb9459a26b088" 2024-05-28T05:52:21.046 INFO:teuthology.orchestra.run.smithi123.stdout: cp -r "/lib/modules/6.9.0-gb9459a26b088/dtb" "/boot/dtb-6.9.0-gb9459a26b088" 2024-05-28T05:52:21.046 INFO:teuthology.orchestra.run.smithi123.stdout:fi 2024-05-28T05:52:21.046 INFO:teuthology.orchestra.run.smithi123.stdout:if [ ! -e "/lib/modules/6.9.0-gb9459a26b088/modules.dep" ]; then 2024-05-28T05:52:21.047 INFO:teuthology.orchestra.run.smithi123.stdout: /usr/sbin/depmod 6.9.0-gb9459a26b088 2024-05-28T05:52:21.047 INFO:teuthology.orchestra.run.smithi123.stdout:fi 2024-05-28T05:52:21.047 INFO:teuthology.orchestra.run.smithi123.stdout:preuninstall scriptlet (using /bin/sh): 2024-05-28T05:52:21.047 INFO:teuthology.orchestra.run.smithi123.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-05-28T05:52:21.047 INFO:teuthology.orchestra.run.smithi123.stdout:kernel-install remove 6.9.0-gb9459a26b088 2024-05-28T05:52:21.047 INFO:teuthology.orchestra.run.smithi123.stdout:fi 2024-05-28T05:52:21.048 DEBUG:teuthology.orchestra.run.smithi123:> sudo rpm -qi grub2-tools 2024-05-28T05:52:21.120 INFO:teuthology.orchestra.run.smithi123.stdout:Name : grub2-tools 2024-05-28T05:52:21.120 INFO:teuthology.orchestra.run.smithi123.stdout:Epoch : 1 2024-05-28T05:52:21.120 INFO:teuthology.orchestra.run.smithi123.stdout:Version : 2.06 2024-05-28T05:52:21.120 INFO:teuthology.orchestra.run.smithi123.stdout:Release : 68.el9 2024-05-28T05:52:21.121 INFO:teuthology.orchestra.run.smithi123.stdout:Architecture: x86_64 2024-05-28T05:52:21.121 INFO:teuthology.orchestra.run.smithi123.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-05-28T05:52:21.121 INFO:teuthology.orchestra.run.smithi123.stdout:Group : Unspecified 2024-05-28T05:52:21.121 INFO:teuthology.orchestra.run.smithi123.stdout:Size : 8264254 2024-05-28T05:52:21.121 INFO:teuthology.orchestra.run.smithi123.stdout:License : GPLv3+ 2024-05-28T05:52:21.121 INFO:teuthology.orchestra.run.smithi123.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-05-28T05:52:21.121 INFO:teuthology.orchestra.run.smithi123.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-05-28T05:52:21.121 INFO:teuthology.orchestra.run.smithi123.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-05-28T05:52:21.121 INFO:teuthology.orchestra.run.smithi123.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-05-28T05:52:21.121 INFO:teuthology.orchestra.run.smithi123.stdout:Packager : builder@centos.org 2024-05-28T05:52:21.121 INFO:teuthology.orchestra.run.smithi123.stdout:Vendor : CentOS 2024-05-28T05:52:21.121 INFO:teuthology.orchestra.run.smithi123.stdout:URL : http://www.gnu.org/software/grub/ 2024-05-28T05:52:21.121 INFO:teuthology.orchestra.run.smithi123.stdout:Summary : Support tools for GRUB. 2024-05-28T05:52:21.121 INFO:teuthology.orchestra.run.smithi123.stdout:Description : 2024-05-28T05:52:21.122 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-05-28T05:52:21.122 INFO:teuthology.orchestra.run.smithi123.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-05-28T05:52:21.122 INFO:teuthology.orchestra.run.smithi123.stdout:customizable bootloader with modular architecture. It supports a rich 2024-05-28T05:52:21.122 INFO:teuthology.orchestra.run.smithi123.stdout:variety of kernel formats, file systems, computer architectures and 2024-05-28T05:52:21.122 INFO:teuthology.orchestra.run.smithi123.stdout:hardware devices. 2024-05-28T05:52:21.122 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-05-28T05:52:21.122 INFO:teuthology.orchestra.run.smithi123.stdout:This subpackage provides tools for support of all platforms. 2024-05-28T05:52:21.123 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-05-28T05:52:21.123 INFO:teuthology.task.kernel:Updating grub on smithi123 to boot 6.9.0-gb9459a26b088 2024-05-28T05:52:21.123 DEBUG:teuthology.orchestra.run.smithi123:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-05-28T05:52:21.640 INFO:teuthology.orchestra.run.smithi123.stderr:Generating grub configuration file ... 2024-05-28T05:52:23.019 INFO:teuthology.orchestra.run.smithi123.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-05-28T05:52:23.057 INFO:teuthology.orchestra.run.smithi123.stderr:done 2024-05-28T05:52:23.060 DEBUG:teuthology.orchestra.run.smithi123:> mktemp 2024-05-28T05:52:23.074 INFO:teuthology.orchestra.run.smithi123.stdout:/tmp/tmp.oH2iO23bda 2024-05-28T05:52:23.074 DEBUG:teuthology.orchestra.run.smithi123:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.oH2iO23bda 2024-05-28T05:52:23.142 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 0666 /tmp/tmp.oH2iO23bda 2024-05-28T05:52:23.282 DEBUG:teuthology.orchestra.remote:smithi123:/tmp/tmp.oH2iO23bda is 6KB 2024-05-28T05:52:23.330 DEBUG:teuthology.orchestra.run.smithi123:> rm -fr /tmp/tmp.oH2iO23bda 2024-05-28T05:52:23.344 DEBUG:teuthology.orchestra.run.smithi123:> sudo /bin/ls /boot/loader/entries || true 2024-05-28T05:52:23.405 INFO:teuthology.orchestra.run.smithi123.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-05-28T05:52:23.405 INFO:teuthology.orchestra.run.smithi123.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-05-28T05:52:23.405 INFO:teuthology.orchestra.run.smithi123.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-05-28T05:52:23.405 INFO:teuthology.orchestra.run.smithi123.stdout:7fc8828e06d0482694bcf4a447d13cb6-6.9.0-gb9459a26b088.conf 2024-05-28T05:52:23.407 DEBUG:teuthology.orchestra.run.smithi123:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-6.9.0-gb9459a26b088 2024-05-28T05:52:23.571 DEBUG:teuthology.orchestra.run.smithi123:> sudo shutdown -r now 2024-05-28T05:52:23.653 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2024-05-28T05:52:46.422 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-05-28T05:52:46.423 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi096.front.sepia.ceph.com' 2024-05-28T05:52:46.424 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi096.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-28T05:52:53.574 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-05-28T05:52:53.574 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi123.front.sepia.ceph.com' 2024-05-28T05:52:53.575 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-28T05:53:46.479 DEBUG:teuthology.orchestra.remote:timed out 2024-05-28T05:53:53.582 DEBUG:teuthology.orchestra.remote:timed out 2024-05-28T05:53:55.480 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi096.front.sepia.ceph.com' 2024-05-28T05:53:55.482 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi096.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-28T05:53:58.879 DEBUG:teuthology.orchestra.run.smithi096:> true 2024-05-28T05:53:59.513 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi096.front.sepia.ceph.com' 2024-05-28T05:53:59.513 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "6.9.0-gb9459a26b088"... 2024-05-28T05:53:59.514 DEBUG:teuthology.orchestra.run.smithi096:> uname -r 2024-05-28T05:53:59.554 INFO:teuthology.orchestra.run.smithi096.stdout:6.9.0-gb9459a26b088 2024-05-28T05:53:59.554 DEBUG:teuthology.task.kernel:current kernel version is 6.9.0-gb9459a26b088 vs 6.9.0-gb9459a26b088 2024-05-28T05:53:59.554 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-05-28T05:53:59.554 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-05-28T05:54:00.555 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-05-28T05:54:00.556 DEBUG:teuthology.orchestra.run.smithi096:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-05-28T05:54:00.680 INFO:teuthology.orchestra.run.smithi096.stdout:ttyS1 2024-05-28T05:54:00.723 DEBUG:teuthology.parallel:result is None 2024-05-28T05:54:02.583 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi123.front.sepia.ceph.com' 2024-05-28T05:54:02.584 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-28T05:54:04.173 DEBUG:teuthology.orchestra.run.smithi123:> true 2024-05-28T05:54:04.836 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi123.front.sepia.ceph.com' 2024-05-28T05:54:04.836 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "6.9.0-gb9459a26b088"... 2024-05-28T05:54:04.836 DEBUG:teuthology.orchestra.run.smithi123:> uname -r 2024-05-28T05:54:04.877 INFO:teuthology.orchestra.run.smithi123.stdout:6.9.0-gb9459a26b088 2024-05-28T05:54:04.877 DEBUG:teuthology.task.kernel:current kernel version is 6.9.0-gb9459a26b088 vs 6.9.0-gb9459a26b088 2024-05-28T05:54:04.877 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-05-28T05:54:04.877 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-05-28T05:54:05.878 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-05-28T05:54:05.878 DEBUG:teuthology.orchestra.run.smithi123:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-05-28T05:54:06.055 INFO:teuthology.orchestra.run.smithi123.stdout:ttyS1 2024-05-28T05:54:06.095 DEBUG:teuthology.parallel:result is None 2024-05-28T05:54:06.095 INFO:teuthology.run_tasks:Running task internal.base... 2024-05-28T05:54:06.102 INFO:teuthology.task.internal:Creating test directory... 2024-05-28T05:54:06.102 DEBUG:teuthology.orchestra.run.smithi096:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-28T05:54:06.107 DEBUG:teuthology.orchestra.run.smithi123:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-28T05:54:06.151 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-05-28T05:54:06.172 INFO:teuthology.run_tasks:Running task internal.archive... 2024-05-28T05:54:06.374 INFO:teuthology.task.internal:Creating archive directory... 2024-05-28T05:54:06.375 DEBUG:teuthology.orchestra.run.smithi096:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-28T05:54:06.379 DEBUG:teuthology.orchestra.run.smithi123:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-28T05:54:06.432 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-05-28T05:54:06.438 INFO:teuthology.task.internal:Enabling coredump saving... 2024-05-28T05:54:06.438 DEBUG:teuthology.orchestra.run.smithi096:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-28T05:54:06.457 DEBUG:teuthology.orchestra.run.smithi123:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-28T05:54:06.529 INFO:teuthology.orchestra.run.smithi096.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-28T05:54:06.552 INFO:teuthology.orchestra.run.smithi096.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-28T05:54:06.583 INFO:teuthology.orchestra.run.smithi123.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-28T05:54:06.614 INFO:teuthology.orchestra.run.smithi123.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-28T05:54:06.619 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-05-28T05:54:06.648 INFO:teuthology.task.internal:Configuring sudo... 2024-05-28T05:54:06.648 DEBUG:teuthology.orchestra.run.smithi096:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-28T05:54:06.653 DEBUG:teuthology.orchestra.run.smithi123:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-28T05:54:06.720 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-05-28T05:54:06.737 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-05-28T05:54:06.737 DEBUG:teuthology.orchestra.run.smithi096:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-28T05:54:06.741 DEBUG:teuthology.orchestra.run.smithi123:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-28T05:54:06.783 DEBUG:teuthology.orchestra.run.smithi096:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-28T05:54:06.888 DEBUG:teuthology.orchestra.run.smithi096:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-28T05:54:06.973 DEBUG:teuthology.orchestra.run.smithi096:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-28T05:54:07.045 DEBUG:teuthology.orchestra.run.smithi096:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-28T05:54:07.122 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T05:54:07.122 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-28T05:54:07.184 DEBUG:teuthology.orchestra.run.smithi123:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-28T05:54:07.263 DEBUG:teuthology.orchestra.run.smithi123:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-28T05:54:07.348 DEBUG:teuthology.orchestra.run.smithi123:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-28T05:54:07.416 DEBUG:teuthology.orchestra.run.smithi123:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-28T05:54:07.499 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-05-28T05:54:07.499 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-28T05:54:07.567 DEBUG:teuthology.orchestra.run.smithi096:> sudo service rsyslog restart 2024-05-28T05:54:07.574 DEBUG:teuthology.orchestra.run.smithi123:> sudo service rsyslog restart 2024-05-28T05:54:07.667 INFO:teuthology.orchestra.run.smithi096.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-28T05:54:07.705 INFO:teuthology.orchestra.run.smithi123.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-28T05:54:08.189 INFO:teuthology.run_tasks:Running task internal.timer... 2024-05-28T05:54:08.197 INFO:teuthology.task.internal:Starting timer... 2024-05-28T05:54:08.197 INFO:teuthology.run_tasks:Running task pcp... 2024-05-28T05:54:08.210 INFO:teuthology.run_tasks:Running task selinux... 2024-05-28T05:54:08.220 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-05-28T05:54:08.221 DEBUG:teuthology.orchestra.run.smithi096:> sudo service auditd rotate 2024-05-28T05:54:08.372 INFO:teuthology.orchestra.run.smithi096.stdout:Rotating logs: 2024-05-28T05:54:08.376 DEBUG:teuthology.orchestra.run.smithi123:> sudo service auditd rotate 2024-05-28T05:54:08.480 INFO:teuthology.orchestra.run.smithi123.stdout:Rotating logs: 2024-05-28T05:54:08.484 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-05-28T05:54:08.484 DEBUG:teuthology.orchestra.run.smithi096:> /usr/sbin/getenforce 2024-05-28T05:54:08.546 INFO:teuthology.orchestra.run.smithi096.stdout:Permissive 2024-05-28T05:54:08.546 DEBUG:teuthology.orchestra.run.smithi123:> /usr/sbin/getenforce 2024-05-28T05:54:08.611 INFO:teuthology.orchestra.run.smithi123.stdout:Permissive 2024-05-28T05:54:08.612 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi096.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi123.front.sepia.ceph.com': 'permissive'} 2024-05-28T05:54:08.612 DEBUG:teuthology.orchestra.run.smithi096:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-28T05:54:08.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-28T05:54:08.672 DEBUG:teuthology.orchestra.run.smithi123:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-28T05:54:08.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-28T05:54:08.742 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-05-28T05:54:08.742 DEBUG:teuthology.orchestra.run.smithi096:> sudo /usr/sbin/setenforce permissive 2024-05-28T05:54:08.799 DEBUG:teuthology.orchestra.run.smithi123:> sudo /usr/sbin/setenforce permissive 2024-05-28T05:54:08.874 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-05-28T05:54:08.884 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-05-28T05:54:08.887 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-05-28T05:54:09.003 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-05-28T05:54:09.014 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-05-28T05:54:09.014 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi096.front.sepia.ceph.com,smithi123.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-05-28T06:00:30.798 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi096.front.sepia.ceph.com'), Remote(name='ubuntu@smithi123.front.sepia.ceph.com')] 2024-05-28T06:00:30.800 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi096.front.sepia.ceph.com' 2024-05-28T06:00:30.801 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi096.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-28T06:00:30.910 DEBUG:teuthology.orchestra.run.smithi096:> true 2024-05-28T06:00:31.059 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi096.front.sepia.ceph.com' 2024-05-28T06:00:31.059 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi123.front.sepia.ceph.com' 2024-05-28T06:00:31.060 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-28T06:00:31.164 DEBUG:teuthology.orchestra.run.smithi123:> true 2024-05-28T06:00:31.317 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi123.front.sepia.ceph.com' 2024-05-28T06:00:31.318 INFO:teuthology.run_tasks:Running task clock... 2024-05-28T06:00:31.327 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-05-28T06:00:31.328 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-28T06:00:31.328 DEBUG:teuthology.orchestra.run.smithi096:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-28T06:00:31.336 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-28T06:00:31.336 DEBUG:teuthology.orchestra.run.smithi123:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-28T06:00:31.425 INFO:teuthology.orchestra.run.smithi123.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-28T06:00:31.427 INFO:teuthology.orchestra.run.smithi096.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-28T06:00:31.468 INFO:teuthology.orchestra.run.smithi123.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-28T06:00:31.469 INFO:teuthology.orchestra.run.smithi096.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-28T06:00:31.541 INFO:teuthology.orchestra.run.smithi123.stderr:sudo: ntpd: command not found 2024-05-28T06:00:31.542 INFO:teuthology.orchestra.run.smithi096.stderr:sudo: ntpd: command not found 2024-05-28T06:00:31.573 INFO:teuthology.orchestra.run.smithi123.stdout:506 Cannot talk to daemon 2024-05-28T06:00:31.575 INFO:teuthology.orchestra.run.smithi096.stdout:506 Cannot talk to daemon 2024-05-28T06:00:31.613 INFO:teuthology.orchestra.run.smithi123.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-28T06:00:31.619 INFO:teuthology.orchestra.run.smithi096.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-28T06:00:31.655 INFO:teuthology.orchestra.run.smithi123.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-28T06:00:31.662 INFO:teuthology.orchestra.run.smithi096.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-28T06:00:31.791 INFO:teuthology.orchestra.run.smithi123.stderr:bash: line 1: ntpq: command not found 2024-05-28T06:00:31.800 INFO:teuthology.orchestra.run.smithi096.stderr:bash: line 1: ntpq: command not found 2024-05-28T06:00:31.802 INFO:teuthology.orchestra.run.smithi123.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-28T06:00:31.802 INFO:teuthology.orchestra.run.smithi123.stdout:=============================================================================== 2024-05-28T06:00:31.802 INFO:teuthology.orchestra.run.smithi123.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-28T06:00:31.802 INFO:teuthology.orchestra.run.smithi123.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-28T06:00:31.802 INFO:teuthology.orchestra.run.smithi123.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-28T06:00:31.802 INFO:teuthology.orchestra.run.smithi123.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-28T06:00:31.809 INFO:teuthology.orchestra.run.smithi096.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-28T06:00:31.809 INFO:teuthology.orchestra.run.smithi096.stdout:=============================================================================== 2024-05-28T06:00:31.809 INFO:teuthology.orchestra.run.smithi096.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-28T06:00:31.809 INFO:teuthology.orchestra.run.smithi096.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-28T06:00:31.809 INFO:teuthology.orchestra.run.smithi096.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-28T06:00:31.810 INFO:teuthology.orchestra.run.smithi096.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-28T06:00:31.811 INFO:teuthology.run_tasks:Running task install... 2024-05-28T06:00:31.819 DEBUG:teuthology.task.install:project ceph 2024-05-28T06:00:31.819 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '9ed025fbf4406fb54980a0647506b15895b6d863'}} 2024-05-28T06:00:31.819 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '9ed025fbf4406fb54980a0647506b15895b6d863'} 2024-05-28T06:00:31.819 INFO:teuthology.task.install:Using flavor: default 2024-05-28T06:00:31.825 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-05-28T06:00:31.825 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-05-28T06:00:31.825 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '9ed025fbf4406fb54980a0647506b15895b6d863', 'tag': None, 'wait_for_package': False} 2024-05-28T06:00:31.826 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=9ed025fbf4406fb54980a0647506b15895b6d863 2024-05-28T06:00:31.827 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '9ed025fbf4406fb54980a0647506b15895b6d863', 'tag': None, 'wait_for_package': False} 2024-05-28T06:00:31.828 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=9ed025fbf4406fb54980a0647506b15895b6d863 2024-05-28T06:00:31.976 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/rishabh-clone-stats/9ed025fbf4406fb54980a0647506b15895b6d863/centos/9/flavors/default/ 2024-05-28T06:00:31.976 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3154.g9ed025fb 2024-05-28T06:00:31.979 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/rishabh-clone-stats/9ed025fbf4406fb54980a0647506b15895b6d863/centos/9/flavors/default/ 2024-05-28T06:00:31.979 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3154.g9ed025fb 2024-05-28T06:00:32.090 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/rishabh-clone-stats/9ed025fbf4406fb54980a0647506b15895b6d863/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/rishabh-clone-stats/9ed025fbf4406fb54980a0647506b15895b6d863/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/rishabh-clone-stats/9ed025fbf4406fb54980a0647506b15895b6d863/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-28T06:00:32.091 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-05-28T06:00:32.091 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-28T06:00:32.151 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/rishabh-clone-stats/9ed025fbf4406fb54980a0647506b15895b6d863/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/rishabh-clone-stats/9ed025fbf4406fb54980a0647506b15895b6d863/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/rishabh-clone-stats/9ed025fbf4406fb54980a0647506b15895b6d863/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-28T06:00:32.151 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:00:32.151 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-28T06:00:32.181 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-05-28T06:00:32.181 DEBUG:teuthology.orchestra.run.smithi123:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/9ed025fbf4406fb54980a0647506b15895b6d863/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-28T06:00:32.232 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-05-28T06:00:32.232 DEBUG:teuthology.orchestra.run.smithi096:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/9ed025fbf4406fb54980a0647506b15895b6d863/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-28T06:00:32.307 DEBUG:teuthology.orchestra.run.smithi123:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-28T06:00:32.361 DEBUG:teuthology.orchestra.run.smithi096:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-28T06:00:32.419 DEBUG:teuthology.orchestra.run.smithi123:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-28T06:00:32.467 DEBUG:teuthology.orchestra.run.smithi096:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-28T06:00:32.497 INFO:teuthology.orchestra.run.smithi123.stdout:check_obsoletes = 1 2024-05-28T06:00:32.501 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum clean all 2024-05-28T06:00:32.540 INFO:teuthology.orchestra.run.smithi096.stdout:check_obsoletes = 1 2024-05-28T06:00:32.543 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum clean all 2024-05-28T06:00:32.813 INFO:teuthology.orchestra.run.smithi123.stdout:45 files removed 2024-05-28T06:00:32.852 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-05-28T06:00:32.853 INFO:teuthology.orchestra.run.smithi096.stdout:45 files removed 2024-05-28T06:00:32.891 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-05-28T06:00:33.806 INFO:teuthology.orchestra.run.smithi123.stdout:ceph packages for x86_64 160 kB/s | 83 kB 00:00 2024-05-28T06:00:33.886 INFO:teuthology.orchestra.run.smithi096.stdout:ceph packages for x86_64 149 kB/s | 83 kB 00:00 2024-05-28T06:00:34.298 INFO:teuthology.orchestra.run.smithi123.stdout:ceph noarch packages 30 kB/s | 12 kB 00:00 2024-05-28T06:00:34.403 INFO:teuthology.orchestra.run.smithi096.stdout:ceph noarch packages 28 kB/s | 12 kB 00:00 2024-05-28T06:00:34.716 INFO:teuthology.orchestra.run.smithi123.stdout:ceph source packages 5.1 kB/s | 1.9 kB 00:00 2024-05-28T06:00:34.870 INFO:teuthology.orchestra.run.smithi096.stdout:ceph source packages 5.0 kB/s | 1.9 kB 00:00 2024-05-28T06:00:35.397 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.1 MB 00:00 2024-05-28T06:00:37.118 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - BaseOS 3.7 MB/s | 8.1 MB 00:02 2024-05-28T06:00:39.248 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 9 - AppStream 7.3 MB/s | 19 MB 00:02 2024-05-28T06:00:39.494 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - AppStream 16 MB/s | 19 MB 00:01 2024-05-28T06:00:44.653 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - CRB 4.9 MB/s | 6.2 MB 00:01 2024-05-28T06:00:45.307 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 9 - CRB 2.9 MB/s | 6.2 MB 00:02 2024-05-28T06:00:46.495 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 17 kB 00:00 2024-05-28T06:00:46.916 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 9 - Extras packages 73 kB/s | 17 kB 00:00 2024-05-28T06:00:48.627 INFO:teuthology.orchestra.run.smithi096.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-05-28T06:00:54.879 INFO:teuthology.orchestra.run.smithi096.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-05-28T06:00:57.364 INFO:teuthology.orchestra.run.smithi096.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-28T06:00:57.365 INFO:teuthology.orchestra.run.smithi096.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-28T06:00:57.386 INFO:teuthology.orchestra.run.smithi096.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-05-28T06:00:57.388 INFO:teuthology.orchestra.run.smithi096.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-05-28T06:00:57.389 INFO:teuthology.orchestra.run.smithi096.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-05-28T06:00:57.390 INFO:teuthology.orchestra.run.smithi096.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-05-28T06:00:57.393 INFO:teuthology.orchestra.run.smithi096.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-05-28T06:00:57.396 INFO:teuthology.orchestra.run.smithi096.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-05-28T06:00:57.397 INFO:teuthology.orchestra.run.smithi096.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-05-28T06:00:57.472 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-05-28T06:00:57.494 INFO:teuthology.orchestra.run.smithi096.stdout:======================================================================================= 2024-05-28T06:00:57.494 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repository Size 2024-05-28T06:00:57.494 INFO:teuthology.orchestra.run.smithi096.stdout:======================================================================================= 2024-05-28T06:00:57.494 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-05-28T06:00:57.494 INFO:teuthology.orchestra.run.smithi096.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-05-28T06:00:57.494 INFO:teuthology.orchestra.run.smithi096.stdout: ceph x86_64 2:19.0.0-3154.g9ed025fb.el9 ceph 6.6 k 2024-05-28T06:00:57.494 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-base x86_64 2:19.0.0-3154.g9ed025fb.el9 ceph 5.4 M 2024-05-28T06:00:57.494 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-fuse x86_64 2:19.0.0-3154.g9ed025fb.el9 ceph 890 k 2024-05-28T06:00:57.494 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3154.g9ed025fb.el9 ceph 145 k 2024-05-28T06:00:57.494 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr x86_64 2:19.0.0-3154.g9ed025fb.el9 ceph 1.5 M 2024-05-28T06:00:57.495 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3154.g9ed025fb.el9 ceph-noarch 142 k 2024-05-28T06:00:57.495 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3154.g9ed025fb.el9 ceph-noarch 3.5 M 2024-05-28T06:00:57.495 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3154.g9ed025fb.el9 ceph-noarch 7.4 M 2024-05-28T06:00:57.497 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-rook noarch 2:19.0.0-3154.g9ed025fb.el9 ceph-noarch 49 k 2024-05-28T06:00:57.497 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-radosgw x86_64 2:19.0.0-3154.g9ed025fb.el9 ceph 11 M 2024-05-28T06:00:57.497 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-test x86_64 2:19.0.0-3154.g9ed025fb.el9 ceph 49 M 2024-05-28T06:00:57.497 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-volume noarch 2:19.0.0-3154.g9ed025fb.el9 ceph-noarch 273 k 2024-05-28T06:00:57.497 INFO:teuthology.orchestra.run.smithi096.stdout: cephadm noarch 2:19.0.0-3154.g9ed025fb.el9 ceph-noarch 762 k 2024-05-28T06:00:57.498 INFO:teuthology.orchestra.run.smithi096.stdout: cephfs-mirror x86_64 2:19.0.0-3154.g9ed025fb.el9 ceph 225 k 2024-05-28T06:00:57.498 INFO:teuthology.orchestra.run.smithi096.stdout: cephfs-shell x86_64 2:19.0.0-3154.g9ed025fb.el9 ceph 27 k 2024-05-28T06:00:57.498 INFO:teuthology.orchestra.run.smithi096.stdout: cephfs-top noarch 2:19.0.0-3154.g9ed025fb.el9 ceph-noarch 18 k 2024-05-28T06:00:57.498 INFO:teuthology.orchestra.run.smithi096.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-05-28T06:00:57.498 INFO:teuthology.orchestra.run.smithi096.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-05-28T06:00:57.498 INFO:teuthology.orchestra.run.smithi096.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-05-28T06:00:57.498 INFO:teuthology.orchestra.run.smithi096.stdout: libcephfs-devel x86_64 2:19.0.0-3154.g9ed025fb.el9 ceph 32 k 2024-05-28T06:00:57.498 INFO:teuthology.orchestra.run.smithi096.stdout: libcephfs2 x86_64 2:19.0.0-3154.g9ed025fb.el9 ceph 746 k 2024-05-28T06:00:57.498 INFO:teuthology.orchestra.run.smithi096.stdout: librados-devel x86_64 2:19.0.0-3154.g9ed025fb.el9 ceph 127 k 2024-05-28T06:00:57.498 INFO:teuthology.orchestra.run.smithi096.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-05-28T06:00:57.498 INFO:teuthology.orchestra.run.smithi096.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-05-28T06:00:57.498 INFO:teuthology.orchestra.run.smithi096.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-05-28T06:00:57.499 INFO:teuthology.orchestra.run.smithi096.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-05-28T06:00:57.499 INFO:teuthology.orchestra.run.smithi096.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-05-28T06:00:57.499 INFO:teuthology.orchestra.run.smithi096.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-05-28T06:00:57.499 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cephfs x86_64 2:19.0.0-3154.g9ed025fb.el9 ceph 161 k 2024-05-28T06:00:57.499 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rados x86_64 2:19.0.0-3154.g9ed025fb.el9 ceph 321 k 2024-05-28T06:00:57.499 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rbd x86_64 2:19.0.0-3154.g9ed025fb.el9 ceph 298 k 2024-05-28T06:00:57.499 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rgw x86_64 2:19.0.0-3154.g9ed025fb.el9 ceph 100 k 2024-05-28T06:00:57.499 INFO:teuthology.orchestra.run.smithi096.stdout: rbd-fuse x86_64 2:19.0.0-3154.g9ed025fb.el9 ceph 85 k 2024-05-28T06:00:57.499 INFO:teuthology.orchestra.run.smithi096.stdout: rbd-mirror x86_64 2:19.0.0-3154.g9ed025fb.el9 ceph 3.1 M 2024-05-28T06:00:57.499 INFO:teuthology.orchestra.run.smithi096.stdout: rbd-nbd x86_64 2:19.0.0-3154.g9ed025fb.el9 ceph 170 k 2024-05-28T06:00:57.499 INFO:teuthology.orchestra.run.smithi096.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-05-28T06:00:57.499 INFO:teuthology.orchestra.run.smithi096.stdout:Upgrading: 2024-05-28T06:00:57.500 INFO:teuthology.orchestra.run.smithi096.stdout: NetworkManager x86_64 1:1.47.90-1.el9 baseos 2.3 M 2024-05-28T06:00:57.500 INFO:teuthology.orchestra.run.smithi096.stdout: NetworkManager-libnm x86_64 1:1.47.90-1.el9 baseos 1.8 M 2024-05-28T06:00:57.500 INFO:teuthology.orchestra.run.smithi096.stdout: NetworkManager-team x86_64 1:1.47.90-1.el9 baseos 41 k 2024-05-28T06:00:57.500 INFO:teuthology.orchestra.run.smithi096.stdout: NetworkManager-tui x86_64 1:1.47.90-1.el9 baseos 248 k 2024-05-28T06:00:57.500 INFO:teuthology.orchestra.run.smithi096.stdout: elfutils-debuginfod-client x86_64 0.191-4.el9 baseos 37 k 2024-05-28T06:00:57.500 INFO:teuthology.orchestra.run.smithi096.stdout: elfutils-libelf x86_64 0.191-4.el9 baseos 207 k 2024-05-28T06:00:57.500 INFO:teuthology.orchestra.run.smithi096.stdout: elfutils-libs x86_64 0.191-4.el9 baseos 260 k 2024-05-28T06:00:57.500 INFO:teuthology.orchestra.run.smithi096.stdout: librados2 x86_64 2:19.0.0-3154.g9ed025fb.el9 ceph 3.4 M 2024-05-28T06:00:57.500 INFO:teuthology.orchestra.run.smithi096.stdout: librbd1 x86_64 2:19.0.0-3154.g9ed025fb.el9 ceph 3.1 M 2024-05-28T06:00:57.500 INFO:teuthology.orchestra.run.smithi096.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-05-28T06:00:57.500 INFO:teuthology.orchestra.run.smithi096.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-05-28T06:00:57.500 INFO:teuthology.orchestra.run.smithi096.stdout:Installing dependencies: 2024-05-28T06:00:57.501 INFO:teuthology.orchestra.run.smithi096.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-05-28T06:00:57.501 INFO:teuthology.orchestra.run.smithi096.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-05-28T06:00:57.501 INFO:teuthology.orchestra.run.smithi096.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-05-28T06:00:57.501 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-common x86_64 2:19.0.0-3154.g9ed025fb.el9 ceph 21 M 2024-05-28T06:00:57.501 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3154.g9ed025fb.el9 ceph-noarch 34 k 2024-05-28T06:00:57.501 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mds x86_64 2:19.0.0-3154.g9ed025fb.el9 ceph 2.4 M 2024-05-28T06:00:57.501 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3154.g9ed025fb.el9 ceph-noarch 252 k 2024-05-28T06:00:57.501 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mon x86_64 2:19.0.0-3154.g9ed025fb.el9 ceph 4.7 M 2024-05-28T06:00:57.501 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-osd x86_64 2:19.0.0-3154.g9ed025fb.el9 ceph 17 M 2024-05-28T06:00:57.501 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3154.g9ed025fb.el9 ceph-noarch 16 k 2024-05-28T06:00:57.501 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-selinux x86_64 2:19.0.0-3154.g9ed025fb.el9 ceph 25 k 2024-05-28T06:00:57.501 INFO:teuthology.orchestra.run.smithi096.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-05-28T06:00:57.502 INFO:teuthology.orchestra.run.smithi096.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-05-28T06:00:57.502 INFO:teuthology.orchestra.run.smithi096.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-05-28T06:00:57.502 INFO:teuthology.orchestra.run.smithi096.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-05-28T06:00:57.502 INFO:teuthology.orchestra.run.smithi096.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-05-28T06:00:57.502 INFO:teuthology.orchestra.run.smithi096.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-05-28T06:00:57.502 INFO:teuthology.orchestra.run.smithi096.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-05-28T06:00:57.502 INFO:teuthology.orchestra.run.smithi096.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-05-28T06:00:57.502 INFO:teuthology.orchestra.run.smithi096.stdout: libcephsqlite x86_64 2:19.0.0-3154.g9ed025fb.el9 ceph 164 k 2024-05-28T06:00:57.502 INFO:teuthology.orchestra.run.smithi096.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-05-28T06:00:57.502 INFO:teuthology.orchestra.run.smithi096.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-05-28T06:00:57.502 INFO:teuthology.orchestra.run.smithi096.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-05-28T06:00:57.503 INFO:teuthology.orchestra.run.smithi096.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-05-28T06:00:57.503 INFO:teuthology.orchestra.run.smithi096.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-05-28T06:00:57.503 INFO:teuthology.orchestra.run.smithi096.stdout: libradosstriper1 x86_64 2:19.0.0-3154.g9ed025fb.el9 ceph 501 k 2024-05-28T06:00:57.503 INFO:teuthology.orchestra.run.smithi096.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-05-28T06:00:57.503 INFO:teuthology.orchestra.run.smithi096.stdout: librgw2 x86_64 2:19.0.0-3154.g9ed025fb.el9 ceph 5.4 M 2024-05-28T06:00:57.503 INFO:teuthology.orchestra.run.smithi096.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-05-28T06:00:57.503 INFO:teuthology.orchestra.run.smithi096.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-05-28T06:00:57.503 INFO:teuthology.orchestra.run.smithi096.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-05-28T06:00:57.503 INFO:teuthology.orchestra.run.smithi096.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-05-28T06:00:57.503 INFO:teuthology.orchestra.run.smithi096.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-05-28T06:00:57.503 INFO:teuthology.orchestra.run.smithi096.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-05-28T06:00:57.503 INFO:teuthology.orchestra.run.smithi096.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-05-28T06:00:57.503 INFO:teuthology.orchestra.run.smithi096.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-05-28T06:00:57.503 INFO:teuthology.orchestra.run.smithi096.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-05-28T06:00:57.503 INFO:teuthology.orchestra.run.smithi096.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-05-28T06:00:57.504 INFO:teuthology.orchestra.run.smithi096.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-05-28T06:00:57.504 INFO:teuthology.orchestra.run.smithi096.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-05-28T06:00:57.504 INFO:teuthology.orchestra.run.smithi096.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-05-28T06:00:57.504 INFO:teuthology.orchestra.run.smithi096.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-05-28T06:00:57.504 INFO:teuthology.orchestra.run.smithi096.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-05-28T06:00:57.504 INFO:teuthology.orchestra.run.smithi096.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-05-28T06:00:57.504 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-05-28T06:00:57.504 INFO:teuthology.orchestra.run.smithi096.stdout: python3-ceph-argparse x86_64 2:19.0.0-3154.g9ed025fb.el9 ceph 45 k 2024-05-28T06:00:57.504 INFO:teuthology.orchestra.run.smithi096.stdout: python3-ceph-common x86_64 2:19.0.0-3154.g9ed025fb.el9 ceph 135 k 2024-05-28T06:00:57.504 INFO:teuthology.orchestra.run.smithi096.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-05-28T06:00:57.504 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-05-28T06:00:57.504 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-05-28T06:00:57.505 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-05-28T06:00:57.505 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-05-28T06:00:57.505 INFO:teuthology.orchestra.run.smithi096.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-05-28T06:00:57.505 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-05-28T06:00:57.505 INFO:teuthology.orchestra.run.smithi096.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-05-28T06:00:57.505 INFO:teuthology.orchestra.run.smithi096.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-05-28T06:00:57.505 INFO:teuthology.orchestra.run.smithi096.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-05-28T06:00:57.505 INFO:teuthology.orchestra.run.smithi096.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-05-28T06:00:57.505 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-05-28T06:00:57.505 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-05-28T06:00:57.505 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-05-28T06:00:57.505 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-05-28T06:00:57.506 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-05-28T06:00:57.506 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-05-28T06:00:57.506 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-05-28T06:00:57.506 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-05-28T06:00:57.506 INFO:teuthology.orchestra.run.smithi096.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-05-28T06:00:57.506 INFO:teuthology.orchestra.run.smithi096.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-05-28T06:00:57.506 INFO:teuthology.orchestra.run.smithi096.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-05-28T06:00:57.506 INFO:teuthology.orchestra.run.smithi096.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-05-28T06:00:57.506 INFO:teuthology.orchestra.run.smithi096.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-05-28T06:00:57.506 INFO:teuthology.orchestra.run.smithi096.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-05-28T06:00:57.506 INFO:teuthology.orchestra.run.smithi096.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-05-28T06:00:57.506 INFO:teuthology.orchestra.run.smithi096.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-05-28T06:00:57.506 INFO:teuthology.orchestra.run.smithi096.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-05-28T06:00:57.507 INFO:teuthology.orchestra.run.smithi096.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-05-28T06:00:57.507 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-05-28T06:00:57.507 INFO:teuthology.orchestra.run.smithi096.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-05-28T06:00:57.507 INFO:teuthology.orchestra.run.smithi096.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-05-28T06:00:57.507 INFO:teuthology.orchestra.run.smithi096.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-05-28T06:00:57.507 INFO:teuthology.orchestra.run.smithi096.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-05-28T06:00:57.507 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-05-28T06:00:57.507 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-05-28T06:00:57.507 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-05-28T06:00:57.507 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-05-28T06:00:57.507 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-05-28T06:00:57.507 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-05-28T06:00:57.507 INFO:teuthology.orchestra.run.smithi096.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-05-28T06:00:57.507 INFO:teuthology.orchestra.run.smithi096.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-05-28T06:00:57.508 INFO:teuthology.orchestra.run.smithi096.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-05-28T06:00:57.508 INFO:teuthology.orchestra.run.smithi096.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-05-28T06:00:57.508 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-05-28T06:00:57.508 INFO:teuthology.orchestra.run.smithi096.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-05-28T06:00:57.508 INFO:teuthology.orchestra.run.smithi096.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-05-28T06:00:57.508 INFO:teuthology.orchestra.run.smithi096.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-05-28T06:00:57.508 INFO:teuthology.orchestra.run.smithi096.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-05-28T06:00:57.508 INFO:teuthology.orchestra.run.smithi096.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-05-28T06:00:57.508 INFO:teuthology.orchestra.run.smithi096.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-05-28T06:00:57.508 INFO:teuthology.orchestra.run.smithi096.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-05-28T06:00:57.508 INFO:teuthology.orchestra.run.smithi096.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-05-28T06:00:57.508 INFO:teuthology.orchestra.run.smithi096.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-05-28T06:00:57.509 INFO:teuthology.orchestra.run.smithi096.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-05-28T06:00:57.509 INFO:teuthology.orchestra.run.smithi096.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-05-28T06:00:57.509 INFO:teuthology.orchestra.run.smithi096.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-05-28T06:00:57.509 INFO:teuthology.orchestra.run.smithi096.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-05-28T06:00:57.509 INFO:teuthology.orchestra.run.smithi096.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-05-28T06:00:57.509 INFO:teuthology.orchestra.run.smithi096.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-05-28T06:00:57.509 INFO:teuthology.orchestra.run.smithi096.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-05-28T06:00:57.509 INFO:teuthology.orchestra.run.smithi096.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-05-28T06:00:57.509 INFO:teuthology.orchestra.run.smithi096.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-05-28T06:00:57.509 INFO:teuthology.orchestra.run.smithi096.stdout:Installing weak dependencies: 2024-05-28T06:00:57.509 INFO:teuthology.orchestra.run.smithi096.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-05-28T06:00:57.509 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:00:57.509 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-05-28T06:00:57.510 INFO:teuthology.orchestra.run.smithi096.stdout:======================================================================================= 2024-05-28T06:00:57.510 INFO:teuthology.orchestra.run.smithi096.stdout:Install 148 Packages 2024-05-28T06:00:57.510 INFO:teuthology.orchestra.run.smithi096.stdout:Upgrade 11 Packages 2024-05-28T06:00:57.510 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:00:57.510 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 227 M 2024-05-28T06:00:57.510 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-05-28T06:00:58.522 INFO:teuthology.orchestra.run.smithi096.stdout:(1/159): ceph-19.0.0-3154.g9ed025fb.el9.x86_64. 23 kB/s | 6.6 kB 00:00 2024-05-28T06:00:58.822 INFO:teuthology.orchestra.run.smithi096.stdout:(2/159): ceph-fuse-19.0.0-3154.g9ed025fb.el9.x8 2.9 MB/s | 890 kB 00:00 2024-05-28T06:00:58.948 INFO:teuthology.orchestra.run.smithi096.stdout:(3/159): ceph-base-19.0.0-3154.g9ed025fb.el9.x8 7.6 MB/s | 5.4 MB 00:00 2024-05-28T06:00:59.056 INFO:teuthology.orchestra.run.smithi096.stdout:(4/159): ceph-immutable-object-cache-19.0.0-315 622 kB/s | 145 kB 00:00 2024-05-28T06:00:59.165 INFO:teuthology.orchestra.run.smithi096.stdout:(5/159): ceph-mds-19.0.0-3154.g9ed025fb.el9.x86 11 MB/s | 2.4 MB 00:00 2024-05-28T06:00:59.357 INFO:teuthology.orchestra.run.smithi096.stdout:(6/159): ceph-mgr-19.0.0-3154.g9ed025fb.el9.x86 5.0 MB/s | 1.5 MB 00:00 2024-05-28T06:00:59.482 INFO:teuthology.orchestra.run.smithi096.stdout:(7/159): ceph-mon-19.0.0-3154.g9ed025fb.el9.x86 15 MB/s | 4.7 MB 00:00 2024-05-28T06:01:00.216 INFO:teuthology.orchestra.run.smithi096.stdout:(8/159): ceph-radosgw-19.0.0-3154.g9ed025fb.el9 14 MB/s | 11 MB 00:00 2024-05-28T06:01:00.282 INFO:teuthology.orchestra.run.smithi096.stdout:(9/159): ceph-selinux-19.0.0-3154.g9ed025fb.el9 384 kB/s | 25 kB 00:00 2024-05-28T06:01:01.358 INFO:teuthology.orchestra.run.smithi096.stdout:(10/159): ceph-osd-19.0.0-3154.g9ed025fb.el9.x8 8.4 MB/s | 17 MB 00:01 2024-05-28T06:01:01.433 INFO:teuthology.orchestra.run.smithi096.stdout:(11/159): cephfs-mirror-19.0.0-3154.g9ed025fb.e 3.0 MB/s | 225 kB 00:00 2024-05-28T06:01:01.508 INFO:teuthology.orchestra.run.smithi096.stdout:(12/159): cephfs-shell-19.0.0-3154.g9ed025fb.el 365 kB/s | 27 kB 00:00 2024-05-28T06:01:01.584 INFO:teuthology.orchestra.run.smithi096.stdout:(13/159): libcephfs-devel-19.0.0-3154.g9ed025fb 428 kB/s | 32 kB 00:00 2024-05-28T06:01:01.616 INFO:teuthology.orchestra.run.smithi123.stdout:Extra Packages for Enterprise Linux 1.5 MB/s | 21 MB 00:14 2024-05-28T06:01:01.767 INFO:teuthology.orchestra.run.smithi096.stdout:(14/159): libcephfs2-19.0.0-3154.g9ed025fb.el9. 4.0 MB/s | 746 kB 00:00 2024-05-28T06:01:02.026 INFO:teuthology.orchestra.run.smithi096.stdout:(15/159): ceph-common-19.0.0-3154.g9ed025fb.el9 5.6 MB/s | 21 MB 00:03 2024-05-28T06:01:02.060 INFO:teuthology.orchestra.run.smithi096.stdout:(16/159): libcephsqlite-19.0.0-3154.g9ed025fb.e 562 kB/s | 164 kB 00:00 2024-05-28T06:01:02.093 INFO:teuthology.orchestra.run.smithi096.stdout:(17/159): librados-devel-19.0.0-3154.g9ed025fb. 1.9 MB/s | 127 kB 00:00 2024-05-28T06:01:02.143 INFO:teuthology.orchestra.run.smithi096.stdout:(18/159): libradosstriper1-19.0.0-3154.g9ed025f 5.9 MB/s | 501 kB 00:00 2024-05-28T06:01:02.210 INFO:teuthology.orchestra.run.smithi096.stdout:(19/159): python3-ceph-argparse-19.0.0-3154.g9e 687 kB/s | 45 kB 00:00 2024-05-28T06:01:02.294 INFO:teuthology.orchestra.run.smithi096.stdout:(20/159): python3-ceph-common-19.0.0-3154.g9ed0 1.6 MB/s | 135 kB 00:00 2024-05-28T06:01:02.377 INFO:teuthology.orchestra.run.smithi096.stdout:(21/159): python3-cephfs-19.0.0-3154.g9ed025fb. 1.9 MB/s | 161 kB 00:00 2024-05-28T06:01:02.478 INFO:teuthology.orchestra.run.smithi096.stdout:(22/159): python3-rados-19.0.0-3154.g9ed025fb.e 3.2 MB/s | 321 kB 00:00 2024-05-28T06:01:02.603 INFO:teuthology.orchestra.run.smithi096.stdout:(23/159): librgw2-19.0.0-3154.g9ed025fb.el9.x86 11 MB/s | 5.4 MB 00:00 2024-05-28T06:01:02.653 INFO:teuthology.orchestra.run.smithi096.stdout:(24/159): python3-rbd-19.0.0-3154.g9ed025fb.el9 1.7 MB/s | 298 kB 00:00 2024-05-28T06:01:02.687 INFO:teuthology.orchestra.run.smithi096.stdout:(25/159): python3-rgw-19.0.0-3154.g9ed025fb.el9 1.2 MB/s | 100 kB 00:00 2024-05-28T06:01:02.712 INFO:teuthology.orchestra.run.smithi096.stdout:(26/159): rbd-fuse-19.0.0-3154.g9ed025fb.el9.x8 1.4 MB/s | 85 kB 00:00 2024-05-28T06:01:02.805 INFO:teuthology.orchestra.run.smithi096.stdout:(27/159): rbd-nbd-19.0.0-3154.g9ed025fb.el9.x86 1.8 MB/s | 170 kB 00:00 2024-05-28T06:01:02.880 INFO:teuthology.orchestra.run.smithi096.stdout:(28/159): ceph-grafana-dashboards-19.0.0-3154.g 461 kB/s | 34 kB 00:00 2024-05-28T06:01:02.972 INFO:teuthology.orchestra.run.smithi096.stdout:(29/159): rbd-mirror-19.0.0-3154.g9ed025fb.el9. 11 MB/s | 3.1 MB 00:00 2024-05-28T06:01:02.997 INFO:teuthology.orchestra.run.smithi096.stdout:(30/159): ceph-mgr-cephadm-19.0.0-3154.g9ed025f 1.2 MB/s | 142 kB 00:00 2024-05-28T06:01:03.356 INFO:teuthology.orchestra.run.smithi096.stdout:(31/159): ceph-mgr-dashboard-19.0.0-3154.g9ed02 9.3 MB/s | 3.5 MB 00:00 2024-05-28T06:01:03.456 INFO:teuthology.orchestra.run.smithi096.stdout:(32/159): ceph-mgr-modules-core-19.0.0-3154.g9e 2.5 MB/s | 252 kB 00:00 2024-05-28T06:01:03.556 INFO:teuthology.orchestra.run.smithi096.stdout:(33/159): ceph-mgr-rook-19.0.0-3154.g9ed025fb.e 498 kB/s | 49 kB 00:00 2024-05-28T06:01:03.632 INFO:teuthology.orchestra.run.smithi096.stdout:(34/159): ceph-prometheus-alerts-19.0.0-3154.g9 222 kB/s | 16 kB 00:00 2024-05-28T06:01:03.724 INFO:teuthology.orchestra.run.smithi096.stdout:(35/159): ceph-volume-19.0.0-3154.g9ed025fb.el9 2.9 MB/s | 273 kB 00:00 2024-05-28T06:01:03.849 INFO:teuthology.orchestra.run.smithi096.stdout:(36/159): ceph-mgr-diskprediction-local-19.0.0- 8.7 MB/s | 7.4 MB 00:00 2024-05-28T06:01:03.899 INFO:teuthology.orchestra.run.smithi096.stdout:(37/159): cephadm-19.0.0-3154.g9ed025fb.el9.noa 4.3 MB/s | 762 kB 00:00 2024-05-28T06:01:03.924 INFO:teuthology.orchestra.run.smithi096.stdout:(38/159): cephfs-top-19.0.0-3154.g9ed025fb.el9. 245 kB/s | 18 kB 00:00 2024-05-28T06:01:04.750 INFO:teuthology.orchestra.run.smithi096.stdout:(39/159): ceph-test-19.0.0-3154.g9ed025fb.el9.x 11 MB/s | 49 MB 00:04 2024-05-28T06:01:05.183 INFO:teuthology.orchestra.run.smithi096.stdout:(40/159): fuse-2.9.9-15.el9.x86_64.rpm 63 kB/s | 80 kB 00:01 2024-05-28T06:01:05.509 INFO:teuthology.orchestra.run.smithi096.stdout:(41/159): libgfortran-11.4.1-3.el9.x86_64.rpm 507 kB/s | 803 kB 00:01 2024-05-28T06:01:05.625 INFO:teuthology.orchestra.run.smithi096.stdout:(42/159): libquadmath-11.4.1-3.el9.x86_64.rpm 225 kB/s | 195 kB 00:00 2024-05-28T06:01:06.159 INFO:teuthology.orchestra.run.smithi096.stdout:(43/159): python3-cffi-1.14.5-5.el9.x86_64.rpm 260 kB/s | 253 kB 00:00 2024-05-28T06:01:06.476 INFO:teuthology.orchestra.run.smithi096.stdout:(44/159): python3-ply-3.11-14.el9.noarch.rpm 125 kB/s | 106 kB 00:00 2024-05-28T06:01:06.685 INFO:teuthology.orchestra.run.smithi096.stdout:(45/159): python3-cryptography-36.0.1-4.el9.x86 1.0 MB/s | 1.2 MB 00:01 2024-05-28T06:01:07.010 INFO:teuthology.orchestra.run.smithi096.stdout:(46/159): python3-pycparser-2.20-6.el9.noarch.r 159 kB/s | 135 kB 00:00 2024-05-28T06:01:07.177 INFO:teuthology.orchestra.run.smithi096.stdout:(47/159): automake-1.16.2-8.el9.noarch.rpm 4.1 MB/s | 689 kB 00:00 2024-05-28T06:01:07.269 INFO:teuthology.orchestra.run.smithi096.stdout:(48/159): bison-3.7.4-5.el9.x86_64.rpm 10 MB/s | 943 kB 00:00 2024-05-28T06:01:07.302 INFO:teuthology.orchestra.run.smithi096.stdout:(49/159): python3-requests-2.25.1-8.el9.noarch. 151 kB/s | 125 kB 00:00 2024-05-28T06:01:07.336 INFO:teuthology.orchestra.run.smithi096.stdout:(50/159): boost-program-options-1.75.0-8.el9.x8 1.6 MB/s | 107 kB 00:00 2024-05-28T06:01:07.386 INFO:teuthology.orchestra.run.smithi096.stdout:(51/159): flex-2.6.4-9.el9.x86_64.rpm 6.2 MB/s | 313 kB 00:00 2024-05-28T06:01:07.412 INFO:teuthology.orchestra.run.smithi096.stdout:(52/159): elfutils-libelf-devel-0.191-4.el9.x86 201 kB/s | 22 kB 00:00 2024-05-28T06:01:07.437 INFO:teuthology.orchestra.run.smithi096.stdout:(53/159): flexiblas-3.0.4-8.el9.x86_64.rpm 659 kB/s | 33 kB 00:00 2024-05-28T06:01:07.479 INFO:teuthology.orchestra.run.smithi096.stdout:(54/159): flexiblas-openblas-openmp-3.0.4-8.el9 446 kB/s | 18 kB 00:00 2024-05-28T06:01:07.522 INFO:teuthology.orchestra.run.smithi096.stdout:(55/159): libaio-devel-0.3.111-13.el9.x86_64.rp 265 kB/s | 11 kB 00:00 2024-05-28T06:01:07.563 INFO:teuthology.orchestra.run.smithi096.stdout:(56/159): libpmemobj-1.12.1-1.el9.x86_64.rpm 4.0 MB/s | 160 kB 00:00 2024-05-28T06:01:07.597 INFO:teuthology.orchestra.run.smithi096.stdout:(57/159): python3-urllib3-1.26.5-5.el9.noarch.r 236 kB/s | 215 kB 00:00 2024-05-28T06:01:07.622 INFO:teuthology.orchestra.run.smithi096.stdout:(58/159): librabbitmq-0.11.0-7.el9.x86_64.rpm 783 kB/s | 45 kB 00:00 2024-05-28T06:01:07.722 INFO:teuthology.orchestra.run.smithi096.stdout:(59/159): flexiblas-netlib-3.0.4-8.el9.x86_64.r 9.7 MB/s | 3.0 MB 00:00 2024-05-28T06:01:07.741 INFO:teuthology.orchestra.run.smithi123.stdout:lab-extras 26 kB/s | 1.7 kB 00:00 2024-05-28T06:01:07.764 INFO:teuthology.orchestra.run.smithi096.stdout:(60/159): libtool-2.4.6-45.el9.x86_64.rpm 4.0 MB/s | 582 kB 00:00 2024-05-28T06:01:07.789 INFO:teuthology.orchestra.run.smithi096.stdout:(61/159): libuuid-devel-2.37.4-18.el9.x86_64.rp 385 kB/s | 25 kB 00:00 2024-05-28T06:01:07.815 INFO:teuthology.orchestra.run.smithi096.stdout:(62/159): libzstd-devel-1.5.1-2.el9.x86_64.rpm 960 kB/s | 47 kB 00:00 2024-05-28T06:01:07.848 INFO:teuthology.orchestra.run.smithi096.stdout:(63/159): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.9 MB/s | 292 kB 00:00 2024-05-28T06:01:07.882 INFO:teuthology.orchestra.run.smithi096.stdout:(64/159): lua-5.4.4-4.el9.x86_64.rpm 2.8 MB/s | 188 kB 00:00 2024-05-28T06:01:07.916 INFO:teuthology.orchestra.run.smithi096.stdout:(65/159): librdkafka-1.6.1-102.el9.x86_64.rpm 2.0 MB/s | 662 kB 00:00 2024-05-28T06:01:07.941 INFO:teuthology.orchestra.run.smithi096.stdout:(66/159): openblas-0.3.21-2.el9.x86_64.rpm 386 kB/s | 35 kB 00:00 2024-05-28T06:01:08.041 INFO:teuthology.orchestra.run.smithi096.stdout:(67/159): postgresql-13.14-1.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-05-28T06:01:08.233 INFO:teuthology.orchestra.run.smithi096.stdout:(68/159): openssl-devel-3.2.1-1.el9.x86_64.rpm 14 MB/s | 4.4 MB 00:00 2024-05-28T06:01:08.333 INFO:teuthology.orchestra.run.smithi096.stdout:(69/159): openblas-openmp-0.3.21-2.el9.x86_64.r 11 MB/s | 4.7 MB 00:00 2024-05-28T06:01:08.384 INFO:teuthology.orchestra.run.smithi096.stdout:(70/159): postgresql-contrib-13.14-1.el9.x86_64 2.6 MB/s | 911 kB 00:00 2024-05-28T06:01:08.400 INFO:teuthology.orchestra.run.smithi096.stdout:(71/159): postgresql-private-libs-13.14-1.el9.x 800 kB/s | 133 kB 00:00 2024-05-28T06:01:08.442 INFO:teuthology.orchestra.run.smithi096.stdout:(72/159): python3-attrs-20.3.0-7.el9.noarch.rpm 2.1 MB/s | 87 kB 00:00 2024-05-28T06:01:08.509 INFO:teuthology.orchestra.run.smithi096.stdout:(73/159): protobuf-3.14.0-13.el9.x86_64.rpm 9.4 MB/s | 1.0 MB 00:00 2024-05-28T06:01:08.551 INFO:teuthology.orchestra.run.smithi096.stdout:(74/159): python3-colorama-0.4.6-3.el9.noarch.r 1.5 MB/s | 61 kB 00:00 2024-05-28T06:01:08.610 INFO:teuthology.orchestra.run.smithi096.stdout:(75/159): python3-devel-3.9.18-3.el9.x86_64.rpm 4.1 MB/s | 244 kB 00:00 2024-05-28T06:01:08.660 INFO:teuthology.orchestra.run.smithi096.stdout:(76/159): python3-jinja2-2.11.3-6.el9.noarch.rp 5.0 MB/s | 248 kB 00:00 2024-05-28T06:01:08.702 INFO:teuthology.orchestra.run.smithi096.stdout:(77/159): python3-jsonpatch-1.21-16.el9.noarch. 619 kB/s | 26 kB 00:00 2024-05-28T06:01:08.744 INFO:teuthology.orchestra.run.smithi096.stdout:(78/159): python3-jsonpointer-2.0-4.el9.noarch. 469 kB/s | 19 kB 00:00 2024-05-28T06:01:08.852 INFO:teuthology.orchestra.run.smithi096.stdout:(79/159): postgresql-server-13.14-1.el9.x86_64. 11 MB/s | 5.8 MB 00:00 2024-05-28T06:01:08.886 INFO:teuthology.orchestra.run.smithi096.stdout:(80/159): python3-mako-1.1.4-6.el9.noarch.rpm 1.2 MB/s | 172 kB 00:00 2024-05-28T06:01:08.912 INFO:teuthology.orchestra.run.smithi096.stdout:(81/159): python3-markupsafe-1.1.1-12.el9.x86_6 598 kB/s | 35 kB 00:00 2024-05-28T06:01:08.962 INFO:teuthology.orchestra.run.smithi096.stdout:(82/159): python3-numpy-f2py-1.20.1-5.el9.x86_6 5.0 MB/s | 250 kB 00:00 2024-05-28T06:01:09.087 INFO:teuthology.orchestra.run.smithi096.stdout:(83/159): python3-babel-2.9.1-2.el9.noarch.rpm 9.3 MB/s | 6.0 MB 00:00 2024-05-28T06:01:09.121 INFO:teuthology.orchestra.run.smithi096.stdout:(84/159): python3-oauthlib-3.1.1-5.el9.noarch.r 1.4 MB/s | 222 kB 00:00 2024-05-28T06:01:09.146 INFO:teuthology.orchestra.run.smithi096.stdout:(85/159): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 77 kB 00:00 2024-05-28T06:01:09.172 INFO:teuthology.orchestra.run.smithi096.stdout:(86/159): python3-prettytable-0.7.2-27.el9.noar 851 kB/s | 42 kB 00:00 2024-05-28T06:01:09.205 INFO:teuthology.orchestra.run.smithi096.stdout:(87/159): python3-protobuf-3.14.0-13.el9.noarch 4.6 MB/s | 269 kB 00:00 2024-05-28T06:01:09.239 INFO:teuthology.orchestra.run.smithi096.stdout:(88/159): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.3 MB/s | 159 kB 00:00 2024-05-28T06:01:09.272 INFO:teuthology.orchestra.run.smithi096.stdout:(89/159): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 279 kB 00:00 2024-05-28T06:01:09.289 INFO:teuthology.orchestra.run.smithi096.stdout:(90/159): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-05-28T06:01:09.306 INFO:teuthology.orchestra.run.smithi096.stdout:(91/159): python3-requests-oauthlib-1.3.0-12.el 1.6 MB/s | 54 kB 00:00 2024-05-28T06:01:09.357 INFO:teuthology.orchestra.run.smithi096.stdout:(92/159): python3-toml-0.10.2-6.el9.noarch.rpm 850 kB/s | 42 kB 00:00 2024-05-28T06:01:09.399 INFO:teuthology.orchestra.run.smithi096.stdout:(93/159): python3-wcwidth-0.2.5-8.el9.noarch.rp 1.1 MB/s | 44 kB 00:00 2024-05-28T06:01:09.449 INFO:teuthology.orchestra.run.smithi096.stdout:(94/159): qatlib-24.02.0-1.el9.x86_64.rpm 4.4 MB/s | 221 kB 00:00 2024-05-28T06:01:09.549 INFO:teuthology.orchestra.run.smithi096.stdout:(95/159): python3-numpy-1.20.1-5.el9.x86_64.rpm 7.7 MB/s | 5.1 MB 00:00 2024-05-28T06:01:09.574 INFO:teuthology.orchestra.run.smithi096.stdout:(96/159): qatlib-service-24.02.0-1.el9.x86_64.r 288 kB/s | 36 kB 00:00 2024-05-28T06:01:09.600 INFO:teuthology.orchestra.run.smithi096.stdout:(97/159): qatzip-libs-1.2.0-1.el9.x86_64.rpm 954 kB/s | 47 kB 00:00 2024-05-28T06:01:09.634 INFO:teuthology.orchestra.run.smithi096.stdout:(98/159): socat-1.7.4.1-5.el9.x86_64.rpm 5.2 MB/s | 305 kB 00:00 2024-05-28T06:01:09.650 INFO:teuthology.orchestra.run.smithi096.stdout:(99/159): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2024-05-28T06:01:09.684 INFO:teuthology.orchestra.run.smithi096.stdout:(100/159): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.9 MB/s | 64 kB 00:00 2024-05-28T06:01:09.718 INFO:teuthology.orchestra.run.smithi096.stdout:(101/159): zlib-devel-1.2.11-41.el9.x86_64.rpm 1.3 MB/s | 45 kB 00:00 2024-05-28T06:01:09.743 INFO:teuthology.orchestra.run.smithi096.stdout:(102/159): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 864 kB/s | 94 kB 00:00 2024-05-28T06:01:09.802 INFO:teuthology.orchestra.run.smithi096.stdout:(103/159): lua-devel-5.4.4-4.el9.x86_64.rpm 270 kB/s | 22 kB 00:00 2024-05-28T06:01:10.078 INFO:teuthology.orchestra.run.smithi096.stdout:(104/159): python3-scipy-1.6.2-8.el9.x86_64.rpm 21 MB/s | 16 MB 00:00 2024-05-28T06:01:10.127 INFO:teuthology.orchestra.run.smithi096.stdout:(105/159): protobuf-compiler-3.14.0-13.el9.x86_ 2.2 MB/s | 863 kB 00:00 2024-05-28T06:01:10.151 INFO:teuthology.orchestra.run.smithi123.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-28T06:01:10.152 INFO:teuthology.orchestra.run.smithi123.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-28T06:01:10.172 INFO:teuthology.orchestra.run.smithi123.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-05-28T06:01:10.174 INFO:teuthology.orchestra.run.smithi123.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-05-28T06:01:10.175 INFO:teuthology.orchestra.run.smithi123.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-05-28T06:01:10.176 INFO:teuthology.orchestra.run.smithi123.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-05-28T06:01:10.178 INFO:teuthology.orchestra.run.smithi123.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-05-28T06:01:10.182 INFO:teuthology.orchestra.run.smithi123.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-05-28T06:01:10.183 INFO:teuthology.orchestra.run.smithi123.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-05-28T06:01:10.254 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-05-28T06:01:10.273 INFO:teuthology.orchestra.run.smithi123.stdout:======================================================================================= 2024-05-28T06:01:10.274 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repository Size 2024-05-28T06:01:10.274 INFO:teuthology.orchestra.run.smithi123.stdout:======================================================================================= 2024-05-28T06:01:10.274 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-05-28T06:01:10.274 INFO:teuthology.orchestra.run.smithi123.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-05-28T06:01:10.274 INFO:teuthology.orchestra.run.smithi123.stdout: ceph x86_64 2:19.0.0-3154.g9ed025fb.el9 ceph 6.6 k 2024-05-28T06:01:10.274 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-base x86_64 2:19.0.0-3154.g9ed025fb.el9 ceph 5.4 M 2024-05-28T06:01:10.274 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-fuse x86_64 2:19.0.0-3154.g9ed025fb.el9 ceph 890 k 2024-05-28T06:01:10.274 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3154.g9ed025fb.el9 ceph 145 k 2024-05-28T06:01:10.274 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr x86_64 2:19.0.0-3154.g9ed025fb.el9 ceph 1.5 M 2024-05-28T06:01:10.274 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3154.g9ed025fb.el9 ceph-noarch 142 k 2024-05-28T06:01:10.274 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3154.g9ed025fb.el9 ceph-noarch 3.5 M 2024-05-28T06:01:10.275 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3154.g9ed025fb.el9 ceph-noarch 7.4 M 2024-05-28T06:01:10.275 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-rook noarch 2:19.0.0-3154.g9ed025fb.el9 ceph-noarch 49 k 2024-05-28T06:01:10.275 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-radosgw x86_64 2:19.0.0-3154.g9ed025fb.el9 ceph 11 M 2024-05-28T06:01:10.275 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-test x86_64 2:19.0.0-3154.g9ed025fb.el9 ceph 49 M 2024-05-28T06:01:10.275 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-volume noarch 2:19.0.0-3154.g9ed025fb.el9 ceph-noarch 273 k 2024-05-28T06:01:10.275 INFO:teuthology.orchestra.run.smithi123.stdout: cephadm noarch 2:19.0.0-3154.g9ed025fb.el9 ceph-noarch 762 k 2024-05-28T06:01:10.275 INFO:teuthology.orchestra.run.smithi123.stdout: cephfs-mirror x86_64 2:19.0.0-3154.g9ed025fb.el9 ceph 225 k 2024-05-28T06:01:10.275 INFO:teuthology.orchestra.run.smithi123.stdout: cephfs-shell x86_64 2:19.0.0-3154.g9ed025fb.el9 ceph 27 k 2024-05-28T06:01:10.275 INFO:teuthology.orchestra.run.smithi123.stdout: cephfs-top noarch 2:19.0.0-3154.g9ed025fb.el9 ceph-noarch 18 k 2024-05-28T06:01:10.275 INFO:teuthology.orchestra.run.smithi123.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-05-28T06:01:10.275 INFO:teuthology.orchestra.run.smithi123.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-05-28T06:01:10.275 INFO:teuthology.orchestra.run.smithi123.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-05-28T06:01:10.275 INFO:teuthology.orchestra.run.smithi123.stdout: libcephfs-devel x86_64 2:19.0.0-3154.g9ed025fb.el9 ceph 32 k 2024-05-28T06:01:10.276 INFO:teuthology.orchestra.run.smithi123.stdout: libcephfs2 x86_64 2:19.0.0-3154.g9ed025fb.el9 ceph 746 k 2024-05-28T06:01:10.276 INFO:teuthology.orchestra.run.smithi123.stdout: librados-devel x86_64 2:19.0.0-3154.g9ed025fb.el9 ceph 127 k 2024-05-28T06:01:10.276 INFO:teuthology.orchestra.run.smithi123.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-05-28T06:01:10.276 INFO:teuthology.orchestra.run.smithi123.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-05-28T06:01:10.276 INFO:teuthology.orchestra.run.smithi123.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-05-28T06:01:10.276 INFO:teuthology.orchestra.run.smithi123.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-05-28T06:01:10.276 INFO:teuthology.orchestra.run.smithi123.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-05-28T06:01:10.276 INFO:teuthology.orchestra.run.smithi123.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-05-28T06:01:10.276 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cephfs x86_64 2:19.0.0-3154.g9ed025fb.el9 ceph 161 k 2024-05-28T06:01:10.276 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rados x86_64 2:19.0.0-3154.g9ed025fb.el9 ceph 321 k 2024-05-28T06:01:10.276 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rbd x86_64 2:19.0.0-3154.g9ed025fb.el9 ceph 298 k 2024-05-28T06:01:10.276 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rgw x86_64 2:19.0.0-3154.g9ed025fb.el9 ceph 100 k 2024-05-28T06:01:10.276 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-fuse x86_64 2:19.0.0-3154.g9ed025fb.el9 ceph 85 k 2024-05-28T06:01:10.277 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-mirror x86_64 2:19.0.0-3154.g9ed025fb.el9 ceph 3.1 M 2024-05-28T06:01:10.277 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-nbd x86_64 2:19.0.0-3154.g9ed025fb.el9 ceph 170 k 2024-05-28T06:01:10.277 INFO:teuthology.orchestra.run.smithi123.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-05-28T06:01:10.277 INFO:teuthology.orchestra.run.smithi123.stdout:Upgrading: 2024-05-28T06:01:10.277 INFO:teuthology.orchestra.run.smithi123.stdout: NetworkManager x86_64 1:1.47.90-1.el9 baseos 2.3 M 2024-05-28T06:01:10.277 INFO:teuthology.orchestra.run.smithi123.stdout: NetworkManager-libnm x86_64 1:1.47.90-1.el9 baseos 1.8 M 2024-05-28T06:01:10.277 INFO:teuthology.orchestra.run.smithi123.stdout: NetworkManager-team x86_64 1:1.47.90-1.el9 baseos 41 k 2024-05-28T06:01:10.277 INFO:teuthology.orchestra.run.smithi123.stdout: NetworkManager-tui x86_64 1:1.47.90-1.el9 baseos 248 k 2024-05-28T06:01:10.277 INFO:teuthology.orchestra.run.smithi123.stdout: elfutils-debuginfod-client x86_64 0.191-4.el9 baseos 37 k 2024-05-28T06:01:10.277 INFO:teuthology.orchestra.run.smithi123.stdout: elfutils-libelf x86_64 0.191-4.el9 baseos 207 k 2024-05-28T06:01:10.277 INFO:teuthology.orchestra.run.smithi123.stdout: elfutils-libs x86_64 0.191-4.el9 baseos 260 k 2024-05-28T06:01:10.277 INFO:teuthology.orchestra.run.smithi123.stdout: librados2 x86_64 2:19.0.0-3154.g9ed025fb.el9 ceph 3.4 M 2024-05-28T06:01:10.277 INFO:teuthology.orchestra.run.smithi123.stdout: librbd1 x86_64 2:19.0.0-3154.g9ed025fb.el9 ceph 3.1 M 2024-05-28T06:01:10.278 INFO:teuthology.orchestra.run.smithi123.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-05-28T06:01:10.278 INFO:teuthology.orchestra.run.smithi123.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-05-28T06:01:10.278 INFO:teuthology.orchestra.run.smithi123.stdout:Installing dependencies: 2024-05-28T06:01:10.278 INFO:teuthology.orchestra.run.smithi123.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-05-28T06:01:10.278 INFO:teuthology.orchestra.run.smithi123.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-05-28T06:01:10.278 INFO:teuthology.orchestra.run.smithi123.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-05-28T06:01:10.278 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-common x86_64 2:19.0.0-3154.g9ed025fb.el9 ceph 21 M 2024-05-28T06:01:10.278 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3154.g9ed025fb.el9 ceph-noarch 34 k 2024-05-28T06:01:10.278 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mds x86_64 2:19.0.0-3154.g9ed025fb.el9 ceph 2.4 M 2024-05-28T06:01:10.278 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3154.g9ed025fb.el9 ceph-noarch 252 k 2024-05-28T06:01:10.278 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mon x86_64 2:19.0.0-3154.g9ed025fb.el9 ceph 4.7 M 2024-05-28T06:01:10.278 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-osd x86_64 2:19.0.0-3154.g9ed025fb.el9 ceph 17 M 2024-05-28T06:01:10.278 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3154.g9ed025fb.el9 ceph-noarch 16 k 2024-05-28T06:01:10.278 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-selinux x86_64 2:19.0.0-3154.g9ed025fb.el9 ceph 25 k 2024-05-28T06:01:10.279 INFO:teuthology.orchestra.run.smithi123.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-05-28T06:01:10.279 INFO:teuthology.orchestra.run.smithi123.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-05-28T06:01:10.279 INFO:teuthology.orchestra.run.smithi123.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-05-28T06:01:10.279 INFO:teuthology.orchestra.run.smithi123.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-05-28T06:01:10.279 INFO:teuthology.orchestra.run.smithi123.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-05-28T06:01:10.279 INFO:teuthology.orchestra.run.smithi123.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-05-28T06:01:10.279 INFO:teuthology.orchestra.run.smithi123.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-05-28T06:01:10.279 INFO:teuthology.orchestra.run.smithi123.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-05-28T06:01:10.279 INFO:teuthology.orchestra.run.smithi123.stdout: libcephsqlite x86_64 2:19.0.0-3154.g9ed025fb.el9 ceph 164 k 2024-05-28T06:01:10.279 INFO:teuthology.orchestra.run.smithi123.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-05-28T06:01:10.279 INFO:teuthology.orchestra.run.smithi123.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-05-28T06:01:10.279 INFO:teuthology.orchestra.run.smithi123.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-05-28T06:01:10.279 INFO:teuthology.orchestra.run.smithi123.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-05-28T06:01:10.280 INFO:teuthology.orchestra.run.smithi123.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-05-28T06:01:10.280 INFO:teuthology.orchestra.run.smithi123.stdout: libradosstriper1 x86_64 2:19.0.0-3154.g9ed025fb.el9 ceph 501 k 2024-05-28T06:01:10.280 INFO:teuthology.orchestra.run.smithi123.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-05-28T06:01:10.280 INFO:teuthology.orchestra.run.smithi123.stdout: librgw2 x86_64 2:19.0.0-3154.g9ed025fb.el9 ceph 5.4 M 2024-05-28T06:01:10.280 INFO:teuthology.orchestra.run.smithi123.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-05-28T06:01:10.280 INFO:teuthology.orchestra.run.smithi123.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-05-28T06:01:10.280 INFO:teuthology.orchestra.run.smithi123.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-05-28T06:01:10.280 INFO:teuthology.orchestra.run.smithi123.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-05-28T06:01:10.280 INFO:teuthology.orchestra.run.smithi123.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-05-28T06:01:10.280 INFO:teuthology.orchestra.run.smithi123.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-05-28T06:01:10.280 INFO:teuthology.orchestra.run.smithi123.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-05-28T06:01:10.280 INFO:teuthology.orchestra.run.smithi123.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-05-28T06:01:10.280 INFO:teuthology.orchestra.run.smithi123.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-05-28T06:01:10.280 INFO:teuthology.orchestra.run.smithi123.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-05-28T06:01:10.281 INFO:teuthology.orchestra.run.smithi123.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-05-28T06:01:10.281 INFO:teuthology.orchestra.run.smithi123.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-05-28T06:01:10.281 INFO:teuthology.orchestra.run.smithi123.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-05-28T06:01:10.281 INFO:teuthology.orchestra.run.smithi123.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-05-28T06:01:10.281 INFO:teuthology.orchestra.run.smithi123.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-05-28T06:01:10.281 INFO:teuthology.orchestra.run.smithi123.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-05-28T06:01:10.281 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-05-28T06:01:10.281 INFO:teuthology.orchestra.run.smithi123.stdout: python3-ceph-argparse x86_64 2:19.0.0-3154.g9ed025fb.el9 ceph 45 k 2024-05-28T06:01:10.281 INFO:teuthology.orchestra.run.smithi123.stdout: python3-ceph-common x86_64 2:19.0.0-3154.g9ed025fb.el9 ceph 135 k 2024-05-28T06:01:10.281 INFO:teuthology.orchestra.run.smithi123.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-05-28T06:01:10.281 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-05-28T06:01:10.281 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-05-28T06:01:10.281 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-05-28T06:01:10.281 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-05-28T06:01:10.282 INFO:teuthology.orchestra.run.smithi123.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-05-28T06:01:10.282 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-05-28T06:01:10.282 INFO:teuthology.orchestra.run.smithi123.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-05-28T06:01:10.282 INFO:teuthology.orchestra.run.smithi123.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-05-28T06:01:10.282 INFO:teuthology.orchestra.run.smithi123.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-05-28T06:01:10.282 INFO:teuthology.orchestra.run.smithi123.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-05-28T06:01:10.282 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-05-28T06:01:10.282 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-05-28T06:01:10.282 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-05-28T06:01:10.282 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-05-28T06:01:10.282 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-05-28T06:01:10.282 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-05-28T06:01:10.282 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-05-28T06:01:10.283 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-05-28T06:01:10.283 INFO:teuthology.orchestra.run.smithi123.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-05-28T06:01:10.283 INFO:teuthology.orchestra.run.smithi123.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-05-28T06:01:10.283 INFO:teuthology.orchestra.run.smithi123.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-05-28T06:01:10.283 INFO:teuthology.orchestra.run.smithi123.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-05-28T06:01:10.283 INFO:teuthology.orchestra.run.smithi123.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-05-28T06:01:10.283 INFO:teuthology.orchestra.run.smithi123.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-05-28T06:01:10.283 INFO:teuthology.orchestra.run.smithi123.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-05-28T06:01:10.283 INFO:teuthology.orchestra.run.smithi123.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-05-28T06:01:10.283 INFO:teuthology.orchestra.run.smithi123.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-05-28T06:01:10.283 INFO:teuthology.orchestra.run.smithi123.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-05-28T06:01:10.283 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-05-28T06:01:10.283 INFO:teuthology.orchestra.run.smithi123.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-05-28T06:01:10.283 INFO:teuthology.orchestra.run.smithi123.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-05-28T06:01:10.284 INFO:teuthology.orchestra.run.smithi123.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-05-28T06:01:10.284 INFO:teuthology.orchestra.run.smithi123.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-05-28T06:01:10.284 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-05-28T06:01:10.284 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-05-28T06:01:10.284 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-05-28T06:01:10.284 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-05-28T06:01:10.284 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-05-28T06:01:10.284 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-05-28T06:01:10.284 INFO:teuthology.orchestra.run.smithi123.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-05-28T06:01:10.284 INFO:teuthology.orchestra.run.smithi123.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-05-28T06:01:10.284 INFO:teuthology.orchestra.run.smithi123.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-05-28T06:01:10.284 INFO:teuthology.orchestra.run.smithi123.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-05-28T06:01:10.284 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-05-28T06:01:10.284 INFO:teuthology.orchestra.run.smithi123.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-05-28T06:01:10.285 INFO:teuthology.orchestra.run.smithi123.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-05-28T06:01:10.285 INFO:teuthology.orchestra.run.smithi123.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-05-28T06:01:10.285 INFO:teuthology.orchestra.run.smithi123.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-05-28T06:01:10.285 INFO:teuthology.orchestra.run.smithi123.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-05-28T06:01:10.285 INFO:teuthology.orchestra.run.smithi123.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-05-28T06:01:10.285 INFO:teuthology.orchestra.run.smithi123.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-05-28T06:01:10.285 INFO:teuthology.orchestra.run.smithi123.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-05-28T06:01:10.285 INFO:teuthology.orchestra.run.smithi123.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-05-28T06:01:10.285 INFO:teuthology.orchestra.run.smithi123.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-05-28T06:01:10.285 INFO:teuthology.orchestra.run.smithi123.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-05-28T06:01:10.285 INFO:teuthology.orchestra.run.smithi123.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-05-28T06:01:10.285 INFO:teuthology.orchestra.run.smithi123.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-05-28T06:01:10.286 INFO:teuthology.orchestra.run.smithi123.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-05-28T06:01:10.286 INFO:teuthology.orchestra.run.smithi123.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-05-28T06:01:10.286 INFO:teuthology.orchestra.run.smithi123.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-05-28T06:01:10.286 INFO:teuthology.orchestra.run.smithi123.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-05-28T06:01:10.286 INFO:teuthology.orchestra.run.smithi123.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-05-28T06:01:10.286 INFO:teuthology.orchestra.run.smithi123.stdout:Installing weak dependencies: 2024-05-28T06:01:10.286 INFO:teuthology.orchestra.run.smithi123.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-05-28T06:01:10.286 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-05-28T06:01:10.286 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-05-28T06:01:10.286 INFO:teuthology.orchestra.run.smithi123.stdout:======================================================================================= 2024-05-28T06:01:10.286 INFO:teuthology.orchestra.run.smithi123.stdout:Install 148 Packages 2024-05-28T06:01:10.286 INFO:teuthology.orchestra.run.smithi123.stdout:Upgrade 11 Packages 2024-05-28T06:01:10.286 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-05-28T06:01:10.287 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 227 M 2024-05-28T06:01:10.287 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-05-28T06:01:10.302 INFO:teuthology.orchestra.run.smithi096.stdout:(106/159): abseil-cpp-20211102.0-3.el9.x86_64.r 1.1 MB/s | 548 kB 00:00 2024-05-28T06:01:10.336 INFO:teuthology.orchestra.run.smithi096.stdout:(107/159): gperftools-libs-2.9.1-2.el9.x86_64.r 1.2 MB/s | 309 kB 00:00 2024-05-28T06:01:10.361 INFO:teuthology.orchestra.run.smithi096.stdout:(108/159): grpc-data-1.46.7-10.el9.noarch.rpm 83 kB/s | 19 kB 00:00 2024-05-28T06:01:10.503 INFO:teuthology.orchestra.run.smithi096.stdout:(109/159): libarrow-9.0.0-11.el9.x86_64.rpm 22 MB/s | 4.4 MB 00:00 2024-05-28T06:01:10.528 INFO:teuthology.orchestra.run.smithi096.stdout:(110/159): libarrow-doc-9.0.0-11.el9.noarch.rpm 171 kB/s | 26 kB 00:00 2024-05-28T06:01:10.553 INFO:teuthology.orchestra.run.smithi096.stdout:(111/159): liboath-2.6.7-2.el9.x86_64.rpm 275 kB/s | 49 kB 00:00 2024-05-28T06:01:10.612 INFO:teuthology.orchestra.run.smithi096.stdout:(112/159): libunwind-1.6.2-1.el9.x86_64.rpm 1.2 MB/s | 67 kB 00:00 2024-05-28T06:01:10.637 INFO:teuthology.orchestra.run.smithi096.stdout:(113/159): luarocks-3.9.2-5.el9.noarch.rpm 1.8 MB/s | 151 kB 00:00 2024-05-28T06:01:10.687 INFO:teuthology.orchestra.run.smithi096.stdout:(114/159): parquet-libs-9.0.0-11.el9.x86_64.rpm 6.2 MB/s | 839 kB 00:00 2024-05-28T06:01:10.754 INFO:teuthology.orchestra.run.smithi096.stdout:(115/159): python3-asyncssh-2.13.2-5.el9.noarch 8.2 MB/s | 548 kB 00:00 2024-05-28T06:01:10.779 INFO:teuthology.orchestra.run.smithi096.stdout:(116/159): python3-bcrypt-3.2.2-1.el9.x86_64.rp 481 kB/s | 43 kB 00:00 2024-05-28T06:01:10.827 INFO:teuthology.orchestra.run.smithi096.stdout:(117/159): python3-cachetools-4.2.4-1.el9.noarc 234 kB/s | 32 kB 00:00 2024-05-28T06:01:10.855 INFO:teuthology.orchestra.run.smithi096.stdout:(118/159): python3-certifi-2021.10.8-2.el9.noar 150 kB/s | 15 kB 00:00 2024-05-28T06:01:10.914 INFO:teuthology.orchestra.run.smithi096.stdout:(119/159): python3-cheroot-8.6.0-4.el9.noarch.r 2.0 MB/s | 172 kB 00:00 2024-05-28T06:01:10.947 INFO:teuthology.orchestra.run.smithi096.stdout:(120/159): python3-cherrypy-18.6.1-2.el9.noarch 2.9 MB/s | 358 kB 00:00 2024-05-28T06:01:10.972 INFO:teuthology.orchestra.run.smithi096.stdout:(121/159): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.5 MB/s | 303 kB 00:00 2024-05-28T06:01:11.006 INFO:teuthology.orchestra.run.smithi096.stdout:(122/159): python3-google-auth-2.29.0-1.el9.noa 2.4 MB/s | 206 kB 00:00 2024-05-28T06:01:11.098 INFO:teuthology.orchestra.run.smithi096.stdout:(123/159): python3-grpcio-1.46.7-10.el9.x86_64. 17 MB/s | 2.0 MB 00:00 2024-05-28T06:01:11.123 INFO:teuthology.orchestra.run.smithi096.stdout:(124/159): python3-grpcio-tools-1.46.7-10.el9.x 981 kB/s | 144 kB 00:00 2024-05-28T06:01:11.148 INFO:teuthology.orchestra.run.smithi096.stdout:(125/159): python3-jaraco-8.2.1-3.el9.noarch.rp 85 kB/s | 11 kB 00:00 2024-05-28T06:01:11.184 INFO:teuthology.orchestra.run.smithi096.stdout:(126/159): python3-jaraco-classes-3.2.1-5.el9.n 306 kB/s | 18 kB 00:00 2024-05-28T06:01:11.207 INFO:teuthology.orchestra.run.smithi096.stdout:(127/159): python3-jaraco-collections-3.0.0-8.e 280 kB/s | 23 kB 00:00 2024-05-28T06:01:11.232 INFO:teuthology.orchestra.run.smithi096.stdout:(128/159): python3-jaraco-functools-3.5.0-2.el9 234 kB/s | 19 kB 00:00 2024-05-28T06:01:11.258 INFO:teuthology.orchestra.run.smithi096.stdout:(129/159): python3-jaraco-text-3.2.0-6.el9.noar 265 kB/s | 20 kB 00:00 2024-05-28T06:01:11.308 INFO:teuthology.orchestra.run.smithi096.stdout:(130/159): python3-kubernetes-26.1.0-2.el9.noar 14 MB/s | 1.0 MB 00:00 2024-05-28T06:01:11.333 INFO:teuthology.orchestra.run.smithi096.stdout:(131/159): python3-logutils-0.3.5-21.el9.noarch 468 kB/s | 46 kB 00:00 2024-05-28T06:01:11.358 INFO:teuthology.orchestra.run.smithi096.stdout:(132/159): python3-more-itertools-8.12.0-2.el9. 894 kB/s | 79 kB 00:00 2024-05-28T06:01:11.392 INFO:teuthology.orchestra.run.smithi096.stdout:(133/159): python3-natsort-7.1.1-5.el9.noarch.r 1.0 MB/s | 58 kB 00:00 2024-05-28T06:01:11.417 INFO:teuthology.orchestra.run.smithi096.stdout:(134/159): python3-pecan-1.4.2-3.el9.noarch.rpm 3.3 MB/s | 272 kB 00:00 2024-05-28T06:01:11.442 INFO:teuthology.orchestra.run.smithi096.stdout:(135/159): python3-portend-3.1.0-2.el9.noarch.r 206 kB/s | 16 kB 00:00 2024-05-28T06:01:11.468 INFO:teuthology.orchestra.run.smithi096.stdout:(136/159): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.8 MB/s | 90 kB 00:00 2024-05-28T06:01:11.493 INFO:teuthology.orchestra.run.smithi096.stdout:(137/159): python3-pyperclip-1.8.2-2.el9.noarch 381 kB/s | 28 kB 00:00 2024-05-28T06:01:11.518 INFO:teuthology.orchestra.run.smithi096.stdout:(138/159): python3-repoze-lru-0.7-16.el9.noarch 418 kB/s | 31 kB 00:00 2024-05-28T06:01:11.552 INFO:teuthology.orchestra.run.smithi096.stdout:(139/159): python3-routes-2.5.1-5.el9.noarch.rp 3.2 MB/s | 188 kB 00:00 2024-05-28T06:01:11.577 INFO:teuthology.orchestra.run.smithi096.stdout:(140/159): python3-rsa-4.9-2.el9.noarch.rpm 720 kB/s | 59 kB 00:00 2024-05-28T06:01:11.602 INFO:teuthology.orchestra.run.smithi096.stdout:(141/159): python3-tempora-5.0.0-2.el9.noarch.r 443 kB/s | 36 kB 00:00 2024-05-28T06:01:11.627 INFO:teuthology.orchestra.run.smithi096.stdout:(142/159): python3-typing-extensions-4.4.0-2.el 1.0 MB/s | 51 kB 00:00 2024-05-28T06:01:11.652 INFO:teuthology.orchestra.run.smithi096.stdout:(143/159): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-05-28T06:01:11.677 INFO:teuthology.orchestra.run.smithi096.stdout:(144/159): python3-websocket-client-1.2.3-2.el9 1.2 MB/s | 90 kB 00:00 2024-05-28T06:01:11.711 INFO:teuthology.orchestra.run.smithi096.stdout:(145/159): python3-werkzeug-2.0.3-3.el9.1.noarc 7.4 MB/s | 427 kB 00:00 2024-05-28T06:01:11.728 INFO:teuthology.orchestra.run.smithi096.stdout:(146/159): python3-zc-lockfile-2.0-10.el9.noarc 272 kB/s | 20 kB 00:00 2024-05-28T06:01:11.787 INFO:teuthology.orchestra.run.smithi096.stdout:(147/159): re2-20211101-3.el9.x86_64.rpm 1.8 MB/s | 192 kB 00:00 2024-05-28T06:01:11.854 INFO:teuthology.orchestra.run.smithi096.stdout:(148/159): thrift-0.15.0-2.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-05-28T06:01:12.121 INFO:teuthology.orchestra.run.smithi096.stdout:(149/159): librados2-19.0.0-3154.g9ed025fb.el9. 8.7 MB/s | 3.4 MB 00:00 2024-05-28T06:01:12.246 INFO:teuthology.orchestra.run.smithi096.stdout:(150/159): librbd1-19.0.0-3154.g9ed025fb.el9.x8 7.1 MB/s | 3.1 MB 00:00 2024-05-28T06:01:12.930 INFO:teuthology.orchestra.run.smithi096.stdout:(151/159): NetworkManager-team-1.47.90-1.el9.x8 61 kB/s | 41 kB 00:00 2024-05-28T06:01:13.121 INFO:teuthology.orchestra.run.smithi096.stdout:(152/159): NetworkManager-1.47.90-1.el9.x86_64. 1.8 MB/s | 2.3 MB 00:01 2024-05-28T06:01:13.472 INFO:teuthology.orchestra.run.smithi096.stdout:(153/159): NetworkManager-libnm-1.47.90-1.el9.x 1.4 MB/s | 1.8 MB 00:01 2024-05-28T06:01:13.905 INFO:teuthology.orchestra.run.smithi096.stdout:(154/159): NetworkManager-tui-1.47.90-1.el9.x86 255 kB/s | 248 kB 00:00 2024-05-28T06:01:13.939 INFO:teuthology.orchestra.run.smithi096.stdout:(155/159): elfutils-debuginfod-client-0.191-4.e 45 kB/s | 37 kB 00:00 2024-05-28T06:01:14.267 INFO:teuthology.orchestra.run.smithi123.stdout:(1/159): ceph-19.0.0-3154.g9ed025fb.el9.x86_64. 26 kB/s | 6.6 kB 00:00 2024-05-28T06:01:14.456 INFO:teuthology.orchestra.run.smithi096.stdout:(156/159): elfutils-libelf-0.191-4.el9.x86_64.r 227 kB/s | 207 kB 00:00 2024-05-28T06:01:14.584 INFO:teuthology.orchestra.run.smithi123.stdout:(2/159): ceph-fuse-19.0.0-3154.g9ed025fb.el9.x8 2.8 MB/s | 890 kB 00:00 2024-05-28T06:01:14.651 INFO:teuthology.orchestra.run.smithi123.stdout:(3/159): ceph-immutable-object-cache-19.0.0-315 2.2 MB/s | 145 kB 00:00 2024-05-28T06:01:14.800 INFO:teuthology.orchestra.run.smithi096.stdout:(157/159): elfutils-libs-0.191-4.el9.x86_64.rpm 292 kB/s | 260 kB 00:00 2024-05-28T06:01:15.081 INFO:teuthology.orchestra.run.smithi096.stdout:(158/159): openssl-3.2.1-1.el9.x86_64.rpm 1.2 MB/s | 1.3 MB 00:01 2024-05-28T06:01:15.476 INFO:teuthology.orchestra.run.smithi123.stdout:(4/159): ceph-common-19.0.0-3154.g9ed025fb.el9. 14 MB/s | 21 MB 00:01 2024-05-28T06:01:15.543 INFO:teuthology.orchestra.run.smithi123.stdout:(5/159): ceph-mds-19.0.0-3154.g9ed025fb.el9.x86 2.7 MB/s | 2.4 MB 00:00 2024-05-28T06:01:15.606 INFO:teuthology.orchestra.run.smithi096.stdout:(159/159): openssl-libs-3.2.1-1.el9.x86_64.rpm 2.1 MB/s | 2.4 MB 00:01 2024-05-28T06:01:15.617 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-05-28T06:01:15.617 INFO:teuthology.orchestra.run.smithi096.stdout:Total 13 MB/s | 227 MB 00:18 2024-05-28T06:01:15.627 INFO:teuthology.orchestra.run.smithi123.stdout:(6/159): ceph-mgr-19.0.0-3154.g9ed025fb.el9.x86 10 MB/s | 1.5 MB 00:00 2024-05-28T06:01:16.319 INFO:teuthology.orchestra.run.smithi123.stdout:(7/159): ceph-base-19.0.0-3154.g9ed025fb.el9.x8 2.3 MB/s | 5.4 MB 00:02 2024-05-28T06:01:16.536 INFO:teuthology.orchestra.run.smithi123.stdout:(8/159): ceph-mon-19.0.0-3154.g9ed025fb.el9.x86 4.7 MB/s | 4.7 MB 00:00 2024-05-28T06:01:16.744 INFO:teuthology.orchestra.run.smithi123.stdout:(9/159): ceph-osd-19.0.0-3154.g9ed025fb.el9.x86 15 MB/s | 17 MB 00:01 2024-05-28T06:01:16.769 INFO:teuthology.orchestra.run.smithi123.stdout:(10/159): ceph-selinux-19.0.0-3154.g9ed025fb.el 108 kB/s | 25 kB 00:00 2024-05-28T06:01:16.828 INFO:teuthology.orchestra.run.smithi123.stdout:(11/159): cephfs-mirror-19.0.0-3154.g9ed025fb.e 3.8 MB/s | 225 kB 00:00 2024-05-28T06:01:16.887 INFO:teuthology.orchestra.run.smithi123.stdout:(12/159): cephfs-shell-19.0.0-3154.g9ed025fb.el 470 kB/s | 27 kB 00:00 2024-05-28T06:01:16.945 INFO:teuthology.orchestra.run.smithi123.stdout:(13/159): libcephfs-devel-19.0.0-3154.g9ed025fb 553 kB/s | 32 kB 00:00 2024-05-28T06:01:17.087 INFO:teuthology.orchestra.run.smithi123.stdout:(14/159): libcephfs2-19.0.0-3154.g9ed025fb.el9. 5.2 MB/s | 746 kB 00:00 2024-05-28T06:01:17.154 INFO:teuthology.orchestra.run.smithi123.stdout:(15/159): libcephsqlite-19.0.0-3154.g9ed025fb.e 2.4 MB/s | 164 kB 00:00 2024-05-28T06:01:17.229 INFO:teuthology.orchestra.run.smithi123.stdout:(16/159): librados-devel-19.0.0-3154.g9ed025fb. 1.7 MB/s | 127 kB 00:00 2024-05-28T06:01:17.388 INFO:teuthology.orchestra.run.smithi123.stdout:(17/159): libradosstriper1-19.0.0-3154.g9ed025f 3.1 MB/s | 501 kB 00:00 2024-05-28T06:01:18.096 INFO:teuthology.orchestra.run.smithi123.stdout:(18/159): librgw2-19.0.0-3154.g9ed025fb.el9.x86 7.6 MB/s | 5.4 MB 00:00 2024-05-28T06:01:18.180 INFO:teuthology.orchestra.run.smithi123.stdout:(19/159): python3-ceph-argparse-19.0.0-3154.g9e 547 kB/s | 45 kB 00:00 2024-05-28T06:01:18.322 INFO:teuthology.orchestra.run.smithi123.stdout:(20/159): python3-ceph-common-19.0.0-3154.g9ed0 956 kB/s | 135 kB 00:00 2024-05-28T06:01:18.406 INFO:teuthology.orchestra.run.smithi123.stdout:(21/159): python3-cephfs-19.0.0-3154.g9ed025fb. 1.9 MB/s | 161 kB 00:00 2024-05-28T06:01:18.506 INFO:teuthology.orchestra.run.smithi123.stdout:(22/159): python3-rados-19.0.0-3154.g9ed025fb.e 3.2 MB/s | 321 kB 00:00 2024-05-28T06:01:18.598 INFO:teuthology.orchestra.run.smithi123.stdout:(23/159): python3-rbd-19.0.0-3154.g9ed025fb.el9 3.2 MB/s | 298 kB 00:00 2024-05-28T06:01:18.682 INFO:teuthology.orchestra.run.smithi123.stdout:(24/159): python3-rgw-19.0.0-3154.g9ed025fb.el9 1.2 MB/s | 100 kB 00:00 2024-05-28T06:01:18.807 INFO:teuthology.orchestra.run.smithi123.stdout:(25/159): rbd-fuse-19.0.0-3154.g9ed025fb.el9.x8 687 kB/s | 85 kB 00:00 2024-05-28T06:01:18.807 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-05-28T06:01:19.057 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-05-28T06:01:19.057 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-05-28T06:01:19.091 INFO:teuthology.orchestra.run.smithi123.stdout:(26/159): ceph-radosgw-19.0.0-3154.g9ed025fb.el 3.8 MB/s | 11 MB 00:02 2024-05-28T06:01:19.166 INFO:teuthology.orchestra.run.smithi123.stdout:(27/159): rbd-nbd-19.0.0-3154.g9ed025fb.el9.x86 2.3 MB/s | 170 kB 00:00 2024-05-28T06:01:19.249 INFO:teuthology.orchestra.run.smithi123.stdout:(28/159): rbd-mirror-19.0.0-3154.g9ed025fb.el9. 7.0 MB/s | 3.1 MB 00:00 2024-05-28T06:01:19.275 INFO:teuthology.orchestra.run.smithi123.stdout:(29/159): ceph-grafana-dashboards-19.0.0-3154.g 317 kB/s | 34 kB 00:00 2024-05-28T06:01:19.308 INFO:teuthology.orchestra.run.smithi123.stdout:(30/159): ceph-mgr-cephadm-19.0.0-3154.g9ed025f 2.4 MB/s | 142 kB 00:00 2024-05-28T06:01:19.851 INFO:teuthology.orchestra.run.smithi123.stdout:(31/159): ceph-test-19.0.0-3154.g9ed025fb.el9.x 16 MB/s | 49 MB 00:03 2024-05-28T06:01:19.934 INFO:teuthology.orchestra.run.smithi123.stdout:(32/159): ceph-mgr-modules-core-19.0.0-3154.g9e 3.0 MB/s | 252 kB 00:00 2024-05-28T06:01:20.001 INFO:teuthology.orchestra.run.smithi123.stdout:(33/159): ceph-mgr-rook-19.0.0-3154.g9ed025fb.e 750 kB/s | 49 kB 00:00 2024-05-28T06:01:20.076 INFO:teuthology.orchestra.run.smithi123.stdout:(34/159): ceph-mgr-dashboard-19.0.0-3154.g9ed02 4.4 MB/s | 3.5 MB 00:00 2024-05-28T06:01:20.101 INFO:teuthology.orchestra.run.smithi123.stdout:(35/159): ceph-prometheus-alerts-19.0.0-3154.g9 165 kB/s | 16 kB 00:00 2024-05-28T06:01:20.151 INFO:teuthology.orchestra.run.smithi123.stdout:(36/159): ceph-volume-19.0.0-3154.g9ed025fb.el9 3.6 MB/s | 273 kB 00:00 2024-05-28T06:01:20.285 INFO:teuthology.orchestra.run.smithi123.stdout:(37/159): ceph-mgr-diskprediction-local-19.0.0- 7.6 MB/s | 7.4 MB 00:00 2024-05-28T06:01:20.311 INFO:teuthology.orchestra.run.smithi123.stdout:(38/159): cephfs-top-19.0.0-3154.g9ed025fb.el9. 115 kB/s | 18 kB 00:00 2024-05-28T06:01:20.361 INFO:teuthology.orchestra.run.smithi123.stdout:(39/159): cephadm-19.0.0-3154.g9ed025fb.el9.noa 2.9 MB/s | 762 kB 00:00 2024-05-28T06:01:20.478 INFO:teuthology.orchestra.run.smithi123.stdout:(40/159): fuse-2.9.9-15.el9.x86_64.rpm 420 kB/s | 80 kB 00:00 2024-05-28T06:01:20.511 INFO:teuthology.orchestra.run.smithi123.stdout:(41/159): libquadmath-11.4.1-3.el9.x86_64.rpm 1.3 MB/s | 195 kB 00:00 2024-05-28T06:01:20.553 INFO:teuthology.orchestra.run.smithi123.stdout:(42/159): libgfortran-11.4.1-3.el9.x86_64.rpm 3.3 MB/s | 803 kB 00:00 2024-05-28T06:01:20.604 INFO:teuthology.orchestra.run.smithi123.stdout:(43/159): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.0 MB/s | 253 kB 00:00 2024-05-28T06:01:20.629 INFO:teuthology.orchestra.run.smithi123.stdout:(44/159): python3-ply-3.11-14.el9.noarch.rpm 1.4 MB/s | 106 kB 00:00 2024-05-28T06:01:20.662 INFO:teuthology.orchestra.run.smithi123.stdout:(45/159): python3-pycparser-2.20-6.el9.noarch.r 2.3 MB/s | 135 kB 00:00 2024-05-28T06:01:20.713 INFO:teuthology.orchestra.run.smithi123.stdout:(46/159): python3-requests-2.25.1-8.el9.noarch. 1.5 MB/s | 125 kB 00:00 2024-05-28T06:01:20.747 INFO:teuthology.orchestra.run.smithi123.stdout:(47/159): python3-urllib3-1.26.5-5.el9.noarch.r 2.5 MB/s | 215 kB 00:00 2024-05-28T06:01:20.797 INFO:teuthology.orchestra.run.smithi123.stdout:(48/159): python3-cryptography-36.0.1-4.el9.x86 4.3 MB/s | 1.2 MB 00:00 2024-05-28T06:01:21.606 INFO:teuthology.orchestra.run.smithi123.stdout:(49/159): boost-program-options-1.75.0-8.el9.x8 132 kB/s | 107 kB 00:00 2024-05-28T06:01:21.747 INFO:teuthology.orchestra.run.smithi123.stdout:(50/159): automake-1.16.2-8.el9.noarch.rpm 667 kB/s | 689 kB 00:01 2024-05-28T06:01:21.779 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-05-28T06:01:21.780 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-05-28T06:01:21.823 INFO:teuthology.orchestra.run.smithi123.stdout:(51/159): bison-3.7.4-5.el9.x86_64.rpm 877 kB/s | 943 kB 00:01 2024-05-28T06:01:22.265 INFO:teuthology.orchestra.run.smithi123.stdout:(52/159): elfutils-libelf-devel-0.191-4.el9.x86 33 kB/s | 22 kB 00:00 2024-05-28T06:01:22.598 INFO:teuthology.orchestra.run.smithi123.stdout:(53/159): flexiblas-3.0.4-8.el9.x86_64.rpm 42 kB/s | 33 kB 00:00 2024-05-28T06:01:22.690 INFO:teuthology.orchestra.run.smithi123.stdout:(54/159): flex-2.6.4-9.el9.x86_64.rpm 333 kB/s | 313 kB 00:00 2024-05-28T06:01:23.274 INFO:teuthology.orchestra.run.smithi123.stdout:(55/159): flexiblas-openblas-openmp-3.0.4-8.el9 27 kB/s | 18 kB 00:00 2024-05-28T06:01:23.324 INFO:teuthology.orchestra.run.smithi123.stdout:(56/159): libaio-devel-0.3.111-13.el9.x86_64.rp 18 kB/s | 11 kB 00:00 2024-05-28T06:01:23.474 INFO:teuthology.orchestra.run.smithi123.stdout:(57/159): flexiblas-netlib-3.0.4-8.el9.x86_64.r 2.5 MB/s | 3.0 MB 00:01 2024-05-28T06:01:24.074 INFO:teuthology.orchestra.run.smithi123.stdout:(58/159): librabbitmq-0.11.0-7.el9.x86_64.rpm 60 kB/s | 45 kB 00:00 2024-05-28T06:01:24.166 INFO:teuthology.orchestra.run.smithi123.stdout:(59/159): libpmemobj-1.12.1-1.el9.x86_64.rpm 180 kB/s | 160 kB 00:00 2024-05-28T06:01:24.708 INFO:teuthology.orchestra.run.smithi123.stdout:(60/159): librdkafka-1.6.1-102.el9.x86_64.rpm 538 kB/s | 662 kB 00:01 2024-05-28T06:01:24.875 INFO:teuthology.orchestra.run.smithi123.stdout:(61/159): libuuid-devel-2.37.4-18.el9.x86_64.rp 36 kB/s | 25 kB 00:00 2024-05-28T06:01:25.142 INFO:teuthology.orchestra.run.smithi123.stdout:(62/159): libtool-2.4.6-45.el9.x86_64.rpm 546 kB/s | 582 kB 00:01 2024-05-28T06:01:25.492 INFO:teuthology.orchestra.run.smithi123.stdout:(63/159): libzstd-devel-1.5.1-2.el9.x86_64.rpm 61 kB/s | 47 kB 00:00 2024-05-28T06:01:25.909 INFO:teuthology.orchestra.run.smithi123.stdout:(64/159): lttng-ust-2.12.0-6.el9.x86_64.rpm 283 kB/s | 292 kB 00:01 2024-05-28T06:01:26.085 INFO:teuthology.orchestra.run.smithi123.stdout:(65/159): lua-5.4.4-4.el9.x86_64.rpm 200 kB/s | 188 kB 00:00 2024-05-28T06:01:26.151 INFO:teuthology.orchestra.run.smithi123.stdout:(66/159): openblas-0.3.21-2.el9.x86_64.rpm 54 kB/s | 35 kB 00:00 2024-05-28T06:01:27.227 INFO:teuthology.orchestra.run.smithi123.stdout:(67/159): openblas-openmp-0.3.21-2.el9.x86_64.r 3.6 MB/s | 4.7 MB 00:01 2024-05-28T06:01:27.302 INFO:teuthology.orchestra.run.smithi123.stdout:(68/159): postgresql-13.14-1.el9.x86_64.rpm 1.4 MB/s | 1.6 MB 00:01 2024-05-28T06:01:27.394 INFO:teuthology.orchestra.run.smithi123.stdout:(69/159): openssl-devel-3.2.1-1.el9.x86_64.rpm 3.4 MB/s | 4.4 MB 00:01 2024-05-28T06:01:28.102 INFO:teuthology.orchestra.run.smithi123.stdout:(70/159): postgresql-private-libs-13.14-1.el9.x 167 kB/s | 133 kB 00:00 2024-05-28T06:01:28.361 INFO:teuthology.orchestra.run.smithi123.stdout:(71/159): postgresql-contrib-13.14-1.el9.x86_64 804 kB/s | 911 kB 00:01 2024-05-28T06:01:28.409 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-05-28T06:01:28.545 INFO:teuthology.orchestra.run.smithi096.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/170 2024-05-28T06:01:28.627 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/170 2024-05-28T06:01:28.703 INFO:teuthology.orchestra.run.smithi123.stdout:(72/159): postgresql-server-13.14-1.el9.x86_64. 4.4 MB/s | 5.8 MB 00:01 2024-05-28T06:01:28.801 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/170 2024-05-28T06:01:28.858 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/170 2024-05-28T06:01:28.889 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/170 2024-05-28T06:01:28.984 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/170 2024-05-28T06:01:29.220 INFO:teuthology.orchestra.run.smithi123.stdout:(73/159): protobuf-3.14.0-13.el9.x86_64.rpm 926 kB/s | 1.0 MB 00:01 2024-05-28T06:01:29.262 INFO:teuthology.orchestra.run.smithi123.stdout:(74/159): python3-attrs-20.3.0-7.el9.noarch.rpm 105 kB/s | 87 kB 00:00 2024-05-28T06:01:29.409 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/170 2024-05-28T06:01:29.545 INFO:teuthology.orchestra.run.smithi096.stdout: Upgrading : librados2-2:19.0.0-3154.g9ed025fb.el9.x86_64 8/170 2024-05-28T06:01:29.714 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: librados2-2:19.0.0-3154.g9ed025fb.el9.x86_64 8/170 2024-05-28T06:01:29.746 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libcephfs2-2:19.0.0-3154.g9ed025fb.el9.x86_64 9/170 2024-05-28T06:01:29.870 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: libcephfs2-2:19.0.0-3154.g9ed025fb.el9.x86_64 9/170 2024-05-28T06:01:29.903 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-rados-2:19.0.0-3154.g9ed025fb.el9.x86_64 10/170 2024-05-28T06:01:29.962 INFO:teuthology.orchestra.run.smithi123.stdout:(75/159): python3-colorama-0.4.6-3.el9.noarch.r 83 kB/s | 61 kB 00:00 2024-05-28T06:01:29.983 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/170 2024-05-28T06:01:30.154 INFO:teuthology.orchestra.run.smithi123.stdout:(76/159): python3-babel-2.9.1-2.el9.noarch.rpm 4.3 MB/s | 6.0 MB 00:01 2024-05-28T06:01:30.187 INFO:teuthology.orchestra.run.smithi123.stdout:(77/159): python3-devel-3.9.18-3.el9.x86_64.rpm 264 kB/s | 244 kB 00:00 2024-05-28T06:01:30.323 INFO:teuthology.orchestra.run.smithi096.stdout: Upgrading : elfutils-libelf-0.191-4.el9.x86_64 12/170 2024-05-28T06:01:30.402 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : postgresql-13.14-1.el9.x86_64 13/170 2024-05-28T06:01:30.426 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libcephsqlite-2:19.0.0-3154.g9ed025fb.el9.x86_64 14/170 2024-05-28T06:01:30.577 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3154.g9ed025fb.el9.x86_64 14/170 2024-05-28T06:01:30.611 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libradosstriper1-2:19.0.0-3154.g9ed025fb.el9.x86 15/170 2024-05-28T06:01:30.696 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3154.g9ed025fb.el9.x86 15/170 2024-05-28T06:01:30.745 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/170 2024-05-28T06:01:30.824 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/170 2024-05-28T06:01:30.863 INFO:teuthology.orchestra.run.smithi123.stdout:(78/159): python3-jsonpatch-1.21-16.el9.noarch. 36 kB/s | 26 kB 00:00 2024-05-28T06:01:30.896 INFO:teuthology.orchestra.run.smithi123.stdout:(79/159): python3-jsonpointer-2.0-4.el9.noarch. 27 kB/s | 19 kB 00:00 2024-05-28T06:01:30.996 INFO:teuthology.orchestra.run.smithi123.stdout:(80/159): python3-jinja2-2.11.3-6.el9.noarch.rp 240 kB/s | 248 kB 00:01 2024-05-28T06:01:31.092 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/170 2024-05-28T06:01:31.441 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/170 2024-05-28T06:01:31.597 INFO:teuthology.orchestra.run.smithi123.stdout:(81/159): python3-markupsafe-1.1.1-12.el9.x86_6 50 kB/s | 35 kB 00:00 2024-05-28T06:01:31.628 INFO:teuthology.orchestra.run.smithi096.stdout: Upgrading : NetworkManager-libnm-1:1.47.90-1.el9.x86_64 18/170 2024-05-28T06:01:31.716 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.90-1.el9.x86_64 18/170 2024-05-28T06:01:31.764 INFO:teuthology.orchestra.run.smithi123.stdout:(82/159): python3-mako-1.1.4-6.el9.noarch.rpm 192 kB/s | 172 kB 00:00 2024-05-28T06:01:31.899 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: NetworkManager-1:1.47.90-1.el9.x86_64 19/170 2024-05-28T06:01:32.006 INFO:teuthology.orchestra.run.smithi096.stdout: Upgrading : NetworkManager-1:1.47.90-1.el9.x86_64 19/170 2024-05-28T06:01:32.342 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: NetworkManager-1:1.47.90-1.el9.x86_64 19/170 2024-05-28T06:01:32.522 INFO:teuthology.orchestra.run.smithi123.stdout:(83/159): python3-numpy-f2py-1.20.1-5.el9.x86_6 271 kB/s | 250 kB 00:00 2024-05-28T06:01:32.526 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : re2-1:20211101-3.el9.x86_64 20/170 2024-05-28T06:01:32.630 INFO:teuthology.orchestra.run.smithi123.stdout:(84/159): python3-numpy-1.20.1-5.el9.x86_64.rpm 3.1 MB/s | 5.1 MB 00:01 2024-05-28T06:01:32.660 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 21/170 2024-05-28T06:01:32.664 INFO:teuthology.orchestra.run.smithi123.stdout:(85/159): python3-oauthlib-3.1.1-5.el9.noarch.r 247 kB/s | 222 kB 00:00 2024-05-28T06:01:32.704 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 22/170 2024-05-28T06:01:32.753 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : liboath-2.6.7-2.el9.x86_64 23/170 2024-05-28T06:01:32.853 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/170 2024-05-28T06:01:32.907 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 25/170 2024-05-28T06:01:32.973 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 26/170 2024-05-28T06:01:33.022 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 27/170 2024-05-28T06:01:33.069 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : lua-5.4.4-4.el9.x86_64 28/170 2024-05-28T06:01:33.183 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 29/170 2024-05-28T06:01:33.269 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 30/170 2024-05-28T06:01:33.306 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 31/170 2024-05-28T06:01:33.314 INFO:teuthology.orchestra.run.smithi123.stdout:(86/159): python3-packaging-20.9-5.el9.noarch.r 98 kB/s | 77 kB 00:00 2024-05-28T06:01:33.360 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 32/170 2024-05-28T06:01:33.440 INFO:teuthology.orchestra.run.smithi123.stdout:(87/159): python3-prettytable-0.7.2-27.el9.noar 52 kB/s | 42 kB 00:00 2024-05-28T06:01:33.467 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 33/170 2024-05-28T06:01:33.509 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-ceph-common-2:19.0.0-3154.g9ed025fb.el9. 34/170 2024-05-28T06:01:33.560 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-ceph-argparse-2:19.0.0-3154.g9ed025fb.el 35/170 2024-05-28T06:01:33.597 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-cephfs-2:19.0.0-3154.g9ed025fb.el9.x86_6 36/170 2024-05-28T06:01:33.682 INFO:teuthology.orchestra.run.smithi123.stdout:(88/159): python3-protobuf-3.14.0-13.el9.noarch 265 kB/s | 269 kB 00:01 2024-05-28T06:01:33.714 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 37/170 2024-05-28T06:01:33.767 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : luarocks-3.9.2-5.el9.noarch 38/170 2024-05-28T06:01:33.868 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 39/170 2024-05-28T06:01:34.051 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 40/170 2024-05-28T06:01:34.123 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 41/170 2024-05-28T06:01:34.149 INFO:teuthology.orchestra.run.smithi123.stdout:(89/159): python3-pyasn1-0.4.8-6.el9.noarch.rpm 191 kB/s | 159 kB 00:00 2024-05-28T06:01:34.399 INFO:teuthology.orchestra.run.smithi123.stdout:(90/159): python3-pyasn1-modules-0.4.8-6.el9.no 291 kB/s | 279 kB 00:00 2024-05-28T06:01:34.499 INFO:teuthology.orchestra.run.smithi123.stdout:(91/159): python3-pytz-2021.1-5.el9.noarch.rpm 67 kB/s | 51 kB 00:00 2024-05-28T06:01:34.813 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-rsa-4.9-2.el9.noarch 42/170 2024-05-28T06:01:34.908 INFO:teuthology.orchestra.run.smithi123.stdout:(92/159): python3-requests-oauthlib-1.3.0-12.el 71 kB/s | 54 kB 00:00 2024-05-28T06:01:34.918 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 43/170 2024-05-28T06:01:34.962 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 44/170 2024-05-28T06:01:35.001 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 45/170 2024-05-28T06:01:35.067 INFO:teuthology.orchestra.run.smithi096.stdout: Upgrading : elfutils-libs-0.191-4.el9.x86_64 46/170 2024-05-28T06:01:35.115 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : librados-devel-2:19.0.0-3154.g9ed025fb.el9.x86_6 47/170 2024-05-28T06:01:35.190 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 48/170 2024-05-28T06:01:35.233 INFO:teuthology.orchestra.run.smithi123.stdout:(93/159): python3-toml-0.10.2-6.el9.noarch.rpm 57 kB/s | 42 kB 00:00 2024-05-28T06:01:35.396 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 49/170 2024-05-28T06:01:35.475 INFO:teuthology.orchestra.run.smithi096.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 50/170 2024-05-28T06:01:35.554 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 51/170 2024-05-28T06:01:35.608 INFO:teuthology.orchestra.run.smithi123.stdout:(94/159): python3-wcwidth-0.2.5-8.el9.noarch.rp 63 kB/s | 44 kB 00:00 2024-05-28T06:01:35.677 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 52/170 2024-05-28T06:01:35.717 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 53/170 2024-05-28T06:01:35.757 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 54/170 2024-05-28T06:01:35.824 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 55/170 2024-05-28T06:01:35.906 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 56/170 2024-05-28T06:01:35.975 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 57/170 2024-05-28T06:01:36.039 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 58/170 2024-05-28T06:01:36.134 INFO:teuthology.orchestra.run.smithi123.stdout:(95/159): python3-scipy-1.6.2-8.el9.x86_64.rpm 9.8 MB/s | 16 MB 00:01 2024-05-28T06:01:36.176 INFO:teuthology.orchestra.run.smithi123.stdout:(96/159): qatlib-24.02.0-1.el9.x86_64.rpm 234 kB/s | 221 kB 00:00 2024-05-28T06:01:36.254 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 59/170 2024-05-28T06:01:36.308 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 60/170 2024-05-28T06:01:36.367 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 61/170 2024-05-28T06:01:36.493 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 62/170 2024-05-28T06:01:36.509 INFO:teuthology.orchestra.run.smithi123.stdout:(97/159): qatlib-service-24.02.0-1.el9.x86_64.r 40 kB/s | 36 kB 00:00 2024-05-28T06:01:36.532 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 63/170 2024-05-28T06:01:36.577 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 64/170 2024-05-28T06:01:36.613 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 65/170 2024-05-28T06:01:36.645 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 66/170 2024-05-28T06:01:36.843 INFO:teuthology.orchestra.run.smithi123.stdout:(98/159): qatzip-libs-1.2.0-1.el9.x86_64.rpm 67 kB/s | 47 kB 00:00 2024-05-28T06:01:36.846 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 67/170 2024-05-28T06:01:37.028 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 68/170 2024-05-28T06:01:37.078 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 69/170 2024-05-28T06:01:37.118 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 70/170 2024-05-28T06:01:37.163 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 71/170 2024-05-28T06:01:37.176 INFO:teuthology.orchestra.run.smithi123.stdout:(99/159): socat-1.7.4.1-5.el9.x86_64.rpm 305 kB/s | 305 kB 00:00 2024-05-28T06:01:37.216 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : uuid-1.6.2-55.el9.x86_64 72/170 2024-05-28T06:01:37.234 INFO:teuthology.orchestra.run.smithi123.stdout:(100/159): uuid-1.6.2-55.el9.x86_64.rpm 80 kB/s | 58 kB 00:00 2024-05-28T06:01:37.263 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 73/170 2024-05-28T06:01:37.304 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 74/170 2024-05-28T06:01:37.350 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 75/170 2024-05-28T06:01:37.392 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 76/170 2024-05-28T06:01:37.457 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 77/170 2024-05-28T06:01:37.508 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 78/170 2024-05-28T06:01:37.649 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 79/170 2024-05-28T06:01:37.715 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 80/170 2024-05-28T06:01:37.715 INFO:teuthology.orchestra.run.smithi123.stdout:(101/159): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 107 kB/s | 94 kB 00:00 2024-05-28T06:01:37.758 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 81/170 2024-05-28T06:01:37.793 INFO:teuthology.orchestra.run.smithi123.stdout:(102/159): lua-devel-5.4.4-4.el9.x86_64.rpm 291 kB/s | 22 kB 00:00 2024-05-28T06:01:37.828 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 82/170 2024-05-28T06:01:37.919 INFO:teuthology.orchestra.run.smithi123.stdout:(103/159): xmlstarlet-1.6.1-20.el9.x86_64.rpm 86 kB/s | 64 kB 00:00 2024-05-28T06:01:37.952 INFO:teuthology.orchestra.run.smithi123.stdout:(104/159): zlib-devel-1.2.11-41.el9.x86_64.rpm 63 kB/s | 45 kB 00:00 2024-05-28T06:01:37.978 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-packaging-20.9-5.el9.noarch 83/170 2024-05-28T06:01:38.004 INFO:teuthology.orchestra.run.smithi123.stdout:(105/159): protobuf-compiler-3.14.0-13.el9.x86_ 4.1 MB/s | 863 kB 00:00 2024-05-28T06:01:38.055 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 84/170 2024-05-28T06:01:38.412 INFO:teuthology.orchestra.run.smithi123.stdout:(106/159): grpc-data-1.46.7-10.el9.noarch.rpm 66 kB/s | 19 kB 00:00 2024-05-28T06:01:38.445 INFO:teuthology.orchestra.run.smithi123.stdout:(107/159): gperftools-libs-2.9.1-2.el9.x86_64.r 629 kB/s | 309 kB 00:00 2024-05-28T06:01:38.487 INFO:teuthology.orchestra.run.smithi123.stdout:(108/159): abseil-cpp-20211102.0-3.el9.x86_64.r 968 kB/s | 548 kB 00:00 2024-05-28T06:01:38.629 INFO:teuthology.orchestra.run.smithi123.stdout:(109/159): libarrow-doc-9.0.0-11.el9.noarch.rpm 143 kB/s | 26 kB 00:00 2024-05-28T06:01:38.671 INFO:teuthology.orchestra.run.smithi123.stdout:(110/159): liboath-2.6.7-2.el9.x86_64.rpm 270 kB/s | 49 kB 00:00 2024-05-28T06:01:38.985 INFO:teuthology.orchestra.run.smithi123.stdout:(111/159): libarrow-9.0.0-11.el9.x86_64.rpm 7.8 MB/s | 4.4 MB 00:00 2024-05-28T06:01:39.013 INFO:teuthology.orchestra.run.smithi123.stdout:(112/159): libunwind-1.6.2-1.el9.x86_64.rpm 181 kB/s | 67 kB 00:00 2024-05-28T06:01:39.038 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 85/170 2024-05-28T06:01:39.039 INFO:teuthology.orchestra.run.smithi123.stdout:(113/159): luarocks-3.9.2-5.el9.noarch.rpm 414 kB/s | 151 kB 00:00 2024-05-28T06:01:39.088 INFO:teuthology.orchestra.run.smithi123.stdout:(114/159): parquet-libs-9.0.0-11.el9.x86_64.rpm 8.1 MB/s | 839 kB 00:00 2024-05-28T06:01:39.095 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 86/170 2024-05-28T06:01:39.128 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 87/170 2024-05-28T06:01:39.139 INFO:teuthology.orchestra.run.smithi123.stdout:(115/159): python3-asyncssh-2.13.2-5.el9.noarch 5.8 MB/s | 548 kB 00:00 2024-05-28T06:01:39.164 INFO:teuthology.orchestra.run.smithi123.stdout:(116/159): python3-bcrypt-3.2.2-1.el9.x86_64.rp 373 kB/s | 43 kB 00:00 2024-05-28T06:01:39.189 INFO:teuthology.orchestra.run.smithi123.stdout:(117/159): python3-cachetools-4.2.4-1.el9.noarc 322 kB/s | 32 kB 00:00 2024-05-28T06:01:39.239 INFO:teuthology.orchestra.run.smithi123.stdout:(118/159): python3-certifi-2021.10.8-2.el9.noar 203 kB/s | 15 kB 00:00 2024-05-28T06:01:39.246 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 88/170 2024-05-28T06:01:39.265 INFO:teuthology.orchestra.run.smithi123.stdout:(119/159): python3-cheroot-8.6.0-4.el9.noarch.r 1.7 MB/s | 172 kB 00:00 2024-05-28T06:01:39.298 INFO:teuthology.orchestra.run.smithi123.stdout:(120/159): python3-cherrypy-18.6.1-2.el9.noarch 3.2 MB/s | 358 kB 00:00 2024-05-28T06:01:39.303 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 89/170 2024-05-28T06:01:39.340 INFO:teuthology.orchestra.run.smithi123.stdout:(121/159): python3-google-auth-2.29.0-1.el9.noa 2.9 MB/s | 206 kB 00:00 2024-05-28T06:01:39.357 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 90/170 2024-05-28T06:01:39.424 INFO:teuthology.orchestra.run.smithi123.stdout:(122/159): python3-grpcio-1.46.7-10.el9.x86_64. 16 MB/s | 2.0 MB 00:00 2024-05-28T06:01:39.449 INFO:teuthology.orchestra.run.smithi123.stdout:(123/159): python3-grpcio-tools-1.46.7-10.el9.x 1.3 MB/s | 144 kB 00:00 2024-05-28T06:01:39.467 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 91/170 2024-05-28T06:01:39.519 INFO:teuthology.orchestra.run.smithi123.stdout:(124/159): python3-jaraco-8.2.1-3.el9.noarch.rp 115 kB/s | 11 kB 00:00 2024-05-28T06:01:39.531 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 92/170 2024-05-28T06:01:39.550 INFO:teuthology.orchestra.run.smithi123.stdout:(125/159): python3-jaraco-classes-3.2.1-5.el9.n 177 kB/s | 18 kB 00:00 2024-05-28T06:01:39.591 INFO:teuthology.orchestra.run.smithi123.stdout:(126/159): python3-jaraco-collections-3.0.0-8.e 327 kB/s | 23 kB 00:00 2024-05-28T06:01:39.617 INFO:teuthology.orchestra.run.smithi123.stdout:(127/159): python3-jaraco-functools-3.5.0-2.el9 293 kB/s | 19 kB 00:00 2024-05-28T06:01:39.659 INFO:teuthology.orchestra.run.smithi123.stdout:(128/159): python3-jaraco-text-3.2.0-6.el9.noar 296 kB/s | 20 kB 00:00 2024-05-28T06:01:39.717 INFO:teuthology.orchestra.run.smithi123.stdout:(129/159): python3-kubernetes-26.1.0-2.el9.noar 10 MB/s | 1.0 MB 00:00 2024-05-28T06:01:39.742 INFO:teuthology.orchestra.run.smithi123.stdout:(130/159): python3-logutils-0.3.5-21.el9.noarch 558 kB/s | 46 kB 00:00 2024-05-28T06:01:39.748 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : openblas-0.3.21-2.el9.x86_64 93/170 2024-05-28T06:01:39.786 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 94/170 2024-05-28T06:01:39.793 INFO:teuthology.orchestra.run.smithi123.stdout:(131/159): python3-more-itertools-8.12.0-2.el9. 1.0 MB/s | 79 kB 00:00 2024-05-28T06:01:39.818 INFO:teuthology.orchestra.run.smithi123.stdout:(132/159): python3-natsort-7.1.1-5.el9.noarch.r 772 kB/s | 58 kB 00:00 2024-05-28T06:01:39.876 INFO:teuthology.orchestra.run.smithi123.stdout:(133/159): python3-pecan-1.4.2-3.el9.noarch.rpm 3.2 MB/s | 272 kB 00:00 2024-05-28T06:01:39.890 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 95/170 2024-05-28T06:01:39.902 INFO:teuthology.orchestra.run.smithi123.stdout:(134/159): python3-portend-3.1.0-2.el9.noarch.r 198 kB/s | 16 kB 00:00 2024-05-28T06:01:39.952 INFO:teuthology.orchestra.run.smithi123.stdout:(135/159): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.2 MB/s | 90 kB 00:00 2024-05-28T06:01:40.036 INFO:teuthology.orchestra.run.smithi123.stdout:(136/159): python3-repoze-lru-0.7-16.el9.noarch 371 kB/s | 31 kB 00:00 2024-05-28T06:01:40.111 INFO:teuthology.orchestra.run.smithi123.stdout:(137/159): python3-routes-2.5.1-5.el9.noarch.rp 2.5 MB/s | 188 kB 00:00 2024-05-28T06:01:40.187 INFO:teuthology.orchestra.run.smithi123.stdout:(138/159): python3-rsa-4.9-2.el9.noarch.rpm 794 kB/s | 59 kB 00:00 2024-05-28T06:01:40.262 INFO:teuthology.orchestra.run.smithi123.stdout:(139/159): python3-tempora-5.0.0-2.el9.noarch.r 481 kB/s | 36 kB 00:00 2024-05-28T06:01:40.337 INFO:teuthology.orchestra.run.smithi123.stdout:(140/159): python3-typing-extensions-4.4.0-2.el 685 kB/s | 51 kB 00:00 2024-05-28T06:01:40.421 INFO:teuthology.orchestra.run.smithi123.stdout:(141/159): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-05-28T06:01:40.498 INFO:teuthology.orchestra.run.smithi123.stdout:(142/159): python3-websocket-client-1.2.3-2.el9 1.2 MB/s | 90 kB 00:00 2024-05-28T06:01:40.581 INFO:teuthology.orchestra.run.smithi123.stdout:(143/159): python3-werkzeug-2.0.3-3.el9.1.noarc 5.0 MB/s | 427 kB 00:00 2024-05-28T06:01:40.638 INFO:teuthology.orchestra.run.smithi123.stdout:(144/159): python3-zc-lockfile-2.0-10.el9.noarc 346 kB/s | 20 kB 00:00 2024-05-28T06:01:40.713 INFO:teuthology.orchestra.run.smithi123.stdout:(145/159): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-05-28T06:01:40.717 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 96/170 2024-05-28T06:01:40.747 INFO:teuthology.orchestra.run.smithi123.stdout:(146/159): python3-pyperclip-1.8.2-2.el9.noarch 33 kB/s | 28 kB 00:00 2024-05-28T06:01:40.823 INFO:teuthology.orchestra.run.smithi123.stdout:(147/159): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-05-28T06:01:40.861 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 97/170 2024-05-28T06:01:41.049 INFO:teuthology.orchestra.run.smithi123.stdout:(148/159): librados2-19.0.0-3154.g9ed025fb.el9. 11 MB/s | 3.4 MB 00:00 2024-05-28T06:01:41.373 INFO:teuthology.orchestra.run.smithi123.stdout:(149/159): NetworkManager-1.47.90-1.el9.x86_64. 7.1 MB/s | 2.3 MB 00:00 2024-05-28T06:01:41.406 INFO:teuthology.orchestra.run.smithi123.stdout:(150/159): python3-cmd2-2.4.2-2.el9.noarch.rpm 142 kB/s | 303 kB 00:02 2024-05-28T06:01:41.482 INFO:teuthology.orchestra.run.smithi123.stdout:(151/159): librbd1-19.0.0-3154.g9ed025fb.el9.x8 4.8 MB/s | 3.1 MB 00:00 2024-05-28T06:01:41.557 INFO:teuthology.orchestra.run.smithi123.stdout:(152/159): NetworkManager-libnm-1.47.90-1.el9.x 9.9 MB/s | 1.8 MB 00:00 2024-05-28T06:01:41.582 INFO:teuthology.orchestra.run.smithi123.stdout:(153/159): NetworkManager-team-1.47.90-1.el9.x8 237 kB/s | 41 kB 00:00 2024-05-28T06:01:41.658 INFO:teuthology.orchestra.run.smithi123.stdout:(154/159): NetworkManager-tui-1.47.90-1.el9.x86 1.4 MB/s | 248 kB 00:00 2024-05-28T06:01:41.691 INFO:teuthology.orchestra.run.smithi123.stdout:(155/159): elfutils-libelf-0.191-4.el9.x86_64.r 1.9 MB/s | 207 kB 00:00 2024-05-28T06:01:41.716 INFO:teuthology.orchestra.run.smithi123.stdout:(156/159): elfutils-debuginfod-client-0.191-4.e 232 kB/s | 37 kB 00:00 2024-05-28T06:01:41.750 INFO:teuthology.orchestra.run.smithi123.stdout:(157/159): elfutils-libs-0.191-4.el9.x86_64.rpm 2.8 MB/s | 260 kB 00:00 2024-05-28T06:01:41.900 INFO:teuthology.orchestra.run.smithi123.stdout:(158/159): openssl-libs-3.2.1-1.el9.x86_64.rpm 13 MB/s | 2.4 MB 00:00 2024-05-28T06:01:42.275 INFO:teuthology.orchestra.run.smithi123.stdout:(159/159): openssl-3.2.1-1.el9.x86_64.rpm 2.3 MB/s | 1.3 MB 00:00 2024-05-28T06:01:42.286 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-05-28T06:01:42.287 INFO:teuthology.orchestra.run.smithi123.stdout:Total 7.1 MB/s | 227 MB 00:32 2024-05-28T06:01:42.496 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 98/170 2024-05-28T06:01:42.591 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 99/170 2024-05-28T06:01:42.627 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 100/170 2024-05-28T06:01:42.659 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 101/170 2024-05-28T06:01:43.023 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 102/170 2024-05-28T06:01:43.087 INFO:teuthology.orchestra.run.smithi096.stdout: Upgrading : librbd1-2:19.0.0-3154.g9ed025fb.el9.x86_64 103/170 2024-05-28T06:01:43.212 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: librbd1-2:19.0.0-3154.g9ed025fb.el9.x86_64 103/170 2024-05-28T06:01:43.244 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-rbd-2:19.0.0-3154.g9ed025fb.el9.x86_64 104/170 2024-05-28T06:01:43.301 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 105/170 2024-05-28T06:01:43.824 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 106/170 2024-05-28T06:01:43.857 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : librgw2-2:19.0.0-3154.g9ed025fb.el9.x86_64 107/170 2024-05-28T06:01:43.965 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: librgw2-2:19.0.0-3154.g9ed025fb.el9.x86_64 107/170 2024-05-28T06:01:43.992 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-rgw-2:19.0.0-3154.g9ed025fb.el9.x86_64 108/170 2024-05-28T06:01:45.475 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-05-28T06:01:45.727 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-05-28T06:01:45.727 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-05-28T06:01:46.654 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-common-2:19.0.0-3154.g9ed025fb.el9.x86_64 109/170 2024-05-28T06:01:46.705 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-common-2:19.0.0-3154.g9ed025fb.el9.x86_64 109/170 2024-05-28T06:01:47.355 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-common-2:19.0.0-3154.g9ed025fb.el9.x86_64 109/170 2024-05-28T06:01:47.400 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-base-2:19.0.0-3154.g9ed025fb.el9.x86_64 110/170 2024-05-28T06:01:47.573 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-base-2:19.0.0-3154.g9ed025fb.el9.x86_64 110/170 2024-05-28T06:01:47.573 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-05-28T06:01:47.574 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-05-28T06:01:47.574 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:01:47.609 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-selinux-2:19.0.0-3154.g9ed025fb.el9.x86_64 111/170 2024-05-28T06:01:48.336 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-05-28T06:01:48.337 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-05-28T06:01:54.887 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-05-28T06:01:54.982 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/170 2024-05-28T06:01:55.064 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/170 2024-05-28T06:01:55.167 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/170 2024-05-28T06:01:55.226 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/170 2024-05-28T06:01:55.259 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/170 2024-05-28T06:01:55.365 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/170 2024-05-28T06:01:55.789 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/170 2024-05-28T06:01:55.890 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : librados2-2:19.0.0-3154.g9ed025fb.el9.x86_64 8/170 2024-05-28T06:01:56.065 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: librados2-2:19.0.0-3154.g9ed025fb.el9.x86_64 8/170 2024-05-28T06:01:56.090 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libcephfs2-2:19.0.0-3154.g9ed025fb.el9.x86_64 9/170 2024-05-28T06:01:56.218 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libcephfs2-2:19.0.0-3154.g9ed025fb.el9.x86_64 9/170 2024-05-28T06:01:56.249 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-rados-2:19.0.0-3154.g9ed025fb.el9.x86_64 10/170 2024-05-28T06:01:56.333 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/170 2024-05-28T06:01:56.677 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : elfutils-libelf-0.191-4.el9.x86_64 12/170 2024-05-28T06:01:56.750 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : postgresql-13.14-1.el9.x86_64 13/170 2024-05-28T06:01:56.780 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libcephsqlite-2:19.0.0-3154.g9ed025fb.el9.x86_64 14/170 2024-05-28T06:01:56.929 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3154.g9ed025fb.el9.x86_64 14/170 2024-05-28T06:01:56.955 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libradosstriper1-2:19.0.0-3154.g9ed025fb.el9.x86 15/170 2024-05-28T06:01:57.027 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3154.g9ed025fb.el9.x86 15/170 2024-05-28T06:01:57.057 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/170 2024-05-28T06:01:57.121 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/170 2024-05-28T06:01:57.431 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/170 2024-05-28T06:01:57.774 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/170 2024-05-28T06:01:57.955 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : NetworkManager-libnm-1:1.47.90-1.el9.x86_64 18/170 2024-05-28T06:01:58.043 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.90-1.el9.x86_64 18/170 2024-05-28T06:01:58.234 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: NetworkManager-1:1.47.90-1.el9.x86_64 19/170 2024-05-28T06:01:58.341 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : NetworkManager-1:1.47.90-1.el9.x86_64 19/170 2024-05-28T06:01:58.669 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: NetworkManager-1:1.47.90-1.el9.x86_64 19/170 2024-05-28T06:01:58.921 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : re2-1:20211101-3.el9.x86_64 20/170 2024-05-28T06:01:59.103 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 21/170 2024-05-28T06:01:59.148 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 22/170 2024-05-28T06:01:59.197 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : liboath-2.6.7-2.el9.x86_64 23/170 2024-05-28T06:01:59.318 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/170 2024-05-28T06:01:59.370 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 25/170 2024-05-28T06:01:59.432 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 26/170 2024-05-28T06:01:59.481 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 27/170 2024-05-28T06:01:59.513 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : lua-5.4.4-4.el9.x86_64 28/170 2024-05-28T06:01:59.619 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 29/170 2024-05-28T06:01:59.690 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 30/170 2024-05-28T06:01:59.730 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 31/170 2024-05-28T06:01:59.784 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 32/170 2024-05-28T06:01:59.895 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 33/170 2024-05-28T06:01:59.935 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-ceph-common-2:19.0.0-3154.g9ed025fb.el9. 34/170 2024-05-28T06:02:00.001 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-ceph-argparse-2:19.0.0-3154.g9ed025fb.el 35/170 2024-05-28T06:02:00.034 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cephfs-2:19.0.0-3154.g9ed025fb.el9.x86_6 36/170 2024-05-28T06:02:00.155 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 37/170 2024-05-28T06:02:00.211 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : luarocks-3.9.2-5.el9.noarch 38/170 2024-05-28T06:02:00.311 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 39/170 2024-05-28T06:02:00.425 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3154.g9ed025fb.el9.x86_64 111/170 2024-05-28T06:02:00.425 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /sys 2024-05-28T06:02:00.426 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /proc 2024-05-28T06:02:00.426 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /mnt 2024-05-28T06:02:00.426 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /var/tmp 2024-05-28T06:02:00.426 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /home 2024-05-28T06:02:00.426 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /root 2024-05-28T06:02:00.426 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /tmp 2024-05-28T06:02:00.426 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:02:00.494 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 40/170 2024-05-28T06:02:00.556 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 41/170 2024-05-28T06:02:00.673 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-volume-2:19.0.0-3154.g9ed025fb.el9.noarch 112/170 2024-05-28T06:02:00.722 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-volume-2:19.0.0-3154.g9ed025fb.el9.noarch 112/170 2024-05-28T06:02:00.722 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-28T06:02:00.722 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-05-28T06:02:00.722 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:02:01.222 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-rsa-4.9-2.el9.noarch 42/170 2024-05-28T06:02:01.321 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 43/170 2024-05-28T06:02:01.364 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 44/170 2024-05-28T06:02:01.409 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 45/170 2024-05-28T06:02:01.478 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : elfutils-libs-0.191-4.el9.x86_64 46/170 2024-05-28T06:02:01.533 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : librados-devel-2:19.0.0-3154.g9ed025fb.el9.x86_6 47/170 2024-05-28T06:02:01.605 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 48/170 2024-05-28T06:02:01.824 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 49/170 2024-05-28T06:02:01.934 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 50/170 2024-05-28T06:02:02.015 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 51/170 2024-05-28T06:02:02.119 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 52/170 2024-05-28T06:02:02.158 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 53/170 2024-05-28T06:02:02.198 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 54/170 2024-05-28T06:02:02.265 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 55/170 2024-05-28T06:02:02.306 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 56/170 2024-05-28T06:02:02.317 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-osd-2:19.0.0-3154.g9ed025fb.el9.x86_64 113/170 2024-05-28T06:02:02.367 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 57/170 2024-05-28T06:02:02.397 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-osd-2:19.0.0-3154.g9ed025fb.el9.x86_64 113/170 2024-05-28T06:02:02.397 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-28T06:02:02.397 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-05-28T06:02:02.397 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-28T06:02:02.397 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-28T06:02:02.397 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:02:02.426 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 58/170 2024-05-28T06:02:02.615 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 59/170 2024-05-28T06:02:02.659 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mds-2:19.0.0-3154.g9ed025fb.el9.x86_64 114/170 2024-05-28T06:02:02.665 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 60/170 2024-05-28T06:02:02.711 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 61/170 2024-05-28T06:02:02.730 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mds-2:19.0.0-3154.g9ed025fb.el9.x86_64 114/170 2024-05-28T06:02:02.730 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-28T06:02:02.730 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-05-28T06:02:02.730 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-28T06:02:02.730 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-28T06:02:02.730 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:02:02.825 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 62/170 2024-05-28T06:02:02.864 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 63/170 2024-05-28T06:02:02.908 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 64/170 2024-05-28T06:02:02.938 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 65/170 2024-05-28T06:02:02.971 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 66/170 2024-05-28T06:02:03.152 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 67/170 2024-05-28T06:02:03.226 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mon-2:19.0.0-3154.g9ed025fb.el9.x86_64 115/170 2024-05-28T06:02:03.293 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mon-2:19.0.0-3154.g9ed025fb.el9.x86_64 115/170 2024-05-28T06:02:03.293 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-28T06:02:03.293 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-05-28T06:02:03.293 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-28T06:02:03.293 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-28T06:02:03.294 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:02:03.336 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 68/170 2024-05-28T06:02:03.380 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 69/170 2024-05-28T06:02:03.419 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 70/170 2024-05-28T06:02:03.443 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : automake-1.16.2-8.el9.noarch 116/170 2024-05-28T06:02:03.473 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 71/170 2024-05-28T06:02:03.510 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-ply-3.11-14.el9.noarch 117/170 2024-05-28T06:02:03.519 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : uuid-1.6.2-55.el9.x86_64 72/170 2024-05-28T06:02:03.564 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 73/170 2024-05-28T06:02:03.584 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 118/170 2024-05-28T06:02:03.606 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 74/170 2024-05-28T06:02:03.660 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 75/170 2024-05-28T06:02:03.760 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 76/170 2024-05-28T06:02:03.839 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 77/170 2024-05-28T06:02:03.850 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 119/170 2024-05-28T06:02:03.888 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 78/170 2024-05-28T06:02:03.914 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 120/170 2024-05-28T06:02:04.001 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 121/170 2024-05-28T06:02:04.043 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 79/170 2024-05-28T06:02:04.103 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 80/170 2024-05-28T06:02:04.110 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 122/170 2024-05-28T06:02:04.152 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 81/170 2024-05-28T06:02:04.223 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 82/170 2024-05-28T06:02:04.277 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 123/170 2024-05-28T06:02:04.330 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 124/170 2024-05-28T06:02:04.370 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 125/170 2024-05-28T06:02:04.372 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-packaging-20.9-5.el9.noarch 83/170 2024-05-28T06:02:04.404 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : fuse-2.9.9-15.el9.x86_64 126/170 2024-05-28T06:02:04.440 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 84/170 2024-05-28T06:02:05.068 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: cephadm-2:19.0.0-3154.g9ed025fb.el9.noarch 127/170 2024-05-28T06:02:05.113 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : cephadm-2:19.0.0-3154.g9ed025fb.el9.noarch 127/170 2024-05-28T06:02:05.169 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3154.g9ed025fb.e 128/170 2024-05-28T06:02:05.280 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3154.g9ed025fb. 129/170 2024-05-28T06:02:05.313 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3154.g9ed025fb.el9.noa 130/170 2024-05-28T06:02:05.412 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 85/170 2024-05-28T06:02:05.472 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 86/170 2024-05-28T06:02:05.506 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 87/170 2024-05-28T06:02:05.625 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 88/170 2024-05-28T06:02:05.714 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 89/170 2024-05-28T06:02:05.770 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 90/170 2024-05-28T06:02:05.882 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 91/170 2024-05-28T06:02:05.932 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 92/170 2024-05-28T06:02:06.167 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : openblas-0.3.21-2.el9.x86_64 93/170 2024-05-28T06:02:06.196 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 94/170 2024-05-28T06:02:06.307 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 95/170 2024-05-28T06:02:06.636 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3154.g9ed025fb.el9.noa 130/170 2024-05-28T06:02:06.676 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3154.g9ed025fb.el9.n 131/170 2024-05-28T06:02:07.162 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 96/170 2024-05-28T06:02:07.302 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 97/170 2024-05-28T06:02:07.790 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3154.g9ed025fb.el9.n 131/170 2024-05-28T06:02:07.815 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3154.g9ed 132/170 2024-05-28T06:02:07.962 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3154.g9ed 132/170 2024-05-28T06:02:08.139 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3154.g9ed025fb.el 133/170 2024-05-28T06:02:08.167 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mgr-2:19.0.0-3154.g9ed025fb.el9.x86_64 134/170 2024-05-28T06:02:08.235 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3154.g9ed025fb.el9.x86_64 134/170 2024-05-28T06:02:08.236 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-28T06:02:08.236 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-05-28T06:02:08.236 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-28T06:02:08.236 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-28T06:02:08.236 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:02:08.293 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mgr-rook-2:19.0.0-3154.g9ed025fb.el9.noarch 135/170 2024-05-28T06:02:08.329 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3154.g9ed025fb.el9.noarch 135/170 2024-05-28T06:02:08.451 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-2:19.0.0-3154.g9ed025fb.el9.x86_64 136/170 2024-05-28T06:02:08.531 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-fuse-2:19.0.0-3154.g9ed025fb.el9.x86_64 137/170 2024-05-28T06:02:08.955 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 98/170 2024-05-28T06:02:09.052 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 99/170 2024-05-28T06:02:09.088 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 100/170 2024-05-28T06:02:09.126 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 101/170 2024-05-28T06:02:09.504 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 102/170 2024-05-28T06:02:09.544 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libtool-2.4.6-45.el9.x86_64 138/170 2024-05-28T06:02:09.547 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : librbd1-2:19.0.0-3154.g9ed025fb.el9.x86_64 103/170 2024-05-28T06:02:09.579 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-radosgw-2:19.0.0-3154.g9ed025fb.el9.x86_64 139/170 2024-05-28T06:02:09.647 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3154.g9ed025fb.el9.x86_64 139/170 2024-05-28T06:02:09.647 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-28T06:02:09.648 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-05-28T06:02:09.648 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-28T06:02:09.648 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-28T06:02:09.648 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:02:09.668 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: librbd1-2:19.0.0-3154.g9ed025fb.el9.x86_64 103/170 2024-05-28T06:02:09.696 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3154.g9ed02 140/170 2024-05-28T06:02:09.704 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-rbd-2:19.0.0-3154.g9ed025fb.el9.x86_64 104/170 2024-05-28T06:02:09.758 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 105/170 2024-05-28T06:02:09.766 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3154.g9ed02 140/170 2024-05-28T06:02:09.766 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-28T06:02:09.766 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-05-28T06:02:09.766 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:02:09.847 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : cephfs-mirror-2:19.0.0-3154.g9ed025fb.el9.x86_64 141/170 2024-05-28T06:02:09.923 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3154.g9ed025fb.el9.x86_64 141/170 2024-05-28T06:02:09.923 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-28T06:02:09.923 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-05-28T06:02:09.923 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-28T06:02:09.923 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-28T06:02:09.923 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:02:10.289 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : rbd-mirror-2:19.0.0-3154.g9ed025fb.el9.x86_64 142/170 2024-05-28T06:02:10.290 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 106/170 2024-05-28T06:02:10.334 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : librgw2-2:19.0.0-3154.g9ed025fb.el9.x86_64 107/170 2024-05-28T06:02:10.360 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3154.g9ed025fb.el9.x86_64 142/170 2024-05-28T06:02:10.360 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-28T06:02:10.361 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-05-28T06:02:10.361 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-28T06:02:10.361 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-28T06:02:10.361 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:02:10.425 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: librgw2-2:19.0.0-3154.g9ed025fb.el9.x86_64 107/170 2024-05-28T06:02:10.452 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-rgw-2:19.0.0-3154.g9ed025fb.el9.x86_64 108/170 2024-05-28T06:02:13.159 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-common-2:19.0.0-3154.g9ed025fb.el9.x86_64 109/170 2024-05-28T06:02:13.198 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-common-2:19.0.0-3154.g9ed025fb.el9.x86_64 109/170 2024-05-28T06:02:13.850 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-common-2:19.0.0-3154.g9ed025fb.el9.x86_64 109/170 2024-05-28T06:02:13.894 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-base-2:19.0.0-3154.g9ed025fb.el9.x86_64 110/170 2024-05-28T06:02:14.054 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-base-2:19.0.0-3154.g9ed025fb.el9.x86_64 110/170 2024-05-28T06:02:14.054 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-05-28T06:02:14.054 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-05-28T06:02:14.054 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-05-28T06:02:14.086 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-selinux-2:19.0.0-3154.g9ed025fb.el9.x86_64 111/170 2024-05-28T06:02:15.475 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-test-2:19.0.0-3154.g9ed025fb.el9.x86_64 143/170 2024-05-28T06:02:15.535 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : rbd-fuse-2:19.0.0-3154.g9ed025fb.el9.x86_64 144/170 2024-05-28T06:02:15.612 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : rbd-nbd-2:19.0.0-3154.g9ed025fb.el9.x86_64 145/170 2024-05-28T06:02:15.655 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 146/170 2024-05-28T06:02:15.691 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 147/170 2024-05-28T06:02:16.013 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : cephfs-shell-2:19.0.0-3154.g9ed025fb.el9.x86_64 148/170 2024-05-28T06:02:16.068 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 149/170 2024-05-28T06:02:16.100 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libcephfs-devel-2:19.0.0-3154.g9ed025fb.el9.x86_ 150/170 2024-05-28T06:02:16.140 INFO:teuthology.orchestra.run.smithi096.stdout: Upgrading : elfutils-debuginfod-client-0.191-4.el9.x86_64 151/170 2024-05-28T06:02:16.187 INFO:teuthology.orchestra.run.smithi096.stdout: Upgrading : NetworkManager-team-1:1.47.90-1.el9.x86_64 152/170 2024-05-28T06:02:16.233 INFO:teuthology.orchestra.run.smithi096.stdout: Upgrading : NetworkManager-tui-1:1.47.90-1.el9.x86_64 153/170 2024-05-28T06:02:17.136 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 154/170 2024-05-28T06:02:17.202 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 154/170 2024-05-28T06:02:17.269 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 154/170 2024-05-28T06:02:18.631 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : cephfs-top-2:19.0.0-3154.g9ed025fb.el9.noarch 155/170 2024-05-28T06:02:18.746 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 156/170 2024-05-28T06:02:18.800 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 157/170 2024-05-28T06:02:18.938 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : flex-2.6.4-9.el9.x86_64 158/170 2024-05-28T06:02:18.981 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : bison-3.7.4-5.el9.x86_64 159/170 2024-05-28T06:02:19.014 INFO:teuthology.orchestra.run.smithi096.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/170 2024-05-28T06:02:19.065 INFO:teuthology.orchestra.run.smithi096.stdout: Cleanup : elfutils-debuginfod-client-0.190-2.el9.x86_64 161/170 2024-05-28T06:02:19.105 INFO:teuthology.orchestra.run.smithi096.stdout: Cleanup : elfutils-libs-0.190-2.el9.x86_64 162/170 2024-05-28T06:02:19.105 INFO:teuthology.orchestra.run.smithi096.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 163/170 2024-05-28T06:02:19.239 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 163/170 2024-05-28T06:02:19.285 INFO:teuthology.orchestra.run.smithi096.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 164/170 2024-05-28T06:02:19.286 INFO:teuthology.orchestra.run.smithi096.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 165/170 2024-05-28T06:02:19.405 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 165/170 2024-05-28T06:02:19.437 INFO:teuthology.orchestra.run.smithi096.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 166/170 2024-05-28T06:02:19.470 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-05-28T06:02:19.471 INFO:teuthology.orchestra.run.smithi096.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-05-28T06:02:19.799 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-05-28T06:02:19.800 INFO:teuthology.orchestra.run.smithi096.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 168/170 2024-05-28T06:02:19.969 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 168/170 2024-05-28T06:02:20.014 INFO:teuthology.orchestra.run.smithi096.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 169/170 2024-05-28T06:02:20.133 INFO:teuthology.orchestra.run.smithi096.stdout: Cleanup : elfutils-libelf-0.190-2.el9.x86_64 170/170 2024-05-28T06:02:20.401 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: elfutils-libelf-0.190-2.el9.x86_64 170/170 2024-05-28T06:02:20.401 INFO:teuthology.orchestra.run.smithi096.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-05-28T06:02:20.401 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:02:26.398 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-2:19.0.0-3154.g9ed025fb.el9.x86_64 1/170 2024-05-28T06:02:26.398 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-base-2:19.0.0-3154.g9ed025fb.el9.x86_64 2/170 2024-05-28T06:02:26.398 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-common-2:19.0.0-3154.g9ed025fb.el9.x86_64 3/170 2024-05-28T06:02:26.398 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-fuse-2:19.0.0-3154.g9ed025fb.el9.x86_64 4/170 2024-05-28T06:02:26.398 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3154.g9ed02 5/170 2024-05-28T06:02:26.398 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mds-2:19.0.0-3154.g9ed025fb.el9.x86_64 6/170 2024-05-28T06:02:26.398 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mgr-2:19.0.0-3154.g9ed025fb.el9.x86_64 7/170 2024-05-28T06:02:26.398 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mon-2:19.0.0-3154.g9ed025fb.el9.x86_64 8/170 2024-05-28T06:02:26.399 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-osd-2:19.0.0-3154.g9ed025fb.el9.x86_64 9/170 2024-05-28T06:02:26.399 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-radosgw-2:19.0.0-3154.g9ed025fb.el9.x86_64 10/170 2024-05-28T06:02:26.399 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-selinux-2:19.0.0-3154.g9ed025fb.el9.x86_64 11/170 2024-05-28T06:02:26.399 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-test-2:19.0.0-3154.g9ed025fb.el9.x86_64 12/170 2024-05-28T06:02:26.399 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : cephfs-mirror-2:19.0.0-3154.g9ed025fb.el9.x86_64 13/170 2024-05-28T06:02:26.399 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : cephfs-shell-2:19.0.0-3154.g9ed025fb.el9.x86_64 14/170 2024-05-28T06:02:26.399 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libcephfs-devel-2:19.0.0-3154.g9ed025fb.el9.x86_ 15/170 2024-05-28T06:02:26.399 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libcephfs2-2:19.0.0-3154.g9ed025fb.el9.x86_64 16/170 2024-05-28T06:02:26.399 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libcephsqlite-2:19.0.0-3154.g9ed025fb.el9.x86_64 17/170 2024-05-28T06:02:26.399 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librados-devel-2:19.0.0-3154.g9ed025fb.el9.x86_6 18/170 2024-05-28T06:02:26.399 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libradosstriper1-2:19.0.0-3154.g9ed025fb.el9.x86 19/170 2024-05-28T06:02:26.399 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librgw2-2:19.0.0-3154.g9ed025fb.el9.x86_64 20/170 2024-05-28T06:02:26.399 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3154.g9ed025fb.el 21/170 2024-05-28T06:02:26.399 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-ceph-common-2:19.0.0-3154.g9ed025fb.el9. 22/170 2024-05-28T06:02:26.400 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-cephfs-2:19.0.0-3154.g9ed025fb.el9.x86_6 23/170 2024-05-28T06:02:26.400 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-rados-2:19.0.0-3154.g9ed025fb.el9.x86_64 24/170 2024-05-28T06:02:26.400 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-rbd-2:19.0.0-3154.g9ed025fb.el9.x86_64 25/170 2024-05-28T06:02:26.400 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-rgw-2:19.0.0-3154.g9ed025fb.el9.x86_64 26/170 2024-05-28T06:02:26.400 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : rbd-fuse-2:19.0.0-3154.g9ed025fb.el9.x86_64 27/170 2024-05-28T06:02:26.400 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : rbd-mirror-2:19.0.0-3154.g9ed025fb.el9.x86_64 28/170 2024-05-28T06:02:26.400 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : rbd-nbd-2:19.0.0-3154.g9ed025fb.el9.x86_64 29/170 2024-05-28T06:02:26.400 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3154.g9ed025fb. 30/170 2024-05-28T06:02:26.400 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3154.g9ed025fb.el9.noa 31/170 2024-05-28T06:02:26.400 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3154.g9ed025fb.el9.n 32/170 2024-05-28T06:02:26.400 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3154.g9ed 33/170 2024-05-28T06:02:26.400 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3154.g9ed025fb.el 34/170 2024-05-28T06:02:26.400 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3154.g9ed025fb.el9.noarch 35/170 2024-05-28T06:02:26.401 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3154.g9ed025fb.e 36/170 2024-05-28T06:02:26.401 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-volume-2:19.0.0-3154.g9ed025fb.el9.noarch 37/170 2024-05-28T06:02:26.401 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : cephadm-2:19.0.0-3154.g9ed025fb.el9.noarch 38/170 2024-05-28T06:02:26.401 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : cephfs-top-2:19.0.0-3154.g9ed025fb.el9.noarch 39/170 2024-05-28T06:02:26.401 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/170 2024-05-28T06:02:26.401 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/170 2024-05-28T06:02:26.401 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/170 2024-05-28T06:02:26.401 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/170 2024-05-28T06:02:26.401 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/170 2024-05-28T06:02:26.401 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/170 2024-05-28T06:02:26.401 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/170 2024-05-28T06:02:26.401 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/170 2024-05-28T06:02:26.402 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/170 2024-05-28T06:02:26.402 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/170 2024-05-28T06:02:26.402 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/170 2024-05-28T06:02:26.402 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/170 2024-05-28T06:02:26.402 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/170 2024-05-28T06:02:26.402 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/170 2024-05-28T06:02:26.402 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/170 2024-05-28T06:02:26.402 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/170 2024-05-28T06:02:26.402 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/170 2024-05-28T06:02:26.402 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/170 2024-05-28T06:02:26.402 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/170 2024-05-28T06:02:26.403 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/170 2024-05-28T06:02:26.403 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/170 2024-05-28T06:02:26.403 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/170 2024-05-28T06:02:26.403 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/170 2024-05-28T06:02:26.403 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/170 2024-05-28T06:02:26.403 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/170 2024-05-28T06:02:26.403 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/170 2024-05-28T06:02:26.403 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/170 2024-05-28T06:02:26.403 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/170 2024-05-28T06:02:26.403 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/170 2024-05-28T06:02:26.403 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/170 2024-05-28T06:02:26.404 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/170 2024-05-28T06:02:26.404 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/170 2024-05-28T06:02:26.404 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/170 2024-05-28T06:02:26.404 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/170 2024-05-28T06:02:26.404 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/170 2024-05-28T06:02:26.405 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/170 2024-05-28T06:02:26.405 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/170 2024-05-28T06:02:26.405 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 77/170 2024-05-28T06:02:26.405 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/170 2024-05-28T06:02:26.405 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/170 2024-05-28T06:02:26.405 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/170 2024-05-28T06:02:26.405 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/170 2024-05-28T06:02:26.405 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/170 2024-05-28T06:02:26.405 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/170 2024-05-28T06:02:26.405 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/170 2024-05-28T06:02:26.405 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/170 2024-05-28T06:02:26.405 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/170 2024-05-28T06:02:26.405 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/170 2024-05-28T06:02:26.406 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/170 2024-05-28T06:02:26.406 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/170 2024-05-28T06:02:26.406 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/170 2024-05-28T06:02:26.406 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/170 2024-05-28T06:02:26.406 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/170 2024-05-28T06:02:26.406 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/170 2024-05-28T06:02:26.406 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/170 2024-05-28T06:02:26.406 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/170 2024-05-28T06:02:26.406 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/170 2024-05-28T06:02:26.406 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/170 2024-05-28T06:02:26.406 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/170 2024-05-28T06:02:26.406 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 99/170 2024-05-28T06:02:26.406 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/170 2024-05-28T06:02:26.407 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 101/170 2024-05-28T06:02:26.407 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/170 2024-05-28T06:02:26.407 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/170 2024-05-28T06:02:26.407 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/170 2024-05-28T06:02:26.407 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/170 2024-05-28T06:02:26.407 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/170 2024-05-28T06:02:26.407 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 107/170 2024-05-28T06:02:26.407 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/170 2024-05-28T06:02:26.407 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/170 2024-05-28T06:02:26.407 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/170 2024-05-28T06:02:26.407 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/170 2024-05-28T06:02:26.408 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/170 2024-05-28T06:02:26.408 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/170 2024-05-28T06:02:26.408 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/170 2024-05-28T06:02:26.408 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/170 2024-05-28T06:02:26.408 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/170 2024-05-28T06:02:26.408 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/170 2024-05-28T06:02:26.408 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 118/170 2024-05-28T06:02:26.408 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/170 2024-05-28T06:02:26.408 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/170 2024-05-28T06:02:26.408 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/170 2024-05-28T06:02:26.408 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/170 2024-05-28T06:02:26.408 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/170 2024-05-28T06:02:26.408 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/170 2024-05-28T06:02:26.409 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/170 2024-05-28T06:02:26.409 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/170 2024-05-28T06:02:26.409 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/170 2024-05-28T06:02:26.409 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/170 2024-05-28T06:02:26.409 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/170 2024-05-28T06:02:26.409 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/170 2024-05-28T06:02:26.409 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/170 2024-05-28T06:02:26.409 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/170 2024-05-28T06:02:26.409 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/170 2024-05-28T06:02:26.409 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/170 2024-05-28T06:02:26.409 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/170 2024-05-28T06:02:26.409 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/170 2024-05-28T06:02:26.409 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/170 2024-05-28T06:02:26.409 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/170 2024-05-28T06:02:26.410 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/170 2024-05-28T06:02:26.410 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/170 2024-05-28T06:02:26.410 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/170 2024-05-28T06:02:26.410 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/170 2024-05-28T06:02:26.410 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/170 2024-05-28T06:02:26.410 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/170 2024-05-28T06:02:26.410 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/170 2024-05-28T06:02:26.410 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/170 2024-05-28T06:02:26.410 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/170 2024-05-28T06:02:26.410 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/170 2024-05-28T06:02:26.410 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librados2-2:19.0.0-3154.g9ed025fb.el9.x86_64 149/170 2024-05-28T06:02:26.410 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/170 2024-05-28T06:02:26.410 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librbd1-2:19.0.0-3154.g9ed025fb.el9.x86_64 151/170 2024-05-28T06:02:26.411 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/170 2024-05-28T06:02:26.411 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : NetworkManager-1:1.47.90-1.el9.x86_64 153/170 2024-05-28T06:02:26.411 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/170 2024-05-28T06:02:26.411 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : NetworkManager-libnm-1:1.47.90-1.el9.x86_64 155/170 2024-05-28T06:02:26.411 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/170 2024-05-28T06:02:26.411 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : NetworkManager-team-1:1.47.90-1.el9.x86_64 157/170 2024-05-28T06:02:26.411 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/170 2024-05-28T06:02:26.411 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : NetworkManager-tui-1:1.47.90-1.el9.x86_64 159/170 2024-05-28T06:02:26.411 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/170 2024-05-28T06:02:26.411 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : elfutils-debuginfod-client-0.191-4.el9.x86_64 161/170 2024-05-28T06:02:26.411 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/170 2024-05-28T06:02:26.411 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : elfutils-libelf-0.191-4.el9.x86_64 163/170 2024-05-28T06:02:26.411 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 164/170 2024-05-28T06:02:26.411 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : elfutils-libs-0.191-4.el9.x86_64 165/170 2024-05-28T06:02:26.412 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 166/170 2024-05-28T06:02:26.412 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 167/170 2024-05-28T06:02:26.412 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 168/170 2024-05-28T06:02:26.412 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 169/170 2024-05-28T06:02:27.082 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3154.g9ed025fb.el9.x86_64 111/170 2024-05-28T06:02:27.082 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /sys 2024-05-28T06:02:27.082 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /proc 2024-05-28T06:02:27.082 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /mnt 2024-05-28T06:02:27.082 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /var/tmp 2024-05-28T06:02:27.082 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /home 2024-05-28T06:02:27.082 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /root 2024-05-28T06:02:27.082 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /tmp 2024-05-28T06:02:27.082 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-05-28T06:02:27.310 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-volume-2:19.0.0-3154.g9ed025fb.el9.noarch 112/170 2024-05-28T06:02:27.363 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-volume-2:19.0.0-3154.g9ed025fb.el9.noarch 112/170 2024-05-28T06:02:27.364 INFO:teuthology.orchestra.run.smithi123.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-28T06:02:27.364 INFO:teuthology.orchestra.run.smithi123.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-05-28T06:02:27.364 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-05-28T06:02:28.644 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 170/170 2024-05-28T06:02:28.644 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:02:28.644 INFO:teuthology.orchestra.run.smithi096.stdout:Upgraded: 2024-05-28T06:02:28.644 INFO:teuthology.orchestra.run.smithi096.stdout: NetworkManager-1:1.47.90-1.el9.x86_64 2024-05-28T06:02:28.644 INFO:teuthology.orchestra.run.smithi096.stdout: NetworkManager-libnm-1:1.47.90-1.el9.x86_64 2024-05-28T06:02:28.644 INFO:teuthology.orchestra.run.smithi096.stdout: NetworkManager-team-1:1.47.90-1.el9.x86_64 2024-05-28T06:02:28.645 INFO:teuthology.orchestra.run.smithi096.stdout: NetworkManager-tui-1:1.47.90-1.el9.x86_64 2024-05-28T06:02:28.645 INFO:teuthology.orchestra.run.smithi096.stdout: elfutils-debuginfod-client-0.191-4.el9.x86_64 2024-05-28T06:02:28.645 INFO:teuthology.orchestra.run.smithi096.stdout: elfutils-libelf-0.191-4.el9.x86_64 2024-05-28T06:02:28.645 INFO:teuthology.orchestra.run.smithi096.stdout: elfutils-libs-0.191-4.el9.x86_64 2024-05-28T06:02:28.645 INFO:teuthology.orchestra.run.smithi096.stdout: librados2-2:19.0.0-3154.g9ed025fb.el9.x86_64 2024-05-28T06:02:28.645 INFO:teuthology.orchestra.run.smithi096.stdout: librbd1-2:19.0.0-3154.g9ed025fb.el9.x86_64 2024-05-28T06:02:28.645 INFO:teuthology.orchestra.run.smithi096.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-05-28T06:02:28.645 INFO:teuthology.orchestra.run.smithi096.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-05-28T06:02:28.645 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-05-28T06:02:28.645 INFO:teuthology.orchestra.run.smithi096.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-05-28T06:02:28.645 INFO:teuthology.orchestra.run.smithi096.stdout: automake-1.16.2-8.el9.noarch 2024-05-28T06:02:28.645 INFO:teuthology.orchestra.run.smithi096.stdout: bison-3.7.4-5.el9.x86_64 2024-05-28T06:02:28.645 INFO:teuthology.orchestra.run.smithi096.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-05-28T06:02:28.645 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-2:19.0.0-3154.g9ed025fb.el9.x86_64 2024-05-28T06:02:28.646 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-base-2:19.0.0-3154.g9ed025fb.el9.x86_64 2024-05-28T06:02:28.646 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-common-2:19.0.0-3154.g9ed025fb.el9.x86_64 2024-05-28T06:02:28.646 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-fuse-2:19.0.0-3154.g9ed025fb.el9.x86_64 2024-05-28T06:02:28.646 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-grafana-dashboards-2:19.0.0-3154.g9ed025fb.el9.noarch 2024-05-28T06:02:28.646 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-immutable-object-cache-2:19.0.0-3154.g9ed025fb.el9.x86_64 2024-05-28T06:02:28.646 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mds-2:19.0.0-3154.g9ed025fb.el9.x86_64 2024-05-28T06:02:28.646 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-2:19.0.0-3154.g9ed025fb.el9.x86_64 2024-05-28T06:02:28.646 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-cephadm-2:19.0.0-3154.g9ed025fb.el9.noarch 2024-05-28T06:02:28.646 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-dashboard-2:19.0.0-3154.g9ed025fb.el9.noarch 2024-05-28T06:02:28.646 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3154.g9ed025fb.el9.noarch 2024-05-28T06:02:28.646 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-modules-core-2:19.0.0-3154.g9ed025fb.el9.noarch 2024-05-28T06:02:28.646 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-rook-2:19.0.0-3154.g9ed025fb.el9.noarch 2024-05-28T06:02:28.646 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mon-2:19.0.0-3154.g9ed025fb.el9.x86_64 2024-05-28T06:02:28.647 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-osd-2:19.0.0-3154.g9ed025fb.el9.x86_64 2024-05-28T06:02:28.647 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-prometheus-alerts-2:19.0.0-3154.g9ed025fb.el9.noarch 2024-05-28T06:02:28.647 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-radosgw-2:19.0.0-3154.g9ed025fb.el9.x86_64 2024-05-28T06:02:28.647 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-selinux-2:19.0.0-3154.g9ed025fb.el9.x86_64 2024-05-28T06:02:28.647 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-test-2:19.0.0-3154.g9ed025fb.el9.x86_64 2024-05-28T06:02:28.647 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-volume-2:19.0.0-3154.g9ed025fb.el9.noarch 2024-05-28T06:02:28.647 INFO:teuthology.orchestra.run.smithi096.stdout: cephadm-2:19.0.0-3154.g9ed025fb.el9.noarch 2024-05-28T06:02:28.647 INFO:teuthology.orchestra.run.smithi096.stdout: cephfs-mirror-2:19.0.0-3154.g9ed025fb.el9.x86_64 2024-05-28T06:02:28.647 INFO:teuthology.orchestra.run.smithi096.stdout: cephfs-shell-2:19.0.0-3154.g9ed025fb.el9.x86_64 2024-05-28T06:02:28.647 INFO:teuthology.orchestra.run.smithi096.stdout: cephfs-top-2:19.0.0-3154.g9ed025fb.el9.noarch 2024-05-28T06:02:28.647 INFO:teuthology.orchestra.run.smithi096.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-05-28T06:02:28.647 INFO:teuthology.orchestra.run.smithi096.stdout: flex-2.6.4-9.el9.x86_64 2024-05-28T06:02:28.647 INFO:teuthology.orchestra.run.smithi096.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-05-28T06:02:28.647 INFO:teuthology.orchestra.run.smithi096.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-05-28T06:02:28.648 INFO:teuthology.orchestra.run.smithi096.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-05-28T06:02:28.648 INFO:teuthology.orchestra.run.smithi096.stdout: fuse-2.9.9-15.el9.x86_64 2024-05-28T06:02:28.648 INFO:teuthology.orchestra.run.smithi096.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-05-28T06:02:28.648 INFO:teuthology.orchestra.run.smithi096.stdout: grpc-data-1.46.7-10.el9.noarch 2024-05-28T06:02:28.648 INFO:teuthology.orchestra.run.smithi096.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-05-28T06:02:28.648 INFO:teuthology.orchestra.run.smithi096.stdout: libarrow-9.0.0-11.el9.x86_64 2024-05-28T06:02:28.648 INFO:teuthology.orchestra.run.smithi096.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-05-28T06:02:28.648 INFO:teuthology.orchestra.run.smithi096.stdout: libcephfs-devel-2:19.0.0-3154.g9ed025fb.el9.x86_64 2024-05-28T06:02:28.648 INFO:teuthology.orchestra.run.smithi096.stdout: libcephfs2-2:19.0.0-3154.g9ed025fb.el9.x86_64 2024-05-28T06:02:28.648 INFO:teuthology.orchestra.run.smithi096.stdout: libcephsqlite-2:19.0.0-3154.g9ed025fb.el9.x86_64 2024-05-28T06:02:28.648 INFO:teuthology.orchestra.run.smithi096.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-05-28T06:02:28.648 INFO:teuthology.orchestra.run.smithi096.stdout: liboath-2.6.7-2.el9.x86_64 2024-05-28T06:02:28.648 INFO:teuthology.orchestra.run.smithi096.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-05-28T06:02:28.649 INFO:teuthology.orchestra.run.smithi096.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-05-28T06:02:28.649 INFO:teuthology.orchestra.run.smithi096.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-05-28T06:02:28.649 INFO:teuthology.orchestra.run.smithi096.stdout: librados-devel-2:19.0.0-3154.g9ed025fb.el9.x86_64 2024-05-28T06:02:28.649 INFO:teuthology.orchestra.run.smithi096.stdout: libradosstriper1-2:19.0.0-3154.g9ed025fb.el9.x86_64 2024-05-28T06:02:28.649 INFO:teuthology.orchestra.run.smithi096.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-05-28T06:02:28.649 INFO:teuthology.orchestra.run.smithi096.stdout: librgw2-2:19.0.0-3154.g9ed025fb.el9.x86_64 2024-05-28T06:02:28.649 INFO:teuthology.orchestra.run.smithi096.stdout: libtool-2.4.6-45.el9.x86_64 2024-05-28T06:02:28.649 INFO:teuthology.orchestra.run.smithi096.stdout: libunwind-1.6.2-1.el9.x86_64 2024-05-28T06:02:28.649 INFO:teuthology.orchestra.run.smithi096.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-05-28T06:02:28.649 INFO:teuthology.orchestra.run.smithi096.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-05-28T06:02:28.649 INFO:teuthology.orchestra.run.smithi096.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-05-28T06:02:28.649 INFO:teuthology.orchestra.run.smithi096.stdout: lua-5.4.4-4.el9.x86_64 2024-05-28T06:02:28.649 INFO:teuthology.orchestra.run.smithi096.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-05-28T06:02:28.650 INFO:teuthology.orchestra.run.smithi096.stdout: luarocks-3.9.2-5.el9.noarch 2024-05-28T06:02:28.650 INFO:teuthology.orchestra.run.smithi096.stdout: openblas-0.3.21-2.el9.x86_64 2024-05-28T06:02:28.650 INFO:teuthology.orchestra.run.smithi096.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-05-28T06:02:28.650 INFO:teuthology.orchestra.run.smithi096.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-05-28T06:02:28.650 INFO:teuthology.orchestra.run.smithi096.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-05-28T06:02:28.650 INFO:teuthology.orchestra.run.smithi096.stdout: postgresql-13.14-1.el9.x86_64 2024-05-28T06:02:28.650 INFO:teuthology.orchestra.run.smithi096.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-05-28T06:02:28.650 INFO:teuthology.orchestra.run.smithi096.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-05-28T06:02:28.650 INFO:teuthology.orchestra.run.smithi096.stdout: postgresql-server-13.14-1.el9.x86_64 2024-05-28T06:02:28.650 INFO:teuthology.orchestra.run.smithi096.stdout: protobuf-3.14.0-13.el9.x86_64 2024-05-28T06:02:28.650 INFO:teuthology.orchestra.run.smithi096.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-05-28T06:02:28.650 INFO:teuthology.orchestra.run.smithi096.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-05-28T06:02:28.651 INFO:teuthology.orchestra.run.smithi096.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-05-28T06:02:28.651 INFO:teuthology.orchestra.run.smithi096.stdout: python3-babel-2.9.1-2.el9.noarch 2024-05-28T06:02:28.651 INFO:teuthology.orchestra.run.smithi096.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-05-28T06:02:28.651 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-05-28T06:02:28.651 INFO:teuthology.orchestra.run.smithi096.stdout: python3-ceph-argparse-2:19.0.0-3154.g9ed025fb.el9.x86_64 2024-05-28T06:02:28.651 INFO:teuthology.orchestra.run.smithi096.stdout: python3-ceph-common-2:19.0.0-3154.g9ed025fb.el9.x86_64 2024-05-28T06:02:28.651 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cephfs-2:19.0.0-3154.g9ed025fb.el9.x86_64 2024-05-28T06:02:28.651 INFO:teuthology.orchestra.run.smithi096.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-05-28T06:02:28.651 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-05-28T06:02:28.651 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-05-28T06:02:28.651 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-05-28T06:02:28.651 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-05-28T06:02:28.651 INFO:teuthology.orchestra.run.smithi096.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-05-28T06:02:28.652 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-05-28T06:02:28.652 INFO:teuthology.orchestra.run.smithi096.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-05-28T06:02:28.652 INFO:teuthology.orchestra.run.smithi096.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-05-28T06:02:28.652 INFO:teuthology.orchestra.run.smithi096.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-05-28T06:02:28.652 INFO:teuthology.orchestra.run.smithi096.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-05-28T06:02:28.652 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-05-28T06:02:28.652 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-05-28T06:02:28.652 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-05-28T06:02:28.652 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-05-28T06:02:28.652 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-05-28T06:02:28.652 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-05-28T06:02:28.652 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-05-28T06:02:28.652 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-05-28T06:02:28.652 INFO:teuthology.orchestra.run.smithi096.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-05-28T06:02:28.653 INFO:teuthology.orchestra.run.smithi096.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-05-28T06:02:28.653 INFO:teuthology.orchestra.run.smithi096.stdout: python3-mako-1.1.4-6.el9.noarch 2024-05-28T06:02:28.653 INFO:teuthology.orchestra.run.smithi096.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-05-28T06:02:28.653 INFO:teuthology.orchestra.run.smithi096.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-05-28T06:02:28.653 INFO:teuthology.orchestra.run.smithi096.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-05-28T06:02:28.653 INFO:teuthology.orchestra.run.smithi096.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-05-28T06:02:28.653 INFO:teuthology.orchestra.run.smithi096.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-05-28T06:02:28.653 INFO:teuthology.orchestra.run.smithi096.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-05-28T06:02:28.653 INFO:teuthology.orchestra.run.smithi096.stdout: python3-packaging-20.9-5.el9.noarch 2024-05-28T06:02:28.653 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-05-28T06:02:28.653 INFO:teuthology.orchestra.run.smithi096.stdout: python3-ply-3.11-14.el9.noarch 2024-05-28T06:02:28.653 INFO:teuthology.orchestra.run.smithi096.stdout: python3-portend-3.1.0-2.el9.noarch 2024-05-28T06:02:28.653 INFO:teuthology.orchestra.run.smithi096.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-05-28T06:02:28.653 INFO:teuthology.orchestra.run.smithi096.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-05-28T06:02:28.654 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-05-28T06:02:28.654 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-05-28T06:02:28.654 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-05-28T06:02:28.654 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pycparser-2.20-6.el9.noarch 2024-05-28T06:02:28.654 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-05-28T06:02:28.654 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pytz-2021.1-5.el9.noarch 2024-05-28T06:02:28.654 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rados-2:19.0.0-3154.g9ed025fb.el9.x86_64 2024-05-28T06:02:28.654 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rbd-2:19.0.0-3154.g9ed025fb.el9.x86_64 2024-05-28T06:02:28.654 INFO:teuthology.orchestra.run.smithi096.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-05-28T06:02:28.654 INFO:teuthology.orchestra.run.smithi096.stdout: python3-requests-2.25.1-8.el9.noarch 2024-05-28T06:02:28.654 INFO:teuthology.orchestra.run.smithi096.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-05-28T06:02:28.654 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rgw-2:19.0.0-3154.g9ed025fb.el9.x86_64 2024-05-28T06:02:28.654 INFO:teuthology.orchestra.run.smithi096.stdout: python3-routes-2.5.1-5.el9.noarch 2024-05-28T06:02:28.654 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rsa-4.9-2.el9.noarch 2024-05-28T06:02:28.655 INFO:teuthology.orchestra.run.smithi096.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-05-28T06:02:28.655 INFO:teuthology.orchestra.run.smithi096.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-05-28T06:02:28.655 INFO:teuthology.orchestra.run.smithi096.stdout: python3-toml-0.10.2-6.el9.noarch 2024-05-28T06:02:28.655 INFO:teuthology.orchestra.run.smithi096.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-05-28T06:02:28.655 INFO:teuthology.orchestra.run.smithi096.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-05-28T06:02:28.655 INFO:teuthology.orchestra.run.smithi096.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-05-28T06:02:28.655 INFO:teuthology.orchestra.run.smithi096.stdout: python3-webob-1.8.7-6.el9.noarch 2024-05-28T06:02:28.655 INFO:teuthology.orchestra.run.smithi096.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-05-28T06:02:28.655 INFO:teuthology.orchestra.run.smithi096.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-05-28T06:02:28.655 INFO:teuthology.orchestra.run.smithi096.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-05-28T06:02:28.655 INFO:teuthology.orchestra.run.smithi096.stdout: qatlib-24.02.0-1.el9.x86_64 2024-05-28T06:02:28.655 INFO:teuthology.orchestra.run.smithi096.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-05-28T06:02:28.655 INFO:teuthology.orchestra.run.smithi096.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-05-28T06:02:28.655 INFO:teuthology.orchestra.run.smithi096.stdout: rbd-fuse-2:19.0.0-3154.g9ed025fb.el9.x86_64 2024-05-28T06:02:28.656 INFO:teuthology.orchestra.run.smithi096.stdout: rbd-mirror-2:19.0.0-3154.g9ed025fb.el9.x86_64 2024-05-28T06:02:28.656 INFO:teuthology.orchestra.run.smithi096.stdout: rbd-nbd-2:19.0.0-3154.g9ed025fb.el9.x86_64 2024-05-28T06:02:28.656 INFO:teuthology.orchestra.run.smithi096.stdout: re2-1:20211101-3.el9.x86_64 2024-05-28T06:02:28.656 INFO:teuthology.orchestra.run.smithi096.stdout: socat-1.7.4.1-5.el9.x86_64 2024-05-28T06:02:28.656 INFO:teuthology.orchestra.run.smithi096.stdout: thrift-0.15.0-2.el9.x86_64 2024-05-28T06:02:28.656 INFO:teuthology.orchestra.run.smithi096.stdout: uuid-1.6.2-55.el9.x86_64 2024-05-28T06:02:28.656 INFO:teuthology.orchestra.run.smithi096.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-05-28T06:02:28.656 INFO:teuthology.orchestra.run.smithi096.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-05-28T06:02:28.656 INFO:teuthology.orchestra.run.smithi096.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-05-28T06:02:28.657 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:02:28.657 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-05-28T06:02:28.979 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-osd-2:19.0.0-3154.g9ed025fb.el9.x86_64 113/170 2024-05-28T06:02:29.054 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-osd-2:19.0.0-3154.g9ed025fb.el9.x86_64 113/170 2024-05-28T06:02:29.054 INFO:teuthology.orchestra.run.smithi123.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-28T06:02:29.054 INFO:teuthology.orchestra.run.smithi123.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-05-28T06:02:29.054 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-28T06:02:29.054 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-28T06:02:29.054 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-05-28T06:02:29.321 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mds-2:19.0.0-3154.g9ed025fb.el9.x86_64 114/170 2024-05-28T06:02:29.352 DEBUG:teuthology.parallel:result is None 2024-05-28T06:02:29.389 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mds-2:19.0.0-3154.g9ed025fb.el9.x86_64 114/170 2024-05-28T06:02:29.389 INFO:teuthology.orchestra.run.smithi123.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-28T06:02:29.390 INFO:teuthology.orchestra.run.smithi123.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-05-28T06:02:29.390 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-28T06:02:29.390 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-28T06:02:29.390 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-05-28T06:02:29.879 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mon-2:19.0.0-3154.g9ed025fb.el9.x86_64 115/170 2024-05-28T06:02:29.948 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mon-2:19.0.0-3154.g9ed025fb.el9.x86_64 115/170 2024-05-28T06:02:29.948 INFO:teuthology.orchestra.run.smithi123.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-28T06:02:29.948 INFO:teuthology.orchestra.run.smithi123.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-05-28T06:02:29.949 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-28T06:02:29.949 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-28T06:02:29.949 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-05-28T06:02:30.096 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : automake-1.16.2-8.el9.noarch 116/170 2024-05-28T06:02:30.161 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-ply-3.11-14.el9.noarch 117/170 2024-05-28T06:02:30.236 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 118/170 2024-05-28T06:02:30.467 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 119/170 2024-05-28T06:02:30.525 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 120/170 2024-05-28T06:02:30.611 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 121/170 2024-05-28T06:02:30.721 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 122/170 2024-05-28T06:02:30.888 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 123/170 2024-05-28T06:02:30.960 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 124/170 2024-05-28T06:02:31.192 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 125/170 2024-05-28T06:02:31.224 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : fuse-2.9.9-15.el9.x86_64 126/170 2024-05-28T06:02:31.894 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: cephadm-2:19.0.0-3154.g9ed025fb.el9.noarch 127/170 2024-05-28T06:02:31.944 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : cephadm-2:19.0.0-3154.g9ed025fb.el9.noarch 127/170 2024-05-28T06:02:32.003 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3154.g9ed025fb.e 128/170 2024-05-28T06:02:32.121 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3154.g9ed025fb. 129/170 2024-05-28T06:02:32.177 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3154.g9ed025fb.el9.noa 130/170 2024-05-28T06:02:33.467 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3154.g9ed025fb.el9.noa 130/170 2024-05-28T06:02:33.509 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3154.g9ed025fb.el9.n 131/170 2024-05-28T06:02:34.608 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3154.g9ed025fb.el9.n 131/170 2024-05-28T06:02:34.642 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3154.g9ed 132/170 2024-05-28T06:02:34.790 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3154.g9ed 132/170 2024-05-28T06:02:34.966 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3154.g9ed025fb.el 133/170 2024-05-28T06:02:34.994 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mgr-2:19.0.0-3154.g9ed025fb.el9.x86_64 134/170 2024-05-28T06:02:35.069 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3154.g9ed025fb.el9.x86_64 134/170 2024-05-28T06:02:35.069 INFO:teuthology.orchestra.run.smithi123.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-28T06:02:35.069 INFO:teuthology.orchestra.run.smithi123.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-05-28T06:02:35.069 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-28T06:02:35.070 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-28T06:02:35.070 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-05-28T06:02:35.128 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mgr-rook-2:19.0.0-3154.g9ed025fb.el9.noarch 135/170 2024-05-28T06:02:35.165 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3154.g9ed025fb.el9.noarch 135/170 2024-05-28T06:02:35.278 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-2:19.0.0-3154.g9ed025fb.el9.x86_64 136/170 2024-05-28T06:02:35.347 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-fuse-2:19.0.0-3154.g9ed025fb.el9.x86_64 137/170 2024-05-28T06:02:36.339 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libtool-2.4.6-45.el9.x86_64 138/170 2024-05-28T06:02:36.372 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-radosgw-2:19.0.0-3154.g9ed025fb.el9.x86_64 139/170 2024-05-28T06:02:36.449 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3154.g9ed025fb.el9.x86_64 139/170 2024-05-28T06:02:36.449 INFO:teuthology.orchestra.run.smithi123.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-28T06:02:36.449 INFO:teuthology.orchestra.run.smithi123.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-05-28T06:02:36.449 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-28T06:02:36.449 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-28T06:02:36.449 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-05-28T06:02:36.498 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3154.g9ed02 140/170 2024-05-28T06:02:36.581 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3154.g9ed02 140/170 2024-05-28T06:02:36.581 INFO:teuthology.orchestra.run.smithi123.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-28T06:02:36.581 INFO:teuthology.orchestra.run.smithi123.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-05-28T06:02:36.581 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-05-28T06:02:36.657 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : cephfs-mirror-2:19.0.0-3154.g9ed025fb.el9.x86_64 141/170 2024-05-28T06:02:36.758 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3154.g9ed025fb.el9.x86_64 141/170 2024-05-28T06:02:36.759 INFO:teuthology.orchestra.run.smithi123.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-28T06:02:36.759 INFO:teuthology.orchestra.run.smithi123.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-05-28T06:02:36.759 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-28T06:02:36.759 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-28T06:02:36.759 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-05-28T06:02:37.140 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : rbd-mirror-2:19.0.0-3154.g9ed025fb.el9.x86_64 142/170 2024-05-28T06:02:37.223 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3154.g9ed025fb.el9.x86_64 142/170 2024-05-28T06:02:37.223 INFO:teuthology.orchestra.run.smithi123.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-28T06:02:37.224 INFO:teuthology.orchestra.run.smithi123.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-05-28T06:02:37.224 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-28T06:02:37.224 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-28T06:02:37.224 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-05-28T06:02:42.291 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-test-2:19.0.0-3154.g9ed025fb.el9.x86_64 143/170 2024-05-28T06:02:42.346 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : rbd-fuse-2:19.0.0-3154.g9ed025fb.el9.x86_64 144/170 2024-05-28T06:02:42.418 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : rbd-nbd-2:19.0.0-3154.g9ed025fb.el9.x86_64 145/170 2024-05-28T06:02:42.455 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 146/170 2024-05-28T06:02:42.493 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 147/170 2024-05-28T06:02:42.787 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : cephfs-shell-2:19.0.0-3154.g9ed025fb.el9.x86_64 148/170 2024-05-28T06:02:42.834 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 149/170 2024-05-28T06:02:42.866 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libcephfs-devel-2:19.0.0-3154.g9ed025fb.el9.x86_ 150/170 2024-05-28T06:02:42.898 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : elfutils-debuginfod-client-0.191-4.el9.x86_64 151/170 2024-05-28T06:02:42.945 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : NetworkManager-team-1:1.47.90-1.el9.x86_64 152/170 2024-05-28T06:02:42.983 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : NetworkManager-tui-1:1.47.90-1.el9.x86_64 153/170 2024-05-28T06:02:43.886 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 154/170 2024-05-28T06:02:43.937 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 154/170 2024-05-28T06:02:43.993 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 154/170 2024-05-28T06:02:45.309 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : cephfs-top-2:19.0.0-3154.g9ed025fb.el9.noarch 155/170 2024-05-28T06:02:45.428 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 156/170 2024-05-28T06:02:45.483 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 157/170 2024-05-28T06:02:45.620 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : flex-2.6.4-9.el9.x86_64 158/170 2024-05-28T06:02:45.662 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : bison-3.7.4-5.el9.x86_64 159/170 2024-05-28T06:02:45.695 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/170 2024-05-28T06:02:45.730 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : elfutils-debuginfod-client-0.190-2.el9.x86_64 161/170 2024-05-28T06:02:45.770 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : elfutils-libs-0.190-2.el9.x86_64 162/170 2024-05-28T06:02:45.770 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 163/170 2024-05-28T06:02:45.902 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 163/170 2024-05-28T06:02:45.950 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 164/170 2024-05-28T06:02:45.951 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 165/170 2024-05-28T06:02:46.087 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 165/170 2024-05-28T06:02:46.126 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 166/170 2024-05-28T06:02:46.159 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-05-28T06:02:46.160 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-05-28T06:02:46.497 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-05-28T06:02:46.497 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 168/170 2024-05-28T06:02:46.641 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 168/170 2024-05-28T06:02:46.687 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 169/170 2024-05-28T06:02:46.806 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : elfutils-libelf-0.190-2.el9.x86_64 170/170 2024-05-28T06:02:47.080 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: elfutils-libelf-0.190-2.el9.x86_64 170/170 2024-05-28T06:02:47.080 INFO:teuthology.orchestra.run.smithi123.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-05-28T06:02:47.080 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-05-28T06:02:52.676 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-2:19.0.0-3154.g9ed025fb.el9.x86_64 1/170 2024-05-28T06:02:52.676 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-base-2:19.0.0-3154.g9ed025fb.el9.x86_64 2/170 2024-05-28T06:02:52.677 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-common-2:19.0.0-3154.g9ed025fb.el9.x86_64 3/170 2024-05-28T06:02:52.677 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-fuse-2:19.0.0-3154.g9ed025fb.el9.x86_64 4/170 2024-05-28T06:02:52.677 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3154.g9ed02 5/170 2024-05-28T06:02:52.677 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mds-2:19.0.0-3154.g9ed025fb.el9.x86_64 6/170 2024-05-28T06:02:52.677 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mgr-2:19.0.0-3154.g9ed025fb.el9.x86_64 7/170 2024-05-28T06:02:52.677 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mon-2:19.0.0-3154.g9ed025fb.el9.x86_64 8/170 2024-05-28T06:02:52.677 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-osd-2:19.0.0-3154.g9ed025fb.el9.x86_64 9/170 2024-05-28T06:02:52.677 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-radosgw-2:19.0.0-3154.g9ed025fb.el9.x86_64 10/170 2024-05-28T06:02:52.677 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-selinux-2:19.0.0-3154.g9ed025fb.el9.x86_64 11/170 2024-05-28T06:02:52.677 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-test-2:19.0.0-3154.g9ed025fb.el9.x86_64 12/170 2024-05-28T06:02:52.677 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : cephfs-mirror-2:19.0.0-3154.g9ed025fb.el9.x86_64 13/170 2024-05-28T06:02:52.678 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : cephfs-shell-2:19.0.0-3154.g9ed025fb.el9.x86_64 14/170 2024-05-28T06:02:52.678 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libcephfs-devel-2:19.0.0-3154.g9ed025fb.el9.x86_ 15/170 2024-05-28T06:02:52.678 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libcephfs2-2:19.0.0-3154.g9ed025fb.el9.x86_64 16/170 2024-05-28T06:02:52.678 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libcephsqlite-2:19.0.0-3154.g9ed025fb.el9.x86_64 17/170 2024-05-28T06:02:52.678 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librados-devel-2:19.0.0-3154.g9ed025fb.el9.x86_6 18/170 2024-05-28T06:02:52.678 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libradosstriper1-2:19.0.0-3154.g9ed025fb.el9.x86 19/170 2024-05-28T06:02:52.678 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librgw2-2:19.0.0-3154.g9ed025fb.el9.x86_64 20/170 2024-05-28T06:02:52.678 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3154.g9ed025fb.el 21/170 2024-05-28T06:02:52.678 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-ceph-common-2:19.0.0-3154.g9ed025fb.el9. 22/170 2024-05-28T06:02:52.678 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cephfs-2:19.0.0-3154.g9ed025fb.el9.x86_6 23/170 2024-05-28T06:02:52.678 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-rados-2:19.0.0-3154.g9ed025fb.el9.x86_64 24/170 2024-05-28T06:02:52.678 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-rbd-2:19.0.0-3154.g9ed025fb.el9.x86_64 25/170 2024-05-28T06:02:52.679 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-rgw-2:19.0.0-3154.g9ed025fb.el9.x86_64 26/170 2024-05-28T06:02:52.679 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : rbd-fuse-2:19.0.0-3154.g9ed025fb.el9.x86_64 27/170 2024-05-28T06:02:52.679 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : rbd-mirror-2:19.0.0-3154.g9ed025fb.el9.x86_64 28/170 2024-05-28T06:02:52.679 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : rbd-nbd-2:19.0.0-3154.g9ed025fb.el9.x86_64 29/170 2024-05-28T06:02:52.679 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3154.g9ed025fb. 30/170 2024-05-28T06:02:52.679 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3154.g9ed025fb.el9.noa 31/170 2024-05-28T06:02:52.679 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3154.g9ed025fb.el9.n 32/170 2024-05-28T06:02:52.679 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3154.g9ed 33/170 2024-05-28T06:02:52.679 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3154.g9ed025fb.el 34/170 2024-05-28T06:02:52.679 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3154.g9ed025fb.el9.noarch 35/170 2024-05-28T06:02:52.679 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3154.g9ed025fb.e 36/170 2024-05-28T06:02:52.680 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-volume-2:19.0.0-3154.g9ed025fb.el9.noarch 37/170 2024-05-28T06:02:52.680 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : cephadm-2:19.0.0-3154.g9ed025fb.el9.noarch 38/170 2024-05-28T06:02:52.680 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : cephfs-top-2:19.0.0-3154.g9ed025fb.el9.noarch 39/170 2024-05-28T06:02:52.680 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/170 2024-05-28T06:02:52.680 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/170 2024-05-28T06:02:52.680 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/170 2024-05-28T06:02:52.680 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/170 2024-05-28T06:02:52.680 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/170 2024-05-28T06:02:52.680 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/170 2024-05-28T06:02:52.680 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/170 2024-05-28T06:02:52.680 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/170 2024-05-28T06:02:52.680 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/170 2024-05-28T06:02:52.680 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/170 2024-05-28T06:02:52.680 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/170 2024-05-28T06:02:52.681 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/170 2024-05-28T06:02:52.681 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/170 2024-05-28T06:02:52.681 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/170 2024-05-28T06:02:52.681 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/170 2024-05-28T06:02:52.681 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/170 2024-05-28T06:02:52.681 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/170 2024-05-28T06:02:52.681 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/170 2024-05-28T06:02:52.681 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/170 2024-05-28T06:02:52.681 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/170 2024-05-28T06:02:52.681 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/170 2024-05-28T06:02:52.681 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/170 2024-05-28T06:02:52.681 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/170 2024-05-28T06:02:52.681 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/170 2024-05-28T06:02:52.682 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/170 2024-05-28T06:02:52.682 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/170 2024-05-28T06:02:52.682 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/170 2024-05-28T06:02:52.682 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/170 2024-05-28T06:02:52.682 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/170 2024-05-28T06:02:52.682 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/170 2024-05-28T06:02:52.682 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/170 2024-05-28T06:02:52.682 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/170 2024-05-28T06:02:52.682 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/170 2024-05-28T06:02:52.682 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/170 2024-05-28T06:02:52.682 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/170 2024-05-28T06:02:52.683 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/170 2024-05-28T06:02:52.683 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/170 2024-05-28T06:02:52.683 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 77/170 2024-05-28T06:02:52.683 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/170 2024-05-28T06:02:52.683 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/170 2024-05-28T06:02:52.683 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/170 2024-05-28T06:02:52.683 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/170 2024-05-28T06:02:52.683 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/170 2024-05-28T06:02:52.683 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/170 2024-05-28T06:02:52.683 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/170 2024-05-28T06:02:52.683 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/170 2024-05-28T06:02:52.684 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/170 2024-05-28T06:02:52.684 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/170 2024-05-28T06:02:52.684 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/170 2024-05-28T06:02:52.684 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/170 2024-05-28T06:02:52.684 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/170 2024-05-28T06:02:52.684 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/170 2024-05-28T06:02:52.684 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/170 2024-05-28T06:02:52.684 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/170 2024-05-28T06:02:52.684 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/170 2024-05-28T06:02:52.684 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/170 2024-05-28T06:02:52.684 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/170 2024-05-28T06:02:52.684 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/170 2024-05-28T06:02:52.684 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/170 2024-05-28T06:02:52.685 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 99/170 2024-05-28T06:02:52.685 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/170 2024-05-28T06:02:52.685 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 101/170 2024-05-28T06:02:52.685 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/170 2024-05-28T06:02:52.685 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/170 2024-05-28T06:02:52.685 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/170 2024-05-28T06:02:52.685 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/170 2024-05-28T06:02:52.685 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/170 2024-05-28T06:02:52.685 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 107/170 2024-05-28T06:02:52.685 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/170 2024-05-28T06:02:52.685 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/170 2024-05-28T06:02:52.685 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/170 2024-05-28T06:02:52.685 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/170 2024-05-28T06:02:52.686 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/170 2024-05-28T06:02:52.686 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/170 2024-05-28T06:02:52.686 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/170 2024-05-28T06:02:52.686 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/170 2024-05-28T06:02:52.686 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/170 2024-05-28T06:02:52.686 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/170 2024-05-28T06:02:52.686 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 118/170 2024-05-28T06:02:52.686 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/170 2024-05-28T06:02:52.686 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/170 2024-05-28T06:02:52.686 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/170 2024-05-28T06:02:52.686 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/170 2024-05-28T06:02:52.686 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/170 2024-05-28T06:02:52.686 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/170 2024-05-28T06:02:52.687 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/170 2024-05-28T06:02:52.687 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/170 2024-05-28T06:02:52.687 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/170 2024-05-28T06:02:52.687 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/170 2024-05-28T06:02:52.687 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/170 2024-05-28T06:02:52.687 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/170 2024-05-28T06:02:52.687 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/170 2024-05-28T06:02:52.687 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/170 2024-05-28T06:02:52.687 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/170 2024-05-28T06:02:52.687 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/170 2024-05-28T06:02:52.687 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/170 2024-05-28T06:02:52.687 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/170 2024-05-28T06:02:52.688 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/170 2024-05-28T06:02:52.688 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/170 2024-05-28T06:02:52.688 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/170 2024-05-28T06:02:52.688 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/170 2024-05-28T06:02:52.688 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/170 2024-05-28T06:02:52.688 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/170 2024-05-28T06:02:52.688 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/170 2024-05-28T06:02:52.688 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/170 2024-05-28T06:02:52.688 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/170 2024-05-28T06:02:52.688 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/170 2024-05-28T06:02:52.688 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/170 2024-05-28T06:02:52.689 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/170 2024-05-28T06:02:52.689 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librados2-2:19.0.0-3154.g9ed025fb.el9.x86_64 149/170 2024-05-28T06:02:52.689 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/170 2024-05-28T06:02:52.689 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librbd1-2:19.0.0-3154.g9ed025fb.el9.x86_64 151/170 2024-05-28T06:02:52.689 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/170 2024-05-28T06:02:52.689 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : NetworkManager-1:1.47.90-1.el9.x86_64 153/170 2024-05-28T06:02:52.689 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/170 2024-05-28T06:02:52.689 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : NetworkManager-libnm-1:1.47.90-1.el9.x86_64 155/170 2024-05-28T06:02:52.689 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/170 2024-05-28T06:02:52.689 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : NetworkManager-team-1:1.47.90-1.el9.x86_64 157/170 2024-05-28T06:02:52.689 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/170 2024-05-28T06:02:52.689 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : NetworkManager-tui-1:1.47.90-1.el9.x86_64 159/170 2024-05-28T06:02:52.689 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/170 2024-05-28T06:02:52.690 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : elfutils-debuginfod-client-0.191-4.el9.x86_64 161/170 2024-05-28T06:02:52.690 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/170 2024-05-28T06:02:52.690 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : elfutils-libelf-0.191-4.el9.x86_64 163/170 2024-05-28T06:02:52.690 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 164/170 2024-05-28T06:02:52.690 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : elfutils-libs-0.191-4.el9.x86_64 165/170 2024-05-28T06:02:52.690 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 166/170 2024-05-28T06:02:52.690 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 167/170 2024-05-28T06:02:52.690 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 168/170 2024-05-28T06:02:52.690 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 169/170 2024-05-28T06:02:54.895 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 170/170 2024-05-28T06:02:54.895 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-05-28T06:02:54.895 INFO:teuthology.orchestra.run.smithi123.stdout:Upgraded: 2024-05-28T06:02:54.895 INFO:teuthology.orchestra.run.smithi123.stdout: NetworkManager-1:1.47.90-1.el9.x86_64 2024-05-28T06:02:54.896 INFO:teuthology.orchestra.run.smithi123.stdout: NetworkManager-libnm-1:1.47.90-1.el9.x86_64 2024-05-28T06:02:54.896 INFO:teuthology.orchestra.run.smithi123.stdout: NetworkManager-team-1:1.47.90-1.el9.x86_64 2024-05-28T06:02:54.896 INFO:teuthology.orchestra.run.smithi123.stdout: NetworkManager-tui-1:1.47.90-1.el9.x86_64 2024-05-28T06:02:54.896 INFO:teuthology.orchestra.run.smithi123.stdout: elfutils-debuginfod-client-0.191-4.el9.x86_64 2024-05-28T06:02:54.896 INFO:teuthology.orchestra.run.smithi123.stdout: elfutils-libelf-0.191-4.el9.x86_64 2024-05-28T06:02:54.896 INFO:teuthology.orchestra.run.smithi123.stdout: elfutils-libs-0.191-4.el9.x86_64 2024-05-28T06:02:54.896 INFO:teuthology.orchestra.run.smithi123.stdout: librados2-2:19.0.0-3154.g9ed025fb.el9.x86_64 2024-05-28T06:02:54.896 INFO:teuthology.orchestra.run.smithi123.stdout: librbd1-2:19.0.0-3154.g9ed025fb.el9.x86_64 2024-05-28T06:02:54.896 INFO:teuthology.orchestra.run.smithi123.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-05-28T06:02:54.896 INFO:teuthology.orchestra.run.smithi123.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-05-28T06:02:54.896 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-05-28T06:02:54.897 INFO:teuthology.orchestra.run.smithi123.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-05-28T06:02:54.897 INFO:teuthology.orchestra.run.smithi123.stdout: automake-1.16.2-8.el9.noarch 2024-05-28T06:02:54.897 INFO:teuthology.orchestra.run.smithi123.stdout: bison-3.7.4-5.el9.x86_64 2024-05-28T06:02:54.897 INFO:teuthology.orchestra.run.smithi123.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-05-28T06:02:54.897 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-2:19.0.0-3154.g9ed025fb.el9.x86_64 2024-05-28T06:02:54.897 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-base-2:19.0.0-3154.g9ed025fb.el9.x86_64 2024-05-28T06:02:54.897 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-common-2:19.0.0-3154.g9ed025fb.el9.x86_64 2024-05-28T06:02:54.897 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-fuse-2:19.0.0-3154.g9ed025fb.el9.x86_64 2024-05-28T06:02:54.897 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-grafana-dashboards-2:19.0.0-3154.g9ed025fb.el9.noarch 2024-05-28T06:02:54.897 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-immutable-object-cache-2:19.0.0-3154.g9ed025fb.el9.x86_64 2024-05-28T06:02:54.898 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mds-2:19.0.0-3154.g9ed025fb.el9.x86_64 2024-05-28T06:02:54.898 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-2:19.0.0-3154.g9ed025fb.el9.x86_64 2024-05-28T06:02:54.898 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-cephadm-2:19.0.0-3154.g9ed025fb.el9.noarch 2024-05-28T06:02:54.898 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-dashboard-2:19.0.0-3154.g9ed025fb.el9.noarch 2024-05-28T06:02:54.898 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3154.g9ed025fb.el9.noarch 2024-05-28T06:02:54.898 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-modules-core-2:19.0.0-3154.g9ed025fb.el9.noarch 2024-05-28T06:02:54.899 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-rook-2:19.0.0-3154.g9ed025fb.el9.noarch 2024-05-28T06:02:54.899 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mon-2:19.0.0-3154.g9ed025fb.el9.x86_64 2024-05-28T06:02:54.899 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-osd-2:19.0.0-3154.g9ed025fb.el9.x86_64 2024-05-28T06:02:54.899 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-prometheus-alerts-2:19.0.0-3154.g9ed025fb.el9.noarch 2024-05-28T06:02:54.899 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-radosgw-2:19.0.0-3154.g9ed025fb.el9.x86_64 2024-05-28T06:02:54.899 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-selinux-2:19.0.0-3154.g9ed025fb.el9.x86_64 2024-05-28T06:02:54.900 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-test-2:19.0.0-3154.g9ed025fb.el9.x86_64 2024-05-28T06:02:54.900 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-volume-2:19.0.0-3154.g9ed025fb.el9.noarch 2024-05-28T06:02:54.900 INFO:teuthology.orchestra.run.smithi123.stdout: cephadm-2:19.0.0-3154.g9ed025fb.el9.noarch 2024-05-28T06:02:54.900 INFO:teuthology.orchestra.run.smithi123.stdout: cephfs-mirror-2:19.0.0-3154.g9ed025fb.el9.x86_64 2024-05-28T06:02:54.900 INFO:teuthology.orchestra.run.smithi123.stdout: cephfs-shell-2:19.0.0-3154.g9ed025fb.el9.x86_64 2024-05-28T06:02:54.900 INFO:teuthology.orchestra.run.smithi123.stdout: cephfs-top-2:19.0.0-3154.g9ed025fb.el9.noarch 2024-05-28T06:02:54.900 INFO:teuthology.orchestra.run.smithi123.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-05-28T06:02:54.900 INFO:teuthology.orchestra.run.smithi123.stdout: flex-2.6.4-9.el9.x86_64 2024-05-28T06:02:54.901 INFO:teuthology.orchestra.run.smithi123.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-05-28T06:02:54.901 INFO:teuthology.orchestra.run.smithi123.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-05-28T06:02:54.901 INFO:teuthology.orchestra.run.smithi123.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-05-28T06:02:54.901 INFO:teuthology.orchestra.run.smithi123.stdout: fuse-2.9.9-15.el9.x86_64 2024-05-28T06:02:54.901 INFO:teuthology.orchestra.run.smithi123.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-05-28T06:02:54.901 INFO:teuthology.orchestra.run.smithi123.stdout: grpc-data-1.46.7-10.el9.noarch 2024-05-28T06:02:54.902 INFO:teuthology.orchestra.run.smithi123.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-05-28T06:02:54.902 INFO:teuthology.orchestra.run.smithi123.stdout: libarrow-9.0.0-11.el9.x86_64 2024-05-28T06:02:54.902 INFO:teuthology.orchestra.run.smithi123.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-05-28T06:02:54.902 INFO:teuthology.orchestra.run.smithi123.stdout: libcephfs-devel-2:19.0.0-3154.g9ed025fb.el9.x86_64 2024-05-28T06:02:54.902 INFO:teuthology.orchestra.run.smithi123.stdout: libcephfs2-2:19.0.0-3154.g9ed025fb.el9.x86_64 2024-05-28T06:02:54.902 INFO:teuthology.orchestra.run.smithi123.stdout: libcephsqlite-2:19.0.0-3154.g9ed025fb.el9.x86_64 2024-05-28T06:02:54.902 INFO:teuthology.orchestra.run.smithi123.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-05-28T06:02:54.903 INFO:teuthology.orchestra.run.smithi123.stdout: liboath-2.6.7-2.el9.x86_64 2024-05-28T06:02:54.903 INFO:teuthology.orchestra.run.smithi123.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-05-28T06:02:54.903 INFO:teuthology.orchestra.run.smithi123.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-05-28T06:02:54.903 INFO:teuthology.orchestra.run.smithi123.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-05-28T06:02:54.903 INFO:teuthology.orchestra.run.smithi123.stdout: librados-devel-2:19.0.0-3154.g9ed025fb.el9.x86_64 2024-05-28T06:02:54.903 INFO:teuthology.orchestra.run.smithi123.stdout: libradosstriper1-2:19.0.0-3154.g9ed025fb.el9.x86_64 2024-05-28T06:02:54.903 INFO:teuthology.orchestra.run.smithi123.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-05-28T06:02:54.903 INFO:teuthology.orchestra.run.smithi123.stdout: librgw2-2:19.0.0-3154.g9ed025fb.el9.x86_64 2024-05-28T06:02:54.903 INFO:teuthology.orchestra.run.smithi123.stdout: libtool-2.4.6-45.el9.x86_64 2024-05-28T06:02:54.903 INFO:teuthology.orchestra.run.smithi123.stdout: libunwind-1.6.2-1.el9.x86_64 2024-05-28T06:02:54.904 INFO:teuthology.orchestra.run.smithi123.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-05-28T06:02:54.904 INFO:teuthology.orchestra.run.smithi123.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-05-28T06:02:54.904 INFO:teuthology.orchestra.run.smithi123.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-05-28T06:02:54.904 INFO:teuthology.orchestra.run.smithi123.stdout: lua-5.4.4-4.el9.x86_64 2024-05-28T06:02:54.904 INFO:teuthology.orchestra.run.smithi123.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-05-28T06:02:54.904 INFO:teuthology.orchestra.run.smithi123.stdout: luarocks-3.9.2-5.el9.noarch 2024-05-28T06:02:54.904 INFO:teuthology.orchestra.run.smithi123.stdout: openblas-0.3.21-2.el9.x86_64 2024-05-28T06:02:54.904 INFO:teuthology.orchestra.run.smithi123.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-05-28T06:02:54.904 INFO:teuthology.orchestra.run.smithi123.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-05-28T06:02:54.904 INFO:teuthology.orchestra.run.smithi123.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-05-28T06:02:54.904 INFO:teuthology.orchestra.run.smithi123.stdout: postgresql-13.14-1.el9.x86_64 2024-05-28T06:02:54.905 INFO:teuthology.orchestra.run.smithi123.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-05-28T06:02:54.905 INFO:teuthology.orchestra.run.smithi123.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-05-28T06:02:54.905 INFO:teuthology.orchestra.run.smithi123.stdout: postgresql-server-13.14-1.el9.x86_64 2024-05-28T06:02:54.905 INFO:teuthology.orchestra.run.smithi123.stdout: protobuf-3.14.0-13.el9.x86_64 2024-05-28T06:02:54.905 INFO:teuthology.orchestra.run.smithi123.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-05-28T06:02:54.905 INFO:teuthology.orchestra.run.smithi123.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-05-28T06:02:54.905 INFO:teuthology.orchestra.run.smithi123.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-05-28T06:02:54.905 INFO:teuthology.orchestra.run.smithi123.stdout: python3-babel-2.9.1-2.el9.noarch 2024-05-28T06:02:54.905 INFO:teuthology.orchestra.run.smithi123.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-05-28T06:02:54.905 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-05-28T06:02:54.905 INFO:teuthology.orchestra.run.smithi123.stdout: python3-ceph-argparse-2:19.0.0-3154.g9ed025fb.el9.x86_64 2024-05-28T06:02:54.905 INFO:teuthology.orchestra.run.smithi123.stdout: python3-ceph-common-2:19.0.0-3154.g9ed025fb.el9.x86_64 2024-05-28T06:02:54.906 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cephfs-2:19.0.0-3154.g9ed025fb.el9.x86_64 2024-05-28T06:02:54.906 INFO:teuthology.orchestra.run.smithi123.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-05-28T06:02:54.906 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-05-28T06:02:54.906 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-05-28T06:02:54.906 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-05-28T06:02:54.906 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-05-28T06:02:54.906 INFO:teuthology.orchestra.run.smithi123.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-05-28T06:02:54.906 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-05-28T06:02:54.906 INFO:teuthology.orchestra.run.smithi123.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-05-28T06:02:54.906 INFO:teuthology.orchestra.run.smithi123.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-05-28T06:02:54.907 INFO:teuthology.orchestra.run.smithi123.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-05-28T06:02:54.907 INFO:teuthology.orchestra.run.smithi123.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-05-28T06:02:54.907 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-05-28T06:02:54.907 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-05-28T06:02:54.907 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-05-28T06:02:54.907 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-05-28T06:02:54.907 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-05-28T06:02:54.907 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-05-28T06:02:54.907 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-05-28T06:02:54.908 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-05-28T06:02:54.908 INFO:teuthology.orchestra.run.smithi123.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-05-28T06:02:54.908 INFO:teuthology.orchestra.run.smithi123.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-05-28T06:02:54.908 INFO:teuthology.orchestra.run.smithi123.stdout: python3-mako-1.1.4-6.el9.noarch 2024-05-28T06:02:54.908 INFO:teuthology.orchestra.run.smithi123.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-05-28T06:02:54.908 INFO:teuthology.orchestra.run.smithi123.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-05-28T06:02:54.908 INFO:teuthology.orchestra.run.smithi123.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-05-28T06:02:54.908 INFO:teuthology.orchestra.run.smithi123.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-05-28T06:02:54.909 INFO:teuthology.orchestra.run.smithi123.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-05-28T06:02:54.909 INFO:teuthology.orchestra.run.smithi123.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-05-28T06:02:54.909 INFO:teuthology.orchestra.run.smithi123.stdout: python3-packaging-20.9-5.el9.noarch 2024-05-28T06:02:54.909 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-05-28T06:02:54.909 INFO:teuthology.orchestra.run.smithi123.stdout: python3-ply-3.11-14.el9.noarch 2024-05-28T06:02:54.909 INFO:teuthology.orchestra.run.smithi123.stdout: python3-portend-3.1.0-2.el9.noarch 2024-05-28T06:02:54.909 INFO:teuthology.orchestra.run.smithi123.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-05-28T06:02:54.909 INFO:teuthology.orchestra.run.smithi123.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-05-28T06:02:54.909 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-05-28T06:02:54.909 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-05-28T06:02:54.909 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-05-28T06:02:54.910 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pycparser-2.20-6.el9.noarch 2024-05-28T06:02:54.910 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-05-28T06:02:54.910 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pytz-2021.1-5.el9.noarch 2024-05-28T06:02:54.910 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rados-2:19.0.0-3154.g9ed025fb.el9.x86_64 2024-05-28T06:02:54.910 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rbd-2:19.0.0-3154.g9ed025fb.el9.x86_64 2024-05-28T06:02:54.911 INFO:teuthology.orchestra.run.smithi123.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-05-28T06:02:54.911 INFO:teuthology.orchestra.run.smithi123.stdout: python3-requests-2.25.1-8.el9.noarch 2024-05-28T06:02:54.911 INFO:teuthology.orchestra.run.smithi123.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-05-28T06:02:54.911 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rgw-2:19.0.0-3154.g9ed025fb.el9.x86_64 2024-05-28T06:02:54.912 INFO:teuthology.orchestra.run.smithi123.stdout: python3-routes-2.5.1-5.el9.noarch 2024-05-28T06:02:54.912 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rsa-4.9-2.el9.noarch 2024-05-28T06:02:54.912 INFO:teuthology.orchestra.run.smithi123.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-05-28T06:02:54.912 INFO:teuthology.orchestra.run.smithi123.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-05-28T06:02:54.912 INFO:teuthology.orchestra.run.smithi123.stdout: python3-toml-0.10.2-6.el9.noarch 2024-05-28T06:02:54.912 INFO:teuthology.orchestra.run.smithi123.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-05-28T06:02:54.912 INFO:teuthology.orchestra.run.smithi123.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-05-28T06:02:54.912 INFO:teuthology.orchestra.run.smithi123.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-05-28T06:02:54.912 INFO:teuthology.orchestra.run.smithi123.stdout: python3-webob-1.8.7-6.el9.noarch 2024-05-28T06:02:54.912 INFO:teuthology.orchestra.run.smithi123.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-05-28T06:02:54.913 INFO:teuthology.orchestra.run.smithi123.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-05-28T06:02:54.913 INFO:teuthology.orchestra.run.smithi123.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-05-28T06:02:54.913 INFO:teuthology.orchestra.run.smithi123.stdout: qatlib-24.02.0-1.el9.x86_64 2024-05-28T06:02:54.913 INFO:teuthology.orchestra.run.smithi123.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-05-28T06:02:54.913 INFO:teuthology.orchestra.run.smithi123.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-05-28T06:02:54.914 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-fuse-2:19.0.0-3154.g9ed025fb.el9.x86_64 2024-05-28T06:02:54.914 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-mirror-2:19.0.0-3154.g9ed025fb.el9.x86_64 2024-05-28T06:02:54.914 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-nbd-2:19.0.0-3154.g9ed025fb.el9.x86_64 2024-05-28T06:02:54.914 INFO:teuthology.orchestra.run.smithi123.stdout: re2-1:20211101-3.el9.x86_64 2024-05-28T06:02:54.914 INFO:teuthology.orchestra.run.smithi123.stdout: socat-1.7.4.1-5.el9.x86_64 2024-05-28T06:02:54.914 INFO:teuthology.orchestra.run.smithi123.stdout: thrift-0.15.0-2.el9.x86_64 2024-05-28T06:02:54.915 INFO:teuthology.orchestra.run.smithi123.stdout: uuid-1.6.2-55.el9.x86_64 2024-05-28T06:02:54.915 INFO:teuthology.orchestra.run.smithi123.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-05-28T06:02:54.915 INFO:teuthology.orchestra.run.smithi123.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-05-28T06:02:54.915 INFO:teuthology.orchestra.run.smithi123.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-05-28T06:02:54.915 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-05-28T06:02:54.915 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-05-28T06:02:55.597 DEBUG:teuthology.parallel:result is None 2024-05-28T06:02:55.597 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=9ed025fbf4406fb54980a0647506b15895b6d863 2024-05-28T06:02:55.822 DEBUG:teuthology.orchestra.run.smithi096:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-28T06:02:55.896 INFO:teuthology.orchestra.run.smithi096.stdout:19.0.0-3154.g9ed025fb.el9 2024-05-28T06:02:55.897 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3154.g9ed025fb.el9 2024-05-28T06:02:55.897 INFO:teuthology.task.install:The correct ceph version 19.0.0-3154.g9ed025fb is installed. 2024-05-28T06:02:55.900 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=9ed025fbf4406fb54980a0647506b15895b6d863 2024-05-28T06:02:56.126 DEBUG:teuthology.orchestra.run.smithi123:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-28T06:02:56.180 INFO:teuthology.orchestra.run.smithi123.stdout:19.0.0-3154.g9ed025fb.el9 2024-05-28T06:02:56.180 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3154.g9ed025fb.el9 2024-05-28T06:02:56.180 INFO:teuthology.task.install:The correct ceph version 19.0.0-3154.g9ed025fb is installed. 2024-05-28T06:02:56.182 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-05-28T06:02:56.182 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:02:56.182 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-28T06:02:56.268 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-05-28T06:02:56.268 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-28T06:02:56.330 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-05-28T06:02:56.330 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:02:56.330 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/usr/bin/daemon-helper 2024-05-28T06:02:56.417 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-28T06:02:56.539 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-05-28T06:02:56.539 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/usr/bin/daemon-helper 2024-05-28T06:02:56.607 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-28T06:02:56.719 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-05-28T06:02:56.719 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:02:56.719 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-28T06:02:56.799 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-28T06:02:56.927 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-05-28T06:02:56.927 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-28T06:02:57.004 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-28T06:02:57.109 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-05-28T06:02:57.109 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:02:57.109 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/usr/bin/stdin-killer 2024-05-28T06:02:57.193 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-28T06:02:57.277 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-05-28T06:02:57.277 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/usr/bin/stdin-killer 2024-05-28T06:02:57.347 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-28T06:02:57.455 INFO:teuthology.run_tasks:Running task ceph... 2024-05-28T06:02:57.554 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-05-28T06:02:57.554 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod 777 /var/log/ceph 2024-05-28T06:02:57.563 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 777 /var/log/ceph 2024-05-28T06:02:57.642 INFO:tasks.ceph:Disabling ceph logrotate... 2024-05-28T06:02:57.642 DEBUG:teuthology.orchestra.run.smithi096:> sudo rm -f -- /etc/logrotate.d/ceph 2024-05-28T06:02:57.649 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -f -- /etc/logrotate.d/ceph 2024-05-28T06:02:57.727 INFO:tasks.ceph:Creating extra log directories... 2024-05-28T06:02:57.728 DEBUG:teuthology.orchestra.run.smithi096:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-05-28T06:02:57.735 DEBUG:teuthology.orchestra.run.smithi123:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-05-28T06:02:57.821 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-05-28T06:02:57.821 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-05-28T06:02:57.821 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-05-28T06:02:57.822 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:02:57.822 DEBUG:teuthology.orchestra.run.smithi096:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-05-28T06:02:57.876 DEBUG:teuthology.orchestra.run.smithi096:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-05-28T06:02:58.092 DEBUG:teuthology.orchestra.run.smithi096:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-05-28T06:02:58.179 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-05-28T06:02:58.180 DEBUG:teuthology.orchestra.run.smithi123:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-05-28T06:02:58.219 DEBUG:teuthology.orchestra.run.smithi123:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-05-28T06:02:58.399 DEBUG:teuthology.orchestra.run.smithi123:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-05-28T06:02:58.468 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-05-28T06:02:58.468 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-05-28T06:02:58.468 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/rishabh-2024-05-28_05:40:25-fs:volumes-rishabh-clone-stats-testing-default-smithi/7729374', 'branch': 'rishabh-clone-stats', 'description': 'fs:volumes/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/volumes/{overrides test/clone-progress}}', 'email': None, 'first_in_suite': False, 'job_id': '7729374', 'kernel': {'flavor': 'default', 'kdb': 1, 'sha1': 'b9459a26b0888785d05c47deb35abc9d0354ab93'}, 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'mgrmodules': {'sequential': [{'print': 'Enabling mgr modules'}]}, 'name': 'rishabh-2024-05-28_05:40:25-fs:volumes-rishabh-clone-stats-testing-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'rishabh-clone-stats'}, 'ceph': {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'sha1': '9ed025fbf4406fb54980a0647506b15895b6d863'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '9ed025fbf4406fb54980a0647506b15895b6d863'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'mgr-vol-clone-stats', 'sha1': '9ed025fbf4406fb54980a0647506b15895b6d863'}}, 'owner': 'scheduled_rishabh@teuthology', 'priority': 50, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 8528, 'sha1': '9ed025fbf4406fb54980a0647506b15895b6d863', 'sleep_before_teardown': 0, 'suite': 'fs:volumes', 'suite_branch': 'mgr-vol-clone-stats', 'suite_path': '/home/teuthworker/src/github.com_rishabh-d-dave_ceph_9ed025fbf4406fb54980a0647506b15895b6d863/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/rishabh-d-dave/ceph', 'suite_sha1': '9ed025fbf4406fb54980a0647506b15895b6d863', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi096.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL4jJA/0Z9YK9hfc6842L/ak9ogtcxjH3nLInKMwUJjkE2BSbM6O+geRv29I7bIO27Bmr6wNs2kEMVNHKHiW//k=', 'smithi123.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPlSxAwcnXg7ZNeI06pVJq57T5CWYy1BuOlwhUhGt6s9B0g4MLXL9KbpN/TiOZ+dk3kD7+DwK5iYwTQ9FclVPP4='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'flavor': 'default', 'kdb': 1, 'sha1': 'b9459a26b0888785d05c47deb35abc9d0354ab93'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '9ed025fbf4406fb54980a0647506b15895b6d863'}}, {'ceph': {'create_rbd_pool': False, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'sha1': '9ed025fbf4406fb54980a0647506b15895b6d863', 'cluster': 'ceph'}}, {'sequential': ['mgrmodules']}, {'ceph-fuse': None}, {'cephfs_test_runner': {'fail_on_skip': False, 'modules': ['tasks.cephfs.test_volumes.TestCloneProgressReporter']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ['local function is_kupstream() return false end', 'local function is_kdistro() return false end', 'local function is_fuse() return true end', "local function syntax_version() return '' end"]}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e7ae7580c417d6ab27fda98fecadca5d9f85d63e', 'timestamp': '2024-05-28_05:40:25', 'tube': 'smithi', 'user': 'rishabh', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2698841'} 2024-05-28T06:02:58.469 DEBUG:teuthology.orchestra.run.smithi096:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-05-28T06:02:58.477 DEBUG:teuthology.orchestra.run.smithi123:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-05-28T06:02:58.527 DEBUG:teuthology.orchestra.run.smithi096:> sudo install -d -m0777 -- /var/run/ceph 2024-05-28T06:02:58.532 DEBUG:teuthology.orchestra.run.smithi123:> sudo install -d -m0777 -- /var/run/ceph 2024-05-28T06:02:58.626 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:02:58.626 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/scratch_devs of=/dev/stdout 2024-05-28T06:02:58.682 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-28T06:02:58.682 DEBUG:teuthology.orchestra.run.smithi096:> stat /dev/vg_nvme/lv_1 2024-05-28T06:02:58.732 INFO:teuthology.orchestra.run.smithi096.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-28T06:02:58.732 INFO:teuthology.orchestra.run.smithi096.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-28T06:02:58.733 INFO:teuthology.orchestra.run.smithi096.stdout:Device: 6h/6d Inode: 633 Links: 1 2024-05-28T06:02:58.733 INFO:teuthology.orchestra.run.smithi096.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-28T06:02:58.733 INFO:teuthology.orchestra.run.smithi096.stdout:Context: system_u:object_r:device_t:s0 2024-05-28T06:02:58.733 INFO:teuthology.orchestra.run.smithi096.stdout:Access: 2024-05-28 06:02:20.839692784 +0000 2024-05-28T06:02:58.733 INFO:teuthology.orchestra.run.smithi096.stdout:Modify: 2024-05-28 05:59:38.608955060 +0000 2024-05-28T06:02:58.733 INFO:teuthology.orchestra.run.smithi096.stdout:Change: 2024-05-28 05:59:38.608955060 +0000 2024-05-28T06:02:58.733 INFO:teuthology.orchestra.run.smithi096.stdout: Birth: 2024-05-28 05:59:38.608955060 +0000 2024-05-28T06:02:58.733 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-28T06:02:58.810 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records in 2024-05-28T06:02:58.810 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records out 2024-05-28T06:02:58.810 INFO:teuthology.orchestra.run.smithi096.stderr:512 bytes copied, 0.000367174 s, 1.4 MB/s 2024-05-28T06:02:58.814 DEBUG:teuthology.orchestra.run.smithi096:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-28T06:02:58.869 DEBUG:teuthology.orchestra.run.smithi096:> stat /dev/vg_nvme/lv_2 2024-05-28T06:02:58.917 INFO:teuthology.orchestra.run.smithi096.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-28T06:02:58.917 INFO:teuthology.orchestra.run.smithi096.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-28T06:02:58.918 INFO:teuthology.orchestra.run.smithi096.stdout:Device: 6h/6d Inode: 650 Links: 1 2024-05-28T06:02:58.918 INFO:teuthology.orchestra.run.smithi096.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-28T06:02:58.918 INFO:teuthology.orchestra.run.smithi096.stdout:Context: system_u:object_r:device_t:s0 2024-05-28T06:02:58.918 INFO:teuthology.orchestra.run.smithi096.stdout:Access: 2024-05-28 06:02:20.840692762 +0000 2024-05-28T06:02:58.918 INFO:teuthology.orchestra.run.smithi096.stdout:Modify: 2024-05-28 05:59:39.179942136 +0000 2024-05-28T06:02:58.918 INFO:teuthology.orchestra.run.smithi096.stdout:Change: 2024-05-28 05:59:39.179942136 +0000 2024-05-28T06:02:58.918 INFO:teuthology.orchestra.run.smithi096.stdout: Birth: 2024-05-28 05:59:39.179942136 +0000 2024-05-28T06:02:58.918 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-28T06:02:58.997 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records in 2024-05-28T06:02:58.997 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records out 2024-05-28T06:02:58.997 INFO:teuthology.orchestra.run.smithi096.stderr:512 bytes copied, 0.000391117 s, 1.3 MB/s 2024-05-28T06:02:59.001 DEBUG:teuthology.orchestra.run.smithi096:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-28T06:02:59.056 DEBUG:teuthology.orchestra.run.smithi096:> stat /dev/vg_nvme/lv_3 2024-05-28T06:02:59.107 INFO:teuthology.orchestra.run.smithi096.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-28T06:02:59.107 INFO:teuthology.orchestra.run.smithi096.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-28T06:02:59.107 INFO:teuthology.orchestra.run.smithi096.stdout:Device: 6h/6d Inode: 669 Links: 1 2024-05-28T06:02:59.107 INFO:teuthology.orchestra.run.smithi096.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-28T06:02:59.107 INFO:teuthology.orchestra.run.smithi096.stdout:Context: system_u:object_r:device_t:s0 2024-05-28T06:02:59.107 INFO:teuthology.orchestra.run.smithi096.stdout:Access: 2024-05-28 06:02:20.840692762 +0000 2024-05-28T06:02:59.107 INFO:teuthology.orchestra.run.smithi096.stdout:Modify: 2024-05-28 05:59:39.758929032 +0000 2024-05-28T06:02:59.107 INFO:teuthology.orchestra.run.smithi096.stdout:Change: 2024-05-28 05:59:39.758929032 +0000 2024-05-28T06:02:59.107 INFO:teuthology.orchestra.run.smithi096.stdout: Birth: 2024-05-28 05:59:39.758929032 +0000 2024-05-28T06:02:59.108 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-28T06:02:59.185 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records in 2024-05-28T06:02:59.185 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records out 2024-05-28T06:02:59.185 INFO:teuthology.orchestra.run.smithi096.stderr:512 bytes copied, 0.000342717 s, 1.5 MB/s 2024-05-28T06:02:59.188 DEBUG:teuthology.orchestra.run.smithi096:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-28T06:02:59.246 DEBUG:teuthology.orchestra.run.smithi096:> stat /dev/vg_nvme/lv_4 2024-05-28T06:02:59.295 INFO:teuthology.orchestra.run.smithi096.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-28T06:02:59.295 INFO:teuthology.orchestra.run.smithi096.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-28T06:02:59.295 INFO:teuthology.orchestra.run.smithi096.stdout:Device: 6h/6d Inode: 689 Links: 1 2024-05-28T06:02:59.295 INFO:teuthology.orchestra.run.smithi096.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-28T06:02:59.295 INFO:teuthology.orchestra.run.smithi096.stdout:Context: system_u:object_r:device_t:s0 2024-05-28T06:02:59.295 INFO:teuthology.orchestra.run.smithi096.stdout:Access: 2024-05-28 06:02:20.841692739 +0000 2024-05-28T06:02:59.295 INFO:teuthology.orchestra.run.smithi096.stdout:Modify: 2024-05-28 05:59:40.369915204 +0000 2024-05-28T06:02:59.295 INFO:teuthology.orchestra.run.smithi096.stdout:Change: 2024-05-28 05:59:40.369915204 +0000 2024-05-28T06:02:59.296 INFO:teuthology.orchestra.run.smithi096.stdout: Birth: 2024-05-28 05:59:40.369915204 +0000 2024-05-28T06:02:59.296 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-28T06:02:59.363 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records in 2024-05-28T06:02:59.363 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records out 2024-05-28T06:02:59.363 INFO:teuthology.orchestra.run.smithi096.stderr:512 bytes copied, 0.000340183 s, 1.5 MB/s 2024-05-28T06:02:59.366 DEBUG:teuthology.orchestra.run.smithi096:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-28T06:02:59.416 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-05-28T06:02:59.416 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-05-28T06:02:59.417 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/scratch_devs of=/dev/stdout 2024-05-28T06:02:59.461 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-28T06:02:59.462 DEBUG:teuthology.orchestra.run.smithi123:> stat /dev/vg_nvme/lv_1 2024-05-28T06:02:59.543 INFO:teuthology.orchestra.run.smithi123.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-28T06:02:59.543 INFO:teuthology.orchestra.run.smithi123.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-28T06:02:59.543 INFO:teuthology.orchestra.run.smithi123.stdout:Device: 6h/6d Inode: 631 Links: 1 2024-05-28T06:02:59.544 INFO:teuthology.orchestra.run.smithi123.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-28T06:02:59.544 INFO:teuthology.orchestra.run.smithi123.stdout:Context: system_u:object_r:device_t:s0 2024-05-28T06:02:59.544 INFO:teuthology.orchestra.run.smithi123.stdout:Access: 2024-05-28 06:02:47.506108856 +0000 2024-05-28T06:02:59.544 INFO:teuthology.orchestra.run.smithi123.stdout:Modify: 2024-05-28 05:59:40.332255802 +0000 2024-05-28T06:02:59.544 INFO:teuthology.orchestra.run.smithi123.stdout:Change: 2024-05-28 05:59:40.332255802 +0000 2024-05-28T06:02:59.544 INFO:teuthology.orchestra.run.smithi123.stdout: Birth: 2024-05-28 05:59:40.332255802 +0000 2024-05-28T06:02:59.544 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-28T06:02:59.641 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records in 2024-05-28T06:02:59.641 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records out 2024-05-28T06:02:59.642 INFO:teuthology.orchestra.run.smithi123.stderr:512 bytes copied, 0.000372427 s, 1.4 MB/s 2024-05-28T06:02:59.644 DEBUG:teuthology.orchestra.run.smithi123:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-28T06:02:59.683 DEBUG:teuthology.orchestra.run.smithi123:> stat /dev/vg_nvme/lv_2 2024-05-28T06:02:59.764 INFO:teuthology.orchestra.run.smithi123.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-28T06:02:59.764 INFO:teuthology.orchestra.run.smithi123.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-28T06:02:59.765 INFO:teuthology.orchestra.run.smithi123.stdout:Device: 6h/6d Inode: 653 Links: 1 2024-05-28T06:02:59.765 INFO:teuthology.orchestra.run.smithi123.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-28T06:02:59.765 INFO:teuthology.orchestra.run.smithi123.stdout:Context: system_u:object_r:device_t:s0 2024-05-28T06:02:59.765 INFO:teuthology.orchestra.run.smithi123.stdout:Access: 2024-05-28 06:02:47.507108833 +0000 2024-05-28T06:02:59.765 INFO:teuthology.orchestra.run.smithi123.stdout:Modify: 2024-05-28 05:59:40.922242907 +0000 2024-05-28T06:02:59.765 INFO:teuthology.orchestra.run.smithi123.stdout:Change: 2024-05-28 05:59:40.922242907 +0000 2024-05-28T06:02:59.765 INFO:teuthology.orchestra.run.smithi123.stdout: Birth: 2024-05-28 05:59:40.922242907 +0000 2024-05-28T06:02:59.765 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-28T06:02:59.876 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records in 2024-05-28T06:02:59.876 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records out 2024-05-28T06:02:59.876 INFO:teuthology.orchestra.run.smithi123.stderr:512 bytes copied, 0.000328272 s, 1.6 MB/s 2024-05-28T06:02:59.880 DEBUG:teuthology.orchestra.run.smithi123:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-28T06:02:59.924 DEBUG:teuthology.orchestra.run.smithi123:> stat /dev/vg_nvme/lv_3 2024-05-28T06:03:00.007 INFO:teuthology.orchestra.run.smithi123.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-28T06:03:00.008 INFO:teuthology.orchestra.run.smithi123.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-28T06:03:00.008 INFO:teuthology.orchestra.run.smithi123.stdout:Device: 6h/6d Inode: 671 Links: 1 2024-05-28T06:03:00.008 INFO:teuthology.orchestra.run.smithi123.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-28T06:03:00.008 INFO:teuthology.orchestra.run.smithi123.stdout:Context: system_u:object_r:device_t:s0 2024-05-28T06:03:00.008 INFO:teuthology.orchestra.run.smithi123.stdout:Access: 2024-05-28 06:02:47.508108809 +0000 2024-05-28T06:03:00.008 INFO:teuthology.orchestra.run.smithi123.stdout:Modify: 2024-05-28 05:59:41.540229400 +0000 2024-05-28T06:03:00.008 INFO:teuthology.orchestra.run.smithi123.stdout:Change: 2024-05-28 05:59:41.540229400 +0000 2024-05-28T06:03:00.008 INFO:teuthology.orchestra.run.smithi123.stdout: Birth: 2024-05-28 05:59:41.540229400 +0000 2024-05-28T06:03:00.008 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-28T06:03:00.116 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records in 2024-05-28T06:03:00.117 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records out 2024-05-28T06:03:00.117 INFO:teuthology.orchestra.run.smithi123.stderr:512 bytes copied, 0.000359695 s, 1.4 MB/s 2024-05-28T06:03:00.120 DEBUG:teuthology.orchestra.run.smithi123:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-28T06:03:00.174 DEBUG:teuthology.orchestra.run.smithi123:> stat /dev/vg_nvme/lv_4 2024-05-28T06:03:00.221 INFO:teuthology.orchestra.run.smithi123.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-28T06:03:00.221 INFO:teuthology.orchestra.run.smithi123.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-28T06:03:00.221 INFO:teuthology.orchestra.run.smithi123.stdout:Device: 6h/6d Inode: 690 Links: 1 2024-05-28T06:03:00.221 INFO:teuthology.orchestra.run.smithi123.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-28T06:03:00.221 INFO:teuthology.orchestra.run.smithi123.stdout:Context: system_u:object_r:device_t:s0 2024-05-28T06:03:00.221 INFO:teuthology.orchestra.run.smithi123.stdout:Access: 2024-05-28 06:02:47.509108786 +0000 2024-05-28T06:03:00.221 INFO:teuthology.orchestra.run.smithi123.stdout:Modify: 2024-05-28 05:59:42.141216265 +0000 2024-05-28T06:03:00.222 INFO:teuthology.orchestra.run.smithi123.stdout:Change: 2024-05-28 05:59:42.141216265 +0000 2024-05-28T06:03:00.222 INFO:teuthology.orchestra.run.smithi123.stdout: Birth: 2024-05-28 05:59:42.141216265 +0000 2024-05-28T06:03:00.222 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-28T06:03:00.342 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records in 2024-05-28T06:03:00.342 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records out 2024-05-28T06:03:00.342 INFO:teuthology.orchestra.run.smithi123.stderr:512 bytes copied, 0.000293732 s, 1.7 MB/s 2024-05-28T06:03:00.346 DEBUG:teuthology.orchestra.run.smithi123:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-28T06:03:00.399 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-05-28T06:03:00.399 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi096.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi123.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-05-28T06:03:00.399 INFO:tasks.ceph:Generating config... 2024-05-28T06:03:00.401 INFO:tasks.ceph:[client] client mount timeout = 600 2024-05-28T06:03:00.401 INFO:tasks.ceph:[client] debug client = 20 2024-05-28T06:03:00.401 INFO:tasks.ceph:[client] debug ms = 1 2024-05-28T06:03:00.401 INFO:tasks.ceph:[client] pid file = 2024-05-28T06:03:00.401 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-05-28T06:03:00.401 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-05-28T06:03:00.401 INFO:tasks.ceph:[mds] debug mds = 20 2024-05-28T06:03:00.402 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-05-28T06:03:00.402 INFO:tasks.ceph:[mds] debug ms = 1 2024-05-28T06:03:00.402 INFO:tasks.ceph:[mds] mds debug frag = True 2024-05-28T06:03:00.402 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-05-28T06:03:00.402 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-05-28T06:03:00.402 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-05-28T06:03:00.402 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-05-28T06:03:00.402 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-05-28T06:03:00.402 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-05-28T06:03:00.403 INFO:tasks.ceph:[mgr] debug client = 20 2024-05-28T06:03:00.403 INFO:tasks.ceph:[mgr] debug finisher = 20 2024-05-28T06:03:00.403 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-05-28T06:03:00.403 INFO:tasks.ceph:[mgr] debug ms = 1 2024-05-28T06:03:00.403 INFO:tasks.ceph:[mon] debug mon = 20 2024-05-28T06:03:00.403 INFO:tasks.ceph:[mon] debug ms = 1 2024-05-28T06:03:00.403 INFO:tasks.ceph:[mon] debug paxos = 20 2024-05-28T06:03:00.403 INFO:tasks.ceph:[mon] mon down mkfs grace = 300 2024-05-28T06:03:00.403 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-05-28T06:03:00.404 INFO:tasks.ceph:[osd] bdev async discard = True 2024-05-28T06:03:00.404 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-05-28T06:03:00.404 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-05-28T06:03:00.404 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-05-28T06:03:00.404 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-05-28T06:03:00.404 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-05-28T06:03:00.404 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-05-28T06:03:00.404 INFO:tasks.ceph:[osd] debug ms = 1 2024-05-28T06:03:00.404 INFO:tasks.ceph:[osd] debug osd = 20 2024-05-28T06:03:00.404 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-05-28T06:03:00.404 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-05-28T06:03:00.404 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-05-28T06:03:00.404 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-05-28T06:03:00.404 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-05-28T06:03:00.405 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-05-28T06:03:00.405 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-05-28T06:03:00.405 INFO:tasks.ceph:Setting up mon.a... 2024-05-28T06:03:00.405 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-05-28T06:03:00.553 INFO:teuthology.orchestra.run.smithi096.stdout:creating /etc/ceph/ceph.keyring 2024-05-28T06:03:00.561 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-05-28T06:03:00.684 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-05-28T06:03:00.762 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.96'), ('mon.b', '172.21.15.123'), ('mon.c', '[v2:172.21.15.123:3301,v1:172.21.15.123:6790]')] 2024-05-28T06:03:00.763 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.96,172.21.15.123,[v2:172.21.15.123:3301,v1:172.21.15.123:6790]'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20, 'debug finisher': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-05-28T06:03:00.764 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:03:00.764 DEBUG:teuthology.orchestra.run.smithi096:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-05-28T06:03:00.815 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.96 --add b 172.21.15.123 --addv c '[v2:172.21.15.123:3301,v1:172.21.15.123:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-05-28T06:03:00.955 INFO:teuthology.orchestra.run.smithi096.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-05-28T06:03:00.955 INFO:teuthology.orchestra.run.smithi096.stdout:monmaptool: generated fsid e4a831b5-e6d1-47a7-96cb-bf85154ee0d2 2024-05-28T06:03:00.955 INFO:teuthology.orchestra.run.smithi096.stdout:setting min_mon_release = quincy 2024-05-28T06:03:00.955 INFO:teuthology.orchestra.run.smithi096.stdout:epoch 0 2024-05-28T06:03:00.956 INFO:teuthology.orchestra.run.smithi096.stdout:fsid e4a831b5-e6d1-47a7-96cb-bf85154ee0d2 2024-05-28T06:03:00.956 INFO:teuthology.orchestra.run.smithi096.stdout:last_changed 2024-05-28T06:03:00.955542+0000 2024-05-28T06:03:00.956 INFO:teuthology.orchestra.run.smithi096.stdout:created 2024-05-28T06:03:00.955542+0000 2024-05-28T06:03:00.956 INFO:teuthology.orchestra.run.smithi096.stdout:min_mon_release 17 (quincy) 2024-05-28T06:03:00.956 INFO:teuthology.orchestra.run.smithi096.stdout:election_strategy: 1 2024-05-28T06:03:00.956 INFO:teuthology.orchestra.run.smithi096.stdout:0: [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] mon.a 2024-05-28T06:03:00.956 INFO:teuthology.orchestra.run.smithi096.stdout:1: [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] mon.b 2024-05-28T06:03:00.956 INFO:teuthology.orchestra.run.smithi096.stdout:2: [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] mon.c 2024-05-28T06:03:00.956 INFO:teuthology.orchestra.run.smithi096.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-05-28T06:03:00.959 DEBUG:teuthology.orchestra.run.smithi096:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-05-28T06:03:01.007 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID e4a831b5-e6d1-47a7-96cb-bf85154ee0d2... 2024-05-28T06:03:01.008 DEBUG:teuthology.orchestra.run.smithi096:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-05-28T06:03:01.013 DEBUG:teuthology.orchestra.run.smithi123:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-05-28T06:03:01.136 INFO:teuthology.orchestra.run.smithi096.stdout:[global] 2024-05-28T06:03:01.137 INFO:teuthology.orchestra.run.smithi096.stdout: chdir = "" 2024-05-28T06:03:01.137 INFO:teuthology.orchestra.run.smithi096.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-05-28T06:03:01.137 INFO:teuthology.orchestra.run.smithi096.stdout: auth supported = cephx 2024-05-28T06:03:01.137 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:03:01.137 INFO:teuthology.orchestra.run.smithi096.stdout: filestore xattr use omap = true 2024-05-28T06:03:01.137 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:03:01.137 INFO:teuthology.orchestra.run.smithi096.stdout: mon clock drift allowed = 1.000 2024-05-28T06:03:01.137 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:03:01.137 INFO:teuthology.orchestra.run.smithi096.stdout: osd crush chooseleaf type = 0 2024-05-28T06:03:01.137 INFO:teuthology.orchestra.run.smithi096.stdout: auth debug = true 2024-05-28T06:03:01.137 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:03:01.137 INFO:teuthology.orchestra.run.smithi096.stdout: ms die on old message = true 2024-05-28T06:03:01.138 INFO:teuthology.orchestra.run.smithi096.stdout: ms die on bug = true 2024-05-28T06:03:01.138 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:03:01.138 INFO:teuthology.orchestra.run.smithi096.stdout: mon max pg per osd = 10000 # >= luminous 2024-05-28T06:03:01.138 INFO:teuthology.orchestra.run.smithi096.stdout: mon pg warn max object skew = 0 2024-05-28T06:03:01.138 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:03:01.138 INFO:teuthology.orchestra.run.smithi096.stdout: # disable pg_autoscaler by default for new pools 2024-05-28T06:03:01.138 INFO:teuthology.orchestra.run.smithi096.stdout: osd_pool_default_pg_autoscale_mode = off 2024-05-28T06:03:01.138 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:03:01.139 INFO:teuthology.orchestra.run.smithi096.stdout: osd pool default size = 2 2024-05-28T06:03:01.139 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:03:01.139 INFO:teuthology.orchestra.run.smithi096.stdout: mon osd allow primary affinity = true 2024-05-28T06:03:01.139 INFO:teuthology.orchestra.run.smithi096.stdout: mon osd allow pg remap = true 2024-05-28T06:03:01.139 INFO:teuthology.orchestra.run.smithi096.stdout: mon warn on legacy crush tunables = false 2024-05-28T06:03:01.139 INFO:teuthology.orchestra.run.smithi096.stdout: mon warn on crush straw calc version zero = false 2024-05-28T06:03:01.139 INFO:teuthology.orchestra.run.smithi096.stdout: mon warn on no sortbitwise = false 2024-05-28T06:03:01.139 INFO:teuthology.orchestra.run.smithi096.stdout: mon warn on osd down out interval zero = false 2024-05-28T06:03:01.139 INFO:teuthology.orchestra.run.smithi096.stdout: mon warn on too few osds = false 2024-05-28T06:03:01.139 INFO:teuthology.orchestra.run.smithi096.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-05-28T06:03:01.139 INFO:teuthology.orchestra.run.smithi096.stdout: mon_warn_on_pool_no_redundancy = false 2024-05-28T06:03:01.139 INFO:teuthology.orchestra.run.smithi096.stdout: mon_allow_pool_size_one = true 2024-05-28T06:03:01.139 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:03:01.139 INFO:teuthology.orchestra.run.smithi096.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-05-28T06:03:01.140 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:03:01.140 INFO:teuthology.orchestra.run.smithi096.stdout: osd default data pool replay window = 5 2024-05-28T06:03:01.140 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:03:01.140 INFO:teuthology.orchestra.run.smithi096.stdout: mon allow pool delete = true 2024-05-28T06:03:01.140 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:03:01.140 INFO:teuthology.orchestra.run.smithi096.stdout: mon cluster log file level = debug 2024-05-28T06:03:01.140 INFO:teuthology.orchestra.run.smithi096.stdout: debug asserts on shutdown = true 2024-05-28T06:03:01.140 INFO:teuthology.orchestra.run.smithi096.stdout: mon health detail to clog = false 2024-05-28T06:03:01.140 INFO:teuthology.orchestra.run.smithi096.stdout: mon host = "172.21.15.96,172.21.15.123,[v2:172.21.15.123:3301,v1:172.21.15.123:6790]" 2024-05-28T06:03:01.141 INFO:teuthology.orchestra.run.smithi096.stdout: fsid = e4a831b5-e6d1-47a7-96cb-bf85154ee0d2 2024-05-28T06:03:01.141 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:03:01.141 INFO:teuthology.orchestra.run.smithi096.stdout:[osd] 2024-05-28T06:03:01.141 INFO:teuthology.orchestra.run.smithi096.stdout: osd journal size = 100 2024-05-28T06:03:01.141 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:03:01.141 INFO:teuthology.orchestra.run.smithi096.stdout: osd scrub load threshold = 5.0 2024-05-28T06:03:01.141 INFO:teuthology.orchestra.run.smithi096.stdout: osd scrub max interval = 600 2024-05-28T06:03:01.141 INFO:teuthology.orchestra.run.smithi096.stdout: osd mclock profile = high_recovery_ops 2024-05-28T06:03:01.141 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:03:01.141 INFO:teuthology.orchestra.run.smithi096.stdout: osd recover clone overlap = true 2024-05-28T06:03:01.141 INFO:teuthology.orchestra.run.smithi096.stdout: osd recovery max chunk = 1048576 2024-05-28T06:03:01.141 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:03:01.141 INFO:teuthology.orchestra.run.smithi096.stdout: osd debug shutdown = true 2024-05-28T06:03:01.142 INFO:teuthology.orchestra.run.smithi096.stdout: osd debug op order = true 2024-05-28T06:03:01.142 INFO:teuthology.orchestra.run.smithi096.stdout: osd debug verify stray on activate = true 2024-05-28T06:03:01.142 INFO:teuthology.orchestra.run.smithi096.stdout: osd debug trim objects = true 2024-05-28T06:03:01.142 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:03:01.142 INFO:teuthology.orchestra.run.smithi096.stdout: osd open classes on start = true 2024-05-28T06:03:01.142 INFO:teuthology.orchestra.run.smithi096.stdout: osd debug pg log writeout = true 2024-05-28T06:03:01.142 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:03:01.142 INFO:teuthology.orchestra.run.smithi096.stdout: osd deep scrub update digest min age = 30 2024-05-28T06:03:01.142 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:03:01.142 INFO:teuthology.orchestra.run.smithi096.stdout: osd map max advance = 10 2024-05-28T06:03:01.143 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:03:01.143 INFO:teuthology.orchestra.run.smithi096.stdout: journal zero on create = true 2024-05-28T06:03:01.143 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:03:01.143 INFO:teuthology.orchestra.run.smithi096.stdout: filestore ondisk finisher threads = 3 2024-05-28T06:03:01.143 INFO:teuthology.orchestra.run.smithi096.stdout: filestore apply finisher threads = 3 2024-05-28T06:03:01.143 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:03:01.143 INFO:teuthology.orchestra.run.smithi096.stdout: bdev debug aio = true 2024-05-28T06:03:01.143 INFO:teuthology.orchestra.run.smithi096.stdout: osd debug misdirected ops = true 2024-05-28T06:03:01.143 INFO:teuthology.orchestra.run.smithi096.stdout: bdev async discard = True 2024-05-28T06:03:01.143 INFO:teuthology.orchestra.run.smithi096.stdout: bdev enable discard = True 2024-05-28T06:03:01.144 INFO:teuthology.orchestra.run.smithi096.stdout: bluestore allocator = bitmap 2024-05-28T06:03:01.144 INFO:teuthology.orchestra.run.smithi096.stdout: bluestore block size = 96636764160 2024-05-28T06:03:01.144 INFO:teuthology.orchestra.run.smithi096.stdout: bluestore fsck on mount = True 2024-05-28T06:03:01.144 INFO:teuthology.orchestra.run.smithi096.stdout: debug bluefs = 1/20 2024-05-28T06:03:01.144 INFO:teuthology.orchestra.run.smithi096.stdout: debug bluestore = 1/20 2024-05-28T06:03:01.144 INFO:teuthology.orchestra.run.smithi096.stdout: debug ms = 1 2024-05-28T06:03:01.144 INFO:teuthology.orchestra.run.smithi096.stdout: debug osd = 20 2024-05-28T06:03:01.144 INFO:teuthology.orchestra.run.smithi096.stdout: debug rocksdb = 4/10 2024-05-28T06:03:01.144 INFO:teuthology.orchestra.run.smithi096.stdout: mon osd backfillfull_ratio = 0.85 2024-05-28T06:03:01.144 INFO:teuthology.orchestra.run.smithi096.stdout: mon osd full ratio = 0.9 2024-05-28T06:03:01.145 INFO:teuthology.orchestra.run.smithi096.stdout: mon osd nearfull ratio = 0.8 2024-05-28T06:03:01.145 INFO:teuthology.orchestra.run.smithi096.stdout: osd failsafe full ratio = 0.95 2024-05-28T06:03:01.145 INFO:teuthology.orchestra.run.smithi096.stdout: osd objectstore = bluestore 2024-05-28T06:03:01.145 INFO:teuthology.orchestra.run.smithi096.stdout: osd op complaint time = 180 2024-05-28T06:03:01.145 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:03:01.145 INFO:teuthology.orchestra.run.smithi096.stdout:[mgr] 2024-05-28T06:03:01.145 INFO:teuthology.orchestra.run.smithi096.stdout: debug ms = 1 2024-05-28T06:03:01.145 INFO:teuthology.orchestra.run.smithi096.stdout: debug mgr = 20 2024-05-28T06:03:01.145 INFO:teuthology.orchestra.run.smithi096.stdout: debug mon = 20 2024-05-28T06:03:01.145 INFO:teuthology.orchestra.run.smithi096.stdout: debug auth = 20 2024-05-28T06:03:01.146 INFO:teuthology.orchestra.run.smithi096.stdout: mon reweight min pgs per osd = 4 2024-05-28T06:03:01.146 INFO:teuthology.orchestra.run.smithi096.stdout: mon reweight min bytes per osd = 10 2024-05-28T06:03:01.146 INFO:teuthology.orchestra.run.smithi096.stdout: mgr/telemetry/nag = false 2024-05-28T06:03:01.146 INFO:teuthology.orchestra.run.smithi096.stdout: debug client = 20 2024-05-28T06:03:01.146 INFO:teuthology.orchestra.run.smithi096.stdout: debug finisher = 20 2024-05-28T06:03:01.146 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:03:01.146 INFO:teuthology.orchestra.run.smithi096.stdout:[mon] 2024-05-28T06:03:01.147 INFO:teuthology.orchestra.run.smithi096.stdout: debug ms = 1 2024-05-28T06:03:01.147 INFO:teuthology.orchestra.run.smithi096.stdout: debug mon = 20 2024-05-28T06:03:01.147 INFO:teuthology.orchestra.run.smithi096.stdout: debug paxos = 20 2024-05-28T06:03:01.147 INFO:teuthology.orchestra.run.smithi096.stdout: debug auth = 20 2024-05-28T06:03:01.147 INFO:teuthology.orchestra.run.smithi096.stdout: mon data avail warn = 5 2024-05-28T06:03:01.147 INFO:teuthology.orchestra.run.smithi096.stdout: mon mgr mkfs grace = 240 2024-05-28T06:03:01.147 INFO:teuthology.orchestra.run.smithi096.stdout: mon reweight min pgs per osd = 4 2024-05-28T06:03:01.147 INFO:teuthology.orchestra.run.smithi096.stdout: mon osd reporter subtree level = osd 2024-05-28T06:03:01.148 INFO:teuthology.orchestra.run.smithi096.stdout: mon osd prime pg temp = true 2024-05-28T06:03:01.148 INFO:teuthology.orchestra.run.smithi096.stdout: mon reweight min bytes per osd = 10 2024-05-28T06:03:01.148 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:03:01.148 INFO:teuthology.orchestra.run.smithi096.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-05-28T06:03:01.148 INFO:teuthology.orchestra.run.smithi096.stdout: auth mon ticket ttl = 660 # 11m 2024-05-28T06:03:01.148 INFO:teuthology.orchestra.run.smithi096.stdout: auth service ticket ttl = 240 # 4m 2024-05-28T06:03:01.148 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:03:01.148 INFO:teuthology.orchestra.run.smithi096.stdout: # don't complain about insecure global_id in the test suite 2024-05-28T06:03:01.148 INFO:teuthology.orchestra.run.smithi096.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-05-28T06:03:01.148 INFO:teuthology.orchestra.run.smithi096.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-05-28T06:03:01.149 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:03:01.149 INFO:teuthology.orchestra.run.smithi096.stdout: # 1m isn't quite enough 2024-05-28T06:03:01.149 INFO:teuthology.orchestra.run.smithi096.stdout: mon_down_mkfs_grace = 2m 2024-05-28T06:03:01.149 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:03:01.149 INFO:teuthology.orchestra.run.smithi096.stdout: mon_warn_on_filestore_osds = false 2024-05-28T06:03:01.149 INFO:teuthology.orchestra.run.smithi096.stdout: mon down mkfs grace = 300 2024-05-28T06:03:01.149 INFO:teuthology.orchestra.run.smithi096.stdout: mon op complaint time = 120 2024-05-28T06:03:01.149 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:03:01.149 INFO:teuthology.orchestra.run.smithi096.stdout:[client] 2024-05-28T06:03:01.149 INFO:teuthology.orchestra.run.smithi096.stdout: rgw cache enabled = true 2024-05-28T06:03:01.149 INFO:teuthology.orchestra.run.smithi096.stdout: rgw enable ops log = true 2024-05-28T06:03:01.150 INFO:teuthology.orchestra.run.smithi096.stdout: rgw enable usage log = true 2024-05-28T06:03:01.150 INFO:teuthology.orchestra.run.smithi096.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-05-28T06:03:01.150 INFO:teuthology.orchestra.run.smithi096.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-05-28T06:03:01.150 INFO:teuthology.orchestra.run.smithi096.stdout: client mount timeout = 600 2024-05-28T06:03:01.150 INFO:teuthology.orchestra.run.smithi096.stdout: debug client = 20 2024-05-28T06:03:01.150 INFO:teuthology.orchestra.run.smithi096.stdout: debug ms = 1 2024-05-28T06:03:01.150 INFO:teuthology.orchestra.run.smithi096.stdout: pid file = "" 2024-05-28T06:03:01.150 INFO:teuthology.orchestra.run.smithi096.stdout: rados mon op timeout = 900 2024-05-28T06:03:01.150 INFO:teuthology.orchestra.run.smithi096.stdout: rados osd op timeout = 900 2024-05-28T06:03:01.150 INFO:teuthology.orchestra.run.smithi096.stdout:[mon.a] 2024-05-28T06:03:01.150 INFO:teuthology.orchestra.run.smithi096.stdout:[mon.b] 2024-05-28T06:03:01.151 INFO:teuthology.orchestra.run.smithi096.stdout:[mon.c] 2024-05-28T06:03:01.151 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.a] 2024-05-28T06:03:01.151 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.c] 2024-05-28T06:03:01.151 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.b] 2024-05-28T06:03:01.151 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.d] 2024-05-28T06:03:01.151 INFO:teuthology.orchestra.run.smithi096.stdout:[mds] 2024-05-28T06:03:01.151 INFO:teuthology.orchestra.run.smithi096.stdout: debug mds = 20 2024-05-28T06:03:01.151 INFO:teuthology.orchestra.run.smithi096.stdout: debug mds balancer = 20 2024-05-28T06:03:01.151 INFO:teuthology.orchestra.run.smithi096.stdout: debug ms = 1 2024-05-28T06:03:01.151 INFO:teuthology.orchestra.run.smithi096.stdout: mds debug frag = True 2024-05-28T06:03:01.151 INFO:teuthology.orchestra.run.smithi096.stdout: mds debug scatterstat = True 2024-05-28T06:03:01.151 INFO:teuthology.orchestra.run.smithi096.stdout: mds op complaint time = 180 2024-05-28T06:03:01.152 INFO:teuthology.orchestra.run.smithi096.stdout: mds verify scatter = True 2024-05-28T06:03:01.152 INFO:teuthology.orchestra.run.smithi096.stdout: osd op complaint time = 180 2024-05-28T06:03:01.152 INFO:teuthology.orchestra.run.smithi096.stdout: rados mon op timeout = 900 2024-05-28T06:03:01.152 INFO:teuthology.orchestra.run.smithi096.stdout: rados osd op timeout = 900 2024-05-28T06:03:01.205 INFO:teuthology.orchestra.run.smithi123.stdout:[global] 2024-05-28T06:03:01.205 INFO:teuthology.orchestra.run.smithi123.stdout: chdir = "" 2024-05-28T06:03:01.205 INFO:teuthology.orchestra.run.smithi123.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-05-28T06:03:01.205 INFO:teuthology.orchestra.run.smithi123.stdout: auth supported = cephx 2024-05-28T06:03:01.206 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-05-28T06:03:01.206 INFO:teuthology.orchestra.run.smithi123.stdout: filestore xattr use omap = true 2024-05-28T06:03:01.206 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-05-28T06:03:01.206 INFO:teuthology.orchestra.run.smithi123.stdout: mon clock drift allowed = 1.000 2024-05-28T06:03:01.206 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-05-28T06:03:01.206 INFO:teuthology.orchestra.run.smithi123.stdout: osd crush chooseleaf type = 0 2024-05-28T06:03:01.206 INFO:teuthology.orchestra.run.smithi123.stdout: auth debug = true 2024-05-28T06:03:01.206 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-05-28T06:03:01.206 INFO:teuthology.orchestra.run.smithi123.stdout: ms die on old message = true 2024-05-28T06:03:01.206 INFO:teuthology.orchestra.run.smithi123.stdout: ms die on bug = true 2024-05-28T06:03:01.206 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-05-28T06:03:01.206 INFO:teuthology.orchestra.run.smithi123.stdout: mon max pg per osd = 10000 # >= luminous 2024-05-28T06:03:01.207 INFO:teuthology.orchestra.run.smithi123.stdout: mon pg warn max object skew = 0 2024-05-28T06:03:01.207 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-05-28T06:03:01.207 INFO:teuthology.orchestra.run.smithi123.stdout: # disable pg_autoscaler by default for new pools 2024-05-28T06:03:01.207 INFO:teuthology.orchestra.run.smithi123.stdout: osd_pool_default_pg_autoscale_mode = off 2024-05-28T06:03:01.207 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-05-28T06:03:01.207 INFO:teuthology.orchestra.run.smithi123.stdout: osd pool default size = 2 2024-05-28T06:03:01.207 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-05-28T06:03:01.207 INFO:teuthology.orchestra.run.smithi123.stdout: mon osd allow primary affinity = true 2024-05-28T06:03:01.207 INFO:teuthology.orchestra.run.smithi123.stdout: mon osd allow pg remap = true 2024-05-28T06:03:01.207 INFO:teuthology.orchestra.run.smithi123.stdout: mon warn on legacy crush tunables = false 2024-05-28T06:03:01.207 INFO:teuthology.orchestra.run.smithi123.stdout: mon warn on crush straw calc version zero = false 2024-05-28T06:03:01.207 INFO:teuthology.orchestra.run.smithi123.stdout: mon warn on no sortbitwise = false 2024-05-28T06:03:01.208 INFO:teuthology.orchestra.run.smithi123.stdout: mon warn on osd down out interval zero = false 2024-05-28T06:03:01.208 INFO:teuthology.orchestra.run.smithi123.stdout: mon warn on too few osds = false 2024-05-28T06:03:01.208 INFO:teuthology.orchestra.run.smithi123.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-05-28T06:03:01.208 INFO:teuthology.orchestra.run.smithi123.stdout: mon_warn_on_pool_no_redundancy = false 2024-05-28T06:03:01.208 INFO:teuthology.orchestra.run.smithi123.stdout: mon_allow_pool_size_one = true 2024-05-28T06:03:01.208 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-05-28T06:03:01.208 INFO:teuthology.orchestra.run.smithi123.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-05-28T06:03:01.208 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-05-28T06:03:01.208 INFO:teuthology.orchestra.run.smithi123.stdout: osd default data pool replay window = 5 2024-05-28T06:03:01.208 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-05-28T06:03:01.209 INFO:teuthology.orchestra.run.smithi123.stdout: mon allow pool delete = true 2024-05-28T06:03:01.209 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-05-28T06:03:01.209 INFO:teuthology.orchestra.run.smithi123.stdout: mon cluster log file level = debug 2024-05-28T06:03:01.209 INFO:teuthology.orchestra.run.smithi123.stdout: debug asserts on shutdown = true 2024-05-28T06:03:01.209 INFO:teuthology.orchestra.run.smithi123.stdout: mon health detail to clog = false 2024-05-28T06:03:01.209 INFO:teuthology.orchestra.run.smithi123.stdout: mon host = "172.21.15.96,172.21.15.123,[v2:172.21.15.123:3301,v1:172.21.15.123:6790]" 2024-05-28T06:03:01.209 INFO:teuthology.orchestra.run.smithi123.stdout: fsid = e4a831b5-e6d1-47a7-96cb-bf85154ee0d2 2024-05-28T06:03:01.209 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-05-28T06:03:01.209 INFO:teuthology.orchestra.run.smithi123.stdout:[osd] 2024-05-28T06:03:01.209 INFO:teuthology.orchestra.run.smithi123.stdout: osd journal size = 100 2024-05-28T06:03:01.209 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-05-28T06:03:01.209 INFO:teuthology.orchestra.run.smithi123.stdout: osd scrub load threshold = 5.0 2024-05-28T06:03:01.210 INFO:teuthology.orchestra.run.smithi123.stdout: osd scrub max interval = 600 2024-05-28T06:03:01.210 INFO:teuthology.orchestra.run.smithi123.stdout: osd mclock profile = high_recovery_ops 2024-05-28T06:03:01.210 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-05-28T06:03:01.210 INFO:teuthology.orchestra.run.smithi123.stdout: osd recover clone overlap = true 2024-05-28T06:03:01.210 INFO:teuthology.orchestra.run.smithi123.stdout: osd recovery max chunk = 1048576 2024-05-28T06:03:01.210 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-05-28T06:03:01.210 INFO:teuthology.orchestra.run.smithi123.stdout: osd debug shutdown = true 2024-05-28T06:03:01.210 INFO:teuthology.orchestra.run.smithi123.stdout: osd debug op order = true 2024-05-28T06:03:01.210 INFO:teuthology.orchestra.run.smithi123.stdout: osd debug verify stray on activate = true 2024-05-28T06:03:01.210 INFO:teuthology.orchestra.run.smithi123.stdout: osd debug trim objects = true 2024-05-28T06:03:01.210 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-05-28T06:03:01.210 INFO:teuthology.orchestra.run.smithi123.stdout: osd open classes on start = true 2024-05-28T06:03:01.210 INFO:teuthology.orchestra.run.smithi123.stdout: osd debug pg log writeout = true 2024-05-28T06:03:01.211 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-05-28T06:03:01.211 INFO:teuthology.orchestra.run.smithi123.stdout: osd deep scrub update digest min age = 30 2024-05-28T06:03:01.211 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-05-28T06:03:01.211 INFO:teuthology.orchestra.run.smithi123.stdout: osd map max advance = 10 2024-05-28T06:03:01.211 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-05-28T06:03:01.211 INFO:teuthology.orchestra.run.smithi123.stdout: journal zero on create = true 2024-05-28T06:03:01.211 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-05-28T06:03:01.211 INFO:teuthology.orchestra.run.smithi123.stdout: filestore ondisk finisher threads = 3 2024-05-28T06:03:01.211 INFO:teuthology.orchestra.run.smithi123.stdout: filestore apply finisher threads = 3 2024-05-28T06:03:01.211 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-05-28T06:03:01.211 INFO:teuthology.orchestra.run.smithi123.stdout: bdev debug aio = true 2024-05-28T06:03:01.211 INFO:teuthology.orchestra.run.smithi123.stdout: osd debug misdirected ops = true 2024-05-28T06:03:01.211 INFO:teuthology.orchestra.run.smithi123.stdout: bdev async discard = True 2024-05-28T06:03:01.211 INFO:teuthology.orchestra.run.smithi123.stdout: bdev enable discard = True 2024-05-28T06:03:01.211 INFO:teuthology.orchestra.run.smithi123.stdout: bluestore allocator = bitmap 2024-05-28T06:03:01.212 INFO:teuthology.orchestra.run.smithi123.stdout: bluestore block size = 96636764160 2024-05-28T06:03:01.212 INFO:teuthology.orchestra.run.smithi123.stdout: bluestore fsck on mount = True 2024-05-28T06:03:01.212 INFO:teuthology.orchestra.run.smithi123.stdout: debug bluefs = 1/20 2024-05-28T06:03:01.212 INFO:teuthology.orchestra.run.smithi123.stdout: debug bluestore = 1/20 2024-05-28T06:03:01.212 INFO:teuthology.orchestra.run.smithi123.stdout: debug ms = 1 2024-05-28T06:03:01.212 INFO:teuthology.orchestra.run.smithi123.stdout: debug osd = 20 2024-05-28T06:03:01.212 INFO:teuthology.orchestra.run.smithi123.stdout: debug rocksdb = 4/10 2024-05-28T06:03:01.212 INFO:teuthology.orchestra.run.smithi123.stdout: mon osd backfillfull_ratio = 0.85 2024-05-28T06:03:01.212 INFO:teuthology.orchestra.run.smithi123.stdout: mon osd full ratio = 0.9 2024-05-28T06:03:01.212 INFO:teuthology.orchestra.run.smithi123.stdout: mon osd nearfull ratio = 0.8 2024-05-28T06:03:01.212 INFO:teuthology.orchestra.run.smithi123.stdout: osd failsafe full ratio = 0.95 2024-05-28T06:03:01.212 INFO:teuthology.orchestra.run.smithi123.stdout: osd objectstore = bluestore 2024-05-28T06:03:01.212 INFO:teuthology.orchestra.run.smithi123.stdout: osd op complaint time = 180 2024-05-28T06:03:01.212 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-05-28T06:03:01.213 INFO:teuthology.orchestra.run.smithi123.stdout:[mgr] 2024-05-28T06:03:01.213 INFO:teuthology.orchestra.run.smithi123.stdout: debug ms = 1 2024-05-28T06:03:01.213 INFO:teuthology.orchestra.run.smithi123.stdout: debug mgr = 20 2024-05-28T06:03:01.213 INFO:teuthology.orchestra.run.smithi123.stdout: debug mon = 20 2024-05-28T06:03:01.213 INFO:teuthology.orchestra.run.smithi123.stdout: debug auth = 20 2024-05-28T06:03:01.213 INFO:teuthology.orchestra.run.smithi123.stdout: mon reweight min pgs per osd = 4 2024-05-28T06:03:01.213 INFO:teuthology.orchestra.run.smithi123.stdout: mon reweight min bytes per osd = 10 2024-05-28T06:03:01.213 INFO:teuthology.orchestra.run.smithi123.stdout: mgr/telemetry/nag = false 2024-05-28T06:03:01.213 INFO:teuthology.orchestra.run.smithi123.stdout: debug client = 20 2024-05-28T06:03:01.213 INFO:teuthology.orchestra.run.smithi123.stdout: debug finisher = 20 2024-05-28T06:03:01.213 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-05-28T06:03:01.213 INFO:teuthology.orchestra.run.smithi123.stdout:[mon] 2024-05-28T06:03:01.213 INFO:teuthology.orchestra.run.smithi123.stdout: debug ms = 1 2024-05-28T06:03:01.213 INFO:teuthology.orchestra.run.smithi123.stdout: debug mon = 20 2024-05-28T06:03:01.214 INFO:teuthology.orchestra.run.smithi123.stdout: debug paxos = 20 2024-05-28T06:03:01.214 INFO:teuthology.orchestra.run.smithi123.stdout: debug auth = 20 2024-05-28T06:03:01.214 INFO:teuthology.orchestra.run.smithi123.stdout: mon data avail warn = 5 2024-05-28T06:03:01.214 INFO:teuthology.orchestra.run.smithi123.stdout: mon mgr mkfs grace = 240 2024-05-28T06:03:01.214 INFO:teuthology.orchestra.run.smithi123.stdout: mon reweight min pgs per osd = 4 2024-05-28T06:03:01.214 INFO:teuthology.orchestra.run.smithi123.stdout: mon osd reporter subtree level = osd 2024-05-28T06:03:01.214 INFO:teuthology.orchestra.run.smithi123.stdout: mon osd prime pg temp = true 2024-05-28T06:03:01.214 INFO:teuthology.orchestra.run.smithi123.stdout: mon reweight min bytes per osd = 10 2024-05-28T06:03:01.214 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-05-28T06:03:01.214 INFO:teuthology.orchestra.run.smithi123.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-05-28T06:03:01.214 INFO:teuthology.orchestra.run.smithi123.stdout: auth mon ticket ttl = 660 # 11m 2024-05-28T06:03:01.214 INFO:teuthology.orchestra.run.smithi123.stdout: auth service ticket ttl = 240 # 4m 2024-05-28T06:03:01.214 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-05-28T06:03:01.215 INFO:teuthology.orchestra.run.smithi123.stdout: # don't complain about insecure global_id in the test suite 2024-05-28T06:03:01.215 INFO:teuthology.orchestra.run.smithi123.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-05-28T06:03:01.215 INFO:teuthology.orchestra.run.smithi123.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-05-28T06:03:01.215 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-05-28T06:03:01.215 INFO:teuthology.orchestra.run.smithi123.stdout: # 1m isn't quite enough 2024-05-28T06:03:01.215 INFO:teuthology.orchestra.run.smithi123.stdout: mon_down_mkfs_grace = 2m 2024-05-28T06:03:01.215 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-05-28T06:03:01.215 INFO:teuthology.orchestra.run.smithi123.stdout: mon_warn_on_filestore_osds = false 2024-05-28T06:03:01.215 INFO:teuthology.orchestra.run.smithi123.stdout: mon down mkfs grace = 300 2024-05-28T06:03:01.215 INFO:teuthology.orchestra.run.smithi123.stdout: mon op complaint time = 120 2024-05-28T06:03:01.215 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-05-28T06:03:01.215 INFO:teuthology.orchestra.run.smithi123.stdout:[client] 2024-05-28T06:03:01.215 INFO:teuthology.orchestra.run.smithi123.stdout: rgw cache enabled = true 2024-05-28T06:03:01.215 INFO:teuthology.orchestra.run.smithi123.stdout: rgw enable ops log = true 2024-05-28T06:03:01.216 INFO:teuthology.orchestra.run.smithi123.stdout: rgw enable usage log = true 2024-05-28T06:03:01.216 INFO:teuthology.orchestra.run.smithi123.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-05-28T06:03:01.216 INFO:teuthology.orchestra.run.smithi123.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-05-28T06:03:01.216 INFO:teuthology.orchestra.run.smithi123.stdout: client mount timeout = 600 2024-05-28T06:03:01.216 INFO:teuthology.orchestra.run.smithi123.stdout: debug client = 20 2024-05-28T06:03:01.216 INFO:teuthology.orchestra.run.smithi123.stdout: debug ms = 1 2024-05-28T06:03:01.216 INFO:teuthology.orchestra.run.smithi123.stdout: pid file = "" 2024-05-28T06:03:01.216 INFO:teuthology.orchestra.run.smithi123.stdout: rados mon op timeout = 900 2024-05-28T06:03:01.216 INFO:teuthology.orchestra.run.smithi123.stdout: rados osd op timeout = 900 2024-05-28T06:03:01.216 INFO:teuthology.orchestra.run.smithi123.stdout:[mon.a] 2024-05-28T06:03:01.217 INFO:teuthology.orchestra.run.smithi123.stdout:[mon.b] 2024-05-28T06:03:01.217 INFO:teuthology.orchestra.run.smithi123.stdout:[mon.c] 2024-05-28T06:03:01.217 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.a] 2024-05-28T06:03:01.217 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.c] 2024-05-28T06:03:01.217 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.b] 2024-05-28T06:03:01.217 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.d] 2024-05-28T06:03:01.217 INFO:teuthology.orchestra.run.smithi123.stdout:[mds] 2024-05-28T06:03:01.217 INFO:teuthology.orchestra.run.smithi123.stdout: debug mds = 20 2024-05-28T06:03:01.217 INFO:teuthology.orchestra.run.smithi123.stdout: debug mds balancer = 20 2024-05-28T06:03:01.217 INFO:teuthology.orchestra.run.smithi123.stdout: debug ms = 1 2024-05-28T06:03:01.217 INFO:teuthology.orchestra.run.smithi123.stdout: mds debug frag = True 2024-05-28T06:03:01.217 INFO:teuthology.orchestra.run.smithi123.stdout: mds debug scatterstat = True 2024-05-28T06:03:01.218 INFO:teuthology.orchestra.run.smithi123.stdout: mds op complaint time = 180 2024-05-28T06:03:01.218 INFO:teuthology.orchestra.run.smithi123.stdout: mds verify scatter = True 2024-05-28T06:03:01.218 INFO:teuthology.orchestra.run.smithi123.stdout: osd op complaint time = 180 2024-05-28T06:03:01.218 INFO:teuthology.orchestra.run.smithi123.stdout: rados mon op timeout = 900 2024-05-28T06:03:01.218 INFO:teuthology.orchestra.run.smithi123.stdout: rados osd op timeout = 900 2024-05-28T06:03:01.243 INFO:tasks.ceph:Creating admin key on mon.a... 2024-05-28T06:03:01.243 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-05-28T06:03:01.384 INFO:tasks.ceph:Copying monmap to all nodes... 2024-05-28T06:03:01.384 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:03:01.385 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-05-28T06:03:01.431 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:03:01.431 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-05-28T06:03:01.527 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi096.front.sepia.ceph.com 2024-05-28T06:03:01.527 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:03:01.527 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/etc/ceph/ceph.keyring 2024-05-28T06:03:01.527 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-05-28T06:03:01.641 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:03:01.641 DEBUG:teuthology.orchestra.run.smithi096:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-05-28T06:03:01.739 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi123.front.sepia.ceph.com 2024-05-28T06:03:01.739 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-05-28T06:03:01.739 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/etc/ceph/ceph.keyring 2024-05-28T06:03:01.739 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-05-28T06:03:01.870 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-05-28T06:03:01.870 DEBUG:teuthology.orchestra.run.smithi123:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-05-28T06:03:01.966 INFO:tasks.ceph:Setting up mon nodes... 2024-05-28T06:03:01.966 INFO:tasks.ceph:Setting up mgr nodes... 2024-05-28T06:03:01.966 DEBUG:teuthology.orchestra.run.smithi096:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-05-28T06:03:02.133 INFO:teuthology.orchestra.run.smithi096.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-05-28T06:03:02.144 DEBUG:teuthology.orchestra.run.smithi123:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-05-28T06:03:02.310 INFO:teuthology.orchestra.run.smithi123.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-05-28T06:03:02.321 INFO:tasks.ceph:Setting up mds nodes... 2024-05-28T06:03:02.322 DEBUG:teuthology.orchestra.run.smithi096:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-05-28T06:03:02.474 INFO:teuthology.orchestra.run.smithi096.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-05-28T06:03:02.482 DEBUG:teuthology.orchestra.run.smithi096:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-05-28T06:03:02.564 DEBUG:teuthology.orchestra.run.smithi096:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-05-28T06:03:02.709 INFO:teuthology.orchestra.run.smithi096.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-05-28T06:03:02.717 DEBUG:teuthology.orchestra.run.smithi096:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-05-28T06:03:02.798 DEBUG:teuthology.orchestra.run.smithi123:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-05-28T06:03:02.939 INFO:teuthology.orchestra.run.smithi123.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-05-28T06:03:02.947 DEBUG:teuthology.orchestra.run.smithi123:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-05-28T06:03:03.031 DEBUG:teuthology.orchestra.run.smithi123:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-05-28T06:03:03.193 INFO:teuthology.orchestra.run.smithi123.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-05-28T06:03:03.202 DEBUG:teuthology.orchestra.run.smithi123:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-05-28T06:03:03.287 INFO:tasks.ceph_client:Setting up client nodes... 2024-05-28T06:03:03.287 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-05-28T06:03:03.408 INFO:teuthology.orchestra.run.smithi096.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-05-28T06:03:03.451 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-05-28T06:03:03.562 INFO:teuthology.orchestra.run.smithi096.stdout:creating /etc/ceph/ceph.client.2.keyring 2024-05-28T06:03:03.599 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-05-28T06:03:03.721 INFO:teuthology.orchestra.run.smithi123.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-05-28T06:03:03.773 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2024-05-28T06:03:03.887 INFO:teuthology.orchestra.run.smithi123.stdout:creating /etc/ceph/ceph.client.3.keyring 2024-05-28T06:03:03.925 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-05-28T06:03:03.925 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi096.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi123.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-05-28T06:03:03.926 DEBUG:teuthology.orchestra.run.smithi096:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-05-28T06:03:04.014 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-05-28T06:03:04.014 INFO:tasks.ceph:role: osd.0 2024-05-28T06:03:04.015 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi096.front.sepia.ceph.com 2024-05-28T06:03:04.015 DEBUG:teuthology.orchestra.run.smithi096:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-05-28T06:03:04.101 INFO:teuthology.orchestra.run.smithi096.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-05-28T06:03:04.102 INFO:teuthology.orchestra.run.smithi096.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-28T06:03:04.102 INFO:teuthology.orchestra.run.smithi096.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-28T06:03:04.102 INFO:teuthology.orchestra.run.smithi096.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-28T06:03:04.102 INFO:teuthology.orchestra.run.smithi096.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-28T06:03:04.102 INFO:teuthology.orchestra.run.smithi096.stdout: = sunit=0 swidth=0 blks 2024-05-28T06:03:04.102 INFO:teuthology.orchestra.run.smithi096.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-28T06:03:04.102 INFO:teuthology.orchestra.run.smithi096.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-28T06:03:04.102 INFO:teuthology.orchestra.run.smithi096.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-28T06:03:04.102 INFO:teuthology.orchestra.run.smithi096.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-28T06:03:04.520 INFO:teuthology.orchestra.run.smithi096.stdout:Discarding blocks...Done. 2024-05-28T06:03:04.528 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi096.front.sepia.ceph.com -o noatime 2024-05-28T06:03:04.528 DEBUG:teuthology.orchestra.run.smithi096:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-05-28T06:03:04.672 DEBUG:teuthology.orchestra.run.smithi096:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-05-28T06:03:04.752 DEBUG:teuthology.orchestra.run.smithi096:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-05-28T06:03:04.819 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-05-28T06:03:04.819 INFO:tasks.ceph:role: osd.1 2024-05-28T06:03:04.819 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi096.front.sepia.ceph.com 2024-05-28T06:03:04.820 DEBUG:teuthology.orchestra.run.smithi096:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-05-28T06:03:04.898 INFO:teuthology.orchestra.run.smithi096.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-05-28T06:03:04.898 INFO:teuthology.orchestra.run.smithi096.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-28T06:03:04.898 INFO:teuthology.orchestra.run.smithi096.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-28T06:03:04.898 INFO:teuthology.orchestra.run.smithi096.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-28T06:03:04.898 INFO:teuthology.orchestra.run.smithi096.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-28T06:03:04.898 INFO:teuthology.orchestra.run.smithi096.stdout: = sunit=0 swidth=0 blks 2024-05-28T06:03:04.898 INFO:teuthology.orchestra.run.smithi096.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-28T06:03:04.898 INFO:teuthology.orchestra.run.smithi096.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-28T06:03:04.899 INFO:teuthology.orchestra.run.smithi096.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-28T06:03:04.899 INFO:teuthology.orchestra.run.smithi096.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-28T06:03:05.254 INFO:teuthology.orchestra.run.smithi096.stdout:Discarding blocks...Done. 2024-05-28T06:03:05.262 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi096.front.sepia.ceph.com -o noatime 2024-05-28T06:03:05.262 DEBUG:teuthology.orchestra.run.smithi096:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-05-28T06:03:05.364 DEBUG:teuthology.orchestra.run.smithi096:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-05-28T06:03:05.434 DEBUG:teuthology.orchestra.run.smithi096:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-05-28T06:03:05.498 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-05-28T06:03:05.498 INFO:tasks.ceph:role: osd.2 2024-05-28T06:03:05.498 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi096.front.sepia.ceph.com 2024-05-28T06:03:05.498 DEBUG:teuthology.orchestra.run.smithi096:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-05-28T06:03:05.569 INFO:teuthology.orchestra.run.smithi096.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-05-28T06:03:05.569 INFO:teuthology.orchestra.run.smithi096.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-28T06:03:05.569 INFO:teuthology.orchestra.run.smithi096.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-28T06:03:05.569 INFO:teuthology.orchestra.run.smithi096.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-28T06:03:05.569 INFO:teuthology.orchestra.run.smithi096.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-28T06:03:05.569 INFO:teuthology.orchestra.run.smithi096.stdout: = sunit=0 swidth=0 blks 2024-05-28T06:03:05.570 INFO:teuthology.orchestra.run.smithi096.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-28T06:03:05.570 INFO:teuthology.orchestra.run.smithi096.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-28T06:03:05.570 INFO:teuthology.orchestra.run.smithi096.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-28T06:03:05.570 INFO:teuthology.orchestra.run.smithi096.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-28T06:03:05.951 INFO:teuthology.orchestra.run.smithi096.stdout:Discarding blocks...Done. 2024-05-28T06:03:05.956 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi096.front.sepia.ceph.com -o noatime 2024-05-28T06:03:05.956 DEBUG:teuthology.orchestra.run.smithi096:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-05-28T06:03:06.072 DEBUG:teuthology.orchestra.run.smithi096:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-05-28T06:03:06.141 DEBUG:teuthology.orchestra.run.smithi096:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-05-28T06:03:06.205 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-05-28T06:03:06.205 INFO:tasks.ceph:role: osd.3 2024-05-28T06:03:06.205 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi096.front.sepia.ceph.com 2024-05-28T06:03:06.205 DEBUG:teuthology.orchestra.run.smithi096:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-05-28T06:03:06.278 INFO:teuthology.orchestra.run.smithi096.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-05-28T06:03:06.278 INFO:teuthology.orchestra.run.smithi096.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-28T06:03:06.278 INFO:teuthology.orchestra.run.smithi096.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-28T06:03:06.278 INFO:teuthology.orchestra.run.smithi096.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-28T06:03:06.278 INFO:teuthology.orchestra.run.smithi096.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-28T06:03:06.278 INFO:teuthology.orchestra.run.smithi096.stdout: = sunit=0 swidth=0 blks 2024-05-28T06:03:06.278 INFO:teuthology.orchestra.run.smithi096.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-28T06:03:06.279 INFO:teuthology.orchestra.run.smithi096.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-28T06:03:06.279 INFO:teuthology.orchestra.run.smithi096.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-28T06:03:06.279 INFO:teuthology.orchestra.run.smithi096.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-28T06:03:06.691 INFO:teuthology.orchestra.run.smithi096.stdout:Discarding blocks...Done. 2024-05-28T06:03:06.698 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi096.front.sepia.ceph.com -o noatime 2024-05-28T06:03:06.698 DEBUG:teuthology.orchestra.run.smithi096:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-05-28T06:03:06.841 DEBUG:teuthology.orchestra.run.smithi096:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-05-28T06:03:06.923 DEBUG:teuthology.orchestra.run.smithi096:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-28T06:03:07.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:07.053+0000 7fb9df413640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-05-28T06:03:07.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:07.054+0000 7fb9df413640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-05-28T06:03:07.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:07.054+0000 7fb9df413640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-05-28T06:03:07.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:07.054+0000 7fb9df413640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-05-28T06:03:07.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:07.054+0000 7fb9df413640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-05-28T06:03:07.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:07.057+0000 7fb9df413640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-05-28T06:03:10.028 DEBUG:teuthology.orchestra.run.smithi096:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-05-28T06:03:10.108 DEBUG:teuthology.orchestra.run.smithi096:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-28T06:03:10.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:10.235+0000 7f8507c0f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-05-28T06:03:10.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:10.235+0000 7f8507c0f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-05-28T06:03:10.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:10.235+0000 7f8507c0f640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-05-28T06:03:10.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:10.236+0000 7f8507c0f640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-05-28T06:03:10.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:10.236+0000 7f8507c0f640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-05-28T06:03:10.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:10.238+0000 7f8507c0f640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-05-28T06:03:13.193 DEBUG:teuthology.orchestra.run.smithi096:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-05-28T06:03:13.274 DEBUG:teuthology.orchestra.run.smithi096:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-28T06:03:13.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:13.391+0000 7f45e621a640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-05-28T06:03:13.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:13.391+0000 7f45e621a640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-05-28T06:03:13.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:13.391+0000 7f45e621a640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-05-28T06:03:13.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:13.391+0000 7f45e621a640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-05-28T06:03:13.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:13.391+0000 7f45e621a640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-05-28T06:03:13.395 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:13.394+0000 7f45e621a640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-05-28T06:03:16.367 DEBUG:teuthology.orchestra.run.smithi096:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-05-28T06:03:16.449 DEBUG:teuthology.orchestra.run.smithi096:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-28T06:03:16.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:16.565+0000 7f565685f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-05-28T06:03:16.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:16.566+0000 7f565685f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-05-28T06:03:16.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:16.566+0000 7f565685f640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-05-28T06:03:16.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:16.566+0000 7f565685f640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-05-28T06:03:16.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:16.566+0000 7f565685f640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-05-28T06:03:16.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:16.568+0000 7f565685f640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-05-28T06:03:19.548 DEBUG:teuthology.orchestra.run.smithi096:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-05-28T06:03:19.635 DEBUG:teuthology.orchestra.run.smithi123:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-05-28T06:03:19.726 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-05-28T06:03:19.726 INFO:tasks.ceph:role: osd.4 2024-05-28T06:03:19.726 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi123.front.sepia.ceph.com 2024-05-28T06:03:19.726 DEBUG:teuthology.orchestra.run.smithi123:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-05-28T06:03:19.812 INFO:teuthology.orchestra.run.smithi123.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-05-28T06:03:19.813 INFO:teuthology.orchestra.run.smithi123.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-28T06:03:19.813 INFO:teuthology.orchestra.run.smithi123.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-28T06:03:19.813 INFO:teuthology.orchestra.run.smithi123.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-28T06:03:19.813 INFO:teuthology.orchestra.run.smithi123.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-28T06:03:19.813 INFO:teuthology.orchestra.run.smithi123.stdout: = sunit=0 swidth=0 blks 2024-05-28T06:03:19.813 INFO:teuthology.orchestra.run.smithi123.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-28T06:03:19.813 INFO:teuthology.orchestra.run.smithi123.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-28T06:03:19.814 INFO:teuthology.orchestra.run.smithi123.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-28T06:03:19.814 INFO:teuthology.orchestra.run.smithi123.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-28T06:03:20.222 INFO:teuthology.orchestra.run.smithi123.stdout:Discarding blocks...Done. 2024-05-28T06:03:20.228 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi123.front.sepia.ceph.com -o noatime 2024-05-28T06:03:20.228 DEBUG:teuthology.orchestra.run.smithi123:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-05-28T06:03:20.384 DEBUG:teuthology.orchestra.run.smithi123:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-05-28T06:03:20.473 DEBUG:teuthology.orchestra.run.smithi123:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-05-28T06:03:20.548 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-05-28T06:03:20.548 INFO:tasks.ceph:role: osd.5 2024-05-28T06:03:20.548 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi123.front.sepia.ceph.com 2024-05-28T06:03:20.548 DEBUG:teuthology.orchestra.run.smithi123:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-05-28T06:03:20.633 INFO:teuthology.orchestra.run.smithi123.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-05-28T06:03:20.633 INFO:teuthology.orchestra.run.smithi123.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-28T06:03:20.633 INFO:teuthology.orchestra.run.smithi123.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-28T06:03:20.633 INFO:teuthology.orchestra.run.smithi123.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-28T06:03:20.634 INFO:teuthology.orchestra.run.smithi123.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-28T06:03:20.634 INFO:teuthology.orchestra.run.smithi123.stdout: = sunit=0 swidth=0 blks 2024-05-28T06:03:20.634 INFO:teuthology.orchestra.run.smithi123.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-28T06:03:20.634 INFO:teuthology.orchestra.run.smithi123.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-28T06:03:20.634 INFO:teuthology.orchestra.run.smithi123.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-28T06:03:20.634 INFO:teuthology.orchestra.run.smithi123.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-28T06:03:21.021 INFO:teuthology.orchestra.run.smithi123.stdout:Discarding blocks...Done. 2024-05-28T06:03:21.029 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi123.front.sepia.ceph.com -o noatime 2024-05-28T06:03:21.029 DEBUG:teuthology.orchestra.run.smithi123:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-05-28T06:03:21.143 DEBUG:teuthology.orchestra.run.smithi123:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-05-28T06:03:21.222 DEBUG:teuthology.orchestra.run.smithi123:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-05-28T06:03:21.281 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-05-28T06:03:21.281 INFO:tasks.ceph:role: osd.6 2024-05-28T06:03:21.281 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi123.front.sepia.ceph.com 2024-05-28T06:03:21.281 DEBUG:teuthology.orchestra.run.smithi123:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-05-28T06:03:21.356 INFO:teuthology.orchestra.run.smithi123.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-05-28T06:03:21.357 INFO:teuthology.orchestra.run.smithi123.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-28T06:03:21.357 INFO:teuthology.orchestra.run.smithi123.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-28T06:03:21.357 INFO:teuthology.orchestra.run.smithi123.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-28T06:03:21.357 INFO:teuthology.orchestra.run.smithi123.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-28T06:03:21.357 INFO:teuthology.orchestra.run.smithi123.stdout: = sunit=0 swidth=0 blks 2024-05-28T06:03:21.357 INFO:teuthology.orchestra.run.smithi123.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-28T06:03:21.357 INFO:teuthology.orchestra.run.smithi123.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-28T06:03:21.357 INFO:teuthology.orchestra.run.smithi123.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-28T06:03:21.357 INFO:teuthology.orchestra.run.smithi123.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-28T06:03:21.743 INFO:teuthology.orchestra.run.smithi123.stdout:Discarding blocks...Done. 2024-05-28T06:03:21.749 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi123.front.sepia.ceph.com -o noatime 2024-05-28T06:03:21.749 DEBUG:teuthology.orchestra.run.smithi123:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-05-28T06:03:21.883 DEBUG:teuthology.orchestra.run.smithi123:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-05-28T06:03:21.958 DEBUG:teuthology.orchestra.run.smithi123:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-05-28T06:03:22.022 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-05-28T06:03:22.022 INFO:tasks.ceph:role: osd.7 2024-05-28T06:03:22.022 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi123.front.sepia.ceph.com 2024-05-28T06:03:22.022 DEBUG:teuthology.orchestra.run.smithi123:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-05-28T06:03:22.092 INFO:teuthology.orchestra.run.smithi123.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-05-28T06:03:22.092 INFO:teuthology.orchestra.run.smithi123.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-28T06:03:22.092 INFO:teuthology.orchestra.run.smithi123.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-28T06:03:22.092 INFO:teuthology.orchestra.run.smithi123.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-28T06:03:22.092 INFO:teuthology.orchestra.run.smithi123.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-28T06:03:22.092 INFO:teuthology.orchestra.run.smithi123.stdout: = sunit=0 swidth=0 blks 2024-05-28T06:03:22.092 INFO:teuthology.orchestra.run.smithi123.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-28T06:03:22.093 INFO:teuthology.orchestra.run.smithi123.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-28T06:03:22.093 INFO:teuthology.orchestra.run.smithi123.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-28T06:03:22.093 INFO:teuthology.orchestra.run.smithi123.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-28T06:03:22.504 INFO:teuthology.orchestra.run.smithi123.stdout:Discarding blocks...Done. 2024-05-28T06:03:22.510 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi123.front.sepia.ceph.com -o noatime 2024-05-28T06:03:22.510 DEBUG:teuthology.orchestra.run.smithi123:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-05-28T06:03:22.653 DEBUG:teuthology.orchestra.run.smithi123:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-05-28T06:03:22.732 DEBUG:teuthology.orchestra.run.smithi123:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-28T06:03:22.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:22.864+0000 7f1e9d00d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-05-28T06:03:22.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:22.865+0000 7f1e9d00d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-05-28T06:03:22.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:22.865+0000 7f1e9d00d640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-05-28T06:03:22.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:22.865+0000 7f1e9d00d640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-05-28T06:03:22.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:22.865+0000 7f1e9d00d640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-05-28T06:03:22.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:22.868+0000 7f1e9d00d640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-05-28T06:03:25.852 DEBUG:teuthology.orchestra.run.smithi123:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-05-28T06:03:25.937 DEBUG:teuthology.orchestra.run.smithi123:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-28T06:03:26.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:26.062+0000 7f9fc4edf640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-05-28T06:03:26.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:26.063+0000 7f9fc4edf640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-05-28T06:03:26.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:26.063+0000 7f9fc4edf640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-05-28T06:03:26.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:26.063+0000 7f9fc4edf640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-05-28T06:03:26.065 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:26.063+0000 7f9fc4edf640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-05-28T06:03:26.066 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:26.065+0000 7f9fc4edf640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-05-28T06:03:28.470 DEBUG:teuthology.orchestra.run.smithi096:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:03:28.478 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:03:29.053 DEBUG:teuthology.orchestra.run.smithi123:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-05-28T06:03:29.137 DEBUG:teuthology.orchestra.run.smithi123:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-28T06:03:29.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:29.261+0000 7f1e5d20f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-05-28T06:03:29.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:29.262+0000 7f1e5d20f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-05-28T06:03:29.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:29.262+0000 7f1e5d20f640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-05-28T06:03:29.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:29.262+0000 7f1e5d20f640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-05-28T06:03:29.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:29.262+0000 7f1e5d20f640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-05-28T06:03:29.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:29.264+0000 7f1e5d20f640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-05-28T06:03:32.494 DEBUG:teuthology.orchestra.run.smithi123:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-05-28T06:03:32.581 DEBUG:teuthology.orchestra.run.smithi123:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-28T06:03:32.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:32.706+0000 7f4a94a12640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-05-28T06:03:32.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:32.707+0000 7f4a94a12640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-05-28T06:03:32.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:32.707+0000 7f4a94a12640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-05-28T06:03:32.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:32.707+0000 7f4a94a12640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-05-28T06:03:32.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:32.707+0000 7f4a94a12640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-05-28T06:03:32.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:32.709+0000 7f4a94a12640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-05-28T06:03:35.919 DEBUG:teuthology.orchestra.run.smithi123:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-05-28T06:03:35.999 INFO:tasks.ceph:Reading keys from all nodes... 2024-05-28T06:03:35.999 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:03:36.000 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-05-28T06:03:36.088 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:03:36.089 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-05-28T06:03:36.173 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:03:36.173 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-05-28T06:03:36.259 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:03:36.260 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-05-28T06:03:36.344 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:03:36.344 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-05-28T06:03:36.426 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:03:36.427 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-05-28T06:03:36.508 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:03:36.509 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-05-28T06:03:36.591 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-05-28T06:03:36.591 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-05-28T06:03:36.679 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-05-28T06:03:36.679 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-05-28T06:03:36.767 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-05-28T06:03:36.767 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-05-28T06:03:36.850 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-05-28T06:03:36.850 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-05-28T06:03:36.934 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-05-28T06:03:36.934 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-05-28T06:03:37.014 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-05-28T06:03:37.014 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-05-28T06:03:37.097 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-05-28T06:03:37.097 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-05-28T06:03:37.180 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:03:37.180 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-05-28T06:03:37.236 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:03:37.236 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2024-05-28T06:03:37.287 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-05-28T06:03:37.287 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-05-28T06:03:37.345 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-05-28T06:03:37.345 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2024-05-28T06:03:37.396 INFO:tasks.ceph:Adding keys to all mons... 2024-05-28T06:03:37.396 DEBUG:teuthology.orchestra.run.smithi096:> sudo tee -a /etc/ceph/ceph.keyring 2024-05-28T06:03:37.403 DEBUG:teuthology.orchestra.run.smithi123:> sudo tee -a /etc/ceph/ceph.keyring 2024-05-28T06:03:37.483 INFO:teuthology.orchestra.run.smithi096.stdout:[mgr.y] 2024-05-28T06:03:37.484 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQCWc1Vmh5r7BxAAA74gn+FozehM+34eFik8kw== 2024-05-28T06:03:37.484 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.a] 2024-05-28T06:03:37.484 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQCWc1VmyA9HHBAANJCF8I/RP5ZYviyOzo0JCQ== 2024-05-28T06:03:37.484 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.c] 2024-05-28T06:03:37.484 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQCWc1VmrLxHKhAAhkTRLDKL6RUJQGetB8JQQA== 2024-05-28T06:03:37.484 INFO:teuthology.orchestra.run.smithi096.stdout:[osd.0] 2024-05-28T06:03:37.484 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQCbc1VmEcNAAxAAiB5AsfY2uRY1kc6h80chuw== 2024-05-28T06:03:37.484 INFO:teuthology.orchestra.run.smithi096.stdout:[osd.1] 2024-05-28T06:03:37.484 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQCec1Vm6CAaDhAA1NkYM2q4fTsgSDSwJZL3Qg== 2024-05-28T06:03:37.484 INFO:teuthology.orchestra.run.smithi096.stdout:[osd.2] 2024-05-28T06:03:37.484 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQChc1VmKn1jFxAA+dM4gM7FR8NwBpyd833A/Q== 2024-05-28T06:03:37.484 INFO:teuthology.orchestra.run.smithi096.stdout:[osd.3] 2024-05-28T06:03:37.485 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQCkc1VmV07MIRAAHIXQGDVbL9V5QqexQ89IJA== 2024-05-28T06:03:37.485 INFO:teuthology.orchestra.run.smithi096.stdout:[mgr.x] 2024-05-28T06:03:37.485 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQCWc1VmVMx/EhAAylvi37aq3hbMOhUSQWLVSg== 2024-05-28T06:03:37.485 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.b] 2024-05-28T06:03:37.485 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQCWc1VmtR0EOBAA296qGHTmB/2sMztDbDJ56g== 2024-05-28T06:03:37.485 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.d] 2024-05-28T06:03:37.485 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQCXc1VmW2aQCxAAnkDwQzlgy39vFe2O9Wwf7Q== 2024-05-28T06:03:37.485 INFO:teuthology.orchestra.run.smithi096.stdout:[osd.4] 2024-05-28T06:03:37.485 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQCqc1VmobuZMxAAlY8xhuzqlkuFrcbTouF5rA== 2024-05-28T06:03:37.485 INFO:teuthology.orchestra.run.smithi096.stdout:[osd.5] 2024-05-28T06:03:37.485 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQCuc1Vmwt3QAxAAUFj/e/wJvxljgAbcTb32Ug== 2024-05-28T06:03:37.485 INFO:teuthology.orchestra.run.smithi096.stdout:[osd.6] 2024-05-28T06:03:37.485 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQCxc1VmscerDxAAqEDAqy6/OvdfSJ+ALZ7WeQ== 2024-05-28T06:03:37.485 INFO:teuthology.orchestra.run.smithi096.stdout:[osd.7] 2024-05-28T06:03:37.486 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQC0c1VmgxkbKhAA7dFzDTBKFmKiFFxPsyiT4Q== 2024-05-28T06:03:37.486 INFO:teuthology.orchestra.run.smithi096.stdout:[client.0] 2024-05-28T06:03:37.486 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQCXc1Vm91hbGBAASTL3HQR8XOs7wVQEHbPl/g== 2024-05-28T06:03:37.486 INFO:teuthology.orchestra.run.smithi096.stdout:[client.2] 2024-05-28T06:03:37.486 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQCXc1VmCS+OIRAAMV3Au6KuchLhA59QTGbv1w== 2024-05-28T06:03:37.486 INFO:teuthology.orchestra.run.smithi096.stdout:[client.1] 2024-05-28T06:03:37.486 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQCXc1VmIc77KhAAw2e0alTmVROhyCIFW4H3Sg== 2024-05-28T06:03:37.486 INFO:teuthology.orchestra.run.smithi096.stdout:[client.3] 2024-05-28T06:03:37.486 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQCXc1Vm5r3pNBAAkVZ3arjywHQkEP5lmS8xqg== 2024-05-28T06:03:37.488 INFO:teuthology.orchestra.run.smithi123.stdout:[mgr.y] 2024-05-28T06:03:37.488 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQCWc1Vmh5r7BxAAA74gn+FozehM+34eFik8kw== 2024-05-28T06:03:37.488 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.a] 2024-05-28T06:03:37.488 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQCWc1VmyA9HHBAANJCF8I/RP5ZYviyOzo0JCQ== 2024-05-28T06:03:37.488 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.c] 2024-05-28T06:03:37.488 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQCWc1VmrLxHKhAAhkTRLDKL6RUJQGetB8JQQA== 2024-05-28T06:03:37.488 INFO:teuthology.orchestra.run.smithi123.stdout:[osd.0] 2024-05-28T06:03:37.488 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQCbc1VmEcNAAxAAiB5AsfY2uRY1kc6h80chuw== 2024-05-28T06:03:37.488 INFO:teuthology.orchestra.run.smithi123.stdout:[osd.1] 2024-05-28T06:03:37.488 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQCec1Vm6CAaDhAA1NkYM2q4fTsgSDSwJZL3Qg== 2024-05-28T06:03:37.488 INFO:teuthology.orchestra.run.smithi123.stdout:[osd.2] 2024-05-28T06:03:37.488 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQChc1VmKn1jFxAA+dM4gM7FR8NwBpyd833A/Q== 2024-05-28T06:03:37.489 INFO:teuthology.orchestra.run.smithi123.stdout:[osd.3] 2024-05-28T06:03:37.489 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQCkc1VmV07MIRAAHIXQGDVbL9V5QqexQ89IJA== 2024-05-28T06:03:37.489 INFO:teuthology.orchestra.run.smithi123.stdout:[mgr.x] 2024-05-28T06:03:37.489 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQCWc1VmVMx/EhAAylvi37aq3hbMOhUSQWLVSg== 2024-05-28T06:03:37.489 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.b] 2024-05-28T06:03:37.489 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQCWc1VmtR0EOBAA296qGHTmB/2sMztDbDJ56g== 2024-05-28T06:03:37.489 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.d] 2024-05-28T06:03:37.489 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQCXc1VmW2aQCxAAnkDwQzlgy39vFe2O9Wwf7Q== 2024-05-28T06:03:37.489 INFO:teuthology.orchestra.run.smithi123.stdout:[osd.4] 2024-05-28T06:03:37.489 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQCqc1VmobuZMxAAlY8xhuzqlkuFrcbTouF5rA== 2024-05-28T06:03:37.489 INFO:teuthology.orchestra.run.smithi123.stdout:[osd.5] 2024-05-28T06:03:37.489 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQCuc1Vmwt3QAxAAUFj/e/wJvxljgAbcTb32Ug== 2024-05-28T06:03:37.489 INFO:teuthology.orchestra.run.smithi123.stdout:[osd.6] 2024-05-28T06:03:37.490 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQCxc1VmscerDxAAqEDAqy6/OvdfSJ+ALZ7WeQ== 2024-05-28T06:03:37.490 INFO:teuthology.orchestra.run.smithi123.stdout:[osd.7] 2024-05-28T06:03:37.490 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQC0c1VmgxkbKhAA7dFzDTBKFmKiFFxPsyiT4Q== 2024-05-28T06:03:37.490 INFO:teuthology.orchestra.run.smithi123.stdout:[client.0] 2024-05-28T06:03:37.490 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQCXc1Vm91hbGBAASTL3HQR8XOs7wVQEHbPl/g== 2024-05-28T06:03:37.490 INFO:teuthology.orchestra.run.smithi123.stdout:[client.2] 2024-05-28T06:03:37.490 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQCXc1VmCS+OIRAAMV3Au6KuchLhA59QTGbv1w== 2024-05-28T06:03:37.490 INFO:teuthology.orchestra.run.smithi123.stdout:[client.1] 2024-05-28T06:03:37.490 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQCXc1VmIc77KhAAw2e0alTmVROhyCIFW4H3Sg== 2024-05-28T06:03:37.490 INFO:teuthology.orchestra.run.smithi123.stdout:[client.3] 2024-05-28T06:03:37.490 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQCXc1Vm5r3pNBAAkVZ3arjywHQkEP5lmS8xqg== 2024-05-28T06:03:37.491 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-05-28T06:03:37.538 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-05-28T06:03:37.675 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-05-28T06:03:37.681 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-05-28T06:03:37.814 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-05-28T06:03:37.819 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-05-28T06:03:37.947 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-28T06:03:37.952 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-28T06:03:38.082 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-28T06:03:38.087 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-28T06:03:38.216 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-28T06:03:38.223 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-28T06:03:38.355 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-28T06:03:38.361 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-28T06:03:38.489 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-05-28T06:03:38.496 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-05-28T06:03:38.628 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-05-28T06:03:38.635 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-05-28T06:03:38.771 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-05-28T06:03:38.776 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-05-28T06:03:38.912 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-28T06:03:38.918 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-28T06:03:39.042 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-28T06:03:39.052 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-28T06:03:39.179 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-28T06:03:39.187 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-28T06:03:39.323 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-28T06:03:39.329 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-28T06:03:39.457 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-28T06:03:39.466 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-28T06:03:39.597 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-28T06:03:39.603 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-28T06:03:39.742 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-28T06:03:39.767 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-28T06:03:39.907 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-28T06:03:39.913 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-28T06:03:40.046 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-05-28T06:03:40.046 DEBUG:teuthology.orchestra.run.smithi096:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-05-28T06:03:40.121 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-05-28T06:03:40.271 DEBUG:teuthology.orchestra.run.smithi096:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-05-28T06:03:40.349 DEBUG:teuthology.orchestra.run.smithi123:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-05-28T06:03:40.433 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-05-28T06:03:40.585 DEBUG:teuthology.orchestra.run.smithi123:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-05-28T06:03:40.652 DEBUG:teuthology.orchestra.run.smithi123:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-05-28T06:03:40.732 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-05-28T06:03:40.876 DEBUG:teuthology.orchestra.run.smithi123:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-05-28T06:03:40.960 DEBUG:teuthology.orchestra.run.smithi096:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-05-28T06:03:40.968 DEBUG:teuthology.orchestra.run.smithi123:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-05-28T06:03:41.024 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-05-28T06:03:41.025 INFO:tasks.ceph.mon.a:Restarting daemon 2024-05-28T06:03:41.025 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-05-28T06:03:41.032 INFO:tasks.ceph.mon.a:Started 2024-05-28T06:03:41.032 INFO:tasks.ceph.mon.b:Restarting daemon 2024-05-28T06:03:41.033 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-05-28T06:03:41.041 INFO:tasks.ceph.mon.b:Started 2024-05-28T06:03:41.041 INFO:tasks.ceph.mon.c:Restarting daemon 2024-05-28T06:03:41.041 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-05-28T06:03:41.046 INFO:tasks.ceph.mon.c:Started 2024-05-28T06:03:41.046 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-05-28T06:03:41.047 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-05-28T06:03:41.047 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-05-28T06:03:41.052 INFO:tasks.ceph.mgr.y:Started 2024-05-28T06:03:41.053 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-05-28T06:03:41.053 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-05-28T06:03:41.058 INFO:tasks.ceph.mgr.x:Started 2024-05-28T06:03:41.059 INFO:tasks.ceph:Setting crush tunables to default 2024-05-28T06:03:41.059 DEBUG:teuthology.orchestra.run.smithi096:> sudo ceph --cluster ceph osd crush tunables default 2024-05-28T06:03:41.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:41.223+0000 7f3533e00640 1 Processor -- start 2024-05-28T06:03:41.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:41.224+0000 7f3533e00640 1 -- start start 2024-05-28T06:03:41.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:41.225+0000 7f3533e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f353415ea40 0x7f35340cfab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:03:41.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:41.225+0000 7f3533e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f353415ee20 0x7f353415fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:03:41.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:41.225+0000 7f3533e00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35340d37b0 con 0x7f35340d5190 2024-05-28T06:03:41.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:41.225+0000 7f3533e00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f35340cecf0 con 0x7f353415ee20 2024-05-28T06:03:41.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:41.225+0000 7f3533e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f35340cefd0 con 0x7f353415ea40 2024-05-28T06:03:41.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:41.225+0000 7f352a000640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f35340d5190 0x7f35340d30a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:41362/0 (socket says 172.21.15.96:41362) 2024-05-28T06:03:41.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:41.225+0000 7f352a000640 1 -- 172.21.15.96:0/1852670942 learned_addr learned my addr 172.21.15.96:0/1852670942 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:03:41.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:41.225+0000 7f3533400640 1 --2- 172.21.15.96:0/1852670942 >> v2:172.21.15.96:3300/0 conn(0x7f353415ee20 0x7f353415fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:03:41.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:41.225+0000 7f3532a00640 1 --2- 172.21.15.96:0/1852670942 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f353415ea40 0x7f35340cfab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:03:41.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:41.225+0000 7f3533400640 1 -- 172.21.15.96:0/1852670942 >> v2:172.21.15.96:3300/0 conn(0x7f353415ee20 msgr2=0x7f353415fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-05-28T06:03:41.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:41.225+0000 7f3533400640 1 -- 172.21.15.96:0/1852670942 >> v2:172.21.15.96:3300/0 conn(0x7f353415ee20 msgr2=0x7f353415fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-28T06:03:41.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:41.225+0000 7f3533400640 1 --2- 172.21.15.96:0/1852670942 >> v2:172.21.15.96:3300/0 conn(0x7f353415ee20 0x7f353415fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-28T06:03:41.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:41.225+0000 7f3532a00640 1 -- 172.21.15.96:0/1852670942 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f353415ea40 msgr2=0x7f35340cfab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-28T06:03:41.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:41.225+0000 7f3532a00640 1 -- 172.21.15.96:0/1852670942 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f353415ea40 msgr2=0x7f35340cfab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-28T06:03:41.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:41.225+0000 7f3532a00640 1 --2- 172.21.15.96:0/1852670942 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f353415ea40 0x7f35340cfab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-28T06:03:41.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:41.225+0000 7f3533400640 1 --2- 172.21.15.96:0/1852670942 >> v2:172.21.15.96:3300/0 conn(0x7f353415ee20 0x7f353415fff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-28T06:03:41.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:41.225+0000 7f3532a00640 1 --2- 172.21.15.96:0/1852670942 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f353415ea40 0x7f35340cfab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-28T06:03:41.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:41.426+0000 7f3532a00640 1 --2- 172.21.15.96:0/1852670942 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f353415ea40 0x7f35340cfab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:03:41.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:41.427+0000 7f3532a00640 1 -- 172.21.15.96:0/1852670942 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f353415ea40 msgr2=0x7f35340cfab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-28T06:03:41.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:41.427+0000 7f3532a00640 1 -- 172.21.15.96:0/1852670942 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f353415ea40 msgr2=0x7f35340cfab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-28T06:03:41.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:41.427+0000 7f3532a00640 1 --2- 172.21.15.96:0/1852670942 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f353415ea40 0x7f35340cfab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-28T06:03:41.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:41.427+0000 7f3532a00640 1 --2- 172.21.15.96:0/1852670942 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f353415ea40 0x7f35340cfab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-28T06:03:41.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:41.427+0000 7f3533400640 1 --2- 172.21.15.96:0/1852670942 >> v2:172.21.15.96:3300/0 conn(0x7f353415ee20 0x7f353415fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:03:41.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:41.428+0000 7f3533400640 1 -- 172.21.15.96:0/1852670942 >> v2:172.21.15.96:3300/0 conn(0x7f353415ee20 msgr2=0x7f353415fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-05-28T06:03:41.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:41.428+0000 7f3533400640 1 -- 172.21.15.96:0/1852670942 >> v2:172.21.15.96:3300/0 conn(0x7f353415ee20 msgr2=0x7f353415fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-28T06:03:41.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:41.428+0000 7f3533400640 1 --2- 172.21.15.96:0/1852670942 >> v2:172.21.15.96:3300/0 conn(0x7f353415ee20 0x7f353415fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-28T06:03:41.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:41.428+0000 7f3533400640 1 --2- 172.21.15.96:0/1852670942 >> v2:172.21.15.96:3300/0 conn(0x7f353415ee20 0x7f353415fff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-28T06:03:41.830 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:41.828+0000 7f3532a00640 1 --2- 172.21.15.96:0/1852670942 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f353415ea40 0x7f35340cfab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:03:41.830 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:41.829+0000 7f3532a00640 1 -- 172.21.15.96:0/1852670942 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f353415ea40 msgr2=0x7f35340cfab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-28T06:03:41.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:41.829+0000 7f3532a00640 1 -- 172.21.15.96:0/1852670942 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f353415ea40 msgr2=0x7f35340cfab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-28T06:03:41.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:41.829+0000 7f3532a00640 1 --2- 172.21.15.96:0/1852670942 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f353415ea40 0x7f35340cfab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-28T06:03:41.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:41.829+0000 7f3532a00640 1 --2- 172.21.15.96:0/1852670942 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f353415ea40 0x7f35340cfab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-28T06:03:41.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:41.829+0000 7f3533400640 1 --2- 172.21.15.96:0/1852670942 >> v2:172.21.15.96:3300/0 conn(0x7f353415ee20 0x7f353415fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:03:41.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:41.830+0000 7f3533400640 1 -- 172.21.15.96:0/1852670942 >> v2:172.21.15.96:3300/0 conn(0x7f353415ee20 msgr2=0x7f353415fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-05-28T06:03:41.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:41.830+0000 7f3533400640 1 -- 172.21.15.96:0/1852670942 >> v2:172.21.15.96:3300/0 conn(0x7f353415ee20 msgr2=0x7f353415fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-28T06:03:41.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:41.830+0000 7f3533400640 1 --2- 172.21.15.96:0/1852670942 >> v2:172.21.15.96:3300/0 conn(0x7f353415ee20 0x7f353415fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-28T06:03:41.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:41.830+0000 7f3533400640 1 --2- 172.21.15.96:0/1852670942 >> v2:172.21.15.96:3300/0 conn(0x7f353415ee20 0x7f353415fff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-28T06:03:42.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:42.631+0000 7f3532a00640 1 --2- 172.21.15.96:0/1852670942 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f353415ea40 0x7f35340cfab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:03:42.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:42.631+0000 7f3532a00640 1 -- 172.21.15.96:0/1852670942 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f353415ea40 msgr2=0x7f35340cfab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-28T06:03:42.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:42.631+0000 7f3532a00640 1 -- 172.21.15.96:0/1852670942 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f353415ea40 msgr2=0x7f35340cfab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-28T06:03:42.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:42.631+0000 7f3532a00640 1 --2- 172.21.15.96:0/1852670942 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f353415ea40 0x7f35340cfab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-28T06:03:42.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:42.631+0000 7f3532a00640 1 --2- 172.21.15.96:0/1852670942 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f353415ea40 0x7f35340cfab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-28T06:03:42.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:42.631+0000 7f3533400640 1 --2- 172.21.15.96:0/1852670942 >> v2:172.21.15.96:3300/0 conn(0x7f353415ee20 0x7f353415fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:03:42.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:42.632+0000 7f3533400640 1 -- 172.21.15.96:0/1852670942 >> v2:172.21.15.96:3300/0 conn(0x7f353415ee20 msgr2=0x7f353415fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-05-28T06:03:42.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:42.632+0000 7f3533400640 1 -- 172.21.15.96:0/1852670942 >> v2:172.21.15.96:3300/0 conn(0x7f353415ee20 msgr2=0x7f353415fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-28T06:03:42.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:42.632+0000 7f3533400640 1 --2- 172.21.15.96:0/1852670942 >> v2:172.21.15.96:3300/0 conn(0x7f353415ee20 0x7f353415fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-28T06:03:42.635 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:42.632+0000 7f3533400640 1 --2- 172.21.15.96:0/1852670942 >> v2:172.21.15.96:3300/0 conn(0x7f353415ee20 0x7f353415fff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-28T06:03:44.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:44.225+0000 7f3530c00640 1 -- 172.21.15.96:0/1852670942 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f353415ea40 msgr2=0x7f35340cfab0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-28T06:03:44.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:44.225+0000 7f3530c00640 1 --2- 172.21.15.96:0/1852670942 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f353415ea40 0x7f35340cfab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:44.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:44.225+0000 7f3530c00640 1 -- 172.21.15.96:0/1852670942 >> v2:172.21.15.96:3300/0 conn(0x7f353415ee20 msgr2=0x7f353415fff0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-28T06:03:44.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:44.225+0000 7f3530c00640 1 --2- 172.21.15.96:0/1852670942 >> v2:172.21.15.96:3300/0 conn(0x7f353415ee20 0x7f353415fff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:44.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:44.225+0000 7f3530c00640 1 -- 172.21.15.96:0/1852670942 >> v1:172.21.15.96:6789/0 conn(0x7f35340d5190 legacy=0x7f35340d30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:03:44.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:44.225+0000 7f3530c00640 1 --2- 172.21.15.96:0/1852670942 >> v2:172.21.15.123:3300/0 conn(0x7f35100036c0 0x7f3510005b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:03:44.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:44.225+0000 7f3530c00640 1 -- 172.21.15.96:0/1852670942 --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35340cf2b0 con 0x7f3510006190 2024-05-28T06:03:44.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:44.225+0000 7f3530c00640 1 -- 172.21.15.96:0/1852670942 --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f35340cecf0 con 0x7f35100036c0 2024-05-28T06:03:44.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:44.226+0000 7f3532a00640 1 --2- 172.21.15.96:0/1852670942 >> v2:172.21.15.123:3300/0 conn(0x7f35100036c0 0x7f3510005b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:03:44.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:44.226+0000 7f3532a00640 1 -- 172.21.15.96:0/1852670942 >> v2:172.21.15.123:3300/0 conn(0x7f35100036c0 msgr2=0x7f3510005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-28T06:03:44.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:44.226+0000 7f3532a00640 1 -- 172.21.15.96:0/1852670942 >> v2:172.21.15.123:3300/0 conn(0x7f35100036c0 msgr2=0x7f3510005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-28T06:03:44.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:44.226+0000 7f3532a00640 1 --2- 172.21.15.96:0/1852670942 >> v2:172.21.15.123:3300/0 conn(0x7f35100036c0 0x7f3510005b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-28T06:03:44.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:44.226+0000 7f3532a00640 1 --2- 172.21.15.96:0/1852670942 >> v2:172.21.15.123:3300/0 conn(0x7f35100036c0 0x7f3510005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-28T06:03:44.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:44.428+0000 7f3532a00640 1 --2- 172.21.15.96:0/1852670942 >> v2:172.21.15.123:3300/0 conn(0x7f35100036c0 0x7f3510005b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:03:44.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:44.428+0000 7f3532a00640 1 -- 172.21.15.96:0/1852670942 >> v2:172.21.15.123:3300/0 conn(0x7f35100036c0 msgr2=0x7f3510005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-28T06:03:44.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:44.428+0000 7f3532a00640 1 -- 172.21.15.96:0/1852670942 >> v2:172.21.15.123:3300/0 conn(0x7f35100036c0 msgr2=0x7f3510005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-28T06:03:44.430 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:44.428+0000 7f3532a00640 1 --2- 172.21.15.96:0/1852670942 >> v2:172.21.15.123:3300/0 conn(0x7f35100036c0 0x7f3510005b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-28T06:03:44.430 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:44.428+0000 7f3532a00640 1 --2- 172.21.15.96:0/1852670942 >> v2:172.21.15.123:3300/0 conn(0x7f35100036c0 0x7f3510005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-28T06:03:44.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:44.830+0000 7f3532a00640 1 --2- 172.21.15.96:0/1852670942 >> v2:172.21.15.123:3300/0 conn(0x7f35100036c0 0x7f3510005b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:03:44.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:44.830+0000 7f3532a00640 1 -- 172.21.15.96:0/1852670942 >> v2:172.21.15.123:3300/0 conn(0x7f35100036c0 msgr2=0x7f3510005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-28T06:03:44.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:44.830+0000 7f3532a00640 1 -- 172.21.15.96:0/1852670942 >> v2:172.21.15.123:3300/0 conn(0x7f35100036c0 msgr2=0x7f3510005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-28T06:03:44.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:44.830+0000 7f3532a00640 1 --2- 172.21.15.96:0/1852670942 >> v2:172.21.15.123:3300/0 conn(0x7f35100036c0 0x7f3510005b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-28T06:03:44.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:44.830+0000 7f3532a00640 1 --2- 172.21.15.96:0/1852670942 >> v2:172.21.15.123:3300/0 conn(0x7f35100036c0 0x7f3510005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-28T06:03:45.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:45.632+0000 7f3532a00640 1 --2- 172.21.15.96:0/1852670942 >> v2:172.21.15.123:3300/0 conn(0x7f35100036c0 0x7f3510005b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:03:45.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:45.632+0000 7f3532a00640 1 -- 172.21.15.96:0/1852670942 >> v2:172.21.15.123:3300/0 conn(0x7f35100036c0 msgr2=0x7f3510005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-28T06:03:45.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:45.632+0000 7f3532a00640 1 -- 172.21.15.96:0/1852670942 >> v2:172.21.15.123:3300/0 conn(0x7f35100036c0 msgr2=0x7f3510005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-28T06:03:45.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:45.632+0000 7f3532a00640 1 --2- 172.21.15.96:0/1852670942 >> v2:172.21.15.123:3300/0 conn(0x7f35100036c0 0x7f3510005b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-28T06:03:45.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:45.632+0000 7f3532a00640 1 --2- 172.21.15.96:0/1852670942 >> v2:172.21.15.123:3300/0 conn(0x7f35100036c0 0x7f3510005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-28T06:03:46.490 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:46.488+0000 7f3532000640 1 -- 172.21.15.96:0/1852670942 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1450160919 0 0) 0x7f35340cf2b0 con 0x7f3510006190 2024-05-28T06:03:46.490 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:46.489+0000 7f3532000640 1 -- 172.21.15.96:0/1852670942 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3518003520 con 0x7f3510006190 2024-05-28T06:03:46.496 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:46.495+0000 7f3532000640 1 -- 172.21.15.96:0/1852670942 <== mon.1 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3359967059 0 0) 0x7f3518003520 con 0x7f3510006190 2024-05-28T06:03:46.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:46.495+0000 7f3532000640 1 -- 172.21.15.96:0/1852670942 >> v2:172.21.15.123:3300/0 conn(0x7f35100036c0 msgr2=0x7f3510005b40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-28T06:03:46.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:46.495+0000 7f3532000640 1 --2- 172.21.15.96:0/1852670942 >> v2:172.21.15.123:3300/0 conn(0x7f35100036c0 0x7f3510005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:46.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:46.495+0000 7f3532000640 1 -- 172.21.15.96:0/1852670942 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f351000af10 con 0x7f3510006190 2024-05-28T06:03:46.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:46.495+0000 7f3532000640 1 -- 172.21.15.96:0/1852670942 <== mon.1 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f351c001870 con 0x7f3510006190 2024-05-28T06:03:46.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:46.495+0000 7f3532000640 1 -- 172.21.15.96:0/1852670942 >> v1:172.21.15.123:6789/0 conn(0x7f3510006190 legacy=0x7f3510008630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:03:46.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:46.495+0000 7f3533400640 1 -- 172.21.15.96:0/1852670942 reap_dead start 2024-05-28T06:03:46.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:46.495+0000 7f3532000640 1 --2- 172.21.15.96:0/1852670942 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3518003a00 0x7f3518003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:03:46.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:46.495+0000 7f3532000640 1 --2- 172.21.15.96:0/1852670942 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3518004800 0x7f3518004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:03:46.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:46.495+0000 7f3532000640 1 --2- 172.21.15.96:0/1852670942 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f35180053c0 0x7f351800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:03:46.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:46.495+0000 7f3532000640 1 -- 172.21.15.96:0/1852670942 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f35340cecf0 con 0x7f35180053c0 2024-05-28T06:03:46.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:46.495+0000 7f3532000640 1 -- 172.21.15.96:0/1852670942 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f351800c280 con 0x7f3518003a00 2024-05-28T06:03:46.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:46.495+0000 7f3532000640 1 -- 172.21.15.96:0/1852670942 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f351800c480 con 0x7f3518004800 2024-05-28T06:03:46.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:46.496+0000 7f3533400640 1 --2- 172.21.15.96:0/1852670942 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f35180053c0 0x7f351800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:03:46.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:46.496+0000 7f3532a00640 1 --2- 172.21.15.96:0/1852670942 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3518003a00 0x7f3518003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:03:46.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:46.496+0000 7f352a000640 1 --2- 172.21.15.96:0/1852670942 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3518004800 0x7f3518004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:03:46.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:46.497+0000 7f3533400640 1 -- 172.21.15.96:0/1852670942 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3518004800 msgr2=0x7f3518004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:03:46.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:46.497+0000 7f3533400640 1 --2- 172.21.15.96:0/1852670942 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3518004800 0x7f3518004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:46.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:46.497+0000 7f3533400640 1 -- 172.21.15.96:0/1852670942 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3518003a00 msgr2=0x7f3518003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:03:46.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:46.497+0000 7f3533400640 1 --2- 172.21.15.96:0/1852670942 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3518003a00 0x7f3518003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:46.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:46.497+0000 7f3533400640 1 -- 172.21.15.96:0/1852670942 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3518003520 con 0x7f35180053c0 2024-05-28T06:03:46.501 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:46.499+0000 7f3533400640 1 --2- 172.21.15.96:0/1852670942 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f35180053c0 0x7f351800b9c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f35341632b0 tx=0x7f3518005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b96b4b2b807dc875 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:03:46.502 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:46.501+0000 7f3532000640 1 -- 172.21.15.96:0/1852670942 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f352c031070 con 0x7f35180053c0 2024-05-28T06:03:46.502 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:46.501+0000 7f3532000640 1 -- 172.21.15.96:0/1852670942 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f352c02b720 con 0x7f35180053c0 2024-05-28T06:03:46.502 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:46.501+0000 7f3532000640 1 -- 172.21.15.96:0/1852670942 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f352c003de0 con 0x7f35180053c0 2024-05-28T06:03:46.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:46.502+0000 7f3533e00640 1 -- 172.21.15.96:0/1852670942 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f35180053c0 msgr2=0x7f351800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:03:46.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:46.502+0000 7f3533e00640 1 --2- 172.21.15.96:0/1852670942 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f35180053c0 0x7f351800b9c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f35341632b0 tx=0x7f3518005830 comp rx=0 tx=0).stop 2024-05-28T06:03:46.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:46.502+0000 7f3533e00640 1 -- 172.21.15.96:0/1852670942 shutdown_connections 2024-05-28T06:03:46.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:46.502+0000 7f3533e00640 1 --2- 172.21.15.96:0/1852670942 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f35180053c0 0x7f351800b9c0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:46.505 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:46.502+0000 7f3533e00640 1 --2- 172.21.15.96:0/1852670942 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3518004800 0x7f3518004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:46.505 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:46.502+0000 7f3533e00640 1 --2- 172.21.15.96:0/1852670942 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3518003a00 0x7f3518003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:46.505 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:46.502+0000 7f3533e00640 1 -- 172.21.15.96:0/1852670942 >> 172.21.15.96:0/1852670942 conn(0x7f353408bfc0 msgr2=0x7f35340d4500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:03:46.505 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:46.503+0000 7f3533e00640 1 -- 172.21.15.96:0/1852670942 shutdown_connections 2024-05-28T06:03:46.505 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:46.503+0000 7f3533e00640 1 -- 172.21.15.96:0/1852670942 wait complete. 2024-05-28T06:03:46.507 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:46.506+0000 7f3533e00640 1 Processor -- start 2024-05-28T06:03:46.507 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:46.506+0000 7f3533e00640 1 -- start start 2024-05-28T06:03:46.509 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:46.508+0000 7f3533e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3518003a00 0x7f35340c8010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:03:46.509 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:46.508+0000 7f3533e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3518004800 0x7f35341f3790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:03:46.509 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:46.508+0000 7f3533e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f35180053c0 0x7f35340b70b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:03:46.510 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:46.508+0000 7f3533e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f353415f450 con 0x7f3518003a00 2024-05-28T06:03:46.510 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:46.508+0000 7f3533e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f353415f2d0 con 0x7f3518004800 2024-05-28T06:03:46.510 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:46.508+0000 7f3533e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f353415f5d0 con 0x7f35180053c0 2024-05-28T06:03:46.510 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:46.508+0000 7f3532a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3518003a00 0x7f35340c8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:03:46.512 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:46.508+0000 7f3532a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3518003a00 0x7f35340c8010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:46846/0 (socket says 172.21.15.96:46846) 2024-05-28T06:03:46.512 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:46.508+0000 7f3532a00640 1 -- 172.21.15.96:0/3687671557 learned_addr learned my addr 172.21.15.96:0/3687671557 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:03:46.512 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:46.508+0000 7f352a000640 1 --2- 172.21.15.96:0/3687671557 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3518004800 0x7f35341f3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:03:46.512 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:46.508+0000 7f3533400640 1 --2- 172.21.15.96:0/3687671557 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f35180053c0 0x7f35340b70b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:03:46.512 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:46.509+0000 7f3532a00640 1 -- 172.21.15.96:0/3687671557 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f35180053c0 msgr2=0x7f35340b70b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:03:46.512 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:46.509+0000 7f3532a00640 1 --2- 172.21.15.96:0/3687671557 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f35180053c0 0x7f35340b70b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:46.512 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:46.509+0000 7f3532a00640 1 -- 172.21.15.96:0/3687671557 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3518004800 msgr2=0x7f35341f3790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:03:46.512 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:46.509+0000 7f3532a00640 1 --2- 172.21.15.96:0/3687671557 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3518004800 0x7f35341f3790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:46.513 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:46.509+0000 7f3532a00640 1 -- 172.21.15.96:0/3687671557 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35340c8f20 con 0x7f3518003a00 2024-05-28T06:03:46.513 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:46.509+0000 7f352a000640 1 --2- 172.21.15.96:0/3687671557 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3518004800 0x7f35341f3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-28T06:03:46.513 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:46.509+0000 7f3532a00640 1 --2- 172.21.15.96:0/3687671557 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3518003a00 0x7f35340c8010 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3524017a60 tx=0x7f3524017f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:03:46.513 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:46.510+0000 7f352b400640 1 -- 172.21.15.96:0/3687671557 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3524016d80 con 0x7f3518003a00 2024-05-28T06:03:46.513 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:46.510+0000 7f352b400640 1 -- 172.21.15.96:0/3687671557 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3524004020 con 0x7f3518003a00 2024-05-28T06:03:46.513 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:46.510+0000 7f352b400640 1 -- 172.21.15.96:0/3687671557 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3524004210 con 0x7f3518003a00 2024-05-28T06:03:46.513 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:46.510+0000 7f3533e00640 1 -- 172.21.15.96:0/3687671557 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35340c8680 con 0x7f3518003a00 2024-05-28T06:03:46.513 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:46.510+0000 7f3533e00640 1 -- 172.21.15.96:0/3687671557 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35340c8c40 con 0x7f3518003a00 2024-05-28T06:03:46.513 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:46.510+0000 7f352b400640 1 -- 172.21.15.96:0/3687671557 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f352402a020 con 0x7f3518003a00 2024-05-28T06:03:46.514 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:46.511+0000 7f352b400640 1 -- 172.21.15.96:0/3687671557 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f35240043b0 con 0x7f3518003a00 2024-05-28T06:03:46.514 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:46.512+0000 7f3533e00640 1 -- 172.21.15.96:0/3687671557 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35340690f0 con 0x7f3518003a00 2024-05-28T06:03:46.517 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:46.516+0000 7f352b400640 1 -- 172.21.15.96:0/3687671557 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f3524031020 con 0x7f3518003a00 2024-05-28T06:03:46.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:46.590+0000 7f3533e00640 1 -- 172.21.15.96:0/3687671557 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f35340b6c60 con 0x7f3518003a00 2024-05-28T06:03:46.596 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:46.595+0000 7f352b400640 1 -- 172.21.15.96:0/3687671557 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7f3524020070 con 0x7f3518003a00 2024-05-28T06:03:46.596 INFO:teuthology.orchestra.run.smithi096.stderr:adjusted tunables profile to default 2024-05-28T06:03:46.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:46.596+0000 7f3533e00640 1 -- 172.21.15.96:0/3687671557 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3518003a00 msgr2=0x7f35340c8010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:03:46.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:46.596+0000 7f3533e00640 1 --2- 172.21.15.96:0/3687671557 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3518003a00 0x7f35340c8010 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3524017a60 tx=0x7f3524017f20 comp rx=0 tx=0).stop 2024-05-28T06:03:46.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:46.596+0000 7f3533e00640 1 -- 172.21.15.96:0/3687671557 shutdown_connections 2024-05-28T06:03:46.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:46.596+0000 7f3533e00640 1 --2- 172.21.15.96:0/3687671557 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f35180053c0 0x7f35340b70b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:46.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:46.596+0000 7f3533e00640 1 --2- 172.21.15.96:0/3687671557 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3518004800 0x7f35341f3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:46.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:46.596+0000 7f3533e00640 1 --2- 172.21.15.96:0/3687671557 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3518003a00 0x7f35340c8010 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:46.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:46.596+0000 7f3533e00640 1 -- 172.21.15.96:0/3687671557 >> 172.21.15.96:0/3687671557 conn(0x7f353408bfc0 msgr2=0x7f35340d2770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:03:46.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:46.597+0000 7f3533e00640 1 -- 172.21.15.96:0/3687671557 shutdown_connections 2024-05-28T06:03:46.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:46.597+0000 7f3533e00640 1 -- 172.21.15.96:0/3687671557 wait complete. 2024-05-28T06:03:46.616 INFO:tasks.ceph:check_enable_crimson: False 2024-05-28T06:03:46.617 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-05-28T06:03:46.617 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:03:46.617 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-05-28T06:03:46.648 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:03:46.646+0000 7feae3572040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-28T06:03:46.695 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:03:46.696 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-05-28T06:03:46.822 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:03:46.823 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-05-28T06:03:46.851 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:03:46.849+0000 7feae3572040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-28T06:03:46.904 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:03:46.904 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-05-28T06:03:46.917 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:03:46.916+0000 7feae3572040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-28T06:03:46.984 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-05-28T06:03:46.984 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-05-28T06:03:47.061 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-05-28T06:03:47.061 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-05-28T06:03:47.132 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-05-28T06:03:47.132 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-05-28T06:03:47.214 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:03:47.213+0000 7feae3572040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-28T06:03:47.243 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-05-28T06:03:47.244 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-05-28T06:03:47.271 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:03:47.270+0000 7feae3572040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-28T06:03:47.343 DEBUG:teuthology.orchestra.run.smithi123:> sudo ceph --cluster ceph osd new a6cb568f-6a4b-4921-942f-3595eac140a0 0 2024-05-28T06:03:47.370 INFO:tasks.ceph.mgr.y.smithi096.stderr:2024-05-28T06:03:47.369+0000 7f6101dec040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-28T06:03:47.558 INFO:tasks.ceph.mgr.y.smithi096.stderr:2024-05-28T06:03:47.557+0000 7f6101dec040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-28T06:03:47.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.568+0000 7f8adf400640 1 Processor -- start 2024-05-28T06:03:47.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.568+0000 7f8adf400640 1 -- start start 2024-05-28T06:03:47.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.569+0000 7f8adf400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f8ad80cef30 0x7f8ad80d1640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:03:47.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.569+0000 7f8adf400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f8ad80d1b80 0x7f8ad80b90a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:03:47.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.569+0000 7f8adf400640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ad80d1f60 con 0x7f8ad815e850 2024-05-28T06:03:47.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.569+0000 7f8adf400640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f8ad815ec30 con 0x7f8ad80cef30 2024-05-28T06:03:47.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.569+0000 7f8adf400640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f8ad80b8000 con 0x7f8ad80d1b80 2024-05-28T06:03:47.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.569+0000 7f8add400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f8ad80d1b80 0x7f8ad80b90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:03:47.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.570+0000 7f8add400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f8ad80d1b80 0x7f8ad80b90a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:56312/0 (socket says 172.21.15.123:56312) 2024-05-28T06:03:47.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.570+0000 7f8add400640 1 -- 172.21.15.123:0/3371709923 learned_addr learned my addr 172.21.15.123:0/3371709923 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-05-28T06:03:47.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.570+0000 7f8adca00640 1 --1- 172.21.15.123:0/3371709923 >> v1:172.21.15.123:6789/0 conn(0x7f8ad815e850 0x7f8ad80b77b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:36888/0 (socket says 172.21.15.123:36888) 2024-05-28T06:03:47.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.570+0000 7f8ad7e00640 1 --2- 172.21.15.123:0/3371709923 >> v2:172.21.15.96:3300/0 conn(0x7f8ad80cef30 0x7f8ad80d1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:03:47.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.570+0000 7f8add400640 1 -- 172.21.15.123:0/3371709923 >> v2:172.21.15.96:3300/0 conn(0x7f8ad80cef30 msgr2=0x7f8ad80d1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:03:47.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.570+0000 7f8add400640 1 --2- 172.21.15.123:0/3371709923 >> v2:172.21.15.96:3300/0 conn(0x7f8ad80cef30 0x7f8ad80d1640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:47.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.570+0000 7f8add400640 1 -- 172.21.15.123:0/3371709923 >> v1:172.21.15.123:6789/0 conn(0x7f8ad815e850 legacy=0x7f8ad80b77b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:03:47.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.570+0000 7f8add400640 1 -- 172.21.15.123:0/3371709923 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ad80b82e0 con 0x7f8ad80d1b80 2024-05-28T06:03:47.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.571+0000 7f8add400640 1 --2- 172.21.15.123:0/3371709923 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8ad80d1b80 0x7f8ad80b90a0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f8ad80cb280 tx=0x7f8ad002a6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=91fb6fbcdcde64eb server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:03:47.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.571+0000 7f8ad7400640 1 -- 172.21.15.123:0/3371709923 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ad0036070 con 0x7f8ad80d1b80 2024-05-28T06:03:47.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.571+0000 7f8ad7400640 1 -- 172.21.15.123:0/3371709923 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ad0031600 con 0x7f8ad80d1b80 2024-05-28T06:03:47.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.571+0000 7f8ad7400640 1 -- 172.21.15.123:0/3371709923 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ad00319d0 con 0x7f8ad80d1b80 2024-05-28T06:03:47.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.572+0000 7f8adf400640 1 -- 172.21.15.123:0/3371709923 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8ad80d1b80 msgr2=0x7f8ad80b90a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:03:47.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.572+0000 7f8adf400640 1 --2- 172.21.15.123:0/3371709923 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8ad80d1b80 0x7f8ad80b90a0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f8ad80cb280 tx=0x7f8ad002a6a0 comp rx=0 tx=0).stop 2024-05-28T06:03:47.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.572+0000 7f8adf400640 1 -- 172.21.15.123:0/3371709923 shutdown_connections 2024-05-28T06:03:47.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.572+0000 7f8adf400640 1 --2- 172.21.15.123:0/3371709923 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8ad80d1b80 0x7f8ad80b90a0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:47.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.572+0000 7f8adf400640 1 --2- 172.21.15.123:0/3371709923 >> v2:172.21.15.96:3300/0 conn(0x7f8ad80cef30 0x7f8ad80d1640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:47.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.572+0000 7f8adf400640 1 -- 172.21.15.123:0/3371709923 >> 172.21.15.123:0/3371709923 conn(0x7f8ad8067aa0 msgr2=0x7f8ad8067eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:03:47.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.572+0000 7f8adf400640 1 -- 172.21.15.123:0/3371709923 shutdown_connections 2024-05-28T06:03:47.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.572+0000 7f8adf400640 1 -- 172.21.15.123:0/3371709923 wait complete. 2024-05-28T06:03:47.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.574+0000 7f8adf400640 1 Processor -- start 2024-05-28T06:03:47.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.574+0000 7f8adf400640 1 -- start start 2024-05-28T06:03:47.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.575+0000 7f8adf400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ad80cef30 0x7f8ad8154e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:03:47.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.575+0000 7f8adf400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8ad80d1b80 0x7f8ad80d3e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:03:47.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.575+0000 7f8adf400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8ad815e850 0x7f8ad80d43d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:03:47.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.575+0000 7f8adf400640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f8ad80b95e0 con 0x7f8ad80cef30 2024-05-28T06:03:47.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.575+0000 7f8adf400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f8ad80b88a0 con 0x7f8ad815e850 2024-05-28T06:03:47.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.575+0000 7f8adf400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f8ad80b9760 con 0x7f8ad80d1b80 2024-05-28T06:03:47.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.575+0000 7f8add400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8ad815e850 0x7f8ad80d43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:03:47.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.575+0000 7f8ad7e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8ad80d1b80 0x7f8ad80d3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:03:47.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.575+0000 7f8add400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8ad815e850 0x7f8ad80d43d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:56316/0 (socket says 172.21.15.123:56316) 2024-05-28T06:03:47.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.575+0000 7f8add400640 1 -- 172.21.15.123:0/1442600269 learned_addr learned my addr 172.21.15.123:0/1442600269 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-05-28T06:03:47.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.575+0000 7f8adca00640 1 --2- 172.21.15.123:0/1442600269 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ad80cef30 0x7f8ad8154e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:03:47.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.575+0000 7f8ad7e00640 1 -- 172.21.15.123:0/1442600269 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8ad815e850 msgr2=0x7f8ad80d43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:03:47.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.575+0000 7f8ad7e00640 1 --2- 172.21.15.123:0/1442600269 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8ad815e850 0x7f8ad80d43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:47.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.575+0000 7f8ad7e00640 1 -- 172.21.15.123:0/1442600269 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ad80cef30 msgr2=0x7f8ad8154e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:03:47.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.575+0000 7f8ad7e00640 1 --2- 172.21.15.123:0/1442600269 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ad80cef30 0x7f8ad8154e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:47.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.575+0000 7f8ad7e00640 1 -- 172.21.15.123:0/1442600269 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ad80d4fc0 con 0x7f8ad80d1b80 2024-05-28T06:03:47.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.575+0000 7f8add400640 1 --2- 172.21.15.123:0/1442600269 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8ad815e850 0x7f8ad80d43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-28T06:03:47.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.575+0000 7f8adca00640 1 --2- 172.21.15.123:0/1442600269 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ad80cef30 0x7f8ad8154e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-28T06:03:47.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.576+0000 7f8ad7e00640 1 --2- 172.21.15.123:0/1442600269 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8ad80d1b80 0x7f8ad80d3e90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f8ac400d5f0 tx=0x7f8ac400dab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:03:47.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.576+0000 7f8ad4c00640 1 -- 172.21.15.123:0/1442600269 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ac40125f0 con 0x7f8ad80d1b80 2024-05-28T06:03:47.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.576+0000 7f8ad4c00640 1 -- 172.21.15.123:0/1442600269 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ac4019070 con 0x7f8ad80d1b80 2024-05-28T06:03:47.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.576+0000 7f8ad4c00640 1 -- 172.21.15.123:0/1442600269 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ac40129d0 con 0x7f8ad80d1b80 2024-05-28T06:03:47.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.576+0000 7f8adf400640 1 -- 172.21.15.123:0/1442600269 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ad80d4a00 con 0x7f8ad80d1b80 2024-05-28T06:03:47.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.576+0000 7f8adf400640 1 -- 172.21.15.123:0/1442600269 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ad80cb840 con 0x7f8ad80d1b80 2024-05-28T06:03:47.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.576+0000 7f8ad4c00640 1 -- 172.21.15.123:0/1442600269 <== mon.2 v2:172.21.15.123:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f8ac4012b70 con 0x7f8ad80d1b80 2024-05-28T06:03:47.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.577+0000 7f8ad4c00640 1 -- 172.21.15.123:0/1442600269 <== mon.2 v2:172.21.15.123:3301/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7f8ac4014500 con 0x7f8ad80d1b80 2024-05-28T06:03:47.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.577+0000 7f8adf400640 1 -- 172.21.15.123:0/1442600269 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ad80689e0 con 0x7f8ad80d1b80 2024-05-28T06:03:47.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.580+0000 7f8ad4c00640 1 -- 172.21.15.123:0/1442600269 <== mon.2 v2:172.21.15.123:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f8ac401e4b0 con 0x7f8ad80d1b80 2024-05-28T06:03:47.625 INFO:tasks.ceph.mgr.y.smithi096.stderr:2024-05-28T06:03:47.624+0000 7f6101dec040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-28T06:03:47.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.658+0000 7f8adf400640 1 -- 172.21.15.123:0/1442600269 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_command({"prefix": "osd new", "uuid": "a6cb568f-6a4b-4921-942f-3595eac140a0", "id": 0} v 0) -- 0x7f8ad80b7690 con 0x7f8ad80d1b80 2024-05-28T06:03:47.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.665+0000 7f8ad4c00640 1 -- 172.21.15.123:0/1442600269 <== mon.2 v2:172.21.15.123:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a6cb568f-6a4b-4921-942f-3595eac140a0", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7f8ac401e690 con 0x7f8ad80d1b80 2024-05-28T06:03:47.666 INFO:teuthology.orchestra.run.smithi123.stdout:0 2024-05-28T06:03:47.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.666+0000 7f8adf400640 1 -- 172.21.15.123:0/1442600269 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8ad80d1b80 msgr2=0x7f8ad80d3e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:03:47.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.666+0000 7f8adf400640 1 --2- 172.21.15.123:0/1442600269 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8ad80d1b80 0x7f8ad80d3e90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f8ac400d5f0 tx=0x7f8ac400dab0 comp rx=0 tx=0).stop 2024-05-28T06:03:47.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.666+0000 7f8adf400640 1 -- 172.21.15.123:0/1442600269 shutdown_connections 2024-05-28T06:03:47.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.666+0000 7f8adf400640 1 --2- 172.21.15.123:0/1442600269 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8ad815e850 0x7f8ad80d43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:47.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.666+0000 7f8adf400640 1 --2- 172.21.15.123:0/1442600269 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8ad80d1b80 0x7f8ad80d3e90 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:47.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.666+0000 7f8adf400640 1 --2- 172.21.15.123:0/1442600269 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ad80cef30 0x7f8ad8154e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:47.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.666+0000 7f8adf400640 1 -- 172.21.15.123:0/1442600269 >> 172.21.15.123:0/1442600269 conn(0x7f8ad8067aa0 msgr2=0x7f8ad80c8200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:03:47.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.667+0000 7f8adf400640 1 -- 172.21.15.123:0/1442600269 shutdown_connections 2024-05-28T06:03:47.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.667+0000 7f8adf400640 1 -- 172.21.15.123:0/1442600269 wait complete. 2024-05-28T06:03:47.685 DEBUG:teuthology.orchestra.run.smithi123:> sudo ceph --cluster ceph osd new 50ee4c07-7f99-4217-8245-964866eecd53 1 2024-05-28T06:03:47.858 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.857+0000 7f40b9e00640 1 Processor -- start 2024-05-28T06:03:47.859 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.857+0000 7f40b9e00640 1 -- start start 2024-05-28T06:03:47.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.859+0000 7f40b9e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f40b415b6f0 0x7f40b406e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:03:47.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.859+0000 7f40b9e00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40b40ade90 con 0x7f40b40725e0 2024-05-28T06:03:47.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.859+0000 7f40b9e00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40b4071e70 con 0x7f40b40d0640 2024-05-28T06:03:47.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.859+0000 7f40b9e00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f40b4071c40 con 0x7f40b415b6f0 2024-05-28T06:03:47.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.859+0000 7f40b2a00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f40b40d0640 0x7f40b40d0a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:36904/0 (socket says 172.21.15.123:36904) 2024-05-28T06:03:47.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.859+0000 7f40b2a00640 1 -- 172.21.15.123:0/3439065953 learned_addr learned my addr 172.21.15.123:0/3439065953 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-05-28T06:03:47.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.860+0000 7f40abe00640 1 --2- 172.21.15.123:0/3439065953 >> v2:172.21.15.96:3300/0 conn(0x7f40b415b6f0 0x7f40b406e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:03:47.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.860+0000 7f40b2000640 1 -- 172.21.15.123:0/3439065953 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1956328460 0 0) 0x7f40b4071e70 con 0x7f40b40d0640 2024-05-28T06:03:47.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.860+0000 7f40b2000640 1 -- 172.21.15.123:0/3439065953 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4098003520 con 0x7f40b40d0640 2024-05-28T06:03:47.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.860+0000 7f40b2000640 1 -- 172.21.15.123:0/3439065953 <== mon.1 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2344186530 0 0) 0x7f4098003520 con 0x7f40b40d0640 2024-05-28T06:03:47.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.860+0000 7f40b2000640 1 -- 172.21.15.123:0/3439065953 >> v2:172.21.15.96:3300/0 conn(0x7f40b415b6f0 msgr2=0x7f40b406e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:03:47.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.860+0000 7f40b2000640 1 --2- 172.21.15.123:0/3439065953 >> v2:172.21.15.96:3300/0 conn(0x7f40b415b6f0 0x7f40b406e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:47.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.860+0000 7f40b2000640 1 -- 172.21.15.123:0/3439065953 >> v1:172.21.15.96:6789/0 conn(0x7f40b40725e0 legacy=0x7f40b40ae650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:03:47.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.860+0000 7f40b2000640 1 -- 172.21.15.123:0/3439065953 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40b4072130 con 0x7f40b40d0640 2024-05-28T06:03:47.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.860+0000 7f40b2000640 1 -- 172.21.15.123:0/3439065953 <== mon.1 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f40a4001670 con 0x7f40b40d0640 2024-05-28T06:03:47.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.860+0000 7f40b2000640 1 -- 172.21.15.123:0/3439065953 >> v1:172.21.15.123:6789/0 conn(0x7f40b40d0640 legacy=0x7f40b40d0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:03:47.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.860+0000 7f40b2000640 1 --2- 172.21.15.123:0/3439065953 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40980039b0 0x7f4098003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:03:47.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.860+0000 7f40b2000640 1 --2- 172.21.15.123:0/3439065953 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40980043b0 0x7f4098004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:03:47.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.860+0000 7f40b2000640 1 --2- 172.21.15.123:0/3439065953 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4098005080 0x7f409800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:03:47.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.860+0000 7f40b2000640 1 -- 172.21.15.123:0/3439065953 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f409800bfb0 con 0x7f40980043b0 2024-05-28T06:03:47.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.860+0000 7f40b2000640 1 -- 172.21.15.123:0/3439065953 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f409800e370 con 0x7f40980039b0 2024-05-28T06:03:47.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.860+0000 7f40b2000640 1 -- 172.21.15.123:0/3439065953 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f409800e570 con 0x7f4098005080 2024-05-28T06:03:47.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.861+0000 7f40b2a00640 1 --2- 172.21.15.123:0/3439065953 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40980039b0 0x7f4098003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:03:47.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.861+0000 7f40b3400640 1 --2- 172.21.15.123:0/3439065953 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4098005080 0x7f409800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:03:47.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.861+0000 7f40abe00640 1 --2- 172.21.15.123:0/3439065953 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40980043b0 0x7f4098004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:03:47.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.861+0000 7f40b2a00640 1 -- 172.21.15.123:0/3439065953 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4098005080 msgr2=0x7f409800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:03:47.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.861+0000 7f40b2a00640 1 --2- 172.21.15.123:0/3439065953 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4098005080 0x7f409800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:47.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.861+0000 7f40b2a00640 1 -- 172.21.15.123:0/3439065953 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40980043b0 msgr2=0x7f4098004880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:03:47.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.861+0000 7f40b2a00640 1 --2- 172.21.15.123:0/3439065953 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40980043b0 0x7f4098004880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:47.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.861+0000 7f40b2a00640 1 -- 172.21.15.123:0/3439065953 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4098003520 con 0x7f40980039b0 2024-05-28T06:03:47.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.861+0000 7f40b3400640 1 --2- 172.21.15.123:0/3439065953 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4098005080 0x7f409800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-28T06:03:47.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.861+0000 7f40b3400640 1 -- 172.21.15.123:0/3439065953 reap_dead start 2024-05-28T06:03:47.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.862+0000 7f40b2a00640 1 --2- 172.21.15.123:0/3439065953 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40980039b0 0x7f4098003d90 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f40a400c290 tx=0x7f4098005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a508b59129706563 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:03:47.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.863+0000 7f40b2000640 1 -- 172.21.15.123:0/3439065953 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40a4007d50 con 0x7f40980039b0 2024-05-28T06:03:47.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.863+0000 7f40b2000640 1 -- 172.21.15.123:0/3439065953 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40a4005af0 con 0x7f40980039b0 2024-05-28T06:03:47.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.863+0000 7f40b2000640 1 -- 172.21.15.123:0/3439065953 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40a4007070 con 0x7f40980039b0 2024-05-28T06:03:47.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.863+0000 7f40b9e00640 1 -- 172.21.15.123:0/3439065953 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40980039b0 msgr2=0x7f4098003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:03:47.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.863+0000 7f40b9e00640 1 --2- 172.21.15.123:0/3439065953 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40980039b0 0x7f4098003d90 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f40a400c290 tx=0x7f4098005830 comp rx=0 tx=0).stop 2024-05-28T06:03:47.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.863+0000 7f40b9e00640 1 -- 172.21.15.123:0/3439065953 shutdown_connections 2024-05-28T06:03:47.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.863+0000 7f40b9e00640 1 --2- 172.21.15.123:0/3439065953 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40980039b0 0x7f4098003d90 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:47.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.863+0000 7f40b9e00640 1 -- 172.21.15.123:0/3439065953 >> 172.21.15.123:0/3439065953 conn(0x7f40b406bd20 msgr2=0x7f40b406c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:03:47.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.863+0000 7f40b9e00640 1 -- 172.21.15.123:0/3439065953 shutdown_connections 2024-05-28T06:03:47.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.864+0000 7f40b9e00640 1 -- 172.21.15.123:0/3439065953 wait complete. 2024-05-28T06:03:47.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.865+0000 7f40b9e00640 1 Processor -- start 2024-05-28T06:03:47.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.865+0000 7f40b9e00640 1 -- start start 2024-05-28T06:03:47.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.866+0000 7f40b9e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40980039b0 0x7f40b40cbd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:03:47.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.866+0000 7f40b9e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f40b415bad0 0x7f40b415d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:03:47.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.866+0000 7f40b9e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40b41594e0 0x7f40b41f71c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:03:47.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.866+0000 7f40b9e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f40b40d0ba0 con 0x7f40b41594e0 2024-05-28T06:03:47.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.866+0000 7f40b9e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f40b40d0a20 con 0x7f40980039b0 2024-05-28T06:03:47.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.866+0000 7f40b9e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f40b40d0d20 con 0x7f40b415bad0 2024-05-28T06:03:47.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.866+0000 7f40b2a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40980039b0 0x7f40b40cbd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:03:47.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.867+0000 7f40b2a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40980039b0 0x7f40b40cbd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:56328/0 (socket says 172.21.15.123:56328) 2024-05-28T06:03:47.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.867+0000 7f40b2a00640 1 -- 172.21.15.123:0/953724786 learned_addr learned my addr 172.21.15.123:0/953724786 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-05-28T06:03:47.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.867+0000 7f40abe00640 1 --2- 172.21.15.123:0/953724786 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f40b415bad0 0x7f40b415d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:03:47.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.867+0000 7f40b3400640 1 --2- 172.21.15.123:0/953724786 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40b41594e0 0x7f40b41f71c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:03:47.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.867+0000 7f40abe00640 1 -- 172.21.15.123:0/953724786 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40980039b0 msgr2=0x7f40b40cbd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:03:47.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.867+0000 7f40abe00640 1 --2- 172.21.15.123:0/953724786 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40980039b0 0x7f40b40cbd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:47.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.867+0000 7f40abe00640 1 -- 172.21.15.123:0/953724786 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40b41594e0 msgr2=0x7f40b41f71c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:03:47.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.867+0000 7f40abe00640 1 --2- 172.21.15.123:0/953724786 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40b41594e0 0x7f40b41f71c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:47.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.867+0000 7f40abe00640 1 -- 172.21.15.123:0/953724786 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40b415d820 con 0x7f40b415bad0 2024-05-28T06:03:47.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.867+0000 7f40b2a00640 1 --2- 172.21.15.123:0/953724786 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40980039b0 0x7f40b40cbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-28T06:03:47.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.867+0000 7f40abe00640 1 --2- 172.21.15.123:0/953724786 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f40b415bad0 0x7f40b415d2e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f409c011590 tx=0x7f409c011a50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:03:47.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.867+0000 7f40ab400640 1 -- 172.21.15.123:0/953724786 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f409c019070 con 0x7f40b415bad0 2024-05-28T06:03:47.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.868+0000 7f40ab400640 1 -- 172.21.15.123:0/953724786 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f409c003610 con 0x7f40b415bad0 2024-05-28T06:03:47.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.868+0000 7f40ab400640 1 -- 172.21.15.123:0/953724786 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f409c0039c0 con 0x7f40b415bad0 2024-05-28T06:03:47.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.868+0000 7f40b9e00640 1 -- 172.21.15.123:0/953724786 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40b415c240 con 0x7f40b415bad0 2024-05-28T06:03:47.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.868+0000 7f40b9e00640 1 -- 172.21.15.123:0/953724786 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f40b415c800 con 0x7f40b415bad0 2024-05-28T06:03:47.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.868+0000 7f40ab400640 1 -- 172.21.15.123:0/953724786 <== mon.2 v2:172.21.15.123:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f409c01c070 con 0x7f40b415bad0 2024-05-28T06:03:47.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.868+0000 7f40ab400640 1 -- 172.21.15.123:0/953724786 <== mon.2 v2:172.21.15.123:3301/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7f409c006510 con 0x7f40b415bad0 2024-05-28T06:03:47.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.869+0000 7f40b9e00640 1 -- 172.21.15.123:0/953724786 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40b40b5290 con 0x7f40b415bad0 2024-05-28T06:03:47.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.871+0000 7f40ab400640 1 -- 172.21.15.123:0/953724786 <== mon.2 v2:172.21.15.123:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f40b40b5290 con 0x7f40b415bad0 2024-05-28T06:03:47.923 INFO:tasks.ceph.mgr.y.smithi096.stderr:2024-05-28T06:03:47.922+0000 7f6101dec040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-28T06:03:47.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.969+0000 7f40b9e00640 1 -- 172.21.15.123:0/953724786 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_command({"prefix": "osd new", "uuid": "50ee4c07-7f99-4217-8245-964866eecd53", "id": 1} v 0) -- 0x7f40b40689e0 con 0x7f40b415bad0 2024-05-28T06:03:47.978 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.977+0000 7f40ab400640 1 -- 172.21.15.123:0/953724786 <== mon.2 v2:172.21.15.123:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "50ee4c07-7f99-4217-8245-964866eecd53", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7f409c0067e0 con 0x7f40b415bad0 2024-05-28T06:03:47.978 INFO:teuthology.orchestra.run.smithi123.stdout:1 2024-05-28T06:03:47.979 INFO:tasks.ceph.mgr.y.smithi096.stderr:2024-05-28T06:03:47.977+0000 7f6101dec040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-28T06:03:47.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.978+0000 7f40b9e00640 1 -- 172.21.15.123:0/953724786 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f40b415bad0 msgr2=0x7f40b415d2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:03:47.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.978+0000 7f40b9e00640 1 --2- 172.21.15.123:0/953724786 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f40b415bad0 0x7f40b415d2e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f409c011590 tx=0x7f409c011a50 comp rx=0 tx=0).stop 2024-05-28T06:03:47.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.979+0000 7f40b9e00640 1 -- 172.21.15.123:0/953724786 shutdown_connections 2024-05-28T06:03:47.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.979+0000 7f40b9e00640 1 --2- 172.21.15.123:0/953724786 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40b41594e0 0x7f40b41f71c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:47.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.979+0000 7f40b9e00640 1 --2- 172.21.15.123:0/953724786 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f40b415bad0 0x7f40b415d2e0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:47.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.979+0000 7f40b9e00640 1 --2- 172.21.15.123:0/953724786 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40980039b0 0x7f40b40cbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:47.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.979+0000 7f40b9e00640 1 -- 172.21.15.123:0/953724786 >> 172.21.15.123:0/953724786 conn(0x7f40b406bd20 msgr2=0x7f40b40afe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:03:47.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.979+0000 7f40b9e00640 1 -- 172.21.15.123:0/953724786 shutdown_connections 2024-05-28T06:03:47.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:47.979+0000 7f40b9e00640 1 -- 172.21.15.123:0/953724786 wait complete. 2024-05-28T06:03:47.997 DEBUG:teuthology.orchestra.run.smithi123:> sudo ceph --cluster ceph osd new 98441e44-a96f-4969-a671-947eb707a6f5 2 2024-05-28T06:03:48.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.174+0000 7f8555c00640 1 Processor -- start 2024-05-28T06:03:48.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.174+0000 7f8555c00640 1 -- start start 2024-05-28T06:03:48.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.176+0000 7f8555c00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f855015e600 0x7f85500b39f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:03:48.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.176+0000 7f8555c00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f855006edf0 con 0x7f855006c010 2024-05-28T06:03:48.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.176+0000 7f8555c00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f855006f090 con 0x7f855006fb70 2024-05-28T06:03:48.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.176+0000 7f8555c00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f855006f370 con 0x7f855015e600 2024-05-28T06:03:48.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.176+0000 7f854f400640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f855006fb70 0x7f855006b900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:36920/0 (socket says 172.21.15.123:36920) 2024-05-28T06:03:48.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.176+0000 7f854f400640 1 -- 172.21.15.123:0/973586087 learned_addr learned my addr 172.21.15.123:0/973586087 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-05-28T06:03:48.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.177+0000 7f854fe00640 1 --2- 172.21.15.123:0/973586087 >> v2:172.21.15.96:3300/0 conn(0x7f855015e600 0x7f85500b39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:03:48.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.177+0000 7f854ea00640 1 -- 172.21.15.123:0/973586087 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2763019310 0 0) 0x7f855006f090 con 0x7f855006fb70 2024-05-28T06:03:48.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.177+0000 7f854ea00640 1 -- 172.21.15.123:0/973586087 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8534003520 con 0x7f855006fb70 2024-05-28T06:03:48.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.177+0000 7f854ea00640 1 -- 172.21.15.123:0/973586087 <== mon.1 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1592865916 0 0) 0x7f8534003520 con 0x7f855006fb70 2024-05-28T06:03:48.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.177+0000 7f854ea00640 1 -- 172.21.15.123:0/973586087 >> v2:172.21.15.96:3300/0 conn(0x7f855015e600 msgr2=0x7f85500b39f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:03:48.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.177+0000 7f854ea00640 1 --2- 172.21.15.123:0/973586087 >> v2:172.21.15.96:3300/0 conn(0x7f855015e600 0x7f85500b39f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:48.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.177+0000 7f854ea00640 1 -- 172.21.15.123:0/973586087 >> v1:172.21.15.96:6789/0 conn(0x7f855006c010 legacy=0x7f85500cf100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:03:48.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.177+0000 7f854ea00640 1 -- 172.21.15.123:0/973586087 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f855006f650 con 0x7f855006fb70 2024-05-28T06:03:48.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.177+0000 7f854ea00640 1 -- 172.21.15.123:0/973586087 <== mon.1 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f853c001670 con 0x7f855006fb70 2024-05-28T06:03:48.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.177+0000 7f854ea00640 1 -- 172.21.15.123:0/973586087 >> v1:172.21.15.123:6789/0 conn(0x7f855006fb70 legacy=0x7f855006b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:03:48.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.177+0000 7f854ea00640 1 --2- 172.21.15.123:0/973586087 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8534003a00 0x7f8534003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:03:48.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.177+0000 7f854ea00640 1 --2- 172.21.15.123:0/973586087 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8534004800 0x7f8534004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:03:48.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.177+0000 7f854ea00640 1 --2- 172.21.15.123:0/973586087 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f85340051c0 0x7f853400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:03:48.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.177+0000 7f854ea00640 1 -- 172.21.15.123:0/973586087 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f855006f370 con 0x7f8534003a00 2024-05-28T06:03:48.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.177+0000 7f854ea00640 1 -- 172.21.15.123:0/973586087 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f853400c0c0 con 0x7f8534004800 2024-05-28T06:03:48.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.177+0000 7f854ea00640 1 -- 172.21.15.123:0/973586087 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f853400c2c0 con 0x7f85340051c0 2024-05-28T06:03:48.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.178+0000 7f854f400640 1 --2- 172.21.15.123:0/973586087 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8534004800 0x7f8534004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:03:48.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.178+0000 7f8555200640 1 --2- 172.21.15.123:0/973586087 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f85340051c0 0x7f853400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:03:48.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.178+0000 7f854fe00640 1 --2- 172.21.15.123:0/973586087 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8534003a00 0x7f8534003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:03:48.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.178+0000 7f854f400640 1 -- 172.21.15.123:0/973586087 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f85340051c0 msgr2=0x7f853400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:03:48.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.178+0000 7f854f400640 1 --2- 172.21.15.123:0/973586087 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f85340051c0 0x7f853400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:48.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.178+0000 7f854f400640 1 -- 172.21.15.123:0/973586087 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8534003a00 msgr2=0x7f8534003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:03:48.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.178+0000 7f854f400640 1 --2- 172.21.15.123:0/973586087 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8534003a00 0x7f8534003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:48.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.178+0000 7f854f400640 1 -- 172.21.15.123:0/973586087 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8534003520 con 0x7f8534004800 2024-05-28T06:03:48.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.178+0000 7f854fe00640 1 --2- 172.21.15.123:0/973586087 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8534003a00 0x7f8534003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-28T06:03:48.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.178+0000 7f8555200640 1 -- 172.21.15.123:0/973586087 reap_dead start 2024-05-28T06:03:48.183 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.179+0000 7f854f400640 1 --2- 172.21.15.123:0/973586087 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8534004800 0x7f8534004c80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f853c00c110 tx=0x7f8534005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f70353a61aca3e6 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:03:48.183 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.179+0000 7f854ea00640 1 -- 172.21.15.123:0/973586087 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f853c007c40 con 0x7f8534004800 2024-05-28T06:03:48.183 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.179+0000 7f854ea00640 1 -- 172.21.15.123:0/973586087 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f853c005af0 con 0x7f8534004800 2024-05-28T06:03:48.183 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.179+0000 7f854ea00640 1 -- 172.21.15.123:0/973586087 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f853c007070 con 0x7f8534004800 2024-05-28T06:03:48.183 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.180+0000 7f8555c00640 1 -- 172.21.15.123:0/973586087 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8534004800 msgr2=0x7f8534004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:03:48.183 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.180+0000 7f8555c00640 1 --2- 172.21.15.123:0/973586087 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8534004800 0x7f8534004c80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f853c00c110 tx=0x7f8534005830 comp rx=0 tx=0).stop 2024-05-28T06:03:48.183 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.180+0000 7f8555c00640 1 -- 172.21.15.123:0/973586087 shutdown_connections 2024-05-28T06:03:48.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.180+0000 7f8555c00640 1 --2- 172.21.15.123:0/973586087 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8534004800 0x7f8534004c80 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:48.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.180+0000 7f8555c00640 1 -- 172.21.15.123:0/973586087 >> 172.21.15.123:0/973586087 conn(0x7f85500b2ed0 msgr2=0x7f85500b32e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:03:48.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.180+0000 7f8555c00640 1 -- 172.21.15.123:0/973586087 shutdown_connections 2024-05-28T06:03:48.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.180+0000 7f8555c00640 1 -- 172.21.15.123:0/973586087 wait complete. 2024-05-28T06:03:48.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.182+0000 7f8555c00640 1 Processor -- start 2024-05-28T06:03:48.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.182+0000 7f8555c00640 1 -- start start 2024-05-28T06:03:48.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.183+0000 7f8555c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8534004800 0x7f85501f3980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:03:48.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.183+0000 7f8555c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f85501f5050 0x7f85500b5e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:03:48.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.183+0000 7f8555c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f85500b63a0 0x7f85500b0e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:03:48.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.183+0000 7f8555c00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f855015e9e0 con 0x7f85501f5050 2024-05-28T06:03:48.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.183+0000 7f8555c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f85500d0f50 con 0x7f85500b63a0 2024-05-28T06:03:48.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.183+0000 7f8555c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f855015eb60 con 0x7f8534004800 2024-05-28T06:03:48.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.183+0000 7f8555200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f85500b63a0 0x7f85500b0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:03:48.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.183+0000 7f854fe00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8534004800 0x7f85501f3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:03:48.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.183+0000 7f8555200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f85500b63a0 0x7f85500b0e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:56346/0 (socket says 172.21.15.123:56346) 2024-05-28T06:03:48.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.183+0000 7f8555200640 1 -- 172.21.15.123:0/2425298167 learned_addr learned my addr 172.21.15.123:0/2425298167 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-05-28T06:03:48.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.183+0000 7f854f400640 1 --2- 172.21.15.123:0/2425298167 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f85501f5050 0x7f85500b5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:03:48.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.183+0000 7f8555200640 1 -- 172.21.15.123:0/2425298167 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8534004800 msgr2=0x7f85501f3980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:03:48.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.183+0000 7f8555200640 1 --2- 172.21.15.123:0/2425298167 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8534004800 0x7f85501f3980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:48.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.183+0000 7f8555200640 1 -- 172.21.15.123:0/2425298167 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f85501f5050 msgr2=0x7f85500b5e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:03:48.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.183+0000 7f8555200640 1 --2- 172.21.15.123:0/2425298167 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f85501f5050 0x7f85500b5e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:48.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.184+0000 7f8555200640 1 -- 172.21.15.123:0/2425298167 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85500b6e70 con 0x7f85500b63a0 2024-05-28T06:03:48.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.184+0000 7f8555200640 1 --2- 172.21.15.123:0/2425298167 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f85500b63a0 0x7f85500b0e20 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f85500b0df0 tx=0x7f854800fd40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:03:48.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.184+0000 7f8547e00640 1 -- 172.21.15.123:0/2425298167 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8548003ba0 con 0x7f85500b63a0 2024-05-28T06:03:48.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.184+0000 7f8547e00640 1 -- 172.21.15.123:0/2425298167 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8548018070 con 0x7f85500b63a0 2024-05-28T06:03:48.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.184+0000 7f8547e00640 1 -- 172.21.15.123:0/2425298167 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85480050d0 con 0x7f85500b63a0 2024-05-28T06:03:48.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.184+0000 7f8555c00640 1 -- 172.21.15.123:0/2425298167 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85500b68b0 con 0x7f85500b63a0 2024-05-28T06:03:48.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.184+0000 7f8555c00640 1 -- 172.21.15.123:0/2425298167 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85500b4160 con 0x7f85500b63a0 2024-05-28T06:03:48.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.184+0000 7f8547e00640 1 -- 172.21.15.123:0/2425298167 <== mon.1 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f8548005270 con 0x7f85500b63a0 2024-05-28T06:03:48.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.185+0000 7f8547e00640 1 -- 172.21.15.123:0/2425298167 <== mon.1 v2:172.21.15.123:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7f854800dd50 con 0x7f85500b63a0 2024-05-28T06:03:48.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.185+0000 7f8555c00640 1 -- 172.21.15.123:0/2425298167 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85500690f0 con 0x7f85500b63a0 2024-05-28T06:03:48.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.188+0000 7f8547e00640 1 -- 172.21.15.123:0/2425298167 <== mon.1 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f85500b6e70 con 0x7f85500b63a0 2024-05-28T06:03:48.199 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:03:48.198+0000 7feae3572040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-28T06:03:48.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.267+0000 7f8555c00640 1 -- 172.21.15.123:0/2425298167 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd new", "uuid": "98441e44-a96f-4969-a671-947eb707a6f5", "id": 2} v 0) -- 0x7f8550072f90 con 0x7f85500b63a0 2024-05-28T06:03:48.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.273+0000 7f8547e00640 1 -- 172.21.15.123:0/2425298167 <== mon.1 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "98441e44-a96f-4969-a671-947eb707a6f5", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7f8548005560 con 0x7f85500b63a0 2024-05-28T06:03:48.274 INFO:teuthology.orchestra.run.smithi123.stdout:2 2024-05-28T06:03:48.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.274+0000 7f8555c00640 1 -- 172.21.15.123:0/2425298167 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f85500b63a0 msgr2=0x7f85500b0e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:03:48.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.274+0000 7f8555c00640 1 --2- 172.21.15.123:0/2425298167 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f85500b63a0 0x7f85500b0e20 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f85500b0df0 tx=0x7f854800fd40 comp rx=0 tx=0).stop 2024-05-28T06:03:48.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.274+0000 7f8555c00640 1 -- 172.21.15.123:0/2425298167 shutdown_connections 2024-05-28T06:03:48.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.274+0000 7f8555c00640 1 --2- 172.21.15.123:0/2425298167 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f85500b63a0 0x7f85500b0e20 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:48.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.274+0000 7f8555c00640 1 --2- 172.21.15.123:0/2425298167 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f85501f5050 0x7f85500b5e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:48.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.274+0000 7f8555c00640 1 --2- 172.21.15.123:0/2425298167 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8534004800 0x7f85501f3980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:48.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.274+0000 7f8555c00640 1 -- 172.21.15.123:0/2425298167 >> 172.21.15.123:0/2425298167 conn(0x7f85500b2ed0 msgr2=0x7f85500cf4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:03:48.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.274+0000 7f8555c00640 1 -- 172.21.15.123:0/2425298167 shutdown_connections 2024-05-28T06:03:48.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.274+0000 7f8555c00640 1 -- 172.21.15.123:0/2425298167 wait complete. 2024-05-28T06:03:48.291 DEBUG:teuthology.orchestra.run.smithi123:> sudo ceph --cluster ceph osd new bf43d98d-903b-4ba0-b8a7-040a9bfbe05d 3 2024-05-28T06:03:48.306 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:03:48.305+0000 7feae3572040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-28T06:03:48.360 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:03:48.359+0000 7feae3572040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-28T06:03:48.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.458+0000 7fcb1be00640 1 Processor -- start 2024-05-28T06:03:48.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.458+0000 7fcb1be00640 1 -- start start 2024-05-28T06:03:48.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.459+0000 7fcb1be00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fcb1c15e870 0x7fcb1c0d0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:03:48.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.459+0000 7fcb1be00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb1c15ec50 con 0x7fcb1c0cfa40 2024-05-28T06:03:48.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.460+0000 7fcb1be00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb1c0cff80 con 0x7fcb1c0d14d0 2024-05-28T06:03:48.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.460+0000 7fcb1be00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fcb1c0d2150 con 0x7fcb1c15e870 2024-05-28T06:03:48.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.460+0000 7fcb1a000640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7fcb1c0d14d0 0x7fcb1c0cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:36934/0 (socket says 172.21.15.123:36934) 2024-05-28T06:03:48.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.460+0000 7fcb1a000640 1 -- 172.21.15.123:0/2048799524 learned_addr learned my addr 172.21.15.123:0/2048799524 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-05-28T06:03:48.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.460+0000 7fcb1aa00640 1 --2- 172.21.15.123:0/2048799524 >> v2:172.21.15.96:3300/0 conn(0x7fcb1c15e870 0x7fcb1c0d0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:03:48.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.460+0000 7fcb19600640 1 -- 172.21.15.123:0/2048799524 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1630422675 0 0) 0x7fcb1c0cff80 con 0x7fcb1c0d14d0 2024-05-28T06:03:48.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.460+0000 7fcb19600640 1 -- 172.21.15.123:0/2048799524 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcb00003520 con 0x7fcb1c0d14d0 2024-05-28T06:03:48.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.460+0000 7fcb19600640 1 -- 172.21.15.123:0/2048799524 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1136634481 0 0) 0x7fcb1c15ec50 con 0x7fcb1c0cfa40 2024-05-28T06:03:48.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.460+0000 7fcb19600640 1 -- 172.21.15.123:0/2048799524 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcb1c0cff80 con 0x7fcb1c0cfa40 2024-05-28T06:03:48.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.461+0000 7fcb19600640 1 -- 172.21.15.123:0/2048799524 <== mon.1 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 408486338 0 0) 0x7fcb00003520 con 0x7fcb1c0d14d0 2024-05-28T06:03:48.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.461+0000 7fcb19600640 1 -- 172.21.15.123:0/2048799524 >> v2:172.21.15.96:3300/0 conn(0x7fcb1c15e870 msgr2=0x7fcb1c0d0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:03:48.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.461+0000 7fcb19600640 1 --2- 172.21.15.123:0/2048799524 >> v2:172.21.15.96:3300/0 conn(0x7fcb1c15e870 0x7fcb1c0d0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:48.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.461+0000 7fcb19600640 1 -- 172.21.15.123:0/2048799524 >> v1:172.21.15.96:6789/0 conn(0x7fcb1c0cfa40 legacy=0x7fcb1c0d25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:03:48.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.461+0000 7fcb19600640 1 -- 172.21.15.123:0/2048799524 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb1c0d0260 con 0x7fcb1c0d14d0 2024-05-28T06:03:48.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.461+0000 7fcb19600640 1 -- 172.21.15.123:0/2048799524 <== mon.1 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7fcb080017f0 con 0x7fcb1c0d14d0 2024-05-28T06:03:48.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.461+0000 7fcb19600640 1 -- 172.21.15.123:0/2048799524 >> v1:172.21.15.123:6789/0 conn(0x7fcb1c0d14d0 legacy=0x7fcb1c0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:03:48.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.461+0000 7fcb19600640 1 --2- 172.21.15.123:0/2048799524 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcb00003b90 0x7fcb00003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:03:48.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.461+0000 7fcb19600640 1 --2- 172.21.15.123:0/2048799524 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcb00004800 0x7fcb00004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:03:48.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.461+0000 7fcb19600640 1 --2- 172.21.15.123:0/2048799524 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcb000051c0 0x7fcb0000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:03:48.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.461+0000 7fcb19600640 1 -- 172.21.15.123:0/2048799524 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fcb1c0d2150 con 0x7fcb000051c0 2024-05-28T06:03:48.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.461+0000 7fcb19600640 1 -- 172.21.15.123:0/2048799524 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fcb0000c090 con 0x7fcb00004800 2024-05-28T06:03:48.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.461+0000 7fcb19600640 1 -- 172.21.15.123:0/2048799524 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fcb0000c290 con 0x7fcb00003b90 2024-05-28T06:03:48.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.461+0000 7fcb1aa00640 1 --2- 172.21.15.123:0/2048799524 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcb00003b90 0x7fcb00003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:03:48.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.461+0000 7fcb1a000640 1 --2- 172.21.15.123:0/2048799524 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcb00004800 0x7fcb00004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:03:48.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.461+0000 7fcb1b400640 1 --2- 172.21.15.123:0/2048799524 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcb000051c0 0x7fcb0000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:03:48.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.461+0000 7fcb1aa00640 1 -- 172.21.15.123:0/2048799524 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcb00004800 msgr2=0x7fcb00004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:03:48.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.461+0000 7fcb1aa00640 1 --2- 172.21.15.123:0/2048799524 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcb00004800 0x7fcb00004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:48.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.461+0000 7fcb1aa00640 1 -- 172.21.15.123:0/2048799524 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcb000051c0 msgr2=0x7fcb0000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:03:48.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.461+0000 7fcb1aa00640 1 --2- 172.21.15.123:0/2048799524 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcb000051c0 0x7fcb0000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:48.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.461+0000 7fcb1aa00640 1 -- 172.21.15.123:0/2048799524 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb00003520 con 0x7fcb00003b90 2024-05-28T06:03:48.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.461+0000 7fcb1b400640 1 -- 172.21.15.123:0/2048799524 reap_dead start 2024-05-28T06:03:48.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.461+0000 7fcb1a000640 1 --2- 172.21.15.123:0/2048799524 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcb00004800 0x7fcb00004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-28T06:03:48.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.462+0000 7fcb1aa00640 1 --2- 172.21.15.123:0/2048799524 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcb00003b90 0x7fcb00003f90 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fcb0c009ac0 tx=0x7fcb00005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d40590dba2bf9beb server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:03:48.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.463+0000 7fcb19600640 1 -- 172.21.15.123:0/2048799524 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb0c036070 con 0x7fcb00003b90 2024-05-28T06:03:48.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.463+0000 7fcb19600640 1 -- 172.21.15.123:0/2048799524 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb0c031400 con 0x7fcb00003b90 2024-05-28T06:03:48.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.463+0000 7fcb19600640 1 -- 172.21.15.123:0/2048799524 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb0c009690 con 0x7fcb00003b90 2024-05-28T06:03:48.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.463+0000 7fcb1be00640 1 -- 172.21.15.123:0/2048799524 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcb00003b90 msgr2=0x7fcb00003f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:03:48.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.463+0000 7fcb1be00640 1 --2- 172.21.15.123:0/2048799524 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcb00003b90 0x7fcb00003f90 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fcb0c009ac0 tx=0x7fcb00005830 comp rx=0 tx=0).stop 2024-05-28T06:03:48.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.463+0000 7fcb1be00640 1 -- 172.21.15.123:0/2048799524 shutdown_connections 2024-05-28T06:03:48.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.463+0000 7fcb1be00640 1 --2- 172.21.15.123:0/2048799524 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcb00003b90 0x7fcb00003f90 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:48.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.463+0000 7fcb1be00640 1 -- 172.21.15.123:0/2048799524 >> 172.21.15.123:0/2048799524 conn(0x7fcb1c08bfc0 msgr2=0x7fcb1c0d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:03:48.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.463+0000 7fcb1be00640 1 -- 172.21.15.123:0/2048799524 shutdown_connections 2024-05-28T06:03:48.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.463+0000 7fcb1be00640 1 -- 172.21.15.123:0/2048799524 wait complete. 2024-05-28T06:03:48.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.465+0000 7fcb1be00640 1 Processor -- start 2024-05-28T06:03:48.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.465+0000 7fcb1be00640 1 -- start start 2024-05-28T06:03:48.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.466+0000 7fcb1be00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcb00003b90 0x7fcb1c15d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:03:48.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.466+0000 7fcb1be00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcb1c1f5010 0x7fcb1c0cabb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:03:48.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.466+0000 7fcb1be00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcb1c0ca0a0 0x7fcb1c0ccb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:03:48.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.466+0000 7fcb1be00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fcb1c0d4fb0 con 0x7fcb00003b90 2024-05-28T06:03:48.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.466+0000 7fcb1be00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fcb1c0d0820 con 0x7fcb1c0ca0a0 2024-05-28T06:03:48.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.466+0000 7fcb1be00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fcb1c0d0f90 con 0x7fcb1c1f5010 2024-05-28T06:03:48.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.466+0000 7fcb1a000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcb1c1f5010 0x7fcb1c0cabb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:03:48.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.467+0000 7fcb1a000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcb1c1f5010 0x7fcb1c0cabb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.123:52714/0 (socket says 172.21.15.123:52714) 2024-05-28T06:03:48.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.467+0000 7fcb1b400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcb1c0ca0a0 0x7fcb1c0ccb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:03:48.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.467+0000 7fcb1a000640 1 -- 172.21.15.123:0/4056591221 learned_addr learned my addr 172.21.15.123:0/4056591221 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-05-28T06:03:48.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.467+0000 7fcb1aa00640 1 --2- 172.21.15.123:0/4056591221 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcb00003b90 0x7fcb1c15d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:03:48.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.467+0000 7fcb1a000640 1 -- 172.21.15.123:0/4056591221 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcb1c0ca0a0 msgr2=0x7fcb1c0ccb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:03:48.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.467+0000 7fcb1a000640 1 --2- 172.21.15.123:0/4056591221 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcb1c0ca0a0 0x7fcb1c0ccb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:48.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.467+0000 7fcb1a000640 1 -- 172.21.15.123:0/4056591221 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcb00003b90 msgr2=0x7fcb1c15d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:03:48.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.467+0000 7fcb1a000640 1 --2- 172.21.15.123:0/4056591221 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcb00003b90 0x7fcb1c15d360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:48.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.467+0000 7fcb1a000640 1 -- 172.21.15.123:0/4056591221 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb1c15d8a0 con 0x7fcb1c1f5010 2024-05-28T06:03:48.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.467+0000 7fcb1aa00640 1 --2- 172.21.15.123:0/4056591221 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcb00003b90 0x7fcb1c15d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:03:48.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.467+0000 7fcb1a000640 1 --2- 172.21.15.123:0/4056591221 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcb1c1f5010 0x7fcb1c0cabb0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fcb0800e7b0 tx=0x7fcb0800ebb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:03:48.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.467+0000 7fcb13400640 1 -- 172.21.15.123:0/4056591221 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb080070d0 con 0x7fcb1c1f5010 2024-05-28T06:03:48.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.467+0000 7fcb13400640 1 -- 172.21.15.123:0/4056591221 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb08018070 con 0x7fcb1c1f5010 2024-05-28T06:03:48.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.467+0000 7fcb13400640 1 -- 172.21.15.123:0/4056591221 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb08007490 con 0x7fcb1c1f5010 2024-05-28T06:03:48.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.467+0000 7fcb1be00640 1 -- 172.21.15.123:0/4056591221 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb1c15c5a0 con 0x7fcb1c1f5010 2024-05-28T06:03:48.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.468+0000 7fcb1be00640 1 -- 172.21.15.123:0/4056591221 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb1c15cb60 con 0x7fcb1c1f5010 2024-05-28T06:03:48.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.468+0000 7fcb13400640 1 -- 172.21.15.123:0/4056591221 <== mon.2 v2:172.21.15.123:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fcb080057a0 con 0x7fcb1c1f5010 2024-05-28T06:03:48.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.468+0000 7fcb13400640 1 -- 172.21.15.123:0/4056591221 <== mon.2 v2:172.21.15.123:3301/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7fcb0800c6c0 con 0x7fcb1c1f5010 2024-05-28T06:03:48.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.469+0000 7fcb1be00640 1 -- 172.21.15.123:0/4056591221 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb1c0689e0 con 0x7fcb1c1f5010 2024-05-28T06:03:48.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.471+0000 7fcb13400640 1 -- 172.21.15.123:0/4056591221 <== mon.2 v2:172.21.15.123:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fcb0800cb70 con 0x7fcb1c1f5010 2024-05-28T06:03:48.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.552+0000 7fcb1be00640 1 -- 172.21.15.123:0/4056591221 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_command({"prefix": "osd new", "uuid": "bf43d98d-903b-4ba0-b8a7-040a9bfbe05d", "id": 3} v 0) -- 0x7fcb1c0b1f00 con 0x7fcb1c1f5010 2024-05-28T06:03:48.558 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:03:48.557+0000 7feae3572040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-28T06:03:48.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.559+0000 7fcb13400640 1 -- 172.21.15.123:0/4056591221 <== mon.2 v2:172.21.15.123:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "bf43d98d-903b-4ba0-b8a7-040a9bfbe05d", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7fcb0800cdb0 con 0x7fcb1c1f5010 2024-05-28T06:03:48.560 INFO:teuthology.orchestra.run.smithi123.stdout:3 2024-05-28T06:03:48.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.560+0000 7fcb1be00640 1 -- 172.21.15.123:0/4056591221 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcb1c1f5010 msgr2=0x7fcb1c0cabb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:03:48.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.560+0000 7fcb1be00640 1 --2- 172.21.15.123:0/4056591221 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcb1c1f5010 0x7fcb1c0cabb0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fcb0800e7b0 tx=0x7fcb0800ebb0 comp rx=0 tx=0).stop 2024-05-28T06:03:48.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.560+0000 7fcb1be00640 1 -- 172.21.15.123:0/4056591221 shutdown_connections 2024-05-28T06:03:48.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.560+0000 7fcb1be00640 1 --2- 172.21.15.123:0/4056591221 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcb1c0ca0a0 0x7fcb1c0ccb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:48.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.560+0000 7fcb1be00640 1 --2- 172.21.15.123:0/4056591221 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcb1c1f5010 0x7fcb1c0cabb0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:48.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.560+0000 7fcb1be00640 1 --2- 172.21.15.123:0/4056591221 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcb00003b90 0x7fcb1c15d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:48.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.560+0000 7fcb1be00640 1 -- 172.21.15.123:0/4056591221 >> 172.21.15.123:0/4056591221 conn(0x7fcb1c08bfc0 msgr2=0x7fcb1c0b4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:03:48.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.561+0000 7fcb1be00640 1 -- 172.21.15.123:0/4056591221 shutdown_connections 2024-05-28T06:03:48.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.561+0000 7fcb1be00640 1 -- 172.21.15.123:0/4056591221 wait complete. 2024-05-28T06:03:48.580 DEBUG:teuthology.orchestra.run.smithi123:> sudo ceph --cluster ceph osd new 275f4739-a589-49e9-9c6d-671ab5716338 4 2024-05-28T06:03:48.627 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:03:48.623+0000 7feae3572040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-28T06:03:48.690 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:03:48.689+0000 7feae3572040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-28T06:03:48.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.801+0000 7f1737200640 1 Processor -- start 2024-05-28T06:03:48.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.801+0000 7f1737200640 1 -- start start 2024-05-28T06:03:48.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.803+0000 7f1737200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f173815e870 0x7f17380d0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:03:48.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.803+0000 7f1737200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f17380cfa40 0x7f17380d25e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:03:48.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.803+0000 7f1737200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f173815ec50 con 0x7f17380d14d0 2024-05-28T06:03:48.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.803+0000 7f1737200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f17380d2150 con 0x7f17380cfa40 2024-05-28T06:03:48.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.803+0000 7f1737200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f17380cff80 con 0x7f173815e870 2024-05-28T06:03:48.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.803+0000 7f1735e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f173815e870 0x7f17380d0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:03:48.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.803+0000 7f172fe00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f17380d14d0 0x7f17380cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:36944/0 (socket says 172.21.15.123:36944) 2024-05-28T06:03:48.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.803+0000 7f1735e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f173815e870 0x7f17380d0f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.123:52720/0 (socket says 172.21.15.123:52720) 2024-05-28T06:03:48.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.803+0000 7f172fe00640 1 -- 172.21.15.123:0/3386231668 learned_addr learned my addr 172.21.15.123:0/3386231668 (peer_addr_for_me v1:172.21.15.123:0/0) 2024-05-28T06:03:48.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.804+0000 7f1736800640 1 --2- 172.21.15.123:0/3386231668 >> v2:172.21.15.96:3300/0 conn(0x7f17380cfa40 0x7f17380d25e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:03:48.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.804+0000 7f1735e00640 1 -- 172.21.15.123:0/3386231668 >> v2:172.21.15.96:3300/0 conn(0x7f17380cfa40 msgr2=0x7f17380d25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:03:48.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.804+0000 7f1735e00640 1 --2- 172.21.15.123:0/3386231668 >> v2:172.21.15.96:3300/0 conn(0x7f17380cfa40 0x7f17380d25e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:48.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.804+0000 7f1735e00640 1 -- 172.21.15.123:0/3386231668 >> v1:172.21.15.123:6789/0 conn(0x7f17380d14d0 legacy=0x7f17380cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:03:48.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.804+0000 7f1735400640 1 -- 172.21.15.123:0/3386231668 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 107602233 0 0) 0x7f173815ec50 con 0x7f17380d14d0 2024-05-28T06:03:48.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.804+0000 7f1735e00640 1 -- 172.21.15.123:0/3386231668 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17380d0260 con 0x7f173815e870 2024-05-28T06:03:48.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.805+0000 7f1735e00640 1 --2- 172.21.15.123:0/3386231668 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f173815e870 0x7f17380d0f90 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f17280097a0 tx=0x7f172802a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=deab15d6e9a4ca0a server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:03:48.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.805+0000 7f1735400640 1 -- 172.21.15.123:0/3386231668 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1728037070 con 0x7f173815e870 2024-05-28T06:03:48.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.805+0000 7f1735400640 1 -- 172.21.15.123:0/3386231668 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1728032440 con 0x7f173815e870 2024-05-28T06:03:48.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.805+0000 7f1735400640 1 -- 172.21.15.123:0/3386231668 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1728032810 con 0x7f173815e870 2024-05-28T06:03:48.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.806+0000 7f1737200640 1 -- 172.21.15.123:0/3386231668 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f173815e870 msgr2=0x7f17380d0f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:03:48.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.806+0000 7f1737200640 1 --2- 172.21.15.123:0/3386231668 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f173815e870 0x7f17380d0f90 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f17280097a0 tx=0x7f172802a180 comp rx=0 tx=0).stop 2024-05-28T06:03:48.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.806+0000 7f1737200640 1 -- 172.21.15.123:0/3386231668 shutdown_connections 2024-05-28T06:03:48.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.806+0000 7f1737200640 1 --2- 172.21.15.123:0/3386231668 >> v2:172.21.15.96:3300/0 conn(0x7f17380cfa40 0x7f17380d25e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:48.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.806+0000 7f1737200640 1 --2- 172.21.15.123:0/3386231668 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f173815e870 0x7f17380d0f90 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:48.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.806+0000 7f1737200640 1 -- 172.21.15.123:0/3386231668 >> 172.21.15.123:0/3386231668 conn(0x7f173808bfc0 msgr2=0x7f17380d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:03:48.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.806+0000 7f1737200640 1 -- 172.21.15.123:0/3386231668 shutdown_connections 2024-05-28T06:03:48.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.806+0000 7f1737200640 1 -- 172.21.15.123:0/3386231668 wait complete. 2024-05-28T06:03:48.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.808+0000 7f1737200640 1 Processor -- start 2024-05-28T06:03:48.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.808+0000 7f1737200640 1 -- start start 2024-05-28T06:03:48.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.809+0000 7f1737200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f17380cfa40 0x7f17380ce410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:03:48.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.809+0000 7f1737200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f17380d14d0 0x7f173815c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:03:48.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.809+0000 7f1737200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f173815e870 0x7f17380b9b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:03:48.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.809+0000 7f1737200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f17380d4fb0 con 0x7f17380cfa40 2024-05-28T06:03:48.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.809+0000 7f1737200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f17380d0820 con 0x7f17380d14d0 2024-05-28T06:03:48.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.809+0000 7f1737200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f17380d2b20 con 0x7f173815e870 2024-05-28T06:03:48.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.809+0000 7f172fe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f17380d14d0 0x7f173815c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:03:48.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.809+0000 7f1736800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f173815e870 0x7f17380b9b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:03:48.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.809+0000 7f172fe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f17380d14d0 0x7f173815c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:56380/0 (socket says 172.21.15.123:56380) 2024-05-28T06:03:48.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.809+0000 7f172fe00640 1 -- 172.21.15.123:0/4278931721 learned_addr learned my addr 172.21.15.123:0/4278931721 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-05-28T06:03:48.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.809+0000 7f1735e00640 1 --2- 172.21.15.123:0/4278931721 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f17380cfa40 0x7f17380ce410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:03:48.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.809+0000 7f172fe00640 1 -- 172.21.15.123:0/4278931721 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f173815e870 msgr2=0x7f17380b9b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:03:48.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.809+0000 7f172fe00640 1 --2- 172.21.15.123:0/4278931721 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f173815e870 0x7f17380b9b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:48.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.810+0000 7f172fe00640 1 -- 172.21.15.123:0/4278931721 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f17380cfa40 msgr2=0x7f17380ce410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:03:48.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.810+0000 7f172fe00640 1 --2- 172.21.15.123:0/4278931721 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f17380cfa40 0x7f17380ce410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:48.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.810+0000 7f172fe00640 1 -- 172.21.15.123:0/4278931721 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17380b8a60 con 0x7f17380d14d0 2024-05-28T06:03:48.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.810+0000 7f172fe00640 1 --2- 172.21.15.123:0/4278931721 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f17380d14d0 0x7f173815c760 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f172000c4d0 tx=0x7f172000c990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:03:48.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.810+0000 7f172e000640 1 -- 172.21.15.123:0/4278931721 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1720007480 con 0x7f17380d14d0 2024-05-28T06:03:48.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.810+0000 7f172e000640 1 -- 172.21.15.123:0/4278931721 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17200057a0 con 0x7f17380d14d0 2024-05-28T06:03:48.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.810+0000 7f1737200640 1 -- 172.21.15.123:0/4278931721 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17380b8d40 con 0x7f17380d14d0 2024-05-28T06:03:48.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.810+0000 7f172e000640 1 -- 172.21.15.123:0/4278931721 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1720007620 con 0x7f17380d14d0 2024-05-28T06:03:48.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.810+0000 7f1737200640 1 -- 172.21.15.123:0/4278931721 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17380b9300 con 0x7f17380d14d0 2024-05-28T06:03:48.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.810+0000 7f172e000640 1 -- 172.21.15.123:0/4278931721 <== mon.1 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f1720005b50 con 0x7f17380d14d0 2024-05-28T06:03:48.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.811+0000 7f172e000640 1 -- 172.21.15.123:0/4278931721 <== mon.1 v2:172.21.15.123:3300/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7f172001b910 con 0x7f17380d14d0 2024-05-28T06:03:48.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.811+0000 7f1737200640 1 -- 172.21.15.123:0/4278931721 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17380689e0 con 0x7f17380d14d0 2024-05-28T06:03:48.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.814+0000 7f172e000640 1 -- 172.21.15.123:0/4278931721 <== mon.1 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f172001b400 con 0x7f17380d14d0 2024-05-28T06:03:48.823 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:03:48.823+0000 7feae3572040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-28T06:03:48.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.893+0000 7f1737200640 1 -- 172.21.15.123:0/4278931721 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd new", "uuid": "275f4739-a589-49e9-9c6d-671ab5716338", "id": 4} v 0) -- 0x7f17380ba500 con 0x7f17380d14d0 2024-05-28T06:03:48.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.904+0000 7f172e000640 1 -- 172.21.15.123:0/4278931721 <== mon.1 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "275f4739-a589-49e9-9c6d-671ab5716338", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7f172001b5e0 con 0x7f17380d14d0 2024-05-28T06:03:48.905 INFO:teuthology.orchestra.run.smithi123.stdout:4 2024-05-28T06:03:48.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.905+0000 7f1737200640 1 -- 172.21.15.123:0/4278931721 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f17380d14d0 msgr2=0x7f173815c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:03:48.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.905+0000 7f1737200640 1 --2- 172.21.15.123:0/4278931721 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f17380d14d0 0x7f173815c760 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f172000c4d0 tx=0x7f172000c990 comp rx=0 tx=0).stop 2024-05-28T06:03:48.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.905+0000 7f1737200640 1 -- 172.21.15.123:0/4278931721 shutdown_connections 2024-05-28T06:03:48.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.905+0000 7f1737200640 1 --2- 172.21.15.123:0/4278931721 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f173815e870 0x7f17380b9b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:48.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.905+0000 7f1737200640 1 --2- 172.21.15.123:0/4278931721 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f17380d14d0 0x7f173815c760 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:48.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.905+0000 7f1737200640 1 --2- 172.21.15.123:0/4278931721 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f17380cfa40 0x7f17380ce410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:48.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.905+0000 7f1737200640 1 -- 172.21.15.123:0/4278931721 >> 172.21.15.123:0/4278931721 conn(0x7f173808bfc0 msgr2=0x7f17380b2d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:03:48.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.905+0000 7f1737200640 1 -- 172.21.15.123:0/4278931721 shutdown_connections 2024-05-28T06:03:48.908 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:48.905+0000 7f1737200640 1 -- 172.21.15.123:0/4278931721 wait complete. 2024-05-28T06:03:48.923 DEBUG:teuthology.orchestra.run.smithi123:> sudo ceph --cluster ceph osd new be79dc9d-ad70-484b-8341-0e0bd36de687 5 2024-05-28T06:03:48.963 INFO:tasks.ceph.mgr.y.smithi096.stderr:2024-05-28T06:03:48.961+0000 7f6101dec040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-28T06:03:49.075 INFO:tasks.ceph.mgr.y.smithi096.stderr:2024-05-28T06:03:49.074+0000 7f6101dec040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-28T06:03:49.103 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.102+0000 7fbb24800640 1 Processor -- start 2024-05-28T06:03:49.103 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.102+0000 7fbb24800640 1 -- start start 2024-05-28T06:03:49.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.103+0000 7fbb24800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fbb1c15e850 0x7fbb1c0b77b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:03:49.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.103+0000 7fbb24800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fbb1c0d1d50 0x7fbb1c0b90a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:03:49.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.103+0000 7fbb24800640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb1c0b8000 con 0x7fbb1c0cef30 2024-05-28T06:03:49.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.103+0000 7fbb24800640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fbb1c15ec30 con 0x7fbb1c0d1d50 2024-05-28T06:03:49.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.103+0000 7fbb24800640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fbb1c0b82e0 con 0x7fbb1c15e850 2024-05-28T06:03:49.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.104+0000 7fbb21e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fbb1c15e850 0x7fbb1c0b77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:03:49.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.104+0000 7fbb21e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fbb1c15e850 0x7fbb1c0b77b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:56390/0 (socket says 172.21.15.123:56390) 2024-05-28T06:03:49.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.104+0000 7fbb21e00640 1 -- 172.21.15.123:0/3756081043 learned_addr learned my addr 172.21.15.123:0/3756081043 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-05-28T06:03:49.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.104+0000 7fbb22800640 1 --2- 172.21.15.123:0/3756081043 >> v2:172.21.15.96:3300/0 conn(0x7fbb1c0d1d50 0x7fbb1c0b90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:03:49.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.104+0000 7fbb21e00640 1 -- 172.21.15.123:0/3756081043 >> v2:172.21.15.96:3300/0 conn(0x7fbb1c0d1d50 msgr2=0x7fbb1c0b90a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:03:49.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.104+0000 7fbb21e00640 1 --2- 172.21.15.123:0/3756081043 >> v2:172.21.15.96:3300/0 conn(0x7fbb1c0d1d50 0x7fbb1c0b90a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:49.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.104+0000 7fbb21e00640 1 -- 172.21.15.123:0/3756081043 >> v1:172.21.15.123:6789/0 conn(0x7fbb1c0cef30 legacy=0x7fbb1c0d1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:03:49.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.104+0000 7fbb21e00640 1 -- 172.21.15.123:0/3756081043 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb1c0b85c0 con 0x7fbb1c15e850 2024-05-28T06:03:49.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.104+0000 7fbb20a00640 1 -- 172.21.15.123:0/3756081043 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1475902731 0 0) 0x7fbb1c0b8000 con 0x7fbb1c0cef30 2024-05-28T06:03:49.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.105+0000 7fbb21e00640 1 --2- 172.21.15.123:0/3756081043 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbb1c15e850 0x7fbb1c0b77b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fbb1c06dba0 tx=0x7fbb1002a1e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cfcf1acc0c97889e server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:03:49.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.106+0000 7fbb20a00640 1 -- 172.21.15.123:0/3756081043 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb10037070 con 0x7fbb1c15e850 2024-05-28T06:03:49.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.106+0000 7fbb20a00640 1 -- 172.21.15.123:0/3756081043 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb10032440 con 0x7fbb1c15e850 2024-05-28T06:03:49.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.106+0000 7fbb20a00640 1 -- 172.21.15.123:0/3756081043 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb10032810 con 0x7fbb1c15e850 2024-05-28T06:03:49.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.106+0000 7fbb24800640 1 -- 172.21.15.123:0/3756081043 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbb1c15e850 msgr2=0x7fbb1c0b77b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:03:49.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.106+0000 7fbb24800640 1 --2- 172.21.15.123:0/3756081043 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbb1c15e850 0x7fbb1c0b77b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fbb1c06dba0 tx=0x7fbb1002a1e0 comp rx=0 tx=0).stop 2024-05-28T06:03:49.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.106+0000 7fbb24800640 1 -- 172.21.15.123:0/3756081043 shutdown_connections 2024-05-28T06:03:49.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.106+0000 7fbb24800640 1 --2- 172.21.15.123:0/3756081043 >> v2:172.21.15.96:3300/0 conn(0x7fbb1c0d1d50 0x7fbb1c0b90a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:49.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.107+0000 7fbb24800640 1 --2- 172.21.15.123:0/3756081043 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbb1c15e850 0x7fbb1c0b77b0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:49.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.107+0000 7fbb24800640 1 -- 172.21.15.123:0/3756081043 >> 172.21.15.123:0/3756081043 conn(0x7fbb1c067aa0 msgr2=0x7fbb1c067eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:03:49.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.107+0000 7fbb24800640 1 -- 172.21.15.123:0/3756081043 shutdown_connections 2024-05-28T06:03:49.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.107+0000 7fbb24800640 1 -- 172.21.15.123:0/3756081043 wait complete. 2024-05-28T06:03:49.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.108+0000 7fbb24800640 1 Processor -- start 2024-05-28T06:03:49.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.108+0000 7fbb24800640 1 -- start start 2024-05-28T06:03:49.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.109+0000 7fbb24800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbb1c0cef30 0x7fbb1c1f17d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:03:49.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.109+0000 7fbb24800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbb1c0d1d50 0x7fbb1c0cd200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:03:49.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.109+0000 7fbb24800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbb1c15e850 0x7fbb1c0abb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:03:49.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.109+0000 7fbb24800640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fbb1c0b95e0 con 0x7fbb1c0cef30 2024-05-28T06:03:49.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.109+0000 7fbb24800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fbb1c0b7cf0 con 0x7fbb1c15e850 2024-05-28T06:03:49.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.109+0000 7fbb24800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fbb1c0b9760 con 0x7fbb1c0d1d50 2024-05-28T06:03:49.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.110+0000 7fbb22800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbb1c15e850 0x7fbb1c0abb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:03:49.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.110+0000 7fbb22800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbb1c15e850 0x7fbb1c0abb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:56392/0 (socket says 172.21.15.123:56392) 2024-05-28T06:03:49.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.110+0000 7fbb22800640 1 -- 172.21.15.123:0/3429501963 learned_addr learned my addr 172.21.15.123:0/3429501963 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-05-28T06:03:49.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.110+0000 7fbb21400640 1 --2- 172.21.15.123:0/3429501963 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbb1c0d1d50 0x7fbb1c0cd200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:03:49.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.110+0000 7fbb21e00640 1 --2- 172.21.15.123:0/3429501963 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbb1c0cef30 0x7fbb1c1f17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:03:49.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.110+0000 7fbb22800640 1 -- 172.21.15.123:0/3429501963 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbb1c0d1d50 msgr2=0x7fbb1c0cd200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:03:49.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.110+0000 7fbb22800640 1 --2- 172.21.15.123:0/3429501963 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbb1c0d1d50 0x7fbb1c0cd200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:49.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.110+0000 7fbb22800640 1 -- 172.21.15.123:0/3429501963 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbb1c0cef30 msgr2=0x7fbb1c1f17d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:03:49.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.110+0000 7fbb22800640 1 --2- 172.21.15.123:0/3429501963 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbb1c0cef30 0x7fbb1c1f17d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:49.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.110+0000 7fbb22800640 1 -- 172.21.15.123:0/3429501963 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb1c0ac0d0 con 0x7fbb1c15e850 2024-05-28T06:03:49.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.110+0000 7fbb21400640 1 --2- 172.21.15.123:0/3429501963 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbb1c0d1d50 0x7fbb1c0cd200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-28T06:03:49.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.110+0000 7fbb22800640 1 --2- 172.21.15.123:0/3429501963 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbb1c15e850 0x7fbb1c0abb90 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fbb18007e40 tx=0x7fbb1800e4b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:03:49.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.111+0000 7fbb16000640 1 -- 172.21.15.123:0/3429501963 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb18014620 con 0x7fbb1c15e850 2024-05-28T06:03:49.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.111+0000 7fbb16000640 1 -- 172.21.15.123:0/3429501963 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb18019070 con 0x7fbb1c15e850 2024-05-28T06:03:49.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.111+0000 7fbb16000640 1 -- 172.21.15.123:0/3429501963 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb18014a20 con 0x7fbb1c15e850 2024-05-28T06:03:49.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.111+0000 7fbb24800640 1 -- 172.21.15.123:0/3429501963 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb1c0cd900 con 0x7fbb1c15e850 2024-05-28T06:03:49.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.111+0000 7fbb24800640 1 -- 172.21.15.123:0/3429501963 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb1c0cdec0 con 0x7fbb1c15e850 2024-05-28T06:03:49.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.111+0000 7fbb16000640 1 -- 172.21.15.123:0/3429501963 <== mon.1 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fbb18014bc0 con 0x7fbb1c15e850 2024-05-28T06:03:49.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.111+0000 7fbb16000640 1 -- 172.21.15.123:0/3429501963 <== mon.1 v2:172.21.15.123:3300/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7fbb1801e9e0 con 0x7fbb1c15e850 2024-05-28T06:03:49.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.112+0000 7fbb24800640 1 -- 172.21.15.123:0/3429501963 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb1c0689e0 con 0x7fbb1c15e850 2024-05-28T06:03:49.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.114+0000 7fbb16000640 1 -- 172.21.15.123:0/3429501963 <== mon.1 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fbb1c0ac0d0 con 0x7fbb1c15e850 2024-05-28T06:03:49.131 INFO:tasks.ceph.mgr.y.smithi096.stderr:2024-05-28T06:03:49.130+0000 7f6101dec040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-28T06:03:49.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.192+0000 7fbb24800640 1 -- 172.21.15.123:0/3429501963 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd new", "uuid": "be79dc9d-ad70-484b-8341-0e0bd36de687", "id": 5} v 0) -- 0x7fbb1c074d10 con 0x7fbb1c15e850 2024-05-28T06:03:49.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.203+0000 7fbb16000640 1 -- 172.21.15.123:0/3429501963 <== mon.1 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "be79dc9d-ad70-484b-8341-0e0bd36de687", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7fbb1801e460 con 0x7fbb1c15e850 2024-05-28T06:03:49.204 INFO:teuthology.orchestra.run.smithi123.stdout:5 2024-05-28T06:03:49.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.204+0000 7fbb24800640 1 -- 172.21.15.123:0/3429501963 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbb1c15e850 msgr2=0x7fbb1c0abb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:03:49.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.204+0000 7fbb24800640 1 --2- 172.21.15.123:0/3429501963 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbb1c15e850 0x7fbb1c0abb90 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fbb18007e40 tx=0x7fbb1800e4b0 comp rx=0 tx=0).stop 2024-05-28T06:03:49.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.204+0000 7fbb24800640 1 -- 172.21.15.123:0/3429501963 shutdown_connections 2024-05-28T06:03:49.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.204+0000 7fbb24800640 1 --2- 172.21.15.123:0/3429501963 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbb1c15e850 0x7fbb1c0abb90 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:49.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.204+0000 7fbb24800640 1 --2- 172.21.15.123:0/3429501963 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbb1c0d1d50 0x7fbb1c0cd200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:49.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.204+0000 7fbb24800640 1 --2- 172.21.15.123:0/3429501963 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbb1c0cef30 0x7fbb1c1f17d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:49.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.204+0000 7fbb24800640 1 -- 172.21.15.123:0/3429501963 >> 172.21.15.123:0/3429501963 conn(0x7fbb1c067aa0 msgr2=0x7fbb1c15f640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:03:49.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.205+0000 7fbb24800640 1 -- 172.21.15.123:0/3429501963 shutdown_connections 2024-05-28T06:03:49.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.205+0000 7fbb24800640 1 -- 172.21.15.123:0/3429501963 wait complete. 2024-05-28T06:03:49.222 DEBUG:teuthology.orchestra.run.smithi123:> sudo ceph --cluster ceph osd new 6397e1cd-f914-4f53-87b7-c95a3dcb16f0 6 2024-05-28T06:03:49.299 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:03:49.298+0000 7feae3572040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-28T06:03:49.341 INFO:tasks.ceph.mgr.y.smithi096.stderr:2024-05-28T06:03:49.340+0000 7f6101dec040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-28T06:03:49.400 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.399+0000 7fb893e00640 1 Processor -- start 2024-05-28T06:03:49.400 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.399+0000 7fb893e00640 1 -- start start 2024-05-28T06:03:49.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.400+0000 7fb893e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fb89415ea40 0x7fb8940cfab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:03:49.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.401+0000 7fb893e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fb89415ee20 0x7fb89415fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:03:49.404 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.401+0000 7fb893e00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8940d37b0 con 0x7fb8940d5190 2024-05-28T06:03:49.404 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.401+0000 7fb893e00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fb8940cecf0 con 0x7fb89415ea40 2024-05-28T06:03:49.404 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.401+0000 7fb893e00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fb8940cefd0 con 0x7fb89415ee20 2024-05-28T06:03:49.404 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.401+0000 7fb893400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fb89415ee20 0x7fb89415fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:03:49.404 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.401+0000 7fb893400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fb89415ee20 0x7fb89415fff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:56404/0 (socket says 172.21.15.123:56404) 2024-05-28T06:03:49.404 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.401+0000 7fb893400640 1 -- 172.21.15.123:0/1927631325 learned_addr learned my addr 172.21.15.123:0/1927631325 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-05-28T06:03:49.404 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.401+0000 7fb892a00640 1 --2- 172.21.15.123:0/1927631325 >> v2:172.21.15.96:3300/0 conn(0x7fb89415ea40 0x7fb8940cfab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:03:49.404 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.401+0000 7fb893400640 1 -- 172.21.15.123:0/1927631325 >> v2:172.21.15.96:3300/0 conn(0x7fb89415ea40 msgr2=0x7fb8940cfab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:03:49.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.401+0000 7fb893400640 1 --2- 172.21.15.123:0/1927631325 >> v2:172.21.15.96:3300/0 conn(0x7fb89415ea40 0x7fb8940cfab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:49.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.401+0000 7fb893400640 1 -- 172.21.15.123:0/1927631325 >> v1:172.21.15.96:6789/0 conn(0x7fb8940d5190 legacy=0x7fb8940d30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:03:49.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.401+0000 7fb893400640 1 -- 172.21.15.123:0/1927631325 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8940cf2b0 con 0x7fb89415ee20 2024-05-28T06:03:49.406 INFO:tasks.ceph.mgr.y.smithi096.stderr:2024-05-28T06:03:49.401+0000 7f6101dec040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-28T06:03:49.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.403+0000 7fb893400640 1 --2- 172.21.15.123:0/1927631325 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb89415ee20 0x7fb89415fff0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fb894162140 tx=0x7fb88c02a6f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b84342df2eac7d4e server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:03:49.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.403+0000 7fb891600640 1 -- 172.21.15.123:0/1927631325 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb88c036070 con 0x7fb89415ee20 2024-05-28T06:03:49.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.403+0000 7fb891600640 1 -- 172.21.15.123:0/1927631325 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb88c031600 con 0x7fb89415ee20 2024-05-28T06:03:49.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.403+0000 7fb891600640 1 -- 172.21.15.123:0/1927631325 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb88c0319d0 con 0x7fb89415ee20 2024-05-28T06:03:49.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.404+0000 7fb893e00640 1 -- 172.21.15.123:0/1927631325 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb89415ee20 msgr2=0x7fb89415fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:03:49.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.404+0000 7fb893e00640 1 --2- 172.21.15.123:0/1927631325 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb89415ee20 0x7fb89415fff0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fb894162140 tx=0x7fb88c02a6f0 comp rx=0 tx=0).stop 2024-05-28T06:03:49.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.404+0000 7fb893e00640 1 -- 172.21.15.123:0/1927631325 shutdown_connections 2024-05-28T06:03:49.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.404+0000 7fb893e00640 1 --2- 172.21.15.123:0/1927631325 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb89415ee20 0x7fb89415fff0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:49.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.404+0000 7fb893e00640 1 --2- 172.21.15.123:0/1927631325 >> v2:172.21.15.96:3300/0 conn(0x7fb89415ea40 0x7fb8940cfab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:49.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.404+0000 7fb893e00640 1 -- 172.21.15.123:0/1927631325 >> 172.21.15.123:0/1927631325 conn(0x7fb89408bfc0 msgr2=0x7fb8940d4500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:03:49.409 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.404+0000 7fb893e00640 1 -- 172.21.15.123:0/1927631325 shutdown_connections 2024-05-28T06:03:49.409 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.404+0000 7fb893e00640 1 -- 172.21.15.123:0/1927631325 wait complete. 2024-05-28T06:03:49.409 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.406+0000 7fb893e00640 1 Processor -- start 2024-05-28T06:03:49.409 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.406+0000 7fb893e00640 1 -- start start 2024-05-28T06:03:49.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.407+0000 7fb893e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb8940d5190 0x7fb894157580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:03:49.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.407+0000 7fb893e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb89415ea40 0x7fb8940ca040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:03:49.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.408+0000 7fb893e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb89415ee20 0x7fb8940cab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:03:49.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.408+0000 7fb893e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb89415f2d0 con 0x7fb8940d5190 2024-05-28T06:03:49.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.408+0000 7fb893e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fb8940cfff0 con 0x7fb89415ea40 2024-05-28T06:03:49.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.408+0000 7fb893e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fb89415f450 con 0x7fb89415ee20 2024-05-28T06:03:49.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.408+0000 7fb892000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb89415ea40 0x7fb8940ca040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:03:49.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.408+0000 7fb893400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb89415ee20 0x7fb8940cab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:03:49.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.408+0000 7fb892000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb89415ea40 0x7fb8940ca040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:56418/0 (socket says 172.21.15.123:56418) 2024-05-28T06:03:49.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.408+0000 7fb893400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb89415ee20 0x7fb8940cab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.123:52728/0 (socket says 172.21.15.123:52728) 2024-05-28T06:03:49.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.408+0000 7fb892000640 1 -- 172.21.15.123:0/2288753672 learned_addr learned my addr 172.21.15.123:0/2288753672 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-05-28T06:03:49.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.408+0000 7fb892a00640 1 --2- 172.21.15.123:0/2288753672 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb8940d5190 0x7fb894157580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:03:49.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.408+0000 7fb893400640 1 -- 172.21.15.123:0/2288753672 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb89415ea40 msgr2=0x7fb8940ca040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:03:49.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.408+0000 7fb893400640 1 --2- 172.21.15.123:0/2288753672 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb89415ea40 0x7fb8940ca040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:49.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.408+0000 7fb893400640 1 -- 172.21.15.123:0/2288753672 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb8940d5190 msgr2=0x7fb894157580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:03:49.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.408+0000 7fb893400640 1 --2- 172.21.15.123:0/2288753672 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb8940d5190 0x7fb894157580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:49.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.408+0000 7fb893400640 1 -- 172.21.15.123:0/2288753672 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8940c8fa0 con 0x7fb89415ee20 2024-05-28T06:03:49.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.409+0000 7fb893400640 1 --2- 172.21.15.123:0/2288753672 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb89415ee20 0x7fb8940cab40 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fb88c030000 tx=0x7fb88c004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:03:49.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.409+0000 7fb88b400640 1 -- 172.21.15.123:0/2288753672 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb88c036050 con 0x7fb89415ee20 2024-05-28T06:03:49.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.409+0000 7fb88b400640 1 -- 172.21.15.123:0/2288753672 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb88c042070 con 0x7fb89415ee20 2024-05-28T06:03:49.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.409+0000 7fb88b400640 1 -- 172.21.15.123:0/2288753672 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb88c009510 con 0x7fb89415ee20 2024-05-28T06:03:49.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.409+0000 7fb893e00640 1 -- 172.21.15.123:0/2288753672 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8940c9280 con 0x7fb89415ee20 2024-05-28T06:03:49.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.409+0000 7fb893e00640 1 -- 172.21.15.123:0/2288753672 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8940c9840 con 0x7fb89415ee20 2024-05-28T06:03:49.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.409+0000 7fb88b400640 1 -- 172.21.15.123:0/2288753672 <== mon.2 v2:172.21.15.123:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fb88c007bd0 con 0x7fb89415ee20 2024-05-28T06:03:49.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.410+0000 7fb88b400640 1 -- 172.21.15.123:0/2288753672 <== mon.2 v2:172.21.15.123:3301/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7fb88c047b40 con 0x7fb89415ee20 2024-05-28T06:03:49.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.410+0000 7fb893e00640 1 -- 172.21.15.123:0/2288753672 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8940690f0 con 0x7fb89415ee20 2024-05-28T06:03:49.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.414+0000 7fb88b400640 1 -- 172.21.15.123:0/2288753672 <== mon.2 v2:172.21.15.123:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fb88c047460 con 0x7fb89415ee20 2024-05-28T06:03:49.471 INFO:tasks.ceph.mgr.y.smithi096.stderr:2024-05-28T06:03:49.469+0000 7f6101dec040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-28T06:03:49.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.494+0000 7fb893e00640 1 -- 172.21.15.123:0/2288753672 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_command({"prefix": "osd new", "uuid": "6397e1cd-f914-4f53-87b7-c95a3dcb16f0", "id": 6} v 0) -- 0x7fb8940b6c60 con 0x7fb89415ee20 2024-05-28T06:03:49.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.505+0000 7fb88b400640 1 -- 172.21.15.123:0/2288753672 <== mon.2 v2:172.21.15.123:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6397e1cd-f914-4f53-87b7-c95a3dcb16f0", "id": 6}]=0 v9) ==== 112+0+2 (secure 0 0 0) 0x7fb88c047640 con 0x7fb89415ee20 2024-05-28T06:03:49.506 INFO:teuthology.orchestra.run.smithi123.stdout:6 2024-05-28T06:03:49.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.507+0000 7fb893e00640 1 -- 172.21.15.123:0/2288753672 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb89415ee20 msgr2=0x7fb8940cab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:03:49.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.507+0000 7fb893e00640 1 --2- 172.21.15.123:0/2288753672 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb89415ee20 0x7fb8940cab40 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fb88c030000 tx=0x7fb88c004a40 comp rx=0 tx=0).stop 2024-05-28T06:03:49.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.507+0000 7fb893e00640 1 -- 172.21.15.123:0/2288753672 shutdown_connections 2024-05-28T06:03:49.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.507+0000 7fb893e00640 1 --2- 172.21.15.123:0/2288753672 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb89415ee20 0x7fb8940cab40 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:49.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.507+0000 7fb893e00640 1 --2- 172.21.15.123:0/2288753672 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb89415ea40 0x7fb8940ca040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:49.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.507+0000 7fb893e00640 1 --2- 172.21.15.123:0/2288753672 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb8940d5190 0x7fb894157580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:49.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.507+0000 7fb893e00640 1 -- 172.21.15.123:0/2288753672 >> 172.21.15.123:0/2288753672 conn(0x7fb89408bfc0 msgr2=0x7fb8940b7790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:03:49.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.507+0000 7fb893e00640 1 -- 172.21.15.123:0/2288753672 shutdown_connections 2024-05-28T06:03:49.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.507+0000 7fb893e00640 1 -- 172.21.15.123:0/2288753672 wait complete. 2024-05-28T06:03:49.524 DEBUG:teuthology.orchestra.run.smithi123:> sudo ceph --cluster ceph osd new be2a4947-16f1-4d46-b732-d511a2c19fae 7 2024-05-28T06:03:49.606 INFO:tasks.ceph.mgr.y.smithi096.stderr:2024-05-28T06:03:49.604+0000 7f6101dec040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-28T06:03:49.614 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:03:49.613+0000 7feae3572040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-28T06:03:49.676 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:03:49.675+0000 7feae3572040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-28T06:03:49.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.691+0000 7f6ef4a00640 1 Processor -- start 2024-05-28T06:03:49.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.691+0000 7f6ef4a00640 1 -- start start 2024-05-28T06:03:49.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.693+0000 7f6ef4a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f6ef015e870 0x7f6ef00d0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:03:49.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.693+0000 7f6ef4a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6ef00cfa40 0x7f6ef00d25e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:03:49.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.693+0000 7f6ef4a00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ef015ec50 con 0x7f6ef00d14d0 2024-05-28T06:03:49.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.693+0000 7f6ef4a00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f6ef00d2150 con 0x7f6ef015e870 2024-05-28T06:03:49.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.693+0000 7f6ef4a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f6ef00cff80 con 0x7f6ef00cfa40 2024-05-28T06:03:49.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.693+0000 7f6eef200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6ef00cfa40 0x7f6ef00d25e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:03:49.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.693+0000 7f6eef200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6ef00cfa40 0x7f6ef00d25e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.123:52738/0 (socket says 172.21.15.123:52738) 2024-05-28T06:03:49.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.693+0000 7f6eef200640 1 -- 172.21.15.123:0/2755230319 learned_addr learned my addr 172.21.15.123:0/2755230319 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-05-28T06:03:49.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.693+0000 7f6eee800640 1 --2- 172.21.15.123:0/2755230319 >> v2:172.21.15.96:3300/0 conn(0x7f6ef015e870 0x7f6ef00d0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:03:49.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.694+0000 7f6eef200640 1 -- 172.21.15.123:0/2755230319 >> v2:172.21.15.96:3300/0 conn(0x7f6ef015e870 msgr2=0x7f6ef00d0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:03:49.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.694+0000 7f6eef200640 1 --2- 172.21.15.123:0/2755230319 >> v2:172.21.15.96:3300/0 conn(0x7f6ef015e870 0x7f6ef00d0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:49.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.694+0000 7f6eef200640 1 -- 172.21.15.123:0/2755230319 >> v1:172.21.15.96:6789/0 conn(0x7f6ef00d14d0 legacy=0x7f6ef00cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:03:49.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.694+0000 7f6eef200640 1 -- 172.21.15.123:0/2755230319 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ef00d0260 con 0x7f6ef00cfa40 2024-05-28T06:03:49.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.695+0000 7f6eef200640 1 --2- 172.21.15.123:0/2755230319 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6ef00cfa40 0x7f6ef00d25e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f6ee8009e50 tx=0x7f6ee802a6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=602d94ec20637b1e server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:03:49.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.695+0000 7f6eede00640 1 -- 172.21.15.123:0/2755230319 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ee8036070 con 0x7f6ef00cfa40 2024-05-28T06:03:49.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.695+0000 7f6eede00640 1 -- 172.21.15.123:0/2755230319 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ee8031600 con 0x7f6ef00cfa40 2024-05-28T06:03:49.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.695+0000 7f6eede00640 1 -- 172.21.15.123:0/2755230319 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ee80319d0 con 0x7f6ef00cfa40 2024-05-28T06:03:49.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.696+0000 7f6ef4a00640 1 -- 172.21.15.123:0/2755230319 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6ef00cfa40 msgr2=0x7f6ef00d25e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:03:49.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.696+0000 7f6ef4a00640 1 --2- 172.21.15.123:0/2755230319 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6ef00cfa40 0x7f6ef00d25e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f6ee8009e50 tx=0x7f6ee802a6c0 comp rx=0 tx=0).stop 2024-05-28T06:03:49.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.696+0000 7f6ef4a00640 1 -- 172.21.15.123:0/2755230319 shutdown_connections 2024-05-28T06:03:49.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.696+0000 7f6ef4a00640 1 --2- 172.21.15.123:0/2755230319 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6ef00cfa40 0x7f6ef00d25e0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:49.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.696+0000 7f6ef4a00640 1 --2- 172.21.15.123:0/2755230319 >> v2:172.21.15.96:3300/0 conn(0x7f6ef015e870 0x7f6ef00d0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:49.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.696+0000 7f6ef4a00640 1 -- 172.21.15.123:0/2755230319 >> 172.21.15.123:0/2755230319 conn(0x7f6ef008bfc0 msgr2=0x7f6ef00d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:03:49.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.696+0000 7f6ef4a00640 1 -- 172.21.15.123:0/2755230319 shutdown_connections 2024-05-28T06:03:49.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.696+0000 7f6ef4a00640 1 -- 172.21.15.123:0/2755230319 wait complete. 2024-05-28T06:03:49.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.698+0000 7f6ef4a00640 1 Processor -- start 2024-05-28T06:03:49.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.698+0000 7f6ef4a00640 1 -- start start 2024-05-28T06:03:49.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.699+0000 7f6ef4a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6ef00cfa40 0x7f6ef014faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:03:49.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.699+0000 7f6ef4a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6ef00d14d0 0x7f6ef01f47f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:03:49.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.699+0000 7f6ef4a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6ef015e870 0x7f6ef00b5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:03:49.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.699+0000 7f6ef4a00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f6ef00d4fb0 con 0x7f6ef015e870 2024-05-28T06:03:49.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.699+0000 7f6ef4a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f6ef00d0820 con 0x7f6ef00cfa40 2024-05-28T06:03:49.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.699+0000 7f6ef4a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f6ef00d2b20 con 0x7f6ef00d14d0 2024-05-28T06:03:49.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.699+0000 7f6eee800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6ef00cfa40 0x7f6ef014faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:03:49.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.699+0000 7f6ee7e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6ef00d14d0 0x7f6ef01f47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:03:49.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.699+0000 7f6eee800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6ef00cfa40 0x7f6ef014faf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:56432/0 (socket says 172.21.15.123:56432) 2024-05-28T06:03:49.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.699+0000 7f6ee7e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6ef00d14d0 0x7f6ef01f47f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.123:52752/0 (socket says 172.21.15.123:52752) 2024-05-28T06:03:49.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.699+0000 7f6eee800640 1 -- 172.21.15.123:0/3002844715 learned_addr learned my addr 172.21.15.123:0/3002844715 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-05-28T06:03:49.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.700+0000 7f6eef200640 1 --2- 172.21.15.123:0/3002844715 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6ef015e870 0x7f6ef00b5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:03:49.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.700+0000 7f6eee800640 1 -- 172.21.15.123:0/3002844715 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6ef00d14d0 msgr2=0x7f6ef01f47f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:03:49.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.700+0000 7f6eee800640 1 --2- 172.21.15.123:0/3002844715 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6ef00d14d0 0x7f6ef01f47f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:49.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.700+0000 7f6eee800640 1 -- 172.21.15.123:0/3002844715 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6ef015e870 msgr2=0x7f6ef00b5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:03:49.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.700+0000 7f6eee800640 1 --2- 172.21.15.123:0/3002844715 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6ef015e870 0x7f6ef00b5670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:49.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.700+0000 7f6eee800640 1 -- 172.21.15.123:0/3002844715 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ef015fb30 con 0x7f6ef00cfa40 2024-05-28T06:03:49.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.700+0000 7f6ee7e00640 1 --2- 172.21.15.123:0/3002844715 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6ef00d14d0 0x7f6ef01f47f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-28T06:03:49.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.700+0000 7f6eee800640 1 --2- 172.21.15.123:0/3002844715 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6ef00cfa40 0x7f6ef014faf0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f6ee000d620 tx=0x7f6ee000dae0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:03:49.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.700+0000 7f6ee6a00640 1 -- 172.21.15.123:0/3002844715 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ee000b620 con 0x7f6ef00cfa40 2024-05-28T06:03:49.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.700+0000 7f6ee6a00640 1 -- 172.21.15.123:0/3002844715 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ee0019070 con 0x7f6ef00cfa40 2024-05-28T06:03:49.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.700+0000 7f6ee6a00640 1 -- 172.21.15.123:0/3002844715 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ee000ba00 con 0x7f6ef00cfa40 2024-05-28T06:03:49.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.700+0000 7f6ef4a00640 1 -- 172.21.15.123:0/3002844715 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ef0160830 con 0x7f6ef00cfa40 2024-05-28T06:03:49.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.701+0000 7f6ef4a00640 1 -- 172.21.15.123:0/3002844715 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ef01f4d30 con 0x7f6ef00cfa40 2024-05-28T06:03:49.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.701+0000 7f6ee6a00640 1 -- 172.21.15.123:0/3002844715 <== mon.1 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f6ee000bba0 con 0x7f6ef00cfa40 2024-05-28T06:03:49.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.701+0000 7f6ee6a00640 1 -- 172.21.15.123:0/3002844715 <== mon.1 v2:172.21.15.123:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1530+0+0 (secure 0 0 0) 0x7f6ee001eac0 con 0x7f6ef00cfa40 2024-05-28T06:03:49.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.702+0000 7f6ef4a00640 1 -- 172.21.15.123:0/3002844715 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ef00689e0 con 0x7f6ef00cfa40 2024-05-28T06:03:49.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.704+0000 7f6ee6a00640 1 -- 172.21.15.123:0/3002844715 <== mon.1 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f6ee001e460 con 0x7f6ef00cfa40 2024-05-28T06:03:49.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.783+0000 7f6ef4a00640 1 -- 172.21.15.123:0/3002844715 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd new", "uuid": "be2a4947-16f1-4d46-b732-d511a2c19fae", "id": 7} v 0) -- 0x7f6ef00ba500 con 0x7f6ef00cfa40 2024-05-28T06:03:49.791 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.790+0000 7f6ee6a00640 1 -- 172.21.15.123:0/3002844715 <== mon.1 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "be2a4947-16f1-4d46-b732-d511a2c19fae", "id": 7}]=0 v10) ==== 112+0+2 (secure 0 0 0) 0x7f6ee001e640 con 0x7f6ef00cfa40 2024-05-28T06:03:49.791 INFO:teuthology.orchestra.run.smithi123.stdout:7 2024-05-28T06:03:49.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.791+0000 7f6ef4a00640 1 -- 172.21.15.123:0/3002844715 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6ef00cfa40 msgr2=0x7f6ef014faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:03:49.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.791+0000 7f6ef4a00640 1 --2- 172.21.15.123:0/3002844715 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6ef00cfa40 0x7f6ef014faf0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f6ee000d620 tx=0x7f6ee000dae0 comp rx=0 tx=0).stop 2024-05-28T06:03:49.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.791+0000 7f6ef4a00640 1 -- 172.21.15.123:0/3002844715 shutdown_connections 2024-05-28T06:03:49.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.791+0000 7f6ef4a00640 1 --2- 172.21.15.123:0/3002844715 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6ef015e870 0x7f6ef00b5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:49.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.791+0000 7f6ef4a00640 1 --2- 172.21.15.123:0/3002844715 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6ef00d14d0 0x7f6ef01f47f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:49.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.791+0000 7f6ef4a00640 1 --2- 172.21.15.123:0/3002844715 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6ef00cfa40 0x7f6ef014faf0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:49.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.791+0000 7f6ef4a00640 1 -- 172.21.15.123:0/3002844715 >> 172.21.15.123:0/3002844715 conn(0x7f6ef008bfc0 msgr2=0x7f6ef00caf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:03:49.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.791+0000 7f6ef4a00640 1 -- 172.21.15.123:0/3002844715 shutdown_connections 2024-05-28T06:03:49.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:03:49.791+0000 7f6ef4a00640 1 -- 172.21.15.123:0/3002844715 wait complete. 2024-05-28T06:03:49.809 INFO:tasks.ceph.osd.0:Restarting daemon 2024-05-28T06:03:49.809 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-05-28T06:03:49.816 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:03:49.815+0000 7feae3572040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-28T06:03:49.817 INFO:tasks.ceph.osd.0:Started 2024-05-28T06:03:49.817 INFO:tasks.ceph.osd.1:Restarting daemon 2024-05-28T06:03:49.817 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-05-28T06:03:49.825 INFO:tasks.ceph.osd.1:Started 2024-05-28T06:03:49.825 INFO:tasks.ceph.osd.2:Restarting daemon 2024-05-28T06:03:49.825 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-05-28T06:03:49.833 INFO:tasks.ceph.osd.2:Started 2024-05-28T06:03:49.833 INFO:tasks.ceph.osd.3:Restarting daemon 2024-05-28T06:03:49.833 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-05-28T06:03:49.837 INFO:tasks.ceph.osd.3:Started 2024-05-28T06:03:49.837 INFO:tasks.ceph.osd.4:Restarting daemon 2024-05-28T06:03:49.838 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-05-28T06:03:49.844 INFO:tasks.ceph.osd.4:Started 2024-05-28T06:03:49.845 INFO:tasks.ceph.osd.5:Restarting daemon 2024-05-28T06:03:49.845 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-05-28T06:03:49.851 INFO:tasks.ceph.osd.5:Started 2024-05-28T06:03:49.851 INFO:tasks.ceph.osd.6:Restarting daemon 2024-05-28T06:03:49.852 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-05-28T06:03:49.857 INFO:tasks.ceph.osd.6:Started 2024-05-28T06:03:49.857 INFO:tasks.ceph.osd.7:Restarting daemon 2024-05-28T06:03:49.858 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-05-28T06:03:49.863 INFO:tasks.ceph.osd.7:Started 2024-05-28T06:03:49.863 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:03:50.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:50.074+0000 7fe1a4800640 1 Processor -- start 2024-05-28T06:03:50.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:50.075+0000 7fe1a4800640 1 -- start start 2024-05-28T06:03:50.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:50.076+0000 7fe1a4800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe19c0b3b50 0x7fe19c0d3190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:03:50.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:50.076+0000 7fe1a4800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fe19c15ed20 0x7fe19c15f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:03:50.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:50.076+0000 7fe1a4800640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe19c0d36d0 con 0x7fe19c0d2db0 2024-05-28T06:03:50.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:50.076+0000 7fe1a4800640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fe19c0d5540 con 0x7fe19c15ed20 2024-05-28T06:03:50.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:50.076+0000 7fe1a4800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fe19c163500 con 0x7fe19c0b3b50 2024-05-28T06:03:50.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:50.076+0000 7fe1a1e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fe19c0d2db0 0x7fe19c0d4d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:50054/0 (socket says 172.21.15.96:50054) 2024-05-28T06:03:50.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:50.076+0000 7fe1a1e00640 1 -- 172.21.15.96:0/426370224 learned_addr learned my addr 172.21.15.96:0/426370224 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:03:50.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:50.076+0000 7fe1a2800640 1 --2- 172.21.15.96:0/426370224 >> v2:172.21.15.96:3300/0 conn(0x7fe19c15ed20 0x7fe19c15f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:03:50.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:50.076+0000 7fe1a1400640 1 --2- 172.21.15.96:0/426370224 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe19c0b3b50 0x7fe19c0d3190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:03:50.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:50.076+0000 7fe1a0a00640 1 -- 172.21.15.96:0/426370224 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2522650008 0 0) 0x7fe19c0d36d0 con 0x7fe19c0d2db0 2024-05-28T06:03:50.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:50.076+0000 7fe1a2800640 1 -- 172.21.15.96:0/426370224 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe19c0b3b50 msgr2=0x7fe19c0d3190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:03:50.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:50.076+0000 7fe1a2800640 1 --2- 172.21.15.96:0/426370224 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe19c0b3b50 0x7fe19c0d3190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:50.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:50.076+0000 7fe1a2800640 1 -- 172.21.15.96:0/426370224 >> v1:172.21.15.96:6789/0 conn(0x7fe19c0d2db0 legacy=0x7fe19c0d4d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:03:50.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:50.076+0000 7fe1a2800640 1 -- 172.21.15.96:0/426370224 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe19c162740 con 0x7fe19c15ed20 2024-05-28T06:03:50.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:50.077+0000 7fe1a2800640 1 --2- 172.21.15.96:0/426370224 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe19c15ed20 0x7fe19c15f1b0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe198009d90 tx=0x7fe19802a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3932e7c21550020c server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:03:50.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:50.078+0000 7fe1a0a00640 1 -- 172.21.15.96:0/426370224 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe198031600 con 0x7fe19c15ed20 2024-05-28T06:03:50.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:50.078+0000 7fe1a0a00640 1 -- 172.21.15.96:0/426370224 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1980317a0 con 0x7fe19c15ed20 2024-05-28T06:03:50.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:50.078+0000 7fe1a0a00640 1 -- 172.21.15.96:0/426370224 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe198031b70 con 0x7fe19c15ed20 2024-05-28T06:03:50.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:50.078+0000 7fe1a4800640 1 -- 172.21.15.96:0/426370224 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe19c15ed20 msgr2=0x7fe19c15f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:03:50.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:50.078+0000 7fe1a4800640 1 --2- 172.21.15.96:0/426370224 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe19c15ed20 0x7fe19c15f1b0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe198009d90 tx=0x7fe19802a5d0 comp rx=0 tx=0).stop 2024-05-28T06:03:50.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:50.078+0000 7fe1a4800640 1 -- 172.21.15.96:0/426370224 shutdown_connections 2024-05-28T06:03:50.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:50.078+0000 7fe1a4800640 1 --2- 172.21.15.96:0/426370224 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe19c15ed20 0x7fe19c15f1b0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:50.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:50.078+0000 7fe1a4800640 1 --2- 172.21.15.96:0/426370224 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe19c0b3b50 0x7fe19c0d3190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:50.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:50.078+0000 7fe1a4800640 1 -- 172.21.15.96:0/426370224 >> 172.21.15.96:0/426370224 conn(0x7fe19c08bfc0 msgr2=0x7fe19c0d4650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:03:50.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:50.078+0000 7fe1a4800640 1 -- 172.21.15.96:0/426370224 shutdown_connections 2024-05-28T06:03:50.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:50.078+0000 7fe1a4800640 1 -- 172.21.15.96:0/426370224 wait complete. 2024-05-28T06:03:50.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:50.080+0000 7fe1a4800640 1 Processor -- start 2024-05-28T06:03:50.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:50.080+0000 7fe1a4800640 1 -- start start 2024-05-28T06:03:50.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:50.081+0000 7fe1a4800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe19c0b3b50 0x7fe19c14f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:03:50.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:50.081+0000 7fe1a4800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe19c0d2db0 0x7fe19c0b2250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:03:50.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:50.081+0000 7fe1a4800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe19c15ed20 0x7fe19c0b2790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:03:50.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:50.081+0000 7fe1a4800640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe19c162fe0 con 0x7fe19c0b3b50 2024-05-28T06:03:50.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:50.081+0000 7fe1a4800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe19c162d00 con 0x7fe19c0d2db0 2024-05-28T06:03:50.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:50.081+0000 7fe1a4800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fe19c0d04a0 con 0x7fe19c15ed20 2024-05-28T06:03:50.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:50.081+0000 7fe1a1e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe19c0b3b50 0x7fe19c14f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:03:50.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:50.081+0000 7fe1a1e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe19c0b3b50 0x7fe19c14f970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:42960/0 (socket says 172.21.15.96:42960) 2024-05-28T06:03:50.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:50.081+0000 7fe1a1e00640 1 -- 172.21.15.96:0/3654720563 learned_addr learned my addr 172.21.15.96:0/3654720563 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:03:50.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:50.082+0000 7fe1a1400640 1 --2- 172.21.15.96:0/3654720563 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe19c0d2db0 0x7fe19c0b2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:03:50.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:50.082+0000 7fe1a2800640 1 --2- 172.21.15.96:0/3654720563 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe19c15ed20 0x7fe19c0b2790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:03:50.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:50.082+0000 7fe1a1e00640 1 -- 172.21.15.96:0/3654720563 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe19c15ed20 msgr2=0x7fe19c0b2790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:03:50.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:50.082+0000 7fe1a1e00640 1 --2- 172.21.15.96:0/3654720563 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe19c15ed20 0x7fe19c0b2790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:50.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:50.082+0000 7fe1a1e00640 1 -- 172.21.15.96:0/3654720563 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe19c0d2db0 msgr2=0x7fe19c0b2250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:03:50.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:50.082+0000 7fe1a1e00640 1 --2- 172.21.15.96:0/3654720563 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe19c0d2db0 0x7fe19c0b2250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:50.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:50.082+0000 7fe1a1e00640 1 -- 172.21.15.96:0/3654720563 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe19c0cd030 con 0x7fe19c0b3b50 2024-05-28T06:03:50.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:50.082+0000 7fe1a1e00640 1 --2- 172.21.15.96:0/3654720563 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe19c0b3b50 0x7fe19c14f970 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe19000c4d0 tx=0x7fe19000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:03:50.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:50.082+0000 7fe196000640 1 -- 172.21.15.96:0/3654720563 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe190018070 con 0x7fe19c0b3b50 2024-05-28T06:03:50.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:50.082+0000 7fe196000640 1 -- 172.21.15.96:0/3654720563 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe190007500 con 0x7fe19c0b3b50 2024-05-28T06:03:50.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:50.082+0000 7fe196000640 1 -- 172.21.15.96:0/3654720563 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe190004a20 con 0x7fe19c0b3b50 2024-05-28T06:03:50.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:50.082+0000 7fe1a4800640 1 -- 172.21.15.96:0/3654720563 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe19c0cd310 con 0x7fe19c0b3b50 2024-05-28T06:03:50.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:50.083+0000 7fe1a4800640 1 -- 172.21.15.96:0/3654720563 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe19c0cd8d0 con 0x7fe19c0b3b50 2024-05-28T06:03:50.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:50.083+0000 7fe196000640 1 -- 172.21.15.96:0/3654720563 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fe190004bc0 con 0x7fe19c0b3b50 2024-05-28T06:03:50.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:50.083+0000 7fe196000640 1 -- 172.21.15.96:0/3654720563 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7fe19001ba30 con 0x7fe19c0b3b50 2024-05-28T06:03:50.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:50.084+0000 7fe1a4800640 1 -- 172.21.15.96:0/3654720563 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe19c0689e0 con 0x7fe19c0b3b50 2024-05-28T06:03:50.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:50.086+0000 7fe196000640 1 -- 172.21.15.96:0/3654720563 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fe19001bc30 con 0x7fe19c0b3b50 2024-05-28T06:03:50.127 INFO:tasks.ceph.mgr.y.smithi096.stderr:2024-05-28T06:03:50.126+0000 7f6101dec040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-28T06:03:50.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:50.149+0000 7fe1a4800640 1 -- 172.21.15.96:0/3654720563 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe19c0d2af0 con 0x7fe19c0b3b50 2024-05-28T06:03:50.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:50.150+0000 7fe196000640 1 -- 172.21.15.96:0/3654720563 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7fe19001be10 con 0x7fe19c0b3b50 2024-05-28T06:03:50.152 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:03:50.152 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":10,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:03:49.786426+0000","last_up_change":"0.000000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:03:50.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:50.152+0000 7fe1a4800640 1 -- 172.21.15.96:0/3654720563 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe19c0b3b50 msgr2=0x7fe19c14f970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:03:50.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:50.152+0000 7fe1a4800640 1 --2- 172.21.15.96:0/3654720563 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe19c0b3b50 0x7fe19c14f970 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe19000c4d0 tx=0x7fe19000c990 comp rx=0 tx=0).stop 2024-05-28T06:03:50.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:50.152+0000 7fe1a4800640 1 -- 172.21.15.96:0/3654720563 shutdown_connections 2024-05-28T06:03:50.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:50.152+0000 7fe1a4800640 1 --2- 172.21.15.96:0/3654720563 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe19c15ed20 0x7fe19c0b2790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:50.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:50.152+0000 7fe1a4800640 1 --2- 172.21.15.96:0/3654720563 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe19c0d2db0 0x7fe19c0b2250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:50.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:50.152+0000 7fe1a4800640 1 --2- 172.21.15.96:0/3654720563 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe19c0b3b50 0x7fe19c14f970 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:50.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:50.152+0000 7fe1a4800640 1 -- 172.21.15.96:0/3654720563 >> 172.21.15.96:0/3654720563 conn(0x7fe19c08bfc0 msgr2=0x7fe19c15c710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:03:50.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:50.152+0000 7fe1a4800640 1 -- 172.21.15.96:0/3654720563 shutdown_connections 2024-05-28T06:03:50.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:03:50.152+0000 7fe1a4800640 1 -- 172.21.15.96:0/3654720563 wait complete. 2024-05-28T06:03:50.169 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-05-28T06:03:50.169 INFO:tasks.ceph:Waiting for OSDs to come up 2024-05-28T06:03:50.183 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:03:50.182+0000 7feae3572040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-28T06:03:50.238 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:03:50.237+0000 7feae3572040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-28T06:03:50.420 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:03:50.418+0000 7feae3572040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-28T06:03:50.429 INFO:tasks.ceph.mgr.y.smithi096.stderr:2024-05-28T06:03:50.428+0000 7f6101dec040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-28T06:03:50.489 INFO:tasks.ceph.mgr.y.smithi096.stderr:2024-05-28T06:03:50.488+0000 7f6101dec040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-28T06:03:50.507 INFO:tasks.ceph.osd.1.smithi096.stderr:2024-05-28T06:03:50.506+0000 7fa4e1c86640 -1 Falling back to public interface 2024-05-28T06:03:50.531 INFO:tasks.ceph.osd.4.smithi123.stderr:2024-05-28T06:03:50.530+0000 7faf6f714640 -1 Falling back to public interface 2024-05-28T06:03:50.540 INFO:tasks.ceph.osd.6.smithi123.stderr:2024-05-28T06:03:50.539+0000 7f792420d640 -1 Falling back to public interface 2024-05-28T06:03:50.548 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-05-28T06:03:50.547+0000 7f697c00d640 -1 Falling back to public interface 2024-05-28T06:03:50.549 INFO:tasks.ceph.osd.0.smithi096.stderr:2024-05-28T06:03:50.547+0000 7f258f614640 -1 Falling back to public interface 2024-05-28T06:03:50.554 INFO:tasks.ceph.osd.2.smithi096.stderr:2024-05-28T06:03:50.553+0000 7f3c43410640 -1 Falling back to public interface 2024-05-28T06:03:50.561 INFO:tasks.ceph.osd.7.smithi123.stderr:2024-05-28T06:03:50.560+0000 7febac816640 -1 Falling back to public interface 2024-05-28T06:03:50.561 INFO:tasks.ceph.osd.5.smithi123.stderr:2024-05-28T06:03:50.560+0000 7f4380e0d640 -1 Falling back to public interface 2024-05-28T06:03:50.592 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:03:50.590+0000 7feae3572040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-28T06:03:50.635 INFO:tasks.ceph.mgr.y.smithi096.stderr:2024-05-28T06:03:50.633+0000 7f6101dec040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-28T06:03:50.649 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:03:50.648+0000 7feae3572040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-28T06:03:50.728 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:03:50.727+0000 7feae3572040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-28T06:03:50.781 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:03:50.780+0000 7feae3572040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-28T06:03:50.971 INFO:tasks.ceph.mgr.y.smithi096.stderr:2024-05-28T06:03:50.969+0000 7f6101dec040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-28T06:03:50.975 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-28T06:03:51.029 INFO:tasks.ceph.mgr.y.smithi096.stderr:2024-05-28T06:03:51.028+0000 7f6101dec040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-28T06:03:51.134 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:51.132+0000 7ff843e00640 1 Processor -- start 2024-05-28T06:03:51.134 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:51.132+0000 7ff843e00640 1 -- start start 2024-05-28T06:03:51.135 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:51.134+0000 7ff843e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff8440d2a60 0x7ff8440d2e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:03:51.135 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:51.134+0000 7ff843e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7ff8440cefb0 0x7ff8440cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:03:51.135 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:51.134+0000 7ff843e00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff84415ee40 con 0x7ff8440d4150 2024-05-28T06:03:51.135 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:51.134+0000 7ff843e00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7ff8440d3380 con 0x7ff8440cefb0 2024-05-28T06:03:51.135 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:51.134+0000 7ff843e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ff84415f0e0 con 0x7ff8440d2a60 2024-05-28T06:03:51.135 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:51.134+0000 7ff842000640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7ff8440d4150 0x7ff8440d2070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:50070/0 (socket says 172.21.15.96:50070) 2024-05-28T06:03:51.136 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:51.134+0000 7ff842000640 1 -- 172.21.15.96:0/56280903 learned_addr learned my addr 172.21.15.96:0/56280903 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:03:51.136 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:51.134+0000 7ff843400640 1 --2- 172.21.15.96:0/56280903 >> v2:172.21.15.96:3300/0 conn(0x7ff8440cefb0 0x7ff8440cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:03:51.136 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:51.134+0000 7ff842a00640 1 --2- 172.21.15.96:0/56280903 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff8440d2a60 0x7ff8440d2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:03:51.136 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:51.134+0000 7ff841600640 1 -- 172.21.15.96:0/56280903 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2448899817 0 0) 0x7ff84415ee40 con 0x7ff8440d4150 2024-05-28T06:03:51.137 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:51.134+0000 7ff841600640 1 -- 172.21.15.96:0/56280903 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff828003520 con 0x7ff8440d4150 2024-05-28T06:03:51.137 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:51.135+0000 7ff842a00640 1 -- 172.21.15.96:0/56280903 >> v2:172.21.15.96:3300/0 conn(0x7ff8440cefb0 msgr2=0x7ff8440cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:03:51.137 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:51.135+0000 7ff842a00640 1 --2- 172.21.15.96:0/56280903 >> v2:172.21.15.96:3300/0 conn(0x7ff8440cefb0 0x7ff8440cf440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:51.137 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:51.135+0000 7ff842a00640 1 -- 172.21.15.96:0/56280903 >> v1:172.21.15.96:6789/0 conn(0x7ff8440d4150 legacy=0x7ff8440d2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:03:51.137 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:51.135+0000 7ff842a00640 1 -- 172.21.15.96:0/56280903 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff84415f3c0 con 0x7ff8440d2a60 2024-05-28T06:03:51.137 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:51.136+0000 7ff842a00640 1 --2- 172.21.15.96:0/56280903 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff8440d2a60 0x7ff8440d2e40 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff8340097a0 tx=0x7ff83402a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7d5f0108daffb072 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:03:51.138 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:51.136+0000 7ff841600640 1 -- 172.21.15.96:0/56280903 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff834037070 con 0x7ff8440d2a60 2024-05-28T06:03:51.138 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:51.136+0000 7ff841600640 1 -- 172.21.15.96:0/56280903 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff834032440 con 0x7ff8440d2a60 2024-05-28T06:03:51.138 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:51.136+0000 7ff841600640 1 -- 172.21.15.96:0/56280903 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff834032810 con 0x7ff8440d2a60 2024-05-28T06:03:51.138 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:51.136+0000 7ff843e00640 1 -- 172.21.15.96:0/56280903 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff8440d2a60 msgr2=0x7ff8440d2e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:03:51.138 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:51.136+0000 7ff843e00640 1 --2- 172.21.15.96:0/56280903 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff8440d2a60 0x7ff8440d2e40 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff8340097a0 tx=0x7ff83402a180 comp rx=0 tx=0).stop 2024-05-28T06:03:51.138 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:51.137+0000 7ff843e00640 1 -- 172.21.15.96:0/56280903 shutdown_connections 2024-05-28T06:03:51.138 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:51.137+0000 7ff843e00640 1 --2- 172.21.15.96:0/56280903 >> v2:172.21.15.96:3300/0 conn(0x7ff8440cefb0 0x7ff8440cf440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:51.138 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:51.137+0000 7ff843e00640 1 --2- 172.21.15.96:0/56280903 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff8440d2a60 0x7ff8440d2e40 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:51.139 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:51.137+0000 7ff843e00640 1 -- 172.21.15.96:0/56280903 >> 172.21.15.96:0/56280903 conn(0x7ff84408bf60 msgr2=0x7ff8440d4530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:03:51.139 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:51.137+0000 7ff843e00640 1 -- 172.21.15.96:0/56280903 shutdown_connections 2024-05-28T06:03:51.139 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:51.137+0000 7ff843e00640 1 -- 172.21.15.96:0/56280903 wait complete. 2024-05-28T06:03:51.139 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:51.138+0000 7ff843e00640 1 Processor -- start 2024-05-28T06:03:51.139 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:51.138+0000 7ff843e00640 1 -- start start 2024-05-28T06:03:51.141 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:51.139+0000 7ff843e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff8440cefb0 0x7ff84414a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:03:51.141 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:51.139+0000 7ff843e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff8440d2a60 0x7ff8440b9840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:03:51.141 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:51.139+0000 7ff843e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8440d4150 0x7ff8440c9a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:03:51.141 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:51.139+0000 7ff843e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff84415e9e0 con 0x7ff8440d2a60 2024-05-28T06:03:51.141 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:51.139+0000 7ff843e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff8440d2780 con 0x7ff8440d4150 2024-05-28T06:03:51.141 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:51.139+0000 7ff843e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ff84415eb60 con 0x7ff8440cefb0 2024-05-28T06:03:51.141 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:51.140+0000 7ff842000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff8440d2a60 0x7ff8440b9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:03:51.141 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:51.140+0000 7ff842a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff8440cefb0 0x7ff84414a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:03:51.141 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:51.140+0000 7ff842a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff8440cefb0 0x7ff84414a8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.96:59512/0 (socket says 172.21.15.96:59512) 2024-05-28T06:03:51.141 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:51.140+0000 7ff842a00640 1 -- 172.21.15.96:0/610740452 learned_addr learned my addr 172.21.15.96:0/610740452 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:03:51.141 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:51.140+0000 7ff843400640 1 --2- 172.21.15.96:0/610740452 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8440d4150 0x7ff8440c9a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:03:51.143 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:51.140+0000 7ff842a00640 1 -- 172.21.15.96:0/610740452 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8440d4150 msgr2=0x7ff8440c9a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:03:51.143 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:51.140+0000 7ff842a00640 1 --2- 172.21.15.96:0/610740452 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8440d4150 0x7ff8440c9a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:51.143 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:51.140+0000 7ff842a00640 1 -- 172.21.15.96:0/610740452 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff8440d2a60 msgr2=0x7ff8440b9840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:03:51.143 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:51.140+0000 7ff842a00640 1 --2- 172.21.15.96:0/610740452 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff8440d2a60 0x7ff8440b9840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:51.143 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:51.140+0000 7ff842a00640 1 -- 172.21.15.96:0/610740452 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff84415b0e0 con 0x7ff8440cefb0 2024-05-28T06:03:51.143 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:51.140+0000 7ff842a00640 1 --2- 172.21.15.96:0/610740452 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff8440cefb0 0x7ff84414a8c0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff834035040 tx=0x7ff8340046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:03:51.143 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:51.141+0000 7ff83aa00640 1 -- 172.21.15.96:0/610740452 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff834041030 con 0x7ff8440cefb0 2024-05-28T06:03:51.143 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:51.141+0000 7ff83aa00640 1 -- 172.21.15.96:0/610740452 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff83403d4e0 con 0x7ff8440cefb0 2024-05-28T06:03:51.143 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:51.141+0000 7ff83aa00640 1 -- 172.21.15.96:0/610740452 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff834037040 con 0x7ff8440cefb0 2024-05-28T06:03:51.143 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:51.141+0000 7ff843e00640 1 -- 172.21.15.96:0/610740452 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8440b8120 con 0x7ff8440cefb0 2024-05-28T06:03:51.143 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:51.141+0000 7ff843e00640 1 -- 172.21.15.96:0/610740452 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8440b86e0 con 0x7ff8440cefb0 2024-05-28T06:03:51.144 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:51.141+0000 7ff83aa00640 1 -- 172.21.15.96:0/610740452 <== mon.2 v2:172.21.15.123:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7ff834044070 con 0x7ff8440cefb0 2024-05-28T06:03:51.144 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:51.141+0000 7ff83aa00640 1 -- 172.21.15.96:0/610740452 <== mon.2 v2:172.21.15.123:3301/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7ff834046690 con 0x7ff8440cefb0 2024-05-28T06:03:51.144 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:51.142+0000 7ff843e00640 1 -- 172.21.15.96:0/610740452 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8440a2440 con 0x7ff8440cefb0 2024-05-28T06:03:51.146 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:51.145+0000 7ff83aa00640 1 -- 172.21.15.96:0/610740452 <== mon.2 v2:172.21.15.123:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7ff834046960 con 0x7ff8440cefb0 2024-05-28T06:03:51.212 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:51.210+0000 7ff843e00640 1 -- 172.21.15.96:0/610740452 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff8440b2f00 con 0x7ff8440cefb0 2024-05-28T06:03:51.213 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:51.212+0000 7ff83aa00640 1 -- 172.21.15.96:0/610740452 <== mon.2 v2:172.21.15.123:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7ff834046b40 con 0x7ff8440cefb0 2024-05-28T06:03:51.214 INFO:teuthology.misc.health.smithi096.stdout: 2024-05-28T06:03:51.214 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":10,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:03:49.786426+0000","last_up_change":"0.000000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:03:51.216 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:51.214+0000 7ff843e00640 1 -- 172.21.15.96:0/610740452 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff8440cefb0 msgr2=0x7ff84414a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:03:51.216 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:51.214+0000 7ff843e00640 1 --2- 172.21.15.96:0/610740452 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff8440cefb0 0x7ff84414a8c0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff834035040 tx=0x7ff8340046c0 comp rx=0 tx=0).stop 2024-05-28T06:03:51.216 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:51.214+0000 7ff843e00640 1 -- 172.21.15.96:0/610740452 shutdown_connections 2024-05-28T06:03:51.216 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:51.214+0000 7ff843e00640 1 --2- 172.21.15.96:0/610740452 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8440d4150 0x7ff8440c9a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:51.216 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:51.214+0000 7ff843e00640 1 --2- 172.21.15.96:0/610740452 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff8440d2a60 0x7ff8440b9840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:51.216 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:51.214+0000 7ff843e00640 1 --2- 172.21.15.96:0/610740452 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff8440cefb0 0x7ff84414a8c0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:51.216 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:51.214+0000 7ff843e00640 1 -- 172.21.15.96:0/610740452 >> 172.21.15.96:0/610740452 conn(0x7ff84408bf60 msgr2=0x7ff844159bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:03:51.216 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:51.214+0000 7ff843e00640 1 -- 172.21.15.96:0/610740452 shutdown_connections 2024-05-28T06:03:51.216 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:51.214+0000 7ff843e00640 1 -- 172.21.15.96:0/610740452 wait complete. 2024-05-28T06:03:51.220 INFO:tasks.ceph.mgr.y.smithi096.stderr:2024-05-28T06:03:51.219+0000 7f6101dec040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-28T06:03:51.228 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-05-28T06:03:51.321 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:03:51.319+0000 7feae3572040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-28T06:03:51.390 INFO:tasks.ceph.mgr.y.smithi096.stderr:2024-05-28T06:03:51.389+0000 7f6101dec040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-28T06:03:51.451 INFO:tasks.ceph.mgr.y.smithi096.stderr:2024-05-28T06:03:51.449+0000 7f6101dec040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-28T06:03:51.454 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:03:51.453+0000 7feae3572040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-28T06:03:51.539 INFO:tasks.ceph.mgr.y.smithi096.stderr:2024-05-28T06:03:51.537+0000 7f6101dec040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-28T06:03:51.592 INFO:tasks.ceph.mgr.y.smithi096.stderr:2024-05-28T06:03:51.591+0000 7f6101dec040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-28T06:03:52.138 INFO:tasks.ceph.mgr.y.smithi096.stderr:2024-05-28T06:03:52.137+0000 7f6101dec040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-28T06:03:52.272 INFO:tasks.ceph.mgr.y.smithi096.stderr:2024-05-28T06:03:52.271+0000 7f6101dec040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-28T06:03:52.496 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:03:52.495+0000 7fea84600640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-05-28T06:03:52.497 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:03:52.496+0000 7fea84600640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-05-28T06:03:53.923 INFO:tasks.ceph.osd.6.smithi123.stderr:2024-05-28T06:03:53.922+0000 7f792420d640 -1 osd.6 0 log_to_monitors true 2024-05-28T06:03:54.021 INFO:tasks.ceph.osd.2.smithi096.stderr:2024-05-28T06:03:54.020+0000 7f3c43410640 -1 osd.2 0 log_to_monitors true 2024-05-28T06:03:54.115 INFO:tasks.ceph.osd.4.smithi123.stderr:2024-05-28T06:03:54.114+0000 7faf6f714640 -1 osd.4 0 log_to_monitors true 2024-05-28T06:03:54.163 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-05-28T06:03:54.162+0000 7f697c00d640 -1 osd.3 0 log_to_monitors true 2024-05-28T06:03:54.178 INFO:tasks.ceph.osd.0.smithi096.stderr:2024-05-28T06:03:54.177+0000 7f258f614640 -1 osd.0 0 log_to_monitors true 2024-05-28T06:03:54.182 INFO:tasks.ceph.osd.5.smithi123.stderr:2024-05-28T06:03:54.182+0000 7f4380e0d640 -1 osd.5 0 log_to_monitors true 2024-05-28T06:03:54.209 INFO:tasks.ceph.osd.7.smithi123.stderr:2024-05-28T06:03:54.208+0000 7febac816640 -1 osd.7 0 log_to_monitors true 2024-05-28T06:03:54.380 INFO:tasks.ceph.osd.1.smithi096.stderr:2024-05-28T06:03:54.379+0000 7fa4e1c86640 -1 osd.1 0 log_to_monitors true 2024-05-28T06:03:57.829 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-05-28T06:03:57.827+0000 7f6976a00640 -1 osd.3 0 waiting for initial osdmap 2024-05-28T06:03:57.837 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-05-28T06:03:57.836+0000 7f696ca00640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-28T06:03:57.857 INFO:tasks.ceph.osd.2.smithi096.stderr:2024-05-28T06:03:57.856+0000 7f3c3de00640 -1 osd.2 0 waiting for initial osdmap 2024-05-28T06:03:57.858 INFO:tasks.ceph.osd.4.smithi123.stderr:2024-05-28T06:03:57.855+0000 7faf6a600640 -1 osd.4 0 waiting for initial osdmap 2024-05-28T06:03:57.861 INFO:tasks.ceph.osd.7.smithi123.stderr:2024-05-28T06:03:57.860+0000 7feba7000640 -1 osd.7 0 waiting for initial osdmap 2024-05-28T06:03:57.862 INFO:tasks.ceph.osd.4.smithi123.stderr:2024-05-28T06:03:57.861+0000 7faf60600640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-28T06:03:57.866 INFO:tasks.ceph.osd.2.smithi096.stderr:2024-05-28T06:03:57.865+0000 7f3c33e00640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-28T06:03:57.870 INFO:tasks.ceph.osd.7.smithi123.stderr:2024-05-28T06:03:57.868+0000 7feb9d000640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-28T06:03:57.879 INFO:tasks.ceph.osd.0.smithi096.stderr:2024-05-28T06:03:57.878+0000 7f258a800640 -1 osd.0 0 waiting for initial osdmap 2024-05-28T06:03:57.885 INFO:tasks.ceph.osd.0.smithi096.stderr:2024-05-28T06:03:57.884+0000 7f257fe00640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-28T06:03:57.888 INFO:tasks.ceph.osd.1.smithi096.stderr:2024-05-28T06:03:57.886+0000 7fa4dcc00640 -1 osd.1 0 waiting for initial osdmap 2024-05-28T06:03:57.902 INFO:tasks.ceph.osd.5.smithi123.stderr:2024-05-28T06:03:57.901+0000 7f437b600640 -1 osd.5 0 waiting for initial osdmap 2024-05-28T06:03:57.907 INFO:tasks.ceph.osd.1.smithi096.stderr:2024-05-28T06:03:57.906+0000 7fa4d2c00640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-28T06:03:57.908 INFO:tasks.ceph.osd.5.smithi123.stderr:2024-05-28T06:03:57.908+0000 7f4371600640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-28T06:03:57.928 INFO:tasks.ceph.osd.6.smithi123.stderr:2024-05-28T06:03:57.927+0000 7f791f600640 -1 osd.6 0 waiting for initial osdmap 2024-05-28T06:03:57.943 INFO:tasks.ceph.osd.6.smithi123.stderr:2024-05-28T06:03:57.942+0000 7f7914c00640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-28T06:03:58.034 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-28T06:03:58.205 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:58.204+0000 7ff581000640 1 Processor -- start 2024-05-28T06:03:58.205 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:58.204+0000 7ff581000640 1 -- start start 2024-05-28T06:03:58.207 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:58.205+0000 7ff581000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff57c158640 0x7ff57c06dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:03:58.207 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:58.205+0000 7ff581000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7ff57c06e7d0 0x7ff57c06ec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:03:58.207 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:58.205+0000 7ff581000640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff57c15b3d0 con 0x7ff57c0b3ce0 2024-05-28T06:03:58.207 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:58.205+0000 7ff581000640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7ff57c15b6b0 con 0x7ff57c06e7d0 2024-05-28T06:03:58.207 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:58.205+0000 7ff581000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ff57c15b990 con 0x7ff57c158640 2024-05-28T06:03:58.208 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:58.206+0000 7ff57b400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff57c158640 0x7ff57c06dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:03:58.208 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:58.206+0000 7ff57aa00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7ff57c06e7d0 0x7ff57c06ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:03:58.208 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:58.206+0000 7ff57b400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff57c158640 0x7ff57c06dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.96:59554/0 (socket says 172.21.15.96:59554) 2024-05-28T06:03:58.208 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:58.206+0000 7ff57b400640 1 -- 172.21.15.96:0/16452408 learned_addr learned my addr 172.21.15.96:0/16452408 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:03:58.208 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:58.207+0000 7ff57b400640 1 -- 172.21.15.96:0/16452408 >> v2:172.21.15.123:3300/0 conn(0x7ff57c06e7d0 msgr2=0x7ff57c06ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:03:58.209 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:58.207+0000 7ff57b400640 1 --2- 172.21.15.96:0/16452408 >> v2:172.21.15.123:3300/0 conn(0x7ff57c06e7d0 0x7ff57c06ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:58.209 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:58.207+0000 7ff57b400640 1 -- 172.21.15.96:0/16452408 >> v1:172.21.15.123:6789/0 conn(0x7ff57c0b3ce0 legacy=0x7ff57c0b4130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:03:58.209 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:58.207+0000 7ff57b400640 1 -- 172.21.15.96:0/16452408 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff57c15bc70 con 0x7ff57c158640 2024-05-28T06:03:58.209 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:58.208+0000 7ff57b400640 1 --2- 172.21.15.96:0/16452408 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff57c158640 0x7ff57c06dd10 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff56c009870 tx=0x7ff56c02a250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6fb042f97b3ac36e server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:03:58.210 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:58.208+0000 7ff57a000640 1 -- 172.21.15.96:0/16452408 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff56c037070 con 0x7ff57c158640 2024-05-28T06:03:58.210 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:58.209+0000 7ff57a000640 1 -- 172.21.15.96:0/16452408 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff56c032440 con 0x7ff57c158640 2024-05-28T06:03:58.210 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:58.209+0000 7ff57a000640 1 -- 172.21.15.96:0/16452408 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff56c032810 con 0x7ff57c158640 2024-05-28T06:03:58.211 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:58.209+0000 7ff581000640 1 -- 172.21.15.96:0/16452408 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff57c158640 msgr2=0x7ff57c06dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:03:58.211 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:58.209+0000 7ff581000640 1 --2- 172.21.15.96:0/16452408 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff57c158640 0x7ff57c06dd10 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff56c009870 tx=0x7ff56c02a250 comp rx=0 tx=0).stop 2024-05-28T06:03:58.211 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:58.209+0000 7ff581000640 1 -- 172.21.15.96:0/16452408 shutdown_connections 2024-05-28T06:03:58.211 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:58.209+0000 7ff581000640 1 --2- 172.21.15.96:0/16452408 >> v2:172.21.15.123:3300/0 conn(0x7ff57c06e7d0 0x7ff57c06ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:58.211 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:58.209+0000 7ff581000640 1 --2- 172.21.15.96:0/16452408 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff57c158640 0x7ff57c06dd10 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:58.211 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:58.209+0000 7ff581000640 1 -- 172.21.15.96:0/16452408 >> 172.21.15.96:0/16452408 conn(0x7ff57c0b1f60 msgr2=0x7ff57c0b2340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:03:58.211 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:58.209+0000 7ff581000640 1 -- 172.21.15.96:0/16452408 shutdown_connections 2024-05-28T06:03:58.211 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:58.209+0000 7ff581000640 1 -- 172.21.15.96:0/16452408 wait complete. 2024-05-28T06:03:58.212 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:58.211+0000 7ff581000640 1 Processor -- start 2024-05-28T06:03:58.212 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:58.211+0000 7ff581000640 1 -- start start 2024-05-28T06:03:58.214 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:58.212+0000 7ff581000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff57c06e7d0 0x7ff57c0ce7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:03:58.214 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:58.212+0000 7ff581000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff57c0b3ce0 0x7ff57c09da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:03:58.214 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:58.212+0000 7ff581000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff57c158640 0x7ff57c0cf930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:03:58.214 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:58.212+0000 7ff581000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff57c158a20 con 0x7ff57c06e7d0 2024-05-28T06:03:58.214 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:58.212+0000 7ff581000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff57c15b0c0 con 0x7ff57c158640 2024-05-28T06:03:58.214 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:58.212+0000 7ff581000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ff57c06e250 con 0x7ff57c0b3ce0 2024-05-28T06:03:58.214 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:58.212+0000 7ff57b400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff57c06e7d0 0x7ff57c0ce7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:03:58.214 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:58.212+0000 7ff57b400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff57c06e7d0 0x7ff57c0ce7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:43018/0 (socket says 172.21.15.96:43018) 2024-05-28T06:03:58.214 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:58.212+0000 7ff57b400640 1 -- 172.21.15.96:0/3203338923 learned_addr learned my addr 172.21.15.96:0/3203338923 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:03:58.214 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:58.213+0000 7ff57b400640 1 -- 172.21.15.96:0/3203338923 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff57c0b3ce0 msgr2=0x7ff57c09da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:03:58.215 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:58.213+0000 7ff57b400640 1 --2- 172.21.15.96:0/3203338923 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff57c0b3ce0 0x7ff57c09da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:58.215 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:58.213+0000 7ff57b400640 1 -- 172.21.15.96:0/3203338923 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff57c158640 msgr2=0x7ff57c0cf930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:03:58.215 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:58.213+0000 7ff57b400640 1 --2- 172.21.15.96:0/3203338923 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff57c158640 0x7ff57c0cf930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:58.215 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:58.213+0000 7ff57b400640 1 -- 172.21.15.96:0/3203338923 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff57c0ced00 con 0x7ff57c06e7d0 2024-05-28T06:03:58.215 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:58.213+0000 7ff57b400640 1 --2- 172.21.15.96:0/3203338923 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff57c06e7d0 0x7ff57c0ce7c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7ff56c02a290 tx=0x7ff56c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:03:58.215 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:58.213+0000 7ff573400640 1 -- 172.21.15.96:0/3203338923 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff56c041030 con 0x7ff57c06e7d0 2024-05-28T06:03:58.216 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:58.213+0000 7ff573400640 1 -- 172.21.15.96:0/3203338923 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff56c030440 con 0x7ff57c06e7d0 2024-05-28T06:03:58.216 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:58.213+0000 7ff573400640 1 -- 172.21.15.96:0/3203338923 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff56c037040 con 0x7ff57c06e7d0 2024-05-28T06:03:58.216 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:58.213+0000 7ff581000640 1 -- 172.21.15.96:0/3203338923 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff57c0cefa0 con 0x7ff57c06e7d0 2024-05-28T06:03:58.216 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:58.213+0000 7ff581000640 1 -- 172.21.15.96:0/3203338923 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff57c0cf560 con 0x7ff57c06e7d0 2024-05-28T06:03:58.216 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:58.214+0000 7ff581000640 1 -- 172.21.15.96:0/3203338923 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff57c09e550 con 0x7ff57c06e7d0 2024-05-28T06:03:58.216 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:58.215+0000 7ff573400640 1 -- 172.21.15.96:0/3203338923 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 4) ==== 90984+0+0 (secure 0 0 0) 0x7ff56c044070 con 0x7ff57c06e7d0 2024-05-28T06:03:58.217 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:58.216+0000 7ff573400640 1 --2- 172.21.15.96:0/3203338923 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff55c06c860 0x7ff55c06ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:03:58.217 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:58.216+0000 7ff573400640 1 -- 172.21.15.96:0/3203338923 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(12..12 src has 1..12) ==== 2231+0+0 (secure 0 0 0) 0x7ff56c0ab420 con 0x7ff57c06e7d0 2024-05-28T06:03:58.218 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:58.216+0000 7ff57aa00640 1 --2- 172.21.15.96:0/3203338923 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff55c06c860 0x7ff55c06ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:03:58.219 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:58.217+0000 7ff57aa00640 1 --2- 172.21.15.96:0/3203338923 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff55c06c860 0x7ff55c06ed20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ff568006a90 tx=0x7ff568006a20 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:03:58.223 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:58.222+0000 7ff573400640 1 -- 172.21.15.96:0/3203338923 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff56c07a620 con 0x7ff57c06e7d0 2024-05-28T06:03:58.409 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:58.407+0000 7ff581000640 1 -- 172.21.15.96:0/3203338923 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff57c075e70 con 0x7ff57c06e7d0 2024-05-28T06:03:58.410 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:58.409+0000 7ff573400640 1 -- 172.21.15.96:0/3203338923 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) ==== 74+0+7058 (secure 0 0 0) 0x7ff56c079f20 con 0x7ff57c06e7d0 2024-05-28T06:03:58.410 INFO:teuthology.misc.health.smithi096.stdout: 2024-05-28T06:03:58.410 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":12,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:03:55.507144+0000","last_up_change":"0.000000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:03:58.413 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:58.411+0000 7ff581000640 1 -- 172.21.15.96:0/3203338923 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff55c06c860 msgr2=0x7ff55c06ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:03:58.413 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:58.411+0000 7ff581000640 1 --2- 172.21.15.96:0/3203338923 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff55c06c860 0x7ff55c06ed20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ff568006a90 tx=0x7ff568006a20 comp rx=0 tx=0).stop 2024-05-28T06:03:58.413 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:58.411+0000 7ff581000640 1 -- 172.21.15.96:0/3203338923 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff57c06e7d0 msgr2=0x7ff57c0ce7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:03:58.413 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:58.411+0000 7ff581000640 1 --2- 172.21.15.96:0/3203338923 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff57c06e7d0 0x7ff57c0ce7c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7ff56c02a290 tx=0x7ff56c0046c0 comp rx=0 tx=0).stop 2024-05-28T06:03:58.413 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:58.412+0000 7ff581000640 1 -- 172.21.15.96:0/3203338923 shutdown_connections 2024-05-28T06:03:58.413 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:58.412+0000 7ff581000640 1 --2- 172.21.15.96:0/3203338923 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff55c06c860 0x7ff55c06ed20 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:58.413 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:58.412+0000 7ff581000640 1 --2- 172.21.15.96:0/3203338923 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff57c158640 0x7ff57c0cf930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:58.414 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:58.412+0000 7ff581000640 1 --2- 172.21.15.96:0/3203338923 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff57c0b3ce0 0x7ff57c09da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:58.414 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:58.412+0000 7ff581000640 1 --2- 172.21.15.96:0/3203338923 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff57c06e7d0 0x7ff57c0ce7c0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:03:58.414 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:58.412+0000 7ff581000640 1 -- 172.21.15.96:0/3203338923 >> 172.21.15.96:0/3203338923 conn(0x7ff57c0b1f60 msgr2=0x7ff57c157970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:03:58.414 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:58.412+0000 7ff581000640 1 -- 172.21.15.96:0/3203338923 shutdown_connections 2024-05-28T06:03:58.414 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:03:58.412+0000 7ff581000640 1 -- 172.21.15.96:0/3203338923 wait complete. 2024-05-28T06:03:58.426 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-05-28T06:03:58.617 DEBUG:teuthology.orchestra.run.smithi096:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:03:58.625 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:04:05.232 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-28T06:04:05.420 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:04:05.418+0000 7fb137200640 1 Processor -- start 2024-05-28T06:04:05.420 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:04:05.418+0000 7fb137200640 1 -- start start 2024-05-28T06:04:05.423 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:04:05.419+0000 7fb137200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fb1380d27b0 0x7fb1380b5f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:05.423 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:04:05.419+0000 7fb137200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1380d0250 con 0x7fb13815e9a0 2024-05-28T06:04:05.423 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:04:05.419+0000 7fb137200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1380d2e90 con 0x7fb1380b6490 2024-05-28T06:04:05.423 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:04:05.419+0000 7fb137200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fb1380d2c40 con 0x7fb1380d27b0 2024-05-28T06:04:05.423 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:04:05.419+0000 7fb135e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fb13815e9a0 0x7fb1380cfb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:42134/0 (socket says 172.21.15.96:42134) 2024-05-28T06:04:05.424 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:04:05.419+0000 7fb135e00640 1 -- 172.21.15.96:0/126893998 learned_addr learned my addr 172.21.15.96:0/126893998 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:04:05.424 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:04:05.420+0000 7fb135400640 1 --2- 172.21.15.96:0/126893998 >> v2:172.21.15.96:3300/0 conn(0x7fb1380d27b0 0x7fb1380b5f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:05.424 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:04:05.420+0000 7fb135400640 1 -- 172.21.15.96:0/126893998 >> v1:172.21.15.123:6789/0 conn(0x7fb1380b6490 legacy=0x7fb13806ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:05.424 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:04:05.420+0000 7fb135400640 1 -- 172.21.15.96:0/126893998 >> v1:172.21.15.96:6789/0 conn(0x7fb13815e9a0 legacy=0x7fb1380cfb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:05.424 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:04:05.420+0000 7fb135400640 1 -- 172.21.15.96:0/126893998 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1380d3170 con 0x7fb1380d27b0 2024-05-28T06:04:05.425 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:04:05.420+0000 7fb134a00640 1 -- 172.21.15.96:0/126893998 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3948984069 0 0) 0x7fb1380d0250 con 0x7fb13815e9a0 2024-05-28T06:04:05.425 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:04:05.421+0000 7fb135400640 1 --2- 172.21.15.96:0/126893998 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb1380d27b0 0x7fb1380b5f50 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb1240097a0 tx=0x7fb12402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d99e9ce112303ae server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:05.425 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:04:05.421+0000 7fb134a00640 1 -- 172.21.15.96:0/126893998 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb124037070 con 0x7fb1380d27b0 2024-05-28T06:04:05.425 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:04:05.421+0000 7fb134a00640 1 -- 172.21.15.96:0/126893998 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb124032440 con 0x7fb1380d27b0 2024-05-28T06:04:05.425 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:04:05.421+0000 7fb134a00640 1 -- 172.21.15.96:0/126893998 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb124032810 con 0x7fb1380d27b0 2024-05-28T06:04:05.432 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:04:05.422+0000 7fb137200640 1 -- 172.21.15.96:0/126893998 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb1380d27b0 msgr2=0x7fb1380b5f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:05.433 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:04:05.422+0000 7fb137200640 1 --2- 172.21.15.96:0/126893998 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb1380d27b0 0x7fb1380b5f50 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb1240097a0 tx=0x7fb12402a1e0 comp rx=0 tx=0).stop 2024-05-28T06:04:05.434 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:04:05.422+0000 7fb137200640 1 -- 172.21.15.96:0/126893998 shutdown_connections 2024-05-28T06:04:05.434 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:04:05.422+0000 7fb137200640 1 --2- 172.21.15.96:0/126893998 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb1380d27b0 0x7fb1380b5f50 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:05.434 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:04:05.422+0000 7fb137200640 1 -- 172.21.15.96:0/126893998 >> 172.21.15.96:0/126893998 conn(0x7fb13808bf60 msgr2=0x7fb1380d44d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:05.434 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:04:05.422+0000 7fb137200640 1 -- 172.21.15.96:0/126893998 shutdown_connections 2024-05-28T06:04:05.435 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:04:05.422+0000 7fb137200640 1 -- 172.21.15.96:0/126893998 wait complete. 2024-05-28T06:04:05.435 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:04:05.424+0000 7fb137200640 1 Processor -- start 2024-05-28T06:04:05.435 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:04:05.424+0000 7fb137200640 1 -- start start 2024-05-28T06:04:05.435 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:04:05.425+0000 7fb137200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb1380b6490 0x7fb1380c9a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:05.435 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:04:05.425+0000 7fb137200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb1380d27b0 0x7fb1381560a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:05.436 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:04:05.425+0000 7fb137200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb13815e9a0 0x7fb1381f7e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:05.436 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:04:05.425+0000 7fb137200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb1380d3a10 con 0x7fb1380b6490 2024-05-28T06:04:05.436 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:04:05.425+0000 7fb137200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fb1380d3730 con 0x7fb1380d27b0 2024-05-28T06:04:05.436 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:04:05.425+0000 7fb137200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fb1380d3f30 con 0x7fb13815e9a0 2024-05-28T06:04:05.437 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:04:05.425+0000 7fb135e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb1380b6490 0x7fb1380c9a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:05.437 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:04:05.425+0000 7fb135e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb1380b6490 0x7fb1380c9a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:33772/0 (socket says 172.21.15.96:33772) 2024-05-28T06:04:05.437 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:04:05.425+0000 7fb135e00640 1 -- 172.21.15.96:0/3072991025 learned_addr learned my addr 172.21.15.96:0/3072991025 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:05.437 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:04:05.426+0000 7fb135400640 1 --2- 172.21.15.96:0/3072991025 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb1380d27b0 0x7fb1381560a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:05.437 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:04:05.426+0000 7fb135e00640 1 -- 172.21.15.96:0/3072991025 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb13815e9a0 msgr2=0x7fb1381f7e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:05.437 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:04:05.426+0000 7fb135e00640 1 --2- 172.21.15.96:0/3072991025 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb13815e9a0 0x7fb1381f7e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:05.438 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:04:05.426+0000 7fb135e00640 1 -- 172.21.15.96:0/3072991025 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb1380d27b0 msgr2=0x7fb1381560a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:05.438 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:04:05.426+0000 7fb135e00640 1 --2- 172.21.15.96:0/3072991025 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb1380d27b0 0x7fb1381560a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:05.438 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:04:05.426+0000 7fb135e00640 1 -- 172.21.15.96:0/3072991025 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1380c9f60 con 0x7fb1380b6490 2024-05-28T06:04:05.438 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:04:05.426+0000 7fb135e00640 1 --2- 172.21.15.96:0/3072991025 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb1380b6490 0x7fb1380c9a20 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb12800c4d0 tx=0x7fb12800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:05.438 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:04:05.426+0000 7fb12e000640 1 -- 172.21.15.96:0/3072991025 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb128018070 con 0x7fb1380b6490 2024-05-28T06:04:05.438 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:04:05.426+0000 7fb12e000640 1 -- 172.21.15.96:0/3072991025 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb128007500 con 0x7fb1380b6490 2024-05-28T06:04:05.438 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:04:05.426+0000 7fb137200640 1 -- 172.21.15.96:0/3072991025 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb13815f490 con 0x7fb1380b6490 2024-05-28T06:04:05.438 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:04:05.426+0000 7fb12e000640 1 -- 172.21.15.96:0/3072991025 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb128004a20 con 0x7fb1380b6490 2024-05-28T06:04:05.438 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:04:05.426+0000 7fb137200640 1 -- 172.21.15.96:0/3072991025 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb13815fd30 con 0x7fb1380b6490 2024-05-28T06:04:05.438 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:04:05.427+0000 7fb137200640 1 -- 172.21.15.96:0/3072991025 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb13815f880 con 0x7fb1380b6490 2024-05-28T06:04:05.439 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:04:05.429+0000 7fb12e000640 1 -- 172.21.15.96:0/3072991025 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb12801b070 con 0x7fb1380b6490 2024-05-28T06:04:05.439 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:04:05.429+0000 7fb12e000640 1 --2- 172.21.15.96:0/3072991025 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb11806c8f0 0x7fb11806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:05.439 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:04:05.429+0000 7fb12e000640 1 -- 172.21.15.96:0/3072991025 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7fb12808ac00 con 0x7fb1380b6490 2024-05-28T06:04:05.439 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:04:05.430+0000 7fb135400640 1 --2- 172.21.15.96:0/3072991025 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb11806c8f0 0x7fb11806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:05.439 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:04:05.431+0000 7fb135400640 1 --2- 172.21.15.96:0/3072991025 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb11806c8f0 0x7fb11806edb0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb124034000 tx=0x7fb12402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:05.439 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:04:05.435+0000 7fb12e000640 1 -- 172.21.15.96:0/3072991025 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb1280592e0 con 0x7fb1380b6490 2024-05-28T06:04:05.636 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:04:05.635+0000 7fb137200640 1 -- 172.21.15.96:0/3072991025 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb138075e70 con 0x7fb1380b6490 2024-05-28T06:04:05.639 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:04:05.637+0000 7fb12e000640 1 -- 172.21.15.96:0/3072991025 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) ==== 74+0+12896 (secure 0 0 0) 0x7fb128058be0 con 0x7fb1380b6490 2024-05-28T06:04:05.639 INFO:teuthology.misc.health.smithi096.stdout: 2024-05-28T06:04:05.639 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":16,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:04:01.891977+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:04:05.642 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:04:05.640+0000 7fb137200640 1 -- 172.21.15.96:0/3072991025 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb11806c8f0 msgr2=0x7fb11806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:05.642 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:04:05.640+0000 7fb137200640 1 --2- 172.21.15.96:0/3072991025 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb11806c8f0 0x7fb11806edb0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb124034000 tx=0x7fb12402e000 comp rx=0 tx=0).stop 2024-05-28T06:04:05.642 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:04:05.640+0000 7fb137200640 1 -- 172.21.15.96:0/3072991025 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb1380b6490 msgr2=0x7fb1380c9a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:05.642 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:04:05.640+0000 7fb137200640 1 --2- 172.21.15.96:0/3072991025 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb1380b6490 0x7fb1380c9a20 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb12800c4d0 tx=0x7fb12800c990 comp rx=0 tx=0).stop 2024-05-28T06:04:05.642 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:04:05.640+0000 7fb137200640 1 -- 172.21.15.96:0/3072991025 shutdown_connections 2024-05-28T06:04:05.642 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:04:05.640+0000 7fb137200640 1 --2- 172.21.15.96:0/3072991025 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb11806c8f0 0x7fb11806edb0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:05.642 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:04:05.640+0000 7fb137200640 1 --2- 172.21.15.96:0/3072991025 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb13815e9a0 0x7fb1381f7e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:05.642 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:04:05.640+0000 7fb137200640 1 --2- 172.21.15.96:0/3072991025 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb1380d27b0 0x7fb1381560a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:05.642 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:04:05.640+0000 7fb137200640 1 --2- 172.21.15.96:0/3072991025 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb1380b6490 0x7fb1380c9a20 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:05.643 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:04:05.640+0000 7fb137200640 1 -- 172.21.15.96:0/3072991025 >> 172.21.15.96:0/3072991025 conn(0x7fb13808bf60 msgr2=0x7fb13815cb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:05.643 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:04:05.640+0000 7fb137200640 1 -- 172.21.15.96:0/3072991025 shutdown_connections 2024-05-28T06:04:05.643 INFO:teuthology.misc.health.smithi096.stderr:2024-05-28T06:04:05.640+0000 7fb137200640 1 -- 172.21.15.96:0/3072991025 wait complete. 2024-05-28T06:04:05.655 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-05-28T06:04:05.656 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-05-28T06:04:05.656 INFO:tasks.ceph.mds.a:Restarting daemon 2024-05-28T06:04:05.656 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-05-28T06:04:05.661 INFO:tasks.ceph.mds.a:Started 2024-05-28T06:04:05.661 INFO:tasks.ceph.mds.c:Restarting daemon 2024-05-28T06:04:05.661 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-05-28T06:04:05.666 INFO:tasks.ceph.mds.c:Started 2024-05-28T06:04:05.667 INFO:tasks.ceph.mds.b:Restarting daemon 2024-05-28T06:04:05.667 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-05-28T06:04:05.675 INFO:tasks.ceph.mds.b:Started 2024-05-28T06:04:05.675 INFO:tasks.ceph.mds.d:Restarting daemon 2024-05-28T06:04:05.675 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-05-28T06:04:05.683 INFO:tasks.ceph.mds.d:Started 2024-05-28T06:04:05.683 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-05-28T06:04:05.683 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:04:05.799 INFO:tasks.ceph.mds.a.smithi096.stdout:starting mds.a at 2024-05-28T06:04:05.801 INFO:tasks.ceph.mds.c.smithi096.stdout:starting mds.c at 2024-05-28T06:04:05.832 INFO:tasks.ceph.mds.b.smithi123.stdout:starting mds.b at 2024-05-28T06:04:05.837 INFO:tasks.ceph.mds.d.smithi123.stdout:starting mds.d at 2024-05-28T06:04:05.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:05.852+0000 7f232a200640 1 Processor -- start 2024-05-28T06:04:05.854 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:05.852+0000 7f232a200640 1 -- start start 2024-05-28T06:04:05.855 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:05.853+0000 7f232a200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f23240d2db0 0x7f23240b3b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:05.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:05.853+0000 7f232a200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f23240d4d60 0x7f23240d3190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:05.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:05.853+0000 7f232a200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f23240cf490 0x7f23240cf920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:05.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:05.853+0000 7f232a200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f23240d5540 con 0x7f23240d2db0 2024-05-28T06:04:05.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:05.853+0000 7f232a200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f23240d36d0 con 0x7f23240d4d60 2024-05-28T06:04:05.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:05.853+0000 7f232a200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f2324163500 con 0x7f23240cf490 2024-05-28T06:04:05.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:05.853+0000 7f2323400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f23240d2db0 0x7f23240b3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:05.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:05.853+0000 7f2323400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f23240d2db0 0x7f23240b3b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:33796/0 (socket says 172.21.15.96:33796) 2024-05-28T06:04:05.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:05.853+0000 7f2323400640 1 -- 172.21.15.96:0/1975689345 learned_addr learned my addr 172.21.15.96:0/1975689345 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:05.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:05.854+0000 7f2322a00640 1 --2- 172.21.15.96:0/1975689345 >> v2:172.21.15.123:3300/0 conn(0x7f23240d4d60 0x7f23240d3190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:05.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:05.854+0000 7f2323e00640 1 --2- 172.21.15.96:0/1975689345 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f23240cf490 0x7f23240cf920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:05.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:05.854+0000 7f2323400640 1 -- 172.21.15.96:0/1975689345 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f23240cf490 msgr2=0x7f23240cf920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:05.857 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:05.854+0000 7f2323400640 1 --2- 172.21.15.96:0/1975689345 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f23240cf490 0x7f23240cf920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:05.857 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:05.854+0000 7f2323400640 1 -- 172.21.15.96:0/1975689345 >> v2:172.21.15.123:3300/0 conn(0x7f23240d4d60 msgr2=0x7f23240d3190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:05.858 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:05.854+0000 7f2323400640 1 --2- 172.21.15.96:0/1975689345 >> v2:172.21.15.123:3300/0 conn(0x7f23240d4d60 0x7f23240d3190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:05.858 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:05.854+0000 7f2323400640 1 -- 172.21.15.96:0/1975689345 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2324162740 con 0x7f23240d2db0 2024-05-28T06:04:05.858 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:05.854+0000 7f2322a00640 1 --2- 172.21.15.96:0/1975689345 >> v2:172.21.15.123:3300/0 conn(0x7f23240d4d60 0x7f23240d3190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-28T06:04:05.860 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:05.855+0000 7f2323400640 1 --2- 172.21.15.96:0/1975689345 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f23240d2db0 0x7f23240b3b50 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f23240b71b0 tx=0x7f231402a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa40861fbb4b9f6e server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:05.860 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:05.855+0000 7f2322000640 1 -- 172.21.15.96:0/1975689345 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2314037070 con 0x7f23240d2db0 2024-05-28T06:04:05.860 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:05.855+0000 7f2322000640 1 -- 172.21.15.96:0/1975689345 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2314032440 con 0x7f23240d2db0 2024-05-28T06:04:05.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:05.855+0000 7f2322000640 1 -- 172.21.15.96:0/1975689345 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2314032810 con 0x7f23240d2db0 2024-05-28T06:04:05.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:05.855+0000 7f232a200640 1 -- 172.21.15.96:0/1975689345 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f23240d2db0 msgr2=0x7f23240b3b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:05.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:05.855+0000 7f232a200640 1 --2- 172.21.15.96:0/1975689345 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f23240d2db0 0x7f23240b3b50 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f23240b71b0 tx=0x7f231402a140 comp rx=0 tx=0).stop 2024-05-28T06:04:05.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:05.856+0000 7f232a200640 1 -- 172.21.15.96:0/1975689345 shutdown_connections 2024-05-28T06:04:05.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:05.856+0000 7f232a200640 1 --2- 172.21.15.96:0/1975689345 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f23240cf490 0x7f23240cf920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:05.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:05.856+0000 7f232a200640 1 --2- 172.21.15.96:0/1975689345 >> v2:172.21.15.123:3300/0 conn(0x7f23240d4d60 0x7f23240d3190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:05.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:05.856+0000 7f232a200640 1 --2- 172.21.15.96:0/1975689345 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f23240d2db0 0x7f23240b3b50 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:05.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:05.856+0000 7f232a200640 1 -- 172.21.15.96:0/1975689345 >> 172.21.15.96:0/1975689345 conn(0x7f232408bfc0 msgr2=0x7f23240d4650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:05.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:05.856+0000 7f232a200640 1 -- 172.21.15.96:0/1975689345 shutdown_connections 2024-05-28T06:04:05.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:05.856+0000 7f232a200640 1 -- 172.21.15.96:0/1975689345 wait complete. 2024-05-28T06:04:05.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:05.857+0000 7f232a200640 1 Processor -- start 2024-05-28T06:04:05.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:05.858+0000 7f232a200640 1 -- start start 2024-05-28T06:04:05.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:05.858+0000 7f232a200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f23240cf490 0x7f23240ca4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:05.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:05.858+0000 7f232a200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f23240d2db0 0x7f2324154250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:05.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:05.858+0000 7f232a200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f23240d4d60 0x7f23240cd810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:05.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:05.858+0000 7f232a200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f2324162fe0 con 0x7f23240d4d60 2024-05-28T06:04:05.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:05.859+0000 7f232a200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f2324162d00 con 0x7f23240d2db0 2024-05-28T06:04:05.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:05.859+0000 7f232a200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f23240d51a0 con 0x7f23240cf490 2024-05-28T06:04:05.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:05.859+0000 7f2323e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f23240d4d60 0x7f23240cd810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:05.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:05.859+0000 7f2323400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f23240cf490 0x7f23240ca4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:05.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:05.859+0000 7f2322a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f23240d2db0 0x7f2324154250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:05.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:05.859+0000 7f2323e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f23240d4d60 0x7f23240cd810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:33806/0 (socket says 172.21.15.96:33806) 2024-05-28T06:04:05.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:05.859+0000 7f2323e00640 1 -- 172.21.15.96:0/1759225461 learned_addr learned my addr 172.21.15.96:0/1759225461 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:05.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:05.859+0000 7f2323e00640 1 -- 172.21.15.96:0/1759225461 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f23240cf490 msgr2=0x7f23240ca4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:05.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:05.859+0000 7f2323e00640 1 --2- 172.21.15.96:0/1759225461 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f23240cf490 0x7f23240ca4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:05.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:05.859+0000 7f2323e00640 1 -- 172.21.15.96:0/1759225461 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f23240d2db0 msgr2=0x7f2324154250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:05.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:05.859+0000 7f2323e00640 1 --2- 172.21.15.96:0/1759225461 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f23240d2db0 0x7f2324154250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:05.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:05.859+0000 7f2323e00640 1 -- 172.21.15.96:0/1759225461 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23240c9410 con 0x7f23240d4d60 2024-05-28T06:04:05.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:05.860+0000 7f2323e00640 1 --2- 172.21.15.96:0/1759225461 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f23240d4d60 0x7f23240cd810 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f231c007d20 tx=0x7f231c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:05.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:05.860+0000 7f231b400640 1 -- 172.21.15.96:0/1759225461 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f231c014420 con 0x7f23240d4d60 2024-05-28T06:04:05.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:05.860+0000 7f231b400640 1 -- 172.21.15.96:0/1759225461 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f231c019070 con 0x7f23240d4d60 2024-05-28T06:04:05.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:05.860+0000 7f231b400640 1 -- 172.21.15.96:0/1759225461 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f231c014820 con 0x7f23240d4d60 2024-05-28T06:04:05.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:05.860+0000 7f232a200640 1 -- 172.21.15.96:0/1759225461 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23240c96f0 con 0x7f23240d4d60 2024-05-28T06:04:05.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:05.860+0000 7f232a200640 1 -- 172.21.15.96:0/1759225461 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23240c9cb0 con 0x7f23240d4d60 2024-05-28T06:04:05.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:05.861+0000 7f232a200640 1 -- 172.21.15.96:0/1759225461 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23240ce960 con 0x7f23240d4d60 2024-05-28T06:04:05.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:05.862+0000 7f231b400640 1 -- 172.21.15.96:0/1759225461 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f231c0149c0 con 0x7f23240d4d60 2024-05-28T06:04:05.865 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:05.863+0000 7f231b400640 1 --2- 172.21.15.96:0/1759225461 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f230406c920 0x7f230406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:05.865 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:05.863+0000 7f231b400640 1 -- 172.21.15.96:0/1759225461 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f231c08be50 con 0x7f23240d4d60 2024-05-28T06:04:05.865 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:05.863+0000 7f2323400640 1 --2- 172.21.15.96:0/1759225461 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f230406c920 0x7f230406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:05.865 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:05.864+0000 7f2323400640 1 --2- 172.21.15.96:0/1759225461 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f230406c920 0x7f230406ede0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2314034000 tx=0x7f231402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:05.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:05.868+0000 7f231b400640 1 -- 172.21.15.96:0/1759225461 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f23240ce960 con 0x7f23240d4d60 2024-05-28T06:04:06.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.066+0000 7f232a200640 1 -- 172.21.15.96:0/1759225461 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f23240689e0 con 0x7f23240d4d60 2024-05-28T06:04:06.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.068+0000 7f231b400640 1 -- 172.21.15.96:0/1759225461 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) ==== 74+0+12896 (secure 0 0 0) 0x7f231c05a010 con 0x7f23240d4d60 2024-05-28T06:04:06.069 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:04:06.069 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":16,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:04:01.891977+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:04:06.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.070+0000 7f232a200640 1 -- 172.21.15.96:0/1759225461 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f230406c920 msgr2=0x7f230406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:06.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.070+0000 7f232a200640 1 --2- 172.21.15.96:0/1759225461 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f230406c920 0x7f230406ede0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2314034000 tx=0x7f231402e000 comp rx=0 tx=0).stop 2024-05-28T06:04:06.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.070+0000 7f232a200640 1 -- 172.21.15.96:0/1759225461 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f23240d4d60 msgr2=0x7f23240cd810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:06.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.070+0000 7f232a200640 1 --2- 172.21.15.96:0/1759225461 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f23240d4d60 0x7f23240cd810 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f231c007d20 tx=0x7f231c00e4b0 comp rx=0 tx=0).stop 2024-05-28T06:04:06.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.070+0000 7f232a200640 1 -- 172.21.15.96:0/1759225461 shutdown_connections 2024-05-28T06:04:06.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.070+0000 7f232a200640 1 --2- 172.21.15.96:0/1759225461 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f230406c920 0x7f230406ede0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:06.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.070+0000 7f232a200640 1 --2- 172.21.15.96:0/1759225461 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f23240d4d60 0x7f23240cd810 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:06.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.070+0000 7f232a200640 1 --2- 172.21.15.96:0/1759225461 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f23240d2db0 0x7f2324154250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:06.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.070+0000 7f232a200640 1 --2- 172.21.15.96:0/1759225461 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f23240cf490 0x7f23240ca4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:06.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.070+0000 7f232a200640 1 -- 172.21.15.96:0/1759225461 >> 172.21.15.96:0/1759225461 conn(0x7f232408bfc0 msgr2=0x7f232415c9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:06.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.070+0000 7f232a200640 1 -- 172.21.15.96:0/1759225461 shutdown_connections 2024-05-28T06:04:06.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.070+0000 7f232a200640 1 -- 172.21.15.96:0/1759225461 wait complete. 2024-05-28T06:04:06.089 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-28T06:03:59.517102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-28T06:04:06.090 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-28T06:04:06.292 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.291+0000 7f97dd400640 1 Processor -- start 2024-05-28T06:04:06.292 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.291+0000 7f97dd400640 1 -- start start 2024-05-28T06:04:06.294 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.292+0000 7f97dd400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f97d80d0640 0x7f97d815d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:06.294 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.292+0000 7f97dd400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f97d80d0a20 0x7f97d80cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:06.294 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.292+0000 7f97dd400640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97d815cb10 con 0x7f97d80b32c0 2024-05-28T06:04:06.294 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.292+0000 7f97dd400640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f97d80adbb0 con 0x7f97d80d0a20 2024-05-28T06:04:06.294 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.292+0000 7f97dd400640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f97d815db10 con 0x7f97d80d0640 2024-05-28T06:04:06.294 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.293+0000 7f97cfe00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f97d80d0a20 0x7f97d80cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:06.294 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.293+0000 7f97cfe00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f97d80d0a20 0x7f97d80cf950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:33814/0 (socket says 172.21.15.96:33814) 2024-05-28T06:04:06.294 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.293+0000 7f97cfe00640 1 -- 172.21.15.96:0/2724332769 learned_addr learned my addr 172.21.15.96:0/2724332769 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:06.295 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.293+0000 7f97d6800640 1 --2- 172.21.15.96:0/2724332769 >> v2:172.21.15.123:3300/0 conn(0x7f97d80d0640 0x7f97d815d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:06.295 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.293+0000 7f97cfe00640 1 -- 172.21.15.96:0/2724332769 >> v2:172.21.15.123:3300/0 conn(0x7f97d80d0640 msgr2=0x7f97d815d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:06.295 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.293+0000 7f97cfe00640 1 --2- 172.21.15.96:0/2724332769 >> v2:172.21.15.123:3300/0 conn(0x7f97d80d0640 0x7f97d815d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:06.295 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.293+0000 7f97cfe00640 1 -- 172.21.15.96:0/2724332769 >> v1:172.21.15.123:6789/0 conn(0x7f97d80b32c0 legacy=0x7f97d80b3730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:06.295 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.293+0000 7f97cfe00640 1 -- 172.21.15.96:0/2724332769 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97d80ad5f0 con 0x7f97d80d0a20 2024-05-28T06:04:06.295 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.293+0000 7f97d6800640 1 --2- 172.21.15.96:0/2724332769 >> v2:172.21.15.123:3300/0 conn(0x7f97d80d0640 0x7f97d815d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:04:06.297 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.294+0000 7f97cfe00640 1 --2- 172.21.15.96:0/2724332769 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97d80d0a20 0x7f97d80cf950 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f97d8074150 tx=0x7f97c002a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b470e133ebe17b4f server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:06.297 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.295+0000 7f97d5e00640 1 -- 172.21.15.96:0/2724332769 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97c0037070 con 0x7f97d80d0a20 2024-05-28T06:04:06.297 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.295+0000 7f97d5e00640 1 -- 172.21.15.96:0/2724332769 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97c0032440 con 0x7f97d80d0a20 2024-05-28T06:04:06.297 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.295+0000 7f97d5e00640 1 -- 172.21.15.96:0/2724332769 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97c0032810 con 0x7f97d80d0a20 2024-05-28T06:04:06.297 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.295+0000 7f97dd400640 1 -- 172.21.15.96:0/2724332769 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97d80d0a20 msgr2=0x7f97d80cf950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:06.297 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.295+0000 7f97dd400640 1 --2- 172.21.15.96:0/2724332769 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97d80d0a20 0x7f97d80cf950 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f97d8074150 tx=0x7f97c002a1c0 comp rx=0 tx=0).stop 2024-05-28T06:04:06.297 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.295+0000 7f97dd400640 1 -- 172.21.15.96:0/2724332769 shutdown_connections 2024-05-28T06:04:06.297 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.295+0000 7f97dd400640 1 --2- 172.21.15.96:0/2724332769 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97d80d0a20 0x7f97d80cf950 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:06.297 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.295+0000 7f97dd400640 1 --2- 172.21.15.96:0/2724332769 >> v2:172.21.15.123:3300/0 conn(0x7f97d80d0640 0x7f97d815d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:06.297 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.295+0000 7f97dd400640 1 -- 172.21.15.96:0/2724332769 >> 172.21.15.96:0/2724332769 conn(0x7f97d806bd20 msgr2=0x7f97d806c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:06.297 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.295+0000 7f97dd400640 1 -- 172.21.15.96:0/2724332769 shutdown_connections 2024-05-28T06:04:06.298 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.295+0000 7f97dd400640 1 -- 172.21.15.96:0/2724332769 wait complete. 2024-05-28T06:04:06.298 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.297+0000 7f97dd400640 1 Processor -- start 2024-05-28T06:04:06.298 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.297+0000 7f97dd400640 1 -- start start 2024-05-28T06:04:06.301 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.298+0000 7f97dd400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97d80b32c0 0x7f97d815b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:06.301 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.298+0000 7f97dd400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f97d80d0640 0x7f97d806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:06.301 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.298+0000 7f97dd400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f97d80d0a20 0x7f97d81f6d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:06.301 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.298+0000 7f97dd400640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f97d815d0b0 con 0x7f97d80b32c0 2024-05-28T06:04:06.301 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.298+0000 7f97dd400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f97d815cdd0 con 0x7f97d80d0a20 2024-05-28T06:04:06.301 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.298+0000 7f97dd400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f97d80b3e40 con 0x7f97d80d0640 2024-05-28T06:04:06.302 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.299+0000 7f97d6800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97d80b32c0 0x7f97d815b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:06.302 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.299+0000 7f97d6800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97d80b32c0 0x7f97d815b4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:33820/0 (socket says 172.21.15.96:33820) 2024-05-28T06:04:06.302 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.299+0000 7f97d6800640 1 -- 172.21.15.96:0/757314831 learned_addr learned my addr 172.21.15.96:0/757314831 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:06.302 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.299+0000 7f97cfe00640 1 --2- 172.21.15.96:0/757314831 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f97d80d0640 0x7f97d806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:06.302 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.299+0000 7f97d7200640 1 --2- 172.21.15.96:0/757314831 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f97d80d0a20 0x7f97d81f6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:06.302 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.299+0000 7f97d6800640 1 -- 172.21.15.96:0/757314831 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f97d80d0640 msgr2=0x7f97d806d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:06.302 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.299+0000 7f97d6800640 1 --2- 172.21.15.96:0/757314831 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f97d80d0640 0x7f97d806d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:06.302 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.299+0000 7f97d6800640 1 -- 172.21.15.96:0/757314831 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f97d80d0a20 msgr2=0x7f97d81f6d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:06.302 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.299+0000 7f97d6800640 1 --2- 172.21.15.96:0/757314831 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f97d80d0a20 0x7f97d81f6d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:06.302 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.299+0000 7f97d6800640 1 -- 172.21.15.96:0/757314831 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97d806e460 con 0x7f97d80b32c0 2024-05-28T06:04:06.302 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.299+0000 7f97cfe00640 1 --2- 172.21.15.96:0/757314831 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f97d80d0640 0x7f97d806d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:04:06.303 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.299+0000 7f97d7200640 1 --2- 172.21.15.96:0/757314831 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f97d80d0a20 0x7f97d81f6d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:04:06.303 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.299+0000 7f97d6800640 1 --2- 172.21.15.96:0/757314831 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97d80b32c0 0x7f97d815b4b0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f97c800c820 tx=0x7f97c800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:06.303 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.300+0000 7f97cea00640 1 -- 172.21.15.96:0/757314831 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97c8015070 con 0x7f97d80b32c0 2024-05-28T06:04:06.303 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.300+0000 7f97cea00640 1 -- 172.21.15.96:0/757314831 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97c8007a30 con 0x7f97d80b32c0 2024-05-28T06:04:06.303 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.300+0000 7f97cea00640 1 -- 172.21.15.96:0/757314831 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97c8007e00 con 0x7f97d80b32c0 2024-05-28T06:04:06.303 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.300+0000 7f97dd400640 1 -- 172.21.15.96:0/757314831 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97d806e740 con 0x7f97d80b32c0 2024-05-28T06:04:06.303 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.300+0000 7f97dd400640 1 -- 172.21.15.96:0/757314831 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97d806ed00 con 0x7f97d80b32c0 2024-05-28T06:04:06.304 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.301+0000 7f97dd400640 1 -- 172.21.15.96:0/757314831 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97d80689e0 con 0x7f97d80b32c0 2024-05-28T06:04:06.304 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.302+0000 7f97cea00640 1 -- 172.21.15.96:0/757314831 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f97c8018070 con 0x7f97d80b32c0 2024-05-28T06:04:06.304 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.303+0000 7f97cea00640 1 --2- 172.21.15.96:0/757314831 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f97b406c8f0 0x7f97b406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:06.304 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.303+0000 7f97cea00640 1 -- 172.21.15.96:0/757314831 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f97c808b7e0 con 0x7f97d80b32c0 2024-05-28T06:04:06.304 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.303+0000 7f97cfe00640 1 --2- 172.21.15.96:0/757314831 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f97b406c8f0 0x7f97b406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:06.305 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.304+0000 7f97cfe00640 1 --2- 172.21.15.96:0/757314831 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f97b406c8f0 0x7f97b406edb0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f97c0034000 tx=0x7f97c002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:06.308 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.307+0000 7f97cea00640 1 -- 172.21.15.96:0/757314831 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f97c8059f50 con 0x7f97d80b32c0 2024-05-28T06:04:06.514 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.513+0000 7f97dd400640 1 -- 172.21.15.96:0/757314831 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f97d8071ef0 con 0x7f97d80b32c0 2024-05-28T06:04:06.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.514+0000 7f97cea00640 1 -- 172.21.15.96:0/757314831 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) ==== 93+0+10 (secure 0 0 0) 0x7f97c8059850 con 0x7f97d80b32c0 2024-05-28T06:04:06.515 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 1 2024-05-28T06:04:06.518 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.516+0000 7f97dd400640 1 -- 172.21.15.96:0/757314831 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f97b406c8f0 msgr2=0x7f97b406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:06.518 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.516+0000 7f97dd400640 1 --2- 172.21.15.96:0/757314831 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f97b406c8f0 0x7f97b406edb0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f97c0034000 tx=0x7f97c002e000 comp rx=0 tx=0).stop 2024-05-28T06:04:06.518 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.516+0000 7f97dd400640 1 -- 172.21.15.96:0/757314831 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97d80b32c0 msgr2=0x7f97d815b4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:06.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.516+0000 7f97dd400640 1 --2- 172.21.15.96:0/757314831 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97d80b32c0 0x7f97d815b4b0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f97c800c820 tx=0x7f97c800cc20 comp rx=0 tx=0).stop 2024-05-28T06:04:06.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.517+0000 7f97dd400640 1 -- 172.21.15.96:0/757314831 shutdown_connections 2024-05-28T06:04:06.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.517+0000 7f97dd400640 1 --2- 172.21.15.96:0/757314831 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f97b406c8f0 0x7f97b406edb0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:06.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.517+0000 7f97dd400640 1 --2- 172.21.15.96:0/757314831 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f97d80d0a20 0x7f97d81f6d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:06.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.517+0000 7f97dd400640 1 --2- 172.21.15.96:0/757314831 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f97d80d0640 0x7f97d806d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:06.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.517+0000 7f97dd400640 1 --2- 172.21.15.96:0/757314831 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97d80b32c0 0x7f97d815b4b0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:06.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.517+0000 7f97dd400640 1 -- 172.21.15.96:0/757314831 >> 172.21.15.96:0/757314831 conn(0x7f97d806bd20 msgr2=0x7f97d806c100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:06.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.517+0000 7f97dd400640 1 -- 172.21.15.96:0/757314831 shutdown_connections 2024-05-28T06:04:06.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.517+0000 7f97dd400640 1 -- 172.21.15.96:0/757314831 wait complete. 2024-05-28T06:04:06.537 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-28T06:04:06.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.734+0000 7fa372000640 1 Processor -- start 2024-05-28T06:04:06.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.734+0000 7fa372000640 1 -- start start 2024-05-28T06:04:06.737 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.736+0000 7fa372000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fa36c15e8a0 0x7fa36c0d0090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:06.737 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.736+0000 7fa372000640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa36c163330 con 0x7fa36c162cd0 2024-05-28T06:04:06.737 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.736+0000 7fa372000640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa36c0cf2d0 con 0x7fa36c0d2b70 2024-05-28T06:04:06.737 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.736+0000 7fa372000640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fa36c0cf5b0 con 0x7fa36c15e8a0 2024-05-28T06:04:06.737 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.736+0000 7fa371600640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fa36c162cd0 0x7fa36c06cb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:42148/0 (socket says 172.21.15.96:42148) 2024-05-28T06:04:06.737 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.736+0000 7fa371600640 1 -- 172.21.15.96:0/297215604 learned_addr learned my addr 172.21.15.96:0/297215604 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:04:06.740 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.736+0000 7fa370c00640 1 --2- 172.21.15.96:0/297215604 >> v2:172.21.15.123:3300/0 conn(0x7fa36c15e8a0 0x7fa36c0d0090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:06.740 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.736+0000 7fa36b400640 1 -- 172.21.15.96:0/297215604 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 40737319 0 0) 0x7fa36c163330 con 0x7fa36c162cd0 2024-05-28T06:04:06.740 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.736+0000 7fa36b400640 1 -- 172.21.15.96:0/297215604 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa354003520 con 0x7fa36c162cd0 2024-05-28T06:04:06.740 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.737+0000 7fa36b400640 1 -- 172.21.15.96:0/297215604 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 121184746 0 0) 0x7fa354003520 con 0x7fa36c162cd0 2024-05-28T06:04:06.741 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.737+0000 7fa36b400640 1 -- 172.21.15.96:0/297215604 >> v2:172.21.15.123:3300/0 conn(0x7fa36c15e8a0 msgr2=0x7fa36c0d0090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:06.741 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.737+0000 7fa36b400640 1 --2- 172.21.15.96:0/297215604 >> v2:172.21.15.123:3300/0 conn(0x7fa36c15e8a0 0x7fa36c0d0090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:06.741 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.737+0000 7fa36b400640 1 -- 172.21.15.96:0/297215604 >> v1:172.21.15.123:6789/0 conn(0x7fa36c0d2b70 legacy=0x7fa36c1625c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:06.741 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.737+0000 7fa36b400640 1 -- 172.21.15.96:0/297215604 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa36c0cf890 con 0x7fa36c162cd0 2024-05-28T06:04:06.741 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.737+0000 7fa36b400640 1 -- 172.21.15.96:0/297215604 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7fa3640019f0 con 0x7fa36c162cd0 2024-05-28T06:04:06.741 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.737+0000 7fa36b400640 1 -- 172.21.15.96:0/297215604 >> v1:172.21.15.96:6789/0 conn(0x7fa36c162cd0 legacy=0x7fa36c06cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:06.741 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.737+0000 7fa36b400640 1 --2- 172.21.15.96:0/297215604 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa3540039b0 0x7fa354003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:06.741 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.737+0000 7fa36b400640 1 --2- 172.21.15.96:0/297215604 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa354004800 0x7fa354004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:06.741 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.737+0000 7fa36b400640 1 --2- 172.21.15.96:0/297215604 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa3540051c0 0x7fa35400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:06.741 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.737+0000 7fa36b400640 1 -- 172.21.15.96:0/297215604 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa36c0cf5b0 con 0x7fa3540051c0 2024-05-28T06:04:06.742 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.737+0000 7fa36b400640 1 -- 172.21.15.96:0/297215604 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa35400c060 con 0x7fa3540039b0 2024-05-28T06:04:06.742 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.737+0000 7fa36b400640 1 -- 172.21.15.96:0/297215604 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fa35400c260 con 0x7fa354004800 2024-05-28T06:04:06.742 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.737+0000 7fa371600640 1 --2- 172.21.15.96:0/297215604 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa3540051c0 0x7fa35400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:06.742 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.737+0000 7fa370c00640 1 --2- 172.21.15.96:0/297215604 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa3540039b0 0x7fa354003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:06.742 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.737+0000 7fa36be00640 1 --2- 172.21.15.96:0/297215604 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa354004800 0x7fa354004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:06.742 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.738+0000 7fa371600640 1 -- 172.21.15.96:0/297215604 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa354004800 msgr2=0x7fa354004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:06.742 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.738+0000 7fa371600640 1 --2- 172.21.15.96:0/297215604 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa354004800 0x7fa354004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:06.742 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.738+0000 7fa371600640 1 -- 172.21.15.96:0/297215604 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa3540039b0 msgr2=0x7fa354003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:06.743 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.738+0000 7fa371600640 1 --2- 172.21.15.96:0/297215604 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa3540039b0 0x7fa354003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:06.743 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.738+0000 7fa371600640 1 -- 172.21.15.96:0/297215604 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa354003520 con 0x7fa3540051c0 2024-05-28T06:04:06.743 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.738+0000 7fa36be00640 1 --2- 172.21.15.96:0/297215604 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa354004800 0x7fa354004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:04:06.743 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.738+0000 7fa371600640 1 -- 172.21.15.96:0/297215604 reap_dead start 2024-05-28T06:04:06.743 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.738+0000 7fa371600640 1 --2- 172.21.15.96:0/297215604 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa3540051c0 0x7fa35400b9c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa36400a850 tx=0x7fa354005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7020b7502744a862 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:06.743 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.739+0000 7fa36b400640 1 -- 172.21.15.96:0/297215604 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa36400a4a0 con 0x7fa3540051c0 2024-05-28T06:04:06.743 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.739+0000 7fa36b400640 1 -- 172.21.15.96:0/297215604 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa36402f070 con 0x7fa3540051c0 2024-05-28T06:04:06.743 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.739+0000 7fa36b400640 1 -- 172.21.15.96:0/297215604 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa36402bce0 con 0x7fa3540051c0 2024-05-28T06:04:06.746 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.739+0000 7fa372000640 1 -- 172.21.15.96:0/297215604 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa3540051c0 msgr2=0x7fa35400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:06.746 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.739+0000 7fa372000640 1 --2- 172.21.15.96:0/297215604 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa3540051c0 0x7fa35400b9c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa36400a850 tx=0x7fa354005830 comp rx=0 tx=0).stop 2024-05-28T06:04:06.746 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.739+0000 7fa372000640 1 -- 172.21.15.96:0/297215604 shutdown_connections 2024-05-28T06:04:06.746 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.739+0000 7fa372000640 1 --2- 172.21.15.96:0/297215604 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa3540051c0 0x7fa35400b9c0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:06.746 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.739+0000 7fa372000640 1 -- 172.21.15.96:0/297215604 >> 172.21.15.96:0/297215604 conn(0x7fa36c08bfc0 msgr2=0x7fa36c0d4590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:06.747 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.739+0000 7fa372000640 1 -- 172.21.15.96:0/297215604 shutdown_connections 2024-05-28T06:04:06.747 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.739+0000 7fa372000640 1 -- 172.21.15.96:0/297215604 wait complete. 2024-05-28T06:04:06.747 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.741+0000 7fa372000640 1 Processor -- start 2024-05-28T06:04:06.747 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.741+0000 7fa372000640 1 -- start start 2024-05-28T06:04:06.747 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.742+0000 7fa372000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa3540051c0 0x7fa36c0b8250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:06.747 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.742+0000 7fa372000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa36c0cd540 0x7fa36c157c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:06.747 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.742+0000 7fa372000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa36c0cdd30 0x7fa36c1ea450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:06.747 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.742+0000 7fa372000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa36c15ee00 con 0x7fa36c0cd540 2024-05-28T06:04:06.747 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.742+0000 7fa372000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa36c15ec80 con 0x7fa3540051c0 2024-05-28T06:04:06.747 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.742+0000 7fa372000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fa36c0d0090 con 0x7fa36c0cdd30 2024-05-28T06:04:06.747 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.742+0000 7fa36be00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa36c0cd540 0x7fa36c157c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:06.747 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.743+0000 7fa36be00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa36c0cd540 0x7fa36c157c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:33850/0 (socket says 172.21.15.96:33850) 2024-05-28T06:04:06.748 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.743+0000 7fa36be00640 1 -- 172.21.15.96:0/415183329 learned_addr learned my addr 172.21.15.96:0/415183329 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:06.748 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.743+0000 7fa370c00640 1 --2- 172.21.15.96:0/415183329 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa3540051c0 0x7fa36c0b8250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:06.748 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.743+0000 7fa371600640 1 --2- 172.21.15.96:0/415183329 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa36c0cdd30 0x7fa36c1ea450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:06.748 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.743+0000 7fa36be00640 1 -- 172.21.15.96:0/415183329 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa36c0cdd30 msgr2=0x7fa36c1ea450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:06.748 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.743+0000 7fa36be00640 1 --2- 172.21.15.96:0/415183329 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa36c0cdd30 0x7fa36c1ea450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:06.748 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.743+0000 7fa36be00640 1 -- 172.21.15.96:0/415183329 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa3540051c0 msgr2=0x7fa36c0b8250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:06.748 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.743+0000 7fa36be00640 1 --2- 172.21.15.96:0/415183329 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa3540051c0 0x7fa36c0b8250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:06.748 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.743+0000 7fa36be00640 1 -- 172.21.15.96:0/415183329 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa36c0b71b0 con 0x7fa36c0cd540 2024-05-28T06:04:06.748 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.743+0000 7fa36be00640 1 --2- 172.21.15.96:0/415183329 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa36c0cd540 0x7fa36c157c60 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa35800d770 tx=0x7fa35800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:06.748 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.744+0000 7fa368c00640 1 -- 172.21.15.96:0/415183329 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa358005270 con 0x7fa36c0cd540 2024-05-28T06:04:06.748 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.744+0000 7fa368c00640 1 -- 172.21.15.96:0/415183329 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa358017070 con 0x7fa36c0cd540 2024-05-28T06:04:06.748 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.744+0000 7fa368c00640 1 -- 172.21.15.96:0/415183329 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa35800b6b0 con 0x7fa36c0cd540 2024-05-28T06:04:06.748 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.744+0000 7fa372000640 1 -- 172.21.15.96:0/415183329 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa36c0b7490 con 0x7fa36c0cd540 2024-05-28T06:04:06.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.744+0000 7fa372000640 1 -- 172.21.15.96:0/415183329 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa36c0b7a50 con 0x7fa36c0cd540 2024-05-28T06:04:06.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.745+0000 7fa372000640 1 -- 172.21.15.96:0/415183329 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa36c0689e0 con 0x7fa36c0cd540 2024-05-28T06:04:06.750 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.746+0000 7fa368c00640 1 -- 172.21.15.96:0/415183329 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa35800b850 con 0x7fa36c0cd540 2024-05-28T06:04:06.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.747+0000 7fa368c00640 1 --2- 172.21.15.96:0/415183329 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa34c06c8f0 0x7fa34c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:06.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.747+0000 7fa368c00640 1 -- 172.21.15.96:0/415183329 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7fa35808b670 con 0x7fa36c0cd540 2024-05-28T06:04:06.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.747+0000 7fa370c00640 1 --2- 172.21.15.96:0/415183329 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa34c06c8f0 0x7fa34c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:06.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.748+0000 7fa370c00640 1 --2- 172.21.15.96:0/415183329 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa34c06c8f0 0x7fa34c06edb0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa360009990 tx=0x7fa360008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:06.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.751+0000 7fa368c00640 1 -- 172.21.15.96:0/415183329 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa358019310 con 0x7fa36c0cd540 2024-05-28T06:04:06.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.989+0000 7fa372000640 1 -- 172.21.15.96:0/415183329 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa36c0d2950 con 0x7fa36c0cd540 2024-05-28T06:04:06.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.990+0000 7fa368c00640 1 -- 172.21.15.96:0/415183329 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+3867 (secure 0 0 0) 0x7fa358001230 con 0x7fa36c0cd540 2024-05-28T06:04:06.991 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:04:06.991 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/3349661842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":3349661842},{"type":"v1","addr":"172.21.15.96:6833","nonce":3349661842}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4270,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/3383264102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":3383264102},{"type":"v1","addr":"172.21.15.96:6835","nonce":3383264102}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4279,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/3068402544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":3068402544},{"type":"v1","addr":"172.21.15.123:6837","nonce":3068402544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/605814935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":605814935},{"type":"v1","addr":"172.21.15.123:6835","nonce":605814935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-05-28T06:04:06.991 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 3 2024-05-28T06:04:06.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.992+0000 7fa372000640 1 -- 172.21.15.96:0/415183329 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa34c06c8f0 msgr2=0x7fa34c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:06.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.992+0000 7fa372000640 1 --2- 172.21.15.96:0/415183329 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa34c06c8f0 0x7fa34c06edb0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa360009990 tx=0x7fa360008040 comp rx=0 tx=0).stop 2024-05-28T06:04:06.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.992+0000 7fa372000640 1 -- 172.21.15.96:0/415183329 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa36c0cd540 msgr2=0x7fa36c157c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:06.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.992+0000 7fa372000640 1 --2- 172.21.15.96:0/415183329 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa36c0cd540 0x7fa36c157c60 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa35800d770 tx=0x7fa35800dc30 comp rx=0 tx=0).stop 2024-05-28T06:04:06.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.992+0000 7fa372000640 1 -- 172.21.15.96:0/415183329 shutdown_connections 2024-05-28T06:04:06.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.992+0000 7fa372000640 1 --2- 172.21.15.96:0/415183329 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa34c06c8f0 0x7fa34c06edb0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:06.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.992+0000 7fa372000640 1 --2- 172.21.15.96:0/415183329 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa36c0cdd30 0x7fa36c1ea450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:06.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.992+0000 7fa372000640 1 --2- 172.21.15.96:0/415183329 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa36c0cd540 0x7fa36c157c60 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:06.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.992+0000 7fa372000640 1 --2- 172.21.15.96:0/415183329 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa3540051c0 0x7fa36c0b8250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:06.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.992+0000 7fa372000640 1 -- 172.21.15.96:0/415183329 >> 172.21.15.96:0/415183329 conn(0x7fa36c08bfc0 msgr2=0x7fa36c160480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:06.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.992+0000 7fa372000640 1 -- 172.21.15.96:0/415183329 shutdown_connections 2024-05-28T06:04:06.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:06.992+0000 7fa372000640 1 -- 172.21.15.96:0/415183329 wait complete. 2024-05-28T06:04:07.012 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:04:07.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.202+0000 7fb64b200640 1 Processor -- start 2024-05-28T06:04:07.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.202+0000 7fb64b200640 1 -- start start 2024-05-28T06:04:07.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.204+0000 7fb64b200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb64406f860 0x7fb6440b3150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:07.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.204+0000 7fb64b200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fb6440d0bb0 0x7fb6440ad130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:07.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.204+0000 7fb64b200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb64406ed10 con 0x7fb6440b3690 2024-05-28T06:04:07.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.204+0000 7fb64b200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fb64415cb00 con 0x7fb6440d0bb0 2024-05-28T06:04:07.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.204+0000 7fb64b200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fb6440b3cc0 con 0x7fb64406f860 2024-05-28T06:04:07.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.204+0000 7fb643e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb64406f860 0x7fb6440b3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:07.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.205+0000 7fb643e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb64406f860 0x7fb6440b3150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.96:59800/0 (socket says 172.21.15.96:59800) 2024-05-28T06:04:07.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.205+0000 7fb643e00640 1 -- 172.21.15.96:0/879585968 learned_addr learned my addr 172.21.15.96:0/879585968 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:07.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.205+0000 7fb649200640 1 --2- 172.21.15.96:0/879585968 >> v2:172.21.15.123:3300/0 conn(0x7fb6440d0bb0 0x7fb6440ad130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:07.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.205+0000 7fb643e00640 1 -- 172.21.15.96:0/879585968 >> v2:172.21.15.123:3300/0 conn(0x7fb6440d0bb0 msgr2=0x7fb6440ad130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:07.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.205+0000 7fb643e00640 1 --2- 172.21.15.96:0/879585968 >> v2:172.21.15.123:3300/0 conn(0x7fb6440d0bb0 0x7fb6440ad130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:07.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.205+0000 7fb643e00640 1 -- 172.21.15.96:0/879585968 >> v1:172.21.15.123:6789/0 conn(0x7fb6440b3690 legacy=0x7fb6440d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:07.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.205+0000 7fb643e00640 1 -- 172.21.15.96:0/879585968 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6440b3fa0 con 0x7fb64406f860 2024-05-28T06:04:07.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.207+0000 7fb643e00640 1 --2- 172.21.15.96:0/879585968 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb64406f860 0x7fb6440b3150 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb634009870 tx=0x7fb63402a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce6319bdded50f1d server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:07.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.208+0000 7fb643400640 1 -- 172.21.15.96:0/879585968 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb634037070 con 0x7fb64406f860 2024-05-28T06:04:07.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.208+0000 7fb643400640 1 -- 172.21.15.96:0/879585968 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb634032440 con 0x7fb64406f860 2024-05-28T06:04:07.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.208+0000 7fb643400640 1 -- 172.21.15.96:0/879585968 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb634032810 con 0x7fb64406f860 2024-05-28T06:04:07.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.208+0000 7fb64b200640 1 -- 172.21.15.96:0/879585968 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb64406f860 msgr2=0x7fb6440b3150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:07.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.208+0000 7fb64b200640 1 --2- 172.21.15.96:0/879585968 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb64406f860 0x7fb6440b3150 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb634009870 tx=0x7fb63402a270 comp rx=0 tx=0).stop 2024-05-28T06:04:07.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.208+0000 7fb64b200640 1 -- 172.21.15.96:0/879585968 shutdown_connections 2024-05-28T06:04:07.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.208+0000 7fb64b200640 1 --2- 172.21.15.96:0/879585968 >> v2:172.21.15.123:3300/0 conn(0x7fb6440d0bb0 0x7fb6440ad130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:07.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.208+0000 7fb64b200640 1 --2- 172.21.15.96:0/879585968 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb64406f860 0x7fb6440b3150 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:07.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.208+0000 7fb64b200640 1 -- 172.21.15.96:0/879585968 >> 172.21.15.96:0/879585968 conn(0x7fb64406bae0 msgr2=0x7fb64406bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:07.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.208+0000 7fb64b200640 1 -- 172.21.15.96:0/879585968 shutdown_connections 2024-05-28T06:04:07.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.209+0000 7fb64b200640 1 -- 172.21.15.96:0/879585968 wait complete. 2024-05-28T06:04:07.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.210+0000 7fb64b200640 1 Processor -- start 2024-05-28T06:04:07.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.211+0000 7fb64b200640 1 -- start start 2024-05-28T06:04:07.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.212+0000 7fb64b200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb64406f860 0x7fb6440cde30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:07.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.212+0000 7fb64b200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb6440b3690 0x7fb6440cec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:07.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.212+0000 7fb64b200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb6440d0bb0 0x7fb6440c7330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:07.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.212+0000 7fb64b200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb6440b4840 con 0x7fb6440d0bb0 2024-05-28T06:04:07.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.212+0000 7fb64b200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fb6440b4560 con 0x7fb64406f860 2024-05-28T06:04:07.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.212+0000 7fb64b200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fb6440ad670 con 0x7fb6440b3690 2024-05-28T06:04:07.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.212+0000 7fb649200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb6440d0bb0 0x7fb6440c7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:07.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.212+0000 7fb649200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb6440d0bb0 0x7fb6440c7330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:33866/0 (socket says 172.21.15.96:33866) 2024-05-28T06:04:07.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.212+0000 7fb649200640 1 -- 172.21.15.96:0/4232354351 learned_addr learned my addr 172.21.15.96:0/4232354351 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:07.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.212+0000 7fb63be00640 1 --2- 172.21.15.96:0/4232354351 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb6440b3690 0x7fb6440cec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:07.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.213+0000 7fb643e00640 1 --2- 172.21.15.96:0/4232354351 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb64406f860 0x7fb6440cde30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:07.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.213+0000 7fb649200640 1 -- 172.21.15.96:0/4232354351 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb6440b3690 msgr2=0x7fb6440cec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:07.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.213+0000 7fb649200640 1 --2- 172.21.15.96:0/4232354351 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb6440b3690 0x7fb6440cec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:07.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.213+0000 7fb649200640 1 -- 172.21.15.96:0/4232354351 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb64406f860 msgr2=0x7fb6440cde30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:07.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.213+0000 7fb649200640 1 --2- 172.21.15.96:0/4232354351 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb64406f860 0x7fb6440cde30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:07.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.213+0000 7fb649200640 1 -- 172.21.15.96:0/4232354351 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb64415f1a0 con 0x7fb6440d0bb0 2024-05-28T06:04:07.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.213+0000 7fb649200640 1 --2- 172.21.15.96:0/4232354351 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb6440d0bb0 0x7fb6440c7330 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb63c00ef90 tx=0x7fb63c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:07.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.213+0000 7fb640c00640 1 -- 172.21.15.96:0/4232354351 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb63c018020 con 0x7fb6440d0bb0 2024-05-28T06:04:07.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.213+0000 7fb640c00640 1 -- 172.21.15.96:0/4232354351 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb63c014480 con 0x7fb6440d0bb0 2024-05-28T06:04:07.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.213+0000 7fb640c00640 1 -- 172.21.15.96:0/4232354351 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb63c009e60 con 0x7fb6440d0bb0 2024-05-28T06:04:07.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.213+0000 7fb64b200640 1 -- 172.21.15.96:0/4232354351 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb64415f480 con 0x7fb6440d0bb0 2024-05-28T06:04:07.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.213+0000 7fb64b200640 1 -- 172.21.15.96:0/4232354351 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb64415fa40 con 0x7fb6440d0bb0 2024-05-28T06:04:07.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.215+0000 7fb64b200640 1 -- 172.21.15.96:0/4232354351 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6440cf220 con 0x7fb6440d0bb0 2024-05-28T06:04:07.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.217+0000 7fb640c00640 1 -- 172.21.15.96:0/4232354351 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb63c005d10 con 0x7fb6440d0bb0 2024-05-28T06:04:07.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.217+0000 7fb640c00640 1 --2- 172.21.15.96:0/4232354351 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb62006c8f0 0x7fb62006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:07.218 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.217+0000 7fb640c00640 1 -- 172.21.15.96:0/4232354351 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7fb63c08c570 con 0x7fb6440d0bb0 2024-05-28T06:04:07.218 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.217+0000 7fb643e00640 1 --2- 172.21.15.96:0/4232354351 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb62006c8f0 0x7fb62006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:07.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.219+0000 7fb643e00640 1 --2- 172.21.15.96:0/4232354351 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb62006c8f0 0x7fb62006edb0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb634034000 tx=0x7fb63402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:07.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.221+0000 7fb640c00640 1 -- 172.21.15.96:0/4232354351 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb63c022030 con 0x7fb6440d0bb0 2024-05-28T06:04:07.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.420+0000 7fb64b200640 1 -- 172.21.15.96:0/4232354351 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb6440689e0 con 0x7fb6440d0bb0 2024-05-28T06:04:07.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.422+0000 7fb640c00640 1 -- 172.21.15.96:0/4232354351 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) ==== 74+0+12896 (secure 0 0 0) 0x7fb63c010040 con 0x7fb6440d0bb0 2024-05-28T06:04:07.423 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:04:07.424 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":16,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:04:01.891977+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:04:07.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.425+0000 7fb64b200640 1 -- 172.21.15.96:0/4232354351 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb62006c8f0 msgr2=0x7fb62006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:07.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.425+0000 7fb64b200640 1 --2- 172.21.15.96:0/4232354351 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb62006c8f0 0x7fb62006edb0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb634034000 tx=0x7fb63402e000 comp rx=0 tx=0).stop 2024-05-28T06:04:07.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.425+0000 7fb64b200640 1 -- 172.21.15.96:0/4232354351 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb6440d0bb0 msgr2=0x7fb6440c7330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:07.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.425+0000 7fb64b200640 1 --2- 172.21.15.96:0/4232354351 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb6440d0bb0 0x7fb6440c7330 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb63c00ef90 tx=0x7fb63c00c490 comp rx=0 tx=0).stop 2024-05-28T06:04:07.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.425+0000 7fb64b200640 1 -- 172.21.15.96:0/4232354351 shutdown_connections 2024-05-28T06:04:07.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.425+0000 7fb64b200640 1 --2- 172.21.15.96:0/4232354351 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb62006c8f0 0x7fb62006edb0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:07.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.425+0000 7fb64b200640 1 --2- 172.21.15.96:0/4232354351 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb6440d0bb0 0x7fb6440c7330 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:07.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.425+0000 7fb64b200640 1 --2- 172.21.15.96:0/4232354351 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb6440b3690 0x7fb6440cec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:07.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.425+0000 7fb64b200640 1 --2- 172.21.15.96:0/4232354351 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb64406f860 0x7fb6440cde30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:07.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.425+0000 7fb64b200640 1 -- 172.21.15.96:0/4232354351 >> 172.21.15.96:0/4232354351 conn(0x7fb64406bae0 msgr2=0x7fb644073550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:07.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.425+0000 7fb64b200640 1 -- 172.21.15.96:0/4232354351 shutdown_connections 2024-05-28T06:04:07.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.425+0000 7fb64b200640 1 -- 172.21.15.96:0/4232354351 wait complete. 2024-05-28T06:04:07.443 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-28T06:03:59.517102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-28T06:04:07.444 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-28T06:04:07.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.636+0000 7f8d94e00640 1 Processor -- start 2024-05-28T06:04:07.638 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.636+0000 7f8d94e00640 1 -- start start 2024-05-28T06:04:07.639 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.637+0000 7f8d94e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8d9015d5d0 0x7f8d900cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:07.639 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.638+0000 7f8d94e00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d9015cb10 con 0x7f8d900b32c0 2024-05-28T06:04:07.639 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.638+0000 7f8d94e00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d900ad5f0 con 0x7f8d900d0640 2024-05-28T06:04:07.639 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.638+0000 7f8d94e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f8d900adbb0 con 0x7f8d9015d5d0 2024-05-28T06:04:07.639 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.638+0000 7f8d8e800640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f8d900b32c0 0x7f8d900b3750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:42152/0 (socket says 172.21.15.96:42152) 2024-05-28T06:04:07.639 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.638+0000 7f8d8e800640 1 -- 172.21.15.96:0/1566152159 learned_addr learned my addr 172.21.15.96:0/1566152159 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:04:07.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.638+0000 7f8d85400640 1 --2- 172.21.15.96:0/1566152159 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8d9015d5d0 0x7f8d900cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:07.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.638+0000 7f8d8d400640 1 -- 172.21.15.96:0/1566152159 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 61520151 0 0) 0x7f8d9015cb10 con 0x7f8d900b32c0 2024-05-28T06:04:07.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.638+0000 7f8d8d400640 1 -- 172.21.15.96:0/1566152159 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8d74003520 con 0x7f8d900b32c0 2024-05-28T06:04:07.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.639+0000 7f8d8d400640 1 -- 172.21.15.96:0/1566152159 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 241968703 0 0) 0x7f8d74003520 con 0x7f8d900b32c0 2024-05-28T06:04:07.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.639+0000 7f8d8d400640 1 -- 172.21.15.96:0/1566152159 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8d9015d5d0 msgr2=0x7f8d900cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:07.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.639+0000 7f8d8d400640 1 --2- 172.21.15.96:0/1566152159 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8d9015d5d0 0x7f8d900cf950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:07.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.639+0000 7f8d8d400640 1 -- 172.21.15.96:0/1566152159 >> v1:172.21.15.123:6789/0 conn(0x7f8d900d0640 legacy=0x7f8d900d0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:07.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.639+0000 7f8d8d400640 1 -- 172.21.15.96:0/1566152159 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d900ad8d0 con 0x7f8d900b32c0 2024-05-28T06:04:07.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.639+0000 7f8d8d400640 1 -- 172.21.15.96:0/1566152159 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f8d880019f0 con 0x7f8d900b32c0 2024-05-28T06:04:07.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.639+0000 7f8d8d400640 1 -- 172.21.15.96:0/1566152159 >> v1:172.21.15.96:6789/0 conn(0x7f8d900b32c0 legacy=0x7f8d900b3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:07.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.639+0000 7f8d8d400640 1 --2- 172.21.15.96:0/1566152159 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8d9015d5d0 0x7f8d740039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:07.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.639+0000 7f8d8d400640 1 --2- 172.21.15.96:0/1566152159 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d74003f60 0x7f8d74004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:07.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.639+0000 7f8d8d400640 1 --2- 172.21.15.96:0/1566152159 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8d74004d40 0x7f8d740051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:07.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.639+0000 7f8d8d400640 1 -- 172.21.15.96:0/1566152159 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f8d7400ba70 con 0x7f8d74003f60 2024-05-28T06:04:07.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.639+0000 7f8d8d400640 1 -- 172.21.15.96:0/1566152159 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f8d7400bc20 con 0x7f8d74004d40 2024-05-28T06:04:07.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.639+0000 7f8d8d400640 1 -- 172.21.15.96:0/1566152159 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f8d7400be20 con 0x7f8d9015d5d0 2024-05-28T06:04:07.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.639+0000 7f8d85400640 1 --2- 172.21.15.96:0/1566152159 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d74003f60 0x7f8d74004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:07.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.639+0000 7f8d8de00640 1 --2- 172.21.15.96:0/1566152159 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8d9015d5d0 0x7f8d740039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:07.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.639+0000 7f8d85400640 1 -- 172.21.15.96:0/1566152159 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8d9015d5d0 msgr2=0x7f8d740039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:07.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.639+0000 7f8d85400640 1 --2- 172.21.15.96:0/1566152159 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8d9015d5d0 0x7f8d740039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:07.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.639+0000 7f8d85400640 1 -- 172.21.15.96:0/1566152159 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8d74004d40 msgr2=0x7f8d740051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:07.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.639+0000 7f8d85400640 1 --2- 172.21.15.96:0/1566152159 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8d74004d40 0x7f8d740051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:07.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.640+0000 7f8d85400640 1 -- 172.21.15.96:0/1566152159 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d74003520 con 0x7f8d74003f60 2024-05-28T06:04:07.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.640+0000 7f8d8de00640 1 --2- 172.21.15.96:0/1566152159 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8d9015d5d0 0x7f8d740039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:04:07.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.640+0000 7f8d85400640 1 --2- 172.21.15.96:0/1566152159 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d74003f60 0x7f8d74004800 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8d78009ac0 tx=0x7f8d74005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5feb2b8f9146b09d server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:07.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.641+0000 7f8d8d400640 1 -- 172.21.15.96:0/1566152159 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d78036070 con 0x7f8d74003f60 2024-05-28T06:04:07.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.641+0000 7f8d8d400640 1 -- 172.21.15.96:0/1566152159 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d78031400 con 0x7f8d74003f60 2024-05-28T06:04:07.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.641+0000 7f8d8d400640 1 -- 172.21.15.96:0/1566152159 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d78009690 con 0x7f8d74003f60 2024-05-28T06:04:07.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.641+0000 7f8d94e00640 1 -- 172.21.15.96:0/1566152159 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d74003f60 msgr2=0x7f8d74004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:07.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.641+0000 7f8d94e00640 1 --2- 172.21.15.96:0/1566152159 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d74003f60 0x7f8d74004800 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8d78009ac0 tx=0x7f8d74005830 comp rx=0 tx=0).stop 2024-05-28T06:04:07.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.641+0000 7f8d8e800640 1 -- 172.21.15.96:0/1566152159 reap_dead start 2024-05-28T06:04:07.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.641+0000 7f8d94e00640 1 -- 172.21.15.96:0/1566152159 shutdown_connections 2024-05-28T06:04:07.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.641+0000 7f8d94e00640 1 -- 172.21.15.96:0/1566152159 >> 172.21.15.96:0/1566152159 conn(0x7f8d9006bd20 msgr2=0x7f8d9006c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:07.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.641+0000 7f8d94e00640 1 -- 172.21.15.96:0/1566152159 shutdown_connections 2024-05-28T06:04:07.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.641+0000 7f8d94e00640 1 -- 172.21.15.96:0/1566152159 wait complete. 2024-05-28T06:04:07.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.643+0000 7f8d94e00640 1 Processor -- start 2024-05-28T06:04:07.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.643+0000 7f8d94e00640 1 -- start start 2024-05-28T06:04:07.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.644+0000 7f8d94e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8d901477f0 0x7f8d9006f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:07.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.644+0000 7f8d94e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8d90147df0 0x7f8d9015b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:07.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.644+0000 7f8d94e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d9015b780 0x7f8d901e5f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:07.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.644+0000 7f8d94e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f8d900b3e60 con 0x7f8d9015b780 2024-05-28T06:04:07.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.644+0000 7f8d94e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f8d9015d0b0 con 0x7f8d90147df0 2024-05-28T06:04:07.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.644+0000 7f8d94e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f8d9015da50 con 0x7f8d901477f0 2024-05-28T06:04:07.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.644+0000 7f8d8e800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d9015b780 0x7f8d901e5f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:07.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.644+0000 7f8d8e800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d9015b780 0x7f8d901e5f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:33886/0 (socket says 172.21.15.96:33886) 2024-05-28T06:04:07.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.644+0000 7f8d8e800640 1 -- 172.21.15.96:0/373683589 learned_addr learned my addr 172.21.15.96:0/373683589 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:07.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.645+0000 7f8d8de00640 1 --2- 172.21.15.96:0/373683589 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8d901477f0 0x7f8d9006f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:07.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.645+0000 7f8d85400640 1 --2- 172.21.15.96:0/373683589 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8d90147df0 0x7f8d9015b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:07.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.645+0000 7f8d8e800640 1 -- 172.21.15.96:0/373683589 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8d901477f0 msgr2=0x7f8d9006f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:07.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.645+0000 7f8d8e800640 1 --2- 172.21.15.96:0/373683589 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8d901477f0 0x7f8d9006f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:07.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.645+0000 7f8d8e800640 1 -- 172.21.15.96:0/373683589 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8d90147df0 msgr2=0x7f8d9015b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:07.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.645+0000 7f8d8e800640 1 --2- 172.21.15.96:0/373683589 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8d90147df0 0x7f8d9015b240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:07.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.645+0000 7f8d8e800640 1 -- 172.21.15.96:0/373683589 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d90147bd0 con 0x7f8d9015b780 2024-05-28T06:04:07.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.645+0000 7f8d8e800640 1 --2- 172.21.15.96:0/373683589 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d9015b780 0x7f8d901e5f40 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8d900d1350 tx=0x7f8d8800ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:07.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.645+0000 7f8d86a00640 1 -- 172.21.15.96:0/373683589 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d88018070 con 0x7f8d9015b780 2024-05-28T06:04:07.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.645+0000 7f8d86a00640 1 -- 172.21.15.96:0/373683589 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d8800a9e0 con 0x7f8d9015b780 2024-05-28T06:04:07.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.645+0000 7f8d86a00640 1 -- 172.21.15.96:0/373683589 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d8800add0 con 0x7f8d9015b780 2024-05-28T06:04:07.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.645+0000 7f8d94e00640 1 -- 172.21.15.96:0/373683589 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d9006da60 con 0x7f8d9015b780 2024-05-28T06:04:07.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.646+0000 7f8d94e00640 1 -- 172.21.15.96:0/373683589 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d9006e020 con 0x7f8d9015b780 2024-05-28T06:04:07.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.647+0000 7f8d94e00640 1 -- 172.21.15.96:0/373683589 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d900b2e70 con 0x7f8d9015b780 2024-05-28T06:04:07.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.647+0000 7f8d86a00640 1 -- 172.21.15.96:0/373683589 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8d880070d0 con 0x7f8d9015b780 2024-05-28T06:04:07.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.650+0000 7f8d86a00640 1 --2- 172.21.15.96:0/373683589 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8d6c06c920 0x7f8d6c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:07.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.650+0000 7f8d86a00640 1 -- 172.21.15.96:0/373683589 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f8d8808af20 con 0x7f8d9015b780 2024-05-28T06:04:07.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.650+0000 7f8d8de00640 1 --2- 172.21.15.96:0/373683589 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8d6c06c920 0x7f8d6c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:07.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.650+0000 7f8d8de00640 1 --2- 172.21.15.96:0/373683589 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8d6c06c920 0x7f8d6c06ede0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8d80001750 tx=0x7f8d80009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:07.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.654+0000 7f8d86a00640 1 -- 172.21.15.96:0/373683589 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8d900b2e70 con 0x7f8d9015b780 2024-05-28T06:04:07.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.840+0000 7f8d94e00640 1 -- 172.21.15.96:0/373683589 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f8d900689e0 con 0x7f8d9015b780 2024-05-28T06:04:07.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.841+0000 7f8d86a00640 1 -- 172.21.15.96:0/373683589 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) ==== 93+0+10 (secure 0 0 0) 0x7f8d88059690 con 0x7f8d9015b780 2024-05-28T06:04:07.842 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 1 2024-05-28T06:04:07.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.843+0000 7f8d94e00640 1 -- 172.21.15.96:0/373683589 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8d6c06c920 msgr2=0x7f8d6c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:07.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.843+0000 7f8d94e00640 1 --2- 172.21.15.96:0/373683589 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8d6c06c920 0x7f8d6c06ede0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8d80001750 tx=0x7f8d80009040 comp rx=0 tx=0).stop 2024-05-28T06:04:07.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.843+0000 7f8d94e00640 1 -- 172.21.15.96:0/373683589 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d9015b780 msgr2=0x7f8d901e5f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:07.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.843+0000 7f8d94e00640 1 --2- 172.21.15.96:0/373683589 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d9015b780 0x7f8d901e5f40 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8d900d1350 tx=0x7f8d8800ccd0 comp rx=0 tx=0).stop 2024-05-28T06:04:07.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.843+0000 7f8d94e00640 1 -- 172.21.15.96:0/373683589 shutdown_connections 2024-05-28T06:04:07.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.843+0000 7f8d94e00640 1 --2- 172.21.15.96:0/373683589 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8d6c06c920 0x7f8d6c06ede0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:07.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.843+0000 7f8d94e00640 1 --2- 172.21.15.96:0/373683589 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d9015b780 0x7f8d901e5f40 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:07.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.843+0000 7f8d94e00640 1 --2- 172.21.15.96:0/373683589 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8d90147df0 0x7f8d9015b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:07.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.843+0000 7f8d94e00640 1 --2- 172.21.15.96:0/373683589 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8d901477f0 0x7f8d9006f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:07.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.843+0000 7f8d94e00640 1 -- 172.21.15.96:0/373683589 >> 172.21.15.96:0/373683589 conn(0x7f8d9006bd20 msgr2=0x7f8d901524b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:07.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.843+0000 7f8d94e00640 1 -- 172.21.15.96:0/373683589 shutdown_connections 2024-05-28T06:04:07.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:07.843+0000 7f8d94e00640 1 -- 172.21.15.96:0/373683589 wait complete. 2024-05-28T06:04:07.863 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-05-28T06:04:08.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.060+0000 7f7665000640 1 Processor -- start 2024-05-28T06:04:08.061 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.060+0000 7f7665000640 1 -- start start 2024-05-28T06:04:08.062 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.062+0000 7f7665000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f766006ee40 0x7f76600ad230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:08.062 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.062+0000 7f7665000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f766006f2c0 0x7f76600ad770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:08.062 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.062+0000 7f7665000640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7660156f10 con 0x7f7660156b30 2024-05-28T06:04:08.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.062+0000 7f7665000640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f766006dda0 con 0x7f766006f2c0 2024-05-28T06:04:08.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.062+0000 7f7665000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f766006e080 con 0x7f766006ee40 2024-05-28T06:04:08.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.062+0000 7f765fe00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f766006f2c0 0x7f76600ad770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:08.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.062+0000 7f765f400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f7660156b30 0x7f76600af6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:42160/0 (socket says 172.21.15.96:42160) 2024-05-28T06:04:08.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.062+0000 7f765f400640 1 -- 172.21.15.96:0/1195948973 learned_addr learned my addr 172.21.15.96:0/1195948973 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:04:08.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.062+0000 7f765ea00640 1 --2- 172.21.15.96:0/1195948973 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f766006ee40 0x7f76600ad230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:08.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.062+0000 7f765fe00640 1 -- 172.21.15.96:0/1195948973 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f766006ee40 msgr2=0x7f76600ad230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:08.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.062+0000 7f765fe00640 1 --2- 172.21.15.96:0/1195948973 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f766006ee40 0x7f76600ad230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:08.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.062+0000 7f765fe00640 1 -- 172.21.15.96:0/1195948973 >> v1:172.21.15.96:6789/0 conn(0x7f7660156b30 legacy=0x7f76600af6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:08.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.062+0000 7f765fe00640 1 -- 172.21.15.96:0/1195948973 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f766006e360 con 0x7f766006f2c0 2024-05-28T06:04:08.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.062+0000 7f765ea00640 1 --2- 172.21.15.96:0/1195948973 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f766006ee40 0x7f76600ad230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:04:08.065 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.063+0000 7f765fe00640 1 --2- 172.21.15.96:0/1195948973 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f766006f2c0 0x7f76600ad770 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7658009d90 tx=0x7f765802a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34f53b05cc17db5b server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:08.065 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.063+0000 7f765e000640 1 -- 172.21.15.96:0/1195948973 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7658031600 con 0x7f766006f2c0 2024-05-28T06:04:08.065 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.063+0000 7f765e000640 1 -- 172.21.15.96:0/1195948973 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76580317a0 con 0x7f766006f2c0 2024-05-28T06:04:08.065 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.063+0000 7f765e000640 1 -- 172.21.15.96:0/1195948973 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7658031b70 con 0x7f766006f2c0 2024-05-28T06:04:08.065 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.063+0000 7f7665000640 1 -- 172.21.15.96:0/1195948973 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f766006f2c0 msgr2=0x7f76600ad770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:08.065 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.063+0000 7f7665000640 1 --2- 172.21.15.96:0/1195948973 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f766006f2c0 0x7f76600ad770 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7658009d90 tx=0x7f765802a5d0 comp rx=0 tx=0).stop 2024-05-28T06:04:08.065 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.064+0000 7f7665000640 1 -- 172.21.15.96:0/1195948973 shutdown_connections 2024-05-28T06:04:08.065 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.064+0000 7f7665000640 1 --2- 172.21.15.96:0/1195948973 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f766006f2c0 0x7f76600ad770 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:08.065 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.064+0000 7f7665000640 1 --2- 172.21.15.96:0/1195948973 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f766006ee40 0x7f76600ad230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:08.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.064+0000 7f7665000640 1 -- 172.21.15.96:0/1195948973 >> 172.21.15.96:0/1195948973 conn(0x7f76600b1fc0 msgr2=0x7f76600b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:08.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.064+0000 7f7665000640 1 -- 172.21.15.96:0/1195948973 shutdown_connections 2024-05-28T06:04:08.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.064+0000 7f7665000640 1 -- 172.21.15.96:0/1195948973 wait complete. 2024-05-28T06:04:08.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.065+0000 7f7665000640 1 Processor -- start 2024-05-28T06:04:08.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.066+0000 7f7665000640 1 -- start start 2024-05-28T06:04:08.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.066+0000 7f7665000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f766006ee40 0x7f76600730a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:08.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.066+0000 7f7665000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f766006f2c0 0x7f7660072960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:08.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.067+0000 7f7665000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7660156b30 0x7f76601f52c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:08.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.067+0000 7f7665000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f76600b2db0 con 0x7f7660156b30 2024-05-28T06:04:08.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.067+0000 7f7665000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f76600b2c30 con 0x7f766006ee40 2024-05-28T06:04:08.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.067+0000 7f7665000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f76600b32c0 con 0x7f766006f2c0 2024-05-28T06:04:08.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.067+0000 7f765fe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7660156b30 0x7f76601f52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:08.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.067+0000 7f765fe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7660156b30 0x7f76601f52c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:33900/0 (socket says 172.21.15.96:33900) 2024-05-28T06:04:08.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.067+0000 7f765fe00640 1 -- 172.21.15.96:0/50449302 learned_addr learned my addr 172.21.15.96:0/50449302 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:08.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.067+0000 7f765f400640 1 --2- 172.21.15.96:0/50449302 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f766006ee40 0x7f76600730a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:08.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.067+0000 7f765ea00640 1 --2- 172.21.15.96:0/50449302 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f766006f2c0 0x7f7660072960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:08.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.067+0000 7f765fe00640 1 -- 172.21.15.96:0/50449302 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f766006f2c0 msgr2=0x7f7660072960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:08.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.067+0000 7f765fe00640 1 --2- 172.21.15.96:0/50449302 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f766006f2c0 0x7f7660072960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:08.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.067+0000 7f765fe00640 1 -- 172.21.15.96:0/50449302 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f766006ee40 msgr2=0x7f76600730a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:08.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.067+0000 7f765fe00640 1 --2- 172.21.15.96:0/50449302 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f766006ee40 0x7f76600730a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:08.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.067+0000 7f765fe00640 1 -- 172.21.15.96:0/50449302 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7660072ea0 con 0x7f7660156b30 2024-05-28T06:04:08.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.067+0000 7f765f400640 1 --2- 172.21.15.96:0/50449302 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f766006ee40 0x7f76600730a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:04:08.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.068+0000 7f765fe00640 1 --2- 172.21.15.96:0/50449302 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7660156b30 0x7f76601f52c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7658009eb0 tx=0x7f7658004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:08.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.068+0000 7f7657400640 1 -- 172.21.15.96:0/50449302 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7658040070 con 0x7f7660156b30 2024-05-28T06:04:08.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.068+0000 7f7657400640 1 -- 172.21.15.96:0/50449302 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7658030b60 con 0x7f7660156b30 2024-05-28T06:04:08.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.068+0000 7f7657400640 1 -- 172.21.15.96:0/50449302 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7658045460 con 0x7f7660156b30 2024-05-28T06:04:08.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.068+0000 7f7665000640 1 -- 172.21.15.96:0/50449302 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76600736d0 con 0x7f7660156b30 2024-05-28T06:04:08.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.068+0000 7f7665000640 1 -- 172.21.15.96:0/50449302 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7660157280 con 0x7f7660156b30 2024-05-28T06:04:08.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.069+0000 7f7665000640 1 -- 172.21.15.96:0/50449302 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7660073bf0 con 0x7f7660156b30 2024-05-28T06:04:08.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.070+0000 7f7657400640 1 -- 172.21.15.96:0/50449302 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7658039040 con 0x7f7660156b30 2024-05-28T06:04:08.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.071+0000 7f7657400640 1 --2- 172.21.15.96:0/50449302 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f764006c920 0x7f764006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:08.075 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.071+0000 7f7657400640 1 -- 172.21.15.96:0/50449302 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f76580acfa0 con 0x7f7660156b30 2024-05-28T06:04:08.075 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.072+0000 7f765f400640 1 --2- 172.21.15.96:0/50449302 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f764006c920 0x7f764006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:08.075 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.073+0000 7f765f400640 1 --2- 172.21.15.96:0/50449302 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f764006c920 0x7f764006ede0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7650000f60 tx=0x7f7650009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:08.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.076+0000 7f7657400640 1 -- 172.21.15.96:0/50449302 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7660073bf0 con 0x7f7660156b30 2024-05-28T06:04:08.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.278+0000 7f7665000640 1 -- 172.21.15.96:0/50449302 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f76600689e0 con 0x7f7660156b30 2024-05-28T06:04:08.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.279+0000 7f7657400640 1 -- 172.21.15.96:0/50449302 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v16) ==== 84+0+65 (secure 0 0 0) 0x7f7660072ea0 con 0x7f7660156b30 2024-05-28T06:04:08.279 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:04:08.279 INFO:teuthology.orchestra.run.smithi096.stdout:[ 2024-05-28T06:04:08.279 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:04:08.279 INFO:teuthology.orchestra.run.smithi096.stdout: "poolnum": 1, 2024-05-28T06:04:08.279 INFO:teuthology.orchestra.run.smithi096.stdout: "poolname": ".mgr" 2024-05-28T06:04:08.280 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:04:08.280 INFO:teuthology.orchestra.run.smithi096.stdout:] 2024-05-28T06:04:08.282 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.281+0000 7f7665000640 1 -- 172.21.15.96:0/50449302 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f764006c920 msgr2=0x7f764006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:08.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.281+0000 7f7665000640 1 --2- 172.21.15.96:0/50449302 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f764006c920 0x7f764006ede0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7650000f60 tx=0x7f7650009040 comp rx=0 tx=0).stop 2024-05-28T06:04:08.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.282+0000 7f7665000640 1 -- 172.21.15.96:0/50449302 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7660156b30 msgr2=0x7f76601f52c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:08.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.282+0000 7f7665000640 1 --2- 172.21.15.96:0/50449302 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7660156b30 0x7f76601f52c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7658009eb0 tx=0x7f7658004a40 comp rx=0 tx=0).stop 2024-05-28T06:04:08.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.282+0000 7f7665000640 1 -- 172.21.15.96:0/50449302 shutdown_connections 2024-05-28T06:04:08.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.282+0000 7f7665000640 1 --2- 172.21.15.96:0/50449302 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f764006c920 0x7f764006ede0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:08.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.282+0000 7f7665000640 1 --2- 172.21.15.96:0/50449302 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7660156b30 0x7f76601f52c0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:08.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.282+0000 7f7665000640 1 --2- 172.21.15.96:0/50449302 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f766006f2c0 0x7f7660072960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:08.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.282+0000 7f7665000640 1 --2- 172.21.15.96:0/50449302 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f766006ee40 0x7f76600730a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:08.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.282+0000 7f7665000640 1 -- 172.21.15.96:0/50449302 >> 172.21.15.96:0/50449302 conn(0x7f76600b1fc0 msgr2=0x7f76600b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:08.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.282+0000 7f7665000640 1 -- 172.21.15.96:0/50449302 shutdown_connections 2024-05-28T06:04:08.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.282+0000 7f7665000640 1 -- 172.21.15.96:0/50449302 wait complete. 2024-05-28T06:04:08.302 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-05-28T06:04:08.303 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-05-28T06:04:08.513 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.513+0000 7efc51600640 1 Processor -- start 2024-05-28T06:04:08.513 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.513+0000 7efc51600640 1 -- start start 2024-05-28T06:04:08.514 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.514+0000 7efc51600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7efc4c06ef00 0x7efc4c0d0320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:08.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.514+0000 7efc51600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7efc4c0b4790 0x7efc4c0d0860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:08.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.514+0000 7efc51600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc4c15c7d0 con 0x7efc4c15c3f0 2024-05-28T06:04:08.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.514+0000 7efc51600640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7efc4c0ad9a0 con 0x7efc4c0b4790 2024-05-28T06:04:08.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.514+0000 7efc51600640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7efc4c0ad3e0 con 0x7efc4c06ef00 2024-05-28T06:04:08.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.514+0000 7efc50c00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7efc4c0b4790 0x7efc4c0d0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:08.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.514+0000 7efc50c00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7efc4c0b4790 0x7efc4c0d0860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:33908/0 (socket says 172.21.15.96:33908) 2024-05-28T06:04:08.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.514+0000 7efc50c00640 1 -- 172.21.15.96:0/3990427382 learned_addr learned my addr 172.21.15.96:0/3990427382 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:08.516 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.515+0000 7efc43400640 1 --2- 172.21.15.96:0/3990427382 >> v2:172.21.15.123:3300/0 conn(0x7efc4c06ef00 0x7efc4c0d0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:08.516 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.515+0000 7efc50c00640 1 -- 172.21.15.96:0/3990427382 >> v2:172.21.15.123:3300/0 conn(0x7efc4c06ef00 msgr2=0x7efc4c0d0320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:08.516 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.515+0000 7efc50c00640 1 --2- 172.21.15.96:0/3990427382 >> v2:172.21.15.123:3300/0 conn(0x7efc4c06ef00 0x7efc4c0d0320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:08.516 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.515+0000 7efc50c00640 1 -- 172.21.15.96:0/3990427382 >> v1:172.21.15.123:6789/0 conn(0x7efc4c15c3f0 legacy=0x7efc4c1608d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:08.516 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.515+0000 7efc50c00640 1 -- 172.21.15.96:0/3990427382 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc4c0ad6c0 con 0x7efc4c0b4790 2024-05-28T06:04:08.516 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.515+0000 7efc43400640 1 --2- 172.21.15.96:0/3990427382 >> v2:172.21.15.123:3300/0 conn(0x7efc4c06ef00 0x7efc4c0d0320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:04:08.516 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.516+0000 7efc50c00640 1 --2- 172.21.15.96:0/3990427382 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efc4c0b4790 0x7efc4c0d0860 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7efc4c0d0e70 tx=0x7efc4402a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c997da2b7a8cb259 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:08.517 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.516+0000 7efc4b400640 1 -- 172.21.15.96:0/3990427382 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc44036070 con 0x7efc4c0b4790 2024-05-28T06:04:08.517 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.516+0000 7efc4b400640 1 -- 172.21.15.96:0/3990427382 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc44031600 con 0x7efc4c0b4790 2024-05-28T06:04:08.518 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.516+0000 7efc4b400640 1 -- 172.21.15.96:0/3990427382 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc440319d0 con 0x7efc4c0b4790 2024-05-28T06:04:08.518 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.517+0000 7efc51600640 1 -- 172.21.15.96:0/3990427382 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efc4c0b4790 msgr2=0x7efc4c0d0860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:08.518 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.517+0000 7efc51600640 1 --2- 172.21.15.96:0/3990427382 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efc4c0b4790 0x7efc4c0d0860 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7efc4c0d0e70 tx=0x7efc4402a6f0 comp rx=0 tx=0).stop 2024-05-28T06:04:08.518 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.517+0000 7efc51600640 1 -- 172.21.15.96:0/3990427382 shutdown_connections 2024-05-28T06:04:08.518 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.517+0000 7efc51600640 1 --2- 172.21.15.96:0/3990427382 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efc4c0b4790 0x7efc4c0d0860 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:08.518 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.517+0000 7efc51600640 1 --2- 172.21.15.96:0/3990427382 >> v2:172.21.15.123:3300/0 conn(0x7efc4c06ef00 0x7efc4c0d0320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:08.518 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.517+0000 7efc51600640 1 -- 172.21.15.96:0/3990427382 >> 172.21.15.96:0/3990427382 conn(0x7efc4c06bae0 msgr2=0x7efc4c06bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:08.518 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.517+0000 7efc51600640 1 -- 172.21.15.96:0/3990427382 shutdown_connections 2024-05-28T06:04:08.518 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.517+0000 7efc51600640 1 -- 172.21.15.96:0/3990427382 wait complete. 2024-05-28T06:04:08.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.519+0000 7efc51600640 1 Processor -- start 2024-05-28T06:04:08.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.519+0000 7efc51600640 1 -- start start 2024-05-28T06:04:08.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.520+0000 7efc51600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efc4c06ef00 0x7efc4c0999f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:08.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.520+0000 7efc51600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc4c0b4790 0x7efc4c14e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:08.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.520+0000 7efc51600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efc4c15c3f0 0x7efc4c14e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:08.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.520+0000 7efc51600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7efc4c0ad090 con 0x7efc4c06ef00 2024-05-28T06:04:08.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.520+0000 7efc51600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7efc4c1603b0 con 0x7efc4c0b4790 2024-05-28T06:04:08.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.520+0000 7efc51600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7efc4c0b30b0 con 0x7efc4c15c3f0 2024-05-28T06:04:08.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.520+0000 7efc4be00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efc4c06ef00 0x7efc4c0999f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:08.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.520+0000 7efc4be00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efc4c06ef00 0x7efc4c0999f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:33920/0 (socket says 172.21.15.96:33920) 2024-05-28T06:04:08.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.520+0000 7efc4be00640 1 -- 172.21.15.96:0/300383538 learned_addr learned my addr 172.21.15.96:0/300383538 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:08.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.520+0000 7efc4be00640 1 -- 172.21.15.96:0/300383538 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efc4c15c3f0 msgr2=0x7efc4c14e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:08.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.521+0000 7efc4be00640 1 --2- 172.21.15.96:0/300383538 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efc4c15c3f0 0x7efc4c14e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:08.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.521+0000 7efc4be00640 1 -- 172.21.15.96:0/300383538 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc4c0b4790 msgr2=0x7efc4c14e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:08.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.521+0000 7efc4be00640 1 --2- 172.21.15.96:0/300383538 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc4c0b4790 0x7efc4c14e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:08.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.521+0000 7efc4be00640 1 -- 172.21.15.96:0/300383538 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc4c14ed10 con 0x7efc4c06ef00 2024-05-28T06:04:08.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.521+0000 7efc4be00640 1 --2- 172.21.15.96:0/300383538 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efc4c06ef00 0x7efc4c0999f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7efc3c00b3f0 tx=0x7efc3c00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:08.523 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.521+0000 7efc48c00640 1 -- 172.21.15.96:0/300383538 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc3c015020 con 0x7efc4c06ef00 2024-05-28T06:04:08.523 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.521+0000 7efc48c00640 1 -- 172.21.15.96:0/300383538 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc3c011830 con 0x7efc4c06ef00 2024-05-28T06:04:08.523 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.521+0000 7efc51600640 1 -- 172.21.15.96:0/300383538 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc4c14eff0 con 0x7efc4c06ef00 2024-05-28T06:04:08.523 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.521+0000 7efc48c00640 1 -- 172.21.15.96:0/300383538 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc3c011c20 con 0x7efc4c06ef00 2024-05-28T06:04:08.523 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.521+0000 7efc51600640 1 -- 172.21.15.96:0/300383538 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc4c14f5b0 con 0x7efc4c06ef00 2024-05-28T06:04:08.523 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.522+0000 7efc51600640 1 -- 172.21.15.96:0/300383538 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc4c0689e0 con 0x7efc4c06ef00 2024-05-28T06:04:08.524 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.523+0000 7efc48c00640 1 -- 172.21.15.96:0/300383538 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7efc3c018020 con 0x7efc4c06ef00 2024-05-28T06:04:08.524 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.524+0000 7efc48c00640 1 --2- 172.21.15.96:0/300383538 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7efc2806c8f0 0x7efc2806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:08.524 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.524+0000 7efc48c00640 1 -- 172.21.15.96:0/300383538 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7efc3c08b540 con 0x7efc4c06ef00 2024-05-28T06:04:08.525 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.525+0000 7efc43400640 1 --2- 172.21.15.96:0/300383538 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7efc2806c8f0 0x7efc2806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:08.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.526+0000 7efc43400640 1 --2- 172.21.15.96:0/300383538 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7efc2806c8f0 0x7efc2806edb0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7efc34006fd0 tx=0x7efc34008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:08.528 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.528+0000 7efc48c00640 1 -- 172.21.15.96:0/300383538 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7efc3c01f030 con 0x7efc4c06ef00 2024-05-28T06:04:08.729 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.729+0000 7efc51600640 1 -- 172.21.15.96:0/300383538 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7efc4c0d0b60 con 0x7efc4c06ef00 2024-05-28T06:04:08.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.837+0000 7efc48c00640 1 -- 172.21.15.96:0/300383538 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v17) ==== 138+0+0 (secure 0 0 0) 0x7efc3c0172c0 con 0x7efc4c06ef00 2024-05-28T06:04:08.838 INFO:teuthology.orchestra.run.smithi096.stderr:pool 'cephfs_metadata' created 2024-05-28T06:04:08.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.841+0000 7efc51600640 1 -- 172.21.15.96:0/300383538 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7efc2806c8f0 msgr2=0x7efc2806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:08.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.841+0000 7efc51600640 1 --2- 172.21.15.96:0/300383538 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7efc2806c8f0 0x7efc2806edb0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7efc34006fd0 tx=0x7efc34008040 comp rx=0 tx=0).stop 2024-05-28T06:04:08.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.841+0000 7efc51600640 1 -- 172.21.15.96:0/300383538 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efc4c06ef00 msgr2=0x7efc4c0999f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:08.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.841+0000 7efc51600640 1 --2- 172.21.15.96:0/300383538 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efc4c06ef00 0x7efc4c0999f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7efc3c00b3f0 tx=0x7efc3c00b8b0 comp rx=0 tx=0).stop 2024-05-28T06:04:08.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.841+0000 7efc51600640 1 -- 172.21.15.96:0/300383538 shutdown_connections 2024-05-28T06:04:08.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.841+0000 7efc51600640 1 --2- 172.21.15.96:0/300383538 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7efc2806c8f0 0x7efc2806edb0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:08.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.841+0000 7efc51600640 1 --2- 172.21.15.96:0/300383538 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efc4c15c3f0 0x7efc4c14e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:08.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.841+0000 7efc51600640 1 --2- 172.21.15.96:0/300383538 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc4c0b4790 0x7efc4c14e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:08.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.841+0000 7efc51600640 1 --2- 172.21.15.96:0/300383538 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efc4c06ef00 0x7efc4c0999f0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:08.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.841+0000 7efc51600640 1 -- 172.21.15.96:0/300383538 >> 172.21.15.96:0/300383538 conn(0x7efc4c06bae0 msgr2=0x7efc4c0d02f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:08.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.841+0000 7efc51600640 1 -- 172.21.15.96:0/300383538 shutdown_connections 2024-05-28T06:04:08.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:08.842+0000 7efc51600640 1 -- 172.21.15.96:0/300383538 wait complete. 2024-05-28T06:04:08.870 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-05-28T06:04:09.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.068+0000 7f0451e00640 1 Processor -- start 2024-05-28T06:04:09.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.068+0000 7f0451e00640 1 -- start start 2024-05-28T06:04:09.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.070+0000 7f0451e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f044c0d0640 0x7f044c15d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:09.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.070+0000 7f0451e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f044c0d0a20 0x7f044c0cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:09.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.070+0000 7f0451e00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f044c15cb10 con 0x7f044c0b32c0 2024-05-28T06:04:09.071 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.070+0000 7f0451e00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f044c0adbb0 con 0x7f044c0d0640 2024-05-28T06:04:09.071 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.070+0000 7f0451e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f044c15db10 con 0x7f044c0d0a20 2024-05-28T06:04:09.071 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.070+0000 7f044b400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f044c0b32c0 0x7f044c0b3750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:55366/0 (socket says 172.21.15.96:55366) 2024-05-28T06:04:09.071 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.070+0000 7f044b400640 1 -- 172.21.15.96:0/1989238964 learned_addr learned my addr 172.21.15.96:0/1989238964 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:04:09.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.070+0000 7f044a000640 1 --2- 172.21.15.96:0/1989238964 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f044c0d0a20 0x7f044c0cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:09.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.070+0000 7f044aa00640 1 --2- 172.21.15.96:0/1989238964 >> v2:172.21.15.123:3300/0 conn(0x7f044c0d0640 0x7f044c15d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:09.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.071+0000 7f0449600640 1 -- 172.21.15.96:0/1989238964 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3844177719 0 0) 0x7f044c15cb10 con 0x7f044c0b32c0 2024-05-28T06:04:09.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.071+0000 7f0449600640 1 -- 172.21.15.96:0/1989238964 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0430003520 con 0x7f044c0b32c0 2024-05-28T06:04:09.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.071+0000 7f0449600640 1 -- 172.21.15.96:0/1989238964 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1284008610 0 0) 0x7f0430003520 con 0x7f044c0b32c0 2024-05-28T06:04:09.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.071+0000 7f0449600640 1 -- 172.21.15.96:0/1989238964 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f044c0d0a20 msgr2=0x7f044c0cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:09.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.071+0000 7f0449600640 1 --2- 172.21.15.96:0/1989238964 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f044c0d0a20 0x7f044c0cf950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:09.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.071+0000 7f0449600640 1 -- 172.21.15.96:0/1989238964 >> v2:172.21.15.123:3300/0 conn(0x7f044c0d0640 msgr2=0x7f044c15d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:09.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.071+0000 7f0449600640 1 --2- 172.21.15.96:0/1989238964 >> v2:172.21.15.123:3300/0 conn(0x7f044c0d0640 0x7f044c15d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:09.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.071+0000 7f0449600640 1 -- 172.21.15.96:0/1989238964 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f044c0ad5f0 con 0x7f044c0b32c0 2024-05-28T06:04:09.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.071+0000 7f0449600640 1 -- 172.21.15.96:0/1989238964 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f04440019f0 con 0x7f044c0b32c0 2024-05-28T06:04:09.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.071+0000 7f0449600640 1 -- 172.21.15.96:0/1989238964 >> v1:172.21.15.96:6789/0 conn(0x7f044c0b32c0 legacy=0x7f044c0b3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:09.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.071+0000 7f0449600640 1 --2- 172.21.15.96:0/1989238964 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f044c0d0a20 0x7f04300039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:09.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.071+0000 7f0449600640 1 --2- 172.21.15.96:0/1989238964 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0430003f80 0x7f0430004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:09.075 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.071+0000 7f0449600640 1 --2- 172.21.15.96:0/1989238964 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0430004d40 0x7f04300051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:09.075 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.071+0000 7f0449600640 1 -- 172.21.15.96:0/1989238964 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f044c15db10 con 0x7f0430004d40 2024-05-28T06:04:09.075 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.071+0000 7f0449600640 1 -- 172.21.15.96:0/1989238964 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f043000b9c0 con 0x7f0430003f80 2024-05-28T06:04:09.075 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.071+0000 7f0449600640 1 -- 172.21.15.96:0/1989238964 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f043000bb40 con 0x7f044c0d0a20 2024-05-28T06:04:09.075 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.072+0000 7f044b400640 1 --2- 172.21.15.96:0/1989238964 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0430004d40 0x7f04300051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:09.075 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.072+0000 7f044aa00640 1 --2- 172.21.15.96:0/1989238964 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f044c0d0a20 0x7f04300039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:09.075 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.072+0000 7f044a000640 1 --2- 172.21.15.96:0/1989238964 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0430003f80 0x7f0430004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:09.075 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.072+0000 7f044b400640 1 -- 172.21.15.96:0/1989238964 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f044c0d0a20 msgr2=0x7f04300039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:09.075 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.072+0000 7f044b400640 1 --2- 172.21.15.96:0/1989238964 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f044c0d0a20 0x7f04300039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:09.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.072+0000 7f044b400640 1 -- 172.21.15.96:0/1989238964 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0430003f80 msgr2=0x7f0430004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:09.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.072+0000 7f044b400640 1 --2- 172.21.15.96:0/1989238964 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0430003f80 0x7f0430004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:09.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.072+0000 7f044b400640 1 -- 172.21.15.96:0/1989238964 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0430003520 con 0x7f0430004d40 2024-05-28T06:04:09.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.072+0000 7f044aa00640 1 --2- 172.21.15.96:0/1989238964 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f044c0d0a20 0x7f04300039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:04:09.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.073+0000 7f044b400640 1 --2- 172.21.15.96:0/1989238964 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0430004d40 0x7f04300051c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f044400a7d0 tx=0x7f0430005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=898b71451d476593 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:09.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.073+0000 7f0449600640 1 -- 172.21.15.96:0/1989238964 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f044400a3f0 con 0x7f0430004d40 2024-05-28T06:04:09.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.073+0000 7f0449600640 1 -- 172.21.15.96:0/1989238964 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f044402f070 con 0x7f0430004d40 2024-05-28T06:04:09.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.073+0000 7f0449600640 1 -- 172.21.15.96:0/1989238964 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f044402bc10 con 0x7f0430004d40 2024-05-28T06:04:09.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.074+0000 7f0451e00640 1 -- 172.21.15.96:0/1989238964 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0430004d40 msgr2=0x7f04300051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:09.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.074+0000 7f0451e00640 1 --2- 172.21.15.96:0/1989238964 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0430004d40 0x7f04300051c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f044400a7d0 tx=0x7f0430005830 comp rx=0 tx=0).stop 2024-05-28T06:04:09.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.074+0000 7f044b400640 1 -- 172.21.15.96:0/1989238964 reap_dead start 2024-05-28T06:04:09.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.074+0000 7f0451e00640 1 -- 172.21.15.96:0/1989238964 shutdown_connections 2024-05-28T06:04:09.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.074+0000 7f0451e00640 1 -- 172.21.15.96:0/1989238964 >> 172.21.15.96:0/1989238964 conn(0x7f044c06bd20 msgr2=0x7f044c06c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:09.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.074+0000 7f0451e00640 1 -- 172.21.15.96:0/1989238964 shutdown_connections 2024-05-28T06:04:09.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.074+0000 7f0451e00640 1 -- 172.21.15.96:0/1989238964 wait complete. 2024-05-28T06:04:09.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.076+0000 7f0451e00640 1 Processor -- start 2024-05-28T06:04:09.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.076+0000 7f0451e00640 1 -- start start 2024-05-28T06:04:09.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.077+0000 7f0451e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0430004d40 0x7f044c148c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:09.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.077+0000 7f0451e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f044c06e380 0x7f044c155f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:09.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.077+0000 7f0451e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f044c156490 0x7f044c15b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:09.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.077+0000 7f0451e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f044c15d0b0 con 0x7f0430004d40 2024-05-28T06:04:09.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.077+0000 7f0451e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f044c15cdd0 con 0x7f044c06e380 2024-05-28T06:04:09.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.077+0000 7f0451e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f044c0b3e60 con 0x7f044c156490 2024-05-28T06:04:09.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.077+0000 7f044aa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0430004d40 0x7f044c148c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:09.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.077+0000 7f044aa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0430004d40 0x7f044c148c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:41370/0 (socket says 172.21.15.96:41370) 2024-05-28T06:04:09.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.077+0000 7f044aa00640 1 -- 172.21.15.96:0/9611190 learned_addr learned my addr 172.21.15.96:0/9611190 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:09.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.078+0000 7f044aa00640 1 -- 172.21.15.96:0/9611190 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f044c156490 msgr2=0x7f044c15b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:09.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.078+0000 7f044aa00640 1 --2- 172.21.15.96:0/9611190 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f044c156490 0x7f044c15b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:09.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.078+0000 7f044a000640 1 --2- 172.21.15.96:0/9611190 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f044c06e380 0x7f044c155f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:09.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.078+0000 7f044aa00640 1 -- 172.21.15.96:0/9611190 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f044c06e380 msgr2=0x7f044c155f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:09.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.078+0000 7f044aa00640 1 --2- 172.21.15.96:0/9611190 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f044c06e380 0x7f044c155f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:09.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.078+0000 7f044aa00640 1 -- 172.21.15.96:0/9611190 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f044c06f970 con 0x7f0430004d40 2024-05-28T06:04:09.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.078+0000 7f044aa00640 1 --2- 172.21.15.96:0/9611190 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0430004d40 0x7f044c148c30 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f043c010980 tx=0x7f043c010e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:09.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.078+0000 7f0443400640 1 -- 172.21.15.96:0/9611190 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f043c00f040 con 0x7f0430004d40 2024-05-28T06:04:09.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.078+0000 7f0443400640 1 -- 172.21.15.96:0/9611190 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f043c00ec20 con 0x7f0430004d40 2024-05-28T06:04:09.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.078+0000 7f0443400640 1 -- 172.21.15.96:0/9611190 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f043c01c400 con 0x7f0430004d40 2024-05-28T06:04:09.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.078+0000 7f0451e00640 1 -- 172.21.15.96:0/9611190 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f044c06e8d0 con 0x7f0430004d40 2024-05-28T06:04:09.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.078+0000 7f0451e00640 1 -- 172.21.15.96:0/9611190 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f044c06ee90 con 0x7f0430004d40 2024-05-28T06:04:09.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.080+0000 7f0451e00640 1 -- 172.21.15.96:0/9611190 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f044c0689e0 con 0x7f0430004d40 2024-05-28T06:04:09.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.081+0000 7f0443400640 1 -- 172.21.15.96:0/9611190 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f043c01c5a0 con 0x7f0430004d40 2024-05-28T06:04:09.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.082+0000 7f0443400640 1 --2- 172.21.15.96:0/9611190 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f042c06c8f0 0x7f042c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:09.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.082+0000 7f0443400640 1 -- 172.21.15.96:0/9611190 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 5227+0+0 (secure 0 0 0) 0x7f043c08c0d0 con 0x7f0430004d40 2024-05-28T06:04:09.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.082+0000 7f044a000640 1 --2- 172.21.15.96:0/9611190 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f042c06c8f0 0x7f042c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:09.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.083+0000 7f044a000640 1 --2- 172.21.15.96:0/9611190 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f042c06c8f0 0x7f042c06edb0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f044c1f7b60 tx=0x7f0438008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:09.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.087+0000 7f0443400640 1 -- 172.21.15.96:0/9611190 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f043c018030 con 0x7f0430004d40 2024-05-28T06:04:09.286 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.286+0000 7f0451e00640 1 -- 172.21.15.96:0/9611190 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f044c071ef0 con 0x7f0430004d40 2024-05-28T06:04:09.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.843+0000 7f0443400640 1 -- 172.21.15.96:0/9611190 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v18) ==== 170+0+0 (secure 0 0 0) 0x7f043c05a640 con 0x7f0430004d40 2024-05-28T06:04:09.844 INFO:teuthology.orchestra.run.smithi096.stderr:pool 'cephfs_data' created 2024-05-28T06:04:09.847 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.847+0000 7f0451e00640 1 -- 172.21.15.96:0/9611190 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f042c06c8f0 msgr2=0x7f042c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:09.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.847+0000 7f0451e00640 1 --2- 172.21.15.96:0/9611190 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f042c06c8f0 0x7f042c06edb0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f044c1f7b60 tx=0x7f0438008040 comp rx=0 tx=0).stop 2024-05-28T06:04:09.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.847+0000 7f0451e00640 1 -- 172.21.15.96:0/9611190 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0430004d40 msgr2=0x7f044c148c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:09.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.847+0000 7f0451e00640 1 --2- 172.21.15.96:0/9611190 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0430004d40 0x7f044c148c30 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f043c010980 tx=0x7f043c010e40 comp rx=0 tx=0).stop 2024-05-28T06:04:09.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.847+0000 7f0451e00640 1 -- 172.21.15.96:0/9611190 shutdown_connections 2024-05-28T06:04:09.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.847+0000 7f0451e00640 1 --2- 172.21.15.96:0/9611190 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f042c06c8f0 0x7f042c06edb0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:09.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.847+0000 7f0451e00640 1 --2- 172.21.15.96:0/9611190 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f044c156490 0x7f044c15b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:09.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.847+0000 7f0451e00640 1 --2- 172.21.15.96:0/9611190 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f044c06e380 0x7f044c155f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:09.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.847+0000 7f0451e00640 1 --2- 172.21.15.96:0/9611190 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0430004d40 0x7f044c148c30 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:09.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.847+0000 7f0451e00640 1 -- 172.21.15.96:0/9611190 >> 172.21.15.96:0/9611190 conn(0x7f044c06bd20 msgr2=0x7f044c15edd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:09.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.847+0000 7f0451e00640 1 -- 172.21.15.96:0/9611190 shutdown_connections 2024-05-28T06:04:09.849 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:09.848+0000 7f0451e00640 1 -- 172.21.15.96:0/9611190 wait complete. 2024-05-28T06:04:09.888 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-05-28T06:04:10.062 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.061+0000 7fce23c00640 1 Processor -- start 2024-05-28T06:04:10.062 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.062+0000 7fce23c00640 1 -- start start 2024-05-28T06:04:10.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.063+0000 7fce23c00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fce1c0d17b0 0x7fce1c0cf2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:10.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.063+0000 7fce23c00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce1c0b13d0 con 0x7fce1c162130 2024-05-28T06:04:10.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.063+0000 7fce23c00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce1c0d33b0 con 0x7fce1c0b2e50 2024-05-28T06:04:10.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.063+0000 7fce23c00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fce1c0b1640 con 0x7fce1c0d17b0 2024-05-28T06:04:10.065 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.063+0000 7fce21200640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fce1c162130 0x7fce1c0d10a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:55370/0 (socket says 172.21.15.96:55370) 2024-05-28T06:04:10.065 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.063+0000 7fce21200640 1 -- 172.21.15.96:0/1827389094 learned_addr learned my addr 172.21.15.96:0/1827389094 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:04:10.065 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.064+0000 7fce1be00640 1 --2- 172.21.15.96:0/1827389094 >> v2:172.21.15.123:3300/0 conn(0x7fce1c0d17b0 0x7fce1c0cf2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:10.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.064+0000 7fce1b400640 1 -- 172.21.15.96:0/1827389094 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3023431571 0 0) 0x7fce1c0b13d0 con 0x7fce1c162130 2024-05-28T06:04:10.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.064+0000 7fce1b400640 1 -- 172.21.15.96:0/1827389094 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fce00003520 con 0x7fce1c162130 2024-05-28T06:04:10.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.064+0000 7fce1b400640 1 -- 172.21.15.96:0/1827389094 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2155253931 0 0) 0x7fce00003520 con 0x7fce1c162130 2024-05-28T06:04:10.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.064+0000 7fce1b400640 1 -- 172.21.15.96:0/1827389094 >> v2:172.21.15.123:3300/0 conn(0x7fce1c0d17b0 msgr2=0x7fce1c0cf2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:10.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.064+0000 7fce1b400640 1 --2- 172.21.15.96:0/1827389094 >> v2:172.21.15.123:3300/0 conn(0x7fce1c0d17b0 0x7fce1c0cf2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:10.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.064+0000 7fce1b400640 1 -- 172.21.15.96:0/1827389094 >> v1:172.21.15.123:6789/0 conn(0x7fce1c0b2e50 legacy=0x7fce1c0cf810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:10.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.064+0000 7fce1b400640 1 -- 172.21.15.96:0/1827389094 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce1c0b1920 con 0x7fce1c162130 2024-05-28T06:04:10.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.064+0000 7fce1b400640 1 -- 172.21.15.96:0/1827389094 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7fce04001670 con 0x7fce1c162130 2024-05-28T06:04:10.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.064+0000 7fce1b400640 1 -- 172.21.15.96:0/1827389094 >> v1:172.21.15.96:6789/0 conn(0x7fce1c162130 legacy=0x7fce1c0d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:10.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.064+0000 7fce1b400640 1 --2- 172.21.15.96:0/1827389094 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce000039b0 0x7fce00003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:10.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.064+0000 7fce1b400640 1 --2- 172.21.15.96:0/1827389094 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fce00004800 0x7fce00004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:10.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.064+0000 7fce1b400640 1 --2- 172.21.15.96:0/1827389094 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fce000051c0 0x7fce0000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:10.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.065+0000 7fce1b400640 1 -- 172.21.15.96:0/1827389094 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fce1c0b1640 con 0x7fce00004800 2024-05-28T06:04:10.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.065+0000 7fce1b400640 1 -- 172.21.15.96:0/1827389094 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fce0000c160 con 0x7fce000039b0 2024-05-28T06:04:10.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.065+0000 7fce1b400640 1 -- 172.21.15.96:0/1827389094 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fce0000c360 con 0x7fce000051c0 2024-05-28T06:04:10.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.065+0000 7fce1be00640 1 --2- 172.21.15.96:0/1827389094 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fce00004800 0x7fce00004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:10.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.065+0000 7fce1be00640 1 -- 172.21.15.96:0/1827389094 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fce000051c0 msgr2=0x7fce0000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:10.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.065+0000 7fce1be00640 1 --2- 172.21.15.96:0/1827389094 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fce000051c0 0x7fce0000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:10.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.065+0000 7fce1be00640 1 -- 172.21.15.96:0/1827389094 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce000039b0 msgr2=0x7fce00003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:10.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.065+0000 7fce1be00640 1 --2- 172.21.15.96:0/1827389094 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce000039b0 0x7fce00003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:10.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.065+0000 7fce1be00640 1 -- 172.21.15.96:0/1827389094 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce00003520 con 0x7fce00004800 2024-05-28T06:04:10.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.065+0000 7fce21c00640 1 -- 172.21.15.96:0/1827389094 reap_dead start 2024-05-28T06:04:10.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.066+0000 7fce1be00640 1 --2- 172.21.15.96:0/1827389094 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fce00004800 0x7fce00004c80 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fce0c009fc0 tx=0x7fce00005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b78304ef177c842 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:10.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.066+0000 7fce1b400640 1 -- 172.21.15.96:0/1827389094 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce0c034020 con 0x7fce00004800 2024-05-28T06:04:10.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.066+0000 7fce1b400640 1 -- 172.21.15.96:0/1827389094 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce0c0305f0 con 0x7fce00004800 2024-05-28T06:04:10.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.066+0000 7fce1b400640 1 -- 172.21.15.96:0/1827389094 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce0c009c10 con 0x7fce00004800 2024-05-28T06:04:10.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.067+0000 7fce23c00640 1 -- 172.21.15.96:0/1827389094 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fce00004800 msgr2=0x7fce00004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:10.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.067+0000 7fce23c00640 1 --2- 172.21.15.96:0/1827389094 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fce00004800 0x7fce00004c80 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fce0c009fc0 tx=0x7fce00005830 comp rx=0 tx=0).stop 2024-05-28T06:04:10.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.067+0000 7fce23c00640 1 -- 172.21.15.96:0/1827389094 shutdown_connections 2024-05-28T06:04:10.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.067+0000 7fce23c00640 1 --2- 172.21.15.96:0/1827389094 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fce00004800 0x7fce00004c80 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:10.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.067+0000 7fce23c00640 1 -- 172.21.15.96:0/1827389094 >> 172.21.15.96:0/1827389094 conn(0x7fce1c08bfc0 msgr2=0x7fce1c0d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:10.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.067+0000 7fce23c00640 1 -- 172.21.15.96:0/1827389094 shutdown_connections 2024-05-28T06:04:10.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.067+0000 7fce23c00640 1 -- 172.21.15.96:0/1827389094 wait complete. 2024-05-28T06:04:10.071 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.069+0000 7fce23c00640 1 Processor -- start 2024-05-28T06:04:10.071 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.069+0000 7fce23c00640 1 -- start start 2024-05-28T06:04:10.071 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.070+0000 7fce23c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fce00004800 0x7fce1c0c8a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:10.071 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.070+0000 7fce23c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce1c0cb7b0 0x7fce1c15af90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:10.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.070+0000 7fce23c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fce1c15b790 0x7fce1c074990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:10.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.070+0000 7fce23c00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fce1c162690 con 0x7fce00004800 2024-05-28T06:04:10.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.070+0000 7fce23c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fce1c162510 con 0x7fce1c0cb7b0 2024-05-28T06:04:10.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.070+0000 7fce23c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fce1c0d3690 con 0x7fce1c15b790 2024-05-28T06:04:10.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.070+0000 7fce21200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fce00004800 0x7fce1c0c8a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:10.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.070+0000 7fce21200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fce00004800 0x7fce1c0c8a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:41394/0 (socket says 172.21.15.96:41394) 2024-05-28T06:04:10.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.070+0000 7fce21200640 1 -- 172.21.15.96:0/1843233761 learned_addr learned my addr 172.21.15.96:0/1843233761 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:10.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.070+0000 7fce21200640 1 -- 172.21.15.96:0/1843233761 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fce1c15b790 msgr2=0x7fce1c074990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:10.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.070+0000 7fce21c00640 1 --2- 172.21.15.96:0/1843233761 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fce1c15b790 0x7fce1c074990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:10.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.070+0000 7fce21200640 1 --2- 172.21.15.96:0/1843233761 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fce1c15b790 0x7fce1c074990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:10.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.070+0000 7fce1be00640 1 --2- 172.21.15.96:0/1843233761 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce1c0cb7b0 0x7fce1c15af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:10.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.070+0000 7fce21200640 1 -- 172.21.15.96:0/1843233761 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce1c0cb7b0 msgr2=0x7fce1c15af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:10.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.071+0000 7fce21200640 1 --2- 172.21.15.96:0/1843233761 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce1c0cb7b0 0x7fce1c15af90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:10.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.071+0000 7fce21200640 1 -- 172.21.15.96:0/1843233761 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce1c15c010 con 0x7fce00004800 2024-05-28T06:04:10.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.071+0000 7fce21200640 1 --2- 172.21.15.96:0/1843233761 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fce00004800 0x7fce1c0c8a40 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fce0400c680 tx=0x7fce0400cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:10.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.071+0000 7fce18c00640 1 -- 172.21.15.96:0/1843233761 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce04018070 con 0x7fce00004800 2024-05-28T06:04:10.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.071+0000 7fce18c00640 1 -- 172.21.15.96:0/1843233761 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce040072d0 con 0x7fce00004800 2024-05-28T06:04:10.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.071+0000 7fce18c00640 1 -- 172.21.15.96:0/1843233761 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce040047e0 con 0x7fce00004800 2024-05-28T06:04:10.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.071+0000 7fce23c00640 1 -- 172.21.15.96:0/1843233761 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce1c0c8f80 con 0x7fce00004800 2024-05-28T06:04:10.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.071+0000 7fce23c00640 1 -- 172.21.15.96:0/1843233761 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce1c0c9520 con 0x7fce00004800 2024-05-28T06:04:10.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.072+0000 7fce23c00640 1 -- 172.21.15.96:0/1843233761 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce1c0690f0 con 0x7fce00004800 2024-05-28T06:04:10.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.073+0000 7fce18c00640 1 -- 172.21.15.96:0/1843233761 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fce0401b070 con 0x7fce00004800 2024-05-28T06:04:10.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.074+0000 7fce18c00640 1 --2- 172.21.15.96:0/1843233761 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fcdf806c920 0x7fcdf806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:10.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.074+0000 7fce18c00640 1 -- 172.21.15.96:0/1843233761 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 5609+0+0 (secure 0 0 0) 0x7fce0408afe0 con 0x7fce00004800 2024-05-28T06:04:10.075 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.075+0000 7fce1be00640 1 --2- 172.21.15.96:0/1843233761 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fcdf806c920 0x7fcdf806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:10.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.076+0000 7fce1be00640 1 --2- 172.21.15.96:0/1843233761 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fcdf806c920 0x7fcdf806ede0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fce0c035000 tx=0x7fce0c02f000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:10.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.079+0000 7fce18c00640 1 -- 172.21.15.96:0/1843233761 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fce1c15c010 con 0x7fce00004800 2024-05-28T06:04:10.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.318+0000 7fce23c00640 1 -- 172.21.15.96:0/1843233761 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fce1c15e820 con 0x7fce00004800 2024-05-28T06:04:10.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.883+0000 7fce18c00640 1 -- 172.21.15.96:0/1843233761 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) ==== 172+0+0 (secure 0 0 0) 0x7fce04059410 con 0x7fce00004800 2024-05-28T06:04:10.889 INFO:teuthology.orchestra.run.smithi096.stderr:new fs with metadata pool 2 and data pool 3 2024-05-28T06:04:10.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.894+0000 7fce23c00640 1 -- 172.21.15.96:0/1843233761 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fcdf806c920 msgr2=0x7fcdf806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:10.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.895+0000 7fce23c00640 1 --2- 172.21.15.96:0/1843233761 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fcdf806c920 0x7fcdf806ede0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fce0c035000 tx=0x7fce0c02f000 comp rx=0 tx=0).stop 2024-05-28T06:04:10.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.895+0000 7fce23c00640 1 -- 172.21.15.96:0/1843233761 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fce00004800 msgr2=0x7fce1c0c8a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:10.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.895+0000 7fce23c00640 1 --2- 172.21.15.96:0/1843233761 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fce00004800 0x7fce1c0c8a40 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fce0400c680 tx=0x7fce0400cb40 comp rx=0 tx=0).stop 2024-05-28T06:04:10.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.895+0000 7fce23c00640 1 -- 172.21.15.96:0/1843233761 shutdown_connections 2024-05-28T06:04:10.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.895+0000 7fce23c00640 1 --2- 172.21.15.96:0/1843233761 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fcdf806c920 0x7fcdf806ede0 secure :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fce0c035000 tx=0x7fce0c02f000 comp rx=0 tx=0).stop 2024-05-28T06:04:10.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.895+0000 7fce23c00640 1 --2- 172.21.15.96:0/1843233761 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fce1c15b790 0x7fce1c074990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:10.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.895+0000 7fce23c00640 1 --2- 172.21.15.96:0/1843233761 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce1c0cb7b0 0x7fce1c15af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:10.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.895+0000 7fce23c00640 1 --2- 172.21.15.96:0/1843233761 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fce00004800 0x7fce1c0c8a40 secure :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fce0400c680 tx=0x7fce0400cb40 comp rx=0 tx=0).stop 2024-05-28T06:04:10.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.895+0000 7fce23c00640 1 -- 172.21.15.96:0/1843233761 >> 172.21.15.96:0/1843233761 conn(0x7fce1c08bfc0 msgr2=0x7fce1c15db00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:10.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.896+0000 7fce23c00640 1 -- 172.21.15.96:0/1843233761 shutdown_connections 2024-05-28T06:04:10.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:10.897+0000 7fce23c00640 1 -- 172.21.15.96:0/1843233761 wait complete. 2024-05-28T06:04:10.928 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:04:11.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.096+0000 7fb6f2600640 1 Processor -- start 2024-05-28T06:04:11.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.096+0000 7fb6f2600640 1 -- start start 2024-05-28T06:04:11.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.097+0000 7fb6f2600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fb6ec0b4e10 0x7fb6ec0cde30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:11.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.097+0000 7fb6f2600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb6ec0d0ca0 0x7fb6ec099090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:11.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.097+0000 7fb6f2600640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6ec0b51f0 con 0x7fb6ec0ce370 2024-05-28T06:04:11.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.097+0000 7fb6f2600640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fb6ec0aef60 con 0x7fb6ec0b4e10 2024-05-28T06:04:11.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.097+0000 7fb6f2600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fb6ec0ad1d0 con 0x7fb6ec0d0ca0 2024-05-28T06:04:11.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.097+0000 7fb6eb400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fb6ec0b4e10 0x7fb6ec0cde30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:11.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.097+0000 7fb6e2a00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fb6ec0ce370 0x7fb6ec0d0590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:55380/0 (socket says 172.21.15.96:55380) 2024-05-28T06:04:11.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.097+0000 7fb6e2a00640 1 -- 172.21.15.96:0/473809397 learned_addr learned my addr 172.21.15.96:0/473809397 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:04:11.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.098+0000 7fb6eb400640 1 -- 172.21.15.96:0/473809397 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb6ec0d0ca0 msgr2=0x7fb6ec099090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:11.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.098+0000 7fb6eb400640 1 --2- 172.21.15.96:0/473809397 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb6ec0d0ca0 0x7fb6ec099090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:11.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.098+0000 7fb6eb400640 1 -- 172.21.15.96:0/473809397 >> v1:172.21.15.96:6789/0 conn(0x7fb6ec0ce370 legacy=0x7fb6ec0d0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:11.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.098+0000 7fb6eb400640 1 -- 172.21.15.96:0/473809397 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6ec0ad470 con 0x7fb6ec0b4e10 2024-05-28T06:04:11.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.099+0000 7fb6eb400640 1 --2- 172.21.15.96:0/473809397 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb6ec0b4e10 0x7fb6ec0cde30 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb6dc0097a0 tx=0x7fb6dc02c320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b79a54830be3e4b server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:11.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.099+0000 7fb6eaa00640 1 -- 172.21.15.96:0/473809397 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6dc039070 con 0x7fb6ec0b4e10 2024-05-28T06:04:11.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.099+0000 7fb6eaa00640 1 -- 172.21.15.96:0/473809397 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6dc034640 con 0x7fb6ec0b4e10 2024-05-28T06:04:11.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.099+0000 7fb6eaa00640 1 -- 172.21.15.96:0/473809397 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6dc034a10 con 0x7fb6ec0b4e10 2024-05-28T06:04:11.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.100+0000 7fb6f2600640 1 -- 172.21.15.96:0/473809397 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb6ec0b4e10 msgr2=0x7fb6ec0cde30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:11.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.100+0000 7fb6f2600640 1 --2- 172.21.15.96:0/473809397 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb6ec0b4e10 0x7fb6ec0cde30 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb6dc0097a0 tx=0x7fb6dc02c320 comp rx=0 tx=0).stop 2024-05-28T06:04:11.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.100+0000 7fb6f2600640 1 -- 172.21.15.96:0/473809397 shutdown_connections 2024-05-28T06:04:11.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.100+0000 7fb6f2600640 1 --2- 172.21.15.96:0/473809397 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb6ec0d0ca0 0x7fb6ec099090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:11.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.100+0000 7fb6f2600640 1 --2- 172.21.15.96:0/473809397 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb6ec0b4e10 0x7fb6ec0cde30 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:11.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.100+0000 7fb6f2600640 1 -- 172.21.15.96:0/473809397 >> 172.21.15.96:0/473809397 conn(0x7fb6ec06bae0 msgr2=0x7fb6ec06bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:11.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.100+0000 7fb6f2600640 1 -- 172.21.15.96:0/473809397 shutdown_connections 2024-05-28T06:04:11.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.100+0000 7fb6f2600640 1 -- 172.21.15.96:0/473809397 wait complete. 2024-05-28T06:04:11.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.101+0000 7fb6f2600640 1 Processor -- start 2024-05-28T06:04:11.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.102+0000 7fb6f2600640 1 -- start start 2024-05-28T06:04:11.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.102+0000 7fb6f2600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb6ec0b4e10 0x7fb6ec0afdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:11.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.102+0000 7fb6f2600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb6ec0ce370 0x7fb6ec072b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:11.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.102+0000 7fb6f2600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb6ec0d0ca0 0x7fb6ec0730c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:11.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.102+0000 7fb6f2600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb6ec0cea00 con 0x7fb6ec0b4e10 2024-05-28T06:04:11.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.102+0000 7fb6f2600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fb6ec0ce880 con 0x7fb6ec0d0ca0 2024-05-28T06:04:11.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.102+0000 7fb6f2600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fb6ec0b3150 con 0x7fb6ec0ce370 2024-05-28T06:04:11.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.103+0000 7fb6eb400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb6ec0b4e10 0x7fb6ec0afdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:11.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.103+0000 7fb6eb400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb6ec0b4e10 0x7fb6ec0afdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:41418/0 (socket says 172.21.15.96:41418) 2024-05-28T06:04:11.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.103+0000 7fb6eb400640 1 -- 172.21.15.96:0/646806620 learned_addr learned my addr 172.21.15.96:0/646806620 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:11.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.103+0000 7fb6eb400640 1 -- 172.21.15.96:0/646806620 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb6ec0ce370 msgr2=0x7fb6ec072b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:11.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.103+0000 7fb6eb400640 1 --2- 172.21.15.96:0/646806620 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb6ec0ce370 0x7fb6ec072b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:11.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.103+0000 7fb6eb400640 1 -- 172.21.15.96:0/646806620 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb6ec0d0ca0 msgr2=0x7fb6ec0730c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:11.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.103+0000 7fb6eb400640 1 --2- 172.21.15.96:0/646806620 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb6ec0d0ca0 0x7fb6ec0730c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:11.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.103+0000 7fb6eb400640 1 -- 172.21.15.96:0/646806620 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6ec1540a0 con 0x7fb6ec0b4e10 2024-05-28T06:04:11.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.103+0000 7fb6eb400640 1 --2- 172.21.15.96:0/646806620 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb6ec0b4e10 0x7fb6ec0afdc0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb6dc0097a0 tx=0x7fb6dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:11.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.103+0000 7fb6e3e00640 1 -- 172.21.15.96:0/646806620 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6dc039040 con 0x7fb6ec0b4e10 2024-05-28T06:04:11.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.104+0000 7fb6e3e00640 1 -- 172.21.15.96:0/646806620 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6dc044070 con 0x7fb6ec0b4e10 2024-05-28T06:04:11.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.104+0000 7fb6e3e00640 1 -- 172.21.15.96:0/646806620 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6dc032990 con 0x7fb6ec0b4e10 2024-05-28T06:04:11.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.104+0000 7fb6f2600640 1 -- 172.21.15.96:0/646806620 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6ec0b2860 con 0x7fb6ec0b4e10 2024-05-28T06:04:11.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.104+0000 7fb6f2600640 1 -- 172.21.15.96:0/646806620 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6ec154340 con 0x7fb6ec0b4e10 2024-05-28T06:04:11.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.105+0000 7fb6f2600640 1 -- 172.21.15.96:0/646806620 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6ec0689e0 con 0x7fb6ec0b4e10 2024-05-28T06:04:11.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.106+0000 7fb6e3e00640 1 -- 172.21.15.96:0/646806620 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb6dc032b30 con 0x7fb6ec0b4e10 2024-05-28T06:04:11.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.106+0000 7fb6e3e00640 1 --2- 172.21.15.96:0/646806620 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb6c806c920 0x7fb6c806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:11.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.107+0000 7fb6e3e00640 1 -- 172.21.15.96:0/646806620 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 5709+0+0 (secure 0 0 0) 0x7fb6dc0ac6a0 con 0x7fb6ec0b4e10 2024-05-28T06:04:11.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.107+0000 7fb6e2a00640 1 --2- 172.21.15.96:0/646806620 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb6c806c920 0x7fb6c806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:11.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.108+0000 7fb6e2a00640 1 --2- 172.21.15.96:0/646806620 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb6c806c920 0x7fb6c806ede0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb6d4000f60 tx=0x7fb6d4007a80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:11.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.111+0000 7fb6e3e00640 1 -- 172.21.15.96:0/646806620 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb6dc032e20 con 0x7fb6ec0b4e10 2024-05-28T06:04:11.308 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.308+0000 7fb6f2600640 1 -- 172.21.15.96:0/646806620 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb6ec0cd450 con 0x7fb6ec0b4e10 2024-05-28T06:04:11.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.311+0000 7fb6e3e00640 1 -- 172.21.15.96:0/646806620 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) ==== 74+0+16630 (secure 0 0 0) 0x7fb6dc07aad0 con 0x7fb6ec0b4e10 2024-05-28T06:04:11.312 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:04:11.312 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":19,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:04:10.319832+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-28T06:04:08.741938+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-28T06:04:09.301470+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:04:11.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.313+0000 7fb6f2600640 1 -- 172.21.15.96:0/646806620 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb6c806c920 msgr2=0x7fb6c806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:11.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.313+0000 7fb6f2600640 1 --2- 172.21.15.96:0/646806620 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb6c806c920 0x7fb6c806ede0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb6d4000f60 tx=0x7fb6d4007a80 comp rx=0 tx=0).stop 2024-05-28T06:04:11.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.313+0000 7fb6f2600640 1 -- 172.21.15.96:0/646806620 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb6ec0b4e10 msgr2=0x7fb6ec0afdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:11.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.313+0000 7fb6f2600640 1 --2- 172.21.15.96:0/646806620 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb6ec0b4e10 0x7fb6ec0afdc0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb6dc0097a0 tx=0x7fb6dc0046c0 comp rx=0 tx=0).stop 2024-05-28T06:04:11.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.313+0000 7fb6f2600640 1 -- 172.21.15.96:0/646806620 shutdown_connections 2024-05-28T06:04:11.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.313+0000 7fb6f2600640 1 --2- 172.21.15.96:0/646806620 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb6c806c920 0x7fb6c806ede0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:11.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.313+0000 7fb6f2600640 1 --2- 172.21.15.96:0/646806620 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb6ec0d0ca0 0x7fb6ec0730c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:11.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.313+0000 7fb6f2600640 1 --2- 172.21.15.96:0/646806620 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb6ec0ce370 0x7fb6ec072b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:11.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.313+0000 7fb6f2600640 1 --2- 172.21.15.96:0/646806620 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb6ec0b4e10 0x7fb6ec0afdc0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:11.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.313+0000 7fb6f2600640 1 -- 172.21.15.96:0/646806620 >> 172.21.15.96:0/646806620 conn(0x7fb6ec06bae0 msgr2=0x7fb6ec0ccc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:11.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.313+0000 7fb6f2600640 1 -- 172.21.15.96:0/646806620 shutdown_connections 2024-05-28T06:04:11.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.313+0000 7fb6f2600640 1 -- 172.21.15.96:0/646806620 wait complete. 2024-05-28T06:04:11.331 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:04:11.530 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.530+0000 7fd75e800640 1 Processor -- start 2024-05-28T06:04:11.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.530+0000 7fd75e800640 1 -- start start 2024-05-28T06:04:11.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.531+0000 7fd75e800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fd7580b4b60 0x7fd7580cfaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:11.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.532+0000 7fd75e800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd7580b3b70 0x7fd7580d0930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:11.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.532+0000 7fd75e800640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7580ced40 con 0x7fd7580d04a0 2024-05-28T06:04:11.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.532+0000 7fd75e800640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fd7580b3fe0 con 0x7fd7580b4b60 2024-05-28T06:04:11.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.532+0000 7fd75e800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fd75815fbb0 con 0x7fd7580b3b70 2024-05-28T06:04:11.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.532+0000 7fd757400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fd7580b4b60 0x7fd7580cfaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:11.533 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.532+0000 7fd757400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fd7580b4b60 0x7fd7580cfaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:41422/0 (socket says 172.21.15.96:41422) 2024-05-28T06:04:11.533 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.532+0000 7fd757400640 1 -- 172.21.15.96:0/2057414530 learned_addr learned my addr 172.21.15.96:0/2057414530 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:11.533 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.532+0000 7fd757400640 1 -- 172.21.15.96:0/2057414530 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd7580b3b70 msgr2=0x7fd7580d0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:11.533 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.532+0000 7fd757400640 1 --2- 172.21.15.96:0/2057414530 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd7580b3b70 0x7fd7580d0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:11.533 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.532+0000 7fd757400640 1 -- 172.21.15.96:0/2057414530 >> v1:172.21.15.123:6789/0 conn(0x7fd7580d04a0 legacy=0x7fd7580b3460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:11.533 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.532+0000 7fd757400640 1 -- 172.21.15.96:0/2057414530 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7580cefc0 con 0x7fd7580b4b60 2024-05-28T06:04:11.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.534+0000 7fd757400640 1 --2- 172.21.15.96:0/2057414530 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd7580b4b60 0x7fd7580cfaa0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd7580cdd80 tx=0x7fd74802a200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f975b773d8eb358d server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:11.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.534+0000 7fd756000640 1 -- 172.21.15.96:0/2057414530 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd748037070 con 0x7fd7580b4b60 2024-05-28T06:04:11.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.534+0000 7fd756000640 1 -- 172.21.15.96:0/2057414530 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd748032440 con 0x7fd7580b4b60 2024-05-28T06:04:11.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.534+0000 7fd756000640 1 -- 172.21.15.96:0/2057414530 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd748032810 con 0x7fd7580b4b60 2024-05-28T06:04:11.536 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.535+0000 7fd75e800640 1 -- 172.21.15.96:0/2057414530 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd7580b4b60 msgr2=0x7fd7580cfaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:11.536 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.535+0000 7fd75e800640 1 --2- 172.21.15.96:0/2057414530 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd7580b4b60 0x7fd7580cfaa0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd7580cdd80 tx=0x7fd74802a200 comp rx=0 tx=0).stop 2024-05-28T06:04:11.536 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.535+0000 7fd75e800640 1 -- 172.21.15.96:0/2057414530 shutdown_connections 2024-05-28T06:04:11.536 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.535+0000 7fd75e800640 1 --2- 172.21.15.96:0/2057414530 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd7580b3b70 0x7fd7580d0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:11.536 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.535+0000 7fd75e800640 1 --2- 172.21.15.96:0/2057414530 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd7580b4b60 0x7fd7580cfaa0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:11.536 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.535+0000 7fd75e800640 1 -- 172.21.15.96:0/2057414530 >> 172.21.15.96:0/2057414530 conn(0x7fd75806bae0 msgr2=0x7fd75806bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:11.536 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.535+0000 7fd75e800640 1 -- 172.21.15.96:0/2057414530 shutdown_connections 2024-05-28T06:04:11.536 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.535+0000 7fd75e800640 1 -- 172.21.15.96:0/2057414530 wait complete. 2024-05-28T06:04:11.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.537+0000 7fd75e800640 1 Processor -- start 2024-05-28T06:04:11.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.537+0000 7fd75e800640 1 -- start start 2024-05-28T06:04:11.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.538+0000 7fd75e800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd7580b3b70 0x7fd7580c3d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:11.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.538+0000 7fd75e800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd7580b4b60 0x7fd758099090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:11.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.538+0000 7fd75e800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7580d04a0 0x7fd7580b05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:11.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.538+0000 7fd75e800640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd7580cffe0 con 0x7fd7580b4b60 2024-05-28T06:04:11.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.538+0000 7fd75e800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd7580cf580 con 0x7fd7580d04a0 2024-05-28T06:04:11.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.538+0000 7fd75e800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fd75815ede0 con 0x7fd7580b3b70 2024-05-28T06:04:11.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.538+0000 7fd756a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd7580b4b60 0x7fd758099090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:11.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.539+0000 7fd756a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd7580b4b60 0x7fd758099090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:41428/0 (socket says 172.21.15.96:41428) 2024-05-28T06:04:11.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.539+0000 7fd756a00640 1 -- 172.21.15.96:0/1111825969 learned_addr learned my addr 172.21.15.96:0/1111825969 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:11.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.539+0000 7fd757e00640 1 --2- 172.21.15.96:0/1111825969 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7580d04a0 0x7fd7580b05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:11.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.539+0000 7fd756a00640 1 -- 172.21.15.96:0/1111825969 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd7580b3b70 msgr2=0x7fd7580c3d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:11.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.539+0000 7fd756a00640 1 --2- 172.21.15.96:0/1111825969 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd7580b3b70 0x7fd7580c3d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:11.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.539+0000 7fd756a00640 1 -- 172.21.15.96:0/1111825969 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7580d04a0 msgr2=0x7fd7580b05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:11.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.539+0000 7fd756a00640 1 --2- 172.21.15.96:0/1111825969 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7580d04a0 0x7fd7580b05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:11.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.539+0000 7fd756a00640 1 -- 172.21.15.96:0/1111825969 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7580c4350 con 0x7fd7580b4b60 2024-05-28T06:04:11.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.539+0000 7fd757e00640 1 --2- 172.21.15.96:0/1111825969 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7580d04a0 0x7fd7580b05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:04:11.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.539+0000 7fd756a00640 1 --2- 172.21.15.96:0/1111825969 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd7580b4b60 0x7fd758099090 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd74400b3e0 tx=0x7fd74400b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:11.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.539+0000 7fd74f400640 1 -- 172.21.15.96:0/1111825969 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd744015020 con 0x7fd7580b4b60 2024-05-28T06:04:11.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.540+0000 7fd74f400640 1 -- 172.21.15.96:0/1111825969 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd744011850 con 0x7fd7580b4b60 2024-05-28T06:04:11.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.540+0000 7fd74f400640 1 -- 172.21.15.96:0/1111825969 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd744011c40 con 0x7fd7580b4b60 2024-05-28T06:04:11.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.540+0000 7fd75e800640 1 -- 172.21.15.96:0/1111825969 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7580c4630 con 0x7fd7580b4b60 2024-05-28T06:04:11.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.540+0000 7fd75e800640 1 -- 172.21.15.96:0/1111825969 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7580c4bf0 con 0x7fd7580b4b60 2024-05-28T06:04:11.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.541+0000 7fd75e800640 1 -- 172.21.15.96:0/1111825969 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7580689e0 con 0x7fd7580b4b60 2024-05-28T06:04:11.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.542+0000 7fd74f400640 1 -- 172.21.15.96:0/1111825969 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd744018020 con 0x7fd7580b4b60 2024-05-28T06:04:11.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.543+0000 7fd74f400640 1 --2- 172.21.15.96:0/1111825969 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd73806c920 0x7fd73806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:11.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.543+0000 7fd74f400640 1 -- 172.21.15.96:0/1111825969 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 5709+0+0 (secure 0 0 0) 0x7fd74408bbc0 con 0x7fd7580b4b60 2024-05-28T06:04:11.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.543+0000 7fd757400640 1 --2- 172.21.15.96:0/1111825969 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd73806c920 0x7fd73806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:11.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.544+0000 7fd757400640 1 --2- 172.21.15.96:0/1111825969 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd73806c920 0x7fd73806ede0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd748034000 tx=0x7fd74802c000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:11.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.547+0000 7fd74f400640 1 -- 172.21.15.96:0/1111825969 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd74401f020 con 0x7fd7580b4b60 2024-05-28T06:04:11.746 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.746+0000 7fd75e800640 1 -- 172.21.15.96:0/1111825969 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd758161550 con 0x7fd7580b4b60 2024-05-28T06:04:11.750 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.749+0000 7fd74f400640 1 -- 172.21.15.96:0/1111825969 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) ==== 74+0+16630 (secure 0 0 0) 0x7fd7440172c0 con 0x7fd7580b4b60 2024-05-28T06:04:11.751 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:04:11.751 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":19,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:04:10.319832+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-28T06:04:08.741938+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-28T06:04:09.301470+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:04:11.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.752+0000 7fd75e800640 1 -- 172.21.15.96:0/1111825969 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd73806c920 msgr2=0x7fd73806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:11.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.752+0000 7fd75e800640 1 --2- 172.21.15.96:0/1111825969 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd73806c920 0x7fd73806ede0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd748034000 tx=0x7fd74802c000 comp rx=0 tx=0).stop 2024-05-28T06:04:11.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.752+0000 7fd75e800640 1 -- 172.21.15.96:0/1111825969 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd7580b4b60 msgr2=0x7fd758099090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:11.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.752+0000 7fd75e800640 1 --2- 172.21.15.96:0/1111825969 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd7580b4b60 0x7fd758099090 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd74400b3e0 tx=0x7fd74400b8a0 comp rx=0 tx=0).stop 2024-05-28T06:04:11.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.752+0000 7fd75e800640 1 -- 172.21.15.96:0/1111825969 shutdown_connections 2024-05-28T06:04:11.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.752+0000 7fd75e800640 1 --2- 172.21.15.96:0/1111825969 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd73806c920 0x7fd73806ede0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:11.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.752+0000 7fd75e800640 1 --2- 172.21.15.96:0/1111825969 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7580d04a0 0x7fd7580b05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:11.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.752+0000 7fd75e800640 1 --2- 172.21.15.96:0/1111825969 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd7580b4b60 0x7fd758099090 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:11.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.752+0000 7fd75e800640 1 --2- 172.21.15.96:0/1111825969 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd7580b3b70 0x7fd7580c3d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:11.755 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.752+0000 7fd75e800640 1 -- 172.21.15.96:0/1111825969 >> 172.21.15.96:0/1111825969 conn(0x7fd75806bae0 msgr2=0x7fd7580b2ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:11.755 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.752+0000 7fd75e800640 1 -- 172.21.15.96:0/1111825969 shutdown_connections 2024-05-28T06:04:11.755 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.752+0000 7fd75e800640 1 -- 172.21.15.96:0/1111825969 wait complete. 2024-05-28T06:04:11.771 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-05-28T06:04:11.824 INFO:tasks.ceph.mds.b.smithi123.stderr:2024-05-28T06:04:11.823+0000 7fda5f400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-28T06:04:11.973 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.972+0000 7f69b3200640 1 Processor -- start 2024-05-28T06:04:11.973 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.973+0000 7f69b3200640 1 -- start start 2024-05-28T06:04:11.974 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.974+0000 7f69b3200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f69b4161ed0 0x7f69b40b49e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:11.974 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.974+0000 7f69b3200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f69b40ced60 0x7f69b40cf210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:11.974 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.974+0000 7f69b3200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69b40b2290 con 0x7f69b40d1bb0 2024-05-28T06:04:11.974 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.974+0000 7f69b3200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f69b40d2000 con 0x7f69b4161ed0 2024-05-28T06:04:11.974 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.974+0000 7f69b3200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f69b40d0690 con 0x7f69b40ced60 2024-05-28T06:04:11.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.975+0000 7f69b2800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f69b40ced60 0x7f69b40cf210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:11.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.975+0000 7f69b2800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f69b40ced60 0x7f69b40cf210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.96:58202/0 (socket says 172.21.15.96:58202) 2024-05-28T06:04:11.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.975+0000 7f69b1400640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f69b40d1bb0 0x7f69b406ca10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.96:41538/0 (socket says 172.21.15.96:41538) 2024-05-28T06:04:11.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.975+0000 7f69b2800640 1 -- 172.21.15.96:0/845885420 learned_addr learned my addr 172.21.15.96:0/845885420 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:11.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.975+0000 7f69b1e00640 1 --2- 172.21.15.96:0/845885420 >> v2:172.21.15.123:3300/0 conn(0x7f69b4161ed0 0x7f69b40b49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:11.976 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.975+0000 7f69b0a00640 1 -- 172.21.15.96:0/845885420 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3268023232 0 0) 0x7f69a0001110 con 0x7f69b40d1bb0 2024-05-28T06:04:11.976 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.975+0000 7f69b0a00640 1 -- 172.21.15.96:0/845885420 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6998003520 con 0x7f69b40d1bb0 2024-05-28T06:04:11.976 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.976+0000 7f69b2800640 1 -- 172.21.15.96:0/845885420 >> v2:172.21.15.123:3300/0 conn(0x7f69b4161ed0 msgr2=0x7f69b40b49e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:11.976 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.976+0000 7f69b2800640 1 --2- 172.21.15.96:0/845885420 >> v2:172.21.15.123:3300/0 conn(0x7f69b4161ed0 0x7f69b40b49e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:11.976 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.976+0000 7f69b2800640 1 -- 172.21.15.96:0/845885420 >> v1:172.21.15.123:6789/0 conn(0x7f69b40d1bb0 legacy=0x7f69b406ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:11.976 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.976+0000 7f69b2800640 1 -- 172.21.15.96:0/845885420 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69b40d03f0 con 0x7f69b40ced60 2024-05-28T06:04:11.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.977+0000 7f69b2800640 1 --2- 172.21.15.96:0/845885420 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f69b40ced60 0x7f69b40cf210 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f69ac009e50 tx=0x7f69ac02a5f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e8fb6b4d00992c60 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:11.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.977+0000 7f69b0a00640 1 -- 172.21.15.96:0/845885420 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69ac036070 con 0x7f69b40ced60 2024-05-28T06:04:11.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.977+0000 7f69b0a00640 1 -- 172.21.15.96:0/845885420 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69ac031600 con 0x7f69b40ced60 2024-05-28T06:04:11.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.977+0000 7f69b0a00640 1 -- 172.21.15.96:0/845885420 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69ac0319b0 con 0x7f69b40ced60 2024-05-28T06:04:11.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.978+0000 7f69b3200640 1 -- 172.21.15.96:0/845885420 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f69b40ced60 msgr2=0x7f69b40cf210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:11.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.978+0000 7f69b3200640 1 --2- 172.21.15.96:0/845885420 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f69b40ced60 0x7f69b40cf210 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f69ac009e50 tx=0x7f69ac02a5f0 comp rx=0 tx=0).stop 2024-05-28T06:04:11.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.978+0000 7f69b3200640 1 -- 172.21.15.96:0/845885420 shutdown_connections 2024-05-28T06:04:11.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.978+0000 7f69b3200640 1 --2- 172.21.15.96:0/845885420 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f69b40ced60 0x7f69b40cf210 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:11.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.978+0000 7f69b3200640 1 --2- 172.21.15.96:0/845885420 >> v2:172.21.15.123:3300/0 conn(0x7f69b4161ed0 0x7f69b40b49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:11.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.978+0000 7f69b3200640 1 -- 172.21.15.96:0/845885420 >> 172.21.15.96:0/845885420 conn(0x7f69b40b3ec0 msgr2=0x7f69b40b42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:11.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.978+0000 7f69b3200640 1 -- 172.21.15.96:0/845885420 shutdown_connections 2024-05-28T06:04:11.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.978+0000 7f69b3200640 1 -- 172.21.15.96:0/845885420 wait complete. 2024-05-28T06:04:11.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.980+0000 7f69b3200640 1 Processor -- start 2024-05-28T06:04:11.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.980+0000 7f69b3200640 1 -- start start 2024-05-28T06:04:11.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.981+0000 7f69b3200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f69b40ced60 0x7f69b409c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:11.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.981+0000 7f69b3200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69b40d1bb0 0x7f69b41510c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:11.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.981+0000 7f69b3200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f69b4161ed0 0x7f69b4070290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:11.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.981+0000 7f69b3200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f69b40b3a60 con 0x7f69b4161ed0 2024-05-28T06:04:11.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.981+0000 7f69b3200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f69b40b1700 con 0x7f69b40d1bb0 2024-05-28T06:04:11.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.981+0000 7f69b3200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f69b40b2750 con 0x7f69b40ced60 2024-05-28T06:04:11.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.981+0000 7f69b2800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f69b4161ed0 0x7f69b4070290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:11.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.981+0000 7f69b2800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f69b4161ed0 0x7f69b4070290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:41434/0 (socket says 172.21.15.96:41434) 2024-05-28T06:04:11.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.981+0000 7f69b2800640 1 -- 172.21.15.96:0/1735103144 learned_addr learned my addr 172.21.15.96:0/1735103144 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:11.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.981+0000 7f69b1e00640 1 --2- 172.21.15.96:0/1735103144 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f69b40ced60 0x7f69b409c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:11.986 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.981+0000 7f69b2800640 1 -- 172.21.15.96:0/1735103144 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f69b40ced60 msgr2=0x7f69b409c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:11.986 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.981+0000 7f69b1400640 1 --2- 172.21.15.96:0/1735103144 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69b40d1bb0 0x7f69b41510c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:11.986 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.981+0000 7f69b2800640 1 --2- 172.21.15.96:0/1735103144 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f69b40ced60 0x7f69b409c2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:11.986 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.981+0000 7f69b2800640 1 -- 172.21.15.96:0/1735103144 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69b40d1bb0 msgr2=0x7f69b41510c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:11.986 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.981+0000 7f69b2800640 1 --2- 172.21.15.96:0/1735103144 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69b40d1bb0 0x7f69b41510c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:11.986 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.981+0000 7f69b2800640 1 -- 172.21.15.96:0/1735103144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69b414e700 con 0x7f69b4161ed0 2024-05-28T06:04:11.986 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.982+0000 7f69b2800640 1 --2- 172.21.15.96:0/1735103144 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f69b4161ed0 0x7f69b4070290 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f69ac034040 tx=0x7f69ac004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:11.986 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.982+0000 7f69aa000640 1 -- 172.21.15.96:0/1735103144 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69ac036050 con 0x7f69b4161ed0 2024-05-28T06:04:11.986 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.982+0000 7f69aa000640 1 -- 172.21.15.96:0/1735103144 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69ac042070 con 0x7f69b4161ed0 2024-05-28T06:04:11.986 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.982+0000 7f69b3200640 1 -- 172.21.15.96:0/1735103144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69b4151600 con 0x7f69b4161ed0 2024-05-28T06:04:11.986 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.982+0000 7f69aa000640 1 -- 172.21.15.96:0/1735103144 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69ac004450 con 0x7f69b4161ed0 2024-05-28T06:04:11.986 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.982+0000 7f69b3200640 1 -- 172.21.15.96:0/1735103144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69b414ec80 con 0x7f69b4161ed0 2024-05-28T06:04:11.987 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.983+0000 7f69b3200640 1 -- 172.21.15.96:0/1735103144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69b40689e0 con 0x7f69b4161ed0 2024-05-28T06:04:11.987 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.984+0000 7f69aa000640 1 -- 172.21.15.96:0/1735103144 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f69ac0045f0 con 0x7f69b4161ed0 2024-05-28T06:04:11.987 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.985+0000 7f69aa000640 1 --2- 172.21.15.96:0/1735103144 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f699406c8f0 0x7f699406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:11.988 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.985+0000 7f69aa000640 1 -- 172.21.15.96:0/1735103144 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f69ac0acac0 con 0x7f69b4161ed0 2024-05-28T06:04:11.988 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.986+0000 7f69b1e00640 1 --2- 172.21.15.96:0/1735103144 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f699406c8f0 0x7f699406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:11.988 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.986+0000 7f69b1e00640 1 --2- 172.21.15.96:0/1735103144 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f699406c8f0 0x7f699406edb0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f69a4009970 tx=0x7f69a4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:11.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:11.989+0000 7f69aa000640 1 -- 172.21.15.96:0/1735103144 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f69ac045360 con 0x7f69b4161ed0 2024-05-28T06:04:12.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:12.207+0000 7f69b3200640 1 -- 172.21.15.96:0/1735103144 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f69b4074c70 con 0x7f69b4161ed0 2024-05-28T06:04:13.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:13.207+0000 7f69aa000640 1 -- 172.21.15.96:0/1735103144 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v6) ==== 118+0+0 (secure 0 0 0) 0x7f69ac07ae80 con 0x7f69b4161ed0 2024-05-28T06:04:13.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:13.211+0000 7f69b3200640 1 -- 172.21.15.96:0/1735103144 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f699406c8f0 msgr2=0x7f699406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:13.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:13.211+0000 7f69b3200640 1 --2- 172.21.15.96:0/1735103144 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f699406c8f0 0x7f699406edb0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f69a4009970 tx=0x7f69a4008040 comp rx=0 tx=0).stop 2024-05-28T06:04:13.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:13.211+0000 7f69b3200640 1 -- 172.21.15.96:0/1735103144 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f69b4161ed0 msgr2=0x7f69b4070290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:13.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:13.211+0000 7f69b3200640 1 --2- 172.21.15.96:0/1735103144 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f69b4161ed0 0x7f69b4070290 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f69ac034040 tx=0x7f69ac004a40 comp rx=0 tx=0).stop 2024-05-28T06:04:13.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:13.211+0000 7f69b3200640 1 -- 172.21.15.96:0/1735103144 shutdown_connections 2024-05-28T06:04:13.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:13.211+0000 7f69b3200640 1 --2- 172.21.15.96:0/1735103144 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f699406c8f0 0x7f699406edb0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:13.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:13.211+0000 7f69b3200640 1 --2- 172.21.15.96:0/1735103144 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f69b4161ed0 0x7f69b4070290 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:13.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:13.211+0000 7f69b3200640 1 --2- 172.21.15.96:0/1735103144 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69b40d1bb0 0x7f69b41510c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:13.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:13.211+0000 7f69b3200640 1 --2- 172.21.15.96:0/1735103144 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f69b40ced60 0x7f69b409c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:13.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:13.211+0000 7f69b3200640 1 -- 172.21.15.96:0/1735103144 >> 172.21.15.96:0/1735103144 conn(0x7f69b40b3ec0 msgr2=0x7f69b40713d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:13.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:13.211+0000 7f69b3200640 1 -- 172.21.15.96:0/1735103144 shutdown_connections 2024-05-28T06:04:13.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:13.211+0000 7f69b3200640 1 -- 172.21.15.96:0/1735103144 wait complete. 2024-05-28T06:04:13.233 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-05-28T06:04:13.233 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-05-28T06:04:13.439 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:13.438+0000 7fa0d6600640 1 Processor -- start 2024-05-28T06:04:13.439 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:13.438+0000 7fa0d6600640 1 -- start start 2024-05-28T06:04:13.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:13.440+0000 7fa0d6600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa0d015d270 0x7fa0d00b4d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:13.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:13.440+0000 7fa0d6600640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0d015d900 con 0x7fa0d0153510 2024-05-28T06:04:13.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:13.440+0000 7fa0d6600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0d00b3f90 con 0x7fa0d00cd2a0 2024-05-28T06:04:13.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:13.440+0000 7fa0d6600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fa0d01538f0 con 0x7fa0d015d270 2024-05-28T06:04:13.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:13.440+0000 7fa0d5200640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fa0d0153510 0x7fa0d00ccb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:55384/0 (socket says 172.21.15.96:55384) 2024-05-28T06:04:13.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:13.440+0000 7fa0d5200640 1 -- 172.21.15.96:0/1693934912 learned_addr learned my addr 172.21.15.96:0/1693934912 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:04:13.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:13.440+0000 7fa0d5c00640 1 --2- 172.21.15.96:0/1693934912 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa0d015d270 0x7fa0d00b4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:13.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:13.440+0000 7fa0cf400640 1 -- 172.21.15.96:0/1693934912 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1078686194 0 0) 0x7fa0d015d900 con 0x7fa0d0153510 2024-05-28T06:04:13.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:13.441+0000 7fa0cf400640 1 -- 172.21.15.96:0/1693934912 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa0b8003520 con 0x7fa0d0153510 2024-05-28T06:04:13.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:13.441+0000 7fa0cf400640 1 -- 172.21.15.96:0/1693934912 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3767841453 0 0) 0x7fa0b8003520 con 0x7fa0d0153510 2024-05-28T06:04:13.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:13.441+0000 7fa0cf400640 1 -- 172.21.15.96:0/1693934912 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa0d015d270 msgr2=0x7fa0d00b4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:13.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:13.441+0000 7fa0cf400640 1 --2- 172.21.15.96:0/1693934912 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa0d015d270 0x7fa0d00b4d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:13.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:13.441+0000 7fa0cf400640 1 -- 172.21.15.96:0/1693934912 >> v1:172.21.15.123:6789/0 conn(0x7fa0d00cd2a0 legacy=0x7fa0d015cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:13.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:13.441+0000 7fa0cf400640 1 -- 172.21.15.96:0/1693934912 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0d00b4270 con 0x7fa0d0153510 2024-05-28T06:04:13.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:13.441+0000 7fa0cf400640 1 -- 172.21.15.96:0/1693934912 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7fa0c4001670 con 0x7fa0d0153510 2024-05-28T06:04:13.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:13.441+0000 7fa0cf400640 1 -- 172.21.15.96:0/1693934912 >> v1:172.21.15.96:6789/0 conn(0x7fa0d0153510 legacy=0x7fa0d00ccb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:13.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:13.441+0000 7fa0cf400640 1 --2- 172.21.15.96:0/1693934912 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa0b80039e0 0x7fa0b8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:13.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:13.441+0000 7fa0cf400640 1 --2- 172.21.15.96:0/1693934912 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa0b8004800 0x7fa0b8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:13.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:13.441+0000 7fa0cf400640 1 --2- 172.21.15.96:0/1693934912 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa0d015d270 0x7fa0b8005270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:13.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:13.441+0000 7fa0cf400640 1 -- 172.21.15.96:0/1693934912 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa0b800bb50 con 0x7fa0b80039e0 2024-05-28T06:04:13.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:13.441+0000 7fa0cf400640 1 -- 172.21.15.96:0/1693934912 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa0b800bcd0 con 0x7fa0b8004800 2024-05-28T06:04:13.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:13.441+0000 7fa0cf400640 1 -- 172.21.15.96:0/1693934912 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fa0b800bed0 con 0x7fa0d015d270 2024-05-28T06:04:13.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:13.441+0000 7fa0d5200640 1 --2- 172.21.15.96:0/1693934912 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa0b80039e0 0x7fa0b8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:13.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:13.442+0000 7fa0d5200640 1 -- 172.21.15.96:0/1693934912 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa0d015d270 msgr2=0x7fa0b8005270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:13.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:13.442+0000 7fa0d5200640 1 --2- 172.21.15.96:0/1693934912 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa0d015d270 0x7fa0b8005270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:13.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:13.442+0000 7fa0d5200640 1 -- 172.21.15.96:0/1693934912 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa0b8004800 msgr2=0x7fa0b8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:13.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:13.442+0000 7fa0d5200640 1 --2- 172.21.15.96:0/1693934912 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa0b8004800 0x7fa0b8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:13.451 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:13.442+0000 7fa0d5200640 1 -- 172.21.15.96:0/1693934912 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0b8003520 con 0x7fa0b80039e0 2024-05-28T06:04:13.451 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:13.443+0000 7fa0d5200640 1 --2- 172.21.15.96:0/1693934912 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa0b80039e0 0x7fa0b8003e00 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa0c400c190 tx=0x7fa0b8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78b00e7cfa4feb2c server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:13.451 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:13.443+0000 7fa0cf400640 1 -- 172.21.15.96:0/1693934912 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0c4007c60 con 0x7fa0b80039e0 2024-05-28T06:04:13.451 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:13.443+0000 7fa0cf400640 1 -- 172.21.15.96:0/1693934912 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0c4005ad0 con 0x7fa0b80039e0 2024-05-28T06:04:13.451 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:13.443+0000 7fa0cf400640 1 -- 172.21.15.96:0/1693934912 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0c4007070 con 0x7fa0b80039e0 2024-05-28T06:04:13.451 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:13.443+0000 7fa0d6600640 1 -- 172.21.15.96:0/1693934912 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa0b80039e0 msgr2=0x7fa0b8003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:13.452 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:13.443+0000 7fa0d6600640 1 --2- 172.21.15.96:0/1693934912 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa0b80039e0 0x7fa0b8003e00 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa0c400c190 tx=0x7fa0b8005830 comp rx=0 tx=0).stop 2024-05-28T06:04:13.452 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:13.443+0000 7fa0d5c00640 1 -- 172.21.15.96:0/1693934912 reap_dead start 2024-05-28T06:04:13.452 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:13.444+0000 7fa0d6600640 1 -- 172.21.15.96:0/1693934912 shutdown_connections 2024-05-28T06:04:13.452 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:13.444+0000 7fa0d6600640 1 -- 172.21.15.96:0/1693934912 >> 172.21.15.96:0/1693934912 conn(0x7fa0d006b900 msgr2=0x7fa0d006bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:13.452 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:13.444+0000 7fa0d6600640 1 -- 172.21.15.96:0/1693934912 shutdown_connections 2024-05-28T06:04:13.452 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:13.444+0000 7fa0d6600640 1 -- 172.21.15.96:0/1693934912 wait complete. 2024-05-28T06:04:13.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:13.445+0000 7fa0d6600640 1 Processor -- start 2024-05-28T06:04:13.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:13.445+0000 7fa0d6600640 1 -- start start 2024-05-28T06:04:13.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:13.446+0000 7fa0d6600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa0b80039e0 0x7fa0d0149650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:13.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:13.446+0000 7fa0d6600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa0d014a860 0x7fa0d0149b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:13.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:13.446+0000 7fa0d6600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa0d014ebc0 0x7fa0d00cdfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:13.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:13.446+0000 7fa0d6600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa0d00b48c0 con 0x7fa0d014ebc0 2024-05-28T06:04:13.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:13.446+0000 7fa0d6600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa0d015d6c0 con 0x7fa0d014a860 2024-05-28T06:04:13.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:13.446+0000 7fa0d6600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fa0d00b4d50 con 0x7fa0b80039e0 2024-05-28T06:04:13.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:13.447+0000 7fa0d5c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa0d014ebc0 0x7fa0d00cdfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:13.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:13.447+0000 7fa0d5c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa0d014ebc0 0x7fa0d00cdfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:41450/0 (socket says 172.21.15.96:41450) 2024-05-28T06:04:13.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:13.447+0000 7fa0d5c00640 1 -- 172.21.15.96:0/3538015201 learned_addr learned my addr 172.21.15.96:0/3538015201 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:13.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:13.447+0000 7fa0cfe00640 1 --2- 172.21.15.96:0/3538015201 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa0d014a860 0x7fa0d0149b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:13.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:13.447+0000 7fa0d5c00640 1 -- 172.21.15.96:0/3538015201 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa0b80039e0 msgr2=0x7fa0d0149650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:13.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:13.447+0000 7fa0d5c00640 1 --2- 172.21.15.96:0/3538015201 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa0b80039e0 0x7fa0d0149650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:13.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:13.447+0000 7fa0d5c00640 1 -- 172.21.15.96:0/3538015201 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa0d014a860 msgr2=0x7fa0d0149b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:13.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:13.447+0000 7fa0d5c00640 1 --2- 172.21.15.96:0/3538015201 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa0d014a860 0x7fa0d0149b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:13.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:13.447+0000 7fa0d5c00640 1 -- 172.21.15.96:0/3538015201 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0d00ced80 con 0x7fa0d014ebc0 2024-05-28T06:04:13.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:13.447+0000 7fa0d5c00640 1 --2- 172.21.15.96:0/3538015201 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa0d014ebc0 0x7fa0d00cdfc0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa0d00afcf0 tx=0x7fa0c800b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:13.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:13.448+0000 7fa0ccc00640 1 -- 172.21.15.96:0/3538015201 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0c8018020 con 0x7fa0d014ebc0 2024-05-28T06:04:13.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:13.448+0000 7fa0ccc00640 1 -- 172.21.15.96:0/3538015201 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0c800bd00 con 0x7fa0d014ebc0 2024-05-28T06:04:13.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:13.448+0000 7fa0ccc00640 1 -- 172.21.15.96:0/3538015201 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0c8014640 con 0x7fa0d014ebc0 2024-05-28T06:04:13.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:13.448+0000 7fa0d6600640 1 -- 172.21.15.96:0/3538015201 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0d015dc20 con 0x7fa0d014ebc0 2024-05-28T06:04:13.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:13.448+0000 7fa0d6600640 1 -- 172.21.15.96:0/3538015201 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0d015e1e0 con 0x7fa0d014ebc0 2024-05-28T06:04:13.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:13.449+0000 7fa0d6600640 1 -- 172.21.15.96:0/3538015201 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0d015e3e0 con 0x7fa0d014ebc0 2024-05-28T06:04:13.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:13.450+0000 7fa0ccc00640 1 -- 172.21.15.96:0/3538015201 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa0c8005d10 con 0x7fa0d014ebc0 2024-05-28T06:04:13.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:13.451+0000 7fa0ccc00640 1 --2- 172.21.15.96:0/3538015201 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa0b006c8f0 0x7fa0b006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:13.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:13.451+0000 7fa0ccc00640 1 -- 172.21.15.96:0/3538015201 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fa0c808c1e0 con 0x7fa0d014ebc0 2024-05-28T06:04:13.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:13.451+0000 7fa0d5200640 1 --2- 172.21.15.96:0/3538015201 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa0b006c8f0 0x7fa0b006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:13.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:13.452+0000 7fa0d5200640 1 --2- 172.21.15.96:0/3538015201 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa0b006c8f0 0x7fa0b006edb0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa0c4032000 tx=0x7fa0c4007000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:13.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:13.455+0000 7fa0ccc00640 1 -- 172.21.15.96:0/3538015201 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa0d015e3e0 con 0x7fa0d014ebc0 2024-05-28T06:04:13.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:13.703+0000 7fa0d6600640 1 -- 172.21.15.96:0/3538015201 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fa0d00689e0 con 0x7fa0d014ebc0 2024-05-28T06:04:14.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.219+0000 7fa0ccc00640 1 -- 172.21.15.96:0/3538015201 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v7) ==== 122+0+0 (secure 0 0 0) 0x7fa0c8022030 con 0x7fa0d014ebc0 2024-05-28T06:04:14.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.223+0000 7fa0d6600640 1 -- 172.21.15.96:0/3538015201 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa0b006c8f0 msgr2=0x7fa0b006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:14.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.223+0000 7fa0d6600640 1 --2- 172.21.15.96:0/3538015201 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa0b006c8f0 0x7fa0b006edb0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa0c4032000 tx=0x7fa0c4007000 comp rx=0 tx=0).stop 2024-05-28T06:04:14.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.223+0000 7fa0d6600640 1 -- 172.21.15.96:0/3538015201 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa0d014ebc0 msgr2=0x7fa0d00cdfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:14.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.223+0000 7fa0d6600640 1 --2- 172.21.15.96:0/3538015201 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa0d014ebc0 0x7fa0d00cdfc0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa0d00afcf0 tx=0x7fa0c800b480 comp rx=0 tx=0).stop 2024-05-28T06:04:14.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.223+0000 7fa0d6600640 1 -- 172.21.15.96:0/3538015201 shutdown_connections 2024-05-28T06:04:14.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.223+0000 7fa0d6600640 1 --2- 172.21.15.96:0/3538015201 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa0b006c8f0 0x7fa0b006edb0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:14.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.223+0000 7fa0d6600640 1 --2- 172.21.15.96:0/3538015201 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa0d014ebc0 0x7fa0d00cdfc0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:14.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.223+0000 7fa0d6600640 1 --2- 172.21.15.96:0/3538015201 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa0d014a860 0x7fa0d0149b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:14.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.223+0000 7fa0d6600640 1 --2- 172.21.15.96:0/3538015201 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa0b80039e0 0x7fa0d0149650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:14.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.223+0000 7fa0d6600640 1 -- 172.21.15.96:0/3538015201 >> 172.21.15.96:0/3538015201 conn(0x7fa0d006b900 msgr2=0x7fa0d006f560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:14.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.223+0000 7fa0d6600640 1 -- 172.21.15.96:0/3538015201 shutdown_connections 2024-05-28T06:04:14.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.223+0000 7fa0d6600640 1 -- 172.21.15.96:0/3538015201 wait complete. 2024-05-28T06:04:14.244 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-05-28T06:04:14.244 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-28T06:04:14.439 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.439+0000 7f2969e00640 1 Processor -- start 2024-05-28T06:04:14.439 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.439+0000 7f2969e00640 1 -- start start 2024-05-28T06:04:14.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.440+0000 7f2969e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f296406f860 0x7f29640b3150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:14.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.440+0000 7f2969e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f29640b3690 0x7f29640d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:14.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.440+0000 7f2969e00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f296406ed10 con 0x7f29640d09e0 2024-05-28T06:04:14.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.440+0000 7f2969e00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f29640d0df0 con 0x7f29640b3690 2024-05-28T06:04:14.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.440+0000 7f2969e00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f296415cb00 con 0x7f296406f860 2024-05-28T06:04:14.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.440+0000 7f295a000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f29640b3690 0x7f29640d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:14.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.440+0000 7f2963400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f29640d09e0 0x7f2964160210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:55400/0 (socket says 172.21.15.96:55400) 2024-05-28T06:04:14.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.440+0000 7f2963400640 1 -- 172.21.15.96:0/414741068 learned_addr learned my addr 172.21.15.96:0/414741068 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:04:14.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.441+0000 7f2962a00640 1 --2- 172.21.15.96:0/414741068 >> v2:172.21.15.123:3300/0 conn(0x7f296406f860 0x7f29640b3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:14.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.441+0000 7f295a000640 1 -- 172.21.15.96:0/414741068 >> v2:172.21.15.123:3300/0 conn(0x7f296406f860 msgr2=0x7f29640b3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:14.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.441+0000 7f295a000640 1 --2- 172.21.15.96:0/414741068 >> v2:172.21.15.123:3300/0 conn(0x7f296406f860 0x7f29640b3150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:14.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.441+0000 7f295a000640 1 -- 172.21.15.96:0/414741068 >> v1:172.21.15.96:6789/0 conn(0x7f29640d09e0 legacy=0x7f2964160210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:14.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.441+0000 7f295a000640 1 -- 172.21.15.96:0/414741068 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29640b3cc0 con 0x7f29640b3690 2024-05-28T06:04:14.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.441+0000 7f2962000640 1 -- 172.21.15.96:0/414741068 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3679036780 0 0) 0x7f295c001490 con 0x7f29640d09e0 2024-05-28T06:04:14.444 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.442+0000 7f295a000640 1 --2- 172.21.15.96:0/414741068 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f29640b3690 0x7f29640d04a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f296406c8d0 tx=0x7f294c02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6eff55fdbed14de0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:14.444 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.442+0000 7f2962000640 1 -- 172.21.15.96:0/414741068 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f294c037070 con 0x7f29640b3690 2024-05-28T06:04:14.444 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.442+0000 7f2962000640 1 -- 172.21.15.96:0/414741068 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f294c032440 con 0x7f29640b3690 2024-05-28T06:04:14.444 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.442+0000 7f2962000640 1 -- 172.21.15.96:0/414741068 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f294c032810 con 0x7f29640b3690 2024-05-28T06:04:14.444 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.443+0000 7f2969e00640 1 -- 172.21.15.96:0/414741068 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f29640b3690 msgr2=0x7f29640d04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:14.444 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.443+0000 7f2969e00640 1 --2- 172.21.15.96:0/414741068 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f29640b3690 0x7f29640d04a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f296406c8d0 tx=0x7f294c02a1a0 comp rx=0 tx=0).stop 2024-05-28T06:04:14.444 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.443+0000 7f2969e00640 1 -- 172.21.15.96:0/414741068 shutdown_connections 2024-05-28T06:04:14.444 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.443+0000 7f2969e00640 1 --2- 172.21.15.96:0/414741068 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f29640b3690 0x7f29640d04a0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:14.444 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.443+0000 7f2969e00640 1 --2- 172.21.15.96:0/414741068 >> v2:172.21.15.123:3300/0 conn(0x7f296406f860 0x7f29640b3150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:14.444 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.443+0000 7f2969e00640 1 -- 172.21.15.96:0/414741068 >> 172.21.15.96:0/414741068 conn(0x7f296406bae0 msgr2=0x7f296406bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:14.444 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.443+0000 7f2969e00640 1 -- 172.21.15.96:0/414741068 shutdown_connections 2024-05-28T06:04:14.444 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.443+0000 7f2969e00640 1 -- 172.21.15.96:0/414741068 wait complete. 2024-05-28T06:04:14.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.445+0000 7f2969e00640 1 Processor -- start 2024-05-28T06:04:14.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.445+0000 7f2969e00640 1 -- start start 2024-05-28T06:04:14.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.446+0000 7f2969e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f296406f860 0x7f296415f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:14.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.446+0000 7f2969e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f29640b3690 0x7f29640999f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:14.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.446+0000 7f2969e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29640d09e0 0x7f29640c4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:14.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.446+0000 7f2969e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f29640b4560 con 0x7f296406f860 2024-05-28T06:04:14.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.446+0000 7f2969e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f29640b4280 con 0x7f29640d09e0 2024-05-28T06:04:14.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.446+0000 7f2969e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f29640b4840 con 0x7f29640b3690 2024-05-28T06:04:14.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.446+0000 7f2962a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f296406f860 0x7f296415f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:14.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.446+0000 7f2962a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f296406f860 0x7f296415f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:41468/0 (socket says 172.21.15.96:41468) 2024-05-28T06:04:14.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.446+0000 7f2962a00640 1 -- 172.21.15.96:0/3822564397 learned_addr learned my addr 172.21.15.96:0/3822564397 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:14.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.446+0000 7f2962a00640 1 -- 172.21.15.96:0/3822564397 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f29640b3690 msgr2=0x7f29640999f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:14.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.446+0000 7f2962a00640 1 --2- 172.21.15.96:0/3822564397 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f29640b3690 0x7f29640999f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:14.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.446+0000 7f2962a00640 1 -- 172.21.15.96:0/3822564397 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29640d09e0 msgr2=0x7f29640c4cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:14.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.446+0000 7f2962a00640 1 --2- 172.21.15.96:0/3822564397 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29640d09e0 0x7f29640c4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:14.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.446+0000 7f2962a00640 1 -- 172.21.15.96:0/3822564397 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29640c5430 con 0x7f296406f860 2024-05-28T06:04:14.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.447+0000 7f2962a00640 1 --2- 172.21.15.96:0/3822564397 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f296406f860 0x7f296415f710 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f295400d620 tx=0x7f295400dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:14.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.447+0000 7f295be00640 1 -- 172.21.15.96:0/3822564397 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f295400b620 con 0x7f296406f860 2024-05-28T06:04:14.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.447+0000 7f295be00640 1 -- 172.21.15.96:0/3822564397 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2954019070 con 0x7f296406f860 2024-05-28T06:04:14.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.447+0000 7f295be00640 1 -- 172.21.15.96:0/3822564397 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f295400ba00 con 0x7f296406f860 2024-05-28T06:04:14.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.447+0000 7f2969e00640 1 -- 172.21.15.96:0/3822564397 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29640c5710 con 0x7f296406f860 2024-05-28T06:04:14.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.447+0000 7f2969e00640 1 -- 172.21.15.96:0/3822564397 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29640c5cd0 con 0x7f296406f860 2024-05-28T06:04:14.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.448+0000 7f2969e00640 1 -- 172.21.15.96:0/3822564397 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29640689e0 con 0x7f296406f860 2024-05-28T06:04:14.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.449+0000 7f295be00640 1 -- 172.21.15.96:0/3822564397 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f295400bba0 con 0x7f296406f860 2024-05-28T06:04:14.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.450+0000 7f295be00640 1 --2- 172.21.15.96:0/3822564397 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f294006c8f0 0x7f294006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:14.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.450+0000 7f295be00640 1 -- 172.21.15.96:0/3822564397 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f295408c020 con 0x7f296406f860 2024-05-28T06:04:14.451 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.451+0000 7f295a000640 1 --2- 172.21.15.96:0/3822564397 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f294006c8f0 0x7f294006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:14.452 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.452+0000 7f295a000640 1 --2- 172.21.15.96:0/3822564397 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f294006c8f0 0x7f294006edb0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f294c034000 tx=0x7f294c02c000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:14.454 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.454+0000 7f295be00640 1 -- 172.21.15.96:0/3822564397 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f295405a3b0 con 0x7f296406f860 2024-05-28T06:04:14.698 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.698+0000 7f2969e00640 1 -- 172.21.15.96:0/3822564397 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f296406e2f0 con 0x7f296406f860 2024-05-28T06:04:14.699 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.699+0000 7f295be00640 1 -- 172.21.15.96:0/3822564397 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5178 (secure 0 0 0) 0x7f2954059cb0 con 0x7f296406f860 2024-05-28T06:04:14.700 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:04:14.700 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/3349661842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":3349661842},{"type":"v1","addr":"172.21.15.96:6833","nonce":3349661842}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4270,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/3383264102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":3383264102},{"type":"v1","addr":"172.21.15.96:6835","nonce":3383264102}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4279,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/3068402544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":3068402544},{"type":"v1","addr":"172.21.15.123:6837","nonce":3068402544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-28T06:04:10.319738+0000","modified":"2024-05-28T06:04:13.704080+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4281},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4281":{"gid":4281,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.123:6835/605814935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":605814935},{"type":"v1","addr":"172.21.15.123:6835","nonce":605814935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4281,"qdb_cluster":[4281]},"id":1}]} 2024-05-28T06:04:14.700 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 7 2024-05-28T06:04:14.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.702+0000 7f2969e00640 1 -- 172.21.15.96:0/3822564397 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f294006c8f0 msgr2=0x7f294006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:14.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.702+0000 7f2969e00640 1 --2- 172.21.15.96:0/3822564397 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f294006c8f0 0x7f294006edb0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f294c034000 tx=0x7f294c02c000 comp rx=0 tx=0).stop 2024-05-28T06:04:14.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.702+0000 7f2969e00640 1 -- 172.21.15.96:0/3822564397 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f296406f860 msgr2=0x7f296415f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:14.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.702+0000 7f2969e00640 1 --2- 172.21.15.96:0/3822564397 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f296406f860 0x7f296415f710 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f295400d620 tx=0x7f295400dae0 comp rx=0 tx=0).stop 2024-05-28T06:04:14.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.702+0000 7f2969e00640 1 -- 172.21.15.96:0/3822564397 shutdown_connections 2024-05-28T06:04:14.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.702+0000 7f2969e00640 1 --2- 172.21.15.96:0/3822564397 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f294006c8f0 0x7f294006edb0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:14.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.702+0000 7f2969e00640 1 --2- 172.21.15.96:0/3822564397 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29640d09e0 0x7f29640c4cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:14.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.702+0000 7f2969e00640 1 --2- 172.21.15.96:0/3822564397 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f29640b3690 0x7f29640999f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:14.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.702+0000 7f2969e00640 1 --2- 172.21.15.96:0/3822564397 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f296406f860 0x7f296415f710 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:14.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.702+0000 7f2969e00640 1 -- 172.21.15.96:0/3822564397 >> 172.21.15.96:0/3822564397 conn(0x7f296406bae0 msgr2=0x7f29640adfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:14.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.702+0000 7f2969e00640 1 -- 172.21.15.96:0/3822564397 shutdown_connections 2024-05-28T06:04:14.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.702+0000 7f2969e00640 1 -- 172.21.15.96:0/3822564397 wait complete. 2024-05-28T06:04:14.721 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:04:14.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.920+0000 7f6cf4a00640 1 Processor -- start 2024-05-28T06:04:14.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.920+0000 7f6cf4a00640 1 -- start start 2024-05-28T06:04:14.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.921+0000 7f6cf4a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f6cf015b3a0 0x7f6cf00b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:14.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.921+0000 7f6cf4a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6cf006dc20 0x7f6cf015f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:14.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.921+0000 7f6cf4a00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f6cf0072c00 0x7f6cf0073050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:14.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.921+0000 7f6cf4a00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f6cf006f980 con 0x7f6cf015b3a0 2024-05-28T06:04:14.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.921+0000 7f6cf4a00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f6cf0073590 con 0x7f6cf0072c00 2024-05-28T06:04:14.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.921+0000 7f6cf4a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f6cf00ae0c0 con 0x7f6cf006dc20 2024-05-28T06:04:14.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.922+0000 7f6cef400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f6cf015b3a0 0x7f6cf00b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:14.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.922+0000 7f6cef400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f6cf015b3a0 0x7f6cf00b32c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:41476/0 (socket says 172.21.15.96:41476) 2024-05-28T06:04:14.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.922+0000 7f6cef400640 1 -- 172.21.15.96:0/2434589650 learned_addr learned my addr 172.21.15.96:0/2434589650 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:14.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.922+0000 7f6cefe00640 1 --2- 172.21.15.96:0/2434589650 >> v2:172.21.15.123:3300/0 conn(0x7f6cf0072c00 0x7f6cf0073050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:14.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.922+0000 7f6cef400640 1 -- 172.21.15.96:0/2434589650 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6cf006dc20 msgr2=0x7f6cf015f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:14.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.922+0000 7f6cef400640 1 --2- 172.21.15.96:0/2434589650 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6cf006dc20 0x7f6cf015f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:14.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.922+0000 7f6cef400640 1 -- 172.21.15.96:0/2434589650 >> v2:172.21.15.123:3300/0 conn(0x7f6cf0072c00 msgr2=0x7f6cf0073050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:14.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.922+0000 7f6cef400640 1 --2- 172.21.15.96:0/2434589650 >> v2:172.21.15.123:3300/0 conn(0x7f6cf0072c00 0x7f6cf0073050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:14.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.922+0000 7f6cef400640 1 -- 172.21.15.96:0/2434589650 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cf00afd60 con 0x7f6cf015b3a0 2024-05-28T06:04:14.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.923+0000 7f6cef400640 1 --2- 172.21.15.96:0/2434589650 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6cf015b3a0 0x7f6cf00b32c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f6cf006fd40 tx=0x7f6ce002a1d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0fe949ecfa6c3a5 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:14.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.923+0000 7f6ceea00640 1 -- 172.21.15.96:0/2434589650 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ce0037070 con 0x7f6cf015b3a0 2024-05-28T06:04:14.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.923+0000 7f6ceea00640 1 -- 172.21.15.96:0/2434589650 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ce0032440 con 0x7f6cf015b3a0 2024-05-28T06:04:14.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.923+0000 7f6ceea00640 1 -- 172.21.15.96:0/2434589650 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ce0032810 con 0x7f6cf015b3a0 2024-05-28T06:04:14.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.924+0000 7f6cf4a00640 1 -- 172.21.15.96:0/2434589650 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6cf015b3a0 msgr2=0x7f6cf00b32c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:14.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.924+0000 7f6cf4a00640 1 --2- 172.21.15.96:0/2434589650 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6cf015b3a0 0x7f6cf00b32c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f6cf006fd40 tx=0x7f6ce002a1d0 comp rx=0 tx=0).stop 2024-05-28T06:04:14.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.924+0000 7f6cf4a00640 1 -- 172.21.15.96:0/2434589650 shutdown_connections 2024-05-28T06:04:14.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.924+0000 7f6cf4a00640 1 --2- 172.21.15.96:0/2434589650 >> v2:172.21.15.123:3300/0 conn(0x7f6cf0072c00 0x7f6cf0073050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:14.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.924+0000 7f6cf4a00640 1 --2- 172.21.15.96:0/2434589650 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6cf006dc20 0x7f6cf015f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:14.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.924+0000 7f6cf4a00640 1 --2- 172.21.15.96:0/2434589650 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6cf015b3a0 0x7f6cf00b32c0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:14.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.924+0000 7f6cf4a00640 1 -- 172.21.15.96:0/2434589650 >> 172.21.15.96:0/2434589650 conn(0x7f6cf00b1fc0 msgr2=0x7f6cf00b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:14.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.924+0000 7f6cf4a00640 1 -- 172.21.15.96:0/2434589650 shutdown_connections 2024-05-28T06:04:14.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.924+0000 7f6cf4a00640 1 -- 172.21.15.96:0/2434589650 wait complete. 2024-05-28T06:04:14.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.926+0000 7f6cf4a00640 1 Processor -- start 2024-05-28T06:04:14.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.926+0000 7f6cf4a00640 1 -- start start 2024-05-28T06:04:14.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.927+0000 7f6cf4a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6cf006dc20 0x7f6cf009dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:14.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.927+0000 7f6cf4a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6cf0072c00 0x7f6cf0149230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:14.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.927+0000 7f6cf4a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6cf015b3a0 0x7f6cf0149770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:14.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.927+0000 7f6cf4a00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f6cf00afa80 con 0x7f6cf015b3a0 2024-05-28T06:04:14.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.927+0000 7f6cf4a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f6cf015fee0 con 0x7f6cf0072c00 2024-05-28T06:04:14.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.927+0000 7f6cf4a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f6cf015b780 con 0x7f6cf006dc20 2024-05-28T06:04:14.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.927+0000 7f6cefe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6cf015b3a0 0x7f6cf0149770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:14.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.928+0000 7f6cefe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6cf015b3a0 0x7f6cf0149770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:41492/0 (socket says 172.21.15.96:41492) 2024-05-28T06:04:14.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.928+0000 7f6cefe00640 1 -- 172.21.15.96:0/2141525450 learned_addr learned my addr 172.21.15.96:0/2141525450 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:14.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.928+0000 7f6ce7e00640 1 --2- 172.21.15.96:0/2141525450 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6cf0072c00 0x7f6cf0149230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:14.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.928+0000 7f6cefe00640 1 -- 172.21.15.96:0/2141525450 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6cf006dc20 msgr2=0x7f6cf009dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:14.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.928+0000 7f6cefe00640 1 --2- 172.21.15.96:0/2141525450 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6cf006dc20 0x7f6cf009dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:14.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.928+0000 7f6cefe00640 1 -- 172.21.15.96:0/2141525450 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6cf0072c00 msgr2=0x7f6cf0149230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:14.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.928+0000 7f6cefe00640 1 --2- 172.21.15.96:0/2141525450 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6cf0072c00 0x7f6cf0149230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:14.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.928+0000 7f6cefe00640 1 -- 172.21.15.96:0/2141525450 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cf0149cb0 con 0x7f6cf015b3a0 2024-05-28T06:04:14.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.928+0000 7f6ce7e00640 1 --2- 172.21.15.96:0/2141525450 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6cf0072c00 0x7f6cf0149230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:04:14.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.928+0000 7f6cefe00640 1 --2- 172.21.15.96:0/2141525450 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6cf015b3a0 0x7f6cf0149770 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f6ce800cc80 tx=0x7f6ce800a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:14.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.928+0000 7f6ce7400640 1 -- 172.21.15.96:0/2141525450 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ce800ce30 con 0x7f6cf015b3a0 2024-05-28T06:04:14.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.928+0000 7f6ce7400640 1 -- 172.21.15.96:0/2141525450 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ce8003990 con 0x7f6cf015b3a0 2024-05-28T06:04:14.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.928+0000 7f6ce7400640 1 -- 172.21.15.96:0/2141525450 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ce8003d60 con 0x7f6cf015b3a0 2024-05-28T06:04:14.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.929+0000 7f6cf4a00640 1 -- 172.21.15.96:0/2141525450 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cf015a440 con 0x7f6cf015b3a0 2024-05-28T06:04:14.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.929+0000 7f6cf4a00640 1 -- 172.21.15.96:0/2141525450 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cf014a700 con 0x7f6cf015b3a0 2024-05-28T06:04:14.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.930+0000 7f6cf4a00640 1 -- 172.21.15.96:0/2141525450 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6cf00689e0 con 0x7f6cf015b3a0 2024-05-28T06:04:14.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.931+0000 7f6ce7400640 1 -- 172.21.15.96:0/2141525450 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6ce8003f00 con 0x7f6cf015b3a0 2024-05-28T06:04:14.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.932+0000 7f6ce7400640 1 --2- 172.21.15.96:0/2141525450 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6ccc06c8f0 0x7f6ccc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:14.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.932+0000 7f6ce7400640 1 -- 172.21.15.96:0/2141525450 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f6ce808bf40 con 0x7f6cf015b3a0 2024-05-28T06:04:14.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.932+0000 7f6cef400640 1 --2- 172.21.15.96:0/2141525450 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6ccc06c8f0 0x7f6ccc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:14.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.933+0000 7f6cef400640 1 --2- 172.21.15.96:0/2141525450 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6ccc06c8f0 0x7f6ccc06edb0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6ce0034000 tx=0x7f6ce002c000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:14.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:14.936+0000 7f6ce7400640 1 -- 172.21.15.96:0/2141525450 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6ce8014030 con 0x7f6cf015b3a0 2024-05-28T06:04:15.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.136+0000 7f6cf4a00640 1 -- 172.21.15.96:0/2141525450 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6cf006e4a0 con 0x7f6cf015b3a0 2024-05-28T06:04:15.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.138+0000 7f6ce7400640 1 -- 172.21.15.96:0/2141525450 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16617 (secure 0 0 0) 0x7f6ce805a360 con 0x7f6cf015b3a0 2024-05-28T06:04:15.139 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:04:15.139 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":20,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:04:11.882922+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-28T06:04:08.741938+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-28T06:04:09.301470+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:04:15.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.141+0000 7f6cf4a00640 1 -- 172.21.15.96:0/2141525450 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6ccc06c8f0 msgr2=0x7f6ccc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:15.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.141+0000 7f6cf4a00640 1 --2- 172.21.15.96:0/2141525450 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6ccc06c8f0 0x7f6ccc06edb0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6ce0034000 tx=0x7f6ce002c000 comp rx=0 tx=0).stop 2024-05-28T06:04:15.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.141+0000 7f6cf4a00640 1 -- 172.21.15.96:0/2141525450 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6cf015b3a0 msgr2=0x7f6cf0149770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:15.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.141+0000 7f6cf4a00640 1 --2- 172.21.15.96:0/2141525450 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6cf015b3a0 0x7f6cf0149770 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f6ce800cc80 tx=0x7f6ce800a5c0 comp rx=0 tx=0).stop 2024-05-28T06:04:15.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.141+0000 7f6cf4a00640 1 -- 172.21.15.96:0/2141525450 shutdown_connections 2024-05-28T06:04:15.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.141+0000 7f6cf4a00640 1 --2- 172.21.15.96:0/2141525450 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6ccc06c8f0 0x7f6ccc06edb0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:15.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.141+0000 7f6cf4a00640 1 --2- 172.21.15.96:0/2141525450 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6cf015b3a0 0x7f6cf0149770 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:15.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.141+0000 7f6cf4a00640 1 --2- 172.21.15.96:0/2141525450 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6cf0072c00 0x7f6cf0149230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:15.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.141+0000 7f6cf4a00640 1 --2- 172.21.15.96:0/2141525450 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6cf006dc20 0x7f6cf009dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:15.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.141+0000 7f6cf4a00640 1 -- 172.21.15.96:0/2141525450 >> 172.21.15.96:0/2141525450 conn(0x7f6cf00b1fc0 msgr2=0x7f6cf006e9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:15.143 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.141+0000 7f6cf4a00640 1 -- 172.21.15.96:0/2141525450 shutdown_connections 2024-05-28T06:04:15.143 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.141+0000 7f6cf4a00640 1 -- 172.21.15.96:0/2141525450 wait complete. 2024-05-28T06:04:15.162 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-05-28T06:04:15.163 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-28T06:04:15.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.358+0000 7f428cc00640 1 Processor -- start 2024-05-28T06:04:15.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.358+0000 7f428cc00640 1 -- start start 2024-05-28T06:04:15.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.360+0000 7f428cc00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f42880b4e10 0x7f42880cde30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:15.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.360+0000 7f428cc00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f42880ce370 0x7f42880d0590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:15.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.360+0000 7f428cc00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42880b51f0 con 0x7f42880d0ad0 2024-05-28T06:04:15.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.360+0000 7f428cc00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f42880d0f00 con 0x7f42880b4e10 2024-05-28T06:04:15.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.360+0000 7f428cc00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f42880aef60 con 0x7f42880ce370 2024-05-28T06:04:15.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.360+0000 7f4285e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f42880b4e10 0x7f42880cde30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:15.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.360+0000 7f4285e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f42880b4e10 0x7f42880cde30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:41498/0 (socket says 172.21.15.96:41498) 2024-05-28T06:04:15.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.360+0000 7f4285e00640 1 -- 172.21.15.96:0/1904072023 learned_addr learned my addr 172.21.15.96:0/1904072023 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:15.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.360+0000 7f4285400640 1 --2- 172.21.15.96:0/1904072023 >> v2:172.21.15.123:3300/0 conn(0x7f42880ce370 0x7f42880d0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:15.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.360+0000 7f4285e00640 1 -- 172.21.15.96:0/1904072023 >> v2:172.21.15.123:3300/0 conn(0x7f42880ce370 msgr2=0x7f42880d0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:15.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.360+0000 7f4285e00640 1 --2- 172.21.15.96:0/1904072023 >> v2:172.21.15.123:3300/0 conn(0x7f42880ce370 0x7f42880d0590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:15.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.360+0000 7f4285e00640 1 -- 172.21.15.96:0/1904072023 >> v1:172.21.15.123:6789/0 conn(0x7f42880d0ad0 legacy=0x7f4288099090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:15.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.360+0000 7f4285e00640 1 -- 172.21.15.96:0/1904072023 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42880ad1d0 con 0x7f42880b4e10 2024-05-28T06:04:15.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.361+0000 7f4285e00640 1 --2- 172.21.15.96:0/1904072023 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f42880b4e10 0x7f42880cde30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f42880ca860 tx=0x7f427802a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e514c73159f36447 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:15.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.362+0000 7f4284a00640 1 -- 172.21.15.96:0/1904072023 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4278036070 con 0x7f42880b4e10 2024-05-28T06:04:15.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.362+0000 7f4284a00640 1 -- 172.21.15.96:0/1904072023 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4278031600 con 0x7f42880b4e10 2024-05-28T06:04:15.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.362+0000 7f4284a00640 1 -- 172.21.15.96:0/1904072023 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42780319d0 con 0x7f42880b4e10 2024-05-28T06:04:15.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.362+0000 7f428cc00640 1 -- 172.21.15.96:0/1904072023 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f42880b4e10 msgr2=0x7f42880cde30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:15.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.362+0000 7f428cc00640 1 --2- 172.21.15.96:0/1904072023 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f42880b4e10 0x7f42880cde30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f42880ca860 tx=0x7f427802a660 comp rx=0 tx=0).stop 2024-05-28T06:04:15.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.362+0000 7f428cc00640 1 -- 172.21.15.96:0/1904072023 shutdown_connections 2024-05-28T06:04:15.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.362+0000 7f428cc00640 1 --2- 172.21.15.96:0/1904072023 >> v2:172.21.15.123:3300/0 conn(0x7f42880ce370 0x7f42880d0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:15.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.362+0000 7f428cc00640 1 --2- 172.21.15.96:0/1904072023 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f42880b4e10 0x7f42880cde30 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:15.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.362+0000 7f428cc00640 1 -- 172.21.15.96:0/1904072023 >> 172.21.15.96:0/1904072023 conn(0x7f428806bae0 msgr2=0x7f428806bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:15.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.363+0000 7f428cc00640 1 -- 172.21.15.96:0/1904072023 shutdown_connections 2024-05-28T06:04:15.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.363+0000 7f428cc00640 1 -- 172.21.15.96:0/1904072023 wait complete. 2024-05-28T06:04:15.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.364+0000 7f428cc00640 1 Processor -- start 2024-05-28T06:04:15.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.364+0000 7f428cc00640 1 -- start start 2024-05-28T06:04:15.367 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.365+0000 7f428cc00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f42880b4e10 0x7f4288099a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:15.367 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.365+0000 7f428cc00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f42880ce370 0x7f42880c8fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:15.367 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.365+0000 7f428cc00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42880d0ad0 0x7f42880c9ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:15.367 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.365+0000 7f428cc00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f42880ce880 con 0x7f42880ce370 2024-05-28T06:04:15.367 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.365+0000 7f428cc00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f42880ad750 con 0x7f42880d0ad0 2024-05-28T06:04:15.367 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.365+0000 7f428cc00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f42880cea00 con 0x7f42880b4e10 2024-05-28T06:04:15.367 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.365+0000 7f4285400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f42880ce370 0x7f42880c8fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:15.367 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.366+0000 7f4285400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f42880ce370 0x7f42880c8fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:41500/0 (socket says 172.21.15.96:41500) 2024-05-28T06:04:15.367 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.366+0000 7f4285400640 1 -- 172.21.15.96:0/1810305546 learned_addr learned my addr 172.21.15.96:0/1810305546 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:15.368 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.366+0000 7f4285e00640 1 --2- 172.21.15.96:0/1810305546 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f42880b4e10 0x7f4288099a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:15.368 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.366+0000 7f4286800640 1 --2- 172.21.15.96:0/1810305546 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42880d0ad0 0x7f42880c9ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:15.368 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.366+0000 7f4285400640 1 -- 172.21.15.96:0/1810305546 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f42880b4e10 msgr2=0x7f4288099a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:15.368 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.366+0000 7f4285400640 1 --2- 172.21.15.96:0/1810305546 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f42880b4e10 0x7f4288099a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:15.368 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.366+0000 7f4285400640 1 -- 172.21.15.96:0/1810305546 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42880d0ad0 msgr2=0x7f42880c9ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:15.368 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.366+0000 7f4285400640 1 --2- 172.21.15.96:0/1810305546 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42880d0ad0 0x7f42880c9ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:15.368 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.366+0000 7f4285400640 1 -- 172.21.15.96:0/1810305546 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42880c7f70 con 0x7f42880ce370 2024-05-28T06:04:15.368 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.366+0000 7f4286800640 1 --2- 172.21.15.96:0/1810305546 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42880d0ad0 0x7f42880c9ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:04:15.368 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.366+0000 7f4285400640 1 --2- 172.21.15.96:0/1810305546 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f42880ce370 0x7f42880c8fd0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f427400c7f0 tx=0x7f427400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:15.368 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.367+0000 7f427e000640 1 -- 172.21.15.96:0/1810305546 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4274015070 con 0x7f42880ce370 2024-05-28T06:04:15.368 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.367+0000 7f427e000640 1 -- 172.21.15.96:0/1810305546 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4274004240 con 0x7f42880ce370 2024-05-28T06:04:15.368 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.367+0000 7f427e000640 1 -- 172.21.15.96:0/1810305546 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4274007620 con 0x7f42880ce370 2024-05-28T06:04:15.368 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.367+0000 7f428cc00640 1 -- 172.21.15.96:0/1810305546 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42880c8210 con 0x7f42880ce370 2024-05-28T06:04:15.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.367+0000 7f428cc00640 1 -- 172.21.15.96:0/1810305546 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42880c87d0 con 0x7f42880ce370 2024-05-28T06:04:15.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.368+0000 7f428cc00640 1 -- 172.21.15.96:0/1810305546 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42880689e0 con 0x7f42880ce370 2024-05-28T06:04:15.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.369+0000 7f427e000640 1 -- 172.21.15.96:0/1810305546 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f42740079d0 con 0x7f42880ce370 2024-05-28T06:04:15.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.370+0000 7f427e000640 1 --2- 172.21.15.96:0/1810305546 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f426806c8f0 0x7f426806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:15.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.370+0000 7f427e000640 1 -- 172.21.15.96:0/1810305546 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f427408bb90 con 0x7f42880ce370 2024-05-28T06:04:15.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.370+0000 7f4285e00640 1 --2- 172.21.15.96:0/1810305546 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f426806c8f0 0x7f426806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:15.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.372+0000 7f4285e00640 1 --2- 172.21.15.96:0/1810305546 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f426806c8f0 0x7f426806edb0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4278033000 tx=0x7f4278009000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:15.375 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.374+0000 7f427e000640 1 -- 172.21.15.96:0/1810305546 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4274059ff0 con 0x7f42880ce370 2024-05-28T06:04:15.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.570+0000 7f428cc00640 1 -- 172.21.15.96:0/1810305546 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f428815e990 con 0x7f426806c8f0 2024-05-28T06:04:15.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.592+0000 7f427e000640 1 -- 172.21.15.96:0/1810305546 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251948 (secure 0 0 0) 0x7f428815e990 con 0x7f426806c8f0 2024-05-28T06:04:15.595 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:04:15.598 INFO:teuthology.orchestra.run.smithi096.stderr:dumped all 2024-05-28T06:04:15.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.599+0000 7f428cc00640 1 -- 172.21.15.96:0/1810305546 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f426806c8f0 msgr2=0x7f426806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:15.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.599+0000 7f428cc00640 1 --2- 172.21.15.96:0/1810305546 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f426806c8f0 0x7f426806edb0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4278033000 tx=0x7f4278009000 comp rx=0 tx=0).stop 2024-05-28T06:04:15.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.599+0000 7f428cc00640 1 -- 172.21.15.96:0/1810305546 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f42880ce370 msgr2=0x7f42880c8fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:15.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.599+0000 7f428cc00640 1 --2- 172.21.15.96:0/1810305546 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f42880ce370 0x7f42880c8fd0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f427400c7f0 tx=0x7f427400ccb0 comp rx=0 tx=0).stop 2024-05-28T06:04:15.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.599+0000 7f428cc00640 1 -- 172.21.15.96:0/1810305546 shutdown_connections 2024-05-28T06:04:15.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.599+0000 7f428cc00640 1 --2- 172.21.15.96:0/1810305546 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f426806c8f0 0x7f426806edb0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:15.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.599+0000 7f428cc00640 1 --2- 172.21.15.96:0/1810305546 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42880d0ad0 0x7f42880c9ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:15.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.599+0000 7f428cc00640 1 --2- 172.21.15.96:0/1810305546 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f42880ce370 0x7f42880c8fd0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:15.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.599+0000 7f428cc00640 1 --2- 172.21.15.96:0/1810305546 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f42880b4e10 0x7f4288099a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:15.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.599+0000 7f428cc00640 1 -- 172.21.15.96:0/1810305546 >> 172.21.15.96:0/1810305546 conn(0x7f428806bae0 msgr2=0x7f42880cba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:15.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.599+0000 7f428cc00640 1 -- 172.21.15.96:0/1810305546 shutdown_connections 2024-05-28T06:04:15.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.599+0000 7f428cc00640 1 -- 172.21.15.96:0/1810305546 wait complete. 2024-05-28T06:04:15.620 INFO:teuthology.orchestra.run.smithi096.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-05-28T06:04:15.483643+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":185,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218992,"kb_used_data":2824,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755728,"statfs":{"total":773094113280,"available":772869865472,"internally_reserved":0,"allocated":2891776,"data_stored":1753028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001905"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.997911+0000","last_change":"2024-05-28T06:04:10.916483+0000","last_active":"2024-05-28T06:04:12.997911+0000","last_peered":"2024-05-28T06:04:12.997911+0000","last_clean":"2024-05-28T06:04:12.997911+0000","last_became_active":"2024-05-28T06:04:10.916214+0000","last_became_peered":"2024-05-28T06:04:10.916214+0000","last_unstale":"2024-05-28T06:04:12.997911+0000","last_undegraded":"2024-05-28T06:04:12.997911+0000","last_fullsized":"2024-05-28T06:04:12.997911+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:43:05.061767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.889968+0000","last_change":"2024-05-28T06:04:10.900791+0000","last_active":"2024-05-28T06:04:11.889968+0000","last_peered":"2024-05-28T06:04:11.889968+0000","last_clean":"2024-05-28T06:04:11.889968+0000","last_became_active":"2024-05-28T06:04:10.900489+0000","last_became_peered":"2024-05-28T06:04:10.900489+0000","last_unstale":"2024-05-28T06:04:11.889968+0000","last_undegraded":"2024-05-28T06:04:11.889968+0000","last_fullsized":"2024-05-28T06:04:11.889968+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:53:07.733019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.794222+0000","last_change":"2024-05-28T06:04:10.901559+0000","last_active":"2024-05-28T06:04:12.794222+0000","last_peered":"2024-05-28T06:04:12.794222+0000","last_clean":"2024-05-28T06:04:12.794222+0000","last_became_active":"2024-05-28T06:04:10.899022+0000","last_became_peered":"2024-05-28T06:04:10.899022+0000","last_unstale":"2024-05-28T06:04:12.794222+0000","last_undegraded":"2024-05-28T06:04:12.794222+0000","last_fullsized":"2024-05-28T06:04:12.794222+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:28:21.123299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.997115+0000","last_change":"2024-05-28T06:04:10.923988+0000","last_active":"2024-05-28T06:04:12.997115+0000","last_peered":"2024-05-28T06:04:12.997115+0000","last_clean":"2024-05-28T06:04:12.997115+0000","last_became_active":"2024-05-28T06:04:10.923718+0000","last_became_peered":"2024-05-28T06:04:10.923718+0000","last_unstale":"2024-05-28T06:04:12.997115+0000","last_undegraded":"2024-05-28T06:04:12.997115+0000","last_fullsized":"2024-05-28T06:04:12.997115+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:45:47.884578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.996357+0000","last_change":"2024-05-28T06:04:10.917922+0000","last_active":"2024-05-28T06:04:12.996357+0000","last_peered":"2024-05-28T06:04:12.996357+0000","last_clean":"2024-05-28T06:04:12.996357+0000","last_became_active":"2024-05-28T06:04:10.917253+0000","last_became_peered":"2024-05-28T06:04:10.917253+0000","last_unstale":"2024-05-28T06:04:12.996357+0000","last_undegraded":"2024-05-28T06:04:12.996357+0000","last_fullsized":"2024-05-28T06:04:12.996357+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:04:49.415234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.996757+0000","last_change":"2024-05-28T06:04:10.907704+0000","last_active":"2024-05-28T06:04:12.996757+0000","last_peered":"2024-05-28T06:04:12.996757+0000","last_clean":"2024-05-28T06:04:12.996757+0000","last_became_active":"2024-05-28T06:04:10.907065+0000","last_became_peered":"2024-05-28T06:04:10.907065+0000","last_unstale":"2024-05-28T06:04:12.996757+0000","last_undegraded":"2024-05-28T06:04:12.996757+0000","last_fullsized":"2024-05-28T06:04:12.996757+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:28:21.123299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.995527+0000","last_change":"2024-05-28T06:04:10.905438+0000","last_active":"2024-05-28T06:04:12.995527+0000","last_peered":"2024-05-28T06:04:12.995527+0000","last_clean":"2024-05-28T06:04:12.995527+0000","last_became_active":"2024-05-28T06:04:10.904842+0000","last_became_peered":"2024-05-28T06:04:10.904842+0000","last_unstale":"2024-05-28T06:04:12.995527+0000","last_undegraded":"2024-05-28T06:04:12.995527+0000","last_fullsized":"2024-05-28T06:04:12.995527+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:46:27.923407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.995330+0000","last_change":"2024-05-28T06:04:10.899401+0000","last_active":"2024-05-28T06:04:12.995330+0000","last_peered":"2024-05-28T06:04:12.995330+0000","last_clean":"2024-05-28T06:04:12.995330+0000","last_became_active":"2024-05-28T06:04:10.898742+0000","last_became_peered":"2024-05-28T06:04:10.898742+0000","last_unstale":"2024-05-28T06:04:12.995330+0000","last_undegraded":"2024-05-28T06:04:12.995330+0000","last_fullsized":"2024-05-28T06:04:12.995330+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:39:46.389531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.997773+0000","last_change":"2024-05-28T06:04:10.907973+0000","last_active":"2024-05-28T06:04:12.997773+0000","last_peered":"2024-05-28T06:04:12.997773+0000","last_clean":"2024-05-28T06:04:12.997773+0000","last_became_active":"2024-05-28T06:04:10.905186+0000","last_became_peered":"2024-05-28T06:04:10.905186+0000","last_unstale":"2024-05-28T06:04:12.997773+0000","last_undegraded":"2024-05-28T06:04:12.997773+0000","last_fullsized":"2024-05-28T06:04:12.997773+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:39:13.976770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.909549+0000","last_change":"2024-05-28T06:04:10.927703+0000","last_active":"2024-05-28T06:04:11.909549+0000","last_peered":"2024-05-28T06:04:11.909549+0000","last_clean":"2024-05-28T06:04:11.909549+0000","last_became_active":"2024-05-28T06:04:10.927383+0000","last_became_peered":"2024-05-28T06:04:10.927383+0000","last_unstale":"2024-05-28T06:04:11.909549+0000","last_undegraded":"2024-05-28T06:04:11.909549+0000","last_fullsized":"2024-05-28T06:04:11.909549+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:11:51.651640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.909515+0000","last_change":"2024-05-28T06:04:10.926573+0000","last_active":"2024-05-28T06:04:11.909515+0000","last_peered":"2024-05-28T06:04:11.909515+0000","last_clean":"2024-05-28T06:04:11.909515+0000","last_became_active":"2024-05-28T06:04:10.926364+0000","last_became_peered":"2024-05-28T06:04:10.926364+0000","last_unstale":"2024-05-28T06:04:11.909515+0000","last_undegraded":"2024-05-28T06:04:11.909515+0000","last_fullsized":"2024-05-28T06:04:11.909515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:28:21.123299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.893744+0000","last_change":"2024-05-28T06:04:10.913441+0000","last_active":"2024-05-28T06:04:11.893744+0000","last_peered":"2024-05-28T06:04:11.893744+0000","last_clean":"2024-05-28T06:04:11.893744+0000","last_became_active":"2024-05-28T06:04:10.912847+0000","last_became_peered":"2024-05-28T06:04:10.912847+0000","last_unstale":"2024-05-28T06:04:11.893744+0000","last_undegraded":"2024-05-28T06:04:11.893744+0000","last_fullsized":"2024-05-28T06:04:11.893744+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:58:51.819866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.996503+0000","last_change":"2024-05-28T06:04:10.909268+0000","last_active":"2024-05-28T06:04:12.996503+0000","last_peered":"2024-05-28T06:04:12.996503+0000","last_clean":"2024-05-28T06:04:12.996503+0000","last_became_active":"2024-05-28T06:04:10.908954+0000","last_became_peered":"2024-05-28T06:04:10.908954+0000","last_unstale":"2024-05-28T06:04:12.996503+0000","last_undegraded":"2024-05-28T06:04:12.996503+0000","last_fullsized":"2024-05-28T06:04:12.996503+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:28:21.123299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.793137+0000","last_change":"2024-05-28T06:04:10.899509+0000","last_active":"2024-05-28T06:04:12.793137+0000","last_peered":"2024-05-28T06:04:12.793137+0000","last_clean":"2024-05-28T06:04:12.793137+0000","last_became_active":"2024-05-28T06:04:10.896990+0000","last_became_peered":"2024-05-28T06:04:10.896990+0000","last_unstale":"2024-05-28T06:04:12.793137+0000","last_undegraded":"2024-05-28T06:04:12.793137+0000","last_fullsized":"2024-05-28T06:04:12.793137+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:17:36.664466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.890266+0000","last_change":"2024-05-28T06:04:10.906808+0000","last_active":"2024-05-28T06:04:11.890266+0000","last_peered":"2024-05-28T06:04:11.890266+0000","last_clean":"2024-05-28T06:04:11.890266+0000","last_became_active":"2024-05-28T06:04:10.906579+0000","last_became_peered":"2024-05-28T06:04:10.906579+0000","last_unstale":"2024-05-28T06:04:11.890266+0000","last_undegraded":"2024-05-28T06:04:11.890266+0000","last_fullsized":"2024-05-28T06:04:11.890266+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:28:21.123299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.997630+0000","last_change":"2024-05-28T06:04:10.904806+0000","last_active":"2024-05-28T06:04:12.997630+0000","last_peered":"2024-05-28T06:04:12.997630+0000","last_clean":"2024-05-28T06:04:12.997630+0000","last_became_active":"2024-05-28T06:04:10.904450+0000","last_became_peered":"2024-05-28T06:04:10.904450+0000","last_unstale":"2024-05-28T06:04:12.997630+0000","last_undegraded":"2024-05-28T06:04:12.997630+0000","last_fullsized":"2024-05-28T06:04:12.997630+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:51:12.839226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.999350+0000","last_change":"2024-05-28T06:04:10.918376+0000","last_active":"2024-05-28T06:04:12.999350+0000","last_peered":"2024-05-28T06:04:12.999350+0000","last_clean":"2024-05-28T06:04:12.999350+0000","last_became_active":"2024-05-28T06:04:10.918042+0000","last_became_peered":"2024-05-28T06:04:10.918042+0000","last_unstale":"2024-05-28T06:04:12.999350+0000","last_undegraded":"2024-05-28T06:04:12.999350+0000","last_fullsized":"2024-05-28T06:04:12.999350+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:39:13.976770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.996501+0000","last_change":"2024-05-28T06:04:10.910995+0000","last_active":"2024-05-28T06:04:12.996501+0000","last_peered":"2024-05-28T06:04:12.996501+0000","last_clean":"2024-05-28T06:04:12.996501+0000","last_became_active":"2024-05-28T06:04:10.910672+0000","last_became_peered":"2024-05-28T06:04:10.910672+0000","last_unstale":"2024-05-28T06:04:12.996501+0000","last_undegraded":"2024-05-28T06:04:12.996501+0000","last_fullsized":"2024-05-28T06:04:12.996501+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:58:51.819866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.997411+0000","last_change":"2024-05-28T06:04:10.911244+0000","last_active":"2024-05-28T06:04:12.997411+0000","last_peered":"2024-05-28T06:04:12.997411+0000","last_clean":"2024-05-28T06:04:12.997411+0000","last_became_active":"2024-05-28T06:04:10.910921+0000","last_became_peered":"2024-05-28T06:04:10.910921+0000","last_unstale":"2024-05-28T06:04:12.997411+0000","last_undegraded":"2024-05-28T06:04:12.997411+0000","last_fullsized":"2024-05-28T06:04:12.997411+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:05:14.695994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.909154+0000","last_change":"2024-05-28T06:04:10.921344+0000","last_active":"2024-05-28T06:04:11.909154+0000","last_peered":"2024-05-28T06:04:11.909154+0000","last_clean":"2024-05-28T06:04:11.909154+0000","last_became_active":"2024-05-28T06:04:10.920438+0000","last_became_peered":"2024-05-28T06:04:10.920438+0000","last_unstale":"2024-05-28T06:04:11.909154+0000","last_undegraded":"2024-05-28T06:04:11.909154+0000","last_fullsized":"2024-05-28T06:04:11.909154+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:53:07.733019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.793512+0000","last_change":"2024-05-28T06:04:10.902770+0000","last_active":"2024-05-28T06:04:12.793512+0000","last_peered":"2024-05-28T06:04:12.793512+0000","last_clean":"2024-05-28T06:04:12.793512+0000","last_became_active":"2024-05-28T06:04:10.901534+0000","last_became_peered":"2024-05-28T06:04:10.901534+0000","last_unstale":"2024-05-28T06:04:12.793512+0000","last_undegraded":"2024-05-28T06:04:12.793512+0000","last_fullsized":"2024-05-28T06:04:12.793512+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:00:40.966129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.996957+0000","last_change":"2024-05-28T06:04:10.915708+0000","last_active":"2024-05-28T06:04:12.996957+0000","last_peered":"2024-05-28T06:04:12.996957+0000","last_clean":"2024-05-28T06:04:12.996957+0000","last_became_active":"2024-05-28T06:04:10.913058+0000","last_became_peered":"2024-05-28T06:04:10.913058+0000","last_unstale":"2024-05-28T06:04:12.996957+0000","last_undegraded":"2024-05-28T06:04:12.996957+0000","last_fullsized":"2024-05-28T06:04:12.996957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:11:51.651640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.995352+0000","last_change":"2024-05-28T06:04:10.904255+0000","last_active":"2024-05-28T06:04:12.995352+0000","last_peered":"2024-05-28T06:04:12.995352+0000","last_clean":"2024-05-28T06:04:12.995352+0000","last_became_active":"2024-05-28T06:04:10.903840+0000","last_became_peered":"2024-05-28T06:04:10.903840+0000","last_unstale":"2024-05-28T06:04:12.995352+0000","last_undegraded":"2024-05-28T06:04:12.995352+0000","last_fullsized":"2024-05-28T06:04:12.995352+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:06:44.034610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.997441+0000","last_change":"2024-05-28T06:04:10.902458+0000","last_active":"2024-05-28T06:04:12.997441+0000","last_peered":"2024-05-28T06:04:12.997441+0000","last_clean":"2024-05-28T06:04:12.997441+0000","last_became_active":"2024-05-28T06:04:10.902054+0000","last_became_peered":"2024-05-28T06:04:10.902054+0000","last_unstale":"2024-05-28T06:04:12.997441+0000","last_undegraded":"2024-05-28T06:04:12.997441+0000","last_fullsized":"2024-05-28T06:04:12.997441+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:46:27.923407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.997558+0000","last_change":"2024-05-28T06:04:10.895892+0000","last_active":"2024-05-28T06:04:12.997558+0000","last_peered":"2024-05-28T06:04:12.997558+0000","last_clean":"2024-05-28T06:04:12.997558+0000","last_became_active":"2024-05-28T06:04:10.895579+0000","last_became_peered":"2024-05-28T06:04:10.895579+0000","last_unstale":"2024-05-28T06:04:12.997558+0000","last_undegraded":"2024-05-28T06:04:12.997558+0000","last_fullsized":"2024-05-28T06:04:12.997558+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:51:12.839226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.909428+0000","last_change":"2024-05-28T06:04:10.914830+0000","last_active":"2024-05-28T06:04:11.909428+0000","last_peered":"2024-05-28T06:04:11.909428+0000","last_clean":"2024-05-28T06:04:11.909428+0000","last_became_active":"2024-05-28T06:04:10.913651+0000","last_became_peered":"2024-05-28T06:04:10.913651+0000","last_unstale":"2024-05-28T06:04:11.909428+0000","last_undegraded":"2024-05-28T06:04:11.909428+0000","last_fullsized":"2024-05-28T06:04:11.909428+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:46:29.255545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.889962+0000","last_change":"2024-05-28T06:04:10.903385+0000","last_active":"2024-05-28T06:04:11.889962+0000","last_peered":"2024-05-28T06:04:11.889962+0000","last_clean":"2024-05-28T06:04:11.889962+0000","last_became_active":"2024-05-28T06:04:10.903078+0000","last_became_peered":"2024-05-28T06:04:10.903078+0000","last_unstale":"2024-05-28T06:04:11.889962+0000","last_undegraded":"2024-05-28T06:04:11.889962+0000","last_fullsized":"2024-05-28T06:04:11.889962+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:06:44.034610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.997438+0000","last_change":"2024-05-28T06:04:10.894062+0000","last_active":"2024-05-28T06:04:12.997438+0000","last_peered":"2024-05-28T06:04:12.997438+0000","last_clean":"2024-05-28T06:04:12.997438+0000","last_became_active":"2024-05-28T06:04:10.893763+0000","last_became_peered":"2024-05-28T06:04:10.893763+0000","last_unstale":"2024-05-28T06:04:12.997438+0000","last_undegraded":"2024-05-28T06:04:12.997438+0000","last_fullsized":"2024-05-28T06:04:12.997438+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:46:27.923407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.793745+0000","last_change":"2024-05-28T06:04:10.897684+0000","last_active":"2024-05-28T06:04:12.793745+0000","last_peered":"2024-05-28T06:04:12.793745+0000","last_clean":"2024-05-28T06:04:12.793745+0000","last_became_active":"2024-05-28T06:04:10.897382+0000","last_became_peered":"2024-05-28T06:04:10.897382+0000","last_unstale":"2024-05-28T06:04:12.793745+0000","last_undegraded":"2024-05-28T06:04:12.793745+0000","last_fullsized":"2024-05-28T06:04:12.793745+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:39:46.389531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.997085+0000","last_change":"2024-05-28T06:04:10.908012+0000","last_active":"2024-05-28T06:04:12.997085+0000","last_peered":"2024-05-28T06:04:12.997085+0000","last_clean":"2024-05-28T06:04:12.997085+0000","last_became_active":"2024-05-28T06:04:10.907436+0000","last_became_peered":"2024-05-28T06:04:10.907436+0000","last_unstale":"2024-05-28T06:04:12.997085+0000","last_undegraded":"2024-05-28T06:04:12.997085+0000","last_fullsized":"2024-05-28T06:04:12.997085+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:58:16.009219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.996141+0000","last_change":"2024-05-28T06:04:10.905729+0000","last_active":"2024-05-28T06:04:12.996141+0000","last_peered":"2024-05-28T06:04:12.996141+0000","last_clean":"2024-05-28T06:04:12.996141+0000","last_became_active":"2024-05-28T06:04:10.905325+0000","last_became_peered":"2024-05-28T06:04:10.905325+0000","last_unstale":"2024-05-28T06:04:12.996141+0000","last_undegraded":"2024-05-28T06:04:12.996141+0000","last_fullsized":"2024-05-28T06:04:12.996141+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:03:46.394497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.908750+0000","last_change":"2024-05-28T06:04:10.915117+0000","last_active":"2024-05-28T06:04:11.908750+0000","last_peered":"2024-05-28T06:04:11.908750+0000","last_clean":"2024-05-28T06:04:11.908750+0000","last_became_active":"2024-05-28T06:04:10.914761+0000","last_became_peered":"2024-05-28T06:04:10.914761+0000","last_unstale":"2024-05-28T06:04:11.908750+0000","last_undegraded":"2024-05-28T06:04:11.908750+0000","last_fullsized":"2024-05-28T06:04:11.908750+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:00:40.966129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.892804+0000","last_change":"2024-05-28T06:04:10.897395+0000","last_active":"2024-05-28T06:04:11.892804+0000","last_peered":"2024-05-28T06:04:11.892804+0000","last_clean":"2024-05-28T06:04:11.892804+0000","last_became_active":"2024-05-28T06:04:10.897108+0000","last_became_peered":"2024-05-28T06:04:10.897108+0000","last_unstale":"2024-05-28T06:04:11.892804+0000","last_undegraded":"2024-05-28T06:04:11.892804+0000","last_fullsized":"2024-05-28T06:04:11.892804+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:16:53.718673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.996650+0000","last_change":"2024-05-28T06:04:09.890037+0000","last_active":"2024-05-28T06:04:12.996650+0000","last_peered":"2024-05-28T06:04:12.996650+0000","last_clean":"2024-05-28T06:04:12.996650+0000","last_became_active":"2024-05-28T06:04:09.889696+0000","last_became_peered":"2024-05-28T06:04:09.889696+0000","last_unstale":"2024-05-28T06:04:12.996650+0000","last_undegraded":"2024-05-28T06:04:12.996650+0000","last_fullsized":"2024-05-28T06:04:12.996650+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:29:32.319083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.794133+0000","last_change":"2024-05-28T06:04:10.896136+0000","last_active":"2024-05-28T06:04:12.794133+0000","last_peered":"2024-05-28T06:04:12.794133+0000","last_clean":"2024-05-28T06:04:12.794133+0000","last_became_active":"2024-05-28T06:04:10.895631+0000","last_became_peered":"2024-05-28T06:04:10.895631+0000","last_unstale":"2024-05-28T06:04:12.794133+0000","last_undegraded":"2024-05-28T06:04:12.794133+0000","last_fullsized":"2024-05-28T06:04:12.794133+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:46:29.255545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.908566+0000","last_change":"2024-05-28T06:04:09.879488+0000","last_active":"2024-05-28T06:04:11.908566+0000","last_peered":"2024-05-28T06:04:11.908566+0000","last_clean":"2024-05-28T06:04:11.908566+0000","last_became_active":"2024-05-28T06:04:09.879182+0000","last_became_peered":"2024-05-28T06:04:09.879182+0000","last_unstale":"2024-05-28T06:04:11.908566+0000","last_undegraded":"2024-05-28T06:04:11.908566+0000","last_fullsized":"2024-05-28T06:04:11.908566+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:29:32.319083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.995389+0000","last_change":"2024-05-28T06:04:09.884424+0000","last_active":"2024-05-28T06:04:12.995389+0000","last_peered":"2024-05-28T06:04:12.995389+0000","last_clean":"2024-05-28T06:04:12.995389+0000","last_became_active":"2024-05-28T06:04:09.884088+0000","last_became_peered":"2024-05-28T06:04:09.884088+0000","last_unstale":"2024-05-28T06:04:12.995389+0000","last_undegraded":"2024-05-28T06:04:12.995389+0000","last_fullsized":"2024-05-28T06:04:12.995389+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:27:30.379288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.996856+0000","last_change":"2024-05-28T06:04:10.890237+0000","last_active":"2024-05-28T06:04:12.996856+0000","last_peered":"2024-05-28T06:04:12.996856+0000","last_clean":"2024-05-28T06:04:12.996856+0000","last_became_active":"2024-05-28T06:04:10.889644+0000","last_became_peered":"2024-05-28T06:04:10.889644+0000","last_unstale":"2024-05-28T06:04:12.996856+0000","last_undegraded":"2024-05-28T06:04:12.996856+0000","last_fullsized":"2024-05-28T06:04:12.996856+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:58:16.009219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.997564+0000","last_change":"2024-05-28T06:04:09.875990+0000","last_active":"2024-05-28T06:04:12.997564+0000","last_peered":"2024-05-28T06:04:12.997564+0000","last_clean":"2024-05-28T06:04:12.997564+0000","last_became_active":"2024-05-28T06:04:09.875644+0000","last_became_peered":"2024-05-28T06:04:09.875644+0000","last_unstale":"2024-05-28T06:04:12.997564+0000","last_undegraded":"2024-05-28T06:04:12.997564+0000","last_fullsized":"2024-05-28T06:04:12.997564+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:27:30.379288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.889859+0000","last_change":"2024-05-28T06:04:10.899922+0000","last_active":"2024-05-28T06:04:11.889859+0000","last_peered":"2024-05-28T06:04:11.889859+0000","last_clean":"2024-05-28T06:04:11.889859+0000","last_became_active":"2024-05-28T06:04:10.899617+0000","last_became_peered":"2024-05-28T06:04:10.899617+0000","last_unstale":"2024-05-28T06:04:11.889859+0000","last_undegraded":"2024-05-28T06:04:11.889859+0000","last_fullsized":"2024-05-28T06:04:11.889859+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:39:46.389531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.909050+0000","last_change":"2024-05-28T06:04:09.887084+0000","last_active":"2024-05-28T06:04:11.909050+0000","last_peered":"2024-05-28T06:04:11.909050+0000","last_clean":"2024-05-28T06:04:11.909050+0000","last_became_active":"2024-05-28T06:04:09.886821+0000","last_became_peered":"2024-05-28T06:04:09.886821+0000","last_unstale":"2024-05-28T06:04:11.909050+0000","last_undegraded":"2024-05-28T06:04:11.909050+0000","last_fullsized":"2024-05-28T06:04:11.909050+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:58:06.720498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.998553+0000","last_change":"2024-05-28T06:04:10.921008+0000","last_active":"2024-05-28T06:04:12.998553+0000","last_peered":"2024-05-28T06:04:12.998553+0000","last_clean":"2024-05-28T06:04:12.998553+0000","last_became_active":"2024-05-28T06:04:10.913026+0000","last_became_peered":"2024-05-28T06:04:10.913026+0000","last_unstale":"2024-05-28T06:04:12.998553+0000","last_undegraded":"2024-05-28T06:04:12.998553+0000","last_fullsized":"2024-05-28T06:04:12.998553+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:34:44.387474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.998439+0000","last_change":"2024-05-28T06:04:09.887088+0000","last_active":"2024-05-28T06:04:12.998439+0000","last_peered":"2024-05-28T06:04:12.998439+0000","last_clean":"2024-05-28T06:04:12.998439+0000","last_became_active":"2024-05-28T06:04:09.886809+0000","last_became_peered":"2024-05-28T06:04:09.886809+0000","last_unstale":"2024-05-28T06:04:12.998439+0000","last_undegraded":"2024-05-28T06:04:12.998439+0000","last_fullsized":"2024-05-28T06:04:12.998439+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:16:52.703842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.794379+0000","last_change":"2024-05-28T06:04:10.904810+0000","last_active":"2024-05-28T06:04:12.794379+0000","last_peered":"2024-05-28T06:04:12.794379+0000","last_clean":"2024-05-28T06:04:12.794379+0000","last_became_active":"2024-05-28T06:04:10.904445+0000","last_became_peered":"2024-05-28T06:04:10.904445+0000","last_unstale":"2024-05-28T06:04:12.794379+0000","last_undegraded":"2024-05-28T06:04:12.794379+0000","last_fullsized":"2024-05-28T06:04:12.794379+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:58:16.009219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.890096+0000","last_change":"2024-05-28T06:04:09.882535+0000","last_active":"2024-05-28T06:04:11.890096+0000","last_peered":"2024-05-28T06:04:11.890096+0000","last_clean":"2024-05-28T06:04:11.890096+0000","last_became_active":"2024-05-28T06:04:09.882159+0000","last_became_peered":"2024-05-28T06:04:09.882159+0000","last_unstale":"2024-05-28T06:04:11.890096+0000","last_undegraded":"2024-05-28T06:04:11.890096+0000","last_fullsized":"2024-05-28T06:04:11.890096+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:03:45.379665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.893646+0000","last_change":"2024-05-28T06:04:10.901897+0000","last_active":"2024-05-28T06:04:11.893646+0000","last_peered":"2024-05-28T06:04:11.893646+0000","last_clean":"2024-05-28T06:04:11.893646+0000","last_became_active":"2024-05-28T06:04:10.901156+0000","last_became_peered":"2024-05-28T06:04:10.901156+0000","last_unstale":"2024-05-28T06:04:11.893646+0000","last_undegraded":"2024-05-28T06:04:11.893646+0000","last_fullsized":"2024-05-28T06:04:11.893646+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:03:46.394497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.909296+0000","last_change":"2024-05-28T06:04:09.878442+0000","last_active":"2024-05-28T06:04:11.909296+0000","last_peered":"2024-05-28T06:04:11.909296+0000","last_clean":"2024-05-28T06:04:11.909296+0000","last_became_active":"2024-05-28T06:04:09.878148+0000","last_became_peered":"2024-05-28T06:04:09.878148+0000","last_unstale":"2024-05-28T06:04:11.909296+0000","last_undegraded":"2024-05-28T06:04:11.909296+0000","last_fullsized":"2024-05-28T06:04:11.909296+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:38:20.882939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.996201+0000","last_change":"2024-05-28T06:04:10.911058+0000","last_active":"2024-05-28T06:04:12.996201+0000","last_peered":"2024-05-28T06:04:12.996201+0000","last_clean":"2024-05-28T06:04:12.996201+0000","last_became_active":"2024-05-28T06:04:10.910775+0000","last_became_peered":"2024-05-28T06:04:10.910775+0000","last_unstale":"2024-05-28T06:04:12.996201+0000","last_undegraded":"2024-05-28T06:04:12.996201+0000","last_fullsized":"2024-05-28T06:04:12.996201+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:03:18.610996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.794179+0000","last_change":"2024-05-28T06:04:09.878327+0000","last_active":"2024-05-28T06:04:12.794179+0000","last_peered":"2024-05-28T06:04:12.794179+0000","last_clean":"2024-05-28T06:04:12.794179+0000","last_became_active":"2024-05-28T06:04:09.878028+0000","last_became_peered":"2024-05-28T06:04:09.878028+0000","last_unstale":"2024-05-28T06:04:12.794179+0000","last_undegraded":"2024-05-28T06:04:12.794179+0000","last_fullsized":"2024-05-28T06:04:12.794179+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:12:29.496721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.793473+0000","last_change":"2024-05-28T06:04:10.902163+0000","last_active":"2024-05-28T06:04:12.793473+0000","last_peered":"2024-05-28T06:04:12.793473+0000","last_clean":"2024-05-28T06:04:12.793473+0000","last_became_active":"2024-05-28T06:04:10.901846+0000","last_became_peered":"2024-05-28T06:04:10.901846+0000","last_unstale":"2024-05-28T06:04:12.793473+0000","last_undegraded":"2024-05-28T06:04:12.793473+0000","last_fullsized":"2024-05-28T06:04:12.793473+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:11:13.101661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.997163+0000","last_change":"2024-05-28T06:04:09.884756+0000","last_active":"2024-05-28T06:04:12.997163+0000","last_peered":"2024-05-28T06:04:12.997163+0000","last_clean":"2024-05-28T06:04:12.997163+0000","last_became_active":"2024-05-28T06:04:09.884354+0000","last_became_peered":"2024-05-28T06:04:09.884354+0000","last_unstale":"2024-05-28T06:04:12.997163+0000","last_undegraded":"2024-05-28T06:04:12.997163+0000","last_fullsized":"2024-05-28T06:04:12.997163+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:36:53.154893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.996772+0000","last_change":"2024-05-28T06:04:10.916180+0000","last_active":"2024-05-28T06:04:12.996772+0000","last_peered":"2024-05-28T06:04:12.996772+0000","last_clean":"2024-05-28T06:04:12.996772+0000","last_became_active":"2024-05-28T06:04:10.915630+0000","last_became_peered":"2024-05-28T06:04:10.915630+0000","last_unstale":"2024-05-28T06:04:12.996772+0000","last_undegraded":"2024-05-28T06:04:12.996772+0000","last_fullsized":"2024-05-28T06:04:12.996772+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:34:44.387474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.908470+0000","last_change":"2024-05-28T06:04:09.883806+0000","last_active":"2024-05-28T06:04:11.908470+0000","last_peered":"2024-05-28T06:04:11.908470+0000","last_clean":"2024-05-28T06:04:11.908470+0000","last_became_active":"2024-05-28T06:04:09.883352+0000","last_became_peered":"2024-05-28T06:04:09.883352+0000","last_unstale":"2024-05-28T06:04:11.908470+0000","last_undegraded":"2024-05-28T06:04:11.908470+0000","last_fullsized":"2024-05-28T06:04:11.908470+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:36:53.154893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.908799+0000","last_change":"2024-05-28T06:04:10.908671+0000","last_active":"2024-05-28T06:04:11.908799+0000","last_peered":"2024-05-28T06:04:11.908799+0000","last_clean":"2024-05-28T06:04:11.908799+0000","last_became_active":"2024-05-28T06:04:10.908402+0000","last_became_peered":"2024-05-28T06:04:10.908402+0000","last_unstale":"2024-05-28T06:04:11.908799+0000","last_undegraded":"2024-05-28T06:04:11.908799+0000","last_fullsized":"2024-05-28T06:04:11.908799+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:06:44.034610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.995124+0000","last_change":"2024-05-28T06:04:09.868302+0000","last_active":"2024-05-28T06:04:12.995124+0000","last_peered":"2024-05-28T06:04:12.995124+0000","last_clean":"2024-05-28T06:04:12.995124+0000","last_became_active":"2024-05-28T06:04:09.868003+0000","last_became_peered":"2024-05-28T06:04:09.868003+0000","last_unstale":"2024-05-28T06:04:12.995124+0000","last_undegraded":"2024-05-28T06:04:12.995124+0000","last_fullsized":"2024-05-28T06:04:12.995124+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:29:32.319083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.995729+0000","last_change":"2024-05-28T06:04:10.916061+0000","last_active":"2024-05-28T06:04:12.995729+0000","last_peered":"2024-05-28T06:04:12.995729+0000","last_clean":"2024-05-28T06:04:12.995729+0000","last_became_active":"2024-05-28T06:04:10.915607+0000","last_became_peered":"2024-05-28T06:04:10.915607+0000","last_unstale":"2024-05-28T06:04:12.995729+0000","last_undegraded":"2024-05-28T06:04:12.995729+0000","last_fullsized":"2024-05-28T06:04:12.995729+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:51:12.839226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.893456+0000","last_change":"2024-05-28T06:04:09.874073+0000","last_active":"2024-05-28T06:04:11.893456+0000","last_peered":"2024-05-28T06:04:11.893456+0000","last_clean":"2024-05-28T06:04:11.893456+0000","last_became_active":"2024-05-28T06:04:09.872906+0000","last_became_peered":"2024-05-28T06:04:09.872906+0000","last_unstale":"2024-05-28T06:04:11.893456+0000","last_undegraded":"2024-05-28T06:04:11.893456+0000","last_fullsized":"2024-05-28T06:04:11.893456+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:06:04.907427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.998407+0000","last_change":"2024-05-28T06:04:10.914311+0000","last_active":"2024-05-28T06:04:12.998407+0000","last_peered":"2024-05-28T06:04:12.998407+0000","last_clean":"2024-05-28T06:04:12.998407+0000","last_became_active":"2024-05-28T06:04:10.913972+0000","last_became_peered":"2024-05-28T06:04:10.913972+0000","last_unstale":"2024-05-28T06:04:12.998407+0000","last_undegraded":"2024-05-28T06:04:12.998407+0000","last_fullsized":"2024-05-28T06:04:12.998407+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:11:58.309255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.996846+0000","last_change":"2024-05-28T06:04:09.867608+0000","last_active":"2024-05-28T06:04:12.996846+0000","last_peered":"2024-05-28T06:04:12.996846+0000","last_clean":"2024-05-28T06:04:12.996846+0000","last_became_active":"2024-05-28T06:04:09.865750+0000","last_became_peered":"2024-05-28T06:04:09.865750+0000","last_unstale":"2024-05-28T06:04:12.996846+0000","last_undegraded":"2024-05-28T06:04:12.996846+0000","last_fullsized":"2024-05-28T06:04:12.996846+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:16:52.703842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.998212+0000","last_change":"2024-05-28T06:04:10.914237+0000","last_active":"2024-05-28T06:04:12.998212+0000","last_peered":"2024-05-28T06:04:12.998212+0000","last_clean":"2024-05-28T06:04:12.998212+0000","last_became_active":"2024-05-28T06:04:10.913957+0000","last_became_peered":"2024-05-28T06:04:10.913957+0000","last_unstale":"2024-05-28T06:04:12.998212+0000","last_undegraded":"2024-05-28T06:04:12.998212+0000","last_fullsized":"2024-05-28T06:04:12.998212+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:28:36.252853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.890109+0000","last_change":"2024-05-28T06:04:09.877173+0000","last_active":"2024-05-28T06:04:11.890109+0000","last_peered":"2024-05-28T06:04:11.890109+0000","last_clean":"2024-05-28T06:04:11.890109+0000","last_became_active":"2024-05-28T06:04:09.876534+0000","last_became_peered":"2024-05-28T06:04:09.876534+0000","last_unstale":"2024-05-28T06:04:11.890109+0000","last_undegraded":"2024-05-28T06:04:11.890109+0000","last_fullsized":"2024-05-28T06:04:11.890109+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:58:50.805034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.998844+0000","last_change":"2024-05-28T06:04:10.923771+0000","last_active":"2024-05-28T06:04:12.998844+0000","last_peered":"2024-05-28T06:04:12.998844+0000","last_clean":"2024-05-28T06:04:12.998844+0000","last_became_active":"2024-05-28T06:04:10.923494+0000","last_became_peered":"2024-05-28T06:04:10.923494+0000","last_unstale":"2024-05-28T06:04:12.998844+0000","last_undegraded":"2024-05-28T06:04:12.998844+0000","last_fullsized":"2024-05-28T06:04:12.998844+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:05:14.695994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.995877+0000","last_change":"2024-05-28T06:04:09.880725+0000","last_active":"2024-05-28T06:04:12.995877+0000","last_peered":"2024-05-28T06:04:12.995877+0000","last_clean":"2024-05-28T06:04:12.995877+0000","last_became_active":"2024-05-28T06:04:09.880430+0000","last_became_peered":"2024-05-28T06:04:09.880430+0000","last_unstale":"2024-05-28T06:04:12.995877+0000","last_undegraded":"2024-05-28T06:04:12.995877+0000","last_fullsized":"2024-05-28T06:04:12.995877+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:27:30.379288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.996167+0000","last_change":"2024-05-28T06:04:10.904576+0000","last_active":"2024-05-28T06:04:12.996167+0000","last_peered":"2024-05-28T06:04:12.996167+0000","last_clean":"2024-05-28T06:04:12.996167+0000","last_became_active":"2024-05-28T06:04:10.904257+0000","last_became_peered":"2024-05-28T06:04:10.904257+0000","last_unstale":"2024-05-28T06:04:12.996167+0000","last_undegraded":"2024-05-28T06:04:12.996167+0000","last_fullsized":"2024-05-28T06:04:12.996167+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:49:22.704773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":170,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.995506+0000","last_change":"2024-05-28T06:04:01.039129+0000","last_active":"2024-05-28T06:04:12.995506+0000","last_peered":"2024-05-28T06:04:12.995506+0000","last_clean":"2024-05-28T06:04:12.995506+0000","last_became_active":"2024-05-28T06:04:01.033906+0000","last_became_peered":"2024-05-28T06:04:01.033906+0000","last_unstale":"2024-05-28T06:04:12.995506+0000","last_undegraded":"2024-05-28T06:04:12.995506+0000","last_fullsized":"2024-05-28T06:04:12.995506+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:03:59.864237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:03:59.864237+0000","last_clean_scrub_stamp":"2024-05-28T06:03:59.864237+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:17:36.440245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.996021+0000","last_change":"2024-05-28T06:04:09.879679+0000","last_active":"2024-05-28T06:04:12.996021+0000","last_peered":"2024-05-28T06:04:12.996021+0000","last_clean":"2024-05-28T06:04:12.996021+0000","last_became_active":"2024-05-28T06:04:09.877412+0000","last_became_peered":"2024-05-28T06:04:09.877412+0000","last_unstale":"2024-05-28T06:04:12.996021+0000","last_undegraded":"2024-05-28T06:04:12.996021+0000","last_fullsized":"2024-05-28T06:04:12.996021+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:58:06.720498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.996386+0000","last_change":"2024-05-28T06:04:10.901746+0000","last_active":"2024-05-28T06:04:12.996386+0000","last_peered":"2024-05-28T06:04:12.996386+0000","last_clean":"2024-05-28T06:04:12.996386+0000","last_became_active":"2024-05-28T06:04:10.901222+0000","last_became_peered":"2024-05-28T06:04:10.901222+0000","last_unstale":"2024-05-28T06:04:12.996386+0000","last_undegraded":"2024-05-28T06:04:12.996386+0000","last_fullsized":"2024-05-28T06:04:12.996386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:46:29.255545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.995167+0000","last_change":"2024-05-28T06:04:09.890312+0000","last_active":"2024-05-28T06:04:12.995167+0000","last_peered":"2024-05-28T06:04:12.995167+0000","last_clean":"2024-05-28T06:04:12.995167+0000","last_became_active":"2024-05-28T06:04:09.889754+0000","last_became_peered":"2024-05-28T06:04:09.889754+0000","last_unstale":"2024-05-28T06:04:12.995167+0000","last_undegraded":"2024-05-28T06:04:12.995167+0000","last_fullsized":"2024-05-28T06:04:12.995167+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:38:20.882939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.793983+0000","last_change":"2024-05-28T06:04:10.886055+0000","last_active":"2024-05-28T06:04:12.793983+0000","last_peered":"2024-05-28T06:04:12.793983+0000","last_clean":"2024-05-28T06:04:12.793983+0000","last_became_active":"2024-05-28T06:04:10.885596+0000","last_became_peered":"2024-05-28T06:04:10.885596+0000","last_unstale":"2024-05-28T06:04:12.793983+0000","last_undegraded":"2024-05-28T06:04:12.793983+0000","last_fullsized":"2024-05-28T06:04:12.793983+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:06:44.034610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.995673+0000","last_change":"2024-05-28T06:04:09.871789+0000","last_active":"2024-05-28T06:04:12.995673+0000","last_peered":"2024-05-28T06:04:12.995673+0000","last_clean":"2024-05-28T06:04:12.995673+0000","last_became_active":"2024-05-28T06:04:09.870232+0000","last_became_peered":"2024-05-28T06:04:09.870232+0000","last_unstale":"2024-05-28T06:04:12.995673+0000","last_undegraded":"2024-05-28T06:04:12.995673+0000","last_fullsized":"2024-05-28T06:04:12.995673+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:36:53.154893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.890126+0000","last_change":"2024-05-28T06:04:10.898534+0000","last_active":"2024-05-28T06:04:11.890126+0000","last_peered":"2024-05-28T06:04:11.890126+0000","last_clean":"2024-05-28T06:04:11.890126+0000","last_became_active":"2024-05-28T06:04:10.898237+0000","last_became_peered":"2024-05-28T06:04:10.898237+0000","last_unstale":"2024-05-28T06:04:11.890126+0000","last_undegraded":"2024-05-28T06:04:11.890126+0000","last_fullsized":"2024-05-28T06:04:11.890126+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:46:29.255545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.997224+0000","last_change":"2024-05-28T06:04:09.884779+0000","last_active":"2024-05-28T06:04:12.997224+0000","last_peered":"2024-05-28T06:04:12.997224+0000","last_clean":"2024-05-28T06:04:12.997224+0000","last_became_active":"2024-05-28T06:04:09.882344+0000","last_became_peered":"2024-05-28T06:04:09.882344+0000","last_unstale":"2024-05-28T06:04:12.997224+0000","last_undegraded":"2024-05-28T06:04:12.997224+0000","last_fullsized":"2024-05-28T06:04:12.997224+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:29:32.319083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.893200+0000","last_change":"2024-05-28T06:04:10.890674+0000","last_active":"2024-05-28T06:04:11.893200+0000","last_peered":"2024-05-28T06:04:11.893200+0000","last_clean":"2024-05-28T06:04:11.893200+0000","last_became_active":"2024-05-28T06:04:10.890144+0000","last_became_peered":"2024-05-28T06:04:10.890144+0000","last_unstale":"2024-05-28T06:04:11.893200+0000","last_undegraded":"2024-05-28T06:04:11.893200+0000","last_fullsized":"2024-05-28T06:04:11.893200+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:38:21.897771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.890349+0000","last_change":"2024-05-28T06:04:09.871025+0000","last_active":"2024-05-28T06:04:11.890349+0000","last_peered":"2024-05-28T06:04:11.890349+0000","last_clean":"2024-05-28T06:04:11.890349+0000","last_became_active":"2024-05-28T06:04:09.870646+0000","last_became_peered":"2024-05-28T06:04:09.870646+0000","last_unstale":"2024-05-28T06:04:11.890349+0000","last_undegraded":"2024-05-28T06:04:11.890349+0000","last_fullsized":"2024-05-28T06:04:11.890349+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:16:52.703842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.998525+0000","last_change":"2024-05-28T06:04:10.912585+0000","last_active":"2024-05-28T06:04:12.998525+0000","last_peered":"2024-05-28T06:04:12.998525+0000","last_clean":"2024-05-28T06:04:12.998525+0000","last_became_active":"2024-05-28T06:04:10.912038+0000","last_became_peered":"2024-05-28T06:04:10.912038+0000","last_unstale":"2024-05-28T06:04:12.998525+0000","last_undegraded":"2024-05-28T06:04:12.998525+0000","last_fullsized":"2024-05-28T06:04:12.998525+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:04:49.415234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.997550+0000","last_change":"2024-05-28T06:04:09.890971+0000","last_active":"2024-05-28T06:04:12.997550+0000","last_peered":"2024-05-28T06:04:12.997550+0000","last_clean":"2024-05-28T06:04:12.997550+0000","last_became_active":"2024-05-28T06:04:09.890401+0000","last_became_peered":"2024-05-28T06:04:09.890401+0000","last_unstale":"2024-05-28T06:04:12.997550+0000","last_undegraded":"2024-05-28T06:04:12.997550+0000","last_fullsized":"2024-05-28T06:04:12.997550+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:05:29.419386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.890261+0000","last_change":"2024-05-28T06:04:10.903420+0000","last_active":"2024-05-28T06:04:11.890261+0000","last_peered":"2024-05-28T06:04:11.890261+0000","last_clean":"2024-05-28T06:04:11.890261+0000","last_became_active":"2024-05-28T06:04:10.898157+0000","last_became_peered":"2024-05-28T06:04:10.898157+0000","last_unstale":"2024-05-28T06:04:11.890261+0000","last_undegraded":"2024-05-28T06:04:11.890261+0000","last_fullsized":"2024-05-28T06:04:11.890261+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:11:51.651640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.890155+0000","last_change":"2024-05-28T06:04:09.886223+0000","last_active":"2024-05-28T06:04:11.890155+0000","last_peered":"2024-05-28T06:04:11.890155+0000","last_clean":"2024-05-28T06:04:11.890155+0000","last_became_active":"2024-05-28T06:04:09.885439+0000","last_became_peered":"2024-05-28T06:04:09.885439+0000","last_unstale":"2024-05-28T06:04:11.890155+0000","last_undegraded":"2024-05-28T06:04:11.890155+0000","last_fullsized":"2024-05-28T06:04:11.890155+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:38:20.882939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.997783+0000","last_change":"2024-05-28T06:04:10.901609+0000","last_active":"2024-05-28T06:04:12.997783+0000","last_peered":"2024-05-28T06:04:12.997783+0000","last_clean":"2024-05-28T06:04:12.997783+0000","last_became_active":"2024-05-28T06:04:10.901302+0000","last_became_peered":"2024-05-28T06:04:10.901302+0000","last_unstale":"2024-05-28T06:04:12.997783+0000","last_undegraded":"2024-05-28T06:04:12.997783+0000","last_fullsized":"2024-05-28T06:04:12.997783+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:49:22.704773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.909384+0000","last_change":"2024-05-28T06:04:09.888169+0000","last_active":"2024-05-28T06:04:11.909384+0000","last_peered":"2024-05-28T06:04:11.909384+0000","last_clean":"2024-05-28T06:04:11.909384+0000","last_became_active":"2024-05-28T06:04:09.887867+0000","last_became_peered":"2024-05-28T06:04:09.887867+0000","last_unstale":"2024-05-28T06:04:11.909384+0000","last_undegraded":"2024-05-28T06:04:11.909384+0000","last_fullsized":"2024-05-28T06:04:11.909384+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:16:52.703842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.793993+0000","last_change":"2024-05-28T06:04:10.897718+0000","last_active":"2024-05-28T06:04:12.793993+0000","last_peered":"2024-05-28T06:04:12.793993+0000","last_clean":"2024-05-28T06:04:12.793993+0000","last_became_active":"2024-05-28T06:04:10.896414+0000","last_became_peered":"2024-05-28T06:04:10.896414+0000","last_unstale":"2024-05-28T06:04:12.793993+0000","last_undegraded":"2024-05-28T06:04:12.793993+0000","last_fullsized":"2024-05-28T06:04:12.793993+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:53:07.733019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.995167+0000","last_change":"2024-05-28T06:04:09.889726+0000","last_active":"2024-05-28T06:04:12.995167+0000","last_peered":"2024-05-28T06:04:12.995167+0000","last_clean":"2024-05-28T06:04:12.995167+0000","last_became_active":"2024-05-28T06:04:09.889394+0000","last_became_peered":"2024-05-28T06:04:09.889394+0000","last_unstale":"2024-05-28T06:04:12.995167+0000","last_undegraded":"2024-05-28T06:04:12.995167+0000","last_fullsized":"2024-05-28T06:04:12.995167+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:58:06.720498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.892869+0000","last_change":"2024-05-28T06:04:10.908922+0000","last_active":"2024-05-28T06:04:11.892869+0000","last_peered":"2024-05-28T06:04:11.892869+0000","last_clean":"2024-05-28T06:04:11.892869+0000","last_became_active":"2024-05-28T06:04:10.905738+0000","last_became_peered":"2024-05-28T06:04:10.905738+0000","last_unstale":"2024-05-28T06:04:11.892869+0000","last_undegraded":"2024-05-28T06:04:11.892869+0000","last_fullsized":"2024-05-28T06:04:11.892869+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:58:07.735330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.890426+0000","last_change":"2024-05-28T06:04:09.886430+0000","last_active":"2024-05-28T06:04:11.890426+0000","last_peered":"2024-05-28T06:04:11.890426+0000","last_clean":"2024-05-28T06:04:11.890426+0000","last_became_active":"2024-05-28T06:04:09.886072+0000","last_became_peered":"2024-05-28T06:04:09.886072+0000","last_unstale":"2024-05-28T06:04:11.890426+0000","last_undegraded":"2024-05-28T06:04:11.890426+0000","last_fullsized":"2024-05-28T06:04:11.890426+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:29:32.319083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.995901+0000","last_change":"2024-05-28T06:04:10.895054+0000","last_active":"2024-05-28T06:04:12.995901+0000","last_peered":"2024-05-28T06:04:12.995901+0000","last_clean":"2024-05-28T06:04:12.995901+0000","last_became_active":"2024-05-28T06:04:10.894384+0000","last_became_peered":"2024-05-28T06:04:10.894384+0000","last_unstale":"2024-05-28T06:04:12.995901+0000","last_undegraded":"2024-05-28T06:04:12.995901+0000","last_fullsized":"2024-05-28T06:04:12.995901+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:53:07.733019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.889822+0000","last_change":"2024-05-28T06:04:09.873566+0000","last_active":"2024-05-28T06:04:11.889822+0000","last_peered":"2024-05-28T06:04:11.889822+0000","last_clean":"2024-05-28T06:04:11.889822+0000","last_became_active":"2024-05-28T06:04:09.872928+0000","last_became_peered":"2024-05-28T06:04:09.872928+0000","last_unstale":"2024-05-28T06:04:11.889822+0000","last_undegraded":"2024-05-28T06:04:11.889822+0000","last_fullsized":"2024-05-28T06:04:11.889822+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:58:06.720498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.996688+0000","last_change":"2024-05-28T06:04:10.905645+0000","last_active":"2024-05-28T06:04:12.996688+0000","last_peered":"2024-05-28T06:04:12.996688+0000","last_clean":"2024-05-28T06:04:12.996688+0000","last_became_active":"2024-05-28T06:04:10.905308+0000","last_became_peered":"2024-05-28T06:04:10.905308+0000","last_unstale":"2024-05-28T06:04:12.996688+0000","last_undegraded":"2024-05-28T06:04:12.996688+0000","last_fullsized":"2024-05-28T06:04:12.996688+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:11:51.651640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.997139+0000","last_change":"2024-05-28T06:04:09.893691+0000","last_active":"2024-05-28T06:04:12.997139+0000","last_peered":"2024-05-28T06:04:12.997139+0000","last_clean":"2024-05-28T06:04:12.997139+0000","last_became_active":"2024-05-28T06:04:09.893416+0000","last_became_peered":"2024-05-28T06:04:09.893416+0000","last_unstale":"2024-05-28T06:04:12.997139+0000","last_undegraded":"2024-05-28T06:04:12.997139+0000","last_fullsized":"2024-05-28T06:04:12.997139+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:53:48.086901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.908760+0000","last_change":"2024-05-28T06:04:10.919265+0000","last_active":"2024-05-28T06:04:11.908760+0000","last_peered":"2024-05-28T06:04:11.908760+0000","last_clean":"2024-05-28T06:04:11.908760+0000","last_became_active":"2024-05-28T06:04:10.918268+0000","last_became_peered":"2024-05-28T06:04:10.918268+0000","last_unstale":"2024-05-28T06:04:11.908760+0000","last_undegraded":"2024-05-28T06:04:11.908760+0000","last_fullsized":"2024-05-28T06:04:11.908760+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:39:46.389531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.998229+0000","last_change":"2024-05-28T06:04:10.912331+0000","last_active":"2024-05-28T06:04:12.998229+0000","last_peered":"2024-05-28T06:04:12.998229+0000","last_clean":"2024-05-28T06:04:12.998229+0000","last_became_active":"2024-05-28T06:04:10.907887+0000","last_became_peered":"2024-05-28T06:04:10.907887+0000","last_unstale":"2024-05-28T06:04:12.998229+0000","last_undegraded":"2024-05-28T06:04:12.998229+0000","last_fullsized":"2024-05-28T06:04:12.998229+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:03:18.610996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.996617+0000","last_change":"2024-05-28T06:04:09.892486+0000","last_active":"2024-05-28T06:04:12.996617+0000","last_peered":"2024-05-28T06:04:12.996617+0000","last_clean":"2024-05-28T06:04:12.996617+0000","last_became_active":"2024-05-28T06:04:09.892160+0000","last_became_peered":"2024-05-28T06:04:09.892160+0000","last_unstale":"2024-05-28T06:04:12.996617+0000","last_undegraded":"2024-05-28T06:04:12.996617+0000","last_fullsized":"2024-05-28T06:04:12.996617+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:03:45.379665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.997770+0000","last_change":"2024-05-28T06:04:10.909079+0000","last_active":"2024-05-28T06:04:12.997770+0000","last_peered":"2024-05-28T06:04:12.997770+0000","last_clean":"2024-05-28T06:04:12.997770+0000","last_became_active":"2024-05-28T06:04:10.907979+0000","last_became_peered":"2024-05-28T06:04:10.907979+0000","last_unstale":"2024-05-28T06:04:12.997770+0000","last_undegraded":"2024-05-28T06:04:12.997770+0000","last_fullsized":"2024-05-28T06:04:12.997770+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:49:22.704773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.996620+0000","last_change":"2024-05-28T06:04:09.890388+0000","last_active":"2024-05-28T06:04:12.996620+0000","last_peered":"2024-05-28T06:04:12.996620+0000","last_clean":"2024-05-28T06:04:12.996620+0000","last_became_active":"2024-05-28T06:04:09.890086+0000","last_became_peered":"2024-05-28T06:04:09.890086+0000","last_unstale":"2024-05-28T06:04:12.996620+0000","last_undegraded":"2024-05-28T06:04:12.996620+0000","last_fullsized":"2024-05-28T06:04:12.996620+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:58:50.805034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.794239+0000","last_change":"2024-05-28T06:04:10.904802+0000","last_active":"2024-05-28T06:04:12.794239+0000","last_peered":"2024-05-28T06:04:12.794239+0000","last_clean":"2024-05-28T06:04:12.794239+0000","last_became_active":"2024-05-28T06:04:10.904438+0000","last_became_peered":"2024-05-28T06:04:10.904438+0000","last_unstale":"2024-05-28T06:04:12.794239+0000","last_undegraded":"2024-05-28T06:04:12.794239+0000","last_fullsized":"2024-05-28T06:04:12.794239+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:11:51.651640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.997391+0000","last_change":"2024-05-28T06:04:09.874413+0000","last_active":"2024-05-28T06:04:12.997391+0000","last_peered":"2024-05-28T06:04:12.997391+0000","last_clean":"2024-05-28T06:04:12.997391+0000","last_became_active":"2024-05-28T06:04:09.872117+0000","last_became_peered":"2024-05-28T06:04:09.872117+0000","last_unstale":"2024-05-28T06:04:12.997391+0000","last_undegraded":"2024-05-28T06:04:12.997391+0000","last_fullsized":"2024-05-28T06:04:12.997391+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:36:53.154893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.997324+0000","last_change":"2024-05-28T06:04:10.909242+0000","last_active":"2024-05-28T06:04:12.997324+0000","last_peered":"2024-05-28T06:04:12.997324+0000","last_clean":"2024-05-28T06:04:12.997324+0000","last_became_active":"2024-05-28T06:04:10.908934+0000","last_became_peered":"2024-05-28T06:04:10.908934+0000","last_unstale":"2024-05-28T06:04:12.997324+0000","last_undegraded":"2024-05-28T06:04:12.997324+0000","last_fullsized":"2024-05-28T06:04:12.997324+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:45:47.884578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.794420+0000","last_change":"2024-05-28T06:04:09.887679+0000","last_active":"2024-05-28T06:04:12.794420+0000","last_peered":"2024-05-28T06:04:12.794420+0000","last_clean":"2024-05-28T06:04:12.794420+0000","last_became_active":"2024-05-28T06:04:09.887386+0000","last_became_peered":"2024-05-28T06:04:09.887386+0000","last_unstale":"2024-05-28T06:04:12.794420+0000","last_undegraded":"2024-05-28T06:04:12.794420+0000","last_fullsized":"2024-05-28T06:04:12.794420+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:38:33.383965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574853,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":802816,"data_stored":661884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":208896,"data_stored":71024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":71060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":70934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":212992,"data_stored":71094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":811008,"data_stored":661940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-28T06:04:15.625 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-28T06:04:15.858 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.858+0000 7ff57b600640 1 Processor -- start 2024-05-28T06:04:15.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.858+0000 7ff57b600640 1 -- start start 2024-05-28T06:04:15.860 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.859+0000 7ff57b600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7ff5740d04a0 0x7ff5740d0930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:15.860 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.859+0000 7ff57b600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff5740ad130 0x7ff5740ad580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:15.860 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.859+0000 7ff57b600640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff57406ed10 con 0x7ff57406f860 2024-05-28T06:04:15.860 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.859+0000 7ff57b600640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7ff57415cb00 con 0x7ff5740d04a0 2024-05-28T06:04:15.860 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.859+0000 7ff57b600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ff5740b3cc0 con 0x7ff5740ad130 2024-05-28T06:04:15.860 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.860+0000 7ff578c00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7ff57406f860 0x7ff5740b3150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:55416/0 (socket says 172.21.15.96:55416) 2024-05-28T06:04:15.860 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.860+0000 7ff578c00640 1 -- 172.21.15.96:0/3238827392 learned_addr learned my addr 172.21.15.96:0/3238827392 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:04:15.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.860+0000 7ff573e00640 1 --2- 172.21.15.96:0/3238827392 >> v2:172.21.15.123:3300/0 conn(0x7ff5740d04a0 0x7ff5740d0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:15.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.860+0000 7ff573400640 1 -- 172.21.15.96:0/3238827392 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1865307648 0 0) 0x7ff568001110 con 0x7ff57406f860 2024-05-28T06:04:15.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.860+0000 7ff573400640 1 -- 172.21.15.96:0/3238827392 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff55c003520 con 0x7ff57406f860 2024-05-28T06:04:15.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.860+0000 7ff579600640 1 --2- 172.21.15.96:0/3238827392 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff5740ad130 0x7ff5740ad580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:15.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.860+0000 7ff573400640 1 -- 172.21.15.96:0/3238827392 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3602395569 0 0) 0x7ff55c003520 con 0x7ff57406f860 2024-05-28T06:04:15.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.861+0000 7ff573400640 1 -- 172.21.15.96:0/3238827392 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff5740ad130 msgr2=0x7ff5740ad580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:15.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.861+0000 7ff573400640 1 --2- 172.21.15.96:0/3238827392 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff5740ad130 0x7ff5740ad580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:15.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.861+0000 7ff573400640 1 -- 172.21.15.96:0/3238827392 >> v2:172.21.15.123:3300/0 conn(0x7ff5740d04a0 msgr2=0x7ff5740d0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:15.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.861+0000 7ff573400640 1 --2- 172.21.15.96:0/3238827392 >> v2:172.21.15.123:3300/0 conn(0x7ff5740d04a0 0x7ff5740d0930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:15.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.861+0000 7ff573400640 1 -- 172.21.15.96:0/3238827392 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5740b3fa0 con 0x7ff57406f860 2024-05-28T06:04:15.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.861+0000 7ff573400640 1 -- 172.21.15.96:0/3238827392 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7ff568001840 con 0x7ff57406f860 2024-05-28T06:04:15.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.861+0000 7ff573400640 1 -- 172.21.15.96:0/3238827392 >> v1:172.21.15.96:6789/0 conn(0x7ff57406f860 legacy=0x7ff5740b3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:15.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.861+0000 7ff573400640 1 --2- 172.21.15.96:0/3238827392 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff55c0039b0 0x7ff55c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:15.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.861+0000 7ff573400640 1 --2- 172.21.15.96:0/3238827392 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff55c004800 0x7ff55c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:15.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.861+0000 7ff573400640 1 --2- 172.21.15.96:0/3238827392 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff5740ad130 0x7ff55c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:15.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.861+0000 7ff573400640 1 -- 172.21.15.96:0/3238827392 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff57415cb00 con 0x7ff55c0039b0 2024-05-28T06:04:15.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.861+0000 7ff573400640 1 -- 172.21.15.96:0/3238827392 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff5740b3cc0 con 0x7ff55c004800 2024-05-28T06:04:15.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.861+0000 7ff573400640 1 -- 172.21.15.96:0/3238827392 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ff55c00b9c0 con 0x7ff5740ad130 2024-05-28T06:04:15.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.861+0000 7ff578c00640 1 --2- 172.21.15.96:0/3238827392 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff55c0039b0 0x7ff55c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:15.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.861+0000 7ff573e00640 1 --2- 172.21.15.96:0/3238827392 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff55c004800 0x7ff55c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:15.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.861+0000 7ff578c00640 1 -- 172.21.15.96:0/3238827392 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff5740ad130 msgr2=0x7ff55c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:15.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.861+0000 7ff579600640 1 --2- 172.21.15.96:0/3238827392 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff5740ad130 0x7ff55c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:15.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.861+0000 7ff578c00640 1 --2- 172.21.15.96:0/3238827392 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff5740ad130 0x7ff55c0051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:15.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.861+0000 7ff578c00640 1 -- 172.21.15.96:0/3238827392 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff55c004800 msgr2=0x7ff55c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:15.865 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.861+0000 7ff578c00640 1 --2- 172.21.15.96:0/3238827392 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff55c004800 0x7ff55c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:15.865 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.862+0000 7ff578c00640 1 -- 172.21.15.96:0/3238827392 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff55c003520 con 0x7ff55c0039b0 2024-05-28T06:04:15.865 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.862+0000 7ff573e00640 1 --2- 172.21.15.96:0/3238827392 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff55c004800 0x7ff55c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:04:15.867 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.862+0000 7ff578c00640 1 --2- 172.21.15.96:0/3238827392 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff55c0039b0 0x7ff55c003dd0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff56800c2f0 tx=0x7ff55c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9bb76b3bdf189cd server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:15.867 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.863+0000 7ff573400640 1 -- 172.21.15.96:0/3238827392 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff568035070 con 0x7ff55c0039b0 2024-05-28T06:04:15.867 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.863+0000 7ff573400640 1 -- 172.21.15.96:0/3238827392 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff568005ad0 con 0x7ff55c0039b0 2024-05-28T06:04:15.867 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.863+0000 7ff573400640 1 -- 172.21.15.96:0/3238827392 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff568007e60 con 0x7ff55c0039b0 2024-05-28T06:04:15.867 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.863+0000 7ff57b600640 1 -- 172.21.15.96:0/3238827392 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff55c0039b0 msgr2=0x7ff55c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:15.867 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.863+0000 7ff57b600640 1 --2- 172.21.15.96:0/3238827392 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff55c0039b0 0x7ff55c003dd0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff56800c2f0 tx=0x7ff55c005830 comp rx=0 tx=0).stop 2024-05-28T06:04:15.867 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.863+0000 7ff579600640 1 -- 172.21.15.96:0/3238827392 reap_dead start 2024-05-28T06:04:15.867 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.864+0000 7ff57b600640 1 -- 172.21.15.96:0/3238827392 shutdown_connections 2024-05-28T06:04:15.867 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.864+0000 7ff57b600640 1 -- 172.21.15.96:0/3238827392 >> 172.21.15.96:0/3238827392 conn(0x7ff57406bae0 msgr2=0x7ff57406bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:15.867 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.864+0000 7ff57b600640 1 -- 172.21.15.96:0/3238827392 shutdown_connections 2024-05-28T06:04:15.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.864+0000 7ff57b600640 1 -- 172.21.15.96:0/3238827392 wait complete. 2024-05-28T06:04:15.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.865+0000 7ff57b600640 1 Processor -- start 2024-05-28T06:04:15.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.866+0000 7ff57b600640 1 -- start start 2024-05-28T06:04:15.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.866+0000 7ff57b600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff55c0039b0 0x7ff57415f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:15.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.866+0000 7ff57b600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff57415cd10 0x7ff5741f3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:15.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.866+0000 7ff57b600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff5741f4410 0x7ff5740c8d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:15.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.866+0000 7ff57b600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff5740b4840 con 0x7ff57415cd10 2024-05-28T06:04:15.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.866+0000 7ff57b600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff5740b4560 con 0x7ff55c0039b0 2024-05-28T06:04:15.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.867+0000 7ff57b600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ff5740b3860 con 0x7ff5741f4410 2024-05-28T06:04:15.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.867+0000 7ff573e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff57415cd10 0x7ff5741f3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:15.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.867+0000 7ff573e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff57415cd10 0x7ff5741f3ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:41512/0 (socket says 172.21.15.96:41512) 2024-05-28T06:04:15.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.867+0000 7ff573e00640 1 -- 172.21.15.96:0/45991818 learned_addr learned my addr 172.21.15.96:0/45991818 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:15.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.867+0000 7ff578c00640 1 --2- 172.21.15.96:0/45991818 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff55c0039b0 0x7ff57415f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:15.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.867+0000 7ff573e00640 1 -- 172.21.15.96:0/45991818 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff5741f4410 msgr2=0x7ff5740c8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:15.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.867+0000 7ff579600640 1 --2- 172.21.15.96:0/45991818 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff5741f4410 0x7ff5740c8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:15.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.867+0000 7ff573e00640 1 --2- 172.21.15.96:0/45991818 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff5741f4410 0x7ff5740c8d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:15.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.867+0000 7ff573e00640 1 -- 172.21.15.96:0/45991818 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff55c0039b0 msgr2=0x7ff57415f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:15.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.867+0000 7ff573e00640 1 --2- 172.21.15.96:0/45991818 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff55c0039b0 0x7ff57415f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:15.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.867+0000 7ff573e00640 1 -- 172.21.15.96:0/45991818 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff57415d580 con 0x7ff57415cd10 2024-05-28T06:04:15.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.867+0000 7ff578c00640 1 --2- 172.21.15.96:0/45991818 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff55c0039b0 0x7ff57415f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:04:15.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.867+0000 7ff573e00640 1 --2- 172.21.15.96:0/45991818 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff57415cd10 0x7ff5741f3ed0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff5600109d0 tx=0x7ff560010e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:15.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.868+0000 7ff570c00640 1 -- 172.21.15.96:0/45991818 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff56000f040 con 0x7ff57415cd10 2024-05-28T06:04:15.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.868+0000 7ff570c00640 1 -- 172.21.15.96:0/45991818 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff56000ec20 con 0x7ff57415cd10 2024-05-28T06:04:15.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.868+0000 7ff570c00640 1 -- 172.21.15.96:0/45991818 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff56001c400 con 0x7ff57415cd10 2024-05-28T06:04:15.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.868+0000 7ff57b600640 1 -- 172.21.15.96:0/45991818 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff57415d860 con 0x7ff57415cd10 2024-05-28T06:04:15.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.868+0000 7ff57b600640 1 -- 172.21.15.96:0/45991818 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff57415de20 con 0x7ff57415cd10 2024-05-28T06:04:15.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.869+0000 7ff57b600640 1 -- 172.21.15.96:0/45991818 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff57406dc10 con 0x7ff57415cd10 2024-05-28T06:04:15.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.870+0000 7ff570c00640 1 -- 172.21.15.96:0/45991818 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff56001c5a0 con 0x7ff57415cd10 2024-05-28T06:04:15.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.871+0000 7ff570c00640 1 --2- 172.21.15.96:0/45991818 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff55406c8f0 0x7ff55406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:15.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.871+0000 7ff570c00640 1 -- 172.21.15.96:0/45991818 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7ff56008c200 con 0x7ff57415cd10 2024-05-28T06:04:15.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.871+0000 7ff578c00640 1 --2- 172.21.15.96:0/45991818 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff55406c8f0 0x7ff55406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:15.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.872+0000 7ff578c00640 1 --2- 172.21.15.96:0/45991818 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff55406c8f0 0x7ff55406edb0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff568032000 tx=0x7ff568009030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:15.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:15.875+0000 7ff570c00640 1 -- 172.21.15.96:0/45991818 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff57406dc10 con 0x7ff57415cd10 2024-05-28T06:04:16.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.072+0000 7ff57b600640 1 -- 172.21.15.96:0/45991818 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff574068240 con 0x7ff55406c8f0 2024-05-28T06:04:16.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.093+0000 7ff570c00640 1 -- 172.21.15.96:0/45991818 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251948 (secure 0 0 0) 0x7ff574068240 con 0x7ff55406c8f0 2024-05-28T06:04:16.095 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:04:16.098 INFO:teuthology.orchestra.run.smithi096.stderr:dumped all 2024-05-28T06:04:16.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.099+0000 7ff57b600640 1 -- 172.21.15.96:0/45991818 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff55406c8f0 msgr2=0x7ff55406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:16.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.099+0000 7ff57b600640 1 --2- 172.21.15.96:0/45991818 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff55406c8f0 0x7ff55406edb0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff568032000 tx=0x7ff568009030 comp rx=0 tx=0).stop 2024-05-28T06:04:16.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.099+0000 7ff57b600640 1 -- 172.21.15.96:0/45991818 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff57415cd10 msgr2=0x7ff5741f3ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:16.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.099+0000 7ff57b600640 1 --2- 172.21.15.96:0/45991818 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff57415cd10 0x7ff5741f3ed0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff5600109d0 tx=0x7ff560010e90 comp rx=0 tx=0).stop 2024-05-28T06:04:16.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.099+0000 7ff57b600640 1 -- 172.21.15.96:0/45991818 shutdown_connections 2024-05-28T06:04:16.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.099+0000 7ff57b600640 1 --2- 172.21.15.96:0/45991818 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff55406c8f0 0x7ff55406edb0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:16.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.099+0000 7ff57b600640 1 --2- 172.21.15.96:0/45991818 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff5741f4410 0x7ff5740c8d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:16.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.099+0000 7ff57b600640 1 --2- 172.21.15.96:0/45991818 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff57415cd10 0x7ff5741f3ed0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:16.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.099+0000 7ff57b600640 1 --2- 172.21.15.96:0/45991818 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff55c0039b0 0x7ff57415f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:16.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.099+0000 7ff57b600640 1 -- 172.21.15.96:0/45991818 >> 172.21.15.96:0/45991818 conn(0x7ff57406bae0 msgr2=0x7ff5740cb590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:16.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.099+0000 7ff57b600640 1 -- 172.21.15.96:0/45991818 shutdown_connections 2024-05-28T06:04:16.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.100+0000 7ff57b600640 1 -- 172.21.15.96:0/45991818 wait complete. 2024-05-28T06:04:16.118 INFO:teuthology.orchestra.run.smithi096.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-05-28T06:04:15.483643+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":185,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218992,"kb_used_data":2824,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755728,"statfs":{"total":773094113280,"available":772869865472,"internally_reserved":0,"allocated":2891776,"data_stored":1753028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001905"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.997911+0000","last_change":"2024-05-28T06:04:10.916483+0000","last_active":"2024-05-28T06:04:12.997911+0000","last_peered":"2024-05-28T06:04:12.997911+0000","last_clean":"2024-05-28T06:04:12.997911+0000","last_became_active":"2024-05-28T06:04:10.916214+0000","last_became_peered":"2024-05-28T06:04:10.916214+0000","last_unstale":"2024-05-28T06:04:12.997911+0000","last_undegraded":"2024-05-28T06:04:12.997911+0000","last_fullsized":"2024-05-28T06:04:12.997911+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:43:05.061767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.889968+0000","last_change":"2024-05-28T06:04:10.900791+0000","last_active":"2024-05-28T06:04:11.889968+0000","last_peered":"2024-05-28T06:04:11.889968+0000","last_clean":"2024-05-28T06:04:11.889968+0000","last_became_active":"2024-05-28T06:04:10.900489+0000","last_became_peered":"2024-05-28T06:04:10.900489+0000","last_unstale":"2024-05-28T06:04:11.889968+0000","last_undegraded":"2024-05-28T06:04:11.889968+0000","last_fullsized":"2024-05-28T06:04:11.889968+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:53:07.733019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.794222+0000","last_change":"2024-05-28T06:04:10.901559+0000","last_active":"2024-05-28T06:04:12.794222+0000","last_peered":"2024-05-28T06:04:12.794222+0000","last_clean":"2024-05-28T06:04:12.794222+0000","last_became_active":"2024-05-28T06:04:10.899022+0000","last_became_peered":"2024-05-28T06:04:10.899022+0000","last_unstale":"2024-05-28T06:04:12.794222+0000","last_undegraded":"2024-05-28T06:04:12.794222+0000","last_fullsized":"2024-05-28T06:04:12.794222+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:28:21.123299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.997115+0000","last_change":"2024-05-28T06:04:10.923988+0000","last_active":"2024-05-28T06:04:12.997115+0000","last_peered":"2024-05-28T06:04:12.997115+0000","last_clean":"2024-05-28T06:04:12.997115+0000","last_became_active":"2024-05-28T06:04:10.923718+0000","last_became_peered":"2024-05-28T06:04:10.923718+0000","last_unstale":"2024-05-28T06:04:12.997115+0000","last_undegraded":"2024-05-28T06:04:12.997115+0000","last_fullsized":"2024-05-28T06:04:12.997115+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:45:47.884578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.996357+0000","last_change":"2024-05-28T06:04:10.917922+0000","last_active":"2024-05-28T06:04:12.996357+0000","last_peered":"2024-05-28T06:04:12.996357+0000","last_clean":"2024-05-28T06:04:12.996357+0000","last_became_active":"2024-05-28T06:04:10.917253+0000","last_became_peered":"2024-05-28T06:04:10.917253+0000","last_unstale":"2024-05-28T06:04:12.996357+0000","last_undegraded":"2024-05-28T06:04:12.996357+0000","last_fullsized":"2024-05-28T06:04:12.996357+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:04:49.415234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.996757+0000","last_change":"2024-05-28T06:04:10.907704+0000","last_active":"2024-05-28T06:04:12.996757+0000","last_peered":"2024-05-28T06:04:12.996757+0000","last_clean":"2024-05-28T06:04:12.996757+0000","last_became_active":"2024-05-28T06:04:10.907065+0000","last_became_peered":"2024-05-28T06:04:10.907065+0000","last_unstale":"2024-05-28T06:04:12.996757+0000","last_undegraded":"2024-05-28T06:04:12.996757+0000","last_fullsized":"2024-05-28T06:04:12.996757+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:28:21.123299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.995527+0000","last_change":"2024-05-28T06:04:10.905438+0000","last_active":"2024-05-28T06:04:12.995527+0000","last_peered":"2024-05-28T06:04:12.995527+0000","last_clean":"2024-05-28T06:04:12.995527+0000","last_became_active":"2024-05-28T06:04:10.904842+0000","last_became_peered":"2024-05-28T06:04:10.904842+0000","last_unstale":"2024-05-28T06:04:12.995527+0000","last_undegraded":"2024-05-28T06:04:12.995527+0000","last_fullsized":"2024-05-28T06:04:12.995527+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:46:27.923407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.995330+0000","last_change":"2024-05-28T06:04:10.899401+0000","last_active":"2024-05-28T06:04:12.995330+0000","last_peered":"2024-05-28T06:04:12.995330+0000","last_clean":"2024-05-28T06:04:12.995330+0000","last_became_active":"2024-05-28T06:04:10.898742+0000","last_became_peered":"2024-05-28T06:04:10.898742+0000","last_unstale":"2024-05-28T06:04:12.995330+0000","last_undegraded":"2024-05-28T06:04:12.995330+0000","last_fullsized":"2024-05-28T06:04:12.995330+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:39:46.389531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.997773+0000","last_change":"2024-05-28T06:04:10.907973+0000","last_active":"2024-05-28T06:04:12.997773+0000","last_peered":"2024-05-28T06:04:12.997773+0000","last_clean":"2024-05-28T06:04:12.997773+0000","last_became_active":"2024-05-28T06:04:10.905186+0000","last_became_peered":"2024-05-28T06:04:10.905186+0000","last_unstale":"2024-05-28T06:04:12.997773+0000","last_undegraded":"2024-05-28T06:04:12.997773+0000","last_fullsized":"2024-05-28T06:04:12.997773+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:39:13.976770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.909549+0000","last_change":"2024-05-28T06:04:10.927703+0000","last_active":"2024-05-28T06:04:11.909549+0000","last_peered":"2024-05-28T06:04:11.909549+0000","last_clean":"2024-05-28T06:04:11.909549+0000","last_became_active":"2024-05-28T06:04:10.927383+0000","last_became_peered":"2024-05-28T06:04:10.927383+0000","last_unstale":"2024-05-28T06:04:11.909549+0000","last_undegraded":"2024-05-28T06:04:11.909549+0000","last_fullsized":"2024-05-28T06:04:11.909549+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:11:51.651640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.909515+0000","last_change":"2024-05-28T06:04:10.926573+0000","last_active":"2024-05-28T06:04:11.909515+0000","last_peered":"2024-05-28T06:04:11.909515+0000","last_clean":"2024-05-28T06:04:11.909515+0000","last_became_active":"2024-05-28T06:04:10.926364+0000","last_became_peered":"2024-05-28T06:04:10.926364+0000","last_unstale":"2024-05-28T06:04:11.909515+0000","last_undegraded":"2024-05-28T06:04:11.909515+0000","last_fullsized":"2024-05-28T06:04:11.909515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:28:21.123299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.893744+0000","last_change":"2024-05-28T06:04:10.913441+0000","last_active":"2024-05-28T06:04:11.893744+0000","last_peered":"2024-05-28T06:04:11.893744+0000","last_clean":"2024-05-28T06:04:11.893744+0000","last_became_active":"2024-05-28T06:04:10.912847+0000","last_became_peered":"2024-05-28T06:04:10.912847+0000","last_unstale":"2024-05-28T06:04:11.893744+0000","last_undegraded":"2024-05-28T06:04:11.893744+0000","last_fullsized":"2024-05-28T06:04:11.893744+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:58:51.819866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.996503+0000","last_change":"2024-05-28T06:04:10.909268+0000","last_active":"2024-05-28T06:04:12.996503+0000","last_peered":"2024-05-28T06:04:12.996503+0000","last_clean":"2024-05-28T06:04:12.996503+0000","last_became_active":"2024-05-28T06:04:10.908954+0000","last_became_peered":"2024-05-28T06:04:10.908954+0000","last_unstale":"2024-05-28T06:04:12.996503+0000","last_undegraded":"2024-05-28T06:04:12.996503+0000","last_fullsized":"2024-05-28T06:04:12.996503+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:28:21.123299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.793137+0000","last_change":"2024-05-28T06:04:10.899509+0000","last_active":"2024-05-28T06:04:12.793137+0000","last_peered":"2024-05-28T06:04:12.793137+0000","last_clean":"2024-05-28T06:04:12.793137+0000","last_became_active":"2024-05-28T06:04:10.896990+0000","last_became_peered":"2024-05-28T06:04:10.896990+0000","last_unstale":"2024-05-28T06:04:12.793137+0000","last_undegraded":"2024-05-28T06:04:12.793137+0000","last_fullsized":"2024-05-28T06:04:12.793137+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:17:36.664466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.890266+0000","last_change":"2024-05-28T06:04:10.906808+0000","last_active":"2024-05-28T06:04:11.890266+0000","last_peered":"2024-05-28T06:04:11.890266+0000","last_clean":"2024-05-28T06:04:11.890266+0000","last_became_active":"2024-05-28T06:04:10.906579+0000","last_became_peered":"2024-05-28T06:04:10.906579+0000","last_unstale":"2024-05-28T06:04:11.890266+0000","last_undegraded":"2024-05-28T06:04:11.890266+0000","last_fullsized":"2024-05-28T06:04:11.890266+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:28:21.123299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.997630+0000","last_change":"2024-05-28T06:04:10.904806+0000","last_active":"2024-05-28T06:04:12.997630+0000","last_peered":"2024-05-28T06:04:12.997630+0000","last_clean":"2024-05-28T06:04:12.997630+0000","last_became_active":"2024-05-28T06:04:10.904450+0000","last_became_peered":"2024-05-28T06:04:10.904450+0000","last_unstale":"2024-05-28T06:04:12.997630+0000","last_undegraded":"2024-05-28T06:04:12.997630+0000","last_fullsized":"2024-05-28T06:04:12.997630+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:51:12.839226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.999350+0000","last_change":"2024-05-28T06:04:10.918376+0000","last_active":"2024-05-28T06:04:12.999350+0000","last_peered":"2024-05-28T06:04:12.999350+0000","last_clean":"2024-05-28T06:04:12.999350+0000","last_became_active":"2024-05-28T06:04:10.918042+0000","last_became_peered":"2024-05-28T06:04:10.918042+0000","last_unstale":"2024-05-28T06:04:12.999350+0000","last_undegraded":"2024-05-28T06:04:12.999350+0000","last_fullsized":"2024-05-28T06:04:12.999350+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:39:13.976770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.996501+0000","last_change":"2024-05-28T06:04:10.910995+0000","last_active":"2024-05-28T06:04:12.996501+0000","last_peered":"2024-05-28T06:04:12.996501+0000","last_clean":"2024-05-28T06:04:12.996501+0000","last_became_active":"2024-05-28T06:04:10.910672+0000","last_became_peered":"2024-05-28T06:04:10.910672+0000","last_unstale":"2024-05-28T06:04:12.996501+0000","last_undegraded":"2024-05-28T06:04:12.996501+0000","last_fullsized":"2024-05-28T06:04:12.996501+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:58:51.819866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.997411+0000","last_change":"2024-05-28T06:04:10.911244+0000","last_active":"2024-05-28T06:04:12.997411+0000","last_peered":"2024-05-28T06:04:12.997411+0000","last_clean":"2024-05-28T06:04:12.997411+0000","last_became_active":"2024-05-28T06:04:10.910921+0000","last_became_peered":"2024-05-28T06:04:10.910921+0000","last_unstale":"2024-05-28T06:04:12.997411+0000","last_undegraded":"2024-05-28T06:04:12.997411+0000","last_fullsized":"2024-05-28T06:04:12.997411+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:05:14.695994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.909154+0000","last_change":"2024-05-28T06:04:10.921344+0000","last_active":"2024-05-28T06:04:11.909154+0000","last_peered":"2024-05-28T06:04:11.909154+0000","last_clean":"2024-05-28T06:04:11.909154+0000","last_became_active":"2024-05-28T06:04:10.920438+0000","last_became_peered":"2024-05-28T06:04:10.920438+0000","last_unstale":"2024-05-28T06:04:11.909154+0000","last_undegraded":"2024-05-28T06:04:11.909154+0000","last_fullsized":"2024-05-28T06:04:11.909154+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:53:07.733019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.793512+0000","last_change":"2024-05-28T06:04:10.902770+0000","last_active":"2024-05-28T06:04:12.793512+0000","last_peered":"2024-05-28T06:04:12.793512+0000","last_clean":"2024-05-28T06:04:12.793512+0000","last_became_active":"2024-05-28T06:04:10.901534+0000","last_became_peered":"2024-05-28T06:04:10.901534+0000","last_unstale":"2024-05-28T06:04:12.793512+0000","last_undegraded":"2024-05-28T06:04:12.793512+0000","last_fullsized":"2024-05-28T06:04:12.793512+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:00:40.966129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.996957+0000","last_change":"2024-05-28T06:04:10.915708+0000","last_active":"2024-05-28T06:04:12.996957+0000","last_peered":"2024-05-28T06:04:12.996957+0000","last_clean":"2024-05-28T06:04:12.996957+0000","last_became_active":"2024-05-28T06:04:10.913058+0000","last_became_peered":"2024-05-28T06:04:10.913058+0000","last_unstale":"2024-05-28T06:04:12.996957+0000","last_undegraded":"2024-05-28T06:04:12.996957+0000","last_fullsized":"2024-05-28T06:04:12.996957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:11:51.651640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.995352+0000","last_change":"2024-05-28T06:04:10.904255+0000","last_active":"2024-05-28T06:04:12.995352+0000","last_peered":"2024-05-28T06:04:12.995352+0000","last_clean":"2024-05-28T06:04:12.995352+0000","last_became_active":"2024-05-28T06:04:10.903840+0000","last_became_peered":"2024-05-28T06:04:10.903840+0000","last_unstale":"2024-05-28T06:04:12.995352+0000","last_undegraded":"2024-05-28T06:04:12.995352+0000","last_fullsized":"2024-05-28T06:04:12.995352+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:06:44.034610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.997441+0000","last_change":"2024-05-28T06:04:10.902458+0000","last_active":"2024-05-28T06:04:12.997441+0000","last_peered":"2024-05-28T06:04:12.997441+0000","last_clean":"2024-05-28T06:04:12.997441+0000","last_became_active":"2024-05-28T06:04:10.902054+0000","last_became_peered":"2024-05-28T06:04:10.902054+0000","last_unstale":"2024-05-28T06:04:12.997441+0000","last_undegraded":"2024-05-28T06:04:12.997441+0000","last_fullsized":"2024-05-28T06:04:12.997441+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:46:27.923407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.997558+0000","last_change":"2024-05-28T06:04:10.895892+0000","last_active":"2024-05-28T06:04:12.997558+0000","last_peered":"2024-05-28T06:04:12.997558+0000","last_clean":"2024-05-28T06:04:12.997558+0000","last_became_active":"2024-05-28T06:04:10.895579+0000","last_became_peered":"2024-05-28T06:04:10.895579+0000","last_unstale":"2024-05-28T06:04:12.997558+0000","last_undegraded":"2024-05-28T06:04:12.997558+0000","last_fullsized":"2024-05-28T06:04:12.997558+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:51:12.839226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.909428+0000","last_change":"2024-05-28T06:04:10.914830+0000","last_active":"2024-05-28T06:04:11.909428+0000","last_peered":"2024-05-28T06:04:11.909428+0000","last_clean":"2024-05-28T06:04:11.909428+0000","last_became_active":"2024-05-28T06:04:10.913651+0000","last_became_peered":"2024-05-28T06:04:10.913651+0000","last_unstale":"2024-05-28T06:04:11.909428+0000","last_undegraded":"2024-05-28T06:04:11.909428+0000","last_fullsized":"2024-05-28T06:04:11.909428+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:46:29.255545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.889962+0000","last_change":"2024-05-28T06:04:10.903385+0000","last_active":"2024-05-28T06:04:11.889962+0000","last_peered":"2024-05-28T06:04:11.889962+0000","last_clean":"2024-05-28T06:04:11.889962+0000","last_became_active":"2024-05-28T06:04:10.903078+0000","last_became_peered":"2024-05-28T06:04:10.903078+0000","last_unstale":"2024-05-28T06:04:11.889962+0000","last_undegraded":"2024-05-28T06:04:11.889962+0000","last_fullsized":"2024-05-28T06:04:11.889962+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:06:44.034610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.997438+0000","last_change":"2024-05-28T06:04:10.894062+0000","last_active":"2024-05-28T06:04:12.997438+0000","last_peered":"2024-05-28T06:04:12.997438+0000","last_clean":"2024-05-28T06:04:12.997438+0000","last_became_active":"2024-05-28T06:04:10.893763+0000","last_became_peered":"2024-05-28T06:04:10.893763+0000","last_unstale":"2024-05-28T06:04:12.997438+0000","last_undegraded":"2024-05-28T06:04:12.997438+0000","last_fullsized":"2024-05-28T06:04:12.997438+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:46:27.923407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.793745+0000","last_change":"2024-05-28T06:04:10.897684+0000","last_active":"2024-05-28T06:04:12.793745+0000","last_peered":"2024-05-28T06:04:12.793745+0000","last_clean":"2024-05-28T06:04:12.793745+0000","last_became_active":"2024-05-28T06:04:10.897382+0000","last_became_peered":"2024-05-28T06:04:10.897382+0000","last_unstale":"2024-05-28T06:04:12.793745+0000","last_undegraded":"2024-05-28T06:04:12.793745+0000","last_fullsized":"2024-05-28T06:04:12.793745+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:39:46.389531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.997085+0000","last_change":"2024-05-28T06:04:10.908012+0000","last_active":"2024-05-28T06:04:12.997085+0000","last_peered":"2024-05-28T06:04:12.997085+0000","last_clean":"2024-05-28T06:04:12.997085+0000","last_became_active":"2024-05-28T06:04:10.907436+0000","last_became_peered":"2024-05-28T06:04:10.907436+0000","last_unstale":"2024-05-28T06:04:12.997085+0000","last_undegraded":"2024-05-28T06:04:12.997085+0000","last_fullsized":"2024-05-28T06:04:12.997085+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:58:16.009219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.996141+0000","last_change":"2024-05-28T06:04:10.905729+0000","last_active":"2024-05-28T06:04:12.996141+0000","last_peered":"2024-05-28T06:04:12.996141+0000","last_clean":"2024-05-28T06:04:12.996141+0000","last_became_active":"2024-05-28T06:04:10.905325+0000","last_became_peered":"2024-05-28T06:04:10.905325+0000","last_unstale":"2024-05-28T06:04:12.996141+0000","last_undegraded":"2024-05-28T06:04:12.996141+0000","last_fullsized":"2024-05-28T06:04:12.996141+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:03:46.394497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.908750+0000","last_change":"2024-05-28T06:04:10.915117+0000","last_active":"2024-05-28T06:04:11.908750+0000","last_peered":"2024-05-28T06:04:11.908750+0000","last_clean":"2024-05-28T06:04:11.908750+0000","last_became_active":"2024-05-28T06:04:10.914761+0000","last_became_peered":"2024-05-28T06:04:10.914761+0000","last_unstale":"2024-05-28T06:04:11.908750+0000","last_undegraded":"2024-05-28T06:04:11.908750+0000","last_fullsized":"2024-05-28T06:04:11.908750+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:00:40.966129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.892804+0000","last_change":"2024-05-28T06:04:10.897395+0000","last_active":"2024-05-28T06:04:11.892804+0000","last_peered":"2024-05-28T06:04:11.892804+0000","last_clean":"2024-05-28T06:04:11.892804+0000","last_became_active":"2024-05-28T06:04:10.897108+0000","last_became_peered":"2024-05-28T06:04:10.897108+0000","last_unstale":"2024-05-28T06:04:11.892804+0000","last_undegraded":"2024-05-28T06:04:11.892804+0000","last_fullsized":"2024-05-28T06:04:11.892804+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:16:53.718673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.996650+0000","last_change":"2024-05-28T06:04:09.890037+0000","last_active":"2024-05-28T06:04:12.996650+0000","last_peered":"2024-05-28T06:04:12.996650+0000","last_clean":"2024-05-28T06:04:12.996650+0000","last_became_active":"2024-05-28T06:04:09.889696+0000","last_became_peered":"2024-05-28T06:04:09.889696+0000","last_unstale":"2024-05-28T06:04:12.996650+0000","last_undegraded":"2024-05-28T06:04:12.996650+0000","last_fullsized":"2024-05-28T06:04:12.996650+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:29:32.319083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.794133+0000","last_change":"2024-05-28T06:04:10.896136+0000","last_active":"2024-05-28T06:04:12.794133+0000","last_peered":"2024-05-28T06:04:12.794133+0000","last_clean":"2024-05-28T06:04:12.794133+0000","last_became_active":"2024-05-28T06:04:10.895631+0000","last_became_peered":"2024-05-28T06:04:10.895631+0000","last_unstale":"2024-05-28T06:04:12.794133+0000","last_undegraded":"2024-05-28T06:04:12.794133+0000","last_fullsized":"2024-05-28T06:04:12.794133+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:46:29.255545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.908566+0000","last_change":"2024-05-28T06:04:09.879488+0000","last_active":"2024-05-28T06:04:11.908566+0000","last_peered":"2024-05-28T06:04:11.908566+0000","last_clean":"2024-05-28T06:04:11.908566+0000","last_became_active":"2024-05-28T06:04:09.879182+0000","last_became_peered":"2024-05-28T06:04:09.879182+0000","last_unstale":"2024-05-28T06:04:11.908566+0000","last_undegraded":"2024-05-28T06:04:11.908566+0000","last_fullsized":"2024-05-28T06:04:11.908566+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:29:32.319083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.995389+0000","last_change":"2024-05-28T06:04:09.884424+0000","last_active":"2024-05-28T06:04:12.995389+0000","last_peered":"2024-05-28T06:04:12.995389+0000","last_clean":"2024-05-28T06:04:12.995389+0000","last_became_active":"2024-05-28T06:04:09.884088+0000","last_became_peered":"2024-05-28T06:04:09.884088+0000","last_unstale":"2024-05-28T06:04:12.995389+0000","last_undegraded":"2024-05-28T06:04:12.995389+0000","last_fullsized":"2024-05-28T06:04:12.995389+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:27:30.379288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.996856+0000","last_change":"2024-05-28T06:04:10.890237+0000","last_active":"2024-05-28T06:04:12.996856+0000","last_peered":"2024-05-28T06:04:12.996856+0000","last_clean":"2024-05-28T06:04:12.996856+0000","last_became_active":"2024-05-28T06:04:10.889644+0000","last_became_peered":"2024-05-28T06:04:10.889644+0000","last_unstale":"2024-05-28T06:04:12.996856+0000","last_undegraded":"2024-05-28T06:04:12.996856+0000","last_fullsized":"2024-05-28T06:04:12.996856+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:58:16.009219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.997564+0000","last_change":"2024-05-28T06:04:09.875990+0000","last_active":"2024-05-28T06:04:12.997564+0000","last_peered":"2024-05-28T06:04:12.997564+0000","last_clean":"2024-05-28T06:04:12.997564+0000","last_became_active":"2024-05-28T06:04:09.875644+0000","last_became_peered":"2024-05-28T06:04:09.875644+0000","last_unstale":"2024-05-28T06:04:12.997564+0000","last_undegraded":"2024-05-28T06:04:12.997564+0000","last_fullsized":"2024-05-28T06:04:12.997564+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:27:30.379288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.889859+0000","last_change":"2024-05-28T06:04:10.899922+0000","last_active":"2024-05-28T06:04:11.889859+0000","last_peered":"2024-05-28T06:04:11.889859+0000","last_clean":"2024-05-28T06:04:11.889859+0000","last_became_active":"2024-05-28T06:04:10.899617+0000","last_became_peered":"2024-05-28T06:04:10.899617+0000","last_unstale":"2024-05-28T06:04:11.889859+0000","last_undegraded":"2024-05-28T06:04:11.889859+0000","last_fullsized":"2024-05-28T06:04:11.889859+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:39:46.389531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.909050+0000","last_change":"2024-05-28T06:04:09.887084+0000","last_active":"2024-05-28T06:04:11.909050+0000","last_peered":"2024-05-28T06:04:11.909050+0000","last_clean":"2024-05-28T06:04:11.909050+0000","last_became_active":"2024-05-28T06:04:09.886821+0000","last_became_peered":"2024-05-28T06:04:09.886821+0000","last_unstale":"2024-05-28T06:04:11.909050+0000","last_undegraded":"2024-05-28T06:04:11.909050+0000","last_fullsized":"2024-05-28T06:04:11.909050+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:58:06.720498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.998553+0000","last_change":"2024-05-28T06:04:10.921008+0000","last_active":"2024-05-28T06:04:12.998553+0000","last_peered":"2024-05-28T06:04:12.998553+0000","last_clean":"2024-05-28T06:04:12.998553+0000","last_became_active":"2024-05-28T06:04:10.913026+0000","last_became_peered":"2024-05-28T06:04:10.913026+0000","last_unstale":"2024-05-28T06:04:12.998553+0000","last_undegraded":"2024-05-28T06:04:12.998553+0000","last_fullsized":"2024-05-28T06:04:12.998553+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:34:44.387474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.998439+0000","last_change":"2024-05-28T06:04:09.887088+0000","last_active":"2024-05-28T06:04:12.998439+0000","last_peered":"2024-05-28T06:04:12.998439+0000","last_clean":"2024-05-28T06:04:12.998439+0000","last_became_active":"2024-05-28T06:04:09.886809+0000","last_became_peered":"2024-05-28T06:04:09.886809+0000","last_unstale":"2024-05-28T06:04:12.998439+0000","last_undegraded":"2024-05-28T06:04:12.998439+0000","last_fullsized":"2024-05-28T06:04:12.998439+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:16:52.703842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.794379+0000","last_change":"2024-05-28T06:04:10.904810+0000","last_active":"2024-05-28T06:04:12.794379+0000","last_peered":"2024-05-28T06:04:12.794379+0000","last_clean":"2024-05-28T06:04:12.794379+0000","last_became_active":"2024-05-28T06:04:10.904445+0000","last_became_peered":"2024-05-28T06:04:10.904445+0000","last_unstale":"2024-05-28T06:04:12.794379+0000","last_undegraded":"2024-05-28T06:04:12.794379+0000","last_fullsized":"2024-05-28T06:04:12.794379+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:58:16.009219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.890096+0000","last_change":"2024-05-28T06:04:09.882535+0000","last_active":"2024-05-28T06:04:11.890096+0000","last_peered":"2024-05-28T06:04:11.890096+0000","last_clean":"2024-05-28T06:04:11.890096+0000","last_became_active":"2024-05-28T06:04:09.882159+0000","last_became_peered":"2024-05-28T06:04:09.882159+0000","last_unstale":"2024-05-28T06:04:11.890096+0000","last_undegraded":"2024-05-28T06:04:11.890096+0000","last_fullsized":"2024-05-28T06:04:11.890096+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:03:45.379665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.893646+0000","last_change":"2024-05-28T06:04:10.901897+0000","last_active":"2024-05-28T06:04:11.893646+0000","last_peered":"2024-05-28T06:04:11.893646+0000","last_clean":"2024-05-28T06:04:11.893646+0000","last_became_active":"2024-05-28T06:04:10.901156+0000","last_became_peered":"2024-05-28T06:04:10.901156+0000","last_unstale":"2024-05-28T06:04:11.893646+0000","last_undegraded":"2024-05-28T06:04:11.893646+0000","last_fullsized":"2024-05-28T06:04:11.893646+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:03:46.394497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.909296+0000","last_change":"2024-05-28T06:04:09.878442+0000","last_active":"2024-05-28T06:04:11.909296+0000","last_peered":"2024-05-28T06:04:11.909296+0000","last_clean":"2024-05-28T06:04:11.909296+0000","last_became_active":"2024-05-28T06:04:09.878148+0000","last_became_peered":"2024-05-28T06:04:09.878148+0000","last_unstale":"2024-05-28T06:04:11.909296+0000","last_undegraded":"2024-05-28T06:04:11.909296+0000","last_fullsized":"2024-05-28T06:04:11.909296+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:38:20.882939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.996201+0000","last_change":"2024-05-28T06:04:10.911058+0000","last_active":"2024-05-28T06:04:12.996201+0000","last_peered":"2024-05-28T06:04:12.996201+0000","last_clean":"2024-05-28T06:04:12.996201+0000","last_became_active":"2024-05-28T06:04:10.910775+0000","last_became_peered":"2024-05-28T06:04:10.910775+0000","last_unstale":"2024-05-28T06:04:12.996201+0000","last_undegraded":"2024-05-28T06:04:12.996201+0000","last_fullsized":"2024-05-28T06:04:12.996201+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:03:18.610996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.794179+0000","last_change":"2024-05-28T06:04:09.878327+0000","last_active":"2024-05-28T06:04:12.794179+0000","last_peered":"2024-05-28T06:04:12.794179+0000","last_clean":"2024-05-28T06:04:12.794179+0000","last_became_active":"2024-05-28T06:04:09.878028+0000","last_became_peered":"2024-05-28T06:04:09.878028+0000","last_unstale":"2024-05-28T06:04:12.794179+0000","last_undegraded":"2024-05-28T06:04:12.794179+0000","last_fullsized":"2024-05-28T06:04:12.794179+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:12:29.496721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.793473+0000","last_change":"2024-05-28T06:04:10.902163+0000","last_active":"2024-05-28T06:04:12.793473+0000","last_peered":"2024-05-28T06:04:12.793473+0000","last_clean":"2024-05-28T06:04:12.793473+0000","last_became_active":"2024-05-28T06:04:10.901846+0000","last_became_peered":"2024-05-28T06:04:10.901846+0000","last_unstale":"2024-05-28T06:04:12.793473+0000","last_undegraded":"2024-05-28T06:04:12.793473+0000","last_fullsized":"2024-05-28T06:04:12.793473+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:11:13.101661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.997163+0000","last_change":"2024-05-28T06:04:09.884756+0000","last_active":"2024-05-28T06:04:12.997163+0000","last_peered":"2024-05-28T06:04:12.997163+0000","last_clean":"2024-05-28T06:04:12.997163+0000","last_became_active":"2024-05-28T06:04:09.884354+0000","last_became_peered":"2024-05-28T06:04:09.884354+0000","last_unstale":"2024-05-28T06:04:12.997163+0000","last_undegraded":"2024-05-28T06:04:12.997163+0000","last_fullsized":"2024-05-28T06:04:12.997163+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:36:53.154893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.996772+0000","last_change":"2024-05-28T06:04:10.916180+0000","last_active":"2024-05-28T06:04:12.996772+0000","last_peered":"2024-05-28T06:04:12.996772+0000","last_clean":"2024-05-28T06:04:12.996772+0000","last_became_active":"2024-05-28T06:04:10.915630+0000","last_became_peered":"2024-05-28T06:04:10.915630+0000","last_unstale":"2024-05-28T06:04:12.996772+0000","last_undegraded":"2024-05-28T06:04:12.996772+0000","last_fullsized":"2024-05-28T06:04:12.996772+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:34:44.387474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.908470+0000","last_change":"2024-05-28T06:04:09.883806+0000","last_active":"2024-05-28T06:04:11.908470+0000","last_peered":"2024-05-28T06:04:11.908470+0000","last_clean":"2024-05-28T06:04:11.908470+0000","last_became_active":"2024-05-28T06:04:09.883352+0000","last_became_peered":"2024-05-28T06:04:09.883352+0000","last_unstale":"2024-05-28T06:04:11.908470+0000","last_undegraded":"2024-05-28T06:04:11.908470+0000","last_fullsized":"2024-05-28T06:04:11.908470+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:36:53.154893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.908799+0000","last_change":"2024-05-28T06:04:10.908671+0000","last_active":"2024-05-28T06:04:11.908799+0000","last_peered":"2024-05-28T06:04:11.908799+0000","last_clean":"2024-05-28T06:04:11.908799+0000","last_became_active":"2024-05-28T06:04:10.908402+0000","last_became_peered":"2024-05-28T06:04:10.908402+0000","last_unstale":"2024-05-28T06:04:11.908799+0000","last_undegraded":"2024-05-28T06:04:11.908799+0000","last_fullsized":"2024-05-28T06:04:11.908799+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:06:44.034610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.995124+0000","last_change":"2024-05-28T06:04:09.868302+0000","last_active":"2024-05-28T06:04:12.995124+0000","last_peered":"2024-05-28T06:04:12.995124+0000","last_clean":"2024-05-28T06:04:12.995124+0000","last_became_active":"2024-05-28T06:04:09.868003+0000","last_became_peered":"2024-05-28T06:04:09.868003+0000","last_unstale":"2024-05-28T06:04:12.995124+0000","last_undegraded":"2024-05-28T06:04:12.995124+0000","last_fullsized":"2024-05-28T06:04:12.995124+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:29:32.319083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.995729+0000","last_change":"2024-05-28T06:04:10.916061+0000","last_active":"2024-05-28T06:04:12.995729+0000","last_peered":"2024-05-28T06:04:12.995729+0000","last_clean":"2024-05-28T06:04:12.995729+0000","last_became_active":"2024-05-28T06:04:10.915607+0000","last_became_peered":"2024-05-28T06:04:10.915607+0000","last_unstale":"2024-05-28T06:04:12.995729+0000","last_undegraded":"2024-05-28T06:04:12.995729+0000","last_fullsized":"2024-05-28T06:04:12.995729+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:51:12.839226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.893456+0000","last_change":"2024-05-28T06:04:09.874073+0000","last_active":"2024-05-28T06:04:11.893456+0000","last_peered":"2024-05-28T06:04:11.893456+0000","last_clean":"2024-05-28T06:04:11.893456+0000","last_became_active":"2024-05-28T06:04:09.872906+0000","last_became_peered":"2024-05-28T06:04:09.872906+0000","last_unstale":"2024-05-28T06:04:11.893456+0000","last_undegraded":"2024-05-28T06:04:11.893456+0000","last_fullsized":"2024-05-28T06:04:11.893456+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:06:04.907427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.998407+0000","last_change":"2024-05-28T06:04:10.914311+0000","last_active":"2024-05-28T06:04:12.998407+0000","last_peered":"2024-05-28T06:04:12.998407+0000","last_clean":"2024-05-28T06:04:12.998407+0000","last_became_active":"2024-05-28T06:04:10.913972+0000","last_became_peered":"2024-05-28T06:04:10.913972+0000","last_unstale":"2024-05-28T06:04:12.998407+0000","last_undegraded":"2024-05-28T06:04:12.998407+0000","last_fullsized":"2024-05-28T06:04:12.998407+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:11:58.309255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.996846+0000","last_change":"2024-05-28T06:04:09.867608+0000","last_active":"2024-05-28T06:04:12.996846+0000","last_peered":"2024-05-28T06:04:12.996846+0000","last_clean":"2024-05-28T06:04:12.996846+0000","last_became_active":"2024-05-28T06:04:09.865750+0000","last_became_peered":"2024-05-28T06:04:09.865750+0000","last_unstale":"2024-05-28T06:04:12.996846+0000","last_undegraded":"2024-05-28T06:04:12.996846+0000","last_fullsized":"2024-05-28T06:04:12.996846+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:16:52.703842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.998212+0000","last_change":"2024-05-28T06:04:10.914237+0000","last_active":"2024-05-28T06:04:12.998212+0000","last_peered":"2024-05-28T06:04:12.998212+0000","last_clean":"2024-05-28T06:04:12.998212+0000","last_became_active":"2024-05-28T06:04:10.913957+0000","last_became_peered":"2024-05-28T06:04:10.913957+0000","last_unstale":"2024-05-28T06:04:12.998212+0000","last_undegraded":"2024-05-28T06:04:12.998212+0000","last_fullsized":"2024-05-28T06:04:12.998212+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:28:36.252853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.890109+0000","last_change":"2024-05-28T06:04:09.877173+0000","last_active":"2024-05-28T06:04:11.890109+0000","last_peered":"2024-05-28T06:04:11.890109+0000","last_clean":"2024-05-28T06:04:11.890109+0000","last_became_active":"2024-05-28T06:04:09.876534+0000","last_became_peered":"2024-05-28T06:04:09.876534+0000","last_unstale":"2024-05-28T06:04:11.890109+0000","last_undegraded":"2024-05-28T06:04:11.890109+0000","last_fullsized":"2024-05-28T06:04:11.890109+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:58:50.805034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.998844+0000","last_change":"2024-05-28T06:04:10.923771+0000","last_active":"2024-05-28T06:04:12.998844+0000","last_peered":"2024-05-28T06:04:12.998844+0000","last_clean":"2024-05-28T06:04:12.998844+0000","last_became_active":"2024-05-28T06:04:10.923494+0000","last_became_peered":"2024-05-28T06:04:10.923494+0000","last_unstale":"2024-05-28T06:04:12.998844+0000","last_undegraded":"2024-05-28T06:04:12.998844+0000","last_fullsized":"2024-05-28T06:04:12.998844+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:05:14.695994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.995877+0000","last_change":"2024-05-28T06:04:09.880725+0000","last_active":"2024-05-28T06:04:12.995877+0000","last_peered":"2024-05-28T06:04:12.995877+0000","last_clean":"2024-05-28T06:04:12.995877+0000","last_became_active":"2024-05-28T06:04:09.880430+0000","last_became_peered":"2024-05-28T06:04:09.880430+0000","last_unstale":"2024-05-28T06:04:12.995877+0000","last_undegraded":"2024-05-28T06:04:12.995877+0000","last_fullsized":"2024-05-28T06:04:12.995877+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:27:30.379288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.996167+0000","last_change":"2024-05-28T06:04:10.904576+0000","last_active":"2024-05-28T06:04:12.996167+0000","last_peered":"2024-05-28T06:04:12.996167+0000","last_clean":"2024-05-28T06:04:12.996167+0000","last_became_active":"2024-05-28T06:04:10.904257+0000","last_became_peered":"2024-05-28T06:04:10.904257+0000","last_unstale":"2024-05-28T06:04:12.996167+0000","last_undegraded":"2024-05-28T06:04:12.996167+0000","last_fullsized":"2024-05-28T06:04:12.996167+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:49:22.704773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":170,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.995506+0000","last_change":"2024-05-28T06:04:01.039129+0000","last_active":"2024-05-28T06:04:12.995506+0000","last_peered":"2024-05-28T06:04:12.995506+0000","last_clean":"2024-05-28T06:04:12.995506+0000","last_became_active":"2024-05-28T06:04:01.033906+0000","last_became_peered":"2024-05-28T06:04:01.033906+0000","last_unstale":"2024-05-28T06:04:12.995506+0000","last_undegraded":"2024-05-28T06:04:12.995506+0000","last_fullsized":"2024-05-28T06:04:12.995506+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:03:59.864237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:03:59.864237+0000","last_clean_scrub_stamp":"2024-05-28T06:03:59.864237+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:17:36.440245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.996021+0000","last_change":"2024-05-28T06:04:09.879679+0000","last_active":"2024-05-28T06:04:12.996021+0000","last_peered":"2024-05-28T06:04:12.996021+0000","last_clean":"2024-05-28T06:04:12.996021+0000","last_became_active":"2024-05-28T06:04:09.877412+0000","last_became_peered":"2024-05-28T06:04:09.877412+0000","last_unstale":"2024-05-28T06:04:12.996021+0000","last_undegraded":"2024-05-28T06:04:12.996021+0000","last_fullsized":"2024-05-28T06:04:12.996021+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:58:06.720498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.996386+0000","last_change":"2024-05-28T06:04:10.901746+0000","last_active":"2024-05-28T06:04:12.996386+0000","last_peered":"2024-05-28T06:04:12.996386+0000","last_clean":"2024-05-28T06:04:12.996386+0000","last_became_active":"2024-05-28T06:04:10.901222+0000","last_became_peered":"2024-05-28T06:04:10.901222+0000","last_unstale":"2024-05-28T06:04:12.996386+0000","last_undegraded":"2024-05-28T06:04:12.996386+0000","last_fullsized":"2024-05-28T06:04:12.996386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:46:29.255545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.995167+0000","last_change":"2024-05-28T06:04:09.890312+0000","last_active":"2024-05-28T06:04:12.995167+0000","last_peered":"2024-05-28T06:04:12.995167+0000","last_clean":"2024-05-28T06:04:12.995167+0000","last_became_active":"2024-05-28T06:04:09.889754+0000","last_became_peered":"2024-05-28T06:04:09.889754+0000","last_unstale":"2024-05-28T06:04:12.995167+0000","last_undegraded":"2024-05-28T06:04:12.995167+0000","last_fullsized":"2024-05-28T06:04:12.995167+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:38:20.882939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.793983+0000","last_change":"2024-05-28T06:04:10.886055+0000","last_active":"2024-05-28T06:04:12.793983+0000","last_peered":"2024-05-28T06:04:12.793983+0000","last_clean":"2024-05-28T06:04:12.793983+0000","last_became_active":"2024-05-28T06:04:10.885596+0000","last_became_peered":"2024-05-28T06:04:10.885596+0000","last_unstale":"2024-05-28T06:04:12.793983+0000","last_undegraded":"2024-05-28T06:04:12.793983+0000","last_fullsized":"2024-05-28T06:04:12.793983+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:06:44.034610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.995673+0000","last_change":"2024-05-28T06:04:09.871789+0000","last_active":"2024-05-28T06:04:12.995673+0000","last_peered":"2024-05-28T06:04:12.995673+0000","last_clean":"2024-05-28T06:04:12.995673+0000","last_became_active":"2024-05-28T06:04:09.870232+0000","last_became_peered":"2024-05-28T06:04:09.870232+0000","last_unstale":"2024-05-28T06:04:12.995673+0000","last_undegraded":"2024-05-28T06:04:12.995673+0000","last_fullsized":"2024-05-28T06:04:12.995673+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:36:53.154893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.890126+0000","last_change":"2024-05-28T06:04:10.898534+0000","last_active":"2024-05-28T06:04:11.890126+0000","last_peered":"2024-05-28T06:04:11.890126+0000","last_clean":"2024-05-28T06:04:11.890126+0000","last_became_active":"2024-05-28T06:04:10.898237+0000","last_became_peered":"2024-05-28T06:04:10.898237+0000","last_unstale":"2024-05-28T06:04:11.890126+0000","last_undegraded":"2024-05-28T06:04:11.890126+0000","last_fullsized":"2024-05-28T06:04:11.890126+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:46:29.255545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.997224+0000","last_change":"2024-05-28T06:04:09.884779+0000","last_active":"2024-05-28T06:04:12.997224+0000","last_peered":"2024-05-28T06:04:12.997224+0000","last_clean":"2024-05-28T06:04:12.997224+0000","last_became_active":"2024-05-28T06:04:09.882344+0000","last_became_peered":"2024-05-28T06:04:09.882344+0000","last_unstale":"2024-05-28T06:04:12.997224+0000","last_undegraded":"2024-05-28T06:04:12.997224+0000","last_fullsized":"2024-05-28T06:04:12.997224+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:29:32.319083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.893200+0000","last_change":"2024-05-28T06:04:10.890674+0000","last_active":"2024-05-28T06:04:11.893200+0000","last_peered":"2024-05-28T06:04:11.893200+0000","last_clean":"2024-05-28T06:04:11.893200+0000","last_became_active":"2024-05-28T06:04:10.890144+0000","last_became_peered":"2024-05-28T06:04:10.890144+0000","last_unstale":"2024-05-28T06:04:11.893200+0000","last_undegraded":"2024-05-28T06:04:11.893200+0000","last_fullsized":"2024-05-28T06:04:11.893200+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:38:21.897771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.890349+0000","last_change":"2024-05-28T06:04:09.871025+0000","last_active":"2024-05-28T06:04:11.890349+0000","last_peered":"2024-05-28T06:04:11.890349+0000","last_clean":"2024-05-28T06:04:11.890349+0000","last_became_active":"2024-05-28T06:04:09.870646+0000","last_became_peered":"2024-05-28T06:04:09.870646+0000","last_unstale":"2024-05-28T06:04:11.890349+0000","last_undegraded":"2024-05-28T06:04:11.890349+0000","last_fullsized":"2024-05-28T06:04:11.890349+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:16:52.703842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.998525+0000","last_change":"2024-05-28T06:04:10.912585+0000","last_active":"2024-05-28T06:04:12.998525+0000","last_peered":"2024-05-28T06:04:12.998525+0000","last_clean":"2024-05-28T06:04:12.998525+0000","last_became_active":"2024-05-28T06:04:10.912038+0000","last_became_peered":"2024-05-28T06:04:10.912038+0000","last_unstale":"2024-05-28T06:04:12.998525+0000","last_undegraded":"2024-05-28T06:04:12.998525+0000","last_fullsized":"2024-05-28T06:04:12.998525+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:04:49.415234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.997550+0000","last_change":"2024-05-28T06:04:09.890971+0000","last_active":"2024-05-28T06:04:12.997550+0000","last_peered":"2024-05-28T06:04:12.997550+0000","last_clean":"2024-05-28T06:04:12.997550+0000","last_became_active":"2024-05-28T06:04:09.890401+0000","last_became_peered":"2024-05-28T06:04:09.890401+0000","last_unstale":"2024-05-28T06:04:12.997550+0000","last_undegraded":"2024-05-28T06:04:12.997550+0000","last_fullsized":"2024-05-28T06:04:12.997550+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:05:29.419386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.890261+0000","last_change":"2024-05-28T06:04:10.903420+0000","last_active":"2024-05-28T06:04:11.890261+0000","last_peered":"2024-05-28T06:04:11.890261+0000","last_clean":"2024-05-28T06:04:11.890261+0000","last_became_active":"2024-05-28T06:04:10.898157+0000","last_became_peered":"2024-05-28T06:04:10.898157+0000","last_unstale":"2024-05-28T06:04:11.890261+0000","last_undegraded":"2024-05-28T06:04:11.890261+0000","last_fullsized":"2024-05-28T06:04:11.890261+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:11:51.651640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.890155+0000","last_change":"2024-05-28T06:04:09.886223+0000","last_active":"2024-05-28T06:04:11.890155+0000","last_peered":"2024-05-28T06:04:11.890155+0000","last_clean":"2024-05-28T06:04:11.890155+0000","last_became_active":"2024-05-28T06:04:09.885439+0000","last_became_peered":"2024-05-28T06:04:09.885439+0000","last_unstale":"2024-05-28T06:04:11.890155+0000","last_undegraded":"2024-05-28T06:04:11.890155+0000","last_fullsized":"2024-05-28T06:04:11.890155+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:38:20.882939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.997783+0000","last_change":"2024-05-28T06:04:10.901609+0000","last_active":"2024-05-28T06:04:12.997783+0000","last_peered":"2024-05-28T06:04:12.997783+0000","last_clean":"2024-05-28T06:04:12.997783+0000","last_became_active":"2024-05-28T06:04:10.901302+0000","last_became_peered":"2024-05-28T06:04:10.901302+0000","last_unstale":"2024-05-28T06:04:12.997783+0000","last_undegraded":"2024-05-28T06:04:12.997783+0000","last_fullsized":"2024-05-28T06:04:12.997783+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:49:22.704773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.909384+0000","last_change":"2024-05-28T06:04:09.888169+0000","last_active":"2024-05-28T06:04:11.909384+0000","last_peered":"2024-05-28T06:04:11.909384+0000","last_clean":"2024-05-28T06:04:11.909384+0000","last_became_active":"2024-05-28T06:04:09.887867+0000","last_became_peered":"2024-05-28T06:04:09.887867+0000","last_unstale":"2024-05-28T06:04:11.909384+0000","last_undegraded":"2024-05-28T06:04:11.909384+0000","last_fullsized":"2024-05-28T06:04:11.909384+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:16:52.703842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.793993+0000","last_change":"2024-05-28T06:04:10.897718+0000","last_active":"2024-05-28T06:04:12.793993+0000","last_peered":"2024-05-28T06:04:12.793993+0000","last_clean":"2024-05-28T06:04:12.793993+0000","last_became_active":"2024-05-28T06:04:10.896414+0000","last_became_peered":"2024-05-28T06:04:10.896414+0000","last_unstale":"2024-05-28T06:04:12.793993+0000","last_undegraded":"2024-05-28T06:04:12.793993+0000","last_fullsized":"2024-05-28T06:04:12.793993+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:53:07.733019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.995167+0000","last_change":"2024-05-28T06:04:09.889726+0000","last_active":"2024-05-28T06:04:12.995167+0000","last_peered":"2024-05-28T06:04:12.995167+0000","last_clean":"2024-05-28T06:04:12.995167+0000","last_became_active":"2024-05-28T06:04:09.889394+0000","last_became_peered":"2024-05-28T06:04:09.889394+0000","last_unstale":"2024-05-28T06:04:12.995167+0000","last_undegraded":"2024-05-28T06:04:12.995167+0000","last_fullsized":"2024-05-28T06:04:12.995167+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:58:06.720498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.892869+0000","last_change":"2024-05-28T06:04:10.908922+0000","last_active":"2024-05-28T06:04:11.892869+0000","last_peered":"2024-05-28T06:04:11.892869+0000","last_clean":"2024-05-28T06:04:11.892869+0000","last_became_active":"2024-05-28T06:04:10.905738+0000","last_became_peered":"2024-05-28T06:04:10.905738+0000","last_unstale":"2024-05-28T06:04:11.892869+0000","last_undegraded":"2024-05-28T06:04:11.892869+0000","last_fullsized":"2024-05-28T06:04:11.892869+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:58:07.735330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.890426+0000","last_change":"2024-05-28T06:04:09.886430+0000","last_active":"2024-05-28T06:04:11.890426+0000","last_peered":"2024-05-28T06:04:11.890426+0000","last_clean":"2024-05-28T06:04:11.890426+0000","last_became_active":"2024-05-28T06:04:09.886072+0000","last_became_peered":"2024-05-28T06:04:09.886072+0000","last_unstale":"2024-05-28T06:04:11.890426+0000","last_undegraded":"2024-05-28T06:04:11.890426+0000","last_fullsized":"2024-05-28T06:04:11.890426+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:29:32.319083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.995901+0000","last_change":"2024-05-28T06:04:10.895054+0000","last_active":"2024-05-28T06:04:12.995901+0000","last_peered":"2024-05-28T06:04:12.995901+0000","last_clean":"2024-05-28T06:04:12.995901+0000","last_became_active":"2024-05-28T06:04:10.894384+0000","last_became_peered":"2024-05-28T06:04:10.894384+0000","last_unstale":"2024-05-28T06:04:12.995901+0000","last_undegraded":"2024-05-28T06:04:12.995901+0000","last_fullsized":"2024-05-28T06:04:12.995901+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:53:07.733019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.889822+0000","last_change":"2024-05-28T06:04:09.873566+0000","last_active":"2024-05-28T06:04:11.889822+0000","last_peered":"2024-05-28T06:04:11.889822+0000","last_clean":"2024-05-28T06:04:11.889822+0000","last_became_active":"2024-05-28T06:04:09.872928+0000","last_became_peered":"2024-05-28T06:04:09.872928+0000","last_unstale":"2024-05-28T06:04:11.889822+0000","last_undegraded":"2024-05-28T06:04:11.889822+0000","last_fullsized":"2024-05-28T06:04:11.889822+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:58:06.720498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.996688+0000","last_change":"2024-05-28T06:04:10.905645+0000","last_active":"2024-05-28T06:04:12.996688+0000","last_peered":"2024-05-28T06:04:12.996688+0000","last_clean":"2024-05-28T06:04:12.996688+0000","last_became_active":"2024-05-28T06:04:10.905308+0000","last_became_peered":"2024-05-28T06:04:10.905308+0000","last_unstale":"2024-05-28T06:04:12.996688+0000","last_undegraded":"2024-05-28T06:04:12.996688+0000","last_fullsized":"2024-05-28T06:04:12.996688+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:11:51.651640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.997139+0000","last_change":"2024-05-28T06:04:09.893691+0000","last_active":"2024-05-28T06:04:12.997139+0000","last_peered":"2024-05-28T06:04:12.997139+0000","last_clean":"2024-05-28T06:04:12.997139+0000","last_became_active":"2024-05-28T06:04:09.893416+0000","last_became_peered":"2024-05-28T06:04:09.893416+0000","last_unstale":"2024-05-28T06:04:12.997139+0000","last_undegraded":"2024-05-28T06:04:12.997139+0000","last_fullsized":"2024-05-28T06:04:12.997139+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:53:48.086901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.908760+0000","last_change":"2024-05-28T06:04:10.919265+0000","last_active":"2024-05-28T06:04:11.908760+0000","last_peered":"2024-05-28T06:04:11.908760+0000","last_clean":"2024-05-28T06:04:11.908760+0000","last_became_active":"2024-05-28T06:04:10.918268+0000","last_became_peered":"2024-05-28T06:04:10.918268+0000","last_unstale":"2024-05-28T06:04:11.908760+0000","last_undegraded":"2024-05-28T06:04:11.908760+0000","last_fullsized":"2024-05-28T06:04:11.908760+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:39:46.389531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.998229+0000","last_change":"2024-05-28T06:04:10.912331+0000","last_active":"2024-05-28T06:04:12.998229+0000","last_peered":"2024-05-28T06:04:12.998229+0000","last_clean":"2024-05-28T06:04:12.998229+0000","last_became_active":"2024-05-28T06:04:10.907887+0000","last_became_peered":"2024-05-28T06:04:10.907887+0000","last_unstale":"2024-05-28T06:04:12.998229+0000","last_undegraded":"2024-05-28T06:04:12.998229+0000","last_fullsized":"2024-05-28T06:04:12.998229+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:03:18.610996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.996617+0000","last_change":"2024-05-28T06:04:09.892486+0000","last_active":"2024-05-28T06:04:12.996617+0000","last_peered":"2024-05-28T06:04:12.996617+0000","last_clean":"2024-05-28T06:04:12.996617+0000","last_became_active":"2024-05-28T06:04:09.892160+0000","last_became_peered":"2024-05-28T06:04:09.892160+0000","last_unstale":"2024-05-28T06:04:12.996617+0000","last_undegraded":"2024-05-28T06:04:12.996617+0000","last_fullsized":"2024-05-28T06:04:12.996617+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:03:45.379665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.997770+0000","last_change":"2024-05-28T06:04:10.909079+0000","last_active":"2024-05-28T06:04:12.997770+0000","last_peered":"2024-05-28T06:04:12.997770+0000","last_clean":"2024-05-28T06:04:12.997770+0000","last_became_active":"2024-05-28T06:04:10.907979+0000","last_became_peered":"2024-05-28T06:04:10.907979+0000","last_unstale":"2024-05-28T06:04:12.997770+0000","last_undegraded":"2024-05-28T06:04:12.997770+0000","last_fullsized":"2024-05-28T06:04:12.997770+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:49:22.704773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.996620+0000","last_change":"2024-05-28T06:04:09.890388+0000","last_active":"2024-05-28T06:04:12.996620+0000","last_peered":"2024-05-28T06:04:12.996620+0000","last_clean":"2024-05-28T06:04:12.996620+0000","last_became_active":"2024-05-28T06:04:09.890086+0000","last_became_peered":"2024-05-28T06:04:09.890086+0000","last_unstale":"2024-05-28T06:04:12.996620+0000","last_undegraded":"2024-05-28T06:04:12.996620+0000","last_fullsized":"2024-05-28T06:04:12.996620+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:58:50.805034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.794239+0000","last_change":"2024-05-28T06:04:10.904802+0000","last_active":"2024-05-28T06:04:12.794239+0000","last_peered":"2024-05-28T06:04:12.794239+0000","last_clean":"2024-05-28T06:04:12.794239+0000","last_became_active":"2024-05-28T06:04:10.904438+0000","last_became_peered":"2024-05-28T06:04:10.904438+0000","last_unstale":"2024-05-28T06:04:12.794239+0000","last_undegraded":"2024-05-28T06:04:12.794239+0000","last_fullsized":"2024-05-28T06:04:12.794239+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:11:51.651640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.997391+0000","last_change":"2024-05-28T06:04:09.874413+0000","last_active":"2024-05-28T06:04:12.997391+0000","last_peered":"2024-05-28T06:04:12.997391+0000","last_clean":"2024-05-28T06:04:12.997391+0000","last_became_active":"2024-05-28T06:04:09.872117+0000","last_became_peered":"2024-05-28T06:04:09.872117+0000","last_unstale":"2024-05-28T06:04:12.997391+0000","last_undegraded":"2024-05-28T06:04:12.997391+0000","last_fullsized":"2024-05-28T06:04:12.997391+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:36:53.154893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.997324+0000","last_change":"2024-05-28T06:04:10.909242+0000","last_active":"2024-05-28T06:04:12.997324+0000","last_peered":"2024-05-28T06:04:12.997324+0000","last_clean":"2024-05-28T06:04:12.997324+0000","last_became_active":"2024-05-28T06:04:10.908934+0000","last_became_peered":"2024-05-28T06:04:10.908934+0000","last_unstale":"2024-05-28T06:04:12.997324+0000","last_undegraded":"2024-05-28T06:04:12.997324+0000","last_fullsized":"2024-05-28T06:04:12.997324+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:45:47.884578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.794420+0000","last_change":"2024-05-28T06:04:09.887679+0000","last_active":"2024-05-28T06:04:12.794420+0000","last_peered":"2024-05-28T06:04:12.794420+0000","last_clean":"2024-05-28T06:04:12.794420+0000","last_became_active":"2024-05-28T06:04:09.887386+0000","last_became_peered":"2024-05-28T06:04:09.887386+0000","last_unstale":"2024-05-28T06:04:12.794420+0000","last_undegraded":"2024-05-28T06:04:12.794420+0000","last_fullsized":"2024-05-28T06:04:12.794420+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:38:33.383965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574853,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":802816,"data_stored":661884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":208896,"data_stored":71024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":71060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":70934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":212992,"data_stored":71094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":811008,"data_stored":661940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-28T06:04:16.123 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-05-28T06:04:16.123 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-05-28T06:04:16.123 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-05-28T06:04:16.123 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-05-28T06:04:16.124 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-05-28T06:04:16.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.364+0000 7f0865a00640 1 Processor -- start 2024-05-28T06:04:16.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.364+0000 7f0865a00640 1 -- start start 2024-05-28T06:04:16.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.365+0000 7f0865a00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f08600b4e10 0x7f08600cde30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:16.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.365+0000 7f0865a00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08600b51f0 con 0x7f08600ce370 2024-05-28T06:04:16.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.365+0000 7f0865a00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08600aef60 con 0x7f08600d0ca0 2024-05-28T06:04:16.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.365+0000 7f0865a00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f08600ad1d0 con 0x7f08600b4e10 2024-05-28T06:04:16.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.366+0000 7f085e000640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f08600ce370 0x7f08600d0590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:55428/0 (socket says 172.21.15.96:55428) 2024-05-28T06:04:16.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.366+0000 7f085e000640 1 -- 172.21.15.96:0/3125171763 learned_addr learned my addr 172.21.15.96:0/3125171763 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:04:16.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.366+0000 7f085ea00640 1 --2- 172.21.15.96:0/3125171763 >> v2:172.21.15.123:3300/0 conn(0x7f08600b4e10 0x7f08600cde30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:16.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.366+0000 7f085d600640 1 -- 172.21.15.96:0/3125171763 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2971142206 0 0) 0x7f08600b51f0 con 0x7f08600ce370 2024-05-28T06:04:16.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.366+0000 7f085d600640 1 -- 172.21.15.96:0/3125171763 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0844003520 con 0x7f08600ce370 2024-05-28T06:04:16.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.367+0000 7f085d600640 1 -- 172.21.15.96:0/3125171763 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3000728163 0 0) 0x7f0844003520 con 0x7f08600ce370 2024-05-28T06:04:16.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.367+0000 7f085d600640 1 -- 172.21.15.96:0/3125171763 >> v2:172.21.15.123:3300/0 conn(0x7f08600b4e10 msgr2=0x7f08600cde30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:16.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.367+0000 7f085d600640 1 --2- 172.21.15.96:0/3125171763 >> v2:172.21.15.123:3300/0 conn(0x7f08600b4e10 0x7f08600cde30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:16.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.367+0000 7f085d600640 1 -- 172.21.15.96:0/3125171763 >> v1:172.21.15.123:6789/0 conn(0x7f08600d0ca0 legacy=0x7f0860099090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:16.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.367+0000 7f085d600640 1 -- 172.21.15.96:0/3125171763 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08600ad470 con 0x7f08600ce370 2024-05-28T06:04:16.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.367+0000 7f085d600640 1 -- 172.21.15.96:0/3125171763 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f084c001670 con 0x7f08600ce370 2024-05-28T06:04:16.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.367+0000 7f085d600640 1 -- 172.21.15.96:0/3125171763 >> v1:172.21.15.96:6789/0 conn(0x7f08600ce370 legacy=0x7f08600d0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:16.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.367+0000 7f085d600640 1 --2- 172.21.15.96:0/3125171763 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f08440039b0 0x7f0844003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:16.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.367+0000 7f085d600640 1 --2- 172.21.15.96:0/3125171763 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0844004800 0x7f0844004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:16.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.367+0000 7f085d600640 1 --2- 172.21.15.96:0/3125171763 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08440051c0 0x7f084400dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:16.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.367+0000 7f085d600640 1 -- 172.21.15.96:0/3125171763 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f08600ad1d0 con 0x7f08440039b0 2024-05-28T06:04:16.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.367+0000 7f085d600640 1 -- 172.21.15.96:0/3125171763 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f084400e370 con 0x7f08440051c0 2024-05-28T06:04:16.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.367+0000 7f085d600640 1 -- 172.21.15.96:0/3125171763 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f084400e570 con 0x7f0844004800 2024-05-28T06:04:16.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.367+0000 7f085ea00640 1 --2- 172.21.15.96:0/3125171763 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f08440039b0 0x7f0844003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:16.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.367+0000 7f085f400640 1 --2- 172.21.15.96:0/3125171763 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08440051c0 0x7f084400dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:16.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.367+0000 7f085e000640 1 --2- 172.21.15.96:0/3125171763 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0844004800 0x7f0844004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:16.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.368+0000 7f085ea00640 1 -- 172.21.15.96:0/3125171763 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0844004800 msgr2=0x7f0844004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:16.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.368+0000 7f085ea00640 1 --2- 172.21.15.96:0/3125171763 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0844004800 0x7f0844004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:16.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.368+0000 7f085ea00640 1 -- 172.21.15.96:0/3125171763 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08440051c0 msgr2=0x7f084400dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:16.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.368+0000 7f085ea00640 1 --2- 172.21.15.96:0/3125171763 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08440051c0 0x7f084400dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:16.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.368+0000 7f085ea00640 1 -- 172.21.15.96:0/3125171763 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0844003520 con 0x7f08440039b0 2024-05-28T06:04:16.373 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.368+0000 7f085f400640 1 -- 172.21.15.96:0/3125171763 reap_dead start 2024-05-28T06:04:16.373 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.368+0000 7f085e000640 1 --2- 172.21.15.96:0/3125171763 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0844004800 0x7f0844004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:04:16.373 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.369+0000 7f085ea00640 1 --2- 172.21.15.96:0/3125171763 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f08440039b0 0x7f0844003dd0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f0850009ac0 tx=0x7f0844005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fdc36ba45aba085 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:16.373 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.369+0000 7f085d600640 1 -- 172.21.15.96:0/3125171763 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0850036070 con 0x7f08440039b0 2024-05-28T06:04:16.373 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.369+0000 7f085d600640 1 -- 172.21.15.96:0/3125171763 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0850031400 con 0x7f08440039b0 2024-05-28T06:04:16.373 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.369+0000 7f085d600640 1 -- 172.21.15.96:0/3125171763 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0850009690 con 0x7f08440039b0 2024-05-28T06:04:16.373 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.369+0000 7f0865a00640 1 -- 172.21.15.96:0/3125171763 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f08440039b0 msgr2=0x7f0844003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:16.373 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.369+0000 7f0865a00640 1 --2- 172.21.15.96:0/3125171763 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f08440039b0 0x7f0844003dd0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f0850009ac0 tx=0x7f0844005830 comp rx=0 tx=0).stop 2024-05-28T06:04:16.373 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.370+0000 7f0865a00640 1 -- 172.21.15.96:0/3125171763 shutdown_connections 2024-05-28T06:04:16.373 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.370+0000 7f0865a00640 1 --2- 172.21.15.96:0/3125171763 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f08440039b0 0x7f0844003dd0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:16.374 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.370+0000 7f0865a00640 1 -- 172.21.15.96:0/3125171763 >> 172.21.15.96:0/3125171763 conn(0x7f086006bae0 msgr2=0x7f086006bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:16.374 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.370+0000 7f0865a00640 1 -- 172.21.15.96:0/3125171763 shutdown_connections 2024-05-28T06:04:16.374 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.370+0000 7f0865a00640 1 -- 172.21.15.96:0/3125171763 wait complete. 2024-05-28T06:04:16.377 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.371+0000 7f0865a00640 1 Processor -- start 2024-05-28T06:04:16.377 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.371+0000 7f0865a00640 1 -- start start 2024-05-28T06:04:16.377 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.372+0000 7f0865a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f08440039b0 0x7f08600c9fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:16.377 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.372+0000 7f0865a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f08601f4580 0x7f08601f82a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:16.377 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.372+0000 7f0865a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08601f87e0 0x7f086006dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:16.377 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.372+0000 7f0865a00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f08600cea00 con 0x7f08440039b0 2024-05-28T06:04:16.377 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.372+0000 7f0865a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f08600ce880 con 0x7f08601f87e0 2024-05-28T06:04:16.377 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.372+0000 7f0865a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f08600b3150 con 0x7f08601f4580 2024-05-28T06:04:16.378 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.373+0000 7f085ea00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f08440039b0 0x7f08600c9fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:16.378 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.373+0000 7f085ea00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f08440039b0 0x7f08600c9fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:41536/0 (socket says 172.21.15.96:41536) 2024-05-28T06:04:16.378 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.373+0000 7f085ea00640 1 -- 172.21.15.96:0/2974746710 learned_addr learned my addr 172.21.15.96:0/2974746710 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:16.378 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.373+0000 7f085e000640 1 --2- 172.21.15.96:0/2974746710 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f08601f4580 0x7f08601f82a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:16.378 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.373+0000 7f085f400640 1 --2- 172.21.15.96:0/2974746710 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08601f87e0 0x7f086006dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:16.378 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.373+0000 7f085ea00640 1 -- 172.21.15.96:0/2974746710 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f08601f4580 msgr2=0x7f08601f82a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:16.378 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.373+0000 7f085ea00640 1 --2- 172.21.15.96:0/2974746710 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f08601f4580 0x7f08601f82a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:16.378 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.373+0000 7f085ea00640 1 -- 172.21.15.96:0/2974746710 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08601f87e0 msgr2=0x7f086006dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:16.378 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.373+0000 7f085ea00640 1 --2- 172.21.15.96:0/2974746710 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08601f87e0 0x7f086006dbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:16.378 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.373+0000 7f085ea00640 1 -- 172.21.15.96:0/2974746710 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08601f4d60 con 0x7f08440039b0 2024-05-28T06:04:16.378 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.373+0000 7f085e000640 1 --2- 172.21.15.96:0/2974746710 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f08601f4580 0x7f08601f82a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:04:16.378 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.373+0000 7f085ea00640 1 --2- 172.21.15.96:0/2974746710 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f08440039b0 0x7f08600c9fb0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f0850034040 tx=0x7f0850004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:16.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.374+0000 7f0856a00640 1 -- 172.21.15.96:0/2974746710 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f085003f020 con 0x7f08440039b0 2024-05-28T06:04:16.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.374+0000 7f0856a00640 1 -- 172.21.15.96:0/2974746710 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0850037040 con 0x7f08440039b0 2024-05-28T06:04:16.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.374+0000 7f0856a00640 1 -- 172.21.15.96:0/2974746710 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0850036070 con 0x7f08440039b0 2024-05-28T06:04:16.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.374+0000 7f0865a00640 1 -- 172.21.15.96:0/2974746710 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08601f5040 con 0x7f08440039b0 2024-05-28T06:04:16.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.374+0000 7f0865a00640 1 -- 172.21.15.96:0/2974746710 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08601f4a00 con 0x7f08440039b0 2024-05-28T06:04:16.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.375+0000 7f0865a00640 1 -- 172.21.15.96:0/2974746710 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08600689e0 con 0x7f08440039b0 2024-05-28T06:04:16.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.376+0000 7f0856a00640 1 -- 172.21.15.96:0/2974746710 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0850042020 con 0x7f08440039b0 2024-05-28T06:04:16.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.377+0000 7f0856a00640 1 --2- 172.21.15.96:0/2974746710 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f084006c8f0 0x7f084006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:16.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.377+0000 7f0856a00640 1 -- 172.21.15.96:0/2974746710 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f08500abb80 con 0x7f08440039b0 2024-05-28T06:04:16.380 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.377+0000 7f085e000640 1 --2- 172.21.15.96:0/2974746710 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f084006c8f0 0x7f084006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:16.380 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.379+0000 7f085e000640 1 --2- 172.21.15.96:0/2974746710 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f084006c8f0 0x7f084006edb0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f08600c3040 tx=0x7f084c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:16.381 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.381+0000 7f0856a00640 1 -- 172.21.15.96:0/2974746710 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0850049030 con 0x7f08440039b0 2024-05-28T06:04:16.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.633+0000 7f0865a00640 1 -- 172.21.15.96:0/2974746710 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f086006f540 con 0x7f08440039b0 2024-05-28T06:04:16.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.642+0000 7f0856a00640 1 -- 172.21.15.96:0/2974746710 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+174140 (secure 0 0 0) 0x7f085007a060 con 0x7f08440039b0 2024-05-28T06:04:16.646 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:04:16.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.646+0000 7f0865a00640 1 -- 172.21.15.96:0/2974746710 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f084006c8f0 msgr2=0x7f084006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:16.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.646+0000 7f0865a00640 1 --2- 172.21.15.96:0/2974746710 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f084006c8f0 0x7f084006edb0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f08600c3040 tx=0x7f084c009040 comp rx=0 tx=0).stop 2024-05-28T06:04:16.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.646+0000 7f0865a00640 1 -- 172.21.15.96:0/2974746710 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f08440039b0 msgr2=0x7f08600c9fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:16.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.646+0000 7f0865a00640 1 --2- 172.21.15.96:0/2974746710 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f08440039b0 0x7f08600c9fb0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f0850034040 tx=0x7f0850004730 comp rx=0 tx=0).stop 2024-05-28T06:04:16.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.647+0000 7f0865a00640 1 -- 172.21.15.96:0/2974746710 shutdown_connections 2024-05-28T06:04:16.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.647+0000 7f0865a00640 1 --2- 172.21.15.96:0/2974746710 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f084006c8f0 0x7f084006edb0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:16.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.647+0000 7f0865a00640 1 --2- 172.21.15.96:0/2974746710 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08601f87e0 0x7f086006dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:16.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.647+0000 7f0865a00640 1 --2- 172.21.15.96:0/2974746710 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f08601f4580 0x7f08601f82a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:16.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.647+0000 7f0865a00640 1 --2- 172.21.15.96:0/2974746710 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f08440039b0 0x7f08600c9fb0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:16.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.647+0000 7f0865a00640 1 -- 172.21.15.96:0/2974746710 >> 172.21.15.96:0/2974746710 conn(0x7f086006bae0 msgr2=0x7f08600cb7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:16.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.647+0000 7f0865a00640 1 -- 172.21.15.96:0/2974746710 shutdown_connections 2024-05-28T06:04:16.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.647+0000 7f0865a00640 1 -- 172.21.15.96:0/2974746710 wait complete. 2024-05-28T06:04:16.666 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":5,"flags":0,"active_gid":4105,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":259466469},{"type":"v1","addr":"172.21.15.123:6833","nonce":259466469}]},"active_addr":"172.21.15.123:6833/259466469","active_change":"2024-05-28T06:03:51.458249+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4107,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.123:0","nonce":2479167541}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.123:0","nonce":1060307833}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.123:0","nonce":2066801952}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.123:0","nonce":383357061}]}]} 2024-05-28T06:04:16.670 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-05-28T06:04:16.670 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-05-28T06:04:16.671 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:04:16.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.874+0000 7fbdcbc00640 1 Processor -- start 2024-05-28T06:04:16.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.874+0000 7fbdcbc00640 1 -- start start 2024-05-28T06:04:16.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.876+0000 7fbdcbc00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fbdc40d04a0 0x7fbdc40d0930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:16.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.876+0000 7fbdcbc00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fbdc40ad130 0x7fbdc40ad540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:16.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.876+0000 7fbdcbc00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbdc406ed10 con 0x7fbdc406f860 2024-05-28T06:04:16.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.876+0000 7fbdcbc00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fbdc415cb00 con 0x7fbdc40d04a0 2024-05-28T06:04:16.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.876+0000 7fbdcbc00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fbdc40b3cc0 con 0x7fbdc40ad130 2024-05-28T06:04:16.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.876+0000 7fbdc3e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fbdc40d04a0 0x7fbdc40d0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:16.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.876+0000 7fbdc9200640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fbdc406f860 0x7fbdc40b3150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:55438/0 (socket says 172.21.15.96:55438) 2024-05-28T06:04:16.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.876+0000 7fbdc9200640 1 -- 172.21.15.96:0/2420435505 learned_addr learned my addr 172.21.15.96:0/2420435505 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:04:16.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.876+0000 7fbdc9c00640 1 --2- 172.21.15.96:0/2420435505 >> v2:172.21.15.123:3300/0 conn(0x7fbdc40ad130 0x7fbdc40ad540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:16.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.877+0000 7fbdc3e00640 1 -- 172.21.15.96:0/2420435505 >> v2:172.21.15.123:3300/0 conn(0x7fbdc40ad130 msgr2=0x7fbdc40ad540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:16.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.877+0000 7fbdc3e00640 1 --2- 172.21.15.96:0/2420435505 >> v2:172.21.15.123:3300/0 conn(0x7fbdc40ad130 0x7fbdc40ad540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:16.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.877+0000 7fbdc3e00640 1 -- 172.21.15.96:0/2420435505 >> v1:172.21.15.96:6789/0 conn(0x7fbdc406f860 legacy=0x7fbdc40b3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:16.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.877+0000 7fbdc3e00640 1 -- 172.21.15.96:0/2420435505 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdc40b3fa0 con 0x7fbdc40d04a0 2024-05-28T06:04:16.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.878+0000 7fbdc3e00640 1 --2- 172.21.15.96:0/2420435505 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbdc40d04a0 0x7fbdc40d0930 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fbdc40af6e0 tx=0x7fbdac02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3dbf44c0b9a47a6 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:16.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.878+0000 7fbdc3400640 1 -- 172.21.15.96:0/2420435505 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdac037070 con 0x7fbdc40d04a0 2024-05-28T06:04:16.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.878+0000 7fbdc3400640 1 -- 172.21.15.96:0/2420435505 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbdac032440 con 0x7fbdc40d04a0 2024-05-28T06:04:16.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.878+0000 7fbdc3400640 1 -- 172.21.15.96:0/2420435505 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdac032810 con 0x7fbdc40d04a0 2024-05-28T06:04:16.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.879+0000 7fbdcbc00640 1 -- 172.21.15.96:0/2420435505 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbdc40d04a0 msgr2=0x7fbdc40d0930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:16.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.879+0000 7fbdcbc00640 1 --2- 172.21.15.96:0/2420435505 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbdc40d04a0 0x7fbdc40d0930 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fbdc40af6e0 tx=0x7fbdac02a1a0 comp rx=0 tx=0).stop 2024-05-28T06:04:16.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.879+0000 7fbdcbc00640 1 -- 172.21.15.96:0/2420435505 shutdown_connections 2024-05-28T06:04:16.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.879+0000 7fbdcbc00640 1 --2- 172.21.15.96:0/2420435505 >> v2:172.21.15.123:3300/0 conn(0x7fbdc40ad130 0x7fbdc40ad540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:16.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.879+0000 7fbdcbc00640 1 --2- 172.21.15.96:0/2420435505 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbdc40d04a0 0x7fbdc40d0930 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:16.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.879+0000 7fbdcbc00640 1 -- 172.21.15.96:0/2420435505 >> 172.21.15.96:0/2420435505 conn(0x7fbdc406bae0 msgr2=0x7fbdc406bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:16.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.879+0000 7fbdcbc00640 1 -- 172.21.15.96:0/2420435505 shutdown_connections 2024-05-28T06:04:16.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.879+0000 7fbdcbc00640 1 -- 172.21.15.96:0/2420435505 wait complete. 2024-05-28T06:04:16.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.880+0000 7fbdcbc00640 1 Processor -- start 2024-05-28T06:04:16.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.881+0000 7fbdcbc00640 1 -- start start 2024-05-28T06:04:16.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.881+0000 7fbdcbc00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbdc406f860 0x7fbdc40c5510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:16.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.881+0000 7fbdcbc00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbdc40ad130 0x7fbdc40b0d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:16.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.882+0000 7fbdcbc00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbdc40d04a0 0x7fbdc40b12a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:16.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.882+0000 7fbdcbc00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fbdc40b4840 con 0x7fbdc40d04a0 2024-05-28T06:04:16.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.882+0000 7fbdcbc00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fbdc40b4560 con 0x7fbdc406f860 2024-05-28T06:04:16.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.882+0000 7fbdcbc00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fbdc40b3860 con 0x7fbdc40ad130 2024-05-28T06:04:16.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.882+0000 7fbdc9c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbdc40d04a0 0x7fbdc40b12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:16.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.882+0000 7fbdc9c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbdc40d04a0 0x7fbdc40b12a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:41558/0 (socket says 172.21.15.96:41558) 2024-05-28T06:04:16.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.882+0000 7fbdc9c00640 1 -- 172.21.15.96:0/684157599 learned_addr learned my addr 172.21.15.96:0/684157599 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:16.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.882+0000 7fbdc3e00640 1 --2- 172.21.15.96:0/684157599 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbdc40ad130 0x7fbdc40b0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:16.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.882+0000 7fbdc9200640 1 --2- 172.21.15.96:0/684157599 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbdc406f860 0x7fbdc40c5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:16.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.882+0000 7fbdc9c00640 1 -- 172.21.15.96:0/684157599 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbdc40ad130 msgr2=0x7fbdc40b0d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:16.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.882+0000 7fbdc9c00640 1 --2- 172.21.15.96:0/684157599 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbdc40ad130 0x7fbdc40b0d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:16.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.882+0000 7fbdc9c00640 1 -- 172.21.15.96:0/684157599 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbdc406f860 msgr2=0x7fbdc40c5510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:16.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.882+0000 7fbdc9c00640 1 --2- 172.21.15.96:0/684157599 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbdc406f860 0x7fbdc40c5510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:16.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.882+0000 7fbdc9c00640 1 -- 172.21.15.96:0/684157599 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdc415e810 con 0x7fbdc40d04a0 2024-05-28T06:04:16.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.883+0000 7fbdc9c00640 1 --2- 172.21.15.96:0/684157599 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbdc40d04a0 0x7fbdc40b12a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fbdbc00d830 tx=0x7fbdbc00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:16.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.883+0000 7fbdc0c00640 1 -- 172.21.15.96:0/684157599 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdbc00b6c0 con 0x7fbdc40d04a0 2024-05-28T06:04:16.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.883+0000 7fbdc0c00640 1 -- 172.21.15.96:0/684157599 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbdbc019070 con 0x7fbdc40d04a0 2024-05-28T06:04:16.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.883+0000 7fbdc0c00640 1 -- 172.21.15.96:0/684157599 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdbc00bac0 con 0x7fbdc40d04a0 2024-05-28T06:04:16.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.883+0000 7fbdcbc00640 1 -- 172.21.15.96:0/684157599 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbdc415eaf0 con 0x7fbdc40d04a0 2024-05-28T06:04:16.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.883+0000 7fbdcbc00640 1 -- 172.21.15.96:0/684157599 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbdc415f0b0 con 0x7fbdc40d04a0 2024-05-28T06:04:16.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.884+0000 7fbdcbc00640 1 -- 172.21.15.96:0/684157599 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbdc40689e0 con 0x7fbdc40d04a0 2024-05-28T06:04:16.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.885+0000 7fbdc0c00640 1 -- 172.21.15.96:0/684157599 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fbdbc01e460 con 0x7fbdc40d04a0 2024-05-28T06:04:16.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.886+0000 7fbdc0c00640 1 --2- 172.21.15.96:0/684157599 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbda006c8f0 0x7fbda006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:16.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.886+0000 7fbdc0c00640 1 -- 172.21.15.96:0/684157599 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fbdbc08c310 con 0x7fbdc40d04a0 2024-05-28T06:04:16.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.887+0000 7fbdc9200640 1 --2- 172.21.15.96:0/684157599 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbda006c8f0 0x7fbda006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:16.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.888+0000 7fbdc9200640 1 --2- 172.21.15.96:0/684157599 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbda006c8f0 0x7fbda006edb0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fbdb40012e0 tx=0x7fbdb4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:16.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:16.891+0000 7fbdc0c00640 1 -- 172.21.15.96:0/684157599 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbdbc05a6a0 con 0x7fbdc40d04a0 2024-05-28T06:04:17.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.091+0000 7fbdcbc00640 1 -- 172.21.15.96:0/684157599 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbdc406e2f0 con 0x7fbdc40d04a0 2024-05-28T06:04:17.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.092+0000 7fbdc0c00640 1 -- 172.21.15.96:0/684157599 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16617 (secure 0 0 0) 0x7fbdbc059fa0 con 0x7fbdc40d04a0 2024-05-28T06:04:17.093 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:04:17.093 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":20,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:04:11.882922+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-28T06:04:08.741938+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-28T06:04:09.301470+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:04:17.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.095+0000 7fbdcbc00640 1 -- 172.21.15.96:0/684157599 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbda006c8f0 msgr2=0x7fbda006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:17.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.095+0000 7fbdcbc00640 1 --2- 172.21.15.96:0/684157599 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbda006c8f0 0x7fbda006edb0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fbdb40012e0 tx=0x7fbdb4009040 comp rx=0 tx=0).stop 2024-05-28T06:04:17.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.095+0000 7fbdcbc00640 1 -- 172.21.15.96:0/684157599 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbdc40d04a0 msgr2=0x7fbdc40b12a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:17.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.095+0000 7fbdcbc00640 1 --2- 172.21.15.96:0/684157599 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbdc40d04a0 0x7fbdc40b12a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fbdbc00d830 tx=0x7fbdbc00dcf0 comp rx=0 tx=0).stop 2024-05-28T06:04:17.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.095+0000 7fbdcbc00640 1 -- 172.21.15.96:0/684157599 shutdown_connections 2024-05-28T06:04:17.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.095+0000 7fbdcbc00640 1 --2- 172.21.15.96:0/684157599 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbda006c8f0 0x7fbda006edb0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:17.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.095+0000 7fbdcbc00640 1 --2- 172.21.15.96:0/684157599 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbdc40d04a0 0x7fbdc40b12a0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:17.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.095+0000 7fbdcbc00640 1 --2- 172.21.15.96:0/684157599 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbdc40ad130 0x7fbdc40b0d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:17.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.095+0000 7fbdcbc00640 1 --2- 172.21.15.96:0/684157599 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbdc406f860 0x7fbdc40c5510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:17.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.095+0000 7fbdcbc00640 1 -- 172.21.15.96:0/684157599 >> 172.21.15.96:0/684157599 conn(0x7fbdc406bae0 msgr2=0x7fbdc40adc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:17.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.095+0000 7fbdcbc00640 1 -- 172.21.15.96:0/684157599 shutdown_connections 2024-05-28T06:04:17.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.095+0000 7fbdcbc00640 1 -- 172.21.15.96:0/684157599 wait complete. 2024-05-28T06:04:17.114 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-05-28T06:04:17.114 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:04:17.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.316+0000 7f8d8c600640 1 Processor -- start 2024-05-28T06:04:17.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.316+0000 7f8d8c600640 1 -- start start 2024-05-28T06:04:17.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.318+0000 7f8d8c600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8d840b7f80 0x7f8d840d1280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:17.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.318+0000 7f8d8c600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f8d840d17c0 0x7f8d8409aa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:17.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.318+0000 7f8d8c600640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d840ccd00 con 0x7f8d8409af70 2024-05-28T06:04:17.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.318+0000 7f8d8c600640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f8d840cedb0 con 0x7f8d840d17c0 2024-05-28T06:04:17.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.318+0000 7f8d8c600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f8d840ccfe0 con 0x7f8d840b7f80 2024-05-28T06:04:17.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.318+0000 7f8d8a600640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f8d8409af70 0x7f8d8415e160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:55444/0 (socket says 172.21.15.96:55444) 2024-05-28T06:04:17.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.318+0000 7f8d8a600640 1 -- 172.21.15.96:0/3983761250 learned_addr learned my addr 172.21.15.96:0/3983761250 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:04:17.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.318+0000 7f8d89200640 1 --2- 172.21.15.96:0/3983761250 >> v2:172.21.15.123:3300/0 conn(0x7f8d840d17c0 0x7f8d8409aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:17.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.319+0000 7f8d89c00640 1 --2- 172.21.15.96:0/3983761250 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8d840b7f80 0x7f8d840d1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:17.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.319+0000 7f8d83e00640 1 -- 172.21.15.96:0/3983761250 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1844998428 0 0) 0x7f8d840ccd00 con 0x7f8d8409af70 2024-05-28T06:04:17.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.319+0000 7f8d83e00640 1 -- 172.21.15.96:0/3983761250 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8d68003520 con 0x7f8d8409af70 2024-05-28T06:04:17.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.319+0000 7f8d83e00640 1 -- 172.21.15.96:0/3983761250 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2217843665 0 0) 0x7f8d68003520 con 0x7f8d8409af70 2024-05-28T06:04:17.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.319+0000 7f8d83e00640 1 -- 172.21.15.96:0/3983761250 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8d840b7f80 msgr2=0x7f8d840d1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:17.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.319+0000 7f8d83e00640 1 --2- 172.21.15.96:0/3983761250 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8d840b7f80 0x7f8d840d1280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:17.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.319+0000 7f8d83e00640 1 -- 172.21.15.96:0/3983761250 >> v2:172.21.15.123:3300/0 conn(0x7f8d840d17c0 msgr2=0x7f8d8409aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:17.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.319+0000 7f8d83e00640 1 --2- 172.21.15.96:0/3983761250 >> v2:172.21.15.123:3300/0 conn(0x7f8d840d17c0 0x7f8d8409aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:17.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.319+0000 7f8d83e00640 1 -- 172.21.15.96:0/3983761250 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d840cd2c0 con 0x7f8d8409af70 2024-05-28T06:04:17.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.319+0000 7f8d83e00640 1 -- 172.21.15.96:0/3983761250 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f8d7c001670 con 0x7f8d8409af70 2024-05-28T06:04:17.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.319+0000 7f8d83e00640 1 -- 172.21.15.96:0/3983761250 >> v1:172.21.15.96:6789/0 conn(0x7f8d8409af70 legacy=0x7f8d8415e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:17.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.319+0000 7f8d83e00640 1 --2- 172.21.15.96:0/3983761250 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8d840b7f80 0x7f8d680039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:17.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.319+0000 7f8d83e00640 1 --2- 172.21.15.96:0/3983761250 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8d68003f30 0x7f8d68004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:17.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.319+0000 7f8d83e00640 1 --2- 172.21.15.96:0/3983761250 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d68004d40 0x7f8d680051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:17.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.319+0000 7f8d83e00640 1 -- 172.21.15.96:0/3983761250 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f8d840ccfe0 con 0x7f8d68004d40 2024-05-28T06:04:17.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.319+0000 7f8d83e00640 1 -- 172.21.15.96:0/3983761250 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f8d6800b9c0 con 0x7f8d68003f30 2024-05-28T06:04:17.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.319+0000 7f8d83e00640 1 -- 172.21.15.96:0/3983761250 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f8d6800bb40 con 0x7f8d840b7f80 2024-05-28T06:04:17.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.320+0000 7f8d8a600640 1 --2- 172.21.15.96:0/3983761250 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d68004d40 0x7f8d680051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:17.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.320+0000 7f8d89c00640 1 --2- 172.21.15.96:0/3983761250 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8d840b7f80 0x7f8d680039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:17.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.320+0000 7f8d89200640 1 --2- 172.21.15.96:0/3983761250 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8d68003f30 0x7f8d68004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:17.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.320+0000 7f8d8a600640 1 -- 172.21.15.96:0/3983761250 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8d840b7f80 msgr2=0x7f8d680039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:17.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.320+0000 7f8d8a600640 1 --2- 172.21.15.96:0/3983761250 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8d840b7f80 0x7f8d680039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:17.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.320+0000 7f8d8a600640 1 -- 172.21.15.96:0/3983761250 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8d68003f30 msgr2=0x7f8d68004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:17.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.320+0000 7f8d8a600640 1 --2- 172.21.15.96:0/3983761250 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8d68003f30 0x7f8d68004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:17.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.320+0000 7f8d8a600640 1 -- 172.21.15.96:0/3983761250 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d68003520 con 0x7f8d68004d40 2024-05-28T06:04:17.328 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.321+0000 7f8d8a600640 1 --2- 172.21.15.96:0/3983761250 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d68004d40 0x7f8d680051c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c001050 tx=0x7f8d68005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddc5a7603fcc375e server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:17.328 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.321+0000 7f8d83e00640 1 -- 172.21.15.96:0/3983761250 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d7c007d70 con 0x7f8d68004d40 2024-05-28T06:04:17.328 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.321+0000 7f8d83e00640 1 -- 172.21.15.96:0/3983761250 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d7c005af0 con 0x7f8d68004d40 2024-05-28T06:04:17.328 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.321+0000 7f8d83e00640 1 -- 172.21.15.96:0/3983761250 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d7c007070 con 0x7f8d68004d40 2024-05-28T06:04:17.328 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.322+0000 7f8d8c600640 1 -- 172.21.15.96:0/3983761250 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d68004d40 msgr2=0x7f8d680051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:17.328 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.322+0000 7f8d8c600640 1 --2- 172.21.15.96:0/3983761250 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d68004d40 0x7f8d680051c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c001050 tx=0x7f8d68005830 comp rx=0 tx=0).stop 2024-05-28T06:04:17.328 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.322+0000 7f8d8a600640 1 -- 172.21.15.96:0/3983761250 reap_dead start 2024-05-28T06:04:17.328 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.322+0000 7f8d8c600640 1 -- 172.21.15.96:0/3983761250 shutdown_connections 2024-05-28T06:04:17.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.322+0000 7f8d8c600640 1 -- 172.21.15.96:0/3983761250 >> 172.21.15.96:0/3983761250 conn(0x7f8d840b3fe0 msgr2=0x7f8d840b43f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:17.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.322+0000 7f8d8c600640 1 -- 172.21.15.96:0/3983761250 shutdown_connections 2024-05-28T06:04:17.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.322+0000 7f8d8c600640 1 -- 172.21.15.96:0/3983761250 wait complete. 2024-05-28T06:04:17.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.324+0000 7f8d8c600640 1 Processor -- start 2024-05-28T06:04:17.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.324+0000 7f8d8c600640 1 -- start start 2024-05-28T06:04:17.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.325+0000 7f8d8c600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8d84149010 0x7f8d8414b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:17.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.325+0000 7f8d8c600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8d841493f0 0x7f8d8414f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:17.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.325+0000 7f8d8c600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d84149920 0x7f8d84159880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:17.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.325+0000 7f8d8c600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f8d840b8360 con 0x7f8d84149920 2024-05-28T06:04:17.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.325+0000 7f8d8c600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f8d840cd880 con 0x7f8d841493f0 2024-05-28T06:04:17.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.325+0000 7f8d8c600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f8d840b5340 con 0x7f8d84149010 2024-05-28T06:04:17.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.325+0000 7f8d8a600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d84149920 0x7f8d84159880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:17.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.325+0000 7f8d8a600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d84149920 0x7f8d84159880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:41566/0 (socket says 172.21.15.96:41566) 2024-05-28T06:04:17.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.325+0000 7f8d8a600640 1 -- 172.21.15.96:0/3409178293 learned_addr learned my addr 172.21.15.96:0/3409178293 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:17.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.325+0000 7f8d89c00640 1 --2- 172.21.15.96:0/3409178293 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8d84149010 0x7f8d8414b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:17.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.325+0000 7f8d89200640 1 --2- 172.21.15.96:0/3409178293 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8d841493f0 0x7f8d8414f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:17.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.325+0000 7f8d8a600640 1 -- 172.21.15.96:0/3409178293 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8d84149010 msgr2=0x7f8d8414b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:17.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.325+0000 7f8d8a600640 1 --2- 172.21.15.96:0/3409178293 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8d84149010 0x7f8d8414b9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:17.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.325+0000 7f8d8a600640 1 -- 172.21.15.96:0/3409178293 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8d841493f0 msgr2=0x7f8d8414f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:17.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.325+0000 7f8d8a600640 1 --2- 172.21.15.96:0/3409178293 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8d841493f0 0x7f8d8414f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:17.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.325+0000 7f8d8a600640 1 -- 172.21.15.96:0/3409178293 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d8415b930 con 0x7f8d84149920 2024-05-28T06:04:17.331 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.325+0000 7f8d89c00640 1 --2- 172.21.15.96:0/3409178293 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8d84149010 0x7f8d8414b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:04:17.331 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.326+0000 7f8d89200640 1 --2- 172.21.15.96:0/3409178293 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8d841493f0 0x7f8d8414f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:04:17.331 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.326+0000 7f8d8a600640 1 --2- 172.21.15.96:0/3409178293 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d84149920 0x7f8d84159880 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c001050 tx=0x7f8d7c000fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:17.331 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.326+0000 7f8d81600640 1 -- 172.21.15.96:0/3409178293 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d7c001cc0 con 0x7f8d84149920 2024-05-28T06:04:17.331 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.326+0000 7f8d81600640 1 -- 172.21.15.96:0/3409178293 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d7c044070 con 0x7f8d84149920 2024-05-28T06:04:17.331 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.326+0000 7f8d81600640 1 -- 172.21.15.96:0/3409178293 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d7c007d50 con 0x7f8d84149920 2024-05-28T06:04:17.331 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.326+0000 7f8d8c600640 1 -- 172.21.15.96:0/3409178293 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d8414bf90 con 0x7f8d84149920 2024-05-28T06:04:17.331 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.326+0000 7f8d8c600640 1 -- 172.21.15.96:0/3409178293 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d8414c550 con 0x7f8d84149920 2024-05-28T06:04:17.331 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.327+0000 7f8d8c600640 1 -- 172.21.15.96:0/3409178293 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d840689e0 con 0x7f8d84149920 2024-05-28T06:04:17.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.328+0000 7f8d81600640 1 -- 172.21.15.96:0/3409178293 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8d7c03b040 con 0x7f8d84149920 2024-05-28T06:04:17.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.329+0000 7f8d81600640 1 --2- 172.21.15.96:0/3409178293 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8d6406c920 0x7f8d6406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:17.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.329+0000 7f8d81600640 1 -- 172.21.15.96:0/3409178293 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f8d7c0abc10 con 0x7f8d84149920 2024-05-28T06:04:17.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.330+0000 7f8d89c00640 1 --2- 172.21.15.96:0/3409178293 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8d6406c920 0x7f8d6406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:17.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.331+0000 7f8d89c00640 1 --2- 172.21.15.96:0/3409178293 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8d6406c920 0x7f8d6406ede0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f8d74007440 tx=0x7f8d7400b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:17.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.333+0000 7f8d81600640 1 -- 172.21.15.96:0/3409178293 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8d7c079fa0 con 0x7f8d84149920 2024-05-28T06:04:17.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.535+0000 7f8d8c600640 1 -- 172.21.15.96:0/3409178293 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8d840719b0 con 0x7f8d84149920 2024-05-28T06:04:17.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.538+0000 7f8d81600640 1 -- 172.21.15.96:0/3409178293 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16617 (secure 0 0 0) 0x7f8d7c0798a0 con 0x7f8d84149920 2024-05-28T06:04:17.542 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:04:17.542 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":20,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:04:11.882922+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-28T06:04:08.741938+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-28T06:04:09.301470+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:04:17.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.540+0000 7f8d8c600640 1 -- 172.21.15.96:0/3409178293 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8d6406c920 msgr2=0x7f8d6406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:17.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.540+0000 7f8d8c600640 1 --2- 172.21.15.96:0/3409178293 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8d6406c920 0x7f8d6406ede0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f8d74007440 tx=0x7f8d7400b040 comp rx=0 tx=0).stop 2024-05-28T06:04:17.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.540+0000 7f8d8c600640 1 -- 172.21.15.96:0/3409178293 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d84149920 msgr2=0x7f8d84159880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:17.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.540+0000 7f8d8c600640 1 --2- 172.21.15.96:0/3409178293 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d84149920 0x7f8d84159880 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c001050 tx=0x7f8d7c000fe0 comp rx=0 tx=0).stop 2024-05-28T06:04:17.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.541+0000 7f8d8c600640 1 -- 172.21.15.96:0/3409178293 shutdown_connections 2024-05-28T06:04:17.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.541+0000 7f8d8c600640 1 --2- 172.21.15.96:0/3409178293 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8d6406c920 0x7f8d6406ede0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:17.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.541+0000 7f8d8c600640 1 --2- 172.21.15.96:0/3409178293 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d84149920 0x7f8d84159880 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:17.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.541+0000 7f8d8c600640 1 --2- 172.21.15.96:0/3409178293 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8d841493f0 0x7f8d8414f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:17.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.541+0000 7f8d8c600640 1 --2- 172.21.15.96:0/3409178293 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8d84149010 0x7f8d8414b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:17.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.541+0000 7f8d8c600640 1 -- 172.21.15.96:0/3409178293 >> 172.21.15.96:0/3409178293 conn(0x7f8d840b3fe0 msgr2=0x7f8d840b5640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:17.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.541+0000 7f8d8c600640 1 -- 172.21.15.96:0/3409178293 shutdown_connections 2024-05-28T06:04:17.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.541+0000 7f8d8c600640 1 -- 172.21.15.96:0/3409178293 wait complete. 2024-05-28T06:04:17.561 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-05-28T06:04:17.561 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-05-28T06:04:17.562 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-05-28T06:04:17.562 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-05-28T06:04:17.562 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-05-28T06:04:17.563 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-05-28T06:04:17.563 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-05-28T06:04:17.563 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-05-28T06:04:17.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.840+0000 7f96e3400640 1 Processor -- start 2024-05-28T06:04:17.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.840+0000 7f96e3400640 1 -- start start 2024-05-28T06:04:17.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.842+0000 7f96e3400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f96dc15ea40 0x7f96dc0d4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:17.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.842+0000 7f96e3400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f96dc0d1490 0x7f96dc162c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:17.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.842+0000 7f96e3400640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96dc0cf7e0 con 0x7f96dc0d1040 2024-05-28T06:04:17.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.842+0000 7f96e3400640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f96dc15ee20 con 0x7f96dc15ea40 2024-05-28T06:04:17.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.842+0000 7f96e3400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f96dc15fe50 con 0x7f96dc0d1490 2024-05-28T06:04:17.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.843+0000 7f96e0a00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f96dc15ea40 0x7f96dc0d4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:17.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.843+0000 7f96dbe00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f96dc0d1040 0x7f96dc162520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:55446/0 (socket says 172.21.15.96:55446) 2024-05-28T06:04:17.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.844+0000 7f96dbe00640 1 -- 172.21.15.96:0/1059256663 learned_addr learned my addr 172.21.15.96:0/1059256663 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:04:17.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.844+0000 7f96e1400640 1 --2- 172.21.15.96:0/1059256663 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f96dc0d1490 0x7f96dc162c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:17.847 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.845+0000 7f96e0a00640 1 -- 172.21.15.96:0/1059256663 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f96dc0d1490 msgr2=0x7f96dc162c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:17.847 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.845+0000 7f96e0a00640 1 --2- 172.21.15.96:0/1059256663 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f96dc0d1490 0x7f96dc162c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:17.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.845+0000 7f96e0a00640 1 -- 172.21.15.96:0/1059256663 >> v1:172.21.15.96:6789/0 conn(0x7f96dc0d1040 legacy=0x7f96dc162520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:17.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.845+0000 7f96db400640 1 -- 172.21.15.96:0/1059256663 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 830702394 0 0) 0x7f96dc0cf7e0 con 0x7f96dc0d1040 2024-05-28T06:04:17.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.845+0000 7f96e0a00640 1 -- 172.21.15.96:0/1059256663 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96dc0cfa80 con 0x7f96dc15ea40 2024-05-28T06:04:17.850 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.847+0000 7f96e0a00640 1 --2- 172.21.15.96:0/1059256663 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f96dc15ea40 0x7f96dc0d4f00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f96cc0097a0 tx=0x7f96cc02a100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d4d3b6c3dbd7ba3 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:17.850 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.848+0000 7f96db400640 1 -- 172.21.15.96:0/1059256663 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96cc037070 con 0x7f96dc15ea40 2024-05-28T06:04:17.850 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.848+0000 7f96db400640 1 -- 172.21.15.96:0/1059256663 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96cc02ae50 con 0x7f96dc15ea40 2024-05-28T06:04:17.851 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.848+0000 7f96db400640 1 -- 172.21.15.96:0/1059256663 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96cc037070 con 0x7f96dc15ea40 2024-05-28T06:04:17.851 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.849+0000 7f96e3400640 1 -- 172.21.15.96:0/1059256663 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f96dc15ea40 msgr2=0x7f96dc0d4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:17.851 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.849+0000 7f96e3400640 1 --2- 172.21.15.96:0/1059256663 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f96dc15ea40 0x7f96dc0d4f00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f96cc0097a0 tx=0x7f96cc02a100 comp rx=0 tx=0).stop 2024-05-28T06:04:17.851 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.850+0000 7f96e3400640 1 -- 172.21.15.96:0/1059256663 shutdown_connections 2024-05-28T06:04:17.852 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.850+0000 7f96e3400640 1 --2- 172.21.15.96:0/1059256663 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f96dc0d1490 0x7f96dc162c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:17.852 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.850+0000 7f96e3400640 1 --2- 172.21.15.96:0/1059256663 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f96dc15ea40 0x7f96dc0d4f00 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:17.852 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.850+0000 7f96e3400640 1 -- 172.21.15.96:0/1059256663 >> 172.21.15.96:0/1059256663 conn(0x7f96dc08bfc0 msgr2=0x7f96dc0d4630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:17.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.851+0000 7f96e3400640 1 -- 172.21.15.96:0/1059256663 shutdown_connections 2024-05-28T06:04:17.854 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.851+0000 7f96e3400640 1 -- 172.21.15.96:0/1059256663 wait complete. 2024-05-28T06:04:17.855 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.854+0000 7f96e3400640 1 Processor -- start 2024-05-28T06:04:17.855 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.854+0000 7f96e3400640 1 -- start start 2024-05-28T06:04:17.857 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.855+0000 7f96e3400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f96dc0d1040 0x7f96dc0ce8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:17.858 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.855+0000 7f413da00640 1 Processor -- start 2024-05-28T06:04:17.858 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.856+0000 7f413da00640 1 -- start start 2024-05-28T06:04:17.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.856+0000 7f96e3400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f96dc0d1490 0x7f96dc15cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:17.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.856+0000 7f96e3400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f96dc15ea40 0x7f96dc0ca310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:17.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.856+0000 7f96e3400640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f96dc0d0320 con 0x7f96dc15ea40 2024-05-28T06:04:17.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.856+0000 7f96e3400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f96dc0d0040 con 0x7f96dc0d1040 2024-05-28T06:04:17.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.856+0000 7f96e3400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f96dc0d18e0 con 0x7f96dc0d1490 2024-05-28T06:04:17.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.856+0000 7f96e1400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f96dc15ea40 0x7f96dc0ca310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:17.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.856+0000 7f96e1400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f96dc15ea40 0x7f96dc0ca310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:41570/0 (socket says 172.21.15.96:41570) 2024-05-28T06:04:17.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.856+0000 7f96e1400640 1 -- 172.21.15.96:0/3483519320 learned_addr learned my addr 172.21.15.96:0/3483519320 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:17.860 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.856+0000 7f96dbe00640 1 --2- 172.21.15.96:0/3483519320 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f96dc0d1490 0x7f96dc15cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:17.860 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.857+0000 7f29b0c00640 1 Processor -- start 2024-05-28T06:04:17.860 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.857+0000 7f96e1400640 1 -- 172.21.15.96:0/3483519320 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f96dc0d1490 msgr2=0x7f96dc15cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:17.860 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.857+0000 7f96e1400640 1 --2- 172.21.15.96:0/3483519320 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f96dc0d1490 0x7f96dc15cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:17.860 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.857+0000 7f96e1400640 1 -- 172.21.15.96:0/3483519320 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f96dc0d1040 msgr2=0x7f96dc0ce8d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-28T06:04:17.860 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.857+0000 7f96e1400640 1 --2- 172.21.15.96:0/3483519320 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f96dc0d1040 0x7f96dc0ce8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:17.860 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.857+0000 7f96e1400640 1 -- 172.21.15.96:0/3483519320 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96dc0b9590 con 0x7f96dc15ea40 2024-05-28T06:04:17.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.858+0000 7f413da00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f4138163170 0x7f413809b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:17.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.858+0000 7f413da00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f41380d07e0 0x7f41380d0c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:17.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.857+0000 7f29b0c00640 1 -- start start 2024-05-28T06:04:17.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.858+0000 7f413da00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f413806d210 con 0x7f4138162480 2024-05-28T06:04:17.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.858+0000 7f413da00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f413806cd80 con 0x7f4138163170 2024-05-28T06:04:17.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.858+0000 7f413da00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f413806d4c0 con 0x7f41380d07e0 2024-05-28T06:04:17.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.859+0000 7f4137400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f41380d07e0 0x7f41380d0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:17.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.859+0000 7f4137400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f41380d07e0 0x7f41380d0c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.96:58378/0 (socket says 172.21.15.96:58378) 2024-05-28T06:04:17.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.859+0000 7f4137400640 1 -- 172.21.15.96:0/3730781912 learned_addr learned my addr 172.21.15.96:0/3730781912 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:17.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.858+0000 7f4136000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f4138163170 0x7f413809b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:17.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.859+0000 7f29b0c00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f29a4006200 0x7f29a4056fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:17.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.858+0000 7f4136a00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f4138162480 0x7f4138162860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.96:41578/0 (socket says 172.21.15.96:41578) 2024-05-28T06:04:17.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.860+0000 7f4137400640 1 -- 172.21.15.96:0/3730781912 >> v2:172.21.15.123:3300/0 conn(0x7f4138163170 msgr2=0x7f413809b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:17.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.860+0000 7f4137400640 1 --2- 172.21.15.96:0/3730781912 >> v2:172.21.15.123:3300/0 conn(0x7f4138163170 0x7f413809b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:17.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.860+0000 7f4137400640 1 -- 172.21.15.96:0/3730781912 >> v1:172.21.15.123:6789/0 conn(0x7f4138162480 legacy=0x7f4138162860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:17.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.860+0000 7f4137400640 1 -- 172.21.15.96:0/3730781912 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f413806d760 con 0x7f41380d07e0 2024-05-28T06:04:17.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.860+0000 7f29ab400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f29a4006200 0x7f29a4056fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:17.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.860+0000 7f29b0c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f29a40075f0 0x7f29a403c720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:17.865 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.860+0000 7f29b0c00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29a4057a70 con 0x7f29a40574f0 2024-05-28T06:04:17.865 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.860+0000 7f29b0c00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f29a4057d50 con 0x7f29a4006200 2024-05-28T06:04:17.865 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.860+0000 7f29ab400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f29a4006200 0x7f29a4056fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:41580/0 (socket says 172.21.15.96:41580) 2024-05-28T06:04:17.865 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.861+0000 7f96e1400640 1 --2- 172.21.15.96:0/3483519320 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f96dc15ea40 0x7f96dc0ca310 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f96c800e930 tx=0x7f96c800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:17.865 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.860+0000 7f29ab400640 1 -- 172.21.15.96:0/3143593544 learned_addr learned my addr 172.21.15.96:0/3143593544 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:17.865 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.860+0000 7f29b0c00640 1 -- 172.21.15.96:0/3143593544 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f29a4058030 con 0x7f29a40075f0 2024-05-28T06:04:17.866 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.861+0000 7f4f5e400640 1 Processor -- start 2024-05-28T06:04:17.866 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.861+0000 7f4f5e400640 1 -- start start 2024-05-28T06:04:17.866 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.861+0000 7f96d8c00640 1 -- 172.21.15.96:0/3483519320 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96c8017070 con 0x7f96dc15ea40 2024-05-28T06:04:17.867 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.862+0000 7f96d8c00640 1 -- 172.21.15.96:0/3483519320 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96c800c920 con 0x7f96dc15ea40 2024-05-28T06:04:17.867 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.862+0000 7f96d8c00640 1 -- 172.21.15.96:0/3483519320 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96c800cd10 con 0x7f96dc15ea40 2024-05-28T06:04:17.867 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.862+0000 7f4137400640 1 --2- 172.21.15.96:0/3730781912 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f41380d07e0 0x7f41380d0c70 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4124009870 tx=0x7f412402a250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7cdf8a757c79eecc server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:17.867 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.861+0000 7f29abe00640 1 --2- 172.21.15.96:0/3143593544 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f29a40075f0 0x7f29a403c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:17.867 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.862+0000 7f29aa000640 1 -- 172.21.15.96:0/3143593544 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3610567210 0 0) 0x7f29a4057a70 con 0x7f29a40574f0 2024-05-28T06:04:17.867 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.862+0000 7f29aa000640 1 -- 172.21.15.96:0/3143593544 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2994003520 con 0x7f29a40574f0 2024-05-28T06:04:17.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.862+0000 7f96e3400640 1 -- 172.21.15.96:0/3483519320 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96dc0b9870 con 0x7f96dc15ea40 2024-05-28T06:04:17.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.862+0000 7f96e3400640 1 -- 172.21.15.96:0/3483519320 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96dc0b9e30 con 0x7f96dc15ea40 2024-05-28T06:04:17.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.862+0000 7f29abe00640 1 -- 172.21.15.96:0/3143593544 >> v2:172.21.15.96:3300/0 conn(0x7f29a4006200 msgr2=0x7f29a4056fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:17.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.862+0000 7f29abe00640 1 --2- 172.21.15.96:0/3143593544 >> v2:172.21.15.96:3300/0 conn(0x7f29a4006200 0x7f29a4056fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:17.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.862+0000 7f29abe00640 1 -- 172.21.15.96:0/3143593544 >> v1:172.21.15.123:6789/0 conn(0x7f29a40574f0 legacy=0x7f29a403c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:17.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.862+0000 7f29abe00640 1 -- 172.21.15.96:0/3143593544 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29a4058310 con 0x7f29a40075f0 2024-05-28T06:04:17.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.863+0000 7f4135600640 1 -- 172.21.15.96:0/3730781912 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4124037070 con 0x7f41380d07e0 2024-05-28T06:04:17.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.863+0000 7f4135600640 1 -- 172.21.15.96:0/3730781912 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4124032440 con 0x7f41380d07e0 2024-05-28T06:04:17.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.863+0000 7f4135600640 1 -- 172.21.15.96:0/3730781912 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4124032810 con 0x7f41380d07e0 2024-05-28T06:04:17.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.864+0000 7f29abe00640 1 --2- 172.21.15.96:0/3143593544 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f29a40075f0 0x7f29a403c720 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f29ac02db50 tx=0x7f29ac0a5ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=96787bf72645e1ef server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:17.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.864+0000 7f96e3400640 1 -- 172.21.15.96:0/3483519320 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f96dc0d0560 con 0x7f96dc15ea40 2024-05-28T06:04:17.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.864+0000 7f29aa000640 1 -- 172.21.15.96:0/3143593544 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29ac0a7980 con 0x7f29a40075f0 2024-05-28T06:04:17.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.865+0000 7f29aa000640 1 -- 172.21.15.96:0/3143593544 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29ac0a7b20 con 0x7f29a40075f0 2024-05-28T06:04:17.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.865+0000 7f413da00640 1 -- 172.21.15.96:0/3730781912 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f41380d07e0 msgr2=0x7f41380d0c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:17.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.865+0000 7f413da00640 1 --2- 172.21.15.96:0/3730781912 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f41380d07e0 0x7f41380d0c70 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4124009870 tx=0x7f412402a250 comp rx=0 tx=0).stop 2024-05-28T06:04:17.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.864+0000 7f4f5e400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f4f58162d20 0x7f4f58071bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:17.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.864+0000 7f4f5e400640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f5806f590 con 0x7f4f58072100 2024-05-28T06:04:17.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.864+0000 7f4f5e400640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f5806f870 con 0x7f4f58162940 2024-05-28T06:04:17.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.864+0000 7f4f5e400640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f4f58072530 con 0x7f4f58162d20 2024-05-28T06:04:17.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.865+0000 7f4f5d000640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f4f58162940 0x7f4f5806eb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.96:41588/0 (socket says 172.21.15.96:41588) 2024-05-28T06:04:17.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.865+0000 7f4f5d000640 1 -- 172.21.15.96:0/3551824148 learned_addr learned my addr 172.21.15.96:0/3551824148 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:04:17.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.865+0000 7f4f57e00640 1 --2- 172.21.15.96:0/3551824148 >> v2:172.21.15.96:3300/0 conn(0x7f4f58162d20 0x7f4f58071bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:17.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.865+0000 7f220ce00640 1 Processor -- start 2024-05-28T06:04:17.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.865+0000 7f29aa000640 1 -- 172.21.15.96:0/3143593544 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29ac0af420 con 0x7f29a40075f0 2024-05-28T06:04:17.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.865+0000 7f413da00640 1 -- 172.21.15.96:0/3730781912 shutdown_connections 2024-05-28T06:04:17.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.865+0000 7f413da00640 1 --2- 172.21.15.96:0/3730781912 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f41380d07e0 0x7f41380d0c70 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:17.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.865+0000 7f413da00640 1 --2- 172.21.15.96:0/3730781912 >> v2:172.21.15.123:3300/0 conn(0x7f4138163170 0x7f413809b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:17.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.865+0000 7f413da00640 1 -- 172.21.15.96:0/3730781912 >> 172.21.15.96:0/3730781912 conn(0x7f41380b4000 msgr2=0x7f41380b4410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:17.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.866+0000 7f29b0c00640 1 -- 172.21.15.96:0/3143593544 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f29a40075f0 msgr2=0x7f29a403c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:17.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.866+0000 7f29b0c00640 1 --2- 172.21.15.96:0/3143593544 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f29a40075f0 0x7f29a403c720 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f29ac02db50 tx=0x7f29ac0a5ba0 comp rx=0 tx=0).stop 2024-05-28T06:04:17.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.866+0000 7f4f57e00640 1 -- 172.21.15.96:0/3551824148 >> v1:172.21.15.123:6789/0 conn(0x7f4f58162940 legacy=0x7f4f5806eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:17.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.866+0000 7f4f57e00640 1 -- 172.21.15.96:0/3551824148 >> v1:172.21.15.96:6789/0 conn(0x7f4f58072100 legacy=0x7f4f580adbb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-28T06:04:17.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.866+0000 7f4f57e00640 1 -- 172.21.15.96:0/3551824148 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f580ad3d0 con 0x7f4f58162d20 2024-05-28T06:04:17.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.866+0000 7f96d8c00640 1 -- 172.21.15.96:0/3483519320 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f96c801a070 con 0x7f96dc15ea40 2024-05-28T06:04:17.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.866+0000 7f413da00640 1 -- 172.21.15.96:0/3730781912 shutdown_connections 2024-05-28T06:04:17.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.866+0000 7f413da00640 1 -- 172.21.15.96:0/3730781912 wait complete. 2024-05-28T06:04:17.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.865+0000 7f220ce00640 1 -- start start 2024-05-28T06:04:17.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.867+0000 7f29b0c00640 1 -- 172.21.15.96:0/3143593544 shutdown_connections 2024-05-28T06:04:17.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.867+0000 7f29b0c00640 1 --2- 172.21.15.96:0/3143593544 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f29a40075f0 0x7f29a403c720 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:17.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.867+0000 7f29b0c00640 1 --2- 172.21.15.96:0/3143593544 >> v2:172.21.15.96:3300/0 conn(0x7f29a4006200 0x7f29a4056fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:17.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.867+0000 7f29b0c00640 1 -- 172.21.15.96:0/3143593544 >> 172.21.15.96:0/3143593544 conn(0x7f29a4058ee0 msgr2=0x7f29a40592f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:17.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.867+0000 7f220ce00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f2208162940 0x7f220806eb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:17.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.867+0000 7f220ce00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f220806f590 con 0x7f22080722d0 2024-05-28T06:04:17.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.867+0000 7f220ce00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f220806f870 con 0x7f2208162d20 2024-05-28T06:04:17.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.867+0000 7f220ce00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f22080ad3d0 con 0x7f2208162940 2024-05-28T06:04:17.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.868+0000 7f96d8c00640 1 --2- 172.21.15.96:0/3483519320 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f96bc06c9c0 0x7f96bc06ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:17.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.868+0000 7f96d8c00640 1 -- 172.21.15.96:0/3483519320 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f96c808bdf0 con 0x7f96dc15ea40 2024-05-28T06:04:17.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.868+0000 7f96d8c00640 1 --2- 172.21.15.96:0/3483519320 >> [v2:172.21.15.96:6809/1243655428,v1:172.21.15.96:6811/1243655428] conn(0x7f96bc074be0 0x7f96bc077000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:17.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.868+0000 7f96d8c00640 1 -- 172.21.15.96:0/3483519320 --> [v2:172.21.15.96:6809/1243655428,v1:172.21.15.96:6811/1243655428] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f96c8026e50 con 0x7f96bc074be0 2024-05-28T06:04:17.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.868+0000 7f96d8c00640 1 -- 172.21.15.96:0/3483519320 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f96c808c1e0 con 0x7f96dc15ea40 2024-05-28T06:04:17.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.868+0000 7f2207400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f2208162940 0x7f220806eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:17.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.868+0000 7f4f57e00640 1 --2- 172.21.15.96:0/3551824148 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f58162d20 0x7f4f58071bc0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4f480097a0 tx=0x7f4f4802a0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e34e4b8cc562fb5 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:17.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.868+0000 7f2207e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f22080722d0 0x7f22080adbb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:55462/0 (socket says 172.21.15.96:55462) 2024-05-28T06:04:17.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.868+0000 7f2207e00640 1 -- 172.21.15.96:0/3455755852 learned_addr learned my addr 172.21.15.96:0/3455755852 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:04:17.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.869+0000 7f96dbe00640 1 --2- 172.21.15.96:0/3483519320 >> [v2:172.21.15.96:6809/1243655428,v1:172.21.15.96:6811/1243655428] conn(0x7f96bc074be0 0x7f96bc077000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:17.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.869+0000 7f96e0a00640 1 --2- 172.21.15.96:0/3483519320 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f96bc06c9c0 0x7f96bc06ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:17.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.869+0000 7f413da00640 1 Processor -- start 2024-05-28T06:04:17.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.869+0000 7f29b0c00640 1 -- 172.21.15.96:0/3143593544 shutdown_connections 2024-05-28T06:04:17.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.869+0000 7f96dbe00640 1 --2- 172.21.15.96:0/3483519320 >> [v2:172.21.15.96:6809/1243655428,v1:172.21.15.96:6811/1243655428] conn(0x7f96bc074be0 0x7f96bc077000 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:17.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.870+0000 7f2206000640 1 -- 172.21.15.96:0/3455755852 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3708086614 0 0) 0x7f220806f870 con 0x7f2208162d20 2024-05-28T06:04:17.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.870+0000 7f413da00640 1 -- start start 2024-05-28T06:04:17.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.870+0000 7f2206000640 1 -- 172.21.15.96:0/3455755852 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f21ec003520 con 0x7f2208162d20 2024-05-28T06:04:17.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.870+0000 7f2206000640 1 -- 172.21.15.96:0/3455755852 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 45422183 0 0) 0x7f220806f590 con 0x7f22080722d0 2024-05-28T06:04:17.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.870+0000 7f2206000640 1 -- 172.21.15.96:0/3455755852 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f220806f870 con 0x7f22080722d0 2024-05-28T06:04:17.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.869+0000 7f4f57400640 1 -- 172.21.15.96:0/3551824148 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f48037070 con 0x7f4f58162d20 2024-05-28T06:04:17.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.869+0000 7f4f57400640 1 -- 172.21.15.96:0/3551824148 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f4802adc0 con 0x7f4f58162d20 2024-05-28T06:04:17.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.870+0000 7f4f57400640 1 -- 172.21.15.96:0/3551824148 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f48032660 con 0x7f4f58162d20 2024-05-28T06:04:17.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.870+0000 7f2207400640 1 -- 172.21.15.96:0/3455755852 >> v1:172.21.15.123:6789/0 conn(0x7f2208162d20 legacy=0x7f2208071bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:17.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.870+0000 7f2207400640 1 -- 172.21.15.96:0/3455755852 >> v1:172.21.15.96:6789/0 conn(0x7f22080722d0 legacy=0x7f22080adbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:17.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.870+0000 7f2207400640 1 -- 172.21.15.96:0/3455755852 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f220806dde0 con 0x7f2208162940 2024-05-28T06:04:17.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.871+0000 7f96e0a00640 1 --2- 172.21.15.96:0/3483519320 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f96bc06c9c0 0x7f96bc06ee80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f96cc034000 tx=0x7f96cc02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:17.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.871+0000 7f4f5e400640 1 -- 172.21.15.96:0/3551824148 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f58162d20 msgr2=0x7f4f58071bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:17.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.871+0000 7f4f5e400640 1 --2- 172.21.15.96:0/3551824148 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f58162d20 0x7f4f58071bc0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4f480097a0 tx=0x7f4f4802a0b0 comp rx=0 tx=0).stop 2024-05-28T06:04:17.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.869+0000 7f29b0c00640 1 -- 172.21.15.96:0/3143593544 wait complete. 2024-05-28T06:04:17.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.872+0000 7f413da00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41380d07e0 0x7f41380ccb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:17.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.872+0000 7f413da00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4138162480 0x7f41380b5f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:17.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.872+0000 7f4f5e400640 1 -- 172.21.15.96:0/3551824148 shutdown_connections 2024-05-28T06:04:17.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.872+0000 7f4f5e400640 1 --2- 172.21.15.96:0/3551824148 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f58162d20 0x7f4f58071bc0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:17.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.872+0000 7f4f5e400640 1 -- 172.21.15.96:0/3551824148 >> 172.21.15.96:0/3551824148 conn(0x7f4f580b1fc0 msgr2=0x7f4f580b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:17.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.872+0000 7f413da00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4138163170 0x7f41380b64c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:17.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.872+0000 7f2207400640 1 --2- 172.21.15.96:0/3455755852 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2208162940 0x7f220806eb00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f21f80097a0 tx=0x7f21f802a120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3c6dd643ac657cf1 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:17.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.872+0000 7f413da00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f413809b980 con 0x7f41380d07e0 2024-05-28T06:04:17.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.873+0000 7f413da00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f413806ca10 con 0x7f4138162480 2024-05-28T06:04:17.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.873+0000 7f413da00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f413809bb00 con 0x7f4138163170 2024-05-28T06:04:17.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.873+0000 7f2206000640 1 -- 172.21.15.96:0/3455755852 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21f8037070 con 0x7f2208162940 2024-05-28T06:04:17.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.873+0000 7f2206000640 1 -- 172.21.15.96:0/3455755852 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21f8032440 con 0x7f2208162940 2024-05-28T06:04:17.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.873+0000 7f2206000640 1 -- 172.21.15.96:0/3455755852 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21f8032850 con 0x7f2208162940 2024-05-28T06:04:17.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.873+0000 7f4136000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4138162480 0x7f41380b5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:17.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.873+0000 7f4136000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4138162480 0x7f41380b5f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.96:58460/0 (socket says 172.21.15.96:58460) 2024-05-28T06:04:17.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.873+0000 7f4136000640 1 -- 172.21.15.96:0/691666412 learned_addr learned my addr 172.21.15.96:0/691666412 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:17.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.873+0000 7f4137400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4138163170 0x7f41380b64c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:17.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.873+0000 7f4136a00640 1 --2- 172.21.15.96:0/691666412 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41380d07e0 0x7f41380ccb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:17.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.873+0000 7f96d8c00640 1 -- 172.21.15.96:0/3483519320 <== osd.0 v2:172.21.15.96:6809/1243655428 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f96c8026e50 con 0x7f96bc074be0 2024-05-28T06:04:17.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.874+0000 7f29b0c00640 1 Processor -- start 2024-05-28T06:04:17.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.874+0000 7f29b0c00640 1 -- start start 2024-05-28T06:04:17.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.874+0000 7f4f5e400640 1 -- 172.21.15.96:0/3551824148 shutdown_connections 2024-05-28T06:04:17.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.874+0000 7f4137400640 1 -- 172.21.15.96:0/691666412 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4138162480 msgr2=0x7f41380b5f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:17.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.874+0000 7f220ce00640 1 -- 172.21.15.96:0/3455755852 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2208162940 msgr2=0x7f220806eb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:17.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.874+0000 7f220ce00640 1 --2- 172.21.15.96:0/3455755852 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2208162940 0x7f220806eb00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f21f80097a0 tx=0x7f21f802a120 comp rx=0 tx=0).stop 2024-05-28T06:04:17.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.874+0000 7f4137400640 1 --2- 172.21.15.96:0/691666412 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4138162480 0x7f41380b5f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:17.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.874+0000 7f4137400640 1 -- 172.21.15.96:0/691666412 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41380d07e0 msgr2=0x7f41380ccb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:17.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.874+0000 7f4137400640 1 --2- 172.21.15.96:0/691666412 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41380d07e0 0x7f41380ccb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:17.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.874+0000 7f4137400640 1 -- 172.21.15.96:0/691666412 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41380b7d90 con 0x7f4138163170 2024-05-28T06:04:17.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.875+0000 7f4137400640 1 --2- 172.21.15.96:0/691666412 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4138163170 0x7f41380b64c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4124031000 tx=0x7f41240046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:17.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.875+0000 7f220ce00640 1 -- 172.21.15.96:0/3455755852 shutdown_connections 2024-05-28T06:04:17.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.875+0000 7f29b0c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29a4006200 0x7f29a40dca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:17.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.875+0000 7f29b0c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f29a40075f0 0x7f29a40e4180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:17.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.875+0000 7f29b0c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f29a40574f0 0x7f29a40e4c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:17.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.875+0000 7f220ce00640 1 --2- 172.21.15.96:0/3455755852 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2208162940 0x7f220806eb00 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:17.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.875+0000 7f220ce00640 1 -- 172.21.15.96:0/3455755852 >> 172.21.15.96:0/3455755852 conn(0x7f22080b1fc0 msgr2=0x7f22080b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:17.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.875+0000 7f412ea00640 1 -- 172.21.15.96:0/691666412 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4124041020 con 0x7f4138163170 2024-05-28T06:04:17.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.876+0000 7f412ea00640 1 -- 172.21.15.96:0/691666412 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f412403d6e0 con 0x7f4138163170 2024-05-28T06:04:17.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.876+0000 7f412ea00640 1 -- 172.21.15.96:0/691666412 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4124041020 con 0x7f4138163170 2024-05-28T06:04:17.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.876+0000 7f29ab400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29a4006200 0x7f29a40dca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:17.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.876+0000 7f29ab400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29a4006200 0x7f29a40dca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.96:58476/0 (socket says 172.21.15.96:58476) 2024-05-28T06:04:17.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.876+0000 7f29aaa00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f29a40075f0 0x7f29a40e4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:17.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.874+0000 7f4f5e400640 1 -- 172.21.15.96:0/3551824148 wait complete. 2024-05-28T06:04:17.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.876+0000 7f29ab400640 1 -- 172.21.15.96:0/853690032 learned_addr learned my addr 172.21.15.96:0/853690032 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:17.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.876+0000 7f29b0c00640 1 -- 172.21.15.96:0/853690032 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f29a403edd0 con 0x7f29a40574f0 2024-05-28T06:04:17.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.876+0000 7f29b0c00640 1 -- 172.21.15.96:0/853690032 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f29a40e3520 con 0x7f29a4006200 2024-05-28T06:04:17.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.876+0000 7f29b0c00640 1 -- 172.21.15.96:0/853690032 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f29a4007a80 con 0x7f29a40075f0 2024-05-28T06:04:17.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.876+0000 7f29abe00640 1 --2- 172.21.15.96:0/853690032 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f29a40574f0 0x7f29a40e4c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:17.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.876+0000 7f413da00640 1 -- 172.21.15.96:0/691666412 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41380cf310 con 0x7f4138163170 2024-05-28T06:04:17.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.876+0000 7f413da00640 1 -- 172.21.15.96:0/691666412 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f41380b6a00 con 0x7f4138163170 2024-05-28T06:04:17.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.877+0000 7f29aaa00640 1 -- 172.21.15.96:0/853690032 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29a4006200 msgr2=0x7f29a40dca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:17.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.877+0000 7f29aaa00640 1 --2- 172.21.15.96:0/853690032 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29a4006200 0x7f29a40dca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:17.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.877+0000 7f29aaa00640 1 -- 172.21.15.96:0/853690032 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f29a40574f0 msgr2=0x7f29a40e4c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:17.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.877+0000 7f29aaa00640 1 --2- 172.21.15.96:0/853690032 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f29a40574f0 0x7f29a40e4c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:17.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.877+0000 7f29aaa00640 1 -- 172.21.15.96:0/853690032 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29a40e5510 con 0x7f29a40075f0 2024-05-28T06:04:17.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.878+0000 7f4152c00640 1 Processor -- start 2024-05-28T06:04:17.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.878+0000 7f4f5e400640 1 Processor -- start 2024-05-28T06:04:17.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.877+0000 7f29abe00640 1 --2- 172.21.15.96:0/853690032 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f29a40574f0 0x7f29a40e4c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-28T06:04:17.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.878+0000 7f29aaa00640 1 --2- 172.21.15.96:0/853690032 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f29a40075f0 0x7f29a40e4180 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f299800c430 tx=0x7f299800c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:17.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.878+0000 7f299f400640 1 -- 172.21.15.96:0/853690032 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2998017070 con 0x7f29a40075f0 2024-05-28T06:04:17.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.878+0000 7f220ce00640 1 -- 172.21.15.96:0/3455755852 shutdown_connections 2024-05-28T06:04:17.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.878+0000 7f299f400640 1 -- 172.21.15.96:0/853690032 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29980052d0 con 0x7f29a40075f0 2024-05-28T06:04:17.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.878+0000 7f299f400640 1 -- 172.21.15.96:0/853690032 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2998003810 con 0x7f29a40075f0 2024-05-28T06:04:17.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.879+0000 7f4f5e400640 1 -- start start 2024-05-28T06:04:17.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.879+0000 7f413da00640 1 -- 172.21.15.96:0/691666412 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f41380689e0 con 0x7f4138163170 2024-05-28T06:04:17.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.879+0000 7f29b0c00640 1 -- 172.21.15.96:0/853690032 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29a40e57f0 con 0x7f29a40075f0 2024-05-28T06:04:17.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.879+0000 7f29b0c00640 1 -- 172.21.15.96:0/853690032 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f29a40e3980 con 0x7f29a40075f0 2024-05-28T06:04:17.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.878+0000 7f4152c00640 1 -- start start 2024-05-28T06:04:17.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.879+0000 7f4152c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f414c0d1040 0x7f414c0d1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:17.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.880+0000 7f4152c00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f414c0d19a0 con 0x7f414c06c950 2024-05-28T06:04:17.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.880+0000 7f4152c00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f414c0cf7e0 con 0x7f414c15ea40 2024-05-28T06:04:17.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.880+0000 7f4152c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f414c15ee20 con 0x7f414c0d1040 2024-05-28T06:04:17.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.880+0000 7f220ce00640 1 -- 172.21.15.96:0/3455755852 wait complete. 2024-05-28T06:04:17.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.880+0000 7f4f5e400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4f58072100 0x7f4f5815e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:17.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.880+0000 7f4f5e400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f58162940 0x7f4f58164c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:17.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.880+0000 7f4f5e400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4f58162d20 0x7f4f580cece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:17.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.880+0000 7f4f5e400640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4f5806df60 con 0x7f4f58162940 2024-05-28T06:04:17.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.880+0000 7f4f5e400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f4f5806dde0 con 0x7f4f58072100 2024-05-28T06:04:17.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.880+0000 7f4f5e400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f4f580ae2c0 con 0x7f4f58162d20 2024-05-28T06:04:17.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.881+0000 7f414be00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f414c15ea40 0x7f414c0d4f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.96:41610/0 (socket says 172.21.15.96:41610) 2024-05-28T06:04:17.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.881+0000 7f414be00640 1 -- 172.21.15.96:0/620639020 learned_addr learned my addr 172.21.15.96:0/620639020 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:04:17.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.881+0000 7f4f5d000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4f58072100 0x7f4f5815e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:17.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.881+0000 7f4f5d000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4f58072100 0x7f4f5815e480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.96:58488/0 (socket says 172.21.15.96:58488) 2024-05-28T06:04:17.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.881+0000 7f4f5d000640 1 -- 172.21.15.96:0/469072148 learned_addr learned my addr 172.21.15.96:0/469072148 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:17.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.882+0000 7f98e5800640 1 Processor -- start 2024-05-28T06:04:17.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.882+0000 7f29b0c00640 1 -- 172.21.15.96:0/853690032 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f29a40e59f0 con 0x7f29a40075f0 2024-05-28T06:04:17.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.881+0000 7f4f57e00640 1 --2- 172.21.15.96:0/469072148 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f58162940 0x7f4f58164c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:17.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.882+0000 7f414aa00640 1 -- 172.21.15.96:0/620639020 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3860244375 0 0) 0x7f414c0cf7e0 con 0x7f414c15ea40 2024-05-28T06:04:17.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.882+0000 7f412ea00640 1 -- 172.21.15.96:0/691666412 <== mon.2 v2:172.21.15.123:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4124044020 con 0x7f4138163170 2024-05-28T06:04:17.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.882+0000 7f414aa00640 1 -- 172.21.15.96:0/620639020 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4130003520 con 0x7f414c15ea40 2024-05-28T06:04:17.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.882+0000 7f220ce00640 1 Processor -- start 2024-05-28T06:04:17.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.882+0000 7f414aa00640 1 -- 172.21.15.96:0/620639020 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3797916627 0 0) 0x7f414c0d19a0 con 0x7f414c06c950 2024-05-28T06:04:17.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.882+0000 7f414aa00640 1 -- 172.21.15.96:0/620639020 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f414c0cf7e0 con 0x7f414c06c950 2024-05-28T06:04:17.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.882+0000 7f98e5800640 1 -- start start 2024-05-28T06:04:17.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.883+0000 7f4f57e00640 1 -- 172.21.15.96:0/469072148 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4f58162d20 msgr2=0x7f4f580cece0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-28T06:04:17.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.883+0000 7f4f57e00640 1 --2- 172.21.15.96:0/469072148 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4f58162d20 0x7f4f580cece0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:17.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.883+0000 7f4f57e00640 1 -- 172.21.15.96:0/469072148 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4f58072100 msgr2=0x7f4f5815e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:17.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.883+0000 7f4f57e00640 1 --2- 172.21.15.96:0/469072148 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4f58072100 0x7f4f5815e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:17.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.883+0000 7f4f57e00640 1 -- 172.21.15.96:0/469072148 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f58165560 con 0x7f4f58162940 2024-05-28T06:04:17.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.883+0000 7f414b400640 1 --2- 172.21.15.96:0/620639020 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f414c0d1040 0x7f414c0d1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:17.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.883+0000 7f414aa00640 1 -- 172.21.15.96:0/620639020 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 775461427 0 0) 0x7f414c0cf7e0 con 0x7f414c06c950 2024-05-28T06:04:17.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.883+0000 7f414aa00640 1 -- 172.21.15.96:0/620639020 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f414c0d1040 msgr2=0x7f414c0d1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:17.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.883+0000 7f414aa00640 1 --2- 172.21.15.96:0/620639020 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f414c0d1040 0x7f414c0d1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:17.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.883+0000 7f414aa00640 1 -- 172.21.15.96:0/620639020 >> v1:172.21.15.123:6789/0 conn(0x7f414c15ea40 legacy=0x7f414c0d4f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:17.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.883+0000 7f414aa00640 1 -- 172.21.15.96:0/620639020 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f414c15fe50 con 0x7f414c06c950 2024-05-28T06:04:17.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.883+0000 7f414aa00640 1 -- 172.21.15.96:0/620639020 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f41440019f0 con 0x7f414c06c950 2024-05-28T06:04:17.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.883+0000 7f414aa00640 1 -- 172.21.15.96:0/620639020 >> v1:172.21.15.96:6789/0 conn(0x7f414c06c950 legacy=0x7f414c06cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:17.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.883+0000 7f414aa00640 1 --2- 172.21.15.96:0/620639020 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4130004bd0 0x7f4130004fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:17.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.883+0000 7f414aa00640 1 --2- 172.21.15.96:0/620639020 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41300037a0 0x7f4130003c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:17.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.883+0000 7f414aa00640 1 --2- 172.21.15.96:0/620639020 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4130004280 0x7f413000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:17.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.883+0000 7faf5aa00640 1 Processor -- start 2024-05-28T06:04:17.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.883+0000 7f4f5d000640 1 --2- 172.21.15.96:0/469072148 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4f58072100 0x7f4f5815e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-28T06:04:17.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.883+0000 7f414aa00640 1 -- 172.21.15.96:0/620639020 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f414c15ee20 con 0x7f41300037a0 2024-05-28T06:04:17.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.883+0000 7f414aa00640 1 -- 172.21.15.96:0/620639020 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f413000c1e0 con 0x7f4130004280 2024-05-28T06:04:17.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.883+0000 7f414aa00640 1 -- 172.21.15.96:0/620639020 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f413000c3e0 con 0x7f4130004bd0 2024-05-28T06:04:17.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.883+0000 7f414b400640 1 --2- 172.21.15.96:0/620639020 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f414c0d1040 0x7f414c0d1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:04:17.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.883+0000 7f4f57e00640 1 --2- 172.21.15.96:0/469072148 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f58162940 0x7f4f58164c50 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f4f48031000 tx=0x7f4f480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:17.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.884+0000 7f414b400640 1 --2- 172.21.15.96:0/620639020 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41300037a0 0x7f4130003c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:17.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.884+0000 7faf5aa00640 1 -- start start 2024-05-28T06:04:17.906 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.884+0000 7f4f54c00640 1 -- 172.21.15.96:0/469072148 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f48042020 con 0x7f4f58162940 2024-05-28T06:04:17.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.884+0000 7f4f54c00640 1 -- 172.21.15.96:0/469072148 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f4803d6b0 con 0x7f4f58162940 2024-05-28T06:04:17.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.884+0000 7f4f54c00640 1 -- 172.21.15.96:0/469072148 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f48037040 con 0x7f4f58162940 2024-05-28T06:04:17.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.884+0000 7f414b400640 1 -- 172.21.15.96:0/620639020 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4130004bd0 msgr2=0x7f4130004fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:17.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.884+0000 7f414be00640 1 --2- 172.21.15.96:0/620639020 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4130004bd0 0x7f4130004fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:17.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.884+0000 7f4f5e400640 1 -- 172.21.15.96:0/469072148 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f58165840 con 0x7f4f58162940 2024-05-28T06:04:17.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.884+0000 7f4f5e400640 1 -- 172.21.15.96:0/469072148 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f581660e0 con 0x7f4f58162940 2024-05-28T06:04:17.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.884+0000 7f414b400640 1 --2- 172.21.15.96:0/620639020 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4130004bd0 0x7f4130004fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:17.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.884+0000 7f414b400640 1 -- 172.21.15.96:0/620639020 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4130004280 msgr2=0x7f413000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:17.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.884+0000 7f414b400640 1 --2- 172.21.15.96:0/620639020 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4130004280 0x7f413000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:17.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.884+0000 7f414b400640 1 -- 172.21.15.96:0/620639020 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f414c15fe50 con 0x7f41300037a0 2024-05-28T06:04:17.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.884+0000 7f414be00640 1 --2- 172.21.15.96:0/620639020 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4130004bd0 0x7f4130004fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:04:17.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.886+0000 7f4f5e400640 1 -- 172.21.15.96:0/469072148 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4f58165c40 con 0x7f4f58162940 2024-05-28T06:04:17.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.886+0000 7f98e5800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f98e00723d0 0x7f98e00b3380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:17.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.886+0000 7f98e5800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f98e015b850 0x7f98e01653e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:17.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.886+0000 7faf5aa00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faf540b65a0 0x7faf540d4e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:17.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.886+0000 7faf5aa00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7faf54167470 0x7faf54166860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:17.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.886+0000 7faf5aa00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faf540d3170 con 0x7faf54166da0 2024-05-28T06:04:17.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.886+0000 7faf5aa00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7faf540d3450 con 0x7faf54167470 2024-05-28T06:04:17.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.886+0000 7faf5aa00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7faf540d3730 con 0x7faf540b65a0 2024-05-28T06:04:17.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.886+0000 7f299f400640 1 -- 172.21.15.96:0/853690032 <== mon.2 v2:172.21.15.123:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f299801a070 con 0x7f29a40075f0 2024-05-28T06:04:17.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.886+0000 7f412ea00640 1 --2- 172.21.15.96:0/691666412 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f411806caf0 0x7f411806efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:17.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.887+0000 7f412ea00640 1 -- 172.21.15.96:0/691666412 <== mon.2 v2:172.21.15.123:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f41240abf90 con 0x7f4138163170 2024-05-28T06:04:17.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.887+0000 7f412ea00640 1 --2- 172.21.15.96:0/691666412 >> [v2:172.21.15.96:6820/3070158265,v1:172.21.15.96:6823/3070158265] conn(0x7f4118074d10 0x7f4118077130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:17.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.887+0000 7f412ea00640 1 -- 172.21.15.96:0/691666412 --> [v2:172.21.15.96:6820/3070158265,v1:172.21.15.96:6823/3070158265] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4124001d30 con 0x7f4118074d10 2024-05-28T06:04:17.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.887+0000 7f412ea00640 1 -- 172.21.15.96:0/691666412 <== mon.2 v2:172.21.15.123:3301/0 6 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f41240ac380 con 0x7f4138163170 2024-05-28T06:04:17.914 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.887+0000 7f414b400640 1 --2- 172.21.15.96:0/620639020 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41300037a0 0x7f4130003c20 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f413c00a860 tx=0x7f4130005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9979917e929a3309 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:17.914 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.886+0000 7f98e5800640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f98e00b3a10 con 0x7f98e00d02d0 2024-05-28T06:04:17.914 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.886+0000 7f98e5800640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f98e006e190 con 0x7f98e015b850 2024-05-28T06:04:17.914 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.886+0000 7f98e5800640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f98e00b3cf0 con 0x7f98e00723d0 2024-05-28T06:04:17.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.886+0000 7f98df400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f98e015b850 0x7f98e01653e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:17.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.887+0000 7f98df400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f98e015b850 0x7f98e01653e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:41638/0 (socket says 172.21.15.96:41638) 2024-05-28T06:04:17.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.887+0000 7f98df400640 1 -- 172.21.15.96:0/217516172 learned_addr learned my addr 172.21.15.96:0/217516172 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:17.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.887+0000 7f98de000640 1 --2- 172.21.15.96:0/217516172 >> v2:172.21.15.123:3300/0 conn(0x7f98e00723d0 0x7f98e00b3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:17.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.887+0000 7f98df400640 1 -- 172.21.15.96:0/217516172 >> v2:172.21.15.123:3300/0 conn(0x7f98e00723d0 msgr2=0x7f98e00b3380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:17.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.887+0000 7f98df400640 1 --2- 172.21.15.96:0/217516172 >> v2:172.21.15.123:3300/0 conn(0x7f98e00723d0 0x7f98e00b3380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:17.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.887+0000 7f98df400640 1 -- 172.21.15.96:0/217516172 >> v1:172.21.15.123:6789/0 conn(0x7f98e00d02d0 legacy=0x7f98e0071cc0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-28T06:04:17.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.887+0000 7f98df400640 1 -- 172.21.15.96:0/217516172 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98e00b3fd0 con 0x7f98e015b850 2024-05-28T06:04:17.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.888+0000 7f4136000640 1 --2- 172.21.15.96:0/691666412 >> [v2:172.21.15.96:6820/3070158265,v1:172.21.15.96:6823/3070158265] conn(0x7f4118074d10 0x7f4118077130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:17.917 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.888+0000 7f414aa00640 1 -- 172.21.15.96:0/620639020 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f413c037070 con 0x7f41300037a0 2024-05-28T06:04:17.917 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.888+0000 7f4136a00640 1 --2- 172.21.15.96:0/691666412 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f411806caf0 0x7f411806efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:17.917 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.888+0000 7f414aa00640 1 -- 172.21.15.96:0/620639020 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f413c02aa90 con 0x7f41300037a0 2024-05-28T06:04:17.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.888+0000 7f414aa00640 1 -- 172.21.15.96:0/620639020 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f413c00a4b0 con 0x7f41300037a0 2024-05-28T06:04:17.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.887+0000 7faf53400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7faf54167470 0x7faf54166860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:17.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.888+0000 7f299f400640 1 --2- 172.21.15.96:0/853690032 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f298c06c9c0 0x7f298c06ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:17.919 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.887+0000 7faf53400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7faf54167470 0x7faf54166860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.96:58510/0 (socket says 172.21.15.96:58510) 2024-05-28T06:04:17.919 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.887+0000 7faf53400640 1 -- 172.21.15.96:0/2317309657 learned_addr learned my addr 172.21.15.96:0/2317309657 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:17.919 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.888+0000 7faf53400640 1 -- 172.21.15.96:0/2317309657 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faf540b65a0 msgr2=0x7faf540d4e00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-28T06:04:17.919 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.888+0000 7faf53400640 1 --2- 172.21.15.96:0/2317309657 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faf540b65a0 0x7faf540d4e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:17.919 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.888+0000 7faf53400640 1 -- 172.21.15.96:0/2317309657 >> v1:172.21.15.96:6789/0 conn(0x7faf54166da0 legacy=0x7faf540cedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:17.919 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.888+0000 7faf53400640 1 -- 172.21.15.96:0/2317309657 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf540d3a10 con 0x7faf54167470 2024-05-28T06:04:17.920 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.886+0000 7f220ce00640 1 -- start start 2024-05-28T06:04:17.920 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.887+0000 7f220ce00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f22080722d0 0x7f22081f7d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:17.920 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.888+0000 7f299f400640 1 -- 172.21.15.96:0/853690032 <== mon.2 v2:172.21.15.123:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f299808b090 con 0x7f29a40075f0 2024-05-28T06:04:17.921 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.887+0000 7f220ce00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2208162940 0x7f220809dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:17.921 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.888+0000 7f220ce00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2208162d20 0x7f2208165110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:17.921 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.888+0000 7f220ce00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f22080ae2c0 con 0x7f2208162d20 2024-05-28T06:04:17.921 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.889+0000 7f4152c00640 1 -- 172.21.15.96:0/620639020 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41300037a0 msgr2=0x7f4130003c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:17.921 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.889+0000 7f4152c00640 1 --2- 172.21.15.96:0/620639020 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41300037a0 0x7f4130003c20 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f413c00a860 tx=0x7f4130005830 comp rx=0 tx=0).stop 2024-05-28T06:04:17.922 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.888+0000 7f220ce00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f220806f250 con 0x7f2208162940 2024-05-28T06:04:17.922 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.889+0000 7f4150c00640 1 -- 172.21.15.96:0/620639020 reap_dead start 2024-05-28T06:04:17.922 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.888+0000 7f220ce00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f22080ae440 con 0x7f22080722d0 2024-05-28T06:04:17.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.889+0000 7f299f400640 1 --2- 172.21.15.96:0/853690032 >> [v2:172.21.15.96:6800/4059252009,v1:172.21.15.96:6801/4059252009] conn(0x7f298c074be0 0x7f298c077000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:17.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.889+0000 7f299f400640 1 -- 172.21.15.96:0/853690032 --> [v2:172.21.15.96:6800/4059252009,v1:172.21.15.96:6801/4059252009] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f299801ae50 con 0x7f298c074be0 2024-05-28T06:04:17.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.889+0000 7f299f400640 1 -- 172.21.15.96:0/853690032 <== mon.2 v2:172.21.15.123:3301/0 6 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f299808b480 con 0x7f29a40075f0 2024-05-28T06:04:17.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.888+0000 7f2207e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2208162d20 0x7f2208165110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:17.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.889+0000 7f29ab400640 1 --2- 172.21.15.96:0/853690032 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f298c06c9c0 0x7f298c06ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:17.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.888+0000 7f2207e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2208162d20 0x7f2208165110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:41648/0 (socket says 172.21.15.96:41648) 2024-05-28T06:04:17.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.889+0000 7f4152c00640 1 -- 172.21.15.96:0/620639020 shutdown_connections 2024-05-28T06:04:17.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.889+0000 7f4152c00640 1 -- 172.21.15.96:0/620639020 >> 172.21.15.96:0/620639020 conn(0x7f414c08bfc0 msgr2=0x7f414c0d4630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:17.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.890+0000 7faf53400640 1 --2- 172.21.15.96:0/2317309657 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faf54167470 0x7faf54166860 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7faf40009870 tx=0x7faf4002a230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=58bd1692e937211d server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:17.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.889+0000 7f29abe00640 1 --2- 172.21.15.96:0/853690032 >> [v2:172.21.15.96:6800/4059252009,v1:172.21.15.96:6801/4059252009] conn(0x7f298c074be0 0x7f298c077000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:17.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.888+0000 7f2207e00640 1 -- 172.21.15.96:0/905762090 learned_addr learned my addr 172.21.15.96:0/905762090 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:17.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.889+0000 7f4136000640 1 --2- 172.21.15.96:0/691666412 >> [v2:172.21.15.96:6820/3070158265,v1:172.21.15.96:6823/3070158265] conn(0x7f4118074d10 0x7f4118077130 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:17.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.890+0000 7f4152c00640 1 -- 172.21.15.96:0/620639020 shutdown_connections 2024-05-28T06:04:17.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.888+0000 7f2207e00640 1 -- 172.21.15.96:0/905762090 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f22080722d0 msgr2=0x7f22081f7d60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-28T06:04:17.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.889+0000 7f2207e00640 1 --2- 172.21.15.96:0/905762090 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f22080722d0 0x7f22081f7d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:17.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.889+0000 7f2207e00640 1 -- 172.21.15.96:0/905762090 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2208162940 msgr2=0x7f220809dea0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-28T06:04:17.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.889+0000 7f2207e00640 1 --2- 172.21.15.96:0/905762090 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2208162940 0x7f220809dea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:17.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.889+0000 7f2207e00640 1 -- 172.21.15.96:0/905762090 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22080cdc70 con 0x7f2208162d20 2024-05-28T06:04:17.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.890+0000 7f2207e00640 1 --2- 172.21.15.96:0/905762090 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2208162d20 0x7f2208165110 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f220000cad0 tx=0x7f220000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:17.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.890+0000 7f98df400640 1 --2- 172.21.15.96:0/217516172 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f98e015b850 0x7f98e01653e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f98e00d0120 tx=0x7f98d802a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5da48097bf5c5a97 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:17.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.890+0000 7f29abe00640 1 --2- 172.21.15.96:0/853690032 >> [v2:172.21.15.96:6800/4059252009,v1:172.21.15.96:6801/4059252009] conn(0x7f298c074be0 0x7f298c077000 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:17.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.891+0000 7faf52a00640 1 -- 172.21.15.96:0/2317309657 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf40037070 con 0x7faf54167470 2024-05-28T06:04:17.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.891+0000 7faf52a00640 1 -- 172.21.15.96:0/2317309657 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf40032440 con 0x7faf54167470 2024-05-28T06:04:17.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.891+0000 7faf52a00640 1 -- 172.21.15.96:0/2317309657 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf40032810 con 0x7faf54167470 2024-05-28T06:04:17.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.891+0000 7f29ab400640 1 --2- 172.21.15.96:0/853690032 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f298c06c9c0 0x7f298c06ee80 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f29a400dc00 tx=0x7f29a0009240 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:17.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.890+0000 7f4152c00640 1 -- 172.21.15.96:0/620639020 wait complete. 2024-05-28T06:04:17.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.891+0000 7f98dd600640 1 -- 172.21.15.96:0/217516172 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98d8037070 con 0x7f98e015b850 2024-05-28T06:04:17.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.891+0000 7f98dd600640 1 -- 172.21.15.96:0/217516172 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98d8032440 con 0x7f98e015b850 2024-05-28T06:04:17.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.891+0000 7f98dd600640 1 -- 172.21.15.96:0/217516172 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98d8032810 con 0x7f98e015b850 2024-05-28T06:04:17.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.892+0000 7faf5aa00640 1 -- 172.21.15.96:0/2317309657 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faf54167470 msgr2=0x7faf54166860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:17.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.892+0000 7faf5aa00640 1 --2- 172.21.15.96:0/2317309657 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faf54167470 0x7faf54166860 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7faf40009870 tx=0x7faf4002a230 comp rx=0 tx=0).stop 2024-05-28T06:04:17.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.892+0000 7faf5aa00640 1 -- 172.21.15.96:0/2317309657 shutdown_connections 2024-05-28T06:04:17.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.892+0000 7faf5aa00640 1 --2- 172.21.15.96:0/2317309657 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faf54167470 0x7faf54166860 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:17.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.892+0000 7faf5aa00640 1 --2- 172.21.15.96:0/2317309657 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faf540b65a0 0x7faf540d4e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:17.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.892+0000 7faf5aa00640 1 -- 172.21.15.96:0/2317309657 >> 172.21.15.96:0/2317309657 conn(0x7faf5408bfc0 msgr2=0x7faf540d44e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:17.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.892+0000 7f412ea00640 1 -- 172.21.15.96:0/691666412 <== osd.2 v2:172.21.15.96:6820/3070158265 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4124001d30 con 0x7f4118074d10 2024-05-28T06:04:17.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.892+0000 7f4136a00640 1 --2- 172.21.15.96:0/691666412 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f411806caf0 0x7f411806efb0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4128000f60 tx=0x7f4128007c80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:17.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.892+0000 7f98e5800640 1 -- 172.21.15.96:0/217516172 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f98e015b850 msgr2=0x7f98e01653e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:17.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.892+0000 7f98e5800640 1 --2- 172.21.15.96:0/217516172 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f98e015b850 0x7f98e01653e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f98e00d0120 tx=0x7f98d802a230 comp rx=0 tx=0).stop 2024-05-28T06:04:17.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.892+0000 7f21ff400640 1 -- 172.21.15.96:0/905762090 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2200017070 con 0x7f2208162d20 2024-05-28T06:04:17.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.892+0000 7f21ff400640 1 -- 172.21.15.96:0/905762090 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f220000abd0 con 0x7f2208162d20 2024-05-28T06:04:17.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.892+0000 7f21ff400640 1 -- 172.21.15.96:0/905762090 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2200012400 con 0x7f2208162d20 2024-05-28T06:04:17.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.892+0000 7f4f54c00640 1 -- 172.21.15.96:0/469072148 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4f48045020 con 0x7f4f58162940 2024-05-28T06:04:17.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.892+0000 7faf5aa00640 1 -- 172.21.15.96:0/2317309657 shutdown_connections 2024-05-28T06:04:17.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.892+0000 7faf5aa00640 1 -- 172.21.15.96:0/2317309657 wait complete. 2024-05-28T06:04:17.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.892+0000 7f98e5800640 1 -- 172.21.15.96:0/217516172 shutdown_connections 2024-05-28T06:04:17.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.892+0000 7f98e5800640 1 --2- 172.21.15.96:0/217516172 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f98e015b850 0x7f98e01653e0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:17.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.892+0000 7f98e5800640 1 --2- 172.21.15.96:0/217516172 >> v2:172.21.15.123:3300/0 conn(0x7f98e00723d0 0x7f98e00b3380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:17.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.892+0000 7f98e5800640 1 -- 172.21.15.96:0/217516172 >> 172.21.15.96:0/217516172 conn(0x7f98e006bae0 msgr2=0x7f98e006bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:17.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.893+0000 7f299f400640 1 -- 172.21.15.96:0/853690032 <== osd.1 v2:172.21.15.96:6800/4059252009 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f299801ae50 con 0x7f298c074be0 2024-05-28T06:04:17.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.893+0000 7f98e5800640 1 -- 172.21.15.96:0/217516172 shutdown_connections 2024-05-28T06:04:17.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.893+0000 7f98e5800640 1 -- 172.21.15.96:0/217516172 wait complete. 2024-05-28T06:04:17.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.893+0000 7f4152c00640 1 Processor -- start 2024-05-28T06:04:17.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.893+0000 7f4152c00640 1 -- start start 2024-05-28T06:04:17.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.894+0000 7f4f54c00640 1 --2- 172.21.15.96:0/469072148 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4f3806c9c0 0x7f4f3806ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:17.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.894+0000 7f4f54c00640 1 -- 172.21.15.96:0/469072148 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f4f480ac8a0 con 0x7f4f58162940 2024-05-28T06:04:17.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.893+0000 7f220ce00640 1 -- 172.21.15.96:0/905762090 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f220809d0e0 con 0x7f2208162d20 2024-05-28T06:04:17.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.894+0000 7f4f54c00640 1 --2- 172.21.15.96:0/469072148 >> [v2:172.21.15.123:6800/3445772216,v1:172.21.15.123:6801/3445772216] conn(0x7f4f38074be0 0x7f4f38077000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:17.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.894+0000 7f4f54c00640 1 -- 172.21.15.96:0/469072148 --> [v2:172.21.15.123:6800/3445772216,v1:172.21.15.123:6801/3445772216] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4f48001d30 con 0x7f4f38074be0 2024-05-28T06:04:17.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.894+0000 7f4f54c00640 1 -- 172.21.15.96:0/469072148 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f4f480acc90 con 0x7f4f58162940 2024-05-28T06:04:17.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.895+0000 7faf5aa00640 1 Processor -- start 2024-05-28T06:04:17.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.895+0000 7f4f5da00640 1 --2- 172.21.15.96:0/469072148 >> [v2:172.21.15.123:6800/3445772216,v1:172.21.15.123:6801/3445772216] conn(0x7f4f38074be0 0x7f4f38077000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:17.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.895+0000 7f4152c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41300037a0 0x7f414c15d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:17.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.895+0000 7f220ce00640 1 -- 172.21.15.96:0/905762090 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f220809d6a0 con 0x7f2208162d20 2024-05-28T06:04:17.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.895+0000 7f98e5800640 1 Processor -- start 2024-05-28T06:04:17.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.897+0000 7f220ce00640 1 -- 172.21.15.96:0/905762090 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f220809da80 con 0x7f2208162d20 2024-05-28T06:04:17.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.896+0000 7f4f5d000640 1 --2- 172.21.15.96:0/469072148 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4f3806c9c0 0x7f4f3806ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:17.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.896+0000 7f4152c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f414c0c9ab0 0x7f414c0b5950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:17.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.896+0000 7f4152c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f414c15ccc0 0x7f414c0b8d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:17.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.896+0000 7f4152c00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f414c0d0040 con 0x7f414c0c9ab0 2024-05-28T06:04:17.943 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.896+0000 7f4152c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f414c0cfd60 con 0x7f41300037a0 2024-05-28T06:04:17.943 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.896+0000 7f4152c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f414c0d0320 con 0x7f414c15ccc0 2024-05-28T06:04:17.943 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.896+0000 7f414be00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41300037a0 0x7f414c15d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:17.943 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.896+0000 7f98e5800640 1 -- start start 2024-05-28T06:04:17.943 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.895+0000 7faf5aa00640 1 -- start start 2024-05-28T06:04:17.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.897+0000 7f4f5da00640 1 --2- 172.21.15.96:0/469072148 >> [v2:172.21.15.123:6800/3445772216,v1:172.21.15.123:6801/3445772216] conn(0x7f4f38074be0 0x7f4f38077000 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:17.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.897+0000 7f4f5d000640 1 --2- 172.21.15.96:0/469072148 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4f3806c9c0 0x7f4f3806ee80 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f4f44004e60 tx=0x7f4f44007f10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:17.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.896+0000 7f414be00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41300037a0 0x7f414c15d500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.96:58530/0 (socket says 172.21.15.96:58530) 2024-05-28T06:04:17.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.896+0000 7f414be00640 1 -- 172.21.15.96:0/204945456 learned_addr learned my addr 172.21.15.96:0/204945456 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:17.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.897+0000 7faf5aa00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faf540b65a0 0x7faf540ce080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:17.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.897+0000 7f414be00640 1 -- 172.21.15.96:0/204945456 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f414c15ccc0 msgr2=0x7f414c0b8d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:17.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.897+0000 7f98e5800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f98e00723d0 0x7f98e0152270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:17.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.897+0000 7faf5aa00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faf54166da0 0x7faf540d0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:17.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.898+0000 7f21ff400640 1 -- 172.21.15.96:0/905762090 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f220001a070 con 0x7f2208162d20 2024-05-28T06:04:17.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.898+0000 7faf5aa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf54167470 0x7faf540d1430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:17.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.898+0000 7f98e5800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f98e00d02d0 0x7f98e01f59b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:17.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.898+0000 7faf58a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf54167470 0x7faf540d1430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:17.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.898+0000 7faf53400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faf54166da0 0x7faf540d0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:17.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.898+0000 7faf58a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf54167470 0x7faf540d1430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:41676/0 (socket says 172.21.15.96:41676) 2024-05-28T06:04:17.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.898+0000 7faf58a00640 1 -- 172.21.15.96:0/1533109368 learned_addr learned my addr 172.21.15.96:0/1533109368 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:17.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.898+0000 7faf5aa00640 1 -- 172.21.15.96:0/1533109368 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7faf540b6b00 con 0x7faf54167470 2024-05-28T06:04:17.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.898+0000 7faf53e00640 1 --2- 172.21.15.96:0/1533109368 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faf540b65a0 0x7faf540ce080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:17.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.898+0000 7f98e5800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f98e015b850 0x7f98e01f5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:17.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.898+0000 7f98dea00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f98e00723d0 0x7f98e0152270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:17.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.898+0000 7f98dea00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f98e00723d0 0x7f98e0152270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.96:58552/0 (socket says 172.21.15.96:58552) 2024-05-28T06:04:17.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.898+0000 7f98e5800640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f98e01550d0 con 0x7f98e015b850 2024-05-28T06:04:17.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.898+0000 7f98e5800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f98e015bca0 con 0x7f98e00723d0 2024-05-28T06:04:17.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.898+0000 7f98df400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f98e015b850 0x7f98e01f5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:17.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.899+0000 7f98df400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f98e015b850 0x7f98e01f5ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:41688/0 (socket says 172.21.15.96:41688) 2024-05-28T06:04:17.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.899+0000 7f98df400640 1 -- 172.21.15.96:0/3683964447 learned_addr learned my addr 172.21.15.96:0/3683964447 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:17.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.899+0000 7faf5aa00640 1 -- 172.21.15.96:0/1533109368 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7faf540b6980 con 0x7faf540b65a0 2024-05-28T06:04:17.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.899+0000 7faf5aa00640 1 -- 172.21.15.96:0/1533109368 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7faf540b8790 con 0x7faf54166da0 2024-05-28T06:04:17.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.897+0000 7f414be00640 1 --2- 172.21.15.96:0/204945456 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f414c15ccc0 0x7f414c0b8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:17.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.898+0000 7f414be00640 1 -- 172.21.15.96:0/204945456 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f414c0c9ab0 msgr2=0x7f414c0b5950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-28T06:04:17.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.898+0000 7f414b400640 1 --2- 172.21.15.96:0/204945456 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f414c0c9ab0 0x7f414c0b5950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:17.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.899+0000 7f414be00640 1 --2- 172.21.15.96:0/204945456 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f414c0c9ab0 0x7f414c0b5950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:17.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.899+0000 7f414be00640 1 -- 172.21.15.96:0/204945456 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f414c0c8cf0 con 0x7f41300037a0 2024-05-28T06:04:17.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.899+0000 7faf58a00640 1 -- 172.21.15.96:0/1533109368 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faf54166da0 msgr2=0x7faf540d0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:17.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.899+0000 7faf58a00640 1 --2- 172.21.15.96:0/1533109368 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faf54166da0 0x7faf540d0ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:17.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.900+0000 7faf58a00640 1 -- 172.21.15.96:0/1533109368 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faf540b65a0 msgr2=0x7faf540ce080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:17.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.900+0000 7f21ff400640 1 --2- 172.21.15.96:0/905762090 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f21e806c940 0x7f21e806ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:17.953 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.900+0000 7faf58a00640 1 --2- 172.21.15.96:0/1533109368 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faf540b65a0 0x7faf540ce080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:17.953 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.900+0000 7faf58a00640 1 -- 172.21.15.96:0/1533109368 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf54165e40 con 0x7faf54167470 2024-05-28T06:04:17.953 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.900+0000 7f21ff400640 1 -- 172.21.15.96:0/905762090 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f2200090060 con 0x7f2208162d20 2024-05-28T06:04:17.954 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.900+0000 7f4f54c00640 1 -- 172.21.15.96:0/469072148 <== osd.4 v2:172.21.15.123:6800/3445772216 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4f48001d30 con 0x7f4f38074be0 2024-05-28T06:04:17.954 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.898+0000 7f98e5800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f98e0155250 con 0x7f98e00d02d0 2024-05-28T06:04:17.954 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.899+0000 7f98df400640 1 -- 172.21.15.96:0/3683964447 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f98e00d02d0 msgr2=0x7f98e01f59b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-28T06:04:17.954 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.900+0000 7f98de000640 1 --2- 172.21.15.96:0/3683964447 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f98e00d02d0 0x7f98e01f59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:17.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.900+0000 7faf53e00640 1 --2- 172.21.15.96:0/1533109368 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faf540b65a0 0x7faf540ce080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-28T06:04:17.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.900+0000 7f98df400640 1 --2- 172.21.15.96:0/3683964447 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f98e00d02d0 0x7f98e01f59b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:17.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.900+0000 7f98df400640 1 -- 172.21.15.96:0/3683964447 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f98e00723d0 msgr2=0x7f98e0152270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:17.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.900+0000 7f98df400640 1 --2- 172.21.15.96:0/3683964447 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f98e00723d0 0x7f98e0152270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:17.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.900+0000 7f98df400640 1 -- 172.21.15.96:0/3683964447 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98e00ce570 con 0x7f98e015b850 2024-05-28T06:04:17.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.900+0000 7f98dea00640 1 --2- 172.21.15.96:0/3683964447 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f98e00723d0 0x7f98e0152270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-28T06:04:17.956 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.900+0000 7f21ff400640 1 --2- 172.21.15.96:0/905762090 >> [v2:172.21.15.96:6808/2384313931,v1:172.21.15.96:6810/2384313931] conn(0x7f21e8074b60 0x7f21e8076f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:17.956 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.900+0000 7faf58a00640 1 --2- 172.21.15.96:0/1533109368 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf54167470 0x7faf540d1430 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7faf4c00c7b0 tx=0x7faf4c00cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:17.956 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.900+0000 7f21ff400640 1 -- 172.21.15.96:0/905762090 --> [v2:172.21.15.96:6808/2384313931,v1:172.21.15.96:6810/2384313931] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f21e8077630 con 0x7f21e8074b60 2024-05-28T06:04:17.956 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.901+0000 7f21ff400640 1 -- 172.21.15.96:0/905762090 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f220008b610 con 0x7f2208162d20 2024-05-28T06:04:17.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.901+0000 7f414be00640 1 --2- 172.21.15.96:0/204945456 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41300037a0 0x7f414c15d500 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f41340010e0 tx=0x7f4134015420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:17.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.900+0000 7f98df400640 1 --2- 172.21.15.96:0/3683964447 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f98e015b850 0x7f98e01f5ef0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f98d802a820 tx=0x7f98d80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:17.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.901+0000 7f2206a00640 1 --2- 172.21.15.96:0/905762090 >> [v2:172.21.15.96:6808/2384313931,v1:172.21.15.96:6810/2384313931] conn(0x7f21e8074b60 0x7f21e8076f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:17.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.901+0000 7faf4be00640 1 -- 172.21.15.96:0/1533109368 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf4c017070 con 0x7faf54167470 2024-05-28T06:04:17.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.901+0000 7faf4be00640 1 -- 172.21.15.96:0/1533109368 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf4c003da0 con 0x7faf54167470 2024-05-28T06:04:17.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.901+0000 7f98d6a00640 1 -- 172.21.15.96:0/3683964447 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98d8041020 con 0x7f98e015b850 2024-05-28T06:04:17.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.901+0000 7faf4be00640 1 -- 172.21.15.96:0/1533109368 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf4c0052b0 con 0x7faf54167470 2024-05-28T06:04:17.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.901+0000 7faf5aa00640 1 -- 172.21.15.96:0/1533109368 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf54166120 con 0x7faf54167470 2024-05-28T06:04:17.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.901+0000 7faf5aa00640 1 -- 172.21.15.96:0/1533109368 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf540c80c0 con 0x7faf54167470 2024-05-28T06:04:17.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.901+0000 7f2207400640 1 --2- 172.21.15.96:0/905762090 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f21e806c940 0x7f21e806ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:17.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.901+0000 7f4143e00640 1 -- 172.21.15.96:0/204945456 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4134015d50 con 0x7f41300037a0 2024-05-28T06:04:17.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.902+0000 7f4143e00640 1 -- 172.21.15.96:0/204945456 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41340058d0 con 0x7f41300037a0 2024-05-28T06:04:17.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.902+0000 7f4143e00640 1 -- 172.21.15.96:0/204945456 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4134005ca0 con 0x7f41300037a0 2024-05-28T06:04:17.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.901+0000 7f98d6a00640 1 -- 172.21.15.96:0/3683964447 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98d8030640 con 0x7f98e015b850 2024-05-28T06:04:17.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.901+0000 7f98d6a00640 1 -- 172.21.15.96:0/3683964447 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98d8037040 con 0x7f98e015b850 2024-05-28T06:04:17.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.902+0000 7f98e5800640 1 -- 172.21.15.96:0/3683964447 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98e00ce850 con 0x7f98e015b850 2024-05-28T06:04:17.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.902+0000 7f98e5800640 1 -- 172.21.15.96:0/3683964447 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98e01527b0 con 0x7f98e015b850 2024-05-28T06:04:17.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.903+0000 7faf5aa00640 1 -- 172.21.15.96:0/1533109368 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7faf540690f0 con 0x7faf54167470 2024-05-28T06:04:17.962 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.902+0000 7f4152c00640 1 -- 172.21.15.96:0/204945456 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f414c0c8fd0 con 0x7f41300037a0 2024-05-28T06:04:17.962 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.902+0000 7f4152c00640 1 -- 172.21.15.96:0/204945456 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f414c0b5e90 con 0x7f41300037a0 2024-05-28T06:04:17.962 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.906+0000 7f4152c00640 1 -- 172.21.15.96:0/204945456 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4124001030 con 0x7f41300037a0 2024-05-28T06:04:17.962 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.906+0000 7f98e5800640 1 -- 172.21.15.96:0/3683964447 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f98e00689e0 con 0x7f98e015b850 2024-05-28T06:04:17.962 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.906+0000 7faf4be00640 1 -- 172.21.15.96:0/1533109368 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7faf4c01a070 con 0x7faf54167470 2024-05-28T06:04:17.963 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.903+0000 7f2207400640 1 --2- 172.21.15.96:0/905762090 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f21e806c940 0x7f21e806ee00 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f21f8034000 tx=0x7f21f802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:17.963 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.904+0000 7f2206a00640 1 --2- 172.21.15.96:0/905762090 >> [v2:172.21.15.96:6808/2384313931,v1:172.21.15.96:6810/2384313931] conn(0x7f21e8074b60 0x7f21e8076f80 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:17.963 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.905+0000 7f21ff400640 1 -- 172.21.15.96:0/905762090 <== osd.3 v2:172.21.15.96:6808/2384313931 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f21e8077630 con 0x7f21e8074b60 2024-05-28T06:04:17.963 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.907+0000 7f98d6a00640 1 -- 172.21.15.96:0/3683964447 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f98d8044020 con 0x7f98e015b850 2024-05-28T06:04:17.963 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.908+0000 7faf4be00640 1 --2- 172.21.15.96:0/1533109368 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7faf3406c940 0x7faf3406ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:17.964 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.909+0000 7f98d6a00640 1 --2- 172.21.15.96:0/3683964447 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f98c006c940 0x7f98c006ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:17.964 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.909+0000 7faf4be00640 1 -- 172.21.15.96:0/1533109368 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7faf4c08b0c0 con 0x7faf54167470 2024-05-28T06:04:17.964 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.909+0000 7f98d6a00640 1 -- 172.21.15.96:0/3683964447 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f98d80ac640 con 0x7f98e015b850 2024-05-28T06:04:17.964 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.909+0000 7f98d6a00640 1 --2- 172.21.15.96:0/3683964447 >> [v2:172.21.15.123:6808/1217470164,v1:172.21.15.123:6809/1217470164] conn(0x7f98c0074b60 0x7f98c0076f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:17.964 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.909+0000 7f98d6a00640 1 -- 172.21.15.96:0/3683964447 --> [v2:172.21.15.123:6808/1217470164,v1:172.21.15.123:6809/1217470164] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f98d8036e90 con 0x7f98c0074b60 2024-05-28T06:04:17.964 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.909+0000 7f98d6a00640 1 -- 172.21.15.96:0/3683964447 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f98d80ac9e0 con 0x7f98e015b850 2024-05-28T06:04:17.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.909+0000 7faf53e00640 1 --2- 172.21.15.96:0/1533109368 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7faf3406c940 0x7faf3406ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:17.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.909+0000 7faf4be00640 1 --2- 172.21.15.96:0/1533109368 >> [v2:172.21.15.123:6816/2085300276,v1:172.21.15.123:6818/2085300276] conn(0x7faf34074b60 0x7faf34076f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:17.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.910+0000 7faf4be00640 1 -- 172.21.15.96:0/1533109368 --> [v2:172.21.15.123:6816/2085300276,v1:172.21.15.123:6818/2085300276] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7faf34077630 con 0x7faf34074b60 2024-05-28T06:04:17.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.910+0000 7faf4be00640 1 -- 172.21.15.96:0/1533109368 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7faf4c092050 con 0x7faf54167470 2024-05-28T06:04:17.966 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.908+0000 7f4143e00640 1 -- 172.21.15.96:0/204945456 <== mon.1 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f41340118c0 con 0x7f41300037a0 2024-05-28T06:04:17.966 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.909+0000 7f4143e00640 1 --2- 172.21.15.96:0/204945456 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f412806c9f0 0x7f412806eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:17.966 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.910+0000 7f4143e00640 1 -- 172.21.15.96:0/204945456 <== mon.1 v2:172.21.15.123:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f4134093af0 con 0x7f41300037a0 2024-05-28T06:04:17.966 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.910+0000 7f4143e00640 1 --2- 172.21.15.96:0/204945456 >> [v2:172.21.15.123:6817/2041107525,v1:172.21.15.123:6819/2041107525] conn(0x7f4128074b90 0x7f4128076fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:17.966 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.910+0000 7f4143e00640 1 -- 172.21.15.96:0/204945456 --> [v2:172.21.15.123:6817/2041107525,v1:172.21.15.123:6819/2041107525] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4128077660 con 0x7f4128074b90 2024-05-28T06:04:17.966 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.910+0000 7f4143e00640 1 -- 172.21.15.96:0/204945456 <== mon.1 v2:172.21.15.123:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f413405daa0 con 0x7f41300037a0 2024-05-28T06:04:17.967 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.910+0000 7f98dea00640 1 --2- 172.21.15.96:0/3683964447 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f98c006c940 0x7f98c006ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:17.967 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.911+0000 7f98de000640 1 --2- 172.21.15.96:0/3683964447 >> [v2:172.21.15.123:6808/1217470164,v1:172.21.15.123:6809/1217470164] conn(0x7f98c0074b60 0x7f98c0076f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:17.967 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.911+0000 7faf53e00640 1 --2- 172.21.15.96:0/1533109368 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7faf3406c940 0x7faf3406ee00 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7faf44006fd0 tx=0x7faf44008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:17.967 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.911+0000 7f414b400640 1 --2- 172.21.15.96:0/204945456 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f412806c9f0 0x7f412806eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:17.967 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.911+0000 7f4150c00640 1 --2- 172.21.15.96:0/204945456 >> [v2:172.21.15.123:6817/2041107525,v1:172.21.15.123:6819/2041107525] conn(0x7f4128074b90 0x7f4128076fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:17.968 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.912+0000 7faf53400640 1 --2- 172.21.15.96:0/1533109368 >> [v2:172.21.15.123:6816/2085300276,v1:172.21.15.123:6818/2085300276] conn(0x7faf34074b60 0x7faf34076f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:17.968 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.912+0000 7f98de000640 1 --2- 172.21.15.96:0/3683964447 >> [v2:172.21.15.123:6808/1217470164,v1:172.21.15.123:6809/1217470164] conn(0x7f98c0074b60 0x7f98c0076f80 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:17.968 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.912+0000 7f414b400640 1 --2- 172.21.15.96:0/204945456 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f412806c9f0 0x7f412806eeb0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f413c034000 tx=0x7f413c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:17.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.913+0000 7f96e3400640 1 -- 172.21.15.96:0/3483519320 --> [v2:172.21.15.96:6809/1243655428,v1:172.21.15.96:6811/1243655428] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f96dc0689e0 con 0x7f96bc074be0 2024-05-28T06:04:17.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.913+0000 7faf53400640 1 --2- 172.21.15.96:0/1533109368 >> [v2:172.21.15.123:6816/2085300276,v1:172.21.15.123:6818/2085300276] conn(0x7faf34074b60 0x7faf34076f80 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:17.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.913+0000 7f98dea00640 1 --2- 172.21.15.96:0/3683964447 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f98c006c940 0x7f98c006ee00 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f98e0166900 tx=0x7f98d0009590 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:17.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.913+0000 7f4150c00640 1 --2- 172.21.15.96:0/204945456 >> [v2:172.21.15.123:6817/2041107525,v1:172.21.15.123:6819/2041107525] conn(0x7f4128074b90 0x7f4128076fb0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:17.970 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.914+0000 7f96d8c00640 1 -- 172.21.15.96:0/3483519320 <== osd.0 v2:172.21.15.96:6809/1243655428 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f96dc0689e0 con 0x7f96bc074be0 2024-05-28T06:04:17.970 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.915+0000 7f96e3400640 1 -- 172.21.15.96:0/3483519320 >> [v2:172.21.15.96:6809/1243655428,v1:172.21.15.96:6811/1243655428] conn(0x7f96bc074be0 msgr2=0x7f96bc077000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:17.970 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.915+0000 7f96e3400640 1 --2- 172.21.15.96:0/3483519320 >> [v2:172.21.15.96:6809/1243655428,v1:172.21.15.96:6811/1243655428] conn(0x7f96bc074be0 0x7f96bc077000 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:17.970 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.915+0000 7f96e3400640 1 -- 172.21.15.96:0/3483519320 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f96bc06c9c0 msgr2=0x7f96bc06ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:17.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.915+0000 7f96e3400640 1 --2- 172.21.15.96:0/3483519320 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f96bc06c9c0 0x7f96bc06ee80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f96cc034000 tx=0x7f96cc02e000 comp rx=0 tx=0).stop 2024-05-28T06:04:17.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.915+0000 7f96e3400640 1 -- 172.21.15.96:0/3483519320 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f96dc15ea40 msgr2=0x7f96dc0ca310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:17.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.915+0000 7f96e3400640 1 --2- 172.21.15.96:0/3483519320 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f96dc15ea40 0x7f96dc0ca310 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f96c800e930 tx=0x7f96c800edf0 comp rx=0 tx=0).stop 2024-05-28T06:04:17.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.916+0000 7f98d6a00640 1 -- 172.21.15.96:0/3683964447 <== osd.6 v2:172.21.15.123:6808/1217470164 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f98d8036e90 con 0x7f98c0074b60 2024-05-28T06:04:17.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.916+0000 7f96e1400640 1 -- 172.21.15.96:0/3483519320 reap_dead start 2024-05-28T06:04:17.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.917+0000 7f96e3400640 1 -- 172.21.15.96:0/3483519320 shutdown_connections 2024-05-28T06:04:17.972 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.917+0000 7f96e3400640 1 -- 172.21.15.96:0/3483519320 >> 172.21.15.96:0/3483519320 conn(0x7f96dc08bfc0 msgr2=0x7f96dc0b1ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:17.972 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.917+0000 7faf4be00640 1 -- 172.21.15.96:0/1533109368 <== osd.7 v2:172.21.15.123:6816/2085300276 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7faf34077630 con 0x7faf34074b60 2024-05-28T06:04:17.972 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.917+0000 7f96e3400640 1 -- 172.21.15.96:0/3483519320 shutdown_connections 2024-05-28T06:04:17.972 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.917+0000 7f96e3400640 1 -- 172.21.15.96:0/3483519320 wait complete. 2024-05-28T06:04:17.972 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.917+0000 7f4143e00640 1 -- 172.21.15.96:0/204945456 <== osd.5 v2:172.21.15.123:6817/2041107525 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4128077660 con 0x7f4128074b90 2024-05-28T06:04:17.973 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.924+0000 7f29b0c00640 1 -- 172.21.15.96:0/853690032 --> [v2:172.21.15.96:6800/4059252009,v1:172.21.15.96:6801/4059252009] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f29a40057e0 con 0x7f298c074be0 2024-05-28T06:04:17.973 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.925+0000 7f299f400640 1 -- 172.21.15.96:0/853690032 <== osd.1 v2:172.21.15.96:6800/4059252009 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f29a40057e0 con 0x7f298c074be0 2024-05-28T06:04:17.973 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.926+0000 7f413da00640 1 -- 172.21.15.96:0/691666412 --> [v2:172.21.15.96:6820/3070158265,v1:172.21.15.96:6823/3070158265] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f41380b5af0 con 0x7f4118074d10 2024-05-28T06:04:17.974 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.927+0000 7f299cc00640 1 -- 172.21.15.96:0/853690032 >> [v2:172.21.15.96:6800/4059252009,v1:172.21.15.96:6801/4059252009] conn(0x7f298c074be0 msgr2=0x7f298c077000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:17.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.927+0000 7f299cc00640 1 --2- 172.21.15.96:0/853690032 >> [v2:172.21.15.96:6800/4059252009,v1:172.21.15.96:6801/4059252009] conn(0x7f298c074be0 0x7f298c077000 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:17.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.928+0000 7f299cc00640 1 -- 172.21.15.96:0/853690032 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f298c06c9c0 msgr2=0x7f298c06ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:17.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.928+0000 7f299cc00640 1 --2- 172.21.15.96:0/853690032 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f298c06c9c0 0x7f298c06ee80 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f29a400dc00 tx=0x7f29a0009240 comp rx=0 tx=0).stop 2024-05-28T06:04:17.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.928+0000 7f299cc00640 1 -- 172.21.15.96:0/853690032 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f29a40075f0 msgr2=0x7f29a40e4180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:17.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.928+0000 7f299cc00640 1 --2- 172.21.15.96:0/853690032 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f29a40075f0 0x7f29a40e4180 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f299800c430 tx=0x7f299800c8f0 comp rx=0 tx=0).stop 2024-05-28T06:04:17.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.928+0000 7f29abe00640 1 -- 172.21.15.96:0/853690032 reap_dead start 2024-05-28T06:04:17.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.928+0000 7f412ea00640 1 -- 172.21.15.96:0/691666412 <== osd.2 v2:172.21.15.96:6820/3070158265 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f41380b5af0 con 0x7f4118074d10 2024-05-28T06:04:17.976 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.928+0000 7f299cc00640 1 -- 172.21.15.96:0/853690032 shutdown_connections 2024-05-28T06:04:17.976 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.928+0000 7f299cc00640 1 -- 172.21.15.96:0/853690032 >> 172.21.15.96:0/853690032 conn(0x7f29a4058ee0 msgr2=0x7f29a4050ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:17.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.929+0000 7f413da00640 1 -- 172.21.15.96:0/691666412 >> [v2:172.21.15.96:6820/3070158265,v1:172.21.15.96:6823/3070158265] conn(0x7f4118074d10 msgr2=0x7f4118077130 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:17.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.929+0000 7f413da00640 1 --2- 172.21.15.96:0/691666412 >> [v2:172.21.15.96:6820/3070158265,v1:172.21.15.96:6823/3070158265] conn(0x7f4118074d10 0x7f4118077130 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:17.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.929+0000 7f413da00640 1 -- 172.21.15.96:0/691666412 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f411806caf0 msgr2=0x7f411806efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:17.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.929+0000 7f413da00640 1 --2- 172.21.15.96:0/691666412 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f411806caf0 0x7f411806efb0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4128000f60 tx=0x7f4128007c80 comp rx=0 tx=0).stop 2024-05-28T06:04:17.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.929+0000 7f413da00640 1 -- 172.21.15.96:0/691666412 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4138163170 msgr2=0x7f41380b64c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:17.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.929+0000 7f413da00640 1 --2- 172.21.15.96:0/691666412 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4138163170 0x7f41380b64c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4124031000 tx=0x7f41240046c0 comp rx=0 tx=0).stop 2024-05-28T06:04:17.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.929+0000 7f4137400640 1 -- 172.21.15.96:0/691666412 reap_dead start 2024-05-28T06:04:17.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.930+0000 7f413da00640 1 -- 172.21.15.96:0/691666412 shutdown_connections 2024-05-28T06:04:17.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.930+0000 7f413da00640 1 -- 172.21.15.96:0/691666412 >> 172.21.15.96:0/691666412 conn(0x7f41380b4000 msgr2=0x7f41380c54e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:17.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.930+0000 7f413da00640 1 -- 172.21.15.96:0/691666412 shutdown_connections 2024-05-28T06:04:17.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.930+0000 7f413da00640 1 -- 172.21.15.96:0/691666412 wait complete. 2024-05-28T06:04:17.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.931+0000 7f4f5e400640 1 -- 172.21.15.96:0/469072148 --> [v2:172.21.15.123:6800/3445772216,v1:172.21.15.123:6801/3445772216] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f4f580689e0 con 0x7f4f38074be0 2024-05-28T06:04:17.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.932+0000 7f4f54c00640 1 -- 172.21.15.96:0/469072148 <== osd.4 v2:172.21.15.123:6800/3445772216 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f4f580689e0 con 0x7f4f38074be0 2024-05-28T06:04:17.979 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.932+0000 7f299cc00640 1 -- 172.21.15.96:0/853690032 shutdown_connections 2024-05-28T06:04:17.979 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.932+0000 7f299cc00640 1 -- 172.21.15.96:0/853690032 wait complete. 2024-05-28T06:04:17.979 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.932+0000 7f220ce00640 1 -- 172.21.15.96:0/905762090 --> [v2:172.21.15.96:6808/2384313931,v1:172.21.15.96:6810/2384313931] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f22081658d0 con 0x7f21e8074b60 2024-05-28T06:04:17.980 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.932+0000 7f4f4e000640 1 -- 172.21.15.96:0/469072148 >> [v2:172.21.15.123:6800/3445772216,v1:172.21.15.123:6801/3445772216] conn(0x7f4f38074be0 msgr2=0x7f4f38077000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:17.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.932+0000 7f4f4e000640 1 --2- 172.21.15.96:0/469072148 >> [v2:172.21.15.123:6800/3445772216,v1:172.21.15.123:6801/3445772216] conn(0x7f4f38074be0 0x7f4f38077000 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:17.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.933+0000 7f4f4e000640 1 -- 172.21.15.96:0/469072148 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4f3806c9c0 msgr2=0x7f4f3806ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:17.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.933+0000 7f4f4e000640 1 --2- 172.21.15.96:0/469072148 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4f3806c9c0 0x7f4f3806ee80 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f4f44004e60 tx=0x7f4f44007f10 comp rx=0 tx=0).stop 2024-05-28T06:04:17.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.933+0000 7f4f4e000640 1 -- 172.21.15.96:0/469072148 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f58162940 msgr2=0x7f4f58164c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:17.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.933+0000 7f4f4e000640 1 --2- 172.21.15.96:0/469072148 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f58162940 0x7f4f58164c50 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f4f48031000 tx=0x7f4f480046c0 comp rx=0 tx=0).stop 2024-05-28T06:04:17.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.933+0000 7f4f5da00640 1 -- 172.21.15.96:0/469072148 reap_dead start 2024-05-28T06:04:17.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.933+0000 7f4f4e000640 1 -- 172.21.15.96:0/469072148 shutdown_connections 2024-05-28T06:04:17.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.933+0000 7f4f4e000640 1 -- 172.21.15.96:0/469072148 >> 172.21.15.96:0/469072148 conn(0x7f4f580b1fc0 msgr2=0x7f4f580ae6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:17.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.933+0000 7f4f4e000640 1 -- 172.21.15.96:0/469072148 shutdown_connections 2024-05-28T06:04:17.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.933+0000 7f4f4e000640 1 -- 172.21.15.96:0/469072148 wait complete. 2024-05-28T06:04:17.982 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.934+0000 7f21ff400640 1 -- 172.21.15.96:0/905762090 <== osd.3 v2:172.21.15.96:6808/2384313931 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f22081658d0 con 0x7f21e8074b60 2024-05-28T06:04:17.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.934+0000 7f220ce00640 1 -- 172.21.15.96:0/905762090 >> [v2:172.21.15.96:6808/2384313931,v1:172.21.15.96:6810/2384313931] conn(0x7f21e8074b60 msgr2=0x7f21e8076f80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:17.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.934+0000 7f220ce00640 1 --2- 172.21.15.96:0/905762090 >> [v2:172.21.15.96:6808/2384313931,v1:172.21.15.96:6810/2384313931] conn(0x7f21e8074b60 0x7f21e8076f80 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:17.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.934+0000 7f220ce00640 1 -- 172.21.15.96:0/905762090 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f21e806c940 msgr2=0x7f21e806ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:17.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.934+0000 7f220ce00640 1 --2- 172.21.15.96:0/905762090 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f21e806c940 0x7f21e806ee00 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f21f8034000 tx=0x7f21f802e000 comp rx=0 tx=0).stop 2024-05-28T06:04:17.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.934+0000 7f220ce00640 1 -- 172.21.15.96:0/905762090 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2208162d20 msgr2=0x7f2208165110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:17.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.934+0000 7f220ce00640 1 --2- 172.21.15.96:0/905762090 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2208162d20 0x7f2208165110 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f220000cad0 tx=0x7f220000cf90 comp rx=0 tx=0).stop 2024-05-28T06:04:17.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.934+0000 7f2207e00640 1 -- 172.21.15.96:0/905762090 reap_dead start 2024-05-28T06:04:17.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.935+0000 7f220ce00640 1 -- 172.21.15.96:0/905762090 shutdown_connections 2024-05-28T06:04:17.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.935+0000 7f220ce00640 1 -- 172.21.15.96:0/905762090 >> 172.21.15.96:0/905762090 conn(0x7f22080b1fc0 msgr2=0x7f22080ae830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:17.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.938+0000 7f220ce00640 1 -- 172.21.15.96:0/905762090 shutdown_connections 2024-05-28T06:04:17.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.938+0000 7f220ce00640 1 -- 172.21.15.96:0/905762090 wait complete. 2024-05-28T06:04:17.985 INFO:teuthology.orchestra.run.smithi096.stdout:55834574854 2024-05-28T06:04:17.986 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-05-28T06:04:17.987 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.944+0000 7f98e5800640 1 -- 172.21.15.96:0/3683964447 --> [v2:172.21.15.123:6808/1217470164,v1:172.21.15.123:6809/1217470164] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f98e0148040 con 0x7f98c0074b60 2024-05-28T06:04:17.987 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.945+0000 7f98d6a00640 1 -- 172.21.15.96:0/3683964447 <== osd.6 v2:172.21.15.123:6808/1217470164 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f98cc009480 con 0x7f98c0074b60 2024-05-28T06:04:17.988 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.946+0000 7f98e5800640 1 -- 172.21.15.96:0/3683964447 >> [v2:172.21.15.123:6808/1217470164,v1:172.21.15.123:6809/1217470164] conn(0x7f98c0074b60 msgr2=0x7f98c0076f80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:17.988 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.946+0000 7f98e5800640 1 --2- 172.21.15.96:0/3683964447 >> [v2:172.21.15.123:6808/1217470164,v1:172.21.15.123:6809/1217470164] conn(0x7f98c0074b60 0x7f98c0076f80 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:17.988 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.946+0000 7f98e5800640 1 -- 172.21.15.96:0/3683964447 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f98c006c940 msgr2=0x7f98c006ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:17.988 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.946+0000 7f98e5800640 1 --2- 172.21.15.96:0/3683964447 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f98c006c940 0x7f98c006ee00 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f98e0166900 tx=0x7f98d0009590 comp rx=0 tx=0).stop 2024-05-28T06:04:17.988 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.947+0000 7f98e5800640 1 -- 172.21.15.96:0/3683964447 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f98e015b850 msgr2=0x7f98e01f5ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:17.988 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.947+0000 7f98e5800640 1 --2- 172.21.15.96:0/3683964447 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f98e015b850 0x7f98e01f5ef0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f98d802a820 tx=0x7f98d80046c0 comp rx=0 tx=0).stop 2024-05-28T06:04:17.988 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.947+0000 7f98df400640 1 -- 172.21.15.96:0/3683964447 reap_dead start 2024-05-28T06:04:17.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.947+0000 7f98e5800640 1 -- 172.21.15.96:0/3683964447 shutdown_connections 2024-05-28T06:04:17.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.947+0000 7f98e5800640 1 -- 172.21.15.96:0/3683964447 >> 172.21.15.96:0/3683964447 conn(0x7f98e006bae0 msgr2=0x7f98e006eeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:17.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.947+0000 7f98e5800640 1 -- 172.21.15.96:0/3683964447 shutdown_connections 2024-05-28T06:04:17.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.947+0000 7f98e5800640 1 -- 172.21.15.96:0/3683964447 wait complete. 2024-05-28T06:04:17.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.950+0000 7f4152c00640 1 -- 172.21.15.96:0/204945456 --> [v2:172.21.15.123:6817/2041107525,v1:172.21.15.123:6819/2041107525] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f4124002dd0 con 0x7f4128074b90 2024-05-28T06:04:17.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.951+0000 7f4143e00640 1 -- 172.21.15.96:0/204945456 <== osd.5 v2:172.21.15.123:6817/2041107525 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f4124002dd0 con 0x7f4128074b90 2024-05-28T06:04:17.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.951+0000 7faf5aa00640 1 -- 172.21.15.96:0/1533109368 --> [v2:172.21.15.123:6816/2085300276,v1:172.21.15.123:6818/2085300276] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7faf540cae70 con 0x7faf34074b60 2024-05-28T06:04:17.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.952+0000 7f4152c00640 1 -- 172.21.15.96:0/204945456 >> [v2:172.21.15.123:6817/2041107525,v1:172.21.15.123:6819/2041107525] conn(0x7f4128074b90 msgr2=0x7f4128076fb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:17.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.952+0000 7f4152c00640 1 --2- 172.21.15.96:0/204945456 >> [v2:172.21.15.123:6817/2041107525,v1:172.21.15.123:6819/2041107525] conn(0x7f4128074b90 0x7f4128076fb0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:17.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.953+0000 7f4152c00640 1 -- 172.21.15.96:0/204945456 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f412806c9f0 msgr2=0x7f412806eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:17.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.953+0000 7f4152c00640 1 --2- 172.21.15.96:0/204945456 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f412806c9f0 0x7f412806eeb0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f413c034000 tx=0x7f413c02e000 comp rx=0 tx=0).stop 2024-05-28T06:04:17.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.953+0000 7f4152c00640 1 -- 172.21.15.96:0/204945456 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41300037a0 msgr2=0x7f414c15d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:17.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.953+0000 7f4152c00640 1 --2- 172.21.15.96:0/204945456 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41300037a0 0x7f414c15d500 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f41340010e0 tx=0x7f4134015420 comp rx=0 tx=0).stop 2024-05-28T06:04:17.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.953+0000 7f4150c00640 1 -- 172.21.15.96:0/204945456 reap_dead start 2024-05-28T06:04:17.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.953+0000 7f4152c00640 1 -- 172.21.15.96:0/204945456 shutdown_connections 2024-05-28T06:04:17.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.953+0000 7f4152c00640 1 -- 172.21.15.96:0/204945456 >> 172.21.15.96:0/204945456 conn(0x7f414c08bfc0 msgr2=0x7f414c0d3a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:17.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.953+0000 7f4152c00640 1 -- 172.21.15.96:0/204945456 shutdown_connections 2024-05-28T06:04:17.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.953+0000 7f4152c00640 1 -- 172.21.15.96:0/204945456 wait complete. 2024-05-28T06:04:17.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.954+0000 7faf4be00640 1 -- 172.21.15.96:0/1533109368 <== osd.7 v2:172.21.15.123:6816/2085300276 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7faf540cae70 con 0x7faf34074b60 2024-05-28T06:04:17.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.954+0000 7faf5aa00640 1 -- 172.21.15.96:0/1533109368 >> [v2:172.21.15.123:6816/2085300276,v1:172.21.15.123:6818/2085300276] conn(0x7faf34074b60 msgr2=0x7faf34076f80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:17.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.954+0000 7faf5aa00640 1 --2- 172.21.15.96:0/1533109368 >> [v2:172.21.15.123:6816/2085300276,v1:172.21.15.123:6818/2085300276] conn(0x7faf34074b60 0x7faf34076f80 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:17.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.955+0000 7faf5aa00640 1 -- 172.21.15.96:0/1533109368 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7faf3406c940 msgr2=0x7faf3406ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:17.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.955+0000 7faf5aa00640 1 --2- 172.21.15.96:0/1533109368 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7faf3406c940 0x7faf3406ee00 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7faf44006fd0 tx=0x7faf44008040 comp rx=0 tx=0).stop 2024-05-28T06:04:17.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.955+0000 7faf5aa00640 1 -- 172.21.15.96:0/1533109368 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf54167470 msgr2=0x7faf540d1430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:17.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.955+0000 7faf5aa00640 1 --2- 172.21.15.96:0/1533109368 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf54167470 0x7faf540d1430 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7faf4c00c7b0 tx=0x7faf4c00cc70 comp rx=0 tx=0).stop 2024-05-28T06:04:17.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.955+0000 7faf58a00640 1 -- 172.21.15.96:0/1533109368 reap_dead start 2024-05-28T06:04:17.993 INFO:teuthology.orchestra.run.smithi096.stdout:55834574855 2024-05-28T06:04:17.994 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-05-28T06:04:17.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.955+0000 7faf5aa00640 1 -- 172.21.15.96:0/1533109368 shutdown_connections 2024-05-28T06:04:17.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.955+0000 7faf5aa00640 1 -- 172.21.15.96:0/1533109368 >> 172.21.15.96:0/1533109368 conn(0x7faf5408bfc0 msgr2=0x7faf54161c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:17.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.956+0000 7faf5aa00640 1 -- 172.21.15.96:0/1533109368 shutdown_connections 2024-05-28T06:04:17.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:17.956+0000 7faf5aa00640 1 -- 172.21.15.96:0/1533109368 wait complete. 2024-05-28T06:04:17.995 INFO:teuthology.orchestra.run.smithi096.stdout:55834574855 2024-05-28T06:04:17.995 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-05-28T06:04:17.996 INFO:teuthology.orchestra.run.smithi096.stdout:55834574854 2024-05-28T06:04:17.996 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-05-28T06:04:17.997 INFO:teuthology.orchestra.run.smithi096.stdout:55834574856 2024-05-28T06:04:17.997 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-05-28T06:04:17.998 INFO:teuthology.orchestra.run.smithi096.stdout:55834574856 2024-05-28T06:04:17.998 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-05-28T06:04:17.999 INFO:teuthology.orchestra.run.smithi096.stdout:55834574854 2024-05-28T06:04:18.000 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-05-28T06:04:18.000 INFO:teuthology.orchestra.run.smithi096.stdout:55834574854 2024-05-28T06:04:18.001 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-05-28T06:04:18.325 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.325+0000 7fc450a00640 1 Processor -- start 2024-05-28T06:04:18.325 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.325+0000 7fc450a00640 1 -- start start 2024-05-28T06:04:18.328 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.327+0000 7fc450a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc44c072eb0 0x7fc44c073300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:18.328 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.327+0000 7fc450a00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fc44c15a490 0x7fc44c15a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:18.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.327+0000 7fc450a00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc44c15c290 con 0x7fc44c072000 2024-05-28T06:04:18.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.327+0000 7fc450a00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fc44c073840 con 0x7fc44c15a490 2024-05-28T06:04:18.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.327+0000 7fc450a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fc44c15c570 con 0x7fc44c072eb0 2024-05-28T06:04:18.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.328+0000 7fc44b200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fc44c15a490 0x7fc44c15a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:18.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.328+0000 7fc44b200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fc44c15a490 0x7fc44c15a8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.96:58564/0 (socket says 172.21.15.96:58564) 2024-05-28T06:04:18.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.328+0000 7fc449e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc44c072eb0 0x7fc44c073300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:18.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.328+0000 7fc44b200640 1 -- 172.21.15.96:0/974672398 learned_addr learned my addr 172.21.15.96:0/974672398 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:18.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.329+0000 7fc44b200640 1 -- 172.21.15.96:0/974672398 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc44c072eb0 msgr2=0x7fc44c073300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:18.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.329+0000 7fc44b200640 1 --2- 172.21.15.96:0/974672398 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc44c072eb0 0x7fc44c073300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:18.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.329+0000 7fc44b200640 1 -- 172.21.15.96:0/974672398 >> v1:172.21.15.123:6789/0 conn(0x7fc44c072000 legacy=0x7fc44c0b3aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:18.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.329+0000 7fc44b200640 1 -- 172.21.15.96:0/974672398 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc44c15c850 con 0x7fc44c15a490 2024-05-28T06:04:18.331 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.330+0000 7f10e4a00640 1 Processor -- start 2024-05-28T06:04:18.331 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.331+0000 7f10e4a00640 1 -- start start 2024-05-28T06:04:18.331 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.331+0000 7fc44b200640 1 --2- 172.21.15.96:0/974672398 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc44c15a490 0x7fc44c15a8e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc444009e50 tx=0x7fc44402a710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9741dca16cf2d1e1 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:18.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.331+0000 7f5650c00640 1 Processor -- start 2024-05-28T06:04:18.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.331+0000 7fc449400640 1 -- 172.21.15.96:0/974672398 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc444036070 con 0x7fc44c15a490 2024-05-28T06:04:18.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.331+0000 7fc449400640 1 -- 172.21.15.96:0/974672398 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc444031600 con 0x7fc44c15a490 2024-05-28T06:04:18.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.331+0000 7fc449400640 1 -- 172.21.15.96:0/974672398 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4440319d0 con 0x7fc44c15a490 2024-05-28T06:04:18.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.332+0000 7fc450a00640 1 -- 172.21.15.96:0/974672398 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc44c15a490 msgr2=0x7fc44c15a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:18.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.332+0000 7fc450a00640 1 --2- 172.21.15.96:0/974672398 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc44c15a490 0x7fc44c15a8e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc444009e50 tx=0x7fc44402a710 comp rx=0 tx=0).stop 2024-05-28T06:04:18.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.332+0000 7f5df2800640 1 Processor -- start 2024-05-28T06:04:18.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.332+0000 7f5df2800640 1 -- start start 2024-05-28T06:04:18.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.333+0000 7f10e4a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f10d80e1080 0x7f10d8038620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:18.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.333+0000 7f10e4a00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f10d8050dc0 0x7f10d8039eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:18.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.333+0000 7f10e4a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f10d803a3f0 0x7f10d80e4a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:18.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.332+0000 7ff8b1800640 1 Processor -- start 2024-05-28T06:04:18.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.332+0000 7ff8b1800640 1 -- start start 2024-05-28T06:04:18.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.333+0000 7fc450a00640 1 -- 172.21.15.96:0/974672398 shutdown_connections 2024-05-28T06:04:18.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.333+0000 7fc450a00640 1 --2- 172.21.15.96:0/974672398 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc44c15a490 0x7fc44c15a8e0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:18.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.333+0000 7fc450a00640 1 --2- 172.21.15.96:0/974672398 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc44c072eb0 0x7fc44c073300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:18.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.333+0000 7fc450a00640 1 -- 172.21.15.96:0/974672398 >> 172.21.15.96:0/974672398 conn(0x7fc44c06c380 msgr2=0x7fc44c0b1db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:18.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.333+0000 7f10e4a00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f10d804f530 con 0x7f10d80e1080 2024-05-28T06:04:18.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.333+0000 7f10e4a00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f10d804f810 con 0x7f10d8050dc0 2024-05-28T06:04:18.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.333+0000 7f10e4a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f10d804faf0 con 0x7f10d803a3f0 2024-05-28T06:04:18.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.334+0000 7f10df200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f10d803a3f0 0x7f10d80e4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:18.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.334+0000 7f10df200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f10d803a3f0 0x7f10d80e4a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.96:58472/0 (socket says 172.21.15.96:58472) 2024-05-28T06:04:18.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.334+0000 7f10df200640 1 -- 172.21.15.96:0/3266152020 learned_addr learned my addr 172.21.15.96:0/3266152020 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:18.337 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.334+0000 7f5df2800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5dec0d12f0 0x7f5dec0d16d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:18.337 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.334+0000 7f5df2800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f5dec0d3600 0x7f5dec0cfad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:18.337 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.334+0000 7f5df2800640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5dec0d2000 con 0x7f5dec0d05b0 2024-05-28T06:04:18.337 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.334+0000 7f5df2800640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f5dec0d2820 con 0x7f5dec0d3600 2024-05-28T06:04:18.337 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.334+0000 7f5df2800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f5dec0d2b00 con 0x7f5dec0d12f0 2024-05-28T06:04:18.337 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.334+0000 7f10de800640 1 --2- 172.21.15.96:0/3266152020 >> v2:172.21.15.96:3300/0 conn(0x7f10d80e1080 0x7f10d8038620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:18.337 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.334+0000 7f10df200640 1 -- 172.21.15.96:0/3266152020 >> v2:172.21.15.123:3300/0 conn(0x7f10d8050dc0 msgr2=0x7f10d8039eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:18.337 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.334+0000 7f10df200640 1 --2- 172.21.15.96:0/3266152020 >> v2:172.21.15.123:3300/0 conn(0x7f10d8050dc0 0x7f10d8039eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:18.337 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.335+0000 7f10df200640 1 -- 172.21.15.96:0/3266152020 >> v2:172.21.15.96:3300/0 conn(0x7f10d80e1080 msgr2=0x7f10d8038620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:18.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.334+0000 7ff8b1800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff8ac163870 0x7ff8ac163d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:18.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.335+0000 7ff8b1800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7ff8ac0cf490 0x7ff8ac0cf920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:18.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.335+0000 7ff8b1800640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8ac163480 con 0x7ff8ac0d1020 2024-05-28T06:04:18.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.335+0000 7ff8b1800640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7ff8ac0d3450 con 0x7ff8ac0cf490 2024-05-28T06:04:18.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.335+0000 7ff8b1800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ff8ac0d2b20 con 0x7ff8ac163870 2024-05-28T06:04:18.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.331+0000 7f5650c00640 1 -- start start 2024-05-28T06:04:18.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.333+0000 7f5650c00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f564c0d04a0 0x7f564c0d0880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:18.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.333+0000 7f5650c00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f564c0b4350 con 0x7f564c162e70 2024-05-28T06:04:18.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.333+0000 7f5650c00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f564c0b32d0 con 0x7f564c06f6e0 2024-05-28T06:04:18.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.333+0000 7f5650c00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f564c0b3ec0 con 0x7f564c0d04a0 2024-05-28T06:04:18.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.334+0000 7f5649e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f564c0d04a0 0x7f564c0d0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:18.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.334+0000 7f5649e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f564c0d04a0 0x7f564c0d0880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:41694/0 (socket says 172.21.15.96:41694) 2024-05-28T06:04:18.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.334+0000 7f5649e00640 1 -- 172.21.15.96:0/2216500621 learned_addr learned my addr 172.21.15.96:0/2216500621 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:18.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.334+0000 7f5649e00640 1 -- 172.21.15.96:0/2216500621 >> v1:172.21.15.123:6789/0 conn(0x7f564c06f6e0 legacy=0x7f564c0cd4f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-28T06:04:18.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.334+0000 7f5649e00640 1 -- 172.21.15.96:0/2216500621 >> v1:172.21.15.96:6789/0 conn(0x7f564c162e70 legacy=0x7f564c0ce230 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-28T06:04:18.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.334+0000 7f5649e00640 1 -- 172.21.15.96:0/2216500621 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f564c0b3590 con 0x7f564c0d04a0 2024-05-28T06:04:18.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.335+0000 7f5deb400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5dec0d12f0 0x7f5dec0d16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:18.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.335+0000 7f5deb400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5dec0d12f0 0x7f5dec0d16d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.96:58480/0 (socket says 172.21.15.96:58480) 2024-05-28T06:04:18.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.335+0000 7f5deb400640 1 -- 172.21.15.96:0/218672419 learned_addr learned my addr 172.21.15.96:0/218672419 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:18.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.335+0000 7f6489400640 1 Processor -- start 2024-05-28T06:04:18.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.336+0000 7f6489400640 1 -- start start 2024-05-28T06:04:18.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.336+0000 7ff8aa000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff8ac163870 0x7ff8ac163d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:18.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.336+0000 7ff8aaa00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7ff8ac0d1020 0x7ff8ac0d1400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:55508/0 (socket says 172.21.15.96:55508) 2024-05-28T06:04:18.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.336+0000 7ff8aaa00640 1 -- 172.21.15.96:0/487261255 learned_addr learned my addr 172.21.15.96:0/487261255 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:04:18.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.336+0000 7f5debe00640 1 --2- 172.21.15.96:0/218672419 >> v2:172.21.15.96:3300/0 conn(0x7f5dec0d3600 0x7f5dec0cfad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:18.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.336+0000 7ff8ab400640 1 --2- 172.21.15.96:0/487261255 >> v2:172.21.15.123:3300/0 conn(0x7ff8ac0cf490 0x7ff8ac0cf920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:18.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.336+0000 7f8f9a000640 1 Processor -- start 2024-05-28T06:04:18.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.336+0000 7ff8a9600640 1 -- 172.21.15.96:0/487261255 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1371402712 0 0) 0x7ff8ac163480 con 0x7ff8ac0d1020 2024-05-28T06:04:18.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.336+0000 7ff8a9600640 1 -- 172.21.15.96:0/487261255 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff890003520 con 0x7ff8ac0d1020 2024-05-28T06:04:18.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.335+0000 7f10df200640 1 --2- 172.21.15.96:0/3266152020 >> v2:172.21.15.96:3300/0 conn(0x7f10d80e1080 0x7f10d8038620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:18.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.335+0000 7f10df200640 1 -- 172.21.15.96:0/3266152020 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10d804fdd0 con 0x7f10d803a3f0 2024-05-28T06:04:18.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.337+0000 7f10de800640 1 --2- 172.21.15.96:0/3266152020 >> v2:172.21.15.96:3300/0 conn(0x7f10d80e1080 0x7f10d8038620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:04:18.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.336+0000 7fc450a00640 1 -- 172.21.15.96:0/974672398 shutdown_connections 2024-05-28T06:04:18.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.336+0000 7fc450a00640 1 -- 172.21.15.96:0/974672398 wait complete. 2024-05-28T06:04:18.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.337+0000 7ff8aa000640 1 -- 172.21.15.96:0/487261255 >> v2:172.21.15.123:3300/0 conn(0x7ff8ac0cf490 msgr2=0x7ff8ac0cf920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:18.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.337+0000 7ff8aa000640 1 --2- 172.21.15.96:0/487261255 >> v2:172.21.15.123:3300/0 conn(0x7ff8ac0cf490 0x7ff8ac0cf920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:18.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.337+0000 7ff8aa000640 1 -- 172.21.15.96:0/487261255 >> v1:172.21.15.96:6789/0 conn(0x7ff8ac0d1020 legacy=0x7ff8ac0d1400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:18.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.337+0000 7ff8aa000640 1 -- 172.21.15.96:0/487261255 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8ac0d2e00 con 0x7ff8ac163870 2024-05-28T06:04:18.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.337+0000 7ff8ab400640 1 --2- 172.21.15.96:0/487261255 >> v2:172.21.15.123:3300/0 conn(0x7ff8ac0cf490 0x7ff8ac0cf920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-28T06:04:18.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.337+0000 7f5649e00640 1 --2- 172.21.15.96:0/2216500621 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f564c0d04a0 0x7f564c0d0880 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f563c01a870 tx=0x7f563c03b1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e1fa40fa61a5fa6 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:18.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.337+0000 7f8f9a000640 1 -- start start 2024-05-28T06:04:18.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.338+0000 7fc450a00640 1 Processor -- start 2024-05-28T06:04:18.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.337+0000 7f6489400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f64840b34e0 0x7f64840725d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:18.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.337+0000 7f6489400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f648406fcb0 0x7f648415a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:18.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.338+0000 7f6489400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f648415a7f0 0x7f64840ad230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:18.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.338+0000 7f6489400640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f64840b3c50 con 0x7f648406fcb0 2024-05-28T06:04:18.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.338+0000 7f6489400640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f64840b3f30 con 0x7f64840b34e0 2024-05-28T06:04:18.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.338+0000 7f6489400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f64840b4210 con 0x7f648415a7f0 2024-05-28T06:04:18.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.338+0000 7f6483400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f648406fcb0 0x7f648415a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:18.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.338+0000 7f5648a00640 1 -- 172.21.15.96:0/2216500621 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f563c048070 con 0x7f564c0d04a0 2024-05-28T06:04:18.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.338+0000 7f6483400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f648406fcb0 0x7f648415a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:41728/0 (socket says 172.21.15.96:41728) 2024-05-28T06:04:18.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.338+0000 7f6483400640 1 -- 172.21.15.96:0/1961509221 learned_addr learned my addr 172.21.15.96:0/1961509221 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:18.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.338+0000 7f5648a00640 1 -- 172.21.15.96:0/2216500621 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f563c043440 con 0x7f564c0d04a0 2024-05-28T06:04:18.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.338+0000 7f5648a00640 1 -- 172.21.15.96:0/2216500621 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f563c043810 con 0x7f564c0d04a0 2024-05-28T06:04:18.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.337+0000 7f5deb400640 1 -- 172.21.15.96:0/218672419 >> v2:172.21.15.96:3300/0 conn(0x7f5dec0d3600 msgr2=0x7f5dec0cfad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:18.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.337+0000 7f5deb400640 1 --2- 172.21.15.96:0/218672419 >> v2:172.21.15.96:3300/0 conn(0x7f5dec0d3600 0x7f5dec0cfad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:18.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.337+0000 7f5deb400640 1 -- 172.21.15.96:0/218672419 >> v1:172.21.15.96:6789/0 conn(0x7f5dec0d05b0 legacy=0x7f5dec0cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:18.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.337+0000 7f5deb400640 1 -- 172.21.15.96:0/218672419 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dec0d1c10 con 0x7f5dec0d12f0 2024-05-28T06:04:18.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.339+0000 7ff8aa000640 1 --2- 172.21.15.96:0/487261255 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff8ac163870 0x7ff8ac163d00 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff8980097a0 tx=0x7ff89802a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=480fa529d60b4b02 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:18.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.338+0000 7f8f9a000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f8f9415aa20 0x7f8f9415ae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:18.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.339+0000 7f10df200640 1 --2- 172.21.15.96:0/3266152020 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f10d803a3f0 0x7f10d80e4a20 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f10d8052560 tx=0x7f10cc02a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e4d0cbd2fcb0803 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:18.349 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.339+0000 7fc450a00640 1 -- start start 2024-05-28T06:04:18.349 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.339+0000 7f6483400640 1 -- 172.21.15.96:0/1961509221 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f648415a7f0 msgr2=0x7f64840ad230 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-28T06:04:18.349 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.339+0000 7f5deb400640 1 --2- 172.21.15.96:0/218672419 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5dec0d12f0 0x7f5dec0d16d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5ddc009870 tx=0x7f5ddc02a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=98a7785c5302885e server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:18.349 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.338+0000 7f8f9a000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f8f940d0d80 0x7f8f940cde30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:18.350 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.339+0000 7ff8a9600640 1 -- 172.21.15.96:0/487261255 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff898037070 con 0x7ff8ac163870 2024-05-28T06:04:18.350 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.339+0000 7f5650c00640 1 -- 172.21.15.96:0/2216500621 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f564c0d04a0 msgr2=0x7f564c0d0880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:18.350 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.339+0000 7f5650c00640 1 --2- 172.21.15.96:0/2216500621 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f564c0d04a0 0x7f564c0d0880 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f563c01a870 tx=0x7f563c03b1c0 comp rx=0 tx=0).stop 2024-05-28T06:04:18.350 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.339+0000 7ff8a9600640 1 -- 172.21.15.96:0/487261255 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff898032440 con 0x7ff8ac163870 2024-05-28T06:04:18.351 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.339+0000 7ff8a9600640 1 -- 172.21.15.96:0/487261255 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff898032810 con 0x7ff8ac163870 2024-05-28T06:04:18.351 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.339+0000 7f8f9a000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8f94073020 0x7f8f940ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:18.351 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.339+0000 7f6483400640 1 --2- 172.21.15.96:0/1961509221 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f648415a7f0 0x7f64840ad230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:18.351 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.339+0000 7f6483400640 1 -- 172.21.15.96:0/1961509221 >> v2:172.21.15.123:3300/0 conn(0x7f64840b34e0 msgr2=0x7f64840725d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:18.352 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.339+0000 7f5650c00640 1 -- 172.21.15.96:0/2216500621 shutdown_connections 2024-05-28T06:04:18.352 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.339+0000 7f5650c00640 1 --2- 172.21.15.96:0/2216500621 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f564c0d04a0 0x7f564c0d0880 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:18.352 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.339+0000 7f5650c00640 1 -- 172.21.15.96:0/2216500621 >> 172.21.15.96:0/2216500621 conn(0x7f564c06bae0 msgr2=0x7f564c06bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:18.352 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.339+0000 7f8f9a000640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f8f940ae520 con 0x7f8f9415aa20 2024-05-28T06:04:18.352 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.339+0000 7f8f9a000640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f8f940ae800 con 0x7f8f940d0d80 2024-05-28T06:04:18.352 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.339+0000 7f8f9a000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f8f940aeae0 con 0x7f8f94073020 2024-05-28T06:04:18.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.340+0000 7f5650c00640 1 -- 172.21.15.96:0/2216500621 shutdown_connections 2024-05-28T06:04:18.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.339+0000 7f8f93e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8f94073020 0x7f8f940ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:18.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.340+0000 7f8f93400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f8f9415aa20 0x7f8f9415ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:18.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.340+0000 7f8f92a00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f8f940d0d80 0x7f8f940cde30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:18.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.340+0000 7f8f93e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8f94073020 0x7f8f940ce370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.96:58490/0 (socket says 172.21.15.96:58490) 2024-05-28T06:04:18.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.340+0000 7f5deaa00640 1 -- 172.21.15.96:0/218672419 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ddc037070 con 0x7f5dec0d12f0 2024-05-28T06:04:18.354 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.340+0000 7f8f93e00640 1 -- 172.21.15.96:0/2583421462 learned_addr learned my addr 172.21.15.96:0/2583421462 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:18.354 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.340+0000 7f5650c00640 1 -- 172.21.15.96:0/2216500621 wait complete. 2024-05-28T06:04:18.354 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.340+0000 7f6483e00640 1 --2- 172.21.15.96:0/1961509221 >> v2:172.21.15.123:3300/0 conn(0x7f64840b34e0 0x7f64840725d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:18.355 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.340+0000 7f6483400640 1 --2- 172.21.15.96:0/1961509221 >> v2:172.21.15.123:3300/0 conn(0x7f64840b34e0 0x7f64840725d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:18.355 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.340+0000 7f5deaa00640 1 -- 172.21.15.96:0/218672419 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ddc02ae10 con 0x7f5dec0d12f0 2024-05-28T06:04:18.355 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.340+0000 7f10dd400640 1 -- 172.21.15.96:0/3266152020 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10cc037070 con 0x7f10d803a3f0 2024-05-28T06:04:18.355 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.340+0000 7f10dd400640 1 -- 172.21.15.96:0/3266152020 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10cc032440 con 0x7f10d803a3f0 2024-05-28T06:04:18.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.340+0000 7f10dd400640 1 -- 172.21.15.96:0/3266152020 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10cc032850 con 0x7f10d803a3f0 2024-05-28T06:04:18.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.340+0000 7f6483400640 1 -- 172.21.15.96:0/1961509221 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64840b44f0 con 0x7f648406fcb0 2024-05-28T06:04:18.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.340+0000 7f5deaa00640 1 -- 172.21.15.96:0/218672419 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ddc032660 con 0x7f5dec0d12f0 2024-05-28T06:04:18.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.341+0000 7f8f93400640 1 -- 172.21.15.96:0/2583421462 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8f94073020 msgr2=0x7f8f940ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:18.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.341+0000 7f8f93400640 1 --2- 172.21.15.96:0/2583421462 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8f94073020 0x7f8f940ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:18.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.341+0000 7fc450a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc44c072000 0x7fc44c15b110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:18.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.341+0000 7f5df2800640 1 -- 172.21.15.96:0/218672419 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5dec0d12f0 msgr2=0x7f5dec0d16d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:18.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.341+0000 7f5df2800640 1 --2- 172.21.15.96:0/218672419 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5dec0d12f0 0x7f5dec0d16d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5ddc009870 tx=0x7f5ddc02a180 comp rx=0 tx=0).stop 2024-05-28T06:04:18.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.341+0000 7f10e4a00640 1 -- 172.21.15.96:0/3266152020 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f10d803a3f0 msgr2=0x7f10d80e4a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:18.358 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.341+0000 7f10e4a00640 1 --2- 172.21.15.96:0/3266152020 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f10d803a3f0 0x7f10d80e4a20 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f10d8052560 tx=0x7f10cc02a210 comp rx=0 tx=0).stop 2024-05-28T06:04:18.358 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.341+0000 7f8f93400640 1 -- 172.21.15.96:0/2583421462 >> v2:172.21.15.123:3300/0 conn(0x7f8f940d0d80 msgr2=0x7f8f940cde30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:18.358 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.341+0000 7f8f93400640 1 --2- 172.21.15.96:0/2583421462 >> v2:172.21.15.123:3300/0 conn(0x7f8f940d0d80 0x7f8f940cde30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:18.358 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.341+0000 7f8f93400640 1 -- 172.21.15.96:0/2583421462 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f940aedc0 con 0x7f8f9415aa20 2024-05-28T06:04:18.358 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.341+0000 7fc450a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc44c072eb0 0x7fc44c1584c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:18.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.341+0000 7ff8b1800640 1 -- 172.21.15.96:0/487261255 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff8ac163870 msgr2=0x7ff8ac163d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:18.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.341+0000 7ff8b1800640 1 --2- 172.21.15.96:0/487261255 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff8ac163870 0x7ff8ac163d00 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff8980097a0 tx=0x7ff89802a180 comp rx=0 tx=0).stop 2024-05-28T06:04:18.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.341+0000 7f10e4a00640 1 -- 172.21.15.96:0/3266152020 shutdown_connections 2024-05-28T06:04:18.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.341+0000 7f10e4a00640 1 --2- 172.21.15.96:0/3266152020 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f10d803a3f0 0x7f10d80e4a20 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:18.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.341+0000 7f10e4a00640 1 --2- 172.21.15.96:0/3266152020 >> v2:172.21.15.123:3300/0 conn(0x7f10d8050dc0 0x7f10d8039eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:18.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.341+0000 7f10e4a00640 1 --2- 172.21.15.96:0/3266152020 >> v2:172.21.15.96:3300/0 conn(0x7f10d80e1080 0x7f10d8038620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:18.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.341+0000 7f10e4a00640 1 -- 172.21.15.96:0/3266152020 >> 172.21.15.96:0/3266152020 conn(0x7f10d8009fa0 msgr2=0x7f10d800a380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:18.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.342+0000 7f10e4a00640 1 -- 172.21.15.96:0/3266152020 shutdown_connections 2024-05-28T06:04:18.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.341+0000 7fc450a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc44c15a490 0x7fc44c148390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:18.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.341+0000 7fc450a00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc44c15e060 con 0x7fc44c072000 2024-05-28T06:04:18.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.341+0000 7fc450a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc44c0cc700 con 0x7fc44c072eb0 2024-05-28T06:04:18.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.341+0000 7fc450a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fc44c0b2a80 con 0x7fc44c15a490 2024-05-28T06:04:18.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.342+0000 7fc449e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc44c072eb0 0x7fc44c1584c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:18.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.342+0000 7fc449e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc44c072eb0 0x7fc44c1584c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.96:58616/0 (socket says 172.21.15.96:58616) 2024-05-28T06:04:18.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.342+0000 7fc449e00640 1 -- 172.21.15.96:0/1652150309 learned_addr learned my addr 172.21.15.96:0/1652150309 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:18.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.342+0000 7ff8b1800640 1 -- 172.21.15.96:0/487261255 shutdown_connections 2024-05-28T06:04:18.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.342+0000 7ff8b1800640 1 --2- 172.21.15.96:0/487261255 >> v2:172.21.15.123:3300/0 conn(0x7ff8ac0cf490 0x7ff8ac0cf920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:18.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.342+0000 7ff8b1800640 1 --2- 172.21.15.96:0/487261255 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff8ac163870 0x7ff8ac163d00 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:18.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.342+0000 7ff8b1800640 1 -- 172.21.15.96:0/487261255 >> 172.21.15.96:0/487261255 conn(0x7ff8ac08bfc0 msgr2=0x7ff8ac0d4650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:18.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.342+0000 7f5df2800640 1 -- 172.21.15.96:0/218672419 shutdown_connections 2024-05-28T06:04:18.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.342+0000 7f5df2800640 1 --2- 172.21.15.96:0/218672419 >> v2:172.21.15.96:3300/0 conn(0x7f5dec0d3600 0x7f5dec0cfad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:18.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.342+0000 7f5df2800640 1 --2- 172.21.15.96:0/218672419 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5dec0d12f0 0x7f5dec0d16d0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:18.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.342+0000 7f5df2800640 1 -- 172.21.15.96:0/218672419 >> 172.21.15.96:0/218672419 conn(0x7f5dec08bfc0 msgr2=0x7f5dec0d4670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:18.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.342+0000 7f6483400640 1 --2- 172.21.15.96:0/1961509221 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f648406fcb0 0x7f648415a2b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f64840b3080 tx=0x7f647002a280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bbc3b5796c8f69b server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:18.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.342+0000 7f10e4a00640 1 -- 172.21.15.96:0/3266152020 wait complete. 2024-05-28T06:04:18.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.342+0000 7fc44b200640 1 --2- 172.21.15.96:0/1652150309 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc44c15a490 0x7fc44c148390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:18.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.342+0000 7fc44a800640 1 --2- 172.21.15.96:0/1652150309 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc44c072000 0x7fc44c15b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:18.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.342+0000 7ff8b1800640 1 -- 172.21.15.96:0/487261255 shutdown_connections 2024-05-28T06:04:18.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.342+0000 7f5df2800640 1 -- 172.21.15.96:0/218672419 shutdown_connections 2024-05-28T06:04:18.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.342+0000 7f5650c00640 1 Processor -- start 2024-05-28T06:04:18.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.342+0000 7f5650c00640 1 -- start start 2024-05-28T06:04:18.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.342+0000 7f6482a00640 1 -- 172.21.15.96:0/1961509221 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6470037070 con 0x7f648406fcb0 2024-05-28T06:04:18.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.342+0000 7f6482a00640 1 -- 172.21.15.96:0/1961509221 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6470032440 con 0x7f648406fcb0 2024-05-28T06:04:18.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.342+0000 7f6482a00640 1 -- 172.21.15.96:0/1961509221 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6470032810 con 0x7f648406fcb0 2024-05-28T06:04:18.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.343+0000 7f6489400640 1 -- 172.21.15.96:0/1961509221 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f648406fcb0 msgr2=0x7f648415a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:18.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.343+0000 7f6489400640 1 --2- 172.21.15.96:0/1961509221 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f648406fcb0 0x7f648415a2b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f64840b3080 tx=0x7f647002a280 comp rx=0 tx=0).stop 2024-05-28T06:04:18.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.343+0000 7f6489400640 1 -- 172.21.15.96:0/1961509221 shutdown_connections 2024-05-28T06:04:18.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.343+0000 7f6489400640 1 --2- 172.21.15.96:0/1961509221 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f648415a7f0 0x7f64840ad230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:18.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.343+0000 7f6489400640 1 --2- 172.21.15.96:0/1961509221 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f648406fcb0 0x7f648415a2b0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:18.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.343+0000 7f6489400640 1 --2- 172.21.15.96:0/1961509221 >> v2:172.21.15.123:3300/0 conn(0x7f64840b34e0 0x7f64840725d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:18.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.343+0000 7f6489400640 1 -- 172.21.15.96:0/1961509221 >> 172.21.15.96:0/1961509221 conn(0x7f64840b1fc0 msgr2=0x7f64840b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:18.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.343+0000 7f6489400640 1 -- 172.21.15.96:0/1961509221 shutdown_connections 2024-05-28T06:04:18.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.344+0000 7f6489400640 1 -- 172.21.15.96:0/1961509221 wait complete. 2024-05-28T06:04:18.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.344+0000 7fc449e00640 1 -- 172.21.15.96:0/1652150309 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc44c15a490 msgr2=0x7fc44c148390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:18.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.344+0000 7f8f93400640 1 --2- 172.21.15.96:0/2583421462 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8f9415aa20 0x7f8f9415ae00 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f8f94072100 tx=0x7f8f8402a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97303fa9e488cbc4 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:18.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.344+0000 7f5650c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f564c06f6e0 0x7f564c155da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:18.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.344+0000 7f8f92000640 1 -- 172.21.15.96:0/2583421462 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f84031600 con 0x7f8f9415aa20 2024-05-28T06:04:18.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.344+0000 7f8f92000640 1 -- 172.21.15.96:0/2583421462 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f840317a0 con 0x7f8f9415aa20 2024-05-28T06:04:18.367 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.344+0000 7f8f92000640 1 -- 172.21.15.96:0/2583421462 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f84031b70 con 0x7f8f9415aa20 2024-05-28T06:04:18.367 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.344+0000 7fc449e00640 1 --2- 172.21.15.96:0/1652150309 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc44c15a490 0x7fc44c148390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:18.367 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.344+0000 7fc449e00640 1 -- 172.21.15.96:0/1652150309 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc44c072000 msgr2=0x7fc44c15b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:18.367 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.344+0000 7fc449e00640 1 --2- 172.21.15.96:0/1652150309 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc44c072000 0x7fc44c15b110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:18.367 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.344+0000 7fc449e00640 1 -- 172.21.15.96:0/1652150309 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc44c0adfa0 con 0x7fc44c072eb0 2024-05-28T06:04:18.367 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.344+0000 7f5650c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f564c0d04a0 0x7f564c0ad130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:18.368 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.345+0000 7f5650c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f564c162e70 0x7f564c0ad670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:18.368 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.345+0000 7f5650c00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f564c0d0dc0 con 0x7f564c06f6e0 2024-05-28T06:04:18.368 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.345+0000 7f5650c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f564c0b3b50 con 0x7f564c0d04a0 2024-05-28T06:04:18.368 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.345+0000 7f5650c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f564c0ce940 con 0x7f564c162e70 2024-05-28T06:04:18.368 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.345+0000 7f10e4a00640 1 Processor -- start 2024-05-28T06:04:18.368 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.342+0000 7ff8b1800640 1 -- 172.21.15.96:0/487261255 wait complete. 2024-05-28T06:04:18.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.345+0000 7f10e4a00640 1 -- start start 2024-05-28T06:04:18.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.343+0000 7f5df2800640 1 -- 172.21.15.96:0/218672419 wait complete. 2024-05-28T06:04:18.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.345+0000 7f5649e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f564c06f6e0 0x7f564c155da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:18.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.346+0000 7f8f9a000640 1 -- 172.21.15.96:0/2583421462 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8f9415aa20 msgr2=0x7f8f9415ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:18.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.346+0000 7f8f9a000640 1 --2- 172.21.15.96:0/2583421462 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8f9415aa20 0x7f8f9415ae00 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f8f94072100 tx=0x7f8f8402a5d0 comp rx=0 tx=0).stop 2024-05-28T06:04:18.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.346+0000 7fc449e00640 1 --2- 172.21.15.96:0/1652150309 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc44c072eb0 0x7fc44c1584c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc43800e9b0 tx=0x7fc43800ee70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:18.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.346+0000 7fc442a00640 1 -- 172.21.15.96:0/1652150309 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc438017070 con 0x7fc44c072eb0 2024-05-28T06:04:18.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.346+0000 7fc442a00640 1 -- 172.21.15.96:0/1652150309 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc43800c9a0 con 0x7fc44c072eb0 2024-05-28T06:04:18.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.345+0000 7f5df2800640 1 Processor -- start 2024-05-28T06:04:18.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.346+0000 7f5df2800640 1 -- start start 2024-05-28T06:04:18.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.345+0000 7f5649400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f564c0d04a0 0x7f564c0ad130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:18.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.345+0000 7f5649400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f564c0d04a0 0x7f564c0ad130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.96:58628/0 (socket says 172.21.15.96:58628) 2024-05-28T06:04:18.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.345+0000 7f5649400640 1 -- 172.21.15.96:0/735651242 learned_addr learned my addr 172.21.15.96:0/735651242 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:18.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.346+0000 7f6489400640 1 Processor -- start 2024-05-28T06:04:18.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.346+0000 7f8f9a000640 1 -- 172.21.15.96:0/2583421462 shutdown_connections 2024-05-28T06:04:18.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.346+0000 7f8f9a000640 1 --2- 172.21.15.96:0/2583421462 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8f94073020 0x7f8f940ce370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:18.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.346+0000 7f8f9a000640 1 --2- 172.21.15.96:0/2583421462 >> v2:172.21.15.123:3300/0 conn(0x7f8f940d0d80 0x7f8f940cde30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:18.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.346+0000 7f8f9a000640 1 --2- 172.21.15.96:0/2583421462 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8f9415aa20 0x7f8f9415ae00 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:18.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.346+0000 7fc442a00640 1 -- 172.21.15.96:0/1652150309 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc438017070 con 0x7fc44c072eb0 2024-05-28T06:04:18.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.346+0000 7f8f9a000640 1 -- 172.21.15.96:0/2583421462 >> 172.21.15.96:0/2583421462 conn(0x7f8f940d0260 msgr2=0x7f8f940d0670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:18.373 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.346+0000 7fc450a00640 1 -- 172.21.15.96:0/1652150309 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc44c0ae280 con 0x7fc44c072eb0 2024-05-28T06:04:18.373 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.346+0000 7ff8b1800640 1 Processor -- start 2024-05-28T06:04:18.373 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.347+0000 7f8f9a000640 1 -- 172.21.15.96:0/2583421462 shutdown_connections 2024-05-28T06:04:18.373 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.347+0000 7f8f9a000640 1 -- 172.21.15.96:0/2583421462 wait complete. 2024-05-28T06:04:18.373 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.347+0000 7fc450a00640 1 -- 172.21.15.96:0/1652150309 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc44c0ae840 con 0x7fc44c072eb0 2024-05-28T06:04:18.374 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.347+0000 7f5df2800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5dec0d05b0 0x7f5dec1eaca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:18.374 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.347+0000 7ff8b1800640 1 -- start start 2024-05-28T06:04:18.374 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.347+0000 7f10e4a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f10d803a3f0 0x7f10d80df880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:18.375 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.347+0000 7f10e4a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f10d8050dc0 0x7f10d8185c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:18.375 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.347+0000 7f5df2800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5dec0d12f0 0x7f5dec1e9a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:18.375 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.347+0000 7f5df2800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5dec0d3600 0x7f5dec0b8de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:18.375 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.347+0000 7f5df2800640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f5dec0b3a00 con 0x7f5dec0d3600 2024-05-28T06:04:18.375 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.347+0000 7f5df2800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5dec0d02a0 con 0x7f5dec0d12f0 2024-05-28T06:04:18.375 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.347+0000 7f5df2800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f5dec0b3b80 con 0x7f5dec0d05b0 2024-05-28T06:04:18.376 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.346+0000 7f6489400640 1 -- start start 2024-05-28T06:04:18.376 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.346+0000 7f564a800640 1 --2- 172.21.15.96:0/735651242 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f564c162e70 0x7f564c0ad670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:18.376 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.347+0000 7f564a800640 1 -- 172.21.15.96:0/735651242 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f564c0d04a0 msgr2=0x7f564c0ad130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:18.376 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.347+0000 7f564a800640 1 --2- 172.21.15.96:0/735651242 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f564c0d04a0 0x7f564c0ad130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:18.376 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.347+0000 7f564a800640 1 -- 172.21.15.96:0/735651242 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f564c06f6e0 msgr2=0x7f564c155da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:18.376 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.347+0000 7f564a800640 1 --2- 172.21.15.96:0/735651242 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f564c06f6e0 0x7f564c155da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:18.376 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.347+0000 7f564a800640 1 -- 172.21.15.96:0/735651242 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f564c0c3d90 con 0x7f564c162e70 2024-05-28T06:04:18.377 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.348+0000 7f10e4a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f10d80e1080 0x7f10d804c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:18.377 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.348+0000 7f5debe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5dec0d3600 0x7f5dec0b8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:18.377 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.348+0000 7f6489400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f648406fcb0 0x7f6484145fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:18.377 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.348+0000 7f6489400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f648415a7f0 0x7f6484146510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:18.377 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.348+0000 7f0998600640 1 Processor -- start 2024-05-28T06:04:18.378 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.348+0000 7f6489400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f64841595f0 0x7f6484201290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:18.378 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.348+0000 7f6489400640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f64840ad8f0 con 0x7f64841595f0 2024-05-28T06:04:18.378 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.348+0000 7f6489400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f64840ad770 con 0x7f648406fcb0 2024-05-28T06:04:18.378 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.348+0000 7f6489400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f64840ada70 con 0x7f648415a7f0 2024-05-28T06:04:18.378 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.348+0000 7f5debe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5dec0d3600 0x7f5dec0b8de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:41768/0 (socket says 172.21.15.96:41768) 2024-05-28T06:04:18.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.348+0000 7f5debe00640 1 -- 172.21.15.96:0/1380600615 learned_addr learned my addr 172.21.15.96:0/1380600615 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:18.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.349+0000 7fc437e00640 1 -- 172.21.15.96:0/1652150309 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc420005290 con 0x7fc44c072eb0 2024-05-28T06:04:18.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.348+0000 7f5deb400640 1 --2- 172.21.15.96:0/1380600615 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5dec0d05b0 0x7f5dec1eaca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:18.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.349+0000 7f0998600640 1 -- start start 2024-05-28T06:04:18.380 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.349+0000 7f6483400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f648415a7f0 0x7f6484146510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:18.380 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.349+0000 7f6483e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f648406fcb0 0x7f6484145fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:18.380 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.349+0000 7f6483400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f648415a7f0 0x7f6484146510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.96:58522/0 (socket says 172.21.15.96:58522) 2024-05-28T06:04:18.380 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.349+0000 7f6483400640 1 -- 172.21.15.96:0/1802077102 learned_addr learned my addr 172.21.15.96:0/1802077102 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:18.381 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.349+0000 7f10de800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f10d803a3f0 0x7f10d80df880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:18.381 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.349+0000 7f10de800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f10d803a3f0 0x7f10d80df880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.96:58642/0 (socket says 172.21.15.96:58642) 2024-05-28T06:04:18.381 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.349+0000 7f10dde00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f10d8050dc0 0x7f10d8185c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:18.381 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.349+0000 7f10dde00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f10d8050dc0 0x7f10d8185c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:41758/0 (socket says 172.21.15.96:41758) 2024-05-28T06:04:18.381 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.349+0000 7f10e4a00640 1 -- 172.21.15.96:0/2835480415 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f10d803a840 con 0x7f10d8050dc0 2024-05-28T06:04:18.381 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.349+0000 7f10de800640 1 -- 172.21.15.96:0/2835480415 learned_addr learned my addr 172.21.15.96:0/2835480415 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:18.381 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.349+0000 7f10e4a00640 1 -- 172.21.15.96:0/2835480415 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f10d8050390 con 0x7f10d803a3f0 2024-05-28T06:04:18.381 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.349+0000 7f10e4a00640 1 -- 172.21.15.96:0/2835480415 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f10d80e4f60 con 0x7f10d80e1080 2024-05-28T06:04:18.381 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.349+0000 7f10df200640 1 --2- 172.21.15.96:0/2835480415 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f10d80e1080 0x7f10d804c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:18.382 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.349+0000 7f5debe00640 1 -- 172.21.15.96:0/1380600615 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5dec0d05b0 msgr2=0x7f5dec1eaca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:18.382 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.349+0000 7f5debe00640 1 --2- 172.21.15.96:0/1380600615 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5dec0d05b0 0x7f5dec1eaca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:18.382 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.349+0000 7f5debe00640 1 -- 172.21.15.96:0/1380600615 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5dec0d12f0 msgr2=0x7f5dec1e9a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:18.382 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.349+0000 7f5debe00640 1 --2- 172.21.15.96:0/1380600615 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5dec0d12f0 0x7f5dec1e9a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:18.382 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.349+0000 7f5debe00640 1 -- 172.21.15.96:0/1380600615 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dec0b7720 con 0x7f5dec0d3600 2024-05-28T06:04:18.382 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.350+0000 7f6488a00640 1 --2- 172.21.15.96:0/1802077102 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f64841595f0 0x7f6484201290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:18.382 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.350+0000 7ff8b1800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff8ac0cf490 0x7ff8ac15ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:18.383 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.350+0000 7f8f9a000640 1 Processor -- start 2024-05-28T06:04:18.383 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.350+0000 7f8f9a000640 1 -- start start 2024-05-28T06:04:18.383 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.350+0000 7ff8b1800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8ac0d1020 0x7ff8ac0cd520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:18.383 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.350+0000 7f0998600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f099015aab0 0x7f09900cde30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:18.384 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.350+0000 7f0998600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f099015ae90 0x7f09900ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:18.384 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.350+0000 7f6483e00640 1 -- 172.21.15.96:0/1802077102 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f648415a7f0 msgr2=0x7f6484146510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:18.384 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.350+0000 7f6483e00640 1 --2- 172.21.15.96:0/1802077102 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f648415a7f0 0x7f6484146510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:18.384 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.350+0000 7f6483e00640 1 -- 172.21.15.96:0/1802077102 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f64841595f0 msgr2=0x7f6484201290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:18.384 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.351+0000 7f6483e00640 1 --2- 172.21.15.96:0/1802077102 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f64841595f0 0x7f6484201290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:18.384 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.350+0000 7f5debe00640 1 --2- 172.21.15.96:0/1380600615 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5dec0d3600 0x7f5dec0b8de0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f5de4009dd0 tx=0x7f5de400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:18.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.351+0000 7ff8b1800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff8ac163870 0x7ff8ac0cda60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:18.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.351+0000 7ff8aaa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff8ac0cf490 0x7ff8ac15ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:18.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.351+0000 7f0998600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09900ae520 con 0x7f09900d0d80 2024-05-28T06:04:18.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.351+0000 7f10df200640 1 -- 172.21.15.96:0/2835480415 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f10d803a3f0 msgr2=0x7f10d80df880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:18.386 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.351+0000 7f10df200640 1 --2- 172.21.15.96:0/2835480415 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f10d803a3f0 0x7f10d80df880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:18.386 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.351+0000 7f10df200640 1 -- 172.21.15.96:0/2835480415 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f10d8050dc0 msgr2=0x7f10d8185c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:18.386 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.351+0000 7f10df200640 1 --2- 172.21.15.96:0/2835480415 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f10d8050dc0 0x7f10d8185c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:18.386 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.351+0000 7f10df200640 1 -- 172.21.15.96:0/2835480415 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10d80e03a0 con 0x7f10d80e1080 2024-05-28T06:04:18.386 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.351+0000 7f0998600640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f09900ae800 con 0x7f099015ae90 2024-05-28T06:04:18.386 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.351+0000 7f0998600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f09900aeae0 con 0x7f099015aab0 2024-05-28T06:04:18.386 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.351+0000 7f0995200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f099015ae90 0x7f09900ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:18.386 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.351+0000 7f0995200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f099015ae90 0x7f09900ce370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:41796/0 (socket says 172.21.15.96:41796) 2024-05-28T06:04:18.386 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.351+0000 7f0995200640 1 -- 172.21.15.96:0/1918279462 learned_addr learned my addr 172.21.15.96:0/1918279462 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:18.387 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.351+0000 7f564a800640 1 --2- 172.21.15.96:0/735651242 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f564c162e70 0x7f564c0ad670 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f564400bdf0 tx=0x7f5644009ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:18.387 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.351+0000 7f6483e00640 1 -- 172.21.15.96:0/1802077102 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6484099e10 con 0x7f648406fcb0 2024-05-28T06:04:18.387 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.351+0000 7f6483400640 1 --2- 172.21.15.96:0/1802077102 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f648415a7f0 0x7f6484146510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-28T06:04:18.387 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.351+0000 7f6488a00640 1 --2- 172.21.15.96:0/1802077102 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f64841595f0 0x7f6484201290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-28T06:04:18.387 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.351+0000 7ff8b1800640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff8ac164240 con 0x7ff8ac0cf490 2024-05-28T06:04:18.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.351+0000 7f0995c00640 1 --2- 172.21.15.96:0/1918279462 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f099015aab0 0x7f09900cde30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:18.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.351+0000 7f5de3400640 1 -- 172.21.15.96:0/1380600615 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5de4018020 con 0x7f5dec0d3600 2024-05-28T06:04:18.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.351+0000 7f5de3400640 1 -- 172.21.15.96:0/1380600615 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5de4014480 con 0x7f5dec0d3600 2024-05-28T06:04:18.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.351+0000 7f5de3400640 1 -- 172.21.15.96:0/1380600615 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5de4014870 con 0x7f5dec0d3600 2024-05-28T06:04:18.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.352+0000 7ff8b1800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff8ac0d38e0 con 0x7ff8ac0d1020 2024-05-28T06:04:18.389 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.351+0000 7f10df200640 1 --2- 172.21.15.96:0/2835480415 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f10d80e1080 0x7f10d804c560 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f10cc031000 tx=0x7f10cc0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:18.389 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.351+0000 7ff8aaa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff8ac0cf490 0x7ff8ac15ba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:41788/0 (socket says 172.21.15.96:41788) 2024-05-28T06:04:18.389 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.352+0000 7ff8b1800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ff8ac1643c0 con 0x7ff8ac163870 2024-05-28T06:04:18.389 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.352+0000 7ff8aaa00640 1 -- 172.21.15.96:0/1668559030 learned_addr learned my addr 172.21.15.96:0/1668559030 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:18.389 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.352+0000 7ff8ab400640 1 --2- 172.21.15.96:0/1668559030 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff8ac163870 0x7ff8ac0cda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:18.390 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.352+0000 7f5df2800640 1 -- 172.21.15.96:0/1380600615 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5dec0b7a00 con 0x7f5dec0d3600 2024-05-28T06:04:18.390 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.352+0000 7f5df2800640 1 -- 172.21.15.96:0/1380600615 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5dec0b7fc0 con 0x7f5dec0d3600 2024-05-28T06:04:18.390 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.352+0000 7f6483e00640 1 --2- 172.21.15.96:0/1802077102 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f648406fcb0 0x7f6484145fd0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f647400c610 tx=0x7f647400cad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:18.390 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.352+0000 7f5642000640 1 -- 172.21.15.96:0/735651242 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5644019070 con 0x7f564c162e70 2024-05-28T06:04:18.391 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.352+0000 7f5642000640 1 -- 172.21.15.96:0/735651242 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5644014440 con 0x7f564c162e70 2024-05-28T06:04:18.391 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.352+0000 7f5642000640 1 -- 172.21.15.96:0/735651242 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5644014830 con 0x7f564c162e70 2024-05-28T06:04:18.391 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.352+0000 7f8f9a000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8f94073020 0x7f8f940c8870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:18.391 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.352+0000 7f8f9a000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8f940d0d80 0x7f8f940c75a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:18.391 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.352+0000 7f8f9a000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f9415aa20 0x7f8f9415ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:18.391 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.352+0000 7f5650c00640 1 -- 172.21.15.96:0/735651242 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f564c0c4070 con 0x7f564c162e70 2024-05-28T06:04:18.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.352+0000 7f8f9a000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f8f940cec40 con 0x7f8f94073020 2024-05-28T06:04:18.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.352+0000 7f8f9a000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f8f940ceac0 con 0x7f8f9415aa20 2024-05-28T06:04:18.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.352+0000 7f8f9a000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f8f940ad130 con 0x7f8f940d0d80 2024-05-28T06:04:18.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.352+0000 7f0995200640 1 -- 172.21.15.96:0/1918279462 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f099015aab0 msgr2=0x7f09900cde30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:18.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.352+0000 7f0995200640 1 --2- 172.21.15.96:0/1918279462 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f099015aab0 0x7f09900cde30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:18.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.352+0000 7f0995200640 1 -- 172.21.15.96:0/1918279462 >> v1:172.21.15.123:6789/0 conn(0x7f09900d0d80 legacy=0x7f09900ad9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:18.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.352+0000 7f0995200640 1 -- 172.21.15.96:0/1918279462 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09900aedc0 con 0x7f099015ae90 2024-05-28T06:04:18.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.353+0000 7f5650c00640 1 -- 172.21.15.96:0/735651242 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f564c0ae680 con 0x7f564c162e70 2024-05-28T06:04:18.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.353+0000 7f8f93400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8f94073020 0x7f8f940c8870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:18.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.353+0000 7f8f93400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8f94073020 0x7f8f940c8870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:41798/0 (socket says 172.21.15.96:41798) 2024-05-28T06:04:18.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.353+0000 7f8f93400640 1 -- 172.21.15.96:0/1821957869 learned_addr learned my addr 172.21.15.96:0/1821957869 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:18.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.353+0000 7f8f92a00640 1 --2- 172.21.15.96:0/1821957869 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8f940d0d80 0x7f8f940c75a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:18.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.353+0000 7f8f93e00640 1 --2- 172.21.15.96:0/1821957869 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f9415aa20 0x7f8f9415ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:18.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.354+0000 7f8f93400640 1 -- 172.21.15.96:0/1821957869 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8f940d0d80 msgr2=0x7f8f940c75a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:18.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.354+0000 7f8f93400640 1 --2- 172.21.15.96:0/1821957869 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8f940d0d80 0x7f8f940c75a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:18.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.354+0000 7f8f93400640 1 -- 172.21.15.96:0/1821957869 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f9415aa20 msgr2=0x7f8f9415ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:18.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.353+0000 7f10d6a00640 1 -- 172.21.15.96:0/2835480415 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10cc041020 con 0x7f10d80e1080 2024-05-28T06:04:18.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.353+0000 7f10d6a00640 1 -- 172.21.15.96:0/2835480415 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10cc0306c0 con 0x7f10d80e1080 2024-05-28T06:04:18.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.353+0000 7f10d6a00640 1 -- 172.21.15.96:0/2835480415 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10cc037040 con 0x7f10d80e1080 2024-05-28T06:04:18.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.352+0000 7fc442a00640 1 -- 172.21.15.96:0/1652150309 <== mon.1 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc43801a070 con 0x7fc44c072eb0 2024-05-28T06:04:18.395 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.353+0000 7fc442a00640 1 --2- 172.21.15.96:0/1652150309 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc42c06caf0 0x7fc42c06efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:18.395 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.354+0000 7fc442a00640 1 -- 172.21.15.96:0/1652150309 <== mon.1 v2:172.21.15.123:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fc43808bb00 con 0x7fc44c072eb0 2024-05-28T06:04:18.395 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.354+0000 7ff8ab400640 1 -- 172.21.15.96:0/1668559030 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8ac0d1020 msgr2=0x7ff8ac0cd520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:18.395 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.354+0000 7ff8ab400640 1 --2- 172.21.15.96:0/1668559030 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8ac0d1020 0x7ff8ac0cd520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:18.395 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.354+0000 7f8f93400640 1 --2- 172.21.15.96:0/1821957869 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f9415aa20 0x7f8f9415ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:18.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.354+0000 7ff8ab400640 1 -- 172.21.15.96:0/1668559030 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff8ac0cf490 msgr2=0x7ff8ac15ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:18.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.354+0000 7ff8ab400640 1 --2- 172.21.15.96:0/1668559030 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff8ac0cf490 0x7ff8ac15ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:18.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.354+0000 7f0995c00640 1 --2- 172.21.15.96:0/1918279462 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f099015aab0 0x7f09900cde30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-28T06:04:18.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.354+0000 7ff8ab400640 1 -- 172.21.15.96:0/1668559030 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8ac15bf70 con 0x7ff8ac163870 2024-05-28T06:04:18.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.354+0000 7f5df2800640 1 -- 172.21.15.96:0/1380600615 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5dec0b81c0 con 0x7f5dec0d3600 2024-05-28T06:04:18.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.354+0000 7ff8aaa00640 1 --2- 172.21.15.96:0/1668559030 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff8ac0cf490 0x7ff8ac15ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-28T06:04:18.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.354+0000 7f8f93400640 1 -- 172.21.15.96:0/1821957869 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f94098320 con 0x7f8f94073020 2024-05-28T06:04:18.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.354+0000 7f10e4a00640 1 -- 172.21.15.96:0/2835480415 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10d804d170 con 0x7f10d80e1080 2024-05-28T06:04:18.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.354+0000 7f10e4a00640 1 -- 172.21.15.96:0/2835480415 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f10d804d730 con 0x7f10d80e1080 2024-05-28T06:04:18.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.354+0000 7f8f93e00640 1 --2- 172.21.15.96:0/1821957869 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f9415aa20 0x7f8f9415ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:04:18.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.354+0000 7f5650c00640 1 -- 172.21.15.96:0/735651242 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f564c0689e0 con 0x7f564c162e70 2024-05-28T06:04:18.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.355+0000 7fc44a800640 1 --2- 172.21.15.96:0/1652150309 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc42c06caf0 0x7fc42c06efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:18.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.355+0000 7f8f93400640 1 --2- 172.21.15.96:0/1821957869 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8f94073020 0x7f8f940c8870 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f8f8402abc0 tx=0x7f8f84004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:18.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.355+0000 7ff8ab400640 1 --2- 172.21.15.96:0/1668559030 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff8ac163870 0x7ff8ac0cda60 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff8a4007e60 tx=0x7ff8a400d4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:18.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.355+0000 7f0995200640 1 --2- 172.21.15.96:0/1918279462 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f099015ae90 0x7f09900ce370 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f0988009e50 tx=0x7f098802c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=402d4456370f0b49 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:18.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.355+0000 7f8f8b400640 1 -- 172.21.15.96:0/1821957869 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f8403f070 con 0x7f8f94073020 2024-05-28T06:04:18.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.356+0000 7f8f8b400640 1 -- 172.21.15.96:0/1821957869 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f84030b60 con 0x7f8f94073020 2024-05-28T06:04:18.400 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.356+0000 7f8f8b400640 1 -- 172.21.15.96:0/1821957869 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f84043410 con 0x7f8f94073020 2024-05-28T06:04:18.400 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.356+0000 7f8f9a000640 1 -- 172.21.15.96:0/1821957869 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f940ca300 con 0x7f8f94073020 2024-05-28T06:04:18.400 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.356+0000 7ff8a2a00640 1 -- 172.21.15.96:0/1668559030 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8a401a070 con 0x7ff8ac163870 2024-05-28T06:04:18.400 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.356+0000 7ff8b1800640 1 -- 172.21.15.96:0/1668559030 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8ac0b7e30 con 0x7ff8ac163870 2024-05-28T06:04:18.400 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.356+0000 7f8f9a000640 1 -- 172.21.15.96:0/1821957869 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f940985e0 con 0x7f8f94073020 2024-05-28T06:04:18.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.356+0000 7ff8b1800640 1 -- 172.21.15.96:0/1668559030 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8ac0b83f0 con 0x7ff8ac163870 2024-05-28T06:04:18.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.356+0000 7f10e4a00640 1 -- 172.21.15.96:0/2835480415 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10d8005380 con 0x7f10d80e1080 2024-05-28T06:04:18.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.356+0000 7ff8a2a00640 1 -- 172.21.15.96:0/1668559030 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8a4003b90 con 0x7ff8ac163870 2024-05-28T06:04:18.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.356+0000 7ff8a2a00640 1 -- 172.21.15.96:0/1668559030 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8a401a070 con 0x7ff8ac163870 2024-05-28T06:04:18.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.357+0000 7f8f9a000640 1 -- 172.21.15.96:0/1821957869 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f94072790 con 0x7f8f94073020 2024-05-28T06:04:18.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.357+0000 7f5de3400640 1 -- 172.21.15.96:0/1380600615 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5de4005d10 con 0x7f5dec0d3600 2024-05-28T06:04:18.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.357+0000 7f098fe00640 1 -- 172.21.15.96:0/1918279462 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f098802e040 con 0x7f099015ae90 2024-05-28T06:04:18.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.357+0000 7f098fe00640 1 -- 172.21.15.96:0/1918279462 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0988037070 con 0x7f099015ae90 2024-05-28T06:04:18.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.357+0000 7f098fe00640 1 -- 172.21.15.96:0/1918279462 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0988032660 con 0x7f099015ae90 2024-05-28T06:04:18.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.358+0000 7f5642000640 1 -- 172.21.15.96:0/735651242 <== mon.2 v2:172.21.15.123:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f564401c070 con 0x7f564c162e70 2024-05-28T06:04:18.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.358+0000 7ff8b1800640 1 -- 172.21.15.96:0/1668559030 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff884005290 con 0x7ff8ac163870 2024-05-28T06:04:18.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.358+0000 7fc44a800640 1 --2- 172.21.15.96:0/1652150309 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc42c06caf0 0x7fc42c06efb0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc43c000f60 tx=0x7fc43c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:18.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.358+0000 7f0998600640 1 -- 172.21.15.96:0/1918279462 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f099015ae90 msgr2=0x7f09900ce370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:18.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.358+0000 7f0998600640 1 --2- 172.21.15.96:0/1918279462 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f099015ae90 0x7f09900ce370 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f0988009e50 tx=0x7f098802c950 comp rx=0 tx=0).stop 2024-05-28T06:04:18.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.359+0000 7f0998600640 1 -- 172.21.15.96:0/1918279462 shutdown_connections 2024-05-28T06:04:18.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.359+0000 7f0998600640 1 --2- 172.21.15.96:0/1918279462 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f099015ae90 0x7f09900ce370 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:18.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.359+0000 7f0998600640 1 --2- 172.21.15.96:0/1918279462 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f099015aab0 0x7f09900cde30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:18.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.359+0000 7f0998600640 1 -- 172.21.15.96:0/1918279462 >> 172.21.15.96:0/1918279462 conn(0x7f09900d0260 msgr2=0x7f09900d0670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:18.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.359+0000 7f5642000640 1 --2- 172.21.15.96:0/735651242 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f562c06c9c0 0x7f562c06ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:18.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.359+0000 7f0998600640 1 -- 172.21.15.96:0/1918279462 shutdown_connections 2024-05-28T06:04:18.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.359+0000 7f5642000640 1 -- 172.21.15.96:0/735651242 <== mon.2 v2:172.21.15.123:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f564408c1f0 con 0x7f564c162e70 2024-05-28T06:04:18.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.359+0000 7f5de3400640 1 --2- 172.21.15.96:0/1380600615 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5dc806c9c0 0x7f5dc806ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:18.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.359+0000 7f5de3400640 1 -- 172.21.15.96:0/1380600615 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f5de408c6a0 con 0x7f5dec0d3600 2024-05-28T06:04:18.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.359+0000 7f0998600640 1 -- 172.21.15.96:0/1918279462 wait complete. 2024-05-28T06:04:18.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.359+0000 7f10d6a00640 1 -- 172.21.15.96:0/2835480415 <== mon.2 v2:172.21.15.123:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f10cc044020 con 0x7f10d80e1080 2024-05-28T06:04:18.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.360+0000 7f8f8b400640 1 -- 172.21.15.96:0/1821957869 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8f840435b0 con 0x7f8f94073020 2024-05-28T06:04:18.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.360+0000 7f5deb400640 1 --2- 172.21.15.96:0/1380600615 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5dc806c9c0 0x7f5dc806ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:18.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.360+0000 7f10d6a00640 1 --2- 172.21.15.96:0/2835480415 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f10c006c940 0x7f10c006ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:18.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.360+0000 7f5649e00640 1 --2- 172.21.15.96:0/735651242 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f562c06c9c0 0x7f562c06ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:18.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.360+0000 7f10d6a00640 1 -- 172.21.15.96:0/2835480415 <== mon.2 v2:172.21.15.123:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f10cc0ac750 con 0x7f10d80e1080 2024-05-28T06:04:18.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.361+0000 7f10de800640 1 --2- 172.21.15.96:0/2835480415 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f10c006c940 0x7f10c006ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:18.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.361+0000 7f0998600640 1 Processor -- start 2024-05-28T06:04:18.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.361+0000 7f5deb400640 1 --2- 172.21.15.96:0/1380600615 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5dc806c9c0 0x7f5dc806ee80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5ddc034000 tx=0x7f5ddc02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:18.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.361+0000 7f0998600640 1 -- start start 2024-05-28T06:04:18.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.361+0000 7f8f8b400640 1 --2- 172.21.15.96:0/1821957869 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8f7406c940 0x7f8f7406ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:18.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.361+0000 7f5649e00640 1 --2- 172.21.15.96:0/735651242 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f562c06c9c0 0x7f562c06ee80 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f563c045000 tx=0x7f563c03f000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:18.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.362+0000 7f8f8b400640 1 -- 172.21.15.96:0/1821957869 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f8f840acfb0 con 0x7f8f94073020 2024-05-28T06:04:18.410 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.362+0000 7f8f92a00640 1 --2- 172.21.15.96:0/1821957869 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8f7406c940 0x7f8f7406ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:18.410 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.362+0000 7f10de800640 1 --2- 172.21.15.96:0/2835480415 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f10c006c940 0x7f10c006ee00 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f10d8179010 tx=0x7f10d0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:18.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.362+0000 7f0998600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f09900d0d80 0x7f09900b18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:18.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.362+0000 7f0998600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f099015aab0 0x7f09900c3d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:18.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.362+0000 7f0998600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f099015ae90 0x7f09900c42b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:18.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.362+0000 7f0998600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f09900cbe90 con 0x7f09900d0d80 2024-05-28T06:04:18.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.362+0000 7f0998600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f09900cbd10 con 0x7f099015aab0 2024-05-28T06:04:18.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.362+0000 7f0998600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f09900cc010 con 0x7f099015ae90 2024-05-28T06:04:18.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.362+0000 7f0995c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f09900d0d80 0x7f09900b18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:18.412 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.363+0000 7f0995c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f09900d0d80 0x7f09900b18a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:41800/0 (socket says 172.21.15.96:41800) 2024-05-28T06:04:18.412 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.363+0000 7f0995c00640 1 -- 172.21.15.96:0/2634038741 learned_addr learned my addr 172.21.15.96:0/2634038741 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:18.412 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.363+0000 7f0995200640 1 --2- 172.21.15.96:0/2634038741 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f099015aab0 0x7f09900c3d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:18.412 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.363+0000 7f0996600640 1 --2- 172.21.15.96:0/2634038741 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f099015ae90 0x7f09900c42b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:18.412 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.363+0000 7f0995c00640 1 -- 172.21.15.96:0/2634038741 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f099015ae90 msgr2=0x7f09900c42b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:18.412 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.363+0000 7f0995c00640 1 --2- 172.21.15.96:0/2634038741 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f099015ae90 0x7f09900c42b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:18.412 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.363+0000 7f0995c00640 1 -- 172.21.15.96:0/2634038741 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f099015aab0 msgr2=0x7f09900c3d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:18.412 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.363+0000 7f0995c00640 1 --2- 172.21.15.96:0/2634038741 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f099015aab0 0x7f09900c3d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:18.412 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.363+0000 7f0995c00640 1 -- 172.21.15.96:0/2634038741 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09901f8b80 con 0x7f09900d0d80 2024-05-28T06:04:18.412 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.363+0000 7f8f92a00640 1 --2- 172.21.15.96:0/1821957869 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8f7406c940 0x7f8f7406ee00 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8f9416bb30 tx=0x7f8f80009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:18.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.363+0000 7f0995c00640 1 --2- 172.21.15.96:0/2634038741 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f09900d0d80 0x7f09900b18a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f097c00e7e0 tx=0x7f097c00eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:18.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.364+0000 7f5de3400640 1 -- 172.21.15.96:0/1380600615 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5dec0b81c0 con 0x7f5dec0d3600 2024-05-28T06:04:18.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.369+0000 7f8f8b400640 1 -- 172.21.15.96:0/1821957869 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8f840dbb80 con 0x7f8f94073020 2024-05-28T06:04:18.414 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.369+0000 7f098d600640 1 -- 172.21.15.96:0/2634038741 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f097c00f040 con 0x7f09900d0d80 2024-05-28T06:04:18.414 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.369+0000 7f098d600640 1 -- 172.21.15.96:0/2634038741 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f097c004030 con 0x7f09900d0d80 2024-05-28T06:04:18.414 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.369+0000 7f098d600640 1 -- 172.21.15.96:0/2634038741 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f097c004400 con 0x7f09900d0d80 2024-05-28T06:04:18.414 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.369+0000 7f0998600640 1 -- 172.21.15.96:0/2634038741 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09901f8e60 con 0x7f09900d0d80 2024-05-28T06:04:18.414 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.369+0000 7f0998600640 1 -- 172.21.15.96:0/2634038741 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09901f9420 con 0x7f09900d0d80 2024-05-28T06:04:18.414 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.370+0000 7f0998600640 1 -- 172.21.15.96:0/2634038741 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09900689e0 con 0x7f09900d0d80 2024-05-28T06:04:18.415 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.371+0000 7f647be00640 1 -- 172.21.15.96:0/1802077102 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6474015070 con 0x7f648406fcb0 2024-05-28T06:04:18.415 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.371+0000 7f6489400640 1 -- 172.21.15.96:0/1802077102 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6484144c60 con 0x7f648406fcb0 2024-05-28T06:04:18.415 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.371+0000 7f647be00640 1 -- 172.21.15.96:0/1802077102 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6474004220 con 0x7f648406fcb0 2024-05-28T06:04:18.415 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.371+0000 7f6489400640 1 -- 172.21.15.96:0/1802077102 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64840b05e0 con 0x7f648406fcb0 2024-05-28T06:04:18.415 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.372+0000 7f647be00640 1 -- 172.21.15.96:0/1802077102 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6474015070 con 0x7f648406fcb0 2024-05-28T06:04:18.415 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.372+0000 7f098d600640 1 -- 172.21.15.96:0/2634038741 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f097c00c6d0 con 0x7f09900d0d80 2024-05-28T06:04:18.416 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.372+0000 7fc442a00640 1 -- 172.21.15.96:0/1652150309 <== mon.1 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc4380598c0 con 0x7fc44c072eb0 2024-05-28T06:04:18.416 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.374+0000 7f6489400640 1 -- 172.21.15.96:0/1802077102 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64841478d0 con 0x7f648406fcb0 2024-05-28T06:04:18.416 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.374+0000 7f098d600640 1 --2- 172.21.15.96:0/2634038741 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f097006c9f0 0x7f097006eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:18.416 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.374+0000 7f098d600640 1 -- 172.21.15.96:0/2634038741 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f097c08b8a0 con 0x7f09900d0d80 2024-05-28T06:04:18.417 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.375+0000 7f5642000640 1 -- 172.21.15.96:0/735651242 <== mon.2 v2:172.21.15.123:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f564405a040 con 0x7f564c162e70 2024-05-28T06:04:18.417 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.375+0000 7f0995200640 1 --2- 172.21.15.96:0/2634038741 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f097006c9f0 0x7f097006eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:18.417 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.376+0000 7f647be00640 1 -- 172.21.15.96:0/1802077102 <== mon.1 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6474003610 con 0x7f648406fcb0 2024-05-28T06:04:18.417 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.377+0000 7f0995200640 1 --2- 172.21.15.96:0/2634038741 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f097006c9f0 0x7f097006eeb0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f0988036000 tx=0x7f0988009000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:18.418 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.377+0000 7f647be00640 1 --2- 172.21.15.96:0/1802077102 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f646406caf0 0x7f646406efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:18.418 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.377+0000 7f647be00640 1 -- 172.21.15.96:0/1802077102 <== mon.1 v2:172.21.15.123:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f647408b4a0 con 0x7f648406fcb0 2024-05-28T06:04:18.418 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.379+0000 7f6483400640 1 --2- 172.21.15.96:0/1802077102 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f646406caf0 0x7f646406efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:18.418 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.379+0000 7ff8a2a00640 1 -- 172.21.15.96:0/1668559030 <== mon.2 v2:172.21.15.123:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff8a4005d10 con 0x7ff8ac163870 2024-05-28T06:04:18.419 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.380+0000 7f6483400640 1 --2- 172.21.15.96:0/1802077102 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f646406caf0 0x7f646406efb0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f64840b0e30 tx=0x7f647002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:18.419 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.380+0000 7ff8a2a00640 1 --2- 172.21.15.96:0/1668559030 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff88c06cb20 0x7ff88c06efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:18.419 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.381+0000 7ff8aaa00640 1 --2- 172.21.15.96:0/1668559030 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff88c06cb20 0x7ff88c06efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:18.419 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.381+0000 7f098d600640 1 -- 172.21.15.96:0/2634038741 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f097c017020 con 0x7f09900d0d80 2024-05-28T06:04:18.420 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.382+0000 7ff8aaa00640 1 --2- 172.21.15.96:0/1668559030 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff88c06cb20 0x7ff88c06efe0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ff89c005880 tx=0x7ff89c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:18.420 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.390+0000 7ff8a2a00640 1 -- 172.21.15.96:0/1668559030 <== mon.2 v2:172.21.15.123:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7ff8a4057c10 con 0x7ff8ac163870 2024-05-28T06:04:18.420 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.390+0000 7f10d6a00640 1 -- 172.21.15.96:0/2835480415 <== mon.2 v2:172.21.15.123:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f10cc04b030 con 0x7f10d80e1080 2024-05-28T06:04:18.420 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.391+0000 7f647be00640 1 -- 172.21.15.96:0/1802077102 <== mon.1 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6474059200 con 0x7f648406fcb0 2024-05-28T06:04:18.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.403+0000 7ff8a2a00640 1 -- 172.21.15.96:0/1668559030 <== mon.2 v2:172.21.15.123:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff8a4010070 con 0x7ff8ac163870 2024-05-28T06:04:18.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.781+0000 7fc437e00640 1 -- 172.21.15.96:0/1652150309 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fc420005850 con 0x7fc44c072eb0 2024-05-28T06:04:18.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.782+0000 7fc442a00640 1 -- 172.21.15.96:0/1652150309 <== mon.1 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fc43805dbc0 con 0x7fc44c072eb0 2024-05-28T06:04:18.785 INFO:teuthology.orchestra.run.smithi096.stdout:55834574854 2024-05-28T06:04:18.787 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.786+0000 7fc437e00640 1 -- 172.21.15.96:0/1652150309 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc42c06caf0 msgr2=0x7fc42c06efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:18.787 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.786+0000 7fc437e00640 1 --2- 172.21.15.96:0/1652150309 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc42c06caf0 0x7fc42c06efb0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc43c000f60 tx=0x7fc43c009040 comp rx=0 tx=0).stop 2024-05-28T06:04:18.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.786+0000 7fc437e00640 1 -- 172.21.15.96:0/1652150309 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc44c072eb0 msgr2=0x7fc44c1584c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:18.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.786+0000 7fc437e00640 1 --2- 172.21.15.96:0/1652150309 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc44c072eb0 0x7fc44c1584c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc43800e9b0 tx=0x7fc43800ee70 comp rx=0 tx=0).stop 2024-05-28T06:04:18.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.786+0000 7fc437e00640 1 -- 172.21.15.96:0/1652150309 shutdown_connections 2024-05-28T06:04:18.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.786+0000 7fc437e00640 1 --2- 172.21.15.96:0/1652150309 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc42c06caf0 0x7fc42c06efb0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:18.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.786+0000 7fc437e00640 1 --2- 172.21.15.96:0/1652150309 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc44c15a490 0x7fc44c148390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:18.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.786+0000 7fc437e00640 1 --2- 172.21.15.96:0/1652150309 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc44c072eb0 0x7fc44c1584c0 secure :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc43800e9b0 tx=0x7fc43800ee70 comp rx=0 tx=0).stop 2024-05-28T06:04:18.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.786+0000 7fc437e00640 1 --2- 172.21.15.96:0/1652150309 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc44c072000 0x7fc44c15b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:18.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.786+0000 7fc437e00640 1 -- 172.21.15.96:0/1652150309 >> 172.21.15.96:0/1652150309 conn(0x7fc44c06c380 msgr2=0x7fc44c09c1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:18.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.787+0000 7fc437e00640 1 -- 172.21.15.96:0/1652150309 shutdown_connections 2024-05-28T06:04:18.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.787+0000 7fc437e00640 1 -- 172.21.15.96:0/1652150309 wait complete. 2024-05-28T06:04:18.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.792+0000 7f8f9a000640 1 -- 172.21.15.96:0/1821957869 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f8f940ca820 con 0x7f8f94073020 2024-05-28T06:04:18.793 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.793+0000 7f8f8b400640 1 -- 172.21.15.96:0/1821957869 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f8f94072790 con 0x7f8f94073020 2024-05-28T06:04:18.793 INFO:teuthology.orchestra.run.smithi096.stdout:55834574854 2024-05-28T06:04:18.797 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.796+0000 7f5df2800640 1 -- 172.21.15.96:0/1380600615 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f5dec0cb500 con 0x7f5dec0d3600 2024-05-28T06:04:18.797 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.797+0000 7f8f88c00640 1 -- 172.21.15.96:0/1821957869 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8f7406c940 msgr2=0x7f8f7406ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:18.798 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.797+0000 7f8f88c00640 1 --2- 172.21.15.96:0/1821957869 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8f7406c940 0x7f8f7406ee00 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8f9416bb30 tx=0x7f8f80009190 comp rx=0 tx=0).stop 2024-05-28T06:04:18.798 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.797+0000 7f8f88c00640 1 -- 172.21.15.96:0/1821957869 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8f94073020 msgr2=0x7f8f940c8870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:18.798 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.797+0000 7f8f88c00640 1 --2- 172.21.15.96:0/1821957869 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8f94073020 0x7f8f940c8870 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f8f8402abc0 tx=0x7f8f84004a40 comp rx=0 tx=0).stop 2024-05-28T06:04:18.798 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.797+0000 7f8f88c00640 1 -- 172.21.15.96:0/1821957869 shutdown_connections 2024-05-28T06:04:18.798 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.797+0000 7f5de3400640 1 -- 172.21.15.96:0/1380600615 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f5de4022070 con 0x7f5dec0d3600 2024-05-28T06:04:18.798 INFO:teuthology.orchestra.run.smithi096.stdout:55834574855 2024-05-28T06:04:18.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.797+0000 7f8f88c00640 1 --2- 172.21.15.96:0/1821957869 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8f7406c940 0x7f8f7406ee00 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:18.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.797+0000 7f8f88c00640 1 --2- 172.21.15.96:0/1821957869 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f9415aa20 0x7f8f9415ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:18.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.797+0000 7f8f88c00640 1 --2- 172.21.15.96:0/1821957869 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8f940d0d80 0x7f8f940c75a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:18.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.797+0000 7f8f88c00640 1 --2- 172.21.15.96:0/1821957869 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8f94073020 0x7f8f940c8870 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:18.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.797+0000 7f8f88c00640 1 -- 172.21.15.96:0/1821957869 >> 172.21.15.96:0/1821957869 conn(0x7f8f940d0260 msgr2=0x7f8f940b0e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:18.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.801+0000 7f8f88c00640 1 -- 172.21.15.96:0/1821957869 shutdown_connections 2024-05-28T06:04:18.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.801+0000 7f8f88c00640 1 -- 172.21.15.96:0/1821957869 wait complete. 2024-05-28T06:04:18.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.801+0000 7f5df2800640 1 -- 172.21.15.96:0/1380600615 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5dc806c9c0 msgr2=0x7f5dc806ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:18.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.801+0000 7f5df2800640 1 --2- 172.21.15.96:0/1380600615 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5dc806c9c0 0x7f5dc806ee80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5ddc034000 tx=0x7f5ddc02e000 comp rx=0 tx=0).stop 2024-05-28T06:04:18.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.801+0000 7f5df2800640 1 -- 172.21.15.96:0/1380600615 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5dec0d3600 msgr2=0x7f5dec0b8de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:18.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.801+0000 7f5df2800640 1 --2- 172.21.15.96:0/1380600615 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5dec0d3600 0x7f5dec0b8de0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f5de4009dd0 tx=0x7f5de400c490 comp rx=0 tx=0).stop 2024-05-28T06:04:18.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.802+0000 7f5df2800640 1 -- 172.21.15.96:0/1380600615 shutdown_connections 2024-05-28T06:04:18.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.802+0000 7f5df2800640 1 --2- 172.21.15.96:0/1380600615 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5dc806c9c0 0x7f5dc806ee80 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:18.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.802+0000 7f5df2800640 1 --2- 172.21.15.96:0/1380600615 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5dec0d3600 0x7f5dec0b8de0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:18.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.802+0000 7f5df2800640 1 --2- 172.21.15.96:0/1380600615 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5dec0d12f0 0x7f5dec1e9a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:18.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.802+0000 7f5df2800640 1 --2- 172.21.15.96:0/1380600615 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5dec0d05b0 0x7f5dec1eaca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:18.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.802+0000 7f5df2800640 1 -- 172.21.15.96:0/1380600615 >> 172.21.15.96:0/1380600615 conn(0x7f5dec08bfc0 msgr2=0x7f5dec06ddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:18.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.803+0000 7f5df2800640 1 -- 172.21.15.96:0/1380600615 shutdown_connections 2024-05-28T06:04:18.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.803+0000 7f5df2800640 1 -- 172.21.15.96:0/1380600615 wait complete. 2024-05-28T06:04:18.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.804+0000 7f0998600640 1 -- 172.21.15.96:0/2634038741 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f099015a070 con 0x7f09900d0d80 2024-05-28T06:04:18.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.805+0000 7f098d600640 1 -- 172.21.15.96:0/2634038741 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f097c059d40 con 0x7f09900d0d80 2024-05-28T06:04:18.808 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.807+0000 7f5650c00640 1 -- 172.21.15.96:0/735651242 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f564c15ce40 con 0x7f564c162e70 2024-05-28T06:04:18.808 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.808+0000 7f5642000640 1 -- 172.21.15.96:0/735651242 <== mon.2 v2:172.21.15.123:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f564405e340 con 0x7f564c162e70 2024-05-28T06:04:18.808 INFO:teuthology.orchestra.run.smithi096.stdout:55834574853 2024-05-28T06:04:18.811 INFO:teuthology.orchestra.run.smithi096.stdout:55834574855 2024-05-28T06:04:18.812 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.808+0000 7f0998600640 1 -- 172.21.15.96:0/2634038741 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f097006c9f0 msgr2=0x7f097006eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:18.812 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.808+0000 7f0998600640 1 --2- 172.21.15.96:0/2634038741 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f097006c9f0 0x7f097006eeb0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f0988036000 tx=0x7f0988009000 comp rx=0 tx=0).stop 2024-05-28T06:04:18.812 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.808+0000 7f0998600640 1 -- 172.21.15.96:0/2634038741 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f09900d0d80 msgr2=0x7f09900b18a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:18.812 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.808+0000 7f0998600640 1 --2- 172.21.15.96:0/2634038741 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f09900d0d80 0x7f09900b18a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f097c00e7e0 tx=0x7f097c00eca0 comp rx=0 tx=0).stop 2024-05-28T06:04:18.812 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.809+0000 7f0998600640 1 -- 172.21.15.96:0/2634038741 shutdown_connections 2024-05-28T06:04:18.812 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.809+0000 7f0998600640 1 --2- 172.21.15.96:0/2634038741 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f097006c9f0 0x7f097006eeb0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:18.812 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.809+0000 7f0998600640 1 --2- 172.21.15.96:0/2634038741 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f099015ae90 0x7f09900c42b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:18.812 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.809+0000 7f0998600640 1 --2- 172.21.15.96:0/2634038741 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f099015aab0 0x7f09900c3d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:18.812 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.809+0000 7f0998600640 1 --2- 172.21.15.96:0/2634038741 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f09900d0d80 0x7f09900b18a0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:18.812 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.809+0000 7f0998600640 1 -- 172.21.15.96:0/2634038741 >> 172.21.15.96:0/2634038741 conn(0x7f09900d0260 msgr2=0x7f09900cad00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:18.812 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.811+0000 7f0998600640 1 -- 172.21.15.96:0/2634038741 shutdown_connections 2024-05-28T06:04:18.813 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.811+0000 7f0998600640 1 -- 172.21.15.96:0/2634038741 wait complete. 2024-05-28T06:04:18.813 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.812+0000 7f5650c00640 1 -- 172.21.15.96:0/735651242 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f562c06c9c0 msgr2=0x7f562c06ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:18.813 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.812+0000 7f5650c00640 1 --2- 172.21.15.96:0/735651242 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f562c06c9c0 0x7f562c06ee80 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f563c045000 tx=0x7f563c03f000 comp rx=0 tx=0).stop 2024-05-28T06:04:18.813 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.812+0000 7f5650c00640 1 -- 172.21.15.96:0/735651242 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f564c162e70 msgr2=0x7f564c0ad670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:18.813 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.812+0000 7f5650c00640 1 --2- 172.21.15.96:0/735651242 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f564c162e70 0x7f564c0ad670 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f564400bdf0 tx=0x7f5644009ef0 comp rx=0 tx=0).stop 2024-05-28T06:04:18.813 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.812+0000 7f5650c00640 1 -- 172.21.15.96:0/735651242 shutdown_connections 2024-05-28T06:04:18.814 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.812+0000 7f5650c00640 1 --2- 172.21.15.96:0/735651242 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f562c06c9c0 0x7f562c06ee80 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:18.814 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.812+0000 7f5650c00640 1 --2- 172.21.15.96:0/735651242 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f564c162e70 0x7f564c0ad670 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:18.814 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.812+0000 7f5650c00640 1 --2- 172.21.15.96:0/735651242 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f564c0d04a0 0x7f564c0ad130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:18.814 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.812+0000 7f5650c00640 1 --2- 172.21.15.96:0/735651242 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f564c06f6e0 0x7f564c155da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:18.814 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.812+0000 7f5650c00640 1 -- 172.21.15.96:0/735651242 >> 172.21.15.96:0/735651242 conn(0x7f564c06bae0 msgr2=0x7f564c15d210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:18.814 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.812+0000 7f5650c00640 1 -- 172.21.15.96:0/735651242 shutdown_connections 2024-05-28T06:04:18.814 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.812+0000 7f5650c00640 1 -- 172.21.15.96:0/735651242 wait complete. 2024-05-28T06:04:18.820 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.1 2024-05-28T06:04:18.823 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.823+0000 7f10e4a00640 1 -- 172.21.15.96:0/2835480415 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f10d8053e10 con 0x7f10d80e1080 2024-05-28T06:04:18.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.824+0000 7f10d6a00640 1 -- 172.21.15.96:0/2835480415 <== mon.2 v2:172.21.15.123:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f10d80e03a0 con 0x7f10d80e1080 2024-05-28T06:04:18.824 INFO:teuthology.orchestra.run.smithi096.stdout:55834574853 2024-05-28T06:04:18.827 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.826+0000 7f10e4a00640 1 -- 172.21.15.96:0/2835480415 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f10c006c940 msgr2=0x7f10c006ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:18.827 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.826+0000 7f10e4a00640 1 --2- 172.21.15.96:0/2835480415 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f10c006c940 0x7f10c006ee00 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f10d8179010 tx=0x7f10d0008040 comp rx=0 tx=0).stop 2024-05-28T06:04:18.827 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.826+0000 7f10e4a00640 1 -- 172.21.15.96:0/2835480415 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f10d80e1080 msgr2=0x7f10d804c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:18.827 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.826+0000 7f10e4a00640 1 --2- 172.21.15.96:0/2835480415 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f10d80e1080 0x7f10d804c560 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f10cc031000 tx=0x7f10cc0046c0 comp rx=0 tx=0).stop 2024-05-28T06:04:18.827 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.827+0000 7f10e4a00640 1 -- 172.21.15.96:0/2835480415 shutdown_connections 2024-05-28T06:04:18.827 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.827+0000 7f10e4a00640 1 --2- 172.21.15.96:0/2835480415 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f10c006c940 0x7f10c006ee00 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:18.827 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.827+0000 7f10e4a00640 1 --2- 172.21.15.96:0/2835480415 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f10d80e1080 0x7f10d804c560 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:18.827 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.827+0000 7f10e4a00640 1 --2- 172.21.15.96:0/2835480415 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f10d8050dc0 0x7f10d8185c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:18.827 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.827+0000 7f10e4a00640 1 --2- 172.21.15.96:0/2835480415 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f10d803a3f0 0x7f10d80df880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:18.827 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.827+0000 7f10e4a00640 1 -- 172.21.15.96:0/2835480415 >> 172.21.15.96:0/2835480415 conn(0x7f10d8009fa0 msgr2=0x7f10d80376d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:18.828 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.828+0000 7f10e4a00640 1 -- 172.21.15.96:0/2835480415 shutdown_connections 2024-05-28T06:04:18.828 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.828+0000 7f10e4a00640 1 -- 172.21.15.96:0/2835480415 wait complete. 2024-05-28T06:04:18.832 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.2 2024-05-28T06:04:18.832 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.3 2024-05-28T06:04:18.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.832+0000 7f6489400640 1 -- 172.21.15.96:0/1802077102 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f64840b03d0 con 0x7f648406fcb0 2024-05-28T06:04:18.833 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.833+0000 7f647be00640 1 -- 172.21.15.96:0/1802077102 <== mon.1 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f647405d500 con 0x7f648406fcb0 2024-05-28T06:04:18.833 INFO:teuthology.orchestra.run.smithi096.stdout:55834574853 2024-05-28T06:04:18.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.836+0000 7f6489400640 1 -- 172.21.15.96:0/1802077102 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f646406caf0 msgr2=0x7f646406efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:18.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.836+0000 7f6489400640 1 --2- 172.21.15.96:0/1802077102 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f646406caf0 0x7f646406efb0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f64840b0e30 tx=0x7f647002e000 comp rx=0 tx=0).stop 2024-05-28T06:04:18.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.836+0000 7f6489400640 1 -- 172.21.15.96:0/1802077102 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f648406fcb0 msgr2=0x7f6484145fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:18.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.836+0000 7f6489400640 1 --2- 172.21.15.96:0/1802077102 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f648406fcb0 0x7f6484145fd0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f647400c610 tx=0x7f647400cad0 comp rx=0 tx=0).stop 2024-05-28T06:04:18.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.836+0000 7f6489400640 1 -- 172.21.15.96:0/1802077102 shutdown_connections 2024-05-28T06:04:18.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.836+0000 7f6489400640 1 --2- 172.21.15.96:0/1802077102 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f646406caf0 0x7f646406efb0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:18.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.836+0000 7f6489400640 1 --2- 172.21.15.96:0/1802077102 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f64841595f0 0x7f6484201290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:18.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.836+0000 7f6489400640 1 --2- 172.21.15.96:0/1802077102 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f648415a7f0 0x7f6484146510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:18.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.836+0000 7f6489400640 1 --2- 172.21.15.96:0/1802077102 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f648406fcb0 0x7f6484145fd0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:18.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.836+0000 7f6489400640 1 -- 172.21.15.96:0/1802077102 >> 172.21.15.96:0/1802077102 conn(0x7f64840b1fc0 msgr2=0x7f648406e320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:18.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.836+0000 7f6489400640 1 -- 172.21.15.96:0/1802077102 shutdown_connections 2024-05-28T06:04:18.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.837+0000 7f6489400640 1 -- 172.21.15.96:0/1802077102 wait complete. 2024-05-28T06:04:18.839 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.5 2024-05-28T06:04:18.839 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.6 2024-05-28T06:04:18.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.841+0000 7ff8b1800640 1 -- 172.21.15.96:0/1668559030 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7ff884005850 con 0x7ff8ac163870 2024-05-28T06:04:18.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.842+0000 7ff8a2a00640 1 -- 172.21.15.96:0/1668559030 <== mon.2 v2:172.21.15.123:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7ff8a4059b70 con 0x7ff8ac163870 2024-05-28T06:04:18.843 INFO:teuthology.orchestra.run.smithi096.stdout:55834574853 2024-05-28T06:04:18.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.844+0000 7ff8b1800640 1 -- 172.21.15.96:0/1668559030 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff88c06cb20 msgr2=0x7ff88c06efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:18.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.844+0000 7ff8b1800640 1 --2- 172.21.15.96:0/1668559030 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff88c06cb20 0x7ff88c06efe0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ff89c005880 tx=0x7ff89c009040 comp rx=0 tx=0).stop 2024-05-28T06:04:18.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.844+0000 7ff8b1800640 1 -- 172.21.15.96:0/1668559030 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff8ac163870 msgr2=0x7ff8ac0cda60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:18.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.844+0000 7ff8b1800640 1 --2- 172.21.15.96:0/1668559030 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff8ac163870 0x7ff8ac0cda60 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff8a4007e60 tx=0x7ff8a400d4a0 comp rx=0 tx=0).stop 2024-05-28T06:04:18.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.845+0000 7ff8b1800640 1 -- 172.21.15.96:0/1668559030 shutdown_connections 2024-05-28T06:04:18.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.845+0000 7ff8b1800640 1 --2- 172.21.15.96:0/1668559030 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff88c06cb20 0x7ff88c06efe0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:18.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.845+0000 7ff8b1800640 1 --2- 172.21.15.96:0/1668559030 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff8ac163870 0x7ff8ac0cda60 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:18.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.845+0000 7ff8b1800640 1 --2- 172.21.15.96:0/1668559030 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8ac0d1020 0x7ff8ac0cd520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:18.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.845+0000 7ff8b1800640 1 --2- 172.21.15.96:0/1668559030 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff8ac0cf490 0x7ff8ac15ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:18.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.845+0000 7ff8b1800640 1 -- 172.21.15.96:0/1668559030 >> 172.21.15.96:0/1668559030 conn(0x7ff8ac08bfc0 msgr2=0x7ff8ac0b2a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:18.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.845+0000 7ff8b1800640 1 -- 172.21.15.96:0/1668559030 shutdown_connections 2024-05-28T06:04:18.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:18.845+0000 7ff8b1800640 1 -- 172.21.15.96:0/1668559030 wait complete. 2024-05-28T06:04:18.849 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.0 2024-05-28T06:04:18.857 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.7 2024-05-28T06:04:18.864 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.4 2024-05-28T06:04:19.821 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-05-28T06:04:19.833 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-05-28T06:04:19.833 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-05-28T06:04:19.839 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-05-28T06:04:19.839 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-05-28T06:04:19.850 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-05-28T06:04:19.857 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-05-28T06:04:19.866 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-05-28T06:04:20.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.105+0000 7ff337e00640 1 Processor -- start 2024-05-28T06:04:20.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.106+0000 7ff337e00640 1 -- start start 2024-05-28T06:04:20.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.107+0000 7ff337e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7ff338073120 0x7ff3380b3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:20.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.108+0000 7ff337e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7ff338073500 0x7ff33815b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:20.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.109+0000 7ff336000640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7ff3380b3fe0 0x7ff33815d050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.96:49206/0 (socket says 172.21.15.96:49206) 2024-05-28T06:04:20.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.109+0000 7ff336000640 1 -- 172.21.15.96:0/4228700175 learned_addr learned my addr 172.21.15.96:0/4228700175 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:04:20.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.109+0000 7ff337400640 1 --2- 172.21.15.96:0/4228700175 >> v2:172.21.15.96:3300/0 conn(0x7ff338073500 0x7ff33815b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:20.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.109+0000 7ff337e00640 1 -- 172.21.15.96:0/4228700175 --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff33815bff0 con 0x7ff3380b3fe0 2024-05-28T06:04:20.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.109+0000 7ff337e00640 1 -- 172.21.15.96:0/4228700175 --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7ff33815c290 con 0x7ff338073500 2024-05-28T06:04:20.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.109+0000 7ff337e00640 1 -- 172.21.15.96:0/4228700175 --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7ff33815c570 con 0x7ff338073120 2024-05-28T06:04:20.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.111+0000 7ff337400640 1 -- 172.21.15.96:0/4228700175 >> v2:172.21.15.123:3300/0 conn(0x7ff338073120 msgr2=0x7ff3380b3aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:20.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.111+0000 7ff337400640 1 --2- 172.21.15.96:0/4228700175 >> v2:172.21.15.123:3300/0 conn(0x7ff338073120 0x7ff3380b3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:20.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.111+0000 7ff337400640 1 -- 172.21.15.96:0/4228700175 >> v1:172.21.15.123:6789/0 conn(0x7ff3380b3fe0 legacy=0x7ff33815d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:20.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.111+0000 7ff337400640 1 -- 172.21.15.96:0/4228700175 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff33815c850 con 0x7ff338073500 2024-05-28T06:04:20.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.113+0000 7ff337400640 1 --2- 172.21.15.96:0/4228700175 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff338073500 0x7ff33815b4a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7ff338075a20 tx=0x7ff32402a010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b1219c7766ee45d server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:20.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.114+0000 7ff335600640 1 -- 172.21.15.96:0/4228700175 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff324037070 con 0x7ff338073500 2024-05-28T06:04:20.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.114+0000 7ff335600640 1 -- 172.21.15.96:0/4228700175 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff32402aca0 con 0x7ff338073500 2024-05-28T06:04:20.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.114+0000 7ff335600640 1 -- 172.21.15.96:0/4228700175 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff324032660 con 0x7ff338073500 2024-05-28T06:04:20.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.116+0000 7ff337e00640 1 -- 172.21.15.96:0/4228700175 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff338073500 msgr2=0x7ff33815b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:20.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.116+0000 7ff337e00640 1 --2- 172.21.15.96:0/4228700175 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff338073500 0x7ff33815b4a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7ff338075a20 tx=0x7ff32402a010 comp rx=0 tx=0).stop 2024-05-28T06:04:20.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.116+0000 7ff337e00640 1 -- 172.21.15.96:0/4228700175 shutdown_connections 2024-05-28T06:04:20.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.117+0000 7ff337e00640 1 --2- 172.21.15.96:0/4228700175 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff338073500 0x7ff33815b4a0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:20.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.117+0000 7ff337e00640 1 --2- 172.21.15.96:0/4228700175 >> v2:172.21.15.123:3300/0 conn(0x7ff338073120 0x7ff3380b3aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:20.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.117+0000 7ff337e00640 1 -- 172.21.15.96:0/4228700175 >> 172.21.15.96:0/4228700175 conn(0x7ff33806c380 msgr2=0x7ff3380b1db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:20.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.117+0000 7ff337e00640 1 -- 172.21.15.96:0/4228700175 shutdown_connections 2024-05-28T06:04:20.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.118+0000 7ff337e00640 1 -- 172.21.15.96:0/4228700175 wait complete. 2024-05-28T06:04:20.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.121+0000 7ff337e00640 1 Processor -- start 2024-05-28T06:04:20.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.121+0000 7ff337e00640 1 -- start start 2024-05-28T06:04:20.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.123+0000 7ff337e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff338073120 0x7ff338152640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:20.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.124+0000 7ff337e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff338073500 0x7ff338153cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:20.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.124+0000 7ff337e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff3380b3fe0 0x7ff3380cedb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:20.125 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.125+0000 7ff336000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff338073500 0x7ff338153cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:20.125 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.125+0000 7ff336000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff338073500 0x7ff338153cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.96:39768/0 (socket says 172.21.15.96:39768) 2024-05-28T06:04:20.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.125+0000 7ff336000640 1 -- 172.21.15.96:0/230603747 learned_addr learned my addr 172.21.15.96:0/230603747 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:20.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.125+0000 7ff337400640 1 --2- 172.21.15.96:0/230603747 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff3380b3fe0 0x7ff3380cedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:20.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.125+0000 7ff337e00640 1 -- 172.21.15.96:0/230603747 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff3380b2a80 con 0x7ff338073120 2024-05-28T06:04:20.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.126+0000 7ff336a00640 1 --2- 172.21.15.96:0/230603747 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff338073120 0x7ff338152640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:20.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.126+0000 7ff337e00640 1 -- 172.21.15.96:0/230603747 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff3380cc700 con 0x7ff338073500 2024-05-28T06:04:20.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.126+0000 7ff337e00640 1 -- 172.21.15.96:0/230603747 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ff33815b9e0 con 0x7ff3380b3fe0 2024-05-28T06:04:20.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.127+0000 7ff336a00640 1 -- 172.21.15.96:0/230603747 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff3380b3fe0 msgr2=0x7ff3380cedb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:20.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.127+0000 7ff336a00640 1 --2- 172.21.15.96:0/230603747 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff3380b3fe0 0x7ff3380cedb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:20.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.128+0000 7ff336a00640 1 -- 172.21.15.96:0/230603747 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff338073500 msgr2=0x7ff338153cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:20.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.128+0000 7ff336a00640 1 --2- 172.21.15.96:0/230603747 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff338073500 0x7ff338153cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:20.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.128+0000 7ff336000640 1 --2- 172.21.15.96:0/230603747 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff338073500 0x7ff338153cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:04:20.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.128+0000 7ff336a00640 1 -- 172.21.15.96:0/230603747 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff338152c20 con 0x7ff338073120 2024-05-28T06:04:20.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.129+0000 7ff336a00640 1 --2- 172.21.15.96:0/230603747 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff338073120 0x7ff338152640 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7ff33000bd70 tx=0x7ff33000be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:20.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.130+0000 7ff32ea00640 1 -- 172.21.15.96:0/230603747 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff330019070 con 0x7ff338073120 2024-05-28T06:04:20.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.130+0000 7ff337e00640 1 -- 172.21.15.96:0/230603747 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff338152f00 con 0x7ff338073120 2024-05-28T06:04:20.131 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.130+0000 7ff337e00640 1 -- 172.21.15.96:0/230603747 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3381537a0 con 0x7ff338073120 2024-05-28T06:04:20.133 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.130+0000 7ff32ea00640 1 -- 172.21.15.96:0/230603747 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff330014440 con 0x7ff338073120 2024-05-28T06:04:20.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.130+0000 7ff32ea00640 1 -- 172.21.15.96:0/230603747 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff330019070 con 0x7ff338073120 2024-05-28T06:04:20.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.133+0000 7ff337e00640 1 -- 172.21.15.96:0/230603747 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff310005290 con 0x7ff338073120 2024-05-28T06:04:20.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.139+0000 7ff32ea00640 1 -- 172.21.15.96:0/230603747 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff33001c070 con 0x7ff338073120 2024-05-28T06:04:20.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.140+0000 7ff32ea00640 1 --2- 172.21.15.96:0/230603747 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff31806caf0 0x7ff31806efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:20.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.141+0000 7ff32ea00640 1 -- 172.21.15.96:0/230603747 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7ff33008bb30 con 0x7ff338073120 2024-05-28T06:04:20.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.142+0000 7ff336000640 1 --2- 172.21.15.96:0/230603747 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff31806caf0 0x7ff31806efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:20.145 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.145+0000 7ff32ea00640 1 -- 172.21.15.96:0/230603747 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff330023020 con 0x7ff338073120 2024-05-28T06:04:20.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.149+0000 7ff336000640 1 --2- 172.21.15.96:0/230603747 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff31806caf0 0x7ff31806efb0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff3381551a0 tx=0x7ff328009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:20.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.153+0000 7fa158c00640 1 Processor -- start 2024-05-28T06:04:20.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.154+0000 7fa158c00640 1 -- start start 2024-05-28T06:04:20.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.156+0000 7fa158c00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fa1540d4130 0x7fa1540d4510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:20.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.156+0000 7fa158c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa1540cfd50 0x7fa1540d2090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:20.159 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.157+0000 7fa151e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fa1540d4130 0x7fa1540d4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:20.159 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.157+0000 7fa151e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fa1540d4130 0x7fa1540d4510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48012/0 (socket says 172.21.15.96:48012) 2024-05-28T06:04:20.159 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.157+0000 7fa158c00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fa1540d25d0 0x7fa154162270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:20.159 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.157+0000 7fa158c00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fa1540cefd0 con 0x7fa1540d4130 2024-05-28T06:04:20.159 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.157+0000 7fa151400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa1540cfd50 0x7fa1540d2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:20.159 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.157+0000 7fa158c00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fa154162fa0 con 0x7fa1540d25d0 2024-05-28T06:04:20.159 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.157+0000 7fa158c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fa1540cf270 con 0x7fa1540cfd50 2024-05-28T06:04:20.159 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.157+0000 7fa151400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa1540cfd50 0x7fa1540d2090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.96:59914/0 (socket says 172.21.15.96:59914) 2024-05-28T06:04:20.159 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.157+0000 7fa151400640 1 -- 172.21.15.96:0/629618055 learned_addr learned my addr 172.21.15.96:0/629618055 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:20.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.158+0000 7fa151e00640 1 -- 172.21.15.96:0/629618055 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa1540cfd50 msgr2=0x7fa1540d2090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:20.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.158+0000 7fa151e00640 1 --2- 172.21.15.96:0/629618055 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa1540cfd50 0x7fa1540d2090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:20.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.158+0000 7fa151e00640 1 -- 172.21.15.96:0/629618055 >> v2:172.21.15.123:3300/0 conn(0x7fa1540d25d0 msgr2=0x7fa154162270 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-28T06:04:20.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.158+0000 7fa151e00640 1 --2- 172.21.15.96:0/629618055 >> v2:172.21.15.123:3300/0 conn(0x7fa1540d25d0 0x7fa154162270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:20.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.158+0000 7fa151e00640 1 -- 172.21.15.96:0/629618055 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1540cf550 con 0x7fa1540d4130 2024-05-28T06:04:20.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.160+0000 7f3d0ee00640 1 Processor -- start 2024-05-28T06:04:20.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.160+0000 7fa151e00640 1 --2- 172.21.15.96:0/629618055 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa1540d4130 0x7fa1540d4510 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fa15406cba0 tx=0x7fa14002a120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=743b2b40603e419d server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:20.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.161+0000 7fa150a00640 1 -- 172.21.15.96:0/629618055 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa140037070 con 0x7fa1540d4130 2024-05-28T06:04:20.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.161+0000 7fa150a00640 1 -- 172.21.15.96:0/629618055 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa140032440 con 0x7fa1540d4130 2024-05-28T06:04:20.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.161+0000 7fa150a00640 1 -- 172.21.15.96:0/629618055 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa140032810 con 0x7fa1540d4130 2024-05-28T06:04:20.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.160+0000 7f3d0ee00640 1 -- start start 2024-05-28T06:04:20.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.161+0000 7fe6b4600640 1 Processor -- start 2024-05-28T06:04:20.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.161+0000 7fe6b4600640 1 -- start start 2024-05-28T06:04:20.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.162+0000 7fa158c00640 1 -- 172.21.15.96:0/629618055 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa1540d4130 msgr2=0x7fa1540d4510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:20.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.162+0000 7fa158c00640 1 --2- 172.21.15.96:0/629618055 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa1540d4130 0x7fa1540d4510 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fa15406cba0 tx=0x7fa14002a120 comp rx=0 tx=0).stop 2024-05-28T06:04:20.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.162+0000 7f3d0ee00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f3d0815e870 0x7f3d080d0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:20.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.162+0000 7f3d0ee00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d0815ec50 con 0x7f3d080cfa40 2024-05-28T06:04:20.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.163+0000 7f3d0ee00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d080cff80 con 0x7f3d080d14d0 2024-05-28T06:04:20.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.163+0000 7f3d0ee00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f3d080d2150 con 0x7f3d0815e870 2024-05-28T06:04:20.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.163+0000 7f3d0e400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f3d080cfa40 0x7f3d080d25e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:35192/0 (socket says 172.21.15.96:35192) 2024-05-28T06:04:20.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.163+0000 7f3d0da00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f3d0815e870 0x7f3d080d0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:20.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.163+0000 7f3d0e400640 1 -- 172.21.15.96:0/4044591752 learned_addr learned my addr 172.21.15.96:0/4044591752 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:04:20.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.163+0000 7fe6b4600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fe6ac0d04a0 0x7fe6ac0d0880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:20.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.163+0000 7fe6b4600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fe6ac162e70 0x7fe6ac0ce230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:20.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.163+0000 7fe6b4600640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6ac0b4350 con 0x7fe6ac06f6e0 2024-05-28T06:04:20.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.163+0000 7fe6b4600640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fe6ac0b3ec0 con 0x7fe6ac162e70 2024-05-28T06:04:20.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.163+0000 7fe6b4600640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fe6ac0b32d0 con 0x7fe6ac0d04a0 2024-05-28T06:04:20.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.164+0000 7fa158c00640 1 -- 172.21.15.96:0/629618055 shutdown_connections 2024-05-28T06:04:20.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.164+0000 7fa158c00640 1 --2- 172.21.15.96:0/629618055 >> v2:172.21.15.123:3300/0 conn(0x7fa1540d25d0 0x7fa154162270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:20.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.164+0000 7fa158c00640 1 --2- 172.21.15.96:0/629618055 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa1540cfd50 0x7fa1540d2090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:20.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.164+0000 7fa158c00640 1 --2- 172.21.15.96:0/629618055 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa1540d4130 0x7fa1540d4510 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:20.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.164+0000 7fa158c00640 1 -- 172.21.15.96:0/629618055 >> 172.21.15.96:0/629618055 conn(0x7fa15408bfc0 msgr2=0x7fa15409d820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:20.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.164+0000 7f3d07e00640 1 -- 172.21.15.96:0/4044591752 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3417365161 0 0) 0x7f3d0815ec50 con 0x7f3d080cfa40 2024-05-28T06:04:20.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.164+0000 7f3d07e00640 1 -- 172.21.15.96:0/4044591752 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3cf0003520 con 0x7f3d080cfa40 2024-05-28T06:04:20.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.164+0000 7fb083800640 1 Processor -- start 2024-05-28T06:04:20.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.164+0000 7f3d0da00640 1 -- 172.21.15.96:0/4044591752 >> v1:172.21.15.123:6789/0 conn(0x7f3d080d14d0 legacy=0x7f3d080cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:20.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.164+0000 7f3d0da00640 1 -- 172.21.15.96:0/4044591752 >> v1:172.21.15.96:6789/0 conn(0x7f3d080cfa40 legacy=0x7f3d080d25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:20.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.164+0000 7f3d0da00640 1 -- 172.21.15.96:0/4044591752 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d080d0260 con 0x7f3d0815e870 2024-05-28T06:04:20.171 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.164+0000 7fe6b1c00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fe6ac0d04a0 0x7fe6ac0d0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:20.171 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.164+0000 7fe6b2600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fe6ac162e70 0x7fe6ac0ce230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:20.171 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.165+0000 7fe6b2600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fe6ac162e70 0x7fe6ac0ce230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48024/0 (socket says 172.21.15.96:48024) 2024-05-28T06:04:20.171 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.165+0000 7fe6b2600640 1 -- 172.21.15.96:0/1450362462 learned_addr learned my addr 172.21.15.96:0/1450362462 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:20.172 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.164+0000 7fb083800640 1 -- start start 2024-05-28T06:04:20.172 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.166+0000 7fa158c00640 1 -- 172.21.15.96:0/629618055 shutdown_connections 2024-05-28T06:04:20.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.165+0000 7fe6b2600640 1 -- 172.21.15.96:0/1450362462 >> v2:172.21.15.123:3300/0 conn(0x7fe6ac0d04a0 msgr2=0x7fe6ac0d0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:20.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.166+0000 7fe6b2600640 1 --2- 172.21.15.96:0/1450362462 >> v2:172.21.15.123:3300/0 conn(0x7fe6ac0d04a0 0x7fe6ac0d0880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:20.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.166+0000 7fa158c00640 1 -- 172.21.15.96:0/629618055 wait complete. 2024-05-28T06:04:20.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.166+0000 7f3d0da00640 1 --2- 172.21.15.96:0/4044591752 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d0815e870 0x7f3d080d0f90 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3cfc0097a0 tx=0x7f3cfc02a1e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eaa558a8702d9d08 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:20.174 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.166+0000 7fb083800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb07c0c5500 0x7fb07c0c58e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:20.174 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.166+0000 7fb083800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fb07c09aa30 0x7fb07c166a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:20.174 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.166+0000 7fb083800640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb07c163380 con 0x7fb07c09ae90 2024-05-28T06:04:20.174 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.166+0000 7f2e88600640 1 Processor -- start 2024-05-28T06:04:20.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.166+0000 7fb083800640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fb07c09bc40 con 0x7fb07c09aa30 2024-05-28T06:04:20.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.166+0000 7fb083800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fb07c0ccd40 con 0x7fb07c0c5500 2024-05-28T06:04:20.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.166+0000 7fe6b2600640 1 -- 172.21.15.96:0/1450362462 >> v1:172.21.15.96:6789/0 conn(0x7fe6ac06f6e0 legacy=0x7fe6ac0cd4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:20.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.166+0000 7fe6b2600640 1 -- 172.21.15.96:0/1450362462 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6ac0b3590 con 0x7fe6ac162e70 2024-05-28T06:04:20.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.167+0000 7fb080e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb07c0c5500 0x7fb07c0c58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:20.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.167+0000 7fb080e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb07c0c5500 0x7fb07c0c58e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.96:59928/0 (socket says 172.21.15.96:59928) 2024-05-28T06:04:20.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.167+0000 7fb080e00640 1 -- 172.21.15.96:0/408747247 learned_addr learned my addr 172.21.15.96:0/408747247 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:20.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.167+0000 7f2e88600640 1 -- start start 2024-05-28T06:04:20.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.167+0000 7f3d07e00640 1 -- 172.21.15.96:0/4044591752 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cfc037070 con 0x7f3d0815e870 2024-05-28T06:04:20.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.167+0000 7f3d07e00640 1 -- 172.21.15.96:0/4044591752 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3cfc032440 con 0x7f3d0815e870 2024-05-28T06:04:20.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.167+0000 7f3d07e00640 1 -- 172.21.15.96:0/4044591752 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cfc032810 con 0x7f3d0815e870 2024-05-28T06:04:20.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.167+0000 7fe6b1c00640 1 --2- 172.21.15.96:0/1450362462 >> v2:172.21.15.123:3300/0 conn(0x7fe6ac0d04a0 0x7fe6ac0d0880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-28T06:04:20.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.168+0000 7fb07be00640 1 --2- 172.21.15.96:0/408747247 >> v2:172.21.15.96:3300/0 conn(0x7fb07c09aa30 0x7fb07c166a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:20.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.168+0000 7fb080e00640 1 -- 172.21.15.96:0/408747247 >> v2:172.21.15.96:3300/0 conn(0x7fb07c09aa30 msgr2=0x7fb07c166a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:20.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.168+0000 7fb080e00640 1 --2- 172.21.15.96:0/408747247 >> v2:172.21.15.96:3300/0 conn(0x7fb07c09aa30 0x7fb07c166a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:20.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.168+0000 7fb080e00640 1 -- 172.21.15.96:0/408747247 >> v1:172.21.15.96:6789/0 conn(0x7fb07c09ae90 legacy=0x7fb07c166fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:20.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.168+0000 7fb080e00640 1 -- 172.21.15.96:0/408747247 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb07c0c5e20 con 0x7fb07c0c5500 2024-05-28T06:04:20.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.169+0000 7f2e88600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f2e800d0d80 0x7f2e8015a840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:20.180 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.169+0000 7f3d0ee00640 1 -- 172.21.15.96:0/4044591752 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d0815e870 msgr2=0x7f3d080d0f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:20.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.169+0000 7f2e88600640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e800cec50 con 0x7f2e800cbd10 2024-05-28T06:04:20.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.169+0000 7f2e88600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e800ae800 con 0x7f2e80073020 2024-05-28T06:04:20.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.169+0000 7f2e88600640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f2e800aeae0 con 0x7f2e800d0d80 2024-05-28T06:04:20.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.168+0000 7fe6b2600640 1 --2- 172.21.15.96:0/1450362462 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe6ac162e70 0x7fe6ac0ce230 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe6ac0c3470 tx=0x7fe6a402a6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52a5470877213d54 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:20.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.169+0000 7f3d0ee00640 1 --2- 172.21.15.96:0/4044591752 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d0815e870 0x7f3d080d0f90 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3cfc0097a0 tx=0x7f3cfc02a1e0 comp rx=0 tx=0).stop 2024-05-28T06:04:20.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.169+0000 7f3d0ee00640 1 -- 172.21.15.96:0/4044591752 shutdown_connections 2024-05-28T06:04:20.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.169+0000 7f3d0ee00640 1 --2- 172.21.15.96:0/4044591752 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d0815e870 0x7f3d080d0f90 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:20.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.169+0000 7f3d0ee00640 1 -- 172.21.15.96:0/4044591752 >> 172.21.15.96:0/4044591752 conn(0x7f3d0808bfc0 msgr2=0x7f3d080d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:20.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.169+0000 7f2e85200640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f2e80073020 0x7f2e800cde30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.96:49222/0 (socket says 172.21.15.96:49222) 2024-05-28T06:04:20.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.169+0000 7f2e85200640 1 -- 172.21.15.96:0/2738906583 learned_addr learned my addr 172.21.15.96:0/2738906583 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:04:20.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.169+0000 7f2e85c00640 1 --2- 172.21.15.96:0/2738906583 >> v2:172.21.15.123:3300/0 conn(0x7f2e800d0d80 0x7f2e8015a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:20.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.170+0000 7f3d0ee00640 1 -- 172.21.15.96:0/4044591752 shutdown_connections 2024-05-28T06:04:20.185 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.170+0000 7fb080e00640 1 --2- 172.21.15.96:0/408747247 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb07c0c5500 0x7fb07c0c58e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb0700098a0 tx=0x7fb07002a1a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c613b325d6615501 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:20.185 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.170+0000 7fe6abe00640 1 -- 172.21.15.96:0/1450362462 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6a4036070 con 0x7fe6ac162e70 2024-05-28T06:04:20.185 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.170+0000 7f3d0ee00640 1 -- 172.21.15.96:0/4044591752 wait complete. 2024-05-28T06:04:20.185 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.170+0000 7fb07b400640 1 -- 172.21.15.96:0/408747247 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb070037070 con 0x7fb07c0c5500 2024-05-28T06:04:20.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.170+0000 7f2e7fe00640 1 -- 172.21.15.96:0/2738906583 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 701266387 0 0) 0x7f2e800ae800 con 0x7f2e80073020 2024-05-28T06:04:20.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.170+0000 7f2e7fe00640 1 -- 172.21.15.96:0/2738906583 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2e68005730 con 0x7f2e80073020 2024-05-28T06:04:20.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.170+0000 7fe6abe00640 1 -- 172.21.15.96:0/1450362462 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6a4031600 con 0x7fe6ac162e70 2024-05-28T06:04:20.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.170+0000 7fe6abe00640 1 -- 172.21.15.96:0/1450362462 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6a4036070 con 0x7fe6ac162e70 2024-05-28T06:04:20.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.171+0000 7f2e85c00640 1 -- 172.21.15.96:0/2738906583 >> v1:172.21.15.123:6789/0 conn(0x7f2e80073020 legacy=0x7f2e800cde30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:20.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.171+0000 7fb9dd600640 1 Processor -- start 2024-05-28T06:04:20.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.171+0000 7f2e85c00640 1 -- 172.21.15.96:0/2738906583 >> v1:172.21.15.96:6789/0 conn(0x7f2e800cbd10 legacy=0x7f2e800ce540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:20.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.171+0000 7f2e85c00640 1 -- 172.21.15.96:0/2738906583 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e800aedc0 con 0x7f2e800d0d80 2024-05-28T06:04:20.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.171+0000 7fb9dd600640 1 -- start start 2024-05-28T06:04:20.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.171+0000 7fa158c00640 1 Processor -- start 2024-05-28T06:04:20.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.172+0000 7fa158c00640 1 -- start start 2024-05-28T06:04:20.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.172+0000 7fe6b4600640 1 -- 172.21.15.96:0/1450362462 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe6ac162e70 msgr2=0x7fe6ac0ce230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:20.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.172+0000 7fe6b4600640 1 --2- 172.21.15.96:0/1450362462 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe6ac162e70 0x7fe6ac0ce230 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe6ac0c3470 tx=0x7fe6a402a6b0 comp rx=0 tx=0).stop 2024-05-28T06:04:20.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.172+0000 7fb07b400640 1 -- 172.21.15.96:0/408747247 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb070032440 con 0x7fb07c0c5500 2024-05-28T06:04:20.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.172+0000 7fb07b400640 1 -- 172.21.15.96:0/408747247 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb070032810 con 0x7fb07c0c5500 2024-05-28T06:04:20.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.172+0000 7f2e85c00640 1 --2- 172.21.15.96:0/2738906583 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e800d0d80 0x7f2e8015a840 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f2e700097a0 tx=0x7f2e7002a250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ba18603a4462dbf server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:20.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.172+0000 7f2e7fe00640 1 -- 172.21.15.96:0/2738906583 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e70037070 con 0x7f2e800d0d80 2024-05-28T06:04:20.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.172+0000 7fe6b4600640 1 -- 172.21.15.96:0/1450362462 shutdown_connections 2024-05-28T06:04:20.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.172+0000 7fe6b4600640 1 --2- 172.21.15.96:0/1450362462 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe6ac162e70 0x7fe6ac0ce230 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:20.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.172+0000 7fe6b4600640 1 --2- 172.21.15.96:0/1450362462 >> v2:172.21.15.123:3300/0 conn(0x7fe6ac0d04a0 0x7fe6ac0d0880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:20.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.172+0000 7fe6b4600640 1 -- 172.21.15.96:0/1450362462 >> 172.21.15.96:0/1450362462 conn(0x7fe6ac06bae0 msgr2=0x7fe6ac06bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:20.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.173+0000 7fb9dd600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fb9d80d2db0 0x7fb9d80b3b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:20.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.173+0000 7f3d0ee00640 1 Processor -- start 2024-05-28T06:04:20.192 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.173+0000 7f2e7fe00640 1 -- 172.21.15.96:0/2738906583 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e70032440 con 0x7f2e800d0d80 2024-05-28T06:04:20.192 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.173+0000 7fb9dd600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb9d815ed20 0x7fb9d815f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:20.193 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.173+0000 7fb9dd600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9d8163500 con 0x7fb9d80d4d60 2024-05-28T06:04:20.193 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.173+0000 7fb9dd600640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fb9d80d5540 con 0x7fb9d80d2db0 2024-05-28T06:04:20.193 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.173+0000 7fb9dd600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fb9d8162740 con 0x7fb9d815ed20 2024-05-28T06:04:20.193 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.173+0000 7fb083800640 1 -- 172.21.15.96:0/408747247 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb07c0c5500 msgr2=0x7fb07c0c58e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:20.193 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.173+0000 7fb083800640 1 --2- 172.21.15.96:0/408747247 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb07c0c5500 0x7fb07c0c58e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb0700098a0 tx=0x7fb07002a1a0 comp rx=0 tx=0).stop 2024-05-28T06:04:20.194 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.173+0000 7f3d0ee00640 1 -- start start 2024-05-28T06:04:20.194 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.173+0000 7f2e7fe00640 1 -- 172.21.15.96:0/2738906583 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e700327f0 con 0x7f2e800d0d80 2024-05-28T06:04:20.194 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.173+0000 7fb9d6a00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fb9d80d2db0 0x7fb9d80b3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:20.194 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.173+0000 7fb9d6a00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fb9d80d2db0 0x7fb9d80b3b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.96:39804/0 (socket says 172.21.15.96:39804) 2024-05-28T06:04:20.194 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.173+0000 7fb9d6a00640 1 -- 172.21.15.96:0/2222913307 learned_addr learned my addr 172.21.15.96:0/2222913307 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:20.195 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.174+0000 7f2e88600640 1 -- 172.21.15.96:0/2738906583 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e800d0d80 msgr2=0x7f2e8015a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:20.195 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.174+0000 7fb9d7400640 1 --2- 172.21.15.96:0/2222913307 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb9d815ed20 0x7fb9d815f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:20.195 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.174+0000 7f2e88600640 1 --2- 172.21.15.96:0/2738906583 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e800d0d80 0x7f2e8015a840 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f2e700097a0 tx=0x7f2e7002a250 comp rx=0 tx=0).stop 2024-05-28T06:04:20.196 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.174+0000 7fb083800640 1 -- 172.21.15.96:0/408747247 shutdown_connections 2024-05-28T06:04:20.196 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.174+0000 7fb083800640 1 --2- 172.21.15.96:0/408747247 >> v2:172.21.15.96:3300/0 conn(0x7fb07c09aa30 0x7fb07c166a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:20.196 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.174+0000 7fb083800640 1 --2- 172.21.15.96:0/408747247 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb07c0c5500 0x7fb07c0c58e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:20.196 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.174+0000 7fb083800640 1 -- 172.21.15.96:0/408747247 >> 172.21.15.96:0/408747247 conn(0x7fb07c0b4020 msgr2=0x7fb07c0b4430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:20.196 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.174+0000 7fb083800640 1 -- 172.21.15.96:0/408747247 shutdown_connections 2024-05-28T06:04:20.196 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.174+0000 7fb9d6a00640 1 -- 172.21.15.96:0/2222913307 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb9d815ed20 msgr2=0x7fb9d815f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:20.196 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.174+0000 7fb9d6a00640 1 --2- 172.21.15.96:0/2222913307 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb9d815ed20 0x7fb9d815f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:20.196 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.174+0000 7fb9d6a00640 1 -- 172.21.15.96:0/2222913307 >> v1:172.21.15.123:6789/0 conn(0x7fb9d80d4d60 legacy=0x7fb9d80d3190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:20.196 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.174+0000 7fb9d6a00640 1 -- 172.21.15.96:0/2222913307 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9d8162a20 con 0x7fb9d80d2db0 2024-05-28T06:04:20.197 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.174+0000 7fa158c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa1540cfd50 0x7fa154154960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:20.197 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.174+0000 7fa158c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa1540d25d0 0x7fa1540cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:20.197 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.174+0000 7fa158c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa1540d4130 0x7fa1540cd720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:20.197 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.174+0000 7fa158c00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa1540d2a20 con 0x7fa1540d25d0 2024-05-28T06:04:20.197 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.174+0000 7fa158c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa1540d4a50 con 0x7fa1540cfd50 2024-05-28T06:04:20.197 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.174+0000 7fa158c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fa15415c590 con 0x7fa1540d4130 2024-05-28T06:04:20.197 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.174+0000 7fb083800640 1 -- 172.21.15.96:0/408747247 wait complete. 2024-05-28T06:04:20.198 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.173+0000 7fe6b4600640 1 -- 172.21.15.96:0/1450362462 shutdown_connections 2024-05-28T06:04:20.198 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.173+0000 7fe6b4600640 1 -- 172.21.15.96:0/1450362462 wait complete. 2024-05-28T06:04:20.198 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.175+0000 7f2e88600640 1 -- 172.21.15.96:0/2738906583 shutdown_connections 2024-05-28T06:04:20.198 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.175+0000 7f2e88600640 1 --2- 172.21.15.96:0/2738906583 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e800d0d80 0x7f2e8015a840 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:20.198 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.175+0000 7f2e88600640 1 -- 172.21.15.96:0/2738906583 >> 172.21.15.96:0/2738906583 conn(0x7f2e800d0260 msgr2=0x7f2e800d0670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:20.199 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.175+0000 7fa151e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa1540cfd50 0x7fa154154960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:20.199 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.175+0000 7fa151e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa1540cfd50 0x7fa154154960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.96:39806/0 (socket says 172.21.15.96:39806) 2024-05-28T06:04:20.199 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.175+0000 7fa151e00640 1 -- 172.21.15.96:0/2271613702 learned_addr learned my addr 172.21.15.96:0/2271613702 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:20.200 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.175+0000 7fa151400640 1 --2- 172.21.15.96:0/2271613702 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa1540d25d0 0x7fa1540cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:20.200 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.176+0000 7fa152800640 1 --2- 172.21.15.96:0/2271613702 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa1540d4130 0x7fa1540cd720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:20.200 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.176+0000 7fa151e00640 1 -- 172.21.15.96:0/2271613702 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa1540d4130 msgr2=0x7fa1540cd720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:20.200 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.176+0000 7fa151e00640 1 --2- 172.21.15.96:0/2271613702 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa1540d4130 0x7fa1540cd720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:20.200 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.176+0000 7fa151e00640 1 -- 172.21.15.96:0/2271613702 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa1540d25d0 msgr2=0x7fa1540cd1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:20.200 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.176+0000 7fa151e00640 1 --2- 172.21.15.96:0/2271613702 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa1540d25d0 0x7fa1540cd1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:20.200 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.176+0000 7fa151e00640 1 -- 172.21.15.96:0/2271613702 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1540c8120 con 0x7fa1540cfd50 2024-05-28T06:04:20.200 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.176+0000 7fa152800640 1 --2- 172.21.15.96:0/2271613702 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa1540d4130 0x7fa1540cd720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:04:20.200 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.176+0000 7fa151400640 1 --2- 172.21.15.96:0/2271613702 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa1540d25d0 0x7fa1540cd1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-28T06:04:20.200 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.176+0000 7fa151e00640 1 --2- 172.21.15.96:0/2271613702 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa1540cfd50 0x7fa154154960 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa14002a710 tx=0x7fa1400046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:20.200 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.177+0000 7fa14a000640 1 -- 172.21.15.96:0/2271613702 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa140037050 con 0x7fa1540cfd50 2024-05-28T06:04:20.200 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.177+0000 7fa14a000640 1 -- 172.21.15.96:0/2271613702 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa140038040 con 0x7fa1540cfd50 2024-05-28T06:04:20.201 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.177+0000 7fa158c00640 1 -- 172.21.15.96:0/2271613702 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1540c8400 con 0x7fa1540cfd50 2024-05-28T06:04:20.201 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.178+0000 7fa14a000640 1 -- 172.21.15.96:0/2271613702 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa14003cb70 con 0x7fa1540cfd50 2024-05-28T06:04:20.201 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.178+0000 7fa158c00640 1 -- 172.21.15.96:0/2271613702 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1540c89c0 con 0x7fa1540cfd50 2024-05-28T06:04:20.201 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.176+0000 7fe6b4600640 1 Processor -- start 2024-05-28T06:04:20.201 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.178+0000 7fe6b4600640 1 -- start start 2024-05-28T06:04:20.201 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.177+0000 7fb083800640 1 Processor -- start 2024-05-28T06:04:20.201 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.177+0000 7fb083800640 1 -- start start 2024-05-28T06:04:20.202 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.177+0000 7f3d0ee00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3d080cfa40 0x7f3d080b1a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:20.202 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.177+0000 7f3d0ee00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d080d14d0 0x7f3d080b1fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:20.202 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.177+0000 7f3d0ee00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3d0815e870 0x7f3d08158b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:20.202 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.177+0000 7f3d0ee00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f3d080d4fb0 con 0x7f3d080cfa40 2024-05-28T06:04:20.202 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.177+0000 7f3d0ee00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f3d080d0820 con 0x7f3d080d14d0 2024-05-28T06:04:20.202 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.177+0000 7f3d0ee00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f3d080d2cf0 con 0x7f3d0815e870 2024-05-28T06:04:20.202 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.177+0000 7f3d0da00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3d080cfa40 0x7f3d080b1a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:20.202 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.178+0000 7f3d0da00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3d080cfa40 0x7f3d080b1a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48044/0 (socket says 172.21.15.96:48044) 2024-05-28T06:04:20.202 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.178+0000 7f3d0da00640 1 -- 172.21.15.96:0/1396976690 learned_addr learned my addr 172.21.15.96:0/1396976690 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:20.202 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.178+0000 7f3d0e400640 1 --2- 172.21.15.96:0/1396976690 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3d0815e870 0x7f3d08158b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:20.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.175+0000 7f2e88600640 1 -- 172.21.15.96:0/2738906583 shutdown_connections 2024-05-28T06:04:20.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.176+0000 7fb9d6a00640 1 --2- 172.21.15.96:0/2222913307 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb9d80d2db0 0x7fb9d80b3b50 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb9c8009870 tx=0x7fb9c802a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e36f63b7be3d74f8 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:20.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.176+0000 7fb9d6000640 1 -- 172.21.15.96:0/2222913307 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9c8037070 con 0x7fb9d80d2db0 2024-05-28T06:04:20.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.176+0000 7fb9d6000640 1 -- 172.21.15.96:0/2222913307 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9c8032440 con 0x7fb9d80d2db0 2024-05-28T06:04:20.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.176+0000 7fb9d6000640 1 -- 172.21.15.96:0/2222913307 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9c8032810 con 0x7fb9d80d2db0 2024-05-28T06:04:20.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.177+0000 7fb9dd600640 1 -- 172.21.15.96:0/2222913307 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb9d80d2db0 msgr2=0x7fb9d80b3b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:20.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.177+0000 7fb9dd600640 1 --2- 172.21.15.96:0/2222913307 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb9d80d2db0 0x7fb9d80b3b50 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb9c8009870 tx=0x7fb9c802a180 comp rx=0 tx=0).stop 2024-05-28T06:04:20.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.175+0000 7f2e88600640 1 -- 172.21.15.96:0/2738906583 wait complete. 2024-05-28T06:04:20.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.178+0000 7f2e88600640 1 Processor -- start 2024-05-28T06:04:20.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.179+0000 7f3d0d000640 1 --2- 172.21.15.96:0/1396976690 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d080d14d0 0x7f3d080b1fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:20.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.179+0000 7f86a7200640 1 Processor -- start 2024-05-28T06:04:20.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.179+0000 7f2e88600640 1 -- start start 2024-05-28T06:04:20.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.179+0000 7fb9dd600640 1 -- 172.21.15.96:0/2222913307 shutdown_connections 2024-05-28T06:04:20.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.179+0000 7fb9dd600640 1 --2- 172.21.15.96:0/2222913307 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb9d815ed20 0x7fb9d815f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:20.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.179+0000 7fb9dd600640 1 --2- 172.21.15.96:0/2222913307 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb9d80d2db0 0x7fb9d80b3b50 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:20.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.179+0000 7fb9dd600640 1 -- 172.21.15.96:0/2222913307 >> 172.21.15.96:0/2222913307 conn(0x7fb9d808bfc0 msgr2=0x7fb9d80d4650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:20.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.179+0000 7f3d0da00640 1 -- 172.21.15.96:0/1396976690 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3d0815e870 msgr2=0x7f3d08158b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:20.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.179+0000 7f3d0da00640 1 --2- 172.21.15.96:0/1396976690 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3d0815e870 0x7f3d08158b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:20.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.179+0000 7f3d0da00640 1 -- 172.21.15.96:0/1396976690 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d080d14d0 msgr2=0x7f3d080b1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:20.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.179+0000 7f3d0da00640 1 --2- 172.21.15.96:0/1396976690 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d080d14d0 0x7f3d080b1fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:20.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.179+0000 7f3d0da00640 1 -- 172.21.15.96:0/1396976690 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d080b3d20 con 0x7f3d080cfa40 2024-05-28T06:04:20.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.179+0000 7f3d0d000640 1 --2- 172.21.15.96:0/1396976690 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d080d14d0 0x7f3d080b1fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:04:20.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.180+0000 7fb083800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb07c09aa30 0x7fb07c0cfe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:20.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.180+0000 7fe6b4600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe6ac06f6e0 0x7fe6ac0c3d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:20.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.180+0000 7f3d0da00640 1 --2- 172.21.15.96:0/1396976690 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3d080cfa40 0x7f3d080b1a70 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f3cfc0098c0 tx=0x7f3cfc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:20.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.180+0000 7fe6b4600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe6ac0d04a0 0x7fe6ac202310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:20.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.180+0000 7fb083800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb07c09ae90 0x7fb07c0d03a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:20.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.181+0000 7fe6b4600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe6ac162e70 0x7fe6ac0aecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:20.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.181+0000 7fb083800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb07c0c5500 0x7fb07c1f8bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:20.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.181+0000 7f3d05600640 1 -- 172.21.15.96:0/1396976690 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cfc037050 con 0x7f3d080cfa40 2024-05-28T06:04:20.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.181+0000 7fe6b2600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe6ac162e70 0x7fe6ac0aecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:20.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.181+0000 7fb080e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb07c09aa30 0x7fb07c0cfe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:20.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.181+0000 7f2e88600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2e80073020 0x7f2e8006f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:20.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.181+0000 7fa13f400640 1 -- 172.21.15.96:0/2271613702 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa128005290 con 0x7fa1540cfd50 2024-05-28T06:04:20.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.181+0000 7fe6b1c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe6ac06f6e0 0x7fe6ac0c3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:20.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.181+0000 7fe6b1c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe6ac06f6e0 0x7fe6ac0c3d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.96:39820/0 (socket says 172.21.15.96:39820) 2024-05-28T06:04:20.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.181+0000 7fe6b1c00640 1 -- 172.21.15.96:0/2577148256 learned_addr learned my addr 172.21.15.96:0/2577148256 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:20.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.181+0000 7fe6b4600640 1 -- 172.21.15.96:0/2577148256 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe6ac0d0dc0 con 0x7fe6ac162e70 2024-05-28T06:04:20.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.181+0000 7fe6b4600640 1 -- 172.21.15.96:0/2577148256 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe6ac0b3b50 con 0x7fe6ac06f6e0 2024-05-28T06:04:20.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.181+0000 7fe6b4600640 1 -- 172.21.15.96:0/2577148256 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fe6ac0ce770 con 0x7fe6ac0d04a0 2024-05-28T06:04:20.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.181+0000 7f3d0ee00640 1 -- 172.21.15.96:0/1396976690 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d080b4000 con 0x7f3d080cfa40 2024-05-28T06:04:20.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.181+0000 7f3d0ee00640 1 -- 172.21.15.96:0/1396976690 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d080b45c0 con 0x7f3d080cfa40 2024-05-28T06:04:20.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.181+0000 7f3d05600640 1 -- 172.21.15.96:0/1396976690 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3cfc038040 con 0x7f3d080cfa40 2024-05-28T06:04:20.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.181+0000 7f2e88600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2e800cbd10 0x7f2e800c9770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:20.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.182+0000 7f2e88600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e800d0d80 0x7f2e801f7ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:20.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.182+0000 7f2e85c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2e80073020 0x7f2e8006f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:20.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.182+0000 7f2e85c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2e80073020 0x7f2e8006f3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.96:59986/0 (socket says 172.21.15.96:59986) 2024-05-28T06:04:20.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.182+0000 7f2e88600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f2e8015b150 con 0x7f2e800cbd10 2024-05-28T06:04:20.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.182+0000 7f2e88600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f2e8015afd0 con 0x7f2e800d0d80 2024-05-28T06:04:20.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.182+0000 7f2e88600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f2e800ad130 con 0x7f2e80073020 2024-05-28T06:04:20.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.182+0000 7f2e85c00640 1 -- 172.21.15.96:0/1810321307 learned_addr learned my addr 172.21.15.96:0/1810321307 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:20.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.181+0000 7fb9dd600640 1 -- 172.21.15.96:0/2222913307 shutdown_connections 2024-05-28T06:04:20.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.181+0000 7fb9dd600640 1 -- 172.21.15.96:0/2222913307 wait complete. 2024-05-28T06:04:20.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.179+0000 7f86a7200640 1 -- start start 2024-05-28T06:04:20.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.181+0000 7f86a7200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f86a00d10a0 0x7f86a00b3620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:20.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.181+0000 7f86a7200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f86a0162210 0x7f86a00cf550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:20.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.181+0000 7f86a7200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86a00d2930 con 0x7f86a00d1480 2024-05-28T06:04:20.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.181+0000 7f86a7200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f86a00d2c10 con 0x7f86a00d10a0 2024-05-28T06:04:20.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.181+0000 7f86a7200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f86a00d2ef0 con 0x7f86a0162210 2024-05-28T06:04:20.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.182+0000 7fb080e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb07c09aa30 0x7fb07c0cfe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48046/0 (socket says 172.21.15.96:48046) 2024-05-28T06:04:20.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.182+0000 7fb080e00640 1 -- 172.21.15.96:0/1976078779 learned_addr learned my addr 172.21.15.96:0/1976078779 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:20.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.182+0000 7f8697e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f86a00d1480 0x7f86a0161b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:35230/0 (socket says 172.21.15.96:35230) 2024-05-28T06:04:20.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.182+0000 7f8697e00640 1 -- 172.21.15.96:0/914271507 learned_addr learned my addr 172.21.15.96:0/914271507 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:04:20.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.182+0000 7fa14a000640 1 -- 172.21.15.96:0/2271613702 <== mon.1 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa14003cd10 con 0x7fa1540cfd50 2024-05-28T06:04:20.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.183+0000 7fe6b2600640 1 -- 172.21.15.96:0/2577148256 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe6ac0d04a0 msgr2=0x7fe6ac202310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:20.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.182+0000 7fe6b1200640 1 --2- 172.21.15.96:0/2577148256 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe6ac0d04a0 0x7fe6ac202310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:20.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.183+0000 7fe6b2600640 1 --2- 172.21.15.96:0/2577148256 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe6ac0d04a0 0x7fe6ac202310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:20.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.183+0000 7fe6b2600640 1 -- 172.21.15.96:0/2577148256 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe6ac06f6e0 msgr2=0x7fe6ac0c3d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:20.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.183+0000 7fe6b2600640 1 --2- 172.21.15.96:0/2577148256 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe6ac06f6e0 0x7fe6ac0c3d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:20.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.183+0000 7fe6b2600640 1 -- 172.21.15.96:0/2577148256 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6ac0c4400 con 0x7fe6ac162e70 2024-05-28T06:04:20.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.183+0000 7fe6b1c00640 1 --2- 172.21.15.96:0/2577148256 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe6ac06f6e0 0x7fe6ac0c3d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:04:20.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.182+0000 7fb083800640 1 -- 172.21.15.96:0/1976078779 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb07c09b680 con 0x7fb07c09aa30 2024-05-28T06:04:20.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.183+0000 7fb083800640 1 -- 172.21.15.96:0/1976078779 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fb07c09b3a0 con 0x7fb07c0c5500 2024-05-28T06:04:20.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.183+0000 7fb07be00640 1 --2- 172.21.15.96:0/1976078779 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb07c09ae90 0x7fb07c0d03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:20.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.183+0000 7fb083800640 1 -- 172.21.15.96:0/1976078779 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fb07c09b960 con 0x7fb07c09ae90 2024-05-28T06:04:20.218 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.182+0000 7f2e85200640 1 --2- 172.21.15.96:0/1810321307 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2e800cbd10 0x7f2e800c9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:20.218 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.182+0000 7f2e86600640 1 --2- 172.21.15.96:0/1810321307 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e800d0d80 0x7f2e801f7ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:20.218 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.183+0000 7f2e85c00640 1 -- 172.21.15.96:0/1810321307 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e800d0d80 msgr2=0x7f2e801f7ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:20.218 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.183+0000 7f2e85c00640 1 --2- 172.21.15.96:0/1810321307 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e800d0d80 0x7f2e801f7ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:20.218 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.183+0000 7f2e85c00640 1 -- 172.21.15.96:0/1810321307 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2e800cbd10 msgr2=0x7f2e800c9770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:20.218 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.183+0000 7f2e85c00640 1 --2- 172.21.15.96:0/1810321307 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2e800cbd10 0x7f2e800c9770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:20.218 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.183+0000 7f2e85c00640 1 -- 172.21.15.96:0/1810321307 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e800b1230 con 0x7f2e80073020 2024-05-28T06:04:20.218 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.183+0000 7f2e85200640 1 --2- 172.21.15.96:0/1810321307 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2e800cbd10 0x7f2e800c9770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-28T06:04:20.218 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.183+0000 7f2e85c00640 1 --2- 172.21.15.96:0/1810321307 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2e80073020 0x7f2e8006f3a0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2e70035040 tx=0x7f2e700046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:20.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.182+0000 7f869fe00640 1 --2- 172.21.15.96:0/914271507 >> v2:172.21.15.96:3300/0 conn(0x7f86a00d10a0 0x7f86a00b3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:20.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.182+0000 7f86a5200640 1 --2- 172.21.15.96:0/914271507 >> v2:172.21.15.123:3300/0 conn(0x7f86a0162210 0x7f86a00cf550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:20.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.183+0000 7f869fe00640 1 -- 172.21.15.96:0/914271507 >> v2:172.21.15.123:3300/0 conn(0x7f86a0162210 msgr2=0x7f86a00cf550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:20.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.183+0000 7f869fe00640 1 --2- 172.21.15.96:0/914271507 >> v2:172.21.15.123:3300/0 conn(0x7f86a0162210 0x7f86a00cf550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:20.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.183+0000 7f869fe00640 1 -- 172.21.15.96:0/914271507 >> v1:172.21.15.96:6789/0 conn(0x7f86a00d1480 legacy=0x7f86a0161b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:20.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.183+0000 7f869fe00640 1 -- 172.21.15.96:0/914271507 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86a00d31d0 con 0x7f86a00d10a0 2024-05-28T06:04:20.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.183+0000 7fe6b1200640 1 --2- 172.21.15.96:0/2577148256 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe6ac0d04a0 0x7fe6ac202310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:04:20.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.184+0000 7fe6b2600640 1 --2- 172.21.15.96:0/2577148256 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe6ac162e70 0x7fe6ac0aecd0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe6ac0b5390 tx=0x7fe6a4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:20.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.184+0000 7f2e7d600640 1 -- 172.21.15.96:0/1810321307 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e70041030 con 0x7f2e80073020 2024-05-28T06:04:20.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.184+0000 7f2e7d600640 1 -- 172.21.15.96:0/1810321307 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e70032e10 con 0x7f2e80073020 2024-05-28T06:04:20.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.184+0000 7f2e7d600640 1 -- 172.21.15.96:0/1810321307 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e70037040 con 0x7f2e80073020 2024-05-28T06:04:20.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.185+0000 7fb081800640 1 --2- 172.21.15.96:0/1976078779 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb07c0c5500 0x7fb07c1f8bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:20.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.185+0000 7fa14a000640 1 --2- 172.21.15.96:0/2271613702 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa13406c940 0x7fa13406ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:20.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.185+0000 7fa14a000640 1 -- 172.21.15.96:0/2271613702 <== mon.1 v2:172.21.15.123:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fa1400ac550 con 0x7fa1540cfd50 2024-05-28T06:04:20.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.185+0000 7f869fe00640 1 --2- 172.21.15.96:0/914271507 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f86a00d10a0 0x7f86a00b3620 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f86a00d4640 tx=0x7f868802a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d24f0c7dbe417ea3 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:20.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.184+0000 7f2e88600640 1 -- 172.21.15.96:0/1810321307 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e80201ab0 con 0x7f2e80073020 2024-05-28T06:04:20.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.184+0000 7f2e88600640 1 -- 172.21.15.96:0/1810321307 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e80202070 con 0x7f2e80073020 2024-05-28T06:04:20.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.186+0000 7fb07be00640 1 -- 172.21.15.96:0/1976078779 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb07c0c5500 msgr2=0x7fb07c1f8bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:20.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.186+0000 7fb07be00640 1 --2- 172.21.15.96:0/1976078779 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb07c0c5500 0x7fb07c1f8bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:20.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.182+0000 7f3d05600640 1 -- 172.21.15.96:0/1396976690 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cfc03cb70 con 0x7f3d080cfa40 2024-05-28T06:04:20.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.186+0000 7f869f400640 1 -- 172.21.15.96:0/914271507 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8688037070 con 0x7f86a00d10a0 2024-05-28T06:04:20.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.186+0000 7f869f400640 1 -- 172.21.15.96:0/914271507 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8688032440 con 0x7f86a00d10a0 2024-05-28T06:04:20.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.186+0000 7f869f400640 1 -- 172.21.15.96:0/914271507 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8688032810 con 0x7f86a00d10a0 2024-05-28T06:04:20.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.187+0000 7fb07be00640 1 -- 172.21.15.96:0/1976078779 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb07c09aa30 msgr2=0x7fb07c0cfe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:20.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.187+0000 7f3d0ee00640 1 -- 172.21.15.96:0/1396976690 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ce4005290 con 0x7f3d080cfa40 2024-05-28T06:04:20.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.187+0000 7f2e88600640 1 -- 172.21.15.96:0/1810321307 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e800689e0 con 0x7f2e80073020 2024-05-28T06:04:20.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.184+0000 7fe6a9600640 1 -- 172.21.15.96:0/2577148256 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6a4036050 con 0x7fe6ac162e70 2024-05-28T06:04:20.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.185+0000 7fe6b4600640 1 -- 172.21.15.96:0/2577148256 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6ac0c46e0 con 0x7fe6ac162e70 2024-05-28T06:04:20.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.185+0000 7fe6b4600640 1 -- 172.21.15.96:0/2577148256 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6ac0cd9e0 con 0x7fe6ac162e70 2024-05-28T06:04:20.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.185+0000 7fe6a9600640 1 -- 172.21.15.96:0/2577148256 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6a4042070 con 0x7fe6ac162e70 2024-05-28T06:04:20.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.185+0000 7fe6a9600640 1 -- 172.21.15.96:0/2577148256 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6a4036050 con 0x7fe6ac162e70 2024-05-28T06:04:20.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.187+0000 7fb07be00640 1 --2- 172.21.15.96:0/1976078779 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb07c09aa30 0x7fb07c0cfe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:20.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.187+0000 7f86a7200640 1 -- 172.21.15.96:0/914271507 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f86a00d10a0 msgr2=0x7f86a00b3620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:20.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.187+0000 7f86a7200640 1 --2- 172.21.15.96:0/914271507 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f86a00d10a0 0x7f86a00b3620 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f86a00d4640 tx=0x7f868802a250 comp rx=0 tx=0).stop 2024-05-28T06:04:20.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.187+0000 7fa151400640 1 --2- 172.21.15.96:0/2271613702 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa13406c940 0x7fa13406ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:20.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.185+0000 7fb9dd600640 1 Processor -- start 2024-05-28T06:04:20.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.185+0000 7fb9dd600640 1 -- start start 2024-05-28T06:04:20.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.187+0000 7fb9dd600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb9d80d2db0 0x7fb9d80cd0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:20.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.187+0000 7fb9dd600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb9d80d4d60 0x7fb9d80b9420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:20.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.188+0000 7fb07be00640 1 -- 172.21.15.96:0/1976078779 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb07c0cf2b0 con 0x7fb07c09ae90 2024-05-28T06:04:20.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.188+0000 7f2e7d600640 1 -- 172.21.15.96:0/1810321307 <== mon.2 v2:172.21.15.123:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2e70044070 con 0x7f2e80073020 2024-05-28T06:04:20.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.188+0000 7fb081800640 1 --2- 172.21.15.96:0/1976078779 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb07c0c5500 0x7fb07c1f8bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-28T06:04:20.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.188+0000 7fe6b4600640 1 -- 172.21.15.96:0/2577148256 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe688005290 con 0x7fe6ac162e70 2024-05-28T06:04:20.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.188+0000 7fb9dd600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb9d815ed20 0x7fb9d80b9960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:20.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.188+0000 7f86a7200640 1 -- 172.21.15.96:0/914271507 shutdown_connections 2024-05-28T06:04:20.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.188+0000 7f86a7200640 1 --2- 172.21.15.96:0/914271507 >> v2:172.21.15.123:3300/0 conn(0x7f86a0162210 0x7f86a00cf550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:20.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.188+0000 7f86a7200640 1 --2- 172.21.15.96:0/914271507 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f86a00d10a0 0x7f86a00b3620 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:20.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.188+0000 7f86a7200640 1 -- 172.21.15.96:0/914271507 >> 172.21.15.96:0/914271507 conn(0x7f86a008bfc0 msgr2=0x7f86a00d4670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:20.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.189+0000 7f86a7200640 1 -- 172.21.15.96:0/914271507 shutdown_connections 2024-05-28T06:04:20.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.189+0000 7f86a7200640 1 -- 172.21.15.96:0/914271507 wait complete. 2024-05-28T06:04:20.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.189+0000 7fa151400640 1 --2- 172.21.15.96:0/2271613702 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa13406c940 0x7fa13406ee00 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fa154161230 tx=0x7fa144008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:20.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.189+0000 7f3d05600640 1 -- 172.21.15.96:0/1396976690 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3cfc03cd10 con 0x7f3d080cfa40 2024-05-28T06:04:20.231 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.189+0000 7fb9d7400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb9d815ed20 0x7fb9d80b9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:20.231 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.189+0000 7fb9d7400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb9d815ed20 0x7fb9d80b9960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48096/0 (socket says 172.21.15.96:48096) 2024-05-28T06:04:20.231 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.189+0000 7fb07be00640 1 --2- 172.21.15.96:0/1976078779 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb07c09ae90 0x7fb07c0d03a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb06800d520 tx=0x7fb06800d9e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:20.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.189+0000 7fb9ce000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb9d80d4d60 0x7fb9d80b9420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:20.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.189+0000 7fb9ce000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb9d80d4d60 0x7fb9d80b9420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.96:59996/0 (socket says 172.21.15.96:59996) 2024-05-28T06:04:20.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.190+0000 7fb9d7400640 1 -- 172.21.15.96:0/4276302645 learned_addr learned my addr 172.21.15.96:0/4276302645 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:20.233 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.190+0000 7fb9dd600640 1 -- 172.21.15.96:0/4276302645 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb9d80d51a0 con 0x7fb9d815ed20 2024-05-28T06:04:20.233 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.190+0000 7fb9dd600640 1 -- 172.21.15.96:0/4276302645 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fb9d8162fe0 con 0x7fb9d80d2db0 2024-05-28T06:04:20.233 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.190+0000 7fb9dd600640 1 -- 172.21.15.96:0/4276302645 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fb9d80d5320 con 0x7fb9d80d4d60 2024-05-28T06:04:20.233 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.189+0000 7f2e7d600640 1 --2- 172.21.15.96:0/1810321307 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2e6406c8f0 0x7f2e6406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:20.233 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.190+0000 7f2e7d600640 1 -- 172.21.15.96:0/1810321307 <== mon.2 v2:172.21.15.123:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f2e700ac2a0 con 0x7f2e80073020 2024-05-28T06:04:20.233 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.191+0000 7f3d05600640 1 --2- 172.21.15.96:0/1396976690 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3ce806c8f0 0x7f3ce806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:20.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.191+0000 7f3d05600640 1 -- 172.21.15.96:0/1396976690 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f3cfc0ac620 con 0x7f3d080cfa40 2024-05-28T06:04:20.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.191+0000 7fe6a9600640 1 -- 172.21.15.96:0/2577148256 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe6a4004250 con 0x7fe6ac162e70 2024-05-28T06:04:20.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.191+0000 7fb9d6a00640 1 --2- 172.21.15.96:0/4276302645 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb9d80d2db0 0x7fb9d80cd0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:20.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.191+0000 7fb9d7400640 1 -- 172.21.15.96:0/4276302645 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb9d80d4d60 msgr2=0x7fb9d80b9420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:20.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.191+0000 7fb9d7400640 1 --2- 172.21.15.96:0/4276302645 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb9d80d4d60 0x7fb9d80b9420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:20.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.191+0000 7fb9d7400640 1 -- 172.21.15.96:0/4276302645 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb9d80d2db0 msgr2=0x7fb9d80cd0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:20.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.191+0000 7fb9d7400640 1 --2- 172.21.15.96:0/4276302645 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb9d80d2db0 0x7fb9d80cd0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:20.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.191+0000 7fb9d7400640 1 -- 172.21.15.96:0/4276302645 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9d80b1880 con 0x7fb9d815ed20 2024-05-28T06:04:20.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.192+0000 7fb9ce000640 1 --2- 172.21.15.96:0/4276302645 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb9d80d4d60 0x7fb9d80b9420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello state changed while learned_addr, mark_down or replacing must be happened just now 2024-05-28T06:04:20.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.192+0000 7fb9d6a00640 1 --2- 172.21.15.96:0/4276302645 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb9d80d2db0 0x7fb9d80cd0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:04:20.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.191+0000 7f2e85200640 1 --2- 172.21.15.96:0/1810321307 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2e6406c8f0 0x7f2e6406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:20.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.192+0000 7f86a7200640 1 Processor -- start 2024-05-28T06:04:20.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.192+0000 7fb9d7400640 1 --2- 172.21.15.96:0/4276302645 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb9d815ed20 0x7fb9d80b9960 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fb9d0007e90 tx=0x7fb9d000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:20.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.192+0000 7f3d0d000640 1 --2- 172.21.15.96:0/1396976690 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3ce806c8f0 0x7f3ce806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:20.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.193+0000 7fe6a9600640 1 --2- 172.21.15.96:0/2577148256 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe68c06cb20 0x7fe68c06efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:20.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.192+0000 7f86a7200640 1 -- start start 2024-05-28T06:04:20.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.193+0000 7fe6a9600640 1 -- 172.21.15.96:0/2577148256 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fe6a40abe40 con 0x7fe6ac162e70 2024-05-28T06:04:20.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.193+0000 7fe6b1c00640 1 --2- 172.21.15.96:0/2577148256 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe68c06cb20 0x7fe68c06efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:20.238 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.193+0000 7f2e85200640 1 --2- 172.21.15.96:0/1810321307 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2e6406c8f0 0x7f2e6406edb0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f2e801ea7b0 tx=0x7f2e74007d30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:20.238 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.193+0000 7f3d0d000640 1 --2- 172.21.15.96:0/1396976690 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3ce806c8f0 0x7f3ce806edb0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3d080b27a0 tx=0x7f3cf4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:20.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.194+0000 7f86a7200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f86a00d10a0 0x7f86a00cc150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:20.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.194+0000 7f86a7200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f86a00d1480 0x7f86a0156de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:20.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.194+0000 7f86a7200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f86a0162210 0x7f86a00ca710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:20.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.194+0000 7f86a7200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f86a00d26f0 con 0x7f86a0162210 2024-05-28T06:04:20.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.194+0000 7f86a7200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f86a00d2570 con 0x7f86a00d1480 2024-05-28T06:04:20.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.194+0000 7f86a7200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f86a00b3b60 con 0x7f86a00d10a0 2024-05-28T06:04:20.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.194+0000 7fe6b1c00640 1 --2- 172.21.15.96:0/2577148256 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe68c06cb20 0x7fe68c06efe0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe6a0009b90 tx=0x7fe6a0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:20.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.194+0000 7f86a5200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f86a0162210 0x7f86a00ca710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:20.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.194+0000 7f86a5200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f86a0162210 0x7f86a00ca710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48110/0 (socket says 172.21.15.96:48110) 2024-05-28T06:04:20.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.194+0000 7f86a5200640 1 -- 172.21.15.96:0/90192308 learned_addr learned my addr 172.21.15.96:0/90192308 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:20.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.195+0000 7f869fe00640 1 --2- 172.21.15.96:0/90192308 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f86a00d10a0 0x7f86a00cc150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:20.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.195+0000 7f8697e00640 1 --2- 172.21.15.96:0/90192308 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f86a00d1480 0x7f86a0156de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:20.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.195+0000 7f86a5200640 1 -- 172.21.15.96:0/90192308 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f86a00d10a0 msgr2=0x7f86a00cc150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:20.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.195+0000 7f86a5200640 1 --2- 172.21.15.96:0/90192308 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f86a00d10a0 0x7f86a00cc150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:20.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.195+0000 7f86a5200640 1 -- 172.21.15.96:0/90192308 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f86a00d1480 msgr2=0x7f86a0156de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:20.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.195+0000 7f86a5200640 1 --2- 172.21.15.96:0/90192308 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f86a00d1480 0x7f86a0156de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:20.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.195+0000 7f86a5200640 1 -- 172.21.15.96:0/90192308 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86a00cac50 con 0x7f86a0162210 2024-05-28T06:04:20.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.195+0000 7f86a5200640 1 --2- 172.21.15.96:0/90192308 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f86a0162210 0x7f86a00ca710 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f869800efc0 tx=0x7f869800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:20.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.198+0000 7f3d05600640 1 -- 172.21.15.96:0/1396976690 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3cfc07a930 con 0x7f3d080cfa40 2024-05-28T06:04:20.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.199+0000 7fa14a000640 1 -- 172.21.15.96:0/2271613702 <== mon.1 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa14007a9f0 con 0x7fa1540cfd50 2024-05-28T06:04:20.243 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.204+0000 7fb9cf400640 1 -- 172.21.15.96:0/4276302645 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9d0014420 con 0x7fb9d815ed20 2024-05-28T06:04:20.243 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.204+0000 7fb9cf400640 1 -- 172.21.15.96:0/4276302645 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9d0019070 con 0x7fb9d815ed20 2024-05-28T06:04:20.243 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.204+0000 7fb9cf400640 1 -- 172.21.15.96:0/4276302645 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9d0014800 con 0x7fb9d815ed20 2024-05-28T06:04:20.243 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.204+0000 7fb9dd600640 1 -- 172.21.15.96:0/4276302645 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9d80b75e0 con 0x7fb9d815ed20 2024-05-28T06:04:20.243 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.204+0000 7fb9dd600640 1 -- 172.21.15.96:0/4276302645 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9d80b7ba0 con 0x7fb9d815ed20 2024-05-28T06:04:20.244 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.204+0000 7fe6a9600640 1 -- 172.21.15.96:0/2577148256 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe6a404b020 con 0x7fe6ac162e70 2024-05-28T06:04:20.244 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.204+0000 7f869cc00640 1 -- 172.21.15.96:0/90192308 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8698018020 con 0x7f86a0162210 2024-05-28T06:04:20.244 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.204+0000 7f869cc00640 1 -- 172.21.15.96:0/90192308 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8698014480 con 0x7f86a0162210 2024-05-28T06:04:20.244 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.204+0000 7f869cc00640 1 -- 172.21.15.96:0/90192308 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8698014870 con 0x7f86a0162210 2024-05-28T06:04:20.244 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.204+0000 7f86a7200640 1 -- 172.21.15.96:0/90192308 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86a00cc7e0 con 0x7f86a0162210 2024-05-28T06:04:20.245 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.205+0000 7fb078c00640 1 -- 172.21.15.96:0/1976078779 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0680125f0 con 0x7fb07c09ae90 2024-05-28T06:04:20.245 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.205+0000 7fb078c00640 1 -- 172.21.15.96:0/1976078779 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb068019070 con 0x7fb07c09ae90 2024-05-28T06:04:20.245 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.205+0000 7fb078c00640 1 -- 172.21.15.96:0/1976078779 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0680129b0 con 0x7fb07c09ae90 2024-05-28T06:04:20.245 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.205+0000 7f2e7d600640 1 -- 172.21.15.96:0/1810321307 <== mon.2 v2:172.21.15.123:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2e7007a7e0 con 0x7f2e80073020 2024-05-28T06:04:20.245 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.206+0000 7fb9dd600640 1 -- 172.21.15.96:0/4276302645 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9d80cffa0 con 0x7fb9d815ed20 2024-05-28T06:04:20.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.206+0000 7f86a7200640 1 -- 172.21.15.96:0/90192308 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86a00ccda0 con 0x7f86a0162210 2024-05-28T06:04:20.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.208+0000 7f869cc00640 1 -- 172.21.15.96:0/90192308 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f869801b020 con 0x7f86a0162210 2024-05-28T06:04:20.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.206+0000 7fb083800640 1 -- 172.21.15.96:0/1976078779 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb07c1f9fc0 con 0x7fb07c09ae90 2024-05-28T06:04:20.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.206+0000 7fb083800640 1 -- 172.21.15.96:0/1976078779 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb07c1fa580 con 0x7fb07c09ae90 2024-05-28T06:04:20.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.209+0000 7fb9cf400640 1 -- 172.21.15.96:0/4276302645 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb9d00149a0 con 0x7fb9d815ed20 2024-05-28T06:04:20.247 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.209+0000 7f86a7200640 1 -- 172.21.15.96:0/90192308 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86a00690f0 con 0x7f86a0162210 2024-05-28T06:04:20.247 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.209+0000 7fb083800640 1 -- 172.21.15.96:0/1976078779 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb07c0690f0 con 0x7fb07c09ae90 2024-05-28T06:04:20.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.210+0000 7f869cc00640 1 --2- 172.21.15.96:0/90192308 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f867c06c940 0x7f867c06ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:20.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.210+0000 7f869cc00640 1 -- 172.21.15.96:0/90192308 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f869808ddc0 con 0x7f86a0162210 2024-05-28T06:04:20.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.211+0000 7fb9cf400640 1 --2- 172.21.15.96:0/4276302645 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb9b406c9f0 0x7fb9b406eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:20.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.211+0000 7fb9cf400640 1 -- 172.21.15.96:0/4276302645 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fb9d008c3a0 con 0x7fb9d815ed20 2024-05-28T06:04:20.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.212+0000 7fb9d6a00640 1 --2- 172.21.15.96:0/4276302645 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb9b406c9f0 0x7fb9b406eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:20.249 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.213+0000 7f869fe00640 1 --2- 172.21.15.96:0/90192308 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f867c06c940 0x7f867c06ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:20.249 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.214+0000 7f869fe00640 1 --2- 172.21.15.96:0/90192308 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f867c06c940 0x7f867c06ee00 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f8688034000 tx=0x7f868802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:20.249 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.215+0000 7fb9d6a00640 1 --2- 172.21.15.96:0/4276302645 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb9b406c9f0 0x7fb9b406eeb0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fb9c8034000 tx=0x7fb9c802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:20.250 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.215+0000 7fb078c00640 1 -- 172.21.15.96:0/1976078779 <== mon.2 v2:172.21.15.123:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb068012b50 con 0x7fb07c09ae90 2024-05-28T06:04:20.250 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.216+0000 7fb078c00640 1 --2- 172.21.15.96:0/1976078779 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb05c06c9c0 0x7fb05c06ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:20.250 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.217+0000 7fb078c00640 1 -- 172.21.15.96:0/1976078779 <== mon.2 v2:172.21.15.123:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fb06808bdd0 con 0x7fb07c09ae90 2024-05-28T06:04:20.250 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.218+0000 7fb080e00640 1 --2- 172.21.15.96:0/1976078779 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb05c06c9c0 0x7fb05c06ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:20.250 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.219+0000 7fb080e00640 1 --2- 172.21.15.96:0/1976078779 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb05c06c9c0 0x7fb05c06ee80 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fb0700098a0 tx=0x7fb07002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:20.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.220+0000 7fb9cf400640 1 -- 172.21.15.96:0/4276302645 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb9d80cffa0 con 0x7fb9d815ed20 2024-05-28T06:04:20.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.229+0000 7fb078c00640 1 -- 172.21.15.96:0/1976078779 <== mon.2 v2:172.21.15.123:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb06808f030 con 0x7fb07c09ae90 2024-05-28T06:04:20.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.230+0000 7f869cc00640 1 -- 172.21.15.96:0/90192308 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8698010040 con 0x7f86a0162210 2024-05-28T06:04:20.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.575+0000 7ff337e00640 1 -- 172.21.15.96:0/230603747 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7ff310005850 con 0x7ff338073120 2024-05-28T06:04:20.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.576+0000 7ff32ea00640 1 -- 172.21.15.96:0/230603747 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7ff3300598f0 con 0x7ff338073120 2024-05-28T06:04:20.577 INFO:teuthology.orchestra.run.smithi096.stdout:55834574855 2024-05-28T06:04:20.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.579+0000 7ff337e00640 1 -- 172.21.15.96:0/230603747 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff31806caf0 msgr2=0x7ff31806efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:20.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.580+0000 7ff337e00640 1 --2- 172.21.15.96:0/230603747 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff31806caf0 0x7ff31806efb0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff3381551a0 tx=0x7ff328009040 comp rx=0 tx=0).stop 2024-05-28T06:04:20.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.580+0000 7ff337e00640 1 -- 172.21.15.96:0/230603747 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff338073120 msgr2=0x7ff338152640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:20.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.580+0000 7ff337e00640 1 --2- 172.21.15.96:0/230603747 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff338073120 0x7ff338152640 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7ff33000bd70 tx=0x7ff33000be60 comp rx=0 tx=0).stop 2024-05-28T06:04:20.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.581+0000 7ff337e00640 1 -- 172.21.15.96:0/230603747 shutdown_connections 2024-05-28T06:04:20.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.581+0000 7ff337e00640 1 --2- 172.21.15.96:0/230603747 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff31806caf0 0x7ff31806efb0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:20.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.581+0000 7ff337e00640 1 --2- 172.21.15.96:0/230603747 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff3380b3fe0 0x7ff3380cedb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:20.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.581+0000 7ff337e00640 1 --2- 172.21.15.96:0/230603747 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff338073500 0x7ff338153cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:20.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.581+0000 7ff337e00640 1 --2- 172.21.15.96:0/230603747 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff338073120 0x7ff338152640 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:20.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.581+0000 7ff337e00640 1 -- 172.21.15.96:0/230603747 >> 172.21.15.96:0/230603747 conn(0x7ff33806c380 msgr2=0x7ff33806db60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:20.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.582+0000 7ff337e00640 1 -- 172.21.15.96:0/230603747 shutdown_connections 2024-05-28T06:04:20.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.582+0000 7ff337e00640 1 -- 172.21.15.96:0/230603747 wait complete. 2024-05-28T06:04:20.610 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.1 2024-05-28T06:04:20.610 DEBUG:teuthology.parallel:result is None 2024-05-28T06:04:20.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.616+0000 7fe6b4600640 1 -- 172.21.15.96:0/2577148256 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fe688005850 con 0x7fe6ac162e70 2024-05-28T06:04:20.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.617+0000 7fe6a9600640 1 -- 172.21.15.96:0/2577148256 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fe6a407a300 con 0x7fe6ac162e70 2024-05-28T06:04:20.619 INFO:teuthology.orchestra.run.smithi096.stdout:55834574856 2024-05-28T06:04:20.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.619+0000 7f3d0ee00640 1 -- 172.21.15.96:0/1396976690 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f3ce4005580 con 0x7f3d080cfa40 2024-05-28T06:04:20.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.619+0000 7f3d05600640 1 -- 172.21.15.96:0/1396976690 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f3cfc07a230 con 0x7f3d080cfa40 2024-05-28T06:04:20.621 INFO:teuthology.orchestra.run.smithi096.stdout:55834574856 2024-05-28T06:04:20.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.620+0000 7fe6b4600640 1 -- 172.21.15.96:0/2577148256 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe68c06cb20 msgr2=0x7fe68c06efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:20.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.620+0000 7fe6b4600640 1 --2- 172.21.15.96:0/2577148256 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe68c06cb20 0x7fe68c06efe0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe6a0009b90 tx=0x7fe6a0008040 comp rx=0 tx=0).stop 2024-05-28T06:04:20.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.620+0000 7fe6b4600640 1 -- 172.21.15.96:0/2577148256 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe6ac162e70 msgr2=0x7fe6ac0aecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:20.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.620+0000 7fe6b4600640 1 --2- 172.21.15.96:0/2577148256 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe6ac162e70 0x7fe6ac0aecd0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe6ac0b5390 tx=0x7fe6a4004a40 comp rx=0 tx=0).stop 2024-05-28T06:04:20.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.621+0000 7fe6b4600640 1 -- 172.21.15.96:0/2577148256 shutdown_connections 2024-05-28T06:04:20.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.621+0000 7fe6b4600640 1 --2- 172.21.15.96:0/2577148256 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe68c06cb20 0x7fe68c06efe0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:20.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.621+0000 7fe6b4600640 1 --2- 172.21.15.96:0/2577148256 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe6ac162e70 0x7fe6ac0aecd0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:20.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.621+0000 7fe6b4600640 1 --2- 172.21.15.96:0/2577148256 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe6ac0d04a0 0x7fe6ac202310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:20.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.621+0000 7fe6b4600640 1 --2- 172.21.15.96:0/2577148256 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe6ac06f6e0 0x7fe6ac0c3d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:20.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.621+0000 7fe6b4600640 1 -- 172.21.15.96:0/2577148256 >> 172.21.15.96:0/2577148256 conn(0x7fe6ac06bae0 msgr2=0x7fe6ac071b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:20.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.621+0000 7fe6b4600640 1 -- 172.21.15.96:0/2577148256 shutdown_connections 2024-05-28T06:04:20.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.621+0000 7fe6b4600640 1 -- 172.21.15.96:0/2577148256 wait complete. 2024-05-28T06:04:20.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.622+0000 7f3d0ee00640 1 -- 172.21.15.96:0/1396976690 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3ce806c8f0 msgr2=0x7f3ce806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:20.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.622+0000 7f3d0ee00640 1 --2- 172.21.15.96:0/1396976690 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3ce806c8f0 0x7f3ce806edb0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3d080b27a0 tx=0x7f3cf4009040 comp rx=0 tx=0).stop 2024-05-28T06:04:20.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.623+0000 7f3d0ee00640 1 -- 172.21.15.96:0/1396976690 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3d080cfa40 msgr2=0x7f3d080b1a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:20.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.623+0000 7f3d0ee00640 1 --2- 172.21.15.96:0/1396976690 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3d080cfa40 0x7f3d080b1a70 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f3cfc0098c0 tx=0x7f3cfc0046c0 comp rx=0 tx=0).stop 2024-05-28T06:04:20.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.623+0000 7f3d0ee00640 1 -- 172.21.15.96:0/1396976690 shutdown_connections 2024-05-28T06:04:20.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.623+0000 7f3d0ee00640 1 --2- 172.21.15.96:0/1396976690 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3ce806c8f0 0x7f3ce806edb0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:20.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.623+0000 7f3d0ee00640 1 --2- 172.21.15.96:0/1396976690 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3d0815e870 0x7f3d08158b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:20.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.623+0000 7f3d0ee00640 1 --2- 172.21.15.96:0/1396976690 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d080d14d0 0x7f3d080b1fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:20.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.623+0000 7f3d0ee00640 1 --2- 172.21.15.96:0/1396976690 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3d080cfa40 0x7f3d080b1a70 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:20.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.623+0000 7f3d0ee00640 1 -- 172.21.15.96:0/1396976690 >> 172.21.15.96:0/1396976690 conn(0x7f3d0808bfc0 msgr2=0x7f3d080b2c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:20.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.623+0000 7f3d0ee00640 1 -- 172.21.15.96:0/1396976690 shutdown_connections 2024-05-28T06:04:20.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.624+0000 7f3d0ee00640 1 -- 172.21.15.96:0/1396976690 wait complete. 2024-05-28T06:04:20.641 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.6 2024-05-28T06:04:20.642 DEBUG:teuthology.parallel:result is None 2024-05-28T06:04:20.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.644+0000 7f2e88600640 1 -- 172.21.15.96:0/1810321307 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f2e800d0640 con 0x7f2e80073020 2024-05-28T06:04:20.646 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.3 2024-05-28T06:04:20.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.645+0000 7fb9dd600640 1 -- 172.21.15.96:0/4276302645 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7fb9d80b1a50 con 0x7fb9d815ed20 2024-05-28T06:04:20.647 DEBUG:teuthology.parallel:result is None 2024-05-28T06:04:20.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.645+0000 7fa13f400640 1 -- 172.21.15.96:0/2271613702 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fa128005580 con 0x7fa1540cfd50 2024-05-28T06:04:20.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.645+0000 7fb9cf400640 1 -- 172.21.15.96:0/4276302645 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fb9d005a7b0 con 0x7fb9d815ed20 2024-05-28T06:04:20.648 INFO:teuthology.orchestra.run.smithi096.stdout:55834574854 2024-05-28T06:04:20.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.645+0000 7f2e7d600640 1 -- 172.21.15.96:0/1810321307 <== mon.2 v2:172.21.15.123:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f2e7007a0e0 con 0x7f2e80073020 2024-05-28T06:04:20.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.646+0000 7fa14a000640 1 -- 172.21.15.96:0/2271613702 <== mon.1 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fa14007a2f0 con 0x7fa1540cfd50 2024-05-28T06:04:20.649 INFO:teuthology.orchestra.run.smithi096.stdout:55834574854 2024-05-28T06:04:20.649 INFO:teuthology.orchestra.run.smithi096.stdout:55834574855 2024-05-28T06:04:20.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.648+0000 7fb9dd600640 1 -- 172.21.15.96:0/4276302645 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb9b406c9f0 msgr2=0x7fb9b406eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:20.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.648+0000 7fb9dd600640 1 --2- 172.21.15.96:0/4276302645 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb9b406c9f0 0x7fb9b406eeb0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fb9c8034000 tx=0x7fb9c802e000 comp rx=0 tx=0).stop 2024-05-28T06:04:20.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.648+0000 7fb9dd600640 1 -- 172.21.15.96:0/4276302645 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb9d815ed20 msgr2=0x7fb9d80b9960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:20.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.648+0000 7fb9dd600640 1 --2- 172.21.15.96:0/4276302645 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb9d815ed20 0x7fb9d80b9960 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fb9d0007e90 tx=0x7fb9d000e4b0 comp rx=0 tx=0).stop 2024-05-28T06:04:20.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.648+0000 7fb9dd600640 1 -- 172.21.15.96:0/4276302645 shutdown_connections 2024-05-28T06:04:20.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.648+0000 7fb9dd600640 1 --2- 172.21.15.96:0/4276302645 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb9b406c9f0 0x7fb9b406eeb0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:20.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.648+0000 7fb9dd600640 1 --2- 172.21.15.96:0/4276302645 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb9d815ed20 0x7fb9d80b9960 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:20.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.648+0000 7fb9dd600640 1 --2- 172.21.15.96:0/4276302645 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb9d80d4d60 0x7fb9d80b9420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:20.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.648+0000 7fb9dd600640 1 --2- 172.21.15.96:0/4276302645 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb9d80d2db0 0x7fb9d80cd0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:20.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.648+0000 7fb9dd600640 1 -- 172.21.15.96:0/4276302645 >> 172.21.15.96:0/4276302645 conn(0x7fb9d808bfc0 msgr2=0x7fb9d80d0690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:20.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.648+0000 7fb9dd600640 1 -- 172.21.15.96:0/4276302645 shutdown_connections 2024-05-28T06:04:20.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.648+0000 7fb9dd600640 1 -- 172.21.15.96:0/4276302645 wait complete. 2024-05-28T06:04:20.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.649+0000 7f2e88600640 1 -- 172.21.15.96:0/1810321307 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2e6406c8f0 msgr2=0x7f2e6406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:20.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.649+0000 7f2e88600640 1 --2- 172.21.15.96:0/1810321307 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2e6406c8f0 0x7f2e6406edb0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f2e801ea7b0 tx=0x7f2e74007d30 comp rx=0 tx=0).stop 2024-05-28T06:04:20.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.649+0000 7f2e88600640 1 -- 172.21.15.96:0/1810321307 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2e80073020 msgr2=0x7f2e8006f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:20.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.649+0000 7f2e88600640 1 --2- 172.21.15.96:0/1810321307 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2e80073020 0x7f2e8006f3a0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2e70035040 tx=0x7f2e700046c0 comp rx=0 tx=0).stop 2024-05-28T06:04:20.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.649+0000 7f2e88600640 1 -- 172.21.15.96:0/1810321307 shutdown_connections 2024-05-28T06:04:20.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.649+0000 7f2e88600640 1 --2- 172.21.15.96:0/1810321307 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2e6406c8f0 0x7f2e6406edb0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:20.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.649+0000 7f2e88600640 1 --2- 172.21.15.96:0/1810321307 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e800d0d80 0x7f2e801f7ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:20.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.649+0000 7f2e88600640 1 --2- 172.21.15.96:0/1810321307 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2e800cbd10 0x7f2e800c9770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:20.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.649+0000 7f2e88600640 1 --2- 172.21.15.96:0/1810321307 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2e80073020 0x7f2e8006f3a0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:20.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.649+0000 7f2e88600640 1 -- 172.21.15.96:0/1810321307 >> 172.21.15.96:0/1810321307 conn(0x7f2e800d0260 msgr2=0x7f2e8006dcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:20.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.649+0000 7f2e88600640 1 -- 172.21.15.96:0/1810321307 shutdown_connections 2024-05-28T06:04:20.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.649+0000 7f2e88600640 1 -- 172.21.15.96:0/1810321307 wait complete. 2024-05-28T06:04:20.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.650+0000 7fa13f400640 1 -- 172.21.15.96:0/2271613702 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa13406c940 msgr2=0x7fa13406ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:20.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.650+0000 7fa13f400640 1 --2- 172.21.15.96:0/2271613702 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa13406c940 0x7fa13406ee00 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fa154161230 tx=0x7fa144008040 comp rx=0 tx=0).stop 2024-05-28T06:04:20.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.650+0000 7fa13f400640 1 -- 172.21.15.96:0/2271613702 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa1540cfd50 msgr2=0x7fa154154960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:20.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.650+0000 7fa13f400640 1 --2- 172.21.15.96:0/2271613702 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa1540cfd50 0x7fa154154960 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa14002a710 tx=0x7fa1400046c0 comp rx=0 tx=0).stop 2024-05-28T06:04:20.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.650+0000 7fa13f400640 1 -- 172.21.15.96:0/2271613702 shutdown_connections 2024-05-28T06:04:20.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.650+0000 7fa13f400640 1 --2- 172.21.15.96:0/2271613702 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa13406c940 0x7fa13406ee00 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:20.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.650+0000 7fa13f400640 1 --2- 172.21.15.96:0/2271613702 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa1540d4130 0x7fa1540cd720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:20.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.650+0000 7fa13f400640 1 --2- 172.21.15.96:0/2271613702 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa1540d25d0 0x7fa1540cd1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:20.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.650+0000 7fa13f400640 1 --2- 172.21.15.96:0/2271613702 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa1540cfd50 0x7fa154154960 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:20.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.650+0000 7fa13f400640 1 -- 172.21.15.96:0/2271613702 >> 172.21.15.96:0/2271613702 conn(0x7fa15408bfc0 msgr2=0x7fa1540c7270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:20.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.650+0000 7fa13f400640 1 -- 172.21.15.96:0/2271613702 shutdown_connections 2024-05-28T06:04:20.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.650+0000 7fa13f400640 1 -- 172.21.15.96:0/2271613702 wait complete. 2024-05-28T06:04:20.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.659+0000 7f86a7200640 1 -- 172.21.15.96:0/90192308 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f86a015afa0 con 0x7f86a0162210 2024-05-28T06:04:20.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.659+0000 7f869cc00640 1 -- 172.21.15.96:0/90192308 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f8698059ff0 con 0x7f86a0162210 2024-05-28T06:04:20.660 INFO:teuthology.orchestra.run.smithi096.stdout:55834574854 2024-05-28T06:04:20.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.661+0000 7f86a7200640 1 -- 172.21.15.96:0/90192308 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f867c06c940 msgr2=0x7f867c06ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:20.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.661+0000 7f86a7200640 1 --2- 172.21.15.96:0/90192308 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f867c06c940 0x7f867c06ee00 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f8688034000 tx=0x7f868802e000 comp rx=0 tx=0).stop 2024-05-28T06:04:20.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.661+0000 7f86a7200640 1 -- 172.21.15.96:0/90192308 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f86a0162210 msgr2=0x7f86a00ca710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:20.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.661+0000 7f86a7200640 1 --2- 172.21.15.96:0/90192308 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f86a0162210 0x7f86a00ca710 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f869800efc0 tx=0x7f869800c490 comp rx=0 tx=0).stop 2024-05-28T06:04:20.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.662+0000 7f86a7200640 1 -- 172.21.15.96:0/90192308 shutdown_connections 2024-05-28T06:04:20.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.662+0000 7f86a7200640 1 --2- 172.21.15.96:0/90192308 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f867c06c940 0x7f867c06ee00 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:20.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.662+0000 7f86a7200640 1 --2- 172.21.15.96:0/90192308 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f86a0162210 0x7f86a00ca710 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:20.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.662+0000 7f86a7200640 1 --2- 172.21.15.96:0/90192308 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f86a00d1480 0x7f86a0156de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:20.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.662+0000 7f86a7200640 1 --2- 172.21.15.96:0/90192308 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f86a00d10a0 0x7f86a00cc150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:20.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.662+0000 7f86a7200640 1 -- 172.21.15.96:0/90192308 >> 172.21.15.96:0/90192308 conn(0x7f86a008bfc0 msgr2=0x7f86a00cfb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:20.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.662+0000 7f86a7200640 1 -- 172.21.15.96:0/90192308 shutdown_connections 2024-05-28T06:04:20.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.662+0000 7f86a7200640 1 -- 172.21.15.96:0/90192308 wait complete. 2024-05-28T06:04:20.672 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.7 2024-05-28T06:04:20.672 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.670+0000 7fb083800640 1 -- 172.21.15.96:0/1976078779 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fb07c162d00 con 0x7fb07c09ae90 2024-05-28T06:04:20.672 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.671+0000 7fb078c00640 1 -- 172.21.15.96:0/1976078779 <== mon.2 v2:172.21.15.123:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fb06805a1e0 con 0x7fb07c09ae90 2024-05-28T06:04:20.672 DEBUG:teuthology.parallel:result is None 2024-05-28T06:04:20.673 INFO:teuthology.orchestra.run.smithi096.stdout:55834574854 2024-05-28T06:04:20.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.674+0000 7fb083800640 1 -- 172.21.15.96:0/1976078779 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb05c06c9c0 msgr2=0x7fb05c06ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:20.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.674+0000 7fb083800640 1 --2- 172.21.15.96:0/1976078779 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb05c06c9c0 0x7fb05c06ee80 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fb0700098a0 tx=0x7fb07002e000 comp rx=0 tx=0).stop 2024-05-28T06:04:20.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.674+0000 7fb083800640 1 -- 172.21.15.96:0/1976078779 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb07c09ae90 msgr2=0x7fb07c0d03a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:20.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.674+0000 7fb083800640 1 --2- 172.21.15.96:0/1976078779 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb07c09ae90 0x7fb07c0d03a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb06800d520 tx=0x7fb06800d9e0 comp rx=0 tx=0).stop 2024-05-28T06:04:20.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.674+0000 7fb083800640 1 -- 172.21.15.96:0/1976078779 shutdown_connections 2024-05-28T06:04:20.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.674+0000 7fb083800640 1 --2- 172.21.15.96:0/1976078779 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb05c06c9c0 0x7fb05c06ee80 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:20.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.674+0000 7fb083800640 1 --2- 172.21.15.96:0/1976078779 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb07c0c5500 0x7fb07c1f8bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:20.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.674+0000 7fb083800640 1 --2- 172.21.15.96:0/1976078779 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb07c09ae90 0x7fb07c0d03a0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:20.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.674+0000 7fb083800640 1 --2- 172.21.15.96:0/1976078779 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb07c09aa30 0x7fb07c0cfe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:20.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.674+0000 7fb083800640 1 -- 172.21.15.96:0/1976078779 >> 172.21.15.96:0/1976078779 conn(0x7fb07c0b4020 msgr2=0x7fb07c0aff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:20.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.674+0000 7fb083800640 1 -- 172.21.15.96:0/1976078779 shutdown_connections 2024-05-28T06:04:20.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.674+0000 7fb083800640 1 -- 172.21.15.96:0/1976078779 wait complete. 2024-05-28T06:04:20.677 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.0 2024-05-28T06:04:20.678 DEBUG:teuthology.parallel:result is None 2024-05-28T06:04:20.678 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.2 2024-05-28T06:04:20.678 DEBUG:teuthology.parallel:result is None 2024-05-28T06:04:20.683 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.4 2024-05-28T06:04:20.683 DEBUG:teuthology.parallel:result is None 2024-05-28T06:04:20.694 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.5 2024-05-28T06:04:20.694 DEBUG:teuthology.parallel:result is None 2024-05-28T06:04:20.694 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-05-28T06:04:20.694 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-28T06:04:20.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.924+0000 7f14dd000640 1 Processor -- start 2024-05-28T06:04:20.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.924+0000 7f14dd000640 1 -- start start 2024-05-28T06:04:20.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.925+0000 7f14dd000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f14d806f860 0x7f14d80b3150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:20.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.925+0000 7f14dd000640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14d806ed10 con 0x7f14d80b3690 2024-05-28T06:04:20.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.925+0000 7f14dd000640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14d815cb00 con 0x7f14d80d0bb0 2024-05-28T06:04:20.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.925+0000 7f14dd000640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f14d80b3cc0 con 0x7f14d806f860 2024-05-28T06:04:20.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.926+0000 7f14d5400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f14d80b3690 0x7f14d80d04a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:35244/0 (socket says 172.21.15.96:35244) 2024-05-28T06:04:20.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.926+0000 7f14d5400640 1 -- 172.21.15.96:0/91109455 learned_addr learned my addr 172.21.15.96:0/91109455 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:04:20.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.926+0000 7f14d5e00640 1 --2- 172.21.15.96:0/91109455 >> v2:172.21.15.123:3300/0 conn(0x7f14d806f860 0x7f14d80b3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:20.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.926+0000 7f14d4a00640 1 -- 172.21.15.96:0/91109455 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3164692146 0 0) 0x7f14c4001110 con 0x7f14d80b3690 2024-05-28T06:04:20.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.926+0000 7f14d4a00640 1 -- 172.21.15.96:0/91109455 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f14bc003520 con 0x7f14d80b3690 2024-05-28T06:04:20.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.926+0000 7f14d4a00640 1 -- 172.21.15.96:0/91109455 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1707048401 0 0) 0x7f14bc003520 con 0x7f14d80b3690 2024-05-28T06:04:20.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.926+0000 7f14d4a00640 1 -- 172.21.15.96:0/91109455 >> v2:172.21.15.123:3300/0 conn(0x7f14d806f860 msgr2=0x7f14d80b3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:20.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.926+0000 7f14d4a00640 1 --2- 172.21.15.96:0/91109455 >> v2:172.21.15.123:3300/0 conn(0x7f14d806f860 0x7f14d80b3150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:20.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.926+0000 7f14d4a00640 1 -- 172.21.15.96:0/91109455 >> v1:172.21.15.123:6789/0 conn(0x7f14d80d0bb0 legacy=0x7f14d80ad130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:20.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.926+0000 7f14d4a00640 1 -- 172.21.15.96:0/91109455 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14d80b3fa0 con 0x7f14d80b3690 2024-05-28T06:04:20.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.926+0000 7f14d4a00640 1 -- 172.21.15.96:0/91109455 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f14c4001840 con 0x7f14d80b3690 2024-05-28T06:04:20.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.926+0000 7f14d4a00640 1 -- 172.21.15.96:0/91109455 >> v1:172.21.15.96:6789/0 conn(0x7f14d80b3690 legacy=0x7f14d80d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:20.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.926+0000 7f14d4a00640 1 --2- 172.21.15.96:0/91109455 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f14bc0039b0 0x7f14bc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:20.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.926+0000 7f14d4a00640 1 --2- 172.21.15.96:0/91109455 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f14bc004800 0x7f14bc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:20.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.926+0000 7f14d4a00640 1 --2- 172.21.15.96:0/91109455 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f14bc0051c0 0x7f14bc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:20.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.927+0000 7f14d4a00640 1 -- 172.21.15.96:0/91109455 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f14d80b3cc0 con 0x7f14bc0039b0 2024-05-28T06:04:20.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.927+0000 7f14d4a00640 1 -- 172.21.15.96:0/91109455 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f14bc00c060 con 0x7f14bc004800 2024-05-28T06:04:20.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.927+0000 7f14d4a00640 1 -- 172.21.15.96:0/91109455 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f14bc00c260 con 0x7f14bc0051c0 2024-05-28T06:04:20.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.927+0000 7f14d5e00640 1 --2- 172.21.15.96:0/91109455 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f14bc0039b0 0x7f14bc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:20.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.927+0000 7f14d5400640 1 --2- 172.21.15.96:0/91109455 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f14bc004800 0x7f14bc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:20.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.927+0000 7f14d5e00640 1 -- 172.21.15.96:0/91109455 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f14bc0051c0 msgr2=0x7f14bc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:20.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.927+0000 7f14d5e00640 1 --2- 172.21.15.96:0/91109455 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f14bc0051c0 0x7f14bc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:20.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.927+0000 7f14d5e00640 1 -- 172.21.15.96:0/91109455 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f14bc004800 msgr2=0x7f14bc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:20.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.927+0000 7f14d5e00640 1 --2- 172.21.15.96:0/91109455 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f14bc004800 0x7f14bc004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:20.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.927+0000 7f14d5e00640 1 -- 172.21.15.96:0/91109455 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14bc003520 con 0x7f14bc0039b0 2024-05-28T06:04:20.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.927+0000 7f14d6800640 1 -- 172.21.15.96:0/91109455 reap_dead start 2024-05-28T06:04:20.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.928+0000 7f14d5e00640 1 --2- 172.21.15.96:0/91109455 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f14bc0039b0 0x7f14bc003dd0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f14c8009ac0 tx=0x7f14bc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2288ff381669ef2 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:20.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.928+0000 7f14d4a00640 1 -- 172.21.15.96:0/91109455 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14c8036070 con 0x7f14bc0039b0 2024-05-28T06:04:20.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.928+0000 7f14d4a00640 1 -- 172.21.15.96:0/91109455 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14c8031400 con 0x7f14bc0039b0 2024-05-28T06:04:20.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.928+0000 7f14d4a00640 1 -- 172.21.15.96:0/91109455 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14c8009690 con 0x7f14bc0039b0 2024-05-28T06:04:20.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.929+0000 7f14dd000640 1 -- 172.21.15.96:0/91109455 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f14bc0039b0 msgr2=0x7f14bc003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:20.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.929+0000 7f14dd000640 1 --2- 172.21.15.96:0/91109455 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f14bc0039b0 0x7f14bc003dd0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f14c8009ac0 tx=0x7f14bc005830 comp rx=0 tx=0).stop 2024-05-28T06:04:20.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.929+0000 7f14dd000640 1 -- 172.21.15.96:0/91109455 shutdown_connections 2024-05-28T06:04:20.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.929+0000 7f14dd000640 1 --2- 172.21.15.96:0/91109455 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f14bc0039b0 0x7f14bc003dd0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:20.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.929+0000 7f14dd000640 1 -- 172.21.15.96:0/91109455 >> 172.21.15.96:0/91109455 conn(0x7f14d806bae0 msgr2=0x7f14d806bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:20.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.929+0000 7f14dd000640 1 -- 172.21.15.96:0/91109455 shutdown_connections 2024-05-28T06:04:20.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.929+0000 7f14dd000640 1 -- 172.21.15.96:0/91109455 wait complete. 2024-05-28T06:04:20.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.930+0000 7f14dd000640 1 Processor -- start 2024-05-28T06:04:20.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.931+0000 7f14dd000640 1 -- start start 2024-05-28T06:04:20.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.931+0000 7f14dd000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f14bc0039b0 0x7f14d81e6c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:20.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.931+0000 7f14dd000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f14d815eeb0 0x7f14d80cc620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:20.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.931+0000 7f14dd000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f14d815f310 0x7f14d80ccb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:20.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.931+0000 7f14dd000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f14d80b4840 con 0x7f14bc0039b0 2024-05-28T06:04:20.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.931+0000 7f14dd000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f14d80b4560 con 0x7f14d815f310 2024-05-28T06:04:20.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.931+0000 7f14dd000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f14d80b3150 con 0x7f14d815eeb0 2024-05-28T06:04:20.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.932+0000 7f14d5e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f14bc0039b0 0x7f14d81e6c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:20.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.932+0000 7f14d5e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f14bc0039b0 0x7f14d81e6c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48126/0 (socket says 172.21.15.96:48126) 2024-05-28T06:04:20.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.932+0000 7f14d5e00640 1 -- 172.21.15.96:0/643903618 learned_addr learned my addr 172.21.15.96:0/643903618 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:20.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.932+0000 7f14d5400640 1 --2- 172.21.15.96:0/643903618 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f14d815eeb0 0x7f14d80cc620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:20.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.932+0000 7f14d5e00640 1 -- 172.21.15.96:0/643903618 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f14d815eeb0 msgr2=0x7f14d80cc620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:20.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.932+0000 7f14d5e00640 1 --2- 172.21.15.96:0/643903618 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f14d815eeb0 0x7f14d80cc620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:20.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.932+0000 7f14d5e00640 1 -- 172.21.15.96:0/643903618 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f14d815f310 msgr2=0x7f14d80ccb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:20.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.932+0000 7f14d5e00640 1 --2- 172.21.15.96:0/643903618 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f14d815f310 0x7f14d80ccb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:20.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.932+0000 7f14d5e00640 1 -- 172.21.15.96:0/643903618 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14d81e5ba0 con 0x7f14bc0039b0 2024-05-28T06:04:20.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.932+0000 7f14d5e00640 1 --2- 172.21.15.96:0/643903618 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f14bc0039b0 0x7f14d81e6c40 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f14c8034040 tx=0x7f14c8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:20.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.933+0000 7f14ce000640 1 -- 172.21.15.96:0/643903618 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14c803f020 con 0x7f14bc0039b0 2024-05-28T06:04:20.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.933+0000 7f14ce000640 1 -- 172.21.15.96:0/643903618 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14c8037040 con 0x7f14bc0039b0 2024-05-28T06:04:20.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.933+0000 7f14ce000640 1 -- 172.21.15.96:0/643903618 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14c8036070 con 0x7f14bc0039b0 2024-05-28T06:04:20.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.933+0000 7f14dd000640 1 -- 172.21.15.96:0/643903618 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14d81e5e80 con 0x7f14bc0039b0 2024-05-28T06:04:20.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.933+0000 7f14dd000640 1 -- 172.21.15.96:0/643903618 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14d81e6440 con 0x7f14bc0039b0 2024-05-28T06:04:20.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.934+0000 7f14dd000640 1 -- 172.21.15.96:0/643903618 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14d80689e0 con 0x7f14bc0039b0 2024-05-28T06:04:20.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.935+0000 7f14ce000640 1 -- 172.21.15.96:0/643903618 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f14c8042020 con 0x7f14bc0039b0 2024-05-28T06:04:20.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.935+0000 7f14ce000640 1 --2- 172.21.15.96:0/643903618 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f14b806c8f0 0x7f14b806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:20.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.936+0000 7f14ce000640 1 -- 172.21.15.96:0/643903618 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f14c80ab980 con 0x7f14bc0039b0 2024-05-28T06:04:20.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.936+0000 7f14d5400640 1 --2- 172.21.15.96:0/643903618 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f14b806c8f0 0x7f14b806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:20.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.937+0000 7f14d5400640 1 --2- 172.21.15.96:0/643903618 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f14b806c8f0 0x7f14b806edb0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f14c40012b0 tx=0x7f14c4001e00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:20.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:20.940+0000 7f14ce000640 1 -- 172.21.15.96:0/643903618 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f14c8049070 con 0x7f14bc0039b0 2024-05-28T06:04:21.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.123+0000 7f14dd000640 1 -- 172.21.15.96:0/643903618 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f14d806e2f0 con 0x7f14b806c8f0 2024-05-28T06:04:21.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.141+0000 7f14ce000640 1 -- 172.21.15.96:0/643903618 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251836 (secure 0 0 0) 0x7f14d806e2f0 con 0x7f14b806c8f0 2024-05-28T06:04:21.145 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:04:21.147 INFO:teuthology.orchestra.run.smithi096.stderr:dumped all 2024-05-28T06:04:21.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.146+0000 7f14dd000640 1 -- 172.21.15.96:0/643903618 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f14b806c8f0 msgr2=0x7f14b806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:21.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.146+0000 7f14dd000640 1 --2- 172.21.15.96:0/643903618 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f14b806c8f0 0x7f14b806edb0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f14c40012b0 tx=0x7f14c4001e00 comp rx=0 tx=0).stop 2024-05-28T06:04:21.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.146+0000 7f14dd000640 1 -- 172.21.15.96:0/643903618 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f14bc0039b0 msgr2=0x7f14d81e6c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:21.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.146+0000 7f14dd000640 1 --2- 172.21.15.96:0/643903618 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f14bc0039b0 0x7f14d81e6c40 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f14c8034040 tx=0x7f14c8004730 comp rx=0 tx=0).stop 2024-05-28T06:04:21.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.146+0000 7f14dd000640 1 -- 172.21.15.96:0/643903618 shutdown_connections 2024-05-28T06:04:21.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.146+0000 7f14dd000640 1 --2- 172.21.15.96:0/643903618 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f14b806c8f0 0x7f14b806edb0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:21.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.146+0000 7f14dd000640 1 --2- 172.21.15.96:0/643903618 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f14d815f310 0x7f14d80ccb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:21.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.146+0000 7f14dd000640 1 --2- 172.21.15.96:0/643903618 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f14d815eeb0 0x7f14d80cc620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:21.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.146+0000 7f14dd000640 1 --2- 172.21.15.96:0/643903618 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f14bc0039b0 0x7f14d81e6c40 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:21.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.146+0000 7f14dd000640 1 -- 172.21.15.96:0/643903618 >> 172.21.15.96:0/643903618 conn(0x7f14d806bae0 msgr2=0x7f14d8072ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:21.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.146+0000 7f14dd000640 1 -- 172.21.15.96:0/643903618 shutdown_connections 2024-05-28T06:04:21.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.146+0000 7f14dd000640 1 -- 172.21.15.96:0/643903618 wait complete. 2024-05-28T06:04:21.164 INFO:teuthology.orchestra.run.smithi096.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-05-28T06:04:19.485243+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218992,"kb_used_data":2824,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755728,"statfs":{"total":773094113280,"available":772869865472,"internally_reserved":0,"allocated":2891776,"data_stored":1753028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.615721"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.997911+0000","last_change":"2024-05-28T06:04:10.916483+0000","last_active":"2024-05-28T06:04:12.997911+0000","last_peered":"2024-05-28T06:04:12.997911+0000","last_clean":"2024-05-28T06:04:12.997911+0000","last_became_active":"2024-05-28T06:04:10.916214+0000","last_became_peered":"2024-05-28T06:04:10.916214+0000","last_unstale":"2024-05-28T06:04:12.997911+0000","last_undegraded":"2024-05-28T06:04:12.997911+0000","last_fullsized":"2024-05-28T06:04:12.997911+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:43:05.061767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.889968+0000","last_change":"2024-05-28T06:04:10.900791+0000","last_active":"2024-05-28T06:04:11.889968+0000","last_peered":"2024-05-28T06:04:11.889968+0000","last_clean":"2024-05-28T06:04:11.889968+0000","last_became_active":"2024-05-28T06:04:10.900489+0000","last_became_peered":"2024-05-28T06:04:10.900489+0000","last_unstale":"2024-05-28T06:04:11.889968+0000","last_undegraded":"2024-05-28T06:04:11.889968+0000","last_fullsized":"2024-05-28T06:04:11.889968+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:53:07.733019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.794222+0000","last_change":"2024-05-28T06:04:10.901559+0000","last_active":"2024-05-28T06:04:12.794222+0000","last_peered":"2024-05-28T06:04:12.794222+0000","last_clean":"2024-05-28T06:04:12.794222+0000","last_became_active":"2024-05-28T06:04:10.899022+0000","last_became_peered":"2024-05-28T06:04:10.899022+0000","last_unstale":"2024-05-28T06:04:12.794222+0000","last_undegraded":"2024-05-28T06:04:12.794222+0000","last_fullsized":"2024-05-28T06:04:12.794222+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:28:21.123299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.997115+0000","last_change":"2024-05-28T06:04:10.923988+0000","last_active":"2024-05-28T06:04:12.997115+0000","last_peered":"2024-05-28T06:04:12.997115+0000","last_clean":"2024-05-28T06:04:12.997115+0000","last_became_active":"2024-05-28T06:04:10.923718+0000","last_became_peered":"2024-05-28T06:04:10.923718+0000","last_unstale":"2024-05-28T06:04:12.997115+0000","last_undegraded":"2024-05-28T06:04:12.997115+0000","last_fullsized":"2024-05-28T06:04:12.997115+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:45:47.884578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.996357+0000","last_change":"2024-05-28T06:04:10.917922+0000","last_active":"2024-05-28T06:04:12.996357+0000","last_peered":"2024-05-28T06:04:12.996357+0000","last_clean":"2024-05-28T06:04:12.996357+0000","last_became_active":"2024-05-28T06:04:10.917253+0000","last_became_peered":"2024-05-28T06:04:10.917253+0000","last_unstale":"2024-05-28T06:04:12.996357+0000","last_undegraded":"2024-05-28T06:04:12.996357+0000","last_fullsized":"2024-05-28T06:04:12.996357+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:04:49.415234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.996757+0000","last_change":"2024-05-28T06:04:10.907704+0000","last_active":"2024-05-28T06:04:12.996757+0000","last_peered":"2024-05-28T06:04:12.996757+0000","last_clean":"2024-05-28T06:04:12.996757+0000","last_became_active":"2024-05-28T06:04:10.907065+0000","last_became_peered":"2024-05-28T06:04:10.907065+0000","last_unstale":"2024-05-28T06:04:12.996757+0000","last_undegraded":"2024-05-28T06:04:12.996757+0000","last_fullsized":"2024-05-28T06:04:12.996757+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:28:21.123299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.995527+0000","last_change":"2024-05-28T06:04:10.905438+0000","last_active":"2024-05-28T06:04:12.995527+0000","last_peered":"2024-05-28T06:04:12.995527+0000","last_clean":"2024-05-28T06:04:12.995527+0000","last_became_active":"2024-05-28T06:04:10.904842+0000","last_became_peered":"2024-05-28T06:04:10.904842+0000","last_unstale":"2024-05-28T06:04:12.995527+0000","last_undegraded":"2024-05-28T06:04:12.995527+0000","last_fullsized":"2024-05-28T06:04:12.995527+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:46:27.923407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.995330+0000","last_change":"2024-05-28T06:04:10.899401+0000","last_active":"2024-05-28T06:04:12.995330+0000","last_peered":"2024-05-28T06:04:12.995330+0000","last_clean":"2024-05-28T06:04:12.995330+0000","last_became_active":"2024-05-28T06:04:10.898742+0000","last_became_peered":"2024-05-28T06:04:10.898742+0000","last_unstale":"2024-05-28T06:04:12.995330+0000","last_undegraded":"2024-05-28T06:04:12.995330+0000","last_fullsized":"2024-05-28T06:04:12.995330+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:39:46.389531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.997773+0000","last_change":"2024-05-28T06:04:10.907973+0000","last_active":"2024-05-28T06:04:12.997773+0000","last_peered":"2024-05-28T06:04:12.997773+0000","last_clean":"2024-05-28T06:04:12.997773+0000","last_became_active":"2024-05-28T06:04:10.905186+0000","last_became_peered":"2024-05-28T06:04:10.905186+0000","last_unstale":"2024-05-28T06:04:12.997773+0000","last_undegraded":"2024-05-28T06:04:12.997773+0000","last_fullsized":"2024-05-28T06:04:12.997773+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:39:13.976770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.909549+0000","last_change":"2024-05-28T06:04:10.927703+0000","last_active":"2024-05-28T06:04:11.909549+0000","last_peered":"2024-05-28T06:04:11.909549+0000","last_clean":"2024-05-28T06:04:11.909549+0000","last_became_active":"2024-05-28T06:04:10.927383+0000","last_became_peered":"2024-05-28T06:04:10.927383+0000","last_unstale":"2024-05-28T06:04:11.909549+0000","last_undegraded":"2024-05-28T06:04:11.909549+0000","last_fullsized":"2024-05-28T06:04:11.909549+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:11:51.651640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.909515+0000","last_change":"2024-05-28T06:04:10.926573+0000","last_active":"2024-05-28T06:04:11.909515+0000","last_peered":"2024-05-28T06:04:11.909515+0000","last_clean":"2024-05-28T06:04:11.909515+0000","last_became_active":"2024-05-28T06:04:10.926364+0000","last_became_peered":"2024-05-28T06:04:10.926364+0000","last_unstale":"2024-05-28T06:04:11.909515+0000","last_undegraded":"2024-05-28T06:04:11.909515+0000","last_fullsized":"2024-05-28T06:04:11.909515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:28:21.123299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.893744+0000","last_change":"2024-05-28T06:04:10.913441+0000","last_active":"2024-05-28T06:04:11.893744+0000","last_peered":"2024-05-28T06:04:11.893744+0000","last_clean":"2024-05-28T06:04:11.893744+0000","last_became_active":"2024-05-28T06:04:10.912847+0000","last_became_peered":"2024-05-28T06:04:10.912847+0000","last_unstale":"2024-05-28T06:04:11.893744+0000","last_undegraded":"2024-05-28T06:04:11.893744+0000","last_fullsized":"2024-05-28T06:04:11.893744+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:58:51.819866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.996503+0000","last_change":"2024-05-28T06:04:10.909268+0000","last_active":"2024-05-28T06:04:12.996503+0000","last_peered":"2024-05-28T06:04:12.996503+0000","last_clean":"2024-05-28T06:04:12.996503+0000","last_became_active":"2024-05-28T06:04:10.908954+0000","last_became_peered":"2024-05-28T06:04:10.908954+0000","last_unstale":"2024-05-28T06:04:12.996503+0000","last_undegraded":"2024-05-28T06:04:12.996503+0000","last_fullsized":"2024-05-28T06:04:12.996503+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:28:21.123299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.793137+0000","last_change":"2024-05-28T06:04:10.899509+0000","last_active":"2024-05-28T06:04:12.793137+0000","last_peered":"2024-05-28T06:04:12.793137+0000","last_clean":"2024-05-28T06:04:12.793137+0000","last_became_active":"2024-05-28T06:04:10.896990+0000","last_became_peered":"2024-05-28T06:04:10.896990+0000","last_unstale":"2024-05-28T06:04:12.793137+0000","last_undegraded":"2024-05-28T06:04:12.793137+0000","last_fullsized":"2024-05-28T06:04:12.793137+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:17:36.664466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.890266+0000","last_change":"2024-05-28T06:04:10.906808+0000","last_active":"2024-05-28T06:04:11.890266+0000","last_peered":"2024-05-28T06:04:11.890266+0000","last_clean":"2024-05-28T06:04:11.890266+0000","last_became_active":"2024-05-28T06:04:10.906579+0000","last_became_peered":"2024-05-28T06:04:10.906579+0000","last_unstale":"2024-05-28T06:04:11.890266+0000","last_undegraded":"2024-05-28T06:04:11.890266+0000","last_fullsized":"2024-05-28T06:04:11.890266+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:28:21.123299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.997630+0000","last_change":"2024-05-28T06:04:10.904806+0000","last_active":"2024-05-28T06:04:12.997630+0000","last_peered":"2024-05-28T06:04:12.997630+0000","last_clean":"2024-05-28T06:04:12.997630+0000","last_became_active":"2024-05-28T06:04:10.904450+0000","last_became_peered":"2024-05-28T06:04:10.904450+0000","last_unstale":"2024-05-28T06:04:12.997630+0000","last_undegraded":"2024-05-28T06:04:12.997630+0000","last_fullsized":"2024-05-28T06:04:12.997630+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:51:12.839226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.999350+0000","last_change":"2024-05-28T06:04:10.918376+0000","last_active":"2024-05-28T06:04:12.999350+0000","last_peered":"2024-05-28T06:04:12.999350+0000","last_clean":"2024-05-28T06:04:12.999350+0000","last_became_active":"2024-05-28T06:04:10.918042+0000","last_became_peered":"2024-05-28T06:04:10.918042+0000","last_unstale":"2024-05-28T06:04:12.999350+0000","last_undegraded":"2024-05-28T06:04:12.999350+0000","last_fullsized":"2024-05-28T06:04:12.999350+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:39:13.976770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.996501+0000","last_change":"2024-05-28T06:04:10.910995+0000","last_active":"2024-05-28T06:04:12.996501+0000","last_peered":"2024-05-28T06:04:12.996501+0000","last_clean":"2024-05-28T06:04:12.996501+0000","last_became_active":"2024-05-28T06:04:10.910672+0000","last_became_peered":"2024-05-28T06:04:10.910672+0000","last_unstale":"2024-05-28T06:04:12.996501+0000","last_undegraded":"2024-05-28T06:04:12.996501+0000","last_fullsized":"2024-05-28T06:04:12.996501+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:58:51.819866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.997411+0000","last_change":"2024-05-28T06:04:10.911244+0000","last_active":"2024-05-28T06:04:12.997411+0000","last_peered":"2024-05-28T06:04:12.997411+0000","last_clean":"2024-05-28T06:04:12.997411+0000","last_became_active":"2024-05-28T06:04:10.910921+0000","last_became_peered":"2024-05-28T06:04:10.910921+0000","last_unstale":"2024-05-28T06:04:12.997411+0000","last_undegraded":"2024-05-28T06:04:12.997411+0000","last_fullsized":"2024-05-28T06:04:12.997411+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:05:14.695994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.909154+0000","last_change":"2024-05-28T06:04:10.921344+0000","last_active":"2024-05-28T06:04:11.909154+0000","last_peered":"2024-05-28T06:04:11.909154+0000","last_clean":"2024-05-28T06:04:11.909154+0000","last_became_active":"2024-05-28T06:04:10.920438+0000","last_became_peered":"2024-05-28T06:04:10.920438+0000","last_unstale":"2024-05-28T06:04:11.909154+0000","last_undegraded":"2024-05-28T06:04:11.909154+0000","last_fullsized":"2024-05-28T06:04:11.909154+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:53:07.733019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.793512+0000","last_change":"2024-05-28T06:04:10.902770+0000","last_active":"2024-05-28T06:04:12.793512+0000","last_peered":"2024-05-28T06:04:12.793512+0000","last_clean":"2024-05-28T06:04:12.793512+0000","last_became_active":"2024-05-28T06:04:10.901534+0000","last_became_peered":"2024-05-28T06:04:10.901534+0000","last_unstale":"2024-05-28T06:04:12.793512+0000","last_undegraded":"2024-05-28T06:04:12.793512+0000","last_fullsized":"2024-05-28T06:04:12.793512+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:00:40.966129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.996957+0000","last_change":"2024-05-28T06:04:10.915708+0000","last_active":"2024-05-28T06:04:12.996957+0000","last_peered":"2024-05-28T06:04:12.996957+0000","last_clean":"2024-05-28T06:04:12.996957+0000","last_became_active":"2024-05-28T06:04:10.913058+0000","last_became_peered":"2024-05-28T06:04:10.913058+0000","last_unstale":"2024-05-28T06:04:12.996957+0000","last_undegraded":"2024-05-28T06:04:12.996957+0000","last_fullsized":"2024-05-28T06:04:12.996957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:11:51.651640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.995352+0000","last_change":"2024-05-28T06:04:10.904255+0000","last_active":"2024-05-28T06:04:12.995352+0000","last_peered":"2024-05-28T06:04:12.995352+0000","last_clean":"2024-05-28T06:04:12.995352+0000","last_became_active":"2024-05-28T06:04:10.903840+0000","last_became_peered":"2024-05-28T06:04:10.903840+0000","last_unstale":"2024-05-28T06:04:12.995352+0000","last_undegraded":"2024-05-28T06:04:12.995352+0000","last_fullsized":"2024-05-28T06:04:12.995352+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:06:44.034610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.997441+0000","last_change":"2024-05-28T06:04:10.902458+0000","last_active":"2024-05-28T06:04:12.997441+0000","last_peered":"2024-05-28T06:04:12.997441+0000","last_clean":"2024-05-28T06:04:12.997441+0000","last_became_active":"2024-05-28T06:04:10.902054+0000","last_became_peered":"2024-05-28T06:04:10.902054+0000","last_unstale":"2024-05-28T06:04:12.997441+0000","last_undegraded":"2024-05-28T06:04:12.997441+0000","last_fullsized":"2024-05-28T06:04:12.997441+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:46:27.923407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.997558+0000","last_change":"2024-05-28T06:04:10.895892+0000","last_active":"2024-05-28T06:04:12.997558+0000","last_peered":"2024-05-28T06:04:12.997558+0000","last_clean":"2024-05-28T06:04:12.997558+0000","last_became_active":"2024-05-28T06:04:10.895579+0000","last_became_peered":"2024-05-28T06:04:10.895579+0000","last_unstale":"2024-05-28T06:04:12.997558+0000","last_undegraded":"2024-05-28T06:04:12.997558+0000","last_fullsized":"2024-05-28T06:04:12.997558+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:51:12.839226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.909428+0000","last_change":"2024-05-28T06:04:10.914830+0000","last_active":"2024-05-28T06:04:11.909428+0000","last_peered":"2024-05-28T06:04:11.909428+0000","last_clean":"2024-05-28T06:04:11.909428+0000","last_became_active":"2024-05-28T06:04:10.913651+0000","last_became_peered":"2024-05-28T06:04:10.913651+0000","last_unstale":"2024-05-28T06:04:11.909428+0000","last_undegraded":"2024-05-28T06:04:11.909428+0000","last_fullsized":"2024-05-28T06:04:11.909428+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:46:29.255545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.889962+0000","last_change":"2024-05-28T06:04:10.903385+0000","last_active":"2024-05-28T06:04:11.889962+0000","last_peered":"2024-05-28T06:04:11.889962+0000","last_clean":"2024-05-28T06:04:11.889962+0000","last_became_active":"2024-05-28T06:04:10.903078+0000","last_became_peered":"2024-05-28T06:04:10.903078+0000","last_unstale":"2024-05-28T06:04:11.889962+0000","last_undegraded":"2024-05-28T06:04:11.889962+0000","last_fullsized":"2024-05-28T06:04:11.889962+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:06:44.034610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.997438+0000","last_change":"2024-05-28T06:04:10.894062+0000","last_active":"2024-05-28T06:04:12.997438+0000","last_peered":"2024-05-28T06:04:12.997438+0000","last_clean":"2024-05-28T06:04:12.997438+0000","last_became_active":"2024-05-28T06:04:10.893763+0000","last_became_peered":"2024-05-28T06:04:10.893763+0000","last_unstale":"2024-05-28T06:04:12.997438+0000","last_undegraded":"2024-05-28T06:04:12.997438+0000","last_fullsized":"2024-05-28T06:04:12.997438+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:46:27.923407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.793745+0000","last_change":"2024-05-28T06:04:10.897684+0000","last_active":"2024-05-28T06:04:12.793745+0000","last_peered":"2024-05-28T06:04:12.793745+0000","last_clean":"2024-05-28T06:04:12.793745+0000","last_became_active":"2024-05-28T06:04:10.897382+0000","last_became_peered":"2024-05-28T06:04:10.897382+0000","last_unstale":"2024-05-28T06:04:12.793745+0000","last_undegraded":"2024-05-28T06:04:12.793745+0000","last_fullsized":"2024-05-28T06:04:12.793745+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:39:46.389531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.997085+0000","last_change":"2024-05-28T06:04:10.908012+0000","last_active":"2024-05-28T06:04:12.997085+0000","last_peered":"2024-05-28T06:04:12.997085+0000","last_clean":"2024-05-28T06:04:12.997085+0000","last_became_active":"2024-05-28T06:04:10.907436+0000","last_became_peered":"2024-05-28T06:04:10.907436+0000","last_unstale":"2024-05-28T06:04:12.997085+0000","last_undegraded":"2024-05-28T06:04:12.997085+0000","last_fullsized":"2024-05-28T06:04:12.997085+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:58:16.009219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.996141+0000","last_change":"2024-05-28T06:04:10.905729+0000","last_active":"2024-05-28T06:04:12.996141+0000","last_peered":"2024-05-28T06:04:12.996141+0000","last_clean":"2024-05-28T06:04:12.996141+0000","last_became_active":"2024-05-28T06:04:10.905325+0000","last_became_peered":"2024-05-28T06:04:10.905325+0000","last_unstale":"2024-05-28T06:04:12.996141+0000","last_undegraded":"2024-05-28T06:04:12.996141+0000","last_fullsized":"2024-05-28T06:04:12.996141+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:03:46.394497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.908750+0000","last_change":"2024-05-28T06:04:10.915117+0000","last_active":"2024-05-28T06:04:11.908750+0000","last_peered":"2024-05-28T06:04:11.908750+0000","last_clean":"2024-05-28T06:04:11.908750+0000","last_became_active":"2024-05-28T06:04:10.914761+0000","last_became_peered":"2024-05-28T06:04:10.914761+0000","last_unstale":"2024-05-28T06:04:11.908750+0000","last_undegraded":"2024-05-28T06:04:11.908750+0000","last_fullsized":"2024-05-28T06:04:11.908750+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:00:40.966129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.892804+0000","last_change":"2024-05-28T06:04:10.897395+0000","last_active":"2024-05-28T06:04:11.892804+0000","last_peered":"2024-05-28T06:04:11.892804+0000","last_clean":"2024-05-28T06:04:11.892804+0000","last_became_active":"2024-05-28T06:04:10.897108+0000","last_became_peered":"2024-05-28T06:04:10.897108+0000","last_unstale":"2024-05-28T06:04:11.892804+0000","last_undegraded":"2024-05-28T06:04:11.892804+0000","last_fullsized":"2024-05-28T06:04:11.892804+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:16:53.718673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.996650+0000","last_change":"2024-05-28T06:04:09.890037+0000","last_active":"2024-05-28T06:04:12.996650+0000","last_peered":"2024-05-28T06:04:12.996650+0000","last_clean":"2024-05-28T06:04:12.996650+0000","last_became_active":"2024-05-28T06:04:09.889696+0000","last_became_peered":"2024-05-28T06:04:09.889696+0000","last_unstale":"2024-05-28T06:04:12.996650+0000","last_undegraded":"2024-05-28T06:04:12.996650+0000","last_fullsized":"2024-05-28T06:04:12.996650+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:29:32.319083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.794133+0000","last_change":"2024-05-28T06:04:10.896136+0000","last_active":"2024-05-28T06:04:12.794133+0000","last_peered":"2024-05-28T06:04:12.794133+0000","last_clean":"2024-05-28T06:04:12.794133+0000","last_became_active":"2024-05-28T06:04:10.895631+0000","last_became_peered":"2024-05-28T06:04:10.895631+0000","last_unstale":"2024-05-28T06:04:12.794133+0000","last_undegraded":"2024-05-28T06:04:12.794133+0000","last_fullsized":"2024-05-28T06:04:12.794133+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:46:29.255545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.908566+0000","last_change":"2024-05-28T06:04:09.879488+0000","last_active":"2024-05-28T06:04:11.908566+0000","last_peered":"2024-05-28T06:04:11.908566+0000","last_clean":"2024-05-28T06:04:11.908566+0000","last_became_active":"2024-05-28T06:04:09.879182+0000","last_became_peered":"2024-05-28T06:04:09.879182+0000","last_unstale":"2024-05-28T06:04:11.908566+0000","last_undegraded":"2024-05-28T06:04:11.908566+0000","last_fullsized":"2024-05-28T06:04:11.908566+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:29:32.319083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.995389+0000","last_change":"2024-05-28T06:04:09.884424+0000","last_active":"2024-05-28T06:04:12.995389+0000","last_peered":"2024-05-28T06:04:12.995389+0000","last_clean":"2024-05-28T06:04:12.995389+0000","last_became_active":"2024-05-28T06:04:09.884088+0000","last_became_peered":"2024-05-28T06:04:09.884088+0000","last_unstale":"2024-05-28T06:04:12.995389+0000","last_undegraded":"2024-05-28T06:04:12.995389+0000","last_fullsized":"2024-05-28T06:04:12.995389+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:27:30.379288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.996856+0000","last_change":"2024-05-28T06:04:10.890237+0000","last_active":"2024-05-28T06:04:12.996856+0000","last_peered":"2024-05-28T06:04:12.996856+0000","last_clean":"2024-05-28T06:04:12.996856+0000","last_became_active":"2024-05-28T06:04:10.889644+0000","last_became_peered":"2024-05-28T06:04:10.889644+0000","last_unstale":"2024-05-28T06:04:12.996856+0000","last_undegraded":"2024-05-28T06:04:12.996856+0000","last_fullsized":"2024-05-28T06:04:12.996856+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:58:16.009219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.997564+0000","last_change":"2024-05-28T06:04:09.875990+0000","last_active":"2024-05-28T06:04:12.997564+0000","last_peered":"2024-05-28T06:04:12.997564+0000","last_clean":"2024-05-28T06:04:12.997564+0000","last_became_active":"2024-05-28T06:04:09.875644+0000","last_became_peered":"2024-05-28T06:04:09.875644+0000","last_unstale":"2024-05-28T06:04:12.997564+0000","last_undegraded":"2024-05-28T06:04:12.997564+0000","last_fullsized":"2024-05-28T06:04:12.997564+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:27:30.379288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.889859+0000","last_change":"2024-05-28T06:04:10.899922+0000","last_active":"2024-05-28T06:04:11.889859+0000","last_peered":"2024-05-28T06:04:11.889859+0000","last_clean":"2024-05-28T06:04:11.889859+0000","last_became_active":"2024-05-28T06:04:10.899617+0000","last_became_peered":"2024-05-28T06:04:10.899617+0000","last_unstale":"2024-05-28T06:04:11.889859+0000","last_undegraded":"2024-05-28T06:04:11.889859+0000","last_fullsized":"2024-05-28T06:04:11.889859+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:39:46.389531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.909050+0000","last_change":"2024-05-28T06:04:09.887084+0000","last_active":"2024-05-28T06:04:11.909050+0000","last_peered":"2024-05-28T06:04:11.909050+0000","last_clean":"2024-05-28T06:04:11.909050+0000","last_became_active":"2024-05-28T06:04:09.886821+0000","last_became_peered":"2024-05-28T06:04:09.886821+0000","last_unstale":"2024-05-28T06:04:11.909050+0000","last_undegraded":"2024-05-28T06:04:11.909050+0000","last_fullsized":"2024-05-28T06:04:11.909050+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:58:06.720498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.998553+0000","last_change":"2024-05-28T06:04:10.921008+0000","last_active":"2024-05-28T06:04:12.998553+0000","last_peered":"2024-05-28T06:04:12.998553+0000","last_clean":"2024-05-28T06:04:12.998553+0000","last_became_active":"2024-05-28T06:04:10.913026+0000","last_became_peered":"2024-05-28T06:04:10.913026+0000","last_unstale":"2024-05-28T06:04:12.998553+0000","last_undegraded":"2024-05-28T06:04:12.998553+0000","last_fullsized":"2024-05-28T06:04:12.998553+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:34:44.387474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.998439+0000","last_change":"2024-05-28T06:04:09.887088+0000","last_active":"2024-05-28T06:04:12.998439+0000","last_peered":"2024-05-28T06:04:12.998439+0000","last_clean":"2024-05-28T06:04:12.998439+0000","last_became_active":"2024-05-28T06:04:09.886809+0000","last_became_peered":"2024-05-28T06:04:09.886809+0000","last_unstale":"2024-05-28T06:04:12.998439+0000","last_undegraded":"2024-05-28T06:04:12.998439+0000","last_fullsized":"2024-05-28T06:04:12.998439+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:16:52.703842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.794379+0000","last_change":"2024-05-28T06:04:10.904810+0000","last_active":"2024-05-28T06:04:12.794379+0000","last_peered":"2024-05-28T06:04:12.794379+0000","last_clean":"2024-05-28T06:04:12.794379+0000","last_became_active":"2024-05-28T06:04:10.904445+0000","last_became_peered":"2024-05-28T06:04:10.904445+0000","last_unstale":"2024-05-28T06:04:12.794379+0000","last_undegraded":"2024-05-28T06:04:12.794379+0000","last_fullsized":"2024-05-28T06:04:12.794379+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:58:16.009219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.890096+0000","last_change":"2024-05-28T06:04:09.882535+0000","last_active":"2024-05-28T06:04:11.890096+0000","last_peered":"2024-05-28T06:04:11.890096+0000","last_clean":"2024-05-28T06:04:11.890096+0000","last_became_active":"2024-05-28T06:04:09.882159+0000","last_became_peered":"2024-05-28T06:04:09.882159+0000","last_unstale":"2024-05-28T06:04:11.890096+0000","last_undegraded":"2024-05-28T06:04:11.890096+0000","last_fullsized":"2024-05-28T06:04:11.890096+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:03:45.379665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.893646+0000","last_change":"2024-05-28T06:04:10.901897+0000","last_active":"2024-05-28T06:04:11.893646+0000","last_peered":"2024-05-28T06:04:11.893646+0000","last_clean":"2024-05-28T06:04:11.893646+0000","last_became_active":"2024-05-28T06:04:10.901156+0000","last_became_peered":"2024-05-28T06:04:10.901156+0000","last_unstale":"2024-05-28T06:04:11.893646+0000","last_undegraded":"2024-05-28T06:04:11.893646+0000","last_fullsized":"2024-05-28T06:04:11.893646+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:03:46.394497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.909296+0000","last_change":"2024-05-28T06:04:09.878442+0000","last_active":"2024-05-28T06:04:11.909296+0000","last_peered":"2024-05-28T06:04:11.909296+0000","last_clean":"2024-05-28T06:04:11.909296+0000","last_became_active":"2024-05-28T06:04:09.878148+0000","last_became_peered":"2024-05-28T06:04:09.878148+0000","last_unstale":"2024-05-28T06:04:11.909296+0000","last_undegraded":"2024-05-28T06:04:11.909296+0000","last_fullsized":"2024-05-28T06:04:11.909296+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:38:20.882939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.996201+0000","last_change":"2024-05-28T06:04:10.911058+0000","last_active":"2024-05-28T06:04:12.996201+0000","last_peered":"2024-05-28T06:04:12.996201+0000","last_clean":"2024-05-28T06:04:12.996201+0000","last_became_active":"2024-05-28T06:04:10.910775+0000","last_became_peered":"2024-05-28T06:04:10.910775+0000","last_unstale":"2024-05-28T06:04:12.996201+0000","last_undegraded":"2024-05-28T06:04:12.996201+0000","last_fullsized":"2024-05-28T06:04:12.996201+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:03:18.610996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.794179+0000","last_change":"2024-05-28T06:04:09.878327+0000","last_active":"2024-05-28T06:04:12.794179+0000","last_peered":"2024-05-28T06:04:12.794179+0000","last_clean":"2024-05-28T06:04:12.794179+0000","last_became_active":"2024-05-28T06:04:09.878028+0000","last_became_peered":"2024-05-28T06:04:09.878028+0000","last_unstale":"2024-05-28T06:04:12.794179+0000","last_undegraded":"2024-05-28T06:04:12.794179+0000","last_fullsized":"2024-05-28T06:04:12.794179+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:12:29.496721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.793473+0000","last_change":"2024-05-28T06:04:10.902163+0000","last_active":"2024-05-28T06:04:12.793473+0000","last_peered":"2024-05-28T06:04:12.793473+0000","last_clean":"2024-05-28T06:04:12.793473+0000","last_became_active":"2024-05-28T06:04:10.901846+0000","last_became_peered":"2024-05-28T06:04:10.901846+0000","last_unstale":"2024-05-28T06:04:12.793473+0000","last_undegraded":"2024-05-28T06:04:12.793473+0000","last_fullsized":"2024-05-28T06:04:12.793473+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:11:13.101661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.997163+0000","last_change":"2024-05-28T06:04:09.884756+0000","last_active":"2024-05-28T06:04:12.997163+0000","last_peered":"2024-05-28T06:04:12.997163+0000","last_clean":"2024-05-28T06:04:12.997163+0000","last_became_active":"2024-05-28T06:04:09.884354+0000","last_became_peered":"2024-05-28T06:04:09.884354+0000","last_unstale":"2024-05-28T06:04:12.997163+0000","last_undegraded":"2024-05-28T06:04:12.997163+0000","last_fullsized":"2024-05-28T06:04:12.997163+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:36:53.154893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.996772+0000","last_change":"2024-05-28T06:04:10.916180+0000","last_active":"2024-05-28T06:04:12.996772+0000","last_peered":"2024-05-28T06:04:12.996772+0000","last_clean":"2024-05-28T06:04:12.996772+0000","last_became_active":"2024-05-28T06:04:10.915630+0000","last_became_peered":"2024-05-28T06:04:10.915630+0000","last_unstale":"2024-05-28T06:04:12.996772+0000","last_undegraded":"2024-05-28T06:04:12.996772+0000","last_fullsized":"2024-05-28T06:04:12.996772+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:34:44.387474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.908470+0000","last_change":"2024-05-28T06:04:09.883806+0000","last_active":"2024-05-28T06:04:11.908470+0000","last_peered":"2024-05-28T06:04:11.908470+0000","last_clean":"2024-05-28T06:04:11.908470+0000","last_became_active":"2024-05-28T06:04:09.883352+0000","last_became_peered":"2024-05-28T06:04:09.883352+0000","last_unstale":"2024-05-28T06:04:11.908470+0000","last_undegraded":"2024-05-28T06:04:11.908470+0000","last_fullsized":"2024-05-28T06:04:11.908470+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:36:53.154893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.908799+0000","last_change":"2024-05-28T06:04:10.908671+0000","last_active":"2024-05-28T06:04:11.908799+0000","last_peered":"2024-05-28T06:04:11.908799+0000","last_clean":"2024-05-28T06:04:11.908799+0000","last_became_active":"2024-05-28T06:04:10.908402+0000","last_became_peered":"2024-05-28T06:04:10.908402+0000","last_unstale":"2024-05-28T06:04:11.908799+0000","last_undegraded":"2024-05-28T06:04:11.908799+0000","last_fullsized":"2024-05-28T06:04:11.908799+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:06:44.034610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.995124+0000","last_change":"2024-05-28T06:04:09.868302+0000","last_active":"2024-05-28T06:04:12.995124+0000","last_peered":"2024-05-28T06:04:12.995124+0000","last_clean":"2024-05-28T06:04:12.995124+0000","last_became_active":"2024-05-28T06:04:09.868003+0000","last_became_peered":"2024-05-28T06:04:09.868003+0000","last_unstale":"2024-05-28T06:04:12.995124+0000","last_undegraded":"2024-05-28T06:04:12.995124+0000","last_fullsized":"2024-05-28T06:04:12.995124+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:29:32.319083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.995729+0000","last_change":"2024-05-28T06:04:10.916061+0000","last_active":"2024-05-28T06:04:12.995729+0000","last_peered":"2024-05-28T06:04:12.995729+0000","last_clean":"2024-05-28T06:04:12.995729+0000","last_became_active":"2024-05-28T06:04:10.915607+0000","last_became_peered":"2024-05-28T06:04:10.915607+0000","last_unstale":"2024-05-28T06:04:12.995729+0000","last_undegraded":"2024-05-28T06:04:12.995729+0000","last_fullsized":"2024-05-28T06:04:12.995729+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:51:12.839226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.893456+0000","last_change":"2024-05-28T06:04:09.874073+0000","last_active":"2024-05-28T06:04:11.893456+0000","last_peered":"2024-05-28T06:04:11.893456+0000","last_clean":"2024-05-28T06:04:11.893456+0000","last_became_active":"2024-05-28T06:04:09.872906+0000","last_became_peered":"2024-05-28T06:04:09.872906+0000","last_unstale":"2024-05-28T06:04:11.893456+0000","last_undegraded":"2024-05-28T06:04:11.893456+0000","last_fullsized":"2024-05-28T06:04:11.893456+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:06:04.907427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.998407+0000","last_change":"2024-05-28T06:04:10.914311+0000","last_active":"2024-05-28T06:04:12.998407+0000","last_peered":"2024-05-28T06:04:12.998407+0000","last_clean":"2024-05-28T06:04:12.998407+0000","last_became_active":"2024-05-28T06:04:10.913972+0000","last_became_peered":"2024-05-28T06:04:10.913972+0000","last_unstale":"2024-05-28T06:04:12.998407+0000","last_undegraded":"2024-05-28T06:04:12.998407+0000","last_fullsized":"2024-05-28T06:04:12.998407+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:11:58.309255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.996846+0000","last_change":"2024-05-28T06:04:09.867608+0000","last_active":"2024-05-28T06:04:12.996846+0000","last_peered":"2024-05-28T06:04:12.996846+0000","last_clean":"2024-05-28T06:04:12.996846+0000","last_became_active":"2024-05-28T06:04:09.865750+0000","last_became_peered":"2024-05-28T06:04:09.865750+0000","last_unstale":"2024-05-28T06:04:12.996846+0000","last_undegraded":"2024-05-28T06:04:12.996846+0000","last_fullsized":"2024-05-28T06:04:12.996846+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:16:52.703842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.998212+0000","last_change":"2024-05-28T06:04:10.914237+0000","last_active":"2024-05-28T06:04:12.998212+0000","last_peered":"2024-05-28T06:04:12.998212+0000","last_clean":"2024-05-28T06:04:12.998212+0000","last_became_active":"2024-05-28T06:04:10.913957+0000","last_became_peered":"2024-05-28T06:04:10.913957+0000","last_unstale":"2024-05-28T06:04:12.998212+0000","last_undegraded":"2024-05-28T06:04:12.998212+0000","last_fullsized":"2024-05-28T06:04:12.998212+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:28:36.252853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.890109+0000","last_change":"2024-05-28T06:04:09.877173+0000","last_active":"2024-05-28T06:04:11.890109+0000","last_peered":"2024-05-28T06:04:11.890109+0000","last_clean":"2024-05-28T06:04:11.890109+0000","last_became_active":"2024-05-28T06:04:09.876534+0000","last_became_peered":"2024-05-28T06:04:09.876534+0000","last_unstale":"2024-05-28T06:04:11.890109+0000","last_undegraded":"2024-05-28T06:04:11.890109+0000","last_fullsized":"2024-05-28T06:04:11.890109+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:58:50.805034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.998844+0000","last_change":"2024-05-28T06:04:10.923771+0000","last_active":"2024-05-28T06:04:12.998844+0000","last_peered":"2024-05-28T06:04:12.998844+0000","last_clean":"2024-05-28T06:04:12.998844+0000","last_became_active":"2024-05-28T06:04:10.923494+0000","last_became_peered":"2024-05-28T06:04:10.923494+0000","last_unstale":"2024-05-28T06:04:12.998844+0000","last_undegraded":"2024-05-28T06:04:12.998844+0000","last_fullsized":"2024-05-28T06:04:12.998844+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:05:14.695994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.995877+0000","last_change":"2024-05-28T06:04:09.880725+0000","last_active":"2024-05-28T06:04:12.995877+0000","last_peered":"2024-05-28T06:04:12.995877+0000","last_clean":"2024-05-28T06:04:12.995877+0000","last_became_active":"2024-05-28T06:04:09.880430+0000","last_became_peered":"2024-05-28T06:04:09.880430+0000","last_unstale":"2024-05-28T06:04:12.995877+0000","last_undegraded":"2024-05-28T06:04:12.995877+0000","last_fullsized":"2024-05-28T06:04:12.995877+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:27:30.379288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.996167+0000","last_change":"2024-05-28T06:04:10.904576+0000","last_active":"2024-05-28T06:04:12.996167+0000","last_peered":"2024-05-28T06:04:12.996167+0000","last_clean":"2024-05-28T06:04:12.996167+0000","last_became_active":"2024-05-28T06:04:10.904257+0000","last_became_peered":"2024-05-28T06:04:10.904257+0000","last_unstale":"2024-05-28T06:04:12.996167+0000","last_undegraded":"2024-05-28T06:04:12.996167+0000","last_fullsized":"2024-05-28T06:04:12.996167+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:49:22.704773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":170,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.995506+0000","last_change":"2024-05-28T06:04:01.039129+0000","last_active":"2024-05-28T06:04:12.995506+0000","last_peered":"2024-05-28T06:04:12.995506+0000","last_clean":"2024-05-28T06:04:12.995506+0000","last_became_active":"2024-05-28T06:04:01.033906+0000","last_became_peered":"2024-05-28T06:04:01.033906+0000","last_unstale":"2024-05-28T06:04:12.995506+0000","last_undegraded":"2024-05-28T06:04:12.995506+0000","last_fullsized":"2024-05-28T06:04:12.995506+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:03:59.864237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:03:59.864237+0000","last_clean_scrub_stamp":"2024-05-28T06:03:59.864237+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:17:36.440245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.996021+0000","last_change":"2024-05-28T06:04:09.879679+0000","last_active":"2024-05-28T06:04:12.996021+0000","last_peered":"2024-05-28T06:04:12.996021+0000","last_clean":"2024-05-28T06:04:12.996021+0000","last_became_active":"2024-05-28T06:04:09.877412+0000","last_became_peered":"2024-05-28T06:04:09.877412+0000","last_unstale":"2024-05-28T06:04:12.996021+0000","last_undegraded":"2024-05-28T06:04:12.996021+0000","last_fullsized":"2024-05-28T06:04:12.996021+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:58:06.720498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.996386+0000","last_change":"2024-05-28T06:04:10.901746+0000","last_active":"2024-05-28T06:04:12.996386+0000","last_peered":"2024-05-28T06:04:12.996386+0000","last_clean":"2024-05-28T06:04:12.996386+0000","last_became_active":"2024-05-28T06:04:10.901222+0000","last_became_peered":"2024-05-28T06:04:10.901222+0000","last_unstale":"2024-05-28T06:04:12.996386+0000","last_undegraded":"2024-05-28T06:04:12.996386+0000","last_fullsized":"2024-05-28T06:04:12.996386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:46:29.255545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.995167+0000","last_change":"2024-05-28T06:04:09.890312+0000","last_active":"2024-05-28T06:04:12.995167+0000","last_peered":"2024-05-28T06:04:12.995167+0000","last_clean":"2024-05-28T06:04:12.995167+0000","last_became_active":"2024-05-28T06:04:09.889754+0000","last_became_peered":"2024-05-28T06:04:09.889754+0000","last_unstale":"2024-05-28T06:04:12.995167+0000","last_undegraded":"2024-05-28T06:04:12.995167+0000","last_fullsized":"2024-05-28T06:04:12.995167+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:38:20.882939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.793983+0000","last_change":"2024-05-28T06:04:10.886055+0000","last_active":"2024-05-28T06:04:12.793983+0000","last_peered":"2024-05-28T06:04:12.793983+0000","last_clean":"2024-05-28T06:04:12.793983+0000","last_became_active":"2024-05-28T06:04:10.885596+0000","last_became_peered":"2024-05-28T06:04:10.885596+0000","last_unstale":"2024-05-28T06:04:12.793983+0000","last_undegraded":"2024-05-28T06:04:12.793983+0000","last_fullsized":"2024-05-28T06:04:12.793983+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:06:44.034610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.995673+0000","last_change":"2024-05-28T06:04:09.871789+0000","last_active":"2024-05-28T06:04:12.995673+0000","last_peered":"2024-05-28T06:04:12.995673+0000","last_clean":"2024-05-28T06:04:12.995673+0000","last_became_active":"2024-05-28T06:04:09.870232+0000","last_became_peered":"2024-05-28T06:04:09.870232+0000","last_unstale":"2024-05-28T06:04:12.995673+0000","last_undegraded":"2024-05-28T06:04:12.995673+0000","last_fullsized":"2024-05-28T06:04:12.995673+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:36:53.154893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.890126+0000","last_change":"2024-05-28T06:04:10.898534+0000","last_active":"2024-05-28T06:04:11.890126+0000","last_peered":"2024-05-28T06:04:11.890126+0000","last_clean":"2024-05-28T06:04:11.890126+0000","last_became_active":"2024-05-28T06:04:10.898237+0000","last_became_peered":"2024-05-28T06:04:10.898237+0000","last_unstale":"2024-05-28T06:04:11.890126+0000","last_undegraded":"2024-05-28T06:04:11.890126+0000","last_fullsized":"2024-05-28T06:04:11.890126+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:46:29.255545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.997224+0000","last_change":"2024-05-28T06:04:09.884779+0000","last_active":"2024-05-28T06:04:12.997224+0000","last_peered":"2024-05-28T06:04:12.997224+0000","last_clean":"2024-05-28T06:04:12.997224+0000","last_became_active":"2024-05-28T06:04:09.882344+0000","last_became_peered":"2024-05-28T06:04:09.882344+0000","last_unstale":"2024-05-28T06:04:12.997224+0000","last_undegraded":"2024-05-28T06:04:12.997224+0000","last_fullsized":"2024-05-28T06:04:12.997224+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:29:32.319083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.893200+0000","last_change":"2024-05-28T06:04:10.890674+0000","last_active":"2024-05-28T06:04:11.893200+0000","last_peered":"2024-05-28T06:04:11.893200+0000","last_clean":"2024-05-28T06:04:11.893200+0000","last_became_active":"2024-05-28T06:04:10.890144+0000","last_became_peered":"2024-05-28T06:04:10.890144+0000","last_unstale":"2024-05-28T06:04:11.893200+0000","last_undegraded":"2024-05-28T06:04:11.893200+0000","last_fullsized":"2024-05-28T06:04:11.893200+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:38:21.897771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.890349+0000","last_change":"2024-05-28T06:04:09.871025+0000","last_active":"2024-05-28T06:04:11.890349+0000","last_peered":"2024-05-28T06:04:11.890349+0000","last_clean":"2024-05-28T06:04:11.890349+0000","last_became_active":"2024-05-28T06:04:09.870646+0000","last_became_peered":"2024-05-28T06:04:09.870646+0000","last_unstale":"2024-05-28T06:04:11.890349+0000","last_undegraded":"2024-05-28T06:04:11.890349+0000","last_fullsized":"2024-05-28T06:04:11.890349+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:16:52.703842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.998525+0000","last_change":"2024-05-28T06:04:10.912585+0000","last_active":"2024-05-28T06:04:12.998525+0000","last_peered":"2024-05-28T06:04:12.998525+0000","last_clean":"2024-05-28T06:04:12.998525+0000","last_became_active":"2024-05-28T06:04:10.912038+0000","last_became_peered":"2024-05-28T06:04:10.912038+0000","last_unstale":"2024-05-28T06:04:12.998525+0000","last_undegraded":"2024-05-28T06:04:12.998525+0000","last_fullsized":"2024-05-28T06:04:12.998525+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:04:49.415234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.997550+0000","last_change":"2024-05-28T06:04:09.890971+0000","last_active":"2024-05-28T06:04:12.997550+0000","last_peered":"2024-05-28T06:04:12.997550+0000","last_clean":"2024-05-28T06:04:12.997550+0000","last_became_active":"2024-05-28T06:04:09.890401+0000","last_became_peered":"2024-05-28T06:04:09.890401+0000","last_unstale":"2024-05-28T06:04:12.997550+0000","last_undegraded":"2024-05-28T06:04:12.997550+0000","last_fullsized":"2024-05-28T06:04:12.997550+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:05:29.419386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.890261+0000","last_change":"2024-05-28T06:04:10.903420+0000","last_active":"2024-05-28T06:04:11.890261+0000","last_peered":"2024-05-28T06:04:11.890261+0000","last_clean":"2024-05-28T06:04:11.890261+0000","last_became_active":"2024-05-28T06:04:10.898157+0000","last_became_peered":"2024-05-28T06:04:10.898157+0000","last_unstale":"2024-05-28T06:04:11.890261+0000","last_undegraded":"2024-05-28T06:04:11.890261+0000","last_fullsized":"2024-05-28T06:04:11.890261+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:11:51.651640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.890155+0000","last_change":"2024-05-28T06:04:09.886223+0000","last_active":"2024-05-28T06:04:11.890155+0000","last_peered":"2024-05-28T06:04:11.890155+0000","last_clean":"2024-05-28T06:04:11.890155+0000","last_became_active":"2024-05-28T06:04:09.885439+0000","last_became_peered":"2024-05-28T06:04:09.885439+0000","last_unstale":"2024-05-28T06:04:11.890155+0000","last_undegraded":"2024-05-28T06:04:11.890155+0000","last_fullsized":"2024-05-28T06:04:11.890155+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:38:20.882939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.997783+0000","last_change":"2024-05-28T06:04:10.901609+0000","last_active":"2024-05-28T06:04:12.997783+0000","last_peered":"2024-05-28T06:04:12.997783+0000","last_clean":"2024-05-28T06:04:12.997783+0000","last_became_active":"2024-05-28T06:04:10.901302+0000","last_became_peered":"2024-05-28T06:04:10.901302+0000","last_unstale":"2024-05-28T06:04:12.997783+0000","last_undegraded":"2024-05-28T06:04:12.997783+0000","last_fullsized":"2024-05-28T06:04:12.997783+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:49:22.704773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.909384+0000","last_change":"2024-05-28T06:04:09.888169+0000","last_active":"2024-05-28T06:04:11.909384+0000","last_peered":"2024-05-28T06:04:11.909384+0000","last_clean":"2024-05-28T06:04:11.909384+0000","last_became_active":"2024-05-28T06:04:09.887867+0000","last_became_peered":"2024-05-28T06:04:09.887867+0000","last_unstale":"2024-05-28T06:04:11.909384+0000","last_undegraded":"2024-05-28T06:04:11.909384+0000","last_fullsized":"2024-05-28T06:04:11.909384+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:16:52.703842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.793993+0000","last_change":"2024-05-28T06:04:10.897718+0000","last_active":"2024-05-28T06:04:12.793993+0000","last_peered":"2024-05-28T06:04:12.793993+0000","last_clean":"2024-05-28T06:04:12.793993+0000","last_became_active":"2024-05-28T06:04:10.896414+0000","last_became_peered":"2024-05-28T06:04:10.896414+0000","last_unstale":"2024-05-28T06:04:12.793993+0000","last_undegraded":"2024-05-28T06:04:12.793993+0000","last_fullsized":"2024-05-28T06:04:12.793993+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:53:07.733019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.995167+0000","last_change":"2024-05-28T06:04:09.889726+0000","last_active":"2024-05-28T06:04:12.995167+0000","last_peered":"2024-05-28T06:04:12.995167+0000","last_clean":"2024-05-28T06:04:12.995167+0000","last_became_active":"2024-05-28T06:04:09.889394+0000","last_became_peered":"2024-05-28T06:04:09.889394+0000","last_unstale":"2024-05-28T06:04:12.995167+0000","last_undegraded":"2024-05-28T06:04:12.995167+0000","last_fullsized":"2024-05-28T06:04:12.995167+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:58:06.720498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.892869+0000","last_change":"2024-05-28T06:04:10.908922+0000","last_active":"2024-05-28T06:04:11.892869+0000","last_peered":"2024-05-28T06:04:11.892869+0000","last_clean":"2024-05-28T06:04:11.892869+0000","last_became_active":"2024-05-28T06:04:10.905738+0000","last_became_peered":"2024-05-28T06:04:10.905738+0000","last_unstale":"2024-05-28T06:04:11.892869+0000","last_undegraded":"2024-05-28T06:04:11.892869+0000","last_fullsized":"2024-05-28T06:04:11.892869+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:58:07.735330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.890426+0000","last_change":"2024-05-28T06:04:09.886430+0000","last_active":"2024-05-28T06:04:11.890426+0000","last_peered":"2024-05-28T06:04:11.890426+0000","last_clean":"2024-05-28T06:04:11.890426+0000","last_became_active":"2024-05-28T06:04:09.886072+0000","last_became_peered":"2024-05-28T06:04:09.886072+0000","last_unstale":"2024-05-28T06:04:11.890426+0000","last_undegraded":"2024-05-28T06:04:11.890426+0000","last_fullsized":"2024-05-28T06:04:11.890426+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:29:32.319083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.995901+0000","last_change":"2024-05-28T06:04:10.895054+0000","last_active":"2024-05-28T06:04:12.995901+0000","last_peered":"2024-05-28T06:04:12.995901+0000","last_clean":"2024-05-28T06:04:12.995901+0000","last_became_active":"2024-05-28T06:04:10.894384+0000","last_became_peered":"2024-05-28T06:04:10.894384+0000","last_unstale":"2024-05-28T06:04:12.995901+0000","last_undegraded":"2024-05-28T06:04:12.995901+0000","last_fullsized":"2024-05-28T06:04:12.995901+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:53:07.733019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.889822+0000","last_change":"2024-05-28T06:04:09.873566+0000","last_active":"2024-05-28T06:04:11.889822+0000","last_peered":"2024-05-28T06:04:11.889822+0000","last_clean":"2024-05-28T06:04:11.889822+0000","last_became_active":"2024-05-28T06:04:09.872928+0000","last_became_peered":"2024-05-28T06:04:09.872928+0000","last_unstale":"2024-05-28T06:04:11.889822+0000","last_undegraded":"2024-05-28T06:04:11.889822+0000","last_fullsized":"2024-05-28T06:04:11.889822+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:58:06.720498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.996688+0000","last_change":"2024-05-28T06:04:10.905645+0000","last_active":"2024-05-28T06:04:12.996688+0000","last_peered":"2024-05-28T06:04:12.996688+0000","last_clean":"2024-05-28T06:04:12.996688+0000","last_became_active":"2024-05-28T06:04:10.905308+0000","last_became_peered":"2024-05-28T06:04:10.905308+0000","last_unstale":"2024-05-28T06:04:12.996688+0000","last_undegraded":"2024-05-28T06:04:12.996688+0000","last_fullsized":"2024-05-28T06:04:12.996688+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:11:51.651640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.997139+0000","last_change":"2024-05-28T06:04:09.893691+0000","last_active":"2024-05-28T06:04:12.997139+0000","last_peered":"2024-05-28T06:04:12.997139+0000","last_clean":"2024-05-28T06:04:12.997139+0000","last_became_active":"2024-05-28T06:04:09.893416+0000","last_became_peered":"2024-05-28T06:04:09.893416+0000","last_unstale":"2024-05-28T06:04:12.997139+0000","last_undegraded":"2024-05-28T06:04:12.997139+0000","last_fullsized":"2024-05-28T06:04:12.997139+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:53:48.086901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.908760+0000","last_change":"2024-05-28T06:04:10.919265+0000","last_active":"2024-05-28T06:04:11.908760+0000","last_peered":"2024-05-28T06:04:11.908760+0000","last_clean":"2024-05-28T06:04:11.908760+0000","last_became_active":"2024-05-28T06:04:10.918268+0000","last_became_peered":"2024-05-28T06:04:10.918268+0000","last_unstale":"2024-05-28T06:04:11.908760+0000","last_undegraded":"2024-05-28T06:04:11.908760+0000","last_fullsized":"2024-05-28T06:04:11.908760+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:39:46.389531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.998229+0000","last_change":"2024-05-28T06:04:10.912331+0000","last_active":"2024-05-28T06:04:12.998229+0000","last_peered":"2024-05-28T06:04:12.998229+0000","last_clean":"2024-05-28T06:04:12.998229+0000","last_became_active":"2024-05-28T06:04:10.907887+0000","last_became_peered":"2024-05-28T06:04:10.907887+0000","last_unstale":"2024-05-28T06:04:12.998229+0000","last_undegraded":"2024-05-28T06:04:12.998229+0000","last_fullsized":"2024-05-28T06:04:12.998229+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:03:18.610996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.996617+0000","last_change":"2024-05-28T06:04:09.892486+0000","last_active":"2024-05-28T06:04:12.996617+0000","last_peered":"2024-05-28T06:04:12.996617+0000","last_clean":"2024-05-28T06:04:12.996617+0000","last_became_active":"2024-05-28T06:04:09.892160+0000","last_became_peered":"2024-05-28T06:04:09.892160+0000","last_unstale":"2024-05-28T06:04:12.996617+0000","last_undegraded":"2024-05-28T06:04:12.996617+0000","last_fullsized":"2024-05-28T06:04:12.996617+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:03:45.379665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.997770+0000","last_change":"2024-05-28T06:04:10.909079+0000","last_active":"2024-05-28T06:04:12.997770+0000","last_peered":"2024-05-28T06:04:12.997770+0000","last_clean":"2024-05-28T06:04:12.997770+0000","last_became_active":"2024-05-28T06:04:10.907979+0000","last_became_peered":"2024-05-28T06:04:10.907979+0000","last_unstale":"2024-05-28T06:04:12.997770+0000","last_undegraded":"2024-05-28T06:04:12.997770+0000","last_fullsized":"2024-05-28T06:04:12.997770+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:49:22.704773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.996620+0000","last_change":"2024-05-28T06:04:09.890388+0000","last_active":"2024-05-28T06:04:12.996620+0000","last_peered":"2024-05-28T06:04:12.996620+0000","last_clean":"2024-05-28T06:04:12.996620+0000","last_became_active":"2024-05-28T06:04:09.890086+0000","last_became_peered":"2024-05-28T06:04:09.890086+0000","last_unstale":"2024-05-28T06:04:12.996620+0000","last_undegraded":"2024-05-28T06:04:12.996620+0000","last_fullsized":"2024-05-28T06:04:12.996620+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:58:50.805034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.794239+0000","last_change":"2024-05-28T06:04:10.904802+0000","last_active":"2024-05-28T06:04:12.794239+0000","last_peered":"2024-05-28T06:04:12.794239+0000","last_clean":"2024-05-28T06:04:12.794239+0000","last_became_active":"2024-05-28T06:04:10.904438+0000","last_became_peered":"2024-05-28T06:04:10.904438+0000","last_unstale":"2024-05-28T06:04:12.794239+0000","last_undegraded":"2024-05-28T06:04:12.794239+0000","last_fullsized":"2024-05-28T06:04:12.794239+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:11:51.651640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.997391+0000","last_change":"2024-05-28T06:04:09.874413+0000","last_active":"2024-05-28T06:04:12.997391+0000","last_peered":"2024-05-28T06:04:12.997391+0000","last_clean":"2024-05-28T06:04:12.997391+0000","last_became_active":"2024-05-28T06:04:09.872117+0000","last_became_peered":"2024-05-28T06:04:09.872117+0000","last_unstale":"2024-05-28T06:04:12.997391+0000","last_undegraded":"2024-05-28T06:04:12.997391+0000","last_fullsized":"2024-05-28T06:04:12.997391+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:36:53.154893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.997324+0000","last_change":"2024-05-28T06:04:10.909242+0000","last_active":"2024-05-28T06:04:12.997324+0000","last_peered":"2024-05-28T06:04:12.997324+0000","last_clean":"2024-05-28T06:04:12.997324+0000","last_became_active":"2024-05-28T06:04:10.908934+0000","last_became_peered":"2024-05-28T06:04:10.908934+0000","last_unstale":"2024-05-28T06:04:12.997324+0000","last_undegraded":"2024-05-28T06:04:12.997324+0000","last_fullsized":"2024-05-28T06:04:12.997324+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:45:47.884578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.794420+0000","last_change":"2024-05-28T06:04:09.887679+0000","last_active":"2024-05-28T06:04:12.794420+0000","last_peered":"2024-05-28T06:04:12.794420+0000","last_clean":"2024-05-28T06:04:12.794420+0000","last_became_active":"2024-05-28T06:04:09.887386+0000","last_became_peered":"2024-05-28T06:04:09.887386+0000","last_unstale":"2024-05-28T06:04:12.794420+0000","last_undegraded":"2024-05-28T06:04:12.794420+0000","last_fullsized":"2024-05-28T06:04:12.794420+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:38:33.383965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":802816,"data_stored":661884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":208896,"data_stored":71024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":71060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":70934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":212992,"data_stored":71094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":811008,"data_stored":661940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-28T06:04:21.168 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-28T06:04:21.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.401+0000 7f2951200640 1 Processor -- start 2024-05-28T06:04:21.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.401+0000 7f2951200640 1 -- start start 2024-05-28T06:04:21.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.403+0000 7f2951200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f294c0d2db0 0x7f294c0b3b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:21.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.403+0000 7f2951200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f294c0d4d60 0x7f294c0d3190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:21.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.403+0000 7f2951200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f294c0d36d0 con 0x7f294c0cf490 2024-05-28T06:04:21.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.403+0000 7f2951200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f294c0d5540 con 0x7f294c0d2db0 2024-05-28T06:04:21.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.403+0000 7f2951200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f294c163500 con 0x7f294c0d4d60 2024-05-28T06:04:21.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.403+0000 7f2949e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f294c0d2db0 0x7f294c0b3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:21.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.403+0000 7f2949e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f294c0d2db0 0x7f294c0b3b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48142/0 (socket says 172.21.15.96:48142) 2024-05-28T06:04:21.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.403+0000 7f2949e00640 1 -- 172.21.15.96:0/3850909841 learned_addr learned my addr 172.21.15.96:0/3850909841 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:21.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.404+0000 7f2949400640 1 --2- 172.21.15.96:0/3850909841 >> v2:172.21.15.123:3300/0 conn(0x7f294c0d4d60 0x7f294c0d3190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:21.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.404+0000 7f2949e00640 1 -- 172.21.15.96:0/3850909841 >> v2:172.21.15.123:3300/0 conn(0x7f294c0d4d60 msgr2=0x7f294c0d3190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:21.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.404+0000 7f2949e00640 1 --2- 172.21.15.96:0/3850909841 >> v2:172.21.15.123:3300/0 conn(0x7f294c0d4d60 0x7f294c0d3190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:21.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.404+0000 7f2949e00640 1 -- 172.21.15.96:0/3850909841 >> v1:172.21.15.123:6789/0 conn(0x7f294c0cf490 legacy=0x7f294c0cf920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:21.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.404+0000 7f2949e00640 1 -- 172.21.15.96:0/3850909841 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f294c162740 con 0x7f294c0d2db0 2024-05-28T06:04:21.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.405+0000 7f2949e00640 1 --2- 172.21.15.96:0/3850909841 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f294c0d2db0 0x7f294c0b3b50 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f294c0b71b0 tx=0x7f293c02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e715708344c94e09 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:21.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.405+0000 7f2948a00640 1 -- 172.21.15.96:0/3850909841 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f293c037070 con 0x7f294c0d2db0 2024-05-28T06:04:21.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.405+0000 7f2948a00640 1 -- 172.21.15.96:0/3850909841 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f293c032440 con 0x7f294c0d2db0 2024-05-28T06:04:21.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.405+0000 7f2948a00640 1 -- 172.21.15.96:0/3850909841 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f293c032810 con 0x7f294c0d2db0 2024-05-28T06:04:21.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.406+0000 7f2951200640 1 -- 172.21.15.96:0/3850909841 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f294c0d2db0 msgr2=0x7f294c0b3b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:21.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.406+0000 7f2951200640 1 --2- 172.21.15.96:0/3850909841 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f294c0d2db0 0x7f294c0b3b50 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f294c0b71b0 tx=0x7f293c02a1a0 comp rx=0 tx=0).stop 2024-05-28T06:04:21.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.406+0000 7f2951200640 1 -- 172.21.15.96:0/3850909841 shutdown_connections 2024-05-28T06:04:21.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.406+0000 7f2951200640 1 --2- 172.21.15.96:0/3850909841 >> v2:172.21.15.123:3300/0 conn(0x7f294c0d4d60 0x7f294c0d3190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:21.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.406+0000 7f2951200640 1 --2- 172.21.15.96:0/3850909841 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f294c0d2db0 0x7f294c0b3b50 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:21.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.406+0000 7f2951200640 1 -- 172.21.15.96:0/3850909841 >> 172.21.15.96:0/3850909841 conn(0x7f294c08bfc0 msgr2=0x7f294c0d4650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:21.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.406+0000 7f2951200640 1 -- 172.21.15.96:0/3850909841 shutdown_connections 2024-05-28T06:04:21.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.406+0000 7f2951200640 1 -- 172.21.15.96:0/3850909841 wait complete. 2024-05-28T06:04:21.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.407+0000 7f2951200640 1 Processor -- start 2024-05-28T06:04:21.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.407+0000 7f2951200640 1 -- start start 2024-05-28T06:04:21.410 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.408+0000 7f2951200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f294c0cf490 0x7f294c0b9ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:21.410 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.408+0000 7f2951200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f294c0d2db0 0x7f294c1527a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:21.410 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.408+0000 7f2951200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f294c0d4d60 0x7f294c0c8180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:21.410 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.408+0000 7f2951200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f294c162fe0 con 0x7f294c0cf490 2024-05-28T06:04:21.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.408+0000 7f2951200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f294c162d00 con 0x7f294c0d2db0 2024-05-28T06:04:21.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.408+0000 7f2951200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f294c0d51a0 con 0x7f294c0d4d60 2024-05-28T06:04:21.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.409+0000 7f2949e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f294c0cf490 0x7f294c0b9ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:21.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.409+0000 7f2949e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f294c0cf490 0x7f294c0b9ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48148/0 (socket says 172.21.15.96:48148) 2024-05-28T06:04:21.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.409+0000 7f2949e00640 1 -- 172.21.15.96:0/183750105 learned_addr learned my addr 172.21.15.96:0/183750105 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:21.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.409+0000 7f2949400640 1 --2- 172.21.15.96:0/183750105 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f294c0d2db0 0x7f294c1527a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:21.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.409+0000 7f294a800640 1 --2- 172.21.15.96:0/183750105 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f294c0d4d60 0x7f294c0c8180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:21.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.409+0000 7f2949e00640 1 -- 172.21.15.96:0/183750105 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f294c0d4d60 msgr2=0x7f294c0c8180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:21.412 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.409+0000 7f2949e00640 1 --2- 172.21.15.96:0/183750105 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f294c0d4d60 0x7f294c0c8180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:21.412 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.409+0000 7f2949e00640 1 -- 172.21.15.96:0/183750105 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f294c0d2db0 msgr2=0x7f294c1527a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:21.412 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.409+0000 7f2949e00640 1 --2- 172.21.15.96:0/183750105 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f294c0d2db0 0x7f294c1527a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:21.412 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.409+0000 7f2949e00640 1 -- 172.21.15.96:0/183750105 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f294c0ce100 con 0x7f294c0cf490 2024-05-28T06:04:21.412 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.409+0000 7f294a800640 1 --2- 172.21.15.96:0/183750105 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f294c0d4d60 0x7f294c0c8180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:04:21.412 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.409+0000 7f2949e00640 1 --2- 172.21.15.96:0/183750105 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f294c0cf490 0x7f294c0b9ce0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f293c000c00 tx=0x7f293c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:21.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.410+0000 7f2942000640 1 -- 172.21.15.96:0/183750105 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f293c037040 con 0x7f294c0cf490 2024-05-28T06:04:21.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.410+0000 7f2942000640 1 -- 172.21.15.96:0/183750105 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f293c042070 con 0x7f294c0cf490 2024-05-28T06:04:21.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.410+0000 7f2951200640 1 -- 172.21.15.96:0/183750105 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f294c0b40b0 con 0x7f294c0cf490 2024-05-28T06:04:21.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.410+0000 7f2942000640 1 -- 172.21.15.96:0/183750105 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f293c030bb0 con 0x7f294c0cf490 2024-05-28T06:04:21.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.410+0000 7f2951200640 1 -- 172.21.15.96:0/183750105 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f294c0cde20 con 0x7f294c0cf490 2024-05-28T06:04:21.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.411+0000 7f2951200640 1 -- 172.21.15.96:0/183750105 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f294c0689e0 con 0x7f294c0cf490 2024-05-28T06:04:21.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.412+0000 7f2942000640 1 -- 172.21.15.96:0/183750105 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f293c047460 con 0x7f294c0cf490 2024-05-28T06:04:21.414 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.413+0000 7f2942000640 1 --2- 172.21.15.96:0/183750105 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f292806c8f0 0x7f292806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:21.414 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.413+0000 7f2942000640 1 -- 172.21.15.96:0/183750105 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f293c0ac7d0 con 0x7f294c0cf490 2024-05-28T06:04:21.414 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.413+0000 7f2949400640 1 --2- 172.21.15.96:0/183750105 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f292806c8f0 0x7f292806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:21.415 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.414+0000 7f2949400640 1 --2- 172.21.15.96:0/183750105 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f292806c8f0 0x7f292806edb0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f294c1f8090 tx=0x7f2934008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:21.418 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.417+0000 7f2942000640 1 -- 172.21.15.96:0/183750105 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f293c045360 con 0x7f294c0cf490 2024-05-28T06:04:21.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.598+0000 7f2951200640 1 -- 172.21.15.96:0/183750105 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f294c0d2af0 con 0x7f292806c8f0 2024-05-28T06:04:21.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.616+0000 7f2942000640 1 -- 172.21.15.96:0/183750105 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251837 (secure 0 0 0) 0x7f294c0d2af0 con 0x7f292806c8f0 2024-05-28T06:04:21.621 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:04:21.622 INFO:teuthology.orchestra.run.smithi096.stderr:dumped all 2024-05-28T06:04:21.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.622+0000 7f2951200640 1 -- 172.21.15.96:0/183750105 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f292806c8f0 msgr2=0x7f292806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:21.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.622+0000 7f2951200640 1 --2- 172.21.15.96:0/183750105 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f292806c8f0 0x7f292806edb0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f294c1f8090 tx=0x7f2934008040 comp rx=0 tx=0).stop 2024-05-28T06:04:21.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.622+0000 7f2951200640 1 -- 172.21.15.96:0/183750105 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f294c0cf490 msgr2=0x7f294c0b9ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:21.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.622+0000 7f2951200640 1 --2- 172.21.15.96:0/183750105 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f294c0cf490 0x7f294c0b9ce0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f293c000c00 tx=0x7f293c0046c0 comp rx=0 tx=0).stop 2024-05-28T06:04:21.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.622+0000 7f2951200640 1 -- 172.21.15.96:0/183750105 shutdown_connections 2024-05-28T06:04:21.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.622+0000 7f2951200640 1 --2- 172.21.15.96:0/183750105 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f292806c8f0 0x7f292806edb0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:21.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.622+0000 7f2951200640 1 --2- 172.21.15.96:0/183750105 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f294c0d4d60 0x7f294c0c8180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:21.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.622+0000 7f2951200640 1 --2- 172.21.15.96:0/183750105 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f294c0d2db0 0x7f294c1527a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:21.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.622+0000 7f2951200640 1 --2- 172.21.15.96:0/183750105 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f294c0cf490 0x7f294c0b9ce0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:21.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.622+0000 7f2951200640 1 -- 172.21.15.96:0/183750105 >> 172.21.15.96:0/183750105 conn(0x7f294c08bfc0 msgr2=0x7f294c15de90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:21.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.622+0000 7f2951200640 1 -- 172.21.15.96:0/183750105 shutdown_connections 2024-05-28T06:04:21.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.622+0000 7f2951200640 1 -- 172.21.15.96:0/183750105 wait complete. 2024-05-28T06:04:21.640 INFO:teuthology.orchestra.run.smithi096.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-05-28T06:04:21.486190+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218992,"kb_used_data":2824,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755728,"statfs":{"total":773094113280,"available":772869865472,"internally_reserved":0,"allocated":2891776,"data_stored":1753028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003807"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.997911+0000","last_change":"2024-05-28T06:04:10.916483+0000","last_active":"2024-05-28T06:04:12.997911+0000","last_peered":"2024-05-28T06:04:12.997911+0000","last_clean":"2024-05-28T06:04:12.997911+0000","last_became_active":"2024-05-28T06:04:10.916214+0000","last_became_peered":"2024-05-28T06:04:10.916214+0000","last_unstale":"2024-05-28T06:04:12.997911+0000","last_undegraded":"2024-05-28T06:04:12.997911+0000","last_fullsized":"2024-05-28T06:04:12.997911+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:43:05.061767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.889968+0000","last_change":"2024-05-28T06:04:10.900791+0000","last_active":"2024-05-28T06:04:11.889968+0000","last_peered":"2024-05-28T06:04:11.889968+0000","last_clean":"2024-05-28T06:04:11.889968+0000","last_became_active":"2024-05-28T06:04:10.900489+0000","last_became_peered":"2024-05-28T06:04:10.900489+0000","last_unstale":"2024-05-28T06:04:11.889968+0000","last_undegraded":"2024-05-28T06:04:11.889968+0000","last_fullsized":"2024-05-28T06:04:11.889968+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:53:07.733019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.794222+0000","last_change":"2024-05-28T06:04:10.901559+0000","last_active":"2024-05-28T06:04:12.794222+0000","last_peered":"2024-05-28T06:04:12.794222+0000","last_clean":"2024-05-28T06:04:12.794222+0000","last_became_active":"2024-05-28T06:04:10.899022+0000","last_became_peered":"2024-05-28T06:04:10.899022+0000","last_unstale":"2024-05-28T06:04:12.794222+0000","last_undegraded":"2024-05-28T06:04:12.794222+0000","last_fullsized":"2024-05-28T06:04:12.794222+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:28:21.123299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.997115+0000","last_change":"2024-05-28T06:04:10.923988+0000","last_active":"2024-05-28T06:04:12.997115+0000","last_peered":"2024-05-28T06:04:12.997115+0000","last_clean":"2024-05-28T06:04:12.997115+0000","last_became_active":"2024-05-28T06:04:10.923718+0000","last_became_peered":"2024-05-28T06:04:10.923718+0000","last_unstale":"2024-05-28T06:04:12.997115+0000","last_undegraded":"2024-05-28T06:04:12.997115+0000","last_fullsized":"2024-05-28T06:04:12.997115+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:45:47.884578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.996357+0000","last_change":"2024-05-28T06:04:10.917922+0000","last_active":"2024-05-28T06:04:12.996357+0000","last_peered":"2024-05-28T06:04:12.996357+0000","last_clean":"2024-05-28T06:04:12.996357+0000","last_became_active":"2024-05-28T06:04:10.917253+0000","last_became_peered":"2024-05-28T06:04:10.917253+0000","last_unstale":"2024-05-28T06:04:12.996357+0000","last_undegraded":"2024-05-28T06:04:12.996357+0000","last_fullsized":"2024-05-28T06:04:12.996357+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:04:49.415234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.996757+0000","last_change":"2024-05-28T06:04:10.907704+0000","last_active":"2024-05-28T06:04:12.996757+0000","last_peered":"2024-05-28T06:04:12.996757+0000","last_clean":"2024-05-28T06:04:12.996757+0000","last_became_active":"2024-05-28T06:04:10.907065+0000","last_became_peered":"2024-05-28T06:04:10.907065+0000","last_unstale":"2024-05-28T06:04:12.996757+0000","last_undegraded":"2024-05-28T06:04:12.996757+0000","last_fullsized":"2024-05-28T06:04:12.996757+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:28:21.123299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.995527+0000","last_change":"2024-05-28T06:04:10.905438+0000","last_active":"2024-05-28T06:04:12.995527+0000","last_peered":"2024-05-28T06:04:12.995527+0000","last_clean":"2024-05-28T06:04:12.995527+0000","last_became_active":"2024-05-28T06:04:10.904842+0000","last_became_peered":"2024-05-28T06:04:10.904842+0000","last_unstale":"2024-05-28T06:04:12.995527+0000","last_undegraded":"2024-05-28T06:04:12.995527+0000","last_fullsized":"2024-05-28T06:04:12.995527+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:46:27.923407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.995330+0000","last_change":"2024-05-28T06:04:10.899401+0000","last_active":"2024-05-28T06:04:12.995330+0000","last_peered":"2024-05-28T06:04:12.995330+0000","last_clean":"2024-05-28T06:04:12.995330+0000","last_became_active":"2024-05-28T06:04:10.898742+0000","last_became_peered":"2024-05-28T06:04:10.898742+0000","last_unstale":"2024-05-28T06:04:12.995330+0000","last_undegraded":"2024-05-28T06:04:12.995330+0000","last_fullsized":"2024-05-28T06:04:12.995330+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:39:46.389531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.997773+0000","last_change":"2024-05-28T06:04:10.907973+0000","last_active":"2024-05-28T06:04:12.997773+0000","last_peered":"2024-05-28T06:04:12.997773+0000","last_clean":"2024-05-28T06:04:12.997773+0000","last_became_active":"2024-05-28T06:04:10.905186+0000","last_became_peered":"2024-05-28T06:04:10.905186+0000","last_unstale":"2024-05-28T06:04:12.997773+0000","last_undegraded":"2024-05-28T06:04:12.997773+0000","last_fullsized":"2024-05-28T06:04:12.997773+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:39:13.976770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.909549+0000","last_change":"2024-05-28T06:04:10.927703+0000","last_active":"2024-05-28T06:04:11.909549+0000","last_peered":"2024-05-28T06:04:11.909549+0000","last_clean":"2024-05-28T06:04:11.909549+0000","last_became_active":"2024-05-28T06:04:10.927383+0000","last_became_peered":"2024-05-28T06:04:10.927383+0000","last_unstale":"2024-05-28T06:04:11.909549+0000","last_undegraded":"2024-05-28T06:04:11.909549+0000","last_fullsized":"2024-05-28T06:04:11.909549+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:11:51.651640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.909515+0000","last_change":"2024-05-28T06:04:10.926573+0000","last_active":"2024-05-28T06:04:11.909515+0000","last_peered":"2024-05-28T06:04:11.909515+0000","last_clean":"2024-05-28T06:04:11.909515+0000","last_became_active":"2024-05-28T06:04:10.926364+0000","last_became_peered":"2024-05-28T06:04:10.926364+0000","last_unstale":"2024-05-28T06:04:11.909515+0000","last_undegraded":"2024-05-28T06:04:11.909515+0000","last_fullsized":"2024-05-28T06:04:11.909515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:28:21.123299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.893744+0000","last_change":"2024-05-28T06:04:10.913441+0000","last_active":"2024-05-28T06:04:11.893744+0000","last_peered":"2024-05-28T06:04:11.893744+0000","last_clean":"2024-05-28T06:04:11.893744+0000","last_became_active":"2024-05-28T06:04:10.912847+0000","last_became_peered":"2024-05-28T06:04:10.912847+0000","last_unstale":"2024-05-28T06:04:11.893744+0000","last_undegraded":"2024-05-28T06:04:11.893744+0000","last_fullsized":"2024-05-28T06:04:11.893744+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:58:51.819866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.996503+0000","last_change":"2024-05-28T06:04:10.909268+0000","last_active":"2024-05-28T06:04:12.996503+0000","last_peered":"2024-05-28T06:04:12.996503+0000","last_clean":"2024-05-28T06:04:12.996503+0000","last_became_active":"2024-05-28T06:04:10.908954+0000","last_became_peered":"2024-05-28T06:04:10.908954+0000","last_unstale":"2024-05-28T06:04:12.996503+0000","last_undegraded":"2024-05-28T06:04:12.996503+0000","last_fullsized":"2024-05-28T06:04:12.996503+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:28:21.123299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.793137+0000","last_change":"2024-05-28T06:04:10.899509+0000","last_active":"2024-05-28T06:04:12.793137+0000","last_peered":"2024-05-28T06:04:12.793137+0000","last_clean":"2024-05-28T06:04:12.793137+0000","last_became_active":"2024-05-28T06:04:10.896990+0000","last_became_peered":"2024-05-28T06:04:10.896990+0000","last_unstale":"2024-05-28T06:04:12.793137+0000","last_undegraded":"2024-05-28T06:04:12.793137+0000","last_fullsized":"2024-05-28T06:04:12.793137+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:17:36.664466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.890266+0000","last_change":"2024-05-28T06:04:10.906808+0000","last_active":"2024-05-28T06:04:11.890266+0000","last_peered":"2024-05-28T06:04:11.890266+0000","last_clean":"2024-05-28T06:04:11.890266+0000","last_became_active":"2024-05-28T06:04:10.906579+0000","last_became_peered":"2024-05-28T06:04:10.906579+0000","last_unstale":"2024-05-28T06:04:11.890266+0000","last_undegraded":"2024-05-28T06:04:11.890266+0000","last_fullsized":"2024-05-28T06:04:11.890266+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:28:21.123299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.997630+0000","last_change":"2024-05-28T06:04:10.904806+0000","last_active":"2024-05-28T06:04:12.997630+0000","last_peered":"2024-05-28T06:04:12.997630+0000","last_clean":"2024-05-28T06:04:12.997630+0000","last_became_active":"2024-05-28T06:04:10.904450+0000","last_became_peered":"2024-05-28T06:04:10.904450+0000","last_unstale":"2024-05-28T06:04:12.997630+0000","last_undegraded":"2024-05-28T06:04:12.997630+0000","last_fullsized":"2024-05-28T06:04:12.997630+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:51:12.839226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.999350+0000","last_change":"2024-05-28T06:04:10.918376+0000","last_active":"2024-05-28T06:04:12.999350+0000","last_peered":"2024-05-28T06:04:12.999350+0000","last_clean":"2024-05-28T06:04:12.999350+0000","last_became_active":"2024-05-28T06:04:10.918042+0000","last_became_peered":"2024-05-28T06:04:10.918042+0000","last_unstale":"2024-05-28T06:04:12.999350+0000","last_undegraded":"2024-05-28T06:04:12.999350+0000","last_fullsized":"2024-05-28T06:04:12.999350+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:39:13.976770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.996501+0000","last_change":"2024-05-28T06:04:10.910995+0000","last_active":"2024-05-28T06:04:12.996501+0000","last_peered":"2024-05-28T06:04:12.996501+0000","last_clean":"2024-05-28T06:04:12.996501+0000","last_became_active":"2024-05-28T06:04:10.910672+0000","last_became_peered":"2024-05-28T06:04:10.910672+0000","last_unstale":"2024-05-28T06:04:12.996501+0000","last_undegraded":"2024-05-28T06:04:12.996501+0000","last_fullsized":"2024-05-28T06:04:12.996501+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:58:51.819866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.997411+0000","last_change":"2024-05-28T06:04:10.911244+0000","last_active":"2024-05-28T06:04:12.997411+0000","last_peered":"2024-05-28T06:04:12.997411+0000","last_clean":"2024-05-28T06:04:12.997411+0000","last_became_active":"2024-05-28T06:04:10.910921+0000","last_became_peered":"2024-05-28T06:04:10.910921+0000","last_unstale":"2024-05-28T06:04:12.997411+0000","last_undegraded":"2024-05-28T06:04:12.997411+0000","last_fullsized":"2024-05-28T06:04:12.997411+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:05:14.695994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.909154+0000","last_change":"2024-05-28T06:04:10.921344+0000","last_active":"2024-05-28T06:04:11.909154+0000","last_peered":"2024-05-28T06:04:11.909154+0000","last_clean":"2024-05-28T06:04:11.909154+0000","last_became_active":"2024-05-28T06:04:10.920438+0000","last_became_peered":"2024-05-28T06:04:10.920438+0000","last_unstale":"2024-05-28T06:04:11.909154+0000","last_undegraded":"2024-05-28T06:04:11.909154+0000","last_fullsized":"2024-05-28T06:04:11.909154+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:53:07.733019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.793512+0000","last_change":"2024-05-28T06:04:10.902770+0000","last_active":"2024-05-28T06:04:12.793512+0000","last_peered":"2024-05-28T06:04:12.793512+0000","last_clean":"2024-05-28T06:04:12.793512+0000","last_became_active":"2024-05-28T06:04:10.901534+0000","last_became_peered":"2024-05-28T06:04:10.901534+0000","last_unstale":"2024-05-28T06:04:12.793512+0000","last_undegraded":"2024-05-28T06:04:12.793512+0000","last_fullsized":"2024-05-28T06:04:12.793512+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:00:40.966129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.996957+0000","last_change":"2024-05-28T06:04:10.915708+0000","last_active":"2024-05-28T06:04:12.996957+0000","last_peered":"2024-05-28T06:04:12.996957+0000","last_clean":"2024-05-28T06:04:12.996957+0000","last_became_active":"2024-05-28T06:04:10.913058+0000","last_became_peered":"2024-05-28T06:04:10.913058+0000","last_unstale":"2024-05-28T06:04:12.996957+0000","last_undegraded":"2024-05-28T06:04:12.996957+0000","last_fullsized":"2024-05-28T06:04:12.996957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:11:51.651640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.995352+0000","last_change":"2024-05-28T06:04:10.904255+0000","last_active":"2024-05-28T06:04:12.995352+0000","last_peered":"2024-05-28T06:04:12.995352+0000","last_clean":"2024-05-28T06:04:12.995352+0000","last_became_active":"2024-05-28T06:04:10.903840+0000","last_became_peered":"2024-05-28T06:04:10.903840+0000","last_unstale":"2024-05-28T06:04:12.995352+0000","last_undegraded":"2024-05-28T06:04:12.995352+0000","last_fullsized":"2024-05-28T06:04:12.995352+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:06:44.034610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.997441+0000","last_change":"2024-05-28T06:04:10.902458+0000","last_active":"2024-05-28T06:04:12.997441+0000","last_peered":"2024-05-28T06:04:12.997441+0000","last_clean":"2024-05-28T06:04:12.997441+0000","last_became_active":"2024-05-28T06:04:10.902054+0000","last_became_peered":"2024-05-28T06:04:10.902054+0000","last_unstale":"2024-05-28T06:04:12.997441+0000","last_undegraded":"2024-05-28T06:04:12.997441+0000","last_fullsized":"2024-05-28T06:04:12.997441+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:46:27.923407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.997558+0000","last_change":"2024-05-28T06:04:10.895892+0000","last_active":"2024-05-28T06:04:12.997558+0000","last_peered":"2024-05-28T06:04:12.997558+0000","last_clean":"2024-05-28T06:04:12.997558+0000","last_became_active":"2024-05-28T06:04:10.895579+0000","last_became_peered":"2024-05-28T06:04:10.895579+0000","last_unstale":"2024-05-28T06:04:12.997558+0000","last_undegraded":"2024-05-28T06:04:12.997558+0000","last_fullsized":"2024-05-28T06:04:12.997558+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:51:12.839226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.909428+0000","last_change":"2024-05-28T06:04:10.914830+0000","last_active":"2024-05-28T06:04:11.909428+0000","last_peered":"2024-05-28T06:04:11.909428+0000","last_clean":"2024-05-28T06:04:11.909428+0000","last_became_active":"2024-05-28T06:04:10.913651+0000","last_became_peered":"2024-05-28T06:04:10.913651+0000","last_unstale":"2024-05-28T06:04:11.909428+0000","last_undegraded":"2024-05-28T06:04:11.909428+0000","last_fullsized":"2024-05-28T06:04:11.909428+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:46:29.255545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.889962+0000","last_change":"2024-05-28T06:04:10.903385+0000","last_active":"2024-05-28T06:04:11.889962+0000","last_peered":"2024-05-28T06:04:11.889962+0000","last_clean":"2024-05-28T06:04:11.889962+0000","last_became_active":"2024-05-28T06:04:10.903078+0000","last_became_peered":"2024-05-28T06:04:10.903078+0000","last_unstale":"2024-05-28T06:04:11.889962+0000","last_undegraded":"2024-05-28T06:04:11.889962+0000","last_fullsized":"2024-05-28T06:04:11.889962+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:06:44.034610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.997438+0000","last_change":"2024-05-28T06:04:10.894062+0000","last_active":"2024-05-28T06:04:12.997438+0000","last_peered":"2024-05-28T06:04:12.997438+0000","last_clean":"2024-05-28T06:04:12.997438+0000","last_became_active":"2024-05-28T06:04:10.893763+0000","last_became_peered":"2024-05-28T06:04:10.893763+0000","last_unstale":"2024-05-28T06:04:12.997438+0000","last_undegraded":"2024-05-28T06:04:12.997438+0000","last_fullsized":"2024-05-28T06:04:12.997438+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:46:27.923407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.793745+0000","last_change":"2024-05-28T06:04:10.897684+0000","last_active":"2024-05-28T06:04:12.793745+0000","last_peered":"2024-05-28T06:04:12.793745+0000","last_clean":"2024-05-28T06:04:12.793745+0000","last_became_active":"2024-05-28T06:04:10.897382+0000","last_became_peered":"2024-05-28T06:04:10.897382+0000","last_unstale":"2024-05-28T06:04:12.793745+0000","last_undegraded":"2024-05-28T06:04:12.793745+0000","last_fullsized":"2024-05-28T06:04:12.793745+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:39:46.389531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.997085+0000","last_change":"2024-05-28T06:04:10.908012+0000","last_active":"2024-05-28T06:04:12.997085+0000","last_peered":"2024-05-28T06:04:12.997085+0000","last_clean":"2024-05-28T06:04:12.997085+0000","last_became_active":"2024-05-28T06:04:10.907436+0000","last_became_peered":"2024-05-28T06:04:10.907436+0000","last_unstale":"2024-05-28T06:04:12.997085+0000","last_undegraded":"2024-05-28T06:04:12.997085+0000","last_fullsized":"2024-05-28T06:04:12.997085+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:58:16.009219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.996141+0000","last_change":"2024-05-28T06:04:10.905729+0000","last_active":"2024-05-28T06:04:12.996141+0000","last_peered":"2024-05-28T06:04:12.996141+0000","last_clean":"2024-05-28T06:04:12.996141+0000","last_became_active":"2024-05-28T06:04:10.905325+0000","last_became_peered":"2024-05-28T06:04:10.905325+0000","last_unstale":"2024-05-28T06:04:12.996141+0000","last_undegraded":"2024-05-28T06:04:12.996141+0000","last_fullsized":"2024-05-28T06:04:12.996141+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:03:46.394497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.908750+0000","last_change":"2024-05-28T06:04:10.915117+0000","last_active":"2024-05-28T06:04:11.908750+0000","last_peered":"2024-05-28T06:04:11.908750+0000","last_clean":"2024-05-28T06:04:11.908750+0000","last_became_active":"2024-05-28T06:04:10.914761+0000","last_became_peered":"2024-05-28T06:04:10.914761+0000","last_unstale":"2024-05-28T06:04:11.908750+0000","last_undegraded":"2024-05-28T06:04:11.908750+0000","last_fullsized":"2024-05-28T06:04:11.908750+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:00:40.966129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.892804+0000","last_change":"2024-05-28T06:04:10.897395+0000","last_active":"2024-05-28T06:04:11.892804+0000","last_peered":"2024-05-28T06:04:11.892804+0000","last_clean":"2024-05-28T06:04:11.892804+0000","last_became_active":"2024-05-28T06:04:10.897108+0000","last_became_peered":"2024-05-28T06:04:10.897108+0000","last_unstale":"2024-05-28T06:04:11.892804+0000","last_undegraded":"2024-05-28T06:04:11.892804+0000","last_fullsized":"2024-05-28T06:04:11.892804+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:16:53.718673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.996650+0000","last_change":"2024-05-28T06:04:09.890037+0000","last_active":"2024-05-28T06:04:12.996650+0000","last_peered":"2024-05-28T06:04:12.996650+0000","last_clean":"2024-05-28T06:04:12.996650+0000","last_became_active":"2024-05-28T06:04:09.889696+0000","last_became_peered":"2024-05-28T06:04:09.889696+0000","last_unstale":"2024-05-28T06:04:12.996650+0000","last_undegraded":"2024-05-28T06:04:12.996650+0000","last_fullsized":"2024-05-28T06:04:12.996650+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:29:32.319083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.794133+0000","last_change":"2024-05-28T06:04:10.896136+0000","last_active":"2024-05-28T06:04:12.794133+0000","last_peered":"2024-05-28T06:04:12.794133+0000","last_clean":"2024-05-28T06:04:12.794133+0000","last_became_active":"2024-05-28T06:04:10.895631+0000","last_became_peered":"2024-05-28T06:04:10.895631+0000","last_unstale":"2024-05-28T06:04:12.794133+0000","last_undegraded":"2024-05-28T06:04:12.794133+0000","last_fullsized":"2024-05-28T06:04:12.794133+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:46:29.255545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.908566+0000","last_change":"2024-05-28T06:04:09.879488+0000","last_active":"2024-05-28T06:04:11.908566+0000","last_peered":"2024-05-28T06:04:11.908566+0000","last_clean":"2024-05-28T06:04:11.908566+0000","last_became_active":"2024-05-28T06:04:09.879182+0000","last_became_peered":"2024-05-28T06:04:09.879182+0000","last_unstale":"2024-05-28T06:04:11.908566+0000","last_undegraded":"2024-05-28T06:04:11.908566+0000","last_fullsized":"2024-05-28T06:04:11.908566+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:29:32.319083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.995389+0000","last_change":"2024-05-28T06:04:09.884424+0000","last_active":"2024-05-28T06:04:12.995389+0000","last_peered":"2024-05-28T06:04:12.995389+0000","last_clean":"2024-05-28T06:04:12.995389+0000","last_became_active":"2024-05-28T06:04:09.884088+0000","last_became_peered":"2024-05-28T06:04:09.884088+0000","last_unstale":"2024-05-28T06:04:12.995389+0000","last_undegraded":"2024-05-28T06:04:12.995389+0000","last_fullsized":"2024-05-28T06:04:12.995389+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:27:30.379288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.996856+0000","last_change":"2024-05-28T06:04:10.890237+0000","last_active":"2024-05-28T06:04:12.996856+0000","last_peered":"2024-05-28T06:04:12.996856+0000","last_clean":"2024-05-28T06:04:12.996856+0000","last_became_active":"2024-05-28T06:04:10.889644+0000","last_became_peered":"2024-05-28T06:04:10.889644+0000","last_unstale":"2024-05-28T06:04:12.996856+0000","last_undegraded":"2024-05-28T06:04:12.996856+0000","last_fullsized":"2024-05-28T06:04:12.996856+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:58:16.009219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.997564+0000","last_change":"2024-05-28T06:04:09.875990+0000","last_active":"2024-05-28T06:04:12.997564+0000","last_peered":"2024-05-28T06:04:12.997564+0000","last_clean":"2024-05-28T06:04:12.997564+0000","last_became_active":"2024-05-28T06:04:09.875644+0000","last_became_peered":"2024-05-28T06:04:09.875644+0000","last_unstale":"2024-05-28T06:04:12.997564+0000","last_undegraded":"2024-05-28T06:04:12.997564+0000","last_fullsized":"2024-05-28T06:04:12.997564+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:27:30.379288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.889859+0000","last_change":"2024-05-28T06:04:10.899922+0000","last_active":"2024-05-28T06:04:11.889859+0000","last_peered":"2024-05-28T06:04:11.889859+0000","last_clean":"2024-05-28T06:04:11.889859+0000","last_became_active":"2024-05-28T06:04:10.899617+0000","last_became_peered":"2024-05-28T06:04:10.899617+0000","last_unstale":"2024-05-28T06:04:11.889859+0000","last_undegraded":"2024-05-28T06:04:11.889859+0000","last_fullsized":"2024-05-28T06:04:11.889859+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:39:46.389531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.909050+0000","last_change":"2024-05-28T06:04:09.887084+0000","last_active":"2024-05-28T06:04:11.909050+0000","last_peered":"2024-05-28T06:04:11.909050+0000","last_clean":"2024-05-28T06:04:11.909050+0000","last_became_active":"2024-05-28T06:04:09.886821+0000","last_became_peered":"2024-05-28T06:04:09.886821+0000","last_unstale":"2024-05-28T06:04:11.909050+0000","last_undegraded":"2024-05-28T06:04:11.909050+0000","last_fullsized":"2024-05-28T06:04:11.909050+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:58:06.720498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.998553+0000","last_change":"2024-05-28T06:04:10.921008+0000","last_active":"2024-05-28T06:04:12.998553+0000","last_peered":"2024-05-28T06:04:12.998553+0000","last_clean":"2024-05-28T06:04:12.998553+0000","last_became_active":"2024-05-28T06:04:10.913026+0000","last_became_peered":"2024-05-28T06:04:10.913026+0000","last_unstale":"2024-05-28T06:04:12.998553+0000","last_undegraded":"2024-05-28T06:04:12.998553+0000","last_fullsized":"2024-05-28T06:04:12.998553+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:34:44.387474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.998439+0000","last_change":"2024-05-28T06:04:09.887088+0000","last_active":"2024-05-28T06:04:12.998439+0000","last_peered":"2024-05-28T06:04:12.998439+0000","last_clean":"2024-05-28T06:04:12.998439+0000","last_became_active":"2024-05-28T06:04:09.886809+0000","last_became_peered":"2024-05-28T06:04:09.886809+0000","last_unstale":"2024-05-28T06:04:12.998439+0000","last_undegraded":"2024-05-28T06:04:12.998439+0000","last_fullsized":"2024-05-28T06:04:12.998439+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:16:52.703842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.794379+0000","last_change":"2024-05-28T06:04:10.904810+0000","last_active":"2024-05-28T06:04:12.794379+0000","last_peered":"2024-05-28T06:04:12.794379+0000","last_clean":"2024-05-28T06:04:12.794379+0000","last_became_active":"2024-05-28T06:04:10.904445+0000","last_became_peered":"2024-05-28T06:04:10.904445+0000","last_unstale":"2024-05-28T06:04:12.794379+0000","last_undegraded":"2024-05-28T06:04:12.794379+0000","last_fullsized":"2024-05-28T06:04:12.794379+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:58:16.009219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.890096+0000","last_change":"2024-05-28T06:04:09.882535+0000","last_active":"2024-05-28T06:04:11.890096+0000","last_peered":"2024-05-28T06:04:11.890096+0000","last_clean":"2024-05-28T06:04:11.890096+0000","last_became_active":"2024-05-28T06:04:09.882159+0000","last_became_peered":"2024-05-28T06:04:09.882159+0000","last_unstale":"2024-05-28T06:04:11.890096+0000","last_undegraded":"2024-05-28T06:04:11.890096+0000","last_fullsized":"2024-05-28T06:04:11.890096+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:03:45.379665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.893646+0000","last_change":"2024-05-28T06:04:10.901897+0000","last_active":"2024-05-28T06:04:11.893646+0000","last_peered":"2024-05-28T06:04:11.893646+0000","last_clean":"2024-05-28T06:04:11.893646+0000","last_became_active":"2024-05-28T06:04:10.901156+0000","last_became_peered":"2024-05-28T06:04:10.901156+0000","last_unstale":"2024-05-28T06:04:11.893646+0000","last_undegraded":"2024-05-28T06:04:11.893646+0000","last_fullsized":"2024-05-28T06:04:11.893646+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:03:46.394497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.909296+0000","last_change":"2024-05-28T06:04:09.878442+0000","last_active":"2024-05-28T06:04:11.909296+0000","last_peered":"2024-05-28T06:04:11.909296+0000","last_clean":"2024-05-28T06:04:11.909296+0000","last_became_active":"2024-05-28T06:04:09.878148+0000","last_became_peered":"2024-05-28T06:04:09.878148+0000","last_unstale":"2024-05-28T06:04:11.909296+0000","last_undegraded":"2024-05-28T06:04:11.909296+0000","last_fullsized":"2024-05-28T06:04:11.909296+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:38:20.882939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.996201+0000","last_change":"2024-05-28T06:04:10.911058+0000","last_active":"2024-05-28T06:04:12.996201+0000","last_peered":"2024-05-28T06:04:12.996201+0000","last_clean":"2024-05-28T06:04:12.996201+0000","last_became_active":"2024-05-28T06:04:10.910775+0000","last_became_peered":"2024-05-28T06:04:10.910775+0000","last_unstale":"2024-05-28T06:04:12.996201+0000","last_undegraded":"2024-05-28T06:04:12.996201+0000","last_fullsized":"2024-05-28T06:04:12.996201+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:03:18.610996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.794179+0000","last_change":"2024-05-28T06:04:09.878327+0000","last_active":"2024-05-28T06:04:12.794179+0000","last_peered":"2024-05-28T06:04:12.794179+0000","last_clean":"2024-05-28T06:04:12.794179+0000","last_became_active":"2024-05-28T06:04:09.878028+0000","last_became_peered":"2024-05-28T06:04:09.878028+0000","last_unstale":"2024-05-28T06:04:12.794179+0000","last_undegraded":"2024-05-28T06:04:12.794179+0000","last_fullsized":"2024-05-28T06:04:12.794179+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:12:29.496721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.793473+0000","last_change":"2024-05-28T06:04:10.902163+0000","last_active":"2024-05-28T06:04:12.793473+0000","last_peered":"2024-05-28T06:04:12.793473+0000","last_clean":"2024-05-28T06:04:12.793473+0000","last_became_active":"2024-05-28T06:04:10.901846+0000","last_became_peered":"2024-05-28T06:04:10.901846+0000","last_unstale":"2024-05-28T06:04:12.793473+0000","last_undegraded":"2024-05-28T06:04:12.793473+0000","last_fullsized":"2024-05-28T06:04:12.793473+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:11:13.101661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.997163+0000","last_change":"2024-05-28T06:04:09.884756+0000","last_active":"2024-05-28T06:04:12.997163+0000","last_peered":"2024-05-28T06:04:12.997163+0000","last_clean":"2024-05-28T06:04:12.997163+0000","last_became_active":"2024-05-28T06:04:09.884354+0000","last_became_peered":"2024-05-28T06:04:09.884354+0000","last_unstale":"2024-05-28T06:04:12.997163+0000","last_undegraded":"2024-05-28T06:04:12.997163+0000","last_fullsized":"2024-05-28T06:04:12.997163+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:36:53.154893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.996772+0000","last_change":"2024-05-28T06:04:10.916180+0000","last_active":"2024-05-28T06:04:12.996772+0000","last_peered":"2024-05-28T06:04:12.996772+0000","last_clean":"2024-05-28T06:04:12.996772+0000","last_became_active":"2024-05-28T06:04:10.915630+0000","last_became_peered":"2024-05-28T06:04:10.915630+0000","last_unstale":"2024-05-28T06:04:12.996772+0000","last_undegraded":"2024-05-28T06:04:12.996772+0000","last_fullsized":"2024-05-28T06:04:12.996772+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:34:44.387474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.908470+0000","last_change":"2024-05-28T06:04:09.883806+0000","last_active":"2024-05-28T06:04:11.908470+0000","last_peered":"2024-05-28T06:04:11.908470+0000","last_clean":"2024-05-28T06:04:11.908470+0000","last_became_active":"2024-05-28T06:04:09.883352+0000","last_became_peered":"2024-05-28T06:04:09.883352+0000","last_unstale":"2024-05-28T06:04:11.908470+0000","last_undegraded":"2024-05-28T06:04:11.908470+0000","last_fullsized":"2024-05-28T06:04:11.908470+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:36:53.154893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.908799+0000","last_change":"2024-05-28T06:04:10.908671+0000","last_active":"2024-05-28T06:04:11.908799+0000","last_peered":"2024-05-28T06:04:11.908799+0000","last_clean":"2024-05-28T06:04:11.908799+0000","last_became_active":"2024-05-28T06:04:10.908402+0000","last_became_peered":"2024-05-28T06:04:10.908402+0000","last_unstale":"2024-05-28T06:04:11.908799+0000","last_undegraded":"2024-05-28T06:04:11.908799+0000","last_fullsized":"2024-05-28T06:04:11.908799+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:06:44.034610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.995124+0000","last_change":"2024-05-28T06:04:09.868302+0000","last_active":"2024-05-28T06:04:12.995124+0000","last_peered":"2024-05-28T06:04:12.995124+0000","last_clean":"2024-05-28T06:04:12.995124+0000","last_became_active":"2024-05-28T06:04:09.868003+0000","last_became_peered":"2024-05-28T06:04:09.868003+0000","last_unstale":"2024-05-28T06:04:12.995124+0000","last_undegraded":"2024-05-28T06:04:12.995124+0000","last_fullsized":"2024-05-28T06:04:12.995124+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:29:32.319083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.995729+0000","last_change":"2024-05-28T06:04:10.916061+0000","last_active":"2024-05-28T06:04:12.995729+0000","last_peered":"2024-05-28T06:04:12.995729+0000","last_clean":"2024-05-28T06:04:12.995729+0000","last_became_active":"2024-05-28T06:04:10.915607+0000","last_became_peered":"2024-05-28T06:04:10.915607+0000","last_unstale":"2024-05-28T06:04:12.995729+0000","last_undegraded":"2024-05-28T06:04:12.995729+0000","last_fullsized":"2024-05-28T06:04:12.995729+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:51:12.839226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.893456+0000","last_change":"2024-05-28T06:04:09.874073+0000","last_active":"2024-05-28T06:04:11.893456+0000","last_peered":"2024-05-28T06:04:11.893456+0000","last_clean":"2024-05-28T06:04:11.893456+0000","last_became_active":"2024-05-28T06:04:09.872906+0000","last_became_peered":"2024-05-28T06:04:09.872906+0000","last_unstale":"2024-05-28T06:04:11.893456+0000","last_undegraded":"2024-05-28T06:04:11.893456+0000","last_fullsized":"2024-05-28T06:04:11.893456+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:06:04.907427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.998407+0000","last_change":"2024-05-28T06:04:10.914311+0000","last_active":"2024-05-28T06:04:12.998407+0000","last_peered":"2024-05-28T06:04:12.998407+0000","last_clean":"2024-05-28T06:04:12.998407+0000","last_became_active":"2024-05-28T06:04:10.913972+0000","last_became_peered":"2024-05-28T06:04:10.913972+0000","last_unstale":"2024-05-28T06:04:12.998407+0000","last_undegraded":"2024-05-28T06:04:12.998407+0000","last_fullsized":"2024-05-28T06:04:12.998407+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:11:58.309255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.996846+0000","last_change":"2024-05-28T06:04:09.867608+0000","last_active":"2024-05-28T06:04:12.996846+0000","last_peered":"2024-05-28T06:04:12.996846+0000","last_clean":"2024-05-28T06:04:12.996846+0000","last_became_active":"2024-05-28T06:04:09.865750+0000","last_became_peered":"2024-05-28T06:04:09.865750+0000","last_unstale":"2024-05-28T06:04:12.996846+0000","last_undegraded":"2024-05-28T06:04:12.996846+0000","last_fullsized":"2024-05-28T06:04:12.996846+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:16:52.703842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.998212+0000","last_change":"2024-05-28T06:04:10.914237+0000","last_active":"2024-05-28T06:04:12.998212+0000","last_peered":"2024-05-28T06:04:12.998212+0000","last_clean":"2024-05-28T06:04:12.998212+0000","last_became_active":"2024-05-28T06:04:10.913957+0000","last_became_peered":"2024-05-28T06:04:10.913957+0000","last_unstale":"2024-05-28T06:04:12.998212+0000","last_undegraded":"2024-05-28T06:04:12.998212+0000","last_fullsized":"2024-05-28T06:04:12.998212+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:28:36.252853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.890109+0000","last_change":"2024-05-28T06:04:09.877173+0000","last_active":"2024-05-28T06:04:11.890109+0000","last_peered":"2024-05-28T06:04:11.890109+0000","last_clean":"2024-05-28T06:04:11.890109+0000","last_became_active":"2024-05-28T06:04:09.876534+0000","last_became_peered":"2024-05-28T06:04:09.876534+0000","last_unstale":"2024-05-28T06:04:11.890109+0000","last_undegraded":"2024-05-28T06:04:11.890109+0000","last_fullsized":"2024-05-28T06:04:11.890109+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:58:50.805034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.998844+0000","last_change":"2024-05-28T06:04:10.923771+0000","last_active":"2024-05-28T06:04:12.998844+0000","last_peered":"2024-05-28T06:04:12.998844+0000","last_clean":"2024-05-28T06:04:12.998844+0000","last_became_active":"2024-05-28T06:04:10.923494+0000","last_became_peered":"2024-05-28T06:04:10.923494+0000","last_unstale":"2024-05-28T06:04:12.998844+0000","last_undegraded":"2024-05-28T06:04:12.998844+0000","last_fullsized":"2024-05-28T06:04:12.998844+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:05:14.695994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.995877+0000","last_change":"2024-05-28T06:04:09.880725+0000","last_active":"2024-05-28T06:04:12.995877+0000","last_peered":"2024-05-28T06:04:12.995877+0000","last_clean":"2024-05-28T06:04:12.995877+0000","last_became_active":"2024-05-28T06:04:09.880430+0000","last_became_peered":"2024-05-28T06:04:09.880430+0000","last_unstale":"2024-05-28T06:04:12.995877+0000","last_undegraded":"2024-05-28T06:04:12.995877+0000","last_fullsized":"2024-05-28T06:04:12.995877+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:27:30.379288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.996167+0000","last_change":"2024-05-28T06:04:10.904576+0000","last_active":"2024-05-28T06:04:12.996167+0000","last_peered":"2024-05-28T06:04:12.996167+0000","last_clean":"2024-05-28T06:04:12.996167+0000","last_became_active":"2024-05-28T06:04:10.904257+0000","last_became_peered":"2024-05-28T06:04:10.904257+0000","last_unstale":"2024-05-28T06:04:12.996167+0000","last_undegraded":"2024-05-28T06:04:12.996167+0000","last_fullsized":"2024-05-28T06:04:12.996167+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:49:22.704773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":170,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.995506+0000","last_change":"2024-05-28T06:04:01.039129+0000","last_active":"2024-05-28T06:04:12.995506+0000","last_peered":"2024-05-28T06:04:12.995506+0000","last_clean":"2024-05-28T06:04:12.995506+0000","last_became_active":"2024-05-28T06:04:01.033906+0000","last_became_peered":"2024-05-28T06:04:01.033906+0000","last_unstale":"2024-05-28T06:04:12.995506+0000","last_undegraded":"2024-05-28T06:04:12.995506+0000","last_fullsized":"2024-05-28T06:04:12.995506+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:03:59.864237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:03:59.864237+0000","last_clean_scrub_stamp":"2024-05-28T06:03:59.864237+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:17:36.440245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.996021+0000","last_change":"2024-05-28T06:04:09.879679+0000","last_active":"2024-05-28T06:04:12.996021+0000","last_peered":"2024-05-28T06:04:12.996021+0000","last_clean":"2024-05-28T06:04:12.996021+0000","last_became_active":"2024-05-28T06:04:09.877412+0000","last_became_peered":"2024-05-28T06:04:09.877412+0000","last_unstale":"2024-05-28T06:04:12.996021+0000","last_undegraded":"2024-05-28T06:04:12.996021+0000","last_fullsized":"2024-05-28T06:04:12.996021+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:58:06.720498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.996386+0000","last_change":"2024-05-28T06:04:10.901746+0000","last_active":"2024-05-28T06:04:12.996386+0000","last_peered":"2024-05-28T06:04:12.996386+0000","last_clean":"2024-05-28T06:04:12.996386+0000","last_became_active":"2024-05-28T06:04:10.901222+0000","last_became_peered":"2024-05-28T06:04:10.901222+0000","last_unstale":"2024-05-28T06:04:12.996386+0000","last_undegraded":"2024-05-28T06:04:12.996386+0000","last_fullsized":"2024-05-28T06:04:12.996386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:46:29.255545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.995167+0000","last_change":"2024-05-28T06:04:09.890312+0000","last_active":"2024-05-28T06:04:12.995167+0000","last_peered":"2024-05-28T06:04:12.995167+0000","last_clean":"2024-05-28T06:04:12.995167+0000","last_became_active":"2024-05-28T06:04:09.889754+0000","last_became_peered":"2024-05-28T06:04:09.889754+0000","last_unstale":"2024-05-28T06:04:12.995167+0000","last_undegraded":"2024-05-28T06:04:12.995167+0000","last_fullsized":"2024-05-28T06:04:12.995167+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:38:20.882939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.793983+0000","last_change":"2024-05-28T06:04:10.886055+0000","last_active":"2024-05-28T06:04:12.793983+0000","last_peered":"2024-05-28T06:04:12.793983+0000","last_clean":"2024-05-28T06:04:12.793983+0000","last_became_active":"2024-05-28T06:04:10.885596+0000","last_became_peered":"2024-05-28T06:04:10.885596+0000","last_unstale":"2024-05-28T06:04:12.793983+0000","last_undegraded":"2024-05-28T06:04:12.793983+0000","last_fullsized":"2024-05-28T06:04:12.793983+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:06:44.034610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.995673+0000","last_change":"2024-05-28T06:04:09.871789+0000","last_active":"2024-05-28T06:04:12.995673+0000","last_peered":"2024-05-28T06:04:12.995673+0000","last_clean":"2024-05-28T06:04:12.995673+0000","last_became_active":"2024-05-28T06:04:09.870232+0000","last_became_peered":"2024-05-28T06:04:09.870232+0000","last_unstale":"2024-05-28T06:04:12.995673+0000","last_undegraded":"2024-05-28T06:04:12.995673+0000","last_fullsized":"2024-05-28T06:04:12.995673+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:36:53.154893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.890126+0000","last_change":"2024-05-28T06:04:10.898534+0000","last_active":"2024-05-28T06:04:11.890126+0000","last_peered":"2024-05-28T06:04:11.890126+0000","last_clean":"2024-05-28T06:04:11.890126+0000","last_became_active":"2024-05-28T06:04:10.898237+0000","last_became_peered":"2024-05-28T06:04:10.898237+0000","last_unstale":"2024-05-28T06:04:11.890126+0000","last_undegraded":"2024-05-28T06:04:11.890126+0000","last_fullsized":"2024-05-28T06:04:11.890126+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:46:29.255545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.997224+0000","last_change":"2024-05-28T06:04:09.884779+0000","last_active":"2024-05-28T06:04:12.997224+0000","last_peered":"2024-05-28T06:04:12.997224+0000","last_clean":"2024-05-28T06:04:12.997224+0000","last_became_active":"2024-05-28T06:04:09.882344+0000","last_became_peered":"2024-05-28T06:04:09.882344+0000","last_unstale":"2024-05-28T06:04:12.997224+0000","last_undegraded":"2024-05-28T06:04:12.997224+0000","last_fullsized":"2024-05-28T06:04:12.997224+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:29:32.319083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.893200+0000","last_change":"2024-05-28T06:04:10.890674+0000","last_active":"2024-05-28T06:04:11.893200+0000","last_peered":"2024-05-28T06:04:11.893200+0000","last_clean":"2024-05-28T06:04:11.893200+0000","last_became_active":"2024-05-28T06:04:10.890144+0000","last_became_peered":"2024-05-28T06:04:10.890144+0000","last_unstale":"2024-05-28T06:04:11.893200+0000","last_undegraded":"2024-05-28T06:04:11.893200+0000","last_fullsized":"2024-05-28T06:04:11.893200+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:38:21.897771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.890349+0000","last_change":"2024-05-28T06:04:09.871025+0000","last_active":"2024-05-28T06:04:11.890349+0000","last_peered":"2024-05-28T06:04:11.890349+0000","last_clean":"2024-05-28T06:04:11.890349+0000","last_became_active":"2024-05-28T06:04:09.870646+0000","last_became_peered":"2024-05-28T06:04:09.870646+0000","last_unstale":"2024-05-28T06:04:11.890349+0000","last_undegraded":"2024-05-28T06:04:11.890349+0000","last_fullsized":"2024-05-28T06:04:11.890349+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:16:52.703842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.998525+0000","last_change":"2024-05-28T06:04:10.912585+0000","last_active":"2024-05-28T06:04:12.998525+0000","last_peered":"2024-05-28T06:04:12.998525+0000","last_clean":"2024-05-28T06:04:12.998525+0000","last_became_active":"2024-05-28T06:04:10.912038+0000","last_became_peered":"2024-05-28T06:04:10.912038+0000","last_unstale":"2024-05-28T06:04:12.998525+0000","last_undegraded":"2024-05-28T06:04:12.998525+0000","last_fullsized":"2024-05-28T06:04:12.998525+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:04:49.415234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.997550+0000","last_change":"2024-05-28T06:04:09.890971+0000","last_active":"2024-05-28T06:04:12.997550+0000","last_peered":"2024-05-28T06:04:12.997550+0000","last_clean":"2024-05-28T06:04:12.997550+0000","last_became_active":"2024-05-28T06:04:09.890401+0000","last_became_peered":"2024-05-28T06:04:09.890401+0000","last_unstale":"2024-05-28T06:04:12.997550+0000","last_undegraded":"2024-05-28T06:04:12.997550+0000","last_fullsized":"2024-05-28T06:04:12.997550+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:05:29.419386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.890261+0000","last_change":"2024-05-28T06:04:10.903420+0000","last_active":"2024-05-28T06:04:11.890261+0000","last_peered":"2024-05-28T06:04:11.890261+0000","last_clean":"2024-05-28T06:04:11.890261+0000","last_became_active":"2024-05-28T06:04:10.898157+0000","last_became_peered":"2024-05-28T06:04:10.898157+0000","last_unstale":"2024-05-28T06:04:11.890261+0000","last_undegraded":"2024-05-28T06:04:11.890261+0000","last_fullsized":"2024-05-28T06:04:11.890261+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:11:51.651640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.890155+0000","last_change":"2024-05-28T06:04:09.886223+0000","last_active":"2024-05-28T06:04:11.890155+0000","last_peered":"2024-05-28T06:04:11.890155+0000","last_clean":"2024-05-28T06:04:11.890155+0000","last_became_active":"2024-05-28T06:04:09.885439+0000","last_became_peered":"2024-05-28T06:04:09.885439+0000","last_unstale":"2024-05-28T06:04:11.890155+0000","last_undegraded":"2024-05-28T06:04:11.890155+0000","last_fullsized":"2024-05-28T06:04:11.890155+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:38:20.882939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.997783+0000","last_change":"2024-05-28T06:04:10.901609+0000","last_active":"2024-05-28T06:04:12.997783+0000","last_peered":"2024-05-28T06:04:12.997783+0000","last_clean":"2024-05-28T06:04:12.997783+0000","last_became_active":"2024-05-28T06:04:10.901302+0000","last_became_peered":"2024-05-28T06:04:10.901302+0000","last_unstale":"2024-05-28T06:04:12.997783+0000","last_undegraded":"2024-05-28T06:04:12.997783+0000","last_fullsized":"2024-05-28T06:04:12.997783+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:49:22.704773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.909384+0000","last_change":"2024-05-28T06:04:09.888169+0000","last_active":"2024-05-28T06:04:11.909384+0000","last_peered":"2024-05-28T06:04:11.909384+0000","last_clean":"2024-05-28T06:04:11.909384+0000","last_became_active":"2024-05-28T06:04:09.887867+0000","last_became_peered":"2024-05-28T06:04:09.887867+0000","last_unstale":"2024-05-28T06:04:11.909384+0000","last_undegraded":"2024-05-28T06:04:11.909384+0000","last_fullsized":"2024-05-28T06:04:11.909384+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:16:52.703842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.793993+0000","last_change":"2024-05-28T06:04:10.897718+0000","last_active":"2024-05-28T06:04:12.793993+0000","last_peered":"2024-05-28T06:04:12.793993+0000","last_clean":"2024-05-28T06:04:12.793993+0000","last_became_active":"2024-05-28T06:04:10.896414+0000","last_became_peered":"2024-05-28T06:04:10.896414+0000","last_unstale":"2024-05-28T06:04:12.793993+0000","last_undegraded":"2024-05-28T06:04:12.793993+0000","last_fullsized":"2024-05-28T06:04:12.793993+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:53:07.733019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.995167+0000","last_change":"2024-05-28T06:04:09.889726+0000","last_active":"2024-05-28T06:04:12.995167+0000","last_peered":"2024-05-28T06:04:12.995167+0000","last_clean":"2024-05-28T06:04:12.995167+0000","last_became_active":"2024-05-28T06:04:09.889394+0000","last_became_peered":"2024-05-28T06:04:09.889394+0000","last_unstale":"2024-05-28T06:04:12.995167+0000","last_undegraded":"2024-05-28T06:04:12.995167+0000","last_fullsized":"2024-05-28T06:04:12.995167+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:58:06.720498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.892869+0000","last_change":"2024-05-28T06:04:10.908922+0000","last_active":"2024-05-28T06:04:11.892869+0000","last_peered":"2024-05-28T06:04:11.892869+0000","last_clean":"2024-05-28T06:04:11.892869+0000","last_became_active":"2024-05-28T06:04:10.905738+0000","last_became_peered":"2024-05-28T06:04:10.905738+0000","last_unstale":"2024-05-28T06:04:11.892869+0000","last_undegraded":"2024-05-28T06:04:11.892869+0000","last_fullsized":"2024-05-28T06:04:11.892869+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:58:07.735330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.890426+0000","last_change":"2024-05-28T06:04:09.886430+0000","last_active":"2024-05-28T06:04:11.890426+0000","last_peered":"2024-05-28T06:04:11.890426+0000","last_clean":"2024-05-28T06:04:11.890426+0000","last_became_active":"2024-05-28T06:04:09.886072+0000","last_became_peered":"2024-05-28T06:04:09.886072+0000","last_unstale":"2024-05-28T06:04:11.890426+0000","last_undegraded":"2024-05-28T06:04:11.890426+0000","last_fullsized":"2024-05-28T06:04:11.890426+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:29:32.319083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.995901+0000","last_change":"2024-05-28T06:04:10.895054+0000","last_active":"2024-05-28T06:04:12.995901+0000","last_peered":"2024-05-28T06:04:12.995901+0000","last_clean":"2024-05-28T06:04:12.995901+0000","last_became_active":"2024-05-28T06:04:10.894384+0000","last_became_peered":"2024-05-28T06:04:10.894384+0000","last_unstale":"2024-05-28T06:04:12.995901+0000","last_undegraded":"2024-05-28T06:04:12.995901+0000","last_fullsized":"2024-05-28T06:04:12.995901+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:53:07.733019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.889822+0000","last_change":"2024-05-28T06:04:09.873566+0000","last_active":"2024-05-28T06:04:11.889822+0000","last_peered":"2024-05-28T06:04:11.889822+0000","last_clean":"2024-05-28T06:04:11.889822+0000","last_became_active":"2024-05-28T06:04:09.872928+0000","last_became_peered":"2024-05-28T06:04:09.872928+0000","last_unstale":"2024-05-28T06:04:11.889822+0000","last_undegraded":"2024-05-28T06:04:11.889822+0000","last_fullsized":"2024-05-28T06:04:11.889822+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:58:06.720498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.996688+0000","last_change":"2024-05-28T06:04:10.905645+0000","last_active":"2024-05-28T06:04:12.996688+0000","last_peered":"2024-05-28T06:04:12.996688+0000","last_clean":"2024-05-28T06:04:12.996688+0000","last_became_active":"2024-05-28T06:04:10.905308+0000","last_became_peered":"2024-05-28T06:04:10.905308+0000","last_unstale":"2024-05-28T06:04:12.996688+0000","last_undegraded":"2024-05-28T06:04:12.996688+0000","last_fullsized":"2024-05-28T06:04:12.996688+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:11:51.651640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.997139+0000","last_change":"2024-05-28T06:04:09.893691+0000","last_active":"2024-05-28T06:04:12.997139+0000","last_peered":"2024-05-28T06:04:12.997139+0000","last_clean":"2024-05-28T06:04:12.997139+0000","last_became_active":"2024-05-28T06:04:09.893416+0000","last_became_peered":"2024-05-28T06:04:09.893416+0000","last_unstale":"2024-05-28T06:04:12.997139+0000","last_undegraded":"2024-05-28T06:04:12.997139+0000","last_fullsized":"2024-05-28T06:04:12.997139+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:53:48.086901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:11.908760+0000","last_change":"2024-05-28T06:04:10.919265+0000","last_active":"2024-05-28T06:04:11.908760+0000","last_peered":"2024-05-28T06:04:11.908760+0000","last_clean":"2024-05-28T06:04:11.908760+0000","last_became_active":"2024-05-28T06:04:10.918268+0000","last_became_peered":"2024-05-28T06:04:10.918268+0000","last_unstale":"2024-05-28T06:04:11.908760+0000","last_undegraded":"2024-05-28T06:04:11.908760+0000","last_fullsized":"2024-05-28T06:04:11.908760+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:39:46.389531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.998229+0000","last_change":"2024-05-28T06:04:10.912331+0000","last_active":"2024-05-28T06:04:12.998229+0000","last_peered":"2024-05-28T06:04:12.998229+0000","last_clean":"2024-05-28T06:04:12.998229+0000","last_became_active":"2024-05-28T06:04:10.907887+0000","last_became_peered":"2024-05-28T06:04:10.907887+0000","last_unstale":"2024-05-28T06:04:12.998229+0000","last_undegraded":"2024-05-28T06:04:12.998229+0000","last_fullsized":"2024-05-28T06:04:12.998229+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:03:18.610996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.996617+0000","last_change":"2024-05-28T06:04:09.892486+0000","last_active":"2024-05-28T06:04:12.996617+0000","last_peered":"2024-05-28T06:04:12.996617+0000","last_clean":"2024-05-28T06:04:12.996617+0000","last_became_active":"2024-05-28T06:04:09.892160+0000","last_became_peered":"2024-05-28T06:04:09.892160+0000","last_unstale":"2024-05-28T06:04:12.996617+0000","last_undegraded":"2024-05-28T06:04:12.996617+0000","last_fullsized":"2024-05-28T06:04:12.996617+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:03:45.379665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.997770+0000","last_change":"2024-05-28T06:04:10.909079+0000","last_active":"2024-05-28T06:04:12.997770+0000","last_peered":"2024-05-28T06:04:12.997770+0000","last_clean":"2024-05-28T06:04:12.997770+0000","last_became_active":"2024-05-28T06:04:10.907979+0000","last_became_peered":"2024-05-28T06:04:10.907979+0000","last_unstale":"2024-05-28T06:04:12.997770+0000","last_undegraded":"2024-05-28T06:04:12.997770+0000","last_fullsized":"2024-05-28T06:04:12.997770+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:49:22.704773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.996620+0000","last_change":"2024-05-28T06:04:09.890388+0000","last_active":"2024-05-28T06:04:12.996620+0000","last_peered":"2024-05-28T06:04:12.996620+0000","last_clean":"2024-05-28T06:04:12.996620+0000","last_became_active":"2024-05-28T06:04:09.890086+0000","last_became_peered":"2024-05-28T06:04:09.890086+0000","last_unstale":"2024-05-28T06:04:12.996620+0000","last_undegraded":"2024-05-28T06:04:12.996620+0000","last_fullsized":"2024-05-28T06:04:12.996620+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:58:50.805034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.794239+0000","last_change":"2024-05-28T06:04:10.904802+0000","last_active":"2024-05-28T06:04:12.794239+0000","last_peered":"2024-05-28T06:04:12.794239+0000","last_clean":"2024-05-28T06:04:12.794239+0000","last_became_active":"2024-05-28T06:04:10.904438+0000","last_became_peered":"2024-05-28T06:04:10.904438+0000","last_unstale":"2024-05-28T06:04:12.794239+0000","last_undegraded":"2024-05-28T06:04:12.794239+0000","last_fullsized":"2024-05-28T06:04:12.794239+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:11:51.651640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.997391+0000","last_change":"2024-05-28T06:04:09.874413+0000","last_active":"2024-05-28T06:04:12.997391+0000","last_peered":"2024-05-28T06:04:12.997391+0000","last_clean":"2024-05-28T06:04:12.997391+0000","last_became_active":"2024-05-28T06:04:09.872117+0000","last_became_peered":"2024-05-28T06:04:09.872117+0000","last_unstale":"2024-05-28T06:04:12.997391+0000","last_undegraded":"2024-05-28T06:04:12.997391+0000","last_fullsized":"2024-05-28T06:04:12.997391+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:36:53.154893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.997324+0000","last_change":"2024-05-28T06:04:10.909242+0000","last_active":"2024-05-28T06:04:12.997324+0000","last_peered":"2024-05-28T06:04:12.997324+0000","last_clean":"2024-05-28T06:04:12.997324+0000","last_became_active":"2024-05-28T06:04:10.908934+0000","last_became_peered":"2024-05-28T06:04:10.908934+0000","last_unstale":"2024-05-28T06:04:12.997324+0000","last_undegraded":"2024-05-28T06:04:12.997324+0000","last_fullsized":"2024-05-28T06:04:12.997324+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:09.837240+0000","last_clean_scrub_stamp":"2024-05-28T06:04:09.837240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:45:47.884578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-28T06:04:12.794420+0000","last_change":"2024-05-28T06:04:09.887679+0000","last_active":"2024-05-28T06:04:12.794420+0000","last_peered":"2024-05-28T06:04:12.794420+0000","last_clean":"2024-05-28T06:04:12.794420+0000","last_became_active":"2024-05-28T06:04:09.887386+0000","last_became_peered":"2024-05-28T06:04:09.887386+0000","last_unstale":"2024-05-28T06:04:12.794420+0000","last_undegraded":"2024-05-28T06:04:12.794420+0000","last_fullsized":"2024-05-28T06:04:12.794420+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:04:08.822408+0000","last_clean_scrub_stamp":"2024-05-28T06:04:08.822408+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:38:33.383965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":802816,"data_stored":661884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":208896,"data_stored":71024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":71060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574857,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574856,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":70934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":212992,"data_stored":71094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":811008,"data_stored":661940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-28T06:04:21.644 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-05-28T06:04:21.644 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-05-28T06:04:21.644 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-05-28T06:04:21.644 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-05-28T06:04:21.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.878+0000 7fd906a00640 1 Processor -- start 2024-05-28T06:04:21.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.878+0000 7fd906a00640 1 -- start start 2024-05-28T06:04:21.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.879+0000 7fd906a00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fd9000d4c40 0x7fd9000d2330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:21.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.879+0000 7fd906a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fd9000d1ca0 0x7fd9000d2870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:21.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.879+0000 7fd906a00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9000d2db0 con 0x7fd90015e690 2024-05-28T06:04:21.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.880+0000 7fd906a00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fd9000d3060 con 0x7fd9000d1ca0 2024-05-28T06:04:21.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.880+0000 7fd906a00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fd9000d3340 con 0x7fd9000d4c40 2024-05-28T06:04:21.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.880+0000 7fd906000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fd9000d1ca0 0x7fd9000d2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:21.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.880+0000 7fd906000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fd9000d1ca0 0x7fd9000d2870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48158/0 (socket says 172.21.15.96:48158) 2024-05-28T06:04:21.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.880+0000 7fd906000640 1 -- 172.21.15.96:0/3371877029 learned_addr learned my addr 172.21.15.96:0/3371877029 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:21.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.880+0000 7fd904c00640 1 --2- 172.21.15.96:0/3371877029 >> v2:172.21.15.123:3300/0 conn(0x7fd9000d4c40 0x7fd9000d2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:21.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.880+0000 7fd906000640 1 -- 172.21.15.96:0/3371877029 >> v2:172.21.15.123:3300/0 conn(0x7fd9000d4c40 msgr2=0x7fd9000d2330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:21.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.880+0000 7fd906000640 1 --2- 172.21.15.96:0/3371877029 >> v2:172.21.15.123:3300/0 conn(0x7fd9000d4c40 0x7fd9000d2330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:21.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.880+0000 7fd906000640 1 -- 172.21.15.96:0/3371877029 >> v1:172.21.15.123:6789/0 conn(0x7fd90015e690 legacy=0x7fd9000d4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:21.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.880+0000 7fd906000640 1 -- 172.21.15.96:0/3371877029 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9000d3620 con 0x7fd9000d1ca0 2024-05-28T06:04:21.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.881+0000 7fd906000640 1 --2- 172.21.15.96:0/3371877029 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9000d1ca0 0x7fd9000d2870 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd90006d130 tx=0x7fd8f002a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fcb9d8982799a92 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:21.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.881+0000 7fd8ffe00640 1 -- 172.21.15.96:0/3371877029 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8f0037070 con 0x7fd9000d1ca0 2024-05-28T06:04:21.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.882+0000 7fd8ffe00640 1 -- 172.21.15.96:0/3371877029 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8f0032440 con 0x7fd9000d1ca0 2024-05-28T06:04:21.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.882+0000 7fd8ffe00640 1 -- 172.21.15.96:0/3371877029 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8f0032810 con 0x7fd9000d1ca0 2024-05-28T06:04:21.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.882+0000 7fd906a00640 1 -- 172.21.15.96:0/3371877029 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9000d1ca0 msgr2=0x7fd9000d2870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:21.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.882+0000 7fd906a00640 1 --2- 172.21.15.96:0/3371877029 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9000d1ca0 0x7fd9000d2870 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd90006d130 tx=0x7fd8f002a230 comp rx=0 tx=0).stop 2024-05-28T06:04:21.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.882+0000 7fd906a00640 1 -- 172.21.15.96:0/3371877029 shutdown_connections 2024-05-28T06:04:21.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.882+0000 7fd906a00640 1 --2- 172.21.15.96:0/3371877029 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9000d1ca0 0x7fd9000d2870 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:21.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.882+0000 7fd906a00640 1 --2- 172.21.15.96:0/3371877029 >> v2:172.21.15.123:3300/0 conn(0x7fd9000d4c40 0x7fd9000d2330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:21.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.882+0000 7fd906a00640 1 -- 172.21.15.96:0/3371877029 >> 172.21.15.96:0/3371877029 conn(0x7fd90009d910 msgr2=0x7fd90008bfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:21.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.882+0000 7fd906a00640 1 -- 172.21.15.96:0/3371877029 shutdown_connections 2024-05-28T06:04:21.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.882+0000 7fd906a00640 1 -- 172.21.15.96:0/3371877029 wait complete. 2024-05-28T06:04:21.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.884+0000 7fd906a00640 1 Processor -- start 2024-05-28T06:04:21.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.884+0000 7fd906a00640 1 -- start start 2024-05-28T06:04:21.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.885+0000 7fd906a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9000d1ca0 0x7fd90015d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:21.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.885+0000 7fd906a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9000d4c40 0x7fd90015fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:21.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.885+0000 7fd906a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd90015e690 0x7fd9000b9610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:21.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.885+0000 7fd906a00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd9000cf510 con 0x7fd9000d4c40 2024-05-28T06:04:21.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.885+0000 7fd906a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd9000cf390 con 0x7fd9000d1ca0 2024-05-28T06:04:21.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.885+0000 7fd906a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fd90015ea70 con 0x7fd90015e690 2024-05-28T06:04:21.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.885+0000 7fd904c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9000d4c40 0x7fd90015fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:21.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.885+0000 7fd904c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9000d4c40 0x7fd90015fdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48172/0 (socket says 172.21.15.96:48172) 2024-05-28T06:04:21.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.885+0000 7fd904c00640 1 -- 172.21.15.96:0/1214394638 learned_addr learned my addr 172.21.15.96:0/1214394638 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:21.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.885+0000 7fd906000640 1 --2- 172.21.15.96:0/1214394638 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd90015e690 0x7fd9000b9610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:21.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.885+0000 7fd905600640 1 --2- 172.21.15.96:0/1214394638 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9000d1ca0 0x7fd90015d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:21.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.885+0000 7fd904c00640 1 -- 172.21.15.96:0/1214394638 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd90015e690 msgr2=0x7fd9000b9610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:21.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.885+0000 7fd904c00640 1 --2- 172.21.15.96:0/1214394638 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd90015e690 0x7fd9000b9610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:21.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.885+0000 7fd904c00640 1 -- 172.21.15.96:0/1214394638 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9000d1ca0 msgr2=0x7fd90015d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:21.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.885+0000 7fd904c00640 1 --2- 172.21.15.96:0/1214394638 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9000d1ca0 0x7fd90015d660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:21.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.885+0000 7fd904c00640 1 -- 172.21.15.96:0/1214394638 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9000ce680 con 0x7fd9000d4c40 2024-05-28T06:04:21.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.886+0000 7fd904c00640 1 --2- 172.21.15.96:0/1214394638 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9000d4c40 0x7fd90015fdd0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec00d800 tx=0x7fd8ec00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:21.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.886+0000 7fd8fd600640 1 -- 172.21.15.96:0/1214394638 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8ec00b6c0 con 0x7fd9000d4c40 2024-05-28T06:04:21.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.886+0000 7fd8fd600640 1 -- 172.21.15.96:0/1214394638 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8ec019070 con 0x7fd9000d4c40 2024-05-28T06:04:21.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.886+0000 7fd906a00640 1 -- 172.21.15.96:0/1214394638 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd90015f010 con 0x7fd9000d4c40 2024-05-28T06:04:21.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.886+0000 7fd8fd600640 1 -- 172.21.15.96:0/1214394638 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8ec00baa0 con 0x7fd9000d4c40 2024-05-28T06:04:21.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.886+0000 7fd906a00640 1 -- 172.21.15.96:0/1214394638 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd90015f5d0 con 0x7fd9000d4c40 2024-05-28T06:04:21.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.887+0000 7fd906a00640 1 -- 172.21.15.96:0/1214394638 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9000689e0 con 0x7fd9000d4c40 2024-05-28T06:04:21.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.888+0000 7fd8fd600640 1 -- 172.21.15.96:0/1214394638 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd8ec00bc40 con 0x7fd9000d4c40 2024-05-28T06:04:21.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.889+0000 7fd8fd600640 1 --2- 172.21.15.96:0/1214394638 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd8e006c8f0 0x7fd8e006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:21.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.889+0000 7fd8fd600640 1 -- 172.21.15.96:0/1214394638 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fd8ec08bfc0 con 0x7fd9000d4c40 2024-05-28T06:04:21.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.890+0000 7fd905600640 1 --2- 172.21.15.96:0/1214394638 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd8e006c8f0 0x7fd8e006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:21.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.891+0000 7fd905600640 1 --2- 172.21.15.96:0/1214394638 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd8e006c8f0 0x7fd8e006edb0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd8f8007bf0 tx=0x7fd8f8007240 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:21.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:21.893+0000 7fd8fd600640 1 -- 172.21.15.96:0/1214394638 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd9000ce680 con 0x7fd9000d4c40 2024-05-28T06:04:22.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.116+0000 7fd906a00640 1 -- 172.21.15.96:0/1214394638 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fd9000b2f40 con 0x7fd9000d4c40 2024-05-28T06:04:22.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.117+0000 7fd8fd600640 1 -- 172.21.15.96:0/1214394638 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7fd8ec05a3e0 con 0x7fd9000d4c40 2024-05-28T06:04:22.121 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:04:22.121 INFO:teuthology.orchestra.run.smithi096.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-05-28T06:04:22.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.120+0000 7fd906a00640 1 -- 172.21.15.96:0/1214394638 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd8e006c8f0 msgr2=0x7fd8e006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:22.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.120+0000 7fd906a00640 1 --2- 172.21.15.96:0/1214394638 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd8e006c8f0 0x7fd8e006edb0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd8f8007bf0 tx=0x7fd8f8007240 comp rx=0 tx=0).stop 2024-05-28T06:04:22.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.120+0000 7fd906a00640 1 -- 172.21.15.96:0/1214394638 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9000d4c40 msgr2=0x7fd90015fdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:22.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.120+0000 7fd906a00640 1 --2- 172.21.15.96:0/1214394638 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9000d4c40 0x7fd90015fdd0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec00d800 tx=0x7fd8ec00dcc0 comp rx=0 tx=0).stop 2024-05-28T06:04:22.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.120+0000 7fd906a00640 1 -- 172.21.15.96:0/1214394638 shutdown_connections 2024-05-28T06:04:22.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.120+0000 7fd906a00640 1 --2- 172.21.15.96:0/1214394638 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd8e006c8f0 0x7fd8e006edb0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:22.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.120+0000 7fd906a00640 1 --2- 172.21.15.96:0/1214394638 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd90015e690 0x7fd9000b9610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:22.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.120+0000 7fd906a00640 1 --2- 172.21.15.96:0/1214394638 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9000d4c40 0x7fd90015fdd0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:22.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.120+0000 7fd906a00640 1 --2- 172.21.15.96:0/1214394638 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9000d1ca0 0x7fd90015d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:22.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.120+0000 7fd906a00640 1 -- 172.21.15.96:0/1214394638 >> 172.21.15.96:0/1214394638 conn(0x7fd90009d910 msgr2=0x7fd9000b2470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:22.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.120+0000 7fd906a00640 1 -- 172.21.15.96:0/1214394638 shutdown_connections 2024-05-28T06:04:22.125 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.120+0000 7fd906a00640 1 -- 172.21.15.96:0/1214394638 wait complete. 2024-05-28T06:04:22.139 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-05-28T06:04:22.139 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:04:22.337 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.336+0000 7f0ce2200640 1 Processor -- start 2024-05-28T06:04:22.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.336+0000 7f0ce2200640 1 -- start start 2024-05-28T06:04:22.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.337+0000 7f0ce2200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f0cdc0b4e10 0x7f0cdc0cde30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:22.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.337+0000 7f0ce2200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0cdc0ce370 0x7f0cdc0d0590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:22.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.337+0000 7f0ce2200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0cdc0b51f0 con 0x7f0cdc0d0ad0 2024-05-28T06:04:22.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.337+0000 7f0ce2200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f0cdc0d0f20 con 0x7f0cdc0b4e10 2024-05-28T06:04:22.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.337+0000 7f0ce2200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f0cdc0aef60 con 0x7f0cdc0ce370 2024-05-28T06:04:22.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.338+0000 7f0cdaa00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0cdc0ce370 0x7f0cdc0d0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:22.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.338+0000 7f0cdaa00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0cdc0ce370 0x7f0cdc0d0590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.96:60034/0 (socket says 172.21.15.96:60034) 2024-05-28T06:04:22.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.338+0000 7f0cdaa00640 1 -- 172.21.15.96:0/753103388 learned_addr learned my addr 172.21.15.96:0/753103388 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:22.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.338+0000 7f0cdb400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f0cdc0b4e10 0x7f0cdc0cde30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:22.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.339+0000 7f0cdaa00640 1 -- 172.21.15.96:0/753103388 >> v2:172.21.15.123:3300/0 conn(0x7f0cdc0b4e10 msgr2=0x7f0cdc0cde30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:22.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.339+0000 7f0cdaa00640 1 --2- 172.21.15.96:0/753103388 >> v2:172.21.15.123:3300/0 conn(0x7f0cdc0b4e10 0x7f0cdc0cde30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:22.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.339+0000 7f0cdaa00640 1 -- 172.21.15.96:0/753103388 >> v1:172.21.15.123:6789/0 conn(0x7f0cdc0d0ad0 legacy=0x7f0cdc099090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:22.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.339+0000 7f0cdaa00640 1 -- 172.21.15.96:0/753103388 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cdc0ad1d0 con 0x7f0cdc0ce370 2024-05-28T06:04:22.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.340+0000 7f0cdaa00640 1 --2- 172.21.15.96:0/753103388 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0cdc0ce370 0x7f0cdc0d0590 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f0cc800bb10 tx=0x7f0cc802c890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a6c9f7a7901f0531 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:22.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.341+0000 7f0cda000640 1 -- 172.21.15.96:0/753103388 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cc8032790 con 0x7f0cdc0ce370 2024-05-28T06:04:22.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.341+0000 7f0cda000640 1 -- 172.21.15.96:0/753103388 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0cc8038070 con 0x7f0cdc0ce370 2024-05-28T06:04:22.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.341+0000 7f0cda000640 1 -- 172.21.15.96:0/753103388 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cc8032b50 con 0x7f0cdc0ce370 2024-05-28T06:04:22.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.341+0000 7f0ce2200640 1 -- 172.21.15.96:0/753103388 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0cdc0ce370 msgr2=0x7f0cdc0d0590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:22.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.341+0000 7f0ce2200640 1 --2- 172.21.15.96:0/753103388 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0cdc0ce370 0x7f0cdc0d0590 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f0cc800bb10 tx=0x7f0cc802c890 comp rx=0 tx=0).stop 2024-05-28T06:04:22.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.341+0000 7f0ce2200640 1 -- 172.21.15.96:0/753103388 shutdown_connections 2024-05-28T06:04:22.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.341+0000 7f0ce2200640 1 --2- 172.21.15.96:0/753103388 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0cdc0ce370 0x7f0cdc0d0590 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:22.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.341+0000 7f0ce2200640 1 --2- 172.21.15.96:0/753103388 >> v2:172.21.15.123:3300/0 conn(0x7f0cdc0b4e10 0x7f0cdc0cde30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:22.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.341+0000 7f0ce2200640 1 -- 172.21.15.96:0/753103388 >> 172.21.15.96:0/753103388 conn(0x7f0cdc06bae0 msgr2=0x7f0cdc06bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:22.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.342+0000 7f0ce2200640 1 -- 172.21.15.96:0/753103388 shutdown_connections 2024-05-28T06:04:22.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.342+0000 7f0ce2200640 1 -- 172.21.15.96:0/753103388 wait complete. 2024-05-28T06:04:22.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.343+0000 7f0ce2200640 1 Processor -- start 2024-05-28T06:04:22.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.343+0000 7f0ce2200640 1 -- start start 2024-05-28T06:04:22.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.344+0000 7f0ce2200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0cdc0b4e10 0x7f0cdc15c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:22.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.344+0000 7f0ce2200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0cdc0ce370 0x7f0cdc1f7920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:22.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.344+0000 7f0ce2200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0cdc0d0ad0 0x7f0cdc0c7cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:22.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.344+0000 7f0ce2200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f0cdc0ce880 con 0x7f0cdc0b4e10 2024-05-28T06:04:22.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.344+0000 7f0ce2200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f0cdc0ad750 con 0x7f0cdc0ce370 2024-05-28T06:04:22.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.344+0000 7f0ce2200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f0cdc0cea00 con 0x7f0cdc0d0ad0 2024-05-28T06:04:22.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.344+0000 7f0cdb400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0cdc0b4e10 0x7f0cdc15c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:22.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.345+0000 7f0cdb400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0cdc0b4e10 0x7f0cdc15c590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48182/0 (socket says 172.21.15.96:48182) 2024-05-28T06:04:22.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.345+0000 7f0cdb400640 1 -- 172.21.15.96:0/3392719012 learned_addr learned my addr 172.21.15.96:0/3392719012 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:22.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.345+0000 7f0cdb400640 1 -- 172.21.15.96:0/3392719012 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0cdc0d0ad0 msgr2=0x7f0cdc0c7cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:22.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.345+0000 7f0cdaa00640 1 --2- 172.21.15.96:0/3392719012 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0cdc0ce370 0x7f0cdc1f7920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:22.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.345+0000 7f0cdbe00640 1 --2- 172.21.15.96:0/3392719012 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0cdc0d0ad0 0x7f0cdc0c7cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:22.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.345+0000 7f0cdb400640 1 --2- 172.21.15.96:0/3392719012 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0cdc0d0ad0 0x7f0cdc0c7cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:22.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.345+0000 7f0cdb400640 1 -- 172.21.15.96:0/3392719012 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0cdc0ce370 msgr2=0x7f0cdc1f7920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:22.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.345+0000 7f0cdb400640 1 --2- 172.21.15.96:0/3392719012 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0cdc0ce370 0x7f0cdc1f7920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:22.349 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.345+0000 7f0cdb400640 1 -- 172.21.15.96:0/3392719012 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cdc0b1850 con 0x7f0cdc0b4e10 2024-05-28T06:04:22.349 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.345+0000 7f0cdaa00640 1 --2- 172.21.15.96:0/3392719012 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0cdc0ce370 0x7f0cdc1f7920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:04:22.349 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.345+0000 7f0cdb400640 1 --2- 172.21.15.96:0/3392719012 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0cdc0b4e10 0x7f0cdc15c590 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f0ccc00e870 tx=0x7f0ccc00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:22.349 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.345+0000 7f0cd3400640 1 -- 172.21.15.96:0/3392719012 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ccc017070 con 0x7f0cdc0b4e10 2024-05-28T06:04:22.349 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.346+0000 7f0cd3400640 1 -- 172.21.15.96:0/3392719012 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ccc00c940 con 0x7f0cdc0b4e10 2024-05-28T06:04:22.349 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.346+0000 7f0cd3400640 1 -- 172.21.15.96:0/3392719012 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ccc00cd30 con 0x7f0cdc0b4e10 2024-05-28T06:04:22.349 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.346+0000 7f0ce2200640 1 -- 172.21.15.96:0/3392719012 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0cdc0b1b30 con 0x7f0cdc0b4e10 2024-05-28T06:04:22.349 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.346+0000 7f0ce2200640 1 -- 172.21.15.96:0/3392719012 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0cdc0b20f0 con 0x7f0cdc0b4e10 2024-05-28T06:04:22.350 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.347+0000 7f0ce2200640 1 -- 172.21.15.96:0/3392719012 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0cdc0689e0 con 0x7f0cdc0b4e10 2024-05-28T06:04:22.350 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.348+0000 7f0cd3400640 1 -- 172.21.15.96:0/3392719012 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0ccc01a070 con 0x7f0cdc0b4e10 2024-05-28T06:04:22.350 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.348+0000 7f0cd3400640 1 --2- 172.21.15.96:0/3392719012 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0cbc06c8f0 0x7f0cbc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:22.350 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.348+0000 7f0cd3400640 1 -- 172.21.15.96:0/3392719012 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f0ccc08bd60 con 0x7f0cdc0b4e10 2024-05-28T06:04:22.350 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.349+0000 7f0cdaa00640 1 --2- 172.21.15.96:0/3392719012 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0cbc06c8f0 0x7f0cbc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:22.351 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.350+0000 7f0cdaa00640 1 --2- 172.21.15.96:0/3392719012 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0cbc06c8f0 0x7f0cbc06edb0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0cc8037000 tx=0x7f0cc8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:22.354 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.353+0000 7f0cd3400640 1 -- 172.21.15.96:0/3392719012 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0ccc05a180 con 0x7f0cdc0b4e10 2024-05-28T06:04:22.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.554+0000 7f0ce2200640 1 -- 172.21.15.96:0/3392719012 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0cdc06f540 con 0x7f0cdc0b4e10 2024-05-28T06:04:22.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.557+0000 7f0cd3400640 1 -- 172.21.15.96:0/3392719012 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16617 (secure 0 0 0) 0x7f0ccc059a80 con 0x7f0cdc0b4e10 2024-05-28T06:04:22.559 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:04:22.559 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":20,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:04:11.882922+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-28T06:04:08.741938+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-28T06:04:09.301470+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:04:22.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.560+0000 7f0ce2200640 1 -- 172.21.15.96:0/3392719012 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0cbc06c8f0 msgr2=0x7f0cbc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:22.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.560+0000 7f0ce2200640 1 --2- 172.21.15.96:0/3392719012 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0cbc06c8f0 0x7f0cbc06edb0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0cc8037000 tx=0x7f0cc8008040 comp rx=0 tx=0).stop 2024-05-28T06:04:22.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.560+0000 7f0ce2200640 1 -- 172.21.15.96:0/3392719012 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0cdc0b4e10 msgr2=0x7f0cdc15c590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:22.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.560+0000 7f0ce2200640 1 --2- 172.21.15.96:0/3392719012 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0cdc0b4e10 0x7f0cdc15c590 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f0ccc00e870 tx=0x7f0ccc00ed30 comp rx=0 tx=0).stop 2024-05-28T06:04:22.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.560+0000 7f0ce2200640 1 -- 172.21.15.96:0/3392719012 shutdown_connections 2024-05-28T06:04:22.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.560+0000 7f0ce2200640 1 --2- 172.21.15.96:0/3392719012 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0cbc06c8f0 0x7f0cbc06edb0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:22.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.560+0000 7f0ce2200640 1 --2- 172.21.15.96:0/3392719012 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0cdc0d0ad0 0x7f0cdc0c7cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:22.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.560+0000 7f0ce2200640 1 --2- 172.21.15.96:0/3392719012 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0cdc0ce370 0x7f0cdc1f7920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:22.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.560+0000 7f0ce2200640 1 --2- 172.21.15.96:0/3392719012 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0cdc0b4e10 0x7f0cdc15c590 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:22.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.560+0000 7f0ce2200640 1 -- 172.21.15.96:0/3392719012 >> 172.21.15.96:0/3392719012 conn(0x7f0cdc06bae0 msgr2=0x7f0cdc071ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:22.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.560+0000 7f0ce2200640 1 -- 172.21.15.96:0/3392719012 shutdown_connections 2024-05-28T06:04:22.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.560+0000 7f0ce2200640 1 -- 172.21.15.96:0/3392719012 wait complete. 2024-05-28T06:04:22.583 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-28T06:03:59.517102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-28T06:04:08.741938+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-28T06:04:09.301470+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-28T06:04:22.583 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-28T06:04:22.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.793+0000 7f81b8a00640 1 Processor -- start 2024-05-28T06:04:22.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.794+0000 7f81b8a00640 1 -- start start 2024-05-28T06:04:22.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.795+0000 7f81b8a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f81b40cd2a0 0x7f81b40b4d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:22.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.795+0000 7f81b8a00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81b415d900 con 0x7f81b415cb60 2024-05-28T06:04:22.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.795+0000 7f81b8a00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81b40b3f90 con 0x7f81b4153510 2024-05-28T06:04:22.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.795+0000 7f81b8a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f81b415d6e0 con 0x7f81b40cd2a0 2024-05-28T06:04:22.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.796+0000 7f81b3e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f81b415cb60 0x7f81b415cfd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:35254/0 (socket says 172.21.15.96:35254) 2024-05-28T06:04:22.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.796+0000 7f81b3e00640 1 -- 172.21.15.96:0/3141299576 learned_addr learned my addr 172.21.15.96:0/3141299576 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:04:22.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.796+0000 7f81abe00640 1 --2- 172.21.15.96:0/3141299576 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f81b40cd2a0 0x7f81b40b4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:22.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.796+0000 7f81b2a00640 1 -- 172.21.15.96:0/3141299576 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 198706048 0 0) 0x7f81b415d900 con 0x7f81b415cb60 2024-05-28T06:04:22.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.796+0000 7f81b2a00640 1 -- 172.21.15.96:0/3141299576 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8198003520 con 0x7f81b415cb60 2024-05-28T06:04:22.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.796+0000 7f81b2a00640 1 -- 172.21.15.96:0/3141299576 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 886543243 0 0) 0x7f8198003520 con 0x7f81b415cb60 2024-05-28T06:04:22.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.796+0000 7f81b2a00640 1 -- 172.21.15.96:0/3141299576 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f81b40cd2a0 msgr2=0x7f81b40b4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:22.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.796+0000 7f81b2a00640 1 --2- 172.21.15.96:0/3141299576 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f81b40cd2a0 0x7f81b40b4d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:22.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.796+0000 7f81b2a00640 1 -- 172.21.15.96:0/3141299576 >> v1:172.21.15.123:6789/0 conn(0x7f81b4153510 legacy=0x7f81b40ccb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:22.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.796+0000 7f81b2a00640 1 -- 172.21.15.96:0/3141299576 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81b40b4270 con 0x7f81b415cb60 2024-05-28T06:04:22.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.796+0000 7f81b2a00640 1 -- 172.21.15.96:0/3141299576 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f81ac0019f0 con 0x7f81b415cb60 2024-05-28T06:04:22.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.796+0000 7f81b2a00640 1 -- 172.21.15.96:0/3141299576 >> v1:172.21.15.96:6789/0 conn(0x7f81b415cb60 legacy=0x7f81b415cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:22.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.796+0000 7f81b2a00640 1 --2- 172.21.15.96:0/3141299576 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f81980039e0 0x7f8198003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:22.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.796+0000 7f81b2a00640 1 --2- 172.21.15.96:0/3141299576 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8198004800 0x7f8198004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:22.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.797+0000 7f81b2a00640 1 --2- 172.21.15.96:0/3141299576 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f81b40cd2a0 0x7f8198005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:22.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.797+0000 7f81b2a00640 1 -- 172.21.15.96:0/3141299576 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f81b415d6e0 con 0x7f8198004800 2024-05-28T06:04:22.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.797+0000 7f81b2a00640 1 -- 172.21.15.96:0/3141299576 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f819800baf0 con 0x7f81980039e0 2024-05-28T06:04:22.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.797+0000 7f81b2a00640 1 -- 172.21.15.96:0/3141299576 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f819800bcf0 con 0x7f81b40cd2a0 2024-05-28T06:04:22.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.797+0000 7f81abe00640 1 --2- 172.21.15.96:0/3141299576 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8198004800 0x7f8198004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:22.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.797+0000 7f81b3400640 1 --2- 172.21.15.96:0/3141299576 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f81980039e0 0x7f8198003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:22.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.797+0000 7f81b3e00640 1 --2- 172.21.15.96:0/3141299576 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f81b40cd2a0 0x7f8198005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:22.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.797+0000 7f81abe00640 1 -- 172.21.15.96:0/3141299576 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f81b40cd2a0 msgr2=0x7f8198005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:22.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.797+0000 7f81abe00640 1 --2- 172.21.15.96:0/3141299576 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f81b40cd2a0 0x7f8198005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:22.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.797+0000 7f81abe00640 1 -- 172.21.15.96:0/3141299576 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f81980039e0 msgr2=0x7f8198003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:22.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.797+0000 7f81abe00640 1 --2- 172.21.15.96:0/3141299576 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f81980039e0 0x7f8198003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:22.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.797+0000 7f81abe00640 1 -- 172.21.15.96:0/3141299576 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8198003520 con 0x7f8198004800 2024-05-28T06:04:22.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.797+0000 7f81b3400640 1 --2- 172.21.15.96:0/3141299576 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f81980039e0 0x7f8198003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-28T06:04:22.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.797+0000 7f81b3e00640 1 --2- 172.21.15.96:0/3141299576 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f81b40cd2a0 0x7f8198005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:04:22.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.798+0000 7f81abe00640 1 --2- 172.21.15.96:0/3141299576 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8198004800 0x7f8198004c80 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f819c009ac0 tx=0x7f8198005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=713dd9a0e24007f1 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:22.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.798+0000 7f81b2a00640 1 -- 172.21.15.96:0/3141299576 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f819c036070 con 0x7f8198004800 2024-05-28T06:04:22.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.798+0000 7f81b2a00640 1 -- 172.21.15.96:0/3141299576 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f819c031400 con 0x7f8198004800 2024-05-28T06:04:22.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.798+0000 7f81b2a00640 1 -- 172.21.15.96:0/3141299576 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f819c009690 con 0x7f8198004800 2024-05-28T06:04:22.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.799+0000 7f81b8a00640 1 -- 172.21.15.96:0/3141299576 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8198004800 msgr2=0x7f8198004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:22.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.799+0000 7f81b8a00640 1 --2- 172.21.15.96:0/3141299576 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8198004800 0x7f8198004c80 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f819c009ac0 tx=0x7f8198005830 comp rx=0 tx=0).stop 2024-05-28T06:04:22.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.799+0000 7f81b3e00640 1 -- 172.21.15.96:0/3141299576 reap_dead start 2024-05-28T06:04:22.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.799+0000 7f81b8a00640 1 -- 172.21.15.96:0/3141299576 shutdown_connections 2024-05-28T06:04:22.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.799+0000 7f81b8a00640 1 -- 172.21.15.96:0/3141299576 >> 172.21.15.96:0/3141299576 conn(0x7f81b406b900 msgr2=0x7f81b406bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:22.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.799+0000 7f81b8a00640 1 -- 172.21.15.96:0/3141299576 shutdown_connections 2024-05-28T06:04:22.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.799+0000 7f81b8a00640 1 -- 172.21.15.96:0/3141299576 wait complete. 2024-05-28T06:04:22.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.801+0000 7f81b8a00640 1 Processor -- start 2024-05-28T06:04:22.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.801+0000 7f81b8a00640 1 -- start start 2024-05-28T06:04:22.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.802+0000 7f81b8a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8198004800 0x7f81b41f81a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:22.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.802+0000 7f81b8a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f81b40adb40 0x7f81b40adfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:22.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.802+0000 7f81b8a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f81b4072500 0x7f81b4072980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:22.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.802+0000 7f81b8a00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f81b40b48c0 con 0x7f81b4072500 2024-05-28T06:04:22.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.802+0000 7f81b8a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f81b40b4d50 con 0x7f8198004800 2024-05-28T06:04:22.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.802+0000 7f81b8a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f81b40b4ed0 con 0x7f81b40adb40 2024-05-28T06:04:22.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.802+0000 7f81b3e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f81b4072500 0x7f81b4072980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:22.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.802+0000 7f81b3e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f81b4072500 0x7f81b4072980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48196/0 (socket says 172.21.15.96:48196) 2024-05-28T06:04:22.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.802+0000 7f81b3e00640 1 -- 172.21.15.96:0/1982004415 learned_addr learned my addr 172.21.15.96:0/1982004415 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:22.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.802+0000 7f81abe00640 1 --2- 172.21.15.96:0/1982004415 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f81b40adb40 0x7f81b40adfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:22.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.802+0000 7f81b3400640 1 --2- 172.21.15.96:0/1982004415 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8198004800 0x7f81b41f81a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:22.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.803+0000 7f81b3e00640 1 -- 172.21.15.96:0/1982004415 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f81b40adb40 msgr2=0x7f81b40adfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:22.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.803+0000 7f81b3e00640 1 --2- 172.21.15.96:0/1982004415 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f81b40adb40 0x7f81b40adfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:22.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.803+0000 7f81b3e00640 1 -- 172.21.15.96:0/1982004415 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8198004800 msgr2=0x7f81b41f81a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:22.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.803+0000 7f81b3e00640 1 --2- 172.21.15.96:0/1982004415 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8198004800 0x7f81b41f81a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:22.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.803+0000 7f81b3e00640 1 -- 172.21.15.96:0/1982004415 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81b40ae600 con 0x7f81b4072500 2024-05-28T06:04:22.807 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.803+0000 7f81b3e00640 1 --2- 172.21.15.96:0/1982004415 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f81b4072500 0x7f81b4072980 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f81ac000f80 tx=0x7f81ac00b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:22.807 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.803+0000 7f81ab400640 1 -- 172.21.15.96:0/1982004415 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81ac00bb80 con 0x7f81b4072500 2024-05-28T06:04:22.807 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.803+0000 7f81ab400640 1 -- 172.21.15.96:0/1982004415 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81ac00bd40 con 0x7f81b4072500 2024-05-28T06:04:22.807 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.803+0000 7f81ab400640 1 -- 172.21.15.96:0/1982004415 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81ac007350 con 0x7f81b4072500 2024-05-28T06:04:22.807 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.803+0000 7f81b8a00640 1 -- 172.21.15.96:0/1982004415 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81b40ae8e0 con 0x7f81b4072500 2024-05-28T06:04:22.807 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.803+0000 7f81b8a00640 1 -- 172.21.15.96:0/1982004415 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81b4152600 con 0x7f81b4072500 2024-05-28T06:04:22.808 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.804+0000 7f81b8a00640 1 -- 172.21.15.96:0/1982004415 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81b40689e0 con 0x7f81b4072500 2024-05-28T06:04:22.808 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.805+0000 7f81ab400640 1 -- 172.21.15.96:0/1982004415 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f81ac01b020 con 0x7f81b4072500 2024-05-28T06:04:22.808 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.806+0000 7f81ab400640 1 --2- 172.21.15.96:0/1982004415 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f819006c920 0x7f819006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:22.808 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.806+0000 7f81ab400640 1 -- 172.21.15.96:0/1982004415 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f81ac08bda0 con 0x7f81b4072500 2024-05-28T06:04:22.808 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.807+0000 7f81b3400640 1 --2- 172.21.15.96:0/1982004415 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f819006c920 0x7f819006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:22.809 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.808+0000 7f81b3400640 1 --2- 172.21.15.96:0/1982004415 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f819006c920 0x7f819006ede0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f81a4007fd0 tx=0x7f81a4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:22.811 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:22.810+0000 7f81ab400640 1 -- 172.21.15.96:0/1982004415 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f81ac05a280 con 0x7f81b4072500 2024-05-28T06:04:23.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.012+0000 7f81b8a00640 1 -- 172.21.15.96:0/1982004415 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f81b40aeab0 con 0x7f81b4072500 2024-05-28T06:04:23.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.013+0000 7f81ab400640 1 -- 172.21.15.96:0/1982004415 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) ==== 93+0+10 (secure 0 0 0) 0x7f81ac059b80 con 0x7f81b4072500 2024-05-28T06:04:23.016 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 1 2024-05-28T06:04:23.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.016+0000 7f81b8a00640 1 -- 172.21.15.96:0/1982004415 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f819006c920 msgr2=0x7f819006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:23.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.016+0000 7f81b8a00640 1 --2- 172.21.15.96:0/1982004415 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f819006c920 0x7f819006ede0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f81a4007fd0 tx=0x7f81a4009040 comp rx=0 tx=0).stop 2024-05-28T06:04:23.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.016+0000 7f81b8a00640 1 -- 172.21.15.96:0/1982004415 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f81b4072500 msgr2=0x7f81b4072980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:23.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.016+0000 7f81b8a00640 1 --2- 172.21.15.96:0/1982004415 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f81b4072500 0x7f81b4072980 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f81ac000f80 tx=0x7f81ac00b4a0 comp rx=0 tx=0).stop 2024-05-28T06:04:23.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.016+0000 7f81b8a00640 1 -- 172.21.15.96:0/1982004415 shutdown_connections 2024-05-28T06:04:23.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.016+0000 7f81b8a00640 1 --2- 172.21.15.96:0/1982004415 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f819006c920 0x7f819006ede0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:23.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.016+0000 7f81b8a00640 1 --2- 172.21.15.96:0/1982004415 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f81b4072500 0x7f81b4072980 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:23.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.016+0000 7f81b8a00640 1 --2- 172.21.15.96:0/1982004415 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f81b40adb40 0x7f81b40adfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:23.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.016+0000 7f81b8a00640 1 --2- 172.21.15.96:0/1982004415 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8198004800 0x7f81b41f81a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:23.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.016+0000 7f81b8a00640 1 -- 172.21.15.96:0/1982004415 >> 172.21.15.96:0/1982004415 conn(0x7f81b406b900 msgr2=0x7f81b40b1e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:23.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.016+0000 7f81b8a00640 1 -- 172.21.15.96:0/1982004415 shutdown_connections 2024-05-28T06:04:23.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.016+0000 7f81b8a00640 1 -- 172.21.15.96:0/1982004415 wait complete. 2024-05-28T06:04:23.037 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-28T06:04:23.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.240+0000 7f4f19400640 1 Processor -- start 2024-05-28T06:04:23.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.241+0000 7f4f19400640 1 -- start start 2024-05-28T06:04:23.257 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.242+0000 7f4f19400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f4f140cd2a0 0x7f4f140b4d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:23.258 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.242+0000 7f4f19400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4f1415cb60 0x7f4f1415cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:23.258 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.242+0000 7f4f19400640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f1415d900 con 0x7f4f14153510 2024-05-28T06:04:23.259 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.242+0000 7f4f19400640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f4f141538f0 con 0x7f4f140cd2a0 2024-05-28T06:04:23.259 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.242+0000 7f4f19400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f4f140b48c0 con 0x7f4f1415cb60 2024-05-28T06:04:23.259 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.242+0000 7f4f13400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f4f140cd2a0 0x7f4f140b4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:23.260 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.242+0000 7f4f13400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f4f140cd2a0 0x7f4f140b4d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48212/0 (socket says 172.21.15.96:48212) 2024-05-28T06:04:23.260 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.242+0000 7f4f13400640 1 -- 172.21.15.96:0/1105478806 learned_addr learned my addr 172.21.15.96:0/1105478806 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:23.260 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.243+0000 7f4f18a00640 1 --2- 172.21.15.96:0/1105478806 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4f1415cb60 0x7f4f1415cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:23.260 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.243+0000 7f4f13400640 1 -- 172.21.15.96:0/1105478806 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4f1415cb60 msgr2=0x7f4f1415cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:23.260 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.243+0000 7f4f13400640 1 --2- 172.21.15.96:0/1105478806 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4f1415cb60 0x7f4f1415cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:23.261 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.243+0000 7f4f13400640 1 -- 172.21.15.96:0/1105478806 >> v1:172.21.15.123:6789/0 conn(0x7f4f14153510 legacy=0x7f4f140ccb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:23.262 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.243+0000 7f4f13400640 1 -- 172.21.15.96:0/1105478806 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f140b3f90 con 0x7f4f140cd2a0 2024-05-28T06:04:23.262 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.243+0000 7f4f18a00640 1 --2- 172.21.15.96:0/1105478806 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4f1415cb60 0x7f4f1415cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:04:23.263 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.244+0000 7f4f13400640 1 --2- 172.21.15.96:0/1105478806 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f140cd2a0 0x7f4f140b4d50 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f4f00009870 tx=0x7f4f0002a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0d9f79c08ba1fdf server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:23.263 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.244+0000 7f4f12a00640 1 -- 172.21.15.96:0/1105478806 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f00037070 con 0x7f4f140cd2a0 2024-05-28T06:04:23.263 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.244+0000 7f4f12a00640 1 -- 172.21.15.96:0/1105478806 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f00032440 con 0x7f4f140cd2a0 2024-05-28T06:04:23.263 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.244+0000 7f4f12a00640 1 -- 172.21.15.96:0/1105478806 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f00032810 con 0x7f4f140cd2a0 2024-05-28T06:04:23.266 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.245+0000 7f4f19400640 1 -- 172.21.15.96:0/1105478806 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f140cd2a0 msgr2=0x7f4f140b4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:23.266 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.245+0000 7f4f19400640 1 --2- 172.21.15.96:0/1105478806 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f140cd2a0 0x7f4f140b4d50 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f4f00009870 tx=0x7f4f0002a180 comp rx=0 tx=0).stop 2024-05-28T06:04:23.266 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.245+0000 7f4f19400640 1 -- 172.21.15.96:0/1105478806 shutdown_connections 2024-05-28T06:04:23.266 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.245+0000 7f4f19400640 1 --2- 172.21.15.96:0/1105478806 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4f1415cb60 0x7f4f1415cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:23.266 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.245+0000 7f4f19400640 1 --2- 172.21.15.96:0/1105478806 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f140cd2a0 0x7f4f140b4d50 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:23.266 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.245+0000 7f4f19400640 1 -- 172.21.15.96:0/1105478806 >> 172.21.15.96:0/1105478806 conn(0x7f4f1406b900 msgr2=0x7f4f1406bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:23.269 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.245+0000 7f4f19400640 1 -- 172.21.15.96:0/1105478806 shutdown_connections 2024-05-28T06:04:23.269 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.245+0000 7f4f19400640 1 -- 172.21.15.96:0/1105478806 wait complete. 2024-05-28T06:04:23.269 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.247+0000 7f4f19400640 1 Processor -- start 2024-05-28T06:04:23.269 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.247+0000 7f4f19400640 1 -- start start 2024-05-28T06:04:23.269 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.248+0000 7f4f19400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4f140cd2a0 0x7f4f140cf960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:23.269 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.248+0000 7f4f19400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f14153510 0x7f4f14151e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:23.269 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.248+0000 7f4f19400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4f1415cb60 0x7f4f140ae2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:23.269 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.248+0000 7f4f19400640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4f1415d4f0 con 0x7f4f14153510 2024-05-28T06:04:23.269 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.248+0000 7f4f19400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f4f140b4550 con 0x7f4f1415cb60 2024-05-28T06:04:23.269 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.248+0000 7f4f19400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f4f1415d670 con 0x7f4f140cd2a0 2024-05-28T06:04:23.270 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.248+0000 7f4f13400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f14153510 0x7f4f14151e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:23.270 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.248+0000 7f4f13400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f14153510 0x7f4f14151e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48226/0 (socket says 172.21.15.96:48226) 2024-05-28T06:04:23.270 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.248+0000 7f4f13400640 1 -- 172.21.15.96:0/1888176440 learned_addr learned my addr 172.21.15.96:0/1888176440 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:23.270 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.248+0000 7f4f13e00640 1 --2- 172.21.15.96:0/1888176440 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4f140cd2a0 0x7f4f140cf960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:23.270 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.248+0000 7f4f13400640 1 -- 172.21.15.96:0/1888176440 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4f140cd2a0 msgr2=0x7f4f140cf960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:23.270 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.248+0000 7f4f18a00640 1 --2- 172.21.15.96:0/1888176440 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4f1415cb60 0x7f4f140ae2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:23.270 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.249+0000 7f4f13400640 1 --2- 172.21.15.96:0/1888176440 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4f140cd2a0 0x7f4f140cf960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:23.270 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.249+0000 7f4f13400640 1 -- 172.21.15.96:0/1888176440 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4f1415cb60 msgr2=0x7f4f140ae2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:23.270 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.249+0000 7f4f13400640 1 --2- 172.21.15.96:0/1888176440 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4f1415cb60 0x7f4f140ae2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:23.270 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.249+0000 7f4f13400640 1 -- 172.21.15.96:0/1888176440 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f140ce900 con 0x7f4f14153510 2024-05-28T06:04:23.270 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.249+0000 7f4f13e00640 1 --2- 172.21.15.96:0/1888176440 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4f140cd2a0 0x7f4f140cf960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:04:23.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.249+0000 7f4f13400640 1 --2- 172.21.15.96:0/1888176440 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f14153510 0x7f4f14151e90 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f4f00009870 tx=0x7f4f000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:23.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.249+0000 7f4f0be00640 1 -- 172.21.15.96:0/1888176440 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f00041020 con 0x7f4f14153510 2024-05-28T06:04:23.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.249+0000 7f4f0be00640 1 -- 172.21.15.96:0/1888176440 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f00030970 con 0x7f4f14153510 2024-05-28T06:04:23.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.249+0000 7f4f0be00640 1 -- 172.21.15.96:0/1888176440 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f00037040 con 0x7f4f14153510 2024-05-28T06:04:23.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.249+0000 7f4f19400640 1 -- 172.21.15.96:0/1888176440 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f140ceba0 con 0x7f4f14153510 2024-05-28T06:04:23.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.249+0000 7f4f19400640 1 -- 172.21.15.96:0/1888176440 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f140cf160 con 0x7f4f14153510 2024-05-28T06:04:23.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.251+0000 7f4f19400640 1 -- 172.21.15.96:0/1888176440 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f140689e0 con 0x7f4f14153510 2024-05-28T06:04:23.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.253+0000 7f4f0be00640 1 -- 172.21.15.96:0/1888176440 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4f00044020 con 0x7f4f14153510 2024-05-28T06:04:23.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.253+0000 7f4f0be00640 1 --2- 172.21.15.96:0/1888176440 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4ef406c8f0 0x7f4ef406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:23.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.254+0000 7f4f0be00640 1 -- 172.21.15.96:0/1888176440 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f4f000ac540 con 0x7f4f14153510 2024-05-28T06:04:23.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.254+0000 7f4f13e00640 1 --2- 172.21.15.96:0/1888176440 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4ef406c8f0 0x7f4ef406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:23.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.255+0000 7f4f13e00640 1 --2- 172.21.15.96:0/1888176440 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4ef406c8f0 0x7f4ef406edb0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f4f04003eb0 tx=0x7f4f04009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:23.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.260+0000 7f4f0be00640 1 -- 172.21.15.96:0/1888176440 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4f0004b030 con 0x7f4f14153510 2024-05-28T06:04:23.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.460+0000 7f4f19400640 1 -- 172.21.15.96:0/1888176440 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f4f140caa70 con 0x7f4f14153510 2024-05-28T06:04:23.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.461+0000 7f4f0be00640 1 -- 172.21.15.96:0/1888176440 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) ==== 104+0+11 (secure 0 0 0) 0x7f4f0007a970 con 0x7f4f14153510 2024-05-28T06:04:23.463 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 32 2024-05-28T06:04:23.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.464+0000 7f4f19400640 1 -- 172.21.15.96:0/1888176440 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4ef406c8f0 msgr2=0x7f4ef406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:23.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.464+0000 7f4f19400640 1 --2- 172.21.15.96:0/1888176440 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4ef406c8f0 0x7f4ef406edb0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f4f04003eb0 tx=0x7f4f04009040 comp rx=0 tx=0).stop 2024-05-28T06:04:23.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.464+0000 7f4f19400640 1 -- 172.21.15.96:0/1888176440 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f14153510 msgr2=0x7f4f14151e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:23.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.464+0000 7f4f19400640 1 --2- 172.21.15.96:0/1888176440 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f14153510 0x7f4f14151e90 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f4f00009870 tx=0x7f4f000046c0 comp rx=0 tx=0).stop 2024-05-28T06:04:23.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.464+0000 7f4f19400640 1 -- 172.21.15.96:0/1888176440 shutdown_connections 2024-05-28T06:04:23.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.464+0000 7f4f19400640 1 --2- 172.21.15.96:0/1888176440 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4ef406c8f0 0x7f4ef406edb0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:23.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.464+0000 7f4f19400640 1 --2- 172.21.15.96:0/1888176440 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4f1415cb60 0x7f4f140ae2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:23.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.464+0000 7f4f19400640 1 --2- 172.21.15.96:0/1888176440 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f14153510 0x7f4f14151e90 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:23.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.464+0000 7f4f19400640 1 --2- 172.21.15.96:0/1888176440 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4f140cd2a0 0x7f4f140cf960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:23.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.464+0000 7f4f19400640 1 -- 172.21.15.96:0/1888176440 >> 172.21.15.96:0/1888176440 conn(0x7f4f1406b900 msgr2=0x7f4f1406f2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:23.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.464+0000 7f4f19400640 1 -- 172.21.15.96:0/1888176440 shutdown_connections 2024-05-28T06:04:23.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.464+0000 7f4f19400640 1 -- 172.21.15.96:0/1888176440 wait complete. 2024-05-28T06:04:23.487 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-28T06:04:23.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.691+0000 7f721f200640 1 Processor -- start 2024-05-28T06:04:23.693 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.691+0000 7f721f200640 1 -- start start 2024-05-28T06:04:23.694 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.692+0000 7f721f200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f721815d270 0x7f72180b4d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:23.695 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.692+0000 7f721f200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f721815d680 con 0x7f7218153510 2024-05-28T06:04:23.695 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.692+0000 7f721f200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f721815d900 con 0x7f72180cd2a0 2024-05-28T06:04:23.695 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.692+0000 7f721f200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f72181538f0 con 0x7f721815d270 2024-05-28T06:04:23.695 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.693+0000 7f721de00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f7218153510 0x7f72180ccb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:35260/0 (socket says 172.21.15.96:35260) 2024-05-28T06:04:23.695 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.693+0000 7f721de00640 1 -- 172.21.15.96:0/54747129 learned_addr learned my addr 172.21.15.96:0/54747129 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:04:23.695 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.693+0000 7f721e800640 1 --2- 172.21.15.96:0/54747129 >> v2:172.21.15.96:3300/0 conn(0x7f721815d270 0x7f72180b4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:23.695 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.693+0000 7f721ca00640 1 -- 172.21.15.96:0/54747129 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 835541962 0 0) 0x7f720c001110 con 0x7f7218153510 2024-05-28T06:04:23.695 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.693+0000 7f721ca00640 1 -- 172.21.15.96:0/54747129 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7200003520 con 0x7f7218153510 2024-05-28T06:04:23.695 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.693+0000 7f721e800640 1 -- 172.21.15.96:0/54747129 >> v1:172.21.15.123:6789/0 conn(0x7f72180cd2a0 legacy=0x7f721815cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:23.695 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.693+0000 7f721e800640 1 -- 172.21.15.96:0/54747129 >> v1:172.21.15.96:6789/0 conn(0x7f7218153510 legacy=0x7f72180ccb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:23.695 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.693+0000 7f721e800640 1 -- 172.21.15.96:0/54747129 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72180b3f90 con 0x7f721815d270 2024-05-28T06:04:23.696 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.694+0000 7f721e800640 1 --2- 172.21.15.96:0/54747129 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f721815d270 0x7f72180b4d50 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f7214009d90 tx=0x7f721402a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f792ac699eb9b0e server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:23.696 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.695+0000 7f721ca00640 1 -- 172.21.15.96:0/54747129 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7214007740 con 0x7f721815d270 2024-05-28T06:04:23.696 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.695+0000 7f721ca00640 1 -- 172.21.15.96:0/54747129 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72140078e0 con 0x7f721815d270 2024-05-28T06:04:23.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.695+0000 7f721ca00640 1 -- 172.21.15.96:0/54747129 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7214007cb0 con 0x7f721815d270 2024-05-28T06:04:23.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.695+0000 7f721f200640 1 -- 172.21.15.96:0/54747129 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f721815d270 msgr2=0x7f72180b4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:23.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.695+0000 7f721f200640 1 --2- 172.21.15.96:0/54747129 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f721815d270 0x7f72180b4d50 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f7214009d90 tx=0x7f721402a660 comp rx=0 tx=0).stop 2024-05-28T06:04:23.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.695+0000 7f721f200640 1 -- 172.21.15.96:0/54747129 shutdown_connections 2024-05-28T06:04:23.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.695+0000 7f721f200640 1 --2- 172.21.15.96:0/54747129 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f721815d270 0x7f72180b4d50 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:23.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.695+0000 7f721f200640 1 -- 172.21.15.96:0/54747129 >> 172.21.15.96:0/54747129 conn(0x7f721806b900 msgr2=0x7f721806bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:23.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.695+0000 7f721f200640 1 -- 172.21.15.96:0/54747129 shutdown_connections 2024-05-28T06:04:23.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.695+0000 7f721f200640 1 -- 172.21.15.96:0/54747129 wait complete. 2024-05-28T06:04:23.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.697+0000 7f721f200640 1 Processor -- start 2024-05-28T06:04:23.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.697+0000 7f721f200640 1 -- start start 2024-05-28T06:04:23.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.698+0000 7f721f200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f72180cd2a0 0x7f721814a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:23.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.698+0000 7f721f200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7218153510 0x7f721815b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:23.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.698+0000 7f721f200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f721815d270 0x7f721815b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:23.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.698+0000 7f721f200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f72180b4550 con 0x7f721815d270 2024-05-28T06:04:23.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.698+0000 7f721f200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f72180b48c0 con 0x7f7218153510 2024-05-28T06:04:23.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.698+0000 7f721f200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f721806e940 con 0x7f72180cd2a0 2024-05-28T06:04:23.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.698+0000 7f721e800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f721815d270 0x7f721815b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:23.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.698+0000 7f721e800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f721815d270 0x7f721815b970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48254/0 (socket says 172.21.15.96:48254) 2024-05-28T06:04:23.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.698+0000 7f721e800640 1 -- 172.21.15.96:0/3154274541 learned_addr learned my addr 172.21.15.96:0/3154274541 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:23.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.699+0000 7f721d400640 1 --2- 172.21.15.96:0/3154274541 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7218153510 0x7f721815b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:23.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.699+0000 7f721e800640 1 -- 172.21.15.96:0/3154274541 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f72180cd2a0 msgr2=0x7f721814a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:23.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.699+0000 7f721de00640 1 --2- 172.21.15.96:0/3154274541 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f72180cd2a0 0x7f721814a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:23.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.699+0000 7f721e800640 1 --2- 172.21.15.96:0/3154274541 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f72180cd2a0 0x7f721814a4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:23.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.699+0000 7f721e800640 1 -- 172.21.15.96:0/3154274541 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7218153510 msgr2=0x7f721815b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:23.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.699+0000 7f721e800640 1 --2- 172.21.15.96:0/3154274541 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7218153510 0x7f721815b430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:23.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.699+0000 7f721e800640 1 -- 172.21.15.96:0/3154274541 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72180cdd50 con 0x7f721815d270 2024-05-28T06:04:23.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.699+0000 7f721e800640 1 --2- 172.21.15.96:0/3154274541 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f721815d270 0x7f721815b970 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f721402ac50 tx=0x7f7214004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:23.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.699+0000 7f7212000640 1 -- 172.21.15.96:0/3154274541 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f721403f070 con 0x7f721815d270 2024-05-28T06:04:23.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.699+0000 7f7212000640 1 -- 172.21.15.96:0/3154274541 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7214031bc0 con 0x7f721815d270 2024-05-28T06:04:23.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.699+0000 7f7212000640 1 -- 172.21.15.96:0/3154274541 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7214009450 con 0x7f721815d270 2024-05-28T06:04:23.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.699+0000 7f721f200640 1 -- 172.21.15.96:0/3154274541 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72181f8410 con 0x7f721815d270 2024-05-28T06:04:23.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.699+0000 7f721f200640 1 -- 172.21.15.96:0/3154274541 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72181f89d0 con 0x7f721815d270 2024-05-28T06:04:23.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.701+0000 7f721f200640 1 -- 172.21.15.96:0/3154274541 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72180689e0 con 0x7f721815d270 2024-05-28T06:04:23.705 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.701+0000 7f7212000640 1 -- 172.21.15.96:0/3154274541 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f721403b410 con 0x7f721815d270 2024-05-28T06:04:23.705 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.702+0000 7f7212000640 1 --2- 172.21.15.96:0/3154274541 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f71fc06c920 0x7f71fc06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:23.705 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.702+0000 7f7212000640 1 -- 172.21.15.96:0/3154274541 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f72140ad330 con 0x7f721815d270 2024-05-28T06:04:23.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.703+0000 7f721de00640 1 --2- 172.21.15.96:0/3154274541 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f71fc06c920 0x7f71fc06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:23.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.704+0000 7f721de00640 1 --2- 172.21.15.96:0/3154274541 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f71fc06c920 0x7f71fc06ede0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f720c0012b0 tx=0x7f720c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:23.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.706+0000 7f7212000640 1 -- 172.21.15.96:0/3154274541 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f721407b6c0 con 0x7f721815d270 2024-05-28T06:04:23.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.904+0000 7f721f200640 1 -- 172.21.15.96:0/3154274541 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f72180b3c40 con 0x7f721815d270 2024-05-28T06:04:23.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.905+0000 7f7212000640 1 -- 172.21.15.96:0/3154274541 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) ==== 100+0+11 (secure 0 0 0) 0x7f721407afc0 con 0x7f721815d270 2024-05-28T06:04:23.907 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 64 2024-05-28T06:04:23.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.907+0000 7f721f200640 1 -- 172.21.15.96:0/3154274541 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f71fc06c920 msgr2=0x7f71fc06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:23.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.907+0000 7f721f200640 1 --2- 172.21.15.96:0/3154274541 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f71fc06c920 0x7f71fc06ede0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f720c0012b0 tx=0x7f720c009040 comp rx=0 tx=0).stop 2024-05-28T06:04:23.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.907+0000 7f721f200640 1 -- 172.21.15.96:0/3154274541 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f721815d270 msgr2=0x7f721815b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:23.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.907+0000 7f721f200640 1 --2- 172.21.15.96:0/3154274541 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f721815d270 0x7f721815b970 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f721402ac50 tx=0x7f7214004a40 comp rx=0 tx=0).stop 2024-05-28T06:04:23.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.907+0000 7f721f200640 1 -- 172.21.15.96:0/3154274541 shutdown_connections 2024-05-28T06:04:23.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.907+0000 7f721f200640 1 --2- 172.21.15.96:0/3154274541 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f71fc06c920 0x7f71fc06ede0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:23.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.908+0000 7f721f200640 1 --2- 172.21.15.96:0/3154274541 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f721815d270 0x7f721815b970 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:23.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.908+0000 7f721f200640 1 --2- 172.21.15.96:0/3154274541 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7218153510 0x7f721815b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:23.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.908+0000 7f721f200640 1 --2- 172.21.15.96:0/3154274541 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f72180cd2a0 0x7f721814a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:23.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.908+0000 7f721f200640 1 -- 172.21.15.96:0/3154274541 >> 172.21.15.96:0/3154274541 conn(0x7f721806b900 msgr2=0x7f721815fee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:23.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.908+0000 7f721f200640 1 -- 172.21.15.96:0/3154274541 shutdown_connections 2024-05-28T06:04:23.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:23.908+0000 7f721f200640 1 -- 172.21.15.96:0/3154274541 wait complete. 2024-05-28T06:04:23.927 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-28T06:04:24.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.120+0000 7f3ccc600640 1 Processor -- start 2024-05-28T06:04:24.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.121+0000 7f3ccc600640 1 -- start start 2024-05-28T06:04:24.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.122+0000 7f3ccc600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f3cc415c900 0x7f3cc40d0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:24.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.122+0000 7f3ccc600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f3cc409bb50 0x7f3cc40b29e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:24.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.122+0000 7f3ccc600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3cc40d1430 con 0x7f3cc40d1010 2024-05-28T06:04:24.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.122+0000 7f3ccc600640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f3cc40b2550 con 0x7f3cc409bb50 2024-05-28T06:04:24.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.122+0000 7f3ccc600640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f3cc40b1960 con 0x7f3cc415c900 2024-05-28T06:04:24.125 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.122+0000 7f3cca600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f3cc409bb50 0x7f3cc40b29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:24.125 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.122+0000 7f3cca600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f3cc409bb50 0x7f3cc40b29e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48264/0 (socket says 172.21.15.96:48264) 2024-05-28T06:04:24.125 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.122+0000 7f3cca600640 1 -- 172.21.15.96:0/603276682 learned_addr learned my addr 172.21.15.96:0/603276682 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:24.125 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.123+0000 7f3cc9c00640 1 --2- 172.21.15.96:0/603276682 >> v2:172.21.15.123:3300/0 conn(0x7f3cc415c900 0x7f3cc40d0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:24.125 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.123+0000 7f3cca600640 1 -- 172.21.15.96:0/603276682 >> v2:172.21.15.123:3300/0 conn(0x7f3cc415c900 msgr2=0x7f3cc40d0ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:24.125 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.123+0000 7f3cca600640 1 --2- 172.21.15.96:0/603276682 >> v2:172.21.15.123:3300/0 conn(0x7f3cc415c900 0x7f3cc40d0ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:24.125 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.123+0000 7f3cca600640 1 -- 172.21.15.96:0/603276682 >> v1:172.21.15.123:6789/0 conn(0x7f3cc40d1010 legacy=0x7f3cc409b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:24.125 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.123+0000 7f3cca600640 1 -- 172.21.15.96:0/603276682 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cc40b1c20 con 0x7f3cc409bb50 2024-05-28T06:04:24.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.124+0000 7f3cca600640 1 --2- 172.21.15.96:0/603276682 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3cc409bb50 0x7f3cc40b29e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f3cc40730a0 tx=0x7f3cbc02a2c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4c73ec1b6b830b4 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:24.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.124+0000 7f3cc3e00640 1 -- 172.21.15.96:0/603276682 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cbc037070 con 0x7f3cc409bb50 2024-05-28T06:04:24.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.124+0000 7f3cc3e00640 1 -- 172.21.15.96:0/603276682 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3cbc032440 con 0x7f3cc409bb50 2024-05-28T06:04:24.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.124+0000 7f3cc3e00640 1 -- 172.21.15.96:0/603276682 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cbc032810 con 0x7f3cc409bb50 2024-05-28T06:04:24.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.125+0000 7f3ccc600640 1 -- 172.21.15.96:0/603276682 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3cc409bb50 msgr2=0x7f3cc40b29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:24.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.125+0000 7f3ccc600640 1 --2- 172.21.15.96:0/603276682 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3cc409bb50 0x7f3cc40b29e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f3cc40730a0 tx=0x7f3cbc02a2c0 comp rx=0 tx=0).stop 2024-05-28T06:04:24.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.125+0000 7f3ccc600640 1 -- 172.21.15.96:0/603276682 shutdown_connections 2024-05-28T06:04:24.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.125+0000 7f3ccc600640 1 --2- 172.21.15.96:0/603276682 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3cc409bb50 0x7f3cc40b29e0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:24.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.125+0000 7f3ccc600640 1 --2- 172.21.15.96:0/603276682 >> v2:172.21.15.123:3300/0 conn(0x7f3cc415c900 0x7f3cc40d0ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:24.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.125+0000 7f3ccc600640 1 -- 172.21.15.96:0/603276682 >> 172.21.15.96:0/603276682 conn(0x7f3cc40b3fe0 msgr2=0x7f3cc40b43f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:24.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.125+0000 7f3ccc600640 1 -- 172.21.15.96:0/603276682 shutdown_connections 2024-05-28T06:04:24.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.125+0000 7f3ccc600640 1 -- 172.21.15.96:0/603276682 wait complete. 2024-05-28T06:04:24.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.127+0000 7f3ccc600640 1 Processor -- start 2024-05-28T06:04:24.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.127+0000 7f3ccc600640 1 -- start start 2024-05-28T06:04:24.131 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.128+0000 7f3ccc600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3cc409bb50 0x7f3cc40c95c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:24.131 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.128+0000 7f3ccc600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3cc40d1010 0x7f3cc40cac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:24.131 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.128+0000 7f3ccc600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3cc415c900 0x7f3cc415f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:24.131 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.128+0000 7f3ccc600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f3cc415cce0 con 0x7f3cc409bb50 2024-05-28T06:04:24.131 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.128+0000 7f3ccc600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f3cc40b21e0 con 0x7f3cc415c900 2024-05-28T06:04:24.132 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.128+0000 7f3ccc600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f3cc40b2f20 con 0x7f3cc40d1010 2024-05-28T06:04:24.132 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.128+0000 7f3cc9c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3cc409bb50 0x7f3cc40c95c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:24.132 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.128+0000 7f3cc9c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3cc409bb50 0x7f3cc40c95c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48266/0 (socket says 172.21.15.96:48266) 2024-05-28T06:04:24.132 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.128+0000 7f3cc9c00640 1 -- 172.21.15.96:0/1927010083 learned_addr learned my addr 172.21.15.96:0/1927010083 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:24.133 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.129+0000 7f3cc9c00640 1 -- 172.21.15.96:0/1927010083 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3cc40d1010 msgr2=0x7f3cc40cac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:24.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.129+0000 7f3cca600640 1 --2- 172.21.15.96:0/1927010083 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3cc415c900 0x7f3cc415f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:24.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.129+0000 7f3cc9200640 1 --2- 172.21.15.96:0/1927010083 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3cc40d1010 0x7f3cc40cac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:24.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.129+0000 7f3cc9c00640 1 --2- 172.21.15.96:0/1927010083 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3cc40d1010 0x7f3cc40cac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:24.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.129+0000 7f3cc9c00640 1 -- 172.21.15.96:0/1927010083 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3cc415c900 msgr2=0x7f3cc415f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:24.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.129+0000 7f3cc9c00640 1 --2- 172.21.15.96:0/1927010083 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3cc415c900 0x7f3cc415f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:24.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.129+0000 7f3cc9c00640 1 -- 172.21.15.96:0/1927010083 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cc415e3f0 con 0x7f3cc409bb50 2024-05-28T06:04:24.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.129+0000 7f3cca600640 1 --2- 172.21.15.96:0/1927010083 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3cc415c900 0x7f3cc415f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:04:24.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.129+0000 7f3cc9c00640 1 --2- 172.21.15.96:0/1927010083 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3cc409bb50 0x7f3cc40c95c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f3cb4007e40 tx=0x7f3cb400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:24.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.129+0000 7f3cc1600640 1 -- 172.21.15.96:0/1927010083 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cb4014620 con 0x7f3cc409bb50 2024-05-28T06:04:24.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.129+0000 7f3cc1600640 1 -- 172.21.15.96:0/1927010083 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3cb4019070 con 0x7f3cc409bb50 2024-05-28T06:04:24.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.129+0000 7f3ccc600640 1 -- 172.21.15.96:0/1927010083 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3cc415e6d0 con 0x7f3cc409bb50 2024-05-28T06:04:24.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.129+0000 7f3cc1600640 1 -- 172.21.15.96:0/1927010083 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cb4014a00 con 0x7f3cc409bb50 2024-05-28T06:04:24.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.129+0000 7f3ccc600640 1 -- 172.21.15.96:0/1927010083 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3cc415ef70 con 0x7f3cc409bb50 2024-05-28T06:04:24.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.131+0000 7f3ccc600640 1 -- 172.21.15.96:0/1927010083 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3cc40689e0 con 0x7f3cc409bb50 2024-05-28T06:04:24.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.132+0000 7f3cc1600640 1 -- 172.21.15.96:0/1927010083 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3cb4014ba0 con 0x7f3cc409bb50 2024-05-28T06:04:24.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.133+0000 7f3cc1600640 1 --2- 172.21.15.96:0/1927010083 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3ca406c920 0x7f3ca406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:24.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.133+0000 7f3cc1600640 1 -- 172.21.15.96:0/1927010083 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f3cb408c500 con 0x7f3cc409bb50 2024-05-28T06:04:24.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.133+0000 7f3cc9200640 1 --2- 172.21.15.96:0/1927010083 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3ca406c920 0x7f3ca406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:24.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.134+0000 7f3cc9200640 1 --2- 172.21.15.96:0/1927010083 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3ca406c920 0x7f3ca406ede0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f3cc40b77c0 tx=0x7f3cb0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:24.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.137+0000 7f3cc1600640 1 -- 172.21.15.96:0/1927010083 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3cb405a890 con 0x7f3cc409bb50 2024-05-28T06:04:24.374 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.372+0000 7f3ccc600640 1 -- 172.21.15.96:0/1927010083 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3cc40cfdd0 con 0x7f3cc409bb50 2024-05-28T06:04:24.376 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.373+0000 7f3cc1600640 1 -- 172.21.15.96:0/1927010083 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5178 (secure 0 0 0) 0x7f3cb405a190 con 0x7f3cc409bb50 2024-05-28T06:04:24.376 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:04:24.377 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/3349661842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":3349661842},{"type":"v1","addr":"172.21.15.96:6833","nonce":3349661842}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4270,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/3383264102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":3383264102},{"type":"v1","addr":"172.21.15.96:6835","nonce":3383264102}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4279,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/3068402544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":3068402544},{"type":"v1","addr":"172.21.15.123:6837","nonce":3068402544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-28T06:04:10.319738+0000","modified":"2024-05-28T06:04:13.704080+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4281},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4281":{"gid":4281,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.123:6835/605814935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":605814935},{"type":"v1","addr":"172.21.15.123:6835","nonce":605814935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4281,"qdb_cluster":[4281]},"id":1}]} 2024-05-28T06:04:24.378 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 7 2024-05-28T06:04:24.378 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.376+0000 7f3ccc600640 1 -- 172.21.15.96:0/1927010083 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3ca406c920 msgr2=0x7f3ca406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:24.378 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.376+0000 7f3ccc600640 1 --2- 172.21.15.96:0/1927010083 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3ca406c920 0x7f3ca406ede0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f3cc40b77c0 tx=0x7f3cb0009040 comp rx=0 tx=0).stop 2024-05-28T06:04:24.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.376+0000 7f3ccc600640 1 -- 172.21.15.96:0/1927010083 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3cc409bb50 msgr2=0x7f3cc40c95c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:24.380 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.376+0000 7f3ccc600640 1 --2- 172.21.15.96:0/1927010083 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3cc409bb50 0x7f3cc40c95c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f3cb4007e40 tx=0x7f3cb400e4b0 comp rx=0 tx=0).stop 2024-05-28T06:04:24.380 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.376+0000 7f3ccc600640 1 -- 172.21.15.96:0/1927010083 shutdown_connections 2024-05-28T06:04:24.380 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.376+0000 7f3ccc600640 1 --2- 172.21.15.96:0/1927010083 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3ca406c920 0x7f3ca406ede0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:24.380 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.376+0000 7f3ccc600640 1 --2- 172.21.15.96:0/1927010083 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3cc415c900 0x7f3cc415f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:24.380 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.376+0000 7f3ccc600640 1 --2- 172.21.15.96:0/1927010083 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3cc40d1010 0x7f3cc40cac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:24.380 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.376+0000 7f3ccc600640 1 --2- 172.21.15.96:0/1927010083 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3cc409bb50 0x7f3cc40c95c0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:24.380 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.376+0000 7f3ccc600640 1 -- 172.21.15.96:0/1927010083 >> 172.21.15.96:0/1927010083 conn(0x7f3cc40b3fe0 msgr2=0x7f3cc40b1150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:24.380 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.376+0000 7f3ccc600640 1 -- 172.21.15.96:0/1927010083 shutdown_connections 2024-05-28T06:04:24.380 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.377+0000 7f3ccc600640 1 -- 172.21.15.96:0/1927010083 wait complete. 2024-05-28T06:04:24.396 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:04:24.594 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.593+0000 7fd6ff600640 1 Processor -- start 2024-05-28T06:04:24.595 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.593+0000 7fd6ff600640 1 -- start start 2024-05-28T06:04:24.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.594+0000 7fd6ff600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fd6f806f860 0x7fd6f80b3150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:24.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.594+0000 7fd6ff600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fd6f80b3690 0x7fd6f80d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:24.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.594+0000 7fd6ff600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd6f80d09e0 0x7fd6f8160210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:24.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.594+0000 7fd6ff600640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fd6f80d0e30 con 0x7fd6f806f860 2024-05-28T06:04:24.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.594+0000 7fd6ff600640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fd6f806ed10 con 0x7fd6f80b3690 2024-05-28T06:04:24.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.594+0000 7fd6ff600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fd6f815cb00 con 0x7fd6f80d09e0 2024-05-28T06:04:24.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.594+0000 7fd6fcc00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fd6f806f860 0x7fd6f80b3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:24.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.595+0000 7fd6fcc00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fd6f806f860 0x7fd6f80b3150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48276/0 (socket says 172.21.15.96:48276) 2024-05-28T06:04:24.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.595+0000 7fd6fcc00640 1 -- 172.21.15.96:0/225636829 learned_addr learned my addr 172.21.15.96:0/225636829 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:24.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.595+0000 7fd6f7e00640 1 --2- 172.21.15.96:0/225636829 >> v2:172.21.15.123:3300/0 conn(0x7fd6f80b3690 0x7fd6f80d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:24.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.595+0000 7fd6fd600640 1 --2- 172.21.15.96:0/225636829 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd6f80d09e0 0x7fd6f8160210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:24.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.595+0000 7fd6fcc00640 1 -- 172.21.15.96:0/225636829 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd6f80d09e0 msgr2=0x7fd6f8160210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:24.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.595+0000 7fd6fcc00640 1 --2- 172.21.15.96:0/225636829 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd6f80d09e0 0x7fd6f8160210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:24.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.595+0000 7fd6fcc00640 1 -- 172.21.15.96:0/225636829 >> v2:172.21.15.123:3300/0 conn(0x7fd6f80b3690 msgr2=0x7fd6f80d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:24.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.595+0000 7fd6fcc00640 1 --2- 172.21.15.96:0/225636829 >> v2:172.21.15.123:3300/0 conn(0x7fd6f80b3690 0x7fd6f80d04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:24.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.595+0000 7fd6fcc00640 1 -- 172.21.15.96:0/225636829 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6f80b3cc0 con 0x7fd6f806f860 2024-05-28T06:04:24.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.595+0000 7fd6f7e00640 1 --2- 172.21.15.96:0/225636829 >> v2:172.21.15.123:3300/0 conn(0x7fd6f80b3690 0x7fd6f80d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:04:24.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.596+0000 7fd6fcc00640 1 --2- 172.21.15.96:0/225636829 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd6f806f860 0x7fd6f80b3150 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fd6f8070120 tx=0x7fd6ec02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a60f2ce1c46ec70 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:24.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.596+0000 7fd6f7400640 1 -- 172.21.15.96:0/225636829 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6ec037070 con 0x7fd6f806f860 2024-05-28T06:04:24.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.597+0000 7fd6f7400640 1 -- 172.21.15.96:0/225636829 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6ec032440 con 0x7fd6f806f860 2024-05-28T06:04:24.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.597+0000 7fd6f7400640 1 -- 172.21.15.96:0/225636829 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6ec032810 con 0x7fd6f806f860 2024-05-28T06:04:24.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.597+0000 7fd6ff600640 1 -- 172.21.15.96:0/225636829 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd6f806f860 msgr2=0x7fd6f80b3150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:24.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.597+0000 7fd6ff600640 1 --2- 172.21.15.96:0/225636829 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd6f806f860 0x7fd6f80b3150 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fd6f8070120 tx=0x7fd6ec02a1e0 comp rx=0 tx=0).stop 2024-05-28T06:04:24.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.597+0000 7fd6ff600640 1 -- 172.21.15.96:0/225636829 shutdown_connections 2024-05-28T06:04:24.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.597+0000 7fd6ff600640 1 --2- 172.21.15.96:0/225636829 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd6f80d09e0 0x7fd6f8160210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:24.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.597+0000 7fd6ff600640 1 --2- 172.21.15.96:0/225636829 >> v2:172.21.15.123:3300/0 conn(0x7fd6f80b3690 0x7fd6f80d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:24.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.597+0000 7fd6ff600640 1 --2- 172.21.15.96:0/225636829 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd6f806f860 0x7fd6f80b3150 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:24.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.597+0000 7fd6ff600640 1 -- 172.21.15.96:0/225636829 >> 172.21.15.96:0/225636829 conn(0x7fd6f806bae0 msgr2=0x7fd6f806bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:24.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.597+0000 7fd6ff600640 1 -- 172.21.15.96:0/225636829 shutdown_connections 2024-05-28T06:04:24.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.597+0000 7fd6ff600640 1 -- 172.21.15.96:0/225636829 wait complete. 2024-05-28T06:04:24.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.599+0000 7fd6ff600640 1 Processor -- start 2024-05-28T06:04:24.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.599+0000 7fd6ff600640 1 -- start start 2024-05-28T06:04:24.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.600+0000 7fd6ff600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd6f806f860 0x7fd6f80c6030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:24.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.600+0000 7fd6ff600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd6f80b3690 0x7fd6f815d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:24.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.600+0000 7fd6ff600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd6f80d09e0 0x7fd6f81f7790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:24.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.600+0000 7fd6ff600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd6f80b4560 con 0x7fd6f806f860 2024-05-28T06:04:24.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.600+0000 7fd6ff600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd6f80b4280 con 0x7fd6f80d09e0 2024-05-28T06:04:24.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.600+0000 7fd6ff600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fd6f80b4840 con 0x7fd6f80b3690 2024-05-28T06:04:24.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.600+0000 7fd6fcc00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd6f806f860 0x7fd6f80c6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:24.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.600+0000 7fd6fcc00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd6f806f860 0x7fd6f80c6030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48278/0 (socket says 172.21.15.96:48278) 2024-05-28T06:04:24.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.600+0000 7fd6fcc00640 1 -- 172.21.15.96:0/677319284 learned_addr learned my addr 172.21.15.96:0/677319284 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:24.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.600+0000 7fd6fcc00640 1 -- 172.21.15.96:0/677319284 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd6f80b3690 msgr2=0x7fd6f815d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:24.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.600+0000 7fd6fcc00640 1 --2- 172.21.15.96:0/677319284 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd6f80b3690 0x7fd6f815d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:24.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.601+0000 7fd6fcc00640 1 -- 172.21.15.96:0/677319284 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd6f80d09e0 msgr2=0x7fd6f81f7790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:24.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.601+0000 7fd6fcc00640 1 --2- 172.21.15.96:0/677319284 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd6f80d09e0 0x7fd6f81f7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:24.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.601+0000 7fd6fcc00640 1 -- 172.21.15.96:0/677319284 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6f80c82b0 con 0x7fd6f806f860 2024-05-28T06:04:24.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.601+0000 7fd6fcc00640 1 --2- 172.21.15.96:0/677319284 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd6f806f860 0x7fd6f80c6030 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fd6ec02a220 tx=0x7fd6ec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:24.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.601+0000 7fd6f4c00640 1 -- 172.21.15.96:0/677319284 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6ec037040 con 0x7fd6f806f860 2024-05-28T06:04:24.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.601+0000 7fd6f4c00640 1 -- 172.21.15.96:0/677319284 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6ec042070 con 0x7fd6f806f860 2024-05-28T06:04:24.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.601+0000 7fd6f4c00640 1 -- 172.21.15.96:0/677319284 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6ec030bb0 con 0x7fd6f806f860 2024-05-28T06:04:24.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.601+0000 7fd6ff600640 1 -- 172.21.15.96:0/677319284 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6f80c6570 con 0x7fd6f806f860 2024-05-28T06:04:24.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.601+0000 7fd6ff600640 1 -- 172.21.15.96:0/677319284 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6f815e300 con 0x7fd6f806f860 2024-05-28T06:04:24.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.602+0000 7fd6ff600640 1 -- 172.21.15.96:0/677319284 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6f80689e0 con 0x7fd6f806f860 2024-05-28T06:04:24.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.603+0000 7fd6f4c00640 1 -- 172.21.15.96:0/677319284 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd6ec047460 con 0x7fd6f806f860 2024-05-28T06:04:24.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.604+0000 7fd6f4c00640 1 --2- 172.21.15.96:0/677319284 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd6d806c8f0 0x7fd6d806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:24.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.604+0000 7fd6f4c00640 1 -- 172.21.15.96:0/677319284 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fd6ec0ac7d0 con 0x7fd6f806f860 2024-05-28T06:04:24.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.605+0000 7fd6f7e00640 1 --2- 172.21.15.96:0/677319284 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd6d806c8f0 0x7fd6d806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:24.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.606+0000 7fd6f7e00640 1 --2- 172.21.15.96:0/677319284 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd6d806c8f0 0x7fd6d806edb0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fd6e4006fd0 tx=0x7fd6e4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:24.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.609+0000 7fd6f4c00640 1 -- 172.21.15.96:0/677319284 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd6ec045360 con 0x7fd6f806f860 2024-05-28T06:04:24.808 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.807+0000 7fd6ff600640 1 -- 172.21.15.96:0/677319284 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd6f806de40 con 0x7fd6f806f860 2024-05-28T06:04:24.811 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.810+0000 7fd6f4c00640 1 -- 172.21.15.96:0/677319284 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16617 (secure 0 0 0) 0x7fd6ec07abf0 con 0x7fd6f806f860 2024-05-28T06:04:24.812 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:04:24.812 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":20,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:04:11.882922+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-28T06:04:08.741938+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-28T06:04:09.301470+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:04:24.814 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.813+0000 7fd6ff600640 1 -- 172.21.15.96:0/677319284 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd6d806c8f0 msgr2=0x7fd6d806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:24.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.813+0000 7fd6ff600640 1 --2- 172.21.15.96:0/677319284 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd6d806c8f0 0x7fd6d806edb0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fd6e4006fd0 tx=0x7fd6e4008040 comp rx=0 tx=0).stop 2024-05-28T06:04:24.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.813+0000 7fd6ff600640 1 -- 172.21.15.96:0/677319284 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd6f806f860 msgr2=0x7fd6f80c6030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:24.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.813+0000 7fd6ff600640 1 --2- 172.21.15.96:0/677319284 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd6f806f860 0x7fd6f80c6030 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fd6ec02a220 tx=0x7fd6ec0046c0 comp rx=0 tx=0).stop 2024-05-28T06:04:24.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.813+0000 7fd6ff600640 1 -- 172.21.15.96:0/677319284 shutdown_connections 2024-05-28T06:04:24.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.813+0000 7fd6ff600640 1 --2- 172.21.15.96:0/677319284 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd6d806c8f0 0x7fd6d806edb0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:24.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.813+0000 7fd6ff600640 1 --2- 172.21.15.96:0/677319284 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd6f80d09e0 0x7fd6f81f7790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:24.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.813+0000 7fd6ff600640 1 --2- 172.21.15.96:0/677319284 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd6f80b3690 0x7fd6f815d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:24.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.813+0000 7fd6ff600640 1 --2- 172.21.15.96:0/677319284 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd6f806f860 0x7fd6f80c6030 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:24.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.813+0000 7fd6ff600640 1 -- 172.21.15.96:0/677319284 >> 172.21.15.96:0/677319284 conn(0x7fd6f806bae0 msgr2=0x7fd6f80cf870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:24.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.813+0000 7fd6ff600640 1 -- 172.21.15.96:0/677319284 shutdown_connections 2024-05-28T06:04:24.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:24.813+0000 7fd6ff600640 1 -- 172.21.15.96:0/677319284 wait complete. 2024-05-28T06:04:24.833 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-28T06:03:59.517102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-28T06:04:08.741938+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-28T06:04:09.301470+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-28T06:04:24.834 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-28T06:04:25.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.037+0000 7f478f800640 1 Processor -- start 2024-05-28T06:04:25.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.038+0000 7f478f800640 1 -- start start 2024-05-28T06:04:25.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.039+0000 7f478f800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f478815b6f0 0x7f478806e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:25.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.039+0000 7f478f800640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47880ade90 con 0x7f47880d0640 2024-05-28T06:04:25.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.039+0000 7f478f800640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4788071e70 con 0x7f47880725e0 2024-05-28T06:04:25.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.039+0000 7f478f800640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f4788071c40 con 0x7f478815b6f0 2024-05-28T06:04:25.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.039+0000 7f4787e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f478815b6f0 0x7f478806e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:25.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.039+0000 7f4787e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f478815b6f0 0x7f478806e350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48288/0 (socket says 172.21.15.96:48288) 2024-05-28T06:04:25.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.039+0000 7f478ce00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f47880d0640 0x7f47880d0a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:35268/0 (socket says 172.21.15.96:35268) 2024-05-28T06:04:25.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.039+0000 7f4787e00640 1 -- 172.21.15.96:0/3155137466 learned_addr learned my addr 172.21.15.96:0/3155137466 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:25.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.040+0000 7f4787e00640 1 -- 172.21.15.96:0/3155137466 >> v1:172.21.15.123:6789/0 conn(0x7f47880725e0 legacy=0x7f47880ae650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:25.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.040+0000 7f4787e00640 1 -- 172.21.15.96:0/3155137466 >> v1:172.21.15.96:6789/0 conn(0x7f47880d0640 legacy=0x7f47880d0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:25.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.040+0000 7f4787e00640 1 -- 172.21.15.96:0/3155137466 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4788072130 con 0x7f478815b6f0 2024-05-28T06:04:25.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.041+0000 7f4787e00640 1 --2- 172.21.15.96:0/3155137466 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f478815b6f0 0x7f478806e350 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f4770009870 tx=0x7f477002a2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a43de1af24627f3f server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:25.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.041+0000 7f4787400640 1 -- 172.21.15.96:0/3155137466 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4770037070 con 0x7f478815b6f0 2024-05-28T06:04:25.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.041+0000 7f4787400640 1 -- 172.21.15.96:0/3155137466 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4770032440 con 0x7f478815b6f0 2024-05-28T06:04:25.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.041+0000 7f4787400640 1 -- 172.21.15.96:0/3155137466 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4770032810 con 0x7f478815b6f0 2024-05-28T06:04:25.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.042+0000 7f478f800640 1 -- 172.21.15.96:0/3155137466 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f478815b6f0 msgr2=0x7f478806e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:25.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.042+0000 7f478f800640 1 --2- 172.21.15.96:0/3155137466 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f478815b6f0 0x7f478806e350 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f4770009870 tx=0x7f477002a2a0 comp rx=0 tx=0).stop 2024-05-28T06:04:25.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.042+0000 7f478f800640 1 -- 172.21.15.96:0/3155137466 shutdown_connections 2024-05-28T06:04:25.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.042+0000 7f478f800640 1 --2- 172.21.15.96:0/3155137466 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f478815b6f0 0x7f478806e350 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:25.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.042+0000 7f478f800640 1 -- 172.21.15.96:0/3155137466 >> 172.21.15.96:0/3155137466 conn(0x7f478806bd20 msgr2=0x7f478806c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:25.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.042+0000 7f478f800640 1 -- 172.21.15.96:0/3155137466 shutdown_connections 2024-05-28T06:04:25.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.042+0000 7f478f800640 1 -- 172.21.15.96:0/3155137466 wait complete. 2024-05-28T06:04:25.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.044+0000 7f478f800640 1 Processor -- start 2024-05-28T06:04:25.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.044+0000 7f478f800640 1 -- start start 2024-05-28T06:04:25.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.045+0000 7f478f800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f47880725e0 0x7f47881f7390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:25.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.045+0000 7f478f800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f47880d0640 0x7f47881e80c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:25.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.045+0000 7f478f800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f478815b6f0 0x7f47881e8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:25.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.045+0000 7f478f800640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f47880aed60 con 0x7f47880725e0 2024-05-28T06:04:25.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.045+0000 7f478f800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f478806e890 con 0x7f478815b6f0 2024-05-28T06:04:25.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.045+0000 7f478f800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f47880aeee0 con 0x7f47880d0640 2024-05-28T06:04:25.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.045+0000 7f478ce00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f47880725e0 0x7f47881f7390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:25.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.045+0000 7f478ce00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f47880725e0 0x7f47881f7390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48302/0 (socket says 172.21.15.96:48302) 2024-05-28T06:04:25.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.045+0000 7f478ce00640 1 -- 172.21.15.96:0/2646711031 learned_addr learned my addr 172.21.15.96:0/2646711031 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:25.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.045+0000 7f478d800640 1 --2- 172.21.15.96:0/2646711031 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f478815b6f0 0x7f47881e8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:25.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.045+0000 7f4787e00640 1 --2- 172.21.15.96:0/2646711031 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f47880d0640 0x7f47881e80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:25.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.045+0000 7f478ce00640 1 -- 172.21.15.96:0/2646711031 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f47880d0640 msgr2=0x7f47881e80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:25.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.045+0000 7f478ce00640 1 --2- 172.21.15.96:0/2646711031 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f47880d0640 0x7f47881e80c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:25.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.045+0000 7f478ce00640 1 -- 172.21.15.96:0/2646711031 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f478815b6f0 msgr2=0x7f47881e8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:25.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.045+0000 7f478ce00640 1 --2- 172.21.15.96:0/2646711031 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f478815b6f0 0x7f47881e8600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:25.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.045+0000 7f478ce00640 1 -- 172.21.15.96:0/2646711031 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47881f78d0 con 0x7f47880725e0 2024-05-28T06:04:25.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.046+0000 7f478ce00640 1 --2- 172.21.15.96:0/2646711031 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f47880725e0 0x7f47881f7390 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f477800c430 tx=0x7f477800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:25.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.046+0000 7f4784c00640 1 -- 172.21.15.96:0/2646711031 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47780074e0 con 0x7f47880725e0 2024-05-28T06:04:25.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.046+0000 7f4784c00640 1 -- 172.21.15.96:0/2646711031 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4778007680 con 0x7f47880725e0 2024-05-28T06:04:25.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.046+0000 7f4784c00640 1 -- 172.21.15.96:0/2646711031 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4778004a00 con 0x7f47880725e0 2024-05-28T06:04:25.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.046+0000 7f478f800640 1 -- 172.21.15.96:0/2646711031 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f478815d820 con 0x7f47880725e0 2024-05-28T06:04:25.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.046+0000 7f478f800640 1 -- 172.21.15.96:0/2646711031 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4788156bf0 con 0x7f47880725e0 2024-05-28T06:04:25.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.047+0000 7f478f800640 1 -- 172.21.15.96:0/2646711031 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47880689e0 con 0x7f47880725e0 2024-05-28T06:04:25.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.048+0000 7f4784c00640 1 -- 172.21.15.96:0/2646711031 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f477801a020 con 0x7f47880725e0 2024-05-28T06:04:25.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.049+0000 7f4784c00640 1 --2- 172.21.15.96:0/2646711031 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f476406c8f0 0x7f476406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:25.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.050+0000 7f4784c00640 1 -- 172.21.15.96:0/2646711031 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f477808b350 con 0x7f47880725e0 2024-05-28T06:04:25.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.050+0000 7f4787e00640 1 --2- 172.21.15.96:0/2646711031 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f476406c8f0 0x7f476406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:25.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.051+0000 7f4787e00640 1 --2- 172.21.15.96:0/2646711031 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f476406c8f0 0x7f476406edb0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f4770034000 tx=0x7f477002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:25.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.054+0000 7f4784c00640 1 -- 172.21.15.96:0/2646711031 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f47780597b0 con 0x7f47880725e0 2024-05-28T06:04:25.254 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.253+0000 7f478f800640 1 -- 172.21.15.96:0/2646711031 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f47880af3d0 con 0x7f47880725e0 2024-05-28T06:04:25.255 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.254+0000 7f4784c00640 1 -- 172.21.15.96:0/2646711031 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) ==== 93+0+10 (secure 0 0 0) 0x7f47780590b0 con 0x7f47880725e0 2024-05-28T06:04:25.256 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 1 2024-05-28T06:04:25.259 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.257+0000 7f478f800640 1 -- 172.21.15.96:0/2646711031 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f476406c8f0 msgr2=0x7f476406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:25.259 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.257+0000 7f478f800640 1 --2- 172.21.15.96:0/2646711031 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f476406c8f0 0x7f476406edb0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f4770034000 tx=0x7f477002e000 comp rx=0 tx=0).stop 2024-05-28T06:04:25.259 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.257+0000 7f478f800640 1 -- 172.21.15.96:0/2646711031 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f47880725e0 msgr2=0x7f47881f7390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:25.259 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.257+0000 7f478f800640 1 --2- 172.21.15.96:0/2646711031 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f47880725e0 0x7f47881f7390 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f477800c430 tx=0x7f477800c8f0 comp rx=0 tx=0).stop 2024-05-28T06:04:25.259 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.257+0000 7f478f800640 1 -- 172.21.15.96:0/2646711031 shutdown_connections 2024-05-28T06:04:25.259 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.257+0000 7f478f800640 1 --2- 172.21.15.96:0/2646711031 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f476406c8f0 0x7f476406edb0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:25.259 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.257+0000 7f478f800640 1 --2- 172.21.15.96:0/2646711031 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f478815b6f0 0x7f47881e8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:25.259 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.257+0000 7f478f800640 1 --2- 172.21.15.96:0/2646711031 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f47880d0640 0x7f47881e80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:25.259 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.257+0000 7f478f800640 1 --2- 172.21.15.96:0/2646711031 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f47880725e0 0x7f47881f7390 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:25.259 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.257+0000 7f478f800640 1 -- 172.21.15.96:0/2646711031 >> 172.21.15.96:0/2646711031 conn(0x7f478806bd20 msgr2=0x7f4788072c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:25.260 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.257+0000 7f478f800640 1 -- 172.21.15.96:0/2646711031 shutdown_connections 2024-05-28T06:04:25.260 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.257+0000 7f478f800640 1 -- 172.21.15.96:0/2646711031 wait complete. 2024-05-28T06:04:25.277 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-28T06:04:25.478 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.477+0000 7f9e42e00640 1 Processor -- start 2024-05-28T06:04:25.479 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.477+0000 7f9e42e00640 1 -- start start 2024-05-28T06:04:25.480 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.479+0000 7f9e42e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9e3c15e850 0x7f9e3c0b77b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:25.480 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.479+0000 7f9e42e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f9e3c0cef30 0x7f9e3c0d1640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:25.481 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.479+0000 7f9e42e00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e3c0b8000 con 0x7f9e3c0d1b80 2024-05-28T06:04:25.481 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.479+0000 7f9e42e00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f9e3c15ec30 con 0x7f9e3c0cef30 2024-05-28T06:04:25.481 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.479+0000 7f9e42e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f9e3c0d1fd0 con 0x7f9e3c15e850 2024-05-28T06:04:25.481 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.479+0000 7f9e3b400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f9e3c0cef30 0x7f9e3c0d1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:25.481 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.479+0000 7f9e3b400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f9e3c0cef30 0x7f9e3c0d1640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48312/0 (socket says 172.21.15.96:48312) 2024-05-28T06:04:25.481 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.479+0000 7f9e3b400640 1 -- 172.21.15.96:0/2089523628 learned_addr learned my addr 172.21.15.96:0/2089523628 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:25.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.480+0000 7f9e3b400640 1 -- 172.21.15.96:0/2089523628 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9e3c15e850 msgr2=0x7f9e3c0b77b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:25.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.480+0000 7f9e3b400640 1 --2- 172.21.15.96:0/2089523628 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9e3c15e850 0x7f9e3c0b77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:25.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.480+0000 7f9e3b400640 1 -- 172.21.15.96:0/2089523628 >> v1:172.21.15.123:6789/0 conn(0x7f9e3c0d1b80 legacy=0x7f9e3c162340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:25.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.480+0000 7f9e3b400640 1 -- 172.21.15.96:0/2089523628 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e3c0b82e0 con 0x7f9e3c0cef30 2024-05-28T06:04:25.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.481+0000 7f9e3b400640 1 --2- 172.21.15.96:0/2089523628 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9e3c0cef30 0x7f9e3c0d1640 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f9e28009870 tx=0x7f9e2802a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9692d960abe4e72 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:25.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.481+0000 7f9e3aa00640 1 -- 172.21.15.96:0/2089523628 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e28037070 con 0x7f9e3c0cef30 2024-05-28T06:04:25.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.481+0000 7f9e3aa00640 1 -- 172.21.15.96:0/2089523628 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e28032440 con 0x7f9e3c0cef30 2024-05-28T06:04:25.483 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.481+0000 7f9e3aa00640 1 -- 172.21.15.96:0/2089523628 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e28032810 con 0x7f9e3c0cef30 2024-05-28T06:04:25.483 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.482+0000 7f9e42e00640 1 -- 172.21.15.96:0/2089523628 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9e3c0cef30 msgr2=0x7f9e3c0d1640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:25.483 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.482+0000 7f9e42e00640 1 --2- 172.21.15.96:0/2089523628 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9e3c0cef30 0x7f9e3c0d1640 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f9e28009870 tx=0x7f9e2802a180 comp rx=0 tx=0).stop 2024-05-28T06:04:25.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.482+0000 7f9e42e00640 1 -- 172.21.15.96:0/2089523628 shutdown_connections 2024-05-28T06:04:25.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.482+0000 7f9e42e00640 1 --2- 172.21.15.96:0/2089523628 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9e3c0cef30 0x7f9e3c0d1640 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:25.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.482+0000 7f9e42e00640 1 --2- 172.21.15.96:0/2089523628 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9e3c15e850 0x7f9e3c0b77b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:25.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.482+0000 7f9e42e00640 1 -- 172.21.15.96:0/2089523628 >> 172.21.15.96:0/2089523628 conn(0x7f9e3c067aa0 msgr2=0x7f9e3c067eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:25.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.482+0000 7f9e42e00640 1 -- 172.21.15.96:0/2089523628 shutdown_connections 2024-05-28T06:04:25.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.482+0000 7f9e42e00640 1 -- 172.21.15.96:0/2089523628 wait complete. 2024-05-28T06:04:25.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.484+0000 7f9e42e00640 1 Processor -- start 2024-05-28T06:04:25.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.484+0000 7f9e42e00640 1 -- start start 2024-05-28T06:04:25.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.484+0000 7f9e42e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9e3c0cef30 0x7f9e3c0cdc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:25.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.485+0000 7f9e42e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9e3c0d1b80 0x7f9e3c1f4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:25.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.485+0000 7f9e42e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9e3c15e850 0x7f9e3c1f5520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:25.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.485+0000 7f9e42e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f9e3c0b7cf0 con 0x7f9e3c15e850 2024-05-28T06:04:25.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.485+0000 7f9e42e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f9e3c0b88a0 con 0x7f9e3c0cef30 2024-05-28T06:04:25.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.485+0000 7f9e42e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f9e3c0ba0b0 con 0x7f9e3c0d1b80 2024-05-28T06:04:25.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.485+0000 7f9e40e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9e3c15e850 0x7f9e3c1f5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:25.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.485+0000 7f9e40e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9e3c15e850 0x7f9e3c1f5520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48326/0 (socket says 172.21.15.96:48326) 2024-05-28T06:04:25.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.485+0000 7f9e40e00640 1 -- 172.21.15.96:0/1793137945 learned_addr learned my addr 172.21.15.96:0/1793137945 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:25.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.485+0000 7f9e3be00640 1 --2- 172.21.15.96:0/1793137945 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9e3c0cef30 0x7f9e3c0cdc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:25.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.485+0000 7f9e3b400640 1 --2- 172.21.15.96:0/1793137945 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9e3c0d1b80 0x7f9e3c1f4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:25.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.485+0000 7f9e40e00640 1 -- 172.21.15.96:0/1793137945 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9e3c0d1b80 msgr2=0x7f9e3c1f4de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:25.489 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.485+0000 7f9e40e00640 1 --2- 172.21.15.96:0/1793137945 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9e3c0d1b80 0x7f9e3c1f4de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:25.489 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.485+0000 7f9e40e00640 1 -- 172.21.15.96:0/1793137945 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9e3c0cef30 msgr2=0x7f9e3c0cdc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:25.489 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.485+0000 7f9e40e00640 1 --2- 172.21.15.96:0/1793137945 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9e3c0cef30 0x7f9e3c0cdc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:25.489 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.485+0000 7f9e40e00640 1 -- 172.21.15.96:0/1793137945 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e3c1f5320 con 0x7f9e3c15e850 2024-05-28T06:04:25.489 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.485+0000 7f9e3b400640 1 --2- 172.21.15.96:0/1793137945 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9e3c0d1b80 0x7f9e3c1f4de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:04:25.489 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.485+0000 7f9e3be00640 1 --2- 172.21.15.96:0/1793137945 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9e3c0cef30 0x7f9e3c0cdc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:04:25.489 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.486+0000 7f9e40e00640 1 --2- 172.21.15.96:0/1793137945 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9e3c15e850 0x7f9e3c1f5520 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f9e3400c7b0 tx=0x7f9e3400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:25.489 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.486+0000 7f9e33e00640 1 -- 172.21.15.96:0/1793137945 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e34017070 con 0x7f9e3c15e850 2024-05-28T06:04:25.489 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.486+0000 7f9e33e00640 1 -- 172.21.15.96:0/1793137945 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e34003da0 con 0x7f9e3c15e850 2024-05-28T06:04:25.489 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.486+0000 7f9e33e00640 1 -- 172.21.15.96:0/1793137945 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e340052b0 con 0x7f9e3c15e850 2024-05-28T06:04:25.490 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.486+0000 7f9e42e00640 1 -- 172.21.15.96:0/1793137945 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e3c153a10 con 0x7f9e3c15e850 2024-05-28T06:04:25.490 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.486+0000 7f9e42e00640 1 -- 172.21.15.96:0/1793137945 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e3c153fd0 con 0x7f9e3c15e850 2024-05-28T06:04:25.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.487+0000 7f9e42e00640 1 -- 172.21.15.96:0/1793137945 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e3c0689e0 con 0x7f9e3c15e850 2024-05-28T06:04:25.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.488+0000 7f9e33e00640 1 -- 172.21.15.96:0/1793137945 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9e3401a070 con 0x7f9e3c15e850 2024-05-28T06:04:25.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.489+0000 7f9e33e00640 1 --2- 172.21.15.96:0/1793137945 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9e1c06c920 0x7f9e1c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:25.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.490+0000 7f9e33e00640 1 -- 172.21.15.96:0/1793137945 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f9e3408b150 con 0x7f9e3c15e850 2024-05-28T06:04:25.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.490+0000 7f9e3be00640 1 --2- 172.21.15.96:0/1793137945 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9e1c06c920 0x7f9e1c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:25.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.491+0000 7f9e3be00640 1 --2- 172.21.15.96:0/1793137945 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9e1c06c920 0x7f9e1c06ede0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f9e2c006fd0 tx=0x7f9e2c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:25.495 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.494+0000 7f9e33e00640 1 -- 172.21.15.96:0/1793137945 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9e3c1f5320 con 0x7f9e3c15e850 2024-05-28T06:04:25.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.696+0000 7f9e42e00640 1 -- 172.21.15.96:0/1793137945 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f9e3c068020 con 0x7f9e3c15e850 2024-05-28T06:04:25.698 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.696+0000 7f9e33e00640 1 -- 172.21.15.96:0/1793137945 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) ==== 104+0+11 (secure 0 0 0) 0x7f9e34059570 con 0x7f9e3c15e850 2024-05-28T06:04:25.698 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 32 2024-05-28T06:04:25.700 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.699+0000 7f9e42e00640 1 -- 172.21.15.96:0/1793137945 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9e1c06c920 msgr2=0x7f9e1c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:25.700 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.699+0000 7f9e42e00640 1 --2- 172.21.15.96:0/1793137945 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9e1c06c920 0x7f9e1c06ede0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f9e2c006fd0 tx=0x7f9e2c008040 comp rx=0 tx=0).stop 2024-05-28T06:04:25.700 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.699+0000 7f9e42e00640 1 -- 172.21.15.96:0/1793137945 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9e3c15e850 msgr2=0x7f9e3c1f5520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:25.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.699+0000 7f9e42e00640 1 --2- 172.21.15.96:0/1793137945 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9e3c15e850 0x7f9e3c1f5520 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f9e3400c7b0 tx=0x7f9e3400cc70 comp rx=0 tx=0).stop 2024-05-28T06:04:25.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.699+0000 7f9e42e00640 1 -- 172.21.15.96:0/1793137945 shutdown_connections 2024-05-28T06:04:25.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.699+0000 7f9e42e00640 1 --2- 172.21.15.96:0/1793137945 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9e1c06c920 0x7f9e1c06ede0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:25.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.699+0000 7f9e42e00640 1 --2- 172.21.15.96:0/1793137945 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9e3c15e850 0x7f9e3c1f5520 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:25.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.699+0000 7f9e42e00640 1 --2- 172.21.15.96:0/1793137945 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9e3c0d1b80 0x7f9e3c1f4de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:25.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.699+0000 7f9e42e00640 1 --2- 172.21.15.96:0/1793137945 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9e3c0cef30 0x7f9e3c0cdc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:25.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.699+0000 7f9e42e00640 1 -- 172.21.15.96:0/1793137945 >> 172.21.15.96:0/1793137945 conn(0x7f9e3c067aa0 msgr2=0x7f9e3c0b5630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:25.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.699+0000 7f9e42e00640 1 -- 172.21.15.96:0/1793137945 shutdown_connections 2024-05-28T06:04:25.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.699+0000 7f9e42e00640 1 -- 172.21.15.96:0/1793137945 wait complete. 2024-05-28T06:04:25.720 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-28T06:04:25.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.925+0000 7f445b000640 1 Processor -- start 2024-05-28T06:04:25.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.925+0000 7f445b000640 1 -- start start 2024-05-28T06:04:25.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.926+0000 7f445b000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f44540d0640 0x7f445415d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:25.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.926+0000 7f445b000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f44540d0a20 0x7f44540cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:25.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.926+0000 7f445b000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f44540b32c0 0x7f44540b3730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:25.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.926+0000 7f445b000640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f44540adbb0 con 0x7f44540d0640 2024-05-28T06:04:25.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.926+0000 7f445b000640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f445415db10 con 0x7f44540d0a20 2024-05-28T06:04:25.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.926+0000 7f445b000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f445415cb10 con 0x7f44540b32c0 2024-05-28T06:04:25.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.927+0000 7f4453e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f44540d0640 0x7f445415d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:25.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.927+0000 7f4453e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f44540d0640 0x7f445415d5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48334/0 (socket says 172.21.15.96:48334) 2024-05-28T06:04:25.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.927+0000 7f4453e00640 1 -- 172.21.15.96:0/2117814552 learned_addr learned my addr 172.21.15.96:0/2117814552 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:25.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.927+0000 7f4453400640 1 --2- 172.21.15.96:0/2117814552 >> v2:172.21.15.123:3300/0 conn(0x7f44540d0a20 0x7f44540cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:25.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.927+0000 7f4459000640 1 --2- 172.21.15.96:0/2117814552 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f44540b32c0 0x7f44540b3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:25.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.927+0000 7f4453e00640 1 -- 172.21.15.96:0/2117814552 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f44540b32c0 msgr2=0x7f44540b3730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:25.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.927+0000 7f4453e00640 1 --2- 172.21.15.96:0/2117814552 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f44540b32c0 0x7f44540b3730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:25.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.927+0000 7f4453e00640 1 -- 172.21.15.96:0/2117814552 >> v2:172.21.15.123:3300/0 conn(0x7f44540d0a20 msgr2=0x7f44540cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:25.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.927+0000 7f4453e00640 1 --2- 172.21.15.96:0/2117814552 >> v2:172.21.15.123:3300/0 conn(0x7f44540d0a20 0x7f44540cf950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:25.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.927+0000 7f4453e00640 1 -- 172.21.15.96:0/2117814552 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44540ad5f0 con 0x7f44540d0640 2024-05-28T06:04:25.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.927+0000 7f4459000640 1 --2- 172.21.15.96:0/2117814552 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f44540b32c0 0x7f44540b3730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:04:25.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.928+0000 7f4453e00640 1 --2- 172.21.15.96:0/2117814552 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f44540d0640 0x7f445415d5d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f4454074120 tx=0x7f444402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99f20b0c5ba10091 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:25.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.929+0000 7f4452a00640 1 -- 172.21.15.96:0/2117814552 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4444037070 con 0x7f44540d0640 2024-05-28T06:04:25.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.929+0000 7f4452a00640 1 -- 172.21.15.96:0/2117814552 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4444032440 con 0x7f44540d0640 2024-05-28T06:04:25.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.929+0000 7f4452a00640 1 -- 172.21.15.96:0/2117814552 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4444032810 con 0x7f44540d0640 2024-05-28T06:04:25.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.929+0000 7f445b000640 1 -- 172.21.15.96:0/2117814552 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f44540d0640 msgr2=0x7f445415d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:25.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.929+0000 7f445b000640 1 --2- 172.21.15.96:0/2117814552 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f44540d0640 0x7f445415d5d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f4454074120 tx=0x7f444402a1e0 comp rx=0 tx=0).stop 2024-05-28T06:04:25.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.930+0000 7f445b000640 1 -- 172.21.15.96:0/2117814552 shutdown_connections 2024-05-28T06:04:25.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.930+0000 7f445b000640 1 --2- 172.21.15.96:0/2117814552 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f44540b32c0 0x7f44540b3730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:25.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.930+0000 7f445b000640 1 --2- 172.21.15.96:0/2117814552 >> v2:172.21.15.123:3300/0 conn(0x7f44540d0a20 0x7f44540cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:25.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.930+0000 7f445b000640 1 --2- 172.21.15.96:0/2117814552 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f44540d0640 0x7f445415d5d0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:25.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.930+0000 7f445b000640 1 -- 172.21.15.96:0/2117814552 >> 172.21.15.96:0/2117814552 conn(0x7f445406bd20 msgr2=0x7f445406c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:25.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.930+0000 7f445b000640 1 -- 172.21.15.96:0/2117814552 shutdown_connections 2024-05-28T06:04:25.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.930+0000 7f445b000640 1 -- 172.21.15.96:0/2117814552 wait complete. 2024-05-28T06:04:25.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.932+0000 7f445b000640 1 Processor -- start 2024-05-28T06:04:25.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.932+0000 7f445b000640 1 -- start start 2024-05-28T06:04:25.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.933+0000 7f445b000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f44540b32c0 0x7f44541f3080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:25.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.933+0000 7f445b000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f44540d0640 0x7f44541f21c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:25.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.933+0000 7f445b000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44540d0a20 0x7f4454142780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:25.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.933+0000 7f445b000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f445415d0b0 con 0x7f44540d0640 2024-05-28T06:04:25.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.933+0000 7f445b000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f445415cdd0 con 0x7f44540d0a20 2024-05-28T06:04:25.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.933+0000 7f445b000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f44540ad2a0 con 0x7f44540b32c0 2024-05-28T06:04:25.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.933+0000 7f4453400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f44540d0640 0x7f44541f21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:25.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.933+0000 7f4453400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f44540d0640 0x7f44541f21c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48348/0 (socket says 172.21.15.96:48348) 2024-05-28T06:04:25.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.933+0000 7f4453400640 1 -- 172.21.15.96:0/815309398 learned_addr learned my addr 172.21.15.96:0/815309398 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:25.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.933+0000 7f4453e00640 1 --2- 172.21.15.96:0/815309398 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f44540b32c0 0x7f44541f3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:25.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.933+0000 7f4453400640 1 -- 172.21.15.96:0/815309398 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f44540b32c0 msgr2=0x7f44541f3080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:25.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.933+0000 7f4453400640 1 --2- 172.21.15.96:0/815309398 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f44540b32c0 0x7f44541f3080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:25.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.933+0000 7f4459000640 1 --2- 172.21.15.96:0/815309398 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44540d0a20 0x7f4454142780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:25.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.933+0000 7f4453400640 1 -- 172.21.15.96:0/815309398 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44540d0a20 msgr2=0x7f4454142780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:25.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.933+0000 7f4453400640 1 --2- 172.21.15.96:0/815309398 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44540d0a20 0x7f4454142780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:25.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.933+0000 7f4453400640 1 -- 172.21.15.96:0/815309398 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44541416e0 con 0x7f44540d0640 2024-05-28T06:04:25.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.934+0000 7f4453400640 1 --2- 172.21.15.96:0/815309398 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f44540d0640 0x7f44541f21c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f444000d5f0 tx=0x7f444000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:25.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.934+0000 7f444be00640 1 -- 172.21.15.96:0/815309398 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44400125f0 con 0x7f44540d0640 2024-05-28T06:04:25.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.934+0000 7f444be00640 1 -- 172.21.15.96:0/815309398 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4440019070 con 0x7f44540d0640 2024-05-28T06:04:25.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.934+0000 7f445b000640 1 -- 172.21.15.96:0/815309398 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44541419c0 con 0x7f44540d0640 2024-05-28T06:04:25.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.934+0000 7f444be00640 1 -- 172.21.15.96:0/815309398 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44400129d0 con 0x7f44540d0640 2024-05-28T06:04:25.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.934+0000 7f445b000640 1 -- 172.21.15.96:0/815309398 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4454141f80 con 0x7f44540d0640 2024-05-28T06:04:25.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.935+0000 7f445b000640 1 -- 172.21.15.96:0/815309398 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44540689e0 con 0x7f44540d0640 2024-05-28T06:04:25.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.936+0000 7f444be00640 1 -- 172.21.15.96:0/815309398 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4440012b70 con 0x7f44540d0640 2024-05-28T06:04:25.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.937+0000 7f444be00640 1 --2- 172.21.15.96:0/815309398 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f443406c8f0 0x7f443406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:25.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.937+0000 7f444be00640 1 -- 172.21.15.96:0/815309398 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f444008bff0 con 0x7f44540d0640 2024-05-28T06:04:25.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.938+0000 7f4453e00640 1 --2- 172.21.15.96:0/815309398 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f443406c8f0 0x7f443406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:25.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.939+0000 7f4453e00640 1 --2- 172.21.15.96:0/815309398 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f443406c8f0 0x7f443406edb0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f4444034000 tx=0x7f444402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:25.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:25.941+0000 7f444be00640 1 -- 172.21.15.96:0/815309398 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f444005a380 con 0x7f44540d0640 2024-05-28T06:04:26.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.135+0000 7f445b000640 1 -- 172.21.15.96:0/815309398 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f4454071ef0 con 0x7f44540d0640 2024-05-28T06:04:26.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.136+0000 7f444be00640 1 -- 172.21.15.96:0/815309398 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) ==== 100+0+11 (secure 0 0 0) 0x7f4440059c80 con 0x7f44540d0640 2024-05-28T06:04:26.138 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 64 2024-05-28T06:04:26.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.138+0000 7f445b000640 1 -- 172.21.15.96:0/815309398 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f443406c8f0 msgr2=0x7f443406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:26.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.138+0000 7f445b000640 1 --2- 172.21.15.96:0/815309398 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f443406c8f0 0x7f443406edb0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f4444034000 tx=0x7f444402e000 comp rx=0 tx=0).stop 2024-05-28T06:04:26.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.138+0000 7f445b000640 1 -- 172.21.15.96:0/815309398 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f44540d0640 msgr2=0x7f44541f21c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:26.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.138+0000 7f445b000640 1 --2- 172.21.15.96:0/815309398 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f44540d0640 0x7f44541f21c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f444000d5f0 tx=0x7f444000dab0 comp rx=0 tx=0).stop 2024-05-28T06:04:26.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.139+0000 7f445b000640 1 -- 172.21.15.96:0/815309398 shutdown_connections 2024-05-28T06:04:26.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.139+0000 7f445b000640 1 --2- 172.21.15.96:0/815309398 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f443406c8f0 0x7f443406edb0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:26.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.139+0000 7f445b000640 1 --2- 172.21.15.96:0/815309398 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44540d0a20 0x7f4454142780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:26.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.139+0000 7f445b000640 1 --2- 172.21.15.96:0/815309398 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f44540d0640 0x7f44541f21c0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:26.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.139+0000 7f445b000640 1 --2- 172.21.15.96:0/815309398 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f44540b32c0 0x7f44541f3080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:26.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.139+0000 7f445b000640 1 -- 172.21.15.96:0/815309398 >> 172.21.15.96:0/815309398 conn(0x7f445406bd20 msgr2=0x7f44540b28b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:26.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.139+0000 7f445b000640 1 -- 172.21.15.96:0/815309398 shutdown_connections 2024-05-28T06:04:26.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.139+0000 7f445b000640 1 -- 172.21.15.96:0/815309398 wait complete. 2024-05-28T06:04:26.158 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-28T06:04:26.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.339+0000 7f1207e00640 1 Processor -- start 2024-05-28T06:04:26.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.339+0000 7f1207e00640 1 -- start start 2024-05-28T06:04:26.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.340+0000 7f1207e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f120006f6e0 0x7f12000cd4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:26.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.340+0000 7f1207e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f1200162e70 0x7f12000ce230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:26.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.341+0000 7f1207e00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12000b4350 con 0x7f12000d04a0 2024-05-28T06:04:26.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.341+0000 7f1207e00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f12000b3ec0 con 0x7f1200162e70 2024-05-28T06:04:26.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.341+0000 7f1207e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f12000b32d0 con 0x7f120006f6e0 2024-05-28T06:04:26.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.341+0000 7f1205400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f12000d04a0 0x7f12000d0880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:35274/0 (socket says 172.21.15.96:35274) 2024-05-28T06:04:26.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.341+0000 7f1205400640 1 -- 172.21.15.96:0/2673767213 learned_addr learned my addr 172.21.15.96:0/2673767213 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:04:26.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.341+0000 7f1205e00640 1 --2- 172.21.15.96:0/2673767213 >> v2:172.21.15.123:3300/0 conn(0x7f1200162e70 0x7f12000ce230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:26.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.341+0000 7f1204a00640 1 --2- 172.21.15.96:0/2673767213 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f120006f6e0 0x7f12000cd4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:26.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.341+0000 7f11ffe00640 1 -- 172.21.15.96:0/2673767213 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2577059692 0 0) 0x7f12000b4350 con 0x7f12000d04a0 2024-05-28T06:04:26.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.341+0000 7f11ffe00640 1 -- 172.21.15.96:0/2673767213 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f11e4003520 con 0x7f12000d04a0 2024-05-28T06:04:26.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.342+0000 7f11ffe00640 1 -- 172.21.15.96:0/2673767213 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2011984514 0 0) 0x7f11e4003520 con 0x7f12000d04a0 2024-05-28T06:04:26.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.342+0000 7f11ffe00640 1 -- 172.21.15.96:0/2673767213 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f120006f6e0 msgr2=0x7f12000cd4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:26.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.342+0000 7f11ffe00640 1 --2- 172.21.15.96:0/2673767213 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f120006f6e0 0x7f12000cd4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:26.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.342+0000 7f11ffe00640 1 -- 172.21.15.96:0/2673767213 >> v2:172.21.15.123:3300/0 conn(0x7f1200162e70 msgr2=0x7f12000ce230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:26.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.342+0000 7f11ffe00640 1 --2- 172.21.15.96:0/2673767213 >> v2:172.21.15.123:3300/0 conn(0x7f1200162e70 0x7f12000ce230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:26.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.342+0000 7f11ffe00640 1 -- 172.21.15.96:0/2673767213 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12000b3590 con 0x7f12000d04a0 2024-05-28T06:04:26.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.342+0000 7f11ffe00640 1 -- 172.21.15.96:0/2673767213 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f11f0001670 con 0x7f12000d04a0 2024-05-28T06:04:26.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.342+0000 7f11ffe00640 1 -- 172.21.15.96:0/2673767213 >> v1:172.21.15.96:6789/0 conn(0x7f12000d04a0 legacy=0x7f12000d0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:26.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.342+0000 7f11ffe00640 1 --2- 172.21.15.96:0/2673767213 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11e40039b0 0x7f11e4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:26.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.342+0000 7f11ffe00640 1 --2- 172.21.15.96:0/2673767213 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f120006f6e0 0x7f11e4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:26.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.342+0000 7f11ffe00640 1 --2- 172.21.15.96:0/2673767213 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f11e4004d40 0x7f11e40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:26.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.342+0000 7f11ffe00640 1 -- 172.21.15.96:0/2673767213 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f12000b3ec0 con 0x7f11e40039b0 2024-05-28T06:04:26.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.342+0000 7f11ffe00640 1 -- 172.21.15.96:0/2673767213 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f12000b32d0 con 0x7f11e4004d40 2024-05-28T06:04:26.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.342+0000 7f11ffe00640 1 -- 172.21.15.96:0/2673767213 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f11e400b9c0 con 0x7f120006f6e0 2024-05-28T06:04:26.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.342+0000 7f1205400640 1 --2- 172.21.15.96:0/2673767213 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11e40039b0 0x7f11e4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:26.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.342+0000 7f1204a00640 1 --2- 172.21.15.96:0/2673767213 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f120006f6e0 0x7f11e4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:26.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.342+0000 7f1205400640 1 -- 172.21.15.96:0/2673767213 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f120006f6e0 msgr2=0x7f11e4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:26.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.342+0000 7f1205400640 1 --2- 172.21.15.96:0/2673767213 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f120006f6e0 0x7f11e4004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:26.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.342+0000 7f1205400640 1 -- 172.21.15.96:0/2673767213 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f11e4004d40 msgr2=0x7f11e40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:26.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.342+0000 7f1205400640 1 --2- 172.21.15.96:0/2673767213 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f11e4004d40 0x7f11e40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:26.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.342+0000 7f1205400640 1 -- 172.21.15.96:0/2673767213 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11e4003520 con 0x7f11e40039b0 2024-05-28T06:04:26.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.343+0000 7f1205400640 1 --2- 172.21.15.96:0/2673767213 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11e40039b0 0x7f11e4003dd0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f11f000c210 tx=0x7f11e4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e8392c9662754c3 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:26.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.344+0000 7f11ffe00640 1 -- 172.21.15.96:0/2673767213 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11f0007d50 con 0x7f11e40039b0 2024-05-28T06:04:26.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.344+0000 7f11ffe00640 1 -- 172.21.15.96:0/2673767213 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11f0005af0 con 0x7f11e40039b0 2024-05-28T06:04:26.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.344+0000 7f11ffe00640 1 -- 172.21.15.96:0/2673767213 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11f0007070 con 0x7f11e40039b0 2024-05-28T06:04:26.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.344+0000 7f1207e00640 1 -- 172.21.15.96:0/2673767213 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11e40039b0 msgr2=0x7f11e4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:26.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.344+0000 7f1207e00640 1 --2- 172.21.15.96:0/2673767213 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11e40039b0 0x7f11e4003dd0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f11f000c210 tx=0x7f11e4005830 comp rx=0 tx=0).stop 2024-05-28T06:04:26.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.344+0000 7f1205e00640 1 -- 172.21.15.96:0/2673767213 reap_dead start 2024-05-28T06:04:26.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.344+0000 7f1207e00640 1 -- 172.21.15.96:0/2673767213 shutdown_connections 2024-05-28T06:04:26.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.344+0000 7f1207e00640 1 -- 172.21.15.96:0/2673767213 >> 172.21.15.96:0/2673767213 conn(0x7f120006bae0 msgr2=0x7f120006bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:26.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.344+0000 7f1207e00640 1 -- 172.21.15.96:0/2673767213 shutdown_connections 2024-05-28T06:04:26.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.344+0000 7f1207e00640 1 -- 172.21.15.96:0/2673767213 wait complete. 2024-05-28T06:04:26.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.346+0000 7f1207e00640 1 Processor -- start 2024-05-28T06:04:26.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.346+0000 7f1207e00640 1 -- start start 2024-05-28T06:04:26.350 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.347+0000 7f1207e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f11e40039b0 0x7f1200163250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:26.350 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.347+0000 7f1207e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f12001652d0 0x7f12000c5890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:26.350 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.347+0000 7f1207e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1200152cb0 0x7f1200153130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:26.350 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.347+0000 7f1207e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f12000cd4f0 con 0x7f12001652d0 2024-05-28T06:04:26.350 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.347+0000 7f1207e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f12000b3b50 con 0x7f1200152cb0 2024-05-28T06:04:26.350 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.347+0000 7f1207e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f12000cd670 con 0x7f11e40039b0 2024-05-28T06:04:26.351 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.348+0000 7f1205400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f11e40039b0 0x7f1200163250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:26.351 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.348+0000 7f1204a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f12001652d0 0x7f12000c5890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:26.351 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.348+0000 7f1205400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f11e40039b0 0x7f1200163250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.96:60190/0 (socket says 172.21.15.96:60190) 2024-05-28T06:04:26.351 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.348+0000 7f1205400640 1 -- 172.21.15.96:0/1926996817 learned_addr learned my addr 172.21.15.96:0/1926996817 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:26.351 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.348+0000 7f1205e00640 1 --2- 172.21.15.96:0/1926996817 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1200152cb0 0x7f1200153130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:26.351 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.348+0000 7f1204a00640 1 -- 172.21.15.96:0/1926996817 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f11e40039b0 msgr2=0x7f1200163250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:26.351 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.348+0000 7f1204a00640 1 --2- 172.21.15.96:0/1926996817 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f11e40039b0 0x7f1200163250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:26.351 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.348+0000 7f1204a00640 1 -- 172.21.15.96:0/1926996817 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1200152cb0 msgr2=0x7f1200153130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:26.351 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.348+0000 7f1204a00640 1 --2- 172.21.15.96:0/1926996817 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1200152cb0 0x7f1200153130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:26.351 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.348+0000 7f1204a00640 1 -- 172.21.15.96:0/1926996817 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1200163a80 con 0x7f12001652d0 2024-05-28T06:04:26.351 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.348+0000 7f1204a00640 1 --2- 172.21.15.96:0/1926996817 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f12001652d0 0x7f12000c5890 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f11ec010860 tx=0x7f11ec010d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:26.351 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.349+0000 7f11fd600640 1 -- 172.21.15.96:0/1926996817 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11ec00f040 con 0x7f12001652d0 2024-05-28T06:04:26.351 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.349+0000 7f11fd600640 1 -- 172.21.15.96:0/1926996817 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11ec004220 con 0x7f12001652d0 2024-05-28T06:04:26.352 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.349+0000 7f11fd600640 1 -- 172.21.15.96:0/1926996817 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11ec00e900 con 0x7f12001652d0 2024-05-28T06:04:26.352 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.349+0000 7f1207e00640 1 -- 172.21.15.96:0/1926996817 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1200163d60 con 0x7f12001652d0 2024-05-28T06:04:26.352 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.349+0000 7f1207e00640 1 -- 172.21.15.96:0/1926996817 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1200164320 con 0x7f12001652d0 2024-05-28T06:04:26.352 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.350+0000 7f1207e00640 1 -- 172.21.15.96:0/1926996817 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12000689e0 con 0x7f12001652d0 2024-05-28T06:04:26.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.351+0000 7f11fd600640 1 -- 172.21.15.96:0/1926996817 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f11ec00eaa0 con 0x7f12001652d0 2024-05-28T06:04:26.354 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.351+0000 7f11fd600640 1 --2- 172.21.15.96:0/1926996817 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f11e006c8f0 0x7f11e006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:26.354 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.352+0000 7f11fd600640 1 -- 172.21.15.96:0/1926996817 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f11ec08bdd0 con 0x7f12001652d0 2024-05-28T06:04:26.354 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.352+0000 7f1205400640 1 --2- 172.21.15.96:0/1926996817 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f11e006c8f0 0x7f11e006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:26.354 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.353+0000 7f1205400640 1 --2- 172.21.15.96:0/1926996817 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f11e006c8f0 0x7f11e006edb0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f11f0000fd0 tx=0x7f11f0009030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:26.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.356+0000 7f11fd600640 1 -- 172.21.15.96:0/1926996817 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f11ec018030 con 0x7f12001652d0 2024-05-28T06:04:26.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.595+0000 7f1207e00640 1 -- 172.21.15.96:0/1926996817 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f12000af5d0 con 0x7f12001652d0 2024-05-28T06:04:26.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.596+0000 7f11fd600640 1 -- 172.21.15.96:0/1926996817 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5178 (secure 0 0 0) 0x7f11ec059ba0 con 0x7f12001652d0 2024-05-28T06:04:26.598 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:04:26.598 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/3349661842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":3349661842},{"type":"v1","addr":"172.21.15.96:6833","nonce":3349661842}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4270,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/3383264102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":3383264102},{"type":"v1","addr":"172.21.15.96:6835","nonce":3383264102}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4279,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/3068402544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":3068402544},{"type":"v1","addr":"172.21.15.123:6837","nonce":3068402544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-28T06:04:10.319738+0000","modified":"2024-05-28T06:04:13.704080+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4281},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4281":{"gid":4281,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.123:6835/605814935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":605814935},{"type":"v1","addr":"172.21.15.123:6835","nonce":605814935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4281,"qdb_cluster":[4281]},"id":1}]} 2024-05-28T06:04:26.598 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 7 2024-05-28T06:04:26.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.599+0000 7f1207e00640 1 -- 172.21.15.96:0/1926996817 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f11e006c8f0 msgr2=0x7f11e006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:26.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.599+0000 7f1207e00640 1 --2- 172.21.15.96:0/1926996817 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f11e006c8f0 0x7f11e006edb0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f11f0000fd0 tx=0x7f11f0009030 comp rx=0 tx=0).stop 2024-05-28T06:04:26.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.599+0000 7f1207e00640 1 -- 172.21.15.96:0/1926996817 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f12001652d0 msgr2=0x7f12000c5890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:26.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.599+0000 7f1207e00640 1 --2- 172.21.15.96:0/1926996817 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f12001652d0 0x7f12000c5890 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f11ec010860 tx=0x7f11ec010d20 comp rx=0 tx=0).stop 2024-05-28T06:04:26.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.599+0000 7f1207e00640 1 -- 172.21.15.96:0/1926996817 shutdown_connections 2024-05-28T06:04:26.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.599+0000 7f1207e00640 1 --2- 172.21.15.96:0/1926996817 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f11e006c8f0 0x7f11e006edb0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:26.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.599+0000 7f1207e00640 1 --2- 172.21.15.96:0/1926996817 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1200152cb0 0x7f1200153130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:26.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.599+0000 7f1207e00640 1 --2- 172.21.15.96:0/1926996817 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f12001652d0 0x7f12000c5890 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:26.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.599+0000 7f1207e00640 1 --2- 172.21.15.96:0/1926996817 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f11e40039b0 0x7f1200163250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:26.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.599+0000 7f1207e00640 1 -- 172.21.15.96:0/1926996817 >> 172.21.15.96:0/1926996817 conn(0x7f120006bae0 msgr2=0x7f12000c3430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:26.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.600+0000 7f1207e00640 1 -- 172.21.15.96:0/1926996817 shutdown_connections 2024-05-28T06:04:26.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.600+0000 7f1207e00640 1 -- 172.21.15.96:0/1926996817 wait complete. 2024-05-28T06:04:26.619 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:04:26.822 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.820+0000 7ffa24200640 1 Processor -- start 2024-05-28T06:04:26.822 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.821+0000 7ffa24200640 1 -- start start 2024-05-28T06:04:26.823 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.822+0000 7ffa24200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffa1c0d24d0 0x7ffa1c0d52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:26.823 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.822+0000 7ffa24200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa1c0d4320 con 0x7ffa1c0d1650 2024-05-28T06:04:26.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.822+0000 7ffa24200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa1c0d2b70 con 0x7ffa1c15e9d0 2024-05-28T06:04:26.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.822+0000 7ffa24200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ffa1c15edb0 con 0x7ffa1c0d24d0 2024-05-28T06:04:26.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.822+0000 7ffa22200640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7ffa1c0d1650 0x7ffa1c0d1aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:35286/0 (socket says 172.21.15.96:35286) 2024-05-28T06:04:26.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.822+0000 7ffa22200640 1 -- 172.21.15.96:0/1146915142 learned_addr learned my addr 172.21.15.96:0/1146915142 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:04:26.825 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.823+0000 7ffa20e00640 1 --2- 172.21.15.96:0/1146915142 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffa1c0d24d0 0x7ffa1c0d52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:26.825 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.823+0000 7ffa1be00640 1 -- 172.21.15.96:0/1146915142 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 851684900 0 0) 0x7ffa1c0d4320 con 0x7ffa1c0d1650 2024-05-28T06:04:26.825 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.823+0000 7ffa1be00640 1 -- 172.21.15.96:0/1146915142 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffa04003520 con 0x7ffa1c0d1650 2024-05-28T06:04:26.826 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.823+0000 7ffa1be00640 1 -- 172.21.15.96:0/1146915142 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 819362728 0 0) 0x7ffa04003520 con 0x7ffa1c0d1650 2024-05-28T06:04:26.826 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.823+0000 7ffa1be00640 1 -- 172.21.15.96:0/1146915142 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffa1c0d24d0 msgr2=0x7ffa1c0d52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:26.826 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.823+0000 7ffa1be00640 1 --2- 172.21.15.96:0/1146915142 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffa1c0d24d0 0x7ffa1c0d52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:26.826 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.823+0000 7ffa1be00640 1 -- 172.21.15.96:0/1146915142 >> v1:172.21.15.123:6789/0 conn(0x7ffa1c15e9d0 legacy=0x7ffa1c0d3c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:26.826 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.823+0000 7ffa1be00640 1 -- 172.21.15.96:0/1146915142 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa1c0d2e50 con 0x7ffa1c0d1650 2024-05-28T06:04:26.826 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.823+0000 7ffa1be00640 1 -- 172.21.15.96:0/1146915142 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7ffa140019f0 con 0x7ffa1c0d1650 2024-05-28T06:04:26.826 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.823+0000 7ffa1be00640 1 -- 172.21.15.96:0/1146915142 >> v1:172.21.15.96:6789/0 conn(0x7ffa1c0d1650 legacy=0x7ffa1c0d1aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:26.826 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.823+0000 7ffa1be00640 1 --2- 172.21.15.96:0/1146915142 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa040039e0 0x7ffa04003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:26.826 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.823+0000 7ffa1be00640 1 --2- 172.21.15.96:0/1146915142 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa04004800 0x7ffa04004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:26.826 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.823+0000 7ffa1be00640 1 --2- 172.21.15.96:0/1146915142 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffa1c0d24d0 0x7ffa04005270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:26.826 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.823+0000 7ffa1be00640 1 -- 172.21.15.96:0/1146915142 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ffa1c15edb0 con 0x7ffa040039e0 2024-05-28T06:04:26.826 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.823+0000 7ffa1be00640 1 -- 172.21.15.96:0/1146915142 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ffa0400bb80 con 0x7ffa04004800 2024-05-28T06:04:26.826 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.823+0000 7ffa1be00640 1 -- 172.21.15.96:0/1146915142 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ffa0400bd80 con 0x7ffa1c0d24d0 2024-05-28T06:04:26.827 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.823+0000 7ffa21800640 1 --2- 172.21.15.96:0/1146915142 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa040039e0 0x7ffa04003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:26.827 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.824+0000 7ffa20e00640 1 --2- 172.21.15.96:0/1146915142 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa04004800 0x7ffa04004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:26.827 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.824+0000 7ffa21800640 1 -- 172.21.15.96:0/1146915142 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffa1c0d24d0 msgr2=0x7ffa04005270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:26.827 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.824+0000 7ffa21800640 1 --2- 172.21.15.96:0/1146915142 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffa1c0d24d0 0x7ffa04005270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:26.827 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.824+0000 7ffa21800640 1 -- 172.21.15.96:0/1146915142 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa04004800 msgr2=0x7ffa04004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:26.827 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.824+0000 7ffa21800640 1 --2- 172.21.15.96:0/1146915142 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa04004800 0x7ffa04004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:26.827 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.824+0000 7ffa21800640 1 -- 172.21.15.96:0/1146915142 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa04003520 con 0x7ffa040039e0 2024-05-28T06:04:26.828 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.825+0000 7ffa21800640 1 --2- 172.21.15.96:0/1146915142 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa040039e0 0x7ffa04003e00 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ffa10001750 tx=0x7ffa04005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8182a7954b880075 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:26.828 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.825+0000 7ffa1be00640 1 -- 172.21.15.96:0/1146915142 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa10001370 con 0x7ffa040039e0 2024-05-28T06:04:26.828 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.825+0000 7ffa1be00640 1 -- 172.21.15.96:0/1146915142 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa100052d0 con 0x7ffa040039e0 2024-05-28T06:04:26.828 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.825+0000 7ffa1be00640 1 -- 172.21.15.96:0/1146915142 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa1002abe0 con 0x7ffa040039e0 2024-05-28T06:04:26.828 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.826+0000 7ffa24200640 1 -- 172.21.15.96:0/1146915142 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa040039e0 msgr2=0x7ffa04003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:26.828 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.826+0000 7ffa24200640 1 --2- 172.21.15.96:0/1146915142 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa040039e0 0x7ffa04003e00 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ffa10001750 tx=0x7ffa04005830 comp rx=0 tx=0).stop 2024-05-28T06:04:26.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.826+0000 7ffa22200640 1 -- 172.21.15.96:0/1146915142 reap_dead start 2024-05-28T06:04:26.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.826+0000 7ffa24200640 1 -- 172.21.15.96:0/1146915142 shutdown_connections 2024-05-28T06:04:26.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.826+0000 7ffa24200640 1 -- 172.21.15.96:0/1146915142 >> 172.21.15.96:0/1146915142 conn(0x7ffa1c067890 msgr2=0x7ffa1c067ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:26.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.826+0000 7ffa24200640 1 -- 172.21.15.96:0/1146915142 shutdown_connections 2024-05-28T06:04:26.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.826+0000 7ffa24200640 1 -- 172.21.15.96:0/1146915142 wait complete. 2024-05-28T06:04:26.830 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.828+0000 7ffa24200640 1 Processor -- start 2024-05-28T06:04:26.830 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.828+0000 7ffa24200640 1 -- start start 2024-05-28T06:04:26.830 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.828+0000 7ffa24200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa040039e0 0x7ffa1c1e9e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:26.830 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.829+0000 7ffa24200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa1c1f7f30 0x7ffa1c161040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:26.830 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.829+0000 7ffa24200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffa1c1f83b0 0x7ffa1c161580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:26.830 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.829+0000 7ffa24200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ffa1c0d36f0 con 0x7ffa040039e0 2024-05-28T06:04:26.830 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.829+0000 7ffa24200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ffa1c0d3410 con 0x7ffa1c1f7f30 2024-05-28T06:04:26.830 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.829+0000 7ffa24200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ffa1c0d52a0 con 0x7ffa1c1f83b0 2024-05-28T06:04:26.830 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.829+0000 7ffa21800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa040039e0 0x7ffa1c1e9e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:26.830 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.829+0000 7ffa21800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa040039e0 0x7ffa1c1e9e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48376/0 (socket says 172.21.15.96:48376) 2024-05-28T06:04:26.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.829+0000 7ffa21800640 1 -- 172.21.15.96:0/3389594850 learned_addr learned my addr 172.21.15.96:0/3389594850 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:26.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.829+0000 7ffa20e00640 1 --2- 172.21.15.96:0/3389594850 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa1c1f7f30 0x7ffa1c161040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:26.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.829+0000 7ffa22200640 1 --2- 172.21.15.96:0/3389594850 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffa1c1f83b0 0x7ffa1c161580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:26.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.829+0000 7ffa21800640 1 -- 172.21.15.96:0/3389594850 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffa1c1f83b0 msgr2=0x7ffa1c161580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:26.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.829+0000 7ffa21800640 1 --2- 172.21.15.96:0/3389594850 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffa1c1f83b0 0x7ffa1c161580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:26.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.829+0000 7ffa21800640 1 -- 172.21.15.96:0/3389594850 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa1c1f7f30 msgr2=0x7ffa1c161040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:26.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.829+0000 7ffa21800640 1 --2- 172.21.15.96:0/3389594850 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa1c1f7f30 0x7ffa1c161040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:26.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.829+0000 7ffa21800640 1 -- 172.21.15.96:0/3389594850 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa1c1f8910 con 0x7ffa040039e0 2024-05-28T06:04:26.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.829+0000 7ffa20e00640 1 --2- 172.21.15.96:0/3389594850 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa1c1f7f30 0x7ffa1c161040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:04:26.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.830+0000 7ffa21800640 1 --2- 172.21.15.96:0/3389594850 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa040039e0 0x7ffa1c1e9e50 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ffa10008fd0 tx=0x7ffa10001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:26.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.830+0000 7ffa19600640 1 -- 172.21.15.96:0/3389594850 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa10037040 con 0x7ffa040039e0 2024-05-28T06:04:26.833 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.830+0000 7ffa19600640 1 -- 172.21.15.96:0/3389594850 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa10005480 con 0x7ffa040039e0 2024-05-28T06:04:26.833 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.830+0000 7ffa19600640 1 -- 172.21.15.96:0/3389594850 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa10007a30 con 0x7ffa040039e0 2024-05-28T06:04:26.833 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.830+0000 7ffa24200640 1 -- 172.21.15.96:0/3389594850 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa1c1f8bf0 con 0x7ffa040039e0 2024-05-28T06:04:26.833 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.830+0000 7ffa24200640 1 -- 172.21.15.96:0/3389594850 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa1c1e76a0 con 0x7ffa040039e0 2024-05-28T06:04:26.833 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.831+0000 7ffa24200640 1 -- 172.21.15.96:0/3389594850 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa1c0689e0 con 0x7ffa040039e0 2024-05-28T06:04:26.833 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.832+0000 7ffa19600640 1 -- 172.21.15.96:0/3389594850 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ffa10044020 con 0x7ffa040039e0 2024-05-28T06:04:26.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.833+0000 7ffa19600640 1 --2- 172.21.15.96:0/3389594850 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ffa0006c920 0x7ffa0006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:26.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.833+0000 7ffa19600640 1 -- 172.21.15.96:0/3389594850 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7ffa100ab7f0 con 0x7ffa040039e0 2024-05-28T06:04:26.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.834+0000 7ffa20e00640 1 --2- 172.21.15.96:0/3389594850 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ffa0006c920 0x7ffa0006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:26.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.835+0000 7ffa20e00640 1 --2- 172.21.15.96:0/3389594850 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ffa0006c920 0x7ffa0006ede0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c0cfae0 tx=0x7ffa0c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:26.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:26.837+0000 7ffa19600640 1 -- 172.21.15.96:0/3389594850 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ffa10079c10 con 0x7ffa040039e0 2024-05-28T06:04:27.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.040+0000 7ffa24200640 1 -- 172.21.15.96:0/3389594850 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ffa1c069190 con 0x7ffa040039e0 2024-05-28T06:04:27.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.043+0000 7ffa19600640 1 -- 172.21.15.96:0/3389594850 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16617 (secure 0 0 0) 0x7ffa10079510 con 0x7ffa040039e0 2024-05-28T06:04:27.045 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:04:27.045 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":20,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:04:11.882922+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-28T06:04:08.741938+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-28T06:04:09.301470+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:04:27.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.046+0000 7ffa24200640 1 -- 172.21.15.96:0/3389594850 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ffa0006c920 msgr2=0x7ffa0006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:27.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.046+0000 7ffa24200640 1 --2- 172.21.15.96:0/3389594850 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ffa0006c920 0x7ffa0006ede0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c0cfae0 tx=0x7ffa0c008040 comp rx=0 tx=0).stop 2024-05-28T06:04:27.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.046+0000 7ffa24200640 1 -- 172.21.15.96:0/3389594850 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa040039e0 msgr2=0x7ffa1c1e9e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:27.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.046+0000 7ffa24200640 1 --2- 172.21.15.96:0/3389594850 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa040039e0 0x7ffa1c1e9e50 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ffa10008fd0 tx=0x7ffa10001120 comp rx=0 tx=0).stop 2024-05-28T06:04:27.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.046+0000 7ffa24200640 1 -- 172.21.15.96:0/3389594850 shutdown_connections 2024-05-28T06:04:27.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.046+0000 7ffa24200640 1 --2- 172.21.15.96:0/3389594850 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ffa0006c920 0x7ffa0006ede0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:27.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.046+0000 7ffa24200640 1 --2- 172.21.15.96:0/3389594850 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffa1c1f83b0 0x7ffa1c161580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:27.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.046+0000 7ffa24200640 1 --2- 172.21.15.96:0/3389594850 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa1c1f7f30 0x7ffa1c161040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:27.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.046+0000 7ffa24200640 1 --2- 172.21.15.96:0/3389594850 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa040039e0 0x7ffa1c1e9e50 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:27.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.046+0000 7ffa24200640 1 -- 172.21.15.96:0/3389594850 >> 172.21.15.96:0/3389594850 conn(0x7ffa1c067890 msgr2=0x7ffa1c15da60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:27.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.046+0000 7ffa24200640 1 -- 172.21.15.96:0/3389594850 shutdown_connections 2024-05-28T06:04:27.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.046+0000 7ffa24200640 1 -- 172.21.15.96:0/3389594850 wait complete. 2024-05-28T06:04:27.065 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-28T06:04:27.255 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.254+0000 7f2a9e000640 1 Processor -- start 2024-05-28T06:04:27.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.254+0000 7f2a9e000640 1 -- start start 2024-05-28T06:04:27.257 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.255+0000 7f2a9e000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f2a9806f860 0x7f2a980b3150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:27.257 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.256+0000 7f2a9e000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f2a980d0bb0 0x7f2a980ad130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:27.257 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.256+0000 7f2a9e000640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a9806ed10 con 0x7f2a980b3690 2024-05-28T06:04:27.257 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.256+0000 7f2a9e000640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f2a9815cb00 con 0x7f2a9806f860 2024-05-28T06:04:27.257 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.256+0000 7f2a9e000640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f2a980b3cc0 con 0x7f2a980d0bb0 2024-05-28T06:04:27.257 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.256+0000 7f2a97400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f2a9806f860 0x7f2a980b3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:27.257 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.256+0000 7f2a8ea00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f2a980b3690 0x7f2a980d04a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:35300/0 (socket says 172.21.15.96:35300) 2024-05-28T06:04:27.257 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.256+0000 7f2a8ea00640 1 -- 172.21.15.96:0/184668436 learned_addr learned my addr 172.21.15.96:0/184668436 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:04:27.258 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.256+0000 7f2a97e00640 1 --2- 172.21.15.96:0/184668436 >> v2:172.21.15.123:3300/0 conn(0x7f2a980d0bb0 0x7f2a980ad130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:27.258 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.256+0000 7f2a97400640 1 -- 172.21.15.96:0/184668436 >> v2:172.21.15.123:3300/0 conn(0x7f2a980d0bb0 msgr2=0x7f2a980ad130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:27.258 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.256+0000 7f2a97400640 1 --2- 172.21.15.96:0/184668436 >> v2:172.21.15.123:3300/0 conn(0x7f2a980d0bb0 0x7f2a980ad130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:27.259 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.256+0000 7f2a97400640 1 -- 172.21.15.96:0/184668436 >> v1:172.21.15.96:6789/0 conn(0x7f2a980b3690 legacy=0x7f2a980d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:27.259 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.256+0000 7f2a97400640 1 -- 172.21.15.96:0/184668436 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a980b3fa0 con 0x7f2a9806f860 2024-05-28T06:04:27.259 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.256+0000 7f2a96a00640 1 -- 172.21.15.96:0/184668436 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2221139018 0 0) 0x7f2a80001110 con 0x7f2a980b3690 2024-05-28T06:04:27.259 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.256+0000 7f2a97e00640 1 --2- 172.21.15.96:0/184668436 >> v2:172.21.15.123:3300/0 conn(0x7f2a980d0bb0 0x7f2a980ad130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:04:27.260 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.257+0000 7f2a97400640 1 --2- 172.21.15.96:0/184668436 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2a9806f860 0x7f2a980b3150 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f2a980af6e0 tx=0x7f2a8802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=135fa5e30b897224 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:27.260 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.258+0000 7f2a96a00640 1 -- 172.21.15.96:0/184668436 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a88037070 con 0x7f2a9806f860 2024-05-28T06:04:27.261 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.258+0000 7f2a96a00640 1 -- 172.21.15.96:0/184668436 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a88032440 con 0x7f2a9806f860 2024-05-28T06:04:27.261 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.258+0000 7f2a96a00640 1 -- 172.21.15.96:0/184668436 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a88032810 con 0x7f2a9806f860 2024-05-28T06:04:27.261 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.258+0000 7f2a9e000640 1 -- 172.21.15.96:0/184668436 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2a9806f860 msgr2=0x7f2a980b3150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:27.261 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.258+0000 7f2a9e000640 1 --2- 172.21.15.96:0/184668436 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2a9806f860 0x7f2a980b3150 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f2a980af6e0 tx=0x7f2a8802a1a0 comp rx=0 tx=0).stop 2024-05-28T06:04:27.261 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.258+0000 7f2a9e000640 1 -- 172.21.15.96:0/184668436 shutdown_connections 2024-05-28T06:04:27.261 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.258+0000 7f2a9e000640 1 --2- 172.21.15.96:0/184668436 >> v2:172.21.15.123:3300/0 conn(0x7f2a980d0bb0 0x7f2a980ad130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:27.261 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.258+0000 7f2a9e000640 1 --2- 172.21.15.96:0/184668436 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2a9806f860 0x7f2a980b3150 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:27.261 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.258+0000 7f2a9e000640 1 -- 172.21.15.96:0/184668436 >> 172.21.15.96:0/184668436 conn(0x7f2a9806bae0 msgr2=0x7f2a9806bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:27.261 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.259+0000 7f2a9e000640 1 -- 172.21.15.96:0/184668436 shutdown_connections 2024-05-28T06:04:27.261 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.259+0000 7f2a9e000640 1 -- 172.21.15.96:0/184668436 wait complete. 2024-05-28T06:04:27.262 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.260+0000 7f2a9e000640 1 Processor -- start 2024-05-28T06:04:27.262 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.260+0000 7f2a9e000640 1 -- start start 2024-05-28T06:04:27.264 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.261+0000 7f2a9e000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2a9806f860 0x7f2a980c6a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:27.264 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.261+0000 7f2a9e000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2a980b3690 0x7f2a980c7330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:27.264 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.261+0000 7f2a9e000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2a980d0bb0 0x7f2a98099a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:27.264 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.261+0000 7f2a9e000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f2a980b4840 con 0x7f2a980d0bb0 2024-05-28T06:04:27.264 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.261+0000 7f2a9e000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f2a980b4560 con 0x7f2a9806f860 2024-05-28T06:04:27.264 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.261+0000 7f2a9e000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f2a980ad670 con 0x7f2a980b3690 2024-05-28T06:04:27.264 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.262+0000 7f2a97e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2a980d0bb0 0x7f2a98099a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:27.264 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.262+0000 7f2a97e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2a980d0bb0 0x7f2a98099a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48398/0 (socket says 172.21.15.96:48398) 2024-05-28T06:04:27.264 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.262+0000 7f2a97e00640 1 -- 172.21.15.96:0/606846619 learned_addr learned my addr 172.21.15.96:0/606846619 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:27.264 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.262+0000 7f2a97400640 1 --2- 172.21.15.96:0/606846619 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2a9806f860 0x7f2a980c6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:27.265 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.262+0000 7f2a8ea00640 1 --2- 172.21.15.96:0/606846619 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2a980b3690 0x7f2a980c7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:27.265 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.262+0000 7f2a97e00640 1 -- 172.21.15.96:0/606846619 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2a980b3690 msgr2=0x7f2a980c7330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:27.265 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.262+0000 7f2a97e00640 1 --2- 172.21.15.96:0/606846619 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2a980b3690 0x7f2a980c7330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:27.265 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.262+0000 7f2a97e00640 1 -- 172.21.15.96:0/606846619 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2a9806f860 msgr2=0x7f2a980c6a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:27.265 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.262+0000 7f2a97e00640 1 --2- 172.21.15.96:0/606846619 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2a9806f860 0x7f2a980c6a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:27.265 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.262+0000 7f2a97e00640 1 -- 172.21.15.96:0/606846619 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a9815fb20 con 0x7f2a980d0bb0 2024-05-28T06:04:27.265 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.262+0000 7f2a97e00640 1 --2- 172.21.15.96:0/606846619 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2a980d0bb0 0x7f2a98099a10 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f2a9000cd80 tx=0x7f2a9000a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:27.265 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.263+0000 7f2a8fe00640 1 -- 172.21.15.96:0/606846619 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a9000f680 con 0x7f2a980d0bb0 2024-05-28T06:04:27.265 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.263+0000 7f2a8fe00640 1 -- 172.21.15.96:0/606846619 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a9000f840 con 0x7f2a980d0bb0 2024-05-28T06:04:27.265 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.263+0000 7f2a9e000640 1 -- 172.21.15.96:0/606846619 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a980c7a30 con 0x7f2a980d0bb0 2024-05-28T06:04:27.265 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.263+0000 7f2a8fe00640 1 -- 172.21.15.96:0/606846619 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a9000fc30 con 0x7f2a980d0bb0 2024-05-28T06:04:27.265 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.263+0000 7f2a9e000640 1 -- 172.21.15.96:0/606846619 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a980c7ff0 con 0x7f2a980d0bb0 2024-05-28T06:04:27.266 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.264+0000 7f2a9e000640 1 -- 172.21.15.96:0/606846619 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a980689e0 con 0x7f2a980d0bb0 2024-05-28T06:04:27.266 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.265+0000 7f2a8fe00640 1 -- 172.21.15.96:0/606846619 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2a900213f0 con 0x7f2a980d0bb0 2024-05-28T06:04:27.267 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.266+0000 7f2a8fe00640 1 --2- 172.21.15.96:0/606846619 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2a7406c920 0x7f2a7406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:27.267 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.266+0000 7f2a8fe00640 1 -- 172.21.15.96:0/606846619 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f2a9008c950 con 0x7f2a980d0bb0 2024-05-28T06:04:27.267 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.266+0000 7f2a97400640 1 --2- 172.21.15.96:0/606846619 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2a7406c920 0x7f2a7406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:27.268 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.267+0000 7f2a97400640 1 --2- 172.21.15.96:0/606846619 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2a7406c920 0x7f2a7406ede0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2a88034000 tx=0x7f2a8802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:27.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.270+0000 7f2a8fe00640 1 -- 172.21.15.96:0/606846619 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2a90014030 con 0x7f2a980d0bb0 2024-05-28T06:04:27.510 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.509+0000 7f2a9e000640 1 -- 172.21.15.96:0/606846619 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2a9806e2f0 con 0x7f2a980d0bb0 2024-05-28T06:04:27.512 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.510+0000 7f2a8fe00640 1 -- 172.21.15.96:0/606846619 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5178 (secure 0 0 0) 0x7f2a9005a690 con 0x7f2a980d0bb0 2024-05-28T06:04:27.512 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:04:27.512 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/3349661842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":3349661842},{"type":"v1","addr":"172.21.15.96:6833","nonce":3349661842}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4270,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/3383264102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":3383264102},{"type":"v1","addr":"172.21.15.96:6835","nonce":3383264102}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4279,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/3068402544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":3068402544},{"type":"v1","addr":"172.21.15.123:6837","nonce":3068402544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-28T06:04:10.319738+0000","modified":"2024-05-28T06:04:13.704080+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4281},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4281":{"gid":4281,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.123:6835/605814935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":605814935},{"type":"v1","addr":"172.21.15.123:6835","nonce":605814935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4281,"qdb_cluster":[4281]},"id":1}]} 2024-05-28T06:04:27.512 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 7 2024-05-28T06:04:27.514 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.513+0000 7f2a9e000640 1 -- 172.21.15.96:0/606846619 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2a7406c920 msgr2=0x7f2a7406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:27.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.513+0000 7f2a9e000640 1 --2- 172.21.15.96:0/606846619 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2a7406c920 0x7f2a7406ede0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2a88034000 tx=0x7f2a8802e000 comp rx=0 tx=0).stop 2024-05-28T06:04:27.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.513+0000 7f2a9e000640 1 -- 172.21.15.96:0/606846619 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2a980d0bb0 msgr2=0x7f2a98099a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:27.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.513+0000 7f2a9e000640 1 --2- 172.21.15.96:0/606846619 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2a980d0bb0 0x7f2a98099a10 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f2a9000cd80 tx=0x7f2a9000a650 comp rx=0 tx=0).stop 2024-05-28T06:04:27.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.513+0000 7f2a9e000640 1 -- 172.21.15.96:0/606846619 shutdown_connections 2024-05-28T06:04:27.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.513+0000 7f2a9e000640 1 --2- 172.21.15.96:0/606846619 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2a7406c920 0x7f2a7406ede0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:27.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.513+0000 7f2a9e000640 1 --2- 172.21.15.96:0/606846619 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2a980d0bb0 0x7f2a98099a10 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:27.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.513+0000 7f2a9e000640 1 --2- 172.21.15.96:0/606846619 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2a980b3690 0x7f2a980c7330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:27.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.513+0000 7f2a9e000640 1 --2- 172.21.15.96:0/606846619 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2a9806f860 0x7f2a980c6a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:27.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.513+0000 7f2a9e000640 1 -- 172.21.15.96:0/606846619 >> 172.21.15.96:0/606846619 conn(0x7f2a9806bae0 msgr2=0x7f2a980b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:27.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.513+0000 7f2a9e000640 1 -- 172.21.15.96:0/606846619 shutdown_connections 2024-05-28T06:04:27.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:27.513+0000 7f2a9e000640 1 -- 172.21.15.96:0/606846619 wait complete. 2024-05-28T06:04:27.935 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 7, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-05-28T06:04:10.319738+0000', 'modified': '2024-05-28T06:04:13.704080+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4281}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4281': {'gid': 4281, 'name': 'b', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.123:6835/605814935', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.123:6834', 'nonce': 605814935}, {'type': 'v1', 'addr': '172.21.15.123:6835', 'nonce': 605814935}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4281, 'qdb_cluster': [4281]} 2024-05-28T06:04:27.936 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-05-28T06:04:27.936 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-05-28T06:04:28.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.143+0000 7f09b5400640 1 Processor -- start 2024-05-28T06:04:28.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.143+0000 7f09b5400640 1 -- start start 2024-05-28T06:04:28.145 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.144+0000 7f09b5400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f09b006f860 0x7f09b00b3150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:28.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.144+0000 7f09b5400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f09b00d0bb0 0x7f09b00ad130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:28.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.144+0000 7f09b5400640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09b006ed10 con 0x7f09b00b3690 2024-05-28T06:04:28.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.144+0000 7f09b5400640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f09b015cb00 con 0x7f09b006f860 2024-05-28T06:04:28.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.144+0000 7f09b5400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f09b00b3cc0 con 0x7f09b00d0bb0 2024-05-28T06:04:28.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.145+0000 7f09ae800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f09b006f860 0x7f09b00b3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:28.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.145+0000 7f09af200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f09b00d0bb0 0x7f09b00ad130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:28.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.145+0000 7f09ae800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f09b006f860 0x7f09b00b3150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.96:40090/0 (socket says 172.21.15.96:40090) 2024-05-28T06:04:28.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.145+0000 7f09ae800640 1 -- 172.21.15.96:0/3809611861 learned_addr learned my addr 172.21.15.96:0/3809611861 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:28.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.146+0000 7f09af200640 1 -- 172.21.15.96:0/3809611861 >> v2:172.21.15.123:3300/0 conn(0x7f09b006f860 msgr2=0x7f09b00b3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:28.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.146+0000 7f09af200640 1 --2- 172.21.15.96:0/3809611861 >> v2:172.21.15.123:3300/0 conn(0x7f09b006f860 0x7f09b00b3150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:28.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.146+0000 7f09af200640 1 -- 172.21.15.96:0/3809611861 >> v1:172.21.15.123:6789/0 conn(0x7f09b00b3690 legacy=0x7f09b00d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:28.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.146+0000 7f09af200640 1 -- 172.21.15.96:0/3809611861 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09b00b3fa0 con 0x7f09b00d0bb0 2024-05-28T06:04:28.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.146+0000 7f09ae800640 1 --2- 172.21.15.96:0/3809611861 >> v2:172.21.15.123:3300/0 conn(0x7f09b006f860 0x7f09b00b3150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-28T06:04:28.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.147+0000 7f09af200640 1 --2- 172.21.15.96:0/3809611861 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f09b00d0bb0 0x7f09b00ad130 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f09a8009d90 tx=0x7f09a802a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cbb38c8642d2922b server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:28.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.147+0000 7f09ad400640 1 -- 172.21.15.96:0/3809611861 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a8007740 con 0x7f09b00d0bb0 2024-05-28T06:04:28.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.148+0000 7f09ad400640 1 -- 172.21.15.96:0/3809611861 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09a80078e0 con 0x7f09b00d0bb0 2024-05-28T06:04:28.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.148+0000 7f09ad400640 1 -- 172.21.15.96:0/3809611861 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a8007cb0 con 0x7f09b00d0bb0 2024-05-28T06:04:28.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.148+0000 7f09b5400640 1 -- 172.21.15.96:0/3809611861 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f09b00d0bb0 msgr2=0x7f09b00ad130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:28.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.148+0000 7f09b5400640 1 --2- 172.21.15.96:0/3809611861 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f09b00d0bb0 0x7f09b00ad130 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f09a8009d90 tx=0x7f09a802a660 comp rx=0 tx=0).stop 2024-05-28T06:04:28.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.148+0000 7f09b5400640 1 -- 172.21.15.96:0/3809611861 shutdown_connections 2024-05-28T06:04:28.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.148+0000 7f09b5400640 1 --2- 172.21.15.96:0/3809611861 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f09b00d0bb0 0x7f09b00ad130 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:28.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.148+0000 7f09b5400640 1 --2- 172.21.15.96:0/3809611861 >> v2:172.21.15.123:3300/0 conn(0x7f09b006f860 0x7f09b00b3150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:28.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.148+0000 7f09b5400640 1 -- 172.21.15.96:0/3809611861 >> 172.21.15.96:0/3809611861 conn(0x7f09b006bae0 msgr2=0x7f09b006bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:28.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.148+0000 7f09b5400640 1 -- 172.21.15.96:0/3809611861 shutdown_connections 2024-05-28T06:04:28.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.148+0000 7f09b5400640 1 -- 172.21.15.96:0/3809611861 wait complete. 2024-05-28T06:04:28.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.150+0000 7f09b5400640 1 Processor -- start 2024-05-28T06:04:28.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.150+0000 7f09b5400640 1 -- start start 2024-05-28T06:04:28.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.151+0000 7f09b5400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f09b006f860 0x7f09b01596e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:28.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.151+0000 7f09b5400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f09b00b3690 0x7f09b01f7270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:28.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.151+0000 7f09b5400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09b00d0bb0 0x7f09b00c7330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:28.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.151+0000 7f09b5400640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f09b00b4840 con 0x7f09b006f860 2024-05-28T06:04:28.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.151+0000 7f09b5400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f09b00b4560 con 0x7f09b00d0bb0 2024-05-28T06:04:28.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.151+0000 7f09b5400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f09b00ad670 con 0x7f09b00b3690 2024-05-28T06:04:28.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.151+0000 7f09ae800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f09b006f860 0x7f09b01596e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:28.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.151+0000 7f09ae800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f09b006f860 0x7f09b01596e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48400/0 (socket says 172.21.15.96:48400) 2024-05-28T06:04:28.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.151+0000 7f09ae800640 1 -- 172.21.15.96:0/290539950 learned_addr learned my addr 172.21.15.96:0/290539950 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:28.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.152+0000 7f09af200640 1 --2- 172.21.15.96:0/290539950 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09b00d0bb0 0x7f09b00c7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:28.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.152+0000 7f09ade00640 1 --2- 172.21.15.96:0/290539950 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f09b00b3690 0x7f09b01f7270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:28.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.152+0000 7f09ae800640 1 -- 172.21.15.96:0/290539950 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f09b00b3690 msgr2=0x7f09b01f7270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:28.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.152+0000 7f09ae800640 1 --2- 172.21.15.96:0/290539950 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f09b00b3690 0x7f09b01f7270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:28.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.152+0000 7f09ae800640 1 -- 172.21.15.96:0/290539950 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09b00d0bb0 msgr2=0x7f09b00c7330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:28.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.152+0000 7f09ae800640 1 --2- 172.21.15.96:0/290539950 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09b00d0bb0 0x7f09b00c7330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:28.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.152+0000 7f09ae800640 1 -- 172.21.15.96:0/290539950 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09b00cdf60 con 0x7f09b006f860 2024-05-28T06:04:28.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.152+0000 7f09ade00640 1 --2- 172.21.15.96:0/290539950 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f09b00b3690 0x7f09b01f7270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:04:28.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.152+0000 7f09ae800640 1 --2- 172.21.15.96:0/290539950 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f09b006f860 0x7f09b01596e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f09a000e9d0 tx=0x7f09a000ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:28.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.152+0000 7f09a6a00640 1 -- 172.21.15.96:0/290539950 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a0017070 con 0x7f09b006f860 2024-05-28T06:04:28.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.152+0000 7f09a6a00640 1 -- 172.21.15.96:0/290539950 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09a000ca60 con 0x7f09b006f860 2024-05-28T06:04:28.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.152+0000 7f09b5400640 1 -- 172.21.15.96:0/290539950 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09b00ce240 con 0x7f09b006f860 2024-05-28T06:04:28.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.152+0000 7f09a6a00640 1 -- 172.21.15.96:0/290539950 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a000ce10 con 0x7f09b006f860 2024-05-28T06:04:28.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.153+0000 7f09b5400640 1 -- 172.21.15.96:0/290539950 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09b00ce800 con 0x7f09b006f860 2024-05-28T06:04:28.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.155+0000 7f09a6a00640 1 -- 172.21.15.96:0/290539950 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f09a001a070 con 0x7f09b006f860 2024-05-28T06:04:28.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.156+0000 7f09a6a00640 1 --2- 172.21.15.96:0/290539950 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f099006c8f0 0x7f099006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:28.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.156+0000 7f09a6a00640 1 -- 172.21.15.96:0/290539950 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f09a008be50 con 0x7f09b006f860 2024-05-28T06:04:28.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.157+0000 7f09b83a9740 1 Processor -- start 2024-05-28T06:04:28.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.157+0000 7f09ade00640 1 --2- 172.21.15.96:0/290539950 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f099006c8f0 0x7f099006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:28.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.157+0000 7f09b83a9740 1 -- start start 2024-05-28T06:04:28.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.158+0000 7f09b83a9740 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x55f68265e2a0 0x55f682661a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:28.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.158+0000 7f09b83a9740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f682661f60 0x55f68267dea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:28.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.158+0000 7f09b83a9740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f68267e3e0 0x55f68267e840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:28.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.158+0000 7f09b83a9740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55f68259bb10 con 0x55f68267e3e0 2024-05-28T06:04:28.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.158+0000 7f09b83a9740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55f6825422b0 con 0x55f682661f60 2024-05-28T06:04:28.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.158+0000 7f09b83a9740 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x55f682597ad0 con 0x55f68265e2a0 2024-05-28T06:04:28.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.158+0000 7f09ade00640 1 --2- 172.21.15.96:0/290539950 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f099006c8f0 0x7f099006edb0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f09b0159c20 tx=0x7f099c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:28.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.158+0000 7f09af200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f68267e3e0 0x55f68267e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:28.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.158+0000 7f09af200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f68267e3e0 0x55f68267e840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48408/0 (socket says 172.21.15.96:48408) 2024-05-28T06:04:28.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.158+0000 7f09af200640 1 -- 172.21.15.96:0/1114318102 learned_addr learned my addr 172.21.15.96:0/1114318102 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:28.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.158+0000 7f09ade00640 1 --2- 172.21.15.96:0/1114318102 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f682661f60 0x55f68267dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:28.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.158+0000 7f09ae800640 1 --2- 172.21.15.96:0/1114318102 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x55f68265e2a0 0x55f682661a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:28.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.158+0000 7f09af200640 1 -- 172.21.15.96:0/1114318102 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x55f68265e2a0 msgr2=0x55f682661a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:28.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.158+0000 7f09af200640 1 --2- 172.21.15.96:0/1114318102 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x55f68265e2a0 0x55f682661a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:28.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.158+0000 7f09af200640 1 -- 172.21.15.96:0/1114318102 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f682661f60 msgr2=0x55f68267dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:28.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.158+0000 7f09af200640 1 --2- 172.21.15.96:0/1114318102 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f682661f60 0x55f68267dea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:28.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.158+0000 7f09af200640 1 -- 172.21.15.96:0/1114318102 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f6822ae2d0 con 0x55f68267e3e0 2024-05-28T06:04:28.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.159+0000 7f09af200640 1 --2- 172.21.15.96:0/1114318102 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f68267e3e0 0x55f68267e840 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f09a802ac50 tx=0x7f09a802f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20b67677d35e09e5 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:28.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.159+0000 7f099b000640 1 -- 172.21.15.96:0/1114318102 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a8007840 con 0x55f68267e3e0 2024-05-28T06:04:28.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.159+0000 7f099b000640 1 -- 172.21.15.96:0/1114318102 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09a80079e0 con 0x55f68267e3e0 2024-05-28T06:04:28.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.159+0000 7f099b000640 1 -- 172.21.15.96:0/1114318102 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a8007d90 con 0x55f68267e3e0 2024-05-28T06:04:28.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.160+0000 7f09b83a9740 1 -- 172.21.15.96:0/1114318102 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f68267e3e0 msgr2=0x55f68267e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:28.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.160+0000 7f09b83a9740 1 --2- 172.21.15.96:0/1114318102 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f68267e3e0 0x55f68267e840 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f09a802ac50 tx=0x7f09a802f000 comp rx=0 tx=0).stop 2024-05-28T06:04:28.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.160+0000 7f09b83a9740 1 -- 172.21.15.96:0/1114318102 shutdown_connections 2024-05-28T06:04:28.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.160+0000 7f09b83a9740 1 --2- 172.21.15.96:0/1114318102 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f68267e3e0 0x55f68267e840 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:28.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.160+0000 7f09b83a9740 1 --2- 172.21.15.96:0/1114318102 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f682661f60 0x55f68267dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:28.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.160+0000 7f09b83a9740 1 --2- 172.21.15.96:0/1114318102 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x55f68265e2a0 0x55f682661a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:28.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.160+0000 7f09b83a9740 1 -- 172.21.15.96:0/1114318102 >> 172.21.15.96:0/1114318102 conn(0x55f682634280 msgr2=0x55f682635280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:28.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.160+0000 7f09b83a9740 1 -- 172.21.15.96:0/1114318102 shutdown_connections 2024-05-28T06:04:28.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.160+0000 7f09b83a9740 1 -- 172.21.15.96:0/1114318102 wait complete. 2024-05-28T06:04:28.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.160+0000 7f09b83a9740 1 -- start start 2024-05-28T06:04:28.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.161+0000 7f09b83a9740 1 Processor -- start 2024-05-28T06:04:28.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.161+0000 7f09b83a9740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f682661f60 0x55f68265cea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:28.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.162+0000 7f09b83a9740 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x55f68267e3e0 0x55f682680540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:28.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.162+0000 7f09b83a9740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f6826659b0 0x55f682680a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:28.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.162+0000 7f09b83a9740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55f6826320b0 con 0x55f6826659b0 2024-05-28T06:04:28.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.162+0000 7f09b83a9740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55f682555570 con 0x55f682661f60 2024-05-28T06:04:28.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.162+0000 7f09b83a9740 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x55f682632390 con 0x55f68267e3e0 2024-05-28T06:04:28.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.162+0000 7f09b83a9740 10 client.0 _ll_register_callbacks cb 0x55f682523fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-28T06:04:28.166 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.162+0000 7f09af200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f6826659b0 0x55f682680a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:28.166 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.162+0000 7f09af200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f6826659b0 0x55f682680a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48414/0 (socket says 172.21.15.96:48414) 2024-05-28T06:04:28.166 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.162+0000 7f09af200640 1 -- 172.21.15.96:0/4103827063 learned_addr learned my addr 172.21.15.96:0/4103827063 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:28.166 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.162+0000 7f09ae800640 1 --2- 172.21.15.96:0/4103827063 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f682661f60 0x55f68265cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:28.166 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.162+0000 7f09af200640 1 -- 172.21.15.96:0/4103827063 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x55f68267e3e0 msgr2=0x55f682680540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:28.166 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.162+0000 7f09ade00640 1 --2- 172.21.15.96:0/4103827063 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x55f68267e3e0 0x55f682680540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:28.166 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.162+0000 7f09af200640 1 --2- 172.21.15.96:0/4103827063 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x55f68267e3e0 0x55f682680540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:28.166 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.162+0000 7f09af200640 1 -- 172.21.15.96:0/4103827063 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f682661f60 msgr2=0x55f68265cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:28.166 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.162+0000 7f09af200640 1 --2- 172.21.15.96:0/4103827063 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f682661f60 0x55f68265cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:28.166 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.162+0000 7f09af200640 1 -- 172.21.15.96:0/4103827063 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f6823786b0 con 0x55f6826659b0 2024-05-28T06:04:28.166 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.162+0000 7f09ae800640 1 --2- 172.21.15.96:0/4103827063 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f682661f60 0x55f68265cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:04:28.166 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.162+0000 7f09af200640 1 --2- 172.21.15.96:0/4103827063 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f6826659b0 0x55f682680a80 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f09a8009e80 tx=0x7f09a8004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=780111403c24c17c server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:28.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.162+0000 7f098f400640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-05-28T06:04:28.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.163+0000 7f098f400640 1 -- 172.21.15.96:0/4103827063 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a8007da0 con 0x55f6826659b0 2024-05-28T06:04:28.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.163+0000 7f098f400640 1 -- 172.21.15.96:0/4103827063 <== mon.0 v2:172.21.15.96:3300/0 2 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f09a8003990 con 0x55f6826659b0 2024-05-28T06:04:28.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.163+0000 7f09b83a9740 1 -- 172.21.15.96:0/4103827063 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f09a80079e0 con 0x55f6826659b0 2024-05-28T06:04:28.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.163+0000 7f098f400640 1 -- 172.21.15.96:0/4103827063 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) ==== 24+0+0 (secure 0 0 0) 0x7f09a8057050 con 0x55f6826659b0 2024-05-28T06:04:28.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.163+0000 7f09b83a9740 10 client.4647 fetch_fsmap learned FSMap version 7 2024-05-28T06:04:28.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.163+0000 7f09b83a9740 1 -- 172.21.15.96:0/4103827063 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) -- 0x55f682662c30 con 0x55f6826659b0 2024-05-28T06:04:28.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.163+0000 7f098f400640 1 -- 172.21.15.96:0/4103827063 <== mon.0 v2:172.21.15.96:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09a80079e0 con 0x55f6826659b0 2024-05-28T06:04:28.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.163+0000 7f098f400640 1 -- 172.21.15.96:0/4103827063 <== mon.0 v2:172.21.15.96:3300/0 5 ==== fsmap(e 7) ==== 3494+0+0 (secure 0 0 0) 0x7f09a805a020 con 0x55f6826659b0 2024-05-28T06:04:28.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.163+0000 7f09b83a9740 10 client.4647 fetch_fsmap finished waiting for FSMap version 7 2024-05-28T06:04:28.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.163+0000 7f09b83a9740 10 client.4647 resolve_mds: resolved name 'b' to daemon mds.b 2024-05-28T06:04:28.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.164+0000 7f09b83a9740 20 client.4647 populate_metadata read hostname 'smithi096' 2024-05-28T06:04:28.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.164+0000 7f09b83a9740 1 --2- 172.21.15.96:0/4103827063 >> [v2:172.21.15.123:6834/605814935,v1:172.21.15.123:6835/605814935] conn(0x55f682663ec0 0x55f6826642a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:28.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.164+0000 7f09b83a9740 4 client.4647 mds_command: new command op to 4281 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-28T06:04:28.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.164+0000 7f09b83a9740 1 -- 172.21.15.96:0/4103827063 --> [v2:172.21.15.123:6834/605814935,v1:172.21.15.123:6835/605814935] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f6825a9530 con 0x55f682663ec0 2024-05-28T06:04:28.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.164+0000 7f09ae800640 1 --2- 172.21.15.96:0/4103827063 >> [v2:172.21.15.123:6834/605814935,v1:172.21.15.123:6835/605814935] conn(0x55f682663ec0 0x55f6826642a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:28.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.165+0000 7f09ae800640 1 --2- 172.21.15.96:0/4103827063 >> [v2:172.21.15.123:6834/605814935,v1:172.21.15.123:6835/605814935] conn(0x55f682663ec0 0x55f6826642a0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b5ebeac9cf5a0430 server_cookie=5e8dfb607cae72fb in_seq=0 out_seq=0 2024-05-28T06:04:28.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.165+0000 7f098f400640 10 client.4647 ms_handle_connect on v2:172.21.15.123:6834/605814935 2024-05-28T06:04:28.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.166+0000 7f098f400640 1 -- 172.21.15.96:0/4103827063 <== mds.0 v2:172.21.15.123:6834/605814935 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f6825a9530 con 0x55f682663ec0 2024-05-28T06:04:28.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.166+0000 7f098f400640 10 client.4647 handle_command_reply: tid=0 2024-05-28T06:04:28.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.167+0000 7f09b83a9740 1 client.4647 shutdown 2024-05-28T06:04:28.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.167+0000 7f09b83a9740 1 -- 172.21.15.96:0/4103827063 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f6826659b0 msgr2=0x55f682680a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:28.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.167+0000 7f09b83a9740 1 --2- 172.21.15.96:0/4103827063 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f6826659b0 0x55f682680a80 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f09a8009e80 tx=0x7f09a8004ab0 comp rx=0 tx=0).stop 2024-05-28T06:04:28.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.167+0000 7f09b83a9740 1 -- 172.21.15.96:0/4103827063 shutdown_connections 2024-05-28T06:04:28.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.167+0000 7f09b83a9740 1 --2- 172.21.15.96:0/4103827063 >> [v2:172.21.15.123:6834/605814935,v1:172.21.15.123:6835/605814935] conn(0x55f682663ec0 0x55f6826642a0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:28.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.167+0000 7f09b83a9740 1 --2- 172.21.15.96:0/4103827063 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f6826659b0 0x55f682680a80 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:28.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.167+0000 7f09b83a9740 1 --2- 172.21.15.96:0/4103827063 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x55f68267e3e0 0x55f682680540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:28.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.167+0000 7f09b83a9740 1 --2- 172.21.15.96:0/4103827063 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f682661f60 0x55f68265cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:28.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.167+0000 7f09b83a9740 1 -- 172.21.15.96:0/4103827063 >> 172.21.15.96:0/4103827063 conn(0x55f68265e2a0 msgr2=0x55f682630ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:28.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.167+0000 7f098f400640 0 client.4647 ms_handle_reset on v2:172.21.15.123:6834/605814935 2024-05-28T06:04:28.171 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.167+0000 7f09b83a9740 1 -- 172.21.15.96:0/4103827063 shutdown_connections 2024-05-28T06:04:28.171 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.167+0000 7f09b83a9740 1 -- 172.21.15.96:0/4103827063 wait complete. 2024-05-28T06:04:28.171 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.167+0000 7f09b83a9740 20 client.4647 trim_cache size 0 max 16384 2024-05-28T06:04:28.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.187+0000 7f09b83a9740 1 Processor -- start 2024-05-28T06:04:28.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.187+0000 7f09b83a9740 1 -- start start 2024-05-28T06:04:28.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.188+0000 7f09b83a9740 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x55f682661f60 0x55f682661940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:28.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.188+0000 7f09b83a9740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f6826659b0 0x55f6826335b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:28.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.188+0000 7f09b83a9740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f68267e3e0 0x55f682638910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:28.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.188+0000 7f09b83a9740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f09b0200030 con 0x55f6826659b0 2024-05-28T06:04:28.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.188+0000 7f09b83a9740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f09b00cf7d0 con 0x55f68267e3e0 2024-05-28T06:04:28.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.188+0000 7f09b83a9740 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f09b00cfe20 con 0x55f682661f60 2024-05-28T06:04:28.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.188+0000 7f09ade00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f6826659b0 0x55f6826335b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:28.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.188+0000 7f09ade00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f6826659b0 0x55f6826335b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48424/0 (socket says 172.21.15.96:48424) 2024-05-28T06:04:28.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.188+0000 7f09ade00640 1 -- 172.21.15.96:0/2816138167 learned_addr learned my addr 172.21.15.96:0/2816138167 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:28.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.188+0000 7f09af200640 1 --2- 172.21.15.96:0/2816138167 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f68267e3e0 0x55f682638910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:28.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.188+0000 7f09ae800640 1 --2- 172.21.15.96:0/2816138167 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x55f682661f60 0x55f682661940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:28.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.188+0000 7f09ade00640 1 -- 172.21.15.96:0/2816138167 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x55f682661f60 msgr2=0x55f682661940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:28.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.188+0000 7f09ade00640 1 --2- 172.21.15.96:0/2816138167 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x55f682661f60 0x55f682661940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:28.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.189+0000 7f09ade00640 1 -- 172.21.15.96:0/2816138167 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f68267e3e0 msgr2=0x55f682638910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:28.192 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.189+0000 7f09ade00640 1 --2- 172.21.15.96:0/2816138167 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f68267e3e0 0x55f682638910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:28.192 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.189+0000 7f09ade00640 1 -- 172.21.15.96:0/2816138167 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f68262caa0 con 0x55f6826659b0 2024-05-28T06:04:28.192 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.189+0000 7f09ade00640 1 --2- 172.21.15.96:0/2816138167 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f6826659b0 0x55f6826335b0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f099c00d780 tx=0x7f099c013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4c3dce75aabcc90 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:28.192 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.189+0000 7f099a600640 1 -- 172.21.15.96:0/2816138167 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f099c01d040 con 0x55f6826659b0 2024-05-28T06:04:28.192 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.189+0000 7f099a600640 1 -- 172.21.15.96:0/2816138167 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f099c026070 con 0x55f6826659b0 2024-05-28T06:04:28.192 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.189+0000 7f099a600640 1 -- 172.21.15.96:0/2816138167 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f099c0149e0 con 0x55f6826659b0 2024-05-28T06:04:28.192 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.190+0000 7f09b83a9740 1 -- 172.21.15.96:0/2816138167 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f6826659b0 msgr2=0x55f6826335b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:28.192 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.190+0000 7f09b83a9740 1 --2- 172.21.15.96:0/2816138167 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f6826659b0 0x55f6826335b0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f099c00d780 tx=0x7f099c013000 comp rx=0 tx=0).stop 2024-05-28T06:04:28.192 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.190+0000 7f09b83a9740 1 -- 172.21.15.96:0/2816138167 shutdown_connections 2024-05-28T06:04:28.192 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.190+0000 7f09b83a9740 1 --2- 172.21.15.96:0/2816138167 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f68267e3e0 0x55f682638910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:28.192 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.190+0000 7f09b83a9740 1 --2- 172.21.15.96:0/2816138167 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f6826659b0 0x55f6826335b0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:28.192 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.190+0000 7f09b83a9740 1 --2- 172.21.15.96:0/2816138167 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x55f682661f60 0x55f682661940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:28.192 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.190+0000 7f09b83a9740 1 -- 172.21.15.96:0/2816138167 >> 172.21.15.96:0/2816138167 conn(0x55f68265e2a0 msgr2=0x55f6826642a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:28.192 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.190+0000 7f09b83a9740 1 -- 172.21.15.96:0/2816138167 shutdown_connections 2024-05-28T06:04:28.193 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.190+0000 7f09b83a9740 1 -- 172.21.15.96:0/2816138167 wait complete. 2024-05-28T06:04:28.195 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.190+0000 7f09b83a9740 1 -- start start 2024-05-28T06:04:28.195 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.191+0000 7f09b83a9740 1 Processor -- start 2024-05-28T06:04:28.195 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.191+0000 7f09b83a9740 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x55f682661f60 0x55f6826fe740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:28.195 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.192+0000 7f09b83a9740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f6826659b0 0x55f68267a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:28.195 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.192+0000 7f09b83a9740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f68267e3e0 0x55f6826763d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:28.195 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.192+0000 7f09b83a9740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55f6826320b0 con 0x55f68267e3e0 2024-05-28T06:04:28.196 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.192+0000 7f09b83a9740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55f682555570 con 0x55f6826659b0 2024-05-28T06:04:28.196 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.192+0000 7f09b83a9740 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x55f682662f10 con 0x55f682661f60 2024-05-28T06:04:28.196 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.192+0000 7f09b83a9740 10 client.0 _ll_register_callbacks cb 0x7f09a8007640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-28T06:04:28.196 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.192+0000 7f09af200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f68267e3e0 0x55f6826763d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:28.196 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.192+0000 7f09af200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f68267e3e0 0x55f6826763d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48436/0 (socket says 172.21.15.96:48436) 2024-05-28T06:04:28.196 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.192+0000 7f09af200640 1 -- 172.21.15.96:0/1573091984 learned_addr learned my addr 172.21.15.96:0/1573091984 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:28.196 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.192+0000 7f09ae800640 1 --2- 172.21.15.96:0/1573091984 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x55f682661f60 0x55f6826fe740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:28.196 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.192+0000 7f09ade00640 1 --2- 172.21.15.96:0/1573091984 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f6826659b0 0x55f68267a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:28.196 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.192+0000 7f09af200640 1 -- 172.21.15.96:0/1573091984 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x55f682661f60 msgr2=0x55f6826fe740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:28.196 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.192+0000 7f09af200640 1 --2- 172.21.15.96:0/1573091984 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x55f682661f60 0x55f6826fe740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:28.196 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.192+0000 7f09af200640 1 -- 172.21.15.96:0/1573091984 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f6826659b0 msgr2=0x55f68267a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:28.196 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.192+0000 7f09af200640 1 --2- 172.21.15.96:0/1573091984 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f6826659b0 0x55f68267a9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:28.196 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.192+0000 7f09af200640 1 -- 172.21.15.96:0/1573091984 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f6826a0800 con 0x55f68267e3e0 2024-05-28T06:04:28.197 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.192+0000 7f09af200640 1 --2- 172.21.15.96:0/1573091984 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f68267e3e0 0x55f6826763d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f09a802a6a0 tx=0x7f09a8033040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f6133450f572d7f server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:28.197 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.192+0000 7f098f400640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-05-28T06:04:28.197 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.193+0000 7f098f400640 1 -- 172.21.15.96:0/1573091984 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a8007da0 con 0x55f68267e3e0 2024-05-28T06:04:28.197 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.193+0000 7f09b83a9740 1 -- 172.21.15.96:0/1573091984 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f099c026070 con 0x55f68267e3e0 2024-05-28T06:04:28.197 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.193+0000 7f098f400640 1 -- 172.21.15.96:0/1573091984 <== mon.0 v2:172.21.15.96:3300/0 2 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f09a8004250 con 0x55f68267e3e0 2024-05-28T06:04:28.197 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.193+0000 7f098f400640 1 -- 172.21.15.96:0/1573091984 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) ==== 24+0+0 (secure 0 0 0) 0x7f09a8067050 con 0x55f68267e3e0 2024-05-28T06:04:28.197 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.193+0000 7f09b83a9740 10 client.4653 fetch_fsmap learned FSMap version 7 2024-05-28T06:04:28.197 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.193+0000 7f09b83a9740 1 -- 172.21.15.96:0/1573091984 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) -- 0x55f682683190 con 0x55f68267e3e0 2024-05-28T06:04:28.197 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.193+0000 7f098f400640 1 -- 172.21.15.96:0/1573091984 <== mon.0 v2:172.21.15.96:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f099c026070 con 0x55f68267e3e0 2024-05-28T06:04:28.197 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.193+0000 7f098f400640 1 -- 172.21.15.96:0/1573091984 <== mon.0 v2:172.21.15.96:3300/0 5 ==== fsmap(e 7) ==== 3494+0+0 (secure 0 0 0) 0x7f09a805a030 con 0x55f68267e3e0 2024-05-28T06:04:28.197 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.193+0000 7f09b83a9740 10 client.4653 fetch_fsmap finished waiting for FSMap version 7 2024-05-28T06:04:28.197 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.193+0000 7f09b83a9740 10 client.4653 resolve_mds: resolved name 'b' to daemon mds.b 2024-05-28T06:04:28.197 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.193+0000 7f09b83a9740 20 client.4653 populate_metadata read hostname 'smithi096' 2024-05-28T06:04:28.198 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.193+0000 7f09b83a9740 1 --2- 172.21.15.96:0/1573091984 >> [v2:172.21.15.123:6834/605814935,v1:172.21.15.123:6835/605814935] conn(0x55f68268eaf0 0x55f682678ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:28.198 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.193+0000 7f09b83a9740 4 client.4653 mds_command: new command op to 4281 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-28T06:04:28.198 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.193+0000 7f09b83a9740 1 -- 172.21.15.96:0/1573091984 --> [v2:172.21.15.123:6834/605814935,v1:172.21.15.123:6835/605814935] -- command(tid 0: {"prefix": "status"}) -- 0x55f6825a33a0 con 0x55f68268eaf0 2024-05-28T06:04:28.198 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.194+0000 7f09ae800640 1 --2- 172.21.15.96:0/1573091984 >> [v2:172.21.15.123:6834/605814935,v1:172.21.15.123:6835/605814935] conn(0x55f68268eaf0 0x55f682678ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:28.199 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.195+0000 7f09ae800640 1 --2- 172.21.15.96:0/1573091984 >> [v2:172.21.15.123:6834/605814935,v1:172.21.15.123:6835/605814935] conn(0x55f68268eaf0 0x55f682678ff0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d1007c73a9d67ac2 server_cookie=361befb3efbd20d5 in_seq=0 out_seq=0 2024-05-28T06:04:28.199 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.195+0000 7f098f400640 10 client.4653 ms_handle_connect on v2:172.21.15.123:6834/605814935 2024-05-28T06:04:28.199 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.195+0000 7f098f400640 1 -- 172.21.15.96:0/1573091984 <== mds.0 v2:172.21.15.123:6834/605814935 1 ==== command_reply(tid 0: 0 ) ==== 8+0+330 (crc 0 0 0) 0x55f6825a33a0 con 0x55f68268eaf0 2024-05-28T06:04:28.199 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.195+0000 7f098f400640 10 client.4653 handle_command_reply: tid=0 2024-05-28T06:04:28.199 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.195+0000 7f09b83a9740 1 client.4653 shutdown 2024-05-28T06:04:28.199 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.196+0000 7f09b83a9740 1 -- 172.21.15.96:0/1573091984 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f68267e3e0 msgr2=0x55f6826763d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:28.199 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.196+0000 7f09b83a9740 1 --2- 172.21.15.96:0/1573091984 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f68267e3e0 0x55f6826763d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f09a802a6a0 tx=0x7f09a8033040 comp rx=0 tx=0).stop 2024-05-28T06:04:28.200 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.196+0000 7f09b83a9740 1 -- 172.21.15.96:0/1573091984 shutdown_connections 2024-05-28T06:04:28.200 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.196+0000 7f09b83a9740 1 --2- 172.21.15.96:0/1573091984 >> [v2:172.21.15.123:6834/605814935,v1:172.21.15.123:6835/605814935] conn(0x55f68268eaf0 0x55f682678ff0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:28.200 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.196+0000 7f09b83a9740 1 --2- 172.21.15.96:0/1573091984 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f68267e3e0 0x55f6826763d0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:28.200 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.196+0000 7f09b83a9740 1 --2- 172.21.15.96:0/1573091984 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f6826659b0 0x55f68267a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:28.200 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.196+0000 7f09b83a9740 1 --2- 172.21.15.96:0/1573091984 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x55f682661f60 0x55f6826fe740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:28.200 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.196+0000 7f09b83a9740 1 -- 172.21.15.96:0/1573091984 >> 172.21.15.96:0/1573091984 conn(0x55f68265e2a0 msgr2=0x55f682662340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:28.200 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.196+0000 7f098f400640 0 client.4653 ms_handle_reset on v2:172.21.15.123:6834/605814935 2024-05-28T06:04:28.200 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.196+0000 7f09b83a9740 1 -- 172.21.15.96:0/1573091984 shutdown_connections 2024-05-28T06:04:28.200 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.196+0000 7f09b83a9740 1 -- 172.21.15.96:0/1573091984 wait complete. 2024-05-28T06:04:28.200 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.196+0000 7f09b83a9740 20 client.4653 trim_cache size 0 max 16384 2024-05-28T06:04:28.200 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:04:28.201 INFO:teuthology.orchestra.run.smithi096.stdout: "cluster_fsid": "e4a831b5-e6d1-47a7-96cb-bf85154ee0d2", 2024-05-28T06:04:28.201 INFO:teuthology.orchestra.run.smithi096.stdout: "whoami": 0, 2024-05-28T06:04:28.201 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 4281, 2024-05-28T06:04:28.201 INFO:teuthology.orchestra.run.smithi096.stdout: "want_state": "up:active", 2024-05-28T06:04:28.201 INFO:teuthology.orchestra.run.smithi096.stdout: "state": "up:active", 2024-05-28T06:04:28.201 INFO:teuthology.orchestra.run.smithi096.stdout: "fs_name": "cephfs", 2024-05-28T06:04:28.201 INFO:teuthology.orchestra.run.smithi096.stdout: "rank_uptime": 17.308209987000001, 2024-05-28T06:04:28.201 INFO:teuthology.orchestra.run.smithi096.stdout: "mdsmap_epoch": 7, 2024-05-28T06:04:28.201 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap_epoch": 20, 2024-05-28T06:04:28.201 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap_epoch_barrier": 20, 2024-05-28T06:04:28.201 INFO:teuthology.orchestra.run.smithi096.stdout: "uptime": 22.360282601000002 2024-05-28T06:04:28.201 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:04:28.202 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.196+0000 7f09b5400640 1 -- 172.21.15.96:0/290539950 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f099006c8f0 msgr2=0x7f099006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:28.202 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.196+0000 7f09b5400640 1 --2- 172.21.15.96:0/290539950 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f099006c8f0 0x7f099006edb0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f09b0159c20 tx=0x7f099c009040 comp rx=0 tx=0).stop 2024-05-28T06:04:28.202 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.196+0000 7f09b5400640 1 -- 172.21.15.96:0/290539950 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f09b006f860 msgr2=0x7f09b01596e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:28.202 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.196+0000 7f09b5400640 1 --2- 172.21.15.96:0/290539950 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f09b006f860 0x7f09b01596e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f09a000e9d0 tx=0x7f09a000ee90 comp rx=0 tx=0).stop 2024-05-28T06:04:28.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.197+0000 7f09b5400640 1 -- 172.21.15.96:0/290539950 shutdown_connections 2024-05-28T06:04:28.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.197+0000 7f09b5400640 1 --2- 172.21.15.96:0/290539950 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f099006c8f0 0x7f099006edb0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:28.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.197+0000 7f09b5400640 1 --2- 172.21.15.96:0/290539950 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09b00d0bb0 0x7f09b00c7330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:28.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.197+0000 7f09b5400640 1 --2- 172.21.15.96:0/290539950 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f09b00b3690 0x7f09b01f7270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:28.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.197+0000 7f09b5400640 1 --2- 172.21.15.96:0/290539950 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f09b006f860 0x7f09b01596e0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:28.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.197+0000 7f09b5400640 1 -- 172.21.15.96:0/290539950 >> 172.21.15.96:0/290539950 conn(0x7f09b006bae0 msgr2=0x7f09b0158810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:28.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.197+0000 7f09b5400640 1 -- 172.21.15.96:0/290539950 shutdown_connections 2024-05-28T06:04:28.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.197+0000 7f09b5400640 1 -- 172.21.15.96:0/290539950 wait complete. 2024-05-28T06:04:28.215 INFO:teuthology.run_tasks:Running task sequential... 2024-05-28T06:04:28.229 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-05-28T06:04:28.229 INFO:teuthology.task.sequential:In sequential, running task print... 2024-05-28T06:04:28.229 INFO:teuthology.task.print:Enabling mgr modules 2024-05-28T06:04:28.230 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-05-28T06:04:28.278 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-05-28T06:04:28.278 INFO:tasks.ceph_fuse:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2024-05-28T06:04:28.279 INFO:tasks.ceph_fuse:client.0 config is {} 2024-05-28T06:04:28.279 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-05-28T06:04:28.279 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-28T06:04:28.279 INFO:tasks.ceph_fuse:client.2 config is {} 2024-05-28T06:04:28.279 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-05-28T06:04:28.279 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-05-28T06:04:28.279 INFO:tasks.ceph_fuse:client.1 config is {} 2024-05-28T06:04:28.279 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-05-28T06:04:28.279 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-05-28T06:04:28.280 INFO:tasks.ceph_fuse:client.3 config is {} 2024-05-28T06:04:28.280 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-05-28T06:04:28.280 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-05-28T06:04:28.280 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:04:28.280 DEBUG:teuthology.orchestra.run.smithi123:> ip netns list 2024-05-28T06:04:28.340 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:04:28.340 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip link delete ceph-brx 2024-05-28T06:04:28.425 INFO:teuthology.orchestra.run.smithi123.stderr:Cannot find device "ceph-brx" 2024-05-28T06:04:28.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-28T06:04:28.430 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:04:28.430 DEBUG:teuthology.orchestra.run.smithi096:> ip netns list 2024-05-28T06:04:28.487 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:04:28.487 DEBUG:teuthology.orchestra.run.smithi096:> sudo ip link delete ceph-brx 2024-05-28T06:04:28.571 INFO:teuthology.orchestra.run.smithi096.stderr:Cannot find device "ceph-brx" 2024-05-28T06:04:28.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-28T06:04:28.574 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-05-28T06:04:28.575 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-05-28T06:04:28.743 DEBUG:teuthology.orchestra.run.smithi096:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:04:28.751 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:04:28.770 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.769+0000 7f71e3800640 1 Processor -- start 2024-05-28T06:04:28.770 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.769+0000 7f71e3800640 1 -- start start 2024-05-28T06:04:28.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.770+0000 7f71e3800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f71dc143cf0 0x7f71dc06c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:28.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.770+0000 7f71e3800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f71dc0b4920 0x7f71dc06ce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:28.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.770+0000 7f71e3800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f71dc15e800 0x7f71dc15ec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:28.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.770+0000 7f71e3800640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f71dc0b3b80 con 0x7f71dc143cf0 2024-05-28T06:04:28.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.770+0000 7f71e3800640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f71dc15f570 con 0x7f71dc15e800 2024-05-28T06:04:28.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.770+0000 7f71e3800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f71dc0b3e40 con 0x7f71dc0b4920 2024-05-28T06:04:28.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.770+0000 7f71e0e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f71dc143cf0 0x7f71dc06c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:28.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.770+0000 7f71e0e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f71dc143cf0 0x7f71dc06c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48442/0 (socket says 172.21.15.96:48442) 2024-05-28T06:04:28.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.770+0000 7f71e0e00640 1 -- 172.21.15.96:0/2648164054 learned_addr learned my addr 172.21.15.96:0/2648164054 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:28.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.771+0000 7f71d3e00640 1 --2- 172.21.15.96:0/2648164054 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f71dc0b4920 0x7f71dc06ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:28.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.771+0000 7f71e1800640 1 --2- 172.21.15.96:0/2648164054 >> v2:172.21.15.123:3300/0 conn(0x7f71dc15e800 0x7f71dc15ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:28.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.771+0000 7f71e0e00640 1 -- 172.21.15.96:0/2648164054 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f71dc0b4920 msgr2=0x7f71dc06ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:28.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.771+0000 7f71e0e00640 1 --2- 172.21.15.96:0/2648164054 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f71dc0b4920 0x7f71dc06ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:28.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.771+0000 7f71e0e00640 1 -- 172.21.15.96:0/2648164054 >> v2:172.21.15.123:3300/0 conn(0x7f71dc15e800 msgr2=0x7f71dc15ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:28.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.771+0000 7f71e0e00640 1 --2- 172.21.15.96:0/2648164054 >> v2:172.21.15.123:3300/0 conn(0x7f71dc15e800 0x7f71dc15ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:28.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.771+0000 7f71e0e00640 1 -- 172.21.15.96:0/2648164054 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71dc0b4120 con 0x7f71dc143cf0 2024-05-28T06:04:28.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.771+0000 7f71d3e00640 1 --2- 172.21.15.96:0/2648164054 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f71dc0b4920 0x7f71dc06ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-28T06:04:28.775 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.772+0000 7f71e0e00640 1 --2- 172.21.15.96:0/2648164054 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f71dc143cf0 0x7f71dc06c910 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f71dc072340 tx=0x7f71cc02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ee8e15a1aa4e235 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:28.775 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.772+0000 7f71dbe00640 1 -- 172.21.15.96:0/2648164054 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71cc037070 con 0x7f71dc143cf0 2024-05-28T06:04:28.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.772+0000 7f71dbe00640 1 -- 172.21.15.96:0/2648164054 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71cc032440 con 0x7f71dc143cf0 2024-05-28T06:04:28.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.772+0000 7f71dbe00640 1 -- 172.21.15.96:0/2648164054 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71cc032810 con 0x7f71dc143cf0 2024-05-28T06:04:28.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.773+0000 7f71e3800640 1 -- 172.21.15.96:0/2648164054 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f71dc143cf0 msgr2=0x7f71dc06c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:28.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.773+0000 7f71e3800640 1 --2- 172.21.15.96:0/2648164054 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f71dc143cf0 0x7f71dc06c910 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f71dc072340 tx=0x7f71cc02a1e0 comp rx=0 tx=0).stop 2024-05-28T06:04:28.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.773+0000 7f71e3800640 1 -- 172.21.15.96:0/2648164054 shutdown_connections 2024-05-28T06:04:28.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.773+0000 7f71e3800640 1 --2- 172.21.15.96:0/2648164054 >> v2:172.21.15.123:3300/0 conn(0x7f71dc15e800 0x7f71dc15ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:28.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.773+0000 7f71e3800640 1 --2- 172.21.15.96:0/2648164054 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f71dc0b4920 0x7f71dc06ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:28.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.773+0000 7f71e3800640 1 --2- 172.21.15.96:0/2648164054 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f71dc143cf0 0x7f71dc06c910 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:28.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.773+0000 7f71e3800640 1 -- 172.21.15.96:0/2648164054 >> 172.21.15.96:0/2648164054 conn(0x7f71dc0724f0 msgr2=0x7f71dc0728d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:28.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.773+0000 7f71e3800640 1 -- 172.21.15.96:0/2648164054 shutdown_connections 2024-05-28T06:04:28.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.773+0000 7f71e3800640 1 -- 172.21.15.96:0/2648164054 wait complete. 2024-05-28T06:04:28.777 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.775+0000 7f71e3800640 1 Processor -- start 2024-05-28T06:04:28.777 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.775+0000 7f71e3800640 1 -- start start 2024-05-28T06:04:28.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.776+0000 7f71e3800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f71dc0b4920 0x7f71dc140350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:28.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.776+0000 7f71e3800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f71dc143cf0 0x7f71dc15cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:28.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.776+0000 7f71e3800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f71dc15e800 0x7f71dc159170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:28.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.776+0000 7f71e3800640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f71dc15f350 con 0x7f71dc143cf0 2024-05-28T06:04:28.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.776+0000 7f71e3800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f71dc15f1d0 con 0x7f71dc0b4920 2024-05-28T06:04:28.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.776+0000 7f71e3800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f71dc0676b0 con 0x7f71dc15e800 2024-05-28T06:04:28.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.776+0000 7f71d3e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f71dc143cf0 0x7f71dc15cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:28.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.776+0000 7f71d3e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f71dc143cf0 0x7f71dc15cc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48446/0 (socket says 172.21.15.96:48446) 2024-05-28T06:04:28.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.776+0000 7f71d3e00640 1 -- 172.21.15.96:0/3729673858 learned_addr learned my addr 172.21.15.96:0/3729673858 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:28.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.777+0000 7f71e1800640 1 --2- 172.21.15.96:0/3729673858 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f71dc15e800 0x7f71dc159170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:28.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.777+0000 7f71e0e00640 1 --2- 172.21.15.96:0/3729673858 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f71dc0b4920 0x7f71dc140350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:28.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.777+0000 7f71d3e00640 1 -- 172.21.15.96:0/3729673858 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f71dc15e800 msgr2=0x7f71dc159170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:28.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.777+0000 7f71d3e00640 1 --2- 172.21.15.96:0/3729673858 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f71dc15e800 0x7f71dc159170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:28.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.777+0000 7f71d3e00640 1 -- 172.21.15.96:0/3729673858 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f71dc0b4920 msgr2=0x7f71dc140350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:28.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.777+0000 7f71d3e00640 1 --2- 172.21.15.96:0/3729673858 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f71dc0b4920 0x7f71dc140350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:28.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.777+0000 7f71d3e00640 1 -- 172.21.15.96:0/3729673858 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71dc13e6e0 con 0x7f71dc143cf0 2024-05-28T06:04:28.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.777+0000 7f71e1800640 1 --2- 172.21.15.96:0/3729673858 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f71dc15e800 0x7f71dc159170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-28T06:04:28.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.777+0000 7f71d3e00640 1 --2- 172.21.15.96:0/3729673858 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f71dc143cf0 0x7f71dc15cc70 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f71c400e730 tx=0x7f71c400ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:28.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.777+0000 7f71d9600640 1 -- 172.21.15.96:0/3729673858 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71c400f040 con 0x7f71dc143cf0 2024-05-28T06:04:28.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.777+0000 7f71d9600640 1 -- 172.21.15.96:0/3729673858 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71c4004230 con 0x7f71dc143cf0 2024-05-28T06:04:28.781 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.777+0000 7f71e3800640 1 -- 172.21.15.96:0/3729673858 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71dc13e9c0 con 0x7f71dc143cf0 2024-05-28T06:04:28.781 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.777+0000 7f71d9600640 1 -- 172.21.15.96:0/3729673858 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71c400ee40 con 0x7f71dc143cf0 2024-05-28T06:04:28.781 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.778+0000 7f71e3800640 1 -- 172.21.15.96:0/3729673858 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71dc0ca110 con 0x7f71dc143cf0 2024-05-28T06:04:28.781 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.779+0000 7f71e3800640 1 -- 172.21.15.96:0/3729673858 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71dc13eee0 con 0x7f71dc143cf0 2024-05-28T06:04:28.781 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.780+0000 7f71d9600640 1 -- 172.21.15.96:0/3729673858 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f71c400c6d0 con 0x7f71dc143cf0 2024-05-28T06:04:28.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.781+0000 7f71d9600640 1 --2- 172.21.15.96:0/3729673858 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f71b806c8f0 0x7f71b806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:28.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.781+0000 7f71d9600640 1 -- 172.21.15.96:0/3729673858 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f71c408b900 con 0x7f71dc143cf0 2024-05-28T06:04:28.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.781+0000 7f71e0e00640 1 --2- 172.21.15.96:0/3729673858 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f71b806c8f0 0x7f71b806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:28.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.782+0000 7f71e0e00640 1 --2- 172.21.15.96:0/3729673858 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f71b806c8f0 0x7f71b806edb0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f71cc034000 tx=0x7f71cc02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:28.786 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:28.785+0000 7f71d9600640 1 -- 172.21.15.96:0/3729673858 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f71c4017020 con 0x7f71dc143cf0 2024-05-28T06:04:29.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:29.026+0000 7f71e3800640 1 -- 172.21.15.96:0/3729673858 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f71dc0b5200 con 0x7f71dc143cf0 2024-05-28T06:04:29.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:29.027+0000 7f71d9600640 1 -- 172.21.15.96:0/3729673858 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) ==== 53+0+73 (secure 0 0 0) 0x7f71c4059d30 con 0x7f71dc143cf0 2024-05-28T06:04:29.028 INFO:teuthology.orchestra.run.smithi096.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-05-28T06:04:29.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:29.029+0000 7f71e3800640 1 -- 172.21.15.96:0/3729673858 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f71b806c8f0 msgr2=0x7f71b806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:29.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:29.029+0000 7f71e3800640 1 --2- 172.21.15.96:0/3729673858 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f71b806c8f0 0x7f71b806edb0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f71cc034000 tx=0x7f71cc02e000 comp rx=0 tx=0).stop 2024-05-28T06:04:29.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:29.029+0000 7f71e3800640 1 -- 172.21.15.96:0/3729673858 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f71dc143cf0 msgr2=0x7f71dc15cc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:29.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:29.029+0000 7f71e3800640 1 --2- 172.21.15.96:0/3729673858 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f71dc143cf0 0x7f71dc15cc70 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f71c400e730 tx=0x7f71c400ebf0 comp rx=0 tx=0).stop 2024-05-28T06:04:29.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:29.030+0000 7f71e3800640 1 -- 172.21.15.96:0/3729673858 shutdown_connections 2024-05-28T06:04:29.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:29.030+0000 7f71e3800640 1 --2- 172.21.15.96:0/3729673858 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f71b806c8f0 0x7f71b806edb0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:29.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:29.030+0000 7f71e3800640 1 --2- 172.21.15.96:0/3729673858 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f71dc15e800 0x7f71dc159170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:29.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:29.030+0000 7f71e3800640 1 --2- 172.21.15.96:0/3729673858 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f71dc143cf0 0x7f71dc15cc70 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:29.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:29.030+0000 7f71e3800640 1 --2- 172.21.15.96:0/3729673858 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f71dc0b4920 0x7f71dc140350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:29.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:29.030+0000 7f71e3800640 1 -- 172.21.15.96:0/3729673858 >> 172.21.15.96:0/3729673858 conn(0x7f71dc0724f0 msgr2=0x7f71dc0728d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:29.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:29.030+0000 7f71e3800640 1 -- 172.21.15.96:0/3729673858 shutdown_connections 2024-05-28T06:04:29.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:29.030+0000 7f71e3800640 1 -- 172.21.15.96:0/3729673858 wait complete. 2024-05-28T06:04:29.050 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-05-28T06:04:29.051 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-28T06:04:29.051 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi096.front.sepia.ceph.com 2024-05-28T06:04:29.051 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-05-28T06:04:29.051 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-28T06:04:29.051 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-05-28T06:04:29.051 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-05-28T06:04:29.051 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-28T06:04:29.051 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-28T06:04:29.051 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:04:29.051 DEBUG:teuthology.orchestra.run.smithi096:> ip addr 2024-05-28T06:04:29.101 INFO:teuthology.orchestra.run.smithi096.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-28T06:04:29.101 INFO:teuthology.orchestra.run.smithi096.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-28T06:04:29.101 INFO:teuthology.orchestra.run.smithi096.stdout: inet 127.0.0.1/8 scope host lo 2024-05-28T06:04:29.102 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:04:29.102 INFO:teuthology.orchestra.run.smithi096.stdout: inet6 ::1/128 scope host 2024-05-28T06:04:29.102 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:04:29.102 INFO:teuthology.orchestra.run.smithi096.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-28T06:04:29.102 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether 0c:c4:7a:d8:d9:bc brd ff:ff:ff:ff:ff:ff 2024-05-28T06:04:29.102 INFO:teuthology.orchestra.run.smithi096.stdout: altname enp7s0f0 2024-05-28T06:04:29.102 INFO:teuthology.orchestra.run.smithi096.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-28T06:04:29.102 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether 0c:c4:7a:d8:d9:bd brd ff:ff:ff:ff:ff:ff 2024-05-28T06:04:29.102 INFO:teuthology.orchestra.run.smithi096.stdout: altname enp7s0f1 2024-05-28T06:04:29.102 INFO:teuthology.orchestra.run.smithi096.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-28T06:04:29.102 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether 0c:c4:7a:8f:cc:0c brd ff:ff:ff:ff:ff:ff 2024-05-28T06:04:29.102 INFO:teuthology.orchestra.run.smithi096.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-28T06:04:29.102 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether 0c:c4:7a:8f:cc:0d brd ff:ff:ff:ff:ff:ff 2024-05-28T06:04:29.103 INFO:teuthology.orchestra.run.smithi096.stdout: inet 172.21.15.96/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-05-28T06:04:29.103 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft 42568sec preferred_lft 42568sec 2024-05-28T06:04:29.103 INFO:teuthology.orchestra.run.smithi096.stdout: inet6 fe80::ec4:7aff:fe8f:cc0d/64 scope link 2024-05-28T06:04:29.103 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:04:29.104 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-05-28T06:04:29.104 DEBUG:teuthology.orchestra.run.smithi096:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-28T06:04:29.104 DEBUG:teuthology.orchestra.run.smithi096:> set -e 2024-05-28T06:04:29.104 DEBUG:teuthology.orchestra.run.smithi096:> sudo ip link add name ceph-brx type bridge 2024-05-28T06:04:29.104 DEBUG:teuthology.orchestra.run.smithi096:> sudo ip addr flush dev ceph-brx 2024-05-28T06:04:29.104 DEBUG:teuthology.orchestra.run.smithi096:> sudo ip link set ceph-brx up 2024-05-28T06:04:29.104 DEBUG:teuthology.orchestra.run.smithi096:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-05-28T06:04:29.105 DEBUG:teuthology.orchestra.run.smithi096:> ') 2024-05-28T06:04:29.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:29 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-28T06:04:29.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-28T06:04:29.585 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:04:29.585 DEBUG:teuthology.orchestra.run.smithi096:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-28T06:04:29.648 INFO:teuthology.orchestra.run.smithi096.stdout:1 2024-05-28T06:04:29.651 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:04:29.651 DEBUG:teuthology.orchestra.run.smithi096:> route 2024-05-28T06:04:29.705 INFO:teuthology.orchestra.run.smithi096.stdout:Kernel IP routing table 2024-05-28T06:04:29.705 INFO:teuthology.orchestra.run.smithi096.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-28T06:04:29.705 INFO:teuthology.orchestra.run.smithi096.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-05-28T06:04:29.705 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-05-28T06:04:29.705 INFO:teuthology.orchestra.run.smithi096.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-28T06:04:29.706 DEBUG:teuthology.orchestra.run.smithi096:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-28T06:04:29.706 DEBUG:teuthology.orchestra.run.smithi096:> set -e 2024-05-28T06:04:29.706 DEBUG:teuthology.orchestra.run.smithi096:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-05-28T06:04:29.706 DEBUG:teuthology.orchestra.run.smithi096:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-05-28T06:04:29.706 DEBUG:teuthology.orchestra.run.smithi096:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-05-28T06:04:29.706 DEBUG:teuthology.orchestra.run.smithi096:> ') 2024-05-28T06:04:29.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:29 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-28T06:04:30.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:30 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-28T06:04:30.451 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:04:30.452 DEBUG:teuthology.orchestra.run.smithi096:> ip netns list 2024-05-28T06:04:30.504 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:04:30.505 DEBUG:teuthology.orchestra.run.smithi096:> ip netns list-id 2024-05-28T06:04:30.557 DEBUG:teuthology.orchestra.run.smithi096:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-28T06:04:30.557 DEBUG:teuthology.orchestra.run.smithi096:> set -e 2024-05-28T06:04:30.557 DEBUG:teuthology.orchestra.run.smithi096:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-28T06:04:30.557 DEBUG:teuthology.orchestra.run.smithi096:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-05-28T06:04:30.557 DEBUG:teuthology.orchestra.run.smithi096:> ') 2024-05-28T06:04:30.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:30 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-28T06:04:30.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:30 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-28T06:04:30.745 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-05-28T06:04:30.745 DEBUG:teuthology.orchestra.run.smithi096:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-28T06:04:30.745 DEBUG:teuthology.orchestra.run.smithi096:> set -e 2024-05-28T06:04:30.746 DEBUG:teuthology.orchestra.run.smithi096:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-05-28T06:04:30.746 DEBUG:teuthology.orchestra.run.smithi096:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-05-28T06:04:30.746 DEBUG:teuthology.orchestra.run.smithi096:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-05-28T06:04:30.746 DEBUG:teuthology.orchestra.run.smithi096:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-05-28T06:04:30.746 DEBUG:teuthology.orchestra.run.smithi096:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-05-28T06:04:30.746 DEBUG:teuthology.orchestra.run.smithi096:> ') 2024-05-28T06:04:30.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:30 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-28T06:04:31.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:31 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-28T06:04:31.042 DEBUG:teuthology.orchestra.run.smithi096:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-28T06:04:31.042 DEBUG:teuthology.orchestra.run.smithi096:> set -e 2024-05-28T06:04:31.042 DEBUG:teuthology.orchestra.run.smithi096:> sudo ip link set brx.0 up 2024-05-28T06:04:31.042 DEBUG:teuthology.orchestra.run.smithi096:> sudo ip link set dev brx.0 master ceph-brx 2024-05-28T06:04:31.042 DEBUG:teuthology.orchestra.run.smithi096:> ') 2024-05-28T06:04:31.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:31 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-28T06:04:31.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:31 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-28T06:04:31.241 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-05-28T06:04:31.241 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-28T06:04:31.241 DEBUG:teuthology.orchestra.run.smithi096:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-05-28T06:04:31.286 INFO:teuthology.orchestra.run.smithi096.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-05-28T06:04:31.286 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-28T06:04:31.286 DEBUG:teuthology.orchestra.run.smithi096:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-05-28T06:04:31.379 DEBUG:teuthology.orchestra.run.smithi096:> sudo modprobe fuse 2024-05-28T06:04:31.454 DEBUG:teuthology.orchestra.run.smithi096:> cat /proc/self/mounts | awk '{print $2}' 2024-05-28T06:04:31.502 INFO:teuthology.orchestra.run.smithi096.stdout:/proc 2024-05-28T06:04:31.503 INFO:teuthology.orchestra.run.smithi096.stdout:/sys 2024-05-28T06:04:31.503 INFO:teuthology.orchestra.run.smithi096.stdout:/dev 2024-05-28T06:04:31.503 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/security 2024-05-28T06:04:31.503 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/shm 2024-05-28T06:04:31.503 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/pts 2024-05-28T06:04:31.503 INFO:teuthology.orchestra.run.smithi096.stdout:/run 2024-05-28T06:04:31.503 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/cgroup 2024-05-28T06:04:31.503 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/pstore 2024-05-28T06:04:31.503 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/bpf 2024-05-28T06:04:31.503 INFO:teuthology.orchestra.run.smithi096.stdout:/ 2024-05-28T06:04:31.503 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/selinux 2024-05-28T06:04:31.503 INFO:teuthology.orchestra.run.smithi096.stdout:/proc/sys/fs/binfmt_misc 2024-05-28T06:04:31.503 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/mqueue 2024-05-28T06:04:31.504 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/debug 2024-05-28T06:04:31.504 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/hugepages 2024-05-28T06:04:31.504 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/tracing 2024-05-28T06:04:31.504 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/fuse/connections 2024-05-28T06:04:31.504 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/config 2024-05-28T06:04:31.504 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-sysctl.service 2024-05-28T06:04:31.504 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-28T06:04:31.504 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-28T06:04:31.504 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/nfs/rpc_pipefs 2024-05-28T06:04:31.504 INFO:teuthology.orchestra.run.smithi096.stdout:/run/user/1000 2024-05-28T06:04:31.504 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph 2024-05-28T06:04:31.504 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-0 2024-05-28T06:04:31.504 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-1 2024-05-28T06:04:31.504 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-2 2024-05-28T06:04:31.505 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-3 2024-05-28T06:04:31.505 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns 2024-05-28T06:04:31.505 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-28T06:04:31.505 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-28T06:04:31.506 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:04:31.507 DEBUG:teuthology.orchestra.run.smithi096:> ls /sys/fs/fuse/connections 2024-05-28T06:04:31.596 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-05-28T06:04:31.596 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-05-28T06:04:31.601 DEBUG:teuthology.orchestra.run.smithi096:> sudo modprobe fuse 2024-05-28T06:04:31.666 DEBUG:teuthology.orchestra.run.smithi096:> cat /proc/self/mounts | awk '{print $2}' 2024-05-28T06:04:31.708 INFO:teuthology.orchestra.run.smithi096.stdout:/proc 2024-05-28T06:04:31.709 INFO:teuthology.orchestra.run.smithi096.stdout:/sys 2024-05-28T06:04:31.709 INFO:teuthology.orchestra.run.smithi096.stdout:/dev 2024-05-28T06:04:31.709 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/security 2024-05-28T06:04:31.709 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/shm 2024-05-28T06:04:31.709 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/pts 2024-05-28T06:04:31.709 INFO:teuthology.orchestra.run.smithi096.stdout:/run 2024-05-28T06:04:31.709 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/cgroup 2024-05-28T06:04:31.709 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/pstore 2024-05-28T06:04:31.709 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/bpf 2024-05-28T06:04:31.709 INFO:teuthology.orchestra.run.smithi096.stdout:/ 2024-05-28T06:04:31.709 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/selinux 2024-05-28T06:04:31.709 INFO:teuthology.orchestra.run.smithi096.stdout:/proc/sys/fs/binfmt_misc 2024-05-28T06:04:31.709 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/mqueue 2024-05-28T06:04:31.710 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/debug 2024-05-28T06:04:31.710 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/hugepages 2024-05-28T06:04:31.710 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/tracing 2024-05-28T06:04:31.710 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/fuse/connections 2024-05-28T06:04:31.710 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/config 2024-05-28T06:04:31.710 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-sysctl.service 2024-05-28T06:04:31.710 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-28T06:04:31.710 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-28T06:04:31.710 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/nfs/rpc_pipefs 2024-05-28T06:04:31.710 INFO:teuthology.orchestra.run.smithi096.stdout:/run/user/1000 2024-05-28T06:04:31.710 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph 2024-05-28T06:04:31.710 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-0 2024-05-28T06:04:31.710 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-1 2024-05-28T06:04:31.711 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-2 2024-05-28T06:04:31.711 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-3 2024-05-28T06:04:31.711 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns 2024-05-28T06:04:31.711 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-28T06:04:31.711 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-28T06:04:31.712 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:04:31.712 DEBUG:teuthology.orchestra.run.smithi096:> ls /sys/fs/fuse/connections 2024-05-28T06:04:31.787 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi096.stderr:2024-05-28T06:04:31.786+0000 7ff2e5815480 -1 init, newargv = 0x5592e0ac7110 newargc=15 2024-05-28T06:04:31.787 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi096.stderr:ceph-fuse[45410]: starting ceph client 2024-05-28T06:04:31.800 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi096.stderr:ceph-fuse[45410]: starting fuse 2024-05-28T06:04:32.897 DEBUG:teuthology.orchestra.run.smithi096:> sudo modprobe fuse 2024-05-28T06:04:32.987 DEBUG:teuthology.orchestra.run.smithi096:> cat /proc/self/mounts | awk '{print $2}' 2024-05-28T06:04:33.043 INFO:teuthology.orchestra.run.smithi096.stdout:/proc 2024-05-28T06:04:33.043 INFO:teuthology.orchestra.run.smithi096.stdout:/sys 2024-05-28T06:04:33.044 INFO:teuthology.orchestra.run.smithi096.stdout:/dev 2024-05-28T06:04:33.044 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/security 2024-05-28T06:04:33.044 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/shm 2024-05-28T06:04:33.044 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/pts 2024-05-28T06:04:33.044 INFO:teuthology.orchestra.run.smithi096.stdout:/run 2024-05-28T06:04:33.044 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/cgroup 2024-05-28T06:04:33.044 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/pstore 2024-05-28T06:04:33.044 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/bpf 2024-05-28T06:04:33.044 INFO:teuthology.orchestra.run.smithi096.stdout:/ 2024-05-28T06:04:33.044 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/selinux 2024-05-28T06:04:33.044 INFO:teuthology.orchestra.run.smithi096.stdout:/proc/sys/fs/binfmt_misc 2024-05-28T06:04:33.044 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/mqueue 2024-05-28T06:04:33.044 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/debug 2024-05-28T06:04:33.045 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/hugepages 2024-05-28T06:04:33.045 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/tracing 2024-05-28T06:04:33.045 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/fuse/connections 2024-05-28T06:04:33.045 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/config 2024-05-28T06:04:33.045 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-sysctl.service 2024-05-28T06:04:33.045 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-28T06:04:33.045 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-28T06:04:33.045 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/nfs/rpc_pipefs 2024-05-28T06:04:33.045 INFO:teuthology.orchestra.run.smithi096.stdout:/run/user/1000 2024-05-28T06:04:33.045 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph 2024-05-28T06:04:33.045 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-0 2024-05-28T06:04:33.045 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-1 2024-05-28T06:04:33.045 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-2 2024-05-28T06:04:33.045 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-3 2024-05-28T06:04:33.046 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns 2024-05-28T06:04:33.046 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-28T06:04:33.046 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-28T06:04:33.046 INFO:teuthology.orchestra.run.smithi096.stdout:/home/ubuntu/cephtest/mnt.0 2024-05-28T06:04:33.047 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:04:33.047 DEBUG:teuthology.orchestra.run.smithi096:> ls /sys/fs/fuse/connections 2024-05-28T06:04:33.095 INFO:teuthology.orchestra.run.smithi096.stdout:40 2024-05-28T06:04:33.095 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40] 2024-05-28T06:04:33.095 DEBUG:teuthology.orchestra.run.smithi096:> sudo stdin-killer -- python3 -c ' 2024-05-28T06:04:33.095 DEBUG:teuthology.orchestra.run.smithi096:> import glob 2024-05-28T06:04:33.095 DEBUG:teuthology.orchestra.run.smithi096:> import re 2024-05-28T06:04:33.095 DEBUG:teuthology.orchestra.run.smithi096:> import os 2024-05-28T06:04:33.096 DEBUG:teuthology.orchestra.run.smithi096:> import subprocess 2024-05-28T06:04:33.096 DEBUG:teuthology.orchestra.run.smithi096:> 2024-05-28T06:04:33.096 DEBUG:teuthology.orchestra.run.smithi096:> def _find_admin_socket(client_name): 2024-05-28T06:04:33.096 DEBUG:teuthology.orchestra.run.smithi096:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-05-28T06:04:33.096 DEBUG:teuthology.orchestra.run.smithi096:> files = glob.glob(asok_path) 2024-05-28T06:04:33.096 DEBUG:teuthology.orchestra.run.smithi096:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-05-28T06:04:33.096 DEBUG:teuthology.orchestra.run.smithi096:> 2024-05-28T06:04:33.096 DEBUG:teuthology.orchestra.run.smithi096:> # Given a non-glob path, it better be there 2024-05-28T06:04:33.096 DEBUG:teuthology.orchestra.run.smithi096:> if "*" not in asok_path: 2024-05-28T06:04:33.096 DEBUG:teuthology.orchestra.run.smithi096:> assert(len(files) == 1) 2024-05-28T06:04:33.096 DEBUG:teuthology.orchestra.run.smithi096:> return files[0] 2024-05-28T06:04:33.096 DEBUG:teuthology.orchestra.run.smithi096:> 2024-05-28T06:04:33.096 DEBUG:teuthology.orchestra.run.smithi096:> for f in files: 2024-05-28T06:04:33.096 DEBUG:teuthology.orchestra.run.smithi096:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-05-28T06:04:33.096 DEBUG:teuthology.orchestra.run.smithi096:> if os.path.exists("/proc/{0}".format(pid)): 2024-05-28T06:04:33.097 DEBUG:teuthology.orchestra.run.smithi096:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-05-28T06:04:33.097 DEBUG:teuthology.orchestra.run.smithi096:> contents = proc_f.read() 2024-05-28T06:04:33.097 DEBUG:teuthology.orchestra.run.smithi096:> if mountpoint in contents: 2024-05-28T06:04:33.097 DEBUG:teuthology.orchestra.run.smithi096:> return f 2024-05-28T06:04:33.097 DEBUG:teuthology.orchestra.run.smithi096:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-05-28T06:04:33.097 DEBUG:teuthology.orchestra.run.smithi096:> 2024-05-28T06:04:33.097 DEBUG:teuthology.orchestra.run.smithi096:> print(_find_admin_socket("client.0")) 2024-05-28T06:04:33.097 DEBUG:teuthology.orchestra.run.smithi096:> ' 2024-05-28T06:04:33.256 INFO:teuthology.orchestra.run.smithi096.stdout:/var/run/ceph/ceph-client.0.45410.asok 2024-05-28T06:04:33.261 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-28T06:04:33.273 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.45410.asok 2024-05-28T06:04:33.274 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:04:33.274 DEBUG:teuthology.orchestra.run.smithi096:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.45410.asok status 2024-05-28T06:04:33.498 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:04:33.499 INFO:teuthology.orchestra.run.smithi096.stdout: "metadata": { 2024-05-28T06:04:33.499 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph_sha1": "9ed025fbf4406fb54980a0647506b15895b6d863", 2024-05-28T06:04:33.499 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph_version": "ceph version 19.0.0-3154-g9ed025fb (9ed025fbf4406fb54980a0647506b15895b6d863) squid (dev)", 2024-05-28T06:04:33.499 INFO:teuthology.orchestra.run.smithi096.stdout: "entity_id": "0", 2024-05-28T06:04:33.499 INFO:teuthology.orchestra.run.smithi096.stdout: "hostname": "smithi096", 2024-05-28T06:04:33.499 INFO:teuthology.orchestra.run.smithi096.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-05-28T06:04:33.499 INFO:teuthology.orchestra.run.smithi096.stdout: "pid": "45410", 2024-05-28T06:04:33.499 INFO:teuthology.orchestra.run.smithi096.stdout: "root": "/" 2024-05-28T06:04:33.499 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:04:33.499 INFO:teuthology.orchestra.run.smithi096.stdout: "dentry_count": 0, 2024-05-28T06:04:33.499 INFO:teuthology.orchestra.run.smithi096.stdout: "dentry_pinned_count": 0, 2024-05-28T06:04:33.499 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 4665, 2024-05-28T06:04:33.500 INFO:teuthology.orchestra.run.smithi096.stdout: "inst": { 2024-05-28T06:04:33.500 INFO:teuthology.orchestra.run.smithi096.stdout: "name": { 2024-05-28T06:04:33.500 INFO:teuthology.orchestra.run.smithi096.stdout: "type": "client", 2024-05-28T06:04:33.500 INFO:teuthology.orchestra.run.smithi096.stdout: "num": 4665 2024-05-28T06:04:33.500 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:04:33.500 INFO:teuthology.orchestra.run.smithi096.stdout: "addr": { 2024-05-28T06:04:33.500 INFO:teuthology.orchestra.run.smithi096.stdout: "type": "v1", 2024-05-28T06:04:33.500 INFO:teuthology.orchestra.run.smithi096.stdout: "addr": "192.168.0.1:0", 2024-05-28T06:04:33.500 INFO:teuthology.orchestra.run.smithi096.stdout: "nonce": 740079297 2024-05-28T06:04:33.500 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:04:33.500 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:04:33.500 INFO:teuthology.orchestra.run.smithi096.stdout: "addr": { 2024-05-28T06:04:33.500 INFO:teuthology.orchestra.run.smithi096.stdout: "type": "v1", 2024-05-28T06:04:33.501 INFO:teuthology.orchestra.run.smithi096.stdout: "addr": "192.168.0.1:0", 2024-05-28T06:04:33.501 INFO:teuthology.orchestra.run.smithi096.stdout: "nonce": 740079297 2024-05-28T06:04:33.501 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:04:33.501 INFO:teuthology.orchestra.run.smithi096.stdout: "inst_str": "client.4665 192.168.0.1:0/740079297", 2024-05-28T06:04:33.501 INFO:teuthology.orchestra.run.smithi096.stdout: "addr_str": "192.168.0.1:0/740079297", 2024-05-28T06:04:33.501 INFO:teuthology.orchestra.run.smithi096.stdout: "inode_count": 1, 2024-05-28T06:04:33.501 INFO:teuthology.orchestra.run.smithi096.stdout: "mds_epoch": 7, 2024-05-28T06:04:33.501 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_epoch": 20, 2024-05-28T06:04:33.501 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_epoch_barrier": 0, 2024-05-28T06:04:33.501 INFO:teuthology.orchestra.run.smithi096.stdout: "blocklisted": false, 2024-05-28T06:04:33.501 INFO:teuthology.orchestra.run.smithi096.stdout: "fs_name": "cephfs" 2024-05-28T06:04:33.501 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:04:33.513 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-05-28T06:04:33.716 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.714+0000 7f61efe00640 1 Processor -- start 2024-05-28T06:04:33.716 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.715+0000 7f61efe00640 1 -- start start 2024-05-28T06:04:33.718 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.716+0000 7f61efe00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f61f00d16a0 0x7f61f00cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:33.718 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.716+0000 7f61efe00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61f015ec50 con 0x7f61f00cf870 2024-05-28T06:04:33.718 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.716+0000 7f61efe00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61f00cff80 con 0x7f61f015e870 2024-05-28T06:04:33.718 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.716+0000 7f61efe00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f61f00d2150 con 0x7f61f00d16a0 2024-05-28T06:04:33.718 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.716+0000 7f61ef400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f61f00cf870 0x7f61f00d25e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:41016/0 (socket says 172.21.15.96:41016) 2024-05-28T06:04:33.718 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.717+0000 7f61ef400640 1 -- 172.21.15.96:0/3368656013 learned_addr learned my addr 172.21.15.96:0/3368656013 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:04:33.718 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.717+0000 7f61ee000640 1 --2- 172.21.15.96:0/3368656013 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f61f00d16a0 0x7f61f00cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:33.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.717+0000 7f61ed600640 1 -- 172.21.15.96:0/3368656013 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4131616330 0 0) 0x7f61f015ec50 con 0x7f61f00cf870 2024-05-28T06:04:33.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.717+0000 7f61ed600640 1 -- 172.21.15.96:0/3368656013 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f61d0003520 con 0x7f61f00cf870 2024-05-28T06:04:33.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.717+0000 7f61ed600640 1 -- 172.21.15.96:0/3368656013 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1178839416 0 0) 0x7f61f00cff80 con 0x7f61f015e870 2024-05-28T06:04:33.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.717+0000 7f61ed600640 1 -- 172.21.15.96:0/3368656013 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f61f015ec50 con 0x7f61f015e870 2024-05-28T06:04:33.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.717+0000 7f61ed600640 1 -- 172.21.15.96:0/3368656013 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2219613336 0 0) 0x7f61d0003520 con 0x7f61f00cf870 2024-05-28T06:04:33.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.717+0000 7f61ed600640 1 -- 172.21.15.96:0/3368656013 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f61f00d16a0 msgr2=0x7f61f00cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:33.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.717+0000 7f61ed600640 1 --2- 172.21.15.96:0/3368656013 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f61f00d16a0 0x7f61f00cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:33.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.717+0000 7f61ed600640 1 -- 172.21.15.96:0/3368656013 >> v1:172.21.15.123:6789/0 conn(0x7f61f015e870 legacy=0x7f61f00d0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:33.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.718+0000 7f61ed600640 1 -- 172.21.15.96:0/3368656013 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61f00d0260 con 0x7f61f00cf870 2024-05-28T06:04:33.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.718+0000 7f61ed600640 1 -- 172.21.15.96:0/3368656013 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f61e8001670 con 0x7f61f00cf870 2024-05-28T06:04:33.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.718+0000 7f61ed600640 1 -- 172.21.15.96:0/3368656013 >> v1:172.21.15.96:6789/0 conn(0x7f61f00cf870 legacy=0x7f61f00d25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:33.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.718+0000 7f61ed600640 1 --2- 172.21.15.96:0/3368656013 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61d0003b70 0x7f61d0003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:33.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.718+0000 7f61ed600640 1 --2- 172.21.15.96:0/3368656013 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f61f00d16a0 0x7f61d0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:33.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.718+0000 7f61ed600640 1 --2- 172.21.15.96:0/3368656013 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f61d0004d40 0x7f61d00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:33.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.718+0000 7f61ed600640 1 -- 172.21.15.96:0/3368656013 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f61f00d2150 con 0x7f61d0004d40 2024-05-28T06:04:33.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.718+0000 7f61ed600640 1 -- 172.21.15.96:0/3368656013 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f61d000ba70 con 0x7f61d0003b70 2024-05-28T06:04:33.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.718+0000 7f61ed600640 1 -- 172.21.15.96:0/3368656013 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f61d000bc70 con 0x7f61f00d16a0 2024-05-28T06:04:33.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.718+0000 7f61ef400640 1 --2- 172.21.15.96:0/3368656013 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f61d0004d40 0x7f61d00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:33.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.718+0000 7f61eea00640 1 --2- 172.21.15.96:0/3368656013 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61d0003b70 0x7f61d0003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:33.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.718+0000 7f61ee000640 1 --2- 172.21.15.96:0/3368656013 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f61f00d16a0 0x7f61d0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:33.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.718+0000 7f61ef400640 1 -- 172.21.15.96:0/3368656013 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f61f00d16a0 msgr2=0x7f61d0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:33.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.718+0000 7f61ef400640 1 --2- 172.21.15.96:0/3368656013 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f61f00d16a0 0x7f61d0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:33.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.718+0000 7f61ef400640 1 -- 172.21.15.96:0/3368656013 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61d0003b70 msgr2=0x7f61d0003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:33.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.718+0000 7f61ef400640 1 --2- 172.21.15.96:0/3368656013 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61d0003b70 0x7f61d0003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:33.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.718+0000 7f61ef400640 1 -- 172.21.15.96:0/3368656013 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61d0003520 con 0x7f61d0004d40 2024-05-28T06:04:33.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.718+0000 7f61ee000640 1 --2- 172.21.15.96:0/3368656013 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f61f00d16a0 0x7f61d0004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:04:33.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.719+0000 7f61ef400640 1 --2- 172.21.15.96:0/3368656013 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f61d0004d40 0x7f61d00051c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f61e800c0e0 tx=0x7f61d0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1650a867b9ba820 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:33.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.720+0000 7f61ed600640 1 -- 172.21.15.96:0/3368656013 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61e8007c40 con 0x7f61d0004d40 2024-05-28T06:04:33.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.720+0000 7f61ed600640 1 -- 172.21.15.96:0/3368656013 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61e8005ad0 con 0x7f61d0004d40 2024-05-28T06:04:33.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.720+0000 7f61ed600640 1 -- 172.21.15.96:0/3368656013 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61e8007070 con 0x7f61d0004d40 2024-05-28T06:04:33.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.720+0000 7f61efe00640 1 -- 172.21.15.96:0/3368656013 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f61d0004d40 msgr2=0x7f61d00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:33.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.720+0000 7f61efe00640 1 --2- 172.21.15.96:0/3368656013 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f61d0004d40 0x7f61d00051c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f61e800c0e0 tx=0x7f61d0005830 comp rx=0 tx=0).stop 2024-05-28T06:04:33.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.720+0000 7f61ef400640 1 -- 172.21.15.96:0/3368656013 reap_dead start 2024-05-28T06:04:33.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.720+0000 7f61efe00640 1 -- 172.21.15.96:0/3368656013 shutdown_connections 2024-05-28T06:04:33.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.720+0000 7f61efe00640 1 -- 172.21.15.96:0/3368656013 >> 172.21.15.96:0/3368656013 conn(0x7f61f008bfc0 msgr2=0x7f61f00d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:33.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.720+0000 7f61efe00640 1 -- 172.21.15.96:0/3368656013 shutdown_connections 2024-05-28T06:04:33.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.720+0000 7f61efe00640 1 -- 172.21.15.96:0/3368656013 wait complete. 2024-05-28T06:04:33.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.722+0000 7f61efe00640 1 Processor -- start 2024-05-28T06:04:33.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.722+0000 7f61efe00640 1 -- start start 2024-05-28T06:04:33.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.723+0000 7f61efe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f61d0004d40 0x7f61f0152a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:33.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.723+0000 7f61efe00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f61f015ad50 0x7f61f015b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:33.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.723+0000 7f61efe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61f00c9950 0x7f61f00c9dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:33.727 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.723+0000 7f61efe00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f61f00d0820 con 0x7f61d0004d40 2024-05-28T06:04:33.727 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.723+0000 7f61efe00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f61f00cfd20 con 0x7f61f00c9950 2024-05-28T06:04:33.727 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.723+0000 7f61efe00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f61f00d4fb0 con 0x7f61f015ad50 2024-05-28T06:04:33.727 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.723+0000 7f61eea00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f61d0004d40 0x7f61f0152a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:33.727 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.724+0000 7f61eea00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f61d0004d40 0x7f61f0152a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58752/0 (socket says 172.21.15.96:58752) 2024-05-28T06:04:33.727 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.724+0000 7f61eea00640 1 -- 172.21.15.96:0/3244950492 learned_addr learned my addr 172.21.15.96:0/3244950492 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:33.727 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.724+0000 7f61ef400640 1 --2- 172.21.15.96:0/3244950492 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61f00c9950 0x7f61f00c9dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:33.727 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.724+0000 7f61ee000640 1 --2- 172.21.15.96:0/3244950492 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f61f015ad50 0x7f61f015b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:33.727 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.724+0000 7f61eea00640 1 -- 172.21.15.96:0/3244950492 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f61f015ad50 msgr2=0x7f61f015b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:33.727 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.724+0000 7f61eea00640 1 --2- 172.21.15.96:0/3244950492 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f61f015ad50 0x7f61f015b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:33.727 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.724+0000 7f61eea00640 1 -- 172.21.15.96:0/3244950492 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61f00c9950 msgr2=0x7f61f00c9dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:33.727 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.724+0000 7f61eea00640 1 --2- 172.21.15.96:0/3244950492 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61f00c9950 0x7f61f00c9dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:33.728 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.724+0000 7f61eea00640 1 -- 172.21.15.96:0/3244950492 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61f015b7b0 con 0x7f61d0004d40 2024-05-28T06:04:33.728 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.724+0000 7f61ee000640 1 --2- 172.21.15.96:0/3244950492 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f61f015ad50 0x7f61f015b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:04:33.728 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.724+0000 7f61eea00640 1 --2- 172.21.15.96:0/3244950492 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f61d0004d40 0x7f61f0152a70 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f61e000e810 tx=0x7f61e000ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:33.728 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.725+0000 7f61e6a00640 1 -- 172.21.15.96:0/3244950492 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61e00050d0 con 0x7f61d0004d40 2024-05-28T06:04:33.728 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.725+0000 7f61e6a00640 1 -- 172.21.15.96:0/3244950492 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61e0018070 con 0x7f61d0004d40 2024-05-28T06:04:33.728 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.725+0000 7f61e6a00640 1 -- 172.21.15.96:0/3244950492 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61e000c6c0 con 0x7f61d0004d40 2024-05-28T06:04:33.728 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.725+0000 7f61efe00640 1 -- 172.21.15.96:0/3244950492 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61f0160560 con 0x7f61d0004d40 2024-05-28T06:04:33.728 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.725+0000 7f61efe00640 1 -- 172.21.15.96:0/3244950492 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61f0153340 con 0x7f61d0004d40 2024-05-28T06:04:33.729 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.726+0000 7f61efe00640 1 -- 172.21.15.96:0/3244950492 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61f00d4250 con 0x7f61d0004d40 2024-05-28T06:04:33.729 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.727+0000 7f61e6a00640 1 -- 172.21.15.96:0/3244950492 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f61e000c860 con 0x7f61d0004d40 2024-05-28T06:04:33.729 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.728+0000 7f61e6a00640 1 --2- 172.21.15.96:0/3244950492 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f61cc06c920 0x7f61cc06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:33.729 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.728+0000 7f61e6a00640 1 -- 172.21.15.96:0/3244950492 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f61e008b9b0 con 0x7f61d0004d40 2024-05-28T06:04:33.729 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.728+0000 7f61ee000640 1 --2- 172.21.15.96:0/3244950492 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f61cc06c920 0x7f61cc06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:33.730 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.729+0000 7f61ee000640 1 --2- 172.21.15.96:0/3244950492 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f61cc06c920 0x7f61cc06ede0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f61d8007580 tx=0x7f61d800d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:33.733 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.732+0000 7f61e6a00640 1 -- 172.21.15.96:0/3244950492 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f61f00d4250 con 0x7f61d0004d40 2024-05-28T06:04:33.980 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.978+0000 7f61efe00640 1 -- 172.21.15.96:0/3244950492 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f61f00689e0 con 0x7f61d0004d40 2024-05-28T06:04:33.980 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.979+0000 7f61e6a00640 1 -- 172.21.15.96:0/3244950492 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) ==== 53+0+73 (secure 0 0 0) 0x7f61e0059d40 con 0x7f61d0004d40 2024-05-28T06:04:33.980 INFO:teuthology.orchestra.run.smithi096.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-05-28T06:04:33.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.981+0000 7f61efe00640 1 -- 172.21.15.96:0/3244950492 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f61cc06c920 msgr2=0x7f61cc06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:33.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.981+0000 7f61efe00640 1 --2- 172.21.15.96:0/3244950492 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f61cc06c920 0x7f61cc06ede0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f61d8007580 tx=0x7f61d800d040 comp rx=0 tx=0).stop 2024-05-28T06:04:33.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.981+0000 7f61efe00640 1 -- 172.21.15.96:0/3244950492 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f61d0004d40 msgr2=0x7f61f0152a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:33.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.981+0000 7f61efe00640 1 --2- 172.21.15.96:0/3244950492 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f61d0004d40 0x7f61f0152a70 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f61e000e810 tx=0x7f61e000ecd0 comp rx=0 tx=0).stop 2024-05-28T06:04:33.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.981+0000 7f61efe00640 1 -- 172.21.15.96:0/3244950492 shutdown_connections 2024-05-28T06:04:33.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.981+0000 7f61efe00640 1 --2- 172.21.15.96:0/3244950492 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f61cc06c920 0x7f61cc06ede0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:33.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.981+0000 7f61efe00640 1 --2- 172.21.15.96:0/3244950492 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61f00c9950 0x7f61f00c9dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:33.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.981+0000 7f61efe00640 1 --2- 172.21.15.96:0/3244950492 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f61f015ad50 0x7f61f015b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:33.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.981+0000 7f61efe00640 1 --2- 172.21.15.96:0/3244950492 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f61d0004d40 0x7f61f0152a70 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:33.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.982+0000 7f61efe00640 1 -- 172.21.15.96:0/3244950492 >> 172.21.15.96:0/3244950492 conn(0x7f61f008bfc0 msgr2=0x7f61f00b1990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:33.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.982+0000 7f61efe00640 1 -- 172.21.15.96:0/3244950492 shutdown_connections 2024-05-28T06:04:33.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:33.982+0000 7f61efe00640 1 -- 172.21.15.96:0/3244950492 wait complete. 2024-05-28T06:04:34.002 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-05-28T06:04:34.002 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-28T06:04:34.002 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi096.front.sepia.ceph.com 2024-05-28T06:04:34.002 INFO:tasks.cephfs.mount:self.client.name = client.2 2024-05-28T06:04:34.002 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-05-28T06:04:34.003 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-05-28T06:04:34.003 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-05-28T06:04:34.003 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-28T06:04:34.003 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2024-05-28T06:04:34.003 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:04:34.003 DEBUG:teuthology.orchestra.run.smithi096:> ip addr 2024-05-28T06:04:34.053 INFO:teuthology.orchestra.run.smithi096.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-28T06:04:34.054 INFO:teuthology.orchestra.run.smithi096.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-28T06:04:34.054 INFO:teuthology.orchestra.run.smithi096.stdout: inet 127.0.0.1/8 scope host lo 2024-05-28T06:04:34.054 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:04:34.054 INFO:teuthology.orchestra.run.smithi096.stdout: inet6 ::1/128 scope host 2024-05-28T06:04:34.054 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:04:34.054 INFO:teuthology.orchestra.run.smithi096.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-28T06:04:34.054 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether 0c:c4:7a:d8:d9:bc brd ff:ff:ff:ff:ff:ff 2024-05-28T06:04:34.054 INFO:teuthology.orchestra.run.smithi096.stdout: altname enp7s0f0 2024-05-28T06:04:34.054 INFO:teuthology.orchestra.run.smithi096.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-28T06:04:34.054 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether 0c:c4:7a:d8:d9:bd brd ff:ff:ff:ff:ff:ff 2024-05-28T06:04:34.054 INFO:teuthology.orchestra.run.smithi096.stdout: altname enp7s0f1 2024-05-28T06:04:34.054 INFO:teuthology.orchestra.run.smithi096.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-28T06:04:34.055 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether 0c:c4:7a:8f:cc:0c brd ff:ff:ff:ff:ff:ff 2024-05-28T06:04:34.055 INFO:teuthology.orchestra.run.smithi096.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-28T06:04:34.055 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether 0c:c4:7a:8f:cc:0d brd ff:ff:ff:ff:ff:ff 2024-05-28T06:04:34.055 INFO:teuthology.orchestra.run.smithi096.stdout: inet 172.21.15.96/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-05-28T06:04:34.055 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft 42563sec preferred_lft 42563sec 2024-05-28T06:04:34.055 INFO:teuthology.orchestra.run.smithi096.stdout: inet6 fe80::ec4:7aff:fe8f:cc0d/64 scope link 2024-05-28T06:04:34.055 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:04:34.055 INFO:teuthology.orchestra.run.smithi096.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-05-28T06:04:34.055 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether ce:6f:19:93:44:f9 brd ff:ff:ff:ff:ff:ff 2024-05-28T06:04:34.055 INFO:teuthology.orchestra.run.smithi096.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-05-28T06:04:34.055 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:04:34.055 INFO:teuthology.orchestra.run.smithi096.stdout: inet6 fe80::604e:fcff:fe60:7892/64 scope link 2024-05-28T06:04:34.055 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:04:34.056 INFO:teuthology.orchestra.run.smithi096.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-28T06:04:34.056 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether ce:6f:19:93:44:f9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-28T06:04:34.056 INFO:teuthology.orchestra.run.smithi096.stdout: inet6 fe80::cc6f:19ff:fe93:44f9/64 scope link 2024-05-28T06:04:34.056 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:04:34.057 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:04:34.057 DEBUG:teuthology.orchestra.run.smithi096:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-28T06:04:34.136 INFO:teuthology.orchestra.run.smithi096.stdout:1 2024-05-28T06:04:34.140 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:04:34.140 DEBUG:teuthology.orchestra.run.smithi096:> route 2024-05-28T06:04:34.190 INFO:teuthology.orchestra.run.smithi096.stdout:Kernel IP routing table 2024-05-28T06:04:34.191 INFO:teuthology.orchestra.run.smithi096.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-28T06:04:34.191 INFO:teuthology.orchestra.run.smithi096.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-05-28T06:04:34.191 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-05-28T06:04:34.191 INFO:teuthology.orchestra.run.smithi096.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-28T06:04:34.191 DEBUG:teuthology.orchestra.run.smithi096:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-28T06:04:34.191 DEBUG:teuthology.orchestra.run.smithi096:> set -e 2024-05-28T06:04:34.191 DEBUG:teuthology.orchestra.run.smithi096:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-05-28T06:04:34.191 DEBUG:teuthology.orchestra.run.smithi096:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-05-28T06:04:34.192 DEBUG:teuthology.orchestra.run.smithi096:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-05-28T06:04:34.192 DEBUG:teuthology.orchestra.run.smithi096:> ') 2024-05-28T06:04:34.295 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:34 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-28T06:04:34.391 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:34 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-28T06:04:34.400 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:04:34.401 DEBUG:teuthology.orchestra.run.smithi096:> ip netns list 2024-05-28T06:04:34.447 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-05-28T06:04:34.447 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:04:34.448 DEBUG:teuthology.orchestra.run.smithi096:> ip netns list-id 2024-05-28T06:04:34.544 INFO:teuthology.orchestra.run.smithi096.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2024-05-28T06:04:34.545 DEBUG:teuthology.orchestra.run.smithi096:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-28T06:04:34.545 DEBUG:teuthology.orchestra.run.smithi096:> set -e 2024-05-28T06:04:34.545 DEBUG:teuthology.orchestra.run.smithi096:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-28T06:04:34.545 DEBUG:teuthology.orchestra.run.smithi096:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2024-05-28T06:04:34.545 DEBUG:teuthology.orchestra.run.smithi096:> ') 2024-05-28T06:04:34.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:34 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-28T06:04:34.715 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:34 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-28T06:04:34.725 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:04:34.726 DEBUG:teuthology.orchestra.run.smithi096:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-05-28T06:04:34.809 INFO:teuthology.orchestra.run.smithi096.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-28T06:04:34.809 INFO:teuthology.orchestra.run.smithi096.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-28T06:04:34.809 INFO:teuthology.orchestra.run.smithi096.stdout: inet 127.0.0.1/8 scope host lo 2024-05-28T06:04:34.809 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:04:34.809 INFO:teuthology.orchestra.run.smithi096.stdout: inet6 ::1/128 scope host 2024-05-28T06:04:34.810 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:04:34.810 INFO:teuthology.orchestra.run.smithi096.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-05-28T06:04:34.810 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether 1e:4e:97:31:21:35 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-05-28T06:04:34.810 INFO:teuthology.orchestra.run.smithi096.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-05-28T06:04:34.810 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:04:34.810 INFO:teuthology.orchestra.run.smithi096.stdout: inet6 fe80::1c4e:97ff:fe31:2135/64 scope link 2024-05-28T06:04:34.810 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:04:34.813 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:04:34.813 DEBUG:teuthology.orchestra.run.smithi096:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-05-28T06:04:34.895 INFO:teuthology.orchestra.run.smithi096.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-28T06:04:34.896 INFO:teuthology.orchestra.run.smithi096.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-28T06:04:34.896 INFO:teuthology.orchestra.run.smithi096.stdout: inet 127.0.0.1/8 scope host lo 2024-05-28T06:04:34.896 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:04:34.896 INFO:teuthology.orchestra.run.smithi096.stdout: inet6 ::1/128 scope host 2024-05-28T06:04:34.896 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:04:34.896 INFO:teuthology.orchestra.run.smithi096.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-05-28T06:04:34.896 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether 1e:4e:97:31:21:35 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-05-28T06:04:34.896 INFO:teuthology.orchestra.run.smithi096.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-05-28T06:04:34.896 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:04:34.896 INFO:teuthology.orchestra.run.smithi096.stdout: inet6 fe80::1c4e:97ff:fe31:2135/64 scope link 2024-05-28T06:04:34.896 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:04:34.901 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2024-05-28T06:04:34.902 DEBUG:teuthology.orchestra.run.smithi096:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-28T06:04:34.902 DEBUG:teuthology.orchestra.run.smithi096:> set -e 2024-05-28T06:04:34.902 DEBUG:teuthology.orchestra.run.smithi096:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2024-05-28T06:04:34.902 DEBUG:teuthology.orchestra.run.smithi096:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-05-28T06:04:34.902 DEBUG:teuthology.orchestra.run.smithi096:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2024-05-28T06:04:34.902 DEBUG:teuthology.orchestra.run.smithi096:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2024-05-28T06:04:34.903 DEBUG:teuthology.orchestra.run.smithi096:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2024-05-28T06:04:34.903 DEBUG:teuthology.orchestra.run.smithi096:> ') 2024-05-28T06:04:35.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:35 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-28T06:04:35.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:35 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-28T06:04:35.174 DEBUG:teuthology.orchestra.run.smithi096:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-28T06:04:35.174 DEBUG:teuthology.orchestra.run.smithi096:> set -e 2024-05-28T06:04:35.174 DEBUG:teuthology.orchestra.run.smithi096:> sudo ip link set brx.1 up 2024-05-28T06:04:35.174 DEBUG:teuthology.orchestra.run.smithi096:> sudo ip link set dev brx.1 master ceph-brx 2024-05-28T06:04:35.174 DEBUG:teuthology.orchestra.run.smithi096:> ') 2024-05-28T06:04:35.268 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:35 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-28T06:04:35.367 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:35 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-28T06:04:35.376 INFO:tasks.cephfs.fuse_mount:Client client.2 config is {} 2024-05-28T06:04:35.376 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-28T06:04:35.376 DEBUG:teuthology.orchestra.run.smithi096:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2024-05-28T06:04:35.419 INFO:teuthology.orchestra.run.smithi096.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2024-05-28T06:04:35.419 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-28T06:04:35.419 DEBUG:teuthology.orchestra.run.smithi096:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2024-05-28T06:04:35.511 DEBUG:teuthology.orchestra.run.smithi096:> sudo modprobe fuse 2024-05-28T06:04:35.587 DEBUG:teuthology.orchestra.run.smithi096:> cat /proc/self/mounts | awk '{print $2}' 2024-05-28T06:04:35.639 INFO:teuthology.orchestra.run.smithi096.stdout:/proc 2024-05-28T06:04:35.639 INFO:teuthology.orchestra.run.smithi096.stdout:/sys 2024-05-28T06:04:35.639 INFO:teuthology.orchestra.run.smithi096.stdout:/dev 2024-05-28T06:04:35.640 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/security 2024-05-28T06:04:35.640 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/shm 2024-05-28T06:04:35.640 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/pts 2024-05-28T06:04:35.640 INFO:teuthology.orchestra.run.smithi096.stdout:/run 2024-05-28T06:04:35.640 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/cgroup 2024-05-28T06:04:35.640 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/pstore 2024-05-28T06:04:35.640 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/bpf 2024-05-28T06:04:35.640 INFO:teuthology.orchestra.run.smithi096.stdout:/ 2024-05-28T06:04:35.640 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/selinux 2024-05-28T06:04:35.640 INFO:teuthology.orchestra.run.smithi096.stdout:/proc/sys/fs/binfmt_misc 2024-05-28T06:04:35.640 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/mqueue 2024-05-28T06:04:35.640 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/debug 2024-05-28T06:04:35.640 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/hugepages 2024-05-28T06:04:35.641 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/tracing 2024-05-28T06:04:35.641 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/fuse/connections 2024-05-28T06:04:35.641 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/config 2024-05-28T06:04:35.641 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-sysctl.service 2024-05-28T06:04:35.641 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-28T06:04:35.641 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-28T06:04:35.641 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/nfs/rpc_pipefs 2024-05-28T06:04:35.641 INFO:teuthology.orchestra.run.smithi096.stdout:/run/user/1000 2024-05-28T06:04:35.642 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph 2024-05-28T06:04:35.642 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-0 2024-05-28T06:04:35.642 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-1 2024-05-28T06:04:35.642 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-2 2024-05-28T06:04:35.642 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-3 2024-05-28T06:04:35.642 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns 2024-05-28T06:04:35.643 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-28T06:04:35.643 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-28T06:04:35.643 INFO:teuthology.orchestra.run.smithi096.stdout:/home/ubuntu/cephtest/mnt.0 2024-05-28T06:04:35.643 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-28T06:04:35.643 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-28T06:04:35.645 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:04:35.645 DEBUG:teuthology.orchestra.run.smithi096:> ls /sys/fs/fuse/connections 2024-05-28T06:04:35.698 INFO:teuthology.orchestra.run.smithi096.stdout:40 2024-05-28T06:04:35.698 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [40] 2024-05-28T06:04:35.698 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.2 --id 2) 2024-05-28T06:04:35.706 DEBUG:teuthology.orchestra.run.smithi096:> sudo modprobe fuse 2024-05-28T06:04:35.777 DEBUG:teuthology.orchestra.run.smithi096:> cat /proc/self/mounts | awk '{print $2}' 2024-05-28T06:04:35.819 INFO:teuthology.orchestra.run.smithi096.stdout:/proc 2024-05-28T06:04:35.819 INFO:teuthology.orchestra.run.smithi096.stdout:/sys 2024-05-28T06:04:35.820 INFO:teuthology.orchestra.run.smithi096.stdout:/dev 2024-05-28T06:04:35.820 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/security 2024-05-28T06:04:35.820 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/shm 2024-05-28T06:04:35.820 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/pts 2024-05-28T06:04:35.820 INFO:teuthology.orchestra.run.smithi096.stdout:/run 2024-05-28T06:04:35.820 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/cgroup 2024-05-28T06:04:35.820 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/pstore 2024-05-28T06:04:35.820 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/bpf 2024-05-28T06:04:35.820 INFO:teuthology.orchestra.run.smithi096.stdout:/ 2024-05-28T06:04:35.820 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/selinux 2024-05-28T06:04:35.820 INFO:teuthology.orchestra.run.smithi096.stdout:/proc/sys/fs/binfmt_misc 2024-05-28T06:04:35.820 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/mqueue 2024-05-28T06:04:35.820 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/debug 2024-05-28T06:04:35.821 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/hugepages 2024-05-28T06:04:35.821 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/tracing 2024-05-28T06:04:35.821 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/fuse/connections 2024-05-28T06:04:35.821 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/config 2024-05-28T06:04:35.821 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-sysctl.service 2024-05-28T06:04:35.821 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-28T06:04:35.821 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-28T06:04:35.821 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/nfs/rpc_pipefs 2024-05-28T06:04:35.821 INFO:teuthology.orchestra.run.smithi096.stdout:/run/user/1000 2024-05-28T06:04:35.821 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph 2024-05-28T06:04:35.821 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-0 2024-05-28T06:04:35.821 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-1 2024-05-28T06:04:35.821 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-2 2024-05-28T06:04:35.821 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-3 2024-05-28T06:04:35.822 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns 2024-05-28T06:04:35.822 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-28T06:04:35.822 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-28T06:04:35.822 INFO:teuthology.orchestra.run.smithi096.stdout:/home/ubuntu/cephtest/mnt.0 2024-05-28T06:04:35.822 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-28T06:04:35.822 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-28T06:04:35.823 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:04:35.823 DEBUG:teuthology.orchestra.run.smithi096:> ls /sys/fs/fuse/connections 2024-05-28T06:04:35.875 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi096.stderr:2024-05-28T06:04:35.874+0000 7fcce9e64480 -1 init, newargv = 0x5580fe3dbeb0 newargc=15 2024-05-28T06:04:35.875 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi096.stderr:ceph-fuse[45976]: starting ceph client 2024-05-28T06:04:35.885 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi096.stderr:ceph-fuse[45976]: starting fuse 2024-05-28T06:04:35.906 INFO:teuthology.orchestra.run.smithi096.stdout:40 2024-05-28T06:04:35.906 INFO:teuthology.orchestra.run.smithi096.stdout:43 2024-05-28T06:04:35.907 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40, 43] 2024-05-28T06:04:35.907 DEBUG:teuthology.orchestra.run.smithi096:> sudo stdin-killer -- python3 -c ' 2024-05-28T06:04:35.907 DEBUG:teuthology.orchestra.run.smithi096:> import glob 2024-05-28T06:04:35.907 DEBUG:teuthology.orchestra.run.smithi096:> import re 2024-05-28T06:04:35.907 DEBUG:teuthology.orchestra.run.smithi096:> import os 2024-05-28T06:04:35.907 DEBUG:teuthology.orchestra.run.smithi096:> import subprocess 2024-05-28T06:04:35.907 DEBUG:teuthology.orchestra.run.smithi096:> 2024-05-28T06:04:35.907 DEBUG:teuthology.orchestra.run.smithi096:> def _find_admin_socket(client_name): 2024-05-28T06:04:35.907 DEBUG:teuthology.orchestra.run.smithi096:> asok_path = "/var/run/ceph/ceph-client.2.*.asok" 2024-05-28T06:04:35.907 DEBUG:teuthology.orchestra.run.smithi096:> files = glob.glob(asok_path) 2024-05-28T06:04:35.907 DEBUG:teuthology.orchestra.run.smithi096:> mountpoint = "/home/ubuntu/cephtest/mnt.2" 2024-05-28T06:04:35.908 DEBUG:teuthology.orchestra.run.smithi096:> 2024-05-28T06:04:35.908 DEBUG:teuthology.orchestra.run.smithi096:> # Given a non-glob path, it better be there 2024-05-28T06:04:35.908 DEBUG:teuthology.orchestra.run.smithi096:> if "*" not in asok_path: 2024-05-28T06:04:35.908 DEBUG:teuthology.orchestra.run.smithi096:> assert(len(files) == 1) 2024-05-28T06:04:35.908 DEBUG:teuthology.orchestra.run.smithi096:> return files[0] 2024-05-28T06:04:35.908 DEBUG:teuthology.orchestra.run.smithi096:> 2024-05-28T06:04:35.908 DEBUG:teuthology.orchestra.run.smithi096:> for f in files: 2024-05-28T06:04:35.908 DEBUG:teuthology.orchestra.run.smithi096:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-05-28T06:04:35.908 DEBUG:teuthology.orchestra.run.smithi096:> if os.path.exists("/proc/{0}".format(pid)): 2024-05-28T06:04:35.908 DEBUG:teuthology.orchestra.run.smithi096:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-05-28T06:04:35.908 DEBUG:teuthology.orchestra.run.smithi096:> contents = proc_f.read() 2024-05-28T06:04:35.908 DEBUG:teuthology.orchestra.run.smithi096:> if mountpoint in contents: 2024-05-28T06:04:35.908 DEBUG:teuthology.orchestra.run.smithi096:> return f 2024-05-28T06:04:35.908 DEBUG:teuthology.orchestra.run.smithi096:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-05-28T06:04:35.908 DEBUG:teuthology.orchestra.run.smithi096:> 2024-05-28T06:04:35.908 DEBUG:teuthology.orchestra.run.smithi096:> print(_find_admin_socket("client.2")) 2024-05-28T06:04:35.909 DEBUG:teuthology.orchestra.run.smithi096:> ' 2024-05-28T06:04:36.090 INFO:teuthology.orchestra.run.smithi096.stdout:/var/run/ceph/ceph-client.2.45976.asok 2024-05-28T06:04:36.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-28T06:04:36.108 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.2.45976.asok 2024-05-28T06:04:36.108 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:04:36.108 DEBUG:teuthology.orchestra.run.smithi096:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.2.45976.asok status 2024-05-28T06:04:36.334 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:04:36.334 INFO:teuthology.orchestra.run.smithi096.stdout: "metadata": { 2024-05-28T06:04:36.334 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph_sha1": "9ed025fbf4406fb54980a0647506b15895b6d863", 2024-05-28T06:04:36.334 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph_version": "ceph version 19.0.0-3154-g9ed025fb (9ed025fbf4406fb54980a0647506b15895b6d863) squid (dev)", 2024-05-28T06:04:36.334 INFO:teuthology.orchestra.run.smithi096.stdout: "entity_id": "2", 2024-05-28T06:04:36.334 INFO:teuthology.orchestra.run.smithi096.stdout: "hostname": "smithi096", 2024-05-28T06:04:36.335 INFO:teuthology.orchestra.run.smithi096.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.2", 2024-05-28T06:04:36.335 INFO:teuthology.orchestra.run.smithi096.stdout: "pid": "45976", 2024-05-28T06:04:36.335 INFO:teuthology.orchestra.run.smithi096.stdout: "root": "/" 2024-05-28T06:04:36.335 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:04:36.335 INFO:teuthology.orchestra.run.smithi096.stdout: "dentry_count": 0, 2024-05-28T06:04:36.335 INFO:teuthology.orchestra.run.smithi096.stdout: "dentry_pinned_count": 0, 2024-05-28T06:04:36.335 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 4680, 2024-05-28T06:04:36.335 INFO:teuthology.orchestra.run.smithi096.stdout: "inst": { 2024-05-28T06:04:36.335 INFO:teuthology.orchestra.run.smithi096.stdout: "name": { 2024-05-28T06:04:36.335 INFO:teuthology.orchestra.run.smithi096.stdout: "type": "client", 2024-05-28T06:04:36.335 INFO:teuthology.orchestra.run.smithi096.stdout: "num": 4680 2024-05-28T06:04:36.335 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:04:36.335 INFO:teuthology.orchestra.run.smithi096.stdout: "addr": { 2024-05-28T06:04:36.335 INFO:teuthology.orchestra.run.smithi096.stdout: "type": "v1", 2024-05-28T06:04:36.336 INFO:teuthology.orchestra.run.smithi096.stdout: "addr": "192.168.0.2:0", 2024-05-28T06:04:36.336 INFO:teuthology.orchestra.run.smithi096.stdout: "nonce": 78391829 2024-05-28T06:04:36.336 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:04:36.336 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:04:36.336 INFO:teuthology.orchestra.run.smithi096.stdout: "addr": { 2024-05-28T06:04:36.336 INFO:teuthology.orchestra.run.smithi096.stdout: "type": "v1", 2024-05-28T06:04:36.336 INFO:teuthology.orchestra.run.smithi096.stdout: "addr": "192.168.0.2:0", 2024-05-28T06:04:36.336 INFO:teuthology.orchestra.run.smithi096.stdout: "nonce": 78391829 2024-05-28T06:04:36.336 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:04:36.336 INFO:teuthology.orchestra.run.smithi096.stdout: "inst_str": "client.4680 192.168.0.2:0/78391829", 2024-05-28T06:04:36.336 INFO:teuthology.orchestra.run.smithi096.stdout: "addr_str": "192.168.0.2:0/78391829", 2024-05-28T06:04:36.336 INFO:teuthology.orchestra.run.smithi096.stdout: "inode_count": 1, 2024-05-28T06:04:36.336 INFO:teuthology.orchestra.run.smithi096.stdout: "mds_epoch": 7, 2024-05-28T06:04:36.336 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_epoch": 20, 2024-05-28T06:04:36.337 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_epoch_barrier": 0, 2024-05-28T06:04:36.337 INFO:teuthology.orchestra.run.smithi096.stdout: "blocklisted": false, 2024-05-28T06:04:36.337 INFO:teuthology.orchestra.run.smithi096.stdout: "fs_name": "cephfs" 2024-05-28T06:04:36.337 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:04:36.349 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-05-28T06:04:36.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.541+0000 7f4ab0400640 1 Processor -- start 2024-05-28T06:04:36.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.541+0000 7f4ab0400640 1 -- start start 2024-05-28T06:04:36.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.542+0000 7f4ab0400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4aa8162680 0x7f4aa8162a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:36.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.543+0000 7f4ab0400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f4aa80cdbb0 0x7f4aa80ce040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:36.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.543+0000 7f4ab0400640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4aa8162fa0 con 0x7f4aa80d0a60 2024-05-28T06:04:36.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.543+0000 7f4ab0400640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f4aa806b900 con 0x7f4aa80cdbb0 2024-05-28T06:04:36.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.543+0000 7f4ab0400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f4aa81634d0 con 0x7f4aa8162680 2024-05-28T06:04:36.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.543+0000 7f4aad000640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f4aa80d0a60 0x7f4aa80b3860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:41024/0 (socket says 172.21.15.96:41024) 2024-05-28T06:04:36.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.543+0000 7f4aad000640 1 -- 172.21.15.96:0/1102325659 learned_addr learned my addr 172.21.15.96:0/1102325659 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:04:36.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.543+0000 7f4aae400640 1 --2- 172.21.15.96:0/1102325659 >> v2:172.21.15.123:3300/0 conn(0x7f4aa80cdbb0 0x7f4aa80ce040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:36.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.543+0000 7f4aada00640 1 --2- 172.21.15.96:0/1102325659 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4aa8162680 0x7f4aa8162a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:36.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.543+0000 7f4aa7e00640 1 -- 172.21.15.96:0/1102325659 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 600969744 0 0) 0x7f4aa8162fa0 con 0x7f4aa80d0a60 2024-05-28T06:04:36.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.544+0000 7f4aa7e00640 1 -- 172.21.15.96:0/1102325659 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4a90003520 con 0x7f4aa80d0a60 2024-05-28T06:04:36.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.544+0000 7f4aa7e00640 1 -- 172.21.15.96:0/1102325659 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2457572597 0 0) 0x7f4a90003520 con 0x7f4aa80d0a60 2024-05-28T06:04:36.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.544+0000 7f4aa7e00640 1 -- 172.21.15.96:0/1102325659 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4aa8162680 msgr2=0x7f4aa8162a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:36.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.544+0000 7f4aa7e00640 1 --2- 172.21.15.96:0/1102325659 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4aa8162680 0x7f4aa8162a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:36.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.544+0000 7f4aa7e00640 1 -- 172.21.15.96:0/1102325659 >> v2:172.21.15.123:3300/0 conn(0x7f4aa80cdbb0 msgr2=0x7f4aa80ce040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:36.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.544+0000 7f4aa7e00640 1 --2- 172.21.15.96:0/1102325659 >> v2:172.21.15.123:3300/0 conn(0x7f4aa80cdbb0 0x7f4aa80ce040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:36.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.544+0000 7f4aa7e00640 1 -- 172.21.15.96:0/1102325659 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4aa80d0fb0 con 0x7f4aa80d0a60 2024-05-28T06:04:36.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.544+0000 7f4aa7e00640 1 -- 172.21.15.96:0/1102325659 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f4a94001670 con 0x7f4aa80d0a60 2024-05-28T06:04:36.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.544+0000 7f4aae400640 1 --2- 172.21.15.96:0/1102325659 >> v2:172.21.15.123:3300/0 conn(0x7f4aa80cdbb0 0x7f4aa80ce040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-28T06:04:36.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.544+0000 7f4aa7e00640 1 -- 172.21.15.96:0/1102325659 >> v1:172.21.15.96:6789/0 conn(0x7f4aa80d0a60 legacy=0x7f4aa80b3860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:36.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.544+0000 7f4aa7e00640 1 --2- 172.21.15.96:0/1102325659 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4aa8162680 0x7f4a900039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:36.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.544+0000 7f4aa7e00640 1 --2- 172.21.15.96:0/1102325659 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4a90003f30 0x7f4a90004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:36.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.544+0000 7f4aa7e00640 1 --2- 172.21.15.96:0/1102325659 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4a90004d40 0x7f4a900051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:36.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.544+0000 7f4aa7e00640 1 -- 172.21.15.96:0/1102325659 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4aa806b900 con 0x7f4a90004d40 2024-05-28T06:04:36.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.544+0000 7f4aa7e00640 1 -- 172.21.15.96:0/1102325659 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f4aa81634d0 con 0x7f4a90003f30 2024-05-28T06:04:36.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.544+0000 7f4aa7e00640 1 -- 172.21.15.96:0/1102325659 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f4a9000b9c0 con 0x7f4aa8162680 2024-05-28T06:04:36.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.545+0000 7f4aae400640 1 --2- 172.21.15.96:0/1102325659 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4a90004d40 0x7f4a900051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:36.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.545+0000 7f4aada00640 1 --2- 172.21.15.96:0/1102325659 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4aa8162680 0x7f4a900039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:36.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.545+0000 7f4aad000640 1 --2- 172.21.15.96:0/1102325659 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4a90003f30 0x7f4a90004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:36.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.545+0000 7f4aae400640 1 -- 172.21.15.96:0/1102325659 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4aa8162680 msgr2=0x7f4a900039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:36.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.545+0000 7f4aae400640 1 --2- 172.21.15.96:0/1102325659 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4aa8162680 0x7f4a900039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:36.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.545+0000 7f4aae400640 1 -- 172.21.15.96:0/1102325659 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4a90003f30 msgr2=0x7f4a90004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:36.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.545+0000 7f4aae400640 1 --2- 172.21.15.96:0/1102325659 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4a90003f30 0x7f4a90004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:36.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.545+0000 7f4aae400640 1 -- 172.21.15.96:0/1102325659 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a90003520 con 0x7f4a90004d40 2024-05-28T06:04:36.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.546+0000 7f4aae400640 1 --2- 172.21.15.96:0/1102325659 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4a90004d40 0x7f4a900051c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f4aa000e350 tx=0x7f4a90005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3811bf90ca294fba server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:36.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.546+0000 7f4aa7e00640 1 -- 172.21.15.96:0/1102325659 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4aa0009de0 con 0x7f4a90004d40 2024-05-28T06:04:36.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.546+0000 7f4aa7e00640 1 -- 172.21.15.96:0/1102325659 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4aa002e650 con 0x7f4a90004d40 2024-05-28T06:04:36.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.546+0000 7f4aa7e00640 1 -- 172.21.15.96:0/1102325659 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4aa002ea00 con 0x7f4a90004d40 2024-05-28T06:04:36.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.547+0000 7f4ab0400640 1 -- 172.21.15.96:0/1102325659 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4a90004d40 msgr2=0x7f4a900051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:36.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.547+0000 7f4ab0400640 1 --2- 172.21.15.96:0/1102325659 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4a90004d40 0x7f4a900051c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f4aa000e350 tx=0x7f4a90005830 comp rx=0 tx=0).stop 2024-05-28T06:04:36.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.547+0000 7f4aae400640 1 -- 172.21.15.96:0/1102325659 reap_dead start 2024-05-28T06:04:36.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.547+0000 7f4ab0400640 1 -- 172.21.15.96:0/1102325659 shutdown_connections 2024-05-28T06:04:36.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.547+0000 7f4ab0400640 1 -- 172.21.15.96:0/1102325659 >> 172.21.15.96:0/1102325659 conn(0x7f4aa809aa30 msgr2=0x7f4aa80b2f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:36.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.547+0000 7f4ab0400640 1 -- 172.21.15.96:0/1102325659 shutdown_connections 2024-05-28T06:04:36.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.547+0000 7f4ab0400640 1 -- 172.21.15.96:0/1102325659 wait complete. 2024-05-28T06:04:36.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.549+0000 7f4ab0400640 1 Processor -- start 2024-05-28T06:04:36.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.549+0000 7f4ab0400640 1 -- start start 2024-05-28T06:04:36.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.550+0000 7f4ab0400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4a90004d40 0x7f4aa81ea910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:36.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.550+0000 7f4ab0400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4aa81e93b0 0x7f4aa80cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:36.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.550+0000 7f4ab0400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4aa80b6ad0 0x7f4aa80b4030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:36.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.550+0000 7f4ab0400640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4aa806c110 con 0x7f4aa80b6ad0 2024-05-28T06:04:36.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.550+0000 7f4ab0400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f4aa806be30 con 0x7f4a90004d40 2024-05-28T06:04:36.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.550+0000 7f4ab0400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f4aa806c630 con 0x7f4aa81e93b0 2024-05-28T06:04:36.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.550+0000 7f4aae400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4aa80b6ad0 0x7f4aa80b4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:36.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.550+0000 7f4aae400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4aa80b6ad0 0x7f4aa80b4030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58768/0 (socket says 172.21.15.96:58768) 2024-05-28T06:04:36.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.550+0000 7f4aae400640 1 -- 172.21.15.96:0/2097966997 learned_addr learned my addr 172.21.15.96:0/2097966997 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:36.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.551+0000 7f4aada00640 1 --2- 172.21.15.96:0/2097966997 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4a90004d40 0x7f4aa81ea910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:36.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.551+0000 7f4aad000640 1 --2- 172.21.15.96:0/2097966997 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4aa81e93b0 0x7f4aa80cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:36.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.551+0000 7f4aae400640 1 -- 172.21.15.96:0/2097966997 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4aa81e93b0 msgr2=0x7f4aa80cbd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:36.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.551+0000 7f4aae400640 1 --2- 172.21.15.96:0/2097966997 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4aa81e93b0 0x7f4aa80cbd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:36.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.551+0000 7f4aae400640 1 -- 172.21.15.96:0/2097966997 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4a90004d40 msgr2=0x7f4aa81ea910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:36.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.551+0000 7f4aae400640 1 --2- 172.21.15.96:0/2097966997 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4a90004d40 0x7f4aa81ea910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:36.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.551+0000 7f4aae400640 1 -- 172.21.15.96:0/2097966997 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4aa81e9870 con 0x7f4aa80b6ad0 2024-05-28T06:04:36.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.551+0000 7f4aada00640 1 --2- 172.21.15.96:0/2097966997 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4a90004d40 0x7f4aa81ea910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:04:36.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.551+0000 7f4aae400640 1 --2- 172.21.15.96:0/2097966997 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4aa80b6ad0 0x7f4aa80b4030 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f4aa000e470 tx=0x7f4aa0004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:36.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.551+0000 7f4aa5600640 1 -- 172.21.15.96:0/2097966997 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4aa0009de0 con 0x7f4aa80b6ad0 2024-05-28T06:04:36.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.551+0000 7f4aa5600640 1 -- 172.21.15.96:0/2097966997 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4aa0003990 con 0x7f4aa80b6ad0 2024-05-28T06:04:36.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.551+0000 7f4aa5600640 1 -- 172.21.15.96:0/2097966997 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4aa0003d60 con 0x7f4aa80b6ad0 2024-05-28T06:04:36.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.551+0000 7f4ab0400640 1 -- 172.21.15.96:0/2097966997 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4aa81e9b50 con 0x7f4aa80b6ad0 2024-05-28T06:04:36.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.552+0000 7f4ab0400640 1 -- 172.21.15.96:0/2097966997 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4aa81ea110 con 0x7f4aa80b6ad0 2024-05-28T06:04:36.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.553+0000 7f4ab0400640 1 -- 172.21.15.96:0/2097966997 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4aa80683b0 con 0x7f4aa80b6ad0 2024-05-28T06:04:36.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.554+0000 7f4aa5600640 1 -- 172.21.15.96:0/2097966997 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4aa002c040 con 0x7f4aa80b6ad0 2024-05-28T06:04:36.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.555+0000 7f4aa5600640 1 --2- 172.21.15.96:0/2097966997 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4a8806c920 0x7f4a8806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:36.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.555+0000 7f4aa5600640 1 -- 172.21.15.96:0/2097966997 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f4aa00aca00 con 0x7f4aa80b6ad0 2024-05-28T06:04:36.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.556+0000 7f4aada00640 1 --2- 172.21.15.96:0/2097966997 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4a8806c920 0x7f4a8806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:36.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.557+0000 7f4aada00640 1 --2- 172.21.15.96:0/2097966997 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4a8806c920 0x7f4a8806ede0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c009a00 tx=0x7f4a9c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:36.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.560+0000 7f4aa5600640 1 -- 172.21.15.96:0/2097966997 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4aa0032030 con 0x7f4aa80b6ad0 2024-05-28T06:04:36.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.802+0000 7f4ab0400640 1 -- 172.21.15.96:0/2097966997 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f4aa809aea0 con 0x7f4aa80b6ad0 2024-05-28T06:04:36.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.803+0000 7f4aa5600640 1 -- 172.21.15.96:0/2097966997 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) ==== 53+0+73 (secure 0 0 0) 0x7f4aa007ad90 con 0x7f4aa80b6ad0 2024-05-28T06:04:36.805 INFO:teuthology.orchestra.run.smithi096.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-05-28T06:04:36.807 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.806+0000 7f4ab0400640 1 -- 172.21.15.96:0/2097966997 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4a8806c920 msgr2=0x7f4a8806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:36.807 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.806+0000 7f4ab0400640 1 --2- 172.21.15.96:0/2097966997 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4a8806c920 0x7f4a8806ede0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c009a00 tx=0x7f4a9c008040 comp rx=0 tx=0).stop 2024-05-28T06:04:36.807 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.806+0000 7f4ab0400640 1 -- 172.21.15.96:0/2097966997 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4aa80b6ad0 msgr2=0x7f4aa80b4030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:36.807 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.806+0000 7f4ab0400640 1 --2- 172.21.15.96:0/2097966997 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4aa80b6ad0 0x7f4aa80b4030 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f4aa000e470 tx=0x7f4aa0004ab0 comp rx=0 tx=0).stop 2024-05-28T06:04:36.808 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.806+0000 7f4ab0400640 1 -- 172.21.15.96:0/2097966997 shutdown_connections 2024-05-28T06:04:36.808 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.806+0000 7f4ab0400640 1 --2- 172.21.15.96:0/2097966997 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4a8806c920 0x7f4a8806ede0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:36.808 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.806+0000 7f4ab0400640 1 --2- 172.21.15.96:0/2097966997 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4aa80b6ad0 0x7f4aa80b4030 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:36.808 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.806+0000 7f4ab0400640 1 --2- 172.21.15.96:0/2097966997 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4aa81e93b0 0x7f4aa80cbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:36.808 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.806+0000 7f4ab0400640 1 --2- 172.21.15.96:0/2097966997 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4a90004d40 0x7f4aa81ea910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:36.808 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.806+0000 7f4ab0400640 1 -- 172.21.15.96:0/2097966997 >> 172.21.15.96:0/2097966997 conn(0x7f4aa809aa30 msgr2=0x7f4aa806eed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:36.809 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.806+0000 7f4ab0400640 1 -- 172.21.15.96:0/2097966997 shutdown_connections 2024-05-28T06:04:36.809 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:36.806+0000 7f4ab0400640 1 -- 172.21.15.96:0/2097966997 wait complete. 2024-05-28T06:04:36.825 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-05-28T06:04:36.825 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-28T06:04:36.826 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi123.front.sepia.ceph.com 2024-05-28T06:04:36.826 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-05-28T06:04:36.826 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-05-28T06:04:36.826 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-05-28T06:04:36.826 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-05-28T06:04:36.826 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-28T06:04:36.826 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-05-28T06:04:36.826 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:04:36.826 DEBUG:teuthology.orchestra.run.smithi123:> ip addr 2024-05-28T06:04:36.883 INFO:teuthology.orchestra.run.smithi123.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-28T06:04:36.883 INFO:teuthology.orchestra.run.smithi123.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-28T06:04:36.883 INFO:teuthology.orchestra.run.smithi123.stdout: inet 127.0.0.1/8 scope host lo 2024-05-28T06:04:36.883 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:04:36.883 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 ::1/128 scope host 2024-05-28T06:04:36.884 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:04:36.884 INFO:teuthology.orchestra.run.smithi123.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-28T06:04:36.884 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:d8:da:6c brd ff:ff:ff:ff:ff:ff 2024-05-28T06:04:36.884 INFO:teuthology.orchestra.run.smithi123.stdout: altname enp7s0f0 2024-05-28T06:04:36.884 INFO:teuthology.orchestra.run.smithi123.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-28T06:04:36.884 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:d8:da:6d brd ff:ff:ff:ff:ff:ff 2024-05-28T06:04:36.884 INFO:teuthology.orchestra.run.smithi123.stdout: altname enp7s0f1 2024-05-28T06:04:36.884 INFO:teuthology.orchestra.run.smithi123.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-28T06:04:36.884 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:8f:d3:f0 brd ff:ff:ff:ff:ff:ff 2024-05-28T06:04:36.884 INFO:teuthology.orchestra.run.smithi123.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-28T06:04:36.884 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:8f:d3:f1 brd ff:ff:ff:ff:ff:ff 2024-05-28T06:04:36.884 INFO:teuthology.orchestra.run.smithi123.stdout: inet 172.21.15.123/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-05-28T06:04:36.884 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft 42567sec preferred_lft 42567sec 2024-05-28T06:04:36.885 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 fe80::ec4:7aff:fe8f:d3f1/64 scope link 2024-05-28T06:04:36.885 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:04:36.886 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-05-28T06:04:36.886 DEBUG:teuthology.orchestra.run.smithi123:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-28T06:04:36.886 DEBUG:teuthology.orchestra.run.smithi123:> set -e 2024-05-28T06:04:36.886 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip link add name ceph-brx type bridge 2024-05-28T06:04:36.886 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip addr flush dev ceph-brx 2024-05-28T06:04:36.886 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip link set ceph-brx up 2024-05-28T06:04:36.886 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-05-28T06:04:36.886 DEBUG:teuthology.orchestra.run.smithi123:> ') 2024-05-28T06:04:36.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:04:36 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-28T06:04:37.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:04:37 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-28T06:04:37.374 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:04:37.374 DEBUG:teuthology.orchestra.run.smithi123:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-28T06:04:37.440 INFO:teuthology.orchestra.run.smithi123.stdout:1 2024-05-28T06:04:37.443 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:04:37.443 DEBUG:teuthology.orchestra.run.smithi123:> route 2024-05-28T06:04:37.495 INFO:teuthology.orchestra.run.smithi123.stdout:Kernel IP routing table 2024-05-28T06:04:37.495 INFO:teuthology.orchestra.run.smithi123.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-28T06:04:37.495 INFO:teuthology.orchestra.run.smithi123.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-05-28T06:04:37.495 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-05-28T06:04:37.496 INFO:teuthology.orchestra.run.smithi123.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-28T06:04:37.496 DEBUG:teuthology.orchestra.run.smithi123:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-28T06:04:37.496 DEBUG:teuthology.orchestra.run.smithi123:> set -e 2024-05-28T06:04:37.496 DEBUG:teuthology.orchestra.run.smithi123:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-05-28T06:04:37.497 DEBUG:teuthology.orchestra.run.smithi123:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-05-28T06:04:37.497 DEBUG:teuthology.orchestra.run.smithi123:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-05-28T06:04:37.497 DEBUG:teuthology.orchestra.run.smithi123:> ') 2024-05-28T06:04:37.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:04:37 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-28T06:04:38.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:04:38 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-28T06:04:38.256 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:04:38.256 DEBUG:teuthology.orchestra.run.smithi123:> ip netns list 2024-05-28T06:04:38.312 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:04:38.312 DEBUG:teuthology.orchestra.run.smithi123:> ip netns list-id 2024-05-28T06:04:38.366 DEBUG:teuthology.orchestra.run.smithi123:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-28T06:04:38.366 DEBUG:teuthology.orchestra.run.smithi123:> set -e 2024-05-28T06:04:38.367 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-05-28T06:04:38.367 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-05-28T06:04:38.367 DEBUG:teuthology.orchestra.run.smithi123:> ') 2024-05-28T06:04:38.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:04:38 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-28T06:04:38.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:04:38 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-28T06:04:38.551 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-05-28T06:04:38.551 DEBUG:teuthology.orchestra.run.smithi123:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-28T06:04:38.551 DEBUG:teuthology.orchestra.run.smithi123:> set -e 2024-05-28T06:04:38.551 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-05-28T06:04:38.551 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-05-28T06:04:38.551 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-05-28T06:04:38.551 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-05-28T06:04:38.551 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-05-28T06:04:38.552 DEBUG:teuthology.orchestra.run.smithi123:> ') 2024-05-28T06:04:38.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:04:38 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-28T06:04:38.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:04:38 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-28T06:04:38.837 DEBUG:teuthology.orchestra.run.smithi123:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-28T06:04:38.838 DEBUG:teuthology.orchestra.run.smithi123:> set -e 2024-05-28T06:04:38.838 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip link set brx.0 up 2024-05-28T06:04:38.838 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip link set dev brx.0 master ceph-brx 2024-05-28T06:04:38.838 DEBUG:teuthology.orchestra.run.smithi123:> ') 2024-05-28T06:04:38.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:04:38 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-28T06:04:39.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:04:39 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-28T06:04:39.042 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-05-28T06:04:39.042 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-28T06:04:39.043 DEBUG:teuthology.orchestra.run.smithi123:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-05-28T06:04:39.091 INFO:teuthology.orchestra.run.smithi123.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-05-28T06:04:39.092 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-28T06:04:39.092 DEBUG:teuthology.orchestra.run.smithi123:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-05-28T06:04:39.141 DEBUG:teuthology.orchestra.run.smithi123:> sudo modprobe fuse 2024-05-28T06:04:39.219 DEBUG:teuthology.orchestra.run.smithi123:> cat /proc/self/mounts | awk '{print $2}' 2024-05-28T06:04:39.270 INFO:teuthology.orchestra.run.smithi123.stdout:/proc 2024-05-28T06:04:39.270 INFO:teuthology.orchestra.run.smithi123.stdout:/sys 2024-05-28T06:04:39.270 INFO:teuthology.orchestra.run.smithi123.stdout:/dev 2024-05-28T06:04:39.270 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/security 2024-05-28T06:04:39.270 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/shm 2024-05-28T06:04:39.271 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/pts 2024-05-28T06:04:39.271 INFO:teuthology.orchestra.run.smithi123.stdout:/run 2024-05-28T06:04:39.271 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/cgroup 2024-05-28T06:04:39.271 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/pstore 2024-05-28T06:04:39.271 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/bpf 2024-05-28T06:04:39.271 INFO:teuthology.orchestra.run.smithi123.stdout:/ 2024-05-28T06:04:39.271 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/selinux 2024-05-28T06:04:39.271 INFO:teuthology.orchestra.run.smithi123.stdout:/proc/sys/fs/binfmt_misc 2024-05-28T06:04:39.271 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/mqueue 2024-05-28T06:04:39.271 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/hugepages 2024-05-28T06:04:39.271 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/debug 2024-05-28T06:04:39.271 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/tracing 2024-05-28T06:04:39.272 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/fuse/connections 2024-05-28T06:04:39.272 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/config 2024-05-28T06:04:39.272 INFO:teuthology.orchestra.run.smithi123.stdout:/run/credentials/systemd-sysctl.service 2024-05-28T06:04:39.272 INFO:teuthology.orchestra.run.smithi123.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-28T06:04:39.272 INFO:teuthology.orchestra.run.smithi123.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-28T06:04:39.272 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/nfs/rpc_pipefs 2024-05-28T06:04:39.272 INFO:teuthology.orchestra.run.smithi123.stdout:/run/user/1000 2024-05-28T06:04:39.272 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph 2024-05-28T06:04:39.272 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-4 2024-05-28T06:04:39.272 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-5 2024-05-28T06:04:39.272 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-6 2024-05-28T06:04:39.273 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-7 2024-05-28T06:04:39.273 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns 2024-05-28T06:04:39.273 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-05-28T06:04:39.273 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-05-28T06:04:39.274 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:04:39.274 DEBUG:teuthology.orchestra.run.smithi123:> ls /sys/fs/fuse/connections 2024-05-28T06:04:39.320 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-05-28T06:04:39.320 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-05-28T06:04:39.369 DEBUG:teuthology.orchestra.run.smithi123:> sudo modprobe fuse 2024-05-28T06:04:39.439 DEBUG:teuthology.orchestra.run.smithi123:> cat /proc/self/mounts | awk '{print $2}' 2024-05-28T06:04:39.478 INFO:teuthology.orchestra.run.smithi123.stdout:/proc 2024-05-28T06:04:39.479 INFO:teuthology.orchestra.run.smithi123.stdout:/sys 2024-05-28T06:04:39.479 INFO:teuthology.orchestra.run.smithi123.stdout:/dev 2024-05-28T06:04:39.479 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/security 2024-05-28T06:04:39.479 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/shm 2024-05-28T06:04:39.479 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/pts 2024-05-28T06:04:39.479 INFO:teuthology.orchestra.run.smithi123.stdout:/run 2024-05-28T06:04:39.479 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/cgroup 2024-05-28T06:04:39.479 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/pstore 2024-05-28T06:04:39.479 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/bpf 2024-05-28T06:04:39.479 INFO:teuthology.orchestra.run.smithi123.stdout:/ 2024-05-28T06:04:39.479 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/selinux 2024-05-28T06:04:39.479 INFO:teuthology.orchestra.run.smithi123.stdout:/proc/sys/fs/binfmt_misc 2024-05-28T06:04:39.480 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/mqueue 2024-05-28T06:04:39.480 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/hugepages 2024-05-28T06:04:39.480 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/debug 2024-05-28T06:04:39.480 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/tracing 2024-05-28T06:04:39.480 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/fuse/connections 2024-05-28T06:04:39.480 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/config 2024-05-28T06:04:39.480 INFO:teuthology.orchestra.run.smithi123.stdout:/run/credentials/systemd-sysctl.service 2024-05-28T06:04:39.480 INFO:teuthology.orchestra.run.smithi123.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-28T06:04:39.480 INFO:teuthology.orchestra.run.smithi123.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-28T06:04:39.480 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/nfs/rpc_pipefs 2024-05-28T06:04:39.480 INFO:teuthology.orchestra.run.smithi123.stdout:/run/user/1000 2024-05-28T06:04:39.480 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph 2024-05-28T06:04:39.480 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-4 2024-05-28T06:04:39.481 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-5 2024-05-28T06:04:39.481 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-6 2024-05-28T06:04:39.481 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-7 2024-05-28T06:04:39.481 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns 2024-05-28T06:04:39.481 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-05-28T06:04:39.481 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-05-28T06:04:39.482 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:04:39.482 DEBUG:teuthology.orchestra.run.smithi123:> ls /sys/fs/fuse/connections 2024-05-28T06:04:39.545 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi123.stderr:2024-05-28T06:04:39.544+0000 7f2438705480 -1 init, newargv = 0x55a14f426b90 newargc=15 2024-05-28T06:04:39.545 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi123.stderr:ceph-fuse[43096]: starting ceph client 2024-05-28T06:04:39.556 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi123.stderr:ceph-fuse[43096]: starting fuse 2024-05-28T06:04:39.570 INFO:teuthology.orchestra.run.smithi123.stdout:40 2024-05-28T06:04:39.570 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40] 2024-05-28T06:04:39.570 DEBUG:teuthology.orchestra.run.smithi123:> sudo stdin-killer -- python3 -c ' 2024-05-28T06:04:39.570 DEBUG:teuthology.orchestra.run.smithi123:> import glob 2024-05-28T06:04:39.570 DEBUG:teuthology.orchestra.run.smithi123:> import re 2024-05-28T06:04:39.570 DEBUG:teuthology.orchestra.run.smithi123:> import os 2024-05-28T06:04:39.571 DEBUG:teuthology.orchestra.run.smithi123:> import subprocess 2024-05-28T06:04:39.571 DEBUG:teuthology.orchestra.run.smithi123:> 2024-05-28T06:04:39.571 DEBUG:teuthology.orchestra.run.smithi123:> def _find_admin_socket(client_name): 2024-05-28T06:04:39.571 DEBUG:teuthology.orchestra.run.smithi123:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-05-28T06:04:39.571 DEBUG:teuthology.orchestra.run.smithi123:> files = glob.glob(asok_path) 2024-05-28T06:04:39.571 DEBUG:teuthology.orchestra.run.smithi123:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-05-28T06:04:39.571 DEBUG:teuthology.orchestra.run.smithi123:> 2024-05-28T06:04:39.571 DEBUG:teuthology.orchestra.run.smithi123:> # Given a non-glob path, it better be there 2024-05-28T06:04:39.571 DEBUG:teuthology.orchestra.run.smithi123:> if "*" not in asok_path: 2024-05-28T06:04:39.571 DEBUG:teuthology.orchestra.run.smithi123:> assert(len(files) == 1) 2024-05-28T06:04:39.571 DEBUG:teuthology.orchestra.run.smithi123:> return files[0] 2024-05-28T06:04:39.571 DEBUG:teuthology.orchestra.run.smithi123:> 2024-05-28T06:04:39.572 DEBUG:teuthology.orchestra.run.smithi123:> for f in files: 2024-05-28T06:04:39.572 DEBUG:teuthology.orchestra.run.smithi123:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-05-28T06:04:39.572 DEBUG:teuthology.orchestra.run.smithi123:> if os.path.exists("/proc/{0}".format(pid)): 2024-05-28T06:04:39.572 DEBUG:teuthology.orchestra.run.smithi123:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-05-28T06:04:39.572 DEBUG:teuthology.orchestra.run.smithi123:> contents = proc_f.read() 2024-05-28T06:04:39.572 DEBUG:teuthology.orchestra.run.smithi123:> if mountpoint in contents: 2024-05-28T06:04:39.572 DEBUG:teuthology.orchestra.run.smithi123:> return f 2024-05-28T06:04:39.572 DEBUG:teuthology.orchestra.run.smithi123:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-05-28T06:04:39.572 DEBUG:teuthology.orchestra.run.smithi123:> 2024-05-28T06:04:39.572 DEBUG:teuthology.orchestra.run.smithi123:> print(_find_admin_socket("client.1")) 2024-05-28T06:04:39.572 DEBUG:teuthology.orchestra.run.smithi123:> ' 2024-05-28T06:04:39.752 INFO:teuthology.orchestra.run.smithi123.stdout:/var/run/ceph/ceph-client.1.43096.asok 2024-05-28T06:04:39.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:04:39 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-28T06:04:39.769 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.43096.asok 2024-05-28T06:04:39.770 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:04:39.770 DEBUG:teuthology.orchestra.run.smithi123:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.43096.asok status 2024-05-28T06:04:39.995 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-05-28T06:04:39.995 INFO:teuthology.orchestra.run.smithi123.stdout: "metadata": { 2024-05-28T06:04:39.996 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph_sha1": "9ed025fbf4406fb54980a0647506b15895b6d863", 2024-05-28T06:04:39.996 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph_version": "ceph version 19.0.0-3154-g9ed025fb (9ed025fbf4406fb54980a0647506b15895b6d863) squid (dev)", 2024-05-28T06:04:39.996 INFO:teuthology.orchestra.run.smithi123.stdout: "entity_id": "1", 2024-05-28T06:04:39.996 INFO:teuthology.orchestra.run.smithi123.stdout: "hostname": "smithi123", 2024-05-28T06:04:39.996 INFO:teuthology.orchestra.run.smithi123.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-05-28T06:04:39.996 INFO:teuthology.orchestra.run.smithi123.stdout: "pid": "43096", 2024-05-28T06:04:39.996 INFO:teuthology.orchestra.run.smithi123.stdout: "root": "/" 2024-05-28T06:04:39.996 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-05-28T06:04:39.996 INFO:teuthology.orchestra.run.smithi123.stdout: "dentry_count": 0, 2024-05-28T06:04:39.996 INFO:teuthology.orchestra.run.smithi123.stdout: "dentry_pinned_count": 0, 2024-05-28T06:04:39.996 INFO:teuthology.orchestra.run.smithi123.stdout: "id": 4537, 2024-05-28T06:04:39.997 INFO:teuthology.orchestra.run.smithi123.stdout: "inst": { 2024-05-28T06:04:39.997 INFO:teuthology.orchestra.run.smithi123.stdout: "name": { 2024-05-28T06:04:39.997 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "client", 2024-05-28T06:04:39.997 INFO:teuthology.orchestra.run.smithi123.stdout: "num": 4537 2024-05-28T06:04:39.997 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-05-28T06:04:39.997 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": { 2024-05-28T06:04:39.997 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "v1", 2024-05-28T06:04:39.997 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": "192.168.0.1:0", 2024-05-28T06:04:39.997 INFO:teuthology.orchestra.run.smithi123.stdout: "nonce": 3970451305 2024-05-28T06:04:39.997 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-05-28T06:04:39.997 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-05-28T06:04:39.997 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": { 2024-05-28T06:04:39.998 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "v1", 2024-05-28T06:04:39.998 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": "192.168.0.1:0", 2024-05-28T06:04:39.998 INFO:teuthology.orchestra.run.smithi123.stdout: "nonce": 3970451305 2024-05-28T06:04:39.998 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-05-28T06:04:39.998 INFO:teuthology.orchestra.run.smithi123.stdout: "inst_str": "client.4537 192.168.0.1:0/3970451305", 2024-05-28T06:04:39.998 INFO:teuthology.orchestra.run.smithi123.stdout: "addr_str": "192.168.0.1:0/3970451305", 2024-05-28T06:04:39.998 INFO:teuthology.orchestra.run.smithi123.stdout: "inode_count": 1, 2024-05-28T06:04:39.998 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_epoch": 7, 2024-05-28T06:04:39.998 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_epoch": 20, 2024-05-28T06:04:39.998 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_epoch_barrier": 0, 2024-05-28T06:04:39.998 INFO:teuthology.orchestra.run.smithi123.stdout: "blocklisted": false, 2024-05-28T06:04:39.999 INFO:teuthology.orchestra.run.smithi123.stdout: "fs_name": "cephfs" 2024-05-28T06:04:39.999 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-05-28T06:04:40.010 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-05-28T06:04:40.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:40.219+0000 7f93f5800640 1 Processor -- start 2024-05-28T06:04:40.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:40.219+0000 7f93f5800640 1 -- start start 2024-05-28T06:04:40.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:40.220+0000 7f93f5800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f93f006f860 0x7f93f00b3150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:40.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:40.220+0000 7f93f5800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f93f00d0bb0 0x7f93f00ad130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:40.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:40.220+0000 7f93f5800640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93f006ed10 con 0x7f93f00b3690 2024-05-28T06:04:40.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:40.220+0000 7f93f5800640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f93f015cb00 con 0x7f93f00d0bb0 2024-05-28T06:04:40.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:40.220+0000 7f93f5800640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f93f00b3cc0 con 0x7f93f006f860 2024-05-28T06:04:40.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:40.221+0000 7f93ef400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f93f00d0bb0 0x7f93f00ad130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:40.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:40.221+0000 7f93ef400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f93f00d0bb0 0x7f93f00ad130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47654/0 (socket says 172.21.15.96:47654) 2024-05-28T06:04:40.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:40.221+0000 7f93ef400640 1 -- 172.21.15.96:0/1257134322 learned_addr learned my addr 172.21.15.96:0/1257134322 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:40.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:40.221+0000 7f93eea00640 1 --2- 172.21.15.96:0/1257134322 >> v2:172.21.15.123:3300/0 conn(0x7f93f006f860 0x7f93f00b3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:40.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:40.221+0000 7f93ef400640 1 -- 172.21.15.96:0/1257134322 >> v2:172.21.15.123:3300/0 conn(0x7f93f006f860 msgr2=0x7f93f00b3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:40.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:40.221+0000 7f93ef400640 1 --2- 172.21.15.96:0/1257134322 >> v2:172.21.15.123:3300/0 conn(0x7f93f006f860 0x7f93f00b3150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:40.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:40.221+0000 7f93ef400640 1 -- 172.21.15.96:0/1257134322 >> v1:172.21.15.123:6789/0 conn(0x7f93f00b3690 legacy=0x7f93f00d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:40.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:40.221+0000 7f93ef400640 1 -- 172.21.15.96:0/1257134322 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93f00b3fa0 con 0x7f93f00d0bb0 2024-05-28T06:04:40.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:40.222+0000 7f93ef400640 1 --2- 172.21.15.96:0/1257134322 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f93f00d0bb0 0x7f93f00ad130 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f93f006c8d0 tx=0x7f93e802a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ec620dc8818612b server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:40.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:40.223+0000 7f93ee000640 1 -- 172.21.15.96:0/1257134322 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93e8036070 con 0x7f93f00d0bb0 2024-05-28T06:04:40.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:40.223+0000 7f93ee000640 1 -- 172.21.15.96:0/1257134322 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93e8031600 con 0x7f93f00d0bb0 2024-05-28T06:04:40.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:40.223+0000 7f93ee000640 1 -- 172.21.15.96:0/1257134322 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93e80319d0 con 0x7f93f00d0bb0 2024-05-28T06:04:40.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:40.223+0000 7f93f5800640 1 -- 172.21.15.96:0/1257134322 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f93f00d0bb0 msgr2=0x7f93f00ad130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:40.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:40.223+0000 7f93f5800640 1 --2- 172.21.15.96:0/1257134322 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f93f00d0bb0 0x7f93f00ad130 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f93f006c8d0 tx=0x7f93e802a6f0 comp rx=0 tx=0).stop 2024-05-28T06:04:40.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:40.223+0000 7f93f5800640 1 -- 172.21.15.96:0/1257134322 shutdown_connections 2024-05-28T06:04:40.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:40.223+0000 7f93f5800640 1 --2- 172.21.15.96:0/1257134322 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f93f00d0bb0 0x7f93f00ad130 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:40.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:40.223+0000 7f93f5800640 1 --2- 172.21.15.96:0/1257134322 >> v2:172.21.15.123:3300/0 conn(0x7f93f006f860 0x7f93f00b3150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:40.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:40.223+0000 7f93f5800640 1 -- 172.21.15.96:0/1257134322 >> 172.21.15.96:0/1257134322 conn(0x7f93f006bae0 msgr2=0x7f93f006bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:40.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:40.223+0000 7f93f5800640 1 -- 172.21.15.96:0/1257134322 shutdown_connections 2024-05-28T06:04:40.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:40.223+0000 7f93f5800640 1 -- 172.21.15.96:0/1257134322 wait complete. 2024-05-28T06:04:40.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:40.225+0000 7f93f5800640 1 Processor -- start 2024-05-28T06:04:40.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:40.225+0000 7f93f5800640 1 -- start start 2024-05-28T06:04:40.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:40.226+0000 7f93f5800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f93f006f860 0x7f93f01590c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:40.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:40.226+0000 7f93f5800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f93f00b3690 0x7f93f015e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:40.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:40.226+0000 7f93f5800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f93f00d0bb0 0x7f93f00c2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:40.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:40.226+0000 7f93f5800640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f93f00b4840 con 0x7f93f00d0bb0 2024-05-28T06:04:40.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:40.226+0000 7f93f5800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f93f00b4560 con 0x7f93f00b3690 2024-05-28T06:04:40.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:40.226+0000 7f93f5800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f93f00ad670 con 0x7f93f006f860 2024-05-28T06:04:40.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:40.226+0000 7f93ef400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f93f00d0bb0 0x7f93f00c2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:40.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:40.227+0000 7f93ef400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f93f00d0bb0 0x7f93f00c2ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47656/0 (socket says 172.21.15.96:47656) 2024-05-28T06:04:40.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:40.227+0000 7f93ef400640 1 -- 172.21.15.96:0/1801245168 learned_addr learned my addr 172.21.15.96:0/1801245168 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:40.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:40.227+0000 7f93e6000640 1 --2- 172.21.15.96:0/1801245168 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f93f00b3690 0x7f93f015e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:40.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:40.227+0000 7f93eea00640 1 --2- 172.21.15.96:0/1801245168 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f93f006f860 0x7f93f01590c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:40.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:40.227+0000 7f93ef400640 1 -- 172.21.15.96:0/1801245168 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f93f006f860 msgr2=0x7f93f01590c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:40.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:40.227+0000 7f93ef400640 1 --2- 172.21.15.96:0/1801245168 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f93f006f860 0x7f93f01590c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:40.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:40.227+0000 7f93ef400640 1 -- 172.21.15.96:0/1801245168 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f93f00b3690 msgr2=0x7f93f015e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:40.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:40.227+0000 7f93ef400640 1 --2- 172.21.15.96:0/1801245168 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f93f00b3690 0x7f93f015e720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:40.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:40.227+0000 7f93ef400640 1 -- 172.21.15.96:0/1801245168 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93f015dfd0 con 0x7f93f00d0bb0 2024-05-28T06:04:40.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:40.227+0000 7f93ef400640 1 --2- 172.21.15.96:0/1801245168 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f93f00d0bb0 0x7f93f00c2ca0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f93e8009f40 tx=0x7f93e8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:40.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:40.227+0000 7f93e7400640 1 -- 172.21.15.96:0/1801245168 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93e8036050 con 0x7f93f00d0bb0 2024-05-28T06:04:40.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:40.228+0000 7f93e7400640 1 -- 172.21.15.96:0/1801245168 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93e8042070 con 0x7f93f00d0bb0 2024-05-28T06:04:40.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:40.228+0000 7f93f5800640 1 -- 172.21.15.96:0/1801245168 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93f015d210 con 0x7f93f00d0bb0 2024-05-28T06:04:40.231 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:40.228+0000 7f93e7400640 1 -- 172.21.15.96:0/1801245168 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93e8009510 con 0x7f93f00d0bb0 2024-05-28T06:04:40.231 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:40.228+0000 7f93f5800640 1 -- 172.21.15.96:0/1801245168 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93f015d7d0 con 0x7f93f00d0bb0 2024-05-28T06:04:40.231 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:40.229+0000 7f93f5800640 1 -- 172.21.15.96:0/1801245168 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93f00689e0 con 0x7f93f00d0bb0 2024-05-28T06:04:40.231 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:40.230+0000 7f93e7400640 1 -- 172.21.15.96:0/1801245168 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f93e8007bd0 con 0x7f93f00d0bb0 2024-05-28T06:04:40.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:40.231+0000 7f93e7400640 1 --2- 172.21.15.96:0/1801245168 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f93cc06c920 0x7f93cc06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:40.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:40.231+0000 7f93e7400640 1 -- 172.21.15.96:0/1801245168 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f93e80acb10 con 0x7f93f00d0bb0 2024-05-28T06:04:40.233 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:40.232+0000 7f93eea00640 1 --2- 172.21.15.96:0/1801245168 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f93cc06c920 0x7f93cc06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:40.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:40.233+0000 7f93eea00640 1 --2- 172.21.15.96:0/1801245168 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f93cc06c920 0x7f93cc06ede0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f93e0006fd0 tx=0x7f93e0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:40.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:40.235+0000 7f93e7400640 1 -- 172.21.15.96:0/1801245168 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f93f006ed10 con 0x7f93f00d0bb0 2024-05-28T06:04:40.475 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:40.474+0000 7f93f5800640 1 -- 172.21.15.96:0/1801245168 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f93f015cf50 con 0x7f93f00d0bb0 2024-05-28T06:04:40.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:40.475+0000 7f93e7400640 1 -- 172.21.15.96:0/1801245168 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) ==== 53+0+73 (secure 0 0 0) 0x7f93e8045360 con 0x7f93f00d0bb0 2024-05-28T06:04:40.476 INFO:teuthology.orchestra.run.smithi096.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-05-28T06:04:40.479 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:40.477+0000 7f93f5800640 1 -- 172.21.15.96:0/1801245168 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f93cc06c920 msgr2=0x7f93cc06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:40.479 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:40.477+0000 7f93f5800640 1 --2- 172.21.15.96:0/1801245168 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f93cc06c920 0x7f93cc06ede0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f93e0006fd0 tx=0x7f93e0008040 comp rx=0 tx=0).stop 2024-05-28T06:04:40.479 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:40.477+0000 7f93f5800640 1 -- 172.21.15.96:0/1801245168 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f93f00d0bb0 msgr2=0x7f93f00c2ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:40.479 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:40.477+0000 7f93f5800640 1 --2- 172.21.15.96:0/1801245168 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f93f00d0bb0 0x7f93f00c2ca0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f93e8009f40 tx=0x7f93e8004a40 comp rx=0 tx=0).stop 2024-05-28T06:04:40.479 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:40.478+0000 7f93f5800640 1 -- 172.21.15.96:0/1801245168 shutdown_connections 2024-05-28T06:04:40.479 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:40.478+0000 7f93f5800640 1 --2- 172.21.15.96:0/1801245168 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f93cc06c920 0x7f93cc06ede0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:40.479 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:40.478+0000 7f93f5800640 1 --2- 172.21.15.96:0/1801245168 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f93f00d0bb0 0x7f93f00c2ca0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:40.479 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:40.478+0000 7f93f5800640 1 --2- 172.21.15.96:0/1801245168 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f93f00b3690 0x7f93f015e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:40.480 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:40.478+0000 7f93f5800640 1 --2- 172.21.15.96:0/1801245168 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f93f006f860 0x7f93f01590c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:40.480 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:40.478+0000 7f93f5800640 1 -- 172.21.15.96:0/1801245168 >> 172.21.15.96:0/1801245168 conn(0x7f93f006bae0 msgr2=0x7f93f00d0470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:40.480 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:40.478+0000 7f93f5800640 1 -- 172.21.15.96:0/1801245168 shutdown_connections 2024-05-28T06:04:40.480 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:40.478+0000 7f93f5800640 1 -- 172.21.15.96:0/1801245168 wait complete. 2024-05-28T06:04:40.497 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-05-28T06:04:40.497 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-28T06:04:40.497 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi123.front.sepia.ceph.com 2024-05-28T06:04:40.497 INFO:tasks.cephfs.mount:self.client.name = client.3 2024-05-28T06:04:40.497 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-05-28T06:04:40.497 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-05-28T06:04:40.497 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-05-28T06:04:40.497 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-28T06:04:40.497 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2024-05-28T06:04:40.498 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:04:40.498 DEBUG:teuthology.orchestra.run.smithi123:> ip addr 2024-05-28T06:04:40.555 INFO:teuthology.orchestra.run.smithi123.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-28T06:04:40.555 INFO:teuthology.orchestra.run.smithi123.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-28T06:04:40.555 INFO:teuthology.orchestra.run.smithi123.stdout: inet 127.0.0.1/8 scope host lo 2024-05-28T06:04:40.556 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:04:40.556 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 ::1/128 scope host 2024-05-28T06:04:40.556 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:04:40.556 INFO:teuthology.orchestra.run.smithi123.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-28T06:04:40.556 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:d8:da:6c brd ff:ff:ff:ff:ff:ff 2024-05-28T06:04:40.556 INFO:teuthology.orchestra.run.smithi123.stdout: altname enp7s0f0 2024-05-28T06:04:40.556 INFO:teuthology.orchestra.run.smithi123.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-28T06:04:40.556 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:d8:da:6d brd ff:ff:ff:ff:ff:ff 2024-05-28T06:04:40.556 INFO:teuthology.orchestra.run.smithi123.stdout: altname enp7s0f1 2024-05-28T06:04:40.556 INFO:teuthology.orchestra.run.smithi123.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-28T06:04:40.556 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:8f:d3:f0 brd ff:ff:ff:ff:ff:ff 2024-05-28T06:04:40.556 INFO:teuthology.orchestra.run.smithi123.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-28T06:04:40.557 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:8f:d3:f1 brd ff:ff:ff:ff:ff:ff 2024-05-28T06:04:40.557 INFO:teuthology.orchestra.run.smithi123.stdout: inet 172.21.15.123/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-05-28T06:04:40.557 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft 42564sec preferred_lft 42564sec 2024-05-28T06:04:40.557 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 fe80::ec4:7aff:fe8f:d3f1/64 scope link 2024-05-28T06:04:40.557 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:04:40.557 INFO:teuthology.orchestra.run.smithi123.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-05-28T06:04:40.557 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether d2:de:1e:c6:20:df brd ff:ff:ff:ff:ff:ff 2024-05-28T06:04:40.557 INFO:teuthology.orchestra.run.smithi123.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-05-28T06:04:40.557 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:04:40.557 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 fe80::b89a:3bff:fe17:fd30/64 scope link 2024-05-28T06:04:40.557 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:04:40.557 INFO:teuthology.orchestra.run.smithi123.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-28T06:04:40.557 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether d2:de:1e:c6:20:df brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-05-28T06:04:40.557 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 fe80::d0de:1eff:fec6:20df/64 scope link tentative 2024-05-28T06:04:40.558 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:04:40.558 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:04:40.559 DEBUG:teuthology.orchestra.run.smithi123:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-28T06:04:40.640 INFO:teuthology.orchestra.run.smithi123.stdout:1 2024-05-28T06:04:40.643 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:04:40.643 DEBUG:teuthology.orchestra.run.smithi123:> route 2024-05-28T06:04:40.692 INFO:teuthology.orchestra.run.smithi123.stdout:Kernel IP routing table 2024-05-28T06:04:40.693 INFO:teuthology.orchestra.run.smithi123.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-28T06:04:40.693 INFO:teuthology.orchestra.run.smithi123.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-05-28T06:04:40.693 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-05-28T06:04:40.693 INFO:teuthology.orchestra.run.smithi123.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-28T06:04:40.693 DEBUG:teuthology.orchestra.run.smithi123:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-28T06:04:40.693 DEBUG:teuthology.orchestra.run.smithi123:> set -e 2024-05-28T06:04:40.693 DEBUG:teuthology.orchestra.run.smithi123:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-05-28T06:04:40.693 DEBUG:teuthology.orchestra.run.smithi123:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-05-28T06:04:40.694 DEBUG:teuthology.orchestra.run.smithi123:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-05-28T06:04:40.694 DEBUG:teuthology.orchestra.run.smithi123:> ') 2024-05-28T06:04:40.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:04:40 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-28T06:04:40.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:04:40 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-28T06:04:40.902 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:04:40.902 DEBUG:teuthology.orchestra.run.smithi123:> ip netns list 2024-05-28T06:04:40.950 INFO:teuthology.orchestra.run.smithi123.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-05-28T06:04:40.950 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:04:40.950 DEBUG:teuthology.orchestra.run.smithi123:> ip netns list-id 2024-05-28T06:04:41.044 INFO:teuthology.orchestra.run.smithi123.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2024-05-28T06:04:41.045 DEBUG:teuthology.orchestra.run.smithi123:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-28T06:04:41.045 DEBUG:teuthology.orchestra.run.smithi123:> set -e 2024-05-28T06:04:41.045 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2024-05-28T06:04:41.045 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2024-05-28T06:04:41.045 DEBUG:teuthology.orchestra.run.smithi123:> ') 2024-05-28T06:04:41.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:04:41 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-28T06:04:41.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:04:41 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-28T06:04:41.230 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:04:41.231 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-05-28T06:04:41.301 INFO:teuthology.orchestra.run.smithi123.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-28T06:04:41.301 INFO:teuthology.orchestra.run.smithi123.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-28T06:04:41.301 INFO:teuthology.orchestra.run.smithi123.stdout: inet 127.0.0.1/8 scope host lo 2024-05-28T06:04:41.301 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:04:41.301 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 ::1/128 scope host 2024-05-28T06:04:41.301 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:04:41.301 INFO:teuthology.orchestra.run.smithi123.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-05-28T06:04:41.301 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether a2:92:40:0e:2a:52 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-05-28T06:04:41.301 INFO:teuthology.orchestra.run.smithi123.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-05-28T06:04:41.301 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:04:41.301 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 fe80::a092:40ff:fe0e:2a52/64 scope link 2024-05-28T06:04:41.302 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:04:41.304 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:04:41.304 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-05-28T06:04:41.384 INFO:teuthology.orchestra.run.smithi123.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-28T06:04:41.384 INFO:teuthology.orchestra.run.smithi123.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-28T06:04:41.385 INFO:teuthology.orchestra.run.smithi123.stdout: inet 127.0.0.1/8 scope host lo 2024-05-28T06:04:41.385 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:04:41.385 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 ::1/128 scope host 2024-05-28T06:04:41.385 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:04:41.385 INFO:teuthology.orchestra.run.smithi123.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-05-28T06:04:41.385 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether a2:92:40:0e:2a:52 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-05-28T06:04:41.385 INFO:teuthology.orchestra.run.smithi123.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-05-28T06:04:41.385 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:04:41.385 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 fe80::a092:40ff:fe0e:2a52/64 scope link 2024-05-28T06:04:41.385 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:04:41.391 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2024-05-28T06:04:41.392 DEBUG:teuthology.orchestra.run.smithi123:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-28T06:04:41.392 DEBUG:teuthology.orchestra.run.smithi123:> set -e 2024-05-28T06:04:41.392 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2024-05-28T06:04:41.392 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-05-28T06:04:41.392 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2024-05-28T06:04:41.392 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2024-05-28T06:04:41.392 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2024-05-28T06:04:41.392 DEBUG:teuthology.orchestra.run.smithi123:> ') 2024-05-28T06:04:41.500 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:04:41 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-28T06:04:41.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:04:41 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-28T06:04:41.677 DEBUG:teuthology.orchestra.run.smithi123:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-28T06:04:41.678 DEBUG:teuthology.orchestra.run.smithi123:> set -e 2024-05-28T06:04:41.678 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip link set brx.1 up 2024-05-28T06:04:41.678 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip link set dev brx.1 master ceph-brx 2024-05-28T06:04:41.678 DEBUG:teuthology.orchestra.run.smithi123:> ') 2024-05-28T06:04:41.774 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:04:41 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-28T06:04:41.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:04:41 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-28T06:04:41.879 INFO:tasks.cephfs.fuse_mount:Client client.3 config is {} 2024-05-28T06:04:41.880 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-28T06:04:41.880 DEBUG:teuthology.orchestra.run.smithi123:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2024-05-28T06:04:41.923 INFO:teuthology.orchestra.run.smithi123.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2024-05-28T06:04:41.923 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-28T06:04:41.923 DEBUG:teuthology.orchestra.run.smithi123:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2024-05-28T06:04:42.015 DEBUG:teuthology.orchestra.run.smithi123:> sudo modprobe fuse 2024-05-28T06:04:42.099 DEBUG:teuthology.orchestra.run.smithi123:> cat /proc/self/mounts | awk '{print $2}' 2024-05-28T06:04:42.155 INFO:teuthology.orchestra.run.smithi123.stdout:/proc 2024-05-28T06:04:42.155 INFO:teuthology.orchestra.run.smithi123.stdout:/sys 2024-05-28T06:04:42.155 INFO:teuthology.orchestra.run.smithi123.stdout:/dev 2024-05-28T06:04:42.155 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/security 2024-05-28T06:04:42.155 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/shm 2024-05-28T06:04:42.156 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/pts 2024-05-28T06:04:42.156 INFO:teuthology.orchestra.run.smithi123.stdout:/run 2024-05-28T06:04:42.156 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/cgroup 2024-05-28T06:04:42.156 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/pstore 2024-05-28T06:04:42.156 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/bpf 2024-05-28T06:04:42.156 INFO:teuthology.orchestra.run.smithi123.stdout:/ 2024-05-28T06:04:42.156 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/selinux 2024-05-28T06:04:42.156 INFO:teuthology.orchestra.run.smithi123.stdout:/proc/sys/fs/binfmt_misc 2024-05-28T06:04:42.156 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/mqueue 2024-05-28T06:04:42.156 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/hugepages 2024-05-28T06:04:42.156 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/debug 2024-05-28T06:04:42.156 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/tracing 2024-05-28T06:04:42.156 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/fuse/connections 2024-05-28T06:04:42.156 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/config 2024-05-28T06:04:42.157 INFO:teuthology.orchestra.run.smithi123.stdout:/run/credentials/systemd-sysctl.service 2024-05-28T06:04:42.157 INFO:teuthology.orchestra.run.smithi123.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-28T06:04:42.157 INFO:teuthology.orchestra.run.smithi123.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-28T06:04:42.157 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/nfs/rpc_pipefs 2024-05-28T06:04:42.157 INFO:teuthology.orchestra.run.smithi123.stdout:/run/user/1000 2024-05-28T06:04:42.157 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph 2024-05-28T06:04:42.157 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-4 2024-05-28T06:04:42.157 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-5 2024-05-28T06:04:42.157 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-6 2024-05-28T06:04:42.157 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-7 2024-05-28T06:04:42.157 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns 2024-05-28T06:04:42.157 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-05-28T06:04:42.157 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-05-28T06:04:42.157 INFO:teuthology.orchestra.run.smithi123.stdout:/home/ubuntu/cephtest/mnt.1 2024-05-28T06:04:42.158 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-05-28T06:04:42.158 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-05-28T06:04:42.158 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:04:42.159 DEBUG:teuthology.orchestra.run.smithi123:> ls /sys/fs/fuse/connections 2024-05-28T06:04:42.208 INFO:teuthology.orchestra.run.smithi123.stdout:40 2024-05-28T06:04:42.208 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [40] 2024-05-28T06:04:42.209 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.3 --id 3) 2024-05-28T06:04:42.213 DEBUG:teuthology.orchestra.run.smithi123:> sudo modprobe fuse 2024-05-28T06:04:42.280 DEBUG:teuthology.orchestra.run.smithi123:> cat /proc/self/mounts | awk '{print $2}' 2024-05-28T06:04:42.320 INFO:teuthology.orchestra.run.smithi123.stdout:/proc 2024-05-28T06:04:42.320 INFO:teuthology.orchestra.run.smithi123.stdout:/sys 2024-05-28T06:04:42.320 INFO:teuthology.orchestra.run.smithi123.stdout:/dev 2024-05-28T06:04:42.321 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/security 2024-05-28T06:04:42.321 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/shm 2024-05-28T06:04:42.321 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/pts 2024-05-28T06:04:42.321 INFO:teuthology.orchestra.run.smithi123.stdout:/run 2024-05-28T06:04:42.321 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/cgroup 2024-05-28T06:04:42.321 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/pstore 2024-05-28T06:04:42.321 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/bpf 2024-05-28T06:04:42.321 INFO:teuthology.orchestra.run.smithi123.stdout:/ 2024-05-28T06:04:42.321 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/selinux 2024-05-28T06:04:42.321 INFO:teuthology.orchestra.run.smithi123.stdout:/proc/sys/fs/binfmt_misc 2024-05-28T06:04:42.321 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/mqueue 2024-05-28T06:04:42.321 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/hugepages 2024-05-28T06:04:42.322 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/debug 2024-05-28T06:04:42.322 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/tracing 2024-05-28T06:04:42.322 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/fuse/connections 2024-05-28T06:04:42.322 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/config 2024-05-28T06:04:42.322 INFO:teuthology.orchestra.run.smithi123.stdout:/run/credentials/systemd-sysctl.service 2024-05-28T06:04:42.322 INFO:teuthology.orchestra.run.smithi123.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-28T06:04:42.322 INFO:teuthology.orchestra.run.smithi123.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-28T06:04:42.322 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/nfs/rpc_pipefs 2024-05-28T06:04:42.322 INFO:teuthology.orchestra.run.smithi123.stdout:/run/user/1000 2024-05-28T06:04:42.322 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph 2024-05-28T06:04:42.323 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-4 2024-05-28T06:04:42.323 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-5 2024-05-28T06:04:42.323 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-6 2024-05-28T06:04:42.323 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-7 2024-05-28T06:04:42.323 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns 2024-05-28T06:04:42.323 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-05-28T06:04:42.323 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-05-28T06:04:42.323 INFO:teuthology.orchestra.run.smithi123.stdout:/home/ubuntu/cephtest/mnt.1 2024-05-28T06:04:42.324 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-05-28T06:04:42.324 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-05-28T06:04:42.325 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:04:42.325 DEBUG:teuthology.orchestra.run.smithi123:> ls /sys/fs/fuse/connections 2024-05-28T06:04:42.377 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi123.stderr:2024-05-28T06:04:42.376+0000 7f3e3be24480 -1 init, newargv = 0x55a35b138a80 newargc=15 2024-05-28T06:04:42.378 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi123.stderr:ceph-fuse[43552]: starting ceph client 2024-05-28T06:04:42.387 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi123.stderr:ceph-fuse[43552]: starting fuse 2024-05-28T06:04:42.412 INFO:teuthology.orchestra.run.smithi123.stdout:40 2024-05-28T06:04:42.412 INFO:teuthology.orchestra.run.smithi123.stdout:43 2024-05-28T06:04:42.412 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40, 43] 2024-05-28T06:04:42.413 DEBUG:teuthology.orchestra.run.smithi123:> sudo stdin-killer -- python3 -c ' 2024-05-28T06:04:42.413 DEBUG:teuthology.orchestra.run.smithi123:> import glob 2024-05-28T06:04:42.413 DEBUG:teuthology.orchestra.run.smithi123:> import re 2024-05-28T06:04:42.413 DEBUG:teuthology.orchestra.run.smithi123:> import os 2024-05-28T06:04:42.413 DEBUG:teuthology.orchestra.run.smithi123:> import subprocess 2024-05-28T06:04:42.413 DEBUG:teuthology.orchestra.run.smithi123:> 2024-05-28T06:04:42.413 DEBUG:teuthology.orchestra.run.smithi123:> def _find_admin_socket(client_name): 2024-05-28T06:04:42.413 DEBUG:teuthology.orchestra.run.smithi123:> asok_path = "/var/run/ceph/ceph-client.3.*.asok" 2024-05-28T06:04:42.413 DEBUG:teuthology.orchestra.run.smithi123:> files = glob.glob(asok_path) 2024-05-28T06:04:42.414 DEBUG:teuthology.orchestra.run.smithi123:> mountpoint = "/home/ubuntu/cephtest/mnt.3" 2024-05-28T06:04:42.414 DEBUG:teuthology.orchestra.run.smithi123:> 2024-05-28T06:04:42.414 DEBUG:teuthology.orchestra.run.smithi123:> # Given a non-glob path, it better be there 2024-05-28T06:04:42.414 DEBUG:teuthology.orchestra.run.smithi123:> if "*" not in asok_path: 2024-05-28T06:04:42.414 DEBUG:teuthology.orchestra.run.smithi123:> assert(len(files) == 1) 2024-05-28T06:04:42.414 DEBUG:teuthology.orchestra.run.smithi123:> return files[0] 2024-05-28T06:04:42.414 DEBUG:teuthology.orchestra.run.smithi123:> 2024-05-28T06:04:42.414 DEBUG:teuthology.orchestra.run.smithi123:> for f in files: 2024-05-28T06:04:42.414 DEBUG:teuthology.orchestra.run.smithi123:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-05-28T06:04:42.414 DEBUG:teuthology.orchestra.run.smithi123:> if os.path.exists("/proc/{0}".format(pid)): 2024-05-28T06:04:42.414 DEBUG:teuthology.orchestra.run.smithi123:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-05-28T06:04:42.414 DEBUG:teuthology.orchestra.run.smithi123:> contents = proc_f.read() 2024-05-28T06:04:42.414 DEBUG:teuthology.orchestra.run.smithi123:> if mountpoint in contents: 2024-05-28T06:04:42.415 DEBUG:teuthology.orchestra.run.smithi123:> return f 2024-05-28T06:04:42.415 DEBUG:teuthology.orchestra.run.smithi123:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-05-28T06:04:42.415 DEBUG:teuthology.orchestra.run.smithi123:> 2024-05-28T06:04:42.415 DEBUG:teuthology.orchestra.run.smithi123:> print(_find_admin_socket("client.3")) 2024-05-28T06:04:42.415 DEBUG:teuthology.orchestra.run.smithi123:> ' 2024-05-28T06:04:42.607 INFO:teuthology.orchestra.run.smithi123.stdout:/var/run/ceph/ceph-client.3.43552.asok 2024-05-28T06:04:42.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-05-28T06:04:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-28T06:04:42.624 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.3.43552.asok 2024-05-28T06:04:42.624 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:04:42.625 DEBUG:teuthology.orchestra.run.smithi123:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.3.43552.asok status 2024-05-28T06:04:42.852 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-05-28T06:04:42.852 INFO:teuthology.orchestra.run.smithi123.stdout: "metadata": { 2024-05-28T06:04:42.852 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph_sha1": "9ed025fbf4406fb54980a0647506b15895b6d863", 2024-05-28T06:04:42.852 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph_version": "ceph version 19.0.0-3154-g9ed025fb (9ed025fbf4406fb54980a0647506b15895b6d863) squid (dev)", 2024-05-28T06:04:42.852 INFO:teuthology.orchestra.run.smithi123.stdout: "entity_id": "3", 2024-05-28T06:04:42.852 INFO:teuthology.orchestra.run.smithi123.stdout: "hostname": "smithi123", 2024-05-28T06:04:42.852 INFO:teuthology.orchestra.run.smithi123.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.3", 2024-05-28T06:04:42.852 INFO:teuthology.orchestra.run.smithi123.stdout: "pid": "43552", 2024-05-28T06:04:42.853 INFO:teuthology.orchestra.run.smithi123.stdout: "root": "/" 2024-05-28T06:04:42.853 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-05-28T06:04:42.853 INFO:teuthology.orchestra.run.smithi123.stdout: "dentry_count": 0, 2024-05-28T06:04:42.853 INFO:teuthology.orchestra.run.smithi123.stdout: "dentry_pinned_count": 0, 2024-05-28T06:04:42.853 INFO:teuthology.orchestra.run.smithi123.stdout: "id": 4546, 2024-05-28T06:04:42.853 INFO:teuthology.orchestra.run.smithi123.stdout: "inst": { 2024-05-28T06:04:42.853 INFO:teuthology.orchestra.run.smithi123.stdout: "name": { 2024-05-28T06:04:42.853 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "client", 2024-05-28T06:04:42.853 INFO:teuthology.orchestra.run.smithi123.stdout: "num": 4546 2024-05-28T06:04:42.853 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-05-28T06:04:42.853 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": { 2024-05-28T06:04:42.853 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "v1", 2024-05-28T06:04:42.853 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": "192.168.0.2:0", 2024-05-28T06:04:42.853 INFO:teuthology.orchestra.run.smithi123.stdout: "nonce": 3230335154 2024-05-28T06:04:42.854 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-05-28T06:04:42.854 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-05-28T06:04:42.854 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": { 2024-05-28T06:04:42.854 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "v1", 2024-05-28T06:04:42.854 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": "192.168.0.2:0", 2024-05-28T06:04:42.854 INFO:teuthology.orchestra.run.smithi123.stdout: "nonce": 3230335154 2024-05-28T06:04:42.854 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-05-28T06:04:42.854 INFO:teuthology.orchestra.run.smithi123.stdout: "inst_str": "client.4546 192.168.0.2:0/3230335154", 2024-05-28T06:04:42.854 INFO:teuthology.orchestra.run.smithi123.stdout: "addr_str": "192.168.0.2:0/3230335154", 2024-05-28T06:04:42.854 INFO:teuthology.orchestra.run.smithi123.stdout: "inode_count": 1, 2024-05-28T06:04:42.854 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_epoch": 7, 2024-05-28T06:04:42.854 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_epoch": 20, 2024-05-28T06:04:42.854 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_epoch_barrier": 0, 2024-05-28T06:04:42.855 INFO:teuthology.orchestra.run.smithi123.stdout: "blocklisted": false, 2024-05-28T06:04:42.855 INFO:teuthology.orchestra.run.smithi123.stdout: "fs_name": "cephfs" 2024-05-28T06:04:42.855 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-05-28T06:04:42.867 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:04:42.867 DEBUG:teuthology.orchestra.run.smithi096:> stat --file-system '--printf=%T 2024-05-28T06:04:42.867 DEBUG:teuthology.orchestra.run.smithi096:> ' -- /home/ubuntu/cephtest/mnt.0 2024-05-28T06:04:42.927 INFO:teuthology.orchestra.run.smithi096.stdout:fuseblk 2024-05-28T06:04:42.927 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-05-28T06:04:42.927 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:04:42.927 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-05-28T06:04:43.011 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:04:43.011 DEBUG:teuthology.orchestra.run.smithi096:> stat --file-system '--printf=%T 2024-05-28T06:04:43.011 DEBUG:teuthology.orchestra.run.smithi096:> ' -- /home/ubuntu/cephtest/mnt.2 2024-05-28T06:04:43.061 INFO:teuthology.orchestra.run.smithi096.stdout:fuseblk 2024-05-28T06:04:43.062 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.2 2024-05-28T06:04:43.062 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:04:43.062 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2024-05-28T06:04:43.146 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:04:43.146 DEBUG:teuthology.orchestra.run.smithi123:> stat --file-system '--printf=%T 2024-05-28T06:04:43.146 DEBUG:teuthology.orchestra.run.smithi123:> ' -- /home/ubuntu/cephtest/mnt.1 2024-05-28T06:04:43.204 INFO:teuthology.orchestra.run.smithi123.stdout:fuseblk 2024-05-28T06:04:43.204 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-05-28T06:04:43.204 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:04:43.205 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-05-28T06:04:43.294 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:04:43.294 DEBUG:teuthology.orchestra.run.smithi123:> stat --file-system '--printf=%T 2024-05-28T06:04:43.294 DEBUG:teuthology.orchestra.run.smithi123:> ' -- /home/ubuntu/cephtest/mnt.3 2024-05-28T06:04:43.347 INFO:teuthology.orchestra.run.smithi123.stdout:fuseblk 2024-05-28T06:04:43.347 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.3 2024-05-28T06:04:43.347 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:04:43.347 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2024-05-28T06:04:43.431 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2024-05-28T06:04:43.461 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:04:43.669 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:43.668+0000 7faec2200640 1 Processor -- start 2024-05-28T06:04:43.670 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:43.668+0000 7faec2200640 1 -- start start 2024-05-28T06:04:43.671 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:43.670+0000 7faec2200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faebc06ee40 0x7faebc0ad230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:43.671 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:43.670+0000 7faec2200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7faebc06f2c0 0x7faebc0ad770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:43.671 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:43.670+0000 7faec2200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faebc156f10 con 0x7faebc156b30 2024-05-28T06:04:43.671 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:43.670+0000 7faec2200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7faebc06dda0 con 0x7faebc06f2c0 2024-05-28T06:04:43.671 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:43.670+0000 7faec2200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7faebc06e080 con 0x7faebc06ee40 2024-05-28T06:04:43.671 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:43.670+0000 7faec0e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7faebc156b30 0x7faebc0af6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:33438/0 (socket says 172.21.15.96:33438) 2024-05-28T06:04:43.672 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:43.670+0000 7faec0e00640 1 -- 172.21.15.96:0/352585560 learned_addr learned my addr 172.21.15.96:0/352585560 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:04:43.672 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:43.670+0000 7faec1800640 1 --2- 172.21.15.96:0/352585560 >> v2:172.21.15.96:3300/0 conn(0x7faebc06f2c0 0x7faebc0ad770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:43.672 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:43.670+0000 7faebb400640 1 -- 172.21.15.96:0/352585560 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4247453313 0 0) 0x7faebc156f10 con 0x7faebc156b30 2024-05-28T06:04:43.673 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:43.670+0000 7faebbe00640 1 --2- 172.21.15.96:0/352585560 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faebc06ee40 0x7faebc0ad230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:43.673 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:43.670+0000 7faebb400640 1 -- 172.21.15.96:0/352585560 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faea4003520 con 0x7faebc156b30 2024-05-28T06:04:43.673 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:43.670+0000 7faec1800640 1 -- 172.21.15.96:0/352585560 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faebc06ee40 msgr2=0x7faebc0ad230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:43.673 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:43.670+0000 7faec1800640 1 --2- 172.21.15.96:0/352585560 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faebc06ee40 0x7faebc0ad230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:43.673 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:43.671+0000 7faec1800640 1 -- 172.21.15.96:0/352585560 >> v1:172.21.15.96:6789/0 conn(0x7faebc156b30 legacy=0x7faebc0af6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:43.673 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:43.671+0000 7faec1800640 1 -- 172.21.15.96:0/352585560 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faebc06e360 con 0x7faebc06f2c0 2024-05-28T06:04:43.673 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:43.671+0000 7faebbe00640 1 --2- 172.21.15.96:0/352585560 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faebc06ee40 0x7faebc0ad230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:04:43.673 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:43.672+0000 7faec1800640 1 --2- 172.21.15.96:0/352585560 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faebc06f2c0 0x7faebc0ad770 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7faeb4009d90 tx=0x7faeb402a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f71a1df9caf8131 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:43.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:43.672+0000 7faebb400640 1 -- 172.21.15.96:0/352585560 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeb4031600 con 0x7faebc06f2c0 2024-05-28T06:04:43.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:43.672+0000 7faebb400640 1 -- 172.21.15.96:0/352585560 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faeb40317a0 con 0x7faebc06f2c0 2024-05-28T06:04:43.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:43.672+0000 7faebb400640 1 -- 172.21.15.96:0/352585560 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeb4031b70 con 0x7faebc06f2c0 2024-05-28T06:04:43.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:43.672+0000 7faec2200640 1 -- 172.21.15.96:0/352585560 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faebc06f2c0 msgr2=0x7faebc0ad770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:43.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:43.673+0000 7faec2200640 1 --2- 172.21.15.96:0/352585560 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faebc06f2c0 0x7faebc0ad770 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7faeb4009d90 tx=0x7faeb402a5d0 comp rx=0 tx=0).stop 2024-05-28T06:04:43.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:43.673+0000 7faec2200640 1 -- 172.21.15.96:0/352585560 shutdown_connections 2024-05-28T06:04:43.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:43.673+0000 7faec2200640 1 --2- 172.21.15.96:0/352585560 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faebc06f2c0 0x7faebc0ad770 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:43.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:43.673+0000 7faec2200640 1 --2- 172.21.15.96:0/352585560 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faebc06ee40 0x7faebc0ad230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:43.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:43.673+0000 7faec2200640 1 -- 172.21.15.96:0/352585560 >> 172.21.15.96:0/352585560 conn(0x7faebc0b1fc0 msgr2=0x7faebc0b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:43.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:43.673+0000 7faec2200640 1 -- 172.21.15.96:0/352585560 shutdown_connections 2024-05-28T06:04:43.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:43.673+0000 7faec2200640 1 -- 172.21.15.96:0/352585560 wait complete. 2024-05-28T06:04:43.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:43.674+0000 7faec2200640 1 Processor -- start 2024-05-28T06:04:43.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:43.675+0000 7faec2200640 1 -- start start 2024-05-28T06:04:43.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:43.675+0000 7faec2200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faebc06ee40 0x7faebc099a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:43.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:43.676+0000 7faec2200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faebc06f2c0 0x7faebc15f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:43.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:43.676+0000 7faec2200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faebc156b30 0x7faebc15d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:43.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:43.676+0000 7faec2200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7faebc0b2db0 con 0x7faebc156b30 2024-05-28T06:04:43.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:43.676+0000 7faec2200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7faebc0b2c30 con 0x7faebc06f2c0 2024-05-28T06:04:43.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:43.676+0000 7faec2200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7faebc0b32c0 con 0x7faebc06ee40 2024-05-28T06:04:43.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:43.676+0000 7faec1800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faebc156b30 0x7faebc15d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:43.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:43.676+0000 7faec1800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faebc156b30 0x7faebc15d2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47674/0 (socket says 172.21.15.96:47674) 2024-05-28T06:04:43.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:43.676+0000 7faec1800640 1 -- 172.21.15.96:0/2703425162 learned_addr learned my addr 172.21.15.96:0/2703425162 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:43.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:43.676+0000 7faec1800640 1 -- 172.21.15.96:0/2703425162 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faebc06ee40 msgr2=0x7faebc099a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:43.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:43.676+0000 7faec0e00640 1 --2- 172.21.15.96:0/2703425162 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faebc06ee40 0x7faebc099a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:43.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:43.676+0000 7faebbe00640 1 --2- 172.21.15.96:0/2703425162 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faebc06f2c0 0x7faebc15f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:43.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:43.676+0000 7faec1800640 1 --2- 172.21.15.96:0/2703425162 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faebc06ee40 0x7faebc099a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:43.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:43.676+0000 7faec1800640 1 -- 172.21.15.96:0/2703425162 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faebc06f2c0 msgr2=0x7faebc15f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:43.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:43.676+0000 7faec1800640 1 --2- 172.21.15.96:0/2703425162 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faebc06f2c0 0x7faebc15f450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:43.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:43.676+0000 7faec1800640 1 -- 172.21.15.96:0/2703425162 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faebc157e20 con 0x7faebc156b30 2024-05-28T06:04:43.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:43.677+0000 7faec1800640 1 --2- 172.21.15.96:0/2703425162 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faebc156b30 0x7faebc15d2f0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7faeb402abc0 tx=0x7faeb4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:43.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:43.677+0000 7faeb8c00640 1 -- 172.21.15.96:0/2703425162 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeb4040070 con 0x7faebc156b30 2024-05-28T06:04:43.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:43.677+0000 7faeb8c00640 1 -- 172.21.15.96:0/2703425162 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faeb403c830 con 0x7faebc156b30 2024-05-28T06:04:43.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:43.677+0000 7faeb8c00640 1 -- 172.21.15.96:0/2703425162 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeb403cbe0 con 0x7faebc156b30 2024-05-28T06:04:43.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:43.677+0000 7faec2200640 1 -- 172.21.15.96:0/2703425162 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faebc158100 con 0x7faebc156b30 2024-05-28T06:04:43.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:43.677+0000 7faec2200640 1 -- 172.21.15.96:0/2703425162 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faebc1586c0 con 0x7faebc156b30 2024-05-28T06:04:43.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:43.678+0000 7faec2200640 1 -- 172.21.15.96:0/2703425162 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faebc0689e0 con 0x7faebc156b30 2024-05-28T06:04:43.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:43.679+0000 7faeb8c00640 1 -- 172.21.15.96:0/2703425162 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7faeb4039040 con 0x7faebc156b30 2024-05-28T06:04:43.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:43.681+0000 7faeb8c00640 1 --2- 172.21.15.96:0/2703425162 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fae9c06c8f0 0x7fae9c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:43.683 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:43.681+0000 7faeb8c00640 1 -- 172.21.15.96:0/2703425162 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7faeb40ad150 con 0x7faebc156b30 2024-05-28T06:04:43.683 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:43.681+0000 7faec0e00640 1 --2- 172.21.15.96:0/2703425162 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fae9c06c8f0 0x7fae9c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:43.683 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:43.682+0000 7faec0e00640 1 --2- 172.21.15.96:0/2703425162 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fae9c06c8f0 0x7fae9c06edb0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7faeb0000f60 tx=0x7faeb0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:43.687 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:43.686+0000 7faeb8c00640 1 -- 172.21.15.96:0/2703425162 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7faeb407b600 con 0x7faebc156b30 2024-05-28T06:04:43.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:43.886+0000 7faec2200640 1 -- 172.21.15.96:0/2703425162 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7faebc14bd20 con 0x7faebc156b30 2024-05-28T06:04:43.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:43.889+0000 7faeb8c00640 1 -- 172.21.15.96:0/2703425162 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16617 (secure 0 0 0) 0x7faeb407af00 con 0x7faebc156b30 2024-05-28T06:04:43.891 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:04:43.891 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":20,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:04:11.882922+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-28T06:04:08.741938+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-28T06:04:09.301470+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:04:43.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:43.891+0000 7faec2200640 1 -- 172.21.15.96:0/2703425162 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fae9c06c8f0 msgr2=0x7fae9c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:43.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:43.891+0000 7faec2200640 1 --2- 172.21.15.96:0/2703425162 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fae9c06c8f0 0x7fae9c06edb0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7faeb0000f60 tx=0x7faeb0009040 comp rx=0 tx=0).stop 2024-05-28T06:04:43.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:43.891+0000 7faec2200640 1 -- 172.21.15.96:0/2703425162 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faebc156b30 msgr2=0x7faebc15d2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:43.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:43.891+0000 7faec2200640 1 --2- 172.21.15.96:0/2703425162 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faebc156b30 0x7faebc15d2f0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7faeb402abc0 tx=0x7faeb4004a40 comp rx=0 tx=0).stop 2024-05-28T06:04:43.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:43.892+0000 7faec2200640 1 -- 172.21.15.96:0/2703425162 shutdown_connections 2024-05-28T06:04:43.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:43.892+0000 7faec2200640 1 --2- 172.21.15.96:0/2703425162 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fae9c06c8f0 0x7fae9c06edb0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:43.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:43.892+0000 7faec2200640 1 --2- 172.21.15.96:0/2703425162 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faebc156b30 0x7faebc15d2f0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:43.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:43.892+0000 7faec2200640 1 --2- 172.21.15.96:0/2703425162 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faebc06f2c0 0x7faebc15f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:43.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:43.892+0000 7faec2200640 1 --2- 172.21.15.96:0/2703425162 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faebc06ee40 0x7faebc099a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:43.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:43.892+0000 7faec2200640 1 -- 172.21.15.96:0/2703425162 >> 172.21.15.96:0/2703425162 conn(0x7faebc0b1fc0 msgr2=0x7faebc15b7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:43.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:43.892+0000 7faec2200640 1 -- 172.21.15.96:0/2703425162 shutdown_connections 2024-05-28T06:04:43.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:43.892+0000 7faec2200640 1 -- 172.21.15.96:0/2703425162 wait complete. 2024-05-28T06:04:43.913 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-28T06:03:59.517102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-28T06:04:08.741938+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-28T06:04:09.301470+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-28T06:04:43.914 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-28T06:04:44.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.106+0000 7f3dbf200640 1 Processor -- start 2024-05-28T06:04:44.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.106+0000 7f3dbf200640 1 -- start start 2024-05-28T06:04:44.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.108+0000 7f3dbf200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3dc0153510 0x7f3dc00b4d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:44.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.108+0000 7f3dbf200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f3dc015d270 0x7f3dc00ccff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:44.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.108+0000 7f3dbf200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3dc015d900 con 0x7f3dc00ccb90 2024-05-28T06:04:44.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.108+0000 7f3dbf200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f3dc01538f0 con 0x7f3dc015d270 2024-05-28T06:04:44.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.108+0000 7f3dbf200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f3dc00cd530 con 0x7f3dc0153510 2024-05-28T06:04:44.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.108+0000 7f3dbe800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f3dc015d270 0x7f3dc00ccff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:44.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.108+0000 7f3dbe800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f3dc015d270 0x7f3dc00ccff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47684/0 (socket says 172.21.15.96:47684) 2024-05-28T06:04:44.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.108+0000 7f3dbe800640 1 -- 172.21.15.96:0/2256261483 learned_addr learned my addr 172.21.15.96:0/2256261483 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:44.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.108+0000 7f3dbde00640 1 --2- 172.21.15.96:0/2256261483 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3dc0153510 0x7f3dc00b4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:44.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.108+0000 7f3dbe800640 1 -- 172.21.15.96:0/2256261483 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3dc0153510 msgr2=0x7f3dc00b4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:44.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.109+0000 7f3dbe800640 1 --2- 172.21.15.96:0/2256261483 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3dc0153510 0x7f3dc00b4d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:44.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.109+0000 7f3dbe800640 1 -- 172.21.15.96:0/2256261483 >> v1:172.21.15.96:6789/0 conn(0x7f3dc00ccb90 legacy=0x7f3dc015cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:44.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.109+0000 7f3dbe800640 1 -- 172.21.15.96:0/2256261483 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dc00b3f90 con 0x7f3dc015d270 2024-05-28T06:04:44.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.109+0000 7f3dbde00640 1 --2- 172.21.15.96:0/2256261483 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3dc0153510 0x7f3dc00b4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:04:44.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.110+0000 7f3dbe800640 1 --2- 172.21.15.96:0/2256261483 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3dc015d270 0x7f3dc00ccff0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f3db8009870 tx=0x7f3db802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65430b9a1452d8a1 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:44.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.110+0000 7f3dbca00640 1 -- 172.21.15.96:0/2256261483 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3db8037070 con 0x7f3dc015d270 2024-05-28T06:04:44.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.110+0000 7f3dbca00640 1 -- 172.21.15.96:0/2256261483 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3db8032440 con 0x7f3dc015d270 2024-05-28T06:04:44.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.110+0000 7f3dbca00640 1 -- 172.21.15.96:0/2256261483 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3db8032810 con 0x7f3dc015d270 2024-05-28T06:04:44.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.110+0000 7f3dbf200640 1 -- 172.21.15.96:0/2256261483 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3dc015d270 msgr2=0x7f3dc00ccff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:44.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.110+0000 7f3dbf200640 1 --2- 172.21.15.96:0/2256261483 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3dc015d270 0x7f3dc00ccff0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f3db8009870 tx=0x7f3db802a1e0 comp rx=0 tx=0).stop 2024-05-28T06:04:44.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.111+0000 7f3dbf200640 1 -- 172.21.15.96:0/2256261483 shutdown_connections 2024-05-28T06:04:44.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.111+0000 7f3dbf200640 1 --2- 172.21.15.96:0/2256261483 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3dc015d270 0x7f3dc00ccff0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:44.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.111+0000 7f3dbf200640 1 --2- 172.21.15.96:0/2256261483 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3dc0153510 0x7f3dc00b4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:44.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.111+0000 7f3dbf200640 1 -- 172.21.15.96:0/2256261483 >> 172.21.15.96:0/2256261483 conn(0x7f3dc006b900 msgr2=0x7f3dc006bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:44.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.111+0000 7f3dbf200640 1 -- 172.21.15.96:0/2256261483 shutdown_connections 2024-05-28T06:04:44.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.111+0000 7f3dbf200640 1 -- 172.21.15.96:0/2256261483 wait complete. 2024-05-28T06:04:44.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.112+0000 7f3dbf200640 1 Processor -- start 2024-05-28T06:04:44.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.113+0000 7f3dbf200640 1 -- start start 2024-05-28T06:04:44.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.113+0000 7f3dbf200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3dc00ccb90 0x7f3dc00ad130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:44.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.113+0000 7f3dbf200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3dc0153510 0x7f3dc00ce890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:44.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.113+0000 7f3dbf200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3dc015d270 0x7f3dc00cedd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:44.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.113+0000 7f3dbf200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f3dc00b4550 con 0x7f3dc015d270 2024-05-28T06:04:44.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.113+0000 7f3dbf200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f3dc00b48c0 con 0x7f3dc00ccb90 2024-05-28T06:04:44.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.113+0000 7f3dbf200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f3dc006e940 con 0x7f3dc0153510 2024-05-28T06:04:44.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.114+0000 7f3dbe800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3dc015d270 0x7f3dc00cedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:44.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.114+0000 7f3dbe800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3dc015d270 0x7f3dc00cedd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47690/0 (socket says 172.21.15.96:47690) 2024-05-28T06:04:44.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.114+0000 7f3dbe800640 1 -- 172.21.15.96:0/515361668 learned_addr learned my addr 172.21.15.96:0/515361668 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:44.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.114+0000 7f3dbe800640 1 -- 172.21.15.96:0/515361668 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3dc0153510 msgr2=0x7f3dc00ce890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:44.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.114+0000 7f3dbd400640 1 --2- 172.21.15.96:0/515361668 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3dc0153510 0x7f3dc00ce890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:44.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.114+0000 7f3dbde00640 1 --2- 172.21.15.96:0/515361668 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3dc00ccb90 0x7f3dc00ad130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:44.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.114+0000 7f3dbe800640 1 --2- 172.21.15.96:0/515361668 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3dc0153510 0x7f3dc00ce890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:44.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.114+0000 7f3dbe800640 1 -- 172.21.15.96:0/515361668 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3dc00ccb90 msgr2=0x7f3dc00ad130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:44.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.114+0000 7f3dbe800640 1 --2- 172.21.15.96:0/515361668 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3dc00ccb90 0x7f3dc00ad130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:44.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.114+0000 7f3dbe800640 1 -- 172.21.15.96:0/515361668 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dc00cf650 con 0x7f3dc015d270 2024-05-28T06:04:44.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.114+0000 7f3dbde00640 1 --2- 172.21.15.96:0/515361668 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3dc00ccb90 0x7f3dc00ad130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:04:44.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.115+0000 7f3dbe800640 1 --2- 172.21.15.96:0/515361668 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3dc015d270 0x7f3dc00cedd0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f3db8009870 tx=0x7f3db80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:44.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.115+0000 7f3db6000640 1 -- 172.21.15.96:0/515361668 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3db8037040 con 0x7f3dc015d270 2024-05-28T06:04:44.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.115+0000 7f3db6000640 1 -- 172.21.15.96:0/515361668 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3db8042070 con 0x7f3dc015d270 2024-05-28T06:04:44.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.115+0000 7f3dbf200640 1 -- 172.21.15.96:0/515361668 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3dc0154bc0 con 0x7f3dc015d270 2024-05-28T06:04:44.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.115+0000 7f3db6000640 1 -- 172.21.15.96:0/515361668 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3db8030bd0 con 0x7f3dc015d270 2024-05-28T06:04:44.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.115+0000 7f3dbf200640 1 -- 172.21.15.96:0/515361668 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3dc0155180 con 0x7f3dc015d270 2024-05-28T06:04:44.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.116+0000 7f3dbf200640 1 -- 172.21.15.96:0/515361668 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3dc00689e0 con 0x7f3dc015d270 2024-05-28T06:04:44.119 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.117+0000 7f3db6000640 1 -- 172.21.15.96:0/515361668 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3db8047460 con 0x7f3dc015d270 2024-05-28T06:04:44.119 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.118+0000 7f3db6000640 1 --2- 172.21.15.96:0/515361668 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3d9c06c8f0 0x7f3d9c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:44.119 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.118+0000 7f3db6000640 1 -- 172.21.15.96:0/515361668 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f3db80ac890 con 0x7f3dc015d270 2024-05-28T06:04:44.120 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.119+0000 7f3dbde00640 1 --2- 172.21.15.96:0/515361668 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3d9c06c8f0 0x7f3d9c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:44.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.120+0000 7f3dbde00640 1 --2- 172.21.15.96:0/515361668 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3d9c06c8f0 0x7f3d9c06edb0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f3da8006fd0 tx=0x7f3da8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:44.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.123+0000 7f3db6000640 1 -- 172.21.15.96:0/515361668 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3db807ac20 con 0x7f3dc015d270 2024-05-28T06:04:44.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.317+0000 7f3dbf200640 1 -- 172.21.15.96:0/515361668 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f3dc006be10 con 0x7f3dc015d270 2024-05-28T06:04:44.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.318+0000 7f3db6000640 1 -- 172.21.15.96:0/515361668 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) ==== 93+0+10 (secure 0 0 0) 0x7f3db807a520 con 0x7f3dc015d270 2024-05-28T06:04:44.319 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 1 2024-05-28T06:04:44.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.320+0000 7f3dbf200640 1 -- 172.21.15.96:0/515361668 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3d9c06c8f0 msgr2=0x7f3d9c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:44.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.320+0000 7f3dbf200640 1 --2- 172.21.15.96:0/515361668 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3d9c06c8f0 0x7f3d9c06edb0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f3da8006fd0 tx=0x7f3da8008040 comp rx=0 tx=0).stop 2024-05-28T06:04:44.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.320+0000 7f3dbf200640 1 -- 172.21.15.96:0/515361668 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3dc015d270 msgr2=0x7f3dc00cedd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:44.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.320+0000 7f3dbf200640 1 --2- 172.21.15.96:0/515361668 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3dc015d270 0x7f3dc00cedd0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f3db8009870 tx=0x7f3db80046c0 comp rx=0 tx=0).stop 2024-05-28T06:04:44.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.320+0000 7f3dbf200640 1 -- 172.21.15.96:0/515361668 shutdown_connections 2024-05-28T06:04:44.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.320+0000 7f3dbf200640 1 --2- 172.21.15.96:0/515361668 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3d9c06c8f0 0x7f3d9c06edb0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:44.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.320+0000 7f3dbf200640 1 --2- 172.21.15.96:0/515361668 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3dc015d270 0x7f3dc00cedd0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:44.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.320+0000 7f3dbf200640 1 --2- 172.21.15.96:0/515361668 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3dc0153510 0x7f3dc00ce890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:44.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.320+0000 7f3dbf200640 1 --2- 172.21.15.96:0/515361668 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3dc00ccb90 0x7f3dc00ad130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:44.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.320+0000 7f3dbf200640 1 -- 172.21.15.96:0/515361668 >> 172.21.15.96:0/515361668 conn(0x7f3dc006b900 msgr2=0x7f3dc015c7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:44.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.321+0000 7f3dbf200640 1 -- 172.21.15.96:0/515361668 shutdown_connections 2024-05-28T06:04:44.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.321+0000 7f3dbf200640 1 -- 172.21.15.96:0/515361668 wait complete. 2024-05-28T06:04:44.340 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-28T06:04:44.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.540+0000 7f75da000640 1 Processor -- start 2024-05-28T06:04:44.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.540+0000 7f75da000640 1 -- start start 2024-05-28T06:04:44.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.542+0000 7f75da000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f75d4144da0 0x7f75d4070750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:44.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.542+0000 7f75da000640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75d415f310 con 0x7f75d415ee00 2024-05-28T06:04:44.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.542+0000 7f75da000640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75d415f5f0 con 0x7f75d406c010 2024-05-28T06:04:44.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.542+0000 7f75da000640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f75d415f8d0 con 0x7f75d4144da0 2024-05-28T06:04:44.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.542+0000 7f75cbe00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f75d415ee00 0x7f75d406b900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:33470/0 (socket says 172.21.15.96:33470) 2024-05-28T06:04:44.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.542+0000 7f75cbe00640 1 -- 172.21.15.96:0/537570655 learned_addr learned my addr 172.21.15.96:0/537570655 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:04:44.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.543+0000 7f75d3e00640 1 -- 172.21.15.96:0/537570655 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2301466828 0 0) 0x7f75d415f310 con 0x7f75d415ee00 2024-05-28T06:04:44.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.543+0000 7f75d8c00640 1 --2- 172.21.15.96:0/537570655 >> v2:172.21.15.123:3300/0 conn(0x7f75d4144da0 0x7f75d4070750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:44.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.543+0000 7f75d3e00640 1 -- 172.21.15.96:0/537570655 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f75b8003520 con 0x7f75d415ee00 2024-05-28T06:04:44.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.543+0000 7f75d3e00640 1 -- 172.21.15.96:0/537570655 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3541513318 0 0) 0x7f75b8003520 con 0x7f75d415ee00 2024-05-28T06:04:44.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.543+0000 7f75d3e00640 1 -- 172.21.15.96:0/537570655 >> v2:172.21.15.123:3300/0 conn(0x7f75d4144da0 msgr2=0x7f75d4070750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:44.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.543+0000 7f75d3e00640 1 --2- 172.21.15.96:0/537570655 >> v2:172.21.15.123:3300/0 conn(0x7f75d4144da0 0x7f75d4070750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:44.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.543+0000 7f75d3e00640 1 -- 172.21.15.96:0/537570655 >> v1:172.21.15.123:6789/0 conn(0x7f75d406c010 legacy=0x7f75d406f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:44.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.543+0000 7f75d3e00640 1 -- 172.21.15.96:0/537570655 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75d415fbb0 con 0x7f75d415ee00 2024-05-28T06:04:44.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.543+0000 7f75d3e00640 1 -- 172.21.15.96:0/537570655 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f75bc001670 con 0x7f75d415ee00 2024-05-28T06:04:44.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.543+0000 7f75d3e00640 1 -- 172.21.15.96:0/537570655 >> v1:172.21.15.96:6789/0 conn(0x7f75d415ee00 legacy=0x7f75d406b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:44.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.543+0000 7f75d3e00640 1 --2- 172.21.15.96:0/537570655 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f75b8003a00 0x7f75b8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:44.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.544+0000 7f75d3e00640 1 --2- 172.21.15.96:0/537570655 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f75b8004800 0x7f75b8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:44.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.544+0000 7f75d3e00640 1 --2- 172.21.15.96:0/537570655 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f75b80052e0 0x7f75b800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:44.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.544+0000 7f75d3e00640 1 -- 172.21.15.96:0/537570655 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f75d415f8d0 con 0x7f75b80052e0 2024-05-28T06:04:44.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.544+0000 7f75d3e00640 1 -- 172.21.15.96:0/537570655 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f75b800c200 con 0x7f75b8003a00 2024-05-28T06:04:44.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.544+0000 7f75d3e00640 1 -- 172.21.15.96:0/537570655 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f75b800c400 con 0x7f75b8004800 2024-05-28T06:04:44.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.544+0000 7f75d9600640 1 --2- 172.21.15.96:0/537570655 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f75b80052e0 0x7f75b800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:44.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.544+0000 7f75d8c00640 1 --2- 172.21.15.96:0/537570655 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f75b8003a00 0x7f75b8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:44.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.544+0000 7f75cbe00640 1 --2- 172.21.15.96:0/537570655 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f75b8004800 0x7f75b8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:44.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.544+0000 7f75d9600640 1 -- 172.21.15.96:0/537570655 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f75b8004800 msgr2=0x7f75b8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:44.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.544+0000 7f75d9600640 1 --2- 172.21.15.96:0/537570655 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f75b8004800 0x7f75b8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:44.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.544+0000 7f75d9600640 1 -- 172.21.15.96:0/537570655 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f75b8003a00 msgr2=0x7f75b8003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:44.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.544+0000 7f75d9600640 1 --2- 172.21.15.96:0/537570655 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f75b8003a00 0x7f75b8003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:44.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.544+0000 7f75d9600640 1 -- 172.21.15.96:0/537570655 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75b8003520 con 0x7f75b80052e0 2024-05-28T06:04:44.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.545+0000 7f75d9600640 1 -- 172.21.15.96:0/537570655 reap_dead start 2024-05-28T06:04:44.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.545+0000 7f75d9600640 1 --2- 172.21.15.96:0/537570655 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f75b80052e0 0x7f75b800b9c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f75cc001e50 tx=0x7f75b8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84c010e3a82b9ff9 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:44.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.546+0000 7f75d3e00640 1 -- 172.21.15.96:0/537570655 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75cc001a70 con 0x7f75b80052e0 2024-05-28T06:04:44.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.546+0000 7f75d3e00640 1 -- 172.21.15.96:0/537570655 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75cc003970 con 0x7f75b80052e0 2024-05-28T06:04:44.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.546+0000 7f75d3e00640 1 -- 172.21.15.96:0/537570655 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75cc003d60 con 0x7f75b80052e0 2024-05-28T06:04:44.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.546+0000 7f75da000640 1 -- 172.21.15.96:0/537570655 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f75b80052e0 msgr2=0x7f75b800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:44.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.546+0000 7f75da000640 1 --2- 172.21.15.96:0/537570655 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f75b80052e0 0x7f75b800b9c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f75cc001e50 tx=0x7f75b8005830 comp rx=0 tx=0).stop 2024-05-28T06:04:44.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.547+0000 7f75da000640 1 -- 172.21.15.96:0/537570655 shutdown_connections 2024-05-28T06:04:44.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.547+0000 7f75da000640 1 --2- 172.21.15.96:0/537570655 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f75b80052e0 0x7f75b800b9c0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:44.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.547+0000 7f75da000640 1 -- 172.21.15.96:0/537570655 >> 172.21.15.96:0/537570655 conn(0x7f75d40b67f0 msgr2=0x7f75d40b6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:44.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.547+0000 7f75da000640 1 -- 172.21.15.96:0/537570655 shutdown_connections 2024-05-28T06:04:44.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.547+0000 7f75da000640 1 -- 172.21.15.96:0/537570655 wait complete. 2024-05-28T06:04:44.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.548+0000 7f75da000640 1 Processor -- start 2024-05-28T06:04:44.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.549+0000 7f75da000640 1 -- start start 2024-05-28T06:04:44.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.549+0000 7f75da000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f75b80052e0 0x7f75d413c210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:44.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.549+0000 7f75da000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f75d41f78e0 0x7f75d41f80b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:44.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.549+0000 7f75da000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f75d40b0530 0x7f75d4138820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:44.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.549+0000 7f75da000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f75d4160190 con 0x7f75d40b0530 2024-05-28T06:04:44.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.549+0000 7f75da000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f75d4160010 con 0x7f75d41f78e0 2024-05-28T06:04:44.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.549+0000 7f75da000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f75d406b900 con 0x7f75b80052e0 2024-05-28T06:04:44.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.550+0000 7f75d9600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f75d40b0530 0x7f75d4138820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:44.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.550+0000 7f75d9600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f75d40b0530 0x7f75d4138820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47714/0 (socket says 172.21.15.96:47714) 2024-05-28T06:04:44.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.550+0000 7f75d9600640 1 -- 172.21.15.96:0/2519904894 learned_addr learned my addr 172.21.15.96:0/2519904894 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:44.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.550+0000 7f75d8c00640 1 --2- 172.21.15.96:0/2519904894 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f75b80052e0 0x7f75d413c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:44.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.550+0000 7f75d9600640 1 -- 172.21.15.96:0/2519904894 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f75b80052e0 msgr2=0x7f75d413c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:44.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.550+0000 7f75d9600640 1 --2- 172.21.15.96:0/2519904894 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f75b80052e0 0x7f75d413c210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:44.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.550+0000 7f75d9600640 1 -- 172.21.15.96:0/2519904894 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f75d41f78e0 msgr2=0x7f75d41f80b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:44.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.550+0000 7f75d9600640 1 --2- 172.21.15.96:0/2519904894 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f75d41f78e0 0x7f75d41f80b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:44.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.550+0000 7f75d9600640 1 -- 172.21.15.96:0/2519904894 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75d413b170 con 0x7f75d40b0530 2024-05-28T06:04:44.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.550+0000 7f75d8c00640 1 --2- 172.21.15.96:0/2519904894 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f75b80052e0 0x7f75d413c210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:04:44.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.550+0000 7f75d9600640 1 --2- 172.21.15.96:0/2519904894 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f75d40b0530 0x7f75d4138820 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f75cc001f70 tx=0x7f75cc0014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:44.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.551+0000 7f75d1600640 1 -- 172.21.15.96:0/2519904894 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75cc001a70 con 0x7f75d40b0530 2024-05-28T06:04:44.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.551+0000 7f75d1600640 1 -- 172.21.15.96:0/2519904894 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75cc02ac70 con 0x7f75d40b0530 2024-05-28T06:04:44.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.551+0000 7f75d1600640 1 -- 172.21.15.96:0/2519904894 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75cc006d70 con 0x7f75d40b0530 2024-05-28T06:04:44.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.551+0000 7f75da000640 1 -- 172.21.15.96:0/2519904894 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75d413b450 con 0x7f75d40b0530 2024-05-28T06:04:44.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.551+0000 7f75da000640 1 -- 172.21.15.96:0/2519904894 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75d413ba10 con 0x7f75d40b0530 2024-05-28T06:04:44.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.552+0000 7f75da000640 1 -- 172.21.15.96:0/2519904894 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75d40681e0 con 0x7f75d40b0530 2024-05-28T06:04:44.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.553+0000 7f75d1600640 1 -- 172.21.15.96:0/2519904894 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f75cc007030 con 0x7f75d40b0530 2024-05-28T06:04:44.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.554+0000 7f75d1600640 1 --2- 172.21.15.96:0/2519904894 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f75b006c8f0 0x7f75b006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:44.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.554+0000 7f75d1600640 1 -- 172.21.15.96:0/2519904894 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f75cc0ab2f0 con 0x7f75d40b0530 2024-05-28T06:04:44.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.555+0000 7f75d8c00640 1 --2- 172.21.15.96:0/2519904894 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f75b006c8f0 0x7f75b006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:44.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.556+0000 7f75d8c00640 1 --2- 172.21.15.96:0/2519904894 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f75b006c8f0 0x7f75b006edb0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f75c4006fd0 tx=0x7f75c4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:44.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.558+0000 7f75d1600640 1 -- 172.21.15.96:0/2519904894 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f75d40681e0 con 0x7f75d40b0530 2024-05-28T06:04:44.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.760+0000 7f75da000640 1 -- 172.21.15.96:0/2519904894 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f75d40b26f0 con 0x7f75d40b0530 2024-05-28T06:04:44.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.760+0000 7f75d1600640 1 -- 172.21.15.96:0/2519904894 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) ==== 104+0+11 (secure 0 0 0) 0x7f75cc0790c0 con 0x7f75d40b0530 2024-05-28T06:04:44.763 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 32 2024-05-28T06:04:44.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.763+0000 7f75da000640 1 -- 172.21.15.96:0/2519904894 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f75b006c8f0 msgr2=0x7f75b006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:44.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.763+0000 7f75da000640 1 --2- 172.21.15.96:0/2519904894 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f75b006c8f0 0x7f75b006edb0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f75c4006fd0 tx=0x7f75c4008040 comp rx=0 tx=0).stop 2024-05-28T06:04:44.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.763+0000 7f75da000640 1 -- 172.21.15.96:0/2519904894 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f75d40b0530 msgr2=0x7f75d4138820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:44.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.763+0000 7f75da000640 1 --2- 172.21.15.96:0/2519904894 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f75d40b0530 0x7f75d4138820 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f75cc001f70 tx=0x7f75cc0014a0 comp rx=0 tx=0).stop 2024-05-28T06:04:44.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.763+0000 7f75da000640 1 -- 172.21.15.96:0/2519904894 shutdown_connections 2024-05-28T06:04:44.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.763+0000 7f75da000640 1 --2- 172.21.15.96:0/2519904894 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f75b006c8f0 0x7f75b006edb0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:44.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.763+0000 7f75da000640 1 --2- 172.21.15.96:0/2519904894 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f75d40b0530 0x7f75d4138820 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:44.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.763+0000 7f75da000640 1 --2- 172.21.15.96:0/2519904894 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f75d41f78e0 0x7f75d41f80b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:44.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.763+0000 7f75da000640 1 --2- 172.21.15.96:0/2519904894 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f75b80052e0 0x7f75d413c210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:44.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.763+0000 7f75da000640 1 -- 172.21.15.96:0/2519904894 >> 172.21.15.96:0/2519904894 conn(0x7f75d40b67f0 msgr2=0x7f75d40b6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:44.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.763+0000 7f75da000640 1 -- 172.21.15.96:0/2519904894 shutdown_connections 2024-05-28T06:04:44.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.763+0000 7f75da000640 1 -- 172.21.15.96:0/2519904894 wait complete. 2024-05-28T06:04:44.784 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-28T06:04:44.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.993+0000 7ff286600640 1 Processor -- start 2024-05-28T06:04:44.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.993+0000 7ff286600640 1 -- start start 2024-05-28T06:04:44.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.994+0000 7ff286600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7ff28015b6f0 0x7ff28006e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:44.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.994+0000 7ff286600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7ff2800725e0 0x7ff2800ae650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:44.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.994+0000 7ff286600640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2800ade90 con 0x7ff2800d0640 2024-05-28T06:04:44.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.994+0000 7ff286600640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7ff280071c40 con 0x7ff28015b6f0 2024-05-28T06:04:44.998 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.994+0000 7ff286600640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7ff280071e70 con 0x7ff2800725e0 2024-05-28T06:04:44.998 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.995+0000 7ff27f400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7ff2800d0640 0x7ff2800d0a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:33482/0 (socket says 172.21.15.96:33482) 2024-05-28T06:04:44.998 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.995+0000 7ff27f400640 1 -- 172.21.15.96:0/3192952922 learned_addr learned my addr 172.21.15.96:0/3192952922 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:04:44.998 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.995+0000 7ff276a00640 1 --2- 172.21.15.96:0/3192952922 >> v2:172.21.15.96:3300/0 conn(0x7ff28015b6f0 0x7ff28006e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:44.998 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.995+0000 7ff27fe00640 1 --2- 172.21.15.96:0/3192952922 >> v2:172.21.15.123:3300/0 conn(0x7ff2800725e0 0x7ff2800ae650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:44.998 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.995+0000 7ff27ea00640 1 -- 172.21.15.96:0/3192952922 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2344003049 0 0) 0x7ff2800ade90 con 0x7ff2800d0640 2024-05-28T06:04:44.998 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.995+0000 7ff27ea00640 1 -- 172.21.15.96:0/3192952922 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff264003420 con 0x7ff2800d0640 2024-05-28T06:04:44.998 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.995+0000 7ff276a00640 1 -- 172.21.15.96:0/3192952922 >> v2:172.21.15.123:3300/0 conn(0x7ff2800725e0 msgr2=0x7ff2800ae650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:44.998 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.995+0000 7ff276a00640 1 --2- 172.21.15.96:0/3192952922 >> v2:172.21.15.123:3300/0 conn(0x7ff2800725e0 0x7ff2800ae650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:44.999 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.995+0000 7ff276a00640 1 -- 172.21.15.96:0/3192952922 >> v1:172.21.15.96:6789/0 conn(0x7ff2800d0640 legacy=0x7ff2800d0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:44.999 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.995+0000 7ff276a00640 1 -- 172.21.15.96:0/3192952922 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff280072130 con 0x7ff28015b6f0 2024-05-28T06:04:45.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.996+0000 7ff276a00640 1 --2- 172.21.15.96:0/3192952922 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff28015b6f0 0x7ff28006e350 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff2800b3370 tx=0x7ff26802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3941b557accb3178 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:45.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.997+0000 7ff27ea00640 1 -- 172.21.15.96:0/3192952922 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff268037070 con 0x7ff28015b6f0 2024-05-28T06:04:45.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.997+0000 7ff27ea00640 1 -- 172.21.15.96:0/3192952922 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff268032440 con 0x7ff28015b6f0 2024-05-28T06:04:45.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.997+0000 7ff27ea00640 1 -- 172.21.15.96:0/3192952922 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff268032810 con 0x7ff28015b6f0 2024-05-28T06:04:45.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.997+0000 7ff286600640 1 -- 172.21.15.96:0/3192952922 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff28015b6f0 msgr2=0x7ff28006e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:45.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.997+0000 7ff286600640 1 --2- 172.21.15.96:0/3192952922 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff28015b6f0 0x7ff28006e350 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff2800b3370 tx=0x7ff26802a1a0 comp rx=0 tx=0).stop 2024-05-28T06:04:45.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.998+0000 7ff286600640 1 -- 172.21.15.96:0/3192952922 shutdown_connections 2024-05-28T06:04:45.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.998+0000 7ff286600640 1 --2- 172.21.15.96:0/3192952922 >> v2:172.21.15.123:3300/0 conn(0x7ff2800725e0 0x7ff2800ae650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:45.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.998+0000 7ff286600640 1 --2- 172.21.15.96:0/3192952922 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff28015b6f0 0x7ff28006e350 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:45.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.998+0000 7ff286600640 1 -- 172.21.15.96:0/3192952922 >> 172.21.15.96:0/3192952922 conn(0x7ff28006bd20 msgr2=0x7ff28006c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:45.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.998+0000 7ff286600640 1 -- 172.21.15.96:0/3192952922 shutdown_connections 2024-05-28T06:04:45.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.998+0000 7ff286600640 1 -- 172.21.15.96:0/3192952922 wait complete. 2024-05-28T06:04:45.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:44.999+0000 7ff286600640 1 Processor -- start 2024-05-28T06:04:45.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.000+0000 7ff286600640 1 -- start start 2024-05-28T06:04:45.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.000+0000 7ff286600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff2800725e0 0x7ff28014b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:45.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.000+0000 7ff286600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff2800d0640 0x7ff2800cee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:45.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.000+0000 7ff286600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff28015b6f0 0x7ff28015db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:45.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.000+0000 7ff286600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff2800aeb90 con 0x7ff2800d0640 2024-05-28T06:04:45.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.000+0000 7ff286600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff28006e890 con 0x7ff2800725e0 2024-05-28T06:04:45.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.000+0000 7ff286600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ff2800aed10 con 0x7ff28015b6f0 2024-05-28T06:04:45.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.001+0000 7ff276a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff2800d0640 0x7ff2800cee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:45.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.001+0000 7ff276a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff2800d0640 0x7ff2800cee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47738/0 (socket says 172.21.15.96:47738) 2024-05-28T06:04:45.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.001+0000 7ff276a00640 1 -- 172.21.15.96:0/4016164121 learned_addr learned my addr 172.21.15.96:0/4016164121 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:45.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.001+0000 7ff27f400640 1 --2- 172.21.15.96:0/4016164121 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff2800725e0 0x7ff28014b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:45.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.001+0000 7ff276a00640 1 -- 172.21.15.96:0/4016164121 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff28015b6f0 msgr2=0x7ff28015db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:45.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.001+0000 7ff276a00640 1 --2- 172.21.15.96:0/4016164121 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff28015b6f0 0x7ff28015db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:45.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.001+0000 7ff276a00640 1 -- 172.21.15.96:0/4016164121 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff2800725e0 msgr2=0x7ff28014b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:45.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.001+0000 7ff276a00640 1 --2- 172.21.15.96:0/4016164121 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff2800725e0 0x7ff28014b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:45.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.001+0000 7ff276a00640 1 -- 172.21.15.96:0/4016164121 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff280157680 con 0x7ff2800d0640 2024-05-28T06:04:45.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.002+0000 7ff276a00640 1 --2- 172.21.15.96:0/4016164121 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff2800d0640 0x7ff2800cee80 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7ff268000c00 tx=0x7ff2680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:45.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.002+0000 7ff277e00640 1 -- 172.21.15.96:0/4016164121 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff268037040 con 0x7ff2800d0640 2024-05-28T06:04:45.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.002+0000 7ff277e00640 1 -- 172.21.15.96:0/4016164121 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff268042070 con 0x7ff2800d0640 2024-05-28T06:04:45.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.002+0000 7ff277e00640 1 -- 172.21.15.96:0/4016164121 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff268030900 con 0x7ff2800d0640 2024-05-28T06:04:45.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.002+0000 7ff286600640 1 -- 172.21.15.96:0/4016164121 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff28015c8c0 con 0x7ff2800d0640 2024-05-28T06:04:45.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.002+0000 7ff286600640 1 -- 172.21.15.96:0/4016164121 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff280157920 con 0x7ff2800d0640 2024-05-28T06:04:45.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.003+0000 7ff286600640 1 -- 172.21.15.96:0/4016164121 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff28015cde0 con 0x7ff2800d0640 2024-05-28T06:04:45.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.004+0000 7ff277e00640 1 -- 172.21.15.96:0/4016164121 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff268030aa0 con 0x7ff2800d0640 2024-05-28T06:04:45.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.005+0000 7ff277e00640 1 --2- 172.21.15.96:0/4016164121 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff25c06c920 0x7ff25c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:45.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.005+0000 7ff277e00640 1 -- 172.21.15.96:0/4016164121 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7ff2680ac510 con 0x7ff2800d0640 2024-05-28T06:04:45.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.006+0000 7ff27f400640 1 --2- 172.21.15.96:0/4016164121 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff25c06c920 0x7ff25c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:45.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.007+0000 7ff27f400640 1 --2- 172.21.15.96:0/4016164121 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff25c06c920 0x7ff25c06ede0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ff270000f60 tx=0x7ff270009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:45.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.009+0000 7ff277e00640 1 -- 172.21.15.96:0/4016164121 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff28015cde0 con 0x7ff2800d0640 2024-05-28T06:04:45.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.209+0000 7ff286600640 1 -- 172.21.15.96:0/4016164121 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7ff28006c5d0 con 0x7ff2800d0640 2024-05-28T06:04:45.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.210+0000 7ff277e00640 1 -- 172.21.15.96:0/4016164121 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) ==== 100+0+11 (secure 0 0 0) 0x7ff26807a8a0 con 0x7ff2800d0640 2024-05-28T06:04:45.212 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 64 2024-05-28T06:04:45.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.212+0000 7ff286600640 1 -- 172.21.15.96:0/4016164121 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff25c06c920 msgr2=0x7ff25c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:45.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.212+0000 7ff286600640 1 --2- 172.21.15.96:0/4016164121 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff25c06c920 0x7ff25c06ede0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ff270000f60 tx=0x7ff270009040 comp rx=0 tx=0).stop 2024-05-28T06:04:45.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.212+0000 7ff286600640 1 -- 172.21.15.96:0/4016164121 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff2800d0640 msgr2=0x7ff2800cee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:45.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.212+0000 7ff286600640 1 --2- 172.21.15.96:0/4016164121 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff2800d0640 0x7ff2800cee80 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7ff268000c00 tx=0x7ff2680046c0 comp rx=0 tx=0).stop 2024-05-28T06:04:45.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.212+0000 7ff286600640 1 -- 172.21.15.96:0/4016164121 shutdown_connections 2024-05-28T06:04:45.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.212+0000 7ff286600640 1 --2- 172.21.15.96:0/4016164121 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff25c06c920 0x7ff25c06ede0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:45.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.212+0000 7ff286600640 1 --2- 172.21.15.96:0/4016164121 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff28015b6f0 0x7ff28015db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:45.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.212+0000 7ff286600640 1 --2- 172.21.15.96:0/4016164121 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff2800d0640 0x7ff2800cee80 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:45.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.212+0000 7ff286600640 1 --2- 172.21.15.96:0/4016164121 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff2800725e0 0x7ff28014b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:45.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.213+0000 7ff286600640 1 -- 172.21.15.96:0/4016164121 >> 172.21.15.96:0/4016164121 conn(0x7ff28006bd20 msgr2=0x7ff28006f550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:45.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.213+0000 7ff286600640 1 -- 172.21.15.96:0/4016164121 shutdown_connections 2024-05-28T06:04:45.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.213+0000 7ff286600640 1 -- 172.21.15.96:0/4016164121 wait complete. 2024-05-28T06:04:45.234 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:04:45.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.435+0000 7f45d2a00640 1 Processor -- start 2024-05-28T06:04:45.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.435+0000 7f45d2a00640 1 -- start start 2024-05-28T06:04:45.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.436+0000 7f45d2a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f45cc156b30 0x7f45cc0af6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:45.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.436+0000 7f45d2a00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f45cc06ee40 0x7f45cc0ad230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:45.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.436+0000 7f45d2a00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45cc156f10 con 0x7f45cc06f2f0 2024-05-28T06:04:45.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.436+0000 7f45d2a00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f45cc06dda0 con 0x7f45cc156b30 2024-05-28T06:04:45.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.436+0000 7f45d2a00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f45cc06e080 con 0x7f45cc06ee40 2024-05-28T06:04:45.439 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.437+0000 7f45d1600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f45cc156b30 0x7f45cc0af6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:45.439 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.437+0000 7f45d1600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f45cc156b30 0x7f45cc0af6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47748/0 (socket says 172.21.15.96:47748) 2024-05-28T06:04:45.439 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.437+0000 7f45d1600640 1 -- 172.21.15.96:0/2358489226 learned_addr learned my addr 172.21.15.96:0/2358489226 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:45.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.437+0000 7f45d0c00640 1 --2- 172.21.15.96:0/2358489226 >> v2:172.21.15.123:3300/0 conn(0x7f45cc06ee40 0x7f45cc0ad230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:45.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.437+0000 7f45d1600640 1 -- 172.21.15.96:0/2358489226 >> v2:172.21.15.123:3300/0 conn(0x7f45cc06ee40 msgr2=0x7f45cc0ad230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:45.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.437+0000 7f45d1600640 1 --2- 172.21.15.96:0/2358489226 >> v2:172.21.15.123:3300/0 conn(0x7f45cc06ee40 0x7f45cc0ad230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:45.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.437+0000 7f45d1600640 1 -- 172.21.15.96:0/2358489226 >> v1:172.21.15.123:6789/0 conn(0x7f45cc06f2f0 legacy=0x7f45cc0b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:45.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.437+0000 7f45d1600640 1 -- 172.21.15.96:0/2358489226 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45cc06e360 con 0x7f45cc156b30 2024-05-28T06:04:45.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.439+0000 7f45d1600640 1 --2- 172.21.15.96:0/2358489226 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f45cc156b30 0x7f45cc0af6e0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f45cc06fdf0 tx=0x7f45c002a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23e5ae14e879778b server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:45.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.439+0000 7f45cbe00640 1 -- 172.21.15.96:0/2358489226 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45c0037070 con 0x7f45cc156b30 2024-05-28T06:04:45.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.439+0000 7f45cbe00640 1 -- 172.21.15.96:0/2358489226 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45c0032440 con 0x7f45cc156b30 2024-05-28T06:04:45.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.439+0000 7f45cbe00640 1 -- 172.21.15.96:0/2358489226 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45c0032810 con 0x7f45cc156b30 2024-05-28T06:04:45.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.439+0000 7f45d2a00640 1 -- 172.21.15.96:0/2358489226 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f45cc156b30 msgr2=0x7f45cc0af6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:45.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.439+0000 7f45d2a00640 1 --2- 172.21.15.96:0/2358489226 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f45cc156b30 0x7f45cc0af6e0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f45cc06fdf0 tx=0x7f45c002a1a0 comp rx=0 tx=0).stop 2024-05-28T06:04:45.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.440+0000 7f45d2a00640 1 -- 172.21.15.96:0/2358489226 shutdown_connections 2024-05-28T06:04:45.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.440+0000 7f45d2a00640 1 --2- 172.21.15.96:0/2358489226 >> v2:172.21.15.123:3300/0 conn(0x7f45cc06ee40 0x7f45cc0ad230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:45.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.440+0000 7f45d2a00640 1 --2- 172.21.15.96:0/2358489226 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f45cc156b30 0x7f45cc0af6e0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:45.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.440+0000 7f45d2a00640 1 -- 172.21.15.96:0/2358489226 >> 172.21.15.96:0/2358489226 conn(0x7f45cc0b1fc0 msgr2=0x7f45cc0b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:45.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.440+0000 7f45d2a00640 1 -- 172.21.15.96:0/2358489226 shutdown_connections 2024-05-28T06:04:45.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.440+0000 7f45d2a00640 1 -- 172.21.15.96:0/2358489226 wait complete. 2024-05-28T06:04:45.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.441+0000 7f45d2a00640 1 Processor -- start 2024-05-28T06:04:45.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.442+0000 7f45d2a00640 1 -- start start 2024-05-28T06:04:45.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.442+0000 7f45d2a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f45cc06ee40 0x7f45cc145060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:45.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.442+0000 7f45d2a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f45cc06f2f0 0x7f45cc1e6970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:45.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.442+0000 7f45d2a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f45cc156b30 0x7f45cc15ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:45.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.442+0000 7f45d2a00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f45cc0afc20 con 0x7f45cc156b30 2024-05-28T06:04:45.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.442+0000 7f45d2a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f45cc0b2c30 con 0x7f45cc06f2f0 2024-05-28T06:04:45.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.442+0000 7f45d2a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f45cc1455a0 con 0x7f45cc06ee40 2024-05-28T06:04:45.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.443+0000 7f45d2000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f45cc156b30 0x7f45cc15ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:45.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.443+0000 7f45d2000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f45cc156b30 0x7f45cc15ed20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47756/0 (socket says 172.21.15.96:47756) 2024-05-28T06:04:45.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.443+0000 7f45d2000640 1 -- 172.21.15.96:0/4234005841 learned_addr learned my addr 172.21.15.96:0/4234005841 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:45.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.443+0000 7f45d1600640 1 --2- 172.21.15.96:0/4234005841 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f45cc06ee40 0x7f45cc145060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:45.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.443+0000 7f45d0c00640 1 --2- 172.21.15.96:0/4234005841 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f45cc06f2f0 0x7f45cc1e6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:45.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.443+0000 7f45d2000640 1 -- 172.21.15.96:0/4234005841 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f45cc06ee40 msgr2=0x7f45cc145060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:45.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.443+0000 7f45d2000640 1 --2- 172.21.15.96:0/4234005841 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f45cc06ee40 0x7f45cc145060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:45.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.443+0000 7f45d2000640 1 -- 172.21.15.96:0/4234005841 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f45cc06f2f0 msgr2=0x7f45cc1e6970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:45.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.443+0000 7f45d2000640 1 --2- 172.21.15.96:0/4234005841 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f45cc06f2f0 0x7f45cc1e6970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:45.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.443+0000 7f45d2000640 1 -- 172.21.15.96:0/4234005841 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45cc159bd0 con 0x7f45cc156b30 2024-05-28T06:04:45.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.443+0000 7f45d1600640 1 --2- 172.21.15.96:0/4234005841 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f45cc06ee40 0x7f45cc145060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:04:45.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.443+0000 7f45d0c00640 1 --2- 172.21.15.96:0/4234005841 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f45cc06f2f0 0x7f45cc1e6970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:04:45.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.443+0000 7f45d2000640 1 --2- 172.21.15.96:0/4234005841 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f45cc156b30 0x7f45cc15ed20 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f45c400c7b0 tx=0x7f45c400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:45.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.444+0000 7f45c9600640 1 -- 172.21.15.96:0/4234005841 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45c400ce60 con 0x7f45cc156b30 2024-05-28T06:04:45.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.444+0000 7f45c9600640 1 -- 172.21.15.96:0/4234005841 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45c4004b10 con 0x7f45cc156b30 2024-05-28T06:04:45.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.444+0000 7f45c9600640 1 -- 172.21.15.96:0/4234005841 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45c400a850 con 0x7f45cc156b30 2024-05-28T06:04:45.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.444+0000 7f45d2a00640 1 -- 172.21.15.96:0/4234005841 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45cc159050 con 0x7f45cc156b30 2024-05-28T06:04:45.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.444+0000 7f45d2a00640 1 -- 172.21.15.96:0/4234005841 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45cc159610 con 0x7f45cc156b30 2024-05-28T06:04:45.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.445+0000 7f45d2a00640 1 -- 172.21.15.96:0/4234005841 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45cc0689e0 con 0x7f45cc156b30 2024-05-28T06:04:45.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.446+0000 7f45c9600640 1 -- 172.21.15.96:0/4234005841 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f45c400a9f0 con 0x7f45cc156b30 2024-05-28T06:04:45.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.447+0000 7f45c9600640 1 --2- 172.21.15.96:0/4234005841 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f45ac06c8f0 0x7f45ac06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:45.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.447+0000 7f45c9600640 1 -- 172.21.15.96:0/4234005841 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f45c408bdb0 con 0x7f45cc156b30 2024-05-28T06:04:45.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.448+0000 7f45d1600640 1 --2- 172.21.15.96:0/4234005841 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f45ac06c8f0 0x7f45ac06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:45.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.449+0000 7f45d1600640 1 --2- 172.21.15.96:0/4234005841 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f45ac06c8f0 0x7f45ac06edb0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f45c0034000 tx=0x7f45c002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:45.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.452+0000 7f45c9600640 1 -- 172.21.15.96:0/4234005841 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f45c4090070 con 0x7f45cc156b30 2024-05-28T06:04:45.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.657+0000 7f45d2a00640 1 -- 172.21.15.96:0/4234005841 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f45cc06f7f0 con 0x7f45cc156b30 2024-05-28T06:04:45.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.658+0000 7f45c9600640 1 -- 172.21.15.96:0/4234005841 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16617 (secure 0 0 0) 0x7f45c4020070 con 0x7f45cc156b30 2024-05-28T06:04:45.660 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:04:45.660 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":20,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:04:11.882922+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-28T06:04:08.741938+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-28T06:04:09.301470+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:04:45.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.661+0000 7f45d2a00640 1 -- 172.21.15.96:0/4234005841 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f45ac06c8f0 msgr2=0x7f45ac06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:45.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.661+0000 7f45d2a00640 1 --2- 172.21.15.96:0/4234005841 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f45ac06c8f0 0x7f45ac06edb0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f45c0034000 tx=0x7f45c002e000 comp rx=0 tx=0).stop 2024-05-28T06:04:45.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.661+0000 7f45d2a00640 1 -- 172.21.15.96:0/4234005841 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f45cc156b30 msgr2=0x7f45cc15ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:45.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.661+0000 7f45d2a00640 1 --2- 172.21.15.96:0/4234005841 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f45cc156b30 0x7f45cc15ed20 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f45c400c7b0 tx=0x7f45c400cc70 comp rx=0 tx=0).stop 2024-05-28T06:04:45.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.661+0000 7f45d2a00640 1 -- 172.21.15.96:0/4234005841 shutdown_connections 2024-05-28T06:04:45.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.661+0000 7f45d2a00640 1 --2- 172.21.15.96:0/4234005841 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f45ac06c8f0 0x7f45ac06edb0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:45.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.661+0000 7f45d2a00640 1 --2- 172.21.15.96:0/4234005841 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f45cc156b30 0x7f45cc15ed20 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:45.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.661+0000 7f45d2a00640 1 --2- 172.21.15.96:0/4234005841 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f45cc06f2f0 0x7f45cc1e6970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:45.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.661+0000 7f45d2a00640 1 --2- 172.21.15.96:0/4234005841 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f45cc06ee40 0x7f45cc145060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:45.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.661+0000 7f45d2a00640 1 -- 172.21.15.96:0/4234005841 >> 172.21.15.96:0/4234005841 conn(0x7f45cc0b1fc0 msgr2=0x7f45cc0b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:45.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.661+0000 7f45d2a00640 1 -- 172.21.15.96:0/4234005841 shutdown_connections 2024-05-28T06:04:45.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.661+0000 7f45d2a00640 1 -- 172.21.15.96:0/4234005841 wait complete. 2024-05-28T06:04:45.682 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-28T06:03:59.517102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-28T06:04:08.741938+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-28T06:04:09.301470+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-28T06:04:45.683 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-28T06:04:45.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.884+0000 7f003c200640 1 Processor -- start 2024-05-28T06:04:45.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.884+0000 7f003c200640 1 -- start start 2024-05-28T06:04:45.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.886+0000 7f003c200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f00340b68f0 0x7f00340cff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:45.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.886+0000 7f003c200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f00340b7e50 0x7f00340cedb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:45.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.886+0000 7f003c200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f003415f160 con 0x7f003415ed80 2024-05-28T06:04:45.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.886+0000 7f003c200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f0034163540 con 0x7f00340b68f0 2024-05-28T06:04:45.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.886+0000 7f003c200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f0034162930 con 0x7f00340b7e50 2024-05-28T06:04:45.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.886+0000 7f0039800640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f003415ed80 0x7f00340b61e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:33484/0 (socket says 172.21.15.96:33484) 2024-05-28T06:04:45.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.886+0000 7f0039800640 1 -- 172.21.15.96:0/3578590728 learned_addr learned my addr 172.21.15.96:0/3578590728 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:04:45.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.886+0000 7f0038e00640 1 --2- 172.21.15.96:0/3578590728 >> v2:172.21.15.96:3300/0 conn(0x7f00340b68f0 0x7f00340cff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:45.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.887+0000 7f0033e00640 1 -- 172.21.15.96:0/3578590728 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3979496458 0 0) 0x7f003415f160 con 0x7f003415ed80 2024-05-28T06:04:45.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.887+0000 7f003a200640 1 --2- 172.21.15.96:0/3578590728 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f00340b7e50 0x7f00340cedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:45.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.887+0000 7f0038e00640 1 -- 172.21.15.96:0/3578590728 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f00340b7e50 msgr2=0x7f00340cedb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:45.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.887+0000 7f0038e00640 1 --2- 172.21.15.96:0/3578590728 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f00340b7e50 0x7f00340cedb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:45.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.887+0000 7f0038e00640 1 -- 172.21.15.96:0/3578590728 >> v1:172.21.15.96:6789/0 conn(0x7f003415ed80 legacy=0x7f00340b61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:45.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.887+0000 7f0038e00640 1 -- 172.21.15.96:0/3578590728 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0034162c10 con 0x7f00340b68f0 2024-05-28T06:04:45.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.887+0000 7f003a200640 1 --2- 172.21.15.96:0/3578590728 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f00340b7e50 0x7f00340cedb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:04:45.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.888+0000 7f0038e00640 1 --2- 172.21.15.96:0/3578590728 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00340b68f0 0x7f00340cff80 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f0028009d90 tx=0x7f002802a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5234177eeb1622a1 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:45.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.888+0000 7f0033e00640 1 -- 172.21.15.96:0/3578590728 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0028007740 con 0x7f00340b68f0 2024-05-28T06:04:45.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.888+0000 7f0033e00640 1 -- 172.21.15.96:0/3578590728 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00280078e0 con 0x7f00340b68f0 2024-05-28T06:04:45.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.888+0000 7f0033e00640 1 -- 172.21.15.96:0/3578590728 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0028007cb0 con 0x7f00340b68f0 2024-05-28T06:04:45.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.889+0000 7f003c200640 1 -- 172.21.15.96:0/3578590728 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00340b68f0 msgr2=0x7f00340cff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:45.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.889+0000 7f003c200640 1 --2- 172.21.15.96:0/3578590728 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00340b68f0 0x7f00340cff80 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f0028009d90 tx=0x7f002802a660 comp rx=0 tx=0).stop 2024-05-28T06:04:45.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.889+0000 7f003c200640 1 -- 172.21.15.96:0/3578590728 shutdown_connections 2024-05-28T06:04:45.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.889+0000 7f003c200640 1 --2- 172.21.15.96:0/3578590728 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f00340b7e50 0x7f00340cedb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:45.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.889+0000 7f003c200640 1 --2- 172.21.15.96:0/3578590728 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00340b68f0 0x7f00340cff80 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:45.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.889+0000 7f003c200640 1 -- 172.21.15.96:0/3578590728 >> 172.21.15.96:0/3578590728 conn(0x7f00340d13c0 msgr2=0x7f00340d17d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:45.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.889+0000 7f003c200640 1 -- 172.21.15.96:0/3578590728 shutdown_connections 2024-05-28T06:04:45.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.889+0000 7f003c200640 1 -- 172.21.15.96:0/3578590728 wait complete. 2024-05-28T06:04:45.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.891+0000 7f003c200640 1 Processor -- start 2024-05-28T06:04:45.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.891+0000 7f003c200640 1 -- start start 2024-05-28T06:04:45.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.892+0000 7f003c200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f00340b68f0 0x7f00341f79f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:45.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.892+0000 7f003c200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f00340b7e50 0x7f00341527a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:45.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.892+0000 7f003c200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f003415ed80 0x7f0034152ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:45.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.892+0000 7f003c200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f00340677b0 con 0x7f003415ed80 2024-05-28T06:04:45.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.892+0000 7f003c200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f00341631d0 con 0x7f00340b68f0 2024-05-28T06:04:45.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.892+0000 7f003c200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f0034067930 con 0x7f00340b7e50 2024-05-28T06:04:45.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.893+0000 7f003a200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f003415ed80 0x7f0034152ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:45.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.893+0000 7f003a200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f003415ed80 0x7f0034152ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47780/0 (socket says 172.21.15.96:47780) 2024-05-28T06:04:45.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.893+0000 7f003a200640 1 -- 172.21.15.96:0/3796284125 learned_addr learned my addr 172.21.15.96:0/3796284125 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:45.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.893+0000 7f0039800640 1 --2- 172.21.15.96:0/3796284125 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f00340b68f0 0x7f00341f79f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:45.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.893+0000 7f0038e00640 1 --2- 172.21.15.96:0/3796284125 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f00340b7e50 0x7f00341527a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:45.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.893+0000 7f003a200640 1 -- 172.21.15.96:0/3796284125 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f00340b7e50 msgr2=0x7f00341527a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:45.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.893+0000 7f003a200640 1 --2- 172.21.15.96:0/3796284125 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f00340b7e50 0x7f00341527a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:45.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.893+0000 7f003a200640 1 -- 172.21.15.96:0/3796284125 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f00340b68f0 msgr2=0x7f00341f79f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:45.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.893+0000 7f003a200640 1 --2- 172.21.15.96:0/3796284125 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f00340b68f0 0x7f00341f79f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:45.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.893+0000 7f003a200640 1 -- 172.21.15.96:0/3796284125 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0034153340 con 0x7f003415ed80 2024-05-28T06:04:45.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.894+0000 7f003a200640 1 --2- 172.21.15.96:0/3796284125 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f003415ed80 0x7f0034152ce0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f002c00c7f0 tx=0x7f002c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:45.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.894+0000 7f0031600640 1 -- 172.21.15.96:0/3796284125 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f002c015070 con 0x7f003415ed80 2024-05-28T06:04:45.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.894+0000 7f0031600640 1 -- 172.21.15.96:0/3796284125 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f002c007820 con 0x7f003415ed80 2024-05-28T06:04:45.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.894+0000 7f0031600640 1 -- 172.21.15.96:0/3796284125 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f002c007c50 con 0x7f003415ed80 2024-05-28T06:04:45.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.894+0000 7f003c200640 1 -- 172.21.15.96:0/3796284125 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00340c8b20 con 0x7f003415ed80 2024-05-28T06:04:45.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.894+0000 7f003c200640 1 -- 172.21.15.96:0/3796284125 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00340c90e0 con 0x7f003415ed80 2024-05-28T06:04:45.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.895+0000 7f003c200640 1 -- 172.21.15.96:0/3796284125 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00340689e0 con 0x7f003415ed80 2024-05-28T06:04:45.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.896+0000 7f0031600640 1 -- 172.21.15.96:0/3796284125 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f002c018070 con 0x7f003415ed80 2024-05-28T06:04:45.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.898+0000 7f0031600640 1 --2- 172.21.15.96:0/3796284125 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f001806c8f0 0x7f001806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:45.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.898+0000 7f0031600640 1 -- 172.21.15.96:0/3796284125 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f002c08bdc0 con 0x7f003415ed80 2024-05-28T06:04:45.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.898+0000 7f0039800640 1 --2- 172.21.15.96:0/3796284125 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f001806c8f0 0x7f001806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:45.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.899+0000 7f0039800640 1 --2- 172.21.15.96:0/3796284125 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f001806c8f0 0x7f001806edb0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f0024000f60 tx=0x7f0024007c00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:45.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:45.901+0000 7f0031600640 1 -- 172.21.15.96:0/3796284125 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f002c05a150 con 0x7f003415ed80 2024-05-28T06:04:46.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.099+0000 7f003c200640 1 -- 172.21.15.96:0/3796284125 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f00340d2910 con 0x7f003415ed80 2024-05-28T06:04:46.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.100+0000 7f0031600640 1 -- 172.21.15.96:0/3796284125 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) ==== 93+0+10 (secure 0 0 0) 0x7f002c059a50 con 0x7f003415ed80 2024-05-28T06:04:46.101 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 1 2024-05-28T06:04:46.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.102+0000 7f003c200640 1 -- 172.21.15.96:0/3796284125 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f001806c8f0 msgr2=0x7f001806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:46.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.102+0000 7f003c200640 1 --2- 172.21.15.96:0/3796284125 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f001806c8f0 0x7f001806edb0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f0024000f60 tx=0x7f0024007c00 comp rx=0 tx=0).stop 2024-05-28T06:04:46.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.102+0000 7f003c200640 1 -- 172.21.15.96:0/3796284125 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f003415ed80 msgr2=0x7f0034152ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:46.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.102+0000 7f003c200640 1 --2- 172.21.15.96:0/3796284125 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f003415ed80 0x7f0034152ce0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f002c00c7f0 tx=0x7f002c00ccb0 comp rx=0 tx=0).stop 2024-05-28T06:04:46.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.102+0000 7f003c200640 1 -- 172.21.15.96:0/3796284125 shutdown_connections 2024-05-28T06:04:46.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.102+0000 7f003c200640 1 --2- 172.21.15.96:0/3796284125 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f001806c8f0 0x7f001806edb0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:46.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.103+0000 7f003c200640 1 --2- 172.21.15.96:0/3796284125 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f003415ed80 0x7f0034152ce0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:46.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.103+0000 7f003c200640 1 --2- 172.21.15.96:0/3796284125 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f00340b7e50 0x7f00341527a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:46.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.103+0000 7f003c200640 1 --2- 172.21.15.96:0/3796284125 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f00340b68f0 0x7f00341f79f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:46.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.103+0000 7f003c200640 1 -- 172.21.15.96:0/3796284125 >> 172.21.15.96:0/3796284125 conn(0x7f00340d13c0 msgr2=0x7f00340b4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:46.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.103+0000 7f003c200640 1 -- 172.21.15.96:0/3796284125 shutdown_connections 2024-05-28T06:04:46.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.103+0000 7f003c200640 1 -- 172.21.15.96:0/3796284125 wait complete. 2024-05-28T06:04:46.122 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-28T06:04:46.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.322+0000 7ff64d600640 1 Processor -- start 2024-05-28T06:04:46.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.323+0000 7ff64d600640 1 -- start start 2024-05-28T06:04:46.325 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.324+0000 7ff64d600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff6480cef30 0x7ff6480d1640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:46.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.324+0000 7ff64d600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7ff6480d1b80 0x7ff6480b90a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:46.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.324+0000 7ff64d600640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6480b8000 con 0x7ff64815e850 2024-05-28T06:04:46.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.324+0000 7ff64d600640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7ff6480d1ff0 con 0x7ff6480d1b80 2024-05-28T06:04:46.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.324+0000 7ff64d600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ff64815ec30 con 0x7ff6480cef30 2024-05-28T06:04:46.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.324+0000 7ff646a00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7ff64815e850 0x7ff6480b77b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:33492/0 (socket says 172.21.15.96:33492) 2024-05-28T06:04:46.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.324+0000 7ff646a00640 1 -- 172.21.15.96:0/2922260842 learned_addr learned my addr 172.21.15.96:0/2922260842 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:04:46.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.324+0000 7ff647400640 1 --2- 172.21.15.96:0/2922260842 >> v2:172.21.15.96:3300/0 conn(0x7ff6480d1b80 0x7ff6480b90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:46.327 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.325+0000 7ff646000640 1 --2- 172.21.15.96:0/2922260842 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff6480cef30 0x7ff6480d1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:46.327 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.325+0000 7ff647400640 1 -- 172.21.15.96:0/2922260842 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff6480cef30 msgr2=0x7ff6480d1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:46.327 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.325+0000 7ff647400640 1 --2- 172.21.15.96:0/2922260842 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff6480cef30 0x7ff6480d1640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:46.327 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.325+0000 7ff645600640 1 -- 172.21.15.96:0/2922260842 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2483942735 0 0) 0x7ff6480b8000 con 0x7ff64815e850 2024-05-28T06:04:46.327 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.325+0000 7ff647400640 1 -- 172.21.15.96:0/2922260842 >> v1:172.21.15.96:6789/0 conn(0x7ff64815e850 legacy=0x7ff6480b77b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:46.327 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.325+0000 7ff647400640 1 -- 172.21.15.96:0/2922260842 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6480b82e0 con 0x7ff6480d1b80 2024-05-28T06:04:46.327 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.325+0000 7ff646000640 1 --2- 172.21.15.96:0/2922260842 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff6480cef30 0x7ff6480d1640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:04:46.328 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.326+0000 7ff647400640 1 --2- 172.21.15.96:0/2922260842 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff6480d1b80 0x7ff6480b90a0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7ff640009d90 tx=0x7ff64002a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2204841aa4e8655a server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:46.328 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.326+0000 7ff645600640 1 -- 172.21.15.96:0/2922260842 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff640031600 con 0x7ff6480d1b80 2024-05-28T06:04:46.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.326+0000 7ff645600640 1 -- 172.21.15.96:0/2922260842 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6400317a0 con 0x7ff6480d1b80 2024-05-28T06:04:46.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.326+0000 7ff645600640 1 -- 172.21.15.96:0/2922260842 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff640031b70 con 0x7ff6480d1b80 2024-05-28T06:04:46.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.327+0000 7ff64d600640 1 -- 172.21.15.96:0/2922260842 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff6480d1b80 msgr2=0x7ff6480b90a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:46.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.327+0000 7ff64d600640 1 --2- 172.21.15.96:0/2922260842 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff6480d1b80 0x7ff6480b90a0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7ff640009d90 tx=0x7ff64002a5d0 comp rx=0 tx=0).stop 2024-05-28T06:04:46.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.327+0000 7ff64d600640 1 -- 172.21.15.96:0/2922260842 shutdown_connections 2024-05-28T06:04:46.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.327+0000 7ff64d600640 1 --2- 172.21.15.96:0/2922260842 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff6480d1b80 0x7ff6480b90a0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:46.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.327+0000 7ff64d600640 1 --2- 172.21.15.96:0/2922260842 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff6480cef30 0x7ff6480d1640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:46.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.327+0000 7ff64d600640 1 -- 172.21.15.96:0/2922260842 >> 172.21.15.96:0/2922260842 conn(0x7ff648067aa0 msgr2=0x7ff648067eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:46.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.327+0000 7ff64d600640 1 -- 172.21.15.96:0/2922260842 shutdown_connections 2024-05-28T06:04:46.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.327+0000 7ff64d600640 1 -- 172.21.15.96:0/2922260842 wait complete. 2024-05-28T06:04:46.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.329+0000 7ff64d600640 1 Processor -- start 2024-05-28T06:04:46.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.329+0000 7ff64d600640 1 -- start start 2024-05-28T06:04:46.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.330+0000 7ff64d600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff6480cef30 0x7ff64815bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:46.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.330+0000 7ff64d600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff6480d1b80 0x7ff6481f42b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:46.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.330+0000 7ff64d600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff64815e850 0x7ff648160790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:46.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.330+0000 7ff64d600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff6480b95e0 con 0x7ff6480cef30 2024-05-28T06:04:46.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.330+0000 7ff64d600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff6480b88a0 con 0x7ff6480d1b80 2024-05-28T06:04:46.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.330+0000 7ff64d600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ff6480b9760 con 0x7ff64815e850 2024-05-28T06:04:46.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.330+0000 7ff646a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff6480cef30 0x7ff64815bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:46.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.330+0000 7ff646a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff6480cef30 0x7ff64815bb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47804/0 (socket says 172.21.15.96:47804) 2024-05-28T06:04:46.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.330+0000 7ff646a00640 1 -- 172.21.15.96:0/3422006838 learned_addr learned my addr 172.21.15.96:0/3422006838 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:46.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.330+0000 7ff646a00640 1 -- 172.21.15.96:0/3422006838 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff64815e850 msgr2=0x7ff648160790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:46.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.330+0000 7ff646a00640 1 --2- 172.21.15.96:0/3422006838 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff64815e850 0x7ff648160790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:46.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.330+0000 7ff646a00640 1 -- 172.21.15.96:0/3422006838 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff6480d1b80 msgr2=0x7ff6481f42b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:46.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.330+0000 7ff646a00640 1 --2- 172.21.15.96:0/3422006838 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff6480d1b80 0x7ff6481f42b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:46.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.331+0000 7ff646a00640 1 -- 172.21.15.96:0/3422006838 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff64815f6f0 con 0x7ff6480cef30 2024-05-28T06:04:46.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.331+0000 7ff646a00640 1 --2- 172.21.15.96:0/3422006838 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff6480cef30 0x7ff64815bb50 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7ff63800c430 tx=0x7ff63800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:46.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.331+0000 7ff63ea00640 1 -- 172.21.15.96:0/3422006838 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6380074e0 con 0x7ff6480cef30 2024-05-28T06:04:46.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.331+0000 7ff63ea00640 1 -- 172.21.15.96:0/3422006838 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6380047e0 con 0x7ff6480cef30 2024-05-28T06:04:46.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.331+0000 7ff63ea00640 1 -- 172.21.15.96:0/3422006838 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff638004bd0 con 0x7ff6480cef30 2024-05-28T06:04:46.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.331+0000 7ff64d600640 1 -- 172.21.15.96:0/3422006838 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff64815f9d0 con 0x7ff6480cef30 2024-05-28T06:04:46.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.331+0000 7ff64d600640 1 -- 172.21.15.96:0/3422006838 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff64815ff90 con 0x7ff6480cef30 2024-05-28T06:04:46.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.332+0000 7ff64d600640 1 -- 172.21.15.96:0/3422006838 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6480689e0 con 0x7ff6480cef30 2024-05-28T06:04:46.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.333+0000 7ff63ea00640 1 -- 172.21.15.96:0/3422006838 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff638011070 con 0x7ff6480cef30 2024-05-28T06:04:46.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.334+0000 7ff63ea00640 1 --2- 172.21.15.96:0/3422006838 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff62406c8f0 0x7ff62406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:46.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.334+0000 7ff63ea00640 1 -- 172.21.15.96:0/3422006838 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7ff63808b6f0 con 0x7ff6480cef30 2024-05-28T06:04:46.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.335+0000 7ff646000640 1 --2- 172.21.15.96:0/3422006838 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff62406c8f0 0x7ff62406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:46.337 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.336+0000 7ff646000640 1 --2- 172.21.15.96:0/3422006838 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff62406c8f0 0x7ff62406edb0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff630006fd0 tx=0x7ff630008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:46.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.339+0000 7ff63ea00640 1 -- 172.21.15.96:0/3422006838 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff63800e280 con 0x7ff6480cef30 2024-05-28T06:04:46.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.539+0000 7ff64d600640 1 -- 172.21.15.96:0/3422006838 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ff64815dcd0 con 0x7ff6480cef30 2024-05-28T06:04:46.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.540+0000 7ff63ea00640 1 -- 172.21.15.96:0/3422006838 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) ==== 104+0+11 (secure 0 0 0) 0x7ff638059bb0 con 0x7ff6480cef30 2024-05-28T06:04:46.542 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 32 2024-05-28T06:04:46.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.543+0000 7ff64d600640 1 -- 172.21.15.96:0/3422006838 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff62406c8f0 msgr2=0x7ff62406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:46.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.543+0000 7ff64d600640 1 --2- 172.21.15.96:0/3422006838 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff62406c8f0 0x7ff62406edb0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff630006fd0 tx=0x7ff630008040 comp rx=0 tx=0).stop 2024-05-28T06:04:46.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.543+0000 7ff64d600640 1 -- 172.21.15.96:0/3422006838 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff6480cef30 msgr2=0x7ff64815bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:46.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.543+0000 7ff64d600640 1 --2- 172.21.15.96:0/3422006838 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff6480cef30 0x7ff64815bb50 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7ff63800c430 tx=0x7ff63800c8f0 comp rx=0 tx=0).stop 2024-05-28T06:04:46.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.543+0000 7ff64d600640 1 -- 172.21.15.96:0/3422006838 shutdown_connections 2024-05-28T06:04:46.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.543+0000 7ff64d600640 1 --2- 172.21.15.96:0/3422006838 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff62406c8f0 0x7ff62406edb0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:46.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.543+0000 7ff64d600640 1 --2- 172.21.15.96:0/3422006838 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff64815e850 0x7ff648160790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:46.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.543+0000 7ff64d600640 1 --2- 172.21.15.96:0/3422006838 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff6480d1b80 0x7ff6481f42b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:46.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.543+0000 7ff64d600640 1 --2- 172.21.15.96:0/3422006838 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff6480cef30 0x7ff64815bb50 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:46.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.543+0000 7ff64d600640 1 -- 172.21.15.96:0/3422006838 >> 172.21.15.96:0/3422006838 conn(0x7ff648067aa0 msgr2=0x7ff6480ca660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:46.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.543+0000 7ff64d600640 1 -- 172.21.15.96:0/3422006838 shutdown_connections 2024-05-28T06:04:46.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.544+0000 7ff64d600640 1 -- 172.21.15.96:0/3422006838 wait complete. 2024-05-28T06:04:46.563 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-28T06:04:46.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.765+0000 7f1df3e00640 1 Processor -- start 2024-05-28T06:04:46.767 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.765+0000 7f1df3e00640 1 -- start start 2024-05-28T06:04:46.768 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.767+0000 7f1df3e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1df4163d90 0x7f1df40ce760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:46.768 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.767+0000 7f1df3e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f1df40ceee0 0x7f1df40cf370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:46.768 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.767+0000 7f1df3e00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1df41670c0 con 0x7f1df41639b0 2024-05-28T06:04:46.768 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.767+0000 7f1df3e00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f1df40ceca0 con 0x7f1df40ceee0 2024-05-28T06:04:46.768 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.767+0000 7f1df3e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f1df41673a0 con 0x7f1df4163d90 2024-05-28T06:04:46.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.767+0000 7f1df3400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f1df40ceee0 0x7f1df40cf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:46.770 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.767+0000 7f1df3400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f1df40ceee0 0x7f1df40cf370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47810/0 (socket says 172.21.15.96:47810) 2024-05-28T06:04:46.770 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.767+0000 7f1df3400640 1 -- 172.21.15.96:0/4122892763 learned_addr learned my addr 172.21.15.96:0/4122892763 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:46.770 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.768+0000 7f1df2000640 1 --2- 172.21.15.96:0/4122892763 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1df4163d90 0x7f1df40ce760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:46.770 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.768+0000 7f1df3400640 1 -- 172.21.15.96:0/4122892763 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1df4163d90 msgr2=0x7f1df40ce760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:46.770 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.768+0000 7f1df3400640 1 --2- 172.21.15.96:0/4122892763 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1df4163d90 0x7f1df40ce760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:46.770 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.768+0000 7f1df3400640 1 -- 172.21.15.96:0/4122892763 >> v1:172.21.15.123:6789/0 conn(0x7f1df41639b0 legacy=0x7f1df4166670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:46.770 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.768+0000 7f1df3400640 1 -- 172.21.15.96:0/4122892763 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1df40cdf60 con 0x7f1df40ceee0 2024-05-28T06:04:46.770 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.768+0000 7f1df2000640 1 --2- 172.21.15.96:0/4122892763 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1df4163d90 0x7f1df40ce760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-28T06:04:46.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.769+0000 7f1df3400640 1 --2- 172.21.15.96:0/4122892763 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1df40ceee0 0x7f1df40cf370 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f1dec009e80 tx=0x7f1dec02a750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22b23b78bbf48138 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:46.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.770+0000 7f1df1600640 1 -- 172.21.15.96:0/4122892763 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dec036070 con 0x7f1df40ceee0 2024-05-28T06:04:46.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.770+0000 7f1df1600640 1 -- 172.21.15.96:0/4122892763 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1dec031600 con 0x7f1df40ceee0 2024-05-28T06:04:46.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.770+0000 7f1df1600640 1 -- 172.21.15.96:0/4122892763 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dec0319d0 con 0x7f1df40ceee0 2024-05-28T06:04:46.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.770+0000 7f1df3e00640 1 -- 172.21.15.96:0/4122892763 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1df40ceee0 msgr2=0x7f1df40cf370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:46.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.770+0000 7f1df3e00640 1 --2- 172.21.15.96:0/4122892763 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1df40ceee0 0x7f1df40cf370 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f1dec009e80 tx=0x7f1dec02a750 comp rx=0 tx=0).stop 2024-05-28T06:04:46.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.770+0000 7f1df3e00640 1 -- 172.21.15.96:0/4122892763 shutdown_connections 2024-05-28T06:04:46.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.771+0000 7f1df3e00640 1 --2- 172.21.15.96:0/4122892763 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1df40ceee0 0x7f1df40cf370 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:46.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.771+0000 7f1df3e00640 1 --2- 172.21.15.96:0/4122892763 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1df4163d90 0x7f1df40ce760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:46.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.771+0000 7f1df3e00640 1 -- 172.21.15.96:0/4122892763 >> 172.21.15.96:0/4122892763 conn(0x7f1df406b900 msgr2=0x7f1df406bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:46.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.771+0000 7f1df3e00640 1 -- 172.21.15.96:0/4122892763 shutdown_connections 2024-05-28T06:04:46.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.771+0000 7f1df3e00640 1 -- 172.21.15.96:0/4122892763 wait complete. 2024-05-28T06:04:46.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.772+0000 7f1df3e00640 1 Processor -- start 2024-05-28T06:04:46.774 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.772+0000 7f1df3e00640 1 -- start start 2024-05-28T06:04:46.775 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.773+0000 7f1df3e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1df40ceee0 0x7f1df406e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:46.775 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.773+0000 7f1df3e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1df41639b0 0x7f1df415eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:46.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.773+0000 7f1df3e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1df4163d90 0x7f1df415f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:46.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.773+0000 7f1df3e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f1df40afa70 con 0x7f1df40ceee0 2024-05-28T06:04:46.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.773+0000 7f1df3e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f1df4166d80 con 0x7f1df4163d90 2024-05-28T06:04:46.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.773+0000 7f1df3e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f1df40afbf0 con 0x7f1df41639b0 2024-05-28T06:04:46.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.774+0000 7f1df2a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1df40ceee0 0x7f1df406e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:46.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.774+0000 7f1df2a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1df40ceee0 0x7f1df406e3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47822/0 (socket says 172.21.15.96:47822) 2024-05-28T06:04:46.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.774+0000 7f1df2a00640 1 -- 172.21.15.96:0/586152475 learned_addr learned my addr 172.21.15.96:0/586152475 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:46.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.774+0000 7f1df2000640 1 --2- 172.21.15.96:0/586152475 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1df41639b0 0x7f1df415eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:46.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.774+0000 7f1df3400640 1 --2- 172.21.15.96:0/586152475 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1df4163d90 0x7f1df415f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:46.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.774+0000 7f1df2a00640 1 -- 172.21.15.96:0/586152475 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1df41639b0 msgr2=0x7f1df415eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:46.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.774+0000 7f1df2a00640 1 --2- 172.21.15.96:0/586152475 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1df41639b0 0x7f1df415eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:46.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.774+0000 7f1df2a00640 1 -- 172.21.15.96:0/586152475 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1df4163d90 msgr2=0x7f1df415f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:46.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.774+0000 7f1df2a00640 1 --2- 172.21.15.96:0/586152475 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1df4163d90 0x7f1df415f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:46.777 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.774+0000 7f1df2a00640 1 -- 172.21.15.96:0/586152475 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1df41fc970 con 0x7f1df40ceee0 2024-05-28T06:04:46.777 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.775+0000 7f1df2a00640 1 --2- 172.21.15.96:0/586152475 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1df40ceee0 0x7f1df406e3f0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc00c440 tx=0x7f1ddc00c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:46.777 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.775+0000 7f1deaa00640 1 -- 172.21.15.96:0/586152475 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ddc0052d0 con 0x7f1df40ceee0 2024-05-28T06:04:46.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.775+0000 7f1deaa00640 1 -- 172.21.15.96:0/586152475 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ddc005470 con 0x7f1df40ceee0 2024-05-28T06:04:46.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.775+0000 7f1deaa00640 1 -- 172.21.15.96:0/586152475 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ddc012870 con 0x7f1df40ceee0 2024-05-28T06:04:46.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.775+0000 7f1df3e00640 1 -- 172.21.15.96:0/586152475 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1df41f8880 con 0x7f1df40ceee0 2024-05-28T06:04:46.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.775+0000 7f1df3e00640 1 -- 172.21.15.96:0/586152475 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1df41f84e0 con 0x7f1df40ceee0 2024-05-28T06:04:46.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.776+0000 7f1df3e00640 1 -- 172.21.15.96:0/586152475 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1df40689e0 con 0x7f1df40ceee0 2024-05-28T06:04:46.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.777+0000 7f1deaa00640 1 -- 172.21.15.96:0/586152475 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1ddc012a10 con 0x7f1df40ceee0 2024-05-28T06:04:46.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.778+0000 7f1deaa00640 1 --2- 172.21.15.96:0/586152475 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1dd006c920 0x7f1dd006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:46.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.778+0000 7f1deaa00640 1 -- 172.21.15.96:0/586152475 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f1ddc08b830 con 0x7f1df40ceee0 2024-05-28T06:04:46.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.779+0000 7f1df2000640 1 --2- 172.21.15.96:0/586152475 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1dd006c920 0x7f1dd006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:46.781 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.780+0000 7f1df2000640 1 --2- 172.21.15.96:0/586152475 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1dd006c920 0x7f1dd006ede0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f1df40b2170 tx=0x7f1de4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:46.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.782+0000 7f1deaa00640 1 -- 172.21.15.96:0/586152475 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1ddc059bc0 con 0x7f1df40ceee0 2024-05-28T06:04:46.988 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.987+0000 7f1df3e00640 1 -- 172.21.15.96:0/586152475 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f1df40b3df0 con 0x7f1df40ceee0 2024-05-28T06:04:46.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.988+0000 7f1deaa00640 1 -- 172.21.15.96:0/586152475 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) ==== 100+0+11 (secure 0 0 0) 0x7f1ddc0594c0 con 0x7f1df40ceee0 2024-05-28T06:04:46.989 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 64 2024-05-28T06:04:46.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.990+0000 7f1df3e00640 1 -- 172.21.15.96:0/586152475 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1dd006c920 msgr2=0x7f1dd006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:46.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.990+0000 7f1df3e00640 1 --2- 172.21.15.96:0/586152475 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1dd006c920 0x7f1dd006ede0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f1df40b2170 tx=0x7f1de4008040 comp rx=0 tx=0).stop 2024-05-28T06:04:46.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.991+0000 7f1df3e00640 1 -- 172.21.15.96:0/586152475 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1df40ceee0 msgr2=0x7f1df406e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:46.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.991+0000 7f1df3e00640 1 --2- 172.21.15.96:0/586152475 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1df40ceee0 0x7f1df406e3f0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc00c440 tx=0x7f1ddc00c900 comp rx=0 tx=0).stop 2024-05-28T06:04:46.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.991+0000 7f1df3e00640 1 -- 172.21.15.96:0/586152475 shutdown_connections 2024-05-28T06:04:46.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.991+0000 7f1df3e00640 1 --2- 172.21.15.96:0/586152475 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1dd006c920 0x7f1dd006ede0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:46.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.991+0000 7f1df3e00640 1 --2- 172.21.15.96:0/586152475 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1df4163d90 0x7f1df415f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:46.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.991+0000 7f1df3e00640 1 --2- 172.21.15.96:0/586152475 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1df41639b0 0x7f1df415eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:46.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.991+0000 7f1df3e00640 1 --2- 172.21.15.96:0/586152475 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1df40ceee0 0x7f1df406e3f0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:46.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.991+0000 7f1df3e00640 1 -- 172.21.15.96:0/586152475 >> 172.21.15.96:0/586152475 conn(0x7f1df406b900 msgr2=0x7f1df4165420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:46.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.991+0000 7f1df3e00640 1 -- 172.21.15.96:0/586152475 shutdown_connections 2024-05-28T06:04:46.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:46.991+0000 7f1df3e00640 1 -- 172.21.15.96:0/586152475 wait complete. 2024-05-28T06:04:47.011 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:04:47.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.212+0000 7f8121a00640 1 Processor -- start 2024-05-28T06:04:47.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.212+0000 7f8121a00640 1 -- start start 2024-05-28T06:04:47.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.214+0000 7f8121a00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f811c0b4e10 0x7f811c0cde30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:47.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.214+0000 7f8121a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f811c0d0ca0 0x7f811c099090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:47.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.214+0000 7f8121a00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f811c0b51f0 con 0x7f811c0ce370 2024-05-28T06:04:47.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.214+0000 7f8121a00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f811c0aef60 con 0x7f811c0d0ca0 2024-05-28T06:04:47.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.214+0000 7f8121a00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f811c0ad1d0 con 0x7f811c0b4e10 2024-05-28T06:04:47.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.214+0000 7f811b400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f811c0d0ca0 0x7f811c099090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:47.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.214+0000 7f8113e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f811c0ce370 0x7f811c0d0590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:33506/0 (socket says 172.21.15.96:33506) 2024-05-28T06:04:47.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.214+0000 7f8113e00640 1 -- 172.21.15.96:0/1959892364 learned_addr learned my addr 172.21.15.96:0/1959892364 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:04:47.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.214+0000 7f811aa00640 1 --2- 172.21.15.96:0/1959892364 >> v2:172.21.15.123:3300/0 conn(0x7f811c0b4e10 0x7f811c0cde30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:47.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.215+0000 7f811b400640 1 -- 172.21.15.96:0/1959892364 >> v2:172.21.15.123:3300/0 conn(0x7f811c0b4e10 msgr2=0x7f811c0cde30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:47.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.215+0000 7f811b400640 1 --2- 172.21.15.96:0/1959892364 >> v2:172.21.15.123:3300/0 conn(0x7f811c0b4e10 0x7f811c0cde30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:47.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.215+0000 7f811b400640 1 -- 172.21.15.96:0/1959892364 >> v1:172.21.15.96:6789/0 conn(0x7f811c0ce370 legacy=0x7f811c0d0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:47.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.215+0000 7f811a000640 1 -- 172.21.15.96:0/1959892364 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3509583747 0 0) 0x7f811c0b51f0 con 0x7f811c0ce370 2024-05-28T06:04:47.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.215+0000 7f811b400640 1 -- 172.21.15.96:0/1959892364 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f811c0ad470 con 0x7f811c0d0ca0 2024-05-28T06:04:47.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.216+0000 7f811b400640 1 --2- 172.21.15.96:0/1959892364 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f811c0d0ca0 0x7f811c099090 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f811c06f080 tx=0x7f811402a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5c23992ad9e493f server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:47.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.216+0000 7f811a000640 1 -- 172.21.15.96:0/1959892364 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8114007740 con 0x7f811c0d0ca0 2024-05-28T06:04:47.218 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.216+0000 7f811a000640 1 -- 172.21.15.96:0/1959892364 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81140078e0 con 0x7f811c0d0ca0 2024-05-28T06:04:47.218 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.216+0000 7f811a000640 1 -- 172.21.15.96:0/1959892364 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8114007cb0 con 0x7f811c0d0ca0 2024-05-28T06:04:47.218 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.217+0000 7f8121a00640 1 -- 172.21.15.96:0/1959892364 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f811c0d0ca0 msgr2=0x7f811c099090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:47.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.217+0000 7f8121a00640 1 --2- 172.21.15.96:0/1959892364 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f811c0d0ca0 0x7f811c099090 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f811c06f080 tx=0x7f811402a5f0 comp rx=0 tx=0).stop 2024-05-28T06:04:47.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.217+0000 7f8121a00640 1 -- 172.21.15.96:0/1959892364 shutdown_connections 2024-05-28T06:04:47.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.217+0000 7f8121a00640 1 --2- 172.21.15.96:0/1959892364 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f811c0d0ca0 0x7f811c099090 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:47.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.217+0000 7f8121a00640 1 --2- 172.21.15.96:0/1959892364 >> v2:172.21.15.123:3300/0 conn(0x7f811c0b4e10 0x7f811c0cde30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:47.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.217+0000 7f8121a00640 1 -- 172.21.15.96:0/1959892364 >> 172.21.15.96:0/1959892364 conn(0x7f811c06bae0 msgr2=0x7f811c06bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:47.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.217+0000 7f8121a00640 1 -- 172.21.15.96:0/1959892364 shutdown_connections 2024-05-28T06:04:47.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.217+0000 7f8121a00640 1 -- 172.21.15.96:0/1959892364 wait complete. 2024-05-28T06:04:47.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.219+0000 7f8121a00640 1 Processor -- start 2024-05-28T06:04:47.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.219+0000 7f8121a00640 1 -- start start 2024-05-28T06:04:47.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.220+0000 7f8121a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f811c0b4e10 0x7f811c158210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:47.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.220+0000 7f8121a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f811c0ce370 0x7f811c0c7330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:47.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.220+0000 7f8121a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f811c0d0ca0 0x7f811c0b0b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:47.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.220+0000 7f8121a00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f811c0cea00 con 0x7f811c0ce370 2024-05-28T06:04:47.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.220+0000 7f8121a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f811c0ce880 con 0x7f811c0d0ca0 2024-05-28T06:04:47.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.220+0000 7f8121a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f811c0b3150 con 0x7f811c0b4e10 2024-05-28T06:04:47.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.220+0000 7f8113e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f811c0ce370 0x7f811c0c7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:47.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.220+0000 7f8113e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f811c0ce370 0x7f811c0c7330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47846/0 (socket says 172.21.15.96:47846) 2024-05-28T06:04:47.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.220+0000 7f8113e00640 1 -- 172.21.15.96:0/68695850 learned_addr learned my addr 172.21.15.96:0/68695850 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:47.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.220+0000 7f811b400640 1 --2- 172.21.15.96:0/68695850 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f811c0d0ca0 0x7f811c0b0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:47.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.220+0000 7f811aa00640 1 --2- 172.21.15.96:0/68695850 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f811c0b4e10 0x7f811c158210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:47.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.220+0000 7f8113e00640 1 -- 172.21.15.96:0/68695850 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f811c0b4e10 msgr2=0x7f811c158210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:47.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.220+0000 7f8113e00640 1 --2- 172.21.15.96:0/68695850 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f811c0b4e10 0x7f811c158210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:47.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.221+0000 7f8113e00640 1 -- 172.21.15.96:0/68695850 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f811c0d0ca0 msgr2=0x7f811c0b0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:47.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.221+0000 7f8113e00640 1 --2- 172.21.15.96:0/68695850 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f811c0d0ca0 0x7f811c0b0b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:47.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.221+0000 7f8113e00640 1 -- 172.21.15.96:0/68695850 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f811c0b1500 con 0x7f811c0ce370 2024-05-28T06:04:47.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.221+0000 7f811aa00640 1 --2- 172.21.15.96:0/68695850 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f811c0b4e10 0x7f811c158210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:04:47.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.221+0000 7f8113e00640 1 --2- 172.21.15.96:0/68695850 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f811c0ce370 0x7f811c0c7330 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f810400c4d0 tx=0x7f810400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:47.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.221+0000 7f8112a00640 1 -- 172.21.15.96:0/68695850 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8104018070 con 0x7f811c0ce370 2024-05-28T06:04:47.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.221+0000 7f8112a00640 1 -- 172.21.15.96:0/68695850 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8104007500 con 0x7f811c0ce370 2024-05-28T06:04:47.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.221+0000 7f8112a00640 1 -- 172.21.15.96:0/68695850 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8104004a20 con 0x7f811c0ce370 2024-05-28T06:04:47.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.221+0000 7f8121a00640 1 -- 172.21.15.96:0/68695850 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f811c0c5650 con 0x7f811c0ce370 2024-05-28T06:04:47.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.221+0000 7f8121a00640 1 -- 172.21.15.96:0/68695850 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f811c0c5c10 con 0x7f811c0ce370 2024-05-28T06:04:47.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.223+0000 7f8121a00640 1 -- 172.21.15.96:0/68695850 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f811c0689e0 con 0x7f811c0ce370 2024-05-28T06:04:47.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.224+0000 7f8112a00640 1 -- 172.21.15.96:0/68695850 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8104004bc0 con 0x7f811c0ce370 2024-05-28T06:04:47.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.225+0000 7f8112a00640 1 --2- 172.21.15.96:0/68695850 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f80f806c8f0 0x7f80f806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:47.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.225+0000 7f8112a00640 1 -- 172.21.15.96:0/68695850 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f810408adb0 con 0x7f811c0ce370 2024-05-28T06:04:47.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.225+0000 7f811aa00640 1 --2- 172.21.15.96:0/68695850 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f80f806c8f0 0x7f80f806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:47.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.226+0000 7f811aa00640 1 --2- 172.21.15.96:0/68695850 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f80f806c8f0 0x7f80f806edb0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f810c006fd0 tx=0x7f810c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:47.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.229+0000 7f8112a00640 1 -- 172.21.15.96:0/68695850 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8104059270 con 0x7f811c0ce370 2024-05-28T06:04:47.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.433+0000 7f8121a00640 1 -- 172.21.15.96:0/68695850 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f811c06f540 con 0x7f811c0ce370 2024-05-28T06:04:47.436 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.435+0000 7f8112a00640 1 -- 172.21.15.96:0/68695850 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16617 (secure 0 0 0) 0x7f8104058b70 con 0x7f811c0ce370 2024-05-28T06:04:47.437 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:04:47.437 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":20,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:04:11.882922+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-28T06:04:08.741938+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-28T06:04:09.301470+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:04:47.439 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.437+0000 7f8121a00640 1 -- 172.21.15.96:0/68695850 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f80f806c8f0 msgr2=0x7f80f806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:47.439 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.437+0000 7f8121a00640 1 --2- 172.21.15.96:0/68695850 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f80f806c8f0 0x7f80f806edb0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f810c006fd0 tx=0x7f810c008040 comp rx=0 tx=0).stop 2024-05-28T06:04:47.439 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.437+0000 7f8121a00640 1 -- 172.21.15.96:0/68695850 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f811c0ce370 msgr2=0x7f811c0c7330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:47.439 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.437+0000 7f8121a00640 1 --2- 172.21.15.96:0/68695850 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f811c0ce370 0x7f811c0c7330 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f810400c4d0 tx=0x7f810400c990 comp rx=0 tx=0).stop 2024-05-28T06:04:47.439 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.438+0000 7f8121a00640 1 -- 172.21.15.96:0/68695850 shutdown_connections 2024-05-28T06:04:47.439 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.438+0000 7f8121a00640 1 --2- 172.21.15.96:0/68695850 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f80f806c8f0 0x7f80f806edb0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:47.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.438+0000 7f8121a00640 1 --2- 172.21.15.96:0/68695850 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f811c0d0ca0 0x7f811c0b0b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:47.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.438+0000 7f8121a00640 1 --2- 172.21.15.96:0/68695850 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f811c0ce370 0x7f811c0c7330 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:47.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.438+0000 7f8121a00640 1 --2- 172.21.15.96:0/68695850 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f811c0b4e10 0x7f811c158210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:47.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.438+0000 7f8121a00640 1 -- 172.21.15.96:0/68695850 >> 172.21.15.96:0/68695850 conn(0x7f811c06bae0 msgr2=0x7f811c06bec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:47.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.438+0000 7f8121a00640 1 -- 172.21.15.96:0/68695850 shutdown_connections 2024-05-28T06:04:47.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.438+0000 7f8121a00640 1 -- 172.21.15.96:0/68695850 wait complete. 2024-05-28T06:04:47.458 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-28T06:03:59.517102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-28T06:04:08.741938+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-28T06:04:09.301470+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-28T06:04:47.459 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-28T06:04:47.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.650+0000 7f5bbca00640 1 Processor -- start 2024-05-28T06:04:47.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.650+0000 7f5bbca00640 1 -- start start 2024-05-28T06:04:47.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.652+0000 7f5bbca00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f5bb815e8e0 0x7f5bb80cb5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:47.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.652+0000 7f5bbca00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5bb80cbd10 con 0x7f5bb80d31a0 2024-05-28T06:04:47.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.652+0000 7f5bbca00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5bb80cbff0 con 0x7f5bb80d2600 2024-05-28T06:04:47.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.652+0000 7f5bbca00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f5bb80cc2d0 con 0x7f5bb815e8e0 2024-05-28T06:04:47.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.652+0000 7f5bb6800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f5bb815e8e0 0x7f5bb80cb5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:47.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.652+0000 7f5bb7200640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f5bb80d31a0 0x7f5bb80ccdb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:33518/0 (socket says 172.21.15.96:33518) 2024-05-28T06:04:47.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.652+0000 7f5bb7200640 1 -- 172.21.15.96:0/1006010013 learned_addr learned my addr 172.21.15.96:0/1006010013 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:04:47.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.653+0000 7f5bb6800640 1 -- 172.21.15.96:0/1006010013 >> v1:172.21.15.123:6789/0 conn(0x7f5bb80d2600 legacy=0x7f5bb80d2a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:47.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.653+0000 7f5bb6800640 1 -- 172.21.15.96:0/1006010013 >> v1:172.21.15.96:6789/0 conn(0x7f5bb80d31a0 legacy=0x7f5bb80ccdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:47.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.653+0000 7f5bb5e00640 1 -- 172.21.15.96:0/1006010013 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2137335218 0 0) 0x7f5bb80cbd10 con 0x7f5bb80d31a0 2024-05-28T06:04:47.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.653+0000 7f5bb6800640 1 -- 172.21.15.96:0/1006010013 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bb80cc5b0 con 0x7f5bb815e8e0 2024-05-28T06:04:47.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.654+0000 7f5bb6800640 1 --2- 172.21.15.96:0/1006010013 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5bb815e8e0 0x7f5bb80cb5c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f5ba80097a0 tx=0x7f5ba802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d71525dcfe8075fe server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:47.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.654+0000 7f5bb5e00640 1 -- 172.21.15.96:0/1006010013 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ba8037070 con 0x7f5bb815e8e0 2024-05-28T06:04:47.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.654+0000 7f5bb5e00640 1 -- 172.21.15.96:0/1006010013 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ba8032440 con 0x7f5bb815e8e0 2024-05-28T06:04:47.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.654+0000 7f5bb5e00640 1 -- 172.21.15.96:0/1006010013 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ba8032810 con 0x7f5bb815e8e0 2024-05-28T06:04:47.657 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.655+0000 7f5bbca00640 1 -- 172.21.15.96:0/1006010013 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5bb815e8e0 msgr2=0x7f5bb80cb5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:47.657 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.655+0000 7f5bbca00640 1 --2- 172.21.15.96:0/1006010013 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5bb815e8e0 0x7f5bb80cb5c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f5ba80097a0 tx=0x7f5ba802a1e0 comp rx=0 tx=0).stop 2024-05-28T06:04:47.657 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.655+0000 7f5bbca00640 1 -- 172.21.15.96:0/1006010013 shutdown_connections 2024-05-28T06:04:47.657 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.655+0000 7f5bbca00640 1 --2- 172.21.15.96:0/1006010013 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5bb815e8e0 0x7f5bb80cb5c0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:47.657 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.655+0000 7f5bbca00640 1 -- 172.21.15.96:0/1006010013 >> 172.21.15.96:0/1006010013 conn(0x7f5bb808bfc0 msgr2=0x7f5bb80d44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:47.657 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.655+0000 7f5bbca00640 1 -- 172.21.15.96:0/1006010013 shutdown_connections 2024-05-28T06:04:47.657 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.655+0000 7f5bbca00640 1 -- 172.21.15.96:0/1006010013 wait complete. 2024-05-28T06:04:47.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.657+0000 7f5bbca00640 1 Processor -- start 2024-05-28T06:04:47.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.657+0000 7f5bbca00640 1 -- start start 2024-05-28T06:04:47.659 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.658+0000 7f5bbca00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5bb80d2600 0x7f5bb8153520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:47.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.658+0000 7f5bbca00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5bb80d31a0 0x7f5bb81f4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:47.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.658+0000 7f5bbca00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5bb815e8e0 0x7f5bb8152e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:47.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.658+0000 7f5bbca00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f5bb815ee40 con 0x7f5bb80d31a0 2024-05-28T06:04:47.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.658+0000 7f5bbca00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5bb815ecc0 con 0x7f5bb815e8e0 2024-05-28T06:04:47.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.658+0000 7f5bbca00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f5bb80cd4c0 con 0x7f5bb80d2600 2024-05-28T06:04:47.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.658+0000 7f5bafe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5bb80d31a0 0x7f5bb81f4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:47.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.658+0000 7f5bafe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5bb80d31a0 0x7f5bb81f4160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47868/0 (socket says 172.21.15.96:47868) 2024-05-28T06:04:47.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.658+0000 7f5bafe00640 1 -- 172.21.15.96:0/511932228 learned_addr learned my addr 172.21.15.96:0/511932228 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:47.661 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.659+0000 7f5bb7200640 1 --2- 172.21.15.96:0/511932228 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5bb815e8e0 0x7f5bb8152e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:47.661 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.659+0000 7f5bafe00640 1 -- 172.21.15.96:0/511932228 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5bb80d2600 msgr2=0x7f5bb8153520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:47.661 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.659+0000 7f5bafe00640 1 --2- 172.21.15.96:0/511932228 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5bb80d2600 0x7f5bb8153520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:47.661 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.659+0000 7f5bafe00640 1 -- 172.21.15.96:0/511932228 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5bb815e8e0 msgr2=0x7f5bb8152e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:47.661 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.659+0000 7f5bafe00640 1 --2- 172.21.15.96:0/511932228 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5bb815e8e0 0x7f5bb8152e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:47.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.659+0000 7f5bafe00640 1 -- 172.21.15.96:0/511932228 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bb80a02d0 con 0x7f5bb80d31a0 2024-05-28T06:04:47.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.659+0000 7f5bb7200640 1 --2- 172.21.15.96:0/511932228 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5bb815e8e0 0x7f5bb8152e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:04:47.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.659+0000 7f5bafe00640 1 --2- 172.21.15.96:0/511932228 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5bb80d31a0 0x7f5bb81f4160 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f5ba000c430 tx=0x7f5ba000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:47.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.659+0000 7f5baea00640 1 -- 172.21.15.96:0/511932228 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ba0011790 con 0x7f5bb80d31a0 2024-05-28T06:04:47.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.659+0000 7f5baea00640 1 -- 172.21.15.96:0/511932228 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ba0017070 con 0x7f5bb80d31a0 2024-05-28T06:04:47.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.659+0000 7f5baea00640 1 -- 172.21.15.96:0/511932228 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ba0011bb0 con 0x7f5bb80d31a0 2024-05-28T06:04:47.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.659+0000 7f5bbca00640 1 -- 172.21.15.96:0/511932228 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bb80a0590 con 0x7f5bb80d31a0 2024-05-28T06:04:47.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.659+0000 7f5bbca00640 1 -- 172.21.15.96:0/511932228 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bb80a0b50 con 0x7f5bb80d31a0 2024-05-28T06:04:47.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.661+0000 7f5bbca00640 1 -- 172.21.15.96:0/511932228 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5bb80689e0 con 0x7f5bb80d31a0 2024-05-28T06:04:47.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.662+0000 7f5baea00640 1 -- 172.21.15.96:0/511932228 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5ba001c460 con 0x7f5bb80d31a0 2024-05-28T06:04:47.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.663+0000 7f5baea00640 1 --2- 172.21.15.96:0/511932228 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5b9406c8f0 0x7f5b9406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:47.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.663+0000 7f5baea00640 1 -- 172.21.15.96:0/511932228 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f5ba008ba70 con 0x7f5bb80d31a0 2024-05-28T06:04:47.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.663+0000 7f5bb6800640 1 --2- 172.21.15.96:0/511932228 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5b9406c8f0 0x7f5b9406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:47.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.664+0000 7f5bb6800640 1 --2- 172.21.15.96:0/511932228 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5b9406c8f0 0x7f5b9406edb0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f5ba8034000 tx=0x7f5ba802c000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:47.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.667+0000 7f5baea00640 1 -- 172.21.15.96:0/511932228 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5ba0059ea0 con 0x7f5bb80d31a0 2024-05-28T06:04:47.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.866+0000 7f5bbca00640 1 -- 172.21.15.96:0/511932228 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5bb8158540 con 0x7f5bb80d31a0 2024-05-28T06:04:47.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.867+0000 7f5baea00640 1 -- 172.21.15.96:0/511932228 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) ==== 93+0+10 (secure 0 0 0) 0x7f5ba00597a0 con 0x7f5bb80d31a0 2024-05-28T06:04:47.869 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 1 2024-05-28T06:04:47.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.870+0000 7f5bbca00640 1 -- 172.21.15.96:0/511932228 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5b9406c8f0 msgr2=0x7f5b9406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:47.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.870+0000 7f5bbca00640 1 --2- 172.21.15.96:0/511932228 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5b9406c8f0 0x7f5b9406edb0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f5ba8034000 tx=0x7f5ba802c000 comp rx=0 tx=0).stop 2024-05-28T06:04:47.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.870+0000 7f5bbca00640 1 -- 172.21.15.96:0/511932228 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5bb80d31a0 msgr2=0x7f5bb81f4160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:47.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.870+0000 7f5bbca00640 1 --2- 172.21.15.96:0/511932228 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5bb80d31a0 0x7f5bb81f4160 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f5ba000c430 tx=0x7f5ba000c8f0 comp rx=0 tx=0).stop 2024-05-28T06:04:47.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.870+0000 7f5bbca00640 1 -- 172.21.15.96:0/511932228 shutdown_connections 2024-05-28T06:04:47.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.870+0000 7f5bbca00640 1 --2- 172.21.15.96:0/511932228 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5b9406c8f0 0x7f5b9406edb0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:47.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.870+0000 7f5bbca00640 1 --2- 172.21.15.96:0/511932228 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5bb815e8e0 0x7f5bb8152e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:47.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.870+0000 7f5bbca00640 1 --2- 172.21.15.96:0/511932228 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5bb80d31a0 0x7f5bb81f4160 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:47.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.870+0000 7f5bbca00640 1 --2- 172.21.15.96:0/511932228 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5bb80d2600 0x7f5bb8153520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:47.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.870+0000 7f5bbca00640 1 -- 172.21.15.96:0/511932228 >> 172.21.15.96:0/511932228 conn(0x7f5bb808bfc0 msgr2=0x7f5bb80d0f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:47.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.870+0000 7f5bbca00640 1 -- 172.21.15.96:0/511932228 shutdown_connections 2024-05-28T06:04:47.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:47.870+0000 7f5bbca00640 1 -- 172.21.15.96:0/511932228 wait complete. 2024-05-28T06:04:47.891 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-28T06:04:48.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.124+0000 7f920b200640 1 Processor -- start 2024-05-28T06:04:48.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.125+0000 7f920b200640 1 -- start start 2024-05-28T06:04:48.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.126+0000 7f920b200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f92040d0640 0x7f92040d0a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:48.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.126+0000 7f920b200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f92040725e0 0x7f92040ae650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:48.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.126+0000 7f920b200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92040ade90 con 0x7f920415b6f0 2024-05-28T06:04:48.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.126+0000 7f920b200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f9204071c40 con 0x7f92040725e0 2024-05-28T06:04:48.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.126+0000 7f920b200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f92040d0f60 con 0x7f92040d0640 2024-05-28T06:04:48.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.126+0000 7f9209200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f92040725e0 0x7f92040ae650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:48.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.126+0000 7f9209200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f92040725e0 0x7f92040ae650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47872/0 (socket says 172.21.15.96:47872) 2024-05-28T06:04:48.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.126+0000 7f9209200640 1 -- 172.21.15.96:0/4073448211 learned_addr learned my addr 172.21.15.96:0/4073448211 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:48.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.127+0000 7f9203e00640 1 --2- 172.21.15.96:0/4073448211 >> v2:172.21.15.123:3300/0 conn(0x7f92040d0640 0x7f92040d0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:48.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.127+0000 7f9209200640 1 -- 172.21.15.96:0/4073448211 >> v2:172.21.15.123:3300/0 conn(0x7f92040d0640 msgr2=0x7f92040d0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:48.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.127+0000 7f9209200640 1 --2- 172.21.15.96:0/4073448211 >> v2:172.21.15.123:3300/0 conn(0x7f92040d0640 0x7f92040d0a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:48.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.127+0000 7f9209200640 1 -- 172.21.15.96:0/4073448211 >> v1:172.21.15.96:6789/0 conn(0x7f920415b6f0 legacy=0x7f920406e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:48.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.127+0000 7f9209200640 1 -- 172.21.15.96:0/4073448211 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9204071e70 con 0x7f92040725e0 2024-05-28T06:04:48.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.127+0000 7f9203e00640 1 --2- 172.21.15.96:0/4073448211 >> v2:172.21.15.123:3300/0 conn(0x7f92040d0640 0x7f92040d0a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:04:48.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.128+0000 7f9209200640 1 --2- 172.21.15.96:0/4073448211 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f92040725e0 0x7f92040ae650 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f920406fe40 tx=0x7f91fc02a740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=810fa183ce2d4a70 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:48.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.128+0000 7f9203400640 1 -- 172.21.15.96:0/4073448211 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91fc036070 con 0x7f92040725e0 2024-05-28T06:04:48.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.128+0000 7f9203400640 1 -- 172.21.15.96:0/4073448211 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91fc031600 con 0x7f92040725e0 2024-05-28T06:04:48.131 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.128+0000 7f9203400640 1 -- 172.21.15.96:0/4073448211 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91fc0319d0 con 0x7f92040725e0 2024-05-28T06:04:48.132 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.129+0000 7f920b200640 1 -- 172.21.15.96:0/4073448211 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f92040725e0 msgr2=0x7f92040ae650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:48.132 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.129+0000 7f920b200640 1 --2- 172.21.15.96:0/4073448211 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f92040725e0 0x7f92040ae650 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f920406fe40 tx=0x7f91fc02a740 comp rx=0 tx=0).stop 2024-05-28T06:04:48.132 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.129+0000 7f920b200640 1 -- 172.21.15.96:0/4073448211 shutdown_connections 2024-05-28T06:04:48.132 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.129+0000 7f920b200640 1 --2- 172.21.15.96:0/4073448211 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f92040725e0 0x7f92040ae650 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:48.132 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.129+0000 7f920b200640 1 --2- 172.21.15.96:0/4073448211 >> v2:172.21.15.123:3300/0 conn(0x7f92040d0640 0x7f92040d0a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:48.132 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.129+0000 7f920b200640 1 -- 172.21.15.96:0/4073448211 >> 172.21.15.96:0/4073448211 conn(0x7f920406bd20 msgr2=0x7f920406c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:48.132 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.129+0000 7f920b200640 1 -- 172.21.15.96:0/4073448211 shutdown_connections 2024-05-28T06:04:48.132 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.129+0000 7f920b200640 1 -- 172.21.15.96:0/4073448211 wait complete. 2024-05-28T06:04:48.132 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.131+0000 7f920b200640 1 Processor -- start 2024-05-28T06:04:48.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.131+0000 7f920b200640 1 -- start start 2024-05-28T06:04:48.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.132+0000 7f920b200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f92040725e0 0x7f92040b03a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:48.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.132+0000 7f920b200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92040d0640 0x7f9204144d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:48.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.132+0000 7f920b200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f920415b6f0 0x7f9204142ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:48.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.132+0000 7f920b200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f92040aeb90 con 0x7f920415b6f0 2024-05-28T06:04:48.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.132+0000 7f920b200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f92040ae130 con 0x7f92040d0640 2024-05-28T06:04:48.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.132+0000 7f920b200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f92040aed10 con 0x7f92040725e0 2024-05-28T06:04:48.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.132+0000 7f9209200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f920415b6f0 0x7f9204142ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:48.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.132+0000 7f9209200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f920415b6f0 0x7f9204142ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47888/0 (socket says 172.21.15.96:47888) 2024-05-28T06:04:48.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.132+0000 7f9209200640 1 -- 172.21.15.96:0/824214446 learned_addr learned my addr 172.21.15.96:0/824214446 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:48.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.133+0000 7f91fb400640 1 --2- 172.21.15.96:0/824214446 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92040d0640 0x7f9204144d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:48.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.133+0000 7f9203e00640 1 --2- 172.21.15.96:0/824214446 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f92040725e0 0x7f92040b03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:48.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.133+0000 7f9209200640 1 -- 172.21.15.96:0/824214446 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f92040725e0 msgr2=0x7f92040b03a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:48.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.133+0000 7f9209200640 1 --2- 172.21.15.96:0/824214446 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f92040725e0 0x7f92040b03a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:48.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.133+0000 7f9209200640 1 -- 172.21.15.96:0/824214446 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92040d0640 msgr2=0x7f9204144d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:48.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.133+0000 7f9209200640 1 --2- 172.21.15.96:0/824214446 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92040d0640 0x7f9204144d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:48.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.133+0000 7f9209200640 1 -- 172.21.15.96:0/824214446 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92041510c0 con 0x7f920415b6f0 2024-05-28T06:04:48.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.133+0000 7f91fb400640 1 --2- 172.21.15.96:0/824214446 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92040d0640 0x7f9204144d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:04:48.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.133+0000 7f9209200640 1 --2- 172.21.15.96:0/824214446 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f920415b6f0 0x7f9204142ca0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f91fc034040 tx=0x7f91fc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:48.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.133+0000 7f9200c00640 1 -- 172.21.15.96:0/824214446 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91fc036050 con 0x7f920415b6f0 2024-05-28T06:04:48.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.133+0000 7f9200c00640 1 -- 172.21.15.96:0/824214446 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91fc042070 con 0x7f920415b6f0 2024-05-28T06:04:48.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.133+0000 7f9200c00640 1 -- 172.21.15.96:0/824214446 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91fc009510 con 0x7f920415b6f0 2024-05-28T06:04:48.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.133+0000 7f920b200640 1 -- 172.21.15.96:0/824214446 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9204145480 con 0x7f920415b6f0 2024-05-28T06:04:48.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.134+0000 7f920b200640 1 -- 172.21.15.96:0/824214446 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9204145a40 con 0x7f920415b6f0 2024-05-28T06:04:48.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.135+0000 7f920b200640 1 -- 172.21.15.96:0/824214446 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92040689e0 con 0x7f920415b6f0 2024-05-28T06:04:48.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.136+0000 7f9200c00640 1 -- 172.21.15.96:0/824214446 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f91fc007bd0 con 0x7f920415b6f0 2024-05-28T06:04:48.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.136+0000 7f9200c00640 1 --2- 172.21.15.96:0/824214446 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f91e006c8f0 0x7f91e006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:48.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.137+0000 7f9200c00640 1 -- 172.21.15.96:0/824214446 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f91fc0acb50 con 0x7f920415b6f0 2024-05-28T06:04:48.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.137+0000 7f9203e00640 1 --2- 172.21.15.96:0/824214446 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f91e006c8f0 0x7f91e006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:48.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.138+0000 7f9203e00640 1 --2- 172.21.15.96:0/824214446 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f91e006c8f0 0x7f91e006edb0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f91f4008850 tx=0x7f91f4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:48.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.141+0000 7f9200c00640 1 -- 172.21.15.96:0/824214446 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f91fc07a890 con 0x7f920415b6f0 2024-05-28T06:04:48.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.338+0000 7f920b200640 1 -- 172.21.15.96:0/824214446 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f92040b0e70 con 0x7f920415b6f0 2024-05-28T06:04:48.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.339+0000 7f9200c00640 1 -- 172.21.15.96:0/824214446 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) ==== 104+0+11 (secure 0 0 0) 0x7f91fc07eb90 con 0x7f920415b6f0 2024-05-28T06:04:48.341 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 32 2024-05-28T06:04:48.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.342+0000 7f920b200640 1 -- 172.21.15.96:0/824214446 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f91e006c8f0 msgr2=0x7f91e006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:48.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.342+0000 7f920b200640 1 --2- 172.21.15.96:0/824214446 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f91e006c8f0 0x7f91e006edb0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f91f4008850 tx=0x7f91f4008040 comp rx=0 tx=0).stop 2024-05-28T06:04:48.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.342+0000 7f920b200640 1 -- 172.21.15.96:0/824214446 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f920415b6f0 msgr2=0x7f9204142ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:48.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.342+0000 7f920b200640 1 --2- 172.21.15.96:0/824214446 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f920415b6f0 0x7f9204142ca0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f91fc034040 tx=0x7f91fc004a40 comp rx=0 tx=0).stop 2024-05-28T06:04:48.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.342+0000 7f920b200640 1 -- 172.21.15.96:0/824214446 shutdown_connections 2024-05-28T06:04:48.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.342+0000 7f920b200640 1 --2- 172.21.15.96:0/824214446 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f91e006c8f0 0x7f91e006edb0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:48.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.342+0000 7f920b200640 1 --2- 172.21.15.96:0/824214446 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f920415b6f0 0x7f9204142ca0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:48.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.342+0000 7f920b200640 1 --2- 172.21.15.96:0/824214446 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92040d0640 0x7f9204144d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:48.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.342+0000 7f920b200640 1 --2- 172.21.15.96:0/824214446 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f92040725e0 0x7f92040b03a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:48.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.342+0000 7f920b200640 1 -- 172.21.15.96:0/824214446 >> 172.21.15.96:0/824214446 conn(0x7f920406bd20 msgr2=0x7f9204072c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:48.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.342+0000 7f920b200640 1 -- 172.21.15.96:0/824214446 shutdown_connections 2024-05-28T06:04:48.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.342+0000 7f920b200640 1 -- 172.21.15.96:0/824214446 wait complete. 2024-05-28T06:04:48.362 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-28T06:04:48.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.563+0000 7f55cca00640 1 Processor -- start 2024-05-28T06:04:48.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.563+0000 7f55cca00640 1 -- start start 2024-05-28T06:04:48.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.565+0000 7f55cca00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f55c80ce1f0 0x7f55c80b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:48.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.565+0000 7f55cca00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f55c80ce5d0 0x7f55c815ec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:48.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.565+0000 7f55cca00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f55c80b3d60 0x7f55c80b41e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:48.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.565+0000 7f55cca00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f55c80b3580 con 0x7f55c80ce1f0 2024-05-28T06:04:48.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.565+0000 7f55cca00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f55c815f490 con 0x7f55c80ce5d0 2024-05-28T06:04:48.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.565+0000 7f55cca00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f55c815f770 con 0x7f55c80b3d60 2024-05-28T06:04:48.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.565+0000 7f55c7400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f55c80ce1f0 0x7f55c80b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:48.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.565+0000 7f55c7400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f55c80ce1f0 0x7f55c80b4dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47892/0 (socket says 172.21.15.96:47892) 2024-05-28T06:04:48.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.565+0000 7f55c7400640 1 -- 172.21.15.96:0/2256809397 learned_addr learned my addr 172.21.15.96:0/2256809397 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:48.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.566+0000 7f55c7e00640 1 --2- 172.21.15.96:0/2256809397 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f55c80b3d60 0x7f55c80b41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:48.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.566+0000 7f55c6a00640 1 --2- 172.21.15.96:0/2256809397 >> v2:172.21.15.123:3300/0 conn(0x7f55c80ce5d0 0x7f55c815ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:48.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.566+0000 7f55c7400640 1 -- 172.21.15.96:0/2256809397 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f55c80b3d60 msgr2=0x7f55c80b41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:48.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.566+0000 7f55c7400640 1 --2- 172.21.15.96:0/2256809397 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f55c80b3d60 0x7f55c80b41e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:48.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.566+0000 7f55c7400640 1 -- 172.21.15.96:0/2256809397 >> v2:172.21.15.123:3300/0 conn(0x7f55c80ce5d0 msgr2=0x7f55c815ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:48.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.566+0000 7f55c7400640 1 --2- 172.21.15.96:0/2256809397 >> v2:172.21.15.123:3300/0 conn(0x7f55c80ce5d0 0x7f55c815ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:48.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.566+0000 7f55c7400640 1 -- 172.21.15.96:0/2256809397 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55c815fa50 con 0x7f55c80ce1f0 2024-05-28T06:04:48.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.567+0000 7f55c7400640 1 --2- 172.21.15.96:0/2256809397 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f55c80ce1f0 0x7f55c80b4dd0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f55c8070060 tx=0x7f55b402a0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6ca20bac52fefbe server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:48.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.568+0000 7f55c6000640 1 -- 172.21.15.96:0/2256809397 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55b4037070 con 0x7f55c80ce1f0 2024-05-28T06:04:48.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.568+0000 7f55c6000640 1 -- 172.21.15.96:0/2256809397 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55b402ae00 con 0x7f55c80ce1f0 2024-05-28T06:04:48.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.568+0000 7f55c6000640 1 -- 172.21.15.96:0/2256809397 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55b4032660 con 0x7f55c80ce1f0 2024-05-28T06:04:48.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.569+0000 7f55cca00640 1 -- 172.21.15.96:0/2256809397 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f55c80ce1f0 msgr2=0x7f55c80b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:48.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.569+0000 7f55cca00640 1 --2- 172.21.15.96:0/2256809397 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f55c80ce1f0 0x7f55c80b4dd0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f55c8070060 tx=0x7f55b402a0b0 comp rx=0 tx=0).stop 2024-05-28T06:04:48.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.569+0000 7f55cca00640 1 -- 172.21.15.96:0/2256809397 shutdown_connections 2024-05-28T06:04:48.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.569+0000 7f55cca00640 1 --2- 172.21.15.96:0/2256809397 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f55c80b3d60 0x7f55c80b41e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:48.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.569+0000 7f55cca00640 1 --2- 172.21.15.96:0/2256809397 >> v2:172.21.15.123:3300/0 conn(0x7f55c80ce5d0 0x7f55c815ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:48.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.569+0000 7f55cca00640 1 --2- 172.21.15.96:0/2256809397 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f55c80ce1f0 0x7f55c80b4dd0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:48.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.569+0000 7f55cca00640 1 -- 172.21.15.96:0/2256809397 >> 172.21.15.96:0/2256809397 conn(0x7f55c806b900 msgr2=0x7f55c806bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:48.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.569+0000 7f55cca00640 1 -- 172.21.15.96:0/2256809397 shutdown_connections 2024-05-28T06:04:48.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.569+0000 7f55cca00640 1 -- 172.21.15.96:0/2256809397 wait complete. 2024-05-28T06:04:48.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.571+0000 7f55cca00640 1 Processor -- start 2024-05-28T06:04:48.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.571+0000 7f55cca00640 1 -- start start 2024-05-28T06:04:48.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.572+0000 7f55cca00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f55c80b3d60 0x7f55c80b1e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:48.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.572+0000 7f55cca00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f55c80ce1f0 0x7f55c8160780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:48.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.572+0000 7f55cca00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f55c80ce5d0 0x7f55c81f39d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:48.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.572+0000 7f55cca00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f55c80ccb90 con 0x7f55c80ce1f0 2024-05-28T06:04:48.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.572+0000 7f55cca00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f55c815f180 con 0x7f55c80ce5d0 2024-05-28T06:04:48.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.572+0000 7f55cca00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f55c80ccd10 con 0x7f55c80b3d60 2024-05-28T06:04:48.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.572+0000 7f55c6a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f55c80ce1f0 0x7f55c8160780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:48.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.572+0000 7f55c6a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f55c80ce1f0 0x7f55c8160780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47906/0 (socket says 172.21.15.96:47906) 2024-05-28T06:04:48.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.572+0000 7f55c6a00640 1 -- 172.21.15.96:0/3173639573 learned_addr learned my addr 172.21.15.96:0/3173639573 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:48.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.573+0000 7f55c7400640 1 --2- 172.21.15.96:0/3173639573 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f55c80b3d60 0x7f55c80b1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:48.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.573+0000 7f55c7e00640 1 --2- 172.21.15.96:0/3173639573 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f55c80ce5d0 0x7f55c81f39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:48.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.573+0000 7f55c6a00640 1 -- 172.21.15.96:0/3173639573 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f55c80b3d60 msgr2=0x7f55c80b1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:48.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.573+0000 7f55c6a00640 1 --2- 172.21.15.96:0/3173639573 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f55c80b3d60 0x7f55c80b1e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:48.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.573+0000 7f55c6a00640 1 -- 172.21.15.96:0/3173639573 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f55c80ce5d0 msgr2=0x7f55c81f39d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:48.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.573+0000 7f55c6a00640 1 --2- 172.21.15.96:0/3173639573 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f55c80ce5d0 0x7f55c81f39d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:48.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.573+0000 7f55c6a00640 1 -- 172.21.15.96:0/3173639573 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55c8160cc0 con 0x7f55c80ce1f0 2024-05-28T06:04:48.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.573+0000 7f55c6a00640 1 --2- 172.21.15.96:0/3173639573 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f55c80ce1f0 0x7f55c8160780 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f55b800d5f0 tx=0x7f55b800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:48.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.573+0000 7f55bf400640 1 -- 172.21.15.96:0/3173639573 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55b80125f0 con 0x7f55c80ce1f0 2024-05-28T06:04:48.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.573+0000 7f55bf400640 1 -- 172.21.15.96:0/3173639573 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55b8019070 con 0x7f55c80ce1f0 2024-05-28T06:04:48.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.573+0000 7f55bf400640 1 -- 172.21.15.96:0/3173639573 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55b80129d0 con 0x7f55c80ce1f0 2024-05-28T06:04:48.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.573+0000 7f55cca00640 1 -- 172.21.15.96:0/3173639573 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55c8160f60 con 0x7f55c80ce1f0 2024-05-28T06:04:48.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.574+0000 7f55cca00640 1 -- 172.21.15.96:0/3173639573 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55c8161520 con 0x7f55c80ce1f0 2024-05-28T06:04:48.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.575+0000 7f55cca00640 1 -- 172.21.15.96:0/3173639573 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55c80689e0 con 0x7f55c80ce1f0 2024-05-28T06:04:48.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.576+0000 7f55bf400640 1 -- 172.21.15.96:0/3173639573 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f55b8012b70 con 0x7f55c80ce1f0 2024-05-28T06:04:48.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.577+0000 7f55bf400640 1 --2- 172.21.15.96:0/3173639573 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f55a806c8f0 0x7f55a806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:48.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.577+0000 7f55bf400640 1 -- 172.21.15.96:0/3173639573 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f55b808bfb0 con 0x7f55c80ce1f0 2024-05-28T06:04:48.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.577+0000 7f55c7400640 1 --2- 172.21.15.96:0/3173639573 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f55a806c8f0 0x7f55a806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:48.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.579+0000 7f55c7400640 1 --2- 172.21.15.96:0/3173639573 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f55a806c8f0 0x7f55a806edb0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f55b4034000 tx=0x7f55b402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:48.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.581+0000 7f55bf400640 1 -- 172.21.15.96:0/3173639573 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f55b805a3d0 con 0x7f55c80ce1f0 2024-05-28T06:04:48.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.782+0000 7f55cca00640 1 -- 172.21.15.96:0/3173639573 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f55c80ca070 con 0x7f55c80ce1f0 2024-05-28T06:04:48.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.783+0000 7f55bf400640 1 -- 172.21.15.96:0/3173639573 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) ==== 100+0+11 (secure 0 0 0) 0x7f55b8059cd0 con 0x7f55c80ce1f0 2024-05-28T06:04:48.785 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 64 2024-05-28T06:04:48.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.786+0000 7f55cca00640 1 -- 172.21.15.96:0/3173639573 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f55a806c8f0 msgr2=0x7f55a806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:48.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.786+0000 7f55cca00640 1 --2- 172.21.15.96:0/3173639573 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f55a806c8f0 0x7f55a806edb0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f55b4034000 tx=0x7f55b402e000 comp rx=0 tx=0).stop 2024-05-28T06:04:48.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.786+0000 7f55cca00640 1 -- 172.21.15.96:0/3173639573 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f55c80ce1f0 msgr2=0x7f55c8160780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:48.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.786+0000 7f55cca00640 1 --2- 172.21.15.96:0/3173639573 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f55c80ce1f0 0x7f55c8160780 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f55b800d5f0 tx=0x7f55b800dab0 comp rx=0 tx=0).stop 2024-05-28T06:04:48.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.786+0000 7f55cca00640 1 -- 172.21.15.96:0/3173639573 shutdown_connections 2024-05-28T06:04:48.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.786+0000 7f55cca00640 1 --2- 172.21.15.96:0/3173639573 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f55a806c8f0 0x7f55a806edb0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:48.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.786+0000 7f55cca00640 1 --2- 172.21.15.96:0/3173639573 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f55c80ce5d0 0x7f55c81f39d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:48.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.786+0000 7f55cca00640 1 --2- 172.21.15.96:0/3173639573 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f55c80ce1f0 0x7f55c8160780 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:48.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.786+0000 7f55cca00640 1 --2- 172.21.15.96:0/3173639573 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f55c80b3d60 0x7f55c80b1e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:48.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.786+0000 7f55cca00640 1 -- 172.21.15.96:0/3173639573 >> 172.21.15.96:0/3173639573 conn(0x7f55c806b900 msgr2=0x7f55c80c94d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:48.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.786+0000 7f55cca00640 1 -- 172.21.15.96:0/3173639573 shutdown_connections 2024-05-28T06:04:48.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:48.786+0000 7f55cca00640 1 -- 172.21.15.96:0/3173639573 wait complete. 2024-05-28T06:04:48.808 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:04:49.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.008+0000 7fc663e00640 1 Processor -- start 2024-05-28T06:04:49.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.008+0000 7fc663e00640 1 -- start start 2024-05-28T06:04:49.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.010+0000 7fc663e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fc664156b30 0x7fc6640af6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:49.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.010+0000 7fc663e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc66406ee40 0x7fc6640ad230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:49.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.010+0000 7fc663e00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc664156f10 con 0x7fc66406f2c0 2024-05-28T06:04:49.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.010+0000 7fc663e00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fc66406dda0 con 0x7fc664156b30 2024-05-28T06:04:49.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.010+0000 7fc663e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fc66406e080 con 0x7fc66406ee40 2024-05-28T06:04:49.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.010+0000 7fc663400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fc66406f2c0 0x7fc6640b32c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:58464/0 (socket says 172.21.15.96:58464) 2024-05-28T06:04:49.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.010+0000 7fc663400640 1 -- 172.21.15.96:0/2400733417 learned_addr learned my addr 172.21.15.96:0/2400733417 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:04:49.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.010+0000 7fc662a00640 1 --2- 172.21.15.96:0/2400733417 >> v2:172.21.15.123:3300/0 conn(0x7fc664156b30 0x7fc6640af6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:49.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.010+0000 7fc662000640 1 --2- 172.21.15.96:0/2400733417 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc66406ee40 0x7fc6640ad230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:49.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.011+0000 7fc661600640 1 -- 172.21.15.96:0/2400733417 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3908685943 0 0) 0x7fc664156f10 con 0x7fc66406f2c0 2024-05-28T06:04:49.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.011+0000 7fc661600640 1 -- 172.21.15.96:0/2400733417 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc648003520 con 0x7fc66406f2c0 2024-05-28T06:04:49.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.011+0000 7fc661600640 1 -- 172.21.15.96:0/2400733417 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2163552980 0 0) 0x7fc648003520 con 0x7fc66406f2c0 2024-05-28T06:04:49.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.011+0000 7fc661600640 1 -- 172.21.15.96:0/2400733417 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc66406ee40 msgr2=0x7fc6640ad230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:49.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.011+0000 7fc661600640 1 --2- 172.21.15.96:0/2400733417 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc66406ee40 0x7fc6640ad230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:49.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.011+0000 7fc661600640 1 -- 172.21.15.96:0/2400733417 >> v2:172.21.15.123:3300/0 conn(0x7fc664156b30 msgr2=0x7fc6640af6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:49.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.011+0000 7fc661600640 1 --2- 172.21.15.96:0/2400733417 >> v2:172.21.15.123:3300/0 conn(0x7fc664156b30 0x7fc6640af6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:49.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.011+0000 7fc661600640 1 -- 172.21.15.96:0/2400733417 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc66406e360 con 0x7fc66406f2c0 2024-05-28T06:04:49.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.011+0000 7fc661600640 1 -- 172.21.15.96:0/2400733417 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7fc65c0019f0 con 0x7fc66406f2c0 2024-05-28T06:04:49.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.011+0000 7fc661600640 1 -- 172.21.15.96:0/2400733417 >> v1:172.21.15.96:6789/0 conn(0x7fc66406f2c0 legacy=0x7fc6640b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:49.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.011+0000 7fc661600640 1 --2- 172.21.15.96:0/2400733417 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc6480039b0 0x7fc648003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:49.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.011+0000 7fc661600640 1 --2- 172.21.15.96:0/2400733417 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc648004800 0x7fc648004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:49.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.011+0000 7fc661600640 1 --2- 172.21.15.96:0/2400733417 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc66406ee40 0x7fc6480051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:49.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.011+0000 7fc661600640 1 -- 172.21.15.96:0/2400733417 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc66406dda0 con 0x7fc6480039b0 2024-05-28T06:04:49.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.011+0000 7fc661600640 1 -- 172.21.15.96:0/2400733417 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc66406e080 con 0x7fc648004800 2024-05-28T06:04:49.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.011+0000 7fc661600640 1 -- 172.21.15.96:0/2400733417 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fc64800b9f0 con 0x7fc66406ee40 2024-05-28T06:04:49.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.011+0000 7fc662a00640 1 --2- 172.21.15.96:0/2400733417 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc6480039b0 0x7fc648003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:49.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.012+0000 7fc662000640 1 --2- 172.21.15.96:0/2400733417 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc648004800 0x7fc648004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:49.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.012+0000 7fc663400640 1 --2- 172.21.15.96:0/2400733417 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc66406ee40 0x7fc6480051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:49.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.012+0000 7fc662a00640 1 -- 172.21.15.96:0/2400733417 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc66406ee40 msgr2=0x7fc6480051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:49.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.012+0000 7fc662a00640 1 --2- 172.21.15.96:0/2400733417 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc66406ee40 0x7fc6480051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:49.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.012+0000 7fc662a00640 1 -- 172.21.15.96:0/2400733417 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc648004800 msgr2=0x7fc648004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:49.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.012+0000 7fc662a00640 1 --2- 172.21.15.96:0/2400733417 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc648004800 0x7fc648004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:49.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.012+0000 7fc662a00640 1 -- 172.21.15.96:0/2400733417 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc648003520 con 0x7fc6480039b0 2024-05-28T06:04:49.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.012+0000 7fc663400640 1 --2- 172.21.15.96:0/2400733417 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc66406ee40 0x7fc6480051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:04:49.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.013+0000 7fc662a00640 1 --2- 172.21.15.96:0/2400733417 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc6480039b0 0x7fc648003dd0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fc64c009ac0 tx=0x7fc648005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff9f829983f466dc server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:49.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.013+0000 7fc661600640 1 -- 172.21.15.96:0/2400733417 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc64c036070 con 0x7fc6480039b0 2024-05-28T06:04:49.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.013+0000 7fc661600640 1 -- 172.21.15.96:0/2400733417 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc64c031400 con 0x7fc6480039b0 2024-05-28T06:04:49.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.013+0000 7fc661600640 1 -- 172.21.15.96:0/2400733417 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc64c009690 con 0x7fc6480039b0 2024-05-28T06:04:49.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.014+0000 7fc663e00640 1 -- 172.21.15.96:0/2400733417 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc6480039b0 msgr2=0x7fc648003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:49.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.014+0000 7fc663e00640 1 --2- 172.21.15.96:0/2400733417 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc6480039b0 0x7fc648003dd0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fc64c009ac0 tx=0x7fc648005830 comp rx=0 tx=0).stop 2024-05-28T06:04:49.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.014+0000 7fc663400640 1 -- 172.21.15.96:0/2400733417 reap_dead start 2024-05-28T06:04:49.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.014+0000 7fc663e00640 1 -- 172.21.15.96:0/2400733417 shutdown_connections 2024-05-28T06:04:49.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.014+0000 7fc663e00640 1 -- 172.21.15.96:0/2400733417 >> 172.21.15.96:0/2400733417 conn(0x7fc6640b1fc0 msgr2=0x7fc6640b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:49.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.014+0000 7fc663e00640 1 -- 172.21.15.96:0/2400733417 shutdown_connections 2024-05-28T06:04:49.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.014+0000 7fc663e00640 1 -- 172.21.15.96:0/2400733417 wait complete. 2024-05-28T06:04:49.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.016+0000 7fc663e00640 1 Processor -- start 2024-05-28T06:04:49.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.016+0000 7fc663e00640 1 -- start start 2024-05-28T06:04:49.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.017+0000 7fc663e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc6480039b0 0x7fc6641f3150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:49.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.017+0000 7fc663e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc6640995c0 0x7fc66415da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:49.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.017+0000 7fc663e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc66415df80 0x7fc664157f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:49.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.017+0000 7fc663e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc6640afc20 con 0x7fc6640995c0 2024-05-28T06:04:49.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.017+0000 7fc663e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc6640b2c30 con 0x7fc66415df80 2024-05-28T06:04:49.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.017+0000 7fc663e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fc66415c9a0 con 0x7fc6480039b0 2024-05-28T06:04:49.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.017+0000 7fc662000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc6640995c0 0x7fc66415da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:49.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.017+0000 7fc662000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc6640995c0 0x7fc66415da40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:36192/0 (socket says 172.21.15.96:36192) 2024-05-28T06:04:49.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.017+0000 7fc662000640 1 -- 172.21.15.96:0/3270695188 learned_addr learned my addr 172.21.15.96:0/3270695188 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:49.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.018+0000 7fc662a00640 1 --2- 172.21.15.96:0/3270695188 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc6480039b0 0x7fc6641f3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:49.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.018+0000 7fc663400640 1 --2- 172.21.15.96:0/3270695188 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc66415df80 0x7fc664157f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:49.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.018+0000 7fc662000640 1 -- 172.21.15.96:0/3270695188 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc6480039b0 msgr2=0x7fc6641f3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:49.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.018+0000 7fc662000640 1 --2- 172.21.15.96:0/3270695188 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc6480039b0 0x7fc6641f3150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:49.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.018+0000 7fc662000640 1 -- 172.21.15.96:0/3270695188 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc66415df80 msgr2=0x7fc664157f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:49.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.018+0000 7fc662000640 1 --2- 172.21.15.96:0/3270695188 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc66415df80 0x7fc664157f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:49.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.018+0000 7fc662000640 1 -- 172.21.15.96:0/3270695188 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc66415cc80 con 0x7fc6640995c0 2024-05-28T06:04:49.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.018+0000 7fc662a00640 1 --2- 172.21.15.96:0/3270695188 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc6480039b0 0x7fc6641f3150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:04:49.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.018+0000 7fc663400640 1 --2- 172.21.15.96:0/3270695188 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc66415df80 0x7fc664157f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:04:49.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.018+0000 7fc662000640 1 --2- 172.21.15.96:0/3270695188 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc6640995c0 0x7fc66415da40 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fc654011630 tx=0x7fc654011af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:49.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.018+0000 7fc65aa00640 1 -- 172.21.15.96:0/3270695188 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc65400f920 con 0x7fc6640995c0 2024-05-28T06:04:49.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.018+0000 7fc65aa00640 1 -- 172.21.15.96:0/3270695188 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc65400fac0 con 0x7fc6640995c0 2024-05-28T06:04:49.022 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.018+0000 7fc65aa00640 1 -- 172.21.15.96:0/3270695188 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc65400fe70 con 0x7fc6640995c0 2024-05-28T06:04:49.022 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.018+0000 7fc663e00640 1 -- 172.21.15.96:0/3270695188 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc66415cf60 con 0x7fc6640995c0 2024-05-28T06:04:49.022 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.019+0000 7fc663e00640 1 -- 172.21.15.96:0/3270695188 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc66415d520 con 0x7fc6640995c0 2024-05-28T06:04:49.022 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.020+0000 7fc663e00640 1 -- 172.21.15.96:0/3270695188 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6640689e0 con 0x7fc6640995c0 2024-05-28T06:04:49.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.021+0000 7fc65aa00640 1 -- 172.21.15.96:0/3270695188 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc654015460 con 0x7fc6640995c0 2024-05-28T06:04:49.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.021+0000 7fc65aa00640 1 --2- 172.21.15.96:0/3270695188 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc64006c8f0 0x7fc64006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:49.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.022+0000 7fc65aa00640 1 -- 172.21.15.96:0/3270695188 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fc65408c5c0 con 0x7fc6640995c0 2024-05-28T06:04:49.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.022+0000 7fc662a00640 1 --2- 172.21.15.96:0/3270695188 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc64006c8f0 0x7fc64006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:49.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.023+0000 7fc662a00640 1 --2- 172.21.15.96:0/3270695188 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc64006c8f0 0x7fc64006edb0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fc64c033000 tx=0x7fc64c008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:49.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.026+0000 7fc65aa00640 1 -- 172.21.15.96:0/3270695188 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc654019030 con 0x7fc6640995c0 2024-05-28T06:04:49.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.228+0000 7fc663e00640 1 -- 172.21.15.96:0/3270695188 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc664155050 con 0x7fc6640995c0 2024-05-28T06:04:49.231 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.230+0000 7fc65aa00640 1 -- 172.21.15.96:0/3270695188 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16617 (secure 0 0 0) 0x7fc65405a950 con 0x7fc6640995c0 2024-05-28T06:04:49.232 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:04:49.232 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":20,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:04:11.882922+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-28T06:04:08.741938+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-28T06:04:09.301470+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:04:49.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.233+0000 7fc663e00640 1 -- 172.21.15.96:0/3270695188 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc64006c8f0 msgr2=0x7fc64006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:49.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.233+0000 7fc663e00640 1 --2- 172.21.15.96:0/3270695188 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc64006c8f0 0x7fc64006edb0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fc64c033000 tx=0x7fc64c008030 comp rx=0 tx=0).stop 2024-05-28T06:04:49.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.233+0000 7fc663e00640 1 -- 172.21.15.96:0/3270695188 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc6640995c0 msgr2=0x7fc66415da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:49.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.233+0000 7fc663e00640 1 --2- 172.21.15.96:0/3270695188 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc6640995c0 0x7fc66415da40 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fc654011630 tx=0x7fc654011af0 comp rx=0 tx=0).stop 2024-05-28T06:04:49.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.233+0000 7fc663e00640 1 -- 172.21.15.96:0/3270695188 shutdown_connections 2024-05-28T06:04:49.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.233+0000 7fc663e00640 1 --2- 172.21.15.96:0/3270695188 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc64006c8f0 0x7fc64006edb0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:49.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.233+0000 7fc663e00640 1 --2- 172.21.15.96:0/3270695188 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc66415df80 0x7fc664157f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:49.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.233+0000 7fc663e00640 1 --2- 172.21.15.96:0/3270695188 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc6640995c0 0x7fc66415da40 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:49.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.233+0000 7fc663e00640 1 --2- 172.21.15.96:0/3270695188 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc6480039b0 0x7fc6641f3150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:49.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.233+0000 7fc663e00640 1 -- 172.21.15.96:0/3270695188 >> 172.21.15.96:0/3270695188 conn(0x7fc6640b1fc0 msgr2=0x7fc6640b4be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:49.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.233+0000 7fc663e00640 1 -- 172.21.15.96:0/3270695188 shutdown_connections 2024-05-28T06:04:49.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.233+0000 7fc663e00640 1 -- 172.21.15.96:0/3270695188 wait complete. 2024-05-28T06:04:49.255 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-28T06:03:59.517102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-28T06:04:08.741938+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-28T06:04:09.301470+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-28T06:04:49.256 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-28T06:04:49.469 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.467+0000 7f91b7600640 1 Processor -- start 2024-05-28T06:04:49.469 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.468+0000 7f91b7600640 1 -- start start 2024-05-28T06:04:49.470 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.469+0000 7f91b7600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f91b00d0640 0x7f91b015d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:49.471 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.469+0000 7f91b7600640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91b00d0060 con 0x7f91b00d0a20 2024-05-28T06:04:49.471 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.469+0000 7f91b7600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91b015cb10 con 0x7f91b00b32c0 2024-05-28T06:04:49.471 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.469+0000 7f91b7600640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f91b00adbb0 con 0x7f91b00d0640 2024-05-28T06:04:49.471 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.469+0000 7f91b4c00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f91b00d0640 0x7f91b015d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:49.471 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.470+0000 7f91b4c00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f91b00d0640 0x7f91b015d5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:36194/0 (socket says 172.21.15.96:36194) 2024-05-28T06:04:49.471 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.470+0000 7f91b4c00640 1 -- 172.21.15.96:0/925538521 learned_addr learned my addr 172.21.15.96:0/925538521 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:49.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.470+0000 7f91a7e00640 1 --1- 172.21.15.96:0/925538521 >> v1:172.21.15.96:6789/0 conn(0x7f91b00d0a20 0x7f91b00cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:58468/0 (socket says 172.21.15.96:58468) 2024-05-28T06:04:49.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.470+0000 7f91b4c00640 1 -- 172.21.15.96:0/925538521 >> v1:172.21.15.123:6789/0 conn(0x7f91b00b32c0 legacy=0x7f91b00b3730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:49.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.470+0000 7f91b4c00640 1 -- 172.21.15.96:0/925538521 >> v1:172.21.15.96:6789/0 conn(0x7f91b00d0a20 legacy=0x7f91b00cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:49.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.470+0000 7f91b4c00640 1 -- 172.21.15.96:0/925538521 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91b00ad5f0 con 0x7f91b00d0640 2024-05-28T06:04:49.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.470+0000 7f91afe00640 1 -- 172.21.15.96:0/925538521 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2572294861 0 0) 0x7f91b00d0060 con 0x7f91b00d0a20 2024-05-28T06:04:49.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.471+0000 7f91b4c00640 1 --2- 172.21.15.96:0/925538521 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f91b00d0640 0x7f91b015d5d0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f91a0009870 tx=0x7f91a002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c558aa506acbc7b0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:49.473 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.471+0000 7f91afe00640 1 -- 172.21.15.96:0/925538521 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91a0037070 con 0x7f91b00d0640 2024-05-28T06:04:49.473 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.471+0000 7f91afe00640 1 -- 172.21.15.96:0/925538521 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91a0032440 con 0x7f91b00d0640 2024-05-28T06:04:49.473 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.471+0000 7f91afe00640 1 -- 172.21.15.96:0/925538521 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91a0032810 con 0x7f91b00d0640 2024-05-28T06:04:49.474 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.472+0000 7f91b7600640 1 -- 172.21.15.96:0/925538521 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f91b00d0640 msgr2=0x7f91b015d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:49.474 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.472+0000 7f91b7600640 1 --2- 172.21.15.96:0/925538521 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f91b00d0640 0x7f91b015d5d0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f91a0009870 tx=0x7f91a002a270 comp rx=0 tx=0).stop 2024-05-28T06:04:49.474 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.472+0000 7f91b7600640 1 -- 172.21.15.96:0/925538521 shutdown_connections 2024-05-28T06:04:49.474 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.472+0000 7f91b7600640 1 --2- 172.21.15.96:0/925538521 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f91b00d0640 0x7f91b015d5d0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:49.474 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.472+0000 7f91b7600640 1 -- 172.21.15.96:0/925538521 >> 172.21.15.96:0/925538521 conn(0x7f91b006bd20 msgr2=0x7f91b006c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:49.474 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.472+0000 7f91b7600640 1 -- 172.21.15.96:0/925538521 shutdown_connections 2024-05-28T06:04:49.474 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.472+0000 7f91b7600640 1 -- 172.21.15.96:0/925538521 wait complete. 2024-05-28T06:04:49.475 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.474+0000 7f91b7600640 1 Processor -- start 2024-05-28T06:04:49.475 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.474+0000 7f91b7600640 1 -- start start 2024-05-28T06:04:49.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.475+0000 7f91b7600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f91b00b32c0 0x7f91b014aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:49.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.475+0000 7f91b7600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f91b00d0640 0x7f91b014f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:49.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.475+0000 7f91b7600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91b00d0a20 0x7f91b014d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:49.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.475+0000 7f91b7600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f91b015cdd0 con 0x7f91b00b32c0 2024-05-28T06:04:49.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.475+0000 7f91b7600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f91b015db10 con 0x7f91b00d0a20 2024-05-28T06:04:49.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.475+0000 7f91b7600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f91b015d0b0 con 0x7f91b00d0640 2024-05-28T06:04:49.478 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.475+0000 7f91b4c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f91b00b32c0 0x7f91b014aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:49.478 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.475+0000 7f91b4c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f91b00b32c0 0x7f91b014aab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:36198/0 (socket says 172.21.15.96:36198) 2024-05-28T06:04:49.478 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.475+0000 7f91b4c00640 1 -- 172.21.15.96:0/3979091528 learned_addr learned my addr 172.21.15.96:0/3979091528 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:49.478 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.476+0000 7f91b4c00640 1 -- 172.21.15.96:0/3979091528 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f91b00d0640 msgr2=0x7f91b014f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:49.478 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.476+0000 7f91b4c00640 1 --2- 172.21.15.96:0/3979091528 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f91b00d0640 0x7f91b014f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:49.478 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.476+0000 7f91b4c00640 1 -- 172.21.15.96:0/3979091528 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91b00d0a20 msgr2=0x7f91b014d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:49.478 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.476+0000 7f91b4c00640 1 --2- 172.21.15.96:0/3979091528 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91b00d0a20 0x7f91b014d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:49.479 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.476+0000 7f91b4c00640 1 -- 172.21.15.96:0/3979091528 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91b01f59d0 con 0x7f91b00b32c0 2024-05-28T06:04:49.479 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.476+0000 7f91b4c00640 1 --2- 172.21.15.96:0/3979091528 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f91b00b32c0 0x7f91b014aab0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f91a0009870 tx=0x7f91a00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:49.479 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.476+0000 7f91ad600640 1 -- 172.21.15.96:0/3979091528 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91a0037040 con 0x7f91b00b32c0 2024-05-28T06:04:49.479 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.476+0000 7f91ad600640 1 -- 172.21.15.96:0/3979091528 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91a0042070 con 0x7f91b00b32c0 2024-05-28T06:04:49.479 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.476+0000 7f91ad600640 1 -- 172.21.15.96:0/3979091528 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91a0030bb0 con 0x7f91b00b32c0 2024-05-28T06:04:49.479 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.476+0000 7f91b7600640 1 -- 172.21.15.96:0/3979091528 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91b014be20 con 0x7f91b00b32c0 2024-05-28T06:04:49.479 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.477+0000 7f91b7600640 1 -- 172.21.15.96:0/3979091528 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91b014f760 con 0x7f91b00b32c0 2024-05-28T06:04:49.479 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.478+0000 7f91b7600640 1 -- 172.21.15.96:0/3979091528 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91b00689e0 con 0x7f91b00b32c0 2024-05-28T06:04:49.481 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.478+0000 7f91ad600640 1 -- 172.21.15.96:0/3979091528 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f91a0047460 con 0x7f91b00b32c0 2024-05-28T06:04:49.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.480+0000 7f91ad600640 1 --2- 172.21.15.96:0/3979091528 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f918c06c8f0 0x7f918c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:49.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.480+0000 7f91ad600640 1 -- 172.21.15.96:0/3979091528 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f91a00ac710 con 0x7f91b00b32c0 2024-05-28T06:04:49.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.480+0000 7f91a7e00640 1 --2- 172.21.15.96:0/3979091528 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f918c06c8f0 0x7f918c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:49.483 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.481+0000 7f91a7e00640 1 --2- 172.21.15.96:0/3979091528 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f918c06c8f0 0x7f918c06edb0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f9198000c00 tx=0x7f91980079c0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:49.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.484+0000 7f91ad600640 1 -- 172.21.15.96:0/3979091528 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f91a007abd0 con 0x7f91b00b32c0 2024-05-28T06:04:49.687 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.686+0000 7f91b7600640 1 -- 172.21.15.96:0/3979091528 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f91b015c250 con 0x7f91b00b32c0 2024-05-28T06:04:49.688 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.687+0000 7f91ad600640 1 -- 172.21.15.96:0/3979091528 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) ==== 93+0+10 (secure 0 0 0) 0x7f91a007a4d0 con 0x7f91b00b32c0 2024-05-28T06:04:49.689 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 1 2024-05-28T06:04:49.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.690+0000 7f91b7600640 1 -- 172.21.15.96:0/3979091528 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f918c06c8f0 msgr2=0x7f918c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:49.693 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.690+0000 7f91b7600640 1 --2- 172.21.15.96:0/3979091528 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f918c06c8f0 0x7f918c06edb0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f9198000c00 tx=0x7f91980079c0 comp rx=0 tx=0).stop 2024-05-28T06:04:49.693 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.690+0000 7f91b7600640 1 -- 172.21.15.96:0/3979091528 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f91b00b32c0 msgr2=0x7f91b014aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:49.693 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.690+0000 7f91b7600640 1 --2- 172.21.15.96:0/3979091528 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f91b00b32c0 0x7f91b014aab0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f91a0009870 tx=0x7f91a00046c0 comp rx=0 tx=0).stop 2024-05-28T06:04:49.693 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.690+0000 7f91b7600640 1 -- 172.21.15.96:0/3979091528 shutdown_connections 2024-05-28T06:04:49.693 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.690+0000 7f91b7600640 1 --2- 172.21.15.96:0/3979091528 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f918c06c8f0 0x7f918c06edb0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:49.693 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.690+0000 7f91b7600640 1 --2- 172.21.15.96:0/3979091528 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91b00d0a20 0x7f91b014d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:49.693 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.690+0000 7f91b7600640 1 --2- 172.21.15.96:0/3979091528 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f91b00d0640 0x7f91b014f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:49.694 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.690+0000 7f91b7600640 1 --2- 172.21.15.96:0/3979091528 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f91b00b32c0 0x7f91b014aab0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:49.694 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.690+0000 7f91b7600640 1 -- 172.21.15.96:0/3979091528 >> 172.21.15.96:0/3979091528 conn(0x7f91b006bd20 msgr2=0x7f91b014b270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:49.694 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.690+0000 7f91b7600640 1 -- 172.21.15.96:0/3979091528 shutdown_connections 2024-05-28T06:04:49.694 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.690+0000 7f91b7600640 1 -- 172.21.15.96:0/3979091528 wait complete. 2024-05-28T06:04:49.710 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-28T06:04:49.917 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.914+0000 7f4137e00640 1 Processor -- start 2024-05-28T06:04:49.917 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.914+0000 7f4137e00640 1 -- start start 2024-05-28T06:04:49.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.916+0000 7f4137e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f41300d0640 0x7f413015d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:49.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.916+0000 7f4137e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f41300d0a20 0x7f41300cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:49.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.916+0000 7f4137e00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f413015cb10 con 0x7f41300b32c0 2024-05-28T06:04:49.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.916+0000 7f4137e00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f41300adbb0 con 0x7f41300d0640 2024-05-28T06:04:49.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.916+0000 7f4137e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f413015db10 con 0x7f41300d0a20 2024-05-28T06:04:49.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.916+0000 7f4135400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f41300d0640 0x7f413015d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:49.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.916+0000 7f4135e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f41300b32c0 0x7f41300b3750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:58480/0 (socket says 172.21.15.96:58480) 2024-05-28T06:04:49.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.916+0000 7f4135e00640 1 -- 172.21.15.96:0/3349208219 learned_addr learned my addr 172.21.15.96:0/3349208219 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:04:49.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.917+0000 7f4134a00640 1 --2- 172.21.15.96:0/3349208219 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f41300d0a20 0x7f41300cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:49.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.917+0000 7f4135400640 1 -- 172.21.15.96:0/3349208219 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f41300d0a20 msgr2=0x7f41300cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:49.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.917+0000 7f4135400640 1 --2- 172.21.15.96:0/3349208219 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f41300d0a20 0x7f41300cf950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:49.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.917+0000 7f4135400640 1 -- 172.21.15.96:0/3349208219 >> v1:172.21.15.96:6789/0 conn(0x7f41300b32c0 legacy=0x7f41300b3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:49.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.917+0000 7f412fe00640 1 -- 172.21.15.96:0/3349208219 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3759865871 0 0) 0x7f413015cb10 con 0x7f41300b32c0 2024-05-28T06:04:49.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.917+0000 7f4135400640 1 -- 172.21.15.96:0/3349208219 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41300ad5f0 con 0x7f41300d0640 2024-05-28T06:04:49.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.917+0000 7f4134a00640 1 --2- 172.21.15.96:0/3349208219 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f41300d0a20 0x7f41300cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:04:49.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.918+0000 7f4135400640 1 --2- 172.21.15.96:0/3349208219 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41300d0640 0x7f413015d5d0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f4128009d90 tx=0x7f412802a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=199c065fbe5d53ac server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:49.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.918+0000 7f412fe00640 1 -- 172.21.15.96:0/3349208219 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4128007740 con 0x7f41300d0640 2024-05-28T06:04:49.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.918+0000 7f412fe00640 1 -- 172.21.15.96:0/3349208219 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41280078e0 con 0x7f41300d0640 2024-05-28T06:04:49.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.918+0000 7f412fe00640 1 -- 172.21.15.96:0/3349208219 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4128007cb0 con 0x7f41300d0640 2024-05-28T06:04:49.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.919+0000 7f4137e00640 1 -- 172.21.15.96:0/3349208219 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41300d0640 msgr2=0x7f413015d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:49.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.919+0000 7f4137e00640 1 --2- 172.21.15.96:0/3349208219 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41300d0640 0x7f413015d5d0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f4128009d90 tx=0x7f412802a660 comp rx=0 tx=0).stop 2024-05-28T06:04:49.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.919+0000 7f4137e00640 1 -- 172.21.15.96:0/3349208219 shutdown_connections 2024-05-28T06:04:49.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.919+0000 7f4137e00640 1 --2- 172.21.15.96:0/3349208219 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f41300d0a20 0x7f41300cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:49.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.919+0000 7f4137e00640 1 --2- 172.21.15.96:0/3349208219 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41300d0640 0x7f413015d5d0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:49.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.919+0000 7f4137e00640 1 -- 172.21.15.96:0/3349208219 >> 172.21.15.96:0/3349208219 conn(0x7f413006bd20 msgr2=0x7f413006c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:49.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.919+0000 7f4137e00640 1 -- 172.21.15.96:0/3349208219 shutdown_connections 2024-05-28T06:04:49.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.919+0000 7f4137e00640 1 -- 172.21.15.96:0/3349208219 wait complete. 2024-05-28T06:04:49.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.921+0000 7f4137e00640 1 Processor -- start 2024-05-28T06:04:49.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.921+0000 7f4137e00640 1 -- start start 2024-05-28T06:04:49.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.922+0000 7f4137e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41300b32c0 0x7f4130144070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:49.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.922+0000 7f4137e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f41300d0640 0x7f413014e330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:49.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.922+0000 7f4137e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41300d0a20 0x7f413015ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:49.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.922+0000 7f4137e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f413015d0b0 con 0x7f41300d0a20 2024-05-28T06:04:49.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.922+0000 7f4137e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f413015cdd0 con 0x7f41300b32c0 2024-05-28T06:04:49.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.922+0000 7f4137e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f41300b3e60 con 0x7f41300d0640 2024-05-28T06:04:49.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.922+0000 7f4135e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41300d0a20 0x7f413015ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:49.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.922+0000 7f4135e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41300d0a20 0x7f413015ef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:36208/0 (socket says 172.21.15.96:36208) 2024-05-28T06:04:49.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.922+0000 7f4135e00640 1 -- 172.21.15.96:0/1894330706 learned_addr learned my addr 172.21.15.96:0/1894330706 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:49.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.922+0000 7f4135e00640 1 -- 172.21.15.96:0/1894330706 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f41300d0640 msgr2=0x7f413014e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:49.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.922+0000 7f4135e00640 1 --2- 172.21.15.96:0/1894330706 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f41300d0640 0x7f413014e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:49.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.922+0000 7f4135e00640 1 -- 172.21.15.96:0/1894330706 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41300b32c0 msgr2=0x7f4130144070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:49.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.922+0000 7f4135e00640 1 --2- 172.21.15.96:0/1894330706 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41300b32c0 0x7f4130144070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:49.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.922+0000 7f4135e00640 1 -- 172.21.15.96:0/1894330706 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f413006e030 con 0x7f41300d0a20 2024-05-28T06:04:49.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.923+0000 7f4135e00640 1 --2- 172.21.15.96:0/1894330706 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41300d0a20 0x7f413015ef70 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f412400c450 tx=0x7f412400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:49.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.923+0000 7f412d600640 1 -- 172.21.15.96:0/1894330706 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4124018070 con 0x7f41300d0a20 2024-05-28T06:04:49.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.923+0000 7f412d600640 1 -- 172.21.15.96:0/1894330706 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4124007500 con 0x7f41300d0a20 2024-05-28T06:04:49.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.923+0000 7f4137e00640 1 -- 172.21.15.96:0/1894330706 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f413006e310 con 0x7f41300d0a20 2024-05-28T06:04:49.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.923+0000 7f412d600640 1 -- 172.21.15.96:0/1894330706 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4124004a20 con 0x7f41300d0a20 2024-05-28T06:04:49.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.923+0000 7f4137e00640 1 -- 172.21.15.96:0/1894330706 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f413006e8d0 con 0x7f41300d0a20 2024-05-28T06:04:49.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.924+0000 7f4137e00640 1 -- 172.21.15.96:0/1894330706 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41300689e0 con 0x7f41300d0a20 2024-05-28T06:04:49.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.925+0000 7f412d600640 1 -- 172.21.15.96:0/1894330706 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f412401b070 con 0x7f41300d0a20 2024-05-28T06:04:49.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.926+0000 7f412d600640 1 --2- 172.21.15.96:0/1894330706 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f411006c940 0x7f411006ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:49.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.926+0000 7f412d600640 1 -- 172.21.15.96:0/1894330706 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f412408af80 con 0x7f41300d0a20 2024-05-28T06:04:49.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.927+0000 7f4135400640 1 --2- 172.21.15.96:0/1894330706 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f411006c940 0x7f411006ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:49.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.928+0000 7f4135400640 1 --2- 172.21.15.96:0/1894330706 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f411006c940 0x7f411006ee00 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f4128034000 tx=0x7f412802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:49.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:49.931+0000 7f412d600640 1 -- 172.21.15.96:0/1894330706 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4124059450 con 0x7f41300d0a20 2024-05-28T06:04:50.131 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.129+0000 7f4137e00640 1 -- 172.21.15.96:0/1894330706 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f413006eaa0 con 0x7f41300d0a20 2024-05-28T06:04:50.132 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.130+0000 7f412d600640 1 -- 172.21.15.96:0/1894330706 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) ==== 104+0+11 (secure 0 0 0) 0x7f4124058d50 con 0x7f41300d0a20 2024-05-28T06:04:50.132 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 32 2024-05-28T06:04:50.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.132+0000 7f4137e00640 1 -- 172.21.15.96:0/1894330706 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f411006c940 msgr2=0x7f411006ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:50.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.132+0000 7f4137e00640 1 --2- 172.21.15.96:0/1894330706 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f411006c940 0x7f411006ee00 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f4128034000 tx=0x7f412802e000 comp rx=0 tx=0).stop 2024-05-28T06:04:50.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.133+0000 7f4137e00640 1 -- 172.21.15.96:0/1894330706 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41300d0a20 msgr2=0x7f413015ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:50.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.133+0000 7f4137e00640 1 --2- 172.21.15.96:0/1894330706 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41300d0a20 0x7f413015ef70 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f412400c450 tx=0x7f412400c910 comp rx=0 tx=0).stop 2024-05-28T06:04:50.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.133+0000 7f4137e00640 1 -- 172.21.15.96:0/1894330706 shutdown_connections 2024-05-28T06:04:50.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.133+0000 7f4137e00640 1 --2- 172.21.15.96:0/1894330706 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f411006c940 0x7f411006ee00 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:50.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.133+0000 7f4137e00640 1 --2- 172.21.15.96:0/1894330706 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41300d0a20 0x7f413015ef70 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:50.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.133+0000 7f4137e00640 1 --2- 172.21.15.96:0/1894330706 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f41300d0640 0x7f413014e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:50.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.133+0000 7f4137e00640 1 --2- 172.21.15.96:0/1894330706 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41300b32c0 0x7f4130144070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:50.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.133+0000 7f4137e00640 1 -- 172.21.15.96:0/1894330706 >> 172.21.15.96:0/1894330706 conn(0x7f413006bd20 msgr2=0x7f41300b3750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:50.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.133+0000 7f4137e00640 1 -- 172.21.15.96:0/1894330706 shutdown_connections 2024-05-28T06:04:50.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.133+0000 7f4137e00640 1 -- 172.21.15.96:0/1894330706 wait complete. 2024-05-28T06:04:50.156 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-28T06:04:50.355 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.354+0000 7fa330600640 1 Processor -- start 2024-05-28T06:04:50.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.354+0000 7fa330600640 1 -- start start 2024-05-28T06:04:50.358 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.356+0000 7fa330600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa32815d5d0 0x7fa3280cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:50.358 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.356+0000 7fa330600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fa3280b32c0 0x7fa3280b3750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:50.358 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.356+0000 7fa330600640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa32815cb10 con 0x7fa3280d0640 2024-05-28T06:04:50.358 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.356+0000 7fa330600640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fa3280adbb0 con 0x7fa3280b32c0 2024-05-28T06:04:50.358 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.356+0000 7fa330600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fa3280ad5f0 con 0x7fa32815d5d0 2024-05-28T06:04:50.358 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.356+0000 7fa32e600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fa3280b32c0 0x7fa3280b3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:50.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.356+0000 7fa32e600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fa3280b32c0 0x7fa3280b3750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:36212/0 (socket says 172.21.15.96:36212) 2024-05-28T06:04:50.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.356+0000 7fa32e600640 1 -- 172.21.15.96:0/1912955266 learned_addr learned my addr 172.21.15.96:0/1912955266 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:50.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.356+0000 7fa32d200640 1 --2- 172.21.15.96:0/1912955266 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa32815d5d0 0x7fa3280cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:50.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.357+0000 7fa32e600640 1 -- 172.21.15.96:0/1912955266 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa32815d5d0 msgr2=0x7fa3280cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:50.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.357+0000 7fa32e600640 1 --2- 172.21.15.96:0/1912955266 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa32815d5d0 0x7fa3280cf950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:50.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.357+0000 7fa32e600640 1 -- 172.21.15.96:0/1912955266 >> v1:172.21.15.96:6789/0 conn(0x7fa3280d0640 legacy=0x7fa3280d0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:50.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.357+0000 7fa32e600640 1 -- 172.21.15.96:0/1912955266 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3280ad8d0 con 0x7fa3280b32c0 2024-05-28T06:04:50.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.358+0000 7fa32e600640 1 --2- 172.21.15.96:0/1912955266 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa3280b32c0 0x7fa3280b3750 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fa320009e50 tx=0x7fa32002a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcbad37f683ed442 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:50.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.358+0000 7fa327e00640 1 -- 172.21.15.96:0/1912955266 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa320036070 con 0x7fa3280b32c0 2024-05-28T06:04:50.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.358+0000 7fa327e00640 1 -- 172.21.15.96:0/1912955266 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa320031600 con 0x7fa3280b32c0 2024-05-28T06:04:50.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.358+0000 7fa327e00640 1 -- 172.21.15.96:0/1912955266 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3200319d0 con 0x7fa3280b32c0 2024-05-28T06:04:50.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.359+0000 7fa330600640 1 -- 172.21.15.96:0/1912955266 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa3280b32c0 msgr2=0x7fa3280b3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:50.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.359+0000 7fa330600640 1 --2- 172.21.15.96:0/1912955266 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa3280b32c0 0x7fa3280b3750 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fa320009e50 tx=0x7fa32002a710 comp rx=0 tx=0).stop 2024-05-28T06:04:50.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.359+0000 7fa330600640 1 -- 172.21.15.96:0/1912955266 shutdown_connections 2024-05-28T06:04:50.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.359+0000 7fa330600640 1 --2- 172.21.15.96:0/1912955266 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa3280b32c0 0x7fa3280b3750 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:50.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.359+0000 7fa330600640 1 --2- 172.21.15.96:0/1912955266 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa32815d5d0 0x7fa3280cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:50.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.359+0000 7fa330600640 1 -- 172.21.15.96:0/1912955266 >> 172.21.15.96:0/1912955266 conn(0x7fa32806bd20 msgr2=0x7fa32806c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:50.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.359+0000 7fa330600640 1 -- 172.21.15.96:0/1912955266 shutdown_connections 2024-05-28T06:04:50.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.359+0000 7fa330600640 1 -- 172.21.15.96:0/1912955266 wait complete. 2024-05-28T06:04:50.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.360+0000 7fa330600640 1 Processor -- start 2024-05-28T06:04:50.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.361+0000 7fa330600640 1 -- start start 2024-05-28T06:04:50.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.361+0000 7fa330600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa3280b32c0 0x7fa3280b05c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:50.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.361+0000 7fa330600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa3280d0640 0x7fa32814c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:50.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.362+0000 7fa330600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa32815d5d0 0x7fa32814c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:50.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.362+0000 7fa330600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa32815da50 con 0x7fa3280d0640 2024-05-28T06:04:50.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.362+0000 7fa330600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa32815d0b0 con 0x7fa32815d5d0 2024-05-28T06:04:50.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.362+0000 7fa330600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fa3280ad2a0 con 0x7fa3280b32c0 2024-05-28T06:04:50.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.362+0000 7fa32d200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa3280d0640 0x7fa32814c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:50.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.362+0000 7fa32d200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa3280d0640 0x7fa32814c4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:36218/0 (socket says 172.21.15.96:36218) 2024-05-28T06:04:50.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.362+0000 7fa32d200640 1 -- 172.21.15.96:0/3959529343 learned_addr learned my addr 172.21.15.96:0/3959529343 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:50.368 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.362+0000 7fa32e600640 1 --2- 172.21.15.96:0/3959529343 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa32815d5d0 0x7fa32814c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:50.368 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.362+0000 7fa32d200640 1 -- 172.21.15.96:0/3959529343 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa3280b32c0 msgr2=0x7fa3280b05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:50.368 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.362+0000 7fa32dc00640 1 --2- 172.21.15.96:0/3959529343 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa3280b32c0 0x7fa3280b05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:50.368 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.362+0000 7fa32d200640 1 --2- 172.21.15.96:0/3959529343 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa3280b32c0 0x7fa3280b05c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:50.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.362+0000 7fa32d200640 1 -- 172.21.15.96:0/3959529343 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa32815d5d0 msgr2=0x7fa32814c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:50.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.362+0000 7fa32d200640 1 --2- 172.21.15.96:0/3959529343 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa32815d5d0 0x7fa32814c9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:50.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.362+0000 7fa32d200640 1 -- 172.21.15.96:0/3959529343 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa32806e3d0 con 0x7fa3280d0640 2024-05-28T06:04:50.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.362+0000 7fa32e600640 1 --2- 172.21.15.96:0/3959529343 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa32815d5d0 0x7fa32814c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:04:50.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.362+0000 7fa32dc00640 1 --2- 172.21.15.96:0/3959529343 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa3280b32c0 0x7fa3280b05c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:04:50.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.363+0000 7fa32d200640 1 --2- 172.21.15.96:0/3959529343 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa3280d0640 0x7fa32814c4a0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fa31400d5f0 tx=0x7fa31400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:50.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.363+0000 7fa325600640 1 -- 172.21.15.96:0/3959529343 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3140125f0 con 0x7fa3280d0640 2024-05-28T06:04:50.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.363+0000 7fa325600640 1 -- 172.21.15.96:0/3959529343 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa314019070 con 0x7fa3280d0640 2024-05-28T06:04:50.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.363+0000 7fa325600640 1 -- 172.21.15.96:0/3959529343 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3140129d0 con 0x7fa3280d0640 2024-05-28T06:04:50.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.363+0000 7fa330600640 1 -- 172.21.15.96:0/3959529343 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa32806e6b0 con 0x7fa3280d0640 2024-05-28T06:04:50.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.363+0000 7fa330600640 1 -- 172.21.15.96:0/3959529343 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa32806ec70 con 0x7fa3280d0640 2024-05-28T06:04:50.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.364+0000 7fa330600640 1 -- 172.21.15.96:0/3959529343 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3280689e0 con 0x7fa3280d0640 2024-05-28T06:04:50.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.365+0000 7fa325600640 1 -- 172.21.15.96:0/3959529343 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa314012b70 con 0x7fa3280d0640 2024-05-28T06:04:50.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.366+0000 7fa325600640 1 --2- 172.21.15.96:0/3959529343 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa30806c8f0 0x7fa30806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:50.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.366+0000 7fa325600640 1 -- 172.21.15.96:0/3959529343 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fa31408bf20 con 0x7fa3280d0640 2024-05-28T06:04:50.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.367+0000 7fa32dc00640 1 --2- 172.21.15.96:0/3959529343 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa30806c8f0 0x7fa30806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:50.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.368+0000 7fa32dc00640 1 --2- 172.21.15.96:0/3959529343 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa30806c8f0 0x7fa30806edb0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fa32814e2f0 tx=0x7fa31c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:50.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.370+0000 7fa325600640 1 -- 172.21.15.96:0/3959529343 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa31405a340 con 0x7fa3280d0640 2024-05-28T06:04:50.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.569+0000 7fa330600640 1 -- 172.21.15.96:0/3959529343 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fa328071ef0 con 0x7fa3280d0640 2024-05-28T06:04:50.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.570+0000 7fa325600640 1 -- 172.21.15.96:0/3959529343 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) ==== 100+0+11 (secure 0 0 0) 0x7fa314059c40 con 0x7fa3280d0640 2024-05-28T06:04:50.571 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 64 2024-05-28T06:04:50.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.572+0000 7fa330600640 1 -- 172.21.15.96:0/3959529343 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa30806c8f0 msgr2=0x7fa30806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:50.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.572+0000 7fa330600640 1 --2- 172.21.15.96:0/3959529343 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa30806c8f0 0x7fa30806edb0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fa32814e2f0 tx=0x7fa31c009040 comp rx=0 tx=0).stop 2024-05-28T06:04:50.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.573+0000 7fa330600640 1 -- 172.21.15.96:0/3959529343 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa3280d0640 msgr2=0x7fa32814c4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:50.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.573+0000 7fa330600640 1 --2- 172.21.15.96:0/3959529343 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa3280d0640 0x7fa32814c4a0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fa31400d5f0 tx=0x7fa31400dab0 comp rx=0 tx=0).stop 2024-05-28T06:04:50.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.573+0000 7fa330600640 1 -- 172.21.15.96:0/3959529343 shutdown_connections 2024-05-28T06:04:50.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.573+0000 7fa330600640 1 --2- 172.21.15.96:0/3959529343 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa30806c8f0 0x7fa30806edb0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:50.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.573+0000 7fa330600640 1 --2- 172.21.15.96:0/3959529343 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa32815d5d0 0x7fa32814c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:50.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.573+0000 7fa330600640 1 --2- 172.21.15.96:0/3959529343 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa3280d0640 0x7fa32814c4a0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:50.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.573+0000 7fa330600640 1 --2- 172.21.15.96:0/3959529343 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa3280b32c0 0x7fa3280b05c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:50.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.573+0000 7fa330600640 1 -- 172.21.15.96:0/3959529343 >> 172.21.15.96:0/3959529343 conn(0x7fa32806bd20 msgr2=0x7fa32815f300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:50.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.573+0000 7fa330600640 1 -- 172.21.15.96:0/3959529343 shutdown_connections 2024-05-28T06:04:50.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.573+0000 7fa330600640 1 -- 172.21.15.96:0/3959529343 wait complete. 2024-05-28T06:04:50.682 INFO:tasks.cephfs_test_runner:Starting test: test_progress_bar_for_1_clone (tasks.cephfs.test_volumes.TestCloneProgressReporter) Test that when one clone is in progress, one progress bar is printed 2024-05-28T06:04:50.682 INFO:tasks.cephfs_test_runner:test_progress_bar_for_1_clone (tasks.cephfs.test_volumes.TestCloneProgressReporter) 2024-05-28T06:04:50.682 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:04:50.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.887+0000 7f2c74a00640 1 Processor -- start 2024-05-28T06:04:50.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.887+0000 7f2c74a00640 1 -- start start 2024-05-28T06:04:50.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.889+0000 7f2c74a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f2c700d16a0 0x7f2c700cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:50.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.889+0000 7f2c74a00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f2c700cf870 0x7f2c700d25e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:50.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.889+0000 7f2c74a00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c700cfce0 con 0x7f2c7015e870 2024-05-28T06:04:50.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.889+0000 7f2c74a00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f2c700d2150 con 0x7f2c700d16a0 2024-05-28T06:04:50.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.889+0000 7f2c74a00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f2c7015ec50 con 0x7f2c700cf870 2024-05-28T06:04:50.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.889+0000 7f2c6f400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f2c7015e870 0x7f2c700d0f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:58494/0 (socket says 172.21.15.96:58494) 2024-05-28T06:04:50.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.889+0000 7f2c6f400640 1 -- 172.21.15.96:0/3508413866 learned_addr learned my addr 172.21.15.96:0/3508413866 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:04:50.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.889+0000 7f2c67e00640 1 --2- 172.21.15.96:0/3508413866 >> v2:172.21.15.96:3300/0 conn(0x7f2c700d16a0 0x7f2c700cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:50.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.890+0000 7f2c6fe00640 1 --2- 172.21.15.96:0/3508413866 >> v2:172.21.15.123:3300/0 conn(0x7f2c700cf870 0x7f2c700d25e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:50.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.890+0000 7f2c67e00640 1 -- 172.21.15.96:0/3508413866 >> v2:172.21.15.123:3300/0 conn(0x7f2c700cf870 msgr2=0x7f2c700d25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:50.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.890+0000 7f2c67e00640 1 --2- 172.21.15.96:0/3508413866 >> v2:172.21.15.123:3300/0 conn(0x7f2c700cf870 0x7f2c700d25e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:50.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.890+0000 7f2c67e00640 1 -- 172.21.15.96:0/3508413866 >> v1:172.21.15.96:6789/0 conn(0x7f2c7015e870 legacy=0x7f2c700d0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:50.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.890+0000 7f2c67e00640 1 -- 172.21.15.96:0/3508413866 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c700cff80 con 0x7f2c700d16a0 2024-05-28T06:04:50.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.890+0000 7f2c6ea00640 1 -- 172.21.15.96:0/3508413866 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 827671542 0 0) 0x7f2c700cfce0 con 0x7f2c7015e870 2024-05-28T06:04:50.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.891+0000 7f2c67e00640 1 --2- 172.21.15.96:0/3508413866 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2c700d16a0 0x7f2c700cf330 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f2c70163400 tx=0x7f2c5802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25667c2fc12cdde4 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:50.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.891+0000 7f2c6ea00640 1 -- 172.21.15.96:0/3508413866 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c58037070 con 0x7f2c700d16a0 2024-05-28T06:04:50.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.891+0000 7f2c6ea00640 1 -- 172.21.15.96:0/3508413866 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c58032440 con 0x7f2c700d16a0 2024-05-28T06:04:50.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.891+0000 7f2c6ea00640 1 -- 172.21.15.96:0/3508413866 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c58032810 con 0x7f2c700d16a0 2024-05-28T06:04:50.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.892+0000 7f2c74a00640 1 -- 172.21.15.96:0/3508413866 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2c700d16a0 msgr2=0x7f2c700cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:50.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.892+0000 7f2c74a00640 1 --2- 172.21.15.96:0/3508413866 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2c700d16a0 0x7f2c700cf330 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f2c70163400 tx=0x7f2c5802a1a0 comp rx=0 tx=0).stop 2024-05-28T06:04:50.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.892+0000 7f2c74a00640 1 -- 172.21.15.96:0/3508413866 shutdown_connections 2024-05-28T06:04:50.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.892+0000 7f2c74a00640 1 --2- 172.21.15.96:0/3508413866 >> v2:172.21.15.123:3300/0 conn(0x7f2c700cf870 0x7f2c700d25e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:50.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.892+0000 7f2c74a00640 1 --2- 172.21.15.96:0/3508413866 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2c700d16a0 0x7f2c700cf330 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:50.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.892+0000 7f2c74a00640 1 -- 172.21.15.96:0/3508413866 >> 172.21.15.96:0/3508413866 conn(0x7f2c7008bfc0 msgr2=0x7f2c700d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:50.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.892+0000 7f2c74a00640 1 -- 172.21.15.96:0/3508413866 shutdown_connections 2024-05-28T06:04:50.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.892+0000 7f2c74a00640 1 -- 172.21.15.96:0/3508413866 wait complete. 2024-05-28T06:04:50.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.894+0000 7f2c74a00640 1 Processor -- start 2024-05-28T06:04:50.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.894+0000 7f2c74a00640 1 -- start start 2024-05-28T06:04:50.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.895+0000 7f2c74a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c700cf870 0x7f2c700caa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:50.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.895+0000 7f2c74a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2c700d16a0 0x7f2c70153650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:50.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.895+0000 7f2c74a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2c7015e870 0x7f2c700d2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:50.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.895+0000 7f2c74a00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f2c700d0820 con 0x7f2c700d16a0 2024-05-28T06:04:50.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.895+0000 7f2c74a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f2c700d0540 con 0x7f2c700cf870 2024-05-28T06:04:50.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.895+0000 7f2c74a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f2c700d4fb0 con 0x7f2c7015e870 2024-05-28T06:04:50.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.895+0000 7f2c67e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2c700d16a0 0x7f2c70153650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:50.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.895+0000 7f2c67e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2c700d16a0 0x7f2c70153650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:36224/0 (socket says 172.21.15.96:36224) 2024-05-28T06:04:50.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.895+0000 7f2c67e00640 1 -- 172.21.15.96:0/2957311320 learned_addr learned my addr 172.21.15.96:0/2957311320 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:50.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.896+0000 7f2c67e00640 1 -- 172.21.15.96:0/2957311320 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2c7015e870 msgr2=0x7f2c700d2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:50.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.896+0000 7f2c67e00640 1 --2- 172.21.15.96:0/2957311320 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2c7015e870 0x7f2c700d2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:50.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.896+0000 7f2c67e00640 1 -- 172.21.15.96:0/2957311320 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c700cf870 msgr2=0x7f2c700caa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:50.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.896+0000 7f2c6f400640 1 --2- 172.21.15.96:0/2957311320 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c700cf870 0x7f2c700caa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:50.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.896+0000 7f2c67e00640 1 --2- 172.21.15.96:0/2957311320 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c700cf870 0x7f2c700caa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:50.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.896+0000 7f2c67e00640 1 -- 172.21.15.96:0/2957311320 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c701e9050 con 0x7f2c700d16a0 2024-05-28T06:04:50.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.896+0000 7f2c67e00640 1 --2- 172.21.15.96:0/2957311320 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2c700d16a0 0x7f2c70153650 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f2c58000c00 tx=0x7f2c580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:50.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.897+0000 7f2c67400640 1 -- 172.21.15.96:0/2957311320 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c58037040 con 0x7f2c700d16a0 2024-05-28T06:04:50.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.897+0000 7f2c67400640 1 -- 172.21.15.96:0/2957311320 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c58042070 con 0x7f2c700d16a0 2024-05-28T06:04:50.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.897+0000 7f2c74a00640 1 -- 172.21.15.96:0/2957311320 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c700c9f50 con 0x7f2c700d16a0 2024-05-28T06:04:50.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.897+0000 7f2c67400640 1 -- 172.21.15.96:0/2957311320 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c58030bd0 con 0x7f2c700d16a0 2024-05-28T06:04:50.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.897+0000 7f2c74a00640 1 -- 172.21.15.96:0/2957311320 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c700ca510 con 0x7f2c700d16a0 2024-05-28T06:04:50.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.898+0000 7f2c74a00640 1 -- 172.21.15.96:0/2957311320 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c700689e0 con 0x7f2c700d16a0 2024-05-28T06:04:50.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.899+0000 7f2c67400640 1 -- 172.21.15.96:0/2957311320 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2c58047460 con 0x7f2c700d16a0 2024-05-28T06:04:50.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.900+0000 7f2c67400640 1 --2- 172.21.15.96:0/2957311320 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2c4c06c8f0 0x7f2c4c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:50.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.901+0000 7f2c67400640 1 -- 172.21.15.96:0/2957311320 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f2c580ac710 con 0x7f2c700d16a0 2024-05-28T06:04:50.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.901+0000 7f2c6f400640 1 --2- 172.21.15.96:0/2957311320 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2c4c06c8f0 0x7f2c4c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:50.905 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.903+0000 7f2c6f400640 1 --2- 172.21.15.96:0/2957311320 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2c4c06c8f0 0x7f2c4c06edb0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f2c60007fd0 tx=0x7f2c60009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:50.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:50.905+0000 7f2c67400640 1 -- 172.21.15.96:0/2957311320 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2c58032e20 con 0x7f2c700d16a0 2024-05-28T06:04:51.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.106+0000 7f2c74a00640 1 -- 172.21.15.96:0/2957311320 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2c700ba260 con 0x7f2c700d16a0 2024-05-28T06:04:51.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.109+0000 7f2c67400640 1 -- 172.21.15.96:0/2957311320 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16617 (secure 0 0 0) 0x7f2c5807ab30 con 0x7f2c700d16a0 2024-05-28T06:04:51.112 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:04:51.112 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":20,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:04:11.882922+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-28T06:04:08.741938+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-28T06:04:09.301470+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:04:51.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.112+0000 7f2c74a00640 1 -- 172.21.15.96:0/2957311320 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2c4c06c8f0 msgr2=0x7f2c4c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:51.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.112+0000 7f2c74a00640 1 --2- 172.21.15.96:0/2957311320 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2c4c06c8f0 0x7f2c4c06edb0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f2c60007fd0 tx=0x7f2c60009040 comp rx=0 tx=0).stop 2024-05-28T06:04:51.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.112+0000 7f2c74a00640 1 -- 172.21.15.96:0/2957311320 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2c700d16a0 msgr2=0x7f2c70153650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:51.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.112+0000 7f2c74a00640 1 --2- 172.21.15.96:0/2957311320 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2c700d16a0 0x7f2c70153650 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f2c58000c00 tx=0x7f2c580046c0 comp rx=0 tx=0).stop 2024-05-28T06:04:51.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.112+0000 7f2c74a00640 1 -- 172.21.15.96:0/2957311320 shutdown_connections 2024-05-28T06:04:51.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.112+0000 7f2c74a00640 1 --2- 172.21.15.96:0/2957311320 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2c4c06c8f0 0x7f2c4c06edb0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:51.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.112+0000 7f2c74a00640 1 --2- 172.21.15.96:0/2957311320 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2c7015e870 0x7f2c700d2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:51.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.112+0000 7f2c74a00640 1 --2- 172.21.15.96:0/2957311320 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2c700d16a0 0x7f2c70153650 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:51.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.112+0000 7f2c74a00640 1 --2- 172.21.15.96:0/2957311320 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c700cf870 0x7f2c700caa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:51.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.112+0000 7f2c74a00640 1 -- 172.21.15.96:0/2957311320 >> 172.21.15.96:0/2957311320 conn(0x7f2c7008bfc0 msgr2=0x7f2c700d25e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:51.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.112+0000 7f2c74a00640 1 -- 172.21.15.96:0/2957311320 shutdown_connections 2024-05-28T06:04:51.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.112+0000 7f2c74a00640 1 -- 172.21.15.96:0/2957311320 wait complete. 2024-05-28T06:04:51.135 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-28T06:03:59.517102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-28T06:04:08.741938+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-28T06:04:09.301470+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-28T06:04:51.135 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-28T06:04:51.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.341+0000 7f741d600640 1 Processor -- start 2024-05-28T06:04:51.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.341+0000 7f741d600640 1 -- start start 2024-05-28T06:04:51.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.343+0000 7f741d600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f7418161a90 0x7f741806e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:51.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.343+0000 7f741d600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7418161e70 0x7f74180b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:51.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.343+0000 7f741d600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f741806e9b0 0x7f74180ad2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:51.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.343+0000 7f741d600640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f74180b3d40 con 0x7f7418161a90 2024-05-28T06:04:51.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.343+0000 7f741d600640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f74180b4020 con 0x7f741806e9b0 2024-05-28T06:04:51.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.343+0000 7f741d600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f74180b4300 con 0x7f7418161e70 2024-05-28T06:04:51.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.343+0000 7f7416a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f7418161a90 0x7f741806e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:51.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.343+0000 7f7416a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f7418161a90 0x7f741806e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:36238/0 (socket says 172.21.15.96:36238) 2024-05-28T06:04:51.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.343+0000 7f7416a00640 1 -- 172.21.15.96:0/308263968 learned_addr learned my addr 172.21.15.96:0/308263968 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:51.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.344+0000 7f740e000640 1 --2- 172.21.15.96:0/308263968 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7418161e70 0x7f74180b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:51.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.344+0000 7f7416a00640 1 -- 172.21.15.96:0/308263968 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7418161e70 msgr2=0x7f74180b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:51.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.344+0000 7f7416a00640 1 --2- 172.21.15.96:0/308263968 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7418161e70 0x7f74180b32c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:51.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.344+0000 7f7416a00640 1 -- 172.21.15.96:0/308263968 >> v2:172.21.15.123:3300/0 conn(0x7f741806e9b0 msgr2=0x7f74180ad2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:51.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.344+0000 7f7416a00640 1 --2- 172.21.15.96:0/308263968 >> v2:172.21.15.123:3300/0 conn(0x7f741806e9b0 0x7f74180ad2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:51.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.344+0000 7f7416a00640 1 -- 172.21.15.96:0/308263968 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f741806dc70 con 0x7f7418161a90 2024-05-28T06:04:51.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.344+0000 7f740e000640 1 --2- 172.21.15.96:0/308263968 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7418161e70 0x7f74180b32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:04:51.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.345+0000 7f7416a00640 1 --2- 172.21.15.96:0/308263968 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7418161a90 0x7f741806e470 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f74180d1320 tx=0x7f740802a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb6f792e593f341b server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:51.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.345+0000 7f7416000640 1 -- 172.21.15.96:0/308263968 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7408036070 con 0x7f7418161a90 2024-05-28T06:04:51.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.345+0000 7f7416000640 1 -- 172.21.15.96:0/308263968 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7408031600 con 0x7f7418161a90 2024-05-28T06:04:51.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.345+0000 7f7416000640 1 -- 172.21.15.96:0/308263968 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74080319d0 con 0x7f7418161a90 2024-05-28T06:04:51.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.346+0000 7f741d600640 1 -- 172.21.15.96:0/308263968 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7418161a90 msgr2=0x7f741806e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:51.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.346+0000 7f741d600640 1 --2- 172.21.15.96:0/308263968 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7418161a90 0x7f741806e470 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f74180d1320 tx=0x7f740802a6a0 comp rx=0 tx=0).stop 2024-05-28T06:04:51.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.346+0000 7f741d600640 1 -- 172.21.15.96:0/308263968 shutdown_connections 2024-05-28T06:04:51.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.346+0000 7f741d600640 1 --2- 172.21.15.96:0/308263968 >> v2:172.21.15.123:3300/0 conn(0x7f741806e9b0 0x7f74180ad2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:51.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.346+0000 7f741d600640 1 --2- 172.21.15.96:0/308263968 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7418161e70 0x7f74180b32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:51.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.346+0000 7f741d600640 1 --2- 172.21.15.96:0/308263968 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7418161a90 0x7f741806e470 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:51.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.346+0000 7f741d600640 1 -- 172.21.15.96:0/308263968 >> 172.21.15.96:0/308263968 conn(0x7f741806bd20 msgr2=0x7f741806c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:51.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.346+0000 7f741d600640 1 -- 172.21.15.96:0/308263968 shutdown_connections 2024-05-28T06:04:51.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.346+0000 7f741d600640 1 -- 172.21.15.96:0/308263968 wait complete. 2024-05-28T06:04:51.349 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.348+0000 7f741d600640 1 Processor -- start 2024-05-28T06:04:51.349 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.348+0000 7f741d600640 1 -- start start 2024-05-28T06:04:51.350 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.349+0000 7f741d600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f741806e9b0 0x7f7418154bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:51.350 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.349+0000 7f741d600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7418161a90 0x7f74181f3d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:51.350 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.349+0000 7f741d600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7418161e70 0x7f74181f42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:51.350 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.349+0000 7f741d600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f74180b3800 con 0x7f741806e9b0 2024-05-28T06:04:51.351 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.349+0000 7f741d600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f74180b5020 con 0x7f7418161e70 2024-05-28T06:04:51.351 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.349+0000 7f741d600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f74180b3980 con 0x7f7418161a90 2024-05-28T06:04:51.352 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.349+0000 7f7416a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f741806e9b0 0x7f7418154bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:51.352 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.349+0000 7f7416a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f741806e9b0 0x7f7418154bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:36252/0 (socket says 172.21.15.96:36252) 2024-05-28T06:04:51.352 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.349+0000 7f7416a00640 1 -- 172.21.15.96:0/131665476 learned_addr learned my addr 172.21.15.96:0/131665476 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:51.352 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.350+0000 7f740e000640 1 --2- 172.21.15.96:0/131665476 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7418161a90 0x7f74181f3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:51.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.350+0000 7f7417400640 1 --2- 172.21.15.96:0/131665476 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7418161e70 0x7f74181f42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:51.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.350+0000 7f7416a00640 1 -- 172.21.15.96:0/131665476 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7418161a90 msgr2=0x7f74181f3d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:51.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.350+0000 7f7416a00640 1 --2- 172.21.15.96:0/131665476 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7418161a90 0x7f74181f3d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:51.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.350+0000 7f7416a00640 1 -- 172.21.15.96:0/131665476 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7418161e70 msgr2=0x7f74181f42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:51.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.350+0000 7f7416a00640 1 --2- 172.21.15.96:0/131665476 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7418161e70 0x7f74181f42c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:51.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.350+0000 7f7416a00640 1 -- 172.21.15.96:0/131665476 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74181538e0 con 0x7f741806e9b0 2024-05-28T06:04:51.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.350+0000 7f7417400640 1 --2- 172.21.15.96:0/131665476 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7418161e70 0x7f74181f42c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:04:51.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.350+0000 7f7416a00640 1 --2- 172.21.15.96:0/131665476 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f741806e9b0 0x7f7418154bf0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f7408034040 tx=0x7f7408004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:51.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.351+0000 7f740f400640 1 -- 172.21.15.96:0/131665476 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7408036050 con 0x7f741806e9b0 2024-05-28T06:04:51.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.351+0000 7f740f400640 1 -- 172.21.15.96:0/131665476 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7408042070 con 0x7f741806e9b0 2024-05-28T06:04:51.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.351+0000 7f740f400640 1 -- 172.21.15.96:0/131665476 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7408009510 con 0x7f741806e9b0 2024-05-28T06:04:51.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.351+0000 7f741d600640 1 -- 172.21.15.96:0/131665476 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7418153bc0 con 0x7f741806e9b0 2024-05-28T06:04:51.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.351+0000 7f741d600640 1 -- 172.21.15.96:0/131665476 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7418154180 con 0x7f741806e9b0 2024-05-28T06:04:51.354 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.352+0000 7f741d600640 1 -- 172.21.15.96:0/131665476 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74180689e0 con 0x7f741806e9b0 2024-05-28T06:04:51.354 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.353+0000 7f740f400640 1 -- 172.21.15.96:0/131665476 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7408007c50 con 0x7f741806e9b0 2024-05-28T06:04:51.355 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.354+0000 7f740f400640 1 --2- 172.21.15.96:0/131665476 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f73f406c920 0x7f73f406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:51.355 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.354+0000 7f740f400640 1 -- 172.21.15.96:0/131665476 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f74080aca80 con 0x7f741806e9b0 2024-05-28T06:04:51.355 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.354+0000 7f740e000640 1 --2- 172.21.15.96:0/131665476 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f73f406c920 0x7f73f406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:51.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.356+0000 7f740e000640 1 --2- 172.21.15.96:0/131665476 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f73f406c920 0x7f73f406ede0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f74180ad350 tx=0x7f7400008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:51.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.358+0000 7f740f400640 1 -- 172.21.15.96:0/131665476 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f740807af40 con 0x7f741806e9b0 2024-05-28T06:04:51.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.559+0000 7f741d600640 1 -- 172.21.15.96:0/131665476 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f741806cc20 con 0x7f741806e9b0 2024-05-28T06:04:51.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.560+0000 7f740f400640 1 -- 172.21.15.96:0/131665476 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) ==== 93+0+10 (secure 0 0 0) 0x7f740807a840 con 0x7f741806e9b0 2024-05-28T06:04:51.561 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 1 2024-05-28T06:04:51.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.562+0000 7f741d600640 1 -- 172.21.15.96:0/131665476 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f73f406c920 msgr2=0x7f73f406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:51.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.562+0000 7f741d600640 1 --2- 172.21.15.96:0/131665476 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f73f406c920 0x7f73f406ede0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f74180ad350 tx=0x7f7400008040 comp rx=0 tx=0).stop 2024-05-28T06:04:51.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.562+0000 7f741d600640 1 -- 172.21.15.96:0/131665476 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f741806e9b0 msgr2=0x7f7418154bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:51.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.562+0000 7f741d600640 1 --2- 172.21.15.96:0/131665476 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f741806e9b0 0x7f7418154bf0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f7408034040 tx=0x7f7408004a40 comp rx=0 tx=0).stop 2024-05-28T06:04:51.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.563+0000 7f741d600640 1 -- 172.21.15.96:0/131665476 shutdown_connections 2024-05-28T06:04:51.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.563+0000 7f741d600640 1 --2- 172.21.15.96:0/131665476 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f73f406c920 0x7f73f406ede0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:51.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.563+0000 7f741d600640 1 --2- 172.21.15.96:0/131665476 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7418161e70 0x7f74181f42c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:51.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.563+0000 7f741d600640 1 --2- 172.21.15.96:0/131665476 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7418161a90 0x7f74181f3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:51.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.563+0000 7f741d600640 1 --2- 172.21.15.96:0/131665476 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f741806e9b0 0x7f7418154bf0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:51.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.563+0000 7f741d600640 1 -- 172.21.15.96:0/131665476 >> 172.21.15.96:0/131665476 conn(0x7f741806bd20 msgr2=0x7f74180ad3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:51.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.563+0000 7f741d600640 1 -- 172.21.15.96:0/131665476 shutdown_connections 2024-05-28T06:04:51.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.563+0000 7f741d600640 1 -- 172.21.15.96:0/131665476 wait complete. 2024-05-28T06:04:51.583 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-28T06:04:51.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.778+0000 7f8241000640 1 Processor -- start 2024-05-28T06:04:51.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.780+0000 7f8241000640 1 -- start start 2024-05-28T06:04:51.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.780+0000 7f8241000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f823c0d0640 0x7f823c0d0a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:51.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.780+0000 7f8241000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f823c0725e0 0x7f823c0ae650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:51.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.780+0000 7f8241000640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f823c0ade90 con 0x7f823c15b6f0 2024-05-28T06:04:51.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.780+0000 7f8241000640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f823c071c40 con 0x7f823c0d0640 2024-05-28T06:04:51.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.780+0000 7f8241000640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f823c0d0f60 con 0x7f823c0725e0 2024-05-28T06:04:51.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.780+0000 7f8239e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f823c0d0640 0x7f823c0d0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:51.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.781+0000 7f8239e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f823c0d0640 0x7f823c0d0a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:36256/0 (socket says 172.21.15.96:36256) 2024-05-28T06:04:51.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.781+0000 7f8239e00640 1 -- 172.21.15.96:0/3865947262 learned_addr learned my addr 172.21.15.96:0/3865947262 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:51.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.781+0000 7f8239e00640 1 -- 172.21.15.96:0/3865947262 >> v2:172.21.15.123:3300/0 conn(0x7f823c0725e0 msgr2=0x7f823c0ae650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:51.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.781+0000 7f8239e00640 1 --2- 172.21.15.96:0/3865947262 >> v2:172.21.15.123:3300/0 conn(0x7f823c0725e0 0x7f823c0ae650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:51.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.781+0000 7f8239e00640 1 -- 172.21.15.96:0/3865947262 >> v1:172.21.15.123:6789/0 conn(0x7f823c15b6f0 legacy=0x7f823c06e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:51.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.781+0000 7f8239e00640 1 -- 172.21.15.96:0/3865947262 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f823c071e70 con 0x7f823c0d0640 2024-05-28T06:04:51.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.782+0000 7f8239e00640 1 --2- 172.21.15.96:0/3865947262 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f823c0d0640 0x7f823c0d0a20 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f823c0cfea0 tx=0x7f822c02a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9550a22cec4e4cb6 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:51.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.782+0000 7f8239400640 1 -- 172.21.15.96:0/3865947262 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f822c037070 con 0x7f823c0d0640 2024-05-28T06:04:51.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.782+0000 7f8239400640 1 -- 172.21.15.96:0/3865947262 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f822c032440 con 0x7f823c0d0640 2024-05-28T06:04:51.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.782+0000 7f8239400640 1 -- 172.21.15.96:0/3865947262 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f822c032810 con 0x7f823c0d0640 2024-05-28T06:04:51.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.783+0000 7f8241000640 1 -- 172.21.15.96:0/3865947262 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f823c0d0640 msgr2=0x7f823c0d0a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:51.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.783+0000 7f8241000640 1 --2- 172.21.15.96:0/3865947262 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f823c0d0640 0x7f823c0d0a20 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f823c0cfea0 tx=0x7f822c02a230 comp rx=0 tx=0).stop 2024-05-28T06:04:51.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.783+0000 7f8241000640 1 -- 172.21.15.96:0/3865947262 shutdown_connections 2024-05-28T06:04:51.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.783+0000 7f8241000640 1 --2- 172.21.15.96:0/3865947262 >> v2:172.21.15.123:3300/0 conn(0x7f823c0725e0 0x7f823c0ae650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:51.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.783+0000 7f8241000640 1 --2- 172.21.15.96:0/3865947262 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f823c0d0640 0x7f823c0d0a20 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:51.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.783+0000 7f8241000640 1 -- 172.21.15.96:0/3865947262 >> 172.21.15.96:0/3865947262 conn(0x7f823c06bd20 msgr2=0x7f823c06c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:51.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.783+0000 7f8241000640 1 -- 172.21.15.96:0/3865947262 shutdown_connections 2024-05-28T06:04:51.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.783+0000 7f8241000640 1 -- 172.21.15.96:0/3865947262 wait complete. 2024-05-28T06:04:51.786 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.785+0000 7f8241000640 1 Processor -- start 2024-05-28T06:04:51.786 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.785+0000 7f8241000640 1 -- start start 2024-05-28T06:04:51.787 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.786+0000 7f8241000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f823c0725e0 0x7f823c1f51a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:51.787 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.786+0000 7f8241000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f823c0d0640 0x7f823c0cefc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:51.787 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.786+0000 7f8241000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f823c15b6f0 0x7f823c15e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:51.787 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.786+0000 7f8241000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f823c0aeb90 con 0x7f823c0725e0 2024-05-28T06:04:51.787 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.786+0000 7f8241000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f823c0ae130 con 0x7f823c0d0640 2024-05-28T06:04:51.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.786+0000 7f8241000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f823c0aed10 con 0x7f823c15b6f0 2024-05-28T06:04:51.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.786+0000 7f8239e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f823c0725e0 0x7f823c1f51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:51.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.786+0000 7f8239e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f823c0725e0 0x7f823c1f51a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:36266/0 (socket says 172.21.15.96:36266) 2024-05-28T06:04:51.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.786+0000 7f8239e00640 1 -- 172.21.15.96:0/436538934 learned_addr learned my addr 172.21.15.96:0/436538934 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:51.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.786+0000 7f8239e00640 1 -- 172.21.15.96:0/436538934 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f823c15b6f0 msgr2=0x7f823c15e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:51.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.786+0000 7f8239e00640 1 --2- 172.21.15.96:0/436538934 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f823c15b6f0 0x7f823c15e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:51.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.786+0000 7f8239e00640 1 -- 172.21.15.96:0/436538934 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f823c0d0640 msgr2=0x7f823c0cefc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:51.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.786+0000 7f8239e00640 1 --2- 172.21.15.96:0/436538934 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f823c0d0640 0x7f823c0cefc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:51.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.786+0000 7f8239e00640 1 -- 172.21.15.96:0/436538934 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f823c0cf500 con 0x7f823c0725e0 2024-05-28T06:04:51.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.787+0000 7f8239e00640 1 --2- 172.21.15.96:0/436538934 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f823c0725e0 0x7f823c1f51a0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f822c000c00 tx=0x7f822c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:51.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.787+0000 7f8232000640 1 -- 172.21.15.96:0/436538934 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f822c041070 con 0x7f823c0725e0 2024-05-28T06:04:51.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.787+0000 7f8232000640 1 -- 172.21.15.96:0/436538934 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f822c03c870 con 0x7f823c0725e0 2024-05-28T06:04:51.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.787+0000 7f8241000640 1 -- 172.21.15.96:0/436538934 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f823c0cf7a0 con 0x7f823c0725e0 2024-05-28T06:04:51.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.787+0000 7f8232000640 1 -- 172.21.15.96:0/436538934 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f822c037040 con 0x7f823c0725e0 2024-05-28T06:04:51.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.787+0000 7f8241000640 1 -- 172.21.15.96:0/436538934 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f823c1e5180 con 0x7f823c0725e0 2024-05-28T06:04:51.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.788+0000 7f8241000640 1 -- 172.21.15.96:0/436538934 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f823c0689e0 con 0x7f823c0725e0 2024-05-28T06:04:51.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.789+0000 7f8232000640 1 -- 172.21.15.96:0/436538934 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f822c044070 con 0x7f823c0725e0 2024-05-28T06:04:51.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.790+0000 7f8232000640 1 --2- 172.21.15.96:0/436538934 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f821806c8f0 0x7f821806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:51.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.791+0000 7f8232000640 1 -- 172.21.15.96:0/436538934 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5825+0+0 (secure 0 0 0) 0x7f822c0ac7f0 con 0x7f823c0725e0 2024-05-28T06:04:51.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.791+0000 7f8233e00640 1 --2- 172.21.15.96:0/436538934 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f821806c8f0 0x7f821806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:51.793 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.792+0000 7f8233e00640 1 --2- 172.21.15.96:0/436538934 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f821806c8f0 0x7f821806edb0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f823c0b1e30 tx=0x7f8224006ba0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:51.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.796+0000 7f8232000640 1 -- 172.21.15.96:0/436538934 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f822c07ab90 con 0x7f823c0725e0 2024-05-28T06:04:51.995 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.993+0000 7f8241000640 1 -- 172.21.15.96:0/436538934 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f823c0b5010 con 0x7f823c0725e0 2024-05-28T06:04:51.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.994+0000 7f8232000640 1 -- 172.21.15.96:0/436538934 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7f822c07a490 con 0x7f823c0725e0 2024-05-28T06:04:51.997 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 32 2024-05-28T06:04:51.998 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.997+0000 7f8241000640 1 -- 172.21.15.96:0/436538934 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f821806c8f0 msgr2=0x7f821806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:51.998 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.997+0000 7f8241000640 1 --2- 172.21.15.96:0/436538934 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f821806c8f0 0x7f821806edb0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f823c0b1e30 tx=0x7f8224006ba0 comp rx=0 tx=0).stop 2024-05-28T06:04:51.998 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.997+0000 7f8241000640 1 -- 172.21.15.96:0/436538934 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f823c0725e0 msgr2=0x7f823c1f51a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:51.998 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.997+0000 7f8241000640 1 --2- 172.21.15.96:0/436538934 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f823c0725e0 0x7f823c1f51a0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f822c000c00 tx=0x7f822c0046c0 comp rx=0 tx=0).stop 2024-05-28T06:04:51.998 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.997+0000 7f8241000640 1 -- 172.21.15.96:0/436538934 shutdown_connections 2024-05-28T06:04:51.999 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.997+0000 7f8241000640 1 --2- 172.21.15.96:0/436538934 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f821806c8f0 0x7f821806edb0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:51.999 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.997+0000 7f8241000640 1 --2- 172.21.15.96:0/436538934 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f823c15b6f0 0x7f823c15e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:51.999 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.997+0000 7f8241000640 1 --2- 172.21.15.96:0/436538934 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f823c0d0640 0x7f823c0cefc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:51.999 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.997+0000 7f8241000640 1 --2- 172.21.15.96:0/436538934 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f823c0725e0 0x7f823c1f51a0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:51.999 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.997+0000 7f8241000640 1 -- 172.21.15.96:0/436538934 >> 172.21.15.96:0/436538934 conn(0x7f823c06bd20 msgr2=0x7f823c0b3400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:51.999 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.997+0000 7f8241000640 1 -- 172.21.15.96:0/436538934 shutdown_connections 2024-05-28T06:04:51.999 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:51.997+0000 7f8241000640 1 -- 172.21.15.96:0/436538934 wait complete. 2024-05-28T06:04:52.016 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-28T06:04:52.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.214+0000 7f3cd9600640 1 Processor -- start 2024-05-28T06:04:52.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.214+0000 7f3cd9600640 1 -- start start 2024-05-28T06:04:52.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.215+0000 7f3cd9600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f3cd415e8e0 0x7f3cd40cb5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:52.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.215+0000 7f3cd9600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3cd40d2600 0x7f3cd40d2a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:52.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.216+0000 7f3cd9600640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3cd40d3430 con 0x7f3cd40d2fa0 2024-05-28T06:04:52.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.216+0000 7f3cd9600640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f3cd40cbd10 con 0x7f3cd415e8e0 2024-05-28T06:04:52.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.216+0000 7f3cd9600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f3cd40cbff0 con 0x7f3cd40d2600 2024-05-28T06:04:52.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.216+0000 7f3cd8c00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f3cd40d2fa0 0x7f3cd40cf2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:58502/0 (socket says 172.21.15.96:58502) 2024-05-28T06:04:52.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.216+0000 7f3cd8c00640 1 -- 172.21.15.96:0/3951872360 learned_addr learned my addr 172.21.15.96:0/3951872360 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:04:52.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.216+0000 7f3cd3e00640 1 --2- 172.21.15.96:0/3951872360 >> v2:172.21.15.123:3300/0 conn(0x7f3cd415e8e0 0x7f3cd40cb5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:52.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.216+0000 7f3cd2a00640 1 -- 172.21.15.96:0/3951872360 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3249162039 0 0) 0x7f3ccc001490 con 0x7f3cd40d2fa0 2024-05-28T06:04:52.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.216+0000 7f3cd2a00640 1 -- 172.21.15.96:0/3951872360 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3cb8003520 con 0x7f3cd40d2fa0 2024-05-28T06:04:52.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.216+0000 7f3cd3400640 1 --2- 172.21.15.96:0/3951872360 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3cd40d2600 0x7f3cd40d2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:52.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.216+0000 7f3cd2a00640 1 -- 172.21.15.96:0/3951872360 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3916529915 0 0) 0x7f3cb8003520 con 0x7f3cd40d2fa0 2024-05-28T06:04:52.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.216+0000 7f3cd2a00640 1 -- 172.21.15.96:0/3951872360 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3cd40d2600 msgr2=0x7f3cd40d2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:52.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.216+0000 7f3cd2a00640 1 --2- 172.21.15.96:0/3951872360 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3cd40d2600 0x7f3cd40d2a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:52.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.216+0000 7f3cd2a00640 1 -- 172.21.15.96:0/3951872360 >> v2:172.21.15.123:3300/0 conn(0x7f3cd415e8e0 msgr2=0x7f3cd40cb5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:52.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.216+0000 7f3cd2a00640 1 --2- 172.21.15.96:0/3951872360 >> v2:172.21.15.123:3300/0 conn(0x7f3cd415e8e0 0x7f3cd40cb5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:52.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.216+0000 7f3cd2a00640 1 -- 172.21.15.96:0/3951872360 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cd40cc2d0 con 0x7f3cd40d2fa0 2024-05-28T06:04:52.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.216+0000 7f3cd2a00640 1 -- 172.21.15.96:0/3951872360 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f3ccc001bc0 con 0x7f3cd40d2fa0 2024-05-28T06:04:52.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.216+0000 7f3cd2a00640 1 -- 172.21.15.96:0/3951872360 >> v1:172.21.15.96:6789/0 conn(0x7f3cd40d2fa0 legacy=0x7f3cd40cf2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:52.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.216+0000 7f3cd2a00640 1 --2- 172.21.15.96:0/3951872360 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3cb80039b0 0x7f3cb8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:52.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.216+0000 7f3cd2a00640 1 --2- 172.21.15.96:0/3951872360 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3cb8004800 0x7f3cb8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:52.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.216+0000 7f3cd2a00640 1 --2- 172.21.15.96:0/3951872360 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3cd40d2600 0x7f3cb80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:52.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.216+0000 7f3cd2a00640 1 -- 172.21.15.96:0/3951872360 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f3cd40cbd10 con 0x7f3cb80039b0 2024-05-28T06:04:52.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.216+0000 7f3cd2a00640 1 -- 172.21.15.96:0/3951872360 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f3cd40cbff0 con 0x7f3cb8004800 2024-05-28T06:04:52.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.216+0000 7f3cd2a00640 1 -- 172.21.15.96:0/3951872360 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f3cb800b9f0 con 0x7f3cd40d2600 2024-05-28T06:04:52.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.216+0000 7f3cd3e00640 1 --2- 172.21.15.96:0/3951872360 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3cb80039b0 0x7f3cb8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:52.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.217+0000 7f3cd3e00640 1 -- 172.21.15.96:0/3951872360 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3cd40d2600 msgr2=0x7f3cb80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:52.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.217+0000 7f3cd8c00640 1 --2- 172.21.15.96:0/3951872360 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3cd40d2600 0x7f3cb80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:52.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.217+0000 7f3cd3e00640 1 --2- 172.21.15.96:0/3951872360 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3cd40d2600 0x7f3cb80051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:52.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.217+0000 7f3cd3e00640 1 -- 172.21.15.96:0/3951872360 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3cb8004800 msgr2=0x7f3cb8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:52.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.217+0000 7f3cd3e00640 1 --2- 172.21.15.96:0/3951872360 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3cb8004800 0x7f3cb8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:52.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.217+0000 7f3cd3e00640 1 -- 172.21.15.96:0/3951872360 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cb8003520 con 0x7f3cb80039b0 2024-05-28T06:04:52.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.218+0000 7f3cd3e00640 1 --2- 172.21.15.96:0/3951872360 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3cb80039b0 0x7f3cb8003dd0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f3cc4009ac0 tx=0x7f3cb8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=593a28438d6b9022 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:52.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.218+0000 7f3cd2a00640 1 -- 172.21.15.96:0/3951872360 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cc4036070 con 0x7f3cb80039b0 2024-05-28T06:04:52.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.218+0000 7f3cd2a00640 1 -- 172.21.15.96:0/3951872360 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3cc4031400 con 0x7f3cb80039b0 2024-05-28T06:04:52.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.218+0000 7f3cd2a00640 1 -- 172.21.15.96:0/3951872360 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cc4009690 con 0x7f3cb80039b0 2024-05-28T06:04:52.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.219+0000 7f3cd9600640 1 -- 172.21.15.96:0/3951872360 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3cb80039b0 msgr2=0x7f3cb8003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:52.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.219+0000 7f3cd9600640 1 --2- 172.21.15.96:0/3951872360 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3cb80039b0 0x7f3cb8003dd0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f3cc4009ac0 tx=0x7f3cb8005830 comp rx=0 tx=0).stop 2024-05-28T06:04:52.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.219+0000 7f3cd8c00640 1 -- 172.21.15.96:0/3951872360 reap_dead start 2024-05-28T06:04:52.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.219+0000 7f3cd9600640 1 -- 172.21.15.96:0/3951872360 shutdown_connections 2024-05-28T06:04:52.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.219+0000 7f3cd9600640 1 -- 172.21.15.96:0/3951872360 >> 172.21.15.96:0/3951872360 conn(0x7f3cd408bfc0 msgr2=0x7f3cd40d44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:52.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.219+0000 7f3cd9600640 1 -- 172.21.15.96:0/3951872360 shutdown_connections 2024-05-28T06:04:52.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.219+0000 7f3cd9600640 1 -- 172.21.15.96:0/3951872360 wait complete. 2024-05-28T06:04:52.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.220+0000 7f3cd9600640 1 Processor -- start 2024-05-28T06:04:52.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.221+0000 7f3cd9600640 1 -- start start 2024-05-28T06:04:52.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.221+0000 7f3cd9600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3cb80039b0 0x7f3cd41e7b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:52.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.221+0000 7f3cd9600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3cd4147e30 0x7f3cd41f3c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:52.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.221+0000 7f3cd9600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3cd415c890 0x7f3cd415cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:52.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.222+0000 7f3cd9600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f3cd415ecc0 con 0x7f3cd415c890 2024-05-28T06:04:52.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.222+0000 7f3cd9600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f3cd40cc890 con 0x7f3cb80039b0 2024-05-28T06:04:52.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.222+0000 7f3cd9600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f3cd415ee40 con 0x7f3cd4147e30 2024-05-28T06:04:52.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.222+0000 7f3cd8c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3cd415c890 0x7f3cd415cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:52.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.222+0000 7f3cd8c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3cd415c890 0x7f3cd415cd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:36282/0 (socket says 172.21.15.96:36282) 2024-05-28T06:04:52.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.222+0000 7f3cd8c00640 1 -- 172.21.15.96:0/2831314752 learned_addr learned my addr 172.21.15.96:0/2831314752 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:52.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.222+0000 7f3cd3e00640 1 --2- 172.21.15.96:0/2831314752 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3cb80039b0 0x7f3cd41e7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:52.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.222+0000 7f3cd3400640 1 --2- 172.21.15.96:0/2831314752 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3cd4147e30 0x7f3cd41f3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:52.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.222+0000 7f3cd8c00640 1 -- 172.21.15.96:0/2831314752 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3cd4147e30 msgr2=0x7f3cd41f3c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:52.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.222+0000 7f3cd8c00640 1 --2- 172.21.15.96:0/2831314752 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3cd4147e30 0x7f3cd41f3c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:52.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.222+0000 7f3cd8c00640 1 -- 172.21.15.96:0/2831314752 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3cb80039b0 msgr2=0x7f3cd41e7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:52.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.222+0000 7f3cd8c00640 1 --2- 172.21.15.96:0/2831314752 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3cb80039b0 0x7f3cd41e7b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:52.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.222+0000 7f3cd8c00640 1 -- 172.21.15.96:0/2831314752 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cd41485d0 con 0x7f3cd415c890 2024-05-28T06:04:52.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.222+0000 7f3cd3e00640 1 --2- 172.21.15.96:0/2831314752 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3cb80039b0 0x7f3cd41e7b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:04:52.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.223+0000 7f3cd8c00640 1 --2- 172.21.15.96:0/2831314752 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3cd415c890 0x7f3cd415cd10 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f3ccc0012e0 tx=0x7f3ccc00b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:52.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.223+0000 7f3ccbe00640 1 -- 172.21.15.96:0/2831314752 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ccc00bb80 con 0x7f3cd415c890 2024-05-28T06:04:52.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.223+0000 7f3ccbe00640 1 -- 172.21.15.96:0/2831314752 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ccc00bd20 con 0x7f3cd415c890 2024-05-28T06:04:52.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.223+0000 7f3ccbe00640 1 -- 172.21.15.96:0/2831314752 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ccc0072b0 con 0x7f3cd415c890 2024-05-28T06:04:52.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.223+0000 7f3cd9600640 1 -- 172.21.15.96:0/2831314752 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3cd41488b0 con 0x7f3cd415c890 2024-05-28T06:04:52.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.223+0000 7f3cd9600640 1 -- 172.21.15.96:0/2831314752 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3cd4148e70 con 0x7f3cd415c890 2024-05-28T06:04:52.231 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.224+0000 7f3cd9600640 1 -- 172.21.15.96:0/2831314752 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3cd40689e0 con 0x7f3cd415c890 2024-05-28T06:04:52.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.225+0000 7f3ccbe00640 1 -- 172.21.15.96:0/2831314752 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3ccc01b060 con 0x7f3cd415c890 2024-05-28T06:04:52.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.226+0000 7f3ccbe00640 1 --2- 172.21.15.96:0/2831314752 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3cb406c920 0x7f3cb406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:52.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.227+0000 7f3ccbe00640 1 -- 172.21.15.96:0/2831314752 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5825+0+0 (secure 0 0 0) 0x7f3ccc08bc80 con 0x7f3cd415c890 2024-05-28T06:04:52.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.227+0000 7f3cd3e00640 1 --2- 172.21.15.96:0/2831314752 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3cb406c920 0x7f3cb406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:52.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.228+0000 7f3cd3e00640 1 --2- 172.21.15.96:0/2831314752 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3cb406c920 0x7f3cb406ede0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f3cc4033000 tx=0x7f3cc4008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:52.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.231+0000 7f3ccbe00640 1 -- 172.21.15.96:0/2831314752 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3cd40d3430 con 0x7f3cd415c890 2024-05-28T06:04:52.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.426+0000 7f3cd9600640 1 -- 172.21.15.96:0/2831314752 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f3cd40cd9a0 con 0x7f3cd415c890 2024-05-28T06:04:52.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.427+0000 7f3ccbe00640 1 -- 172.21.15.96:0/2831314752 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7f3ccc059f90 con 0x7f3cd415c890 2024-05-28T06:04:52.429 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 64 2024-05-28T06:04:52.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.430+0000 7f3cd9600640 1 -- 172.21.15.96:0/2831314752 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3cb406c920 msgr2=0x7f3cb406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:52.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.430+0000 7f3cd9600640 1 --2- 172.21.15.96:0/2831314752 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3cb406c920 0x7f3cb406ede0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f3cc4033000 tx=0x7f3cc4008030 comp rx=0 tx=0).stop 2024-05-28T06:04:52.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.430+0000 7f3cd9600640 1 -- 172.21.15.96:0/2831314752 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3cd415c890 msgr2=0x7f3cd415cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:52.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.430+0000 7f3cd9600640 1 --2- 172.21.15.96:0/2831314752 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3cd415c890 0x7f3cd415cd10 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f3ccc0012e0 tx=0x7f3ccc00b4a0 comp rx=0 tx=0).stop 2024-05-28T06:04:52.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.430+0000 7f3cd9600640 1 -- 172.21.15.96:0/2831314752 shutdown_connections 2024-05-28T06:04:52.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.430+0000 7f3cd9600640 1 --2- 172.21.15.96:0/2831314752 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3cb406c920 0x7f3cb406ede0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:52.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.430+0000 7f3cd9600640 1 --2- 172.21.15.96:0/2831314752 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3cd415c890 0x7f3cd415cd10 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:52.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.430+0000 7f3cd9600640 1 --2- 172.21.15.96:0/2831314752 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3cd4147e30 0x7f3cd41f3c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:52.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.430+0000 7f3cd9600640 1 --2- 172.21.15.96:0/2831314752 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3cb80039b0 0x7f3cd41e7b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:52.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.430+0000 7f3cd9600640 1 -- 172.21.15.96:0/2831314752 >> 172.21.15.96:0/2831314752 conn(0x7f3cd408bfc0 msgr2=0x7f3cd40cf2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:52.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.430+0000 7f3cd9600640 1 -- 172.21.15.96:0/2831314752 shutdown_connections 2024-05-28T06:04:52.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.430+0000 7f3cd9600640 1 -- 172.21.15.96:0/2831314752 wait complete. 2024-05-28T06:04:52.450 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestCloneProgressReporter.test_progress_bar_for_1_clone' 2024-05-28T06:04:52.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.641+0000 7f74eb600640 1 Processor -- start 2024-05-28T06:04:52.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.641+0000 7f74eb600640 1 -- start start 2024-05-28T06:04:52.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.643+0000 7f74eb600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f74e40d2db0 0x7f74e40b3b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:52.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.643+0000 7f74eb600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f74e40d4d60 0x7f74e40d3190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:52.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.643+0000 7f74eb600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f74e40cf490 0x7f74e40cf920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:52.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.643+0000 7f74eb600640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f74e40d5540 con 0x7f74e40d4d60 2024-05-28T06:04:52.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.643+0000 7f74eb600640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f74e40d36d0 con 0x7f74e40cf490 2024-05-28T06:04:52.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.643+0000 7f74eb600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f74e4163500 con 0x7f74e40d2db0 2024-05-28T06:04:52.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.643+0000 7f74e3e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f74e40d4d60 0x7f74e40d3190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:52.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.643+0000 7f74e3e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f74e40d4d60 0x7f74e40d3190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:36284/0 (socket says 172.21.15.96:36284) 2024-05-28T06:04:52.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.643+0000 7f74e3e00640 1 -- 172.21.15.96:0/815993192 learned_addr learned my addr 172.21.15.96:0/815993192 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:52.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.644+0000 7f74e9600640 1 --2- 172.21.15.96:0/815993192 >> v2:172.21.15.123:3300/0 conn(0x7f74e40cf490 0x7f74e40cf920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:52.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.644+0000 7f74e3e00640 1 -- 172.21.15.96:0/815993192 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f74e40d2db0 msgr2=0x7f74e40b3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:52.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.644+0000 7f74e3e00640 1 --2- 172.21.15.96:0/815993192 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f74e40d2db0 0x7f74e40b3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:52.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.644+0000 7f74e3e00640 1 -- 172.21.15.96:0/815993192 >> v2:172.21.15.123:3300/0 conn(0x7f74e40cf490 msgr2=0x7f74e40cf920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:52.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.644+0000 7f74e3e00640 1 --2- 172.21.15.96:0/815993192 >> v2:172.21.15.123:3300/0 conn(0x7f74e40cf490 0x7f74e40cf920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:52.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.644+0000 7f74e3e00640 1 -- 172.21.15.96:0/815993192 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74e4162740 con 0x7f74e40d4d60 2024-05-28T06:04:52.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.644+0000 7f74e9600640 1 --2- 172.21.15.96:0/815993192 >> v2:172.21.15.123:3300/0 conn(0x7f74e40cf490 0x7f74e40cf920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:04:52.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.645+0000 7f74e3e00640 1 --2- 172.21.15.96:0/815993192 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f74e40d4d60 0x7f74e40d3190 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f74e41601b0 tx=0x7f74d002a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c059641f583a0a9 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:52.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.646+0000 7f74e3400640 1 -- 172.21.15.96:0/815993192 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74d0037070 con 0x7f74e40d4d60 2024-05-28T06:04:52.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.646+0000 7f74e3400640 1 -- 172.21.15.96:0/815993192 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74d0032440 con 0x7f74e40d4d60 2024-05-28T06:04:52.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.646+0000 7f74e3400640 1 -- 172.21.15.96:0/815993192 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74d0032810 con 0x7f74e40d4d60 2024-05-28T06:04:52.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.646+0000 7f74eb600640 1 -- 172.21.15.96:0/815993192 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f74e40d4d60 msgr2=0x7f74e40d3190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:52.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.646+0000 7f74eb600640 1 --2- 172.21.15.96:0/815993192 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f74e40d4d60 0x7f74e40d3190 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f74e41601b0 tx=0x7f74d002a140 comp rx=0 tx=0).stop 2024-05-28T06:04:52.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.646+0000 7f74eb600640 1 -- 172.21.15.96:0/815993192 shutdown_connections 2024-05-28T06:04:52.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.647+0000 7f74eb600640 1 --2- 172.21.15.96:0/815993192 >> v2:172.21.15.123:3300/0 conn(0x7f74e40cf490 0x7f74e40cf920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:52.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.647+0000 7f74eb600640 1 --2- 172.21.15.96:0/815993192 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f74e40d4d60 0x7f74e40d3190 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:52.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.647+0000 7f74eb600640 1 --2- 172.21.15.96:0/815993192 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f74e40d2db0 0x7f74e40b3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:52.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.647+0000 7f74eb600640 1 -- 172.21.15.96:0/815993192 >> 172.21.15.96:0/815993192 conn(0x7f74e408bfc0 msgr2=0x7f74e40d4650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:52.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.647+0000 7f74eb600640 1 -- 172.21.15.96:0/815993192 shutdown_connections 2024-05-28T06:04:52.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.647+0000 7f74eb600640 1 -- 172.21.15.96:0/815993192 wait complete. 2024-05-28T06:04:52.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.648+0000 7f74eb600640 1 Processor -- start 2024-05-28T06:04:52.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.648+0000 7f74eb600640 1 -- start start 2024-05-28T06:04:52.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.649+0000 7f74eb600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f74e40cf490 0x7f74e41549c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:52.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.649+0000 7f74eb600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f74e40d2db0 0x7f74e40cd3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:52.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.649+0000 7f74eb600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f74e40d4d60 0x7f74e40cd920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:52.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.649+0000 7f74eb600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f74e4162fe0 con 0x7f74e40d2db0 2024-05-28T06:04:52.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.649+0000 7f74eb600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f74e4162d00 con 0x7f74e40d4d60 2024-05-28T06:04:52.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.649+0000 7f74eb600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f74e40d51a0 con 0x7f74e40cf490 2024-05-28T06:04:52.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.650+0000 7f74e3e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f74e40d2db0 0x7f74e40cd3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:52.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.650+0000 7f74e3e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f74e40d2db0 0x7f74e40cd3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:36290/0 (socket says 172.21.15.96:36290) 2024-05-28T06:04:52.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.650+0000 7f74e3e00640 1 -- 172.21.15.96:0/4235024595 learned_addr learned my addr 172.21.15.96:0/4235024595 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:52.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.650+0000 7f74e8c00640 1 --2- 172.21.15.96:0/4235024595 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f74e40cf490 0x7f74e41549c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:52.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.650+0000 7f74e3e00640 1 -- 172.21.15.96:0/4235024595 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f74e40cf490 msgr2=0x7f74e41549c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:52.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.650+0000 7f74e3e00640 1 --2- 172.21.15.96:0/4235024595 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f74e40cf490 0x7f74e41549c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:52.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.650+0000 7f74e3e00640 1 -- 172.21.15.96:0/4235024595 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f74e40d4d60 msgr2=0x7f74e40cd920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:52.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.650+0000 7f74e3e00640 1 --2- 172.21.15.96:0/4235024595 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f74e40d4d60 0x7f74e40cd920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:52.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.650+0000 7f74e3e00640 1 -- 172.21.15.96:0/4235024595 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74e40cc2d0 con 0x7f74e40d2db0 2024-05-28T06:04:52.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.650+0000 7f74e3e00640 1 --2- 172.21.15.96:0/4235024595 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f74e40d2db0 0x7f74e40cd3e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f74d0000c00 tx=0x7f74d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:52.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.651+0000 7f74e0c00640 1 -- 172.21.15.96:0/4235024595 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74d0041070 con 0x7f74e40d2db0 2024-05-28T06:04:52.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.651+0000 7f74e0c00640 1 -- 172.21.15.96:0/4235024595 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74d003c870 con 0x7f74e40d2db0 2024-05-28T06:04:52.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.651+0000 7f74e0c00640 1 -- 172.21.15.96:0/4235024595 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74d0037040 con 0x7f74e40d2db0 2024-05-28T06:04:52.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.651+0000 7f74eb600640 1 -- 172.21.15.96:0/4235024595 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74e40cc5b0 con 0x7f74e40d2db0 2024-05-28T06:04:52.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.651+0000 7f74eb600640 1 -- 172.21.15.96:0/4235024595 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74e40ccb70 con 0x7f74e40d2db0 2024-05-28T06:04:52.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.652+0000 7f74eb600640 1 -- 172.21.15.96:0/4235024595 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74e4069450 con 0x7f74e40d2db0 2024-05-28T06:04:52.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.653+0000 7f74e0c00640 1 -- 172.21.15.96:0/4235024595 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f74d0044070 con 0x7f74e40d2db0 2024-05-28T06:04:52.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.654+0000 7f74e0c00640 1 --2- 172.21.15.96:0/4235024595 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f74c406c8f0 0x7f74c406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:52.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.654+0000 7f74e0c00640 1 -- 172.21.15.96:0/4235024595 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5825+0+0 (secure 0 0 0) 0x7f74d00ac720 con 0x7f74e40d2db0 2024-05-28T06:04:52.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.655+0000 7f74e8c00640 1 --2- 172.21.15.96:0/4235024595 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f74c406c8f0 0x7f74c406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:52.657 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.656+0000 7f74e8c00640 1 --2- 172.21.15.96:0/4235024595 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f74c406c8f0 0x7f74c406edb0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f74e415bd50 tx=0x7f74d8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:52.659 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.658+0000 7f74e0c00640 1 -- 172.21.15.96:0/4235024595 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f74e4069450 con 0x7f74e40d2db0 2024-05-28T06:04:52.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:52.940+0000 7f74eb600640 1 -- 172.21.15.96:0/4235024595 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgressReporter.test_progress_bar_for_1_clone"]} v 0) -- 0x7f74e40689e0 con 0x7f74e40d2db0 2024-05-28T06:04:53.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.642+0000 7f74e0c00640 1 -- 172.21.15.96:0/4235024595 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgressReporter.test_progress_bar_for_1_clone"]}]=0 v63) ==== 163+0+0 (secure 0 0 0) 0x7f74d007ab60 con 0x7f74e40d2db0 2024-05-28T06:04:53.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.646+0000 7f74eb600640 1 -- 172.21.15.96:0/4235024595 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f74c406c8f0 msgr2=0x7f74c406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:53.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.646+0000 7f74eb600640 1 --2- 172.21.15.96:0/4235024595 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f74c406c8f0 0x7f74c406edb0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f74e415bd50 tx=0x7f74d8008040 comp rx=0 tx=0).stop 2024-05-28T06:04:53.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.646+0000 7f74eb600640 1 -- 172.21.15.96:0/4235024595 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f74e40d2db0 msgr2=0x7f74e40cd3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:53.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.646+0000 7f74eb600640 1 --2- 172.21.15.96:0/4235024595 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f74e40d2db0 0x7f74e40cd3e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f74d0000c00 tx=0x7f74d00046c0 comp rx=0 tx=0).stop 2024-05-28T06:04:53.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.646+0000 7f74eb600640 1 -- 172.21.15.96:0/4235024595 shutdown_connections 2024-05-28T06:04:53.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.646+0000 7f74eb600640 1 --2- 172.21.15.96:0/4235024595 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f74c406c8f0 0x7f74c406edb0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:53.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.646+0000 7f74eb600640 1 --2- 172.21.15.96:0/4235024595 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f74e40d4d60 0x7f74e40cd920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:53.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.646+0000 7f74eb600640 1 --2- 172.21.15.96:0/4235024595 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f74e40d2db0 0x7f74e40cd3e0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:53.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.646+0000 7f74eb600640 1 --2- 172.21.15.96:0/4235024595 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f74e40cf490 0x7f74e41549c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:53.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.646+0000 7f74eb600640 1 -- 172.21.15.96:0/4235024595 >> 172.21.15.96:0/4235024595 conn(0x7f74e408bfc0 msgr2=0x7f74e415ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:53.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.647+0000 7f74eb600640 1 -- 172.21.15.96:0/4235024595 shutdown_connections 2024-05-28T06:04:53.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.647+0000 7f74eb600640 1 -- 172.21.15.96:0/4235024595 wait complete. 2024-05-28T06:04:53.668 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-05-28T06:04:53.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.870+0000 7faa51800640 1 Processor -- start 2024-05-28T06:04:53.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.871+0000 7faa51800640 1 -- start start 2024-05-28T06:04:53.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.872+0000 7faa51800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7faa4c06c910 0x7faa4c06cd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:53.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.872+0000 7faa51800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faa4c06ec80 0x7faa4c06f110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:53.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.872+0000 7faa51800640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa4c073010 con 0x7faa4c0b0320 2024-05-28T06:04:53.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.872+0000 7faa51800640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7faa4c0732b0 con 0x7faa4c06c910 2024-05-28T06:04:53.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.872+0000 7faa51800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7faa4c073590 con 0x7faa4c06ec80 2024-05-28T06:04:53.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.873+0000 7faa4aa00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7faa4c0b0320 0x7faa4c15f830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:58510/0 (socket says 172.21.15.96:58510) 2024-05-28T06:04:53.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.873+0000 7faa4aa00640 1 -- 172.21.15.96:0/3269495528 learned_addr learned my addr 172.21.15.96:0/3269495528 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:04:53.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.873+0000 7faa4b400640 1 --2- 172.21.15.96:0/3269495528 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faa4c06ec80 0x7faa4c06f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:53.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.873+0000 7faa4a000640 1 --2- 172.21.15.96:0/3269495528 >> v2:172.21.15.123:3300/0 conn(0x7faa4c06c910 0x7faa4c06cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:53.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.873+0000 7faa49600640 1 -- 172.21.15.96:0/3269495528 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 594431999 0 0) 0x7faa4c073010 con 0x7faa4c0b0320 2024-05-28T06:04:53.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.873+0000 7faa49600640 1 -- 172.21.15.96:0/3269495528 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa30003520 con 0x7faa4c0b0320 2024-05-28T06:04:53.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.874+0000 7faa49600640 1 -- 172.21.15.96:0/3269495528 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3541622449 0 0) 0x7faa30003520 con 0x7faa4c0b0320 2024-05-28T06:04:53.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.874+0000 7faa49600640 1 -- 172.21.15.96:0/3269495528 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faa4c06ec80 msgr2=0x7faa4c06f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:53.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.874+0000 7faa49600640 1 --2- 172.21.15.96:0/3269495528 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faa4c06ec80 0x7faa4c06f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:53.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.874+0000 7faa49600640 1 -- 172.21.15.96:0/3269495528 >> v2:172.21.15.123:3300/0 conn(0x7faa4c06c910 msgr2=0x7faa4c06cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:53.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.874+0000 7faa49600640 1 --2- 172.21.15.96:0/3269495528 >> v2:172.21.15.123:3300/0 conn(0x7faa4c06c910 0x7faa4c06cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:53.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.874+0000 7faa49600640 1 -- 172.21.15.96:0/3269495528 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa4c06e480 con 0x7faa4c0b0320 2024-05-28T06:04:53.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.874+0000 7faa49600640 1 -- 172.21.15.96:0/3269495528 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7faa3c001670 con 0x7faa4c0b0320 2024-05-28T06:04:53.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.874+0000 7faa49600640 1 -- 172.21.15.96:0/3269495528 >> v1:172.21.15.96:6789/0 conn(0x7faa4c0b0320 legacy=0x7faa4c15f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:53.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.874+0000 7faa49600640 1 --2- 172.21.15.96:0/3269495528 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa300039b0 0x7faa30003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:53.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.874+0000 7faa49600640 1 --2- 172.21.15.96:0/3269495528 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa30004800 0x7faa30004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:53.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.874+0000 7faa49600640 1 --2- 172.21.15.96:0/3269495528 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faa4c06ec80 0x7faa300051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:53.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.874+0000 7faa49600640 1 -- 172.21.15.96:0/3269495528 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7faa4c0732b0 con 0x7faa30004800 2024-05-28T06:04:53.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.874+0000 7faa49600640 1 -- 172.21.15.96:0/3269495528 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7faa4c073590 con 0x7faa300039b0 2024-05-28T06:04:53.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.874+0000 7faa49600640 1 -- 172.21.15.96:0/3269495528 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7faa3000ba40 con 0x7faa4c06ec80 2024-05-28T06:04:53.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.874+0000 7faa4a000640 1 --2- 172.21.15.96:0/3269495528 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa30004800 0x7faa30004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:53.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.874+0000 7faa4a000640 1 -- 172.21.15.96:0/3269495528 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faa4c06ec80 msgr2=0x7faa300051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:53.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.874+0000 7faa4a000640 1 --2- 172.21.15.96:0/3269495528 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faa4c06ec80 0x7faa300051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:53.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.874+0000 7faa4a000640 1 -- 172.21.15.96:0/3269495528 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa300039b0 msgr2=0x7faa30003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:53.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.874+0000 7faa4a000640 1 --2- 172.21.15.96:0/3269495528 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa300039b0 0x7faa30003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:53.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.874+0000 7faa4a000640 1 -- 172.21.15.96:0/3269495528 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa30003520 con 0x7faa30004800 2024-05-28T06:04:53.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.875+0000 7faa4a000640 1 --2- 172.21.15.96:0/3269495528 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa30004800 0x7faa30004c80 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7faa34009ac0 tx=0x7faa30005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4782cbae9a4097c server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:53.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.876+0000 7faa49600640 1 -- 172.21.15.96:0/3269495528 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa34036070 con 0x7faa30004800 2024-05-28T06:04:53.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.876+0000 7faa49600640 1 -- 172.21.15.96:0/3269495528 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa34031400 con 0x7faa30004800 2024-05-28T06:04:53.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.876+0000 7faa49600640 1 -- 172.21.15.96:0/3269495528 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa34009690 con 0x7faa30004800 2024-05-28T06:04:53.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.876+0000 7faa51800640 1 -- 172.21.15.96:0/3269495528 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa30004800 msgr2=0x7faa30004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:53.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.876+0000 7faa51800640 1 --2- 172.21.15.96:0/3269495528 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa30004800 0x7faa30004c80 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7faa34009ac0 tx=0x7faa30005830 comp rx=0 tx=0).stop 2024-05-28T06:04:53.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.876+0000 7faa4b400640 1 -- 172.21.15.96:0/3269495528 reap_dead start 2024-05-28T06:04:53.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.876+0000 7faa51800640 1 -- 172.21.15.96:0/3269495528 shutdown_connections 2024-05-28T06:04:53.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.876+0000 7faa51800640 1 -- 172.21.15.96:0/3269495528 >> 172.21.15.96:0/3269495528 conn(0x7faa4c0724f0 msgr2=0x7faa4c072900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:53.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.877+0000 7faa51800640 1 -- 172.21.15.96:0/3269495528 shutdown_connections 2024-05-28T06:04:53.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.877+0000 7faa51800640 1 -- 172.21.15.96:0/3269495528 wait complete. 2024-05-28T06:04:53.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.878+0000 7faa51800640 1 Processor -- start 2024-05-28T06:04:53.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.878+0000 7faa51800640 1 -- start start 2024-05-28T06:04:53.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.879+0000 7faa51800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faa30004800 0x7faa4c1449e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:53.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.879+0000 7faa51800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa4c143980 0x7faa4c1f7d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:53.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.879+0000 7faa51800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa4c1f8270 0x7faa4c160d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:53.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.879+0000 7faa51800640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7faa4c0b4930 con 0x7faa4c1f8270 2024-05-28T06:04:53.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.879+0000 7faa51800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7faa4c0b47b0 con 0x7faa4c143980 2024-05-28T06:04:53.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.879+0000 7faa51800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7faa4c06f110 con 0x7faa30004800 2024-05-28T06:04:53.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.880+0000 7faa4b400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa4c1f8270 0x7faa4c160d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:53.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.880+0000 7faa4b400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa4c1f8270 0x7faa4c160d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:36298/0 (socket says 172.21.15.96:36298) 2024-05-28T06:04:53.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.880+0000 7faa4b400640 1 -- 172.21.15.96:0/1728332862 learned_addr learned my addr 172.21.15.96:0/1728332862 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:53.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.880+0000 7faa4aa00640 1 --2- 172.21.15.96:0/1728332862 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faa30004800 0x7faa4c1449e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:53.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.880+0000 7faa4a000640 1 --2- 172.21.15.96:0/1728332862 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa4c143980 0x7faa4c1f7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:53.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.880+0000 7faa4b400640 1 -- 172.21.15.96:0/1728332862 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faa30004800 msgr2=0x7faa4c1449e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:53.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.880+0000 7faa4b400640 1 --2- 172.21.15.96:0/1728332862 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faa30004800 0x7faa4c1449e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:53.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.880+0000 7faa4b400640 1 -- 172.21.15.96:0/1728332862 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa4c143980 msgr2=0x7faa4c1f7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:53.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.880+0000 7faa4b400640 1 --2- 172.21.15.96:0/1728332862 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa4c143980 0x7faa4c1f7d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:53.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.880+0000 7faa4b400640 1 -- 172.21.15.96:0/1728332862 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa4c0d0030 con 0x7faa4c1f8270 2024-05-28T06:04:53.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.881+0000 7faa4b400640 1 --2- 172.21.15.96:0/1728332862 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa4c1f8270 0x7faa4c160d30 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7faa4c0b3c30 tx=0x7faa4400b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:53.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.881+0000 7faa42a00640 1 -- 172.21.15.96:0/1728332862 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa44018020 con 0x7faa4c1f8270 2024-05-28T06:04:53.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.881+0000 7faa42a00640 1 -- 172.21.15.96:0/1728332862 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa4400bb10 con 0x7faa4c1f8270 2024-05-28T06:04:53.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.881+0000 7faa42a00640 1 -- 172.21.15.96:0/1728332862 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa44014400 con 0x7faa4c1f8270 2024-05-28T06:04:53.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.881+0000 7faa51800640 1 -- 172.21.15.96:0/1728332862 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa4c1f8880 con 0x7faa4c1f8270 2024-05-28T06:04:53.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.881+0000 7faa51800640 1 -- 172.21.15.96:0/1728332862 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa4c1441e0 con 0x7faa4c1f8270 2024-05-28T06:04:53.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.882+0000 7faa51800640 1 -- 172.21.15.96:0/1728332862 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa4c0690f0 con 0x7faa4c1f8270 2024-05-28T06:04:53.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.883+0000 7faa42a00640 1 -- 172.21.15.96:0/1728332862 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7faa44005d10 con 0x7faa4c1f8270 2024-05-28T06:04:53.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.884+0000 7faa42a00640 1 --2- 172.21.15.96:0/1728332862 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7faa2806c8f0 0x7faa2806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:53.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.884+0000 7faa42a00640 1 -- 172.21.15.96:0/1728332862 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5825+0+0 (secure 0 0 0) 0x7faa4408c160 con 0x7faa4c1f8270 2024-05-28T06:04:53.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.885+0000 7faa4aa00640 1 --2- 172.21.15.96:0/1728332862 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7faa2806c8f0 0x7faa2806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:53.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.886+0000 7faa4aa00640 1 --2- 172.21.15.96:0/1728332862 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7faa2806c8f0 0x7faa2806edb0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7faa3c0079b0 tx=0x7faa3c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:53.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:53.888+0000 7faa42a00640 1 -- 172.21.15.96:0/1728332862 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7faa44022030 con 0x7faa4c1f8270 2024-05-28T06:04:54.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:54.089+0000 7faa51800640 1 -- 172.21.15.96:0/1728332862 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7faa4c067b70 con 0x7faa4c1f8270 2024-05-28T06:04:54.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:54.096+0000 7faa42a00640 1 -- 172.21.15.96:0/1728332862 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) ==== 122+0+0 (secure 0 0 0) 0x7faa4405a500 con 0x7faa4c1f8270 2024-05-28T06:04:54.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:54.098+0000 7faa51800640 1 -- 172.21.15.96:0/1728332862 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7faa2806c8f0 msgr2=0x7faa2806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:54.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:54.098+0000 7faa51800640 1 --2- 172.21.15.96:0/1728332862 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7faa2806c8f0 0x7faa2806edb0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7faa3c0079b0 tx=0x7faa3c009040 comp rx=0 tx=0).stop 2024-05-28T06:04:54.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:54.098+0000 7faa51800640 1 -- 172.21.15.96:0/1728332862 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa4c1f8270 msgr2=0x7faa4c160d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:54.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:54.098+0000 7faa51800640 1 --2- 172.21.15.96:0/1728332862 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa4c1f8270 0x7faa4c160d30 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7faa4c0b3c30 tx=0x7faa4400b480 comp rx=0 tx=0).stop 2024-05-28T06:04:54.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:54.098+0000 7faa51800640 1 -- 172.21.15.96:0/1728332862 shutdown_connections 2024-05-28T06:04:54.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:54.098+0000 7faa51800640 1 --2- 172.21.15.96:0/1728332862 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7faa2806c8f0 0x7faa2806edb0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:54.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:54.098+0000 7faa51800640 1 --2- 172.21.15.96:0/1728332862 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa4c1f8270 0x7faa4c160d30 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:54.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:54.098+0000 7faa51800640 1 --2- 172.21.15.96:0/1728332862 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa4c143980 0x7faa4c1f7d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:54.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:54.098+0000 7faa51800640 1 --2- 172.21.15.96:0/1728332862 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faa30004800 0x7faa4c1449e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:54.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:54.098+0000 7faa51800640 1 -- 172.21.15.96:0/1728332862 >> 172.21.15.96:0/1728332862 conn(0x7faa4c0724f0 msgr2=0x7faa4c06f540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:54.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:54.099+0000 7faa51800640 1 -- 172.21.15.96:0/1728332862 shutdown_connections 2024-05-28T06:04:54.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:54.099+0000 7faa51800640 1 -- 172.21.15.96:0/1728332862 wait complete. 2024-05-28T06:04:54.118 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-05-28T06:04:54.119 DEBUG:teuthology.orchestra.run.smithi096:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-05-28T06:04:54.230 DEBUG:teuthology.orchestra.run.smithi123:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-05-28T06:04:54.349 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-05-28T06:04:54.350 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:04:54.350 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:04:54.409 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:04:54.409 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:04:54.465 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:04:54.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:54.661+0000 7f79e7800640 1 Processor -- start 2024-05-28T06:04:54.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:54.661+0000 7f79e7800640 1 -- start start 2024-05-28T06:04:54.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:54.663+0000 7f79e7800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f79e006f6e0 0x7f79e00cd4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:54.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:54.663+0000 7f79e7800640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79e00b4350 con 0x7f79e00d04a0 2024-05-28T06:04:54.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:54.663+0000 7f79e7800640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79e00b32d0 con 0x7f79e0162e70 2024-05-28T06:04:54.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:54.663+0000 7f79e7800640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f79e00b3ec0 con 0x7f79e006f6e0 2024-05-28T06:04:54.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:54.663+0000 7f79e4e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f79e00d04a0 0x7f79e00d0880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:58518/0 (socket says 172.21.15.96:58518) 2024-05-28T06:04:54.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:54.663+0000 7f79e4e00640 1 -- 172.21.15.96:0/995825924 learned_addr learned my addr 172.21.15.96:0/995825924 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:04:54.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:54.663+0000 7f79dfe00640 1 --2- 172.21.15.96:0/995825924 >> v2:172.21.15.96:3300/0 conn(0x7f79e006f6e0 0x7f79e00cd4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:54.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:54.664+0000 7f79dfe00640 1 -- 172.21.15.96:0/995825924 >> v1:172.21.15.123:6789/0 conn(0x7f79e0162e70 legacy=0x7f79e00ce230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:54.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:54.664+0000 7f79dfe00640 1 -- 172.21.15.96:0/995825924 >> v1:172.21.15.96:6789/0 conn(0x7f79e00d04a0 legacy=0x7f79e00d0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:54.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:54.664+0000 7f79dfe00640 1 -- 172.21.15.96:0/995825924 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79e00b3590 con 0x7f79e006f6e0 2024-05-28T06:04:54.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:54.665+0000 7f79dfe00640 1 --2- 172.21.15.96:0/995825924 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f79e006f6e0 0x7f79e00cd4f0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f79cc0097a0 tx=0x7f79cc02a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53fbff723a90cb58 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:54.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:54.665+0000 7f79df400640 1 -- 172.21.15.96:0/995825924 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79cc037070 con 0x7f79e006f6e0 2024-05-28T06:04:54.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:54.665+0000 7f79df400640 1 -- 172.21.15.96:0/995825924 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79cc02ae50 con 0x7f79e006f6e0 2024-05-28T06:04:54.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:54.665+0000 7f79df400640 1 -- 172.21.15.96:0/995825924 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79cc032660 con 0x7f79e006f6e0 2024-05-28T06:04:54.666 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:54.666+0000 7f79e7800640 1 -- 172.21.15.96:0/995825924 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f79e006f6e0 msgr2=0x7f79e00cd4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:54.666 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:54.666+0000 7f79e7800640 1 --2- 172.21.15.96:0/995825924 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f79e006f6e0 0x7f79e00cd4f0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f79cc0097a0 tx=0x7f79cc02a140 comp rx=0 tx=0).stop 2024-05-28T06:04:54.666 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:54.666+0000 7f79e7800640 1 -- 172.21.15.96:0/995825924 shutdown_connections 2024-05-28T06:04:54.666 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:54.666+0000 7f79e7800640 1 --2- 172.21.15.96:0/995825924 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f79e006f6e0 0x7f79e00cd4f0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:54.667 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:54.666+0000 7f79e7800640 1 -- 172.21.15.96:0/995825924 >> 172.21.15.96:0/995825924 conn(0x7f79e006bae0 msgr2=0x7f79e006bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:54.667 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:54.666+0000 7f79e7800640 1 -- 172.21.15.96:0/995825924 shutdown_connections 2024-05-28T06:04:54.667 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:54.666+0000 7f79e7800640 1 -- 172.21.15.96:0/995825924 wait complete. 2024-05-28T06:04:54.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:54.668+0000 7f79e7800640 1 Processor -- start 2024-05-28T06:04:54.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:54.668+0000 7f79e7800640 1 -- start start 2024-05-28T06:04:54.669 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:54.669+0000 7f79e7800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f79e006f6e0 0x7f79e00998f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:54.669 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:54.669+0000 7f79e7800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f79e00d04a0 0x7f79e00c3d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:54.669 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:54.669+0000 7f79e7800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f79e0162e70 0x7f79e0165600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:54.669 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:54.669+0000 7f79e7800640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f79e00cda30 con 0x7f79e00d04a0 2024-05-28T06:04:54.670 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:54.669+0000 7f79e7800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f79e00b3b50 con 0x7f79e006f6e0 2024-05-28T06:04:54.670 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:54.669+0000 7f79e7800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f79e00ce940 con 0x7f79e0162e70 2024-05-28T06:04:54.670 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:54.669+0000 7f79dfe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f79e00d04a0 0x7f79e00c3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:54.670 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:54.669+0000 7f79dfe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f79e00d04a0 0x7f79e00c3d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:36326/0 (socket says 172.21.15.96:36326) 2024-05-28T06:04:54.670 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:54.669+0000 7f79dfe00640 1 -- 172.21.15.96:0/2177890868 learned_addr learned my addr 172.21.15.96:0/2177890868 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:54.671 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:54.669+0000 7f79dfe00640 1 -- 172.21.15.96:0/2177890868 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f79e0162e70 msgr2=0x7f79e0165600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:54.671 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:54.669+0000 7f79dfe00640 1 --2- 172.21.15.96:0/2177890868 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f79e0162e70 0x7f79e0165600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:54.671 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:54.670+0000 7f79e4e00640 1 --2- 172.21.15.96:0/2177890868 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f79e006f6e0 0x7f79e00998f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:54.671 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:54.670+0000 7f79dfe00640 1 -- 172.21.15.96:0/2177890868 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f79e006f6e0 msgr2=0x7f79e00998f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:54.671 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:54.670+0000 7f79dfe00640 1 --2- 172.21.15.96:0/2177890868 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f79e006f6e0 0x7f79e00998f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:54.671 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:54.670+0000 7f79dfe00640 1 -- 172.21.15.96:0/2177890868 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79e00c4670 con 0x7f79e00d04a0 2024-05-28T06:04:54.672 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:54.670+0000 7f79dfe00640 1 --2- 172.21.15.96:0/2177890868 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f79e00d04a0 0x7f79e00c3d90 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f79cc000c00 tx=0x7f79cc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:54.672 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:54.670+0000 7f79dcc00640 1 -- 172.21.15.96:0/2177890868 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79cc042070 con 0x7f79e00d04a0 2024-05-28T06:04:54.672 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:54.670+0000 7f79dcc00640 1 -- 172.21.15.96:0/2177890868 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79cc03e670 con 0x7f79e00d04a0 2024-05-28T06:04:54.672 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:54.670+0000 7f79dcc00640 1 -- 172.21.15.96:0/2177890868 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79cc037040 con 0x7f79e00d04a0 2024-05-28T06:04:54.672 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:54.670+0000 7f79e7800640 1 -- 172.21.15.96:0/2177890868 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79e00c4950 con 0x7f79e00d04a0 2024-05-28T06:04:54.672 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:54.670+0000 7f79e7800640 1 -- 172.21.15.96:0/2177890868 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79e00c4f10 con 0x7f79e00d04a0 2024-05-28T06:04:54.672 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:54.672+0000 7f79e7800640 1 -- 172.21.15.96:0/2177890868 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79e00af5a0 con 0x7f79e00d04a0 2024-05-28T06:04:54.672 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:54.672+0000 7f79dcc00640 1 -- 172.21.15.96:0/2177890868 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f79cc045070 con 0x7f79e00d04a0 2024-05-28T06:04:54.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:54.673+0000 7f79dcc00640 1 --2- 172.21.15.96:0/2177890868 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f79c006c8f0 0x7f79c006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:54.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:54.674+0000 7f79dcc00640 1 -- 172.21.15.96:0/2177890868 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5825+0+0 (secure 0 0 0) 0x7f79cc0ac630 con 0x7f79e00d04a0 2024-05-28T06:04:54.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:54.674+0000 7f79e4e00640 1 --2- 172.21.15.96:0/2177890868 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f79c006c8f0 0x7f79c006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:54.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:54.675+0000 7f79e4e00640 1 --2- 172.21.15.96:0/2177890868 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f79c006c8f0 0x7f79c006edb0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f79d0000f60 tx=0x7f79d0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:54.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:54.678+0000 7f79dcc00640 1 -- 172.21.15.96:0/2177890868 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f79e00af5a0 con 0x7f79e00d04a0 2024-05-28T06:04:54.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:54.877+0000 7f79e7800640 1 -- 172.21.15.96:0/2177890868 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f79e00689e0 con 0x7f79e00d04a0 2024-05-28T06:04:54.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:54.880+0000 7f79dcc00640 1 -- 172.21.15.96:0/2177890868 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) ==== 74+0+16865 (secure 0 0 0) 0x7f79cc07aa60 con 0x7f79e00d04a0 2024-05-28T06:04:54.881 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:04:54.882 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":22,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:04:52.624604+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-28T06:04:08.741938+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-28T06:04:09.301470+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.b","mappings":[{"from":3,"to":1}]},{"pgid":"3.d","mappings":[{"from":7,"to":2}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:04:54.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:54.883+0000 7f79e7800640 1 -- 172.21.15.96:0/2177890868 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f79c006c8f0 msgr2=0x7f79c006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:54.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:54.883+0000 7f79e7800640 1 --2- 172.21.15.96:0/2177890868 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f79c006c8f0 0x7f79c006edb0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f79d0000f60 tx=0x7f79d0009040 comp rx=0 tx=0).stop 2024-05-28T06:04:54.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:54.883+0000 7f79e7800640 1 -- 172.21.15.96:0/2177890868 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f79e00d04a0 msgr2=0x7f79e00c3d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:54.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:54.883+0000 7f79e7800640 1 --2- 172.21.15.96:0/2177890868 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f79e00d04a0 0x7f79e00c3d90 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f79cc000c00 tx=0x7f79cc0046c0 comp rx=0 tx=0).stop 2024-05-28T06:04:54.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:54.883+0000 7f79e7800640 1 -- 172.21.15.96:0/2177890868 shutdown_connections 2024-05-28T06:04:54.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:54.883+0000 7f79e7800640 1 --2- 172.21.15.96:0/2177890868 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f79c006c8f0 0x7f79c006edb0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:54.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:54.883+0000 7f79e7800640 1 --2- 172.21.15.96:0/2177890868 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f79e0162e70 0x7f79e0165600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:54.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:54.883+0000 7f79e7800640 1 --2- 172.21.15.96:0/2177890868 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f79e00d04a0 0x7f79e00c3d90 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:54.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:54.883+0000 7f79e7800640 1 --2- 172.21.15.96:0/2177890868 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f79e006f6e0 0x7f79e00998f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:54.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:54.883+0000 7f79e7800640 1 -- 172.21.15.96:0/2177890868 >> 172.21.15.96:0/2177890868 conn(0x7f79e006bae0 msgr2=0x7f79e00b1590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:54.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:54.883+0000 7f79e7800640 1 -- 172.21.15.96:0/2177890868 shutdown_connections 2024-05-28T06:04:54.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:54.883+0000 7f79e7800640 1 -- 172.21.15.96:0/2177890868 wait complete. 2024-05-28T06:04:54.903 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-28T06:03:59.517102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-28T06:04:08.741938+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-28T06:04:09.301470+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-28T06:04:54.904 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-28T06:04:55.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.097+0000 7f05a6a00640 1 Processor -- start 2024-05-28T06:04:55.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.098+0000 7f05a6a00640 1 -- start start 2024-05-28T06:04:55.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.099+0000 7f05a6a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f05a015ed80 0x7f05a00cff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:55.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.099+0000 7f05a6a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f05a00b7e50 0x7f05a00cedb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:55.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.099+0000 7f05a6a00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05a015f160 con 0x7f05a00b61e0 2024-05-28T06:04:55.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.099+0000 7f05a6a00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f05a0163540 con 0x7f05a00b7e50 2024-05-28T06:04:55.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.099+0000 7f05a6a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f05a0162930 con 0x7f05a015ed80 2024-05-28T06:04:55.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.100+0000 7f059f400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f05a00b7e50 0x7f05a00cedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:55.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.100+0000 7f059f400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f05a00b7e50 0x7f05a00cedb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:36334/0 (socket says 172.21.15.96:36334) 2024-05-28T06:04:55.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.100+0000 7f059f400640 1 -- 172.21.15.96:0/1570599279 learned_addr learned my addr 172.21.15.96:0/1570599279 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:55.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.100+0000 7f059fe00640 1 --2- 172.21.15.96:0/1570599279 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f05a015ed80 0x7f05a00cff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:55.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.100+0000 7f059f400640 1 -- 172.21.15.96:0/1570599279 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f05a015ed80 msgr2=0x7f05a00cff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:55.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.100+0000 7f059f400640 1 --2- 172.21.15.96:0/1570599279 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f05a015ed80 0x7f05a00cff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:55.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.100+0000 7f059f400640 1 -- 172.21.15.96:0/1570599279 >> v1:172.21.15.123:6789/0 conn(0x7f05a00b61e0 legacy=0x7f05a00d2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:55.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.101+0000 7f059f400640 1 -- 172.21.15.96:0/1570599279 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05a0162c10 con 0x7f05a00b7e50 2024-05-28T06:04:55.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.102+0000 7f059f400640 1 --2- 172.21.15.96:0/1570599279 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f05a00b7e50 0x7f05a00cedb0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f058c009870 tx=0x7f058c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=535798706a54fe1b server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:55.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.102+0000 7f059ea00640 1 -- 172.21.15.96:0/1570599279 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f058c037070 con 0x7f05a00b7e50 2024-05-28T06:04:55.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.102+0000 7f059ea00640 1 -- 172.21.15.96:0/1570599279 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f058c032440 con 0x7f05a00b7e50 2024-05-28T06:04:55.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.102+0000 7f059ea00640 1 -- 172.21.15.96:0/1570599279 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f058c032810 con 0x7f05a00b7e50 2024-05-28T06:04:55.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.103+0000 7f05a6a00640 1 -- 172.21.15.96:0/1570599279 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f05a00b7e50 msgr2=0x7f05a00cedb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:55.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.103+0000 7f05a6a00640 1 --2- 172.21.15.96:0/1570599279 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f05a00b7e50 0x7f05a00cedb0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f058c009870 tx=0x7f058c02a270 comp rx=0 tx=0).stop 2024-05-28T06:04:55.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.103+0000 7f05a6a00640 1 -- 172.21.15.96:0/1570599279 shutdown_connections 2024-05-28T06:04:55.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.103+0000 7f05a6a00640 1 --2- 172.21.15.96:0/1570599279 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f05a00b7e50 0x7f05a00cedb0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:55.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.103+0000 7f05a6a00640 1 --2- 172.21.15.96:0/1570599279 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f05a015ed80 0x7f05a00cff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:55.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.103+0000 7f05a6a00640 1 -- 172.21.15.96:0/1570599279 >> 172.21.15.96:0/1570599279 conn(0x7f05a00d13c0 msgr2=0x7f05a00d17d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:55.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.103+0000 7f05a6a00640 1 -- 172.21.15.96:0/1570599279 shutdown_connections 2024-05-28T06:04:55.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.103+0000 7f05a6a00640 1 -- 172.21.15.96:0/1570599279 wait complete. 2024-05-28T06:04:55.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.105+0000 7f05a6a00640 1 Processor -- start 2024-05-28T06:04:55.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.105+0000 7f05a6a00640 1 -- start start 2024-05-28T06:04:55.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.106+0000 7f05a6a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f05a00b61e0 0x7f05a01439c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:55.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.106+0000 7f05a6a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f05a00b7e50 0x7f05a015fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:55.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.106+0000 7f05a6a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f05a015ed80 0x7f05a01617f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:55.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.106+0000 7f05a6a00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f05a00b6670 con 0x7f05a00b61e0 2024-05-28T06:04:55.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.106+0000 7f05a6a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f05a01631d0 con 0x7f05a00b7e50 2024-05-28T06:04:55.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.106+0000 7f05a6a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f05a00b67f0 con 0x7f05a015ed80 2024-05-28T06:04:55.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.106+0000 7f059fe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f05a00b61e0 0x7f05a01439c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:55.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.106+0000 7f059fe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f05a00b61e0 0x7f05a01439c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:36338/0 (socket says 172.21.15.96:36338) 2024-05-28T06:04:55.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.106+0000 7f059fe00640 1 -- 172.21.15.96:0/3585454542 learned_addr learned my addr 172.21.15.96:0/3585454542 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:55.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.107+0000 7f05a4a00640 1 --2- 172.21.15.96:0/3585454542 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f05a015ed80 0x7f05a01617f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:55.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.107+0000 7f059fe00640 1 -- 172.21.15.96:0/3585454542 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f05a015ed80 msgr2=0x7f05a01617f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:55.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.107+0000 7f059fe00640 1 --2- 172.21.15.96:0/3585454542 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f05a015ed80 0x7f05a01617f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:55.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.107+0000 7f059fe00640 1 -- 172.21.15.96:0/3585454542 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f05a00b7e50 msgr2=0x7f05a015fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:55.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.107+0000 7f059fe00640 1 --2- 172.21.15.96:0/3585454542 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f05a00b7e50 0x7f05a015fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:55.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.107+0000 7f059fe00640 1 -- 172.21.15.96:0/3585454542 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05a0160780 con 0x7f05a00b61e0 2024-05-28T06:04:55.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.107+0000 7f05a4a00640 1 --2- 172.21.15.96:0/3585454542 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f05a015ed80 0x7f05a01617f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:04:55.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.107+0000 7f059fe00640 1 --2- 172.21.15.96:0/3585454542 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f05a00b61e0 0x7f05a01439c0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f059000efc0 tx=0x7f059000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:55.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.107+0000 7f0597e00640 1 -- 172.21.15.96:0/3585454542 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0590018020 con 0x7f05a00b61e0 2024-05-28T06:04:55.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.108+0000 7f0597e00640 1 -- 172.21.15.96:0/3585454542 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0590014480 con 0x7f05a00b61e0 2024-05-28T06:04:55.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.108+0000 7f0597e00640 1 -- 172.21.15.96:0/3585454542 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0590014850 con 0x7f05a00b61e0 2024-05-28T06:04:55.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.108+0000 7f05a6a00640 1 -- 172.21.15.96:0/3585454542 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05a0160a60 con 0x7f05a00b61e0 2024-05-28T06:04:55.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.108+0000 7f05a6a00640 1 -- 172.21.15.96:0/3585454542 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05a00ca0e0 con 0x7f05a00b61e0 2024-05-28T06:04:55.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.109+0000 7f05a6a00640 1 -- 172.21.15.96:0/3585454542 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05a00689e0 con 0x7f05a00b61e0 2024-05-28T06:04:55.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.110+0000 7f0597e00640 1 -- 172.21.15.96:0/3585454542 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0590005d10 con 0x7f05a00b61e0 2024-05-28T06:04:55.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.111+0000 7f0597e00640 1 --2- 172.21.15.96:0/3585454542 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f058006c8f0 0x7f058006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:55.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.111+0000 7f0597e00640 1 -- 172.21.15.96:0/3585454542 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5825+0+0 (secure 0 0 0) 0x7f059008ca90 con 0x7f05a00b61e0 2024-05-28T06:04:55.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.111+0000 7f059f400640 1 --2- 172.21.15.96:0/3585454542 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f058006c8f0 0x7f058006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:55.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.112+0000 7f059f400640 1 --2- 172.21.15.96:0/3585454542 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f058006c8f0 0x7f058006edb0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f058c034000 tx=0x7f058c02c000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:55.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.115+0000 7f0597e00640 1 -- 172.21.15.96:0/3585454542 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0590022030 con 0x7f05a00b61e0 2024-05-28T06:04:55.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.316+0000 7f05a6a00640 1 -- 172.21.15.96:0/3585454542 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f05a00d2910 con 0x7f05a00b61e0 2024-05-28T06:04:55.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.317+0000 7f0597e00640 1 -- 172.21.15.96:0/3585454542 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) ==== 93+0+10 (secure 0 0 0) 0x7f0590010040 con 0x7f05a00b61e0 2024-05-28T06:04:55.317 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 1 2024-05-28T06:04:55.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.320+0000 7f05a6a00640 1 -- 172.21.15.96:0/3585454542 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f058006c8f0 msgr2=0x7f058006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:55.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.320+0000 7f05a6a00640 1 --2- 172.21.15.96:0/3585454542 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f058006c8f0 0x7f058006edb0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f058c034000 tx=0x7f058c02c000 comp rx=0 tx=0).stop 2024-05-28T06:04:55.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.320+0000 7f05a6a00640 1 -- 172.21.15.96:0/3585454542 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f05a00b61e0 msgr2=0x7f05a01439c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:55.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.320+0000 7f05a6a00640 1 --2- 172.21.15.96:0/3585454542 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f05a00b61e0 0x7f05a01439c0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f059000efc0 tx=0x7f059000c490 comp rx=0 tx=0).stop 2024-05-28T06:04:55.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.320+0000 7f05a6a00640 1 -- 172.21.15.96:0/3585454542 shutdown_connections 2024-05-28T06:04:55.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.320+0000 7f05a6a00640 1 --2- 172.21.15.96:0/3585454542 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f058006c8f0 0x7f058006edb0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:55.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.320+0000 7f05a6a00640 1 --2- 172.21.15.96:0/3585454542 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f05a015ed80 0x7f05a01617f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:55.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.320+0000 7f05a6a00640 1 --2- 172.21.15.96:0/3585454542 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f05a00b7e50 0x7f05a015fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:55.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.320+0000 7f05a6a00640 1 --2- 172.21.15.96:0/3585454542 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f05a00b61e0 0x7f05a01439c0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:55.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.320+0000 7f05a6a00640 1 -- 172.21.15.96:0/3585454542 >> 172.21.15.96:0/3585454542 conn(0x7f05a00d13c0 msgr2=0x7f05a00b8230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:55.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.320+0000 7f05a6a00640 1 -- 172.21.15.96:0/3585454542 shutdown_connections 2024-05-28T06:04:55.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.320+0000 7f05a6a00640 1 -- 172.21.15.96:0/3585454542 wait complete. 2024-05-28T06:04:55.339 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-28T06:04:55.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.540+0000 7f738bc00640 1 Processor -- start 2024-05-28T06:04:55.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.540+0000 7f738bc00640 1 -- start start 2024-05-28T06:04:55.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.542+0000 7f738bc00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f738415abe0 0x7f738406f380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:55.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.542+0000 7f738bc00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f73840725c0 0x7f7384072a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:55.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.542+0000 7f738bc00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f738406e2e0 con 0x7f738406f8c0 2024-05-28T06:04:55.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.542+0000 7f738bc00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f7384072f70 con 0x7f738415abe0 2024-05-28T06:04:55.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.542+0000 7f738bc00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f738406e5c0 con 0x7f73840725c0 2024-05-28T06:04:55.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.542+0000 7f7389200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f738415abe0 0x7f738406f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:55.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.542+0000 7f7383e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f738406f8c0 0x7f73840b45d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:58526/0 (socket says 172.21.15.96:58526) 2024-05-28T06:04:55.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.542+0000 7f7383e00640 1 -- 172.21.15.96:0/224684883 learned_addr learned my addr 172.21.15.96:0/224684883 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:04:55.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.542+0000 7f7389200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f738415abe0 0x7f738406f380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:36342/0 (socket says 172.21.15.96:36342) 2024-05-28T06:04:55.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.542+0000 7f7389c00640 1 --2- 172.21.15.96:0/224684883 >> v2:172.21.15.123:3300/0 conn(0x7f73840725c0 0x7f7384072a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:55.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.542+0000 7f7389200640 1 -- 172.21.15.96:0/224684883 >> v2:172.21.15.123:3300/0 conn(0x7f73840725c0 msgr2=0x7f7384072a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:55.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.542+0000 7f7389200640 1 --2- 172.21.15.96:0/224684883 >> v2:172.21.15.123:3300/0 conn(0x7f73840725c0 0x7f7384072a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:55.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.542+0000 7f7389200640 1 -- 172.21.15.96:0/224684883 >> v1:172.21.15.96:6789/0 conn(0x7f738406f8c0 legacy=0x7f73840b45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:55.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.543+0000 7f7389200640 1 -- 172.21.15.96:0/224684883 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f738406e8a0 con 0x7f738415abe0 2024-05-28T06:04:55.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.544+0000 7f7389200640 1 --2- 172.21.15.96:0/224684883 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f738415abe0 0x7f738406f380 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f73840cfcf0 tx=0x7f737402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c1209da3842bba4 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:55.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.544+0000 7f7383400640 1 -- 172.21.15.96:0/224684883 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7374037070 con 0x7f738415abe0 2024-05-28T06:04:55.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.544+0000 7f7383400640 1 -- 172.21.15.96:0/224684883 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7374032440 con 0x7f738415abe0 2024-05-28T06:04:55.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.544+0000 7f7383400640 1 -- 172.21.15.96:0/224684883 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7374032810 con 0x7f738415abe0 2024-05-28T06:04:55.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.544+0000 7f738bc00640 1 -- 172.21.15.96:0/224684883 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f738415abe0 msgr2=0x7f738406f380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:55.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.545+0000 7f738bc00640 1 --2- 172.21.15.96:0/224684883 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f738415abe0 0x7f738406f380 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f73840cfcf0 tx=0x7f737402a1a0 comp rx=0 tx=0).stop 2024-05-28T06:04:55.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.545+0000 7f738bc00640 1 -- 172.21.15.96:0/224684883 shutdown_connections 2024-05-28T06:04:55.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.545+0000 7f738bc00640 1 --2- 172.21.15.96:0/224684883 >> v2:172.21.15.123:3300/0 conn(0x7f73840725c0 0x7f7384072a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:55.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.545+0000 7f738bc00640 1 --2- 172.21.15.96:0/224684883 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f738415abe0 0x7f738406f380 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:55.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.545+0000 7f738bc00640 1 -- 172.21.15.96:0/224684883 >> 172.21.15.96:0/224684883 conn(0x7f738406bd20 msgr2=0x7f738406c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:55.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.545+0000 7f738bc00640 1 -- 172.21.15.96:0/224684883 shutdown_connections 2024-05-28T06:04:55.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.545+0000 7f738bc00640 1 -- 172.21.15.96:0/224684883 wait complete. 2024-05-28T06:04:55.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.547+0000 7f738bc00640 1 Processor -- start 2024-05-28T06:04:55.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.547+0000 7f738bc00640 1 -- start start 2024-05-28T06:04:55.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.547+0000 7f738bc00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f738406f8c0 0x7f738414b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:55.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.548+0000 7f738bc00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f73840725c0 0x7f7384141d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:55.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.548+0000 7f738bc00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f738415abe0 0x7f7384142260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:55.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.548+0000 7f738bc00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f738415afc0 con 0x7f738406f8c0 2024-05-28T06:04:55.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.548+0000 7f738bc00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f738406ee60 con 0x7f73840725c0 2024-05-28T06:04:55.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.548+0000 7f738bc00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f73840b4ce0 con 0x7f738415abe0 2024-05-28T06:04:55.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.548+0000 7f7389200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f738406f8c0 0x7f738414b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:55.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.548+0000 7f7389200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f738406f8c0 0x7f738414b160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:36344/0 (socket says 172.21.15.96:36344) 2024-05-28T06:04:55.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.548+0000 7f7389200640 1 -- 172.21.15.96:0/1789055386 learned_addr learned my addr 172.21.15.96:0/1789055386 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:55.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.548+0000 7f7389200640 1 -- 172.21.15.96:0/1789055386 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f738415abe0 msgr2=0x7f7384142260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:55.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.548+0000 7f7389200640 1 --2- 172.21.15.96:0/1789055386 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f738415abe0 0x7f7384142260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:55.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.548+0000 7f7389200640 1 -- 172.21.15.96:0/1789055386 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f73840725c0 msgr2=0x7f7384141d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:55.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.548+0000 7f7389200640 1 --2- 172.21.15.96:0/1789055386 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f73840725c0 0x7f7384141d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:55.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.548+0000 7f7389200640 1 -- 172.21.15.96:0/1789055386 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7384099ed0 con 0x7f738406f8c0 2024-05-28T06:04:55.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.549+0000 7f7389200640 1 --2- 172.21.15.96:0/1789055386 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f738406f8c0 0x7f738414b160 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f7374009890 tx=0x7f73740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:55.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.549+0000 7f7380c00640 1 -- 172.21.15.96:0/1789055386 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7374037040 con 0x7f738406f8c0 2024-05-28T06:04:55.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.549+0000 7f7380c00640 1 -- 172.21.15.96:0/1789055386 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7374042070 con 0x7f738406f8c0 2024-05-28T06:04:55.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.549+0000 7f738bc00640 1 -- 172.21.15.96:0/1789055386 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7384149cc0 con 0x7f738406f8c0 2024-05-28T06:04:55.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.549+0000 7f7380c00640 1 -- 172.21.15.96:0/1789055386 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7374030a70 con 0x7f738406f8c0 2024-05-28T06:04:55.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.549+0000 7f738bc00640 1 -- 172.21.15.96:0/1789055386 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7384099bf0 con 0x7f738406f8c0 2024-05-28T06:04:55.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.550+0000 7f738bc00640 1 -- 172.21.15.96:0/1789055386 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73840682d0 con 0x7f738406f8c0 2024-05-28T06:04:55.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.551+0000 7f7380c00640 1 -- 172.21.15.96:0/1789055386 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7374030c10 con 0x7f738406f8c0 2024-05-28T06:04:55.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.552+0000 7f7380c00640 1 --2- 172.21.15.96:0/1789055386 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f736006c920 0x7f736006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:55.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.552+0000 7f7380c00640 1 -- 172.21.15.96:0/1789055386 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5825+0+0 (secure 0 0 0) 0x7f73740ac5d0 con 0x7f738406f8c0 2024-05-28T06:04:55.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.553+0000 7f7383e00640 1 --2- 172.21.15.96:0/1789055386 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f736006c920 0x7f736006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:55.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.554+0000 7f7383e00640 1 --2- 172.21.15.96:0/1789055386 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f736006c920 0x7f736006ede0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f736c0012e0 tx=0x7f736c001630 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:55.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.556+0000 7f7380c00640 1 -- 172.21.15.96:0/1789055386 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f73840682d0 con 0x7f738406f8c0 2024-05-28T06:04:55.757 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.757+0000 7f738bc00640 1 -- 172.21.15.96:0/1789055386 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f73840689e0 con 0x7f738406f8c0 2024-05-28T06:04:55.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.758+0000 7f7380c00640 1 -- 172.21.15.96:0/1789055386 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) ==== 104+0+11 (secure 0 0 0) 0x7f737407aa10 con 0x7f738406f8c0 2024-05-28T06:04:55.758 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 32 2024-05-28T06:04:55.761 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.760+0000 7f738bc00640 1 -- 172.21.15.96:0/1789055386 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f736006c920 msgr2=0x7f736006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:55.761 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.760+0000 7f738bc00640 1 --2- 172.21.15.96:0/1789055386 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f736006c920 0x7f736006ede0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f736c0012e0 tx=0x7f736c001630 comp rx=0 tx=0).stop 2024-05-28T06:04:55.761 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.760+0000 7f738bc00640 1 -- 172.21.15.96:0/1789055386 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f738406f8c0 msgr2=0x7f738414b160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:55.761 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.760+0000 7f738bc00640 1 --2- 172.21.15.96:0/1789055386 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f738406f8c0 0x7f738414b160 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f7374009890 tx=0x7f73740046c0 comp rx=0 tx=0).stop 2024-05-28T06:04:55.761 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.760+0000 7f738bc00640 1 -- 172.21.15.96:0/1789055386 shutdown_connections 2024-05-28T06:04:55.761 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.760+0000 7f738bc00640 1 --2- 172.21.15.96:0/1789055386 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f736006c920 0x7f736006ede0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:55.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.760+0000 7f738bc00640 1 --2- 172.21.15.96:0/1789055386 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f738415abe0 0x7f7384142260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:55.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.760+0000 7f738bc00640 1 --2- 172.21.15.96:0/1789055386 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f73840725c0 0x7f7384141d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:55.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.760+0000 7f738bc00640 1 --2- 172.21.15.96:0/1789055386 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f738406f8c0 0x7f738414b160 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:55.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.760+0000 7f738bc00640 1 -- 172.21.15.96:0/1789055386 >> 172.21.15.96:0/1789055386 conn(0x7f738406bd20 msgr2=0x7f73840b0880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:55.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.760+0000 7f738bc00640 1 -- 172.21.15.96:0/1789055386 shutdown_connections 2024-05-28T06:04:55.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.760+0000 7f738bc00640 1 -- 172.21.15.96:0/1789055386 wait complete. 2024-05-28T06:04:55.780 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-28T06:04:55.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.974+0000 7f60f2600640 1 Processor -- start 2024-05-28T06:04:55.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.975+0000 7f60f2600640 1 -- start start 2024-05-28T06:04:55.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.976+0000 7f60f2600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f60ec15b6f0 0x7f60ec06e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:55.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.976+0000 7f60f2600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f60ec0725e0 0x7f60ec0ae650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:55.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.976+0000 7f60f2600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60ec0ade90 con 0x7f60ec0d0640 2024-05-28T06:04:55.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.976+0000 7f60f2600640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f60ec071c40 con 0x7f60ec15b6f0 2024-05-28T06:04:55.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.976+0000 7f60f2600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f60ec071e70 con 0x7f60ec0725e0 2024-05-28T06:04:55.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.977+0000 7f60eaa00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f60ec15b6f0 0x7f60ec06e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:55.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.977+0000 7f60eaa00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f60ec15b6f0 0x7f60ec06e350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:36346/0 (socket says 172.21.15.96:36346) 2024-05-28T06:04:55.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.977+0000 7f60eaa00640 1 -- 172.21.15.96:0/2188090612 learned_addr learned my addr 172.21.15.96:0/2188090612 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:55.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.977+0000 7f60eaa00640 1 -- 172.21.15.96:0/2188090612 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f60ec0725e0 msgr2=0x7f60ec0ae650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:55.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.977+0000 7f60eaa00640 1 --2- 172.21.15.96:0/2188090612 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f60ec0725e0 0x7f60ec0ae650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:55.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.977+0000 7f60eaa00640 1 -- 172.21.15.96:0/2188090612 >> v1:172.21.15.123:6789/0 conn(0x7f60ec0d0640 legacy=0x7f60ec0d0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:55.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.977+0000 7f60eaa00640 1 -- 172.21.15.96:0/2188090612 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60ec072130 con 0x7f60ec15b6f0 2024-05-28T06:04:55.979 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.978+0000 7f60eaa00640 1 --2- 172.21.15.96:0/2188090612 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f60ec15b6f0 0x7f60ec06e350 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f60d800bb10 tx=0x7f60d802c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1a01217618a56bf server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:55.979 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.978+0000 7f60ea000640 1 -- 172.21.15.96:0/2188090612 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60d8032790 con 0x7f60ec15b6f0 2024-05-28T06:04:55.979 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.978+0000 7f60ea000640 1 -- 172.21.15.96:0/2188090612 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60d8038070 con 0x7f60ec15b6f0 2024-05-28T06:04:55.979 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.978+0000 7f60ea000640 1 -- 172.21.15.96:0/2188090612 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60d8032b50 con 0x7f60ec15b6f0 2024-05-28T06:04:55.980 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.979+0000 7f60f2600640 1 -- 172.21.15.96:0/2188090612 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f60ec15b6f0 msgr2=0x7f60ec06e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:55.980 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.979+0000 7f60f2600640 1 --2- 172.21.15.96:0/2188090612 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f60ec15b6f0 0x7f60ec06e350 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f60d800bb10 tx=0x7f60d802c8d0 comp rx=0 tx=0).stop 2024-05-28T06:04:55.980 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.979+0000 7f60f2600640 1 -- 172.21.15.96:0/2188090612 shutdown_connections 2024-05-28T06:04:55.980 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.979+0000 7f60f2600640 1 --2- 172.21.15.96:0/2188090612 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f60ec0725e0 0x7f60ec0ae650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:55.980 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.979+0000 7f60f2600640 1 --2- 172.21.15.96:0/2188090612 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f60ec15b6f0 0x7f60ec06e350 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:55.980 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.979+0000 7f60f2600640 1 -- 172.21.15.96:0/2188090612 >> 172.21.15.96:0/2188090612 conn(0x7f60ec06bd20 msgr2=0x7f60ec06c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:55.980 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.979+0000 7f60f2600640 1 -- 172.21.15.96:0/2188090612 shutdown_connections 2024-05-28T06:04:55.980 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.979+0000 7f60f2600640 1 -- 172.21.15.96:0/2188090612 wait complete. 2024-05-28T06:04:55.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.981+0000 7f60f2600640 1 Processor -- start 2024-05-28T06:04:55.982 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.982+0000 7f60f2600640 1 -- start start 2024-05-28T06:04:55.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.983+0000 7f60f2600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f60ec0725e0 0x7f60ec15e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:55.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.983+0000 7f60f2600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60ec0d0640 0x7f60ec1531f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:55.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.983+0000 7f60f2600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f60ec15b6f0 0x7f60ec0afea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:55.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.983+0000 7f60f2600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f60ec0aeb90 con 0x7f60ec0725e0 2024-05-28T06:04:55.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.983+0000 7f60f2600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f60ec06e890 con 0x7f60ec0d0640 2024-05-28T06:04:55.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.983+0000 7f60f2600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f60ec0aed10 con 0x7f60ec15b6f0 2024-05-28T06:04:55.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.983+0000 7f60eb400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f60ec0725e0 0x7f60ec15e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:55.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.983+0000 7f60eb400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f60ec0725e0 0x7f60ec15e120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:36358/0 (socket says 172.21.15.96:36358) 2024-05-28T06:04:55.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.983+0000 7f60eb400640 1 -- 172.21.15.96:0/727042414 learned_addr learned my addr 172.21.15.96:0/727042414 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:55.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.983+0000 7f60eb400640 1 -- 172.21.15.96:0/727042414 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f60ec15b6f0 msgr2=0x7f60ec0afea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:55.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.983+0000 7f60eb400640 1 --2- 172.21.15.96:0/727042414 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f60ec15b6f0 0x7f60ec0afea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:55.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.983+0000 7f60eb400640 1 -- 172.21.15.96:0/727042414 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60ec0d0640 msgr2=0x7f60ec1531f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:55.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.983+0000 7f60eb400640 1 --2- 172.21.15.96:0/727042414 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60ec0d0640 0x7f60ec1531f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:55.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.983+0000 7f60eb400640 1 -- 172.21.15.96:0/727042414 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60ec1f4830 con 0x7f60ec0725e0 2024-05-28T06:04:55.986 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.984+0000 7f60eb400640 1 --2- 172.21.15.96:0/727042414 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f60ec0725e0 0x7f60ec15e120 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f60dc00c7c0 tx=0x7f60dc00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:55.986 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.984+0000 7f60e3400640 1 -- 172.21.15.96:0/727042414 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60dc016020 con 0x7f60ec0725e0 2024-05-28T06:04:55.986 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.984+0000 7f60e3400640 1 -- 172.21.15.96:0/727042414 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60dc00a9f0 con 0x7f60ec0725e0 2024-05-28T06:04:55.986 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.984+0000 7f60f2600640 1 -- 172.21.15.96:0/727042414 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60ec1f4b10 con 0x7f60ec0725e0 2024-05-28T06:04:55.986 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.984+0000 7f60e3400640 1 -- 172.21.15.96:0/727042414 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60dc00ae20 con 0x7f60ec0725e0 2024-05-28T06:04:55.986 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.984+0000 7f60f2600640 1 -- 172.21.15.96:0/727042414 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60ec1f50d0 con 0x7f60ec0725e0 2024-05-28T06:04:55.986 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.985+0000 7f60f2600640 1 -- 172.21.15.96:0/727042414 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60ec0689e0 con 0x7f60ec0725e0 2024-05-28T06:04:55.987 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.986+0000 7f60e3400640 1 -- 172.21.15.96:0/727042414 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f60dc019020 con 0x7f60ec0725e0 2024-05-28T06:04:55.987 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.987+0000 7f60e3400640 1 --2- 172.21.15.96:0/727042414 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f60cc06c8f0 0x7f60cc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:55.987 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.987+0000 7f60e3400640 1 -- 172.21.15.96:0/727042414 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5825+0+0 (secure 0 0 0) 0x7f60dc08b990 con 0x7f60ec0725e0 2024-05-28T06:04:55.988 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.988+0000 7f60eaa00640 1 --2- 172.21.15.96:0/727042414 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f60cc06c8f0 0x7f60cc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:55.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.989+0000 7f60eaa00640 1 --2- 172.21.15.96:0/727042414 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f60cc06c8f0 0x7f60cc06edb0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f60d8037000 tx=0x7f60d8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:55.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:55.991+0000 7f60e3400640 1 -- 172.21.15.96:0/727042414 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f60dc020070 con 0x7f60ec0725e0 2024-05-28T06:04:56.193 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.193+0000 7f60f2600640 1 -- 172.21.15.96:0/727042414 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f60ec15a470 con 0x7f60ec0725e0 2024-05-28T06:04:56.194 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.194+0000 7f60e3400640 1 -- 172.21.15.96:0/727042414 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) ==== 100+0+11 (secure 0 0 0) 0x7f60dc059ca0 con 0x7f60ec0725e0 2024-05-28T06:04:56.195 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 64 2024-05-28T06:04:56.197 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.196+0000 7f60f2600640 1 -- 172.21.15.96:0/727042414 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f60cc06c8f0 msgr2=0x7f60cc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:56.197 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.196+0000 7f60f2600640 1 --2- 172.21.15.96:0/727042414 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f60cc06c8f0 0x7f60cc06edb0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f60d8037000 tx=0x7f60d8008040 comp rx=0 tx=0).stop 2024-05-28T06:04:56.197 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.196+0000 7f60f2600640 1 -- 172.21.15.96:0/727042414 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f60ec0725e0 msgr2=0x7f60ec15e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:56.197 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.196+0000 7f60f2600640 1 --2- 172.21.15.96:0/727042414 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f60ec0725e0 0x7f60ec15e120 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f60dc00c7c0 tx=0x7f60dc00cc80 comp rx=0 tx=0).stop 2024-05-28T06:04:56.197 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.197+0000 7f60f2600640 1 -- 172.21.15.96:0/727042414 shutdown_connections 2024-05-28T06:04:56.197 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.197+0000 7f60f2600640 1 --2- 172.21.15.96:0/727042414 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f60cc06c8f0 0x7f60cc06edb0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:56.197 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.197+0000 7f60f2600640 1 --2- 172.21.15.96:0/727042414 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f60ec15b6f0 0x7f60ec0afea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:56.197 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.197+0000 7f60f2600640 1 --2- 172.21.15.96:0/727042414 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60ec0d0640 0x7f60ec1531f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:56.198 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.197+0000 7f60f2600640 1 --2- 172.21.15.96:0/727042414 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f60ec0725e0 0x7f60ec15e120 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:56.198 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.197+0000 7f60f2600640 1 -- 172.21.15.96:0/727042414 >> 172.21.15.96:0/727042414 conn(0x7f60ec06bd20 msgr2=0x7f60ec06c100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:56.198 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.197+0000 7f60f2600640 1 -- 172.21.15.96:0/727042414 shutdown_connections 2024-05-28T06:04:56.198 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.197+0000 7f60f2600640 1 -- 172.21.15.96:0/727042414 wait complete. 2024-05-28T06:04:56.217 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-28T06:04:56.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.408+0000 7f8ddfc00640 1 Processor -- start 2024-05-28T06:04:56.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.408+0000 7f8ddfc00640 1 -- start start 2024-05-28T06:04:56.410 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.410+0000 7f8ddfc00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8dd80d0590 0x7f8dd80d09f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:56.410 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.410+0000 7f8ddfc00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8dd80b51f0 con 0x7f8dd8099090 2024-05-28T06:04:56.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.410+0000 7f8ddfc00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8dd80aef60 con 0x7f8dd80b4e10 2024-05-28T06:04:56.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.410+0000 7f8ddfc00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f8dd80d0f30 con 0x7f8dd80d0590 2024-05-28T06:04:56.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.410+0000 7f8dddc00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f8dd8099090 0x7f8dd80994e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:58536/0 (socket says 172.21.15.96:58536) 2024-05-28T06:04:56.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.410+0000 7f8dddc00640 1 -- 172.21.15.96:0/4248164745 learned_addr learned my addr 172.21.15.96:0/4248164745 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:04:56.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.411+0000 7f8dd7e00640 1 --2- 172.21.15.96:0/4248164745 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8dd80d0590 0x7f8dd80d09f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:56.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.411+0000 7f8dd7400640 1 -- 172.21.15.96:0/4248164745 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1991815701 0 0) 0x7f8dd80b51f0 con 0x7f8dd8099090 2024-05-28T06:04:56.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.411+0000 7f8dd7400640 1 -- 172.21.15.96:0/4248164745 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8dbc003520 con 0x7f8dd8099090 2024-05-28T06:04:56.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.411+0000 7f8dd7400640 1 -- 172.21.15.96:0/4248164745 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 567523279 0 0) 0x7f8dbc003520 con 0x7f8dd8099090 2024-05-28T06:04:56.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.411+0000 7f8dd7400640 1 -- 172.21.15.96:0/4248164745 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8dd80d0590 msgr2=0x7f8dd80d09f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:56.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.411+0000 7f8dd7400640 1 --2- 172.21.15.96:0/4248164745 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8dd80d0590 0x7f8dd80d09f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:56.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.411+0000 7f8dd7400640 1 -- 172.21.15.96:0/4248164745 >> v1:172.21.15.123:6789/0 conn(0x7f8dd80b4e10 legacy=0x7f8dd80cde30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:56.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.411+0000 7f8dd7400640 1 -- 172.21.15.96:0/4248164745 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8dd80ad1d0 con 0x7f8dd8099090 2024-05-28T06:04:56.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.411+0000 7f8dd7400640 1 -- 172.21.15.96:0/4248164745 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f8dd0001670 con 0x7f8dd8099090 2024-05-28T06:04:56.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.411+0000 7f8dd7400640 1 -- 172.21.15.96:0/4248164745 >> v1:172.21.15.96:6789/0 conn(0x7f8dd8099090 legacy=0x7f8dd80994e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:56.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.411+0000 7f8dd7400640 1 --2- 172.21.15.96:0/4248164745 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8dbc0039e0 0x7f8dbc003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:56.414 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.411+0000 7f8dd7400640 1 --2- 172.21.15.96:0/4248164745 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8dbc004800 0x7f8dbc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:56.414 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.411+0000 7f8dd7400640 1 --2- 172.21.15.96:0/4248164745 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8dd80d0590 0x7f8dbc005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:56.414 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.411+0000 7f8dd7400640 1 -- 172.21.15.96:0/4248164745 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f8dbc00bac0 con 0x7f8dbc0039e0 2024-05-28T06:04:56.414 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.411+0000 7f8dd7400640 1 -- 172.21.15.96:0/4248164745 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f8dbc00bc70 con 0x7f8dbc004800 2024-05-28T06:04:56.414 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.411+0000 7f8dd7400640 1 -- 172.21.15.96:0/4248164745 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f8dbc00be70 con 0x7f8dd80d0590 2024-05-28T06:04:56.414 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.411+0000 7f8ddd200640 1 --2- 172.21.15.96:0/4248164745 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8dbc0039e0 0x7f8dbc003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:56.414 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.412+0000 7f8dddc00640 1 --2- 172.21.15.96:0/4248164745 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8dd80d0590 0x7f8dbc005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:56.414 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.412+0000 7f8ddd200640 1 -- 172.21.15.96:0/4248164745 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8dd80d0590 msgr2=0x7f8dbc005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:56.414 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.412+0000 7f8ddd200640 1 --2- 172.21.15.96:0/4248164745 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8dd80d0590 0x7f8dbc005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:56.415 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.412+0000 7f8ddd200640 1 -- 172.21.15.96:0/4248164745 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8dbc004800 msgr2=0x7f8dbc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:56.415 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.412+0000 7f8ddd200640 1 --2- 172.21.15.96:0/4248164745 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8dbc004800 0x7f8dbc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:56.415 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.412+0000 7f8ddd200640 1 -- 172.21.15.96:0/4248164745 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8dbc003520 con 0x7f8dbc0039e0 2024-05-28T06:04:56.415 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.412+0000 7f8dddc00640 1 --2- 172.21.15.96:0/4248164745 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8dd80d0590 0x7f8dbc005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:04:56.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.413+0000 7f8ddd200640 1 --2- 172.21.15.96:0/4248164745 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8dbc0039e0 0x7f8dbc003e00 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f8dc8001e50 tx=0x7f8dbc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c8481e9acc9c18d server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:56.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.413+0000 7f8dd7400640 1 -- 172.21.15.96:0/4248164745 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dc8001a70 con 0x7f8dbc0039e0 2024-05-28T06:04:56.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.413+0000 7f8dd7400640 1 -- 172.21.15.96:0/4248164745 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8dc8003970 con 0x7f8dbc0039e0 2024-05-28T06:04:56.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.413+0000 7f8dd7400640 1 -- 172.21.15.96:0/4248164745 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dc8003d60 con 0x7f8dbc0039e0 2024-05-28T06:04:56.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.414+0000 7f8ddfc00640 1 -- 172.21.15.96:0/4248164745 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8dbc0039e0 msgr2=0x7f8dbc003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:56.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.414+0000 7f8ddfc00640 1 --2- 172.21.15.96:0/4248164745 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8dbc0039e0 0x7f8dbc003e00 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f8dc8001e50 tx=0x7f8dbc005830 comp rx=0 tx=0).stop 2024-05-28T06:04:56.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.414+0000 7f8dddc00640 1 -- 172.21.15.96:0/4248164745 reap_dead start 2024-05-28T06:04:56.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.414+0000 7f8ddfc00640 1 -- 172.21.15.96:0/4248164745 shutdown_connections 2024-05-28T06:04:56.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.414+0000 7f8ddfc00640 1 -- 172.21.15.96:0/4248164745 >> 172.21.15.96:0/4248164745 conn(0x7f8dd806bae0 msgr2=0x7f8dd806bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:56.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.414+0000 7f8ddfc00640 1 -- 172.21.15.96:0/4248164745 shutdown_connections 2024-05-28T06:04:56.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.414+0000 7f8ddfc00640 1 -- 172.21.15.96:0/4248164745 wait complete. 2024-05-28T06:04:56.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.416+0000 7f8ddfc00640 1 Processor -- start 2024-05-28T06:04:56.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.416+0000 7f8ddfc00640 1 -- start start 2024-05-28T06:04:56.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.417+0000 7f8ddfc00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8dbc0039e0 0x7f8dd81595b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:56.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.417+0000 7f8ddfc00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8dd81e7f10 0x7f8dd81e8370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:56.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.417+0000 7f8ddfc00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8dd8073140 0x7f8dd80735c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:56.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.417+0000 7f8ddfc00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f8dd80ce540 con 0x7f8dd8073140 2024-05-28T06:04:56.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.417+0000 7f8ddfc00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f8dd80ad750 con 0x7f8dd81e7f10 2024-05-28T06:04:56.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.417+0000 7f8ddfc00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f8dd80ce880 con 0x7f8dbc0039e0 2024-05-28T06:04:56.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.417+0000 7f8dddc00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8dd8073140 0x7f8dd80735c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:56.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.417+0000 7f8dddc00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8dd8073140 0x7f8dd80735c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:36376/0 (socket says 172.21.15.96:36376) 2024-05-28T06:04:56.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.417+0000 7f8dddc00640 1 -- 172.21.15.96:0/3471386292 learned_addr learned my addr 172.21.15.96:0/3471386292 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:56.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.417+0000 7f8dddc00640 1 -- 172.21.15.96:0/3471386292 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8dbc0039e0 msgr2=0x7f8dd81595b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:56.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.417+0000 7f8dddc00640 1 --2- 172.21.15.96:0/3471386292 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8dbc0039e0 0x7f8dd81595b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:56.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.417+0000 7f8dddc00640 1 -- 172.21.15.96:0/3471386292 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8dd81e7f10 msgr2=0x7f8dd81e8370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:56.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.417+0000 7f8dddc00640 1 --2- 172.21.15.96:0/3471386292 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8dd81e7f10 0x7f8dd81e8370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:56.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.417+0000 7f8dddc00640 1 -- 172.21.15.96:0/3471386292 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8dd8071cd0 con 0x7f8dd8073140 2024-05-28T06:04:56.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.418+0000 7f8dddc00640 1 --2- 172.21.15.96:0/3471386292 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8dd8073140 0x7f8dd80735c0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f8dbc004780 tx=0x7f8dd000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:56.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.418+0000 7f8dd4c00640 1 -- 172.21.15.96:0/3471386292 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dd0007230 con 0x7f8dd8073140 2024-05-28T06:04:56.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.418+0000 7f8dd4c00640 1 -- 172.21.15.96:0/3471386292 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8dd00073d0 con 0x7f8dd8073140 2024-05-28T06:04:56.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.418+0000 7f8dd4c00640 1 -- 172.21.15.96:0/3471386292 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dd00047e0 con 0x7f8dd8073140 2024-05-28T06:04:56.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.418+0000 7f8ddfc00640 1 -- 172.21.15.96:0/3471386292 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8dd8071fb0 con 0x7f8dd8073140 2024-05-28T06:04:56.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.418+0000 7f8ddfc00640 1 -- 172.21.15.96:0/3471386292 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8dd8072570 con 0x7f8dd8073140 2024-05-28T06:04:56.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.419+0000 7f8ddfc00640 1 -- 172.21.15.96:0/3471386292 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8dd80689e0 con 0x7f8dd8073140 2024-05-28T06:04:56.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.420+0000 7f8dd4c00640 1 -- 172.21.15.96:0/3471386292 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8dd0004980 con 0x7f8dd8073140 2024-05-28T06:04:56.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.421+0000 7f8dd4c00640 1 --2- 172.21.15.96:0/3471386292 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8db806c920 0x7f8db806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:56.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.421+0000 7f8dd4c00640 1 -- 172.21.15.96:0/3471386292 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5825+0+0 (secure 0 0 0) 0x7f8dd008b7a0 con 0x7f8dd8073140 2024-05-28T06:04:56.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.422+0000 7f8ddd200640 1 --2- 172.21.15.96:0/3471386292 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8db806c920 0x7f8db806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:56.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.423+0000 7f8ddd200640 1 --2- 172.21.15.96:0/3471386292 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8db806c920 0x7f8db806ede0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f8dc8032000 tx=0x7f8dc802c000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:56.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.426+0000 7f8dd4c00640 1 -- 172.21.15.96:0/3471386292 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8dd0059b70 con 0x7f8dd8073140 2024-05-28T06:04:56.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.641+0000 7f8ddfc00640 1 -- 172.21.15.96:0/3471386292 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f8dd8073b00 con 0x7f8dd8073140 2024-05-28T06:04:56.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.642+0000 7f8dd4c00640 1 -- 172.21.15.96:0/3471386292 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) ==== 80+0+0 (secure 0 0 0) 0x7f8dd0011e10 con 0x7f8dd8073140 2024-05-28T06:04:56.643 INFO:teuthology.orchestra.run.smithi096.stderr:listed 0 entries 2024-05-28T06:04:56.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.644+0000 7f8ddfc00640 1 -- 172.21.15.96:0/3471386292 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8db806c920 msgr2=0x7f8db806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:56.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.644+0000 7f8ddfc00640 1 --2- 172.21.15.96:0/3471386292 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8db806c920 0x7f8db806ede0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f8dc8032000 tx=0x7f8dc802c000 comp rx=0 tx=0).stop 2024-05-28T06:04:56.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.644+0000 7f8ddfc00640 1 -- 172.21.15.96:0/3471386292 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8dd8073140 msgr2=0x7f8dd80735c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:56.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.644+0000 7f8ddfc00640 1 --2- 172.21.15.96:0/3471386292 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8dd8073140 0x7f8dd80735c0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f8dbc004780 tx=0x7f8dd000dc00 comp rx=0 tx=0).stop 2024-05-28T06:04:56.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.645+0000 7f8ddfc00640 1 -- 172.21.15.96:0/3471386292 shutdown_connections 2024-05-28T06:04:56.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.645+0000 7f8ddfc00640 1 --2- 172.21.15.96:0/3471386292 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8db806c920 0x7f8db806ede0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:56.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.645+0000 7f8ddfc00640 1 --2- 172.21.15.96:0/3471386292 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8dd8073140 0x7f8dd80735c0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:56.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.645+0000 7f8ddfc00640 1 --2- 172.21.15.96:0/3471386292 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8dd81e7f10 0x7f8dd81e8370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:56.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.645+0000 7f8ddfc00640 1 --2- 172.21.15.96:0/3471386292 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8dbc0039e0 0x7f8dd81595b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:56.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.645+0000 7f8ddfc00640 1 -- 172.21.15.96:0/3471386292 >> 172.21.15.96:0/3471386292 conn(0x7f8dd806bae0 msgr2=0x7f8dd815a980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:56.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.645+0000 7f8ddfc00640 1 -- 172.21.15.96:0/3471386292 shutdown_connections 2024-05-28T06:04:56.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.645+0000 7f8ddfc00640 1 -- 172.21.15.96:0/3471386292 wait complete. 2024-05-28T06:04:56.664 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:04:56.664 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:04:56.709 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:04:56.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.951+0000 7f48b3200640 1 Processor -- start 2024-05-28T06:04:56.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.951+0000 7f48b3200640 1 -- start start 2024-05-28T06:04:56.953 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.952+0000 7f48b3200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f48b4156b30 0x7f48b40af6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:56.953 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.953+0000 7f48b3200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48b4156f10 con 0x7f48b40ad940 2024-05-28T06:04:56.953 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.953+0000 7f48b3200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48b406dda0 con 0x7f48b406ee40 2024-05-28T06:04:56.953 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.953+0000 7f48b3200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f48b406e080 con 0x7f48b4156b30 2024-05-28T06:04:56.953 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.953+0000 7f48b2800640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f48b40ad940 0x7f48b40b32c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:58546/0 (socket says 172.21.15.96:58546) 2024-05-28T06:04:56.953 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.953+0000 7f48b2800640 1 -- 172.21.15.96:0/1984784988 learned_addr learned my addr 172.21.15.96:0/1984784988 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:04:56.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.953+0000 7f48b1400640 1 -- 172.21.15.96:0/1984784988 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2046753323 0 0) 0x7f48b4156f10 con 0x7f48b40ad940 2024-05-28T06:04:56.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.953+0000 7f48b1400640 1 -- 172.21.15.96:0/1984784988 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4898003520 con 0x7f48b40ad940 2024-05-28T06:04:56.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.953+0000 7f48b1e00640 1 --2- 172.21.15.96:0/1984784988 >> v2:172.21.15.123:3300/0 conn(0x7f48b4156b30 0x7f48b40af6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:56.956 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.954+0000 7f48b1400640 1 -- 172.21.15.96:0/1984784988 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3483694718 0 0) 0x7f4898003520 con 0x7f48b40ad940 2024-05-28T06:04:56.956 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.954+0000 7f48b1400640 1 -- 172.21.15.96:0/1984784988 >> v2:172.21.15.123:3300/0 conn(0x7f48b4156b30 msgr2=0x7f48b40af6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:56.956 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.954+0000 7f48b1400640 1 --2- 172.21.15.96:0/1984784988 >> v2:172.21.15.123:3300/0 conn(0x7f48b4156b30 0x7f48b40af6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:56.956 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.954+0000 7f48b1400640 1 -- 172.21.15.96:0/1984784988 >> v1:172.21.15.123:6789/0 conn(0x7f48b406ee40 legacy=0x7f48b40ad230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:56.956 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.954+0000 7f48b1400640 1 -- 172.21.15.96:0/1984784988 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48b406e360 con 0x7f48b40ad940 2024-05-28T06:04:56.956 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.954+0000 7f48b1400640 1 -- 172.21.15.96:0/1984784988 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f48ac0019f0 con 0x7f48b40ad940 2024-05-28T06:04:56.956 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.954+0000 7f48b1400640 1 -- 172.21.15.96:0/1984784988 >> v1:172.21.15.96:6789/0 conn(0x7f48b40ad940 legacy=0x7f48b40b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:56.956 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.954+0000 7f48b1400640 1 --2- 172.21.15.96:0/1984784988 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f48980039b0 0x7f4898003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:56.956 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.954+0000 7f48b1400640 1 --2- 172.21.15.96:0/1984784988 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4898004800 0x7f4898004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:56.956 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.954+0000 7f48b1400640 1 --2- 172.21.15.96:0/1984784988 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f48980052e0 0x7f489800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:56.956 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.954+0000 7f48b1400640 1 -- 172.21.15.96:0/1984784988 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f48b406e080 con 0x7f48980039b0 2024-05-28T06:04:56.956 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.954+0000 7f48b1400640 1 -- 172.21.15.96:0/1984784988 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f489800c1b0 con 0x7f48980052e0 2024-05-28T06:04:56.956 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.954+0000 7f48b1400640 1 -- 172.21.15.96:0/1984784988 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f489800c3b0 con 0x7f4898004800 2024-05-28T06:04:56.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.954+0000 7f48b1e00640 1 --2- 172.21.15.96:0/1984784988 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f48980039b0 0x7f4898003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:56.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.955+0000 7f48b2800640 1 --2- 172.21.15.96:0/1984784988 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f48980052e0 0x7f489800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:56.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.955+0000 7f48abe00640 1 --2- 172.21.15.96:0/1984784988 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4898004800 0x7f4898004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:56.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.955+0000 7f48b1e00640 1 -- 172.21.15.96:0/1984784988 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4898004800 msgr2=0x7f4898004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:56.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.955+0000 7f48b1e00640 1 --2- 172.21.15.96:0/1984784988 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4898004800 0x7f4898004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:56.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.955+0000 7f48b1e00640 1 -- 172.21.15.96:0/1984784988 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f48980052e0 msgr2=0x7f489800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:56.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.955+0000 7f48b1e00640 1 --2- 172.21.15.96:0/1984784988 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f48980052e0 0x7f489800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:56.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.955+0000 7f48b1e00640 1 -- 172.21.15.96:0/1984784988 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4898003520 con 0x7f48980039b0 2024-05-28T06:04:56.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.955+0000 7f48b2800640 1 -- 172.21.15.96:0/1984784988 reap_dead start 2024-05-28T06:04:56.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.956+0000 7f48b1e00640 1 --2- 172.21.15.96:0/1984784988 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f48980039b0 0x7f4898003dd0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f48a400ba00 tx=0x7f4898005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d19f82cbb80d58d5 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:56.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.956+0000 7f48b1400640 1 -- 172.21.15.96:0/1984784988 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48a4030070 con 0x7f48980039b0 2024-05-28T06:04:56.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.956+0000 7f48b1400640 1 -- 172.21.15.96:0/1984784988 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48a402adb0 con 0x7f48980039b0 2024-05-28T06:04:56.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.956+0000 7f48b1400640 1 -- 172.21.15.96:0/1984784988 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48a400b600 con 0x7f48980039b0 2024-05-28T06:04:56.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.957+0000 7f48b3200640 1 -- 172.21.15.96:0/1984784988 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f48980039b0 msgr2=0x7f4898003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:56.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.957+0000 7f48b3200640 1 --2- 172.21.15.96:0/1984784988 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f48980039b0 0x7f4898003dd0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f48a400ba00 tx=0x7f4898005830 comp rx=0 tx=0).stop 2024-05-28T06:04:56.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.957+0000 7f48b3200640 1 -- 172.21.15.96:0/1984784988 shutdown_connections 2024-05-28T06:04:56.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.957+0000 7f48b3200640 1 --2- 172.21.15.96:0/1984784988 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f48980039b0 0x7f4898003dd0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:56.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.957+0000 7f48b3200640 1 -- 172.21.15.96:0/1984784988 >> 172.21.15.96:0/1984784988 conn(0x7f48b40b1fc0 msgr2=0x7f48b40b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:56.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.957+0000 7f48b3200640 1 -- 172.21.15.96:0/1984784988 shutdown_connections 2024-05-28T06:04:56.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.957+0000 7f48b3200640 1 -- 172.21.15.96:0/1984784988 wait complete. 2024-05-28T06:04:56.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.959+0000 7f48b3200640 1 Processor -- start 2024-05-28T06:04:56.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.959+0000 7f48b3200640 1 -- start start 2024-05-28T06:04:56.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.960+0000 7f48b3200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f48980039b0 0x7f48b40b3c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:56.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.960+0000 7f48b3200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f48b41f4210 0x7f48b40b4b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:56.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.960+0000 7f48b3200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f48b40b00f0 0x7f48b40b0570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:56.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.960+0000 7f48b3200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f48b40b2db0 con 0x7f48980039b0 2024-05-28T06:04:56.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.960+0000 7f48b3200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f48b40b2c30 con 0x7f48b40b00f0 2024-05-28T06:04:56.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.960+0000 7f48b3200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f48b406f2f0 con 0x7f48b41f4210 2024-05-28T06:04:56.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.960+0000 7f48b1e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f48980039b0 0x7f48b40b3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:56.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.960+0000 7f48b1e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f48980039b0 0x7f48b40b3c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:36400/0 (socket says 172.21.15.96:36400) 2024-05-28T06:04:56.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.960+0000 7f48b1e00640 1 -- 172.21.15.96:0/169198505 learned_addr learned my addr 172.21.15.96:0/169198505 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:56.962 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.961+0000 7f48b2800640 1 --2- 172.21.15.96:0/169198505 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f48b40b00f0 0x7f48b40b0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:56.962 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.961+0000 7f48b1e00640 1 -- 172.21.15.96:0/169198505 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f48b41f4210 msgr2=0x7f48b40b4b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:56.962 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.961+0000 7f48abe00640 1 --2- 172.21.15.96:0/169198505 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f48b41f4210 0x7f48b40b4b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:56.962 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.961+0000 7f48b1e00640 1 --2- 172.21.15.96:0/169198505 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f48b41f4210 0x7f48b40b4b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:56.962 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.961+0000 7f48b1e00640 1 -- 172.21.15.96:0/169198505 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f48b40b00f0 msgr2=0x7f48b40b0570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:56.962 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.961+0000 7f48b1e00640 1 --2- 172.21.15.96:0/169198505 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f48b40b00f0 0x7f48b40b0570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:56.962 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.961+0000 7f48b1e00640 1 -- 172.21.15.96:0/169198505 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48b4151e40 con 0x7f48980039b0 2024-05-28T06:04:56.962 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.961+0000 7f48b2800640 1 --2- 172.21.15.96:0/169198505 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f48b40b00f0 0x7f48b40b0570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:04:56.963 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.961+0000 7f48b1e00640 1 --2- 172.21.15.96:0/169198505 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f48980039b0 0x7f48b40b3c10 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f48a400bb20 tx=0x7f48a4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:56.963 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.961+0000 7f48aa000640 1 -- 172.21.15.96:0/169198505 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48a4043020 con 0x7f48980039b0 2024-05-28T06:04:56.963 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.961+0000 7f48aa000640 1 -- 172.21.15.96:0/169198505 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48a403f680 con 0x7f48980039b0 2024-05-28T06:04:56.963 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.961+0000 7f48aa000640 1 -- 172.21.15.96:0/169198505 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48a400b600 con 0x7f48980039b0 2024-05-28T06:04:56.963 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.961+0000 7f48b3200640 1 -- 172.21.15.96:0/169198505 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48b41f8130 con 0x7f48980039b0 2024-05-28T06:04:56.963 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.962+0000 7f48b3200640 1 -- 172.21.15.96:0/169198505 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48b41f86f0 con 0x7f48980039b0 2024-05-28T06:04:56.964 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.963+0000 7f48b3200640 1 -- 172.21.15.96:0/169198505 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48b40b0ab0 con 0x7f48980039b0 2024-05-28T06:04:56.964 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.964+0000 7f48aa000640 1 -- 172.21.15.96:0/169198505 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f48a4046020 con 0x7f48980039b0 2024-05-28T06:04:56.964 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.964+0000 7f48aa000640 1 --2- 172.21.15.96:0/169198505 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f489006c8f0 0x7f489006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:56.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.965+0000 7f48aa000640 1 -- 172.21.15.96:0/169198505 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5825+0+0 (secure 0 0 0) 0x7f48a40ac430 con 0x7f48980039b0 2024-05-28T06:04:56.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.965+0000 7f48abe00640 1 --2- 172.21.15.96:0/169198505 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f489006c8f0 0x7f489006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:56.966 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.966+0000 7f48abe00640 1 --2- 172.21.15.96:0/169198505 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f489006c8f0 0x7f489006edb0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f489c007fd0 tx=0x7f489c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:56.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:56.969+0000 7f48aa000640 1 -- 172.21.15.96:0/169198505 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f48a404d070 con 0x7f48980039b0 2024-05-28T06:04:57.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.170+0000 7f48b3200640 1 -- 172.21.15.96:0/169198505 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f48b40689e0 con 0x7f48980039b0 2024-05-28T06:04:57.172 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.172+0000 7f48aa000640 1 -- 172.21.15.96:0/169198505 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) ==== 74+0+16865 (secure 0 0 0) 0x7f48b4151e40 con 0x7f48980039b0 2024-05-28T06:04:57.172 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:04:57.173 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":22,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:04:52.624604+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-28T06:04:08.741938+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-28T06:04:09.301470+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.b","mappings":[{"from":3,"to":1}]},{"pgid":"3.d","mappings":[{"from":7,"to":2}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:04:57.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.174+0000 7f48b3200640 1 -- 172.21.15.96:0/169198505 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f489006c8f0 msgr2=0x7f489006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:57.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.174+0000 7f48b3200640 1 --2- 172.21.15.96:0/169198505 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f489006c8f0 0x7f489006edb0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f489c007fd0 tx=0x7f489c009040 comp rx=0 tx=0).stop 2024-05-28T06:04:57.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.174+0000 7f48b3200640 1 -- 172.21.15.96:0/169198505 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f48980039b0 msgr2=0x7f48b40b3c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:57.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.174+0000 7f48b3200640 1 --2- 172.21.15.96:0/169198505 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f48980039b0 0x7f48b40b3c10 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f48a400bb20 tx=0x7f48a4004730 comp rx=0 tx=0).stop 2024-05-28T06:04:57.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.174+0000 7f48b3200640 1 -- 172.21.15.96:0/169198505 shutdown_connections 2024-05-28T06:04:57.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.174+0000 7f48b3200640 1 --2- 172.21.15.96:0/169198505 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f489006c8f0 0x7f489006edb0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:57.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.174+0000 7f48b3200640 1 --2- 172.21.15.96:0/169198505 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f48b40b00f0 0x7f48b40b0570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:57.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.174+0000 7f48b3200640 1 --2- 172.21.15.96:0/169198505 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f48b41f4210 0x7f48b40b4b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:57.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.174+0000 7f48b3200640 1 --2- 172.21.15.96:0/169198505 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f48980039b0 0x7f48b40b3c10 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:57.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.174+0000 7f48b3200640 1 -- 172.21.15.96:0/169198505 >> 172.21.15.96:0/169198505 conn(0x7f48b40b1fc0 msgr2=0x7f48b415b2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:57.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.174+0000 7f48b3200640 1 -- 172.21.15.96:0/169198505 shutdown_connections 2024-05-28T06:04:57.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.175+0000 7f48b3200640 1 -- 172.21.15.96:0/169198505 wait complete. 2024-05-28T06:04:57.194 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-28T06:03:59.517102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-28T06:04:08.741938+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-28T06:04:09.301470+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-28T06:04:57.195 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-28T06:04:57.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.396+0000 7f6e8de00640 1 Processor -- start 2024-05-28T06:04:57.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.396+0000 7f6e8de00640 1 -- start start 2024-05-28T06:04:57.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.397+0000 7f6e8de00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f6e880d0640 0x7f6e8815d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:57.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.397+0000 7f6e8de00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f6e880d0a20 0x7f6e880cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:57.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.397+0000 7f6e8de00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e8815cb10 con 0x7f6e880b32c0 2024-05-28T06:04:57.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.397+0000 7f6e8de00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f6e880adbb0 con 0x7f6e880d0640 2024-05-28T06:04:57.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.397+0000 7f6e8de00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f6e8815db10 con 0x7f6e880d0a20 2024-05-28T06:04:57.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.398+0000 7f6e86a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f6e880d0640 0x7f6e8815d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:57.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.398+0000 7f6e87400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f6e880b32c0 0x7f6e880b3730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:58552/0 (socket says 172.21.15.96:58552) 2024-05-28T06:04:57.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.398+0000 7f6e87400640 1 -- 172.21.15.96:0/432882642 learned_addr learned my addr 172.21.15.96:0/432882642 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:04:57.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.398+0000 7f6e86000640 1 --2- 172.21.15.96:0/432882642 >> v2:172.21.15.123:3300/0 conn(0x7f6e880d0a20 0x7f6e880cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:57.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.398+0000 7f6e86a00640 1 -- 172.21.15.96:0/432882642 >> v2:172.21.15.123:3300/0 conn(0x7f6e880d0a20 msgr2=0x7f6e880cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:57.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.398+0000 7f6e86a00640 1 --2- 172.21.15.96:0/432882642 >> v2:172.21.15.123:3300/0 conn(0x7f6e880d0a20 0x7f6e880cf950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:57.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.398+0000 7f6e86a00640 1 -- 172.21.15.96:0/432882642 >> v1:172.21.15.96:6789/0 conn(0x7f6e880b32c0 legacy=0x7f6e880b3730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:57.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.398+0000 7f6e86a00640 1 -- 172.21.15.96:0/432882642 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e880ad5f0 con 0x7f6e880d0640 2024-05-28T06:04:57.400 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.398+0000 7f6e85600640 1 -- 172.21.15.96:0/432882642 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3734839905 0 0) 0x7f6e8815cb10 con 0x7f6e880b32c0 2024-05-28T06:04:57.400 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.399+0000 7f6e86a00640 1 --2- 172.21.15.96:0/432882642 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6e880d0640 0x7f6e8815d5d0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f6e88074120 tx=0x7f6e7802a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cae3f56f4c801fb0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:57.400 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.399+0000 7f6e85600640 1 -- 172.21.15.96:0/432882642 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e78037070 con 0x7f6e880d0640 2024-05-28T06:04:57.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.399+0000 7f6e85600640 1 -- 172.21.15.96:0/432882642 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e7802ae50 con 0x7f6e880d0640 2024-05-28T06:04:57.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.399+0000 7f6e85600640 1 -- 172.21.15.96:0/432882642 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e78032660 con 0x7f6e880d0640 2024-05-28T06:04:57.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.400+0000 7f6e8de00640 1 -- 172.21.15.96:0/432882642 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6e880d0640 msgr2=0x7f6e8815d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:57.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.400+0000 7f6e8de00640 1 --2- 172.21.15.96:0/432882642 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6e880d0640 0x7f6e8815d5d0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f6e88074120 tx=0x7f6e7802a140 comp rx=0 tx=0).stop 2024-05-28T06:04:57.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.400+0000 7f6e8de00640 1 -- 172.21.15.96:0/432882642 shutdown_connections 2024-05-28T06:04:57.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.400+0000 7f6e8de00640 1 --2- 172.21.15.96:0/432882642 >> v2:172.21.15.123:3300/0 conn(0x7f6e880d0a20 0x7f6e880cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:57.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.400+0000 7f6e8de00640 1 --2- 172.21.15.96:0/432882642 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6e880d0640 0x7f6e8815d5d0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:57.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.400+0000 7f6e8de00640 1 -- 172.21.15.96:0/432882642 >> 172.21.15.96:0/432882642 conn(0x7f6e8806bd20 msgr2=0x7f6e8806c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:57.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.400+0000 7f6e8de00640 1 -- 172.21.15.96:0/432882642 shutdown_connections 2024-05-28T06:04:57.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.400+0000 7f6e8de00640 1 -- 172.21.15.96:0/432882642 wait complete. 2024-05-28T06:04:57.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.402+0000 7f6e8de00640 1 Processor -- start 2024-05-28T06:04:57.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.402+0000 7f6e8de00640 1 -- start start 2024-05-28T06:04:57.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.403+0000 7f6e8de00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6e880b32c0 0x7f6e88143690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:57.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.403+0000 7f6e8de00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6e880d0640 0x7f6e88156e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:57.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.403+0000 7f6e8de00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6e880d0a20 0x7f6e88157370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:57.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.403+0000 7f6e8de00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f6e8815d0b0 con 0x7f6e880b32c0 2024-05-28T06:04:57.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.403+0000 7f6e8de00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f6e8815cdd0 con 0x7f6e880d0640 2024-05-28T06:04:57.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.403+0000 7f6e8de00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f6e880b3e40 con 0x7f6e880d0a20 2024-05-28T06:04:57.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.403+0000 7f6e86a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6e880b32c0 0x7f6e88143690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:57.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.403+0000 7f6e86a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6e880b32c0 0x7f6e88143690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:36420/0 (socket says 172.21.15.96:36420) 2024-05-28T06:04:57.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.403+0000 7f6e86a00640 1 -- 172.21.15.96:0/3585946752 learned_addr learned my addr 172.21.15.96:0/3585946752 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:57.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.404+0000 7f6e86a00640 1 -- 172.21.15.96:0/3585946752 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6e880d0a20 msgr2=0x7f6e88157370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:57.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.404+0000 7f6e86a00640 1 --2- 172.21.15.96:0/3585946752 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6e880d0a20 0x7f6e88157370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:57.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.404+0000 7f6e86a00640 1 -- 172.21.15.96:0/3585946752 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6e880d0640 msgr2=0x7f6e88156e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:57.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.404+0000 7f6e86a00640 1 --2- 172.21.15.96:0/3585946752 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6e880d0640 0x7f6e88156e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:57.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.404+0000 7f6e86a00640 1 -- 172.21.15.96:0/3585946752 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e880b22a0 con 0x7f6e880b32c0 2024-05-28T06:04:57.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.404+0000 7f6e86a00640 1 --2- 172.21.15.96:0/3585946752 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6e880b32c0 0x7f6e88143690 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f6e78035040 tx=0x7f6e780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:57.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.404+0000 7f6e7f400640 1 -- 172.21.15.96:0/3585946752 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e78041070 con 0x7f6e880b32c0 2024-05-28T06:04:57.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.404+0000 7f6e7f400640 1 -- 172.21.15.96:0/3585946752 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e7803c670 con 0x7f6e880b32c0 2024-05-28T06:04:57.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.404+0000 7f6e7f400640 1 -- 172.21.15.96:0/3585946752 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e78037040 con 0x7f6e880b32c0 2024-05-28T06:04:57.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.404+0000 7f6e8de00640 1 -- 172.21.15.96:0/3585946752 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e88143c70 con 0x7f6e880b32c0 2024-05-28T06:04:57.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.404+0000 7f6e8de00640 1 -- 172.21.15.96:0/3585946752 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e88144230 con 0x7f6e880b32c0 2024-05-28T06:04:57.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.405+0000 7f6e8de00640 1 -- 172.21.15.96:0/3585946752 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e880689e0 con 0x7f6e880b32c0 2024-05-28T06:04:57.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.406+0000 7f6e7f400640 1 -- 172.21.15.96:0/3585946752 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6e7803a070 con 0x7f6e880b32c0 2024-05-28T06:04:57.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.407+0000 7f6e7f400640 1 --2- 172.21.15.96:0/3585946752 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6e6806c8f0 0x7f6e6806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:57.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.407+0000 7f6e7f400640 1 -- 172.21.15.96:0/3585946752 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5825+0+0 (secure 0 0 0) 0x7f6e780abff0 con 0x7f6e880b32c0 2024-05-28T06:04:57.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.408+0000 7f6e86000640 1 --2- 172.21.15.96:0/3585946752 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6e6806c8f0 0x7f6e6806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:57.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.409+0000 7f6e86000640 1 --2- 172.21.15.96:0/3585946752 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6e6806c8f0 0x7f6e6806edb0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f6e88073610 tx=0x7f6e74008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:57.412 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.412+0000 7f6e7f400640 1 -- 172.21.15.96:0/3585946752 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6e7807a340 con 0x7f6e880b32c0 2024-05-28T06:04:57.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.608+0000 7f6e8de00640 1 -- 172.21.15.96:0/3585946752 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f6e88071ef0 con 0x7f6e880b32c0 2024-05-28T06:04:57.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.609+0000 7f6e7f400640 1 -- 172.21.15.96:0/3585946752 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) ==== 93+0+10 (secure 0 0 0) 0x7f6e78079c40 con 0x7f6e880b32c0 2024-05-28T06:04:57.609 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 1 2024-05-28T06:04:57.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.611+0000 7f6e8de00640 1 -- 172.21.15.96:0/3585946752 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6e6806c8f0 msgr2=0x7f6e6806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:57.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.611+0000 7f6e8de00640 1 --2- 172.21.15.96:0/3585946752 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6e6806c8f0 0x7f6e6806edb0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f6e88073610 tx=0x7f6e74008040 comp rx=0 tx=0).stop 2024-05-28T06:04:57.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.611+0000 7f6e8de00640 1 -- 172.21.15.96:0/3585946752 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6e880b32c0 msgr2=0x7f6e88143690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:57.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.611+0000 7f6e8de00640 1 --2- 172.21.15.96:0/3585946752 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6e880b32c0 0x7f6e88143690 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f6e78035040 tx=0x7f6e780046c0 comp rx=0 tx=0).stop 2024-05-28T06:04:57.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.611+0000 7f6e8de00640 1 -- 172.21.15.96:0/3585946752 shutdown_connections 2024-05-28T06:04:57.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.611+0000 7f6e8de00640 1 --2- 172.21.15.96:0/3585946752 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6e6806c8f0 0x7f6e6806edb0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:57.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.611+0000 7f6e8de00640 1 --2- 172.21.15.96:0/3585946752 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6e880d0a20 0x7f6e88157370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:57.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.611+0000 7f6e8de00640 1 --2- 172.21.15.96:0/3585946752 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6e880d0640 0x7f6e88156e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:57.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.611+0000 7f6e8de00640 1 --2- 172.21.15.96:0/3585946752 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6e880b32c0 0x7f6e88143690 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:57.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.611+0000 7f6e8de00640 1 -- 172.21.15.96:0/3585946752 >> 172.21.15.96:0/3585946752 conn(0x7f6e8806bd20 msgr2=0x7f6e880b3730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:57.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.612+0000 7f6e8de00640 1 -- 172.21.15.96:0/3585946752 shutdown_connections 2024-05-28T06:04:57.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.612+0000 7f6e8de00640 1 -- 172.21.15.96:0/3585946752 wait complete. 2024-05-28T06:04:57.631 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-28T06:04:57.825 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.825+0000 7f35e9a00640 1 Processor -- start 2024-05-28T06:04:57.825 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.825+0000 7f35e9a00640 1 -- start start 2024-05-28T06:04:57.827 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.826+0000 7f35e9a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f35e40ce5d0 0x7f35e40b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:57.827 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.826+0000 7f35e9a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f35e40b3d60 0x7f35e40b41e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:57.827 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.827+0000 7f35e9a00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35e40b3580 con 0x7f35e40ce1f0 2024-05-28T06:04:57.827 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.827+0000 7f35e9a00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f35e415f490 con 0x7f35e40ce5d0 2024-05-28T06:04:57.827 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.827+0000 7f35e9a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f35e415f770 con 0x7f35e40b3d60 2024-05-28T06:04:57.828 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.827+0000 7f35db400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f35e40ce5d0 0x7f35e40b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:57.828 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.827+0000 7f35db400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f35e40ce5d0 0x7f35e40b4dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:36426/0 (socket says 172.21.15.96:36426) 2024-05-28T06:04:57.828 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.827+0000 7f35db400640 1 -- 172.21.15.96:0/3206353055 learned_addr learned my addr 172.21.15.96:0/3206353055 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:57.828 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.827+0000 7f35e9000640 1 --2- 172.21.15.96:0/3206353055 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f35e40b3d60 0x7f35e40b41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:57.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.827+0000 7f35db400640 1 -- 172.21.15.96:0/3206353055 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f35e40b3d60 msgr2=0x7f35e40b41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:57.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.827+0000 7f35db400640 1 --2- 172.21.15.96:0/3206353055 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f35e40b3d60 0x7f35e40b41e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:57.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.827+0000 7f35db400640 1 -- 172.21.15.96:0/3206353055 >> v1:172.21.15.123:6789/0 conn(0x7f35e40ce1f0 legacy=0x7f35e415ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:57.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.827+0000 7f35db400640 1 -- 172.21.15.96:0/3206353055 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35e415fa50 con 0x7f35e40ce5d0 2024-05-28T06:04:57.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.828+0000 7f35db400640 1 --2- 172.21.15.96:0/3206353055 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f35e40ce5d0 0x7f35e40b4dd0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f35cc009870 tx=0x7f35cc02a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e66c655c933760bb server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:57.830 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.829+0000 7f35e3400640 1 -- 172.21.15.96:0/3206353055 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35cc037070 con 0x7f35e40ce5d0 2024-05-28T06:04:57.830 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.829+0000 7f35e3400640 1 -- 172.21.15.96:0/3206353055 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35cc02ae10 con 0x7f35e40ce5d0 2024-05-28T06:04:57.830 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.829+0000 7f35e3400640 1 -- 172.21.15.96:0/3206353055 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35cc032660 con 0x7f35e40ce5d0 2024-05-28T06:04:57.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.830+0000 7f35e9a00640 1 -- 172.21.15.96:0/3206353055 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f35e40ce5d0 msgr2=0x7f35e40b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:57.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.830+0000 7f35e9a00640 1 --2- 172.21.15.96:0/3206353055 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f35e40ce5d0 0x7f35e40b4dd0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f35cc009870 tx=0x7f35cc02a100 comp rx=0 tx=0).stop 2024-05-28T06:04:57.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.830+0000 7f35e9a00640 1 -- 172.21.15.96:0/3206353055 shutdown_connections 2024-05-28T06:04:57.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.830+0000 7f35e9a00640 1 --2- 172.21.15.96:0/3206353055 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f35e40b3d60 0x7f35e40b41e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:57.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.830+0000 7f35e9a00640 1 --2- 172.21.15.96:0/3206353055 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f35e40ce5d0 0x7f35e40b4dd0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:57.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.830+0000 7f35e9a00640 1 -- 172.21.15.96:0/3206353055 >> 172.21.15.96:0/3206353055 conn(0x7f35e406b900 msgr2=0x7f35e406bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:57.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.830+0000 7f35e9a00640 1 -- 172.21.15.96:0/3206353055 shutdown_connections 2024-05-28T06:04:57.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.830+0000 7f35e9a00640 1 -- 172.21.15.96:0/3206353055 wait complete. 2024-05-28T06:04:57.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.832+0000 7f35e9a00640 1 Processor -- start 2024-05-28T06:04:57.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.832+0000 7f35e9a00640 1 -- start start 2024-05-28T06:04:57.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.832+0000 7f35e9a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f35e40b3d60 0x7f35e415a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:57.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.832+0000 7f35e9a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f35e40ce1f0 0x7f35e4148690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:57.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.833+0000 7f35e9a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f35e40ce5d0 0x7f35e406ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:57.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.833+0000 7f35e9a00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f35e40ccd10 con 0x7f35e40ce1f0 2024-05-28T06:04:57.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.833+0000 7f35e9a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f35e40ccb90 con 0x7f35e40ce5d0 2024-05-28T06:04:57.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.833+0000 7f35e9a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f35e40cce90 con 0x7f35e40b3d60 2024-05-28T06:04:57.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.833+0000 7f35db400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f35e40ce1f0 0x7f35e4148690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:57.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.833+0000 7f35db400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f35e40ce1f0 0x7f35e4148690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:36440/0 (socket says 172.21.15.96:36440) 2024-05-28T06:04:57.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.833+0000 7f35db400640 1 -- 172.21.15.96:0/2608944822 learned_addr learned my addr 172.21.15.96:0/2608944822 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:57.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.833+0000 7f35db400640 1 -- 172.21.15.96:0/2608944822 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f35e40b3d60 msgr2=0x7f35e415a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:57.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.833+0000 7f35e3e00640 1 --2- 172.21.15.96:0/2608944822 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f35e40b3d60 0x7f35e415a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:57.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.833+0000 7f35db400640 1 --2- 172.21.15.96:0/2608944822 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f35e40b3d60 0x7f35e415a520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:57.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.833+0000 7f35e9000640 1 --2- 172.21.15.96:0/2608944822 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f35e40ce5d0 0x7f35e406ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:57.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.833+0000 7f35db400640 1 -- 172.21.15.96:0/2608944822 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f35e40ce5d0 msgr2=0x7f35e406ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:57.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.833+0000 7f35db400640 1 --2- 172.21.15.96:0/2608944822 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f35e40ce5d0 0x7f35e406ec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:57.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.833+0000 7f35db400640 1 -- 172.21.15.96:0/2608944822 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35e4159480 con 0x7f35e40ce1f0 2024-05-28T06:04:57.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.834+0000 7f35db400640 1 --2- 172.21.15.96:0/2608944822 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f35e40ce1f0 0x7f35e4148690 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f35cc009870 tx=0x7f35cc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:57.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.834+0000 7f35e0c00640 1 -- 172.21.15.96:0/2608944822 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35cc041070 con 0x7f35e40ce1f0 2024-05-28T06:04:57.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.834+0000 7f35e0c00640 1 -- 172.21.15.96:0/2608944822 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35cc03c670 con 0x7f35e40ce1f0 2024-05-28T06:04:57.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.834+0000 7f35e9a00640 1 -- 172.21.15.96:0/2608944822 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35e4159760 con 0x7f35e40ce1f0 2024-05-28T06:04:57.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.834+0000 7f35e0c00640 1 -- 172.21.15.96:0/2608944822 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35cc037040 con 0x7f35e40ce1f0 2024-05-28T06:04:57.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.834+0000 7f35e9a00640 1 -- 172.21.15.96:0/2608944822 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35e4159d20 con 0x7f35e40ce1f0 2024-05-28T06:04:57.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.835+0000 7f35e9a00640 1 -- 172.21.15.96:0/2608944822 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35e40689e0 con 0x7f35e40ce1f0 2024-05-28T06:04:57.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.836+0000 7f35e0c00640 1 -- 172.21.15.96:0/2608944822 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f35cc044070 con 0x7f35e40ce1f0 2024-05-28T06:04:57.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.837+0000 7f35e0c00640 1 --2- 172.21.15.96:0/2608944822 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f35c006c8f0 0x7f35c006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:57.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.837+0000 7f35e0c00640 1 -- 172.21.15.96:0/2608944822 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5825+0+0 (secure 0 0 0) 0x7f35cc0abfe0 con 0x7f35e40ce1f0 2024-05-28T06:04:57.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.838+0000 7f35e3e00640 1 --2- 172.21.15.96:0/2608944822 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f35c006c8f0 0x7f35c006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:57.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.839+0000 7f35e3e00640 1 --2- 172.21.15.96:0/2608944822 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f35c006c8f0 0x7f35c006edb0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f35d4003d90 tx=0x7f35d4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:57.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:57.841+0000 7f35e0c00640 1 -- 172.21.15.96:0/2608944822 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f35cc07a3c0 con 0x7f35e40ce1f0 2024-05-28T06:04:58.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.044+0000 7f35e9a00640 1 -- 172.21.15.96:0/2608944822 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f35e40ca070 con 0x7f35e40ce1f0 2024-05-28T06:04:58.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.045+0000 7f35e0c00640 1 -- 172.21.15.96:0/2608944822 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) ==== 104+0+11 (secure 0 0 0) 0x7f35cc079cc0 con 0x7f35e40ce1f0 2024-05-28T06:04:58.045 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 32 2024-05-28T06:04:58.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.047+0000 7f35e9a00640 1 -- 172.21.15.96:0/2608944822 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f35c006c8f0 msgr2=0x7f35c006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:58.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.047+0000 7f35e9a00640 1 --2- 172.21.15.96:0/2608944822 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f35c006c8f0 0x7f35c006edb0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f35d4003d90 tx=0x7f35d4009040 comp rx=0 tx=0).stop 2024-05-28T06:04:58.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.047+0000 7f35e9a00640 1 -- 172.21.15.96:0/2608944822 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f35e40ce1f0 msgr2=0x7f35e4148690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:58.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.047+0000 7f35e9a00640 1 --2- 172.21.15.96:0/2608944822 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f35e40ce1f0 0x7f35e4148690 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f35cc009870 tx=0x7f35cc0046c0 comp rx=0 tx=0).stop 2024-05-28T06:04:58.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.047+0000 7f35e9a00640 1 -- 172.21.15.96:0/2608944822 shutdown_connections 2024-05-28T06:04:58.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.047+0000 7f35e9a00640 1 --2- 172.21.15.96:0/2608944822 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f35c006c8f0 0x7f35c006edb0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:58.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.047+0000 7f35e9a00640 1 --2- 172.21.15.96:0/2608944822 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f35e40ce5d0 0x7f35e406ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:58.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.047+0000 7f35e9a00640 1 --2- 172.21.15.96:0/2608944822 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f35e40ce1f0 0x7f35e4148690 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:58.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.047+0000 7f35e9a00640 1 --2- 172.21.15.96:0/2608944822 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f35e40b3d60 0x7f35e415a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:58.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.047+0000 7f35e9a00640 1 -- 172.21.15.96:0/2608944822 >> 172.21.15.96:0/2608944822 conn(0x7f35e406b900 msgr2=0x7f35e415c660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:58.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.048+0000 7f35e9a00640 1 -- 172.21.15.96:0/2608944822 shutdown_connections 2024-05-28T06:04:58.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.048+0000 7f35e9a00640 1 -- 172.21.15.96:0/2608944822 wait complete. 2024-05-28T06:04:58.066 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-28T06:04:58.265 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.264+0000 7f72bfe00640 1 Processor -- start 2024-05-28T06:04:58.265 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.265+0000 7f72bfe00640 1 -- start start 2024-05-28T06:04:58.267 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.266+0000 7f72bfe00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f72c015e870 0x7f72c00d0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:58.267 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.266+0000 7f72bfe00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f72c00d14d0 0x7f72c00cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:58.267 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.266+0000 7f72bfe00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f72c00cf870 0x7f72c0162360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:58.267 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.266+0000 7f72bfe00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f72c00d2150 con 0x7f72c00cf870 2024-05-28T06:04:58.267 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.266+0000 7f72bfe00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f72c00cfd00 con 0x7f72c015e870 2024-05-28T06:04:58.267 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.266+0000 7f72bfe00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f72c015ec50 con 0x7f72c00d14d0 2024-05-28T06:04:58.267 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.266+0000 7f72bf400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f72c00cf870 0x7f72c0162360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:58.267 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.267+0000 7f72bf400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f72c00cf870 0x7f72c0162360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:36442/0 (socket says 172.21.15.96:36442) 2024-05-28T06:04:58.267 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.267+0000 7f72bf400640 1 -- 172.21.15.96:0/3837298529 learned_addr learned my addr 172.21.15.96:0/3837298529 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:58.269 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.267+0000 7f72be000640 1 --2- 172.21.15.96:0/3837298529 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f72c00d14d0 0x7f72c00cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:58.269 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.267+0000 7f72bea00640 1 --2- 172.21.15.96:0/3837298529 >> v2:172.21.15.123:3300/0 conn(0x7f72c015e870 0x7f72c00d0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:58.269 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.267+0000 7f72bf400640 1 -- 172.21.15.96:0/3837298529 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f72c00d14d0 msgr2=0x7f72c00cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:58.269 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.267+0000 7f72bf400640 1 --2- 172.21.15.96:0/3837298529 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f72c00d14d0 0x7f72c00cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:58.269 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.267+0000 7f72bf400640 1 -- 172.21.15.96:0/3837298529 >> v2:172.21.15.123:3300/0 conn(0x7f72c015e870 msgr2=0x7f72c00d0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:58.269 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.267+0000 7f72bf400640 1 --2- 172.21.15.96:0/3837298529 >> v2:172.21.15.123:3300/0 conn(0x7f72c015e870 0x7f72c00d0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:58.269 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.267+0000 7f72bf400640 1 -- 172.21.15.96:0/3837298529 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72c00cff80 con 0x7f72c00cf870 2024-05-28T06:04:58.269 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.267+0000 7f72be000640 1 --2- 172.21.15.96:0/3837298529 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f72c00d14d0 0x7f72c00cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:04:58.269 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.267+0000 7f72bea00640 1 --2- 172.21.15.96:0/3837298529 >> v2:172.21.15.123:3300/0 conn(0x7f72c015e870 0x7f72c00d0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:04:58.269 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.268+0000 7f72bf400640 1 --2- 172.21.15.96:0/3837298529 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f72c00cf870 0x7f72c0162360 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f72c008e720 tx=0x7f72b802a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83cf196293dd10b8 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:58.270 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.268+0000 7f72bd600640 1 -- 172.21.15.96:0/3837298529 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72b8036070 con 0x7f72c00cf870 2024-05-28T06:04:58.270 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.268+0000 7f72bd600640 1 -- 172.21.15.96:0/3837298529 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72b8031600 con 0x7f72c00cf870 2024-05-28T06:04:58.270 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.268+0000 7f72bd600640 1 -- 172.21.15.96:0/3837298529 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72b80319d0 con 0x7f72c00cf870 2024-05-28T06:04:58.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.269+0000 7f72bfe00640 1 -- 172.21.15.96:0/3837298529 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f72c00cf870 msgr2=0x7f72c0162360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:58.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.269+0000 7f72bfe00640 1 --2- 172.21.15.96:0/3837298529 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f72c00cf870 0x7f72c0162360 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f72c008e720 tx=0x7f72b802a6a0 comp rx=0 tx=0).stop 2024-05-28T06:04:58.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.269+0000 7f72bfe00640 1 -- 172.21.15.96:0/3837298529 shutdown_connections 2024-05-28T06:04:58.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.269+0000 7f72bfe00640 1 --2- 172.21.15.96:0/3837298529 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f72c00cf870 0x7f72c0162360 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:58.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.269+0000 7f72bfe00640 1 --2- 172.21.15.96:0/3837298529 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f72c00d14d0 0x7f72c00cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:58.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.269+0000 7f72bfe00640 1 --2- 172.21.15.96:0/3837298529 >> v2:172.21.15.123:3300/0 conn(0x7f72c015e870 0x7f72c00d0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:58.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.269+0000 7f72bfe00640 1 -- 172.21.15.96:0/3837298529 >> 172.21.15.96:0/3837298529 conn(0x7f72c008bfc0 msgr2=0x7f72c00d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:58.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.269+0000 7f72bfe00640 1 -- 172.21.15.96:0/3837298529 shutdown_connections 2024-05-28T06:04:58.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.269+0000 7f72bfe00640 1 -- 172.21.15.96:0/3837298529 wait complete. 2024-05-28T06:04:58.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.271+0000 7f72bfe00640 1 Processor -- start 2024-05-28T06:04:58.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.271+0000 7f72bfe00640 1 -- start start 2024-05-28T06:04:58.274 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.272+0000 7f72bfe00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f72c00cf870 0x7f72c0159140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:58.274 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.272+0000 7f72bfe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f72c00d14d0 0x7f72c00cd430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:58.274 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.272+0000 7f72bfe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f72c015e870 0x7f72c00b93b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:58.274 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.272+0000 7f72bfe00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f72c00d0820 con 0x7f72c015e870 2024-05-28T06:04:58.274 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.272+0000 7f72bfe00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f72c00d0540 con 0x7f72c00d14d0 2024-05-28T06:04:58.274 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.272+0000 7f72bfe00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f72c00d4fb0 con 0x7f72c00cf870 2024-05-28T06:04:58.274 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.272+0000 7f72bf400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f72c015e870 0x7f72c00b93b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:58.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.272+0000 7f72bf400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f72c015e870 0x7f72c00b93b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:36456/0 (socket says 172.21.15.96:36456) 2024-05-28T06:04:58.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.272+0000 7f72bf400640 1 -- 172.21.15.96:0/3929766487 learned_addr learned my addr 172.21.15.96:0/3929766487 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:58.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.273+0000 7f72be000640 1 --2- 172.21.15.96:0/3929766487 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f72c00d14d0 0x7f72c00cd430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:58.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.273+0000 7f72bea00640 1 --2- 172.21.15.96:0/3929766487 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f72c00cf870 0x7f72c0159140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:58.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.273+0000 7f72bf400640 1 -- 172.21.15.96:0/3929766487 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f72c00cf870 msgr2=0x7f72c0159140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:58.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.273+0000 7f72bf400640 1 --2- 172.21.15.96:0/3929766487 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f72c00cf870 0x7f72c0159140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:58.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.273+0000 7f72bf400640 1 -- 172.21.15.96:0/3929766487 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f72c00d14d0 msgr2=0x7f72c00cd430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:58.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.273+0000 7f72bf400640 1 --2- 172.21.15.96:0/3929766487 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f72c00d14d0 0x7f72c00cd430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:58.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.273+0000 7f72bf400640 1 -- 172.21.15.96:0/3929766487 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72c015fc50 con 0x7f72c015e870 2024-05-28T06:04:58.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.273+0000 7f72bf400640 1 --2- 172.21.15.96:0/3929766487 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f72c015e870 0x7f72c00b93b0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f72b8034040 tx=0x7f72b8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:58.276 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.273+0000 7f72b7400640 1 -- 172.21.15.96:0/3929766487 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72b8041020 con 0x7f72c015e870 2024-05-28T06:04:58.276 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.273+0000 7f72b7400640 1 -- 172.21.15.96:0/3929766487 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72b80077a0 con 0x7f72c015e870 2024-05-28T06:04:58.276 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.273+0000 7f72b7400640 1 -- 172.21.15.96:0/3929766487 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72b8036040 con 0x7f72c015e870 2024-05-28T06:04:58.276 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.273+0000 7f72bfe00640 1 -- 172.21.15.96:0/3929766487 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72c015ff30 con 0x7f72c015e870 2024-05-28T06:04:58.276 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.274+0000 7f72bfe00640 1 -- 172.21.15.96:0/3929766487 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72c01604f0 con 0x7f72c015e870 2024-05-28T06:04:58.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.275+0000 7f72bfe00640 1 -- 172.21.15.96:0/3929766487 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72c00689e0 con 0x7f72c015e870 2024-05-28T06:04:58.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.276+0000 7f72b7400640 1 -- 172.21.15.96:0/3929766487 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f72b8044020 con 0x7f72c015e870 2024-05-28T06:04:58.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.277+0000 7f72b7400640 1 --2- 172.21.15.96:0/3929766487 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f729c06c8f0 0x7f729c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:58.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.277+0000 7f72b7400640 1 -- 172.21.15.96:0/3929766487 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5825+0+0 (secure 0 0 0) 0x7f72b80acfa0 con 0x7f72c015e870 2024-05-28T06:04:58.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.277+0000 7f72bea00640 1 --2- 172.21.15.96:0/3929766487 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f729c06c8f0 0x7f729c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:58.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.279+0000 7f72bea00640 1 --2- 172.21.15.96:0/3929766487 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f729c06c8f0 0x7f729c06edb0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f72b0006fd0 tx=0x7f72b0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:58.281 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.281+0000 7f72b7400640 1 -- 172.21.15.96:0/3929766487 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f72b804b020 con 0x7f72c015e870 2024-05-28T06:04:58.480 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.479+0000 7f72bfe00640 1 -- 172.21.15.96:0/3929766487 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f72c00ba500 con 0x7f72c015e870 2024-05-28T06:04:58.481 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.480+0000 7f72b7400640 1 -- 172.21.15.96:0/3929766487 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) ==== 100+0+11 (secure 0 0 0) 0x7f72b803ae20 con 0x7f72c015e870 2024-05-28T06:04:58.481 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 64 2024-05-28T06:04:58.483 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.483+0000 7f72bfe00640 1 -- 172.21.15.96:0/3929766487 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f729c06c8f0 msgr2=0x7f729c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:58.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.483+0000 7f72bfe00640 1 --2- 172.21.15.96:0/3929766487 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f729c06c8f0 0x7f729c06edb0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f72b0006fd0 tx=0x7f72b0008040 comp rx=0 tx=0).stop 2024-05-28T06:04:58.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.483+0000 7f72bfe00640 1 -- 172.21.15.96:0/3929766487 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f72c015e870 msgr2=0x7f72c00b93b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:58.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.483+0000 7f72bfe00640 1 --2- 172.21.15.96:0/3929766487 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f72c015e870 0x7f72c00b93b0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f72b8034040 tx=0x7f72b8004a40 comp rx=0 tx=0).stop 2024-05-28T06:04:58.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.483+0000 7f72bfe00640 1 -- 172.21.15.96:0/3929766487 shutdown_connections 2024-05-28T06:04:58.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.483+0000 7f72bfe00640 1 --2- 172.21.15.96:0/3929766487 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f729c06c8f0 0x7f729c06edb0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:58.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.483+0000 7f72bfe00640 1 --2- 172.21.15.96:0/3929766487 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f72c015e870 0x7f72c00b93b0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:58.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.483+0000 7f72bfe00640 1 --2- 172.21.15.96:0/3929766487 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f72c00d14d0 0x7f72c00cd430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:58.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.483+0000 7f72bfe00640 1 --2- 172.21.15.96:0/3929766487 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f72c00cf870 0x7f72c0159140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:58.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.483+0000 7f72bfe00640 1 -- 172.21.15.96:0/3929766487 >> 172.21.15.96:0/3929766487 conn(0x7f72c008bfc0 msgr2=0x7f72c006e3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:58.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.483+0000 7f72bfe00640 1 -- 172.21.15.96:0/3929766487 shutdown_connections 2024-05-28T06:04:58.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.483+0000 7f72bfe00640 1 -- 172.21.15.96:0/3929766487 wait complete. 2024-05-28T06:04:58.503 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-28T06:04:58.696 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.695+0000 7f0679600640 1 Processor -- start 2024-05-28T06:04:58.696 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.696+0000 7f0679600640 1 -- start start 2024-05-28T06:04:58.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.697+0000 7f0679600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f06740d16a0 0x7f06740cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:58.698 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.697+0000 7f0679600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f06740cf870 0x7f06740d25e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:58.698 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.697+0000 7f0679600640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f067415ec50 con 0x7f067415e870 2024-05-28T06:04:58.698 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.697+0000 7f0679600640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f06740d2150 con 0x7f06740cf870 2024-05-28T06:04:58.698 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.697+0000 7f0679600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f06740cfd20 con 0x7f06740d16a0 2024-05-28T06:04:58.698 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.697+0000 7f0673e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f067415e870 0x7f06740d0f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:58556/0 (socket says 172.21.15.96:58556) 2024-05-28T06:04:58.698 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.697+0000 7f0673e00640 1 -- 172.21.15.96:0/1028073597 learned_addr learned my addr 172.21.15.96:0/1028073597 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:04:58.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.698+0000 7f0678c00640 1 --2- 172.21.15.96:0/1028073597 >> v2:172.21.15.123:3300/0 conn(0x7f06740cf870 0x7f06740d25e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:58.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.698+0000 7f0673400640 1 --2- 172.21.15.96:0/1028073597 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f06740d16a0 0x7f06740cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:58.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.698+0000 7f0672a00640 1 -- 172.21.15.96:0/1028073597 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1783232969 0 0) 0x7f067415ec50 con 0x7f067415e870 2024-05-28T06:04:58.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.698+0000 7f0672a00640 1 -- 172.21.15.96:0/1028073597 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0654003520 con 0x7f067415e870 2024-05-28T06:04:58.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.698+0000 7f0672a00640 1 -- 172.21.15.96:0/1028073597 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1905428529 0 0) 0x7f0654003520 con 0x7f067415e870 2024-05-28T06:04:58.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.698+0000 7f0672a00640 1 -- 172.21.15.96:0/1028073597 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f06740d16a0 msgr2=0x7f06740cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:58.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.698+0000 7f0672a00640 1 --2- 172.21.15.96:0/1028073597 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f06740d16a0 0x7f06740cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:58.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.698+0000 7f0672a00640 1 -- 172.21.15.96:0/1028073597 >> v2:172.21.15.123:3300/0 conn(0x7f06740cf870 msgr2=0x7f06740d25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:58.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.698+0000 7f0672a00640 1 --2- 172.21.15.96:0/1028073597 >> v2:172.21.15.123:3300/0 conn(0x7f06740cf870 0x7f06740d25e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:58.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.698+0000 7f0672a00640 1 -- 172.21.15.96:0/1028073597 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06740cff80 con 0x7f067415e870 2024-05-28T06:04:58.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.698+0000 7f0672a00640 1 -- 172.21.15.96:0/1028073597 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f0664001670 con 0x7f067415e870 2024-05-28T06:04:58.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.698+0000 7f0672a00640 1 -- 172.21.15.96:0/1028073597 >> v1:172.21.15.96:6789/0 conn(0x7f067415e870 legacy=0x7f06740d0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:58.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.698+0000 7f0672a00640 1 --2- 172.21.15.96:0/1028073597 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f06540039b0 0x7f0654003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:58.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.699+0000 7f0672a00640 1 --2- 172.21.15.96:0/1028073597 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f06740d16a0 0x7f0654004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:58.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.699+0000 7f0672a00640 1 --2- 172.21.15.96:0/1028073597 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0654004d40 0x7f06540051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:58.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.699+0000 7f0672a00640 1 -- 172.21.15.96:0/1028073597 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f06740d2150 con 0x7f0654004d40 2024-05-28T06:04:58.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.699+0000 7f0672a00640 1 -- 172.21.15.96:0/1028073597 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f06740cfd20 con 0x7f06540039b0 2024-05-28T06:04:58.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.699+0000 7f0672a00640 1 -- 172.21.15.96:0/1028073597 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f065400b9c0 con 0x7f06740d16a0 2024-05-28T06:04:58.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.699+0000 7f0678c00640 1 --2- 172.21.15.96:0/1028073597 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0654004d40 0x7f06540051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:58.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.699+0000 7f0673400640 1 --2- 172.21.15.96:0/1028073597 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f06740d16a0 0x7f0654004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:58.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.699+0000 7f0678c00640 1 -- 172.21.15.96:0/1028073597 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f06740d16a0 msgr2=0x7f0654004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:58.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.699+0000 7f0678c00640 1 --2- 172.21.15.96:0/1028073597 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f06740d16a0 0x7f0654004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:58.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.699+0000 7f0678c00640 1 -- 172.21.15.96:0/1028073597 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f06540039b0 msgr2=0x7f0654003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:58.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.699+0000 7f0678c00640 1 --2- 172.21.15.96:0/1028073597 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f06540039b0 0x7f0654003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:58.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.699+0000 7f0678c00640 1 -- 172.21.15.96:0/1028073597 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0654003520 con 0x7f0654004d40 2024-05-28T06:04:58.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.699+0000 7f0673400640 1 --2- 172.21.15.96:0/1028073597 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f06740d16a0 0x7f0654004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:04:58.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.700+0000 7f0678c00640 1 --2- 172.21.15.96:0/1028073597 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0654004d40 0x7f06540051c0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f066c009ac0 tx=0x7f0654005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47487eba33cf1f7b server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:58.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.701+0000 7f0672a00640 1 -- 172.21.15.96:0/1028073597 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f066c036070 con 0x7f0654004d40 2024-05-28T06:04:58.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.701+0000 7f0672a00640 1 -- 172.21.15.96:0/1028073597 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f066c031400 con 0x7f0654004d40 2024-05-28T06:04:58.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.701+0000 7f0672a00640 1 -- 172.21.15.96:0/1028073597 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f066c009690 con 0x7f0654004d40 2024-05-28T06:04:58.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.701+0000 7f0679600640 1 -- 172.21.15.96:0/1028073597 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0654004d40 msgr2=0x7f06540051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:58.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.701+0000 7f0679600640 1 --2- 172.21.15.96:0/1028073597 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0654004d40 0x7f06540051c0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f066c009ac0 tx=0x7f0654005830 comp rx=0 tx=0).stop 2024-05-28T06:04:58.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.701+0000 7f0678c00640 1 -- 172.21.15.96:0/1028073597 reap_dead start 2024-05-28T06:04:58.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.701+0000 7f0679600640 1 -- 172.21.15.96:0/1028073597 shutdown_connections 2024-05-28T06:04:58.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.701+0000 7f0679600640 1 -- 172.21.15.96:0/1028073597 >> 172.21.15.96:0/1028073597 conn(0x7f067408bfc0 msgr2=0x7f06740d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:58.705 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.701+0000 7f0679600640 1 -- 172.21.15.96:0/1028073597 shutdown_connections 2024-05-28T06:04:58.705 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.701+0000 7f0679600640 1 -- 172.21.15.96:0/1028073597 wait complete. 2024-05-28T06:04:58.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.703+0000 7f0679600640 1 Processor -- start 2024-05-28T06:04:58.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.703+0000 7f0679600640 1 -- start start 2024-05-28T06:04:58.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.704+0000 7f0679600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f06740c7260 0x7f06741f4790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:58.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.704+0000 7f0679600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f067406d650 0x7f067406dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:58.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.704+0000 7f0679600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f06740cb850 0x7f06740cbcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:58.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.704+0000 7f0679600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f06740d0820 con 0x7f067406d650 2024-05-28T06:04:58.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.704+0000 7f0679600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f06740d0540 con 0x7f06740cb850 2024-05-28T06:04:58.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.704+0000 7f0679600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f06740d4fb0 con 0x7f06740c7260 2024-05-28T06:04:58.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.704+0000 7f0673400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f067406d650 0x7f067406dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:58.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.705+0000 7f0673400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f067406d650 0x7f067406dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:36472/0 (socket says 172.21.15.96:36472) 2024-05-28T06:04:58.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.705+0000 7f0673400640 1 -- 172.21.15.96:0/407448957 learned_addr learned my addr 172.21.15.96:0/407448957 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:58.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.705+0000 7f0673e00640 1 --2- 172.21.15.96:0/407448957 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f06740c7260 0x7f06741f4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:58.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.705+0000 7f0673400640 1 -- 172.21.15.96:0/407448957 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f06740c7260 msgr2=0x7f06741f4790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:58.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.705+0000 7f0678c00640 1 --2- 172.21.15.96:0/407448957 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f06740cb850 0x7f06740cbcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:58.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.705+0000 7f0673400640 1 --2- 172.21.15.96:0/407448957 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f06740c7260 0x7f06741f4790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:58.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.705+0000 7f0673400640 1 -- 172.21.15.96:0/407448957 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f06740cb850 msgr2=0x7f06740cbcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:58.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.705+0000 7f0673400640 1 --2- 172.21.15.96:0/407448957 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f06740cb850 0x7f06740cbcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:58.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.705+0000 7f0673400640 1 -- 172.21.15.96:0/407448957 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06740ccb60 con 0x7f067406d650 2024-05-28T06:04:58.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.705+0000 7f0673e00640 1 --2- 172.21.15.96:0/407448957 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f06740c7260 0x7f06741f4790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:04:58.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.705+0000 7f0673400640 1 --2- 172.21.15.96:0/407448957 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f067406d650 0x7f067406dad0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f065c00f380 tx=0x7f065c00f840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:58.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.706+0000 7f066be00640 1 -- 172.21.15.96:0/407448957 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f065c003b90 con 0x7f067406d650 2024-05-28T06:04:58.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.706+0000 7f066be00640 1 -- 172.21.15.96:0/407448957 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f065c003d30 con 0x7f067406d650 2024-05-28T06:04:58.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.706+0000 7f0679600640 1 -- 172.21.15.96:0/407448957 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06740cce40 con 0x7f067406d650 2024-05-28T06:04:58.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.706+0000 7f066be00640 1 -- 172.21.15.96:0/407448957 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f065c004220 con 0x7f067406d650 2024-05-28T06:04:58.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.706+0000 7f0679600640 1 -- 172.21.15.96:0/407448957 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06740c7640 con 0x7f067406d650 2024-05-28T06:04:58.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.707+0000 7f0679600640 1 -- 172.21.15.96:0/407448957 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06740689e0 con 0x7f067406d650 2024-05-28T06:04:58.710 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.708+0000 7f066be00640 1 -- 172.21.15.96:0/407448957 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f065c0043c0 con 0x7f067406d650 2024-05-28T06:04:58.710 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.709+0000 7f066be00640 1 --2- 172.21.15.96:0/407448957 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f065006c920 0x7f065006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:58.711 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.709+0000 7f066be00640 1 -- 172.21.15.96:0/407448957 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5825+0+0 (secure 0 0 0) 0x7f065c08c230 con 0x7f067406d650 2024-05-28T06:04:58.711 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.709+0000 7f0673e00640 1 --2- 172.21.15.96:0/407448957 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f065006c920 0x7f065006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:58.711 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.710+0000 7f0673e00640 1 --2- 172.21.15.96:0/407448957 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f065006c920 0x7f065006ede0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f0664001eb0 tx=0x7f0664009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:58.713 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.713+0000 7f066be00640 1 -- 172.21.15.96:0/407448957 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f065c012070 con 0x7f067406d650 2024-05-28T06:04:58.876 DEBUG:teuthology.orchestra.run.smithi096:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:04:58.884 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:04:58.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.931+0000 7f0679600640 1 -- 172.21.15.96:0/407448957 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f06740ba500 con 0x7f067406d650 2024-05-28T06:04:58.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.931+0000 7f066be00640 1 -- 172.21.15.96:0/407448957 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) ==== 80+0+0 (secure 0 0 0) 0x7f065c05a540 con 0x7f067406d650 2024-05-28T06:04:58.932 INFO:teuthology.orchestra.run.smithi096.stderr:listed 0 entries 2024-05-28T06:04:58.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.934+0000 7f0679600640 1 -- 172.21.15.96:0/407448957 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f065006c920 msgr2=0x7f065006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:58.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.934+0000 7f0679600640 1 --2- 172.21.15.96:0/407448957 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f065006c920 0x7f065006ede0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f0664001eb0 tx=0x7f0664009040 comp rx=0 tx=0).stop 2024-05-28T06:04:58.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.934+0000 7f0679600640 1 -- 172.21.15.96:0/407448957 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f067406d650 msgr2=0x7f067406dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:58.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.934+0000 7f0679600640 1 --2- 172.21.15.96:0/407448957 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f067406d650 0x7f067406dad0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f065c00f380 tx=0x7f065c00f840 comp rx=0 tx=0).stop 2024-05-28T06:04:58.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.934+0000 7f0679600640 1 -- 172.21.15.96:0/407448957 shutdown_connections 2024-05-28T06:04:58.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.934+0000 7f0679600640 1 --2- 172.21.15.96:0/407448957 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f065006c920 0x7f065006ede0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:58.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.934+0000 7f0679600640 1 --2- 172.21.15.96:0/407448957 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f06740cb850 0x7f06740cbcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:58.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.934+0000 7f0679600640 1 --2- 172.21.15.96:0/407448957 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f067406d650 0x7f067406dad0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:58.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.934+0000 7f0679600640 1 --2- 172.21.15.96:0/407448957 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f06740c7260 0x7f06741f4790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:58.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.934+0000 7f0679600640 1 -- 172.21.15.96:0/407448957 >> 172.21.15.96:0/407448957 conn(0x7f067408bfc0 msgr2=0x7f06740b2aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:58.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.934+0000 7f0679600640 1 -- 172.21.15.96:0/407448957 shutdown_connections 2024-05-28T06:04:58.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:58.934+0000 7f0679600640 1 -- 172.21.15.96:0/407448957 wait complete. 2024-05-28T06:04:58.951 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi096.front.sepia.ceph.com... 2024-05-28T06:04:58.951 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:04:58.951 DEBUG:teuthology.orchestra.run.smithi096:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-05-28T06:04:59.021 INFO:teuthology.orchestra.run:waiting for 300 2024-05-28T06:04:59.021 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi096.front.sepia.ceph.com 2024-05-28T06:04:59.021 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:04:59.021 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-28T06:04:59.120 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-05-28T06:04:59.120 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:04:59.176 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-05-28T06:04:59.176 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:04:59.231 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:04:59.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.435+0000 7f5142c00640 1 Processor -- start 2024-05-28T06:04:59.436 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.435+0000 7f5142c00640 1 -- start start 2024-05-28T06:04:59.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.436+0000 7f5142c00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f513c073310 0x7f513c0b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:59.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.436+0000 7f5142c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f513c06dc20 0x7f513c15f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:59.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.436+0000 7f5142c00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f513c0afd60 con 0x7f513c15b3a0 2024-05-28T06:04:59.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.436+0000 7f5142c00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f513c06f980 con 0x7f513c073310 2024-05-28T06:04:59.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.436+0000 7f5142c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f513c0ae0c0 con 0x7f513c06dc20 2024-05-28T06:04:59.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.436+0000 7f5141800640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f513c15b3a0 0x7f513c072c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:60418/0 (socket says 172.21.15.96:60418) 2024-05-28T06:04:59.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.436+0000 7f5141800640 1 -- 172.21.15.96:0/2193649176 learned_addr learned my addr 172.21.15.96:0/2193649176 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:04:59.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.437+0000 7f5140e00640 1 --2- 172.21.15.96:0/2193649176 >> v2:172.21.15.123:3300/0 conn(0x7f513c073310 0x7f513c0b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:59.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.437+0000 7f513be00640 1 -- 172.21.15.96:0/2193649176 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3999402853 0 0) 0x7f513c0afd60 con 0x7f513c15b3a0 2024-05-28T06:04:59.444 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.437+0000 7f5142200640 1 --2- 172.21.15.96:0/2193649176 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f513c06dc20 0x7f513c15f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:59.444 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.437+0000 7f513be00640 1 -- 172.21.15.96:0/2193649176 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5124003520 con 0x7f513c15b3a0 2024-05-28T06:04:59.444 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.437+0000 7f513be00640 1 -- 172.21.15.96:0/2193649176 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1274415854 0 0) 0x7f5124003520 con 0x7f513c15b3a0 2024-05-28T06:04:59.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.437+0000 7f513be00640 1 -- 172.21.15.96:0/2193649176 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f513c06dc20 msgr2=0x7f513c15f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:59.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.437+0000 7f513be00640 1 --2- 172.21.15.96:0/2193649176 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f513c06dc20 0x7f513c15f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:59.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.437+0000 7f513be00640 1 -- 172.21.15.96:0/2193649176 >> v2:172.21.15.123:3300/0 conn(0x7f513c073310 msgr2=0x7f513c0b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:59.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.437+0000 7f513be00640 1 --2- 172.21.15.96:0/2193649176 >> v2:172.21.15.123:3300/0 conn(0x7f513c073310 0x7f513c0b32c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:59.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.437+0000 7f513be00640 1 -- 172.21.15.96:0/2193649176 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f513c0af7a0 con 0x7f513c15b3a0 2024-05-28T06:04:59.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.437+0000 7f513be00640 1 -- 172.21.15.96:0/2193649176 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f5130001670 con 0x7f513c15b3a0 2024-05-28T06:04:59.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.437+0000 7f513be00640 1 -- 172.21.15.96:0/2193649176 >> v1:172.21.15.96:6789/0 conn(0x7f513c15b3a0 legacy=0x7f513c072c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:59.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.437+0000 7f513be00640 1 --2- 172.21.15.96:0/2193649176 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f51240039b0 0x7f5124003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:59.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.437+0000 7f513be00640 1 --2- 172.21.15.96:0/2193649176 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f513c06dc20 0x7f5124004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:59.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.437+0000 7f513be00640 1 --2- 172.21.15.96:0/2193649176 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5124004d40 0x7f51240051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:59.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.437+0000 7f513be00640 1 -- 172.21.15.96:0/2193649176 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f513c06f980 con 0x7f5124004d40 2024-05-28T06:04:59.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.437+0000 7f513be00640 1 -- 172.21.15.96:0/2193649176 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f513c0ae0c0 con 0x7f51240039b0 2024-05-28T06:04:59.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.437+0000 7f513be00640 1 -- 172.21.15.96:0/2193649176 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f512400dd30 con 0x7f513c06dc20 2024-05-28T06:04:59.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.437+0000 7f5140e00640 1 --2- 172.21.15.96:0/2193649176 >> v2:172.21.15.123:3300/0 conn(0x7f513c073310 0x7f513c0b32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-28T06:04:59.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.438+0000 7f5142200640 1 --2- 172.21.15.96:0/2193649176 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5124004d40 0x7f51240051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:59.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.438+0000 7f5141800640 1 --2- 172.21.15.96:0/2193649176 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f51240039b0 0x7f5124003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:59.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.438+0000 7f5142200640 1 -- 172.21.15.96:0/2193649176 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f513c06dc20 msgr2=0x7f5124004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:59.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.438+0000 7f5140e00640 1 --2- 172.21.15.96:0/2193649176 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f513c06dc20 0x7f5124004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:59.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.438+0000 7f5142200640 1 --2- 172.21.15.96:0/2193649176 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f513c06dc20 0x7f5124004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:59.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.438+0000 7f5142200640 1 -- 172.21.15.96:0/2193649176 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f51240039b0 msgr2=0x7f5124003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:59.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.438+0000 7f5142200640 1 --2- 172.21.15.96:0/2193649176 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f51240039b0 0x7f5124003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:59.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.438+0000 7f5142200640 1 -- 172.21.15.96:0/2193649176 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5124003520 con 0x7f5124004d40 2024-05-28T06:04:59.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.438+0000 7f5141800640 1 --2- 172.21.15.96:0/2193649176 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f51240039b0 0x7f5124003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:04:59.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.439+0000 7f5142200640 1 --2- 172.21.15.96:0/2193649176 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5124004d40 0x7f51240051c0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f5134007360 tx=0x7f5124005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b6e9c0107d00644 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:59.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.439+0000 7f513be00640 1 -- 172.21.15.96:0/2193649176 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5134009be0 con 0x7f5124004d40 2024-05-28T06:04:59.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.439+0000 7f513be00640 1 -- 172.21.15.96:0/2193649176 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5134036600 con 0x7f5124004d40 2024-05-28T06:04:59.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.439+0000 7f513be00640 1 -- 172.21.15.96:0/2193649176 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51340369d0 con 0x7f5124004d40 2024-05-28T06:04:59.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.440+0000 7f5142c00640 1 -- 172.21.15.96:0/2193649176 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5124004d40 msgr2=0x7f51240051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:59.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.440+0000 7f5142c00640 1 --2- 172.21.15.96:0/2193649176 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5124004d40 0x7f51240051c0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f5134007360 tx=0x7f5124005830 comp rx=0 tx=0).stop 2024-05-28T06:04:59.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.440+0000 7f5142200640 1 -- 172.21.15.96:0/2193649176 reap_dead start 2024-05-28T06:04:59.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.440+0000 7f5142c00640 1 -- 172.21.15.96:0/2193649176 shutdown_connections 2024-05-28T06:04:59.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.440+0000 7f5142c00640 1 -- 172.21.15.96:0/2193649176 >> 172.21.15.96:0/2193649176 conn(0x7f513c0b1fc0 msgr2=0x7f513c0b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:59.451 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.440+0000 7f5142c00640 1 -- 172.21.15.96:0/2193649176 shutdown_connections 2024-05-28T06:04:59.451 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.440+0000 7f5142c00640 1 -- 172.21.15.96:0/2193649176 wait complete. 2024-05-28T06:04:59.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.442+0000 7f5142c00640 1 Processor -- start 2024-05-28T06:04:59.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.442+0000 7f5142c00640 1 -- start start 2024-05-28T06:04:59.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.443+0000 7f5142c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5124004d40 0x7f513c14b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:59.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.443+0000 7f5142c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f513c0b0d10 0x7f513c0b1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:59.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.443+0000 7f5142c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f513c15bb80 0x7f513c15c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:59.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.443+0000 7f5142c00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f513c15f9a0 con 0x7f5124004d40 2024-05-28T06:04:59.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.443+0000 7f5142c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f513c15b780 con 0x7f513c0b0d10 2024-05-28T06:04:59.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.443+0000 7f5142c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f513c15fb20 con 0x7f513c15bb80 2024-05-28T06:04:59.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.443+0000 7f5141800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5124004d40 0x7f513c14b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:59.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.443+0000 7f5141800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5124004d40 0x7f513c14b230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54538/0 (socket says 172.21.15.96:54538) 2024-05-28T06:04:59.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.443+0000 7f5141800640 1 -- 172.21.15.96:0/327272125 learned_addr learned my addr 172.21.15.96:0/327272125 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:59.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.444+0000 7f5140e00640 1 --2- 172.21.15.96:0/327272125 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f513c0b0d10 0x7f513c0b1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:59.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.444+0000 7f5141800640 1 -- 172.21.15.96:0/327272125 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f513c15bb80 msgr2=0x7f513c15c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:59.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.444+0000 7f5141800640 1 --2- 172.21.15.96:0/327272125 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f513c15bb80 0x7f513c15c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:59.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.444+0000 7f5141800640 1 -- 172.21.15.96:0/327272125 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f513c0b0d10 msgr2=0x7f513c0b1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:59.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.444+0000 7f5141800640 1 --2- 172.21.15.96:0/327272125 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f513c0b0d10 0x7f513c0b1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:59.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.444+0000 7f5141800640 1 -- 172.21.15.96:0/327272125 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f513c15c8b0 con 0x7f5124004d40 2024-05-28T06:04:59.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.444+0000 7f5140e00640 1 --2- 172.21.15.96:0/327272125 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f513c0b0d10 0x7f513c0b1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:04:59.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.444+0000 7f5141800640 1 --2- 172.21.15.96:0/327272125 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5124004d40 0x7f513c14b230 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f513000d890 tx=0x7f513000dd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:59.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.444+0000 7f5139600640 1 -- 172.21.15.96:0/327272125 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5130019070 con 0x7f5124004d40 2024-05-28T06:04:59.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.444+0000 7f5139600640 1 -- 172.21.15.96:0/327272125 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51300074e0 con 0x7f5124004d40 2024-05-28T06:04:59.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.444+0000 7f5139600640 1 -- 172.21.15.96:0/327272125 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5130004a00 con 0x7f5124004d40 2024-05-28T06:04:59.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.444+0000 7f5142c00640 1 -- 172.21.15.96:0/327272125 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f513c15cb90 con 0x7f5124004d40 2024-05-28T06:04:59.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.444+0000 7f5142c00640 1 -- 172.21.15.96:0/327272125 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f513c15d150 con 0x7f5124004d40 2024-05-28T06:04:59.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.446+0000 7f5142c00640 1 -- 172.21.15.96:0/327272125 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f513c0b16d0 con 0x7f5124004d40 2024-05-28T06:04:59.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.447+0000 7f5139600640 1 -- 172.21.15.96:0/327272125 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5130004ba0 con 0x7f5124004d40 2024-05-28T06:04:59.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.448+0000 7f5139600640 1 --2- 172.21.15.96:0/327272125 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f511c06c8f0 0x7f511c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:59.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.448+0000 7f5139600640 1 -- 172.21.15.96:0/327272125 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5825+0+0 (secure 0 0 0) 0x7f513008b600 con 0x7f5124004d40 2024-05-28T06:04:59.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.448+0000 7f5140e00640 1 --2- 172.21.15.96:0/327272125 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f511c06c8f0 0x7f511c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:59.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.449+0000 7f5140e00640 1 --2- 172.21.15.96:0/327272125 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f511c06c8f0 0x7f511c06edb0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f513c06ed30 tx=0x7f5128008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:59.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.452+0000 7f5139600640 1 -- 172.21.15.96:0/327272125 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f51300599b0 con 0x7f5124004d40 2024-05-28T06:04:59.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.647+0000 7f5142c00640 1 -- 172.21.15.96:0/327272125 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f513c14b7a0 con 0x7f5124004d40 2024-05-28T06:04:59.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.649+0000 7f5139600640 1 -- 172.21.15.96:0/327272125 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) ==== 74+0+16865 (secure 0 0 0) 0x7f51300592b0 con 0x7f5124004d40 2024-05-28T06:04:59.649 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:04:59.650 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":22,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:04:52.624604+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-28T06:04:08.741938+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-28T06:04:09.301470+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.b","mappings":[{"from":3,"to":1}]},{"pgid":"3.d","mappings":[{"from":7,"to":2}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:04:59.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.651+0000 7f5142c00640 1 -- 172.21.15.96:0/327272125 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f511c06c8f0 msgr2=0x7f511c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:59.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.651+0000 7f5142c00640 1 --2- 172.21.15.96:0/327272125 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f511c06c8f0 0x7f511c06edb0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f513c06ed30 tx=0x7f5128008040 comp rx=0 tx=0).stop 2024-05-28T06:04:59.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.651+0000 7f5142c00640 1 -- 172.21.15.96:0/327272125 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5124004d40 msgr2=0x7f513c14b230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:59.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.651+0000 7f5142c00640 1 --2- 172.21.15.96:0/327272125 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5124004d40 0x7f513c14b230 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f513000d890 tx=0x7f513000dd50 comp rx=0 tx=0).stop 2024-05-28T06:04:59.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.651+0000 7f5142c00640 1 -- 172.21.15.96:0/327272125 shutdown_connections 2024-05-28T06:04:59.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.651+0000 7f5142c00640 1 --2- 172.21.15.96:0/327272125 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f511c06c8f0 0x7f511c06edb0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:59.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.651+0000 7f5142c00640 1 --2- 172.21.15.96:0/327272125 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f513c15bb80 0x7f513c15c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:59.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.651+0000 7f5142c00640 1 --2- 172.21.15.96:0/327272125 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f513c0b0d10 0x7f513c0b1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:59.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.651+0000 7f5142c00640 1 --2- 172.21.15.96:0/327272125 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5124004d40 0x7f513c14b230 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:59.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.651+0000 7f5142c00640 1 -- 172.21.15.96:0/327272125 >> 172.21.15.96:0/327272125 conn(0x7f513c0b1fc0 msgr2=0x7f513c156710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:59.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.651+0000 7f5142c00640 1 -- 172.21.15.96:0/327272125 shutdown_connections 2024-05-28T06:04:59.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.651+0000 7f5142c00640 1 -- 172.21.15.96:0/327272125 wait complete. 2024-05-28T06:04:59.669 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-28T06:03:59.517102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-28T06:04:08.741938+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-28T06:04:09.301470+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-28T06:04:59.669 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-28T06:04:59.860 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.860+0000 7f31db200640 1 Processor -- start 2024-05-28T06:04:59.860 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.860+0000 7f31db200640 1 -- start start 2024-05-28T06:04:59.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.861+0000 7f31db200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f31dc0caa40 0x7f31dc0ce960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:59.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.861+0000 7f31db200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f31dc0cae20 0x7f31dc151db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:59.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.862+0000 7f31db200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31dc152780 con 0x7f31dc1522f0 2024-05-28T06:04:59.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.862+0000 7f31db200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f31dc0d0b30 con 0x7f31dc0caa40 2024-05-28T06:04:59.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.862+0000 7f31db200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f31dc0b4160 con 0x7f31dc0cae20 2024-05-28T06:04:59.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.862+0000 7f31da800640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f31dc1522f0 0x7f31dc0cf710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:60420/0 (socket says 172.21.15.96:60420) 2024-05-28T06:04:59.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.862+0000 7f31da800640 1 -- 172.21.15.96:0/488791414 learned_addr learned my addr 172.21.15.96:0/488791414 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:04:59.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.862+0000 7f31d9e00640 1 --2- 172.21.15.96:0/488791414 >> v2:172.21.15.123:3300/0 conn(0x7f31dc0caa40 0x7f31dc0ce960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:59.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.862+0000 7f31d9400640 1 --2- 172.21.15.96:0/488791414 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f31dc0cae20 0x7f31dc151db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:59.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.863+0000 7f31d8a00640 1 -- 172.21.15.96:0/488791414 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 626693034 0 0) 0x7f31dc152780 con 0x7f31dc1522f0 2024-05-28T06:04:59.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.863+0000 7f31d8a00640 1 -- 172.21.15.96:0/488791414 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f31c0003520 con 0x7f31dc1522f0 2024-05-28T06:04:59.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.863+0000 7f31d9e00640 1 -- 172.21.15.96:0/488791414 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f31dc0cae20 msgr2=0x7f31dc151db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:04:59.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.863+0000 7f31d9e00640 1 --2- 172.21.15.96:0/488791414 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f31dc0cae20 0x7f31dc151db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:59.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.863+0000 7f31d9e00640 1 -- 172.21.15.96:0/488791414 >> v1:172.21.15.96:6789/0 conn(0x7f31dc1522f0 legacy=0x7f31dc0cf710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:59.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.863+0000 7f31d9e00640 1 -- 172.21.15.96:0/488791414 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31dc0b3e80 con 0x7f31dc0caa40 2024-05-28T06:04:59.865 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.864+0000 7f31d9e00640 1 --2- 172.21.15.96:0/488791414 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f31dc0caa40 0x7f31dc0ce960 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f31dc0b42e0 tx=0x7f31cc02a100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3174b9de06db0a33 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:59.865 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.864+0000 7f31d8a00640 1 -- 172.21.15.96:0/488791414 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31cc037070 con 0x7f31dc0caa40 2024-05-28T06:04:59.865 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.864+0000 7f31d8a00640 1 -- 172.21.15.96:0/488791414 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31cc02ae50 con 0x7f31dc0caa40 2024-05-28T06:04:59.865 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.864+0000 7f31d8a00640 1 -- 172.21.15.96:0/488791414 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31cc032640 con 0x7f31dc0caa40 2024-05-28T06:04:59.866 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.865+0000 7f31db200640 1 -- 172.21.15.96:0/488791414 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f31dc0caa40 msgr2=0x7f31dc0ce960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:59.866 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.865+0000 7f31db200640 1 --2- 172.21.15.96:0/488791414 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f31dc0caa40 0x7f31dc0ce960 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f31dc0b42e0 tx=0x7f31cc02a100 comp rx=0 tx=0).stop 2024-05-28T06:04:59.866 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.865+0000 7f31db200640 1 -- 172.21.15.96:0/488791414 shutdown_connections 2024-05-28T06:04:59.866 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.865+0000 7f31db200640 1 --2- 172.21.15.96:0/488791414 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f31dc0cae20 0x7f31dc151db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:59.866 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.865+0000 7f31db200640 1 --2- 172.21.15.96:0/488791414 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f31dc0caa40 0x7f31dc0ce960 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:59.866 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.865+0000 7f31db200640 1 -- 172.21.15.96:0/488791414 >> 172.21.15.96:0/488791414 conn(0x7f31dc06b900 msgr2=0x7f31dc06bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:04:59.866 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.865+0000 7f31db200640 1 -- 172.21.15.96:0/488791414 shutdown_connections 2024-05-28T06:04:59.867 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.865+0000 7f31db200640 1 -- 172.21.15.96:0/488791414 wait complete. 2024-05-28T06:04:59.867 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.867+0000 7f31db200640 1 Processor -- start 2024-05-28T06:04:59.867 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.867+0000 7f31db200640 1 -- start start 2024-05-28T06:04:59.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.868+0000 7f31db200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f31dc0caa40 0x7f31dc147d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:59.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.868+0000 7f31db200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f31dc0cae20 0x7f31dc06e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:59.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.868+0000 7f31db200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f31dc1522f0 0x7f31dc150060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:59.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.868+0000 7f31db200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f31dc0ce440 con 0x7f31dc0caa40 2024-05-28T06:04:59.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.868+0000 7f31db200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f31dc0ce160 con 0x7f31dc1522f0 2024-05-28T06:04:59.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.868+0000 7f31db200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f31dc0ceea0 con 0x7f31dc0cae20 2024-05-28T06:04:59.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.868+0000 7f31d9e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f31dc0caa40 0x7f31dc147d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:59.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.868+0000 7f31d9e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f31dc0caa40 0x7f31dc147d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54540/0 (socket says 172.21.15.96:54540) 2024-05-28T06:04:59.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.868+0000 7f31d9e00640 1 -- 172.21.15.96:0/704383102 learned_addr learned my addr 172.21.15.96:0/704383102 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:04:59.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.868+0000 7f31d9400640 1 --2- 172.21.15.96:0/704383102 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f31dc0cae20 0x7f31dc06e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:59.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.869+0000 7f31da800640 1 --2- 172.21.15.96:0/704383102 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f31dc1522f0 0x7f31dc150060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:59.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.869+0000 7f31d9e00640 1 -- 172.21.15.96:0/704383102 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f31dc0cae20 msgr2=0x7f31dc06e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:59.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.869+0000 7f31d9e00640 1 --2- 172.21.15.96:0/704383102 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f31dc0cae20 0x7f31dc06e1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:59.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.869+0000 7f31d9e00640 1 -- 172.21.15.96:0/704383102 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f31dc1522f0 msgr2=0x7f31dc150060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:04:59.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.869+0000 7f31d9e00640 1 --2- 172.21.15.96:0/704383102 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f31dc1522f0 0x7f31dc150060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:04:59.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.869+0000 7f31d9e00640 1 -- 172.21.15.96:0/704383102 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31dc06ec80 con 0x7f31dc0caa40 2024-05-28T06:04:59.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.869+0000 7f31da800640 1 --2- 172.21.15.96:0/704383102 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f31dc1522f0 0x7f31dc150060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:04:59.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.869+0000 7f31d9e00640 1 --2- 172.21.15.96:0/704383102 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f31dc0caa40 0x7f31dc147d60 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f31cc000c00 tx=0x7f31cc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:59.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.869+0000 7f31d2000640 1 -- 172.21.15.96:0/704383102 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31cc040020 con 0x7f31dc0caa40 2024-05-28T06:04:59.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.869+0000 7f31d2000640 1 -- 172.21.15.96:0/704383102 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31cc038040 con 0x7f31dc0caa40 2024-05-28T06:04:59.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.869+0000 7f31d2000640 1 -- 172.21.15.96:0/704383102 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31cc037070 con 0x7f31dc0caa40 2024-05-28T06:04:59.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.869+0000 7f31db200640 1 -- 172.21.15.96:0/704383102 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31dc06ef60 con 0x7f31dc0caa40 2024-05-28T06:04:59.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.869+0000 7f31db200640 1 -- 172.21.15.96:0/704383102 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31dc06f520 con 0x7f31dc0caa40 2024-05-28T06:04:59.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.871+0000 7f31db200640 1 -- 172.21.15.96:0/704383102 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31dc0689e0 con 0x7f31dc0caa40 2024-05-28T06:04:59.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.872+0000 7f31d2000640 1 -- 172.21.15.96:0/704383102 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f31cc043020 con 0x7f31dc0caa40 2024-05-28T06:04:59.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.872+0000 7f31d2000640 1 --2- 172.21.15.96:0/704383102 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f31bc06c8f0 0x7f31bc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:04:59.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.873+0000 7f31d2000640 1 -- 172.21.15.96:0/704383102 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5825+0+0 (secure 0 0 0) 0x7f31cc0abad0 con 0x7f31dc0caa40 2024-05-28T06:04:59.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.873+0000 7f31d9400640 1 --2- 172.21.15.96:0/704383102 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f31bc06c8f0 0x7f31bc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:04:59.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.874+0000 7f31d9400640 1 --2- 172.21.15.96:0/704383102 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f31bc06c8f0 0x7f31bc06edb0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f31dc15a080 tx=0x7f31c8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:04:59.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:04:59.877+0000 7f31d2000640 1 -- 172.21.15.96:0/704383102 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f31cc04a030 con 0x7f31dc0caa40 2024-05-28T06:05:00.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.076+0000 7f31db200640 1 -- 172.21.15.96:0/704383102 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f31dc15b050 con 0x7f31dc0caa40 2024-05-28T06:05:00.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.077+0000 7f31d2000640 1 -- 172.21.15.96:0/704383102 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) ==== 93+0+10 (secure 0 0 0) 0x7f31cc079e80 con 0x7f31dc0caa40 2024-05-28T06:05:00.078 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 1 2024-05-28T06:05:00.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.080+0000 7f31db200640 1 -- 172.21.15.96:0/704383102 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f31bc06c8f0 msgr2=0x7f31bc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:00.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.080+0000 7f31db200640 1 --2- 172.21.15.96:0/704383102 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f31bc06c8f0 0x7f31bc06edb0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f31dc15a080 tx=0x7f31c8008040 comp rx=0 tx=0).stop 2024-05-28T06:05:00.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.080+0000 7f31db200640 1 -- 172.21.15.96:0/704383102 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f31dc0caa40 msgr2=0x7f31dc147d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:00.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.080+0000 7f31db200640 1 --2- 172.21.15.96:0/704383102 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f31dc0caa40 0x7f31dc147d60 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f31cc000c00 tx=0x7f31cc0046c0 comp rx=0 tx=0).stop 2024-05-28T06:05:00.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.080+0000 7f31db200640 1 -- 172.21.15.96:0/704383102 shutdown_connections 2024-05-28T06:05:00.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.080+0000 7f31db200640 1 --2- 172.21.15.96:0/704383102 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f31bc06c8f0 0x7f31bc06edb0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:00.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.080+0000 7f31db200640 1 --2- 172.21.15.96:0/704383102 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f31dc1522f0 0x7f31dc150060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:00.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.080+0000 7f31db200640 1 --2- 172.21.15.96:0/704383102 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f31dc0cae20 0x7f31dc06e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:00.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.080+0000 7f31db200640 1 --2- 172.21.15.96:0/704383102 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f31dc0caa40 0x7f31dc147d60 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:00.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.080+0000 7f31db200640 1 -- 172.21.15.96:0/704383102 >> 172.21.15.96:0/704383102 conn(0x7f31dc06b900 msgr2=0x7f31dc1598e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:00.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.080+0000 7f31db200640 1 -- 172.21.15.96:0/704383102 shutdown_connections 2024-05-28T06:05:00.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.080+0000 7f31db200640 1 -- 172.21.15.96:0/704383102 wait complete. 2024-05-28T06:05:00.100 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-28T06:05:00.304 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.303+0000 7f03cbe00640 1 Processor -- start 2024-05-28T06:05:00.304 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.303+0000 7f03cbe00640 1 -- start start 2024-05-28T06:05:00.305 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.305+0000 7f03cbe00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f03cc0cd2a0 0x7f03cc0b4d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:00.305 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.305+0000 7f03cbe00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f03cc15cb60 0x7f03cc15cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:00.305 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.305+0000 7f03cbe00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03cc15d900 con 0x7f03cc153510 2024-05-28T06:05:00.305 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.305+0000 7f03cbe00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f03cc1538f0 con 0x7f03cc15cb60 2024-05-28T06:05:00.305 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.305+0000 7f03cbe00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f03cc0b48c0 con 0x7f03cc0cd2a0 2024-05-28T06:05:00.306 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.305+0000 7f03caa00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f03cc153510 0x7f03cc0ccb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:60434/0 (socket says 172.21.15.96:60434) 2024-05-28T06:05:00.306 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.305+0000 7f03caa00640 1 -- 172.21.15.96:0/1792363941 learned_addr learned my addr 172.21.15.96:0/1792363941 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:05:00.307 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.305+0000 7f03ca000640 1 --2- 172.21.15.96:0/1792363941 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f03cc0cd2a0 0x7f03cc0b4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:00.308 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.306+0000 7f03cb400640 1 --2- 172.21.15.96:0/1792363941 >> v2:172.21.15.123:3300/0 conn(0x7f03cc15cb60 0x7f03cc15cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:00.308 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.306+0000 7f03c9600640 1 -- 172.21.15.96:0/1792363941 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 706599060 0 0) 0x7f03cc15d900 con 0x7f03cc153510 2024-05-28T06:05:00.308 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.306+0000 7f03c9600640 1 -- 172.21.15.96:0/1792363941 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f03b0003520 con 0x7f03cc153510 2024-05-28T06:05:00.308 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.306+0000 7f03c9600640 1 -- 172.21.15.96:0/1792363941 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2204502575 0 0) 0x7f03b0003520 con 0x7f03cc153510 2024-05-28T06:05:00.308 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.306+0000 7f03c9600640 1 -- 172.21.15.96:0/1792363941 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f03cc0cd2a0 msgr2=0x7f03cc0b4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:00.308 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.306+0000 7f03c9600640 1 --2- 172.21.15.96:0/1792363941 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f03cc0cd2a0 0x7f03cc0b4d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:00.308 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.306+0000 7f03c9600640 1 -- 172.21.15.96:0/1792363941 >> v2:172.21.15.123:3300/0 conn(0x7f03cc15cb60 msgr2=0x7f03cc15cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:00.308 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.306+0000 7f03c9600640 1 --2- 172.21.15.96:0/1792363941 >> v2:172.21.15.123:3300/0 conn(0x7f03cc15cb60 0x7f03cc15cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:00.308 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.306+0000 7f03c9600640 1 -- 172.21.15.96:0/1792363941 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03cc0b3f90 con 0x7f03cc153510 2024-05-28T06:05:00.308 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.306+0000 7f03c9600640 1 -- 172.21.15.96:0/1792363941 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f03bc001670 con 0x7f03cc153510 2024-05-28T06:05:00.308 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.306+0000 7f03c9600640 1 -- 172.21.15.96:0/1792363941 >> v1:172.21.15.96:6789/0 conn(0x7f03cc153510 legacy=0x7f03cc0ccb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:00.308 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.306+0000 7f03c9600640 1 --2- 172.21.15.96:0/1792363941 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f03b00039b0 0x7f03b0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:00.308 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.306+0000 7f03c9600640 1 --2- 172.21.15.96:0/1792363941 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f03b0004800 0x7f03b0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:00.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.306+0000 7f03c9600640 1 --2- 172.21.15.96:0/1792363941 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f03cc0cd2a0 0x7f03b00051e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:00.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.306+0000 7f03c9600640 1 -- 172.21.15.96:0/1792363941 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f03cc0b48c0 con 0x7f03b00039b0 2024-05-28T06:05:00.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.306+0000 7f03c9600640 1 -- 172.21.15.96:0/1792363941 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f03b000bad0 con 0x7f03b0004800 2024-05-28T06:05:00.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.306+0000 7f03c9600640 1 -- 172.21.15.96:0/1792363941 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f03b000bcd0 con 0x7f03cc0cd2a0 2024-05-28T06:05:00.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.306+0000 7f03caa00640 1 --2- 172.21.15.96:0/1792363941 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f03b00039b0 0x7f03b0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:00.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.307+0000 7f03ca000640 1 --2- 172.21.15.96:0/1792363941 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f03b0004800 0x7f03b0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:00.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.307+0000 7f03cb400640 1 --2- 172.21.15.96:0/1792363941 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f03cc0cd2a0 0x7f03b00051e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:00.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.307+0000 7f03caa00640 1 -- 172.21.15.96:0/1792363941 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f03cc0cd2a0 msgr2=0x7f03b00051e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:00.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.307+0000 7f03caa00640 1 --2- 172.21.15.96:0/1792363941 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f03cc0cd2a0 0x7f03b00051e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:00.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.307+0000 7f03caa00640 1 -- 172.21.15.96:0/1792363941 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f03b0004800 msgr2=0x7f03b0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:00.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.307+0000 7f03caa00640 1 --2- 172.21.15.96:0/1792363941 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f03b0004800 0x7f03b0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:00.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.307+0000 7f03caa00640 1 -- 172.21.15.96:0/1792363941 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03b0003520 con 0x7f03b00039b0 2024-05-28T06:05:00.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.307+0000 7f03cb400640 1 --2- 172.21.15.96:0/1792363941 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f03cc0cd2a0 0x7f03b00051e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:05:00.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.308+0000 7f03caa00640 1 --2- 172.21.15.96:0/1792363941 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f03b00039b0 0x7f03b0003dd0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f03bc00c160 tx=0x7f03b0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ceb9482a918b8a8 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:00.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.308+0000 7f03c9600640 1 -- 172.21.15.96:0/1792363941 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03bc007c80 con 0x7f03b00039b0 2024-05-28T06:05:00.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.308+0000 7f03c9600640 1 -- 172.21.15.96:0/1792363941 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03bc005ad0 con 0x7f03b00039b0 2024-05-28T06:05:00.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.308+0000 7f03c9600640 1 -- 172.21.15.96:0/1792363941 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03bc007070 con 0x7f03b00039b0 2024-05-28T06:05:00.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.309+0000 7f03cbe00640 1 -- 172.21.15.96:0/1792363941 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f03b00039b0 msgr2=0x7f03b0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:00.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.309+0000 7f03cbe00640 1 --2- 172.21.15.96:0/1792363941 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f03b00039b0 0x7f03b0003dd0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f03bc00c160 tx=0x7f03b0005830 comp rx=0 tx=0).stop 2024-05-28T06:05:00.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.309+0000 7f03cb400640 1 -- 172.21.15.96:0/1792363941 reap_dead start 2024-05-28T06:05:00.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.309+0000 7f03cbe00640 1 -- 172.21.15.96:0/1792363941 shutdown_connections 2024-05-28T06:05:00.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.309+0000 7f03cbe00640 1 -- 172.21.15.96:0/1792363941 >> 172.21.15.96:0/1792363941 conn(0x7f03cc06b900 msgr2=0x7f03cc06bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:00.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.309+0000 7f03cbe00640 1 -- 172.21.15.96:0/1792363941 shutdown_connections 2024-05-28T06:05:00.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.309+0000 7f03cbe00640 1 -- 172.21.15.96:0/1792363941 wait complete. 2024-05-28T06:05:00.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.311+0000 7f03cbe00640 1 Processor -- start 2024-05-28T06:05:00.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.311+0000 7f03cbe00640 1 -- start start 2024-05-28T06:05:00.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.312+0000 7f03cbe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f03b00039b0 0x7f03cc0727c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:00.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.312+0000 7f03cbe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f03cc0ad130 0x7f03cc0ad5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:00.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.312+0000 7f03cbe00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f03cc152280 0x7f03cc152700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:00.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.312+0000 7f03cbe00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f03cc15d510 con 0x7f03cc0ad130 2024-05-28T06:05:00.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.312+0000 7f03cbe00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f03cc0b4550 con 0x7f03b00039b0 2024-05-28T06:05:00.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.312+0000 7f03cbe00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f03cc15d690 con 0x7f03cc152280 2024-05-28T06:05:00.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.312+0000 7f03ca000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f03cc0ad130 0x7f03cc0ad5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:00.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.312+0000 7f03ca000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f03cc0ad130 0x7f03cc0ad5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54556/0 (socket says 172.21.15.96:54556) 2024-05-28T06:05:00.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.312+0000 7f03ca000640 1 -- 172.21.15.96:0/3476462247 learned_addr learned my addr 172.21.15.96:0/3476462247 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:00.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.313+0000 7f03cb400640 1 --2- 172.21.15.96:0/3476462247 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f03cc152280 0x7f03cc152700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:00.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.313+0000 7f03caa00640 1 --2- 172.21.15.96:0/3476462247 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f03b00039b0 0x7f03cc0727c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:00.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.313+0000 7f03ca000640 1 -- 172.21.15.96:0/3476462247 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f03cc152280 msgr2=0x7f03cc152700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:00.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.313+0000 7f03ca000640 1 --2- 172.21.15.96:0/3476462247 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f03cc152280 0x7f03cc152700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:00.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.313+0000 7f03ca000640 1 -- 172.21.15.96:0/3476462247 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f03b00039b0 msgr2=0x7f03cc0727c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:00.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.313+0000 7f03ca000640 1 --2- 172.21.15.96:0/3476462247 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f03b00039b0 0x7f03cc0727c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:00.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.313+0000 7f03ca000640 1 -- 172.21.15.96:0/3476462247 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03cc0adaf0 con 0x7f03cc0ad130 2024-05-28T06:05:00.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.313+0000 7f03ca000640 1 --2- 172.21.15.96:0/3476462247 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f03cc0ad130 0x7f03cc0ad5b0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f03b800f5b0 tx=0x7f03b800fa70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:00.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.313+0000 7f03c2a00640 1 -- 172.21.15.96:0/3476462247 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03b8019020 con 0x7f03cc0ad130 2024-05-28T06:05:00.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.313+0000 7f03c2a00640 1 -- 172.21.15.96:0/3476462247 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03b8014870 con 0x7f03cc0ad130 2024-05-28T06:05:00.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.313+0000 7f03cbe00640 1 -- 172.21.15.96:0/3476462247 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03cc152c40 con 0x7f03cc0ad130 2024-05-28T06:05:00.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.313+0000 7f03c2a00640 1 -- 172.21.15.96:0/3476462247 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03b8014c60 con 0x7f03cc0ad130 2024-05-28T06:05:00.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.313+0000 7f03cbe00640 1 -- 172.21.15.96:0/3476462247 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03cc0ae060 con 0x7f03cc0ad130 2024-05-28T06:05:00.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.315+0000 7f03cbe00640 1 -- 172.21.15.96:0/3476462247 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03cc0689e0 con 0x7f03cc0ad130 2024-05-28T06:05:00.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.316+0000 7f03c2a00640 1 -- 172.21.15.96:0/3476462247 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f03b801c020 con 0x7f03cc0ad130 2024-05-28T06:05:00.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.317+0000 7f03c2a00640 1 --2- 172.21.15.96:0/3476462247 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f03ac06c920 0x7f03ac06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:00.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.317+0000 7f03c2a00640 1 -- 172.21.15.96:0/3476462247 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5825+0+0 (secure 0 0 0) 0x7f03b808c410 con 0x7f03cc0ad130 2024-05-28T06:05:00.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.318+0000 7f03caa00640 1 --2- 172.21.15.96:0/3476462247 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f03ac06c920 0x7f03ac06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:00.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.319+0000 7f03caa00640 1 --2- 172.21.15.96:0/3476462247 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f03ac06c920 0x7f03ac06ede0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f03bc032000 tx=0x7f03bc007000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:00.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.322+0000 7f03c2a00640 1 -- 172.21.15.96:0/3476462247 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f03b8023020 con 0x7f03cc0ad130 2024-05-28T06:05:00.524 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.524+0000 7f03cbe00640 1 -- 172.21.15.96:0/3476462247 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f03cc160df0 con 0x7f03cc0ad130 2024-05-28T06:05:00.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.525+0000 7f03c2a00640 1 -- 172.21.15.96:0/3476462247 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) ==== 104+0+11 (secure 0 0 0) 0x7f03b8026350 con 0x7f03cc0ad130 2024-05-28T06:05:00.526 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 32 2024-05-28T06:05:00.529 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.528+0000 7f03cbe00640 1 -- 172.21.15.96:0/3476462247 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f03ac06c920 msgr2=0x7f03ac06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:00.529 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.528+0000 7f03cbe00640 1 --2- 172.21.15.96:0/3476462247 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f03ac06c920 0x7f03ac06ede0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f03bc032000 tx=0x7f03bc007000 comp rx=0 tx=0).stop 2024-05-28T06:05:00.529 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.528+0000 7f03cbe00640 1 -- 172.21.15.96:0/3476462247 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f03cc0ad130 msgr2=0x7f03cc0ad5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:00.529 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.528+0000 7f03cbe00640 1 --2- 172.21.15.96:0/3476462247 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f03cc0ad130 0x7f03cc0ad5b0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f03b800f5b0 tx=0x7f03b800fa70 comp rx=0 tx=0).stop 2024-05-28T06:05:00.529 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.528+0000 7f03cbe00640 1 -- 172.21.15.96:0/3476462247 shutdown_connections 2024-05-28T06:05:00.529 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.528+0000 7f03cbe00640 1 --2- 172.21.15.96:0/3476462247 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f03ac06c920 0x7f03ac06ede0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:00.529 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.528+0000 7f03cbe00640 1 --2- 172.21.15.96:0/3476462247 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f03cc152280 0x7f03cc152700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:00.530 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.528+0000 7f03cbe00640 1 --2- 172.21.15.96:0/3476462247 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f03cc0ad130 0x7f03cc0ad5b0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:00.530 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.528+0000 7f03cbe00640 1 --2- 172.21.15.96:0/3476462247 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f03b00039b0 0x7f03cc0727c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:00.530 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.528+0000 7f03cbe00640 1 -- 172.21.15.96:0/3476462247 >> 172.21.15.96:0/3476462247 conn(0x7f03cc06b900 msgr2=0x7f03cc0ccb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:00.530 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.528+0000 7f03cbe00640 1 -- 172.21.15.96:0/3476462247 shutdown_connections 2024-05-28T06:05:00.530 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.529+0000 7f03cbe00640 1 -- 172.21.15.96:0/3476462247 wait complete. 2024-05-28T06:05:00.547 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-28T06:05:00.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.751+0000 7f5f81000640 1 Processor -- start 2024-05-28T06:05:00.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.751+0000 7f5f81000640 1 -- start start 2024-05-28T06:05:00.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.753+0000 7f5f81000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f5f7c15abe0 0x7f5f7c06f380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:00.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.753+0000 7f5f81000640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f7c06e2e0 con 0x7f5f7c06f8c0 2024-05-28T06:05:00.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.753+0000 7f5f81000640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f7c06e5c0 con 0x7f5f7c0725c0 2024-05-28T06:05:00.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.753+0000 7f5f81000640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f5f7c06e8a0 con 0x7f5f7c15abe0 2024-05-28T06:05:00.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.753+0000 7f5f79400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f5f7c06f8c0 0x7f5f7c0b45d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:60440/0 (socket says 172.21.15.96:60440) 2024-05-28T06:05:00.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.753+0000 7f5f79400640 1 -- 172.21.15.96:0/3606703160 learned_addr learned my addr 172.21.15.96:0/3606703160 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:05:00.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.754+0000 7f5f79e00640 1 --2- 172.21.15.96:0/3606703160 >> v2:172.21.15.123:3300/0 conn(0x7f5f7c15abe0 0x7f5f7c06f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:00.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.754+0000 7f5f78a00640 1 -- 172.21.15.96:0/3606703160 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3072109421 0 0) 0x7f5f7c06e2e0 con 0x7f5f7c06f8c0 2024-05-28T06:05:00.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.754+0000 7f5f78a00640 1 -- 172.21.15.96:0/3606703160 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5f5c003520 con 0x7f5f7c06f8c0 2024-05-28T06:05:00.757 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.754+0000 7f5f78a00640 1 -- 172.21.15.96:0/3606703160 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1347308350 0 0) 0x7f5f7c06e5c0 con 0x7f5f7c0725c0 2024-05-28T06:05:00.757 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.754+0000 7f5f78a00640 1 -- 172.21.15.96:0/3606703160 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5f7c06e2e0 con 0x7f5f7c0725c0 2024-05-28T06:05:00.757 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.754+0000 7f5f78a00640 1 -- 172.21.15.96:0/3606703160 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 614797252 0 0) 0x7f5f5c003520 con 0x7f5f7c06f8c0 2024-05-28T06:05:00.757 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.754+0000 7f5f78a00640 1 -- 172.21.15.96:0/3606703160 >> v2:172.21.15.123:3300/0 conn(0x7f5f7c15abe0 msgr2=0x7f5f7c06f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:00.757 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.754+0000 7f5f78a00640 1 --2- 172.21.15.96:0/3606703160 >> v2:172.21.15.123:3300/0 conn(0x7f5f7c15abe0 0x7f5f7c06f380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:00.757 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.754+0000 7f5f78a00640 1 -- 172.21.15.96:0/3606703160 >> v1:172.21.15.123:6789/0 conn(0x7f5f7c0725c0 legacy=0x7f5f7c0ad680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:00.757 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.754+0000 7f5f78a00640 1 -- 172.21.15.96:0/3606703160 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f7c06eb80 con 0x7f5f7c06f8c0 2024-05-28T06:05:00.757 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.754+0000 7f5f78a00640 1 -- 172.21.15.96:0/3606703160 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f5f640019f0 con 0x7f5f7c06f8c0 2024-05-28T06:05:00.757 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.754+0000 7f5f78a00640 1 -- 172.21.15.96:0/3606703160 >> v1:172.21.15.96:6789/0 conn(0x7f5f7c06f8c0 legacy=0x7f5f7c0b45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:00.757 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.754+0000 7f5f78a00640 1 --2- 172.21.15.96:0/3606703160 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5f5c003b90 0x7f5f5c003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:00.757 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.754+0000 7f5f78a00640 1 --2- 172.21.15.96:0/3606703160 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f5c004800 0x7f5f5c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:00.757 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.754+0000 7f5f78a00640 1 --2- 172.21.15.96:0/3606703160 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5f5c0051c0 0x7f5f5c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:00.757 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.754+0000 7f5f78a00640 1 -- 172.21.15.96:0/3606703160 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f5f7c06e8a0 con 0x7f5f5c0051c0 2024-05-28T06:05:00.757 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.754+0000 7f5f78a00640 1 -- 172.21.15.96:0/3606703160 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5f5c00e2a0 con 0x7f5f5c004800 2024-05-28T06:05:00.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.754+0000 7f5f78a00640 1 -- 172.21.15.96:0/3606703160 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f5f5c00e4a0 con 0x7f5f5c003b90 2024-05-28T06:05:00.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.755+0000 7f5f7a800640 1 --2- 172.21.15.96:0/3606703160 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5f5c0051c0 0x7f5f5c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:00.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.755+0000 7f5f79400640 1 --2- 172.21.15.96:0/3606703160 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f5c004800 0x7f5f5c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:00.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.755+0000 7f5f79e00640 1 --2- 172.21.15.96:0/3606703160 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5f5c003b90 0x7f5f5c003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:00.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.755+0000 7f5f7a800640 1 -- 172.21.15.96:0/3606703160 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5f5c003b90 msgr2=0x7f5f5c003f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:00.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.755+0000 7f5f7a800640 1 --2- 172.21.15.96:0/3606703160 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5f5c003b90 0x7f5f5c003f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:00.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.755+0000 7f5f7a800640 1 -- 172.21.15.96:0/3606703160 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f5c004800 msgr2=0x7f5f5c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:00.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.755+0000 7f5f7a800640 1 --2- 172.21.15.96:0/3606703160 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f5c004800 0x7f5f5c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:00.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.755+0000 7f5f7a800640 1 -- 172.21.15.96:0/3606703160 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f5c003520 con 0x7f5f5c0051c0 2024-05-28T06:05:00.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.755+0000 7f5f79e00640 1 --2- 172.21.15.96:0/3606703160 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5f5c003b90 0x7f5f5c003f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:05:00.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.756+0000 7f5f7a800640 1 -- 172.21.15.96:0/3606703160 reap_dead start 2024-05-28T06:05:00.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.756+0000 7f5f7a800640 1 --2- 172.21.15.96:0/3606703160 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5f5c0051c0 0x7f5f5c00b9c0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f5f740018d0 tx=0x7f5f5c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db450391b2ca2f9c server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:00.760 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.756+0000 7f5f78a00640 1 -- 172.21.15.96:0/3606703160 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f740014f0 con 0x7f5f5c0051c0 2024-05-28T06:05:00.760 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.757+0000 7f5f78a00640 1 -- 172.21.15.96:0/3606703160 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f740052d0 con 0x7f5f5c0051c0 2024-05-28T06:05:00.760 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.757+0000 7f5f78a00640 1 -- 172.21.15.96:0/3606703160 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f7402acc0 con 0x7f5f5c0051c0 2024-05-28T06:05:00.760 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.757+0000 7f5f81000640 1 -- 172.21.15.96:0/3606703160 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5f5c0051c0 msgr2=0x7f5f5c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:00.760 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.757+0000 7f5f81000640 1 --2- 172.21.15.96:0/3606703160 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5f5c0051c0 0x7f5f5c00b9c0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f5f740018d0 tx=0x7f5f5c005830 comp rx=0 tx=0).stop 2024-05-28T06:05:00.760 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.757+0000 7f5f81000640 1 -- 172.21.15.96:0/3606703160 shutdown_connections 2024-05-28T06:05:00.760 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.757+0000 7f5f81000640 1 --2- 172.21.15.96:0/3606703160 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5f5c0051c0 0x7f5f5c00b9c0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:00.760 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.757+0000 7f5f81000640 1 -- 172.21.15.96:0/3606703160 >> 172.21.15.96:0/3606703160 conn(0x7f5f7c06bd20 msgr2=0x7f5f7c06c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:00.760 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.757+0000 7f5f81000640 1 -- 172.21.15.96:0/3606703160 shutdown_connections 2024-05-28T06:05:00.760 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.757+0000 7f5f81000640 1 -- 172.21.15.96:0/3606703160 wait complete. 2024-05-28T06:05:00.760 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.759+0000 7f5f81000640 1 Processor -- start 2024-05-28T06:05:00.760 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.759+0000 7f5f81000640 1 -- start start 2024-05-28T06:05:00.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.760+0000 7f5f81000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5f5c0051c0 0x7f5f7c0ae300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:00.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.760+0000 7f5f81000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f7c1f41a0 0x7f5f7c159010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:00.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.760+0000 7f5f81000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5f7c159550 0x7f5f7c09d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:00.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.760+0000 7f5f81000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f5f7c06df60 con 0x7f5f5c0051c0 2024-05-28T06:05:00.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.760+0000 7f5f81000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5f7c15afc0 con 0x7f5f7c1f41a0 2024-05-28T06:05:00.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.760+0000 7f5f81000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f5f7c06f380 con 0x7f5f7c159550 2024-05-28T06:05:00.763 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.760+0000 7f5f79e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5f5c0051c0 0x7f5f7c0ae300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:00.763 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.760+0000 7f5f79e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5f5c0051c0 0x7f5f7c0ae300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54580/0 (socket says 172.21.15.96:54580) 2024-05-28T06:05:00.763 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.760+0000 7f5f79e00640 1 -- 172.21.15.96:0/2162147775 learned_addr learned my addr 172.21.15.96:0/2162147775 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:00.763 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.761+0000 7f5f79400640 1 --2- 172.21.15.96:0/2162147775 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f7c1f41a0 0x7f5f7c159010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:00.763 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.761+0000 7f5f79e00640 1 -- 172.21.15.96:0/2162147775 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5f7c159550 msgr2=0x7f5f7c09d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:00.763 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.761+0000 7f5f79e00640 1 --2- 172.21.15.96:0/2162147775 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5f7c159550 0x7f5f7c09d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:00.763 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.761+0000 7f5f79e00640 1 -- 172.21.15.96:0/2162147775 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f7c1f41a0 msgr2=0x7f5f7c159010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:00.763 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.761+0000 7f5f79e00640 1 --2- 172.21.15.96:0/2162147775 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f7c1f41a0 0x7f5f7c159010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:00.763 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.761+0000 7f5f79e00640 1 -- 172.21.15.96:0/2162147775 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f7c159cd0 con 0x7f5f5c0051c0 2024-05-28T06:05:00.763 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.761+0000 7f5f79400640 1 --2- 172.21.15.96:0/2162147775 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f7c1f41a0 0x7f5f7c159010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:05:00.763 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.761+0000 7f5f79e00640 1 --2- 172.21.15.96:0/2162147775 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5f5c0051c0 0x7f5f7c0ae300 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f5f6c0108b0 tx=0x7f5f6c010d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:00.763 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.761+0000 7f5f72000640 1 -- 172.21.15.96:0/2162147775 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f6c00f040 con 0x7f5f5c0051c0 2024-05-28T06:05:00.763 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.761+0000 7f5f72000640 1 -- 172.21.15.96:0/2162147775 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f6c00eb60 con 0x7f5f5c0051c0 2024-05-28T06:05:00.764 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.761+0000 7f5f72000640 1 -- 172.21.15.96:0/2162147775 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f6c01c400 con 0x7f5f5c0051c0 2024-05-28T06:05:00.764 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.761+0000 7f5f81000640 1 -- 172.21.15.96:0/2162147775 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f7c1f36c0 con 0x7f5f5c0051c0 2024-05-28T06:05:00.764 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.761+0000 7f5f81000640 1 -- 172.21.15.96:0/2162147775 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f7c1f3c80 con 0x7f5f5c0051c0 2024-05-28T06:05:00.764 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.763+0000 7f5f81000640 1 -- 172.21.15.96:0/2162147775 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f7c159a30 con 0x7f5f5c0051c0 2024-05-28T06:05:00.764 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.764+0000 7f5f72000640 1 -- 172.21.15.96:0/2162147775 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5f6c01c5a0 con 0x7f5f5c0051c0 2024-05-28T06:05:00.764 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.764+0000 7f5f72000640 1 --2- 172.21.15.96:0/2162147775 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5f5806c920 0x7f5f5806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:00.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.765+0000 7f5f72000640 1 -- 172.21.15.96:0/2162147775 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5825+0+0 (secure 0 0 0) 0x7f5f6c08c370 con 0x7f5f5c0051c0 2024-05-28T06:05:00.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.765+0000 7f5f79400640 1 --2- 172.21.15.96:0/2162147775 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5f5806c920 0x7f5f5806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:00.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.766+0000 7f5f79400640 1 --2- 172.21.15.96:0/2162147775 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5f5806c920 0x7f5f5806ede0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f5f64001b60 tx=0x7f5f64001250 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:00.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.769+0000 7f5f72000640 1 -- 172.21.15.96:0/2162147775 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5f6c018030 con 0x7f5f5c0051c0 2024-05-28T06:05:00.974 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.974+0000 7f5f81000640 1 -- 172.21.15.96:0/2162147775 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f5f7c0689e0 con 0x7f5f5c0051c0 2024-05-28T06:05:00.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.974+0000 7f5f72000640 1 -- 172.21.15.96:0/2162147775 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) ==== 100+0+11 (secure 0 0 0) 0x7f5f6c05a030 con 0x7f5f5c0051c0 2024-05-28T06:05:00.975 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 64 2024-05-28T06:05:00.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.977+0000 7f5f81000640 1 -- 172.21.15.96:0/2162147775 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5f5806c920 msgr2=0x7f5f5806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:00.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.977+0000 7f5f81000640 1 --2- 172.21.15.96:0/2162147775 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5f5806c920 0x7f5f5806ede0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f5f64001b60 tx=0x7f5f64001250 comp rx=0 tx=0).stop 2024-05-28T06:05:00.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.977+0000 7f5f81000640 1 -- 172.21.15.96:0/2162147775 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5f5c0051c0 msgr2=0x7f5f7c0ae300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:00.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.977+0000 7f5f81000640 1 --2- 172.21.15.96:0/2162147775 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5f5c0051c0 0x7f5f7c0ae300 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f5f6c0108b0 tx=0x7f5f6c010d70 comp rx=0 tx=0).stop 2024-05-28T06:05:00.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.977+0000 7f5f81000640 1 -- 172.21.15.96:0/2162147775 shutdown_connections 2024-05-28T06:05:00.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.977+0000 7f5f81000640 1 --2- 172.21.15.96:0/2162147775 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5f5806c920 0x7f5f5806ede0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:00.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.977+0000 7f5f81000640 1 --2- 172.21.15.96:0/2162147775 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5f7c159550 0x7f5f7c09d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:00.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.977+0000 7f5f81000640 1 --2- 172.21.15.96:0/2162147775 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f7c1f41a0 0x7f5f7c159010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:00.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.977+0000 7f5f81000640 1 --2- 172.21.15.96:0/2162147775 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5f5c0051c0 0x7f5f7c0ae300 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:00.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.977+0000 7f5f81000640 1 -- 172.21.15.96:0/2162147775 >> 172.21.15.96:0/2162147775 conn(0x7f5f7c06bd20 msgr2=0x7f5f7c15f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:00.979 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.977+0000 7f5f81000640 1 -- 172.21.15.96:0/2162147775 shutdown_connections 2024-05-28T06:05:00.979 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:00.977+0000 7f5f81000640 1 -- 172.21.15.96:0/2162147775 wait complete. 2024-05-28T06:05:00.997 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-28T06:05:01.196 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.195+0000 7f3703c00640 1 Processor -- start 2024-05-28T06:05:01.196 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.196+0000 7f3703c00640 1 -- start start 2024-05-28T06:05:01.198 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.197+0000 7f3703c00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f36fc0d2db0 0x7f36fc0b3b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:01.198 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.197+0000 7f3703c00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f36fc0d4d60 0x7f36fc0d3190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:01.198 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.197+0000 7f3703c00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f36fc0d36d0 con 0x7f36fc0cf490 2024-05-28T06:05:01.198 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.197+0000 7f3703c00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f36fc0d5540 con 0x7f36fc0d2db0 2024-05-28T06:05:01.198 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.197+0000 7f3703c00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f36fc163500 con 0x7f36fc0d4d60 2024-05-28T06:05:01.198 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.198+0000 7f3701200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f36fc0d2db0 0x7f36fc0b3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:01.198 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.198+0000 7f3701200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f36fc0d2db0 0x7f36fc0b3b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54592/0 (socket says 172.21.15.96:54592) 2024-05-28T06:05:01.198 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.198+0000 7f3701200640 1 -- 172.21.15.96:0/104953755 learned_addr learned my addr 172.21.15.96:0/104953755 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:01.199 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.198+0000 7f3701200640 1 -- 172.21.15.96:0/104953755 >> v2:172.21.15.123:3300/0 conn(0x7f36fc0d4d60 msgr2=0x7f36fc0d3190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:01.199 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.198+0000 7f36fbe00640 1 --2- 172.21.15.96:0/104953755 >> v2:172.21.15.123:3300/0 conn(0x7f36fc0d4d60 0x7f36fc0d3190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:01.199 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.198+0000 7f3701200640 1 --2- 172.21.15.96:0/104953755 >> v2:172.21.15.123:3300/0 conn(0x7f36fc0d4d60 0x7f36fc0d3190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:01.199 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.198+0000 7f3701200640 1 -- 172.21.15.96:0/104953755 >> v1:172.21.15.123:6789/0 conn(0x7f36fc0cf490 legacy=0x7f36fc0cf920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:01.199 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.198+0000 7f3701200640 1 -- 172.21.15.96:0/104953755 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36fc162740 con 0x7f36fc0d2db0 2024-05-28T06:05:01.200 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.199+0000 7f3701200640 1 --2- 172.21.15.96:0/104953755 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f36fc0d2db0 0x7f36fc0b3b50 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f36fc0b71b0 tx=0x7f36ec02a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d999cc9e1a2db103 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:01.200 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.199+0000 7f36fb400640 1 -- 172.21.15.96:0/104953755 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36ec036070 con 0x7f36fc0d2db0 2024-05-28T06:05:01.200 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.200+0000 7f36fb400640 1 -- 172.21.15.96:0/104953755 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36ec031600 con 0x7f36fc0d2db0 2024-05-28T06:05:01.200 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.200+0000 7f36fb400640 1 -- 172.21.15.96:0/104953755 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36ec0319d0 con 0x7f36fc0d2db0 2024-05-28T06:05:01.200 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.200+0000 7f3703c00640 1 -- 172.21.15.96:0/104953755 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f36fc0d2db0 msgr2=0x7f36fc0b3b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:01.200 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.200+0000 7f3703c00640 1 --2- 172.21.15.96:0/104953755 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f36fc0d2db0 0x7f36fc0b3b50 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f36fc0b71b0 tx=0x7f36ec02a660 comp rx=0 tx=0).stop 2024-05-28T06:05:01.201 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.200+0000 7f3703c00640 1 -- 172.21.15.96:0/104953755 shutdown_connections 2024-05-28T06:05:01.201 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.200+0000 7f3703c00640 1 --2- 172.21.15.96:0/104953755 >> v2:172.21.15.123:3300/0 conn(0x7f36fc0d4d60 0x7f36fc0d3190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:01.201 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.200+0000 7f3703c00640 1 --2- 172.21.15.96:0/104953755 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f36fc0d2db0 0x7f36fc0b3b50 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:01.201 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.200+0000 7f3703c00640 1 -- 172.21.15.96:0/104953755 >> 172.21.15.96:0/104953755 conn(0x7f36fc08bfc0 msgr2=0x7f36fc0d4650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:01.201 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.200+0000 7f3703c00640 1 -- 172.21.15.96:0/104953755 shutdown_connections 2024-05-28T06:05:01.201 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.200+0000 7f3703c00640 1 -- 172.21.15.96:0/104953755 wait complete. 2024-05-28T06:05:01.202 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.202+0000 7f3703c00640 1 Processor -- start 2024-05-28T06:05:01.202 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.202+0000 7f3703c00640 1 -- start start 2024-05-28T06:05:01.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.203+0000 7f3703c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f36fc0cf490 0x7f36fc0b9ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:01.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.203+0000 7f3703c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f36fc0d2db0 0x7f36fc1527a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:01.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.203+0000 7f3703c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f36fc0d4d60 0x7f36fc0c8180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:01.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.203+0000 7f3703c00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f36fc162fe0 con 0x7f36fc0d2db0 2024-05-28T06:05:01.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.203+0000 7f3703c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f36fc162d00 con 0x7f36fc0cf490 2024-05-28T06:05:01.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.203+0000 7f3703c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f36fc0d51a0 con 0x7f36fc0d4d60 2024-05-28T06:05:01.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.203+0000 7f36fbe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f36fc0d2db0 0x7f36fc1527a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:01.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.203+0000 7f36fbe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f36fc0d2db0 0x7f36fc1527a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54598/0 (socket says 172.21.15.96:54598) 2024-05-28T06:05:01.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.203+0000 7f36fbe00640 1 -- 172.21.15.96:0/705924371 learned_addr learned my addr 172.21.15.96:0/705924371 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:01.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.204+0000 7f3701200640 1 --2- 172.21.15.96:0/705924371 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f36fc0cf490 0x7f36fc0b9ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:01.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.204+0000 7f36fbe00640 1 -- 172.21.15.96:0/705924371 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f36fc0d4d60 msgr2=0x7f36fc0c8180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:01.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.204+0000 7f36fbe00640 1 --2- 172.21.15.96:0/705924371 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f36fc0d4d60 0x7f36fc0c8180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:01.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.204+0000 7f36fbe00640 1 -- 172.21.15.96:0/705924371 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f36fc0cf490 msgr2=0x7f36fc0b9ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:01.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.204+0000 7f36fbe00640 1 --2- 172.21.15.96:0/705924371 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f36fc0cf490 0x7f36fc0b9ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:01.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.204+0000 7f36fbe00640 1 -- 172.21.15.96:0/705924371 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36fc0ce100 con 0x7f36fc0d2db0 2024-05-28T06:05:01.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.204+0000 7f36fbe00640 1 --2- 172.21.15.96:0/705924371 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f36fc0d2db0 0x7f36fc1527a0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f36e400c820 tx=0x7f36e400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:01.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.204+0000 7f36f8c00640 1 -- 172.21.15.96:0/705924371 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36e4015070 con 0x7f36fc0d2db0 2024-05-28T06:05:01.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.204+0000 7f36f8c00640 1 -- 172.21.15.96:0/705924371 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36e4004240 con 0x7f36fc0d2db0 2024-05-28T06:05:01.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.204+0000 7f36f8c00640 1 -- 172.21.15.96:0/705924371 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36e400ce10 con 0x7f36fc0d2db0 2024-05-28T06:05:01.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.204+0000 7f3703c00640 1 -- 172.21.15.96:0/705924371 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36fc0b40b0 con 0x7f36fc0d2db0 2024-05-28T06:05:01.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.205+0000 7f3703c00640 1 -- 172.21.15.96:0/705924371 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36fc0cde20 con 0x7f36fc0d2db0 2024-05-28T06:05:01.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.206+0000 7f3703c00640 1 -- 172.21.15.96:0/705924371 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36fc0689e0 con 0x7f36fc0d2db0 2024-05-28T06:05:01.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.207+0000 7f36f8c00640 1 -- 172.21.15.96:0/705924371 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f36e4007830 con 0x7f36fc0d2db0 2024-05-28T06:05:01.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.207+0000 7f36f8c00640 1 --2- 172.21.15.96:0/705924371 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f36d806c8f0 0x7f36d806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:01.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.208+0000 7f36f8c00640 1 -- 172.21.15.96:0/705924371 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5825+0+0 (secure 0 0 0) 0x7f36e408bac0 con 0x7f36fc0d2db0 2024-05-28T06:05:01.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.208+0000 7f3701200640 1 --2- 172.21.15.96:0/705924371 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f36d806c8f0 0x7f36d806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:01.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.209+0000 7f3701200640 1 --2- 172.21.15.96:0/705924371 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f36d806c8f0 0x7f36d806edb0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f36ec033000 tx=0x7f36ec009000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:01.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.212+0000 7f36f8c00640 1 -- 172.21.15.96:0/705924371 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f36e4059e60 con 0x7f36fc0d2db0 2024-05-28T06:05:01.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.423+0000 7f3703c00640 1 -- 172.21.15.96:0/705924371 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f36fc0d2af0 con 0x7f36fc0d2db0 2024-05-28T06:05:01.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.424+0000 7f36f8c00640 1 -- 172.21.15.96:0/705924371 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) ==== 80+0+0 (secure 0 0 0) 0x7f36e4059760 con 0x7f36fc0d2db0 2024-05-28T06:05:01.424 INFO:teuthology.orchestra.run.smithi096.stderr:listed 0 entries 2024-05-28T06:05:01.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.426+0000 7f3703c00640 1 -- 172.21.15.96:0/705924371 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f36d806c8f0 msgr2=0x7f36d806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:01.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.426+0000 7f3703c00640 1 --2- 172.21.15.96:0/705924371 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f36d806c8f0 0x7f36d806edb0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f36ec033000 tx=0x7f36ec009000 comp rx=0 tx=0).stop 2024-05-28T06:05:01.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.427+0000 7f3703c00640 1 -- 172.21.15.96:0/705924371 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f36fc0d2db0 msgr2=0x7f36fc1527a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:01.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.427+0000 7f3703c00640 1 --2- 172.21.15.96:0/705924371 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f36fc0d2db0 0x7f36fc1527a0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f36e400c820 tx=0x7f36e400cce0 comp rx=0 tx=0).stop 2024-05-28T06:05:01.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.427+0000 7f3703c00640 1 -- 172.21.15.96:0/705924371 shutdown_connections 2024-05-28T06:05:01.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.427+0000 7f3703c00640 1 --2- 172.21.15.96:0/705924371 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f36d806c8f0 0x7f36d806edb0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:01.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.427+0000 7f3703c00640 1 --2- 172.21.15.96:0/705924371 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f36fc0d4d60 0x7f36fc0c8180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:01.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.427+0000 7f3703c00640 1 --2- 172.21.15.96:0/705924371 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f36fc0d2db0 0x7f36fc1527a0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:01.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.427+0000 7f3703c00640 1 --2- 172.21.15.96:0/705924371 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f36fc0cf490 0x7f36fc0b9ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:01.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.427+0000 7f3703c00640 1 -- 172.21.15.96:0/705924371 >> 172.21.15.96:0/705924371 conn(0x7f36fc08bfc0 msgr2=0x7f36fc15de90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:01.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.427+0000 7f3703c00640 1 -- 172.21.15.96:0/705924371 shutdown_connections 2024-05-28T06:05:01.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.427+0000 7f3703c00640 1 -- 172.21.15.96:0/705924371 wait complete. 2024-05-28T06:05:01.447 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-05-28T06:05:01.447 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:05:01.507 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:05:01.711 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.711+0000 7f1131600640 1 Processor -- start 2024-05-28T06:05:01.712 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.711+0000 7f1131600640 1 -- start start 2024-05-28T06:05:01.713 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.712+0000 7f1131600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f112c06f860 0x7f112c0b3150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:01.713 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.712+0000 7f1131600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f112c0b3690 0x7f112c0d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:01.713 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.712+0000 7f1131600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f112c06ed10 con 0x7f112c0d09e0 2024-05-28T06:05:01.713 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.712+0000 7f1131600640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f112c0d0e50 con 0x7f112c0b3690 2024-05-28T06:05:01.713 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.712+0000 7f1131600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f112c15cb00 con 0x7f112c06f860 2024-05-28T06:05:01.713 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.713+0000 7f1123e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f112c0b3690 0x7f112c0d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:01.713 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.713+0000 7f1123e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f112c0b3690 0x7f112c0d04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54614/0 (socket says 172.21.15.96:54614) 2024-05-28T06:05:01.713 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.713+0000 7f1123e00640 1 -- 172.21.15.96:0/3073615315 learned_addr learned my addr 172.21.15.96:0/3073615315 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:01.714 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.713+0000 7f112aa00640 1 --2- 172.21.15.96:0/3073615315 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f112c06f860 0x7f112c0b3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:01.714 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.713+0000 7f1123e00640 1 -- 172.21.15.96:0/3073615315 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f112c06f860 msgr2=0x7f112c0b3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:01.714 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.713+0000 7f1123e00640 1 --2- 172.21.15.96:0/3073615315 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f112c06f860 0x7f112c0b3150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:01.714 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.713+0000 7f1123e00640 1 -- 172.21.15.96:0/3073615315 >> v1:172.21.15.123:6789/0 conn(0x7f112c0d09e0 legacy=0x7f112c160210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:01.714 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.713+0000 7f1123e00640 1 -- 172.21.15.96:0/3073615315 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f112c0b3cc0 con 0x7f112c0b3690 2024-05-28T06:05:01.716 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.714+0000 7f1123e00640 1 --2- 172.21.15.96:0/3073615315 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f112c0b3690 0x7f112c0d04a0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f1114009870 tx=0x7f111402a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b97f3363b639c66 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:01.716 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.714+0000 7f112a000640 1 -- 172.21.15.96:0/3073615315 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1114037070 con 0x7f112c0b3690 2024-05-28T06:05:01.716 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.714+0000 7f112a000640 1 -- 172.21.15.96:0/3073615315 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1114032440 con 0x7f112c0b3690 2024-05-28T06:05:01.716 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.715+0000 7f112a000640 1 -- 172.21.15.96:0/3073615315 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1114032810 con 0x7f112c0b3690 2024-05-28T06:05:01.716 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.715+0000 7f1131600640 1 -- 172.21.15.96:0/3073615315 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f112c0b3690 msgr2=0x7f112c0d04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:01.716 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.715+0000 7f1131600640 1 --2- 172.21.15.96:0/3073615315 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f112c0b3690 0x7f112c0d04a0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f1114009870 tx=0x7f111402a210 comp rx=0 tx=0).stop 2024-05-28T06:05:01.716 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.715+0000 7f1131600640 1 -- 172.21.15.96:0/3073615315 shutdown_connections 2024-05-28T06:05:01.716 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.715+0000 7f1131600640 1 --2- 172.21.15.96:0/3073615315 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f112c0b3690 0x7f112c0d04a0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:01.716 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.715+0000 7f1131600640 1 --2- 172.21.15.96:0/3073615315 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f112c06f860 0x7f112c0b3150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:01.716 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.715+0000 7f1131600640 1 -- 172.21.15.96:0/3073615315 >> 172.21.15.96:0/3073615315 conn(0x7f112c06bae0 msgr2=0x7f112c06bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:01.717 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.715+0000 7f1131600640 1 -- 172.21.15.96:0/3073615315 shutdown_connections 2024-05-28T06:05:01.717 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.715+0000 7f1131600640 1 -- 172.21.15.96:0/3073615315 wait complete. 2024-05-28T06:05:01.717 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.717+0000 7f1131600640 1 Processor -- start 2024-05-28T06:05:01.717 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.717+0000 7f1131600640 1 -- start start 2024-05-28T06:05:01.719 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.718+0000 7f1131600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f112c06f860 0x7f112c099980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:01.719 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.718+0000 7f1131600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f112c0b3690 0x7f112c15ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:01.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.718+0000 7f1131600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f112c0d09e0 0x7f112c0c8e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:01.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.718+0000 7f1131600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f112c0b4560 con 0x7f112c0b3690 2024-05-28T06:05:01.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.718+0000 7f1131600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f112c0b4280 con 0x7f112c0d09e0 2024-05-28T06:05:01.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.718+0000 7f1131600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f112c0b4840 con 0x7f112c06f860 2024-05-28T06:05:01.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.718+0000 7f1123e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f112c0b3690 0x7f112c15ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:01.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.718+0000 7f1123e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f112c0b3690 0x7f112c15ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54626/0 (socket says 172.21.15.96:54626) 2024-05-28T06:05:01.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.718+0000 7f1123e00640 1 -- 172.21.15.96:0/2276512436 learned_addr learned my addr 172.21.15.96:0/2276512436 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:01.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.719+0000 7f112aa00640 1 --2- 172.21.15.96:0/2276512436 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f112c06f860 0x7f112c099980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:01.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.719+0000 7f112b400640 1 --2- 172.21.15.96:0/2276512436 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f112c0d09e0 0x7f112c0c8e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:01.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.719+0000 7f1123e00640 1 -- 172.21.15.96:0/2276512436 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f112c06f860 msgr2=0x7f112c099980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:01.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.719+0000 7f1123e00640 1 --2- 172.21.15.96:0/2276512436 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f112c06f860 0x7f112c099980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:01.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.719+0000 7f1123e00640 1 -- 172.21.15.96:0/2276512436 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f112c0d09e0 msgr2=0x7f112c0c8e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:01.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.719+0000 7f1123e00640 1 --2- 172.21.15.96:0/2276512436 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f112c0d09e0 0x7f112c0c8e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:01.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.719+0000 7f1123e00640 1 -- 172.21.15.96:0/2276512436 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f112c099ec0 con 0x7f112c0b3690 2024-05-28T06:05:01.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.719+0000 7f1123e00640 1 --2- 172.21.15.96:0/2276512436 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f112c0b3690 0x7f112c15ef20 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f1114000c00 tx=0x7f11140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:01.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.720+0000 7f1122a00640 1 -- 172.21.15.96:0/2276512436 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1114041020 con 0x7f112c0b3690 2024-05-28T06:05:01.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.720+0000 7f1122a00640 1 -- 172.21.15.96:0/2276512436 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11140306c0 con 0x7f112c0b3690 2024-05-28T06:05:01.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.720+0000 7f1122a00640 1 -- 172.21.15.96:0/2276512436 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1114037040 con 0x7f112c0b3690 2024-05-28T06:05:01.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.720+0000 7f1131600640 1 -- 172.21.15.96:0/2276512436 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f112c15d850 con 0x7f112c0b3690 2024-05-28T06:05:01.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.720+0000 7f1131600640 1 -- 172.21.15.96:0/2276512436 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f112c15de10 con 0x7f112c0b3690 2024-05-28T06:05:01.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.721+0000 7f1131600640 1 -- 172.21.15.96:0/2276512436 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f112c0689e0 con 0x7f112c0b3690 2024-05-28T06:05:01.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.722+0000 7f1122a00640 1 -- 172.21.15.96:0/2276512436 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1114044020 con 0x7f112c0b3690 2024-05-28T06:05:01.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.723+0000 7f1122a00640 1 --2- 172.21.15.96:0/2276512436 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f110806c8f0 0x7f110806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:01.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.723+0000 7f1122a00640 1 -- 172.21.15.96:0/2276512436 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5825+0+0 (secure 0 0 0) 0x7f11140ac3c0 con 0x7f112c0b3690 2024-05-28T06:05:01.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.724+0000 7f112aa00640 1 --2- 172.21.15.96:0/2276512436 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f110806c8f0 0x7f110806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:01.725 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.725+0000 7f112aa00640 1 --2- 172.21.15.96:0/2276512436 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f110806c8f0 0x7f110806edb0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f111c006fd0 tx=0x7f111c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:01.728 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.728+0000 7f1122a00640 1 -- 172.21.15.96:0/2276512436 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f111404b030 con 0x7f112c0b3690 2024-05-28T06:05:01.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.928+0000 7f1131600640 1 -- 172.21.15.96:0/2276512436 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f112c0ccb00 con 0x7f112c0b3690 2024-05-28T06:05:01.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.930+0000 7f1122a00640 1 -- 172.21.15.96:0/2276512436 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) ==== 74+0+16865 (secure 0 0 0) 0x7f112c099ec0 con 0x7f112c0b3690 2024-05-28T06:05:01.931 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:05:01.931 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":22,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:04:52.624604+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-28T06:04:08.741938+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-28T06:04:09.301470+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.b","mappings":[{"from":3,"to":1}]},{"pgid":"3.d","mappings":[{"from":7,"to":2}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:05:01.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.932+0000 7f1131600640 1 -- 172.21.15.96:0/2276512436 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f110806c8f0 msgr2=0x7f110806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:01.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.932+0000 7f1131600640 1 --2- 172.21.15.96:0/2276512436 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f110806c8f0 0x7f110806edb0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f111c006fd0 tx=0x7f111c008040 comp rx=0 tx=0).stop 2024-05-28T06:05:01.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.933+0000 7f1131600640 1 -- 172.21.15.96:0/2276512436 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f112c0b3690 msgr2=0x7f112c15ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:01.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.933+0000 7f1131600640 1 --2- 172.21.15.96:0/2276512436 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f112c0b3690 0x7f112c15ef20 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f1114000c00 tx=0x7f11140046c0 comp rx=0 tx=0).stop 2024-05-28T06:05:01.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.933+0000 7f1131600640 1 -- 172.21.15.96:0/2276512436 shutdown_connections 2024-05-28T06:05:01.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.933+0000 7f1131600640 1 --2- 172.21.15.96:0/2276512436 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f110806c8f0 0x7f110806edb0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:01.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.933+0000 7f1131600640 1 --2- 172.21.15.96:0/2276512436 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f112c0d09e0 0x7f112c0c8e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:01.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.933+0000 7f1131600640 1 --2- 172.21.15.96:0/2276512436 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f112c0b3690 0x7f112c15ef20 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:01.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.933+0000 7f1131600640 1 --2- 172.21.15.96:0/2276512436 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f112c06f860 0x7f112c099980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:01.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.933+0000 7f1131600640 1 -- 172.21.15.96:0/2276512436 >> 172.21.15.96:0/2276512436 conn(0x7f112c06bae0 msgr2=0x7f112c0b2580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:01.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.933+0000 7f1131600640 1 -- 172.21.15.96:0/2276512436 shutdown_connections 2024-05-28T06:05:01.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:01.933+0000 7f1131600640 1 -- 172.21.15.96:0/2276512436 wait complete. 2024-05-28T06:05:01.951 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-28T06:03:59.517102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-28T06:04:08.741938+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-28T06:04:09.301470+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-28T06:05:01.952 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-28T06:05:02.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.149+0000 7fa8ff200640 1 Processor -- start 2024-05-28T06:05:02.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.149+0000 7fa8ff200640 1 -- start start 2024-05-28T06:05:02.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.150+0000 7fa8ff200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa900153510 0x7fa9000b4d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:02.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.150+0000 7fa8ff200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fa9000ccb90 0x7fa9000ccff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:02.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.151+0000 7fa8ff200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fa90015cb60 0x7fa90015cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:02.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.151+0000 7fa8ff200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fa9001538f0 con 0x7fa9000ccb90 2024-05-28T06:05:02.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.151+0000 7fa8ff200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fa9000cd530 con 0x7fa90015cb60 2024-05-28T06:05:02.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.151+0000 7fa8ff200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fa9000b48c0 con 0x7fa900153510 2024-05-28T06:05:02.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.151+0000 7fa8fd400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fa9000ccb90 0x7fa9000ccff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:02.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.151+0000 7fa8fd400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fa9000ccb90 0x7fa9000ccff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54642/0 (socket says 172.21.15.96:54642) 2024-05-28T06:05:02.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.151+0000 7fa8fd400640 1 -- 172.21.15.96:0/823501941 learned_addr learned my addr 172.21.15.96:0/823501941 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:02.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.151+0000 7fa8fd400640 1 -- 172.21.15.96:0/823501941 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa900153510 msgr2=0x7fa9000b4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:02.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.151+0000 7fa8fd400640 1 --2- 172.21.15.96:0/823501941 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa900153510 0x7fa9000b4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:02.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.151+0000 7fa8fd400640 1 -- 172.21.15.96:0/823501941 >> v2:172.21.15.123:3300/0 conn(0x7fa90015cb60 msgr2=0x7fa90015cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:02.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.151+0000 7fa8fd400640 1 --2- 172.21.15.96:0/823501941 >> v2:172.21.15.123:3300/0 conn(0x7fa90015cb60 0x7fa90015cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:02.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.151+0000 7fa8fd400640 1 -- 172.21.15.96:0/823501941 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa90015d900 con 0x7fa9000ccb90 2024-05-28T06:05:02.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.152+0000 7fa8fd400640 1 --2- 172.21.15.96:0/823501941 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa9000ccb90 0x7fa9000ccff0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fa9000ca760 tx=0x7fa8ec02a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=196616cf152ee08a server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:02.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.153+0000 7fa8fca00640 1 -- 172.21.15.96:0/823501941 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8ec037070 con 0x7fa9000ccb90 2024-05-28T06:05:02.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.153+0000 7fa8fca00640 1 -- 172.21.15.96:0/823501941 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8ec032440 con 0x7fa9000ccb90 2024-05-28T06:05:02.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.153+0000 7fa8fca00640 1 -- 172.21.15.96:0/823501941 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8ec032810 con 0x7fa9000ccb90 2024-05-28T06:05:02.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.153+0000 7fa8ff200640 1 -- 172.21.15.96:0/823501941 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa9000ccb90 msgr2=0x7fa9000ccff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:02.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.153+0000 7fa8ff200640 1 --2- 172.21.15.96:0/823501941 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa9000ccb90 0x7fa9000ccff0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fa9000ca760 tx=0x7fa8ec02a140 comp rx=0 tx=0).stop 2024-05-28T06:05:02.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.153+0000 7fa8ff200640 1 -- 172.21.15.96:0/823501941 shutdown_connections 2024-05-28T06:05:02.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.153+0000 7fa8ff200640 1 --2- 172.21.15.96:0/823501941 >> v2:172.21.15.123:3300/0 conn(0x7fa90015cb60 0x7fa90015cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:02.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.153+0000 7fa8ff200640 1 --2- 172.21.15.96:0/823501941 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa9000ccb90 0x7fa9000ccff0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:02.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.153+0000 7fa8ff200640 1 --2- 172.21.15.96:0/823501941 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa900153510 0x7fa9000b4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:02.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.153+0000 7fa8ff200640 1 -- 172.21.15.96:0/823501941 >> 172.21.15.96:0/823501941 conn(0x7fa90006b900 msgr2=0x7fa90006bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:02.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.153+0000 7fa8ff200640 1 -- 172.21.15.96:0/823501941 shutdown_connections 2024-05-28T06:05:02.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.153+0000 7fa8ff200640 1 -- 172.21.15.96:0/823501941 wait complete. 2024-05-28T06:05:02.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.155+0000 7fa8ff200640 1 Processor -- start 2024-05-28T06:05:02.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.155+0000 7fa8ff200640 1 -- start start 2024-05-28T06:05:02.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.156+0000 7fa8ff200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa9000ccb90 0x7fa9000aed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:02.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.156+0000 7fa8ff200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa900153510 0x7fa9001475d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:02.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.156+0000 7fa8ff200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa90015cb60 0x7fa90015b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:02.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.156+0000 7fa8ff200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa9000b4550 con 0x7fa90015cb60 2024-05-28T06:05:02.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.156+0000 7fa8ff200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa9000b4270 con 0x7fa900153510 2024-05-28T06:05:02.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.156+0000 7fa8ff200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fa90015d530 con 0x7fa9000ccb90 2024-05-28T06:05:02.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.156+0000 7fa8fe800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa90015cb60 0x7fa90015b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:02.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.156+0000 7fa8fe800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa90015cb60 0x7fa90015b430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54650/0 (socket says 172.21.15.96:54650) 2024-05-28T06:05:02.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.156+0000 7fa8fe800640 1 -- 172.21.15.96:0/1984078002 learned_addr learned my addr 172.21.15.96:0/1984078002 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:02.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.157+0000 7fa8fd400640 1 --2- 172.21.15.96:0/1984078002 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa900153510 0x7fa9001475d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:02.159 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.157+0000 7fa8fde00640 1 --2- 172.21.15.96:0/1984078002 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa9000ccb90 0x7fa9000aed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:02.159 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.157+0000 7fa8fe800640 1 -- 172.21.15.96:0/1984078002 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa9000ccb90 msgr2=0x7fa9000aed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:02.159 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.157+0000 7fa8fe800640 1 --2- 172.21.15.96:0/1984078002 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa9000ccb90 0x7fa9000aed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:02.159 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.157+0000 7fa8fe800640 1 -- 172.21.15.96:0/1984078002 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa900153510 msgr2=0x7fa9001475d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:02.159 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.157+0000 7fa8fe800640 1 --2- 172.21.15.96:0/1984078002 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa900153510 0x7fa9001475d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:02.159 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.157+0000 7fa8fe800640 1 -- 172.21.15.96:0/1984078002 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9000cdef0 con 0x7fa90015cb60 2024-05-28T06:05:02.159 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.157+0000 7fa8fde00640 1 --2- 172.21.15.96:0/1984078002 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa9000ccb90 0x7fa9000aed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:05:02.159 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.157+0000 7fa8fe800640 1 --2- 172.21.15.96:0/1984078002 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa90015cb60 0x7fa90015b430 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fa8f800b970 tx=0x7fa8f800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:02.159 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.157+0000 7fa8f6000640 1 -- 172.21.15.96:0/1984078002 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8f8019070 con 0x7fa90015cb60 2024-05-28T06:05:02.159 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.157+0000 7fa8f6000640 1 -- 172.21.15.96:0/1984078002 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8f800cb00 con 0x7fa90015cb60 2024-05-28T06:05:02.159 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.157+0000 7fa8ff200640 1 -- 172.21.15.96:0/1984078002 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9000ce190 con 0x7fa90015cb60 2024-05-28T06:05:02.159 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.157+0000 7fa8f6000640 1 -- 172.21.15.96:0/1984078002 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8f8014440 con 0x7fa90015cb60 2024-05-28T06:05:02.159 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.158+0000 7fa8ff200640 1 -- 172.21.15.96:0/1984078002 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9000cea30 con 0x7fa90015cb60 2024-05-28T06:05:02.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.159+0000 7fa8ff200640 1 -- 172.21.15.96:0/1984078002 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9000ce360 con 0x7fa90015cb60 2024-05-28T06:05:02.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.159+0000 7fa8f6000640 1 -- 172.21.15.96:0/1984078002 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa8f801c070 con 0x7fa90015cb60 2024-05-28T06:05:02.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.160+0000 7fa8f6000640 1 --2- 172.21.15.96:0/1984078002 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa8e006c920 0x7fa8e006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:02.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.160+0000 7fa8f6000640 1 -- 172.21.15.96:0/1984078002 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5825+0+0 (secure 0 0 0) 0x7fa8f808b850 con 0x7fa90015cb60 2024-05-28T06:05:02.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.161+0000 7fa8fde00640 1 --2- 172.21.15.96:0/1984078002 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa8e006c920 0x7fa8e006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:02.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.162+0000 7fa8fde00640 1 --2- 172.21.15.96:0/1984078002 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa8e006c920 0x7fa8e006ede0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa8f0006fd0 tx=0x7fa8f0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:02.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.164+0000 7fa8f6000640 1 -- 172.21.15.96:0/1984078002 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa8f8059bf0 con 0x7fa90015cb60 2024-05-28T06:05:02.352 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.351+0000 7fa8ff200640 1 -- 172.21.15.96:0/1984078002 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fa9000689e0 con 0x7fa90015cb60 2024-05-28T06:05:02.352 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.352+0000 7fa8f6000640 1 -- 172.21.15.96:0/1984078002 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) ==== 93+0+10 (secure 0 0 0) 0x7fa8f80594f0 con 0x7fa90015cb60 2024-05-28T06:05:02.353 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 1 2024-05-28T06:05:02.355 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.354+0000 7fa8ff200640 1 -- 172.21.15.96:0/1984078002 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa8e006c920 msgr2=0x7fa8e006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:02.355 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.354+0000 7fa8ff200640 1 --2- 172.21.15.96:0/1984078002 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa8e006c920 0x7fa8e006ede0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa8f0006fd0 tx=0x7fa8f0008040 comp rx=0 tx=0).stop 2024-05-28T06:05:02.355 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.354+0000 7fa8ff200640 1 -- 172.21.15.96:0/1984078002 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa90015cb60 msgr2=0x7fa90015b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:02.355 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.354+0000 7fa8ff200640 1 --2- 172.21.15.96:0/1984078002 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa90015cb60 0x7fa90015b430 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fa8f800b970 tx=0x7fa8f800be30 comp rx=0 tx=0).stop 2024-05-28T06:05:02.355 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.355+0000 7fa8ff200640 1 -- 172.21.15.96:0/1984078002 shutdown_connections 2024-05-28T06:05:02.355 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.355+0000 7fa8ff200640 1 --2- 172.21.15.96:0/1984078002 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa8e006c920 0x7fa8e006ede0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:02.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.355+0000 7fa8ff200640 1 --2- 172.21.15.96:0/1984078002 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa90015cb60 0x7fa90015b430 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:02.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.355+0000 7fa8ff200640 1 --2- 172.21.15.96:0/1984078002 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa900153510 0x7fa9001475d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:02.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.355+0000 7fa8ff200640 1 --2- 172.21.15.96:0/1984078002 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa9000ccb90 0x7fa9000aed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:02.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.355+0000 7fa8ff200640 1 -- 172.21.15.96:0/1984078002 >> 172.21.15.96:0/1984078002 conn(0x7fa90006b900 msgr2=0x7fa90006e900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:02.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.355+0000 7fa8ff200640 1 -- 172.21.15.96:0/1984078002 shutdown_connections 2024-05-28T06:05:02.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.355+0000 7fa8ff200640 1 -- 172.21.15.96:0/1984078002 wait complete. 2024-05-28T06:05:02.372 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-28T06:05:02.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.572+0000 7fdb59000640 1 Processor -- start 2024-05-28T06:05:02.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.573+0000 7fdb59000640 1 -- start start 2024-05-28T06:05:02.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.574+0000 7fdb59000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fdb5415ed80 0x7fdb540cff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:02.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.574+0000 7fdb59000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdb540b68f0 0x7fdb540cedb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:02.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.574+0000 7fdb59000640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb5415f160 con 0x7fdb540b7e50 2024-05-28T06:05:02.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.574+0000 7fdb59000640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fdb54163540 con 0x7fdb5415ed80 2024-05-28T06:05:02.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.574+0000 7fdb59000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fdb54162930 con 0x7fdb540b68f0 2024-05-28T06:05:02.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.575+0000 7fdb51e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fdb5415ed80 0x7fdb540cff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:02.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.575+0000 7fdb52800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdb540b68f0 0x7fdb540cedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:02.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.575+0000 7fdb52800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdb540b68f0 0x7fdb540cedb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.96:38546/0 (socket says 172.21.15.96:38546) 2024-05-28T06:05:02.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.575+0000 7fdb52800640 1 -- 172.21.15.96:0/3646440391 learned_addr learned my addr 172.21.15.96:0/3646440391 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:02.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.575+0000 7fdb52800640 1 -- 172.21.15.96:0/3646440391 >> v2:172.21.15.123:3300/0 conn(0x7fdb5415ed80 msgr2=0x7fdb540cff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:02.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.575+0000 7fdb52800640 1 --2- 172.21.15.96:0/3646440391 >> v2:172.21.15.123:3300/0 conn(0x7fdb5415ed80 0x7fdb540cff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:02.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.575+0000 7fdb52800640 1 -- 172.21.15.96:0/3646440391 >> v1:172.21.15.123:6789/0 conn(0x7fdb540b7e50 legacy=0x7fdb540b61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:02.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.575+0000 7fdb52800640 1 -- 172.21.15.96:0/3646440391 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb54162c10 con 0x7fdb540b68f0 2024-05-28T06:05:02.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.577+0000 7fdb52800640 1 --2- 172.21.15.96:0/3646440391 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdb540b68f0 0x7fdb540cedb0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c009870 tx=0x7fdb4c02a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9350e536faafc98d server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:02.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.577+0000 7fdb51400640 1 -- 172.21.15.96:0/3646440391 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb4c037070 con 0x7fdb540b68f0 2024-05-28T06:05:02.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.577+0000 7fdb51400640 1 -- 172.21.15.96:0/3646440391 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb4c032440 con 0x7fdb540b68f0 2024-05-28T06:05:02.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.577+0000 7fdb51400640 1 -- 172.21.15.96:0/3646440391 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb4c032810 con 0x7fdb540b68f0 2024-05-28T06:05:02.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.578+0000 7fdb59000640 1 -- 172.21.15.96:0/3646440391 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdb540b68f0 msgr2=0x7fdb540cedb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:02.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.578+0000 7fdb59000640 1 --2- 172.21.15.96:0/3646440391 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdb540b68f0 0x7fdb540cedb0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c009870 tx=0x7fdb4c02a270 comp rx=0 tx=0).stop 2024-05-28T06:05:02.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.578+0000 7fdb59000640 1 -- 172.21.15.96:0/3646440391 shutdown_connections 2024-05-28T06:05:02.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.578+0000 7fdb59000640 1 --2- 172.21.15.96:0/3646440391 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdb540b68f0 0x7fdb540cedb0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:02.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.578+0000 7fdb59000640 1 --2- 172.21.15.96:0/3646440391 >> v2:172.21.15.123:3300/0 conn(0x7fdb5415ed80 0x7fdb540cff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:02.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.578+0000 7fdb59000640 1 -- 172.21.15.96:0/3646440391 >> 172.21.15.96:0/3646440391 conn(0x7fdb540d13c0 msgr2=0x7fdb540d17d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:02.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.578+0000 7fdb59000640 1 -- 172.21.15.96:0/3646440391 shutdown_connections 2024-05-28T06:05:02.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.578+0000 7fdb59000640 1 -- 172.21.15.96:0/3646440391 wait complete. 2024-05-28T06:05:02.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.580+0000 7fdb59000640 1 Processor -- start 2024-05-28T06:05:02.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.580+0000 7fdb59000640 1 -- start start 2024-05-28T06:05:02.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.581+0000 7fdb59000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb540b68f0 0x7fdb54150100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:02.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.581+0000 7fdb59000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb540b7e50 0x7fdb5415ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:02.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.581+0000 7fdb59000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdb5415ed80 0x7fdb5415b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:02.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.581+0000 7fdb59000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fdb540677b0 con 0x7fdb540b68f0 2024-05-28T06:05:02.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.581+0000 7fdb59000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fdb541631d0 con 0x7fdb540b7e50 2024-05-28T06:05:02.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.581+0000 7fdb59000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fdb54067930 con 0x7fdb5415ed80 2024-05-28T06:05:02.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.581+0000 7fdb51e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb540b68f0 0x7fdb54150100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:02.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.581+0000 7fdb51e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb540b68f0 0x7fdb54150100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54662/0 (socket says 172.21.15.96:54662) 2024-05-28T06:05:02.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.581+0000 7fdb51e00640 1 -- 172.21.15.96:0/514799030 learned_addr learned my addr 172.21.15.96:0/514799030 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:02.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.581+0000 7fdb49400640 1 --2- 172.21.15.96:0/514799030 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb540b7e50 0x7fdb5415ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:02.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.581+0000 7fdb52800640 1 --2- 172.21.15.96:0/514799030 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdb5415ed80 0x7fdb5415b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:02.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.581+0000 7fdb51e00640 1 -- 172.21.15.96:0/514799030 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdb5415ed80 msgr2=0x7fdb5415b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:02.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.581+0000 7fdb51e00640 1 --2- 172.21.15.96:0/514799030 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdb5415ed80 0x7fdb5415b8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:02.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.581+0000 7fdb51e00640 1 -- 172.21.15.96:0/514799030 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb540b7e50 msgr2=0x7fdb5415ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:02.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.581+0000 7fdb51e00640 1 --2- 172.21.15.96:0/514799030 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb540b7e50 0x7fdb5415ad30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:02.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.582+0000 7fdb51e00640 1 -- 172.21.15.96:0/514799030 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb5415bfe0 con 0x7fdb540b68f0 2024-05-28T06:05:02.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.582+0000 7fdb51e00640 1 --2- 172.21.15.96:0/514799030 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb540b68f0 0x7fdb54150100 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fdb4400efc0 tx=0x7fdb4400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:02.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.582+0000 7fdb4aa00640 1 -- 172.21.15.96:0/514799030 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb44018020 con 0x7fdb540b68f0 2024-05-28T06:05:02.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.582+0000 7fdb4aa00640 1 -- 172.21.15.96:0/514799030 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb44014480 con 0x7fdb540b68f0 2024-05-28T06:05:02.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.582+0000 7fdb4aa00640 1 -- 172.21.15.96:0/514799030 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb44014850 con 0x7fdb540b68f0 2024-05-28T06:05:02.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.582+0000 7fdb59000640 1 -- 172.21.15.96:0/514799030 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb5415c2c0 con 0x7fdb540b68f0 2024-05-28T06:05:02.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.582+0000 7fdb59000640 1 -- 172.21.15.96:0/514799030 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb540c8610 con 0x7fdb540b68f0 2024-05-28T06:05:02.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.583+0000 7fdb59000640 1 -- 172.21.15.96:0/514799030 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb540689e0 con 0x7fdb540b68f0 2024-05-28T06:05:02.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.584+0000 7fdb4aa00640 1 -- 172.21.15.96:0/514799030 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fdb44005d10 con 0x7fdb540b68f0 2024-05-28T06:05:02.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.585+0000 7fdb4aa00640 1 --2- 172.21.15.96:0/514799030 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdb3006c8f0 0x7fdb3006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:02.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.585+0000 7fdb4aa00640 1 -- 172.21.15.96:0/514799030 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5825+0+0 (secure 0 0 0) 0x7fdb4408c700 con 0x7fdb540b68f0 2024-05-28T06:05:02.586 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.586+0000 7fdb49400640 1 --2- 172.21.15.96:0/514799030 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdb3006c8f0 0x7fdb3006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:02.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.587+0000 7fdb49400640 1 --2- 172.21.15.96:0/514799030 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdb3006c8f0 0x7fdb3006edb0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fdb54150640 tx=0x7fdb3c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:02.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.590+0000 7fdb4aa00640 1 -- 172.21.15.96:0/514799030 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fdb44022070 con 0x7fdb540b68f0 2024-05-28T06:05:02.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.790+0000 7fdb59000640 1 -- 172.21.15.96:0/514799030 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fdb540d2910 con 0x7fdb540b68f0 2024-05-28T06:05:02.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.791+0000 7fdb4aa00640 1 -- 172.21.15.96:0/514799030 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) ==== 104+0+11 (secure 0 0 0) 0x7fdb44010040 con 0x7fdb540b68f0 2024-05-28T06:05:02.792 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 32 2024-05-28T06:05:02.794 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.794+0000 7fdb59000640 1 -- 172.21.15.96:0/514799030 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdb3006c8f0 msgr2=0x7fdb3006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:02.794 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.794+0000 7fdb59000640 1 --2- 172.21.15.96:0/514799030 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdb3006c8f0 0x7fdb3006edb0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fdb54150640 tx=0x7fdb3c009040 comp rx=0 tx=0).stop 2024-05-28T06:05:02.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.794+0000 7fdb59000640 1 -- 172.21.15.96:0/514799030 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb540b68f0 msgr2=0x7fdb54150100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:02.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.794+0000 7fdb59000640 1 --2- 172.21.15.96:0/514799030 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb540b68f0 0x7fdb54150100 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fdb4400efc0 tx=0x7fdb4400c490 comp rx=0 tx=0).stop 2024-05-28T06:05:02.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.794+0000 7fdb59000640 1 -- 172.21.15.96:0/514799030 shutdown_connections 2024-05-28T06:05:02.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.794+0000 7fdb59000640 1 --2- 172.21.15.96:0/514799030 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdb3006c8f0 0x7fdb3006edb0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:02.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.794+0000 7fdb59000640 1 --2- 172.21.15.96:0/514799030 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdb5415ed80 0x7fdb5415b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:02.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.794+0000 7fdb59000640 1 --2- 172.21.15.96:0/514799030 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb540b7e50 0x7fdb5415ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:02.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.794+0000 7fdb59000640 1 --2- 172.21.15.96:0/514799030 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb540b68f0 0x7fdb54150100 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:02.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.794+0000 7fdb59000640 1 -- 172.21.15.96:0/514799030 >> 172.21.15.96:0/514799030 conn(0x7fdb540d13c0 msgr2=0x7fdb5415f460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:02.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.794+0000 7fdb59000640 1 -- 172.21.15.96:0/514799030 shutdown_connections 2024-05-28T06:05:02.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:02.794+0000 7fdb59000640 1 -- 172.21.15.96:0/514799030 wait complete. 2024-05-28T06:05:02.812 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-28T06:05:03.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.018+0000 7f8b15400640 1 Processor -- start 2024-05-28T06:05:03.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.018+0000 7f8b15400640 1 -- start start 2024-05-28T06:05:03.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.020+0000 7f8b15400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f8b1015e870 0x7f8b100d0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:03.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.020+0000 7f8b15400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f8b100cfa40 0x7f8b100d25e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:03.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.020+0000 7f8b15400640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b1015ec50 con 0x7f8b100d14d0 2024-05-28T06:05:03.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.020+0000 7f8b15400640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f8b100d2150 con 0x7f8b1015e870 2024-05-28T06:05:03.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.020+0000 7f8b15400640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f8b100cff80 con 0x7f8b100cfa40 2024-05-28T06:05:03.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.020+0000 7f8b0fe00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f8b1015e870 0x7f8b100d0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:03.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.020+0000 7f8b0fe00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f8b1015e870 0x7f8b100d0f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54678/0 (socket says 172.21.15.96:54678) 2024-05-28T06:05:03.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.020+0000 7f8b0fe00640 1 -- 172.21.15.96:0/3378718819 learned_addr learned my addr 172.21.15.96:0/3378718819 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:03.022 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.020+0000 7f8b14a00640 1 --2- 172.21.15.96:0/3378718819 >> v2:172.21.15.123:3300/0 conn(0x7f8b100cfa40 0x7f8b100d25e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:03.022 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.020+0000 7f8b0fe00640 1 -- 172.21.15.96:0/3378718819 >> v2:172.21.15.123:3300/0 conn(0x7f8b100cfa40 msgr2=0x7f8b100d25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:03.022 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.020+0000 7f8b0fe00640 1 --2- 172.21.15.96:0/3378718819 >> v2:172.21.15.123:3300/0 conn(0x7f8b100cfa40 0x7f8b100d25e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:03.022 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.020+0000 7f8b0fe00640 1 -- 172.21.15.96:0/3378718819 >> v1:172.21.15.123:6789/0 conn(0x7f8b100d14d0 legacy=0x7f8b100cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:03.022 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.020+0000 7f8b0fe00640 1 -- 172.21.15.96:0/3378718819 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b100d0260 con 0x7f8b1015e870 2024-05-28T06:05:03.022 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.022+0000 7f8b0fe00640 1 --2- 172.21.15.96:0/3378718819 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8b1015e870 0x7f8b100d0f90 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f8b1008cad0 tx=0x7f8b0002a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9eac0c83e8259875 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:03.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.022+0000 7f8b0ea00640 1 -- 172.21.15.96:0/3378718819 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b00037070 con 0x7f8b1015e870 2024-05-28T06:05:03.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.022+0000 7f8b0ea00640 1 -- 172.21.15.96:0/3378718819 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b00032440 con 0x7f8b1015e870 2024-05-28T06:05:03.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.022+0000 7f8b0ea00640 1 -- 172.21.15.96:0/3378718819 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b00032810 con 0x7f8b1015e870 2024-05-28T06:05:03.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.022+0000 7f8b15400640 1 -- 172.21.15.96:0/3378718819 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8b1015e870 msgr2=0x7f8b100d0f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:03.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.022+0000 7f8b15400640 1 --2- 172.21.15.96:0/3378718819 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8b1015e870 0x7f8b100d0f90 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f8b1008cad0 tx=0x7f8b0002a230 comp rx=0 tx=0).stop 2024-05-28T06:05:03.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.023+0000 7f8b15400640 1 -- 172.21.15.96:0/3378718819 shutdown_connections 2024-05-28T06:05:03.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.023+0000 7f8b15400640 1 --2- 172.21.15.96:0/3378718819 >> v2:172.21.15.123:3300/0 conn(0x7f8b100cfa40 0x7f8b100d25e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:03.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.023+0000 7f8b15400640 1 --2- 172.21.15.96:0/3378718819 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8b1015e870 0x7f8b100d0f90 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:03.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.023+0000 7f8b15400640 1 -- 172.21.15.96:0/3378718819 >> 172.21.15.96:0/3378718819 conn(0x7f8b1008bfc0 msgr2=0x7f8b100d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:03.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.023+0000 7f8b15400640 1 -- 172.21.15.96:0/3378718819 shutdown_connections 2024-05-28T06:05:03.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.023+0000 7f8b15400640 1 -- 172.21.15.96:0/3378718819 wait complete. 2024-05-28T06:05:03.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.024+0000 7f8b15400640 1 Processor -- start 2024-05-28T06:05:03.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.025+0000 7f8b15400640 1 -- start start 2024-05-28T06:05:03.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.025+0000 7f8b15400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8b100cfa40 0x7f8b100ccca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:03.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.025+0000 7f8b15400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b100d14d0 0x7f8b100cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:03.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.026+0000 7f8b15400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8b1015e870 0x7f8b100c9100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:03.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.026+0000 7f8b15400640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f8b100d4fb0 con 0x7f8b1015e870 2024-05-28T06:05:03.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.026+0000 7f8b15400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f8b100d0820 con 0x7f8b100d14d0 2024-05-28T06:05:03.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.026+0000 7f8b15400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f8b100d2b20 con 0x7f8b100cfa40 2024-05-28T06:05:03.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.026+0000 7f8b14a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8b1015e870 0x7f8b100c9100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:03.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.026+0000 7f8b14a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8b1015e870 0x7f8b100c9100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54680/0 (socket says 172.21.15.96:54680) 2024-05-28T06:05:03.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.026+0000 7f8b14a00640 1 -- 172.21.15.96:0/3323617287 learned_addr learned my addr 172.21.15.96:0/3323617287 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:03.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.026+0000 7f8b0f400640 1 --2- 172.21.15.96:0/3323617287 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b100d14d0 0x7f8b100cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:03.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.026+0000 7f8b0fe00640 1 --2- 172.21.15.96:0/3323617287 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8b100cfa40 0x7f8b100ccca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:03.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.026+0000 7f8b14a00640 1 -- 172.21.15.96:0/3323617287 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8b100cfa40 msgr2=0x7f8b100ccca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:03.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.026+0000 7f8b14a00640 1 --2- 172.21.15.96:0/3323617287 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8b100cfa40 0x7f8b100ccca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:03.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.026+0000 7f8b14a00640 1 -- 172.21.15.96:0/3323617287 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b100d14d0 msgr2=0x7f8b100cd1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:03.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.026+0000 7f8b14a00640 1 --2- 172.21.15.96:0/3323617287 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b100d14d0 0x7f8b100cd1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:03.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.026+0000 7f8b14a00640 1 -- 172.21.15.96:0/3323617287 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b100cda00 con 0x7f8b1015e870 2024-05-28T06:05:03.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.026+0000 7f8b0fe00640 1 --2- 172.21.15.96:0/3323617287 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8b100cfa40 0x7f8b100ccca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:05:03.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.027+0000 7f8b14a00640 1 --2- 172.21.15.96:0/3323617287 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8b1015e870 0x7f8b100c9100 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f8b0800cd60 tx=0x7f8b0800a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:03.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.027+0000 7f8b07e00640 1 -- 172.21.15.96:0/3323617287 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b08003b90 con 0x7f8b1015e870 2024-05-28T06:05:03.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.027+0000 7f8b07e00640 1 -- 172.21.15.96:0/3323617287 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b08003d50 con 0x7f8b1015e870 2024-05-28T06:05:03.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.027+0000 7f8b07e00640 1 -- 172.21.15.96:0/3323617287 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b08004260 con 0x7f8b1015e870 2024-05-28T06:05:03.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.027+0000 7f8b15400640 1 -- 172.21.15.96:0/3323617287 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b100cdce0 con 0x7f8b1015e870 2024-05-28T06:05:03.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.027+0000 7f8b15400640 1 -- 172.21.15.96:0/3323617287 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b100b7240 con 0x7f8b1015e870 2024-05-28T06:05:03.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.028+0000 7f8b15400640 1 -- 172.21.15.96:0/3323617287 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b100cd780 con 0x7f8b1015e870 2024-05-28T06:05:03.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.029+0000 7f8b07e00640 1 -- 172.21.15.96:0/3323617287 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8b08004400 con 0x7f8b1015e870 2024-05-28T06:05:03.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.030+0000 7f8b07e00640 1 --2- 172.21.15.96:0/3323617287 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8af006c8f0 0x7f8af006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:03.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.030+0000 7f8b07e00640 1 -- 172.21.15.96:0/3323617287 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5825+0+0 (secure 0 0 0) 0x7f8b0808c6d0 con 0x7f8b1015e870 2024-05-28T06:05:03.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.031+0000 7f8b0fe00640 1 --2- 172.21.15.96:0/3323617287 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8af006c8f0 0x7f8af006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:03.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.032+0000 7f8b0fe00640 1 --2- 172.21.15.96:0/3323617287 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8af006c8f0 0x7f8af006edb0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f8b00034000 tx=0x7f8b0002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:03.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.034+0000 7f8b07e00640 1 -- 172.21.15.96:0/3323617287 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8b100cd780 con 0x7f8b1015e870 2024-05-28T06:05:03.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.231+0000 7f8b15400640 1 -- 172.21.15.96:0/3323617287 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f8b100689e0 con 0x7f8b1015e870 2024-05-28T06:05:03.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.232+0000 7f8b07e00640 1 -- 172.21.15.96:0/3323617287 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) ==== 100+0+11 (secure 0 0 0) 0x7f8b0805aa80 con 0x7f8b1015e870 2024-05-28T06:05:03.233 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 64 2024-05-28T06:05:03.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.236+0000 7f8b15400640 1 -- 172.21.15.96:0/3323617287 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8af006c8f0 msgr2=0x7f8af006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:03.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.236+0000 7f8b15400640 1 --2- 172.21.15.96:0/3323617287 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8af006c8f0 0x7f8af006edb0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f8b00034000 tx=0x7f8b0002e000 comp rx=0 tx=0).stop 2024-05-28T06:05:03.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.236+0000 7f8b15400640 1 -- 172.21.15.96:0/3323617287 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8b1015e870 msgr2=0x7f8b100c9100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:03.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.236+0000 7f8b15400640 1 --2- 172.21.15.96:0/3323617287 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8b1015e870 0x7f8b100c9100 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f8b0800cd60 tx=0x7f8b0800a660 comp rx=0 tx=0).stop 2024-05-28T06:05:03.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.236+0000 7f8b15400640 1 -- 172.21.15.96:0/3323617287 shutdown_connections 2024-05-28T06:05:03.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.236+0000 7f8b15400640 1 --2- 172.21.15.96:0/3323617287 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8af006c8f0 0x7f8af006edb0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:03.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.236+0000 7f8b15400640 1 --2- 172.21.15.96:0/3323617287 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8b1015e870 0x7f8b100c9100 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:03.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.236+0000 7f8b15400640 1 --2- 172.21.15.96:0/3323617287 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b100d14d0 0x7f8b100cd1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:03.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.236+0000 7f8b15400640 1 --2- 172.21.15.96:0/3323617287 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8b100cfa40 0x7f8b100ccca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:03.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.236+0000 7f8b15400640 1 -- 172.21.15.96:0/3323617287 >> 172.21.15.96:0/3323617287 conn(0x7f8b1008bfc0 msgr2=0x7f8b100b6200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:03.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.236+0000 7f8b15400640 1 -- 172.21.15.96:0/3323617287 shutdown_connections 2024-05-28T06:05:03.238 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.236+0000 7f8b15400640 1 -- 172.21.15.96:0/3323617287 wait complete. 2024-05-28T06:05:03.256 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-28T06:05:03.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.453+0000 7f3f76e00640 1 Processor -- start 2024-05-28T06:05:03.454 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.453+0000 7f3f76e00640 1 -- start start 2024-05-28T06:05:03.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.454+0000 7f3f76e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f3f7015ed00 0x7f3f700b3c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:03.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.454+0000 7f3f76e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f3f700d46c0 0x7f3f700d4b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:03.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.454+0000 7f3f76e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3f700d0d60 0x7f3f700d11f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:03.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.454+0000 7f3f76e00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f3f700d5060 con 0x7f3f700d46c0 2024-05-28T06:05:03.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.454+0000 7f3f76e00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f3f700b41a0 con 0x7f3f7015ed00 2024-05-28T06:05:03.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.454+0000 7f3f76e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f3f700b5070 con 0x7f3f700d0d60 2024-05-28T06:05:03.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.455+0000 7f3f67e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f3f700d46c0 0x7f3f700d4b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:03.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.455+0000 7f3f67e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f3f700d46c0 0x7f3f700d4b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54684/0 (socket says 172.21.15.96:54684) 2024-05-28T06:05:03.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.455+0000 7f3f67e00640 1 -- 172.21.15.96:0/888741779 learned_addr learned my addr 172.21.15.96:0/888741779 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:03.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.455+0000 7f3f6fe00640 1 --2- 172.21.15.96:0/888741779 >> v2:172.21.15.123:3300/0 conn(0x7f3f7015ed00 0x7f3f700b3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:03.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.455+0000 7f3f74e00640 1 --2- 172.21.15.96:0/888741779 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3f700d0d60 0x7f3f700d11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:03.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.455+0000 7f3f67e00640 1 -- 172.21.15.96:0/888741779 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3f700d0d60 msgr2=0x7f3f700d11f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:03.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.455+0000 7f3f67e00640 1 --2- 172.21.15.96:0/888741779 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3f700d0d60 0x7f3f700d11f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:03.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.455+0000 7f3f67e00640 1 -- 172.21.15.96:0/888741779 >> v2:172.21.15.123:3300/0 conn(0x7f3f7015ed00 msgr2=0x7f3f700b3c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:03.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.455+0000 7f3f67e00640 1 --2- 172.21.15.96:0/888741779 >> v2:172.21.15.123:3300/0 conn(0x7f3f7015ed00 0x7f3f700b3c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:03.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.455+0000 7f3f67e00640 1 -- 172.21.15.96:0/888741779 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f700d33b0 con 0x7f3f700d46c0 2024-05-28T06:05:03.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.455+0000 7f3f74e00640 1 --2- 172.21.15.96:0/888741779 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3f700d0d60 0x7f3f700d11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:05:03.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.455+0000 7f3f6fe00640 1 --2- 172.21.15.96:0/888741779 >> v2:172.21.15.123:3300/0 conn(0x7f3f7015ed00 0x7f3f700b3c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:05:03.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.456+0000 7f3f67e00640 1 --2- 172.21.15.96:0/888741779 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3f700d46c0 0x7f3f700d4b20 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f3f700b7320 tx=0x7f3f5802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f07f8d7eca6d4b2 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:03.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.457+0000 7f3f6f400640 1 -- 172.21.15.96:0/888741779 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f58037070 con 0x7f3f700d46c0 2024-05-28T06:05:03.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.457+0000 7f3f6f400640 1 -- 172.21.15.96:0/888741779 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f58032440 con 0x7f3f700d46c0 2024-05-28T06:05:03.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.457+0000 7f3f6f400640 1 -- 172.21.15.96:0/888741779 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f58032810 con 0x7f3f700d46c0 2024-05-28T06:05:03.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.457+0000 7f3f76e00640 1 -- 172.21.15.96:0/888741779 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3f700d46c0 msgr2=0x7f3f700d4b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:03.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.457+0000 7f3f76e00640 1 --2- 172.21.15.96:0/888741779 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3f700d46c0 0x7f3f700d4b20 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f3f700b7320 tx=0x7f3f5802a1e0 comp rx=0 tx=0).stop 2024-05-28T06:05:03.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.458+0000 7f3f76e00640 1 -- 172.21.15.96:0/888741779 shutdown_connections 2024-05-28T06:05:03.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.458+0000 7f3f76e00640 1 --2- 172.21.15.96:0/888741779 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3f700d0d60 0x7f3f700d11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:03.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.458+0000 7f3f76e00640 1 --2- 172.21.15.96:0/888741779 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3f700d46c0 0x7f3f700d4b20 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:03.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.458+0000 7f3f76e00640 1 --2- 172.21.15.96:0/888741779 >> v2:172.21.15.123:3300/0 conn(0x7f3f7015ed00 0x7f3f700b3c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:03.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.458+0000 7f3f76e00640 1 -- 172.21.15.96:0/888741779 >> 172.21.15.96:0/888741779 conn(0x7f3f7009da70 msgr2=0x7f3f7008bfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:03.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.458+0000 7f3f76e00640 1 -- 172.21.15.96:0/888741779 shutdown_connections 2024-05-28T06:05:03.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.458+0000 7f3f76e00640 1 -- 172.21.15.96:0/888741779 wait complete. 2024-05-28T06:05:03.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.459+0000 7f3f76e00640 1 Processor -- start 2024-05-28T06:05:03.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.459+0000 7f3f76e00640 1 -- start start 2024-05-28T06:05:03.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.460+0000 7f3f76e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3f700d0d60 0x7f3f700cfcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:03.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.460+0000 7f3f76e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3f700d46c0 0x7f3f700cd590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:03.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.460+0000 7f3f76e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f7015ed00 0x7f3f700b94f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:03.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.460+0000 7f3f76e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f3f700b5610 con 0x7f3f700d46c0 2024-05-28T06:05:03.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.460+0000 7f3f76e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f3f700b5330 con 0x7f3f7015ed00 2024-05-28T06:05:03.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.460+0000 7f3f76e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f3f700d3040 con 0x7f3f700d0d60 2024-05-28T06:05:03.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.461+0000 7f3f67e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3f700d46c0 0x7f3f700cd590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:03.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.461+0000 7f3f67e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3f700d46c0 0x7f3f700cd590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54700/0 (socket says 172.21.15.96:54700) 2024-05-28T06:05:03.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.461+0000 7f3f67e00640 1 -- 172.21.15.96:0/1007243098 learned_addr learned my addr 172.21.15.96:0/1007243098 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:03.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.461+0000 7f3f67e00640 1 -- 172.21.15.96:0/1007243098 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3f700d0d60 msgr2=0x7f3f700cfcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:03.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.461+0000 7f3f67e00640 1 --2- 172.21.15.96:0/1007243098 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3f700d0d60 0x7f3f700cfcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:03.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.461+0000 7f3f67e00640 1 -- 172.21.15.96:0/1007243098 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f7015ed00 msgr2=0x7f3f700b94f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:03.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.461+0000 7f3f67e00640 1 --2- 172.21.15.96:0/1007243098 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f7015ed00 0x7f3f700b94f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:03.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.461+0000 7f3f67e00640 1 -- 172.21.15.96:0/1007243098 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f700cdec0 con 0x7f3f700d46c0 2024-05-28T06:05:03.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.461+0000 7f3f67e00640 1 --2- 172.21.15.96:0/1007243098 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3f700d46c0 0x7f3f700cd590 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f3f5802a220 tx=0x7f3f580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:03.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.462+0000 7f3f6cc00640 1 -- 172.21.15.96:0/1007243098 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f58037040 con 0x7f3f700d46c0 2024-05-28T06:05:03.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.462+0000 7f3f6cc00640 1 -- 172.21.15.96:0/1007243098 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f58042070 con 0x7f3f700d46c0 2024-05-28T06:05:03.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.462+0000 7f3f6cc00640 1 -- 172.21.15.96:0/1007243098 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f58030bb0 con 0x7f3f700d46c0 2024-05-28T06:05:03.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.462+0000 7f3f76e00640 1 -- 172.21.15.96:0/1007243098 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f700ce1a0 con 0x7f3f700d46c0 2024-05-28T06:05:03.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.462+0000 7f3f76e00640 1 -- 172.21.15.96:0/1007243098 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f700ce760 con 0x7f3f700d46c0 2024-05-28T06:05:03.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.463+0000 7f3f76e00640 1 -- 172.21.15.96:0/1007243098 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f700689e0 con 0x7f3f700d46c0 2024-05-28T06:05:03.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.464+0000 7f3f6cc00640 1 -- 172.21.15.96:0/1007243098 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3f58047460 con 0x7f3f700d46c0 2024-05-28T06:05:03.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.465+0000 7f3f6cc00640 1 --2- 172.21.15.96:0/1007243098 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3f4c06c8f0 0x7f3f4c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:03.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.465+0000 7f3f6cc00640 1 -- 172.21.15.96:0/1007243098 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5825+0+0 (secure 0 0 0) 0x7f3f580ac930 con 0x7f3f700d46c0 2024-05-28T06:05:03.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.465+0000 7f3f6fe00640 1 --2- 172.21.15.96:0/1007243098 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3f4c06c8f0 0x7f3f4c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:03.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.467+0000 7f3f6fe00640 1 --2- 172.21.15.96:0/1007243098 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3f4c06c8f0 0x7f3f4c06edb0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f3f60006fd0 tx=0x7f3f60008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:03.469 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.469+0000 7f3f6cc00640 1 -- 172.21.15.96:0/1007243098 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3f58045360 con 0x7f3f700d46c0 2024-05-28T06:05:03.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.678+0000 7f3f76e00640 1 -- 172.21.15.96:0/1007243098 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f3f700d4cd0 con 0x7f3f700d46c0 2024-05-28T06:05:03.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.679+0000 7f3f6cc00640 1 -- 172.21.15.96:0/1007243098 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) ==== 80+0+0 (secure 0 0 0) 0x7f3f5807ac40 con 0x7f3f700d46c0 2024-05-28T06:05:03.680 INFO:teuthology.orchestra.run.smithi096.stderr:listed 0 entries 2024-05-28T06:05:03.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.682+0000 7f3f76e00640 1 -- 172.21.15.96:0/1007243098 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3f4c06c8f0 msgr2=0x7f3f4c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:03.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.682+0000 7f3f76e00640 1 --2- 172.21.15.96:0/1007243098 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3f4c06c8f0 0x7f3f4c06edb0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f3f60006fd0 tx=0x7f3f60008040 comp rx=0 tx=0).stop 2024-05-28T06:05:03.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.682+0000 7f3f76e00640 1 -- 172.21.15.96:0/1007243098 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3f700d46c0 msgr2=0x7f3f700cd590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:03.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.682+0000 7f3f76e00640 1 --2- 172.21.15.96:0/1007243098 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3f700d46c0 0x7f3f700cd590 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f3f5802a220 tx=0x7f3f580046c0 comp rx=0 tx=0).stop 2024-05-28T06:05:03.683 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.682+0000 7f3f76e00640 1 -- 172.21.15.96:0/1007243098 shutdown_connections 2024-05-28T06:05:03.683 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.682+0000 7f3f76e00640 1 --2- 172.21.15.96:0/1007243098 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3f4c06c8f0 0x7f3f4c06edb0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:03.683 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.682+0000 7f3f76e00640 1 --2- 172.21.15.96:0/1007243098 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f7015ed00 0x7f3f700b94f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:03.683 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.682+0000 7f3f76e00640 1 --2- 172.21.15.96:0/1007243098 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3f700d46c0 0x7f3f700cd590 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:03.683 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.682+0000 7f3f76e00640 1 --2- 172.21.15.96:0/1007243098 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3f700d0d60 0x7f3f700cfcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:03.683 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.682+0000 7f3f76e00640 1 -- 172.21.15.96:0/1007243098 >> 172.21.15.96:0/1007243098 conn(0x7f3f7009da70 msgr2=0x7f3f700b1440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:03.683 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.682+0000 7f3f76e00640 1 -- 172.21.15.96:0/1007243098 shutdown_connections 2024-05-28T06:05:03.683 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:03.682+0000 7f3f76e00640 1 -- 172.21.15.96:0/1007243098 wait complete. 2024-05-28T06:05:03.700 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi123.front.sepia.ceph.com... 2024-05-28T06:05:03.700 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:05:03.701 DEBUG:teuthology.orchestra.run.smithi123:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-05-28T06:05:03.793 INFO:teuthology.orchestra.run:waiting for 300 2024-05-28T06:05:03.793 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi123.front.sepia.ceph.com 2024-05-28T06:05:03.793 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:05:03.793 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-05-28T06:05:03.857 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:05:03.857 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:05:03.914 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:05:03.914 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:05:03.967 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:05:04.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.156+0000 7f34b3200640 1 Processor -- start 2024-05-28T06:05:04.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.157+0000 7f34b3200640 1 -- start start 2024-05-28T06:05:04.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.158+0000 7f34b3200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f34b40ad940 0x7f34b40b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:04.159 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.158+0000 7f34b3200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34b4156f10 con 0x7f34b406ee40 2024-05-28T06:05:04.159 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.158+0000 7f34b3200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34b406dda0 con 0x7f34b4156b30 2024-05-28T06:05:04.159 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.158+0000 7f34b3200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f34b406e080 con 0x7f34b40ad940 2024-05-28T06:05:04.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.159+0000 7f34abe00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f34b406ee40 0x7f34b40ad230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:60450/0 (socket says 172.21.15.96:60450) 2024-05-28T06:05:04.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.159+0000 7f34abe00640 1 -- 172.21.15.96:0/666317754 learned_addr learned my addr 172.21.15.96:0/666317754 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:05:04.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.159+0000 7f34b2800640 1 --2- 172.21.15.96:0/666317754 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f34b40ad940 0x7f34b40b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:04.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.159+0000 7f34b1400640 1 -- 172.21.15.96:0/666317754 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2020567552 0 0) 0x7f34b4156f10 con 0x7f34b406ee40 2024-05-28T06:05:04.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.159+0000 7f34b1400640 1 -- 172.21.15.96:0/666317754 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3494003520 con 0x7f34b406ee40 2024-05-28T06:05:04.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.159+0000 7f34b1400640 1 -- 172.21.15.96:0/666317754 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 676183476 0 0) 0x7f3494003520 con 0x7f34b406ee40 2024-05-28T06:05:04.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.159+0000 7f34b1400640 1 -- 172.21.15.96:0/666317754 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f34b40ad940 msgr2=0x7f34b40b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:04.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.159+0000 7f34b1400640 1 --2- 172.21.15.96:0/666317754 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f34b40ad940 0x7f34b40b32c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:04.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.159+0000 7f34b1400640 1 -- 172.21.15.96:0/666317754 >> v1:172.21.15.123:6789/0 conn(0x7f34b4156b30 legacy=0x7f34b40af6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:04.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.159+0000 7f34b1400640 1 -- 172.21.15.96:0/666317754 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34b406e360 con 0x7f34b406ee40 2024-05-28T06:05:04.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.159+0000 7f34b1400640 1 -- 172.21.15.96:0/666317754 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f349c001670 con 0x7f34b406ee40 2024-05-28T06:05:04.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.160+0000 7f34b1400640 1 -- 172.21.15.96:0/666317754 >> v1:172.21.15.96:6789/0 conn(0x7f34b406ee40 legacy=0x7f34b40ad230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:04.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.160+0000 7f34b1400640 1 --2- 172.21.15.96:0/666317754 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f34b40ad940 0x7f34940039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:04.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.160+0000 7f34b1400640 1 --2- 172.21.15.96:0/666317754 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3494003f60 0x7f3494004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:04.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.160+0000 7f34b1400640 1 --2- 172.21.15.96:0/666317754 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3494004d40 0x7f34940051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:04.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.160+0000 7f34b1400640 1 -- 172.21.15.96:0/666317754 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f34b406e080 con 0x7f3494004d40 2024-05-28T06:05:04.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.160+0000 7f34b1400640 1 -- 172.21.15.96:0/666317754 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f349400bb50 con 0x7f3494003f60 2024-05-28T06:05:04.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.160+0000 7f34b1400640 1 -- 172.21.15.96:0/666317754 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f349400bd50 con 0x7f34b40ad940 2024-05-28T06:05:04.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.160+0000 7f34b2800640 1 --2- 172.21.15.96:0/666317754 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3494004d40 0x7f34940051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:04.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.160+0000 7f34abe00640 1 --2- 172.21.15.96:0/666317754 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3494003f60 0x7f3494004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:04.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.160+0000 7f34b2800640 1 -- 172.21.15.96:0/666317754 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f34b40ad940 msgr2=0x7f34940039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:04.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.160+0000 7f34b1e00640 1 --2- 172.21.15.96:0/666317754 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f34b40ad940 0x7f34940039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:04.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.160+0000 7f34b2800640 1 --2- 172.21.15.96:0/666317754 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f34b40ad940 0x7f34940039e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:04.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.160+0000 7f34b2800640 1 -- 172.21.15.96:0/666317754 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3494003f60 msgr2=0x7f3494004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:04.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.160+0000 7f34b2800640 1 --2- 172.21.15.96:0/666317754 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3494003f60 0x7f3494004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:04.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.160+0000 7f34b2800640 1 -- 172.21.15.96:0/666317754 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3494003520 con 0x7f3494004d40 2024-05-28T06:05:04.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.160+0000 7f34abe00640 1 --2- 172.21.15.96:0/666317754 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3494003f60 0x7f3494004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:05:04.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.161+0000 7f34b2800640 1 --2- 172.21.15.96:0/666317754 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3494004d40 0x7f34940051c0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f34ac009f90 tx=0x7f3494005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8951263e38092051 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:04.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.162+0000 7f34b1400640 1 -- 172.21.15.96:0/666317754 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34ac009c10 con 0x7f3494004d40 2024-05-28T06:05:04.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.162+0000 7f34b1400640 1 -- 172.21.15.96:0/666317754 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34ac036070 con 0x7f3494004d40 2024-05-28T06:05:04.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.162+0000 7f34b1400640 1 -- 172.21.15.96:0/666317754 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34ac031810 con 0x7f3494004d40 2024-05-28T06:05:04.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.162+0000 7f34b3200640 1 -- 172.21.15.96:0/666317754 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3494004d40 msgr2=0x7f34940051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:04.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.162+0000 7f34b3200640 1 --2- 172.21.15.96:0/666317754 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3494004d40 0x7f34940051c0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f34ac009f90 tx=0x7f3494005830 comp rx=0 tx=0).stop 2024-05-28T06:05:04.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.162+0000 7f34b2800640 1 -- 172.21.15.96:0/666317754 reap_dead start 2024-05-28T06:05:04.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.162+0000 7f34b3200640 1 -- 172.21.15.96:0/666317754 shutdown_connections 2024-05-28T06:05:04.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.162+0000 7f34b3200640 1 -- 172.21.15.96:0/666317754 >> 172.21.15.96:0/666317754 conn(0x7f34b40b1fc0 msgr2=0x7f34b40b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:04.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.162+0000 7f34b3200640 1 -- 172.21.15.96:0/666317754 shutdown_connections 2024-05-28T06:05:04.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.162+0000 7f34b3200640 1 -- 172.21.15.96:0/666317754 wait complete. 2024-05-28T06:05:04.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.164+0000 7f34b3200640 1 Processor -- start 2024-05-28T06:05:04.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.164+0000 7f34b3200640 1 -- start start 2024-05-28T06:05:04.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.165+0000 7f34b3200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3494004d40 0x7f34b409de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:04.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.165+0000 7f34b3200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f34b4149810 0x7f34b4148fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:04.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.165+0000 7f34b3200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f34b40b1050 0x7f34b414ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:04.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.165+0000 7f34b3200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f34b40b2db0 con 0x7f34b40b1050 2024-05-28T06:05:04.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.165+0000 7f34b3200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f34b40b2c30 con 0x7f3494004d40 2024-05-28T06:05:04.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.165+0000 7f34b3200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f34b406f2f0 con 0x7f34b4149810 2024-05-28T06:05:04.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.165+0000 7f34b2800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f34b40b1050 0x7f34b414ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:04.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.165+0000 7f34b2800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f34b40b1050 0x7f34b414ba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54720/0 (socket says 172.21.15.96:54720) 2024-05-28T06:05:04.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.165+0000 7f34b2800640 1 -- 172.21.15.96:0/3092023344 learned_addr learned my addr 172.21.15.96:0/3092023344 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:04.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.166+0000 7f34abe00640 1 --2- 172.21.15.96:0/3092023344 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f34b4149810 0x7f34b4148fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:04.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.166+0000 7f34b1e00640 1 --2- 172.21.15.96:0/3092023344 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3494004d40 0x7f34b409de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:04.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.166+0000 7f34b2800640 1 -- 172.21.15.96:0/3092023344 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f34b4149810 msgr2=0x7f34b4148fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:04.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.166+0000 7f34b2800640 1 --2- 172.21.15.96:0/3092023344 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f34b4149810 0x7f34b4148fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:04.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.166+0000 7f34b2800640 1 -- 172.21.15.96:0/3092023344 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3494004d40 msgr2=0x7f34b409de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:04.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.166+0000 7f34b2800640 1 --2- 172.21.15.96:0/3092023344 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3494004d40 0x7f34b409de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:04.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.166+0000 7f34b2800640 1 -- 172.21.15.96:0/3092023344 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34b40b0290 con 0x7f34b40b1050 2024-05-28T06:05:04.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.166+0000 7f34abe00640 1 --2- 172.21.15.96:0/3092023344 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f34b4149810 0x7f34b4148fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:05:04.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.166+0000 7f34b1e00640 1 --2- 172.21.15.96:0/3092023344 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3494004d40 0x7f34b409de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:05:04.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.166+0000 7f34b2800640 1 --2- 172.21.15.96:0/3092023344 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f34b40b1050 0x7f34b414ba60 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f34ac02aeb0 tx=0x7f34ac009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:04.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.166+0000 7f34aa000640 1 -- 172.21.15.96:0/3092023344 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34ac009c10 con 0x7f34b40b1050 2024-05-28T06:05:04.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.166+0000 7f34aa000640 1 -- 172.21.15.96:0/3092023344 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34ac045430 con 0x7f34b40b1050 2024-05-28T06:05:04.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.166+0000 7f34b3200640 1 -- 172.21.15.96:0/3092023344 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34b40b0570 con 0x7f34b40b1050 2024-05-28T06:05:04.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.166+0000 7f34aa000640 1 -- 172.21.15.96:0/3092023344 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34ac045820 con 0x7f34b40b1050 2024-05-28T06:05:04.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.167+0000 7f34b3200640 1 -- 172.21.15.96:0/3092023344 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34b40b0b30 con 0x7f34b40b1050 2024-05-28T06:05:04.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.168+0000 7f34b3200640 1 -- 172.21.15.96:0/3092023344 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34b40b0d30 con 0x7f34b40b1050 2024-05-28T06:05:04.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.169+0000 7f34aa000640 1 -- 172.21.15.96:0/3092023344 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f34ac005d10 con 0x7f34b40b1050 2024-05-28T06:05:04.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.169+0000 7f34aa000640 1 --2- 172.21.15.96:0/3092023344 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f349006c8f0 0x7f349006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:04.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.170+0000 7f34aa000640 1 -- 172.21.15.96:0/3092023344 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5825+0+0 (secure 0 0 0) 0x7f34ac0acc00 con 0x7f34b40b1050 2024-05-28T06:05:04.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.170+0000 7f34b1e00640 1 --2- 172.21.15.96:0/3092023344 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f349006c8f0 0x7f349006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:04.171 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.171+0000 7f34b1e00640 1 --2- 172.21.15.96:0/3092023344 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f349006c8f0 0x7f349006edb0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f34a40016f0 tx=0x7f34a4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:04.174 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.173+0000 7f34aa000640 1 -- 172.21.15.96:0/3092023344 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f34b40b0d30 con 0x7f34b40b1050 2024-05-28T06:05:04.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.361+0000 7f34b3200640 1 -- 172.21.15.96:0/3092023344 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f34b40cfce0 con 0x7f34b40b1050 2024-05-28T06:05:04.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.362+0000 7f34aa000640 1 -- 172.21.15.96:0/3092023344 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) ==== 74+0+16865 (secure 0 0 0) 0x7f34ac02e050 con 0x7f34b40b1050 2024-05-28T06:05:04.363 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:05:04.363 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":22,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:04:52.624604+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-28T06:04:08.741938+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-28T06:04:09.301470+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.b","mappings":[{"from":3,"to":1}]},{"pgid":"3.d","mappings":[{"from":7,"to":2}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:05:04.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.364+0000 7f34b3200640 1 -- 172.21.15.96:0/3092023344 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f349006c8f0 msgr2=0x7f349006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:04.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.364+0000 7f34b3200640 1 --2- 172.21.15.96:0/3092023344 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f349006c8f0 0x7f349006edb0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f34a40016f0 tx=0x7f34a4009040 comp rx=0 tx=0).stop 2024-05-28T06:05:04.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.364+0000 7f34b3200640 1 -- 172.21.15.96:0/3092023344 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f34b40b1050 msgr2=0x7f34b414ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:04.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.364+0000 7f34b3200640 1 --2- 172.21.15.96:0/3092023344 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f34b40b1050 0x7f34b414ba60 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f34ac02aeb0 tx=0x7f34ac009750 comp rx=0 tx=0).stop 2024-05-28T06:05:04.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.365+0000 7f34b3200640 1 -- 172.21.15.96:0/3092023344 shutdown_connections 2024-05-28T06:05:04.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.365+0000 7f34b3200640 1 --2- 172.21.15.96:0/3092023344 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f349006c8f0 0x7f349006edb0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:04.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.365+0000 7f34b3200640 1 --2- 172.21.15.96:0/3092023344 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f34b40b1050 0x7f34b414ba60 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:04.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.365+0000 7f34b3200640 1 --2- 172.21.15.96:0/3092023344 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f34b4149810 0x7f34b4148fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:04.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.365+0000 7f34b3200640 1 --2- 172.21.15.96:0/3092023344 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3494004d40 0x7f34b409de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:04.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.365+0000 7f34b3200640 1 -- 172.21.15.96:0/3092023344 >> 172.21.15.96:0/3092023344 conn(0x7f34b40b1fc0 msgr2=0x7f34b415c920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:04.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.365+0000 7f34b3200640 1 -- 172.21.15.96:0/3092023344 shutdown_connections 2024-05-28T06:05:04.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.365+0000 7f34b3200640 1 -- 172.21.15.96:0/3092023344 wait complete. 2024-05-28T06:05:04.384 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-28T06:03:59.517102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-28T06:04:08.741938+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-28T06:04:09.301470+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-28T06:05:04.384 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-28T06:05:04.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.586+0000 7f08c4a00640 1 Processor -- start 2024-05-28T06:05:04.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.587+0000 7f08c4a00640 1 -- start start 2024-05-28T06:05:04.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.588+0000 7f08c4a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f08c00d16a0 0x7f08c00cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:04.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.588+0000 7f08c4a00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08c015ec50 con 0x7f08c015e870 2024-05-28T06:05:04.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.588+0000 7f08c4a00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08c00cff80 con 0x7f08c00cf870 2024-05-28T06:05:04.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.588+0000 7f08c4a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f08c00d2150 con 0x7f08c00d16a0 2024-05-28T06:05:04.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.588+0000 7f08bf400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f08c015e870 0x7f08c00d0f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:60452/0 (socket says 172.21.15.96:60452) 2024-05-28T06:05:04.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.588+0000 7f08bf400640 1 -- 172.21.15.96:0/399857664 learned_addr learned my addr 172.21.15.96:0/399857664 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:05:04.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.589+0000 7f08b6a00640 1 --2- 172.21.15.96:0/399857664 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f08c00d16a0 0x7f08c00cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:04.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.589+0000 7f08bea00640 1 -- 172.21.15.96:0/399857664 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3395895331 0 0) 0x7f08c015ec50 con 0x7f08c015e870 2024-05-28T06:05:04.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.589+0000 7f08bea00640 1 -- 172.21.15.96:0/399857664 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f08a4003520 con 0x7f08c015e870 2024-05-28T06:05:04.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.589+0000 7f08b6a00640 1 -- 172.21.15.96:0/399857664 >> v1:172.21.15.123:6789/0 conn(0x7f08c00cf870 legacy=0x7f08c00d25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:04.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.589+0000 7f08b6a00640 1 -- 172.21.15.96:0/399857664 >> v1:172.21.15.96:6789/0 conn(0x7f08c015e870 legacy=0x7f08c00d0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:04.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.589+0000 7f08bea00640 1 -- 172.21.15.96:0/399857664 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1748091838 0 0) 0x7f08a4003520 con 0x7f08c015e870 2024-05-28T06:05:04.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.589+0000 7f08b6a00640 1 -- 172.21.15.96:0/399857664 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08c00d0260 con 0x7f08c00d16a0 2024-05-28T06:05:04.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.590+0000 7f08b6a00640 1 --2- 172.21.15.96:0/399857664 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f08c00d16a0 0x7f08c00cf330 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f08a80097c0 tx=0x7f08a802a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4565fdce405d1554 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:04.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.591+0000 7f08bea00640 1 -- 172.21.15.96:0/399857664 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08a8037070 con 0x7f08c00d16a0 2024-05-28T06:05:04.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.591+0000 7f08bea00640 1 -- 172.21.15.96:0/399857664 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08a802ac10 con 0x7f08c00d16a0 2024-05-28T06:05:04.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.591+0000 7f08bea00640 1 -- 172.21.15.96:0/399857664 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08b0001670 con 0x7f08c00d16a0 2024-05-28T06:05:04.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.592+0000 7f08c4a00640 1 -- 172.21.15.96:0/399857664 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f08c00d16a0 msgr2=0x7f08c00cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:04.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.592+0000 7f08c4a00640 1 --2- 172.21.15.96:0/399857664 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f08c00d16a0 0x7f08c00cf330 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f08a80097c0 tx=0x7f08a802a180 comp rx=0 tx=0).stop 2024-05-28T06:05:04.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.592+0000 7f08c4a00640 1 -- 172.21.15.96:0/399857664 shutdown_connections 2024-05-28T06:05:04.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.592+0000 7f08c4a00640 1 --2- 172.21.15.96:0/399857664 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f08c00d16a0 0x7f08c00cf330 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:04.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.592+0000 7f08c4a00640 1 -- 172.21.15.96:0/399857664 >> 172.21.15.96:0/399857664 conn(0x7f08c008bfc0 msgr2=0x7f08c00d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:04.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.592+0000 7f08c4a00640 1 -- 172.21.15.96:0/399857664 shutdown_connections 2024-05-28T06:05:04.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.592+0000 7f08c4a00640 1 -- 172.21.15.96:0/399857664 wait complete. 2024-05-28T06:05:04.594 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.594+0000 7f08c4a00640 1 Processor -- start 2024-05-28T06:05:04.594 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.594+0000 7f08c4a00640 1 -- start start 2024-05-28T06:05:04.595 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.595+0000 7f08c4a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f08c00cf870 0x7f08c0159c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:04.595 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.595+0000 7f08c4a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f08c00d16a0 0x7f08c00cb650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:04.595 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.595+0000 7f08c4a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08c015e870 0x7f08c015f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:04.595 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.595+0000 7f08c4a00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f08c00d0820 con 0x7f08c00cf870 2024-05-28T06:05:04.595 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.595+0000 7f08c4a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f08c00cfd20 con 0x7f08c015e870 2024-05-28T06:05:04.595 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.595+0000 7f08c4a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f08c00d4fb0 con 0x7f08c00d16a0 2024-05-28T06:05:04.595 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.595+0000 7f08bf400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f08c00cf870 0x7f08c0159c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:04.596 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.595+0000 7f08bf400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f08c00cf870 0x7f08c0159c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54724/0 (socket says 172.21.15.96:54724) 2024-05-28T06:05:04.596 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.595+0000 7f08bf400640 1 -- 172.21.15.96:0/1561681894 learned_addr learned my addr 172.21.15.96:0/1561681894 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:04.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.595+0000 7f08bfe00640 1 --2- 172.21.15.96:0/1561681894 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08c015e870 0x7f08c015f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:04.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.595+0000 7f08bf400640 1 -- 172.21.15.96:0/1561681894 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f08c00d16a0 msgr2=0x7f08c00cb650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:04.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.595+0000 7f08bf400640 1 --2- 172.21.15.96:0/1561681894 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f08c00d16a0 0x7f08c00cb650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:04.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.595+0000 7f08bf400640 1 -- 172.21.15.96:0/1561681894 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08c015e870 msgr2=0x7f08c015f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:04.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.595+0000 7f08bf400640 1 --2- 172.21.15.96:0/1561681894 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08c015e870 0x7f08c015f570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:04.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.595+0000 7f08bf400640 1 -- 172.21.15.96:0/1561681894 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08c00cbea0 con 0x7f08c00cf870 2024-05-28T06:05:04.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.596+0000 7f08bf400640 1 --2- 172.21.15.96:0/1561681894 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f08c00cf870 0x7f08c0159c00 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f08b0001eb0 tx=0x7f08b000c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:04.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.596+0000 7f08b7e00640 1 -- 172.21.15.96:0/1561681894 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08b0017070 con 0x7f08c00cf870 2024-05-28T06:05:04.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.596+0000 7f08b7e00640 1 -- 172.21.15.96:0/1561681894 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08b0005ce0 con 0x7f08c00cf870 2024-05-28T06:05:04.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.596+0000 7f08b7e00640 1 -- 172.21.15.96:0/1561681894 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08b0007290 con 0x7f08c00cf870 2024-05-28T06:05:04.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.596+0000 7f08c4a00640 1 -- 172.21.15.96:0/1561681894 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08c00cc180 con 0x7f08c00cf870 2024-05-28T06:05:04.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.596+0000 7f08c4a00640 1 -- 172.21.15.96:0/1561681894 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08c00cbb90 con 0x7f08c00cf870 2024-05-28T06:05:04.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.597+0000 7f08c4a00640 1 -- 172.21.15.96:0/1561681894 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08c00689e0 con 0x7f08c00cf870 2024-05-28T06:05:04.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.598+0000 7f08b7e00640 1 -- 172.21.15.96:0/1561681894 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f08b001a070 con 0x7f08c00cf870 2024-05-28T06:05:04.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.599+0000 7f08b7e00640 1 --2- 172.21.15.96:0/1561681894 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f089c06c920 0x7f089c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:04.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.599+0000 7f08b7e00640 1 -- 172.21.15.96:0/1561681894 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5825+0+0 (secure 0 0 0) 0x7f08b008b080 con 0x7f08c00cf870 2024-05-28T06:05:04.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.600+0000 7f08b6a00640 1 --2- 172.21.15.96:0/1561681894 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f089c06c920 0x7f089c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:04.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.601+0000 7f08b6a00640 1 --2- 172.21.15.96:0/1561681894 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f089c06c920 0x7f089c06ede0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f08a8034000 tx=0x7f08a802c000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:04.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.603+0000 7f08b7e00640 1 -- 172.21.15.96:0/1561681894 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f08b0059390 con 0x7f08c00cf870 2024-05-28T06:05:04.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.800+0000 7f08c4a00640 1 -- 172.21.15.96:0/1561681894 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f08c00ba500 con 0x7f08c00cf870 2024-05-28T06:05:04.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.801+0000 7f08b7e00640 1 -- 172.21.15.96:0/1561681894 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) ==== 93+0+10 (secure 0 0 0) 0x7f08b0058c90 con 0x7f08c00cf870 2024-05-28T06:05:04.802 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 1 2024-05-28T06:05:04.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.804+0000 7f08c4a00640 1 -- 172.21.15.96:0/1561681894 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f089c06c920 msgr2=0x7f089c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:04.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.804+0000 7f08c4a00640 1 --2- 172.21.15.96:0/1561681894 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f089c06c920 0x7f089c06ede0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f08a8034000 tx=0x7f08a802c000 comp rx=0 tx=0).stop 2024-05-28T06:05:04.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.804+0000 7f08c4a00640 1 -- 172.21.15.96:0/1561681894 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f08c00cf870 msgr2=0x7f08c0159c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:04.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.804+0000 7f08c4a00640 1 --2- 172.21.15.96:0/1561681894 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f08c00cf870 0x7f08c0159c00 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f08b0001eb0 tx=0x7f08b000c9b0 comp rx=0 tx=0).stop 2024-05-28T06:05:04.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.804+0000 7f08c4a00640 1 -- 172.21.15.96:0/1561681894 shutdown_connections 2024-05-28T06:05:04.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.804+0000 7f08c4a00640 1 --2- 172.21.15.96:0/1561681894 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f089c06c920 0x7f089c06ede0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:04.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.804+0000 7f08c4a00640 1 --2- 172.21.15.96:0/1561681894 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08c015e870 0x7f08c015f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:04.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.804+0000 7f08c4a00640 1 --2- 172.21.15.96:0/1561681894 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f08c00d16a0 0x7f08c00cb650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:04.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.804+0000 7f08c4a00640 1 --2- 172.21.15.96:0/1561681894 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f08c00cf870 0x7f08c0159c00 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:04.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.804+0000 7f08c4a00640 1 -- 172.21.15.96:0/1561681894 >> 172.21.15.96:0/1561681894 conn(0x7f08c008bfc0 msgr2=0x7f08c00d25e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:04.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.804+0000 7f08c4a00640 1 -- 172.21.15.96:0/1561681894 shutdown_connections 2024-05-28T06:05:04.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:04.804+0000 7f08c4a00640 1 -- 172.21.15.96:0/1561681894 wait complete. 2024-05-28T06:05:04.824 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-28T06:05:05.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.016+0000 7f4468c00640 1 Processor -- start 2024-05-28T06:05:05.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.016+0000 7f4468c00640 1 -- start start 2024-05-28T06:05:05.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.018+0000 7f4468c00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f44640caa40 0x7f44640ce960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:05.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.018+0000 7f4468c00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f44640cae20 0x7f4464151db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:05.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.018+0000 7f4468c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f44641522f0 0x7f44640cf710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:05.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.018+0000 7f4468c00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f44640d0b30 con 0x7f44640caa40 2024-05-28T06:05:05.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.018+0000 7f4468c00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f44640b4160 con 0x7f44640cae20 2024-05-28T06:05:05.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.018+0000 7f4468c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f4464152760 con 0x7f44641522f0 2024-05-28T06:05:05.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.018+0000 7f4463400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f44640caa40 0x7f44640ce960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:05.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.018+0000 7f4463400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f44640caa40 0x7f44640ce960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54732/0 (socket says 172.21.15.96:54732) 2024-05-28T06:05:05.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.018+0000 7f4463400640 1 -- 172.21.15.96:0/376699348 learned_addr learned my addr 172.21.15.96:0/376699348 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:05.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.019+0000 7f4463400640 1 -- 172.21.15.96:0/376699348 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f44641522f0 msgr2=0x7f44640cf710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:05.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.019+0000 7f4463400640 1 --2- 172.21.15.96:0/376699348 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f44641522f0 0x7f44640cf710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:05.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.019+0000 7f4463400640 1 -- 172.21.15.96:0/376699348 >> v2:172.21.15.123:3300/0 conn(0x7f44640cae20 msgr2=0x7f4464151db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:05.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.019+0000 7f445be00640 1 --2- 172.21.15.96:0/376699348 >> v2:172.21.15.123:3300/0 conn(0x7f44640cae20 0x7f4464151db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:05.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.019+0000 7f4463400640 1 --2- 172.21.15.96:0/376699348 >> v2:172.21.15.123:3300/0 conn(0x7f44640cae20 0x7f4464151db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:05.022 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.019+0000 7f4463400640 1 -- 172.21.15.96:0/376699348 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44640b3e80 con 0x7f44640caa40 2024-05-28T06:05:05.022 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.019+0000 7f445be00640 1 --2- 172.21.15.96:0/376699348 >> v2:172.21.15.123:3300/0 conn(0x7f44640cae20 0x7f4464151db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:05:05.022 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.020+0000 7f4463400640 1 --2- 172.21.15.96:0/376699348 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f44640caa40 0x7f44640ce960 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f44640d0fb0 tx=0x7f445402a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e00aa99e3807bb8 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:05.022 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.020+0000 7f4462a00640 1 -- 172.21.15.96:0/376699348 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4454037070 con 0x7f44640caa40 2024-05-28T06:05:05.022 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.020+0000 7f4462a00640 1 -- 172.21.15.96:0/376699348 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4454032440 con 0x7f44640caa40 2024-05-28T06:05:05.022 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.020+0000 7f4462a00640 1 -- 172.21.15.96:0/376699348 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4454032810 con 0x7f44640caa40 2024-05-28T06:05:05.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.021+0000 7f4468c00640 1 -- 172.21.15.96:0/376699348 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f44640caa40 msgr2=0x7f44640ce960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:05.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.021+0000 7f4468c00640 1 --2- 172.21.15.96:0/376699348 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f44640caa40 0x7f44640ce960 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f44640d0fb0 tx=0x7f445402a140 comp rx=0 tx=0).stop 2024-05-28T06:05:05.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.021+0000 7f4468c00640 1 -- 172.21.15.96:0/376699348 shutdown_connections 2024-05-28T06:05:05.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.021+0000 7f4468c00640 1 --2- 172.21.15.96:0/376699348 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f44641522f0 0x7f44640cf710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:05.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.021+0000 7f4468c00640 1 --2- 172.21.15.96:0/376699348 >> v2:172.21.15.123:3300/0 conn(0x7f44640cae20 0x7f4464151db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:05.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.021+0000 7f4468c00640 1 --2- 172.21.15.96:0/376699348 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f44640caa40 0x7f44640ce960 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:05.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.021+0000 7f4468c00640 1 -- 172.21.15.96:0/376699348 >> 172.21.15.96:0/376699348 conn(0x7f446406b900 msgr2=0x7f446406bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:05.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.021+0000 7f4468c00640 1 -- 172.21.15.96:0/376699348 shutdown_connections 2024-05-28T06:05:05.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.021+0000 7f4468c00640 1 -- 172.21.15.96:0/376699348 wait complete. 2024-05-28T06:05:05.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.023+0000 7f4468c00640 1 Processor -- start 2024-05-28T06:05:05.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.023+0000 7f4468c00640 1 -- start start 2024-05-28T06:05:05.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.025+0000 7f4468c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f44640caa40 0x7f4464142470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:05.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.025+0000 7f4468c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44640cae20 0x7f4464072f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:05.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.025+0000 7f4468c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f44641522f0 0x7f4464073450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:05.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.025+0000 7f4468c00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f44640ce440 con 0x7f44641522f0 2024-05-28T06:05:05.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.025+0000 7f4468c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f44640ce160 con 0x7f44640cae20 2024-05-28T06:05:05.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.025+0000 7f4468c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f44640ceea0 con 0x7f44640caa40 2024-05-28T06:05:05.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.025+0000 7f4463e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f44641522f0 0x7f4464073450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:05.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.025+0000 7f4463e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f44641522f0 0x7f4464073450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54748/0 (socket says 172.21.15.96:54748) 2024-05-28T06:05:05.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.025+0000 7f4463e00640 1 -- 172.21.15.96:0/844669236 learned_addr learned my addr 172.21.15.96:0/844669236 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:05.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.025+0000 7f4463400640 1 --2- 172.21.15.96:0/844669236 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f44640caa40 0x7f4464142470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:05.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.025+0000 7f4463e00640 1 -- 172.21.15.96:0/844669236 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f44640caa40 msgr2=0x7f4464142470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:05.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.025+0000 7f445be00640 1 --2- 172.21.15.96:0/844669236 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44640cae20 0x7f4464072f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:05.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.025+0000 7f4463e00640 1 --2- 172.21.15.96:0/844669236 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f44640caa40 0x7f4464142470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:05.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.025+0000 7f4463e00640 1 -- 172.21.15.96:0/844669236 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44640cae20 msgr2=0x7f4464072f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:05.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.025+0000 7f4463e00640 1 --2- 172.21.15.96:0/844669236 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44640cae20 0x7f4464072f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:05.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.025+0000 7f4463e00640 1 -- 172.21.15.96:0/844669236 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44641429b0 con 0x7f44641522f0 2024-05-28T06:05:05.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.025+0000 7f4463400640 1 --2- 172.21.15.96:0/844669236 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f44640caa40 0x7f4464142470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:05:05.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.025+0000 7f445be00640 1 --2- 172.21.15.96:0/844669236 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44640cae20 0x7f4464072f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:05:05.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.026+0000 7f4463e00640 1 --2- 172.21.15.96:0/844669236 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f44641522f0 0x7f4464073450 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f445c00b970 tx=0x7f445c00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:05.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.026+0000 7f445b400640 1 -- 172.21.15.96:0/844669236 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f445c019070 con 0x7f44641522f0 2024-05-28T06:05:05.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.026+0000 7f445b400640 1 -- 172.21.15.96:0/844669236 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f445c00cb60 con 0x7f44641522f0 2024-05-28T06:05:05.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.026+0000 7f445b400640 1 -- 172.21.15.96:0/844669236 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f445c014440 con 0x7f44641522f0 2024-05-28T06:05:05.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.026+0000 7f4468c00640 1 -- 172.21.15.96:0/844669236 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f446409d2e0 con 0x7f44641522f0 2024-05-28T06:05:05.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.026+0000 7f4468c00640 1 -- 172.21.15.96:0/844669236 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f446409d8a0 con 0x7f44641522f0 2024-05-28T06:05:05.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.027+0000 7f4468c00640 1 -- 172.21.15.96:0/844669236 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44640689e0 con 0x7f44641522f0 2024-05-28T06:05:05.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.028+0000 7f445b400640 1 -- 172.21.15.96:0/844669236 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f445c01c070 con 0x7f44641522f0 2024-05-28T06:05:05.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.029+0000 7f445b400640 1 --2- 172.21.15.96:0/844669236 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f444006c8f0 0x7f444006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:05.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.029+0000 7f445b400640 1 -- 172.21.15.96:0/844669236 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5825+0+0 (secure 0 0 0) 0x7f445c08b9b0 con 0x7f44641522f0 2024-05-28T06:05:05.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.030+0000 7f4463400640 1 --2- 172.21.15.96:0/844669236 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f444006c8f0 0x7f444006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:05.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.031+0000 7f4463400640 1 --2- 172.21.15.96:0/844669236 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f444006c8f0 0x7f444006edb0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f4454034000 tx=0x7f445402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:05.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.034+0000 7f445b400640 1 -- 172.21.15.96:0/844669236 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f445c059cc0 con 0x7f44641522f0 2024-05-28T06:05:05.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.234+0000 7f4468c00640 1 -- 172.21.15.96:0/844669236 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f44640cfa30 con 0x7f44641522f0 2024-05-28T06:05:05.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.235+0000 7f445b400640 1 -- 172.21.15.96:0/844669236 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) ==== 104+0+11 (secure 0 0 0) 0x7f445c0595c0 con 0x7f44641522f0 2024-05-28T06:05:05.236 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 32 2024-05-28T06:05:05.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.238+0000 7f4468c00640 1 -- 172.21.15.96:0/844669236 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f444006c8f0 msgr2=0x7f444006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:05.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.238+0000 7f4468c00640 1 --2- 172.21.15.96:0/844669236 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f444006c8f0 0x7f444006edb0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f4454034000 tx=0x7f445402e000 comp rx=0 tx=0).stop 2024-05-28T06:05:05.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.238+0000 7f4468c00640 1 -- 172.21.15.96:0/844669236 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f44641522f0 msgr2=0x7f4464073450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:05.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.238+0000 7f4468c00640 1 --2- 172.21.15.96:0/844669236 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f44641522f0 0x7f4464073450 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f445c00b970 tx=0x7f445c00be30 comp rx=0 tx=0).stop 2024-05-28T06:05:05.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.238+0000 7f4468c00640 1 -- 172.21.15.96:0/844669236 shutdown_connections 2024-05-28T06:05:05.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.238+0000 7f4468c00640 1 --2- 172.21.15.96:0/844669236 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f444006c8f0 0x7f444006edb0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:05.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.238+0000 7f4468c00640 1 --2- 172.21.15.96:0/844669236 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f44641522f0 0x7f4464073450 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:05.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.238+0000 7f4468c00640 1 --2- 172.21.15.96:0/844669236 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44640cae20 0x7f4464072f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:05.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.238+0000 7f4468c00640 1 --2- 172.21.15.96:0/844669236 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f44640caa40 0x7f4464142470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:05.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.238+0000 7f4468c00640 1 -- 172.21.15.96:0/844669236 >> 172.21.15.96:0/844669236 conn(0x7f446406b900 msgr2=0x7f446415a080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:05.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.238+0000 7f4468c00640 1 -- 172.21.15.96:0/844669236 shutdown_connections 2024-05-28T06:05:05.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.238+0000 7f4468c00640 1 -- 172.21.15.96:0/844669236 wait complete. 2024-05-28T06:05:05.258 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-28T06:05:05.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.455+0000 7f3e2f600640 1 Processor -- start 2024-05-28T06:05:05.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.455+0000 7f3e2f600640 1 -- start start 2024-05-28T06:05:05.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.457+0000 7f3e2f600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3e2815ed80 0x7f3e280cff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:05.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.457+0000 7f3e2f600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f3e280b7e50 0x7f3e280cedb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:05.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.457+0000 7f3e2f600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f3e280b61e0 0x7f3e280b6670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:05.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.457+0000 7f3e2f600640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f3e28163540 con 0x7f3e280b61e0 2024-05-28T06:05:05.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.457+0000 7f3e2f600640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f3e2815f160 con 0x7f3e280b7e50 2024-05-28T06:05:05.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.457+0000 7f3e2f600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f3e28162930 con 0x7f3e2815ed80 2024-05-28T06:05:05.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.457+0000 7f3e2d600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f3e280b61e0 0x7f3e280b6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:05.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.457+0000 7f3e2d600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f3e280b61e0 0x7f3e280b6670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54756/0 (socket says 172.21.15.96:54756) 2024-05-28T06:05:05.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.457+0000 7f3e2d600640 1 -- 172.21.15.96:0/1931502221 learned_addr learned my addr 172.21.15.96:0/1931502221 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:05.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.457+0000 7f3e1fe00640 1 --2- 172.21.15.96:0/1931502221 >> v2:172.21.15.123:3300/0 conn(0x7f3e280b7e50 0x7f3e280cedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:05.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.457+0000 7f3e2cc00640 1 --2- 172.21.15.96:0/1931502221 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3e2815ed80 0x7f3e280cff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:05.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.457+0000 7f3e2d600640 1 -- 172.21.15.96:0/1931502221 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3e2815ed80 msgr2=0x7f3e280cff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:05.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.457+0000 7f3e2d600640 1 --2- 172.21.15.96:0/1931502221 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3e2815ed80 0x7f3e280cff80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:05.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.457+0000 7f3e2d600640 1 -- 172.21.15.96:0/1931502221 >> v2:172.21.15.123:3300/0 conn(0x7f3e280b7e50 msgr2=0x7f3e280cedb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:05.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.457+0000 7f3e2d600640 1 --2- 172.21.15.96:0/1931502221 >> v2:172.21.15.123:3300/0 conn(0x7f3e280b7e50 0x7f3e280cedb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:05.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.457+0000 7f3e2d600640 1 -- 172.21.15.96:0/1931502221 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e28162c10 con 0x7f3e280b61e0 2024-05-28T06:05:05.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.458+0000 7f3e2d600640 1 --2- 172.21.15.96:0/1931502221 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3e280b61e0 0x7f3e280b6670 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f3e2808cb90 tx=0x7f3e2002a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f80c1c5c660758cf server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:05.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.459+0000 7f3e27e00640 1 -- 172.21.15.96:0/1931502221 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e20036070 con 0x7f3e280b61e0 2024-05-28T06:05:05.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.459+0000 7f3e27e00640 1 -- 172.21.15.96:0/1931502221 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e20031600 con 0x7f3e280b61e0 2024-05-28T06:05:05.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.459+0000 7f3e27e00640 1 -- 172.21.15.96:0/1931502221 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e200319d0 con 0x7f3e280b61e0 2024-05-28T06:05:05.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.459+0000 7f3e2f600640 1 -- 172.21.15.96:0/1931502221 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3e280b61e0 msgr2=0x7f3e280b6670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:05.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.459+0000 7f3e2f600640 1 --2- 172.21.15.96:0/1931502221 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3e280b61e0 0x7f3e280b6670 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f3e2808cb90 tx=0x7f3e2002a6a0 comp rx=0 tx=0).stop 2024-05-28T06:05:05.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.459+0000 7f3e2f600640 1 -- 172.21.15.96:0/1931502221 shutdown_connections 2024-05-28T06:05:05.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.459+0000 7f3e2f600640 1 --2- 172.21.15.96:0/1931502221 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3e280b61e0 0x7f3e280b6670 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:05.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.459+0000 7f3e2f600640 1 --2- 172.21.15.96:0/1931502221 >> v2:172.21.15.123:3300/0 conn(0x7f3e280b7e50 0x7f3e280cedb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:05.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.459+0000 7f3e2f600640 1 --2- 172.21.15.96:0/1931502221 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3e2815ed80 0x7f3e280cff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:05.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.459+0000 7f3e2f600640 1 -- 172.21.15.96:0/1931502221 >> 172.21.15.96:0/1931502221 conn(0x7f3e280d13c0 msgr2=0x7f3e280d17d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:05.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.460+0000 7f3e2f600640 1 -- 172.21.15.96:0/1931502221 shutdown_connections 2024-05-28T06:05:05.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.460+0000 7f3e2f600640 1 -- 172.21.15.96:0/1931502221 wait complete. 2024-05-28T06:05:05.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.461+0000 7f3e2f600640 1 Processor -- start 2024-05-28T06:05:05.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.461+0000 7f3e2f600640 1 -- start start 2024-05-28T06:05:05.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.462+0000 7f3e2f600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3e280b61e0 0x7f3e281f7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:05.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.462+0000 7f3e2f600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3e280b7e50 0x7f3e28145220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:05.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.462+0000 7f3e2f600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3e2815ed80 0x7f3e28145760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:05.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.462+0000 7f3e2f600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f3e280d2b30 con 0x7f3e2815ed80 2024-05-28T06:05:05.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.462+0000 7f3e2f600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f3e281631d0 con 0x7f3e280b7e50 2024-05-28T06:05:05.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.462+0000 7f3e2f600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f3e280d2cb0 con 0x7f3e280b61e0 2024-05-28T06:05:05.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.462+0000 7f3e2d600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3e2815ed80 0x7f3e28145760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:05.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.462+0000 7f3e2d600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3e2815ed80 0x7f3e28145760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54764/0 (socket says 172.21.15.96:54764) 2024-05-28T06:05:05.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.463+0000 7f3e2d600640 1 -- 172.21.15.96:0/565909094 learned_addr learned my addr 172.21.15.96:0/565909094 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:05.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.463+0000 7f3e2cc00640 1 --2- 172.21.15.96:0/565909094 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3e280b61e0 0x7f3e281f7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:05.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.463+0000 7f3e2d600640 1 -- 172.21.15.96:0/565909094 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3e280b61e0 msgr2=0x7f3e281f7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:05.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.463+0000 7f3e1fe00640 1 --2- 172.21.15.96:0/565909094 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3e280b7e50 0x7f3e28145220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:05.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.463+0000 7f3e2d600640 1 --2- 172.21.15.96:0/565909094 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3e280b61e0 0x7f3e281f7c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:05.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.463+0000 7f3e2d600640 1 -- 172.21.15.96:0/565909094 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3e280b7e50 msgr2=0x7f3e28145220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:05.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.463+0000 7f3e2d600640 1 --2- 172.21.15.96:0/565909094 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3e280b7e50 0x7f3e28145220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:05.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.463+0000 7f3e2d600640 1 -- 172.21.15.96:0/565909094 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e280c9cf0 con 0x7f3e2815ed80 2024-05-28T06:05:05.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.463+0000 7f3e2d600640 1 --2- 172.21.15.96:0/565909094 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3e2815ed80 0x7f3e28145760 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f3e20009f40 tx=0x7f3e20004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:05.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.464+0000 7f3e25600640 1 -- 172.21.15.96:0/565909094 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e20036050 con 0x7f3e2815ed80 2024-05-28T06:05:05.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.464+0000 7f3e25600640 1 -- 172.21.15.96:0/565909094 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e20042070 con 0x7f3e2815ed80 2024-05-28T06:05:05.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.464+0000 7f3e25600640 1 -- 172.21.15.96:0/565909094 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e20009510 con 0x7f3e2815ed80 2024-05-28T06:05:05.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.464+0000 7f3e2f600640 1 -- 172.21.15.96:0/565909094 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e280c9fd0 con 0x7f3e2815ed80 2024-05-28T06:05:05.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.464+0000 7f3e2f600640 1 -- 172.21.15.96:0/565909094 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e280ca590 con 0x7f3e2815ed80 2024-05-28T06:05:05.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.465+0000 7f3e2f600640 1 -- 172.21.15.96:0/565909094 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e280689e0 con 0x7f3e2815ed80 2024-05-28T06:05:05.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.466+0000 7f3e25600640 1 -- 172.21.15.96:0/565909094 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3e20007bd0 con 0x7f3e2815ed80 2024-05-28T06:05:05.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.467+0000 7f3e25600640 1 --2- 172.21.15.96:0/565909094 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3e0406c8f0 0x7f3e0406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:05.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.467+0000 7f3e25600640 1 -- 172.21.15.96:0/565909094 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5825+0+0 (secure 0 0 0) 0x7f3e200ac9e0 con 0x7f3e2815ed80 2024-05-28T06:05:05.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.467+0000 7f3e2cc00640 1 --2- 172.21.15.96:0/565909094 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3e0406c8f0 0x7f3e0406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:05.470 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.469+0000 7f3e2cc00640 1 --2- 172.21.15.96:0/565909094 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3e0406c8f0 0x7f3e0406edb0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f3e28162390 tx=0x7f3e18008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:05.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.471+0000 7f3e25600640 1 -- 172.21.15.96:0/565909094 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3e2007ae20 con 0x7f3e2815ed80 2024-05-28T06:05:05.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.667+0000 7f3e2f600640 1 -- 172.21.15.96:0/565909094 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f3e280d2910 con 0x7f3e2815ed80 2024-05-28T06:05:05.669 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.668+0000 7f3e25600640 1 -- 172.21.15.96:0/565909094 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) ==== 100+0+11 (secure 0 0 0) 0x7f3e2007a720 con 0x7f3e2815ed80 2024-05-28T06:05:05.669 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 64 2024-05-28T06:05:05.672 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.670+0000 7f3e2f600640 1 -- 172.21.15.96:0/565909094 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3e0406c8f0 msgr2=0x7f3e0406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:05.672 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.670+0000 7f3e2f600640 1 --2- 172.21.15.96:0/565909094 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3e0406c8f0 0x7f3e0406edb0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f3e28162390 tx=0x7f3e18008040 comp rx=0 tx=0).stop 2024-05-28T06:05:05.672 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.670+0000 7f3e2f600640 1 -- 172.21.15.96:0/565909094 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3e2815ed80 msgr2=0x7f3e28145760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:05.672 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.670+0000 7f3e2f600640 1 --2- 172.21.15.96:0/565909094 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3e2815ed80 0x7f3e28145760 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f3e20009f40 tx=0x7f3e20004a40 comp rx=0 tx=0).stop 2024-05-28T06:05:05.672 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.670+0000 7f3e2f600640 1 -- 172.21.15.96:0/565909094 shutdown_connections 2024-05-28T06:05:05.672 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.670+0000 7f3e2f600640 1 --2- 172.21.15.96:0/565909094 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3e0406c8f0 0x7f3e0406edb0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:05.672 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.670+0000 7f3e2f600640 1 --2- 172.21.15.96:0/565909094 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3e2815ed80 0x7f3e28145760 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:05.672 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.670+0000 7f3e2f600640 1 --2- 172.21.15.96:0/565909094 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3e280b7e50 0x7f3e28145220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:05.672 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.670+0000 7f3e2f600640 1 --2- 172.21.15.96:0/565909094 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3e280b61e0 0x7f3e281f7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:05.672 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.670+0000 7f3e2f600640 1 -- 172.21.15.96:0/565909094 >> 172.21.15.96:0/565909094 conn(0x7f3e280d13c0 msgr2=0x7f3e280677b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:05.673 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.671+0000 7f3e2f600640 1 -- 172.21.15.96:0/565909094 shutdown_connections 2024-05-28T06:05:05.673 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.671+0000 7f3e2f600640 1 -- 172.21.15.96:0/565909094 wait complete. 2024-05-28T06:05:05.694 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-28T06:05:05.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.896+0000 7f6252000640 1 Processor -- start 2024-05-28T06:05:05.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.896+0000 7f6252000640 1 -- start start 2024-05-28T06:05:05.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.898+0000 7f6252000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f624c15e870 0x7f624c0d0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:05.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.898+0000 7f6252000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f624c0cfa40 0x7f624c0d25e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:05.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.898+0000 7f6252000640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f624c15ec50 con 0x7f624c0d14d0 2024-05-28T06:05:05.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.898+0000 7f6252000640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f624c0d2150 con 0x7f624c15e870 2024-05-28T06:05:05.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.898+0000 7f6252000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f624c0cff80 con 0x7f624c0cfa40 2024-05-28T06:05:05.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.898+0000 7f6251600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f624c0cfa40 0x7f624c0d25e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:05.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.898+0000 7f6250c00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f624c15e870 0x7f624c0d0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:05.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.899+0000 7f6250c00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f624c15e870 0x7f624c0d0f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.96:56172/0 (socket says 172.21.15.96:56172) 2024-05-28T06:05:05.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.899+0000 7f6250c00640 1 -- 172.21.15.96:0/3380510143 learned_addr learned my addr 172.21.15.96:0/3380510143 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:05.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.899+0000 7f6250c00640 1 -- 172.21.15.96:0/3380510143 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f624c0cfa40 msgr2=0x7f624c0d25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:05.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.899+0000 7f6250c00640 1 --2- 172.21.15.96:0/3380510143 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f624c0cfa40 0x7f624c0d25e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:05.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.899+0000 7f6250c00640 1 -- 172.21.15.96:0/3380510143 >> v1:172.21.15.123:6789/0 conn(0x7f624c0d14d0 legacy=0x7f624c0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:05.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.899+0000 7f6250c00640 1 -- 172.21.15.96:0/3380510143 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f624c0d0260 con 0x7f624c15e870 2024-05-28T06:05:05.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.900+0000 7f6250c00640 1 --2- 172.21.15.96:0/3380510143 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f624c15e870 0x7f624c0d0f90 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f6244009870 tx=0x7f624402a270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b2ccd65a80e9cf4 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:05.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.901+0000 7f624b400640 1 -- 172.21.15.96:0/3380510143 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6244037070 con 0x7f624c15e870 2024-05-28T06:05:05.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.901+0000 7f624b400640 1 -- 172.21.15.96:0/3380510143 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6244032440 con 0x7f624c15e870 2024-05-28T06:05:05.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.901+0000 7f624b400640 1 -- 172.21.15.96:0/3380510143 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6244032810 con 0x7f624c15e870 2024-05-28T06:05:05.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.901+0000 7f6252000640 1 -- 172.21.15.96:0/3380510143 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f624c15e870 msgr2=0x7f624c0d0f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:05.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.901+0000 7f6252000640 1 --2- 172.21.15.96:0/3380510143 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f624c15e870 0x7f624c0d0f90 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f6244009870 tx=0x7f624402a270 comp rx=0 tx=0).stop 2024-05-28T06:05:05.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.902+0000 7f6252000640 1 -- 172.21.15.96:0/3380510143 shutdown_connections 2024-05-28T06:05:05.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.902+0000 7f6252000640 1 --2- 172.21.15.96:0/3380510143 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f624c0cfa40 0x7f624c0d25e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:05.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.902+0000 7f6252000640 1 --2- 172.21.15.96:0/3380510143 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f624c15e870 0x7f624c0d0f90 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:05.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.902+0000 7f6252000640 1 -- 172.21.15.96:0/3380510143 >> 172.21.15.96:0/3380510143 conn(0x7f624c08bfc0 msgr2=0x7f624c0d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:05.904 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.902+0000 7f6252000640 1 -- 172.21.15.96:0/3380510143 shutdown_connections 2024-05-28T06:05:05.904 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.902+0000 7f6252000640 1 -- 172.21.15.96:0/3380510143 wait complete. 2024-05-28T06:05:05.904 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.903+0000 7f6252000640 1 Processor -- start 2024-05-28T06:05:05.904 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.903+0000 7f6252000640 1 -- start start 2024-05-28T06:05:05.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.904+0000 7f6252000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f624c0cfa40 0x7f624c1f75a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:05.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.904+0000 7f6252000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f624c0d14d0 0x7f624c0cd130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:05.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.904+0000 7f6252000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f624c15e870 0x7f624c0cd670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:05.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.904+0000 7f6252000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f624c0d4fb0 con 0x7f624c0d14d0 2024-05-28T06:05:05.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.904+0000 7f6252000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f624c0d0820 con 0x7f624c15e870 2024-05-28T06:05:05.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.904+0000 7f6252000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f624c0d2b20 con 0x7f624c0cfa40 2024-05-28T06:05:05.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.904+0000 7f624be00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f624c0d14d0 0x7f624c0cd130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:05.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.905+0000 7f624be00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f624c0d14d0 0x7f624c0cd130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54780/0 (socket says 172.21.15.96:54780) 2024-05-28T06:05:05.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.905+0000 7f624be00640 1 -- 172.21.15.96:0/149223298 learned_addr learned my addr 172.21.15.96:0/149223298 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:05.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.905+0000 7f624be00640 1 -- 172.21.15.96:0/149223298 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f624c0cfa40 msgr2=0x7f624c1f75a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:05.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.905+0000 7f6250c00640 1 --2- 172.21.15.96:0/149223298 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f624c0cfa40 0x7f624c1f75a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:05.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.905+0000 7f624be00640 1 --2- 172.21.15.96:0/149223298 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f624c0cfa40 0x7f624c1f75a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:05.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.905+0000 7f624be00640 1 -- 172.21.15.96:0/149223298 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f624c15e870 msgr2=0x7f624c0cd670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:05.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.905+0000 7f6251600640 1 --2- 172.21.15.96:0/149223298 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f624c15e870 0x7f624c0cd670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:05.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.905+0000 7f624be00640 1 --2- 172.21.15.96:0/149223298 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f624c15e870 0x7f624c0cd670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:05.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.905+0000 7f624be00640 1 -- 172.21.15.96:0/149223298 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f624c0b7d50 con 0x7f624c0d14d0 2024-05-28T06:05:05.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.905+0000 7f624be00640 1 --2- 172.21.15.96:0/149223298 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f624c0d14d0 0x7f624c0cd130 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f623400c770 tx=0x7f623400cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:05.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.905+0000 7f6248c00640 1 -- 172.21.15.96:0/149223298 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6234016020 con 0x7f624c0d14d0 2024-05-28T06:05:05.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.906+0000 7f6248c00640 1 -- 172.21.15.96:0/149223298 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6234006de0 con 0x7f624c0d14d0 2024-05-28T06:05:05.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.906+0000 7f6248c00640 1 -- 172.21.15.96:0/149223298 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6234003c00 con 0x7f624c0d14d0 2024-05-28T06:05:05.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.906+0000 7f6252000640 1 -- 172.21.15.96:0/149223298 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f624c0b8030 con 0x7f624c0d14d0 2024-05-28T06:05:05.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.906+0000 7f6252000640 1 -- 172.21.15.96:0/149223298 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f624c0b85f0 con 0x7f624c0d14d0 2024-05-28T06:05:05.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.907+0000 7f6252000640 1 -- 172.21.15.96:0/149223298 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f624c0689e0 con 0x7f624c0d14d0 2024-05-28T06:05:05.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.908+0000 7f6248c00640 1 -- 172.21.15.96:0/149223298 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f623400a850 con 0x7f624c0d14d0 2024-05-28T06:05:05.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.909+0000 7f6248c00640 1 --2- 172.21.15.96:0/149223298 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f622806c8f0 0x7f622806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:05.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.909+0000 7f6248c00640 1 -- 172.21.15.96:0/149223298 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5825+0+0 (secure 0 0 0) 0x7f623408b8a0 con 0x7f624c0d14d0 2024-05-28T06:05:05.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.909+0000 7f6250c00640 1 --2- 172.21.15.96:0/149223298 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f622806c8f0 0x7f622806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:05.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.910+0000 7f6250c00640 1 --2- 172.21.15.96:0/149223298 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f622806c8f0 0x7f622806edb0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f6244034000 tx=0x7f624402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:05.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:05.915+0000 7f6248c00640 1 -- 172.21.15.96:0/149223298 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6234020020 con 0x7f624c0d14d0 2024-05-28T06:05:06.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.129+0000 7f6252000640 1 -- 172.21.15.96:0/149223298 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f624c0b6570 con 0x7f624c0d14d0 2024-05-28T06:05:06.131 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.130+0000 7f6248c00640 1 -- 172.21.15.96:0/149223298 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) ==== 80+0+0 (secure 0 0 0) 0x7f6234059ce0 con 0x7f624c0d14d0 2024-05-28T06:05:06.131 INFO:teuthology.orchestra.run.smithi096.stderr:listed 0 entries 2024-05-28T06:05:06.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.132+0000 7f6252000640 1 -- 172.21.15.96:0/149223298 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f622806c8f0 msgr2=0x7f622806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:06.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.132+0000 7f6252000640 1 --2- 172.21.15.96:0/149223298 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f622806c8f0 0x7f622806edb0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f6244034000 tx=0x7f624402e000 comp rx=0 tx=0).stop 2024-05-28T06:05:06.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.133+0000 7f6252000640 1 -- 172.21.15.96:0/149223298 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f624c0d14d0 msgr2=0x7f624c0cd130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:06.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.133+0000 7f6252000640 1 --2- 172.21.15.96:0/149223298 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f624c0d14d0 0x7f624c0cd130 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f623400c770 tx=0x7f623400cc30 comp rx=0 tx=0).stop 2024-05-28T06:05:06.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.133+0000 7f6252000640 1 -- 172.21.15.96:0/149223298 shutdown_connections 2024-05-28T06:05:06.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.133+0000 7f6252000640 1 --2- 172.21.15.96:0/149223298 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f622806c8f0 0x7f622806edb0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:06.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.133+0000 7f6252000640 1 --2- 172.21.15.96:0/149223298 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f624c15e870 0x7f624c0cd670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:06.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.133+0000 7f6252000640 1 --2- 172.21.15.96:0/149223298 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f624c0d14d0 0x7f624c0cd130 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:06.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.133+0000 7f6252000640 1 --2- 172.21.15.96:0/149223298 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f624c0cfa40 0x7f624c1f75a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:06.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.133+0000 7f6252000640 1 -- 172.21.15.96:0/149223298 >> 172.21.15.96:0/149223298 conn(0x7f624c08bfc0 msgr2=0x7f624c0b2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:06.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.133+0000 7f6252000640 1 -- 172.21.15.96:0/149223298 shutdown_connections 2024-05-28T06:05:06.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.133+0000 7f6252000640 1 -- 172.21.15.96:0/149223298 wait complete. 2024-05-28T06:05:06.154 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:05:06.154 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:05:06.202 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:05:06.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.443+0000 7f7122c00640 1 Processor -- start 2024-05-28T06:05:06.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.443+0000 7f7122c00640 1 -- start start 2024-05-28T06:05:06.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.445+0000 7f7122c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f711c15e870 0x7f711c0d0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:06.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.445+0000 7f7122c00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f711c0d14d0 0x7f711c0cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:06.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.445+0000 7f7122c00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f711c0cf870 0x7f711c162360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:06.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.445+0000 7f7122c00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f711c0d2150 con 0x7f711c0cf870 2024-05-28T06:05:06.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.445+0000 7f7122c00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f711c0cfd00 con 0x7f711c0d14d0 2024-05-28T06:05:06.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.445+0000 7f7122c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f711c15ec50 con 0x7f711c15e870 2024-05-28T06:05:06.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.445+0000 7f7122200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f711c0cf870 0x7f711c162360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:06.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.445+0000 7f7122200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f711c0cf870 0x7f711c162360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54792/0 (socket says 172.21.15.96:54792) 2024-05-28T06:05:06.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.445+0000 7f7122200640 1 -- 172.21.15.96:0/4081534363 learned_addr learned my addr 172.21.15.96:0/4081534363 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:06.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.445+0000 7f7121800640 1 --2- 172.21.15.96:0/4081534363 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f711c15e870 0x7f711c0d0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:06.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.446+0000 7f7120e00640 1 --2- 172.21.15.96:0/4081534363 >> v2:172.21.15.123:3300/0 conn(0x7f711c0d14d0 0x7f711c0cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:06.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.446+0000 7f7122200640 1 -- 172.21.15.96:0/4081534363 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f711c15e870 msgr2=0x7f711c0d0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:06.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.446+0000 7f7122200640 1 --2- 172.21.15.96:0/4081534363 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f711c15e870 0x7f711c0d0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:06.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.446+0000 7f7122200640 1 -- 172.21.15.96:0/4081534363 >> v2:172.21.15.123:3300/0 conn(0x7f711c0d14d0 msgr2=0x7f711c0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:06.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.446+0000 7f7122200640 1 --2- 172.21.15.96:0/4081534363 >> v2:172.21.15.123:3300/0 conn(0x7f711c0d14d0 0x7f711c0cf330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:06.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.446+0000 7f7122200640 1 -- 172.21.15.96:0/4081534363 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f711c0cff80 con 0x7f711c0cf870 2024-05-28T06:05:06.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.447+0000 7f7122200640 1 --2- 172.21.15.96:0/4081534363 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f711c0cf870 0x7f711c162360 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f711c0b7170 tx=0x7f711402a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbd4e7a43e17f781 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:06.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.447+0000 7f711be00640 1 -- 172.21.15.96:0/4081534363 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7114036070 con 0x7f711c0cf870 2024-05-28T06:05:06.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.447+0000 7f711be00640 1 -- 172.21.15.96:0/4081534363 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7114031600 con 0x7f711c0cf870 2024-05-28T06:05:06.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.447+0000 7f711be00640 1 -- 172.21.15.96:0/4081534363 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71140319d0 con 0x7f711c0cf870 2024-05-28T06:05:06.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.448+0000 7f7122c00640 1 -- 172.21.15.96:0/4081534363 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f711c0cf870 msgr2=0x7f711c162360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:06.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.448+0000 7f7122c00640 1 --2- 172.21.15.96:0/4081534363 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f711c0cf870 0x7f711c162360 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f711c0b7170 tx=0x7f711402a6a0 comp rx=0 tx=0).stop 2024-05-28T06:05:06.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.448+0000 7f7122c00640 1 -- 172.21.15.96:0/4081534363 shutdown_connections 2024-05-28T06:05:06.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.448+0000 7f7122c00640 1 --2- 172.21.15.96:0/4081534363 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f711c0cf870 0x7f711c162360 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:06.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.448+0000 7f7122c00640 1 --2- 172.21.15.96:0/4081534363 >> v2:172.21.15.123:3300/0 conn(0x7f711c0d14d0 0x7f711c0cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:06.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.448+0000 7f7122c00640 1 --2- 172.21.15.96:0/4081534363 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f711c15e870 0x7f711c0d0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:06.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.448+0000 7f7122c00640 1 -- 172.21.15.96:0/4081534363 >> 172.21.15.96:0/4081534363 conn(0x7f711c08bfc0 msgr2=0x7f711c0d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:06.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.448+0000 7f7122c00640 1 -- 172.21.15.96:0/4081534363 shutdown_connections 2024-05-28T06:05:06.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.448+0000 7f7122c00640 1 -- 172.21.15.96:0/4081534363 wait complete. 2024-05-28T06:05:06.451 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.449+0000 7f7122c00640 1 Processor -- start 2024-05-28T06:05:06.451 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.450+0000 7f7122c00640 1 -- start start 2024-05-28T06:05:06.452 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.450+0000 7f7122c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f711c0cf870 0x7f711c1593f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:06.452 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.450+0000 7f7122c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f711c0d14d0 0x7f711c0b8e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:06.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.450+0000 7f7122c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f711c15e870 0x7f711c0cb7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:06.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.451+0000 7f7122c00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f711c0d0820 con 0x7f711c0cf870 2024-05-28T06:05:06.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.451+0000 7f7122c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f711c0d0540 con 0x7f711c0d14d0 2024-05-28T06:05:06.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.451+0000 7f7122c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f711c0d4fb0 con 0x7f711c15e870 2024-05-28T06:05:06.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.451+0000 7f7121800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f711c0cf870 0x7f711c1593f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:06.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.451+0000 7f7121800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f711c0cf870 0x7f711c1593f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54808/0 (socket says 172.21.15.96:54808) 2024-05-28T06:05:06.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.451+0000 7f7121800640 1 -- 172.21.15.96:0/2119221284 learned_addr learned my addr 172.21.15.96:0/2119221284 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:06.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.451+0000 7f7121800640 1 -- 172.21.15.96:0/2119221284 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f711c15e870 msgr2=0x7f711c0cb7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:06.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.451+0000 7f7121800640 1 --2- 172.21.15.96:0/2119221284 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f711c15e870 0x7f711c0cb7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:06.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.451+0000 7f7121800640 1 -- 172.21.15.96:0/2119221284 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f711c0d14d0 msgr2=0x7f711c0b8e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:06.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.451+0000 7f7121800640 1 --2- 172.21.15.96:0/2119221284 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f711c0d14d0 0x7f711c0b8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:06.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.451+0000 7f7121800640 1 -- 172.21.15.96:0/2119221284 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f711c15b410 con 0x7f711c0cf870 2024-05-28T06:05:06.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.451+0000 7f7121800640 1 --2- 172.21.15.96:0/2119221284 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f711c0cf870 0x7f711c1593f0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f711000d800 tx=0x7f711000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:06.454 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.452+0000 7f7119600640 1 -- 172.21.15.96:0/2119221284 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f711000b6c0 con 0x7f711c0cf870 2024-05-28T06:05:06.454 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.452+0000 7f7119600640 1 -- 172.21.15.96:0/2119221284 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7110019070 con 0x7f711c0cf870 2024-05-28T06:05:06.454 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.452+0000 7f7119600640 1 -- 172.21.15.96:0/2119221284 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f711000ba60 con 0x7f711c0cf870 2024-05-28T06:05:06.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.452+0000 7f7122c00640 1 -- 172.21.15.96:0/2119221284 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f711c15b750 con 0x7f711c0cf870 2024-05-28T06:05:06.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.452+0000 7f7122c00640 1 -- 172.21.15.96:0/2119221284 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f711c15bf90 con 0x7f711c0cf870 2024-05-28T06:05:06.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.453+0000 7f7122c00640 1 -- 172.21.15.96:0/2119221284 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f711c0d4250 con 0x7f711c0cf870 2024-05-28T06:05:06.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.454+0000 7f7119600640 1 -- 172.21.15.96:0/2119221284 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f711000bc00 con 0x7f711c0cf870 2024-05-28T06:05:06.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.455+0000 7f7119600640 1 --2- 172.21.15.96:0/2119221284 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f70fc06c920 0x7f70fc06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:06.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.455+0000 7f7119600640 1 -- 172.21.15.96:0/2119221284 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5825+0+0 (secure 0 0 0) 0x7f711008c070 con 0x7f711c0cf870 2024-05-28T06:05:06.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.456+0000 7f7120e00640 1 --2- 172.21.15.96:0/2119221284 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f70fc06c920 0x7f70fc06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:06.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.457+0000 7f7120e00640 1 --2- 172.21.15.96:0/2119221284 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f70fc06c920 0x7f70fc06ede0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f711c161ef0 tx=0x7f7108008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:06.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.460+0000 7f7119600640 1 -- 172.21.15.96:0/2119221284 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f711c0d4250 con 0x7f711c0cf870 2024-05-28T06:05:06.661 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.660+0000 7f7122c00640 1 -- 172.21.15.96:0/2119221284 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f711c0689e0 con 0x7f711c0cf870 2024-05-28T06:05:06.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.663+0000 7f7119600640 1 -- 172.21.15.96:0/2119221284 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) ==== 74+0+16865 (secure 0 0 0) 0x7f711c15b750 con 0x7f711c0cf870 2024-05-28T06:05:06.665 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:05:06.665 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":22,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:04:52.624604+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-28T06:04:08.741938+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-28T06:04:09.301470+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.b","mappings":[{"from":3,"to":1}]},{"pgid":"3.d","mappings":[{"from":7,"to":2}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:05:06.666 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.665+0000 7f7122c00640 1 -- 172.21.15.96:0/2119221284 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f70fc06c920 msgr2=0x7f70fc06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:06.666 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.665+0000 7f7122c00640 1 --2- 172.21.15.96:0/2119221284 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f70fc06c920 0x7f70fc06ede0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f711c161ef0 tx=0x7f7108008040 comp rx=0 tx=0).stop 2024-05-28T06:05:06.667 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.665+0000 7f7122c00640 1 -- 172.21.15.96:0/2119221284 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f711c0cf870 msgr2=0x7f711c1593f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:06.667 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.665+0000 7f7122c00640 1 --2- 172.21.15.96:0/2119221284 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f711c0cf870 0x7f711c1593f0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f711000d800 tx=0x7f711000dcc0 comp rx=0 tx=0).stop 2024-05-28T06:05:06.667 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.666+0000 7f7122c00640 1 -- 172.21.15.96:0/2119221284 shutdown_connections 2024-05-28T06:05:06.667 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.666+0000 7f7122c00640 1 --2- 172.21.15.96:0/2119221284 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f70fc06c920 0x7f70fc06ede0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:06.667 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.666+0000 7f7122c00640 1 --2- 172.21.15.96:0/2119221284 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f711c15e870 0x7f711c0cb7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:06.667 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.666+0000 7f7122c00640 1 --2- 172.21.15.96:0/2119221284 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f711c0d14d0 0x7f711c0b8e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:06.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.666+0000 7f7122c00640 1 --2- 172.21.15.96:0/2119221284 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f711c0cf870 0x7f711c1593f0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:06.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.666+0000 7f7122c00640 1 -- 172.21.15.96:0/2119221284 >> 172.21.15.96:0/2119221284 conn(0x7f711c08bfc0 msgr2=0x7f711c0b2b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:06.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.666+0000 7f7122c00640 1 -- 172.21.15.96:0/2119221284 shutdown_connections 2024-05-28T06:05:06.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.666+0000 7f7122c00640 1 -- 172.21.15.96:0/2119221284 wait complete. 2024-05-28T06:05:06.686 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-28T06:03:59.517102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-28T06:04:08.741938+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-28T06:04:09.301470+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-28T06:05:06.687 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-28T06:05:06.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.884+0000 7f1383e00640 1 Processor -- start 2024-05-28T06:05:06.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.885+0000 7f1383e00640 1 -- start start 2024-05-28T06:05:06.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.886+0000 7f1383e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f137c09b790 0x7f137c160900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:06.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.886+0000 7f1383e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f137c0b13d0 0x7f137c160e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:06.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.886+0000 7f1383e00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f137c0cdb60 con 0x7f137c09aca0 2024-05-28T06:05:06.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.886+0000 7f1383e00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f137c0b1990 con 0x7f137c0b13d0 2024-05-28T06:05:06.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.886+0000 7f1383e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f137c0cde00 con 0x7f137c09b790 2024-05-28T06:05:06.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.887+0000 7f1381e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f137c0b13d0 0x7f137c160e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:06.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.887+0000 7f1380a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f137c09b790 0x7f137c160900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:06.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.887+0000 7f1381e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f137c0b13d0 0x7f137c160e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.96:56202/0 (socket says 172.21.15.96:56202) 2024-05-28T06:05:06.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.887+0000 7f1381e00640 1 -- 172.21.15.96:0/1378731223 learned_addr learned my addr 172.21.15.96:0/1378731223 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:06.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.888+0000 7f1381e00640 1 -- 172.21.15.96:0/1378731223 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f137c09b790 msgr2=0x7f137c160900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:06.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.888+0000 7f1381e00640 1 --2- 172.21.15.96:0/1378731223 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f137c09b790 0x7f137c160900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:06.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.888+0000 7f1381e00640 1 -- 172.21.15.96:0/1378731223 >> v1:172.21.15.123:6789/0 conn(0x7f137c09aca0 legacy=0x7f137c09b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:06.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.888+0000 7f1381e00640 1 -- 172.21.15.96:0/1378731223 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f137c0ce0e0 con 0x7f137c0b13d0 2024-05-28T06:05:06.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.889+0000 7f1381e00640 1 --2- 172.21.15.96:0/1378731223 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f137c0b13d0 0x7f137c160e40 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f137c0d30b0 tx=0x7f137402a730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d34d0b2980fedfba server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:06.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.889+0000 7f137be00640 1 -- 172.21.15.96:0/1378731223 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1374036070 con 0x7f137c0b13d0 2024-05-28T06:05:06.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.889+0000 7f137be00640 1 -- 172.21.15.96:0/1378731223 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1374031600 con 0x7f137c0b13d0 2024-05-28T06:05:06.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.889+0000 7f137be00640 1 -- 172.21.15.96:0/1378731223 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13740319d0 con 0x7f137c0b13d0 2024-05-28T06:05:06.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.890+0000 7f1383e00640 1 -- 172.21.15.96:0/1378731223 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f137c0b13d0 msgr2=0x7f137c160e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:06.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.890+0000 7f1383e00640 1 --2- 172.21.15.96:0/1378731223 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f137c0b13d0 0x7f137c160e40 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f137c0d30b0 tx=0x7f137402a730 comp rx=0 tx=0).stop 2024-05-28T06:05:06.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.890+0000 7f1383e00640 1 -- 172.21.15.96:0/1378731223 shutdown_connections 2024-05-28T06:05:06.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.890+0000 7f1383e00640 1 --2- 172.21.15.96:0/1378731223 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f137c0b13d0 0x7f137c160e40 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:06.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.890+0000 7f1383e00640 1 --2- 172.21.15.96:0/1378731223 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f137c09b790 0x7f137c160900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:06.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.890+0000 7f1383e00640 1 -- 172.21.15.96:0/1378731223 >> 172.21.15.96:0/1378731223 conn(0x7f137c0d24b0 msgr2=0x7f137c0d28c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:06.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.890+0000 7f1383e00640 1 -- 172.21.15.96:0/1378731223 shutdown_connections 2024-05-28T06:05:06.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.890+0000 7f1383e00640 1 -- 172.21.15.96:0/1378731223 wait complete. 2024-05-28T06:05:06.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.892+0000 7f1383e00640 1 Processor -- start 2024-05-28T06:05:06.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.892+0000 7f1383e00640 1 -- start start 2024-05-28T06:05:06.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.893+0000 7f1383e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f137c09aca0 0x7f137c1f3df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:06.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.893+0000 7f1383e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f137c09b790 0x7f137c0cb6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:06.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.893+0000 7f1383e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f137c0b13d0 0x7f137c0c96a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:06.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.893+0000 7f1383e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f137c161500 con 0x7f137c0b13d0 2024-05-28T06:05:06.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.893+0000 7f1383e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f137c161380 con 0x7f137c09aca0 2024-05-28T06:05:06.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.893+0000 7f1383e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f137c161680 con 0x7f137c09b790 2024-05-28T06:05:06.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.893+0000 7f1381e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f137c0b13d0 0x7f137c0c96a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:06.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.893+0000 7f1381e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f137c0b13d0 0x7f137c0c96a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54814/0 (socket says 172.21.15.96:54814) 2024-05-28T06:05:06.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.893+0000 7f1381e00640 1 -- 172.21.15.96:0/3572572411 learned_addr learned my addr 172.21.15.96:0/3572572411 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:06.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.894+0000 7f1381400640 1 --2- 172.21.15.96:0/3572572411 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f137c09aca0 0x7f137c1f3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:06.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.894+0000 7f1380a00640 1 --2- 172.21.15.96:0/3572572411 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f137c09b790 0x7f137c0cb6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:06.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.894+0000 7f1381e00640 1 -- 172.21.15.96:0/3572572411 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f137c09b790 msgr2=0x7f137c0cb6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:06.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.894+0000 7f1381e00640 1 --2- 172.21.15.96:0/3572572411 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f137c09b790 0x7f137c0cb6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:06.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.894+0000 7f1381e00640 1 -- 172.21.15.96:0/3572572411 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f137c09aca0 msgr2=0x7f137c1f3df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:06.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.894+0000 7f1381e00640 1 --2- 172.21.15.96:0/3572572411 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f137c09aca0 0x7f137c1f3df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:06.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.894+0000 7f1381e00640 1 -- 172.21.15.96:0/3572572411 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f137c1f4330 con 0x7f137c0b13d0 2024-05-28T06:05:06.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.894+0000 7f1380a00640 1 --2- 172.21.15.96:0/3572572411 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f137c09b790 0x7f137c0cb6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:05:06.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.894+0000 7f1381e00640 1 --2- 172.21.15.96:0/3572572411 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f137c0b13d0 0x7f137c0c96a0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f1374034040 tx=0x7f1374004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:06.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.895+0000 7f1379600640 1 -- 172.21.15.96:0/3572572411 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1374041020 con 0x7f137c0b13d0 2024-05-28T06:05:06.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.895+0000 7f1379600640 1 -- 172.21.15.96:0/3572572411 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13740077a0 con 0x7f137c0b13d0 2024-05-28T06:05:06.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.895+0000 7f1383e00640 1 -- 172.21.15.96:0/3572572411 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f137c0cc180 con 0x7f137c0b13d0 2024-05-28T06:05:06.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.895+0000 7f1379600640 1 -- 172.21.15.96:0/3572572411 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1374036040 con 0x7f137c0b13d0 2024-05-28T06:05:06.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.895+0000 7f1383e00640 1 -- 172.21.15.96:0/3572572411 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f137c0cc740 con 0x7f137c0b13d0 2024-05-28T06:05:06.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.896+0000 7f1383e00640 1 -- 172.21.15.96:0/3572572411 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f137c0cc980 con 0x7f137c0b13d0 2024-05-28T06:05:06.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.897+0000 7f1379600640 1 -- 172.21.15.96:0/3572572411 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1374044020 con 0x7f137c0b13d0 2024-05-28T06:05:06.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.898+0000 7f1379600640 1 --2- 172.21.15.96:0/3572572411 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f135c06c8f0 0x7f135c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:06.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.899+0000 7f1379600640 1 -- 172.21.15.96:0/3572572411 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5825+0+0 (secure 0 0 0) 0x7f13740ac860 con 0x7f137c0b13d0 2024-05-28T06:05:06.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.899+0000 7f1381400640 1 --2- 172.21.15.96:0/3572572411 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f135c06c8f0 0x7f135c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:06.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.900+0000 7f1381400640 1 --2- 172.21.15.96:0/3572572411 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f135c06c8f0 0x7f135c06edb0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f136c006fd0 tx=0x7f136c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:06.905 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:06.904+0000 7f1379600640 1 -- 172.21.15.96:0/3572572411 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f137c0cc980 con 0x7f137c0b13d0 2024-05-28T06:05:07.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.105+0000 7f1383e00640 1 -- 172.21.15.96:0/3572572411 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f137c0690f0 con 0x7f137c0b13d0 2024-05-28T06:05:07.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.106+0000 7f1379600640 1 -- 172.21.15.96:0/3572572411 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) ==== 93+0+10 (secure 0 0 0) 0x7f137404b030 con 0x7f137c0b13d0 2024-05-28T06:05:07.108 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 1 2024-05-28T06:05:07.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.108+0000 7f1383e00640 1 -- 172.21.15.96:0/3572572411 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f135c06c8f0 msgr2=0x7f135c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:07.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.108+0000 7f1383e00640 1 --2- 172.21.15.96:0/3572572411 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f135c06c8f0 0x7f135c06edb0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f136c006fd0 tx=0x7f136c008040 comp rx=0 tx=0).stop 2024-05-28T06:05:07.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.108+0000 7f1383e00640 1 -- 172.21.15.96:0/3572572411 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f137c0b13d0 msgr2=0x7f137c0c96a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:07.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.108+0000 7f1383e00640 1 --2- 172.21.15.96:0/3572572411 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f137c0b13d0 0x7f137c0c96a0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f1374034040 tx=0x7f1374004a40 comp rx=0 tx=0).stop 2024-05-28T06:05:07.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.109+0000 7f1383e00640 1 -- 172.21.15.96:0/3572572411 shutdown_connections 2024-05-28T06:05:07.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.109+0000 7f1383e00640 1 --2- 172.21.15.96:0/3572572411 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f135c06c8f0 0x7f135c06edb0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:07.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.109+0000 7f1383e00640 1 --2- 172.21.15.96:0/3572572411 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f137c0b13d0 0x7f137c0c96a0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:07.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.109+0000 7f1383e00640 1 --2- 172.21.15.96:0/3572572411 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f137c09b790 0x7f137c0cb6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:07.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.109+0000 7f1383e00640 1 --2- 172.21.15.96:0/3572572411 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f137c09aca0 0x7f137c1f3df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:07.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.109+0000 7f1383e00640 1 -- 172.21.15.96:0/3572572411 >> 172.21.15.96:0/3572572411 conn(0x7f137c0d24b0 msgr2=0x7f137c070c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:07.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.109+0000 7f1383e00640 1 -- 172.21.15.96:0/3572572411 shutdown_connections 2024-05-28T06:05:07.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.109+0000 7f1383e00640 1 -- 172.21.15.96:0/3572572411 wait complete. 2024-05-28T06:05:07.129 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-28T06:05:07.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.329+0000 7f6b07e00640 1 Processor -- start 2024-05-28T06:05:07.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.329+0000 7f6b07e00640 1 -- start start 2024-05-28T06:05:07.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.330+0000 7f6b07e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6b080b4dd0 0x7f6b080ccb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:07.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.330+0000 7f6b07e00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b080b3580 con 0x7f6b080ce1f0 2024-05-28T06:05:07.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.330+0000 7f6b07e00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b0815f490 con 0x7f6b080ce5d0 2024-05-28T06:05:07.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.330+0000 7f6b07e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f6b0815f770 con 0x7f6b080b4dd0 2024-05-28T06:05:07.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.331+0000 7f6b06a00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f6b080ce1f0 0x7f6b0815ec40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:60454/0 (socket says 172.21.15.96:60454) 2024-05-28T06:05:07.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.331+0000 7f6b06a00640 1 -- 172.21.15.96:0/2159801065 learned_addr learned my addr 172.21.15.96:0/2159801065 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:05:07.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.331+0000 7f6b07400640 1 --2- 172.21.15.96:0/2159801065 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6b080b4dd0 0x7f6b080ccb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:07.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.331+0000 7f6b06000640 1 -- 172.21.15.96:0/2159801065 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2636037455 0 0) 0x7f6b080b3580 con 0x7f6b080ce1f0 2024-05-28T06:05:07.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.331+0000 7f6b06000640 1 -- 172.21.15.96:0/2159801065 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6aec003520 con 0x7f6b080ce1f0 2024-05-28T06:05:07.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.331+0000 7f6b06000640 1 -- 172.21.15.96:0/2159801065 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4020874543 0 0) 0x7f6aec003520 con 0x7f6b080ce1f0 2024-05-28T06:05:07.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.331+0000 7f6b06000640 1 -- 172.21.15.96:0/2159801065 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6b080b4dd0 msgr2=0x7f6b080ccb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:07.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.331+0000 7f6b06000640 1 --2- 172.21.15.96:0/2159801065 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6b080b4dd0 0x7f6b080ccb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:07.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.331+0000 7f6b06000640 1 -- 172.21.15.96:0/2159801065 >> v1:172.21.15.123:6789/0 conn(0x7f6b080ce5d0 legacy=0x7f6b080b3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:07.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.331+0000 7f6b06000640 1 -- 172.21.15.96:0/2159801065 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b0815fa50 con 0x7f6b080ce1f0 2024-05-28T06:05:07.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.331+0000 7f6b06000640 1 -- 172.21.15.96:0/2159801065 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f6af8001670 con 0x7f6b080ce1f0 2024-05-28T06:05:07.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.331+0000 7f6b06000640 1 -- 172.21.15.96:0/2159801065 >> v1:172.21.15.96:6789/0 conn(0x7f6b080ce1f0 legacy=0x7f6b0815ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:07.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.332+0000 7f6b06000640 1 --2- 172.21.15.96:0/2159801065 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6b080b4dd0 0x7f6aec0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:07.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.332+0000 7f6b06000640 1 --2- 172.21.15.96:0/2159801065 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6aec004010 0x7f6aec004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:07.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.332+0000 7f6b06000640 1 --2- 172.21.15.96:0/2159801065 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6aec004d90 0x7f6aec005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:07.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.332+0000 7f6b06000640 1 -- 172.21.15.96:0/2159801065 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f6b0815f770 con 0x7f6aec004010 2024-05-28T06:05:07.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.332+0000 7f6b06000640 1 -- 172.21.15.96:0/2159801065 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f6aec00bca0 con 0x7f6aec004d90 2024-05-28T06:05:07.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.332+0000 7f6b06000640 1 -- 172.21.15.96:0/2159801065 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f6aec00bea0 con 0x7f6b080b4dd0 2024-05-28T06:05:07.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.332+0000 7f6affe00640 1 --2- 172.21.15.96:0/2159801065 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6aec004010 0x7f6aec004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:07.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.332+0000 7f6b06a00640 1 --2- 172.21.15.96:0/2159801065 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6b080b4dd0 0x7f6aec0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:07.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.332+0000 7f6b07400640 1 --2- 172.21.15.96:0/2159801065 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6aec004d90 0x7f6aec005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:07.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.332+0000 7f6affe00640 1 -- 172.21.15.96:0/2159801065 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6b080b4dd0 msgr2=0x7f6aec0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:07.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.332+0000 7f6affe00640 1 --2- 172.21.15.96:0/2159801065 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6b080b4dd0 0x7f6aec0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:07.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.332+0000 7f6affe00640 1 -- 172.21.15.96:0/2159801065 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6aec004d90 msgr2=0x7f6aec005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:07.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.332+0000 7f6affe00640 1 --2- 172.21.15.96:0/2159801065 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6aec004d90 0x7f6aec005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:07.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.332+0000 7f6affe00640 1 -- 172.21.15.96:0/2159801065 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6aec003520 con 0x7f6aec004010 2024-05-28T06:05:07.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.332+0000 7f6b07400640 1 --2- 172.21.15.96:0/2159801065 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6aec004d90 0x7f6aec005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:05:07.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.333+0000 7f6affe00640 1 --2- 172.21.15.96:0/2159801065 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6aec004010 0x7f6aec004800 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f6af0001720 tx=0x7f6aec005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e65c36bec6af0cfe server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:07.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.334+0000 7f6b06000640 1 -- 172.21.15.96:0/2159801065 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6af0001370 con 0x7f6aec004010 2024-05-28T06:05:07.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.334+0000 7f6b06000640 1 -- 172.21.15.96:0/2159801065 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6af00052d0 con 0x7f6aec004010 2024-05-28T06:05:07.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.334+0000 7f6b06000640 1 -- 172.21.15.96:0/2159801065 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6af002abc0 con 0x7f6aec004010 2024-05-28T06:05:07.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.334+0000 7f6b07e00640 1 -- 172.21.15.96:0/2159801065 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6aec004010 msgr2=0x7f6aec004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:07.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.334+0000 7f6b07e00640 1 --2- 172.21.15.96:0/2159801065 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6aec004010 0x7f6aec004800 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f6af0001720 tx=0x7f6aec005830 comp rx=0 tx=0).stop 2024-05-28T06:05:07.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.334+0000 7f6b07400640 1 -- 172.21.15.96:0/2159801065 reap_dead start 2024-05-28T06:05:07.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.334+0000 7f6b07e00640 1 -- 172.21.15.96:0/2159801065 shutdown_connections 2024-05-28T06:05:07.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.334+0000 7f6b07e00640 1 -- 172.21.15.96:0/2159801065 >> 172.21.15.96:0/2159801065 conn(0x7f6b0806b900 msgr2=0x7f6b0806bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:07.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.335+0000 7f6b07e00640 1 -- 172.21.15.96:0/2159801065 shutdown_connections 2024-05-28T06:05:07.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.335+0000 7f6b07e00640 1 -- 172.21.15.96:0/2159801065 wait complete. 2024-05-28T06:05:07.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.336+0000 7f6b07e00640 1 Processor -- start 2024-05-28T06:05:07.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.336+0000 7f6b07e00640 1 -- start start 2024-05-28T06:05:07.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.337+0000 7f6b07e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6aec004010 0x7f6b080b11b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:07.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.337+0000 7f6b07e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6b081f7890 0x7f6b080b2130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:07.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.337+0000 7f6b07e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6b080cb7c0 0x7f6b080cbc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:07.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.337+0000 7f6b07e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f6b080b45f0 con 0x7f6aec004010 2024-05-28T06:05:07.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.337+0000 7f6b07e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f6b080b4470 con 0x7f6b081f7890 2024-05-28T06:05:07.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.337+0000 7f6b07e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f6b080ccb90 con 0x7f6b080cb7c0 2024-05-28T06:05:07.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.338+0000 7f6b06a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6aec004010 0x7f6b080b11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:07.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.338+0000 7f6b06a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6aec004010 0x7f6b080b11b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54830/0 (socket says 172.21.15.96:54830) 2024-05-28T06:05:07.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.338+0000 7f6b06a00640 1 -- 172.21.15.96:0/1931599942 learned_addr learned my addr 172.21.15.96:0/1931599942 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:07.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.338+0000 7f6b07400640 1 --2- 172.21.15.96:0/1931599942 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6b080cb7c0 0x7f6b080cbc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:07.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.338+0000 7f6affe00640 1 --2- 172.21.15.96:0/1931599942 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6b081f7890 0x7f6b080b2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:07.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.338+0000 7f6b06a00640 1 -- 172.21.15.96:0/1931599942 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6b080cb7c0 msgr2=0x7f6b080cbc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:07.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.338+0000 7f6b06a00640 1 --2- 172.21.15.96:0/1931599942 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6b080cb7c0 0x7f6b080cbc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:07.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.338+0000 7f6b06a00640 1 -- 172.21.15.96:0/1931599942 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6b081f7890 msgr2=0x7f6b080b2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:07.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.338+0000 7f6b06a00640 1 --2- 172.21.15.96:0/1931599942 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6b081f7890 0x7f6b080b2130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:07.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.338+0000 7f6b06a00640 1 -- 172.21.15.96:0/1931599942 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b0815fd70 con 0x7f6aec004010 2024-05-28T06:05:07.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.338+0000 7f6b06a00640 1 --2- 172.21.15.96:0/1931599942 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6aec004010 0x7f6b080b11b0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f6af800e7e0 tx=0x7f6af800eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:07.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.339+0000 7f6afea00640 1 -- 172.21.15.96:0/1931599942 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6af8019070 con 0x7f6aec004010 2024-05-28T06:05:07.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.339+0000 7f6afea00640 1 -- 172.21.15.96:0/1931599942 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6af800ee90 con 0x7f6aec004010 2024-05-28T06:05:07.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.339+0000 7f6b07e00640 1 -- 172.21.15.96:0/1931599942 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b080b2990 con 0x7f6aec004010 2024-05-28T06:05:07.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.339+0000 7f6afea00640 1 -- 172.21.15.96:0/1931599942 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6af800ca90 con 0x7f6aec004010 2024-05-28T06:05:07.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.339+0000 7f6b07e00640 1 -- 172.21.15.96:0/1931599942 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b081602f0 con 0x7f6aec004010 2024-05-28T06:05:07.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.340+0000 7f6b07e00640 1 -- 172.21.15.96:0/1931599942 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b080689e0 con 0x7f6aec004010 2024-05-28T06:05:07.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.341+0000 7f6afea00640 1 -- 172.21.15.96:0/1931599942 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6af801c070 con 0x7f6aec004010 2024-05-28T06:05:07.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.342+0000 7f6afea00640 1 --2- 172.21.15.96:0/1931599942 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6ae406c8f0 0x7f6ae406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:07.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.342+0000 7f6afea00640 1 -- 172.21.15.96:0/1931599942 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5825+0+0 (secure 0 0 0) 0x7f6af808b430 con 0x7f6aec004010 2024-05-28T06:05:07.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.342+0000 7f6affe00640 1 --2- 172.21.15.96:0/1931599942 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6ae406c8f0 0x7f6ae406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:07.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.344+0000 7f6affe00640 1 --2- 172.21.15.96:0/1931599942 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6ae406c8f0 0x7f6ae406edb0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f6af0032000 tx=0x7f6af002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:07.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.346+0000 7f6afea00640 1 -- 172.21.15.96:0/1931599942 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6af8059740 con 0x7f6aec004010 2024-05-28T06:05:07.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.544+0000 7f6b07e00640 1 -- 172.21.15.96:0/1931599942 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f6b080ca070 con 0x7f6aec004010 2024-05-28T06:05:07.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.545+0000 7f6afea00640 1 -- 172.21.15.96:0/1931599942 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) ==== 104+0+11 (secure 0 0 0) 0x7f6af8059040 con 0x7f6aec004010 2024-05-28T06:05:07.546 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 32 2024-05-28T06:05:07.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.548+0000 7f6b07e00640 1 -- 172.21.15.96:0/1931599942 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6ae406c8f0 msgr2=0x7f6ae406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:07.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.548+0000 7f6b07e00640 1 --2- 172.21.15.96:0/1931599942 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6ae406c8f0 0x7f6ae406edb0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f6af0032000 tx=0x7f6af002e000 comp rx=0 tx=0).stop 2024-05-28T06:05:07.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.548+0000 7f6b07e00640 1 -- 172.21.15.96:0/1931599942 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6aec004010 msgr2=0x7f6b080b11b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:07.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.548+0000 7f6b07e00640 1 --2- 172.21.15.96:0/1931599942 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6aec004010 0x7f6b080b11b0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f6af800e7e0 tx=0x7f6af800eca0 comp rx=0 tx=0).stop 2024-05-28T06:05:07.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.548+0000 7f6b07e00640 1 -- 172.21.15.96:0/1931599942 shutdown_connections 2024-05-28T06:05:07.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.548+0000 7f6b07e00640 1 --2- 172.21.15.96:0/1931599942 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6ae406c8f0 0x7f6ae406edb0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:07.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.548+0000 7f6b07e00640 1 --2- 172.21.15.96:0/1931599942 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6b080cb7c0 0x7f6b080cbc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:07.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.548+0000 7f6b07e00640 1 --2- 172.21.15.96:0/1931599942 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6b081f7890 0x7f6b080b2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:07.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.548+0000 7f6b07e00640 1 --2- 172.21.15.96:0/1931599942 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6aec004010 0x7f6b080b11b0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:07.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.548+0000 7f6b07e00640 1 -- 172.21.15.96:0/1931599942 >> 172.21.15.96:0/1931599942 conn(0x7f6b0806b900 msgr2=0x7f6b0806bce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:07.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.548+0000 7f6b07e00640 1 -- 172.21.15.96:0/1931599942 shutdown_connections 2024-05-28T06:05:07.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.548+0000 7f6b07e00640 1 -- 172.21.15.96:0/1931599942 wait complete. 2024-05-28T06:05:07.567 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-28T06:05:07.770 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.768+0000 7fb7d0e00640 1 Processor -- start 2024-05-28T06:05:07.770 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.768+0000 7fb7d0e00640 1 -- start start 2024-05-28T06:05:07.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.770+0000 7fb7d0e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fb7cc06f380 0x7fb7cc0b45d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:07.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.770+0000 7fb7d0e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb7cc0b4b10 0x7fb7cc0725c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:07.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.770+0000 7fb7d0e00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7cc06e2e0 con 0x7fb7cc15abe0 2024-05-28T06:05:07.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.770+0000 7fb7d0e00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fb7cc0d0d50 con 0x7fb7cc06f380 2024-05-28T06:05:07.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.770+0000 7fb7d0e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fb7cc06e5c0 con 0x7fb7cc0b4b10 2024-05-28T06:05:07.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.770+0000 7fb7c9e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fb7cc15abe0 0x7fb7cc0d0640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:60456/0 (socket says 172.21.15.96:60456) 2024-05-28T06:05:07.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.770+0000 7fb7c9e00640 1 -- 172.21.15.96:0/1046707271 learned_addr learned my addr 172.21.15.96:0/1046707271 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:05:07.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.770+0000 7fb7c9400640 1 --2- 172.21.15.96:0/1046707271 >> v2:172.21.15.96:3300/0 conn(0x7fb7cc06f380 0x7fb7cc0b45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:07.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.771+0000 7fb7ca800640 1 --2- 172.21.15.96:0/1046707271 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb7cc0b4b10 0x7fb7cc0725c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:07.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.771+0000 7fb7c8a00640 1 -- 172.21.15.96:0/1046707271 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1914468093 0 0) 0x7fb7cc06e2e0 con 0x7fb7cc15abe0 2024-05-28T06:05:07.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.771+0000 7fb7c8a00640 1 -- 172.21.15.96:0/1046707271 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb7ac003520 con 0x7fb7cc15abe0 2024-05-28T06:05:07.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.771+0000 7fb7c9400640 1 -- 172.21.15.96:0/1046707271 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb7cc0b4b10 msgr2=0x7fb7cc0725c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:07.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.771+0000 7fb7c9400640 1 --2- 172.21.15.96:0/1046707271 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb7cc0b4b10 0x7fb7cc0725c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:07.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.771+0000 7fb7c9400640 1 -- 172.21.15.96:0/1046707271 >> v1:172.21.15.96:6789/0 conn(0x7fb7cc15abe0 legacy=0x7fb7cc0d0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:07.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.771+0000 7fb7c9400640 1 -- 172.21.15.96:0/1046707271 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7cc06e8a0 con 0x7fb7cc06f380 2024-05-28T06:05:07.774 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.772+0000 7fb7c9400640 1 --2- 172.21.15.96:0/1046707271 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb7cc06f380 0x7fb7cc0b45d0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fb7b4009d90 tx=0x7fb7b402a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afe2998e4c17104c server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:07.774 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.772+0000 7fb7c8a00640 1 -- 172.21.15.96:0/1046707271 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7b4007740 con 0x7fb7cc06f380 2024-05-28T06:05:07.774 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.772+0000 7fb7c8a00640 1 -- 172.21.15.96:0/1046707271 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7b40078e0 con 0x7fb7cc06f380 2024-05-28T06:05:07.774 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.772+0000 7fb7c8a00640 1 -- 172.21.15.96:0/1046707271 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7b4007cb0 con 0x7fb7cc06f380 2024-05-28T06:05:07.775 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.773+0000 7fb7d0e00640 1 -- 172.21.15.96:0/1046707271 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb7cc06f380 msgr2=0x7fb7cc0b45d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:07.775 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.773+0000 7fb7d0e00640 1 --2- 172.21.15.96:0/1046707271 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb7cc06f380 0x7fb7cc0b45d0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fb7b4009d90 tx=0x7fb7b402a660 comp rx=0 tx=0).stop 2024-05-28T06:05:07.775 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.773+0000 7fb7d0e00640 1 -- 172.21.15.96:0/1046707271 shutdown_connections 2024-05-28T06:05:07.775 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.773+0000 7fb7d0e00640 1 --2- 172.21.15.96:0/1046707271 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb7cc0b4b10 0x7fb7cc0725c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:07.775 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.773+0000 7fb7d0e00640 1 --2- 172.21.15.96:0/1046707271 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb7cc06f380 0x7fb7cc0b45d0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:07.775 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.773+0000 7fb7d0e00640 1 -- 172.21.15.96:0/1046707271 >> 172.21.15.96:0/1046707271 conn(0x7fb7cc06bd20 msgr2=0x7fb7cc06c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:07.775 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.773+0000 7fb7d0e00640 1 -- 172.21.15.96:0/1046707271 shutdown_connections 2024-05-28T06:05:07.775 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.773+0000 7fb7d0e00640 1 -- 172.21.15.96:0/1046707271 wait complete. 2024-05-28T06:05:07.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.775+0000 7fb7d0e00640 1 Processor -- start 2024-05-28T06:05:07.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.775+0000 7fb7d0e00640 1 -- start start 2024-05-28T06:05:07.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.776+0000 7fb7d0e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb7cc06f380 0x7fb7cc1558a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:07.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.776+0000 7fb7d0e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb7cc0b4b10 0x7fb7cc15e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:07.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.777+0000 7fb7d0e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb7cc15abe0 0x7fb7cc15e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:07.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.777+0000 7fb7d0e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb7cc15afc0 con 0x7fb7cc06f380 2024-05-28T06:05:07.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.777+0000 7fb7d0e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fb7cc06ee60 con 0x7fb7cc0b4b10 2024-05-28T06:05:07.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.777+0000 7fb7d0e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fb7cc06df60 con 0x7fb7cc15abe0 2024-05-28T06:05:07.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.777+0000 7fb7c9e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb7cc06f380 0x7fb7cc1558a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:07.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.777+0000 7fb7c9e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb7cc06f380 0x7fb7cc1558a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54848/0 (socket says 172.21.15.96:54848) 2024-05-28T06:05:07.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.777+0000 7fb7c9e00640 1 -- 172.21.15.96:0/1986357459 learned_addr learned my addr 172.21.15.96:0/1986357459 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:07.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.777+0000 7fb7c9400640 1 --2- 172.21.15.96:0/1986357459 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb7cc0b4b10 0x7fb7cc15e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:07.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.777+0000 7fb7ca800640 1 --2- 172.21.15.96:0/1986357459 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb7cc15abe0 0x7fb7cc15e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:07.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.777+0000 7fb7c9e00640 1 -- 172.21.15.96:0/1986357459 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb7cc15abe0 msgr2=0x7fb7cc15e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:07.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.777+0000 7fb7c9e00640 1 --2- 172.21.15.96:0/1986357459 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb7cc15abe0 0x7fb7cc15e9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:07.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.777+0000 7fb7c9e00640 1 -- 172.21.15.96:0/1986357459 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb7cc0b4b10 msgr2=0x7fb7cc15e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:07.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.777+0000 7fb7c9e00640 1 --2- 172.21.15.96:0/1986357459 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb7cc0b4b10 0x7fb7cc15e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:07.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.777+0000 7fb7c9e00640 1 -- 172.21.15.96:0/1986357459 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7cc155de0 con 0x7fb7cc06f380 2024-05-28T06:05:07.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.778+0000 7fb7c9e00640 1 --2- 172.21.15.96:0/1986357459 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb7cc06f380 0x7fb7cc1558a0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc00c390 tx=0x7fb7bc00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:07.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.778+0000 7fb7c2000640 1 -- 172.21.15.96:0/1986357459 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7bc017070 con 0x7fb7cc06f380 2024-05-28T06:05:07.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.778+0000 7fb7c2000640 1 -- 172.21.15.96:0/1986357459 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7bc005ce0 con 0x7fb7cc06f380 2024-05-28T06:05:07.781 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.778+0000 7fb7c2000640 1 -- 172.21.15.96:0/1986357459 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7bc0072f0 con 0x7fb7cc06f380 2024-05-28T06:05:07.781 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.778+0000 7fb7d0e00640 1 -- 172.21.15.96:0/1986357459 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7cc1f7f60 con 0x7fb7cc06f380 2024-05-28T06:05:07.781 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.778+0000 7fb7d0e00640 1 -- 172.21.15.96:0/1986357459 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7cc1f8520 con 0x7fb7cc06f380 2024-05-28T06:05:07.781 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.779+0000 7fb7d0e00640 1 -- 172.21.15.96:0/1986357459 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7cc0689e0 con 0x7fb7cc06f380 2024-05-28T06:05:07.781 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.780+0000 7fb7c2000640 1 -- 172.21.15.96:0/1986357459 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb7bc01a070 con 0x7fb7cc06f380 2024-05-28T06:05:07.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.782+0000 7fb7c2000640 1 --2- 172.21.15.96:0/1986357459 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb7a806c8f0 0x7fb7a806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:07.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.782+0000 7fb7c2000640 1 -- 172.21.15.96:0/1986357459 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5825+0+0 (secure 0 0 0) 0x7fb7bc08afe0 con 0x7fb7cc06f380 2024-05-28T06:05:07.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.782+0000 7fb7c9400640 1 --2- 172.21.15.96:0/1986357459 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb7a806c8f0 0x7fb7a806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:07.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.783+0000 7fb7c9400640 1 --2- 172.21.15.96:0/1986357459 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb7a806c8f0 0x7fb7a806edb0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb7b4034000 tx=0x7fb7b402f000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:07.787 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.786+0000 7fb7c2000640 1 -- 172.21.15.96:0/1986357459 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb7bc0593a0 con 0x7fb7cc06f380 2024-05-28T06:05:07.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.992+0000 7fb7d0e00640 1 -- 172.21.15.96:0/1986357459 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fb7cc06dd40 con 0x7fb7cc06f380 2024-05-28T06:05:07.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.993+0000 7fb7c2000640 1 -- 172.21.15.96:0/1986357459 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) ==== 100+0+11 (secure 0 0 0) 0x7fb7bc058ca0 con 0x7fb7cc06f380 2024-05-28T06:05:07.994 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 64 2024-05-28T06:05:07.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.995+0000 7fb7d0e00640 1 -- 172.21.15.96:0/1986357459 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb7a806c8f0 msgr2=0x7fb7a806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:07.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.995+0000 7fb7d0e00640 1 --2- 172.21.15.96:0/1986357459 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb7a806c8f0 0x7fb7a806edb0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb7b4034000 tx=0x7fb7b402f000 comp rx=0 tx=0).stop 2024-05-28T06:05:07.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.995+0000 7fb7d0e00640 1 -- 172.21.15.96:0/1986357459 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb7cc06f380 msgr2=0x7fb7cc1558a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:07.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.995+0000 7fb7d0e00640 1 --2- 172.21.15.96:0/1986357459 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb7cc06f380 0x7fb7cc1558a0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc00c390 tx=0x7fb7bc00c850 comp rx=0 tx=0).stop 2024-05-28T06:05:07.998 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.996+0000 7fb7d0e00640 1 -- 172.21.15.96:0/1986357459 shutdown_connections 2024-05-28T06:05:07.998 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.996+0000 7fb7d0e00640 1 --2- 172.21.15.96:0/1986357459 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb7a806c8f0 0x7fb7a806edb0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:07.998 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.996+0000 7fb7d0e00640 1 --2- 172.21.15.96:0/1986357459 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb7cc15abe0 0x7fb7cc15e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:07.998 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.996+0000 7fb7d0e00640 1 --2- 172.21.15.96:0/1986357459 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb7cc0b4b10 0x7fb7cc15e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:07.998 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.996+0000 7fb7d0e00640 1 --2- 172.21.15.96:0/1986357459 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb7cc06f380 0x7fb7cc1558a0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:07.998 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.996+0000 7fb7d0e00640 1 -- 172.21.15.96:0/1986357459 >> 172.21.15.96:0/1986357459 conn(0x7fb7cc06bd20 msgr2=0x7fb7cc156620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:07.998 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.996+0000 7fb7d0e00640 1 -- 172.21.15.96:0/1986357459 shutdown_connections 2024-05-28T06:05:07.998 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:07.996+0000 7fb7d0e00640 1 -- 172.21.15.96:0/1986357459 wait complete. 2024-05-28T06:05:08.016 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-28T06:05:08.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.213+0000 7f1539400640 1 Processor -- start 2024-05-28T06:05:08.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.214+0000 7f1539400640 1 -- start start 2024-05-28T06:05:08.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.215+0000 7f1539400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f153415e870 0x7f15340d0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:08.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.215+0000 7f1539400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f15340cfa40 0x7f15340d25e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:08.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.215+0000 7f1539400640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f153415ec50 con 0x7f15340d14d0 2024-05-28T06:05:08.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.215+0000 7f1539400640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f15340d2150 con 0x7f153415e870 2024-05-28T06:05:08.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.215+0000 7f1539400640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f15340cff80 con 0x7f15340cfa40 2024-05-28T06:05:08.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.216+0000 7f1533400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f15340d14d0 0x7f15340cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:60472/0 (socket says 172.21.15.96:60472) 2024-05-28T06:05:08.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.216+0000 7f1533400640 1 -- 172.21.15.96:0/3896851159 learned_addr learned my addr 172.21.15.96:0/3896851159 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:05:08.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.216+0000 7f1533e00640 1 --2- 172.21.15.96:0/3896851159 >> v2:172.21.15.96:3300/0 conn(0x7f153415e870 0x7f15340d0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:08.218 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.216+0000 7f1538a00640 1 --2- 172.21.15.96:0/3896851159 >> v2:172.21.15.123:3300/0 conn(0x7f15340cfa40 0x7f15340d25e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:08.218 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.216+0000 7f1533e00640 1 -- 172.21.15.96:0/3896851159 >> v2:172.21.15.123:3300/0 conn(0x7f15340cfa40 msgr2=0x7f15340d25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:08.218 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.216+0000 7f1533e00640 1 --2- 172.21.15.96:0/3896851159 >> v2:172.21.15.123:3300/0 conn(0x7f15340cfa40 0x7f15340d25e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:08.218 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.216+0000 7f1533e00640 1 -- 172.21.15.96:0/3896851159 >> v1:172.21.15.96:6789/0 conn(0x7f15340d14d0 legacy=0x7f15340cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:08.218 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.216+0000 7f1533e00640 1 -- 172.21.15.96:0/3896851159 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15340d0260 con 0x7f153415e870 2024-05-28T06:05:08.218 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.216+0000 7f1532a00640 1 -- 172.21.15.96:0/3896851159 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2046585623 0 0) 0x7f153415ec50 con 0x7f15340d14d0 2024-05-28T06:05:08.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.217+0000 7f1533e00640 1 --2- 172.21.15.96:0/3896851159 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f153415e870 0x7f15340d0f90 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f153408cad0 tx=0x7f152402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e1a655ab580446e server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:08.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.218+0000 7f1532a00640 1 -- 172.21.15.96:0/3896851159 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1524037070 con 0x7f153415e870 2024-05-28T06:05:08.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.218+0000 7f1532a00640 1 -- 172.21.15.96:0/3896851159 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1524032440 con 0x7f153415e870 2024-05-28T06:05:08.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.218+0000 7f1532a00640 1 -- 172.21.15.96:0/3896851159 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1524032810 con 0x7f153415e870 2024-05-28T06:05:08.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.218+0000 7f1539400640 1 -- 172.21.15.96:0/3896851159 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f153415e870 msgr2=0x7f15340d0f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:08.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.218+0000 7f1539400640 1 --2- 172.21.15.96:0/3896851159 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f153415e870 0x7f15340d0f90 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f153408cad0 tx=0x7f152402a1a0 comp rx=0 tx=0).stop 2024-05-28T06:05:08.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.218+0000 7f1539400640 1 -- 172.21.15.96:0/3896851159 shutdown_connections 2024-05-28T06:05:08.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.218+0000 7f1539400640 1 --2- 172.21.15.96:0/3896851159 >> v2:172.21.15.123:3300/0 conn(0x7f15340cfa40 0x7f15340d25e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:08.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.218+0000 7f1539400640 1 --2- 172.21.15.96:0/3896851159 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f153415e870 0x7f15340d0f90 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:08.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.218+0000 7f1539400640 1 -- 172.21.15.96:0/3896851159 >> 172.21.15.96:0/3896851159 conn(0x7f153408bfc0 msgr2=0x7f15340d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:08.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.218+0000 7f1539400640 1 -- 172.21.15.96:0/3896851159 shutdown_connections 2024-05-28T06:05:08.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.218+0000 7f1539400640 1 -- 172.21.15.96:0/3896851159 wait complete. 2024-05-28T06:05:08.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.221+0000 7f1539400640 1 Processor -- start 2024-05-28T06:05:08.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.221+0000 7f1539400640 1 -- start start 2024-05-28T06:05:08.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.222+0000 7f1539400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f15340cfa40 0x7f1534159b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:08.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.222+0000 7f1539400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f15340d14d0 0x7f15340ca410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:08.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.222+0000 7f1539400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f153415e870 0x7f15340ca950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:08.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.222+0000 7f1539400640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f15340d4fb0 con 0x7f153415e870 2024-05-28T06:05:08.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.222+0000 7f1539400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f15340d0820 con 0x7f15340d14d0 2024-05-28T06:05:08.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.222+0000 7f1539400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f15340d2b20 con 0x7f15340cfa40 2024-05-28T06:05:08.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.222+0000 7f1538a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f153415e870 0x7f15340ca950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:08.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.222+0000 7f1538a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f153415e870 0x7f15340ca950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54858/0 (socket says 172.21.15.96:54858) 2024-05-28T06:05:08.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.222+0000 7f1538a00640 1 -- 172.21.15.96:0/591562370 learned_addr learned my addr 172.21.15.96:0/591562370 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:08.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.223+0000 7f1533e00640 1 --2- 172.21.15.96:0/591562370 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f15340cfa40 0x7f1534159b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:08.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.223+0000 7f1533400640 1 --2- 172.21.15.96:0/591562370 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f15340d14d0 0x7f15340ca410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:08.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.223+0000 7f1538a00640 1 -- 172.21.15.96:0/591562370 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f15340cfa40 msgr2=0x7f1534159b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:08.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.223+0000 7f1538a00640 1 --2- 172.21.15.96:0/591562370 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f15340cfa40 0x7f1534159b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:08.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.223+0000 7f1538a00640 1 -- 172.21.15.96:0/591562370 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f15340d14d0 msgr2=0x7f15340ca410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:08.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.223+0000 7f1538a00640 1 --2- 172.21.15.96:0/591562370 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f15340d14d0 0x7f15340ca410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:08.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.223+0000 7f1538a00640 1 -- 172.21.15.96:0/591562370 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15340cdb70 con 0x7f153415e870 2024-05-28T06:05:08.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.223+0000 7f1538a00640 1 --2- 172.21.15.96:0/591562370 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f153415e870 0x7f15340ca950 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f152c00cd60 tx=0x7f152c00a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:08.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.223+0000 7f152be00640 1 -- 172.21.15.96:0/591562370 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f152c003b90 con 0x7f153415e870 2024-05-28T06:05:08.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.223+0000 7f152be00640 1 -- 172.21.15.96:0/591562370 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f152c003d50 con 0x7f153415e870 2024-05-28T06:05:08.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.223+0000 7f152be00640 1 -- 172.21.15.96:0/591562370 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f152c004260 con 0x7f153415e870 2024-05-28T06:05:08.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.223+0000 7f1539400640 1 -- 172.21.15.96:0/591562370 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15340cde50 con 0x7f153415e870 2024-05-28T06:05:08.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.224+0000 7f1539400640 1 -- 172.21.15.96:0/591562370 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15340ce410 con 0x7f153415e870 2024-05-28T06:05:08.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.225+0000 7f1539400640 1 -- 172.21.15.96:0/591562370 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15340689e0 con 0x7f153415e870 2024-05-28T06:05:08.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.226+0000 7f152be00640 1 -- 172.21.15.96:0/591562370 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f152c004400 con 0x7f153415e870 2024-05-28T06:05:08.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.227+0000 7f152be00640 1 --2- 172.21.15.96:0/591562370 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f151406c8f0 0x7f151406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:08.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.227+0000 7f152be00640 1 -- 172.21.15.96:0/591562370 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5825+0+0 (secure 0 0 0) 0x7f152c08c730 con 0x7f153415e870 2024-05-28T06:05:08.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.227+0000 7f1533e00640 1 --2- 172.21.15.96:0/591562370 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f151406c8f0 0x7f151406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:08.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.228+0000 7f1533e00640 1 --2- 172.21.15.96:0/591562370 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f151406c8f0 0x7f151406edb0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1524034000 tx=0x7f152402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:08.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.231+0000 7f152be00640 1 -- 172.21.15.96:0/591562370 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f152c05aa40 con 0x7f153415e870 2024-05-28T06:05:08.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.438+0000 7f1539400640 1 -- 172.21.15.96:0/591562370 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f15340d1950 con 0x7f153415e870 2024-05-28T06:05:08.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.439+0000 7f152be00640 1 -- 172.21.15.96:0/591562370 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) ==== 80+0+0 (secure 0 0 0) 0x7f152c05a340 con 0x7f153415e870 2024-05-28T06:05:08.440 INFO:teuthology.orchestra.run.smithi096.stderr:listed 0 entries 2024-05-28T06:05:08.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.441+0000 7f1539400640 1 -- 172.21.15.96:0/591562370 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f151406c8f0 msgr2=0x7f151406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:08.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.441+0000 7f1539400640 1 --2- 172.21.15.96:0/591562370 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f151406c8f0 0x7f151406edb0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1524034000 tx=0x7f152402e000 comp rx=0 tx=0).stop 2024-05-28T06:05:08.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.441+0000 7f1539400640 1 -- 172.21.15.96:0/591562370 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f153415e870 msgr2=0x7f15340ca950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:08.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.441+0000 7f1539400640 1 --2- 172.21.15.96:0/591562370 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f153415e870 0x7f15340ca950 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f152c00cd60 tx=0x7f152c00a660 comp rx=0 tx=0).stop 2024-05-28T06:05:08.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.442+0000 7f1539400640 1 -- 172.21.15.96:0/591562370 shutdown_connections 2024-05-28T06:05:08.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.442+0000 7f1539400640 1 --2- 172.21.15.96:0/591562370 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f151406c8f0 0x7f151406edb0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:08.444 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.442+0000 7f1539400640 1 --2- 172.21.15.96:0/591562370 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f153415e870 0x7f15340ca950 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:08.444 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.442+0000 7f1539400640 1 --2- 172.21.15.96:0/591562370 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f15340d14d0 0x7f15340ca410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:08.444 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.442+0000 7f1539400640 1 --2- 172.21.15.96:0/591562370 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f15340cfa40 0x7f1534159b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:08.444 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.442+0000 7f1539400640 1 -- 172.21.15.96:0/591562370 >> 172.21.15.96:0/591562370 conn(0x7f153408bfc0 msgr2=0x7f153415d710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:08.444 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.442+0000 7f1539400640 1 -- 172.21.15.96:0/591562370 shutdown_connections 2024-05-28T06:05:08.444 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.442+0000 7f1539400640 1 -- 172.21.15.96:0/591562370 wait complete. 2024-05-28T06:05:08.461 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi096.front.sepia.ceph.com... 2024-05-28T06:05:08.461 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:05:08.462 DEBUG:teuthology.orchestra.run.smithi096:> sudo fusermount -u /home/ubuntu/cephtest/mnt.2 2024-05-28T06:05:08.531 INFO:teuthology.orchestra.run:waiting for 300 2024-05-28T06:05:08.531 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi096.front.sepia.ceph.com 2024-05-28T06:05:08.531 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:05:08.531 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-05-28T06:05:08.585 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-05-28T06:05:08.586 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:05:08.645 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-05-28T06:05:08.645 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:05:08.699 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:05:08.905 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.904+0000 7fc149400640 1 Processor -- start 2024-05-28T06:05:08.905 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.904+0000 7fc149400640 1 -- start start 2024-05-28T06:05:08.906 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.905+0000 7fc149400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fc14406ee40 0x7fc1440ad230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:08.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.905+0000 7fc149400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc14406f2f0 0x7fc1440ad770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:08.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.905+0000 7fc149400640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc144156f10 con 0x7fc144156b30 2024-05-28T06:05:08.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.905+0000 7fc149400640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fc14406dda0 con 0x7fc14406ee40 2024-05-28T06:05:08.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.905+0000 7fc149400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fc14406e080 con 0x7fc14406f2f0 2024-05-28T06:05:08.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.906+0000 7fc143400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fc14406ee40 0x7fc1440ad230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:08.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.906+0000 7fc143400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fc14406ee40 0x7fc1440ad230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51206/0 (socket says 172.21.15.96:51206) 2024-05-28T06:05:08.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.906+0000 7fc143400640 1 -- 172.21.15.96:0/1301757592 learned_addr learned my addr 172.21.15.96:0/1301757592 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:08.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.906+0000 7fc143400640 1 -- 172.21.15.96:0/1301757592 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc14406f2f0 msgr2=0x7fc1440ad770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:08.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.906+0000 7fc143400640 1 --2- 172.21.15.96:0/1301757592 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc14406f2f0 0x7fc1440ad770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:08.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.906+0000 7fc143400640 1 -- 172.21.15.96:0/1301757592 >> v1:172.21.15.123:6789/0 conn(0x7fc144156b30 legacy=0x7fc1440af6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:08.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.906+0000 7fc143400640 1 -- 172.21.15.96:0/1301757592 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc14406e360 con 0x7fc14406ee40 2024-05-28T06:05:08.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.907+0000 7fc143400640 1 --2- 172.21.15.96:0/1301757592 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc14406ee40 0x7fc1440ad230 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fc130009870 tx=0x7fc13002a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc43bacf542fa0e8 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:08.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.908+0000 7fc142a00640 1 -- 172.21.15.96:0/1301757592 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc130037070 con 0x7fc14406ee40 2024-05-28T06:05:08.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.908+0000 7fc142a00640 1 -- 172.21.15.96:0/1301757592 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc130032440 con 0x7fc14406ee40 2024-05-28T06:05:08.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.908+0000 7fc142a00640 1 -- 172.21.15.96:0/1301757592 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc130032810 con 0x7fc14406ee40 2024-05-28T06:05:08.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.908+0000 7fc149400640 1 -- 172.21.15.96:0/1301757592 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc14406ee40 msgr2=0x7fc1440ad230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:08.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.908+0000 7fc149400640 1 --2- 172.21.15.96:0/1301757592 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc14406ee40 0x7fc1440ad230 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fc130009870 tx=0x7fc13002a180 comp rx=0 tx=0).stop 2024-05-28T06:05:08.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.908+0000 7fc149400640 1 -- 172.21.15.96:0/1301757592 shutdown_connections 2024-05-28T06:05:08.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.908+0000 7fc149400640 1 --2- 172.21.15.96:0/1301757592 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc14406f2f0 0x7fc1440ad770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:08.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.908+0000 7fc149400640 1 --2- 172.21.15.96:0/1301757592 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc14406ee40 0x7fc1440ad230 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:08.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.908+0000 7fc149400640 1 -- 172.21.15.96:0/1301757592 >> 172.21.15.96:0/1301757592 conn(0x7fc1440b1fc0 msgr2=0x7fc1440b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:08.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.908+0000 7fc149400640 1 -- 172.21.15.96:0/1301757592 shutdown_connections 2024-05-28T06:05:08.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.908+0000 7fc149400640 1 -- 172.21.15.96:0/1301757592 wait complete. 2024-05-28T06:05:08.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.910+0000 7fc149400640 1 Processor -- start 2024-05-28T06:05:08.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.910+0000 7fc149400640 1 -- start start 2024-05-28T06:05:08.914 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.911+0000 7fc149400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc14406ee40 0x7fc1441f2740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:08.914 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.911+0000 7fc149400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc14406f2f0 0x7fc14415c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:08.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.911+0000 7fc149400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc144156b30 0x7fc144159760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:08.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.911+0000 7fc149400640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc1440b2db0 con 0x7fc14406f2f0 2024-05-28T06:05:08.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.911+0000 7fc149400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc1440b2c30 con 0x7fc14406ee40 2024-05-28T06:05:08.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.911+0000 7fc149400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fc1440b32c0 con 0x7fc144156b30 2024-05-28T06:05:08.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.912+0000 7fc143400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc14406f2f0 0x7fc14415c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:08.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.912+0000 7fc143400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc14406f2f0 0x7fc14415c900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51220/0 (socket says 172.21.15.96:51220) 2024-05-28T06:05:08.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.912+0000 7fc143400640 1 -- 172.21.15.96:0/3741958781 learned_addr learned my addr 172.21.15.96:0/3741958781 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:08.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.912+0000 7fc143e00640 1 --2- 172.21.15.96:0/3741958781 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc14406ee40 0x7fc1441f2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:08.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.912+0000 7fc148a00640 1 --2- 172.21.15.96:0/3741958781 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc144156b30 0x7fc144159760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:08.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.912+0000 7fc143400640 1 -- 172.21.15.96:0/3741958781 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc144156b30 msgr2=0x7fc144159760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:08.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.912+0000 7fc143400640 1 --2- 172.21.15.96:0/3741958781 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc144156b30 0x7fc144159760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:08.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.912+0000 7fc143400640 1 -- 172.21.15.96:0/3741958781 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc14406ee40 msgr2=0x7fc1441f2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:08.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.912+0000 7fc143400640 1 --2- 172.21.15.96:0/3741958781 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc14406ee40 0x7fc1441f2740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:08.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.912+0000 7fc143400640 1 -- 172.21.15.96:0/3741958781 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc144152660 con 0x7fc14406f2f0 2024-05-28T06:05:08.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.912+0000 7fc143400640 1 --2- 172.21.15.96:0/3741958781 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc14406f2f0 0x7fc14415c900 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fc130000c00 tx=0x7fc1300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:08.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.913+0000 7fc13be00640 1 -- 172.21.15.96:0/3741958781 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc130037040 con 0x7fc14406f2f0 2024-05-28T06:05:08.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.913+0000 7fc13be00640 1 -- 172.21.15.96:0/3741958781 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc130042070 con 0x7fc14406f2f0 2024-05-28T06:05:08.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.913+0000 7fc149400640 1 -- 172.21.15.96:0/3741958781 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc144157c00 con 0x7fc14406f2f0 2024-05-28T06:05:08.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.913+0000 7fc13be00640 1 -- 172.21.15.96:0/3741958781 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc130030bb0 con 0x7fc14406f2f0 2024-05-28T06:05:08.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.913+0000 7fc149400640 1 -- 172.21.15.96:0/3741958781 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc14415ce40 con 0x7fc14406f2f0 2024-05-28T06:05:08.917 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.914+0000 7fc149400640 1 -- 172.21.15.96:0/3741958781 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1440689e0 con 0x7fc14406f2f0 2024-05-28T06:05:08.917 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.915+0000 7fc13be00640 1 -- 172.21.15.96:0/3741958781 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc130047460 con 0x7fc14406f2f0 2024-05-28T06:05:08.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.916+0000 7fc13be00640 1 --2- 172.21.15.96:0/3741958781 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc12406c920 0x7fc12406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:08.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.916+0000 7fc13be00640 1 -- 172.21.15.96:0/3741958781 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5825+0+0 (secure 0 0 0) 0x7fc1300ac7e0 con 0x7fc14406f2f0 2024-05-28T06:05:08.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.917+0000 7fc143e00640 1 --2- 172.21.15.96:0/3741958781 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc12406c920 0x7fc12406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:08.919 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.918+0000 7fc143e00640 1 --2- 172.21.15.96:0/3741958781 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc12406c920 0x7fc12406ede0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fc134006fd0 tx=0x7fc134008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:08.921 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:08.920+0000 7fc13be00640 1 -- 172.21.15.96:0/3741958781 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc130045360 con 0x7fc14406f2f0 2024-05-28T06:05:09.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.122+0000 7fc149400640 1 -- 172.21.15.96:0/3741958781 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc14415a900 con 0x7fc14406f2f0 2024-05-28T06:05:09.125 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.124+0000 7fc13be00640 1 -- 172.21.15.96:0/3741958781 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) ==== 74+0+16865 (secure 0 0 0) 0x7fc13007ab80 con 0x7fc14406f2f0 2024-05-28T06:05:09.126 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:05:09.126 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":22,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:04:52.624604+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-28T06:04:08.741938+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-28T06:04:09.301470+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.b","mappings":[{"from":3,"to":1}]},{"pgid":"3.d","mappings":[{"from":7,"to":2}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:05:09.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.127+0000 7fc149400640 1 -- 172.21.15.96:0/3741958781 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc12406c920 msgr2=0x7fc12406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:09.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.127+0000 7fc149400640 1 --2- 172.21.15.96:0/3741958781 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc12406c920 0x7fc12406ede0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fc134006fd0 tx=0x7fc134008040 comp rx=0 tx=0).stop 2024-05-28T06:05:09.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.127+0000 7fc149400640 1 -- 172.21.15.96:0/3741958781 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc14406f2f0 msgr2=0x7fc14415c900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:09.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.127+0000 7fc149400640 1 --2- 172.21.15.96:0/3741958781 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc14406f2f0 0x7fc14415c900 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fc130000c00 tx=0x7fc1300046c0 comp rx=0 tx=0).stop 2024-05-28T06:05:09.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.127+0000 7fc149400640 1 -- 172.21.15.96:0/3741958781 shutdown_connections 2024-05-28T06:05:09.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.127+0000 7fc149400640 1 --2- 172.21.15.96:0/3741958781 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc12406c920 0x7fc12406ede0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:09.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.127+0000 7fc149400640 1 --2- 172.21.15.96:0/3741958781 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc144156b30 0x7fc144159760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:09.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.127+0000 7fc149400640 1 --2- 172.21.15.96:0/3741958781 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc14406f2f0 0x7fc14415c900 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:09.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.127+0000 7fc149400640 1 --2- 172.21.15.96:0/3741958781 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc14406ee40 0x7fc1441f2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:09.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.127+0000 7fc149400640 1 -- 172.21.15.96:0/3741958781 >> 172.21.15.96:0/3741958781 conn(0x7fc1440b1fc0 msgr2=0x7fc14415bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:09.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.127+0000 7fc149400640 1 -- 172.21.15.96:0/3741958781 shutdown_connections 2024-05-28T06:05:09.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.127+0000 7fc149400640 1 -- 172.21.15.96:0/3741958781 wait complete. 2024-05-28T06:05:09.149 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-28T06:03:59.517102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-28T06:04:08.741938+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-28T06:04:09.301470+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-28T06:05:09.149 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-28T06:05:09.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.341+0000 7fe336a00640 1 Processor -- start 2024-05-28T06:05:09.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.341+0000 7fe336a00640 1 -- start start 2024-05-28T06:05:09.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.343+0000 7fe336a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fe33015eed0 0x7fe3300cef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:09.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.343+0000 7fe336a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe3300cf450 0x7fe3300b3e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:09.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.343+0000 7fe336a00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3300cfc60 con 0x7fe330144520 2024-05-28T06:05:09.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.343+0000 7fe336a00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fe330144900 con 0x7fe33015eed0 2024-05-28T06:05:09.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.343+0000 7fe336a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fe33015f620 con 0x7fe3300cf450 2024-05-28T06:05:09.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.343+0000 7fe334c00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fe33015eed0 0x7fe3300cef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:09.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.343+0000 7fe335600640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fe330144520 0x7fe33015e7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:34810/0 (socket says 172.21.15.96:34810) 2024-05-28T06:05:09.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.344+0000 7fe335600640 1 -- 172.21.15.96:0/2321772011 learned_addr learned my addr 172.21.15.96:0/2321772011 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:05:09.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.344+0000 7fe336000640 1 --2- 172.21.15.96:0/2321772011 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe3300cf450 0x7fe3300b3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:09.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.344+0000 7fe334c00640 1 -- 172.21.15.96:0/2321772011 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe3300cf450 msgr2=0x7fe3300b3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:09.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.344+0000 7fe334c00640 1 --2- 172.21.15.96:0/2321772011 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe3300cf450 0x7fe3300b3e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:09.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.344+0000 7fe334c00640 1 -- 172.21.15.96:0/2321772011 >> v1:172.21.15.96:6789/0 conn(0x7fe330144520 legacy=0x7fe33015e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:09.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.344+0000 7fe32fe00640 1 -- 172.21.15.96:0/2321772011 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1393182155 0 0) 0x7fe3300cfc60 con 0x7fe330144520 2024-05-28T06:05:09.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.344+0000 7fe334c00640 1 -- 172.21.15.96:0/2321772011 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3300cff40 con 0x7fe33015eed0 2024-05-28T06:05:09.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.344+0000 7fe336000640 1 --2- 172.21.15.96:0/2321772011 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe3300cf450 0x7fe3300b3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:05:09.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.345+0000 7fe334c00640 1 --2- 172.21.15.96:0/2321772011 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe33015eed0 0x7fe3300cef10 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fe31c00a970 tx=0x7fe31c02b580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea191d650550f075 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:09.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.345+0000 7fe32fe00640 1 -- 172.21.15.96:0/2321772011 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe31c00b040 con 0x7fe33015eed0 2024-05-28T06:05:09.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.345+0000 7fe32fe00640 1 -- 172.21.15.96:0/2321772011 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe31c030710 con 0x7fe33015eed0 2024-05-28T06:05:09.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.346+0000 7fe32fe00640 1 -- 172.21.15.96:0/2321772011 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe31c030ae0 con 0x7fe33015eed0 2024-05-28T06:05:09.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.346+0000 7fe336a00640 1 -- 172.21.15.96:0/2321772011 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe33015eed0 msgr2=0x7fe3300cef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:09.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.346+0000 7fe336a00640 1 --2- 172.21.15.96:0/2321772011 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe33015eed0 0x7fe3300cef10 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fe31c00a970 tx=0x7fe31c02b580 comp rx=0 tx=0).stop 2024-05-28T06:05:09.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.346+0000 7fe336a00640 1 -- 172.21.15.96:0/2321772011 shutdown_connections 2024-05-28T06:05:09.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.346+0000 7fe336a00640 1 --2- 172.21.15.96:0/2321772011 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe3300cf450 0x7fe3300b3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:09.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.346+0000 7fe336a00640 1 --2- 172.21.15.96:0/2321772011 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe33015eed0 0x7fe3300cef10 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:09.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.346+0000 7fe336a00640 1 -- 172.21.15.96:0/2321772011 >> 172.21.15.96:0/2321772011 conn(0x7fe330073280 msgr2=0x7fe33006b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:09.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.346+0000 7fe336a00640 1 -- 172.21.15.96:0/2321772011 shutdown_connections 2024-05-28T06:05:09.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.346+0000 7fe336a00640 1 -- 172.21.15.96:0/2321772011 wait complete. 2024-05-28T06:05:09.349 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.348+0000 7fe336a00640 1 Processor -- start 2024-05-28T06:05:09.349 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.348+0000 7fe336a00640 1 -- start start 2024-05-28T06:05:09.352 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.349+0000 7fe336a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe3300cf450 0x7fe33013c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:09.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.349+0000 7fe336a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe330144520 0x7fe33015a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:09.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.349+0000 7fe336a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe33015eed0 0x7fe33013a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:09.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.349+0000 7fe336a00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe3300d0500 con 0x7fe33015eed0 2024-05-28T06:05:09.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.349+0000 7fe336a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe3300d0870 con 0x7fe330144520 2024-05-28T06:05:09.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.349+0000 7fe336a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fe3300cf8e0 con 0x7fe3300cf450 2024-05-28T06:05:09.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.350+0000 7fe336000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe33015eed0 0x7fe33013a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:09.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.350+0000 7fe336000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe33015eed0 0x7fe33013a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51234/0 (socket says 172.21.15.96:51234) 2024-05-28T06:05:09.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.350+0000 7fe336000640 1 -- 172.21.15.96:0/1129152494 learned_addr learned my addr 172.21.15.96:0/1129152494 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:09.354 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.350+0000 7fe335600640 1 --2- 172.21.15.96:0/1129152494 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe3300cf450 0x7fe33013c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:09.354 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.350+0000 7fe334c00640 1 --2- 172.21.15.96:0/1129152494 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe330144520 0x7fe33015a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:09.354 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.350+0000 7fe336000640 1 -- 172.21.15.96:0/1129152494 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe3300cf450 msgr2=0x7fe33013c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:09.354 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.350+0000 7fe336000640 1 --2- 172.21.15.96:0/1129152494 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe3300cf450 0x7fe33013c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:09.354 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.350+0000 7fe336000640 1 -- 172.21.15.96:0/1129152494 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe330144520 msgr2=0x7fe33015a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:09.354 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.350+0000 7fe336000640 1 --2- 172.21.15.96:0/1129152494 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe330144520 0x7fe33015a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:09.354 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.350+0000 7fe336000640 1 -- 172.21.15.96:0/1129152494 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe33015c1f0 con 0x7fe33015eed0 2024-05-28T06:05:09.354 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.351+0000 7fe336000640 1 --2- 172.21.15.96:0/1129152494 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe33015eed0 0x7fe33013a6c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fe32800cd80 tx=0x7fe32800a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:09.354 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.351+0000 7fe32d600640 1 -- 172.21.15.96:0/1129152494 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe32800f680 con 0x7fe33015eed0 2024-05-28T06:05:09.355 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.351+0000 7fe32d600640 1 -- 172.21.15.96:0/1129152494 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe32800f840 con 0x7fe33015eed0 2024-05-28T06:05:09.355 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.351+0000 7fe336a00640 1 -- 172.21.15.96:0/1129152494 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe33015c4d0 con 0x7fe33015eed0 2024-05-28T06:05:09.355 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.351+0000 7fe32d600640 1 -- 172.21.15.96:0/1129152494 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe32800fc30 con 0x7fe33015eed0 2024-05-28T06:05:09.355 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.351+0000 7fe336a00640 1 -- 172.21.15.96:0/1129152494 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe33015ca90 con 0x7fe33015eed0 2024-05-28T06:05:09.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.352+0000 7fe336a00640 1 -- 172.21.15.96:0/1129152494 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3300689e0 con 0x7fe33015eed0 2024-05-28T06:05:09.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.353+0000 7fe32d600640 1 -- 172.21.15.96:0/1129152494 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe3280213f0 con 0x7fe33015eed0 2024-05-28T06:05:09.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.354+0000 7fe32d600640 1 --2- 172.21.15.96:0/1129152494 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe31006c8f0 0x7fe31006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:09.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.354+0000 7fe32d600640 1 -- 172.21.15.96:0/1129152494 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5825+0+0 (secure 0 0 0) 0x7fe32808cad0 con 0x7fe33015eed0 2024-05-28T06:05:09.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.355+0000 7fe335600640 1 --2- 172.21.15.96:0/1129152494 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe31006c8f0 0x7fe31006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:09.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.356+0000 7fe335600640 1 --2- 172.21.15.96:0/1129152494 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe31006c8f0 0x7fe31006edb0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fe324000f60 tx=0x7fe324007b50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:09.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.358+0000 7fe32d600640 1 -- 172.21.15.96:0/1129152494 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe328014030 con 0x7fe33015eed0 2024-05-28T06:05:09.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.543+0000 7fe336a00640 1 -- 172.21.15.96:0/1129152494 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe33006faf0 con 0x7fe33015eed0 2024-05-28T06:05:09.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.544+0000 7fe32d600640 1 -- 172.21.15.96:0/1129152494 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) ==== 93+0+10 (secure 0 0 0) 0x7fe32805ad50 con 0x7fe33015eed0 2024-05-28T06:05:09.545 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 1 2024-05-28T06:05:09.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.546+0000 7fe336a00640 1 -- 172.21.15.96:0/1129152494 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe31006c8f0 msgr2=0x7fe31006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:09.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.546+0000 7fe336a00640 1 --2- 172.21.15.96:0/1129152494 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe31006c8f0 0x7fe31006edb0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fe324000f60 tx=0x7fe324007b50 comp rx=0 tx=0).stop 2024-05-28T06:05:09.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.546+0000 7fe336a00640 1 -- 172.21.15.96:0/1129152494 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe33015eed0 msgr2=0x7fe33013a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:09.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.546+0000 7fe336a00640 1 --2- 172.21.15.96:0/1129152494 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe33015eed0 0x7fe33013a6c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fe32800cd80 tx=0x7fe32800a650 comp rx=0 tx=0).stop 2024-05-28T06:05:09.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.546+0000 7fe336a00640 1 -- 172.21.15.96:0/1129152494 shutdown_connections 2024-05-28T06:05:09.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.546+0000 7fe336a00640 1 --2- 172.21.15.96:0/1129152494 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe31006c8f0 0x7fe31006edb0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:09.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.546+0000 7fe336a00640 1 --2- 172.21.15.96:0/1129152494 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe33015eed0 0x7fe33013a6c0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:09.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.546+0000 7fe336a00640 1 --2- 172.21.15.96:0/1129152494 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe330144520 0x7fe33015a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:09.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.546+0000 7fe336a00640 1 --2- 172.21.15.96:0/1129152494 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe3300cf450 0x7fe33013c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:09.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.546+0000 7fe336a00640 1 -- 172.21.15.96:0/1129152494 >> 172.21.15.96:0/1129152494 conn(0x7fe330073280 msgr2=0x7fe3300b1d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:09.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.546+0000 7fe336a00640 1 -- 172.21.15.96:0/1129152494 shutdown_connections 2024-05-28T06:05:09.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.546+0000 7fe336a00640 1 -- 172.21.15.96:0/1129152494 wait complete. 2024-05-28T06:05:09.565 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-28T06:05:09.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.751+0000 7f6efb000640 1 Processor -- start 2024-05-28T06:05:09.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.752+0000 7f6efb000640 1 -- start start 2024-05-28T06:05:09.755 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.754+0000 7f6efb000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f6ef415ea40 0x7f6ef40cfab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:09.755 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.754+0000 7f6efb000640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ef40d37b0 con 0x7f6ef415ee20 2024-05-28T06:05:09.755 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.754+0000 7f6efb000640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ef40cecf0 con 0x7f6ef40d5190 2024-05-28T06:05:09.755 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.754+0000 7f6efb000640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f6ef40cefd0 con 0x7f6ef415ea40 2024-05-28T06:05:09.755 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.754+0000 7f6ef9c00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f6ef415ea40 0x7f6ef40cfab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:09.755 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.754+0000 7f6efa600640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f6ef415ee20 0x7f6ef415fff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:34816/0 (socket says 172.21.15.96:34816) 2024-05-28T06:05:09.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.754+0000 7f6efa600640 1 -- 172.21.15.96:0/2915180884 learned_addr learned my addr 172.21.15.96:0/2915180884 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:05:09.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.754+0000 7f6ef9c00640 1 -- 172.21.15.96:0/2915180884 >> v1:172.21.15.123:6789/0 conn(0x7f6ef40d5190 legacy=0x7f6ef40d30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:09.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.755+0000 7f6ef9c00640 1 -- 172.21.15.96:0/2915180884 >> v1:172.21.15.96:6789/0 conn(0x7f6ef415ee20 legacy=0x7f6ef415fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:09.757 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.755+0000 7f6ef9c00640 1 -- 172.21.15.96:0/2915180884 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ef40cf2b0 con 0x7f6ef415ea40 2024-05-28T06:05:09.757 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.755+0000 7f6ef3e00640 1 -- 172.21.15.96:0/2915180884 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1648379780 0 0) 0x7f6ef40d37b0 con 0x7f6ef415ee20 2024-05-28T06:05:09.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.756+0000 7f6ef9c00640 1 --2- 172.21.15.96:0/2915180884 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6ef415ea40 0x7f6ef40cfab0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f6ee8009d90 tx=0x7f6ee802a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cd30ed6159acf32 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:09.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.756+0000 7f6ef3e00640 1 -- 172.21.15.96:0/2915180884 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ee8007740 con 0x7f6ef415ea40 2024-05-28T06:05:09.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.756+0000 7f6ef3e00640 1 -- 172.21.15.96:0/2915180884 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ee80078e0 con 0x7f6ef415ea40 2024-05-28T06:05:09.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.756+0000 7f6ef3e00640 1 -- 172.21.15.96:0/2915180884 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ee8007cb0 con 0x7f6ef415ea40 2024-05-28T06:05:09.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.756+0000 7f6efb000640 1 -- 172.21.15.96:0/2915180884 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6ef415ea40 msgr2=0x7f6ef40cfab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:09.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.756+0000 7f6efb000640 1 --2- 172.21.15.96:0/2915180884 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6ef415ea40 0x7f6ef40cfab0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f6ee8009d90 tx=0x7f6ee802a6a0 comp rx=0 tx=0).stop 2024-05-28T06:05:09.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.757+0000 7f6efb000640 1 -- 172.21.15.96:0/2915180884 shutdown_connections 2024-05-28T06:05:09.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.757+0000 7f6efb000640 1 --2- 172.21.15.96:0/2915180884 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6ef415ea40 0x7f6ef40cfab0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:09.759 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.757+0000 7f6efb000640 1 -- 172.21.15.96:0/2915180884 >> 172.21.15.96:0/2915180884 conn(0x7f6ef408bfc0 msgr2=0x7f6ef40d4500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:09.759 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.757+0000 7f6efb000640 1 -- 172.21.15.96:0/2915180884 shutdown_connections 2024-05-28T06:05:09.759 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.757+0000 7f6efb000640 1 -- 172.21.15.96:0/2915180884 wait complete. 2024-05-28T06:05:09.760 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.758+0000 7f6efb000640 1 Processor -- start 2024-05-28T06:05:09.760 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.759+0000 7f6efb000640 1 -- start start 2024-05-28T06:05:09.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.760+0000 7f6efb000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6ef40d5190 0x7f6ef414a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:09.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.760+0000 7f6efb000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6ef415ea40 0x7f6ef40cd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:09.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.760+0000 7f6efb000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6ef415ee20 0x7f6ef40cdce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:09.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.760+0000 7f6efb000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f6ef415f2d0 con 0x7f6ef40d5190 2024-05-28T06:05:09.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.760+0000 7f6efb000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f6ef40cfff0 con 0x7f6ef415ea40 2024-05-28T06:05:09.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.760+0000 7f6efb000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f6ef415f450 con 0x7f6ef415ee20 2024-05-28T06:05:09.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.760+0000 7f6ef9c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6ef40d5190 0x7f6ef414a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:09.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.760+0000 7f6ef9c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6ef40d5190 0x7f6ef414a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51254/0 (socket says 172.21.15.96:51254) 2024-05-28T06:05:09.763 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.760+0000 7f6ef9c00640 1 -- 172.21.15.96:0/3865391625 learned_addr learned my addr 172.21.15.96:0/3865391625 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:09.763 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.760+0000 7f6ef9c00640 1 -- 172.21.15.96:0/3865391625 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6ef415ee20 msgr2=0x7f6ef40cdce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:09.763 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.760+0000 7f6ef9c00640 1 --2- 172.21.15.96:0/3865391625 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6ef415ee20 0x7f6ef40cdce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:09.763 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.760+0000 7f6ef9200640 1 --2- 172.21.15.96:0/3865391625 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6ef415ea40 0x7f6ef40cd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:09.763 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.760+0000 7f6ef9c00640 1 -- 172.21.15.96:0/3865391625 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6ef415ea40 msgr2=0x7f6ef40cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:09.763 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.760+0000 7f6ef9c00640 1 --2- 172.21.15.96:0/3865391625 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6ef415ea40 0x7f6ef40cd7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:09.763 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.760+0000 7f6ef9c00640 1 -- 172.21.15.96:0/3865391625 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ef4158670 con 0x7f6ef40d5190 2024-05-28T06:05:09.763 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.761+0000 7f6ef9c00640 1 --2- 172.21.15.96:0/3865391625 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6ef40d5190 0x7f6ef414a7d0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f6ee8009d90 tx=0x7f6ee8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:09.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.761+0000 7f6ef1600640 1 -- 172.21.15.96:0/3865391625 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ee8040070 con 0x7f6ef40d5190 2024-05-28T06:05:09.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.761+0000 7f6ef1600640 1 -- 172.21.15.96:0/3865391625 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ee8031c40 con 0x7f6ef40d5190 2024-05-28T06:05:09.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.761+0000 7f6efb000640 1 -- 172.21.15.96:0/3865391625 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ef40b9b30 con 0x7f6ef40d5190 2024-05-28T06:05:09.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.761+0000 7f6ef1600640 1 -- 172.21.15.96:0/3865391625 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ee803c420 con 0x7f6ef40d5190 2024-05-28T06:05:09.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.761+0000 7f6efb000640 1 -- 172.21.15.96:0/3865391625 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ef414ad10 con 0x7f6ef40d5190 2024-05-28T06:05:09.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.762+0000 7f6efb000640 1 -- 172.21.15.96:0/3865391625 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ef40690f0 con 0x7f6ef40d5190 2024-05-28T06:05:09.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.763+0000 7f6ef1600640 1 -- 172.21.15.96:0/3865391625 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6ee8039040 con 0x7f6ef40d5190 2024-05-28T06:05:09.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.764+0000 7f6ef1600640 1 --2- 172.21.15.96:0/3865391625 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6ed406c8f0 0x7f6ed406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:09.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.764+0000 7f6ef1600640 1 -- 172.21.15.96:0/3865391625 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5825+0+0 (secure 0 0 0) 0x7f6ee80ad380 con 0x7f6ef40d5190 2024-05-28T06:05:09.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.765+0000 7f6ef9200640 1 --2- 172.21.15.96:0/3865391625 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6ed406c8f0 0x7f6ed406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:09.767 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.766+0000 7f6ef9200640 1 --2- 172.21.15.96:0/3865391625 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6ed406c8f0 0x7f6ed406edb0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f6ef4161fe0 tx=0x7f6ee0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:09.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.768+0000 7f6ef1600640 1 -- 172.21.15.96:0/3865391625 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6ee807b690 con 0x7f6ef40d5190 2024-05-28T06:05:09.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.959+0000 7f6efb000640 1 -- 172.21.15.96:0/3865391625 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f6ef40b6c60 con 0x7f6ef40d5190 2024-05-28T06:05:09.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.959+0000 7f6ef1600640 1 -- 172.21.15.96:0/3865391625 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) ==== 104+0+11 (secure 0 0 0) 0x7f6ee807af90 con 0x7f6ef40d5190 2024-05-28T06:05:09.961 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 32 2024-05-28T06:05:09.964 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.962+0000 7f6efb000640 1 -- 172.21.15.96:0/3865391625 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6ed406c8f0 msgr2=0x7f6ed406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:09.964 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.962+0000 7f6efb000640 1 --2- 172.21.15.96:0/3865391625 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6ed406c8f0 0x7f6ed406edb0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f6ef4161fe0 tx=0x7f6ee0009040 comp rx=0 tx=0).stop 2024-05-28T06:05:09.964 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.962+0000 7f6efb000640 1 -- 172.21.15.96:0/3865391625 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6ef40d5190 msgr2=0x7f6ef414a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:09.964 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.962+0000 7f6efb000640 1 --2- 172.21.15.96:0/3865391625 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6ef40d5190 0x7f6ef414a7d0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f6ee8009d90 tx=0x7f6ee8004a40 comp rx=0 tx=0).stop 2024-05-28T06:05:09.964 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.962+0000 7f6efb000640 1 -- 172.21.15.96:0/3865391625 shutdown_connections 2024-05-28T06:05:09.964 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.962+0000 7f6efb000640 1 --2- 172.21.15.96:0/3865391625 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6ed406c8f0 0x7f6ed406edb0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:09.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.962+0000 7f6efb000640 1 --2- 172.21.15.96:0/3865391625 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6ef415ee20 0x7f6ef40cdce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:09.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.962+0000 7f6efb000640 1 --2- 172.21.15.96:0/3865391625 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6ef415ea40 0x7f6ef40cd7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:09.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.962+0000 7f6efb000640 1 --2- 172.21.15.96:0/3865391625 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6ef40d5190 0x7f6ef414a7d0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:09.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.962+0000 7f6efb000640 1 -- 172.21.15.96:0/3865391625 >> 172.21.15.96:0/3865391625 conn(0x7f6ef408bfc0 msgr2=0x7f6ef4162170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:09.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.962+0000 7f6efb000640 1 -- 172.21.15.96:0/3865391625 shutdown_connections 2024-05-28T06:05:09.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:09.962+0000 7f6efb000640 1 -- 172.21.15.96:0/3865391625 wait complete. 2024-05-28T06:05:09.982 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-28T06:05:10.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.177+0000 7f11e7200640 1 Processor -- start 2024-05-28T06:05:10.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.178+0000 7f11e7200640 1 -- start start 2024-05-28T06:05:10.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.179+0000 7f11e7200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f11e015c3f0 0x7f11e015ee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:10.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.179+0000 7f11e7200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f11e015f3d0 0x7f11e00b14b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:10.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.179+0000 7f11e7200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f11e00b19f0 0x7f11e00ca6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:10.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.179+0000 7f11e7200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f11e00b1020 con 0x7f11e015f3d0 2024-05-28T06:05:10.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.179+0000 7f11e7200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f11e015f950 con 0x7f11e00b19f0 2024-05-28T06:05:10.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.179+0000 7f11e7200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f11e015fc30 con 0x7f11e015c3f0 2024-05-28T06:05:10.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.180+0000 7f11df400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f11e015f3d0 0x7f11e00b14b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:10.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.180+0000 7f11df400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f11e015f3d0 0x7f11e00b14b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51258/0 (socket says 172.21.15.96:51258) 2024-05-28T06:05:10.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.180+0000 7f11df400640 1 -- 172.21.15.96:0/210725684 learned_addr learned my addr 172.21.15.96:0/210725684 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:10.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.180+0000 7f11dfe00640 1 --2- 172.21.15.96:0/210725684 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f11e015c3f0 0x7f11e015ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:10.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.180+0000 7f11e5200640 1 --2- 172.21.15.96:0/210725684 >> v2:172.21.15.123:3300/0 conn(0x7f11e00b19f0 0x7f11e00ca6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:10.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.180+0000 7f11df400640 1 -- 172.21.15.96:0/210725684 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f11e015c3f0 msgr2=0x7f11e015ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:10.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.180+0000 7f11df400640 1 --2- 172.21.15.96:0/210725684 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f11e015c3f0 0x7f11e015ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:10.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.180+0000 7f11df400640 1 -- 172.21.15.96:0/210725684 >> v2:172.21.15.123:3300/0 conn(0x7f11e00b19f0 msgr2=0x7f11e00ca6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:10.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.180+0000 7f11df400640 1 --2- 172.21.15.96:0/210725684 >> v2:172.21.15.123:3300/0 conn(0x7f11e00b19f0 0x7f11e00ca6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:10.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.180+0000 7f11df400640 1 -- 172.21.15.96:0/210725684 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11e00b0cb0 con 0x7f11e015f3d0 2024-05-28T06:05:10.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.181+0000 7f11df400640 1 --2- 172.21.15.96:0/210725684 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11e015f3d0 0x7f11e00b14b0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f11e00cb930 tx=0x7f11c802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82b45fdc15339b35 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:10.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.181+0000 7f11dea00640 1 -- 172.21.15.96:0/210725684 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11c8037070 con 0x7f11e015f3d0 2024-05-28T06:05:10.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.182+0000 7f11dea00640 1 -- 172.21.15.96:0/210725684 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11c8032440 con 0x7f11e015f3d0 2024-05-28T06:05:10.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.182+0000 7f11dea00640 1 -- 172.21.15.96:0/210725684 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11c8032810 con 0x7f11e015f3d0 2024-05-28T06:05:10.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.182+0000 7f11e7200640 1 -- 172.21.15.96:0/210725684 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11e015f3d0 msgr2=0x7f11e00b14b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:10.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.182+0000 7f11e7200640 1 --2- 172.21.15.96:0/210725684 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11e015f3d0 0x7f11e00b14b0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f11e00cb930 tx=0x7f11c802a270 comp rx=0 tx=0).stop 2024-05-28T06:05:10.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.182+0000 7f11e7200640 1 -- 172.21.15.96:0/210725684 shutdown_connections 2024-05-28T06:05:10.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.182+0000 7f11e7200640 1 --2- 172.21.15.96:0/210725684 >> v2:172.21.15.123:3300/0 conn(0x7f11e00b19f0 0x7f11e00ca6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:10.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.182+0000 7f11e7200640 1 --2- 172.21.15.96:0/210725684 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11e015f3d0 0x7f11e00b14b0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:10.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.182+0000 7f11e7200640 1 --2- 172.21.15.96:0/210725684 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f11e015c3f0 0x7f11e015ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:10.185 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.182+0000 7f11e7200640 1 -- 172.21.15.96:0/210725684 >> 172.21.15.96:0/210725684 conn(0x7f11e009aa30 msgr2=0x7f11e00b3020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:10.185 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.182+0000 7f11e7200640 1 -- 172.21.15.96:0/210725684 shutdown_connections 2024-05-28T06:05:10.185 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.182+0000 7f11e7200640 1 -- 172.21.15.96:0/210725684 wait complete. 2024-05-28T06:05:10.185 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.184+0000 7f11e7200640 1 Processor -- start 2024-05-28T06:05:10.185 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.184+0000 7f11e7200640 1 -- start start 2024-05-28T06:05:10.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.185+0000 7f11e7200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11e00b19f0 0x7f11e014ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:10.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.185+0000 7f11e7200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f11e015c3f0 0x7f11e00aed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:10.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.185+0000 7f11e7200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f11e015f3d0 0x7f11e015a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:10.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.185+0000 7f11e7200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f11e00cabe0 con 0x7f11e00b19f0 2024-05-28T06:05:10.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.185+0000 7f11e7200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f11e00b38e0 con 0x7f11e015f3d0 2024-05-28T06:05:10.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.185+0000 7f11e7200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f11e00cad60 con 0x7f11e015c3f0 2024-05-28T06:05:10.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.185+0000 7f11dfe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11e00b19f0 0x7f11e014ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:10.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.185+0000 7f11dfe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11e00b19f0 0x7f11e014ee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51268/0 (socket says 172.21.15.96:51268) 2024-05-28T06:05:10.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.185+0000 7f11dfe00640 1 -- 172.21.15.96:0/3287195260 learned_addr learned my addr 172.21.15.96:0/3287195260 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:10.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.185+0000 7f11df400640 1 --2- 172.21.15.96:0/3287195260 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f11e015c3f0 0x7f11e00aed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:10.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.186+0000 7f11e5200640 1 --2- 172.21.15.96:0/3287195260 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f11e015f3d0 0x7f11e015a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:10.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.186+0000 7f11dfe00640 1 -- 172.21.15.96:0/3287195260 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f11e015c3f0 msgr2=0x7f11e00aed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:10.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.186+0000 7f11dfe00640 1 --2- 172.21.15.96:0/3287195260 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f11e015c3f0 0x7f11e00aed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:10.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.186+0000 7f11dfe00640 1 -- 172.21.15.96:0/3287195260 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f11e015f3d0 msgr2=0x7f11e015a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:10.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.186+0000 7f11dfe00640 1 --2- 172.21.15.96:0/3287195260 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f11e015f3d0 0x7f11e015a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:10.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.186+0000 7f11dfe00640 1 -- 172.21.15.96:0/3287195260 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11e014f380 con 0x7f11e00b19f0 2024-05-28T06:05:10.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.186+0000 7f11df400640 1 --2- 172.21.15.96:0/3287195260 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f11e015c3f0 0x7f11e00aed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:05:10.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.186+0000 7f11e5200640 1 --2- 172.21.15.96:0/3287195260 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f11e015f3d0 0x7f11e015a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:05:10.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.186+0000 7f11dfe00640 1 --2- 172.21.15.96:0/3287195260 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11e00b19f0 0x7f11e014ee40 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f11d000d830 tx=0x7f11d000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:10.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.186+0000 7f11d7e00640 1 -- 172.21.15.96:0/3287195260 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11d000b6c0 con 0x7f11e00b19f0 2024-05-28T06:05:10.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.186+0000 7f11d7e00640 1 -- 172.21.15.96:0/3287195260 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11d0019070 con 0x7f11e00b19f0 2024-05-28T06:05:10.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.186+0000 7f11d7e00640 1 -- 172.21.15.96:0/3287195260 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11d000ba80 con 0x7f11e00b19f0 2024-05-28T06:05:10.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.186+0000 7f11e7200640 1 -- 172.21.15.96:0/3287195260 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11e00afe60 con 0x7f11e00b19f0 2024-05-28T06:05:10.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.186+0000 7f11e7200640 1 -- 172.21.15.96:0/3287195260 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11e00b47b0 con 0x7f11e00b19f0 2024-05-28T06:05:10.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.187+0000 7f11e7200640 1 -- 172.21.15.96:0/3287195260 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11e00690f0 con 0x7f11e00b19f0 2024-05-28T06:05:10.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.188+0000 7f11d7e00640 1 -- 172.21.15.96:0/3287195260 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f11d001e460 con 0x7f11e00b19f0 2024-05-28T06:05:10.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.189+0000 7f11d7e00640 1 --2- 172.21.15.96:0/3287195260 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f11bc06c920 0x7f11bc06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:10.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.189+0000 7f11d7e00640 1 -- 172.21.15.96:0/3287195260 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5825+0+0 (secure 0 0 0) 0x7f11d008c2a0 con 0x7f11e00b19f0 2024-05-28T06:05:10.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.190+0000 7f11df400640 1 --2- 172.21.15.96:0/3287195260 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f11bc06c920 0x7f11bc06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:10.192 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.191+0000 7f11df400640 1 --2- 172.21.15.96:0/3287195260 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f11bc06c920 0x7f11bc06ede0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f11c8034000 tx=0x7f11c802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:10.194 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.193+0000 7f11d7e00640 1 -- 172.21.15.96:0/3287195260 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f11e014f380 con 0x7f11e00b19f0 2024-05-28T06:05:10.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.391+0000 7f11e7200640 1 -- 172.21.15.96:0/3287195260 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f11e00b5190 con 0x7f11e00b19f0 2024-05-28T06:05:10.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.392+0000 7f11d7e00640 1 -- 172.21.15.96:0/3287195260 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) ==== 100+0+11 (secure 0 0 0) 0x7f11d005a5b0 con 0x7f11e00b19f0 2024-05-28T06:05:10.393 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 64 2024-05-28T06:05:10.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.394+0000 7f11e7200640 1 -- 172.21.15.96:0/3287195260 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f11bc06c920 msgr2=0x7f11bc06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:10.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.394+0000 7f11e7200640 1 --2- 172.21.15.96:0/3287195260 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f11bc06c920 0x7f11bc06ede0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f11c8034000 tx=0x7f11c802e000 comp rx=0 tx=0).stop 2024-05-28T06:05:10.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.394+0000 7f11e7200640 1 -- 172.21.15.96:0/3287195260 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11e00b19f0 msgr2=0x7f11e014ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:10.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.394+0000 7f11e7200640 1 --2- 172.21.15.96:0/3287195260 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11e00b19f0 0x7f11e014ee40 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f11d000d830 tx=0x7f11d000dcf0 comp rx=0 tx=0).stop 2024-05-28T06:05:10.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.394+0000 7f11e7200640 1 -- 172.21.15.96:0/3287195260 shutdown_connections 2024-05-28T06:05:10.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.394+0000 7f11e7200640 1 --2- 172.21.15.96:0/3287195260 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f11bc06c920 0x7f11bc06ede0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:10.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.394+0000 7f11e7200640 1 --2- 172.21.15.96:0/3287195260 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f11e015f3d0 0x7f11e015a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:10.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.394+0000 7f11e7200640 1 --2- 172.21.15.96:0/3287195260 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f11e015c3f0 0x7f11e00aed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:10.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.394+0000 7f11e7200640 1 --2- 172.21.15.96:0/3287195260 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11e00b19f0 0x7f11e014ee40 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:10.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.394+0000 7f11e7200640 1 -- 172.21.15.96:0/3287195260 >> 172.21.15.96:0/3287195260 conn(0x7f11e009aa30 msgr2=0x7f11e006eee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:10.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.395+0000 7f11e7200640 1 -- 172.21.15.96:0/3287195260 shutdown_connections 2024-05-28T06:05:10.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.395+0000 7f11e7200640 1 -- 172.21.15.96:0/3287195260 wait complete. 2024-05-28T06:05:10.414 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-28T06:05:10.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.605+0000 7f6f2aa00640 1 Processor -- start 2024-05-28T06:05:10.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.605+0000 7f6f2aa00640 1 -- start start 2024-05-28T06:05:10.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.606+0000 7f6f2aa00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6f2406ee40 0x7f6f240ad230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:10.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.607+0000 7f6f2aa00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f6f2406f2c0 0x7f6f240ad770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:10.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.607+0000 7f6f2aa00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f24156f10 con 0x7f6f24156b30 2024-05-28T06:05:10.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.607+0000 7f6f2aa00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f6f2406dda0 con 0x7f6f2406f2c0 2024-05-28T06:05:10.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.607+0000 7f6f2aa00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f6f2406e080 con 0x7f6f2406ee40 2024-05-28T06:05:10.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.607+0000 7f6f2a000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f6f2406f2c0 0x7f6f240ad770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:10.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.607+0000 7f6f29600640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f6f24156b30 0x7f6f240af6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:34832/0 (socket says 172.21.15.96:34832) 2024-05-28T06:05:10.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.607+0000 7f6f29600640 1 -- 172.21.15.96:0/3133178963 learned_addr learned my addr 172.21.15.96:0/3133178963 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:05:10.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.607+0000 7f6f28c00640 1 --2- 172.21.15.96:0/3133178963 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6f2406ee40 0x7f6f240ad230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:10.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.607+0000 7f6f2a000640 1 -- 172.21.15.96:0/3133178963 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6f2406ee40 msgr2=0x7f6f240ad230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:10.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.607+0000 7f6f2a000640 1 --2- 172.21.15.96:0/3133178963 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6f2406ee40 0x7f6f240ad230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:10.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.607+0000 7f6f2a000640 1 -- 172.21.15.96:0/3133178963 >> v1:172.21.15.96:6789/0 conn(0x7f6f24156b30 legacy=0x7f6f240af6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:10.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.607+0000 7f6f2a000640 1 -- 172.21.15.96:0/3133178963 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f2406e360 con 0x7f6f2406f2c0 2024-05-28T06:05:10.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.607+0000 7f6f23e00640 1 -- 172.21.15.96:0/3133178963 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1061821599 0 0) 0x7f6f24156f10 con 0x7f6f24156b30 2024-05-28T06:05:10.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.608+0000 7f6f2a000640 1 --2- 172.21.15.96:0/3133178963 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6f2406f2c0 0x7f6f240ad770 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f6f1c009d90 tx=0x7f6f1c02a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4086535ae37cb071 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:10.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.609+0000 7f6f23e00640 1 -- 172.21.15.96:0/3133178963 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f1c031600 con 0x7f6f2406f2c0 2024-05-28T06:05:10.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.609+0000 7f6f23e00640 1 -- 172.21.15.96:0/3133178963 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f1c0317a0 con 0x7f6f2406f2c0 2024-05-28T06:05:10.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.609+0000 7f6f23e00640 1 -- 172.21.15.96:0/3133178963 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f1c031b70 con 0x7f6f2406f2c0 2024-05-28T06:05:10.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.609+0000 7f6f2aa00640 1 -- 172.21.15.96:0/3133178963 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6f2406f2c0 msgr2=0x7f6f240ad770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:10.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.609+0000 7f6f2aa00640 1 --2- 172.21.15.96:0/3133178963 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6f2406f2c0 0x7f6f240ad770 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f6f1c009d90 tx=0x7f6f1c02a5d0 comp rx=0 tx=0).stop 2024-05-28T06:05:10.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.609+0000 7f6f2aa00640 1 -- 172.21.15.96:0/3133178963 shutdown_connections 2024-05-28T06:05:10.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.609+0000 7f6f2aa00640 1 --2- 172.21.15.96:0/3133178963 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6f2406f2c0 0x7f6f240ad770 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:10.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.609+0000 7f6f2aa00640 1 --2- 172.21.15.96:0/3133178963 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6f2406ee40 0x7f6f240ad230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:10.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.609+0000 7f6f2aa00640 1 -- 172.21.15.96:0/3133178963 >> 172.21.15.96:0/3133178963 conn(0x7f6f240b1fc0 msgr2=0x7f6f240b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:10.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.609+0000 7f6f2aa00640 1 -- 172.21.15.96:0/3133178963 shutdown_connections 2024-05-28T06:05:10.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.609+0000 7f6f2aa00640 1 -- 172.21.15.96:0/3133178963 wait complete. 2024-05-28T06:05:10.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.611+0000 7f6f2aa00640 1 Processor -- start 2024-05-28T06:05:10.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.611+0000 7f6f2aa00640 1 -- start start 2024-05-28T06:05:10.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.612+0000 7f6f2aa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6f2406ee40 0x7f6f2414c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:10.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.612+0000 7f6f2aa00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f2406f2c0 0x7f6f24156fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:10.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.612+0000 7f6f2aa00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6f24156b30 0x7f6f241574e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:10.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.612+0000 7f6f2aa00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f6f240b2db0 con 0x7f6f2406ee40 2024-05-28T06:05:10.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.612+0000 7f6f2aa00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f6f240b2c30 con 0x7f6f2406f2c0 2024-05-28T06:05:10.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.612+0000 7f6f2aa00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f6f240b32c0 con 0x7f6f24156b30 2024-05-28T06:05:10.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.612+0000 7f6f29600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6f2406ee40 0x7f6f2414c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:10.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.612+0000 7f6f29600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6f2406ee40 0x7f6f2414c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51290/0 (socket says 172.21.15.96:51290) 2024-05-28T06:05:10.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.612+0000 7f6f29600640 1 -- 172.21.15.96:0/2692602147 learned_addr learned my addr 172.21.15.96:0/2692602147 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:10.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.613+0000 7f6f29600640 1 -- 172.21.15.96:0/2692602147 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6f24156b30 msgr2=0x7f6f241574e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:10.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.613+0000 7f6f29600640 1 --2- 172.21.15.96:0/2692602147 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6f24156b30 0x7f6f241574e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:10.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.613+0000 7f6f29600640 1 -- 172.21.15.96:0/2692602147 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f2406f2c0 msgr2=0x7f6f24156fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:10.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.613+0000 7f6f29600640 1 --2- 172.21.15.96:0/2692602147 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f2406f2c0 0x7f6f24156fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:10.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.613+0000 7f6f29600640 1 -- 172.21.15.96:0/2692602147 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f24157d60 con 0x7f6f2406ee40 2024-05-28T06:05:10.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.613+0000 7f6f29600640 1 --2- 172.21.15.96:0/2692602147 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6f2406ee40 0x7f6f2414c820 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f6f1800c460 tx=0x7f6f1800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:10.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.613+0000 7f6f21600640 1 -- 172.21.15.96:0/2692602147 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f18018070 con 0x7f6f2406ee40 2024-05-28T06:05:10.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.613+0000 7f6f21600640 1 -- 172.21.15.96:0/2692602147 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f18007500 con 0x7f6f2406ee40 2024-05-28T06:05:10.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.613+0000 7f6f2aa00640 1 -- 172.21.15.96:0/2692602147 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f24158040 con 0x7f6f2406ee40 2024-05-28T06:05:10.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.613+0000 7f6f21600640 1 -- 172.21.15.96:0/2692602147 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f18004a20 con 0x7f6f2406ee40 2024-05-28T06:05:10.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.613+0000 7f6f2aa00640 1 -- 172.21.15.96:0/2692602147 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f24158600 con 0x7f6f2406ee40 2024-05-28T06:05:10.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.615+0000 7f6f2aa00640 1 -- 172.21.15.96:0/2692602147 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f240689e0 con 0x7f6f2406ee40 2024-05-28T06:05:10.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.615+0000 7f6f21600640 1 -- 172.21.15.96:0/2692602147 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6f1801b070 con 0x7f6f2406ee40 2024-05-28T06:05:10.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.616+0000 7f6f21600640 1 --2- 172.21.15.96:0/2692602147 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6f0406c8f0 0x7f6f0406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:10.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.617+0000 7f6f21600640 1 -- 172.21.15.96:0/2692602147 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5825+0+0 (secure 0 0 0) 0x7f6f1808adf0 con 0x7f6f2406ee40 2024-05-28T06:05:10.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.617+0000 7f6f28c00640 1 --2- 172.21.15.96:0/2692602147 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6f0406c8f0 0x7f6f0406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:10.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.618+0000 7f6f28c00640 1 --2- 172.21.15.96:0/2692602147 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6f0406c8f0 0x7f6f0406edb0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f6f24071ce0 tx=0x7f6f10008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:10.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.621+0000 7f6f21600640 1 -- 172.21.15.96:0/2692602147 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6f18059230 con 0x7f6f2406ee40 2024-05-28T06:05:10.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.834+0000 7f6f2aa00640 1 -- 172.21.15.96:0/2692602147 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f6f2415c110 con 0x7f6f2406ee40 2024-05-28T06:05:10.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.834+0000 7f6f21600640 1 -- 172.21.15.96:0/2692602147 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) ==== 80+0+0 (secure 0 0 0) 0x7f6f18058b30 con 0x7f6f2406ee40 2024-05-28T06:05:10.836 INFO:teuthology.orchestra.run.smithi096.stderr:listed 0 entries 2024-05-28T06:05:10.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.837+0000 7f6f2aa00640 1 -- 172.21.15.96:0/2692602147 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6f0406c8f0 msgr2=0x7f6f0406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:10.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.837+0000 7f6f2aa00640 1 --2- 172.21.15.96:0/2692602147 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6f0406c8f0 0x7f6f0406edb0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f6f24071ce0 tx=0x7f6f10008040 comp rx=0 tx=0).stop 2024-05-28T06:05:10.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.837+0000 7f6f2aa00640 1 -- 172.21.15.96:0/2692602147 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6f2406ee40 msgr2=0x7f6f2414c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:10.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.837+0000 7f6f2aa00640 1 --2- 172.21.15.96:0/2692602147 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6f2406ee40 0x7f6f2414c820 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f6f1800c460 tx=0x7f6f1800c920 comp rx=0 tx=0).stop 2024-05-28T06:05:10.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.837+0000 7f6f2aa00640 1 -- 172.21.15.96:0/2692602147 shutdown_connections 2024-05-28T06:05:10.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.837+0000 7f6f2aa00640 1 --2- 172.21.15.96:0/2692602147 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6f0406c8f0 0x7f6f0406edb0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:10.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.837+0000 7f6f2aa00640 1 --2- 172.21.15.96:0/2692602147 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6f24156b30 0x7f6f241574e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:10.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.837+0000 7f6f2aa00640 1 --2- 172.21.15.96:0/2692602147 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f2406f2c0 0x7f6f24156fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:10.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.837+0000 7f6f2aa00640 1 --2- 172.21.15.96:0/2692602147 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6f2406ee40 0x7f6f2414c820 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:10.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.837+0000 7f6f2aa00640 1 -- 172.21.15.96:0/2692602147 >> 172.21.15.96:0/2692602147 conn(0x7f6f240b1fc0 msgr2=0x7f6f240d0250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:10.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.837+0000 7f6f2aa00640 1 -- 172.21.15.96:0/2692602147 shutdown_connections 2024-05-28T06:05:10.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:10.837+0000 7f6f2aa00640 1 -- 172.21.15.96:0/2692602147 wait complete. 2024-05-28T06:05:10.856 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-05-28T06:05:10.856 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:05:10.910 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:05:11.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.116+0000 7f64d5600640 1 Processor -- start 2024-05-28T06:05:11.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.116+0000 7f64d5600640 1 -- start start 2024-05-28T06:05:11.119 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.117+0000 7f64d5600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f64d006b900 0x7f64d00708e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:11.119 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.117+0000 7f64d5600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f64d015eb30 0x7f64d006bce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:11.119 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.117+0000 7f64d5600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f64d00b0350 0x7f64d00ae220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:11.119 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.117+0000 7f64d5600640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f64d00b5e30 con 0x7f64d00b0350 2024-05-28T06:05:11.119 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.117+0000 7f64d5600640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f64d00b4730 con 0x7f64d015eb30 2024-05-28T06:05:11.119 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.117+0000 7f64d5600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f64d0070e20 con 0x7f64d006b900 2024-05-28T06:05:11.120 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.118+0000 7f64cf400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f64d00b0350 0x7f64d00ae220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:11.120 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.118+0000 7f64cf400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f64d00b0350 0x7f64d00ae220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51306/0 (socket says 172.21.15.96:51306) 2024-05-28T06:05:11.120 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.118+0000 7f64cf400640 1 -- 172.21.15.96:0/1918354796 learned_addr learned my addr 172.21.15.96:0/1918354796 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:11.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.118+0000 7f64cea00640 1 --2- 172.21.15.96:0/1918354796 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f64d006b900 0x7f64d00708e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:11.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.118+0000 7f64cf400640 1 -- 172.21.15.96:0/1918354796 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f64d006b900 msgr2=0x7f64d00708e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:11.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.118+0000 7f64ce000640 1 --2- 172.21.15.96:0/1918354796 >> v2:172.21.15.123:3300/0 conn(0x7f64d015eb30 0x7f64d006bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:11.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.118+0000 7f64cf400640 1 --2- 172.21.15.96:0/1918354796 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f64d006b900 0x7f64d00708e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:11.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.118+0000 7f64cf400640 1 -- 172.21.15.96:0/1918354796 >> v2:172.21.15.123:3300/0 conn(0x7f64d015eb30 msgr2=0x7f64d006bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:11.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.118+0000 7f64cf400640 1 --2- 172.21.15.96:0/1918354796 >> v2:172.21.15.123:3300/0 conn(0x7f64d015eb30 0x7f64d006bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:11.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.118+0000 7f64cf400640 1 -- 172.21.15.96:0/1918354796 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64d006c220 con 0x7f64d00b0350 2024-05-28T06:05:11.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.118+0000 7f64ce000640 1 --2- 172.21.15.96:0/1918354796 >> v2:172.21.15.123:3300/0 conn(0x7f64d015eb30 0x7f64d006bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:05:11.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.119+0000 7f64cf400640 1 --2- 172.21.15.96:0/1918354796 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f64d00b0350 0x7f64d00ae220 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f64d0075930 tx=0x7f64c802a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85ef5bd1332ed257 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:11.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.120+0000 7f64cd600640 1 -- 172.21.15.96:0/1918354796 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64c8036070 con 0x7f64d00b0350 2024-05-28T06:05:11.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.120+0000 7f64cd600640 1 -- 172.21.15.96:0/1918354796 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64c8031600 con 0x7f64d00b0350 2024-05-28T06:05:11.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.120+0000 7f64cd600640 1 -- 172.21.15.96:0/1918354796 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64c80319d0 con 0x7f64d00b0350 2024-05-28T06:05:11.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.120+0000 7f64d5600640 1 -- 172.21.15.96:0/1918354796 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f64d00b0350 msgr2=0x7f64d00ae220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:11.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.120+0000 7f64d5600640 1 --2- 172.21.15.96:0/1918354796 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f64d00b0350 0x7f64d00ae220 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f64d0075930 tx=0x7f64c802a610 comp rx=0 tx=0).stop 2024-05-28T06:05:11.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.121+0000 7f64d5600640 1 -- 172.21.15.96:0/1918354796 shutdown_connections 2024-05-28T06:05:11.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.121+0000 7f64d5600640 1 --2- 172.21.15.96:0/1918354796 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f64d00b0350 0x7f64d00ae220 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:11.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.121+0000 7f64d5600640 1 --2- 172.21.15.96:0/1918354796 >> v2:172.21.15.123:3300/0 conn(0x7f64d015eb30 0x7f64d006bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:11.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.121+0000 7f64d5600640 1 --2- 172.21.15.96:0/1918354796 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f64d006b900 0x7f64d00708e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:11.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.121+0000 7f64d5600640 1 -- 172.21.15.96:0/1918354796 >> 172.21.15.96:0/1918354796 conn(0x7f64d00b6760 msgr2=0x7f64d00b6b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:11.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.121+0000 7f64d5600640 1 -- 172.21.15.96:0/1918354796 shutdown_connections 2024-05-28T06:05:11.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.121+0000 7f64d5600640 1 -- 172.21.15.96:0/1918354796 wait complete. 2024-05-28T06:05:11.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.122+0000 7f64d5600640 1 Processor -- start 2024-05-28T06:05:11.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.123+0000 7f64d5600640 1 -- start start 2024-05-28T06:05:11.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.123+0000 7f64d5600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f64d006b900 0x7f64d0161b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:11.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.123+0000 7f64d5600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64d00b0350 0x7f64d00b2f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:11.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.124+0000 7f64d5600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f64d015eb30 0x7f64d009da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:11.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.124+0000 7f64d5600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f64d00b43c0 con 0x7f64d015eb30 2024-05-28T06:05:11.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.124+0000 7f64d5600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f64d00b6340 con 0x7f64d00b0350 2024-05-28T06:05:11.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.124+0000 7f64d5600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f64d00ae760 con 0x7f64d006b900 2024-05-28T06:05:11.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.124+0000 7f64cf400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f64d015eb30 0x7f64d009da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:11.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.124+0000 7f64cf400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f64d015eb30 0x7f64d009da40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51320/0 (socket says 172.21.15.96:51320) 2024-05-28T06:05:11.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.124+0000 7f64cf400640 1 -- 172.21.15.96:0/2539045736 learned_addr learned my addr 172.21.15.96:0/2539045736 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:11.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.124+0000 7f64cea00640 1 --2- 172.21.15.96:0/2539045736 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f64d006b900 0x7f64d0161b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:11.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.124+0000 7f64ce000640 1 --2- 172.21.15.96:0/2539045736 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64d00b0350 0x7f64d00b2f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:11.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.124+0000 7f64cf400640 1 -- 172.21.15.96:0/2539045736 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f64d006b900 msgr2=0x7f64d0161b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:11.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.124+0000 7f64cf400640 1 --2- 172.21.15.96:0/2539045736 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f64d006b900 0x7f64d0161b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:11.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.124+0000 7f64cf400640 1 -- 172.21.15.96:0/2539045736 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64d00b0350 msgr2=0x7f64d00b2f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:11.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.124+0000 7f64cf400640 1 --2- 172.21.15.96:0/2539045736 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64d00b0350 0x7f64d00b2f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:11.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.124+0000 7f64cf400640 1 -- 172.21.15.96:0/2539045736 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64d00b2470 con 0x7f64d015eb30 2024-05-28T06:05:11.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.124+0000 7f64cea00640 1 --2- 172.21.15.96:0/2539045736 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f64d006b900 0x7f64d0161b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:05:11.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.124+0000 7f64ce000640 1 --2- 172.21.15.96:0/2539045736 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64d00b0350 0x7f64d00b2f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:05:11.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.125+0000 7f64cf400640 1 --2- 172.21.15.96:0/2539045736 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f64d015eb30 0x7f64d009da40 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f64c8034040 tx=0x7f64c8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:11.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.125+0000 7f64c6a00640 1 -- 172.21.15.96:0/2539045736 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64c8041020 con 0x7f64d015eb30 2024-05-28T06:05:11.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.125+0000 7f64c6a00640 1 -- 172.21.15.96:0/2539045736 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64c80077a0 con 0x7f64d015eb30 2024-05-28T06:05:11.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.125+0000 7f64c6a00640 1 -- 172.21.15.96:0/2539045736 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64c8036040 con 0x7f64d015eb30 2024-05-28T06:05:11.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.125+0000 7f64d5600640 1 -- 172.21.15.96:0/2539045736 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64d009cc10 con 0x7f64d015eb30 2024-05-28T06:05:11.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.125+0000 7f64d5600640 1 -- 172.21.15.96:0/2539045736 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64d00b2710 con 0x7f64d015eb30 2024-05-28T06:05:11.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.126+0000 7f64d5600640 1 -- 172.21.15.96:0/2539045736 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64d00690f0 con 0x7f64d015eb30 2024-05-28T06:05:11.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.127+0000 7f64c6a00640 1 -- 172.21.15.96:0/2539045736 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f64c8044020 con 0x7f64d015eb30 2024-05-28T06:05:11.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.129+0000 7f64c6a00640 1 --2- 172.21.15.96:0/2539045736 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f64ac06c920 0x7f64ac06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:11.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.129+0000 7f64c6a00640 1 -- 172.21.15.96:0/2539045736 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5825+0+0 (secure 0 0 0) 0x7f64c80acb90 con 0x7f64d015eb30 2024-05-28T06:05:11.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.129+0000 7f64cea00640 1 --2- 172.21.15.96:0/2539045736 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f64ac06c920 0x7f64ac06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:11.132 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.131+0000 7f64cea00640 1 --2- 172.21.15.96:0/2539045736 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f64ac06c920 0x7f64ac06ede0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f64c0006fd0 tx=0x7f64c0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:11.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.133+0000 7f64c6a00640 1 -- 172.21.15.96:0/2539045736 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f64c804b030 con 0x7f64d015eb30 2024-05-28T06:05:11.337 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.336+0000 7f64d5600640 1 -- 172.21.15.96:0/2539045736 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f64d00b52b0 con 0x7f64d015eb30 2024-05-28T06:05:11.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.339+0000 7f64c6a00640 1 -- 172.21.15.96:0/2539045736 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) ==== 74+0+16865 (secure 0 0 0) 0x7f64c807aff0 con 0x7f64d015eb30 2024-05-28T06:05:11.341 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:05:11.341 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":22,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:04:52.624604+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-28T06:04:08.741938+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-28T06:04:09.301470+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.b","mappings":[{"from":3,"to":1}]},{"pgid":"3.d","mappings":[{"from":7,"to":2}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:05:11.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.341+0000 7f64d5600640 1 -- 172.21.15.96:0/2539045736 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f64ac06c920 msgr2=0x7f64ac06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:11.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.341+0000 7f64d5600640 1 --2- 172.21.15.96:0/2539045736 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f64ac06c920 0x7f64ac06ede0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f64c0006fd0 tx=0x7f64c0008040 comp rx=0 tx=0).stop 2024-05-28T06:05:11.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.341+0000 7f64d5600640 1 -- 172.21.15.96:0/2539045736 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f64d015eb30 msgr2=0x7f64d009da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:11.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.341+0000 7f64d5600640 1 --2- 172.21.15.96:0/2539045736 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f64d015eb30 0x7f64d009da40 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f64c8034040 tx=0x7f64c8004a40 comp rx=0 tx=0).stop 2024-05-28T06:05:11.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.341+0000 7f64d5600640 1 -- 172.21.15.96:0/2539045736 shutdown_connections 2024-05-28T06:05:11.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.341+0000 7f64d5600640 1 --2- 172.21.15.96:0/2539045736 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f64ac06c920 0x7f64ac06ede0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:11.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.341+0000 7f64d5600640 1 --2- 172.21.15.96:0/2539045736 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f64d015eb30 0x7f64d009da40 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:11.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.341+0000 7f64d5600640 1 --2- 172.21.15.96:0/2539045736 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64d00b0350 0x7f64d00b2f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:11.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.341+0000 7f64d5600640 1 --2- 172.21.15.96:0/2539045736 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f64d006b900 0x7f64d0161b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:11.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.341+0000 7f64d5600640 1 -- 172.21.15.96:0/2539045736 >> 172.21.15.96:0/2539045736 conn(0x7f64d00b6760 msgr2=0x7f64d0160200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:11.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.342+0000 7f64d5600640 1 -- 172.21.15.96:0/2539045736 shutdown_connections 2024-05-28T06:05:11.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.342+0000 7f64d5600640 1 -- 172.21.15.96:0/2539045736 wait complete. 2024-05-28T06:05:11.363 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-28T06:03:59.517102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-28T06:04:08.741938+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-28T06:04:09.301470+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-28T06:05:11.364 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-28T06:05:11.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.571+0000 7f1771800640 1 Processor -- start 2024-05-28T06:05:11.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.572+0000 7f1771800640 1 -- start start 2024-05-28T06:05:11.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.573+0000 7f1771800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f176c0d0640 0x7f176c15d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:11.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.573+0000 7f1771800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f176c0b32c0 0x7f176c0b3750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:11.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.573+0000 7f1771800640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f176c0d0060 con 0x7f176c0d0a20 2024-05-28T06:05:11.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.573+0000 7f1771800640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f176c0adbb0 con 0x7f176c0b32c0 2024-05-28T06:05:11.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.573+0000 7f1771800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f176c15db10 con 0x7f176c0d0640 2024-05-28T06:05:11.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.574+0000 7f176aa00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f176c0d0640 0x7f176c15d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:11.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.574+0000 7f176b400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f176c0b32c0 0x7f176c0b3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:11.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.574+0000 7f176aa00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f176c0d0640 0x7f176c15d5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.96:58520/0 (socket says 172.21.15.96:58520) 2024-05-28T06:05:11.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.574+0000 7f176aa00640 1 -- 172.21.15.96:0/1389886597 learned_addr learned my addr 172.21.15.96:0/1389886597 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:11.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.574+0000 7f176aa00640 1 -- 172.21.15.96:0/1389886597 >> v2:172.21.15.123:3300/0 conn(0x7f176c0b32c0 msgr2=0x7f176c0b3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:11.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.574+0000 7f176aa00640 1 --2- 172.21.15.96:0/1389886597 >> v2:172.21.15.123:3300/0 conn(0x7f176c0b32c0 0x7f176c0b3750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:11.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.574+0000 7f176aa00640 1 -- 172.21.15.96:0/1389886597 >> v1:172.21.15.123:6789/0 conn(0x7f176c0d0a20 legacy=0x7f176c0cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:11.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.574+0000 7f176aa00640 1 -- 172.21.15.96:0/1389886597 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f176c15cb10 con 0x7f176c0d0640 2024-05-28T06:05:11.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.576+0000 7f176aa00640 1 --2- 172.21.15.96:0/1389886597 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f176c0d0640 0x7f176c15d5d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f175c009870 tx=0x7f175c02a250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6fff3a7e7ab75f31 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:11.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.576+0000 7f1769600640 1 -- 172.21.15.96:0/1389886597 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f175c037070 con 0x7f176c0d0640 2024-05-28T06:05:11.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.576+0000 7f1769600640 1 -- 172.21.15.96:0/1389886597 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f175c032440 con 0x7f176c0d0640 2024-05-28T06:05:11.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.576+0000 7f1769600640 1 -- 172.21.15.96:0/1389886597 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f175c032810 con 0x7f176c0d0640 2024-05-28T06:05:11.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.577+0000 7f1771800640 1 -- 172.21.15.96:0/1389886597 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f176c0d0640 msgr2=0x7f176c15d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:11.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.577+0000 7f1771800640 1 --2- 172.21.15.96:0/1389886597 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f176c0d0640 0x7f176c15d5d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f175c009870 tx=0x7f175c02a250 comp rx=0 tx=0).stop 2024-05-28T06:05:11.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.577+0000 7f1771800640 1 -- 172.21.15.96:0/1389886597 shutdown_connections 2024-05-28T06:05:11.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.577+0000 7f1771800640 1 --2- 172.21.15.96:0/1389886597 >> v2:172.21.15.123:3300/0 conn(0x7f176c0b32c0 0x7f176c0b3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:11.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.577+0000 7f1771800640 1 --2- 172.21.15.96:0/1389886597 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f176c0d0640 0x7f176c15d5d0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:11.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.577+0000 7f1771800640 1 -- 172.21.15.96:0/1389886597 >> 172.21.15.96:0/1389886597 conn(0x7f176c06bd20 msgr2=0x7f176c06c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:11.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.577+0000 7f1771800640 1 -- 172.21.15.96:0/1389886597 shutdown_connections 2024-05-28T06:05:11.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.577+0000 7f1771800640 1 -- 172.21.15.96:0/1389886597 wait complete. 2024-05-28T06:05:11.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.579+0000 7f1771800640 1 Processor -- start 2024-05-28T06:05:11.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.579+0000 7f1771800640 1 -- start start 2024-05-28T06:05:11.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.580+0000 7f1771800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f176c0b32c0 0x7f176c09dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:11.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.580+0000 7f1771800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f176c0d0640 0x7f176c1f3c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:11.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.580+0000 7f1771800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f176c0d0a20 0x7f176c09a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:11.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.580+0000 7f1771800640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f176c15cdd0 con 0x7f176c0d0640 2024-05-28T06:05:11.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.580+0000 7f1771800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f176c0ad8d0 con 0x7f176c0d0a20 2024-05-28T06:05:11.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.580+0000 7f1771800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f176c15d0b0 con 0x7f176c0b32c0 2024-05-28T06:05:11.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.580+0000 7f176a000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f176c0d0640 0x7f176c1f3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:11.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.580+0000 7f176a000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f176c0d0640 0x7f176c1f3c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51334/0 (socket says 172.21.15.96:51334) 2024-05-28T06:05:11.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.580+0000 7f176a000640 1 -- 172.21.15.96:0/1147521825 learned_addr learned my addr 172.21.15.96:0/1147521825 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:11.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.581+0000 7f176aa00640 1 --2- 172.21.15.96:0/1147521825 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f176c0b32c0 0x7f176c09dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:11.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.581+0000 7f176b400640 1 --2- 172.21.15.96:0/1147521825 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f176c0d0a20 0x7f176c09a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:11.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.581+0000 7f176a000640 1 -- 172.21.15.96:0/1147521825 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f176c0b32c0 msgr2=0x7f176c09dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:11.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.581+0000 7f176a000640 1 --2- 172.21.15.96:0/1147521825 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f176c0b32c0 0x7f176c09dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:11.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.581+0000 7f176a000640 1 -- 172.21.15.96:0/1147521825 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f176c0d0a20 msgr2=0x7f176c09a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:11.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.581+0000 7f176a000640 1 --2- 172.21.15.96:0/1147521825 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f176c0d0a20 0x7f176c09a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:11.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.581+0000 7f176a000640 1 -- 172.21.15.96:0/1147521825 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f176c09c910 con 0x7f176c0d0640 2024-05-28T06:05:11.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.581+0000 7f176a000640 1 --2- 172.21.15.96:0/1147521825 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f176c0d0640 0x7f176c1f3c60 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f175400c730 tx=0x7f175400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:11.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.581+0000 7f1762a00640 1 -- 172.21.15.96:0/1147521825 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1754016020 con 0x7f176c0d0640 2024-05-28T06:05:11.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.582+0000 7f1762a00640 1 -- 172.21.15.96:0/1147521825 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1754006d50 con 0x7f176c0d0640 2024-05-28T06:05:11.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.582+0000 7f1771800640 1 -- 172.21.15.96:0/1147521825 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f176c09c070 con 0x7f176c0d0640 2024-05-28T06:05:11.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.582+0000 7f1762a00640 1 -- 172.21.15.96:0/1147521825 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1754003bb0 con 0x7f176c0d0640 2024-05-28T06:05:11.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.582+0000 7f1771800640 1 -- 172.21.15.96:0/1147521825 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f176c06db60 con 0x7f176c0d0640 2024-05-28T06:05:11.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.583+0000 7f1771800640 1 -- 172.21.15.96:0/1147521825 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f176c0689e0 con 0x7f176c0d0640 2024-05-28T06:05:11.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.584+0000 7f1762a00640 1 -- 172.21.15.96:0/1147521825 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f175400a850 con 0x7f176c0d0640 2024-05-28T06:05:11.586 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.585+0000 7f1762a00640 1 --2- 172.21.15.96:0/1147521825 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f174806c8f0 0x7f174806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:11.586 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.585+0000 7f1762a00640 1 -- 172.21.15.96:0/1147521825 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5825+0+0 (secure 0 0 0) 0x7f175408b8c0 con 0x7f176c0d0640 2024-05-28T06:05:11.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.586+0000 7f176aa00640 1 --2- 172.21.15.96:0/1147521825 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f174806c8f0 0x7f174806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:11.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.587+0000 7f176aa00640 1 --2- 172.21.15.96:0/1147521825 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f174806c8f0 0x7f174806edb0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f175c034000 tx=0x7f175c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:11.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.590+0000 7f1762a00640 1 -- 172.21.15.96:0/1147521825 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1754020020 con 0x7f176c0d0640 2024-05-28T06:05:11.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.790+0000 7f1771800640 1 -- 172.21.15.96:0/1147521825 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f176c071ef0 con 0x7f176c0d0640 2024-05-28T06:05:11.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.790+0000 7f1762a00640 1 -- 172.21.15.96:0/1147521825 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) ==== 93+0+10 (secure 0 0 0) 0x7f1754019310 con 0x7f176c0d0640 2024-05-28T06:05:11.792 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 1 2024-05-28T06:05:11.794 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.793+0000 7f1771800640 1 -- 172.21.15.96:0/1147521825 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f174806c8f0 msgr2=0x7f174806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:11.794 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.793+0000 7f1771800640 1 --2- 172.21.15.96:0/1147521825 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f174806c8f0 0x7f174806edb0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f175c034000 tx=0x7f175c02e000 comp rx=0 tx=0).stop 2024-05-28T06:05:11.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.793+0000 7f1771800640 1 -- 172.21.15.96:0/1147521825 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f176c0d0640 msgr2=0x7f176c1f3c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:11.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.793+0000 7f1771800640 1 --2- 172.21.15.96:0/1147521825 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f176c0d0640 0x7f176c1f3c60 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f175400c730 tx=0x7f175400cbf0 comp rx=0 tx=0).stop 2024-05-28T06:05:11.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.793+0000 7f1771800640 1 -- 172.21.15.96:0/1147521825 shutdown_connections 2024-05-28T06:05:11.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.793+0000 7f1771800640 1 --2- 172.21.15.96:0/1147521825 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f174806c8f0 0x7f174806edb0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:11.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.793+0000 7f1771800640 1 --2- 172.21.15.96:0/1147521825 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f176c0d0a20 0x7f176c09a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:11.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.793+0000 7f1771800640 1 --2- 172.21.15.96:0/1147521825 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f176c0d0640 0x7f176c1f3c60 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:11.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.793+0000 7f1771800640 1 --2- 172.21.15.96:0/1147521825 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f176c0b32c0 0x7f176c09dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:11.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.793+0000 7f1771800640 1 -- 172.21.15.96:0/1147521825 >> 172.21.15.96:0/1147521825 conn(0x7f176c06bd20 msgr2=0x7f176c0b04f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:11.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.793+0000 7f1771800640 1 -- 172.21.15.96:0/1147521825 shutdown_connections 2024-05-28T06:05:11.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:11.793+0000 7f1771800640 1 -- 172.21.15.96:0/1147521825 wait complete. 2024-05-28T06:05:11.813 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-28T06:05:12.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.023+0000 7fe905600640 1 Processor -- start 2024-05-28T06:05:12.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.023+0000 7fe905600640 1 -- start start 2024-05-28T06:05:12.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.025+0000 7fe905600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe900161e70 0x7fe9000b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:12.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.025+0000 7fe905600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fe9000b3800 0x7fe9000ad2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:12.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.025+0000 7fe905600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9000b3d40 con 0x7fe900161a90 2024-05-28T06:05:12.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.025+0000 7fe905600640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fe9000b4020 con 0x7fe9000b3800 2024-05-28T06:05:12.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.025+0000 7fe905600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fe9000b4300 con 0x7fe900161e70 2024-05-28T06:05:12.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.026+0000 7fe8fe000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe900161e70 0x7fe9000b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:12.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.026+0000 7fe8ff400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fe9000b3800 0x7fe9000ad2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:12.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.026+0000 7fe8fe000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe900161e70 0x7fe9000b32c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.96:58532/0 (socket says 172.21.15.96:58532) 2024-05-28T06:05:12.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.026+0000 7fe8fe000640 1 -- 172.21.15.96:0/2279569705 learned_addr learned my addr 172.21.15.96:0/2279569705 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:12.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.027+0000 7fe8ff400640 1 -- 172.21.15.96:0/2279569705 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe900161e70 msgr2=0x7fe9000b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:12.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.027+0000 7fe8ff400640 1 --2- 172.21.15.96:0/2279569705 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe900161e70 0x7fe9000b32c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:12.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.027+0000 7fe8ff400640 1 -- 172.21.15.96:0/2279569705 >> v1:172.21.15.123:6789/0 conn(0x7fe900161a90 legacy=0x7fe90006e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:12.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.027+0000 7fe8ff400640 1 -- 172.21.15.96:0/2279569705 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe90006dc70 con 0x7fe9000b3800 2024-05-28T06:05:12.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.028+0000 7fe8ff400640 1 --2- 172.21.15.96:0/2279569705 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9000b3800 0x7fe9000ad2a0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fe8f80097a0 tx=0x7fe8f802a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d24bea81a2f63b2 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:12.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.028+0000 7fe8fd600640 1 -- 172.21.15.96:0/2279569705 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8f8037070 con 0x7fe9000b3800 2024-05-28T06:05:12.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.028+0000 7fe8fd600640 1 -- 172.21.15.96:0/2279569705 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8f8032440 con 0x7fe9000b3800 2024-05-28T06:05:12.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.028+0000 7fe8fd600640 1 -- 172.21.15.96:0/2279569705 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8f8032810 con 0x7fe9000b3800 2024-05-28T06:05:12.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.029+0000 7fe905600640 1 -- 172.21.15.96:0/2279569705 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9000b3800 msgr2=0x7fe9000ad2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:12.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.029+0000 7fe905600640 1 --2- 172.21.15.96:0/2279569705 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9000b3800 0x7fe9000ad2a0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fe8f80097a0 tx=0x7fe8f802a180 comp rx=0 tx=0).stop 2024-05-28T06:05:12.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.029+0000 7fe905600640 1 -- 172.21.15.96:0/2279569705 shutdown_connections 2024-05-28T06:05:12.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.029+0000 7fe905600640 1 --2- 172.21.15.96:0/2279569705 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9000b3800 0x7fe9000ad2a0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:12.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.029+0000 7fe905600640 1 --2- 172.21.15.96:0/2279569705 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe900161e70 0x7fe9000b32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:12.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.029+0000 7fe905600640 1 -- 172.21.15.96:0/2279569705 >> 172.21.15.96:0/2279569705 conn(0x7fe90006bd20 msgr2=0x7fe90006c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:12.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.029+0000 7fe905600640 1 -- 172.21.15.96:0/2279569705 shutdown_connections 2024-05-28T06:05:12.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.029+0000 7fe905600640 1 -- 172.21.15.96:0/2279569705 wait complete. 2024-05-28T06:05:12.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.031+0000 7fe905600640 1 Processor -- start 2024-05-28T06:05:12.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.031+0000 7fe905600640 1 -- start start 2024-05-28T06:05:12.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.032+0000 7fe905600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe9000b3800 0x7fe90014bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:12.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.032+0000 7fe905600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe900161a90 0x7fe90015d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:12.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.032+0000 7fe905600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe900161e70 0x7fe90015a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:12.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.032+0000 7fe905600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe90006eb80 con 0x7fe9000b3800 2024-05-28T06:05:12.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.032+0000 7fe905600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe9000b5020 con 0x7fe900161e70 2024-05-28T06:05:12.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.032+0000 7fe905600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fe9000ad7e0 con 0x7fe900161a90 2024-05-28T06:05:12.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.032+0000 7fe8fea00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe9000b3800 0x7fe90014bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:12.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.032+0000 7fe8fea00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe9000b3800 0x7fe90014bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51338/0 (socket says 172.21.15.96:51338) 2024-05-28T06:05:12.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.032+0000 7fe8fea00640 1 -- 172.21.15.96:0/312986898 learned_addr learned my addr 172.21.15.96:0/312986898 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:12.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.033+0000 7fe8fe000640 1 --2- 172.21.15.96:0/312986898 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe900161a90 0x7fe90015d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:12.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.033+0000 7fe8ff400640 1 --2- 172.21.15.96:0/312986898 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe900161e70 0x7fe90015a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:12.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.033+0000 7fe8fea00640 1 -- 172.21.15.96:0/312986898 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe900161a90 msgr2=0x7fe90015d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:12.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.033+0000 7fe8fea00640 1 --2- 172.21.15.96:0/312986898 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe900161a90 0x7fe90015d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:12.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.033+0000 7fe8fea00640 1 -- 172.21.15.96:0/312986898 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe900161e70 msgr2=0x7fe90015a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:12.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.033+0000 7fe8fea00640 1 --2- 172.21.15.96:0/312986898 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe900161e70 0x7fe90015a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:12.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.033+0000 7fe8fea00640 1 -- 172.21.15.96:0/312986898 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe90013e840 con 0x7fe9000b3800 2024-05-28T06:05:12.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.033+0000 7fe8fea00640 1 --2- 172.21.15.96:0/312986898 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe9000b3800 0x7fe90014bc90 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fe8f000c790 tx=0x7fe8f000cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:12.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.033+0000 7fe8f6a00640 1 -- 172.21.15.96:0/312986898 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8f0016020 con 0x7fe9000b3800 2024-05-28T06:05:12.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.034+0000 7fe8f6a00640 1 -- 172.21.15.96:0/312986898 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8f0006d50 con 0x7fe9000b3800 2024-05-28T06:05:12.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.034+0000 7fe8f6a00640 1 -- 172.21.15.96:0/312986898 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8f0003bb0 con 0x7fe9000b3800 2024-05-28T06:05:12.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.034+0000 7fe905600640 1 -- 172.21.15.96:0/312986898 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe90015e3c0 con 0x7fe9000b3800 2024-05-28T06:05:12.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.034+0000 7fe905600640 1 -- 172.21.15.96:0/312986898 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe90015e980 con 0x7fe9000b3800 2024-05-28T06:05:12.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.035+0000 7fe905600640 1 -- 172.21.15.96:0/312986898 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9000689e0 con 0x7fe9000b3800 2024-05-28T06:05:12.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.036+0000 7fe8f6a00640 1 -- 172.21.15.96:0/312986898 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe8f000a850 con 0x7fe9000b3800 2024-05-28T06:05:12.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.037+0000 7fe8f6a00640 1 --2- 172.21.15.96:0/312986898 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe8e006c920 0x7fe8e006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:12.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.037+0000 7fe8f6a00640 1 -- 172.21.15.96:0/312986898 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5825+0+0 (secure 0 0 0) 0x7fe8f008b960 con 0x7fe9000b3800 2024-05-28T06:05:12.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.038+0000 7fe8fe000640 1 --2- 172.21.15.96:0/312986898 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe8e006c920 0x7fe8e006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:12.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.039+0000 7fe8fe000640 1 --2- 172.21.15.96:0/312986898 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe8e006c920 0x7fe8e006ede0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fe9000b0cf0 tx=0x7fe8ec008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:12.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.041+0000 7fe8f6a00640 1 -- 172.21.15.96:0/312986898 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe8f0020020 con 0x7fe9000b3800 2024-05-28T06:05:12.252 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.251+0000 7fe905600640 1 -- 172.21.15.96:0/312986898 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fe900075000 con 0x7fe9000b3800 2024-05-28T06:05:12.253 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.252+0000 7fe8f6a00640 1 -- 172.21.15.96:0/312986898 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) ==== 104+0+11 (secure 0 0 0) 0x7fe8f0019310 con 0x7fe9000b3800 2024-05-28T06:05:12.253 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 32 2024-05-28T06:05:12.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.254+0000 7fe905600640 1 -- 172.21.15.96:0/312986898 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe8e006c920 msgr2=0x7fe8e006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:12.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.254+0000 7fe905600640 1 --2- 172.21.15.96:0/312986898 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe8e006c920 0x7fe8e006ede0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fe9000b0cf0 tx=0x7fe8ec008040 comp rx=0 tx=0).stop 2024-05-28T06:05:12.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.254+0000 7fe905600640 1 -- 172.21.15.96:0/312986898 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe9000b3800 msgr2=0x7fe90014bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:12.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.254+0000 7fe905600640 1 --2- 172.21.15.96:0/312986898 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe9000b3800 0x7fe90014bc90 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fe8f000c790 tx=0x7fe8f000cc50 comp rx=0 tx=0).stop 2024-05-28T06:05:12.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.254+0000 7fe905600640 1 -- 172.21.15.96:0/312986898 shutdown_connections 2024-05-28T06:05:12.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.254+0000 7fe905600640 1 --2- 172.21.15.96:0/312986898 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe8e006c920 0x7fe8e006ede0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:12.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.254+0000 7fe905600640 1 --2- 172.21.15.96:0/312986898 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe900161e70 0x7fe90015a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:12.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.254+0000 7fe905600640 1 --2- 172.21.15.96:0/312986898 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe900161a90 0x7fe90015d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:12.257 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.254+0000 7fe905600640 1 --2- 172.21.15.96:0/312986898 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe9000b3800 0x7fe90014bc90 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:12.257 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.254+0000 7fe905600640 1 -- 172.21.15.96:0/312986898 >> 172.21.15.96:0/312986898 conn(0x7fe90006bd20 msgr2=0x7fe9000d0610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:12.257 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.254+0000 7fe905600640 1 -- 172.21.15.96:0/312986898 shutdown_connections 2024-05-28T06:05:12.257 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.254+0000 7fe905600640 1 -- 172.21.15.96:0/312986898 wait complete. 2024-05-28T06:05:12.276 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-28T06:05:12.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.486+0000 7f8818200640 1 Processor -- start 2024-05-28T06:05:12.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.486+0000 7f8818200640 1 -- start start 2024-05-28T06:05:12.489 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.488+0000 7f8818200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f88100d0640 0x7f881015d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:12.489 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.488+0000 7f8818200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f88100d0a20 0x7f88100cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:12.489 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.488+0000 7f8818200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f881015cb10 con 0x7f88100b32c0 2024-05-28T06:05:12.489 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.488+0000 7f8818200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f88100adbb0 con 0x7f88100d0a20 2024-05-28T06:05:12.489 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.488+0000 7f8818200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f881015db10 con 0x7f88100d0640 2024-05-28T06:05:12.490 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.488+0000 7f8814e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f88100d0a20 0x7f88100cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:12.490 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.488+0000 7f8814e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f88100d0a20 0x7f88100cf950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51340/0 (socket says 172.21.15.96:51340) 2024-05-28T06:05:12.490 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.488+0000 7f8814e00640 1 -- 172.21.15.96:0/3673141621 learned_addr learned my addr 172.21.15.96:0/3673141621 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:12.490 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.488+0000 7f8815800640 1 --2- 172.21.15.96:0/3673141621 >> v2:172.21.15.123:3300/0 conn(0x7f88100d0640 0x7f881015d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:12.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.488+0000 7f8814e00640 1 -- 172.21.15.96:0/3673141621 >> v2:172.21.15.123:3300/0 conn(0x7f88100d0640 msgr2=0x7f881015d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:12.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.488+0000 7f8814e00640 1 --2- 172.21.15.96:0/3673141621 >> v2:172.21.15.123:3300/0 conn(0x7f88100d0640 0x7f881015d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:12.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.488+0000 7f8814e00640 1 -- 172.21.15.96:0/3673141621 >> v1:172.21.15.123:6789/0 conn(0x7f88100b32c0 legacy=0x7f88100b3730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:12.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.489+0000 7f8814e00640 1 -- 172.21.15.96:0/3673141621 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88100ad5f0 con 0x7f88100d0a20 2024-05-28T06:05:12.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.489+0000 7f8815800640 1 --2- 172.21.15.96:0/3673141621 >> v2:172.21.15.123:3300/0 conn(0x7f88100d0640 0x7f881015d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:05:12.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.489+0000 7f8814e00640 1 --2- 172.21.15.96:0/3673141621 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f88100d0a20 0x7f88100cf950 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f8810074150 tx=0x7f87fc02a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1144774c86d6f4fa server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:12.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.490+0000 7f880fe00640 1 -- 172.21.15.96:0/3673141621 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87fc037070 con 0x7f88100d0a20 2024-05-28T06:05:12.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.490+0000 7f880fe00640 1 -- 172.21.15.96:0/3673141621 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87fc032440 con 0x7f88100d0a20 2024-05-28T06:05:12.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.490+0000 7f880fe00640 1 -- 172.21.15.96:0/3673141621 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87fc032810 con 0x7f88100d0a20 2024-05-28T06:05:12.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.491+0000 7f8818200640 1 -- 172.21.15.96:0/3673141621 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f88100d0a20 msgr2=0x7f88100cf950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:12.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.491+0000 7f8818200640 1 --2- 172.21.15.96:0/3673141621 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f88100d0a20 0x7f88100cf950 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f8810074150 tx=0x7f87fc02a1c0 comp rx=0 tx=0).stop 2024-05-28T06:05:12.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.491+0000 7f8818200640 1 -- 172.21.15.96:0/3673141621 shutdown_connections 2024-05-28T06:05:12.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.491+0000 7f8818200640 1 --2- 172.21.15.96:0/3673141621 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f88100d0a20 0x7f88100cf950 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:12.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.491+0000 7f8818200640 1 --2- 172.21.15.96:0/3673141621 >> v2:172.21.15.123:3300/0 conn(0x7f88100d0640 0x7f881015d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:12.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.491+0000 7f8818200640 1 -- 172.21.15.96:0/3673141621 >> 172.21.15.96:0/3673141621 conn(0x7f881006bd20 msgr2=0x7f881006c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:12.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.491+0000 7f8818200640 1 -- 172.21.15.96:0/3673141621 shutdown_connections 2024-05-28T06:05:12.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.491+0000 7f8818200640 1 -- 172.21.15.96:0/3673141621 wait complete. 2024-05-28T06:05:12.494 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.493+0000 7f8818200640 1 Processor -- start 2024-05-28T06:05:12.494 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.493+0000 7f8818200640 1 -- start start 2024-05-28T06:05:12.495 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.494+0000 7f8818200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f88100b32c0 0x7f881006f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:12.495 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.494+0000 7f8818200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f88100d0640 0x7f8810148670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:12.496 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.494+0000 7f8818200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f88100d0a20 0x7f881006dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:12.496 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.494+0000 7f8818200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f881015d0b0 con 0x7f88100d0640 2024-05-28T06:05:12.496 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.494+0000 7f8818200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f881015cdd0 con 0x7f88100d0a20 2024-05-28T06:05:12.496 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.494+0000 7f8818200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f88100b3e40 con 0x7f88100b32c0 2024-05-28T06:05:12.496 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.494+0000 7f8814e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f88100d0640 0x7f8810148670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:12.496 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.494+0000 7f8814e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f88100d0640 0x7f8810148670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51348/0 (socket says 172.21.15.96:51348) 2024-05-28T06:05:12.496 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.494+0000 7f8814e00640 1 -- 172.21.15.96:0/1437980391 learned_addr learned my addr 172.21.15.96:0/1437980391 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:12.496 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.494+0000 7f8815800640 1 --2- 172.21.15.96:0/1437980391 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f88100b32c0 0x7f881006f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:12.496 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.494+0000 7f8814e00640 1 -- 172.21.15.96:0/1437980391 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f88100b32c0 msgr2=0x7f881006f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:12.496 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.494+0000 7f8814e00640 1 --2- 172.21.15.96:0/1437980391 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f88100b32c0 0x7f881006f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:12.496 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.494+0000 7f8814e00640 1 -- 172.21.15.96:0/1437980391 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f88100d0a20 msgr2=0x7f881006dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:12.496 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.494+0000 7f8814e00640 1 --2- 172.21.15.96:0/1437980391 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f88100d0a20 0x7f881006dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:12.496 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.494+0000 7f8814e00640 1 -- 172.21.15.96:0/1437980391 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f881006f9a0 con 0x7f88100d0640 2024-05-28T06:05:12.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.495+0000 7f8814e00640 1 --2- 172.21.15.96:0/1437980391 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f88100d0640 0x7f8810148670 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f87fc0097f0 tx=0x7f87fc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:12.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.495+0000 7f880d600640 1 -- 172.21.15.96:0/1437980391 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87fc037040 con 0x7f88100d0640 2024-05-28T06:05:12.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.495+0000 7f880d600640 1 -- 172.21.15.96:0/1437980391 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87fc004230 con 0x7f88100d0640 2024-05-28T06:05:12.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.495+0000 7f880d600640 1 -- 172.21.15.96:0/1437980391 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87fc030620 con 0x7f88100d0640 2024-05-28T06:05:12.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.495+0000 7f8818200640 1 -- 172.21.15.96:0/1437980391 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f881006e3c0 con 0x7f88100d0640 2024-05-28T06:05:12.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.495+0000 7f8818200640 1 -- 172.21.15.96:0/1437980391 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f881006ec60 con 0x7f88100d0640 2024-05-28T06:05:12.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.496+0000 7f8818200640 1 -- 172.21.15.96:0/1437980391 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88100689e0 con 0x7f88100d0640 2024-05-28T06:05:12.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.497+0000 7f880d600640 1 -- 172.21.15.96:0/1437980391 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f87fc0307c0 con 0x7f88100d0640 2024-05-28T06:05:12.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.498+0000 7f880d600640 1 --2- 172.21.15.96:0/1437980391 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f87f006c8f0 0x7f87f006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:12.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.498+0000 7f880d600640 1 -- 172.21.15.96:0/1437980391 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5825+0+0 (secure 0 0 0) 0x7f87fc0acb20 con 0x7f88100d0640 2024-05-28T06:05:12.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.499+0000 7f8815800640 1 --2- 172.21.15.96:0/1437980391 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f87f006c8f0 0x7f87f006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:12.501 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.500+0000 7f8815800640 1 --2- 172.21.15.96:0/1437980391 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f87f006c8f0 0x7f87f006edb0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f8804008850 tx=0x7f8804008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:12.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.503+0000 7f880d600640 1 -- 172.21.15.96:0/1437980391 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f87fc07ae30 con 0x7f88100d0640 2024-05-28T06:05:12.705 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.703+0000 7f8818200640 1 -- 172.21.15.96:0/1437980391 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f8810071ef0 con 0x7f88100d0640 2024-05-28T06:05:12.705 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.704+0000 7f880d600640 1 -- 172.21.15.96:0/1437980391 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) ==== 100+0+11 (secure 0 0 0) 0x7f87fc07a730 con 0x7f88100d0640 2024-05-28T06:05:12.706 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 64 2024-05-28T06:05:12.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.707+0000 7f8818200640 1 -- 172.21.15.96:0/1437980391 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f87f006c8f0 msgr2=0x7f87f006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:12.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.707+0000 7f8818200640 1 --2- 172.21.15.96:0/1437980391 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f87f006c8f0 0x7f87f006edb0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f8804008850 tx=0x7f8804008040 comp rx=0 tx=0).stop 2024-05-28T06:05:12.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.707+0000 7f8818200640 1 -- 172.21.15.96:0/1437980391 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f88100d0640 msgr2=0x7f8810148670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:12.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.707+0000 7f8818200640 1 --2- 172.21.15.96:0/1437980391 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f88100d0640 0x7f8810148670 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f87fc0097f0 tx=0x7f87fc0046c0 comp rx=0 tx=0).stop 2024-05-28T06:05:12.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.707+0000 7f8818200640 1 -- 172.21.15.96:0/1437980391 shutdown_connections 2024-05-28T06:05:12.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.707+0000 7f8818200640 1 --2- 172.21.15.96:0/1437980391 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f87f006c8f0 0x7f87f006edb0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:12.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.707+0000 7f8818200640 1 --2- 172.21.15.96:0/1437980391 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f88100d0a20 0x7f881006dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:12.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.707+0000 7f8818200640 1 --2- 172.21.15.96:0/1437980391 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f88100d0640 0x7f8810148670 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:12.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.707+0000 7f8818200640 1 --2- 172.21.15.96:0/1437980391 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f88100b32c0 0x7f881006f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:12.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.707+0000 7f8818200640 1 -- 172.21.15.96:0/1437980391 >> 172.21.15.96:0/1437980391 conn(0x7f881006bd20 msgr2=0x7f881015f240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:12.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.707+0000 7f8818200640 1 -- 172.21.15.96:0/1437980391 shutdown_connections 2024-05-28T06:05:12.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.707+0000 7f8818200640 1 -- 172.21.15.96:0/1437980391 wait complete. 2024-05-28T06:05:12.727 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-28T06:05:12.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.923+0000 7f374dc00640 1 Processor -- start 2024-05-28T06:05:12.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.923+0000 7f374dc00640 1 -- start start 2024-05-28T06:05:12.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.925+0000 7f374dc00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3748158b80 0x7f374806dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:12.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.925+0000 7f374dc00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f374806e6e0 con 0x7f37480722d0 2024-05-28T06:05:12.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.925+0000 7f374dc00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f374806e9c0 con 0x7f374806e160 2024-05-28T06:05:12.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.925+0000 7f374dc00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f374806eca0 con 0x7f3748158b80 2024-05-28T06:05:12.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.925+0000 7f374d200640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f37480722d0 0x7f37480b32c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:34838/0 (socket says 172.21.15.96:34838) 2024-05-28T06:05:12.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.925+0000 7f374d200640 1 -- 172.21.15.96:0/2959138652 learned_addr learned my addr 172.21.15.96:0/2959138652 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:05:12.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.925+0000 7f3747e00640 1 --2- 172.21.15.96:0/2959138652 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3748158b80 0x7f374806dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:12.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.926+0000 7f3747400640 1 -- 172.21.15.96:0/2959138652 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2544436038 0 0) 0x7f374806e6e0 con 0x7f37480722d0 2024-05-28T06:05:12.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.926+0000 7f3747400640 1 -- 172.21.15.96:0/2959138652 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f372c003520 con 0x7f37480722d0 2024-05-28T06:05:12.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.926+0000 7f3747400640 1 -- 172.21.15.96:0/2959138652 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2103594550 0 0) 0x7f372c003520 con 0x7f37480722d0 2024-05-28T06:05:12.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.926+0000 7f3747400640 1 -- 172.21.15.96:0/2959138652 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3748158b80 msgr2=0x7f374806dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:12.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.926+0000 7f3747400640 1 --2- 172.21.15.96:0/2959138652 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3748158b80 0x7f374806dc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:12.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.926+0000 7f3747400640 1 -- 172.21.15.96:0/2959138652 >> v1:172.21.15.123:6789/0 conn(0x7f374806e160 legacy=0x7f3748071bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:12.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.926+0000 7f3747400640 1 -- 172.21.15.96:0/2959138652 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f374806ef80 con 0x7f37480722d0 2024-05-28T06:05:12.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.926+0000 7f3747400640 1 -- 172.21.15.96:0/2959138652 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f37400019f0 con 0x7f37480722d0 2024-05-28T06:05:12.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.926+0000 7f3747400640 1 -- 172.21.15.96:0/2959138652 >> v1:172.21.15.96:6789/0 conn(0x7f37480722d0 legacy=0x7f37480b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:12.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.926+0000 7f3747400640 1 --2- 172.21.15.96:0/2959138652 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f372c0039e0 0x7f372c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:12.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.926+0000 7f3747400640 1 --2- 172.21.15.96:0/2959138652 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3748158b80 0x7f372c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:12.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.926+0000 7f3747400640 1 --2- 172.21.15.96:0/2959138652 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f372c004d40 0x7f372c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:12.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.926+0000 7f3747400640 1 -- 172.21.15.96:0/2959138652 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f374806eca0 con 0x7f372c0039e0 2024-05-28T06:05:12.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.926+0000 7f3747400640 1 -- 172.21.15.96:0/2959138652 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f372c00ba70 con 0x7f372c004d40 2024-05-28T06:05:12.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.926+0000 7f3747400640 1 -- 172.21.15.96:0/2959138652 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f372c00bc70 con 0x7f3748158b80 2024-05-28T06:05:12.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.927+0000 7f3747e00640 1 --2- 172.21.15.96:0/2959138652 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f372c0039e0 0x7f372c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:12.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.927+0000 7f373f400640 1 --2- 172.21.15.96:0/2959138652 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3748158b80 0x7f372c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:12.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.927+0000 7f374d200640 1 --2- 172.21.15.96:0/2959138652 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f372c004d40 0x7f372c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:12.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.927+0000 7f3747e00640 1 -- 172.21.15.96:0/2959138652 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3748158b80 msgr2=0x7f372c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:12.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.927+0000 7f3747e00640 1 --2- 172.21.15.96:0/2959138652 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3748158b80 0x7f372c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:12.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.927+0000 7f3747e00640 1 -- 172.21.15.96:0/2959138652 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f372c004d40 msgr2=0x7f372c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:12.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.927+0000 7f3747e00640 1 --2- 172.21.15.96:0/2959138652 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f372c004d40 0x7f372c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:12.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.927+0000 7f3747e00640 1 -- 172.21.15.96:0/2959138652 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f372c003520 con 0x7f372c0039e0 2024-05-28T06:05:12.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.927+0000 7f373f400640 1 --2- 172.21.15.96:0/2959138652 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3748158b80 0x7f372c004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-28T06:05:12.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.928+0000 7f3747e00640 1 --2- 172.21.15.96:0/2959138652 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f372c0039e0 0x7f372c003e00 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f3738009a90 tx=0x7f372c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31ea23673dcce8eb server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:12.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.929+0000 7f3747400640 1 -- 172.21.15.96:0/2959138652 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3738036070 con 0x7f372c0039e0 2024-05-28T06:05:12.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.929+0000 7f3747400640 1 -- 172.21.15.96:0/2959138652 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3738031400 con 0x7f372c0039e0 2024-05-28T06:05:12.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.929+0000 7f3747400640 1 -- 172.21.15.96:0/2959138652 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3738009690 con 0x7f372c0039e0 2024-05-28T06:05:12.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.929+0000 7f374dc00640 1 -- 172.21.15.96:0/2959138652 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f372c0039e0 msgr2=0x7f372c003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:12.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.929+0000 7f374dc00640 1 --2- 172.21.15.96:0/2959138652 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f372c0039e0 0x7f372c003e00 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f3738009a90 tx=0x7f372c005830 comp rx=0 tx=0).stop 2024-05-28T06:05:12.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.929+0000 7f374d200640 1 -- 172.21.15.96:0/2959138652 reap_dead start 2024-05-28T06:05:12.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.929+0000 7f374dc00640 1 -- 172.21.15.96:0/2959138652 shutdown_connections 2024-05-28T06:05:12.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.929+0000 7f374dc00640 1 -- 172.21.15.96:0/2959138652 >> 172.21.15.96:0/2959138652 conn(0x7f37480b1fc0 msgr2=0x7f37480b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:12.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.929+0000 7f374dc00640 1 -- 172.21.15.96:0/2959138652 shutdown_connections 2024-05-28T06:05:12.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.929+0000 7f374dc00640 1 -- 172.21.15.96:0/2959138652 wait complete. 2024-05-28T06:05:12.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.931+0000 7f374dc00640 1 Processor -- start 2024-05-28T06:05:12.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.931+0000 7f374dc00640 1 -- start start 2024-05-28T06:05:12.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.932+0000 7f374dc00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f372c0039e0 0x7f37480b0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:12.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.932+0000 7f374dc00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3748156770 0x7f3748156bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:12.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.932+0000 7f374dc00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3748151ac0 0x7f3748151f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:12.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.932+0000 7f374dc00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f374806f780 con 0x7f372c0039e0 2024-05-28T06:05:12.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.932+0000 7f374dc00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f37480b2d30 con 0x7f3748151ac0 2024-05-28T06:05:12.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.932+0000 7f374dc00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f37481f8100 con 0x7f3748156770 2024-05-28T06:05:12.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.932+0000 7f3747e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f372c0039e0 0x7f37480b0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:12.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.933+0000 7f3747e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f372c0039e0 0x7f37480b0c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51370/0 (socket says 172.21.15.96:51370) 2024-05-28T06:05:12.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.933+0000 7f3747e00640 1 -- 172.21.15.96:0/2261355400 learned_addr learned my addr 172.21.15.96:0/2261355400 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:12.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.933+0000 7f373f400640 1 --2- 172.21.15.96:0/2261355400 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3748156770 0x7f3748156bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:12.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.933+0000 7f374d200640 1 --2- 172.21.15.96:0/2261355400 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3748151ac0 0x7f3748151f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:12.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.933+0000 7f3747e00640 1 -- 172.21.15.96:0/2261355400 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3748156770 msgr2=0x7f3748156bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:12.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.933+0000 7f3747e00640 1 --2- 172.21.15.96:0/2261355400 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3748156770 0x7f3748156bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:12.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.933+0000 7f3747e00640 1 -- 172.21.15.96:0/2261355400 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3748151ac0 msgr2=0x7f3748151f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:12.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.933+0000 7f3747e00640 1 --2- 172.21.15.96:0/2261355400 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3748151ac0 0x7f3748151f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:12.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.933+0000 7f3747e00640 1 -- 172.21.15.96:0/2261355400 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37481f77d0 con 0x7f372c0039e0 2024-05-28T06:05:12.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.933+0000 7f3747e00640 1 --2- 172.21.15.96:0/2261355400 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f372c0039e0 0x7f37480b0c50 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f3738034040 tx=0x7f3738004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:12.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.934+0000 7f3744c00640 1 -- 172.21.15.96:0/2261355400 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3738040020 con 0x7f372c0039e0 2024-05-28T06:05:12.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.934+0000 7f3744c00640 1 -- 172.21.15.96:0/2261355400 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3738003940 con 0x7f372c0039e0 2024-05-28T06:05:12.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.934+0000 7f3744c00640 1 -- 172.21.15.96:0/2261355400 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3738036040 con 0x7f372c0039e0 2024-05-28T06:05:12.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.934+0000 7f374dc00640 1 -- 172.21.15.96:0/2261355400 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37481f7ab0 con 0x7f372c0039e0 2024-05-28T06:05:12.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.934+0000 7f374dc00640 1 -- 172.21.15.96:0/2261355400 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37481f8590 con 0x7f372c0039e0 2024-05-28T06:05:12.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.935+0000 7f374dc00640 1 -- 172.21.15.96:0/2261355400 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37480689e0 con 0x7f372c0039e0 2024-05-28T06:05:12.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.936+0000 7f3744c00640 1 -- 172.21.15.96:0/2261355400 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3738043020 con 0x7f372c0039e0 2024-05-28T06:05:12.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.937+0000 7f3744c00640 1 --2- 172.21.15.96:0/2261355400 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f372406c8f0 0x7f372406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:12.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.937+0000 7f3744c00640 1 -- 172.21.15.96:0/2261355400 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5825+0+0 (secure 0 0 0) 0x7f37380abe00 con 0x7f372c0039e0 2024-05-28T06:05:12.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.937+0000 7f373f400640 1 --2- 172.21.15.96:0/2261355400 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f372406c8f0 0x7f372406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:12.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.938+0000 7f373f400640 1 --2- 172.21.15.96:0/2261355400 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f372406c8f0 0x7f372406edb0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f3730007fd0 tx=0x7f3730009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:12.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:12.941+0000 7f3744c00640 1 -- 172.21.15.96:0/2261355400 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f373804a070 con 0x7f372c0039e0 2024-05-28T06:05:13.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.156+0000 7f374dc00640 1 -- 172.21.15.96:0/2261355400 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f3748150060 con 0x7f372c0039e0 2024-05-28T06:05:13.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.157+0000 7f3744c00640 1 -- 172.21.15.96:0/2261355400 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) ==== 80+0+0 (secure 0 0 0) 0x7f373807a110 con 0x7f372c0039e0 2024-05-28T06:05:13.159 INFO:teuthology.orchestra.run.smithi096.stderr:listed 0 entries 2024-05-28T06:05:13.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.160+0000 7f374dc00640 1 -- 172.21.15.96:0/2261355400 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f372406c8f0 msgr2=0x7f372406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:13.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.160+0000 7f374dc00640 1 --2- 172.21.15.96:0/2261355400 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f372406c8f0 0x7f372406edb0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f3730007fd0 tx=0x7f3730009040 comp rx=0 tx=0).stop 2024-05-28T06:05:13.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.160+0000 7f374dc00640 1 -- 172.21.15.96:0/2261355400 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f372c0039e0 msgr2=0x7f37480b0c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:13.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.160+0000 7f374dc00640 1 --2- 172.21.15.96:0/2261355400 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f372c0039e0 0x7f37480b0c50 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f3738034040 tx=0x7f3738004730 comp rx=0 tx=0).stop 2024-05-28T06:05:13.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.160+0000 7f374dc00640 1 -- 172.21.15.96:0/2261355400 shutdown_connections 2024-05-28T06:05:13.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.160+0000 7f374dc00640 1 --2- 172.21.15.96:0/2261355400 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f372406c8f0 0x7f372406edb0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:13.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.160+0000 7f374dc00640 1 --2- 172.21.15.96:0/2261355400 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3748151ac0 0x7f3748151f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:13.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.160+0000 7f374dc00640 1 --2- 172.21.15.96:0/2261355400 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3748156770 0x7f3748156bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:13.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.160+0000 7f374dc00640 1 --2- 172.21.15.96:0/2261355400 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f372c0039e0 0x7f37480b0c50 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:13.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.160+0000 7f374dc00640 1 -- 172.21.15.96:0/2261355400 >> 172.21.15.96:0/2261355400 conn(0x7f37480b1fc0 msgr2=0x7f374815ec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:13.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.160+0000 7f374dc00640 1 -- 172.21.15.96:0/2261355400 shutdown_connections 2024-05-28T06:05:13.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.160+0000 7f374dc00640 1 -- 172.21.15.96:0/2261355400 wait complete. 2024-05-28T06:05:13.181 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi123.front.sepia.ceph.com... 2024-05-28T06:05:13.182 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:05:13.182 DEBUG:teuthology.orchestra.run.smithi123:> sudo fusermount -u /home/ubuntu/cephtest/mnt.3 2024-05-28T06:05:13.274 INFO:teuthology.orchestra.run:waiting for 300 2024-05-28T06:05:13.274 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi123.front.sepia.ceph.com 2024-05-28T06:05:13.275 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:05:13.275 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-05-28T06:05:13.324 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-05-28T06:05:13.324 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-28T06:05:13.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.532+0000 7f00a0a00640 1 Processor -- start 2024-05-28T06:05:13.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.533+0000 7f00a0a00640 1 -- start start 2024-05-28T06:05:13.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.534+0000 7f00a0a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f009c159090 0x7f009c0aea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:13.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.534+0000 7f00a0a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f009c0b4240 0x7f009c0b46f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:13.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.534+0000 7f00a0a00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f009c06efa0 con 0x7f009c0b4dc0 2024-05-28T06:05:13.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.534+0000 7f00a0a00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f009c06f280 con 0x7f009c159090 2024-05-28T06:05:13.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.534+0000 7f00a0a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f009c06f560 con 0x7f009c0b4240 2024-05-28T06:05:13.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.534+0000 7f0099e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f009c159090 0x7f009c0aea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:13.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.534+0000 7f0099e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f009c159090 0x7f009c0aea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51372/0 (socket says 172.21.15.96:51372) 2024-05-28T06:05:13.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.534+0000 7f0099e00640 1 -- 172.21.15.96:0/1246546159 learned_addr learned my addr 172.21.15.96:0/1246546159 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:13.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.535+0000 7f0099e00640 1 -- 172.21.15.96:0/1246546159 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f009c0b4240 msgr2=0x7f009c0b46f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:13.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.535+0000 7f0099400640 1 --2- 172.21.15.96:0/1246546159 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f009c0b4240 0x7f009c0b46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:13.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.535+0000 7f0099e00640 1 --2- 172.21.15.96:0/1246546159 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f009c0b4240 0x7f009c0b46f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:13.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.535+0000 7f0099e00640 1 -- 172.21.15.96:0/1246546159 >> v1:172.21.15.123:6789/0 conn(0x7f009c0b4dc0 legacy=0x7f009c070040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:13.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.535+0000 7f0099e00640 1 -- 172.21.15.96:0/1246546159 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f009c06f840 con 0x7f009c159090 2024-05-28T06:05:13.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.536+0000 7f0099e00640 1 --2- 172.21.15.96:0/1246546159 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f009c159090 0x7f009c0aea30 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f0084009870 tx=0x7f008402a200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d67e679eaf9cf8d server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:13.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.536+0000 7f0098a00640 1 -- 172.21.15.96:0/1246546159 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0084037070 con 0x7f009c159090 2024-05-28T06:05:13.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.536+0000 7f0098a00640 1 -- 172.21.15.96:0/1246546159 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0084032440 con 0x7f009c159090 2024-05-28T06:05:13.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.536+0000 7f0098a00640 1 -- 172.21.15.96:0/1246546159 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0084032810 con 0x7f009c159090 2024-05-28T06:05:13.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.537+0000 7f00a0a00640 1 -- 172.21.15.96:0/1246546159 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f009c159090 msgr2=0x7f009c0aea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:13.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.537+0000 7f00a0a00640 1 --2- 172.21.15.96:0/1246546159 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f009c159090 0x7f009c0aea30 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f0084009870 tx=0x7f008402a200 comp rx=0 tx=0).stop 2024-05-28T06:05:13.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.537+0000 7f00a0a00640 1 -- 172.21.15.96:0/1246546159 shutdown_connections 2024-05-28T06:05:13.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.537+0000 7f00a0a00640 1 --2- 172.21.15.96:0/1246546159 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f009c0b4240 0x7f009c0b46f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:13.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.537+0000 7f00a0a00640 1 --2- 172.21.15.96:0/1246546159 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f009c159090 0x7f009c0aea30 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:13.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.537+0000 7f00a0a00640 1 -- 172.21.15.96:0/1246546159 >> 172.21.15.96:0/1246546159 conn(0x7f009c0b6760 msgr2=0x7f009c0b6b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:13.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.537+0000 7f00a0a00640 1 -- 172.21.15.96:0/1246546159 shutdown_connections 2024-05-28T06:05:13.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.537+0000 7f00a0a00640 1 -- 172.21.15.96:0/1246546159 wait complete. 2024-05-28T06:05:13.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.539+0000 7f00a0a00640 1 Processor -- start 2024-05-28T06:05:13.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.539+0000 7f00a0a00640 1 -- start start 2024-05-28T06:05:13.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.540+0000 7f00a0a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f009c0b4240 0x7f009c153af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:13.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.540+0000 7f00a0a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f009c0b4dc0 0x7f009c154030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:13.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.540+0000 7f00a0a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f009c159090 0x7f009c14a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:13.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.540+0000 7f00a0a00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f009c0aef70 con 0x7f009c159090 2024-05-28T06:05:13.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.540+0000 7f00a0a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f009c159470 con 0x7f009c0b4dc0 2024-05-28T06:05:13.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.540+0000 7f00a0a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f009c070750 con 0x7f009c0b4240 2024-05-28T06:05:13.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.540+0000 7f009a800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f009c159090 0x7f009c14a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:13.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.540+0000 7f009a800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f009c159090 0x7f009c14a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51374/0 (socket says 172.21.15.96:51374) 2024-05-28T06:05:13.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.540+0000 7f009a800640 1 -- 172.21.15.96:0/516635650 learned_addr learned my addr 172.21.15.96:0/516635650 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:13.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.541+0000 7f009a800640 1 -- 172.21.15.96:0/516635650 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f009c0b4240 msgr2=0x7f009c153af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:13.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.541+0000 7f0099e00640 1 --2- 172.21.15.96:0/516635650 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f009c0b4240 0x7f009c153af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:13.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.541+0000 7f0099400640 1 --2- 172.21.15.96:0/516635650 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f009c0b4dc0 0x7f009c154030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:13.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.541+0000 7f009a800640 1 --2- 172.21.15.96:0/516635650 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f009c0b4240 0x7f009c153af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:13.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.541+0000 7f009a800640 1 -- 172.21.15.96:0/516635650 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f009c0b4dc0 msgr2=0x7f009c154030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:13.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.541+0000 7f009a800640 1 --2- 172.21.15.96:0/516635650 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f009c0b4dc0 0x7f009c154030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:13.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.541+0000 7f009a800640 1 -- 172.21.15.96:0/516635650 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f009c0af660 con 0x7f009c159090 2024-05-28T06:05:13.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.541+0000 7f009a800640 1 --2- 172.21.15.96:0/516635650 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f009c159090 0x7f009c14a300 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f009400c410 tx=0x7f009400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:13.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.541+0000 7f0092000640 1 -- 172.21.15.96:0/516635650 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0094011790 con 0x7f009c159090 2024-05-28T06:05:13.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.541+0000 7f0092000640 1 -- 172.21.15.96:0/516635650 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0094017070 con 0x7f009c159090 2024-05-28T06:05:13.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.541+0000 7f0092000640 1 -- 172.21.15.96:0/516635650 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0094011bb0 con 0x7f009c159090 2024-05-28T06:05:13.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.541+0000 7f00a0a00640 1 -- 172.21.15.96:0/516635650 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f009c0af940 con 0x7f009c159090 2024-05-28T06:05:13.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.541+0000 7f00a0a00640 1 -- 172.21.15.96:0/516635650 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f009c0aff00 con 0x7f009c159090 2024-05-28T06:05:13.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.543+0000 7f00a0a00640 1 -- 172.21.15.96:0/516635650 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f009c0690f0 con 0x7f009c159090 2024-05-28T06:05:13.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.544+0000 7f0092000640 1 -- 172.21.15.96:0/516635650 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f009401c460 con 0x7f009c159090 2024-05-28T06:05:13.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.544+0000 7f0092000640 1 --2- 172.21.15.96:0/516635650 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f007806c920 0x7f007806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:13.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.545+0000 7f0092000640 1 -- 172.21.15.96:0/516635650 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5825+0+0 (secure 0 0 0) 0x7f009408b800 con 0x7f009c159090 2024-05-28T06:05:13.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.545+0000 7f0099e00640 1 --2- 172.21.15.96:0/516635650 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f007806c920 0x7f007806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:13.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.546+0000 7f0099e00640 1 --2- 172.21.15.96:0/516635650 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f007806c920 0x7f007806ede0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f0084034000 tx=0x7f008402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:13.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.549+0000 7f0092000640 1 -- 172.21.15.96:0/516635650 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0094021020 con 0x7f009c159090 2024-05-28T06:05:13.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.789+0000 7f00a0a00640 1 -- 172.21.15.96:0/516635650 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f009c1552c0 con 0x7f009c159090 2024-05-28T06:05:13.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.789+0000 7f0092000640 1 -- 172.21.15.96:0/516635650 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5178 (secure 0 0 0) 0x7f0094059b10 con 0x7f009c159090 2024-05-28T06:05:13.791 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:05:13.792 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/3349661842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":3349661842},{"type":"v1","addr":"172.21.15.96:6833","nonce":3349661842}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4270,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/3383264102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":3383264102},{"type":"v1","addr":"172.21.15.96:6835","nonce":3383264102}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4279,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/3068402544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":3068402544},{"type":"v1","addr":"172.21.15.123:6837","nonce":3068402544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-28T06:04:10.319738+0000","modified":"2024-05-28T06:04:13.704080+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4281},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4281":{"gid":4281,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.123:6835/605814935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":605814935},{"type":"v1","addr":"172.21.15.123:6835","nonce":605814935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4281,"qdb_cluster":[4281]},"id":1}]} 2024-05-28T06:05:13.792 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 7 2024-05-28T06:05:13.794 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.792+0000 7f00a0a00640 1 -- 172.21.15.96:0/516635650 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f007806c920 msgr2=0x7f007806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:13.794 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.792+0000 7f00a0a00640 1 --2- 172.21.15.96:0/516635650 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f007806c920 0x7f007806ede0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f0084034000 tx=0x7f008402e000 comp rx=0 tx=0).stop 2024-05-28T06:05:13.794 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.792+0000 7f00a0a00640 1 -- 172.21.15.96:0/516635650 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f009c159090 msgr2=0x7f009c14a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:13.794 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.792+0000 7f00a0a00640 1 --2- 172.21.15.96:0/516635650 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f009c159090 0x7f009c14a300 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f009400c410 tx=0x7f009400c8d0 comp rx=0 tx=0).stop 2024-05-28T06:05:13.794 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.792+0000 7f00a0a00640 1 -- 172.21.15.96:0/516635650 shutdown_connections 2024-05-28T06:05:13.794 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.792+0000 7f00a0a00640 1 --2- 172.21.15.96:0/516635650 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f007806c920 0x7f007806ede0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:13.794 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.792+0000 7f00a0a00640 1 --2- 172.21.15.96:0/516635650 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f009c159090 0x7f009c14a300 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:13.794 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.792+0000 7f00a0a00640 1 --2- 172.21.15.96:0/516635650 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f009c0b4dc0 0x7f009c154030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:13.794 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.792+0000 7f00a0a00640 1 --2- 172.21.15.96:0/516635650 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f009c0b4240 0x7f009c153af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:13.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.792+0000 7f00a0a00640 1 -- 172.21.15.96:0/516635650 >> 172.21.15.96:0/516635650 conn(0x7f009c0b6760 msgr2=0x7f009c09e590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:13.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.792+0000 7f00a0a00640 1 -- 172.21.15.96:0/516635650 shutdown_connections 2024-05-28T06:05:13.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:13.792+0000 7f00a0a00640 1 -- 172.21.15.96:0/516635650 wait complete. 2024-05-28T06:05:13.811 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:05:14.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.008+0000 7f697f600640 1 Processor -- start 2024-05-28T06:05:14.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.009+0000 7f697f600640 1 -- start start 2024-05-28T06:05:14.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.010+0000 7f697f600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f69780d1480 0x7f6978161b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:14.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.010+0000 7f697f600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f6978162040 0x7f69780cf550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:14.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.010+0000 7f697f600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69780b3d30 con 0x7f69780d10a0 2024-05-28T06:05:14.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.010+0000 7f697f600640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f69780d2930 con 0x7f6978162040 2024-05-28T06:05:14.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.010+0000 7f697f600640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f69780d2c10 con 0x7f69780d1480 2024-05-28T06:05:14.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.011+0000 7f697d600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f6978162040 0x7f69780cf550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:14.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.011+0000 7f697d600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f6978162040 0x7f69780cf550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51382/0 (socket says 172.21.15.96:51382) 2024-05-28T06:05:14.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.011+0000 7f697d600640 1 -- 172.21.15.96:0/3302992067 learned_addr learned my addr 172.21.15.96:0/3302992067 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:14.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.011+0000 7f6977e00640 1 --2- 172.21.15.96:0/3302992067 >> v2:172.21.15.123:3300/0 conn(0x7f69780d1480 0x7f6978161b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:14.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.011+0000 7f697d600640 1 -- 172.21.15.96:0/3302992067 >> v2:172.21.15.123:3300/0 conn(0x7f69780d1480 msgr2=0x7f6978161b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:14.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.011+0000 7f697d600640 1 --2- 172.21.15.96:0/3302992067 >> v2:172.21.15.123:3300/0 conn(0x7f69780d1480 0x7f6978161b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:14.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.011+0000 7f697d600640 1 -- 172.21.15.96:0/3302992067 >> v1:172.21.15.123:6789/0 conn(0x7f69780d10a0 legacy=0x7f69780b3620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:14.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.011+0000 7f697d600640 1 -- 172.21.15.96:0/3302992067 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69780d2ef0 con 0x7f6978162040 2024-05-28T06:05:14.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.012+0000 7f697d600640 1 --2- 172.21.15.96:0/3302992067 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6978162040 0x7f69780cf550 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f69780d4640 tx=0x7f697002a290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7271b2284309f6d8 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:14.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.012+0000 7f6977400640 1 -- 172.21.15.96:0/3302992067 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6970037070 con 0x7f6978162040 2024-05-28T06:05:14.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.012+0000 7f6977400640 1 -- 172.21.15.96:0/3302992067 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6970032440 con 0x7f6978162040 2024-05-28T06:05:14.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.012+0000 7f6977400640 1 -- 172.21.15.96:0/3302992067 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6970032810 con 0x7f6978162040 2024-05-28T06:05:14.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.013+0000 7f697f600640 1 -- 172.21.15.96:0/3302992067 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6978162040 msgr2=0x7f69780cf550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:14.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.013+0000 7f697f600640 1 --2- 172.21.15.96:0/3302992067 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6978162040 0x7f69780cf550 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f69780d4640 tx=0x7f697002a290 comp rx=0 tx=0).stop 2024-05-28T06:05:14.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.013+0000 7f697f600640 1 -- 172.21.15.96:0/3302992067 shutdown_connections 2024-05-28T06:05:14.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.013+0000 7f697f600640 1 --2- 172.21.15.96:0/3302992067 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6978162040 0x7f69780cf550 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:14.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.013+0000 7f697f600640 1 --2- 172.21.15.96:0/3302992067 >> v2:172.21.15.123:3300/0 conn(0x7f69780d1480 0x7f6978161b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:14.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.013+0000 7f697f600640 1 -- 172.21.15.96:0/3302992067 >> 172.21.15.96:0/3302992067 conn(0x7f697808bfc0 msgr2=0x7f69780d4670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:14.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.013+0000 7f697f600640 1 -- 172.21.15.96:0/3302992067 shutdown_connections 2024-05-28T06:05:14.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.013+0000 7f697f600640 1 -- 172.21.15.96:0/3302992067 wait complete. 2024-05-28T06:05:14.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.015+0000 7f697f600640 1 Processor -- start 2024-05-28T06:05:14.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.015+0000 7f697f600640 1 -- start start 2024-05-28T06:05:14.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.016+0000 7f697f600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69780d10a0 0x7f697815d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:14.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.016+0000 7f697f600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f69780d1480 0x7f69780c9620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:14.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.016+0000 7f697f600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6978162040 0x7f69780b96d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:14.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.016+0000 7f697f600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f6978162450 con 0x7f69780d1480 2024-05-28T06:05:14.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.016+0000 7f697f600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f69780d34b0 con 0x7f69780d10a0 2024-05-28T06:05:14.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.016+0000 7f697f600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f69780cb9e0 con 0x7f6978162040 2024-05-28T06:05:14.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.016+0000 7f6977e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f69780d1480 0x7f69780c9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:14.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.016+0000 7f6977e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f69780d1480 0x7f69780c9620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51386/0 (socket says 172.21.15.96:51386) 2024-05-28T06:05:14.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.016+0000 7f6977e00640 1 -- 172.21.15.96:0/3562877126 learned_addr learned my addr 172.21.15.96:0/3562877126 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:14.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.017+0000 7f697cc00640 1 --2- 172.21.15.96:0/3562877126 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69780d10a0 0x7f697815d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:14.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.017+0000 7f697d600640 1 --2- 172.21.15.96:0/3562877126 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6978162040 0x7f69780b96d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:14.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.017+0000 7f6977e00640 1 -- 172.21.15.96:0/3562877126 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6978162040 msgr2=0x7f69780b96d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:14.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.017+0000 7f6977e00640 1 --2- 172.21.15.96:0/3562877126 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6978162040 0x7f69780b96d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:14.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.017+0000 7f6977e00640 1 -- 172.21.15.96:0/3562877126 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69780d10a0 msgr2=0x7f697815d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:14.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.017+0000 7f6977e00640 1 --2- 172.21.15.96:0/3562877126 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69780d10a0 0x7f697815d5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:14.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.017+0000 7f6977e00640 1 -- 172.21.15.96:0/3562877126 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f697815db00 con 0x7f69780d1480 2024-05-28T06:05:14.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.017+0000 7f6977e00640 1 --2- 172.21.15.96:0/3562877126 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f69780d1480 0x7f69780c9620 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f6960007ef0 tx=0x7f696000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:14.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.017+0000 7f6974c00640 1 -- 172.21.15.96:0/3562877126 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6960014620 con 0x7f69780d1480 2024-05-28T06:05:14.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.017+0000 7f6974c00640 1 -- 172.21.15.96:0/3562877126 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6960019070 con 0x7f69780d1480 2024-05-28T06:05:14.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.017+0000 7f6974c00640 1 -- 172.21.15.96:0/3562877126 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6960014a00 con 0x7f69780d1480 2024-05-28T06:05:14.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.017+0000 7f697f600640 1 -- 172.21.15.96:0/3562877126 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69780cbc30 con 0x7f69780d1480 2024-05-28T06:05:14.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.017+0000 7f697f600640 1 -- 172.21.15.96:0/3562877126 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69780cc1f0 con 0x7f69780d1480 2024-05-28T06:05:14.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.019+0000 7f697f600640 1 -- 172.21.15.96:0/3562877126 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69780b1c60 con 0x7f69780d1480 2024-05-28T06:05:14.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.019+0000 7f6974c00640 1 -- 172.21.15.96:0/3562877126 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6960014ba0 con 0x7f69780d1480 2024-05-28T06:05:14.022 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.021+0000 7f6974c00640 1 --2- 172.21.15.96:0/3562877126 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f695406c8f0 0x7f695406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:14.022 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.021+0000 7f6974c00640 1 -- 172.21.15.96:0/3562877126 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5825+0+0 (secure 0 0 0) 0x7f696008c560 con 0x7f69780d1480 2024-05-28T06:05:14.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.022+0000 7f697cc00640 1 --2- 172.21.15.96:0/3562877126 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f695406c8f0 0x7f695406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:14.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.023+0000 7f697cc00640 1 --2- 172.21.15.96:0/3562877126 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f695406c8f0 0x7f695406edb0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f6968003d90 tx=0x7f6968009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:14.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.025+0000 7f6974c00640 1 -- 172.21.15.96:0/3562877126 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f69780b1c60 con 0x7f69780d1480 2024-05-28T06:05:14.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.210+0000 7f697f600640 1 -- 172.21.15.96:0/3562877126 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f69780690f0 con 0x7f69780d1480 2024-05-28T06:05:14.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.212+0000 7f6974c00640 1 -- 172.21.15.96:0/3562877126 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) ==== 74+0+16865 (secure 0 0 0) 0x7f696005a870 con 0x7f69780d1480 2024-05-28T06:05:14.214 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:05:14.214 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":22,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:04:52.624604+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-28T06:04:08.741938+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-28T06:04:09.301470+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.b","mappings":[{"from":3,"to":1}]},{"pgid":"3.d","mappings":[{"from":7,"to":2}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:05:14.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.215+0000 7f697f600640 1 -- 172.21.15.96:0/3562877126 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f695406c8f0 msgr2=0x7f695406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:14.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.215+0000 7f697f600640 1 --2- 172.21.15.96:0/3562877126 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f695406c8f0 0x7f695406edb0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f6968003d90 tx=0x7f6968009040 comp rx=0 tx=0).stop 2024-05-28T06:05:14.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.215+0000 7f697f600640 1 -- 172.21.15.96:0/3562877126 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f69780d1480 msgr2=0x7f69780c9620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:14.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.215+0000 7f697f600640 1 --2- 172.21.15.96:0/3562877126 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f69780d1480 0x7f69780c9620 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f6960007ef0 tx=0x7f696000e4b0 comp rx=0 tx=0).stop 2024-05-28T06:05:14.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.215+0000 7f697f600640 1 -- 172.21.15.96:0/3562877126 shutdown_connections 2024-05-28T06:05:14.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.215+0000 7f697f600640 1 --2- 172.21.15.96:0/3562877126 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f695406c8f0 0x7f695406edb0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:14.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.215+0000 7f697f600640 1 --2- 172.21.15.96:0/3562877126 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6978162040 0x7f69780b96d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:14.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.215+0000 7f697f600640 1 --2- 172.21.15.96:0/3562877126 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f69780d1480 0x7f69780c9620 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:14.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.215+0000 7f697f600640 1 --2- 172.21.15.96:0/3562877126 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69780d10a0 0x7f697815d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:14.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.215+0000 7f697f600640 1 -- 172.21.15.96:0/3562877126 >> 172.21.15.96:0/3562877126 conn(0x7f697808bfc0 msgr2=0x7f69780b1550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:14.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.215+0000 7f697f600640 1 -- 172.21.15.96:0/3562877126 shutdown_connections 2024-05-28T06:05:14.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.215+0000 7f697f600640 1 -- 172.21.15.96:0/3562877126 wait complete. 2024-05-28T06:05:14.234 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-28T06:03:59.517102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-28T06:04:08.741938+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-28T06:04:09.301470+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-28T06:05:14.235 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-28T06:05:14.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.427+0000 7f13f5c00640 1 Processor -- start 2024-05-28T06:05:14.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.428+0000 7f13f5c00640 1 -- start start 2024-05-28T06:05:14.430 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.429+0000 7f13f5c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f13f00ce910 0x7f13f006b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:14.430 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.429+0000 7f13f5c00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13f00ae930 con 0x7f13f015fc60 2024-05-28T06:05:14.430 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.429+0000 7f13f5c00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13f00ccec0 con 0x7f13f00d0140 2024-05-28T06:05:14.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.429+0000 7f13f5c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f13f00cdd70 con 0x7f13f00ce910 2024-05-28T06:05:14.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.429+0000 7f13eea00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f13f015fc60 0x7f13f00ae220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:34852/0 (socket says 172.21.15.96:34852) 2024-05-28T06:05:14.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.429+0000 7f13eea00640 1 -- 172.21.15.96:0/1203633484 learned_addr learned my addr 172.21.15.96:0/1203633484 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:05:14.433 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.430+0000 7f13ef400640 1 --2- 172.21.15.96:0/1203633484 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f13f00ce910 0x7f13f006b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:14.433 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.430+0000 7f13ed600640 1 -- 172.21.15.96:0/1203633484 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3857657467 0 0) 0x7f13f00ae930 con 0x7f13f015fc60 2024-05-28T06:05:14.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.430+0000 7f13ed600640 1 -- 172.21.15.96:0/1203633484 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f13d0003520 con 0x7f13f015fc60 2024-05-28T06:05:14.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.430+0000 7f13ed600640 1 -- 172.21.15.96:0/1203633484 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1547578489 0 0) 0x7f13d0003520 con 0x7f13f015fc60 2024-05-28T06:05:14.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.430+0000 7f13ed600640 1 -- 172.21.15.96:0/1203633484 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f13f00ce910 msgr2=0x7f13f006b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:14.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.430+0000 7f13ed600640 1 --2- 172.21.15.96:0/1203633484 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f13f00ce910 0x7f13f006b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:14.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.430+0000 7f13ed600640 1 -- 172.21.15.96:0/1203633484 >> v1:172.21.15.123:6789/0 conn(0x7f13f00d0140 legacy=0x7f13f00ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:14.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.430+0000 7f13ed600640 1 -- 172.21.15.96:0/1203633484 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13f00cd160 con 0x7f13f015fc60 2024-05-28T06:05:14.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.430+0000 7f13ed600640 1 -- 172.21.15.96:0/1203633484 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f13e0001670 con 0x7f13f015fc60 2024-05-28T06:05:14.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.430+0000 7f13ed600640 1 -- 172.21.15.96:0/1203633484 >> v1:172.21.15.96:6789/0 conn(0x7f13f015fc60 legacy=0x7f13f00ae220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:14.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.430+0000 7f13ed600640 1 --2- 172.21.15.96:0/1203633484 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f13f00ce910 0x7f13d00039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:14.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.430+0000 7f13ed600640 1 --2- 172.21.15.96:0/1203633484 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f13d0003fb0 0x7f13d0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:14.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.430+0000 7f13ed600640 1 --2- 172.21.15.96:0/1203633484 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13d0004d40 0x7f13d00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:14.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.430+0000 7f13ed600640 1 -- 172.21.15.96:0/1203633484 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f13f00cdd70 con 0x7f13d0004d40 2024-05-28T06:05:14.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.430+0000 7f13ed600640 1 -- 172.21.15.96:0/1203633484 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f13d000bb00 con 0x7f13d0003fb0 2024-05-28T06:05:14.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.430+0000 7f13ed600640 1 -- 172.21.15.96:0/1203633484 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f13d000bd00 con 0x7f13f00ce910 2024-05-28T06:05:14.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.431+0000 7f13ef400640 1 --2- 172.21.15.96:0/1203633484 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13d0004d40 0x7f13d00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:14.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.431+0000 7f13eea00640 1 --2- 172.21.15.96:0/1203633484 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f13f00ce910 0x7f13d00039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:14.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.431+0000 7f13ee000640 1 --2- 172.21.15.96:0/1203633484 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f13d0003fb0 0x7f13d0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:14.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.431+0000 7f13ef400640 1 -- 172.21.15.96:0/1203633484 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f13f00ce910 msgr2=0x7f13d00039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:14.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.431+0000 7f13ef400640 1 --2- 172.21.15.96:0/1203633484 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f13f00ce910 0x7f13d00039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:14.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.431+0000 7f13ef400640 1 -- 172.21.15.96:0/1203633484 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f13d0003fb0 msgr2=0x7f13d0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:14.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.431+0000 7f13ef400640 1 --2- 172.21.15.96:0/1203633484 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f13d0003fb0 0x7f13d0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:14.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.431+0000 7f13ef400640 1 -- 172.21.15.96:0/1203633484 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13d0003520 con 0x7f13d0004d40 2024-05-28T06:05:14.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.432+0000 7f13ef400640 1 --2- 172.21.15.96:0/1203633484 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13d0004d40 0x7f13d00051c0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f13e8009ac0 tx=0x7f13d0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76cbf8a137324549 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:14.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.432+0000 7f13ed600640 1 -- 172.21.15.96:0/1203633484 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13e8036070 con 0x7f13d0004d40 2024-05-28T06:05:14.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.432+0000 7f13ed600640 1 -- 172.21.15.96:0/1203633484 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13e8031400 con 0x7f13d0004d40 2024-05-28T06:05:14.436 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.432+0000 7f13ed600640 1 -- 172.21.15.96:0/1203633484 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13e8009690 con 0x7f13d0004d40 2024-05-28T06:05:14.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.433+0000 7f13f5c00640 1 -- 172.21.15.96:0/1203633484 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13d0004d40 msgr2=0x7f13d00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:14.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.433+0000 7f13f5c00640 1 --2- 172.21.15.96:0/1203633484 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13d0004d40 0x7f13d00051c0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f13e8009ac0 tx=0x7f13d0005830 comp rx=0 tx=0).stop 2024-05-28T06:05:14.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.433+0000 7f13ef400640 1 -- 172.21.15.96:0/1203633484 reap_dead start 2024-05-28T06:05:14.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.433+0000 7f13f5c00640 1 -- 172.21.15.96:0/1203633484 shutdown_connections 2024-05-28T06:05:14.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.433+0000 7f13f5c00640 1 -- 172.21.15.96:0/1203633484 >> 172.21.15.96:0/1203633484 conn(0x7f13f00b2f40 msgr2=0x7f13f00b3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:14.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.433+0000 7f13f5c00640 1 -- 172.21.15.96:0/1203633484 shutdown_connections 2024-05-28T06:05:14.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.433+0000 7f13f5c00640 1 -- 172.21.15.96:0/1203633484 wait complete. 2024-05-28T06:05:14.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.435+0000 7f13f5c00640 1 Processor -- start 2024-05-28T06:05:14.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.435+0000 7f13f5c00640 1 -- start start 2024-05-28T06:05:14.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.436+0000 7f13f5c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f13f014e600 0x7f13f015ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:14.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.436+0000 7f13f5c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13f014e9e0 0x7f13f01591a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:14.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.436+0000 7f13f5c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f13f01596e0 0x7f13f00b1e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:14.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.436+0000 7f13f5c00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f13f00cda00 con 0x7f13f014e9e0 2024-05-28T06:05:14.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.436+0000 7f13f5c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f13f00cd720 con 0x7f13f01596e0 2024-05-28T06:05:14.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.436+0000 7f13f5c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f13f00b3d70 con 0x7f13f014e600 2024-05-28T06:05:14.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.436+0000 7f13ee000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13f014e9e0 0x7f13f01591a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:14.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.436+0000 7f13ee000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13f014e9e0 0x7f13f01591a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51392/0 (socket says 172.21.15.96:51392) 2024-05-28T06:05:14.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.436+0000 7f13ee000640 1 -- 172.21.15.96:0/318838066 learned_addr learned my addr 172.21.15.96:0/318838066 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:14.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.437+0000 7f13eea00640 1 --2- 172.21.15.96:0/318838066 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f13f014e600 0x7f13f015ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:14.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.437+0000 7f13ee000640 1 -- 172.21.15.96:0/318838066 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f13f014e600 msgr2=0x7f13f015ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:14.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.437+0000 7f13ee000640 1 --2- 172.21.15.96:0/318838066 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f13f014e600 0x7f13f015ad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:14.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.437+0000 7f13ee000640 1 -- 172.21.15.96:0/318838066 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f13f01596e0 msgr2=0x7f13f00b1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:14.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.437+0000 7f13ef400640 1 --2- 172.21.15.96:0/318838066 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f13f01596e0 0x7f13f00b1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:14.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.437+0000 7f13ee000640 1 --2- 172.21.15.96:0/318838066 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f13f01596e0 0x7f13f00b1e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:14.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.437+0000 7f13ee000640 1 -- 172.21.15.96:0/318838066 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13f0159c60 con 0x7f13f014e9e0 2024-05-28T06:05:14.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.437+0000 7f13ee000640 1 --2- 172.21.15.96:0/318838066 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13f014e9e0 0x7f13f01591a0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f13d800e8f0 tx=0x7f13d800edb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:14.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.437+0000 7f13e6a00640 1 -- 172.21.15.96:0/318838066 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d8003b80 con 0x7f13f014e9e0 2024-05-28T06:05:14.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.437+0000 7f13e6a00640 1 -- 172.21.15.96:0/318838066 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13d8018070 con 0x7f13f014e9e0 2024-05-28T06:05:14.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.437+0000 7f13f5c00640 1 -- 172.21.15.96:0/318838066 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13f0159f40 con 0x7f13f014e9e0 2024-05-28T06:05:14.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.437+0000 7f13e6a00640 1 -- 172.21.15.96:0/318838066 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d80050d0 con 0x7f13f014e9e0 2024-05-28T06:05:14.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.437+0000 7f13f5c00640 1 -- 172.21.15.96:0/318838066 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13f015a500 con 0x7f13f014e9e0 2024-05-28T06:05:14.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.438+0000 7f13f5c00640 1 -- 172.21.15.96:0/318838066 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13f00689e0 con 0x7f13f014e9e0 2024-05-28T06:05:14.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.439+0000 7f13e6a00640 1 -- 172.21.15.96:0/318838066 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f13d8005270 con 0x7f13f014e9e0 2024-05-28T06:05:14.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.441+0000 7f13e6a00640 1 --2- 172.21.15.96:0/318838066 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f13cc06c8f0 0x7f13cc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:14.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.441+0000 7f13e6a00640 1 -- 172.21.15.96:0/318838066 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5825+0+0 (secure 0 0 0) 0x7f13d808bd20 con 0x7f13f014e9e0 2024-05-28T06:05:14.444 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.441+0000 7f13eea00640 1 --2- 172.21.15.96:0/318838066 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f13cc06c8f0 0x7f13cc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:14.444 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.443+0000 7f13eea00640 1 --2- 172.21.15.96:0/318838066 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f13cc06c8f0 0x7f13cc06edb0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f13e0001eb0 tx=0x7f13e0000f50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:14.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.445+0000 7f13e6a00640 1 -- 172.21.15.96:0/318838066 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f13d801a310 con 0x7f13f014e9e0 2024-05-28T06:05:14.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.647+0000 7f13f5c00640 1 -- 172.21.15.96:0/318838066 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f13f00b3560 con 0x7f13f014e9e0 2024-05-28T06:05:14.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.647+0000 7f13e6a00640 1 -- 172.21.15.96:0/318838066 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) ==== 93+0+10 (secure 0 0 0) 0x7f13d8059fa0 con 0x7f13f014e9e0 2024-05-28T06:05:14.649 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 1 2024-05-28T06:05:14.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.650+0000 7f13f5c00640 1 -- 172.21.15.96:0/318838066 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f13cc06c8f0 msgr2=0x7f13cc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:14.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.650+0000 7f13f5c00640 1 --2- 172.21.15.96:0/318838066 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f13cc06c8f0 0x7f13cc06edb0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f13e0001eb0 tx=0x7f13e0000f50 comp rx=0 tx=0).stop 2024-05-28T06:05:14.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.650+0000 7f13f5c00640 1 -- 172.21.15.96:0/318838066 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13f014e9e0 msgr2=0x7f13f01591a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:14.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.650+0000 7f13f5c00640 1 --2- 172.21.15.96:0/318838066 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13f014e9e0 0x7f13f01591a0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f13d800e8f0 tx=0x7f13d800edb0 comp rx=0 tx=0).stop 2024-05-28T06:05:14.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.651+0000 7f13f5c00640 1 -- 172.21.15.96:0/318838066 shutdown_connections 2024-05-28T06:05:14.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.651+0000 7f13f5c00640 1 --2- 172.21.15.96:0/318838066 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f13cc06c8f0 0x7f13cc06edb0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:14.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.651+0000 7f13f5c00640 1 --2- 172.21.15.96:0/318838066 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f13f01596e0 0x7f13f00b1e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:14.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.651+0000 7f13f5c00640 1 --2- 172.21.15.96:0/318838066 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13f014e9e0 0x7f13f01591a0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:14.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.651+0000 7f13f5c00640 1 --2- 172.21.15.96:0/318838066 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f13f014e600 0x7f13f015ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:14.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.651+0000 7f13f5c00640 1 -- 172.21.15.96:0/318838066 >> 172.21.15.96:0/318838066 conn(0x7f13f00b2f40 msgr2=0x7f13f006ffb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:14.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.651+0000 7f13f5c00640 1 -- 172.21.15.96:0/318838066 shutdown_connections 2024-05-28T06:05:14.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.651+0000 7f13f5c00640 1 -- 172.21.15.96:0/318838066 wait complete. 2024-05-28T06:05:14.671 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-28T06:05:14.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.866+0000 7f2207e00640 1 Processor -- start 2024-05-28T06:05:14.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.866+0000 7f2207e00640 1 -- start start 2024-05-28T06:05:14.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.868+0000 7f2207e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f22080ce1f0 0x7f22080b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:14.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.868+0000 7f2207e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f22080ce5d0 0x7f220815ec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:14.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.868+0000 7f2207e00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22080b3580 con 0x7f22080b3d60 2024-05-28T06:05:14.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.868+0000 7f2207e00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f220815f490 con 0x7f22080ce5d0 2024-05-28T06:05:14.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.868+0000 7f2207e00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f220815f770 con 0x7f22080ce1f0 2024-05-28T06:05:14.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.868+0000 7f2207400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f22080b3d60 0x7f22080ccb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:34864/0 (socket says 172.21.15.96:34864) 2024-05-28T06:05:14.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.868+0000 7f21fe000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f22080ce5d0 0x7f220815ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:14.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.868+0000 7f2207400640 1 -- 172.21.15.96:0/2263037708 learned_addr learned my addr 172.21.15.96:0/2263037708 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:05:14.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.869+0000 7f2206a00640 1 --2- 172.21.15.96:0/2263037708 >> v2:172.21.15.123:3300/0 conn(0x7f22080ce1f0 0x7f22080b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:14.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.869+0000 7f21fe000640 1 -- 172.21.15.96:0/2263037708 >> v2:172.21.15.123:3300/0 conn(0x7f22080ce1f0 msgr2=0x7f22080b4dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:14.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.869+0000 7f21fe000640 1 --2- 172.21.15.96:0/2263037708 >> v2:172.21.15.123:3300/0 conn(0x7f22080ce1f0 0x7f22080b4dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:14.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.869+0000 7f21fe000640 1 -- 172.21.15.96:0/2263037708 >> v1:172.21.15.96:6789/0 conn(0x7f22080b3d60 legacy=0x7f22080ccb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:14.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.869+0000 7f2206000640 1 -- 172.21.15.96:0/2263037708 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 547702596 0 0) 0x7f22080b3580 con 0x7f22080b3d60 2024-05-28T06:05:14.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.869+0000 7f21fe000640 1 -- 172.21.15.96:0/2263037708 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f220815fa50 con 0x7f22080ce5d0 2024-05-28T06:05:14.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.870+0000 7f21fe000640 1 --2- 172.21.15.96:0/2263037708 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f22080ce5d0 0x7f220815ec40 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f22080cdd00 tx=0x7f21f002a080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8d2c048c554c178 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:14.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.870+0000 7f2206000640 1 -- 172.21.15.96:0/2263037708 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21f0037070 con 0x7f22080ce5d0 2024-05-28T06:05:14.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.870+0000 7f2206000640 1 -- 172.21.15.96:0/2263037708 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21f002add0 con 0x7f22080ce5d0 2024-05-28T06:05:14.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.870+0000 7f2206000640 1 -- 172.21.15.96:0/2263037708 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21f0032660 con 0x7f22080ce5d0 2024-05-28T06:05:14.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.871+0000 7f2207e00640 1 -- 172.21.15.96:0/2263037708 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f22080ce5d0 msgr2=0x7f220815ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:14.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.871+0000 7f2207e00640 1 --2- 172.21.15.96:0/2263037708 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f22080ce5d0 0x7f220815ec40 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f22080cdd00 tx=0x7f21f002a080 comp rx=0 tx=0).stop 2024-05-28T06:05:14.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.871+0000 7f2207e00640 1 -- 172.21.15.96:0/2263037708 shutdown_connections 2024-05-28T06:05:14.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.871+0000 7f2207e00640 1 --2- 172.21.15.96:0/2263037708 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f22080ce5d0 0x7f220815ec40 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:14.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.871+0000 7f2207e00640 1 --2- 172.21.15.96:0/2263037708 >> v2:172.21.15.123:3300/0 conn(0x7f22080ce1f0 0x7f22080b4dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:14.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.871+0000 7f2207e00640 1 -- 172.21.15.96:0/2263037708 >> 172.21.15.96:0/2263037708 conn(0x7f220806b900 msgr2=0x7f220806bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:14.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.871+0000 7f2207e00640 1 -- 172.21.15.96:0/2263037708 shutdown_connections 2024-05-28T06:05:14.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.871+0000 7f2207e00640 1 -- 172.21.15.96:0/2263037708 wait complete. 2024-05-28T06:05:14.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.873+0000 7f2207e00640 1 Processor -- start 2024-05-28T06:05:14.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.873+0000 7f2207e00640 1 -- start start 2024-05-28T06:05:14.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.874+0000 7f2207e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f22080b3d60 0x7f220814b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:14.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.874+0000 7f2207e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f22080ce1f0 0x7f22080cfce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:14.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.874+0000 7f2207e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f22080ce5d0 0x7f22080d0220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:14.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.874+0000 7f2207e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f22080cd2a0 con 0x7f22080ce1f0 2024-05-28T06:05:14.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.874+0000 7f2207e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f220815f180 con 0x7f22080ce5d0 2024-05-28T06:05:14.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.874+0000 7f2207e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f22080cd420 con 0x7f22080b3d60 2024-05-28T06:05:14.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.874+0000 7f21fe000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f22080ce1f0 0x7f22080cfce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:14.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.874+0000 7f21fe000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f22080ce1f0 0x7f22080cfce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51414/0 (socket says 172.21.15.96:51414) 2024-05-28T06:05:14.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.874+0000 7f21fe000640 1 -- 172.21.15.96:0/2488438967 learned_addr learned my addr 172.21.15.96:0/2488438967 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:14.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.875+0000 7f2207400640 1 --2- 172.21.15.96:0/2488438967 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f22080ce5d0 0x7f22080d0220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:14.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.875+0000 7f21fe000640 1 -- 172.21.15.96:0/2488438967 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f22080b3d60 msgr2=0x7f220814b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:14.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.875+0000 7f2206a00640 1 --2- 172.21.15.96:0/2488438967 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f22080b3d60 0x7f220814b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:14.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.875+0000 7f21fe000640 1 --2- 172.21.15.96:0/2488438967 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f22080b3d60 0x7f220814b5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:14.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.875+0000 7f21fe000640 1 -- 172.21.15.96:0/2488438967 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f22080ce5d0 msgr2=0x7f22080d0220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:14.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.875+0000 7f21fe000640 1 --2- 172.21.15.96:0/2488438967 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f22080ce5d0 0x7f22080d0220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:14.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.875+0000 7f21fe000640 1 -- 172.21.15.96:0/2488438967 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f220814bb00 con 0x7f22080ce1f0 2024-05-28T06:05:14.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.875+0000 7f2207400640 1 --2- 172.21.15.96:0/2488438967 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f22080ce5d0 0x7f22080d0220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:05:14.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.875+0000 7f21fe000640 1 --2- 172.21.15.96:0/2488438967 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f22080ce1f0 0x7f22080cfce0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f21f0009890 tx=0x7f21f00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:14.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.875+0000 7f21ff400640 1 -- 172.21.15.96:0/2488438967 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21f0041020 con 0x7f22080ce1f0 2024-05-28T06:05:14.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.875+0000 7f21ff400640 1 -- 172.21.15.96:0/2488438967 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21f003d420 con 0x7f22080ce1f0 2024-05-28T06:05:14.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.875+0000 7f2207e00640 1 -- 172.21.15.96:0/2488438967 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2208160d40 con 0x7f22080ce1f0 2024-05-28T06:05:14.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.876+0000 7f21ff400640 1 -- 172.21.15.96:0/2488438967 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21f0037040 con 0x7f22080ce1f0 2024-05-28T06:05:14.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.876+0000 7f2207e00640 1 -- 172.21.15.96:0/2488438967 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2208161300 con 0x7f22080ce1f0 2024-05-28T06:05:14.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.877+0000 7f2207e00640 1 -- 172.21.15.96:0/2488438967 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22080689e0 con 0x7f22080ce1f0 2024-05-28T06:05:14.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.878+0000 7f21ff400640 1 -- 172.21.15.96:0/2488438967 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f21f0044020 con 0x7f22080ce1f0 2024-05-28T06:05:14.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.879+0000 7f21ff400640 1 --2- 172.21.15.96:0/2488438967 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f21e406c8f0 0x7f21e406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:14.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.879+0000 7f21ff400640 1 -- 172.21.15.96:0/2488438967 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5825+0+0 (secure 0 0 0) 0x7f21f00ac1c0 con 0x7f22080ce1f0 2024-05-28T06:05:14.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.880+0000 7f2206a00640 1 --2- 172.21.15.96:0/2488438967 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f21e406c8f0 0x7f21e406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:14.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.881+0000 7f2206a00640 1 --2- 172.21.15.96:0/2488438967 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f21e406c8f0 0x7f21e406edb0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f21f8006fd0 tx=0x7f21f8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:14.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:14.884+0000 7f21ff400640 1 -- 172.21.15.96:0/2488438967 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f21f004b030 con 0x7f22080ce1f0 2024-05-28T06:05:15.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.086+0000 7f2207e00640 1 -- 172.21.15.96:0/2488438967 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f22080ca070 con 0x7f22080ce1f0 2024-05-28T06:05:15.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.087+0000 7f21ff400640 1 -- 172.21.15.96:0/2488438967 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) ==== 104+0+11 (secure 0 0 0) 0x7f21f007a4d0 con 0x7f22080ce1f0 2024-05-28T06:05:15.089 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 32 2024-05-28T06:05:15.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.089+0000 7f2207e00640 1 -- 172.21.15.96:0/2488438967 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f21e406c8f0 msgr2=0x7f21e406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:15.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.089+0000 7f2207e00640 1 --2- 172.21.15.96:0/2488438967 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f21e406c8f0 0x7f21e406edb0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f21f8006fd0 tx=0x7f21f8008040 comp rx=0 tx=0).stop 2024-05-28T06:05:15.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.089+0000 7f2207e00640 1 -- 172.21.15.96:0/2488438967 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f22080ce1f0 msgr2=0x7f22080cfce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:15.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.089+0000 7f2207e00640 1 --2- 172.21.15.96:0/2488438967 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f22080ce1f0 0x7f22080cfce0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f21f0009890 tx=0x7f21f00046c0 comp rx=0 tx=0).stop 2024-05-28T06:05:15.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.090+0000 7f2207e00640 1 -- 172.21.15.96:0/2488438967 shutdown_connections 2024-05-28T06:05:15.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.090+0000 7f2207e00640 1 --2- 172.21.15.96:0/2488438967 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f21e406c8f0 0x7f21e406edb0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:15.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.090+0000 7f2207e00640 1 --2- 172.21.15.96:0/2488438967 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f22080ce5d0 0x7f22080d0220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:15.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.090+0000 7f2207e00640 1 --2- 172.21.15.96:0/2488438967 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f22080ce1f0 0x7f22080cfce0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:15.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.090+0000 7f2207e00640 1 --2- 172.21.15.96:0/2488438967 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f22080b3d60 0x7f220814b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:15.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.090+0000 7f2207e00640 1 -- 172.21.15.96:0/2488438967 >> 172.21.15.96:0/2488438967 conn(0x7f220806b900 msgr2=0x7f22080ccb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:15.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.090+0000 7f2207e00640 1 -- 172.21.15.96:0/2488438967 shutdown_connections 2024-05-28T06:05:15.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.090+0000 7f2207e00640 1 -- 172.21.15.96:0/2488438967 wait complete. 2024-05-28T06:05:15.109 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-28T06:05:15.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.308+0000 7f5343200640 1 Processor -- start 2024-05-28T06:05:15.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.308+0000 7f5343200640 1 -- start start 2024-05-28T06:05:15.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.309+0000 7f5343200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f534415e870 0x7f53440d0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:15.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.309+0000 7f5343200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f53440cfa40 0x7f53440d25e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:15.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.309+0000 7f5343200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f534415ec50 con 0x7f53440d14d0 2024-05-28T06:05:15.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.309+0000 7f5343200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f53440d2150 con 0x7f534415e870 2024-05-28T06:05:15.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.309+0000 7f5343200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f53440cff80 con 0x7f53440cfa40 2024-05-28T06:05:15.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.310+0000 7f5341e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f534415e870 0x7f53440d0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:15.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.310+0000 7f5341e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f534415e870 0x7f53440d0f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.96:37576/0 (socket says 172.21.15.96:37576) 2024-05-28T06:05:15.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.310+0000 7f5341e00640 1 -- 172.21.15.96:0/1441654501 learned_addr learned my addr 172.21.15.96:0/1441654501 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:15.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.310+0000 7f5342800640 1 --2- 172.21.15.96:0/1441654501 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f53440cfa40 0x7f53440d25e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:15.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.311+0000 7f5341e00640 1 -- 172.21.15.96:0/1441654501 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f53440cfa40 msgr2=0x7f53440d25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:15.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.311+0000 7f5341e00640 1 --2- 172.21.15.96:0/1441654501 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f53440cfa40 0x7f53440d25e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:15.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.311+0000 7f5341e00640 1 -- 172.21.15.96:0/1441654501 >> v1:172.21.15.123:6789/0 conn(0x7f53440d14d0 legacy=0x7f53440cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:15.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.311+0000 7f5341e00640 1 -- 172.21.15.96:0/1441654501 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53440d0260 con 0x7f534415e870 2024-05-28T06:05:15.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.312+0000 7f5341e00640 1 --2- 172.21.15.96:0/1441654501 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f534415e870 0x7f53440d0f90 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f532c009870 tx=0x7f532c02a270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=20fda71c55d6473b server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:15.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.313+0000 7f5340a00640 1 -- 172.21.15.96:0/1441654501 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f532c037070 con 0x7f534415e870 2024-05-28T06:05:15.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.313+0000 7f5340a00640 1 -- 172.21.15.96:0/1441654501 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f532c032440 con 0x7f534415e870 2024-05-28T06:05:15.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.313+0000 7f5340a00640 1 -- 172.21.15.96:0/1441654501 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f532c032810 con 0x7f534415e870 2024-05-28T06:05:15.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.313+0000 7f5343200640 1 -- 172.21.15.96:0/1441654501 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f534415e870 msgr2=0x7f53440d0f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:15.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.313+0000 7f5343200640 1 --2- 172.21.15.96:0/1441654501 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f534415e870 0x7f53440d0f90 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f532c009870 tx=0x7f532c02a270 comp rx=0 tx=0).stop 2024-05-28T06:05:15.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.313+0000 7f5343200640 1 -- 172.21.15.96:0/1441654501 shutdown_connections 2024-05-28T06:05:15.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.313+0000 7f5343200640 1 --2- 172.21.15.96:0/1441654501 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f53440cfa40 0x7f53440d25e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:15.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.314+0000 7f5343200640 1 --2- 172.21.15.96:0/1441654501 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f534415e870 0x7f53440d0f90 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:15.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.314+0000 7f5343200640 1 -- 172.21.15.96:0/1441654501 >> 172.21.15.96:0/1441654501 conn(0x7f534408bfc0 msgr2=0x7f53440d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:15.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.314+0000 7f5343200640 1 -- 172.21.15.96:0/1441654501 shutdown_connections 2024-05-28T06:05:15.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.314+0000 7f5343200640 1 -- 172.21.15.96:0/1441654501 wait complete. 2024-05-28T06:05:15.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.315+0000 7f5343200640 1 Processor -- start 2024-05-28T06:05:15.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.315+0000 7f5343200640 1 -- start start 2024-05-28T06:05:15.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.316+0000 7f5343200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f53440cfa40 0x7f53440cb650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:15.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.316+0000 7f5343200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f53440d14d0 0x7f53441f3b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:15.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.316+0000 7f5343200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f534415e870 0x7f53440cd0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:15.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.316+0000 7f5343200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f53440d4fb0 con 0x7f534415e870 2024-05-28T06:05:15.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.316+0000 7f5343200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f53440d0820 con 0x7f53440d14d0 2024-05-28T06:05:15.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.316+0000 7f5343200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f53440d2b20 con 0x7f53440cfa40 2024-05-28T06:05:15.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.317+0000 7f5342800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f534415e870 0x7f53440cd0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:15.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.317+0000 7f5342800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f534415e870 0x7f53440cd0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51416/0 (socket says 172.21.15.96:51416) 2024-05-28T06:05:15.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.317+0000 7f5342800640 1 -- 172.21.15.96:0/3076887949 learned_addr learned my addr 172.21.15.96:0/3076887949 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:15.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.317+0000 7f5341400640 1 --2- 172.21.15.96:0/3076887949 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f53440d14d0 0x7f53441f3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:15.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.317+0000 7f5341e00640 1 --2- 172.21.15.96:0/3076887949 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f53440cfa40 0x7f53440cb650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:15.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.317+0000 7f5342800640 1 -- 172.21.15.96:0/3076887949 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f53440cfa40 msgr2=0x7f53440cb650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:15.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.317+0000 7f5342800640 1 --2- 172.21.15.96:0/3076887949 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f53440cfa40 0x7f53440cb650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:15.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.317+0000 7f5342800640 1 -- 172.21.15.96:0/3076887949 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f53440d14d0 msgr2=0x7f53441f3b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:15.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.317+0000 7f5342800640 1 --2- 172.21.15.96:0/3076887949 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f53440d14d0 0x7f53441f3b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:15.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.317+0000 7f5342800640 1 -- 172.21.15.96:0/3076887949 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53440cb460 con 0x7f534415e870 2024-05-28T06:05:15.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.318+0000 7f5342800640 1 --2- 172.21.15.96:0/3076887949 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f534415e870 0x7f53440cd0d0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f533c00e850 tx=0x7f533c00ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:15.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.318+0000 7f533a000640 1 -- 172.21.15.96:0/3076887949 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f533c017070 con 0x7f534415e870 2024-05-28T06:05:15.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.318+0000 7f533a000640 1 -- 172.21.15.96:0/3076887949 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f533c00c940 con 0x7f534415e870 2024-05-28T06:05:15.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.318+0000 7f533a000640 1 -- 172.21.15.96:0/3076887949 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f533c00cd30 con 0x7f534415e870 2024-05-28T06:05:15.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.318+0000 7f5343200640 1 -- 172.21.15.96:0/3076887949 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53440cbd50 con 0x7f534415e870 2024-05-28T06:05:15.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.318+0000 7f5343200640 1 -- 172.21.15.96:0/3076887949 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53440cc310 con 0x7f534415e870 2024-05-28T06:05:15.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.319+0000 7f5343200640 1 -- 172.21.15.96:0/3076887949 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53440689e0 con 0x7f534415e870 2024-05-28T06:05:15.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.320+0000 7f533a000640 1 -- 172.21.15.96:0/3076887949 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f533c01a070 con 0x7f534415e870 2024-05-28T06:05:15.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.321+0000 7f533a000640 1 --2- 172.21.15.96:0/3076887949 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f532006c8f0 0x7f532006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:15.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.321+0000 7f533a000640 1 -- 172.21.15.96:0/3076887949 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5825+0+0 (secure 0 0 0) 0x7f533c08bc20 con 0x7f534415e870 2024-05-28T06:05:15.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.322+0000 7f5341e00640 1 --2- 172.21.15.96:0/3076887949 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f532006c8f0 0x7f532006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:15.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.323+0000 7f5341e00640 1 --2- 172.21.15.96:0/3076887949 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f532006c8f0 0x7f532006edb0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f532c034000 tx=0x7f532c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:15.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.325+0000 7f533a000640 1 -- 172.21.15.96:0/3076887949 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f533c059f30 con 0x7f534415e870 2024-05-28T06:05:15.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.525+0000 7f5343200640 1 -- 172.21.15.96:0/3076887949 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f53440ba500 con 0x7f534415e870 2024-05-28T06:05:15.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.526+0000 7f533a000640 1 -- 172.21.15.96:0/3076887949 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) ==== 100+0+11 (secure 0 0 0) 0x7f533c059830 con 0x7f534415e870 2024-05-28T06:05:15.528 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 64 2024-05-28T06:05:15.530 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.528+0000 7f5343200640 1 -- 172.21.15.96:0/3076887949 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f532006c8f0 msgr2=0x7f532006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:15.530 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.528+0000 7f5343200640 1 --2- 172.21.15.96:0/3076887949 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f532006c8f0 0x7f532006edb0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f532c034000 tx=0x7f532c02e000 comp rx=0 tx=0).stop 2024-05-28T06:05:15.530 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.528+0000 7f5343200640 1 -- 172.21.15.96:0/3076887949 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f534415e870 msgr2=0x7f53440cd0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:15.530 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.528+0000 7f5343200640 1 --2- 172.21.15.96:0/3076887949 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f534415e870 0x7f53440cd0d0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f533c00e850 tx=0x7f533c00ed10 comp rx=0 tx=0).stop 2024-05-28T06:05:15.530 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.528+0000 7f5343200640 1 -- 172.21.15.96:0/3076887949 shutdown_connections 2024-05-28T06:05:15.530 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.528+0000 7f5343200640 1 --2- 172.21.15.96:0/3076887949 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f532006c8f0 0x7f532006edb0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:15.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.529+0000 7f5343200640 1 --2- 172.21.15.96:0/3076887949 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f534415e870 0x7f53440cd0d0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:15.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.529+0000 7f5343200640 1 --2- 172.21.15.96:0/3076887949 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f53440d14d0 0x7f53441f3b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:15.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.529+0000 7f5343200640 1 --2- 172.21.15.96:0/3076887949 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f53440cfa40 0x7f53440cb650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:15.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.529+0000 7f5343200640 1 -- 172.21.15.96:0/3076887949 >> 172.21.15.96:0/3076887949 conn(0x7f534408bfc0 msgr2=0x7f53440b6050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:15.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.529+0000 7f5343200640 1 -- 172.21.15.96:0/3076887949 shutdown_connections 2024-05-28T06:05:15.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.529+0000 7f5343200640 1 -- 172.21.15.96:0/3076887949 wait complete. 2024-05-28T06:05:15.549 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-28T06:05:15.747 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.745+0000 7faaa1000640 1 Processor -- start 2024-05-28T06:05:15.747 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.746+0000 7faaa1000640 1 -- start start 2024-05-28T06:05:15.748 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.747+0000 7faaa1000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7faa9c0d16a0 0x7faa9c0cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:15.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.747+0000 7faaa1000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faa9c0cf870 0x7faa9c0d25e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:15.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.747+0000 7faaa1000640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa9c15ec50 con 0x7faa9c15e870 2024-05-28T06:05:15.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.747+0000 7faaa1000640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7faa9c0d2150 con 0x7faa9c0d16a0 2024-05-28T06:05:15.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.747+0000 7faaa1000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7faa9c0cfd20 con 0x7faa9c0cf870 2024-05-28T06:05:15.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.747+0000 7faa9b400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7faa9c15e870 0x7faa9c0d0f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:34874/0 (socket says 172.21.15.96:34874) 2024-05-28T06:05:15.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.747+0000 7faa9b400640 1 -- 172.21.15.96:0/2069003636 learned_addr learned my addr 172.21.15.96:0/2069003636 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:05:15.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.747+0000 7faa9aa00640 1 --2- 172.21.15.96:0/2069003636 >> v2:172.21.15.96:3300/0 conn(0x7faa9c0d16a0 0x7faa9c0cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:15.750 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.748+0000 7faa9be00640 1 --2- 172.21.15.96:0/2069003636 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faa9c0cf870 0x7faa9c0d25e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:15.750 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.748+0000 7faa9a000640 1 -- 172.21.15.96:0/2069003636 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 865109518 0 0) 0x7faa9c15ec50 con 0x7faa9c15e870 2024-05-28T06:05:15.750 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.748+0000 7faa9a000640 1 -- 172.21.15.96:0/2069003636 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa80003520 con 0x7faa9c15e870 2024-05-28T06:05:15.750 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.748+0000 7faa9aa00640 1 -- 172.21.15.96:0/2069003636 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faa9c0cf870 msgr2=0x7faa9c0d25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:15.750 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.748+0000 7faa9aa00640 1 --2- 172.21.15.96:0/2069003636 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faa9c0cf870 0x7faa9c0d25e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:15.750 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.748+0000 7faa9aa00640 1 -- 172.21.15.96:0/2069003636 >> v1:172.21.15.96:6789/0 conn(0x7faa9c15e870 legacy=0x7faa9c0d0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:15.750 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.748+0000 7faa9aa00640 1 -- 172.21.15.96:0/2069003636 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa9c0cff80 con 0x7faa9c0d16a0 2024-05-28T06:05:15.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.749+0000 7faa9aa00640 1 --2- 172.21.15.96:0/2069003636 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa9c0d16a0 0x7faa9c0cf330 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7faa8c009de0 tx=0x7faa8c02a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b6148d112635355 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:15.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.749+0000 7faa9a000640 1 -- 172.21.15.96:0/2069003636 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa8c007740 con 0x7faa9c0d16a0 2024-05-28T06:05:15.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.749+0000 7faa9a000640 1 -- 172.21.15.96:0/2069003636 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa8c0078e0 con 0x7faa9c0d16a0 2024-05-28T06:05:15.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.749+0000 7faa9a000640 1 -- 172.21.15.96:0/2069003636 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa8c007cb0 con 0x7faa9c0d16a0 2024-05-28T06:05:15.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.750+0000 7faaa1000640 1 -- 172.21.15.96:0/2069003636 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa9c0d16a0 msgr2=0x7faa9c0cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:15.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.750+0000 7faaa1000640 1 --2- 172.21.15.96:0/2069003636 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa9c0d16a0 0x7faa9c0cf330 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7faa8c009de0 tx=0x7faa8c02a6c0 comp rx=0 tx=0).stop 2024-05-28T06:05:15.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.750+0000 7faaa1000640 1 -- 172.21.15.96:0/2069003636 shutdown_connections 2024-05-28T06:05:15.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.750+0000 7faaa1000640 1 --2- 172.21.15.96:0/2069003636 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faa9c0cf870 0x7faa9c0d25e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:15.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.750+0000 7faaa1000640 1 --2- 172.21.15.96:0/2069003636 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa9c0d16a0 0x7faa9c0cf330 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:15.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.750+0000 7faaa1000640 1 -- 172.21.15.96:0/2069003636 >> 172.21.15.96:0/2069003636 conn(0x7faa9c08bfc0 msgr2=0x7faa9c0d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:15.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.750+0000 7faaa1000640 1 -- 172.21.15.96:0/2069003636 shutdown_connections 2024-05-28T06:05:15.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.750+0000 7faaa1000640 1 -- 172.21.15.96:0/2069003636 wait complete. 2024-05-28T06:05:15.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.752+0000 7faaa1000640 1 Processor -- start 2024-05-28T06:05:15.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.752+0000 7faaa1000640 1 -- start start 2024-05-28T06:05:15.755 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.753+0000 7faaa1000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa9c0cf870 0x7faa9c160830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:15.755 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.753+0000 7faaa1000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faa9c0d16a0 0x7faa9c0ccb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:15.755 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.753+0000 7faaa1000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa9c15e870 0x7faa9c14f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:15.755 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.753+0000 7faaa1000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7faa9c0d0820 con 0x7faa9c15e870 2024-05-28T06:05:15.755 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.753+0000 7faaa1000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7faa9c0d0540 con 0x7faa9c0cf870 2024-05-28T06:05:15.755 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.753+0000 7faaa1000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7faa9c0d4fb0 con 0x7faa9c0d16a0 2024-05-28T06:05:15.755 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.753+0000 7faa9be00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa9c15e870 0x7faa9c14f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:15.755 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.753+0000 7faa9be00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa9c15e870 0x7faa9c14f5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51438/0 (socket says 172.21.15.96:51438) 2024-05-28T06:05:15.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.753+0000 7faa9be00640 1 -- 172.21.15.96:0/3202466353 learned_addr learned my addr 172.21.15.96:0/3202466353 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:15.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.754+0000 7faa9aa00640 1 --2- 172.21.15.96:0/3202466353 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faa9c0d16a0 0x7faa9c0ccb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:15.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.754+0000 7faa9b400640 1 --2- 172.21.15.96:0/3202466353 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa9c0cf870 0x7faa9c160830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:15.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.754+0000 7faa9be00640 1 -- 172.21.15.96:0/3202466353 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faa9c0d16a0 msgr2=0x7faa9c0ccb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:15.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.754+0000 7faa9be00640 1 --2- 172.21.15.96:0/3202466353 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faa9c0d16a0 0x7faa9c0ccb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:15.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.754+0000 7faa9be00640 1 -- 172.21.15.96:0/3202466353 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa9c0cf870 msgr2=0x7faa9c160830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:15.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.754+0000 7faa9be00640 1 --2- 172.21.15.96:0/3202466353 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa9c0cf870 0x7faa9c160830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:15.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.754+0000 7faa9be00640 1 -- 172.21.15.96:0/3202466353 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa9c0cd4c0 con 0x7faa9c15e870 2024-05-28T06:05:15.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.754+0000 7faa9be00640 1 --2- 172.21.15.96:0/3202466353 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa9c15e870 0x7faa9c14f5c0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7faa9400d800 tx=0x7faa9400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:15.757 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.754+0000 7faa93400640 1 -- 172.21.15.96:0/3202466353 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa9400b6c0 con 0x7faa9c15e870 2024-05-28T06:05:15.757 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.754+0000 7faa93400640 1 -- 172.21.15.96:0/3202466353 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa94019070 con 0x7faa9c15e870 2024-05-28T06:05:15.757 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.754+0000 7faa93400640 1 -- 172.21.15.96:0/3202466353 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa9400bac0 con 0x7faa9c15e870 2024-05-28T06:05:15.757 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.754+0000 7faaa1000640 1 -- 172.21.15.96:0/3202466353 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa9c0cd7a0 con 0x7faa9c15e870 2024-05-28T06:05:15.757 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.754+0000 7faaa1000640 1 -- 172.21.15.96:0/3202466353 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa9c0cdd60 con 0x7faa9c15e870 2024-05-28T06:05:15.757 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.756+0000 7faaa1000640 1 -- 172.21.15.96:0/3202466353 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa9c0690f0 con 0x7faa9c15e870 2024-05-28T06:05:15.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.756+0000 7faa93400640 1 -- 172.21.15.96:0/3202466353 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7faa9400bc60 con 0x7faa9c15e870 2024-05-28T06:05:15.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.757+0000 7faa93400640 1 --2- 172.21.15.96:0/3202466353 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7faa7806c8f0 0x7faa7806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:15.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.757+0000 7faa93400640 1 -- 172.21.15.96:0/3202466353 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5825+0+0 (secure 0 0 0) 0x7faa9408bff0 con 0x7faa9c15e870 2024-05-28T06:05:15.759 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.758+0000 7faa9b400640 1 --2- 172.21.15.96:0/3202466353 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7faa7806c8f0 0x7faa7806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:15.760 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.759+0000 7faa9b400640 1 --2- 172.21.15.96:0/3202466353 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7faa7806c8f0 0x7faa7806edb0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7faa9c15a1d0 tx=0x7faa84009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:15.763 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:15.762+0000 7faa93400640 1 -- 172.21.15.96:0/3202466353 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7faa9c0690f0 con 0x7faa9c15e870 2024-05-28T06:05:16.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.003+0000 7faaa1000640 1 -- 172.21.15.96:0/3202466353 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faa9c06e6e0 con 0x7faa9c15e870 2024-05-28T06:05:16.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.004+0000 7faa93400640 1 -- 172.21.15.96:0/3202466353 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5178 (secure 0 0 0) 0x7faa9405a3f0 con 0x7faa9c15e870 2024-05-28T06:05:16.005 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:05:16.005 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/3349661842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":3349661842},{"type":"v1","addr":"172.21.15.96:6833","nonce":3349661842}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4270,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/3383264102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":3383264102},{"type":"v1","addr":"172.21.15.96:6835","nonce":3383264102}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4279,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/3068402544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":3068402544},{"type":"v1","addr":"172.21.15.123:6837","nonce":3068402544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-28T06:04:10.319738+0000","modified":"2024-05-28T06:04:13.704080+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4281},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4281":{"gid":4281,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.123:6835/605814935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":605814935},{"type":"v1","addr":"172.21.15.123:6835","nonce":605814935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4281,"qdb_cluster":[4281]},"id":1}]} 2024-05-28T06:05:16.006 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 7 2024-05-28T06:05:16.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.006+0000 7faaa1000640 1 -- 172.21.15.96:0/3202466353 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7faa7806c8f0 msgr2=0x7faa7806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:16.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.006+0000 7faaa1000640 1 --2- 172.21.15.96:0/3202466353 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7faa7806c8f0 0x7faa7806edb0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7faa9c15a1d0 tx=0x7faa84009040 comp rx=0 tx=0).stop 2024-05-28T06:05:16.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.006+0000 7faaa1000640 1 -- 172.21.15.96:0/3202466353 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa9c15e870 msgr2=0x7faa9c14f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:16.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.006+0000 7faaa1000640 1 --2- 172.21.15.96:0/3202466353 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa9c15e870 0x7faa9c14f5c0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7faa9400d800 tx=0x7faa9400dcc0 comp rx=0 tx=0).stop 2024-05-28T06:05:16.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.006+0000 7faaa1000640 1 -- 172.21.15.96:0/3202466353 shutdown_connections 2024-05-28T06:05:16.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.006+0000 7faaa1000640 1 --2- 172.21.15.96:0/3202466353 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7faa7806c8f0 0x7faa7806edb0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:16.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.006+0000 7faaa1000640 1 --2- 172.21.15.96:0/3202466353 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa9c15e870 0x7faa9c14f5c0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:16.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.006+0000 7faaa1000640 1 --2- 172.21.15.96:0/3202466353 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faa9c0d16a0 0x7faa9c0ccb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:16.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.006+0000 7faaa1000640 1 --2- 172.21.15.96:0/3202466353 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa9c0cf870 0x7faa9c160830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:16.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.006+0000 7faaa1000640 1 -- 172.21.15.96:0/3202466353 >> 172.21.15.96:0/3202466353 conn(0x7faa9c08bfc0 msgr2=0x7faa9c0cab40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:16.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.007+0000 7faaa1000640 1 -- 172.21.15.96:0/3202466353 shutdown_connections 2024-05-28T06:05:16.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.007+0000 7faaa1000640 1 -- 172.21.15.96:0/3202466353 wait complete. 2024-05-28T06:05:16.027 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:05:16.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.218+0000 7f7cedc00640 1 Processor -- start 2024-05-28T06:05:16.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.219+0000 7f7cedc00640 1 -- start start 2024-05-28T06:05:16.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.220+0000 7f7cedc00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f7ce815e9d0 0x7f7ce80d24d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:16.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.220+0000 7f7cedc00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f7ce80d52a0 0x7f7ce80d3c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:16.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.220+0000 7f7cedc00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7ce80d4150 0x7f7ce80d1650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:16.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.220+0000 7f7cedc00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f7ce815edb0 con 0x7f7ce815e9d0 2024-05-28T06:05:16.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.220+0000 7f7cedc00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f7ce80d5730 con 0x7f7ce80d52a0 2024-05-28T06:05:16.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.220+0000 7f7cedc00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f7ce80d2b70 con 0x7f7ce80d4150 2024-05-28T06:05:16.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.220+0000 7f7ce6a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f7ce815e9d0 0x7f7ce80d24d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:16.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.220+0000 7f7ce6a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f7ce815e9d0 0x7f7ce80d24d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51450/0 (socket says 172.21.15.96:51450) 2024-05-28T06:05:16.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.220+0000 7f7ce6a00640 1 -- 172.21.15.96:0/2396171445 learned_addr learned my addr 172.21.15.96:0/2396171445 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:16.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.221+0000 7f7ce7400640 1 --2- 172.21.15.96:0/2396171445 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7ce80d4150 0x7f7ce80d1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:16.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.221+0000 7f7ce6000640 1 --2- 172.21.15.96:0/2396171445 >> v2:172.21.15.123:3300/0 conn(0x7f7ce80d52a0 0x7f7ce80d3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:16.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.221+0000 7f7ce6a00640 1 -- 172.21.15.96:0/2396171445 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7ce80d4150 msgr2=0x7f7ce80d1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:16.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.221+0000 7f7ce6a00640 1 --2- 172.21.15.96:0/2396171445 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7ce80d4150 0x7f7ce80d1650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:16.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.221+0000 7f7ce6a00640 1 -- 172.21.15.96:0/2396171445 >> v2:172.21.15.123:3300/0 conn(0x7f7ce80d52a0 msgr2=0x7f7ce80d3c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:16.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.221+0000 7f7ce6a00640 1 --2- 172.21.15.96:0/2396171445 >> v2:172.21.15.123:3300/0 conn(0x7f7ce80d52a0 0x7f7ce80d3c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:16.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.221+0000 7f7ce6a00640 1 -- 172.21.15.96:0/2396171445 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ce80d2e50 con 0x7f7ce815e9d0 2024-05-28T06:05:16.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.221+0000 7f7ce7400640 1 --2- 172.21.15.96:0/2396171445 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7ce80d4150 0x7f7ce80d1650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:05:16.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.222+0000 7f7ce6a00640 1 --2- 172.21.15.96:0/2396171445 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7ce815e9d0 0x7f7ce80d24d0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f7ce80b3dc0 tx=0x7f7cd802a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8aac2cee30c52095 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:16.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.222+0000 7f7ce5600640 1 -- 172.21.15.96:0/2396171445 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cd8037070 con 0x7f7ce815e9d0 2024-05-28T06:05:16.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.222+0000 7f7ce5600640 1 -- 172.21.15.96:0/2396171445 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7cd8032440 con 0x7f7ce815e9d0 2024-05-28T06:05:16.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.222+0000 7f7ce5600640 1 -- 172.21.15.96:0/2396171445 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cd8032810 con 0x7f7ce815e9d0 2024-05-28T06:05:16.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.223+0000 7f7cedc00640 1 -- 172.21.15.96:0/2396171445 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7ce815e9d0 msgr2=0x7f7ce80d24d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:16.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.223+0000 7f7cedc00640 1 --2- 172.21.15.96:0/2396171445 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7ce815e9d0 0x7f7ce80d24d0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f7ce80b3dc0 tx=0x7f7cd802a140 comp rx=0 tx=0).stop 2024-05-28T06:05:16.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.223+0000 7f7cedc00640 1 -- 172.21.15.96:0/2396171445 shutdown_connections 2024-05-28T06:05:16.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.223+0000 7f7cedc00640 1 --2- 172.21.15.96:0/2396171445 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7ce80d4150 0x7f7ce80d1650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:16.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.223+0000 7f7cedc00640 1 --2- 172.21.15.96:0/2396171445 >> v2:172.21.15.123:3300/0 conn(0x7f7ce80d52a0 0x7f7ce80d3c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:16.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.223+0000 7f7cedc00640 1 --2- 172.21.15.96:0/2396171445 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7ce815e9d0 0x7f7ce80d24d0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:16.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.223+0000 7f7cedc00640 1 -- 172.21.15.96:0/2396171445 >> 172.21.15.96:0/2396171445 conn(0x7f7ce8067890 msgr2=0x7f7ce8067ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:16.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.223+0000 7f7cedc00640 1 -- 172.21.15.96:0/2396171445 shutdown_connections 2024-05-28T06:05:16.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.223+0000 7f7cedc00640 1 -- 172.21.15.96:0/2396171445 wait complete. 2024-05-28T06:05:16.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.225+0000 7f7cedc00640 1 Processor -- start 2024-05-28T06:05:16.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.226+0000 7f7cedc00640 1 -- start start 2024-05-28T06:05:16.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.227+0000 7f7cedc00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7ce80d52a0 0x7f7ce80cada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:16.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.227+0000 7f7cedc00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7ce815e9d0 0x7f7ce81f4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:16.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.227+0000 7f7cedc00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7ce8150be0 0x7f7ce81f5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:16.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.227+0000 7f7cedc00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f7ce80d36f0 con 0x7f7ce80d52a0 2024-05-28T06:05:16.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.227+0000 7f7cedc00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f7ce80d3410 con 0x7f7ce8150be0 2024-05-28T06:05:16.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.227+0000 7f7cedc00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f7ce80d1b90 con 0x7f7ce815e9d0 2024-05-28T06:05:16.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.227+0000 7f7ce6a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7ce80d52a0 0x7f7ce80cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:16.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.227+0000 7f7ce6a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7ce80d52a0 0x7f7ce80cada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51464/0 (socket says 172.21.15.96:51464) 2024-05-28T06:05:16.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.227+0000 7f7ce6a00640 1 -- 172.21.15.96:0/381629586 learned_addr learned my addr 172.21.15.96:0/381629586 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:16.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.227+0000 7f7ce7400640 1 --2- 172.21.15.96:0/381629586 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7ce8150be0 0x7f7ce81f5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:16.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.227+0000 7f7ce6a00640 1 -- 172.21.15.96:0/381629586 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7ce815e9d0 msgr2=0x7f7ce81f4fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:16.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.227+0000 7f7ce6a00640 1 --2- 172.21.15.96:0/381629586 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7ce815e9d0 0x7f7ce81f4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:16.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.227+0000 7f7ce6a00640 1 -- 172.21.15.96:0/381629586 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7ce8150be0 msgr2=0x7f7ce81f5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:16.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.228+0000 7f7ce6a00640 1 --2- 172.21.15.96:0/381629586 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7ce8150be0 0x7f7ce81f5700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:16.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.228+0000 7f7ce6a00640 1 -- 172.21.15.96:0/381629586 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ce81f5500 con 0x7f7ce80d52a0 2024-05-28T06:05:16.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.228+0000 7f7ce6a00640 1 --2- 172.21.15.96:0/381629586 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7ce80d52a0 0x7f7ce80cada0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f7cd8000c00 tx=0x7f7cd80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:16.231 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.228+0000 7f7cdea00640 1 -- 172.21.15.96:0/381629586 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cd8037040 con 0x7f7ce80d52a0 2024-05-28T06:05:16.231 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.228+0000 7f7cdea00640 1 -- 172.21.15.96:0/381629586 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7cd8042070 con 0x7f7ce80d52a0 2024-05-28T06:05:16.231 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.228+0000 7f7cdea00640 1 -- 172.21.15.96:0/381629586 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cd8030bb0 con 0x7f7ce80d52a0 2024-05-28T06:05:16.231 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.228+0000 7f7cedc00640 1 -- 172.21.15.96:0/381629586 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ce8151180 con 0x7f7ce80d52a0 2024-05-28T06:05:16.231 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.228+0000 7f7cedc00640 1 -- 172.21.15.96:0/381629586 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ce8151740 con 0x7f7ce80d52a0 2024-05-28T06:05:16.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.230+0000 7f7cedc00640 1 -- 172.21.15.96:0/381629586 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ce80689e0 con 0x7f7ce80d52a0 2024-05-28T06:05:16.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.230+0000 7f7cdea00640 1 -- 172.21.15.96:0/381629586 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7cd8047460 con 0x7f7ce80d52a0 2024-05-28T06:05:16.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.231+0000 7f7cdea00640 1 --2- 172.21.15.96:0/381629586 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7cc406c8f0 0x7f7cc406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:16.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.231+0000 7f7cdea00640 1 -- 172.21.15.96:0/381629586 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5825+0+0 (secure 0 0 0) 0x7f7cd80ac850 con 0x7f7ce80d52a0 2024-05-28T06:05:16.233 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.232+0000 7f7ce6000640 1 --2- 172.21.15.96:0/381629586 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7cc406c8f0 0x7f7cc406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:16.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.233+0000 7f7ce6000640 1 --2- 172.21.15.96:0/381629586 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7cc406c8f0 0x7f7cc406edb0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f7cd0006fd0 tx=0x7f7cd0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:16.238 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.237+0000 7f7cdea00640 1 -- 172.21.15.96:0/381629586 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7ce81f5500 con 0x7f7ce80d52a0 2024-05-28T06:05:16.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.435+0000 7f7cedc00640 1 -- 172.21.15.96:0/381629586 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7ce8160ba0 con 0x7f7ce80d52a0 2024-05-28T06:05:16.439 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.438+0000 7f7cdea00640 1 -- 172.21.15.96:0/381629586 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) ==== 74+0+16865 (secure 0 0 0) 0x7f7cd8045360 con 0x7f7ce80d52a0 2024-05-28T06:05:16.440 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:05:16.440 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":22,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:04:52.624604+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-28T06:04:08.741938+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-28T06:04:09.301470+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.b","mappings":[{"from":3,"to":1}]},{"pgid":"3.d","mappings":[{"from":7,"to":2}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:05:16.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.441+0000 7f7cedc00640 1 -- 172.21.15.96:0/381629586 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7cc406c8f0 msgr2=0x7f7cc406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:16.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.441+0000 7f7cedc00640 1 --2- 172.21.15.96:0/381629586 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7cc406c8f0 0x7f7cc406edb0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f7cd0006fd0 tx=0x7f7cd0008040 comp rx=0 tx=0).stop 2024-05-28T06:05:16.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.441+0000 7f7cedc00640 1 -- 172.21.15.96:0/381629586 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7ce80d52a0 msgr2=0x7f7ce80cada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:16.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.441+0000 7f7cedc00640 1 --2- 172.21.15.96:0/381629586 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7ce80d52a0 0x7f7ce80cada0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f7cd8000c00 tx=0x7f7cd80046c0 comp rx=0 tx=0).stop 2024-05-28T06:05:16.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.441+0000 7f7cedc00640 1 -- 172.21.15.96:0/381629586 shutdown_connections 2024-05-28T06:05:16.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.441+0000 7f7cedc00640 1 --2- 172.21.15.96:0/381629586 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7cc406c8f0 0x7f7cc406edb0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:16.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.441+0000 7f7cedc00640 1 --2- 172.21.15.96:0/381629586 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7ce8150be0 0x7f7ce81f5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:16.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.441+0000 7f7cedc00640 1 --2- 172.21.15.96:0/381629586 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7ce815e9d0 0x7f7ce81f4fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:16.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.441+0000 7f7cedc00640 1 --2- 172.21.15.96:0/381629586 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7ce80d52a0 0x7f7ce80cada0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:16.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.441+0000 7f7cedc00640 1 -- 172.21.15.96:0/381629586 >> 172.21.15.96:0/381629586 conn(0x7f7ce8067890 msgr2=0x7f7ce8067c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:16.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.441+0000 7f7cedc00640 1 -- 172.21.15.96:0/381629586 shutdown_connections 2024-05-28T06:05:16.444 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.441+0000 7f7cedc00640 1 -- 172.21.15.96:0/381629586 wait complete. 2024-05-28T06:05:16.462 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-05-28T06:05:16.462 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-28T06:05:16.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.651+0000 7f2090800640 1 Processor -- start 2024-05-28T06:05:16.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.651+0000 7f2090800640 1 -- start start 2024-05-28T06:05:16.657 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.653+0000 7f2090800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f208806f860 0x7f20880b3150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:16.657 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.653+0000 7f2090800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f20880d0bb0 0x7f20880ad130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:16.657 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.653+0000 7f2090800640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f208806ed10 con 0x7f20880b3690 2024-05-28T06:05:16.657 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.653+0000 7f2090800640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f208815cb00 con 0x7f208806f860 2024-05-28T06:05:16.657 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.653+0000 7f2090800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f20880b3cc0 con 0x7f20880d0bb0 2024-05-28T06:05:16.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.653+0000 7f208d400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f20880b3690 0x7f20880d04a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:34878/0 (socket says 172.21.15.96:34878) 2024-05-28T06:05:16.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.653+0000 7f208d400640 1 -- 172.21.15.96:0/3716484965 learned_addr learned my addr 172.21.15.96:0/3716484965 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:05:16.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.653+0000 7f208de00640 1 --2- 172.21.15.96:0/3716484965 >> v2:172.21.15.123:3300/0 conn(0x7f208806f860 0x7f20880b3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:16.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.654+0000 7f208e800640 1 --2- 172.21.15.96:0/3716484965 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f20880d0bb0 0x7f20880ad130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:16.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.654+0000 7f208ca00640 1 -- 172.21.15.96:0/3716484965 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1452226945 0 0) 0x7f2078001110 con 0x7f20880b3690 2024-05-28T06:05:16.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.654+0000 7f208ca00640 1 -- 172.21.15.96:0/3716484965 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2070003520 con 0x7f20880b3690 2024-05-28T06:05:16.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.654+0000 7f208ca00640 1 -- 172.21.15.96:0/3716484965 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4040899253 0 0) 0x7f2070003520 con 0x7f20880b3690 2024-05-28T06:05:16.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.654+0000 7f208ca00640 1 -- 172.21.15.96:0/3716484965 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f20880d0bb0 msgr2=0x7f20880ad130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:16.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.654+0000 7f208ca00640 1 --2- 172.21.15.96:0/3716484965 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f20880d0bb0 0x7f20880ad130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:16.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.654+0000 7f208ca00640 1 -- 172.21.15.96:0/3716484965 >> v2:172.21.15.123:3300/0 conn(0x7f208806f860 msgr2=0x7f20880b3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:16.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.654+0000 7f208ca00640 1 --2- 172.21.15.96:0/3716484965 >> v2:172.21.15.123:3300/0 conn(0x7f208806f860 0x7f20880b3150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:16.659 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.654+0000 7f208ca00640 1 -- 172.21.15.96:0/3716484965 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20880b3fa0 con 0x7f20880b3690 2024-05-28T06:05:16.659 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.654+0000 7f208ca00640 1 -- 172.21.15.96:0/3716484965 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f2078001840 con 0x7f20880b3690 2024-05-28T06:05:16.659 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.654+0000 7f208ca00640 1 -- 172.21.15.96:0/3716484965 >> v1:172.21.15.96:6789/0 conn(0x7f20880b3690 legacy=0x7f20880d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:16.659 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.654+0000 7f208ca00640 1 --2- 172.21.15.96:0/3716484965 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f20700039b0 0x7f2070003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:16.659 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.654+0000 7f208ca00640 1 --2- 172.21.15.96:0/3716484965 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2070004800 0x7f2070004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:16.659 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.654+0000 7f208ca00640 1 --2- 172.21.15.96:0/3716484965 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f20880d0bb0 0x7f20700051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:16.659 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.654+0000 7f208ca00640 1 -- 172.21.15.96:0/3716484965 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f208815cb00 con 0x7f2070004800 2024-05-28T06:05:16.659 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.654+0000 7f208ca00640 1 -- 172.21.15.96:0/3716484965 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f20880b3cc0 con 0x7f20700039b0 2024-05-28T06:05:16.659 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.654+0000 7f208ca00640 1 -- 172.21.15.96:0/3716484965 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f207000b9c0 con 0x7f20880d0bb0 2024-05-28T06:05:16.659 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.655+0000 7f208d400640 1 --2- 172.21.15.96:0/3716484965 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2070004800 0x7f2070004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:16.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.655+0000 7f208de00640 1 --2- 172.21.15.96:0/3716484965 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f20700039b0 0x7f2070003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:16.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.655+0000 7f208d400640 1 -- 172.21.15.96:0/3716484965 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f20880d0bb0 msgr2=0x7f20700051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:16.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.655+0000 7f208e800640 1 --2- 172.21.15.96:0/3716484965 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f20880d0bb0 0x7f20700051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:16.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.655+0000 7f208d400640 1 --2- 172.21.15.96:0/3716484965 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f20880d0bb0 0x7f20700051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:16.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.655+0000 7f208d400640 1 -- 172.21.15.96:0/3716484965 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f20700039b0 msgr2=0x7f2070003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:16.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.655+0000 7f208d400640 1 --2- 172.21.15.96:0/3716484965 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f20700039b0 0x7f2070003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:16.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.655+0000 7f208d400640 1 -- 172.21.15.96:0/3716484965 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2070003520 con 0x7f2070004800 2024-05-28T06:05:16.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.655+0000 7f208e800640 1 --2- 172.21.15.96:0/3716484965 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f20880d0bb0 0x7f20700051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:05:16.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.656+0000 7f208d400640 1 --2- 172.21.15.96:0/3716484965 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2070004800 0x7f2070004c80 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f207800c490 tx=0x7f2070005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0a77ff606e945a0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:16.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.656+0000 7f208ca00640 1 -- 172.21.15.96:0/3716484965 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2078032020 con 0x7f2070004800 2024-05-28T06:05:16.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.656+0000 7f208ca00640 1 -- 172.21.15.96:0/3716484965 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2078005ad0 con 0x7f2070004800 2024-05-28T06:05:16.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.656+0000 7f208ca00640 1 -- 172.21.15.96:0/3716484965 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f207800c0e0 con 0x7f2070004800 2024-05-28T06:05:16.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.657+0000 7f2090800640 1 -- 172.21.15.96:0/3716484965 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2070004800 msgr2=0x7f2070004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:16.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.657+0000 7f2090800640 1 --2- 172.21.15.96:0/3716484965 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2070004800 0x7f2070004c80 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f207800c490 tx=0x7f2070005830 comp rx=0 tx=0).stop 2024-05-28T06:05:16.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.657+0000 7f208e800640 1 -- 172.21.15.96:0/3716484965 reap_dead start 2024-05-28T06:05:16.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.657+0000 7f2090800640 1 -- 172.21.15.96:0/3716484965 shutdown_connections 2024-05-28T06:05:16.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.657+0000 7f2090800640 1 -- 172.21.15.96:0/3716484965 >> 172.21.15.96:0/3716484965 conn(0x7f208806bae0 msgr2=0x7f208806bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:16.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.657+0000 7f2090800640 1 -- 172.21.15.96:0/3716484965 shutdown_connections 2024-05-28T06:05:16.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.657+0000 7f2090800640 1 -- 172.21.15.96:0/3716484965 wait complete. 2024-05-28T06:05:16.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.659+0000 7f2090800640 1 Processor -- start 2024-05-28T06:05:16.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.659+0000 7f2090800640 1 -- start start 2024-05-28T06:05:16.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.660+0000 7f2090800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2070004800 0x7f20881f31e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:16.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.660+0000 7f2090800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f20881582b0 0x7f208815b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:16.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.660+0000 7f2090800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f208815bd40 0x7f20880c8020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:16.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.660+0000 7f2090800640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f20880b4840 con 0x7f208815bd40 2024-05-28T06:05:16.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.660+0000 7f2090800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f20880b4560 con 0x7f20881582b0 2024-05-28T06:05:16.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.660+0000 7f2090800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f20880ad130 con 0x7f2070004800 2024-05-28T06:05:16.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.661+0000 7f208e800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f208815bd40 0x7f20880c8020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:16.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.661+0000 7f208e800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f208815bd40 0x7f20880c8020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51492/0 (socket says 172.21.15.96:51492) 2024-05-28T06:05:16.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.661+0000 7f208e800640 1 -- 172.21.15.96:0/2410731930 learned_addr learned my addr 172.21.15.96:0/2410731930 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:16.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.661+0000 7f208de00640 1 --2- 172.21.15.96:0/2410731930 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2070004800 0x7f20881f31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:16.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.661+0000 7f208d400640 1 --2- 172.21.15.96:0/2410731930 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f20881582b0 0x7f208815b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:16.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.661+0000 7f208e800640 1 -- 172.21.15.96:0/2410731930 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2070004800 msgr2=0x7f20881f31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:16.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.661+0000 7f208e800640 1 --2- 172.21.15.96:0/2410731930 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2070004800 0x7f20881f31e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:16.666 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.661+0000 7f208e800640 1 -- 172.21.15.96:0/2410731930 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f20881582b0 msgr2=0x7f208815b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:16.666 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.661+0000 7f208e800640 1 --2- 172.21.15.96:0/2410731930 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f20881582b0 0x7f208815b800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:16.666 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.661+0000 7f208e800640 1 -- 172.21.15.96:0/2410731930 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20880c3a50 con 0x7f208815bd40 2024-05-28T06:05:16.666 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.661+0000 7f208e800640 1 --2- 172.21.15.96:0/2410731930 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f208815bd40 0x7f20880c8020 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f20880d0ff0 tx=0x7f208400f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:16.666 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.662+0000 7f2082000640 1 -- 172.21.15.96:0/2410731930 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f208401c070 con 0x7f208815bd40 2024-05-28T06:05:16.666 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.662+0000 7f2082000640 1 -- 172.21.15.96:0/2410731930 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2084017400 con 0x7f208815bd40 2024-05-28T06:05:16.666 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.662+0000 7f2082000640 1 -- 172.21.15.96:0/2410731930 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20840177d0 con 0x7f208815bd40 2024-05-28T06:05:16.666 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.662+0000 7f2090800640 1 -- 172.21.15.96:0/2410731930 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20880c9fb0 con 0x7f208815bd40 2024-05-28T06:05:16.666 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.662+0000 7f2090800640 1 -- 172.21.15.96:0/2410731930 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20880c3d10 con 0x7f208815bd40 2024-05-28T06:05:16.666 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.663+0000 7f2090800640 1 -- 172.21.15.96:0/2410731930 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20880689e0 con 0x7f208815bd40 2024-05-28T06:05:16.666 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.664+0000 7f2082000640 1 -- 172.21.15.96:0/2410731930 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2084005d10 con 0x7f208815bd40 2024-05-28T06:05:16.667 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.665+0000 7f2082000640 1 --2- 172.21.15.96:0/2410731930 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f206c06c830 0x7f206c06ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:16.667 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.665+0000 7f2082000640 1 -- 172.21.15.96:0/2410731930 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5825+0+0 (secure 0 0 0) 0x7f208408be10 con 0x7f208815bd40 2024-05-28T06:05:16.667 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.666+0000 7f208de00640 1 --2- 172.21.15.96:0/2410731930 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f206c06c830 0x7f206c06ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:16.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.667+0000 7f208de00640 1 --2- 172.21.15.96:0/2410731930 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f206c06c830 0x7f206c06ecf0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f207c009990 tx=0x7f207c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:16.670 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.669+0000 7f2082000640 1 -- 172.21.15.96:0/2410731930 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f208408f030 con 0x7f208815bd40 2024-05-28T06:05:16.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.911+0000 7f2090800640 1 -- 172.21.15.96:0/2410731930 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f208806e2f0 con 0x7f208815bd40 2024-05-28T06:05:16.914 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.912+0000 7f2082000640 1 -- 172.21.15.96:0/2410731930 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5178 (secure 0 0 0) 0x7f2084012070 con 0x7f208815bd40 2024-05-28T06:05:16.914 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:05:16.914 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/3349661842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":3349661842},{"type":"v1","addr":"172.21.15.96:6833","nonce":3349661842}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4270,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/3383264102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":3383264102},{"type":"v1","addr":"172.21.15.96:6835","nonce":3383264102}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4279,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/3068402544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":3068402544},{"type":"v1","addr":"172.21.15.123:6837","nonce":3068402544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-28T06:04:10.319738+0000","modified":"2024-05-28T06:04:13.704080+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4281},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4281":{"gid":4281,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.123:6835/605814935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":605814935},{"type":"v1","addr":"172.21.15.123:6835","nonce":605814935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4281,"qdb_cluster":[4281]},"id":1}]} 2024-05-28T06:05:16.915 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 7 2024-05-28T06:05:16.917 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.915+0000 7f2090800640 1 -- 172.21.15.96:0/2410731930 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f206c06c830 msgr2=0x7f206c06ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:16.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.915+0000 7f2090800640 1 --2- 172.21.15.96:0/2410731930 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f206c06c830 0x7f206c06ecf0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f207c009990 tx=0x7f207c008040 comp rx=0 tx=0).stop 2024-05-28T06:05:16.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.915+0000 7f2090800640 1 -- 172.21.15.96:0/2410731930 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f208815bd40 msgr2=0x7f20880c8020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:16.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.915+0000 7f2090800640 1 --2- 172.21.15.96:0/2410731930 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f208815bd40 0x7f20880c8020 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f20880d0ff0 tx=0x7f208400f4a0 comp rx=0 tx=0).stop 2024-05-28T06:05:16.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.915+0000 7f2090800640 1 -- 172.21.15.96:0/2410731930 shutdown_connections 2024-05-28T06:05:16.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.915+0000 7f2090800640 1 --2- 172.21.15.96:0/2410731930 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f206c06c830 0x7f206c06ecf0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:16.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.915+0000 7f2090800640 1 --2- 172.21.15.96:0/2410731930 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f208815bd40 0x7f20880c8020 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:16.919 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.915+0000 7f2090800640 1 --2- 172.21.15.96:0/2410731930 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f20881582b0 0x7f208815b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:16.921 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.915+0000 7f2090800640 1 --2- 172.21.15.96:0/2410731930 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2070004800 0x7f20881f31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:16.921 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.915+0000 7f2090800640 1 -- 172.21.15.96:0/2410731930 >> 172.21.15.96:0/2410731930 conn(0x7f208806bae0 msgr2=0x7f20880aff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:16.922 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.915+0000 7f2090800640 1 -- 172.21.15.96:0/2410731930 shutdown_connections 2024-05-28T06:05:16.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:16.915+0000 7f2090800640 1 -- 172.21.15.96:0/2410731930 wait complete. 2024-05-28T06:05:16.938 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-28T06:05:17.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.137+0000 7f0d32a00640 1 Processor -- start 2024-05-28T06:05:17.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.137+0000 7f0d32a00640 1 -- start start 2024-05-28T06:05:17.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.139+0000 7f0d32a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0d2c156b30 0x7f0d2c0af6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:17.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.139+0000 7f0d32a00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f0d2c06ee40 0x7f0d2c0ad230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:17.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.139+0000 7f0d32a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f0d2c06f2c0 0x7f0d2c0ad770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:17.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.139+0000 7f0d32a00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f0d2c156f10 con 0x7f0d2c06f2c0 2024-05-28T06:05:17.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.139+0000 7f0d32a00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f0d2c06dda0 con 0x7f0d2c06ee40 2024-05-28T06:05:17.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.139+0000 7f0d32a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f0d2c06e080 con 0x7f0d2c156b30 2024-05-28T06:05:17.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.139+0000 7f0d32000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f0d2c06f2c0 0x7f0d2c0ad770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:17.143 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.139+0000 7f0d32000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f0d2c06f2c0 0x7f0d2c0ad770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51500/0 (socket says 172.21.15.96:51500) 2024-05-28T06:05:17.143 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.139+0000 7f0d32000640 1 -- 172.21.15.96:0/2765663731 learned_addr learned my addr 172.21.15.96:0/2765663731 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:17.143 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.140+0000 7f0d30c00640 1 --2- 172.21.15.96:0/2765663731 >> v2:172.21.15.123:3300/0 conn(0x7f0d2c06ee40 0x7f0d2c0ad230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:17.143 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.140+0000 7f0d31600640 1 --2- 172.21.15.96:0/2765663731 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0d2c156b30 0x7f0d2c0af6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:17.143 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.140+0000 7f0d32000640 1 -- 172.21.15.96:0/2765663731 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0d2c156b30 msgr2=0x7f0d2c0af6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:17.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.140+0000 7f0d32000640 1 --2- 172.21.15.96:0/2765663731 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0d2c156b30 0x7f0d2c0af6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:17.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.140+0000 7f0d32000640 1 -- 172.21.15.96:0/2765663731 >> v2:172.21.15.123:3300/0 conn(0x7f0d2c06ee40 msgr2=0x7f0d2c0ad230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:17.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.140+0000 7f0d32000640 1 --2- 172.21.15.96:0/2765663731 >> v2:172.21.15.123:3300/0 conn(0x7f0d2c06ee40 0x7f0d2c0ad230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:17.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.140+0000 7f0d32000640 1 -- 172.21.15.96:0/2765663731 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d2c06e360 con 0x7f0d2c06f2c0 2024-05-28T06:05:17.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.141+0000 7f0d32000640 1 --2- 172.21.15.96:0/2765663731 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0d2c06f2c0 0x7f0d2c0ad770 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f0d2c06c860 tx=0x7f0d2402a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c57a0218045a3dfd server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:17.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.142+0000 7f0d2be00640 1 -- 172.21.15.96:0/2765663731 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d24036070 con 0x7f0d2c06f2c0 2024-05-28T06:05:17.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.142+0000 7f0d2be00640 1 -- 172.21.15.96:0/2765663731 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d24031600 con 0x7f0d2c06f2c0 2024-05-28T06:05:17.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.142+0000 7f0d2be00640 1 -- 172.21.15.96:0/2765663731 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d240319d0 con 0x7f0d2c06f2c0 2024-05-28T06:05:17.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.142+0000 7f0d32a00640 1 -- 172.21.15.96:0/2765663731 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0d2c06f2c0 msgr2=0x7f0d2c0ad770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:17.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.142+0000 7f0d32a00640 1 --2- 172.21.15.96:0/2765663731 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0d2c06f2c0 0x7f0d2c0ad770 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f0d2c06c860 tx=0x7f0d2402a610 comp rx=0 tx=0).stop 2024-05-28T06:05:17.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.142+0000 7f0d32a00640 1 -- 172.21.15.96:0/2765663731 shutdown_connections 2024-05-28T06:05:17.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.142+0000 7f0d32a00640 1 --2- 172.21.15.96:0/2765663731 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0d2c06f2c0 0x7f0d2c0ad770 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:17.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.142+0000 7f0d32a00640 1 --2- 172.21.15.96:0/2765663731 >> v2:172.21.15.123:3300/0 conn(0x7f0d2c06ee40 0x7f0d2c0ad230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:17.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.142+0000 7f0d32a00640 1 --2- 172.21.15.96:0/2765663731 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0d2c156b30 0x7f0d2c0af6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:17.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.142+0000 7f0d32a00640 1 -- 172.21.15.96:0/2765663731 >> 172.21.15.96:0/2765663731 conn(0x7f0d2c0b1fc0 msgr2=0x7f0d2c0b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:17.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.142+0000 7f0d32a00640 1 -- 172.21.15.96:0/2765663731 shutdown_connections 2024-05-28T06:05:17.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.142+0000 7f0d32a00640 1 -- 172.21.15.96:0/2765663731 wait complete. 2024-05-28T06:05:17.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.144+0000 7f0d32a00640 1 Processor -- start 2024-05-28T06:05:17.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.144+0000 7f0d32a00640 1 -- start start 2024-05-28T06:05:17.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.145+0000 7f0d32a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0d2c06ee40 0x7f0d2c0723a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:17.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.145+0000 7f0d32a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d2c06f2c0 0x7f0d2c157500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:17.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.145+0000 7f0d32a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0d2c156b30 0x7f0d2c1553e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:17.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.145+0000 7f0d32a00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f0d2c0afc20 con 0x7f0d2c156b30 2024-05-28T06:05:17.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.145+0000 7f0d32a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f0d2c0b2c30 con 0x7f0d2c06f2c0 2024-05-28T06:05:17.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.145+0000 7f0d32a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f0d2c157a40 con 0x7f0d2c06ee40 2024-05-28T06:05:17.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.145+0000 7f0d32000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0d2c156b30 0x7f0d2c1553e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:17.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.145+0000 7f0d32000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0d2c156b30 0x7f0d2c1553e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51504/0 (socket says 172.21.15.96:51504) 2024-05-28T06:05:17.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.145+0000 7f0d32000640 1 -- 172.21.15.96:0/453748000 learned_addr learned my addr 172.21.15.96:0/453748000 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:17.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.146+0000 7f0d30c00640 1 --2- 172.21.15.96:0/453748000 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d2c06f2c0 0x7f0d2c157500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:17.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.146+0000 7f0d31600640 1 --2- 172.21.15.96:0/453748000 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0d2c06ee40 0x7f0d2c0723a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:17.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.146+0000 7f0d32000640 1 -- 172.21.15.96:0/453748000 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0d2c06ee40 msgr2=0x7f0d2c0723a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:17.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.146+0000 7f0d32000640 1 --2- 172.21.15.96:0/453748000 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0d2c06ee40 0x7f0d2c0723a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:17.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.146+0000 7f0d32000640 1 -- 172.21.15.96:0/453748000 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d2c06f2c0 msgr2=0x7f0d2c157500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:17.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.146+0000 7f0d32000640 1 --2- 172.21.15.96:0/453748000 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d2c06f2c0 0x7f0d2c157500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:17.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.146+0000 7f0d32000640 1 -- 172.21.15.96:0/453748000 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d2c072a30 con 0x7f0d2c156b30 2024-05-28T06:05:17.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.146+0000 7f0d30c00640 1 --2- 172.21.15.96:0/453748000 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d2c06f2c0 0x7f0d2c157500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:05:17.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.146+0000 7f0d32000640 1 --2- 172.21.15.96:0/453748000 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0d2c156b30 0x7f0d2c1553e0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f0d24034040 tx=0x7f0d24004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:17.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.146+0000 7f0d29600640 1 -- 172.21.15.96:0/453748000 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d24036050 con 0x7f0d2c156b30 2024-05-28T06:05:17.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.147+0000 7f0d29600640 1 -- 172.21.15.96:0/453748000 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d24042070 con 0x7f0d2c156b30 2024-05-28T06:05:17.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.147+0000 7f0d29600640 1 -- 172.21.15.96:0/453748000 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d24009510 con 0x7f0d2c156b30 2024-05-28T06:05:17.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.147+0000 7f0d32a00640 1 -- 172.21.15.96:0/453748000 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d2c072d10 con 0x7f0d2c156b30 2024-05-28T06:05:17.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.147+0000 7f0d32a00640 1 -- 172.21.15.96:0/453748000 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d2c0732d0 con 0x7f0d2c156b30 2024-05-28T06:05:17.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.148+0000 7f0d32a00640 1 -- 172.21.15.96:0/453748000 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d2c0689e0 con 0x7f0d2c156b30 2024-05-28T06:05:17.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.149+0000 7f0d29600640 1 -- 172.21.15.96:0/453748000 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0d24007bf0 con 0x7f0d2c156b30 2024-05-28T06:05:17.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.149+0000 7f0d29600640 1 --2- 172.21.15.96:0/453748000 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0d0c06c8f0 0x7f0d0c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:17.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.150+0000 7f0d29600640 1 -- 172.21.15.96:0/453748000 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5825+0+0 (secure 0 0 0) 0x7f0d240aca00 con 0x7f0d2c156b30 2024-05-28T06:05:17.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.150+0000 7f0d31600640 1 --2- 172.21.15.96:0/453748000 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0d0c06c8f0 0x7f0d0c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:17.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.151+0000 7f0d31600640 1 --2- 172.21.15.96:0/453748000 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0d0c06c8f0 0x7f0d0c06edb0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f0d20006fd0 tx=0x7f0d20008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:17.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.154+0000 7f0d29600640 1 -- 172.21.15.96:0/453748000 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0d2407ada0 con 0x7f0d2c156b30 2024-05-28T06:05:17.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.387+0000 7f0d32a00640 1 -- 172.21.15.96:0/453748000 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0d2c06f7f0 con 0x7f0d2c156b30 2024-05-28T06:05:17.390 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.389+0000 7f0d29600640 1 -- 172.21.15.96:0/453748000 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5178 (secure 0 0 0) 0x7f0d2407a6a0 con 0x7f0d2c156b30 2024-05-28T06:05:17.390 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:05:17.390 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/3349661842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":3349661842},{"type":"v1","addr":"172.21.15.96:6833","nonce":3349661842}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4270,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/3383264102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":3383264102},{"type":"v1","addr":"172.21.15.96:6835","nonce":3383264102}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4279,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/3068402544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":3068402544},{"type":"v1","addr":"172.21.15.123:6837","nonce":3068402544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-28T06:04:10.319738+0000","modified":"2024-05-28T06:04:13.704080+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4281},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4281":{"gid":4281,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.123:6835/605814935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":605814935},{"type":"v1","addr":"172.21.15.123:6835","nonce":605814935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4281,"qdb_cluster":[4281]},"id":1}]} 2024-05-28T06:05:17.390 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 7 2024-05-28T06:05:17.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.391+0000 7f0d32a00640 1 -- 172.21.15.96:0/453748000 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0d0c06c8f0 msgr2=0x7f0d0c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:17.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.391+0000 7f0d32a00640 1 --2- 172.21.15.96:0/453748000 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0d0c06c8f0 0x7f0d0c06edb0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f0d20006fd0 tx=0x7f0d20008040 comp rx=0 tx=0).stop 2024-05-28T06:05:17.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.391+0000 7f0d32a00640 1 -- 172.21.15.96:0/453748000 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0d2c156b30 msgr2=0x7f0d2c1553e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:17.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.391+0000 7f0d32a00640 1 --2- 172.21.15.96:0/453748000 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0d2c156b30 0x7f0d2c1553e0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f0d24034040 tx=0x7f0d24004a40 comp rx=0 tx=0).stop 2024-05-28T06:05:17.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.391+0000 7f0d32a00640 1 -- 172.21.15.96:0/453748000 shutdown_connections 2024-05-28T06:05:17.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.391+0000 7f0d32a00640 1 --2- 172.21.15.96:0/453748000 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0d0c06c8f0 0x7f0d0c06edb0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:17.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.391+0000 7f0d32a00640 1 --2- 172.21.15.96:0/453748000 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0d2c156b30 0x7f0d2c1553e0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:17.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.391+0000 7f0d32a00640 1 --2- 172.21.15.96:0/453748000 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d2c06f2c0 0x7f0d2c157500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:17.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.391+0000 7f0d32a00640 1 --2- 172.21.15.96:0/453748000 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0d2c06ee40 0x7f0d2c0723a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:17.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.391+0000 7f0d32a00640 1 -- 172.21.15.96:0/453748000 >> 172.21.15.96:0/453748000 conn(0x7f0d2c0b1fc0 msgr2=0x7f0d2c0b0e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:17.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.392+0000 7f0d32a00640 1 -- 172.21.15.96:0/453748000 shutdown_connections 2024-05-28T06:05:17.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.392+0000 7f0d32a00640 1 -- 172.21.15.96:0/453748000 wait complete. 2024-05-28T06:05:17.410 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:05:17.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.607+0000 7fc757e00640 1 Processor -- start 2024-05-28T06:05:17.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.607+0000 7fc757e00640 1 -- start start 2024-05-28T06:05:17.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.609+0000 7fc757e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc75815e870 0x7fc7580d0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:17.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.609+0000 7fc757e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fc7580cfa40 0x7fc7580d25e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:17.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.609+0000 7fc757e00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc75815ec50 con 0x7fc7580d14d0 2024-05-28T06:05:17.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.609+0000 7fc757e00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fc7580d2150 con 0x7fc7580cfa40 2024-05-28T06:05:17.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.609+0000 7fc757e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fc7580cff80 con 0x7fc75815e870 2024-05-28T06:05:17.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.609+0000 7fc756000640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fc7580d14d0 0x7fc7580cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:34892/0 (socket says 172.21.15.96:34892) 2024-05-28T06:05:17.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.609+0000 7fc756000640 1 -- 172.21.15.96:0/68034413 learned_addr learned my addr 172.21.15.96:0/68034413 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:05:17.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.610+0000 7fc757400640 1 --2- 172.21.15.96:0/68034413 >> v2:172.21.15.123:3300/0 conn(0x7fc7580cfa40 0x7fc7580d25e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:17.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.610+0000 7fc756a00640 1 --2- 172.21.15.96:0/68034413 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc75815e870 0x7fc7580d0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:17.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.610+0000 7fc755600640 1 -- 172.21.15.96:0/68034413 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 491691177 0 0) 0x7fc75815ec50 con 0x7fc7580d14d0 2024-05-28T06:05:17.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.610+0000 7fc755600640 1 -- 172.21.15.96:0/68034413 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc738003520 con 0x7fc7580d14d0 2024-05-28T06:05:17.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.610+0000 7fc755600640 1 -- 172.21.15.96:0/68034413 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 909890417 0 0) 0x7fc738003520 con 0x7fc7580d14d0 2024-05-28T06:05:17.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.610+0000 7fc755600640 1 -- 172.21.15.96:0/68034413 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc75815e870 msgr2=0x7fc7580d0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:17.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.610+0000 7fc755600640 1 --2- 172.21.15.96:0/68034413 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc75815e870 0x7fc7580d0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:17.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.610+0000 7fc755600640 1 -- 172.21.15.96:0/68034413 >> v2:172.21.15.123:3300/0 conn(0x7fc7580cfa40 msgr2=0x7fc7580d25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:17.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.610+0000 7fc755600640 1 --2- 172.21.15.96:0/68034413 >> v2:172.21.15.123:3300/0 conn(0x7fc7580cfa40 0x7fc7580d25e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:17.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.610+0000 7fc755600640 1 -- 172.21.15.96:0/68034413 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7580d0260 con 0x7fc7580d14d0 2024-05-28T06:05:17.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.610+0000 7fc755600640 1 -- 172.21.15.96:0/68034413 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7fc740001b70 con 0x7fc7580d14d0 2024-05-28T06:05:17.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.610+0000 7fc755600640 1 -- 172.21.15.96:0/68034413 >> v1:172.21.15.96:6789/0 conn(0x7fc7580d14d0 legacy=0x7fc7580cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:17.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.610+0000 7fc755600640 1 --2- 172.21.15.96:0/68034413 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc7380039b0 0x7fc738003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:17.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.610+0000 7fc755600640 1 --2- 172.21.15.96:0/68034413 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc738004800 0x7fc738004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:17.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.610+0000 7fc755600640 1 --2- 172.21.15.96:0/68034413 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc75815e870 0x7fc7380051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:17.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.610+0000 7fc755600640 1 -- 172.21.15.96:0/68034413 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc7580d2150 con 0x7fc7380039b0 2024-05-28T06:05:17.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.610+0000 7fc755600640 1 -- 172.21.15.96:0/68034413 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc7580cff80 con 0x7fc738004800 2024-05-28T06:05:17.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.610+0000 7fc755600640 1 -- 172.21.15.96:0/68034413 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fc73800b9c0 con 0x7fc75815e870 2024-05-28T06:05:17.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.611+0000 7fc756a00640 1 --2- 172.21.15.96:0/68034413 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc7380039b0 0x7fc738003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:17.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.611+0000 7fc757400640 1 --2- 172.21.15.96:0/68034413 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc75815e870 0x7fc7380051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:17.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.611+0000 7fc756000640 1 --2- 172.21.15.96:0/68034413 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc738004800 0x7fc738004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:17.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.611+0000 7fc756a00640 1 -- 172.21.15.96:0/68034413 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc75815e870 msgr2=0x7fc7380051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:17.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.611+0000 7fc756a00640 1 --2- 172.21.15.96:0/68034413 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc75815e870 0x7fc7380051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:17.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.611+0000 7fc756a00640 1 -- 172.21.15.96:0/68034413 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc738004800 msgr2=0x7fc738004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:17.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.611+0000 7fc756a00640 1 --2- 172.21.15.96:0/68034413 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc738004800 0x7fc738004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:17.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.611+0000 7fc756a00640 1 -- 172.21.15.96:0/68034413 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc738003520 con 0x7fc7380039b0 2024-05-28T06:05:17.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.612+0000 7fc756a00640 1 --2- 172.21.15.96:0/68034413 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc7380039b0 0x7fc738003dd0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fc748009ac0 tx=0x7fc738005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d126c1e94a71f106 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:17.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.613+0000 7fc755600640 1 -- 172.21.15.96:0/68034413 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc748036070 con 0x7fc7380039b0 2024-05-28T06:05:17.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.613+0000 7fc755600640 1 -- 172.21.15.96:0/68034413 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc748031400 con 0x7fc7380039b0 2024-05-28T06:05:17.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.613+0000 7fc755600640 1 -- 172.21.15.96:0/68034413 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc748009690 con 0x7fc7380039b0 2024-05-28T06:05:17.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.613+0000 7fc757e00640 1 -- 172.21.15.96:0/68034413 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc7380039b0 msgr2=0x7fc738003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:17.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.613+0000 7fc757e00640 1 --2- 172.21.15.96:0/68034413 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc7380039b0 0x7fc738003dd0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fc748009ac0 tx=0x7fc738005830 comp rx=0 tx=0).stop 2024-05-28T06:05:17.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.613+0000 7fc757400640 1 -- 172.21.15.96:0/68034413 reap_dead start 2024-05-28T06:05:17.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.613+0000 7fc757e00640 1 -- 172.21.15.96:0/68034413 shutdown_connections 2024-05-28T06:05:17.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.613+0000 7fc757e00640 1 -- 172.21.15.96:0/68034413 >> 172.21.15.96:0/68034413 conn(0x7fc75808bfc0 msgr2=0x7fc7580d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:17.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.613+0000 7fc757e00640 1 -- 172.21.15.96:0/68034413 shutdown_connections 2024-05-28T06:05:17.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.613+0000 7fc757e00640 1 -- 172.21.15.96:0/68034413 wait complete. 2024-05-28T06:05:17.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.615+0000 7fc757e00640 1 Processor -- start 2024-05-28T06:05:17.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.615+0000 7fc757e00640 1 -- start start 2024-05-28T06:05:17.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.616+0000 7fc757e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc7380039b0 0x7fc758159b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:17.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.616+0000 7fc757e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc7580d24e0 0x7fc7580d2960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:17.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.616+0000 7fc757e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc7581f7960 0x7fc7581f7de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:17.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.616+0000 7fc757e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc7580d4fb0 con 0x7fc7580d24e0 2024-05-28T06:05:17.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.616+0000 7fc757e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc7580d0820 con 0x7fc7380039b0 2024-05-28T06:05:17.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.616+0000 7fc757e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fc7580d0f90 con 0x7fc7581f7960 2024-05-28T06:05:17.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.616+0000 7fc756000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc7580d24e0 0x7fc7580d2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:17.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.616+0000 7fc756000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc7580d24e0 0x7fc7580d2960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51526/0 (socket says 172.21.15.96:51526) 2024-05-28T06:05:17.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.616+0000 7fc756000640 1 -- 172.21.15.96:0/4245017799 learned_addr learned my addr 172.21.15.96:0/4245017799 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:17.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.617+0000 7fc756000640 1 -- 172.21.15.96:0/4245017799 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc7581f7960 msgr2=0x7fc7581f7de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:17.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.617+0000 7fc756a00640 1 --2- 172.21.15.96:0/4245017799 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc7380039b0 0x7fc758159b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:17.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.617+0000 7fc756000640 1 --2- 172.21.15.96:0/4245017799 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc7581f7960 0x7fc7581f7de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:17.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.617+0000 7fc756000640 1 -- 172.21.15.96:0/4245017799 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc7380039b0 msgr2=0x7fc758159b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:17.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.617+0000 7fc756000640 1 --2- 172.21.15.96:0/4245017799 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc7380039b0 0x7fc758159b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:17.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.617+0000 7fc756000640 1 -- 172.21.15.96:0/4245017799 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7581f8700 con 0x7fc7580d24e0 2024-05-28T06:05:17.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.617+0000 7fc756000640 1 --2- 172.21.15.96:0/4245017799 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc7580d24e0 0x7fc7580d2960 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fc74000ea60 tx=0x7fc74000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:17.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.617+0000 7fc74ea00640 1 -- 172.21.15.96:0/4245017799 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc740005d30 con 0x7fc7580d24e0 2024-05-28T06:05:17.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.617+0000 7fc74ea00640 1 -- 172.21.15.96:0/4245017799 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc740018070 con 0x7fc7580d24e0 2024-05-28T06:05:17.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.617+0000 7fc757e00640 1 -- 172.21.15.96:0/4245017799 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7580b71e0 con 0x7fc7580d24e0 2024-05-28T06:05:17.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.617+0000 7fc74ea00640 1 -- 172.21.15.96:0/4245017799 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc74000ca00 con 0x7fc7580d24e0 2024-05-28T06:05:17.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.617+0000 7fc757e00640 1 -- 172.21.15.96:0/4245017799 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7580b77a0 con 0x7fc7580d24e0 2024-05-28T06:05:17.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.618+0000 7fc757e00640 1 -- 172.21.15.96:0/4245017799 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7580d2ea0 con 0x7fc7580d24e0 2024-05-28T06:05:17.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.619+0000 7fc74ea00640 1 -- 172.21.15.96:0/4245017799 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc74000cba0 con 0x7fc7580d24e0 2024-05-28T06:05:17.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.620+0000 7fc74ea00640 1 --2- 172.21.15.96:0/4245017799 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc73406c8f0 0x7fc73406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:17.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.620+0000 7fc74ea00640 1 -- 172.21.15.96:0/4245017799 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5825+0+0 (secure 0 0 0) 0x7fc74008be20 con 0x7fc7580d24e0 2024-05-28T06:05:17.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.621+0000 7fc756a00640 1 --2- 172.21.15.96:0/4245017799 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc73406c8f0 0x7fc73406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:17.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.622+0000 7fc756a00640 1 --2- 172.21.15.96:0/4245017799 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc73406c8f0 0x7fc73406edb0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fc748033000 tx=0x7fc748008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:17.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.624+0000 7fc74ea00640 1 -- 172.21.15.96:0/4245017799 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc74005a130 con 0x7fc7580d24e0 2024-05-28T06:05:17.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.829+0000 7fc757e00640 1 -- 172.21.15.96:0/4245017799 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc7580689e0 con 0x7fc7580d24e0 2024-05-28T06:05:17.833 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.831+0000 7fc74ea00640 1 -- 172.21.15.96:0/4245017799 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) ==== 74+0+16865 (secure 0 0 0) 0x7fc740059a30 con 0x7fc7580d24e0 2024-05-28T06:05:17.834 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:05:17.835 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":22,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:04:52.624604+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-28T06:04:08.741938+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-28T06:04:09.301470+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.b","mappings":[{"from":3,"to":1}]},{"pgid":"3.d","mappings":[{"from":7,"to":2}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:05:17.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.834+0000 7fc757e00640 1 -- 172.21.15.96:0/4245017799 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc73406c8f0 msgr2=0x7fc73406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:17.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.834+0000 7fc757e00640 1 --2- 172.21.15.96:0/4245017799 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc73406c8f0 0x7fc73406edb0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fc748033000 tx=0x7fc748008030 comp rx=0 tx=0).stop 2024-05-28T06:05:17.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.834+0000 7fc757e00640 1 -- 172.21.15.96:0/4245017799 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc7580d24e0 msgr2=0x7fc7580d2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:17.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.834+0000 7fc757e00640 1 --2- 172.21.15.96:0/4245017799 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc7580d24e0 0x7fc7580d2960 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fc74000ea60 tx=0x7fc74000ef20 comp rx=0 tx=0).stop 2024-05-28T06:05:17.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.834+0000 7fc757e00640 1 -- 172.21.15.96:0/4245017799 shutdown_connections 2024-05-28T06:05:17.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.834+0000 7fc757e00640 1 --2- 172.21.15.96:0/4245017799 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc73406c8f0 0x7fc73406edb0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:17.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.834+0000 7fc757e00640 1 --2- 172.21.15.96:0/4245017799 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc7581f7960 0x7fc7581f7de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:17.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.834+0000 7fc757e00640 1 --2- 172.21.15.96:0/4245017799 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc7580d24e0 0x7fc7580d2960 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:17.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.834+0000 7fc757e00640 1 --2- 172.21.15.96:0/4245017799 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc7380039b0 0x7fc758159b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:17.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.834+0000 7fc757e00640 1 -- 172.21.15.96:0/4245017799 >> 172.21.15.96:0/4245017799 conn(0x7fc75808bfc0 msgr2=0x7fc7580b2b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:17.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.835+0000 7fc757e00640 1 -- 172.21.15.96:0/4245017799 shutdown_connections 2024-05-28T06:05:17.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:17.835+0000 7fc757e00640 1 -- 172.21.15.96:0/4245017799 wait complete. 2024-05-28T06:05:17.857 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-05-28T06:05:18.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.054+0000 7ff76be00640 1 Processor -- start 2024-05-28T06:05:18.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.054+0000 7ff76be00640 1 -- start start 2024-05-28T06:05:18.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.055+0000 7ff76be00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff76c0ad940 0x7ff76c0b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:18.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.055+0000 7ff76be00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff76c156f10 con 0x7ff76c06ee40 2024-05-28T06:05:18.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.055+0000 7ff76be00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff76c06dda0 con 0x7ff76c156b30 2024-05-28T06:05:18.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.055+0000 7ff76be00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ff76c06e080 con 0x7ff76c0ad940 2024-05-28T06:05:18.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.056+0000 7ff762000640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7ff76c06ee40 0x7ff76c0ad230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:34898/0 (socket says 172.21.15.96:34898) 2024-05-28T06:05:18.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.056+0000 7ff762000640 1 -- 172.21.15.96:0/620316418 learned_addr learned my addr 172.21.15.96:0/620316418 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:05:18.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.056+0000 7ff76b400640 1 --2- 172.21.15.96:0/620316418 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff76c0ad940 0x7ff76c0b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:18.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.056+0000 7ff76a000640 1 -- 172.21.15.96:0/620316418 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3042205086 0 0) 0x7ff76c156f10 con 0x7ff76c06ee40 2024-05-28T06:05:18.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.056+0000 7ff76a000640 1 -- 172.21.15.96:0/620316418 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff750003520 con 0x7ff76c06ee40 2024-05-28T06:05:18.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.057+0000 7ff76a000640 1 -- 172.21.15.96:0/620316418 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3116745841 0 0) 0x7ff750003520 con 0x7ff76c06ee40 2024-05-28T06:05:18.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.057+0000 7ff76a000640 1 -- 172.21.15.96:0/620316418 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff76c0ad940 msgr2=0x7ff76c0b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:18.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.057+0000 7ff76a000640 1 --2- 172.21.15.96:0/620316418 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff76c0ad940 0x7ff76c0b32c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:18.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.057+0000 7ff76a000640 1 -- 172.21.15.96:0/620316418 >> v1:172.21.15.123:6789/0 conn(0x7ff76c156b30 legacy=0x7ff76c0af6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:18.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.057+0000 7ff76a000640 1 -- 172.21.15.96:0/620316418 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff76c06e360 con 0x7ff76c06ee40 2024-05-28T06:05:18.061 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.057+0000 7ff76a000640 1 -- 172.21.15.96:0/620316418 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7ff754001670 con 0x7ff76c06ee40 2024-05-28T06:05:18.061 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.057+0000 7ff76a000640 1 -- 172.21.15.96:0/620316418 >> v1:172.21.15.96:6789/0 conn(0x7ff76c06ee40 legacy=0x7ff76c0ad230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:18.061 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.057+0000 7ff76a000640 1 --2- 172.21.15.96:0/620316418 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff7500039e0 0x7ff750003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:18.061 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.057+0000 7ff76a000640 1 --2- 172.21.15.96:0/620316418 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff750004800 0x7ff750004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:18.061 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.057+0000 7ff76a000640 1 --2- 172.21.15.96:0/620316418 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff76c0ad940 0x7ff750005270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:18.061 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.057+0000 7ff76a000640 1 -- 172.21.15.96:0/620316418 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff76c06e080 con 0x7ff750004800 2024-05-28T06:05:18.061 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.057+0000 7ff76a000640 1 -- 172.21.15.96:0/620316418 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff75000bb80 con 0x7ff7500039e0 2024-05-28T06:05:18.062 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.057+0000 7ff76a000640 1 -- 172.21.15.96:0/620316418 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ff75000bd80 con 0x7ff76c0ad940 2024-05-28T06:05:18.062 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.057+0000 7ff762000640 1 --2- 172.21.15.96:0/620316418 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff750004800 0x7ff750004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:18.062 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.057+0000 7ff76aa00640 1 --2- 172.21.15.96:0/620316418 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff7500039e0 0x7ff750003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:18.062 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.057+0000 7ff76b400640 1 --2- 172.21.15.96:0/620316418 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff76c0ad940 0x7ff750005270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:18.062 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.057+0000 7ff762000640 1 -- 172.21.15.96:0/620316418 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff76c0ad940 msgr2=0x7ff750005270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:18.062 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.057+0000 7ff762000640 1 --2- 172.21.15.96:0/620316418 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff76c0ad940 0x7ff750005270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:18.062 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.057+0000 7ff762000640 1 -- 172.21.15.96:0/620316418 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff7500039e0 msgr2=0x7ff750003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:18.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.057+0000 7ff762000640 1 --2- 172.21.15.96:0/620316418 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff7500039e0 0x7ff750003e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:18.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.057+0000 7ff762000640 1 -- 172.21.15.96:0/620316418 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff750003520 con 0x7ff750004800 2024-05-28T06:05:18.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.058+0000 7ff762000640 1 --2- 172.21.15.96:0/620316418 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff750004800 0x7ff750004c80 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7ff75400c110 tx=0x7ff750005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a6d0ff932b409da server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:18.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.059+0000 7ff76a000640 1 -- 172.21.15.96:0/620316418 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff754007c40 con 0x7ff750004800 2024-05-28T06:05:18.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.059+0000 7ff76a000640 1 -- 172.21.15.96:0/620316418 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff754005af0 con 0x7ff750004800 2024-05-28T06:05:18.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.059+0000 7ff76a000640 1 -- 172.21.15.96:0/620316418 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff754007070 con 0x7ff750004800 2024-05-28T06:05:18.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.059+0000 7ff76be00640 1 -- 172.21.15.96:0/620316418 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff750004800 msgr2=0x7ff750004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:18.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.059+0000 7ff76be00640 1 --2- 172.21.15.96:0/620316418 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff750004800 0x7ff750004c80 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7ff75400c110 tx=0x7ff750005830 comp rx=0 tx=0).stop 2024-05-28T06:05:18.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.059+0000 7ff76b400640 1 -- 172.21.15.96:0/620316418 reap_dead start 2024-05-28T06:05:18.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.059+0000 7ff76be00640 1 -- 172.21.15.96:0/620316418 shutdown_connections 2024-05-28T06:05:18.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.059+0000 7ff76be00640 1 -- 172.21.15.96:0/620316418 >> 172.21.15.96:0/620316418 conn(0x7ff76c0b1fc0 msgr2=0x7ff76c0b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:18.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.059+0000 7ff76be00640 1 -- 172.21.15.96:0/620316418 shutdown_connections 2024-05-28T06:05:18.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.059+0000 7ff76be00640 1 -- 172.21.15.96:0/620316418 wait complete. 2024-05-28T06:05:18.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.061+0000 7ff76be00640 1 Processor -- start 2024-05-28T06:05:18.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.061+0000 7ff76be00640 1 -- start start 2024-05-28T06:05:18.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.062+0000 7ff76be00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff750004800 0x7ff76c099910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:18.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.062+0000 7ff76be00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff76c141980 0x7ff76c13ff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:18.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.062+0000 7ff76be00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff76c140440 0x7ff76c0b4940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:18.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.062+0000 7ff76be00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff76c0b2db0 con 0x7ff76c140440 2024-05-28T06:05:18.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.062+0000 7ff76be00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff76c0b2c30 con 0x7ff76c141980 2024-05-28T06:05:18.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.062+0000 7ff76be00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ff76c06f2f0 con 0x7ff750004800 2024-05-28T06:05:18.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.062+0000 7ff76b400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff76c140440 0x7ff76c0b4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:18.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.062+0000 7ff76b400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff76c140440 0x7ff76c0b4940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51552/0 (socket says 172.21.15.96:51552) 2024-05-28T06:05:18.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.062+0000 7ff76b400640 1 -- 172.21.15.96:0/1184721799 learned_addr learned my addr 172.21.15.96:0/1184721799 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:18.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.063+0000 7ff76b400640 1 -- 172.21.15.96:0/1184721799 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff750004800 msgr2=0x7ff76c099910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:18.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.063+0000 7ff76b400640 1 --2- 172.21.15.96:0/1184721799 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff750004800 0x7ff76c099910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:18.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.063+0000 7ff76b400640 1 -- 172.21.15.96:0/1184721799 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff76c141980 msgr2=0x7ff76c13ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:18.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.063+0000 7ff762000640 1 --2- 172.21.15.96:0/1184721799 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff76c141980 0x7ff76c13ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:18.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.063+0000 7ff76b400640 1 --2- 172.21.15.96:0/1184721799 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff76c141980 0x7ff76c13ff00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:18.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.063+0000 7ff76b400640 1 -- 172.21.15.96:0/1184721799 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff76c099e50 con 0x7ff76c140440 2024-05-28T06:05:18.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.063+0000 7ff76b400640 1 --2- 172.21.15.96:0/1184721799 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff76c140440 0x7ff76c0b4940 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7ff764009f40 tx=0x7ff76400f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:18.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.063+0000 7ff763400640 1 -- 172.21.15.96:0/1184721799 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff76401c070 con 0x7ff76c140440 2024-05-28T06:05:18.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.063+0000 7ff763400640 1 -- 172.21.15.96:0/1184721799 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff764017400 con 0x7ff76c140440 2024-05-28T06:05:18.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.063+0000 7ff763400640 1 -- 172.21.15.96:0/1184721799 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7640177f0 con 0x7ff76c140440 2024-05-28T06:05:18.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.063+0000 7ff76be00640 1 -- 172.21.15.96:0/1184721799 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff76c1408e0 con 0x7ff76c140440 2024-05-28T06:05:18.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.064+0000 7ff76be00640 1 -- 172.21.15.96:0/1184721799 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff76c140ea0 con 0x7ff76c140440 2024-05-28T06:05:18.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.065+0000 7ff76be00640 1 -- 172.21.15.96:0/1184721799 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff76c15d850 con 0x7ff76c140440 2024-05-28T06:05:18.071 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.066+0000 7ff763400640 1 -- 172.21.15.96:0/1184721799 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff76401f070 con 0x7ff76c140440 2024-05-28T06:05:18.071 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.066+0000 7ff763400640 1 --2- 172.21.15.96:0/1184721799 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff74806c830 0x7ff74806ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:18.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.067+0000 7ff763400640 1 -- 172.21.15.96:0/1184721799 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5825+0+0 (secure 0 0 0) 0x7ff76408bb40 con 0x7ff76c140440 2024-05-28T06:05:18.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.067+0000 7ff76aa00640 1 --2- 172.21.15.96:0/1184721799 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff74806c830 0x7ff74806ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:18.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.068+0000 7ff76aa00640 1 --2- 172.21.15.96:0/1184721799 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff74806c830 0x7ff74806ecf0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7ff75c007fd0 tx=0x7ff75c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:18.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.071+0000 7ff763400640 1 -- 172.21.15.96:0/1184721799 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff76c15d850 con 0x7ff76c140440 2024-05-28T06:05:18.306 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.305+0000 7ff76be00640 1 -- 172.21.15.96:0/1184721799 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7ff76c06f7f0 con 0x7ff76c140440 2024-05-28T06:05:18.308 INFO:tasks.ceph.mon.a.smithi096.stderr:2024-05-28T06:05:18.306+0000 7fb2fca00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-28T06:05:18.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.309+0000 7ff763400640 1 -- 172.21.15.96:0/1184721799 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v8) ==== 161+0+0 (secure 0 0 0) 0x7ff764012070 con 0x7ff76c140440 2024-05-28T06:05:18.311 INFO:teuthology.orchestra.run.smithi096.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-28T06:05:18.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.312+0000 7ff76be00640 1 -- 172.21.15.96:0/1184721799 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff74806c830 msgr2=0x7ff74806ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:18.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.312+0000 7ff76be00640 1 --2- 172.21.15.96:0/1184721799 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff74806c830 0x7ff74806ecf0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7ff75c007fd0 tx=0x7ff75c009040 comp rx=0 tx=0).stop 2024-05-28T06:05:18.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.312+0000 7ff76be00640 1 -- 172.21.15.96:0/1184721799 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff76c140440 msgr2=0x7ff76c0b4940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:18.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.312+0000 7ff76be00640 1 --2- 172.21.15.96:0/1184721799 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff76c140440 0x7ff76c0b4940 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7ff764009f40 tx=0x7ff76400f4a0 comp rx=0 tx=0).stop 2024-05-28T06:05:18.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.312+0000 7ff76be00640 1 -- 172.21.15.96:0/1184721799 shutdown_connections 2024-05-28T06:05:18.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.312+0000 7ff76be00640 1 --2- 172.21.15.96:0/1184721799 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff74806c830 0x7ff74806ecf0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:18.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.312+0000 7ff76be00640 1 --2- 172.21.15.96:0/1184721799 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff76c140440 0x7ff76c0b4940 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:18.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.312+0000 7ff76be00640 1 --2- 172.21.15.96:0/1184721799 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff76c141980 0x7ff76c13ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:18.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.312+0000 7ff76be00640 1 --2- 172.21.15.96:0/1184721799 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff750004800 0x7ff76c099910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:18.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.312+0000 7ff76be00640 1 -- 172.21.15.96:0/1184721799 >> 172.21.15.96:0/1184721799 conn(0x7ff76c0b1fc0 msgr2=0x7ff76c15afd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:18.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.312+0000 7ff76be00640 1 -- 172.21.15.96:0/1184721799 shutdown_connections 2024-05-28T06:05:18.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.312+0000 7ff76be00640 1 -- 172.21.15.96:0/1184721799 wait complete. 2024-05-28T06:05:18.333 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-05-28T06:05:18.340 INFO:tasks.ceph.mds.b.smithi123.stderr: -4231> 2024-05-28T06:04:11.823+0000 7fda5f400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-28T06:05:18.437 INFO:tasks.ceph.mds.b.smithi123.stdout:starting mds.b at 2024-05-28T06:05:18.533 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.531+0000 7f1cd9800640 1 Processor -- start 2024-05-28T06:05:18.533 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.531+0000 7f1cd9800640 1 -- start start 2024-05-28T06:05:18.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.532+0000 7f1cd9800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f1cd40d0640 0x7f1cd415d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:18.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.532+0000 7f1cd9800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1cd40d0a20 0x7f1cd40cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:18.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.533+0000 7f1cd9800640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1cd415cb10 con 0x7f1cd40b32c0 2024-05-28T06:05:18.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.533+0000 7f1cd9800640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f1cd40adbb0 con 0x7f1cd40d0640 2024-05-28T06:05:18.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.533+0000 7f1cd9800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f1cd415db10 con 0x7f1cd40d0a20 2024-05-28T06:05:18.536 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.533+0000 7f1cd3400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f1cd40b32c0 0x7f1cd40b3750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:34914/0 (socket says 172.21.15.96:34914) 2024-05-28T06:05:18.536 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.533+0000 7f1cd3400640 1 -- 172.21.15.96:0/4062955417 learned_addr learned my addr 172.21.15.96:0/4062955417 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:05:18.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.533+0000 7f1cd2a00640 1 --2- 172.21.15.96:0/4062955417 >> v2:172.21.15.123:3300/0 conn(0x7f1cd40d0640 0x7f1cd415d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:18.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.533+0000 7f1cd1600640 1 -- 172.21.15.96:0/4062955417 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2751142633 0 0) 0x7f1cd415cb10 con 0x7f1cd40b32c0 2024-05-28T06:05:18.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.533+0000 7f1cd1600640 1 -- 172.21.15.96:0/4062955417 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1cb8003520 con 0x7f1cd40b32c0 2024-05-28T06:05:18.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.533+0000 7f1cd2000640 1 --2- 172.21.15.96:0/4062955417 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1cd40d0a20 0x7f1cd40cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:18.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.534+0000 7f1cd1600640 1 -- 172.21.15.96:0/4062955417 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2871072240 0 0) 0x7f1cb8003520 con 0x7f1cd40b32c0 2024-05-28T06:05:18.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.534+0000 7f1cd1600640 1 -- 172.21.15.96:0/4062955417 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1cd40d0a20 msgr2=0x7f1cd40cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:18.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.534+0000 7f1cd1600640 1 --2- 172.21.15.96:0/4062955417 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1cd40d0a20 0x7f1cd40cf950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:18.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.534+0000 7f1cd1600640 1 -- 172.21.15.96:0/4062955417 >> v2:172.21.15.123:3300/0 conn(0x7f1cd40d0640 msgr2=0x7f1cd415d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:18.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.534+0000 7f1cd1600640 1 --2- 172.21.15.96:0/4062955417 >> v2:172.21.15.123:3300/0 conn(0x7f1cd40d0640 0x7f1cd415d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:18.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.534+0000 7f1cd1600640 1 -- 172.21.15.96:0/4062955417 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cd40ad5f0 con 0x7f1cd40b32c0 2024-05-28T06:05:18.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.534+0000 7f1cd1600640 1 -- 172.21.15.96:0/4062955417 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f1ccc0019f0 con 0x7f1cd40b32c0 2024-05-28T06:05:18.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.534+0000 7f1cd1600640 1 -- 172.21.15.96:0/4062955417 >> v1:172.21.15.96:6789/0 conn(0x7f1cd40b32c0 legacy=0x7f1cd40b3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:18.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.534+0000 7f1cd1600640 1 --2- 172.21.15.96:0/4062955417 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1cb80039b0 0x7f1cb8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:18.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.534+0000 7f1cd1600640 1 --2- 172.21.15.96:0/4062955417 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1cb8004800 0x7f1cb8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:18.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.534+0000 7f1cd1600640 1 --2- 172.21.15.96:0/4062955417 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1cb80051c0 0x7f1cb800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:18.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.534+0000 7f1cd1600640 1 -- 172.21.15.96:0/4062955417 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f1cd415db10 con 0x7f1cb80051c0 2024-05-28T06:05:18.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.534+0000 7f1cd1600640 1 -- 172.21.15.96:0/4062955417 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f1cb800bfb0 con 0x7f1cb8004800 2024-05-28T06:05:18.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.534+0000 7f1cd1600640 1 -- 172.21.15.96:0/4062955417 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f1cb800c180 con 0x7f1cb80039b0 2024-05-28T06:05:18.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.534+0000 7f1cd2a00640 1 --2- 172.21.15.96:0/4062955417 >> v2:172.21.15.123:3300/0 conn(0x7f1cd40d0640 0x7f1cd415d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-28T06:05:18.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.534+0000 7f1cd3400640 1 --2- 172.21.15.96:0/4062955417 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1cb80051c0 0x7f1cb800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:18.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.534+0000 7f1cd2000640 1 --2- 172.21.15.96:0/4062955417 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1cb8004800 0x7f1cb8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:18.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.535+0000 7f1cd2a00640 1 --2- 172.21.15.96:0/4062955417 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1cb80039b0 0x7f1cb8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:18.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.535+0000 7f1cd3400640 1 -- 172.21.15.96:0/4062955417 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1cb80039b0 msgr2=0x7f1cb8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:18.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.535+0000 7f1cd3400640 1 --2- 172.21.15.96:0/4062955417 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1cb80039b0 0x7f1cb8003dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:18.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.535+0000 7f1cd3400640 1 -- 172.21.15.96:0/4062955417 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1cb8004800 msgr2=0x7f1cb8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:18.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.535+0000 7f1cd3400640 1 --2- 172.21.15.96:0/4062955417 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1cb8004800 0x7f1cb8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:18.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.535+0000 7f1cd3400640 1 -- 172.21.15.96:0/4062955417 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cb8003520 con 0x7f1cb80051c0 2024-05-28T06:05:18.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.536+0000 7f1cd3400640 1 --2- 172.21.15.96:0/4062955417 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1cb80051c0 0x7f1cb800b9c0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f1ccc00a950 tx=0x7f1cb8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca230d4addf29883 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:18.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.536+0000 7f1cd1600640 1 -- 172.21.15.96:0/4062955417 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ccc00a4c0 con 0x7f1cb80051c0 2024-05-28T06:05:18.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.536+0000 7f1cd1600640 1 -- 172.21.15.96:0/4062955417 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ccc02f070 con 0x7f1cb80051c0 2024-05-28T06:05:18.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.536+0000 7f1cd1600640 1 -- 172.21.15.96:0/4062955417 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ccc02bd40 con 0x7f1cb80051c0 2024-05-28T06:05:18.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.536+0000 7f1cd9800640 1 -- 172.21.15.96:0/4062955417 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1cb80051c0 msgr2=0x7f1cb800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:18.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.536+0000 7f1cd9800640 1 --2- 172.21.15.96:0/4062955417 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1cb80051c0 0x7f1cb800b9c0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f1ccc00a950 tx=0x7f1cb8005830 comp rx=0 tx=0).stop 2024-05-28T06:05:18.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.536+0000 7f1cd3400640 1 -- 172.21.15.96:0/4062955417 reap_dead start 2024-05-28T06:05:18.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.537+0000 7f1cd9800640 1 -- 172.21.15.96:0/4062955417 shutdown_connections 2024-05-28T06:05:18.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.537+0000 7f1cd9800640 1 -- 172.21.15.96:0/4062955417 >> 172.21.15.96:0/4062955417 conn(0x7f1cd406bd20 msgr2=0x7f1cd406c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:18.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.537+0000 7f1cd9800640 1 -- 172.21.15.96:0/4062955417 shutdown_connections 2024-05-28T06:05:18.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.537+0000 7f1cd9800640 1 -- 172.21.15.96:0/4062955417 wait complete. 2024-05-28T06:05:18.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.538+0000 7f1cd9800640 1 Processor -- start 2024-05-28T06:05:18.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.538+0000 7f1cd9800640 1 -- start start 2024-05-28T06:05:18.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.539+0000 7f1cd9800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1cb80051c0 0x7f1cd4156030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:18.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.539+0000 7f1cd9800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1cd414daf0 0x7f1cd41465c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:18.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.539+0000 7f1cd9800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1cd40b1220 0x7f1cd40b16a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:18.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.539+0000 7f1cd9800640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f1cd415d0b0 con 0x7f1cb80051c0 2024-05-28T06:05:18.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.539+0000 7f1cd9800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f1cd415cdd0 con 0x7f1cd414daf0 2024-05-28T06:05:18.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.539+0000 7f1cd9800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f1cd40b3e60 con 0x7f1cd40b1220 2024-05-28T06:05:18.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.539+0000 7f1cd2a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1cb80051c0 0x7f1cd4156030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:18.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.539+0000 7f1cd2a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1cb80051c0 0x7f1cd4156030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51572/0 (socket says 172.21.15.96:51572) 2024-05-28T06:05:18.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.540+0000 7f1cd2a00640 1 -- 172.21.15.96:0/1271870272 learned_addr learned my addr 172.21.15.96:0/1271870272 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:18.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.540+0000 7f1cd2a00640 1 -- 172.21.15.96:0/1271870272 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1cd40b1220 msgr2=0x7f1cd40b16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:18.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.540+0000 7f1cd2a00640 1 --2- 172.21.15.96:0/1271870272 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1cd40b1220 0x7f1cd40b16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:18.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.540+0000 7f1cd2a00640 1 -- 172.21.15.96:0/1271870272 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1cd414daf0 msgr2=0x7f1cd41465c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:18.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.540+0000 7f1cd2a00640 1 --2- 172.21.15.96:0/1271870272 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1cd414daf0 0x7f1cd41465c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:18.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.540+0000 7f1cd2a00640 1 -- 172.21.15.96:0/1271870272 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cd414ed00 con 0x7f1cb80051c0 2024-05-28T06:05:18.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.540+0000 7f1cd2a00640 1 --2- 172.21.15.96:0/1271870272 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1cb80051c0 0x7f1cd4156030 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f1cc400f570 tx=0x7f1cc400fa30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:18.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.540+0000 7f1ccaa00640 1 -- 172.21.15.96:0/1271870272 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cc4018070 con 0x7f1cb80051c0 2024-05-28T06:05:18.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.540+0000 7f1ccaa00640 1 -- 172.21.15.96:0/1271870272 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1cc4014890 con 0x7f1cb80051c0 2024-05-28T06:05:18.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.540+0000 7f1ccaa00640 1 -- 172.21.15.96:0/1271870272 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cc4014c80 con 0x7f1cb80051c0 2024-05-28T06:05:18.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.540+0000 7f1cd9800640 1 -- 172.21.15.96:0/1271870272 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1cd414e180 con 0x7f1cb80051c0 2024-05-28T06:05:18.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.541+0000 7f1cd9800640 1 -- 172.21.15.96:0/1271870272 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1cd414e740 con 0x7f1cb80051c0 2024-05-28T06:05:18.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.542+0000 7f1cd9800640 1 -- 172.21.15.96:0/1271870272 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1cd40689e0 con 0x7f1cb80051c0 2024-05-28T06:05:18.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.543+0000 7f1ccaa00640 1 -- 172.21.15.96:0/1271870272 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1cc401b070 con 0x7f1cb80051c0 2024-05-28T06:05:18.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.543+0000 7f1ccaa00640 1 --2- 172.21.15.96:0/1271870272 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1cb406c8f0 0x7f1cb406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:18.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.544+0000 7f1ccaa00640 1 -- 172.21.15.96:0/1271870272 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5911+0+0 (secure 0 0 0) 0x7f1cc408bf30 con 0x7f1cb80051c0 2024-05-28T06:05:18.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.544+0000 7f1cd2000640 1 --2- 172.21.15.96:0/1271870272 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1cb406c8f0 0x7f1cb406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:18.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.545+0000 7f1cd2000640 1 --2- 172.21.15.96:0/1271870272 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1cb406c8f0 0x7f1cb406edb0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f1cc0006a90 tx=0x7f1cc0006a20 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:18.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.548+0000 7f1ccaa00640 1 -- 172.21.15.96:0/1271870272 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1cc405a280 con 0x7f1cb80051c0 2024-05-28T06:05:18.774 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:18.772+0000 7f1cd9800640 1 -- 172.21.15.96:0/1271870272 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f1cd4071ef0 con 0x7f1cb80051c0 2024-05-28T06:05:19.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:19.313+0000 7f1ccaa00640 1 -- 172.21.15.96:0/1271870272 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v9) ==== 104+0+0 (secure 0 0 0) 0x7f1cc4014e20 con 0x7f1cb80051c0 2024-05-28T06:05:19.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:19.318+0000 7f1cd9800640 1 -- 172.21.15.96:0/1271870272 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1cb406c8f0 msgr2=0x7f1cb406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:19.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:19.318+0000 7f1cd9800640 1 --2- 172.21.15.96:0/1271870272 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1cb406c8f0 0x7f1cb406edb0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f1cc0006a90 tx=0x7f1cc0006a20 comp rx=0 tx=0).stop 2024-05-28T06:05:19.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:19.318+0000 7f1cd9800640 1 -- 172.21.15.96:0/1271870272 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1cb80051c0 msgr2=0x7f1cd4156030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:19.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:19.318+0000 7f1cd9800640 1 --2- 172.21.15.96:0/1271870272 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1cb80051c0 0x7f1cd4156030 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f1cc400f570 tx=0x7f1cc400fa30 comp rx=0 tx=0).stop 2024-05-28T06:05:19.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:19.318+0000 7f1cd9800640 1 -- 172.21.15.96:0/1271870272 shutdown_connections 2024-05-28T06:05:19.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:19.318+0000 7f1cd9800640 1 --2- 172.21.15.96:0/1271870272 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1cb406c8f0 0x7f1cb406edb0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:19.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:19.318+0000 7f1cd9800640 1 --2- 172.21.15.96:0/1271870272 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1cd40b1220 0x7f1cd40b16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:19.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:19.318+0000 7f1cd9800640 1 --2- 172.21.15.96:0/1271870272 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1cd414daf0 0x7f1cd41465c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:19.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:19.318+0000 7f1cd9800640 1 --2- 172.21.15.96:0/1271870272 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1cb80051c0 0x7f1cd4156030 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:19.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:19.318+0000 7f1cd9800640 1 -- 172.21.15.96:0/1271870272 >> 172.21.15.96:0/1271870272 conn(0x7f1cd406bd20 msgr2=0x7f1cd415b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:19.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:19.318+0000 7f1cd9800640 1 -- 172.21.15.96:0/1271870272 shutdown_connections 2024-05-28T06:05:19.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:19.318+0000 7f1cd9800640 1 -- 172.21.15.96:0/1271870272 wait complete. 2024-05-28T06:05:19.343 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-05-28T06:05:19.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:19.554+0000 7f4412600640 1 Processor -- start 2024-05-28T06:05:19.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:19.555+0000 7f4412600640 1 -- start start 2024-05-28T06:05:19.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:19.556+0000 7f4412600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f440c0cfb40 0x7f440c0d0d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:19.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:19.556+0000 7f4412600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f440c0cffa0 0x7f440c0d1250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:19.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:19.556+0000 7f4412600640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f440c0d2200 con 0x7f440c15e5a0 2024-05-28T06:05:19.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:19.556+0000 7f4412600640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f440c0d2b30 con 0x7f440c0cffa0 2024-05-28T06:05:19.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:19.556+0000 7f4412600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f440c0d24e0 con 0x7f440c0cfb40 2024-05-28T06:05:19.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:19.556+0000 7f4411c00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f440c0cffa0 0x7f440c0d1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:19.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:19.556+0000 7f4411c00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f440c0cffa0 0x7f440c0d1250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49384/0 (socket says 172.21.15.96:49384) 2024-05-28T06:05:19.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:19.556+0000 7f4411c00640 1 -- 172.21.15.96:0/522541907 learned_addr learned my addr 172.21.15.96:0/522541907 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:19.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:19.557+0000 7f4403e00640 1 --2- 172.21.15.96:0/522541907 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f440c0cfb40 0x7f440c0d0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:19.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:19.557+0000 7f4411c00640 1 -- 172.21.15.96:0/522541907 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f440c0cfb40 msgr2=0x7f440c0d0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:19.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:19.557+0000 7f4411c00640 1 --2- 172.21.15.96:0/522541907 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f440c0cfb40 0x7f440c0d0d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:19.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:19.557+0000 7f4411c00640 1 -- 172.21.15.96:0/522541907 >> v1:172.21.15.96:6789/0 conn(0x7f440c15e5a0 legacy=0x7f440c0d2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:19.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:19.557+0000 7f4411c00640 1 -- 172.21.15.96:0/522541907 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f440c0d27c0 con 0x7f440c0cffa0 2024-05-28T06:05:19.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:19.558+0000 7f4411c00640 1 --2- 172.21.15.96:0/522541907 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f440c0cffa0 0x7f440c0d1250 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f4404009870 tx=0x7f440402a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c14ea88e2100b9d2 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:19.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:19.558+0000 7f440be00640 1 -- 172.21.15.96:0/522541907 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4404037070 con 0x7f440c0cffa0 2024-05-28T06:05:19.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:19.558+0000 7f440be00640 1 -- 172.21.15.96:0/522541907 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4404032440 con 0x7f440c0cffa0 2024-05-28T06:05:19.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:19.558+0000 7f440be00640 1 -- 172.21.15.96:0/522541907 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4404032810 con 0x7f440c0cffa0 2024-05-28T06:05:19.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:19.559+0000 7f4412600640 1 -- 172.21.15.96:0/522541907 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f440c0cffa0 msgr2=0x7f440c0d1250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:19.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:19.559+0000 7f4412600640 1 --2- 172.21.15.96:0/522541907 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f440c0cffa0 0x7f440c0d1250 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f4404009870 tx=0x7f440402a250 comp rx=0 tx=0).stop 2024-05-28T06:05:19.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:19.559+0000 7f4412600640 1 -- 172.21.15.96:0/522541907 shutdown_connections 2024-05-28T06:05:19.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:19.559+0000 7f4412600640 1 --2- 172.21.15.96:0/522541907 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f440c0cffa0 0x7f440c0d1250 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:19.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:19.559+0000 7f4412600640 1 --2- 172.21.15.96:0/522541907 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f440c0cfb40 0x7f440c0d0d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:19.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:19.559+0000 7f4412600640 1 -- 172.21.15.96:0/522541907 >> 172.21.15.96:0/522541907 conn(0x7f440c08bfc0 msgr2=0x7f440c0d44b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:19.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:19.559+0000 7f4412600640 1 -- 172.21.15.96:0/522541907 shutdown_connections 2024-05-28T06:05:19.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:19.559+0000 7f4412600640 1 -- 172.21.15.96:0/522541907 wait complete. 2024-05-28T06:05:19.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:19.561+0000 7f4412600640 1 Processor -- start 2024-05-28T06:05:19.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:19.561+0000 7f4412600640 1 -- start start 2024-05-28T06:05:19.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:19.562+0000 7f4412600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f440c0cfb40 0x7f440c161700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:19.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:19.562+0000 7f4412600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f440c0cffa0 0x7f440c0ca840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:19.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:19.562+0000 7f4412600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f440c15e5a0 0x7f440c0b8a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:19.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:19.562+0000 7f4412600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f440c0d53a0 con 0x7f440c0cffa0 2024-05-28T06:05:19.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:19.562+0000 7f4412600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f440c0d5220 con 0x7f440c0cfb40 2024-05-28T06:05:19.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:19.562+0000 7f4412600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f440c06c740 con 0x7f440c15e5a0 2024-05-28T06:05:19.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:19.562+0000 7f4403e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f440c0cffa0 0x7f440c0ca840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:19.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:19.562+0000 7f4403e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f440c0cffa0 0x7f440c0ca840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49388/0 (socket says 172.21.15.96:49388) 2024-05-28T06:05:19.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:19.562+0000 7f4403e00640 1 -- 172.21.15.96:0/3195308450 learned_addr learned my addr 172.21.15.96:0/3195308450 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:19.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:19.562+0000 7f4411200640 1 --2- 172.21.15.96:0/3195308450 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f440c0cfb40 0x7f440c161700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:19.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:19.562+0000 7f4403e00640 1 -- 172.21.15.96:0/3195308450 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f440c15e5a0 msgr2=0x7f440c0b8a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:19.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:19.562+0000 7f4411c00640 1 --2- 172.21.15.96:0/3195308450 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f440c15e5a0 0x7f440c0b8a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:19.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:19.562+0000 7f4403e00640 1 --2- 172.21.15.96:0/3195308450 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f440c15e5a0 0x7f440c0b8a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:19.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:19.562+0000 7f4403e00640 1 -- 172.21.15.96:0/3195308450 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f440c0cfb40 msgr2=0x7f440c161700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:19.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:19.562+0000 7f4403e00640 1 --2- 172.21.15.96:0/3195308450 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f440c0cfb40 0x7f440c161700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:19.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:19.563+0000 7f4403e00640 1 -- 172.21.15.96:0/3195308450 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f440c161c40 con 0x7f440c0cffa0 2024-05-28T06:05:19.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:19.563+0000 7f4403e00640 1 --2- 172.21.15.96:0/3195308450 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f440c0cffa0 0x7f440c0ca840 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f43f400d800 tx=0x7f43f400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:19.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:19.563+0000 7f4409600640 1 -- 172.21.15.96:0/3195308450 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43f400b6c0 con 0x7f440c0cffa0 2024-05-28T06:05:19.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:19.563+0000 7f4409600640 1 -- 172.21.15.96:0/3195308450 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43f4019070 con 0x7f440c0cffa0 2024-05-28T06:05:19.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:19.563+0000 7f4409600640 1 -- 172.21.15.96:0/3195308450 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43f400ba80 con 0x7f440c0cffa0 2024-05-28T06:05:19.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:19.563+0000 7f4412600640 1 -- 172.21.15.96:0/3195308450 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f440c0c95e0 con 0x7f440c0cffa0 2024-05-28T06:05:19.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:19.563+0000 7f4412600640 1 -- 172.21.15.96:0/3195308450 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f440c0cbc10 con 0x7f440c0cffa0 2024-05-28T06:05:19.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:19.564+0000 7f4412600640 1 -- 172.21.15.96:0/3195308450 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f440c0689e0 con 0x7f440c0cffa0 2024-05-28T06:05:19.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:19.565+0000 7f4409600640 1 -- 172.21.15.96:0/3195308450 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f43f400bc20 con 0x7f440c0cffa0 2024-05-28T06:05:19.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:19.566+0000 7f4409600640 1 --2- 172.21.15.96:0/3195308450 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f43e806c8f0 0x7f43e806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:19.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:19.566+0000 7f4409600640 1 -- 172.21.15.96:0/3195308450 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5911+0+0 (secure 0 0 0) 0x7f43f408c0c0 con 0x7f440c0cffa0 2024-05-28T06:05:19.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:19.567+0000 7f4411200640 1 --2- 172.21.15.96:0/3195308450 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f43e806c8f0 0x7f43e806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:19.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:19.568+0000 7f4411200640 1 --2- 172.21.15.96:0/3195308450 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f43e806c8f0 0x7f43e806edb0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f43fc007fd0 tx=0x7f43fc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:19.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:19.570+0000 7f4409600640 1 -- 172.21.15.96:0/3195308450 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f43f405a410 con 0x7f440c0cffa0 2024-05-28T06:05:19.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:19.771+0000 7f4412600640 1 -- 172.21.15.96:0/3195308450 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f440c0d1d30 con 0x7f440c0cffa0 2024-05-28T06:05:20.325 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.323+0000 7f4409600640 1 -- 172.21.15.96:0/3195308450 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v24) ==== 181+0+0 (secure 0 0 0) 0x7f43f4059d10 con 0x7f440c0cffa0 2024-05-28T06:05:20.325 INFO:teuthology.orchestra.run.smithi096.stderr:pool 'cephfs_metadata' removed 2024-05-28T06:05:20.328 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.327+0000 7f4412600640 1 -- 172.21.15.96:0/3195308450 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f43e806c8f0 msgr2=0x7f43e806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:20.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.327+0000 7f4412600640 1 --2- 172.21.15.96:0/3195308450 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f43e806c8f0 0x7f43e806edb0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f43fc007fd0 tx=0x7f43fc009040 comp rx=0 tx=0).stop 2024-05-28T06:05:20.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.327+0000 7f4412600640 1 -- 172.21.15.96:0/3195308450 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f440c0cffa0 msgr2=0x7f440c0ca840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:20.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.327+0000 7f4412600640 1 --2- 172.21.15.96:0/3195308450 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f440c0cffa0 0x7f440c0ca840 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f43f400d800 tx=0x7f43f400dcc0 comp rx=0 tx=0).stop 2024-05-28T06:05:20.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.327+0000 7f4412600640 1 -- 172.21.15.96:0/3195308450 shutdown_connections 2024-05-28T06:05:20.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.327+0000 7f4412600640 1 --2- 172.21.15.96:0/3195308450 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f43e806c8f0 0x7f43e806edb0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:20.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.327+0000 7f4412600640 1 --2- 172.21.15.96:0/3195308450 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f440c15e5a0 0x7f440c0b8a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:20.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.327+0000 7f4412600640 1 --2- 172.21.15.96:0/3195308450 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f440c0cffa0 0x7f440c0ca840 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:20.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.327+0000 7f4412600640 1 --2- 172.21.15.96:0/3195308450 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f440c0cfb40 0x7f440c161700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:20.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.327+0000 7f4412600640 1 -- 172.21.15.96:0/3195308450 >> 172.21.15.96:0/3195308450 conn(0x7f440c08bfc0 msgr2=0x7f440c0b3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:20.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.327+0000 7f4412600640 1 -- 172.21.15.96:0/3195308450 shutdown_connections 2024-05-28T06:05:20.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.327+0000 7f4412600640 1 -- 172.21.15.96:0/3195308450 wait complete. 2024-05-28T06:05:20.352 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-05-28T06:05:20.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.554+0000 7f6c42c00640 1 Processor -- start 2024-05-28T06:05:20.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.555+0000 7f6c42c00640 1 -- start start 2024-05-28T06:05:20.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.556+0000 7f6c42c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6c3c15c7a0 0x7f6c3c0d0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:20.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.556+0000 7f6c42c00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f6c3c09b140 0x7f6c3c0b7620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:20.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.556+0000 7f6c42c00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c3c0cd240 con 0x7f6c3c09b770 2024-05-28T06:05:20.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.556+0000 7f6c42c00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f6c3c0cde50 con 0x7f6c3c09b140 2024-05-28T06:05:20.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.556+0000 7f6c42c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f6c3c0b6360 con 0x7f6c3c15c7a0 2024-05-28T06:05:20.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.557+0000 7f6c33400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f6c3c09b770 0x7f6c3c09aa30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:48320/0 (socket says 172.21.15.96:48320) 2024-05-28T06:05:20.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.557+0000 7f6c33400640 1 -- 172.21.15.96:0/2011534562 learned_addr learned my addr 172.21.15.96:0/2011534562 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:05:20.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.557+0000 7f6c3be00640 1 --2- 172.21.15.96:0/2011534562 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6c3c15c7a0 0x7f6c3c0d0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:20.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.557+0000 7f6c40c00640 1 --2- 172.21.15.96:0/2011534562 >> v2:172.21.15.123:3300/0 conn(0x7f6c3c09b140 0x7f6c3c0b7620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:20.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.557+0000 7f6c3b400640 1 -- 172.21.15.96:0/2011534562 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2505842373 0 0) 0x7f6c3c0cd240 con 0x7f6c3c09b770 2024-05-28T06:05:20.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.557+0000 7f6c3b400640 1 -- 172.21.15.96:0/2011534562 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6c20003520 con 0x7f6c3c09b770 2024-05-28T06:05:20.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.557+0000 7f6c3b400640 1 -- 172.21.15.96:0/2011534562 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 855061593 0 0) 0x7f6c20003520 con 0x7f6c3c09b770 2024-05-28T06:05:20.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.557+0000 7f6c3b400640 1 -- 172.21.15.96:0/2011534562 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6c3c15c7a0 msgr2=0x7f6c3c0d0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:20.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.557+0000 7f6c3b400640 1 --2- 172.21.15.96:0/2011534562 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6c3c15c7a0 0x7f6c3c0d0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:20.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.557+0000 7f6c3b400640 1 -- 172.21.15.96:0/2011534562 >> v2:172.21.15.123:3300/0 conn(0x7f6c3c09b140 msgr2=0x7f6c3c0b7620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:20.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.557+0000 7f6c3b400640 1 --2- 172.21.15.96:0/2011534562 >> v2:172.21.15.123:3300/0 conn(0x7f6c3c09b140 0x7f6c3c0b7620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:20.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.557+0000 7f6c3b400640 1 -- 172.21.15.96:0/2011534562 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c3c0cd520 con 0x7f6c3c09b770 2024-05-28T06:05:20.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.557+0000 7f6c3b400640 1 -- 172.21.15.96:0/2011534562 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f6c24001670 con 0x7f6c3c09b770 2024-05-28T06:05:20.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.558+0000 7f6c3b400640 1 -- 172.21.15.96:0/2011534562 >> v1:172.21.15.96:6789/0 conn(0x7f6c3c09b770 legacy=0x7f6c3c09aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:20.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.558+0000 7f6c3b400640 1 --2- 172.21.15.96:0/2011534562 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c200039b0 0x7f6c20003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:20.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.558+0000 7f6c3b400640 1 --2- 172.21.15.96:0/2011534562 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c20004800 0x7f6c20004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:20.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.558+0000 7f6c3b400640 1 --2- 172.21.15.96:0/2011534562 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6c3c15c7a0 0x7f6c200051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:20.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.558+0000 7f6c3b400640 1 -- 172.21.15.96:0/2011534562 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f6c3c0cde50 con 0x7f6c20004800 2024-05-28T06:05:20.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.558+0000 7f6c3b400640 1 -- 172.21.15.96:0/2011534562 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f6c3c0b6360 con 0x7f6c200039b0 2024-05-28T06:05:20.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.558+0000 7f6c3b400640 1 -- 172.21.15.96:0/2011534562 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f6c2000ba10 con 0x7f6c3c15c7a0 2024-05-28T06:05:20.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.558+0000 7f6c33400640 1 --2- 172.21.15.96:0/2011534562 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c20004800 0x7f6c20004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:20.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.558+0000 7f6c33400640 1 -- 172.21.15.96:0/2011534562 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6c3c15c7a0 msgr2=0x7f6c200051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:20.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.558+0000 7f6c33400640 1 --2- 172.21.15.96:0/2011534562 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6c3c15c7a0 0x7f6c200051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:20.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.558+0000 7f6c33400640 1 -- 172.21.15.96:0/2011534562 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c200039b0 msgr2=0x7f6c20003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:20.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.558+0000 7f6c33400640 1 --2- 172.21.15.96:0/2011534562 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c200039b0 0x7f6c20003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:20.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.558+0000 7f6c33400640 1 -- 172.21.15.96:0/2011534562 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c20003520 con 0x7f6c20004800 2024-05-28T06:05:20.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.559+0000 7f6c33400640 1 --2- 172.21.15.96:0/2011534562 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c20004800 0x7f6c20004c80 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f6c2400c1c0 tx=0x7f6c20005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=669680f62b80da95 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:20.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.560+0000 7f6c3b400640 1 -- 172.21.15.96:0/2011534562 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c24007c80 con 0x7f6c20004800 2024-05-28T06:05:20.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.560+0000 7f6c3b400640 1 -- 172.21.15.96:0/2011534562 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c2402e070 con 0x7f6c20004800 2024-05-28T06:05:20.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.560+0000 7f6c3b400640 1 -- 172.21.15.96:0/2011534562 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c24005d30 con 0x7f6c20004800 2024-05-28T06:05:20.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.560+0000 7f6c42c00640 1 -- 172.21.15.96:0/2011534562 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c20004800 msgr2=0x7f6c20004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:20.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.560+0000 7f6c42c00640 1 --2- 172.21.15.96:0/2011534562 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c20004800 0x7f6c20004c80 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f6c2400c1c0 tx=0x7f6c20005830 comp rx=0 tx=0).stop 2024-05-28T06:05:20.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.560+0000 7f6c40c00640 1 -- 172.21.15.96:0/2011534562 reap_dead start 2024-05-28T06:05:20.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.560+0000 7f6c42c00640 1 -- 172.21.15.96:0/2011534562 shutdown_connections 2024-05-28T06:05:20.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.560+0000 7f6c42c00640 1 -- 172.21.15.96:0/2011534562 >> 172.21.15.96:0/2011534562 conn(0x7f6c3c0b3ff0 msgr2=0x7f6c3c0b4400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:20.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.560+0000 7f6c42c00640 1 -- 172.21.15.96:0/2011534562 shutdown_connections 2024-05-28T06:05:20.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.561+0000 7f6c42c00640 1 -- 172.21.15.96:0/2011534562 wait complete. 2024-05-28T06:05:20.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.562+0000 7f6c42c00640 1 Processor -- start 2024-05-28T06:05:20.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.562+0000 7f6c42c00640 1 -- start start 2024-05-28T06:05:20.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.563+0000 7f6c42c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c20004800 0x7f6c3c15bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:20.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.563+0000 7f6c42c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6c3c14e630 0x7f6c3c0c5b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:20.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.563+0000 7f6c42c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c3c1e76b0 0x7f6c3c1e7b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:20.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.563+0000 7f6c42c00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f6c3c15cb80 con 0x7f6c20004800 2024-05-28T06:05:20.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.563+0000 7f6c42c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f6c3c0cdae0 con 0x7f6c3c1e76b0 2024-05-28T06:05:20.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.563+0000 7f6c42c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f6c3c0ce2e0 con 0x7f6c3c14e630 2024-05-28T06:05:20.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.564+0000 7f6c3be00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c20004800 0x7f6c3c15bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:20.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.564+0000 7f6c3be00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c20004800 0x7f6c3c15bc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49394/0 (socket says 172.21.15.96:49394) 2024-05-28T06:05:20.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.564+0000 7f6c3be00640 1 -- 172.21.15.96:0/2728167135 learned_addr learned my addr 172.21.15.96:0/2728167135 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:20.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.564+0000 7f6c40c00640 1 --2- 172.21.15.96:0/2728167135 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c3c1e76b0 0x7f6c3c1e7b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:20.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.564+0000 7f6c33400640 1 --2- 172.21.15.96:0/2728167135 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6c3c14e630 0x7f6c3c0c5b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:20.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.564+0000 7f6c3be00640 1 -- 172.21.15.96:0/2728167135 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6c3c14e630 msgr2=0x7f6c3c0c5b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:20.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.564+0000 7f6c3be00640 1 --2- 172.21.15.96:0/2728167135 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6c3c14e630 0x7f6c3c0c5b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:20.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.564+0000 7f6c3be00640 1 -- 172.21.15.96:0/2728167135 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c3c1e76b0 msgr2=0x7f6c3c1e7b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:20.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.564+0000 7f6c3be00640 1 --2- 172.21.15.96:0/2728167135 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c3c1e76b0 0x7f6c3c1e7b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:20.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.564+0000 7f6c3be00640 1 -- 172.21.15.96:0/2728167135 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c3c0c8390 con 0x7f6c20004800 2024-05-28T06:05:20.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.564+0000 7f6c33400640 1 --2- 172.21.15.96:0/2728167135 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6c3c14e630 0x7f6c3c0c5b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:05:20.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.564+0000 7f6c40c00640 1 --2- 172.21.15.96:0/2728167135 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c3c1e76b0 0x7f6c3c1e7b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:05:20.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.565+0000 7f6c3be00640 1 --2- 172.21.15.96:0/2728167135 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c20004800 0x7f6c3c15bc60 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f6c2c00e7d0 tx=0x7f6c2c00ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:20.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.565+0000 7f6c38c00640 1 -- 172.21.15.96:0/2728167135 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c2c017070 con 0x7f6c20004800 2024-05-28T06:05:20.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.565+0000 7f6c38c00640 1 -- 172.21.15.96:0/2728167135 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c2c00ee80 con 0x7f6c20004800 2024-05-28T06:05:20.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.565+0000 7f6c38c00640 1 -- 172.21.15.96:0/2728167135 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c2c00cb00 con 0x7f6c20004800 2024-05-28T06:05:20.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.565+0000 7f6c42c00640 1 -- 172.21.15.96:0/2728167135 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c3c0c8670 con 0x7f6c20004800 2024-05-28T06:05:20.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.565+0000 7f6c42c00640 1 -- 172.21.15.96:0/2728167135 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c3c0c8c30 con 0x7f6c20004800 2024-05-28T06:05:20.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.566+0000 7f6c42c00640 1 -- 172.21.15.96:0/2728167135 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c3c14ea10 con 0x7f6c20004800 2024-05-28T06:05:20.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.567+0000 7f6c38c00640 1 -- 172.21.15.96:0/2728167135 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6c2c01a070 con 0x7f6c20004800 2024-05-28T06:05:20.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.568+0000 7f6c38c00640 1 --2- 172.21.15.96:0/2728167135 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6c1806c920 0x7f6c1806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:20.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.569+0000 7f6c38c00640 1 -- 172.21.15.96:0/2728167135 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5473+0+0 (secure 0 0 0) 0x7f6c2c08bb90 con 0x7f6c20004800 2024-05-28T06:05:20.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.569+0000 7f6c33400640 1 --2- 172.21.15.96:0/2728167135 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6c1806c920 0x7f6c1806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:20.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.570+0000 7f6c33400640 1 --2- 172.21.15.96:0/2728167135 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6c1806c920 0x7f6c1806ede0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f6c24032000 tx=0x7f6c24007000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:20.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.573+0000 7f6c38c00640 1 -- 172.21.15.96:0/2728167135 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6c2c05a000 con 0x7f6c20004800 2024-05-28T06:05:20.786 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:20.785+0000 7f6c42c00640 1 -- 172.21.15.96:0/2728167135 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f6c3c0cd0a0 con 0x7f6c20004800 2024-05-28T06:05:21.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:21.334+0000 7f6c38c00640 1 -- 172.21.15.96:0/2728167135 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v25) ==== 169+0+0 (secure 0 0 0) 0x7f6c3c0cd0a0 con 0x7f6c20004800 2024-05-28T06:05:21.336 INFO:teuthology.orchestra.run.smithi096.stderr:pool 'cephfs_data' removed 2024-05-28T06:05:21.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:21.340+0000 7f6c42c00640 1 -- 172.21.15.96:0/2728167135 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6c1806c920 msgr2=0x7f6c1806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:21.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:21.340+0000 7f6c42c00640 1 --2- 172.21.15.96:0/2728167135 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6c1806c920 0x7f6c1806ede0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f6c24032000 tx=0x7f6c24007000 comp rx=0 tx=0).stop 2024-05-28T06:05:21.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:21.340+0000 7f6c42c00640 1 -- 172.21.15.96:0/2728167135 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c20004800 msgr2=0x7f6c3c15bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:21.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:21.340+0000 7f6c42c00640 1 --2- 172.21.15.96:0/2728167135 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c20004800 0x7f6c3c15bc60 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f6c2c00e7d0 tx=0x7f6c2c00ec90 comp rx=0 tx=0).stop 2024-05-28T06:05:21.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:21.340+0000 7f6c42c00640 1 -- 172.21.15.96:0/2728167135 shutdown_connections 2024-05-28T06:05:21.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:21.340+0000 7f6c42c00640 1 --2- 172.21.15.96:0/2728167135 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6c1806c920 0x7f6c1806ede0 secure :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f6c24032000 tx=0x7f6c24007000 comp rx=0 tx=0).stop 2024-05-28T06:05:21.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:21.340+0000 7f6c42c00640 1 --2- 172.21.15.96:0/2728167135 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c3c1e76b0 0x7f6c3c1e7b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:21.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:21.340+0000 7f6c42c00640 1 --2- 172.21.15.96:0/2728167135 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6c3c14e630 0x7f6c3c0c5b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:21.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:21.340+0000 7f6c42c00640 1 --2- 172.21.15.96:0/2728167135 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c20004800 0x7f6c3c15bc60 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:21.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:21.340+0000 7f6c42c00640 1 -- 172.21.15.96:0/2728167135 >> 172.21.15.96:0/2728167135 conn(0x7f6c3c0b3ff0 msgr2=0x7f6c3c071820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:21.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:21.340+0000 7f6c42c00640 1 -- 172.21.15.96:0/2728167135 shutdown_connections 2024-05-28T06:05:21.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:21.341+0000 7f6c42c00640 1 -- 172.21.15.96:0/2728167135 wait complete. 2024-05-28T06:05:21.366 INFO:tasks.ceph.mds.a:Restarting daemon 2024-05-28T06:05:21.366 INFO:tasks.ceph.mds.a:Stopping old one... 2024-05-28T06:05:21.366 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-05-28T06:05:21.366 INFO:teuthology.orchestra.run:waiting for 300 2024-05-28T06:05:21.366 INFO:tasks.ceph.mds.c:Restarting daemon 2024-05-28T06:05:21.366 INFO:tasks.ceph.mds.c:Stopping old one... 2024-05-28T06:05:21.367 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-05-28T06:05:21.367 INFO:teuthology.orchestra.run:waiting for 300 2024-05-28T06:05:21.367 INFO:tasks.ceph.mds.b:Restarting daemon 2024-05-28T06:05:21.367 INFO:tasks.ceph.mds.b:Stopping old one... 2024-05-28T06:05:21.367 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-05-28T06:05:21.367 INFO:teuthology.orchestra.run:waiting for 300 2024-05-28T06:05:21.367 INFO:tasks.ceph.mds.d:Restarting daemon 2024-05-28T06:05:21.368 INFO:tasks.ceph.mds.d:Stopping old one... 2024-05-28T06:05:21.368 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-05-28T06:05:21.368 INFO:teuthology.orchestra.run:waiting for 300 2024-05-28T06:05:21.385 INFO:tasks.ceph.mds.a:Stopped 2024-05-28T06:05:21.385 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-05-28T06:05:21.387 INFO:tasks.ceph.mds.c:Stopped 2024-05-28T06:05:21.387 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-05-28T06:05:21.393 INFO:tasks.ceph.mds.d:Stopped 2024-05-28T06:05:21.393 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-05-28T06:05:21.393 INFO:tasks.ceph.mds.b:Stopped 2024-05-28T06:05:21.393 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-05-28T06:05:21.431 INFO:tasks.ceph.mds.a:Started 2024-05-28T06:05:21.432 DEBUG:teuthology.parallel:result is None 2024-05-28T06:05:21.435 INFO:tasks.ceph.mds.c:Started 2024-05-28T06:05:21.435 DEBUG:teuthology.parallel:result is None 2024-05-28T06:05:21.440 INFO:tasks.ceph.mds.d:Started 2024-05-28T06:05:21.440 DEBUG:teuthology.parallel:result is None 2024-05-28T06:05:21.446 INFO:tasks.ceph.mds.b:Started 2024-05-28T06:05:21.446 DEBUG:teuthology.parallel:result is None 2024-05-28T06:05:21.446 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-05-28T06:05:21.573 INFO:tasks.ceph.mds.a.smithi096.stdout:starting mds.a at 2024-05-28T06:05:21.574 INFO:tasks.ceph.mds.c.smithi096.stdout:starting mds.c at 2024-05-28T06:05:21.593 INFO:tasks.ceph.mds.b.smithi123.stdout:starting mds.b at 2024-05-28T06:05:21.593 INFO:tasks.ceph.mds.d.smithi123.stdout:starting mds.d at 2024-05-28T06:05:21.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:21.619+0000 7f35cfa00640 1 Processor -- start 2024-05-28T06:05:21.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:21.619+0000 7f35cfa00640 1 -- start start 2024-05-28T06:05:21.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:21.620+0000 7f35cfa00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f35c806f860 0x7f35c80b3150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:21.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:21.620+0000 7f35cfa00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f35c80d0bb0 0x7f35c80ad130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:21.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:21.620+0000 7f35cfa00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35c806ed10 con 0x7f35c80b3690 2024-05-28T06:05:21.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:21.620+0000 7f35cfa00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f35c815cb00 con 0x7f35c806f860 2024-05-28T06:05:21.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:21.620+0000 7f35cfa00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f35c80b3cc0 con 0x7f35c80d0bb0 2024-05-28T06:05:21.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:21.620+0000 7f35c7e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f35c80b3690 0x7f35c80d04a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:48322/0 (socket says 172.21.15.96:48322) 2024-05-28T06:05:21.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:21.620+0000 7f35cd000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f35c806f860 0x7f35c80b3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:21.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:21.620+0000 7f35c7e00640 1 -- 172.21.15.96:0/917981793 learned_addr learned my addr 172.21.15.96:0/917981793 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:05:21.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:21.620+0000 7f35cda00640 1 --2- 172.21.15.96:0/917981793 >> v2:172.21.15.123:3300/0 conn(0x7f35c80d0bb0 0x7f35c80ad130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:21.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:21.621+0000 7f35c7400640 1 -- 172.21.15.96:0/917981793 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1922273655 0 0) 0x7f35b4001110 con 0x7f35c80b3690 2024-05-28T06:05:21.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:21.621+0000 7f35c7400640 1 -- 172.21.15.96:0/917981793 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f35b0003520 con 0x7f35c80b3690 2024-05-28T06:05:21.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:21.621+0000 7f35cd000640 1 -- 172.21.15.96:0/917981793 >> v2:172.21.15.123:3300/0 conn(0x7f35c80d0bb0 msgr2=0x7f35c80ad130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:21.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:21.621+0000 7f35cd000640 1 --2- 172.21.15.96:0/917981793 >> v2:172.21.15.123:3300/0 conn(0x7f35c80d0bb0 0x7f35c80ad130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:21.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:21.621+0000 7f35cd000640 1 -- 172.21.15.96:0/917981793 >> v1:172.21.15.96:6789/0 conn(0x7f35c80b3690 legacy=0x7f35c80d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:21.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:21.621+0000 7f35cd000640 1 -- 172.21.15.96:0/917981793 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35c80b3fa0 con 0x7f35c806f860 2024-05-28T06:05:21.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:21.622+0000 7f35cd000640 1 --2- 172.21.15.96:0/917981793 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f35c806f860 0x7f35c80b3150 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f35c80af6e0 tx=0x7f35bc02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=577cdd3daf500e43 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:21.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:21.622+0000 7f35c7400640 1 -- 172.21.15.96:0/917981793 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35bc037070 con 0x7f35c806f860 2024-05-28T06:05:21.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:21.622+0000 7f35c7400640 1 -- 172.21.15.96:0/917981793 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35bc032440 con 0x7f35c806f860 2024-05-28T06:05:21.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:21.622+0000 7f35c7400640 1 -- 172.21.15.96:0/917981793 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35bc032810 con 0x7f35c806f860 2024-05-28T06:05:21.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:21.622+0000 7f35cfa00640 1 -- 172.21.15.96:0/917981793 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f35c806f860 msgr2=0x7f35c80b3150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:21.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:21.622+0000 7f35cfa00640 1 --2- 172.21.15.96:0/917981793 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f35c806f860 0x7f35c80b3150 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f35c80af6e0 tx=0x7f35bc02a1a0 comp rx=0 tx=0).stop 2024-05-28T06:05:21.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:21.623+0000 7f35cfa00640 1 -- 172.21.15.96:0/917981793 shutdown_connections 2024-05-28T06:05:21.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:21.623+0000 7f35cfa00640 1 --2- 172.21.15.96:0/917981793 >> v2:172.21.15.123:3300/0 conn(0x7f35c80d0bb0 0x7f35c80ad130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:21.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:21.623+0000 7f35cfa00640 1 --2- 172.21.15.96:0/917981793 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f35c806f860 0x7f35c80b3150 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:21.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:21.623+0000 7f35cfa00640 1 -- 172.21.15.96:0/917981793 >> 172.21.15.96:0/917981793 conn(0x7f35c806bae0 msgr2=0x7f35c806bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:21.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:21.623+0000 7f35cfa00640 1 -- 172.21.15.96:0/917981793 shutdown_connections 2024-05-28T06:05:21.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:21.623+0000 7f35cfa00640 1 -- 172.21.15.96:0/917981793 wait complete. 2024-05-28T06:05:21.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:21.624+0000 7f35cfa00640 1 Processor -- start 2024-05-28T06:05:21.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:21.624+0000 7f35cfa00640 1 -- start start 2024-05-28T06:05:21.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:21.625+0000 7f35cfa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f35c806f860 0x7f35c81f3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:21.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:21.625+0000 7f35cfa00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f35c80b3690 0x7f35c80c3a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:21.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:21.625+0000 7f35cfa00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f35c80d0bb0 0x7f35c815e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:21.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:21.625+0000 7f35cfa00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f35c80b4840 con 0x7f35c806f860 2024-05-28T06:05:21.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:21.625+0000 7f35cfa00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f35c80b4560 con 0x7f35c80b3690 2024-05-28T06:05:21.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:21.625+0000 7f35cfa00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f35c80ad670 con 0x7f35c80d0bb0 2024-05-28T06:05:21.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:21.626+0000 7f35cd000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f35c806f860 0x7f35c81f3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:21.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:21.626+0000 7f35cd000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f35c806f860 0x7f35c81f3fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49444/0 (socket says 172.21.15.96:49444) 2024-05-28T06:05:21.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:21.626+0000 7f35cd000640 1 -- 172.21.15.96:0/2056706631 learned_addr learned my addr 172.21.15.96:0/2056706631 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:21.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:21.626+0000 7f35cda00640 1 --2- 172.21.15.96:0/2056706631 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f35c80d0bb0 0x7f35c815e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:21.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:21.626+0000 7f35c7e00640 1 --2- 172.21.15.96:0/2056706631 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f35c80b3690 0x7f35c80c3a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:21.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:21.626+0000 7f35cd000640 1 -- 172.21.15.96:0/2056706631 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f35c80d0bb0 msgr2=0x7f35c815e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:21.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:21.626+0000 7f35cd000640 1 --2- 172.21.15.96:0/2056706631 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f35c80d0bb0 0x7f35c815e7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:21.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:21.626+0000 7f35cd000640 1 -- 172.21.15.96:0/2056706631 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f35c80b3690 msgr2=0x7f35c80c3a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:21.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:21.626+0000 7f35cd000640 1 --2- 172.21.15.96:0/2056706631 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f35c80b3690 0x7f35c80c3a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:21.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:21.626+0000 7f35cd000640 1 -- 172.21.15.96:0/2056706631 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35c81f4500 con 0x7f35c806f860 2024-05-28T06:05:21.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:21.626+0000 7f35cda00640 1 --2- 172.21.15.96:0/2056706631 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f35c80d0bb0 0x7f35c815e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-28T06:05:21.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:21.626+0000 7f35cd000640 1 --2- 172.21.15.96:0/2056706631 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f35c806f860 0x7f35c81f3fc0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f35bc009890 tx=0x7f35bc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:21.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:21.626+0000 7f35c4c00640 1 -- 172.21.15.96:0/2056706631 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35bc041020 con 0x7f35c806f860 2024-05-28T06:05:21.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:21.626+0000 7f35c4c00640 1 -- 172.21.15.96:0/2056706631 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35bc03d700 con 0x7f35c806f860 2024-05-28T06:05:21.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:21.627+0000 7f35c4c00640 1 -- 172.21.15.96:0/2056706631 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35bc037040 con 0x7f35c806f860 2024-05-28T06:05:21.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:21.627+0000 7f35cfa00640 1 -- 172.21.15.96:0/2056706631 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35c80c4350 con 0x7f35c806f860 2024-05-28T06:05:21.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:21.627+0000 7f35cfa00640 1 -- 172.21.15.96:0/2056706631 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35c80c4910 con 0x7f35c806f860 2024-05-28T06:05:21.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:21.628+0000 7f35cfa00640 1 -- 172.21.15.96:0/2056706631 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35c80b3150 con 0x7f35c806f860 2024-05-28T06:05:21.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:21.628+0000 7f35c4c00640 1 -- 172.21.15.96:0/2056706631 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f35bc044020 con 0x7f35c806f860 2024-05-28T06:05:21.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:21.629+0000 7f35c4c00640 1 --2- 172.21.15.96:0/2056706631 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f35a806c8f0 0x7f35a806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:21.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:21.629+0000 7f35c4c00640 1 -- 172.21.15.96:0/2056706631 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 4943+0+0 (secure 0 0 0) 0x7f35bc0ac070 con 0x7f35c806f860 2024-05-28T06:05:21.631 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:21.630+0000 7f35c7e00640 1 --2- 172.21.15.96:0/2056706631 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f35a806c8f0 0x7f35a806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:21.631 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:21.630+0000 7f35c7e00640 1 --2- 172.21.15.96:0/2056706631 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f35a806c8f0 0x7f35a806edb0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f35c80b3050 tx=0x7f35b4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:21.635 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:21.634+0000 7f35c4c00640 1 -- 172.21.15.96:0/2056706631 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f35c80b3150 con 0x7f35c806f860 2024-05-28T06:05:21.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:21.854+0000 7f35cfa00640 1 -- 172.21.15.96:0/2056706631 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f35c80689e0 con 0x7f35c806f860 2024-05-28T06:05:22.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:22.357+0000 7f35c4c00640 1 -- 172.21.15.96:0/2056706631 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v27) ==== 97+0+0 (secure 0 0 0) 0x7f35bc04b030 con 0x7f35c806f860 2024-05-28T06:05:22.359 INFO:teuthology.orchestra.run.smithi096.stderr: removed all blocklist entries 2024-05-28T06:05:22.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:22.361+0000 7f35cfa00640 1 -- 172.21.15.96:0/2056706631 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f35a806c8f0 msgr2=0x7f35a806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:22.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:22.361+0000 7f35cfa00640 1 --2- 172.21.15.96:0/2056706631 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f35a806c8f0 0x7f35a806edb0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f35c80b3050 tx=0x7f35b4009040 comp rx=0 tx=0).stop 2024-05-28T06:05:22.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:22.361+0000 7f35cfa00640 1 -- 172.21.15.96:0/2056706631 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f35c806f860 msgr2=0x7f35c81f3fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:22.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:22.361+0000 7f35cfa00640 1 --2- 172.21.15.96:0/2056706631 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f35c806f860 0x7f35c81f3fc0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f35bc009890 tx=0x7f35bc0046c0 comp rx=0 tx=0).stop 2024-05-28T06:05:22.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:22.361+0000 7f35cfa00640 1 -- 172.21.15.96:0/2056706631 shutdown_connections 2024-05-28T06:05:22.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:22.361+0000 7f35cfa00640 1 --2- 172.21.15.96:0/2056706631 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f35a806c8f0 0x7f35a806edb0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:22.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:22.361+0000 7f35cfa00640 1 --2- 172.21.15.96:0/2056706631 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f35c80d0bb0 0x7f35c815e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:22.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:22.361+0000 7f35cfa00640 1 --2- 172.21.15.96:0/2056706631 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f35c80b3690 0x7f35c80c3a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:22.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:22.361+0000 7f35cfa00640 1 --2- 172.21.15.96:0/2056706631 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f35c806f860 0x7f35c81f3fc0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:22.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:22.361+0000 7f35cfa00640 1 -- 172.21.15.96:0/2056706631 >> 172.21.15.96:0/2056706631 conn(0x7f35c806bae0 msgr2=0x7f35c80ada30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:22.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:22.362+0000 7f35cfa00640 1 -- 172.21.15.96:0/2056706631 shutdown_connections 2024-05-28T06:05:22.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:22.362+0000 7f35cfa00640 1 -- 172.21.15.96:0/2056706631 wait complete. 2024-05-28T06:05:22.385 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-05-28T06:05:22.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:22.571+0000 7f9fa9c00640 1 Processor -- start 2024-05-28T06:05:22.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:22.571+0000 7f9fa9c00640 1 -- start start 2024-05-28T06:05:22.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:22.573+0000 7f9fa9c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9fa406f860 0x7f9fa40b3150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:22.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:22.573+0000 7f9fa9c00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f9fa40b3690 0x7f9fa40d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:22.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:22.573+0000 7f9fa9c00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9fa406ed10 con 0x7f9fa40d09e0 2024-05-28T06:05:22.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:22.573+0000 7f9fa9c00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f9fa40d0e50 con 0x7f9fa40b3690 2024-05-28T06:05:22.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:22.573+0000 7f9fa9c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f9fa415cb00 con 0x7f9fa406f860 2024-05-28T06:05:22.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:22.573+0000 7f9f9a000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f9fa40b3690 0x7f9fa40d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:22.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:22.573+0000 7f9fa3400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f9fa40d09e0 0x7f9fa4160210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:48334/0 (socket says 172.21.15.96:48334) 2024-05-28T06:05:22.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:22.573+0000 7f9fa3400640 1 -- 172.21.15.96:0/3163958866 learned_addr learned my addr 172.21.15.96:0/3163958866 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:05:22.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:22.574+0000 7f9fa2a00640 1 --2- 172.21.15.96:0/3163958866 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9fa406f860 0x7f9fa40b3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:22.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:22.574+0000 7f9f9a000640 1 -- 172.21.15.96:0/3163958866 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9fa406f860 msgr2=0x7f9fa40b3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:22.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:22.574+0000 7f9f9a000640 1 --2- 172.21.15.96:0/3163958866 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9fa406f860 0x7f9fa40b3150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:22.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:22.574+0000 7f9f9a000640 1 -- 172.21.15.96:0/3163958866 >> v1:172.21.15.96:6789/0 conn(0x7f9fa40d09e0 legacy=0x7f9fa4160210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:22.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:22.574+0000 7f9f9a000640 1 -- 172.21.15.96:0/3163958866 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fa40b3cc0 con 0x7f9fa40b3690 2024-05-28T06:05:22.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:22.575+0000 7f9f9a000640 1 --2- 172.21.15.96:0/3163958866 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9fa40b3690 0x7f9fa40d04a0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c0097a0 tx=0x7f9f8c02a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14a663b81d205205 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:22.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:22.575+0000 7f9fa2000640 1 -- 172.21.15.96:0/3163958866 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f8c037070 con 0x7f9fa40b3690 2024-05-28T06:05:22.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:22.575+0000 7f9fa2000640 1 -- 172.21.15.96:0/3163958866 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f8c032440 con 0x7f9fa40b3690 2024-05-28T06:05:22.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:22.575+0000 7f9fa2000640 1 -- 172.21.15.96:0/3163958866 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f8c032810 con 0x7f9fa40b3690 2024-05-28T06:05:22.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:22.576+0000 7f9fa9c00640 1 -- 172.21.15.96:0/3163958866 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9fa40b3690 msgr2=0x7f9fa40d04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:22.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:22.576+0000 7f9fa9c00640 1 --2- 172.21.15.96:0/3163958866 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9fa40b3690 0x7f9fa40d04a0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c0097a0 tx=0x7f9f8c02a180 comp rx=0 tx=0).stop 2024-05-28T06:05:22.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:22.576+0000 7f9fa9c00640 1 -- 172.21.15.96:0/3163958866 shutdown_connections 2024-05-28T06:05:22.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:22.576+0000 7f9fa9c00640 1 --2- 172.21.15.96:0/3163958866 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9fa40b3690 0x7f9fa40d04a0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:22.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:22.576+0000 7f9fa9c00640 1 --2- 172.21.15.96:0/3163958866 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9fa406f860 0x7f9fa40b3150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:22.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:22.576+0000 7f9fa9c00640 1 -- 172.21.15.96:0/3163958866 >> 172.21.15.96:0/3163958866 conn(0x7f9fa406bae0 msgr2=0x7f9fa406bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:22.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:22.576+0000 7f9fa9c00640 1 -- 172.21.15.96:0/3163958866 shutdown_connections 2024-05-28T06:05:22.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:22.576+0000 7f9fa9c00640 1 -- 172.21.15.96:0/3163958866 wait complete. 2024-05-28T06:05:22.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:22.578+0000 7f9fa9c00640 1 Processor -- start 2024-05-28T06:05:22.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:22.578+0000 7f9fa9c00640 1 -- start start 2024-05-28T06:05:22.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:22.579+0000 7f9fa9c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9fa406f860 0x7f9fa415cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:22.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:22.579+0000 7f9fa9c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9fa40b3690 0x7f9fa4099a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:22.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:22.579+0000 7f9fa9c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9fa40d09e0 0x7f9fa40b2580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:22.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:22.579+0000 7f9fa9c00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f9fa40b4560 con 0x7f9fa40d09e0 2024-05-28T06:05:22.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:22.579+0000 7f9fa9c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f9fa40b4280 con 0x7f9fa40b3690 2024-05-28T06:05:22.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:22.579+0000 7f9fa9c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f9fa40b4840 con 0x7f9fa406f860 2024-05-28T06:05:22.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:22.580+0000 7f9fa3400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9fa40d09e0 0x7f9fa40b2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:22.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:22.580+0000 7f9fa3400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9fa40d09e0 0x7f9fa40b2580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49466/0 (socket says 172.21.15.96:49466) 2024-05-28T06:05:22.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:22.580+0000 7f9fa3400640 1 -- 172.21.15.96:0/2242195026 learned_addr learned my addr 172.21.15.96:0/2242195026 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:22.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:22.580+0000 7f9fa2a00640 1 --2- 172.21.15.96:0/2242195026 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9fa406f860 0x7f9fa415cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:22.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:22.580+0000 7f9f9a000640 1 --2- 172.21.15.96:0/2242195026 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9fa40b3690 0x7f9fa4099a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:22.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:22.580+0000 7f9fa3400640 1 -- 172.21.15.96:0/2242195026 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9fa406f860 msgr2=0x7f9fa415cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:22.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:22.580+0000 7f9fa3400640 1 --2- 172.21.15.96:0/2242195026 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9fa406f860 0x7f9fa415cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:22.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:22.580+0000 7f9fa3400640 1 -- 172.21.15.96:0/2242195026 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9fa40b3690 msgr2=0x7f9fa4099a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:22.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:22.580+0000 7f9fa3400640 1 --2- 172.21.15.96:0/2242195026 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9fa40b3690 0x7f9fa4099a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:22.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:22.580+0000 7f9fa3400640 1 -- 172.21.15.96:0/2242195026 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fa415d3f0 con 0x7f9fa40d09e0 2024-05-28T06:05:22.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:22.581+0000 7f9fa3400640 1 --2- 172.21.15.96:0/2242195026 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9fa40d09e0 0x7f9fa40b2580 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f9f9c00c420 tx=0x7f9f9c00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:22.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:22.581+0000 7f9f9b400640 1 -- 172.21.15.96:0/2242195026 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f9c017070 con 0x7f9fa40d09e0 2024-05-28T06:05:22.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:22.581+0000 7f9f9b400640 1 -- 172.21.15.96:0/2242195026 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f9c005ce0 con 0x7f9fa40d09e0 2024-05-28T06:05:22.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:22.581+0000 7f9f9b400640 1 -- 172.21.15.96:0/2242195026 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f9c0072d0 con 0x7f9fa40d09e0 2024-05-28T06:05:22.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:22.581+0000 7f9fa9c00640 1 -- 172.21.15.96:0/2242195026 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fa415d6d0 con 0x7f9fa40d09e0 2024-05-28T06:05:22.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:22.581+0000 7f9fa9c00640 1 -- 172.21.15.96:0/2242195026 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fa415dc90 con 0x7f9fa40d09e0 2024-05-28T06:05:22.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:22.582+0000 7f9fa9c00640 1 -- 172.21.15.96:0/2242195026 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9fa40689e0 con 0x7f9fa40d09e0 2024-05-28T06:05:22.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:22.583+0000 7f9f9b400640 1 -- 172.21.15.96:0/2242195026 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9f9c01a070 con 0x7f9fa40d09e0 2024-05-28T06:05:22.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:22.584+0000 7f9f9b400640 1 --2- 172.21.15.96:0/2242195026 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9f8006c8f0 0x7f9f8006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:22.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:22.584+0000 7f9f9b400640 1 -- 172.21.15.96:0/2242195026 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 4857+0+0 (secure 0 0 0) 0x7f9f9c08aad0 con 0x7f9fa40d09e0 2024-05-28T06:05:22.586 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:22.585+0000 7f9fa2a00640 1 --2- 172.21.15.96:0/2242195026 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9f8006c8f0 0x7f9f8006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:22.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:22.586+0000 7f9fa2a00640 1 --2- 172.21.15.96:0/2242195026 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9f8006c8f0 0x7f9f8006edb0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f9f940099c0 tx=0x7f9f94003990 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:22.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:22.589+0000 7f9f9b400640 1 -- 172.21.15.96:0/2242195026 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9fa406ed10 con 0x7f9fa40d09e0 2024-05-28T06:05:22.860 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:22.859+0000 7f9fa9c00640 1 -- 172.21.15.96:0/2242195026 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f9fa406e2f0 con 0x7f9fa40d09e0 2024-05-28T06:05:22.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:22.860+0000 7f9f9b400640 1 -- 172.21.15.96:0/2242195026 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v6) ==== 80+0+6645 (secure 0 0 0) 0x7f9f9c058bf0 con 0x7f9fa40d09e0 2024-05-28T06:05:22.862 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:05:22.862 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:05:22.862 INFO:teuthology.orchestra.run.smithi096.stdout: "auth_dump": [ 2024-05-28T06:05:22.862 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:05:22.862 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "mds.a", 2024-05-28T06:05:22.862 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCWc1VmyA9HHBAANJCF8I/RP5ZYviyOzo0JCQ==", 2024-05-28T06:05:22.863 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:05:22.863 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow", 2024-05-28T06:05:22.863 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow *", 2024-05-28T06:05:22.863 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow *", 2024-05-28T06:05:22.863 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:05:22.863 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:05:22.863 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:05:22.863 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:05:22.863 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "mds.b", 2024-05-28T06:05:22.863 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCWc1VmtR0EOBAA296qGHTmB/2sMztDbDJ56g==", 2024-05-28T06:05:22.863 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:05:22.863 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow", 2024-05-28T06:05:22.863 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow *", 2024-05-28T06:05:22.863 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow *", 2024-05-28T06:05:22.863 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:05:22.864 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:05:22.864 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:05:22.864 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:05:22.864 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "mds.c", 2024-05-28T06:05:22.864 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCWc1VmrLxHKhAAhkTRLDKL6RUJQGetB8JQQA==", 2024-05-28T06:05:22.864 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:05:22.864 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow", 2024-05-28T06:05:22.864 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow *", 2024-05-28T06:05:22.864 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow *", 2024-05-28T06:05:22.864 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:05:22.864 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:05:22.864 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:05:22.864 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:05:22.865 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "mds.d", 2024-05-28T06:05:22.865 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCXc1VmW2aQCxAAnkDwQzlgy39vFe2O9Wwf7Q==", 2024-05-28T06:05:22.865 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:05:22.865 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow", 2024-05-28T06:05:22.865 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow *", 2024-05-28T06:05:22.865 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow *", 2024-05-28T06:05:22.865 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:05:22.865 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:05:22.865 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:05:22.865 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:05:22.865 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "osd.0", 2024-05-28T06:05:22.865 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCbc1VmEcNAAxAAiB5AsfY2uRY1kc6h80chuw==", 2024-05-28T06:05:22.865 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:05:22.865 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow profile osd", 2024-05-28T06:05:22.866 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile osd", 2024-05-28T06:05:22.866 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:05:22.866 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:05:22.866 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:05:22.866 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:05:22.866 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "osd.1", 2024-05-28T06:05:22.866 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCec1Vm6CAaDhAA1NkYM2q4fTsgSDSwJZL3Qg==", 2024-05-28T06:05:22.866 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:05:22.866 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow profile osd", 2024-05-28T06:05:22.866 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile osd", 2024-05-28T06:05:22.866 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:05:22.866 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:05:22.866 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:05:22.866 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:05:22.867 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "osd.2", 2024-05-28T06:05:22.867 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQChc1VmKn1jFxAA+dM4gM7FR8NwBpyd833A/Q==", 2024-05-28T06:05:22.867 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:05:22.867 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow profile osd", 2024-05-28T06:05:22.867 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile osd", 2024-05-28T06:05:22.867 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:05:22.867 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:05:22.867 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:05:22.867 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:05:22.867 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "osd.3", 2024-05-28T06:05:22.867 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCkc1VmV07MIRAAHIXQGDVbL9V5QqexQ89IJA==", 2024-05-28T06:05:22.867 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:05:22.867 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow profile osd", 2024-05-28T06:05:22.868 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile osd", 2024-05-28T06:05:22.868 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:05:22.868 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:05:22.868 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:05:22.868 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:05:22.868 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "osd.4", 2024-05-28T06:05:22.868 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCqc1VmobuZMxAAlY8xhuzqlkuFrcbTouF5rA==", 2024-05-28T06:05:22.868 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:05:22.868 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow profile osd", 2024-05-28T06:05:22.868 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile osd", 2024-05-28T06:05:22.868 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:05:22.868 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:05:22.868 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:05:22.868 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:05:22.869 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "osd.5", 2024-05-28T06:05:22.869 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCuc1Vmwt3QAxAAUFj/e/wJvxljgAbcTb32Ug==", 2024-05-28T06:05:22.869 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:05:22.869 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow profile osd", 2024-05-28T06:05:22.869 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile osd", 2024-05-28T06:05:22.869 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:05:22.869 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:05:22.869 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:05:22.869 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:05:22.869 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "osd.6", 2024-05-28T06:05:22.869 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCxc1VmscerDxAAqEDAqy6/OvdfSJ+ALZ7WeQ==", 2024-05-28T06:05:22.869 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:05:22.869 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow profile osd", 2024-05-28T06:05:22.870 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile osd", 2024-05-28T06:05:22.870 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:05:22.870 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:05:22.870 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:05:22.870 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:05:22.870 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "osd.7", 2024-05-28T06:05:22.870 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQC0c1VmgxkbKhAA7dFzDTBKFmKiFFxPsyiT4Q==", 2024-05-28T06:05:22.870 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:05:22.870 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow profile osd", 2024-05-28T06:05:22.870 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile osd", 2024-05-28T06:05:22.871 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:05:22.871 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:05:22.871 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:05:22.871 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:05:22.871 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "client.0", 2024-05-28T06:05:22.871 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCXc1Vm91hbGBAASTL3HQR8XOs7wVQEHbPl/g==", 2024-05-28T06:05:22.871 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:05:22.871 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow", 2024-05-28T06:05:22.871 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow r", 2024-05-28T06:05:22.871 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow rw", 2024-05-28T06:05:22.872 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow rwx" 2024-05-28T06:05:22.872 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:05:22.872 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:05:22.872 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:05:22.872 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "client.1", 2024-05-28T06:05:22.872 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCXc1VmIc77KhAAw2e0alTmVROhyCIFW4H3Sg==", 2024-05-28T06:05:22.872 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:05:22.872 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow", 2024-05-28T06:05:22.872 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow r", 2024-05-28T06:05:22.872 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow rw", 2024-05-28T06:05:22.872 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow rwx" 2024-05-28T06:05:22.872 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:05:22.872 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:05:22.873 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:05:22.873 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "client.2", 2024-05-28T06:05:22.873 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCXc1VmCS+OIRAAMV3Au6KuchLhA59QTGbv1w==", 2024-05-28T06:05:22.873 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:05:22.873 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow", 2024-05-28T06:05:22.873 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow r", 2024-05-28T06:05:22.873 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow rw", 2024-05-28T06:05:22.873 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow rwx" 2024-05-28T06:05:22.873 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:05:22.874 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:05:22.874 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:05:22.874 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "client.3", 2024-05-28T06:05:22.874 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCXc1Vm5r3pNBAAkVZ3arjywHQkEP5lmS8xqg==", 2024-05-28T06:05:22.874 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:05:22.874 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow", 2024-05-28T06:05:22.874 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow r", 2024-05-28T06:05:22.874 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow rw", 2024-05-28T06:05:22.874 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow rwx" 2024-05-28T06:05:22.874 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:05:22.874 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:05:22.874 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:05:22.875 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "client.admin", 2024-05-28T06:05:22.875 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCVc1Vm5QaXFhAAQFGKgbZ/kpGisNm2lIw0Uw==", 2024-05-28T06:05:22.875 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:05:22.875 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow *", 2024-05-28T06:05:22.875 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow *", 2024-05-28T06:05:22.875 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow *", 2024-05-28T06:05:22.875 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:05:22.875 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:05:22.875 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:05:22.875 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:05:22.876 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "client.bootstrap-mds", 2024-05-28T06:05:22.876 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQDCc1VmIBo8GxAASc2dtSn9uwhNXln7TKvohg==", 2024-05-28T06:05:22.876 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:05:22.876 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile bootstrap-mds" 2024-05-28T06:05:22.876 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:05:22.876 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:05:22.876 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:05:22.876 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "client.bootstrap-mgr", 2024-05-28T06:05:22.876 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQDCc1Vm3Tc8GxAAWXMX15crmZobtlGjlOTFZQ==", 2024-05-28T06:05:22.876 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:05:22.876 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile bootstrap-mgr" 2024-05-28T06:05:22.877 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:05:22.877 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:05:22.877 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:05:22.877 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "client.bootstrap-osd", 2024-05-28T06:05:22.877 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQDCc1VmiFI8GxAA0kqk9faNKJ0meZ9xc7rjZQ==", 2024-05-28T06:05:22.877 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:05:22.877 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile bootstrap-osd" 2024-05-28T06:05:22.877 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:05:22.877 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:05:22.877 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:05:22.878 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "client.bootstrap-rbd", 2024-05-28T06:05:22.878 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQDCc1VmUG08GxAALviDZvATvnr6/G1CzdjrAw==", 2024-05-28T06:05:22.878 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:05:22.878 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile bootstrap-rbd" 2024-05-28T06:05:22.878 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:05:22.878 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:05:22.878 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:05:22.878 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "client.bootstrap-rbd-mirror", 2024-05-28T06:05:22.878 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQDCc1VmRIc8GxAA9IUWAlbjLJQst0RS5Kr81g==", 2024-05-28T06:05:22.878 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:05:22.878 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2024-05-28T06:05:22.879 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:05:22.879 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:05:22.879 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:05:22.879 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "client.bootstrap-rgw", 2024-05-28T06:05:22.879 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQDCc1VmOqE8GxAAXwRZ7+Qi6wFimBQtDBiE1A==", 2024-05-28T06:05:22.879 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:05:22.879 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile bootstrap-rgw" 2024-05-28T06:05:22.879 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:05:22.879 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:05:22.880 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:05:22.880 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "mgr.x", 2024-05-28T06:05:22.880 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCWc1VmVMx/EhAAylvi37aq3hbMOhUSQWLVSg==", 2024-05-28T06:05:22.880 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:05:22.880 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow *", 2024-05-28T06:05:22.880 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile mgr", 2024-05-28T06:05:22.880 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:05:22.880 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:05:22.880 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:05:22.880 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:05:22.881 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "mgr.y", 2024-05-28T06:05:22.881 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCWc1Vmh5r7BxAAA74gn+FozehM+34eFik8kw==", 2024-05-28T06:05:22.881 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:05:22.881 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow *", 2024-05-28T06:05:22.881 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile mgr", 2024-05-28T06:05:22.881 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:05:22.881 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:05:22.881 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:05:22.881 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-05-28T06:05:22.881 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:05:22.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:22.863+0000 7f9fa9c00640 1 -- 172.21.15.96:0/2242195026 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9f8006c8f0 msgr2=0x7f9f8006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:22.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:22.863+0000 7f9fa9c00640 1 --2- 172.21.15.96:0/2242195026 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9f8006c8f0 0x7f9f8006edb0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f9f940099c0 tx=0x7f9f94003990 comp rx=0 tx=0).stop 2024-05-28T06:05:22.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:22.863+0000 7f9fa9c00640 1 -- 172.21.15.96:0/2242195026 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9fa40d09e0 msgr2=0x7f9fa40b2580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:22.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:22.863+0000 7f9fa9c00640 1 --2- 172.21.15.96:0/2242195026 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9fa40d09e0 0x7f9fa40b2580 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f9f9c00c420 tx=0x7f9f9c00c8e0 comp rx=0 tx=0).stop 2024-05-28T06:05:22.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:22.863+0000 7f9fa9c00640 1 -- 172.21.15.96:0/2242195026 shutdown_connections 2024-05-28T06:05:22.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:22.863+0000 7f9fa9c00640 1 --2- 172.21.15.96:0/2242195026 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9f8006c8f0 0x7f9f8006edb0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:22.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:22.863+0000 7f9fa9c00640 1 --2- 172.21.15.96:0/2242195026 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9fa40d09e0 0x7f9fa40b2580 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:22.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:22.863+0000 7f9fa9c00640 1 --2- 172.21.15.96:0/2242195026 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9fa40b3690 0x7f9fa4099a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:22.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:22.863+0000 7f9fa9c00640 1 --2- 172.21.15.96:0/2242195026 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9fa406f860 0x7f9fa415cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:22.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:22.863+0000 7f9fa9c00640 1 -- 172.21.15.96:0/2242195026 >> 172.21.15.96:0/2242195026 conn(0x7f9fa406bae0 msgr2=0x7f9fa40cb870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:22.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:22.863+0000 7f9fa9c00640 1 -- 172.21.15.96:0/2242195026 shutdown_connections 2024-05-28T06:05:22.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:22.864+0000 7f9fa9c00640 1 -- 172.21.15.96:0/2242195026 wait complete. 2024-05-28T06:05:22.885 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2024-05-28T06:05:23.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.084+0000 7f37a0200640 1 Processor -- start 2024-05-28T06:05:23.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.085+0000 7f37a0200640 1 -- start start 2024-05-28T06:05:23.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.086+0000 7f37a0200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f379815ed00 0x7f37980b3c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:23.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.086+0000 7f37a0200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f37980d0350 0x7f37980d4b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:23.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.086+0000 7f37a0200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37980b5070 con 0x7f37980d46c0 2024-05-28T06:05:23.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.086+0000 7f37a0200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f37980d5060 con 0x7f37980d0350 2024-05-28T06:05:23.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.086+0000 7f37a0200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f37980b41a0 con 0x7f379815ed00 2024-05-28T06:05:23.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.086+0000 7f379e200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f37980d0350 0x7f37980d4b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:23.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.087+0000 7f379e200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f37980d0350 0x7f37980d4b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49478/0 (socket says 172.21.15.96:49478) 2024-05-28T06:05:23.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.087+0000 7f379e200640 1 -- 172.21.15.96:0/426756278 learned_addr learned my addr 172.21.15.96:0/426756278 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:23.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.087+0000 7f379d800640 1 --2- 172.21.15.96:0/426756278 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f379815ed00 0x7f37980b3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:23.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.087+0000 7f379e200640 1 -- 172.21.15.96:0/426756278 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f379815ed00 msgr2=0x7f37980b3c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:23.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.087+0000 7f379e200640 1 --2- 172.21.15.96:0/426756278 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f379815ed00 0x7f37980b3c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:23.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.087+0000 7f379e200640 1 -- 172.21.15.96:0/426756278 >> v1:172.21.15.96:6789/0 conn(0x7f37980d46c0 legacy=0x7f37980d0d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:23.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.087+0000 7f379e200640 1 -- 172.21.15.96:0/426756278 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37980d33b0 con 0x7f37980d0350 2024-05-28T06:05:23.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.088+0000 7f379e200640 1 --2- 172.21.15.96:0/426756278 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f37980d0350 0x7f37980d4b20 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f3790009870 tx=0x7f379002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=870eb429107364ee server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:23.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.088+0000 7f3797e00640 1 -- 172.21.15.96:0/426756278 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3790037070 con 0x7f37980d0350 2024-05-28T06:05:23.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.088+0000 7f3797e00640 1 -- 172.21.15.96:0/426756278 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3790032440 con 0x7f37980d0350 2024-05-28T06:05:23.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.088+0000 7f3797e00640 1 -- 172.21.15.96:0/426756278 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3790032810 con 0x7f37980d0350 2024-05-28T06:05:23.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.089+0000 7f37a0200640 1 -- 172.21.15.96:0/426756278 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f37980d0350 msgr2=0x7f37980d4b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:23.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.089+0000 7f37a0200640 1 --2- 172.21.15.96:0/426756278 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f37980d0350 0x7f37980d4b20 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f3790009870 tx=0x7f379002a270 comp rx=0 tx=0).stop 2024-05-28T06:05:23.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.089+0000 7f37a0200640 1 -- 172.21.15.96:0/426756278 shutdown_connections 2024-05-28T06:05:23.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.089+0000 7f37a0200640 1 --2- 172.21.15.96:0/426756278 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f37980d0350 0x7f37980d4b20 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:23.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.089+0000 7f37a0200640 1 --2- 172.21.15.96:0/426756278 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f379815ed00 0x7f37980b3c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:23.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.089+0000 7f37a0200640 1 -- 172.21.15.96:0/426756278 >> 172.21.15.96:0/426756278 conn(0x7f379809da70 msgr2=0x7f379808bfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:23.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.089+0000 7f37a0200640 1 -- 172.21.15.96:0/426756278 shutdown_connections 2024-05-28T06:05:23.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.089+0000 7f37a0200640 1 -- 172.21.15.96:0/426756278 wait complete. 2024-05-28T06:05:23.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.091+0000 7f37a0200640 1 Processor -- start 2024-05-28T06:05:23.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.091+0000 7f37a0200640 1 -- start start 2024-05-28T06:05:23.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.092+0000 7f37a0200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f37980d0350 0x7f379814e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:23.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.092+0000 7f37a0200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f37980d46c0 0x7f37980b26f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:23.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.092+0000 7f37a0200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f379815ed00 0x7f37980b2c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:23.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.092+0000 7f37a0200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f37980b5610 con 0x7f379815ed00 2024-05-28T06:05:23.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.092+0000 7f37a0200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f37980b5330 con 0x7f37980d0350 2024-05-28T06:05:23.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.092+0000 7f37a0200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f37980d3040 con 0x7f37980d46c0 2024-05-28T06:05:23.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.093+0000 7f379e200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f379815ed00 0x7f37980b2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:23.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.093+0000 7f379e200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f379815ed00 0x7f37980b2c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49494/0 (socket says 172.21.15.96:49494) 2024-05-28T06:05:23.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.093+0000 7f379e200640 1 -- 172.21.15.96:0/2041585475 learned_addr learned my addr 172.21.15.96:0/2041585475 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:23.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.093+0000 7f379d800640 1 --2- 172.21.15.96:0/2041585475 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f37980d0350 0x7f379814e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:23.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.093+0000 7f379ce00640 1 --2- 172.21.15.96:0/2041585475 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f37980d46c0 0x7f37980b26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:23.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.093+0000 7f379e200640 1 -- 172.21.15.96:0/2041585475 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f37980d46c0 msgr2=0x7f37980b26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:23.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.093+0000 7f379e200640 1 --2- 172.21.15.96:0/2041585475 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f37980d46c0 0x7f37980b26f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:23.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.093+0000 7f379e200640 1 -- 172.21.15.96:0/2041585475 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f37980d0350 msgr2=0x7f379814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:23.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.093+0000 7f379e200640 1 --2- 172.21.15.96:0/2041585475 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f37980d0350 0x7f379814e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:23.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.093+0000 7f379e200640 1 -- 172.21.15.96:0/2041585475 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37981f57d0 con 0x7f379815ed00 2024-05-28T06:05:23.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.093+0000 7f379ce00640 1 --2- 172.21.15.96:0/2041585475 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f37980d46c0 0x7f37980b26f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:05:23.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.093+0000 7f379e200640 1 --2- 172.21.15.96:0/2041585475 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f379815ed00 0x7f37980b2c30 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f379002a2b0 tx=0x7f37900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:23.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.094+0000 7f3795600640 1 -- 172.21.15.96:0/2041585475 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3790037040 con 0x7f379815ed00 2024-05-28T06:05:23.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.094+0000 7f3795600640 1 -- 172.21.15.96:0/2041585475 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3790042070 con 0x7f379815ed00 2024-05-28T06:05:23.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.094+0000 7f3795600640 1 -- 172.21.15.96:0/2041585475 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3790030920 con 0x7f379815ed00 2024-05-28T06:05:23.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.094+0000 7f37a0200640 1 -- 172.21.15.96:0/2041585475 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37981f7fe0 con 0x7f379815ed00 2024-05-28T06:05:23.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.094+0000 7f37a0200640 1 -- 172.21.15.96:0/2041585475 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37981f85a0 con 0x7f379815ed00 2024-05-28T06:05:23.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.095+0000 7f37a0200640 1 -- 172.21.15.96:0/2041585475 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37980689e0 con 0x7f379815ed00 2024-05-28T06:05:23.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.096+0000 7f3795600640 1 -- 172.21.15.96:0/2041585475 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3790030ac0 con 0x7f379815ed00 2024-05-28T06:05:23.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.097+0000 7f3795600640 1 --2- 172.21.15.96:0/2041585475 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f377806c940 0x7f377806ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:23.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.097+0000 7f3795600640 1 -- 172.21.15.96:0/2041585475 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 4857+0+0 (secure 0 0 0) 0x7f37900ac120 con 0x7f379815ed00 2024-05-28T06:05:23.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.098+0000 7f379d800640 1 --2- 172.21.15.96:0/2041585475 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f377806c940 0x7f377806ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:23.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.099+0000 7f379d800640 1 --2- 172.21.15.96:0/2041585475 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f377806c940 0x7f377806ee00 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f3788006fd0 tx=0x7f3788008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:23.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.101+0000 7f3795600640 1 -- 172.21.15.96:0/2041585475 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f379007a230 con 0x7f379815ed00 2024-05-28T06:05:23.374 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.373+0000 7f37a0200640 1 -- 172.21.15.96:0/2041585475 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) -- 0x7f379806da90 con 0x7f379815ed00 2024-05-28T06:05:23.377 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.376+0000 7f3795600640 1 -- 172.21.15.96:0/2041585475 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v7) ==== 90+0+0 (secure 0 0 0) 0x7f379007e530 con 0x7f379815ed00 2024-05-28T06:05:23.380 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.379+0000 7f37a0200640 1 -- 172.21.15.96:0/2041585475 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f377806c940 msgr2=0x7f377806ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:23.380 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.379+0000 7f37a0200640 1 --2- 172.21.15.96:0/2041585475 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f377806c940 0x7f377806ee00 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f3788006fd0 tx=0x7f3788008040 comp rx=0 tx=0).stop 2024-05-28T06:05:23.381 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.379+0000 7f37a0200640 1 -- 172.21.15.96:0/2041585475 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f379815ed00 msgr2=0x7f37980b2c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:23.381 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.379+0000 7f37a0200640 1 --2- 172.21.15.96:0/2041585475 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f379815ed00 0x7f37980b2c30 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f379002a2b0 tx=0x7f37900046c0 comp rx=0 tx=0).stop 2024-05-28T06:05:23.381 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.379+0000 7f37a0200640 1 -- 172.21.15.96:0/2041585475 shutdown_connections 2024-05-28T06:05:23.381 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.379+0000 7f37a0200640 1 --2- 172.21.15.96:0/2041585475 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f377806c940 0x7f377806ee00 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:23.381 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.379+0000 7f37a0200640 1 --2- 172.21.15.96:0/2041585475 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f379815ed00 0x7f37980b2c30 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:23.381 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.379+0000 7f37a0200640 1 --2- 172.21.15.96:0/2041585475 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f37980d46c0 0x7f37980b26f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:23.381 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.379+0000 7f37a0200640 1 --2- 172.21.15.96:0/2041585475 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f37980d0350 0x7f379814e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:23.381 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.379+0000 7f37a0200640 1 -- 172.21.15.96:0/2041585475 >> 172.21.15.96:0/2041585475 conn(0x7f379809da70 msgr2=0x7f37981610f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:23.381 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.379+0000 7f37a0200640 1 -- 172.21.15.96:0/2041585475 shutdown_connections 2024-05-28T06:05:23.381 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.379+0000 7f37a0200640 1 -- 172.21.15.96:0/2041585475 wait complete. 2024-05-28T06:05:23.399 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2024-05-28T06:05:23.596 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.594+0000 7fa9dce00640 1 Processor -- start 2024-05-28T06:05:23.596 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.594+0000 7fa9dce00640 1 -- start start 2024-05-28T06:05:23.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.595+0000 7fa9dce00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa9d80b4e10 0x7fa9d80cde30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:23.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.596+0000 7fa9dce00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fa9d80ce370 0x7fa9d80d0590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:23.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.596+0000 7fa9dce00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9d80b51f0 con 0x7fa9d80d0ad0 2024-05-28T06:05:23.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.596+0000 7fa9dce00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fa9d80d0f20 con 0x7fa9d80ce370 2024-05-28T06:05:23.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.596+0000 7fa9dce00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fa9d80aef60 con 0x7fa9d80b4e10 2024-05-28T06:05:23.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.596+0000 7fa9d5e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa9d80b4e10 0x7fa9d80cde30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:23.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.596+0000 7fa9cfe00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fa9d80ce370 0x7fa9d80d0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:23.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.596+0000 7fa9d5e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa9d80b4e10 0x7fa9d80cde30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.96:53448/0 (socket says 172.21.15.96:53448) 2024-05-28T06:05:23.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.596+0000 7fa9d5e00640 1 -- 172.21.15.96:0/361188163 learned_addr learned my addr 172.21.15.96:0/361188163 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:23.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.597+0000 7fa9cfe00640 1 -- 172.21.15.96:0/361188163 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa9d80b4e10 msgr2=0x7fa9d80cde30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:23.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.597+0000 7fa9cfe00640 1 --2- 172.21.15.96:0/361188163 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa9d80b4e10 0x7fa9d80cde30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:23.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.597+0000 7fa9cfe00640 1 -- 172.21.15.96:0/361188163 >> v1:172.21.15.123:6789/0 conn(0x7fa9d80d0ad0 legacy=0x7fa9d8099090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:23.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.597+0000 7fa9cfe00640 1 -- 172.21.15.96:0/361188163 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9d80ad1d0 con 0x7fa9d80ce370 2024-05-28T06:05:23.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.598+0000 7fa9cfe00640 1 --2- 172.21.15.96:0/361188163 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa9d80ce370 0x7fa9d80d0590 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fa9c000ba40 tx=0x7fa9c002c7b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd9f0523cf9bc89 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:23.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.599+0000 7fa9d5400640 1 -- 172.21.15.96:0/361188163 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9c0031710 con 0x7fa9d80ce370 2024-05-28T06:05:23.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.599+0000 7fa9d5400640 1 -- 172.21.15.96:0/361188163 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9c00318b0 con 0x7fa9d80ce370 2024-05-28T06:05:23.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.599+0000 7fa9d5400640 1 -- 172.21.15.96:0/361188163 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9c0031c80 con 0x7fa9d80ce370 2024-05-28T06:05:23.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.600+0000 7fa9dce00640 1 -- 172.21.15.96:0/361188163 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa9d80ce370 msgr2=0x7fa9d80d0590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:23.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.600+0000 7fa9dce00640 1 --2- 172.21.15.96:0/361188163 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa9d80ce370 0x7fa9d80d0590 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fa9c000ba40 tx=0x7fa9c002c7b0 comp rx=0 tx=0).stop 2024-05-28T06:05:23.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.600+0000 7fa9dce00640 1 -- 172.21.15.96:0/361188163 shutdown_connections 2024-05-28T06:05:23.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.600+0000 7fa9dce00640 1 --2- 172.21.15.96:0/361188163 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa9d80ce370 0x7fa9d80d0590 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:23.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.600+0000 7fa9dce00640 1 --2- 172.21.15.96:0/361188163 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa9d80b4e10 0x7fa9d80cde30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:23.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.600+0000 7fa9dce00640 1 -- 172.21.15.96:0/361188163 >> 172.21.15.96:0/361188163 conn(0x7fa9d806bae0 msgr2=0x7fa9d806bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:23.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.600+0000 7fa9dce00640 1 -- 172.21.15.96:0/361188163 shutdown_connections 2024-05-28T06:05:23.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.600+0000 7fa9dce00640 1 -- 172.21.15.96:0/361188163 wait complete. 2024-05-28T06:05:23.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.602+0000 7fa9dce00640 1 Processor -- start 2024-05-28T06:05:23.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.602+0000 7fa9dce00640 1 -- start start 2024-05-28T06:05:23.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.603+0000 7fa9dce00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa9d80b4e10 0x7fa9d8099a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:23.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.603+0000 7fa9dce00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa9d80ce370 0x7fa9d81537d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:23.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.603+0000 7fa9dce00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa9d80d0ad0 0x7fa9d80b23c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:23.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.603+0000 7fa9dce00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa9d80ce880 con 0x7fa9d80d0ad0 2024-05-28T06:05:23.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.603+0000 7fa9dce00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa9d80ad750 con 0x7fa9d80b4e10 2024-05-28T06:05:23.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.603+0000 7fa9dce00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fa9d80cea00 con 0x7fa9d80ce370 2024-05-28T06:05:23.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.603+0000 7fa9d6800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa9d80d0ad0 0x7fa9d80b23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:23.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.603+0000 7fa9d6800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa9d80d0ad0 0x7fa9d80b23c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49496/0 (socket says 172.21.15.96:49496) 2024-05-28T06:05:23.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.603+0000 7fa9d6800640 1 -- 172.21.15.96:0/1547463752 learned_addr learned my addr 172.21.15.96:0/1547463752 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:23.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.603+0000 7fa9d5e00640 1 --2- 172.21.15.96:0/1547463752 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa9d80b4e10 0x7fa9d8099a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:23.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.603+0000 7fa9cfe00640 1 --2- 172.21.15.96:0/1547463752 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa9d80ce370 0x7fa9d81537d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:23.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.604+0000 7fa9d6800640 1 -- 172.21.15.96:0/1547463752 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa9d80ce370 msgr2=0x7fa9d81537d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:23.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.604+0000 7fa9d6800640 1 --2- 172.21.15.96:0/1547463752 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa9d80ce370 0x7fa9d81537d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:23.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.604+0000 7fa9d6800640 1 -- 172.21.15.96:0/1547463752 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa9d80b4e10 msgr2=0x7fa9d8099a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:23.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.604+0000 7fa9d6800640 1 --2- 172.21.15.96:0/1547463752 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa9d80b4e10 0x7fa9d8099a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:23.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.604+0000 7fa9d6800640 1 -- 172.21.15.96:0/1547463752 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9d80b1320 con 0x7fa9d80d0ad0 2024-05-28T06:05:23.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.604+0000 7fa9d6800640 1 --2- 172.21.15.96:0/1547463752 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa9d80d0ad0 0x7fa9d80b23c0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fa9d000c6e0 tx=0x7fa9d000cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:23.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.604+0000 7fa9ce000640 1 -- 172.21.15.96:0/1547463752 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9d0003b70 con 0x7fa9d80d0ad0 2024-05-28T06:05:23.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.604+0000 7fa9ce000640 1 -- 172.21.15.96:0/1547463752 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9d0003d30 con 0x7fa9d80d0ad0 2024-05-28T06:05:23.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.604+0000 7fa9ce000640 1 -- 172.21.15.96:0/1547463752 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9d0005310 con 0x7fa9d80d0ad0 2024-05-28T06:05:23.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.604+0000 7fa9dce00640 1 -- 172.21.15.96:0/1547463752 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9d80b1600 con 0x7fa9d80d0ad0 2024-05-28T06:05:23.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.604+0000 7fa9dce00640 1 -- 172.21.15.96:0/1547463752 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9d80b1bc0 con 0x7fa9d80d0ad0 2024-05-28T06:05:23.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.606+0000 7fa9dce00640 1 -- 172.21.15.96:0/1547463752 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9d80689e0 con 0x7fa9d80d0ad0 2024-05-28T06:05:23.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.606+0000 7fa9ce000640 1 -- 172.21.15.96:0/1547463752 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa9d00054b0 con 0x7fa9d80d0ad0 2024-05-28T06:05:23.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.607+0000 7fa9ce000640 1 --2- 172.21.15.96:0/1547463752 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa9b406c8f0 0x7fa9b406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:23.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.607+0000 7fa9ce000640 1 -- 172.21.15.96:0/1547463752 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 4857+0+0 (secure 0 0 0) 0x7fa9d008b780 con 0x7fa9d80d0ad0 2024-05-28T06:05:23.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.608+0000 7fa9d5e00640 1 --2- 172.21.15.96:0/1547463752 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa9b406c8f0 0x7fa9b406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:23.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.609+0000 7fa9d5e00640 1 --2- 172.21.15.96:0/1547463752 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa9b406c8f0 0x7fa9b406edb0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa9c8009b60 tx=0x7fa9c8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:23.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.612+0000 7fa9ce000640 1 -- 172.21.15.96:0/1547463752 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa9d0059e60 con 0x7fa9d80d0ad0 2024-05-28T06:05:23.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.892+0000 7fa9dce00640 1 -- 172.21.15.96:0/1547463752 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) -- 0x7fa9d806f540 con 0x7fa9d80d0ad0 2024-05-28T06:05:23.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.897+0000 7fa9ce000640 1 -- 172.21.15.96:0/1547463752 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v8) ==== 90+0+0 (secure 0 0 0) 0x7fa9d0059760 con 0x7fa9d80d0ad0 2024-05-28T06:05:23.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.900+0000 7fa9dce00640 1 -- 172.21.15.96:0/1547463752 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa9b406c8f0 msgr2=0x7fa9b406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:23.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.900+0000 7fa9dce00640 1 --2- 172.21.15.96:0/1547463752 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa9b406c8f0 0x7fa9b406edb0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa9c8009b60 tx=0x7fa9c8008040 comp rx=0 tx=0).stop 2024-05-28T06:05:23.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.900+0000 7fa9dce00640 1 -- 172.21.15.96:0/1547463752 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa9d80d0ad0 msgr2=0x7fa9d80b23c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:23.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.900+0000 7fa9dce00640 1 --2- 172.21.15.96:0/1547463752 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa9d80d0ad0 0x7fa9d80b23c0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fa9d000c6e0 tx=0x7fa9d000cba0 comp rx=0 tx=0).stop 2024-05-28T06:05:23.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.901+0000 7fa9dce00640 1 -- 172.21.15.96:0/1547463752 shutdown_connections 2024-05-28T06:05:23.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.901+0000 7fa9dce00640 1 --2- 172.21.15.96:0/1547463752 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa9b406c8f0 0x7fa9b406edb0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:23.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.901+0000 7fa9dce00640 1 --2- 172.21.15.96:0/1547463752 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa9d80d0ad0 0x7fa9d80b23c0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:23.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.901+0000 7fa9dce00640 1 --2- 172.21.15.96:0/1547463752 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa9d80ce370 0x7fa9d81537d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:23.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.901+0000 7fa9dce00640 1 --2- 172.21.15.96:0/1547463752 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa9d80b4e10 0x7fa9d8099a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:23.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.901+0000 7fa9dce00640 1 -- 172.21.15.96:0/1547463752 >> 172.21.15.96:0/1547463752 conn(0x7fa9d806bae0 msgr2=0x7fa9d80ca840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:23.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.901+0000 7fa9dce00640 1 -- 172.21.15.96:0/1547463752 shutdown_connections 2024-05-28T06:05:23.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:23.901+0000 7fa9dce00640 1 -- 172.21.15.96:0/1547463752 wait complete. 2024-05-28T06:05:23.921 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2024-05-28T06:05:24.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.108+0000 7f6ee2200640 1 Processor -- start 2024-05-28T06:05:24.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.109+0000 7f6ee2200640 1 -- start start 2024-05-28T06:05:24.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.110+0000 7f6ee2200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f6edc158b80 0x7f6edc06dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:24.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.110+0000 7f6ee2200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f6edc06e160 0x7f6edc071bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:24.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.110+0000 7f6ee2200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6edc072510 con 0x7f6edc072100 2024-05-28T06:05:24.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.110+0000 7f6ee2200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f6edc0b42d0 con 0x7f6edc06e160 2024-05-28T06:05:24.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.110+0000 7f6ee2200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f6edc06e6e0 con 0x7f6edc158b80 2024-05-28T06:05:24.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.110+0000 7f6ed3e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f6edc06e160 0x7f6edc071bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:24.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.110+0000 7f6ee1800640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f6edc072100 0x7f6edc1621f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:48350/0 (socket says 172.21.15.96:48350) 2024-05-28T06:05:24.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.110+0000 7f6ee1800640 1 -- 172.21.15.96:0/1754728243 learned_addr learned my addr 172.21.15.96:0/1754728243 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:05:24.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.111+0000 7f6ee0e00640 1 --2- 172.21.15.96:0/1754728243 >> v2:172.21.15.123:3300/0 conn(0x7f6edc158b80 0x7f6edc06dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:24.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.111+0000 7f6ed3e00640 1 -- 172.21.15.96:0/1754728243 >> v2:172.21.15.123:3300/0 conn(0x7f6edc158b80 msgr2=0x7f6edc06dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:24.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.111+0000 7f6ed3e00640 1 --2- 172.21.15.96:0/1754728243 >> v2:172.21.15.123:3300/0 conn(0x7f6edc158b80 0x7f6edc06dc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:24.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.111+0000 7f6ed3e00640 1 -- 172.21.15.96:0/1754728243 >> v1:172.21.15.96:6789/0 conn(0x7f6edc072100 legacy=0x7f6edc1621f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:24.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.111+0000 7f6edbe00640 1 -- 172.21.15.96:0/1754728243 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2447034896 0 0) 0x7f6ed4001490 con 0x7f6edc072100 2024-05-28T06:05:24.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.111+0000 7f6ed3e00640 1 -- 172.21.15.96:0/1754728243 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6edc06e9c0 con 0x7f6edc06e160 2024-05-28T06:05:24.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.112+0000 7f6ed3e00640 1 --2- 172.21.15.96:0/1754728243 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6edc06e160 0x7f6edc071bc0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f6edc06c830 tx=0x7f6ecc02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a979f0fd63941ccf server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:24.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.112+0000 7f6edbe00640 1 -- 172.21.15.96:0/1754728243 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ecc037070 con 0x7f6edc06e160 2024-05-28T06:05:24.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.112+0000 7f6edbe00640 1 -- 172.21.15.96:0/1754728243 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ecc032440 con 0x7f6edc06e160 2024-05-28T06:05:24.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.112+0000 7f6edbe00640 1 -- 172.21.15.96:0/1754728243 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ecc032810 con 0x7f6edc06e160 2024-05-28T06:05:24.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.113+0000 7f6ee2200640 1 -- 172.21.15.96:0/1754728243 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6edc06e160 msgr2=0x7f6edc071bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:24.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.113+0000 7f6ee2200640 1 --2- 172.21.15.96:0/1754728243 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6edc06e160 0x7f6edc071bc0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f6edc06c830 tx=0x7f6ecc02a1a0 comp rx=0 tx=0).stop 2024-05-28T06:05:24.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.113+0000 7f6ee2200640 1 -- 172.21.15.96:0/1754728243 shutdown_connections 2024-05-28T06:05:24.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.113+0000 7f6ee2200640 1 --2- 172.21.15.96:0/1754728243 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6edc06e160 0x7f6edc071bc0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:24.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.113+0000 7f6ee2200640 1 --2- 172.21.15.96:0/1754728243 >> v2:172.21.15.123:3300/0 conn(0x7f6edc158b80 0x7f6edc06dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:24.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.113+0000 7f6ee2200640 1 -- 172.21.15.96:0/1754728243 >> 172.21.15.96:0/1754728243 conn(0x7f6edc0b1fc0 msgr2=0x7f6edc0b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:24.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.113+0000 7f6ee2200640 1 -- 172.21.15.96:0/1754728243 shutdown_connections 2024-05-28T06:05:24.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.113+0000 7f6ee2200640 1 -- 172.21.15.96:0/1754728243 wait complete. 2024-05-28T06:05:24.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.115+0000 7f6ee2200640 1 Processor -- start 2024-05-28T06:05:24.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.115+0000 7f6ee2200640 1 -- start start 2024-05-28T06:05:24.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.116+0000 7f6ee2200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6edc06e160 0x7f6edc1477d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:24.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.116+0000 7f6ee2200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6edc072100 0x7f6edc09a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:24.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.116+0000 7f6ee2200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6edc158b80 0x7f6edc153d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:24.119 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.116+0000 7f6ee2200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f6edc06f260 con 0x7f6edc072100 2024-05-28T06:05:24.119 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.116+0000 7f6ee2200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f6edc06ef80 con 0x7f6edc06e160 2024-05-28T06:05:24.119 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.116+0000 7f6ee2200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f6edc0b2d30 con 0x7f6edc158b80 2024-05-28T06:05:24.119 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.116+0000 7f6ed3e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6edc072100 0x7f6edc09a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:24.119 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.116+0000 7f6ed3e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6edc072100 0x7f6edc09a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49510/0 (socket says 172.21.15.96:49510) 2024-05-28T06:05:24.119 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.116+0000 7f6ed3e00640 1 -- 172.21.15.96:0/1848309842 learned_addr learned my addr 172.21.15.96:0/1848309842 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:24.119 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.116+0000 7f6ee0e00640 1 --2- 172.21.15.96:0/1848309842 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6edc06e160 0x7f6edc1477d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:24.119 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.116+0000 7f6ed3e00640 1 -- 172.21.15.96:0/1848309842 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6edc158b80 msgr2=0x7f6edc153d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:24.119 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.116+0000 7f6ed3e00640 1 --2- 172.21.15.96:0/1848309842 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6edc158b80 0x7f6edc153d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:24.119 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.116+0000 7f6ed3e00640 1 -- 172.21.15.96:0/1848309842 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6edc06e160 msgr2=0x7f6edc1477d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:24.119 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.116+0000 7f6ed3e00640 1 --2- 172.21.15.96:0/1848309842 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6edc06e160 0x7f6edc1477d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:24.120 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.116+0000 7f6ed3e00640 1 -- 172.21.15.96:0/1848309842 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6edc09d3d0 con 0x7f6edc072100 2024-05-28T06:05:24.120 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.117+0000 7f6ed3e00640 1 --2- 172.21.15.96:0/1848309842 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6edc072100 0x7f6edc09a7e0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f6ecc000c00 tx=0x7f6ecc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:24.120 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.117+0000 7f6ed9600640 1 -- 172.21.15.96:0/1848309842 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ecc037040 con 0x7f6edc072100 2024-05-28T06:05:24.120 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.117+0000 7f6ed9600640 1 -- 172.21.15.96:0/1848309842 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ecc042070 con 0x7f6edc072100 2024-05-28T06:05:24.120 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.117+0000 7f6ee2200640 1 -- 172.21.15.96:0/1848309842 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6edc09d6b0 con 0x7f6edc072100 2024-05-28T06:05:24.120 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.117+0000 7f6ed9600640 1 -- 172.21.15.96:0/1848309842 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ecc030bd0 con 0x7f6edc072100 2024-05-28T06:05:24.120 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.117+0000 7f6ee2200640 1 -- 172.21.15.96:0/1848309842 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6edc09ad20 con 0x7f6edc072100 2024-05-28T06:05:24.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.118+0000 7f6ee2200640 1 -- 172.21.15.96:0/1848309842 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6edc0689e0 con 0x7f6edc072100 2024-05-28T06:05:24.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.119+0000 7f6ed9600640 1 -- 172.21.15.96:0/1848309842 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6ecc047460 con 0x7f6edc072100 2024-05-28T06:05:24.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.120+0000 7f6ed9600640 1 --2- 172.21.15.96:0/1848309842 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6eb806c920 0x7f6eb806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:24.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.120+0000 7f6ed9600640 1 -- 172.21.15.96:0/1848309842 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 4857+0+0 (secure 0 0 0) 0x7f6ecc0ac3e0 con 0x7f6edc072100 2024-05-28T06:05:24.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.121+0000 7f6ee0e00640 1 --2- 172.21.15.96:0/1848309842 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6eb806c920 0x7f6eb806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:24.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.122+0000 7f6ee0e00640 1 --2- 172.21.15.96:0/1848309842 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6eb806c920 0x7f6eb806ede0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f6ec4006fd0 tx=0x7f6ec4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:24.125 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.124+0000 7f6ed9600640 1 -- 172.21.15.96:0/1848309842 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6ecc032e20 con 0x7f6edc072100 2024-05-28T06:05:24.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.395+0000 7f6ee2200640 1 -- 172.21.15.96:0/1848309842 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) -- 0x7f6edc161ce0 con 0x7f6edc072100 2024-05-28T06:05:24.400 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.399+0000 7f6ed9600640 1 -- 172.21.15.96:0/1848309842 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v9) ==== 90+0+0 (secure 0 0 0) 0x7f6ecc07ab50 con 0x7f6edc072100 2024-05-28T06:05:24.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.401+0000 7f6ee2200640 1 -- 172.21.15.96:0/1848309842 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6eb806c920 msgr2=0x7f6eb806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:24.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.402+0000 7f6ee2200640 1 --2- 172.21.15.96:0/1848309842 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6eb806c920 0x7f6eb806ede0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f6ec4006fd0 tx=0x7f6ec4008040 comp rx=0 tx=0).stop 2024-05-28T06:05:24.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.402+0000 7f6ee2200640 1 -- 172.21.15.96:0/1848309842 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6edc072100 msgr2=0x7f6edc09a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:24.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.402+0000 7f6ee2200640 1 --2- 172.21.15.96:0/1848309842 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6edc072100 0x7f6edc09a7e0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f6ecc000c00 tx=0x7f6ecc0046c0 comp rx=0 tx=0).stop 2024-05-28T06:05:24.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.402+0000 7f6ee2200640 1 -- 172.21.15.96:0/1848309842 shutdown_connections 2024-05-28T06:05:24.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.402+0000 7f6ee2200640 1 --2- 172.21.15.96:0/1848309842 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6eb806c920 0x7f6eb806ede0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:24.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.402+0000 7f6ee2200640 1 --2- 172.21.15.96:0/1848309842 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6edc158b80 0x7f6edc153d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:24.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.402+0000 7f6ee2200640 1 --2- 172.21.15.96:0/1848309842 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6edc072100 0x7f6edc09a7e0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:24.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.402+0000 7f6ee2200640 1 --2- 172.21.15.96:0/1848309842 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6edc06e160 0x7f6edc1477d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:24.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.402+0000 7f6ee2200640 1 -- 172.21.15.96:0/1848309842 >> 172.21.15.96:0/1848309842 conn(0x7f6edc0b1fc0 msgr2=0x7f6edc155810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:24.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.402+0000 7f6ee2200640 1 -- 172.21.15.96:0/1848309842 shutdown_connections 2024-05-28T06:05:24.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.402+0000 7f6ee2200640 1 -- 172.21.15.96:0/1848309842 wait complete. 2024-05-28T06:05:24.421 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2024-05-28T06:05:24.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.612+0000 7f60b3e00640 1 Processor -- start 2024-05-28T06:05:24.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.612+0000 7f60b3e00640 1 -- start start 2024-05-28T06:05:24.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.613+0000 7f60b3e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f60b415e530 0x7f60b406f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:24.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.613+0000 7f60b3e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f60b40afff0 0x7f60b40b5080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:24.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.613+0000 7f60b3e00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60b406e500 con 0x7f60b4070700 2024-05-28T06:05:24.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.613+0000 7f60b3e00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f60b406e7e0 con 0x7f60b40afff0 2024-05-28T06:05:24.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.613+0000 7f60b3e00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f60b406eac0 con 0x7f60b415e530 2024-05-28T06:05:24.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.614+0000 7f60b3400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f60b40afff0 0x7f60b40b5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:24.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.614+0000 7f60b3400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f60b40afff0 0x7f60b40b5080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49514/0 (socket says 172.21.15.96:49514) 2024-05-28T06:05:24.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.614+0000 7f60b3400640 1 -- 172.21.15.96:0/507740373 learned_addr learned my addr 172.21.15.96:0/507740373 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:24.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.614+0000 7f60abe00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f60b4070700 0x7f60b40af8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:48356/0 (socket says 172.21.15.96:48356) 2024-05-28T06:05:24.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.614+0000 7f60b2a00640 1 --2- 172.21.15.96:0/507740373 >> v2:172.21.15.123:3300/0 conn(0x7f60b415e530 0x7f60b406f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:24.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.614+0000 7f60b3400640 1 -- 172.21.15.96:0/507740373 >> v2:172.21.15.123:3300/0 conn(0x7f60b415e530 msgr2=0x7f60b406f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:24.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.614+0000 7f60b3400640 1 --2- 172.21.15.96:0/507740373 >> v2:172.21.15.123:3300/0 conn(0x7f60b415e530 0x7f60b406f5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:24.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.614+0000 7f60b3400640 1 -- 172.21.15.96:0/507740373 >> v1:172.21.15.96:6789/0 conn(0x7f60b4070700 legacy=0x7f60b40af8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:24.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.614+0000 7f60b3400640 1 -- 172.21.15.96:0/507740373 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60b406eda0 con 0x7f60b40afff0 2024-05-28T06:05:24.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.615+0000 7f60b3400640 1 --2- 172.21.15.96:0/507740373 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f60b40afff0 0x7f60b40b5080 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f60b4075940 tx=0x7f60ac02a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bc567bd8b26536a server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:24.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.615+0000 7f60b2000640 1 -- 172.21.15.96:0/507740373 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60ac036070 con 0x7f60b40afff0 2024-05-28T06:05:24.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.615+0000 7f60b2000640 1 -- 172.21.15.96:0/507740373 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60ac031600 con 0x7f60b40afff0 2024-05-28T06:05:24.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.615+0000 7f60b2000640 1 -- 172.21.15.96:0/507740373 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60ac0319d0 con 0x7f60b40afff0 2024-05-28T06:05:24.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.616+0000 7f60b3e00640 1 -- 172.21.15.96:0/507740373 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f60b40afff0 msgr2=0x7f60b40b5080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:24.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.616+0000 7f60b3e00640 1 --2- 172.21.15.96:0/507740373 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f60b40afff0 0x7f60b40b5080 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f60b4075940 tx=0x7f60ac02a6a0 comp rx=0 tx=0).stop 2024-05-28T06:05:24.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.616+0000 7f60b3e00640 1 -- 172.21.15.96:0/507740373 shutdown_connections 2024-05-28T06:05:24.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.616+0000 7f60b3e00640 1 --2- 172.21.15.96:0/507740373 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f60b40afff0 0x7f60b40b5080 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:24.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.616+0000 7f60b3e00640 1 --2- 172.21.15.96:0/507740373 >> v2:172.21.15.123:3300/0 conn(0x7f60b415e530 0x7f60b406f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:24.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.616+0000 7f60b3e00640 1 -- 172.21.15.96:0/507740373 >> 172.21.15.96:0/507740373 conn(0x7f60b406b900 msgr2=0x7f60b406bce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:24.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.616+0000 7f60b3e00640 1 -- 172.21.15.96:0/507740373 shutdown_connections 2024-05-28T06:05:24.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.616+0000 7f60b3e00640 1 -- 172.21.15.96:0/507740373 wait complete. 2024-05-28T06:05:24.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.618+0000 7f60b3e00640 1 Processor -- start 2024-05-28T06:05:24.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.618+0000 7f60b3e00640 1 -- start start 2024-05-28T06:05:24.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.619+0000 7f60b3e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60b4070700 0x7f60b41f8a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:24.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.619+0000 7f60b3e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f60b40afff0 0x7f60b41f4900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:24.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.619+0000 7f60b3e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f60b415e530 0x7f60b41503d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:24.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.619+0000 7f60b3e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f60b40af570 con 0x7f60b40afff0 2024-05-28T06:05:24.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.619+0000 7f60b3e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f60b415e910 con 0x7f60b4070700 2024-05-28T06:05:24.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.619+0000 7f60b3e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f60b40af6f0 con 0x7f60b415e530 2024-05-28T06:05:24.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.619+0000 7f60abe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f60b40afff0 0x7f60b41f4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:24.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.619+0000 7f60abe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f60b40afff0 0x7f60b41f4900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49516/0 (socket says 172.21.15.96:49516) 2024-05-28T06:05:24.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.619+0000 7f60abe00640 1 -- 172.21.15.96:0/3432665472 learned_addr learned my addr 172.21.15.96:0/3432665472 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:24.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.620+0000 7f60abe00640 1 -- 172.21.15.96:0/3432665472 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f60b415e530 msgr2=0x7f60b41503d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:24.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.620+0000 7f60abe00640 1 --2- 172.21.15.96:0/3432665472 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f60b415e530 0x7f60b41503d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:24.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.620+0000 7f60abe00640 1 -- 172.21.15.96:0/3432665472 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60b4070700 msgr2=0x7f60b41f8a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:24.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.620+0000 7f60abe00640 1 --2- 172.21.15.96:0/3432665472 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60b4070700 0x7f60b41f8a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:24.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.620+0000 7f60abe00640 1 -- 172.21.15.96:0/3432665472 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60b41f4e40 con 0x7f60b40afff0 2024-05-28T06:05:24.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.620+0000 7f60abe00640 1 --2- 172.21.15.96:0/3432665472 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f60b40afff0 0x7f60b41f4900 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f609c00c430 tx=0x7f609c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:24.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.620+0000 7f60aaa00640 1 -- 172.21.15.96:0/3432665472 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f609c0074e0 con 0x7f60b40afff0 2024-05-28T06:05:24.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.620+0000 7f60aaa00640 1 -- 172.21.15.96:0/3432665472 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f609c007680 con 0x7f60b40afff0 2024-05-28T06:05:24.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.620+0000 7f60b3e00640 1 -- 172.21.15.96:0/3432665472 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60b41f5110 con 0x7f60b40afff0 2024-05-28T06:05:24.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.620+0000 7f60aaa00640 1 -- 172.21.15.96:0/3432665472 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f609c004a00 con 0x7f60b40afff0 2024-05-28T06:05:24.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.620+0000 7f60b3e00640 1 -- 172.21.15.96:0/3432665472 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60b41f56d0 con 0x7f60b40afff0 2024-05-28T06:05:24.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.622+0000 7f60b3e00640 1 -- 172.21.15.96:0/3432665472 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60b40689e0 con 0x7f60b40afff0 2024-05-28T06:05:24.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.622+0000 7f60aaa00640 1 -- 172.21.15.96:0/3432665472 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f609c01a020 con 0x7f60b40afff0 2024-05-28T06:05:24.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.623+0000 7f60aaa00640 1 --2- 172.21.15.96:0/3432665472 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f609006c8f0 0x7f609006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:24.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.623+0000 7f60aaa00640 1 -- 172.21.15.96:0/3432665472 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 4857+0+0 (secure 0 0 0) 0x7f609c08afa0 con 0x7f60b40afff0 2024-05-28T06:05:24.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.624+0000 7f60b2a00640 1 --2- 172.21.15.96:0/3432665472 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f609006c8f0 0x7f609006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:24.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.625+0000 7f60b2a00640 1 --2- 172.21.15.96:0/3432665472 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f609006c8f0 0x7f609006edb0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f60b40b5c30 tx=0x7f60a4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:24.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.627+0000 7f60aaa00640 1 -- 172.21.15.96:0/3432665472 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f609c0597b0 con 0x7f60b40afff0 2024-05-28T06:05:24.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.877+0000 7f60b3e00640 1 -- 172.21.15.96:0/3432665472 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) -- 0x7f60b4070080 con 0x7f60b40afff0 2024-05-28T06:05:24.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.881+0000 7f60aaa00640 1 -- 172.21.15.96:0/3432665472 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v10) ==== 90+0+0 (secure 0 0 0) 0x7f609c0590b0 con 0x7f60b40afff0 2024-05-28T06:05:24.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.884+0000 7f60b3e00640 1 -- 172.21.15.96:0/3432665472 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f609006c8f0 msgr2=0x7f609006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:24.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.884+0000 7f60b3e00640 1 --2- 172.21.15.96:0/3432665472 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f609006c8f0 0x7f609006edb0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f60b40b5c30 tx=0x7f60a4008040 comp rx=0 tx=0).stop 2024-05-28T06:05:24.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.884+0000 7f60b3e00640 1 -- 172.21.15.96:0/3432665472 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f60b40afff0 msgr2=0x7f60b41f4900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:24.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.884+0000 7f60b3e00640 1 --2- 172.21.15.96:0/3432665472 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f60b40afff0 0x7f60b41f4900 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f609c00c430 tx=0x7f609c00c8f0 comp rx=0 tx=0).stop 2024-05-28T06:05:24.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.884+0000 7f60b3e00640 1 -- 172.21.15.96:0/3432665472 shutdown_connections 2024-05-28T06:05:24.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.884+0000 7f60b3e00640 1 --2- 172.21.15.96:0/3432665472 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f609006c8f0 0x7f609006edb0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:24.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.884+0000 7f60b3e00640 1 --2- 172.21.15.96:0/3432665472 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f60b415e530 0x7f60b41503d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:24.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.884+0000 7f60b3e00640 1 --2- 172.21.15.96:0/3432665472 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f60b40afff0 0x7f60b41f4900 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:24.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.884+0000 7f60b3e00640 1 --2- 172.21.15.96:0/3432665472 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60b4070700 0x7f60b41f8a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:24.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.884+0000 7f60b3e00640 1 -- 172.21.15.96:0/3432665472 >> 172.21.15.96:0/3432665472 conn(0x7f60b406b900 msgr2=0x7f60b40b0640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:24.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.884+0000 7f60b3e00640 1 -- 172.21.15.96:0/3432665472 shutdown_connections 2024-05-28T06:05:24.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:24.884+0000 7f60b3e00640 1 -- 172.21.15.96:0/3432665472 wait complete. 2024-05-28T06:05:24.904 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2024-05-28T06:05:25.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.095+0000 7f8e3be00640 1 Processor -- start 2024-05-28T06:05:25.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.096+0000 7f8e3be00640 1 -- start start 2024-05-28T06:05:25.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.097+0000 7f8e3be00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f8e3c0d16a0 0x7f8e3c0cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:25.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.097+0000 7f8e3be00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e3c0cfce0 con 0x7f8e3c15e870 2024-05-28T06:05:25.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.097+0000 7f8e3be00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e3c15ec50 con 0x7f8e3c0cf870 2024-05-28T06:05:25.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.097+0000 7f8e3be00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f8e3c0d2150 con 0x7f8e3c0d16a0 2024-05-28T06:05:25.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.098+0000 7f8e3a000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f8e3c0d16a0 0x7f8e3c0cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:25.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.098+0000 7f8e3aa00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f8e3c15e870 0x7f8e3c0d0f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:48366/0 (socket says 172.21.15.96:48366) 2024-05-28T06:05:25.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.098+0000 7f8e3aa00640 1 -- 172.21.15.96:0/2508605174 learned_addr learned my addr 172.21.15.96:0/2508605174 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:05:25.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.098+0000 7f8e3a000640 1 -- 172.21.15.96:0/2508605174 >> v1:172.21.15.123:6789/0 conn(0x7f8e3c0cf870 legacy=0x7f8e3c0d25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:25.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.098+0000 7f8e3a000640 1 -- 172.21.15.96:0/2508605174 >> v1:172.21.15.96:6789/0 conn(0x7f8e3c15e870 legacy=0x7f8e3c0d0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:25.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.098+0000 7f8e39600640 1 -- 172.21.15.96:0/2508605174 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 208101717 0 0) 0x7f8e3c0cfce0 con 0x7f8e3c15e870 2024-05-28T06:05:25.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.098+0000 7f8e3a000640 1 -- 172.21.15.96:0/2508605174 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e3c0cff80 con 0x7f8e3c0d16a0 2024-05-28T06:05:25.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.099+0000 7f8e3a000640 1 --2- 172.21.15.96:0/2508605174 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8e3c0d16a0 0x7f8e3c0cf330 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f8e24009de0 tx=0x7f8e2402a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad637ca3078c2f29 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:25.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.100+0000 7f8e39600640 1 -- 172.21.15.96:0/2508605174 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e24007740 con 0x7f8e3c0d16a0 2024-05-28T06:05:25.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.100+0000 7f8e39600640 1 -- 172.21.15.96:0/2508605174 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e240078e0 con 0x7f8e3c0d16a0 2024-05-28T06:05:25.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.100+0000 7f8e39600640 1 -- 172.21.15.96:0/2508605174 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e24007cb0 con 0x7f8e3c0d16a0 2024-05-28T06:05:25.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.100+0000 7f8e3be00640 1 -- 172.21.15.96:0/2508605174 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8e3c0d16a0 msgr2=0x7f8e3c0cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:25.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.100+0000 7f8e3be00640 1 --2- 172.21.15.96:0/2508605174 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8e3c0d16a0 0x7f8e3c0cf330 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f8e24009de0 tx=0x7f8e2402a710 comp rx=0 tx=0).stop 2024-05-28T06:05:25.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.100+0000 7f8e3be00640 1 -- 172.21.15.96:0/2508605174 shutdown_connections 2024-05-28T06:05:25.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.100+0000 7f8e3be00640 1 --2- 172.21.15.96:0/2508605174 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8e3c0d16a0 0x7f8e3c0cf330 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:25.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.100+0000 7f8e3be00640 1 -- 172.21.15.96:0/2508605174 >> 172.21.15.96:0/2508605174 conn(0x7f8e3c08bfc0 msgr2=0x7f8e3c0d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:25.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.100+0000 7f8e3be00640 1 -- 172.21.15.96:0/2508605174 shutdown_connections 2024-05-28T06:05:25.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.100+0000 7f8e3be00640 1 -- 172.21.15.96:0/2508605174 wait complete. 2024-05-28T06:05:25.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.102+0000 7f8e3be00640 1 Processor -- start 2024-05-28T06:05:25.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.102+0000 7f8e3be00640 1 -- start start 2024-05-28T06:05:25.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.103+0000 7f8e3be00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8e3c0cf870 0x7f8e3c1612a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:25.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.103+0000 7f8e3be00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e3c0d16a0 0x7f8e3c0cc100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:25.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.103+0000 7f8e3be00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8e3c15e870 0x7f8e3c159660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:25.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.103+0000 7f8e3be00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f8e3c0d0820 con 0x7f8e3c15e870 2024-05-28T06:05:25.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.103+0000 7f8e3be00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f8e3c0d0540 con 0x7f8e3c0d16a0 2024-05-28T06:05:25.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.103+0000 7f8e3be00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f8e3c0d4fb0 con 0x7f8e3c0cf870 2024-05-28T06:05:25.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.103+0000 7f8e3b400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8e3c15e870 0x7f8e3c159660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:25.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.104+0000 7f8e3b400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8e3c15e870 0x7f8e3c159660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49534/0 (socket says 172.21.15.96:49534) 2024-05-28T06:05:25.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.104+0000 7f8e3b400640 1 -- 172.21.15.96:0/3111142988 learned_addr learned my addr 172.21.15.96:0/3111142988 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:25.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.104+0000 7f8e3aa00640 1 --2- 172.21.15.96:0/3111142988 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8e3c0cf870 0x7f8e3c1612a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:25.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.104+0000 7f8e3a000640 1 --2- 172.21.15.96:0/3111142988 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e3c0d16a0 0x7f8e3c0cc100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:25.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.104+0000 7f8e3b400640 1 -- 172.21.15.96:0/3111142988 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8e3c0cf870 msgr2=0x7f8e3c1612a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:25.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.104+0000 7f8e3b400640 1 --2- 172.21.15.96:0/3111142988 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8e3c0cf870 0x7f8e3c1612a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:25.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.104+0000 7f8e3b400640 1 -- 172.21.15.96:0/3111142988 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e3c0d16a0 msgr2=0x7f8e3c0cc100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:25.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.104+0000 7f8e3b400640 1 --2- 172.21.15.96:0/3111142988 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e3c0d16a0 0x7f8e3c0cc100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:25.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.104+0000 7f8e3b400640 1 -- 172.21.15.96:0/3111142988 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e3c0ccb10 con 0x7f8e3c15e870 2024-05-28T06:05:25.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.104+0000 7f8e3b400640 1 --2- 172.21.15.96:0/3111142988 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8e3c15e870 0x7f8e3c159660 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f8e3400c440 tx=0x7f8e3400c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:25.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.104+0000 7f8e32a00640 1 -- 172.21.15.96:0/3111142988 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e34011790 con 0x7f8e3c15e870 2024-05-28T06:05:25.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.105+0000 7f8e32a00640 1 -- 172.21.15.96:0/3111142988 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e34017070 con 0x7f8e3c15e870 2024-05-28T06:05:25.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.105+0000 7f8e32a00640 1 -- 172.21.15.96:0/3111142988 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e34011b70 con 0x7f8e3c15e870 2024-05-28T06:05:25.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.105+0000 7f8e3be00640 1 -- 172.21.15.96:0/3111142988 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e3c15b8c0 con 0x7f8e3c15e870 2024-05-28T06:05:25.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.105+0000 7f8e3be00640 1 -- 172.21.15.96:0/3111142988 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e3c15be80 con 0x7f8e3c15e870 2024-05-28T06:05:25.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.106+0000 7f8e3be00640 1 -- 172.21.15.96:0/3111142988 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e3c0689e0 con 0x7f8e3c15e870 2024-05-28T06:05:25.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.107+0000 7f8e32a00640 1 -- 172.21.15.96:0/3111142988 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8e34011d10 con 0x7f8e3c15e870 2024-05-28T06:05:25.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.108+0000 7f8e32a00640 1 --2- 172.21.15.96:0/3111142988 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8e1806c8f0 0x7f8e1806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:25.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.108+0000 7f8e32a00640 1 -- 172.21.15.96:0/3111142988 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 4857+0+0 (secure 0 0 0) 0x7f8e3408b680 con 0x7f8e3c15e870 2024-05-28T06:05:25.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.108+0000 7f8e3aa00640 1 --2- 172.21.15.96:0/3111142988 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8e1806c8f0 0x7f8e1806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:25.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.109+0000 7f8e3aa00640 1 --2- 172.21.15.96:0/3111142988 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8e1806c8f0 0x7f8e1806edb0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f8e2c000f60 tx=0x7f8e2c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:25.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.112+0000 7f8e32a00640 1 -- 172.21.15.96:0/3111142988 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8e34059d60 con 0x7f8e3c15e870 2024-05-28T06:05:25.380 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.379+0000 7f8e3be00640 1 -- 172.21.15.96:0/3111142988 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) -- 0x7f8e3c0ba500 con 0x7f8e3c15e870 2024-05-28T06:05:25.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.391+0000 7f8e32a00640 1 -- 172.21.15.96:0/3111142988 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v11) ==== 97+0+0 (secure 0 0 0) 0x7f8e34059660 con 0x7f8e3c15e870 2024-05-28T06:05:25.395 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.393+0000 7f8e3be00640 1 -- 172.21.15.96:0/3111142988 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8e1806c8f0 msgr2=0x7f8e1806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:25.395 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.394+0000 7f8e3be00640 1 --2- 172.21.15.96:0/3111142988 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8e1806c8f0 0x7f8e1806edb0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f8e2c000f60 tx=0x7f8e2c009040 comp rx=0 tx=0).stop 2024-05-28T06:05:25.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.394+0000 7f8e3be00640 1 -- 172.21.15.96:0/3111142988 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8e3c15e870 msgr2=0x7f8e3c159660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:25.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.394+0000 7f8e3be00640 1 --2- 172.21.15.96:0/3111142988 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8e3c15e870 0x7f8e3c159660 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f8e3400c440 tx=0x7f8e3400c900 comp rx=0 tx=0).stop 2024-05-28T06:05:25.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.394+0000 7f8e3be00640 1 -- 172.21.15.96:0/3111142988 shutdown_connections 2024-05-28T06:05:25.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.394+0000 7f8e3be00640 1 --2- 172.21.15.96:0/3111142988 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8e1806c8f0 0x7f8e1806edb0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:25.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.394+0000 7f8e3be00640 1 --2- 172.21.15.96:0/3111142988 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8e3c15e870 0x7f8e3c159660 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:25.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.394+0000 7f8e3be00640 1 --2- 172.21.15.96:0/3111142988 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e3c0d16a0 0x7f8e3c0cc100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:25.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.394+0000 7f8e3be00640 1 --2- 172.21.15.96:0/3111142988 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8e3c0cf870 0x7f8e3c1612a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:25.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.394+0000 7f8e3be00640 1 -- 172.21.15.96:0/3111142988 >> 172.21.15.96:0/3111142988 conn(0x7f8e3c08bfc0 msgr2=0x7f8e3c0b1ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:25.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.394+0000 7f8e3be00640 1 -- 172.21.15.96:0/3111142988 shutdown_connections 2024-05-28T06:05:25.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.394+0000 7f8e3be00640 1 -- 172.21.15.96:0/3111142988 wait complete. 2024-05-28T06:05:25.412 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2024-05-28T06:05:25.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.605+0000 7f5043000640 1 Processor -- start 2024-05-28T06:05:25.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.606+0000 7f5043000640 1 -- start start 2024-05-28T06:05:25.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.607+0000 7f5043000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f503c071bc0 0x7f503c072070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:25.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.607+0000 7f5043000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f503c0b32c0 0x7f503c0b36a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:25.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.607+0000 7f5043000640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f503c06e6e0 con 0x7f503c158b80 2024-05-28T06:05:25.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.607+0000 7f5043000640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f503c06e9c0 con 0x7f503c0b32c0 2024-05-28T06:05:25.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.607+0000 7f5043000640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f503c06eca0 con 0x7f503c071bc0 2024-05-28T06:05:25.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.607+0000 7f5042600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f503c0b32c0 0x7f503c0b36a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:25.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.608+0000 7f5042600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f503c0b32c0 0x7f503c0b36a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49546/0 (socket says 172.21.15.96:49546) 2024-05-28T06:05:25.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.608+0000 7f5042600640 1 -- 172.21.15.96:0/3991729754 learned_addr learned my addr 172.21.15.96:0/3991729754 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:25.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.608+0000 7f5041200640 1 --2- 172.21.15.96:0/3991729754 >> v2:172.21.15.123:3300/0 conn(0x7f503c071bc0 0x7f503c072070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:25.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.608+0000 7f5042600640 1 -- 172.21.15.96:0/3991729754 >> v2:172.21.15.123:3300/0 conn(0x7f503c071bc0 msgr2=0x7f503c072070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:25.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.608+0000 7f5042600640 1 --2- 172.21.15.96:0/3991729754 >> v2:172.21.15.123:3300/0 conn(0x7f503c071bc0 0x7f503c072070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:25.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.608+0000 7f5042600640 1 -- 172.21.15.96:0/3991729754 >> v1:172.21.15.123:6789/0 conn(0x7f503c158b80 legacy=0x7f503c06dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:25.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.608+0000 7f5042600640 1 -- 172.21.15.96:0/3991729754 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f503c06ef80 con 0x7f503c0b32c0 2024-05-28T06:05:25.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.609+0000 7f5042600640 1 --2- 172.21.15.96:0/3991729754 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f503c0b32c0 0x7f503c0b36a0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f503c162130 tx=0x7f503402a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0fae4c1eb97d6bf server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:25.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.609+0000 7f503be00640 1 -- 172.21.15.96:0/3991729754 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5034036070 con 0x7f503c0b32c0 2024-05-28T06:05:25.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.609+0000 7f503be00640 1 -- 172.21.15.96:0/3991729754 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5034031600 con 0x7f503c0b32c0 2024-05-28T06:05:25.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.609+0000 7f503be00640 1 -- 172.21.15.96:0/3991729754 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50340319d0 con 0x7f503c0b32c0 2024-05-28T06:05:25.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.610+0000 7f5043000640 1 -- 172.21.15.96:0/3991729754 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f503c0b32c0 msgr2=0x7f503c0b36a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:25.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.610+0000 7f5043000640 1 --2- 172.21.15.96:0/3991729754 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f503c0b32c0 0x7f503c0b36a0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f503c162130 tx=0x7f503402a6f0 comp rx=0 tx=0).stop 2024-05-28T06:05:25.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.610+0000 7f5043000640 1 -- 172.21.15.96:0/3991729754 shutdown_connections 2024-05-28T06:05:25.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.610+0000 7f5043000640 1 --2- 172.21.15.96:0/3991729754 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f503c0b32c0 0x7f503c0b36a0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:25.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.610+0000 7f5043000640 1 --2- 172.21.15.96:0/3991729754 >> v2:172.21.15.123:3300/0 conn(0x7f503c071bc0 0x7f503c072070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:25.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.610+0000 7f5043000640 1 -- 172.21.15.96:0/3991729754 >> 172.21.15.96:0/3991729754 conn(0x7f503c0b1fc0 msgr2=0x7f503c0b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:25.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.610+0000 7f5043000640 1 -- 172.21.15.96:0/3991729754 shutdown_connections 2024-05-28T06:05:25.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.610+0000 7f5043000640 1 -- 172.21.15.96:0/3991729754 wait complete. 2024-05-28T06:05:25.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.612+0000 7f5043000640 1 Processor -- start 2024-05-28T06:05:25.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.612+0000 7f5043000640 1 -- start start 2024-05-28T06:05:25.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.613+0000 7f5043000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f503c071bc0 0x7f503c1f3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:25.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.613+0000 7f5043000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f503c0b32c0 0x7f503c156d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:25.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.613+0000 7f5043000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f503c158b80 0x7f503c09cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:25.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.613+0000 7f5043000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f503c06f780 con 0x7f503c071bc0 2024-05-28T06:05:25.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.613+0000 7f5043000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f503c0b2d30 con 0x7f503c0b32c0 2024-05-28T06:05:25.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.613+0000 7f5043000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f503c09ad80 con 0x7f503c158b80 2024-05-28T06:05:25.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.613+0000 7f5041c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f503c071bc0 0x7f503c1f3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:25.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.613+0000 7f5041c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f503c071bc0 0x7f503c1f3fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49556/0 (socket says 172.21.15.96:49556) 2024-05-28T06:05:25.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.613+0000 7f5041c00640 1 -- 172.21.15.96:0/1331127969 learned_addr learned my addr 172.21.15.96:0/1331127969 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:25.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.614+0000 7f5041c00640 1 -- 172.21.15.96:0/1331127969 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f503c158b80 msgr2=0x7f503c09cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:25.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.614+0000 7f5041c00640 1 --2- 172.21.15.96:0/1331127969 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f503c158b80 0x7f503c09cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:25.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.614+0000 7f5041c00640 1 -- 172.21.15.96:0/1331127969 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f503c0b32c0 msgr2=0x7f503c156d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:25.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.614+0000 7f5041c00640 1 --2- 172.21.15.96:0/1331127969 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f503c0b32c0 0x7f503c156d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:25.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.614+0000 7f5041c00640 1 -- 172.21.15.96:0/1331127969 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f503c1f4500 con 0x7f503c071bc0 2024-05-28T06:05:25.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.614+0000 7f5041c00640 1 --2- 172.21.15.96:0/1331127969 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f503c071bc0 0x7f503c1f3fc0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f503000b3f0 tx=0x7f503000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:25.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.614+0000 7f5039600640 1 -- 172.21.15.96:0/1331127969 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5030015020 con 0x7f503c071bc0 2024-05-28T06:05:25.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.614+0000 7f5039600640 1 -- 172.21.15.96:0/1331127969 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5030011830 con 0x7f503c071bc0 2024-05-28T06:05:25.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.614+0000 7f5039600640 1 -- 172.21.15.96:0/1331127969 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5030011c20 con 0x7f503c071bc0 2024-05-28T06:05:25.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.614+0000 7f5043000640 1 -- 172.21.15.96:0/1331127969 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f503c09d710 con 0x7f503c071bc0 2024-05-28T06:05:25.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.614+0000 7f5043000640 1 -- 172.21.15.96:0/1331127969 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f503c09aa10 con 0x7f503c071bc0 2024-05-28T06:05:25.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.616+0000 7f5043000640 1 -- 172.21.15.96:0/1331127969 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f503c0689e0 con 0x7f503c071bc0 2024-05-28T06:05:25.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.616+0000 7f5039600640 1 -- 172.21.15.96:0/1331127969 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5030018020 con 0x7f503c071bc0 2024-05-28T06:05:25.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.617+0000 7f5039600640 1 --2- 172.21.15.96:0/1331127969 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f501c06c8f0 0x7f501c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:25.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.617+0000 7f5039600640 1 -- 172.21.15.96:0/1331127969 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 4857+0+0 (secure 0 0 0) 0x7f503008b420 con 0x7f503c071bc0 2024-05-28T06:05:25.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.618+0000 7f5041200640 1 --2- 172.21.15.96:0/1331127969 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f501c06c8f0 0x7f501c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:25.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.619+0000 7f5041200640 1 --2- 172.21.15.96:0/1331127969 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f501c06c8f0 0x7f501c06edb0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f503c156670 tx=0x7f5028008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:25.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.621+0000 7f5039600640 1 -- 172.21.15.96:0/1331127969 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f503001f030 con 0x7f503c071bc0 2024-05-28T06:05:25.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.894+0000 7f5043000640 1 -- 172.21.15.96:0/1331127969 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) -- 0x7f503c09a500 con 0x7f503c071bc0 2024-05-28T06:05:25.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.899+0000 7f5039600640 1 -- 172.21.15.96:0/1331127969 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v12) ==== 90+0+0 (secure 0 0 0) 0x7f5030059b90 con 0x7f503c071bc0 2024-05-28T06:05:25.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.901+0000 7f5043000640 1 -- 172.21.15.96:0/1331127969 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f501c06c8f0 msgr2=0x7f501c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:25.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.901+0000 7f5043000640 1 --2- 172.21.15.96:0/1331127969 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f501c06c8f0 0x7f501c06edb0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f503c156670 tx=0x7f5028008040 comp rx=0 tx=0).stop 2024-05-28T06:05:25.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.901+0000 7f5043000640 1 -- 172.21.15.96:0/1331127969 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f503c071bc0 msgr2=0x7f503c1f3fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:25.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.901+0000 7f5043000640 1 --2- 172.21.15.96:0/1331127969 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f503c071bc0 0x7f503c1f3fc0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f503000b3f0 tx=0x7f503000b8b0 comp rx=0 tx=0).stop 2024-05-28T06:05:25.904 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.902+0000 7f5043000640 1 -- 172.21.15.96:0/1331127969 shutdown_connections 2024-05-28T06:05:25.904 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.902+0000 7f5043000640 1 --2- 172.21.15.96:0/1331127969 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f501c06c8f0 0x7f501c06edb0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:25.904 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.902+0000 7f5043000640 1 --2- 172.21.15.96:0/1331127969 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f503c158b80 0x7f503c09cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:25.904 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.902+0000 7f5043000640 1 --2- 172.21.15.96:0/1331127969 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f503c0b32c0 0x7f503c156d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:25.904 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.902+0000 7f5043000640 1 --2- 172.21.15.96:0/1331127969 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f503c071bc0 0x7f503c1f3fc0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:25.904 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.902+0000 7f5043000640 1 -- 172.21.15.96:0/1331127969 >> 172.21.15.96:0/1331127969 conn(0x7f503c0b1fc0 msgr2=0x7f503c0b4d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:25.904 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.902+0000 7f5043000640 1 -- 172.21.15.96:0/1331127969 shutdown_connections 2024-05-28T06:05:25.904 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:25.902+0000 7f5043000640 1 -- 172.21.15.96:0/1331127969 wait complete. 2024-05-28T06:05:25.924 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:05:26.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.122+0000 7f6979a00640 1 Processor -- start 2024-05-28T06:05:26.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.122+0000 7f6979a00640 1 -- start start 2024-05-28T06:05:26.125 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.123+0000 7f6979a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f69740b4e10 0x7f69740cde30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:26.125 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.123+0000 7f6979a00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f69740ce370 0x7f69740d0590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:26.125 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.123+0000 7f6979a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f69740d0ad0 0x7f6974099090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:26.125 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.123+0000 7f6979a00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f69740d0f40 con 0x7f69740b4e10 2024-05-28T06:05:26.125 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.123+0000 7f6979a00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f69740b51f0 con 0x7f69740ce370 2024-05-28T06:05:26.125 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.123+0000 7f6979a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f69740ad1d0 con 0x7f69740d0ad0 2024-05-28T06:05:26.125 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.123+0000 7f6972a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f69740b4e10 0x7f69740cde30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:26.125 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.124+0000 7f6972a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f69740b4e10 0x7f69740cde30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49570/0 (socket says 172.21.15.96:49570) 2024-05-28T06:05:26.125 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.124+0000 7f6972a00640 1 -- 172.21.15.96:0/4173851267 learned_addr learned my addr 172.21.15.96:0/4173851267 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:26.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.124+0000 7f6972a00640 1 -- 172.21.15.96:0/4173851267 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f69740d0ad0 msgr2=0x7f6974099090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:26.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.124+0000 7f6972a00640 1 --2- 172.21.15.96:0/4173851267 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f69740d0ad0 0x7f6974099090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:26.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.124+0000 7f6972a00640 1 -- 172.21.15.96:0/4173851267 >> v2:172.21.15.123:3300/0 conn(0x7f69740ce370 msgr2=0x7f69740d0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:26.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.124+0000 7f6972a00640 1 --2- 172.21.15.96:0/4173851267 >> v2:172.21.15.123:3300/0 conn(0x7f69740ce370 0x7f69740d0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:26.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.124+0000 7f6972a00640 1 -- 172.21.15.96:0/4173851267 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69740ad470 con 0x7f69740b4e10 2024-05-28T06:05:26.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.125+0000 7f6972a00640 1 --2- 172.21.15.96:0/4173851267 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f69740b4e10 0x7f69740cde30 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f69740ca860 tx=0x7f696402c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53bce02a5b4fe847 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:26.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.125+0000 7f6971600640 1 -- 172.21.15.96:0/4173851267 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6964039070 con 0x7f69740b4e10 2024-05-28T06:05:26.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.125+0000 7f6971600640 1 -- 172.21.15.96:0/4173851267 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f696400b2e0 con 0x7f69740b4e10 2024-05-28T06:05:26.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.125+0000 7f6971600640 1 -- 172.21.15.96:0/4173851267 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6964009820 con 0x7f69740b4e10 2024-05-28T06:05:26.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.126+0000 7f6979a00640 1 -- 172.21.15.96:0/4173851267 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f69740b4e10 msgr2=0x7f69740cde30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:26.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.126+0000 7f6979a00640 1 --2- 172.21.15.96:0/4173851267 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f69740b4e10 0x7f69740cde30 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f69740ca860 tx=0x7f696402c820 comp rx=0 tx=0).stop 2024-05-28T06:05:26.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.126+0000 7f6979a00640 1 -- 172.21.15.96:0/4173851267 shutdown_connections 2024-05-28T06:05:26.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.126+0000 7f6979a00640 1 --2- 172.21.15.96:0/4173851267 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f69740d0ad0 0x7f6974099090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:26.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.126+0000 7f6979a00640 1 --2- 172.21.15.96:0/4173851267 >> v2:172.21.15.123:3300/0 conn(0x7f69740ce370 0x7f69740d0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:26.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.126+0000 7f6979a00640 1 --2- 172.21.15.96:0/4173851267 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f69740b4e10 0x7f69740cde30 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:26.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.126+0000 7f6979a00640 1 -- 172.21.15.96:0/4173851267 >> 172.21.15.96:0/4173851267 conn(0x7f697406bae0 msgr2=0x7f697406bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:26.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.126+0000 7f6979a00640 1 -- 172.21.15.96:0/4173851267 shutdown_connections 2024-05-28T06:05:26.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.126+0000 7f6979a00640 1 -- 172.21.15.96:0/4173851267 wait complete. 2024-05-28T06:05:26.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.128+0000 7f6979a00640 1 Processor -- start 2024-05-28T06:05:26.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.129+0000 7f6979a00640 1 -- start start 2024-05-28T06:05:26.131 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.130+0000 7f6979a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69740b4e10 0x7f69740afe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:26.131 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.130+0000 7f6979a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f69740ce370 0x7f69740b2660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:26.131 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.130+0000 7f6979a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f69740d0ad0 0x7f69741f4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:26.132 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.130+0000 7f6979a00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f69740cea00 con 0x7f69740ce370 2024-05-28T06:05:26.132 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.130+0000 7f6979a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f69740ce880 con 0x7f69740b4e10 2024-05-28T06:05:26.132 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.130+0000 7f6979a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f69740b3150 con 0x7f69740d0ad0 2024-05-28T06:05:26.132 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.130+0000 7f6972000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f69740ce370 0x7f69740b2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:26.132 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.130+0000 7f6972000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f69740ce370 0x7f69740b2660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49576/0 (socket says 172.21.15.96:49576) 2024-05-28T06:05:26.132 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.130+0000 7f6972000640 1 -- 172.21.15.96:0/623995602 learned_addr learned my addr 172.21.15.96:0/623995602 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:26.133 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.130+0000 7f6972000640 1 -- 172.21.15.96:0/623995602 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f69740d0ad0 msgr2=0x7f69741f4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:26.133 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.130+0000 7f6972000640 1 --2- 172.21.15.96:0/623995602 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f69740d0ad0 0x7f69741f4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:26.133 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.130+0000 7f6972000640 1 -- 172.21.15.96:0/623995602 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69740b4e10 msgr2=0x7f69740afe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:26.133 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.130+0000 7f6972000640 1 --2- 172.21.15.96:0/623995602 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69740b4e10 0x7f69740afe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:26.133 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.130+0000 7f6972000640 1 -- 172.21.15.96:0/623995602 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6974158cc0 con 0x7f69740ce370 2024-05-28T06:05:26.133 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.131+0000 7f6972000640 1 --2- 172.21.15.96:0/623995602 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f69740ce370 0x7f69740b2660 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f696000b440 tx=0x7f696000b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:26.133 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.131+0000 7f696aa00640 1 -- 172.21.15.96:0/623995602 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6960015020 con 0x7f69740ce370 2024-05-28T06:05:26.133 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.131+0000 7f696aa00640 1 -- 172.21.15.96:0/623995602 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6960011890 con 0x7f69740ce370 2024-05-28T06:05:26.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.131+0000 7f696aa00640 1 -- 172.21.15.96:0/623995602 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6960011c80 con 0x7f69740ce370 2024-05-28T06:05:26.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.131+0000 7f6979a00640 1 -- 172.21.15.96:0/623995602 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6974158fa0 con 0x7f69740ce370 2024-05-28T06:05:26.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.131+0000 7f6979a00640 1 -- 172.21.15.96:0/623995602 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6974159560 con 0x7f69740ce370 2024-05-28T06:05:26.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.132+0000 7f6979a00640 1 -- 172.21.15.96:0/623995602 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69740705d0 con 0x7f69740ce370 2024-05-28T06:05:26.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.133+0000 7f696aa00640 1 -- 172.21.15.96:0/623995602 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6960018020 con 0x7f69740ce370 2024-05-28T06:05:26.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.134+0000 7f696aa00640 1 --2- 172.21.15.96:0/623995602 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f695406c8f0 0x7f695406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:26.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.134+0000 7f696aa00640 1 -- 172.21.15.96:0/623995602 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 4857+0+0 (secure 0 0 0) 0x7f696008ba20 con 0x7f69740ce370 2024-05-28T06:05:26.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.135+0000 7f6972a00640 1 --2- 172.21.15.96:0/623995602 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f695406c8f0 0x7f695406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:26.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.136+0000 7f6972a00640 1 --2- 172.21.15.96:0/623995602 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f695406c8f0 0x7f695406edb0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f6964036000 tx=0x7f696402f000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:26.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.138+0000 7f696aa00640 1 -- 172.21.15.96:0/623995602 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f69740705d0 con 0x7f69740ce370 2024-05-28T06:05:26.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.337+0000 7f6979a00640 1 -- 172.21.15.96:0/623995602 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f69740689e0 con 0x7f69740ce370 2024-05-28T06:05:26.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.339+0000 7f696aa00640 1 -- 172.21.15.96:0/623995602 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) ==== 74+0+12903 (secure 0 0 0) 0x7f696001f020 con 0x7f69740ce370 2024-05-28T06:05:26.341 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:05:26.341 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":29,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:05:22.372384+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:05:26.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.342+0000 7f6979a00640 1 -- 172.21.15.96:0/623995602 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f695406c8f0 msgr2=0x7f695406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:26.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.342+0000 7f6979a00640 1 --2- 172.21.15.96:0/623995602 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f695406c8f0 0x7f695406edb0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f6964036000 tx=0x7f696402f000 comp rx=0 tx=0).stop 2024-05-28T06:05:26.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.342+0000 7f6979a00640 1 -- 172.21.15.96:0/623995602 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f69740ce370 msgr2=0x7f69740b2660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:26.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.342+0000 7f6979a00640 1 --2- 172.21.15.96:0/623995602 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f69740ce370 0x7f69740b2660 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f696000b440 tx=0x7f696000b900 comp rx=0 tx=0).stop 2024-05-28T06:05:26.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.342+0000 7f6979a00640 1 -- 172.21.15.96:0/623995602 shutdown_connections 2024-05-28T06:05:26.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.342+0000 7f6979a00640 1 --2- 172.21.15.96:0/623995602 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f695406c8f0 0x7f695406edb0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:26.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.342+0000 7f6979a00640 1 --2- 172.21.15.96:0/623995602 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f69740d0ad0 0x7f69741f4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:26.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.342+0000 7f6979a00640 1 --2- 172.21.15.96:0/623995602 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f69740ce370 0x7f69740b2660 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:26.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.342+0000 7f6979a00640 1 --2- 172.21.15.96:0/623995602 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69740b4e10 0x7f69740afe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:26.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.342+0000 7f6979a00640 1 -- 172.21.15.96:0/623995602 >> 172.21.15.96:0/623995602 conn(0x7f697406bae0 msgr2=0x7f6974157290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:26.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.342+0000 7f6979a00640 1 -- 172.21.15.96:0/623995602 shutdown_connections 2024-05-28T06:05:26.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.342+0000 7f6979a00640 1 -- 172.21.15.96:0/623995602 wait complete. 2024-05-28T06:05:26.361 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-28T06:03:59.517102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-28T06:05:26.361 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-28T06:05:26.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.558+0000 7f306e600640 1 Processor -- start 2024-05-28T06:05:26.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.558+0000 7f306e600640 1 -- start start 2024-05-28T06:05:26.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.560+0000 7f306e600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f306815c550 0x7f30680d0250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:26.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.560+0000 7f306e600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f30680b30e0 0x7f30680af1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:26.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.560+0000 7f306e600640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f30680b3710 con 0x7f30680af6e0 2024-05-28T06:05:26.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.560+0000 7f306e600640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f30680d0790 con 0x7f306815c550 2024-05-28T06:05:26.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.560+0000 7f306e600640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f30680b2c50 con 0x7f30680b30e0 2024-05-28T06:05:26.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.560+0000 7f306d200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f306815c550 0x7f30680d0250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:26.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.560+0000 7f306dc00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f30680af6e0 0x7f306809aa30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:48376/0 (socket says 172.21.15.96:48376) 2024-05-28T06:05:26.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.560+0000 7f306dc00640 1 -- 172.21.15.96:0/46629559 learned_addr learned my addr 172.21.15.96:0/46629559 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:05:26.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.561+0000 7f3067e00640 1 --2- 172.21.15.96:0/46629559 >> v2:172.21.15.123:3300/0 conn(0x7f30680b30e0 0x7f30680af1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:26.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.561+0000 7f306d200640 1 -- 172.21.15.96:0/46629559 >> v2:172.21.15.123:3300/0 conn(0x7f30680b30e0 msgr2=0x7f30680af1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:26.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.561+0000 7f306d200640 1 --2- 172.21.15.96:0/46629559 >> v2:172.21.15.123:3300/0 conn(0x7f30680b30e0 0x7f30680af1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:26.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.561+0000 7f306d200640 1 -- 172.21.15.96:0/46629559 >> v1:172.21.15.96:6789/0 conn(0x7f30680af6e0 legacy=0x7f306809aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:26.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.561+0000 7f306d200640 1 -- 172.21.15.96:0/46629559 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30680b39b0 con 0x7f306815c550 2024-05-28T06:05:26.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.561+0000 7f3067400640 1 -- 172.21.15.96:0/46629559 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 261901647 0 0) 0x7f30680b3710 con 0x7f30680af6e0 2024-05-28T06:05:26.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.562+0000 7f306d200640 1 --2- 172.21.15.96:0/46629559 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f306815c550 0x7f30680d0250 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f30680d2230 tx=0x7f305c02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae7cd7f5529bcda3 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:26.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.562+0000 7f3067400640 1 -- 172.21.15.96:0/46629559 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f305c037070 con 0x7f306815c550 2024-05-28T06:05:26.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.562+0000 7f3067400640 1 -- 172.21.15.96:0/46629559 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f305c032440 con 0x7f306815c550 2024-05-28T06:05:26.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.562+0000 7f3067400640 1 -- 172.21.15.96:0/46629559 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f305c032810 con 0x7f306815c550 2024-05-28T06:05:26.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.563+0000 7f306e600640 1 -- 172.21.15.96:0/46629559 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f306815c550 msgr2=0x7f30680d0250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:26.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.563+0000 7f306e600640 1 --2- 172.21.15.96:0/46629559 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f306815c550 0x7f30680d0250 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f30680d2230 tx=0x7f305c02a1a0 comp rx=0 tx=0).stop 2024-05-28T06:05:26.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.563+0000 7f306e600640 1 -- 172.21.15.96:0/46629559 shutdown_connections 2024-05-28T06:05:26.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.563+0000 7f306e600640 1 --2- 172.21.15.96:0/46629559 >> v2:172.21.15.123:3300/0 conn(0x7f30680b30e0 0x7f30680af1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:26.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.563+0000 7f306e600640 1 --2- 172.21.15.96:0/46629559 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f306815c550 0x7f30680d0250 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:26.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.563+0000 7f306e600640 1 -- 172.21.15.96:0/46629559 >> 172.21.15.96:0/46629559 conn(0x7f30680b3ec0 msgr2=0x7f30680b42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:26.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.563+0000 7f306e600640 1 -- 172.21.15.96:0/46629559 shutdown_connections 2024-05-28T06:05:26.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.563+0000 7f306e600640 1 -- 172.21.15.96:0/46629559 wait complete. 2024-05-28T06:05:26.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.565+0000 7f306e600640 1 Processor -- start 2024-05-28T06:05:26.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.565+0000 7f306e600640 1 -- start start 2024-05-28T06:05:26.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.566+0000 7f306e600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f30680af6e0 0x7f306814ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:26.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.566+0000 7f306e600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f30680b30e0 0x7f30681596e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:26.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.566+0000 7f306e600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f306815c550 0x7f30680ccd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:26.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.566+0000 7f306e600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f306815cab0 con 0x7f306815c550 2024-05-28T06:05:26.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.566+0000 7f306e600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f306815c930 con 0x7f30680b30e0 2024-05-28T06:05:26.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.566+0000 7f306e600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f30680afb30 con 0x7f30680af6e0 2024-05-28T06:05:26.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.566+0000 7f306dc00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f306815c550 0x7f30680ccd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:26.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.566+0000 7f306dc00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f306815c550 0x7f30680ccd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49592/0 (socket says 172.21.15.96:49592) 2024-05-28T06:05:26.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.566+0000 7f306dc00640 1 -- 172.21.15.96:0/946603637 learned_addr learned my addr 172.21.15.96:0/946603637 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:26.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.566+0000 7f306d200640 1 --2- 172.21.15.96:0/946603637 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f30680af6e0 0x7f306814ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:26.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.567+0000 7f3067e00640 1 --2- 172.21.15.96:0/946603637 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f30680b30e0 0x7f30681596e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:26.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.567+0000 7f306dc00640 1 -- 172.21.15.96:0/946603637 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f30680af6e0 msgr2=0x7f306814ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:26.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.567+0000 7f306dc00640 1 --2- 172.21.15.96:0/946603637 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f30680af6e0 0x7f306814ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:26.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.567+0000 7f306dc00640 1 -- 172.21.15.96:0/946603637 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f30680b30e0 msgr2=0x7f30681596e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:26.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.567+0000 7f306dc00640 1 --2- 172.21.15.96:0/946603637 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f30680b30e0 0x7f30681596e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:26.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.567+0000 7f306dc00640 1 -- 172.21.15.96:0/946603637 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30680cb8b0 con 0x7f306815c550 2024-05-28T06:05:26.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.567+0000 7f3067e00640 1 --2- 172.21.15.96:0/946603637 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f30680b30e0 0x7f30681596e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:05:26.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.567+0000 7f306dc00640 1 --2- 172.21.15.96:0/946603637 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f306815c550 0x7f30680ccd60 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f306000caf0 tx=0x7f306000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:26.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.567+0000 7f3064c00640 1 -- 172.21.15.96:0/946603637 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f306000ac40 con 0x7f306815c550 2024-05-28T06:05:26.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.567+0000 7f3064c00640 1 -- 172.21.15.96:0/946603637 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f306000ade0 con 0x7f306815c550 2024-05-28T06:05:26.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.567+0000 7f3064c00640 1 -- 172.21.15.96:0/946603637 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3060012640 con 0x7f306815c550 2024-05-28T06:05:26.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.567+0000 7f306e600640 1 -- 172.21.15.96:0/946603637 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3068159e50 con 0x7f306815c550 2024-05-28T06:05:26.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.567+0000 7f306e600640 1 -- 172.21.15.96:0/946603637 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f306815a410 con 0x7f306815c550 2024-05-28T06:05:26.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.568+0000 7f306e600640 1 -- 172.21.15.96:0/946603637 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30680689e0 con 0x7f306815c550 2024-05-28T06:05:26.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.569+0000 7f3064c00640 1 -- 172.21.15.96:0/946603637 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f30600128a0 con 0x7f306815c550 2024-05-28T06:05:26.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.570+0000 7f3064c00640 1 --2- 172.21.15.96:0/946603637 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f304806c8f0 0x7f304806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:26.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.570+0000 7f3064c00640 1 -- 172.21.15.96:0/946603637 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 4857+0+0 (secure 0 0 0) 0x7f306008bee0 con 0x7f306815c550 2024-05-28T06:05:26.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.571+0000 7f306d200640 1 --2- 172.21.15.96:0/946603637 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f304806c8f0 0x7f304806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:26.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.572+0000 7f306d200640 1 --2- 172.21.15.96:0/946603637 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f304806c8f0 0x7f304806edb0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f305c034000 tx=0x7f305c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:26.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.575+0000 7f3064c00640 1 -- 172.21.15.96:0/946603637 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f306005a5c0 con 0x7f306815c550 2024-05-28T06:05:26.775 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.774+0000 7f306e600640 1 -- 172.21.15.96:0/946603637 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f30680b4820 con 0x7f306815c550 2024-05-28T06:05:26.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.775+0000 7f3064c00640 1 -- 172.21.15.96:0/946603637 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v29) ==== 93+0+10 (secure 0 0 0) 0x7f3060059ec0 con 0x7f306815c550 2024-05-28T06:05:26.776 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 1 2024-05-28T06:05:26.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.777+0000 7f306e600640 1 -- 172.21.15.96:0/946603637 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f304806c8f0 msgr2=0x7f304806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:26.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.777+0000 7f306e600640 1 --2- 172.21.15.96:0/946603637 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f304806c8f0 0x7f304806edb0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f305c034000 tx=0x7f305c02e000 comp rx=0 tx=0).stop 2024-05-28T06:05:26.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.777+0000 7f306e600640 1 -- 172.21.15.96:0/946603637 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f306815c550 msgr2=0x7f30680ccd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:26.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.777+0000 7f306e600640 1 --2- 172.21.15.96:0/946603637 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f306815c550 0x7f30680ccd60 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f306000caf0 tx=0x7f306000a060 comp rx=0 tx=0).stop 2024-05-28T06:05:26.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.777+0000 7f306e600640 1 -- 172.21.15.96:0/946603637 shutdown_connections 2024-05-28T06:05:26.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.777+0000 7f306e600640 1 --2- 172.21.15.96:0/946603637 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f304806c8f0 0x7f304806edb0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:26.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.777+0000 7f306e600640 1 --2- 172.21.15.96:0/946603637 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f306815c550 0x7f30680ccd60 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:26.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.777+0000 7f306e600640 1 --2- 172.21.15.96:0/946603637 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f30680b30e0 0x7f30681596e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:26.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.777+0000 7f306e600640 1 --2- 172.21.15.96:0/946603637 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f30680af6e0 0x7f306814ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:26.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.777+0000 7f306e600640 1 -- 172.21.15.96:0/946603637 >> 172.21.15.96:0/946603637 conn(0x7f30680b3ec0 msgr2=0x7f30680713c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:26.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.777+0000 7f306e600640 1 -- 172.21.15.96:0/946603637 shutdown_connections 2024-05-28T06:05:26.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.778+0000 7f306e600640 1 -- 172.21.15.96:0/946603637 wait complete. 2024-05-28T06:05:26.798 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-05-28T06:05:26.998 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.996+0000 7f259ee00640 1 Processor -- start 2024-05-28T06:05:26.998 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.996+0000 7f259ee00640 1 -- start start 2024-05-28T06:05:27.000 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.997+0000 7f259ee00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f25980b4e10 0x7f25980cde30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:27.000 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.997+0000 7f259ee00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f25980ce370 0x7f25980d0590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:27.000 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.997+0000 7f259ee00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25980b51f0 con 0x7f25980d0ad0 2024-05-28T06:05:27.000 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.997+0000 7f259ee00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f25980d0f20 con 0x7f25980ce370 2024-05-28T06:05:27.000 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.997+0000 7f259ee00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f25980aef60 con 0x7f25980b4e10 2024-05-28T06:05:27.000 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.998+0000 7f2597400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f25980ce370 0x7f25980d0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:27.000 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.998+0000 7f2597400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f25980ce370 0x7f25980d0590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49608/0 (socket says 172.21.15.96:49608) 2024-05-28T06:05:27.000 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.998+0000 7f2597400640 1 -- 172.21.15.96:0/3943829725 learned_addr learned my addr 172.21.15.96:0/3943829725 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:27.000 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.998+0000 7f2597e00640 1 --2- 172.21.15.96:0/3943829725 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f25980b4e10 0x7f25980cde30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:27.000 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.998+0000 7f2597400640 1 -- 172.21.15.96:0/3943829725 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f25980b4e10 msgr2=0x7f25980cde30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:27.000 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.998+0000 7f2597400640 1 --2- 172.21.15.96:0/3943829725 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f25980b4e10 0x7f25980cde30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:27.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.998+0000 7f2597400640 1 -- 172.21.15.96:0/3943829725 >> v1:172.21.15.123:6789/0 conn(0x7f25980d0ad0 legacy=0x7f2598099090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:27.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.998+0000 7f2597400640 1 -- 172.21.15.96:0/3943829725 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25980ad1d0 con 0x7f25980ce370 2024-05-28T06:05:27.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.998+0000 7f2597e00640 1 --2- 172.21.15.96:0/3943829725 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f25980b4e10 0x7f25980cde30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:05:27.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:26.999+0000 7f2597400640 1 --2- 172.21.15.96:0/3943829725 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f25980ce370 0x7f25980d0590 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f2584009870 tx=0x7f258402a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c72f444b439f7b7b server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:27.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.000+0000 7f2596a00640 1 -- 172.21.15.96:0/3943829725 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2584037070 con 0x7f25980ce370 2024-05-28T06:05:27.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.000+0000 7f2596a00640 1 -- 172.21.15.96:0/3943829725 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2584032440 con 0x7f25980ce370 2024-05-28T06:05:27.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.000+0000 7f2596a00640 1 -- 172.21.15.96:0/3943829725 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2584032810 con 0x7f25980ce370 2024-05-28T06:05:27.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.000+0000 7f259ee00640 1 -- 172.21.15.96:0/3943829725 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f25980ce370 msgr2=0x7f25980d0590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:27.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.000+0000 7f259ee00640 1 --2- 172.21.15.96:0/3943829725 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f25980ce370 0x7f25980d0590 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f2584009870 tx=0x7f258402a180 comp rx=0 tx=0).stop 2024-05-28T06:05:27.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.000+0000 7f259ee00640 1 -- 172.21.15.96:0/3943829725 shutdown_connections 2024-05-28T06:05:27.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.000+0000 7f259ee00640 1 --2- 172.21.15.96:0/3943829725 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f25980ce370 0x7f25980d0590 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:27.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.000+0000 7f259ee00640 1 --2- 172.21.15.96:0/3943829725 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f25980b4e10 0x7f25980cde30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:27.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.000+0000 7f259ee00640 1 -- 172.21.15.96:0/3943829725 >> 172.21.15.96:0/3943829725 conn(0x7f259806bae0 msgr2=0x7f259806bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:27.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.000+0000 7f259ee00640 1 -- 172.21.15.96:0/3943829725 shutdown_connections 2024-05-28T06:05:27.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.000+0000 7f259ee00640 1 -- 172.21.15.96:0/3943829725 wait complete. 2024-05-28T06:05:27.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.002+0000 7f259ee00640 1 Processor -- start 2024-05-28T06:05:27.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.002+0000 7f259ee00640 1 -- start start 2024-05-28T06:05:27.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.003+0000 7f259ee00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f25980b4e10 0x7f25981594b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:27.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.003+0000 7f259ee00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f25980ce370 0x7f25980afda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:27.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.003+0000 7f259ee00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f25980d0ad0 0x7f25980c7330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:27.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.003+0000 7f259ee00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f25980ce880 con 0x7f25980ce370 2024-05-28T06:05:27.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.003+0000 7f259ee00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f25980ad750 con 0x7f25980d0ad0 2024-05-28T06:05:27.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.003+0000 7f259ee00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f25980cea00 con 0x7f25980b4e10 2024-05-28T06:05:27.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.004+0000 7f2597400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f25980ce370 0x7f25980afda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:27.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.004+0000 7f2597400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f25980ce370 0x7f25980afda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49620/0 (socket says 172.21.15.96:49620) 2024-05-28T06:05:27.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.004+0000 7f2597400640 1 -- 172.21.15.96:0/4286660214 learned_addr learned my addr 172.21.15.96:0/4286660214 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:27.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.004+0000 7f2597e00640 1 --2- 172.21.15.96:0/4286660214 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f25980b4e10 0x7f25981594b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:27.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.004+0000 7f259ce00640 1 --2- 172.21.15.96:0/4286660214 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f25980d0ad0 0x7f25980c7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:27.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.004+0000 7f2597400640 1 -- 172.21.15.96:0/4286660214 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f25980b4e10 msgr2=0x7f25981594b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:27.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.004+0000 7f2597400640 1 --2- 172.21.15.96:0/4286660214 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f25980b4e10 0x7f25981594b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:27.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.004+0000 7f2597400640 1 -- 172.21.15.96:0/4286660214 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f25980d0ad0 msgr2=0x7f25980c7330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:27.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.004+0000 7f2597400640 1 --2- 172.21.15.96:0/4286660214 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f25980d0ad0 0x7f25980c7330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:27.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.004+0000 7f2597400640 1 -- 172.21.15.96:0/4286660214 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2598072210 con 0x7f25980ce370 2024-05-28T06:05:27.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.004+0000 7f259ce00640 1 --2- 172.21.15.96:0/4286660214 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f25980d0ad0 0x7f25980c7330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:05:27.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.004+0000 7f2597e00640 1 --2- 172.21.15.96:0/4286660214 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f25980b4e10 0x7f25981594b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:05:27.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.004+0000 7f2597400640 1 --2- 172.21.15.96:0/4286660214 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f25980ce370 0x7f25980afda0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f2584009870 tx=0x7f25840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:27.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.005+0000 7f258fe00640 1 -- 172.21.15.96:0/4286660214 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2584041020 con 0x7f25980ce370 2024-05-28T06:05:27.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.005+0000 7f258fe00640 1 -- 172.21.15.96:0/4286660214 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2584030970 con 0x7f25980ce370 2024-05-28T06:05:27.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.005+0000 7f259ee00640 1 -- 172.21.15.96:0/4286660214 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25980724f0 con 0x7f25980ce370 2024-05-28T06:05:27.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.005+0000 7f258fe00640 1 -- 172.21.15.96:0/4286660214 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2584037040 con 0x7f25980ce370 2024-05-28T06:05:27.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.005+0000 7f259ee00640 1 -- 172.21.15.96:0/4286660214 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2598072ab0 con 0x7f25980ce370 2024-05-28T06:05:27.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.006+0000 7f259ee00640 1 -- 172.21.15.96:0/4286660214 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25980689e0 con 0x7f25980ce370 2024-05-28T06:05:27.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.007+0000 7f258fe00640 1 -- 172.21.15.96:0/4286660214 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2584044020 con 0x7f25980ce370 2024-05-28T06:05:27.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.008+0000 7f258fe00640 1 --2- 172.21.15.96:0/4286660214 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f257806c8f0 0x7f257806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:27.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.008+0000 7f258fe00640 1 -- 172.21.15.96:0/4286660214 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 4857+0+0 (secure 0 0 0) 0x7f25840ac290 con 0x7f25980ce370 2024-05-28T06:05:27.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.009+0000 7f2597e00640 1 --2- 172.21.15.96:0/4286660214 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f257806c8f0 0x7f257806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:27.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.010+0000 7f2597e00640 1 --2- 172.21.15.96:0/4286660214 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f257806c8f0 0x7f257806edb0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f2588006fd0 tx=0x7f2588008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:27.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.013+0000 7f258fe00640 1 -- 172.21.15.96:0/4286660214 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f258404b030 con 0x7f25980ce370 2024-05-28T06:05:27.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.215+0000 7f259ee00640 1 -- 172.21.15.96:0/4286660214 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f259806cc30 con 0x7f25980ce370 2024-05-28T06:05:27.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.215+0000 7f258fe00640 1 -- 172.21.15.96:0/4286660214 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v29) ==== 84+0+65 (secure 0 0 0) 0x7f258407a970 con 0x7f25980ce370 2024-05-28T06:05:27.217 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:05:27.217 INFO:teuthology.orchestra.run.smithi096.stdout:[ 2024-05-28T06:05:27.217 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:05:27.217 INFO:teuthology.orchestra.run.smithi096.stdout: "poolnum": 1, 2024-05-28T06:05:27.217 INFO:teuthology.orchestra.run.smithi096.stdout: "poolname": ".mgr" 2024-05-28T06:05:27.217 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:05:27.217 INFO:teuthology.orchestra.run.smithi096.stdout:] 2024-05-28T06:05:27.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.218+0000 7f259ee00640 1 -- 172.21.15.96:0/4286660214 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f257806c8f0 msgr2=0x7f257806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:27.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.218+0000 7f259ee00640 1 --2- 172.21.15.96:0/4286660214 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f257806c8f0 0x7f257806edb0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f2588006fd0 tx=0x7f2588008040 comp rx=0 tx=0).stop 2024-05-28T06:05:27.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.218+0000 7f259ee00640 1 -- 172.21.15.96:0/4286660214 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f25980ce370 msgr2=0x7f25980afda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:27.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.218+0000 7f259ee00640 1 --2- 172.21.15.96:0/4286660214 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f25980ce370 0x7f25980afda0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f2584009870 tx=0x7f25840046c0 comp rx=0 tx=0).stop 2024-05-28T06:05:27.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.218+0000 7f259ee00640 1 -- 172.21.15.96:0/4286660214 shutdown_connections 2024-05-28T06:05:27.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.218+0000 7f259ee00640 1 --2- 172.21.15.96:0/4286660214 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f257806c8f0 0x7f257806edb0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:27.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.218+0000 7f259ee00640 1 --2- 172.21.15.96:0/4286660214 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f25980d0ad0 0x7f25980c7330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:27.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.218+0000 7f259ee00640 1 --2- 172.21.15.96:0/4286660214 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f25980ce370 0x7f25980afda0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:27.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.218+0000 7f259ee00640 1 --2- 172.21.15.96:0/4286660214 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f25980b4e10 0x7f25981594b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:27.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.218+0000 7f259ee00640 1 -- 172.21.15.96:0/4286660214 >> 172.21.15.96:0/4286660214 conn(0x7f259806bae0 msgr2=0x7f259815bb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:27.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.219+0000 7f259ee00640 1 -- 172.21.15.96:0/4286660214 shutdown_connections 2024-05-28T06:05:27.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.219+0000 7f259ee00640 1 -- 172.21.15.96:0/4286660214 wait complete. 2024-05-28T06:05:27.239 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-05-28T06:05:27.239 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-05-28T06:05:27.444 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.442+0000 7fa301400640 1 Processor -- start 2024-05-28T06:05:27.444 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.443+0000 7fa301400640 1 -- start start 2024-05-28T06:05:27.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.444+0000 7fa301400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fa2fc15c830 0x7fa2fc0ce230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:27.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.444+0000 7fa301400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fa2fc0d08e0 0x7fa2fc0d0d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:27.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.444+0000 7fa301400640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2fc15cc10 con 0x7fa2fc0ce770 2024-05-28T06:05:27.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.444+0000 7fa301400640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fa2fc0b4b90 con 0x7fa2fc15c830 2024-05-28T06:05:27.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.444+0000 7fa301400640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fa2fc0b4e70 con 0x7fa2fc0d08e0 2024-05-28T06:05:27.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.444+0000 7fa2fa800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fa2fc15c830 0x7fa2fc0ce230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:27.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.444+0000 7fa2fa800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fa2fc15c830 0x7fa2fc0ce230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49630/0 (socket says 172.21.15.96:49630) 2024-05-28T06:05:27.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.444+0000 7fa2fa800640 1 -- 172.21.15.96:0/2022466255 learned_addr learned my addr 172.21.15.96:0/2022466255 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:27.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.445+0000 7fa2fb200640 1 --2- 172.21.15.96:0/2022466255 >> v2:172.21.15.123:3300/0 conn(0x7fa2fc0d08e0 0x7fa2fc0d0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:27.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.445+0000 7fa2fa800640 1 -- 172.21.15.96:0/2022466255 >> v2:172.21.15.123:3300/0 conn(0x7fa2fc0d08e0 msgr2=0x7fa2fc0d0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:27.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.445+0000 7fa2fa800640 1 --2- 172.21.15.96:0/2022466255 >> v2:172.21.15.123:3300/0 conn(0x7fa2fc0d08e0 0x7fa2fc0d0d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:27.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.445+0000 7fa2fa800640 1 -- 172.21.15.96:0/2022466255 >> v1:172.21.15.123:6789/0 conn(0x7fa2fc0ce770 legacy=0x7fa2fc0b40b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:27.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.445+0000 7fa2fa800640 1 -- 172.21.15.96:0/2022466255 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2fc0b5150 con 0x7fa2fc15c830 2024-05-28T06:05:27.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.445+0000 7fa2fb200640 1 --2- 172.21.15.96:0/2022466255 >> v2:172.21.15.123:3300/0 conn(0x7fa2fc0d08e0 0x7fa2fc0d0d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:05:27.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.446+0000 7fa2fa800640 1 --2- 172.21.15.96:0/2022466255 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa2fc15c830 0x7fa2fc0ce230 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fa2fc0afd60 tx=0x7fa2ec02a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=884e391cedd8b0be server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:27.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.446+0000 7fa2f9e00640 1 -- 172.21.15.96:0/2022466255 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2ec037070 con 0x7fa2fc15c830 2024-05-28T06:05:27.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.446+0000 7fa2f9e00640 1 -- 172.21.15.96:0/2022466255 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2ec032440 con 0x7fa2fc15c830 2024-05-28T06:05:27.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.446+0000 7fa2f9e00640 1 -- 172.21.15.96:0/2022466255 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2ec032810 con 0x7fa2fc15c830 2024-05-28T06:05:27.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.447+0000 7fa301400640 1 -- 172.21.15.96:0/2022466255 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa2fc15c830 msgr2=0x7fa2fc0ce230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:27.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.447+0000 7fa301400640 1 --2- 172.21.15.96:0/2022466255 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa2fc15c830 0x7fa2fc0ce230 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fa2fc0afd60 tx=0x7fa2ec02a230 comp rx=0 tx=0).stop 2024-05-28T06:05:27.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.447+0000 7fa301400640 1 -- 172.21.15.96:0/2022466255 shutdown_connections 2024-05-28T06:05:27.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.447+0000 7fa301400640 1 --2- 172.21.15.96:0/2022466255 >> v2:172.21.15.123:3300/0 conn(0x7fa2fc0d08e0 0x7fa2fc0d0d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:27.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.447+0000 7fa301400640 1 --2- 172.21.15.96:0/2022466255 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa2fc15c830 0x7fa2fc0ce230 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:27.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.447+0000 7fa301400640 1 -- 172.21.15.96:0/2022466255 >> 172.21.15.96:0/2022466255 conn(0x7fa2fc0b20a0 msgr2=0x7fa2fc0b24b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:27.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.447+0000 7fa301400640 1 -- 172.21.15.96:0/2022466255 shutdown_connections 2024-05-28T06:05:27.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.447+0000 7fa301400640 1 -- 172.21.15.96:0/2022466255 wait complete. 2024-05-28T06:05:27.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.449+0000 7fa301400640 1 Processor -- start 2024-05-28T06:05:27.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.449+0000 7fa301400640 1 -- start start 2024-05-28T06:05:27.452 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.450+0000 7fa301400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa2fc0ce770 0x7fa2fc15ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:27.452 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.450+0000 7fa301400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa2fc0d08e0 0x7fa2fc072340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:27.452 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.450+0000 7fa301400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa2fc15c830 0x7fa2fc072880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:27.452 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.450+0000 7fa301400640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa2fc0b4940 con 0x7fa2fc0ce770 2024-05-28T06:05:27.452 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.450+0000 7fa301400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa2fc0b47c0 con 0x7fa2fc15c830 2024-05-28T06:05:27.452 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.450+0000 7fa301400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fa2fc0cffe0 con 0x7fa2fc0d08e0 2024-05-28T06:05:27.452 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.450+0000 7fa2fa800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa2fc0ce770 0x7fa2fc15ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:27.452 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.450+0000 7fa2fa800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa2fc0ce770 0x7fa2fc15ea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49646/0 (socket says 172.21.15.96:49646) 2024-05-28T06:05:27.452 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.450+0000 7fa2fa800640 1 -- 172.21.15.96:0/1464162969 learned_addr learned my addr 172.21.15.96:0/1464162969 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:27.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.451+0000 7fa2fa800640 1 -- 172.21.15.96:0/1464162969 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa2fc0d08e0 msgr2=0x7fa2fc072340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:27.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.451+0000 7fa2fa800640 1 --2- 172.21.15.96:0/1464162969 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa2fc0d08e0 0x7fa2fc072340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:27.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.451+0000 7fa2fa800640 1 -- 172.21.15.96:0/1464162969 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa2fc15c830 msgr2=0x7fa2fc072880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:27.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.451+0000 7fa2fa800640 1 --2- 172.21.15.96:0/1464162969 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa2fc15c830 0x7fa2fc072880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:27.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.451+0000 7fa2fa800640 1 -- 172.21.15.96:0/1464162969 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2fc1f84d0 con 0x7fa2fc0ce770 2024-05-28T06:05:27.454 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.451+0000 7fa2fa800640 1 --2- 172.21.15.96:0/1464162969 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa2fc0ce770 0x7fa2fc15ea80 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fa2ec000c00 tx=0x7fa2ec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:27.454 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.451+0000 7fa2f3400640 1 -- 172.21.15.96:0/1464162969 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2ec041020 con 0x7fa2fc0ce770 2024-05-28T06:05:27.454 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.451+0000 7fa2f3400640 1 -- 172.21.15.96:0/1464162969 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2ec030970 con 0x7fa2fc0ce770 2024-05-28T06:05:27.454 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.451+0000 7fa2f3400640 1 -- 172.21.15.96:0/1464162969 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2ec037040 con 0x7fa2fc0ce770 2024-05-28T06:05:27.454 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.451+0000 7fa301400640 1 -- 172.21.15.96:0/1464162969 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2fc15d9e0 con 0x7fa2fc0ce770 2024-05-28T06:05:27.454 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.452+0000 7fa301400640 1 -- 172.21.15.96:0/1464162969 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2fc15dfa0 con 0x7fa2fc0ce770 2024-05-28T06:05:27.454 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.453+0000 7fa301400640 1 -- 172.21.15.96:0/1464162969 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2fc0689e0 con 0x7fa2fc0ce770 2024-05-28T06:05:27.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.454+0000 7fa2f3400640 1 -- 172.21.15.96:0/1464162969 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa2ec044020 con 0x7fa2fc0ce770 2024-05-28T06:05:27.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.455+0000 7fa2f3400640 1 --2- 172.21.15.96:0/1464162969 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa2d806c8f0 0x7fa2d806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:27.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.455+0000 7fa2f3400640 1 -- 172.21.15.96:0/1464162969 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 4857+0+0 (secure 0 0 0) 0x7fa2ec0ac270 con 0x7fa2fc0ce770 2024-05-28T06:05:27.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.456+0000 7fa2f1e00640 1 --2- 172.21.15.96:0/1464162969 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa2d806c8f0 0x7fa2d806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:27.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.457+0000 7fa2f1e00640 1 --2- 172.21.15.96:0/1464162969 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa2d806c8f0 0x7fa2d806edb0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fa2fc072f00 tx=0x7fa2e4007990 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:27.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.460+0000 7fa2f3400640 1 -- 172.21.15.96:0/1464162969 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa2ec04b030 con 0x7fa2fc0ce770 2024-05-28T06:05:27.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:27.660+0000 7fa301400640 1 -- 172.21.15.96:0/1464162969 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fa2fc0b1a40 con 0x7fa2fc0ce770 2024-05-28T06:05:28.417 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:28.415+0000 7fa2f3400640 1 -- 172.21.15.96:0/1464162969 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v30) ==== 138+0+0 (secure 0 0 0) 0x7fa2fc1f84d0 con 0x7fa2fc0ce770 2024-05-28T06:05:28.417 INFO:teuthology.orchestra.run.smithi096.stderr:pool 'cephfs_metadata' created 2024-05-28T06:05:28.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:28.419+0000 7fa301400640 1 -- 172.21.15.96:0/1464162969 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa2d806c8f0 msgr2=0x7fa2d806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:28.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:28.419+0000 7fa301400640 1 --2- 172.21.15.96:0/1464162969 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa2d806c8f0 0x7fa2d806edb0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fa2fc072f00 tx=0x7fa2e4007990 comp rx=0 tx=0).stop 2024-05-28T06:05:28.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:28.419+0000 7fa301400640 1 -- 172.21.15.96:0/1464162969 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa2fc0ce770 msgr2=0x7fa2fc15ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:28.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:28.419+0000 7fa301400640 1 --2- 172.21.15.96:0/1464162969 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa2fc0ce770 0x7fa2fc15ea80 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fa2ec000c00 tx=0x7fa2ec0046c0 comp rx=0 tx=0).stop 2024-05-28T06:05:28.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:28.419+0000 7fa301400640 1 -- 172.21.15.96:0/1464162969 shutdown_connections 2024-05-28T06:05:28.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:28.419+0000 7fa301400640 1 --2- 172.21.15.96:0/1464162969 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa2d806c8f0 0x7fa2d806edb0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:28.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:28.419+0000 7fa301400640 1 --2- 172.21.15.96:0/1464162969 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa2fc15c830 0x7fa2fc072880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:28.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:28.419+0000 7fa301400640 1 --2- 172.21.15.96:0/1464162969 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa2fc0d08e0 0x7fa2fc072340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:28.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:28.419+0000 7fa301400640 1 --2- 172.21.15.96:0/1464162969 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa2fc0ce770 0x7fa2fc15ea80 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:28.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:28.419+0000 7fa301400640 1 -- 172.21.15.96:0/1464162969 >> 172.21.15.96:0/1464162969 conn(0x7fa2fc0b20a0 msgr2=0x7fa2fc0aff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:28.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:28.419+0000 7fa301400640 1 -- 172.21.15.96:0/1464162969 shutdown_connections 2024-05-28T06:05:28.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:28.419+0000 7fa301400640 1 -- 172.21.15.96:0/1464162969 wait complete. 2024-05-28T06:05:28.450 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-05-28T06:05:28.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:28.643+0000 7f868c200640 1 Processor -- start 2024-05-28T06:05:28.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:28.643+0000 7f868c200640 1 -- start start 2024-05-28T06:05:28.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:28.644+0000 7f868c200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f86840d0640 0x7f86840d0a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:28.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:28.645+0000 7f868c200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86840ade90 con 0x7f86840725e0 2024-05-28T06:05:28.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:28.645+0000 7f868c200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86840d0f60 con 0x7f868415b6f0 2024-05-28T06:05:28.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:28.645+0000 7f868c200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f8684071c40 con 0x7f86840d0640 2024-05-28T06:05:28.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:28.645+0000 7f8689800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f86840d0640 0x7f86840d0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:28.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:28.645+0000 7f868a200640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f86840725e0 0x7f86840ae650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:48388/0 (socket says 172.21.15.96:48388) 2024-05-28T06:05:28.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:28.645+0000 7f868a200640 1 -- 172.21.15.96:0/1757018867 learned_addr learned my addr 172.21.15.96:0/1757018867 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:05:28.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:28.645+0000 7f8689800640 1 -- 172.21.15.96:0/1757018867 >> v1:172.21.15.123:6789/0 conn(0x7f868415b6f0 legacy=0x7f868406e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:28.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:28.645+0000 7f8689800640 1 -- 172.21.15.96:0/1757018867 >> v1:172.21.15.96:6789/0 conn(0x7f86840725e0 legacy=0x7f86840ae650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:28.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:28.645+0000 7f8689800640 1 -- 172.21.15.96:0/1757018867 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8684071e70 con 0x7f86840d0640 2024-05-28T06:05:28.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:28.645+0000 7f8683e00640 1 -- 172.21.15.96:0/1757018867 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3922849358 0 0) 0x7f86840ade90 con 0x7f86840725e0 2024-05-28T06:05:28.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:28.647+0000 7f8689800640 1 --2- 172.21.15.96:0/1757018867 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f86840d0640 0x7f86840d0a20 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f868406fe40 tx=0x7f867802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ffc348e0dcde838 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:28.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:28.647+0000 7f8683e00640 1 -- 172.21.15.96:0/1757018867 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8678037070 con 0x7f86840d0640 2024-05-28T06:05:28.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:28.647+0000 7f8683e00640 1 -- 172.21.15.96:0/1757018867 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8678032440 con 0x7f86840d0640 2024-05-28T06:05:28.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:28.647+0000 7f8683e00640 1 -- 172.21.15.96:0/1757018867 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8678032810 con 0x7f86840d0640 2024-05-28T06:05:28.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:28.647+0000 7f868c200640 1 -- 172.21.15.96:0/1757018867 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f86840d0640 msgr2=0x7f86840d0a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:28.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:28.647+0000 7f868c200640 1 --2- 172.21.15.96:0/1757018867 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f86840d0640 0x7f86840d0a20 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f868406fe40 tx=0x7f867802a1a0 comp rx=0 tx=0).stop 2024-05-28T06:05:28.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:28.648+0000 7f868c200640 1 -- 172.21.15.96:0/1757018867 shutdown_connections 2024-05-28T06:05:28.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:28.648+0000 7f868c200640 1 --2- 172.21.15.96:0/1757018867 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f86840d0640 0x7f86840d0a20 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:28.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:28.648+0000 7f868c200640 1 -- 172.21.15.96:0/1757018867 >> 172.21.15.96:0/1757018867 conn(0x7f868406bd20 msgr2=0x7f868406c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:28.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:28.648+0000 7f868c200640 1 -- 172.21.15.96:0/1757018867 shutdown_connections 2024-05-28T06:05:28.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:28.648+0000 7f868c200640 1 -- 172.21.15.96:0/1757018867 wait complete. 2024-05-28T06:05:28.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:28.649+0000 7f868c200640 1 Processor -- start 2024-05-28T06:05:28.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:28.650+0000 7f868c200640 1 -- start start 2024-05-28T06:05:28.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:28.651+0000 7f868c200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f86840725e0 0x7f86840cf6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:28.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:28.651+0000 7f868c200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f86840d0640 0x7f868414b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:28.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:28.651+0000 7f868c200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f868415b6f0 0x7f868414ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:28.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:28.651+0000 7f868c200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f86840aed60 con 0x7f86840725e0 2024-05-28T06:05:28.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:28.651+0000 7f868c200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f86840ae130 con 0x7f868415b6f0 2024-05-28T06:05:28.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:28.651+0000 7f868c200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f86840aeee0 con 0x7f86840d0640 2024-05-28T06:05:28.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:28.651+0000 7f8689800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f86840725e0 0x7f86840cf6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:28.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:28.651+0000 7f8689800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f86840725e0 0x7f86840cf6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49672/0 (socket says 172.21.15.96:49672) 2024-05-28T06:05:28.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:28.651+0000 7f8689800640 1 -- 172.21.15.96:0/1794160895 learned_addr learned my addr 172.21.15.96:0/1794160895 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:28.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:28.651+0000 7f8689800640 1 -- 172.21.15.96:0/1794160895 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f86840d0640 msgr2=0x7f868414b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:28.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:28.651+0000 7f8689800640 1 --2- 172.21.15.96:0/1794160895 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f86840d0640 0x7f868414b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:28.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:28.651+0000 7f8689800640 1 -- 172.21.15.96:0/1794160895 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f868415b6f0 msgr2=0x7f868414ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:28.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:28.651+0000 7f8689800640 1 --2- 172.21.15.96:0/1794160895 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f868415b6f0 0x7f868414ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:28.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:28.651+0000 7f8689800640 1 -- 172.21.15.96:0/1794160895 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8684145eb0 con 0x7f86840725e0 2024-05-28T06:05:28.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:28.652+0000 7f8689800640 1 --2- 172.21.15.96:0/1794160895 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f86840725e0 0x7f86840cf6b0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f8678035040 tx=0x7f86780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:28.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:28.652+0000 7f8681600640 1 -- 172.21.15.96:0/1794160895 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8678037040 con 0x7f86840725e0 2024-05-28T06:05:28.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:28.652+0000 7f8681600640 1 -- 172.21.15.96:0/1794160895 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8678004230 con 0x7f86840725e0 2024-05-28T06:05:28.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:28.652+0000 7f8681600640 1 -- 172.21.15.96:0/1794160895 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8678032e40 con 0x7f86840725e0 2024-05-28T06:05:28.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:28.652+0000 7f868c200640 1 -- 172.21.15.96:0/1794160895 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8684146190 con 0x7f86840725e0 2024-05-28T06:05:28.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:28.652+0000 7f868c200640 1 -- 172.21.15.96:0/1794160895 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86840cd110 con 0x7f86840725e0 2024-05-28T06:05:28.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:28.653+0000 7f868c200640 1 -- 172.21.15.96:0/1794160895 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86840b4b10 con 0x7f86840725e0 2024-05-28T06:05:28.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:28.654+0000 7f8681600640 1 -- 172.21.15.96:0/1794160895 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8678030770 con 0x7f86840725e0 2024-05-28T06:05:28.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:28.655+0000 7f8681600640 1 --2- 172.21.15.96:0/1794160895 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f866406c8f0 0x7f866406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:28.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:28.655+0000 7f8681600640 1 -- 172.21.15.96:0/1794160895 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5227+0+0 (secure 0 0 0) 0x7f86780acca0 con 0x7f86840725e0 2024-05-28T06:05:28.657 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:28.656+0000 7f8688e00640 1 --2- 172.21.15.96:0/1794160895 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f866406c8f0 0x7f866406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:28.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:28.657+0000 7f8688e00640 1 --2- 172.21.15.96:0/1794160895 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f866406c8f0 0x7f866406edb0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f8670006fd0 tx=0x7f8670006bc0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:28.661 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:28.660+0000 7f8681600640 1 -- 172.21.15.96:0/1794160895 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f86840b4b10 con 0x7f86840725e0 2024-05-28T06:05:28.865 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:28.863+0000 7f868c200640 1 -- 172.21.15.96:0/1794160895 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f86840689e0 con 0x7f86840725e0 2024-05-28T06:05:29.010 DEBUG:teuthology.orchestra.run.smithi096:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:05:29.021 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:05:29.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:29.422+0000 7f8681600640 1 -- 172.21.15.96:0/1794160895 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v31) ==== 170+0+0 (secure 0 0 0) 0x7f867807b3c0 con 0x7f86840725e0 2024-05-28T06:05:29.424 INFO:teuthology.orchestra.run.smithi096.stderr:pool 'cephfs_data' created 2024-05-28T06:05:29.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:29.426+0000 7f868c200640 1 -- 172.21.15.96:0/1794160895 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f866406c8f0 msgr2=0x7f866406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:29.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:29.426+0000 7f868c200640 1 --2- 172.21.15.96:0/1794160895 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f866406c8f0 0x7f866406edb0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f8670006fd0 tx=0x7f8670006bc0 comp rx=0 tx=0).stop 2024-05-28T06:05:29.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:29.426+0000 7f868c200640 1 -- 172.21.15.96:0/1794160895 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f86840725e0 msgr2=0x7f86840cf6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:29.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:29.426+0000 7f868c200640 1 --2- 172.21.15.96:0/1794160895 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f86840725e0 0x7f86840cf6b0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f8678035040 tx=0x7f86780046c0 comp rx=0 tx=0).stop 2024-05-28T06:05:29.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:29.426+0000 7f868c200640 1 -- 172.21.15.96:0/1794160895 shutdown_connections 2024-05-28T06:05:29.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:29.426+0000 7f868c200640 1 --2- 172.21.15.96:0/1794160895 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f866406c8f0 0x7f866406edb0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:29.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:29.426+0000 7f868c200640 1 --2- 172.21.15.96:0/1794160895 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f868415b6f0 0x7f868414ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:29.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:29.426+0000 7f868c200640 1 --2- 172.21.15.96:0/1794160895 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f86840d0640 0x7f868414b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:29.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:29.426+0000 7f868c200640 1 --2- 172.21.15.96:0/1794160895 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f86840725e0 0x7f86840cf6b0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:29.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:29.426+0000 7f868c200640 1 -- 172.21.15.96:0/1794160895 >> 172.21.15.96:0/1794160895 conn(0x7f868406bd20 msgr2=0x7f86840b1150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:29.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:29.426+0000 7f868c200640 1 -- 172.21.15.96:0/1794160895 shutdown_connections 2024-05-28T06:05:29.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:29.427+0000 7f868c200640 1 -- 172.21.15.96:0/1794160895 wait complete. 2024-05-28T06:05:29.464 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-05-28T06:05:29.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:29.641+0000 7fc8c4400640 1 Processor -- start 2024-05-28T06:05:29.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:29.641+0000 7fc8c4400640 1 -- start start 2024-05-28T06:05:29.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:29.642+0000 7fc8c4400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fc8bc0d0640 0x7fc8bc15d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:29.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:29.642+0000 7fc8c4400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fc8bc0b32c0 0x7fc8bc0b3730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:29.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:29.642+0000 7fc8c4400640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8bc0d0060 con 0x7fc8bc0d0a20 2024-05-28T06:05:29.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:29.642+0000 7fc8c4400640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fc8bc0adbb0 con 0x7fc8bc0d0640 2024-05-28T06:05:29.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:29.642+0000 7fc8c4400640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fc8bc15db10 con 0x7fc8bc0b32c0 2024-05-28T06:05:29.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:29.643+0000 7fc8c1a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fc8bc0d0640 0x7fc8bc15d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:29.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:29.643+0000 7fc8c1a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fc8bc0d0640 0x7fc8bc15d5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:50764/0 (socket says 172.21.15.96:50764) 2024-05-28T06:05:29.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:29.643+0000 7fc8c1a00640 1 -- 172.21.15.96:0/623212311 learned_addr learned my addr 172.21.15.96:0/623212311 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:29.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:29.643+0000 7fc8c2400640 1 --2- 172.21.15.96:0/623212311 >> v2:172.21.15.123:3300/0 conn(0x7fc8bc0b32c0 0x7fc8bc0b3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:29.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:29.643+0000 7fc8c1a00640 1 -- 172.21.15.96:0/623212311 >> v2:172.21.15.123:3300/0 conn(0x7fc8bc0b32c0 msgr2=0x7fc8bc0b3730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:29.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:29.643+0000 7fc8c1a00640 1 --2- 172.21.15.96:0/623212311 >> v2:172.21.15.123:3300/0 conn(0x7fc8bc0b32c0 0x7fc8bc0b3730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:29.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:29.643+0000 7fc8c1a00640 1 -- 172.21.15.96:0/623212311 >> v1:172.21.15.123:6789/0 conn(0x7fc8bc0d0a20 legacy=0x7fc8bc0cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:29.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:29.643+0000 7fc8c1a00640 1 -- 172.21.15.96:0/623212311 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8bc15cb10 con 0x7fc8bc0d0640 2024-05-28T06:05:29.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:29.644+0000 7fc8c1a00640 1 --2- 172.21.15.96:0/623212311 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc8bc0d0640 0x7fc8bc15d5d0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fc8bc0d1230 tx=0x7fc8b002a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa728a3a01db75cc server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:29.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:29.645+0000 7fc8bbe00640 1 -- 172.21.15.96:0/623212311 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8b0037070 con 0x7fc8bc0d0640 2024-05-28T06:05:29.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:29.645+0000 7fc8bbe00640 1 -- 172.21.15.96:0/623212311 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8b0032440 con 0x7fc8bc0d0640 2024-05-28T06:05:29.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:29.645+0000 7fc8bbe00640 1 -- 172.21.15.96:0/623212311 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8b0032810 con 0x7fc8bc0d0640 2024-05-28T06:05:29.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:29.645+0000 7fc8c4400640 1 -- 172.21.15.96:0/623212311 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc8bc0d0640 msgr2=0x7fc8bc15d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:29.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:29.645+0000 7fc8c4400640 1 --2- 172.21.15.96:0/623212311 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc8bc0d0640 0x7fc8bc15d5d0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fc8bc0d1230 tx=0x7fc8b002a1c0 comp rx=0 tx=0).stop 2024-05-28T06:05:29.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:29.645+0000 7fc8c4400640 1 -- 172.21.15.96:0/623212311 shutdown_connections 2024-05-28T06:05:29.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:29.645+0000 7fc8c4400640 1 --2- 172.21.15.96:0/623212311 >> v2:172.21.15.123:3300/0 conn(0x7fc8bc0b32c0 0x7fc8bc0b3730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:29.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:29.645+0000 7fc8c4400640 1 --2- 172.21.15.96:0/623212311 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc8bc0d0640 0x7fc8bc15d5d0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:29.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:29.645+0000 7fc8c4400640 1 -- 172.21.15.96:0/623212311 >> 172.21.15.96:0/623212311 conn(0x7fc8bc06bd20 msgr2=0x7fc8bc06c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:29.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:29.645+0000 7fc8c4400640 1 -- 172.21.15.96:0/623212311 shutdown_connections 2024-05-28T06:05:29.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:29.645+0000 7fc8c4400640 1 -- 172.21.15.96:0/623212311 wait complete. 2024-05-28T06:05:29.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:29.647+0000 7fc8c4400640 1 Processor -- start 2024-05-28T06:05:29.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:29.647+0000 7fc8c4400640 1 -- start start 2024-05-28T06:05:29.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:29.648+0000 7fc8c4400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc8bc0b32c0 0x7fc8bc099b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:29.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:29.648+0000 7fc8c4400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc8bc0d0640 0x7fc8bc14cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:29.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:29.648+0000 7fc8c4400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc8bc0d0a20 0x7fc8bc13ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:29.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:29.648+0000 7fc8c4400640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc8bc15cdd0 con 0x7fc8bc0b32c0 2024-05-28T06:05:29.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:29.648+0000 7fc8c4400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc8bc0ad8d0 con 0x7fc8bc0d0640 2024-05-28T06:05:29.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:29.648+0000 7fc8c4400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fc8bc15d0b0 con 0x7fc8bc0d0a20 2024-05-28T06:05:29.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:29.648+0000 7fc8c1a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc8bc0b32c0 0x7fc8bc099b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:29.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:29.649+0000 7fc8c1a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc8bc0b32c0 0x7fc8bc099b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:50774/0 (socket says 172.21.15.96:50774) 2024-05-28T06:05:29.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:29.649+0000 7fc8c1a00640 1 -- 172.21.15.96:0/3731054653 learned_addr learned my addr 172.21.15.96:0/3731054653 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:29.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:29.649+0000 7fc8c1000640 1 --2- 172.21.15.96:0/3731054653 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc8bc0d0640 0x7fc8bc14cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:29.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:29.649+0000 7fc8c1a00640 1 -- 172.21.15.96:0/3731054653 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc8bc0d0a20 msgr2=0x7fc8bc13ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:29.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:29.649+0000 7fc8c1a00640 1 --2- 172.21.15.96:0/3731054653 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc8bc0d0a20 0x7fc8bc13ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:29.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:29.649+0000 7fc8c1a00640 1 -- 172.21.15.96:0/3731054653 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc8bc0d0640 msgr2=0x7fc8bc14cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:29.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:29.649+0000 7fc8c1a00640 1 --2- 172.21.15.96:0/3731054653 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc8bc0d0640 0x7fc8bc14cb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:29.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:29.649+0000 7fc8c1a00640 1 -- 172.21.15.96:0/3731054653 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8bc1543b0 con 0x7fc8bc0b32c0 2024-05-28T06:05:29.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:29.649+0000 7fc8c1000640 1 --2- 172.21.15.96:0/3731054653 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc8bc0d0640 0x7fc8bc14cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:05:29.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:29.649+0000 7fc8c1a00640 1 --2- 172.21.15.96:0/3731054653 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc8bc0b32c0 0x7fc8bc099b10 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fc8b00097f0 tx=0x7fc8b00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:29.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:29.650+0000 7fc8b9600640 1 -- 172.21.15.96:0/3731054653 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8b0037040 con 0x7fc8bc0b32c0 2024-05-28T06:05:29.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:29.650+0000 7fc8b9600640 1 -- 172.21.15.96:0/3731054653 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8b0004230 con 0x7fc8bc0b32c0 2024-05-28T06:05:29.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:29.650+0000 7fc8b9600640 1 -- 172.21.15.96:0/3731054653 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8b0003610 con 0x7fc8bc0b32c0 2024-05-28T06:05:29.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:29.650+0000 7fc8c4400640 1 -- 172.21.15.96:0/3731054653 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8bc154690 con 0x7fc8bc0b32c0 2024-05-28T06:05:29.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:29.650+0000 7fc8c4400640 1 -- 172.21.15.96:0/3731054653 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8bc14d040 con 0x7fc8bc0b32c0 2024-05-28T06:05:29.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:29.651+0000 7fc8c4400640 1 -- 172.21.15.96:0/3731054653 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8bc154bb0 con 0x7fc8bc0b32c0 2024-05-28T06:05:29.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:29.652+0000 7fc8b9600640 1 -- 172.21.15.96:0/3731054653 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc8b00037b0 con 0x7fc8bc0b32c0 2024-05-28T06:05:29.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:29.653+0000 7fc8b9600640 1 --2- 172.21.15.96:0/3731054653 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc89c06c8f0 0x7fc89c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:29.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:29.653+0000 7fc8b9600640 1 -- 172.21.15.96:0/3731054653 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 5609+0+0 (secure 0 0 0) 0x7fc8b00ac920 con 0x7fc8bc0b32c0 2024-05-28T06:05:29.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:29.654+0000 7fc8c1000640 1 --2- 172.21.15.96:0/3731054653 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc89c06c8f0 0x7fc89c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:29.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:29.655+0000 7fc8c1000640 1 --2- 172.21.15.96:0/3731054653 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc89c06c8f0 0x7fc89c06edb0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fc8bc14d350 tx=0x7fc8a8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:29.659 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:29.658+0000 7fc8b9600640 1 -- 172.21.15.96:0/3731054653 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc8b007ad10 con 0x7fc8bc0b32c0 2024-05-28T06:05:29.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:29.891+0000 7fc8c4400640 1 -- 172.21.15.96:0/3731054653 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fc8bc0689e0 con 0x7fc8bc0b32c0 2024-05-28T06:05:30.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:30.458+0000 7fc8b9600640 1 -- 172.21.15.96:0/3731054653 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 4 and data pool 5 v12) ==== 172+0+0 (secure 0 0 0) 0x7fc8b007a610 con 0x7fc8bc0b32c0 2024-05-28T06:05:30.463 INFO:teuthology.orchestra.run.smithi096.stderr:new fs with metadata pool 4 and data pool 5 2024-05-28T06:05:30.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:30.467+0000 7fc8c4400640 1 -- 172.21.15.96:0/3731054653 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc89c06c8f0 msgr2=0x7fc89c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:30.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:30.467+0000 7fc8c4400640 1 --2- 172.21.15.96:0/3731054653 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc89c06c8f0 0x7fc89c06edb0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fc8bc14d350 tx=0x7fc8a8008040 comp rx=0 tx=0).stop 2024-05-28T06:05:30.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:30.467+0000 7fc8c4400640 1 -- 172.21.15.96:0/3731054653 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc8bc0b32c0 msgr2=0x7fc8bc099b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:30.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:30.467+0000 7fc8c4400640 1 --2- 172.21.15.96:0/3731054653 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc8bc0b32c0 0x7fc8bc099b10 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fc8b00097f0 tx=0x7fc8b00046c0 comp rx=0 tx=0).stop 2024-05-28T06:05:30.469 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:30.467+0000 7fc8c4400640 1 -- 172.21.15.96:0/3731054653 shutdown_connections 2024-05-28T06:05:30.469 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:30.467+0000 7fc8c4400640 1 --2- 172.21.15.96:0/3731054653 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc89c06c8f0 0x7fc89c06edb0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:30.469 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:30.467+0000 7fc8c4400640 1 --2- 172.21.15.96:0/3731054653 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc8bc0d0a20 0x7fc8bc13ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:30.469 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:30.467+0000 7fc8c4400640 1 --2- 172.21.15.96:0/3731054653 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc8bc0d0640 0x7fc8bc14cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:30.469 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:30.467+0000 7fc8c4400640 1 --2- 172.21.15.96:0/3731054653 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc8bc0b32c0 0x7fc8bc099b10 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:30.469 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:30.467+0000 7fc8c4400640 1 -- 172.21.15.96:0/3731054653 >> 172.21.15.96:0/3731054653 conn(0x7fc8bc06bd20 msgr2=0x7fc8bc0b0d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:30.470 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:30.468+0000 7fc8c4400640 1 -- 172.21.15.96:0/3731054653 shutdown_connections 2024-05-28T06:05:30.471 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:30.470+0000 7fc8c4400640 1 -- 172.21.15.96:0/3731054653 wait complete. 2024-05-28T06:05:30.510 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:05:30.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:30.675+0000 7f9f30200640 1 Processor -- start 2024-05-28T06:05:30.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:30.675+0000 7f9f30200640 1 -- start start 2024-05-28T06:05:30.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:30.676+0000 7f9f30200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f9f280b4b60 0x7f9f280cfaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:30.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:30.677+0000 7f9f30200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9f280b3b70 0x7f9f280d0930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:30.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:30.677+0000 7f9f30200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f280ced40 con 0x7f9f280d04a0 2024-05-28T06:05:30.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:30.677+0000 7f9f30200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f9f280b3fe0 con 0x7f9f280b4b60 2024-05-28T06:05:30.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:30.677+0000 7f9f30200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f9f2815fbb0 con 0x7f9f280b3b70 2024-05-28T06:05:30.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:30.677+0000 7f9f2d800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f9f280b4b60 0x7f9f280cfaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:30.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:30.677+0000 7f9f2d800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f9f280b4b60 0x7f9f280cfaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:50776/0 (socket says 172.21.15.96:50776) 2024-05-28T06:05:30.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:30.677+0000 7f9f2d800640 1 -- 172.21.15.96:0/3646038789 learned_addr learned my addr 172.21.15.96:0/3646038789 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:30.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:30.677+0000 7f9f2d800640 1 -- 172.21.15.96:0/3646038789 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9f280b3b70 msgr2=0x7f9f280d0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:30.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:30.677+0000 7f9f2e200640 1 --2- 172.21.15.96:0/3646038789 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9f280b3b70 0x7f9f280d0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:30.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:30.677+0000 7f9f2d800640 1 --2- 172.21.15.96:0/3646038789 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9f280b3b70 0x7f9f280d0930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:30.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:30.677+0000 7f9f2d800640 1 -- 172.21.15.96:0/3646038789 >> v1:172.21.15.123:6789/0 conn(0x7f9f280d04a0 legacy=0x7f9f280b3460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:30.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:30.677+0000 7f9f2d800640 1 -- 172.21.15.96:0/3646038789 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f280cefc0 con 0x7f9f280b4b60 2024-05-28T06:05:30.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:30.678+0000 7f9f2d800640 1 --2- 172.21.15.96:0/3646038789 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f280b4b60 0x7f9f280cfaa0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f9f280cdd80 tx=0x7f9f1c02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1539756a89b1c7b5 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:30.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:30.679+0000 7f9f27e00640 1 -- 172.21.15.96:0/3646038789 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f1c037070 con 0x7f9f280b4b60 2024-05-28T06:05:30.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:30.679+0000 7f9f27e00640 1 -- 172.21.15.96:0/3646038789 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f1c032440 con 0x7f9f280b4b60 2024-05-28T06:05:30.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:30.679+0000 7f9f27e00640 1 -- 172.21.15.96:0/3646038789 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f1c032810 con 0x7f9f280b4b60 2024-05-28T06:05:30.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:30.679+0000 7f9f30200640 1 -- 172.21.15.96:0/3646038789 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f280b4b60 msgr2=0x7f9f280cfaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:30.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:30.679+0000 7f9f30200640 1 --2- 172.21.15.96:0/3646038789 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f280b4b60 0x7f9f280cfaa0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f9f280cdd80 tx=0x7f9f1c02a1e0 comp rx=0 tx=0).stop 2024-05-28T06:05:30.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:30.679+0000 7f9f30200640 1 -- 172.21.15.96:0/3646038789 shutdown_connections 2024-05-28T06:05:30.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:30.679+0000 7f9f30200640 1 --2- 172.21.15.96:0/3646038789 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9f280b3b70 0x7f9f280d0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:30.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:30.679+0000 7f9f30200640 1 --2- 172.21.15.96:0/3646038789 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f280b4b60 0x7f9f280cfaa0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:30.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:30.679+0000 7f9f30200640 1 -- 172.21.15.96:0/3646038789 >> 172.21.15.96:0/3646038789 conn(0x7f9f2806bae0 msgr2=0x7f9f2806bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:30.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:30.679+0000 7f9f30200640 1 -- 172.21.15.96:0/3646038789 shutdown_connections 2024-05-28T06:05:30.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:30.679+0000 7f9f30200640 1 -- 172.21.15.96:0/3646038789 wait complete. 2024-05-28T06:05:30.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:30.681+0000 7f9f30200640 1 Processor -- start 2024-05-28T06:05:30.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:30.681+0000 7f9f30200640 1 -- start start 2024-05-28T06:05:30.684 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:30.682+0000 7f9f30200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9f280b3b70 0x7f9f281596e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:30.684 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:30.682+0000 7f9f30200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f280b4b60 0x7f9f280c4840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:30.684 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:30.682+0000 7f9f30200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f280d04a0 0x7f9f28159e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:30.684 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:30.682+0000 7f9f30200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f9f280cffe0 con 0x7f9f280d04a0 2024-05-28T06:05:30.684 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:30.682+0000 7f9f30200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f9f280cf580 con 0x7f9f280b4b60 2024-05-28T06:05:30.684 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:30.682+0000 7f9f30200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f9f2815ede0 con 0x7f9f280b3b70 2024-05-28T06:05:30.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:30.683+0000 7f9f2e200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f280d04a0 0x7f9f28159e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:30.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:30.683+0000 7f9f2e200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f280d04a0 0x7f9f28159e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:50780/0 (socket says 172.21.15.96:50780) 2024-05-28T06:05:30.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:30.683+0000 7f9f2e200640 1 -- 172.21.15.96:0/1355315171 learned_addr learned my addr 172.21.15.96:0/1355315171 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:30.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:30.683+0000 7f9f2ce00640 1 --2- 172.21.15.96:0/1355315171 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f280b4b60 0x7f9f280c4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:30.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:30.683+0000 7f9f2d800640 1 --2- 172.21.15.96:0/1355315171 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9f280b3b70 0x7f9f281596e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:30.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:30.683+0000 7f9f2e200640 1 -- 172.21.15.96:0/1355315171 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9f280b3b70 msgr2=0x7f9f281596e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:30.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:30.683+0000 7f9f2e200640 1 --2- 172.21.15.96:0/1355315171 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9f280b3b70 0x7f9f281596e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:30.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:30.683+0000 7f9f2e200640 1 -- 172.21.15.96:0/1355315171 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f280b4b60 msgr2=0x7f9f280c4840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:30.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:30.683+0000 7f9f2e200640 1 --2- 172.21.15.96:0/1355315171 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f280b4b60 0x7f9f280c4840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:30.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:30.683+0000 7f9f2e200640 1 -- 172.21.15.96:0/1355315171 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f28159c20 con 0x7f9f280d04a0 2024-05-28T06:05:30.686 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:30.683+0000 7f9f2e200640 1 --2- 172.21.15.96:0/1355315171 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f280d04a0 0x7f9f28159e20 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f9f2000cd70 tx=0x7f9f2000a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:30.686 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:30.683+0000 7f9f25600640 1 -- 172.21.15.96:0/1355315171 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f20003b90 con 0x7f9f280d04a0 2024-05-28T06:05:30.686 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:30.684+0000 7f9f25600640 1 -- 172.21.15.96:0/1355315171 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f20003d30 con 0x7f9f280d04a0 2024-05-28T06:05:30.686 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:30.684+0000 7f9f25600640 1 -- 172.21.15.96:0/1355315171 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f20004240 con 0x7f9f280d04a0 2024-05-28T06:05:30.687 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:30.684+0000 7f9f30200640 1 -- 172.21.15.96:0/1355315171 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f280c5e30 con 0x7f9f280d04a0 2024-05-28T06:05:30.687 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:30.684+0000 7f9f30200640 1 -- 172.21.15.96:0/1355315171 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f280c63f0 con 0x7f9f280d04a0 2024-05-28T06:05:30.687 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:30.685+0000 7f9f30200640 1 -- 172.21.15.96:0/1355315171 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f280689e0 con 0x7f9f280d04a0 2024-05-28T06:05:30.687 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:30.686+0000 7f9f25600640 1 -- 172.21.15.96:0/1355315171 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9f200043e0 con 0x7f9f280d04a0 2024-05-28T06:05:30.688 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:30.687+0000 7f9f25600640 1 --2- 172.21.15.96:0/1355315171 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9f0806c8f0 0x7f9f0806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:30.688 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:30.687+0000 7f9f25600640 1 -- 172.21.15.96:0/1355315171 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5709+0+0 (secure 0 0 0) 0x7f9f2008c690 con 0x7f9f280d04a0 2024-05-28T06:05:30.689 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:30.688+0000 7f9f2d800640 1 --2- 172.21.15.96:0/1355315171 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9f0806c8f0 0x7f9f0806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:30.689 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:30.688+0000 7f9f2d800640 1 --2- 172.21.15.96:0/1355315171 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9f0806c8f0 0x7f9f0806edb0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f9f1c034000 tx=0x7f9f1c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:30.693 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:30.692+0000 7f9f25600640 1 -- 172.21.15.96:0/1355315171 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9f28159c20 con 0x7f9f280d04a0 2024-05-28T06:05:30.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:30.891+0000 7f9f30200640 1 -- 172.21.15.96:0/1355315171 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9f280adf50 con 0x7f9f280d04a0 2024-05-28T06:05:30.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:30.894+0000 7f9f25600640 1 -- 172.21.15.96:0/1355315171 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v32) ==== 74+0+16694 (secure 0 0 0) 0x7f9f2005aa20 con 0x7f9f280d04a0 2024-05-28T06:05:30.896 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:05:30.896 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":32,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:05:29.892978+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-05-28T06:05:27.674327+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-05-28T06:05:28.878204+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:05:30.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:30.897+0000 7f9f30200640 1 -- 172.21.15.96:0/1355315171 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9f0806c8f0 msgr2=0x7f9f0806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:30.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:30.897+0000 7f9f30200640 1 --2- 172.21.15.96:0/1355315171 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9f0806c8f0 0x7f9f0806edb0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f9f1c034000 tx=0x7f9f1c02e000 comp rx=0 tx=0).stop 2024-05-28T06:05:30.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:30.897+0000 7f9f30200640 1 -- 172.21.15.96:0/1355315171 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f280d04a0 msgr2=0x7f9f28159e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:30.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:30.897+0000 7f9f30200640 1 --2- 172.21.15.96:0/1355315171 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f280d04a0 0x7f9f28159e20 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f9f2000cd70 tx=0x7f9f2000a660 comp rx=0 tx=0).stop 2024-05-28T06:05:30.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:30.897+0000 7f9f30200640 1 -- 172.21.15.96:0/1355315171 shutdown_connections 2024-05-28T06:05:30.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:30.897+0000 7f9f30200640 1 --2- 172.21.15.96:0/1355315171 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9f0806c8f0 0x7f9f0806edb0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:30.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:30.897+0000 7f9f30200640 1 --2- 172.21.15.96:0/1355315171 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f280d04a0 0x7f9f28159e20 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:30.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:30.897+0000 7f9f30200640 1 --2- 172.21.15.96:0/1355315171 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f280b4b60 0x7f9f280c4840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:30.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:30.897+0000 7f9f30200640 1 --2- 172.21.15.96:0/1355315171 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9f280b3b70 0x7f9f281596e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:30.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:30.897+0000 7f9f30200640 1 -- 172.21.15.96:0/1355315171 >> 172.21.15.96:0/1355315171 conn(0x7f9f2806bae0 msgr2=0x7f9f280ad130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:30.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:30.898+0000 7f9f30200640 1 -- 172.21.15.96:0/1355315171 shutdown_connections 2024-05-28T06:05:30.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:30.898+0000 7f9f30200640 1 -- 172.21.15.96:0/1355315171 wait complete. 2024-05-28T06:05:30.918 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:05:31.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.108+0000 7f13b8600640 1 Processor -- start 2024-05-28T06:05:31.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.108+0000 7f13b8600640 1 -- start start 2024-05-28T06:05:31.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.110+0000 7f13b8600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f13b006f860 0x7f13b00b3150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:31.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.110+0000 7f13b8600640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13b006ed10 con 0x7f13b00b3690 2024-05-28T06:05:31.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.110+0000 7f13b8600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13b015cb00 con 0x7f13b00d0bb0 2024-05-28T06:05:31.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.110+0000 7f13b8600640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f13b00b3cc0 con 0x7f13b006f860 2024-05-28T06:05:31.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.110+0000 7f13b5200640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f13b00b3690 0x7f13b00d04a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:40270/0 (socket says 172.21.15.96:40270) 2024-05-28T06:05:31.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.110+0000 7f13b5200640 1 -- 172.21.15.96:0/2855365519 learned_addr learned my addr 172.21.15.96:0/2855365519 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:05:31.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.110+0000 7f13b5c00640 1 --2- 172.21.15.96:0/2855365519 >> v2:172.21.15.123:3300/0 conn(0x7f13b006f860 0x7f13b00b3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:31.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.111+0000 7f13afe00640 1 -- 172.21.15.96:0/2855365519 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3176698670 0 0) 0x7f139c001110 con 0x7f13b00b3690 2024-05-28T06:05:31.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.111+0000 7f13afe00640 1 -- 172.21.15.96:0/2855365519 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1398003520 con 0x7f13b00b3690 2024-05-28T06:05:31.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.111+0000 7f13afe00640 1 -- 172.21.15.96:0/2855365519 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2634695636 0 0) 0x7f1398003520 con 0x7f13b00b3690 2024-05-28T06:05:31.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.111+0000 7f13afe00640 1 -- 172.21.15.96:0/2855365519 >> v2:172.21.15.123:3300/0 conn(0x7f13b006f860 msgr2=0x7f13b00b3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:31.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.111+0000 7f13afe00640 1 --2- 172.21.15.96:0/2855365519 >> v2:172.21.15.123:3300/0 conn(0x7f13b006f860 0x7f13b00b3150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:31.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.111+0000 7f13afe00640 1 -- 172.21.15.96:0/2855365519 >> v1:172.21.15.123:6789/0 conn(0x7f13b00d0bb0 legacy=0x7f13b00ad130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:31.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.111+0000 7f13afe00640 1 -- 172.21.15.96:0/2855365519 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13b00b3fa0 con 0x7f13b00b3690 2024-05-28T06:05:31.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.111+0000 7f13afe00640 1 -- 172.21.15.96:0/2855365519 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f139c001840 con 0x7f13b00b3690 2024-05-28T06:05:31.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.111+0000 7f13afe00640 1 -- 172.21.15.96:0/2855365519 >> v1:172.21.15.96:6789/0 conn(0x7f13b00b3690 legacy=0x7f13b00d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:31.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.111+0000 7f13afe00640 1 --2- 172.21.15.96:0/2855365519 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1398003a00 0x7f1398003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:31.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.111+0000 7f13afe00640 1 --2- 172.21.15.96:0/2855365519 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1398004800 0x7f1398004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:31.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.111+0000 7f13afe00640 1 --2- 172.21.15.96:0/2855365519 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13980051c0 0x7f139800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:31.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.111+0000 7f13afe00640 1 -- 172.21.15.96:0/2855365519 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f13b00b3cc0 con 0x7f13980051c0 2024-05-28T06:05:31.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.111+0000 7f13afe00640 1 -- 172.21.15.96:0/2855365519 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f139800c0c0 con 0x7f1398003a00 2024-05-28T06:05:31.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.111+0000 7f13afe00640 1 -- 172.21.15.96:0/2855365519 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f139800c2c0 con 0x7f1398004800 2024-05-28T06:05:31.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.112+0000 7f13b6600640 1 --2- 172.21.15.96:0/2855365519 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13980051c0 0x7f139800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:31.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.112+0000 7f13b5c00640 1 --2- 172.21.15.96:0/2855365519 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1398003a00 0x7f1398003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:31.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.112+0000 7f13b5200640 1 --2- 172.21.15.96:0/2855365519 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1398004800 0x7f1398004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:31.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.112+0000 7f13b6600640 1 -- 172.21.15.96:0/2855365519 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1398004800 msgr2=0x7f1398004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:31.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.112+0000 7f13b6600640 1 --2- 172.21.15.96:0/2855365519 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1398004800 0x7f1398004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:31.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.112+0000 7f13b6600640 1 -- 172.21.15.96:0/2855365519 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1398003a00 msgr2=0x7f1398003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:31.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.112+0000 7f13b6600640 1 --2- 172.21.15.96:0/2855365519 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1398003a00 0x7f1398003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:31.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.112+0000 7f13b6600640 1 -- 172.21.15.96:0/2855365519 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1398003520 con 0x7f13980051c0 2024-05-28T06:05:31.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.113+0000 7f13b6600640 1 -- 172.21.15.96:0/2855365519 reap_dead start 2024-05-28T06:05:31.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.113+0000 7f13b6600640 1 --2- 172.21.15.96:0/2855365519 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13980051c0 0x7f139800b9c0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f13a8001e50 tx=0x7f1398005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccdcde433d97eafe server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:31.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.113+0000 7f13afe00640 1 -- 172.21.15.96:0/2855365519 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13a8001a70 con 0x7f13980051c0 2024-05-28T06:05:31.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.113+0000 7f13afe00640 1 -- 172.21.15.96:0/2855365519 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13a8003970 con 0x7f13980051c0 2024-05-28T06:05:31.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.113+0000 7f13afe00640 1 -- 172.21.15.96:0/2855365519 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13a8003d60 con 0x7f13980051c0 2024-05-28T06:05:31.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.114+0000 7f13b8600640 1 -- 172.21.15.96:0/2855365519 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13980051c0 msgr2=0x7f139800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:31.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.114+0000 7f13b8600640 1 --2- 172.21.15.96:0/2855365519 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13980051c0 0x7f139800b9c0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f13a8001e50 tx=0x7f1398005830 comp rx=0 tx=0).stop 2024-05-28T06:05:31.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.114+0000 7f13b8600640 1 -- 172.21.15.96:0/2855365519 shutdown_connections 2024-05-28T06:05:31.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.114+0000 7f13b8600640 1 --2- 172.21.15.96:0/2855365519 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13980051c0 0x7f139800b9c0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:31.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.114+0000 7f13b8600640 1 -- 172.21.15.96:0/2855365519 >> 172.21.15.96:0/2855365519 conn(0x7f13b006bae0 msgr2=0x7f13b006bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:31.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.114+0000 7f13b8600640 1 -- 172.21.15.96:0/2855365519 shutdown_connections 2024-05-28T06:05:31.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.114+0000 7f13b8600640 1 -- 172.21.15.96:0/2855365519 wait complete. 2024-05-28T06:05:31.119 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.116+0000 7f13b8600640 1 Processor -- start 2024-05-28T06:05:31.119 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.116+0000 7f13b8600640 1 -- start start 2024-05-28T06:05:31.119 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.117+0000 7f13b8600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f13980051c0 0x7f13b0099530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:31.119 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.117+0000 7f13b8600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13b00cd850 0x7f13b00c5f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:31.119 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.117+0000 7f13b8600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f13b00b11f0 0x7f13b015e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:31.119 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.117+0000 7f13b8600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f13b00b4840 con 0x7f13b00cd850 2024-05-28T06:05:31.119 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.117+0000 7f13b8600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f13b00b4560 con 0x7f13b00b11f0 2024-05-28T06:05:31.119 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.117+0000 7f13b8600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f13b00b3150 con 0x7f13980051c0 2024-05-28T06:05:31.119 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.117+0000 7f13b5200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13b00cd850 0x7f13b00c5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:31.119 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.117+0000 7f13b5200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13b00cd850 0x7f13b00c5f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:50792/0 (socket says 172.21.15.96:50792) 2024-05-28T06:05:31.119 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.117+0000 7f13b5200640 1 -- 172.21.15.96:0/934050897 learned_addr learned my addr 172.21.15.96:0/934050897 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:31.119 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.117+0000 7f13b5c00640 1 --2- 172.21.15.96:0/934050897 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f13980051c0 0x7f13b0099530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:31.120 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.117+0000 7f13b6600640 1 --2- 172.21.15.96:0/934050897 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f13b00b11f0 0x7f13b015e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:31.120 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.118+0000 7f13b5200640 1 -- 172.21.15.96:0/934050897 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f13980051c0 msgr2=0x7f13b0099530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:31.120 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.118+0000 7f13b5200640 1 --2- 172.21.15.96:0/934050897 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f13980051c0 0x7f13b0099530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:31.120 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.118+0000 7f13b5200640 1 -- 172.21.15.96:0/934050897 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f13b00b11f0 msgr2=0x7f13b015e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:31.120 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.118+0000 7f13b5200640 1 --2- 172.21.15.96:0/934050897 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f13b00b11f0 0x7f13b015e980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:31.120 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.118+0000 7f13b5200640 1 -- 172.21.15.96:0/934050897 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13b00c64d0 con 0x7f13b00cd850 2024-05-28T06:05:31.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.118+0000 7f13b5200640 1 --2- 172.21.15.96:0/934050897 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13b00cd850 0x7f13b00c5f90 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f139c001eb0 tx=0x7f139c00eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:31.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.118+0000 7f13ad600640 1 -- 172.21.15.96:0/934050897 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f139c00c6c0 con 0x7f13b00cd850 2024-05-28T06:05:31.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.118+0000 7f13ad600640 1 -- 172.21.15.96:0/934050897 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f139c018070 con 0x7f13b00cd850 2024-05-28T06:05:31.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.118+0000 7f13ad600640 1 -- 172.21.15.96:0/934050897 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f139c00caa0 con 0x7f13b00cd850 2024-05-28T06:05:31.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.118+0000 7f13b8600640 1 -- 172.21.15.96:0/934050897 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13b00c4ef0 con 0x7f13b00cd850 2024-05-28T06:05:31.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.118+0000 7f13b8600640 1 -- 172.21.15.96:0/934050897 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13b00c54b0 con 0x7f13b00cd850 2024-05-28T06:05:31.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.120+0000 7f13b8600640 1 -- 172.21.15.96:0/934050897 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13b015e780 con 0x7f13b00cd850 2024-05-28T06:05:31.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.120+0000 7f13ad600640 1 -- 172.21.15.96:0/934050897 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f139c00cc40 con 0x7f13b00cd850 2024-05-28T06:05:31.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.121+0000 7f13ad600640 1 --2- 172.21.15.96:0/934050897 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f139006c920 0x7f139006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:31.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.121+0000 7f13ad600640 1 -- 172.21.15.96:0/934050897 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5709+0+0 (secure 0 0 0) 0x7f139c08b6e0 con 0x7f13b00cd850 2024-05-28T06:05:31.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.122+0000 7f13b5c00640 1 --2- 172.21.15.96:0/934050897 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f139006c920 0x7f139006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:31.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.123+0000 7f13b5c00640 1 --2- 172.21.15.96:0/934050897 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f139006c920 0x7f139006ede0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f13a4009990 tx=0x7f13a4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:31.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.126+0000 7f13ad600640 1 -- 172.21.15.96:0/934050897 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f13b015e780 con 0x7f13b00cd850 2024-05-28T06:05:31.327 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.326+0000 7f13b8600640 1 -- 172.21.15.96:0/934050897 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f13b00689e0 con 0x7f13b00cd850 2024-05-28T06:05:31.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.328+0000 7f13ad600640 1 -- 172.21.15.96:0/934050897 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v32) ==== 74+0+16694 (secure 0 0 0) 0x7f13b006ed10 con 0x7f13b00cd850 2024-05-28T06:05:31.330 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:05:31.330 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":32,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:05:29.892978+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-05-28T06:05:27.674327+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-05-28T06:05:28.878204+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:05:31.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.331+0000 7f13b8600640 1 -- 172.21.15.96:0/934050897 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f139006c920 msgr2=0x7f139006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:31.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.331+0000 7f13b8600640 1 --2- 172.21.15.96:0/934050897 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f139006c920 0x7f139006ede0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f13a4009990 tx=0x7f13a4008040 comp rx=0 tx=0).stop 2024-05-28T06:05:31.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.331+0000 7f13b8600640 1 -- 172.21.15.96:0/934050897 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13b00cd850 msgr2=0x7f13b00c5f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:31.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.331+0000 7f13b8600640 1 --2- 172.21.15.96:0/934050897 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13b00cd850 0x7f13b00c5f90 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f139c001eb0 tx=0x7f139c00eb90 comp rx=0 tx=0).stop 2024-05-28T06:05:31.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.331+0000 7f13b8600640 1 -- 172.21.15.96:0/934050897 shutdown_connections 2024-05-28T06:05:31.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.331+0000 7f13b8600640 1 --2- 172.21.15.96:0/934050897 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f139006c920 0x7f139006ede0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:31.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.331+0000 7f13b8600640 1 --2- 172.21.15.96:0/934050897 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f13b00b11f0 0x7f13b015e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:31.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.331+0000 7f13b8600640 1 --2- 172.21.15.96:0/934050897 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13b00cd850 0x7f13b00c5f90 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:31.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.331+0000 7f13b8600640 1 --2- 172.21.15.96:0/934050897 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f13980051c0 0x7f13b0099530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:31.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.331+0000 7f13b8600640 1 -- 172.21.15.96:0/934050897 >> 172.21.15.96:0/934050897 conn(0x7f13b006bae0 msgr2=0x7f13b00cbc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:31.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.331+0000 7f13b8600640 1 -- 172.21.15.96:0/934050897 shutdown_connections 2024-05-28T06:05:31.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.331+0000 7f13b8600640 1 -- 172.21.15.96:0/934050897 wait complete. 2024-05-28T06:05:31.351 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-05-28T06:05:31.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.546+0000 7f7128200640 1 Processor -- start 2024-05-28T06:05:31.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.546+0000 7f7128200640 1 -- start start 2024-05-28T06:05:31.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.547+0000 7f7128200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f712015d5d0 0x7f71200cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:31.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.547+0000 7f7128200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f71200b32c0 0x7f71200b3700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:31.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.547+0000 7f7128200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f712015cb10 con 0x7f71200d0640 2024-05-28T06:05:31.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.548+0000 7f7128200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f71200adbb0 con 0x7f712015d5d0 2024-05-28T06:05:31.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.548+0000 7f7128200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f71200ad5f0 con 0x7f71200b32c0 2024-05-28T06:05:31.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.548+0000 7f7124e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f712015d5d0 0x7f71200cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:31.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.548+0000 7f7124e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f712015d5d0 0x7f71200cf950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:50808/0 (socket says 172.21.15.96:50808) 2024-05-28T06:05:31.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.548+0000 7f7124e00640 1 -- 172.21.15.96:0/3334852143 learned_addr learned my addr 172.21.15.96:0/3334852143 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:31.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.548+0000 7f7124e00640 1 -- 172.21.15.96:0/3334852143 >> v2:172.21.15.123:3300/0 conn(0x7f71200b32c0 msgr2=0x7f71200b3700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:31.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.548+0000 7f7124e00640 1 --2- 172.21.15.96:0/3334852143 >> v2:172.21.15.123:3300/0 conn(0x7f71200b32c0 0x7f71200b3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:31.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.548+0000 7f7124e00640 1 -- 172.21.15.96:0/3334852143 >> v1:172.21.15.123:6789/0 conn(0x7f71200d0640 legacy=0x7f71200d0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:31.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.548+0000 7f7124e00640 1 -- 172.21.15.96:0/3334852143 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71200ad8d0 con 0x7f712015d5d0 2024-05-28T06:05:31.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.549+0000 7f7124e00640 1 --2- 172.21.15.96:0/3334852143 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f712015d5d0 0x7f71200cf950 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f71200d1200 tx=0x7f710c02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9360031cbb5ade8 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:31.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.550+0000 7f711fe00640 1 -- 172.21.15.96:0/3334852143 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f710c037070 con 0x7f712015d5d0 2024-05-28T06:05:31.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.550+0000 7f711fe00640 1 -- 172.21.15.96:0/3334852143 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f710c032440 con 0x7f712015d5d0 2024-05-28T06:05:31.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.550+0000 7f711fe00640 1 -- 172.21.15.96:0/3334852143 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f710c032810 con 0x7f712015d5d0 2024-05-28T06:05:31.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.550+0000 7f7128200640 1 -- 172.21.15.96:0/3334852143 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f712015d5d0 msgr2=0x7f71200cf950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:31.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.550+0000 7f7128200640 1 --2- 172.21.15.96:0/3334852143 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f712015d5d0 0x7f71200cf950 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f71200d1200 tx=0x7f710c02a1a0 comp rx=0 tx=0).stop 2024-05-28T06:05:31.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.550+0000 7f7128200640 1 -- 172.21.15.96:0/3334852143 shutdown_connections 2024-05-28T06:05:31.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.550+0000 7f7128200640 1 --2- 172.21.15.96:0/3334852143 >> v2:172.21.15.123:3300/0 conn(0x7f71200b32c0 0x7f71200b3700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:31.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.550+0000 7f7128200640 1 --2- 172.21.15.96:0/3334852143 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f712015d5d0 0x7f71200cf950 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:31.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.550+0000 7f7128200640 1 -- 172.21.15.96:0/3334852143 >> 172.21.15.96:0/3334852143 conn(0x7f712006bd20 msgr2=0x7f712006c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:31.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.551+0000 7f7128200640 1 -- 172.21.15.96:0/3334852143 shutdown_connections 2024-05-28T06:05:31.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.551+0000 7f7128200640 1 -- 172.21.15.96:0/3334852143 wait complete. 2024-05-28T06:05:31.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.552+0000 7f7128200640 1 Processor -- start 2024-05-28T06:05:31.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.552+0000 7f7128200640 1 -- start start 2024-05-28T06:05:31.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.553+0000 7f7128200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f71200b32c0 0x7f712006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:31.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.553+0000 7f7128200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f71200d0640 0x7f71200aff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:31.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.553+0000 7f7128200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f712015d5d0 0x7f712014c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:31.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.553+0000 7f7128200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f712015da50 con 0x7f71200d0640 2024-05-28T06:05:31.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.553+0000 7f7128200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f712015d0b0 con 0x7f71200b32c0 2024-05-28T06:05:31.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.553+0000 7f7128200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f71200ad2a0 con 0x7f712015d5d0 2024-05-28T06:05:31.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.553+0000 7f7124e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f71200d0640 0x7f71200aff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:31.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.554+0000 7f7124e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f71200d0640 0x7f71200aff30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:50824/0 (socket says 172.21.15.96:50824) 2024-05-28T06:05:31.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.554+0000 7f7124e00640 1 -- 172.21.15.96:0/1444512353 learned_addr learned my addr 172.21.15.96:0/1444512353 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:31.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.554+0000 7f7124e00640 1 -- 172.21.15.96:0/1444512353 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f712015d5d0 msgr2=0x7f712014c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:31.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.554+0000 7f7124e00640 1 --2- 172.21.15.96:0/1444512353 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f712015d5d0 0x7f712014c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:31.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.554+0000 7f7124e00640 1 -- 172.21.15.96:0/1444512353 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f71200b32c0 msgr2=0x7f712006edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:31.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.554+0000 7f7124e00640 1 --2- 172.21.15.96:0/1444512353 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f71200b32c0 0x7f712006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:31.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.554+0000 7f7124e00640 1 -- 172.21.15.96:0/1444512353 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71200b0b80 con 0x7f71200d0640 2024-05-28T06:05:31.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.554+0000 7f7124e00640 1 --2- 172.21.15.96:0/1444512353 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f71200d0640 0x7f71200aff30 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f710c02a1e0 tx=0x7f710c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:31.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.555+0000 7f711d600640 1 -- 172.21.15.96:0/1444512353 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f710c041070 con 0x7f71200d0640 2024-05-28T06:05:31.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.555+0000 7f711d600640 1 -- 172.21.15.96:0/1444512353 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f710c03c890 con 0x7f71200d0640 2024-05-28T06:05:31.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.555+0000 7f711d600640 1 -- 172.21.15.96:0/1444512353 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f710c037040 con 0x7f71200d0640 2024-05-28T06:05:31.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.555+0000 7f7128200640 1 -- 172.21.15.96:0/1444512353 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71200b0e60 con 0x7f71200d0640 2024-05-28T06:05:31.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.555+0000 7f7128200640 1 -- 172.21.15.96:0/1444512353 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f712014e4d0 con 0x7f71200d0640 2024-05-28T06:05:31.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.556+0000 7f7128200640 1 -- 172.21.15.96:0/1444512353 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71200b08f0 con 0x7f71200d0640 2024-05-28T06:05:31.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.557+0000 7f711d600640 1 -- 172.21.15.96:0/1444512353 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f710c044070 con 0x7f71200d0640 2024-05-28T06:05:31.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.558+0000 7f711d600640 1 --2- 172.21.15.96:0/1444512353 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f710006c8f0 0x7f710006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:31.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.558+0000 7f711d600640 1 -- 172.21.15.96:0/1444512353 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f710c0ac740 con 0x7f71200d0640 2024-05-28T06:05:31.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.559+0000 7f7125800640 1 --2- 172.21.15.96:0/1444512353 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f710006c8f0 0x7f710006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:31.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.560+0000 7f7125800640 1 --2- 172.21.15.96:0/1444512353 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f710006c8f0 0x7f710006edb0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f7114009650 tx=0x7f7114006ba0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:31.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.563+0000 7f711d600640 1 -- 172.21.15.96:0/1444512353 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f710c07aad0 con 0x7f71200d0640 2024-05-28T06:05:31.808 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:31.806+0000 7f7128200640 1 -- 172.21.15.96:0/1444512353 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f71200b4d40 con 0x7f71200d0640 2024-05-28T06:05:32.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:32.482+0000 7f711d600640 1 -- 172.21.15.96:0/1444512353 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v14) ==== 118+0+0 (secure 0 0 0) 0x7f710c07a3d0 con 0x7f71200d0640 2024-05-28T06:05:32.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:32.486+0000 7f7128200640 1 -- 172.21.15.96:0/1444512353 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f710006c8f0 msgr2=0x7f710006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:32.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:32.486+0000 7f7128200640 1 --2- 172.21.15.96:0/1444512353 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f710006c8f0 0x7f710006edb0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f7114009650 tx=0x7f7114006ba0 comp rx=0 tx=0).stop 2024-05-28T06:05:32.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:32.486+0000 7f7128200640 1 -- 172.21.15.96:0/1444512353 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f71200d0640 msgr2=0x7f71200aff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:32.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:32.486+0000 7f7128200640 1 --2- 172.21.15.96:0/1444512353 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f71200d0640 0x7f71200aff30 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f710c02a1e0 tx=0x7f710c0046c0 comp rx=0 tx=0).stop 2024-05-28T06:05:32.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:32.486+0000 7f7128200640 1 -- 172.21.15.96:0/1444512353 shutdown_connections 2024-05-28T06:05:32.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:32.486+0000 7f7128200640 1 --2- 172.21.15.96:0/1444512353 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f710006c8f0 0x7f710006edb0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:32.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:32.486+0000 7f7128200640 1 --2- 172.21.15.96:0/1444512353 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f712015d5d0 0x7f712014c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:32.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:32.486+0000 7f7128200640 1 --2- 172.21.15.96:0/1444512353 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f71200d0640 0x7f71200aff30 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:32.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:32.486+0000 7f7128200640 1 --2- 172.21.15.96:0/1444512353 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f71200b32c0 0x7f712006edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:32.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:32.486+0000 7f7128200640 1 -- 172.21.15.96:0/1444512353 >> 172.21.15.96:0/1444512353 conn(0x7f712006bd20 msgr2=0x7f712015e4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:32.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:32.486+0000 7f7128200640 1 -- 172.21.15.96:0/1444512353 shutdown_connections 2024-05-28T06:05:32.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:32.486+0000 7f7128200640 1 -- 172.21.15.96:0/1444512353 wait complete. 2024-05-28T06:05:32.509 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-05-28T06:05:32.509 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-05-28T06:05:32.718 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:32.716+0000 7f0cce400640 1 Processor -- start 2024-05-28T06:05:32.718 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:32.717+0000 7f0cce400640 1 -- start start 2024-05-28T06:05:32.719 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:32.718+0000 7f0cce400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f0cc815e870 0x7f0cc80d0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:32.719 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:32.718+0000 7f0cce400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0cc80d14d0 0x7f0cc80cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:32.719 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:32.718+0000 7f0cce400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f0cc80cf870 0x7f0cc8162360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:32.719 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:32.718+0000 7f0cce400640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f0cc80d2150 con 0x7f0cc815e870 2024-05-28T06:05:32.719 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:32.718+0000 7f0cce400640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f0cc80cfd00 con 0x7f0cc80cf870 2024-05-28T06:05:32.719 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:32.718+0000 7f0cce400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f0cc815ec50 con 0x7f0cc80d14d0 2024-05-28T06:05:32.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:32.718+0000 7f0ccd000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f0cc815e870 0x7f0cc80d0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:32.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:32.718+0000 7f0ccd000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f0cc815e870 0x7f0cc80d0f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:50828/0 (socket says 172.21.15.96:50828) 2024-05-28T06:05:32.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:32.718+0000 7f0ccd000640 1 -- 172.21.15.96:0/3284618947 learned_addr learned my addr 172.21.15.96:0/3284618947 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:32.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:32.719+0000 7f0cc7e00640 1 --2- 172.21.15.96:0/3284618947 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0cc80d14d0 0x7f0cc80cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:32.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:32.719+0000 7f0ccd000640 1 -- 172.21.15.96:0/3284618947 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0cc80d14d0 msgr2=0x7f0cc80cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:32.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:32.719+0000 7f0ccda00640 1 --2- 172.21.15.96:0/3284618947 >> v2:172.21.15.123:3300/0 conn(0x7f0cc80cf870 0x7f0cc8162360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:32.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:32.719+0000 7f0ccd000640 1 --2- 172.21.15.96:0/3284618947 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0cc80d14d0 0x7f0cc80cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:32.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:32.719+0000 7f0ccd000640 1 -- 172.21.15.96:0/3284618947 >> v2:172.21.15.123:3300/0 conn(0x7f0cc80cf870 msgr2=0x7f0cc8162360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:32.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:32.719+0000 7f0ccd000640 1 --2- 172.21.15.96:0/3284618947 >> v2:172.21.15.123:3300/0 conn(0x7f0cc80cf870 0x7f0cc8162360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:32.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:32.719+0000 7f0ccd000640 1 -- 172.21.15.96:0/3284618947 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cc80cff80 con 0x7f0cc815e870 2024-05-28T06:05:32.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:32.719+0000 7f0cc7e00640 1 --2- 172.21.15.96:0/3284618947 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0cc80d14d0 0x7f0cc80cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:05:32.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:32.719+0000 7f0ccda00640 1 --2- 172.21.15.96:0/3284618947 >> v2:172.21.15.123:3300/0 conn(0x7f0cc80cf870 0x7f0cc8162360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:05:32.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:32.720+0000 7f0ccd000640 1 --2- 172.21.15.96:0/3284618947 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0cc815e870 0x7f0cc80d0f90 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f0cc80b3f60 tx=0x7f0cb802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb31be0bb0a432bd server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:32.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:32.720+0000 7f0cc7400640 1 -- 172.21.15.96:0/3284618947 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cb8037070 con 0x7f0cc815e870 2024-05-28T06:05:32.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:32.720+0000 7f0cc7400640 1 -- 172.21.15.96:0/3284618947 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0cb8032440 con 0x7f0cc815e870 2024-05-28T06:05:32.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:32.720+0000 7f0cc7400640 1 -- 172.21.15.96:0/3284618947 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cb8032810 con 0x7f0cc815e870 2024-05-28T06:05:32.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:32.721+0000 7f0cce400640 1 -- 172.21.15.96:0/3284618947 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0cc815e870 msgr2=0x7f0cc80d0f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:32.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:32.721+0000 7f0cce400640 1 --2- 172.21.15.96:0/3284618947 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0cc815e870 0x7f0cc80d0f90 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f0cc80b3f60 tx=0x7f0cb802a1e0 comp rx=0 tx=0).stop 2024-05-28T06:05:32.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:32.721+0000 7f0cce400640 1 -- 172.21.15.96:0/3284618947 shutdown_connections 2024-05-28T06:05:32.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:32.721+0000 7f0cce400640 1 --2- 172.21.15.96:0/3284618947 >> v2:172.21.15.123:3300/0 conn(0x7f0cc80cf870 0x7f0cc8162360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:32.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:32.721+0000 7f0cce400640 1 --2- 172.21.15.96:0/3284618947 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0cc80d14d0 0x7f0cc80cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:32.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:32.721+0000 7f0cce400640 1 --2- 172.21.15.96:0/3284618947 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0cc815e870 0x7f0cc80d0f90 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:32.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:32.721+0000 7f0cce400640 1 -- 172.21.15.96:0/3284618947 >> 172.21.15.96:0/3284618947 conn(0x7f0cc808bfc0 msgr2=0x7f0cc80d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:32.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:32.721+0000 7f0cce400640 1 -- 172.21.15.96:0/3284618947 shutdown_connections 2024-05-28T06:05:32.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:32.721+0000 7f0cce400640 1 -- 172.21.15.96:0/3284618947 wait complete. 2024-05-28T06:05:32.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:32.723+0000 7f0cce400640 1 Processor -- start 2024-05-28T06:05:32.725 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:32.723+0000 7f0cce400640 1 -- start start 2024-05-28T06:05:32.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:32.724+0000 7f0cce400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0cc80cf870 0x7f0cc814f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:32.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:32.724+0000 7f0cce400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0cc80d14d0 0x7f0cc80b7210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:32.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:32.724+0000 7f0cce400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0cc815e870 0x7f0cc80b6530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:32.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:32.724+0000 7f0cce400640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f0cc80d0820 con 0x7f0cc80cf870 2024-05-28T06:05:32.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:32.724+0000 7f0cce400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f0cc80d0540 con 0x7f0cc815e870 2024-05-28T06:05:32.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:32.724+0000 7f0cce400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f0cc80d4fb0 con 0x7f0cc80d14d0 2024-05-28T06:05:32.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:32.724+0000 7f0ccd000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0cc80cf870 0x7f0cc814f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:32.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:32.724+0000 7f0ccd000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0cc80cf870 0x7f0cc814f810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:50832/0 (socket says 172.21.15.96:50832) 2024-05-28T06:05:32.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:32.724+0000 7f0ccd000640 1 -- 172.21.15.96:0/3154717481 learned_addr learned my addr 172.21.15.96:0/3154717481 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:32.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:32.724+0000 7f0ccd000640 1 -- 172.21.15.96:0/3154717481 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0cc80d14d0 msgr2=0x7f0cc80b7210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:32.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:32.724+0000 7f0ccd000640 1 --2- 172.21.15.96:0/3154717481 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0cc80d14d0 0x7f0cc80b7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:32.727 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:32.724+0000 7f0ccd000640 1 -- 172.21.15.96:0/3154717481 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0cc815e870 msgr2=0x7f0cc80b6530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:32.727 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:32.724+0000 7f0ccd000640 1 --2- 172.21.15.96:0/3154717481 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0cc815e870 0x7f0cc80b6530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:32.727 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:32.724+0000 7f0ccd000640 1 -- 172.21.15.96:0/3154717481 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cc815a690 con 0x7f0cc80cf870 2024-05-28T06:05:32.727 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:32.725+0000 7f0ccd000640 1 --2- 172.21.15.96:0/3154717481 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0cc80cf870 0x7f0cc814f810 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f0cb8000c00 tx=0x7f0cb80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:32.728 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:32.725+0000 7f0cc4c00640 1 -- 172.21.15.96:0/3154717481 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cb8041070 con 0x7f0cc80cf870 2024-05-28T06:05:32.728 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:32.725+0000 7f0cc4c00640 1 -- 172.21.15.96:0/3154717481 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0cb803c870 con 0x7f0cc80cf870 2024-05-28T06:05:32.728 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:32.725+0000 7f0cce400640 1 -- 172.21.15.96:0/3154717481 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0cc815a970 con 0x7f0cc80cf870 2024-05-28T06:05:32.728 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:32.725+0000 7f0cc4c00640 1 -- 172.21.15.96:0/3154717481 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cb8037040 con 0x7f0cc80cf870 2024-05-28T06:05:32.728 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:32.725+0000 7f0cce400640 1 -- 172.21.15.96:0/3154717481 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0cc815af30 con 0x7f0cc80cf870 2024-05-28T06:05:32.728 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:32.726+0000 7f0cce400640 1 -- 172.21.15.96:0/3154717481 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0cc80689e0 con 0x7f0cc80cf870 2024-05-28T06:05:32.729 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:32.727+0000 7f0cc4c00640 1 -- 172.21.15.96:0/3154717481 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0cb8044070 con 0x7f0cc80cf870 2024-05-28T06:05:32.729 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:32.728+0000 7f0cc4c00640 1 --2- 172.21.15.96:0/3154717481 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0ca406c8f0 0x7f0ca406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:32.729 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:32.728+0000 7f0cc4c00640 1 -- 172.21.15.96:0/3154717481 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f0cb80ac7e0 con 0x7f0cc80cf870 2024-05-28T06:05:32.730 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:32.729+0000 7f0cc7e00640 1 --2- 172.21.15.96:0/3154717481 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0ca406c8f0 0x7f0ca406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:32.731 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:32.730+0000 7f0cc7e00640 1 --2- 172.21.15.96:0/3154717481 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0ca406c8f0 0x7f0ca406edb0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f0cc80ba4d0 tx=0x7f0cb0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:32.733 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:32.732+0000 7f0cc4c00640 1 -- 172.21.15.96:0/3154717481 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0cb807ac00 con 0x7f0cc80cf870 2024-05-28T06:05:32.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:32.974+0000 7f0cce400640 1 -- 172.21.15.96:0/3154717481 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f0cc806e800 con 0x7f0cc80cf870 2024-05-28T06:05:33.374 INFO:tasks.ceph.mds.d.smithi123.stderr:2024-05-28T06:05:33.373+0000 7f7fa5e00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-28T06:05:33.496 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.495+0000 7f0cc4c00640 1 -- 172.21.15.96:0/3154717481 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v15) ==== 122+0+0 (secure 0 0 0) 0x7f0cb807a500 con 0x7f0cc80cf870 2024-05-28T06:05:33.502 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.498+0000 7f0cce400640 1 -- 172.21.15.96:0/3154717481 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0ca406c8f0 msgr2=0x7f0ca406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:33.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.498+0000 7f0cce400640 1 --2- 172.21.15.96:0/3154717481 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0ca406c8f0 0x7f0ca406edb0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f0cc80ba4d0 tx=0x7f0cb0008040 comp rx=0 tx=0).stop 2024-05-28T06:05:33.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.498+0000 7f0cce400640 1 -- 172.21.15.96:0/3154717481 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0cc80cf870 msgr2=0x7f0cc814f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:33.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.498+0000 7f0cce400640 1 --2- 172.21.15.96:0/3154717481 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0cc80cf870 0x7f0cc814f810 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f0cb8000c00 tx=0x7f0cb80046c0 comp rx=0 tx=0).stop 2024-05-28T06:05:33.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.499+0000 7f0cce400640 1 -- 172.21.15.96:0/3154717481 shutdown_connections 2024-05-28T06:05:33.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.499+0000 7f0cce400640 1 --2- 172.21.15.96:0/3154717481 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0ca406c8f0 0x7f0ca406edb0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:33.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.499+0000 7f0cce400640 1 --2- 172.21.15.96:0/3154717481 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0cc815e870 0x7f0cc80b6530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:33.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.499+0000 7f0cce400640 1 --2- 172.21.15.96:0/3154717481 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0cc80d14d0 0x7f0cc80b7210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:33.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.499+0000 7f0cce400640 1 --2- 172.21.15.96:0/3154717481 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0cc80cf870 0x7f0cc814f810 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:33.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.499+0000 7f0cce400640 1 -- 172.21.15.96:0/3154717481 >> 172.21.15.96:0/3154717481 conn(0x7f0cc808bfc0 msgr2=0x7f0cc80d2e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:33.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.499+0000 7f0cce400640 1 -- 172.21.15.96:0/3154717481 shutdown_connections 2024-05-28T06:05:33.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.499+0000 7f0cce400640 1 -- 172.21.15.96:0/3154717481 wait complete. 2024-05-28T06:05:33.522 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-05-28T06:05:33.522 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-28T06:05:33.718 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.717+0000 7fd9c5c00640 1 Processor -- start 2024-05-28T06:05:33.719 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.717+0000 7fd9c5c00640 1 -- start start 2024-05-28T06:05:33.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.718+0000 7fd9c5c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd9c006ee40 0x7fd9c00ad230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:33.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.719+0000 7fd9c5c00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fd9c006f2c0 0x7fd9c00ad770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:33.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.719+0000 7fd9c5c00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9c0156f10 con 0x7fd9c0156b30 2024-05-28T06:05:33.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.719+0000 7fd9c5c00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fd9c006dda0 con 0x7fd9c006f2c0 2024-05-28T06:05:33.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.719+0000 7fd9c5c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fd9c006e080 con 0x7fd9c006ee40 2024-05-28T06:05:33.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.719+0000 7fd9bfe00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7fd9c0156b30 0x7fd9c00af6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.96:50084/0 (socket says 172.21.15.96:50084) 2024-05-28T06:05:33.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.719+0000 7fd9bfe00640 1 -- 172.21.15.96:0/3179702130 learned_addr learned my addr 172.21.15.96:0/3179702130 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:05:33.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.720+0000 7fd9b7e00640 1 --2- 172.21.15.96:0/3179702130 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd9c006ee40 0x7fd9c00ad230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:33.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.720+0000 7fd9c5200640 1 --2- 172.21.15.96:0/3179702130 >> v2:172.21.15.123:3300/0 conn(0x7fd9c006f2c0 0x7fd9c00ad770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:33.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.720+0000 7fd9bf400640 1 -- 172.21.15.96:0/3179702130 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2371503175 0 0) 0x7fd9c0156f10 con 0x7fd9c0156b30 2024-05-28T06:05:33.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.720+0000 7fd9bf400640 1 -- 172.21.15.96:0/3179702130 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd9a4003520 con 0x7fd9c0156b30 2024-05-28T06:05:33.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.720+0000 7fd9b7e00640 1 -- 172.21.15.96:0/3179702130 >> v2:172.21.15.123:3300/0 conn(0x7fd9c006f2c0 msgr2=0x7fd9c00ad770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:33.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.720+0000 7fd9b7e00640 1 --2- 172.21.15.96:0/3179702130 >> v2:172.21.15.123:3300/0 conn(0x7fd9c006f2c0 0x7fd9c00ad770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:33.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.720+0000 7fd9b7e00640 1 -- 172.21.15.96:0/3179702130 >> v1:172.21.15.123:6789/0 conn(0x7fd9c0156b30 legacy=0x7fd9c00af6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:33.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.720+0000 7fd9b7e00640 1 -- 172.21.15.96:0/3179702130 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9c006e360 con 0x7fd9c006ee40 2024-05-28T06:05:33.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.722+0000 7fd9b7e00640 1 --2- 172.21.15.96:0/3179702130 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd9c006ee40 0x7fd9c00ad230 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd9a80097a0 tx=0x7fd9a802a100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f0cca236522e9da server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:33.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.722+0000 7fd9bf400640 1 -- 172.21.15.96:0/3179702130 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9a8037070 con 0x7fd9c006ee40 2024-05-28T06:05:33.725 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.722+0000 7fd9bf400640 1 -- 172.21.15.96:0/3179702130 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9a802ae50 con 0x7fd9c006ee40 2024-05-28T06:05:33.725 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.723+0000 7fd9bf400640 1 -- 172.21.15.96:0/3179702130 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9a8032660 con 0x7fd9c006ee40 2024-05-28T06:05:33.725 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.723+0000 7fd9c5c00640 1 -- 172.21.15.96:0/3179702130 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd9c006ee40 msgr2=0x7fd9c00ad230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:33.725 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.723+0000 7fd9c5c00640 1 --2- 172.21.15.96:0/3179702130 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd9c006ee40 0x7fd9c00ad230 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd9a80097a0 tx=0x7fd9a802a100 comp rx=0 tx=0).stop 2024-05-28T06:05:33.725 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.723+0000 7fd9c5c00640 1 -- 172.21.15.96:0/3179702130 shutdown_connections 2024-05-28T06:05:33.725 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.723+0000 7fd9c5c00640 1 --2- 172.21.15.96:0/3179702130 >> v2:172.21.15.123:3300/0 conn(0x7fd9c006f2c0 0x7fd9c00ad770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:33.725 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.723+0000 7fd9c5c00640 1 --2- 172.21.15.96:0/3179702130 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd9c006ee40 0x7fd9c00ad230 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:33.725 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.723+0000 7fd9c5c00640 1 -- 172.21.15.96:0/3179702130 >> 172.21.15.96:0/3179702130 conn(0x7fd9c00b1fc0 msgr2=0x7fd9c00b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:33.725 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.723+0000 7fd9c5c00640 1 -- 172.21.15.96:0/3179702130 shutdown_connections 2024-05-28T06:05:33.725 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.723+0000 7fd9c5c00640 1 -- 172.21.15.96:0/3179702130 wait complete. 2024-05-28T06:05:33.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.725+0000 7fd9c5c00640 1 Processor -- start 2024-05-28T06:05:33.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.725+0000 7fd9c5c00640 1 -- start start 2024-05-28T06:05:33.728 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.726+0000 7fd9c5c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9c006ee40 0x7fd9c01f3230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:33.728 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.726+0000 7fd9c5c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd9c006f2c0 0x7fd9c015f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:33.728 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.726+0000 7fd9c5c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9c0156b30 0x7fd9c00b4480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:33.728 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.726+0000 7fd9c5c00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd9c00b2db0 con 0x7fd9c0156b30 2024-05-28T06:05:33.729 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.726+0000 7fd9c5c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd9c00b2c30 con 0x7fd9c006ee40 2024-05-28T06:05:33.729 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.726+0000 7fd9c5c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fd9c00b32c0 con 0x7fd9c006f2c0 2024-05-28T06:05:33.729 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.726+0000 7fd9c5200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9c0156b30 0x7fd9c00b4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:33.729 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.726+0000 7fd9c5200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9c0156b30 0x7fd9c00b4480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:50844/0 (socket says 172.21.15.96:50844) 2024-05-28T06:05:33.729 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.726+0000 7fd9c5200640 1 -- 172.21.15.96:0/238853624 learned_addr learned my addr 172.21.15.96:0/238853624 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:33.729 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.727+0000 7fd9b7e00640 1 --2- 172.21.15.96:0/238853624 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd9c006f2c0 0x7fd9c015f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:33.729 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.727+0000 7fd9bfe00640 1 --2- 172.21.15.96:0/238853624 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9c006ee40 0x7fd9c01f3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:33.729 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.727+0000 7fd9c5200640 1 -- 172.21.15.96:0/238853624 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd9c006f2c0 msgr2=0x7fd9c015f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:33.729 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.727+0000 7fd9c5200640 1 --2- 172.21.15.96:0/238853624 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd9c006f2c0 0x7fd9c015f4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:33.729 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.727+0000 7fd9c5200640 1 -- 172.21.15.96:0/238853624 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9c006ee40 msgr2=0x7fd9c01f3230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:33.729 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.727+0000 7fd9c5200640 1 --2- 172.21.15.96:0/238853624 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9c006ee40 0x7fd9c01f3230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:33.729 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.727+0000 7fd9c5200640 1 -- 172.21.15.96:0/238853624 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9c01f3770 con 0x7fd9c0156b30 2024-05-28T06:05:33.730 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.727+0000 7fd9c5200640 1 --2- 172.21.15.96:0/238853624 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9c0156b30 0x7fd9c00b4480 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fd9b8009d20 tx=0x7fd9b800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:33.731 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.728+0000 7fd9bcc00640 1 -- 172.21.15.96:0/238853624 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9b8018020 con 0x7fd9c0156b30 2024-05-28T06:05:33.731 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.728+0000 7fd9bcc00640 1 -- 172.21.15.96:0/238853624 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9b8009b10 con 0x7fd9c0156b30 2024-05-28T06:05:33.731 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.728+0000 7fd9bcc00640 1 -- 172.21.15.96:0/238853624 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9b80146c0 con 0x7fd9c0156b30 2024-05-28T06:05:33.731 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.728+0000 7fd9c5c00640 1 -- 172.21.15.96:0/238853624 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9c015c9a0 con 0x7fd9c0156b30 2024-05-28T06:05:33.731 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.728+0000 7fd9c5c00640 1 -- 172.21.15.96:0/238853624 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9c015d240 con 0x7fd9c0156b30 2024-05-28T06:05:33.731 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.729+0000 7fd9c5c00640 1 -- 172.21.15.96:0/238853624 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9c00689e0 con 0x7fd9c0156b30 2024-05-28T06:05:33.732 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.730+0000 7fd9bcc00640 1 -- 172.21.15.96:0/238853624 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd9b8005d10 con 0x7fd9c0156b30 2024-05-28T06:05:33.733 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.731+0000 7fd9bcc00640 1 --2- 172.21.15.96:0/238853624 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd99c06c8f0 0x7fd99c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:33.733 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.731+0000 7fd9bcc00640 1 -- 172.21.15.96:0/238853624 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7fd9b808c6b0 con 0x7fd9c0156b30 2024-05-28T06:05:33.733 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.732+0000 7fd9bfe00640 1 --2- 172.21.15.96:0/238853624 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd99c06c8f0 0x7fd99c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:33.734 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.733+0000 7fd9bfe00640 1 --2- 172.21.15.96:0/238853624 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd99c06c8f0 0x7fd99c06edb0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fd9b00011e0 tx=0x7fd9b0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:33.737 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.736+0000 7fd9bcc00640 1 -- 172.21.15.96:0/238853624 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd9b8022070 con 0x7fd9c0156b30 2024-05-28T06:05:33.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.991+0000 7fd9c5c00640 1 -- 172.21.15.96:0/238853624 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd9c006f7f0 con 0x7fd9c0156b30 2024-05-28T06:05:33.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.993+0000 7fd9bcc00640 1 -- 172.21.15.96:0/238853624 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) ==== 94+0+5178 (secure 0 0 0) 0x7fd9b8010040 con 0x7fd9c0156b30 2024-05-28T06:05:33.994 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:05:33.994 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4580,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6836/918905879","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":918905879},{"type":"v1","addr":"172.21.15.123:6836","nonce":918905879}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":5223,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6834/697317719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6833","nonce":697317719},{"type":"v1","addr":"172.21.15.96:6834","nonce":697317719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":5226,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/1327552826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":1327552826},{"type":"v1","addr":"172.21.15.96:6835","nonce":1327552826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-28T06:05:29.892912+0000","modified":"2024-05-28T06:05:32.976408+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5238":{"gid":5238,"name":"d","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6837/458334664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6835","nonce":458334664},{"type":"v1","addr":"172.21.15.123:6837","nonce":458334664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5238,"qdb_cluster":[5238]},"id":2}]} 2024-05-28T06:05:33.995 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 15 2024-05-28T06:05:33.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.995+0000 7fd9c5c00640 1 -- 172.21.15.96:0/238853624 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd99c06c8f0 msgr2=0x7fd99c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:33.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.995+0000 7fd9c5c00640 1 --2- 172.21.15.96:0/238853624 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd99c06c8f0 0x7fd99c06edb0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fd9b00011e0 tx=0x7fd9b0009040 comp rx=0 tx=0).stop 2024-05-28T06:05:33.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.995+0000 7fd9c5c00640 1 -- 172.21.15.96:0/238853624 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9c0156b30 msgr2=0x7fd9c00b4480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:33.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.995+0000 7fd9c5c00640 1 --2- 172.21.15.96:0/238853624 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9c0156b30 0x7fd9c00b4480 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fd9b8009d20 tx=0x7fd9b800c490 comp rx=0 tx=0).stop 2024-05-28T06:05:33.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.995+0000 7fd9c5c00640 1 -- 172.21.15.96:0/238853624 shutdown_connections 2024-05-28T06:05:33.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.995+0000 7fd9c5c00640 1 --2- 172.21.15.96:0/238853624 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd99c06c8f0 0x7fd99c06edb0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:33.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.995+0000 7fd9c5c00640 1 --2- 172.21.15.96:0/238853624 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9c0156b30 0x7fd9c00b4480 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:33.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.995+0000 7fd9c5c00640 1 --2- 172.21.15.96:0/238853624 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd9c006f2c0 0x7fd9c015f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:33.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.995+0000 7fd9c5c00640 1 --2- 172.21.15.96:0/238853624 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9c006ee40 0x7fd9c01f3230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:33.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.995+0000 7fd9c5c00640 1 -- 172.21.15.96:0/238853624 >> 172.21.15.96:0/238853624 conn(0x7fd9c00b1fc0 msgr2=0x7fd9c015a6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:33.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.995+0000 7fd9c5c00640 1 -- 172.21.15.96:0/238853624 shutdown_connections 2024-05-28T06:05:33.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:33.995+0000 7fd9c5c00640 1 -- 172.21.15.96:0/238853624 wait complete. 2024-05-28T06:05:34.014 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:05:34.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.203+0000 7f1b43e00640 1 Processor -- start 2024-05-28T06:05:34.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.204+0000 7f1b43e00640 1 -- start start 2024-05-28T06:05:34.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.205+0000 7f1b43e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f1b44158b80 0x7f1b4406dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:34.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.205+0000 7f1b43e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f1b440722d0 0x7f1b440b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:34.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.205+0000 7f1b43e00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b4406e6e0 con 0x7f1b4406e160 2024-05-28T06:05:34.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.205+0000 7f1b43e00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f1b4406e9c0 con 0x7f1b44158b80 2024-05-28T06:05:34.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.205+0000 7f1b43e00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f1b4406eca0 con 0x7f1b440722d0 2024-05-28T06:05:34.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.205+0000 7f1b42a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f1b44158b80 0x7f1b4406dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:34.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.205+0000 7f1b3a000640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f1b4406e160 0x7f1b44071bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:40278/0 (socket says 172.21.15.96:40278) 2024-05-28T06:05:34.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.205+0000 7f1b3a000640 1 -- 172.21.15.96:0/763764707 learned_addr learned my addr 172.21.15.96:0/763764707 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:05:34.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.206+0000 7f1b43400640 1 --2- 172.21.15.96:0/763764707 >> v2:172.21.15.123:3300/0 conn(0x7f1b440722d0 0x7f1b440b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:34.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.206+0000 7f1b42000640 1 -- 172.21.15.96:0/763764707 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1622090685 0 0) 0x7f1b4406e6e0 con 0x7f1b4406e160 2024-05-28T06:05:34.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.206+0000 7f1b42000640 1 -- 172.21.15.96:0/763764707 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1b28003520 con 0x7f1b4406e160 2024-05-28T06:05:34.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.206+0000 7f1b42a00640 1 -- 172.21.15.96:0/763764707 >> v2:172.21.15.123:3300/0 conn(0x7f1b440722d0 msgr2=0x7f1b440b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:34.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.206+0000 7f1b42a00640 1 --2- 172.21.15.96:0/763764707 >> v2:172.21.15.123:3300/0 conn(0x7f1b440722d0 0x7f1b440b32c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:34.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.206+0000 7f1b42a00640 1 -- 172.21.15.96:0/763764707 >> v1:172.21.15.96:6789/0 conn(0x7f1b4406e160 legacy=0x7f1b44071bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:34.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.206+0000 7f1b42a00640 1 -- 172.21.15.96:0/763764707 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b4406ef80 con 0x7f1b44158b80 2024-05-28T06:05:34.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.206+0000 7f1b43400640 1 --2- 172.21.15.96:0/763764707 >> v2:172.21.15.123:3300/0 conn(0x7f1b440722d0 0x7f1b440b32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:05:34.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.207+0000 7f1b42a00640 1 --2- 172.21.15.96:0/763764707 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1b44158b80 0x7f1b4406dc20 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f1b4406c890 tx=0x7f1b3402a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e402b408caec1ff9 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:34.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.208+0000 7f1b42000640 1 -- 172.21.15.96:0/763764707 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b34007740 con 0x7f1b44158b80 2024-05-28T06:05:34.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.208+0000 7f1b42000640 1 -- 172.21.15.96:0/763764707 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b340078e0 con 0x7f1b44158b80 2024-05-28T06:05:34.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.208+0000 7f1b42000640 1 -- 172.21.15.96:0/763764707 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b34007cb0 con 0x7f1b44158b80 2024-05-28T06:05:34.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.208+0000 7f1b43e00640 1 -- 172.21.15.96:0/763764707 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1b44158b80 msgr2=0x7f1b4406dc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:34.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.208+0000 7f1b43e00640 1 --2- 172.21.15.96:0/763764707 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1b44158b80 0x7f1b4406dc20 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f1b4406c890 tx=0x7f1b3402a680 comp rx=0 tx=0).stop 2024-05-28T06:05:34.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.208+0000 7f1b43e00640 1 -- 172.21.15.96:0/763764707 shutdown_connections 2024-05-28T06:05:34.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.209+0000 7f1b43e00640 1 --2- 172.21.15.96:0/763764707 >> v2:172.21.15.123:3300/0 conn(0x7f1b440722d0 0x7f1b440b32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:34.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.209+0000 7f1b43e00640 1 --2- 172.21.15.96:0/763764707 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1b44158b80 0x7f1b4406dc20 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:34.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.209+0000 7f1b43e00640 1 -- 172.21.15.96:0/763764707 >> 172.21.15.96:0/763764707 conn(0x7f1b440b1fc0 msgr2=0x7f1b440b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:34.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.209+0000 7f1b43e00640 1 -- 172.21.15.96:0/763764707 shutdown_connections 2024-05-28T06:05:34.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.209+0000 7f1b43e00640 1 -- 172.21.15.96:0/763764707 wait complete. 2024-05-28T06:05:34.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.210+0000 7f1b43e00640 1 Processor -- start 2024-05-28T06:05:34.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.210+0000 7f1b43e00640 1 -- start start 2024-05-28T06:05:34.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.211+0000 7f1b43e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1b4406e160 0x7f1b44157610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:34.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.211+0000 7f1b43e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1b440722d0 0x7f1b441507d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:34.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.211+0000 7f1b43e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b44158b80 0x7f1b440afd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:34.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.211+0000 7f1b43e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f1b4406f780 con 0x7f1b4406e160 2024-05-28T06:05:34.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.211+0000 7f1b43e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f1b440b2d30 con 0x7f1b44158b80 2024-05-28T06:05:34.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.211+0000 7f1b43e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f1b4409c2e0 con 0x7f1b440722d0 2024-05-28T06:05:34.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.212+0000 7f1b42a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1b4406e160 0x7f1b44157610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:34.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.212+0000 7f1b42a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1b4406e160 0x7f1b44157610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:50862/0 (socket says 172.21.15.96:50862) 2024-05-28T06:05:34.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.212+0000 7f1b42a00640 1 -- 172.21.15.96:0/4204508697 learned_addr learned my addr 172.21.15.96:0/4204508697 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:34.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.212+0000 7f1b3a000640 1 --2- 172.21.15.96:0/4204508697 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1b440722d0 0x7f1b441507d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:34.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.212+0000 7f1b43400640 1 --2- 172.21.15.96:0/4204508697 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b44158b80 0x7f1b440afd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:34.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.212+0000 7f1b42a00640 1 -- 172.21.15.96:0/4204508697 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1b440722d0 msgr2=0x7f1b441507d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:34.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.212+0000 7f1b42a00640 1 --2- 172.21.15.96:0/4204508697 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1b440722d0 0x7f1b441507d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:34.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.212+0000 7f1b42a00640 1 -- 172.21.15.96:0/4204508697 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b44158b80 msgr2=0x7f1b440afd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:34.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.212+0000 7f1b42a00640 1 --2- 172.21.15.96:0/4204508697 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b44158b80 0x7f1b440afd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:34.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.212+0000 7f1b42a00640 1 -- 172.21.15.96:0/4204508697 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b440ad230 con 0x7f1b4406e160 2024-05-28T06:05:34.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.213+0000 7f1b42a00640 1 --2- 172.21.15.96:0/4204508697 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1b4406e160 0x7f1b44157610 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f1b34009e80 tx=0x7f1b34004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:34.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.213+0000 7f1b3b400640 1 -- 172.21.15.96:0/4204508697 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b3403f070 con 0x7f1b4406e160 2024-05-28T06:05:34.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.213+0000 7f1b3b400640 1 -- 172.21.15.96:0/4204508697 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b34031c40 con 0x7f1b4406e160 2024-05-28T06:05:34.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.213+0000 7f1b3b400640 1 -- 172.21.15.96:0/4204508697 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b34009450 con 0x7f1b4406e160 2024-05-28T06:05:34.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.213+0000 7f1b43e00640 1 -- 172.21.15.96:0/4204508697 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b4409b6d0 con 0x7f1b4406e160 2024-05-28T06:05:34.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.213+0000 7f1b43e00640 1 -- 172.21.15.96:0/4204508697 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b4409bc90 con 0x7f1b4406e160 2024-05-28T06:05:34.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.214+0000 7f1b43e00640 1 -- 172.21.15.96:0/4204508697 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b440689e0 con 0x7f1b4406e160 2024-05-28T06:05:34.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.215+0000 7f1b3b400640 1 -- 172.21.15.96:0/4204508697 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1b34043410 con 0x7f1b4406e160 2024-05-28T06:05:34.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.216+0000 7f1b3b400640 1 --2- 172.21.15.96:0/4204508697 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1b2006c8f0 0x7f1b2006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:34.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.216+0000 7f1b3b400640 1 -- 172.21.15.96:0/4204508697 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f1b340acfe0 con 0x7f1b4406e160 2024-05-28T06:05:34.218 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.217+0000 7f1b3a000640 1 --2- 172.21.15.96:0/4204508697 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1b2006c8f0 0x7f1b2006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:34.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.218+0000 7f1b3a000640 1 --2- 172.21.15.96:0/4204508697 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1b2006c8f0 0x7f1b2006edb0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f1b4409a460 tx=0x7f1b2c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:34.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.220+0000 7f1b3b400640 1 -- 172.21.15.96:0/4204508697 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1b3407b3f0 con 0x7f1b4406e160 2024-05-28T06:05:34.419 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.417+0000 7f1b43e00640 1 -- 172.21.15.96:0/4204508697 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1b440d10f0 con 0x7f1b4406e160 2024-05-28T06:05:34.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.421+0000 7f1b3b400640 1 -- 172.21.15.96:0/4204508697 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) ==== 74+0+16681 (secure 0 0 0) 0x7f1b3407acf0 con 0x7f1b4406e160 2024-05-28T06:05:34.422 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:05:34.423 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":33,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:05:31.459066+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-05-28T06:05:27.674327+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-05-28T06:05:28.878204+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:05:34.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.423+0000 7f1b43e00640 1 -- 172.21.15.96:0/4204508697 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1b2006c8f0 msgr2=0x7f1b2006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:34.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.423+0000 7f1b43e00640 1 --2- 172.21.15.96:0/4204508697 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1b2006c8f0 0x7f1b2006edb0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f1b4409a460 tx=0x7f1b2c009040 comp rx=0 tx=0).stop 2024-05-28T06:05:34.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.423+0000 7f1b43e00640 1 -- 172.21.15.96:0/4204508697 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1b4406e160 msgr2=0x7f1b44157610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:34.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.423+0000 7f1b43e00640 1 --2- 172.21.15.96:0/4204508697 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1b4406e160 0x7f1b44157610 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f1b34009e80 tx=0x7f1b34004a40 comp rx=0 tx=0).stop 2024-05-28T06:05:34.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.423+0000 7f1b43e00640 1 -- 172.21.15.96:0/4204508697 shutdown_connections 2024-05-28T06:05:34.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.423+0000 7f1b43e00640 1 --2- 172.21.15.96:0/4204508697 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1b2006c8f0 0x7f1b2006edb0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:34.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.423+0000 7f1b43e00640 1 --2- 172.21.15.96:0/4204508697 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b44158b80 0x7f1b440afd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:34.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.423+0000 7f1b43e00640 1 --2- 172.21.15.96:0/4204508697 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1b440722d0 0x7f1b441507d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:34.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.424+0000 7f1b43e00640 1 --2- 172.21.15.96:0/4204508697 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1b4406e160 0x7f1b44157610 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:34.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.424+0000 7f1b43e00640 1 -- 172.21.15.96:0/4204508697 >> 172.21.15.96:0/4204508697 conn(0x7f1b440b1fc0 msgr2=0x7f1b4415f3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:34.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.424+0000 7f1b43e00640 1 -- 172.21.15.96:0/4204508697 shutdown_connections 2024-05-28T06:05:34.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.424+0000 7f1b43e00640 1 -- 172.21.15.96:0/4204508697 wait complete. 2024-05-28T06:05:34.444 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-05-28T06:05:34.444 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-28T06:05:34.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.635+0000 7ffa7ca00640 1 Processor -- start 2024-05-28T06:05:34.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.635+0000 7ffa7ca00640 1 -- start start 2024-05-28T06:05:34.638 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.637+0000 7ffa7ca00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7ffa780d16a0 0x7ffa780cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:34.638 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.637+0000 7ffa7ca00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa780cfce0 con 0x7ffa780cf870 2024-05-28T06:05:34.638 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.637+0000 7ffa7ca00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa7815ec50 con 0x7ffa7815e870 2024-05-28T06:05:34.639 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.637+0000 7ffa7ca00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7ffa780d2150 con 0x7ffa780d16a0 2024-05-28T06:05:34.639 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.637+0000 7ffa76a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7ffa780d16a0 0x7ffa780cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:34.639 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.637+0000 7ffa77e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7ffa780cf870 0x7ffa780d25e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:40282/0 (socket says 172.21.15.96:40282) 2024-05-28T06:05:34.639 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.637+0000 7ffa77e00640 1 -- 172.21.15.96:0/338816278 learned_addr learned my addr 172.21.15.96:0/338816278 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:05:34.639 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.638+0000 7ffa76a00640 1 -- 172.21.15.96:0/338816278 >> v1:172.21.15.123:6789/0 conn(0x7ffa7815e870 legacy=0x7ffa780d0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:34.640 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.638+0000 7ffa76a00640 1 -- 172.21.15.96:0/338816278 >> v1:172.21.15.96:6789/0 conn(0x7ffa780cf870 legacy=0x7ffa780d25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:34.640 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.638+0000 7ffa76000640 1 -- 172.21.15.96:0/338816278 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3247129527 0 0) 0x7ffa780cfce0 con 0x7ffa780cf870 2024-05-28T06:05:34.640 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.638+0000 7ffa76a00640 1 -- 172.21.15.96:0/338816278 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa780cff80 con 0x7ffa780d16a0 2024-05-28T06:05:34.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.639+0000 7ffa76a00640 1 --2- 172.21.15.96:0/338816278 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa780d16a0 0x7ffa780cf330 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7ffa60009de0 tx=0x7ffa6002a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f260272243d5c23a server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:34.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.639+0000 7ffa76000640 1 -- 172.21.15.96:0/338816278 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa60007740 con 0x7ffa780d16a0 2024-05-28T06:05:34.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.639+0000 7ffa76000640 1 -- 172.21.15.96:0/338816278 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa600078e0 con 0x7ffa780d16a0 2024-05-28T06:05:34.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.639+0000 7ffa76000640 1 -- 172.21.15.96:0/338816278 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa60007cb0 con 0x7ffa780d16a0 2024-05-28T06:05:34.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.639+0000 7ffa7ca00640 1 -- 172.21.15.96:0/338816278 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa780d16a0 msgr2=0x7ffa780cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:34.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.639+0000 7ffa7ca00640 1 --2- 172.21.15.96:0/338816278 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa780d16a0 0x7ffa780cf330 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7ffa60009de0 tx=0x7ffa6002a710 comp rx=0 tx=0).stop 2024-05-28T06:05:34.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.640+0000 7ffa7ca00640 1 -- 172.21.15.96:0/338816278 shutdown_connections 2024-05-28T06:05:34.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.640+0000 7ffa7ca00640 1 --2- 172.21.15.96:0/338816278 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa780d16a0 0x7ffa780cf330 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:34.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.640+0000 7ffa7ca00640 1 -- 172.21.15.96:0/338816278 >> 172.21.15.96:0/338816278 conn(0x7ffa7808bfc0 msgr2=0x7ffa780d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:34.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.640+0000 7ffa7ca00640 1 -- 172.21.15.96:0/338816278 shutdown_connections 2024-05-28T06:05:34.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.640+0000 7ffa7ca00640 1 -- 172.21.15.96:0/338816278 wait complete. 2024-05-28T06:05:34.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.641+0000 7ffa7ca00640 1 Processor -- start 2024-05-28T06:05:34.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.641+0000 7ffa7ca00640 1 -- start start 2024-05-28T06:05:34.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.642+0000 7ffa7ca00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffa780cf870 0x7ffa78159b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:34.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.642+0000 7ffa7ca00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa780d16a0 0x7ffa7815f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:34.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.642+0000 7ffa7ca00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa7815e870 0x7ffa78160e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:34.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.642+0000 7ffa7ca00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ffa780d0820 con 0x7ffa7815e870 2024-05-28T06:05:34.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.642+0000 7ffa7ca00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ffa780d0540 con 0x7ffa780d16a0 2024-05-28T06:05:34.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.642+0000 7ffa7ca00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ffa780d4fb0 con 0x7ffa780cf870 2024-05-28T06:05:34.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.642+0000 7ffa77e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa7815e870 0x7ffa78160e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:34.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.643+0000 7ffa77e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa7815e870 0x7ffa78160e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:50876/0 (socket says 172.21.15.96:50876) 2024-05-28T06:05:34.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.643+0000 7ffa77e00640 1 -- 172.21.15.96:0/268618725 learned_addr learned my addr 172.21.15.96:0/268618725 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:34.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.643+0000 7ffa76a00640 1 --2- 172.21.15.96:0/268618725 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa780d16a0 0x7ffa7815f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:34.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.643+0000 7ffa77400640 1 --2- 172.21.15.96:0/268618725 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffa780cf870 0x7ffa78159b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:34.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.643+0000 7ffa77e00640 1 -- 172.21.15.96:0/268618725 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffa780cf870 msgr2=0x7ffa78159b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:34.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.643+0000 7ffa77e00640 1 --2- 172.21.15.96:0/268618725 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffa780cf870 0x7ffa78159b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:34.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.643+0000 7ffa77e00640 1 -- 172.21.15.96:0/268618725 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa780d16a0 msgr2=0x7ffa7815f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:34.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.643+0000 7ffa77e00640 1 --2- 172.21.15.96:0/268618725 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa780d16a0 0x7ffa7815f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:34.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.643+0000 7ffa77e00640 1 -- 172.21.15.96:0/268618725 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa7815fc40 con 0x7ffa7815e870 2024-05-28T06:05:34.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.643+0000 7ffa76a00640 1 --2- 172.21.15.96:0/268618725 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa780d16a0 0x7ffa7815f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:05:34.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.643+0000 7ffa77400640 1 --2- 172.21.15.96:0/268618725 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffa780cf870 0x7ffa78159b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:05:34.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.643+0000 7ffa77e00640 1 --2- 172.21.15.96:0/268618725 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa7815e870 0x7ffa78160e70 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7ffa7000c480 tx=0x7ffa7000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:34.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.643+0000 7ffa6f400640 1 -- 172.21.15.96:0/268618725 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa70018070 con 0x7ffa7815e870 2024-05-28T06:05:34.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.644+0000 7ffa6f400640 1 -- 172.21.15.96:0/268618725 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa70007500 con 0x7ffa7815e870 2024-05-28T06:05:34.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.644+0000 7ffa6f400640 1 -- 172.21.15.96:0/268618725 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa70004a20 con 0x7ffa7815e870 2024-05-28T06:05:34.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.644+0000 7ffa7ca00640 1 -- 172.21.15.96:0/268618725 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa78161c60 con 0x7ffa7815e870 2024-05-28T06:05:34.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.644+0000 7ffa7ca00640 1 -- 172.21.15.96:0/268618725 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa78161980 con 0x7ffa7815e870 2024-05-28T06:05:34.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.645+0000 7ffa7ca00640 1 -- 172.21.15.96:0/268618725 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa7815f3e0 con 0x7ffa7815e870 2024-05-28T06:05:34.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.646+0000 7ffa6f400640 1 -- 172.21.15.96:0/268618725 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ffa7001b070 con 0x7ffa7815e870 2024-05-28T06:05:34.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.647+0000 7ffa6f400640 1 --2- 172.21.15.96:0/268618725 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ffa5406c8f0 0x7ffa5406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:34.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.647+0000 7ffa6f400640 1 -- 172.21.15.96:0/268618725 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7ffa7008aee0 con 0x7ffa7815e870 2024-05-28T06:05:34.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.647+0000 7ffa77400640 1 --2- 172.21.15.96:0/268618725 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ffa5406c8f0 0x7ffa5406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:34.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.648+0000 7ffa77400640 1 --2- 172.21.15.96:0/268618725 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ffa5406c8f0 0x7ffa5406edb0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ffa6800a830 tx=0x7ffa68008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:34.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.651+0000 7ffa6f400640 1 -- 172.21.15.96:0/268618725 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ffa7815f3e0 con 0x7ffa7815e870 2024-05-28T06:05:34.847 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.845+0000 7ffa7ca00640 1 -- 172.21.15.96:0/268618725 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ffa780689e0 con 0x7ffa5406c8f0 2024-05-28T06:05:34.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.868+0000 7ffa6f400640 1 -- 172.21.15.96:0/268618725 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+246631 (secure 0 0 0) 0x7ffa780689e0 con 0x7ffa5406c8f0 2024-05-28T06:05:34.873 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:05:34.874 INFO:teuthology.orchestra.run.smithi096.stderr:dumped all 2024-05-28T06:05:34.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.873+0000 7ffa7ca00640 1 -- 172.21.15.96:0/268618725 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ffa5406c8f0 msgr2=0x7ffa5406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:34.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.873+0000 7ffa7ca00640 1 --2- 172.21.15.96:0/268618725 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ffa5406c8f0 0x7ffa5406edb0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ffa6800a830 tx=0x7ffa68008040 comp rx=0 tx=0).stop 2024-05-28T06:05:34.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.873+0000 7ffa7ca00640 1 -- 172.21.15.96:0/268618725 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa7815e870 msgr2=0x7ffa78160e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:34.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.873+0000 7ffa7ca00640 1 --2- 172.21.15.96:0/268618725 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa7815e870 0x7ffa78160e70 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7ffa7000c480 tx=0x7ffa7000c940 comp rx=0 tx=0).stop 2024-05-28T06:05:34.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.873+0000 7ffa7ca00640 1 -- 172.21.15.96:0/268618725 shutdown_connections 2024-05-28T06:05:34.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.873+0000 7ffa7ca00640 1 --2- 172.21.15.96:0/268618725 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ffa5406c8f0 0x7ffa5406edb0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:34.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.873+0000 7ffa7ca00640 1 --2- 172.21.15.96:0/268618725 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa7815e870 0x7ffa78160e70 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:34.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.873+0000 7ffa7ca00640 1 --2- 172.21.15.96:0/268618725 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa780d16a0 0x7ffa7815f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:34.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.873+0000 7ffa7ca00640 1 --2- 172.21.15.96:0/268618725 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffa780cf870 0x7ffa78159b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:34.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.873+0000 7ffa7ca00640 1 -- 172.21.15.96:0/268618725 >> 172.21.15.96:0/268618725 conn(0x7ffa7808bfc0 msgr2=0x7ffa780b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:34.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.873+0000 7ffa7ca00640 1 -- 172.21.15.96:0/268618725 shutdown_connections 2024-05-28T06:05:34.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:34.873+0000 7ffa7ca00640 1 -- 172.21.15.96:0/268618725 wait complete. 2024-05-28T06:05:34.894 INFO:teuthology.orchestra.run.smithi096.stdout:{"pg_ready":true,"pg_map":{"version":73,"stamp":"2024-05-28T06:05:33.510905+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":4,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219960,"kb_used_data":3796,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754760,"statfs":{"total":773094113280,"available":772868874240,"internally_reserved":0,"allocated":3887104,"data_stored":2205195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.090214"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.445439+0000","last_change":"2024-05-28T06:05:29.445439+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.445439+0000","last_undegraded":"2024-05-28T06:05:29.445439+0000","last_fullsized":"2024-05-28T06:05:29.445439+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.452319+0000","last_change":"2024-05-28T06:05:29.452319+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.452319+0000","last_undegraded":"2024-05-28T06:05:29.452319+0000","last_fullsized":"2024-05-28T06:05:29.452319+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.455274+0000","last_change":"2024-05-28T06:05:29.455274+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.455274+0000","last_undegraded":"2024-05-28T06:05:29.455274+0000","last_fullsized":"2024-05-28T06:05:29.455274+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.445974+0000","last_change":"2024-05-28T06:05:29.445974+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.445974+0000","last_undegraded":"2024-05-28T06:05:29.445974+0000","last_fullsized":"2024-05-28T06:05:29.445974+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.442069+0000","last_change":"2024-05-28T06:05:29.442069+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.442069+0000","last_undegraded":"2024-05-28T06:05:29.442069+0000","last_fullsized":"2024-05-28T06:05:29.442069+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.453076+0000","last_change":"2024-05-28T06:05:29.453076+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.453076+0000","last_undegraded":"2024-05-28T06:05:29.453076+0000","last_fullsized":"2024-05-28T06:05:29.453076+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.452478+0000","last_change":"2024-05-28T06:05:29.452478+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.452478+0000","last_undegraded":"2024-05-28T06:05:29.452478+0000","last_fullsized":"2024-05-28T06:05:29.452478+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.455664+0000","last_change":"2024-05-28T06:05:29.455664+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.455664+0000","last_undegraded":"2024-05-28T06:05:29.455664+0000","last_fullsized":"2024-05-28T06:05:29.455664+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.434352+0000","last_change":"2024-05-28T06:05:29.434352+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.434352+0000","last_undegraded":"2024-05-28T06:05:29.434352+0000","last_fullsized":"2024-05-28T06:05:29.434352+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.444010+0000","last_change":"2024-05-28T06:05:29.444010+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.444010+0000","last_undegraded":"2024-05-28T06:05:29.444010+0000","last_fullsized":"2024-05-28T06:05:29.444010+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.453472+0000","last_change":"2024-05-28T06:05:29.453472+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.453472+0000","last_undegraded":"2024-05-28T06:05:29.453472+0000","last_fullsized":"2024-05-28T06:05:29.453472+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.456491+0000","last_change":"2024-05-28T06:05:29.456491+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.456491+0000","last_undegraded":"2024-05-28T06:05:29.456491+0000","last_fullsized":"2024-05-28T06:05:29.456491+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.439713+0000","last_change":"2024-05-28T06:05:29.439713+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.439713+0000","last_undegraded":"2024-05-28T06:05:29.439713+0000","last_fullsized":"2024-05-28T06:05:29.439713+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.452561+0000","last_change":"2024-05-28T06:05:29.452561+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.452561+0000","last_undegraded":"2024-05-28T06:05:29.452561+0000","last_fullsized":"2024-05-28T06:05:29.452561+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.456977+0000","last_change":"2024-05-28T06:05:29.456977+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.456977+0000","last_undegraded":"2024-05-28T06:05:29.456977+0000","last_fullsized":"2024-05-28T06:05:29.456977+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.452961+0000","last_change":"2024-05-28T06:05:29.452961+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.452961+0000","last_undegraded":"2024-05-28T06:05:29.452961+0000","last_fullsized":"2024-05-28T06:05:29.452961+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.452936+0000","last_change":"2024-05-28T06:05:29.452936+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.452936+0000","last_undegraded":"2024-05-28T06:05:29.452936+0000","last_fullsized":"2024-05-28T06:05:29.452936+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.451797+0000","last_change":"2024-05-28T06:05:29.451797+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.451797+0000","last_undegraded":"2024-05-28T06:05:29.451797+0000","last_fullsized":"2024-05-28T06:05:29.451797+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.447932+0000","last_change":"2024-05-28T06:05:29.447932+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.447932+0000","last_undegraded":"2024-05-28T06:05:29.447932+0000","last_fullsized":"2024-05-28T06:05:29.447932+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.437958+0000","last_change":"2024-05-28T06:05:29.437958+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.437958+0000","last_undegraded":"2024-05-28T06:05:29.437958+0000","last_fullsized":"2024-05-28T06:05:29.437958+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.436112+0000","last_change":"2024-05-28T06:05:29.436112+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.436112+0000","last_undegraded":"2024-05-28T06:05:29.436112+0000","last_fullsized":"2024-05-28T06:05:29.436112+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.432539+0000","last_change":"2024-05-28T06:05:29.432539+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.432539+0000","last_undegraded":"2024-05-28T06:05:29.432539+0000","last_fullsized":"2024-05-28T06:05:29.432539+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.440180+0000","last_change":"2024-05-28T06:05:29.440180+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.440180+0000","last_undegraded":"2024-05-28T06:05:29.440180+0000","last_fullsized":"2024-05-28T06:05:29.440180+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.454693+0000","last_change":"2024-05-28T06:05:29.454693+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.454693+0000","last_undegraded":"2024-05-28T06:05:29.454693+0000","last_fullsized":"2024-05-28T06:05:29.454693+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.446445+0000","last_change":"2024-05-28T06:05:29.446445+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.446445+0000","last_undegraded":"2024-05-28T06:05:29.446445+0000","last_fullsized":"2024-05-28T06:05:29.446445+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.451346+0000","last_change":"2024-05-28T06:05:29.451346+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.451346+0000","last_undegraded":"2024-05-28T06:05:29.451346+0000","last_fullsized":"2024-05-28T06:05:29.451346+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.444042+0000","last_change":"2024-05-28T06:05:29.444042+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.444042+0000","last_undegraded":"2024-05-28T06:05:29.444042+0000","last_fullsized":"2024-05-28T06:05:29.444042+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.455928+0000","last_change":"2024-05-28T06:05:29.455928+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.455928+0000","last_undegraded":"2024-05-28T06:05:29.455928+0000","last_fullsized":"2024-05-28T06:05:29.455928+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.434836+0000","last_change":"2024-05-28T06:05:29.434836+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.434836+0000","last_undegraded":"2024-05-28T06:05:29.434836+0000","last_fullsized":"2024-05-28T06:05:29.434836+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.454892+0000","last_change":"2024-05-28T06:05:29.454892+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.454892+0000","last_undegraded":"2024-05-28T06:05:29.454892+0000","last_fullsized":"2024-05-28T06:05:29.454892+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.446998+0000","last_change":"2024-05-28T06:05:29.446998+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.446998+0000","last_undegraded":"2024-05-28T06:05:29.446998+0000","last_fullsized":"2024-05-28T06:05:29.446998+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.455194+0000","last_change":"2024-05-28T06:05:29.455194+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.455194+0000","last_undegraded":"2024-05-28T06:05:29.455194+0000","last_fullsized":"2024-05-28T06:05:29.455194+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.438448+0000","last_change":"2024-05-28T06:05:29.438448+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.438448+0000","last_undegraded":"2024-05-28T06:05:29.438448+0000","last_fullsized":"2024-05-28T06:05:29.438448+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-28T06:05:29.477464+0000","last_change":"2024-05-28T06:05:29.474199+0000","last_active":"2024-05-28T06:05:29.477464+0000","last_peered":"2024-05-28T06:05:29.477464+0000","last_clean":"2024-05-28T06:05:29.477464+0000","last_became_active":"2024-05-28T06:05:29.473863+0000","last_became_peered":"2024-05-28T06:05:29.473863+0000","last_unstale":"2024-05-28T06:05:29.477464+0000","last_undegraded":"2024-05-28T06:05:29.477464+0000","last_fullsized":"2024-05-28T06:05:29.477464+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:30:20.399303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.443791+0000","last_change":"2024-05-28T06:05:29.443791+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.443791+0000","last_undegraded":"2024-05-28T06:05:29.443791+0000","last_fullsized":"2024-05-28T06:05:29.443791+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-28T06:05:29.462994+0000","last_change":"2024-05-28T06:05:29.460672+0000","last_active":"2024-05-28T06:05:29.462994+0000","last_peered":"2024-05-28T06:05:29.462994+0000","last_clean":"2024-05-28T06:05:29.462994+0000","last_became_active":"2024-05-28T06:05:29.459991+0000","last_became_peered":"2024-05-28T06:05:29.459991+0000","last_unstale":"2024-05-28T06:05:29.462994+0000","last_undegraded":"2024-05-28T06:05:29.462994+0000","last_fullsized":"2024-05-28T06:05:29.462994+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:05:43.269097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.433757+0000","last_change":"2024-05-28T06:05:29.433757+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.433757+0000","last_undegraded":"2024-05-28T06:05:29.433757+0000","last_fullsized":"2024-05-28T06:05:29.433757+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-28T06:05:29.461037+0000","last_change":"2024-05-28T06:05:29.456735+0000","last_active":"2024-05-28T06:05:29.461037+0000","last_peered":"2024-05-28T06:05:29.461037+0000","last_clean":"2024-05-28T06:05:29.461037+0000","last_became_active":"2024-05-28T06:05:29.456466+0000","last_became_peered":"2024-05-28T06:05:29.456466+0000","last_unstale":"2024-05-28T06:05:29.461037+0000","last_undegraded":"2024-05-28T06:05:29.461037+0000","last_fullsized":"2024-05-28T06:05:29.461037+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:47:21.316456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.443049+0000","last_change":"2024-05-28T06:05:29.443049+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.443049+0000","last_undegraded":"2024-05-28T06:05:29.443049+0000","last_fullsized":"2024-05-28T06:05:29.443049+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-28T06:05:29.455760+0000","last_change":"2024-05-28T06:05:29.451693+0000","last_active":"2024-05-28T06:05:29.455760+0000","last_peered":"2024-05-28T06:05:29.455760+0000","last_clean":"2024-05-28T06:05:29.455760+0000","last_became_active":"2024-05-28T06:05:29.451343+0000","last_became_peered":"2024-05-28T06:05:29.451343+0000","last_unstale":"2024-05-28T06:05:29.455760+0000","last_undegraded":"2024-05-28T06:05:29.455760+0000","last_fullsized":"2024-05-28T06:05:29.455760+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:42:38.699538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.454003+0000","last_change":"2024-05-28T06:05:29.454003+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.454003+0000","last_undegraded":"2024-05-28T06:05:29.454003+0000","last_fullsized":"2024-05-28T06:05:29.454003+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-28T06:05:29.469115+0000","last_change":"2024-05-28T06:05:29.463758+0000","last_active":"2024-05-28T06:05:29.469115+0000","last_peered":"2024-05-28T06:05:29.469115+0000","last_clean":"2024-05-28T06:05:29.469115+0000","last_became_active":"2024-05-28T06:05:29.460297+0000","last_became_peered":"2024-05-28T06:05:29.460297+0000","last_unstale":"2024-05-28T06:05:29.469115+0000","last_undegraded":"2024-05-28T06:05:29.469115+0000","last_fullsized":"2024-05-28T06:05:29.469115+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:45:19.781181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.455037+0000","last_change":"2024-05-28T06:05:29.455037+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.455037+0000","last_undegraded":"2024-05-28T06:05:29.455037+0000","last_fullsized":"2024-05-28T06:05:29.455037+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-28T06:05:29.468155+0000","last_change":"2024-05-28T06:05:29.462024+0000","last_active":"2024-05-28T06:05:29.468155+0000","last_peered":"2024-05-28T06:05:29.468155+0000","last_clean":"2024-05-28T06:05:29.468155+0000","last_became_active":"2024-05-28T06:05:29.460033+0000","last_became_peered":"2024-05-28T06:05:29.460033+0000","last_unstale":"2024-05-28T06:05:29.468155+0000","last_undegraded":"2024-05-28T06:05:29.468155+0000","last_fullsized":"2024-05-28T06:05:29.468155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:43:59.184224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.432275+0000","last_change":"2024-05-28T06:05:29.432275+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.432275+0000","last_undegraded":"2024-05-28T06:05:29.432275+0000","last_fullsized":"2024-05-28T06:05:29.432275+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-28T06:05:29.459996+0000","last_change":"2024-05-28T06:05:29.456433+0000","last_active":"2024-05-28T06:05:29.459996+0000","last_peered":"2024-05-28T06:05:29.459996+0000","last_clean":"2024-05-28T06:05:29.459996+0000","last_became_active":"2024-05-28T06:05:29.456113+0000","last_became_peered":"2024-05-28T06:05:29.456113+0000","last_unstale":"2024-05-28T06:05:29.459996+0000","last_undegraded":"2024-05-28T06:05:29.459996+0000","last_fullsized":"2024-05-28T06:05:29.459996+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:10:48.750962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.445350+0000","last_change":"2024-05-28T06:05:29.445350+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.445350+0000","last_undegraded":"2024-05-28T06:05:29.445350+0000","last_fullsized":"2024-05-28T06:05:29.445350+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-28T06:05:29.468715+0000","last_change":"2024-05-28T06:05:29.462393+0000","last_active":"2024-05-28T06:05:29.468715+0000","last_peered":"2024-05-28T06:05:29.468715+0000","last_clean":"2024-05-28T06:05:29.468715+0000","last_became_active":"2024-05-28T06:05:29.461785+0000","last_became_peered":"2024-05-28T06:05:29.461785+0000","last_unstale":"2024-05-28T06:05:29.468715+0000","last_undegraded":"2024-05-28T06:05:29.468715+0000","last_fullsized":"2024-05-28T06:05:29.468715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:42:15.256795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.458171+0000","last_change":"2024-05-28T06:05:29.458171+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.458171+0000","last_undegraded":"2024-05-28T06:05:29.458171+0000","last_fullsized":"2024-05-28T06:05:29.458171+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-28T06:05:29.463385+0000","last_change":"2024-05-28T06:05:29.461808+0000","last_active":"2024-05-28T06:05:29.463385+0000","last_peered":"2024-05-28T06:05:29.463385+0000","last_clean":"2024-05-28T06:05:29.463385+0000","last_became_active":"2024-05-28T06:05:29.461481+0000","last_became_peered":"2024-05-28T06:05:29.461481+0000","last_unstale":"2024-05-28T06:05:29.463385+0000","last_undegraded":"2024-05-28T06:05:29.463385+0000","last_fullsized":"2024-05-28T06:05:29.463385+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:32:34.183989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.442237+0000","last_change":"2024-05-28T06:05:29.442237+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.442237+0000","last_undegraded":"2024-05-28T06:05:29.442237+0000","last_fullsized":"2024-05-28T06:05:29.442237+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-28T06:05:29.463015+0000","last_change":"2024-05-28T06:05:29.460940+0000","last_active":"2024-05-28T06:05:29.463015+0000","last_peered":"2024-05-28T06:05:29.463015+0000","last_clean":"2024-05-28T06:05:29.463015+0000","last_became_active":"2024-05-28T06:05:29.460433+0000","last_became_peered":"2024-05-28T06:05:29.460433+0000","last_unstale":"2024-05-28T06:05:29.463015+0000","last_undegraded":"2024-05-28T06:05:29.463015+0000","last_fullsized":"2024-05-28T06:05:29.463015+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:30:20.399303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.452611+0000","last_change":"2024-05-28T06:05:29.452611+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.452611+0000","last_undegraded":"2024-05-28T06:05:29.452611+0000","last_fullsized":"2024-05-28T06:05:29.452611+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-28T06:05:29.469706+0000","last_change":"2024-05-28T06:05:29.462646+0000","last_active":"2024-05-28T06:05:29.469706+0000","last_peered":"2024-05-28T06:05:29.469706+0000","last_clean":"2024-05-28T06:05:29.469706+0000","last_became_active":"2024-05-28T06:05:29.462355+0000","last_became_peered":"2024-05-28T06:05:29.462355+0000","last_unstale":"2024-05-28T06:05:29.469706+0000","last_undegraded":"2024-05-28T06:05:29.469706+0000","last_fullsized":"2024-05-28T06:05:29.469706+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:12:52.117940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.455754+0000","last_change":"2024-05-28T06:05:29.455754+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.455754+0000","last_undegraded":"2024-05-28T06:05:29.455754+0000","last_fullsized":"2024-05-28T06:05:29.455754+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-28T06:05:29.460944+0000","last_change":"2024-05-28T06:05:29.458155+0000","last_active":"2024-05-28T06:05:29.460944+0000","last_peered":"2024-05-28T06:05:29.460944+0000","last_clean":"2024-05-28T06:05:29.460944+0000","last_became_active":"2024-05-28T06:05:29.457822+0000","last_became_peered":"2024-05-28T06:05:29.457822+0000","last_unstale":"2024-05-28T06:05:29.460944+0000","last_undegraded":"2024-05-28T06:05:29.460944+0000","last_fullsized":"2024-05-28T06:05:29.460944+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:12:38.485450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.442805+0000","last_change":"2024-05-28T06:05:29.442805+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.442805+0000","last_undegraded":"2024-05-28T06:05:29.442805+0000","last_fullsized":"2024-05-28T06:05:29.442805+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-28T06:05:29.469142+0000","last_change":"2024-05-28T06:05:29.464610+0000","last_active":"2024-05-28T06:05:29.469142+0000","last_peered":"2024-05-28T06:05:29.469142+0000","last_clean":"2024-05-28T06:05:29.469142+0000","last_became_active":"2024-05-28T06:05:29.464074+0000","last_became_peered":"2024-05-28T06:05:29.464074+0000","last_unstale":"2024-05-28T06:05:29.469142+0000","last_undegraded":"2024-05-28T06:05:29.469142+0000","last_fullsized":"2024-05-28T06:05:29.469142+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:31:54.521268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.453696+0000","last_change":"2024-05-28T06:05:29.453696+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.453696+0000","last_undegraded":"2024-05-28T06:05:29.453696+0000","last_fullsized":"2024-05-28T06:05:29.453696+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-28T06:05:29.464678+0000","last_change":"2024-05-28T06:05:29.463194+0000","last_active":"2024-05-28T06:05:29.464678+0000","last_peered":"2024-05-28T06:05:29.464678+0000","last_clean":"2024-05-28T06:05:29.464678+0000","last_became_active":"2024-05-28T06:05:29.462870+0000","last_became_peered":"2024-05-28T06:05:29.462870+0000","last_unstale":"2024-05-28T06:05:29.464678+0000","last_undegraded":"2024-05-28T06:05:29.464678+0000","last_fullsized":"2024-05-28T06:05:29.464678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:30:20.399303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.454681+0000","last_change":"2024-05-28T06:05:29.454681+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.454681+0000","last_undegraded":"2024-05-28T06:05:29.454681+0000","last_fullsized":"2024-05-28T06:05:29.454681+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-28T06:05:29.463771+0000","last_change":"2024-05-28T06:05:29.462298+0000","last_active":"2024-05-28T06:05:29.463771+0000","last_peered":"2024-05-28T06:05:29.463771+0000","last_clean":"2024-05-28T06:05:29.463771+0000","last_became_active":"2024-05-28T06:05:29.462022+0000","last_became_peered":"2024-05-28T06:05:29.462022+0000","last_unstale":"2024-05-28T06:05:29.463771+0000","last_undegraded":"2024-05-28T06:05:29.463771+0000","last_fullsized":"2024-05-28T06:05:29.463771+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:11:47.259186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.441893+0000","last_change":"2024-05-28T06:05:29.441893+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.441893+0000","last_undegraded":"2024-05-28T06:05:29.441893+0000","last_fullsized":"2024-05-28T06:05:29.441893+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-28T06:05:29.463784+0000","last_change":"2024-05-28T06:05:29.460103+0000","last_active":"2024-05-28T06:05:29.463784+0000","last_peered":"2024-05-28T06:05:29.463784+0000","last_clean":"2024-05-28T06:05:29.463784+0000","last_became_active":"2024-05-28T06:05:29.459788+0000","last_became_peered":"2024-05-28T06:05:29.459788+0000","last_unstale":"2024-05-28T06:05:29.463784+0000","last_undegraded":"2024-05-28T06:05:29.463784+0000","last_fullsized":"2024-05-28T06:05:29.463784+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:57:51.752948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.434191+0000","last_change":"2024-05-28T06:05:29.434191+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.434191+0000","last_undegraded":"2024-05-28T06:05:29.434191+0000","last_fullsized":"2024-05-28T06:05:29.434191+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-28T06:05:29.460988+0000","last_change":"2024-05-28T06:05:29.458728+0000","last_active":"2024-05-28T06:05:29.460988+0000","last_peered":"2024-05-28T06:05:29.460988+0000","last_clean":"2024-05-28T06:05:29.460988+0000","last_became_active":"2024-05-28T06:05:29.458154+0000","last_became_peered":"2024-05-28T06:05:29.458154+0000","last_unstale":"2024-05-28T06:05:29.460988+0000","last_undegraded":"2024-05-28T06:05:29.460988+0000","last_fullsized":"2024-05-28T06:05:29.460988+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:05:43.269097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.451184+0000","last_change":"2024-05-28T06:05:29.451184+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.451184+0000","last_undegraded":"2024-05-28T06:05:29.451184+0000","last_fullsized":"2024-05-28T06:05:29.451184+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-28T06:05:29.469005+0000","last_change":"2024-05-28T06:05:29.463122+0000","last_active":"2024-05-28T06:05:29.469005+0000","last_peered":"2024-05-28T06:05:29.469005+0000","last_clean":"2024-05-28T06:05:29.469005+0000","last_became_active":"2024-05-28T06:05:29.462362+0000","last_became_peered":"2024-05-28T06:05:29.462362+0000","last_unstale":"2024-05-28T06:05:29.469005+0000","last_undegraded":"2024-05-28T06:05:29.469005+0000","last_fullsized":"2024-05-28T06:05:29.469005+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:01:56.755222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.442315+0000","last_change":"2024-05-28T06:05:29.442315+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.442315+0000","last_undegraded":"2024-05-28T06:05:29.442315+0000","last_fullsized":"2024-05-28T06:05:29.442315+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-28T06:05:29.459230+0000","last_change":"2024-05-28T06:05:29.450660+0000","last_active":"2024-05-28T06:05:29.459230+0000","last_peered":"2024-05-28T06:05:29.459230+0000","last_clean":"2024-05-28T06:05:29.459230+0000","last_became_active":"2024-05-28T06:05:29.446547+0000","last_became_peered":"2024-05-28T06:05:29.446547+0000","last_unstale":"2024-05-28T06:05:29.459230+0000","last_undegraded":"2024-05-28T06:05:29.459230+0000","last_fullsized":"2024-05-28T06:05:29.459230+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:42:38.699538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.437631+0000","last_change":"2024-05-28T06:05:29.437631+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.437631+0000","last_undegraded":"2024-05-28T06:05:29.437631+0000","last_fullsized":"2024-05-28T06:05:29.437631+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-28T06:05:29.468175+0000","last_change":"2024-05-28T06:05:29.461213+0000","last_active":"2024-05-28T06:05:29.468175+0000","last_peered":"2024-05-28T06:05:29.468175+0000","last_clean":"2024-05-28T06:05:29.468175+0000","last_became_active":"2024-05-28T06:05:29.460899+0000","last_became_peered":"2024-05-28T06:05:29.460899+0000","last_unstale":"2024-05-28T06:05:29.468175+0000","last_undegraded":"2024-05-28T06:05:29.468175+0000","last_fullsized":"2024-05-28T06:05:29.468175+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:07:14.772596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.436257+0000","last_change":"2024-05-28T06:05:29.436257+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.436257+0000","last_undegraded":"2024-05-28T06:05:29.436257+0000","last_fullsized":"2024-05-28T06:05:29.436257+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-28T06:05:29.452420+0000","last_change":"2024-05-28T06:05:29.447607+0000","last_active":"2024-05-28T06:05:29.452420+0000","last_peered":"2024-05-28T06:05:29.452420+0000","last_clean":"2024-05-28T06:05:29.452420+0000","last_became_active":"2024-05-28T06:05:29.447321+0000","last_became_peered":"2024-05-28T06:05:29.447321+0000","last_unstale":"2024-05-28T06:05:29.452420+0000","last_undegraded":"2024-05-28T06:05:29.452420+0000","last_fullsized":"2024-05-28T06:05:29.452420+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:22:38.584903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.437966+0000","last_change":"2024-05-28T06:05:29.437966+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.437966+0000","last_undegraded":"2024-05-28T06:05:29.437966+0000","last_fullsized":"2024-05-28T06:05:29.437966+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-28T06:05:29.470438+0000","last_change":"2024-05-28T06:05:29.463849+0000","last_active":"2024-05-28T06:05:29.470438+0000","last_peered":"2024-05-28T06:05:29.470438+0000","last_clean":"2024-05-28T06:05:29.470438+0000","last_became_active":"2024-05-28T06:05:29.463514+0000","last_became_peered":"2024-05-28T06:05:29.463514+0000","last_unstale":"2024-05-28T06:05:29.470438+0000","last_undegraded":"2024-05-28T06:05:29.470438+0000","last_fullsized":"2024-05-28T06:05:29.470438+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:05:43.269097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.436238+0000","last_change":"2024-05-28T06:05:29.436238+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.436238+0000","last_undegraded":"2024-05-28T06:05:29.436238+0000","last_fullsized":"2024-05-28T06:05:29.436238+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-28T06:05:29.457491+0000","last_change":"2024-05-28T06:05:29.450109+0000","last_active":"2024-05-28T06:05:29.457491+0000","last_peered":"2024-05-28T06:05:29.457491+0000","last_clean":"2024-05-28T06:05:29.457491+0000","last_became_active":"2024-05-28T06:05:29.449618+0000","last_became_peered":"2024-05-28T06:05:29.449618+0000","last_unstale":"2024-05-28T06:05:29.457491+0000","last_undegraded":"2024-05-28T06:05:29.457491+0000","last_fullsized":"2024-05-28T06:05:29.457491+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:09:47.123693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.453182+0000","last_change":"2024-05-28T06:05:29.453182+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.453182+0000","last_undegraded":"2024-05-28T06:05:29.453182+0000","last_fullsized":"2024-05-28T06:05:29.453182+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-28T06:05:29.471119+0000","last_change":"2024-05-28T06:05:29.467769+0000","last_active":"2024-05-28T06:05:29.471119+0000","last_peered":"2024-05-28T06:05:29.471119+0000","last_clean":"2024-05-28T06:05:29.471119+0000","last_became_active":"2024-05-28T06:05:29.467422+0000","last_became_peered":"2024-05-28T06:05:29.467422+0000","last_unstale":"2024-05-28T06:05:29.471119+0000","last_undegraded":"2024-05-28T06:05:29.471119+0000","last_fullsized":"2024-05-28T06:05:29.471119+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:33:35.470452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.436688+0000","last_change":"2024-05-28T06:05:29.436688+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.436688+0000","last_undegraded":"2024-05-28T06:05:29.436688+0000","last_fullsized":"2024-05-28T06:05:29.436688+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-28T06:05:29.472543+0000","last_change":"2024-05-28T06:05:29.471119+0000","last_active":"2024-05-28T06:05:29.472543+0000","last_peered":"2024-05-28T06:05:29.472543+0000","last_clean":"2024-05-28T06:05:29.472543+0000","last_became_active":"2024-05-28T06:05:29.470884+0000","last_became_peered":"2024-05-28T06:05:29.470884+0000","last_unstale":"2024-05-28T06:05:29.472543+0000","last_undegraded":"2024-05-28T06:05:29.472543+0000","last_fullsized":"2024-05-28T06:05:29.472543+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:47:21.316456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.448476+0000","last_change":"2024-05-28T06:05:29.448476+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.448476+0000","last_undegraded":"2024-05-28T06:05:29.448476+0000","last_fullsized":"2024-05-28T06:05:29.448476+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-28T06:05:29.452484+0000","last_change":"2024-05-28T06:05:29.449898+0000","last_active":"2024-05-28T06:05:29.452484+0000","last_peered":"2024-05-28T06:05:29.452484+0000","last_clean":"2024-05-28T06:05:29.452484+0000","last_became_active":"2024-05-28T06:05:29.449322+0000","last_became_peered":"2024-05-28T06:05:29.449322+0000","last_unstale":"2024-05-28T06:05:29.452484+0000","last_undegraded":"2024-05-28T06:05:29.452484+0000","last_fullsized":"2024-05-28T06:05:29.452484+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:16:42.579003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.439531+0000","last_change":"2024-05-28T06:05:29.439531+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.439531+0000","last_undegraded":"2024-05-28T06:05:29.439531+0000","last_fullsized":"2024-05-28T06:05:29.439531+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-28T06:05:29.457020+0000","last_change":"2024-05-28T06:05:29.451001+0000","last_active":"2024-05-28T06:05:29.457020+0000","last_peered":"2024-05-28T06:05:29.457020+0000","last_clean":"2024-05-28T06:05:29.457020+0000","last_became_active":"2024-05-28T06:05:29.450717+0000","last_became_peered":"2024-05-28T06:05:29.450717+0000","last_unstale":"2024-05-28T06:05:29.457020+0000","last_undegraded":"2024-05-28T06:05:29.457020+0000","last_fullsized":"2024-05-28T06:05:29.457020+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:16:42.579003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.448185+0000","last_change":"2024-05-28T06:05:29.448185+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.448185+0000","last_undegraded":"2024-05-28T06:05:29.448185+0000","last_fullsized":"2024-05-28T06:05:29.448185+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-28T06:05:29.470603+0000","last_change":"2024-05-28T06:05:29.469461+0000","last_active":"2024-05-28T06:05:29.470603+0000","last_peered":"2024-05-28T06:05:29.470603+0000","last_clean":"2024-05-28T06:05:29.470603+0000","last_became_active":"2024-05-28T06:05:29.469114+0000","last_became_peered":"2024-05-28T06:05:29.469114+0000","last_unstale":"2024-05-28T06:05:29.470603+0000","last_undegraded":"2024-05-28T06:05:29.470603+0000","last_fullsized":"2024-05-28T06:05:29.470603+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:10:48.750962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.438458+0000","last_change":"2024-05-28T06:05:29.438458+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.438458+0000","last_undegraded":"2024-05-28T06:05:29.438458+0000","last_fullsized":"2024-05-28T06:05:29.438458+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-28T06:05:29.470697+0000","last_change":"2024-05-28T06:05:29.469913+0000","last_active":"2024-05-28T06:05:29.470697+0000","last_peered":"2024-05-28T06:05:29.470697+0000","last_clean":"2024-05-28T06:05:29.470697+0000","last_became_active":"2024-05-28T06:05:29.469460+0000","last_became_peered":"2024-05-28T06:05:29.469460+0000","last_unstale":"2024-05-28T06:05:29.470697+0000","last_undegraded":"2024-05-28T06:05:29.470697+0000","last_fullsized":"2024-05-28T06:05:29.470697+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:10:48.750962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.444841+0000","last_change":"2024-05-28T06:05:29.444841+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.444841+0000","last_undegraded":"2024-05-28T06:05:29.444841+0000","last_fullsized":"2024-05-28T06:05:29.444841+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-28T06:05:29.471575+0000","last_change":"2024-05-28T06:05:29.469678+0000","last_active":"2024-05-28T06:05:29.471575+0000","last_peered":"2024-05-28T06:05:29.471575+0000","last_clean":"2024-05-28T06:05:29.471575+0000","last_became_active":"2024-05-28T06:05:29.469209+0000","last_became_peered":"2024-05-28T06:05:29.469209+0000","last_unstale":"2024-05-28T06:05:29.471575+0000","last_undegraded":"2024-05-28T06:05:29.471575+0000","last_fullsized":"2024-05-28T06:05:29.471575+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:37:09.560235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.433299+0000","last_change":"2024-05-28T06:05:29.433299+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.433299+0000","last_undegraded":"2024-05-28T06:05:29.433299+0000","last_fullsized":"2024-05-28T06:05:29.433299+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-28T06:05:29.467257+0000","last_change":"2024-05-28T06:05:29.465063+0000","last_active":"2024-05-28T06:05:29.467257+0000","last_peered":"2024-05-28T06:05:29.467257+0000","last_clean":"2024-05-28T06:05:29.467257+0000","last_became_active":"2024-05-28T06:05:29.464579+0000","last_became_peered":"2024-05-28T06:05:29.464579+0000","last_unstale":"2024-05-28T06:05:29.467257+0000","last_undegraded":"2024-05-28T06:05:29.467257+0000","last_fullsized":"2024-05-28T06:05:29.467257+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:58:31.330922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.451026+0000","last_change":"2024-05-28T06:05:29.451026+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.451026+0000","last_undegraded":"2024-05-28T06:05:29.451026+0000","last_fullsized":"2024-05-28T06:05:29.451026+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-28T06:05:29.465703+0000","last_change":"2024-05-28T06:05:29.461542+0000","last_active":"2024-05-28T06:05:29.465703+0000","last_peered":"2024-05-28T06:05:29.465703+0000","last_clean":"2024-05-28T06:05:29.465703+0000","last_became_active":"2024-05-28T06:05:29.461155+0000","last_became_peered":"2024-05-28T06:05:29.461155+0000","last_unstale":"2024-05-28T06:05:29.465703+0000","last_undegraded":"2024-05-28T06:05:29.465703+0000","last_fullsized":"2024-05-28T06:05:29.465703+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:16:42.579003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":190,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-28T06:05:29.456002+0000","last_change":"2024-05-28T06:04:01.039129+0000","last_active":"2024-05-28T06:05:29.456002+0000","last_peered":"2024-05-28T06:05:29.456002+0000","last_clean":"2024-05-28T06:05:29.456002+0000","last_became_active":"2024-05-28T06:04:01.033906+0000","last_became_peered":"2024-05-28T06:04:01.033906+0000","last_unstale":"2024-05-28T06:05:29.456002+0000","last_undegraded":"2024-05-28T06:05:29.456002+0000","last_fullsized":"2024-05-28T06:05:29.456002+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:03:59.864237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:03:59.864237+0000","last_clean_scrub_stamp":"2024-05-28T06:03:59.864237+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:17:36.440245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574869,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27928,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343912,"statfs":{"total":96636764160,"available":96608165888,"internally_reserved":0,"allocated":929792,"data_stored":717642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574871,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":348160,"data_stored":133541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574869,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":335872,"data_stored":127274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574869,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":335872,"data_stored":127274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574871,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":335872,"data_stored":127274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574870,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":335872,"data_stored":127274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574870,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":335872,"data_stored":127274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574869,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27928,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343912,"statfs":{"total":96636764160,"available":96608165888,"internally_reserved":0,"allocated":929792,"data_stored":717642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-28T06:05:34.899 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-28T06:05:35.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.134+0000 7fcaca400640 1 Processor -- start 2024-05-28T06:05:35.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.135+0000 7fcaca400640 1 -- start start 2024-05-28T06:05:35.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.136+0000 7fcaca400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fcac4158b80 0x7fcac406dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:35.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.136+0000 7fcaca400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcac406e160 0x7fcac4071bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:35.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.136+0000 7fcaca400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fcac4072100 0x7fcac41621f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:35.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.136+0000 7fcaca400640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fcac406e6e0 con 0x7fcac4072100 2024-05-28T06:05:35.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.136+0000 7fcaca400640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fcac406e9c0 con 0x7fcac4158b80 2024-05-28T06:05:35.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.136+0000 7fcaca400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fcac406eca0 con 0x7fcac406e160 2024-05-28T06:05:35.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.136+0000 7fcac9a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fcac4072100 0x7fcac41621f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:35.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.136+0000 7fcac9a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fcac4072100 0x7fcac41621f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:50890/0 (socket says 172.21.15.96:50890) 2024-05-28T06:05:35.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.136+0000 7fcac9a00640 1 -- 172.21.15.96:0/1229673066 learned_addr learned my addr 172.21.15.96:0/1229673066 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:35.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.137+0000 7fcabbe00640 1 --2- 172.21.15.96:0/1229673066 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcac406e160 0x7fcac4071bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:35.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.137+0000 7fcac9000640 1 --2- 172.21.15.96:0/1229673066 >> v2:172.21.15.123:3300/0 conn(0x7fcac4158b80 0x7fcac406dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:35.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.137+0000 7fcac9a00640 1 -- 172.21.15.96:0/1229673066 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcac406e160 msgr2=0x7fcac4071bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:35.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.137+0000 7fcac9a00640 1 --2- 172.21.15.96:0/1229673066 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcac406e160 0x7fcac4071bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:35.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.137+0000 7fcac9a00640 1 -- 172.21.15.96:0/1229673066 >> v2:172.21.15.123:3300/0 conn(0x7fcac4158b80 msgr2=0x7fcac406dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:35.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.137+0000 7fcac9a00640 1 --2- 172.21.15.96:0/1229673066 >> v2:172.21.15.123:3300/0 conn(0x7fcac4158b80 0x7fcac406dc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:35.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.137+0000 7fcac9a00640 1 -- 172.21.15.96:0/1229673066 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcac406ef80 con 0x7fcac4072100 2024-05-28T06:05:35.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.137+0000 7fcac9000640 1 --2- 172.21.15.96:0/1229673066 >> v2:172.21.15.123:3300/0 conn(0x7fcac4158b80 0x7fcac406dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:05:35.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.138+0000 7fcac9a00640 1 --2- 172.21.15.96:0/1229673066 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcac4072100 0x7fcac41621f0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fcac406c830 tx=0x7fcabc02a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcc5959a2caf23a server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:35.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.138+0000 7fcac3e00640 1 -- 172.21.15.96:0/1229673066 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcabc036070 con 0x7fcac4072100 2024-05-28T06:05:35.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.138+0000 7fcac3e00640 1 -- 172.21.15.96:0/1229673066 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcabc031600 con 0x7fcac4072100 2024-05-28T06:05:35.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.138+0000 7fcac3e00640 1 -- 172.21.15.96:0/1229673066 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcabc0319d0 con 0x7fcac4072100 2024-05-28T06:05:35.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.139+0000 7fcaca400640 1 -- 172.21.15.96:0/1229673066 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcac4072100 msgr2=0x7fcac41621f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:35.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.139+0000 7fcaca400640 1 --2- 172.21.15.96:0/1229673066 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcac4072100 0x7fcac41621f0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fcac406c830 tx=0x7fcabc02a6a0 comp rx=0 tx=0).stop 2024-05-28T06:05:35.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.139+0000 7fcaca400640 1 -- 172.21.15.96:0/1229673066 shutdown_connections 2024-05-28T06:05:35.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.139+0000 7fcaca400640 1 --2- 172.21.15.96:0/1229673066 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcac4072100 0x7fcac41621f0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:35.143 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.139+0000 7fcaca400640 1 --2- 172.21.15.96:0/1229673066 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcac406e160 0x7fcac4071bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:35.143 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.139+0000 7fcaca400640 1 --2- 172.21.15.96:0/1229673066 >> v2:172.21.15.123:3300/0 conn(0x7fcac4158b80 0x7fcac406dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:35.143 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.139+0000 7fcaca400640 1 -- 172.21.15.96:0/1229673066 >> 172.21.15.96:0/1229673066 conn(0x7fcac40b1fc0 msgr2=0x7fcac40b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:35.143 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.139+0000 7fcaca400640 1 -- 172.21.15.96:0/1229673066 shutdown_connections 2024-05-28T06:05:35.143 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.139+0000 7fcaca400640 1 -- 172.21.15.96:0/1229673066 wait complete. 2024-05-28T06:05:35.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.141+0000 7fcaca400640 1 Processor -- start 2024-05-28T06:05:35.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.141+0000 7fcaca400640 1 -- start start 2024-05-28T06:05:35.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.142+0000 7fcaca400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcac406e160 0x7fcac4141ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:35.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.142+0000 7fcaca400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcac4072100 0x7fcac41535a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:35.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.142+0000 7fcaca400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcac4158b80 0x7fcac4153ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:35.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.142+0000 7fcaca400640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fcac406f780 con 0x7fcac4072100 2024-05-28T06:05:35.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.142+0000 7fcaca400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fcac40b2d30 con 0x7fcac406e160 2024-05-28T06:05:35.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.142+0000 7fcaca400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fcac41423c0 con 0x7fcac4158b80 2024-05-28T06:05:35.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.142+0000 7fcabbe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcac4072100 0x7fcac41535a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:35.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.142+0000 7fcabbe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcac4072100 0x7fcac41535a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:50898/0 (socket says 172.21.15.96:50898) 2024-05-28T06:05:35.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.142+0000 7fcabbe00640 1 -- 172.21.15.96:0/1922866591 learned_addr learned my addr 172.21.15.96:0/1922866591 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:35.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.143+0000 7fcac9a00640 1 --2- 172.21.15.96:0/1922866591 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcac4158b80 0x7fcac4153ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:35.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.143+0000 7fcac9000640 1 --2- 172.21.15.96:0/1922866591 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcac406e160 0x7fcac4141ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:35.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.143+0000 7fcabbe00640 1 -- 172.21.15.96:0/1922866591 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcac4158b80 msgr2=0x7fcac4153ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:35.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.143+0000 7fcabbe00640 1 --2- 172.21.15.96:0/1922866591 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcac4158b80 0x7fcac4153ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:35.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.143+0000 7fcabbe00640 1 -- 172.21.15.96:0/1922866591 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcac406e160 msgr2=0x7fcac4141ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:35.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.143+0000 7fcabbe00640 1 --2- 172.21.15.96:0/1922866591 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcac406e160 0x7fcac4141ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:35.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.143+0000 7fcabbe00640 1 -- 172.21.15.96:0/1922866591 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcac41426a0 con 0x7fcac4072100 2024-05-28T06:05:35.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.143+0000 7fcac9a00640 1 --2- 172.21.15.96:0/1922866591 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcac4158b80 0x7fcac4153ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:05:35.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.143+0000 7fcac9000640 1 --2- 172.21.15.96:0/1922866591 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcac406e160 0x7fcac4141ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:05:35.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.143+0000 7fcabbe00640 1 --2- 172.21.15.96:0/1922866591 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcac4072100 0x7fcac41535a0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fcaac00d7d0 tx=0x7fcaac00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:35.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.143+0000 7fcac1600640 1 -- 172.21.15.96:0/1922866591 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaac004020 con 0x7fcac4072100 2024-05-28T06:05:35.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.143+0000 7fcac1600640 1 -- 172.21.15.96:0/1922866591 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcaac0041e0 con 0x7fcac4072100 2024-05-28T06:05:35.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.143+0000 7fcac1600640 1 -- 172.21.15.96:0/1922866591 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaac0043d0 con 0x7fcac4072100 2024-05-28T06:05:35.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.143+0000 7fcaca400640 1 -- 172.21.15.96:0/1922866591 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcac4142980 con 0x7fcac4072100 2024-05-28T06:05:35.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.143+0000 7fcaca400640 1 -- 172.21.15.96:0/1922866591 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcac4142000 con 0x7fcac4072100 2024-05-28T06:05:35.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.145+0000 7fcaca400640 1 -- 172.21.15.96:0/1922866591 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcac40689e0 con 0x7fcac4072100 2024-05-28T06:05:35.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.145+0000 7fcac1600640 1 -- 172.21.15.96:0/1922866591 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fcaac01b020 con 0x7fcac4072100 2024-05-28T06:05:35.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.146+0000 7fcac1600640 1 --2- 172.21.15.96:0/1922866591 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fcaa006c8f0 0x7fcaa006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:35.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.146+0000 7fcac1600640 1 -- 172.21.15.96:0/1922866591 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7fcaac08ba10 con 0x7fcac4072100 2024-05-28T06:05:35.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.147+0000 7fcac9000640 1 --2- 172.21.15.96:0/1922866591 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fcaa006c8f0 0x7fcaa006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:35.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.148+0000 7fcac9000640 1 --2- 172.21.15.96:0/1922866591 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fcaa006c8f0 0x7fcaa006edb0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fcab4006fd0 tx=0x7fcab4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:35.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.150+0000 7fcac1600640 1 -- 172.21.15.96:0/1922866591 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fcaac059e30 con 0x7fcac4072100 2024-05-28T06:05:35.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.347+0000 7fcaca400640 1 -- 172.21.15.96:0/1922866591 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fcac4152c90 con 0x7fcaa006c8f0 2024-05-28T06:05:35.368 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.366+0000 7fcac1600640 1 -- 172.21.15.96:0/1922866591 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+246631 (secure 0 0 0) 0x7fcac4152c90 con 0x7fcaa006c8f0 2024-05-28T06:05:35.371 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:05:35.372 INFO:teuthology.orchestra.run.smithi096.stderr:dumped all 2024-05-28T06:05:35.374 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.372+0000 7fcaca400640 1 -- 172.21.15.96:0/1922866591 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fcaa006c8f0 msgr2=0x7fcaa006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:35.374 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.372+0000 7fcaca400640 1 --2- 172.21.15.96:0/1922866591 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fcaa006c8f0 0x7fcaa006edb0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fcab4006fd0 tx=0x7fcab4008040 comp rx=0 tx=0).stop 2024-05-28T06:05:35.374 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.372+0000 7fcaca400640 1 -- 172.21.15.96:0/1922866591 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcac4072100 msgr2=0x7fcac41535a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:35.374 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.372+0000 7fcaca400640 1 --2- 172.21.15.96:0/1922866591 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcac4072100 0x7fcac41535a0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fcaac00d7d0 tx=0x7fcaac00dc90 comp rx=0 tx=0).stop 2024-05-28T06:05:35.374 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.372+0000 7fcaca400640 1 -- 172.21.15.96:0/1922866591 shutdown_connections 2024-05-28T06:05:35.374 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.372+0000 7fcaca400640 1 --2- 172.21.15.96:0/1922866591 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fcaa006c8f0 0x7fcaa006edb0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:35.374 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.372+0000 7fcaca400640 1 --2- 172.21.15.96:0/1922866591 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcac4158b80 0x7fcac4153ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:35.375 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.372+0000 7fcaca400640 1 --2- 172.21.15.96:0/1922866591 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcac4072100 0x7fcac41535a0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:35.375 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.372+0000 7fcaca400640 1 --2- 172.21.15.96:0/1922866591 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcac406e160 0x7fcac4141ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:35.375 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.372+0000 7fcaca400640 1 -- 172.21.15.96:0/1922866591 >> 172.21.15.96:0/1922866591 conn(0x7fcac40b1fc0 msgr2=0x7fcac4157cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:35.375 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.372+0000 7fcaca400640 1 -- 172.21.15.96:0/1922866591 shutdown_connections 2024-05-28T06:05:35.375 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.372+0000 7fcaca400640 1 -- 172.21.15.96:0/1922866591 wait complete. 2024-05-28T06:05:35.392 INFO:teuthology.orchestra.run.smithi096.stdout:{"pg_ready":true,"pg_map":{"version":73,"stamp":"2024-05-28T06:05:33.510905+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":4,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219960,"kb_used_data":3796,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754760,"statfs":{"total":773094113280,"available":772868874240,"internally_reserved":0,"allocated":3887104,"data_stored":2205195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.090214"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.445439+0000","last_change":"2024-05-28T06:05:29.445439+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.445439+0000","last_undegraded":"2024-05-28T06:05:29.445439+0000","last_fullsized":"2024-05-28T06:05:29.445439+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.452319+0000","last_change":"2024-05-28T06:05:29.452319+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.452319+0000","last_undegraded":"2024-05-28T06:05:29.452319+0000","last_fullsized":"2024-05-28T06:05:29.452319+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.455274+0000","last_change":"2024-05-28T06:05:29.455274+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.455274+0000","last_undegraded":"2024-05-28T06:05:29.455274+0000","last_fullsized":"2024-05-28T06:05:29.455274+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.445974+0000","last_change":"2024-05-28T06:05:29.445974+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.445974+0000","last_undegraded":"2024-05-28T06:05:29.445974+0000","last_fullsized":"2024-05-28T06:05:29.445974+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.442069+0000","last_change":"2024-05-28T06:05:29.442069+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.442069+0000","last_undegraded":"2024-05-28T06:05:29.442069+0000","last_fullsized":"2024-05-28T06:05:29.442069+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.453076+0000","last_change":"2024-05-28T06:05:29.453076+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.453076+0000","last_undegraded":"2024-05-28T06:05:29.453076+0000","last_fullsized":"2024-05-28T06:05:29.453076+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.452478+0000","last_change":"2024-05-28T06:05:29.452478+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.452478+0000","last_undegraded":"2024-05-28T06:05:29.452478+0000","last_fullsized":"2024-05-28T06:05:29.452478+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.455664+0000","last_change":"2024-05-28T06:05:29.455664+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.455664+0000","last_undegraded":"2024-05-28T06:05:29.455664+0000","last_fullsized":"2024-05-28T06:05:29.455664+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.434352+0000","last_change":"2024-05-28T06:05:29.434352+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.434352+0000","last_undegraded":"2024-05-28T06:05:29.434352+0000","last_fullsized":"2024-05-28T06:05:29.434352+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.444010+0000","last_change":"2024-05-28T06:05:29.444010+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.444010+0000","last_undegraded":"2024-05-28T06:05:29.444010+0000","last_fullsized":"2024-05-28T06:05:29.444010+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.453472+0000","last_change":"2024-05-28T06:05:29.453472+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.453472+0000","last_undegraded":"2024-05-28T06:05:29.453472+0000","last_fullsized":"2024-05-28T06:05:29.453472+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.456491+0000","last_change":"2024-05-28T06:05:29.456491+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.456491+0000","last_undegraded":"2024-05-28T06:05:29.456491+0000","last_fullsized":"2024-05-28T06:05:29.456491+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.439713+0000","last_change":"2024-05-28T06:05:29.439713+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.439713+0000","last_undegraded":"2024-05-28T06:05:29.439713+0000","last_fullsized":"2024-05-28T06:05:29.439713+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.452561+0000","last_change":"2024-05-28T06:05:29.452561+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.452561+0000","last_undegraded":"2024-05-28T06:05:29.452561+0000","last_fullsized":"2024-05-28T06:05:29.452561+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.456977+0000","last_change":"2024-05-28T06:05:29.456977+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.456977+0000","last_undegraded":"2024-05-28T06:05:29.456977+0000","last_fullsized":"2024-05-28T06:05:29.456977+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.452961+0000","last_change":"2024-05-28T06:05:29.452961+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.452961+0000","last_undegraded":"2024-05-28T06:05:29.452961+0000","last_fullsized":"2024-05-28T06:05:29.452961+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.452936+0000","last_change":"2024-05-28T06:05:29.452936+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.452936+0000","last_undegraded":"2024-05-28T06:05:29.452936+0000","last_fullsized":"2024-05-28T06:05:29.452936+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.451797+0000","last_change":"2024-05-28T06:05:29.451797+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.451797+0000","last_undegraded":"2024-05-28T06:05:29.451797+0000","last_fullsized":"2024-05-28T06:05:29.451797+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.447932+0000","last_change":"2024-05-28T06:05:29.447932+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.447932+0000","last_undegraded":"2024-05-28T06:05:29.447932+0000","last_fullsized":"2024-05-28T06:05:29.447932+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.437958+0000","last_change":"2024-05-28T06:05:29.437958+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.437958+0000","last_undegraded":"2024-05-28T06:05:29.437958+0000","last_fullsized":"2024-05-28T06:05:29.437958+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.436112+0000","last_change":"2024-05-28T06:05:29.436112+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.436112+0000","last_undegraded":"2024-05-28T06:05:29.436112+0000","last_fullsized":"2024-05-28T06:05:29.436112+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.432539+0000","last_change":"2024-05-28T06:05:29.432539+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.432539+0000","last_undegraded":"2024-05-28T06:05:29.432539+0000","last_fullsized":"2024-05-28T06:05:29.432539+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.440180+0000","last_change":"2024-05-28T06:05:29.440180+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.440180+0000","last_undegraded":"2024-05-28T06:05:29.440180+0000","last_fullsized":"2024-05-28T06:05:29.440180+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.454693+0000","last_change":"2024-05-28T06:05:29.454693+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.454693+0000","last_undegraded":"2024-05-28T06:05:29.454693+0000","last_fullsized":"2024-05-28T06:05:29.454693+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.446445+0000","last_change":"2024-05-28T06:05:29.446445+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.446445+0000","last_undegraded":"2024-05-28T06:05:29.446445+0000","last_fullsized":"2024-05-28T06:05:29.446445+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.451346+0000","last_change":"2024-05-28T06:05:29.451346+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.451346+0000","last_undegraded":"2024-05-28T06:05:29.451346+0000","last_fullsized":"2024-05-28T06:05:29.451346+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.444042+0000","last_change":"2024-05-28T06:05:29.444042+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.444042+0000","last_undegraded":"2024-05-28T06:05:29.444042+0000","last_fullsized":"2024-05-28T06:05:29.444042+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.455928+0000","last_change":"2024-05-28T06:05:29.455928+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.455928+0000","last_undegraded":"2024-05-28T06:05:29.455928+0000","last_fullsized":"2024-05-28T06:05:29.455928+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.434836+0000","last_change":"2024-05-28T06:05:29.434836+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.434836+0000","last_undegraded":"2024-05-28T06:05:29.434836+0000","last_fullsized":"2024-05-28T06:05:29.434836+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.454892+0000","last_change":"2024-05-28T06:05:29.454892+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.454892+0000","last_undegraded":"2024-05-28T06:05:29.454892+0000","last_fullsized":"2024-05-28T06:05:29.454892+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.446998+0000","last_change":"2024-05-28T06:05:29.446998+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.446998+0000","last_undegraded":"2024-05-28T06:05:29.446998+0000","last_fullsized":"2024-05-28T06:05:29.446998+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.455194+0000","last_change":"2024-05-28T06:05:29.455194+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.455194+0000","last_undegraded":"2024-05-28T06:05:29.455194+0000","last_fullsized":"2024-05-28T06:05:29.455194+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.438448+0000","last_change":"2024-05-28T06:05:29.438448+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.438448+0000","last_undegraded":"2024-05-28T06:05:29.438448+0000","last_fullsized":"2024-05-28T06:05:29.438448+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-28T06:05:29.477464+0000","last_change":"2024-05-28T06:05:29.474199+0000","last_active":"2024-05-28T06:05:29.477464+0000","last_peered":"2024-05-28T06:05:29.477464+0000","last_clean":"2024-05-28T06:05:29.477464+0000","last_became_active":"2024-05-28T06:05:29.473863+0000","last_became_peered":"2024-05-28T06:05:29.473863+0000","last_unstale":"2024-05-28T06:05:29.477464+0000","last_undegraded":"2024-05-28T06:05:29.477464+0000","last_fullsized":"2024-05-28T06:05:29.477464+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:30:20.399303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.443791+0000","last_change":"2024-05-28T06:05:29.443791+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.443791+0000","last_undegraded":"2024-05-28T06:05:29.443791+0000","last_fullsized":"2024-05-28T06:05:29.443791+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-28T06:05:29.462994+0000","last_change":"2024-05-28T06:05:29.460672+0000","last_active":"2024-05-28T06:05:29.462994+0000","last_peered":"2024-05-28T06:05:29.462994+0000","last_clean":"2024-05-28T06:05:29.462994+0000","last_became_active":"2024-05-28T06:05:29.459991+0000","last_became_peered":"2024-05-28T06:05:29.459991+0000","last_unstale":"2024-05-28T06:05:29.462994+0000","last_undegraded":"2024-05-28T06:05:29.462994+0000","last_fullsized":"2024-05-28T06:05:29.462994+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:05:43.269097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.433757+0000","last_change":"2024-05-28T06:05:29.433757+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.433757+0000","last_undegraded":"2024-05-28T06:05:29.433757+0000","last_fullsized":"2024-05-28T06:05:29.433757+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-28T06:05:29.461037+0000","last_change":"2024-05-28T06:05:29.456735+0000","last_active":"2024-05-28T06:05:29.461037+0000","last_peered":"2024-05-28T06:05:29.461037+0000","last_clean":"2024-05-28T06:05:29.461037+0000","last_became_active":"2024-05-28T06:05:29.456466+0000","last_became_peered":"2024-05-28T06:05:29.456466+0000","last_unstale":"2024-05-28T06:05:29.461037+0000","last_undegraded":"2024-05-28T06:05:29.461037+0000","last_fullsized":"2024-05-28T06:05:29.461037+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:47:21.316456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.443049+0000","last_change":"2024-05-28T06:05:29.443049+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.443049+0000","last_undegraded":"2024-05-28T06:05:29.443049+0000","last_fullsized":"2024-05-28T06:05:29.443049+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-28T06:05:29.455760+0000","last_change":"2024-05-28T06:05:29.451693+0000","last_active":"2024-05-28T06:05:29.455760+0000","last_peered":"2024-05-28T06:05:29.455760+0000","last_clean":"2024-05-28T06:05:29.455760+0000","last_became_active":"2024-05-28T06:05:29.451343+0000","last_became_peered":"2024-05-28T06:05:29.451343+0000","last_unstale":"2024-05-28T06:05:29.455760+0000","last_undegraded":"2024-05-28T06:05:29.455760+0000","last_fullsized":"2024-05-28T06:05:29.455760+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:42:38.699538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.454003+0000","last_change":"2024-05-28T06:05:29.454003+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.454003+0000","last_undegraded":"2024-05-28T06:05:29.454003+0000","last_fullsized":"2024-05-28T06:05:29.454003+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-28T06:05:29.469115+0000","last_change":"2024-05-28T06:05:29.463758+0000","last_active":"2024-05-28T06:05:29.469115+0000","last_peered":"2024-05-28T06:05:29.469115+0000","last_clean":"2024-05-28T06:05:29.469115+0000","last_became_active":"2024-05-28T06:05:29.460297+0000","last_became_peered":"2024-05-28T06:05:29.460297+0000","last_unstale":"2024-05-28T06:05:29.469115+0000","last_undegraded":"2024-05-28T06:05:29.469115+0000","last_fullsized":"2024-05-28T06:05:29.469115+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:45:19.781181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.455037+0000","last_change":"2024-05-28T06:05:29.455037+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.455037+0000","last_undegraded":"2024-05-28T06:05:29.455037+0000","last_fullsized":"2024-05-28T06:05:29.455037+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-28T06:05:29.468155+0000","last_change":"2024-05-28T06:05:29.462024+0000","last_active":"2024-05-28T06:05:29.468155+0000","last_peered":"2024-05-28T06:05:29.468155+0000","last_clean":"2024-05-28T06:05:29.468155+0000","last_became_active":"2024-05-28T06:05:29.460033+0000","last_became_peered":"2024-05-28T06:05:29.460033+0000","last_unstale":"2024-05-28T06:05:29.468155+0000","last_undegraded":"2024-05-28T06:05:29.468155+0000","last_fullsized":"2024-05-28T06:05:29.468155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:43:59.184224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.432275+0000","last_change":"2024-05-28T06:05:29.432275+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.432275+0000","last_undegraded":"2024-05-28T06:05:29.432275+0000","last_fullsized":"2024-05-28T06:05:29.432275+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-28T06:05:29.459996+0000","last_change":"2024-05-28T06:05:29.456433+0000","last_active":"2024-05-28T06:05:29.459996+0000","last_peered":"2024-05-28T06:05:29.459996+0000","last_clean":"2024-05-28T06:05:29.459996+0000","last_became_active":"2024-05-28T06:05:29.456113+0000","last_became_peered":"2024-05-28T06:05:29.456113+0000","last_unstale":"2024-05-28T06:05:29.459996+0000","last_undegraded":"2024-05-28T06:05:29.459996+0000","last_fullsized":"2024-05-28T06:05:29.459996+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:10:48.750962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.445350+0000","last_change":"2024-05-28T06:05:29.445350+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.445350+0000","last_undegraded":"2024-05-28T06:05:29.445350+0000","last_fullsized":"2024-05-28T06:05:29.445350+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-28T06:05:29.468715+0000","last_change":"2024-05-28T06:05:29.462393+0000","last_active":"2024-05-28T06:05:29.468715+0000","last_peered":"2024-05-28T06:05:29.468715+0000","last_clean":"2024-05-28T06:05:29.468715+0000","last_became_active":"2024-05-28T06:05:29.461785+0000","last_became_peered":"2024-05-28T06:05:29.461785+0000","last_unstale":"2024-05-28T06:05:29.468715+0000","last_undegraded":"2024-05-28T06:05:29.468715+0000","last_fullsized":"2024-05-28T06:05:29.468715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:42:15.256795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.458171+0000","last_change":"2024-05-28T06:05:29.458171+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.458171+0000","last_undegraded":"2024-05-28T06:05:29.458171+0000","last_fullsized":"2024-05-28T06:05:29.458171+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-28T06:05:29.463385+0000","last_change":"2024-05-28T06:05:29.461808+0000","last_active":"2024-05-28T06:05:29.463385+0000","last_peered":"2024-05-28T06:05:29.463385+0000","last_clean":"2024-05-28T06:05:29.463385+0000","last_became_active":"2024-05-28T06:05:29.461481+0000","last_became_peered":"2024-05-28T06:05:29.461481+0000","last_unstale":"2024-05-28T06:05:29.463385+0000","last_undegraded":"2024-05-28T06:05:29.463385+0000","last_fullsized":"2024-05-28T06:05:29.463385+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:32:34.183989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.442237+0000","last_change":"2024-05-28T06:05:29.442237+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.442237+0000","last_undegraded":"2024-05-28T06:05:29.442237+0000","last_fullsized":"2024-05-28T06:05:29.442237+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-28T06:05:29.463015+0000","last_change":"2024-05-28T06:05:29.460940+0000","last_active":"2024-05-28T06:05:29.463015+0000","last_peered":"2024-05-28T06:05:29.463015+0000","last_clean":"2024-05-28T06:05:29.463015+0000","last_became_active":"2024-05-28T06:05:29.460433+0000","last_became_peered":"2024-05-28T06:05:29.460433+0000","last_unstale":"2024-05-28T06:05:29.463015+0000","last_undegraded":"2024-05-28T06:05:29.463015+0000","last_fullsized":"2024-05-28T06:05:29.463015+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:30:20.399303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.452611+0000","last_change":"2024-05-28T06:05:29.452611+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.452611+0000","last_undegraded":"2024-05-28T06:05:29.452611+0000","last_fullsized":"2024-05-28T06:05:29.452611+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-28T06:05:29.469706+0000","last_change":"2024-05-28T06:05:29.462646+0000","last_active":"2024-05-28T06:05:29.469706+0000","last_peered":"2024-05-28T06:05:29.469706+0000","last_clean":"2024-05-28T06:05:29.469706+0000","last_became_active":"2024-05-28T06:05:29.462355+0000","last_became_peered":"2024-05-28T06:05:29.462355+0000","last_unstale":"2024-05-28T06:05:29.469706+0000","last_undegraded":"2024-05-28T06:05:29.469706+0000","last_fullsized":"2024-05-28T06:05:29.469706+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:12:52.117940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.455754+0000","last_change":"2024-05-28T06:05:29.455754+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.455754+0000","last_undegraded":"2024-05-28T06:05:29.455754+0000","last_fullsized":"2024-05-28T06:05:29.455754+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-28T06:05:29.460944+0000","last_change":"2024-05-28T06:05:29.458155+0000","last_active":"2024-05-28T06:05:29.460944+0000","last_peered":"2024-05-28T06:05:29.460944+0000","last_clean":"2024-05-28T06:05:29.460944+0000","last_became_active":"2024-05-28T06:05:29.457822+0000","last_became_peered":"2024-05-28T06:05:29.457822+0000","last_unstale":"2024-05-28T06:05:29.460944+0000","last_undegraded":"2024-05-28T06:05:29.460944+0000","last_fullsized":"2024-05-28T06:05:29.460944+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:12:38.485450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.442805+0000","last_change":"2024-05-28T06:05:29.442805+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.442805+0000","last_undegraded":"2024-05-28T06:05:29.442805+0000","last_fullsized":"2024-05-28T06:05:29.442805+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-28T06:05:29.469142+0000","last_change":"2024-05-28T06:05:29.464610+0000","last_active":"2024-05-28T06:05:29.469142+0000","last_peered":"2024-05-28T06:05:29.469142+0000","last_clean":"2024-05-28T06:05:29.469142+0000","last_became_active":"2024-05-28T06:05:29.464074+0000","last_became_peered":"2024-05-28T06:05:29.464074+0000","last_unstale":"2024-05-28T06:05:29.469142+0000","last_undegraded":"2024-05-28T06:05:29.469142+0000","last_fullsized":"2024-05-28T06:05:29.469142+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:31:54.521268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.453696+0000","last_change":"2024-05-28T06:05:29.453696+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.453696+0000","last_undegraded":"2024-05-28T06:05:29.453696+0000","last_fullsized":"2024-05-28T06:05:29.453696+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-28T06:05:29.464678+0000","last_change":"2024-05-28T06:05:29.463194+0000","last_active":"2024-05-28T06:05:29.464678+0000","last_peered":"2024-05-28T06:05:29.464678+0000","last_clean":"2024-05-28T06:05:29.464678+0000","last_became_active":"2024-05-28T06:05:29.462870+0000","last_became_peered":"2024-05-28T06:05:29.462870+0000","last_unstale":"2024-05-28T06:05:29.464678+0000","last_undegraded":"2024-05-28T06:05:29.464678+0000","last_fullsized":"2024-05-28T06:05:29.464678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:30:20.399303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.454681+0000","last_change":"2024-05-28T06:05:29.454681+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.454681+0000","last_undegraded":"2024-05-28T06:05:29.454681+0000","last_fullsized":"2024-05-28T06:05:29.454681+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-28T06:05:29.463771+0000","last_change":"2024-05-28T06:05:29.462298+0000","last_active":"2024-05-28T06:05:29.463771+0000","last_peered":"2024-05-28T06:05:29.463771+0000","last_clean":"2024-05-28T06:05:29.463771+0000","last_became_active":"2024-05-28T06:05:29.462022+0000","last_became_peered":"2024-05-28T06:05:29.462022+0000","last_unstale":"2024-05-28T06:05:29.463771+0000","last_undegraded":"2024-05-28T06:05:29.463771+0000","last_fullsized":"2024-05-28T06:05:29.463771+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:11:47.259186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.441893+0000","last_change":"2024-05-28T06:05:29.441893+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.441893+0000","last_undegraded":"2024-05-28T06:05:29.441893+0000","last_fullsized":"2024-05-28T06:05:29.441893+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-28T06:05:29.463784+0000","last_change":"2024-05-28T06:05:29.460103+0000","last_active":"2024-05-28T06:05:29.463784+0000","last_peered":"2024-05-28T06:05:29.463784+0000","last_clean":"2024-05-28T06:05:29.463784+0000","last_became_active":"2024-05-28T06:05:29.459788+0000","last_became_peered":"2024-05-28T06:05:29.459788+0000","last_unstale":"2024-05-28T06:05:29.463784+0000","last_undegraded":"2024-05-28T06:05:29.463784+0000","last_fullsized":"2024-05-28T06:05:29.463784+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:57:51.752948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.434191+0000","last_change":"2024-05-28T06:05:29.434191+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.434191+0000","last_undegraded":"2024-05-28T06:05:29.434191+0000","last_fullsized":"2024-05-28T06:05:29.434191+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-28T06:05:29.460988+0000","last_change":"2024-05-28T06:05:29.458728+0000","last_active":"2024-05-28T06:05:29.460988+0000","last_peered":"2024-05-28T06:05:29.460988+0000","last_clean":"2024-05-28T06:05:29.460988+0000","last_became_active":"2024-05-28T06:05:29.458154+0000","last_became_peered":"2024-05-28T06:05:29.458154+0000","last_unstale":"2024-05-28T06:05:29.460988+0000","last_undegraded":"2024-05-28T06:05:29.460988+0000","last_fullsized":"2024-05-28T06:05:29.460988+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:05:43.269097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.451184+0000","last_change":"2024-05-28T06:05:29.451184+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.451184+0000","last_undegraded":"2024-05-28T06:05:29.451184+0000","last_fullsized":"2024-05-28T06:05:29.451184+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-28T06:05:29.469005+0000","last_change":"2024-05-28T06:05:29.463122+0000","last_active":"2024-05-28T06:05:29.469005+0000","last_peered":"2024-05-28T06:05:29.469005+0000","last_clean":"2024-05-28T06:05:29.469005+0000","last_became_active":"2024-05-28T06:05:29.462362+0000","last_became_peered":"2024-05-28T06:05:29.462362+0000","last_unstale":"2024-05-28T06:05:29.469005+0000","last_undegraded":"2024-05-28T06:05:29.469005+0000","last_fullsized":"2024-05-28T06:05:29.469005+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:01:56.755222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.442315+0000","last_change":"2024-05-28T06:05:29.442315+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.442315+0000","last_undegraded":"2024-05-28T06:05:29.442315+0000","last_fullsized":"2024-05-28T06:05:29.442315+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-28T06:05:29.459230+0000","last_change":"2024-05-28T06:05:29.450660+0000","last_active":"2024-05-28T06:05:29.459230+0000","last_peered":"2024-05-28T06:05:29.459230+0000","last_clean":"2024-05-28T06:05:29.459230+0000","last_became_active":"2024-05-28T06:05:29.446547+0000","last_became_peered":"2024-05-28T06:05:29.446547+0000","last_unstale":"2024-05-28T06:05:29.459230+0000","last_undegraded":"2024-05-28T06:05:29.459230+0000","last_fullsized":"2024-05-28T06:05:29.459230+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:42:38.699538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.437631+0000","last_change":"2024-05-28T06:05:29.437631+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.437631+0000","last_undegraded":"2024-05-28T06:05:29.437631+0000","last_fullsized":"2024-05-28T06:05:29.437631+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-28T06:05:29.468175+0000","last_change":"2024-05-28T06:05:29.461213+0000","last_active":"2024-05-28T06:05:29.468175+0000","last_peered":"2024-05-28T06:05:29.468175+0000","last_clean":"2024-05-28T06:05:29.468175+0000","last_became_active":"2024-05-28T06:05:29.460899+0000","last_became_peered":"2024-05-28T06:05:29.460899+0000","last_unstale":"2024-05-28T06:05:29.468175+0000","last_undegraded":"2024-05-28T06:05:29.468175+0000","last_fullsized":"2024-05-28T06:05:29.468175+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:07:14.772596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.436257+0000","last_change":"2024-05-28T06:05:29.436257+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.436257+0000","last_undegraded":"2024-05-28T06:05:29.436257+0000","last_fullsized":"2024-05-28T06:05:29.436257+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-28T06:05:29.452420+0000","last_change":"2024-05-28T06:05:29.447607+0000","last_active":"2024-05-28T06:05:29.452420+0000","last_peered":"2024-05-28T06:05:29.452420+0000","last_clean":"2024-05-28T06:05:29.452420+0000","last_became_active":"2024-05-28T06:05:29.447321+0000","last_became_peered":"2024-05-28T06:05:29.447321+0000","last_unstale":"2024-05-28T06:05:29.452420+0000","last_undegraded":"2024-05-28T06:05:29.452420+0000","last_fullsized":"2024-05-28T06:05:29.452420+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:22:38.584903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.437966+0000","last_change":"2024-05-28T06:05:29.437966+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.437966+0000","last_undegraded":"2024-05-28T06:05:29.437966+0000","last_fullsized":"2024-05-28T06:05:29.437966+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-28T06:05:29.470438+0000","last_change":"2024-05-28T06:05:29.463849+0000","last_active":"2024-05-28T06:05:29.470438+0000","last_peered":"2024-05-28T06:05:29.470438+0000","last_clean":"2024-05-28T06:05:29.470438+0000","last_became_active":"2024-05-28T06:05:29.463514+0000","last_became_peered":"2024-05-28T06:05:29.463514+0000","last_unstale":"2024-05-28T06:05:29.470438+0000","last_undegraded":"2024-05-28T06:05:29.470438+0000","last_fullsized":"2024-05-28T06:05:29.470438+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:05:43.269097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.436238+0000","last_change":"2024-05-28T06:05:29.436238+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.436238+0000","last_undegraded":"2024-05-28T06:05:29.436238+0000","last_fullsized":"2024-05-28T06:05:29.436238+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-28T06:05:29.457491+0000","last_change":"2024-05-28T06:05:29.450109+0000","last_active":"2024-05-28T06:05:29.457491+0000","last_peered":"2024-05-28T06:05:29.457491+0000","last_clean":"2024-05-28T06:05:29.457491+0000","last_became_active":"2024-05-28T06:05:29.449618+0000","last_became_peered":"2024-05-28T06:05:29.449618+0000","last_unstale":"2024-05-28T06:05:29.457491+0000","last_undegraded":"2024-05-28T06:05:29.457491+0000","last_fullsized":"2024-05-28T06:05:29.457491+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:09:47.123693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.453182+0000","last_change":"2024-05-28T06:05:29.453182+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.453182+0000","last_undegraded":"2024-05-28T06:05:29.453182+0000","last_fullsized":"2024-05-28T06:05:29.453182+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-28T06:05:29.471119+0000","last_change":"2024-05-28T06:05:29.467769+0000","last_active":"2024-05-28T06:05:29.471119+0000","last_peered":"2024-05-28T06:05:29.471119+0000","last_clean":"2024-05-28T06:05:29.471119+0000","last_became_active":"2024-05-28T06:05:29.467422+0000","last_became_peered":"2024-05-28T06:05:29.467422+0000","last_unstale":"2024-05-28T06:05:29.471119+0000","last_undegraded":"2024-05-28T06:05:29.471119+0000","last_fullsized":"2024-05-28T06:05:29.471119+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:33:35.470452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.436688+0000","last_change":"2024-05-28T06:05:29.436688+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.436688+0000","last_undegraded":"2024-05-28T06:05:29.436688+0000","last_fullsized":"2024-05-28T06:05:29.436688+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-28T06:05:29.472543+0000","last_change":"2024-05-28T06:05:29.471119+0000","last_active":"2024-05-28T06:05:29.472543+0000","last_peered":"2024-05-28T06:05:29.472543+0000","last_clean":"2024-05-28T06:05:29.472543+0000","last_became_active":"2024-05-28T06:05:29.470884+0000","last_became_peered":"2024-05-28T06:05:29.470884+0000","last_unstale":"2024-05-28T06:05:29.472543+0000","last_undegraded":"2024-05-28T06:05:29.472543+0000","last_fullsized":"2024-05-28T06:05:29.472543+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:47:21.316456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.448476+0000","last_change":"2024-05-28T06:05:29.448476+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.448476+0000","last_undegraded":"2024-05-28T06:05:29.448476+0000","last_fullsized":"2024-05-28T06:05:29.448476+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-28T06:05:29.452484+0000","last_change":"2024-05-28T06:05:29.449898+0000","last_active":"2024-05-28T06:05:29.452484+0000","last_peered":"2024-05-28T06:05:29.452484+0000","last_clean":"2024-05-28T06:05:29.452484+0000","last_became_active":"2024-05-28T06:05:29.449322+0000","last_became_peered":"2024-05-28T06:05:29.449322+0000","last_unstale":"2024-05-28T06:05:29.452484+0000","last_undegraded":"2024-05-28T06:05:29.452484+0000","last_fullsized":"2024-05-28T06:05:29.452484+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:16:42.579003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.439531+0000","last_change":"2024-05-28T06:05:29.439531+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.439531+0000","last_undegraded":"2024-05-28T06:05:29.439531+0000","last_fullsized":"2024-05-28T06:05:29.439531+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-28T06:05:29.457020+0000","last_change":"2024-05-28T06:05:29.451001+0000","last_active":"2024-05-28T06:05:29.457020+0000","last_peered":"2024-05-28T06:05:29.457020+0000","last_clean":"2024-05-28T06:05:29.457020+0000","last_became_active":"2024-05-28T06:05:29.450717+0000","last_became_peered":"2024-05-28T06:05:29.450717+0000","last_unstale":"2024-05-28T06:05:29.457020+0000","last_undegraded":"2024-05-28T06:05:29.457020+0000","last_fullsized":"2024-05-28T06:05:29.457020+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:16:42.579003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.448185+0000","last_change":"2024-05-28T06:05:29.448185+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.448185+0000","last_undegraded":"2024-05-28T06:05:29.448185+0000","last_fullsized":"2024-05-28T06:05:29.448185+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-28T06:05:29.470603+0000","last_change":"2024-05-28T06:05:29.469461+0000","last_active":"2024-05-28T06:05:29.470603+0000","last_peered":"2024-05-28T06:05:29.470603+0000","last_clean":"2024-05-28T06:05:29.470603+0000","last_became_active":"2024-05-28T06:05:29.469114+0000","last_became_peered":"2024-05-28T06:05:29.469114+0000","last_unstale":"2024-05-28T06:05:29.470603+0000","last_undegraded":"2024-05-28T06:05:29.470603+0000","last_fullsized":"2024-05-28T06:05:29.470603+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:10:48.750962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.438458+0000","last_change":"2024-05-28T06:05:29.438458+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.438458+0000","last_undegraded":"2024-05-28T06:05:29.438458+0000","last_fullsized":"2024-05-28T06:05:29.438458+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-28T06:05:29.470697+0000","last_change":"2024-05-28T06:05:29.469913+0000","last_active":"2024-05-28T06:05:29.470697+0000","last_peered":"2024-05-28T06:05:29.470697+0000","last_clean":"2024-05-28T06:05:29.470697+0000","last_became_active":"2024-05-28T06:05:29.469460+0000","last_became_peered":"2024-05-28T06:05:29.469460+0000","last_unstale":"2024-05-28T06:05:29.470697+0000","last_undegraded":"2024-05-28T06:05:29.470697+0000","last_fullsized":"2024-05-28T06:05:29.470697+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:10:48.750962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.444841+0000","last_change":"2024-05-28T06:05:29.444841+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.444841+0000","last_undegraded":"2024-05-28T06:05:29.444841+0000","last_fullsized":"2024-05-28T06:05:29.444841+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-28T06:05:29.471575+0000","last_change":"2024-05-28T06:05:29.469678+0000","last_active":"2024-05-28T06:05:29.471575+0000","last_peered":"2024-05-28T06:05:29.471575+0000","last_clean":"2024-05-28T06:05:29.471575+0000","last_became_active":"2024-05-28T06:05:29.469209+0000","last_became_peered":"2024-05-28T06:05:29.469209+0000","last_unstale":"2024-05-28T06:05:29.471575+0000","last_undegraded":"2024-05-28T06:05:29.471575+0000","last_fullsized":"2024-05-28T06:05:29.471575+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:37:09.560235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.433299+0000","last_change":"2024-05-28T06:05:29.433299+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.433299+0000","last_undegraded":"2024-05-28T06:05:29.433299+0000","last_fullsized":"2024-05-28T06:05:29.433299+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-28T06:05:29.467257+0000","last_change":"2024-05-28T06:05:29.465063+0000","last_active":"2024-05-28T06:05:29.467257+0000","last_peered":"2024-05-28T06:05:29.467257+0000","last_clean":"2024-05-28T06:05:29.467257+0000","last_became_active":"2024-05-28T06:05:29.464579+0000","last_became_peered":"2024-05-28T06:05:29.464579+0000","last_unstale":"2024-05-28T06:05:29.467257+0000","last_undegraded":"2024-05-28T06:05:29.467257+0000","last_fullsized":"2024-05-28T06:05:29.467257+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:58:31.330922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-28T06:05:29.451026+0000","last_change":"2024-05-28T06:05:29.451026+0000","last_active":"2024-05-28T06:05:29.416642+0000","last_peered":"2024-05-28T06:05:29.416642+0000","last_clean":"2024-05-28T06:05:29.416642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:05:29.451026+0000","last_undegraded":"2024-05-28T06:05:29.451026+0000","last_fullsized":"2024-05-28T06:05:29.451026+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-28T06:05:29.465703+0000","last_change":"2024-05-28T06:05:29.461542+0000","last_active":"2024-05-28T06:05:29.465703+0000","last_peered":"2024-05-28T06:05:29.465703+0000","last_clean":"2024-05-28T06:05:29.465703+0000","last_became_active":"2024-05-28T06:05:29.461155+0000","last_became_peered":"2024-05-28T06:05:29.461155+0000","last_unstale":"2024-05-28T06:05:29.465703+0000","last_undegraded":"2024-05-28T06:05:29.465703+0000","last_fullsized":"2024-05-28T06:05:29.465703+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:16:42.579003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":190,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-28T06:05:29.456002+0000","last_change":"2024-05-28T06:04:01.039129+0000","last_active":"2024-05-28T06:05:29.456002+0000","last_peered":"2024-05-28T06:05:29.456002+0000","last_clean":"2024-05-28T06:05:29.456002+0000","last_became_active":"2024-05-28T06:04:01.033906+0000","last_became_peered":"2024-05-28T06:04:01.033906+0000","last_unstale":"2024-05-28T06:05:29.456002+0000","last_undegraded":"2024-05-28T06:05:29.456002+0000","last_fullsized":"2024-05-28T06:05:29.456002+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:03:59.864237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:03:59.864237+0000","last_clean_scrub_stamp":"2024-05-28T06:03:59.864237+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:17:36.440245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574869,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27928,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343912,"statfs":{"total":96636764160,"available":96608165888,"internally_reserved":0,"allocated":929792,"data_stored":717642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574871,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":348160,"data_stored":133541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574869,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":335872,"data_stored":127274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574869,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":335872,"data_stored":127274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574871,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":335872,"data_stored":127274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574870,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":335872,"data_stored":127274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574870,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":335872,"data_stored":127274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574869,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27928,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343912,"statfs":{"total":96636764160,"available":96608165888,"internally_reserved":0,"allocated":929792,"data_stored":717642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-28T06:05:35.396 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-28T06:05:35.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.642+0000 7f08aba00640 1 Processor -- start 2024-05-28T06:05:35.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.642+0000 7f08aba00640 1 -- start start 2024-05-28T06:05:35.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.643+0000 7f08aba00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f08a40b4b60 0x7f08a40cfaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:35.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.643+0000 7f08aba00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f08a40b3b70 0x7f08a40d0930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:35.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.643+0000 7f08aba00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08a40ced40 con 0x7f08a40d04a0 2024-05-28T06:05:35.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.643+0000 7f08aba00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f08a40b3f90 con 0x7f08a40b3b70 2024-05-28T06:05:35.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.643+0000 7f08aba00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f08a415fbb0 con 0x7f08a40b4b60 2024-05-28T06:05:35.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.644+0000 7f08a3e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f08a40d04a0 0x7f08a40b3460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:40298/0 (socket says 172.21.15.96:40298) 2024-05-28T06:05:35.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.644+0000 7f08a9a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f08a40b3b70 0x7f08a40d0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:35.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.644+0000 7f08a3e00640 1 -- 172.21.15.96:0/2158730393 learned_addr learned my addr 172.21.15.96:0/2158730393 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:05:35.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.644+0000 7f08a9000640 1 --2- 172.21.15.96:0/2158730393 >> v2:172.21.15.123:3300/0 conn(0x7f08a40b4b60 0x7f08a40cfaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:35.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.644+0000 7f08a9a00640 1 -- 172.21.15.96:0/2158730393 >> v2:172.21.15.123:3300/0 conn(0x7f08a40b4b60 msgr2=0x7f08a40cfaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:35.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.644+0000 7f08a9a00640 1 --2- 172.21.15.96:0/2158730393 >> v2:172.21.15.123:3300/0 conn(0x7f08a40b4b60 0x7f08a40cfaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:35.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.644+0000 7f08a9a00640 1 -- 172.21.15.96:0/2158730393 >> v1:172.21.15.96:6789/0 conn(0x7f08a40d04a0 legacy=0x7f08a40b3460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:35.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.644+0000 7f08a9a00640 1 -- 172.21.15.96:0/2158730393 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08a40cefc0 con 0x7f08a40b3b70 2024-05-28T06:05:35.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.644+0000 7f08a3400640 1 -- 172.21.15.96:0/2158730393 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 596869306 0 0) 0x7f088c001110 con 0x7f08a40d04a0 2024-05-28T06:05:35.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.644+0000 7f08a9000640 1 --2- 172.21.15.96:0/2158730393 >> v2:172.21.15.123:3300/0 conn(0x7f08a40b4b60 0x7f08a40cfaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-28T06:05:35.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.645+0000 7f08a9a00640 1 --2- 172.21.15.96:0/2158730393 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f08a40b3b70 0x7f08a40d0930 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f08a40cdd80 tx=0x7f089c02a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbf99e8ab24c1540 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:35.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.646+0000 7f08a3400640 1 -- 172.21.15.96:0/2158730393 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f089c007740 con 0x7f08a40b3b70 2024-05-28T06:05:35.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.646+0000 7f08a3400640 1 -- 172.21.15.96:0/2158730393 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f089c0078e0 con 0x7f08a40b3b70 2024-05-28T06:05:35.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.646+0000 7f08a3400640 1 -- 172.21.15.96:0/2158730393 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f089c007cb0 con 0x7f08a40b3b70 2024-05-28T06:05:35.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.646+0000 7f08aba00640 1 -- 172.21.15.96:0/2158730393 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f08a40b3b70 msgr2=0x7f08a40d0930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:35.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.646+0000 7f08aba00640 1 --2- 172.21.15.96:0/2158730393 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f08a40b3b70 0x7f08a40d0930 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f08a40cdd80 tx=0x7f089c02a680 comp rx=0 tx=0).stop 2024-05-28T06:05:35.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.646+0000 7f08aba00640 1 -- 172.21.15.96:0/2158730393 shutdown_connections 2024-05-28T06:05:35.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.646+0000 7f08aba00640 1 --2- 172.21.15.96:0/2158730393 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f08a40b3b70 0x7f08a40d0930 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:35.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.646+0000 7f08aba00640 1 --2- 172.21.15.96:0/2158730393 >> v2:172.21.15.123:3300/0 conn(0x7f08a40b4b60 0x7f08a40cfaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:35.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.646+0000 7f08aba00640 1 -- 172.21.15.96:0/2158730393 >> 172.21.15.96:0/2158730393 conn(0x7f08a406bae0 msgr2=0x7f08a406bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:35.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.646+0000 7f08aba00640 1 -- 172.21.15.96:0/2158730393 shutdown_connections 2024-05-28T06:05:35.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.646+0000 7f08aba00640 1 -- 172.21.15.96:0/2158730393 wait complete. 2024-05-28T06:05:35.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.648+0000 7f08aba00640 1 Processor -- start 2024-05-28T06:05:35.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.648+0000 7f08aba00640 1 -- start start 2024-05-28T06:05:35.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.649+0000 7f08aba00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f08a40b3b70 0x7f08a415b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:35.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.649+0000 7f08aba00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08a40b4b60 0x7f08a41f7660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:35.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.649+0000 7f08aba00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f08a40d04a0 0x7f08a4073630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:35.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.649+0000 7f08aba00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f08a40cffe0 con 0x7f08a40b3b70 2024-05-28T06:05:35.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.649+0000 7f08aba00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f08a40cf580 con 0x7f08a40b4b60 2024-05-28T06:05:35.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.649+0000 7f08aba00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f08a415ede0 con 0x7f08a40d04a0 2024-05-28T06:05:35.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.650+0000 7f08a9000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f08a40b3b70 0x7f08a415b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:35.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.650+0000 7f08a9000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f08a40b3b70 0x7f08a415b2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:50910/0 (socket says 172.21.15.96:50910) 2024-05-28T06:05:35.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.650+0000 7f08a9000640 1 -- 172.21.15.96:0/1816209497 learned_addr learned my addr 172.21.15.96:0/1816209497 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:35.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.650+0000 7f08a3e00640 1 --2- 172.21.15.96:0/1816209497 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08a40b4b60 0x7f08a41f7660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:35.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.650+0000 7f08a9a00640 1 --2- 172.21.15.96:0/1816209497 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f08a40d04a0 0x7f08a4073630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:35.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.650+0000 7f08a9000640 1 -- 172.21.15.96:0/1816209497 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f08a40d04a0 msgr2=0x7f08a4073630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:35.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.650+0000 7f08a9000640 1 --2- 172.21.15.96:0/1816209497 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f08a40d04a0 0x7f08a4073630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:35.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.650+0000 7f08a9000640 1 -- 172.21.15.96:0/1816209497 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08a40b4b60 msgr2=0x7f08a41f7660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:35.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.650+0000 7f08a9000640 1 --2- 172.21.15.96:0/1816209497 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08a40b4b60 0x7f08a41f7660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:35.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.650+0000 7f08a9000640 1 -- 172.21.15.96:0/1816209497 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08a415a230 con 0x7f08a40b3b70 2024-05-28T06:05:35.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.650+0000 7f08a9a00640 1 --2- 172.21.15.96:0/1816209497 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f08a40d04a0 0x7f08a4073630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:05:35.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.650+0000 7f08a9000640 1 --2- 172.21.15.96:0/1816209497 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f08a40b3b70 0x7f08a415b2d0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f089400e7e0 tx=0x7f089400eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:35.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.651+0000 7f08a0c00640 1 -- 172.21.15.96:0/1816209497 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f089400f040 con 0x7f08a40b3b70 2024-05-28T06:05:35.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.651+0000 7f08a0c00640 1 -- 172.21.15.96:0/1816209497 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0894004030 con 0x7f08a40b3b70 2024-05-28T06:05:35.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.651+0000 7f08a0c00640 1 -- 172.21.15.96:0/1816209497 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0894004400 con 0x7f08a40b3b70 2024-05-28T06:05:35.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.651+0000 7f08aba00640 1 -- 172.21.15.96:0/1816209497 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08a415a510 con 0x7f08a40b3b70 2024-05-28T06:05:35.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.651+0000 7f08aba00640 1 -- 172.21.15.96:0/1816209497 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08a415aad0 con 0x7f08a40b3b70 2024-05-28T06:05:35.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.652+0000 7f08aba00640 1 -- 172.21.15.96:0/1816209497 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08a40689e0 con 0x7f08a40b3b70 2024-05-28T06:05:35.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.653+0000 7f08a0c00640 1 -- 172.21.15.96:0/1816209497 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f089400c6d0 con 0x7f08a40b3b70 2024-05-28T06:05:35.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.654+0000 7f08a0c00640 1 --2- 172.21.15.96:0/1816209497 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f088006c8f0 0x7f088006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:35.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.654+0000 7f08a0c00640 1 -- 172.21.15.96:0/1816209497 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f089408b9d0 con 0x7f08a40b3b70 2024-05-28T06:05:35.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.655+0000 7f08a3e00640 1 --2- 172.21.15.96:0/1816209497 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f088006c8f0 0x7f088006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:35.657 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.656+0000 7f08a3e00640 1 --2- 172.21.15.96:0/1816209497 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f088006c8f0 0x7f088006edb0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f08a415fea0 tx=0x7f088c007c00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:35.659 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.658+0000 7f08a0c00640 1 -- 172.21.15.96:0/1816209497 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0894017020 con 0x7f08a40b3b70 2024-05-28T06:05:35.857 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.856+0000 7f08aba00640 1 -- 172.21.15.96:0/1816209497 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f08a40ad2d0 con 0x7f088006c8f0 2024-05-28T06:05:35.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.875+0000 7f08a0c00640 1 -- 172.21.15.96:0/1816209497 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251951 (secure 0 0 0) 0x7f08a40ad2d0 con 0x7f088006c8f0 2024-05-28T06:05:35.879 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:05:35.880 INFO:teuthology.orchestra.run.smithi096.stderr:dumped all 2024-05-28T06:05:35.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.879+0000 7f08aba00640 1 -- 172.21.15.96:0/1816209497 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f088006c8f0 msgr2=0x7f088006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:35.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.880+0000 7f08aba00640 1 --2- 172.21.15.96:0/1816209497 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f088006c8f0 0x7f088006edb0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f08a415fea0 tx=0x7f088c007c00 comp rx=0 tx=0).stop 2024-05-28T06:05:35.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.880+0000 7f08aba00640 1 -- 172.21.15.96:0/1816209497 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f08a40b3b70 msgr2=0x7f08a415b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:35.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.880+0000 7f08aba00640 1 --2- 172.21.15.96:0/1816209497 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f08a40b3b70 0x7f08a415b2d0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f089400e7e0 tx=0x7f089400eca0 comp rx=0 tx=0).stop 2024-05-28T06:05:35.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.880+0000 7f08aba00640 1 -- 172.21.15.96:0/1816209497 shutdown_connections 2024-05-28T06:05:35.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.880+0000 7f08aba00640 1 --2- 172.21.15.96:0/1816209497 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f088006c8f0 0x7f088006edb0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:35.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.880+0000 7f08aba00640 1 --2- 172.21.15.96:0/1816209497 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f08a40d04a0 0x7f08a4073630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:35.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.880+0000 7f08aba00640 1 --2- 172.21.15.96:0/1816209497 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08a40b4b60 0x7f08a41f7660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:35.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.880+0000 7f08aba00640 1 --2- 172.21.15.96:0/1816209497 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f08a40b3b70 0x7f08a415b2d0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:35.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.880+0000 7f08aba00640 1 -- 172.21.15.96:0/1816209497 >> 172.21.15.96:0/1816209497 conn(0x7f08a406bae0 msgr2=0x7f08a4160390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:35.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.880+0000 7f08aba00640 1 -- 172.21.15.96:0/1816209497 shutdown_connections 2024-05-28T06:05:35.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:35.880+0000 7f08aba00640 1 -- 172.21.15.96:0/1816209497 wait complete. 2024-05-28T06:05:35.901 INFO:teuthology.orchestra.run.smithi096.stdout:{"pg_ready":true,"pg_map":{"version":74,"stamp":"2024-05-28T06:05:35.511482+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220332,"kb_used_data":4136,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754388,"statfs":{"total":773094113280,"available":772868493312,"internally_reserved":0,"allocated":4235264,"data_stored":2358484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.084983"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.754089+0000","last_change":"2024-05-28T06:05:30.477204+0000","last_active":"2024-05-28T06:05:31.754089+0000","last_peered":"2024-05-28T06:05:31.754089+0000","last_clean":"2024-05-28T06:05:31.754089+0000","last_became_active":"2024-05-28T06:05:30.476844+0000","last_became_peered":"2024-05-28T06:05:30.476844+0000","last_unstale":"2024-05-28T06:05:31.754089+0000","last_undegraded":"2024-05-28T06:05:31.754089+0000","last_fullsized":"2024-05-28T06:05:31.754089+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:42:04.972881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.753020+0000","last_change":"2024-05-28T06:05:30.477943+0000","last_active":"2024-05-28T06:05:31.753020+0000","last_peered":"2024-05-28T06:05:31.753020+0000","last_clean":"2024-05-28T06:05:31.753020+0000","last_became_active":"2024-05-28T06:05:30.476995+0000","last_became_peered":"2024-05-28T06:05:30.476995+0000","last_unstale":"2024-05-28T06:05:31.753020+0000","last_undegraded":"2024-05-28T06:05:31.753020+0000","last_fullsized":"2024-05-28T06:05:31.753020+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:56:43.306179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.758130+0000","last_change":"2024-05-28T06:05:30.483815+0000","last_active":"2024-05-28T06:05:31.758130+0000","last_peered":"2024-05-28T06:05:31.758130+0000","last_clean":"2024-05-28T06:05:31.758130+0000","last_became_active":"2024-05-28T06:05:30.483476+0000","last_became_peered":"2024-05-28T06:05:30.483476+0000","last_unstale":"2024-05-28T06:05:31.758130+0000","last_undegraded":"2024-05-28T06:05:31.758130+0000","last_fullsized":"2024-05-28T06:05:31.758130+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:43:33.343569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.487451+0000","last_change":"2024-05-28T06:05:30.481651+0000","last_active":"2024-05-28T06:05:31.487451+0000","last_peered":"2024-05-28T06:05:31.487451+0000","last_clean":"2024-05-28T06:05:31.487451+0000","last_became_active":"2024-05-28T06:05:30.477221+0000","last_became_peered":"2024-05-28T06:05:30.477221+0000","last_unstale":"2024-05-28T06:05:31.487451+0000","last_undegraded":"2024-05-28T06:05:31.487451+0000","last_fullsized":"2024-05-28T06:05:31.487451+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:51:16.699034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.755050+0000","last_change":"2024-05-28T06:05:30.486034+0000","last_active":"2024-05-28T06:05:31.755050+0000","last_peered":"2024-05-28T06:05:31.755050+0000","last_clean":"2024-05-28T06:05:31.755050+0000","last_became_active":"2024-05-28T06:05:30.485742+0000","last_became_peered":"2024-05-28T06:05:30.485742+0000","last_unstale":"2024-05-28T06:05:31.755050+0000","last_undegraded":"2024-05-28T06:05:31.755050+0000","last_fullsized":"2024-05-28T06:05:31.755050+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:16:43.593153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.752594+0000","last_change":"2024-05-28T06:05:30.474720+0000","last_active":"2024-05-28T06:05:31.752594+0000","last_peered":"2024-05-28T06:05:31.752594+0000","last_clean":"2024-05-28T06:05:31.752594+0000","last_became_active":"2024-05-28T06:05:30.474189+0000","last_became_peered":"2024-05-28T06:05:30.474189+0000","last_unstale":"2024-05-28T06:05:31.752594+0000","last_undegraded":"2024-05-28T06:05:31.752594+0000","last_fullsized":"2024-05-28T06:05:31.752594+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:26:58.818587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.756498+0000","last_change":"2024-05-28T06:05:30.487197+0000","last_active":"2024-05-28T06:05:31.756498+0000","last_peered":"2024-05-28T06:05:31.756498+0000","last_clean":"2024-05-28T06:05:31.756498+0000","last_became_active":"2024-05-28T06:05:30.486918+0000","last_became_peered":"2024-05-28T06:05:30.486918+0000","last_unstale":"2024-05-28T06:05:31.756498+0000","last_undegraded":"2024-05-28T06:05:31.756498+0000","last_fullsized":"2024-05-28T06:05:31.756498+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:27:30.648202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.754628+0000","last_change":"2024-05-28T06:05:30.489229+0000","last_active":"2024-05-28T06:05:31.754628+0000","last_peered":"2024-05-28T06:05:31.754628+0000","last_clean":"2024-05-28T06:05:31.754628+0000","last_became_active":"2024-05-28T06:05:30.488923+0000","last_became_peered":"2024-05-28T06:05:30.488923+0000","last_unstale":"2024-05-28T06:05:31.754628+0000","last_undegraded":"2024-05-28T06:05:31.754628+0000","last_fullsized":"2024-05-28T06:05:31.754628+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:58:04.060400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.483710+0000","last_change":"2024-05-28T06:05:30.487873+0000","last_active":"2024-05-28T06:05:31.483710+0000","last_peered":"2024-05-28T06:05:31.483710+0000","last_clean":"2024-05-28T06:05:31.483710+0000","last_became_active":"2024-05-28T06:05:30.487617+0000","last_became_peered":"2024-05-28T06:05:30.487617+0000","last_unstale":"2024-05-28T06:05:31.483710+0000","last_undegraded":"2024-05-28T06:05:31.483710+0000","last_fullsized":"2024-05-28T06:05:31.483710+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:24:54.475794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.487607+0000","last_change":"2024-05-28T06:05:30.473105+0000","last_active":"2024-05-28T06:05:31.487607+0000","last_peered":"2024-05-28T06:05:31.487607+0000","last_clean":"2024-05-28T06:05:31.487607+0000","last_became_active":"2024-05-28T06:05:30.472798+0000","last_became_peered":"2024-05-28T06:05:30.472798+0000","last_unstale":"2024-05-28T06:05:31.487607+0000","last_undegraded":"2024-05-28T06:05:31.487607+0000","last_fullsized":"2024-05-28T06:05:31.487607+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:17:42.817810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.755746+0000","last_change":"2024-05-28T06:05:30.486102+0000","last_active":"2024-05-28T06:05:31.755746+0000","last_peered":"2024-05-28T06:05:31.755746+0000","last_clean":"2024-05-28T06:05:31.755746+0000","last_became_active":"2024-05-28T06:05:30.485870+0000","last_became_peered":"2024-05-28T06:05:30.485870+0000","last_unstale":"2024-05-28T06:05:31.755746+0000","last_undegraded":"2024-05-28T06:05:31.755746+0000","last_fullsized":"2024-05-28T06:05:31.755746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:27:15.781576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.757226+0000","last_change":"2024-05-28T06:05:30.478015+0000","last_active":"2024-05-28T06:05:31.757226+0000","last_peered":"2024-05-28T06:05:31.757226+0000","last_clean":"2024-05-28T06:05:31.757226+0000","last_became_active":"2024-05-28T06:05:30.477760+0000","last_became_peered":"2024-05-28T06:05:30.477760+0000","last_unstale":"2024-05-28T06:05:31.757226+0000","last_undegraded":"2024-05-28T06:05:31.757226+0000","last_fullsized":"2024-05-28T06:05:31.757226+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:26:58.818587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.755386+0000","last_change":"2024-05-28T06:05:30.489911+0000","last_active":"2024-05-28T06:05:31.755386+0000","last_peered":"2024-05-28T06:05:31.755386+0000","last_clean":"2024-05-28T06:05:31.755386+0000","last_became_active":"2024-05-28T06:05:30.489638+0000","last_became_peered":"2024-05-28T06:05:30.489638+0000","last_unstale":"2024-05-28T06:05:31.755386+0000","last_undegraded":"2024-05-28T06:05:31.755386+0000","last_fullsized":"2024-05-28T06:05:31.755386+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:30:21.413453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.752710+0000","last_change":"2024-05-28T06:05:30.488321+0000","last_active":"2024-05-28T06:05:31.752710+0000","last_peered":"2024-05-28T06:05:31.752710+0000","last_clean":"2024-05-28T06:05:31.752710+0000","last_became_active":"2024-05-28T06:05:30.487979+0000","last_became_peered":"2024-05-28T06:05:30.487979+0000","last_unstale":"2024-05-28T06:05:31.752710+0000","last_undegraded":"2024-05-28T06:05:31.752710+0000","last_fullsized":"2024-05-28T06:05:31.752710+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:27:15.781576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.755011+0000","last_change":"2024-05-28T06:05:30.469583+0000","last_active":"2024-05-28T06:05:31.755011+0000","last_peered":"2024-05-28T06:05:31.755011+0000","last_clean":"2024-05-28T06:05:31.755011+0000","last_became_active":"2024-05-28T06:05:30.469299+0000","last_became_peered":"2024-05-28T06:05:30.469299+0000","last_unstale":"2024-05-28T06:05:31.755011+0000","last_undegraded":"2024-05-28T06:05:31.755011+0000","last_fullsized":"2024-05-28T06:05:31.755011+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:51:12.201028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.755507+0000","last_change":"2024-05-28T06:05:30.474489+0000","last_active":"2024-05-28T06:05:31.755507+0000","last_peered":"2024-05-28T06:05:31.755507+0000","last_clean":"2024-05-28T06:05:31.755507+0000","last_became_active":"2024-05-28T06:05:30.474215+0000","last_became_peered":"2024-05-28T06:05:30.474215+0000","last_unstale":"2024-05-28T06:05:31.755507+0000","last_undegraded":"2024-05-28T06:05:31.755507+0000","last_fullsized":"2024-05-28T06:05:31.755507+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:26:58.818587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.756720+0000","last_change":"2024-05-28T06:05:30.486573+0000","last_active":"2024-05-28T06:05:31.756720+0000","last_peered":"2024-05-28T06:05:31.756720+0000","last_clean":"2024-05-28T06:05:31.756720+0000","last_became_active":"2024-05-28T06:05:30.486307+0000","last_became_peered":"2024-05-28T06:05:30.486307+0000","last_unstale":"2024-05-28T06:05:31.756720+0000","last_undegraded":"2024-05-28T06:05:31.756720+0000","last_fullsized":"2024-05-28T06:05:31.756720+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:43:33.343569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.753543+0000","last_change":"2024-05-28T06:05:30.477781+0000","last_active":"2024-05-28T06:05:31.753543+0000","last_peered":"2024-05-28T06:05:31.753543+0000","last_clean":"2024-05-28T06:05:31.753543+0000","last_became_active":"2024-05-28T06:05:30.476599+0000","last_became_peered":"2024-05-28T06:05:30.476599+0000","last_unstale":"2024-05-28T06:05:31.753543+0000","last_undegraded":"2024-05-28T06:05:31.753543+0000","last_fullsized":"2024-05-28T06:05:31.753543+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:43:33.343569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.753222+0000","last_change":"2024-05-28T06:05:30.476553+0000","last_active":"2024-05-28T06:05:31.753222+0000","last_peered":"2024-05-28T06:05:31.753222+0000","last_clean":"2024-05-28T06:05:31.753222+0000","last_became_active":"2024-05-28T06:05:30.476290+0000","last_became_peered":"2024-05-28T06:05:30.476290+0000","last_unstale":"2024-05-28T06:05:31.753222+0000","last_undegraded":"2024-05-28T06:05:31.753222+0000","last_fullsized":"2024-05-28T06:05:31.753222+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:27:30.648202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.755494+0000","last_change":"2024-05-28T06:05:30.483659+0000","last_active":"2024-05-28T06:05:31.755494+0000","last_peered":"2024-05-28T06:05:31.755494+0000","last_clean":"2024-05-28T06:05:31.755494+0000","last_became_active":"2024-05-28T06:05:30.483403+0000","last_became_peered":"2024-05-28T06:05:30.483403+0000","last_unstale":"2024-05-28T06:05:31.755494+0000","last_undegraded":"2024-05-28T06:05:31.755494+0000","last_fullsized":"2024-05-28T06:05:31.755494+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:47:22.330606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.482709+0000","last_change":"2024-05-28T06:05:30.472992+0000","last_active":"2024-05-28T06:05:31.482709+0000","last_peered":"2024-05-28T06:05:31.482709+0000","last_clean":"2024-05-28T06:05:31.482709+0000","last_became_active":"2024-05-28T06:05:30.472725+0000","last_became_peered":"2024-05-28T06:05:30.472725+0000","last_unstale":"2024-05-28T06:05:31.482709+0000","last_undegraded":"2024-05-28T06:05:31.482709+0000","last_fullsized":"2024-05-28T06:05:31.482709+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:42:04.972881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.482893+0000","last_change":"2024-05-28T06:05:30.468698+0000","last_active":"2024-05-28T06:05:31.482893+0000","last_peered":"2024-05-28T06:05:31.482893+0000","last_clean":"2024-05-28T06:05:31.482893+0000","last_became_active":"2024-05-28T06:05:30.468120+0000","last_became_peered":"2024-05-28T06:05:30.468120+0000","last_unstale":"2024-05-28T06:05:31.482893+0000","last_undegraded":"2024-05-28T06:05:31.482893+0000","last_fullsized":"2024-05-28T06:05:31.482893+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:41:24.956597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.477622+0000","last_change":"2024-05-28T06:05:30.470896+0000","last_active":"2024-05-28T06:05:31.477622+0000","last_peered":"2024-05-28T06:05:31.477622+0000","last_clean":"2024-05-28T06:05:31.477622+0000","last_became_active":"2024-05-28T06:05:30.470316+0000","last_became_peered":"2024-05-28T06:05:30.470316+0000","last_unstale":"2024-05-28T06:05:31.477622+0000","last_undegraded":"2024-05-28T06:05:31.477622+0000","last_fullsized":"2024-05-28T06:05:31.477622+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:37:19.547704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.754580+0000","last_change":"2024-05-28T06:05:30.483993+0000","last_active":"2024-05-28T06:05:31.754580+0000","last_peered":"2024-05-28T06:05:31.754580+0000","last_clean":"2024-05-28T06:05:31.754580+0000","last_became_active":"2024-05-28T06:05:30.483709+0000","last_became_peered":"2024-05-28T06:05:30.483709+0000","last_unstale":"2024-05-28T06:05:31.754580+0000","last_undegraded":"2024-05-28T06:05:31.754580+0000","last_fullsized":"2024-05-28T06:05:31.754580+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:58:04.060400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.488135+0000","last_change":"2024-05-28T06:05:30.472224+0000","last_active":"2024-05-28T06:05:31.488135+0000","last_peered":"2024-05-28T06:05:31.488135+0000","last_clean":"2024-05-28T06:05:31.488135+0000","last_became_active":"2024-05-28T06:05:30.471939+0000","last_became_peered":"2024-05-28T06:05:30.471939+0000","last_unstale":"2024-05-28T06:05:31.488135+0000","last_undegraded":"2024-05-28T06:05:31.488135+0000","last_fullsized":"2024-05-28T06:05:31.488135+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:58:04.060400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.753861+0000","last_change":"2024-05-28T06:05:30.472699+0000","last_active":"2024-05-28T06:05:31.753861+0000","last_peered":"2024-05-28T06:05:31.753861+0000","last_clean":"2024-05-28T06:05:31.753861+0000","last_became_active":"2024-05-28T06:05:30.472399+0000","last_became_peered":"2024-05-28T06:05:30.472399+0000","last_unstale":"2024-05-28T06:05:31.753861+0000","last_undegraded":"2024-05-28T06:05:31.753861+0000","last_fullsized":"2024-05-28T06:05:31.753861+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:28:45.385929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.488123+0000","last_change":"2024-05-28T06:05:30.477001+0000","last_active":"2024-05-28T06:05:31.488123+0000","last_peered":"2024-05-28T06:05:31.488123+0000","last_clean":"2024-05-28T06:05:31.488123+0000","last_became_active":"2024-05-28T06:05:30.476725+0000","last_became_peered":"2024-05-28T06:05:30.476725+0000","last_unstale":"2024-05-28T06:05:31.488123+0000","last_undegraded":"2024-05-28T06:05:31.488123+0000","last_fullsized":"2024-05-28T06:05:31.488123+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:11:20.837156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.757661+0000","last_change":"2024-05-28T06:05:30.484736+0000","last_active":"2024-05-28T06:05:31.757661+0000","last_peered":"2024-05-28T06:05:31.757661+0000","last_clean":"2024-05-28T06:05:31.757661+0000","last_became_active":"2024-05-28T06:05:30.482722+0000","last_became_peered":"2024-05-28T06:05:30.482722+0000","last_unstale":"2024-05-28T06:05:31.757661+0000","last_undegraded":"2024-05-28T06:05:31.757661+0000","last_fullsized":"2024-05-28T06:05:31.757661+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:56:43.306179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.482975+0000","last_change":"2024-05-28T06:05:30.477442+0000","last_active":"2024-05-28T06:05:31.482975+0000","last_peered":"2024-05-28T06:05:31.482975+0000","last_clean":"2024-05-28T06:05:31.482975+0000","last_became_active":"2024-05-28T06:05:30.477224+0000","last_became_peered":"2024-05-28T06:05:30.477224+0000","last_unstale":"2024-05-28T06:05:31.482975+0000","last_undegraded":"2024-05-28T06:05:31.482975+0000","last_fullsized":"2024-05-28T06:05:31.482975+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:46:23.694062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.756903+0000","last_change":"2024-05-28T06:05:30.484668+0000","last_active":"2024-05-28T06:05:31.756903+0000","last_peered":"2024-05-28T06:05:31.756903+0000","last_clean":"2024-05-28T06:05:31.756903+0000","last_became_active":"2024-05-28T06:05:30.484339+0000","last_became_peered":"2024-05-28T06:05:30.484339+0000","last_unstale":"2024-05-28T06:05:31.756903+0000","last_undegraded":"2024-05-28T06:05:31.756903+0000","last_fullsized":"2024-05-28T06:05:31.756903+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:09:23.562166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.754848+0000","last_change":"2024-05-28T06:05:30.479119+0000","last_active":"2024-05-28T06:05:31.754848+0000","last_peered":"2024-05-28T06:05:31.754848+0000","last_clean":"2024-05-28T06:05:31.754848+0000","last_became_active":"2024-05-28T06:05:30.478742+0000","last_became_peered":"2024-05-28T06:05:30.478742+0000","last_unstale":"2024-05-28T06:05:31.754848+0000","last_undegraded":"2024-05-28T06:05:31.754848+0000","last_fullsized":"2024-05-28T06:05:31.754848+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:41:24.956597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.754808+0000","last_change":"2024-05-28T06:05:30.474293+0000","last_active":"2024-05-28T06:05:31.754808+0000","last_peered":"2024-05-28T06:05:31.754808+0000","last_clean":"2024-05-28T06:05:31.754808+0000","last_became_active":"2024-05-28T06:05:30.473563+0000","last_became_peered":"2024-05-28T06:05:30.473563+0000","last_unstale":"2024-05-28T06:05:31.754808+0000","last_undegraded":"2024-05-28T06:05:31.754808+0000","last_fullsized":"2024-05-28T06:05:31.754808+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:51:12.201028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.478000+0000","last_change":"2024-05-28T06:05:30.473138+0000","last_active":"2024-05-28T06:05:31.478000+0000","last_peered":"2024-05-28T06:05:31.478000+0000","last_clean":"2024-05-28T06:05:31.478000+0000","last_became_active":"2024-05-28T06:05:30.472302+0000","last_became_peered":"2024-05-28T06:05:30.472302+0000","last_unstale":"2024-05-28T06:05:31.478000+0000","last_undegraded":"2024-05-28T06:05:31.478000+0000","last_fullsized":"2024-05-28T06:05:31.478000+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:46:23.694062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"32'4","reported_seq":18,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.754387+0000","last_change":"2024-05-28T06:05:29.474199+0000","last_active":"2024-05-28T06:05:31.754387+0000","last_peered":"2024-05-28T06:05:31.754387+0000","last_clean":"2024-05-28T06:05:31.754387+0000","last_became_active":"2024-05-28T06:05:29.473863+0000","last_became_peered":"2024-05-28T06:05:29.473863+0000","last_unstale":"2024-05-28T06:05:31.754387+0000","last_undegraded":"2024-05-28T06:05:31.754387+0000","last_fullsized":"2024-05-28T06:05:31.754387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:47:21.316456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.754375+0000","last_change":"2024-05-28T06:05:30.473327+0000","last_active":"2024-05-28T06:05:31.754375+0000","last_peered":"2024-05-28T06:05:31.754375+0000","last_clean":"2024-05-28T06:05:31.754375+0000","last_became_active":"2024-05-28T06:05:30.470709+0000","last_became_peered":"2024-05-28T06:05:30.470709+0000","last_unstale":"2024-05-28T06:05:31.754375+0000","last_undegraded":"2024-05-28T06:05:31.754375+0000","last_fullsized":"2024-05-28T06:05:31.754375+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:37:19.547704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.478062+0000","last_change":"2024-05-28T06:05:29.460672+0000","last_active":"2024-05-28T06:05:31.478062+0000","last_peered":"2024-05-28T06:05:31.478062+0000","last_clean":"2024-05-28T06:05:31.478062+0000","last_became_active":"2024-05-28T06:05:29.459991+0000","last_became_peered":"2024-05-28T06:05:29.459991+0000","last_unstale":"2024-05-28T06:05:31.478062+0000","last_undegraded":"2024-05-28T06:05:31.478062+0000","last_fullsized":"2024-05-28T06:05:31.478062+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:57:51.752948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.483444+0000","last_change":"2024-05-28T06:05:30.477437+0000","last_active":"2024-05-28T06:05:31.483444+0000","last_peered":"2024-05-28T06:05:31.483444+0000","last_clean":"2024-05-28T06:05:31.483444+0000","last_became_active":"2024-05-28T06:05:30.477193+0000","last_became_peered":"2024-05-28T06:05:30.477193+0000","last_unstale":"2024-05-28T06:05:31.483444+0000","last_undegraded":"2024-05-28T06:05:31.483444+0000","last_fullsized":"2024-05-28T06:05:31.483444+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:14:29.844730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.753411+0000","last_change":"2024-05-28T06:05:29.456735+0000","last_active":"2024-05-28T06:05:31.753411+0000","last_peered":"2024-05-28T06:05:31.753411+0000","last_clean":"2024-05-28T06:05:31.753411+0000","last_became_active":"2024-05-28T06:05:29.456466+0000","last_became_peered":"2024-05-28T06:05:29.456466+0000","last_unstale":"2024-05-28T06:05:31.753411+0000","last_undegraded":"2024-05-28T06:05:31.753411+0000","last_fullsized":"2024-05-28T06:05:31.753411+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:41:23.942447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.488191+0000","last_change":"2024-05-28T06:05:30.476962+0000","last_active":"2024-05-28T06:05:31.488191+0000","last_peered":"2024-05-28T06:05:31.488191+0000","last_clean":"2024-05-28T06:05:31.488191+0000","last_became_active":"2024-05-28T06:05:30.476657+0000","last_became_peered":"2024-05-28T06:05:30.476657+0000","last_unstale":"2024-05-28T06:05:31.488191+0000","last_undegraded":"2024-05-28T06:05:31.488191+0000","last_fullsized":"2024-05-28T06:05:31.488191+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:51:12.201028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.755643+0000","last_change":"2024-05-28T06:05:29.451693+0000","last_active":"2024-05-28T06:05:31.755643+0000","last_peered":"2024-05-28T06:05:31.755643+0000","last_clean":"2024-05-28T06:05:31.755643+0000","last_became_active":"2024-05-28T06:05:29.451343+0000","last_became_peered":"2024-05-28T06:05:29.451343+0000","last_unstale":"2024-05-28T06:05:31.755643+0000","last_undegraded":"2024-05-28T06:05:31.755643+0000","last_fullsized":"2024-05-28T06:05:31.755643+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:42:15.256795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.756525+0000","last_change":"2024-05-28T06:05:30.477467+0000","last_active":"2024-05-28T06:05:31.756525+0000","last_peered":"2024-05-28T06:05:31.756525+0000","last_clean":"2024-05-28T06:05:31.756525+0000","last_became_active":"2024-05-28T06:05:30.477224+0000","last_became_peered":"2024-05-28T06:05:30.477224+0000","last_unstale":"2024-05-28T06:05:31.756525+0000","last_undegraded":"2024-05-28T06:05:31.756525+0000","last_fullsized":"2024-05-28T06:05:31.756525+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:21:35.302971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.482842+0000","last_change":"2024-05-28T06:05:29.463758+0000","last_active":"2024-05-28T06:05:31.482842+0000","last_peered":"2024-05-28T06:05:31.482842+0000","last_clean":"2024-05-28T06:05:31.482842+0000","last_became_active":"2024-05-28T06:05:29.460297+0000","last_became_peered":"2024-05-28T06:05:29.460297+0000","last_unstale":"2024-05-28T06:05:31.482842+0000","last_undegraded":"2024-05-28T06:05:31.482842+0000","last_fullsized":"2024-05-28T06:05:31.482842+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:05:43.269097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.756966+0000","last_change":"2024-05-28T06:05:30.484901+0000","last_active":"2024-05-28T06:05:31.756966+0000","last_peered":"2024-05-28T06:05:31.756966+0000","last_clean":"2024-05-28T06:05:31.756966+0000","last_became_active":"2024-05-28T06:05:30.484581+0000","last_became_peered":"2024-05-28T06:05:30.484581+0000","last_unstale":"2024-05-28T06:05:31.756966+0000","last_undegraded":"2024-05-28T06:05:31.756966+0000","last_fullsized":"2024-05-28T06:05:31.756966+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:39:33.123768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.482842+0000","last_change":"2024-05-28T06:05:29.462024+0000","last_active":"2024-05-28T06:05:31.482842+0000","last_peered":"2024-05-28T06:05:31.482842+0000","last_clean":"2024-05-28T06:05:31.482842+0000","last_became_active":"2024-05-28T06:05:29.460033+0000","last_became_peered":"2024-05-28T06:05:29.460033+0000","last_unstale":"2024-05-28T06:05:31.482842+0000","last_undegraded":"2024-05-28T06:05:31.482842+0000","last_fullsized":"2024-05-28T06:05:31.482842+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:38:57.785689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.482390+0000","last_change":"2024-05-28T06:05:30.468776+0000","last_active":"2024-05-28T06:05:31.482390+0000","last_peered":"2024-05-28T06:05:31.482390+0000","last_clean":"2024-05-28T06:05:31.482390+0000","last_became_active":"2024-05-28T06:05:30.468551+0000","last_became_peered":"2024-05-28T06:05:30.468551+0000","last_unstale":"2024-05-28T06:05:31.482390+0000","last_undegraded":"2024-05-28T06:05:31.482390+0000","last_fullsized":"2024-05-28T06:05:31.482390+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:33:36.484602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.752569+0000","last_change":"2024-05-28T06:05:29.456433+0000","last_active":"2024-05-28T06:05:31.752569+0000","last_peered":"2024-05-28T06:05:31.752569+0000","last_clean":"2024-05-28T06:05:31.752569+0000","last_became_active":"2024-05-28T06:05:29.456113+0000","last_became_peered":"2024-05-28T06:05:29.456113+0000","last_unstale":"2024-05-28T06:05:31.752569+0000","last_undegraded":"2024-05-28T06:05:31.752569+0000","last_fullsized":"2024-05-28T06:05:31.752569+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:12:52.117940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.486965+0000","last_change":"2024-05-28T06:05:30.478697+0000","last_active":"2024-05-28T06:05:31.486965+0000","last_peered":"2024-05-28T06:05:31.486965+0000","last_clean":"2024-05-28T06:05:31.486965+0000","last_became_active":"2024-05-28T06:05:30.477683+0000","last_became_peered":"2024-05-28T06:05:30.477683+0000","last_unstale":"2024-05-28T06:05:31.486965+0000","last_undegraded":"2024-05-28T06:05:31.486965+0000","last_fullsized":"2024-05-28T06:05:31.486965+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:24:54.475794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.482413+0000","last_change":"2024-05-28T06:05:29.462393+0000","last_active":"2024-05-28T06:05:31.482413+0000","last_peered":"2024-05-28T06:05:31.482413+0000","last_clean":"2024-05-28T06:05:31.482413+0000","last_became_active":"2024-05-28T06:05:29.461785+0000","last_became_peered":"2024-05-28T06:05:29.461785+0000","last_unstale":"2024-05-28T06:05:31.482413+0000","last_undegraded":"2024-05-28T06:05:31.482413+0000","last_fullsized":"2024-05-28T06:05:31.482413+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:58:46.608407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.757287+0000","last_change":"2024-05-28T06:05:30.481878+0000","last_active":"2024-05-28T06:05:31.757287+0000","last_peered":"2024-05-28T06:05:31.757287+0000","last_clean":"2024-05-28T06:05:31.757287+0000","last_became_active":"2024-05-28T06:05:30.481566+0000","last_became_peered":"2024-05-28T06:05:30.481566+0000","last_unstale":"2024-05-28T06:05:31.757287+0000","last_undegraded":"2024-05-28T06:05:31.757287+0000","last_fullsized":"2024-05-28T06:05:31.757287+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:27:15.781576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.753786+0000","last_change":"2024-05-28T06:05:29.461808+0000","last_active":"2024-05-28T06:05:31.753786+0000","last_peered":"2024-05-28T06:05:31.753786+0000","last_clean":"2024-05-28T06:05:31.753786+0000","last_became_active":"2024-05-28T06:05:29.461481+0000","last_became_peered":"2024-05-28T06:05:29.461481+0000","last_unstale":"2024-05-28T06:05:31.753786+0000","last_undegraded":"2024-05-28T06:05:31.753786+0000","last_fullsized":"2024-05-28T06:05:31.753786+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:46:22.679912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.753788+0000","last_change":"2024-05-28T06:05:30.475975+0000","last_active":"2024-05-28T06:05:31.753788+0000","last_peered":"2024-05-28T06:05:31.753788+0000","last_clean":"2024-05-28T06:05:31.753788+0000","last_became_active":"2024-05-28T06:05:30.475693+0000","last_became_peered":"2024-05-28T06:05:30.475693+0000","last_unstale":"2024-05-28T06:05:31.753788+0000","last_undegraded":"2024-05-28T06:05:31.753788+0000","last_fullsized":"2024-05-28T06:05:31.753788+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:09:23.562166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.478140+0000","last_change":"2024-05-28T06:05:29.460940+0000","last_active":"2024-05-28T06:05:31.478140+0000","last_peered":"2024-05-28T06:05:31.478140+0000","last_clean":"2024-05-28T06:05:31.478140+0000","last_became_active":"2024-05-28T06:05:29.460433+0000","last_became_peered":"2024-05-28T06:05:29.460433+0000","last_unstale":"2024-05-28T06:05:31.478140+0000","last_undegraded":"2024-05-28T06:05:31.478140+0000","last_fullsized":"2024-05-28T06:05:31.478140+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:32:34.183989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.753438+0000","last_change":"2024-05-28T06:05:30.472323+0000","last_active":"2024-05-28T06:05:31.753438+0000","last_peered":"2024-05-28T06:05:31.753438+0000","last_clean":"2024-05-28T06:05:31.753438+0000","last_became_active":"2024-05-28T06:05:30.472022+0000","last_became_peered":"2024-05-28T06:05:30.472022+0000","last_unstale":"2024-05-28T06:05:31.753438+0000","last_undegraded":"2024-05-28T06:05:31.753438+0000","last_fullsized":"2024-05-28T06:05:31.753438+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:21:35.302971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"32'3","reported_seq":17,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.757966+0000","last_change":"2024-05-28T06:05:29.462646+0000","last_active":"2024-05-28T06:05:31.757966+0000","last_peered":"2024-05-28T06:05:31.757966+0000","last_clean":"2024-05-28T06:05:31.757966+0000","last_became_active":"2024-05-28T06:05:29.462355+0000","last_became_peered":"2024-05-28T06:05:29.462355+0000","last_unstale":"2024-05-28T06:05:31.757966+0000","last_undegraded":"2024-05-28T06:05:31.757966+0000","last_fullsized":"2024-05-28T06:05:31.757966+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:37:18.533553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.757672+0000","last_change":"2024-05-28T06:05:30.483710+0000","last_active":"2024-05-28T06:05:31.757672+0000","last_peered":"2024-05-28T06:05:31.757672+0000","last_clean":"2024-05-28T06:05:31.757672+0000","last_became_active":"2024-05-28T06:05:30.483250+0000","last_became_peered":"2024-05-28T06:05:30.483250+0000","last_unstale":"2024-05-28T06:05:31.757672+0000","last_undegraded":"2024-05-28T06:05:31.757672+0000","last_fullsized":"2024-05-28T06:05:31.757672+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:38:12.995079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"32'6","reported_seq":20,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.753560+0000","last_change":"2024-05-28T06:05:29.458155+0000","last_active":"2024-05-28T06:05:31.753560+0000","last_peered":"2024-05-28T06:05:31.753560+0000","last_clean":"2024-05-28T06:05:31.753560+0000","last_became_active":"2024-05-28T06:05:29.457822+0000","last_became_peered":"2024-05-28T06:05:29.457822+0000","last_unstale":"2024-05-28T06:05:31.753560+0000","last_undegraded":"2024-05-28T06:05:31.753560+0000","last_fullsized":"2024-05-28T06:05:31.753560+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:37:18.533553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.487453+0000","last_change":"2024-05-28T06:05:30.479836+0000","last_active":"2024-05-28T06:05:31.487453+0000","last_peered":"2024-05-28T06:05:31.487453+0000","last_clean":"2024-05-28T06:05:31.487453+0000","last_became_active":"2024-05-28T06:05:30.479377+0000","last_became_peered":"2024-05-28T06:05:30.479377+0000","last_unstale":"2024-05-28T06:05:31.487453+0000","last_undegraded":"2024-05-28T06:05:31.487453+0000","last_fullsized":"2024-05-28T06:05:31.487453+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:39:39.679412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.483004+0000","last_change":"2024-05-28T06:05:29.464610+0000","last_active":"2024-05-28T06:05:31.483004+0000","last_peered":"2024-05-28T06:05:31.483004+0000","last_clean":"2024-05-28T06:05:31.483004+0000","last_became_active":"2024-05-28T06:05:29.464074+0000","last_became_peered":"2024-05-28T06:05:29.464074+0000","last_unstale":"2024-05-28T06:05:31.483004+0000","last_undegraded":"2024-05-28T06:05:31.483004+0000","last_fullsized":"2024-05-28T06:05:31.483004+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:30:20.399303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.752600+0000","last_change":"2024-05-28T06:05:30.474344+0000","last_active":"2024-05-28T06:05:31.752600+0000","last_peered":"2024-05-28T06:05:31.752600+0000","last_clean":"2024-05-28T06:05:31.752600+0000","last_became_active":"2024-05-28T06:05:30.474074+0000","last_became_peered":"2024-05-28T06:05:30.474074+0000","last_unstale":"2024-05-28T06:05:31.752600+0000","last_undegraded":"2024-05-28T06:05:31.752600+0000","last_fullsized":"2024-05-28T06:05:31.752600+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:39:33.123768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.486959+0000","last_change":"2024-05-28T06:05:29.463194+0000","last_active":"2024-05-28T06:05:31.486959+0000","last_peered":"2024-05-28T06:05:31.486959+0000","last_clean":"2024-05-28T06:05:31.486959+0000","last_became_active":"2024-05-28T06:05:29.462870+0000","last_became_peered":"2024-05-28T06:05:29.462870+0000","last_unstale":"2024-05-28T06:05:31.486959+0000","last_undegraded":"2024-05-28T06:05:31.486959+0000","last_fullsized":"2024-05-28T06:05:31.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:32:34.183989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.755042+0000","last_change":"2024-05-28T06:05:30.478298+0000","last_active":"2024-05-28T06:05:31.755042+0000","last_peered":"2024-05-28T06:05:31.755042+0000","last_clean":"2024-05-28T06:05:31.755042+0000","last_became_active":"2024-05-28T06:05:30.478018+0000","last_became_peered":"2024-05-28T06:05:30.478018+0000","last_unstale":"2024-05-28T06:05:31.755042+0000","last_undegraded":"2024-05-28T06:05:31.755042+0000","last_fullsized":"2024-05-28T06:05:31.755042+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:39:33.123768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.752817+0000","last_change":"2024-05-28T06:05:29.462298+0000","last_active":"2024-05-28T06:05:31.752817+0000","last_peered":"2024-05-28T06:05:31.752817+0000","last_clean":"2024-05-28T06:05:31.752817+0000","last_became_active":"2024-05-28T06:05:29.462022+0000","last_became_peered":"2024-05-28T06:05:29.462022+0000","last_unstale":"2024-05-28T06:05:31.752817+0000","last_undegraded":"2024-05-28T06:05:31.752817+0000","last_fullsized":"2024-05-28T06:05:31.752817+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:33:35.470452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.754851+0000","last_change":"2024-05-28T06:05:30.478395+0000","last_active":"2024-05-28T06:05:31.754851+0000","last_peered":"2024-05-28T06:05:31.754851+0000","last_clean":"2024-05-28T06:05:31.754851+0000","last_became_active":"2024-05-28T06:05:30.470732+0000","last_became_peered":"2024-05-28T06:05:30.470732+0000","last_unstale":"2024-05-28T06:05:31.754851+0000","last_undegraded":"2024-05-28T06:05:31.754851+0000","last_fullsized":"2024-05-28T06:05:31.754851+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:46:23.694062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"32'8","reported_seq":22,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.752843+0000","last_change":"2024-05-28T06:05:29.460103+0000","last_active":"2024-05-28T06:05:31.752843+0000","last_peered":"2024-05-28T06:05:31.752843+0000","last_clean":"2024-05-28T06:05:31.752843+0000","last_became_active":"2024-05-28T06:05:29.459788+0000","last_became_peered":"2024-05-28T06:05:29.459788+0000","last_unstale":"2024-05-28T06:05:31.752843+0000","last_undegraded":"2024-05-28T06:05:31.752843+0000","last_fullsized":"2024-05-28T06:05:31.752843+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:03:08.734203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.483476+0000","last_change":"2024-05-28T06:05:30.471442+0000","last_active":"2024-05-28T06:05:31.483476+0000","last_peered":"2024-05-28T06:05:31.483476+0000","last_clean":"2024-05-28T06:05:31.483476+0000","last_became_active":"2024-05-28T06:05:30.471189+0000","last_became_peered":"2024-05-28T06:05:30.471189+0000","last_unstale":"2024-05-28T06:05:31.483476+0000","last_undegraded":"2024-05-28T06:05:31.483476+0000","last_fullsized":"2024-05-28T06:05:31.483476+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:43:11.326702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.488303+0000","last_change":"2024-05-28T06:05:29.458728+0000","last_active":"2024-05-28T06:05:31.488303+0000","last_peered":"2024-05-28T06:05:31.488303+0000","last_clean":"2024-05-28T06:05:31.488303+0000","last_became_active":"2024-05-28T06:05:29.458154+0000","last_became_peered":"2024-05-28T06:05:29.458154+0000","last_unstale":"2024-05-28T06:05:31.488303+0000","last_undegraded":"2024-05-28T06:05:31.488303+0000","last_fullsized":"2024-05-28T06:05:31.488303+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:12:38.485450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.757818+0000","last_change":"2024-05-28T06:05:30.471043+0000","last_active":"2024-05-28T06:05:31.757818+0000","last_peered":"2024-05-28T06:05:31.757818+0000","last_clean":"2024-05-28T06:05:31.757818+0000","last_became_active":"2024-05-28T06:05:30.470782+0000","last_became_peered":"2024-05-28T06:05:30.470782+0000","last_unstale":"2024-05-28T06:05:31.757818+0000","last_undegraded":"2024-05-28T06:05:31.757818+0000","last_fullsized":"2024-05-28T06:05:31.757818+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:27:30.648202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.483361+0000","last_change":"2024-05-28T06:05:29.463122+0000","last_active":"2024-05-28T06:05:31.483361+0000","last_peered":"2024-05-28T06:05:31.483361+0000","last_clean":"2024-05-28T06:05:31.483361+0000","last_became_active":"2024-05-28T06:05:29.462362+0000","last_became_peered":"2024-05-28T06:05:29.462362+0000","last_unstale":"2024-05-28T06:05:31.483361+0000","last_undegraded":"2024-05-28T06:05:31.483361+0000","last_fullsized":"2024-05-28T06:05:31.483361+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:34:23.160698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.487454+0000","last_change":"2024-05-28T06:05:30.482239+0000","last_active":"2024-05-28T06:05:31.487454+0000","last_peered":"2024-05-28T06:05:31.487454+0000","last_clean":"2024-05-28T06:05:31.487454+0000","last_became_active":"2024-05-28T06:05:30.481947+0000","last_became_peered":"2024-05-28T06:05:30.481947+0000","last_unstale":"2024-05-28T06:05:31.487454+0000","last_undegraded":"2024-05-28T06:05:31.487454+0000","last_fullsized":"2024-05-28T06:05:31.487454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:04:45.854464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.483370+0000","last_change":"2024-05-28T06:05:29.450660+0000","last_active":"2024-05-28T06:05:31.483370+0000","last_peered":"2024-05-28T06:05:31.483370+0000","last_clean":"2024-05-28T06:05:31.483370+0000","last_became_active":"2024-05-28T06:05:29.446547+0000","last_became_peered":"2024-05-28T06:05:29.446547+0000","last_unstale":"2024-05-28T06:05:31.483370+0000","last_undegraded":"2024-05-28T06:05:31.483370+0000","last_fullsized":"2024-05-28T06:05:31.483370+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:33:40.689124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.755147+0000","last_change":"2024-05-28T06:05:30.478730+0000","last_active":"2024-05-28T06:05:31.755147+0000","last_peered":"2024-05-28T06:05:31.755147+0000","last_clean":"2024-05-28T06:05:31.755147+0000","last_became_active":"2024-05-28T06:05:30.478438+0000","last_became_peered":"2024-05-28T06:05:30.478438+0000","last_unstale":"2024-05-28T06:05:31.755147+0000","last_undegraded":"2024-05-28T06:05:31.755147+0000","last_fullsized":"2024-05-28T06:05:31.755147+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:05:44.283248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.483501+0000","last_change":"2024-05-28T06:05:29.461213+0000","last_active":"2024-05-28T06:05:31.483501+0000","last_peered":"2024-05-28T06:05:31.483501+0000","last_clean":"2024-05-28T06:05:31.483501+0000","last_became_active":"2024-05-28T06:05:29.460899+0000","last_became_peered":"2024-05-28T06:05:29.460899+0000","last_unstale":"2024-05-28T06:05:31.483501+0000","last_undegraded":"2024-05-28T06:05:31.483501+0000","last_fullsized":"2024-05-28T06:05:31.483501+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:16:42.579003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.754877+0000","last_change":"2024-05-28T06:05:30.484117+0000","last_active":"2024-05-28T06:05:31.754877+0000","last_peered":"2024-05-28T06:05:31.754877+0000","last_clean":"2024-05-28T06:05:31.754877+0000","last_became_active":"2024-05-28T06:05:30.483835+0000","last_became_peered":"2024-05-28T06:05:30.483835+0000","last_unstale":"2024-05-28T06:05:31.754877+0000","last_undegraded":"2024-05-28T06:05:31.754877+0000","last_fullsized":"2024-05-28T06:05:31.754877+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:41:32.997209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.478083+0000","last_change":"2024-05-28T06:05:29.447607+0000","last_active":"2024-05-28T06:05:31.478083+0000","last_peered":"2024-05-28T06:05:31.478083+0000","last_clean":"2024-05-28T06:05:31.478083+0000","last_became_active":"2024-05-28T06:05:29.447321+0000","last_became_peered":"2024-05-28T06:05:29.447321+0000","last_unstale":"2024-05-28T06:05:31.478083+0000","last_undegraded":"2024-05-28T06:05:31.478083+0000","last_fullsized":"2024-05-28T06:05:31.478083+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:47:21.316456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.755458+0000","last_change":"2024-05-28T06:05:30.484954+0000","last_active":"2024-05-28T06:05:31.755458+0000","last_peered":"2024-05-28T06:05:31.755458+0000","last_clean":"2024-05-28T06:05:31.755458+0000","last_became_active":"2024-05-28T06:05:30.484699+0000","last_became_peered":"2024-05-28T06:05:30.484699+0000","last_unstale":"2024-05-28T06:05:31.755458+0000","last_undegraded":"2024-05-28T06:05:31.755458+0000","last_fullsized":"2024-05-28T06:05:31.755458+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:10:49.765112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.753811+0000","last_change":"2024-05-28T06:05:29.463849+0000","last_active":"2024-05-28T06:05:31.753811+0000","last_peered":"2024-05-28T06:05:31.753811+0000","last_clean":"2024-05-28T06:05:31.753811+0000","last_became_active":"2024-05-28T06:05:29.463514+0000","last_became_peered":"2024-05-28T06:05:29.463514+0000","last_unstale":"2024-05-28T06:05:31.753811+0000","last_undegraded":"2024-05-28T06:05:31.753811+0000","last_fullsized":"2024-05-28T06:05:31.753811+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:10:48.750962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.754811+0000","last_change":"2024-05-28T06:05:30.485046+0000","last_active":"2024-05-28T06:05:31.754811+0000","last_peered":"2024-05-28T06:05:31.754811+0000","last_clean":"2024-05-28T06:05:31.754811+0000","last_became_active":"2024-05-28T06:05:30.484737+0000","last_became_peered":"2024-05-28T06:05:30.484737+0000","last_unstale":"2024-05-28T06:05:31.754811+0000","last_undegraded":"2024-05-28T06:05:31.754811+0000","last_fullsized":"2024-05-28T06:05:31.754811+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:34:24.174848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.482448+0000","last_change":"2024-05-28T06:05:29.450109+0000","last_active":"2024-05-28T06:05:31.482448+0000","last_peered":"2024-05-28T06:05:31.482448+0000","last_clean":"2024-05-28T06:05:31.482448+0000","last_became_active":"2024-05-28T06:05:29.449618+0000","last_became_peered":"2024-05-28T06:05:29.449618+0000","last_unstale":"2024-05-28T06:05:31.482448+0000","last_undegraded":"2024-05-28T06:05:31.482448+0000","last_fullsized":"2024-05-28T06:05:31.482448+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:41:31.983059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.755911+0000","last_change":"2024-05-28T06:05:30.470295+0000","last_active":"2024-05-28T06:05:31.755911+0000","last_peered":"2024-05-28T06:05:31.755911+0000","last_clean":"2024-05-28T06:05:31.755911+0000","last_became_active":"2024-05-28T06:05:30.470025+0000","last_became_peered":"2024-05-28T06:05:30.470025+0000","last_unstale":"2024-05-28T06:05:31.755911+0000","last_undegraded":"2024-05-28T06:05:31.755911+0000","last_fullsized":"2024-05-28T06:05:31.755911+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:38:12.995079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.756136+0000","last_change":"2024-05-28T06:05:29.467769+0000","last_active":"2024-05-28T06:05:31.756136+0000","last_peered":"2024-05-28T06:05:31.756136+0000","last_clean":"2024-05-28T06:05:31.756136+0000","last_became_active":"2024-05-28T06:05:29.467422+0000","last_became_peered":"2024-05-28T06:05:29.467422+0000","last_unstale":"2024-05-28T06:05:31.756136+0000","last_undegraded":"2024-05-28T06:05:31.756136+0000","last_fullsized":"2024-05-28T06:05:31.756136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:42:03.958730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.754971+0000","last_change":"2024-05-28T06:05:30.485976+0000","last_active":"2024-05-28T06:05:31.754971+0000","last_peered":"2024-05-28T06:05:31.754971+0000","last_clean":"2024-05-28T06:05:31.754971+0000","last_became_active":"2024-05-28T06:05:30.485689+0000","last_became_peered":"2024-05-28T06:05:30.485689+0000","last_unstale":"2024-05-28T06:05:31.754971+0000","last_undegraded":"2024-05-28T06:05:31.754971+0000","last_fullsized":"2024-05-28T06:05:31.754971+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:22:39.599053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.488393+0000","last_change":"2024-05-28T06:05:29.471119+0000","last_active":"2024-05-28T06:05:31.488393+0000","last_peered":"2024-05-28T06:05:31.488393+0000","last_clean":"2024-05-28T06:05:31.488393+0000","last_became_active":"2024-05-28T06:05:29.470884+0000","last_became_peered":"2024-05-28T06:05:29.470884+0000","last_unstale":"2024-05-28T06:05:31.488393+0000","last_undegraded":"2024-05-28T06:05:31.488393+0000","last_fullsized":"2024-05-28T06:05:31.488393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:57:51.752948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.757909+0000","last_change":"2024-05-28T06:05:30.471622+0000","last_active":"2024-05-28T06:05:31.757909+0000","last_peered":"2024-05-28T06:05:31.757909+0000","last_clean":"2024-05-28T06:05:31.757909+0000","last_became_active":"2024-05-28T06:05:30.471379+0000","last_became_peered":"2024-05-28T06:05:30.471379+0000","last_unstale":"2024-05-28T06:05:31.757909+0000","last_undegraded":"2024-05-28T06:05:31.757909+0000","last_fullsized":"2024-05-28T06:05:31.757909+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:21:35.302971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"33'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.478413+0000","last_change":"2024-05-28T06:05:29.449898+0000","last_active":"2024-05-28T06:05:31.478413+0000","last_peered":"2024-05-28T06:05:31.478413+0000","last_clean":"2024-05-28T06:05:31.478413+0000","last_became_active":"2024-05-28T06:05:29.449322+0000","last_became_peered":"2024-05-28T06:05:29.449322+0000","last_unstale":"2024-05-28T06:05:31.478413+0000","last_undegraded":"2024-05-28T06:05:31.478413+0000","last_fullsized":"2024-05-28T06:05:31.478413+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:11:47.259186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.477884+0000","last_change":"2024-05-28T06:05:30.487595+0000","last_active":"2024-05-28T06:05:31.477884+0000","last_peered":"2024-05-28T06:05:31.477884+0000","last_clean":"2024-05-28T06:05:31.477884+0000","last_became_active":"2024-05-28T06:05:30.485508+0000","last_became_peered":"2024-05-28T06:05:30.485508+0000","last_unstale":"2024-05-28T06:05:31.477884+0000","last_undegraded":"2024-05-28T06:05:31.477884+0000","last_fullsized":"2024-05-28T06:05:31.477884+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:41:24.956597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.487750+0000","last_change":"2024-05-28T06:05:29.451001+0000","last_active":"2024-05-28T06:05:31.487750+0000","last_peered":"2024-05-28T06:05:31.487750+0000","last_clean":"2024-05-28T06:05:31.487750+0000","last_became_active":"2024-05-28T06:05:29.450717+0000","last_became_peered":"2024-05-28T06:05:29.450717+0000","last_unstale":"2024-05-28T06:05:31.487750+0000","last_undegraded":"2024-05-28T06:05:31.487750+0000","last_fullsized":"2024-05-28T06:05:31.487750+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:58:31.330922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.753649+0000","last_change":"2024-05-28T06:05:30.479494+0000","last_active":"2024-05-28T06:05:31.753649+0000","last_peered":"2024-05-28T06:05:31.753649+0000","last_clean":"2024-05-28T06:05:31.753649+0000","last_became_active":"2024-05-28T06:05:30.479234+0000","last_became_peered":"2024-05-28T06:05:30.479234+0000","last_unstale":"2024-05-28T06:05:31.753649+0000","last_undegraded":"2024-05-28T06:05:31.753649+0000","last_fullsized":"2024-05-28T06:05:31.753649+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:33:36.484602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.478259+0000","last_change":"2024-05-28T06:05:29.469461+0000","last_active":"2024-05-28T06:05:31.478259+0000","last_peered":"2024-05-28T06:05:31.478259+0000","last_clean":"2024-05-28T06:05:31.478259+0000","last_became_active":"2024-05-28T06:05:29.469114+0000","last_became_peered":"2024-05-28T06:05:29.469114+0000","last_unstale":"2024-05-28T06:05:31.478259+0000","last_undegraded":"2024-05-28T06:05:31.478259+0000","last_fullsized":"2024-05-28T06:05:31.478259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:12:38.485450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.478074+0000","last_change":"2024-05-28T06:05:30.472808+0000","last_active":"2024-05-28T06:05:31.478074+0000","last_peered":"2024-05-28T06:05:31.478074+0000","last_clean":"2024-05-28T06:05:31.478074+0000","last_became_active":"2024-05-28T06:05:30.472492+0000","last_became_peered":"2024-05-28T06:05:30.472492+0000","last_unstale":"2024-05-28T06:05:31.478074+0000","last_undegraded":"2024-05-28T06:05:31.478074+0000","last_fullsized":"2024-05-28T06:05:31.478074+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:14:29.844730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.487917+0000","last_change":"2024-05-28T06:05:29.469913+0000","last_active":"2024-05-28T06:05:31.487917+0000","last_peered":"2024-05-28T06:05:31.487917+0000","last_clean":"2024-05-28T06:05:31.487917+0000","last_became_active":"2024-05-28T06:05:29.469460+0000","last_became_peered":"2024-05-28T06:05:29.469460+0000","last_unstale":"2024-05-28T06:05:31.487917+0000","last_undegraded":"2024-05-28T06:05:31.487917+0000","last_fullsized":"2024-05-28T06:05:31.487917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:11:47.259186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.752711+0000","last_change":"2024-05-28T06:05:30.483504+0000","last_active":"2024-05-28T06:05:31.752711+0000","last_peered":"2024-05-28T06:05:31.752711+0000","last_clean":"2024-05-28T06:05:31.752711+0000","last_became_active":"2024-05-28T06:05:30.483140+0000","last_became_peered":"2024-05-28T06:05:30.483140+0000","last_unstale":"2024-05-28T06:05:31.752711+0000","last_undegraded":"2024-05-28T06:05:31.752711+0000","last_fullsized":"2024-05-28T06:05:31.752711+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:38:12.995079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.482596+0000","last_change":"2024-05-28T06:05:29.469678+0000","last_active":"2024-05-28T06:05:31.482596+0000","last_peered":"2024-05-28T06:05:31.482596+0000","last_clean":"2024-05-28T06:05:31.482596+0000","last_became_active":"2024-05-28T06:05:29.469209+0000","last_became_peered":"2024-05-28T06:05:29.469209+0000","last_unstale":"2024-05-28T06:05:31.482596+0000","last_undegraded":"2024-05-28T06:05:31.482596+0000","last_fullsized":"2024-05-28T06:05:31.482596+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:22:38.584903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.482651+0000","last_change":"2024-05-28T06:05:30.474607+0000","last_active":"2024-05-28T06:05:31.482651+0000","last_peered":"2024-05-28T06:05:31.482651+0000","last_clean":"2024-05-28T06:05:31.482651+0000","last_became_active":"2024-05-28T06:05:30.474286+0000","last_became_peered":"2024-05-28T06:05:30.474286+0000","last_unstale":"2024-05-28T06:05:31.482651+0000","last_undegraded":"2024-05-28T06:05:31.482651+0000","last_fullsized":"2024-05-28T06:05:31.482651+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:37:19.547704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.753119+0000","last_change":"2024-05-28T06:05:29.465063+0000","last_active":"2024-05-28T06:05:31.753119+0000","last_peered":"2024-05-28T06:05:31.753119+0000","last_clean":"2024-05-28T06:05:31.753119+0000","last_became_active":"2024-05-28T06:05:29.464579+0000","last_became_peered":"2024-05-28T06:05:29.464579+0000","last_unstale":"2024-05-28T06:05:31.753119+0000","last_undegraded":"2024-05-28T06:05:31.753119+0000","last_fullsized":"2024-05-28T06:05:31.753119+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:42:03.958730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.755773+0000","last_change":"2024-05-28T06:05:30.479434+0000","last_active":"2024-05-28T06:05:31.755773+0000","last_peered":"2024-05-28T06:05:31.755773+0000","last_clean":"2024-05-28T06:05:31.755773+0000","last_became_active":"2024-05-28T06:05:30.479148+0000","last_became_peered":"2024-05-28T06:05:30.479148+0000","last_unstale":"2024-05-28T06:05:31.755773+0000","last_undegraded":"2024-05-28T06:05:31.755773+0000","last_fullsized":"2024-05-28T06:05:31.755773+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:56:43.306179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.753427+0000","last_change":"2024-05-28T06:05:29.461542+0000","last_active":"2024-05-28T06:05:31.753427+0000","last_peered":"2024-05-28T06:05:31.753427+0000","last_clean":"2024-05-28T06:05:31.753427+0000","last_became_active":"2024-05-28T06:05:29.461155+0000","last_became_peered":"2024-05-28T06:05:29.461155+0000","last_unstale":"2024-05-28T06:05:31.753427+0000","last_undegraded":"2024-05-28T06:05:31.753427+0000","last_fullsized":"2024-05-28T06:05:31.753427+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:12:38.485450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":194,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.755508+0000","last_change":"2024-05-28T06:04:01.039129+0000","last_active":"2024-05-28T06:05:31.755508+0000","last_peered":"2024-05-28T06:05:31.755508+0000","last_clean":"2024-05-28T06:05:31.755508+0000","last_became_active":"2024-05-28T06:04:01.033906+0000","last_became_peered":"2024-05-28T06:04:01.033906+0000","last_unstale":"2024-05-28T06:05:31.755508+0000","last_undegraded":"2024-05-28T06:05:31.755508+0000","last_fullsized":"2024-05-28T06:05:31.755508+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:03:59.864237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:03:59.864237+0000","last_clean_scrub_stamp":"2024-05-28T06:03:59.864237+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:17:36.440245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574870,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27968,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343872,"statfs":{"total":96636764160,"available":96608124928,"internally_reserved":0,"allocated":970752,"data_stored":737074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574872,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27396,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":380928,"data_stored":147232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574870,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27396,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":380928,"data_stored":147288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574870,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":380928,"data_stored":146720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574872,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27408,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344432,"statfs":{"total":96636764160,"available":96608698368,"internally_reserved":0,"allocated":389120,"data_stored":148250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574871,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27396,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":376832,"data_stored":146686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574871,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":389120,"data_stored":148250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574870,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27964,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343876,"statfs":{"total":96636764160,"available":96608129024,"internally_reserved":0,"allocated":966656,"data_stored":736984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-28T06:05:35.905 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-05-28T06:05:36.145 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.143+0000 7f7b05600640 1 Processor -- start 2024-05-28T06:05:36.145 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.143+0000 7f7b05600640 1 -- start start 2024-05-28T06:05:36.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.145+0000 7f7b05600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f7b00145320 0x7f7b00071cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:36.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.145+0000 7f7b05600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f7b00072200 0x7f7b000b3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:36.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.145+0000 7f7b05600640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b000cf0c0 con 0x7f7b000b41e0 2024-05-28T06:05:36.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.145+0000 7f7b05600640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f7b000b4a20 con 0x7f7b00145320 2024-05-28T06:05:36.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.145+0000 7f7b05600640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f7b000b4d00 con 0x7f7b00072200 2024-05-28T06:05:36.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.145+0000 7f7aff400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f7b000b41e0 0x7f7b00160240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:40312/0 (socket says 172.21.15.96:40312) 2024-05-28T06:05:36.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.145+0000 7f7afea00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f7b00145320 0x7f7b00071cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:36.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.145+0000 7f7aff400640 1 -- 172.21.15.96:0/2941280509 learned_addr learned my addr 172.21.15.96:0/2941280509 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:05:36.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.145+0000 7f7afe000640 1 --2- 172.21.15.96:0/2941280509 >> v2:172.21.15.123:3300/0 conn(0x7f7b00072200 0x7f7b000b3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:36.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.145+0000 7f7afea00640 1 -- 172.21.15.96:0/2941280509 >> v2:172.21.15.123:3300/0 conn(0x7f7b00072200 msgr2=0x7f7b000b3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:36.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.145+0000 7f7afea00640 1 --2- 172.21.15.96:0/2941280509 >> v2:172.21.15.123:3300/0 conn(0x7f7b00072200 0x7f7b000b3ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:36.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.145+0000 7f7afea00640 1 -- 172.21.15.96:0/2941280509 >> v1:172.21.15.96:6789/0 conn(0x7f7b000b41e0 legacy=0x7f7b00160240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:36.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.146+0000 7f7afd600640 1 -- 172.21.15.96:0/2941280509 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1614660482 0 0) 0x7f7b000cf0c0 con 0x7f7b000b41e0 2024-05-28T06:05:36.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.146+0000 7f7afea00640 1 -- 172.21.15.96:0/2941280509 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b000b4fe0 con 0x7f7b00145320 2024-05-28T06:05:36.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.147+0000 7f7afea00640 1 --2- 172.21.15.96:0/2941280509 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7b00145320 0x7f7b00071cc0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f7b0006db90 tx=0x7f7af002a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14fc4a8cab13c94a server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:36.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.147+0000 7f7afd600640 1 -- 172.21.15.96:0/2941280509 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7af0037070 con 0x7f7b00145320 2024-05-28T06:05:36.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.147+0000 7f7afd600640 1 -- 172.21.15.96:0/2941280509 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7af0032440 con 0x7f7b00145320 2024-05-28T06:05:36.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.147+0000 7f7afd600640 1 -- 172.21.15.96:0/2941280509 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7af0032810 con 0x7f7b00145320 2024-05-28T06:05:36.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.147+0000 7f7b05600640 1 -- 172.21.15.96:0/2941280509 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7b00145320 msgr2=0x7f7b00071cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:36.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.147+0000 7f7b05600640 1 --2- 172.21.15.96:0/2941280509 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7b00145320 0x7f7b00071cc0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f7b0006db90 tx=0x7f7af002a1a0 comp rx=0 tx=0).stop 2024-05-28T06:05:36.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.147+0000 7f7b05600640 1 -- 172.21.15.96:0/2941280509 shutdown_connections 2024-05-28T06:05:36.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.147+0000 7f7b05600640 1 --2- 172.21.15.96:0/2941280509 >> v2:172.21.15.123:3300/0 conn(0x7f7b00072200 0x7f7b000b3ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:36.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.147+0000 7f7b05600640 1 --2- 172.21.15.96:0/2941280509 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7b00145320 0x7f7b00071cc0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:36.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.147+0000 7f7b05600640 1 -- 172.21.15.96:0/2941280509 >> 172.21.15.96:0/2941280509 conn(0x7f7b00067ce0 msgr2=0x7f7b0006b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:36.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.148+0000 7f7b05600640 1 -- 172.21.15.96:0/2941280509 shutdown_connections 2024-05-28T06:05:36.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.148+0000 7f7b05600640 1 -- 172.21.15.96:0/2941280509 wait complete. 2024-05-28T06:05:36.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.149+0000 7f7b05600640 1 Processor -- start 2024-05-28T06:05:36.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.149+0000 7f7b05600640 1 -- start start 2024-05-28T06:05:36.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.150+0000 7f7b05600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7b00072200 0x7f7b0013d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:36.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.150+0000 7f7b05600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7b000b41e0 0x7f7b001f4520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:36.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.150+0000 7f7b05600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7b00145320 0x7f7b0013eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:36.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.150+0000 7f7b05600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f7b000b4650 con 0x7f7b00145320 2024-05-28T06:05:36.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.150+0000 7f7b05600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f7b00145700 con 0x7f7b000b41e0 2024-05-28T06:05:36.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.150+0000 7f7b05600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f7b000b47d0 con 0x7f7b00072200 2024-05-28T06:05:36.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.151+0000 7f7aff400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7b00145320 0x7f7b0013eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:36.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.151+0000 7f7aff400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7b00145320 0x7f7b0013eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:50916/0 (socket says 172.21.15.96:50916) 2024-05-28T06:05:36.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.151+0000 7f7aff400640 1 -- 172.21.15.96:0/1243889474 learned_addr learned my addr 172.21.15.96:0/1243889474 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:36.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.151+0000 7f7afe000640 1 --2- 172.21.15.96:0/1243889474 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7b000b41e0 0x7f7b001f4520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:36.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.151+0000 7f7afea00640 1 --2- 172.21.15.96:0/1243889474 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7b00072200 0x7f7b0013d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:36.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.151+0000 7f7aff400640 1 -- 172.21.15.96:0/1243889474 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7b00072200 msgr2=0x7f7b0013d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:36.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.151+0000 7f7aff400640 1 --2- 172.21.15.96:0/1243889474 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7b00072200 0x7f7b0013d010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:36.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.151+0000 7f7aff400640 1 -- 172.21.15.96:0/1243889474 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7b000b41e0 msgr2=0x7f7b001f4520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:36.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.151+0000 7f7aff400640 1 --2- 172.21.15.96:0/1243889474 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7b000b41e0 0x7f7b001f4520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:36.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.151+0000 7f7aff400640 1 -- 172.21.15.96:0/1243889474 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b001f4f00 con 0x7f7b00145320 2024-05-28T06:05:36.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.151+0000 7f7aff400640 1 --2- 172.21.15.96:0/1243889474 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7b00145320 0x7f7b0013eb60 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f7af800c7a0 tx=0x7f7af800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:36.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.152+0000 7f7af6a00640 1 -- 172.21.15.96:0/1243889474 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7af8018070 con 0x7f7b00145320 2024-05-28T06:05:36.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.152+0000 7f7af6a00640 1 -- 172.21.15.96:0/1243889474 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7af800a5e0 con 0x7f7b00145320 2024-05-28T06:05:36.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.152+0000 7f7b05600640 1 -- 172.21.15.96:0/1243889474 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b001f51e0 con 0x7f7b00145320 2024-05-28T06:05:36.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.152+0000 7f7af6a00640 1 -- 172.21.15.96:0/1243889474 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7af800aa10 con 0x7f7b00145320 2024-05-28T06:05:36.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.152+0000 7f7b05600640 1 -- 172.21.15.96:0/1243889474 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b0013d550 con 0x7f7b00145320 2024-05-28T06:05:36.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.153+0000 7f7b05600640 1 -- 172.21.15.96:0/1243889474 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b000689e0 con 0x7f7b00145320 2024-05-28T06:05:36.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.154+0000 7f7af6a00640 1 -- 172.21.15.96:0/1243889474 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7af801b070 con 0x7f7b00145320 2024-05-28T06:05:36.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.155+0000 7f7af6a00640 1 --2- 172.21.15.96:0/1243889474 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7adc06c8f0 0x7f7adc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:36.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.155+0000 7f7af6a00640 1 -- 172.21.15.96:0/1243889474 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f7af808b1c0 con 0x7f7b00145320 2024-05-28T06:05:36.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.155+0000 7f7afea00640 1 --2- 172.21.15.96:0/1243889474 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7adc06c8f0 0x7f7adc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:36.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.157+0000 7f7afea00640 1 --2- 172.21.15.96:0/1243889474 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7adc06c8f0 0x7f7adc06edb0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f7af0034000 tx=0x7f7af002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:36.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.159+0000 7f7af6a00640 1 -- 172.21.15.96:0/1243889474 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7af8059550 con 0x7f7b00145320 2024-05-28T06:05:36.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.425+0000 7f7b05600640 1 -- 172.21.15.96:0/1243889474 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7f7b000b34e0 con 0x7f7b00145320 2024-05-28T06:05:36.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.427+0000 7f7af6a00640 1 -- 172.21.15.96:0/1243889474 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1013 (secure 0 0 0) 0x7f7af8058e50 con 0x7f7b00145320 2024-05-28T06:05:36.428 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:05:36.428 INFO:teuthology.orchestra.run.smithi096.stdout:{"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":109,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":8,"num_up_osds":8,"osd_up_since":1716876238,"num_in_osds":8,"osd_in_since":1716876229,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":24,"data_bytes":592778,"bytes_used":225619968,"bytes_avail":772868493312,"bytes_total":773094113280,"write_bytes_sec":2355,"read_op_per_sec":0,"write_op_per_sec":7},"fsmap":{"epoch":15,"id":2,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":2,"rank":0,"name":"d","status":"up:active","gid":5238}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-05-28T06:05:17.506749+0000","services":{}},"progress_events":{}} 2024-05-28T06:05:36.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.429+0000 7f7b05600640 1 -- 172.21.15.96:0/1243889474 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7adc06c8f0 msgr2=0x7f7adc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:36.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.429+0000 7f7b05600640 1 --2- 172.21.15.96:0/1243889474 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7adc06c8f0 0x7f7adc06edb0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f7af0034000 tx=0x7f7af002e000 comp rx=0 tx=0).stop 2024-05-28T06:05:36.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.429+0000 7f7b05600640 1 -- 172.21.15.96:0/1243889474 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7b00145320 msgr2=0x7f7b0013eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:36.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.429+0000 7f7b05600640 1 --2- 172.21.15.96:0/1243889474 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7b00145320 0x7f7b0013eb60 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f7af800c7a0 tx=0x7f7af800cc60 comp rx=0 tx=0).stop 2024-05-28T06:05:36.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.430+0000 7f7b05600640 1 -- 172.21.15.96:0/1243889474 shutdown_connections 2024-05-28T06:05:36.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.430+0000 7f7b05600640 1 --2- 172.21.15.96:0/1243889474 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7adc06c8f0 0x7f7adc06edb0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:36.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.430+0000 7f7b05600640 1 --2- 172.21.15.96:0/1243889474 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7b00145320 0x7f7b0013eb60 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:36.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.430+0000 7f7b05600640 1 --2- 172.21.15.96:0/1243889474 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7b000b41e0 0x7f7b001f4520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:36.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.430+0000 7f7b05600640 1 --2- 172.21.15.96:0/1243889474 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7b00072200 0x7f7b0013d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:36.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.430+0000 7f7b05600640 1 -- 172.21.15.96:0/1243889474 >> 172.21.15.96:0/1243889474 conn(0x7f7b00067ce0 msgr2=0x7f7b0015cce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:36.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.430+0000 7f7b05600640 1 -- 172.21.15.96:0/1243889474 shutdown_connections 2024-05-28T06:05:36.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.430+0000 7f7b05600640 1 -- 172.21.15.96:0/1243889474 wait complete. 2024-05-28T06:05:36.449 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-05-28T06:05:36.449 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-28T06:05:36.640 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.640+0000 7f04b4400640 1 Processor -- start 2024-05-28T06:05:36.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.640+0000 7f04b4400640 1 -- start start 2024-05-28T06:05:36.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.640+0000 7f04b4400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f04ac06f860 0x7f04ac0b3150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:36.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.640+0000 7f04b4400640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04ac06ed10 con 0x7f04ac0d0bb0 2024-05-28T06:05:36.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.641+0000 7f04b4400640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04ac15cb00 con 0x7f04ac0b3690 2024-05-28T06:05:36.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.641+0000 7f04b4400640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f04ac0b3cc0 con 0x7f04ac06f860 2024-05-28T06:05:36.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.641+0000 7f04b1a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f04ac06f860 0x7f04ac0b3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:36.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.641+0000 7f04b2400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f04ac0d0bb0 0x7f04ac0ad130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:40322/0 (socket says 172.21.15.96:40322) 2024-05-28T06:05:36.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.641+0000 7f04b2400640 1 -- 172.21.15.96:0/270389296 learned_addr learned my addr 172.21.15.96:0/270389296 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:05:36.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.641+0000 7f04b1a00640 1 -- 172.21.15.96:0/270389296 >> v1:172.21.15.123:6789/0 conn(0x7f04ac0b3690 legacy=0x7f04ac0d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:36.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.641+0000 7f04b1a00640 1 -- 172.21.15.96:0/270389296 >> v1:172.21.15.96:6789/0 conn(0x7f04ac0d0bb0 legacy=0x7f04ac0ad130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:36.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.641+0000 7f04b1a00640 1 -- 172.21.15.96:0/270389296 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04ac0b3fa0 con 0x7f04ac06f860 2024-05-28T06:05:36.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.642+0000 7f04b1a00640 1 --2- 172.21.15.96:0/270389296 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f04ac06f860 0x7f04ac0b3150 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f04a00097a0 tx=0x7f04a002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcabe5117fec00df server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:36.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.643+0000 7f04abe00640 1 -- 172.21.15.96:0/270389296 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04a0037070 con 0x7f04ac06f860 2024-05-28T06:05:36.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.643+0000 7f04abe00640 1 -- 172.21.15.96:0/270389296 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04a0032440 con 0x7f04ac06f860 2024-05-28T06:05:36.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.643+0000 7f04abe00640 1 -- 172.21.15.96:0/270389296 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04a0032810 con 0x7f04ac06f860 2024-05-28T06:05:36.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.643+0000 7f04b4400640 1 -- 172.21.15.96:0/270389296 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f04ac06f860 msgr2=0x7f04ac0b3150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:36.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.643+0000 7f04b4400640 1 --2- 172.21.15.96:0/270389296 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f04ac06f860 0x7f04ac0b3150 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f04a00097a0 tx=0x7f04a002a1e0 comp rx=0 tx=0).stop 2024-05-28T06:05:36.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.643+0000 7f04b4400640 1 -- 172.21.15.96:0/270389296 shutdown_connections 2024-05-28T06:05:36.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.643+0000 7f04b4400640 1 --2- 172.21.15.96:0/270389296 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f04ac06f860 0x7f04ac0b3150 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:36.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.643+0000 7f04b4400640 1 -- 172.21.15.96:0/270389296 >> 172.21.15.96:0/270389296 conn(0x7f04ac06bae0 msgr2=0x7f04ac06bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:36.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.643+0000 7f04b4400640 1 -- 172.21.15.96:0/270389296 shutdown_connections 2024-05-28T06:05:36.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.644+0000 7f04b4400640 1 -- 172.21.15.96:0/270389296 wait complete. 2024-05-28T06:05:36.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.645+0000 7f04b4400640 1 Processor -- start 2024-05-28T06:05:36.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.645+0000 7f04b4400640 1 -- start start 2024-05-28T06:05:36.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.646+0000 7f04b4400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f04ac06f860 0x7f04ac0c9a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:36.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.646+0000 7f04b4400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f04ac0b3690 0x7f04ac1591b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:36.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.646+0000 7f04b4400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f04ac0d0bb0 0x7f04ac0cde30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:36.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.646+0000 7f04b4400640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f04ac0b4840 con 0x7f04ac0b3690 2024-05-28T06:05:36.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.646+0000 7f04b4400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f04ac0b4560 con 0x7f04ac06f860 2024-05-28T06:05:36.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.646+0000 7f04b4400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f04ac0ad840 con 0x7f04ac0d0bb0 2024-05-28T06:05:36.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.646+0000 7f04b1000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f04ac0b3690 0x7f04ac1591b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:36.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.647+0000 7f04b1000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f04ac0b3690 0x7f04ac1591b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:50922/0 (socket says 172.21.15.96:50922) 2024-05-28T06:05:36.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.647+0000 7f04b1000640 1 -- 172.21.15.96:0/1711544416 learned_addr learned my addr 172.21.15.96:0/1711544416 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:36.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.647+0000 7f04b1000640 1 -- 172.21.15.96:0/1711544416 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f04ac0d0bb0 msgr2=0x7f04ac0cde30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:36.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.647+0000 7f04b1000640 1 --2- 172.21.15.96:0/1711544416 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f04ac0d0bb0 0x7f04ac0cde30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:36.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.647+0000 7f04b1000640 1 -- 172.21.15.96:0/1711544416 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f04ac06f860 msgr2=0x7f04ac0c9a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:36.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.647+0000 7f04b1000640 1 --2- 172.21.15.96:0/1711544416 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f04ac06f860 0x7f04ac0c9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:36.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.647+0000 7f04b1000640 1 -- 172.21.15.96:0/1711544416 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04ac0c9f50 con 0x7f04ac0b3690 2024-05-28T06:05:36.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.647+0000 7f04b1000640 1 --2- 172.21.15.96:0/1711544416 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f04ac0b3690 0x7f04ac1591b0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f049800c430 tx=0x7f049800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:36.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.647+0000 7f04a9600640 1 -- 172.21.15.96:0/1711544416 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0498011790 con 0x7f04ac0b3690 2024-05-28T06:05:36.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.648+0000 7f04a9600640 1 -- 172.21.15.96:0/1711544416 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0498017070 con 0x7f04ac0b3690 2024-05-28T06:05:36.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.648+0000 7f04a9600640 1 -- 172.21.15.96:0/1711544416 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0498011b70 con 0x7f04ac0b3690 2024-05-28T06:05:36.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.648+0000 7f04b4400640 1 -- 172.21.15.96:0/1711544416 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04ac0ca220 con 0x7f04ac0b3690 2024-05-28T06:05:36.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.648+0000 7f04b4400640 1 -- 172.21.15.96:0/1711544416 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04ac15d2a0 con 0x7f04ac0b3690 2024-05-28T06:05:36.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.649+0000 7f04b4400640 1 -- 172.21.15.96:0/1711544416 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04ac0689e0 con 0x7f04ac0b3690 2024-05-28T06:05:36.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.650+0000 7f04a9600640 1 -- 172.21.15.96:0/1711544416 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0498011d10 con 0x7f04ac0b3690 2024-05-28T06:05:36.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.651+0000 7f04a9600640 1 --2- 172.21.15.96:0/1711544416 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f048c06c920 0x7f048c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:36.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.651+0000 7f04a9600640 1 -- 172.21.15.96:0/1711544416 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f049808b950 con 0x7f04ac0b3690 2024-05-28T06:05:36.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.652+0000 7f04b1a00640 1 --2- 172.21.15.96:0/1711544416 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f048c06c920 0x7f048c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:36.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.653+0000 7f04b1a00640 1 --2- 172.21.15.96:0/1711544416 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f048c06c920 0x7f048c06ede0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f04a0034000 tx=0x7f04a002c000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:36.657 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.656+0000 7f04a9600640 1 -- 172.21.15.96:0/1711544416 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0498059d70 con 0x7f04ac0b3690 2024-05-28T06:05:36.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.839+0000 7f04b4400640 1 -- 172.21.15.96:0/1711544416 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f04ac06dcb0 con 0x7f048c06c920 2024-05-28T06:05:36.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.858+0000 7f04a9600640 1 -- 172.21.15.96:0/1711544416 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251951 (secure 0 0 0) 0x7f04ac06dcb0 con 0x7f048c06c920 2024-05-28T06:05:36.863 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:05:36.864 INFO:teuthology.orchestra.run.smithi096.stderr:dumped all 2024-05-28T06:05:36.866 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.865+0000 7f04b4400640 1 -- 172.21.15.96:0/1711544416 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f048c06c920 msgr2=0x7f048c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:36.866 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.865+0000 7f04b4400640 1 --2- 172.21.15.96:0/1711544416 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f048c06c920 0x7f048c06ede0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f04a0034000 tx=0x7f04a002c000 comp rx=0 tx=0).stop 2024-05-28T06:05:36.866 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.865+0000 7f04b4400640 1 -- 172.21.15.96:0/1711544416 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f04ac0b3690 msgr2=0x7f04ac1591b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:36.866 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.865+0000 7f04b4400640 1 --2- 172.21.15.96:0/1711544416 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f04ac0b3690 0x7f04ac1591b0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f049800c430 tx=0x7f049800c8f0 comp rx=0 tx=0).stop 2024-05-28T06:05:36.866 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.865+0000 7f04b4400640 1 -- 172.21.15.96:0/1711544416 shutdown_connections 2024-05-28T06:05:36.866 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.865+0000 7f04b4400640 1 --2- 172.21.15.96:0/1711544416 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f048c06c920 0x7f048c06ede0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:36.866 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.865+0000 7f04b4400640 1 --2- 172.21.15.96:0/1711544416 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f04ac0d0bb0 0x7f04ac0cde30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:36.866 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.865+0000 7f04b4400640 1 --2- 172.21.15.96:0/1711544416 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f04ac0b3690 0x7f04ac1591b0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:36.866 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.865+0000 7f04b4400640 1 --2- 172.21.15.96:0/1711544416 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f04ac06f860 0x7f04ac0c9a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:36.866 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.865+0000 7f04b4400640 1 -- 172.21.15.96:0/1711544416 >> 172.21.15.96:0/1711544416 conn(0x7f04ac06bae0 msgr2=0x7f04ac0735c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:36.867 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.865+0000 7f04b4400640 1 -- 172.21.15.96:0/1711544416 shutdown_connections 2024-05-28T06:05:36.867 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:36.865+0000 7f04b4400640 1 -- 172.21.15.96:0/1711544416 wait complete. 2024-05-28T06:05:36.884 INFO:teuthology.orchestra.run.smithi096.stdout:{"pg_ready":true,"pg_map":{"version":74,"stamp":"2024-05-28T06:05:35.511482+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220332,"kb_used_data":4136,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754388,"statfs":{"total":773094113280,"available":772868493312,"internally_reserved":0,"allocated":4235264,"data_stored":2358484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.084983"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.754089+0000","last_change":"2024-05-28T06:05:30.477204+0000","last_active":"2024-05-28T06:05:31.754089+0000","last_peered":"2024-05-28T06:05:31.754089+0000","last_clean":"2024-05-28T06:05:31.754089+0000","last_became_active":"2024-05-28T06:05:30.476844+0000","last_became_peered":"2024-05-28T06:05:30.476844+0000","last_unstale":"2024-05-28T06:05:31.754089+0000","last_undegraded":"2024-05-28T06:05:31.754089+0000","last_fullsized":"2024-05-28T06:05:31.754089+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:42:04.972881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.753020+0000","last_change":"2024-05-28T06:05:30.477943+0000","last_active":"2024-05-28T06:05:31.753020+0000","last_peered":"2024-05-28T06:05:31.753020+0000","last_clean":"2024-05-28T06:05:31.753020+0000","last_became_active":"2024-05-28T06:05:30.476995+0000","last_became_peered":"2024-05-28T06:05:30.476995+0000","last_unstale":"2024-05-28T06:05:31.753020+0000","last_undegraded":"2024-05-28T06:05:31.753020+0000","last_fullsized":"2024-05-28T06:05:31.753020+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:56:43.306179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.758130+0000","last_change":"2024-05-28T06:05:30.483815+0000","last_active":"2024-05-28T06:05:31.758130+0000","last_peered":"2024-05-28T06:05:31.758130+0000","last_clean":"2024-05-28T06:05:31.758130+0000","last_became_active":"2024-05-28T06:05:30.483476+0000","last_became_peered":"2024-05-28T06:05:30.483476+0000","last_unstale":"2024-05-28T06:05:31.758130+0000","last_undegraded":"2024-05-28T06:05:31.758130+0000","last_fullsized":"2024-05-28T06:05:31.758130+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:43:33.343569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.487451+0000","last_change":"2024-05-28T06:05:30.481651+0000","last_active":"2024-05-28T06:05:31.487451+0000","last_peered":"2024-05-28T06:05:31.487451+0000","last_clean":"2024-05-28T06:05:31.487451+0000","last_became_active":"2024-05-28T06:05:30.477221+0000","last_became_peered":"2024-05-28T06:05:30.477221+0000","last_unstale":"2024-05-28T06:05:31.487451+0000","last_undegraded":"2024-05-28T06:05:31.487451+0000","last_fullsized":"2024-05-28T06:05:31.487451+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:51:16.699034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.755050+0000","last_change":"2024-05-28T06:05:30.486034+0000","last_active":"2024-05-28T06:05:31.755050+0000","last_peered":"2024-05-28T06:05:31.755050+0000","last_clean":"2024-05-28T06:05:31.755050+0000","last_became_active":"2024-05-28T06:05:30.485742+0000","last_became_peered":"2024-05-28T06:05:30.485742+0000","last_unstale":"2024-05-28T06:05:31.755050+0000","last_undegraded":"2024-05-28T06:05:31.755050+0000","last_fullsized":"2024-05-28T06:05:31.755050+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:16:43.593153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.752594+0000","last_change":"2024-05-28T06:05:30.474720+0000","last_active":"2024-05-28T06:05:31.752594+0000","last_peered":"2024-05-28T06:05:31.752594+0000","last_clean":"2024-05-28T06:05:31.752594+0000","last_became_active":"2024-05-28T06:05:30.474189+0000","last_became_peered":"2024-05-28T06:05:30.474189+0000","last_unstale":"2024-05-28T06:05:31.752594+0000","last_undegraded":"2024-05-28T06:05:31.752594+0000","last_fullsized":"2024-05-28T06:05:31.752594+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:26:58.818587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.756498+0000","last_change":"2024-05-28T06:05:30.487197+0000","last_active":"2024-05-28T06:05:31.756498+0000","last_peered":"2024-05-28T06:05:31.756498+0000","last_clean":"2024-05-28T06:05:31.756498+0000","last_became_active":"2024-05-28T06:05:30.486918+0000","last_became_peered":"2024-05-28T06:05:30.486918+0000","last_unstale":"2024-05-28T06:05:31.756498+0000","last_undegraded":"2024-05-28T06:05:31.756498+0000","last_fullsized":"2024-05-28T06:05:31.756498+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:27:30.648202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.754628+0000","last_change":"2024-05-28T06:05:30.489229+0000","last_active":"2024-05-28T06:05:31.754628+0000","last_peered":"2024-05-28T06:05:31.754628+0000","last_clean":"2024-05-28T06:05:31.754628+0000","last_became_active":"2024-05-28T06:05:30.488923+0000","last_became_peered":"2024-05-28T06:05:30.488923+0000","last_unstale":"2024-05-28T06:05:31.754628+0000","last_undegraded":"2024-05-28T06:05:31.754628+0000","last_fullsized":"2024-05-28T06:05:31.754628+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:58:04.060400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.483710+0000","last_change":"2024-05-28T06:05:30.487873+0000","last_active":"2024-05-28T06:05:31.483710+0000","last_peered":"2024-05-28T06:05:31.483710+0000","last_clean":"2024-05-28T06:05:31.483710+0000","last_became_active":"2024-05-28T06:05:30.487617+0000","last_became_peered":"2024-05-28T06:05:30.487617+0000","last_unstale":"2024-05-28T06:05:31.483710+0000","last_undegraded":"2024-05-28T06:05:31.483710+0000","last_fullsized":"2024-05-28T06:05:31.483710+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:24:54.475794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.487607+0000","last_change":"2024-05-28T06:05:30.473105+0000","last_active":"2024-05-28T06:05:31.487607+0000","last_peered":"2024-05-28T06:05:31.487607+0000","last_clean":"2024-05-28T06:05:31.487607+0000","last_became_active":"2024-05-28T06:05:30.472798+0000","last_became_peered":"2024-05-28T06:05:30.472798+0000","last_unstale":"2024-05-28T06:05:31.487607+0000","last_undegraded":"2024-05-28T06:05:31.487607+0000","last_fullsized":"2024-05-28T06:05:31.487607+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:17:42.817810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.755746+0000","last_change":"2024-05-28T06:05:30.486102+0000","last_active":"2024-05-28T06:05:31.755746+0000","last_peered":"2024-05-28T06:05:31.755746+0000","last_clean":"2024-05-28T06:05:31.755746+0000","last_became_active":"2024-05-28T06:05:30.485870+0000","last_became_peered":"2024-05-28T06:05:30.485870+0000","last_unstale":"2024-05-28T06:05:31.755746+0000","last_undegraded":"2024-05-28T06:05:31.755746+0000","last_fullsized":"2024-05-28T06:05:31.755746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:27:15.781576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.757226+0000","last_change":"2024-05-28T06:05:30.478015+0000","last_active":"2024-05-28T06:05:31.757226+0000","last_peered":"2024-05-28T06:05:31.757226+0000","last_clean":"2024-05-28T06:05:31.757226+0000","last_became_active":"2024-05-28T06:05:30.477760+0000","last_became_peered":"2024-05-28T06:05:30.477760+0000","last_unstale":"2024-05-28T06:05:31.757226+0000","last_undegraded":"2024-05-28T06:05:31.757226+0000","last_fullsized":"2024-05-28T06:05:31.757226+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:26:58.818587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.755386+0000","last_change":"2024-05-28T06:05:30.489911+0000","last_active":"2024-05-28T06:05:31.755386+0000","last_peered":"2024-05-28T06:05:31.755386+0000","last_clean":"2024-05-28T06:05:31.755386+0000","last_became_active":"2024-05-28T06:05:30.489638+0000","last_became_peered":"2024-05-28T06:05:30.489638+0000","last_unstale":"2024-05-28T06:05:31.755386+0000","last_undegraded":"2024-05-28T06:05:31.755386+0000","last_fullsized":"2024-05-28T06:05:31.755386+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:30:21.413453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.752710+0000","last_change":"2024-05-28T06:05:30.488321+0000","last_active":"2024-05-28T06:05:31.752710+0000","last_peered":"2024-05-28T06:05:31.752710+0000","last_clean":"2024-05-28T06:05:31.752710+0000","last_became_active":"2024-05-28T06:05:30.487979+0000","last_became_peered":"2024-05-28T06:05:30.487979+0000","last_unstale":"2024-05-28T06:05:31.752710+0000","last_undegraded":"2024-05-28T06:05:31.752710+0000","last_fullsized":"2024-05-28T06:05:31.752710+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:27:15.781576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.755011+0000","last_change":"2024-05-28T06:05:30.469583+0000","last_active":"2024-05-28T06:05:31.755011+0000","last_peered":"2024-05-28T06:05:31.755011+0000","last_clean":"2024-05-28T06:05:31.755011+0000","last_became_active":"2024-05-28T06:05:30.469299+0000","last_became_peered":"2024-05-28T06:05:30.469299+0000","last_unstale":"2024-05-28T06:05:31.755011+0000","last_undegraded":"2024-05-28T06:05:31.755011+0000","last_fullsized":"2024-05-28T06:05:31.755011+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:51:12.201028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.755507+0000","last_change":"2024-05-28T06:05:30.474489+0000","last_active":"2024-05-28T06:05:31.755507+0000","last_peered":"2024-05-28T06:05:31.755507+0000","last_clean":"2024-05-28T06:05:31.755507+0000","last_became_active":"2024-05-28T06:05:30.474215+0000","last_became_peered":"2024-05-28T06:05:30.474215+0000","last_unstale":"2024-05-28T06:05:31.755507+0000","last_undegraded":"2024-05-28T06:05:31.755507+0000","last_fullsized":"2024-05-28T06:05:31.755507+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:26:58.818587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.756720+0000","last_change":"2024-05-28T06:05:30.486573+0000","last_active":"2024-05-28T06:05:31.756720+0000","last_peered":"2024-05-28T06:05:31.756720+0000","last_clean":"2024-05-28T06:05:31.756720+0000","last_became_active":"2024-05-28T06:05:30.486307+0000","last_became_peered":"2024-05-28T06:05:30.486307+0000","last_unstale":"2024-05-28T06:05:31.756720+0000","last_undegraded":"2024-05-28T06:05:31.756720+0000","last_fullsized":"2024-05-28T06:05:31.756720+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:43:33.343569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.753543+0000","last_change":"2024-05-28T06:05:30.477781+0000","last_active":"2024-05-28T06:05:31.753543+0000","last_peered":"2024-05-28T06:05:31.753543+0000","last_clean":"2024-05-28T06:05:31.753543+0000","last_became_active":"2024-05-28T06:05:30.476599+0000","last_became_peered":"2024-05-28T06:05:30.476599+0000","last_unstale":"2024-05-28T06:05:31.753543+0000","last_undegraded":"2024-05-28T06:05:31.753543+0000","last_fullsized":"2024-05-28T06:05:31.753543+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:43:33.343569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.753222+0000","last_change":"2024-05-28T06:05:30.476553+0000","last_active":"2024-05-28T06:05:31.753222+0000","last_peered":"2024-05-28T06:05:31.753222+0000","last_clean":"2024-05-28T06:05:31.753222+0000","last_became_active":"2024-05-28T06:05:30.476290+0000","last_became_peered":"2024-05-28T06:05:30.476290+0000","last_unstale":"2024-05-28T06:05:31.753222+0000","last_undegraded":"2024-05-28T06:05:31.753222+0000","last_fullsized":"2024-05-28T06:05:31.753222+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:27:30.648202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.755494+0000","last_change":"2024-05-28T06:05:30.483659+0000","last_active":"2024-05-28T06:05:31.755494+0000","last_peered":"2024-05-28T06:05:31.755494+0000","last_clean":"2024-05-28T06:05:31.755494+0000","last_became_active":"2024-05-28T06:05:30.483403+0000","last_became_peered":"2024-05-28T06:05:30.483403+0000","last_unstale":"2024-05-28T06:05:31.755494+0000","last_undegraded":"2024-05-28T06:05:31.755494+0000","last_fullsized":"2024-05-28T06:05:31.755494+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:47:22.330606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.482709+0000","last_change":"2024-05-28T06:05:30.472992+0000","last_active":"2024-05-28T06:05:31.482709+0000","last_peered":"2024-05-28T06:05:31.482709+0000","last_clean":"2024-05-28T06:05:31.482709+0000","last_became_active":"2024-05-28T06:05:30.472725+0000","last_became_peered":"2024-05-28T06:05:30.472725+0000","last_unstale":"2024-05-28T06:05:31.482709+0000","last_undegraded":"2024-05-28T06:05:31.482709+0000","last_fullsized":"2024-05-28T06:05:31.482709+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:42:04.972881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.482893+0000","last_change":"2024-05-28T06:05:30.468698+0000","last_active":"2024-05-28T06:05:31.482893+0000","last_peered":"2024-05-28T06:05:31.482893+0000","last_clean":"2024-05-28T06:05:31.482893+0000","last_became_active":"2024-05-28T06:05:30.468120+0000","last_became_peered":"2024-05-28T06:05:30.468120+0000","last_unstale":"2024-05-28T06:05:31.482893+0000","last_undegraded":"2024-05-28T06:05:31.482893+0000","last_fullsized":"2024-05-28T06:05:31.482893+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:41:24.956597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.477622+0000","last_change":"2024-05-28T06:05:30.470896+0000","last_active":"2024-05-28T06:05:31.477622+0000","last_peered":"2024-05-28T06:05:31.477622+0000","last_clean":"2024-05-28T06:05:31.477622+0000","last_became_active":"2024-05-28T06:05:30.470316+0000","last_became_peered":"2024-05-28T06:05:30.470316+0000","last_unstale":"2024-05-28T06:05:31.477622+0000","last_undegraded":"2024-05-28T06:05:31.477622+0000","last_fullsized":"2024-05-28T06:05:31.477622+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:37:19.547704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.754580+0000","last_change":"2024-05-28T06:05:30.483993+0000","last_active":"2024-05-28T06:05:31.754580+0000","last_peered":"2024-05-28T06:05:31.754580+0000","last_clean":"2024-05-28T06:05:31.754580+0000","last_became_active":"2024-05-28T06:05:30.483709+0000","last_became_peered":"2024-05-28T06:05:30.483709+0000","last_unstale":"2024-05-28T06:05:31.754580+0000","last_undegraded":"2024-05-28T06:05:31.754580+0000","last_fullsized":"2024-05-28T06:05:31.754580+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:58:04.060400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.488135+0000","last_change":"2024-05-28T06:05:30.472224+0000","last_active":"2024-05-28T06:05:31.488135+0000","last_peered":"2024-05-28T06:05:31.488135+0000","last_clean":"2024-05-28T06:05:31.488135+0000","last_became_active":"2024-05-28T06:05:30.471939+0000","last_became_peered":"2024-05-28T06:05:30.471939+0000","last_unstale":"2024-05-28T06:05:31.488135+0000","last_undegraded":"2024-05-28T06:05:31.488135+0000","last_fullsized":"2024-05-28T06:05:31.488135+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:58:04.060400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.753861+0000","last_change":"2024-05-28T06:05:30.472699+0000","last_active":"2024-05-28T06:05:31.753861+0000","last_peered":"2024-05-28T06:05:31.753861+0000","last_clean":"2024-05-28T06:05:31.753861+0000","last_became_active":"2024-05-28T06:05:30.472399+0000","last_became_peered":"2024-05-28T06:05:30.472399+0000","last_unstale":"2024-05-28T06:05:31.753861+0000","last_undegraded":"2024-05-28T06:05:31.753861+0000","last_fullsized":"2024-05-28T06:05:31.753861+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:28:45.385929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.488123+0000","last_change":"2024-05-28T06:05:30.477001+0000","last_active":"2024-05-28T06:05:31.488123+0000","last_peered":"2024-05-28T06:05:31.488123+0000","last_clean":"2024-05-28T06:05:31.488123+0000","last_became_active":"2024-05-28T06:05:30.476725+0000","last_became_peered":"2024-05-28T06:05:30.476725+0000","last_unstale":"2024-05-28T06:05:31.488123+0000","last_undegraded":"2024-05-28T06:05:31.488123+0000","last_fullsized":"2024-05-28T06:05:31.488123+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:11:20.837156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.757661+0000","last_change":"2024-05-28T06:05:30.484736+0000","last_active":"2024-05-28T06:05:31.757661+0000","last_peered":"2024-05-28T06:05:31.757661+0000","last_clean":"2024-05-28T06:05:31.757661+0000","last_became_active":"2024-05-28T06:05:30.482722+0000","last_became_peered":"2024-05-28T06:05:30.482722+0000","last_unstale":"2024-05-28T06:05:31.757661+0000","last_undegraded":"2024-05-28T06:05:31.757661+0000","last_fullsized":"2024-05-28T06:05:31.757661+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:56:43.306179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.482975+0000","last_change":"2024-05-28T06:05:30.477442+0000","last_active":"2024-05-28T06:05:31.482975+0000","last_peered":"2024-05-28T06:05:31.482975+0000","last_clean":"2024-05-28T06:05:31.482975+0000","last_became_active":"2024-05-28T06:05:30.477224+0000","last_became_peered":"2024-05-28T06:05:30.477224+0000","last_unstale":"2024-05-28T06:05:31.482975+0000","last_undegraded":"2024-05-28T06:05:31.482975+0000","last_fullsized":"2024-05-28T06:05:31.482975+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:46:23.694062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.756903+0000","last_change":"2024-05-28T06:05:30.484668+0000","last_active":"2024-05-28T06:05:31.756903+0000","last_peered":"2024-05-28T06:05:31.756903+0000","last_clean":"2024-05-28T06:05:31.756903+0000","last_became_active":"2024-05-28T06:05:30.484339+0000","last_became_peered":"2024-05-28T06:05:30.484339+0000","last_unstale":"2024-05-28T06:05:31.756903+0000","last_undegraded":"2024-05-28T06:05:31.756903+0000","last_fullsized":"2024-05-28T06:05:31.756903+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:09:23.562166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.754848+0000","last_change":"2024-05-28T06:05:30.479119+0000","last_active":"2024-05-28T06:05:31.754848+0000","last_peered":"2024-05-28T06:05:31.754848+0000","last_clean":"2024-05-28T06:05:31.754848+0000","last_became_active":"2024-05-28T06:05:30.478742+0000","last_became_peered":"2024-05-28T06:05:30.478742+0000","last_unstale":"2024-05-28T06:05:31.754848+0000","last_undegraded":"2024-05-28T06:05:31.754848+0000","last_fullsized":"2024-05-28T06:05:31.754848+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:41:24.956597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.754808+0000","last_change":"2024-05-28T06:05:30.474293+0000","last_active":"2024-05-28T06:05:31.754808+0000","last_peered":"2024-05-28T06:05:31.754808+0000","last_clean":"2024-05-28T06:05:31.754808+0000","last_became_active":"2024-05-28T06:05:30.473563+0000","last_became_peered":"2024-05-28T06:05:30.473563+0000","last_unstale":"2024-05-28T06:05:31.754808+0000","last_undegraded":"2024-05-28T06:05:31.754808+0000","last_fullsized":"2024-05-28T06:05:31.754808+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:51:12.201028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.478000+0000","last_change":"2024-05-28T06:05:30.473138+0000","last_active":"2024-05-28T06:05:31.478000+0000","last_peered":"2024-05-28T06:05:31.478000+0000","last_clean":"2024-05-28T06:05:31.478000+0000","last_became_active":"2024-05-28T06:05:30.472302+0000","last_became_peered":"2024-05-28T06:05:30.472302+0000","last_unstale":"2024-05-28T06:05:31.478000+0000","last_undegraded":"2024-05-28T06:05:31.478000+0000","last_fullsized":"2024-05-28T06:05:31.478000+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:46:23.694062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"32'4","reported_seq":18,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.754387+0000","last_change":"2024-05-28T06:05:29.474199+0000","last_active":"2024-05-28T06:05:31.754387+0000","last_peered":"2024-05-28T06:05:31.754387+0000","last_clean":"2024-05-28T06:05:31.754387+0000","last_became_active":"2024-05-28T06:05:29.473863+0000","last_became_peered":"2024-05-28T06:05:29.473863+0000","last_unstale":"2024-05-28T06:05:31.754387+0000","last_undegraded":"2024-05-28T06:05:31.754387+0000","last_fullsized":"2024-05-28T06:05:31.754387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:47:21.316456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.754375+0000","last_change":"2024-05-28T06:05:30.473327+0000","last_active":"2024-05-28T06:05:31.754375+0000","last_peered":"2024-05-28T06:05:31.754375+0000","last_clean":"2024-05-28T06:05:31.754375+0000","last_became_active":"2024-05-28T06:05:30.470709+0000","last_became_peered":"2024-05-28T06:05:30.470709+0000","last_unstale":"2024-05-28T06:05:31.754375+0000","last_undegraded":"2024-05-28T06:05:31.754375+0000","last_fullsized":"2024-05-28T06:05:31.754375+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:37:19.547704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.478062+0000","last_change":"2024-05-28T06:05:29.460672+0000","last_active":"2024-05-28T06:05:31.478062+0000","last_peered":"2024-05-28T06:05:31.478062+0000","last_clean":"2024-05-28T06:05:31.478062+0000","last_became_active":"2024-05-28T06:05:29.459991+0000","last_became_peered":"2024-05-28T06:05:29.459991+0000","last_unstale":"2024-05-28T06:05:31.478062+0000","last_undegraded":"2024-05-28T06:05:31.478062+0000","last_fullsized":"2024-05-28T06:05:31.478062+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:57:51.752948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.483444+0000","last_change":"2024-05-28T06:05:30.477437+0000","last_active":"2024-05-28T06:05:31.483444+0000","last_peered":"2024-05-28T06:05:31.483444+0000","last_clean":"2024-05-28T06:05:31.483444+0000","last_became_active":"2024-05-28T06:05:30.477193+0000","last_became_peered":"2024-05-28T06:05:30.477193+0000","last_unstale":"2024-05-28T06:05:31.483444+0000","last_undegraded":"2024-05-28T06:05:31.483444+0000","last_fullsized":"2024-05-28T06:05:31.483444+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:14:29.844730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.753411+0000","last_change":"2024-05-28T06:05:29.456735+0000","last_active":"2024-05-28T06:05:31.753411+0000","last_peered":"2024-05-28T06:05:31.753411+0000","last_clean":"2024-05-28T06:05:31.753411+0000","last_became_active":"2024-05-28T06:05:29.456466+0000","last_became_peered":"2024-05-28T06:05:29.456466+0000","last_unstale":"2024-05-28T06:05:31.753411+0000","last_undegraded":"2024-05-28T06:05:31.753411+0000","last_fullsized":"2024-05-28T06:05:31.753411+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:41:23.942447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.488191+0000","last_change":"2024-05-28T06:05:30.476962+0000","last_active":"2024-05-28T06:05:31.488191+0000","last_peered":"2024-05-28T06:05:31.488191+0000","last_clean":"2024-05-28T06:05:31.488191+0000","last_became_active":"2024-05-28T06:05:30.476657+0000","last_became_peered":"2024-05-28T06:05:30.476657+0000","last_unstale":"2024-05-28T06:05:31.488191+0000","last_undegraded":"2024-05-28T06:05:31.488191+0000","last_fullsized":"2024-05-28T06:05:31.488191+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:51:12.201028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.755643+0000","last_change":"2024-05-28T06:05:29.451693+0000","last_active":"2024-05-28T06:05:31.755643+0000","last_peered":"2024-05-28T06:05:31.755643+0000","last_clean":"2024-05-28T06:05:31.755643+0000","last_became_active":"2024-05-28T06:05:29.451343+0000","last_became_peered":"2024-05-28T06:05:29.451343+0000","last_unstale":"2024-05-28T06:05:31.755643+0000","last_undegraded":"2024-05-28T06:05:31.755643+0000","last_fullsized":"2024-05-28T06:05:31.755643+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:42:15.256795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.756525+0000","last_change":"2024-05-28T06:05:30.477467+0000","last_active":"2024-05-28T06:05:31.756525+0000","last_peered":"2024-05-28T06:05:31.756525+0000","last_clean":"2024-05-28T06:05:31.756525+0000","last_became_active":"2024-05-28T06:05:30.477224+0000","last_became_peered":"2024-05-28T06:05:30.477224+0000","last_unstale":"2024-05-28T06:05:31.756525+0000","last_undegraded":"2024-05-28T06:05:31.756525+0000","last_fullsized":"2024-05-28T06:05:31.756525+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:21:35.302971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.482842+0000","last_change":"2024-05-28T06:05:29.463758+0000","last_active":"2024-05-28T06:05:31.482842+0000","last_peered":"2024-05-28T06:05:31.482842+0000","last_clean":"2024-05-28T06:05:31.482842+0000","last_became_active":"2024-05-28T06:05:29.460297+0000","last_became_peered":"2024-05-28T06:05:29.460297+0000","last_unstale":"2024-05-28T06:05:31.482842+0000","last_undegraded":"2024-05-28T06:05:31.482842+0000","last_fullsized":"2024-05-28T06:05:31.482842+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:05:43.269097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.756966+0000","last_change":"2024-05-28T06:05:30.484901+0000","last_active":"2024-05-28T06:05:31.756966+0000","last_peered":"2024-05-28T06:05:31.756966+0000","last_clean":"2024-05-28T06:05:31.756966+0000","last_became_active":"2024-05-28T06:05:30.484581+0000","last_became_peered":"2024-05-28T06:05:30.484581+0000","last_unstale":"2024-05-28T06:05:31.756966+0000","last_undegraded":"2024-05-28T06:05:31.756966+0000","last_fullsized":"2024-05-28T06:05:31.756966+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:39:33.123768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.482842+0000","last_change":"2024-05-28T06:05:29.462024+0000","last_active":"2024-05-28T06:05:31.482842+0000","last_peered":"2024-05-28T06:05:31.482842+0000","last_clean":"2024-05-28T06:05:31.482842+0000","last_became_active":"2024-05-28T06:05:29.460033+0000","last_became_peered":"2024-05-28T06:05:29.460033+0000","last_unstale":"2024-05-28T06:05:31.482842+0000","last_undegraded":"2024-05-28T06:05:31.482842+0000","last_fullsized":"2024-05-28T06:05:31.482842+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:38:57.785689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.482390+0000","last_change":"2024-05-28T06:05:30.468776+0000","last_active":"2024-05-28T06:05:31.482390+0000","last_peered":"2024-05-28T06:05:31.482390+0000","last_clean":"2024-05-28T06:05:31.482390+0000","last_became_active":"2024-05-28T06:05:30.468551+0000","last_became_peered":"2024-05-28T06:05:30.468551+0000","last_unstale":"2024-05-28T06:05:31.482390+0000","last_undegraded":"2024-05-28T06:05:31.482390+0000","last_fullsized":"2024-05-28T06:05:31.482390+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:33:36.484602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.752569+0000","last_change":"2024-05-28T06:05:29.456433+0000","last_active":"2024-05-28T06:05:31.752569+0000","last_peered":"2024-05-28T06:05:31.752569+0000","last_clean":"2024-05-28T06:05:31.752569+0000","last_became_active":"2024-05-28T06:05:29.456113+0000","last_became_peered":"2024-05-28T06:05:29.456113+0000","last_unstale":"2024-05-28T06:05:31.752569+0000","last_undegraded":"2024-05-28T06:05:31.752569+0000","last_fullsized":"2024-05-28T06:05:31.752569+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:12:52.117940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.486965+0000","last_change":"2024-05-28T06:05:30.478697+0000","last_active":"2024-05-28T06:05:31.486965+0000","last_peered":"2024-05-28T06:05:31.486965+0000","last_clean":"2024-05-28T06:05:31.486965+0000","last_became_active":"2024-05-28T06:05:30.477683+0000","last_became_peered":"2024-05-28T06:05:30.477683+0000","last_unstale":"2024-05-28T06:05:31.486965+0000","last_undegraded":"2024-05-28T06:05:31.486965+0000","last_fullsized":"2024-05-28T06:05:31.486965+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:24:54.475794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.482413+0000","last_change":"2024-05-28T06:05:29.462393+0000","last_active":"2024-05-28T06:05:31.482413+0000","last_peered":"2024-05-28T06:05:31.482413+0000","last_clean":"2024-05-28T06:05:31.482413+0000","last_became_active":"2024-05-28T06:05:29.461785+0000","last_became_peered":"2024-05-28T06:05:29.461785+0000","last_unstale":"2024-05-28T06:05:31.482413+0000","last_undegraded":"2024-05-28T06:05:31.482413+0000","last_fullsized":"2024-05-28T06:05:31.482413+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:58:46.608407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.757287+0000","last_change":"2024-05-28T06:05:30.481878+0000","last_active":"2024-05-28T06:05:31.757287+0000","last_peered":"2024-05-28T06:05:31.757287+0000","last_clean":"2024-05-28T06:05:31.757287+0000","last_became_active":"2024-05-28T06:05:30.481566+0000","last_became_peered":"2024-05-28T06:05:30.481566+0000","last_unstale":"2024-05-28T06:05:31.757287+0000","last_undegraded":"2024-05-28T06:05:31.757287+0000","last_fullsized":"2024-05-28T06:05:31.757287+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:27:15.781576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.753786+0000","last_change":"2024-05-28T06:05:29.461808+0000","last_active":"2024-05-28T06:05:31.753786+0000","last_peered":"2024-05-28T06:05:31.753786+0000","last_clean":"2024-05-28T06:05:31.753786+0000","last_became_active":"2024-05-28T06:05:29.461481+0000","last_became_peered":"2024-05-28T06:05:29.461481+0000","last_unstale":"2024-05-28T06:05:31.753786+0000","last_undegraded":"2024-05-28T06:05:31.753786+0000","last_fullsized":"2024-05-28T06:05:31.753786+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:46:22.679912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.753788+0000","last_change":"2024-05-28T06:05:30.475975+0000","last_active":"2024-05-28T06:05:31.753788+0000","last_peered":"2024-05-28T06:05:31.753788+0000","last_clean":"2024-05-28T06:05:31.753788+0000","last_became_active":"2024-05-28T06:05:30.475693+0000","last_became_peered":"2024-05-28T06:05:30.475693+0000","last_unstale":"2024-05-28T06:05:31.753788+0000","last_undegraded":"2024-05-28T06:05:31.753788+0000","last_fullsized":"2024-05-28T06:05:31.753788+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:09:23.562166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.478140+0000","last_change":"2024-05-28T06:05:29.460940+0000","last_active":"2024-05-28T06:05:31.478140+0000","last_peered":"2024-05-28T06:05:31.478140+0000","last_clean":"2024-05-28T06:05:31.478140+0000","last_became_active":"2024-05-28T06:05:29.460433+0000","last_became_peered":"2024-05-28T06:05:29.460433+0000","last_unstale":"2024-05-28T06:05:31.478140+0000","last_undegraded":"2024-05-28T06:05:31.478140+0000","last_fullsized":"2024-05-28T06:05:31.478140+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:32:34.183989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.753438+0000","last_change":"2024-05-28T06:05:30.472323+0000","last_active":"2024-05-28T06:05:31.753438+0000","last_peered":"2024-05-28T06:05:31.753438+0000","last_clean":"2024-05-28T06:05:31.753438+0000","last_became_active":"2024-05-28T06:05:30.472022+0000","last_became_peered":"2024-05-28T06:05:30.472022+0000","last_unstale":"2024-05-28T06:05:31.753438+0000","last_undegraded":"2024-05-28T06:05:31.753438+0000","last_fullsized":"2024-05-28T06:05:31.753438+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:21:35.302971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"32'3","reported_seq":17,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.757966+0000","last_change":"2024-05-28T06:05:29.462646+0000","last_active":"2024-05-28T06:05:31.757966+0000","last_peered":"2024-05-28T06:05:31.757966+0000","last_clean":"2024-05-28T06:05:31.757966+0000","last_became_active":"2024-05-28T06:05:29.462355+0000","last_became_peered":"2024-05-28T06:05:29.462355+0000","last_unstale":"2024-05-28T06:05:31.757966+0000","last_undegraded":"2024-05-28T06:05:31.757966+0000","last_fullsized":"2024-05-28T06:05:31.757966+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:37:18.533553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.757672+0000","last_change":"2024-05-28T06:05:30.483710+0000","last_active":"2024-05-28T06:05:31.757672+0000","last_peered":"2024-05-28T06:05:31.757672+0000","last_clean":"2024-05-28T06:05:31.757672+0000","last_became_active":"2024-05-28T06:05:30.483250+0000","last_became_peered":"2024-05-28T06:05:30.483250+0000","last_unstale":"2024-05-28T06:05:31.757672+0000","last_undegraded":"2024-05-28T06:05:31.757672+0000","last_fullsized":"2024-05-28T06:05:31.757672+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:38:12.995079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"32'6","reported_seq":20,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.753560+0000","last_change":"2024-05-28T06:05:29.458155+0000","last_active":"2024-05-28T06:05:31.753560+0000","last_peered":"2024-05-28T06:05:31.753560+0000","last_clean":"2024-05-28T06:05:31.753560+0000","last_became_active":"2024-05-28T06:05:29.457822+0000","last_became_peered":"2024-05-28T06:05:29.457822+0000","last_unstale":"2024-05-28T06:05:31.753560+0000","last_undegraded":"2024-05-28T06:05:31.753560+0000","last_fullsized":"2024-05-28T06:05:31.753560+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:37:18.533553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.487453+0000","last_change":"2024-05-28T06:05:30.479836+0000","last_active":"2024-05-28T06:05:31.487453+0000","last_peered":"2024-05-28T06:05:31.487453+0000","last_clean":"2024-05-28T06:05:31.487453+0000","last_became_active":"2024-05-28T06:05:30.479377+0000","last_became_peered":"2024-05-28T06:05:30.479377+0000","last_unstale":"2024-05-28T06:05:31.487453+0000","last_undegraded":"2024-05-28T06:05:31.487453+0000","last_fullsized":"2024-05-28T06:05:31.487453+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:39:39.679412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.483004+0000","last_change":"2024-05-28T06:05:29.464610+0000","last_active":"2024-05-28T06:05:31.483004+0000","last_peered":"2024-05-28T06:05:31.483004+0000","last_clean":"2024-05-28T06:05:31.483004+0000","last_became_active":"2024-05-28T06:05:29.464074+0000","last_became_peered":"2024-05-28T06:05:29.464074+0000","last_unstale":"2024-05-28T06:05:31.483004+0000","last_undegraded":"2024-05-28T06:05:31.483004+0000","last_fullsized":"2024-05-28T06:05:31.483004+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:30:20.399303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.752600+0000","last_change":"2024-05-28T06:05:30.474344+0000","last_active":"2024-05-28T06:05:31.752600+0000","last_peered":"2024-05-28T06:05:31.752600+0000","last_clean":"2024-05-28T06:05:31.752600+0000","last_became_active":"2024-05-28T06:05:30.474074+0000","last_became_peered":"2024-05-28T06:05:30.474074+0000","last_unstale":"2024-05-28T06:05:31.752600+0000","last_undegraded":"2024-05-28T06:05:31.752600+0000","last_fullsized":"2024-05-28T06:05:31.752600+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:39:33.123768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.486959+0000","last_change":"2024-05-28T06:05:29.463194+0000","last_active":"2024-05-28T06:05:31.486959+0000","last_peered":"2024-05-28T06:05:31.486959+0000","last_clean":"2024-05-28T06:05:31.486959+0000","last_became_active":"2024-05-28T06:05:29.462870+0000","last_became_peered":"2024-05-28T06:05:29.462870+0000","last_unstale":"2024-05-28T06:05:31.486959+0000","last_undegraded":"2024-05-28T06:05:31.486959+0000","last_fullsized":"2024-05-28T06:05:31.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:32:34.183989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.755042+0000","last_change":"2024-05-28T06:05:30.478298+0000","last_active":"2024-05-28T06:05:31.755042+0000","last_peered":"2024-05-28T06:05:31.755042+0000","last_clean":"2024-05-28T06:05:31.755042+0000","last_became_active":"2024-05-28T06:05:30.478018+0000","last_became_peered":"2024-05-28T06:05:30.478018+0000","last_unstale":"2024-05-28T06:05:31.755042+0000","last_undegraded":"2024-05-28T06:05:31.755042+0000","last_fullsized":"2024-05-28T06:05:31.755042+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:39:33.123768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.752817+0000","last_change":"2024-05-28T06:05:29.462298+0000","last_active":"2024-05-28T06:05:31.752817+0000","last_peered":"2024-05-28T06:05:31.752817+0000","last_clean":"2024-05-28T06:05:31.752817+0000","last_became_active":"2024-05-28T06:05:29.462022+0000","last_became_peered":"2024-05-28T06:05:29.462022+0000","last_unstale":"2024-05-28T06:05:31.752817+0000","last_undegraded":"2024-05-28T06:05:31.752817+0000","last_fullsized":"2024-05-28T06:05:31.752817+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:33:35.470452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.754851+0000","last_change":"2024-05-28T06:05:30.478395+0000","last_active":"2024-05-28T06:05:31.754851+0000","last_peered":"2024-05-28T06:05:31.754851+0000","last_clean":"2024-05-28T06:05:31.754851+0000","last_became_active":"2024-05-28T06:05:30.470732+0000","last_became_peered":"2024-05-28T06:05:30.470732+0000","last_unstale":"2024-05-28T06:05:31.754851+0000","last_undegraded":"2024-05-28T06:05:31.754851+0000","last_fullsized":"2024-05-28T06:05:31.754851+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:46:23.694062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"32'8","reported_seq":22,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.752843+0000","last_change":"2024-05-28T06:05:29.460103+0000","last_active":"2024-05-28T06:05:31.752843+0000","last_peered":"2024-05-28T06:05:31.752843+0000","last_clean":"2024-05-28T06:05:31.752843+0000","last_became_active":"2024-05-28T06:05:29.459788+0000","last_became_peered":"2024-05-28T06:05:29.459788+0000","last_unstale":"2024-05-28T06:05:31.752843+0000","last_undegraded":"2024-05-28T06:05:31.752843+0000","last_fullsized":"2024-05-28T06:05:31.752843+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:03:08.734203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.483476+0000","last_change":"2024-05-28T06:05:30.471442+0000","last_active":"2024-05-28T06:05:31.483476+0000","last_peered":"2024-05-28T06:05:31.483476+0000","last_clean":"2024-05-28T06:05:31.483476+0000","last_became_active":"2024-05-28T06:05:30.471189+0000","last_became_peered":"2024-05-28T06:05:30.471189+0000","last_unstale":"2024-05-28T06:05:31.483476+0000","last_undegraded":"2024-05-28T06:05:31.483476+0000","last_fullsized":"2024-05-28T06:05:31.483476+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:43:11.326702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.488303+0000","last_change":"2024-05-28T06:05:29.458728+0000","last_active":"2024-05-28T06:05:31.488303+0000","last_peered":"2024-05-28T06:05:31.488303+0000","last_clean":"2024-05-28T06:05:31.488303+0000","last_became_active":"2024-05-28T06:05:29.458154+0000","last_became_peered":"2024-05-28T06:05:29.458154+0000","last_unstale":"2024-05-28T06:05:31.488303+0000","last_undegraded":"2024-05-28T06:05:31.488303+0000","last_fullsized":"2024-05-28T06:05:31.488303+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:12:38.485450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.757818+0000","last_change":"2024-05-28T06:05:30.471043+0000","last_active":"2024-05-28T06:05:31.757818+0000","last_peered":"2024-05-28T06:05:31.757818+0000","last_clean":"2024-05-28T06:05:31.757818+0000","last_became_active":"2024-05-28T06:05:30.470782+0000","last_became_peered":"2024-05-28T06:05:30.470782+0000","last_unstale":"2024-05-28T06:05:31.757818+0000","last_undegraded":"2024-05-28T06:05:31.757818+0000","last_fullsized":"2024-05-28T06:05:31.757818+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:27:30.648202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.483361+0000","last_change":"2024-05-28T06:05:29.463122+0000","last_active":"2024-05-28T06:05:31.483361+0000","last_peered":"2024-05-28T06:05:31.483361+0000","last_clean":"2024-05-28T06:05:31.483361+0000","last_became_active":"2024-05-28T06:05:29.462362+0000","last_became_peered":"2024-05-28T06:05:29.462362+0000","last_unstale":"2024-05-28T06:05:31.483361+0000","last_undegraded":"2024-05-28T06:05:31.483361+0000","last_fullsized":"2024-05-28T06:05:31.483361+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:34:23.160698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.487454+0000","last_change":"2024-05-28T06:05:30.482239+0000","last_active":"2024-05-28T06:05:31.487454+0000","last_peered":"2024-05-28T06:05:31.487454+0000","last_clean":"2024-05-28T06:05:31.487454+0000","last_became_active":"2024-05-28T06:05:30.481947+0000","last_became_peered":"2024-05-28T06:05:30.481947+0000","last_unstale":"2024-05-28T06:05:31.487454+0000","last_undegraded":"2024-05-28T06:05:31.487454+0000","last_fullsized":"2024-05-28T06:05:31.487454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:04:45.854464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.483370+0000","last_change":"2024-05-28T06:05:29.450660+0000","last_active":"2024-05-28T06:05:31.483370+0000","last_peered":"2024-05-28T06:05:31.483370+0000","last_clean":"2024-05-28T06:05:31.483370+0000","last_became_active":"2024-05-28T06:05:29.446547+0000","last_became_peered":"2024-05-28T06:05:29.446547+0000","last_unstale":"2024-05-28T06:05:31.483370+0000","last_undegraded":"2024-05-28T06:05:31.483370+0000","last_fullsized":"2024-05-28T06:05:31.483370+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:33:40.689124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.755147+0000","last_change":"2024-05-28T06:05:30.478730+0000","last_active":"2024-05-28T06:05:31.755147+0000","last_peered":"2024-05-28T06:05:31.755147+0000","last_clean":"2024-05-28T06:05:31.755147+0000","last_became_active":"2024-05-28T06:05:30.478438+0000","last_became_peered":"2024-05-28T06:05:30.478438+0000","last_unstale":"2024-05-28T06:05:31.755147+0000","last_undegraded":"2024-05-28T06:05:31.755147+0000","last_fullsized":"2024-05-28T06:05:31.755147+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:05:44.283248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.483501+0000","last_change":"2024-05-28T06:05:29.461213+0000","last_active":"2024-05-28T06:05:31.483501+0000","last_peered":"2024-05-28T06:05:31.483501+0000","last_clean":"2024-05-28T06:05:31.483501+0000","last_became_active":"2024-05-28T06:05:29.460899+0000","last_became_peered":"2024-05-28T06:05:29.460899+0000","last_unstale":"2024-05-28T06:05:31.483501+0000","last_undegraded":"2024-05-28T06:05:31.483501+0000","last_fullsized":"2024-05-28T06:05:31.483501+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:16:42.579003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.754877+0000","last_change":"2024-05-28T06:05:30.484117+0000","last_active":"2024-05-28T06:05:31.754877+0000","last_peered":"2024-05-28T06:05:31.754877+0000","last_clean":"2024-05-28T06:05:31.754877+0000","last_became_active":"2024-05-28T06:05:30.483835+0000","last_became_peered":"2024-05-28T06:05:30.483835+0000","last_unstale":"2024-05-28T06:05:31.754877+0000","last_undegraded":"2024-05-28T06:05:31.754877+0000","last_fullsized":"2024-05-28T06:05:31.754877+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:41:32.997209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.478083+0000","last_change":"2024-05-28T06:05:29.447607+0000","last_active":"2024-05-28T06:05:31.478083+0000","last_peered":"2024-05-28T06:05:31.478083+0000","last_clean":"2024-05-28T06:05:31.478083+0000","last_became_active":"2024-05-28T06:05:29.447321+0000","last_became_peered":"2024-05-28T06:05:29.447321+0000","last_unstale":"2024-05-28T06:05:31.478083+0000","last_undegraded":"2024-05-28T06:05:31.478083+0000","last_fullsized":"2024-05-28T06:05:31.478083+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:47:21.316456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.755458+0000","last_change":"2024-05-28T06:05:30.484954+0000","last_active":"2024-05-28T06:05:31.755458+0000","last_peered":"2024-05-28T06:05:31.755458+0000","last_clean":"2024-05-28T06:05:31.755458+0000","last_became_active":"2024-05-28T06:05:30.484699+0000","last_became_peered":"2024-05-28T06:05:30.484699+0000","last_unstale":"2024-05-28T06:05:31.755458+0000","last_undegraded":"2024-05-28T06:05:31.755458+0000","last_fullsized":"2024-05-28T06:05:31.755458+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:10:49.765112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.753811+0000","last_change":"2024-05-28T06:05:29.463849+0000","last_active":"2024-05-28T06:05:31.753811+0000","last_peered":"2024-05-28T06:05:31.753811+0000","last_clean":"2024-05-28T06:05:31.753811+0000","last_became_active":"2024-05-28T06:05:29.463514+0000","last_became_peered":"2024-05-28T06:05:29.463514+0000","last_unstale":"2024-05-28T06:05:31.753811+0000","last_undegraded":"2024-05-28T06:05:31.753811+0000","last_fullsized":"2024-05-28T06:05:31.753811+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:10:48.750962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.754811+0000","last_change":"2024-05-28T06:05:30.485046+0000","last_active":"2024-05-28T06:05:31.754811+0000","last_peered":"2024-05-28T06:05:31.754811+0000","last_clean":"2024-05-28T06:05:31.754811+0000","last_became_active":"2024-05-28T06:05:30.484737+0000","last_became_peered":"2024-05-28T06:05:30.484737+0000","last_unstale":"2024-05-28T06:05:31.754811+0000","last_undegraded":"2024-05-28T06:05:31.754811+0000","last_fullsized":"2024-05-28T06:05:31.754811+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:34:24.174848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.482448+0000","last_change":"2024-05-28T06:05:29.450109+0000","last_active":"2024-05-28T06:05:31.482448+0000","last_peered":"2024-05-28T06:05:31.482448+0000","last_clean":"2024-05-28T06:05:31.482448+0000","last_became_active":"2024-05-28T06:05:29.449618+0000","last_became_peered":"2024-05-28T06:05:29.449618+0000","last_unstale":"2024-05-28T06:05:31.482448+0000","last_undegraded":"2024-05-28T06:05:31.482448+0000","last_fullsized":"2024-05-28T06:05:31.482448+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:41:31.983059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.755911+0000","last_change":"2024-05-28T06:05:30.470295+0000","last_active":"2024-05-28T06:05:31.755911+0000","last_peered":"2024-05-28T06:05:31.755911+0000","last_clean":"2024-05-28T06:05:31.755911+0000","last_became_active":"2024-05-28T06:05:30.470025+0000","last_became_peered":"2024-05-28T06:05:30.470025+0000","last_unstale":"2024-05-28T06:05:31.755911+0000","last_undegraded":"2024-05-28T06:05:31.755911+0000","last_fullsized":"2024-05-28T06:05:31.755911+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:38:12.995079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.756136+0000","last_change":"2024-05-28T06:05:29.467769+0000","last_active":"2024-05-28T06:05:31.756136+0000","last_peered":"2024-05-28T06:05:31.756136+0000","last_clean":"2024-05-28T06:05:31.756136+0000","last_became_active":"2024-05-28T06:05:29.467422+0000","last_became_peered":"2024-05-28T06:05:29.467422+0000","last_unstale":"2024-05-28T06:05:31.756136+0000","last_undegraded":"2024-05-28T06:05:31.756136+0000","last_fullsized":"2024-05-28T06:05:31.756136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:42:03.958730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.754971+0000","last_change":"2024-05-28T06:05:30.485976+0000","last_active":"2024-05-28T06:05:31.754971+0000","last_peered":"2024-05-28T06:05:31.754971+0000","last_clean":"2024-05-28T06:05:31.754971+0000","last_became_active":"2024-05-28T06:05:30.485689+0000","last_became_peered":"2024-05-28T06:05:30.485689+0000","last_unstale":"2024-05-28T06:05:31.754971+0000","last_undegraded":"2024-05-28T06:05:31.754971+0000","last_fullsized":"2024-05-28T06:05:31.754971+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:22:39.599053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.488393+0000","last_change":"2024-05-28T06:05:29.471119+0000","last_active":"2024-05-28T06:05:31.488393+0000","last_peered":"2024-05-28T06:05:31.488393+0000","last_clean":"2024-05-28T06:05:31.488393+0000","last_became_active":"2024-05-28T06:05:29.470884+0000","last_became_peered":"2024-05-28T06:05:29.470884+0000","last_unstale":"2024-05-28T06:05:31.488393+0000","last_undegraded":"2024-05-28T06:05:31.488393+0000","last_fullsized":"2024-05-28T06:05:31.488393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:57:51.752948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.757909+0000","last_change":"2024-05-28T06:05:30.471622+0000","last_active":"2024-05-28T06:05:31.757909+0000","last_peered":"2024-05-28T06:05:31.757909+0000","last_clean":"2024-05-28T06:05:31.757909+0000","last_became_active":"2024-05-28T06:05:30.471379+0000","last_became_peered":"2024-05-28T06:05:30.471379+0000","last_unstale":"2024-05-28T06:05:31.757909+0000","last_undegraded":"2024-05-28T06:05:31.757909+0000","last_fullsized":"2024-05-28T06:05:31.757909+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:21:35.302971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"33'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.478413+0000","last_change":"2024-05-28T06:05:29.449898+0000","last_active":"2024-05-28T06:05:31.478413+0000","last_peered":"2024-05-28T06:05:31.478413+0000","last_clean":"2024-05-28T06:05:31.478413+0000","last_became_active":"2024-05-28T06:05:29.449322+0000","last_became_peered":"2024-05-28T06:05:29.449322+0000","last_unstale":"2024-05-28T06:05:31.478413+0000","last_undegraded":"2024-05-28T06:05:31.478413+0000","last_fullsized":"2024-05-28T06:05:31.478413+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:11:47.259186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.477884+0000","last_change":"2024-05-28T06:05:30.487595+0000","last_active":"2024-05-28T06:05:31.477884+0000","last_peered":"2024-05-28T06:05:31.477884+0000","last_clean":"2024-05-28T06:05:31.477884+0000","last_became_active":"2024-05-28T06:05:30.485508+0000","last_became_peered":"2024-05-28T06:05:30.485508+0000","last_unstale":"2024-05-28T06:05:31.477884+0000","last_undegraded":"2024-05-28T06:05:31.477884+0000","last_fullsized":"2024-05-28T06:05:31.477884+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:41:24.956597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.487750+0000","last_change":"2024-05-28T06:05:29.451001+0000","last_active":"2024-05-28T06:05:31.487750+0000","last_peered":"2024-05-28T06:05:31.487750+0000","last_clean":"2024-05-28T06:05:31.487750+0000","last_became_active":"2024-05-28T06:05:29.450717+0000","last_became_peered":"2024-05-28T06:05:29.450717+0000","last_unstale":"2024-05-28T06:05:31.487750+0000","last_undegraded":"2024-05-28T06:05:31.487750+0000","last_fullsized":"2024-05-28T06:05:31.487750+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:58:31.330922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.753649+0000","last_change":"2024-05-28T06:05:30.479494+0000","last_active":"2024-05-28T06:05:31.753649+0000","last_peered":"2024-05-28T06:05:31.753649+0000","last_clean":"2024-05-28T06:05:31.753649+0000","last_became_active":"2024-05-28T06:05:30.479234+0000","last_became_peered":"2024-05-28T06:05:30.479234+0000","last_unstale":"2024-05-28T06:05:31.753649+0000","last_undegraded":"2024-05-28T06:05:31.753649+0000","last_fullsized":"2024-05-28T06:05:31.753649+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:33:36.484602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.478259+0000","last_change":"2024-05-28T06:05:29.469461+0000","last_active":"2024-05-28T06:05:31.478259+0000","last_peered":"2024-05-28T06:05:31.478259+0000","last_clean":"2024-05-28T06:05:31.478259+0000","last_became_active":"2024-05-28T06:05:29.469114+0000","last_became_peered":"2024-05-28T06:05:29.469114+0000","last_unstale":"2024-05-28T06:05:31.478259+0000","last_undegraded":"2024-05-28T06:05:31.478259+0000","last_fullsized":"2024-05-28T06:05:31.478259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:12:38.485450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.478074+0000","last_change":"2024-05-28T06:05:30.472808+0000","last_active":"2024-05-28T06:05:31.478074+0000","last_peered":"2024-05-28T06:05:31.478074+0000","last_clean":"2024-05-28T06:05:31.478074+0000","last_became_active":"2024-05-28T06:05:30.472492+0000","last_became_peered":"2024-05-28T06:05:30.472492+0000","last_unstale":"2024-05-28T06:05:31.478074+0000","last_undegraded":"2024-05-28T06:05:31.478074+0000","last_fullsized":"2024-05-28T06:05:31.478074+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:14:29.844730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.487917+0000","last_change":"2024-05-28T06:05:29.469913+0000","last_active":"2024-05-28T06:05:31.487917+0000","last_peered":"2024-05-28T06:05:31.487917+0000","last_clean":"2024-05-28T06:05:31.487917+0000","last_became_active":"2024-05-28T06:05:29.469460+0000","last_became_peered":"2024-05-28T06:05:29.469460+0000","last_unstale":"2024-05-28T06:05:31.487917+0000","last_undegraded":"2024-05-28T06:05:31.487917+0000","last_fullsized":"2024-05-28T06:05:31.487917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:11:47.259186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.752711+0000","last_change":"2024-05-28T06:05:30.483504+0000","last_active":"2024-05-28T06:05:31.752711+0000","last_peered":"2024-05-28T06:05:31.752711+0000","last_clean":"2024-05-28T06:05:31.752711+0000","last_became_active":"2024-05-28T06:05:30.483140+0000","last_became_peered":"2024-05-28T06:05:30.483140+0000","last_unstale":"2024-05-28T06:05:31.752711+0000","last_undegraded":"2024-05-28T06:05:31.752711+0000","last_fullsized":"2024-05-28T06:05:31.752711+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:38:12.995079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.482596+0000","last_change":"2024-05-28T06:05:29.469678+0000","last_active":"2024-05-28T06:05:31.482596+0000","last_peered":"2024-05-28T06:05:31.482596+0000","last_clean":"2024-05-28T06:05:31.482596+0000","last_became_active":"2024-05-28T06:05:29.469209+0000","last_became_peered":"2024-05-28T06:05:29.469209+0000","last_unstale":"2024-05-28T06:05:31.482596+0000","last_undegraded":"2024-05-28T06:05:31.482596+0000","last_fullsized":"2024-05-28T06:05:31.482596+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:22:38.584903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.482651+0000","last_change":"2024-05-28T06:05:30.474607+0000","last_active":"2024-05-28T06:05:31.482651+0000","last_peered":"2024-05-28T06:05:31.482651+0000","last_clean":"2024-05-28T06:05:31.482651+0000","last_became_active":"2024-05-28T06:05:30.474286+0000","last_became_peered":"2024-05-28T06:05:30.474286+0000","last_unstale":"2024-05-28T06:05:31.482651+0000","last_undegraded":"2024-05-28T06:05:31.482651+0000","last_fullsized":"2024-05-28T06:05:31.482651+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:37:19.547704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.753119+0000","last_change":"2024-05-28T06:05:29.465063+0000","last_active":"2024-05-28T06:05:31.753119+0000","last_peered":"2024-05-28T06:05:31.753119+0000","last_clean":"2024-05-28T06:05:31.753119+0000","last_became_active":"2024-05-28T06:05:29.464579+0000","last_became_peered":"2024-05-28T06:05:29.464579+0000","last_unstale":"2024-05-28T06:05:31.753119+0000","last_undegraded":"2024-05-28T06:05:31.753119+0000","last_fullsized":"2024-05-28T06:05:31.753119+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:42:03.958730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.755773+0000","last_change":"2024-05-28T06:05:30.479434+0000","last_active":"2024-05-28T06:05:31.755773+0000","last_peered":"2024-05-28T06:05:31.755773+0000","last_clean":"2024-05-28T06:05:31.755773+0000","last_became_active":"2024-05-28T06:05:30.479148+0000","last_became_peered":"2024-05-28T06:05:30.479148+0000","last_unstale":"2024-05-28T06:05:31.755773+0000","last_undegraded":"2024-05-28T06:05:31.755773+0000","last_fullsized":"2024-05-28T06:05:31.755773+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:56:43.306179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.753427+0000","last_change":"2024-05-28T06:05:29.461542+0000","last_active":"2024-05-28T06:05:31.753427+0000","last_peered":"2024-05-28T06:05:31.753427+0000","last_clean":"2024-05-28T06:05:31.753427+0000","last_became_active":"2024-05-28T06:05:29.461155+0000","last_became_peered":"2024-05-28T06:05:29.461155+0000","last_unstale":"2024-05-28T06:05:31.753427+0000","last_undegraded":"2024-05-28T06:05:31.753427+0000","last_fullsized":"2024-05-28T06:05:31.753427+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:12:38.485450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":194,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.755508+0000","last_change":"2024-05-28T06:04:01.039129+0000","last_active":"2024-05-28T06:05:31.755508+0000","last_peered":"2024-05-28T06:05:31.755508+0000","last_clean":"2024-05-28T06:05:31.755508+0000","last_became_active":"2024-05-28T06:04:01.033906+0000","last_became_peered":"2024-05-28T06:04:01.033906+0000","last_unstale":"2024-05-28T06:05:31.755508+0000","last_undegraded":"2024-05-28T06:05:31.755508+0000","last_fullsized":"2024-05-28T06:05:31.755508+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:03:59.864237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:03:59.864237+0000","last_clean_scrub_stamp":"2024-05-28T06:03:59.864237+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:17:36.440245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574870,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27968,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343872,"statfs":{"total":96636764160,"available":96608124928,"internally_reserved":0,"allocated":970752,"data_stored":737074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574872,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27396,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":380928,"data_stored":147232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574870,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27396,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":380928,"data_stored":147288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574870,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":380928,"data_stored":146720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574872,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27408,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344432,"statfs":{"total":96636764160,"available":96608698368,"internally_reserved":0,"allocated":389120,"data_stored":148250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574871,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27396,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":376832,"data_stored":146686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574871,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":389120,"data_stored":148250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574870,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27964,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343876,"statfs":{"total":96636764160,"available":96608129024,"internally_reserved":0,"allocated":966656,"data_stored":736984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-28T06:05:39.889 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-28T06:05:40.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.101+0000 7ff205c00640 1 Processor -- start 2024-05-28T06:05:40.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.101+0000 7ff205c00640 1 -- start start 2024-05-28T06:05:40.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.102+0000 7ff205c00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7ff20006f860 0x7ff2000b3150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:40.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.102+0000 7ff205c00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7ff2000b3690 0x7ff2000d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:40.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.102+0000 7ff205c00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff20006ed10 con 0x7ff2000d09e0 2024-05-28T06:05:40.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.102+0000 7ff205c00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7ff2000d0df0 con 0x7ff20006f860 2024-05-28T06:05:40.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.102+0000 7ff205c00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7ff20015cb00 con 0x7ff2000b3690 2024-05-28T06:05:40.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.103+0000 7ff1fea00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7ff20006f860 0x7ff2000b3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:40.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.103+0000 7ff1fea00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7ff20006f860 0x7ff2000b3150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:42152/0 (socket says 172.21.15.96:42152) 2024-05-28T06:05:40.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.103+0000 7ff1fea00640 1 -- 172.21.15.96:0/229381173 learned_addr learned my addr 172.21.15.96:0/229381173 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:40.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.103+0000 7ff1f7e00640 1 --2- 172.21.15.96:0/229381173 >> v2:172.21.15.123:3300/0 conn(0x7ff2000b3690 0x7ff2000d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:40.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.103+0000 7ff1fea00640 1 -- 172.21.15.96:0/229381173 >> v2:172.21.15.123:3300/0 conn(0x7ff2000b3690 msgr2=0x7ff2000d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:40.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.103+0000 7ff1fea00640 1 --2- 172.21.15.96:0/229381173 >> v2:172.21.15.123:3300/0 conn(0x7ff2000b3690 0x7ff2000d04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:40.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.103+0000 7ff1fea00640 1 -- 172.21.15.96:0/229381173 >> v1:172.21.15.123:6789/0 conn(0x7ff2000d09e0 legacy=0x7ff200160210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:40.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.103+0000 7ff1fea00640 1 -- 172.21.15.96:0/229381173 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2000b3cc0 con 0x7ff20006f860 2024-05-28T06:05:40.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.104+0000 7ff1fea00640 1 --2- 172.21.15.96:0/229381173 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff20006f860 0x7ff2000b3150 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7ff200070120 tx=0x7ff1f002a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6bec653d2a4c6dd server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:40.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.104+0000 7ff1fe000640 1 -- 172.21.15.96:0/229381173 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1f0037070 con 0x7ff20006f860 2024-05-28T06:05:40.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.105+0000 7ff1fe000640 1 -- 172.21.15.96:0/229381173 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1f0032440 con 0x7ff20006f860 2024-05-28T06:05:40.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.105+0000 7ff1fe000640 1 -- 172.21.15.96:0/229381173 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1f0032810 con 0x7ff20006f860 2024-05-28T06:05:40.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.105+0000 7ff205c00640 1 -- 172.21.15.96:0/229381173 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff20006f860 msgr2=0x7ff2000b3150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:40.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.105+0000 7ff205c00640 1 --2- 172.21.15.96:0/229381173 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff20006f860 0x7ff2000b3150 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7ff200070120 tx=0x7ff1f002a230 comp rx=0 tx=0).stop 2024-05-28T06:05:40.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.105+0000 7ff205c00640 1 -- 172.21.15.96:0/229381173 shutdown_connections 2024-05-28T06:05:40.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.105+0000 7ff205c00640 1 --2- 172.21.15.96:0/229381173 >> v2:172.21.15.123:3300/0 conn(0x7ff2000b3690 0x7ff2000d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:40.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.105+0000 7ff205c00640 1 --2- 172.21.15.96:0/229381173 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff20006f860 0x7ff2000b3150 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:40.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.105+0000 7ff205c00640 1 -- 172.21.15.96:0/229381173 >> 172.21.15.96:0/229381173 conn(0x7ff20006bae0 msgr2=0x7ff20006bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:40.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.105+0000 7ff205c00640 1 -- 172.21.15.96:0/229381173 shutdown_connections 2024-05-28T06:05:40.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.105+0000 7ff205c00640 1 -- 172.21.15.96:0/229381173 wait complete. 2024-05-28T06:05:40.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.107+0000 7ff205c00640 1 Processor -- start 2024-05-28T06:05:40.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.107+0000 7ff205c00640 1 -- start start 2024-05-28T06:05:40.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.108+0000 7ff205c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff20006f860 0x7ff2000c3a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:40.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.108+0000 7ff205c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff2000b3690 0x7ff2001f3e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:40.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.108+0000 7ff205c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff2000d09e0 0x7ff20015f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:40.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.108+0000 7ff205c00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff2000b4560 con 0x7ff2000b3690 2024-05-28T06:05:40.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.108+0000 7ff205c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff2000b4280 con 0x7ff2000d09e0 2024-05-28T06:05:40.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.108+0000 7ff205c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ff2000b4840 con 0x7ff20006f860 2024-05-28T06:05:40.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.108+0000 7ff1f7e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff2000b3690 0x7ff2001f3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:40.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.109+0000 7ff1f7e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff2000b3690 0x7ff2001f3e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:42160/0 (socket says 172.21.15.96:42160) 2024-05-28T06:05:40.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.109+0000 7ff1f7e00640 1 -- 172.21.15.96:0/1232841236 learned_addr learned my addr 172.21.15.96:0/1232841236 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:40.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.109+0000 7ff1ff400640 1 --2- 172.21.15.96:0/1232841236 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff2000d09e0 0x7ff20015f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:40.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.109+0000 7ff1f7e00640 1 -- 172.21.15.96:0/1232841236 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff20006f860 msgr2=0x7ff2000c3a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:40.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.109+0000 7ff1f7e00640 1 --2- 172.21.15.96:0/1232841236 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff20006f860 0x7ff2000c3a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:40.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.109+0000 7ff1f7e00640 1 -- 172.21.15.96:0/1232841236 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff2000d09e0 msgr2=0x7ff20015f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:40.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.109+0000 7ff1f7e00640 1 --2- 172.21.15.96:0/1232841236 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff2000d09e0 0x7ff20015f170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:40.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.109+0000 7ff1f7e00640 1 -- 172.21.15.96:0/1232841236 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2000c43c0 con 0x7ff2000b3690 2024-05-28T06:05:40.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.109+0000 7ff1f7e00640 1 --2- 172.21.15.96:0/1232841236 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff2000b3690 0x7ff2001f3e70 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7ff1e800c820 tx=0x7ff1e800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:40.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.110+0000 7ff1f6a00640 1 -- 172.21.15.96:0/1232841236 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1e8015070 con 0x7ff2000b3690 2024-05-28T06:05:40.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.110+0000 7ff1f6a00640 1 -- 172.21.15.96:0/1232841236 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1e8004240 con 0x7ff2000b3690 2024-05-28T06:05:40.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.110+0000 7ff1f6a00640 1 -- 172.21.15.96:0/1232841236 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1e800ce10 con 0x7ff2000b3690 2024-05-28T06:05:40.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.110+0000 7ff205c00640 1 -- 172.21.15.96:0/1232841236 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2000c46a0 con 0x7ff2000b3690 2024-05-28T06:05:40.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.110+0000 7ff205c00640 1 -- 172.21.15.96:0/1232841236 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2000c4c60 con 0x7ff2000b3690 2024-05-28T06:05:40.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.111+0000 7ff205c00640 1 -- 172.21.15.96:0/1232841236 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2000689e0 con 0x7ff2000b3690 2024-05-28T06:05:40.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.112+0000 7ff1f6a00640 1 -- 172.21.15.96:0/1232841236 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff1e8007830 con 0x7ff2000b3690 2024-05-28T06:05:40.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.113+0000 7ff1f6a00640 1 --2- 172.21.15.96:0/1232841236 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff1dc06c8f0 0x7ff1dc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:40.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.113+0000 7ff1f6a00640 1 -- 172.21.15.96:0/1232841236 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7ff1e808bb10 con 0x7ff2000b3690 2024-05-28T06:05:40.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.114+0000 7ff1fea00640 1 --2- 172.21.15.96:0/1232841236 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff1dc06c8f0 0x7ff1dc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:40.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.115+0000 7ff1fea00640 1 --2- 172.21.15.96:0/1232841236 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff1dc06c8f0 0x7ff1dc06edb0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7ff1f0034000 tx=0x7ff1f002c000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:40.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.117+0000 7ff1f6a00640 1 -- 172.21.15.96:0/1232841236 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff1e8059ea0 con 0x7ff2000b3690 2024-05-28T06:05:40.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.309+0000 7ff205c00640 1 -- 172.21.15.96:0/1232841236 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff20006e2f0 con 0x7ff1dc06c8f0 2024-05-28T06:05:40.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.329+0000 7ff1f6a00640 1 -- 172.21.15.96:0/1232841236 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251951 (secure 0 0 0) 0x7ff20006e2f0 con 0x7ff1dc06c8f0 2024-05-28T06:05:40.332 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:05:40.335 INFO:teuthology.orchestra.run.smithi096.stderr:dumped all 2024-05-28T06:05:40.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.336+0000 7ff205c00640 1 -- 172.21.15.96:0/1232841236 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff1dc06c8f0 msgr2=0x7ff1dc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:40.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.336+0000 7ff205c00640 1 --2- 172.21.15.96:0/1232841236 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff1dc06c8f0 0x7ff1dc06edb0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7ff1f0034000 tx=0x7ff1f002c000 comp rx=0 tx=0).stop 2024-05-28T06:05:40.337 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.336+0000 7ff205c00640 1 -- 172.21.15.96:0/1232841236 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff2000b3690 msgr2=0x7ff2001f3e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:40.337 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.336+0000 7ff205c00640 1 --2- 172.21.15.96:0/1232841236 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff2000b3690 0x7ff2001f3e70 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7ff1e800c820 tx=0x7ff1e800cce0 comp rx=0 tx=0).stop 2024-05-28T06:05:40.337 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.336+0000 7ff205c00640 1 -- 172.21.15.96:0/1232841236 shutdown_connections 2024-05-28T06:05:40.337 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.336+0000 7ff205c00640 1 --2- 172.21.15.96:0/1232841236 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff1dc06c8f0 0x7ff1dc06edb0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:40.337 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.336+0000 7ff205c00640 1 --2- 172.21.15.96:0/1232841236 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff2000d09e0 0x7ff20015f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:40.337 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.336+0000 7ff205c00640 1 --2- 172.21.15.96:0/1232841236 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff2000b3690 0x7ff2001f3e70 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:40.337 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.336+0000 7ff205c00640 1 --2- 172.21.15.96:0/1232841236 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff20006f860 0x7ff2000c3a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:40.337 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.336+0000 7ff205c00640 1 -- 172.21.15.96:0/1232841236 >> 172.21.15.96:0/1232841236 conn(0x7ff20006bae0 msgr2=0x7ff2000b0e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:40.337 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.336+0000 7ff205c00640 1 -- 172.21.15.96:0/1232841236 shutdown_connections 2024-05-28T06:05:40.337 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.336+0000 7ff205c00640 1 -- 172.21.15.96:0/1232841236 wait complete. 2024-05-28T06:05:40.355 INFO:teuthology.orchestra.run.smithi096.stdout:{"pg_ready":true,"pg_map":{"version":76,"stamp":"2024-05-28T06:05:39.513179+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220332,"kb_used_data":4136,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754388,"statfs":{"total":773094113280,"available":772868493312,"internally_reserved":0,"allocated":4235264,"data_stored":2358484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.065580"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.754089+0000","last_change":"2024-05-28T06:05:30.477204+0000","last_active":"2024-05-28T06:05:31.754089+0000","last_peered":"2024-05-28T06:05:31.754089+0000","last_clean":"2024-05-28T06:05:31.754089+0000","last_became_active":"2024-05-28T06:05:30.476844+0000","last_became_peered":"2024-05-28T06:05:30.476844+0000","last_unstale":"2024-05-28T06:05:31.754089+0000","last_undegraded":"2024-05-28T06:05:31.754089+0000","last_fullsized":"2024-05-28T06:05:31.754089+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:42:04.972881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.753020+0000","last_change":"2024-05-28T06:05:30.477943+0000","last_active":"2024-05-28T06:05:31.753020+0000","last_peered":"2024-05-28T06:05:31.753020+0000","last_clean":"2024-05-28T06:05:31.753020+0000","last_became_active":"2024-05-28T06:05:30.476995+0000","last_became_peered":"2024-05-28T06:05:30.476995+0000","last_unstale":"2024-05-28T06:05:31.753020+0000","last_undegraded":"2024-05-28T06:05:31.753020+0000","last_fullsized":"2024-05-28T06:05:31.753020+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:56:43.306179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.758130+0000","last_change":"2024-05-28T06:05:30.483815+0000","last_active":"2024-05-28T06:05:31.758130+0000","last_peered":"2024-05-28T06:05:31.758130+0000","last_clean":"2024-05-28T06:05:31.758130+0000","last_became_active":"2024-05-28T06:05:30.483476+0000","last_became_peered":"2024-05-28T06:05:30.483476+0000","last_unstale":"2024-05-28T06:05:31.758130+0000","last_undegraded":"2024-05-28T06:05:31.758130+0000","last_fullsized":"2024-05-28T06:05:31.758130+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:43:33.343569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.487451+0000","last_change":"2024-05-28T06:05:30.481651+0000","last_active":"2024-05-28T06:05:31.487451+0000","last_peered":"2024-05-28T06:05:31.487451+0000","last_clean":"2024-05-28T06:05:31.487451+0000","last_became_active":"2024-05-28T06:05:30.477221+0000","last_became_peered":"2024-05-28T06:05:30.477221+0000","last_unstale":"2024-05-28T06:05:31.487451+0000","last_undegraded":"2024-05-28T06:05:31.487451+0000","last_fullsized":"2024-05-28T06:05:31.487451+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:51:16.699034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.755050+0000","last_change":"2024-05-28T06:05:30.486034+0000","last_active":"2024-05-28T06:05:31.755050+0000","last_peered":"2024-05-28T06:05:31.755050+0000","last_clean":"2024-05-28T06:05:31.755050+0000","last_became_active":"2024-05-28T06:05:30.485742+0000","last_became_peered":"2024-05-28T06:05:30.485742+0000","last_unstale":"2024-05-28T06:05:31.755050+0000","last_undegraded":"2024-05-28T06:05:31.755050+0000","last_fullsized":"2024-05-28T06:05:31.755050+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:16:43.593153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.752594+0000","last_change":"2024-05-28T06:05:30.474720+0000","last_active":"2024-05-28T06:05:31.752594+0000","last_peered":"2024-05-28T06:05:31.752594+0000","last_clean":"2024-05-28T06:05:31.752594+0000","last_became_active":"2024-05-28T06:05:30.474189+0000","last_became_peered":"2024-05-28T06:05:30.474189+0000","last_unstale":"2024-05-28T06:05:31.752594+0000","last_undegraded":"2024-05-28T06:05:31.752594+0000","last_fullsized":"2024-05-28T06:05:31.752594+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:26:58.818587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.756498+0000","last_change":"2024-05-28T06:05:30.487197+0000","last_active":"2024-05-28T06:05:31.756498+0000","last_peered":"2024-05-28T06:05:31.756498+0000","last_clean":"2024-05-28T06:05:31.756498+0000","last_became_active":"2024-05-28T06:05:30.486918+0000","last_became_peered":"2024-05-28T06:05:30.486918+0000","last_unstale":"2024-05-28T06:05:31.756498+0000","last_undegraded":"2024-05-28T06:05:31.756498+0000","last_fullsized":"2024-05-28T06:05:31.756498+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:27:30.648202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.754628+0000","last_change":"2024-05-28T06:05:30.489229+0000","last_active":"2024-05-28T06:05:31.754628+0000","last_peered":"2024-05-28T06:05:31.754628+0000","last_clean":"2024-05-28T06:05:31.754628+0000","last_became_active":"2024-05-28T06:05:30.488923+0000","last_became_peered":"2024-05-28T06:05:30.488923+0000","last_unstale":"2024-05-28T06:05:31.754628+0000","last_undegraded":"2024-05-28T06:05:31.754628+0000","last_fullsized":"2024-05-28T06:05:31.754628+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:58:04.060400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.483710+0000","last_change":"2024-05-28T06:05:30.487873+0000","last_active":"2024-05-28T06:05:31.483710+0000","last_peered":"2024-05-28T06:05:31.483710+0000","last_clean":"2024-05-28T06:05:31.483710+0000","last_became_active":"2024-05-28T06:05:30.487617+0000","last_became_peered":"2024-05-28T06:05:30.487617+0000","last_unstale":"2024-05-28T06:05:31.483710+0000","last_undegraded":"2024-05-28T06:05:31.483710+0000","last_fullsized":"2024-05-28T06:05:31.483710+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:24:54.475794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.487607+0000","last_change":"2024-05-28T06:05:30.473105+0000","last_active":"2024-05-28T06:05:31.487607+0000","last_peered":"2024-05-28T06:05:31.487607+0000","last_clean":"2024-05-28T06:05:31.487607+0000","last_became_active":"2024-05-28T06:05:30.472798+0000","last_became_peered":"2024-05-28T06:05:30.472798+0000","last_unstale":"2024-05-28T06:05:31.487607+0000","last_undegraded":"2024-05-28T06:05:31.487607+0000","last_fullsized":"2024-05-28T06:05:31.487607+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:17:42.817810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.755746+0000","last_change":"2024-05-28T06:05:30.486102+0000","last_active":"2024-05-28T06:05:31.755746+0000","last_peered":"2024-05-28T06:05:31.755746+0000","last_clean":"2024-05-28T06:05:31.755746+0000","last_became_active":"2024-05-28T06:05:30.485870+0000","last_became_peered":"2024-05-28T06:05:30.485870+0000","last_unstale":"2024-05-28T06:05:31.755746+0000","last_undegraded":"2024-05-28T06:05:31.755746+0000","last_fullsized":"2024-05-28T06:05:31.755746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:27:15.781576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.757226+0000","last_change":"2024-05-28T06:05:30.478015+0000","last_active":"2024-05-28T06:05:31.757226+0000","last_peered":"2024-05-28T06:05:31.757226+0000","last_clean":"2024-05-28T06:05:31.757226+0000","last_became_active":"2024-05-28T06:05:30.477760+0000","last_became_peered":"2024-05-28T06:05:30.477760+0000","last_unstale":"2024-05-28T06:05:31.757226+0000","last_undegraded":"2024-05-28T06:05:31.757226+0000","last_fullsized":"2024-05-28T06:05:31.757226+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:26:58.818587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.755386+0000","last_change":"2024-05-28T06:05:30.489911+0000","last_active":"2024-05-28T06:05:31.755386+0000","last_peered":"2024-05-28T06:05:31.755386+0000","last_clean":"2024-05-28T06:05:31.755386+0000","last_became_active":"2024-05-28T06:05:30.489638+0000","last_became_peered":"2024-05-28T06:05:30.489638+0000","last_unstale":"2024-05-28T06:05:31.755386+0000","last_undegraded":"2024-05-28T06:05:31.755386+0000","last_fullsized":"2024-05-28T06:05:31.755386+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:30:21.413453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.752710+0000","last_change":"2024-05-28T06:05:30.488321+0000","last_active":"2024-05-28T06:05:31.752710+0000","last_peered":"2024-05-28T06:05:31.752710+0000","last_clean":"2024-05-28T06:05:31.752710+0000","last_became_active":"2024-05-28T06:05:30.487979+0000","last_became_peered":"2024-05-28T06:05:30.487979+0000","last_unstale":"2024-05-28T06:05:31.752710+0000","last_undegraded":"2024-05-28T06:05:31.752710+0000","last_fullsized":"2024-05-28T06:05:31.752710+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:27:15.781576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.755011+0000","last_change":"2024-05-28T06:05:30.469583+0000","last_active":"2024-05-28T06:05:31.755011+0000","last_peered":"2024-05-28T06:05:31.755011+0000","last_clean":"2024-05-28T06:05:31.755011+0000","last_became_active":"2024-05-28T06:05:30.469299+0000","last_became_peered":"2024-05-28T06:05:30.469299+0000","last_unstale":"2024-05-28T06:05:31.755011+0000","last_undegraded":"2024-05-28T06:05:31.755011+0000","last_fullsized":"2024-05-28T06:05:31.755011+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:51:12.201028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.755507+0000","last_change":"2024-05-28T06:05:30.474489+0000","last_active":"2024-05-28T06:05:31.755507+0000","last_peered":"2024-05-28T06:05:31.755507+0000","last_clean":"2024-05-28T06:05:31.755507+0000","last_became_active":"2024-05-28T06:05:30.474215+0000","last_became_peered":"2024-05-28T06:05:30.474215+0000","last_unstale":"2024-05-28T06:05:31.755507+0000","last_undegraded":"2024-05-28T06:05:31.755507+0000","last_fullsized":"2024-05-28T06:05:31.755507+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:26:58.818587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.756720+0000","last_change":"2024-05-28T06:05:30.486573+0000","last_active":"2024-05-28T06:05:31.756720+0000","last_peered":"2024-05-28T06:05:31.756720+0000","last_clean":"2024-05-28T06:05:31.756720+0000","last_became_active":"2024-05-28T06:05:30.486307+0000","last_became_peered":"2024-05-28T06:05:30.486307+0000","last_unstale":"2024-05-28T06:05:31.756720+0000","last_undegraded":"2024-05-28T06:05:31.756720+0000","last_fullsized":"2024-05-28T06:05:31.756720+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:43:33.343569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.753543+0000","last_change":"2024-05-28T06:05:30.477781+0000","last_active":"2024-05-28T06:05:31.753543+0000","last_peered":"2024-05-28T06:05:31.753543+0000","last_clean":"2024-05-28T06:05:31.753543+0000","last_became_active":"2024-05-28T06:05:30.476599+0000","last_became_peered":"2024-05-28T06:05:30.476599+0000","last_unstale":"2024-05-28T06:05:31.753543+0000","last_undegraded":"2024-05-28T06:05:31.753543+0000","last_fullsized":"2024-05-28T06:05:31.753543+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:43:33.343569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.753222+0000","last_change":"2024-05-28T06:05:30.476553+0000","last_active":"2024-05-28T06:05:31.753222+0000","last_peered":"2024-05-28T06:05:31.753222+0000","last_clean":"2024-05-28T06:05:31.753222+0000","last_became_active":"2024-05-28T06:05:30.476290+0000","last_became_peered":"2024-05-28T06:05:30.476290+0000","last_unstale":"2024-05-28T06:05:31.753222+0000","last_undegraded":"2024-05-28T06:05:31.753222+0000","last_fullsized":"2024-05-28T06:05:31.753222+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:27:30.648202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.755494+0000","last_change":"2024-05-28T06:05:30.483659+0000","last_active":"2024-05-28T06:05:31.755494+0000","last_peered":"2024-05-28T06:05:31.755494+0000","last_clean":"2024-05-28T06:05:31.755494+0000","last_became_active":"2024-05-28T06:05:30.483403+0000","last_became_peered":"2024-05-28T06:05:30.483403+0000","last_unstale":"2024-05-28T06:05:31.755494+0000","last_undegraded":"2024-05-28T06:05:31.755494+0000","last_fullsized":"2024-05-28T06:05:31.755494+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:47:22.330606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.482709+0000","last_change":"2024-05-28T06:05:30.472992+0000","last_active":"2024-05-28T06:05:31.482709+0000","last_peered":"2024-05-28T06:05:31.482709+0000","last_clean":"2024-05-28T06:05:31.482709+0000","last_became_active":"2024-05-28T06:05:30.472725+0000","last_became_peered":"2024-05-28T06:05:30.472725+0000","last_unstale":"2024-05-28T06:05:31.482709+0000","last_undegraded":"2024-05-28T06:05:31.482709+0000","last_fullsized":"2024-05-28T06:05:31.482709+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:42:04.972881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.482893+0000","last_change":"2024-05-28T06:05:30.468698+0000","last_active":"2024-05-28T06:05:31.482893+0000","last_peered":"2024-05-28T06:05:31.482893+0000","last_clean":"2024-05-28T06:05:31.482893+0000","last_became_active":"2024-05-28T06:05:30.468120+0000","last_became_peered":"2024-05-28T06:05:30.468120+0000","last_unstale":"2024-05-28T06:05:31.482893+0000","last_undegraded":"2024-05-28T06:05:31.482893+0000","last_fullsized":"2024-05-28T06:05:31.482893+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:41:24.956597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.477622+0000","last_change":"2024-05-28T06:05:30.470896+0000","last_active":"2024-05-28T06:05:31.477622+0000","last_peered":"2024-05-28T06:05:31.477622+0000","last_clean":"2024-05-28T06:05:31.477622+0000","last_became_active":"2024-05-28T06:05:30.470316+0000","last_became_peered":"2024-05-28T06:05:30.470316+0000","last_unstale":"2024-05-28T06:05:31.477622+0000","last_undegraded":"2024-05-28T06:05:31.477622+0000","last_fullsized":"2024-05-28T06:05:31.477622+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:37:19.547704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.754580+0000","last_change":"2024-05-28T06:05:30.483993+0000","last_active":"2024-05-28T06:05:31.754580+0000","last_peered":"2024-05-28T06:05:31.754580+0000","last_clean":"2024-05-28T06:05:31.754580+0000","last_became_active":"2024-05-28T06:05:30.483709+0000","last_became_peered":"2024-05-28T06:05:30.483709+0000","last_unstale":"2024-05-28T06:05:31.754580+0000","last_undegraded":"2024-05-28T06:05:31.754580+0000","last_fullsized":"2024-05-28T06:05:31.754580+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:58:04.060400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.488135+0000","last_change":"2024-05-28T06:05:30.472224+0000","last_active":"2024-05-28T06:05:31.488135+0000","last_peered":"2024-05-28T06:05:31.488135+0000","last_clean":"2024-05-28T06:05:31.488135+0000","last_became_active":"2024-05-28T06:05:30.471939+0000","last_became_peered":"2024-05-28T06:05:30.471939+0000","last_unstale":"2024-05-28T06:05:31.488135+0000","last_undegraded":"2024-05-28T06:05:31.488135+0000","last_fullsized":"2024-05-28T06:05:31.488135+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:58:04.060400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.753861+0000","last_change":"2024-05-28T06:05:30.472699+0000","last_active":"2024-05-28T06:05:31.753861+0000","last_peered":"2024-05-28T06:05:31.753861+0000","last_clean":"2024-05-28T06:05:31.753861+0000","last_became_active":"2024-05-28T06:05:30.472399+0000","last_became_peered":"2024-05-28T06:05:30.472399+0000","last_unstale":"2024-05-28T06:05:31.753861+0000","last_undegraded":"2024-05-28T06:05:31.753861+0000","last_fullsized":"2024-05-28T06:05:31.753861+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:28:45.385929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.488123+0000","last_change":"2024-05-28T06:05:30.477001+0000","last_active":"2024-05-28T06:05:31.488123+0000","last_peered":"2024-05-28T06:05:31.488123+0000","last_clean":"2024-05-28T06:05:31.488123+0000","last_became_active":"2024-05-28T06:05:30.476725+0000","last_became_peered":"2024-05-28T06:05:30.476725+0000","last_unstale":"2024-05-28T06:05:31.488123+0000","last_undegraded":"2024-05-28T06:05:31.488123+0000","last_fullsized":"2024-05-28T06:05:31.488123+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:11:20.837156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.757661+0000","last_change":"2024-05-28T06:05:30.484736+0000","last_active":"2024-05-28T06:05:31.757661+0000","last_peered":"2024-05-28T06:05:31.757661+0000","last_clean":"2024-05-28T06:05:31.757661+0000","last_became_active":"2024-05-28T06:05:30.482722+0000","last_became_peered":"2024-05-28T06:05:30.482722+0000","last_unstale":"2024-05-28T06:05:31.757661+0000","last_undegraded":"2024-05-28T06:05:31.757661+0000","last_fullsized":"2024-05-28T06:05:31.757661+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:56:43.306179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.482975+0000","last_change":"2024-05-28T06:05:30.477442+0000","last_active":"2024-05-28T06:05:31.482975+0000","last_peered":"2024-05-28T06:05:31.482975+0000","last_clean":"2024-05-28T06:05:31.482975+0000","last_became_active":"2024-05-28T06:05:30.477224+0000","last_became_peered":"2024-05-28T06:05:30.477224+0000","last_unstale":"2024-05-28T06:05:31.482975+0000","last_undegraded":"2024-05-28T06:05:31.482975+0000","last_fullsized":"2024-05-28T06:05:31.482975+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:46:23.694062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.756903+0000","last_change":"2024-05-28T06:05:30.484668+0000","last_active":"2024-05-28T06:05:31.756903+0000","last_peered":"2024-05-28T06:05:31.756903+0000","last_clean":"2024-05-28T06:05:31.756903+0000","last_became_active":"2024-05-28T06:05:30.484339+0000","last_became_peered":"2024-05-28T06:05:30.484339+0000","last_unstale":"2024-05-28T06:05:31.756903+0000","last_undegraded":"2024-05-28T06:05:31.756903+0000","last_fullsized":"2024-05-28T06:05:31.756903+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:09:23.562166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.754848+0000","last_change":"2024-05-28T06:05:30.479119+0000","last_active":"2024-05-28T06:05:31.754848+0000","last_peered":"2024-05-28T06:05:31.754848+0000","last_clean":"2024-05-28T06:05:31.754848+0000","last_became_active":"2024-05-28T06:05:30.478742+0000","last_became_peered":"2024-05-28T06:05:30.478742+0000","last_unstale":"2024-05-28T06:05:31.754848+0000","last_undegraded":"2024-05-28T06:05:31.754848+0000","last_fullsized":"2024-05-28T06:05:31.754848+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:41:24.956597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.754808+0000","last_change":"2024-05-28T06:05:30.474293+0000","last_active":"2024-05-28T06:05:31.754808+0000","last_peered":"2024-05-28T06:05:31.754808+0000","last_clean":"2024-05-28T06:05:31.754808+0000","last_became_active":"2024-05-28T06:05:30.473563+0000","last_became_peered":"2024-05-28T06:05:30.473563+0000","last_unstale":"2024-05-28T06:05:31.754808+0000","last_undegraded":"2024-05-28T06:05:31.754808+0000","last_fullsized":"2024-05-28T06:05:31.754808+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:51:12.201028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.478000+0000","last_change":"2024-05-28T06:05:30.473138+0000","last_active":"2024-05-28T06:05:31.478000+0000","last_peered":"2024-05-28T06:05:31.478000+0000","last_clean":"2024-05-28T06:05:31.478000+0000","last_became_active":"2024-05-28T06:05:30.472302+0000","last_became_peered":"2024-05-28T06:05:30.472302+0000","last_unstale":"2024-05-28T06:05:31.478000+0000","last_undegraded":"2024-05-28T06:05:31.478000+0000","last_fullsized":"2024-05-28T06:05:31.478000+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:46:23.694062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"32'4","reported_seq":18,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.754387+0000","last_change":"2024-05-28T06:05:29.474199+0000","last_active":"2024-05-28T06:05:31.754387+0000","last_peered":"2024-05-28T06:05:31.754387+0000","last_clean":"2024-05-28T06:05:31.754387+0000","last_became_active":"2024-05-28T06:05:29.473863+0000","last_became_peered":"2024-05-28T06:05:29.473863+0000","last_unstale":"2024-05-28T06:05:31.754387+0000","last_undegraded":"2024-05-28T06:05:31.754387+0000","last_fullsized":"2024-05-28T06:05:31.754387+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:47:21.316456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.754375+0000","last_change":"2024-05-28T06:05:30.473327+0000","last_active":"2024-05-28T06:05:31.754375+0000","last_peered":"2024-05-28T06:05:31.754375+0000","last_clean":"2024-05-28T06:05:31.754375+0000","last_became_active":"2024-05-28T06:05:30.470709+0000","last_became_peered":"2024-05-28T06:05:30.470709+0000","last_unstale":"2024-05-28T06:05:31.754375+0000","last_undegraded":"2024-05-28T06:05:31.754375+0000","last_fullsized":"2024-05-28T06:05:31.754375+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:37:19.547704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.478062+0000","last_change":"2024-05-28T06:05:29.460672+0000","last_active":"2024-05-28T06:05:31.478062+0000","last_peered":"2024-05-28T06:05:31.478062+0000","last_clean":"2024-05-28T06:05:31.478062+0000","last_became_active":"2024-05-28T06:05:29.459991+0000","last_became_peered":"2024-05-28T06:05:29.459991+0000","last_unstale":"2024-05-28T06:05:31.478062+0000","last_undegraded":"2024-05-28T06:05:31.478062+0000","last_fullsized":"2024-05-28T06:05:31.478062+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:57:51.752948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.483444+0000","last_change":"2024-05-28T06:05:30.477437+0000","last_active":"2024-05-28T06:05:31.483444+0000","last_peered":"2024-05-28T06:05:31.483444+0000","last_clean":"2024-05-28T06:05:31.483444+0000","last_became_active":"2024-05-28T06:05:30.477193+0000","last_became_peered":"2024-05-28T06:05:30.477193+0000","last_unstale":"2024-05-28T06:05:31.483444+0000","last_undegraded":"2024-05-28T06:05:31.483444+0000","last_fullsized":"2024-05-28T06:05:31.483444+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:14:29.844730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.753411+0000","last_change":"2024-05-28T06:05:29.456735+0000","last_active":"2024-05-28T06:05:31.753411+0000","last_peered":"2024-05-28T06:05:31.753411+0000","last_clean":"2024-05-28T06:05:31.753411+0000","last_became_active":"2024-05-28T06:05:29.456466+0000","last_became_peered":"2024-05-28T06:05:29.456466+0000","last_unstale":"2024-05-28T06:05:31.753411+0000","last_undegraded":"2024-05-28T06:05:31.753411+0000","last_fullsized":"2024-05-28T06:05:31.753411+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:41:23.942447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.488191+0000","last_change":"2024-05-28T06:05:30.476962+0000","last_active":"2024-05-28T06:05:31.488191+0000","last_peered":"2024-05-28T06:05:31.488191+0000","last_clean":"2024-05-28T06:05:31.488191+0000","last_became_active":"2024-05-28T06:05:30.476657+0000","last_became_peered":"2024-05-28T06:05:30.476657+0000","last_unstale":"2024-05-28T06:05:31.488191+0000","last_undegraded":"2024-05-28T06:05:31.488191+0000","last_fullsized":"2024-05-28T06:05:31.488191+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:51:12.201028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.755643+0000","last_change":"2024-05-28T06:05:29.451693+0000","last_active":"2024-05-28T06:05:31.755643+0000","last_peered":"2024-05-28T06:05:31.755643+0000","last_clean":"2024-05-28T06:05:31.755643+0000","last_became_active":"2024-05-28T06:05:29.451343+0000","last_became_peered":"2024-05-28T06:05:29.451343+0000","last_unstale":"2024-05-28T06:05:31.755643+0000","last_undegraded":"2024-05-28T06:05:31.755643+0000","last_fullsized":"2024-05-28T06:05:31.755643+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:42:15.256795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.756525+0000","last_change":"2024-05-28T06:05:30.477467+0000","last_active":"2024-05-28T06:05:31.756525+0000","last_peered":"2024-05-28T06:05:31.756525+0000","last_clean":"2024-05-28T06:05:31.756525+0000","last_became_active":"2024-05-28T06:05:30.477224+0000","last_became_peered":"2024-05-28T06:05:30.477224+0000","last_unstale":"2024-05-28T06:05:31.756525+0000","last_undegraded":"2024-05-28T06:05:31.756525+0000","last_fullsized":"2024-05-28T06:05:31.756525+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:21:35.302971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.482842+0000","last_change":"2024-05-28T06:05:29.463758+0000","last_active":"2024-05-28T06:05:31.482842+0000","last_peered":"2024-05-28T06:05:31.482842+0000","last_clean":"2024-05-28T06:05:31.482842+0000","last_became_active":"2024-05-28T06:05:29.460297+0000","last_became_peered":"2024-05-28T06:05:29.460297+0000","last_unstale":"2024-05-28T06:05:31.482842+0000","last_undegraded":"2024-05-28T06:05:31.482842+0000","last_fullsized":"2024-05-28T06:05:31.482842+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:05:43.269097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.756966+0000","last_change":"2024-05-28T06:05:30.484901+0000","last_active":"2024-05-28T06:05:31.756966+0000","last_peered":"2024-05-28T06:05:31.756966+0000","last_clean":"2024-05-28T06:05:31.756966+0000","last_became_active":"2024-05-28T06:05:30.484581+0000","last_became_peered":"2024-05-28T06:05:30.484581+0000","last_unstale":"2024-05-28T06:05:31.756966+0000","last_undegraded":"2024-05-28T06:05:31.756966+0000","last_fullsized":"2024-05-28T06:05:31.756966+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:39:33.123768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.482842+0000","last_change":"2024-05-28T06:05:29.462024+0000","last_active":"2024-05-28T06:05:31.482842+0000","last_peered":"2024-05-28T06:05:31.482842+0000","last_clean":"2024-05-28T06:05:31.482842+0000","last_became_active":"2024-05-28T06:05:29.460033+0000","last_became_peered":"2024-05-28T06:05:29.460033+0000","last_unstale":"2024-05-28T06:05:31.482842+0000","last_undegraded":"2024-05-28T06:05:31.482842+0000","last_fullsized":"2024-05-28T06:05:31.482842+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:38:57.785689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.482390+0000","last_change":"2024-05-28T06:05:30.468776+0000","last_active":"2024-05-28T06:05:31.482390+0000","last_peered":"2024-05-28T06:05:31.482390+0000","last_clean":"2024-05-28T06:05:31.482390+0000","last_became_active":"2024-05-28T06:05:30.468551+0000","last_became_peered":"2024-05-28T06:05:30.468551+0000","last_unstale":"2024-05-28T06:05:31.482390+0000","last_undegraded":"2024-05-28T06:05:31.482390+0000","last_fullsized":"2024-05-28T06:05:31.482390+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:33:36.484602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.752569+0000","last_change":"2024-05-28T06:05:29.456433+0000","last_active":"2024-05-28T06:05:31.752569+0000","last_peered":"2024-05-28T06:05:31.752569+0000","last_clean":"2024-05-28T06:05:31.752569+0000","last_became_active":"2024-05-28T06:05:29.456113+0000","last_became_peered":"2024-05-28T06:05:29.456113+0000","last_unstale":"2024-05-28T06:05:31.752569+0000","last_undegraded":"2024-05-28T06:05:31.752569+0000","last_fullsized":"2024-05-28T06:05:31.752569+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:12:52.117940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.486965+0000","last_change":"2024-05-28T06:05:30.478697+0000","last_active":"2024-05-28T06:05:31.486965+0000","last_peered":"2024-05-28T06:05:31.486965+0000","last_clean":"2024-05-28T06:05:31.486965+0000","last_became_active":"2024-05-28T06:05:30.477683+0000","last_became_peered":"2024-05-28T06:05:30.477683+0000","last_unstale":"2024-05-28T06:05:31.486965+0000","last_undegraded":"2024-05-28T06:05:31.486965+0000","last_fullsized":"2024-05-28T06:05:31.486965+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:24:54.475794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.482413+0000","last_change":"2024-05-28T06:05:29.462393+0000","last_active":"2024-05-28T06:05:31.482413+0000","last_peered":"2024-05-28T06:05:31.482413+0000","last_clean":"2024-05-28T06:05:31.482413+0000","last_became_active":"2024-05-28T06:05:29.461785+0000","last_became_peered":"2024-05-28T06:05:29.461785+0000","last_unstale":"2024-05-28T06:05:31.482413+0000","last_undegraded":"2024-05-28T06:05:31.482413+0000","last_fullsized":"2024-05-28T06:05:31.482413+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:58:46.608407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.757287+0000","last_change":"2024-05-28T06:05:30.481878+0000","last_active":"2024-05-28T06:05:31.757287+0000","last_peered":"2024-05-28T06:05:31.757287+0000","last_clean":"2024-05-28T06:05:31.757287+0000","last_became_active":"2024-05-28T06:05:30.481566+0000","last_became_peered":"2024-05-28T06:05:30.481566+0000","last_unstale":"2024-05-28T06:05:31.757287+0000","last_undegraded":"2024-05-28T06:05:31.757287+0000","last_fullsized":"2024-05-28T06:05:31.757287+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:27:15.781576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.753786+0000","last_change":"2024-05-28T06:05:29.461808+0000","last_active":"2024-05-28T06:05:31.753786+0000","last_peered":"2024-05-28T06:05:31.753786+0000","last_clean":"2024-05-28T06:05:31.753786+0000","last_became_active":"2024-05-28T06:05:29.461481+0000","last_became_peered":"2024-05-28T06:05:29.461481+0000","last_unstale":"2024-05-28T06:05:31.753786+0000","last_undegraded":"2024-05-28T06:05:31.753786+0000","last_fullsized":"2024-05-28T06:05:31.753786+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:46:22.679912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.753788+0000","last_change":"2024-05-28T06:05:30.475975+0000","last_active":"2024-05-28T06:05:31.753788+0000","last_peered":"2024-05-28T06:05:31.753788+0000","last_clean":"2024-05-28T06:05:31.753788+0000","last_became_active":"2024-05-28T06:05:30.475693+0000","last_became_peered":"2024-05-28T06:05:30.475693+0000","last_unstale":"2024-05-28T06:05:31.753788+0000","last_undegraded":"2024-05-28T06:05:31.753788+0000","last_fullsized":"2024-05-28T06:05:31.753788+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:09:23.562166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.478140+0000","last_change":"2024-05-28T06:05:29.460940+0000","last_active":"2024-05-28T06:05:31.478140+0000","last_peered":"2024-05-28T06:05:31.478140+0000","last_clean":"2024-05-28T06:05:31.478140+0000","last_became_active":"2024-05-28T06:05:29.460433+0000","last_became_peered":"2024-05-28T06:05:29.460433+0000","last_unstale":"2024-05-28T06:05:31.478140+0000","last_undegraded":"2024-05-28T06:05:31.478140+0000","last_fullsized":"2024-05-28T06:05:31.478140+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:32:34.183989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.753438+0000","last_change":"2024-05-28T06:05:30.472323+0000","last_active":"2024-05-28T06:05:31.753438+0000","last_peered":"2024-05-28T06:05:31.753438+0000","last_clean":"2024-05-28T06:05:31.753438+0000","last_became_active":"2024-05-28T06:05:30.472022+0000","last_became_peered":"2024-05-28T06:05:30.472022+0000","last_unstale":"2024-05-28T06:05:31.753438+0000","last_undegraded":"2024-05-28T06:05:31.753438+0000","last_fullsized":"2024-05-28T06:05:31.753438+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:21:35.302971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"32'3","reported_seq":17,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.757966+0000","last_change":"2024-05-28T06:05:29.462646+0000","last_active":"2024-05-28T06:05:31.757966+0000","last_peered":"2024-05-28T06:05:31.757966+0000","last_clean":"2024-05-28T06:05:31.757966+0000","last_became_active":"2024-05-28T06:05:29.462355+0000","last_became_peered":"2024-05-28T06:05:29.462355+0000","last_unstale":"2024-05-28T06:05:31.757966+0000","last_undegraded":"2024-05-28T06:05:31.757966+0000","last_fullsized":"2024-05-28T06:05:31.757966+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:37:18.533553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.757672+0000","last_change":"2024-05-28T06:05:30.483710+0000","last_active":"2024-05-28T06:05:31.757672+0000","last_peered":"2024-05-28T06:05:31.757672+0000","last_clean":"2024-05-28T06:05:31.757672+0000","last_became_active":"2024-05-28T06:05:30.483250+0000","last_became_peered":"2024-05-28T06:05:30.483250+0000","last_unstale":"2024-05-28T06:05:31.757672+0000","last_undegraded":"2024-05-28T06:05:31.757672+0000","last_fullsized":"2024-05-28T06:05:31.757672+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:38:12.995079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"32'6","reported_seq":20,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.753560+0000","last_change":"2024-05-28T06:05:29.458155+0000","last_active":"2024-05-28T06:05:31.753560+0000","last_peered":"2024-05-28T06:05:31.753560+0000","last_clean":"2024-05-28T06:05:31.753560+0000","last_became_active":"2024-05-28T06:05:29.457822+0000","last_became_peered":"2024-05-28T06:05:29.457822+0000","last_unstale":"2024-05-28T06:05:31.753560+0000","last_undegraded":"2024-05-28T06:05:31.753560+0000","last_fullsized":"2024-05-28T06:05:31.753560+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:37:18.533553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.487453+0000","last_change":"2024-05-28T06:05:30.479836+0000","last_active":"2024-05-28T06:05:31.487453+0000","last_peered":"2024-05-28T06:05:31.487453+0000","last_clean":"2024-05-28T06:05:31.487453+0000","last_became_active":"2024-05-28T06:05:30.479377+0000","last_became_peered":"2024-05-28T06:05:30.479377+0000","last_unstale":"2024-05-28T06:05:31.487453+0000","last_undegraded":"2024-05-28T06:05:31.487453+0000","last_fullsized":"2024-05-28T06:05:31.487453+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:39:39.679412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.483004+0000","last_change":"2024-05-28T06:05:29.464610+0000","last_active":"2024-05-28T06:05:31.483004+0000","last_peered":"2024-05-28T06:05:31.483004+0000","last_clean":"2024-05-28T06:05:31.483004+0000","last_became_active":"2024-05-28T06:05:29.464074+0000","last_became_peered":"2024-05-28T06:05:29.464074+0000","last_unstale":"2024-05-28T06:05:31.483004+0000","last_undegraded":"2024-05-28T06:05:31.483004+0000","last_fullsized":"2024-05-28T06:05:31.483004+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:30:20.399303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.752600+0000","last_change":"2024-05-28T06:05:30.474344+0000","last_active":"2024-05-28T06:05:31.752600+0000","last_peered":"2024-05-28T06:05:31.752600+0000","last_clean":"2024-05-28T06:05:31.752600+0000","last_became_active":"2024-05-28T06:05:30.474074+0000","last_became_peered":"2024-05-28T06:05:30.474074+0000","last_unstale":"2024-05-28T06:05:31.752600+0000","last_undegraded":"2024-05-28T06:05:31.752600+0000","last_fullsized":"2024-05-28T06:05:31.752600+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:39:33.123768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.486959+0000","last_change":"2024-05-28T06:05:29.463194+0000","last_active":"2024-05-28T06:05:31.486959+0000","last_peered":"2024-05-28T06:05:31.486959+0000","last_clean":"2024-05-28T06:05:31.486959+0000","last_became_active":"2024-05-28T06:05:29.462870+0000","last_became_peered":"2024-05-28T06:05:29.462870+0000","last_unstale":"2024-05-28T06:05:31.486959+0000","last_undegraded":"2024-05-28T06:05:31.486959+0000","last_fullsized":"2024-05-28T06:05:31.486959+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:32:34.183989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.755042+0000","last_change":"2024-05-28T06:05:30.478298+0000","last_active":"2024-05-28T06:05:31.755042+0000","last_peered":"2024-05-28T06:05:31.755042+0000","last_clean":"2024-05-28T06:05:31.755042+0000","last_became_active":"2024-05-28T06:05:30.478018+0000","last_became_peered":"2024-05-28T06:05:30.478018+0000","last_unstale":"2024-05-28T06:05:31.755042+0000","last_undegraded":"2024-05-28T06:05:31.755042+0000","last_fullsized":"2024-05-28T06:05:31.755042+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:39:33.123768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.752817+0000","last_change":"2024-05-28T06:05:29.462298+0000","last_active":"2024-05-28T06:05:31.752817+0000","last_peered":"2024-05-28T06:05:31.752817+0000","last_clean":"2024-05-28T06:05:31.752817+0000","last_became_active":"2024-05-28T06:05:29.462022+0000","last_became_peered":"2024-05-28T06:05:29.462022+0000","last_unstale":"2024-05-28T06:05:31.752817+0000","last_undegraded":"2024-05-28T06:05:31.752817+0000","last_fullsized":"2024-05-28T06:05:31.752817+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:33:35.470452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.754851+0000","last_change":"2024-05-28T06:05:30.478395+0000","last_active":"2024-05-28T06:05:31.754851+0000","last_peered":"2024-05-28T06:05:31.754851+0000","last_clean":"2024-05-28T06:05:31.754851+0000","last_became_active":"2024-05-28T06:05:30.470732+0000","last_became_peered":"2024-05-28T06:05:30.470732+0000","last_unstale":"2024-05-28T06:05:31.754851+0000","last_undegraded":"2024-05-28T06:05:31.754851+0000","last_fullsized":"2024-05-28T06:05:31.754851+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:46:23.694062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"32'8","reported_seq":22,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.752843+0000","last_change":"2024-05-28T06:05:29.460103+0000","last_active":"2024-05-28T06:05:31.752843+0000","last_peered":"2024-05-28T06:05:31.752843+0000","last_clean":"2024-05-28T06:05:31.752843+0000","last_became_active":"2024-05-28T06:05:29.459788+0000","last_became_peered":"2024-05-28T06:05:29.459788+0000","last_unstale":"2024-05-28T06:05:31.752843+0000","last_undegraded":"2024-05-28T06:05:31.752843+0000","last_fullsized":"2024-05-28T06:05:31.752843+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:03:08.734203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.483476+0000","last_change":"2024-05-28T06:05:30.471442+0000","last_active":"2024-05-28T06:05:31.483476+0000","last_peered":"2024-05-28T06:05:31.483476+0000","last_clean":"2024-05-28T06:05:31.483476+0000","last_became_active":"2024-05-28T06:05:30.471189+0000","last_became_peered":"2024-05-28T06:05:30.471189+0000","last_unstale":"2024-05-28T06:05:31.483476+0000","last_undegraded":"2024-05-28T06:05:31.483476+0000","last_fullsized":"2024-05-28T06:05:31.483476+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:43:11.326702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.488303+0000","last_change":"2024-05-28T06:05:29.458728+0000","last_active":"2024-05-28T06:05:31.488303+0000","last_peered":"2024-05-28T06:05:31.488303+0000","last_clean":"2024-05-28T06:05:31.488303+0000","last_became_active":"2024-05-28T06:05:29.458154+0000","last_became_peered":"2024-05-28T06:05:29.458154+0000","last_unstale":"2024-05-28T06:05:31.488303+0000","last_undegraded":"2024-05-28T06:05:31.488303+0000","last_fullsized":"2024-05-28T06:05:31.488303+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:12:38.485450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.757818+0000","last_change":"2024-05-28T06:05:30.471043+0000","last_active":"2024-05-28T06:05:31.757818+0000","last_peered":"2024-05-28T06:05:31.757818+0000","last_clean":"2024-05-28T06:05:31.757818+0000","last_became_active":"2024-05-28T06:05:30.470782+0000","last_became_peered":"2024-05-28T06:05:30.470782+0000","last_unstale":"2024-05-28T06:05:31.757818+0000","last_undegraded":"2024-05-28T06:05:31.757818+0000","last_fullsized":"2024-05-28T06:05:31.757818+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:27:30.648202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.483361+0000","last_change":"2024-05-28T06:05:29.463122+0000","last_active":"2024-05-28T06:05:31.483361+0000","last_peered":"2024-05-28T06:05:31.483361+0000","last_clean":"2024-05-28T06:05:31.483361+0000","last_became_active":"2024-05-28T06:05:29.462362+0000","last_became_peered":"2024-05-28T06:05:29.462362+0000","last_unstale":"2024-05-28T06:05:31.483361+0000","last_undegraded":"2024-05-28T06:05:31.483361+0000","last_fullsized":"2024-05-28T06:05:31.483361+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:34:23.160698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.487454+0000","last_change":"2024-05-28T06:05:30.482239+0000","last_active":"2024-05-28T06:05:31.487454+0000","last_peered":"2024-05-28T06:05:31.487454+0000","last_clean":"2024-05-28T06:05:31.487454+0000","last_became_active":"2024-05-28T06:05:30.481947+0000","last_became_peered":"2024-05-28T06:05:30.481947+0000","last_unstale":"2024-05-28T06:05:31.487454+0000","last_undegraded":"2024-05-28T06:05:31.487454+0000","last_fullsized":"2024-05-28T06:05:31.487454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:04:45.854464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.483370+0000","last_change":"2024-05-28T06:05:29.450660+0000","last_active":"2024-05-28T06:05:31.483370+0000","last_peered":"2024-05-28T06:05:31.483370+0000","last_clean":"2024-05-28T06:05:31.483370+0000","last_became_active":"2024-05-28T06:05:29.446547+0000","last_became_peered":"2024-05-28T06:05:29.446547+0000","last_unstale":"2024-05-28T06:05:31.483370+0000","last_undegraded":"2024-05-28T06:05:31.483370+0000","last_fullsized":"2024-05-28T06:05:31.483370+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:33:40.689124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.755147+0000","last_change":"2024-05-28T06:05:30.478730+0000","last_active":"2024-05-28T06:05:31.755147+0000","last_peered":"2024-05-28T06:05:31.755147+0000","last_clean":"2024-05-28T06:05:31.755147+0000","last_became_active":"2024-05-28T06:05:30.478438+0000","last_became_peered":"2024-05-28T06:05:30.478438+0000","last_unstale":"2024-05-28T06:05:31.755147+0000","last_undegraded":"2024-05-28T06:05:31.755147+0000","last_fullsized":"2024-05-28T06:05:31.755147+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:05:44.283248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.483501+0000","last_change":"2024-05-28T06:05:29.461213+0000","last_active":"2024-05-28T06:05:31.483501+0000","last_peered":"2024-05-28T06:05:31.483501+0000","last_clean":"2024-05-28T06:05:31.483501+0000","last_became_active":"2024-05-28T06:05:29.460899+0000","last_became_peered":"2024-05-28T06:05:29.460899+0000","last_unstale":"2024-05-28T06:05:31.483501+0000","last_undegraded":"2024-05-28T06:05:31.483501+0000","last_fullsized":"2024-05-28T06:05:31.483501+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:16:42.579003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.754877+0000","last_change":"2024-05-28T06:05:30.484117+0000","last_active":"2024-05-28T06:05:31.754877+0000","last_peered":"2024-05-28T06:05:31.754877+0000","last_clean":"2024-05-28T06:05:31.754877+0000","last_became_active":"2024-05-28T06:05:30.483835+0000","last_became_peered":"2024-05-28T06:05:30.483835+0000","last_unstale":"2024-05-28T06:05:31.754877+0000","last_undegraded":"2024-05-28T06:05:31.754877+0000","last_fullsized":"2024-05-28T06:05:31.754877+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:41:32.997209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.478083+0000","last_change":"2024-05-28T06:05:29.447607+0000","last_active":"2024-05-28T06:05:31.478083+0000","last_peered":"2024-05-28T06:05:31.478083+0000","last_clean":"2024-05-28T06:05:31.478083+0000","last_became_active":"2024-05-28T06:05:29.447321+0000","last_became_peered":"2024-05-28T06:05:29.447321+0000","last_unstale":"2024-05-28T06:05:31.478083+0000","last_undegraded":"2024-05-28T06:05:31.478083+0000","last_fullsized":"2024-05-28T06:05:31.478083+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:47:21.316456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.755458+0000","last_change":"2024-05-28T06:05:30.484954+0000","last_active":"2024-05-28T06:05:31.755458+0000","last_peered":"2024-05-28T06:05:31.755458+0000","last_clean":"2024-05-28T06:05:31.755458+0000","last_became_active":"2024-05-28T06:05:30.484699+0000","last_became_peered":"2024-05-28T06:05:30.484699+0000","last_unstale":"2024-05-28T06:05:31.755458+0000","last_undegraded":"2024-05-28T06:05:31.755458+0000","last_fullsized":"2024-05-28T06:05:31.755458+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:10:49.765112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.753811+0000","last_change":"2024-05-28T06:05:29.463849+0000","last_active":"2024-05-28T06:05:31.753811+0000","last_peered":"2024-05-28T06:05:31.753811+0000","last_clean":"2024-05-28T06:05:31.753811+0000","last_became_active":"2024-05-28T06:05:29.463514+0000","last_became_peered":"2024-05-28T06:05:29.463514+0000","last_unstale":"2024-05-28T06:05:31.753811+0000","last_undegraded":"2024-05-28T06:05:31.753811+0000","last_fullsized":"2024-05-28T06:05:31.753811+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:10:48.750962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.754811+0000","last_change":"2024-05-28T06:05:30.485046+0000","last_active":"2024-05-28T06:05:31.754811+0000","last_peered":"2024-05-28T06:05:31.754811+0000","last_clean":"2024-05-28T06:05:31.754811+0000","last_became_active":"2024-05-28T06:05:30.484737+0000","last_became_peered":"2024-05-28T06:05:30.484737+0000","last_unstale":"2024-05-28T06:05:31.754811+0000","last_undegraded":"2024-05-28T06:05:31.754811+0000","last_fullsized":"2024-05-28T06:05:31.754811+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:34:24.174848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.482448+0000","last_change":"2024-05-28T06:05:29.450109+0000","last_active":"2024-05-28T06:05:31.482448+0000","last_peered":"2024-05-28T06:05:31.482448+0000","last_clean":"2024-05-28T06:05:31.482448+0000","last_became_active":"2024-05-28T06:05:29.449618+0000","last_became_peered":"2024-05-28T06:05:29.449618+0000","last_unstale":"2024-05-28T06:05:31.482448+0000","last_undegraded":"2024-05-28T06:05:31.482448+0000","last_fullsized":"2024-05-28T06:05:31.482448+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:41:31.983059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.755911+0000","last_change":"2024-05-28T06:05:30.470295+0000","last_active":"2024-05-28T06:05:31.755911+0000","last_peered":"2024-05-28T06:05:31.755911+0000","last_clean":"2024-05-28T06:05:31.755911+0000","last_became_active":"2024-05-28T06:05:30.470025+0000","last_became_peered":"2024-05-28T06:05:30.470025+0000","last_unstale":"2024-05-28T06:05:31.755911+0000","last_undegraded":"2024-05-28T06:05:31.755911+0000","last_fullsized":"2024-05-28T06:05:31.755911+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:38:12.995079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.756136+0000","last_change":"2024-05-28T06:05:29.467769+0000","last_active":"2024-05-28T06:05:31.756136+0000","last_peered":"2024-05-28T06:05:31.756136+0000","last_clean":"2024-05-28T06:05:31.756136+0000","last_became_active":"2024-05-28T06:05:29.467422+0000","last_became_peered":"2024-05-28T06:05:29.467422+0000","last_unstale":"2024-05-28T06:05:31.756136+0000","last_undegraded":"2024-05-28T06:05:31.756136+0000","last_fullsized":"2024-05-28T06:05:31.756136+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:42:03.958730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.754971+0000","last_change":"2024-05-28T06:05:30.485976+0000","last_active":"2024-05-28T06:05:31.754971+0000","last_peered":"2024-05-28T06:05:31.754971+0000","last_clean":"2024-05-28T06:05:31.754971+0000","last_became_active":"2024-05-28T06:05:30.485689+0000","last_became_peered":"2024-05-28T06:05:30.485689+0000","last_unstale":"2024-05-28T06:05:31.754971+0000","last_undegraded":"2024-05-28T06:05:31.754971+0000","last_fullsized":"2024-05-28T06:05:31.754971+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:22:39.599053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.488393+0000","last_change":"2024-05-28T06:05:29.471119+0000","last_active":"2024-05-28T06:05:31.488393+0000","last_peered":"2024-05-28T06:05:31.488393+0000","last_clean":"2024-05-28T06:05:31.488393+0000","last_became_active":"2024-05-28T06:05:29.470884+0000","last_became_peered":"2024-05-28T06:05:29.470884+0000","last_unstale":"2024-05-28T06:05:31.488393+0000","last_undegraded":"2024-05-28T06:05:31.488393+0000","last_fullsized":"2024-05-28T06:05:31.488393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:57:51.752948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.757909+0000","last_change":"2024-05-28T06:05:30.471622+0000","last_active":"2024-05-28T06:05:31.757909+0000","last_peered":"2024-05-28T06:05:31.757909+0000","last_clean":"2024-05-28T06:05:31.757909+0000","last_became_active":"2024-05-28T06:05:30.471379+0000","last_became_peered":"2024-05-28T06:05:30.471379+0000","last_unstale":"2024-05-28T06:05:31.757909+0000","last_undegraded":"2024-05-28T06:05:31.757909+0000","last_fullsized":"2024-05-28T06:05:31.757909+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:21:35.302971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"33'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.478413+0000","last_change":"2024-05-28T06:05:29.449898+0000","last_active":"2024-05-28T06:05:31.478413+0000","last_peered":"2024-05-28T06:05:31.478413+0000","last_clean":"2024-05-28T06:05:31.478413+0000","last_became_active":"2024-05-28T06:05:29.449322+0000","last_became_peered":"2024-05-28T06:05:29.449322+0000","last_unstale":"2024-05-28T06:05:31.478413+0000","last_undegraded":"2024-05-28T06:05:31.478413+0000","last_fullsized":"2024-05-28T06:05:31.478413+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:11:47.259186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.477884+0000","last_change":"2024-05-28T06:05:30.487595+0000","last_active":"2024-05-28T06:05:31.477884+0000","last_peered":"2024-05-28T06:05:31.477884+0000","last_clean":"2024-05-28T06:05:31.477884+0000","last_became_active":"2024-05-28T06:05:30.485508+0000","last_became_peered":"2024-05-28T06:05:30.485508+0000","last_unstale":"2024-05-28T06:05:31.477884+0000","last_undegraded":"2024-05-28T06:05:31.477884+0000","last_fullsized":"2024-05-28T06:05:31.477884+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:41:24.956597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.487750+0000","last_change":"2024-05-28T06:05:29.451001+0000","last_active":"2024-05-28T06:05:31.487750+0000","last_peered":"2024-05-28T06:05:31.487750+0000","last_clean":"2024-05-28T06:05:31.487750+0000","last_became_active":"2024-05-28T06:05:29.450717+0000","last_became_peered":"2024-05-28T06:05:29.450717+0000","last_unstale":"2024-05-28T06:05:31.487750+0000","last_undegraded":"2024-05-28T06:05:31.487750+0000","last_fullsized":"2024-05-28T06:05:31.487750+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:58:31.330922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.753649+0000","last_change":"2024-05-28T06:05:30.479494+0000","last_active":"2024-05-28T06:05:31.753649+0000","last_peered":"2024-05-28T06:05:31.753649+0000","last_clean":"2024-05-28T06:05:31.753649+0000","last_became_active":"2024-05-28T06:05:30.479234+0000","last_became_peered":"2024-05-28T06:05:30.479234+0000","last_unstale":"2024-05-28T06:05:31.753649+0000","last_undegraded":"2024-05-28T06:05:31.753649+0000","last_fullsized":"2024-05-28T06:05:31.753649+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:33:36.484602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.478259+0000","last_change":"2024-05-28T06:05:29.469461+0000","last_active":"2024-05-28T06:05:31.478259+0000","last_peered":"2024-05-28T06:05:31.478259+0000","last_clean":"2024-05-28T06:05:31.478259+0000","last_became_active":"2024-05-28T06:05:29.469114+0000","last_became_peered":"2024-05-28T06:05:29.469114+0000","last_unstale":"2024-05-28T06:05:31.478259+0000","last_undegraded":"2024-05-28T06:05:31.478259+0000","last_fullsized":"2024-05-28T06:05:31.478259+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:12:38.485450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.478074+0000","last_change":"2024-05-28T06:05:30.472808+0000","last_active":"2024-05-28T06:05:31.478074+0000","last_peered":"2024-05-28T06:05:31.478074+0000","last_clean":"2024-05-28T06:05:31.478074+0000","last_became_active":"2024-05-28T06:05:30.472492+0000","last_became_peered":"2024-05-28T06:05:30.472492+0000","last_unstale":"2024-05-28T06:05:31.478074+0000","last_undegraded":"2024-05-28T06:05:31.478074+0000","last_fullsized":"2024-05-28T06:05:31.478074+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:14:29.844730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.487917+0000","last_change":"2024-05-28T06:05:29.469913+0000","last_active":"2024-05-28T06:05:31.487917+0000","last_peered":"2024-05-28T06:05:31.487917+0000","last_clean":"2024-05-28T06:05:31.487917+0000","last_became_active":"2024-05-28T06:05:29.469460+0000","last_became_peered":"2024-05-28T06:05:29.469460+0000","last_unstale":"2024-05-28T06:05:31.487917+0000","last_undegraded":"2024-05-28T06:05:31.487917+0000","last_fullsized":"2024-05-28T06:05:31.487917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:11:47.259186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.752711+0000","last_change":"2024-05-28T06:05:30.483504+0000","last_active":"2024-05-28T06:05:31.752711+0000","last_peered":"2024-05-28T06:05:31.752711+0000","last_clean":"2024-05-28T06:05:31.752711+0000","last_became_active":"2024-05-28T06:05:30.483140+0000","last_became_peered":"2024-05-28T06:05:30.483140+0000","last_unstale":"2024-05-28T06:05:31.752711+0000","last_undegraded":"2024-05-28T06:05:31.752711+0000","last_fullsized":"2024-05-28T06:05:31.752711+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:38:12.995079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.482596+0000","last_change":"2024-05-28T06:05:29.469678+0000","last_active":"2024-05-28T06:05:31.482596+0000","last_peered":"2024-05-28T06:05:31.482596+0000","last_clean":"2024-05-28T06:05:31.482596+0000","last_became_active":"2024-05-28T06:05:29.469209+0000","last_became_peered":"2024-05-28T06:05:29.469209+0000","last_unstale":"2024-05-28T06:05:31.482596+0000","last_undegraded":"2024-05-28T06:05:31.482596+0000","last_fullsized":"2024-05-28T06:05:31.482596+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:22:38.584903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.482651+0000","last_change":"2024-05-28T06:05:30.474607+0000","last_active":"2024-05-28T06:05:31.482651+0000","last_peered":"2024-05-28T06:05:31.482651+0000","last_clean":"2024-05-28T06:05:31.482651+0000","last_became_active":"2024-05-28T06:05:30.474286+0000","last_became_peered":"2024-05-28T06:05:30.474286+0000","last_unstale":"2024-05-28T06:05:31.482651+0000","last_undegraded":"2024-05-28T06:05:31.482651+0000","last_fullsized":"2024-05-28T06:05:31.482651+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:37:19.547704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.753119+0000","last_change":"2024-05-28T06:05:29.465063+0000","last_active":"2024-05-28T06:05:31.753119+0000","last_peered":"2024-05-28T06:05:31.753119+0000","last_clean":"2024-05-28T06:05:31.753119+0000","last_became_active":"2024-05-28T06:05:29.464579+0000","last_became_peered":"2024-05-28T06:05:29.464579+0000","last_unstale":"2024-05-28T06:05:31.753119+0000","last_undegraded":"2024-05-28T06:05:31.753119+0000","last_fullsized":"2024-05-28T06:05:31.753119+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:42:03.958730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.755773+0000","last_change":"2024-05-28T06:05:30.479434+0000","last_active":"2024-05-28T06:05:31.755773+0000","last_peered":"2024-05-28T06:05:31.755773+0000","last_clean":"2024-05-28T06:05:31.755773+0000","last_became_active":"2024-05-28T06:05:30.479148+0000","last_became_peered":"2024-05-28T06:05:30.479148+0000","last_unstale":"2024-05-28T06:05:31.755773+0000","last_undegraded":"2024-05-28T06:05:31.755773+0000","last_fullsized":"2024-05-28T06:05:31.755773+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:29.416642+0000","last_clean_scrub_stamp":"2024-05-28T06:05:29.416642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:56:43.306179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.753427+0000","last_change":"2024-05-28T06:05:29.461542+0000","last_active":"2024-05-28T06:05:31.753427+0000","last_peered":"2024-05-28T06:05:31.753427+0000","last_clean":"2024-05-28T06:05:31.753427+0000","last_became_active":"2024-05-28T06:05:29.461155+0000","last_became_peered":"2024-05-28T06:05:29.461155+0000","last_unstale":"2024-05-28T06:05:31.753427+0000","last_undegraded":"2024-05-28T06:05:31.753427+0000","last_fullsized":"2024-05-28T06:05:31.753427+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:05:28.402491+0000","last_clean_scrub_stamp":"2024-05-28T06:05:28.402491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:12:38.485450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":194,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-28T06:05:31.755508+0000","last_change":"2024-05-28T06:04:01.039129+0000","last_active":"2024-05-28T06:05:31.755508+0000","last_peered":"2024-05-28T06:05:31.755508+0000","last_clean":"2024-05-28T06:05:31.755508+0000","last_became_active":"2024-05-28T06:04:01.033906+0000","last_became_peered":"2024-05-28T06:04:01.033906+0000","last_unstale":"2024-05-28T06:05:31.755508+0000","last_undegraded":"2024-05-28T06:05:31.755508+0000","last_fullsized":"2024-05-28T06:05:31.755508+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:03:59.864237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:03:59.864237+0000","last_clean_scrub_stamp":"2024-05-28T06:03:59.864237+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:17:36.440245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574870,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27968,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343872,"statfs":{"total":96636764160,"available":96608124928,"internally_reserved":0,"allocated":970752,"data_stored":737074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574872,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27396,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":380928,"data_stored":147232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574870,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27396,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":380928,"data_stored":147288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574870,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":380928,"data_stored":146720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574872,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27408,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344432,"statfs":{"total":96636764160,"available":96608698368,"internally_reserved":0,"allocated":389120,"data_stored":148250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574871,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27396,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":376832,"data_stored":146686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574871,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":389120,"data_stored":148250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574870,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27964,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343876,"statfs":{"total":96636764160,"available":96608129024,"internally_reserved":0,"allocated":966656,"data_stored":736984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-28T06:05:40.359 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-05-28T06:05:40.359 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-05-28T06:05:40.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.602+0000 7fbb2fe00640 1 Processor -- start 2024-05-28T06:05:40.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.602+0000 7fbb2fe00640 1 -- start start 2024-05-28T06:05:40.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.603+0000 7fbb2fe00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbb30153510 0x7fbb300b4d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:40.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.603+0000 7fbb2fe00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fbb300ccb90 0x7fbb300ccff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:40.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.603+0000 7fbb2fe00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb3015d900 con 0x7fbb3015cb60 2024-05-28T06:05:40.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.603+0000 7fbb2fe00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fbb301538f0 con 0x7fbb300ccb90 2024-05-28T06:05:40.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.603+0000 7fbb2fe00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fbb300cd530 con 0x7fbb30153510 2024-05-28T06:05:40.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.603+0000 7fbb2f400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fbb3015cb60 0x7fbb3015cff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:59688/0 (socket says 172.21.15.96:59688) 2024-05-28T06:05:40.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.603+0000 7fbb2f400640 1 -- 172.21.15.96:0/1764254174 learned_addr learned my addr 172.21.15.96:0/1764254174 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:05:40.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.604+0000 7fbb27e00640 1 --2- 172.21.15.96:0/1764254174 >> v2:172.21.15.123:3300/0 conn(0x7fbb300ccb90 0x7fbb300ccff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:40.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.604+0000 7fbb2e000640 1 -- 172.21.15.96:0/1764254174 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1081395855 0 0) 0x7fbb3015d900 con 0x7fbb3015cb60 2024-05-28T06:05:40.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.604+0000 7fbb2e000640 1 -- 172.21.15.96:0/1764254174 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbb14003520 con 0x7fbb3015cb60 2024-05-28T06:05:40.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.604+0000 7fbb2ea00640 1 --2- 172.21.15.96:0/1764254174 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbb30153510 0x7fbb300b4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:40.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.604+0000 7fbb2e000640 1 -- 172.21.15.96:0/1764254174 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 760752006 0 0) 0x7fbb14003520 con 0x7fbb3015cb60 2024-05-28T06:05:40.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.604+0000 7fbb2e000640 1 -- 172.21.15.96:0/1764254174 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbb30153510 msgr2=0x7fbb300b4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:40.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.604+0000 7fbb2e000640 1 --2- 172.21.15.96:0/1764254174 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbb30153510 0x7fbb300b4d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:40.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.604+0000 7fbb2e000640 1 -- 172.21.15.96:0/1764254174 >> v2:172.21.15.123:3300/0 conn(0x7fbb300ccb90 msgr2=0x7fbb300ccff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:40.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.604+0000 7fbb2e000640 1 --2- 172.21.15.96:0/1764254174 >> v2:172.21.15.123:3300/0 conn(0x7fbb300ccb90 0x7fbb300ccff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:40.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.604+0000 7fbb2e000640 1 -- 172.21.15.96:0/1764254174 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb300b3f90 con 0x7fbb3015cb60 2024-05-28T06:05:40.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.604+0000 7fbb2e000640 1 -- 172.21.15.96:0/1764254174 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7fbb280019f0 con 0x7fbb3015cb60 2024-05-28T06:05:40.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.604+0000 7fbb2e000640 1 -- 172.21.15.96:0/1764254174 >> v1:172.21.15.96:6789/0 conn(0x7fbb3015cb60 legacy=0x7fbb3015cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:40.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.604+0000 7fbb2e000640 1 --2- 172.21.15.96:0/1764254174 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbb140039b0 0x7fbb14003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:40.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.604+0000 7fbb2e000640 1 --2- 172.21.15.96:0/1764254174 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbb30153510 0x7fbb14004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:40.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.604+0000 7fbb2e000640 1 --2- 172.21.15.96:0/1764254174 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbb14004d40 0x7fbb140051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:40.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.604+0000 7fbb2e000640 1 -- 172.21.15.96:0/1764254174 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fbb1400fe80 con 0x7fbb14004d40 2024-05-28T06:05:40.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.604+0000 7fbb2e000640 1 -- 172.21.15.96:0/1764254174 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fbb14010030 con 0x7fbb140039b0 2024-05-28T06:05:40.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.604+0000 7fbb2e000640 1 -- 172.21.15.96:0/1764254174 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fbb14010230 con 0x7fbb30153510 2024-05-28T06:05:40.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.605+0000 7fbb2f400640 1 --2- 172.21.15.96:0/1764254174 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbb14004d40 0x7fbb140051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:40.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.605+0000 7fbb2ea00640 1 --2- 172.21.15.96:0/1764254174 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbb140039b0 0x7fbb14003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:40.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.605+0000 7fbb27e00640 1 --2- 172.21.15.96:0/1764254174 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbb30153510 0x7fbb14004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:40.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.605+0000 7fbb2f400640 1 -- 172.21.15.96:0/1764254174 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbb30153510 msgr2=0x7fbb14004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:40.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.605+0000 7fbb2f400640 1 --2- 172.21.15.96:0/1764254174 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbb30153510 0x7fbb14004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:40.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.605+0000 7fbb2f400640 1 -- 172.21.15.96:0/1764254174 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbb140039b0 msgr2=0x7fbb14003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:40.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.605+0000 7fbb2f400640 1 --2- 172.21.15.96:0/1764254174 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbb140039b0 0x7fbb14003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:40.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.605+0000 7fbb2f400640 1 -- 172.21.15.96:0/1764254174 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb14003520 con 0x7fbb14004d40 2024-05-28T06:05:40.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.605+0000 7fbb27e00640 1 --2- 172.21.15.96:0/1764254174 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbb30153510 0x7fbb14004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:05:40.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.606+0000 7fbb2f400640 1 --2- 172.21.15.96:0/1764254174 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbb14004d40 0x7fbb140051c0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fbb2800a920 tx=0x7fbb14005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=121ad7faf84833c7 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:40.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.606+0000 7fbb2e000640 1 -- 172.21.15.96:0/1764254174 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb2800a4c0 con 0x7fbb14004d40 2024-05-28T06:05:40.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.606+0000 7fbb2e000640 1 -- 172.21.15.96:0/1764254174 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb2802f070 con 0x7fbb14004d40 2024-05-28T06:05:40.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.606+0000 7fbb2e000640 1 -- 172.21.15.96:0/1764254174 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb2802bd00 con 0x7fbb14004d40 2024-05-28T06:05:40.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.607+0000 7fbb2fe00640 1 -- 172.21.15.96:0/1764254174 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbb14004d40 msgr2=0x7fbb140051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:40.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.607+0000 7fbb2fe00640 1 --2- 172.21.15.96:0/1764254174 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbb14004d40 0x7fbb140051c0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fbb2800a920 tx=0x7fbb14005830 comp rx=0 tx=0).stop 2024-05-28T06:05:40.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.607+0000 7fbb2f400640 1 -- 172.21.15.96:0/1764254174 reap_dead start 2024-05-28T06:05:40.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.607+0000 7fbb2fe00640 1 -- 172.21.15.96:0/1764254174 shutdown_connections 2024-05-28T06:05:40.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.607+0000 7fbb2fe00640 1 -- 172.21.15.96:0/1764254174 >> 172.21.15.96:0/1764254174 conn(0x7fbb3006b900 msgr2=0x7fbb3006bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:40.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.607+0000 7fbb2fe00640 1 -- 172.21.15.96:0/1764254174 shutdown_connections 2024-05-28T06:05:40.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.607+0000 7fbb2fe00640 1 -- 172.21.15.96:0/1764254174 wait complete. 2024-05-28T06:05:40.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.609+0000 7fbb2fe00640 1 Processor -- start 2024-05-28T06:05:40.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.609+0000 7fbb2fe00640 1 -- start start 2024-05-28T06:05:40.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.610+0000 7fbb2fe00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbb14004d40 0x7fbb300727b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:40.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.610+0000 7fbb2fe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbb30149fa0 0x7fbb300ce080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:40.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.610+0000 7fbb2fe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbb30072ef0 0x7fbb3015db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:40.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.610+0000 7fbb2fe00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fbb300b4550 con 0x7fbb30149fa0 2024-05-28T06:05:40.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.610+0000 7fbb2fe00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fbb300b48c0 con 0x7fbb30072ef0 2024-05-28T06:05:40.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.610+0000 7fbb2fe00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fbb300b4d50 con 0x7fbb14004d40 2024-05-28T06:05:40.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.610+0000 7fbb27e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbb30149fa0 0x7fbb300ce080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:40.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.610+0000 7fbb27e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbb30149fa0 0x7fbb300ce080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:42180/0 (socket says 172.21.15.96:42180) 2024-05-28T06:05:40.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.610+0000 7fbb27e00640 1 -- 172.21.15.96:0/526396304 learned_addr learned my addr 172.21.15.96:0/526396304 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:40.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.610+0000 7fbb27e00640 1 -- 172.21.15.96:0/526396304 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbb14004d40 msgr2=0x7fbb300727b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:40.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.610+0000 7fbb27e00640 1 --2- 172.21.15.96:0/526396304 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbb14004d40 0x7fbb300727b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:40.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.610+0000 7fbb27e00640 1 -- 172.21.15.96:0/526396304 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbb30072ef0 msgr2=0x7fbb3015db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:40.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.610+0000 7fbb27e00640 1 --2- 172.21.15.96:0/526396304 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbb30072ef0 0x7fbb3015db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:40.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.610+0000 7fbb27e00640 1 -- 172.21.15.96:0/526396304 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb301ef960 con 0x7fbb30149fa0 2024-05-28T06:05:40.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.611+0000 7fbb27e00640 1 --2- 172.21.15.96:0/526396304 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbb30149fa0 0x7fbb300ce080 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fbb1800e810 tx=0x7fbb1800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:40.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.611+0000 7fbb26a00640 1 -- 172.21.15.96:0/526396304 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb18017070 con 0x7fbb30149fa0 2024-05-28T06:05:40.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.611+0000 7fbb26a00640 1 -- 172.21.15.96:0/526396304 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb1800c850 con 0x7fbb30149fa0 2024-05-28T06:05:40.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.611+0000 7fbb26a00640 1 -- 172.21.15.96:0/526396304 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb1800f040 con 0x7fbb30149fa0 2024-05-28T06:05:40.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.611+0000 7fbb2fe00640 1 -- 172.21.15.96:0/526396304 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb300737b0 con 0x7fbb30149fa0 2024-05-28T06:05:40.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.611+0000 7fbb2fe00640 1 -- 172.21.15.96:0/526396304 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb30071fb0 con 0x7fbb30149fa0 2024-05-28T06:05:40.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.612+0000 7fbb2fe00640 1 -- 172.21.15.96:0/526396304 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb300689e0 con 0x7fbb30149fa0 2024-05-28T06:05:40.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.613+0000 7fbb26a00640 1 -- 172.21.15.96:0/526396304 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fbb1801a070 con 0x7fbb30149fa0 2024-05-28T06:05:40.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.614+0000 7fbb26a00640 1 --2- 172.21.15.96:0/526396304 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbb0c06c8f0 0x7fbb0c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:40.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.614+0000 7fbb26a00640 1 -- 172.21.15.96:0/526396304 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7fbb1808b770 con 0x7fbb30149fa0 2024-05-28T06:05:40.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.615+0000 7fbb2ea00640 1 --2- 172.21.15.96:0/526396304 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbb0c06c8f0 0x7fbb0c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:40.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.616+0000 7fbb2ea00640 1 --2- 172.21.15.96:0/526396304 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbb0c06c8f0 0x7fbb0c06edb0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fbb20006fd0 tx=0x7fbb2000b2e0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:40.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.618+0000 7fbb26a00640 1 -- 172.21.15.96:0/526396304 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbb18059b00 con 0x7fbb30149fa0 2024-05-28T06:05:40.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.871+0000 7fbb2fe00640 1 -- 172.21.15.96:0/526396304 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7fbb3006ecb0 con 0x7fbb30149fa0 2024-05-28T06:05:40.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.876+0000 7fbb26a00640 1 -- 172.21.15.96:0/526396304 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v13) ==== 190+0+147 (secure 0 0 0) 0x7fbb18059400 con 0x7fbb30149fa0 2024-05-28T06:05:40.877 INFO:teuthology.orchestra.run.smithi096.stderr:updated caps for client.0 2024-05-28T06:05:40.877 INFO:teuthology.orchestra.run.smithi096.stdout:[client.0] 2024-05-28T06:05:40.877 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQCXc1Vm91hbGBAASTL3HQR8XOs7wVQEHbPl/g== 2024-05-28T06:05:40.877 INFO:teuthology.orchestra.run.smithi096.stdout: caps mds = "allow" 2024-05-28T06:05:40.877 INFO:teuthology.orchestra.run.smithi096.stdout: caps mon = "allow r" 2024-05-28T06:05:40.877 INFO:teuthology.orchestra.run.smithi096.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-05-28T06:05:40.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.879+0000 7fbb2fe00640 1 -- 172.21.15.96:0/526396304 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbb0c06c8f0 msgr2=0x7fbb0c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:40.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.879+0000 7fbb2fe00640 1 --2- 172.21.15.96:0/526396304 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbb0c06c8f0 0x7fbb0c06edb0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fbb20006fd0 tx=0x7fbb2000b2e0 comp rx=0 tx=0).stop 2024-05-28T06:05:40.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.879+0000 7fbb2fe00640 1 -- 172.21.15.96:0/526396304 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbb30149fa0 msgr2=0x7fbb300ce080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:40.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.879+0000 7fbb2fe00640 1 --2- 172.21.15.96:0/526396304 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbb30149fa0 0x7fbb300ce080 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fbb1800e810 tx=0x7fbb1800ecd0 comp rx=0 tx=0).stop 2024-05-28T06:05:40.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.879+0000 7fbb2fe00640 1 -- 172.21.15.96:0/526396304 shutdown_connections 2024-05-28T06:05:40.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.879+0000 7fbb2fe00640 1 --2- 172.21.15.96:0/526396304 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbb0c06c8f0 0x7fbb0c06edb0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:40.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.879+0000 7fbb2fe00640 1 --2- 172.21.15.96:0/526396304 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbb30072ef0 0x7fbb3015db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:40.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.879+0000 7fbb2fe00640 1 --2- 172.21.15.96:0/526396304 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbb30149fa0 0x7fbb300ce080 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:40.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.879+0000 7fbb2fe00640 1 --2- 172.21.15.96:0/526396304 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbb14004d40 0x7fbb300727b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:40.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.879+0000 7fbb2fe00640 1 -- 172.21.15.96:0/526396304 >> 172.21.15.96:0/526396304 conn(0x7fbb3006b900 msgr2=0x7fbb3015f480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:40.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.879+0000 7fbb2fe00640 1 -- 172.21.15.96:0/526396304 shutdown_connections 2024-05-28T06:05:40.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:40.879+0000 7fbb2fe00640 1 -- 172.21.15.96:0/526396304 wait complete. 2024-05-28T06:05:40.898 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-28T06:05:41.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.095+0000 7ffbabe00640 1 Processor -- start 2024-05-28T06:05:41.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.095+0000 7ffbabe00640 1 -- start start 2024-05-28T06:05:41.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.096+0000 7ffbabe00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7ffbac06ee40 0x7ffbac0ad230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:41.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.096+0000 7ffbabe00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffbac06f2f0 0x7ffbac0ad770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:41.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.096+0000 7ffbabe00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffbac156f10 con 0x7ffbac156b30 2024-05-28T06:05:41.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.096+0000 7ffbabe00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7ffbac06dda0 con 0x7ffbac06ee40 2024-05-28T06:05:41.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.096+0000 7ffbabe00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ffbac06e080 con 0x7ffbac06f2f0 2024-05-28T06:05:41.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.097+0000 7ffbaaa00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7ffbac156b30 0x7ffbac0af6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:59698/0 (socket says 172.21.15.96:59698) 2024-05-28T06:05:41.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.097+0000 7ffbaaa00640 1 -- 172.21.15.96:0/2665907005 learned_addr learned my addr 172.21.15.96:0/2665907005 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:05:41.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.097+0000 7ffbaa000640 1 --2- 172.21.15.96:0/2665907005 >> v2:172.21.15.123:3300/0 conn(0x7ffbac06ee40 0x7ffbac0ad230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:41.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.097+0000 7ffbab400640 1 --2- 172.21.15.96:0/2665907005 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffbac06f2f0 0x7ffbac0ad770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:41.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.098+0000 7ffba9600640 1 -- 172.21.15.96:0/2665907005 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3222850217 0 0) 0x7ffbac156f10 con 0x7ffbac156b30 2024-05-28T06:05:41.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.098+0000 7ffba9600640 1 -- 172.21.15.96:0/2665907005 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffb8c003520 con 0x7ffbac156b30 2024-05-28T06:05:41.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.098+0000 7ffbab400640 1 -- 172.21.15.96:0/2665907005 >> v2:172.21.15.123:3300/0 conn(0x7ffbac06ee40 msgr2=0x7ffbac0ad230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:41.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.098+0000 7ffbab400640 1 --2- 172.21.15.96:0/2665907005 >> v2:172.21.15.123:3300/0 conn(0x7ffbac06ee40 0x7ffbac0ad230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:41.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.098+0000 7ffbab400640 1 -- 172.21.15.96:0/2665907005 >> v1:172.21.15.96:6789/0 conn(0x7ffbac156b30 legacy=0x7ffbac0af6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:41.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.098+0000 7ffba9600640 1 -- 172.21.15.96:0/2665907005 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1245916134 0 0) 0x7ffb8c003520 con 0x7ffbac156b30 2024-05-28T06:05:41.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.098+0000 7ffbab400640 1 -- 172.21.15.96:0/2665907005 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbac06e360 con 0x7ffbac06f2f0 2024-05-28T06:05:41.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.098+0000 7ffbaa000640 1 --2- 172.21.15.96:0/2665907005 >> v2:172.21.15.123:3300/0 conn(0x7ffbac06ee40 0x7ffbac0ad230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-28T06:05:41.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.099+0000 7ffbab400640 1 --2- 172.21.15.96:0/2665907005 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffbac06f2f0 0x7ffbac0ad770 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ffba40097a0 tx=0x7ffba402a080 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea1f6083bb78f2cc server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:41.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.100+0000 7ffba9600640 1 -- 172.21.15.96:0/2665907005 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffba4037070 con 0x7ffbac06f2f0 2024-05-28T06:05:41.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.100+0000 7ffba9600640 1 -- 172.21.15.96:0/2665907005 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffba402ab10 con 0x7ffbac06f2f0 2024-05-28T06:05:41.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.100+0000 7ffba9600640 1 -- 172.21.15.96:0/2665907005 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb9c001670 con 0x7ffbac06f2f0 2024-05-28T06:05:41.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.100+0000 7ffbabe00640 1 -- 172.21.15.96:0/2665907005 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffbac06f2f0 msgr2=0x7ffbac0ad770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:41.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.100+0000 7ffbabe00640 1 --2- 172.21.15.96:0/2665907005 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffbac06f2f0 0x7ffbac0ad770 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ffba40097a0 tx=0x7ffba402a080 comp rx=0 tx=0).stop 2024-05-28T06:05:41.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.101+0000 7ffbabe00640 1 -- 172.21.15.96:0/2665907005 shutdown_connections 2024-05-28T06:05:41.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.101+0000 7ffbabe00640 1 --2- 172.21.15.96:0/2665907005 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffbac06f2f0 0x7ffbac0ad770 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:41.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.101+0000 7ffbabe00640 1 --2- 172.21.15.96:0/2665907005 >> v2:172.21.15.123:3300/0 conn(0x7ffbac06ee40 0x7ffbac0ad230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:41.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.101+0000 7ffbabe00640 1 -- 172.21.15.96:0/2665907005 >> 172.21.15.96:0/2665907005 conn(0x7ffbac0b1fc0 msgr2=0x7ffbac0b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:41.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.101+0000 7ffbabe00640 1 -- 172.21.15.96:0/2665907005 shutdown_connections 2024-05-28T06:05:41.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.101+0000 7ffbabe00640 1 -- 172.21.15.96:0/2665907005 wait complete. 2024-05-28T06:05:41.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.103+0000 7ffbabe00640 1 Processor -- start 2024-05-28T06:05:41.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.103+0000 7ffbabe00640 1 -- start start 2024-05-28T06:05:41.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.104+0000 7ffbabe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffbac06ee40 0x7ffbac1599d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:41.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.104+0000 7ffbabe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffbac06f2f0 0x7ffbac1f74e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:41.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.104+0000 7ffbabe00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffbac156b30 0x7ffbac14d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:41.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.104+0000 7ffbabe00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ffbac0b2db0 con 0x7ffbac06ee40 2024-05-28T06:05:41.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.104+0000 7ffbabe00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ffbac0b2c30 con 0x7ffbac06f2f0 2024-05-28T06:05:41.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.104+0000 7ffbabe00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ffbac0b32c0 con 0x7ffbac156b30 2024-05-28T06:05:41.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.104+0000 7ffbaaa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffbac06ee40 0x7ffbac1599d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:41.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.104+0000 7ffbaaa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffbac06ee40 0x7ffbac1599d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:42190/0 (socket says 172.21.15.96:42190) 2024-05-28T06:05:41.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.104+0000 7ffbaaa00640 1 -- 172.21.15.96:0/2455352301 learned_addr learned my addr 172.21.15.96:0/2455352301 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:41.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.104+0000 7ffbaa000640 1 --2- 172.21.15.96:0/2455352301 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffbac06f2f0 0x7ffbac1f74e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:41.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.104+0000 7ffbab400640 1 --2- 172.21.15.96:0/2455352301 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffbac156b30 0x7ffbac14d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:41.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.104+0000 7ffbaaa00640 1 -- 172.21.15.96:0/2455352301 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffbac156b30 msgr2=0x7ffbac14d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:41.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.104+0000 7ffbaaa00640 1 --2- 172.21.15.96:0/2455352301 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffbac156b30 0x7ffbac14d620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:41.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.104+0000 7ffbaaa00640 1 -- 172.21.15.96:0/2455352301 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffbac06f2f0 msgr2=0x7ffbac1f74e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:41.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.104+0000 7ffbaaa00640 1 --2- 172.21.15.96:0/2455352301 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffbac06f2f0 0x7ffbac1f74e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:41.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.104+0000 7ffbaaa00640 1 -- 172.21.15.96:0/2455352301 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbac14ddc0 con 0x7ffbac06ee40 2024-05-28T06:05:41.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.104+0000 7ffbaa000640 1 --2- 172.21.15.96:0/2455352301 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffbac06f2f0 0x7ffbac1f74e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:05:41.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.105+0000 7ffbaaa00640 1 --2- 172.21.15.96:0/2455352301 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffbac06ee40 0x7ffbac1599d0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7ffb9c00c700 tx=0x7ffb9c00cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:41.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.105+0000 7ffba3400640 1 -- 172.21.15.96:0/2455352301 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb9c017070 con 0x7ffbac06ee40 2024-05-28T06:05:41.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.105+0000 7ffba3400640 1 -- 172.21.15.96:0/2455352301 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb9c005ad0 con 0x7ffbac06ee40 2024-05-28T06:05:41.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.105+0000 7ffbabe00640 1 -- 172.21.15.96:0/2455352301 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffbac14e0a0 con 0x7ffbac06ee40 2024-05-28T06:05:41.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.105+0000 7ffba3400640 1 -- 172.21.15.96:0/2455352301 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb9c007070 con 0x7ffbac06ee40 2024-05-28T06:05:41.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.105+0000 7ffbabe00640 1 -- 172.21.15.96:0/2455352301 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffbac157890 con 0x7ffbac06ee40 2024-05-28T06:05:41.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.106+0000 7ffbabe00640 1 -- 172.21.15.96:0/2455352301 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffbac0689e0 con 0x7ffbac06ee40 2024-05-28T06:05:41.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.107+0000 7ffba3400640 1 -- 172.21.15.96:0/2455352301 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ffb9c01a070 con 0x7ffbac06ee40 2024-05-28T06:05:41.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.108+0000 7ffba3400640 1 --2- 172.21.15.96:0/2455352301 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ffb8806c8f0 0x7ffb8806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:41.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.108+0000 7ffba3400640 1 -- 172.21.15.96:0/2455352301 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7ffb9c08afd0 con 0x7ffbac06ee40 2024-05-28T06:05:41.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.109+0000 7ffbaa000640 1 --2- 172.21.15.96:0/2455352301 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ffb8806c8f0 0x7ffb8806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:41.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.110+0000 7ffbaa000640 1 --2- 172.21.15.96:0/2455352301 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ffb8806c8f0 0x7ffb8806edb0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7ffbac148eb0 tx=0x7ffb9400b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:41.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.113+0000 7ffba3400640 1 -- 172.21.15.96:0/2455352301 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ffb9c058c80 con 0x7ffbac06ee40 2024-05-28T06:05:41.350 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.350+0000 7ffbabe00640 1 -- 172.21.15.96:0/2455352301 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffbac15b620 con 0x7ffbac06ee40 2024-05-28T06:05:41.352 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.351+0000 7ffba3400640 1 -- 172.21.15.96:0/2455352301 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) ==== 94+0+5178 (secure 0 0 0) 0x7ffb9c05cf80 con 0x7ffbac06ee40 2024-05-28T06:05:41.352 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:05:41.352 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4580,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6836/918905879","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":918905879},{"type":"v1","addr":"172.21.15.123:6836","nonce":918905879}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":5223,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6834/697317719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6833","nonce":697317719},{"type":"v1","addr":"172.21.15.96:6834","nonce":697317719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":5226,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/1327552826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":1327552826},{"type":"v1","addr":"172.21.15.96:6835","nonce":1327552826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-28T06:05:29.892912+0000","modified":"2024-05-28T06:05:32.976408+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5238":{"gid":5238,"name":"d","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6837/458334664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6835","nonce":458334664},{"type":"v1","addr":"172.21.15.123:6837","nonce":458334664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5238,"qdb_cluster":[5238]},"id":2}]} 2024-05-28T06:05:41.353 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 15 2024-05-28T06:05:41.355 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.353+0000 7ffbabe00640 1 -- 172.21.15.96:0/2455352301 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ffb8806c8f0 msgr2=0x7ffb8806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:41.355 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.353+0000 7ffbabe00640 1 --2- 172.21.15.96:0/2455352301 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ffb8806c8f0 0x7ffb8806edb0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7ffbac148eb0 tx=0x7ffb9400b040 comp rx=0 tx=0).stop 2024-05-28T06:05:41.355 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.353+0000 7ffbabe00640 1 -- 172.21.15.96:0/2455352301 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffbac06ee40 msgr2=0x7ffbac1599d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:41.355 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.353+0000 7ffbabe00640 1 --2- 172.21.15.96:0/2455352301 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffbac06ee40 0x7ffbac1599d0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7ffb9c00c700 tx=0x7ffb9c00cbc0 comp rx=0 tx=0).stop 2024-05-28T06:05:41.355 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.353+0000 7ffbabe00640 1 -- 172.21.15.96:0/2455352301 shutdown_connections 2024-05-28T06:05:41.355 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.354+0000 7ffbabe00640 1 --2- 172.21.15.96:0/2455352301 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ffb8806c8f0 0x7ffb8806edb0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:41.355 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.354+0000 7ffbabe00640 1 --2- 172.21.15.96:0/2455352301 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffbac156b30 0x7ffbac14d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:41.355 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.354+0000 7ffbabe00640 1 --2- 172.21.15.96:0/2455352301 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffbac06f2f0 0x7ffbac1f74e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:41.355 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.354+0000 7ffbabe00640 1 --2- 172.21.15.96:0/2455352301 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffbac06ee40 0x7ffbac1599d0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:41.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.354+0000 7ffbabe00640 1 -- 172.21.15.96:0/2455352301 >> 172.21.15.96:0/2455352301 conn(0x7ffbac0b1fc0 msgr2=0x7ffbac0ad230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:41.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.354+0000 7ffbabe00640 1 -- 172.21.15.96:0/2455352301 shutdown_connections 2024-05-28T06:05:41.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.354+0000 7ffbabe00640 1 -- 172.21.15.96:0/2455352301 wait complete. 2024-05-28T06:05:41.779 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 15, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-05-28T06:05:29.892912+0000', 'modified': '2024-05-28T06:05:32.976408+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5238}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5238': {'gid': 5238, 'name': 'd', 'rank': 0, 'incarnation': 12, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.123:6837/458334664', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.123:6835', 'nonce': 458334664}, {'type': 'v1', 'addr': '172.21.15.123:6837', 'nonce': 458334664}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5238, 'qdb_cluster': [5238]} 2024-05-28T06:05:41.780 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-05-28T06:05:41.780 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-05-28T06:05:41.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.993+0000 7fc5b1800640 1 Processor -- start 2024-05-28T06:05:41.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.993+0000 7fc5b1800640 1 -- start start 2024-05-28T06:05:41.995 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.994+0000 7fc5b1800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fc5ac0d04a0 0x7fc5ac0d0930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:41.995 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.994+0000 7fc5b1800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc5ac0ad130 0x7fc5ac0ad580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:41.995 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.994+0000 7fc5b1800640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5ac06ed10 con 0x7fc5ac06f860 2024-05-28T06:05:41.995 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.994+0000 7fc5b1800640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fc5ac15cb00 con 0x7fc5ac0d04a0 2024-05-28T06:05:41.995 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.994+0000 7fc5b1800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fc5ac0b3cc0 con 0x7fc5ac0ad130 2024-05-28T06:05:41.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.995+0000 7fc5aa000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fc5ac0d04a0 0x7fc5ac0d0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:41.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.995+0000 7fc5aa000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fc5ac0d04a0 0x7fc5ac0d0930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:42200/0 (socket says 172.21.15.96:42200) 2024-05-28T06:05:41.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.995+0000 7fc5aa000640 1 -- 172.21.15.96:0/2273111555 learned_addr learned my addr 172.21.15.96:0/2273111555 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:41.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.995+0000 7fc5ab400640 1 --2- 172.21.15.96:0/2273111555 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc5ac0ad130 0x7fc5ac0ad580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:41.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.995+0000 7fc5aa000640 1 -- 172.21.15.96:0/2273111555 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc5ac0ad130 msgr2=0x7fc5ac0ad580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:41.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.995+0000 7fc5aa000640 1 --2- 172.21.15.96:0/2273111555 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc5ac0ad130 0x7fc5ac0ad580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:41.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.995+0000 7fc5aa000640 1 -- 172.21.15.96:0/2273111555 >> v1:172.21.15.123:6789/0 conn(0x7fc5ac06f860 legacy=0x7fc5ac0b3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:41.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.995+0000 7fc5aa000640 1 -- 172.21.15.96:0/2273111555 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5ac0b3fa0 con 0x7fc5ac0d04a0 2024-05-28T06:05:41.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.997+0000 7fc5aa000640 1 --2- 172.21.15.96:0/2273111555 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc5ac0d04a0 0x7fc5ac0d0930 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fc598009870 tx=0x7fc59802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af53254a3a216d1 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:41.999 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.997+0000 7fc5a9600640 1 -- 172.21.15.96:0/2273111555 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc598037070 con 0x7fc5ac0d04a0 2024-05-28T06:05:41.999 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.997+0000 7fc5a9600640 1 -- 172.21.15.96:0/2273111555 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc598032440 con 0x7fc5ac0d04a0 2024-05-28T06:05:41.999 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.997+0000 7fc5a9600640 1 -- 172.21.15.96:0/2273111555 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc598032810 con 0x7fc5ac0d04a0 2024-05-28T06:05:41.999 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.998+0000 7fc5b1800640 1 -- 172.21.15.96:0/2273111555 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc5ac0d04a0 msgr2=0x7fc5ac0d0930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:41.999 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.998+0000 7fc5b1800640 1 --2- 172.21.15.96:0/2273111555 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc5ac0d04a0 0x7fc5ac0d0930 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fc598009870 tx=0x7fc59802a270 comp rx=0 tx=0).stop 2024-05-28T06:05:41.999 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.998+0000 7fc5b1800640 1 -- 172.21.15.96:0/2273111555 shutdown_connections 2024-05-28T06:05:42.000 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.998+0000 7fc5b1800640 1 --2- 172.21.15.96:0/2273111555 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc5ac0ad130 0x7fc5ac0ad580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:42.000 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.998+0000 7fc5b1800640 1 --2- 172.21.15.96:0/2273111555 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc5ac0d04a0 0x7fc5ac0d0930 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:42.000 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.998+0000 7fc5b1800640 1 -- 172.21.15.96:0/2273111555 >> 172.21.15.96:0/2273111555 conn(0x7fc5ac06bae0 msgr2=0x7fc5ac06bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:42.000 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.998+0000 7fc5b1800640 1 -- 172.21.15.96:0/2273111555 shutdown_connections 2024-05-28T06:05:42.000 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:41.998+0000 7fc5b1800640 1 -- 172.21.15.96:0/2273111555 wait complete. 2024-05-28T06:05:42.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.000+0000 7fc5b1800640 1 Processor -- start 2024-05-28T06:05:42.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.000+0000 7fc5b1800640 1 -- start start 2024-05-28T06:05:42.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.001+0000 7fc5b1800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc5ac06f860 0x7fc5ac0999f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:42.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.001+0000 7fc5b1800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc5ac0ad130 0x7fc5ac1f5c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:42.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.001+0000 7fc5b1800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5ac0d04a0 0x7fc5ac0cc7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:42.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.001+0000 7fc5b1800640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc5ac0b4840 con 0x7fc5ac06f860 2024-05-28T06:05:42.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.001+0000 7fc5b1800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc5ac0b4560 con 0x7fc5ac0d04a0 2024-05-28T06:05:42.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.001+0000 7fc5b1800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fc5ac0b3860 con 0x7fc5ac0ad130 2024-05-28T06:05:42.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.001+0000 7fc5aaa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc5ac06f860 0x7fc5ac0999f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:42.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.001+0000 7fc5aaa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc5ac06f860 0x7fc5ac0999f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:42202/0 (socket says 172.21.15.96:42202) 2024-05-28T06:05:42.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.001+0000 7fc5aaa00640 1 -- 172.21.15.96:0/3461294253 learned_addr learned my addr 172.21.15.96:0/3461294253 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:42.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.002+0000 7fc5ab400640 1 --2- 172.21.15.96:0/3461294253 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5ac0d04a0 0x7fc5ac0cc7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:42.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.002+0000 7fc5aa000640 1 --2- 172.21.15.96:0/3461294253 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc5ac0ad130 0x7fc5ac1f5c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:42.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.002+0000 7fc5aaa00640 1 -- 172.21.15.96:0/3461294253 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc5ac0ad130 msgr2=0x7fc5ac1f5c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:42.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.002+0000 7fc5aaa00640 1 --2- 172.21.15.96:0/3461294253 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc5ac0ad130 0x7fc5ac1f5c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:42.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.002+0000 7fc5aaa00640 1 -- 172.21.15.96:0/3461294253 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5ac0d04a0 msgr2=0x7fc5ac0cc7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:42.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.002+0000 7fc5aaa00640 1 --2- 172.21.15.96:0/3461294253 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5ac0d04a0 0x7fc5ac0cc7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:42.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.002+0000 7fc5aaa00640 1 -- 172.21.15.96:0/3461294253 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5ac156db0 con 0x7fc5ac06f860 2024-05-28T06:05:42.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.002+0000 7fc5aa000640 1 --2- 172.21.15.96:0/3461294253 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc5ac0ad130 0x7fc5ac1f5c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:05:42.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.002+0000 7fc5aaa00640 1 --2- 172.21.15.96:0/3461294253 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc5ac06f860 0x7fc5ac0999f0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fc59c00c430 tx=0x7fc59c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:42.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.002+0000 7fc5a2a00640 1 -- 172.21.15.96:0/3461294253 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc59c0052d0 con 0x7fc5ac06f860 2024-05-28T06:05:42.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.002+0000 7fc5a2a00640 1 -- 172.21.15.96:0/3461294253 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc59c005490 con 0x7fc5ac06f860 2024-05-28T06:05:42.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.002+0000 7fc5b1800640 1 -- 172.21.15.96:0/3461294253 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5ac157090 con 0x7fc5ac06f860 2024-05-28T06:05:42.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.003+0000 7fc5a2a00640 1 -- 172.21.15.96:0/3461294253 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc59c007260 con 0x7fc5ac06f860 2024-05-28T06:05:42.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.003+0000 7fc5b1800640 1 -- 172.21.15.96:0/3461294253 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5ac1f6150 con 0x7fc5ac06f860 2024-05-28T06:05:42.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.005+0000 7fc5a2a00640 1 -- 172.21.15.96:0/3461294253 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc59c007400 con 0x7fc5ac06f860 2024-05-28T06:05:42.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.006+0000 7fc5a2a00640 1 --2- 172.21.15.96:0/3461294253 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc58c06c8f0 0x7fc58c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:42.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.006+0000 7fc5a2a00640 1 -- 172.21.15.96:0/3461294253 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7fc59c08b110 con 0x7fc5ac06f860 2024-05-28T06:05:42.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.006+0000 7fc5aa000640 1 --2- 172.21.15.96:0/3461294253 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc58c06c8f0 0x7fc58c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:42.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.007+0000 7fc5b45b6740 1 Processor -- start 2024-05-28T06:05:42.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.007+0000 7fc5b45b6740 1 -- start start 2024-05-28T06:05:42.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.007+0000 7fc5b45b6740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557ff70472a0 0x557ff704aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:42.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.007+0000 7fc5aa000640 1 --2- 172.21.15.96:0/3461294253 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc58c06c8f0 0x7fc58c06edb0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fc598034000 tx=0x7fc59802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:42.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.008+0000 7fc5b45b6740 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x557ff704af60 0x557ff7066ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:42.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.008+0000 7fc5b45b6740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557ff70673e0 0x557ff7067840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:42.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.008+0000 7fc5b45b6740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x557ff6f84b10 con 0x557ff70472a0 2024-05-28T06:05:42.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.008+0000 7fc5b45b6740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x557ff6f2b2b0 con 0x557ff70673e0 2024-05-28T06:05:42.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.008+0000 7fc5b45b6740 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x557ff6f80ad0 con 0x557ff704af60 2024-05-28T06:05:42.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.008+0000 7fc5aaa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557ff70472a0 0x557ff704aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:42.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.008+0000 7fc5aaa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557ff70472a0 0x557ff704aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:42210/0 (socket says 172.21.15.96:42210) 2024-05-28T06:05:42.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.008+0000 7fc5aaa00640 1 -- 172.21.15.96:0/1894014278 learned_addr learned my addr 172.21.15.96:0/1894014278 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:42.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.008+0000 7fc5aaa00640 1 -- 172.21.15.96:0/1894014278 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x557ff704af60 msgr2=0x557ff7066ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:42.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.008+0000 7fc5ab400640 1 --2- 172.21.15.96:0/1894014278 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557ff70673e0 0x557ff7067840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:42.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.008+0000 7fc5aaa00640 1 --2- 172.21.15.96:0/1894014278 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x557ff704af60 0x557ff7066ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:42.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.008+0000 7fc5aaa00640 1 -- 172.21.15.96:0/1894014278 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557ff70673e0 msgr2=0x557ff7067840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:42.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.008+0000 7fc5aaa00640 1 --2- 172.21.15.96:0/1894014278 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557ff70673e0 0x557ff7067840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:42.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.008+0000 7fc5aaa00640 1 -- 172.21.15.96:0/1894014278 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557ff6c972d0 con 0x557ff70472a0 2024-05-28T06:05:42.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.008+0000 7fc5aaa00640 1 --2- 172.21.15.96:0/1894014278 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557ff70472a0 0x557ff704aa20 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fc59c055200 tx=0x7fc59c00a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39d6d03ba4a7a3f9 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:42.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.009+0000 7fc597400640 1 -- 172.21.15.96:0/1894014278 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc59c05c070 con 0x557ff70472a0 2024-05-28T06:05:42.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.009+0000 7fc597400640 1 -- 172.21.15.96:0/1894014278 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc59c0570a0 con 0x557ff70472a0 2024-05-28T06:05:42.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.009+0000 7fc597400640 1 -- 172.21.15.96:0/1894014278 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc59c071180 con 0x557ff70472a0 2024-05-28T06:05:42.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.009+0000 7fc5b45b6740 1 -- 172.21.15.96:0/1894014278 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557ff70472a0 msgr2=0x557ff704aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:42.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.009+0000 7fc5b45b6740 1 --2- 172.21.15.96:0/1894014278 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557ff70472a0 0x557ff704aa20 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fc59c055200 tx=0x7fc59c00a000 comp rx=0 tx=0).stop 2024-05-28T06:05:42.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.009+0000 7fc5b45b6740 1 -- 172.21.15.96:0/1894014278 shutdown_connections 2024-05-28T06:05:42.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.009+0000 7fc5b45b6740 1 --2- 172.21.15.96:0/1894014278 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557ff70673e0 0x557ff7067840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:42.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.009+0000 7fc5b45b6740 1 --2- 172.21.15.96:0/1894014278 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x557ff704af60 0x557ff7066ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:42.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.009+0000 7fc5b45b6740 1 --2- 172.21.15.96:0/1894014278 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557ff70472a0 0x557ff704aa20 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:42.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.009+0000 7fc5b45b6740 1 -- 172.21.15.96:0/1894014278 >> 172.21.15.96:0/1894014278 conn(0x557ff701d280 msgr2=0x557ff701e280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:42.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.009+0000 7fc5b45b6740 1 -- 172.21.15.96:0/1894014278 shutdown_connections 2024-05-28T06:05:42.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.009+0000 7fc5b45b6740 1 -- 172.21.15.96:0/1894014278 wait complete. 2024-05-28T06:05:42.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.010+0000 7fc5b45b6740 1 -- start start 2024-05-28T06:05:42.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.010+0000 7fc5b45b6740 1 Processor -- start 2024-05-28T06:05:42.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.011+0000 7fc5b45b6740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557ff704af60 0x557ff7046770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:42.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.011+0000 7fc5b45b6740 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x557ff70673e0 0x557ff704cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:42.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.011+0000 7fc5b45b6740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557ff704d4f0 0x557ff705d610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:42.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.011+0000 7fc5b45b6740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x557ff701b0b0 con 0x557ff704af60 2024-05-28T06:05:42.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.011+0000 7fc5b45b6740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x557ff6f3e570 con 0x557ff704d4f0 2024-05-28T06:05:42.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.011+0000 7fc5b45b6740 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x557ff701b390 con 0x557ff70673e0 2024-05-28T06:05:42.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.011+0000 7fc5b45b6740 10 client.0 _ll_register_callbacks cb 0x557ff6f0cfc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-28T06:05:42.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.012+0000 7fc5aaa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557ff704af60 0x557ff7046770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:42.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.012+0000 7fc5aaa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557ff704af60 0x557ff7046770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:42226/0 (socket says 172.21.15.96:42226) 2024-05-28T06:05:42.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.012+0000 7fc5aaa00640 1 -- 172.21.15.96:0/556139404 learned_addr learned my addr 172.21.15.96:0/556139404 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:42.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.012+0000 7fc5aa000640 1 --2- 172.21.15.96:0/556139404 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x557ff70673e0 0x557ff704cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:42.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.012+0000 7fc5ab400640 1 --2- 172.21.15.96:0/556139404 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557ff704d4f0 0x557ff705d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:42.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.012+0000 7fc5aaa00640 1 -- 172.21.15.96:0/556139404 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x557ff70673e0 msgr2=0x557ff704cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:42.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.012+0000 7fc5aaa00640 1 --2- 172.21.15.96:0/556139404 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x557ff70673e0 0x557ff704cfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:42.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.012+0000 7fc5aaa00640 1 -- 172.21.15.96:0/556139404 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557ff704d4f0 msgr2=0x557ff705d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:42.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.012+0000 7fc5aaa00640 1 --2- 172.21.15.96:0/556139404 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557ff704d4f0 0x557ff705d610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:42.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.012+0000 7fc5aaa00640 1 -- 172.21.15.96:0/556139404 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557ff6d616b0 con 0x557ff704af60 2024-05-28T06:05:42.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.012+0000 7fc5aaa00640 1 --2- 172.21.15.96:0/556139404 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557ff704af60 0x557ff7046770 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fc59c058d30 tx=0x7fc59c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b73eb81e105b5bd3 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:42.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.012+0000 7fc58be00640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-05-28T06:05:42.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.012+0000 7fc58be00640 1 -- 172.21.15.96:0/556139404 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc59c05c040 con 0x557ff704af60 2024-05-28T06:05:42.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.012+0000 7fc58be00640 1 -- 172.21.15.96:0/556139404 <== mon.0 v2:172.21.15.96:3300/0 2 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7fc59c078340 con 0x557ff704af60 2024-05-28T06:05:42.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.012+0000 7fc5b45b6740 1 -- 172.21.15.96:0/556139404 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc59c0570a0 con 0x557ff704af60 2024-05-28T06:05:42.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.013+0000 7fc58be00640 1 -- 172.21.15.96:0/556139404 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=15) ==== 24+0+0 (secure 0 0 0) 0x7fc59c07e050 con 0x557ff704af60 2024-05-28T06:05:42.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.013+0000 7fc5b45b6740 10 client.5451 fetch_fsmap learned FSMap version 15 2024-05-28T06:05:42.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.013+0000 7fc5b45b6740 1 -- 172.21.15.96:0/556139404 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=15,monmap=2+}) -- 0x557ff705f830 con 0x557ff704af60 2024-05-28T06:05:42.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.013+0000 7fc58be00640 1 -- 172.21.15.96:0/556139404 <== mon.0 v2:172.21.15.96:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc59c0570a0 con 0x557ff704af60 2024-05-28T06:05:42.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.013+0000 7fc58be00640 1 -- 172.21.15.96:0/556139404 <== mon.0 v2:172.21.15.96:3300/0 5 ==== fsmap(e 15) ==== 3494+0+0 (secure 0 0 0) 0x7fc59c081020 con 0x557ff704af60 2024-05-28T06:05:42.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.013+0000 7fc5b45b6740 10 client.5451 fetch_fsmap finished waiting for FSMap version 15 2024-05-28T06:05:42.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.013+0000 7fc5b45b6740 10 client.5451 resolve_mds: resolved name 'd' to daemon mds.d 2024-05-28T06:05:42.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.013+0000 7fc5b45b6740 20 client.5451 populate_metadata read hostname 'smithi096' 2024-05-28T06:05:42.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.013+0000 7fc5b45b6740 1 --2- 172.21.15.96:0/556139404 >> [v2:172.21.15.123:6835/458334664,v1:172.21.15.123:6837/458334664] conn(0x557ff7044690 0x557ff7044a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:42.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.013+0000 7fc5b45b6740 4 client.5451 mds_command: new command op to 5238 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-28T06:05:42.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.013+0000 7fc5b45b6740 1 -- 172.21.15.96:0/556139404 --> [v2:172.21.15.123:6835/458334664,v1:172.21.15.123:6837/458334664] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557ff6f92530 con 0x557ff7044690 2024-05-28T06:05:42.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.014+0000 7fc5aa000640 1 --2- 172.21.15.96:0/556139404 >> [v2:172.21.15.123:6835/458334664,v1:172.21.15.123:6837/458334664] conn(0x557ff7044690 0x557ff7044a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:42.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.015+0000 7fc5aa000640 1 --2- 172.21.15.96:0/556139404 >> [v2:172.21.15.123:6835/458334664,v1:172.21.15.123:6837/458334664] conn(0x557ff7044690 0x557ff7044a70 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2ab36525a49e1c70 server_cookie=9aa87124e6990de1 in_seq=0 out_seq=0 2024-05-28T06:05:42.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.015+0000 7fc58be00640 10 client.5451 ms_handle_connect on v2:172.21.15.123:6835/458334664 2024-05-28T06:05:42.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.016+0000 7fc58be00640 1 -- 172.21.15.96:0/556139404 <== mds.0 v2:172.21.15.123:6835/458334664 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557ff6f92530 con 0x557ff7044690 2024-05-28T06:05:42.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.016+0000 7fc58be00640 10 client.5451 handle_command_reply: tid=0 2024-05-28T06:05:42.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.016+0000 7fc5b45b6740 1 client.5451 shutdown 2024-05-28T06:05:42.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.016+0000 7fc5b45b6740 1 -- 172.21.15.96:0/556139404 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557ff704af60 msgr2=0x557ff7046770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:42.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.016+0000 7fc5b45b6740 1 --2- 172.21.15.96:0/556139404 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557ff704af60 0x557ff7046770 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fc59c058d30 tx=0x7fc59c000bc0 comp rx=0 tx=0).stop 2024-05-28T06:05:42.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.016+0000 7fc5b45b6740 1 -- 172.21.15.96:0/556139404 shutdown_connections 2024-05-28T06:05:42.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.016+0000 7fc5b45b6740 1 --2- 172.21.15.96:0/556139404 >> [v2:172.21.15.123:6835/458334664,v1:172.21.15.123:6837/458334664] conn(0x557ff7044690 0x557ff7044a70 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:42.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.016+0000 7fc5b45b6740 1 --2- 172.21.15.96:0/556139404 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557ff704d4f0 0x557ff705d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:42.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.016+0000 7fc5b45b6740 1 --2- 172.21.15.96:0/556139404 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x557ff70673e0 0x557ff704cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:42.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.016+0000 7fc5b45b6740 1 --2- 172.21.15.96:0/556139404 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557ff704af60 0x557ff7046770 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:42.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.016+0000 7fc58be00640 0 client.5451 ms_handle_reset on v2:172.21.15.123:6835/458334664 2024-05-28T06:05:42.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.016+0000 7fc5b45b6740 1 -- 172.21.15.96:0/556139404 >> 172.21.15.96:0/556139404 conn(0x557ff70472a0 msgr2=0x557ff7020c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:42.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.017+0000 7fc5b45b6740 1 -- 172.21.15.96:0/556139404 shutdown_connections 2024-05-28T06:05:42.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.017+0000 7fc5b45b6740 1 -- 172.21.15.96:0/556139404 wait complete. 2024-05-28T06:05:42.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.017+0000 7fc5b45b6740 20 client.5451 trim_cache size 0 max 16384 2024-05-28T06:05:42.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.036+0000 7fc5b45b6740 1 Processor -- start 2024-05-28T06:05:42.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.036+0000 7fc5b45b6740 1 -- start start 2024-05-28T06:05:42.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.037+0000 7fc5b45b6740 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x557ff704af60 0x557ff701bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:42.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.037+0000 7fc5b45b6740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557ff704d4f0 0x557ff7016800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:42.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.037+0000 7fc5b45b6740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557ff70673e0 0x557ff7046b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:42.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.037+0000 7fc5b45b6740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc5ac1fff60 con 0x557ff704d4f0 2024-05-28T06:05:42.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.037+0000 7fc5b45b6740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc5ac1f4b70 con 0x557ff70673e0 2024-05-28T06:05:42.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.037+0000 7fc5b45b6740 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fc5ac1f51c0 con 0x557ff704af60 2024-05-28T06:05:42.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.038+0000 7fc5aa000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557ff704d4f0 0x557ff7016800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:42.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.038+0000 7fc5aa000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557ff704d4f0 0x557ff7016800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:42232/0 (socket says 172.21.15.96:42232) 2024-05-28T06:05:42.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.038+0000 7fc5aa000640 1 -- 172.21.15.96:0/3794060607 learned_addr learned my addr 172.21.15.96:0/3794060607 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:42.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.038+0000 7fc5aaa00640 1 --2- 172.21.15.96:0/3794060607 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x557ff704af60 0x557ff701bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:42.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.038+0000 7fc5ab400640 1 --2- 172.21.15.96:0/3794060607 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557ff70673e0 0x557ff7046b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:42.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.038+0000 7fc5aa000640 1 -- 172.21.15.96:0/3794060607 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x557ff704af60 msgr2=0x557ff701bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:42.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.038+0000 7fc5aa000640 1 --2- 172.21.15.96:0/3794060607 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x557ff704af60 0x557ff701bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:42.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.038+0000 7fc5aa000640 1 -- 172.21.15.96:0/3794060607 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557ff70673e0 msgr2=0x557ff7046b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:42.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.038+0000 7fc5aa000640 1 --2- 172.21.15.96:0/3794060607 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557ff70673e0 0x557ff7046b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:42.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.038+0000 7fc5aa000640 1 -- 172.21.15.96:0/3794060607 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557ff704d320 con 0x557ff704d4f0 2024-05-28T06:05:42.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.038+0000 7fc5aa000640 1 --2- 172.21.15.96:0/3794060607 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557ff704d4f0 0x557ff7016800 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fc598035040 tx=0x7fc59802c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e8a5695de4d9367 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:42.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.039+0000 7fc596a00640 1 -- 172.21.15.96:0/3794060607 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc598037040 con 0x557ff704d4f0 2024-05-28T06:05:42.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.039+0000 7fc596a00640 1 -- 172.21.15.96:0/3794060607 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc59804b0f0 con 0x557ff704d4f0 2024-05-28T06:05:42.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.039+0000 7fc596a00640 1 -- 172.21.15.96:0/3794060607 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc59804b4a0 con 0x557ff704d4f0 2024-05-28T06:05:42.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.039+0000 7fc5b45b6740 1 -- 172.21.15.96:0/3794060607 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557ff704d4f0 msgr2=0x557ff7016800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:42.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.039+0000 7fc5b45b6740 1 --2- 172.21.15.96:0/3794060607 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557ff704d4f0 0x557ff7016800 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fc598035040 tx=0x7fc59802c000 comp rx=0 tx=0).stop 2024-05-28T06:05:42.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.039+0000 7fc5b45b6740 1 -- 172.21.15.96:0/3794060607 shutdown_connections 2024-05-28T06:05:42.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.039+0000 7fc5b45b6740 1 --2- 172.21.15.96:0/3794060607 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557ff70673e0 0x557ff7046b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:42.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.039+0000 7fc5b45b6740 1 --2- 172.21.15.96:0/3794060607 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557ff704d4f0 0x557ff7016800 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:42.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.039+0000 7fc5b45b6740 1 --2- 172.21.15.96:0/3794060607 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x557ff704af60 0x557ff701bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:42.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.039+0000 7fc5b45b6740 1 -- 172.21.15.96:0/3794060607 >> 172.21.15.96:0/3794060607 conn(0x557ff70472a0 msgr2=0x557ff701b2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:42.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.040+0000 7fc5b45b6740 1 -- 172.21.15.96:0/3794060607 shutdown_connections 2024-05-28T06:05:42.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.040+0000 7fc5b45b6740 1 -- 172.21.15.96:0/3794060607 wait complete. 2024-05-28T06:05:42.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.040+0000 7fc5b45b6740 1 -- start start 2024-05-28T06:05:42.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.040+0000 7fc5b45b6740 1 Processor -- start 2024-05-28T06:05:42.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.041+0000 7fc5b45b6740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557ff704af60 0x557ff70df0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:42.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.041+0000 7fc5b45b6740 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x557ff704d4f0 0x557ff705f210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:42.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.041+0000 7fc5b45b6740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557ff70673e0 0x557ff701bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:42.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.041+0000 7fc5b45b6740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x557ff701b0b0 con 0x557ff70673e0 2024-05-28T06:05:42.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.041+0000 7fc5b45b6740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x557ff6f3e570 con 0x557ff704af60 2024-05-28T06:05:42.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.041+0000 7fc5b45b6740 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x557ff705cb30 con 0x557ff704d4f0 2024-05-28T06:05:42.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.041+0000 7fc5b45b6740 10 client.0 _ll_register_callbacks cb 0x7fc59c057250 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-28T06:05:42.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.042+0000 7fc5ab400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557ff70673e0 0x557ff701bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:42.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.042+0000 7fc5ab400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557ff70673e0 0x557ff701bdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:42242/0 (socket says 172.21.15.96:42242) 2024-05-28T06:05:42.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.042+0000 7fc5ab400640 1 -- 172.21.15.96:0/622841532 learned_addr learned my addr 172.21.15.96:0/622841532 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:42.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.042+0000 7fc5aa000640 1 --2- 172.21.15.96:0/622841532 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x557ff704d4f0 0x557ff705f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:42.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.042+0000 7fc5ab400640 1 -- 172.21.15.96:0/622841532 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x557ff704d4f0 msgr2=0x557ff705f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:42.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.042+0000 7fc5ab400640 1 --2- 172.21.15.96:0/622841532 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x557ff704d4f0 0x557ff705f210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:42.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.042+0000 7fc5aaa00640 1 --2- 172.21.15.96:0/622841532 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557ff704af60 0x557ff70df0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:42.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.042+0000 7fc5ab400640 1 -- 172.21.15.96:0/622841532 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557ff704af60 msgr2=0x557ff70df0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:42.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.042+0000 7fc5ab400640 1 --2- 172.21.15.96:0/622841532 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557ff704af60 0x557ff70df0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:42.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.042+0000 7fc5ab400640 1 -- 172.21.15.96:0/622841532 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557ff705e250 con 0x557ff70673e0 2024-05-28T06:05:42.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.042+0000 7fc5aa000640 1 --2- 172.21.15.96:0/622841532 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x557ff704d4f0 0x557ff705f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:05:42.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.042+0000 7fc5aaa00640 1 --2- 172.21.15.96:0/622841532 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557ff704af60 0x557ff70df0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:05:42.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.042+0000 7fc5ab400640 1 --2- 172.21.15.96:0/622841532 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557ff70673e0 0x557ff701bdc0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fc5a40137f0 tx=0x7fc5a4013cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5f87d2b01e49b66 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:42.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.042+0000 7fc58be00640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-05-28T06:05:42.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.043+0000 7fc58be00640 1 -- 172.21.15.96:0/622841532 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5a401d020 con 0x557ff70673e0 2024-05-28T06:05:42.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.043+0000 7fc58be00640 1 -- 172.21.15.96:0/622841532 <== mon.0 v2:172.21.15.96:3300/0 2 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7fc5a4007a40 con 0x557ff70673e0 2024-05-28T06:05:42.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.043+0000 7fc5b45b6740 1 -- 172.21.15.96:0/622841532 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc59804b0f0 con 0x557ff70673e0 2024-05-28T06:05:42.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.043+0000 7fc58be00640 1 -- 172.21.15.96:0/622841532 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=15) ==== 24+0+0 (secure 0 0 0) 0x7fc5a4027050 con 0x557ff70673e0 2024-05-28T06:05:42.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.043+0000 7fc5b45b6740 10 client.5457 fetch_fsmap learned FSMap version 15 2024-05-28T06:05:42.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.043+0000 7fc5b45b6740 1 -- 172.21.15.96:0/622841532 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=15,monmap=2+}) -- 0x557ff704e650 con 0x557ff70673e0 2024-05-28T06:05:42.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.043+0000 7fc58be00640 1 -- 172.21.15.96:0/622841532 <== mon.0 v2:172.21.15.96:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc59804b0f0 con 0x557ff70673e0 2024-05-28T06:05:42.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.043+0000 7fc58be00640 1 -- 172.21.15.96:0/622841532 <== mon.0 v2:172.21.15.96:3300/0 5 ==== fsmap(e 15) ==== 3494+0+0 (secure 0 0 0) 0x7fc5a402a020 con 0x557ff70673e0 2024-05-28T06:05:42.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.043+0000 7fc5b45b6740 10 client.5457 fetch_fsmap finished waiting for FSMap version 15 2024-05-28T06:05:42.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.043+0000 7fc5b45b6740 10 client.5457 resolve_mds: resolved name 'd' to daemon mds.d 2024-05-28T06:05:42.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.043+0000 7fc5b45b6740 20 client.5457 populate_metadata read hostname 'smithi096' 2024-05-28T06:05:42.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.043+0000 7fc5b45b6740 1 --2- 172.21.15.96:0/622841532 >> [v2:172.21.15.123:6835/458334664,v1:172.21.15.123:6837/458334664] conn(0x557ff70661a0 0x557ff705b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:42.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.043+0000 7fc5b45b6740 4 client.5457 mds_command: new command op to 5238 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-28T06:05:42.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.043+0000 7fc5b45b6740 1 -- 172.21.15.96:0/622841532 --> [v2:172.21.15.123:6835/458334664,v1:172.21.15.123:6837/458334664] -- command(tid 0: {"prefix": "status"}) -- 0x557ff6f8c3a0 con 0x557ff70661a0 2024-05-28T06:05:42.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.044+0000 7fc5aaa00640 1 --2- 172.21.15.96:0/622841532 >> [v2:172.21.15.123:6835/458334664,v1:172.21.15.123:6837/458334664] conn(0x557ff70661a0 0x557ff705b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:42.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.045+0000 7fc5aaa00640 1 --2- 172.21.15.96:0/622841532 >> [v2:172.21.15.123:6835/458334664,v1:172.21.15.123:6837/458334664] conn(0x557ff70661a0 0x557ff705b3b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b856fd1dcce093 server_cookie=945c8dfe3f6caacb in_seq=0 out_seq=0 2024-05-28T06:05:42.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.045+0000 7fc58be00640 10 client.5457 ms_handle_connect on v2:172.21.15.123:6835/458334664 2024-05-28T06:05:42.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.046+0000 7fc58be00640 1 -- 172.21.15.96:0/622841532 <== mds.0 v2:172.21.15.123:6835/458334664 1 ==== command_reply(tid 0: 0 ) ==== 8+0+325 (crc 0 0 0) 0x557ff6f8c3a0 con 0x557ff70661a0 2024-05-28T06:05:42.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.046+0000 7fc58be00640 10 client.5457 handle_command_reply: tid=0 2024-05-28T06:05:42.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.046+0000 7fc5b45b6740 1 client.5457 shutdown 2024-05-28T06:05:42.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.046+0000 7fc5b45b6740 1 -- 172.21.15.96:0/622841532 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557ff70673e0 msgr2=0x557ff701bdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:42.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.046+0000 7fc5b45b6740 1 --2- 172.21.15.96:0/622841532 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557ff70673e0 0x557ff701bdc0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fc5a40137f0 tx=0x7fc5a4013cb0 comp rx=0 tx=0).stop 2024-05-28T06:05:42.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.046+0000 7fc5b45b6740 1 -- 172.21.15.96:0/622841532 shutdown_connections 2024-05-28T06:05:42.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.046+0000 7fc5b45b6740 1 --2- 172.21.15.96:0/622841532 >> [v2:172.21.15.123:6835/458334664,v1:172.21.15.123:6837/458334664] conn(0x557ff70661a0 0x557ff705b3b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:42.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.046+0000 7fc5b45b6740 1 --2- 172.21.15.96:0/622841532 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557ff70673e0 0x557ff701bdc0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:42.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.046+0000 7fc5b45b6740 1 --2- 172.21.15.96:0/622841532 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x557ff704d4f0 0x557ff705f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:42.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.046+0000 7fc5b45b6740 1 --2- 172.21.15.96:0/622841532 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557ff704af60 0x557ff70df0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:42.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.046+0000 7fc58be00640 0 client.5457 ms_handle_reset on v2:172.21.15.123:6835/458334664 2024-05-28T06:05:42.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.046+0000 7fc5b45b6740 1 -- 172.21.15.96:0/622841532 >> 172.21.15.96:0/622841532 conn(0x557ff70472a0 msgr2=0x557ff7016650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:42.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.046+0000 7fc5b45b6740 1 -- 172.21.15.96:0/622841532 shutdown_connections 2024-05-28T06:05:42.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.046+0000 7fc5b45b6740 1 -- 172.21.15.96:0/622841532 wait complete. 2024-05-28T06:05:42.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.046+0000 7fc5b45b6740 20 client.5457 trim_cache size 0 max 16384 2024-05-28T06:05:42.051 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:05:42.051 INFO:teuthology.orchestra.run.smithi096.stdout: "cluster_fsid": "e4a831b5-e6d1-47a7-96cb-bf85154ee0d2", 2024-05-28T06:05:42.051 INFO:teuthology.orchestra.run.smithi096.stdout: "whoami": 0, 2024-05-28T06:05:42.051 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 5238, 2024-05-28T06:05:42.051 INFO:teuthology.orchestra.run.smithi096.stdout: "want_state": "up:active", 2024-05-28T06:05:42.052 INFO:teuthology.orchestra.run.smithi096.stdout: "state": "up:active", 2024-05-28T06:05:42.052 INFO:teuthology.orchestra.run.smithi096.stdout: "fs_name": "cephfs", 2024-05-28T06:05:42.052 INFO:teuthology.orchestra.run.smithi096.stdout: "rank_uptime": 11.584917378, 2024-05-28T06:05:42.052 INFO:teuthology.orchestra.run.smithi096.stdout: "mdsmap_epoch": 15, 2024-05-28T06:05:42.052 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap_epoch": 33, 2024-05-28T06:05:42.052 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap_epoch_barrier": 33, 2024-05-28T06:05:42.052 INFO:teuthology.orchestra.run.smithi096.stdout: "uptime": 20.448865777000002 2024-05-28T06:05:42.052 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:05:42.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.047+0000 7fc5b1800640 1 -- 172.21.15.96:0/3461294253 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc58c06c8f0 msgr2=0x7fc58c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:42.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.047+0000 7fc5b1800640 1 --2- 172.21.15.96:0/3461294253 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc58c06c8f0 0x7fc58c06edb0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fc598034000 tx=0x7fc59802e000 comp rx=0 tx=0).stop 2024-05-28T06:05:42.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.047+0000 7fc5b1800640 1 -- 172.21.15.96:0/3461294253 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc5ac06f860 msgr2=0x7fc5ac0999f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:42.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.047+0000 7fc5b1800640 1 --2- 172.21.15.96:0/3461294253 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc5ac06f860 0x7fc5ac0999f0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fc59c00c430 tx=0x7fc59c00c8f0 comp rx=0 tx=0).stop 2024-05-28T06:05:42.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.047+0000 7fc5b1800640 1 -- 172.21.15.96:0/3461294253 shutdown_connections 2024-05-28T06:05:42.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.047+0000 7fc5b1800640 1 --2- 172.21.15.96:0/3461294253 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc58c06c8f0 0x7fc58c06edb0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:42.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.047+0000 7fc5b1800640 1 --2- 172.21.15.96:0/3461294253 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5ac0d04a0 0x7fc5ac0cc7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:42.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.047+0000 7fc5b1800640 1 --2- 172.21.15.96:0/3461294253 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc5ac0ad130 0x7fc5ac1f5c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:42.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.047+0000 7fc5b1800640 1 --2- 172.21.15.96:0/3461294253 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc5ac06f860 0x7fc5ac0999f0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:42.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.047+0000 7fc5b1800640 1 -- 172.21.15.96:0/3461294253 >> 172.21.15.96:0/3461294253 conn(0x7fc5ac06bae0 msgr2=0x7fc5ac0b2cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:42.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.047+0000 7fc5b1800640 1 -- 172.21.15.96:0/3461294253 shutdown_connections 2024-05-28T06:05:42.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.047+0000 7fc5b1800640 1 -- 172.21.15.96:0/3461294253 wait complete. 2024-05-28T06:05:42.065 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-05-28T06:05:42.243 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.243+0000 7f26f9200640 1 Processor -- start 2024-05-28T06:05:42.244 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.243+0000 7f26f9200640 1 -- start start 2024-05-28T06:05:42.245 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.244+0000 7f26f9200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f26f406f860 0x7f26f40b3150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:42.245 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.244+0000 7f26f9200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f26f40b3690 0x7f26f40d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:42.245 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.244+0000 7f26f9200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26f406ed10 con 0x7f26f40d09e0 2024-05-28T06:05:42.245 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.244+0000 7f26f9200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f26f40d0e50 con 0x7f26f406f860 2024-05-28T06:05:42.245 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.244+0000 7f26f9200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f26f415cb00 con 0x7f26f40b3690 2024-05-28T06:05:42.245 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.245+0000 7f26f1e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f26f406f860 0x7f26f40b3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:42.245 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.245+0000 7f26f2800640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f26f40d09e0 0x7f26f4160210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:59706/0 (socket says 172.21.15.96:59706) 2024-05-28T06:05:42.245 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.245+0000 7f26f2800640 1 -- 172.21.15.96:0/3685431246 learned_addr learned my addr 172.21.15.96:0/3685431246 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:05:42.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.245+0000 7f26f1400640 1 --2- 172.21.15.96:0/3685431246 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f26f40b3690 0x7f26f40d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:42.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.245+0000 7f26f1e00640 1 -- 172.21.15.96:0/3685431246 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f26f40b3690 msgr2=0x7f26f40d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:42.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.245+0000 7f26f1e00640 1 --2- 172.21.15.96:0/3685431246 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f26f40b3690 0x7f26f40d04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:42.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.245+0000 7f26f1e00640 1 -- 172.21.15.96:0/3685431246 >> v1:172.21.15.96:6789/0 conn(0x7f26f40d09e0 legacy=0x7f26f4160210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:42.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.245+0000 7f26f0a00640 1 -- 172.21.15.96:0/3685431246 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 807662921 0 0) 0x7f26ec001110 con 0x7f26f40d09e0 2024-05-28T06:05:42.247 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.245+0000 7f26f1e00640 1 -- 172.21.15.96:0/3685431246 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26f40b3cc0 con 0x7f26f406f860 2024-05-28T06:05:42.247 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.246+0000 7f26f1e00640 1 --2- 172.21.15.96:0/3685431246 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f26f406f860 0x7f26f40b3150 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f26e4009d90 tx=0x7f26e402a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77dfca5a2b401ad server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:42.247 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.247+0000 7f26f0a00640 1 -- 172.21.15.96:0/3685431246 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26e4007740 con 0x7f26f406f860 2024-05-28T06:05:42.247 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.247+0000 7f26f0a00640 1 -- 172.21.15.96:0/3685431246 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26e40078e0 con 0x7f26f406f860 2024-05-28T06:05:42.247 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.247+0000 7f26f0a00640 1 -- 172.21.15.96:0/3685431246 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26e4007cb0 con 0x7f26f406f860 2024-05-28T06:05:42.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.247+0000 7f26f9200640 1 -- 172.21.15.96:0/3685431246 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f26f406f860 msgr2=0x7f26f40b3150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:42.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.247+0000 7f26f9200640 1 --2- 172.21.15.96:0/3685431246 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f26f406f860 0x7f26f40b3150 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f26e4009d90 tx=0x7f26e402a660 comp rx=0 tx=0).stop 2024-05-28T06:05:42.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.247+0000 7f26f9200640 1 -- 172.21.15.96:0/3685431246 shutdown_connections 2024-05-28T06:05:42.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.247+0000 7f26f9200640 1 --2- 172.21.15.96:0/3685431246 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f26f40b3690 0x7f26f40d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:42.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.247+0000 7f26f9200640 1 --2- 172.21.15.96:0/3685431246 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f26f406f860 0x7f26f40b3150 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:42.249 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.247+0000 7f26f9200640 1 -- 172.21.15.96:0/3685431246 >> 172.21.15.96:0/3685431246 conn(0x7f26f406bae0 msgr2=0x7f26f406bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:42.249 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.248+0000 7f26f9200640 1 -- 172.21.15.96:0/3685431246 shutdown_connections 2024-05-28T06:05:42.249 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.248+0000 7f26f9200640 1 -- 172.21.15.96:0/3685431246 wait complete. 2024-05-28T06:05:42.249 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.249+0000 7f26f9200640 1 Processor -- start 2024-05-28T06:05:42.249 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.249+0000 7f26f9200640 1 -- start start 2024-05-28T06:05:42.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.250+0000 7f26f9200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f26f406f860 0x7f26f40c3a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:42.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.250+0000 7f26f9200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26f40b3690 0x7f26f41f7b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:42.252 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.250+0000 7f26f9200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f26f40d09e0 0x7f26f40c9760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:42.252 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.250+0000 7f26f9200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f26f40b4560 con 0x7f26f40d09e0 2024-05-28T06:05:42.252 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.250+0000 7f26f9200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f26f40b4280 con 0x7f26f40b3690 2024-05-28T06:05:42.252 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.250+0000 7f26f9200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f26f40b4840 con 0x7f26f406f860 2024-05-28T06:05:42.252 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.251+0000 7f26f2800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f26f40d09e0 0x7f26f40c9760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:42.252 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.251+0000 7f26f2800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f26f40d09e0 0x7f26f40c9760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:42260/0 (socket says 172.21.15.96:42260) 2024-05-28T06:05:42.252 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.251+0000 7f26f2800640 1 -- 172.21.15.96:0/83862405 learned_addr learned my addr 172.21.15.96:0/83862405 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:42.252 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.251+0000 7f26f1400640 1 --2- 172.21.15.96:0/83862405 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26f40b3690 0x7f26f41f7b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:42.252 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.251+0000 7f26f1e00640 1 --2- 172.21.15.96:0/83862405 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f26f406f860 0x7f26f40c3a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:42.252 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.251+0000 7f26f2800640 1 -- 172.21.15.96:0/83862405 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f26f406f860 msgr2=0x7f26f40c3a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:42.252 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.251+0000 7f26f2800640 1 --2- 172.21.15.96:0/83862405 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f26f406f860 0x7f26f40c3a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:42.253 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.251+0000 7f26f2800640 1 -- 172.21.15.96:0/83862405 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26f40b3690 msgr2=0x7f26f41f7b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:42.253 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.251+0000 7f26f2800640 1 --2- 172.21.15.96:0/83862405 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26f40b3690 0x7f26f41f7b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:42.253 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.251+0000 7f26f2800640 1 -- 172.21.15.96:0/83862405 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26f40c9ff0 con 0x7f26f40d09e0 2024-05-28T06:05:42.253 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.251+0000 7f26f1400640 1 --2- 172.21.15.96:0/83862405 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26f40b3690 0x7f26f41f7b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:05:42.253 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.251+0000 7f26f2800640 1 --2- 172.21.15.96:0/83862405 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f26f40d09e0 0x7f26f40c9760 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f26ec00c780 tx=0x7f26ec00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:42.254 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.252+0000 7f26ea000640 1 -- 172.21.15.96:0/83862405 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26ec018070 con 0x7f26f40d09e0 2024-05-28T06:05:42.254 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.252+0000 7f26ea000640 1 -- 172.21.15.96:0/83862405 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26ec0070d0 con 0x7f26f40d09e0 2024-05-28T06:05:42.255 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.252+0000 7f26f9200640 1 -- 172.21.15.96:0/83862405 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26f40ce080 con 0x7f26f40d09e0 2024-05-28T06:05:42.255 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.252+0000 7f26ea000640 1 -- 172.21.15.96:0/83862405 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26ec0074c0 con 0x7f26f40d09e0 2024-05-28T06:05:42.255 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.252+0000 7f26f9200640 1 -- 172.21.15.96:0/83862405 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26f40c9ca0 con 0x7f26f40d09e0 2024-05-28T06:05:42.255 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.253+0000 7f26f9200640 1 -- 172.21.15.96:0/83862405 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26f40689e0 con 0x7f26f40d09e0 2024-05-28T06:05:42.255 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.254+0000 7f26ea000640 1 -- 172.21.15.96:0/83862405 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f26ec01b070 con 0x7f26f40d09e0 2024-05-28T06:05:42.255 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.255+0000 7f26ea000640 1 --2- 172.21.15.96:0/83862405 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f26d006c8f0 0x7f26d006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:42.255 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.255+0000 7f26ea000640 1 -- 172.21.15.96:0/83862405 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f26ec08ae20 con 0x7f26f40d09e0 2024-05-28T06:05:42.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.256+0000 7f26f1e00640 1 --2- 172.21.15.96:0/83862405 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f26d006c8f0 0x7f26d006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:42.257 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.257+0000 7f26f1e00640 1 --2- 172.21.15.96:0/83862405 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f26d006c8f0 0x7f26d006edb0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f26e4034000 tx=0x7f26e4033040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:42.260 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.260+0000 7f26ea000640 1 -- 172.21.15.96:0/83862405 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f26f406ed10 con 0x7f26f40d09e0 2024-05-28T06:05:42.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.502+0000 7f26f9200640 1 -- 172.21.15.96:0/83862405 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f26f40ae4c0 con 0x7f26f40d09e0 2024-05-28T06:05:42.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.503+0000 7f26ea000640 1 -- 172.21.15.96:0/83862405 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v15) ==== 53+0+73 (secure 0 0 0) 0x7f26ec059240 con 0x7f26f40d09e0 2024-05-28T06:05:42.504 INFO:teuthology.orchestra.run.smithi096.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-05-28T06:05:42.506 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.506+0000 7f26f9200640 1 -- 172.21.15.96:0/83862405 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f26d006c8f0 msgr2=0x7f26d006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:42.506 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.506+0000 7f26f9200640 1 --2- 172.21.15.96:0/83862405 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f26d006c8f0 0x7f26d006edb0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f26e4034000 tx=0x7f26e4033040 comp rx=0 tx=0).stop 2024-05-28T06:05:42.507 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.506+0000 7f26f9200640 1 -- 172.21.15.96:0/83862405 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f26f40d09e0 msgr2=0x7f26f40c9760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:42.507 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.506+0000 7f26f9200640 1 --2- 172.21.15.96:0/83862405 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f26f40d09e0 0x7f26f40c9760 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f26ec00c780 tx=0x7f26ec00cc40 comp rx=0 tx=0).stop 2024-05-28T06:05:42.507 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.506+0000 7f26f9200640 1 -- 172.21.15.96:0/83862405 shutdown_connections 2024-05-28T06:05:42.507 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.506+0000 7f26f9200640 1 --2- 172.21.15.96:0/83862405 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f26d006c8f0 0x7f26d006edb0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:42.507 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.506+0000 7f26f9200640 1 --2- 172.21.15.96:0/83862405 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f26f40d09e0 0x7f26f40c9760 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:42.507 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.506+0000 7f26f9200640 1 --2- 172.21.15.96:0/83862405 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26f40b3690 0x7f26f41f7b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:42.507 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.506+0000 7f26f9200640 1 --2- 172.21.15.96:0/83862405 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f26f406f860 0x7f26f40c3a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:42.507 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.506+0000 7f26f9200640 1 -- 172.21.15.96:0/83862405 >> 172.21.15.96:0/83862405 conn(0x7f26f406bae0 msgr2=0x7f26f40b0d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:42.507 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.506+0000 7f26f9200640 1 -- 172.21.15.96:0/83862405 shutdown_connections 2024-05-28T06:05:42.508 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42.506+0000 7f26f9200640 1 -- 172.21.15.96:0/83862405 wait complete. 2024-05-28T06:05:42.524 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-05-28T06:05:42.524 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-28T06:05:42.524 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi096.front.sepia.ceph.com 2024-05-28T06:05:42.524 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-05-28T06:05:42.524 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-28T06:05:42.525 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-05-28T06:05:42.525 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-05-28T06:05:42.525 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-28T06:05:42.525 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-28T06:05:42.525 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:05:42.525 DEBUG:teuthology.orchestra.run.smithi096:> ip addr 2024-05-28T06:05:42.576 INFO:teuthology.orchestra.run.smithi096.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-28T06:05:42.576 INFO:teuthology.orchestra.run.smithi096.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-28T06:05:42.576 INFO:teuthology.orchestra.run.smithi096.stdout: inet 127.0.0.1/8 scope host lo 2024-05-28T06:05:42.576 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:05:42.576 INFO:teuthology.orchestra.run.smithi096.stdout: inet6 ::1/128 scope host 2024-05-28T06:05:42.576 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:05:42.577 INFO:teuthology.orchestra.run.smithi096.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-28T06:05:42.577 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether 0c:c4:7a:d8:d9:bc brd ff:ff:ff:ff:ff:ff 2024-05-28T06:05:42.577 INFO:teuthology.orchestra.run.smithi096.stdout: altname enp7s0f0 2024-05-28T06:05:42.577 INFO:teuthology.orchestra.run.smithi096.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-28T06:05:42.577 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether 0c:c4:7a:d8:d9:bd brd ff:ff:ff:ff:ff:ff 2024-05-28T06:05:42.577 INFO:teuthology.orchestra.run.smithi096.stdout: altname enp7s0f1 2024-05-28T06:05:42.577 INFO:teuthology.orchestra.run.smithi096.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-28T06:05:42.577 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether 0c:c4:7a:8f:cc:0c brd ff:ff:ff:ff:ff:ff 2024-05-28T06:05:42.577 INFO:teuthology.orchestra.run.smithi096.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-28T06:05:42.577 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether 0c:c4:7a:8f:cc:0d brd ff:ff:ff:ff:ff:ff 2024-05-28T06:05:42.577 INFO:teuthology.orchestra.run.smithi096.stdout: inet 172.21.15.96/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-05-28T06:05:42.577 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft 42494sec preferred_lft 42494sec 2024-05-28T06:05:42.577 INFO:teuthology.orchestra.run.smithi096.stdout: inet6 fe80::ec4:7aff:fe8f:cc0d/64 scope link 2024-05-28T06:05:42.577 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:05:42.578 INFO:teuthology.orchestra.run.smithi096.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-05-28T06:05:42.578 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether ae:d9:f5:b0:4f:f3 brd ff:ff:ff:ff:ff:ff 2024-05-28T06:05:42.578 INFO:teuthology.orchestra.run.smithi096.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-05-28T06:05:42.578 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:05:42.578 INFO:teuthology.orchestra.run.smithi096.stdout: inet6 fe80::604e:fcff:fe60:7892/64 scope link 2024-05-28T06:05:42.578 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:05:42.578 INFO:teuthology.orchestra.run.smithi096.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-28T06:05:42.578 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether ce:6f:19:93:44:f9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-28T06:05:42.578 INFO:teuthology.orchestra.run.smithi096.stdout: inet6 fe80::cc6f:19ff:fe93:44f9/64 scope link 2024-05-28T06:05:42.578 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:05:42.578 INFO:teuthology.orchestra.run.smithi096.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-28T06:05:42.578 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether ae:d9:f5:b0:4f:f3 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-28T06:05:42.578 INFO:teuthology.orchestra.run.smithi096.stdout: inet6 fe80::acd9:f5ff:feb0:4ff3/64 scope link 2024-05-28T06:05:42.579 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:05:42.579 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:05:42.579 DEBUG:teuthology.orchestra.run.smithi096:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-28T06:05:42.659 INFO:teuthology.orchestra.run.smithi096.stdout:1 2024-05-28T06:05:42.663 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:05:42.663 DEBUG:teuthology.orchestra.run.smithi096:> route 2024-05-28T06:05:42.718 INFO:teuthology.orchestra.run.smithi096.stdout:Kernel IP routing table 2024-05-28T06:05:42.718 INFO:teuthology.orchestra.run.smithi096.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-28T06:05:42.718 INFO:teuthology.orchestra.run.smithi096.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-05-28T06:05:42.718 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-05-28T06:05:42.718 INFO:teuthology.orchestra.run.smithi096.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-28T06:05:42.719 DEBUG:teuthology.orchestra.run.smithi096:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-28T06:05:42.719 DEBUG:teuthology.orchestra.run.smithi096:> set -e 2024-05-28T06:05:42.719 DEBUG:teuthology.orchestra.run.smithi096:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-05-28T06:05:42.719 DEBUG:teuthology.orchestra.run.smithi096:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-05-28T06:05:42.719 DEBUG:teuthology.orchestra.run.smithi096:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-05-28T06:05:42.719 DEBUG:teuthology.orchestra.run.smithi096:> ') 2024-05-28T06:05:42.819 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-28T06:05:42.919 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-28T06:05:42.930 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:05:42.930 DEBUG:teuthology.orchestra.run.smithi096:> ip netns list 2024-05-28T06:05:42.979 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-05-28T06:05:42.979 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-05-28T06:05:42.980 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-28T06:05:42.980 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:05:42.980 DEBUG:teuthology.orchestra.run.smithi096:> sudo ip link set brx.0 up 2024-05-28T06:05:43.064 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-05-28T06:05:43.065 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-28T06:05:43.065 DEBUG:teuthology.orchestra.run.smithi096:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-05-28T06:05:43.119 INFO:teuthology.orchestra.run.smithi096.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-05-28T06:05:43.120 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-28T06:05:43.120 DEBUG:teuthology.orchestra.run.smithi096:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-05-28T06:05:43.168 DEBUG:teuthology.orchestra.run.smithi096:> sudo modprobe fuse 2024-05-28T06:05:43.299 DEBUG:teuthology.orchestra.run.smithi096:> cat /proc/self/mounts | awk '{print $2}' 2024-05-28T06:05:43.355 INFO:teuthology.orchestra.run.smithi096.stdout:/proc 2024-05-28T06:05:43.355 INFO:teuthology.orchestra.run.smithi096.stdout:/sys 2024-05-28T06:05:43.355 INFO:teuthology.orchestra.run.smithi096.stdout:/dev 2024-05-28T06:05:43.355 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/security 2024-05-28T06:05:43.355 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/shm 2024-05-28T06:05:43.355 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/pts 2024-05-28T06:05:43.356 INFO:teuthology.orchestra.run.smithi096.stdout:/run 2024-05-28T06:05:43.356 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/cgroup 2024-05-28T06:05:43.356 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/pstore 2024-05-28T06:05:43.356 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/bpf 2024-05-28T06:05:43.356 INFO:teuthology.orchestra.run.smithi096.stdout:/ 2024-05-28T06:05:43.356 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/selinux 2024-05-28T06:05:43.356 INFO:teuthology.orchestra.run.smithi096.stdout:/proc/sys/fs/binfmt_misc 2024-05-28T06:05:43.356 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/mqueue 2024-05-28T06:05:43.356 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/debug 2024-05-28T06:05:43.356 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/hugepages 2024-05-28T06:05:43.356 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/tracing 2024-05-28T06:05:43.356 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/fuse/connections 2024-05-28T06:05:43.356 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/config 2024-05-28T06:05:43.357 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-sysctl.service 2024-05-28T06:05:43.357 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-28T06:05:43.357 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-28T06:05:43.357 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/nfs/rpc_pipefs 2024-05-28T06:05:43.357 INFO:teuthology.orchestra.run.smithi096.stdout:/run/user/1000 2024-05-28T06:05:43.357 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph 2024-05-28T06:05:43.357 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-0 2024-05-28T06:05:43.357 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-1 2024-05-28T06:05:43.357 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-2 2024-05-28T06:05:43.357 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-3 2024-05-28T06:05:43.357 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns 2024-05-28T06:05:43.357 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-28T06:05:43.357 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-28T06:05:43.357 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-28T06:05:43.358 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-28T06:05:43.358 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:05:43.359 DEBUG:teuthology.orchestra.run.smithi096:> ls /sys/fs/fuse/connections 2024-05-28T06:05:43.408 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-05-28T06:05:43.408 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-05-28T06:05:43.414 DEBUG:teuthology.orchestra.run.smithi096:> sudo modprobe fuse 2024-05-28T06:05:43.481 DEBUG:teuthology.orchestra.run.smithi096:> cat /proc/self/mounts | awk '{print $2}' 2024-05-28T06:05:43.524 INFO:teuthology.orchestra.run.smithi096.stdout:/proc 2024-05-28T06:05:43.525 INFO:teuthology.orchestra.run.smithi096.stdout:/sys 2024-05-28T06:05:43.525 INFO:teuthology.orchestra.run.smithi096.stdout:/dev 2024-05-28T06:05:43.525 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/security 2024-05-28T06:05:43.525 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/shm 2024-05-28T06:05:43.526 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/pts 2024-05-28T06:05:43.526 INFO:teuthology.orchestra.run.smithi096.stdout:/run 2024-05-28T06:05:43.526 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/cgroup 2024-05-28T06:05:43.526 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/pstore 2024-05-28T06:05:43.526 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/bpf 2024-05-28T06:05:43.527 INFO:teuthology.orchestra.run.smithi096.stdout:/ 2024-05-28T06:05:43.527 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/selinux 2024-05-28T06:05:43.527 INFO:teuthology.orchestra.run.smithi096.stdout:/proc/sys/fs/binfmt_misc 2024-05-28T06:05:43.527 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/mqueue 2024-05-28T06:05:43.527 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/debug 2024-05-28T06:05:43.527 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/hugepages 2024-05-28T06:05:43.527 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/tracing 2024-05-28T06:05:43.528 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/fuse/connections 2024-05-28T06:05:43.528 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/config 2024-05-28T06:05:43.528 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-sysctl.service 2024-05-28T06:05:43.528 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-28T06:05:43.528 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-28T06:05:43.528 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/nfs/rpc_pipefs 2024-05-28T06:05:43.529 INFO:teuthology.orchestra.run.smithi096.stdout:/run/user/1000 2024-05-28T06:05:43.529 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph 2024-05-28T06:05:43.529 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-0 2024-05-28T06:05:43.529 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-1 2024-05-28T06:05:43.529 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-2 2024-05-28T06:05:43.529 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-3 2024-05-28T06:05:43.529 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns 2024-05-28T06:05:43.530 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-28T06:05:43.530 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-28T06:05:43.530 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-28T06:05:43.530 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-28T06:05:43.531 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:05:43.532 DEBUG:teuthology.orchestra.run.smithi096:> ls /sys/fs/fuse/connections 2024-05-28T06:05:43.579 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi096.stderr:2024-05-28T06:05:43.578+0000 7fe975b79480 -1 init, newargv = 0x55bd0ec37b90 newargc=15 2024-05-28T06:05:43.579 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi096.stderr:ceph-fuse[51263]: starting ceph client 2024-05-28T06:05:43.589 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi096.stderr:ceph-fuse[51263]: starting fuse 2024-05-28T06:05:43.613 INFO:teuthology.orchestra.run.smithi096.stdout:40 2024-05-28T06:05:43.613 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40] 2024-05-28T06:05:43.613 DEBUG:teuthology.orchestra.run.smithi096:> sudo stdin-killer -- python3 -c ' 2024-05-28T06:05:43.613 DEBUG:teuthology.orchestra.run.smithi096:> import glob 2024-05-28T06:05:43.613 DEBUG:teuthology.orchestra.run.smithi096:> import re 2024-05-28T06:05:43.614 DEBUG:teuthology.orchestra.run.smithi096:> import os 2024-05-28T06:05:43.614 DEBUG:teuthology.orchestra.run.smithi096:> import subprocess 2024-05-28T06:05:43.614 DEBUG:teuthology.orchestra.run.smithi096:> 2024-05-28T06:05:43.614 DEBUG:teuthology.orchestra.run.smithi096:> def _find_admin_socket(client_name): 2024-05-28T06:05:43.614 DEBUG:teuthology.orchestra.run.smithi096:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-05-28T06:05:43.614 DEBUG:teuthology.orchestra.run.smithi096:> files = glob.glob(asok_path) 2024-05-28T06:05:43.614 DEBUG:teuthology.orchestra.run.smithi096:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-05-28T06:05:43.614 DEBUG:teuthology.orchestra.run.smithi096:> 2024-05-28T06:05:43.614 DEBUG:teuthology.orchestra.run.smithi096:> # Given a non-glob path, it better be there 2024-05-28T06:05:43.614 DEBUG:teuthology.orchestra.run.smithi096:> if "*" not in asok_path: 2024-05-28T06:05:43.614 DEBUG:teuthology.orchestra.run.smithi096:> assert(len(files) == 1) 2024-05-28T06:05:43.614 DEBUG:teuthology.orchestra.run.smithi096:> return files[0] 2024-05-28T06:05:43.614 DEBUG:teuthology.orchestra.run.smithi096:> 2024-05-28T06:05:43.614 DEBUG:teuthology.orchestra.run.smithi096:> for f in files: 2024-05-28T06:05:43.615 DEBUG:teuthology.orchestra.run.smithi096:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-05-28T06:05:43.615 DEBUG:teuthology.orchestra.run.smithi096:> if os.path.exists("/proc/{0}".format(pid)): 2024-05-28T06:05:43.615 DEBUG:teuthology.orchestra.run.smithi096:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-05-28T06:05:43.615 DEBUG:teuthology.orchestra.run.smithi096:> contents = proc_f.read() 2024-05-28T06:05:43.615 DEBUG:teuthology.orchestra.run.smithi096:> if mountpoint in contents: 2024-05-28T06:05:43.615 DEBUG:teuthology.orchestra.run.smithi096:> return f 2024-05-28T06:05:43.615 DEBUG:teuthology.orchestra.run.smithi096:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-05-28T06:05:43.615 DEBUG:teuthology.orchestra.run.smithi096:> 2024-05-28T06:05:43.615 DEBUG:teuthology.orchestra.run.smithi096:> print(_find_admin_socket("client.0")) 2024-05-28T06:05:43.615 DEBUG:teuthology.orchestra.run.smithi096:> ' 2024-05-28T06:05:43.804 INFO:teuthology.orchestra.run.smithi096.stdout:/var/run/ceph/ceph-client.0.51263.asok 2024-05-28T06:05:43.808 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-28T06:05:43.820 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.51263.asok 2024-05-28T06:05:43.820 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:05:43.820 DEBUG:teuthology.orchestra.run.smithi096:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.51263.asok status 2024-05-28T06:05:44.038 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:05:44.038 INFO:teuthology.orchestra.run.smithi096.stdout: "metadata": { 2024-05-28T06:05:44.039 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph_sha1": "9ed025fbf4406fb54980a0647506b15895b6d863", 2024-05-28T06:05:44.039 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph_version": "ceph version 19.0.0-3154-g9ed025fb (9ed025fbf4406fb54980a0647506b15895b6d863) squid (dev)", 2024-05-28T06:05:44.039 INFO:teuthology.orchestra.run.smithi096.stdout: "entity_id": "0", 2024-05-28T06:05:44.039 INFO:teuthology.orchestra.run.smithi096.stdout: "hostname": "smithi096", 2024-05-28T06:05:44.039 INFO:teuthology.orchestra.run.smithi096.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-05-28T06:05:44.039 INFO:teuthology.orchestra.run.smithi096.stdout: "pid": "51263", 2024-05-28T06:05:44.039 INFO:teuthology.orchestra.run.smithi096.stdout: "root": "/" 2024-05-28T06:05:44.039 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:05:44.039 INFO:teuthology.orchestra.run.smithi096.stdout: "dentry_count": 0, 2024-05-28T06:05:44.039 INFO:teuthology.orchestra.run.smithi096.stdout: "dentry_pinned_count": 0, 2024-05-28T06:05:44.040 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 5472, 2024-05-28T06:05:44.040 INFO:teuthology.orchestra.run.smithi096.stdout: "inst": { 2024-05-28T06:05:44.040 INFO:teuthology.orchestra.run.smithi096.stdout: "name": { 2024-05-28T06:05:44.040 INFO:teuthology.orchestra.run.smithi096.stdout: "type": "client", 2024-05-28T06:05:44.040 INFO:teuthology.orchestra.run.smithi096.stdout: "num": 5472 2024-05-28T06:05:44.040 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:05:44.040 INFO:teuthology.orchestra.run.smithi096.stdout: "addr": { 2024-05-28T06:05:44.040 INFO:teuthology.orchestra.run.smithi096.stdout: "type": "v1", 2024-05-28T06:05:44.040 INFO:teuthology.orchestra.run.smithi096.stdout: "addr": "192.168.0.1:0", 2024-05-28T06:05:44.040 INFO:teuthology.orchestra.run.smithi096.stdout: "nonce": 2456297838 2024-05-28T06:05:44.040 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:05:44.040 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:05:44.041 INFO:teuthology.orchestra.run.smithi096.stdout: "addr": { 2024-05-28T06:05:44.041 INFO:teuthology.orchestra.run.smithi096.stdout: "type": "v1", 2024-05-28T06:05:44.041 INFO:teuthology.orchestra.run.smithi096.stdout: "addr": "192.168.0.1:0", 2024-05-28T06:05:44.041 INFO:teuthology.orchestra.run.smithi096.stdout: "nonce": 2456297838 2024-05-28T06:05:44.041 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:05:44.041 INFO:teuthology.orchestra.run.smithi096.stdout: "inst_str": "client.5472 192.168.0.1:0/2456297838", 2024-05-28T06:05:44.041 INFO:teuthology.orchestra.run.smithi096.stdout: "addr_str": "192.168.0.1:0/2456297838", 2024-05-28T06:05:44.041 INFO:teuthology.orchestra.run.smithi096.stdout: "inode_count": 1, 2024-05-28T06:05:44.041 INFO:teuthology.orchestra.run.smithi096.stdout: "mds_epoch": 15, 2024-05-28T06:05:44.041 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_epoch": 33, 2024-05-28T06:05:44.041 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_epoch_barrier": 0, 2024-05-28T06:05:44.041 INFO:teuthology.orchestra.run.smithi096.stdout: "blocklisted": false, 2024-05-28T06:05:44.042 INFO:teuthology.orchestra.run.smithi096.stdout: "fs_name": "cephfs" 2024-05-28T06:05:44.042 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:05:44.053 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:05:44.053 DEBUG:teuthology.orchestra.run.smithi096:> stat --file-system '--printf=%T 2024-05-28T06:05:44.053 DEBUG:teuthology.orchestra.run.smithi096:> ' -- /home/ubuntu/cephtest/mnt.0 2024-05-28T06:05:44.106 INFO:teuthology.orchestra.run.smithi096.stdout:fuseblk 2024-05-28T06:05:44.106 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-05-28T06:05:44.107 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:05:44.107 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-05-28T06:05:44.189 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-05-28T06:05:44.391 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:44.390+0000 7f425a200640 1 Processor -- start 2024-05-28T06:05:44.391 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:44.391+0000 7f425a200640 1 -- start start 2024-05-28T06:05:44.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:44.392+0000 7f425a200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f425415b340 0x7f425406f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:44.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:44.392+0000 7f425a200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f425406fa90 con 0x7f425415b720 2024-05-28T06:05:44.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:44.392+0000 7f425a200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42540aeda0 con 0x7f425415c2c0 2024-05-28T06:05:44.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:44.392+0000 7f425a200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f42540af9b0 con 0x7f425415b340 2024-05-28T06:05:44.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:44.392+0000 7f4253400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f425415b340 0x7f425406f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:44.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:44.392+0000 7f4253400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f425415b340 0x7f425406f550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:42276/0 (socket says 172.21.15.96:42276) 2024-05-28T06:05:44.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:44.392+0000 7f4253400640 1 -- 172.21.15.96:0/959659901 learned_addr learned my addr 172.21.15.96:0/959659901 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:44.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:44.393+0000 7f4253400640 1 -- 172.21.15.96:0/959659901 >> v1:172.21.15.123:6789/0 conn(0x7f425415c2c0 legacy=0x7f42540b3300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:44.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:44.393+0000 7f4253400640 1 -- 172.21.15.96:0/959659901 >> v1:172.21.15.96:6789/0 conn(0x7f425415b720 legacy=0x7f425415bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:44.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:44.393+0000 7f4253400640 1 -- 172.21.15.96:0/959659901 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42540af080 con 0x7f425415b340 2024-05-28T06:05:44.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:44.394+0000 7f4253400640 1 --2- 172.21.15.96:0/959659901 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f425415b340 0x7f425406f550 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f424400d150 tx=0x7f424402c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d0ff7d7f2728215 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:44.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:44.394+0000 7f4252a00640 1 -- 172.21.15.96:0/959659901 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4244009600 con 0x7f425415b340 2024-05-28T06:05:44.395 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:44.394+0000 7f4252a00640 1 -- 172.21.15.96:0/959659901 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4244037070 con 0x7f425415b340 2024-05-28T06:05:44.395 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:44.394+0000 7f4252a00640 1 -- 172.21.15.96:0/959659901 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42440099c0 con 0x7f425415b340 2024-05-28T06:05:44.395 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:44.395+0000 7f425a200640 1 -- 172.21.15.96:0/959659901 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f425415b340 msgr2=0x7f425406f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:44.395 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:44.395+0000 7f425a200640 1 --2- 172.21.15.96:0/959659901 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f425415b340 0x7f425406f550 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f424400d150 tx=0x7f424402c970 comp rx=0 tx=0).stop 2024-05-28T06:05:44.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:44.395+0000 7f425a200640 1 -- 172.21.15.96:0/959659901 shutdown_connections 2024-05-28T06:05:44.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:44.395+0000 7f425a200640 1 --2- 172.21.15.96:0/959659901 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f425415b340 0x7f425406f550 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:44.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:44.395+0000 7f425a200640 1 -- 172.21.15.96:0/959659901 >> 172.21.15.96:0/959659901 conn(0x7f42540b1e50 msgr2=0x7f42540b2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:44.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:44.395+0000 7f425a200640 1 -- 172.21.15.96:0/959659901 shutdown_connections 2024-05-28T06:05:44.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:44.395+0000 7f425a200640 1 -- 172.21.15.96:0/959659901 wait complete. 2024-05-28T06:05:44.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:44.397+0000 7f425a200640 1 Processor -- start 2024-05-28T06:05:44.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:44.397+0000 7f425a200640 1 -- start start 2024-05-28T06:05:44.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:44.398+0000 7f425a200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f425415b340 0x7f42541560e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:44.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:44.398+0000 7f425a200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f425415b720 0x7f425415e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:44.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:44.398+0000 7f425a200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f425415c2c0 0x7f42540afc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:44.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:44.398+0000 7f425a200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f42540aea30 con 0x7f425415c2c0 2024-05-28T06:05:44.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:44.398+0000 7f425a200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f42540af640 con 0x7f425415b720 2024-05-28T06:05:44.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:44.398+0000 7f425a200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f425415c8c0 con 0x7f425415b340 2024-05-28T06:05:44.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:44.398+0000 7f4253e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f425415c2c0 0x7f42540afc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:44.400 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:44.398+0000 7f4253e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f425415c2c0 0x7f42540afc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:42284/0 (socket says 172.21.15.96:42284) 2024-05-28T06:05:44.400 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:44.398+0000 7f4253e00640 1 -- 172.21.15.96:0/914705034 learned_addr learned my addr 172.21.15.96:0/914705034 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:44.400 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:44.399+0000 7f4253400640 1 --2- 172.21.15.96:0/914705034 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f425415b340 0x7f42541560e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:44.400 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:44.399+0000 7f424aa00640 1 --2- 172.21.15.96:0/914705034 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f425415b720 0x7f425415e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:44.400 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:44.399+0000 7f4253e00640 1 -- 172.21.15.96:0/914705034 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f425415b340 msgr2=0x7f42541560e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:44.400 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:44.399+0000 7f4253e00640 1 --2- 172.21.15.96:0/914705034 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f425415b340 0x7f42541560e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:44.400 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:44.399+0000 7f4253e00640 1 -- 172.21.15.96:0/914705034 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f425415b720 msgr2=0x7f425415e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:44.400 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:44.399+0000 7f4253e00640 1 --2- 172.21.15.96:0/914705034 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f425415b720 0x7f425415e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:44.400 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:44.399+0000 7f4253e00640 1 -- 172.21.15.96:0/914705034 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4254152870 con 0x7f425415c2c0 2024-05-28T06:05:44.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:44.399+0000 7f4253e00640 1 --2- 172.21.15.96:0/914705034 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f425415c2c0 0x7f42540afc00 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f424c00c440 tx=0x7f424c00c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:44.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:44.399+0000 7f424be00640 1 -- 172.21.15.96:0/914705034 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f424c011790 con 0x7f425415c2c0 2024-05-28T06:05:44.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:44.400+0000 7f424be00640 1 -- 172.21.15.96:0/914705034 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f424c017070 con 0x7f425415c2c0 2024-05-28T06:05:44.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:44.400+0000 7f425a200640 1 -- 172.21.15.96:0/914705034 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4254156750 con 0x7f425415c2c0 2024-05-28T06:05:44.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:44.400+0000 7f424be00640 1 -- 172.21.15.96:0/914705034 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f424c011b70 con 0x7f425415c2c0 2024-05-28T06:05:44.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:44.400+0000 7f425a200640 1 -- 172.21.15.96:0/914705034 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4254152b30 con 0x7f425415c2c0 2024-05-28T06:05:44.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:44.401+0000 7f425a200640 1 -- 172.21.15.96:0/914705034 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4254152d30 con 0x7f425415c2c0 2024-05-28T06:05:44.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:44.402+0000 7f424be00640 1 -- 172.21.15.96:0/914705034 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f424c011d10 con 0x7f425415c2c0 2024-05-28T06:05:44.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:44.403+0000 7f424be00640 1 --2- 172.21.15.96:0/914705034 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f423006c8f0 0x7f423006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:44.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:44.404+0000 7f424be00640 1 -- 172.21.15.96:0/914705034 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f424c08b940 con 0x7f425415c2c0 2024-05-28T06:05:44.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:44.404+0000 7f4253400640 1 --2- 172.21.15.96:0/914705034 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f423006c8f0 0x7f423006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:44.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:44.405+0000 7f4253400640 1 --2- 172.21.15.96:0/914705034 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f423006c8f0 0x7f423006edb0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f4244036000 tx=0x7f424400b000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:44.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:44.408+0000 7f424be00640 1 -- 172.21.15.96:0/914705034 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4254152d30 con 0x7f425415c2c0 2024-05-28T06:05:44.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:44.651+0000 7f425a200640 1 -- 172.21.15.96:0/914705034 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) -- 0x7f42540689e0 con 0x7f425415c2c0 2024-05-28T06:05:44.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:44.897+0000 7f424be00640 1 -- 172.21.15.96:0/914705034 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v16) ==== 136+0+0 (secure 0 0 0) 0x7f424c059d60 con 0x7f425415c2c0 2024-05-28T06:05:44.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:44.900+0000 7f425a200640 1 -- 172.21.15.96:0/914705034 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f423006c8f0 msgr2=0x7f423006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:44.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:44.900+0000 7f425a200640 1 --2- 172.21.15.96:0/914705034 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f423006c8f0 0x7f423006edb0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f4244036000 tx=0x7f424400b000 comp rx=0 tx=0).stop 2024-05-28T06:05:44.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:44.901+0000 7f425a200640 1 -- 172.21.15.96:0/914705034 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f425415c2c0 msgr2=0x7f42540afc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:44.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:44.901+0000 7f425a200640 1 --2- 172.21.15.96:0/914705034 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f425415c2c0 0x7f42540afc00 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f424c00c440 tx=0x7f424c00c900 comp rx=0 tx=0).stop 2024-05-28T06:05:44.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:44.901+0000 7f425a200640 1 -- 172.21.15.96:0/914705034 shutdown_connections 2024-05-28T06:05:44.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:44.901+0000 7f425a200640 1 --2- 172.21.15.96:0/914705034 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f423006c8f0 0x7f423006edb0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:44.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:44.901+0000 7f425a200640 1 --2- 172.21.15.96:0/914705034 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f425415c2c0 0x7f42540afc00 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:44.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:44.901+0000 7f425a200640 1 --2- 172.21.15.96:0/914705034 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f425415b720 0x7f425415e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:44.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:44.901+0000 7f425a200640 1 --2- 172.21.15.96:0/914705034 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f425415b340 0x7f42541560e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:44.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:44.901+0000 7f425a200640 1 -- 172.21.15.96:0/914705034 >> 172.21.15.96:0/914705034 conn(0x7f42540b1e50 msgr2=0x7f4254072730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:44.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:44.901+0000 7f425a200640 1 -- 172.21.15.96:0/914705034 shutdown_connections 2024-05-28T06:05:44.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:44.901+0000 7f425a200640 1 -- 172.21.15.96:0/914705034 wait complete. 2024-05-28T06:05:44.923 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-05-28T06:05:45.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.116+0000 7f410a000640 1 Processor -- start 2024-05-28T06:05:45.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.116+0000 7f410a000640 1 -- start start 2024-05-28T06:05:45.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.118+0000 7f410a000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f41040b4e10 0x7f41040cde30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:45.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.118+0000 7f410a000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f41040ce370 0x7f41040d0590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:45.119 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.118+0000 7f410a000640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41040b51f0 con 0x7f41040d0ad0 2024-05-28T06:05:45.119 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.118+0000 7f410a000640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f41040d0f00 con 0x7f41040b4e10 2024-05-28T06:05:45.119 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.118+0000 7f410a000640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f41040aef60 con 0x7f41040ce370 2024-05-28T06:05:45.119 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.118+0000 7f4103400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f41040b4e10 0x7f41040cde30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:45.119 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.118+0000 7f4103400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f41040b4e10 0x7f41040cde30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:42300/0 (socket says 172.21.15.96:42300) 2024-05-28T06:05:45.119 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.118+0000 7f4103e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f41040d0ad0 0x7f4104099090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:59726/0 (socket says 172.21.15.96:59726) 2024-05-28T06:05:45.119 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.118+0000 7f4103400640 1 -- 172.21.15.96:0/4201356440 learned_addr learned my addr 172.21.15.96:0/4201356440 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:45.120 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.119+0000 7f4102a00640 1 --2- 172.21.15.96:0/4201356440 >> v2:172.21.15.123:3300/0 conn(0x7f41040ce370 0x7f41040d0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:45.120 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.119+0000 7f4103400640 1 -- 172.21.15.96:0/4201356440 >> v2:172.21.15.123:3300/0 conn(0x7f41040ce370 msgr2=0x7f41040d0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:45.120 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.119+0000 7f4103400640 1 --2- 172.21.15.96:0/4201356440 >> v2:172.21.15.123:3300/0 conn(0x7f41040ce370 0x7f41040d0590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:45.120 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.119+0000 7f4103400640 1 -- 172.21.15.96:0/4201356440 >> v1:172.21.15.96:6789/0 conn(0x7f41040d0ad0 legacy=0x7f4104099090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:45.120 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.119+0000 7f4103400640 1 -- 172.21.15.96:0/4201356440 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41040ad1d0 con 0x7f41040b4e10 2024-05-28T06:05:45.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.120+0000 7f4103400640 1 --2- 172.21.15.96:0/4201356440 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41040b4e10 0x7f41040cde30 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f41040ca860 tx=0x7f40f402a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cc74340e9046505 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:45.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.120+0000 7f4102000640 1 -- 172.21.15.96:0/4201356440 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40f4036070 con 0x7f41040b4e10 2024-05-28T06:05:45.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.120+0000 7f4102000640 1 -- 172.21.15.96:0/4201356440 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40f4031600 con 0x7f41040b4e10 2024-05-28T06:05:45.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.120+0000 7f4102000640 1 -- 172.21.15.96:0/4201356440 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40f40319d0 con 0x7f41040b4e10 2024-05-28T06:05:45.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.121+0000 7f410a000640 1 -- 172.21.15.96:0/4201356440 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41040b4e10 msgr2=0x7f41040cde30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:45.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.121+0000 7f410a000640 1 --2- 172.21.15.96:0/4201356440 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41040b4e10 0x7f41040cde30 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f41040ca860 tx=0x7f40f402a6a0 comp rx=0 tx=0).stop 2024-05-28T06:05:45.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.121+0000 7f410a000640 1 -- 172.21.15.96:0/4201356440 shutdown_connections 2024-05-28T06:05:45.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.121+0000 7f410a000640 1 --2- 172.21.15.96:0/4201356440 >> v2:172.21.15.123:3300/0 conn(0x7f41040ce370 0x7f41040d0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:45.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.121+0000 7f410a000640 1 --2- 172.21.15.96:0/4201356440 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41040b4e10 0x7f41040cde30 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:45.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.121+0000 7f410a000640 1 -- 172.21.15.96:0/4201356440 >> 172.21.15.96:0/4201356440 conn(0x7f410406bae0 msgr2=0x7f410406bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:45.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.121+0000 7f410a000640 1 -- 172.21.15.96:0/4201356440 shutdown_connections 2024-05-28T06:05:45.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.121+0000 7f410a000640 1 -- 172.21.15.96:0/4201356440 wait complete. 2024-05-28T06:05:45.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.123+0000 7f410a000640 1 Processor -- start 2024-05-28T06:05:45.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.123+0000 7f410a000640 1 -- start start 2024-05-28T06:05:45.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.124+0000 7f410a000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f41040b4e10 0x7f4104073740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:45.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.124+0000 7f410a000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41040ce370 0x7f4104158230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:45.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.124+0000 7f410a000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41040d0ad0 0x7f4104071b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:45.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.124+0000 7f410a000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f41040ce880 con 0x7f41040d0ad0 2024-05-28T06:05:45.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.124+0000 7f410a000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f41040ad750 con 0x7f41040ce370 2024-05-28T06:05:45.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.124+0000 7f410a000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f41040cea00 con 0x7f41040b4e10 2024-05-28T06:05:45.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.124+0000 7f4103e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41040d0ad0 0x7f4104071b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:45.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.124+0000 7f4103e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41040d0ad0 0x7f4104071b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:42310/0 (socket says 172.21.15.96:42310) 2024-05-28T06:05:45.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.124+0000 7f4103e00640 1 -- 172.21.15.96:0/505865162 learned_addr learned my addr 172.21.15.96:0/505865162 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:45.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.125+0000 7f4103400640 1 --2- 172.21.15.96:0/505865162 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f41040b4e10 0x7f4104073740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:45.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.125+0000 7f4102a00640 1 --2- 172.21.15.96:0/505865162 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41040ce370 0x7f4104158230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:45.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.125+0000 7f4103e00640 1 -- 172.21.15.96:0/505865162 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f41040b4e10 msgr2=0x7f4104073740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:45.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.125+0000 7f4103e00640 1 --2- 172.21.15.96:0/505865162 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f41040b4e10 0x7f4104073740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:45.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.125+0000 7f4103e00640 1 -- 172.21.15.96:0/505865162 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41040ce370 msgr2=0x7f4104158230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:45.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.125+0000 7f4103e00640 1 --2- 172.21.15.96:0/505865162 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41040ce370 0x7f4104158230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:45.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.125+0000 7f4103e00640 1 -- 172.21.15.96:0/505865162 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4104072580 con 0x7f41040d0ad0 2024-05-28T06:05:45.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.125+0000 7f4103e00640 1 --2- 172.21.15.96:0/505865162 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41040d0ad0 0x7f4104071b30 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f40fc00c3f0 tx=0x7f40fc00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:45.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.125+0000 7f40fb400640 1 -- 172.21.15.96:0/505865162 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40fc0074e0 con 0x7f41040d0ad0 2024-05-28T06:05:45.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.125+0000 7f40fb400640 1 -- 172.21.15.96:0/505865162 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40fc007680 con 0x7f41040d0ad0 2024-05-28T06:05:45.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.125+0000 7f40fb400640 1 -- 172.21.15.96:0/505865162 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40fc004a00 con 0x7f41040d0ad0 2024-05-28T06:05:45.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.125+0000 7f410a000640 1 -- 172.21.15.96:0/505865162 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4104072860 con 0x7f41040d0ad0 2024-05-28T06:05:45.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.126+0000 7f410a000640 1 -- 172.21.15.96:0/505865162 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41040c54c0 con 0x7f41040d0ad0 2024-05-28T06:05:45.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.127+0000 7f410a000640 1 -- 172.21.15.96:0/505865162 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4104072a60 con 0x7f41040d0ad0 2024-05-28T06:05:45.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.128+0000 7f40fb400640 1 -- 172.21.15.96:0/505865162 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f40fc01a020 con 0x7f41040d0ad0 2024-05-28T06:05:45.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.129+0000 7f40fb400640 1 --2- 172.21.15.96:0/505865162 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f40e406c8f0 0x7f40e406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:45.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.129+0000 7f40fb400640 1 -- 172.21.15.96:0/505865162 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f40fc08b2f0 con 0x7f41040d0ad0 2024-05-28T06:05:45.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.130+0000 7f4103400640 1 --2- 172.21.15.96:0/505865162 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f40e406c8f0 0x7f40e406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:45.131 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.131+0000 7f4103400640 1 --2- 172.21.15.96:0/505865162 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f40e406c8f0 0x7f40e406edb0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f40f4033000 tx=0x7f40f4009000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:45.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.134+0000 7f40fb400640 1 -- 172.21.15.96:0/505865162 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4104072a60 con 0x7f41040d0ad0 2024-05-28T06:05:45.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.362+0000 7f410a000640 1 -- 172.21.15.96:0/505865162 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7f41040c56c0 con 0x7f40e406c8f0 2024-05-28T06:05:45.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.366+0000 7f40fb400640 1 -- 172.21.15.96:0/505865162 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7f41040c56c0 con 0x7f40e406c8f0 2024-05-28T06:05:45.366 INFO:teuthology.orchestra.run.smithi096.stdout:[ 2024-05-28T06:05:45.367 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:05:45.367 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "cephfs" 2024-05-28T06:05:45.367 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:05:45.367 INFO:teuthology.orchestra.run.smithi096.stdout:] 2024-05-28T06:05:45.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.368+0000 7f410a000640 1 -- 172.21.15.96:0/505865162 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f40e406c8f0 msgr2=0x7f40e406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:45.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.368+0000 7f410a000640 1 --2- 172.21.15.96:0/505865162 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f40e406c8f0 0x7f40e406edb0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f40f4033000 tx=0x7f40f4009000 comp rx=0 tx=0).stop 2024-05-28T06:05:45.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.368+0000 7f410a000640 1 -- 172.21.15.96:0/505865162 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41040d0ad0 msgr2=0x7f4104071b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:45.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.368+0000 7f410a000640 1 --2- 172.21.15.96:0/505865162 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41040d0ad0 0x7f4104071b30 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f40fc00c3f0 tx=0x7f40fc00c8b0 comp rx=0 tx=0).stop 2024-05-28T06:05:45.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.368+0000 7f410a000640 1 -- 172.21.15.96:0/505865162 shutdown_connections 2024-05-28T06:05:45.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.368+0000 7f410a000640 1 --2- 172.21.15.96:0/505865162 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f40e406c8f0 0x7f40e406edb0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:45.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.368+0000 7f410a000640 1 --2- 172.21.15.96:0/505865162 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41040d0ad0 0x7f4104071b30 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:45.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.368+0000 7f410a000640 1 --2- 172.21.15.96:0/505865162 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41040ce370 0x7f4104158230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:45.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.369+0000 7f410a000640 1 --2- 172.21.15.96:0/505865162 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f41040b4e10 0x7f4104073740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:45.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.369+0000 7f410a000640 1 -- 172.21.15.96:0/505865162 >> 172.21.15.96:0/505865162 conn(0x7f410406bae0 msgr2=0x7f410415cc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:45.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.369+0000 7f410a000640 1 -- 172.21.15.96:0/505865162 shutdown_connections 2024-05-28T06:05:45.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.369+0000 7f410a000640 1 -- 172.21.15.96:0/505865162 wait complete. 2024-05-28T06:05:45.388 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-05-28T06:05:45.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.596+0000 7f11d5800640 1 Processor -- start 2024-05-28T06:05:45.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.597+0000 7f11d5800640 1 -- start start 2024-05-28T06:05:45.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.598+0000 7f11d5800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f11d00cfa40 0x7f11d0169690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:45.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.598+0000 7f11d5800640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11d015ec50 con 0x7f11d015e870 2024-05-28T06:05:45.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.598+0000 7f11d5800640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11d00cff80 con 0x7f11d00d16a0 2024-05-28T06:05:45.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.598+0000 7f11d5800640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f11d00d2150 con 0x7f11d00cfa40 2024-05-28T06:05:45.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.598+0000 7f11cfe00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f11d015e870 0x7f11d00d0f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:59734/0 (socket says 172.21.15.96:59734) 2024-05-28T06:05:45.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.598+0000 7f11cfe00640 1 -- 172.21.15.96:0/295839691 learned_addr learned my addr 172.21.15.96:0/295839691 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:05:45.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.599+0000 7f11cea00640 1 -- 172.21.15.96:0/295839691 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 63561930 0 0) 0x7f11d015ec50 con 0x7f11d015e870 2024-05-28T06:05:45.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.599+0000 7f11cea00640 1 -- 172.21.15.96:0/295839691 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f11b0003520 con 0x7f11d015e870 2024-05-28T06:05:45.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.599+0000 7f11d4e00640 1 --2- 172.21.15.96:0/295839691 >> v2:172.21.15.123:3300/0 conn(0x7f11d00cfa40 0x7f11d0169690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:45.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.599+0000 7f11cea00640 1 -- 172.21.15.96:0/295839691 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 731095402 0 0) 0x7f11b0003520 con 0x7f11d015e870 2024-05-28T06:05:45.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.599+0000 7f11cea00640 1 -- 172.21.15.96:0/295839691 >> v2:172.21.15.123:3300/0 conn(0x7f11d00cfa40 msgr2=0x7f11d0169690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:45.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.599+0000 7f11cea00640 1 --2- 172.21.15.96:0/295839691 >> v2:172.21.15.123:3300/0 conn(0x7f11d00cfa40 0x7f11d0169690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:45.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.599+0000 7f11cea00640 1 -- 172.21.15.96:0/295839691 >> v1:172.21.15.123:6789/0 conn(0x7f11d00d16a0 legacy=0x7f11d00cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:45.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.599+0000 7f11cea00640 1 -- 172.21.15.96:0/295839691 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11d00d0260 con 0x7f11d015e870 2024-05-28T06:05:45.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.599+0000 7f11cea00640 1 -- 172.21.15.96:0/295839691 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f11c0001670 con 0x7f11d015e870 2024-05-28T06:05:45.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.599+0000 7f11cea00640 1 -- 172.21.15.96:0/295839691 >> v1:172.21.15.96:6789/0 conn(0x7f11d015e870 legacy=0x7f11d00d0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:45.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.599+0000 7f11cea00640 1 --2- 172.21.15.96:0/295839691 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11b0003a00 0x7f11b0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:45.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.599+0000 7f11cea00640 1 --2- 172.21.15.96:0/295839691 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f11b0004800 0x7f11b0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:45.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.599+0000 7f11cea00640 1 --2- 172.21.15.96:0/295839691 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f11b0005210 0x7f11b000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:45.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.599+0000 7f11cea00640 1 -- 172.21.15.96:0/295839691 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f11d00d2150 con 0x7f11b0003a00 2024-05-28T06:05:45.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.599+0000 7f11cea00640 1 -- 172.21.15.96:0/295839691 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f11b000c140 con 0x7f11b0004800 2024-05-28T06:05:45.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.599+0000 7f11cea00640 1 -- 172.21.15.96:0/295839691 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f11b000c340 con 0x7f11b0005210 2024-05-28T06:05:45.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.600+0000 7f11cfe00640 1 --2- 172.21.15.96:0/295839691 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11b0003a00 0x7f11b0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:45.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.600+0000 7f11cfe00640 1 -- 172.21.15.96:0/295839691 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f11b0005210 msgr2=0x7f11b000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:45.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.600+0000 7f11cfe00640 1 --2- 172.21.15.96:0/295839691 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f11b0005210 0x7f11b000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:45.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.600+0000 7f11cfe00640 1 -- 172.21.15.96:0/295839691 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f11b0004800 msgr2=0x7f11b0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:45.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.600+0000 7f11cfe00640 1 --2- 172.21.15.96:0/295839691 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f11b0004800 0x7f11b0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:45.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.600+0000 7f11cfe00640 1 -- 172.21.15.96:0/295839691 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11b0003520 con 0x7f11b0003a00 2024-05-28T06:05:45.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.600+0000 7f11d4e00640 1 -- 172.21.15.96:0/295839691 reap_dead start 2024-05-28T06:05:45.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.601+0000 7f11cfe00640 1 --2- 172.21.15.96:0/295839691 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11b0003a00 0x7f11b0003e00 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f11c0001eb0 tx=0x7f11b0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8b2355db4d5a3e1 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:45.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.601+0000 7f11cea00640 1 -- 172.21.15.96:0/295839691 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11c0007c80 con 0x7f11b0003a00 2024-05-28T06:05:45.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.601+0000 7f11cea00640 1 -- 172.21.15.96:0/295839691 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11c0005af0 con 0x7f11b0003a00 2024-05-28T06:05:45.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.601+0000 7f11cea00640 1 -- 172.21.15.96:0/295839691 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11c0007070 con 0x7f11b0003a00 2024-05-28T06:05:45.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.602+0000 7f11d5800640 1 -- 172.21.15.96:0/295839691 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11b0003a00 msgr2=0x7f11b0003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:45.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.602+0000 7f11d5800640 1 --2- 172.21.15.96:0/295839691 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11b0003a00 0x7f11b0003e00 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f11c0001eb0 tx=0x7f11b0005830 comp rx=0 tx=0).stop 2024-05-28T06:05:45.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.602+0000 7f11d5800640 1 -- 172.21.15.96:0/295839691 shutdown_connections 2024-05-28T06:05:45.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.602+0000 7f11d5800640 1 --2- 172.21.15.96:0/295839691 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11b0003a00 0x7f11b0003e00 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:45.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.602+0000 7f11d5800640 1 -- 172.21.15.96:0/295839691 >> 172.21.15.96:0/295839691 conn(0x7f11d008bfc0 msgr2=0x7f11d00d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:45.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.602+0000 7f11d5800640 1 -- 172.21.15.96:0/295839691 shutdown_connections 2024-05-28T06:05:45.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.602+0000 7f11d5800640 1 -- 172.21.15.96:0/295839691 wait complete. 2024-05-28T06:05:45.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.604+0000 7f11d5800640 1 Processor -- start 2024-05-28T06:05:45.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.604+0000 7f11d5800640 1 -- start start 2024-05-28T06:05:45.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.605+0000 7f11d5800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11b0003a00 0x7f11d00c9930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:45.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.605+0000 7f11d5800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f11d00ca350 0x7f11d00ca7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:45.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.605+0000 7f11d5800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f11d01f4580 0x7f11d01f4a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:45.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.605+0000 7f11d5800640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f11d00d4fb0 con 0x7f11b0003a00 2024-05-28T06:05:45.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.605+0000 7f11d5800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f11d00d0820 con 0x7f11d00ca350 2024-05-28T06:05:45.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.605+0000 7f11d5800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f11d00cf330 con 0x7f11d01f4580 2024-05-28T06:05:45.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.605+0000 7f11cfe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11b0003a00 0x7f11d00c9930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:45.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.605+0000 7f11cfe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11b0003a00 0x7f11d00c9930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:42334/0 (socket says 172.21.15.96:42334) 2024-05-28T06:05:45.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.605+0000 7f11cfe00640 1 -- 172.21.15.96:0/554281564 learned_addr learned my addr 172.21.15.96:0/554281564 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:45.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.605+0000 7f11cfe00640 1 -- 172.21.15.96:0/554281564 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f11d01f4580 msgr2=0x7f11d01f4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:45.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.605+0000 7f11cfe00640 1 --2- 172.21.15.96:0/554281564 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f11d01f4580 0x7f11d01f4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:45.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.605+0000 7f11cfe00640 1 -- 172.21.15.96:0/554281564 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f11d00ca350 msgr2=0x7f11d00ca7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:45.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.605+0000 7f11cf400640 1 --2- 172.21.15.96:0/554281564 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f11d00ca350 0x7f11d00ca7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:45.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.606+0000 7f11cfe00640 1 --2- 172.21.15.96:0/554281564 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f11d00ca350 0x7f11d00ca7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:45.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.606+0000 7f11cfe00640 1 -- 172.21.15.96:0/554281564 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11d01f52b0 con 0x7f11b0003a00 2024-05-28T06:05:45.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.606+0000 7f11cfe00640 1 --2- 172.21.15.96:0/554281564 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11b0003a00 0x7f11d00c9930 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f11c000c250 tx=0x7f11c0001970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:45.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.606+0000 7f11c7e00640 1 -- 172.21.15.96:0/554281564 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11c0042070 con 0x7f11b0003a00 2024-05-28T06:05:45.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.606+0000 7f11c7e00640 1 -- 172.21.15.96:0/554281564 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11c00015b0 con 0x7f11b0003a00 2024-05-28T06:05:45.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.606+0000 7f11c7e00640 1 -- 172.21.15.96:0/554281564 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11c0037040 con 0x7f11b0003a00 2024-05-28T06:05:45.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.606+0000 7f11d5800640 1 -- 172.21.15.96:0/554281564 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11d01f7a70 con 0x7f11b0003a00 2024-05-28T06:05:45.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.606+0000 7f11d5800640 1 -- 172.21.15.96:0/554281564 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11d01f8030 con 0x7f11b0003a00 2024-05-28T06:05:45.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.608+0000 7f11d5800640 1 -- 172.21.15.96:0/554281564 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11d00689e0 con 0x7f11b0003a00 2024-05-28T06:05:45.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.608+0000 7f11c7e00640 1 -- 172.21.15.96:0/554281564 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f11c0045070 con 0x7f11b0003a00 2024-05-28T06:05:45.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.609+0000 7f11c7e00640 1 --2- 172.21.15.96:0/554281564 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f11ac06c920 0x7f11ac06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:45.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.610+0000 7f11c7e00640 1 -- 172.21.15.96:0/554281564 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f11c00ab0e0 con 0x7f11b0003a00 2024-05-28T06:05:45.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.610+0000 7f11cf400640 1 --2- 172.21.15.96:0/554281564 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f11ac06c920 0x7f11ac06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:45.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.611+0000 7f11cf400640 1 --2- 172.21.15.96:0/554281564 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f11ac06c920 0x7f11ac06ede0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f11d0050680 tx=0x7f11b8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:45.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.614+0000 7f11c7e00640 1 -- 172.21.15.96:0/554281564 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f11c0078eb0 con 0x7f11b0003a00 2024-05-28T06:05:45.818 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.817+0000 7f11d5800640 1 -- 172.21.15.96:0/554281564 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f11d00ba500 con 0x7f11b0003a00 2024-05-28T06:05:45.818 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.818+0000 7f11c7e00640 1 -- 172.21.15.96:0/554281564 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) ==== 122+0+0 (secure 0 0 0) 0x7f11c007d1b0 con 0x7f11b0003a00 2024-05-28T06:05:45.821 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.821+0000 7f11d5800640 1 -- 172.21.15.96:0/554281564 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f11ac06c920 msgr2=0x7f11ac06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:45.822 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.821+0000 7f11d5800640 1 --2- 172.21.15.96:0/554281564 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f11ac06c920 0x7f11ac06ede0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f11d0050680 tx=0x7f11b8009040 comp rx=0 tx=0).stop 2024-05-28T06:05:45.822 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.821+0000 7f11d5800640 1 -- 172.21.15.96:0/554281564 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11b0003a00 msgr2=0x7f11d00c9930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:45.822 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.821+0000 7f11d5800640 1 --2- 172.21.15.96:0/554281564 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11b0003a00 0x7f11d00c9930 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f11c000c250 tx=0x7f11c0001970 comp rx=0 tx=0).stop 2024-05-28T06:05:45.822 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.821+0000 7f11d5800640 1 -- 172.21.15.96:0/554281564 shutdown_connections 2024-05-28T06:05:45.822 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.821+0000 7f11d5800640 1 --2- 172.21.15.96:0/554281564 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f11ac06c920 0x7f11ac06ede0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:45.822 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.821+0000 7f11d5800640 1 --2- 172.21.15.96:0/554281564 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f11d01f4580 0x7f11d01f4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:45.823 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.821+0000 7f11d5800640 1 --2- 172.21.15.96:0/554281564 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f11d00ca350 0x7f11d00ca7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:45.823 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.821+0000 7f11d5800640 1 --2- 172.21.15.96:0/554281564 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11b0003a00 0x7f11d00c9930 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:45.823 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.821+0000 7f11d5800640 1 -- 172.21.15.96:0/554281564 >> 172.21.15.96:0/554281564 conn(0x7f11d008bfc0 msgr2=0x7f11d00b4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:45.823 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.821+0000 7f11d5800640 1 -- 172.21.15.96:0/554281564 shutdown_connections 2024-05-28T06:05:45.823 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:45.821+0000 7f11d5800640 1 -- 172.21.15.96:0/554281564 wait complete. 2024-05-28T06:05:45.841 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv1 --mode=777 2024-05-28T06:05:46.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.038+0000 7fb72e600640 1 Processor -- start 2024-05-28T06:05:46.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.039+0000 7fb72e600640 1 -- start start 2024-05-28T06:05:46.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.040+0000 7fb72e600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb72815c3f0 0x7fb7281608d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:46.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.040+0000 7fb72e600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fb72806ef00 0x7fb7280d0320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:46.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.040+0000 7fb72e600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb72815c7d0 con 0x7fb7280b4790 2024-05-28T06:05:46.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.040+0000 7fb72e600640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fb7280ad9a0 con 0x7fb72806ef00 2024-05-28T06:05:46.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.040+0000 7fb72e600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fb728160e10 con 0x7fb72815c3f0 2024-05-28T06:05:46.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.040+0000 7fb727e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fb72806ef00 0x7fb7280d0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:46.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.041+0000 7fb727e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fb72806ef00 0x7fb7280d0320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:42336/0 (socket says 172.21.15.96:42336) 2024-05-28T06:05:46.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.041+0000 7fb727e00640 1 -- 172.21.15.96:0/4116726902 learned_addr learned my addr 172.21.15.96:0/4116726902 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:46.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.041+0000 7fb72d200640 1 --2- 172.21.15.96:0/4116726902 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb72815c3f0 0x7fb7281608d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:46.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.041+0000 7fb727e00640 1 -- 172.21.15.96:0/4116726902 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb72815c3f0 msgr2=0x7fb7281608d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:46.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.041+0000 7fb727e00640 1 --2- 172.21.15.96:0/4116726902 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb72815c3f0 0x7fb7281608d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:46.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.041+0000 7fb727e00640 1 -- 172.21.15.96:0/4116726902 >> v1:172.21.15.123:6789/0 conn(0x7fb7280b4790 legacy=0x7fb7280b30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:46.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.041+0000 7fb727e00640 1 -- 172.21.15.96:0/4116726902 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7280ad3e0 con 0x7fb72806ef00 2024-05-28T06:05:46.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.042+0000 7fb727e00640 1 --2- 172.21.15.96:0/4116726902 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb72806ef00 0x7fb7280d0320 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fb714009870 tx=0x7fb71402a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da5c24f4c58f7e4b server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:46.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.042+0000 7fb727400640 1 -- 172.21.15.96:0/4116726902 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb714037070 con 0x7fb72806ef00 2024-05-28T06:05:46.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.042+0000 7fb727400640 1 -- 172.21.15.96:0/4116726902 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb714032440 con 0x7fb72806ef00 2024-05-28T06:05:46.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.042+0000 7fb727400640 1 -- 172.21.15.96:0/4116726902 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb714032810 con 0x7fb72806ef00 2024-05-28T06:05:46.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.043+0000 7fb72e600640 1 -- 172.21.15.96:0/4116726902 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb72806ef00 msgr2=0x7fb7280d0320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:46.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.043+0000 7fb72e600640 1 --2- 172.21.15.96:0/4116726902 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb72806ef00 0x7fb7280d0320 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fb714009870 tx=0x7fb71402a210 comp rx=0 tx=0).stop 2024-05-28T06:05:46.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.043+0000 7fb72e600640 1 -- 172.21.15.96:0/4116726902 shutdown_connections 2024-05-28T06:05:46.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.043+0000 7fb72e600640 1 --2- 172.21.15.96:0/4116726902 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb72806ef00 0x7fb7280d0320 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:46.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.043+0000 7fb72e600640 1 --2- 172.21.15.96:0/4116726902 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb72815c3f0 0x7fb7281608d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:46.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.043+0000 7fb72e600640 1 -- 172.21.15.96:0/4116726902 >> 172.21.15.96:0/4116726902 conn(0x7fb72806bae0 msgr2=0x7fb72806bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:46.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.043+0000 7fb72e600640 1 -- 172.21.15.96:0/4116726902 shutdown_connections 2024-05-28T06:05:46.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.043+0000 7fb72e600640 1 -- 172.21.15.96:0/4116726902 wait complete. 2024-05-28T06:05:46.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.045+0000 7fb72e600640 1 Processor -- start 2024-05-28T06:05:46.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.045+0000 7fb72e600640 1 -- start start 2024-05-28T06:05:46.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.046+0000 7fb72e600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb72806ef00 0x7fb7280b0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:46.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.046+0000 7fb72e600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb7280b4790 0x7fb72815c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:46.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.046+0000 7fb72e600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb72815c3f0 0x7fb728150680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:46.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.046+0000 7fb72e600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb7281603b0 con 0x7fb72806ef00 2024-05-28T06:05:46.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.046+0000 7fb72e600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fb7281600d0 con 0x7fb72815c3f0 2024-05-28T06:05:46.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.046+0000 7fb72e600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fb7280ad090 con 0x7fb7280b4790 2024-05-28T06:05:46.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.046+0000 7fb72d200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb72806ef00 0x7fb7280b0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:46.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.046+0000 7fb72d200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb72806ef00 0x7fb7280b0fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:42340/0 (socket says 172.21.15.96:42340) 2024-05-28T06:05:46.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.046+0000 7fb72d200640 1 -- 172.21.15.96:0/326953731 learned_addr learned my addr 172.21.15.96:0/326953731 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:46.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.047+0000 7fb727e00640 1 --2- 172.21.15.96:0/326953731 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb7280b4790 0x7fb72815c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:46.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.047+0000 7fb72d200640 1 -- 172.21.15.96:0/326953731 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb7280b4790 msgr2=0x7fb72815c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:46.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.047+0000 7fb72d200640 1 --2- 172.21.15.96:0/326953731 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb7280b4790 0x7fb72815c9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:46.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.047+0000 7fb72d200640 1 -- 172.21.15.96:0/326953731 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb72815c3f0 msgr2=0x7fb728150680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:46.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.047+0000 7fb72d200640 1 --2- 172.21.15.96:0/326953731 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb72815c3f0 0x7fb728150680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:46.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.047+0000 7fb72d200640 1 -- 172.21.15.96:0/326953731 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7281539a0 con 0x7fb72806ef00 2024-05-28T06:05:46.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.047+0000 7fb72d200640 1 --2- 172.21.15.96:0/326953731 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb72806ef00 0x7fb7280b0fc0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fb71c00d830 tx=0x7fb71c00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:46.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.047+0000 7fb724c00640 1 -- 172.21.15.96:0/326953731 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb71c00b6c0 con 0x7fb72806ef00 2024-05-28T06:05:46.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.047+0000 7fb724c00640 1 -- 172.21.15.96:0/326953731 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb71c019070 con 0x7fb72806ef00 2024-05-28T06:05:46.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.047+0000 7fb72e600640 1 -- 172.21.15.96:0/326953731 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb72815d0d0 con 0x7fb72806ef00 2024-05-28T06:05:46.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.047+0000 7fb724c00640 1 -- 172.21.15.96:0/326953731 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb71c00ba80 con 0x7fb72806ef00 2024-05-28T06:05:46.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.048+0000 7fb72e600640 1 -- 172.21.15.96:0/326953731 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb72815d690 con 0x7fb72806ef00 2024-05-28T06:05:46.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.049+0000 7fb72e600640 1 -- 172.21.15.96:0/326953731 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7280689e0 con 0x7fb72806ef00 2024-05-28T06:05:46.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.050+0000 7fb724c00640 1 -- 172.21.15.96:0/326953731 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb71c01e460 con 0x7fb72806ef00 2024-05-28T06:05:46.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.051+0000 7fb724c00640 1 --2- 172.21.15.96:0/326953731 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb70806c8f0 0x7fb70806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:46.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.051+0000 7fb724c00640 1 -- 172.21.15.96:0/326953731 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7fb71c08c240 con 0x7fb72806ef00 2024-05-28T06:05:46.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.052+0000 7fb727e00640 1 --2- 172.21.15.96:0/326953731 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb70806c8f0 0x7fb70806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:46.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.053+0000 7fb727e00640 1 --2- 172.21.15.96:0/326953731 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb70806c8f0 0x7fb70806edb0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fb714034000 tx=0x7fb71402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:46.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.056+0000 7fb724c00640 1 -- 172.21.15.96:0/326953731 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb71c05a660 con 0x7fb72806ef00 2024-05-28T06:05:46.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.283+0000 7fb72e600640 1 -- 172.21.15.96:0/326953731 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv1", "mode": "777", "target": ["mon-mgr", ""]}) -- 0x7fb7280733c0 con 0x7fb70806c8f0 2024-05-28T06:05:46.296 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:05:46.295+0000 7fea59e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:05:46.296 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:05:46.295+0000 7fea59e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:05:46.296 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:05:46.295+0000 7fea59e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:05:46.297 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:05:46.295+0000 7fea59e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:05:46.297 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:05:46.295+0000 7fea59e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:05:46.376 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.374+0000 7fb724c00640 1 -- 172.21.15.96:0/326953731 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fb7280733c0 con 0x7fb70806c8f0 2024-05-28T06:05:46.378 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.377+0000 7fb72e600640 1 -- 172.21.15.96:0/326953731 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb70806c8f0 msgr2=0x7fb70806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:46.378 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.377+0000 7fb72e600640 1 --2- 172.21.15.96:0/326953731 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb70806c8f0 0x7fb70806edb0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fb714034000 tx=0x7fb71402e000 comp rx=0 tx=0).stop 2024-05-28T06:05:46.378 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.377+0000 7fb72e600640 1 -- 172.21.15.96:0/326953731 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb72806ef00 msgr2=0x7fb7280b0fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:46.378 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.377+0000 7fb72e600640 1 --2- 172.21.15.96:0/326953731 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb72806ef00 0x7fb7280b0fc0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fb71c00d830 tx=0x7fb71c00dcf0 comp rx=0 tx=0).stop 2024-05-28T06:05:46.378 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.377+0000 7fb72e600640 1 -- 172.21.15.96:0/326953731 shutdown_connections 2024-05-28T06:05:46.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.377+0000 7fb72e600640 1 --2- 172.21.15.96:0/326953731 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb70806c8f0 0x7fb70806edb0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:46.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.377+0000 7fb72e600640 1 --2- 172.21.15.96:0/326953731 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb72815c3f0 0x7fb728150680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:46.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.377+0000 7fb72e600640 1 --2- 172.21.15.96:0/326953731 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb7280b4790 0x7fb72815c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:46.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.377+0000 7fb72e600640 1 --2- 172.21.15.96:0/326953731 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb72806ef00 0x7fb7280b0fc0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:46.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.377+0000 7fb72e600640 1 -- 172.21.15.96:0/326953731 >> 172.21.15.96:0/326953731 conn(0x7fb72806bae0 msgr2=0x7fb728072310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:46.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.377+0000 7fb72e600640 1 -- 172.21.15.96:0/326953731 shutdown_connections 2024-05-28T06:05:46.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.377+0000 7fb72e600640 1 -- 172.21.15.96:0/326953731 wait complete. 2024-05-28T06:05:46.398 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-05-28T06:05:46.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.599+0000 7fc33fa00640 1 Processor -- start 2024-05-28T06:05:46.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.599+0000 7fc33fa00640 1 -- start start 2024-05-28T06:05:46.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.601+0000 7fc33fa00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fc3380d0640 0x7fc3380d0a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:46.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.601+0000 7fc33fa00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc33815b6f0 0x7fc33806e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:46.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.601+0000 7fc33fa00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fc3380725e0 0x7fc3380b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:46.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.601+0000 7fc33fa00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fc338071c40 con 0x7fc3380d0640 2024-05-28T06:05:46.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.601+0000 7fc33fa00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fc3380ade90 con 0x7fc3380725e0 2024-05-28T06:05:46.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.601+0000 7fc33fa00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fc3380d0f60 con 0x7fc33815b6f0 2024-05-28T06:05:46.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.601+0000 7fc33d000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fc3380d0640 0x7fc3380d0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:46.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.601+0000 7fc33d000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fc3380d0640 0x7fc3380d0a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:42350/0 (socket says 172.21.15.96:42350) 2024-05-28T06:05:46.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.601+0000 7fc33d000640 1 -- 172.21.15.96:0/2934320671 learned_addr learned my addr 172.21.15.96:0/2934320671 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:46.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.602+0000 7fc33da00640 1 --2- 172.21.15.96:0/2934320671 >> v2:172.21.15.123:3300/0 conn(0x7fc3380725e0 0x7fc3380b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:46.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.602+0000 7fc32fe00640 1 --2- 172.21.15.96:0/2934320671 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc33815b6f0 0x7fc33806e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:46.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.602+0000 7fc33d000640 1 -- 172.21.15.96:0/2934320671 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc33815b6f0 msgr2=0x7fc33806e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:46.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.602+0000 7fc33d000640 1 --2- 172.21.15.96:0/2934320671 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc33815b6f0 0x7fc33806e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:46.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.602+0000 7fc33d000640 1 -- 172.21.15.96:0/2934320671 >> v2:172.21.15.123:3300/0 conn(0x7fc3380725e0 msgr2=0x7fc3380b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:05:46.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.602+0000 7fc33d000640 1 --2- 172.21.15.96:0/2934320671 >> v2:172.21.15.123:3300/0 conn(0x7fc3380725e0 0x7fc3380b44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:46.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.602+0000 7fc33d000640 1 -- 172.21.15.96:0/2934320671 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc338071e70 con 0x7fc3380d0640 2024-05-28T06:05:46.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.602+0000 7fc33da00640 1 --2- 172.21.15.96:0/2934320671 >> v2:172.21.15.123:3300/0 conn(0x7fc3380725e0 0x7fc3380b44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-28T06:05:46.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.603+0000 7fc33d000640 1 --2- 172.21.15.96:0/2934320671 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc3380d0640 0x7fc3380d0a20 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fc338070130 tx=0x7fc32802c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddd4c256fef6262e server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:46.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.603+0000 7fc337e00640 1 -- 172.21.15.96:0/2934320671 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc328032790 con 0x7fc3380d0640 2024-05-28T06:05:46.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.604+0000 7fc337e00640 1 -- 172.21.15.96:0/2934320671 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc328038070 con 0x7fc3380d0640 2024-05-28T06:05:46.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.604+0000 7fc337e00640 1 -- 172.21.15.96:0/2934320671 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc328032b50 con 0x7fc3380d0640 2024-05-28T06:05:46.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.604+0000 7fc33fa00640 1 -- 172.21.15.96:0/2934320671 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc3380d0640 msgr2=0x7fc3380d0a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:46.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.604+0000 7fc33fa00640 1 --2- 172.21.15.96:0/2934320671 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc3380d0640 0x7fc3380d0a20 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fc338070130 tx=0x7fc32802c890 comp rx=0 tx=0).stop 2024-05-28T06:05:46.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.604+0000 7fc33fa00640 1 -- 172.21.15.96:0/2934320671 shutdown_connections 2024-05-28T06:05:46.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.604+0000 7fc33fa00640 1 --2- 172.21.15.96:0/2934320671 >> v2:172.21.15.123:3300/0 conn(0x7fc3380725e0 0x7fc3380b44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:46.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.604+0000 7fc33fa00640 1 --2- 172.21.15.96:0/2934320671 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc33815b6f0 0x7fc33806e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:46.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.604+0000 7fc33fa00640 1 --2- 172.21.15.96:0/2934320671 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc3380d0640 0x7fc3380d0a20 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:46.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.604+0000 7fc33fa00640 1 -- 172.21.15.96:0/2934320671 >> 172.21.15.96:0/2934320671 conn(0x7fc33806bd20 msgr2=0x7fc33806c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:46.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.604+0000 7fc33fa00640 1 -- 172.21.15.96:0/2934320671 shutdown_connections 2024-05-28T06:05:46.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.604+0000 7fc33fa00640 1 -- 172.21.15.96:0/2934320671 wait complete. 2024-05-28T06:05:46.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.606+0000 7fc33fa00640 1 Processor -- start 2024-05-28T06:05:46.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.606+0000 7fc33fa00640 1 -- start start 2024-05-28T06:05:46.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.607+0000 7fc33fa00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3380725e0 0x7fc33815a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:46.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.607+0000 7fc33fa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc3380d0640 0x7fc338157600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:46.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.607+0000 7fc33fa00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc33815b6f0 0x7fc338157b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:46.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.607+0000 7fc33fa00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc33806e890 con 0x7fc3380d0640 2024-05-28T06:05:46.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.607+0000 7fc33fa00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc3380ae130 con 0x7fc3380725e0 2024-05-28T06:05:46.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.607+0000 7fc33fa00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fc3380b4a10 con 0x7fc33815b6f0 2024-05-28T06:05:46.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.607+0000 7fc32fe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc3380d0640 0x7fc338157600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:46.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.607+0000 7fc32fe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc3380d0640 0x7fc338157600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:42352/0 (socket says 172.21.15.96:42352) 2024-05-28T06:05:46.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.607+0000 7fc32fe00640 1 -- 172.21.15.96:0/3747925564 learned_addr learned my addr 172.21.15.96:0/3747925564 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:05:46.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.608+0000 7fc33d000640 1 --2- 172.21.15.96:0/3747925564 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3380725e0 0x7fc33815a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:46.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.608+0000 7fc32fe00640 1 -- 172.21.15.96:0/3747925564 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc33815b6f0 msgr2=0x7fc338157b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:46.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.608+0000 7fc32fe00640 1 --2- 172.21.15.96:0/3747925564 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc33815b6f0 0x7fc338157b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:46.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.608+0000 7fc32fe00640 1 -- 172.21.15.96:0/3747925564 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3380725e0 msgr2=0x7fc33815a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:46.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.608+0000 7fc32fe00640 1 --2- 172.21.15.96:0/3747925564 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3380725e0 0x7fc33815a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:46.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.608+0000 7fc32fe00640 1 -- 172.21.15.96:0/3747925564 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc33814fd30 con 0x7fc3380d0640 2024-05-28T06:05:46.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.608+0000 7fc33d000640 1 --2- 172.21.15.96:0/3747925564 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3380725e0 0x7fc33815a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:05:46.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.608+0000 7fc32fe00640 1 --2- 172.21.15.96:0/3747925564 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc3380d0640 0x7fc338157600 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fc32000d5f0 tx=0x7fc32000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:46.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.608+0000 7fc335600640 1 -- 172.21.15.96:0/3747925564 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3200125f0 con 0x7fc3380d0640 2024-05-28T06:05:46.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.608+0000 7fc335600640 1 -- 172.21.15.96:0/3747925564 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc320019070 con 0x7fc3380d0640 2024-05-28T06:05:46.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.608+0000 7fc33fa00640 1 -- 172.21.15.96:0/3747925564 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc33815ce10 con 0x7fc3380d0640 2024-05-28T06:05:46.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.609+0000 7fc335600640 1 -- 172.21.15.96:0/3747925564 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3200129d0 con 0x7fc3380d0640 2024-05-28T06:05:46.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.609+0000 7fc33fa00640 1 -- 172.21.15.96:0/3747925564 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc33815d0d0 con 0x7fc3380d0640 2024-05-28T06:05:46.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.610+0000 7fc33fa00640 1 -- 172.21.15.96:0/3747925564 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3380689e0 con 0x7fc3380d0640 2024-05-28T06:05:46.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.611+0000 7fc335600640 1 -- 172.21.15.96:0/3747925564 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc320012b70 con 0x7fc3380d0640 2024-05-28T06:05:46.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.611+0000 7fc335600640 1 --2- 172.21.15.96:0/3747925564 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc31406c8f0 0x7fc31406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:05:46.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.612+0000 7fc335600640 1 -- 172.21.15.96:0/3747925564 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7fc32008bef0 con 0x7fc3380d0640 2024-05-28T06:05:46.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.612+0000 7fc33d000640 1 --2- 172.21.15.96:0/3747925564 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc31406c8f0 0x7fc31406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:05:46.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.613+0000 7fc33d000640 1 --2- 172.21.15.96:0/3747925564 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc31406c8f0 0x7fc31406edb0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fc328037000 tx=0x7fc328008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:05:46.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.616+0000 7fc335600640 1 -- 172.21.15.96:0/3747925564 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc32005a3b0 con 0x7fc3380d0640 2024-05-28T06:05:46.847 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.847+0000 7fc33fa00640 1 -- 172.21.15.96:0/3747925564 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) -- 0x7fc3380af8a0 con 0x7fc31406c8f0 2024-05-28T06:05:46.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.859+0000 7fc335600640 1 -- 172.21.15.96:0/3747925564 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+59 (secure 0 0 0) 0x7fc3380af8a0 con 0x7fc31406c8f0 2024-05-28T06:05:46.859 INFO:teuthology.orchestra.run.smithi096.stdout:/volumes/_nogroup/sv1/72f8c8b7-4e81-4cb9-9368-010031a3b109 2024-05-28T06:05:46.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.861+0000 7fc33fa00640 1 -- 172.21.15.96:0/3747925564 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc31406c8f0 msgr2=0x7fc31406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:46.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.861+0000 7fc33fa00640 1 --2- 172.21.15.96:0/3747925564 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc31406c8f0 0x7fc31406edb0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fc328037000 tx=0x7fc328008040 comp rx=0 tx=0).stop 2024-05-28T06:05:46.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.861+0000 7fc33fa00640 1 -- 172.21.15.96:0/3747925564 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc3380d0640 msgr2=0x7fc338157600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:05:46.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.861+0000 7fc33fa00640 1 --2- 172.21.15.96:0/3747925564 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc3380d0640 0x7fc338157600 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fc32000d5f0 tx=0x7fc32000dab0 comp rx=0 tx=0).stop 2024-05-28T06:05:46.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.862+0000 7fc33fa00640 1 -- 172.21.15.96:0/3747925564 shutdown_connections 2024-05-28T06:05:46.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.862+0000 7fc33fa00640 1 --2- 172.21.15.96:0/3747925564 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc31406c8f0 0x7fc31406edb0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:46.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.862+0000 7fc33fa00640 1 --2- 172.21.15.96:0/3747925564 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc33815b6f0 0x7fc338157b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:46.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.862+0000 7fc33fa00640 1 --2- 172.21.15.96:0/3747925564 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc3380d0640 0x7fc338157600 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:46.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.862+0000 7fc33fa00640 1 --2- 172.21.15.96:0/3747925564 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3380725e0 0x7fc33815a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:05:46.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.862+0000 7fc33fa00640 1 -- 172.21.15.96:0/3747925564 >> 172.21.15.96:0/3747925564 conn(0x7fc33806bd20 msgr2=0x7fc338072c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:05:46.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.862+0000 7fc33fa00640 1 -- 172.21.15.96:0/3747925564 shutdown_connections 2024-05-28T06:05:46.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:05:46.862+0000 7fc33fa00640 1 -- 172.21.15.96:0/3747925564 wait complete. 2024-05-28T06:05:46.882 DEBUG:tasks.cephfs.test_volumes:filling subvolume sv1 with 10 files each 1024MB size under directory volumes/_nogroup/sv1/72f8c8b7-4e81-4cb9-9368-010031a3b109 2024-05-28T06:05:46.882 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:05:46.882 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:05:46.933 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:05:46.933 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/72f8c8b7-4e81-4cb9-9368-010031a3b109/subvolume_file.0 bs=1M conv=fdatasync count=1024 seek=0) 2024-05-28T06:05:59.148 DEBUG:teuthology.orchestra.run.smithi096:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:05:59.156 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:06:01.878 INFO:teuthology.orchestra.run.smithi096.stderr:1024+0 records in 2024-05-28T06:06:01.879 INFO:teuthology.orchestra.run.smithi096.stderr:1024+0 records out 2024-05-28T06:06:01.879 INFO:teuthology.orchestra.run.smithi096.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 14.8599 s, 72.3 MB/s 2024-05-28T06:06:01.881 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:06:01.881 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:06:01.933 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:06:01.933 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/72f8c8b7-4e81-4cb9-9368-010031a3b109/subvolume_file.1 bs=1M conv=fdatasync count=1024 seek=0) 2024-05-28T06:06:13.943 INFO:teuthology.orchestra.run.smithi096.stderr:1024+0 records in 2024-05-28T06:06:13.943 INFO:teuthology.orchestra.run.smithi096.stderr:1024+0 records out 2024-05-28T06:06:13.943 INFO:teuthology.orchestra.run.smithi096.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 11.9063 s, 90.2 MB/s 2024-05-28T06:06:13.945 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:06:13.945 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:06:13.998 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:06:13.999 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/72f8c8b7-4e81-4cb9-9368-010031a3b109/subvolume_file.2 bs=1M conv=fdatasync count=1024 seek=0) 2024-05-28T06:06:25.787 INFO:teuthology.orchestra.run.smithi096.stderr:1024+0 records in 2024-05-28T06:06:25.787 INFO:teuthology.orchestra.run.smithi096.stderr:1024+0 records out 2024-05-28T06:06:25.787 INFO:teuthology.orchestra.run.smithi096.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 11.7239 s, 91.6 MB/s 2024-05-28T06:06:25.789 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:06:25.789 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:06:25.841 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:06:25.841 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/72f8c8b7-4e81-4cb9-9368-010031a3b109/subvolume_file.3 bs=1M conv=fdatasync count=1024 seek=0) 2024-05-28T06:06:29.277 DEBUG:teuthology.orchestra.run.smithi096:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:06:29.285 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:06:36.727 INFO:teuthology.orchestra.run.smithi096.stderr:1024+0 records in 2024-05-28T06:06:36.727 INFO:teuthology.orchestra.run.smithi096.stderr:1024+0 records out 2024-05-28T06:06:36.727 INFO:teuthology.orchestra.run.smithi096.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 10.7792 s, 99.6 MB/s 2024-05-28T06:06:36.728 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:06:36.728 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:06:36.779 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:06:36.779 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/72f8c8b7-4e81-4cb9-9368-010031a3b109/subvolume_file.4 bs=1M conv=fdatasync count=1024 seek=0) 2024-05-28T06:06:48.208 INFO:teuthology.orchestra.run.smithi096.stderr:1024+0 records in 2024-05-28T06:06:48.208 INFO:teuthology.orchestra.run.smithi096.stderr:1024+0 records out 2024-05-28T06:06:48.208 INFO:teuthology.orchestra.run.smithi096.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 11.3258 s, 94.8 MB/s 2024-05-28T06:06:48.209 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:06:48.209 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:06:48.257 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:06:48.257 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/72f8c8b7-4e81-4cb9-9368-010031a3b109/subvolume_file.5 bs=1M conv=fdatasync count=1024 seek=0) 2024-05-28T06:06:59.411 DEBUG:teuthology.orchestra.run.smithi096:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:06:59.420 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:06:59.511 INFO:teuthology.orchestra.run.smithi096.stderr:1024+0 records in 2024-05-28T06:06:59.511 INFO:teuthology.orchestra.run.smithi096.stderr:1024+0 records out 2024-05-28T06:06:59.511 INFO:teuthology.orchestra.run.smithi096.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 11.1502 s, 96.3 MB/s 2024-05-28T06:06:59.512 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:06:59.512 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:06:59.555 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:06:59.555 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/72f8c8b7-4e81-4cb9-9368-010031a3b109/subvolume_file.6 bs=1M conv=fdatasync count=1024 seek=0) 2024-05-28T06:07:11.126 INFO:teuthology.orchestra.run.smithi096.stderr:1024+0 records in 2024-05-28T06:07:11.126 INFO:teuthology.orchestra.run.smithi096.stderr:1024+0 records out 2024-05-28T06:07:11.127 INFO:teuthology.orchestra.run.smithi096.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 11.4639 s, 93.7 MB/s 2024-05-28T06:07:11.128 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:07:11.128 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:07:11.179 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:07:11.179 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/72f8c8b7-4e81-4cb9-9368-010031a3b109/subvolume_file.7 bs=1M conv=fdatasync count=1024 seek=0) 2024-05-28T06:07:23.069 INFO:teuthology.orchestra.run.smithi096.stderr:1024+0 records in 2024-05-28T06:07:23.069 INFO:teuthology.orchestra.run.smithi096.stderr:1024+0 records out 2024-05-28T06:07:23.070 INFO:teuthology.orchestra.run.smithi096.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 11.7843 s, 91.1 MB/s 2024-05-28T06:07:23.071 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:07:23.071 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:07:23.123 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:07:23.123 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/72f8c8b7-4e81-4cb9-9368-010031a3b109/subvolume_file.8 bs=1M conv=fdatasync count=1024 seek=0) 2024-05-28T06:07:29.536 DEBUG:teuthology.orchestra.run.smithi096:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:07:29.542 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:07:34.585 INFO:teuthology.orchestra.run.smithi096.stderr:1024+0 records in 2024-05-28T06:07:34.585 INFO:teuthology.orchestra.run.smithi096.stderr:1024+0 records out 2024-05-28T06:07:34.585 INFO:teuthology.orchestra.run.smithi096.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 11.4004 s, 94.2 MB/s 2024-05-28T06:07:34.587 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:07:34.587 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:07:34.637 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:07:34.637 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/72f8c8b7-4e81-4cb9-9368-010031a3b109/subvolume_file.9 bs=1M conv=fdatasync count=1024 seek=0) 2024-05-28T06:07:45.582 INFO:teuthology.orchestra.run.smithi096.stderr:1024+0 records in 2024-05-28T06:07:45.583 INFO:teuthology.orchestra.run.smithi096.stderr:1024+0 records out 2024-05-28T06:07:45.583 INFO:teuthology.orchestra.run.smithi096.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 10.8875 s, 98.6 MB/s 2024-05-28T06:07:45.585 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot create cephfs sv1 ss1 2024-05-28T06:07:45.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:45.788+0000 7fb585200640 1 Processor -- start 2024-05-28T06:07:45.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:45.788+0000 7fb585200640 1 -- start start 2024-05-28T06:07:45.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:45.790+0000 7fb585200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fb58015fa00 0x7fb58006e970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:07:45.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:45.790+0000 7fb585200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb58006fa00 con 0x7fb5800d0b40 2024-05-28T06:07:45.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:45.790+0000 7fb585200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb58006eeb0 con 0x7fb5801600f0 2024-05-28T06:07:45.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:45.790+0000 7fb585200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fb58006f160 con 0x7fb58015fa00 2024-05-28T06:07:45.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:45.790+0000 7fb57fe00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fb5800d0b40 0x7fb580072190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:49312/0 (socket says 172.21.15.96:49312) 2024-05-28T06:07:45.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:45.790+0000 7fb57fe00640 1 -- 172.21.15.96:0/2415595714 learned_addr learned my addr 172.21.15.96:0/2415595714 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:07:45.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:45.791+0000 7fb57f400640 1 --2- 172.21.15.96:0/2415595714 >> v2:172.21.15.123:3300/0 conn(0x7fb58015fa00 0x7fb58006e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:07:45.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:45.791+0000 7fb57e000640 1 -- 172.21.15.96:0/2415595714 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3214166151 0 0) 0x7fb58006fa00 con 0x7fb5800d0b40 2024-05-28T06:07:45.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:45.791+0000 7fb57e000640 1 -- 172.21.15.96:0/2415595714 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb564003520 con 0x7fb5800d0b40 2024-05-28T06:07:45.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:45.791+0000 7fb57f400640 1 -- 172.21.15.96:0/2415595714 >> v1:172.21.15.123:6789/0 conn(0x7fb5801600f0 legacy=0x7fb5800d0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:07:45.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:45.791+0000 7fb57f400640 1 -- 172.21.15.96:0/2415595714 >> v1:172.21.15.96:6789/0 conn(0x7fb5800d0b40 legacy=0x7fb580072190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:07:45.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:45.791+0000 7fb57e000640 1 -- 172.21.15.96:0/2415595714 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1142731786 0 0) 0x7fb564003520 con 0x7fb5800d0b40 2024-05-28T06:07:45.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:45.791+0000 7fb57f400640 1 -- 172.21.15.96:0/2415595714 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb58006f440 con 0x7fb58015fa00 2024-05-28T06:07:45.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:45.793+0000 7fb57f400640 1 --2- 172.21.15.96:0/2415595714 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb58015fa00 0x7fb58006e970 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fb5700097a0 tx=0x7fb57002a0a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b618c924521a9a07 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:07:45.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:45.793+0000 7fb57e000640 1 -- 172.21.15.96:0/2415595714 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb570037070 con 0x7fb58015fa00 2024-05-28T06:07:45.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:45.794+0000 7fb57e000640 1 -- 172.21.15.96:0/2415595714 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb57002ab30 con 0x7fb58015fa00 2024-05-28T06:07:45.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:45.794+0000 7fb57e000640 1 -- 172.21.15.96:0/2415595714 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb578001a10 con 0x7fb58015fa00 2024-05-28T06:07:45.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:45.794+0000 7fb585200640 1 -- 172.21.15.96:0/2415595714 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb58015fa00 msgr2=0x7fb58006e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:07:45.797 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:45.794+0000 7fb585200640 1 --2- 172.21.15.96:0/2415595714 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb58015fa00 0x7fb58006e970 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fb5700097a0 tx=0x7fb57002a0a0 comp rx=0 tx=0).stop 2024-05-28T06:07:45.797 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:45.794+0000 7fb585200640 1 -- 172.21.15.96:0/2415595714 shutdown_connections 2024-05-28T06:07:45.797 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:45.794+0000 7fb585200640 1 --2- 172.21.15.96:0/2415595714 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb58015fa00 0x7fb58006e970 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:45.797 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:45.794+0000 7fb585200640 1 -- 172.21.15.96:0/2415595714 >> 172.21.15.96:0/2415595714 conn(0x7fb58006bae0 msgr2=0x7fb58006bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:07:45.797 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:45.794+0000 7fb585200640 1 -- 172.21.15.96:0/2415595714 shutdown_connections 2024-05-28T06:07:45.798 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:45.795+0000 7fb585200640 1 -- 172.21.15.96:0/2415595714 wait complete. 2024-05-28T06:07:45.798 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:45.797+0000 7fb585200640 1 Processor -- start 2024-05-28T06:07:45.798 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:45.797+0000 7fb585200640 1 -- start start 2024-05-28T06:07:45.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:45.798+0000 7fb585200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb5800d0b40 0x7fb58014d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:07:45.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:45.798+0000 7fb585200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb58015fa00 0x7fb580159f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:07:45.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:45.798+0000 7fb585200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb5801600f0 0x7fb58015da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:07:45.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:45.798+0000 7fb585200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb58015fde0 con 0x7fb5800d0b40 2024-05-28T06:07:45.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:45.798+0000 7fb585200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fb580071c70 con 0x7fb5801600f0 2024-05-28T06:07:45.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:45.798+0000 7fb585200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fb5800728a0 con 0x7fb58015fa00 2024-05-28T06:07:45.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:45.798+0000 7fb57f400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb5800d0b40 0x7fb58014d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:07:45.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:45.798+0000 7fb57f400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb5800d0b40 0x7fb58014d8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:34396/0 (socket says 172.21.15.96:34396) 2024-05-28T06:07:45.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:45.798+0000 7fb57f400640 1 -- 172.21.15.96:0/3227872740 learned_addr learned my addr 172.21.15.96:0/3227872740 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:07:45.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:45.799+0000 7fb57ea00640 1 --2- 172.21.15.96:0/3227872740 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb58015fa00 0x7fb580159f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:07:45.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:45.799+0000 7fb57f400640 1 -- 172.21.15.96:0/3227872740 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb58015fa00 msgr2=0x7fb580159f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:07:45.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:45.799+0000 7fb57f400640 1 --2- 172.21.15.96:0/3227872740 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb58015fa00 0x7fb580159f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:45.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:45.799+0000 7fb57f400640 1 -- 172.21.15.96:0/3227872740 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb5801600f0 msgr2=0x7fb58015da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:07:45.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:45.799+0000 7fb57f400640 1 --2- 172.21.15.96:0/3227872740 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb5801600f0 0x7fb58015da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:45.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:45.799+0000 7fb57f400640 1 -- 172.21.15.96:0/3227872740 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5801f7fc0 con 0x7fb5800d0b40 2024-05-28T06:07:45.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:45.799+0000 7fb57ea00640 1 --2- 172.21.15.96:0/3227872740 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb58015fa00 0x7fb580159f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:07:45.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:45.799+0000 7fb57f400640 1 --2- 172.21.15.96:0/3227872740 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb5800d0b40 0x7fb58014d8e0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fb570035040 tx=0x7fb5700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:07:45.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:45.799+0000 7fb577400640 1 -- 172.21.15.96:0/3227872740 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb570041020 con 0x7fb5800d0b40 2024-05-28T06:07:45.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:45.799+0000 7fb577400640 1 -- 172.21.15.96:0/3227872740 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb57003d6e0 con 0x7fb5800d0b40 2024-05-28T06:07:45.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:45.799+0000 7fb577400640 1 -- 172.21.15.96:0/3227872740 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb570037040 con 0x7fb5800d0b40 2024-05-28T06:07:45.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:45.799+0000 7fb585200640 1 -- 172.21.15.96:0/3227872740 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5801f82a0 con 0x7fb5800d0b40 2024-05-28T06:07:45.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:45.800+0000 7fb585200640 1 -- 172.21.15.96:0/3227872740 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5801f8860 con 0x7fb5800d0b40 2024-05-28T06:07:45.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:45.801+0000 7fb585200640 1 -- 172.21.15.96:0/3227872740 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5800689e0 con 0x7fb5800d0b40 2024-05-28T06:07:45.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:45.802+0000 7fb577400640 1 -- 172.21.15.96:0/3227872740 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 7) ==== 91040+0+0 (secure 0 0 0) 0x7fb570044020 con 0x7fb5800d0b40 2024-05-28T06:07:45.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:45.803+0000 7fb577400640 1 --2- 172.21.15.96:0/3227872740 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb56006c8f0 0x7fb56006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:07:45.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:45.803+0000 7fb577400640 1 -- 172.21.15.96:0/3227872740 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7fb5700ac280 con 0x7fb5800d0b40 2024-05-28T06:07:45.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:45.803+0000 7fb57ea00640 1 --2- 172.21.15.96:0/3227872740 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb56006c8f0 0x7fb56006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:07:45.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:45.804+0000 7fb57ea00640 1 --2- 172.21.15.96:0/3227872740 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb56006c8f0 0x7fb56006edb0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fb58015aaa0 tx=0x7fb56c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:07:45.810 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:45.809+0000 7fb577400640 1 -- 172.21.15.96:0/3227872740 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb57004b030 con 0x7fb5800d0b40 2024-05-28T06:07:46.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.041+0000 7fb585200640 1 -- 172.21.15.96:0/3227872740 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""]}) -- 0x7fb5800ae0e0 con 0x7fb56006c8f0 2024-05-28T06:07:46.054 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:07:46.054+0000 7fea59e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:07:46.054 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:07:46.054+0000 7fea59e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:07:46.054 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:07:46.054+0000 7fea59e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:07:46.054 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:07:46.054+0000 7fea59e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:07:46.055 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:07:46.054+0000 7fea59e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:07:46.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.087+0000 7fb577400640 1 -- 172.21.15.96:0/3227872740 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fb5800ae0e0 con 0x7fb56006c8f0 2024-05-28T06:07:46.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.089+0000 7fb585200640 1 -- 172.21.15.96:0/3227872740 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb56006c8f0 msgr2=0x7fb56006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:07:46.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.089+0000 7fb585200640 1 --2- 172.21.15.96:0/3227872740 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb56006c8f0 0x7fb56006edb0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fb58015aaa0 tx=0x7fb56c009040 comp rx=0 tx=0).stop 2024-05-28T06:07:46.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.089+0000 7fb585200640 1 -- 172.21.15.96:0/3227872740 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb5800d0b40 msgr2=0x7fb58014d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:07:46.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.089+0000 7fb585200640 1 --2- 172.21.15.96:0/3227872740 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb5800d0b40 0x7fb58014d8e0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fb570035040 tx=0x7fb5700046c0 comp rx=0 tx=0).stop 2024-05-28T06:07:46.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.089+0000 7fb585200640 1 -- 172.21.15.96:0/3227872740 shutdown_connections 2024-05-28T06:07:46.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.090+0000 7fb585200640 1 --2- 172.21.15.96:0/3227872740 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb56006c8f0 0x7fb56006edb0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:46.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.090+0000 7fb585200640 1 --2- 172.21.15.96:0/3227872740 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb5801600f0 0x7fb58015da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:46.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.090+0000 7fb585200640 1 --2- 172.21.15.96:0/3227872740 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb58015fa00 0x7fb580159f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:46.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.090+0000 7fb585200640 1 --2- 172.21.15.96:0/3227872740 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb5800d0b40 0x7fb58014d8e0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:46.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.090+0000 7fb585200640 1 -- 172.21.15.96:0/3227872740 >> 172.21.15.96:0/3227872740 conn(0x7fb58006bae0 msgr2=0x7fb5800d0400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:07:46.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.090+0000 7fb585200640 1 -- 172.21.15.96:0/3227872740 shutdown_connections 2024-05-28T06:07:46.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.090+0000 7fb585200640 1 -- 172.21.15.96:0/3227872740 wait complete. 2024-05-28T06:07:46.109 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-05-28T06:07:46.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.311+0000 7feed3400640 1 Processor -- start 2024-05-28T06:07:46.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.312+0000 7feed3400640 1 -- start start 2024-05-28T06:07:46.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.313+0000 7feed3400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7feecc15b6f0 0x7feecc06e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:07:46.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.313+0000 7feed3400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7feecc0725e0 0x7feecc0ae650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:07:46.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.313+0000 7feed3400640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feecc0ade90 con 0x7feecc0d0640 2024-05-28T06:07:46.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.313+0000 7feed3400640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7feecc071c40 con 0x7feecc15b6f0 2024-05-28T06:07:46.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.313+0000 7feed3400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7feecc071e70 con 0x7feecc0725e0 2024-05-28T06:07:46.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.314+0000 7feed1400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7feecc0725e0 0x7feecc0ae650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:07:46.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.314+0000 7feec3e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7feecc15b6f0 0x7feecc06e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:07:46.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.314+0000 7feed1400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7feecc0725e0 0x7feecc0ae650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.96:35588/0 (socket says 172.21.15.96:35588) 2024-05-28T06:07:46.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.314+0000 7feed1400640 1 -- 172.21.15.96:0/1696217073 learned_addr learned my addr 172.21.15.96:0/1696217073 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:07:46.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.315+0000 7feec3e00640 1 -- 172.21.15.96:0/1696217073 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7feecc0725e0 msgr2=0x7feecc0ae650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:07:46.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.315+0000 7feec3e00640 1 --2- 172.21.15.96:0/1696217073 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7feecc0725e0 0x7feecc0ae650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:46.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.315+0000 7feec3e00640 1 -- 172.21.15.96:0/1696217073 >> v1:172.21.15.123:6789/0 conn(0x7feecc0d0640 legacy=0x7feecc0d0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:07:46.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.315+0000 7feec3e00640 1 -- 172.21.15.96:0/1696217073 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feecc072130 con 0x7feecc15b6f0 2024-05-28T06:07:46.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.316+0000 7feec3e00640 1 --2- 172.21.15.96:0/1696217073 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feecc15b6f0 0x7feecc06e350 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7feeb400dbc0 tx=0x7feeb402e390 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ef9f2cb137e54617 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:07:46.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.317+0000 7feecbe00640 1 -- 172.21.15.96:0/1696217073 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeb402ee10 con 0x7feecc15b6f0 2024-05-28T06:07:46.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.317+0000 7feecbe00640 1 -- 172.21.15.96:0/1696217073 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feeb400b450 con 0x7feecc15b6f0 2024-05-28T06:07:46.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.317+0000 7feecbe00640 1 -- 172.21.15.96:0/1696217073 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeb400b820 con 0x7feecc15b6f0 2024-05-28T06:07:46.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.317+0000 7feed3400640 1 -- 172.21.15.96:0/1696217073 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feecc15b6f0 msgr2=0x7feecc06e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:07:46.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.317+0000 7feed3400640 1 --2- 172.21.15.96:0/1696217073 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feecc15b6f0 0x7feecc06e350 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7feeb400dbc0 tx=0x7feeb402e390 comp rx=0 tx=0).stop 2024-05-28T06:07:46.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.318+0000 7feed3400640 1 -- 172.21.15.96:0/1696217073 shutdown_connections 2024-05-28T06:07:46.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.318+0000 7feed3400640 1 --2- 172.21.15.96:0/1696217073 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7feecc0725e0 0x7feecc0ae650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:46.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.318+0000 7feed3400640 1 --2- 172.21.15.96:0/1696217073 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feecc15b6f0 0x7feecc06e350 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:46.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.318+0000 7feed3400640 1 -- 172.21.15.96:0/1696217073 >> 172.21.15.96:0/1696217073 conn(0x7feecc06bd20 msgr2=0x7feecc06c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:07:46.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.318+0000 7feed3400640 1 -- 172.21.15.96:0/1696217073 shutdown_connections 2024-05-28T06:07:46.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.318+0000 7feed3400640 1 -- 172.21.15.96:0/1696217073 wait complete. 2024-05-28T06:07:46.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.319+0000 7feed3400640 1 Processor -- start 2024-05-28T06:07:46.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.319+0000 7feed3400640 1 -- start start 2024-05-28T06:07:46.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.320+0000 7feed3400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feecc0725e0 0x7feecc1f05c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:07:46.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.320+0000 7feed3400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feecc0d0640 0x7feecc0b3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:07:46.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.320+0000 7feed3400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7feecc15b6f0 0x7feecc141ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:07:46.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.320+0000 7feed3400640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7feecc0aeb90 con 0x7feecc0725e0 2024-05-28T06:07:46.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.320+0000 7feed3400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7feecc06e890 con 0x7feecc0d0640 2024-05-28T06:07:46.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.320+0000 7feed3400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7feecc0aed10 con 0x7feecc15b6f0 2024-05-28T06:07:46.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.321+0000 7feed0a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feecc0725e0 0x7feecc1f05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:07:46.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.321+0000 7feed0a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feecc0725e0 0x7feecc1f05c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:34404/0 (socket says 172.21.15.96:34404) 2024-05-28T06:07:46.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.321+0000 7feed0a00640 1 -- 172.21.15.96:0/1515894527 learned_addr learned my addr 172.21.15.96:0/1515894527 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:07:46.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.321+0000 7feec3e00640 1 --2- 172.21.15.96:0/1515894527 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feecc0d0640 0x7feecc0b3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:07:46.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.321+0000 7feed1400640 1 --2- 172.21.15.96:0/1515894527 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7feecc15b6f0 0x7feecc141ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:07:46.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.321+0000 7feed0a00640 1 -- 172.21.15.96:0/1515894527 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7feecc15b6f0 msgr2=0x7feecc141ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:07:46.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.321+0000 7feed0a00640 1 --2- 172.21.15.96:0/1515894527 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7feecc15b6f0 0x7feecc141ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:46.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.321+0000 7feed0a00640 1 -- 172.21.15.96:0/1515894527 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feecc0d0640 msgr2=0x7feecc0b3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:07:46.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.321+0000 7feed0a00640 1 --2- 172.21.15.96:0/1515894527 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feecc0d0640 0x7feecc0b3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:46.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.321+0000 7feed0a00640 1 -- 172.21.15.96:0/1515894527 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feecc1f37a0 con 0x7feecc0725e0 2024-05-28T06:07:46.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.322+0000 7feed0a00640 1 --2- 172.21.15.96:0/1515894527 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feecc0725e0 0x7feecc1f05c0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7feebc00c750 tx=0x7feebc00cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:07:46.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.322+0000 7feec9600640 1 -- 172.21.15.96:0/1515894527 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feebc017070 con 0x7feecc0725e0 2024-05-28T06:07:46.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.322+0000 7feec9600640 1 -- 172.21.15.96:0/1515894527 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feebc005270 con 0x7feecc0725e0 2024-05-28T06:07:46.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.322+0000 7feec9600640 1 -- 172.21.15.96:0/1515894527 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feebc00a640 con 0x7feecc0725e0 2024-05-28T06:07:46.325 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.322+0000 7feed3400640 1 -- 172.21.15.96:0/1515894527 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feecc1f3a80 con 0x7feecc0725e0 2024-05-28T06:07:46.325 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.322+0000 7feed3400640 1 -- 172.21.15.96:0/1515894527 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feecc1f4040 con 0x7feecc0725e0 2024-05-28T06:07:46.325 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.323+0000 7feed3400640 1 -- 172.21.15.96:0/1515894527 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feecc0689e0 con 0x7feecc0725e0 2024-05-28T06:07:46.325 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.324+0000 7feec9600640 1 -- 172.21.15.96:0/1515894527 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 7) ==== 91040+0+0 (secure 0 0 0) 0x7feebc01a070 con 0x7feecc0725e0 2024-05-28T06:07:46.327 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.326+0000 7feec9600640 1 --2- 172.21.15.96:0/1515894527 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7feea806c8f0 0x7feea806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:07:46.327 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.326+0000 7feec9600640 1 -- 172.21.15.96:0/1515894527 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7feebc08b090 con 0x7feecc0725e0 2024-05-28T06:07:46.327 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.326+0000 7feec3e00640 1 --2- 172.21.15.96:0/1515894527 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7feea806c8f0 0x7feea806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:07:46.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.327+0000 7feec3e00640 1 --2- 172.21.15.96:0/1515894527 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7feea806c8f0 0x7feea806edb0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7feeb4037000 tx=0x7feeb4032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:07:46.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.331+0000 7feec9600640 1 -- 172.21.15.96:0/1515894527 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7feebc0594c0 con 0x7feecc0725e0 2024-05-28T06:07:46.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.567+0000 7feed3400640 1 -- 172.21.15.96:0/1515894527 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) -- 0x7feecc06ee30 con 0x7feea806c8f0 2024-05-28T06:07:46.586 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.585+0000 7feec9600640 1 -- 172.21.15.96:0/1515894527 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+59 (secure 0 0 0) 0x7feecc06ee30 con 0x7feea806c8f0 2024-05-28T06:07:46.587 INFO:teuthology.orchestra.run.smithi096.stdout:/volumes/_nogroup/sv1/72f8c8b7-4e81-4cb9-9368-010031a3b109 2024-05-28T06:07:46.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.587+0000 7feed3400640 1 -- 172.21.15.96:0/1515894527 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7feea806c8f0 msgr2=0x7feea806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:07:46.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.587+0000 7feed3400640 1 --2- 172.21.15.96:0/1515894527 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7feea806c8f0 0x7feea806edb0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7feeb4037000 tx=0x7feeb4032000 comp rx=0 tx=0).stop 2024-05-28T06:07:46.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.587+0000 7feed3400640 1 -- 172.21.15.96:0/1515894527 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feecc0725e0 msgr2=0x7feecc1f05c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:07:46.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.587+0000 7feed3400640 1 --2- 172.21.15.96:0/1515894527 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feecc0725e0 0x7feecc1f05c0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7feebc00c750 tx=0x7feebc00cc10 comp rx=0 tx=0).stop 2024-05-28T06:07:46.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.587+0000 7feed3400640 1 -- 172.21.15.96:0/1515894527 shutdown_connections 2024-05-28T06:07:46.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.587+0000 7feed3400640 1 --2- 172.21.15.96:0/1515894527 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7feea806c8f0 0x7feea806edb0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:46.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.587+0000 7feed3400640 1 --2- 172.21.15.96:0/1515894527 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7feecc15b6f0 0x7feecc141ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:46.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.587+0000 7feed3400640 1 --2- 172.21.15.96:0/1515894527 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feecc0d0640 0x7feecc0b3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:46.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.588+0000 7feed3400640 1 --2- 172.21.15.96:0/1515894527 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feecc0725e0 0x7feecc1f05c0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:46.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.588+0000 7feed3400640 1 -- 172.21.15.96:0/1515894527 >> 172.21.15.96:0/1515894527 conn(0x7feecc06bd20 msgr2=0x7feecc06f450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:07:46.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.588+0000 7feed3400640 1 -- 172.21.15.96:0/1515894527 shutdown_connections 2024-05-28T06:07:46.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.588+0000 7feed3400640 1 -- 172.21.15.96:0/1515894527 wait complete. 2024-05-28T06:07:46.609 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:07:46.609 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/72f8c8b7-4e81-4cb9-9368-010031a3b109 2024-05-28T06:07:46.609 DEBUG:teuthology.orchestra.run.smithi096:> ) 2024-05-28T06:07:46.670 INFO:teuthology.orchestra.run.smithi096.stdout:# file: volumes/_nogroup/sv1/72f8c8b7-4e81-4cb9-9368-010031a3b109 2024-05-28T06:07:46.670 INFO:teuthology.orchestra.run.smithi096.stdout:ceph.dir.rbytes="10737418240" 2024-05-28T06:07:46.670 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:07:46.671 INFO:tasks.cephfs.test_volumes:r_size = 10737418240 exp_size = 10737418240 2024-05-28T06:07:46.672 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 ss1clone1 2024-05-28T06:07:46.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.883+0000 7f0598200640 1 Processor -- start 2024-05-28T06:07:46.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.884+0000 7f0598200640 1 -- start start 2024-05-28T06:07:46.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.885+0000 7f0598200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f05900cd550 0x7f059006b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:07:46.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.885+0000 7f0598200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f05900cb9a0 0x7f05900cbe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:07:46.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.885+0000 7f0598200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f059006ef80 con 0x7f05900ccd20 2024-05-28T06:07:46.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.885+0000 7f0598200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f059006c660 con 0x7f05900cd550 2024-05-28T06:07:46.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.885+0000 7f0598200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f059006f2b0 con 0x7f05900cb9a0 2024-05-28T06:07:46.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.886+0000 7f0594e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f05900cd550 0x7f059006b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:07:46.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.886+0000 7f0596200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f05900cb9a0 0x7f05900cbe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:07:46.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.886+0000 7f0594e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f05900cd550 0x7f059006b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.96:40718/0 (socket says 172.21.15.96:40718) 2024-05-28T06:07:46.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.886+0000 7f0594e00640 1 -- 172.21.15.96:0/1514198828 learned_addr learned my addr 172.21.15.96:0/1514198828 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:07:46.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.886+0000 7f0594e00640 1 -- 172.21.15.96:0/1514198828 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f05900cb9a0 msgr2=0x7f05900cbe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:07:46.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.886+0000 7f0594e00640 1 --2- 172.21.15.96:0/1514198828 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f05900cb9a0 0x7f05900cbe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:46.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.887+0000 7f0594e00640 1 -- 172.21.15.96:0/1514198828 >> v1:172.21.15.123:6789/0 conn(0x7f05900ccd20 legacy=0x7f059006f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:07:46.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.887+0000 7f0594e00640 1 -- 172.21.15.96:0/1514198828 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05900cd100 con 0x7f05900cd550 2024-05-28T06:07:46.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.888+0000 7f0594e00640 1 --2- 172.21.15.96:0/1514198828 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f05900cd550 0x7f059006b900 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f057c009e50 tx=0x7f057c02a680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9a104df9609fa032 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:07:46.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.888+0000 7f058fe00640 1 -- 172.21.15.96:0/1514198828 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f057c036070 con 0x7f05900cd550 2024-05-28T06:07:46.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.888+0000 7f058fe00640 1 -- 172.21.15.96:0/1514198828 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f057c031600 con 0x7f05900cd550 2024-05-28T06:07:46.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.888+0000 7f058fe00640 1 -- 172.21.15.96:0/1514198828 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f057c0319d0 con 0x7f05900cd550 2024-05-28T06:07:46.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.889+0000 7f0598200640 1 -- 172.21.15.96:0/1514198828 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f05900cd550 msgr2=0x7f059006b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:07:46.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.889+0000 7f0598200640 1 --2- 172.21.15.96:0/1514198828 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f05900cd550 0x7f059006b900 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f057c009e50 tx=0x7f057c02a680 comp rx=0 tx=0).stop 2024-05-28T06:07:46.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.889+0000 7f0598200640 1 -- 172.21.15.96:0/1514198828 shutdown_connections 2024-05-28T06:07:46.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.889+0000 7f0598200640 1 --2- 172.21.15.96:0/1514198828 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f05900cb9a0 0x7f05900cbe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:46.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.889+0000 7f0598200640 1 --2- 172.21.15.96:0/1514198828 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f05900cd550 0x7f059006b900 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:46.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.889+0000 7f0598200640 1 -- 172.21.15.96:0/1514198828 >> 172.21.15.96:0/1514198828 conn(0x7f05900b2f40 msgr2=0x7f05900b3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:07:46.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.889+0000 7f0598200640 1 -- 172.21.15.96:0/1514198828 shutdown_connections 2024-05-28T06:07:46.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.889+0000 7f0598200640 1 -- 172.21.15.96:0/1514198828 wait complete. 2024-05-28T06:07:46.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.891+0000 7f0598200640 1 Processor -- start 2024-05-28T06:07:46.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.891+0000 7f0598200640 1 -- start start 2024-05-28T06:07:46.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.892+0000 7f0598200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f05900cb9a0 0x7f05900cad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:07:46.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.892+0000 7f0598200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f05900ccd20 0x7f0590152f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:07:46.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.892+0000 7f0598200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f05900cd550 0x7f0590156a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:07:46.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.892+0000 7f0598200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f059006be40 con 0x7f05900cb9a0 2024-05-28T06:07:46.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.892+0000 7f0598200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f05900cf140 con 0x7f05900ccd20 2024-05-28T06:07:46.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.892+0000 7f0598200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f059006c380 con 0x7f05900cd550 2024-05-28T06:07:46.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.893+0000 7f0595800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f05900cb9a0 0x7f05900cad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:07:46.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.893+0000 7f0595800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f05900cb9a0 0x7f05900cad80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:34408/0 (socket says 172.21.15.96:34408) 2024-05-28T06:07:46.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.893+0000 7f0595800640 1 -- 172.21.15.96:0/2215055805 learned_addr learned my addr 172.21.15.96:0/2215055805 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:07:46.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.893+0000 7f0596200640 1 --2- 172.21.15.96:0/2215055805 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f05900cd550 0x7f0590156a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:07:46.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.893+0000 7f0594e00640 1 --2- 172.21.15.96:0/2215055805 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f05900ccd20 0x7f0590152f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:07:46.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.893+0000 7f0595800640 1 -- 172.21.15.96:0/2215055805 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f05900cd550 msgr2=0x7f0590156a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:07:46.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.893+0000 7f0595800640 1 --2- 172.21.15.96:0/2215055805 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f05900cd550 0x7f0590156a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:46.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.893+0000 7f0595800640 1 -- 172.21.15.96:0/2215055805 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f05900ccd20 msgr2=0x7f0590152f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:07:46.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.893+0000 7f0595800640 1 --2- 172.21.15.96:0/2215055805 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f05900ccd20 0x7f0590152f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:46.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.893+0000 7f0595800640 1 -- 172.21.15.96:0/2215055805 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05900b4c70 con 0x7f05900cb9a0 2024-05-28T06:07:46.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.893+0000 7f0594e00640 1 --2- 172.21.15.96:0/2215055805 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f05900ccd20 0x7f0590152f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:07:46.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.893+0000 7f0596200640 1 --2- 172.21.15.96:0/2215055805 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f05900cd550 0x7f0590156a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:07:46.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.893+0000 7f0595800640 1 --2- 172.21.15.96:0/2215055805 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f05900cb9a0 0x7f05900cad80 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f058000c460 tx=0x7f058000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:07:46.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.894+0000 7f058d600640 1 -- 172.21.15.96:0/2215055805 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05800052d0 con 0x7f05900cb9a0 2024-05-28T06:07:46.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.894+0000 7f058d600640 1 -- 172.21.15.96:0/2215055805 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0580012630 con 0x7f05900cb9a0 2024-05-28T06:07:46.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.894+0000 7f0598200640 1 -- 172.21.15.96:0/2215055805 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05900b4f50 con 0x7f05900cb9a0 2024-05-28T06:07:46.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.894+0000 7f058d600640 1 -- 172.21.15.96:0/2215055805 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05800054b0 con 0x7f05900cb9a0 2024-05-28T06:07:46.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.894+0000 7f0598200640 1 -- 172.21.15.96:0/2215055805 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05900b5510 con 0x7f05900cb9a0 2024-05-28T06:07:46.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.895+0000 7f0598200640 1 -- 172.21.15.96:0/2215055805 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05900689e0 con 0x7f05900cb9a0 2024-05-28T06:07:46.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.896+0000 7f058d600640 1 -- 172.21.15.96:0/2215055805 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 7) ==== 91040+0+0 (secure 0 0 0) 0x7f0580012a20 con 0x7f05900cb9a0 2024-05-28T06:07:46.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.897+0000 7f058d600640 1 --2- 172.21.15.96:0/2215055805 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f057006c8f0 0x7f057006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:07:46.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.898+0000 7f058d600640 1 -- 172.21.15.96:0/2215055805 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f058008b7b0 con 0x7f05900cb9a0 2024-05-28T06:07:46.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.898+0000 7f0594e00640 1 --2- 172.21.15.96:0/2215055805 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f057006c8f0 0x7f057006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:07:46.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.899+0000 7f0594e00640 1 --2- 172.21.15.96:0/2215055805 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f057006c8f0 0x7f057006edb0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f057c033000 tx=0x7f057c009000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:07:46.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:46.902+0000 7f058d600640 1 -- 172.21.15.96:0/2215055805 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0580059b40 con 0x7f05900cb9a0 2024-05-28T06:07:47.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.138+0000 7f0598200640 1 -- 172.21.15.96:0/2215055805 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "ss1clone1", "target": ["mon-mgr", ""]}) -- 0x7f059015ee30 con 0x7f057006c8f0 2024-05-28T06:07:47.367 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.367+0000 7f058d600640 1 -- 172.21.15.96:0/2215055805 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f059015ee30 con 0x7f057006c8f0 2024-05-28T06:07:47.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.370+0000 7f0598200640 1 -- 172.21.15.96:0/2215055805 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f057006c8f0 msgr2=0x7f057006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:07:47.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.370+0000 7f0598200640 1 --2- 172.21.15.96:0/2215055805 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f057006c8f0 0x7f057006edb0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f057c033000 tx=0x7f057c009000 comp rx=0 tx=0).stop 2024-05-28T06:07:47.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.370+0000 7f0598200640 1 -- 172.21.15.96:0/2215055805 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f05900cb9a0 msgr2=0x7f05900cad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:07:47.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.370+0000 7f0598200640 1 --2- 172.21.15.96:0/2215055805 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f05900cb9a0 0x7f05900cad80 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f058000c460 tx=0x7f058000c920 comp rx=0 tx=0).stop 2024-05-28T06:07:47.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.370+0000 7f0598200640 1 -- 172.21.15.96:0/2215055805 shutdown_connections 2024-05-28T06:07:47.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.370+0000 7f0598200640 1 --2- 172.21.15.96:0/2215055805 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f057006c8f0 0x7f057006edb0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:47.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.370+0000 7f0598200640 1 --2- 172.21.15.96:0/2215055805 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f05900cd550 0x7f0590156a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:47.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.370+0000 7f0598200640 1 --2- 172.21.15.96:0/2215055805 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f05900ccd20 0x7f0590152f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:47.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.370+0000 7f0598200640 1 --2- 172.21.15.96:0/2215055805 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f05900cb9a0 0x7f05900cad80 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:47.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.370+0000 7f0598200640 1 -- 172.21.15.96:0/2215055805 >> 172.21.15.96:0/2215055805 conn(0x7f05900b2f40 msgr2=0x7f05900d0660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:07:47.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.370+0000 7f0598200640 1 -- 172.21.15.96:0/2215055805 shutdown_connections 2024-05-28T06:07:47.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.370+0000 7f0598200640 1 -- 172.21.15.96:0/2215055805 wait complete. 2024-05-28T06:07:47.373 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:07:47.372+0000 7fea60c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:07:47.373 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:07:47.372+0000 7fea60c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:07:47.373 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:07:47.372+0000 7fea60c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:07:47.373 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:07:47.372+0000 7fea60c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:07:47.373 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:07:47.372+0000 7fea60c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:07:47.396 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-05-28T06:07:47.397 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:07:47.397+0000 7fea60200640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:07:47.397 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:07:47.397+0000 7fea60200640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:07:47.398 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:07:47.397+0000 7fea60200640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:07:47.398 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:07:47.397+0000 7fea60200640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:07:47.398 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:07:47.397+0000 7fea60200640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:07:47.595 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.593+0000 7fbc8d200640 1 Processor -- start 2024-05-28T06:07:47.595 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.594+0000 7fbc8d200640 1 -- start start 2024-05-28T06:07:47.596 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.595+0000 7fbc8d200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fbc880d4c40 0x7fbc880d2330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:07:47.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.595+0000 7fbc8d200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc880d3060 con 0x7fbc8815e690 2024-05-28T06:07:47.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.595+0000 7fbc8d200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc880d3340 con 0x7fbc880d1ca0 2024-05-28T06:07:47.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.595+0000 7fbc8d200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fbc880d3620 con 0x7fbc880d4c40 2024-05-28T06:07:47.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.595+0000 7fbc87400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fbc8815e690 0x7fbc880d4530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:49326/0 (socket says 172.21.15.96:49326) 2024-05-28T06:07:47.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.595+0000 7fbc87400640 1 -- 172.21.15.96:0/4279517757 learned_addr learned my addr 172.21.15.96:0/4279517757 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:07:47.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.596+0000 7fbc86000640 1 -- 172.21.15.96:0/4279517757 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2717565719 0 0) 0x7fbc880d3060 con 0x7fbc8815e690 2024-05-28T06:07:47.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.596+0000 7fbc86000640 1 -- 172.21.15.96:0/4279517757 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc6c003520 con 0x7fbc8815e690 2024-05-28T06:07:47.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.596+0000 7fbc86a00640 1 --2- 172.21.15.96:0/4279517757 >> v2:172.21.15.123:3300/0 conn(0x7fbc880d4c40 0x7fbc880d2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:07:47.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.596+0000 7fbc86000640 1 -- 172.21.15.96:0/4279517757 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3062440049 0 0) 0x7fbc6c003520 con 0x7fbc8815e690 2024-05-28T06:07:47.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.596+0000 7fbc86000640 1 -- 172.21.15.96:0/4279517757 >> v2:172.21.15.123:3300/0 conn(0x7fbc880d4c40 msgr2=0x7fbc880d2330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:07:47.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.596+0000 7fbc86000640 1 --2- 172.21.15.96:0/4279517757 >> v2:172.21.15.123:3300/0 conn(0x7fbc880d4c40 0x7fbc880d2330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:47.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.596+0000 7fbc86000640 1 -- 172.21.15.96:0/4279517757 >> v1:172.21.15.123:6789/0 conn(0x7fbc880d1ca0 legacy=0x7fbc880d2870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:07:47.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.596+0000 7fbc86000640 1 -- 172.21.15.96:0/4279517757 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc880cf770 con 0x7fbc8815e690 2024-05-28T06:07:47.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.596+0000 7fbc86000640 1 -- 172.21.15.96:0/4279517757 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7fbc78001670 con 0x7fbc8815e690 2024-05-28T06:07:47.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.596+0000 7fbc86000640 1 -- 172.21.15.96:0/4279517757 >> v1:172.21.15.96:6789/0 conn(0x7fbc8815e690 legacy=0x7fbc880d4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:07:47.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.596+0000 7fbc86000640 1 --2- 172.21.15.96:0/4279517757 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc6c0039b0 0x7fbc6c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:07:47.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.597+0000 7fbc86000640 1 --2- 172.21.15.96:0/4279517757 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc6c004800 0x7fbc6c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:07:47.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.597+0000 7fbc86000640 1 --2- 172.21.15.96:0/4279517757 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbc6c0051c0 0x7fbc6c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:07:47.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.597+0000 7fbc86000640 1 -- 172.21.15.96:0/4279517757 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fbc880d3620 con 0x7fbc6c004800 2024-05-28T06:07:47.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.597+0000 7fbc86000640 1 -- 172.21.15.96:0/4279517757 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fbc6c00c110 con 0x7fbc6c0039b0 2024-05-28T06:07:47.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.597+0000 7fbc86000640 1 -- 172.21.15.96:0/4279517757 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fbc6c00c310 con 0x7fbc6c0051c0 2024-05-28T06:07:47.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.597+0000 7fbc86a00640 1 --2- 172.21.15.96:0/4279517757 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc6c004800 0x7fbc6c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:07:47.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.597+0000 7fbc87400640 1 --2- 172.21.15.96:0/4279517757 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc6c0039b0 0x7fbc6c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:07:47.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.597+0000 7fbc87e00640 1 --2- 172.21.15.96:0/4279517757 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbc6c0051c0 0x7fbc6c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:07:47.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.597+0000 7fbc86a00640 1 -- 172.21.15.96:0/4279517757 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbc6c0051c0 msgr2=0x7fbc6c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:07:47.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.597+0000 7fbc86a00640 1 --2- 172.21.15.96:0/4279517757 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbc6c0051c0 0x7fbc6c00b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:47.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.597+0000 7fbc86a00640 1 -- 172.21.15.96:0/4279517757 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc6c0039b0 msgr2=0x7fbc6c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:07:47.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.597+0000 7fbc86a00640 1 --2- 172.21.15.96:0/4279517757 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc6c0039b0 0x7fbc6c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:47.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.597+0000 7fbc86a00640 1 -- 172.21.15.96:0/4279517757 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc6c003520 con 0x7fbc6c004800 2024-05-28T06:07:47.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.597+0000 7fbc87e00640 1 -- 172.21.15.96:0/4279517757 reap_dead start 2024-05-28T06:07:47.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.598+0000 7fbc86a00640 1 --2- 172.21.15.96:0/4279517757 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc6c004800 0x7fbc6c004c80 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fbc7400ba10 tx=0x7fbc6c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d44a3fabdc0ed5f7 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:07:47.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.598+0000 7fbc86000640 1 -- 172.21.15.96:0/4279517757 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc7400b5e0 con 0x7fbc6c004800 2024-05-28T06:07:47.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.598+0000 7fbc86000640 1 -- 172.21.15.96:0/4279517757 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc7402adc0 con 0x7fbc6c004800 2024-05-28T06:07:47.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.599+0000 7fbc86000640 1 -- 172.21.15.96:0/4279517757 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc74005bb0 con 0x7fbc6c004800 2024-05-28T06:07:47.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.599+0000 7fbc8d200640 1 -- 172.21.15.96:0/4279517757 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc6c004800 msgr2=0x7fbc6c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:07:47.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.599+0000 7fbc8d200640 1 --2- 172.21.15.96:0/4279517757 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc6c004800 0x7fbc6c004c80 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fbc7400ba10 tx=0x7fbc6c005830 comp rx=0 tx=0).stop 2024-05-28T06:07:47.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.599+0000 7fbc8d200640 1 -- 172.21.15.96:0/4279517757 shutdown_connections 2024-05-28T06:07:47.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.599+0000 7fbc8d200640 1 --2- 172.21.15.96:0/4279517757 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc6c004800 0x7fbc6c004c80 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:47.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.599+0000 7fbc8d200640 1 -- 172.21.15.96:0/4279517757 >> 172.21.15.96:0/4279517757 conn(0x7fbc8809d910 msgr2=0x7fbc8808bfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:07:47.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.599+0000 7fbc8d200640 1 -- 172.21.15.96:0/4279517757 shutdown_connections 2024-05-28T06:07:47.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.599+0000 7fbc8d200640 1 -- 172.21.15.96:0/4279517757 wait complete. 2024-05-28T06:07:47.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.601+0000 7fbc8d200640 1 Processor -- start 2024-05-28T06:07:47.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.601+0000 7fbc8d200640 1 -- start start 2024-05-28T06:07:47.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.602+0000 7fbc8d200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc6c004800 0x7fbc8815eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:07:47.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.602+0000 7fbc8d200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbc881618f0 0x7fbc880cb670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:07:47.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.602+0000 7fbc8d200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc880cbbb0 0x7fbc880cd270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:07:47.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.602+0000 7fbc8d200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fbc8815ebf0 con 0x7fbc6c004800 2024-05-28T06:07:47.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.602+0000 7fbc8d200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fbc8815ea70 con 0x7fbc880cbbb0 2024-05-28T06:07:47.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.602+0000 7fbc8d200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fbc8815ed70 con 0x7fbc881618f0 2024-05-28T06:07:47.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.602+0000 7fbc87400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc6c004800 0x7fbc8815eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:07:47.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.602+0000 7fbc87400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc6c004800 0x7fbc8815eef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:34428/0 (socket says 172.21.15.96:34428) 2024-05-28T06:07:47.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.602+0000 7fbc87400640 1 -- 172.21.15.96:0/706387067 learned_addr learned my addr 172.21.15.96:0/706387067 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:07:47.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.603+0000 7fbc87400640 1 -- 172.21.15.96:0/706387067 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbc881618f0 msgr2=0x7fbc880cb670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:07:47.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.603+0000 7fbc87400640 1 --2- 172.21.15.96:0/706387067 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbc881618f0 0x7fbc880cb670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:47.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.603+0000 7fbc87400640 1 -- 172.21.15.96:0/706387067 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc880cbbb0 msgr2=0x7fbc880cd270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:07:47.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.603+0000 7fbc87400640 1 --2- 172.21.15.96:0/706387067 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc880cbbb0 0x7fbc880cd270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:47.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.603+0000 7fbc87400640 1 -- 172.21.15.96:0/706387067 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc881e8120 con 0x7fbc6c004800 2024-05-28T06:07:47.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.603+0000 7fbc87400640 1 --2- 172.21.15.96:0/706387067 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc6c004800 0x7fbc8815eef0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fbc7800e870 tx=0x7fbc7800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:07:47.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.603+0000 7fbc7f400640 1 -- 172.21.15.96:0/706387067 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc7800c8d0 con 0x7fbc6c004800 2024-05-28T06:07:47.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.603+0000 7fbc7f400640 1 -- 172.21.15.96:0/706387067 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc78018070 con 0x7fbc6c004800 2024-05-28T06:07:47.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.603+0000 7fbc8d200640 1 -- 172.21.15.96:0/706387067 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc881e8400 con 0x7fbc6c004800 2024-05-28T06:07:47.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.603+0000 7fbc7f400640 1 -- 172.21.15.96:0/706387067 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc7800ccd0 con 0x7fbc6c004800 2024-05-28T06:07:47.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.604+0000 7fbc8d200640 1 -- 172.21.15.96:0/706387067 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc881e89c0 con 0x7fbc6c004800 2024-05-28T06:07:47.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.605+0000 7fbc8d200640 1 -- 172.21.15.96:0/706387067 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc880689e0 con 0x7fbc6c004800 2024-05-28T06:07:47.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.605+0000 7fbc7f400640 1 -- 172.21.15.96:0/706387067 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 7) ==== 91040+0+0 (secure 0 0 0) 0x7fbc7801d460 con 0x7fbc6c004800 2024-05-28T06:07:47.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.607+0000 7fbc7f400640 1 --2- 172.21.15.96:0/706387067 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbc6806c8f0 0x7fbc6806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:07:47.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.607+0000 7fbc7f400640 1 -- 172.21.15.96:0/706387067 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7fbc7808bc00 con 0x7fbc6c004800 2024-05-28T06:07:47.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.607+0000 7fbc86a00640 1 --2- 172.21.15.96:0/706387067 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbc6806c8f0 0x7fbc6806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:07:47.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.608+0000 7fbc86a00640 1 --2- 172.21.15.96:0/706387067 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbc6806c8f0 0x7fbc6806edb0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fbc74034000 tx=0x7fbc7402c000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:07:47.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.611+0000 7fbc7f400640 1 -- 172.21.15.96:0/706387067 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbc7805a020 con 0x7fbc6c004800 2024-05-28T06:07:47.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.771+0000 7fbc7f400640 1 -- 172.21.15.96:0/706387067 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mgrmap(e 8) ==== 91193+0+0 (secure 0 0 0) 0x7fbc78055f90 con 0x7fbc6c004800 2024-05-28T06:07:47.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.889+0000 7fbc8d200640 1 -- 172.21.15.96:0/706387067 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fbc8806cbc0 con 0x7fbc6c004800 2024-05-28T06:07:47.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.891+0000 7fbc7f400640 1 -- 172.21.15.96:0/706387067 <== mon.0 v2:172.21.15.96:3300/0 8 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1829 (secure 0 0 0) 0x7fbc78076fe0 con 0x7fbc6c004800 2024-05-28T06:07:47.892 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:07:47.892 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:07:47.892 INFO:teuthology.orchestra.run.smithi096.stdout: "fsid": "e4a831b5-e6d1-47a7-96cb-bf85154ee0d2", 2024-05-28T06:07:47.893 INFO:teuthology.orchestra.run.smithi096.stdout: "health": { 2024-05-28T06:07:47.893 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "HEALTH_OK", 2024-05-28T06:07:47.893 INFO:teuthology.orchestra.run.smithi096.stdout: "checks": {}, 2024-05-28T06:07:47.893 INFO:teuthology.orchestra.run.smithi096.stdout: "mutes": [] 2024-05-28T06:07:47.893 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:07:47.893 INFO:teuthology.orchestra.run.smithi096.stdout: "election_epoch": 8, 2024-05-28T06:07:47.893 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum": [ 2024-05-28T06:07:47.893 INFO:teuthology.orchestra.run.smithi096.stdout: 0, 2024-05-28T06:07:47.893 INFO:teuthology.orchestra.run.smithi096.stdout: 1, 2024-05-28T06:07:47.893 INFO:teuthology.orchestra.run.smithi096.stdout: 2 2024-05-28T06:07:47.893 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:07:47.893 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_names": [ 2024-05-28T06:07:47.893 INFO:teuthology.orchestra.run.smithi096.stdout: "a", 2024-05-28T06:07:47.893 INFO:teuthology.orchestra.run.smithi096.stdout: "b", 2024-05-28T06:07:47.894 INFO:teuthology.orchestra.run.smithi096.stdout: "c" 2024-05-28T06:07:47.894 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:07:47.894 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_age": 241, 2024-05-28T06:07:47.894 INFO:teuthology.orchestra.run.smithi096.stdout: "monmap": { 2024-05-28T06:07:47.894 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 1, 2024-05-28T06:07:47.894 INFO:teuthology.orchestra.run.smithi096.stdout: "min_mon_release_name": "squid", 2024-05-28T06:07:47.894 INFO:teuthology.orchestra.run.smithi096.stdout: "num_mons": 3 2024-05-28T06:07:47.894 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:07:47.894 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap": { 2024-05-28T06:07:47.894 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 33, 2024-05-28T06:07:47.894 INFO:teuthology.orchestra.run.smithi096.stdout: "num_osds": 8, 2024-05-28T06:07:47.894 INFO:teuthology.orchestra.run.smithi096.stdout: "num_up_osds": 8, 2024-05-28T06:07:47.895 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_up_since": 1716876238, 2024-05-28T06:07:47.895 INFO:teuthology.orchestra.run.smithi096.stdout: "num_in_osds": 8, 2024-05-28T06:07:47.895 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_in_since": 1716876229, 2024-05-28T06:07:47.895 INFO:teuthology.orchestra.run.smithi096.stdout: "num_remapped_pgs": 0 2024-05-28T06:07:47.895 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:07:47.895 INFO:teuthology.orchestra.run.smithi096.stdout: "pgmap": { 2024-05-28T06:07:47.895 INFO:teuthology.orchestra.run.smithi096.stdout: "pgs_by_state": [ 2024-05-28T06:07:47.895 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:07:47.895 INFO:teuthology.orchestra.run.smithi096.stdout: "state_name": "active+clean", 2024-05-28T06:07:47.895 INFO:teuthology.orchestra.run.smithi096.stdout: "count": 97 2024-05-28T06:07:47.895 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:07:47.895 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:07:47.895 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pgs": 97, 2024-05-28T06:07:47.895 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pools": 3, 2024-05-28T06:07:47.896 INFO:teuthology.orchestra.run.smithi096.stdout: "num_objects": 2559, 2024-05-28T06:07:47.896 INFO:teuthology.orchestra.run.smithi096.stdout: "data_bytes": 10629552875, 2024-05-28T06:07:47.896 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_used": 21400117248, 2024-05-28T06:07:47.896 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_avail": 751693996032, 2024-05-28T06:07:47.896 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_total": 773094113280, 2024-05-28T06:07:47.896 INFO:teuthology.orchestra.run.smithi096.stdout: "write_bytes_sec": 80712090, 2024-05-28T06:07:47.896 INFO:teuthology.orchestra.run.smithi096.stdout: "read_op_per_sec": 0, 2024-05-28T06:07:47.896 INFO:teuthology.orchestra.run.smithi096.stdout: "write_op_per_sec": 20 2024-05-28T06:07:47.896 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:07:47.896 INFO:teuthology.orchestra.run.smithi096.stdout: "fsmap": { 2024-05-28T06:07:47.896 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 16, 2024-05-28T06:07:47.896 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 2, 2024-05-28T06:07:47.896 INFO:teuthology.orchestra.run.smithi096.stdout: "up": 1, 2024-05-28T06:07:47.897 INFO:teuthology.orchestra.run.smithi096.stdout: "in": 1, 2024-05-28T06:07:47.897 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 1, 2024-05-28T06:07:47.897 INFO:teuthology.orchestra.run.smithi096.stdout: "by_rank": [ 2024-05-28T06:07:47.897 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:07:47.897 INFO:teuthology.orchestra.run.smithi096.stdout: "filesystem_id": 2, 2024-05-28T06:07:47.897 INFO:teuthology.orchestra.run.smithi096.stdout: "rank": 0, 2024-05-28T06:07:47.897 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "d", 2024-05-28T06:07:47.897 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "up:active", 2024-05-28T06:07:47.897 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 5238 2024-05-28T06:07:47.897 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:07:47.898 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:07:47.898 INFO:teuthology.orchestra.run.smithi096.stdout: "up:standby": 3 2024-05-28T06:07:47.898 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:07:47.898 INFO:teuthology.orchestra.run.smithi096.stdout: "mgrmap": { 2024-05-28T06:07:47.898 INFO:teuthology.orchestra.run.smithi096.stdout: "available": true, 2024-05-28T06:07:47.898 INFO:teuthology.orchestra.run.smithi096.stdout: "num_standbys": 1, 2024-05-28T06:07:47.898 INFO:teuthology.orchestra.run.smithi096.stdout: "modules": [ 2024-05-28T06:07:47.898 INFO:teuthology.orchestra.run.smithi096.stdout: "iostat", 2024-05-28T06:07:47.898 INFO:teuthology.orchestra.run.smithi096.stdout: "nfs", 2024-05-28T06:07:47.898 INFO:teuthology.orchestra.run.smithi096.stdout: "restful" 2024-05-28T06:07:47.898 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:07:47.898 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:07:47.898 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:07:47.899 INFO:teuthology.orchestra.run.smithi096.stdout: "servicemap": { 2024-05-28T06:07:47.899 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 7, 2024-05-28T06:07:47.899 INFO:teuthology.orchestra.run.smithi096.stdout: "modified": "2024-05-28T06:06:33.535436+0000", 2024-05-28T06:07:47.899 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:07:47.899 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:07:47.899 INFO:teuthology.orchestra.run.smithi096.stdout: "progress_events": {} 2024-05-28T06:07:47.899 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:07:47.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.893+0000 7fbc8d200640 1 -- 172.21.15.96:0/706387067 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbc6806c8f0 msgr2=0x7fbc6806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:07:47.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.893+0000 7fbc8d200640 1 --2- 172.21.15.96:0/706387067 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbc6806c8f0 0x7fbc6806edb0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fbc74034000 tx=0x7fbc7402c000 comp rx=0 tx=0).stop 2024-05-28T06:07:47.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.893+0000 7fbc8d200640 1 -- 172.21.15.96:0/706387067 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc6c004800 msgr2=0x7fbc8815eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:07:47.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.893+0000 7fbc8d200640 1 --2- 172.21.15.96:0/706387067 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc6c004800 0x7fbc8815eef0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fbc7800e870 tx=0x7fbc7800ed30 comp rx=0 tx=0).stop 2024-05-28T06:07:47.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.894+0000 7fbc8d200640 1 -- 172.21.15.96:0/706387067 shutdown_connections 2024-05-28T06:07:47.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.894+0000 7fbc8d200640 1 --2- 172.21.15.96:0/706387067 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbc6806c8f0 0x7fbc6806edb0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:47.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.894+0000 7fbc8d200640 1 --2- 172.21.15.96:0/706387067 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc880cbbb0 0x7fbc880cd270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:47.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.894+0000 7fbc8d200640 1 --2- 172.21.15.96:0/706387067 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbc881618f0 0x7fbc880cb670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:47.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.894+0000 7fbc8d200640 1 --2- 172.21.15.96:0/706387067 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc6c004800 0x7fbc8815eef0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:47.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.894+0000 7fbc8d200640 1 -- 172.21.15.96:0/706387067 >> 172.21.15.96:0/706387067 conn(0x7fbc8809d910 msgr2=0x7fbc880d2150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:07:47.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.894+0000 7fbc8d200640 1 -- 172.21.15.96:0/706387067 shutdown_connections 2024-05-28T06:07:47.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:47.894+0000 7fbc8d200640 1 -- 172.21.15.96:0/706387067 wait complete. 2024-05-28T06:07:49.914 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-05-28T06:07:50.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.121+0000 7f4202000640 1 Processor -- start 2024-05-28T06:07:50.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.122+0000 7f4202000640 1 -- start start 2024-05-28T06:07:50.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.124+0000 7f4202000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f41fc15c6b0 0x7f41fc0b3db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:07:50.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.124+0000 7f4202000640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41fc0cd1a0 con 0x7f41fc0ccc30 2024-05-28T06:07:50.125 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.124+0000 7f4202000640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41fc0cd480 con 0x7f41fc0d0460 2024-05-28T06:07:50.125 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.124+0000 7f4202000640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f41fc06e3a0 con 0x7f41fc15c6b0 2024-05-28T06:07:50.125 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.124+0000 7f41fbe00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f41fc0ccc30 0x7f41fc06dc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:50038/0 (socket says 172.21.15.96:50038) 2024-05-28T06:07:50.125 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.124+0000 7f41fbe00640 1 -- 172.21.15.96:0/2808916271 learned_addr learned my addr 172.21.15.96:0/2808916271 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:07:50.125 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.124+0000 7f4200c00640 1 --2- 172.21.15.96:0/2808916271 >> v2:172.21.15.123:3300/0 conn(0x7f41fc15c6b0 0x7f41fc0b3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:07:50.125 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.124+0000 7f41fb400640 1 -- 172.21.15.96:0/2808916271 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3568676804 0 0) 0x7f41fc0cd1a0 con 0x7f41fc0ccc30 2024-05-28T06:07:50.125 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.124+0000 7f41fb400640 1 -- 172.21.15.96:0/2808916271 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f41e4003520 con 0x7f41fc0ccc30 2024-05-28T06:07:50.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.124+0000 7f41fb400640 1 -- 172.21.15.96:0/2808916271 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3472392759 0 0) 0x7f41e4003520 con 0x7f41fc0ccc30 2024-05-28T06:07:50.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.124+0000 7f41fb400640 1 -- 172.21.15.96:0/2808916271 >> v2:172.21.15.123:3300/0 conn(0x7f41fc15c6b0 msgr2=0x7f41fc0b3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:07:50.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.124+0000 7f41fb400640 1 --2- 172.21.15.96:0/2808916271 >> v2:172.21.15.123:3300/0 conn(0x7f41fc15c6b0 0x7f41fc0b3db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:50.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.124+0000 7f41fb400640 1 -- 172.21.15.96:0/2808916271 >> v1:172.21.15.123:6789/0 conn(0x7f41fc0d0460 legacy=0x7f41fc0cbf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:07:50.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.124+0000 7f41fb400640 1 -- 172.21.15.96:0/2808916271 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41fc0cd760 con 0x7f41fc0ccc30 2024-05-28T06:07:50.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.124+0000 7f41fb400640 1 -- 172.21.15.96:0/2808916271 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f41e8001670 con 0x7f41fc0ccc30 2024-05-28T06:07:50.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.124+0000 7f41fb400640 1 -- 172.21.15.96:0/2808916271 >> v1:172.21.15.96:6789/0 conn(0x7f41fc0ccc30 legacy=0x7f41fc06dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:07:50.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.124+0000 7f41fb400640 1 --2- 172.21.15.96:0/2808916271 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f41e40039b0 0x7f41e4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:07:50.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.124+0000 7f41fb400640 1 --2- 172.21.15.96:0/2808916271 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41e4004800 0x7f41e4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:07:50.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.124+0000 7f41fb400640 1 --2- 172.21.15.96:0/2808916271 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41e40051c0 0x7f41e400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:07:50.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.125+0000 7f41fb400640 1 -- 172.21.15.96:0/2808916271 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f41fc06e3a0 con 0x7f41e40051c0 2024-05-28T06:07:50.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.125+0000 7f41fb400640 1 -- 172.21.15.96:0/2808916271 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f41e400c060 con 0x7f41e4004800 2024-05-28T06:07:50.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.125+0000 7f41fb400640 1 -- 172.21.15.96:0/2808916271 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f41e400c260 con 0x7f41e40039b0 2024-05-28T06:07:50.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.125+0000 7f4201600640 1 --2- 172.21.15.96:0/2808916271 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41e40051c0 0x7f41e400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:07:50.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.125+0000 7f4200c00640 1 --2- 172.21.15.96:0/2808916271 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f41e40039b0 0x7f41e4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:07:50.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.125+0000 7f41fbe00640 1 --2- 172.21.15.96:0/2808916271 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41e4004800 0x7f41e4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:07:50.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.125+0000 7f4201600640 1 -- 172.21.15.96:0/2808916271 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f41e40039b0 msgr2=0x7f41e4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:07:50.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.125+0000 7f4201600640 1 --2- 172.21.15.96:0/2808916271 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f41e40039b0 0x7f41e4003dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:50.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.125+0000 7f4201600640 1 -- 172.21.15.96:0/2808916271 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41e4004800 msgr2=0x7f41e4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:07:50.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.125+0000 7f4201600640 1 --2- 172.21.15.96:0/2808916271 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41e4004800 0x7f41e4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:50.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.125+0000 7f4201600640 1 -- 172.21.15.96:0/2808916271 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41e4003520 con 0x7f41e40051c0 2024-05-28T06:07:50.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.125+0000 7f41fbe00640 1 --2- 172.21.15.96:0/2808916271 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41e4004800 0x7f41e4004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:07:50.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.126+0000 7f4201600640 1 -- 172.21.15.96:0/2808916271 reap_dead start 2024-05-28T06:07:50.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.126+0000 7f4201600640 1 --2- 172.21.15.96:0/2808916271 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41e40051c0 0x7f41e400b9c0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f41f4001e50 tx=0x7f41e4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a05dfc5b735034fd server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:07:50.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.127+0000 7f41fb400640 1 -- 172.21.15.96:0/2808916271 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41f4001a70 con 0x7f41e40051c0 2024-05-28T06:07:50.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.127+0000 7f41fb400640 1 -- 172.21.15.96:0/2808916271 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41f4003970 con 0x7f41e40051c0 2024-05-28T06:07:50.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.127+0000 7f41fb400640 1 -- 172.21.15.96:0/2808916271 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41f4003d60 con 0x7f41e40051c0 2024-05-28T06:07:50.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.127+0000 7f4202000640 1 -- 172.21.15.96:0/2808916271 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41e40051c0 msgr2=0x7f41e400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:07:50.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.127+0000 7f4202000640 1 --2- 172.21.15.96:0/2808916271 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41e40051c0 0x7f41e400b9c0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f41f4001e50 tx=0x7f41e4005830 comp rx=0 tx=0).stop 2024-05-28T06:07:50.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.127+0000 7f4202000640 1 -- 172.21.15.96:0/2808916271 shutdown_connections 2024-05-28T06:07:50.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.127+0000 7f4202000640 1 --2- 172.21.15.96:0/2808916271 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41e40051c0 0x7f41e400b9c0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:50.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.127+0000 7f4202000640 1 -- 172.21.15.96:0/2808916271 >> 172.21.15.96:0/2808916271 conn(0x7f41fc06bae0 msgr2=0x7f41fc06bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:07:50.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.127+0000 7f4202000640 1 -- 172.21.15.96:0/2808916271 shutdown_connections 2024-05-28T06:07:50.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.127+0000 7f4202000640 1 -- 172.21.15.96:0/2808916271 wait complete. 2024-05-28T06:07:50.131 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.129+0000 7f4202000640 1 Processor -- start 2024-05-28T06:07:50.131 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.129+0000 7f4202000640 1 -- start start 2024-05-28T06:07:50.132 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.130+0000 7f4202000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41e40051c0 0x7f41fc155320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:07:50.132 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.130+0000 7f4202000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41fc0b1d10 0x7f41fc0c6570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:07:50.132 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.131+0000 7f4202000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f41fc0afd20 0x7f41fc0b0180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:07:50.132 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.131+0000 7f4202000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f41fc0d08b0 con 0x7f41e40051c0 2024-05-28T06:07:50.132 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.131+0000 7f4202000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f41fc0cba10 con 0x7f41fc0b1d10 2024-05-28T06:07:50.132 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.131+0000 7f4202000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f41fc0d0a30 con 0x7f41fc0afd20 2024-05-28T06:07:50.132 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.131+0000 7f4200c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41e40051c0 0x7f41fc155320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:07:50.133 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.131+0000 7f4200c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41e40051c0 0x7f41fc155320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49690/0 (socket says 172.21.15.96:49690) 2024-05-28T06:07:50.133 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.131+0000 7f4200c00640 1 -- 172.21.15.96:0/932425242 learned_addr learned my addr 172.21.15.96:0/932425242 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:07:50.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.131+0000 7f41fbe00640 1 --2- 172.21.15.96:0/932425242 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41fc0b1d10 0x7f41fc0c6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:07:50.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.131+0000 7f4201600640 1 --2- 172.21.15.96:0/932425242 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f41fc0afd20 0x7f41fc0b0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:07:50.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.131+0000 7f4200c00640 1 -- 172.21.15.96:0/932425242 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f41fc0afd20 msgr2=0x7f41fc0b0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:07:50.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.131+0000 7f4200c00640 1 --2- 172.21.15.96:0/932425242 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f41fc0afd20 0x7f41fc0b0180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:50.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.131+0000 7f4200c00640 1 -- 172.21.15.96:0/932425242 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41fc0b1d10 msgr2=0x7f41fc0c6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:07:50.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.131+0000 7f4200c00640 1 --2- 172.21.15.96:0/932425242 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41fc0b1d10 0x7f41fc0c6570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:50.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.131+0000 7f4200c00640 1 -- 172.21.15.96:0/932425242 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41fc0b2480 con 0x7f41e40051c0 2024-05-28T06:07:50.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.131+0000 7f41fbe00640 1 --2- 172.21.15.96:0/932425242 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41fc0b1d10 0x7f41fc0c6570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:07:50.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.132+0000 7f4200c00640 1 --2- 172.21.15.96:0/932425242 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41e40051c0 0x7f41fc155320 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f41f0010890 tx=0x7f41f0010d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:07:50.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.132+0000 7f41f8c00640 1 -- 172.21.15.96:0/932425242 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41f000f040 con 0x7f41e40051c0 2024-05-28T06:07:50.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.132+0000 7f41f8c00640 1 -- 172.21.15.96:0/932425242 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41f000eb60 con 0x7f41e40051c0 2024-05-28T06:07:50.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.132+0000 7f41f8c00640 1 -- 172.21.15.96:0/932425242 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41f001c400 con 0x7f41e40051c0 2024-05-28T06:07:50.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.132+0000 7f4202000640 1 -- 172.21.15.96:0/932425242 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41fc0b2760 con 0x7f41e40051c0 2024-05-28T06:07:50.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.132+0000 7f4202000640 1 -- 172.21.15.96:0/932425242 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41fc0c7ce0 con 0x7f41e40051c0 2024-05-28T06:07:50.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.133+0000 7f4202000640 1 -- 172.21.15.96:0/932425242 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41fc0689e0 con 0x7f41e40051c0 2024-05-28T06:07:50.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.134+0000 7f41f8c00640 1 -- 172.21.15.96:0/932425242 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 8) ==== 91193+0+0 (secure 0 0 0) 0x7f41f001c5a0 con 0x7f41e40051c0 2024-05-28T06:07:50.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.135+0000 7f41f8c00640 1 --2- 172.21.15.96:0/932425242 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f41dc06caa0 0x7f41dc06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:07:50.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.135+0000 7f41f8c00640 1 -- 172.21.15.96:0/932425242 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f41f008c5e0 con 0x7f41e40051c0 2024-05-28T06:07:50.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.136+0000 7f41fbe00640 1 --2- 172.21.15.96:0/932425242 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f41dc06caa0 0x7f41dc06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:07:50.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.137+0000 7f41fbe00640 1 --2- 172.21.15.96:0/932425242 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f41dc06caa0 0x7f41dc06ef60 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f41e8001200 tx=0x7f41e8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:07:50.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.140+0000 7f41f8c00640 1 -- 172.21.15.96:0/932425242 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f41f0018030 con 0x7f41e40051c0 2024-05-28T06:07:50.416 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.414+0000 7f4202000640 1 -- 172.21.15.96:0/932425242 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f41fc072ef0 con 0x7f41e40051c0 2024-05-28T06:07:50.416 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.415+0000 7f41f8c00640 1 -- 172.21.15.96:0/932425242 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1829 (secure 0 0 0) 0x7f41f005a0a0 con 0x7f41e40051c0 2024-05-28T06:07:50.417 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:07:50.417 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:07:50.417 INFO:teuthology.orchestra.run.smithi096.stdout: "fsid": "e4a831b5-e6d1-47a7-96cb-bf85154ee0d2", 2024-05-28T06:07:50.417 INFO:teuthology.orchestra.run.smithi096.stdout: "health": { 2024-05-28T06:07:50.417 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "HEALTH_OK", 2024-05-28T06:07:50.417 INFO:teuthology.orchestra.run.smithi096.stdout: "checks": {}, 2024-05-28T06:07:50.417 INFO:teuthology.orchestra.run.smithi096.stdout: "mutes": [] 2024-05-28T06:07:50.417 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:07:50.417 INFO:teuthology.orchestra.run.smithi096.stdout: "election_epoch": 8, 2024-05-28T06:07:50.417 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum": [ 2024-05-28T06:07:50.417 INFO:teuthology.orchestra.run.smithi096.stdout: 0, 2024-05-28T06:07:50.418 INFO:teuthology.orchestra.run.smithi096.stdout: 1, 2024-05-28T06:07:50.418 INFO:teuthology.orchestra.run.smithi096.stdout: 2 2024-05-28T06:07:50.418 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:07:50.418 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_names": [ 2024-05-28T06:07:50.418 INFO:teuthology.orchestra.run.smithi096.stdout: "a", 2024-05-28T06:07:50.418 INFO:teuthology.orchestra.run.smithi096.stdout: "b", 2024-05-28T06:07:50.418 INFO:teuthology.orchestra.run.smithi096.stdout: "c" 2024-05-28T06:07:50.418 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:07:50.418 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_age": 243, 2024-05-28T06:07:50.418 INFO:teuthology.orchestra.run.smithi096.stdout: "monmap": { 2024-05-28T06:07:50.418 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 1, 2024-05-28T06:07:50.418 INFO:teuthology.orchestra.run.smithi096.stdout: "min_mon_release_name": "squid", 2024-05-28T06:07:50.419 INFO:teuthology.orchestra.run.smithi096.stdout: "num_mons": 3 2024-05-28T06:07:50.419 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:07:50.419 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap": { 2024-05-28T06:07:50.419 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 33, 2024-05-28T06:07:50.419 INFO:teuthology.orchestra.run.smithi096.stdout: "num_osds": 8, 2024-05-28T06:07:50.419 INFO:teuthology.orchestra.run.smithi096.stdout: "num_up_osds": 8, 2024-05-28T06:07:50.419 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_up_since": 1716876238, 2024-05-28T06:07:50.419 INFO:teuthology.orchestra.run.smithi096.stdout: "num_in_osds": 8, 2024-05-28T06:07:50.419 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_in_since": 1716876229, 2024-05-28T06:07:50.419 INFO:teuthology.orchestra.run.smithi096.stdout: "num_remapped_pgs": 0 2024-05-28T06:07:50.419 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:07:50.419 INFO:teuthology.orchestra.run.smithi096.stdout: "pgmap": { 2024-05-28T06:07:50.419 INFO:teuthology.orchestra.run.smithi096.stdout: "pgs_by_state": [ 2024-05-28T06:07:50.419 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:07:50.420 INFO:teuthology.orchestra.run.smithi096.stdout: "state_name": "active+clean", 2024-05-28T06:07:50.420 INFO:teuthology.orchestra.run.smithi096.stdout: "count": 97 2024-05-28T06:07:50.420 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:07:50.420 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:07:50.420 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pgs": 97, 2024-05-28T06:07:50.420 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pools": 3, 2024-05-28T06:07:50.420 INFO:teuthology.orchestra.run.smithi096.stdout: "num_objects": 2559, 2024-05-28T06:07:50.420 INFO:teuthology.orchestra.run.smithi096.stdout: "data_bytes": 10629552875, 2024-05-28T06:07:50.420 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_used": 21400117248, 2024-05-28T06:07:50.420 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_avail": 751693996032, 2024-05-28T06:07:50.420 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_total": 773094113280, 2024-05-28T06:07:50.420 INFO:teuthology.orchestra.run.smithi096.stdout: "write_bytes_sec": 80714756, 2024-05-28T06:07:50.420 INFO:teuthology.orchestra.run.smithi096.stdout: "read_op_per_sec": 0, 2024-05-28T06:07:50.420 INFO:teuthology.orchestra.run.smithi096.stdout: "write_op_per_sec": 20 2024-05-28T06:07:50.421 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:07:50.421 INFO:teuthology.orchestra.run.smithi096.stdout: "fsmap": { 2024-05-28T06:07:50.421 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 16, 2024-05-28T06:07:50.421 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 2, 2024-05-28T06:07:50.421 INFO:teuthology.orchestra.run.smithi096.stdout: "up": 1, 2024-05-28T06:07:50.421 INFO:teuthology.orchestra.run.smithi096.stdout: "in": 1, 2024-05-28T06:07:50.421 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 1, 2024-05-28T06:07:50.421 INFO:teuthology.orchestra.run.smithi096.stdout: "by_rank": [ 2024-05-28T06:07:50.421 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:07:50.421 INFO:teuthology.orchestra.run.smithi096.stdout: "filesystem_id": 2, 2024-05-28T06:07:50.421 INFO:teuthology.orchestra.run.smithi096.stdout: "rank": 0, 2024-05-28T06:07:50.421 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "d", 2024-05-28T06:07:50.421 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "up:active", 2024-05-28T06:07:50.421 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 5238 2024-05-28T06:07:50.422 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:07:50.422 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:07:50.422 INFO:teuthology.orchestra.run.smithi096.stdout: "up:standby": 3 2024-05-28T06:07:50.422 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:07:50.422 INFO:teuthology.orchestra.run.smithi096.stdout: "mgrmap": { 2024-05-28T06:07:50.422 INFO:teuthology.orchestra.run.smithi096.stdout: "available": true, 2024-05-28T06:07:50.422 INFO:teuthology.orchestra.run.smithi096.stdout: "num_standbys": 1, 2024-05-28T06:07:50.422 INFO:teuthology.orchestra.run.smithi096.stdout: "modules": [ 2024-05-28T06:07:50.422 INFO:teuthology.orchestra.run.smithi096.stdout: "iostat", 2024-05-28T06:07:50.422 INFO:teuthology.orchestra.run.smithi096.stdout: "nfs", 2024-05-28T06:07:50.422 INFO:teuthology.orchestra.run.smithi096.stdout: "restful" 2024-05-28T06:07:50.422 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:07:50.422 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:07:50.423 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:07:50.423 INFO:teuthology.orchestra.run.smithi096.stdout: "servicemap": { 2024-05-28T06:07:50.423 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 7, 2024-05-28T06:07:50.423 INFO:teuthology.orchestra.run.smithi096.stdout: "modified": "2024-05-28T06:06:33.535436+0000", 2024-05-28T06:07:50.423 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:07:50.423 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:07:50.423 INFO:teuthology.orchestra.run.smithi096.stdout: "progress_events": {} 2024-05-28T06:07:50.423 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:07:50.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.418+0000 7f4202000640 1 -- 172.21.15.96:0/932425242 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f41dc06caa0 msgr2=0x7f41dc06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:07:50.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.418+0000 7f4202000640 1 --2- 172.21.15.96:0/932425242 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f41dc06caa0 0x7f41dc06ef60 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f41e8001200 tx=0x7f41e8009040 comp rx=0 tx=0).stop 2024-05-28T06:07:50.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.418+0000 7f4202000640 1 -- 172.21.15.96:0/932425242 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41e40051c0 msgr2=0x7f41fc155320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:07:50.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.418+0000 7f4202000640 1 --2- 172.21.15.96:0/932425242 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41e40051c0 0x7f41fc155320 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f41f0010890 tx=0x7f41f0010d50 comp rx=0 tx=0).stop 2024-05-28T06:07:50.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.418+0000 7f4202000640 1 -- 172.21.15.96:0/932425242 shutdown_connections 2024-05-28T06:07:50.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.418+0000 7f4202000640 1 --2- 172.21.15.96:0/932425242 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f41dc06caa0 0x7f41dc06ef60 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:50.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.418+0000 7f4202000640 1 --2- 172.21.15.96:0/932425242 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f41fc0afd20 0x7f41fc0b0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:50.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.418+0000 7f4202000640 1 --2- 172.21.15.96:0/932425242 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41fc0b1d10 0x7f41fc0c6570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:50.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.418+0000 7f4202000640 1 --2- 172.21.15.96:0/932425242 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41e40051c0 0x7f41fc155320 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:50.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.418+0000 7f4202000640 1 -- 172.21.15.96:0/932425242 >> 172.21.15.96:0/932425242 conn(0x7f41fc06bae0 msgr2=0x7f41fc15b8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:07:50.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.418+0000 7f4202000640 1 -- 172.21.15.96:0/932425242 shutdown_connections 2024-05-28T06:07:50.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:50.418+0000 7f4202000640 1 -- 172.21.15.96:0/932425242 wait complete. 2024-05-28T06:07:52.440 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-05-28T06:07:52.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:52.708+0000 7f0bb6200640 1 Processor -- start 2024-05-28T06:07:52.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:52.708+0000 7f0bb6200640 1 -- start start 2024-05-28T06:07:52.710 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:52.709+0000 7f0bb6200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0bb006b900 0x7f0bb006bce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:07:52.711 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:52.709+0000 7f0bb6200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f0bb015ed60 0x7f0bb006f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:07:52.711 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:52.709+0000 7f0bb6200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f0bb00cba20 0x7f0bb006f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:07:52.711 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:52.709+0000 7f0bb6200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f0bb00d05f0 con 0x7f0bb00cba20 2024-05-28T06:07:52.711 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:52.709+0000 7f0bb6200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f0bb006c220 con 0x7f0bb015ed60 2024-05-28T06:07:52.711 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:52.709+0000 7f0bb6200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f0bb00d0890 con 0x7f0bb006b900 2024-05-28T06:07:52.711 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:52.710+0000 7f0bafe00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f0bb00cba20 0x7f0bb006f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:07:52.711 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:52.710+0000 7f0bafe00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f0bb00cba20 0x7f0bb006f620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49694/0 (socket says 172.21.15.96:49694) 2024-05-28T06:07:52.711 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:52.710+0000 7f0bafe00640 1 -- 172.21.15.96:0/3909260395 learned_addr learned my addr 172.21.15.96:0/3909260395 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:07:52.711 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:52.710+0000 7f0bafe00640 1 -- 172.21.15.96:0/3909260395 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0bb006b900 msgr2=0x7f0bb006bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:07:52.711 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:52.710+0000 7f0bafe00640 1 --2- 172.21.15.96:0/3909260395 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0bb006b900 0x7f0bb006bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:52.711 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:52.710+0000 7f0bafe00640 1 -- 172.21.15.96:0/3909260395 >> v2:172.21.15.123:3300/0 conn(0x7f0bb015ed60 msgr2=0x7f0bb006f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:07:52.711 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:52.710+0000 7f0bafe00640 1 --2- 172.21.15.96:0/3909260395 >> v2:172.21.15.123:3300/0 conn(0x7f0bb015ed60 0x7f0bb006f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:52.711 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:52.710+0000 7f0bafe00640 1 -- 172.21.15.96:0/3909260395 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bb00d0b70 con 0x7f0bb00cba20 2024-05-28T06:07:52.712 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:52.711+0000 7f0bafe00640 1 --2- 172.21.15.96:0/3909260395 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0bb00cba20 0x7f0bb006f620 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f0bb00717d0 tx=0x7f0ba802a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7754a685cfb07762 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:07:52.712 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:52.712+0000 7f0bae000640 1 -- 172.21.15.96:0/3909260395 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ba8036070 con 0x7f0bb00cba20 2024-05-28T06:07:52.712 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:52.712+0000 7f0bae000640 1 -- 172.21.15.96:0/3909260395 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ba8031600 con 0x7f0bb00cba20 2024-05-28T06:07:52.712 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:52.712+0000 7f0bae000640 1 -- 172.21.15.96:0/3909260395 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ba80319d0 con 0x7f0bb00cba20 2024-05-28T06:07:52.713 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:52.713+0000 7f0bb6200640 1 -- 172.21.15.96:0/3909260395 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0bb00cba20 msgr2=0x7f0bb006f620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:07:52.713 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:52.713+0000 7f0bb6200640 1 --2- 172.21.15.96:0/3909260395 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0bb00cba20 0x7f0bb006f620 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f0bb00717d0 tx=0x7f0ba802a610 comp rx=0 tx=0).stop 2024-05-28T06:07:52.713 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:52.713+0000 7f0bb6200640 1 -- 172.21.15.96:0/3909260395 shutdown_connections 2024-05-28T06:07:52.713 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:52.713+0000 7f0bb6200640 1 --2- 172.21.15.96:0/3909260395 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0bb00cba20 0x7f0bb006f620 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:52.713 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:52.713+0000 7f0bb6200640 1 --2- 172.21.15.96:0/3909260395 >> v2:172.21.15.123:3300/0 conn(0x7f0bb015ed60 0x7f0bb006f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:52.713 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:52.713+0000 7f0bb6200640 1 --2- 172.21.15.96:0/3909260395 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0bb006b900 0x7f0bb006bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:52.714 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:52.713+0000 7f0bb6200640 1 -- 172.21.15.96:0/3909260395 >> 172.21.15.96:0/3909260395 conn(0x7f0bb009aa30 msgr2=0x7f0bb00b2f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:07:52.714 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:52.713+0000 7f0bb6200640 1 -- 172.21.15.96:0/3909260395 shutdown_connections 2024-05-28T06:07:52.714 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:52.713+0000 7f0bb6200640 1 -- 172.21.15.96:0/3909260395 wait complete. 2024-05-28T06:07:52.715 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:52.715+0000 7f0bb6200640 1 Processor -- start 2024-05-28T06:07:52.715 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:52.715+0000 7f0bb6200640 1 -- start start 2024-05-28T06:07:52.716 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:52.716+0000 7f0bb6200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0bb006b900 0x7f0bb0150940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:07:52.716 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:52.716+0000 7f0bb6200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0bb00cba20 0x7f0bb00b54c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:07:52.717 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:52.716+0000 7f0bb6200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0bb015ed60 0x7f0bb00af270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:07:52.717 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:52.716+0000 7f0bb6200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f0bb00d1130 con 0x7f0bb015ed60 2024-05-28T06:07:52.717 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:52.716+0000 7f0bb6200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f0bb00b3810 con 0x7f0bb006b900 2024-05-28T06:07:52.717 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:52.716+0000 7f0bb6200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f0bb00cd5f0 con 0x7f0bb00cba20 2024-05-28T06:07:52.717 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:52.716+0000 7f0bafe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0bb015ed60 0x7f0bb00af270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:07:52.717 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:52.716+0000 7f0bafe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0bb015ed60 0x7f0bb00af270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49702/0 (socket says 172.21.15.96:49702) 2024-05-28T06:07:52.717 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:52.716+0000 7f0bafe00640 1 -- 172.21.15.96:0/469381316 learned_addr learned my addr 172.21.15.96:0/469381316 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:07:52.718 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:52.717+0000 7f0baea00640 1 --2- 172.21.15.96:0/469381316 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0bb00cba20 0x7f0bb00b54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:07:52.718 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:52.717+0000 7f0baf400640 1 --2- 172.21.15.96:0/469381316 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0bb006b900 0x7f0bb0150940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:07:52.718 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:52.717+0000 7f0bafe00640 1 -- 172.21.15.96:0/469381316 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0bb00cba20 msgr2=0x7f0bb00b54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:07:52.718 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:52.717+0000 7f0bafe00640 1 --2- 172.21.15.96:0/469381316 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0bb00cba20 0x7f0bb00b54c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:52.718 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:52.717+0000 7f0bafe00640 1 -- 172.21.15.96:0/469381316 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0bb006b900 msgr2=0x7f0bb0150940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:07:52.718 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:52.717+0000 7f0bafe00640 1 --2- 172.21.15.96:0/469381316 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0bb006b900 0x7f0bb0150940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:52.719 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:52.717+0000 7f0bafe00640 1 -- 172.21.15.96:0/469381316 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bb0150e80 con 0x7f0bb015ed60 2024-05-28T06:07:52.719 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:52.717+0000 7f0bafe00640 1 --2- 172.21.15.96:0/469381316 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0bb015ed60 0x7f0bb00af270 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f0ba8009e20 tx=0x7f0ba8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:07:52.719 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:52.718+0000 7f0ba7400640 1 -- 172.21.15.96:0/469381316 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ba8041020 con 0x7f0bb015ed60 2024-05-28T06:07:52.719 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:52.718+0000 7f0ba7400640 1 -- 172.21.15.96:0/469381316 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ba8037040 con 0x7f0bb015ed60 2024-05-28T06:07:52.719 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:52.718+0000 7f0ba7400640 1 -- 172.21.15.96:0/469381316 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ba8036070 con 0x7f0bb015ed60 2024-05-28T06:07:52.719 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:52.718+0000 7f0bb6200640 1 -- 172.21.15.96:0/469381316 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0bb00b5da0 con 0x7f0bb015ed60 2024-05-28T06:07:52.719 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:52.718+0000 7f0bb6200640 1 -- 172.21.15.96:0/469381316 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0bb00b6360 con 0x7f0bb015ed60 2024-05-28T06:07:52.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:52.722+0000 7f0ba7400640 1 -- 172.21.15.96:0/469381316 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 8) ==== 91193+0+0 (secure 0 0 0) 0x7f0ba8044020 con 0x7f0bb015ed60 2024-05-28T06:07:52.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:52.724+0000 7f0ba7400640 1 --2- 172.21.15.96:0/469381316 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0b9006caa0 0x7f0b9006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:07:52.725 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:52.724+0000 7f0bb6200640 1 -- 172.21.15.96:0/469381316 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0bb00cd080 con 0x7f0bb015ed60 2024-05-28T06:07:52.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:52.726+0000 7f0ba7400640 1 -- 172.21.15.96:0/469381316 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f0ba80acbe0 con 0x7f0bb015ed60 2024-05-28T06:07:52.737 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:52.736+0000 7f0baf400640 1 --2- 172.21.15.96:0/469381316 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0b9006caa0 0x7f0b9006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:07:52.737 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:52.737+0000 7f0ba7400640 1 -- 172.21.15.96:0/469381316 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0bb00cd080 con 0x7f0bb015ed60 2024-05-28T06:07:52.742 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:52.742+0000 7f0baf400640 1 --2- 172.21.15.96:0/469381316 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0b9006caa0 0x7f0b9006ef60 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f0ba0009890 tx=0x7f0ba0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:07:53.125 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:53.125+0000 7f0bb6200640 1 -- 172.21.15.96:0/469381316 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f0bb0070580 con 0x7f0bb015ed60 2024-05-28T06:07:53.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:53.126+0000 7f0ba7400640 1 -- 172.21.15.96:0/469381316 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1860 (secure 0 0 0) 0x7f0ba804b030 con 0x7f0bb015ed60 2024-05-28T06:07:53.128 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:07:53.128 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:07:53.128 INFO:teuthology.orchestra.run.smithi096.stdout: "fsid": "e4a831b5-e6d1-47a7-96cb-bf85154ee0d2", 2024-05-28T06:07:53.128 INFO:teuthology.orchestra.run.smithi096.stdout: "health": { 2024-05-28T06:07:53.128 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "HEALTH_OK", 2024-05-28T06:07:53.128 INFO:teuthology.orchestra.run.smithi096.stdout: "checks": {}, 2024-05-28T06:07:53.128 INFO:teuthology.orchestra.run.smithi096.stdout: "mutes": [] 2024-05-28T06:07:53.128 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:07:53.128 INFO:teuthology.orchestra.run.smithi096.stdout: "election_epoch": 8, 2024-05-28T06:07:53.129 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum": [ 2024-05-28T06:07:53.129 INFO:teuthology.orchestra.run.smithi096.stdout: 0, 2024-05-28T06:07:53.129 INFO:teuthology.orchestra.run.smithi096.stdout: 1, 2024-05-28T06:07:53.129 INFO:teuthology.orchestra.run.smithi096.stdout: 2 2024-05-28T06:07:53.129 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:07:53.129 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_names": [ 2024-05-28T06:07:53.129 INFO:teuthology.orchestra.run.smithi096.stdout: "a", 2024-05-28T06:07:53.129 INFO:teuthology.orchestra.run.smithi096.stdout: "b", 2024-05-28T06:07:53.129 INFO:teuthology.orchestra.run.smithi096.stdout: "c" 2024-05-28T06:07:53.129 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:07:53.129 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_age": 246, 2024-05-28T06:07:53.129 INFO:teuthology.orchestra.run.smithi096.stdout: "monmap": { 2024-05-28T06:07:53.129 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 1, 2024-05-28T06:07:53.129 INFO:teuthology.orchestra.run.smithi096.stdout: "min_mon_release_name": "squid", 2024-05-28T06:07:53.130 INFO:teuthology.orchestra.run.smithi096.stdout: "num_mons": 3 2024-05-28T06:07:53.130 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:07:53.130 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap": { 2024-05-28T06:07:53.130 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 33, 2024-05-28T06:07:53.130 INFO:teuthology.orchestra.run.smithi096.stdout: "num_osds": 8, 2024-05-28T06:07:53.130 INFO:teuthology.orchestra.run.smithi096.stdout: "num_up_osds": 8, 2024-05-28T06:07:53.130 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_up_since": 1716876238, 2024-05-28T06:07:53.130 INFO:teuthology.orchestra.run.smithi096.stdout: "num_in_osds": 8, 2024-05-28T06:07:53.130 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_in_since": 1716876229, 2024-05-28T06:07:53.130 INFO:teuthology.orchestra.run.smithi096.stdout: "num_remapped_pgs": 0 2024-05-28T06:07:53.130 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:07:53.130 INFO:teuthology.orchestra.run.smithi096.stdout: "pgmap": { 2024-05-28T06:07:53.130 INFO:teuthology.orchestra.run.smithi096.stdout: "pgs_by_state": [ 2024-05-28T06:07:53.131 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:07:53.131 INFO:teuthology.orchestra.run.smithi096.stdout: "state_name": "active+clean", 2024-05-28T06:07:53.131 INFO:teuthology.orchestra.run.smithi096.stdout: "count": 97 2024-05-28T06:07:53.131 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:07:53.131 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:07:53.131 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pgs": 97, 2024-05-28T06:07:53.131 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pools": 3, 2024-05-28T06:07:53.131 INFO:teuthology.orchestra.run.smithi096.stdout: "num_objects": 2588, 2024-05-28T06:07:53.131 INFO:teuthology.orchestra.run.smithi096.stdout: "data_bytes": 10738676087, 2024-05-28T06:07:53.131 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_used": 21735825408, 2024-05-28T06:07:53.131 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_avail": 751358287872, 2024-05-28T06:07:53.131 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_total": 773094113280, 2024-05-28T06:07:53.131 INFO:teuthology.orchestra.run.smithi096.stdout: "read_bytes_sec": 170, 2024-05-28T06:07:53.131 INFO:teuthology.orchestra.run.smithi096.stdout: "write_bytes_sec": 89805726, 2024-05-28T06:07:53.132 INFO:teuthology.orchestra.run.smithi096.stdout: "read_op_per_sec": 0, 2024-05-28T06:07:53.132 INFO:teuthology.orchestra.run.smithi096.stdout: "write_op_per_sec": 24 2024-05-28T06:07:53.132 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:07:53.132 INFO:teuthology.orchestra.run.smithi096.stdout: "fsmap": { 2024-05-28T06:07:53.132 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 16, 2024-05-28T06:07:53.132 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 2, 2024-05-28T06:07:53.132 INFO:teuthology.orchestra.run.smithi096.stdout: "up": 1, 2024-05-28T06:07:53.132 INFO:teuthology.orchestra.run.smithi096.stdout: "in": 1, 2024-05-28T06:07:53.132 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 1, 2024-05-28T06:07:53.132 INFO:teuthology.orchestra.run.smithi096.stdout: "by_rank": [ 2024-05-28T06:07:53.132 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:07:53.132 INFO:teuthology.orchestra.run.smithi096.stdout: "filesystem_id": 2, 2024-05-28T06:07:53.132 INFO:teuthology.orchestra.run.smithi096.stdout: "rank": 0, 2024-05-28T06:07:53.132 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "d", 2024-05-28T06:07:53.133 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "up:active", 2024-05-28T06:07:53.133 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 5238 2024-05-28T06:07:53.133 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:07:53.133 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:07:53.133 INFO:teuthology.orchestra.run.smithi096.stdout: "up:standby": 3 2024-05-28T06:07:53.133 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:07:53.133 INFO:teuthology.orchestra.run.smithi096.stdout: "mgrmap": { 2024-05-28T06:07:53.133 INFO:teuthology.orchestra.run.smithi096.stdout: "available": true, 2024-05-28T06:07:53.133 INFO:teuthology.orchestra.run.smithi096.stdout: "num_standbys": 1, 2024-05-28T06:07:53.133 INFO:teuthology.orchestra.run.smithi096.stdout: "modules": [ 2024-05-28T06:07:53.133 INFO:teuthology.orchestra.run.smithi096.stdout: "iostat", 2024-05-28T06:07:53.133 INFO:teuthology.orchestra.run.smithi096.stdout: "nfs", 2024-05-28T06:07:53.133 INFO:teuthology.orchestra.run.smithi096.stdout: "restful" 2024-05-28T06:07:53.133 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:07:53.134 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:07:53.134 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:07:53.134 INFO:teuthology.orchestra.run.smithi096.stdout: "servicemap": { 2024-05-28T06:07:53.134 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 7, 2024-05-28T06:07:53.134 INFO:teuthology.orchestra.run.smithi096.stdout: "modified": "2024-05-28T06:06:33.535436+0000", 2024-05-28T06:07:53.134 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:07:53.134 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:07:53.134 INFO:teuthology.orchestra.run.smithi096.stdout: "progress_events": {} 2024-05-28T06:07:53.134 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:07:53.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:53.133+0000 7f0bb6200640 1 -- 172.21.15.96:0/469381316 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0b9006caa0 msgr2=0x7f0b9006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:07:53.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:53.133+0000 7f0bb6200640 1 --2- 172.21.15.96:0/469381316 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0b9006caa0 0x7f0b9006ef60 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f0ba0009890 tx=0x7f0ba0008040 comp rx=0 tx=0).stop 2024-05-28T06:07:53.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:53.133+0000 7f0bb6200640 1 -- 172.21.15.96:0/469381316 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0bb015ed60 msgr2=0x7f0bb00af270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:07:53.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:53.133+0000 7f0bb6200640 1 --2- 172.21.15.96:0/469381316 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0bb015ed60 0x7f0bb00af270 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f0ba8009e20 tx=0x7f0ba8004a40 comp rx=0 tx=0).stop 2024-05-28T06:07:53.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:53.133+0000 7f0bb6200640 1 -- 172.21.15.96:0/469381316 shutdown_connections 2024-05-28T06:07:53.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:53.133+0000 7f0bb6200640 1 --2- 172.21.15.96:0/469381316 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0b9006caa0 0x7f0b9006ef60 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:53.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:53.133+0000 7f0bb6200640 1 --2- 172.21.15.96:0/469381316 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0bb015ed60 0x7f0bb00af270 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:53.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:53.133+0000 7f0bb6200640 1 --2- 172.21.15.96:0/469381316 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0bb00cba20 0x7f0bb00b54c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:53.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:53.133+0000 7f0bb6200640 1 --2- 172.21.15.96:0/469381316 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0bb006b900 0x7f0bb0150940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:53.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:53.133+0000 7f0bb6200640 1 -- 172.21.15.96:0/469381316 >> 172.21.15.96:0/469381316 conn(0x7f0bb009aa30 msgr2=0x7f0bb00b0720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:07:53.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:53.133+0000 7f0bb6200640 1 -- 172.21.15.96:0/469381316 shutdown_connections 2024-05-28T06:07:53.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:53.134+0000 7f0bb6200640 1 -- 172.21.15.96:0/469381316 wait complete. 2024-05-28T06:07:55.161 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-05-28T06:07:55.395 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.395+0000 7efc02000640 1 Processor -- start 2024-05-28T06:07:55.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.395+0000 7efc02000640 1 -- start start 2024-05-28T06:07:55.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.398+0000 7efc02000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7efbfc15b850 0x7efbfc06ed30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:07:55.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.398+0000 7efc02000640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efbfc0b3a10 con 0x7efbfc0b3380 2024-05-28T06:07:55.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.398+0000 7efc02000640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efbfc0b3cf0 con 0x7efbfc071cc0 2024-05-28T06:07:55.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.398+0000 7efc02000640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7efbfc06e190 con 0x7efbfc15b850 2024-05-28T06:07:55.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.399+0000 7efbfb400640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7efbfc071cc0 0x7efbfc0720a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.96:37178/0 (socket says 172.21.15.96:37178) 2024-05-28T06:07:55.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.399+0000 7efbfb400640 1 -- 172.21.15.96:0/960966017 learned_addr learned my addr 172.21.15.96:0/960966017 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:07:55.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.399+0000 7efbfbe00640 1 --2- 172.21.15.96:0/960966017 >> v2:172.21.15.96:3300/0 conn(0x7efbfc15b850 0x7efbfc06ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:07:55.400 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.400+0000 7efbfa000640 1 -- 172.21.15.96:0/960966017 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1168999701 0 0) 0x7efbfc0b3a10 con 0x7efbfc0b3380 2024-05-28T06:07:55.400 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.400+0000 7efbfa000640 1 -- 172.21.15.96:0/960966017 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efbe0003520 con 0x7efbfc0b3380 2024-05-28T06:07:55.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.400+0000 7efbfa000640 1 -- 172.21.15.96:0/960966017 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2482553971 0 0) 0x7efbfc0b3cf0 con 0x7efbfc071cc0 2024-05-28T06:07:55.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.400+0000 7efbfa000640 1 -- 172.21.15.96:0/960966017 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efbfc0b3a10 con 0x7efbfc071cc0 2024-05-28T06:07:55.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.400+0000 7efbfa000640 1 -- 172.21.15.96:0/960966017 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 550637158 0 0) 0x7efbe0003520 con 0x7efbfc0b3380 2024-05-28T06:07:55.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.400+0000 7efbfa000640 1 -- 172.21.15.96:0/960966017 >> v2:172.21.15.96:3300/0 conn(0x7efbfc15b850 msgr2=0x7efbfc06ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:07:55.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.400+0000 7efbfa000640 1 --2- 172.21.15.96:0/960966017 >> v2:172.21.15.96:3300/0 conn(0x7efbfc15b850 0x7efbfc06ed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:55.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.400+0000 7efbfa000640 1 -- 172.21.15.96:0/960966017 >> v1:172.21.15.123:6789/0 conn(0x7efbfc071cc0 legacy=0x7efbfc0720a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:07:55.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.400+0000 7efbfa000640 1 -- 172.21.15.96:0/960966017 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efbfc0b3fd0 con 0x7efbfc0b3380 2024-05-28T06:07:55.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.400+0000 7efbfa000640 1 -- 172.21.15.96:0/960966017 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7efbe8001670 con 0x7efbfc0b3380 2024-05-28T06:07:55.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.400+0000 7efbfa000640 1 -- 172.21.15.96:0/960966017 >> v1:172.21.15.96:6789/0 conn(0x7efbfc0b3380 legacy=0x7efbfc06e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:07:55.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.400+0000 7efbfa000640 1 --2- 172.21.15.96:0/960966017 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efbe0003be0 0x7efbe0003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:07:55.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.401+0000 7efbfa000640 1 --2- 172.21.15.96:0/960966017 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efbe0004800 0x7efbe0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:07:55.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.401+0000 7efbfa000640 1 --2- 172.21.15.96:0/960966017 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efbe00051c0 0x7efbe000ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:07:55.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.401+0000 7efbfa000640 1 -- 172.21.15.96:0/960966017 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7efbfc06e190 con 0x7efbe0004800 2024-05-28T06:07:55.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.401+0000 7efbfa000640 1 -- 172.21.15.96:0/960966017 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7efbe0010510 con 0x7efbe00051c0 2024-05-28T06:07:55.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.401+0000 7efbfa000640 1 -- 172.21.15.96:0/960966017 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7efbe0010710 con 0x7efbe0003be0 2024-05-28T06:07:55.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.400+0000 7efbfbe00640 1 --2- 172.21.15.96:0/960966017 >> v2:172.21.15.96:3300/0 conn(0x7efbfc15b850 0x7efbfc06ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-28T06:07:55.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.401+0000 7efbfb400640 1 --2- 172.21.15.96:0/960966017 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efbe0003be0 0x7efbe0003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:07:55.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.402+0000 7efbfbe00640 1 --2- 172.21.15.96:0/960966017 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efbe00051c0 0x7efbe000ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:07:55.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.402+0000 7efbfaa00640 1 --2- 172.21.15.96:0/960966017 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efbe0004800 0x7efbe0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:07:55.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.402+0000 7efbfbe00640 1 -- 172.21.15.96:0/960966017 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efbe0003be0 msgr2=0x7efbe0003fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:07:55.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.402+0000 7efbfbe00640 1 --2- 172.21.15.96:0/960966017 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efbe0003be0 0x7efbe0003fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:55.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.402+0000 7efbfbe00640 1 -- 172.21.15.96:0/960966017 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efbe0004800 msgr2=0x7efbe0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:07:55.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.402+0000 7efbfbe00640 1 --2- 172.21.15.96:0/960966017 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efbe0004800 0x7efbe0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:55.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.402+0000 7efbfbe00640 1 -- 172.21.15.96:0/960966017 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efbe0003520 con 0x7efbe00051c0 2024-05-28T06:07:55.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.403+0000 7efbfaa00640 1 --2- 172.21.15.96:0/960966017 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efbe0004800 0x7efbe0004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-28T06:07:55.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.403+0000 7efbfbe00640 1 -- 172.21.15.96:0/960966017 reap_dead start 2024-05-28T06:07:55.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.404+0000 7efbfbe00640 1 --2- 172.21.15.96:0/960966017 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efbe00051c0 0x7efbe000ffd0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7efbf400e380 tx=0x7efbe0005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1e12d95ed6aed2f6 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:07:55.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.406+0000 7efbfa000640 1 -- 172.21.15.96:0/960966017 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbe8007af0 con 0x7efbe00051c0 2024-05-28T06:07:55.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.406+0000 7efbfa000640 1 -- 172.21.15.96:0/960966017 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efbf402c620 con 0x7efbe00051c0 2024-05-28T06:07:55.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.406+0000 7efbfa000640 1 -- 172.21.15.96:0/960966017 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbf4009de0 con 0x7efbe00051c0 2024-05-28T06:07:55.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.407+0000 7efc02000640 1 -- 172.21.15.96:0/960966017 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efbe00051c0 msgr2=0x7efbe000ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:07:55.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.407+0000 7efc02000640 1 --2- 172.21.15.96:0/960966017 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efbe00051c0 0x7efbe000ffd0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7efbf400e380 tx=0x7efbe0005830 comp rx=0 tx=0).stop 2024-05-28T06:07:55.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.407+0000 7efc02000640 1 -- 172.21.15.96:0/960966017 shutdown_connections 2024-05-28T06:07:55.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.407+0000 7efc02000640 1 --2- 172.21.15.96:0/960966017 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efbe00051c0 0x7efbe000ffd0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:55.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.407+0000 7efc02000640 1 -- 172.21.15.96:0/960966017 >> 172.21.15.96:0/960966017 conn(0x7efbfc06bae0 msgr2=0x7efbfc06bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:07:55.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.408+0000 7efc02000640 1 -- 172.21.15.96:0/960966017 shutdown_connections 2024-05-28T06:07:55.412 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.409+0000 7efc02000640 1 -- 172.21.15.96:0/960966017 wait complete. 2024-05-28T06:07:55.412 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.412+0000 7efc02000640 1 Processor -- start 2024-05-28T06:07:55.412 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.412+0000 7efc02000640 1 -- start start 2024-05-28T06:07:55.415 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.414+0000 7efc02000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efbe00051c0 0x7efbfc0aeda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:07:55.415 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.414+0000 7efc02000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efbfc1506c0 0x7efbfc144f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:07:55.415 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.414+0000 7efc02000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efbfc0a9490 0x7efbfc0a9940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:07:55.415 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.414+0000 7efc02000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7efbfc1550d0 con 0x7efbfc0a9490 2024-05-28T06:07:55.415 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.414+0000 7efc02000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7efbfc15bcc0 con 0x7efbfc1506c0 2024-05-28T06:07:55.415 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.414+0000 7efc02000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7efbfc155250 con 0x7efbe00051c0 2024-05-28T06:07:55.416 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.415+0000 7efbfb400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efbe00051c0 0x7efbfc0aeda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:07:55.416 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.415+0000 7efbfb400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efbe00051c0 0x7efbfc0aeda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.96:54888/0 (socket says 172.21.15.96:54888) 2024-05-28T06:07:55.416 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.415+0000 7efbfb400640 1 -- 172.21.15.96:0/3493841683 learned_addr learned my addr 172.21.15.96:0/3493841683 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:07:55.416 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.415+0000 7efbfbe00640 1 --2- 172.21.15.96:0/3493841683 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efbfc0a9490 0x7efbfc0a9940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:07:55.417 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.415+0000 7efbfaa00640 1 --2- 172.21.15.96:0/3493841683 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efbfc1506c0 0x7efbfc144f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:07:55.417 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.416+0000 7efbfbe00640 1 -- 172.21.15.96:0/3493841683 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efbe00051c0 msgr2=0x7efbfc0aeda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:07:55.417 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.416+0000 7efbfbe00640 1 --2- 172.21.15.96:0/3493841683 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efbe00051c0 0x7efbfc0aeda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:55.417 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.416+0000 7efbfbe00640 1 -- 172.21.15.96:0/3493841683 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efbfc1506c0 msgr2=0x7efbfc144f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:07:55.417 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.416+0000 7efbfbe00640 1 --2- 172.21.15.96:0/3493841683 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efbfc1506c0 0x7efbfc144f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:55.417 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.416+0000 7efbfbe00640 1 -- 172.21.15.96:0/3493841683 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efbfc1464c0 con 0x7efbfc0a9490 2024-05-28T06:07:55.417 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.417+0000 7efbfbe00640 1 --2- 172.21.15.96:0/3493841683 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efbfc0a9490 0x7efbfc0a9940 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7efbfc0cff20 tx=0x7efbf4009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:07:55.418 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.417+0000 7efbf3400640 1 -- 172.21.15.96:0/3493841683 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbf4009de0 con 0x7efbfc0a9490 2024-05-28T06:07:55.418 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.417+0000 7efbf3400640 1 -- 172.21.15.96:0/3493841683 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efbf402c420 con 0x7efbfc0a9490 2024-05-28T06:07:55.418 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.417+0000 7efbf3400640 1 -- 172.21.15.96:0/3493841683 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbf4037af0 con 0x7efbfc0a9490 2024-05-28T06:07:55.418 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.418+0000 7efc02000640 1 -- 172.21.15.96:0/3493841683 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efbfc1458b0 con 0x7efbfc0a9490 2024-05-28T06:07:55.419 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.419+0000 7efc02000640 1 -- 172.21.15.96:0/3493841683 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efbfc145e70 con 0x7efbfc0a9490 2024-05-28T06:07:55.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.421+0000 7efc02000640 1 -- 172.21.15.96:0/3493841683 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efbfc0d02d0 con 0x7efbfc0a9490 2024-05-28T06:07:55.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.423+0000 7efbf3400640 1 -- 172.21.15.96:0/3493841683 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 8) ==== 91193+0+0 (secure 0 0 0) 0x7efbf4037c90 con 0x7efbfc0a9490 2024-05-28T06:07:55.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.425+0000 7efbf3400640 1 --2- 172.21.15.96:0/3493841683 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7efbdc06caf0 0x7efbdc06efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:07:55.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.425+0000 7efbf3400640 1 -- 172.21.15.96:0/3493841683 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7efbf40ac6b0 con 0x7efbfc0a9490 2024-05-28T06:07:55.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.435+0000 7efbfb400640 1 --2- 172.21.15.96:0/3493841683 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7efbdc06caf0 0x7efbdc06efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:07:55.436 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.435+0000 7efbf3400640 1 -- 172.21.15.96:0/3493841683 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7efbfc0d02d0 con 0x7efbfc0a9490 2024-05-28T06:07:55.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.437+0000 7efbfb400640 1 --2- 172.21.15.96:0/3493841683 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7efbdc06caf0 0x7efbdc06efb0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7efbec004f30 tx=0x7efbec007d90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:07:55.852 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.852+0000 7efc02000640 1 -- 172.21.15.96:0/3493841683 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7efbfc06fb00 con 0x7efbfc0a9490 2024-05-28T06:07:55.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.852+0000 7efbf3400640 1 -- 172.21.15.96:0/3493841683 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+2095 (secure 0 0 0) 0x7efbfc06fb00 con 0x7efbfc0a9490 2024-05-28T06:07:55.853 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:07:55.853 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:07:55.853 INFO:teuthology.orchestra.run.smithi096.stdout: "fsid": "e4a831b5-e6d1-47a7-96cb-bf85154ee0d2", 2024-05-28T06:07:55.853 INFO:teuthology.orchestra.run.smithi096.stdout: "health": { 2024-05-28T06:07:55.853 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "HEALTH_OK", 2024-05-28T06:07:55.853 INFO:teuthology.orchestra.run.smithi096.stdout: "checks": {}, 2024-05-28T06:07:55.853 INFO:teuthology.orchestra.run.smithi096.stdout: "mutes": [] 2024-05-28T06:07:55.853 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:07:55.854 INFO:teuthology.orchestra.run.smithi096.stdout: "election_epoch": 8, 2024-05-28T06:07:55.854 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum": [ 2024-05-28T06:07:55.854 INFO:teuthology.orchestra.run.smithi096.stdout: 0, 2024-05-28T06:07:55.854 INFO:teuthology.orchestra.run.smithi096.stdout: 1, 2024-05-28T06:07:55.854 INFO:teuthology.orchestra.run.smithi096.stdout: 2 2024-05-28T06:07:55.854 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:07:55.854 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_names": [ 2024-05-28T06:07:55.854 INFO:teuthology.orchestra.run.smithi096.stdout: "a", 2024-05-28T06:07:55.854 INFO:teuthology.orchestra.run.smithi096.stdout: "b", 2024-05-28T06:07:55.854 INFO:teuthology.orchestra.run.smithi096.stdout: "c" 2024-05-28T06:07:55.854 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:07:55.855 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_age": 249, 2024-05-28T06:07:55.855 INFO:teuthology.orchestra.run.smithi096.stdout: "monmap": { 2024-05-28T06:07:55.855 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 1, 2024-05-28T06:07:55.855 INFO:teuthology.orchestra.run.smithi096.stdout: "min_mon_release_name": "squid", 2024-05-28T06:07:55.855 INFO:teuthology.orchestra.run.smithi096.stdout: "num_mons": 3 2024-05-28T06:07:55.855 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:07:55.855 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap": { 2024-05-28T06:07:55.855 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 33, 2024-05-28T06:07:55.855 INFO:teuthology.orchestra.run.smithi096.stdout: "num_osds": 8, 2024-05-28T06:07:55.855 INFO:teuthology.orchestra.run.smithi096.stdout: "num_up_osds": 8, 2024-05-28T06:07:55.855 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_up_since": 1716876238, 2024-05-28T06:07:55.855 INFO:teuthology.orchestra.run.smithi096.stdout: "num_in_osds": 8, 2024-05-28T06:07:55.855 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_in_since": 1716876229, 2024-05-28T06:07:55.856 INFO:teuthology.orchestra.run.smithi096.stdout: "num_remapped_pgs": 0 2024-05-28T06:07:55.856 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:07:55.856 INFO:teuthology.orchestra.run.smithi096.stdout: "pgmap": { 2024-05-28T06:07:55.856 INFO:teuthology.orchestra.run.smithi096.stdout: "pgs_by_state": [ 2024-05-28T06:07:55.856 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:07:55.856 INFO:teuthology.orchestra.run.smithi096.stdout: "state_name": "active+clean", 2024-05-28T06:07:55.856 INFO:teuthology.orchestra.run.smithi096.stdout: "count": 97 2024-05-28T06:07:55.856 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:07:55.856 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:07:55.856 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pgs": 97, 2024-05-28T06:07:55.856 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pools": 3, 2024-05-28T06:07:55.856 INFO:teuthology.orchestra.run.smithi096.stdout: "num_objects": 2588, 2024-05-28T06:07:55.856 INFO:teuthology.orchestra.run.smithi096.stdout: "data_bytes": 10738676087, 2024-05-28T06:07:55.856 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_used": 21735825408, 2024-05-28T06:07:55.857 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_avail": 751358287872, 2024-05-28T06:07:55.857 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_total": 773094113280, 2024-05-28T06:07:55.857 INFO:teuthology.orchestra.run.smithi096.stdout: "read_bytes_sec": 170, 2024-05-28T06:07:55.857 INFO:teuthology.orchestra.run.smithi096.stdout: "write_bytes_sec": 49271413, 2024-05-28T06:07:55.857 INFO:teuthology.orchestra.run.smithi096.stdout: "read_op_per_sec": 0, 2024-05-28T06:07:55.857 INFO:teuthology.orchestra.run.smithi096.stdout: "write_op_per_sec": 14 2024-05-28T06:07:55.857 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:07:55.857 INFO:teuthology.orchestra.run.smithi096.stdout: "fsmap": { 2024-05-28T06:07:55.857 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 16, 2024-05-28T06:07:55.857 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 2, 2024-05-28T06:07:55.857 INFO:teuthology.orchestra.run.smithi096.stdout: "up": 1, 2024-05-28T06:07:55.857 INFO:teuthology.orchestra.run.smithi096.stdout: "in": 1, 2024-05-28T06:07:55.857 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 1, 2024-05-28T06:07:55.857 INFO:teuthology.orchestra.run.smithi096.stdout: "by_rank": [ 2024-05-28T06:07:55.858 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:07:55.858 INFO:teuthology.orchestra.run.smithi096.stdout: "filesystem_id": 2, 2024-05-28T06:07:55.858 INFO:teuthology.orchestra.run.smithi096.stdout: "rank": 0, 2024-05-28T06:07:55.858 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "d", 2024-05-28T06:07:55.858 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "up:active", 2024-05-28T06:07:55.858 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 5238 2024-05-28T06:07:55.858 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:07:55.858 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:07:55.858 INFO:teuthology.orchestra.run.smithi096.stdout: "up:standby": 3 2024-05-28T06:07:55.858 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:07:55.858 INFO:teuthology.orchestra.run.smithi096.stdout: "mgrmap": { 2024-05-28T06:07:55.858 INFO:teuthology.orchestra.run.smithi096.stdout: "available": true, 2024-05-28T06:07:55.859 INFO:teuthology.orchestra.run.smithi096.stdout: "num_standbys": 1, 2024-05-28T06:07:55.859 INFO:teuthology.orchestra.run.smithi096.stdout: "modules": [ 2024-05-28T06:07:55.859 INFO:teuthology.orchestra.run.smithi096.stdout: "iostat", 2024-05-28T06:07:55.859 INFO:teuthology.orchestra.run.smithi096.stdout: "nfs", 2024-05-28T06:07:55.859 INFO:teuthology.orchestra.run.smithi096.stdout: "restful" 2024-05-28T06:07:55.859 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:07:55.859 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:07:55.859 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:07:55.859 INFO:teuthology.orchestra.run.smithi096.stdout: "servicemap": { 2024-05-28T06:07:55.859 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 7, 2024-05-28T06:07:55.859 INFO:teuthology.orchestra.run.smithi096.stdout: "modified": "2024-05-28T06:06:33.535436+0000", 2024-05-28T06:07:55.859 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:07:55.859 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:07:55.860 INFO:teuthology.orchestra.run.smithi096.stdout: "progress_events": { 2024-05-28T06:07:55.860 INFO:teuthology.orchestra.run.smithi096.stdout: "18e91b7c-9da2-4cf3-88f8-e7e2859c593b": { 2024-05-28T06:07:55.860 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "1 ongoing clones; Avg progress is 0.0% (1s)\n [............................] ", 2024-05-28T06:07:55.860 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": 0, 2024-05-28T06:07:55.860 INFO:teuthology.orchestra.run.smithi096.stdout: "add_to_ceph_s": true 2024-05-28T06:07:55.860 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:07:55.860 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:07:55.860 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:07:55.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.855+0000 7efc02000640 1 -- 172.21.15.96:0/3493841683 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7efbdc06caf0 msgr2=0x7efbdc06efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:07:55.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.855+0000 7efc02000640 1 --2- 172.21.15.96:0/3493841683 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7efbdc06caf0 0x7efbdc06efb0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7efbec004f30 tx=0x7efbec007d90 comp rx=0 tx=0).stop 2024-05-28T06:07:55.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.855+0000 7efc02000640 1 -- 172.21.15.96:0/3493841683 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efbfc0a9490 msgr2=0x7efbfc0a9940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:07:55.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.855+0000 7efc02000640 1 --2- 172.21.15.96:0/3493841683 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efbfc0a9490 0x7efbfc0a9940 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7efbfc0cff20 tx=0x7efbf4009750 comp rx=0 tx=0).stop 2024-05-28T06:07:55.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.855+0000 7efc02000640 1 -- 172.21.15.96:0/3493841683 shutdown_connections 2024-05-28T06:07:55.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.855+0000 7efc02000640 1 --2- 172.21.15.96:0/3493841683 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7efbdc06caf0 0x7efbdc06efb0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:55.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.855+0000 7efc02000640 1 --2- 172.21.15.96:0/3493841683 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efbfc0a9490 0x7efbfc0a9940 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:55.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.855+0000 7efc02000640 1 --2- 172.21.15.96:0/3493841683 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efbfc1506c0 0x7efbfc144f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:55.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.855+0000 7efc02000640 1 --2- 172.21.15.96:0/3493841683 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efbe00051c0 0x7efbfc0aeda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:55.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.855+0000 7efc02000640 1 -- 172.21.15.96:0/3493841683 >> 172.21.15.96:0/3493841683 conn(0x7efbfc06bae0 msgr2=0x7efbfc06e470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:07:55.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.855+0000 7efc02000640 1 -- 172.21.15.96:0/3493841683 shutdown_connections 2024-05-28T06:07:55.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:55.855+0000 7efc02000640 1 -- 172.21.15.96:0/3493841683 wait complete. 2024-05-28T06:07:55.881 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs ss1clone1 2024-05-28T06:07:56.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.139+0000 7f5503c00640 1 Processor -- start 2024-05-28T06:07:56.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.139+0000 7f5503c00640 1 -- start start 2024-05-28T06:07:56.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.140+0000 7f5503c00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f54fc15e850 0x7f54fc0b77b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:07:56.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.140+0000 7f5503c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f54fc0cef30 0x7f54fc0d1640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:07:56.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.140+0000 7f5503c00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f54fc0b8000 con 0x7f54fc0d1b80 2024-05-28T06:07:56.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.140+0000 7f5503c00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f54fc0d1ff0 con 0x7f54fc15e850 2024-05-28T06:07:56.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.140+0000 7f5503c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f54fc15ec30 con 0x7f54fc0cef30 2024-05-28T06:07:56.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.140+0000 7f5501200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f54fc15e850 0x7f54fc0b77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:07:56.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.140+0000 7f5501200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f54fc15e850 0x7f54fc0b77b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49734/0 (socket says 172.21.15.96:49734) 2024-05-28T06:07:56.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.140+0000 7f5501200640 1 -- 172.21.15.96:0/3372303149 learned_addr learned my addr 172.21.15.96:0/3372303149 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:07:56.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.141+0000 7f54fbe00640 1 --2- 172.21.15.96:0/3372303149 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f54fc0cef30 0x7f54fc0d1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:07:56.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.141+0000 7f54fb400640 1 -- 172.21.15.96:0/3372303149 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3656164410 0 0) 0x7f54fc0b8000 con 0x7f54fc0d1b80 2024-05-28T06:07:56.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.141+0000 7f54fb400640 1 -- 172.21.15.96:0/3372303149 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f54e4003520 con 0x7f54fc0d1b80 2024-05-28T06:07:56.143 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.141+0000 7f5501200640 1 -- 172.21.15.96:0/3372303149 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f54fc0cef30 msgr2=0x7f54fc0d1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:07:56.143 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.141+0000 7f5501200640 1 --2- 172.21.15.96:0/3372303149 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f54fc0cef30 0x7f54fc0d1640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:56.143 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.141+0000 7f5501200640 1 -- 172.21.15.96:0/3372303149 >> v1:172.21.15.123:6789/0 conn(0x7f54fc0d1b80 legacy=0x7f54fc162340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:07:56.143 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.141+0000 7f5501200640 1 -- 172.21.15.96:0/3372303149 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54fc0b82e0 con 0x7f54fc15e850 2024-05-28T06:07:56.143 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.142+0000 7f54fbe00640 1 --2- 172.21.15.96:0/3372303149 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f54fc0cef30 0x7f54fc0d1640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-28T06:07:56.143 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.143+0000 7f5501200640 1 --2- 172.21.15.96:0/3372303149 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f54fc15e850 0x7f54fc0b77b0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f54f0009870 tx=0x7f54f002a1d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a10a9380f6afbc3 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:07:56.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.147+0000 7f54fb400640 1 -- 172.21.15.96:0/3372303149 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54f0037070 con 0x7f54fc15e850 2024-05-28T06:07:56.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.147+0000 7f54fb400640 1 -- 172.21.15.96:0/3372303149 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54f0032440 con 0x7f54fc15e850 2024-05-28T06:07:56.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.147+0000 7f54fb400640 1 -- 172.21.15.96:0/3372303149 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54f0032810 con 0x7f54fc15e850 2024-05-28T06:07:56.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.149+0000 7f5503c00640 1 -- 172.21.15.96:0/3372303149 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f54fc15e850 msgr2=0x7f54fc0b77b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:07:56.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.149+0000 7f5503c00640 1 --2- 172.21.15.96:0/3372303149 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f54fc15e850 0x7f54fc0b77b0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f54f0009870 tx=0x7f54f002a1d0 comp rx=0 tx=0).stop 2024-05-28T06:07:56.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.151+0000 7f5503c00640 1 -- 172.21.15.96:0/3372303149 shutdown_connections 2024-05-28T06:07:56.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.151+0000 7f5503c00640 1 --2- 172.21.15.96:0/3372303149 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f54fc0cef30 0x7f54fc0d1640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:56.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.151+0000 7f5503c00640 1 --2- 172.21.15.96:0/3372303149 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f54fc15e850 0x7f54fc0b77b0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:56.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.151+0000 7f5503c00640 1 -- 172.21.15.96:0/3372303149 >> 172.21.15.96:0/3372303149 conn(0x7f54fc067aa0 msgr2=0x7f54fc067eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:07:56.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.151+0000 7f5503c00640 1 -- 172.21.15.96:0/3372303149 shutdown_connections 2024-05-28T06:07:56.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.151+0000 7f5503c00640 1 -- 172.21.15.96:0/3372303149 wait complete. 2024-05-28T06:07:56.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.155+0000 7f5503c00640 1 Processor -- start 2024-05-28T06:07:56.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.155+0000 7f5503c00640 1 -- start start 2024-05-28T06:07:56.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.157+0000 7f5503c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f54fc0cef30 0x7f54fc15bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:07:56.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.157+0000 7f5503c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f54fc0d1b80 0x7f54fc0cd3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:07:56.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.157+0000 7f5503c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f54fc15e850 0x7f54fc06caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:07:56.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.157+0000 7f5503c00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f54fc0b7cf0 con 0x7f54fc0cef30 2024-05-28T06:07:56.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.157+0000 7f5503c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f54fc0b88a0 con 0x7f54fc0d1b80 2024-05-28T06:07:56.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.157+0000 7f5503c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f54fc0ba0b0 con 0x7f54fc15e850 2024-05-28T06:07:56.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.158+0000 7f5501c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f54fc15e850 0x7f54fc06caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:07:56.159 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.158+0000 7f5501c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f54fc15e850 0x7f54fc06caf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.96:54908/0 (socket says 172.21.15.96:54908) 2024-05-28T06:07:56.159 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.158+0000 7f5501c00640 1 -- 172.21.15.96:0/4044607048 learned_addr learned my addr 172.21.15.96:0/4044607048 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:07:56.159 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.158+0000 7f54fbe00640 1 --2- 172.21.15.96:0/4044607048 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f54fc0d1b80 0x7f54fc0cd3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:07:56.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.159+0000 7f5501200640 1 --2- 172.21.15.96:0/4044607048 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f54fc0cef30 0x7f54fc15bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:07:56.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.160+0000 7f5501c00640 1 -- 172.21.15.96:0/4044607048 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f54fc0d1b80 msgr2=0x7f54fc0cd3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:07:56.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.160+0000 7f5501c00640 1 --2- 172.21.15.96:0/4044607048 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f54fc0d1b80 0x7f54fc0cd3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:56.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.160+0000 7f5501c00640 1 -- 172.21.15.96:0/4044607048 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f54fc0cef30 msgr2=0x7f54fc15bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:07:56.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.160+0000 7f5501c00640 1 --2- 172.21.15.96:0/4044607048 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f54fc0cef30 0x7f54fc15bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:56.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.160+0000 7f5501c00640 1 -- 172.21.15.96:0/4044607048 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54fc0cdf00 con 0x7f54fc15e850 2024-05-28T06:07:56.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.161+0000 7f5501c00640 1 --2- 172.21.15.96:0/4044607048 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f54fc15e850 0x7f54fc06caf0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f54f400c6e0 tx=0x7f54f400cba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:07:56.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.161+0000 7f54f8c00640 1 -- 172.21.15.96:0/4044607048 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54f4003b70 con 0x7f54fc15e850 2024-05-28T06:07:56.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.162+0000 7f54f8c00640 1 -- 172.21.15.96:0/4044607048 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54f4003d10 con 0x7f54fc15e850 2024-05-28T06:07:56.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.162+0000 7f5503c00640 1 -- 172.21.15.96:0/4044607048 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54fc0ce1e0 con 0x7f54fc15e850 2024-05-28T06:07:56.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.162+0000 7f54f8c00640 1 -- 172.21.15.96:0/4044607048 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54f40052d0 con 0x7f54fc15e850 2024-05-28T06:07:56.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.162+0000 7f5503c00640 1 -- 172.21.15.96:0/4044607048 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f54fc0ce7a0 con 0x7f54fc15e850 2024-05-28T06:07:56.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.164+0000 7f5503c00640 1 -- 172.21.15.96:0/4044607048 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54fc0689e0 con 0x7f54fc15e850 2024-05-28T06:07:56.166 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.166+0000 7f54f8c00640 1 -- 172.21.15.96:0/4044607048 <== mon.2 v2:172.21.15.123:3301/0 4 ==== mgrmap(e 8) ==== 91193+0+0 (secure 0 0 0) 0x7f54f4005470 con 0x7f54fc15e850 2024-05-28T06:07:56.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.168+0000 7f54f8c00640 1 --2- 172.21.15.96:0/4044607048 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f54dc06caa0 0x7f54dc06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:07:56.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.169+0000 7f54f8c00640 1 -- 172.21.15.96:0/4044607048 <== mon.2 v2:172.21.15.123:3301/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f54f408bea0 con 0x7f54fc15e850 2024-05-28T06:07:56.172 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.172+0000 7f5501200640 1 --2- 172.21.15.96:0/4044607048 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f54dc06caa0 0x7f54dc06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:07:56.174 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.174+0000 7f54f8c00640 1 -- 172.21.15.96:0/4044607048 <== mon.2 v2:172.21.15.123:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f54f4059a70 con 0x7f54fc15e850 2024-05-28T06:07:56.174 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.174+0000 7f5501200640 1 --2- 172.21.15.96:0/4044607048 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f54dc06caa0 0x7f54dc06ef60 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f54f0034000 tx=0x7f54f002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:07:56.512 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.512+0000 7f5503c00640 1 -- 172.21.15.96:0/4044607048 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) -- 0x7f54fc0b79b0 con 0x7f54dc06caa0 2024-05-28T06:07:56.699 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.698+0000 7f54f8c00640 1 -- 172.21.15.96:0/4044607048 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f54fc0b79b0 con 0x7f54dc06caa0 2024-05-28T06:07:56.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.701+0000 7f5503c00640 1 -- 172.21.15.96:0/4044607048 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f54dc06caa0 msgr2=0x7f54dc06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:07:56.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.701+0000 7f5503c00640 1 --2- 172.21.15.96:0/4044607048 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f54dc06caa0 0x7f54dc06ef60 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f54f0034000 tx=0x7f54f002e000 comp rx=0 tx=0).stop 2024-05-28T06:07:56.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.701+0000 7f5503c00640 1 -- 172.21.15.96:0/4044607048 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f54fc15e850 msgr2=0x7f54fc06caf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:07:56.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.701+0000 7f5503c00640 1 --2- 172.21.15.96:0/4044607048 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f54fc15e850 0x7f54fc06caf0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f54f400c6e0 tx=0x7f54f400cba0 comp rx=0 tx=0).stop 2024-05-28T06:07:56.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.701+0000 7f5503c00640 1 -- 172.21.15.96:0/4044607048 shutdown_connections 2024-05-28T06:07:56.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.701+0000 7f5503c00640 1 --2- 172.21.15.96:0/4044607048 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f54dc06caa0 0x7f54dc06ef60 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:56.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.701+0000 7f5503c00640 1 --2- 172.21.15.96:0/4044607048 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f54fc15e850 0x7f54fc06caf0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:56.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.701+0000 7f5503c00640 1 --2- 172.21.15.96:0/4044607048 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f54fc0d1b80 0x7f54fc0cd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:56.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.701+0000 7f5503c00640 1 --2- 172.21.15.96:0/4044607048 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f54fc0cef30 0x7f54fc15bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:56.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.701+0000 7f5503c00640 1 -- 172.21.15.96:0/4044607048 >> 172.21.15.96:0/4044607048 conn(0x7f54fc067aa0 msgr2=0x7f54fc0b59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:07:56.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.702+0000 7f5503c00640 1 -- 172.21.15.96:0/4044607048 shutdown_connections 2024-05-28T06:07:56.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.702+0000 7f5503c00640 1 -- 172.21.15.96:0/4044607048 wait complete. 2024-05-28T06:07:56.724 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-05-28T06:07:56.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.923+0000 7fb544a00640 1 Processor -- start 2024-05-28T06:07:56.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.924+0000 7fb544a00640 1 -- start start 2024-05-28T06:07:56.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.925+0000 7fb544a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb54015e870 0x7fb5400d0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:07:56.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.925+0000 7fb544a00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fb5400cfa40 0x7fb5400d25e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:07:56.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.925+0000 7fb544a00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb54015ec50 con 0x7fb5400d14d0 2024-05-28T06:07:56.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.925+0000 7fb544a00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fb5400d2150 con 0x7fb5400cfa40 2024-05-28T06:07:56.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.925+0000 7fb544a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fb5400cff80 con 0x7fb54015e870 2024-05-28T06:07:56.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.926+0000 7fb53e800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb54015e870 0x7fb5400d0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:07:56.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.926+0000 7fb53de00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7fb5400d14d0 0x7fb5400cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.96:37202/0 (socket says 172.21.15.96:37202) 2024-05-28T06:07:56.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.926+0000 7fb53de00640 1 -- 172.21.15.96:0/2227303518 learned_addr learned my addr 172.21.15.96:0/2227303518 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:07:56.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.926+0000 7fb53f200640 1 --2- 172.21.15.96:0/2227303518 >> v2:172.21.15.123:3300/0 conn(0x7fb5400cfa40 0x7fb5400d25e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:07:56.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.927+0000 7fb53e800640 1 -- 172.21.15.96:0/2227303518 >> v2:172.21.15.123:3300/0 conn(0x7fb5400cfa40 msgr2=0x7fb5400d25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:07:56.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.927+0000 7fb53e800640 1 --2- 172.21.15.96:0/2227303518 >> v2:172.21.15.123:3300/0 conn(0x7fb5400cfa40 0x7fb5400d25e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:56.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.927+0000 7fb53e800640 1 -- 172.21.15.96:0/2227303518 >> v1:172.21.15.123:6789/0 conn(0x7fb5400d14d0 legacy=0x7fb5400cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:07:56.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.927+0000 7fb53e800640 1 -- 172.21.15.96:0/2227303518 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5400d0260 con 0x7fb54015e870 2024-05-28T06:07:56.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.928+0000 7fb53e800640 1 --2- 172.21.15.96:0/2227303518 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb54015e870 0x7fb5400d0f90 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb5300097a0 tx=0x7fb53002a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=62cbd4a35cbc8db3 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:07:56.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.929+0000 7fb53d400640 1 -- 172.21.15.96:0/2227303518 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb530037070 con 0x7fb54015e870 2024-05-28T06:07:56.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.929+0000 7fb53d400640 1 -- 172.21.15.96:0/2227303518 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb530032440 con 0x7fb54015e870 2024-05-28T06:07:56.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.929+0000 7fb53d400640 1 -- 172.21.15.96:0/2227303518 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb530032810 con 0x7fb54015e870 2024-05-28T06:07:56.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.930+0000 7fb544a00640 1 -- 172.21.15.96:0/2227303518 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb54015e870 msgr2=0x7fb5400d0f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:07:56.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.930+0000 7fb544a00640 1 --2- 172.21.15.96:0/2227303518 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb54015e870 0x7fb5400d0f90 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb5300097a0 tx=0x7fb53002a180 comp rx=0 tx=0).stop 2024-05-28T06:07:56.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.930+0000 7fb544a00640 1 -- 172.21.15.96:0/2227303518 shutdown_connections 2024-05-28T06:07:56.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.930+0000 7fb544a00640 1 --2- 172.21.15.96:0/2227303518 >> v2:172.21.15.123:3300/0 conn(0x7fb5400cfa40 0x7fb5400d25e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:56.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.930+0000 7fb544a00640 1 --2- 172.21.15.96:0/2227303518 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb54015e870 0x7fb5400d0f90 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:56.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.930+0000 7fb544a00640 1 -- 172.21.15.96:0/2227303518 >> 172.21.15.96:0/2227303518 conn(0x7fb54008bfc0 msgr2=0x7fb5400d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:07:56.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.930+0000 7fb544a00640 1 -- 172.21.15.96:0/2227303518 shutdown_connections 2024-05-28T06:07:56.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.930+0000 7fb544a00640 1 -- 172.21.15.96:0/2227303518 wait complete. 2024-05-28T06:07:56.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.932+0000 7fb544a00640 1 Processor -- start 2024-05-28T06:07:56.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.932+0000 7fb544a00640 1 -- start start 2024-05-28T06:07:56.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.933+0000 7fb544a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb5400cfa40 0x7fb54015cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:07:56.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.933+0000 7fb544a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb5400d14d0 0x7fb54014f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:07:56.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.933+0000 7fb544a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb54015e870 0x7fb54015b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:07:56.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.933+0000 7fb544a00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb5400d4fb0 con 0x7fb54015e870 2024-05-28T06:07:56.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.933+0000 7fb544a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fb5400d0820 con 0x7fb5400cfa40 2024-05-28T06:07:56.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.933+0000 7fb544a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fb5400d2b20 con 0x7fb5400d14d0 2024-05-28T06:07:56.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.933+0000 7fb53f200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb54015e870 0x7fb54015b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:07:56.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.934+0000 7fb53f200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb54015e870 0x7fb54015b480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49752/0 (socket says 172.21.15.96:49752) 2024-05-28T06:07:56.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.934+0000 7fb53de00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb5400d14d0 0x7fb54014f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:07:56.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.934+0000 7fb53f200640 1 -- 172.21.15.96:0/2218055212 learned_addr learned my addr 172.21.15.96:0/2218055212 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:07:56.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.934+0000 7fb53e800640 1 --2- 172.21.15.96:0/2218055212 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb5400cfa40 0x7fb54015cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:07:56.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.934+0000 7fb53f200640 1 -- 172.21.15.96:0/2218055212 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb5400d14d0 msgr2=0x7fb54014f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:07:56.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.934+0000 7fb53f200640 1 --2- 172.21.15.96:0/2218055212 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb5400d14d0 0x7fb54014f9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:56.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.934+0000 7fb53f200640 1 -- 172.21.15.96:0/2218055212 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb5400cfa40 msgr2=0x7fb54015cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:07:56.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.934+0000 7fb53f200640 1 --2- 172.21.15.96:0/2218055212 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb5400cfa40 0x7fb54015cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:56.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.934+0000 7fb53f200640 1 -- 172.21.15.96:0/2218055212 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb54014ff10 con 0x7fb54015e870 2024-05-28T06:07:56.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.934+0000 7fb53f200640 1 --2- 172.21.15.96:0/2218055212 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb54015e870 0x7fb54015b480 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fb53800efc0 tx=0x7fb53800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:07:56.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.935+0000 7fb536a00640 1 -- 172.21.15.96:0/2218055212 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb538018020 con 0x7fb54015e870 2024-05-28T06:07:56.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.935+0000 7fb536a00640 1 -- 172.21.15.96:0/2218055212 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb538014480 con 0x7fb54015e870 2024-05-28T06:07:56.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.935+0000 7fb536a00640 1 -- 172.21.15.96:0/2218055212 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb538014870 con 0x7fb54015e870 2024-05-28T06:07:56.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.935+0000 7fb544a00640 1 -- 172.21.15.96:0/2218055212 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb54015be60 con 0x7fb54015e870 2024-05-28T06:07:56.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.935+0000 7fb544a00640 1 -- 172.21.15.96:0/2218055212 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb54015c420 con 0x7fb54015e870 2024-05-28T06:07:56.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.936+0000 7fb544a00640 1 -- 172.21.15.96:0/2218055212 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5400689e0 con 0x7fb54015e870 2024-05-28T06:07:56.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.937+0000 7fb536a00640 1 -- 172.21.15.96:0/2218055212 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 8) ==== 91193+0+0 (secure 0 0 0) 0x7fb538005d10 con 0x7fb54015e870 2024-05-28T06:07:56.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.938+0000 7fb536a00640 1 --2- 172.21.15.96:0/2218055212 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb51c06caa0 0x7fb51c06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:07:56.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.938+0000 7fb536a00640 1 -- 172.21.15.96:0/2218055212 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7fb53808cdb0 con 0x7fb54015e870 2024-05-28T06:07:56.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.939+0000 7fb53e800640 1 --2- 172.21.15.96:0/2218055212 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb51c06caa0 0x7fb51c06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:07:56.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.940+0000 7fb53e800640 1 --2- 172.21.15.96:0/2218055212 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb51c06caa0 0x7fb51c06ef60 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fb530034000 tx=0x7fb53002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:07:56.943 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:56.943+0000 7fb536a00640 1 -- 172.21.15.96:0/2218055212 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb538091070 con 0x7fb54015e870 2024-05-28T06:07:57.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:57.168+0000 7fb544a00640 1 -- 172.21.15.96:0/2218055212 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7fb54006e350 con 0x7fb51c06caa0 2024-05-28T06:07:57.172 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:57.172+0000 7fb536a00640 1 -- 172.21.15.96:0/2218055212 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7fb54006e350 con 0x7fb51c06caa0 2024-05-28T06:07:57.172 INFO:teuthology.orchestra.run.smithi096.stdout:[ 2024-05-28T06:07:57.172 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:07:57.172 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "cephfs" 2024-05-28T06:07:57.173 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:07:57.173 INFO:teuthology.orchestra.run.smithi096.stdout:] 2024-05-28T06:07:57.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:57.175+0000 7fb544a00640 1 -- 172.21.15.96:0/2218055212 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb51c06caa0 msgr2=0x7fb51c06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:07:57.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:57.175+0000 7fb544a00640 1 --2- 172.21.15.96:0/2218055212 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb51c06caa0 0x7fb51c06ef60 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fb530034000 tx=0x7fb53002e000 comp rx=0 tx=0).stop 2024-05-28T06:07:57.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:57.175+0000 7fb544a00640 1 -- 172.21.15.96:0/2218055212 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb54015e870 msgr2=0x7fb54015b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:07:57.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:57.175+0000 7fb544a00640 1 --2- 172.21.15.96:0/2218055212 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb54015e870 0x7fb54015b480 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fb53800efc0 tx=0x7fb53800c490 comp rx=0 tx=0).stop 2024-05-28T06:07:57.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:57.175+0000 7fb544a00640 1 -- 172.21.15.96:0/2218055212 shutdown_connections 2024-05-28T06:07:57.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:57.175+0000 7fb544a00640 1 --2- 172.21.15.96:0/2218055212 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb51c06caa0 0x7fb51c06ef60 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:57.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:57.175+0000 7fb544a00640 1 --2- 172.21.15.96:0/2218055212 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb54015e870 0x7fb54015b480 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:57.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:57.175+0000 7fb544a00640 1 --2- 172.21.15.96:0/2218055212 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb5400d14d0 0x7fb54014f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:57.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:57.175+0000 7fb544a00640 1 --2- 172.21.15.96:0/2218055212 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb5400cfa40 0x7fb54015cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:57.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:57.175+0000 7fb544a00640 1 -- 172.21.15.96:0/2218055212 >> 172.21.15.96:0/2218055212 conn(0x7fb54008bfc0 msgr2=0x7fb5400b2f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:07:57.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:57.175+0000 7fb544a00640 1 -- 172.21.15.96:0/2218055212 shutdown_connections 2024-05-28T06:07:57.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:57.175+0000 7fb544a00640 1 -- 172.21.15.96:0/2218055212 wait complete. 2024-05-28T06:07:57.195 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-05-28T06:07:57.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:57.394+0000 7f9c76e00640 1 Processor -- start 2024-05-28T06:07:57.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:57.394+0000 7f9c76e00640 1 -- start start 2024-05-28T06:07:57.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:57.395+0000 7f9c76e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f9c7015baa0 0x7f9c700d0970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:07:57.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:57.395+0000 7f9c76e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9c7006f390 0x7f9c7006f800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:07:57.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:57.395+0000 7f9c76e00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c7006e3d0 con 0x7f9c70155a00 2024-05-28T06:07:57.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:57.395+0000 7f9c76e00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f9c7006fd40 con 0x7f9c7015baa0 2024-05-28T06:07:57.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:57.395+0000 7f9c76e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f9c700ccf20 con 0x7f9c7006f390 2024-05-28T06:07:57.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:57.396+0000 7f9c67400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f9c7015baa0 0x7f9c700d0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:07:57.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:57.396+0000 7f9c67400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f9c7015baa0 0x7f9c700d0970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49756/0 (socket says 172.21.15.96:49756) 2024-05-28T06:07:57.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:57.396+0000 7f9c67400640 1 -- 172.21.15.96:0/1733444188 learned_addr learned my addr 172.21.15.96:0/1733444188 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:07:57.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:57.396+0000 7f9c74e00640 1 --2- 172.21.15.96:0/1733444188 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9c7006f390 0x7f9c7006f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:07:57.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:57.396+0000 7f9c67400640 1 -- 172.21.15.96:0/1733444188 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9c7006f390 msgr2=0x7f9c7006f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:07:57.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:57.396+0000 7f9c67400640 1 --2- 172.21.15.96:0/1733444188 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9c7006f390 0x7f9c7006f800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:57.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:57.396+0000 7f9c67400640 1 -- 172.21.15.96:0/1733444188 >> v1:172.21.15.123:6789/0 conn(0x7f9c70155a00 legacy=0x7f9c7006dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:07:57.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:57.396+0000 7f9c67400640 1 -- 172.21.15.96:0/1733444188 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c7015c0b0 con 0x7f9c7015baa0 2024-05-28T06:07:57.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:57.397+0000 7f9c67400640 1 --2- 172.21.15.96:0/1733444188 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9c7015baa0 0x7f9c700d0970 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f9c58009870 tx=0x7f9c5802a200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b953039e0cdbb0f5 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:07:57.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:57.398+0000 7f9c6f400640 1 -- 172.21.15.96:0/1733444188 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c58037070 con 0x7f9c7015baa0 2024-05-28T06:07:57.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:57.398+0000 7f9c6f400640 1 -- 172.21.15.96:0/1733444188 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c58032440 con 0x7f9c7015baa0 2024-05-28T06:07:57.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:57.398+0000 7f9c6f400640 1 -- 172.21.15.96:0/1733444188 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c58032810 con 0x7f9c7015baa0 2024-05-28T06:07:57.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:57.398+0000 7f9c76e00640 1 -- 172.21.15.96:0/1733444188 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9c7015baa0 msgr2=0x7f9c700d0970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:07:57.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:57.398+0000 7f9c76e00640 1 --2- 172.21.15.96:0/1733444188 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9c7015baa0 0x7f9c700d0970 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f9c58009870 tx=0x7f9c5802a200 comp rx=0 tx=0).stop 2024-05-28T06:07:57.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:57.398+0000 7f9c76e00640 1 -- 172.21.15.96:0/1733444188 shutdown_connections 2024-05-28T06:07:57.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:57.398+0000 7f9c76e00640 1 --2- 172.21.15.96:0/1733444188 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9c7006f390 0x7f9c7006f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:57.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:57.398+0000 7f9c76e00640 1 --2- 172.21.15.96:0/1733444188 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9c7015baa0 0x7f9c700d0970 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:57.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:57.398+0000 7f9c76e00640 1 -- 172.21.15.96:0/1733444188 >> 172.21.15.96:0/1733444188 conn(0x7f9c70067ce0 msgr2=0x7f9c7006b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:07:57.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:57.398+0000 7f9c76e00640 1 -- 172.21.15.96:0/1733444188 shutdown_connections 2024-05-28T06:07:57.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:57.399+0000 7f9c76e00640 1 -- 172.21.15.96:0/1733444188 wait complete. 2024-05-28T06:07:57.400 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:57.400+0000 7f9c76e00640 1 Processor -- start 2024-05-28T06:07:57.400 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:57.400+0000 7f9c76e00640 1 -- start start 2024-05-28T06:07:57.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:57.401+0000 7f9c76e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9c7006f390 0x7f9c701f6840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:07:57.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:57.401+0000 7f9c76e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9c70155a00 0x7f9c701e4230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:07:57.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:57.401+0000 7f9c76e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c7015baa0 0x7f9c700cef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:07:57.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:57.401+0000 7f9c76e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f9c7015c670 con 0x7f9c7006f390 2024-05-28T06:07:57.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:57.401+0000 7f9c76e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f9c7015c950 con 0x7f9c7015baa0 2024-05-28T06:07:57.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:57.401+0000 7f9c76e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f9c7015f090 con 0x7f9c70155a00 2024-05-28T06:07:57.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:57.402+0000 7f9c6fe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9c7006f390 0x7f9c701f6840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:07:57.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:57.402+0000 7f9c6fe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9c7006f390 0x7f9c701f6840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49760/0 (socket says 172.21.15.96:49760) 2024-05-28T06:07:57.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:57.402+0000 7f9c6fe00640 1 -- 172.21.15.96:0/3585518389 learned_addr learned my addr 172.21.15.96:0/3585518389 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:07:57.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:57.402+0000 7f9c6fe00640 1 -- 172.21.15.96:0/3585518389 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9c70155a00 msgr2=0x7f9c701e4230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:07:57.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:57.402+0000 7f9c6fe00640 1 --2- 172.21.15.96:0/3585518389 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9c70155a00 0x7f9c701e4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:57.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:57.402+0000 7f9c6fe00640 1 -- 172.21.15.96:0/3585518389 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c7015baa0 msgr2=0x7f9c700cef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:07:57.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:57.402+0000 7f9c6fe00640 1 --2- 172.21.15.96:0/3585518389 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c7015baa0 0x7f9c700cef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:57.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:57.402+0000 7f9c6fe00640 1 -- 172.21.15.96:0/3585518389 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c701590c0 con 0x7f9c7006f390 2024-05-28T06:07:57.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:57.402+0000 7f9c6fe00640 1 --2- 172.21.15.96:0/3585518389 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9c7006f390 0x7f9c701f6840 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f9c6000c430 tx=0x7f9c6000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:07:57.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:57.403+0000 7f9c6cc00640 1 -- 172.21.15.96:0/3585518389 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c60011790 con 0x7f9c7006f390 2024-05-28T06:07:57.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:57.403+0000 7f9c6cc00640 1 -- 172.21.15.96:0/3585518389 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c60017070 con 0x7f9c7006f390 2024-05-28T06:07:57.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:57.403+0000 7f9c6cc00640 1 -- 172.21.15.96:0/3585518389 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c60011bb0 con 0x7f9c7006f390 2024-05-28T06:07:57.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:57.403+0000 7f9c76e00640 1 -- 172.21.15.96:0/3585518389 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c701593a0 con 0x7f9c7006f390 2024-05-28T06:07:57.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:57.403+0000 7f9c76e00640 1 -- 172.21.15.96:0/3585518389 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c70159960 con 0x7f9c7006f390 2024-05-28T06:07:57.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:57.404+0000 7f9c76e00640 1 -- 172.21.15.96:0/3585518389 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c700689e0 con 0x7f9c7006f390 2024-05-28T06:07:57.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:57.405+0000 7f9c6cc00640 1 -- 172.21.15.96:0/3585518389 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 8) ==== 91193+0+0 (secure 0 0 0) 0x7f9c6001c460 con 0x7f9c7006f390 2024-05-28T06:07:57.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:57.406+0000 7f9c6cc00640 1 --2- 172.21.15.96:0/3585518389 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9c4c06caa0 0x7f9c4c06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:07:57.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:57.406+0000 7f9c6cc00640 1 -- 172.21.15.96:0/3585518389 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f9c6008bd90 con 0x7f9c7006f390 2024-05-28T06:07:57.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:57.406+0000 7f9c67400640 1 --2- 172.21.15.96:0/3585518389 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9c4c06caa0 0x7f9c4c06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:07:57.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:57.407+0000 7f9c67400640 1 --2- 172.21.15.96:0/3585518389 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9c4c06caa0 0x7f9c4c06ef60 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f9c58034000 tx=0x7f9c5802c000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:07:57.410 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:57.410+0000 7f9c6cc00640 1 -- 172.21.15.96:0/3585518389 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9c60059f40 con 0x7f9c7006f390 2024-05-28T06:07:57.690 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:57.689+0000 7f9c76e00640 1 -- 172.21.15.96:0/3585518389 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f9c70159b30 con 0x7f9c7006f390 2024-05-28T06:07:57.690 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:57.690+0000 7f9c6cc00640 1 -- 172.21.15.96:0/3585518389 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+2138 (secure 0 0 0) 0x7f9c60059840 con 0x7f9c7006f390 2024-05-28T06:07:57.690 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:07:57.691 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:07:57.691 INFO:teuthology.orchestra.run.smithi096.stdout: "fsid": "e4a831b5-e6d1-47a7-96cb-bf85154ee0d2", 2024-05-28T06:07:57.691 INFO:teuthology.orchestra.run.smithi096.stdout: "health": { 2024-05-28T06:07:57.691 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "HEALTH_OK", 2024-05-28T06:07:57.691 INFO:teuthology.orchestra.run.smithi096.stdout: "checks": {}, 2024-05-28T06:07:57.691 INFO:teuthology.orchestra.run.smithi096.stdout: "mutes": [] 2024-05-28T06:07:57.691 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:07:57.691 INFO:teuthology.orchestra.run.smithi096.stdout: "election_epoch": 8, 2024-05-28T06:07:57.691 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum": [ 2024-05-28T06:07:57.691 INFO:teuthology.orchestra.run.smithi096.stdout: 0, 2024-05-28T06:07:57.691 INFO:teuthology.orchestra.run.smithi096.stdout: 1, 2024-05-28T06:07:57.691 INFO:teuthology.orchestra.run.smithi096.stdout: 2 2024-05-28T06:07:57.691 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:07:57.692 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_names": [ 2024-05-28T06:07:57.692 INFO:teuthology.orchestra.run.smithi096.stdout: "a", 2024-05-28T06:07:57.692 INFO:teuthology.orchestra.run.smithi096.stdout: "b", 2024-05-28T06:07:57.692 INFO:teuthology.orchestra.run.smithi096.stdout: "c" 2024-05-28T06:07:57.692 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:07:57.692 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_age": 251, 2024-05-28T06:07:57.692 INFO:teuthology.orchestra.run.smithi096.stdout: "monmap": { 2024-05-28T06:07:57.692 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 1, 2024-05-28T06:07:57.692 INFO:teuthology.orchestra.run.smithi096.stdout: "min_mon_release_name": "squid", 2024-05-28T06:07:57.692 INFO:teuthology.orchestra.run.smithi096.stdout: "num_mons": 3 2024-05-28T06:07:57.692 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:07:57.692 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap": { 2024-05-28T06:07:57.692 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 33, 2024-05-28T06:07:57.692 INFO:teuthology.orchestra.run.smithi096.stdout: "num_osds": 8, 2024-05-28T06:07:57.693 INFO:teuthology.orchestra.run.smithi096.stdout: "num_up_osds": 8, 2024-05-28T06:07:57.693 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_up_since": 1716876238, 2024-05-28T06:07:57.693 INFO:teuthology.orchestra.run.smithi096.stdout: "num_in_osds": 8, 2024-05-28T06:07:57.693 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_in_since": 1716876229, 2024-05-28T06:07:57.693 INFO:teuthology.orchestra.run.smithi096.stdout: "num_remapped_pgs": 0 2024-05-28T06:07:57.693 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:07:57.693 INFO:teuthology.orchestra.run.smithi096.stdout: "pgmap": { 2024-05-28T06:07:57.693 INFO:teuthology.orchestra.run.smithi096.stdout: "pgs_by_state": [ 2024-05-28T06:07:57.693 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:07:57.693 INFO:teuthology.orchestra.run.smithi096.stdout: "state_name": "active+clean", 2024-05-28T06:07:57.693 INFO:teuthology.orchestra.run.smithi096.stdout: "count": 97 2024-05-28T06:07:57.693 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:07:57.693 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:07:57.693 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pgs": 97, 2024-05-28T06:07:57.694 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pools": 3, 2024-05-28T06:07:57.694 INFO:teuthology.orchestra.run.smithi096.stdout: "num_objects": 2923, 2024-05-28T06:07:57.694 INFO:teuthology.orchestra.run.smithi096.stdout: "data_bytes": 12139655245, 2024-05-28T06:07:57.694 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_used": 23845675008, 2024-05-28T06:07:57.694 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_avail": 749248438272, 2024-05-28T06:07:57.694 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_total": 773094113280, 2024-05-28T06:07:57.694 INFO:teuthology.orchestra.run.smithi096.stdout: "read_bytes_sec": 120892685, 2024-05-28T06:07:57.694 INFO:teuthology.orchestra.run.smithi096.stdout: "write_bytes_sec": 165980672, 2024-05-28T06:07:57.694 INFO:teuthology.orchestra.run.smithi096.stdout: "read_op_per_sec": 29, 2024-05-28T06:07:57.694 INFO:teuthology.orchestra.run.smithi096.stdout: "write_op_per_sec": 44 2024-05-28T06:07:57.694 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:07:57.694 INFO:teuthology.orchestra.run.smithi096.stdout: "fsmap": { 2024-05-28T06:07:57.694 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 16, 2024-05-28T06:07:57.695 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 2, 2024-05-28T06:07:57.695 INFO:teuthology.orchestra.run.smithi096.stdout: "up": 1, 2024-05-28T06:07:57.695 INFO:teuthology.orchestra.run.smithi096.stdout: "in": 1, 2024-05-28T06:07:57.695 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 1, 2024-05-28T06:07:57.695 INFO:teuthology.orchestra.run.smithi096.stdout: "by_rank": [ 2024-05-28T06:07:57.695 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:07:57.695 INFO:teuthology.orchestra.run.smithi096.stdout: "filesystem_id": 2, 2024-05-28T06:07:57.695 INFO:teuthology.orchestra.run.smithi096.stdout: "rank": 0, 2024-05-28T06:07:57.695 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "d", 2024-05-28T06:07:57.695 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "up:active", 2024-05-28T06:07:57.695 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 5238 2024-05-28T06:07:57.695 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:07:57.696 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:07:57.696 INFO:teuthology.orchestra.run.smithi096.stdout: "up:standby": 3 2024-05-28T06:07:57.696 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:07:57.696 INFO:teuthology.orchestra.run.smithi096.stdout: "mgrmap": { 2024-05-28T06:07:57.696 INFO:teuthology.orchestra.run.smithi096.stdout: "available": true, 2024-05-28T06:07:57.696 INFO:teuthology.orchestra.run.smithi096.stdout: "num_standbys": 1, 2024-05-28T06:07:57.696 INFO:teuthology.orchestra.run.smithi096.stdout: "modules": [ 2024-05-28T06:07:57.696 INFO:teuthology.orchestra.run.smithi096.stdout: "iostat", 2024-05-28T06:07:57.696 INFO:teuthology.orchestra.run.smithi096.stdout: "nfs", 2024-05-28T06:07:57.697 INFO:teuthology.orchestra.run.smithi096.stdout: "restful" 2024-05-28T06:07:57.697 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:07:57.697 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:07:57.697 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:07:57.697 INFO:teuthology.orchestra.run.smithi096.stdout: "servicemap": { 2024-05-28T06:07:57.697 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 7, 2024-05-28T06:07:57.697 INFO:teuthology.orchestra.run.smithi096.stdout: "modified": "2024-05-28T06:06:33.535436+0000", 2024-05-28T06:07:57.697 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:07:57.697 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:07:57.697 INFO:teuthology.orchestra.run.smithi096.stdout: "progress_events": { 2024-05-28T06:07:57.697 INFO:teuthology.orchestra.run.smithi096.stdout: "18e91b7c-9da2-4cf3-88f8-e7e2859c593b": { 2024-05-28T06:07:57.698 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "1 ongoing clones; Avg progress is 10.0% (3s)\n [==..........................] (remaining: 28s)", 2024-05-28T06:07:57.698 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": 0.10000000149011612, 2024-05-28T06:07:57.698 INFO:teuthology.orchestra.run.smithi096.stdout: "add_to_ceph_s": true 2024-05-28T06:07:57.698 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:07:57.698 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:07:57.698 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:07:57.699 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:57.692+0000 7f9c76e00640 1 -- 172.21.15.96:0/3585518389 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9c4c06caa0 msgr2=0x7f9c4c06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:07:57.699 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:57.692+0000 7f9c76e00640 1 --2- 172.21.15.96:0/3585518389 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9c4c06caa0 0x7f9c4c06ef60 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f9c58034000 tx=0x7f9c5802c000 comp rx=0 tx=0).stop 2024-05-28T06:07:57.699 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:57.692+0000 7f9c76e00640 1 -- 172.21.15.96:0/3585518389 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9c7006f390 msgr2=0x7f9c701f6840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:07:57.699 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:57.692+0000 7f9c76e00640 1 --2- 172.21.15.96:0/3585518389 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9c7006f390 0x7f9c701f6840 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f9c6000c430 tx=0x7f9c6000c8f0 comp rx=0 tx=0).stop 2024-05-28T06:07:57.699 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:57.693+0000 7f9c76e00640 1 -- 172.21.15.96:0/3585518389 shutdown_connections 2024-05-28T06:07:57.699 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:57.693+0000 7f9c76e00640 1 --2- 172.21.15.96:0/3585518389 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9c4c06caa0 0x7f9c4c06ef60 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:57.700 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:57.693+0000 7f9c76e00640 1 --2- 172.21.15.96:0/3585518389 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c7015baa0 0x7f9c700cef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:57.700 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:57.693+0000 7f9c76e00640 1 --2- 172.21.15.96:0/3585518389 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9c70155a00 0x7f9c701e4230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:57.700 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:57.693+0000 7f9c76e00640 1 --2- 172.21.15.96:0/3585518389 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9c7006f390 0x7f9c701f6840 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:07:57.700 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:57.693+0000 7f9c76e00640 1 -- 172.21.15.96:0/3585518389 >> 172.21.15.96:0/3585518389 conn(0x7f9c70067ce0 msgr2=0x7f9c7006b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:07:57.700 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:57.693+0000 7f9c76e00640 1 -- 172.21.15.96:0/3585518389 shutdown_connections 2024-05-28T06:07:57.700 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:07:57.693+0000 7f9c76e00640 1 -- 172.21.15.96:0/3585518389 wait complete. 2024-05-28T06:07:59.664 DEBUG:teuthology.orchestra.run.smithi096:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:07:59.672 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:08:02.712 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-05-28T06:08:02.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:02.924+0000 7fa6a3200640 1 Processor -- start 2024-05-28T06:08:02.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:02.925+0000 7fa6a3200640 1 -- start start 2024-05-28T06:08:02.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:02.926+0000 7fa6a3200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fa6a415e690 0x7fa6a40d4530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:02.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:02.926+0000 7fa6a3200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa6a40d1ca0 0x7fa6a40d4a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:02.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:02.926+0000 7fa6a3200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6a40d3060 con 0x7fa6a415ea70 2024-05-28T06:08:02.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:02.926+0000 7fa6a3200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fa6a40d3340 con 0x7fa6a415e690 2024-05-28T06:08:02.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:02.926+0000 7fa6a3200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fa6a40d3620 con 0x7fa6a40d1ca0 2024-05-28T06:08:02.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:02.926+0000 7fa6a1e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fa6a415e690 0x7fa6a40d4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:02.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:02.926+0000 7fa6a2800640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fa6a415ea70 0x7fa6a40d2330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:60486/0 (socket says 172.21.15.96:60486) 2024-05-28T06:08:02.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:02.926+0000 7fa6a1e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fa6a415e690 0x7fa6a40d4530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49158/0 (socket says 172.21.15.96:49158) 2024-05-28T06:08:02.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:02.926+0000 7fa6a2800640 1 -- 172.21.15.96:0/3577785665 learned_addr learned my addr 172.21.15.96:0/3577785665 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:08:02.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:02.927+0000 7fa6a1400640 1 --2- 172.21.15.96:0/3577785665 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa6a40d1ca0 0x7fa6a40d4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:02.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:02.927+0000 7fa6a1e00640 1 -- 172.21.15.96:0/3577785665 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa6a40d1ca0 msgr2=0x7fa6a40d4a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:02.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:02.927+0000 7fa6a1e00640 1 --2- 172.21.15.96:0/3577785665 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa6a40d1ca0 0x7fa6a40d4a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:02.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:02.927+0000 7fa6a1e00640 1 -- 172.21.15.96:0/3577785665 >> v1:172.21.15.96:6789/0 conn(0x7fa6a415ea70 legacy=0x7fa6a40d2330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:02.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:02.927+0000 7fa6a1e00640 1 -- 172.21.15.96:0/3577785665 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6a40cf770 con 0x7fa6a415e690 2024-05-28T06:08:02.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:02.927+0000 7fa6a1400640 1 --2- 172.21.15.96:0/3577785665 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa6a40d1ca0 0x7fa6a40d4a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:08:02.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:02.927+0000 7fa6a0a00640 1 -- 172.21.15.96:0/3577785665 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3626067375 0 0) 0x7fa6a40d3060 con 0x7fa6a415ea70 2024-05-28T06:08:02.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:02.928+0000 7fa6a1e00640 1 --2- 172.21.15.96:0/3577785665 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6a415e690 0x7fa6a40d4530 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fa6940097a0 tx=0x7fa69402a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=863fa0d05db1415f server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:02.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:02.928+0000 7fa6a0a00640 1 -- 172.21.15.96:0/3577785665 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa694037070 con 0x7fa6a415e690 2024-05-28T06:08:02.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:02.928+0000 7fa6a0a00640 1 -- 172.21.15.96:0/3577785665 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa694032440 con 0x7fa6a415e690 2024-05-28T06:08:02.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:02.928+0000 7fa6a0a00640 1 -- 172.21.15.96:0/3577785665 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa694032810 con 0x7fa6a415e690 2024-05-28T06:08:02.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:02.929+0000 7fa6a3200640 1 -- 172.21.15.96:0/3577785665 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6a415e690 msgr2=0x7fa6a40d4530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:02.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:02.929+0000 7fa6a3200640 1 --2- 172.21.15.96:0/3577785665 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6a415e690 0x7fa6a40d4530 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fa6940097a0 tx=0x7fa69402a180 comp rx=0 tx=0).stop 2024-05-28T06:08:02.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:02.929+0000 7fa6a3200640 1 -- 172.21.15.96:0/3577785665 shutdown_connections 2024-05-28T06:08:02.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:02.929+0000 7fa6a3200640 1 --2- 172.21.15.96:0/3577785665 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa6a40d1ca0 0x7fa6a40d4a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:02.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:02.929+0000 7fa6a3200640 1 --2- 172.21.15.96:0/3577785665 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6a415e690 0x7fa6a40d4530 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:02.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:02.929+0000 7fa6a3200640 1 -- 172.21.15.96:0/3577785665 >> 172.21.15.96:0/3577785665 conn(0x7fa6a409d910 msgr2=0x7fa6a408bfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:02.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:02.929+0000 7fa6a3200640 1 -- 172.21.15.96:0/3577785665 shutdown_connections 2024-05-28T06:08:02.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:02.929+0000 7fa6a3200640 1 -- 172.21.15.96:0/3577785665 wait complete. 2024-05-28T06:08:02.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:02.931+0000 7fa6a3200640 1 Processor -- start 2024-05-28T06:08:02.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:02.931+0000 7fa6a3200640 1 -- start start 2024-05-28T06:08:02.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:02.932+0000 7fa6a3200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6a40d1ca0 0x7fa6a40b92b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:02.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:02.932+0000 7fa6a3200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6a415e690 0x7fa6a40ca4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:02.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:02.932+0000 7fa6a3200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa6a415ea70 0x7fa6a4156f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:02.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:02.932+0000 7fa6a3200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa6a40d2bc0 con 0x7fa6a415e690 2024-05-28T06:08:02.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:02.932+0000 7fa6a3200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa6a40d2a40 con 0x7fa6a40d1ca0 2024-05-28T06:08:02.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:02.932+0000 7fa6a3200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fa6a40d2d40 con 0x7fa6a415ea70 2024-05-28T06:08:02.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:02.932+0000 7fa6a1400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6a415e690 0x7fa6a40ca4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:02.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:02.932+0000 7fa6a1400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6a415e690 0x7fa6a40ca4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49164/0 (socket says 172.21.15.96:49164) 2024-05-28T06:08:02.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:02.932+0000 7fa6a1400640 1 -- 172.21.15.96:0/1317404513 learned_addr learned my addr 172.21.15.96:0/1317404513 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:08:02.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:02.933+0000 7fa6a2800640 1 --2- 172.21.15.96:0/1317404513 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa6a415ea70 0x7fa6a4156f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:02.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:02.933+0000 7fa6a1e00640 1 --2- 172.21.15.96:0/1317404513 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6a40d1ca0 0x7fa6a40b92b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:02.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:02.933+0000 7fa6a1400640 1 -- 172.21.15.96:0/1317404513 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa6a415ea70 msgr2=0x7fa6a4156f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:02.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:02.933+0000 7fa6a1400640 1 --2- 172.21.15.96:0/1317404513 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa6a415ea70 0x7fa6a4156f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:02.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:02.933+0000 7fa6a1400640 1 -- 172.21.15.96:0/1317404513 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6a40d1ca0 msgr2=0x7fa6a40b92b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:02.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:02.933+0000 7fa6a1400640 1 --2- 172.21.15.96:0/1317404513 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6a40d1ca0 0x7fa6a40b92b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:02.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:02.933+0000 7fa6a1400640 1 -- 172.21.15.96:0/1317404513 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6a40b8210 con 0x7fa6a415e690 2024-05-28T06:08:02.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:02.933+0000 7fa6a1400640 1 --2- 172.21.15.96:0/1317404513 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6a415e690 0x7fa6a40ca4b0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fa68c00cef0 tx=0x7fa68c00a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:02.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:02.933+0000 7fa69a000640 1 -- 172.21.15.96:0/1317404513 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa68c00e860 con 0x7fa6a415e690 2024-05-28T06:08:02.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:02.933+0000 7fa69a000640 1 -- 172.21.15.96:0/1317404513 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa68c00ea00 con 0x7fa6a415e690 2024-05-28T06:08:02.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:02.933+0000 7fa69a000640 1 -- 172.21.15.96:0/1317404513 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa68c00edb0 con 0x7fa6a415e690 2024-05-28T06:08:02.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:02.933+0000 7fa6a3200640 1 -- 172.21.15.96:0/1317404513 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6a40b84f0 con 0x7fa6a415e690 2024-05-28T06:08:02.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:02.934+0000 7fa6a3200640 1 -- 172.21.15.96:0/1317404513 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6a40b8ab0 con 0x7fa6a415e690 2024-05-28T06:08:02.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:02.935+0000 7fa6a3200640 1 -- 172.21.15.96:0/1317404513 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6a40689e0 con 0x7fa6a415e690 2024-05-28T06:08:02.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:02.936+0000 7fa69a000640 1 -- 172.21.15.96:0/1317404513 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 8) ==== 91193+0+0 (secure 0 0 0) 0x7fa68c016420 con 0x7fa6a415e690 2024-05-28T06:08:02.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:02.936+0000 7fa69a000640 1 --2- 172.21.15.96:0/1317404513 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa68006caa0 0x7fa68006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:02.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:02.937+0000 7fa69a000640 1 -- 172.21.15.96:0/1317404513 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7fa68c08cd40 con 0x7fa6a415e690 2024-05-28T06:08:02.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:02.937+0000 7fa6a1e00640 1 --2- 172.21.15.96:0/1317404513 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa68006caa0 0x7fa68006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:02.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:02.938+0000 7fa6a1e00640 1 --2- 172.21.15.96:0/1317404513 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa68006caa0 0x7fa68006ef60 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fa694034000 tx=0x7fa69402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:02.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:02.941+0000 7fa69a000640 1 -- 172.21.15.96:0/1317404513 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa68c05ad40 con 0x7fa6a415e690 2024-05-28T06:08:03.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:03.221+0000 7fa6a3200640 1 -- 172.21.15.96:0/1317404513 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fa6a406cbc0 con 0x7fa6a415e690 2024-05-28T06:08:03.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:03.222+0000 7fa69a000640 1 -- 172.21.15.96:0/1317404513 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+2122 (secure 0 0 0) 0x7fa68c05a640 con 0x7fa6a415e690 2024-05-28T06:08:03.224 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:08:03.224 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:08:03.224 INFO:teuthology.orchestra.run.smithi096.stdout: "fsid": "e4a831b5-e6d1-47a7-96cb-bf85154ee0d2", 2024-05-28T06:08:03.224 INFO:teuthology.orchestra.run.smithi096.stdout: "health": { 2024-05-28T06:08:03.224 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "HEALTH_OK", 2024-05-28T06:08:03.225 INFO:teuthology.orchestra.run.smithi096.stdout: "checks": {}, 2024-05-28T06:08:03.225 INFO:teuthology.orchestra.run.smithi096.stdout: "mutes": [] 2024-05-28T06:08:03.225 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:08:03.225 INFO:teuthology.orchestra.run.smithi096.stdout: "election_epoch": 8, 2024-05-28T06:08:03.225 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum": [ 2024-05-28T06:08:03.225 INFO:teuthology.orchestra.run.smithi096.stdout: 0, 2024-05-28T06:08:03.225 INFO:teuthology.orchestra.run.smithi096.stdout: 1, 2024-05-28T06:08:03.225 INFO:teuthology.orchestra.run.smithi096.stdout: 2 2024-05-28T06:08:03.225 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:08:03.225 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_names": [ 2024-05-28T06:08:03.225 INFO:teuthology.orchestra.run.smithi096.stdout: "a", 2024-05-28T06:08:03.225 INFO:teuthology.orchestra.run.smithi096.stdout: "b", 2024-05-28T06:08:03.225 INFO:teuthology.orchestra.run.smithi096.stdout: "c" 2024-05-28T06:08:03.226 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:08:03.226 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_age": 256, 2024-05-28T06:08:03.226 INFO:teuthology.orchestra.run.smithi096.stdout: "monmap": { 2024-05-28T06:08:03.226 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 1, 2024-05-28T06:08:03.226 INFO:teuthology.orchestra.run.smithi096.stdout: "min_mon_release_name": "squid", 2024-05-28T06:08:03.226 INFO:teuthology.orchestra.run.smithi096.stdout: "num_mons": 3 2024-05-28T06:08:03.226 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:08:03.226 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap": { 2024-05-28T06:08:03.226 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 33, 2024-05-28T06:08:03.226 INFO:teuthology.orchestra.run.smithi096.stdout: "num_osds": 8, 2024-05-28T06:08:03.226 INFO:teuthology.orchestra.run.smithi096.stdout: "num_up_osds": 8, 2024-05-28T06:08:03.226 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_up_since": 1716876238, 2024-05-28T06:08:03.226 INFO:teuthology.orchestra.run.smithi096.stdout: "num_in_osds": 8, 2024-05-28T06:08:03.227 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_in_since": 1716876229, 2024-05-28T06:08:03.227 INFO:teuthology.orchestra.run.smithi096.stdout: "num_remapped_pgs": 0 2024-05-28T06:08:03.227 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:08:03.227 INFO:teuthology.orchestra.run.smithi096.stdout: "pgmap": { 2024-05-28T06:08:03.227 INFO:teuthology.orchestra.run.smithi096.stdout: "pgs_by_state": [ 2024-05-28T06:08:03.227 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:08:03.227 INFO:teuthology.orchestra.run.smithi096.stdout: "state_name": "active+clean", 2024-05-28T06:08:03.227 INFO:teuthology.orchestra.run.smithi096.stdout: "count": 97 2024-05-28T06:08:03.227 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:08:03.227 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:08:03.227 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pgs": 97, 2024-05-28T06:08:03.228 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pools": 3, 2024-05-28T06:08:03.228 INFO:teuthology.orchestra.run.smithi096.stdout: "num_objects": 3208, 2024-05-28T06:08:03.228 INFO:teuthology.orchestra.run.smithi096.stdout: "data_bytes": 13330917896, 2024-05-28T06:08:03.228 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_used": 26924572672, 2024-05-28T06:08:03.228 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_avail": 746169540608, 2024-05-28T06:08:03.228 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_total": 773094113280, 2024-05-28T06:08:03.228 INFO:teuthology.orchestra.run.smithi096.stdout: "read_bytes_sec": 218035629, 2024-05-28T06:08:03.228 INFO:teuthology.orchestra.run.smithi096.stdout: "write_bytes_sec": 225045591, 2024-05-28T06:08:03.228 INFO:teuthology.orchestra.run.smithi096.stdout: "read_op_per_sec": 52, 2024-05-28T06:08:03.228 INFO:teuthology.orchestra.run.smithi096.stdout: "write_op_per_sec": 61 2024-05-28T06:08:03.228 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:08:03.228 INFO:teuthology.orchestra.run.smithi096.stdout: "fsmap": { 2024-05-28T06:08:03.228 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 16, 2024-05-28T06:08:03.228 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 2, 2024-05-28T06:08:03.229 INFO:teuthology.orchestra.run.smithi096.stdout: "up": 1, 2024-05-28T06:08:03.229 INFO:teuthology.orchestra.run.smithi096.stdout: "in": 1, 2024-05-28T06:08:03.229 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 1, 2024-05-28T06:08:03.229 INFO:teuthology.orchestra.run.smithi096.stdout: "by_rank": [ 2024-05-28T06:08:03.229 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:08:03.229 INFO:teuthology.orchestra.run.smithi096.stdout: "filesystem_id": 2, 2024-05-28T06:08:03.229 INFO:teuthology.orchestra.run.smithi096.stdout: "rank": 0, 2024-05-28T06:08:03.229 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "d", 2024-05-28T06:08:03.229 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "up:active", 2024-05-28T06:08:03.229 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 5238 2024-05-28T06:08:03.229 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:08:03.229 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:08:03.229 INFO:teuthology.orchestra.run.smithi096.stdout: "up:standby": 3 2024-05-28T06:08:03.230 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:08:03.230 INFO:teuthology.orchestra.run.smithi096.stdout: "mgrmap": { 2024-05-28T06:08:03.230 INFO:teuthology.orchestra.run.smithi096.stdout: "available": true, 2024-05-28T06:08:03.230 INFO:teuthology.orchestra.run.smithi096.stdout: "num_standbys": 1, 2024-05-28T06:08:03.230 INFO:teuthology.orchestra.run.smithi096.stdout: "modules": [ 2024-05-28T06:08:03.230 INFO:teuthology.orchestra.run.smithi096.stdout: "iostat", 2024-05-28T06:08:03.230 INFO:teuthology.orchestra.run.smithi096.stdout: "nfs", 2024-05-28T06:08:03.230 INFO:teuthology.orchestra.run.smithi096.stdout: "restful" 2024-05-28T06:08:03.230 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:08:03.230 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:08:03.230 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:08:03.230 INFO:teuthology.orchestra.run.smithi096.stdout: "servicemap": { 2024-05-28T06:08:03.230 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 7, 2024-05-28T06:08:03.231 INFO:teuthology.orchestra.run.smithi096.stdout: "modified": "2024-05-28T06:06:33.535436+0000", 2024-05-28T06:08:03.231 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:08:03.231 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:08:03.231 INFO:teuthology.orchestra.run.smithi096.stdout: "progress_events": { 2024-05-28T06:08:03.231 INFO:teuthology.orchestra.run.smithi096.stdout: "18e91b7c-9da2-4cf3-88f8-e7e2859c593b": { 2024-05-28T06:08:03.231 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "All cloning finished, waiting 26.0 seconds before exiting (9s)\n [============================] ", 2024-05-28T06:08:03.231 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": 1, 2024-05-28T06:08:03.231 INFO:teuthology.orchestra.run.smithi096.stdout: "add_to_ceph_s": true 2024-05-28T06:08:03.231 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:08:03.231 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:08:03.231 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:08:03.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:03.225+0000 7fa6a3200640 1 -- 172.21.15.96:0/1317404513 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa68006caa0 msgr2=0x7fa68006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:03.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:03.225+0000 7fa6a3200640 1 --2- 172.21.15.96:0/1317404513 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa68006caa0 0x7fa68006ef60 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fa694034000 tx=0x7fa69402e000 comp rx=0 tx=0).stop 2024-05-28T06:08:03.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:03.225+0000 7fa6a3200640 1 -- 172.21.15.96:0/1317404513 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6a415e690 msgr2=0x7fa6a40ca4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:03.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:03.225+0000 7fa6a3200640 1 --2- 172.21.15.96:0/1317404513 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6a415e690 0x7fa6a40ca4b0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fa68c00cef0 tx=0x7fa68c00a6c0 comp rx=0 tx=0).stop 2024-05-28T06:08:03.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:03.225+0000 7fa6a3200640 1 -- 172.21.15.96:0/1317404513 shutdown_connections 2024-05-28T06:08:03.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:03.225+0000 7fa6a3200640 1 --2- 172.21.15.96:0/1317404513 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa68006caa0 0x7fa68006ef60 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:03.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:03.225+0000 7fa6a3200640 1 --2- 172.21.15.96:0/1317404513 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa6a415ea70 0x7fa6a4156f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:03.233 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:03.225+0000 7fa6a3200640 1 --2- 172.21.15.96:0/1317404513 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6a415e690 0x7fa6a40ca4b0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:03.233 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:03.225+0000 7fa6a3200640 1 --2- 172.21.15.96:0/1317404513 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6a40d1ca0 0x7fa6a40b92b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:03.233 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:03.225+0000 7fa6a3200640 1 -- 172.21.15.96:0/1317404513 >> 172.21.15.96:0/1317404513 conn(0x7fa6a409d910 msgr2=0x7fa6a40d1380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:03.233 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:03.226+0000 7fa6a3200640 1 -- 172.21.15.96:0/1317404513 shutdown_connections 2024-05-28T06:08:03.233 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:03.226+0000 7fa6a3200640 1 -- 172.21.15.96:0/1317404513 wait complete. 2024-05-28T06:08:08.247 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-05-28T06:08:08.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.455+0000 7fd3a8e00640 1 Processor -- start 2024-05-28T06:08:08.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.455+0000 7fd3a8e00640 1 -- start start 2024-05-28T06:08:08.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.456+0000 7fd3a8e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd3a40d4c40 0x7fd3a40d2330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:08.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.456+0000 7fd3a8e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fd3a40d1ca0 0x7fd3a40d2870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:08.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.456+0000 7fd3a8e00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3a40d2db0 con 0x7fd3a415e690 2024-05-28T06:08:08.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.456+0000 7fd3a8e00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fd3a40d3060 con 0x7fd3a40d1ca0 2024-05-28T06:08:08.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.456+0000 7fd3a8e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fd3a40d3340 con 0x7fd3a40d4c40 2024-05-28T06:08:08.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.457+0000 7fd3a3400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fd3a415e690 0x7fd3a40d4530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:60494/0 (socket says 172.21.15.96:60494) 2024-05-28T06:08:08.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.457+0000 7fd3a3400640 1 -- 172.21.15.96:0/576329684 learned_addr learned my addr 172.21.15.96:0/576329684 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:08:08.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.457+0000 7fd3a2a00640 1 -- 172.21.15.96:0/576329684 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3766998700 0 0) 0x7fd3a40d2db0 con 0x7fd3a415e690 2024-05-28T06:08:08.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.457+0000 7fd39be00640 1 --2- 172.21.15.96:0/576329684 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd3a40d4c40 0x7fd3a40d2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:08.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.457+0000 7fd3a2a00640 1 -- 172.21.15.96:0/576329684 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd388003520 con 0x7fd3a415e690 2024-05-28T06:08:08.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.457+0000 7fd3a3e00640 1 --2- 172.21.15.96:0/576329684 >> v2:172.21.15.123:3300/0 conn(0x7fd3a40d1ca0 0x7fd3a40d2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:08.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.457+0000 7fd3a2a00640 1 -- 172.21.15.96:0/576329684 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3388964622 0 0) 0x7fd388003520 con 0x7fd3a415e690 2024-05-28T06:08:08.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.457+0000 7fd3a2a00640 1 -- 172.21.15.96:0/576329684 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd3a40d4c40 msgr2=0x7fd3a40d2330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:08.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.457+0000 7fd3a2a00640 1 --2- 172.21.15.96:0/576329684 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd3a40d4c40 0x7fd3a40d2330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:08.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.457+0000 7fd3a2a00640 1 -- 172.21.15.96:0/576329684 >> v2:172.21.15.123:3300/0 conn(0x7fd3a40d1ca0 msgr2=0x7fd3a40d2870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:08.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.457+0000 7fd3a2a00640 1 --2- 172.21.15.96:0/576329684 >> v2:172.21.15.123:3300/0 conn(0x7fd3a40d1ca0 0x7fd3a40d2870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:08.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.457+0000 7fd3a2a00640 1 -- 172.21.15.96:0/576329684 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3a40d3620 con 0x7fd3a415e690 2024-05-28T06:08:08.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.457+0000 7fd3a2a00640 1 -- 172.21.15.96:0/576329684 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7fd394001670 con 0x7fd3a415e690 2024-05-28T06:08:08.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.458+0000 7fd3a2a00640 1 -- 172.21.15.96:0/576329684 >> v1:172.21.15.96:6789/0 conn(0x7fd3a415e690 legacy=0x7fd3a40d4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:08.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.458+0000 7fd3a2a00640 1 --2- 172.21.15.96:0/576329684 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd3880039b0 0x7fd388003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:08.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.458+0000 7fd3a2a00640 1 --2- 172.21.15.96:0/576329684 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd3a40d4c40 0x7fd388004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:08.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.458+0000 7fd3a2a00640 1 --2- 172.21.15.96:0/576329684 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd388004d40 0x7fd3880051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:08.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.458+0000 7fd3a2a00640 1 -- 172.21.15.96:0/576329684 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd3a40d3060 con 0x7fd3880039b0 2024-05-28T06:08:08.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.458+0000 7fd3a2a00640 1 -- 172.21.15.96:0/576329684 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd3a40d3340 con 0x7fd388004d40 2024-05-28T06:08:08.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.458+0000 7fd3a2a00640 1 -- 172.21.15.96:0/576329684 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fd38800baa0 con 0x7fd3a40d4c40 2024-05-28T06:08:08.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.458+0000 7fd3a3400640 1 --2- 172.21.15.96:0/576329684 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd3880039b0 0x7fd388003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:08.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.458+0000 7fd39be00640 1 --2- 172.21.15.96:0/576329684 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd3a40d4c40 0x7fd388004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:08.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.458+0000 7fd3a3e00640 1 --2- 172.21.15.96:0/576329684 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd388004d40 0x7fd3880051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:08.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.458+0000 7fd3a3400640 1 -- 172.21.15.96:0/576329684 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd3a40d4c40 msgr2=0x7fd388004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:08.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.458+0000 7fd3a3400640 1 --2- 172.21.15.96:0/576329684 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd3a40d4c40 0x7fd388004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:08.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.458+0000 7fd3a3400640 1 -- 172.21.15.96:0/576329684 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd388004d40 msgr2=0x7fd3880051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:08.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.458+0000 7fd3a3400640 1 --2- 172.21.15.96:0/576329684 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd388004d40 0x7fd3880051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:08.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.458+0000 7fd3a3400640 1 -- 172.21.15.96:0/576329684 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd388003520 con 0x7fd3880039b0 2024-05-28T06:08:08.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.459+0000 7fd3a3400640 1 --2- 172.21.15.96:0/576329684 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd3880039b0 0x7fd388003dd0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fd39400c290 tx=0x7fd388005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73ac7d67b40a36f6 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:08.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.460+0000 7fd3a2a00640 1 -- 172.21.15.96:0/576329684 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd394007d50 con 0x7fd3880039b0 2024-05-28T06:08:08.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.460+0000 7fd3a2a00640 1 -- 172.21.15.96:0/576329684 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd39402e070 con 0x7fd3880039b0 2024-05-28T06:08:08.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.460+0000 7fd3a2a00640 1 -- 172.21.15.96:0/576329684 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd394005d30 con 0x7fd3880039b0 2024-05-28T06:08:08.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.460+0000 7fd3a8e00640 1 -- 172.21.15.96:0/576329684 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd3880039b0 msgr2=0x7fd388003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:08.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.460+0000 7fd3a8e00640 1 --2- 172.21.15.96:0/576329684 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd3880039b0 0x7fd388003dd0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fd39400c290 tx=0x7fd388005830 comp rx=0 tx=0).stop 2024-05-28T06:08:08.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.460+0000 7fd3a3e00640 1 -- 172.21.15.96:0/576329684 reap_dead start 2024-05-28T06:08:08.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.460+0000 7fd3a8e00640 1 -- 172.21.15.96:0/576329684 shutdown_connections 2024-05-28T06:08:08.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.460+0000 7fd3a8e00640 1 -- 172.21.15.96:0/576329684 >> 172.21.15.96:0/576329684 conn(0x7fd3a409d910 msgr2=0x7fd3a408bfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:08.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.460+0000 7fd3a8e00640 1 -- 172.21.15.96:0/576329684 shutdown_connections 2024-05-28T06:08:08.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.460+0000 7fd3a8e00640 1 -- 172.21.15.96:0/576329684 wait complete. 2024-05-28T06:08:08.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.462+0000 7fd3a8e00640 1 Processor -- start 2024-05-28T06:08:08.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.462+0000 7fd3a8e00640 1 -- start start 2024-05-28T06:08:08.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.463+0000 7fd3a8e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd3880039b0 0x7fd3a40c9300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:08.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.463+0000 7fd3a8e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd3a40c9f40 0x7fd3a40ca3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:08.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.463+0000 7fd3a8e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd3a40cc6c0 0x7fd3a40ccb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:08.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.463+0000 7fd3a8e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd3a40cf510 con 0x7fd3880039b0 2024-05-28T06:08:08.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.463+0000 7fd3a8e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd3a40cf390 con 0x7fd3a40cc6c0 2024-05-28T06:08:08.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.463+0000 7fd3a8e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fd3a415ea70 con 0x7fd3a40c9f40 2024-05-28T06:08:08.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.463+0000 7fd3a3400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd3880039b0 0x7fd3a40c9300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:08.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.463+0000 7fd3a3400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd3880039b0 0x7fd3a40c9300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49174/0 (socket says 172.21.15.96:49174) 2024-05-28T06:08:08.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.463+0000 7fd3a3400640 1 -- 172.21.15.96:0/3846905741 learned_addr learned my addr 172.21.15.96:0/3846905741 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:08:08.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.464+0000 7fd39be00640 1 --2- 172.21.15.96:0/3846905741 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd3a40c9f40 0x7fd3a40ca3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:08.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.464+0000 7fd3a3e00640 1 --2- 172.21.15.96:0/3846905741 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd3a40cc6c0 0x7fd3a40ccb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:08.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.464+0000 7fd3a3400640 1 -- 172.21.15.96:0/3846905741 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd3a40c9f40 msgr2=0x7fd3a40ca3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:08.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.464+0000 7fd3a3400640 1 --2- 172.21.15.96:0/3846905741 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd3a40c9f40 0x7fd3a40ca3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:08.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.464+0000 7fd3a3400640 1 -- 172.21.15.96:0/3846905741 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd3a40cc6c0 msgr2=0x7fd3a40ccb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:08.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.464+0000 7fd3a3400640 1 --2- 172.21.15.96:0/3846905741 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd3a40cc6c0 0x7fd3a40ccb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:08.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.464+0000 7fd3a3400640 1 -- 172.21.15.96:0/3846905741 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3a40cd080 con 0x7fd3880039b0 2024-05-28T06:08:08.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.464+0000 7fd39be00640 1 --2- 172.21.15.96:0/3846905741 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd3a40c9f40 0x7fd3a40ca3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:08:08.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.464+0000 7fd3a3400640 1 --2- 172.21.15.96:0/3846905741 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd3880039b0 0x7fd3a40c9300 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fd394001970 tx=0x7fd3a40d37f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:08.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.465+0000 7fd39b400640 1 -- 172.21.15.96:0/3846905741 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3940049f0 con 0x7fd3880039b0 2024-05-28T06:08:08.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.465+0000 7fd39b400640 1 -- 172.21.15.96:0/3846905741 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd394004b90 con 0x7fd3880039b0 2024-05-28T06:08:08.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.465+0000 7fd39b400640 1 -- 172.21.15.96:0/3846905741 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd394007070 con 0x7fd3880039b0 2024-05-28T06:08:08.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.465+0000 7fd3a8e00640 1 -- 172.21.15.96:0/3846905741 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3a40cd340 con 0x7fd3880039b0 2024-05-28T06:08:08.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.465+0000 7fd3a8e00640 1 -- 172.21.15.96:0/3846905741 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3a40cd900 con 0x7fd3880039b0 2024-05-28T06:08:08.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.466+0000 7fd3a8e00640 1 -- 172.21.15.96:0/3846905741 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3a40689e0 con 0x7fd3880039b0 2024-05-28T06:08:08.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.467+0000 7fd39b400640 1 -- 172.21.15.96:0/3846905741 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 8) ==== 91193+0+0 (secure 0 0 0) 0x7fd394046020 con 0x7fd3880039b0 2024-05-28T06:08:08.469 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.468+0000 7fd39b400640 1 --2- 172.21.15.96:0/3846905741 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd38006cad0 0x7fd38006ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:08.469 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.468+0000 7fd39b400640 1 -- 172.21.15.96:0/3846905741 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7fd3940ac260 con 0x7fd3880039b0 2024-05-28T06:08:08.470 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.469+0000 7fd39be00640 1 --2- 172.21.15.96:0/3846905741 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd38006cad0 0x7fd38006ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:08.471 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.470+0000 7fd39be00640 1 --2- 172.21.15.96:0/3846905741 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd38006cad0 0x7fd38006ef90 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fd3a41f7840 tx=0x7fd38c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:08.474 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.473+0000 7fd39b400640 1 -- 172.21.15.96:0/3846905741 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd39407a2e0 con 0x7fd3880039b0 2024-05-28T06:08:08.745 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.743+0000 7fd3a8e00640 1 -- 172.21.15.96:0/3846905741 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fd3a406cbc0 con 0x7fd3880039b0 2024-05-28T06:08:08.745 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.744+0000 7fd39b400640 1 -- 172.21.15.96:0/3846905741 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+2121 (secure 0 0 0) 0x7fd394079be0 con 0x7fd3880039b0 2024-05-28T06:08:08.746 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:08:08.746 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:08:08.746 INFO:teuthology.orchestra.run.smithi096.stdout: "fsid": "e4a831b5-e6d1-47a7-96cb-bf85154ee0d2", 2024-05-28T06:08:08.746 INFO:teuthology.orchestra.run.smithi096.stdout: "health": { 2024-05-28T06:08:08.746 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "HEALTH_OK", 2024-05-28T06:08:08.746 INFO:teuthology.orchestra.run.smithi096.stdout: "checks": {}, 2024-05-28T06:08:08.746 INFO:teuthology.orchestra.run.smithi096.stdout: "mutes": [] 2024-05-28T06:08:08.746 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:08:08.747 INFO:teuthology.orchestra.run.smithi096.stdout: "election_epoch": 8, 2024-05-28T06:08:08.747 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum": [ 2024-05-28T06:08:08.747 INFO:teuthology.orchestra.run.smithi096.stdout: 0, 2024-05-28T06:08:08.747 INFO:teuthology.orchestra.run.smithi096.stdout: 1, 2024-05-28T06:08:08.747 INFO:teuthology.orchestra.run.smithi096.stdout: 2 2024-05-28T06:08:08.747 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:08:08.747 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_names": [ 2024-05-28T06:08:08.747 INFO:teuthology.orchestra.run.smithi096.stdout: "a", 2024-05-28T06:08:08.747 INFO:teuthology.orchestra.run.smithi096.stdout: "b", 2024-05-28T06:08:08.747 INFO:teuthology.orchestra.run.smithi096.stdout: "c" 2024-05-28T06:08:08.747 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:08:08.747 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_age": 262, 2024-05-28T06:08:08.747 INFO:teuthology.orchestra.run.smithi096.stdout: "monmap": { 2024-05-28T06:08:08.747 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 1, 2024-05-28T06:08:08.747 INFO:teuthology.orchestra.run.smithi096.stdout: "min_mon_release_name": "squid", 2024-05-28T06:08:08.748 INFO:teuthology.orchestra.run.smithi096.stdout: "num_mons": 3 2024-05-28T06:08:08.748 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:08:08.748 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap": { 2024-05-28T06:08:08.748 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 33, 2024-05-28T06:08:08.748 INFO:teuthology.orchestra.run.smithi096.stdout: "num_osds": 8, 2024-05-28T06:08:08.748 INFO:teuthology.orchestra.run.smithi096.stdout: "num_up_osds": 8, 2024-05-28T06:08:08.748 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_up_since": 1716876238, 2024-05-28T06:08:08.748 INFO:teuthology.orchestra.run.smithi096.stdout: "num_in_osds": 8, 2024-05-28T06:08:08.748 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_in_since": 1716876229, 2024-05-28T06:08:08.748 INFO:teuthology.orchestra.run.smithi096.stdout: "num_remapped_pgs": 0 2024-05-28T06:08:08.748 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:08:08.748 INFO:teuthology.orchestra.run.smithi096.stdout: "pgmap": { 2024-05-28T06:08:08.748 INFO:teuthology.orchestra.run.smithi096.stdout: "pgs_by_state": [ 2024-05-28T06:08:08.749 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:08:08.749 INFO:teuthology.orchestra.run.smithi096.stdout: "state_name": "active+clean", 2024-05-28T06:08:08.749 INFO:teuthology.orchestra.run.smithi096.stdout: "count": 97 2024-05-28T06:08:08.749 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:08:08.749 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:08:08.749 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pgs": 97, 2024-05-28T06:08:08.749 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pools": 3, 2024-05-28T06:08:08.749 INFO:teuthology.orchestra.run.smithi096.stdout: "num_objects": 3207, 2024-05-28T06:08:08.749 INFO:teuthology.orchestra.run.smithi096.stdout: "data_bytes": 13330945173, 2024-05-28T06:08:08.749 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_used": 26924621824, 2024-05-28T06:08:08.749 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_avail": 746169491456, 2024-05-28T06:08:08.749 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_total": 773094113280, 2024-05-28T06:08:08.749 INFO:teuthology.orchestra.run.smithi096.stdout: "read_bytes_sec": 97132522, 2024-05-28T06:08:08.750 INFO:teuthology.orchestra.run.smithi096.stdout: "write_bytes_sec": 99238706, 2024-05-28T06:08:08.750 INFO:teuthology.orchestra.run.smithi096.stdout: "read_op_per_sec": 23, 2024-05-28T06:08:08.750 INFO:teuthology.orchestra.run.smithi096.stdout: "write_op_per_sec": 27 2024-05-28T06:08:08.750 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:08:08.750 INFO:teuthology.orchestra.run.smithi096.stdout: "fsmap": { 2024-05-28T06:08:08.750 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 16, 2024-05-28T06:08:08.750 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 2, 2024-05-28T06:08:08.750 INFO:teuthology.orchestra.run.smithi096.stdout: "up": 1, 2024-05-28T06:08:08.750 INFO:teuthology.orchestra.run.smithi096.stdout: "in": 1, 2024-05-28T06:08:08.750 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 1, 2024-05-28T06:08:08.750 INFO:teuthology.orchestra.run.smithi096.stdout: "by_rank": [ 2024-05-28T06:08:08.750 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:08:08.750 INFO:teuthology.orchestra.run.smithi096.stdout: "filesystem_id": 2, 2024-05-28T06:08:08.750 INFO:teuthology.orchestra.run.smithi096.stdout: "rank": 0, 2024-05-28T06:08:08.751 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "d", 2024-05-28T06:08:08.751 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "up:active", 2024-05-28T06:08:08.751 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 5238 2024-05-28T06:08:08.751 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:08:08.751 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:08:08.751 INFO:teuthology.orchestra.run.smithi096.stdout: "up:standby": 3 2024-05-28T06:08:08.751 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:08:08.751 INFO:teuthology.orchestra.run.smithi096.stdout: "mgrmap": { 2024-05-28T06:08:08.751 INFO:teuthology.orchestra.run.smithi096.stdout: "available": true, 2024-05-28T06:08:08.751 INFO:teuthology.orchestra.run.smithi096.stdout: "num_standbys": 1, 2024-05-28T06:08:08.751 INFO:teuthology.orchestra.run.smithi096.stdout: "modules": [ 2024-05-28T06:08:08.751 INFO:teuthology.orchestra.run.smithi096.stdout: "iostat", 2024-05-28T06:08:08.751 INFO:teuthology.orchestra.run.smithi096.stdout: "nfs", 2024-05-28T06:08:08.751 INFO:teuthology.orchestra.run.smithi096.stdout: "restful" 2024-05-28T06:08:08.751 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:08:08.752 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:08:08.752 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:08:08.752 INFO:teuthology.orchestra.run.smithi096.stdout: "servicemap": { 2024-05-28T06:08:08.752 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 7, 2024-05-28T06:08:08.752 INFO:teuthology.orchestra.run.smithi096.stdout: "modified": "2024-05-28T06:06:33.535436+0000", 2024-05-28T06:08:08.752 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:08:08.752 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:08:08.752 INFO:teuthology.orchestra.run.smithi096.stdout: "progress_events": { 2024-05-28T06:08:08.752 INFO:teuthology.orchestra.run.smithi096.stdout: "18e91b7c-9da2-4cf3-88f8-e7e2859c593b": { 2024-05-28T06:08:08.752 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "All cloning finished, waiting 20.0 seconds before exiting (15s)\n [============================] ", 2024-05-28T06:08:08.752 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": 1, 2024-05-28T06:08:08.752 INFO:teuthology.orchestra.run.smithi096.stdout: "add_to_ceph_s": true 2024-05-28T06:08:08.752 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:08:08.752 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:08:08.752 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:08:08.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.747+0000 7fd3a8e00640 1 -- 172.21.15.96:0/3846905741 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd38006cad0 msgr2=0x7fd38006ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:08.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.747+0000 7fd3a8e00640 1 --2- 172.21.15.96:0/3846905741 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd38006cad0 0x7fd38006ef90 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fd3a41f7840 tx=0x7fd38c008040 comp rx=0 tx=0).stop 2024-05-28T06:08:08.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.747+0000 7fd3a8e00640 1 -- 172.21.15.96:0/3846905741 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd3880039b0 msgr2=0x7fd3a40c9300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:08.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.747+0000 7fd3a8e00640 1 --2- 172.21.15.96:0/3846905741 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd3880039b0 0x7fd3a40c9300 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fd394001970 tx=0x7fd3a40d37f0 comp rx=0 tx=0).stop 2024-05-28T06:08:08.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.747+0000 7fd3a8e00640 1 -- 172.21.15.96:0/3846905741 shutdown_connections 2024-05-28T06:08:08.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.747+0000 7fd3a8e00640 1 --2- 172.21.15.96:0/3846905741 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd38006cad0 0x7fd38006ef90 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:08.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.747+0000 7fd3a8e00640 1 --2- 172.21.15.96:0/3846905741 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd3a40cc6c0 0x7fd3a40ccb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:08.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.747+0000 7fd3a8e00640 1 --2- 172.21.15.96:0/3846905741 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd3a40c9f40 0x7fd3a40ca3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:08.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.747+0000 7fd3a8e00640 1 --2- 172.21.15.96:0/3846905741 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd3880039b0 0x7fd3a40c9300 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:08.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.747+0000 7fd3a8e00640 1 -- 172.21.15.96:0/3846905741 >> 172.21.15.96:0/3846905741 conn(0x7fd3a409d910 msgr2=0x7fd3a40b1170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:08.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.747+0000 7fd3a8e00640 1 -- 172.21.15.96:0/3846905741 shutdown_connections 2024-05-28T06:08:08.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:08.747+0000 7fd3a8e00640 1 -- 172.21.15.96:0/3846905741 wait complete. 2024-05-28T06:08:13.767 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-05-28T06:08:13.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:13.972+0000 7fe6f3e00640 1 Processor -- start 2024-05-28T06:08:13.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:13.973+0000 7fe6f3e00640 1 -- start start 2024-05-28T06:08:13.976 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:13.974+0000 7fe6f3e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fe6f415d170 0x7fe6f40c50c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:13.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:13.974+0000 7fe6f3e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fe6f4163dc0 0x7fe6f40d0460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:13.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:13.974+0000 7fe6f3e00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6f40d0e10 con 0x7fe6f40d09a0 2024-05-28T06:08:13.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:13.974+0000 7fe6f3e00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fe6f40adb00 con 0x7fe6f415d170 2024-05-28T06:08:13.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:13.974+0000 7fe6f3e00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fe6f40aefa0 con 0x7fe6f4163dc0 2024-05-28T06:08:13.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:13.975+0000 7fe6f2a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fe6f415d170 0x7fe6f40c50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:13.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:13.975+0000 7fe6f2a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fe6f415d170 0x7fe6f40c50c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:59304/0 (socket says 172.21.15.96:59304) 2024-05-28T06:08:13.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:13.975+0000 7fe6f2a00640 1 -- 172.21.15.96:0/2428122639 learned_addr learned my addr 172.21.15.96:0/2428122639 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:08:13.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:13.975+0000 7fe6ea000640 1 --2- 172.21.15.96:0/2428122639 >> v2:172.21.15.123:3300/0 conn(0x7fe6f4163dc0 0x7fe6f40d0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:13.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:13.975+0000 7fe6f2a00640 1 -- 172.21.15.96:0/2428122639 >> v2:172.21.15.123:3300/0 conn(0x7fe6f4163dc0 msgr2=0x7fe6f40d0460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:13.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:13.975+0000 7fe6f2a00640 1 --2- 172.21.15.96:0/2428122639 >> v2:172.21.15.123:3300/0 conn(0x7fe6f4163dc0 0x7fe6f40d0460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:13.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:13.975+0000 7fe6f2a00640 1 -- 172.21.15.96:0/2428122639 >> v1:172.21.15.123:6789/0 conn(0x7fe6f40d09a0 legacy=0x7fe6f40b4470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:13.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:13.975+0000 7fe6f2a00640 1 -- 172.21.15.96:0/2428122639 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6f415d550 con 0x7fe6f415d170 2024-05-28T06:08:13.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:13.975+0000 7fe6ea000640 1 --2- 172.21.15.96:0/2428122639 >> v2:172.21.15.123:3300/0 conn(0x7fe6f4163dc0 0x7fe6f40d0460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:08:13.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:13.976+0000 7fe6f2a00640 1 --2- 172.21.15.96:0/2428122639 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe6f415d170 0x7fe6f40c50c0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fe6f40cdc80 tx=0x7fe6e402a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5774d7f21248a242 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:13.979 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:13.977+0000 7fe6f2000640 1 -- 172.21.15.96:0/2428122639 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6e4037070 con 0x7fe6f415d170 2024-05-28T06:08:13.979 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:13.977+0000 7fe6f2000640 1 -- 172.21.15.96:0/2428122639 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6e4032440 con 0x7fe6f415d170 2024-05-28T06:08:13.979 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:13.977+0000 7fe6f2000640 1 -- 172.21.15.96:0/2428122639 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6e4032810 con 0x7fe6f415d170 2024-05-28T06:08:13.980 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:13.978+0000 7fe6f3e00640 1 -- 172.21.15.96:0/2428122639 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe6f415d170 msgr2=0x7fe6f40c50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:13.980 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:13.978+0000 7fe6f3e00640 1 --2- 172.21.15.96:0/2428122639 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe6f415d170 0x7fe6f40c50c0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fe6f40cdc80 tx=0x7fe6e402a230 comp rx=0 tx=0).stop 2024-05-28T06:08:13.980 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:13.978+0000 7fe6f3e00640 1 -- 172.21.15.96:0/2428122639 shutdown_connections 2024-05-28T06:08:13.980 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:13.978+0000 7fe6f3e00640 1 --2- 172.21.15.96:0/2428122639 >> v2:172.21.15.123:3300/0 conn(0x7fe6f4163dc0 0x7fe6f40d0460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:13.980 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:13.978+0000 7fe6f3e00640 1 --2- 172.21.15.96:0/2428122639 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe6f415d170 0x7fe6f40c50c0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:13.980 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:13.978+0000 7fe6f3e00640 1 -- 172.21.15.96:0/2428122639 >> 172.21.15.96:0/2428122639 conn(0x7fe6f406bae0 msgr2=0x7fe6f406bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:13.980 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:13.978+0000 7fe6f3e00640 1 -- 172.21.15.96:0/2428122639 shutdown_connections 2024-05-28T06:08:13.980 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:13.978+0000 7fe6f3e00640 1 -- 172.21.15.96:0/2428122639 wait complete. 2024-05-28T06:08:13.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:13.979+0000 7fe6f3e00640 1 Processor -- start 2024-05-28T06:08:13.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:13.980+0000 7fe6f3e00640 1 -- start start 2024-05-28T06:08:13.982 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:13.980+0000 7fe6f3e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe6f40d09a0 0x7fe6f415a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:13.982 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:13.981+0000 7fe6f3e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe6f415d170 0x7fe6f40b0480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:13.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:13.981+0000 7fe6f3e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe6f4163dc0 0x7fe6f40b09c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:13.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:13.981+0000 7fe6f3e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe6f40ad790 con 0x7fe6f40d09a0 2024-05-28T06:08:13.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:13.981+0000 7fe6f3e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe6f40ad4b0 con 0x7fe6f4163dc0 2024-05-28T06:08:13.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:13.981+0000 7fe6f3e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fe6f40b3150 con 0x7fe6f415d170 2024-05-28T06:08:13.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:13.981+0000 7fe6f2a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe6f40d09a0 0x7fe6f415a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:13.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:13.981+0000 7fe6f2a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe6f40d09a0 0x7fe6f415a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:59320/0 (socket says 172.21.15.96:59320) 2024-05-28T06:08:13.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:13.981+0000 7fe6f2a00640 1 -- 172.21.15.96:0/3391673426 learned_addr learned my addr 172.21.15.96:0/3391673426 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:08:13.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:13.981+0000 7fe6f2a00640 1 -- 172.21.15.96:0/3391673426 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe6f415d170 msgr2=0x7fe6f40b0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:13.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:13.981+0000 7fe6f2a00640 1 --2- 172.21.15.96:0/3391673426 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe6f415d170 0x7fe6f40b0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:13.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:13.981+0000 7fe6f2a00640 1 -- 172.21.15.96:0/3391673426 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe6f4163dc0 msgr2=0x7fe6f40b09c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:13.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:13.981+0000 7fe6f2a00640 1 --2- 172.21.15.96:0/3391673426 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe6f4163dc0 0x7fe6f40b09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:13.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:13.981+0000 7fe6f2a00640 1 -- 172.21.15.96:0/3391673426 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6f4073d40 con 0x7fe6f40d09a0 2024-05-28T06:08:13.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:13.982+0000 7fe6f2a00640 1 --2- 172.21.15.96:0/3391673426 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe6f40d09a0 0x7fe6f415a120 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fe6e4000c00 tx=0x7fe6e40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:13.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:13.982+0000 7fe6eb400640 1 -- 172.21.15.96:0/3391673426 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6e4037040 con 0x7fe6f40d09a0 2024-05-28T06:08:13.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:13.982+0000 7fe6eb400640 1 -- 172.21.15.96:0/3391673426 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6e4042070 con 0x7fe6f40d09a0 2024-05-28T06:08:13.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:13.982+0000 7fe6eb400640 1 -- 172.21.15.96:0/3391673426 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6e4030bb0 con 0x7fe6f40d09a0 2024-05-28T06:08:13.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:13.982+0000 7fe6f3e00640 1 -- 172.21.15.96:0/3391673426 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6f40729b0 con 0x7fe6f40d09a0 2024-05-28T06:08:13.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:13.982+0000 7fe6f3e00640 1 -- 172.21.15.96:0/3391673426 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6f4071ed0 con 0x7fe6f40d09a0 2024-05-28T06:08:13.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:13.983+0000 7fe6f3e00640 1 -- 172.21.15.96:0/3391673426 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6f40689e0 con 0x7fe6f40d09a0 2024-05-28T06:08:13.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:13.984+0000 7fe6eb400640 1 -- 172.21.15.96:0/3391673426 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 8) ==== 91193+0+0 (secure 0 0 0) 0x7fe6e4047460 con 0x7fe6f40d09a0 2024-05-28T06:08:13.986 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:13.985+0000 7fe6eb400640 1 --2- 172.21.15.96:0/3391673426 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe6d006caa0 0x7fe6d006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:13.986 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:13.985+0000 7fe6eb400640 1 -- 172.21.15.96:0/3391673426 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7fe6e40acb60 con 0x7fe6f40d09a0 2024-05-28T06:08:13.987 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:13.986+0000 7fe6ea000640 1 --2- 172.21.15.96:0/3391673426 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe6d006caa0 0x7fe6d006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:13.988 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:13.987+0000 7fe6ea000640 1 --2- 172.21.15.96:0/3391673426 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe6d006caa0 0x7fe6d006ef60 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fe6dc006fd0 tx=0x7fe6dc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:13.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:13.990+0000 7fe6eb400640 1 -- 172.21.15.96:0/3391673426 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe6e4045360 con 0x7fe6f40d09a0 2024-05-28T06:08:14.257 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:14.255+0000 7fe6f3e00640 1 -- 172.21.15.96:0/3391673426 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fe6f4099810 con 0x7fe6f40d09a0 2024-05-28T06:08:14.257 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:14.256+0000 7fe6eb400640 1 -- 172.21.15.96:0/3391673426 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+2109 (secure 0 0 0) 0x7fe6e407ac70 con 0x7fe6f40d09a0 2024-05-28T06:08:14.258 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:08:14.258 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:08:14.258 INFO:teuthology.orchestra.run.smithi096.stdout: "fsid": "e4a831b5-e6d1-47a7-96cb-bf85154ee0d2", 2024-05-28T06:08:14.258 INFO:teuthology.orchestra.run.smithi096.stdout: "health": { 2024-05-28T06:08:14.258 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "HEALTH_OK", 2024-05-28T06:08:14.258 INFO:teuthology.orchestra.run.smithi096.stdout: "checks": {}, 2024-05-28T06:08:14.258 INFO:teuthology.orchestra.run.smithi096.stdout: "mutes": [] 2024-05-28T06:08:14.258 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:08:14.258 INFO:teuthology.orchestra.run.smithi096.stdout: "election_epoch": 8, 2024-05-28T06:08:14.258 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum": [ 2024-05-28T06:08:14.258 INFO:teuthology.orchestra.run.smithi096.stdout: 0, 2024-05-28T06:08:14.258 INFO:teuthology.orchestra.run.smithi096.stdout: 1, 2024-05-28T06:08:14.259 INFO:teuthology.orchestra.run.smithi096.stdout: 2 2024-05-28T06:08:14.259 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:08:14.259 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_names": [ 2024-05-28T06:08:14.259 INFO:teuthology.orchestra.run.smithi096.stdout: "a", 2024-05-28T06:08:14.259 INFO:teuthology.orchestra.run.smithi096.stdout: "b", 2024-05-28T06:08:14.259 INFO:teuthology.orchestra.run.smithi096.stdout: "c" 2024-05-28T06:08:14.259 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:08:14.259 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_age": 267, 2024-05-28T06:08:14.259 INFO:teuthology.orchestra.run.smithi096.stdout: "monmap": { 2024-05-28T06:08:14.259 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 1, 2024-05-28T06:08:14.259 INFO:teuthology.orchestra.run.smithi096.stdout: "min_mon_release_name": "squid", 2024-05-28T06:08:14.259 INFO:teuthology.orchestra.run.smithi096.stdout: "num_mons": 3 2024-05-28T06:08:14.259 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:08:14.259 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap": { 2024-05-28T06:08:14.260 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 33, 2024-05-28T06:08:14.260 INFO:teuthology.orchestra.run.smithi096.stdout: "num_osds": 8, 2024-05-28T06:08:14.260 INFO:teuthology.orchestra.run.smithi096.stdout: "num_up_osds": 8, 2024-05-28T06:08:14.260 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_up_since": 1716876238, 2024-05-28T06:08:14.260 INFO:teuthology.orchestra.run.smithi096.stdout: "num_in_osds": 8, 2024-05-28T06:08:14.260 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_in_since": 1716876229, 2024-05-28T06:08:14.260 INFO:teuthology.orchestra.run.smithi096.stdout: "num_remapped_pgs": 0 2024-05-28T06:08:14.260 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:08:14.260 INFO:teuthology.orchestra.run.smithi096.stdout: "pgmap": { 2024-05-28T06:08:14.260 INFO:teuthology.orchestra.run.smithi096.stdout: "pgs_by_state": [ 2024-05-28T06:08:14.260 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:08:14.260 INFO:teuthology.orchestra.run.smithi096.stdout: "state_name": "active+clean", 2024-05-28T06:08:14.260 INFO:teuthology.orchestra.run.smithi096.stdout: "count": 97 2024-05-28T06:08:14.261 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:08:14.261 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:08:14.261 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pgs": 97, 2024-05-28T06:08:14.261 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pools": 3, 2024-05-28T06:08:14.261 INFO:teuthology.orchestra.run.smithi096.stdout: "num_objects": 3207, 2024-05-28T06:08:14.261 INFO:teuthology.orchestra.run.smithi096.stdout: "data_bytes": 13330961023, 2024-05-28T06:08:14.261 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_used": 26924654592, 2024-05-28T06:08:14.261 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_avail": 746169458688, 2024-05-28T06:08:14.261 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_total": 773094113280, 2024-05-28T06:08:14.261 INFO:teuthology.orchestra.run.smithi096.stdout: "read_bytes_sec": 85, 2024-05-28T06:08:14.261 INFO:teuthology.orchestra.run.smithi096.stdout: "write_bytes_sec": 4094, 2024-05-28T06:08:14.261 INFO:teuthology.orchestra.run.smithi096.stdout: "read_op_per_sec": 0, 2024-05-28T06:08:14.261 INFO:teuthology.orchestra.run.smithi096.stdout: "write_op_per_sec": 1 2024-05-28T06:08:14.262 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:08:14.262 INFO:teuthology.orchestra.run.smithi096.stdout: "fsmap": { 2024-05-28T06:08:14.262 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 16, 2024-05-28T06:08:14.262 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 2, 2024-05-28T06:08:14.262 INFO:teuthology.orchestra.run.smithi096.stdout: "up": 1, 2024-05-28T06:08:14.262 INFO:teuthology.orchestra.run.smithi096.stdout: "in": 1, 2024-05-28T06:08:14.262 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 1, 2024-05-28T06:08:14.262 INFO:teuthology.orchestra.run.smithi096.stdout: "by_rank": [ 2024-05-28T06:08:14.262 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:08:14.262 INFO:teuthology.orchestra.run.smithi096.stdout: "filesystem_id": 2, 2024-05-28T06:08:14.262 INFO:teuthology.orchestra.run.smithi096.stdout: "rank": 0, 2024-05-28T06:08:14.262 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "d", 2024-05-28T06:08:14.262 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "up:active", 2024-05-28T06:08:14.262 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 5238 2024-05-28T06:08:14.263 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:08:14.263 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:08:14.263 INFO:teuthology.orchestra.run.smithi096.stdout: "up:standby": 3 2024-05-28T06:08:14.263 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:08:14.263 INFO:teuthology.orchestra.run.smithi096.stdout: "mgrmap": { 2024-05-28T06:08:14.263 INFO:teuthology.orchestra.run.smithi096.stdout: "available": true, 2024-05-28T06:08:14.263 INFO:teuthology.orchestra.run.smithi096.stdout: "num_standbys": 1, 2024-05-28T06:08:14.263 INFO:teuthology.orchestra.run.smithi096.stdout: "modules": [ 2024-05-28T06:08:14.263 INFO:teuthology.orchestra.run.smithi096.stdout: "iostat", 2024-05-28T06:08:14.263 INFO:teuthology.orchestra.run.smithi096.stdout: "nfs", 2024-05-28T06:08:14.263 INFO:teuthology.orchestra.run.smithi096.stdout: "restful" 2024-05-28T06:08:14.263 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:08:14.263 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:08:14.263 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:08:14.264 INFO:teuthology.orchestra.run.smithi096.stdout: "servicemap": { 2024-05-28T06:08:14.264 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 7, 2024-05-28T06:08:14.264 INFO:teuthology.orchestra.run.smithi096.stdout: "modified": "2024-05-28T06:06:33.535436+0000", 2024-05-28T06:08:14.264 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:08:14.264 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:08:14.264 INFO:teuthology.orchestra.run.smithi096.stdout: "progress_events": { 2024-05-28T06:08:14.264 INFO:teuthology.orchestra.run.smithi096.stdout: "18e91b7c-9da2-4cf3-88f8-e7e2859c593b": { 2024-05-28T06:08:14.264 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "All cloning finished, waiting 14.0 seconds before exiting (21s)\n [============================] ", 2024-05-28T06:08:14.264 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": 1, 2024-05-28T06:08:14.264 INFO:teuthology.orchestra.run.smithi096.stdout: "add_to_ceph_s": true 2024-05-28T06:08:14.264 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:08:14.264 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:08:14.264 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:08:14.265 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:14.258+0000 7fe6f3e00640 1 -- 172.21.15.96:0/3391673426 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe6d006caa0 msgr2=0x7fe6d006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:14.265 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:14.258+0000 7fe6f3e00640 1 --2- 172.21.15.96:0/3391673426 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe6d006caa0 0x7fe6d006ef60 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fe6dc006fd0 tx=0x7fe6dc008040 comp rx=0 tx=0).stop 2024-05-28T06:08:14.265 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:14.258+0000 7fe6f3e00640 1 -- 172.21.15.96:0/3391673426 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe6f40d09a0 msgr2=0x7fe6f415a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:14.265 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:14.258+0000 7fe6f3e00640 1 --2- 172.21.15.96:0/3391673426 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe6f40d09a0 0x7fe6f415a120 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fe6e4000c00 tx=0x7fe6e40046c0 comp rx=0 tx=0).stop 2024-05-28T06:08:14.265 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:14.258+0000 7fe6f3e00640 1 -- 172.21.15.96:0/3391673426 shutdown_connections 2024-05-28T06:08:14.266 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:14.258+0000 7fe6f3e00640 1 --2- 172.21.15.96:0/3391673426 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe6d006caa0 0x7fe6d006ef60 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:14.266 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:14.259+0000 7fe6f3e00640 1 --2- 172.21.15.96:0/3391673426 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe6f4163dc0 0x7fe6f40b09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:14.266 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:14.259+0000 7fe6f3e00640 1 --2- 172.21.15.96:0/3391673426 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe6f415d170 0x7fe6f40b0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:14.266 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:14.259+0000 7fe6f3e00640 1 --2- 172.21.15.96:0/3391673426 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe6f40d09a0 0x7fe6f415a120 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:14.266 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:14.259+0000 7fe6f3e00640 1 -- 172.21.15.96:0/3391673426 >> 172.21.15.96:0/3391673426 conn(0x7fe6f406bae0 msgr2=0x7fe6f40b26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:14.266 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:14.259+0000 7fe6f3e00640 1 -- 172.21.15.96:0/3391673426 shutdown_connections 2024-05-28T06:08:14.266 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:14.259+0000 7fe6f3e00640 1 -- 172.21.15.96:0/3391673426 wait complete. 2024-05-28T06:08:19.281 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-05-28T06:08:19.483 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.482+0000 7fdef0c00640 1 Processor -- start 2024-05-28T06:08:19.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.482+0000 7fdef0c00640 1 -- start start 2024-05-28T06:08:19.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.484+0000 7fdef0c00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fdeec155a00 0x7fdeec15baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:19.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.484+0000 7fdef0c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdeec0d0970 0x7fdeec06dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:19.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.484+0000 7fdef0c00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdeec15c0b0 con 0x7fdeec06f390 2024-05-28T06:08:19.486 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.484+0000 7fdef0c00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fdeec0ccf20 con 0x7fdeec155a00 2024-05-28T06:08:19.486 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.484+0000 7fdef0c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fdeec0d0490 con 0x7fdeec0d0970 2024-05-28T06:08:19.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.484+0000 7fdeea800640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fdeec06f390 0x7fdeec15f5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:58872/0 (socket says 172.21.15.96:58872) 2024-05-28T06:08:19.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.484+0000 7fdeea800640 1 -- 172.21.15.96:0/3959538335 learned_addr learned my addr 172.21.15.96:0/3959538335 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:08:19.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.485+0000 7fdee9400640 1 --2- 172.21.15.96:0/3959538335 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdeec0d0970 0x7fdeec06dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:19.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.485+0000 7fdee9e00640 1 --2- 172.21.15.96:0/3959538335 >> v2:172.21.15.123:3300/0 conn(0x7fdeec155a00 0x7fdeec15baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:19.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.485+0000 7fdee8a00640 1 -- 172.21.15.96:0/3959538335 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 836521523 0 0) 0x7fdeec15c0b0 con 0x7fdeec06f390 2024-05-28T06:08:19.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.485+0000 7fdee8a00640 1 -- 172.21.15.96:0/3959538335 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fded0003520 con 0x7fdeec06f390 2024-05-28T06:08:19.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.485+0000 7fdee8a00640 1 -- 172.21.15.96:0/3959538335 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3364566268 0 0) 0x7fded0003520 con 0x7fdeec06f390 2024-05-28T06:08:19.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.485+0000 7fdee8a00640 1 -- 172.21.15.96:0/3959538335 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdeec0d0970 msgr2=0x7fdeec06dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:19.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.485+0000 7fdee8a00640 1 --2- 172.21.15.96:0/3959538335 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdeec0d0970 0x7fdeec06dcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:19.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.485+0000 7fdee8a00640 1 -- 172.21.15.96:0/3959538335 >> v2:172.21.15.123:3300/0 conn(0x7fdeec155a00 msgr2=0x7fdeec15baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:19.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.485+0000 7fdee8a00640 1 --2- 172.21.15.96:0/3959538335 >> v2:172.21.15.123:3300/0 conn(0x7fdeec155a00 0x7fdeec15baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:19.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.485+0000 7fdee8a00640 1 -- 172.21.15.96:0/3959538335 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdeec15c390 con 0x7fdeec06f390 2024-05-28T06:08:19.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.485+0000 7fdee8a00640 1 -- 172.21.15.96:0/3959538335 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7fdee40019f0 con 0x7fdeec06f390 2024-05-28T06:08:19.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.485+0000 7fdee8a00640 1 -- 172.21.15.96:0/3959538335 >> v1:172.21.15.96:6789/0 conn(0x7fdeec06f390 legacy=0x7fdeec15f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:19.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.485+0000 7fdee8a00640 1 --2- 172.21.15.96:0/3959538335 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fded00039b0 0x7fded0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:19.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.485+0000 7fdee8a00640 1 --2- 172.21.15.96:0/3959538335 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fded0004800 0x7fded0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:19.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.485+0000 7fdee8a00640 1 --2- 172.21.15.96:0/3959538335 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fded00051c0 0x7fded000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:19.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.485+0000 7fdee8a00640 1 -- 172.21.15.96:0/3959538335 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fded000bfb0 con 0x7fded0004800 2024-05-28T06:08:19.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.485+0000 7fdee8a00640 1 -- 172.21.15.96:0/3959538335 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fded000c130 con 0x7fded00039b0 2024-05-28T06:08:19.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.485+0000 7fdee8a00640 1 -- 172.21.15.96:0/3959538335 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fded000c330 con 0x7fded00051c0 2024-05-28T06:08:19.489 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.485+0000 7fdee9e00640 1 --2- 172.21.15.96:0/3959538335 >> v2:172.21.15.123:3300/0 conn(0x7fdeec155a00 0x7fdeec15baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-28T06:08:19.489 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.486+0000 7fdee9400640 1 --2- 172.21.15.96:0/3959538335 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdeec0d0970 0x7fdeec06dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-28T06:08:19.489 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.486+0000 7fdee9400640 1 --2- 172.21.15.96:0/3959538335 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fded0004800 0x7fded0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:19.490 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.486+0000 7fdeea800640 1 --2- 172.21.15.96:0/3959538335 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fded00051c0 0x7fded000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:19.490 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.486+0000 7fdee9e00640 1 --2- 172.21.15.96:0/3959538335 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fded00039b0 0x7fded0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:19.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.486+0000 7fdee9400640 1 -- 172.21.15.96:0/3959538335 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fded00051c0 msgr2=0x7fded000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:19.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.486+0000 7fdee9400640 1 --2- 172.21.15.96:0/3959538335 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fded00051c0 0x7fded000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:19.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.486+0000 7fdee9400640 1 -- 172.21.15.96:0/3959538335 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fded00039b0 msgr2=0x7fded0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:19.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.486+0000 7fdee9400640 1 --2- 172.21.15.96:0/3959538335 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fded00039b0 0x7fded0003dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:19.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.486+0000 7fdee9400640 1 -- 172.21.15.96:0/3959538335 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fded0003520 con 0x7fded0004800 2024-05-28T06:08:19.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.486+0000 7fdeea800640 1 --2- 172.21.15.96:0/3959538335 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fded00051c0 0x7fded000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:08:19.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.487+0000 7fdee9400640 1 --2- 172.21.15.96:0/3959538335 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fded0004800 0x7fded0004c80 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fdeec0cddc0 tx=0x7fded0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20e1c8550ac80c99 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:19.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.487+0000 7fdee8a00640 1 -- 172.21.15.96:0/3959538335 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fded8037070 con 0x7fded0004800 2024-05-28T06:08:19.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.488+0000 7fdee8a00640 1 -- 172.21.15.96:0/3959538335 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fded802ab10 con 0x7fded0004800 2024-05-28T06:08:19.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.488+0000 7fdee8a00640 1 -- 172.21.15.96:0/3959538335 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fded800f040 con 0x7fded0004800 2024-05-28T06:08:19.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.488+0000 7fdef0c00640 1 -- 172.21.15.96:0/3959538335 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fded0004800 msgr2=0x7fded0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:19.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.488+0000 7fdef0c00640 1 --2- 172.21.15.96:0/3959538335 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fded0004800 0x7fded0004c80 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fdeec0cddc0 tx=0x7fded0005830 comp rx=0 tx=0).stop 2024-05-28T06:08:19.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.488+0000 7fdeea800640 1 -- 172.21.15.96:0/3959538335 reap_dead start 2024-05-28T06:08:19.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.488+0000 7fdef0c00640 1 -- 172.21.15.96:0/3959538335 shutdown_connections 2024-05-28T06:08:19.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.488+0000 7fdef0c00640 1 -- 172.21.15.96:0/3959538335 >> 172.21.15.96:0/3959538335 conn(0x7fdeec067ce0 msgr2=0x7fdeec06b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:19.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.488+0000 7fdef0c00640 1 -- 172.21.15.96:0/3959538335 shutdown_connections 2024-05-28T06:08:19.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.488+0000 7fdef0c00640 1 -- 172.21.15.96:0/3959538335 wait complete. 2024-05-28T06:08:19.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.490+0000 7fdef0c00640 1 Processor -- start 2024-05-28T06:08:19.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.490+0000 7fdef0c00640 1 -- start start 2024-05-28T06:08:19.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.491+0000 7fdef0c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fded0004800 0x7fdeec14af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:19.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.491+0000 7fdef0c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdeec149d90 0x7fdeec14e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:19.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.491+0000 7fdef0c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdeec14eb20 0x7fdeec152210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:19.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.491+0000 7fdef0c00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fdeec15f090 con 0x7fdeec149d90 2024-05-28T06:08:19.494 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.491+0000 7fdef0c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fdeec15c950 con 0x7fded0004800 2024-05-28T06:08:19.494 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.491+0000 7fdef0c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fdeec155de0 con 0x7fdeec14eb20 2024-05-28T06:08:19.494 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.491+0000 7fdee9400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdeec149d90 0x7fdeec14e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:19.494 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.491+0000 7fdee9400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdeec149d90 0x7fdeec14e5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:45412/0 (socket says 172.21.15.96:45412) 2024-05-28T06:08:19.494 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.491+0000 7fdee9400640 1 -- 172.21.15.96:0/3524567048 learned_addr learned my addr 172.21.15.96:0/3524567048 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:08:19.494 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.492+0000 7fdee9e00640 1 --2- 172.21.15.96:0/3524567048 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fded0004800 0x7fdeec14af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:19.494 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.492+0000 7fdeea800640 1 --2- 172.21.15.96:0/3524567048 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdeec14eb20 0x7fdeec152210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:19.494 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.492+0000 7fdee9400640 1 -- 172.21.15.96:0/3524567048 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdeec14eb20 msgr2=0x7fdeec152210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:19.494 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.492+0000 7fdee9400640 1 --2- 172.21.15.96:0/3524567048 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdeec14eb20 0x7fdeec152210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:19.494 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.492+0000 7fdee9400640 1 -- 172.21.15.96:0/3524567048 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fded0004800 msgr2=0x7fdeec14af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:19.494 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.492+0000 7fdee9400640 1 --2- 172.21.15.96:0/3524567048 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fded0004800 0x7fdeec14af50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:19.494 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.492+0000 7fdee9400640 1 -- 172.21.15.96:0/3524567048 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdeec151d60 con 0x7fdeec149d90 2024-05-28T06:08:19.494 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.492+0000 7fdeea800640 1 --2- 172.21.15.96:0/3524567048 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdeec14eb20 0x7fdeec152210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:08:19.495 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.492+0000 7fdee9400640 1 --2- 172.21.15.96:0/3524567048 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdeec149d90 0x7fdeec14e5e0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fded80097c0 tx=0x7fded8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:19.495 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.492+0000 7fdee2000640 1 -- 172.21.15.96:0/3524567048 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fded8037050 con 0x7fdeec149d90 2024-05-28T06:08:19.496 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.493+0000 7fdee2000640 1 -- 172.21.15.96:0/3524567048 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fded8043070 con 0x7fdeec149d90 2024-05-28T06:08:19.496 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.493+0000 7fdee2000640 1 -- 172.21.15.96:0/3524567048 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fded803c850 con 0x7fdeec149d90 2024-05-28T06:08:19.496 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.493+0000 7fdef0c00640 1 -- 172.21.15.96:0/3524567048 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdeec14b530 con 0x7fdeec149d90 2024-05-28T06:08:19.496 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.493+0000 7fdef0c00640 1 -- 172.21.15.96:0/3524567048 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdeec14baf0 con 0x7fdeec149d90 2024-05-28T06:08:19.496 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.494+0000 7fdef0c00640 1 -- 172.21.15.96:0/3524567048 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdeec0689e0 con 0x7fdeec149d90 2024-05-28T06:08:19.496 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.495+0000 7fdee2000640 1 -- 172.21.15.96:0/3524567048 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 8) ==== 91193+0+0 (secure 0 0 0) 0x7fded803c9f0 con 0x7fdeec149d90 2024-05-28T06:08:19.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.496+0000 7fdee2000640 1 --2- 172.21.15.96:0/3524567048 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdecc06cad0 0x7fdecc06ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:19.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.496+0000 7fdee2000640 1 -- 172.21.15.96:0/3524567048 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7fded80ac180 con 0x7fdeec149d90 2024-05-28T06:08:19.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.496+0000 7fdee9e00640 1 --2- 172.21.15.96:0/3524567048 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdecc06cad0 0x7fdecc06ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:19.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.498+0000 7fdee9e00640 1 --2- 172.21.15.96:0/3524567048 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdecc06cad0 0x7fdecc06ef90 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fdedc006fd0 tx=0x7fdedc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:19.502 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.501+0000 7fdee2000640 1 -- 172.21.15.96:0/3524567048 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fded804c020 con 0x7fdeec149d90 2024-05-28T06:08:19.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.778+0000 7fdef0c00640 1 -- 172.21.15.96:0/3524567048 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fdeec06f170 con 0x7fdeec149d90 2024-05-28T06:08:19.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.779+0000 7fdee2000640 1 -- 172.21.15.96:0/3524567048 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+2079 (secure 0 0 0) 0x7fded807a290 con 0x7fdeec149d90 2024-05-28T06:08:19.780 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:08:19.781 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:08:19.781 INFO:teuthology.orchestra.run.smithi096.stdout: "fsid": "e4a831b5-e6d1-47a7-96cb-bf85154ee0d2", 2024-05-28T06:08:19.781 INFO:teuthology.orchestra.run.smithi096.stdout: "health": { 2024-05-28T06:08:19.781 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "HEALTH_OK", 2024-05-28T06:08:19.781 INFO:teuthology.orchestra.run.smithi096.stdout: "checks": {}, 2024-05-28T06:08:19.781 INFO:teuthology.orchestra.run.smithi096.stdout: "mutes": [] 2024-05-28T06:08:19.781 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:08:19.781 INFO:teuthology.orchestra.run.smithi096.stdout: "election_epoch": 8, 2024-05-28T06:08:19.781 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum": [ 2024-05-28T06:08:19.781 INFO:teuthology.orchestra.run.smithi096.stdout: 0, 2024-05-28T06:08:19.781 INFO:teuthology.orchestra.run.smithi096.stdout: 1, 2024-05-28T06:08:19.781 INFO:teuthology.orchestra.run.smithi096.stdout: 2 2024-05-28T06:08:19.782 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:08:19.782 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_names": [ 2024-05-28T06:08:19.782 INFO:teuthology.orchestra.run.smithi096.stdout: "a", 2024-05-28T06:08:19.782 INFO:teuthology.orchestra.run.smithi096.stdout: "b", 2024-05-28T06:08:19.782 INFO:teuthology.orchestra.run.smithi096.stdout: "c" 2024-05-28T06:08:19.782 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:08:19.782 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_age": 273, 2024-05-28T06:08:19.782 INFO:teuthology.orchestra.run.smithi096.stdout: "monmap": { 2024-05-28T06:08:19.782 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 1, 2024-05-28T06:08:19.782 INFO:teuthology.orchestra.run.smithi096.stdout: "min_mon_release_name": "squid", 2024-05-28T06:08:19.782 INFO:teuthology.orchestra.run.smithi096.stdout: "num_mons": 3 2024-05-28T06:08:19.782 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:08:19.782 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap": { 2024-05-28T06:08:19.783 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 33, 2024-05-28T06:08:19.783 INFO:teuthology.orchestra.run.smithi096.stdout: "num_osds": 8, 2024-05-28T06:08:19.783 INFO:teuthology.orchestra.run.smithi096.stdout: "num_up_osds": 8, 2024-05-28T06:08:19.783 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_up_since": 1716876238, 2024-05-28T06:08:19.783 INFO:teuthology.orchestra.run.smithi096.stdout: "num_in_osds": 8, 2024-05-28T06:08:19.783 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_in_since": 1716876229, 2024-05-28T06:08:19.783 INFO:teuthology.orchestra.run.smithi096.stdout: "num_remapped_pgs": 0 2024-05-28T06:08:19.783 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:08:19.783 INFO:teuthology.orchestra.run.smithi096.stdout: "pgmap": { 2024-05-28T06:08:19.783 INFO:teuthology.orchestra.run.smithi096.stdout: "pgs_by_state": [ 2024-05-28T06:08:19.783 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:08:19.783 INFO:teuthology.orchestra.run.smithi096.stdout: "state_name": "active+clean", 2024-05-28T06:08:19.783 INFO:teuthology.orchestra.run.smithi096.stdout: "count": 97 2024-05-28T06:08:19.783 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:08:19.784 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:08:19.784 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pgs": 97, 2024-05-28T06:08:19.784 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pools": 3, 2024-05-28T06:08:19.784 INFO:teuthology.orchestra.run.smithi096.stdout: "num_objects": 3207, 2024-05-28T06:08:19.784 INFO:teuthology.orchestra.run.smithi096.stdout: "data_bytes": 13330961023, 2024-05-28T06:08:19.784 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_used": 26924654592, 2024-05-28T06:08:19.784 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_avail": 746169458688, 2024-05-28T06:08:19.784 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_total": 773094113280, 2024-05-28T06:08:19.784 INFO:teuthology.orchestra.run.smithi096.stdout: "write_bytes_sec": 1535, 2024-05-28T06:08:19.784 INFO:teuthology.orchestra.run.smithi096.stdout: "read_op_per_sec": 0, 2024-05-28T06:08:19.784 INFO:teuthology.orchestra.run.smithi096.stdout: "write_op_per_sec": 0 2024-05-28T06:08:19.784 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:08:19.784 INFO:teuthology.orchestra.run.smithi096.stdout: "fsmap": { 2024-05-28T06:08:19.784 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 16, 2024-05-28T06:08:19.785 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 2, 2024-05-28T06:08:19.785 INFO:teuthology.orchestra.run.smithi096.stdout: "up": 1, 2024-05-28T06:08:19.785 INFO:teuthology.orchestra.run.smithi096.stdout: "in": 1, 2024-05-28T06:08:19.785 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 1, 2024-05-28T06:08:19.785 INFO:teuthology.orchestra.run.smithi096.stdout: "by_rank": [ 2024-05-28T06:08:19.785 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:08:19.785 INFO:teuthology.orchestra.run.smithi096.stdout: "filesystem_id": 2, 2024-05-28T06:08:19.785 INFO:teuthology.orchestra.run.smithi096.stdout: "rank": 0, 2024-05-28T06:08:19.785 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "d", 2024-05-28T06:08:19.785 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "up:active", 2024-05-28T06:08:19.785 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 5238 2024-05-28T06:08:19.785 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:08:19.785 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:08:19.785 INFO:teuthology.orchestra.run.smithi096.stdout: "up:standby": 3 2024-05-28T06:08:19.786 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:08:19.786 INFO:teuthology.orchestra.run.smithi096.stdout: "mgrmap": { 2024-05-28T06:08:19.786 INFO:teuthology.orchestra.run.smithi096.stdout: "available": true, 2024-05-28T06:08:19.786 INFO:teuthology.orchestra.run.smithi096.stdout: "num_standbys": 1, 2024-05-28T06:08:19.786 INFO:teuthology.orchestra.run.smithi096.stdout: "modules": [ 2024-05-28T06:08:19.786 INFO:teuthology.orchestra.run.smithi096.stdout: "iostat", 2024-05-28T06:08:19.786 INFO:teuthology.orchestra.run.smithi096.stdout: "nfs", 2024-05-28T06:08:19.786 INFO:teuthology.orchestra.run.smithi096.stdout: "restful" 2024-05-28T06:08:19.786 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:08:19.786 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:08:19.786 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:08:19.786 INFO:teuthology.orchestra.run.smithi096.stdout: "servicemap": { 2024-05-28T06:08:19.786 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 7, 2024-05-28T06:08:19.786 INFO:teuthology.orchestra.run.smithi096.stdout: "modified": "2024-05-28T06:06:33.535436+0000", 2024-05-28T06:08:19.787 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:08:19.787 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:08:19.787 INFO:teuthology.orchestra.run.smithi096.stdout: "progress_events": { 2024-05-28T06:08:19.787 INFO:teuthology.orchestra.run.smithi096.stdout: "18e91b7c-9da2-4cf3-88f8-e7e2859c593b": { 2024-05-28T06:08:19.787 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "All cloning finished, waiting 10.0 seconds before exiting (25s)\n [============================] ", 2024-05-28T06:08:19.787 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": 1, 2024-05-28T06:08:19.787 INFO:teuthology.orchestra.run.smithi096.stdout: "add_to_ceph_s": true 2024-05-28T06:08:19.787 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:08:19.787 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:08:19.787 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:08:19.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.782+0000 7fdef0c00640 1 -- 172.21.15.96:0/3524567048 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdecc06cad0 msgr2=0x7fdecc06ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:19.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.782+0000 7fdef0c00640 1 --2- 172.21.15.96:0/3524567048 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdecc06cad0 0x7fdecc06ef90 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fdedc006fd0 tx=0x7fdedc008040 comp rx=0 tx=0).stop 2024-05-28T06:08:19.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.782+0000 7fdef0c00640 1 -- 172.21.15.96:0/3524567048 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdeec149d90 msgr2=0x7fdeec14e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:19.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.782+0000 7fdef0c00640 1 --2- 172.21.15.96:0/3524567048 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdeec149d90 0x7fdeec14e5e0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fded80097c0 tx=0x7fded8004730 comp rx=0 tx=0).stop 2024-05-28T06:08:19.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.782+0000 7fdef0c00640 1 -- 172.21.15.96:0/3524567048 shutdown_connections 2024-05-28T06:08:19.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.782+0000 7fdef0c00640 1 --2- 172.21.15.96:0/3524567048 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdecc06cad0 0x7fdecc06ef90 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:19.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.782+0000 7fdef0c00640 1 --2- 172.21.15.96:0/3524567048 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdeec14eb20 0x7fdeec152210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:19.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.782+0000 7fdef0c00640 1 --2- 172.21.15.96:0/3524567048 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdeec149d90 0x7fdeec14e5e0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:19.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.782+0000 7fdef0c00640 1 --2- 172.21.15.96:0/3524567048 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fded0004800 0x7fdeec14af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:19.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.782+0000 7fdef0c00640 1 -- 172.21.15.96:0/3524567048 >> 172.21.15.96:0/3524567048 conn(0x7fdeec067ce0 msgr2=0x7fdeec15ae70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:19.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.782+0000 7fdef0c00640 1 -- 172.21.15.96:0/3524567048 shutdown_connections 2024-05-28T06:08:19.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:19.782+0000 7fdef0c00640 1 -- 172.21.15.96:0/3524567048 wait complete. 2024-05-28T06:08:24.802 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-05-28T06:08:25.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:25.011+0000 7f82b7e00640 1 Processor -- start 2024-05-28T06:08:25.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:25.011+0000 7f82b7e00640 1 -- start start 2024-05-28T06:08:25.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:25.012+0000 7f82b7e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f82b8161df0 0x7f82b8099800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:25.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:25.012+0000 7f82b7e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f82b80d0b70 0x7f82b80cefa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:25.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:25.012+0000 7f82b7e00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82b81621d0 con 0x7f82b80cc2b0 2024-05-28T06:08:25.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:25.012+0000 7f82b7e00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f82b80cd840 con 0x7f82b80d0b70 2024-05-28T06:08:25.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:25.012+0000 7f82b7e00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f82b80ccf10 con 0x7f82b8161df0 2024-05-28T06:08:25.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:25.012+0000 7f82b7400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f82b80d0b70 0x7f82b80cefa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:25.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:25.013+0000 7f82b7400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f82b80d0b70 0x7f82b80cefa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:45426/0 (socket says 172.21.15.96:45426) 2024-05-28T06:08:25.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:25.013+0000 7f82b7400640 1 -- 172.21.15.96:0/813429237 learned_addr learned my addr 172.21.15.96:0/813429237 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:08:25.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:25.013+0000 7f82ae000640 1 --1- 172.21.15.96:0/813429237 >> v1:172.21.15.96:6789/0 conn(0x7f82b80cc2b0 0x7f82b80d0460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:58882/0 (socket says 172.21.15.96:58882) 2024-05-28T06:08:25.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:25.013+0000 7f82b6a00640 1 --2- 172.21.15.96:0/813429237 >> v2:172.21.15.123:3300/0 conn(0x7f82b8161df0 0x7f82b8099800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:25.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:25.013+0000 7f82b7400640 1 -- 172.21.15.96:0/813429237 >> v2:172.21.15.123:3300/0 conn(0x7f82b8161df0 msgr2=0x7f82b8099800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:25.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:25.013+0000 7f82b7400640 1 --2- 172.21.15.96:0/813429237 >> v2:172.21.15.123:3300/0 conn(0x7f82b8161df0 0x7f82b8099800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:25.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:25.013+0000 7f82b7400640 1 -- 172.21.15.96:0/813429237 >> v1:172.21.15.96:6789/0 conn(0x7f82b80cc2b0 legacy=0x7f82b80d0460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:25.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:25.013+0000 7f82b7400640 1 -- 172.21.15.96:0/813429237 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82b80cd1f0 con 0x7f82b80d0b70 2024-05-28T06:08:25.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:25.013+0000 7f82b6000640 1 -- 172.21.15.96:0/813429237 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 817817689 0 0) 0x7f82a0001110 con 0x7f82b80cc2b0 2024-05-28T06:08:25.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:25.014+0000 7f82b7400640 1 --2- 172.21.15.96:0/813429237 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f82b80d0b70 0x7f82b80cefa0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f82b80b3770 tx=0x7f82b002a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=312ed76dafc803e6 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:25.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:25.014+0000 7f82b6000640 1 -- 172.21.15.96:0/813429237 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82b0036070 con 0x7f82b80d0b70 2024-05-28T06:08:25.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:25.014+0000 7f82b6000640 1 -- 172.21.15.96:0/813429237 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82b0031600 con 0x7f82b80d0b70 2024-05-28T06:08:25.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:25.014+0000 7f82b6000640 1 -- 172.21.15.96:0/813429237 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82b00319d0 con 0x7f82b80d0b70 2024-05-28T06:08:25.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:25.015+0000 7f82b7e00640 1 -- 172.21.15.96:0/813429237 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f82b80d0b70 msgr2=0x7f82b80cefa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:25.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:25.015+0000 7f82b7e00640 1 --2- 172.21.15.96:0/813429237 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f82b80d0b70 0x7f82b80cefa0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f82b80b3770 tx=0x7f82b002a730 comp rx=0 tx=0).stop 2024-05-28T06:08:25.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:25.015+0000 7f82b7e00640 1 -- 172.21.15.96:0/813429237 shutdown_connections 2024-05-28T06:08:25.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:25.015+0000 7f82b7e00640 1 --2- 172.21.15.96:0/813429237 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f82b80d0b70 0x7f82b80cefa0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:25.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:25.015+0000 7f82b7e00640 1 --2- 172.21.15.96:0/813429237 >> v2:172.21.15.123:3300/0 conn(0x7f82b8161df0 0x7f82b8099800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:25.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:25.015+0000 7f82b7e00640 1 -- 172.21.15.96:0/813429237 >> 172.21.15.96:0/813429237 conn(0x7f82b806bae0 msgr2=0x7f82b806bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:25.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:25.015+0000 7f82b7e00640 1 -- 172.21.15.96:0/813429237 shutdown_connections 2024-05-28T06:08:25.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:25.015+0000 7f82b7e00640 1 -- 172.21.15.96:0/813429237 wait complete. 2024-05-28T06:08:25.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:25.017+0000 7f82b7e00640 1 Processor -- start 2024-05-28T06:08:25.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:25.017+0000 7f82b7e00640 1 -- start start 2024-05-28T06:08:25.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:25.018+0000 7f82b7e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f82b80cc2b0 0x7f82b81527b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:25.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:25.018+0000 7f82b7e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f82b80d0b70 0x7f82b8160d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:25.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:25.018+0000 7f82b7e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f82b8161df0 0x7f82b80c7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:25.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:25.018+0000 7f82b7e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f82b80ccbc0 con 0x7f82b80cc2b0 2024-05-28T06:08:25.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:25.018+0000 7f82b7e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f82b8099d40 con 0x7f82b8161df0 2024-05-28T06:08:25.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:25.018+0000 7f82b7e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f82b80ca870 con 0x7f82b80d0b70 2024-05-28T06:08:25.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:25.018+0000 7f82b6a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f82b80cc2b0 0x7f82b81527b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:25.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:25.018+0000 7f82b6a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f82b80cc2b0 0x7f82b81527b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:45438/0 (socket says 172.21.15.96:45438) 2024-05-28T06:08:25.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:25.018+0000 7f82b6a00640 1 -- 172.21.15.96:0/3091150411 learned_addr learned my addr 172.21.15.96:0/3091150411 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:08:25.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:25.019+0000 7f82b6a00640 1 -- 172.21.15.96:0/3091150411 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f82b80d0b70 msgr2=0x7f82b8160d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:25.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:25.019+0000 7f82b6a00640 1 --2- 172.21.15.96:0/3091150411 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f82b80d0b70 0x7f82b8160d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:25.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:25.019+0000 7f82b6a00640 1 -- 172.21.15.96:0/3091150411 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f82b8161df0 msgr2=0x7f82b80c7ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:25.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:25.019+0000 7f82b6a00640 1 --2- 172.21.15.96:0/3091150411 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f82b8161df0 0x7f82b80c7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:25.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:25.019+0000 7f82b6a00640 1 -- 172.21.15.96:0/3091150411 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82b815f9d0 con 0x7f82b80cc2b0 2024-05-28T06:08:25.022 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:25.019+0000 7f82b6a00640 1 --2- 172.21.15.96:0/3091150411 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f82b80cc2b0 0x7f82b81527b0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f82a800d5f0 tx=0x7f82a800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:25.022 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:25.019+0000 7f82af400640 1 -- 172.21.15.96:0/3091150411 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82a80125f0 con 0x7f82b80cc2b0 2024-05-28T06:08:25.022 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:25.019+0000 7f82af400640 1 -- 172.21.15.96:0/3091150411 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82a8019070 con 0x7f82b80cc2b0 2024-05-28T06:08:25.022 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:25.019+0000 7f82af400640 1 -- 172.21.15.96:0/3091150411 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82a80129d0 con 0x7f82b80cc2b0 2024-05-28T06:08:25.022 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:25.019+0000 7f82b7e00640 1 -- 172.21.15.96:0/3091150411 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82b815fcb0 con 0x7f82b80cc2b0 2024-05-28T06:08:25.022 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:25.019+0000 7f82b7e00640 1 -- 172.21.15.96:0/3091150411 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82b8160270 con 0x7f82b80cc2b0 2024-05-28T06:08:25.022 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:25.020+0000 7f82b7e00640 1 -- 172.21.15.96:0/3091150411 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82b80aeb90 con 0x7f82b80cc2b0 2024-05-28T06:08:25.022 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:25.021+0000 7f82af400640 1 -- 172.21.15.96:0/3091150411 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 8) ==== 91193+0+0 (secure 0 0 0) 0x7f82a8012b70 con 0x7f82b80cc2b0 2024-05-28T06:08:25.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:25.022+0000 7f82af400640 1 --2- 172.21.15.96:0/3091150411 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f829406caa0 0x7f829406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:25.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:25.022+0000 7f82af400640 1 -- 172.21.15.96:0/3091150411 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f82a808c280 con 0x7f82b80cc2b0 2024-05-28T06:08:25.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:25.023+0000 7f82ae000640 1 --2- 172.21.15.96:0/3091150411 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f829406caa0 0x7f829406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:25.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:25.024+0000 7f82ae000640 1 --2- 172.21.15.96:0/3091150411 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f829406caa0 0x7f829406ef60 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f82b80c8cb0 tx=0x7f82a0001310 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:25.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:25.027+0000 7f82af400640 1 -- 172.21.15.96:0/3091150411 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f82b80aeb90 con 0x7f82b80cc2b0 2024-05-28T06:08:25.297 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:25.295+0000 7f82b7e00640 1 -- 172.21.15.96:0/3091150411 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f82b80689e0 con 0x7f82b80cc2b0 2024-05-28T06:08:25.297 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:25.296+0000 7f82af400640 1 -- 172.21.15.96:0/3091150411 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1984 (secure 0 0 0) 0x7f82a805a420 con 0x7f82b80cc2b0 2024-05-28T06:08:25.298 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:08:25.298 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:08:25.298 INFO:teuthology.orchestra.run.smithi096.stdout: "fsid": "e4a831b5-e6d1-47a7-96cb-bf85154ee0d2", 2024-05-28T06:08:25.298 INFO:teuthology.orchestra.run.smithi096.stdout: "health": { 2024-05-28T06:08:25.298 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "HEALTH_OK", 2024-05-28T06:08:25.298 INFO:teuthology.orchestra.run.smithi096.stdout: "checks": {}, 2024-05-28T06:08:25.299 INFO:teuthology.orchestra.run.smithi096.stdout: "mutes": [] 2024-05-28T06:08:25.299 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:08:25.299 INFO:teuthology.orchestra.run.smithi096.stdout: "election_epoch": 8, 2024-05-28T06:08:25.299 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum": [ 2024-05-28T06:08:25.299 INFO:teuthology.orchestra.run.smithi096.stdout: 0, 2024-05-28T06:08:25.299 INFO:teuthology.orchestra.run.smithi096.stdout: 1, 2024-05-28T06:08:25.299 INFO:teuthology.orchestra.run.smithi096.stdout: 2 2024-05-28T06:08:25.299 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:08:25.299 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_names": [ 2024-05-28T06:08:25.300 INFO:teuthology.orchestra.run.smithi096.stdout: "a", 2024-05-28T06:08:25.300 INFO:teuthology.orchestra.run.smithi096.stdout: "b", 2024-05-28T06:08:25.300 INFO:teuthology.orchestra.run.smithi096.stdout: "c" 2024-05-28T06:08:25.300 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:08:25.300 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_age": 278, 2024-05-28T06:08:25.300 INFO:teuthology.orchestra.run.smithi096.stdout: "monmap": { 2024-05-28T06:08:25.300 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 1, 2024-05-28T06:08:25.300 INFO:teuthology.orchestra.run.smithi096.stdout: "min_mon_release_name": "squid", 2024-05-28T06:08:25.300 INFO:teuthology.orchestra.run.smithi096.stdout: "num_mons": 3 2024-05-28T06:08:25.300 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:08:25.300 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap": { 2024-05-28T06:08:25.300 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 33, 2024-05-28T06:08:25.300 INFO:teuthology.orchestra.run.smithi096.stdout: "num_osds": 8, 2024-05-28T06:08:25.301 INFO:teuthology.orchestra.run.smithi096.stdout: "num_up_osds": 8, 2024-05-28T06:08:25.301 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_up_since": 1716876238, 2024-05-28T06:08:25.301 INFO:teuthology.orchestra.run.smithi096.stdout: "num_in_osds": 8, 2024-05-28T06:08:25.301 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_in_since": 1716876229, 2024-05-28T06:08:25.301 INFO:teuthology.orchestra.run.smithi096.stdout: "num_remapped_pgs": 0 2024-05-28T06:08:25.301 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:08:25.301 INFO:teuthology.orchestra.run.smithi096.stdout: "pgmap": { 2024-05-28T06:08:25.301 INFO:teuthology.orchestra.run.smithi096.stdout: "pgs_by_state": [ 2024-05-28T06:08:25.301 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:08:25.301 INFO:teuthology.orchestra.run.smithi096.stdout: "state_name": "active+clean", 2024-05-28T06:08:25.301 INFO:teuthology.orchestra.run.smithi096.stdout: "count": 97 2024-05-28T06:08:25.301 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:08:25.301 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:08:25.302 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pgs": 97, 2024-05-28T06:08:25.302 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pools": 3, 2024-05-28T06:08:25.302 INFO:teuthology.orchestra.run.smithi096.stdout: "num_objects": 3207, 2024-05-28T06:08:25.302 INFO:teuthology.orchestra.run.smithi096.stdout: "data_bytes": 13330961023, 2024-05-28T06:08:25.302 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_used": 26924654592, 2024-05-28T06:08:25.302 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_avail": 746169458688, 2024-05-28T06:08:25.302 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_total": 773094113280 2024-05-28T06:08:25.302 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:08:25.302 INFO:teuthology.orchestra.run.smithi096.stdout: "fsmap": { 2024-05-28T06:08:25.302 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 16, 2024-05-28T06:08:25.302 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 2, 2024-05-28T06:08:25.302 INFO:teuthology.orchestra.run.smithi096.stdout: "up": 1, 2024-05-28T06:08:25.302 INFO:teuthology.orchestra.run.smithi096.stdout: "in": 1, 2024-05-28T06:08:25.303 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 1, 2024-05-28T06:08:25.303 INFO:teuthology.orchestra.run.smithi096.stdout: "by_rank": [ 2024-05-28T06:08:25.303 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:08:25.303 INFO:teuthology.orchestra.run.smithi096.stdout: "filesystem_id": 2, 2024-05-28T06:08:25.303 INFO:teuthology.orchestra.run.smithi096.stdout: "rank": 0, 2024-05-28T06:08:25.303 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "d", 2024-05-28T06:08:25.303 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "up:active", 2024-05-28T06:08:25.303 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 5238 2024-05-28T06:08:25.303 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:08:25.303 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:08:25.303 INFO:teuthology.orchestra.run.smithi096.stdout: "up:standby": 3 2024-05-28T06:08:25.303 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:08:25.303 INFO:teuthology.orchestra.run.smithi096.stdout: "mgrmap": { 2024-05-28T06:08:25.304 INFO:teuthology.orchestra.run.smithi096.stdout: "available": true, 2024-05-28T06:08:25.304 INFO:teuthology.orchestra.run.smithi096.stdout: "num_standbys": 1, 2024-05-28T06:08:25.304 INFO:teuthology.orchestra.run.smithi096.stdout: "modules": [ 2024-05-28T06:08:25.304 INFO:teuthology.orchestra.run.smithi096.stdout: "iostat", 2024-05-28T06:08:25.304 INFO:teuthology.orchestra.run.smithi096.stdout: "nfs", 2024-05-28T06:08:25.304 INFO:teuthology.orchestra.run.smithi096.stdout: "restful" 2024-05-28T06:08:25.304 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:08:25.304 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:08:25.305 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:08:25.305 INFO:teuthology.orchestra.run.smithi096.stdout: "servicemap": { 2024-05-28T06:08:25.305 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 7, 2024-05-28T06:08:25.305 INFO:teuthology.orchestra.run.smithi096.stdout: "modified": "2024-05-28T06:06:33.535436+0000", 2024-05-28T06:08:25.305 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:08:25.305 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:08:25.305 INFO:teuthology.orchestra.run.smithi096.stdout: "progress_events": { 2024-05-28T06:08:25.305 INFO:teuthology.orchestra.run.smithi096.stdout: "18e91b7c-9da2-4cf3-88f8-e7e2859c593b": { 2024-05-28T06:08:25.305 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "All cloning finished, waiting 4.0 seconds before exiting (31s)\n [============================] ", 2024-05-28T06:08:25.305 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": 1, 2024-05-28T06:08:25.305 INFO:teuthology.orchestra.run.smithi096.stdout: "add_to_ceph_s": true 2024-05-28T06:08:25.305 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:08:25.306 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:08:25.306 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:08:25.307 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:25.298+0000 7f82b7e00640 1 -- 172.21.15.96:0/3091150411 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f829406caa0 msgr2=0x7f829406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:25.307 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:25.299+0000 7f82b7e00640 1 --2- 172.21.15.96:0/3091150411 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f829406caa0 0x7f829406ef60 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f82b80c8cb0 tx=0x7f82a0001310 comp rx=0 tx=0).stop 2024-05-28T06:08:25.307 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:25.299+0000 7f82b7e00640 1 -- 172.21.15.96:0/3091150411 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f82b80cc2b0 msgr2=0x7f82b81527b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:25.307 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:25.299+0000 7f82b7e00640 1 --2- 172.21.15.96:0/3091150411 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f82b80cc2b0 0x7f82b81527b0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f82a800d5f0 tx=0x7f82a800dab0 comp rx=0 tx=0).stop 2024-05-28T06:08:25.307 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:25.299+0000 7f82b7e00640 1 -- 172.21.15.96:0/3091150411 shutdown_connections 2024-05-28T06:08:25.307 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:25.299+0000 7f82b7e00640 1 --2- 172.21.15.96:0/3091150411 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f829406caa0 0x7f829406ef60 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:25.307 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:25.299+0000 7f82b7e00640 1 --2- 172.21.15.96:0/3091150411 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f82b8161df0 0x7f82b80c7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:25.307 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:25.299+0000 7f82b7e00640 1 --2- 172.21.15.96:0/3091150411 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f82b80d0b70 0x7f82b8160d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:25.307 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:25.299+0000 7f82b7e00640 1 --2- 172.21.15.96:0/3091150411 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f82b80cc2b0 0x7f82b81527b0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:25.307 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:25.299+0000 7f82b7e00640 1 -- 172.21.15.96:0/3091150411 >> 172.21.15.96:0/3091150411 conn(0x7f82b806bae0 msgr2=0x7f82b80ad130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:25.308 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:25.299+0000 7f82b7e00640 1 -- 172.21.15.96:0/3091150411 shutdown_connections 2024-05-28T06:08:25.308 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:25.299+0000 7f82b7e00640 1 -- 172.21.15.96:0/3091150411 wait complete. 2024-05-28T06:08:29.791 DEBUG:teuthology.orchestra.run.smithi096:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:08:29.800 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:08:30.321 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-05-28T06:08:30.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:30.524+0000 7fbdf0600640 1 Processor -- start 2024-05-28T06:08:30.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:30.524+0000 7fbdf0600640 1 -- start start 2024-05-28T06:08:30.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:30.526+0000 7fbdf0600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fbde80723d0 0x7fbde80b3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:30.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:30.526+0000 7fbdf0600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbde80b41e0 0x7fbde80cf880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:30.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:30.526+0000 7fbdf0600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbde80cf0c0 con 0x7fbde8145320 2024-05-28T06:08:30.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:30.526+0000 7fbdf0600640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fbde80b4a20 con 0x7fbde80723d0 2024-05-28T06:08:30.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:30.526+0000 7fbdf0600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fbde80b4d00 con 0x7fbde80b41e0 2024-05-28T06:08:30.528 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:30.527+0000 7fbdee600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbde80b41e0 0x7fbde80cf880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:30.529 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:30.527+0000 7fbdedc00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7fbde8145320 0x7fbde8071cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.96:37936/0 (socket says 172.21.15.96:37936) 2024-05-28T06:08:30.529 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:30.527+0000 7fbdedc00640 1 -- 172.21.15.96:0/3071504516 learned_addr learned my addr 172.21.15.96:0/3071504516 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:08:30.529 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:30.527+0000 7fbdee600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbde80b41e0 0x7fbde80cf880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.96:42392/0 (socket says 172.21.15.96:42392) 2024-05-28T06:08:30.529 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:30.527+0000 7fbded200640 1 --2- 172.21.15.96:0/3071504516 >> v2:172.21.15.123:3300/0 conn(0x7fbde80723d0 0x7fbde80b3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:30.529 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:30.528+0000 7fbdee600640 1 -- 172.21.15.96:0/3071504516 >> v2:172.21.15.123:3300/0 conn(0x7fbde80723d0 msgr2=0x7fbde80b3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:30.529 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:30.528+0000 7fbdee600640 1 --2- 172.21.15.96:0/3071504516 >> v2:172.21.15.123:3300/0 conn(0x7fbde80723d0 0x7fbde80b3ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:30.530 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:30.528+0000 7fbdee600640 1 -- 172.21.15.96:0/3071504516 >> v1:172.21.15.123:6789/0 conn(0x7fbde8145320 legacy=0x7fbde8071cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:30.530 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:30.528+0000 7fbdee600640 1 -- 172.21.15.96:0/3071504516 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbde80b4fe0 con 0x7fbde80b41e0 2024-05-28T06:08:30.530 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:30.528+0000 7fbde7e00640 1 -- 172.21.15.96:0/3071504516 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 508794328 0 0) 0x7fbde80cf0c0 con 0x7fbde8145320 2024-05-28T06:08:30.530 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:30.529+0000 7fbdee600640 1 --2- 172.21.15.96:0/3071504516 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbde80b41e0 0x7fbde80cf880 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fbde0009d90 tx=0x7fbde002a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f671a9b566cabe8 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:30.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:30.529+0000 7fbde7e00640 1 -- 172.21.15.96:0/3071504516 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbde0007740 con 0x7fbde80b41e0 2024-05-28T06:08:30.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:30.529+0000 7fbde7e00640 1 -- 172.21.15.96:0/3071504516 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbde00078e0 con 0x7fbde80b41e0 2024-05-28T06:08:30.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:30.529+0000 7fbde7e00640 1 -- 172.21.15.96:0/3071504516 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbde0007cb0 con 0x7fbde80b41e0 2024-05-28T06:08:30.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:30.530+0000 7fbdf0600640 1 -- 172.21.15.96:0/3071504516 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbde80b41e0 msgr2=0x7fbde80cf880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:30.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:30.530+0000 7fbdf0600640 1 --2- 172.21.15.96:0/3071504516 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbde80b41e0 0x7fbde80cf880 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fbde0009d90 tx=0x7fbde002a660 comp rx=0 tx=0).stop 2024-05-28T06:08:30.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:30.530+0000 7fbdf0600640 1 -- 172.21.15.96:0/3071504516 shutdown_connections 2024-05-28T06:08:30.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:30.530+0000 7fbdf0600640 1 --2- 172.21.15.96:0/3071504516 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbde80b41e0 0x7fbde80cf880 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:30.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:30.530+0000 7fbdf0600640 1 --2- 172.21.15.96:0/3071504516 >> v2:172.21.15.123:3300/0 conn(0x7fbde80723d0 0x7fbde80b3ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:30.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:30.530+0000 7fbdf0600640 1 -- 172.21.15.96:0/3071504516 >> 172.21.15.96:0/3071504516 conn(0x7fbde8067ce0 msgr2=0x7fbde806b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:30.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:30.530+0000 7fbdf0600640 1 -- 172.21.15.96:0/3071504516 shutdown_connections 2024-05-28T06:08:30.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:30.530+0000 7fbdf0600640 1 -- 172.21.15.96:0/3071504516 wait complete. 2024-05-28T06:08:30.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:30.532+0000 7fbdf0600640 1 Processor -- start 2024-05-28T06:08:30.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:30.533+0000 7fbdf0600640 1 -- start start 2024-05-28T06:08:30.536 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:30.534+0000 7fbdf0600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbde80723d0 0x7fbde8154f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:30.536 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:30.534+0000 7fbdf0600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbde80b41e0 0x7fbde815ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:30.536 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:30.534+0000 7fbdf0600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbde8145320 0x7fbde815f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:30.536 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:30.534+0000 7fbdf0600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fbde80b4650 con 0x7fbde80723d0 2024-05-28T06:08:30.536 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:30.534+0000 7fbdf0600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fbde8145700 con 0x7fbde80b41e0 2024-05-28T06:08:30.536 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:30.534+0000 7fbdf0600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fbde80b47d0 con 0x7fbde8145320 2024-05-28T06:08:30.536 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:30.534+0000 7fbdedc00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbde80723d0 0x7fbde8154f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:30.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:30.534+0000 7fbdedc00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbde80723d0 0x7fbde8154f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35724/0 (socket says 172.21.15.96:35724) 2024-05-28T06:08:30.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:30.534+0000 7fbdedc00640 1 -- 172.21.15.96:0/314482528 learned_addr learned my addr 172.21.15.96:0/314482528 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:08:30.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:30.535+0000 7fbdee600640 1 --2- 172.21.15.96:0/314482528 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbde8145320 0x7fbde815f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:30.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:30.535+0000 7fbded200640 1 --2- 172.21.15.96:0/314482528 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbde80b41e0 0x7fbde815ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:30.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:30.535+0000 7fbdedc00640 1 -- 172.21.15.96:0/314482528 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbde8145320 msgr2=0x7fbde815f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:30.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:30.535+0000 7fbdedc00640 1 --2- 172.21.15.96:0/314482528 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbde8145320 0x7fbde815f210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:30.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:30.535+0000 7fbdedc00640 1 -- 172.21.15.96:0/314482528 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbde80b41e0 msgr2=0x7fbde815ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:30.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:30.535+0000 7fbdedc00640 1 --2- 172.21.15.96:0/314482528 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbde80b41e0 0x7fbde815ecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:30.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:30.535+0000 7fbdedc00640 1 -- 172.21.15.96:0/314482528 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbde80afd60 con 0x7fbde80723d0 2024-05-28T06:08:30.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:30.535+0000 7fbdedc00640 1 --2- 172.21.15.96:0/314482528 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbde80723d0 0x7fbde8154f20 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fbddc00c460 tx=0x7fbddc00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:30.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:30.535+0000 7fbde5600640 1 -- 172.21.15.96:0/314482528 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbddc018070 con 0x7fbde80723d0 2024-05-28T06:08:30.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:30.535+0000 7fbde5600640 1 -- 172.21.15.96:0/314482528 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbddc007500 con 0x7fbde80723d0 2024-05-28T06:08:30.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:30.535+0000 7fbde5600640 1 -- 172.21.15.96:0/314482528 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbddc004a20 con 0x7fbde80723d0 2024-05-28T06:08:30.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:30.535+0000 7fbdf0600640 1 -- 172.21.15.96:0/314482528 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbde80b0040 con 0x7fbde80723d0 2024-05-28T06:08:30.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:30.535+0000 7fbdf0600640 1 -- 172.21.15.96:0/314482528 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbde80b0600 con 0x7fbde80723d0 2024-05-28T06:08:30.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:30.537+0000 7fbdf0600640 1 -- 172.21.15.96:0/314482528 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbde80689e0 con 0x7fbde80723d0 2024-05-28T06:08:30.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:30.538+0000 7fbde5600640 1 -- 172.21.15.96:0/314482528 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 8) ==== 91193+0+0 (secure 0 0 0) 0x7fbddc004bc0 con 0x7fbde80723d0 2024-05-28T06:08:30.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:30.538+0000 7fbde5600640 1 --2- 172.21.15.96:0/314482528 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbdc806cad0 0x7fbdc806ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:30.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:30.539+0000 7fbde5600640 1 -- 172.21.15.96:0/314482528 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7fbddc08b1d0 con 0x7fbde80723d0 2024-05-28T06:08:30.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:30.539+0000 7fbded200640 1 --2- 172.21.15.96:0/314482528 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbdc806cad0 0x7fbdc806ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:30.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:30.540+0000 7fbded200640 1 --2- 172.21.15.96:0/314482528 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbdc806cad0 0x7fbdc806ef90 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fbde8144830 tx=0x7fbdd4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:30.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:30.545+0000 7fbde5600640 1 -- 172.21.15.96:0/314482528 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbddc0592e0 con 0x7fbde80723d0 2024-05-28T06:08:30.818 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:30.817+0000 7fbdf0600640 1 -- 172.21.15.96:0/314482528 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fbde80b34e0 con 0x7fbde80723d0 2024-05-28T06:08:30.819 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:30.818+0000 7fbde5600640 1 -- 172.21.15.96:0/314482528 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1730 (secure 0 0 0) 0x7fbddc058be0 con 0x7fbde80723d0 2024-05-28T06:08:30.820 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:08:30.820 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:08:30.820 INFO:teuthology.orchestra.run.smithi096.stdout: "fsid": "e4a831b5-e6d1-47a7-96cb-bf85154ee0d2", 2024-05-28T06:08:30.820 INFO:teuthology.orchestra.run.smithi096.stdout: "health": { 2024-05-28T06:08:30.820 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "HEALTH_OK", 2024-05-28T06:08:30.820 INFO:teuthology.orchestra.run.smithi096.stdout: "checks": {}, 2024-05-28T06:08:30.820 INFO:teuthology.orchestra.run.smithi096.stdout: "mutes": [] 2024-05-28T06:08:30.820 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:08:30.820 INFO:teuthology.orchestra.run.smithi096.stdout: "election_epoch": 8, 2024-05-28T06:08:30.820 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum": [ 2024-05-28T06:08:30.820 INFO:teuthology.orchestra.run.smithi096.stdout: 0, 2024-05-28T06:08:30.821 INFO:teuthology.orchestra.run.smithi096.stdout: 1, 2024-05-28T06:08:30.821 INFO:teuthology.orchestra.run.smithi096.stdout: 2 2024-05-28T06:08:30.821 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:08:30.821 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_names": [ 2024-05-28T06:08:30.821 INFO:teuthology.orchestra.run.smithi096.stdout: "a", 2024-05-28T06:08:30.821 INFO:teuthology.orchestra.run.smithi096.stdout: "b", 2024-05-28T06:08:30.821 INFO:teuthology.orchestra.run.smithi096.stdout: "c" 2024-05-28T06:08:30.821 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:08:30.821 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_age": 284, 2024-05-28T06:08:30.821 INFO:teuthology.orchestra.run.smithi096.stdout: "monmap": { 2024-05-28T06:08:30.821 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 1, 2024-05-28T06:08:30.821 INFO:teuthology.orchestra.run.smithi096.stdout: "min_mon_release_name": "squid", 2024-05-28T06:08:30.821 INFO:teuthology.orchestra.run.smithi096.stdout: "num_mons": 3 2024-05-28T06:08:30.821 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:08:30.822 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap": { 2024-05-28T06:08:30.822 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 33, 2024-05-28T06:08:30.822 INFO:teuthology.orchestra.run.smithi096.stdout: "num_osds": 8, 2024-05-28T06:08:30.822 INFO:teuthology.orchestra.run.smithi096.stdout: "num_up_osds": 8, 2024-05-28T06:08:30.822 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_up_since": 1716876238, 2024-05-28T06:08:30.822 INFO:teuthology.orchestra.run.smithi096.stdout: "num_in_osds": 8, 2024-05-28T06:08:30.822 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_in_since": 1716876229, 2024-05-28T06:08:30.822 INFO:teuthology.orchestra.run.smithi096.stdout: "num_remapped_pgs": 0 2024-05-28T06:08:30.822 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:08:30.822 INFO:teuthology.orchestra.run.smithi096.stdout: "pgmap": { 2024-05-28T06:08:30.822 INFO:teuthology.orchestra.run.smithi096.stdout: "pgs_by_state": [ 2024-05-28T06:08:30.822 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:08:30.822 INFO:teuthology.orchestra.run.smithi096.stdout: "state_name": "active+clean", 2024-05-28T06:08:30.823 INFO:teuthology.orchestra.run.smithi096.stdout: "count": 97 2024-05-28T06:08:30.823 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:08:30.823 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:08:30.823 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pgs": 97, 2024-05-28T06:08:30.823 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pools": 3, 2024-05-28T06:08:30.823 INFO:teuthology.orchestra.run.smithi096.stdout: "num_objects": 3207, 2024-05-28T06:08:30.823 INFO:teuthology.orchestra.run.smithi096.stdout: "data_bytes": 13330961023, 2024-05-28T06:08:30.823 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_used": 26924654592, 2024-05-28T06:08:30.823 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_avail": 746169458688, 2024-05-28T06:08:30.823 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_total": 773094113280 2024-05-28T06:08:30.823 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:08:30.823 INFO:teuthology.orchestra.run.smithi096.stdout: "fsmap": { 2024-05-28T06:08:30.823 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 16, 2024-05-28T06:08:30.823 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 2, 2024-05-28T06:08:30.824 INFO:teuthology.orchestra.run.smithi096.stdout: "up": 1, 2024-05-28T06:08:30.824 INFO:teuthology.orchestra.run.smithi096.stdout: "in": 1, 2024-05-28T06:08:30.824 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 1, 2024-05-28T06:08:30.824 INFO:teuthology.orchestra.run.smithi096.stdout: "by_rank": [ 2024-05-28T06:08:30.824 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:08:30.824 INFO:teuthology.orchestra.run.smithi096.stdout: "filesystem_id": 2, 2024-05-28T06:08:30.824 INFO:teuthology.orchestra.run.smithi096.stdout: "rank": 0, 2024-05-28T06:08:30.824 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "d", 2024-05-28T06:08:30.824 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "up:active", 2024-05-28T06:08:30.824 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 5238 2024-05-28T06:08:30.824 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:08:30.824 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:08:30.824 INFO:teuthology.orchestra.run.smithi096.stdout: "up:standby": 3 2024-05-28T06:08:30.824 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:08:30.825 INFO:teuthology.orchestra.run.smithi096.stdout: "mgrmap": { 2024-05-28T06:08:30.825 INFO:teuthology.orchestra.run.smithi096.stdout: "available": true, 2024-05-28T06:08:30.825 INFO:teuthology.orchestra.run.smithi096.stdout: "num_standbys": 1, 2024-05-28T06:08:30.825 INFO:teuthology.orchestra.run.smithi096.stdout: "modules": [ 2024-05-28T06:08:30.825 INFO:teuthology.orchestra.run.smithi096.stdout: "iostat", 2024-05-28T06:08:30.825 INFO:teuthology.orchestra.run.smithi096.stdout: "nfs", 2024-05-28T06:08:30.825 INFO:teuthology.orchestra.run.smithi096.stdout: "restful" 2024-05-28T06:08:30.825 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:08:30.825 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:08:30.825 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:08:30.825 INFO:teuthology.orchestra.run.smithi096.stdout: "servicemap": { 2024-05-28T06:08:30.825 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 7, 2024-05-28T06:08:30.825 INFO:teuthology.orchestra.run.smithi096.stdout: "modified": "2024-05-28T06:06:33.535436+0000", 2024-05-28T06:08:30.825 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:08:30.825 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:08:30.826 INFO:teuthology.orchestra.run.smithi096.stdout: "progress_events": {} 2024-05-28T06:08:30.826 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:08:30.826 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:30.820+0000 7fbdf0600640 1 -- 172.21.15.96:0/314482528 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbdc806cad0 msgr2=0x7fbdc806ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:30.826 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:30.820+0000 7fbdf0600640 1 --2- 172.21.15.96:0/314482528 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbdc806cad0 0x7fbdc806ef90 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fbde8144830 tx=0x7fbdd4008040 comp rx=0 tx=0).stop 2024-05-28T06:08:30.827 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:30.820+0000 7fbdf0600640 1 -- 172.21.15.96:0/314482528 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbde80723d0 msgr2=0x7fbde8154f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:30.827 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:30.820+0000 7fbdf0600640 1 --2- 172.21.15.96:0/314482528 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbde80723d0 0x7fbde8154f20 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fbddc00c460 tx=0x7fbddc00c920 comp rx=0 tx=0).stop 2024-05-28T06:08:30.827 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:30.820+0000 7fbdf0600640 1 -- 172.21.15.96:0/314482528 shutdown_connections 2024-05-28T06:08:30.827 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:30.820+0000 7fbdf0600640 1 --2- 172.21.15.96:0/314482528 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbdc806cad0 0x7fbdc806ef90 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:30.827 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:30.820+0000 7fbdf0600640 1 --2- 172.21.15.96:0/314482528 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbde8145320 0x7fbde815f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:30.827 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:30.820+0000 7fbdf0600640 1 --2- 172.21.15.96:0/314482528 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbde80b41e0 0x7fbde815ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:30.827 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:30.820+0000 7fbdf0600640 1 --2- 172.21.15.96:0/314482528 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbde80723d0 0x7fbde8154f20 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:30.827 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:30.820+0000 7fbdf0600640 1 -- 172.21.15.96:0/314482528 >> 172.21.15.96:0/314482528 conn(0x7fbde8067ce0 msgr2=0x7fbde81431a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:30.827 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:30.821+0000 7fbdf0600640 1 -- 172.21.15.96:0/314482528 shutdown_connections 2024-05-28T06:08:30.827 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:30.821+0000 7fbdf0600640 1 -- 172.21.15.96:0/314482528 wait complete. 2024-05-28T06:08:30.841 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume ls cephfs --format json 2024-05-28T06:08:31.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.041+0000 7f424f000640 1 Processor -- start 2024-05-28T06:08:31.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.041+0000 7f424f000640 1 -- start start 2024-05-28T06:08:31.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.042+0000 7f424f000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f42480ce1f0 0x7f42480b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:31.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.043+0000 7f424f000640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42480b3580 con 0x7f42480ce5d0 2024-05-28T06:08:31.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.043+0000 7f424f000640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f424815f490 con 0x7f42480b3d60 2024-05-28T06:08:31.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.043+0000 7f424f000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f424815f770 con 0x7f42480ce1f0 2024-05-28T06:08:31.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.043+0000 7f424d200640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f42480ce5d0 0x7f424815ec40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:42556/0 (socket says 172.21.15.96:42556) 2024-05-28T06:08:31.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.043+0000 7f424d200640 1 -- 172.21.15.96:0/499632982 learned_addr learned my addr 172.21.15.96:0/499632982 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:08:31.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.043+0000 7f4247e00640 1 -- 172.21.15.96:0/499632982 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2769452414 0 0) 0x7f42480b3580 con 0x7f42480ce5d0 2024-05-28T06:08:31.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.043+0000 7f424dc00640 1 --2- 172.21.15.96:0/499632982 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f42480ce1f0 0x7f42480b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:31.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.043+0000 7f4247e00640 1 -- 172.21.15.96:0/499632982 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4230003520 con 0x7f42480ce5d0 2024-05-28T06:08:31.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.044+0000 7f4247e00640 1 -- 172.21.15.96:0/499632982 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 952822606 0 0) 0x7f4230003520 con 0x7f42480ce5d0 2024-05-28T06:08:31.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.044+0000 7f4247e00640 1 -- 172.21.15.96:0/499632982 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f42480ce1f0 msgr2=0x7f42480b4dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:31.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.044+0000 7f4247e00640 1 --2- 172.21.15.96:0/499632982 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f42480ce1f0 0x7f42480b4dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:31.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.044+0000 7f4247e00640 1 -- 172.21.15.96:0/499632982 >> v1:172.21.15.123:6789/0 conn(0x7f42480b3d60 legacy=0x7f42480ccb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:31.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.044+0000 7f4247e00640 1 -- 172.21.15.96:0/499632982 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f424815fa50 con 0x7f42480ce5d0 2024-05-28T06:08:31.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.044+0000 7f4247e00640 1 -- 172.21.15.96:0/499632982 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f423c001670 con 0x7f42480ce5d0 2024-05-28T06:08:31.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.044+0000 7f4247e00640 1 -- 172.21.15.96:0/499632982 >> v1:172.21.15.96:6789/0 conn(0x7f42480ce5d0 legacy=0x7f424815ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:31.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.044+0000 7f4247e00640 1 --2- 172.21.15.96:0/499632982 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f42300039e0 0x7f4230003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:31.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.044+0000 7f4247e00640 1 --2- 172.21.15.96:0/499632982 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4230004800 0x7f4230004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:31.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.044+0000 7f4247e00640 1 --2- 172.21.15.96:0/499632982 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f42480ce1f0 0x7f423000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:31.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.044+0000 7f4247e00640 1 -- 172.21.15.96:0/499632982 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f424815f770 con 0x7f42300039e0 2024-05-28T06:08:31.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.044+0000 7f4247e00640 1 -- 172.21.15.96:0/499632982 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f4230005520 con 0x7f4230004800 2024-05-28T06:08:31.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.044+0000 7f4247e00640 1 -- 172.21.15.96:0/499632982 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f423000bf00 con 0x7f42480ce1f0 2024-05-28T06:08:31.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.044+0000 7f424dc00640 1 --2- 172.21.15.96:0/499632982 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f42300039e0 0x7f4230003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:31.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.044+0000 7f424dc00640 1 -- 172.21.15.96:0/499632982 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f42480ce1f0 msgr2=0x7f423000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:31.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.044+0000 7f424dc00640 1 --2- 172.21.15.96:0/499632982 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f42480ce1f0 0x7f423000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:31.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.044+0000 7f424dc00640 1 -- 172.21.15.96:0/499632982 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4230004800 msgr2=0x7f4230004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:31.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.044+0000 7f424dc00640 1 --2- 172.21.15.96:0/499632982 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4230004800 0x7f4230004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:31.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.044+0000 7f424dc00640 1 -- 172.21.15.96:0/499632982 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4230003520 con 0x7f42300039e0 2024-05-28T06:08:31.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.045+0000 7f424dc00640 1 --2- 172.21.15.96:0/499632982 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f42300039e0 0x7f4230003e00 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f4238009ac0 tx=0x7f4230005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=daa54529462dab43 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:31.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.046+0000 7f4247e00640 1 -- 172.21.15.96:0/499632982 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4238036070 con 0x7f42300039e0 2024-05-28T06:08:31.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.046+0000 7f4247e00640 1 -- 172.21.15.96:0/499632982 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4238031400 con 0x7f42300039e0 2024-05-28T06:08:31.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.046+0000 7f4247e00640 1 -- 172.21.15.96:0/499632982 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4238009690 con 0x7f42300039e0 2024-05-28T06:08:31.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.046+0000 7f424f000640 1 -- 172.21.15.96:0/499632982 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f42300039e0 msgr2=0x7f4230003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:31.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.046+0000 7f424f000640 1 --2- 172.21.15.96:0/499632982 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f42300039e0 0x7f4230003e00 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f4238009ac0 tx=0x7f4230005830 comp rx=0 tx=0).stop 2024-05-28T06:08:31.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.046+0000 7f424e600640 1 -- 172.21.15.96:0/499632982 reap_dead start 2024-05-28T06:08:31.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.047+0000 7f424f000640 1 -- 172.21.15.96:0/499632982 shutdown_connections 2024-05-28T06:08:31.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.047+0000 7f424f000640 1 -- 172.21.15.96:0/499632982 >> 172.21.15.96:0/499632982 conn(0x7f424806b900 msgr2=0x7f424806bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:31.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.047+0000 7f424f000640 1 -- 172.21.15.96:0/499632982 shutdown_connections 2024-05-28T06:08:31.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.047+0000 7f424f000640 1 -- 172.21.15.96:0/499632982 wait complete. 2024-05-28T06:08:31.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.049+0000 7f424f000640 1 Processor -- start 2024-05-28T06:08:31.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.049+0000 7f424f000640 1 -- start start 2024-05-28T06:08:31.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.050+0000 7f424f000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f42300039e0 0x7f4248144de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:31.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.050+0000 7f424f000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f42480999f0 0x7f4248160630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:31.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.051+0000 7f424f000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4248160b70 0x7f42480ae180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:31.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.051+0000 7f424f000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f42480cd420 con 0x7f42480999f0 2024-05-28T06:08:31.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.051+0000 7f424f000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f42480cd2a0 con 0x7f4248160b70 2024-05-28T06:08:31.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.051+0000 7f424f000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f42480b4dd0 con 0x7f42300039e0 2024-05-28T06:08:31.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.051+0000 7f424d200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f42480999f0 0x7f4248160630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:31.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.051+0000 7f424d200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f42480999f0 0x7f4248160630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35752/0 (socket says 172.21.15.96:35752) 2024-05-28T06:08:31.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.051+0000 7f424d200640 1 -- 172.21.15.96:0/1787059861 learned_addr learned my addr 172.21.15.96:0/1787059861 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:08:31.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.051+0000 7f424dc00640 1 --2- 172.21.15.96:0/1787059861 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f42300039e0 0x7f4248144de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:31.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.051+0000 7f424d200640 1 -- 172.21.15.96:0/1787059861 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f42300039e0 msgr2=0x7f4248144de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:31.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.051+0000 7f424d200640 1 --2- 172.21.15.96:0/1787059861 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f42300039e0 0x7f4248144de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:31.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.051+0000 7f424d200640 1 -- 172.21.15.96:0/1787059861 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4248160b70 msgr2=0x7f42480ae180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:31.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.051+0000 7f424d200640 1 --2- 172.21.15.96:0/1787059861 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4248160b70 0x7f42480ae180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:31.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.051+0000 7f424d200640 1 -- 172.21.15.96:0/1787059861 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42480b0c80 con 0x7f42480999f0 2024-05-28T06:08:31.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.052+0000 7f424d200640 1 --2- 172.21.15.96:0/1787059861 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f42480999f0 0x7f4248160630 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f423c00c700 tx=0x7f423c00cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:31.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.052+0000 7f4245600640 1 -- 172.21.15.96:0/1787059861 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f423c018070 con 0x7f42480999f0 2024-05-28T06:08:31.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.052+0000 7f4245600640 1 -- 172.21.15.96:0/1787059861 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f423c0074e0 con 0x7f42480999f0 2024-05-28T06:08:31.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.052+0000 7f4245600640 1 -- 172.21.15.96:0/1787059861 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f423c004a00 con 0x7f42480999f0 2024-05-28T06:08:31.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.052+0000 7f424f000640 1 -- 172.21.15.96:0/1787059861 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42480b0f60 con 0x7f42480999f0 2024-05-28T06:08:31.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.052+0000 7f424f000640 1 -- 172.21.15.96:0/1787059861 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42480b1520 con 0x7f42480999f0 2024-05-28T06:08:31.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.053+0000 7f424f000640 1 -- 172.21.15.96:0/1787059861 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42480689e0 con 0x7f42480999f0 2024-05-28T06:08:31.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.054+0000 7f4245600640 1 -- 172.21.15.96:0/1787059861 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 8) ==== 91193+0+0 (secure 0 0 0) 0x7f423c01b070 con 0x7f42480999f0 2024-05-28T06:08:31.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.055+0000 7f4245600640 1 --2- 172.21.15.96:0/1787059861 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f422c06caa0 0x7f422c06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:31.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.055+0000 7f4245600640 1 -- 172.21.15.96:0/1787059861 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f423c08b430 con 0x7f42480999f0 2024-05-28T06:08:31.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.056+0000 7f424dc00640 1 --2- 172.21.15.96:0/1787059861 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f422c06caa0 0x7f422c06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:31.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.057+0000 7f424dc00640 1 --2- 172.21.15.96:0/1787059861 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f422c06caa0 0x7f422c06ef60 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f4238033000 tx=0x7f4238008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:31.061 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.060+0000 7f4245600640 1 -- 172.21.15.96:0/1787059861 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f423c059540 con 0x7f42480999f0 2024-05-28T06:08:31.296 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.294+0000 7f424f000640 1 -- 172.21.15.96:0/1787059861 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume ls", "vol_name": "cephfs", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f42480ca070 con 0x7f422c06caa0 2024-05-28T06:08:31.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.311+0000 7f4245600640 1 -- 172.21.15.96:0/1787059861 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+79 (secure 0 0 0) 0x7f42480ca070 con 0x7f422c06caa0 2024-05-28T06:08:31.312 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:08:31.312 INFO:teuthology.orchestra.run.smithi096.stdout:[ 2024-05-28T06:08:31.312 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:08:31.312 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "sv1" 2024-05-28T06:08:31.312 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:08:31.312 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:08:31.312 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "ss1clone1" 2024-05-28T06:08:31.312 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:08:31.312 INFO:teuthology.orchestra.run.smithi096.stdout:] 2024-05-28T06:08:31.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.312+0000 7f424f000640 1 -- 172.21.15.96:0/1787059861 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f422c06caa0 msgr2=0x7f422c06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:31.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.312+0000 7f424f000640 1 --2- 172.21.15.96:0/1787059861 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f422c06caa0 0x7f422c06ef60 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f4238033000 tx=0x7f4238008030 comp rx=0 tx=0).stop 2024-05-28T06:08:31.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.312+0000 7f424f000640 1 -- 172.21.15.96:0/1787059861 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f42480999f0 msgr2=0x7f4248160630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:31.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.312+0000 7f424f000640 1 --2- 172.21.15.96:0/1787059861 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f42480999f0 0x7f4248160630 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f423c00c700 tx=0x7f423c00cbc0 comp rx=0 tx=0).stop 2024-05-28T06:08:31.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.313+0000 7f424f000640 1 -- 172.21.15.96:0/1787059861 shutdown_connections 2024-05-28T06:08:31.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.313+0000 7f424f000640 1 --2- 172.21.15.96:0/1787059861 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f422c06caa0 0x7f422c06ef60 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:31.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.313+0000 7f424f000640 1 --2- 172.21.15.96:0/1787059861 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4248160b70 0x7f42480ae180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:31.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.313+0000 7f424f000640 1 --2- 172.21.15.96:0/1787059861 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f42480999f0 0x7f4248160630 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:31.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.313+0000 7f424f000640 1 --2- 172.21.15.96:0/1787059861 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f42300039e0 0x7f4248144de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:31.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.313+0000 7f424f000640 1 -- 172.21.15.96:0/1787059861 >> 172.21.15.96:0/1787059861 conn(0x7f424806b900 msgr2=0x7f424806e580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:31.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.313+0000 7f424f000640 1 -- 172.21.15.96:0/1787059861 shutdown_connections 2024-05-28T06:08:31.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.313+0000 7f424f000640 1 -- 172.21.15.96:0/1787059861 wait complete. 2024-05-28T06:08:31.333 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot ls cephfs sv1 --format json 2024-05-28T06:08:31.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.530+0000 7f39f7e00640 1 Processor -- start 2024-05-28T06:08:31.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.530+0000 7f39f7e00640 1 -- start start 2024-05-28T06:08:31.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.531+0000 7f39f7e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f39f80ce560 0x7f39f80b4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:31.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.532+0000 7f39f7e00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39f815bb30 con 0x7f39f80d0430 2024-05-28T06:08:31.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.532+0000 7f39f7e00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39f80b3ba0 con 0x7f39f80d0860 2024-05-28T06:08:31.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.532+0000 7f39f7e00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f39f815bdd0 con 0x7f39f80ce560 2024-05-28T06:08:31.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.532+0000 7f39f6a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f39f80ce560 0x7f39f80b4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:31.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.532+0000 7f39f6a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f39f80ce560 0x7f39f80b4b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35768/0 (socket says 172.21.15.96:35768) 2024-05-28T06:08:31.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.532+0000 7f39f6a00640 1 -- 172.21.15.96:0/1879442899 learned_addr learned my addr 172.21.15.96:0/1879442899 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:08:31.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.532+0000 7f39f6000640 1 --1- 172.21.15.96:0/1879442899 >> v1:172.21.15.96:6789/0 conn(0x7f39f80d0430 0x7f39f815f0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:42568/0 (socket says 172.21.15.96:42568) 2024-05-28T06:08:31.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.532+0000 7f39f6a00640 1 -- 172.21.15.96:0/1879442899 >> v1:172.21.15.123:6789/0 conn(0x7f39f80d0860 legacy=0x7f39f815f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:31.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.532+0000 7f39f6a00640 1 -- 172.21.15.96:0/1879442899 >> v1:172.21.15.96:6789/0 conn(0x7f39f80d0430 legacy=0x7f39f815f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:31.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.532+0000 7f39f6a00640 1 -- 172.21.15.96:0/1879442899 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39f815c0b0 con 0x7f39f80ce560 2024-05-28T06:08:31.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.532+0000 7f39f5600640 1 -- 172.21.15.96:0/1879442899 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3377228423 0 0) 0x7f39f815bb30 con 0x7f39f80d0430 2024-05-28T06:08:31.536 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.533+0000 7f39f6a00640 1 --2- 172.21.15.96:0/1879442899 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f39f80ce560 0x7f39f80b4b80 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f39f815ff70 tx=0x7f39e802a290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=269274ba9811d846 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:31.536 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.534+0000 7f39f5600640 1 -- 172.21.15.96:0/1879442899 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39e8037070 con 0x7f39f80ce560 2024-05-28T06:08:31.536 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.534+0000 7f39f5600640 1 -- 172.21.15.96:0/1879442899 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39e8032440 con 0x7f39f80ce560 2024-05-28T06:08:31.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.534+0000 7f39f5600640 1 -- 172.21.15.96:0/1879442899 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39e8032810 con 0x7f39f80ce560 2024-05-28T06:08:31.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.534+0000 7f39f7e00640 1 -- 172.21.15.96:0/1879442899 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f39f80ce560 msgr2=0x7f39f80b4b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:31.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.534+0000 7f39f7e00640 1 --2- 172.21.15.96:0/1879442899 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f39f80ce560 0x7f39f80b4b80 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f39f815ff70 tx=0x7f39e802a290 comp rx=0 tx=0).stop 2024-05-28T06:08:31.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.534+0000 7f39f7e00640 1 -- 172.21.15.96:0/1879442899 shutdown_connections 2024-05-28T06:08:31.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.534+0000 7f39f7e00640 1 --2- 172.21.15.96:0/1879442899 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f39f80ce560 0x7f39f80b4b80 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:31.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.534+0000 7f39f7e00640 1 -- 172.21.15.96:0/1879442899 >> 172.21.15.96:0/1879442899 conn(0x7f39f806bae0 msgr2=0x7f39f806bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:31.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.535+0000 7f39f7e00640 1 -- 172.21.15.96:0/1879442899 shutdown_connections 2024-05-28T06:08:31.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.535+0000 7f39f7e00640 1 -- 172.21.15.96:0/1879442899 wait complete. 2024-05-28T06:08:31.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.536+0000 7f39f7e00640 1 Processor -- start 2024-05-28T06:08:31.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.536+0000 7f39f7e00640 1 -- start start 2024-05-28T06:08:31.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.537+0000 7f39f7e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f39f80ce560 0x7f39f814a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:31.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.537+0000 7f39f7e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f39f80d0430 0x7f39f80b1a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:31.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.537+0000 7f39f7e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f39f80d0860 0x7f39f80ae380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:31.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.537+0000 7f39f7e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f39f80d0cd0 con 0x7f39f80ce560 2024-05-28T06:08:31.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.537+0000 7f39f7e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f39f815c670 con 0x7f39f80d0860 2024-05-28T06:08:31.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.537+0000 7f39f7e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f39f806f630 con 0x7f39f80d0430 2024-05-28T06:08:31.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.538+0000 7f39f6a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f39f80ce560 0x7f39f814a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:31.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.538+0000 7f39f6a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f39f80ce560 0x7f39f814a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35776/0 (socket says 172.21.15.96:35776) 2024-05-28T06:08:31.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.538+0000 7f39f6a00640 1 -- 172.21.15.96:0/2821168533 learned_addr learned my addr 172.21.15.96:0/2821168533 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:08:31.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.538+0000 7f39f6a00640 1 -- 172.21.15.96:0/2821168533 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f39f80d0430 msgr2=0x7f39f80b1a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:31.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.538+0000 7f39f6a00640 1 --2- 172.21.15.96:0/2821168533 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f39f80d0430 0x7f39f80b1a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:31.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.538+0000 7f39f6a00640 1 -- 172.21.15.96:0/2821168533 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f39f80d0860 msgr2=0x7f39f80ae380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:31.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.538+0000 7f39f6a00640 1 --2- 172.21.15.96:0/2821168533 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f39f80d0860 0x7f39f80ae380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:31.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.538+0000 7f39f6a00640 1 -- 172.21.15.96:0/2821168533 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39f80b24a0 con 0x7f39f80ce560 2024-05-28T06:08:31.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.538+0000 7f39f6a00640 1 --2- 172.21.15.96:0/2821168533 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f39f80ce560 0x7f39f814a240 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f39e8000c00 tx=0x7f39e80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:31.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.539+0000 7f39eea00640 1 -- 172.21.15.96:0/2821168533 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39e8041070 con 0x7f39f80ce560 2024-05-28T06:08:31.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.539+0000 7f39eea00640 1 -- 172.21.15.96:0/2821168533 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39e803c870 con 0x7f39f80ce560 2024-05-28T06:08:31.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.539+0000 7f39eea00640 1 -- 172.21.15.96:0/2821168533 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39e8037040 con 0x7f39f80ce560 2024-05-28T06:08:31.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.539+0000 7f39f7e00640 1 -- 172.21.15.96:0/2821168533 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39f80b2780 con 0x7f39f80ce560 2024-05-28T06:08:31.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.539+0000 7f39f7e00640 1 -- 172.21.15.96:0/2821168533 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39f80ad230 con 0x7f39f80ce560 2024-05-28T06:08:31.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.540+0000 7f39f7e00640 1 -- 172.21.15.96:0/2821168533 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39f80b2950 con 0x7f39f80ce560 2024-05-28T06:08:31.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.541+0000 7f39eea00640 1 -- 172.21.15.96:0/2821168533 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 8) ==== 91193+0+0 (secure 0 0 0) 0x7f39e8044070 con 0x7f39f80ce560 2024-05-28T06:08:31.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.542+0000 7f39eea00640 1 --2- 172.21.15.96:0/2821168533 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f39d806caa0 0x7f39d806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:31.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.542+0000 7f39eea00640 1 -- 172.21.15.96:0/2821168533 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f39e80acaf0 con 0x7f39f80ce560 2024-05-28T06:08:31.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.542+0000 7f39f6000640 1 --2- 172.21.15.96:0/2821168533 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f39d806caa0 0x7f39d806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:31.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.544+0000 7f39f6000640 1 --2- 172.21.15.96:0/2821168533 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f39d806caa0 0x7f39d806ef60 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f39e4007fd0 tx=0x7f39e4007920 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:31.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.547+0000 7f39eea00640 1 -- 172.21.15.96:0/2821168533 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f39e807adb0 con 0x7f39f80ce560 2024-05-28T06:08:31.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.787+0000 7f39f7e00640 1 -- 172.21.15.96:0/2821168533 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot ls", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f39f814a780 con 0x7f39d806caa0 2024-05-28T06:08:31.825 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.824+0000 7f39eea00640 1 -- 172.21.15.96:0/2821168533 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f39f814a780 con 0x7f39d806caa0 2024-05-28T06:08:31.825 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:08:31.825 INFO:teuthology.orchestra.run.smithi096.stdout:[ 2024-05-28T06:08:31.826 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:08:31.826 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "ss1" 2024-05-28T06:08:31.826 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:08:31.826 INFO:teuthology.orchestra.run.smithi096.stdout:] 2024-05-28T06:08:31.828 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.826+0000 7f39f7e00640 1 -- 172.21.15.96:0/2821168533 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f39d806caa0 msgr2=0x7f39d806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:31.828 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.826+0000 7f39f7e00640 1 --2- 172.21.15.96:0/2821168533 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f39d806caa0 0x7f39d806ef60 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f39e4007fd0 tx=0x7f39e4007920 comp rx=0 tx=0).stop 2024-05-28T06:08:31.828 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.826+0000 7f39f7e00640 1 -- 172.21.15.96:0/2821168533 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f39f80ce560 msgr2=0x7f39f814a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:31.828 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.826+0000 7f39f7e00640 1 --2- 172.21.15.96:0/2821168533 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f39f80ce560 0x7f39f814a240 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f39e8000c00 tx=0x7f39e80046c0 comp rx=0 tx=0).stop 2024-05-28T06:08:31.828 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.826+0000 7f39f7e00640 1 -- 172.21.15.96:0/2821168533 shutdown_connections 2024-05-28T06:08:31.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.826+0000 7f39f7e00640 1 --2- 172.21.15.96:0/2821168533 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f39d806caa0 0x7f39d806ef60 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:31.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.826+0000 7f39f7e00640 1 --2- 172.21.15.96:0/2821168533 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f39f80d0860 0x7f39f80ae380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:31.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.826+0000 7f39f7e00640 1 --2- 172.21.15.96:0/2821168533 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f39f80d0430 0x7f39f80b1a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:31.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.826+0000 7f39f7e00640 1 --2- 172.21.15.96:0/2821168533 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f39f80ce560 0x7f39f814a240 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:31.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.826+0000 7f39f7e00640 1 -- 172.21.15.96:0/2821168533 >> 172.21.15.96:0/2821168533 conn(0x7f39f806bae0 msgr2=0x7f39f815f0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:31.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.826+0000 7f39f7e00640 1 -- 172.21.15.96:0/2821168533 shutdown_connections 2024-05-28T06:08:31.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:31.826+0000 7f39f7e00640 1 -- 172.21.15.96:0/2821168533 wait complete. 2024-05-28T06:08:31.848 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-05-28T06:08:32.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.052+0000 7f7672200640 1 Processor -- start 2024-05-28T06:08:32.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.052+0000 7f7672200640 1 -- start start 2024-05-28T06:08:32.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.054+0000 7f7672200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f766c0725e0 0x7f766c166050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:32.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.054+0000 7f7672200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f766c0ade90 con 0x7f766c15b6f0 2024-05-28T06:08:32.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.054+0000 7f7672200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f766c071e70 con 0x7f766c0d0640 2024-05-28T06:08:32.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.054+0000 7f7672200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f766c071c40 con 0x7f766c0725e0 2024-05-28T06:08:32.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.054+0000 7f7662a00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f766c15b6f0 0x7f766c06e350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:42584/0 (socket says 172.21.15.96:42584) 2024-05-28T06:08:32.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.054+0000 7f7662a00640 1 -- 172.21.15.96:0/674595534 learned_addr learned my addr 172.21.15.96:0/674595534 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:08:32.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.055+0000 7f766be00640 1 --2- 172.21.15.96:0/674595534 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f766c0725e0 0x7f766c166050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:32.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.055+0000 7f766aa00640 1 -- 172.21.15.96:0/674595534 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1601184772 0 0) 0x7f766c0ade90 con 0x7f766c15b6f0 2024-05-28T06:08:32.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.055+0000 7f766aa00640 1 -- 172.21.15.96:0/674595534 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7650003520 con 0x7f766c15b6f0 2024-05-28T06:08:32.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.055+0000 7f766aa00640 1 -- 172.21.15.96:0/674595534 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 729616130 0 0) 0x7f7650003520 con 0x7f766c15b6f0 2024-05-28T06:08:32.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.055+0000 7f766aa00640 1 -- 172.21.15.96:0/674595534 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f766c0725e0 msgr2=0x7f766c166050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:32.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.055+0000 7f766aa00640 1 --2- 172.21.15.96:0/674595534 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f766c0725e0 0x7f766c166050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:32.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.055+0000 7f766aa00640 1 -- 172.21.15.96:0/674595534 >> v1:172.21.15.123:6789/0 conn(0x7f766c0d0640 legacy=0x7f766c0d0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:32.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.055+0000 7f766aa00640 1 -- 172.21.15.96:0/674595534 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f766c072130 con 0x7f766c15b6f0 2024-05-28T06:08:32.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.055+0000 7f766aa00640 1 -- 172.21.15.96:0/674595534 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f7654001670 con 0x7f766c15b6f0 2024-05-28T06:08:32.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.055+0000 7f766aa00640 1 -- 172.21.15.96:0/674595534 >> v1:172.21.15.96:6789/0 conn(0x7f766c15b6f0 legacy=0x7f766c06e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:32.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.055+0000 7f766aa00640 1 --2- 172.21.15.96:0/674595534 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f76500039e0 0x7f7650003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:32.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.055+0000 7f766aa00640 1 --2- 172.21.15.96:0/674595534 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7650004800 0x7f7650004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:32.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.055+0000 7f766aa00640 1 --2- 172.21.15.96:0/674595534 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f766c0725e0 0x7f7650005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:32.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.055+0000 7f766aa00640 1 -- 172.21.15.96:0/674595534 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f765000bac0 con 0x7f76500039e0 2024-05-28T06:08:32.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.055+0000 7f766aa00640 1 -- 172.21.15.96:0/674595534 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f765000bc70 con 0x7f7650004800 2024-05-28T06:08:32.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.055+0000 7f766aa00640 1 -- 172.21.15.96:0/674595534 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f765000be70 con 0x7f766c0725e0 2024-05-28T06:08:32.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.056+0000 7f766b400640 1 --2- 172.21.15.96:0/674595534 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f76500039e0 0x7f7650003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:32.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.056+0000 7f766b400640 1 -- 172.21.15.96:0/674595534 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f766c0725e0 msgr2=0x7f7650005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:32.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.056+0000 7f766be00640 1 --2- 172.21.15.96:0/674595534 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f766c0725e0 0x7f7650005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:32.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.056+0000 7f7662a00640 1 --2- 172.21.15.96:0/674595534 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7650004800 0x7f7650004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:32.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.056+0000 7f766b400640 1 --2- 172.21.15.96:0/674595534 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f766c0725e0 0x7f7650005200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:32.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.056+0000 7f766b400640 1 -- 172.21.15.96:0/674595534 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7650004800 msgr2=0x7f7650004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:32.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.056+0000 7f766b400640 1 --2- 172.21.15.96:0/674595534 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7650004800 0x7f7650004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:32.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.056+0000 7f766b400640 1 -- 172.21.15.96:0/674595534 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7650003520 con 0x7f76500039e0 2024-05-28T06:08:32.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.056+0000 7f7662a00640 1 --2- 172.21.15.96:0/674595534 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7650004800 0x7f7650004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:08:32.062 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.057+0000 7f766b400640 1 --2- 172.21.15.96:0/674595534 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f76500039e0 0x7f7650003e00 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f765c001e50 tx=0x7f7650005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc8df2b9444036c9 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:32.062 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.057+0000 7f766aa00640 1 -- 172.21.15.96:0/674595534 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f765c001a70 con 0x7f76500039e0 2024-05-28T06:08:32.062 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.057+0000 7f766aa00640 1 -- 172.21.15.96:0/674595534 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f765c003970 con 0x7f76500039e0 2024-05-28T06:08:32.062 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.057+0000 7f766aa00640 1 -- 172.21.15.96:0/674595534 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f765c003d60 con 0x7f76500039e0 2024-05-28T06:08:32.062 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.058+0000 7f7672200640 1 -- 172.21.15.96:0/674595534 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f76500039e0 msgr2=0x7f7650003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:32.062 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.058+0000 7f7672200640 1 --2- 172.21.15.96:0/674595534 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f76500039e0 0x7f7650003e00 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f765c001e50 tx=0x7f7650005830 comp rx=0 tx=0).stop 2024-05-28T06:08:32.062 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.058+0000 7f766be00640 1 -- 172.21.15.96:0/674595534 reap_dead start 2024-05-28T06:08:32.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.058+0000 7f7672200640 1 -- 172.21.15.96:0/674595534 shutdown_connections 2024-05-28T06:08:32.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.058+0000 7f7672200640 1 -- 172.21.15.96:0/674595534 >> 172.21.15.96:0/674595534 conn(0x7f766c06bd20 msgr2=0x7f766c06c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:32.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.058+0000 7f7672200640 1 -- 172.21.15.96:0/674595534 shutdown_connections 2024-05-28T06:08:32.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.058+0000 7f7672200640 1 -- 172.21.15.96:0/674595534 wait complete. 2024-05-28T06:08:32.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.060+0000 7f7672200640 1 Processor -- start 2024-05-28T06:08:32.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.060+0000 7f7672200640 1 -- start start 2024-05-28T06:08:32.065 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.061+0000 7f7672200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f76500039e0 0x7f766c15e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:32.065 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.061+0000 7f7672200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f766c157600 0x7f766c157a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:32.065 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.061+0000 7f7672200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f766c1f3db0 0x7f766c1f4230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:32.065 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.061+0000 7f7672200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f766c0af7e0 con 0x7f766c1f3db0 2024-05-28T06:08:32.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.061+0000 7f7672200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f766c0af660 con 0x7f766c157600 2024-05-28T06:08:32.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.061+0000 7f7672200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f766c0af960 con 0x7f76500039e0 2024-05-28T06:08:32.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.061+0000 7f766be00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f766c1f3db0 0x7f766c1f4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:32.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.061+0000 7f766be00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f766c1f3db0 0x7f766c1f4230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35796/0 (socket says 172.21.15.96:35796) 2024-05-28T06:08:32.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.061+0000 7f766be00640 1 -- 172.21.15.96:0/40700387 learned_addr learned my addr 172.21.15.96:0/40700387 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:08:32.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.062+0000 7f766be00640 1 -- 172.21.15.96:0/40700387 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f76500039e0 msgr2=0x7f766c15e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:32.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.062+0000 7f766b400640 1 --2- 172.21.15.96:0/40700387 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f76500039e0 0x7f766c15e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:32.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.062+0000 7f7662a00640 1 --2- 172.21.15.96:0/40700387 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f766c157600 0x7f766c157a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:32.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.062+0000 7f766be00640 1 --2- 172.21.15.96:0/40700387 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f76500039e0 0x7f766c15e080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:32.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.062+0000 7f766be00640 1 -- 172.21.15.96:0/40700387 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f766c157600 msgr2=0x7f766c157a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:32.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.062+0000 7f766be00640 1 --2- 172.21.15.96:0/40700387 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f766c157600 0x7f766c157a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:32.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.062+0000 7f766be00640 1 -- 172.21.15.96:0/40700387 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f766c15e5c0 con 0x7f766c1f3db0 2024-05-28T06:08:32.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.062+0000 7f7662a00640 1 --2- 172.21.15.96:0/40700387 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f766c157600 0x7f766c157a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:08:32.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.062+0000 7f766be00640 1 --2- 172.21.15.96:0/40700387 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f766c1f3db0 0x7f766c1f4230 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f7664006fd0 tx=0x7f766400e8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:32.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.062+0000 7f7663e00640 1 -- 172.21.15.96:0/40700387 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f766400f040 con 0x7f766c1f3db0 2024-05-28T06:08:32.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.062+0000 7f7663e00640 1 -- 172.21.15.96:0/40700387 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7664019070 con 0x7f766c1f3db0 2024-05-28T06:08:32.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.063+0000 7f7672200640 1 -- 172.21.15.96:0/40700387 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f766c158bc0 con 0x7f766c1f3db0 2024-05-28T06:08:32.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.063+0000 7f7663e00640 1 -- 172.21.15.96:0/40700387 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7664013850 con 0x7f766c1f3db0 2024-05-28T06:08:32.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.063+0000 7f7672200640 1 -- 172.21.15.96:0/40700387 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f766c159180 con 0x7f766c1f3db0 2024-05-28T06:08:32.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.064+0000 7f7672200640 1 -- 172.21.15.96:0/40700387 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f766c157fc0 con 0x7f766c1f3db0 2024-05-28T06:08:32.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.065+0000 7f7663e00640 1 -- 172.21.15.96:0/40700387 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 8) ==== 91193+0+0 (secure 0 0 0) 0x7f76640139f0 con 0x7f766c1f3db0 2024-05-28T06:08:32.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.066+0000 7f7663e00640 1 --2- 172.21.15.96:0/40700387 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f764806caa0 0x7f764806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:32.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.066+0000 7f7663e00640 1 -- 172.21.15.96:0/40700387 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f766408b6d0 con 0x7f766c1f3db0 2024-05-28T06:08:32.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.066+0000 7f766b400640 1 --2- 172.21.15.96:0/40700387 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f764806caa0 0x7f764806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:32.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.067+0000 7f766b400640 1 --2- 172.21.15.96:0/40700387 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f764806caa0 0x7f764806ef60 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f765c032000 tx=0x7f765c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:32.071 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.070+0000 7f7663e00640 1 -- 172.21.15.96:0/40700387 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f76640596d0 con 0x7f766c1f3db0 2024-05-28T06:08:32.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.308+0000 7f7672200640 1 -- 172.21.15.96:0/40700387 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f766c0689e0 con 0x7f764806caa0 2024-05-28T06:08:32.381 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.380+0000 7f7663e00640 1 -- 172.21.15.96:0/40700387 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f766c0689e0 con 0x7f764806caa0 2024-05-28T06:08:32.382 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:08:32.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.383+0000 7f7672200640 1 -- 172.21.15.96:0/40700387 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f764806caa0 msgr2=0x7f764806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:32.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.383+0000 7f7672200640 1 --2- 172.21.15.96:0/40700387 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f764806caa0 0x7f764806ef60 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f765c032000 tx=0x7f765c02e000 comp rx=0 tx=0).stop 2024-05-28T06:08:32.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.383+0000 7f7672200640 1 -- 172.21.15.96:0/40700387 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f766c1f3db0 msgr2=0x7f766c1f4230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:32.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.383+0000 7f7672200640 1 --2- 172.21.15.96:0/40700387 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f766c1f3db0 0x7f766c1f4230 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f7664006fd0 tx=0x7f766400e8b0 comp rx=0 tx=0).stop 2024-05-28T06:08:32.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.383+0000 7f7672200640 1 -- 172.21.15.96:0/40700387 shutdown_connections 2024-05-28T06:08:32.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.383+0000 7f7672200640 1 --2- 172.21.15.96:0/40700387 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f764806caa0 0x7f764806ef60 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:32.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.383+0000 7f7672200640 1 --2- 172.21.15.96:0/40700387 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f766c1f3db0 0x7f766c1f4230 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:32.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.383+0000 7f7672200640 1 --2- 172.21.15.96:0/40700387 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f766c157600 0x7f766c157a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:32.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.383+0000 7f7672200640 1 --2- 172.21.15.96:0/40700387 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f76500039e0 0x7f766c15e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:32.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.383+0000 7f7672200640 1 -- 172.21.15.96:0/40700387 >> 172.21.15.96:0/40700387 conn(0x7f766c06bd20 msgr2=0x7f766c072c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:32.386 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.383+0000 7f7672200640 1 -- 172.21.15.96:0/40700387 shutdown_connections 2024-05-28T06:08:32.386 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.383+0000 7f7672200640 1 -- 172.21.15.96:0/40700387 wait complete. 2024-05-28T06:08:32.404 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs sv1 2024-05-28T06:08:32.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.606+0000 7fd7a1400640 1 Processor -- start 2024-05-28T06:08:32.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.607+0000 7fd7a1400640 1 -- start start 2024-05-28T06:08:32.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.608+0000 7fd7a1400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd79c0d0640 0x7fd79c15d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:32.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.608+0000 7fd7a1400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fd79c0b32c0 0x7fd79c0b3750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:32.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.608+0000 7fd7a1400640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd79c0d0060 con 0x7fd79c0d0a20 2024-05-28T06:08:32.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.608+0000 7fd7a1400640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fd79c0adbb0 con 0x7fd79c0b32c0 2024-05-28T06:08:32.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.608+0000 7fd7a1400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fd79c15db10 con 0x7fd79c0d0640 2024-05-28T06:08:32.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.608+0000 7fd791e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fd79c0d0a20 0x7fd79c0cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:42598/0 (socket says 172.21.15.96:42598) 2024-05-28T06:08:32.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.608+0000 7fd791e00640 1 -- 172.21.15.96:0/103891159 learned_addr learned my addr 172.21.15.96:0/103891159 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:08:32.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.609+0000 7fd79a800640 1 --2- 172.21.15.96:0/103891159 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd79c0d0640 0x7fd79c15d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:32.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.609+0000 7fd79b200640 1 --2- 172.21.15.96:0/103891159 >> v2:172.21.15.123:3300/0 conn(0x7fd79c0b32c0 0x7fd79c0b3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:32.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.609+0000 7fd799e00640 1 -- 172.21.15.96:0/103891159 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2170878531 0 0) 0x7fd79c0d0060 con 0x7fd79c0d0a20 2024-05-28T06:08:32.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.609+0000 7fd799e00640 1 -- 172.21.15.96:0/103891159 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd780003520 con 0x7fd79c0d0a20 2024-05-28T06:08:32.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.609+0000 7fd799e00640 1 -- 172.21.15.96:0/103891159 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 13962664 0 0) 0x7fd780003520 con 0x7fd79c0d0a20 2024-05-28T06:08:32.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.609+0000 7fd799e00640 1 -- 172.21.15.96:0/103891159 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd79c0d0640 msgr2=0x7fd79c15d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:32.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.609+0000 7fd799e00640 1 --2- 172.21.15.96:0/103891159 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd79c0d0640 0x7fd79c15d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:32.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.609+0000 7fd799e00640 1 -- 172.21.15.96:0/103891159 >> v2:172.21.15.123:3300/0 conn(0x7fd79c0b32c0 msgr2=0x7fd79c0b3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:32.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.609+0000 7fd799e00640 1 --2- 172.21.15.96:0/103891159 >> v2:172.21.15.123:3300/0 conn(0x7fd79c0b32c0 0x7fd79c0b3750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:32.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.609+0000 7fd799e00640 1 -- 172.21.15.96:0/103891159 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd79c15cb10 con 0x7fd79c0d0a20 2024-05-28T06:08:32.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.609+0000 7fd799e00640 1 -- 172.21.15.96:0/103891159 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7fd784001670 con 0x7fd79c0d0a20 2024-05-28T06:08:32.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.609+0000 7fd799e00640 1 -- 172.21.15.96:0/103891159 >> v1:172.21.15.96:6789/0 conn(0x7fd79c0d0a20 legacy=0x7fd79c0cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:32.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.609+0000 7fd799e00640 1 --2- 172.21.15.96:0/103891159 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd79c0d0640 0x7fd7800039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:32.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.609+0000 7fd799e00640 1 --2- 172.21.15.96:0/103891159 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd780003f80 0x7fd780004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:32.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.609+0000 7fd799e00640 1 --2- 172.21.15.96:0/103891159 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd780004d40 0x7fd7800051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:32.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.609+0000 7fd799e00640 1 -- 172.21.15.96:0/103891159 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd79c15db10 con 0x7fd780004d40 2024-05-28T06:08:32.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.609+0000 7fd799e00640 1 -- 172.21.15.96:0/103891159 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd78000b9c0 con 0x7fd780003f80 2024-05-28T06:08:32.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.609+0000 7fd799e00640 1 -- 172.21.15.96:0/103891159 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fd78000bb40 con 0x7fd79c0d0640 2024-05-28T06:08:32.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.610+0000 7fd79b200640 1 --2- 172.21.15.96:0/103891159 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd780004d40 0x7fd7800051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:32.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.610+0000 7fd79b200640 1 -- 172.21.15.96:0/103891159 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd79c0d0640 msgr2=0x7fd7800039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:32.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.610+0000 7fd79b200640 1 --2- 172.21.15.96:0/103891159 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd79c0d0640 0x7fd7800039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:32.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.610+0000 7fd79b200640 1 -- 172.21.15.96:0/103891159 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd780003f80 msgr2=0x7fd780004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:32.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.610+0000 7fd79b200640 1 --2- 172.21.15.96:0/103891159 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd780003f80 0x7fd780004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:32.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.610+0000 7fd79b200640 1 -- 172.21.15.96:0/103891159 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd780003520 con 0x7fd780004d40 2024-05-28T06:08:32.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.611+0000 7fd79b200640 1 --2- 172.21.15.96:0/103891159 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd780004d40 0x7fd7800051c0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fd794009af0 tx=0x7fd780005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8566d33c2e09b6e server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:32.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.611+0000 7fd799e00640 1 -- 172.21.15.96:0/103891159 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd794036070 con 0x7fd780004d40 2024-05-28T06:08:32.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.611+0000 7fd799e00640 1 -- 172.21.15.96:0/103891159 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd794031400 con 0x7fd780004d40 2024-05-28T06:08:32.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.611+0000 7fd799e00640 1 -- 172.21.15.96:0/103891159 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd794009690 con 0x7fd780004d40 2024-05-28T06:08:32.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.612+0000 7fd7a1400640 1 -- 172.21.15.96:0/103891159 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd780004d40 msgr2=0x7fd7800051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:32.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.612+0000 7fd7a1400640 1 --2- 172.21.15.96:0/103891159 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd780004d40 0x7fd7800051c0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fd794009af0 tx=0x7fd780005830 comp rx=0 tx=0).stop 2024-05-28T06:08:32.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.612+0000 7fd79b200640 1 -- 172.21.15.96:0/103891159 reap_dead start 2024-05-28T06:08:32.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.612+0000 7fd7a1400640 1 -- 172.21.15.96:0/103891159 shutdown_connections 2024-05-28T06:08:32.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.612+0000 7fd7a1400640 1 -- 172.21.15.96:0/103891159 >> 172.21.15.96:0/103891159 conn(0x7fd79c06bd20 msgr2=0x7fd79c06c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:32.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.612+0000 7fd7a1400640 1 -- 172.21.15.96:0/103891159 shutdown_connections 2024-05-28T06:08:32.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.612+0000 7fd7a1400640 1 -- 172.21.15.96:0/103891159 wait complete. 2024-05-28T06:08:32.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.614+0000 7fd7a1400640 1 Processor -- start 2024-05-28T06:08:32.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.614+0000 7fd7a1400640 1 -- start start 2024-05-28T06:08:32.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.615+0000 7fd7a1400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd780004d40 0x7fd79c1f3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:32.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.615+0000 7fd7a1400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd79c152ec0 0x7fd79c14f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:32.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.615+0000 7fd7a1400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd79c1554d0 0x7fd79c155980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:32.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.615+0000 7fd7a1400640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd79c15cdd0 con 0x7fd79c1554d0 2024-05-28T06:08:32.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.615+0000 7fd7a1400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd79c0ad8d0 con 0x7fd780004d40 2024-05-28T06:08:32.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.615+0000 7fd7a1400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fd79c15d0b0 con 0x7fd79c152ec0 2024-05-28T06:08:32.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.615+0000 7fd79b200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd79c1554d0 0x7fd79c155980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:32.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.615+0000 7fd79b200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd79c1554d0 0x7fd79c155980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35810/0 (socket says 172.21.15.96:35810) 2024-05-28T06:08:32.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.615+0000 7fd79b200640 1 -- 172.21.15.96:0/291582107 learned_addr learned my addr 172.21.15.96:0/291582107 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:08:32.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.616+0000 7fd791e00640 1 --2- 172.21.15.96:0/291582107 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd79c152ec0 0x7fd79c14f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:32.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.616+0000 7fd79b200640 1 -- 172.21.15.96:0/291582107 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd79c152ec0 msgr2=0x7fd79c14f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:32.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.616+0000 7fd79a800640 1 --2- 172.21.15.96:0/291582107 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd780004d40 0x7fd79c1f3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:32.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.616+0000 7fd79b200640 1 --2- 172.21.15.96:0/291582107 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd79c152ec0 0x7fd79c14f870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:32.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.616+0000 7fd79b200640 1 -- 172.21.15.96:0/291582107 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd780004d40 msgr2=0x7fd79c1f3cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:32.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.616+0000 7fd79b200640 1 --2- 172.21.15.96:0/291582107 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd780004d40 0x7fd79c1f3cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:32.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.616+0000 7fd79b200640 1 -- 172.21.15.96:0/291582107 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd79c153630 con 0x7fd79c1554d0 2024-05-28T06:08:32.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.616+0000 7fd79b200640 1 --2- 172.21.15.96:0/291582107 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd79c1554d0 0x7fd79c155980 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fd794034040 tx=0x7fd794004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:32.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.616+0000 7fd793400640 1 -- 172.21.15.96:0/291582107 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd794036040 con 0x7fd79c1554d0 2024-05-28T06:08:32.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.616+0000 7fd793400640 1 -- 172.21.15.96:0/291582107 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd794041070 con 0x7fd79c1554d0 2024-05-28T06:08:32.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.616+0000 7fd793400640 1 -- 172.21.15.96:0/291582107 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd794004520 con 0x7fd79c1554d0 2024-05-28T06:08:32.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.616+0000 7fd7a1400640 1 -- 172.21.15.96:0/291582107 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd79c153910 con 0x7fd79c1554d0 2024-05-28T06:08:32.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.616+0000 7fd7a1400640 1 -- 172.21.15.96:0/291582107 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd79c153ed0 con 0x7fd79c1554d0 2024-05-28T06:08:32.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.617+0000 7fd7a1400640 1 -- 172.21.15.96:0/291582107 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd79c0689e0 con 0x7fd79c1554d0 2024-05-28T06:08:32.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.618+0000 7fd793400640 1 -- 172.21.15.96:0/291582107 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 8) ==== 91193+0+0 (secure 0 0 0) 0x7fd794039070 con 0x7fd79c1554d0 2024-05-28T06:08:32.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.619+0000 7fd793400640 1 --2- 172.21.15.96:0/291582107 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd77806caa0 0x7fd77806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:32.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.619+0000 7fd793400640 1 -- 172.21.15.96:0/291582107 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7fd7940ab330 con 0x7fd79c1554d0 2024-05-28T06:08:32.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.620+0000 7fd79a800640 1 --2- 172.21.15.96:0/291582107 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd77806caa0 0x7fd77806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:32.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.621+0000 7fd79a800640 1 --2- 172.21.15.96:0/291582107 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd77806caa0 0x7fd77806ef60 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fd78c004b30 tx=0x7fd78c00b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:32.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.624+0000 7fd793400640 1 -- 172.21.15.96:0/291582107 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd79404a020 con 0x7fd79c1554d0 2024-05-28T06:08:32.857 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.855+0000 7fd7a1400640 1 -- 172.21.15.96:0/291582107 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) -- 0x7fd79c071ef0 con 0x7fd77806caa0 2024-05-28T06:08:32.905 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.904+0000 7fd793400640 1 -- 172.21.15.96:0/291582107 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fd79c071ef0 con 0x7fd77806caa0 2024-05-28T06:08:32.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.905+0000 7fd7a1400640 1 -- 172.21.15.96:0/291582107 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd77806caa0 msgr2=0x7fd77806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:32.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.906+0000 7fd7a1400640 1 --2- 172.21.15.96:0/291582107 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd77806caa0 0x7fd77806ef60 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fd78c004b30 tx=0x7fd78c00b040 comp rx=0 tx=0).stop 2024-05-28T06:08:32.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.906+0000 7fd7a1400640 1 -- 172.21.15.96:0/291582107 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd79c1554d0 msgr2=0x7fd79c155980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:32.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.906+0000 7fd7a1400640 1 --2- 172.21.15.96:0/291582107 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd79c1554d0 0x7fd79c155980 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fd794034040 tx=0x7fd794004730 comp rx=0 tx=0).stop 2024-05-28T06:08:32.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.906+0000 7fd7a1400640 1 -- 172.21.15.96:0/291582107 shutdown_connections 2024-05-28T06:08:32.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.906+0000 7fd7a1400640 1 --2- 172.21.15.96:0/291582107 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd77806caa0 0x7fd77806ef60 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:32.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.906+0000 7fd7a1400640 1 --2- 172.21.15.96:0/291582107 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd79c1554d0 0x7fd79c155980 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:32.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.906+0000 7fd7a1400640 1 --2- 172.21.15.96:0/291582107 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd79c152ec0 0x7fd79c14f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:32.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.906+0000 7fd7a1400640 1 --2- 172.21.15.96:0/291582107 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd780004d40 0x7fd79c1f3cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:32.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.906+0000 7fd7a1400640 1 -- 172.21.15.96:0/291582107 >> 172.21.15.96:0/291582107 conn(0x7fd79c06bd20 msgr2=0x7fd79c073330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:32.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.906+0000 7fd7a1400640 1 -- 172.21.15.96:0/291582107 shutdown_connections 2024-05-28T06:08:32.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:32.906+0000 7fd7a1400640 1 -- 172.21.15.96:0/291582107 wait complete. 2024-05-28T06:08:32.915 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:08:32.913+0000 7fea5da00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:08:32.915 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:08:32.913+0000 7fea5da00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:08:32.915 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:08:32.913+0000 7fea5da00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:08:32.915 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:08:32.913+0000 7fea5da00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:08:32.915 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:08:32.913+0000 7fea5da00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:08:32.929 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs ss1clone1 2024-05-28T06:08:32.940 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:08:32.939+0000 7fea5bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:08:32.940 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:08:32.939+0000 7fea5bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:08:32.941 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:08:32.939+0000 7fea5bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:08:32.941 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:08:32.939+0000 7fea5bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:08:32.941 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:08:32.939+0000 7fea5bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:08:33.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:33.136+0000 7ff92d400640 1 Processor -- start 2024-05-28T06:08:33.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:33.137+0000 7ff92d400640 1 -- start start 2024-05-28T06:08:33.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:33.139+0000 7ff92d400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7ff9280d34e0 0x7ff9280d4e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:33.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:33.139+0000 7ff92d400640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9280d5500 con 0x7ff9281622b0 2024-05-28T06:08:33.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:33.139+0000 7ff92d400640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9280d2720 con 0x7ff9280b5570 2024-05-28T06:08:33.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:33.139+0000 7ff92d400640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7ff9280d2a00 con 0x7ff9280d34e0 2024-05-28T06:08:33.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:33.139+0000 7ff91fe00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7ff9280d34e0 0x7ff9280d4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:33.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:33.139+0000 7ff926800640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7ff9281622b0 0x7ff9280d1b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:42614/0 (socket says 172.21.15.96:42614) 2024-05-28T06:08:33.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:33.139+0000 7ff926800640 1 -- 172.21.15.96:0/861521572 learned_addr learned my addr 172.21.15.96:0/861521572 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:08:33.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:33.139+0000 7ff91fe00640 1 -- 172.21.15.96:0/861521572 >> v1:172.21.15.123:6789/0 conn(0x7ff9280b5570 legacy=0x7ff9280b3f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:33.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:33.139+0000 7ff91fe00640 1 -- 172.21.15.96:0/861521572 >> v1:172.21.15.96:6789/0 conn(0x7ff9281622b0 legacy=0x7ff9280d1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:33.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:33.139+0000 7ff925e00640 1 -- 172.21.15.96:0/861521572 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2246526713 0 0) 0x7ff9280d5500 con 0x7ff9281622b0 2024-05-28T06:08:33.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:33.139+0000 7ff91fe00640 1 -- 172.21.15.96:0/861521572 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9280d2ce0 con 0x7ff9280d34e0 2024-05-28T06:08:33.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:33.140+0000 7ff91fe00640 1 --2- 172.21.15.96:0/861521572 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff9280d34e0 0x7ff9280d4e00 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7ff9100097a0 tx=0x7ff91002a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49dd442c93d9ccba server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:33.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:33.140+0000 7ff925e00640 1 -- 172.21.15.96:0/861521572 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff910037070 con 0x7ff9280d34e0 2024-05-28T06:08:33.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:33.140+0000 7ff925e00640 1 -- 172.21.15.96:0/861521572 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff91002ae50 con 0x7ff9280d34e0 2024-05-28T06:08:33.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:33.140+0000 7ff925e00640 1 -- 172.21.15.96:0/861521572 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff910032660 con 0x7ff9280d34e0 2024-05-28T06:08:33.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:33.141+0000 7ff92d400640 1 -- 172.21.15.96:0/861521572 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff9280d34e0 msgr2=0x7ff9280d4e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:33.143 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:33.141+0000 7ff92d400640 1 --2- 172.21.15.96:0/861521572 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff9280d34e0 0x7ff9280d4e00 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7ff9100097a0 tx=0x7ff91002a140 comp rx=0 tx=0).stop 2024-05-28T06:08:33.143 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:33.141+0000 7ff92d400640 1 -- 172.21.15.96:0/861521572 shutdown_connections 2024-05-28T06:08:33.143 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:33.141+0000 7ff92d400640 1 --2- 172.21.15.96:0/861521572 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff9280d34e0 0x7ff9280d4e00 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:33.143 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:33.141+0000 7ff92d400640 1 -- 172.21.15.96:0/861521572 >> 172.21.15.96:0/861521572 conn(0x7ff92808bfc0 msgr2=0x7ff9280d46f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:33.143 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:33.141+0000 7ff92d400640 1 -- 172.21.15.96:0/861521572 shutdown_connections 2024-05-28T06:08:33.143 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:33.141+0000 7ff92d400640 1 -- 172.21.15.96:0/861521572 wait complete. 2024-05-28T06:08:33.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:33.143+0000 7ff92d400640 1 Processor -- start 2024-05-28T06:08:33.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:33.143+0000 7ff92d400640 1 -- start start 2024-05-28T06:08:33.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:33.144+0000 7ff92d400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff9280b5570 0x7ff92815a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:33.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:33.144+0000 7ff92d400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff9280d34e0 0x7ff9281f4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:33.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:33.144+0000 7ff92d400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff9281622b0 0x7ff9280b8a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:33.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:33.144+0000 7ff92d400640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff9280b5d90 con 0x7ff9280b5570 2024-05-28T06:08:33.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:33.144+0000 7ff92d400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff9280b5c10 con 0x7ff9281622b0 2024-05-28T06:08:33.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:33.144+0000 7ff92d400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ff9280b93c0 con 0x7ff9280d34e0 2024-05-28T06:08:33.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:33.144+0000 7ff926800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff9280b5570 0x7ff92815a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:33.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:33.144+0000 7ff926800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff9280b5570 0x7ff92815a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35832/0 (socket says 172.21.15.96:35832) 2024-05-28T06:08:33.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:33.144+0000 7ff926800640 1 -- 172.21.15.96:0/1454647384 learned_addr learned my addr 172.21.15.96:0/1454647384 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:08:33.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:33.145+0000 7ff91fe00640 1 --2- 172.21.15.96:0/1454647384 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff9280d34e0 0x7ff9281f4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:33.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:33.145+0000 7ff926800640 1 -- 172.21.15.96:0/1454647384 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff9280d34e0 msgr2=0x7ff9281f4ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:33.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:33.145+0000 7ff927200640 1 --2- 172.21.15.96:0/1454647384 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff9281622b0 0x7ff9280b8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:33.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:33.145+0000 7ff926800640 1 --2- 172.21.15.96:0/1454647384 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff9280d34e0 0x7ff9281f4ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:33.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:33.145+0000 7ff926800640 1 -- 172.21.15.96:0/1454647384 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff9281622b0 msgr2=0x7ff9280b8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:33.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:33.145+0000 7ff926800640 1 --2- 172.21.15.96:0/1454647384 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff9281622b0 0x7ff9280b8a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:33.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:33.145+0000 7ff926800640 1 -- 172.21.15.96:0/1454647384 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff92815a840 con 0x7ff9280b5570 2024-05-28T06:08:33.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:33.145+0000 7ff926800640 1 --2- 172.21.15.96:0/1454647384 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff9280b5570 0x7ff92815a300 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7ff91800c460 tx=0x7ff91800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:33.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:33.145+0000 7ff91ea00640 1 -- 172.21.15.96:0/1454647384 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff918018070 con 0x7ff9280b5570 2024-05-28T06:08:33.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:33.145+0000 7ff91ea00640 1 -- 172.21.15.96:0/1454647384 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff918007500 con 0x7ff9280b5570 2024-05-28T06:08:33.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:33.145+0000 7ff91ea00640 1 -- 172.21.15.96:0/1454647384 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff918004a20 con 0x7ff9280b5570 2024-05-28T06:08:33.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:33.145+0000 7ff92d400640 1 -- 172.21.15.96:0/1454647384 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9280b9610 con 0x7ff9280b5570 2024-05-28T06:08:33.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:33.146+0000 7ff92d400640 1 -- 172.21.15.96:0/1454647384 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9280b9bd0 con 0x7ff9280b5570 2024-05-28T06:08:33.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:33.147+0000 7ff92d400640 1 -- 172.21.15.96:0/1454647384 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9280689e0 con 0x7ff9280b5570 2024-05-28T06:08:33.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:33.148+0000 7ff91ea00640 1 -- 172.21.15.96:0/1454647384 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 8) ==== 91193+0+0 (secure 0 0 0) 0x7ff91801b070 con 0x7ff9280b5570 2024-05-28T06:08:33.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:33.149+0000 7ff91ea00640 1 --2- 172.21.15.96:0/1454647384 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff90406caa0 0x7ff90406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:33.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:33.149+0000 7ff91ea00640 1 -- 172.21.15.96:0/1454647384 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7ff91808b060 con 0x7ff9280b5570 2024-05-28T06:08:33.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:33.149+0000 7ff91fe00640 1 --2- 172.21.15.96:0/1454647384 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff90406caa0 0x7ff90406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:33.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:33.151+0000 7ff91fe00640 1 --2- 172.21.15.96:0/1454647384 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff90406caa0 0x7ff90406ef60 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ff910034000 tx=0x7ff91002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:33.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:33.153+0000 7ff91ea00640 1 -- 172.21.15.96:0/1454647384 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff9180592a0 con 0x7ff9280b5570 2024-05-28T06:08:33.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:33.384+0000 7ff92d400640 1 -- 172.21.15.96:0/1454647384 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "ss1clone1", "target": ["mon-mgr", ""]}) -- 0x7ff92808c6c0 con 0x7ff90406caa0 2024-05-28T06:08:33.436 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:33.435+0000 7ff91ea00640 1 -- 172.21.15.96:0/1454647384 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7ff92808c6c0 con 0x7ff90406caa0 2024-05-28T06:08:33.439 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:33.438+0000 7ff92d400640 1 -- 172.21.15.96:0/1454647384 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff90406caa0 msgr2=0x7ff90406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:33.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:33.438+0000 7ff92d400640 1 --2- 172.21.15.96:0/1454647384 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff90406caa0 0x7ff90406ef60 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ff910034000 tx=0x7ff91002e000 comp rx=0 tx=0).stop 2024-05-28T06:08:33.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:33.438+0000 7ff92d400640 1 -- 172.21.15.96:0/1454647384 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff9280b5570 msgr2=0x7ff92815a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:33.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:33.438+0000 7ff92d400640 1 --2- 172.21.15.96:0/1454647384 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff9280b5570 0x7ff92815a300 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7ff91800c460 tx=0x7ff91800c920 comp rx=0 tx=0).stop 2024-05-28T06:08:33.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:33.438+0000 7ff92d400640 1 -- 172.21.15.96:0/1454647384 shutdown_connections 2024-05-28T06:08:33.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:33.438+0000 7ff92d400640 1 --2- 172.21.15.96:0/1454647384 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff90406caa0 0x7ff90406ef60 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:33.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:33.438+0000 7ff92d400640 1 --2- 172.21.15.96:0/1454647384 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff9281622b0 0x7ff9280b8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:33.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:33.438+0000 7ff92d400640 1 --2- 172.21.15.96:0/1454647384 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff9280d34e0 0x7ff9281f4ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:33.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:33.438+0000 7ff92d400640 1 --2- 172.21.15.96:0/1454647384 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff9280b5570 0x7ff92815a300 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:33.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:33.438+0000 7ff92d400640 1 -- 172.21.15.96:0/1454647384 >> 172.21.15.96:0/1454647384 conn(0x7ff92808bfc0 msgr2=0x7ff92806e120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:33.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:33.438+0000 7ff92d400640 1 -- 172.21.15.96:0/1454647384 shutdown_connections 2024-05-28T06:08:33.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:33.438+0000 7ff92d400640 1 -- 172.21.15.96:0/1454647384 wait complete. 2024-05-28T06:08:33.459 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2024-05-28T06:08:33.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:33 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-28T06:08:33.567 INFO:teuthology.orchestra.run.smithi096.stdout:2 2024-05-28T06:08:33.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:33 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-28T06:08:33.577 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-05-28T06:08:33.577 DEBUG:teuthology.orchestra.run.smithi096:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-05-28T06:08:33.681 DEBUG:teuthology.orchestra.run.smithi123:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-05-28T06:08:33.787 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-05-28T06:08:33.787 INFO:tasks.cephfs.mount:Terminating background process 2024-05-28T06:08:33.788 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:08:33.788 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:08:33.838 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:08:34.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.067+0000 7f1301800640 1 Processor -- start 2024-05-28T06:08:34.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.068+0000 7f1301800640 1 -- start start 2024-05-28T06:08:34.071 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.069+0000 7f1301800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f12fc15e870 0x7f12fc0d0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:34.071 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.069+0000 7f1301800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f12fc0d14d0 0x7f12fc0cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:34.071 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.069+0000 7f1301800640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12fc0cfd00 con 0x7f12fc0cf870 2024-05-28T06:08:34.071 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.069+0000 7f1301800640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f12fc0d2150 con 0x7f12fc0d14d0 2024-05-28T06:08:34.071 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.069+0000 7f1301800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f12fc15ec50 con 0x7f12fc15e870 2024-05-28T06:08:34.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.070+0000 7f12fbe00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f12fc15e870 0x7f12fc0d0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:34.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.070+0000 7f1300e00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f12fc0cf870 0x7f12fc162360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.96:37982/0 (socket says 172.21.15.96:37982) 2024-05-28T06:08:34.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.070+0000 7f12f3e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f12fc0d14d0 0x7f12fc0cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:34.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.070+0000 7f1300e00640 1 -- 172.21.15.96:0/2521415351 learned_addr learned my addr 172.21.15.96:0/2521415351 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:08:34.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.071+0000 7f12fbe00640 1 -- 172.21.15.96:0/2521415351 >> v2:172.21.15.123:3300/0 conn(0x7f12fc0d14d0 msgr2=0x7f12fc0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:34.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.071+0000 7f12fbe00640 1 --2- 172.21.15.96:0/2521415351 >> v2:172.21.15.123:3300/0 conn(0x7f12fc0d14d0 0x7f12fc0cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:34.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.071+0000 7f12fbe00640 1 -- 172.21.15.96:0/2521415351 >> v1:172.21.15.123:6789/0 conn(0x7f12fc0cf870 legacy=0x7f12fc162360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:34.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.071+0000 7f12fbe00640 1 -- 172.21.15.96:0/2521415351 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12fc0cff80 con 0x7f12fc15e870 2024-05-28T06:08:34.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.071+0000 7f12f3e00640 1 --2- 172.21.15.96:0/2521415351 >> v2:172.21.15.123:3300/0 conn(0x7f12fc0d14d0 0x7f12fc0cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-28T06:08:34.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.073+0000 7f12fbe00640 1 --2- 172.21.15.96:0/2521415351 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f12fc15e870 0x7f12fc0d0f90 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f12ec0097a0 tx=0x7f12ec02a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=977b07033e293870 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:34.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.073+0000 7f12fb400640 1 -- 172.21.15.96:0/2521415351 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12ec037070 con 0x7f12fc15e870 2024-05-28T06:08:34.075 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.073+0000 7f12fb400640 1 -- 172.21.15.96:0/2521415351 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12ec032440 con 0x7f12fc15e870 2024-05-28T06:08:34.075 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.073+0000 7f12fb400640 1 -- 172.21.15.96:0/2521415351 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12ec032810 con 0x7f12fc15e870 2024-05-28T06:08:34.075 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.074+0000 7f1301800640 1 -- 172.21.15.96:0/2521415351 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f12fc15e870 msgr2=0x7f12fc0d0f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:34.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.074+0000 7f1301800640 1 --2- 172.21.15.96:0/2521415351 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f12fc15e870 0x7f12fc0d0f90 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f12ec0097a0 tx=0x7f12ec02a180 comp rx=0 tx=0).stop 2024-05-28T06:08:34.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.074+0000 7f1301800640 1 -- 172.21.15.96:0/2521415351 shutdown_connections 2024-05-28T06:08:34.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.074+0000 7f1301800640 1 --2- 172.21.15.96:0/2521415351 >> v2:172.21.15.123:3300/0 conn(0x7f12fc0d14d0 0x7f12fc0cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:34.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.074+0000 7f1301800640 1 --2- 172.21.15.96:0/2521415351 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f12fc15e870 0x7f12fc0d0f90 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:34.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.074+0000 7f1301800640 1 -- 172.21.15.96:0/2521415351 >> 172.21.15.96:0/2521415351 conn(0x7f12fc08bfc0 msgr2=0x7f12fc0d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:34.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.074+0000 7f1301800640 1 -- 172.21.15.96:0/2521415351 shutdown_connections 2024-05-28T06:08:34.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.074+0000 7f1301800640 1 -- 172.21.15.96:0/2521415351 wait complete. 2024-05-28T06:08:34.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.076+0000 7f1301800640 1 Processor -- start 2024-05-28T06:08:34.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.076+0000 7f1301800640 1 -- start start 2024-05-28T06:08:34.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.077+0000 7f1301800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f12fc0cf870 0x7f12fc0b70e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:34.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.077+0000 7f1301800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f12fc0d14d0 0x7f12fc0ca200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:34.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.077+0000 7f1301800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f12fc15e870 0x7f12fc14b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:34.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.077+0000 7f1301800640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f12fc0d0820 con 0x7f12fc0d14d0 2024-05-28T06:08:34.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.077+0000 7f1301800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f12fc0d0540 con 0x7f12fc0cf870 2024-05-28T06:08:34.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.077+0000 7f1301800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f12fc0d4fb0 con 0x7f12fc15e870 2024-05-28T06:08:34.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.078+0000 7f12f3e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f12fc0d14d0 0x7f12fc0ca200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:34.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.078+0000 7f12f3e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f12fc0d14d0 0x7f12fc0ca200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35842/0 (socket says 172.21.15.96:35842) 2024-05-28T06:08:34.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.078+0000 7f12f3e00640 1 -- 172.21.15.96:0/2938185521 learned_addr learned my addr 172.21.15.96:0/2938185521 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:08:34.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.078+0000 7f12fbe00640 1 --2- 172.21.15.96:0/2938185521 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f12fc0cf870 0x7f12fc0b70e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:34.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.078+0000 7f1300e00640 1 --2- 172.21.15.96:0/2938185521 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f12fc15e870 0x7f12fc14b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:34.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.078+0000 7f12f3e00640 1 -- 172.21.15.96:0/2938185521 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f12fc15e870 msgr2=0x7f12fc14b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:34.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.078+0000 7f12f3e00640 1 --2- 172.21.15.96:0/2938185521 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f12fc15e870 0x7f12fc14b820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:34.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.078+0000 7f12f3e00640 1 -- 172.21.15.96:0/2938185521 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f12fc0cf870 msgr2=0x7f12fc0b70e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:34.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.078+0000 7f12f3e00640 1 --2- 172.21.15.96:0/2938185521 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f12fc0cf870 0x7f12fc0b70e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:34.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.078+0000 7f12f3e00640 1 -- 172.21.15.96:0/2938185521 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12fc14c270 con 0x7f12fc0d14d0 2024-05-28T06:08:34.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.079+0000 7f12f3e00640 1 --2- 172.21.15.96:0/2938185521 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f12fc0d14d0 0x7f12fc0ca200 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f12e400ebe0 tx=0x7f12e400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:34.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.079+0000 7f12f8c00640 1 -- 172.21.15.96:0/2938185521 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12e400cc30 con 0x7f12fc0d14d0 2024-05-28T06:08:34.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.079+0000 7f12f8c00640 1 -- 172.21.15.96:0/2938185521 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12e400cdd0 con 0x7f12fc0d14d0 2024-05-28T06:08:34.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.079+0000 7f1301800640 1 -- 172.21.15.96:0/2938185521 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12fc14c550 con 0x7f12fc0d14d0 2024-05-28T06:08:34.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.079+0000 7f12f8c00640 1 -- 172.21.15.96:0/2938185521 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12e4005bb0 con 0x7f12fc0d14d0 2024-05-28T06:08:34.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.079+0000 7f1301800640 1 -- 172.21.15.96:0/2938185521 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12fc160c50 con 0x7f12fc0d14d0 2024-05-28T06:08:34.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.080+0000 7f1301800640 1 -- 172.21.15.96:0/2938185521 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12fc0689e0 con 0x7f12fc0d14d0 2024-05-28T06:08:34.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.082+0000 7f12f8c00640 1 -- 172.21.15.96:0/2938185521 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 8) ==== 91193+0+0 (secure 0 0 0) 0x7f12e4006030 con 0x7f12fc0d14d0 2024-05-28T06:08:34.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.082+0000 7f12f8c00640 1 --2- 172.21.15.96:0/2938185521 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f12d806caa0 0x7f12d806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:34.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.082+0000 7f12f8c00640 1 -- 172.21.15.96:0/2938185521 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f12e408c3b0 con 0x7f12fc0d14d0 2024-05-28T06:08:34.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.083+0000 7f12fbe00640 1 --2- 172.21.15.96:0/2938185521 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f12d806caa0 0x7f12d806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:34.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.084+0000 7f12fbe00640 1 --2- 172.21.15.96:0/2938185521 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f12d806caa0 0x7f12d806ef60 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f12ec034000 tx=0x7f12ec02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:34.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.087+0000 7f12f8c00640 1 -- 172.21.15.96:0/2938185521 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f12e4017020 con 0x7f12fc0d14d0 2024-05-28T06:08:34.290 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.288+0000 7f1301800640 1 -- 172.21.15.96:0/2938185521 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f12fc0ba500 con 0x7f12fc0d14d0 2024-05-28T06:08:34.292 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.291+0000 7f12f8c00640 1 -- 172.21.15.96:0/2938185521 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) ==== 74+0+16681 (secure 0 0 0) 0x7f12e405a550 con 0x7f12fc0d14d0 2024-05-28T06:08:34.293 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:08:34.293 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":33,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:05:31.459066+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-05-28T06:05:27.674327+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-05-28T06:05:28.878204+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:08:34.295 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.293+0000 7f1301800640 1 -- 172.21.15.96:0/2938185521 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f12d806caa0 msgr2=0x7f12d806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:34.296 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.293+0000 7f1301800640 1 --2- 172.21.15.96:0/2938185521 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f12d806caa0 0x7f12d806ef60 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f12ec034000 tx=0x7f12ec02e000 comp rx=0 tx=0).stop 2024-05-28T06:08:34.296 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.294+0000 7f1301800640 1 -- 172.21.15.96:0/2938185521 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f12fc0d14d0 msgr2=0x7f12fc0ca200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:34.296 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.294+0000 7f1301800640 1 --2- 172.21.15.96:0/2938185521 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f12fc0d14d0 0x7f12fc0ca200 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f12e400ebe0 tx=0x7f12e400c520 comp rx=0 tx=0).stop 2024-05-28T06:08:34.296 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.294+0000 7f1301800640 1 -- 172.21.15.96:0/2938185521 shutdown_connections 2024-05-28T06:08:34.296 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.294+0000 7f1301800640 1 --2- 172.21.15.96:0/2938185521 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f12d806caa0 0x7f12d806ef60 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:34.296 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.294+0000 7f1301800640 1 --2- 172.21.15.96:0/2938185521 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f12fc15e870 0x7f12fc14b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:34.296 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.294+0000 7f1301800640 1 --2- 172.21.15.96:0/2938185521 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f12fc0d14d0 0x7f12fc0ca200 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:34.296 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.294+0000 7f1301800640 1 --2- 172.21.15.96:0/2938185521 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f12fc0cf870 0x7f12fc0b70e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:34.296 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.294+0000 7f1301800640 1 -- 172.21.15.96:0/2938185521 >> 172.21.15.96:0/2938185521 conn(0x7f12fc08bfc0 msgr2=0x7f12fc0d0b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:34.296 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.294+0000 7f1301800640 1 -- 172.21.15.96:0/2938185521 shutdown_connections 2024-05-28T06:08:34.296 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.294+0000 7f1301800640 1 -- 172.21.15.96:0/2938185521 wait complete. 2024-05-28T06:08:34.313 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-28T06:03:59.517102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-28T06:05:27.674327+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '32', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-05-28T06:05:28.878204+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-28T06:08:34.313 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-28T06:08:34.508 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.506+0000 7fbf92400640 1 Processor -- start 2024-05-28T06:08:34.508 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.506+0000 7fbf92400640 1 -- start start 2024-05-28T06:08:34.509 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.508+0000 7fbf92400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fbf8c06fb70 0x7fbf8c0b39f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:34.510 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.508+0000 7fbf92400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbf8c06c010 0x7fbf8c0cf100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:34.510 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.508+0000 7fbf92400640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf8c06edf0 con 0x7fbf8c15e600 2024-05-28T06:08:34.510 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.508+0000 7fbf92400640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fbf8c06f090 con 0x7fbf8c06fb70 2024-05-28T06:08:34.510 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.508+0000 7fbf92400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fbf8c06f370 con 0x7fbf8c06c010 2024-05-28T06:08:34.510 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.508+0000 7fbf8be00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fbf8c06fb70 0x7fbf8c0b39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:34.510 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.508+0000 7fbf8be00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fbf8c06fb70 0x7fbf8c0b39f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35848/0 (socket says 172.21.15.96:35848) 2024-05-28T06:08:34.511 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.508+0000 7fbf8be00640 1 -- 172.21.15.96:0/4132717117 learned_addr learned my addr 172.21.15.96:0/4132717117 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:08:34.511 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.509+0000 7fbf8be00640 1 -- 172.21.15.96:0/4132717117 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbf8c06c010 msgr2=0x7fbf8c0cf100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:34.511 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.509+0000 7fbf8be00640 1 --2- 172.21.15.96:0/4132717117 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbf8c06c010 0x7fbf8c0cf100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:34.511 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.509+0000 7fbf8be00640 1 -- 172.21.15.96:0/4132717117 >> v1:172.21.15.123:6789/0 conn(0x7fbf8c15e600 legacy=0x7fbf8c06b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:34.511 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.509+0000 7fbf8be00640 1 -- 172.21.15.96:0/4132717117 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf8c06f650 con 0x7fbf8c06fb70 2024-05-28T06:08:34.511 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.510+0000 7fbf8be00640 1 --2- 172.21.15.96:0/4132717117 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf8c06fb70 0x7fbf8c0b39f0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fbf78009870 tx=0x7fbf7802a200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9ef67fe416f1d93 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:34.512 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.510+0000 7fbf8b400640 1 -- 172.21.15.96:0/4132717117 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf78037070 con 0x7fbf8c06fb70 2024-05-28T06:08:34.512 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.510+0000 7fbf8b400640 1 -- 172.21.15.96:0/4132717117 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf78032440 con 0x7fbf8c06fb70 2024-05-28T06:08:34.513 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.510+0000 7fbf8b400640 1 -- 172.21.15.96:0/4132717117 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf78032810 con 0x7fbf8c06fb70 2024-05-28T06:08:34.513 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.511+0000 7fbf92400640 1 -- 172.21.15.96:0/4132717117 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf8c06fb70 msgr2=0x7fbf8c0b39f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:34.513 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.511+0000 7fbf92400640 1 --2- 172.21.15.96:0/4132717117 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf8c06fb70 0x7fbf8c0b39f0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fbf78009870 tx=0x7fbf7802a200 comp rx=0 tx=0).stop 2024-05-28T06:08:34.513 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.511+0000 7fbf92400640 1 -- 172.21.15.96:0/4132717117 shutdown_connections 2024-05-28T06:08:34.513 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.511+0000 7fbf92400640 1 --2- 172.21.15.96:0/4132717117 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbf8c06c010 0x7fbf8c0cf100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:34.513 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.511+0000 7fbf92400640 1 --2- 172.21.15.96:0/4132717117 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf8c06fb70 0x7fbf8c0b39f0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:34.513 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.511+0000 7fbf92400640 1 -- 172.21.15.96:0/4132717117 >> 172.21.15.96:0/4132717117 conn(0x7fbf8c0b2ed0 msgr2=0x7fbf8c0b32e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:34.513 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.511+0000 7fbf92400640 1 -- 172.21.15.96:0/4132717117 shutdown_connections 2024-05-28T06:08:34.513 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.511+0000 7fbf92400640 1 -- 172.21.15.96:0/4132717117 wait complete. 2024-05-28T06:08:34.514 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.513+0000 7fbf92400640 1 Processor -- start 2024-05-28T06:08:34.514 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.513+0000 7fbf92400640 1 -- start start 2024-05-28T06:08:34.517 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.513+0000 7fbf92400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbf8c06c010 0x7fbf8c15d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:34.518 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.514+0000 7fbf92400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbf8c06fb70 0x7fbf8c142be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:34.518 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.514+0000 7fbf92400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf8c15e600 0x7fbf8c1f7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:34.518 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.514+0000 7fbf92400640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fbf8c15e9e0 con 0x7fbf8c15e600 2024-05-28T06:08:34.518 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.514+0000 7fbf92400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fbf8c0d0f50 con 0x7fbf8c06c010 2024-05-28T06:08:34.518 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.514+0000 7fbf92400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fbf8c15eb60 con 0x7fbf8c06fb70 2024-05-28T06:08:34.518 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.514+0000 7fbf91a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf8c15e600 0x7fbf8c1f7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:34.518 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.514+0000 7fbf91a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf8c15e600 0x7fbf8c1f7c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35862/0 (socket says 172.21.15.96:35862) 2024-05-28T06:08:34.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.514+0000 7fbf91a00640 1 -- 172.21.15.96:0/941813238 learned_addr learned my addr 172.21.15.96:0/941813238 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:08:34.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.514+0000 7fbf8be00640 1 --2- 172.21.15.96:0/941813238 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbf8c06fb70 0x7fbf8c142be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:34.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.514+0000 7fbf91000640 1 --2- 172.21.15.96:0/941813238 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbf8c06c010 0x7fbf8c15d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:34.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.514+0000 7fbf91a00640 1 -- 172.21.15.96:0/941813238 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbf8c06fb70 msgr2=0x7fbf8c142be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:34.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.514+0000 7fbf91a00640 1 --2- 172.21.15.96:0/941813238 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbf8c06fb70 0x7fbf8c142be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:34.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.514+0000 7fbf91a00640 1 -- 172.21.15.96:0/941813238 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbf8c06c010 msgr2=0x7fbf8c15d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:34.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.514+0000 7fbf91a00640 1 --2- 172.21.15.96:0/941813238 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbf8c06c010 0x7fbf8c15d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:34.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.514+0000 7fbf91a00640 1 -- 172.21.15.96:0/941813238 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf8c143120 con 0x7fbf8c15e600 2024-05-28T06:08:34.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.515+0000 7fbf91a00640 1 --2- 172.21.15.96:0/941813238 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf8c15e600 0x7fbf8c1f7c20 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fbf8400b9e0 tx=0x7fbf8400bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:34.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.515+0000 7fbf88c00640 1 -- 172.21.15.96:0/941813238 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf84019070 con 0x7fbf8c15e600 2024-05-28T06:08:34.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.515+0000 7fbf88c00640 1 -- 172.21.15.96:0/941813238 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf8400cb40 con 0x7fbf8c15e600 2024-05-28T06:08:34.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.515+0000 7fbf88c00640 1 -- 172.21.15.96:0/941813238 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf84014440 con 0x7fbf8c15e600 2024-05-28T06:08:34.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.515+0000 7fbf92400640 1 -- 172.21.15.96:0/941813238 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf8c15d8e0 con 0x7fbf8c15e600 2024-05-28T06:08:34.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.515+0000 7fbf92400640 1 -- 172.21.15.96:0/941813238 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf8c15c7a0 con 0x7fbf8c15e600 2024-05-28T06:08:34.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.516+0000 7fbf92400640 1 -- 172.21.15.96:0/941813238 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf8c15c5d0 con 0x7fbf8c15e600 2024-05-28T06:08:34.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.517+0000 7fbf88c00640 1 -- 172.21.15.96:0/941813238 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 8) ==== 91193+0+0 (secure 0 0 0) 0x7fbf8401c070 con 0x7fbf8c15e600 2024-05-28T06:08:34.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.518+0000 7fbf88c00640 1 --2- 172.21.15.96:0/941813238 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbf6c06caa0 0x7fbf6c06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:34.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.518+0000 7fbf88c00640 1 -- 172.21.15.96:0/941813238 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7fbf8408bc80 con 0x7fbf8c15e600 2024-05-28T06:08:34.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.519+0000 7fbf91000640 1 --2- 172.21.15.96:0/941813238 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbf6c06caa0 0x7fbf6c06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:34.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.520+0000 7fbf91000640 1 --2- 172.21.15.96:0/941813238 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbf6c06caa0 0x7fbf6c06ef60 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fbf80006fd0 tx=0x7fbf80008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:34.524 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.522+0000 7fbf88c00640 1 -- 172.21.15.96:0/941813238 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbf84059d90 con 0x7fbf8c15e600 2024-05-28T06:08:34.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.591+0000 7fbf88c00640 1 -- 172.21.15.96:0/941813238 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mgrmap(e 9) ==== 91295+0+0 (secure 0 0 0) 0x7fbf84055d00 con 0x7fbf8c15e600 2024-05-28T06:08:34.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.724+0000 7fbf92400640 1 -- 172.21.15.96:0/941813238 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fbf8c0690f0 con 0x7fbf8c15e600 2024-05-28T06:08:34.727 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.725+0000 7fbf88c00640 1 -- 172.21.15.96:0/941813238 <== mon.0 v2:172.21.15.96:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v33) ==== 93+0+10 (secure 0 0 0) 0x7fbf840780c0 con 0x7fbf8c15e600 2024-05-28T06:08:34.727 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 1 2024-05-28T06:08:34.729 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.728+0000 7fbf92400640 1 -- 172.21.15.96:0/941813238 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbf6c06caa0 msgr2=0x7fbf6c06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:34.729 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.728+0000 7fbf92400640 1 --2- 172.21.15.96:0/941813238 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbf6c06caa0 0x7fbf6c06ef60 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fbf80006fd0 tx=0x7fbf80008040 comp rx=0 tx=0).stop 2024-05-28T06:08:34.730 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.728+0000 7fbf92400640 1 -- 172.21.15.96:0/941813238 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf8c15e600 msgr2=0x7fbf8c1f7c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:34.730 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.728+0000 7fbf92400640 1 --2- 172.21.15.96:0/941813238 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf8c15e600 0x7fbf8c1f7c20 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fbf8400b9e0 tx=0x7fbf8400bea0 comp rx=0 tx=0).stop 2024-05-28T06:08:34.730 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.728+0000 7fbf92400640 1 -- 172.21.15.96:0/941813238 shutdown_connections 2024-05-28T06:08:34.731 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.728+0000 7fbf92400640 1 --2- 172.21.15.96:0/941813238 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbf6c06caa0 0x7fbf6c06ef60 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:34.731 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.728+0000 7fbf92400640 1 --2- 172.21.15.96:0/941813238 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf8c15e600 0x7fbf8c1f7c20 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:34.731 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.728+0000 7fbf92400640 1 --2- 172.21.15.96:0/941813238 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbf8c06fb70 0x7fbf8c142be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:34.731 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.728+0000 7fbf92400640 1 --2- 172.21.15.96:0/941813238 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbf8c06c010 0x7fbf8c15d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:34.731 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.728+0000 7fbf92400640 1 -- 172.21.15.96:0/941813238 >> 172.21.15.96:0/941813238 conn(0x7fbf8c0b2ed0 msgr2=0x7fbf8c0d02e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:34.731 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.728+0000 7fbf92400640 1 -- 172.21.15.96:0/941813238 shutdown_connections 2024-05-28T06:08:34.731 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.728+0000 7fbf92400640 1 -- 172.21.15.96:0/941813238 wait complete. 2024-05-28T06:08:34.748 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-28T06:08:34.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.926+0000 7f7178c00640 1 Processor -- start 2024-05-28T06:08:34.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.926+0000 7f7178c00640 1 -- start start 2024-05-28T06:08:34.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.928+0000 7f7178c00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f7174153510 0x7f71740b4d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:34.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.928+0000 7f7178c00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f717415d900 con 0x7f71740ccb90 2024-05-28T06:08:34.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.928+0000 7f7178c00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71740b3f90 con 0x7f717415d270 2024-05-28T06:08:34.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.928+0000 7f7178c00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f71741538f0 con 0x7f7174153510 2024-05-28T06:08:34.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.928+0000 7f7173400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f7174153510 0x7f71740b4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:34.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.928+0000 7f716aa00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f71740ccb90 0x7f717415cb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:42620/0 (socket says 172.21.15.96:42620) 2024-05-28T06:08:34.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.928+0000 7f716aa00640 1 -- 172.21.15.96:0/2972404699 learned_addr learned my addr 172.21.15.96:0/2972404699 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:08:34.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.928+0000 7f7173400640 1 -- 172.21.15.96:0/2972404699 >> v1:172.21.15.123:6789/0 conn(0x7f717415d270 legacy=0x7f717406e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:34.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.928+0000 7f7172a00640 1 -- 172.21.15.96:0/2972404699 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4118685952 0 0) 0x7f717415d900 con 0x7f71740ccb90 2024-05-28T06:08:34.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.928+0000 7f7173400640 1 -- 172.21.15.96:0/2972404699 >> v1:172.21.15.96:6789/0 conn(0x7f71740ccb90 legacy=0x7f717415cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:34.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.928+0000 7f7173400640 1 -- 172.21.15.96:0/2972404699 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71740b4270 con 0x7f7174153510 2024-05-28T06:08:34.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.929+0000 7f7173400640 1 --2- 172.21.15.96:0/2972404699 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7174153510 0x7f71740b4d50 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f71640097a0 tx=0x7f716402a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85f9580d52246fb9 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:34.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.929+0000 7f7172a00640 1 -- 172.21.15.96:0/2972404699 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7164037070 con 0x7f7174153510 2024-05-28T06:08:34.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.929+0000 7f7172a00640 1 -- 172.21.15.96:0/2972404699 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f716402ae50 con 0x7f7174153510 2024-05-28T06:08:34.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.929+0000 7f7172a00640 1 -- 172.21.15.96:0/2972404699 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7164032660 con 0x7f7174153510 2024-05-28T06:08:34.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.930+0000 7f7178c00640 1 -- 172.21.15.96:0/2972404699 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7174153510 msgr2=0x7f71740b4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:34.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.930+0000 7f7178c00640 1 --2- 172.21.15.96:0/2972404699 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7174153510 0x7f71740b4d50 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f71640097a0 tx=0x7f716402a140 comp rx=0 tx=0).stop 2024-05-28T06:08:34.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.930+0000 7f7178c00640 1 -- 172.21.15.96:0/2972404699 shutdown_connections 2024-05-28T06:08:34.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.930+0000 7f7178c00640 1 --2- 172.21.15.96:0/2972404699 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7174153510 0x7f71740b4d50 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:34.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.930+0000 7f7178c00640 1 -- 172.21.15.96:0/2972404699 >> 172.21.15.96:0/2972404699 conn(0x7f717406b900 msgr2=0x7f717406bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:34.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.930+0000 7f7178c00640 1 -- 172.21.15.96:0/2972404699 shutdown_connections 2024-05-28T06:08:34.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.930+0000 7f7178c00640 1 -- 172.21.15.96:0/2972404699 wait complete. 2024-05-28T06:08:34.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.931+0000 7f7178c00640 1 Processor -- start 2024-05-28T06:08:34.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.932+0000 7f7178c00640 1 -- start start 2024-05-28T06:08:34.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.932+0000 7f7178c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f71740ccb90 0x7f717414ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:34.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.932+0000 7f7178c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7174153510 0x7f71740ad440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:34.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.932+0000 7f7178c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f717415d270 0x7f71740ce260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:34.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.932+0000 7f7178c00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f717415d6a0 con 0x7f7174153510 2024-05-28T06:08:34.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.933+0000 7f7178c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f71740b48c0 con 0x7f71740ccb90 2024-05-28T06:08:34.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.933+0000 7f7178c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f71740ccff0 con 0x7f717415d270 2024-05-28T06:08:34.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.933+0000 7f716aa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7174153510 0x7f71740ad440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:34.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.933+0000 7f716aa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7174153510 0x7f71740ad440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35894/0 (socket says 172.21.15.96:35894) 2024-05-28T06:08:34.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.933+0000 7f716aa00640 1 -- 172.21.15.96:0/2035483632 learned_addr learned my addr 172.21.15.96:0/2035483632 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:08:34.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.933+0000 7f716aa00640 1 -- 172.21.15.96:0/2035483632 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f717415d270 msgr2=0x7f71740ce260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:34.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.933+0000 7f716aa00640 1 --2- 172.21.15.96:0/2035483632 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f717415d270 0x7f71740ce260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:34.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.933+0000 7f716aa00640 1 -- 172.21.15.96:0/2035483632 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f71740ccb90 msgr2=0x7f717414ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:34.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.933+0000 7f716aa00640 1 --2- 172.21.15.96:0/2035483632 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f71740ccb90 0x7f717414ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:34.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.933+0000 7f716aa00640 1 -- 172.21.15.96:0/2035483632 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f717415e500 con 0x7f7174153510 2024-05-28T06:08:34.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.933+0000 7f716aa00640 1 --2- 172.21.15.96:0/2035483632 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7174153510 0x7f71740ad440 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f715c00c4d0 tx=0x7f715c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:34.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.934+0000 7f716be00640 1 -- 172.21.15.96:0/2035483632 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f715c018070 con 0x7f7174153510 2024-05-28T06:08:34.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.934+0000 7f716be00640 1 -- 172.21.15.96:0/2035483632 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f715c007500 con 0x7f7174153510 2024-05-28T06:08:34.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.934+0000 7f716be00640 1 -- 172.21.15.96:0/2035483632 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f715c004a20 con 0x7f7174153510 2024-05-28T06:08:34.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.934+0000 7f7178c00640 1 -- 172.21.15.96:0/2035483632 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f717415e780 con 0x7f7174153510 2024-05-28T06:08:34.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.934+0000 7f7178c00640 1 -- 172.21.15.96:0/2035483632 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f717415ed40 con 0x7f7174153510 2024-05-28T06:08:34.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.935+0000 7f7178c00640 1 -- 172.21.15.96:0/2035483632 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71740689e0 con 0x7f7174153510 2024-05-28T06:08:34.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.936+0000 7f716be00640 1 -- 172.21.15.96:0/2035483632 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 9) ==== 91295+0+0 (secure 0 0 0) 0x7f715c004bc0 con 0x7f7174153510 2024-05-28T06:08:34.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.936+0000 7f716be00640 1 --2- 172.21.15.96:0/2035483632 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f715006cbf0 0x7f715006f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:34.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.937+0000 7f716be00640 1 -- 172.21.15.96:0/2035483632 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f715c08b270 con 0x7f7174153510 2024-05-28T06:08:34.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.937+0000 7f7173400640 1 --2- 172.21.15.96:0/2035483632 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f715006cbf0 0x7f715006f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:34.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.938+0000 7f7173400640 1 --2- 172.21.15.96:0/2035483632 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f715006cbf0 0x7f715006f0b0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f7164034000 tx=0x7f716402c000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:34.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:34.942+0000 7f716be00640 1 -- 172.21.15.96:0/2035483632 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f717415e500 con 0x7f7174153510 2024-05-28T06:08:35.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.135+0000 7f7178c00640 1 -- 172.21.15.96:0/2035483632 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f71740b3c40 con 0x7f7174153510 2024-05-28T06:08:35.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.136+0000 7f716be00640 1 -- 172.21.15.96:0/2035483632 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v33) ==== 104+0+11 (secure 0 0 0) 0x7f715c0592c0 con 0x7f7174153510 2024-05-28T06:08:35.136 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 32 2024-05-28T06:08:35.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.138+0000 7f7178c00640 1 -- 172.21.15.96:0/2035483632 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f715006cbf0 msgr2=0x7f715006f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:35.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.138+0000 7f7178c00640 1 --2- 172.21.15.96:0/2035483632 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f715006cbf0 0x7f715006f0b0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f7164034000 tx=0x7f716402c000 comp rx=0 tx=0).stop 2024-05-28T06:08:35.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.138+0000 7f7178c00640 1 -- 172.21.15.96:0/2035483632 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7174153510 msgr2=0x7f71740ad440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:35.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.138+0000 7f7178c00640 1 --2- 172.21.15.96:0/2035483632 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7174153510 0x7f71740ad440 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f715c00c4d0 tx=0x7f715c00c990 comp rx=0 tx=0).stop 2024-05-28T06:08:35.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.138+0000 7f7178c00640 1 -- 172.21.15.96:0/2035483632 shutdown_connections 2024-05-28T06:08:35.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.138+0000 7f7178c00640 1 --2- 172.21.15.96:0/2035483632 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f715006cbf0 0x7f715006f0b0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:35.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.138+0000 7f7178c00640 1 --2- 172.21.15.96:0/2035483632 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f717415d270 0x7f71740ce260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:35.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.138+0000 7f7178c00640 1 --2- 172.21.15.96:0/2035483632 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7174153510 0x7f71740ad440 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:35.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.138+0000 7f7178c00640 1 --2- 172.21.15.96:0/2035483632 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f71740ccb90 0x7f717414ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:35.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.138+0000 7f7178c00640 1 -- 172.21.15.96:0/2035483632 >> 172.21.15.96:0/2035483632 conn(0x7f717406b900 msgr2=0x7f717415b430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:35.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.139+0000 7f7178c00640 1 -- 172.21.15.96:0/2035483632 shutdown_connections 2024-05-28T06:08:35.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.139+0000 7f7178c00640 1 -- 172.21.15.96:0/2035483632 wait complete. 2024-05-28T06:08:35.158 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-28T06:08:35.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.359+0000 7f218b200640 1 Processor -- start 2024-05-28T06:08:35.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.360+0000 7f218b200640 1 -- start start 2024-05-28T06:08:35.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.361+0000 7f218b200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f21840d16a0 0x7f21840cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:35.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.361+0000 7f218b200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21840cfce0 con 0x7f21840cf870 2024-05-28T06:08:35.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.361+0000 7f218b200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f218415ec50 con 0x7f218415e870 2024-05-28T06:08:35.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.361+0000 7f218b200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f21840d2150 con 0x7f21840d16a0 2024-05-28T06:08:35.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.362+0000 7f218a800640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f21840cf870 0x7f21840d25e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:42628/0 (socket says 172.21.15.96:42628) 2024-05-28T06:08:35.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.362+0000 7f218a800640 1 -- 172.21.15.96:0/843192031 learned_addr learned my addr 172.21.15.96:0/843192031 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:08:35.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.362+0000 7f2189400640 1 --2- 172.21.15.96:0/843192031 >> v2:172.21.15.123:3300/0 conn(0x7f21840d16a0 0x7f21840cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:35.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.362+0000 7f2188a00640 1 -- 172.21.15.96:0/843192031 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3822373813 0 0) 0x7f21840cfce0 con 0x7f21840cf870 2024-05-28T06:08:35.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.362+0000 7f2188a00640 1 -- 172.21.15.96:0/843192031 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2170003520 con 0x7f21840cf870 2024-05-28T06:08:35.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.363+0000 7f2188a00640 1 -- 172.21.15.96:0/843192031 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 538624726 0 0) 0x7f2170003520 con 0x7f21840cf870 2024-05-28T06:08:35.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.363+0000 7f2188a00640 1 -- 172.21.15.96:0/843192031 >> v2:172.21.15.123:3300/0 conn(0x7f21840d16a0 msgr2=0x7f21840cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:35.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.363+0000 7f2188a00640 1 --2- 172.21.15.96:0/843192031 >> v2:172.21.15.123:3300/0 conn(0x7f21840d16a0 0x7f21840cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:35.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.363+0000 7f2188a00640 1 -- 172.21.15.96:0/843192031 >> v1:172.21.15.123:6789/0 conn(0x7f218415e870 legacy=0x7f21840d0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:35.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.363+0000 7f2188a00640 1 -- 172.21.15.96:0/843192031 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21840cff80 con 0x7f21840cf870 2024-05-28T06:08:35.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.363+0000 7f2188a00640 1 -- 172.21.15.96:0/843192031 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f2180001670 con 0x7f21840cf870 2024-05-28T06:08:35.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.363+0000 7f2188a00640 1 -- 172.21.15.96:0/843192031 >> v1:172.21.15.96:6789/0 conn(0x7f21840cf870 legacy=0x7f21840d25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:35.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.363+0000 7f2188a00640 1 --2- 172.21.15.96:0/843192031 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f21700039b0 0x7f2170003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:35.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.363+0000 7f2188a00640 1 --2- 172.21.15.96:0/843192031 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2170004800 0x7f2170004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:35.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.363+0000 7f2188a00640 1 --2- 172.21.15.96:0/843192031 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f21700051c0 0x7f217000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:35.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.363+0000 7f2188a00640 1 -- 172.21.15.96:0/843192031 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f21840d2150 con 0x7f21700051c0 2024-05-28T06:08:35.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.363+0000 7f2188a00640 1 -- 172.21.15.96:0/843192031 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f217000c060 con 0x7f2170004800 2024-05-28T06:08:35.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.363+0000 7f2188a00640 1 -- 172.21.15.96:0/843192031 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f217000c260 con 0x7f21700039b0 2024-05-28T06:08:35.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.363+0000 7f218a800640 1 --2- 172.21.15.96:0/843192031 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f21700051c0 0x7f217000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:35.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.363+0000 7f2189e00640 1 --2- 172.21.15.96:0/843192031 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f21700039b0 0x7f2170003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:35.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.364+0000 7f218a800640 1 -- 172.21.15.96:0/843192031 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f21700039b0 msgr2=0x7f2170003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:35.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.364+0000 7f218a800640 1 --2- 172.21.15.96:0/843192031 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f21700039b0 0x7f2170003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:35.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.364+0000 7f218a800640 1 -- 172.21.15.96:0/843192031 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2170004800 msgr2=0x7f2170004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:35.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.364+0000 7f218a800640 1 --2- 172.21.15.96:0/843192031 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2170004800 0x7f2170004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:35.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.364+0000 7f218a800640 1 -- 172.21.15.96:0/843192031 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2170003520 con 0x7f21700051c0 2024-05-28T06:08:35.367 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.364+0000 7f218a800640 1 -- 172.21.15.96:0/843192031 reap_dead start 2024-05-28T06:08:35.367 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.365+0000 7f218a800640 1 --2- 172.21.15.96:0/843192031 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f21700051c0 0x7f217000b9c0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f218000c210 tx=0x7f2170005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f5211352c87cc28 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:35.367 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.365+0000 7f2188a00640 1 -- 172.21.15.96:0/843192031 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2180007d30 con 0x7f21700051c0 2024-05-28T06:08:35.367 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.365+0000 7f2188a00640 1 -- 172.21.15.96:0/843192031 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2180005ad0 con 0x7f21700051c0 2024-05-28T06:08:35.367 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.365+0000 7f2188a00640 1 -- 172.21.15.96:0/843192031 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2180007070 con 0x7f21700051c0 2024-05-28T06:08:35.368 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.365+0000 7f218b200640 1 -- 172.21.15.96:0/843192031 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f21700051c0 msgr2=0x7f217000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:35.368 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.365+0000 7f218b200640 1 --2- 172.21.15.96:0/843192031 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f21700051c0 0x7f217000b9c0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f218000c210 tx=0x7f2170005830 comp rx=0 tx=0).stop 2024-05-28T06:08:35.368 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.365+0000 7f218b200640 1 -- 172.21.15.96:0/843192031 shutdown_connections 2024-05-28T06:08:35.368 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.365+0000 7f218b200640 1 --2- 172.21.15.96:0/843192031 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f21700051c0 0x7f217000b9c0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:35.368 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.365+0000 7f218b200640 1 -- 172.21.15.96:0/843192031 >> 172.21.15.96:0/843192031 conn(0x7f218408bfc0 msgr2=0x7f21840d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:35.368 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.366+0000 7f218b200640 1 -- 172.21.15.96:0/843192031 shutdown_connections 2024-05-28T06:08:35.368 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.366+0000 7f218b200640 1 -- 172.21.15.96:0/843192031 wait complete. 2024-05-28T06:08:35.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.367+0000 7f218b200640 1 Processor -- start 2024-05-28T06:08:35.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.367+0000 7f218b200640 1 -- start start 2024-05-28T06:08:35.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.368+0000 7f218b200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f21700051c0 0x7f21840ce0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:35.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.368+0000 7f218b200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f21840b7dd0 0x7f21841f47d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:35.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.368+0000 7f218b200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f21841597d0 0x7f21840b96d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:35.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.368+0000 7f218b200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f21840d0820 con 0x7f21841597d0 2024-05-28T06:08:35.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.368+0000 7f218b200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f21840d0540 con 0x7f21840b7dd0 2024-05-28T06:08:35.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.368+0000 7f218b200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f21840d4fb0 con 0x7f21700051c0 2024-05-28T06:08:35.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.369+0000 7f218a800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f21841597d0 0x7f21840b96d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:35.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.369+0000 7f218a800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f21841597d0 0x7f21840b96d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35912/0 (socket says 172.21.15.96:35912) 2024-05-28T06:08:35.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.369+0000 7f218a800640 1 -- 172.21.15.96:0/1599507675 learned_addr learned my addr 172.21.15.96:0/1599507675 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:08:35.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.369+0000 7f2189e00640 1 --2- 172.21.15.96:0/1599507675 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f21700051c0 0x7f21840ce0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:35.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.369+0000 7f2189400640 1 --2- 172.21.15.96:0/1599507675 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f21840b7dd0 0x7f21841f47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:35.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.369+0000 7f218a800640 1 -- 172.21.15.96:0/1599507675 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f21700051c0 msgr2=0x7f21840ce0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:35.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.369+0000 7f218a800640 1 --2- 172.21.15.96:0/1599507675 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f21700051c0 0x7f21840ce0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:35.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.369+0000 7f218a800640 1 -- 172.21.15.96:0/1599507675 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f21840b7dd0 msgr2=0x7f21841f47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:35.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.369+0000 7f218a800640 1 --2- 172.21.15.96:0/1599507675 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f21840b7dd0 0x7f21841f47d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:35.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.369+0000 7f218a800640 1 -- 172.21.15.96:0/1599507675 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21841f4f70 con 0x7f21841597d0 2024-05-28T06:08:35.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.370+0000 7f218a800640 1 --2- 172.21.15.96:0/1599507675 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f21841597d0 0x7f21840b96d0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f218000c210 tx=0x7f2180001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:35.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.370+0000 7f217a000640 1 -- 172.21.15.96:0/1599507675 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2180041020 con 0x7f21841597d0 2024-05-28T06:08:35.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.370+0000 7f217a000640 1 -- 172.21.15.96:0/1599507675 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2180000ed0 con 0x7f21841597d0 2024-05-28T06:08:35.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.370+0000 7f218b200640 1 -- 172.21.15.96:0/1599507675 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21841f5250 con 0x7f21841597d0 2024-05-28T06:08:35.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.370+0000 7f217a000640 1 -- 172.21.15.96:0/1599507675 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2180007290 con 0x7f21841597d0 2024-05-28T06:08:35.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.370+0000 7f218b200640 1 -- 172.21.15.96:0/1599507675 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f218415aaa0 con 0x7f21841597d0 2024-05-28T06:08:35.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.371+0000 7f218b200640 1 -- 172.21.15.96:0/1599507675 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21840689e0 con 0x7f21841597d0 2024-05-28T06:08:35.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.372+0000 7f217a000640 1 -- 172.21.15.96:0/1599507675 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 9) ==== 91295+0+0 (secure 0 0 0) 0x7f2180007430 con 0x7f21841597d0 2024-05-28T06:08:35.373 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.373+0000 7f217a000640 1 --2- 172.21.15.96:0/1599507675 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f216806cbc0 0x7f216806f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:35.373 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.373+0000 7f217a000640 1 -- 172.21.15.96:0/1599507675 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f21800abd20 con 0x7f21841597d0 2024-05-28T06:08:35.374 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.374+0000 7f2189e00640 1 --2- 172.21.15.96:0/1599507675 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f216806cbc0 0x7f216806f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:35.375 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.375+0000 7f2189e00640 1 --2- 172.21.15.96:0/1599507675 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f216806cbc0 0x7f216806f080 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f217c007fd0 tx=0x7f217c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:35.377 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.377+0000 7f217a000640 1 -- 172.21.15.96:0/1599507675 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f218004b020 con 0x7f21841597d0 2024-05-28T06:08:35.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.579+0000 7f218b200640 1 -- 172.21.15.96:0/1599507675 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f21840c7e40 con 0x7f21841597d0 2024-05-28T06:08:35.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.580+0000 7f217a000640 1 -- 172.21.15.96:0/1599507675 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v33) ==== 100+0+11 (secure 0 0 0) 0x7f2180079e90 con 0x7f21841597d0 2024-05-28T06:08:35.580 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 64 2024-05-28T06:08:35.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.582+0000 7f218b200640 1 -- 172.21.15.96:0/1599507675 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f216806cbc0 msgr2=0x7f216806f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:35.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.582+0000 7f218b200640 1 --2- 172.21.15.96:0/1599507675 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f216806cbc0 0x7f216806f080 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f217c007fd0 tx=0x7f217c009040 comp rx=0 tx=0).stop 2024-05-28T06:08:35.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.582+0000 7f218b200640 1 -- 172.21.15.96:0/1599507675 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f21841597d0 msgr2=0x7f21840b96d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:35.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.582+0000 7f218b200640 1 --2- 172.21.15.96:0/1599507675 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f21841597d0 0x7f21840b96d0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f218000c210 tx=0x7f2180001a40 comp rx=0 tx=0).stop 2024-05-28T06:08:35.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.583+0000 7f218b200640 1 -- 172.21.15.96:0/1599507675 shutdown_connections 2024-05-28T06:08:35.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.583+0000 7f218b200640 1 --2- 172.21.15.96:0/1599507675 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f216806cbc0 0x7f216806f080 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:35.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.583+0000 7f218b200640 1 --2- 172.21.15.96:0/1599507675 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f21841597d0 0x7f21840b96d0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:35.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.583+0000 7f218b200640 1 --2- 172.21.15.96:0/1599507675 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f21840b7dd0 0x7f21841f47d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:35.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.583+0000 7f218b200640 1 --2- 172.21.15.96:0/1599507675 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f21700051c0 0x7f21840ce0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:35.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.583+0000 7f218b200640 1 -- 172.21.15.96:0/1599507675 >> 172.21.15.96:0/1599507675 conn(0x7f218408bfc0 msgr2=0x7f21840b10f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:35.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.583+0000 7f218b200640 1 -- 172.21.15.96:0/1599507675 shutdown_connections 2024-05-28T06:08:35.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.583+0000 7f218b200640 1 -- 172.21.15.96:0/1599507675 wait complete. 2024-05-28T06:08:35.603 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-28T06:08:35.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.804+0000 7f5069e00640 1 Processor -- start 2024-05-28T06:08:35.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.804+0000 7f5069e00640 1 -- start start 2024-05-28T06:08:35.807 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.806+0000 7f5069e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f50640d16a0 0x7f50640cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:35.807 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.806+0000 7f5069e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f50640cf870 0x7f50640d25e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:35.807 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.806+0000 7f5069e00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f506415ec50 con 0x7f506415e870 2024-05-28T06:08:35.807 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.806+0000 7f5069e00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f50640d2150 con 0x7f50640cf870 2024-05-28T06:08:35.807 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.806+0000 7f5069e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f50640cfd20 con 0x7f50640d16a0 2024-05-28T06:08:35.807 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.806+0000 7f5069400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f50640cf870 0x7f50640d25e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:35.808 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.806+0000 7f5069400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f50640cf870 0x7f50640d25e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35920/0 (socket says 172.21.15.96:35920) 2024-05-28T06:08:35.808 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.806+0000 7f5069400640 1 -- 172.21.15.96:0/3124504088 learned_addr learned my addr 172.21.15.96:0/3124504088 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:08:35.808 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.807+0000 7f5063e00640 1 --2- 172.21.15.96:0/3124504088 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f50640d16a0 0x7f50640cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:35.808 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.807+0000 7f5069400640 1 -- 172.21.15.96:0/3124504088 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f50640d16a0 msgr2=0x7f50640cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:35.808 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.807+0000 7f5069400640 1 --2- 172.21.15.96:0/3124504088 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f50640d16a0 0x7f50640cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:35.808 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.807+0000 7f5069400640 1 -- 172.21.15.96:0/3124504088 >> v1:172.21.15.123:6789/0 conn(0x7f506415e870 legacy=0x7f50640d0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:35.808 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.807+0000 7f5069400640 1 -- 172.21.15.96:0/3124504088 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50640cff80 con 0x7f50640cf870 2024-05-28T06:08:35.808 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.807+0000 7f5063e00640 1 --2- 172.21.15.96:0/3124504088 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f50640d16a0 0x7f50640cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:08:35.809 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.808+0000 7f5069400640 1 --2- 172.21.15.96:0/3124504088 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f50640cf870 0x7f50640d25e0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f505c009870 tx=0x7f505c02a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b820b912782cc0e8 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:35.809 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.808+0000 7f5063400640 1 -- 172.21.15.96:0/3124504088 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f505c037070 con 0x7f50640cf870 2024-05-28T06:08:35.809 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.808+0000 7f5063400640 1 -- 172.21.15.96:0/3124504088 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f505c032440 con 0x7f50640cf870 2024-05-28T06:08:35.809 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.808+0000 7f5063400640 1 -- 172.21.15.96:0/3124504088 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f505c032810 con 0x7f50640cf870 2024-05-28T06:08:35.810 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.809+0000 7f5069e00640 1 -- 172.21.15.96:0/3124504088 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f50640cf870 msgr2=0x7f50640d25e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:35.810 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.809+0000 7f5069e00640 1 --2- 172.21.15.96:0/3124504088 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f50640cf870 0x7f50640d25e0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f505c009870 tx=0x7f505c02a210 comp rx=0 tx=0).stop 2024-05-28T06:08:35.810 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.809+0000 7f5069e00640 1 -- 172.21.15.96:0/3124504088 shutdown_connections 2024-05-28T06:08:35.810 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.809+0000 7f5069e00640 1 --2- 172.21.15.96:0/3124504088 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f50640cf870 0x7f50640d25e0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:35.810 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.809+0000 7f5069e00640 1 --2- 172.21.15.96:0/3124504088 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f50640d16a0 0x7f50640cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:35.810 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.809+0000 7f5069e00640 1 -- 172.21.15.96:0/3124504088 >> 172.21.15.96:0/3124504088 conn(0x7f506408bfc0 msgr2=0x7f50640d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:35.810 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.809+0000 7f5069e00640 1 -- 172.21.15.96:0/3124504088 shutdown_connections 2024-05-28T06:08:35.810 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.809+0000 7f5069e00640 1 -- 172.21.15.96:0/3124504088 wait complete. 2024-05-28T06:08:35.811 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.811+0000 7f5069e00640 1 Processor -- start 2024-05-28T06:08:35.811 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.811+0000 7f5069e00640 1 -- start start 2024-05-28T06:08:35.813 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.813+0000 7f5069e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50640cf870 0x7f5064160ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:35.813 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.813+0000 7f5069e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f50640d16a0 0x7f5064161530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:35.813 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.813+0000 7f5069e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f506415e870 0x7f506414d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:35.813 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.813+0000 7f5069e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f50640d0820 con 0x7f50640d16a0 2024-05-28T06:08:35.813 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.813+0000 7f5069e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f50640d0540 con 0x7f50640cf870 2024-05-28T06:08:35.814 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.813+0000 7f5069e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f50640d4fb0 con 0x7f506415e870 2024-05-28T06:08:35.814 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.813+0000 7f5063e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f50640d16a0 0x7f5064161530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:35.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.813+0000 7f5063e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f50640d16a0 0x7f5064161530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35934/0 (socket says 172.21.15.96:35934) 2024-05-28T06:08:35.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.813+0000 7f5063e00640 1 -- 172.21.15.96:0/2255256941 learned_addr learned my addr 172.21.15.96:0/2255256941 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:08:35.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.814+0000 7f5068a00640 1 --2- 172.21.15.96:0/2255256941 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50640cf870 0x7f5064160ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:35.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.814+0000 7f5063e00640 1 -- 172.21.15.96:0/2255256941 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f506415e870 msgr2=0x7f506414d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:35.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.814+0000 7f5063e00640 1 --2- 172.21.15.96:0/2255256941 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f506415e870 0x7f506414d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:35.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.814+0000 7f5063e00640 1 -- 172.21.15.96:0/2255256941 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50640cf870 msgr2=0x7f5064160ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:35.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.814+0000 7f5063e00640 1 --2- 172.21.15.96:0/2255256941 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50640cf870 0x7f5064160ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:35.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.814+0000 7f5063e00640 1 -- 172.21.15.96:0/2255256941 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50640b7480 con 0x7f50640d16a0 2024-05-28T06:08:35.816 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.815+0000 7f5063e00640 1 --2- 172.21.15.96:0/2255256941 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f50640d16a0 0x7f5064161530 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f505000c7f0 tx=0x7f505000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:35.816 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.815+0000 7f5060c00640 1 -- 172.21.15.96:0/2255256941 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5050005b90 con 0x7f50640d16a0 2024-05-28T06:08:35.816 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.815+0000 7f5060c00640 1 -- 172.21.15.96:0/2255256941 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5050005d30 con 0x7f50640d16a0 2024-05-28T06:08:35.816 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.815+0000 7f5060c00640 1 -- 172.21.15.96:0/2255256941 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5050007a30 con 0x7f50640d16a0 2024-05-28T06:08:35.816 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.815+0000 7f5069e00640 1 -- 172.21.15.96:0/2255256941 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50640b7720 con 0x7f50640d16a0 2024-05-28T06:08:35.816 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.816+0000 7f5069e00640 1 -- 172.21.15.96:0/2255256941 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50640c9c40 con 0x7f50640d16a0 2024-05-28T06:08:35.818 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.817+0000 7f5069e00640 1 -- 172.21.15.96:0/2255256941 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50640689e0 con 0x7f50640d16a0 2024-05-28T06:08:35.819 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.819+0000 7f5060c00640 1 -- 172.21.15.96:0/2255256941 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 9) ==== 91295+0+0 (secure 0 0 0) 0x7f5050007bd0 con 0x7f50640d16a0 2024-05-28T06:08:35.820 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.820+0000 7f5060c00640 1 --2- 172.21.15.96:0/2255256941 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f504406cbf0 0x7f504406f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:35.820 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.820+0000 7f5060c00640 1 -- 172.21.15.96:0/2255256941 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f505008c170 con 0x7f50640d16a0 2024-05-28T06:08:35.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.824+0000 7f5068a00640 1 --2- 172.21.15.96:0/2255256941 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f504406cbf0 0x7f504406f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:35.826 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.826+0000 7f5060c00640 1 -- 172.21.15.96:0/2255256941 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5050015020 con 0x7f50640d16a0 2024-05-28T06:08:35.827 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:35.827+0000 7f5068a00640 1 --2- 172.21.15.96:0/2255256941 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f504406cbf0 0x7f504406f0b0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f5054009f20 tx=0x7f50540094d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:36.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:36.184+0000 7f5069e00640 1 -- 172.21.15.96:0/2255256941 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f506406e070 con 0x7f50640d16a0 2024-05-28T06:08:36.185 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:36.184+0000 7f5060c00640 1 -- 172.21.15.96:0/2255256941 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v33) ==== 80+0+0 (secure 0 0 0) 0x7f505005a100 con 0x7f50640d16a0 2024-05-28T06:08:36.185 INFO:teuthology.orchestra.run.smithi096.stderr:listed 0 entries 2024-05-28T06:08:36.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:36.188+0000 7f5069e00640 1 -- 172.21.15.96:0/2255256941 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f504406cbf0 msgr2=0x7f504406f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:36.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:36.188+0000 7f5069e00640 1 --2- 172.21.15.96:0/2255256941 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f504406cbf0 0x7f504406f0b0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f5054009f20 tx=0x7f50540094d0 comp rx=0 tx=0).stop 2024-05-28T06:08:36.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:36.188+0000 7f5069e00640 1 -- 172.21.15.96:0/2255256941 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f50640d16a0 msgr2=0x7f5064161530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:36.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:36.188+0000 7f5069e00640 1 --2- 172.21.15.96:0/2255256941 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f50640d16a0 0x7f5064161530 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f505000c7f0 tx=0x7f505000cbf0 comp rx=0 tx=0).stop 2024-05-28T06:08:36.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:36.188+0000 7f5069e00640 1 -- 172.21.15.96:0/2255256941 shutdown_connections 2024-05-28T06:08:36.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:36.188+0000 7f5069e00640 1 --2- 172.21.15.96:0/2255256941 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f504406cbf0 0x7f504406f0b0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:36.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:36.188+0000 7f5069e00640 1 --2- 172.21.15.96:0/2255256941 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f506415e870 0x7f506414d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:36.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:36.188+0000 7f5069e00640 1 --2- 172.21.15.96:0/2255256941 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f50640d16a0 0x7f5064161530 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:36.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:36.188+0000 7f5069e00640 1 --2- 172.21.15.96:0/2255256941 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50640cf870 0x7f5064160ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:36.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:36.188+0000 7f5069e00640 1 -- 172.21.15.96:0/2255256941 >> 172.21.15.96:0/2255256941 conn(0x7f506408bfc0 msgr2=0x7f50640b3010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:36.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:36.189+0000 7f5069e00640 1 -- 172.21.15.96:0/2255256941 shutdown_connections 2024-05-28T06:08:36.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:36.189+0000 7f5069e00640 1 -- 172.21.15.96:0/2255256941 wait complete. 2024-05-28T06:08:36.217 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi096.front.sepia.ceph.com... 2024-05-28T06:08:36.217 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:08:36.217 DEBUG:teuthology.orchestra.run.smithi096:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-05-28T06:08:36.308 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi096.front.sepia.ceph.com 2024-05-28T06:08:36.308 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:08:36.308 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-28T06:08:36.315 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi096.stderr:ceph-fuse[51263]: fuse finished with error 0 and tester_r 0 2024-05-28T06:08:36.512 INFO:tasks.cephfs.mount:Terminating background process 2024-05-28T06:08:36.512 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-05-28T06:08:36.512 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:08:36.566 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052731,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248996k,nr_inodes=812249,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-05-28T06:08:36.567 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi123.front.sepia.ceph.com 2024-05-28T06:08:36.567 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:08:36.567 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-05-28T06:08:36.632 INFO:teuthology.orchestra.run.smithi123.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-05-28T06:08:36.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-28T06:08:36.633 INFO:tasks.cephfs.mount:Terminating background process 2024-05-28T06:08:36.633 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:08:36.633 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:08:36.687 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052731,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248996k,nr_inodes=812249,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-28T06:08:36.688 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi096.front.sepia.ceph.com 2024-05-28T06:08:36.688 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:08:36.688 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-05-28T06:08:36.734 INFO:teuthology.orchestra.run.smithi096.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-05-28T06:08:36.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-28T06:08:36.736 INFO:tasks.cephfs.mount:Terminating background process 2024-05-28T06:08:36.736 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-05-28T06:08:36.736 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:08:36.789 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052731,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248996k,nr_inodes=812249,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-05-28T06:08:36.789 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi123.front.sepia.ceph.com 2024-05-28T06:08:36.789 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:08:36.789 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-05-28T06:08:36.841 INFO:teuthology.orchestra.run.smithi123.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-05-28T06:08:36.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-28T06:08:36.842 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-28T06:08:37.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.083+0000 7f26ff800640 1 Processor -- start 2024-05-28T06:08:37.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.084+0000 7f26ff800640 1 -- start start 2024-05-28T06:08:37.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.085+0000 7f26ff800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f26f806f530 0x7f26f80cf1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:37.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.086+0000 7f26ff800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f26f80d0c20 0x7f26f80cc9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:37.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.086+0000 7f26ff800640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26f80cfb30 con 0x7f26f80af890 2024-05-28T06:08:37.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.086+0000 7f26ff800640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f26f80cfe10 con 0x7f26f80d0c20 2024-05-28T06:08:37.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.086+0000 7f26ff800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f26f80b4680 con 0x7f26f806f530 2024-05-28T06:08:37.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.086+0000 7f26f7e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f26f80af890 0x7f26f80d0510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:42636/0 (socket says 172.21.15.96:42636) 2024-05-28T06:08:37.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.086+0000 7f26f7e00640 1 -- 172.21.15.96:0/2922395563 learned_addr learned my addr 172.21.15.96:0/2922395563 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:08:37.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.087+0000 7f26f7400640 1 -- 172.21.15.96:0/2922395563 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 820233850 0 0) 0x7f26f80cfb30 con 0x7f26f80af890 2024-05-28T06:08:37.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.087+0000 7f26fce00640 1 --2- 172.21.15.96:0/2922395563 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f26f806f530 0x7f26f80cf1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:37.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.087+0000 7f26f7400640 1 -- 172.21.15.96:0/2922395563 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f26e0003520 con 0x7f26f80af890 2024-05-28T06:08:37.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.086+0000 7f26fd800640 1 --2- 172.21.15.96:0/2922395563 >> v2:172.21.15.123:3300/0 conn(0x7f26f80d0c20 0x7f26f80cc9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:37.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.087+0000 7f26f7400640 1 -- 172.21.15.96:0/2922395563 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4290770656 0 0) 0x7f26e0003520 con 0x7f26f80af890 2024-05-28T06:08:37.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.087+0000 7f26f7400640 1 -- 172.21.15.96:0/2922395563 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f26f806f530 msgr2=0x7f26f80cf1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:37.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.087+0000 7f26f7400640 1 --2- 172.21.15.96:0/2922395563 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f26f806f530 0x7f26f80cf1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:37.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.087+0000 7f26f7400640 1 -- 172.21.15.96:0/2922395563 >> v2:172.21.15.123:3300/0 conn(0x7f26f80d0c20 msgr2=0x7f26f80cc9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:37.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.087+0000 7f26f7400640 1 --2- 172.21.15.96:0/2922395563 >> v2:172.21.15.123:3300/0 conn(0x7f26f80d0c20 0x7f26f80cc9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:37.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.087+0000 7f26f7400640 1 -- 172.21.15.96:0/2922395563 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26f80b4960 con 0x7f26f80af890 2024-05-28T06:08:37.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.087+0000 7f26f7400640 1 -- 172.21.15.96:0/2922395563 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f26e4001670 con 0x7f26f80af890 2024-05-28T06:08:37.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.087+0000 7f26f7400640 1 -- 172.21.15.96:0/2922395563 >> v1:172.21.15.96:6789/0 conn(0x7f26f80af890 legacy=0x7f26f80d0510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:37.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.087+0000 7f26f7400640 1 --2- 172.21.15.96:0/2922395563 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26e00039b0 0x7f26e0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:37.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.087+0000 7f26f7400640 1 --2- 172.21.15.96:0/2922395563 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f26e0004800 0x7f26e0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:37.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.087+0000 7f26f7400640 1 --2- 172.21.15.96:0/2922395563 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f26f806f530 0x7f26e00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:37.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.087+0000 7f26f7400640 1 -- 172.21.15.96:0/2922395563 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f26f80cfe10 con 0x7f26e0004800 2024-05-28T06:08:37.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.087+0000 7f26f7400640 1 -- 172.21.15.96:0/2922395563 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f26f80b4680 con 0x7f26e00039b0 2024-05-28T06:08:37.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.087+0000 7f26f7400640 1 -- 172.21.15.96:0/2922395563 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f26e0007ac0 con 0x7f26f806f530 2024-05-28T06:08:37.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.087+0000 7f26fd800640 1 --2- 172.21.15.96:0/2922395563 >> v2:172.21.15.123:3300/0 conn(0x7f26f80d0c20 0x7f26f80cc9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:08:37.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.088+0000 7f26fce00640 1 --2- 172.21.15.96:0/2922395563 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26e00039b0 0x7f26e0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:37.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.088+0000 7f26fd800640 1 --2- 172.21.15.96:0/2922395563 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f26f806f530 0x7f26e00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:37.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.089+0000 7f26fce00640 1 -- 172.21.15.96:0/2922395563 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f26f806f530 msgr2=0x7f26e00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:37.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.089+0000 7f26fce00640 1 --2- 172.21.15.96:0/2922395563 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f26f806f530 0x7f26e00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:37.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.089+0000 7f26fce00640 1 -- 172.21.15.96:0/2922395563 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f26e0004800 msgr2=0x7f26e0004c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-28T06:08:37.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.089+0000 7f26f7e00640 1 --2- 172.21.15.96:0/2922395563 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f26e0004800 0x7f26e0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:37.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.090+0000 7f26fce00640 1 --2- 172.21.15.96:0/2922395563 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f26e0004800 0x7f26e0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:37.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.090+0000 7f26fce00640 1 -- 172.21.15.96:0/2922395563 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26e0003520 con 0x7f26e00039b0 2024-05-28T06:08:37.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.092+0000 7f26f7e00640 1 --2- 172.21.15.96:0/2922395563 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f26e0004800 0x7f26e0004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:08:37.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.092+0000 7f26fce00640 1 --2- 172.21.15.96:0/2922395563 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26e00039b0 0x7f26e0003dd0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f26ec009ac0 tx=0x7f26e0005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ccc2467a1a3c2fc9 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:37.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.092+0000 7f26f7400640 1 -- 172.21.15.96:0/2922395563 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26ec036070 con 0x7f26e00039b0 2024-05-28T06:08:37.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.092+0000 7f26f7400640 1 -- 172.21.15.96:0/2922395563 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26ec031400 con 0x7f26e00039b0 2024-05-28T06:08:37.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.092+0000 7f26f7400640 1 -- 172.21.15.96:0/2922395563 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26ec009690 con 0x7f26e00039b0 2024-05-28T06:08:37.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.093+0000 7f26ff800640 1 -- 172.21.15.96:0/2922395563 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26e00039b0 msgr2=0x7f26e0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:37.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.093+0000 7f26ff800640 1 --2- 172.21.15.96:0/2922395563 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26e00039b0 0x7f26e0003dd0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f26ec009ac0 tx=0x7f26e0005830 comp rx=0 tx=0).stop 2024-05-28T06:08:37.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.093+0000 7f26fd800640 1 -- 172.21.15.96:0/2922395563 reap_dead start 2024-05-28T06:08:37.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.094+0000 7f26ff800640 1 -- 172.21.15.96:0/2922395563 shutdown_connections 2024-05-28T06:08:37.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.094+0000 7f26ff800640 1 -- 172.21.15.96:0/2922395563 >> 172.21.15.96:0/2922395563 conn(0x7f26f806bae0 msgr2=0x7f26f806bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:37.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.094+0000 7f26ff800640 1 -- 172.21.15.96:0/2922395563 shutdown_connections 2024-05-28T06:08:37.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.094+0000 7f26ff800640 1 -- 172.21.15.96:0/2922395563 wait complete. 2024-05-28T06:08:37.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.097+0000 7f26ff800640 1 Processor -- start 2024-05-28T06:08:37.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.097+0000 7f26ff800640 1 -- start start 2024-05-28T06:08:37.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.098+0000 7f26ff800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26e00039b0 0x7f26f815d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:37.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.099+0000 7f26ff800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f26f80cd790 0x7f26f81551e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:37.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.099+0000 7f26ff800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f26f8159760 0x7f26f8159c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:37.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.099+0000 7f26ff800640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f26f806fea0 con 0x7f26f80cd790 2024-05-28T06:08:37.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.099+0000 7f26ff800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f26f806fd20 con 0x7f26e00039b0 2024-05-28T06:08:37.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.099+0000 7f26ff800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f26f80b4e80 con 0x7f26f8159760 2024-05-28T06:08:37.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.099+0000 7f26fce00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26e00039b0 0x7f26f815d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:37.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.099+0000 7f26fce00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26e00039b0 0x7f26f815d1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.96:38820/0 (socket says 172.21.15.96:38820) 2024-05-28T06:08:37.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.099+0000 7f26fce00640 1 -- 172.21.15.96:0/3423136155 learned_addr learned my addr 172.21.15.96:0/3423136155 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:08:37.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.099+0000 7f26f7e00640 1 --2- 172.21.15.96:0/3423136155 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f26f80cd790 0x7f26f81551e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:37.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.099+0000 7f26fd800640 1 --2- 172.21.15.96:0/3423136155 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f26f8159760 0x7f26f8159c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:37.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.100+0000 7f26fce00640 1 -- 172.21.15.96:0/3423136155 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f26f8159760 msgr2=0x7f26f8159c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:37.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.100+0000 7f26fce00640 1 --2- 172.21.15.96:0/3423136155 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f26f8159760 0x7f26f8159c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:37.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.100+0000 7f26fce00640 1 -- 172.21.15.96:0/3423136155 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f26f80cd790 msgr2=0x7f26f81551e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:37.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.100+0000 7f26fce00640 1 --2- 172.21.15.96:0/3423136155 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f26f80cd790 0x7f26f81551e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:37.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.100+0000 7f26fce00640 1 -- 172.21.15.96:0/3423136155 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26f8157fa0 con 0x7f26e00039b0 2024-05-28T06:08:37.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.101+0000 7f26fce00640 1 --2- 172.21.15.96:0/3423136155 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26e00039b0 0x7f26f815d1d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f26ec009be0 tx=0x7f26ec004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:37.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.101+0000 7f26f4c00640 1 -- 172.21.15.96:0/3423136155 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26ec036070 con 0x7f26e00039b0 2024-05-28T06:08:37.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.101+0000 7f26f4c00640 1 -- 172.21.15.96:0/3423136155 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26ec041070 con 0x7f26e00039b0 2024-05-28T06:08:37.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.101+0000 7f26f4c00640 1 -- 172.21.15.96:0/3423136155 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26ec0315b0 con 0x7f26e00039b0 2024-05-28T06:08:37.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.101+0000 7f26ff800640 1 -- 172.21.15.96:0/3423136155 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26f80c3f30 con 0x7f26e00039b0 2024-05-28T06:08:37.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.102+0000 7f26ff800640 1 -- 172.21.15.96:0/3423136155 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26f80c44f0 con 0x7f26e00039b0 2024-05-28T06:08:37.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.107+0000 7f26ff800640 1 -- 172.21.15.96:0/3423136155 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26f80689e0 con 0x7f26e00039b0 2024-05-28T06:08:37.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.108+0000 7f26f4c00640 1 -- 172.21.15.96:0/3423136155 <== mon.1 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 9) ==== 91295+0+0 (secure 0 0 0) 0x7f26ec031770 con 0x7f26e00039b0 2024-05-28T06:08:37.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.109+0000 7f26f4c00640 1 --2- 172.21.15.96:0/3423136155 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f26d806cc40 0x7f26d806f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:37.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.109+0000 7f26f4c00640 1 -- 172.21.15.96:0/3423136155 <== mon.1 v2:172.21.15.123:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f26ec0abcc0 con 0x7f26e00039b0 2024-05-28T06:08:37.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.110+0000 7f26f7e00640 1 --2- 172.21.15.96:0/3423136155 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f26d806cc40 0x7f26d806f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:37.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.111+0000 7f26f7e00640 1 --2- 172.21.15.96:0/3423136155 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f26d806cc40 0x7f26d806f100 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f26e4001820 tx=0x7f26e4001690 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:37.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.118+0000 7f26f4c00640 1 -- 172.21.15.96:0/3423136155 <== mon.1 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f26ec04a020 con 0x7f26e00039b0 2024-05-28T06:08:37.489 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.488+0000 7f26ff800640 1 -- 172.21.15.96:0/3423136155 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f26f80af3f0 con 0x7f26e00039b0 2024-05-28T06:08:37.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.491+0000 7f26f4c00640 1 -- 172.21.15.96:0/3423136155 <== mon.1 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+5178 (secure 0 0 0) 0x7f26ec079ea0 con 0x7f26e00039b0 2024-05-28T06:08:37.492 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:08:37.492 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4580,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6836/918905879","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":918905879},{"type":"v1","addr":"172.21.15.123:6836","nonce":918905879}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":5223,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6834/697317719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6833","nonce":697317719},{"type":"v1","addr":"172.21.15.96:6834","nonce":697317719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":5226,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/1327552826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":1327552826},{"type":"v1","addr":"172.21.15.96:6835","nonce":1327552826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-28T06:05:29.892912+0000","modified":"2024-05-28T06:05:32.976408+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5238":{"gid":5238,"name":"d","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6837/458334664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6835","nonce":458334664},{"type":"v1","addr":"172.21.15.123:6837","nonce":458334664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5238,"qdb_cluster":[5238]},"id":2}]} 2024-05-28T06:08:37.492 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 16 2024-05-28T06:08:37.495 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.495+0000 7f26ff800640 1 -- 172.21.15.96:0/3423136155 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f26d806cc40 msgr2=0x7f26d806f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:37.496 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.495+0000 7f26ff800640 1 --2- 172.21.15.96:0/3423136155 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f26d806cc40 0x7f26d806f100 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f26e4001820 tx=0x7f26e4001690 comp rx=0 tx=0).stop 2024-05-28T06:08:37.496 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.495+0000 7f26ff800640 1 -- 172.21.15.96:0/3423136155 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26e00039b0 msgr2=0x7f26f815d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:37.496 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.495+0000 7f26ff800640 1 --2- 172.21.15.96:0/3423136155 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26e00039b0 0x7f26f815d1d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f26ec009be0 tx=0x7f26ec004730 comp rx=0 tx=0).stop 2024-05-28T06:08:37.496 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.496+0000 7f26ff800640 1 -- 172.21.15.96:0/3423136155 shutdown_connections 2024-05-28T06:08:37.496 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.496+0000 7f26ff800640 1 --2- 172.21.15.96:0/3423136155 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f26d806cc40 0x7f26d806f100 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:37.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.496+0000 7f26ff800640 1 --2- 172.21.15.96:0/3423136155 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f26f8159760 0x7f26f8159c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:37.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.496+0000 7f26ff800640 1 --2- 172.21.15.96:0/3423136155 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f26f80cd790 0x7f26f81551e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:37.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.496+0000 7f26ff800640 1 --2- 172.21.15.96:0/3423136155 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26e00039b0 0x7f26f815d1d0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:37.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.496+0000 7f26ff800640 1 -- 172.21.15.96:0/3423136155 >> 172.21.15.96:0/3423136155 conn(0x7f26f806bae0 msgr2=0x7f26f80cc9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:37.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.496+0000 7f26ff800640 1 -- 172.21.15.96:0/3423136155 shutdown_connections 2024-05-28T06:08:37.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.496+0000 7f26ff800640 1 -- 172.21.15.96:0/3423136155 wait complete. 2024-05-28T06:08:37.522 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:08:37.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.765+0000 7fe7a1200640 1 Processor -- start 2024-05-28T06:08:37.770 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.765+0000 7fe7a1200640 1 -- start start 2024-05-28T06:08:37.770 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.767+0000 7fe7a1200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe79c15f0f0 0x7fe79c0b4320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:37.770 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.767+0000 7fe7a1200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe79c0b73e0 con 0x7fe79c0b6f50 2024-05-28T06:08:37.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.767+0000 7fe7a1200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe79c0b4900 con 0x7fe79c0b59a0 2024-05-28T06:08:37.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.767+0000 7fe7a1200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fe79c0b09f0 con 0x7fe79c15f0f0 2024-05-28T06:08:37.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.767+0000 7fe799e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe79c15f0f0 0x7fe79c0b4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:37.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.767+0000 7fe799400640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7fe79c0b59a0 0x7fe79c15f4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.96:38032/0 (socket says 172.21.15.96:38032) 2024-05-28T06:08:37.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.767+0000 7fe799400640 1 -- 172.21.15.96:0/2262472698 learned_addr learned my addr 172.21.15.96:0/2262472698 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:08:37.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.768+0000 7fe798a00640 1 -- 172.21.15.96:0/2262472698 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2469754436 0 0) 0x7fe79c0b4900 con 0x7fe79c0b59a0 2024-05-28T06:08:37.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.768+0000 7fe798a00640 1 -- 172.21.15.96:0/2262472698 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe780003520 con 0x7fe79c0b59a0 2024-05-28T06:08:37.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.769+0000 7fe798a00640 1 -- 172.21.15.96:0/2262472698 <== mon.1 v1:172.21.15.123:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 367147382 0 0) 0x7fe780003520 con 0x7fe79c0b59a0 2024-05-28T06:08:37.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.769+0000 7fe798a00640 1 -- 172.21.15.96:0/2262472698 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe79c15f0f0 msgr2=0x7fe79c0b4320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:37.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.769+0000 7fe798a00640 1 --2- 172.21.15.96:0/2262472698 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe79c15f0f0 0x7fe79c0b4320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:37.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.769+0000 7fe798a00640 1 -- 172.21.15.96:0/2262472698 >> v1:172.21.15.96:6789/0 conn(0x7fe79c0b6f50 legacy=0x7fe79c06ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:37.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.769+0000 7fe798a00640 1 -- 172.21.15.96:0/2262472698 --> v1:172.21.15.123:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe79c0b4be0 con 0x7fe79c0b59a0 2024-05-28T06:08:37.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.769+0000 7fe798a00640 1 -- 172.21.15.96:0/2262472698 <== mon.1 v1:172.21.15.123:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7fe788001670 con 0x7fe79c0b59a0 2024-05-28T06:08:37.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.769+0000 7fe798a00640 1 -- 172.21.15.96:0/2262472698 >> v1:172.21.15.123:6789/0 conn(0x7fe79c0b59a0 legacy=0x7fe79c15f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:37.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.769+0000 7fe798a00640 1 --2- 172.21.15.96:0/2262472698 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe7800039b0 0x7fe780003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:37.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.769+0000 7fe798a00640 1 --2- 172.21.15.96:0/2262472698 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe780004800 0x7fe780004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:37.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.769+0000 7fe798a00640 1 --2- 172.21.15.96:0/2262472698 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe7800051c0 0x7fe78002d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:37.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.769+0000 7fe798a00640 1 -- 172.21.15.96:0/2262472698 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe79c0b09f0 con 0x7fe7800039b0 2024-05-28T06:08:37.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.769+0000 7fe798a00640 1 -- 172.21.15.96:0/2262472698 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe78002dff0 con 0x7fe780004800 2024-05-28T06:08:37.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.769+0000 7fe798a00640 1 -- 172.21.15.96:0/2262472698 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fe78002e1f0 con 0x7fe7800051c0 2024-05-28T06:08:37.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.769+0000 7fe799400640 1 --2- 172.21.15.96:0/2262472698 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe780004800 0x7fe780004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:37.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.770+0000 7fe79a800640 1 --2- 172.21.15.96:0/2262472698 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe7800051c0 0x7fe78002d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:37.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.770+0000 7fe79a800640 1 -- 172.21.15.96:0/2262472698 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe780004800 msgr2=0x7fe780004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:37.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.770+0000 7fe79a800640 1 --2- 172.21.15.96:0/2262472698 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe780004800 0x7fe780004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:37.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.771+0000 7fe79a800640 1 -- 172.21.15.96:0/2262472698 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe7800039b0 msgr2=0x7fe780003dd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-28T06:08:37.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.771+0000 7fe799e00640 1 --2- 172.21.15.96:0/2262472698 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe7800039b0 0x7fe780003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:37.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.771+0000 7fe79a800640 1 --2- 172.21.15.96:0/2262472698 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe7800039b0 0x7fe780003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:37.774 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.771+0000 7fe79a800640 1 -- 172.21.15.96:0/2262472698 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe780003520 con 0x7fe7800051c0 2024-05-28T06:08:37.774 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.771+0000 7fe799e00640 1 --2- 172.21.15.96:0/2262472698 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe7800039b0 0x7fe780003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:08:37.774 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.772+0000 7fe79a800640 1 --2- 172.21.15.96:0/2262472698 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe7800051c0 0x7fe78002d9d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe794001e50 tx=0x7fe780005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fecfbcfb75c52ab5 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:37.774 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.773+0000 7fe798a00640 1 -- 172.21.15.96:0/2262472698 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe794001a70 con 0x7fe7800051c0 2024-05-28T06:08:37.774 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.773+0000 7fe798a00640 1 -- 172.21.15.96:0/2262472698 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe794003970 con 0x7fe7800051c0 2024-05-28T06:08:37.774 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.773+0000 7fe798a00640 1 -- 172.21.15.96:0/2262472698 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe794003da0 con 0x7fe7800051c0 2024-05-28T06:08:37.775 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.773+0000 7fe7a1200640 1 -- 172.21.15.96:0/2262472698 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe7800051c0 msgr2=0x7fe78002d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:37.775 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.773+0000 7fe7a1200640 1 --2- 172.21.15.96:0/2262472698 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe7800051c0 0x7fe78002d9d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe794001e50 tx=0x7fe780005830 comp rx=0 tx=0).stop 2024-05-28T06:08:37.775 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.773+0000 7fe79a800640 1 -- 172.21.15.96:0/2262472698 reap_dead start 2024-05-28T06:08:37.775 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.773+0000 7fe7a1200640 1 -- 172.21.15.96:0/2262472698 shutdown_connections 2024-05-28T06:08:37.775 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.773+0000 7fe7a1200640 1 -- 172.21.15.96:0/2262472698 >> 172.21.15.96:0/2262472698 conn(0x7fe79c09aa30 msgr2=0x7fe79c0b6840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:37.775 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.774+0000 7fe7a1200640 1 -- 172.21.15.96:0/2262472698 shutdown_connections 2024-05-28T06:08:37.775 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.774+0000 7fe7a1200640 1 -- 172.21.15.96:0/2262472698 wait complete. 2024-05-28T06:08:37.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.775+0000 7fe7a1200640 1 Processor -- start 2024-05-28T06:08:37.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.775+0000 7fe7a1200640 1 -- start start 2024-05-28T06:08:37.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.776+0000 7fe7a1200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe7800051c0 0x7fe79c09be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:37.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.776+0000 7fe7a1200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe79c155ea0 0x7fe79c0aeda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:37.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.776+0000 7fe7a1200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe79c1589a0 0x7fe79c158e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:37.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.776+0000 7fe7a1200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe79c0b51a0 con 0x7fe79c155ea0 2024-05-28T06:08:37.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.776+0000 7fe7a1200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe79c0b5510 con 0x7fe7800051c0 2024-05-28T06:08:37.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.776+0000 7fe7a1200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fe79c15b8a0 con 0x7fe79c1589a0 2024-05-28T06:08:37.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.776+0000 7fe799400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe79c155ea0 0x7fe79c0aeda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:37.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.776+0000 7fe799400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe79c155ea0 0x7fe79c0aeda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35976/0 (socket says 172.21.15.96:35976) 2024-05-28T06:08:37.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.776+0000 7fe799400640 1 -- 172.21.15.96:0/754371487 learned_addr learned my addr 172.21.15.96:0/754371487 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:08:37.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.776+0000 7fe799e00640 1 --2- 172.21.15.96:0/754371487 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe7800051c0 0x7fe79c09be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:37.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.777+0000 7fe79a800640 1 --2- 172.21.15.96:0/754371487 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe79c1589a0 0x7fe79c158e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:37.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.777+0000 7fe799e00640 1 -- 172.21.15.96:0/754371487 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe79c1589a0 msgr2=0x7fe79c158e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:37.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.777+0000 7fe799e00640 1 --2- 172.21.15.96:0/754371487 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe79c1589a0 0x7fe79c158e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:37.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.777+0000 7fe799e00640 1 -- 172.21.15.96:0/754371487 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe79c155ea0 msgr2=0x7fe79c0aeda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:37.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.777+0000 7fe799e00640 1 --2- 172.21.15.96:0/754371487 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe79c155ea0 0x7fe79c0aeda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:37.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.777+0000 7fe799e00640 1 -- 172.21.15.96:0/754371487 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe79c156320 con 0x7fe7800051c0 2024-05-28T06:08:37.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.777+0000 7fe799400640 1 --2- 172.21.15.96:0/754371487 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe79c155ea0 0x7fe79c0aeda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-28T06:08:37.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.777+0000 7fe799e00640 1 --2- 172.21.15.96:0/754371487 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe7800051c0 0x7fe79c09be00 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fe78c010580 tx=0x7fe78c010a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:37.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.777+0000 7fe792000640 1 -- 172.21.15.96:0/754371487 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe78c00f040 con 0x7fe7800051c0 2024-05-28T06:08:37.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.777+0000 7fe792000640 1 -- 172.21.15.96:0/754371487 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe78c014820 con 0x7fe7800051c0 2024-05-28T06:08:37.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.777+0000 7fe7a1200640 1 -- 172.21.15.96:0/754371487 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe79c15aaa0 con 0x7fe7800051c0 2024-05-28T06:08:37.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.777+0000 7fe792000640 1 -- 172.21.15.96:0/754371487 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe78c014c10 con 0x7fe7800051c0 2024-05-28T06:08:37.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.777+0000 7fe7a1200640 1 -- 172.21.15.96:0/754371487 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe79c159fc0 con 0x7fe7800051c0 2024-05-28T06:08:37.781 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.778+0000 7fe7a1200640 1 -- 172.21.15.96:0/754371487 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe79c0689e0 con 0x7fe7800051c0 2024-05-28T06:08:37.781 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.779+0000 7fe792000640 1 -- 172.21.15.96:0/754371487 <== mon.1 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 9) ==== 91295+0+0 (secure 0 0 0) 0x7fe78c01c400 con 0x7fe7800051c0 2024-05-28T06:08:37.781 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.780+0000 7fe792000640 1 --2- 172.21.15.96:0/754371487 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe77c06cbf0 0x7fe77c06f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:37.781 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.780+0000 7fe792000640 1 -- 172.21.15.96:0/754371487 <== mon.1 v2:172.21.15.123:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5765+0+0 (secure 0 0 0) 0x7fe78c08c270 con 0x7fe7800051c0 2024-05-28T06:08:37.781 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.781+0000 7fe799400640 1 --2- 172.21.15.96:0/754371487 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe77c06cbf0 0x7fe77c06f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:37.781 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.781+0000 7fe799400640 1 --2- 172.21.15.96:0/754371487 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe77c06cbf0 0x7fe77c06f0b0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fe7880017e0 tx=0x7fe788001430 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:37.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.785+0000 7fe792000640 1 -- 172.21.15.96:0/754371487 <== mon.1 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe78c018030 con 0x7fe7800051c0 2024-05-28T06:08:37.973 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.972+0000 7fe7a1200640 1 -- 172.21.15.96:0/754371487 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe79c15b050 con 0x7fe7800051c0 2024-05-28T06:08:37.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.976+0000 7fe792000640 1 -- 172.21.15.96:0/754371487 <== mon.1 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) ==== 74+0+16789 (secure 0 0 0) 0x7fe78c059c90 con 0x7fe7800051c0 2024-05-28T06:08:37.977 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:08:37.977 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":34,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:08:37.595959+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-05-28T06:05:27.674327+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-05-28T06:05:28.878204+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":34,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[{"pool":5,"snaps":[{"begin":2,"length":2}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:08:37.979 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.978+0000 7fe7a1200640 1 -- 172.21.15.96:0/754371487 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe77c06cbf0 msgr2=0x7fe77c06f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:37.979 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.978+0000 7fe7a1200640 1 --2- 172.21.15.96:0/754371487 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe77c06cbf0 0x7fe77c06f0b0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fe7880017e0 tx=0x7fe788001430 comp rx=0 tx=0).stop 2024-05-28T06:08:37.979 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.978+0000 7fe7a1200640 1 -- 172.21.15.96:0/754371487 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe7800051c0 msgr2=0x7fe79c09be00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:37.979 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.978+0000 7fe7a1200640 1 --2- 172.21.15.96:0/754371487 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe7800051c0 0x7fe79c09be00 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fe78c010580 tx=0x7fe78c010a40 comp rx=0 tx=0).stop 2024-05-28T06:08:37.980 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.978+0000 7fe7a1200640 1 -- 172.21.15.96:0/754371487 shutdown_connections 2024-05-28T06:08:37.980 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.978+0000 7fe7a1200640 1 --2- 172.21.15.96:0/754371487 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe77c06cbf0 0x7fe77c06f0b0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:37.980 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.978+0000 7fe7a1200640 1 --2- 172.21.15.96:0/754371487 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe79c1589a0 0x7fe79c158e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:37.980 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.978+0000 7fe7a1200640 1 --2- 172.21.15.96:0/754371487 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe79c155ea0 0x7fe79c0aeda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:37.980 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.978+0000 7fe7a1200640 1 --2- 172.21.15.96:0/754371487 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe7800051c0 0x7fe79c09be00 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:37.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.978+0000 7fe7a1200640 1 -- 172.21.15.96:0/754371487 >> 172.21.15.96:0/754371487 conn(0x7fe79c09aa30 msgr2=0x7fe79c06ee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:37.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.978+0000 7fe7a1200640 1 -- 172.21.15.96:0/754371487 shutdown_connections 2024-05-28T06:08:37.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:37.978+0000 7fe7a1200640 1 -- 172.21.15.96:0/754371487 wait complete. 2024-05-28T06:08:37.996 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-28T06:03:59.517102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-28T06:05:27.674327+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '32', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-05-28T06:05:28.878204+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '34', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 34, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-28T06:08:37.997 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-28T06:08:38.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.190+0000 7f3888400640 1 Processor -- start 2024-05-28T06:08:38.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.190+0000 7f3888400640 1 -- start start 2024-05-28T06:08:38.193 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.191+0000 7f3888400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f38800d0640 0x7f38800d0a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:38.193 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.191+0000 7f3888400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f388015b6f0 0x7f388006e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:38.193 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.192+0000 7f3888400640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f38800ade90 con 0x7f38800725e0 2024-05-28T06:08:38.193 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.192+0000 7f3888400640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f3880071c40 con 0x7f38800d0640 2024-05-28T06:08:38.193 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.192+0000 7f3888400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f38800d0f60 con 0x7f388015b6f0 2024-05-28T06:08:38.193 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.192+0000 7f3885a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f38800d0640 0x7f38800d0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:38.193 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.192+0000 7f3885a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f38800d0640 0x7f38800d0a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35980/0 (socket says 172.21.15.96:35980) 2024-05-28T06:08:38.193 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.192+0000 7f3885a00640 1 -- 172.21.15.96:0/3628824433 learned_addr learned my addr 172.21.15.96:0/3628824433 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:08:38.193 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.192+0000 7f3885000640 1 --2- 172.21.15.96:0/3628824433 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f388015b6f0 0x7f388006e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:38.193 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.192+0000 7f3885a00640 1 -- 172.21.15.96:0/3628824433 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f388015b6f0 msgr2=0x7f388006e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:38.194 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.192+0000 7f3885a00640 1 --2- 172.21.15.96:0/3628824433 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f388015b6f0 0x7f388006e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:38.194 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.192+0000 7f3885a00640 1 -- 172.21.15.96:0/3628824433 >> v1:172.21.15.123:6789/0 conn(0x7f38800725e0 legacy=0x7f38800b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:38.194 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.192+0000 7f3885a00640 1 -- 172.21.15.96:0/3628824433 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3880071e70 con 0x7f38800d0640 2024-05-28T06:08:38.194 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.192+0000 7f3885000640 1 --2- 172.21.15.96:0/3628824433 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f388015b6f0 0x7f388006e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:08:38.195 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.193+0000 7f3885a00640 1 --2- 172.21.15.96:0/3628824433 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f38800d0640 0x7f38800d0a20 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f3874009870 tx=0x7f387402a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1104a6d14cb5c02b server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:38.195 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.194+0000 7f387fe00640 1 -- 172.21.15.96:0/3628824433 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3874037070 con 0x7f38800d0640 2024-05-28T06:08:38.195 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.194+0000 7f387fe00640 1 -- 172.21.15.96:0/3628824433 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3874032440 con 0x7f38800d0640 2024-05-28T06:08:38.195 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.194+0000 7f387fe00640 1 -- 172.21.15.96:0/3628824433 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3874032810 con 0x7f38800d0640 2024-05-28T06:08:38.195 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.194+0000 7f3888400640 1 -- 172.21.15.96:0/3628824433 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f38800d0640 msgr2=0x7f38800d0a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:38.195 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.194+0000 7f3888400640 1 --2- 172.21.15.96:0/3628824433 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f38800d0640 0x7f38800d0a20 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f3874009870 tx=0x7f387402a210 comp rx=0 tx=0).stop 2024-05-28T06:08:38.195 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.194+0000 7f3888400640 1 -- 172.21.15.96:0/3628824433 shutdown_connections 2024-05-28T06:08:38.196 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.194+0000 7f3888400640 1 --2- 172.21.15.96:0/3628824433 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f388015b6f0 0x7f388006e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:38.196 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.194+0000 7f3888400640 1 --2- 172.21.15.96:0/3628824433 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f38800d0640 0x7f38800d0a20 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:38.196 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.194+0000 7f3888400640 1 -- 172.21.15.96:0/3628824433 >> 172.21.15.96:0/3628824433 conn(0x7f388006bd20 msgr2=0x7f388006c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:38.196 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.195+0000 7f3888400640 1 -- 172.21.15.96:0/3628824433 shutdown_connections 2024-05-28T06:08:38.196 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.195+0000 7f3888400640 1 -- 172.21.15.96:0/3628824433 wait complete. 2024-05-28T06:08:38.197 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.197+0000 7f3888400640 1 Processor -- start 2024-05-28T06:08:38.197 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.197+0000 7f3888400640 1 -- start start 2024-05-28T06:08:38.198 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.198+0000 7f3888400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f38800725e0 0x7f38801f1800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:38.198 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.198+0000 7f3888400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38800d0640 0x7f38801f5d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:38.198 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.198+0000 7f3888400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f388015b6f0 0x7f3880143090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:38.199 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.198+0000 7f3888400640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f388006e890 con 0x7f388015b6f0 2024-05-28T06:08:38.199 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.198+0000 7f3888400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f38800ae130 con 0x7f38800d0640 2024-05-28T06:08:38.199 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.198+0000 7f3888400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f38800af660 con 0x7f38800725e0 2024-05-28T06:08:38.199 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.198+0000 7f3886400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f388015b6f0 0x7f3880143090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:38.199 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.198+0000 7f3886400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f388015b6f0 0x7f3880143090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35992/0 (socket says 172.21.15.96:35992) 2024-05-28T06:08:38.199 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.198+0000 7f3886400640 1 -- 172.21.15.96:0/1341940301 learned_addr learned my addr 172.21.15.96:0/1341940301 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:08:38.201 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.198+0000 7f3885000640 1 --2- 172.21.15.96:0/1341940301 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38800d0640 0x7f38801f5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:38.201 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.198+0000 7f3885a00640 1 --2- 172.21.15.96:0/1341940301 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f38800725e0 0x7f38801f1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:38.201 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.198+0000 7f3886400640 1 -- 172.21.15.96:0/1341940301 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f38800725e0 msgr2=0x7f38801f1800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:38.201 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.198+0000 7f3886400640 1 --2- 172.21.15.96:0/1341940301 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f38800725e0 0x7f38801f1800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:38.201 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.198+0000 7f3886400640 1 -- 172.21.15.96:0/1341940301 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38800d0640 msgr2=0x7f38801f5d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:38.201 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.198+0000 7f3886400640 1 --2- 172.21.15.96:0/1341940301 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38800d0640 0x7f38801f5d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:38.201 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.198+0000 7f3886400640 1 -- 172.21.15.96:0/1341940301 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3880143a40 con 0x7f388015b6f0 2024-05-28T06:08:38.201 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.199+0000 7f3885a00640 1 --2- 172.21.15.96:0/1341940301 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f38800725e0 0x7f38801f1800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:08:38.202 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.199+0000 7f3885000640 1 --2- 172.21.15.96:0/1341940301 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38800d0640 0x7f38801f5d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:08:38.202 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.199+0000 7f3886400640 1 --2- 172.21.15.96:0/1341940301 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f388015b6f0 0x7f3880143090 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f387800c780 tx=0x7f387800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:38.202 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.199+0000 7f387d600640 1 -- 172.21.15.96:0/1341940301 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3878004b60 con 0x7f388015b6f0 2024-05-28T06:08:38.202 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.199+0000 7f387d600640 1 -- 172.21.15.96:0/1341940301 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3878004d00 con 0x7f388015b6f0 2024-05-28T06:08:38.202 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.199+0000 7f387d600640 1 -- 172.21.15.96:0/1341940301 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f387800aa90 con 0x7f388015b6f0 2024-05-28T06:08:38.202 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.199+0000 7f3888400640 1 -- 172.21.15.96:0/1341940301 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3880143d20 con 0x7f388015b6f0 2024-05-28T06:08:38.202 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.199+0000 7f3888400640 1 -- 172.21.15.96:0/1341940301 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38801f6270 con 0x7f388015b6f0 2024-05-28T06:08:38.202 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.200+0000 7f3888400640 1 -- 172.21.15.96:0/1341940301 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38800689e0 con 0x7f388015b6f0 2024-05-28T06:08:38.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.201+0000 7f387d600640 1 -- 172.21.15.96:0/1341940301 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 9) ==== 91295+0+0 (secure 0 0 0) 0x7f387800ac30 con 0x7f388015b6f0 2024-05-28T06:08:38.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.202+0000 7f387d600640 1 --2- 172.21.15.96:0/1341940301 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f386006cbc0 0x7f386006f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:38.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.202+0000 7f387d600640 1 -- 172.21.15.96:0/1341940301 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5765+0+0 (secure 0 0 0) 0x7f387808c330 con 0x7f388015b6f0 2024-05-28T06:08:38.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.203+0000 7f3885a00640 1 --2- 172.21.15.96:0/1341940301 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f386006cbc0 0x7f386006f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:38.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.204+0000 7f3885a00640 1 --2- 172.21.15.96:0/1341940301 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f386006cbc0 0x7f386006f080 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f3874034000 tx=0x7f387402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:38.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.207+0000 7f387d600640 1 -- 172.21.15.96:0/1341940301 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3878020070 con 0x7f388015b6f0 2024-05-28T06:08:38.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.392+0000 7f3888400640 1 -- 172.21.15.96:0/1341940301 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f388006cc20 con 0x7f388015b6f0 2024-05-28T06:08:38.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.393+0000 7f387d600640 1 -- 172.21.15.96:0/1341940301 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v34) ==== 93+0+10 (secure 0 0 0) 0x7f387805a380 con 0x7f388015b6f0 2024-05-28T06:08:38.394 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 1 2024-05-28T06:08:38.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.395+0000 7f3888400640 1 -- 172.21.15.96:0/1341940301 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f386006cbc0 msgr2=0x7f386006f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:38.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.395+0000 7f3888400640 1 --2- 172.21.15.96:0/1341940301 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f386006cbc0 0x7f386006f080 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f3874034000 tx=0x7f387402e000 comp rx=0 tx=0).stop 2024-05-28T06:08:38.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.395+0000 7f3888400640 1 -- 172.21.15.96:0/1341940301 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f388015b6f0 msgr2=0x7f3880143090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:38.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.395+0000 7f3888400640 1 --2- 172.21.15.96:0/1341940301 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f388015b6f0 0x7f3880143090 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f387800c780 tx=0x7f387800cc40 comp rx=0 tx=0).stop 2024-05-28T06:08:38.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.396+0000 7f3888400640 1 -- 172.21.15.96:0/1341940301 shutdown_connections 2024-05-28T06:08:38.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.396+0000 7f3888400640 1 --2- 172.21.15.96:0/1341940301 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f386006cbc0 0x7f386006f080 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:38.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.396+0000 7f3888400640 1 --2- 172.21.15.96:0/1341940301 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f388015b6f0 0x7f3880143090 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:38.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.396+0000 7f3888400640 1 --2- 172.21.15.96:0/1341940301 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38800d0640 0x7f38801f5d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:38.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.396+0000 7f3888400640 1 --2- 172.21.15.96:0/1341940301 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f38800725e0 0x7f38801f1800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:38.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.396+0000 7f3888400640 1 -- 172.21.15.96:0/1341940301 >> 172.21.15.96:0/1341940301 conn(0x7f388006bd20 msgr2=0x7f3880158450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:38.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.396+0000 7f3888400640 1 -- 172.21.15.96:0/1341940301 shutdown_connections 2024-05-28T06:08:38.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.396+0000 7f3888400640 1 -- 172.21.15.96:0/1341940301 wait complete. 2024-05-28T06:08:38.415 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-28T06:08:38.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.612+0000 7f777e800640 1 Processor -- start 2024-05-28T06:08:38.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.612+0000 7f777e800640 1 -- start start 2024-05-28T06:08:38.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.614+0000 7f777e800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f77780ce1f0 0x7f77780b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:38.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.614+0000 7f777e800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f77780ce5d0 0x7f777815ec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:38.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.614+0000 7f777e800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f77780b3d60 0x7f77780b41d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:38.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.614+0000 7f777e800640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f77780b3580 con 0x7f77780b3d60 2024-05-28T06:08:38.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.614+0000 7f777e800640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f777815f490 con 0x7f77780ce1f0 2024-05-28T06:08:38.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.614+0000 7f777e800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f777815f770 con 0x7f77780ce5d0 2024-05-28T06:08:38.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.614+0000 7f777de00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f77780b3d60 0x7f77780b41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:38.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.614+0000 7f777de00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f77780b3d60 0x7f77780b41d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:36000/0 (socket says 172.21.15.96:36000) 2024-05-28T06:08:38.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.614+0000 7f777de00640 1 -- 172.21.15.96:0/4147895734 learned_addr learned my addr 172.21.15.96:0/4147895734 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:08:38.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.614+0000 7f777ca00640 1 --2- 172.21.15.96:0/4147895734 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f77780ce5d0 0x7f777815ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:38.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.614+0000 7f777d400640 1 --2- 172.21.15.96:0/4147895734 >> v2:172.21.15.123:3300/0 conn(0x7f77780ce1f0 0x7f77780b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:38.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.615+0000 7f777de00640 1 -- 172.21.15.96:0/4147895734 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f77780ce5d0 msgr2=0x7f777815ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:38.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.615+0000 7f777de00640 1 --2- 172.21.15.96:0/4147895734 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f77780ce5d0 0x7f777815ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:38.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.615+0000 7f777de00640 1 -- 172.21.15.96:0/4147895734 >> v2:172.21.15.123:3300/0 conn(0x7f77780ce1f0 msgr2=0x7f77780b4dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:38.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.615+0000 7f777de00640 1 --2- 172.21.15.96:0/4147895734 >> v2:172.21.15.123:3300/0 conn(0x7f77780ce1f0 0x7f77780b4dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:38.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.615+0000 7f777de00640 1 -- 172.21.15.96:0/4147895734 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f777815fa50 con 0x7f77780b3d60 2024-05-28T06:08:38.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.616+0000 7f777de00640 1 --2- 172.21.15.96:0/4147895734 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f77780b3d60 0x7f77780b41d0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f77780cdd00 tx=0x7f777002a570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9586d4ab3807886 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:38.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.616+0000 7f7777e00640 1 -- 172.21.15.96:0/4147895734 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7770036070 con 0x7f77780b3d60 2024-05-28T06:08:38.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.616+0000 7f7777e00640 1 -- 172.21.15.96:0/4147895734 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7770031600 con 0x7f77780b3d60 2024-05-28T06:08:38.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.616+0000 7f7777e00640 1 -- 172.21.15.96:0/4147895734 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77700319d0 con 0x7f77780b3d60 2024-05-28T06:08:38.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.616+0000 7f777e800640 1 -- 172.21.15.96:0/4147895734 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f77780b3d60 msgr2=0x7f77780b41d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:38.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.616+0000 7f777e800640 1 --2- 172.21.15.96:0/4147895734 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f77780b3d60 0x7f77780b41d0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f77780cdd00 tx=0x7f777002a570 comp rx=0 tx=0).stop 2024-05-28T06:08:38.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.617+0000 7f777e800640 1 -- 172.21.15.96:0/4147895734 shutdown_connections 2024-05-28T06:08:38.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.617+0000 7f777e800640 1 --2- 172.21.15.96:0/4147895734 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f77780b3d60 0x7f77780b41d0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:38.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.617+0000 7f777e800640 1 --2- 172.21.15.96:0/4147895734 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f77780ce5d0 0x7f777815ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:38.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.617+0000 7f777e800640 1 --2- 172.21.15.96:0/4147895734 >> v2:172.21.15.123:3300/0 conn(0x7f77780ce1f0 0x7f77780b4dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:38.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.617+0000 7f777e800640 1 -- 172.21.15.96:0/4147895734 >> 172.21.15.96:0/4147895734 conn(0x7f777806b900 msgr2=0x7f777806bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:38.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.617+0000 7f777e800640 1 -- 172.21.15.96:0/4147895734 shutdown_connections 2024-05-28T06:08:38.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.617+0000 7f777e800640 1 -- 172.21.15.96:0/4147895734 wait complete. 2024-05-28T06:08:38.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.619+0000 7f777e800640 1 Processor -- start 2024-05-28T06:08:38.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.619+0000 7f777e800640 1 -- start start 2024-05-28T06:08:38.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.620+0000 7f777e800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f77780b3d60 0x7f77781566c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:38.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.620+0000 7f777e800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f77780ce1f0 0x7f777815a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:38.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.620+0000 7f777e800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77780ce5d0 0x7f7778160bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:38.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.620+0000 7f777e800640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f77780ccb90 con 0x7f77780ce1f0 2024-05-28T06:08:38.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.620+0000 7f777e800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f777815f180 con 0x7f77780ce5d0 2024-05-28T06:08:38.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.620+0000 7f777e800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f77780ccd10 con 0x7f77780b3d60 2024-05-28T06:08:38.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.620+0000 7f777ca00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f77780ce1f0 0x7f777815a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:38.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.620+0000 7f777ca00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f77780ce1f0 0x7f777815a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:36008/0 (socket says 172.21.15.96:36008) 2024-05-28T06:08:38.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.620+0000 7f777ca00640 1 -- 172.21.15.96:0/925111660 learned_addr learned my addr 172.21.15.96:0/925111660 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:08:38.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.621+0000 7f777ca00640 1 -- 172.21.15.96:0/925111660 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f77780b3d60 msgr2=0x7f77781566c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:38.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.621+0000 7f777ca00640 1 --2- 172.21.15.96:0/925111660 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f77780b3d60 0x7f77781566c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:38.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.621+0000 7f777ca00640 1 -- 172.21.15.96:0/925111660 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77780ce5d0 msgr2=0x7f7778160bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:38.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.621+0000 7f777ca00640 1 --2- 172.21.15.96:0/925111660 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77780ce5d0 0x7f7778160bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:38.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.621+0000 7f777ca00640 1 -- 172.21.15.96:0/925111660 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7778156c00 con 0x7f77780ce1f0 2024-05-28T06:08:38.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.621+0000 7f777ca00640 1 --2- 172.21.15.96:0/925111660 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f77780ce1f0 0x7f777815a9b0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f776400d5f0 tx=0x7f776400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:38.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.621+0000 7f7775600640 1 -- 172.21.15.96:0/925111660 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77640125f0 con 0x7f77780ce1f0 2024-05-28T06:08:38.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.621+0000 7f7775600640 1 -- 172.21.15.96:0/925111660 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7764019070 con 0x7f77780ce1f0 2024-05-28T06:08:38.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.621+0000 7f777e800640 1 -- 172.21.15.96:0/925111660 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f777815fe10 con 0x7f77780ce1f0 2024-05-28T06:08:38.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.621+0000 7f7775600640 1 -- 172.21.15.96:0/925111660 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77640129d0 con 0x7f77780ce1f0 2024-05-28T06:08:38.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.621+0000 7f777e800640 1 -- 172.21.15.96:0/925111660 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77781603d0 con 0x7f77780ce1f0 2024-05-28T06:08:38.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.622+0000 7f777e800640 1 -- 172.21.15.96:0/925111660 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77780689e0 con 0x7f77780ce1f0 2024-05-28T06:08:38.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.623+0000 7f7775600640 1 -- 172.21.15.96:0/925111660 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 9) ==== 91295+0+0 (secure 0 0 0) 0x7f7764012b70 con 0x7f77780ce1f0 2024-05-28T06:08:38.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.624+0000 7f7775600640 1 --2- 172.21.15.96:0/925111660 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f775806cbc0 0x7f775806f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:38.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.624+0000 7f7775600640 1 -- 172.21.15.96:0/925111660 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5765+0+0 (secure 0 0 0) 0x7f776408c430 con 0x7f77780ce1f0 2024-05-28T06:08:38.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.625+0000 7f777d400640 1 --2- 172.21.15.96:0/925111660 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f775806cbc0 0x7f775806f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:38.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.626+0000 7f777d400640 1 --2- 172.21.15.96:0/925111660 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f775806cbc0 0x7f775806f080 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f776c006b10 tx=0x7f776c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:38.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.629+0000 7f7775600640 1 -- 172.21.15.96:0/925111660 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f776405a4d0 con 0x7f77780ce1f0 2024-05-28T06:08:38.812 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.812+0000 7f777e800640 1 -- 172.21.15.96:0/925111660 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f77780c9700 con 0x7f77780ce1f0 2024-05-28T06:08:38.813 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.813+0000 7f7775600640 1 -- 172.21.15.96:0/925111660 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v34) ==== 104+0+11 (secure 0 0 0) 0x7f7764059dd0 con 0x7f77780ce1f0 2024-05-28T06:08:38.813 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 32 2024-05-28T06:08:38.816 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.815+0000 7f777e800640 1 -- 172.21.15.96:0/925111660 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f775806cbc0 msgr2=0x7f775806f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:38.816 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.815+0000 7f777e800640 1 --2- 172.21.15.96:0/925111660 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f775806cbc0 0x7f775806f080 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f776c006b10 tx=0x7f776c008040 comp rx=0 tx=0).stop 2024-05-28T06:08:38.816 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.815+0000 7f777e800640 1 -- 172.21.15.96:0/925111660 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f77780ce1f0 msgr2=0x7f777815a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:38.816 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.815+0000 7f777e800640 1 --2- 172.21.15.96:0/925111660 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f77780ce1f0 0x7f777815a9b0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f776400d5f0 tx=0x7f776400dab0 comp rx=0 tx=0).stop 2024-05-28T06:08:38.816 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.815+0000 7f777e800640 1 -- 172.21.15.96:0/925111660 shutdown_connections 2024-05-28T06:08:38.816 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.815+0000 7f777e800640 1 --2- 172.21.15.96:0/925111660 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f775806cbc0 0x7f775806f080 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:38.816 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.815+0000 7f777e800640 1 --2- 172.21.15.96:0/925111660 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77780ce5d0 0x7f7778160bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:38.817 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.815+0000 7f777e800640 1 --2- 172.21.15.96:0/925111660 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f77780ce1f0 0x7f777815a9b0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:38.817 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.815+0000 7f777e800640 1 --2- 172.21.15.96:0/925111660 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f77780b3d60 0x7f77781566c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:38.817 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.815+0000 7f777e800640 1 -- 172.21.15.96:0/925111660 >> 172.21.15.96:0/925111660 conn(0x7f777806b900 msgr2=0x7f777815c660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:38.817 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.816+0000 7f777e800640 1 -- 172.21.15.96:0/925111660 shutdown_connections 2024-05-28T06:08:38.817 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:38.816+0000 7f777e800640 1 -- 172.21.15.96:0/925111660 wait complete. 2024-05-28T06:08:38.833 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-28T06:08:39.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.024+0000 7fa4dd600640 1 Processor -- start 2024-05-28T06:08:39.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.025+0000 7fa4dd600640 1 -- start start 2024-05-28T06:08:39.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.026+0000 7fa4dd600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fa4d80d16a0 0x7fa4d80cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:39.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.026+0000 7fa4dd600640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4d80cfce0 con 0x7fa4d815e870 2024-05-28T06:08:39.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.026+0000 7fa4dd600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4d815ec50 con 0x7fa4d80cf870 2024-05-28T06:08:39.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.026+0000 7fa4dd600640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fa4d80d2150 con 0x7fa4d80d16a0 2024-05-28T06:08:39.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.026+0000 7fa4d7e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fa4d815e870 0x7fa4d80d0f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:45548/0 (socket says 172.21.15.96:45548) 2024-05-28T06:08:39.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.026+0000 7fa4d7e00640 1 -- 172.21.15.96:0/1785818566 learned_addr learned my addr 172.21.15.96:0/1785818566 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:08:39.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.027+0000 7fa4d7400640 1 --2- 172.21.15.96:0/1785818566 >> v2:172.21.15.123:3300/0 conn(0x7fa4d80d16a0 0x7fa4d80cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:39.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.027+0000 7fa4d6a00640 1 -- 172.21.15.96:0/1785818566 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 231402312 0 0) 0x7fa4d80cfce0 con 0x7fa4d815e870 2024-05-28T06:08:39.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.027+0000 7fa4d6a00640 1 -- 172.21.15.96:0/1785818566 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa4b8003520 con 0x7fa4d815e870 2024-05-28T06:08:39.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.027+0000 7fa4d6a00640 1 -- 172.21.15.96:0/1785818566 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1060901331 0 0) 0x7fa4b8003520 con 0x7fa4d815e870 2024-05-28T06:08:39.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.027+0000 7fa4d6a00640 1 -- 172.21.15.96:0/1785818566 >> v2:172.21.15.123:3300/0 conn(0x7fa4d80d16a0 msgr2=0x7fa4d80cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:39.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.027+0000 7fa4d6a00640 1 --2- 172.21.15.96:0/1785818566 >> v2:172.21.15.123:3300/0 conn(0x7fa4d80d16a0 0x7fa4d80cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:39.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.027+0000 7fa4d6a00640 1 -- 172.21.15.96:0/1785818566 >> v1:172.21.15.123:6789/0 conn(0x7fa4d80cf870 legacy=0x7fa4d80d25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:39.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.027+0000 7fa4d6a00640 1 -- 172.21.15.96:0/1785818566 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4d80cff80 con 0x7fa4d815e870 2024-05-28T06:08:39.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.027+0000 7fa4d6a00640 1 -- 172.21.15.96:0/1785818566 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7fa4c8001670 con 0x7fa4d815e870 2024-05-28T06:08:39.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.027+0000 7fa4d6a00640 1 -- 172.21.15.96:0/1785818566 >> v1:172.21.15.96:6789/0 conn(0x7fa4d815e870 legacy=0x7fa4d80d0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:39.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.028+0000 7fa4d6a00640 1 --2- 172.21.15.96:0/1785818566 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa4b80039b0 0x7fa4b8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:39.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.028+0000 7fa4d6a00640 1 --2- 172.21.15.96:0/1785818566 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa4b8004800 0x7fa4b8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:39.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.028+0000 7fa4d6a00640 1 --2- 172.21.15.96:0/1785818566 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa4b80051c0 0x7fa4b800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:39.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.028+0000 7fa4d6a00640 1 -- 172.21.15.96:0/1785818566 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa4d80d2150 con 0x7fa4b80051c0 2024-05-28T06:08:39.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.028+0000 7fa4d6a00640 1 -- 172.21.15.96:0/1785818566 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa4b800c060 con 0x7fa4b80039b0 2024-05-28T06:08:39.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.028+0000 7fa4d6a00640 1 -- 172.21.15.96:0/1785818566 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fa4b800c260 con 0x7fa4b8004800 2024-05-28T06:08:39.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.028+0000 7fa4dcc00640 1 --2- 172.21.15.96:0/1785818566 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa4b80051c0 0x7fa4b800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:39.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.028+0000 7fa4d7400640 1 --2- 172.21.15.96:0/1785818566 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa4b8004800 0x7fa4b8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:39.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.028+0000 7fa4d7e00640 1 --2- 172.21.15.96:0/1785818566 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa4b80039b0 0x7fa4b8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:39.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.028+0000 7fa4dcc00640 1 -- 172.21.15.96:0/1785818566 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa4b8004800 msgr2=0x7fa4b8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:39.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.028+0000 7fa4dcc00640 1 --2- 172.21.15.96:0/1785818566 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa4b8004800 0x7fa4b8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:39.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.028+0000 7fa4dcc00640 1 -- 172.21.15.96:0/1785818566 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa4b80039b0 msgr2=0x7fa4b8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:39.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.028+0000 7fa4dcc00640 1 --2- 172.21.15.96:0/1785818566 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa4b80039b0 0x7fa4b8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:39.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.028+0000 7fa4dcc00640 1 -- 172.21.15.96:0/1785818566 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4b8003520 con 0x7fa4b80051c0 2024-05-28T06:08:39.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.029+0000 7fa4dcc00640 1 -- 172.21.15.96:0/1785818566 reap_dead start 2024-05-28T06:08:39.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.029+0000 7fa4dcc00640 1 --2- 172.21.15.96:0/1785818566 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa4b80051c0 0x7fa4b800b9c0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fa4d0001750 tx=0x7fa4b8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e9eb1132e7da234 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:39.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.030+0000 7fa4d6a00640 1 -- 172.21.15.96:0/1785818566 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4d0001370 con 0x7fa4b80051c0 2024-05-28T06:08:39.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.030+0000 7fa4d6a00640 1 -- 172.21.15.96:0/1785818566 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4d0005310 con 0x7fa4b80051c0 2024-05-28T06:08:39.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.030+0000 7fa4d6a00640 1 -- 172.21.15.96:0/1785818566 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4d002abe0 con 0x7fa4b80051c0 2024-05-28T06:08:39.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.030+0000 7fa4dd600640 1 -- 172.21.15.96:0/1785818566 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa4b80051c0 msgr2=0x7fa4b800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:39.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.030+0000 7fa4dd600640 1 --2- 172.21.15.96:0/1785818566 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa4b80051c0 0x7fa4b800b9c0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fa4d0001750 tx=0x7fa4b8005830 comp rx=0 tx=0).stop 2024-05-28T06:08:39.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.030+0000 7fa4dd600640 1 -- 172.21.15.96:0/1785818566 shutdown_connections 2024-05-28T06:08:39.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.030+0000 7fa4dd600640 1 --2- 172.21.15.96:0/1785818566 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa4b80051c0 0x7fa4b800b9c0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:39.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.030+0000 7fa4dd600640 1 -- 172.21.15.96:0/1785818566 >> 172.21.15.96:0/1785818566 conn(0x7fa4d808bfc0 msgr2=0x7fa4d80d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:39.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.031+0000 7fa4dd600640 1 -- 172.21.15.96:0/1785818566 shutdown_connections 2024-05-28T06:08:39.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.031+0000 7fa4dd600640 1 -- 172.21.15.96:0/1785818566 wait complete. 2024-05-28T06:08:39.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.032+0000 7fa4dd600640 1 Processor -- start 2024-05-28T06:08:39.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.032+0000 7fa4dd600640 1 -- start start 2024-05-28T06:08:39.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.033+0000 7fa4dd600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa4b80051c0 0x7fa4d8153dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:39.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.033+0000 7fa4dd600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa4d81f46f0 0x7fa4d81f4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:39.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.033+0000 7fa4dd600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa4d815b410 0x7fa4d815b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:39.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.033+0000 7fa4dd600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa4d80d0820 con 0x7fa4b80051c0 2024-05-28T06:08:39.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.033+0000 7fa4dd600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa4d80d0540 con 0x7fa4d81f46f0 2024-05-28T06:08:39.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.033+0000 7fa4dd600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fa4d80d4fb0 con 0x7fa4d815b410 2024-05-28T06:08:39.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.034+0000 7fa4d7e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa4b80051c0 0x7fa4d8153dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:39.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.034+0000 7fa4d7e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa4b80051c0 0x7fa4d8153dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58066/0 (socket says 172.21.15.96:58066) 2024-05-28T06:08:39.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.034+0000 7fa4d7e00640 1 -- 172.21.15.96:0/2981242011 learned_addr learned my addr 172.21.15.96:0/2981242011 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:08:39.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.034+0000 7fa4d7e00640 1 -- 172.21.15.96:0/2981242011 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa4d815b410 msgr2=0x7fa4d815b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:39.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.034+0000 7fa4dcc00640 1 --2- 172.21.15.96:0/2981242011 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa4d815b410 0x7fa4d815b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:39.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.034+0000 7fa4d7400640 1 --2- 172.21.15.96:0/2981242011 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa4d81f46f0 0x7fa4d81f4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:39.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.034+0000 7fa4d7e00640 1 --2- 172.21.15.96:0/2981242011 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa4d815b410 0x7fa4d815b8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:39.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.034+0000 7fa4d7e00640 1 -- 172.21.15.96:0/2981242011 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa4d81f46f0 msgr2=0x7fa4d81f4b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:39.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.034+0000 7fa4d7e00640 1 --2- 172.21.15.96:0/2981242011 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa4d81f46f0 0x7fa4d81f4b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:39.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.034+0000 7fa4d7e00640 1 -- 172.21.15.96:0/2981242011 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4d81f50b0 con 0x7fa4b80051c0 2024-05-28T06:08:39.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.034+0000 7fa4d7e00640 1 --2- 172.21.15.96:0/2981242011 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa4b80051c0 0x7fa4d8153dd0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fa4c800e810 tx=0x7fa4c800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:39.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.034+0000 7fa4cfe00640 1 -- 172.21.15.96:0/2981242011 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4c800c870 con 0x7fa4b80051c0 2024-05-28T06:08:39.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.035+0000 7fa4cfe00640 1 -- 172.21.15.96:0/2981242011 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4c8018070 con 0x7fa4b80051c0 2024-05-28T06:08:39.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.035+0000 7fa4cfe00640 1 -- 172.21.15.96:0/2981242011 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4c800cc50 con 0x7fa4b80051c0 2024-05-28T06:08:39.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.035+0000 7fa4dd600640 1 -- 172.21.15.96:0/2981242011 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4d815be00 con 0x7fa4b80051c0 2024-05-28T06:08:39.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.035+0000 7fa4dd600640 1 -- 172.21.15.96:0/2981242011 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4d80b86d0 con 0x7fa4b80051c0 2024-05-28T06:08:39.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.036+0000 7fa4dd600640 1 -- 172.21.15.96:0/2981242011 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4d80b84c0 con 0x7fa4b80051c0 2024-05-28T06:08:39.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.037+0000 7fa4cfe00640 1 -- 172.21.15.96:0/2981242011 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 9) ==== 91295+0+0 (secure 0 0 0) 0x7fa4c8013420 con 0x7fa4b80051c0 2024-05-28T06:08:39.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.037+0000 7fa4cfe00640 1 --2- 172.21.15.96:0/2981242011 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa4b406cbf0 0x7fa4b406f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:39.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.038+0000 7fa4cfe00640 1 -- 172.21.15.96:0/2981242011 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5765+0+0 (secure 0 0 0) 0x7fa4c808bbf0 con 0x7fa4b80051c0 2024-05-28T06:08:39.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.038+0000 7fa4d7400640 1 --2- 172.21.15.96:0/2981242011 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa4b406cbf0 0x7fa4b406f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:39.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.039+0000 7fa4d7400640 1 --2- 172.21.15.96:0/2981242011 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa4b406cbf0 0x7fa4b406f0b0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fa4d808c4c0 tx=0x7fa4c000b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:39.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.041+0000 7fa4cfe00640 1 -- 172.21.15.96:0/2981242011 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa4c8059c50 con 0x7fa4b80051c0 2024-05-28T06:08:39.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.223+0000 7fa4dd600640 1 -- 172.21.15.96:0/2981242011 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fa4d80689e0 con 0x7fa4b80051c0 2024-05-28T06:08:39.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.224+0000 7fa4cfe00640 1 -- 172.21.15.96:0/2981242011 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v34) ==== 100+0+11 (secure 0 0 0) 0x7fa4c8059550 con 0x7fa4b80051c0 2024-05-28T06:08:39.225 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 64 2024-05-28T06:08:39.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.226+0000 7fa4dd600640 1 -- 172.21.15.96:0/2981242011 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa4b406cbf0 msgr2=0x7fa4b406f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:39.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.226+0000 7fa4dd600640 1 --2- 172.21.15.96:0/2981242011 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa4b406cbf0 0x7fa4b406f0b0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fa4d808c4c0 tx=0x7fa4c000b040 comp rx=0 tx=0).stop 2024-05-28T06:08:39.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.226+0000 7fa4dd600640 1 -- 172.21.15.96:0/2981242011 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa4b80051c0 msgr2=0x7fa4d8153dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:39.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.226+0000 7fa4dd600640 1 --2- 172.21.15.96:0/2981242011 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa4b80051c0 0x7fa4d8153dd0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fa4c800e810 tx=0x7fa4c800ecd0 comp rx=0 tx=0).stop 2024-05-28T06:08:39.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.226+0000 7fa4dd600640 1 -- 172.21.15.96:0/2981242011 shutdown_connections 2024-05-28T06:08:39.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.226+0000 7fa4dd600640 1 --2- 172.21.15.96:0/2981242011 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa4b406cbf0 0x7fa4b406f0b0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:39.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.226+0000 7fa4dd600640 1 --2- 172.21.15.96:0/2981242011 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa4d815b410 0x7fa4d815b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:39.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.226+0000 7fa4dd600640 1 --2- 172.21.15.96:0/2981242011 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa4d81f46f0 0x7fa4d81f4b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:39.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.226+0000 7fa4dd600640 1 --2- 172.21.15.96:0/2981242011 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa4b80051c0 0x7fa4d8153dd0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:39.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.226+0000 7fa4dd600640 1 -- 172.21.15.96:0/2981242011 >> 172.21.15.96:0/2981242011 conn(0x7fa4d808bfc0 msgr2=0x7fa4d80b35b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:39.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.227+0000 7fa4dd600640 1 -- 172.21.15.96:0/2981242011 shutdown_connections 2024-05-28T06:08:39.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.227+0000 7fa4dd600640 1 -- 172.21.15.96:0/2981242011 wait complete. 2024-05-28T06:08:39.245 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-28T06:08:39.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.427+0000 7f9761a00640 1 Processor -- start 2024-05-28T06:08:39.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.427+0000 7f9761a00640 1 -- start start 2024-05-28T06:08:39.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.428+0000 7f9761a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f975c06f860 0x7f975c0b3150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:39.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.428+0000 7f9761a00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f975c0b3690 0x7f975c0d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:39.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.428+0000 7f9761a00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f975c06ed10 con 0x7f975c0d09e0 2024-05-28T06:08:39.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.428+0000 7f9761a00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f975c0d0e50 con 0x7f975c0b3690 2024-05-28T06:08:39.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.428+0000 7f9761a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f975c15cb00 con 0x7f975c06f860 2024-05-28T06:08:39.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.429+0000 7f975b400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f975c0d09e0 0x7f975c160210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:45550/0 (socket says 172.21.15.96:45550) 2024-05-28T06:08:39.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.429+0000 7f975b400640 1 -- 172.21.15.96:0/1008227286 learned_addr learned my addr 172.21.15.96:0/1008227286 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:08:39.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.429+0000 7f975aa00640 1 --2- 172.21.15.96:0/1008227286 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f975c06f860 0x7f975c0b3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:39.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.429+0000 7f975a000640 1 --2- 172.21.15.96:0/1008227286 >> v2:172.21.15.123:3300/0 conn(0x7f975c0b3690 0x7f975c0d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:39.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.429+0000 7f9759600640 1 -- 172.21.15.96:0/1008227286 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4236785188 0 0) 0x7f9754001490 con 0x7f975c0d09e0 2024-05-28T06:08:39.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.429+0000 7f9759600640 1 -- 172.21.15.96:0/1008227286 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9740003520 con 0x7f975c0d09e0 2024-05-28T06:08:39.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.430+0000 7f9759600640 1 -- 172.21.15.96:0/1008227286 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 828960474 0 0) 0x7f9740003520 con 0x7f975c0d09e0 2024-05-28T06:08:39.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.430+0000 7f9759600640 1 -- 172.21.15.96:0/1008227286 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f975c06f860 msgr2=0x7f975c0b3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:39.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.430+0000 7f9759600640 1 --2- 172.21.15.96:0/1008227286 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f975c06f860 0x7f975c0b3150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:39.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.430+0000 7f9759600640 1 -- 172.21.15.96:0/1008227286 >> v2:172.21.15.123:3300/0 conn(0x7f975c0b3690 msgr2=0x7f975c0d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:39.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.430+0000 7f9759600640 1 --2- 172.21.15.96:0/1008227286 >> v2:172.21.15.123:3300/0 conn(0x7f975c0b3690 0x7f975c0d04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:39.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.430+0000 7f9759600640 1 -- 172.21.15.96:0/1008227286 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f975c0b3cc0 con 0x7f975c0d09e0 2024-05-28T06:08:39.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.430+0000 7f9759600640 1 -- 172.21.15.96:0/1008227286 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f9754001bc0 con 0x7f975c0d09e0 2024-05-28T06:08:39.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.430+0000 7f9759600640 1 -- 172.21.15.96:0/1008227286 >> v1:172.21.15.96:6789/0 conn(0x7f975c0d09e0 legacy=0x7f975c160210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:39.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.430+0000 7f9759600640 1 --2- 172.21.15.96:0/1008227286 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f975c06f860 0x7f97400039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:39.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.430+0000 7f9759600640 1 --2- 172.21.15.96:0/1008227286 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9740003f30 0x7f9740004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:39.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.430+0000 7f9759600640 1 --2- 172.21.15.96:0/1008227286 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9740004d40 0x7f97400051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:39.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.430+0000 7f9759600640 1 -- 172.21.15.96:0/1008227286 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f975c0d0e50 con 0x7f9740004d40 2024-05-28T06:08:39.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.430+0000 7f9759600640 1 -- 172.21.15.96:0/1008227286 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f975c15cb00 con 0x7f9740003f30 2024-05-28T06:08:39.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.430+0000 7f9759600640 1 -- 172.21.15.96:0/1008227286 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f974000b9c0 con 0x7f975c06f860 2024-05-28T06:08:39.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.430+0000 7f975b400640 1 --2- 172.21.15.96:0/1008227286 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9740004d40 0x7f97400051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:39.433 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.430+0000 7f975aa00640 1 --2- 172.21.15.96:0/1008227286 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f975c06f860 0x7f97400039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:39.433 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.431+0000 7f975b400640 1 -- 172.21.15.96:0/1008227286 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f975c06f860 msgr2=0x7f97400039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:39.433 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.431+0000 7f975b400640 1 --2- 172.21.15.96:0/1008227286 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f975c06f860 0x7f97400039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:39.433 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.431+0000 7f975b400640 1 -- 172.21.15.96:0/1008227286 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9740003f30 msgr2=0x7f9740004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:39.433 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.431+0000 7f975b400640 1 --2- 172.21.15.96:0/1008227286 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9740003f30 0x7f9740004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:39.433 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.431+0000 7f975b400640 1 -- 172.21.15.96:0/1008227286 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9740003520 con 0x7f9740004d40 2024-05-28T06:08:39.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.432+0000 7f975b400640 1 --2- 172.21.15.96:0/1008227286 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9740004d40 0x7f97400051c0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f975400aa70 tx=0x7f9740005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0d9e009fca29313 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:39.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.432+0000 7f9759600640 1 -- 172.21.15.96:0/1008227286 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f975400a690 con 0x7f9740004d40 2024-05-28T06:08:39.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.432+0000 7f9759600640 1 -- 172.21.15.96:0/1008227286 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f975402f070 con 0x7f9740004d40 2024-05-28T06:08:39.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.432+0000 7f9759600640 1 -- 172.21.15.96:0/1008227286 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9754034470 con 0x7f9740004d40 2024-05-28T06:08:39.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.432+0000 7f9761a00640 1 -- 172.21.15.96:0/1008227286 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9740004d40 msgr2=0x7f97400051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:39.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.432+0000 7f9761a00640 1 --2- 172.21.15.96:0/1008227286 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9740004d40 0x7f97400051c0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f975400aa70 tx=0x7f9740005830 comp rx=0 tx=0).stop 2024-05-28T06:08:39.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.432+0000 7f975b400640 1 -- 172.21.15.96:0/1008227286 reap_dead start 2024-05-28T06:08:39.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.432+0000 7f9761a00640 1 -- 172.21.15.96:0/1008227286 shutdown_connections 2024-05-28T06:08:39.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.432+0000 7f9761a00640 1 -- 172.21.15.96:0/1008227286 >> 172.21.15.96:0/1008227286 conn(0x7f975c06bae0 msgr2=0x7f975c06bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:39.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.432+0000 7f9761a00640 1 -- 172.21.15.96:0/1008227286 shutdown_connections 2024-05-28T06:08:39.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.432+0000 7f9761a00640 1 -- 172.21.15.96:0/1008227286 wait complete. 2024-05-28T06:08:39.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.434+0000 7f9761a00640 1 Processor -- start 2024-05-28T06:08:39.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.434+0000 7f9761a00640 1 -- start start 2024-05-28T06:08:39.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.435+0000 7f9761a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9740004d40 0x7f975c15f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:39.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.435+0000 7f9761a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f975c0c7d40 0x7f975c099850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:39.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.435+0000 7f9761a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f975c0b10e0 0x7f975c0b1560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:39.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.435+0000 7f9761a00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f975c0b4560 con 0x7f9740004d40 2024-05-28T06:08:39.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.435+0000 7f9761a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f975c0b4280 con 0x7f975c0c7d40 2024-05-28T06:08:39.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.435+0000 7f9761a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f975c0b4840 con 0x7f975c0b10e0 2024-05-28T06:08:39.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.435+0000 7f975aa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9740004d40 0x7f975c15f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:39.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.435+0000 7f975aa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9740004d40 0x7f975c15f200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58080/0 (socket says 172.21.15.96:58080) 2024-05-28T06:08:39.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.435+0000 7f975aa00640 1 -- 172.21.15.96:0/3758573092 learned_addr learned my addr 172.21.15.96:0/3758573092 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:08:39.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.436+0000 7f975b400640 1 --2- 172.21.15.96:0/3758573092 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f975c0b10e0 0x7f975c0b1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:39.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.436+0000 7f975a000640 1 --2- 172.21.15.96:0/3758573092 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f975c0c7d40 0x7f975c099850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:39.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.436+0000 7f975aa00640 1 -- 172.21.15.96:0/3758573092 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f975c0b10e0 msgr2=0x7f975c0b1560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:39.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.436+0000 7f975aa00640 1 --2- 172.21.15.96:0/3758573092 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f975c0b10e0 0x7f975c0b1560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:39.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.436+0000 7f975aa00640 1 -- 172.21.15.96:0/3758573092 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f975c0c7d40 msgr2=0x7f975c099850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:39.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.436+0000 7f975aa00640 1 --2- 172.21.15.96:0/3758573092 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f975c0c7d40 0x7f975c099850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:39.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.436+0000 7f975aa00640 1 -- 172.21.15.96:0/3758573092 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f975c0c8550 con 0x7f9740004d40 2024-05-28T06:08:39.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.436+0000 7f975aa00640 1 --2- 172.21.15.96:0/3758573092 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9740004d40 0x7f975c15f200 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f974c011660 tx=0x7f974c011b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:39.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.436+0000 7f9752a00640 1 -- 172.21.15.96:0/3758573092 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f974c00f9c0 con 0x7f9740004d40 2024-05-28T06:08:39.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.436+0000 7f9752a00640 1 -- 172.21.15.96:0/3758573092 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f974c00fb60 con 0x7f9740004d40 2024-05-28T06:08:39.439 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.436+0000 7f9761a00640 1 -- 172.21.15.96:0/3758573092 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f975c0c8830 con 0x7f9740004d40 2024-05-28T06:08:39.439 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.437+0000 7f9752a00640 1 -- 172.21.15.96:0/3758573092 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f974c015460 con 0x7f9740004d40 2024-05-28T06:08:39.439 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.437+0000 7f9761a00640 1 -- 172.21.15.96:0/3758573092 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f975c0c8df0 con 0x7f9740004d40 2024-05-28T06:08:39.439 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.438+0000 7f9761a00640 1 -- 172.21.15.96:0/3758573092 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f975c0d07d0 con 0x7f9740004d40 2024-05-28T06:08:39.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.439+0000 7f9752a00640 1 -- 172.21.15.96:0/3758573092 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 9) ==== 91295+0+0 (secure 0 0 0) 0x7f974c015600 con 0x7f9740004d40 2024-05-28T06:08:39.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.440+0000 7f9752a00640 1 --2- 172.21.15.96:0/3758573092 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f973c06cbf0 0x7f973c06f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:39.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.440+0000 7f9752a00640 1 -- 172.21.15.96:0/3758573092 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5765+0+0 (secure 0 0 0) 0x7f974c08cb40 con 0x7f9740004d40 2024-05-28T06:08:39.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.441+0000 7f975a000640 1 --2- 172.21.15.96:0/3758573092 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f973c06cbf0 0x7f973c06f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:39.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.442+0000 7f975a000640 1 --2- 172.21.15.96:0/3758573092 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f973c06cbf0 0x7f973c06f0b0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f9748004770 tx=0x7f9748008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:39.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.444+0000 7f9752a00640 1 -- 172.21.15.96:0/3758573092 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f975c0d07d0 con 0x7f9740004d40 2024-05-28T06:08:39.670 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.670+0000 7f9761a00640 1 -- 172.21.15.96:0/3758573092 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f975c0689e0 con 0x7f9740004d40 2024-05-28T06:08:39.671 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.671+0000 7f9752a00640 1 -- 172.21.15.96:0/3758573092 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+5178 (secure 0 0 0) 0x7f974c019030 con 0x7f9740004d40 2024-05-28T06:08:39.672 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:08:39.672 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4580,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6836/918905879","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":918905879},{"type":"v1","addr":"172.21.15.123:6836","nonce":918905879}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":5223,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6834/697317719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6833","nonce":697317719},{"type":"v1","addr":"172.21.15.96:6834","nonce":697317719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":5226,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/1327552826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":1327552826},{"type":"v1","addr":"172.21.15.96:6835","nonce":1327552826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-28T06:05:29.892912+0000","modified":"2024-05-28T06:05:32.976408+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5238":{"gid":5238,"name":"d","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6837/458334664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6835","nonce":458334664},{"type":"v1","addr":"172.21.15.123:6837","nonce":458334664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5238,"qdb_cluster":[5238]},"id":2}]} 2024-05-28T06:08:39.672 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 16 2024-05-28T06:08:39.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.673+0000 7f9761a00640 1 -- 172.21.15.96:0/3758573092 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f973c06cbf0 msgr2=0x7f973c06f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:39.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.673+0000 7f9761a00640 1 --2- 172.21.15.96:0/3758573092 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f973c06cbf0 0x7f973c06f0b0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f9748004770 tx=0x7f9748008040 comp rx=0 tx=0).stop 2024-05-28T06:08:39.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.673+0000 7f9761a00640 1 -- 172.21.15.96:0/3758573092 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9740004d40 msgr2=0x7f975c15f200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:39.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.673+0000 7f9761a00640 1 --2- 172.21.15.96:0/3758573092 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9740004d40 0x7f975c15f200 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f974c011660 tx=0x7f974c011b20 comp rx=0 tx=0).stop 2024-05-28T06:08:39.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.674+0000 7f9761a00640 1 -- 172.21.15.96:0/3758573092 shutdown_connections 2024-05-28T06:08:39.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.674+0000 7f9761a00640 1 --2- 172.21.15.96:0/3758573092 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f973c06cbf0 0x7f973c06f0b0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:39.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.674+0000 7f9761a00640 1 --2- 172.21.15.96:0/3758573092 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f975c0b10e0 0x7f975c0b1560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:39.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.674+0000 7f9761a00640 1 --2- 172.21.15.96:0/3758573092 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f975c0c7d40 0x7f975c099850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:39.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.674+0000 7f9761a00640 1 --2- 172.21.15.96:0/3758573092 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9740004d40 0x7f975c15f200 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:39.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.674+0000 7f9761a00640 1 -- 172.21.15.96:0/3758573092 >> 172.21.15.96:0/3758573092 conn(0x7f975c06bae0 msgr2=0x7f975c0cbe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:39.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.674+0000 7f9761a00640 1 -- 172.21.15.96:0/3758573092 shutdown_connections 2024-05-28T06:08:39.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.674+0000 7f9761a00640 1 -- 172.21.15.96:0/3758573092 wait complete. 2024-05-28T06:08:39.692 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:08:39.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.884+0000 7fe321a00640 1 Processor -- start 2024-05-28T06:08:39.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.884+0000 7fe321a00640 1 -- start start 2024-05-28T06:08:39.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.885+0000 7fe321a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fe31c158b80 0x7fe31c06dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:39.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.886+0000 7fe321a00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe31c06e6e0 con 0x7fe31c0722d0 2024-05-28T06:08:39.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.886+0000 7fe321a00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe31c06e9c0 con 0x7fe31c06e160 2024-05-28T06:08:39.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.886+0000 7fe321a00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fe31c06eca0 con 0x7fe31c158b80 2024-05-28T06:08:39.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.886+0000 7fe31be00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fe31c158b80 0x7fe31c06dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:39.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.886+0000 7fe321000640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fe31c0722d0 0x7fe31c0b32c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:45558/0 (socket says 172.21.15.96:45558) 2024-05-28T06:08:39.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.886+0000 7fe321000640 1 -- 172.21.15.96:0/116693767 learned_addr learned my addr 172.21.15.96:0/116693767 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:08:39.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.887+0000 7fe31be00640 1 -- 172.21.15.96:0/116693767 >> v1:172.21.15.123:6789/0 conn(0x7fe31c06e160 legacy=0x7fe31c071bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:39.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.887+0000 7fe31be00640 1 -- 172.21.15.96:0/116693767 >> v1:172.21.15.96:6789/0 conn(0x7fe31c0722d0 legacy=0x7fe31c0b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:39.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.887+0000 7fe31be00640 1 -- 172.21.15.96:0/116693767 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe31c06ef80 con 0x7fe31c158b80 2024-05-28T06:08:39.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.888+0000 7fe31be00640 1 --2- 172.21.15.96:0/116693767 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe31c158b80 0x7fe31c06dc20 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fe30c0097a0 tx=0x7fe30c02a0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b5802f510f5f02c server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:39.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.888+0000 7fe31aa00640 1 -- 172.21.15.96:0/116693767 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe30c037070 con 0x7fe31c158b80 2024-05-28T06:08:39.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.888+0000 7fe31aa00640 1 -- 172.21.15.96:0/116693767 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe30c02ad60 con 0x7fe31c158b80 2024-05-28T06:08:39.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.888+0000 7fe31aa00640 1 -- 172.21.15.96:0/116693767 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe30c032660 con 0x7fe31c158b80 2024-05-28T06:08:39.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.889+0000 7fe321a00640 1 -- 172.21.15.96:0/116693767 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe31c158b80 msgr2=0x7fe31c06dc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:39.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.889+0000 7fe321a00640 1 --2- 172.21.15.96:0/116693767 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe31c158b80 0x7fe31c06dc20 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fe30c0097a0 tx=0x7fe30c02a0d0 comp rx=0 tx=0).stop 2024-05-28T06:08:39.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.889+0000 7fe321a00640 1 -- 172.21.15.96:0/116693767 shutdown_connections 2024-05-28T06:08:39.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.889+0000 7fe321a00640 1 --2- 172.21.15.96:0/116693767 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe31c158b80 0x7fe31c06dc20 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:39.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.889+0000 7fe321a00640 1 -- 172.21.15.96:0/116693767 >> 172.21.15.96:0/116693767 conn(0x7fe31c0b1fc0 msgr2=0x7fe31c0b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:39.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.889+0000 7fe321a00640 1 -- 172.21.15.96:0/116693767 shutdown_connections 2024-05-28T06:08:39.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.889+0000 7fe321a00640 1 -- 172.21.15.96:0/116693767 wait complete. 2024-05-28T06:08:39.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.891+0000 7fe321a00640 1 Processor -- start 2024-05-28T06:08:39.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.891+0000 7fe321a00640 1 -- start start 2024-05-28T06:08:39.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.892+0000 7fe321a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe31c06e160 0x7fe31c152780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:39.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.892+0000 7fe321a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe31c0722d0 0x7fe31c1f4800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:39.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.892+0000 7fe321a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe31c158b80 0x7fe31c160660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:39.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.892+0000 7fe321a00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe31c06f780 con 0x7fe31c158b80 2024-05-28T06:08:39.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.892+0000 7fe321a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe31c0b2d30 con 0x7fe31c06e160 2024-05-28T06:08:39.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.892+0000 7fe321a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fe31c0aef40 con 0x7fe31c0722d0 2024-05-28T06:08:39.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.892+0000 7fe321000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe31c158b80 0x7fe31c160660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:39.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.892+0000 7fe321000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe31c158b80 0x7fe31c160660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58092/0 (socket says 172.21.15.96:58092) 2024-05-28T06:08:39.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.892+0000 7fe321000640 1 -- 172.21.15.96:0/2585512712 learned_addr learned my addr 172.21.15.96:0/2585512712 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:08:39.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.892+0000 7fe321000640 1 -- 172.21.15.96:0/2585512712 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe31c0722d0 msgr2=0x7fe31c1f4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:39.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.892+0000 7fe321000640 1 --2- 172.21.15.96:0/2585512712 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe31c0722d0 0x7fe31c1f4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:39.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.892+0000 7fe321000640 1 -- 172.21.15.96:0/2585512712 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe31c06e160 msgr2=0x7fe31c152780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:39.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.892+0000 7fe321000640 1 --2- 172.21.15.96:0/2585512712 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe31c06e160 0x7fe31c152780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:39.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.892+0000 7fe321000640 1 -- 172.21.15.96:0/2585512712 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe31c09c9c0 con 0x7fe31c158b80 2024-05-28T06:08:39.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.893+0000 7fe321000640 1 --2- 172.21.15.96:0/2585512712 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe31c158b80 0x7fe31c160660 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fe31400c890 tx=0x7fe31400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:39.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.893+0000 7fe313e00640 1 -- 172.21.15.96:0/2585512712 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe314018070 con 0x7fe31c158b80 2024-05-28T06:08:39.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.893+0000 7fe313e00640 1 -- 172.21.15.96:0/2585512712 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe31400a640 con 0x7fe31c158b80 2024-05-28T06:08:39.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.893+0000 7fe313e00640 1 -- 172.21.15.96:0/2585512712 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe31400aa30 con 0x7fe31c158b80 2024-05-28T06:08:39.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.893+0000 7fe321a00640 1 -- 172.21.15.96:0/2585512712 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe31c09b540 con 0x7fe31c158b80 2024-05-28T06:08:39.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.893+0000 7fe321a00640 1 -- 172.21.15.96:0/2585512712 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe31c09a5e0 con 0x7fe31c158b80 2024-05-28T06:08:39.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.894+0000 7fe321a00640 1 -- 172.21.15.96:0/2585512712 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe31c0689e0 con 0x7fe31c158b80 2024-05-28T06:08:39.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.895+0000 7fe313e00640 1 -- 172.21.15.96:0/2585512712 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 9) ==== 91295+0+0 (secure 0 0 0) 0x7fe31400abd0 con 0x7fe31c158b80 2024-05-28T06:08:39.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.896+0000 7fe313e00640 1 --2- 172.21.15.96:0/2585512712 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe2fc06cbc0 0x7fe2fc06f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:39.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.896+0000 7fe313e00640 1 -- 172.21.15.96:0/2585512712 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5765+0+0 (secure 0 0 0) 0x7fe31408b800 con 0x7fe31c158b80 2024-05-28T06:08:39.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.897+0000 7fe31be00640 1 --2- 172.21.15.96:0/2585512712 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe2fc06cbc0 0x7fe2fc06f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:39.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.898+0000 7fe31be00640 1 --2- 172.21.15.96:0/2585512712 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe2fc06cbc0 0x7fe2fc06f080 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fe30c034000 tx=0x7fe30c02c000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:39.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:39.900+0000 7fe313e00640 1 -- 172.21.15.96:0/2585512712 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe314059810 con 0x7fe31c158b80 2024-05-28T06:08:40.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.085+0000 7fe321a00640 1 -- 172.21.15.96:0/2585512712 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe31c06cbc0 con 0x7fe31c158b80 2024-05-28T06:08:40.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.088+0000 7fe313e00640 1 -- 172.21.15.96:0/2585512712 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) ==== 74+0+16789 (secure 0 0 0) 0x7fe314059110 con 0x7fe31c158b80 2024-05-28T06:08:40.089 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:08:40.089 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":34,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:08:37.595959+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-05-28T06:05:27.674327+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-05-28T06:05:28.878204+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":34,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[{"pool":5,"snaps":[{"begin":2,"length":2}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:08:40.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.090+0000 7fe321a00640 1 -- 172.21.15.96:0/2585512712 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe2fc06cbc0 msgr2=0x7fe2fc06f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:40.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.090+0000 7fe321a00640 1 --2- 172.21.15.96:0/2585512712 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe2fc06cbc0 0x7fe2fc06f080 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fe30c034000 tx=0x7fe30c02c000 comp rx=0 tx=0).stop 2024-05-28T06:08:40.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.091+0000 7fe321a00640 1 -- 172.21.15.96:0/2585512712 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe31c158b80 msgr2=0x7fe31c160660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:40.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.091+0000 7fe321a00640 1 --2- 172.21.15.96:0/2585512712 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe31c158b80 0x7fe31c160660 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fe31400c890 tx=0x7fe31400cd50 comp rx=0 tx=0).stop 2024-05-28T06:08:40.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.091+0000 7fe321a00640 1 -- 172.21.15.96:0/2585512712 shutdown_connections 2024-05-28T06:08:40.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.091+0000 7fe321a00640 1 --2- 172.21.15.96:0/2585512712 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe2fc06cbc0 0x7fe2fc06f080 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:40.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.091+0000 7fe321a00640 1 --2- 172.21.15.96:0/2585512712 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe31c158b80 0x7fe31c160660 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:40.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.091+0000 7fe321a00640 1 --2- 172.21.15.96:0/2585512712 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe31c0722d0 0x7fe31c1f4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:40.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.091+0000 7fe321a00640 1 --2- 172.21.15.96:0/2585512712 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe31c06e160 0x7fe31c152780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:40.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.091+0000 7fe321a00640 1 -- 172.21.15.96:0/2585512712 >> 172.21.15.96:0/2585512712 conn(0x7fe31c0b1fc0 msgr2=0x7fe31c157990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:40.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.091+0000 7fe321a00640 1 -- 172.21.15.96:0/2585512712 shutdown_connections 2024-05-28T06:08:40.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.091+0000 7fe321a00640 1 -- 172.21.15.96:0/2585512712 wait complete. 2024-05-28T06:08:40.109 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-05-28T06:08:40.109 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-28T06:08:40.305 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.304+0000 7f241fe00640 1 Processor -- start 2024-05-28T06:08:40.305 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.304+0000 7f241fe00640 1 -- start start 2024-05-28T06:08:40.306 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.306+0000 7f241fe00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f24200ad940 0x7f24200b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:40.306 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.306+0000 7f241fe00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2420156f10 con 0x7f242006ee40 2024-05-28T06:08:40.306 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.306+0000 7f241fe00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f242006dda0 con 0x7f2420156b30 2024-05-28T06:08:40.306 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.306+0000 7f241fe00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f242006e080 con 0x7f24200ad940 2024-05-28T06:08:40.306 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.306+0000 7f241e000640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f242006ee40 0x7f24200ad230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:45562/0 (socket says 172.21.15.96:45562) 2024-05-28T06:08:40.307 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.306+0000 7f241e000640 1 -- 172.21.15.96:0/13363638 learned_addr learned my addr 172.21.15.96:0/13363638 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:08:40.308 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.306+0000 7f241d600640 1 -- 172.21.15.96:0/13363638 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 844016006 0 0) 0x7f2420156f10 con 0x7f242006ee40 2024-05-28T06:08:40.308 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.306+0000 7f241f400640 1 --2- 172.21.15.96:0/13363638 >> v2:172.21.15.123:3300/0 conn(0x7f24200ad940 0x7f24200b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:40.308 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.306+0000 7f241d600640 1 -- 172.21.15.96:0/13363638 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2404003520 con 0x7f242006ee40 2024-05-28T06:08:40.308 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.307+0000 7f241d600640 1 -- 172.21.15.96:0/13363638 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 41580473 0 0) 0x7f2404003520 con 0x7f242006ee40 2024-05-28T06:08:40.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.307+0000 7f241d600640 1 -- 172.21.15.96:0/13363638 >> v2:172.21.15.123:3300/0 conn(0x7f24200ad940 msgr2=0x7f24200b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:40.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.307+0000 7f241d600640 1 --2- 172.21.15.96:0/13363638 >> v2:172.21.15.123:3300/0 conn(0x7f24200ad940 0x7f24200b32c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:40.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.307+0000 7f241d600640 1 -- 172.21.15.96:0/13363638 >> v1:172.21.15.123:6789/0 conn(0x7f2420156b30 legacy=0x7f24200af6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:40.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.307+0000 7f241d600640 1 -- 172.21.15.96:0/13363638 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f242006e360 con 0x7f242006ee40 2024-05-28T06:08:40.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.307+0000 7f241d600640 1 -- 172.21.15.96:0/13363638 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f240c001670 con 0x7f242006ee40 2024-05-28T06:08:40.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.307+0000 7f241d600640 1 -- 172.21.15.96:0/13363638 >> v1:172.21.15.96:6789/0 conn(0x7f242006ee40 legacy=0x7f24200ad230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:40.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.307+0000 7f241d600640 1 --2- 172.21.15.96:0/13363638 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2404003a00 0x7f2404003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:40.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.307+0000 7f241d600640 1 --2- 172.21.15.96:0/13363638 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2404004800 0x7f2404004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:40.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.307+0000 7f241d600640 1 --2- 172.21.15.96:0/13363638 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2404005210 0x7f240400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:40.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.307+0000 7f241d600640 1 -- 172.21.15.96:0/13363638 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f242006e080 con 0x7f2404005210 2024-05-28T06:08:40.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.307+0000 7f241d600640 1 -- 172.21.15.96:0/13363638 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f240400c160 con 0x7f2404003a00 2024-05-28T06:08:40.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.307+0000 7f241d600640 1 -- 172.21.15.96:0/13363638 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f240400c360 con 0x7f2404004800 2024-05-28T06:08:40.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.307+0000 7f241f400640 1 --2- 172.21.15.96:0/13363638 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2404005210 0x7f240400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:40.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.308+0000 7f241f400640 1 -- 172.21.15.96:0/13363638 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2404004800 msgr2=0x7f2404004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:40.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.308+0000 7f241f400640 1 --2- 172.21.15.96:0/13363638 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2404004800 0x7f2404004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:40.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.308+0000 7f241f400640 1 -- 172.21.15.96:0/13363638 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2404003a00 msgr2=0x7f2404003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:40.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.308+0000 7f241f400640 1 --2- 172.21.15.96:0/13363638 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2404003a00 0x7f2404003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:40.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.308+0000 7f241f400640 1 -- 172.21.15.96:0/13363638 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2404003520 con 0x7f2404005210 2024-05-28T06:08:40.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.308+0000 7f241f400640 1 -- 172.21.15.96:0/13363638 reap_dead start 2024-05-28T06:08:40.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.309+0000 7f241f400640 1 --2- 172.21.15.96:0/13363638 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2404005210 0x7f240400b9c0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f2418009f90 tx=0x7f2404005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48e8c856b1cf7de3 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:40.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.309+0000 7f241d600640 1 -- 172.21.15.96:0/13363638 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2418035020 con 0x7f2404005210 2024-05-28T06:08:40.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.309+0000 7f241d600640 1 -- 172.21.15.96:0/13363638 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2418003b90 con 0x7f2404005210 2024-05-28T06:08:40.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.309+0000 7f241d600640 1 -- 172.21.15.96:0/13363638 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2418009c10 con 0x7f2404005210 2024-05-28T06:08:40.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.309+0000 7f241fe00640 1 -- 172.21.15.96:0/13363638 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2404005210 msgr2=0x7f240400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:40.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.309+0000 7f241fe00640 1 --2- 172.21.15.96:0/13363638 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2404005210 0x7f240400b9c0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f2418009f90 tx=0x7f2404005830 comp rx=0 tx=0).stop 2024-05-28T06:08:40.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.309+0000 7f241fe00640 1 -- 172.21.15.96:0/13363638 shutdown_connections 2024-05-28T06:08:40.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.309+0000 7f241fe00640 1 --2- 172.21.15.96:0/13363638 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2404005210 0x7f240400b9c0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:40.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.309+0000 7f241fe00640 1 -- 172.21.15.96:0/13363638 >> 172.21.15.96:0/13363638 conn(0x7f24200b1fc0 msgr2=0x7f24200b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:40.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.310+0000 7f241fe00640 1 -- 172.21.15.96:0/13363638 shutdown_connections 2024-05-28T06:08:40.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.310+0000 7f241fe00640 1 -- 172.21.15.96:0/13363638 wait complete. 2024-05-28T06:08:40.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.311+0000 7f241fe00640 1 Processor -- start 2024-05-28T06:08:40.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.311+0000 7f241fe00640 1 -- start start 2024-05-28T06:08:40.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.312+0000 7f241fe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2404005210 0x7f24200b12c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:40.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.312+0000 7f241fe00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2420156f10 0x7f2420155b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:40.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.312+0000 7f241fe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2420142030 0x7f24201424b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:40.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.312+0000 7f241fe00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f24200b2db0 con 0x7f2404005210 2024-05-28T06:08:40.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.312+0000 7f241fe00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f24200b2c30 con 0x7f2420142030 2024-05-28T06:08:40.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.312+0000 7f241fe00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f242006f2f0 con 0x7f2420156f10 2024-05-28T06:08:40.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.312+0000 7f241ea00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2404005210 0x7f24200b12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:40.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.312+0000 7f241ea00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2404005210 0x7f24200b12c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58098/0 (socket says 172.21.15.96:58098) 2024-05-28T06:08:40.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.313+0000 7f241ea00640 1 -- 172.21.15.96:0/2665204318 learned_addr learned my addr 172.21.15.96:0/2665204318 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:08:40.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.313+0000 7f241ea00640 1 -- 172.21.15.96:0/2665204318 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2420156f10 msgr2=0x7f2420155b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:40.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.313+0000 7f241ea00640 1 --2- 172.21.15.96:0/2665204318 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2420156f10 0x7f2420155b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:40.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.313+0000 7f241ea00640 1 -- 172.21.15.96:0/2665204318 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2420142030 msgr2=0x7f24201424b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:40.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.313+0000 7f241ea00640 1 --2- 172.21.15.96:0/2665204318 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2420142030 0x7f24201424b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:40.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.313+0000 7f241ea00640 1 -- 172.21.15.96:0/2665204318 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2420159d00 con 0x7f2404005210 2024-05-28T06:08:40.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.313+0000 7f241ea00640 1 --2- 172.21.15.96:0/2665204318 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2404005210 0x7f24200b12c0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f241000c430 tx=0x7f241000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:40.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.313+0000 7f2416a00640 1 -- 172.21.15.96:0/2665204318 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2410017070 con 0x7f2404005210 2024-05-28T06:08:40.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.313+0000 7f2416a00640 1 -- 172.21.15.96:0/2665204318 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24100052d0 con 0x7f2404005210 2024-05-28T06:08:40.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.313+0000 7f2416a00640 1 -- 172.21.15.96:0/2665204318 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2410003830 con 0x7f2404005210 2024-05-28T06:08:40.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.313+0000 7f241fe00640 1 -- 172.21.15.96:0/2665204318 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f242014bef0 con 0x7f2404005210 2024-05-28T06:08:40.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.314+0000 7f241fe00640 1 -- 172.21.15.96:0/2665204318 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2420159500 con 0x7f2404005210 2024-05-28T06:08:40.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.315+0000 7f241fe00640 1 -- 172.21.15.96:0/2665204318 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f242006faa0 con 0x7f2404005210 2024-05-28T06:08:40.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.316+0000 7f2416a00640 1 -- 172.21.15.96:0/2665204318 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 9) ==== 91295+0+0 (secure 0 0 0) 0x7f24100039d0 con 0x7f2404005210 2024-05-28T06:08:40.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.316+0000 7f2416a00640 1 --2- 172.21.15.96:0/2665204318 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f240006cbf0 0x7f240006f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:40.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.317+0000 7f2416a00640 1 -- 172.21.15.96:0/2665204318 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5765+0+0 (secure 0 0 0) 0x7f241008b490 con 0x7f2404005210 2024-05-28T06:08:40.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.317+0000 7f241e000640 1 --2- 172.21.15.96:0/2665204318 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f240006cbf0 0x7f240006f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:40.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.318+0000 7f241e000640 1 --2- 172.21.15.96:0/2665204318 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f240006cbf0 0x7f240006f0b0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f24200d12e0 tx=0x7f240c0014f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:40.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.321+0000 7f2416a00640 1 -- 172.21.15.96:0/2665204318 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f242006faa0 con 0x7f2404005210 2024-05-28T06:08:40.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.555+0000 7f241fe00640 1 -- 172.21.15.96:0/2665204318 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f24200689e0 con 0x7f2404005210 2024-05-28T06:08:40.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.556+0000 7f2416a00640 1 -- 172.21.15.96:0/2665204318 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+5178 (secure 0 0 0) 0x7f2410059410 con 0x7f2404005210 2024-05-28T06:08:40.557 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:08:40.557 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4580,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6836/918905879","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":918905879},{"type":"v1","addr":"172.21.15.123:6836","nonce":918905879}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":5223,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6834/697317719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6833","nonce":697317719},{"type":"v1","addr":"172.21.15.96:6834","nonce":697317719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":5226,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/1327552826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":1327552826},{"type":"v1","addr":"172.21.15.96:6835","nonce":1327552826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-28T06:05:29.892912+0000","modified":"2024-05-28T06:05:32.976408+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5238":{"gid":5238,"name":"d","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6837/458334664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6835","nonce":458334664},{"type":"v1","addr":"172.21.15.123:6837","nonce":458334664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5238,"qdb_cluster":[5238]},"id":2}]} 2024-05-28T06:08:40.557 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 16 2024-05-28T06:08:40.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.558+0000 7f241fe00640 1 -- 172.21.15.96:0/2665204318 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f240006cbf0 msgr2=0x7f240006f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:40.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.558+0000 7f241fe00640 1 --2- 172.21.15.96:0/2665204318 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f240006cbf0 0x7f240006f0b0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f24200d12e0 tx=0x7f240c0014f0 comp rx=0 tx=0).stop 2024-05-28T06:08:40.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.559+0000 7f241fe00640 1 -- 172.21.15.96:0/2665204318 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2404005210 msgr2=0x7f24200b12c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:40.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.559+0000 7f241fe00640 1 --2- 172.21.15.96:0/2665204318 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2404005210 0x7f24200b12c0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f241000c430 tx=0x7f241000c8f0 comp rx=0 tx=0).stop 2024-05-28T06:08:40.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.559+0000 7f241fe00640 1 -- 172.21.15.96:0/2665204318 shutdown_connections 2024-05-28T06:08:40.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.559+0000 7f241fe00640 1 --2- 172.21.15.96:0/2665204318 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f240006cbf0 0x7f240006f0b0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:40.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.559+0000 7f241fe00640 1 --2- 172.21.15.96:0/2665204318 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2420142030 0x7f24201424b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:40.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.559+0000 7f241fe00640 1 --2- 172.21.15.96:0/2665204318 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2420156f10 0x7f2420155b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:40.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.559+0000 7f241fe00640 1 --2- 172.21.15.96:0/2665204318 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2404005210 0x7f24200b12c0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:40.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.559+0000 7f241fe00640 1 -- 172.21.15.96:0/2665204318 >> 172.21.15.96:0/2665204318 conn(0x7f24200b1fc0 msgr2=0x7f242015b530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:40.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.559+0000 7f241fe00640 1 -- 172.21.15.96:0/2665204318 shutdown_connections 2024-05-28T06:08:40.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.559+0000 7f241fe00640 1 -- 172.21.15.96:0/2665204318 wait complete. 2024-05-28T06:08:40.577 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-28T06:08:40.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.766+0000 7f410ac00640 1 Processor -- start 2024-05-28T06:08:40.767 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.766+0000 7f410ac00640 1 -- start start 2024-05-28T06:08:40.767 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.767+0000 7f410ac00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f41040b4e10 0x7f41040cde30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:40.768 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.767+0000 7f410ac00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f41040d0ca0 0x7f4104099090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:40.768 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.767+0000 7f410ac00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41040b51f0 con 0x7f41040ce370 2024-05-28T06:08:40.768 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.767+0000 7f410ac00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f41040aef60 con 0x7f41040d0ca0 2024-05-28T06:08:40.768 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.767+0000 7f410ac00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f41040ad1d0 con 0x7f41040b4e10 2024-05-28T06:08:40.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.768+0000 7f4103400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f41040ce370 0x7f41040d0590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:45564/0 (socket says 172.21.15.96:45564) 2024-05-28T06:08:40.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.768+0000 7f4108c00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f41040d0ca0 0x7f4104099090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:40.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.768+0000 7f4103400640 1 -- 172.21.15.96:0/1020685689 learned_addr learned my addr 172.21.15.96:0/1020685689 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:08:40.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.768+0000 7f4103e00640 1 --2- 172.21.15.96:0/1020685689 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f41040b4e10 0x7f41040cde30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:40.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.768+0000 7f4108c00640 1 -- 172.21.15.96:0/1020685689 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f41040b4e10 msgr2=0x7f41040cde30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:40.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.768+0000 7f4108c00640 1 --2- 172.21.15.96:0/1020685689 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f41040b4e10 0x7f41040cde30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:40.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.768+0000 7f4108c00640 1 -- 172.21.15.96:0/1020685689 >> v1:172.21.15.96:6789/0 conn(0x7f41040ce370 legacy=0x7f41040d0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:40.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.768+0000 7f4108c00640 1 -- 172.21.15.96:0/1020685689 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41040ad470 con 0x7f41040d0ca0 2024-05-28T06:08:40.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.768+0000 7f4102a00640 1 -- 172.21.15.96:0/1020685689 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3213375650 0 0) 0x7f41040b51f0 con 0x7f41040ce370 2024-05-28T06:08:40.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.769+0000 7f4108c00640 1 --2- 172.21.15.96:0/1020685689 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41040d0ca0 0x7f4104099090 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f40fc009d90 tx=0x7f40fc02a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81f8fedf7c685bf8 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:40.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.770+0000 7f4102a00640 1 -- 172.21.15.96:0/1020685689 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40fc031600 con 0x7f41040d0ca0 2024-05-28T06:08:40.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.770+0000 7f4102a00640 1 -- 172.21.15.96:0/1020685689 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40fc0317a0 con 0x7f41040d0ca0 2024-05-28T06:08:40.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.770+0000 7f4102a00640 1 -- 172.21.15.96:0/1020685689 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40fc031b70 con 0x7f41040d0ca0 2024-05-28T06:08:40.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.770+0000 7f410ac00640 1 -- 172.21.15.96:0/1020685689 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41040d0ca0 msgr2=0x7f4104099090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:40.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.770+0000 7f410ac00640 1 --2- 172.21.15.96:0/1020685689 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41040d0ca0 0x7f4104099090 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f40fc009d90 tx=0x7f40fc02a5d0 comp rx=0 tx=0).stop 2024-05-28T06:08:40.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.770+0000 7f410ac00640 1 -- 172.21.15.96:0/1020685689 shutdown_connections 2024-05-28T06:08:40.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.770+0000 7f410ac00640 1 --2- 172.21.15.96:0/1020685689 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41040d0ca0 0x7f4104099090 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:40.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.770+0000 7f410ac00640 1 --2- 172.21.15.96:0/1020685689 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f41040b4e10 0x7f41040cde30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:40.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.770+0000 7f410ac00640 1 -- 172.21.15.96:0/1020685689 >> 172.21.15.96:0/1020685689 conn(0x7f410406bae0 msgr2=0x7f410406bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:40.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.770+0000 7f410ac00640 1 -- 172.21.15.96:0/1020685689 shutdown_connections 2024-05-28T06:08:40.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.770+0000 7f410ac00640 1 -- 172.21.15.96:0/1020685689 wait complete. 2024-05-28T06:08:40.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.772+0000 7f410ac00640 1 Processor -- start 2024-05-28T06:08:40.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.772+0000 7f410ac00640 1 -- start start 2024-05-28T06:08:40.775 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.773+0000 7f410ac00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41040b4e10 0x7f41041f47b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:40.775 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.773+0000 7f410ac00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f41040ce370 0x7f41040b2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:40.775 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.773+0000 7f410ac00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41040d0ca0 0x7f41040b2740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:40.775 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.773+0000 7f410ac00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f41040cea00 con 0x7f41040d0ca0 2024-05-28T06:08:40.775 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.773+0000 7f410ac00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f41040ce880 con 0x7f41040b4e10 2024-05-28T06:08:40.775 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.773+0000 7f410ac00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f41040b3150 con 0x7f41040ce370 2024-05-28T06:08:40.775 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.774+0000 7f4108c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41040d0ca0 0x7f41040b2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:40.775 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.774+0000 7f4108c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41040d0ca0 0x7f41040b2740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58110/0 (socket says 172.21.15.96:58110) 2024-05-28T06:08:40.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.774+0000 7f4108c00640 1 -- 172.21.15.96:0/303008148 learned_addr learned my addr 172.21.15.96:0/303008148 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:08:40.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.774+0000 7f4103e00640 1 --2- 172.21.15.96:0/303008148 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41040b4e10 0x7f41041f47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:40.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.774+0000 7f4103400640 1 --2- 172.21.15.96:0/303008148 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f41040ce370 0x7f41040b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:40.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.774+0000 7f4108c00640 1 -- 172.21.15.96:0/303008148 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f41040ce370 msgr2=0x7f41040b2200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:40.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.774+0000 7f4108c00640 1 --2- 172.21.15.96:0/303008148 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f41040ce370 0x7f41040b2200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:40.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.774+0000 7f4108c00640 1 -- 172.21.15.96:0/303008148 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41040b4e10 msgr2=0x7f41041f47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:40.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.774+0000 7f4108c00640 1 --2- 172.21.15.96:0/303008148 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41040b4e10 0x7f41041f47b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:40.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.774+0000 7f4108c00640 1 -- 172.21.15.96:0/303008148 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41040725e0 con 0x7f41040d0ca0 2024-05-28T06:08:40.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.774+0000 7f4103e00640 1 --2- 172.21.15.96:0/303008148 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41040b4e10 0x7f41041f47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:08:40.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.774+0000 7f4108c00640 1 --2- 172.21.15.96:0/303008148 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41040d0ca0 0x7f41040b2740 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f40fc009eb0 tx=0x7f40fc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:40.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.774+0000 7f40fbe00640 1 -- 172.21.15.96:0/303008148 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40fc040070 con 0x7f41040d0ca0 2024-05-28T06:08:40.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.775+0000 7f40fbe00640 1 -- 172.21.15.96:0/303008148 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40fc030be0 con 0x7f41040d0ca0 2024-05-28T06:08:40.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.775+0000 7f40fbe00640 1 -- 172.21.15.96:0/303008148 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40fc009450 con 0x7f41040d0ca0 2024-05-28T06:08:40.777 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.775+0000 7f410ac00640 1 -- 172.21.15.96:0/303008148 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4104073640 con 0x7f41040d0ca0 2024-05-28T06:08:40.777 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.775+0000 7f410ac00640 1 -- 172.21.15.96:0/303008148 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4104072b60 con 0x7f41040d0ca0 2024-05-28T06:08:40.777 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.776+0000 7f410ac00640 1 -- 172.21.15.96:0/303008148 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41040ceb80 con 0x7f41040d0ca0 2024-05-28T06:08:40.777 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.777+0000 7f40fbe00640 1 -- 172.21.15.96:0/303008148 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 9) ==== 91295+0+0 (secure 0 0 0) 0x7f40fc039040 con 0x7f41040d0ca0 2024-05-28T06:08:40.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.778+0000 7f40fbe00640 1 --2- 172.21.15.96:0/303008148 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f40e406cbf0 0x7f40e406f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:40.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.778+0000 7f40fbe00640 1 -- 172.21.15.96:0/303008148 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5765+0+0 (secure 0 0 0) 0x7f40fc0ad240 con 0x7f41040d0ca0 2024-05-28T06:08:40.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.778+0000 7f4103e00640 1 --2- 172.21.15.96:0/303008148 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f40e406cbf0 0x7f40e406f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:40.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.780+0000 7f4103e00640 1 --2- 172.21.15.96:0/303008148 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f40e406cbf0 0x7f40e406f0b0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f40f4009a20 tx=0x7f40f4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:40.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:40.782+0000 7f40fbe00640 1 -- 172.21.15.96:0/303008148 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f41040ceb80 con 0x7f41040d0ca0 2024-05-28T06:08:41.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.018+0000 7f410ac00640 1 -- 172.21.15.96:0/303008148 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f41040689e0 con 0x7f41040d0ca0 2024-05-28T06:08:41.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.020+0000 7f40fbe00640 1 -- 172.21.15.96:0/303008148 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+5178 (secure 0 0 0) 0x7f40fc07b020 con 0x7f41040d0ca0 2024-05-28T06:08:41.020 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:08:41.021 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4580,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6836/918905879","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":918905879},{"type":"v1","addr":"172.21.15.123:6836","nonce":918905879}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":5223,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6834/697317719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6833","nonce":697317719},{"type":"v1","addr":"172.21.15.96:6834","nonce":697317719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":5226,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/1327552826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":1327552826},{"type":"v1","addr":"172.21.15.96:6835","nonce":1327552826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-28T06:05:29.892912+0000","modified":"2024-05-28T06:05:32.976408+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5238":{"gid":5238,"name":"d","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.123:6837/458334664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6835","nonce":458334664},{"type":"v1","addr":"172.21.15.123:6837","nonce":458334664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5238,"qdb_cluster":[5238]},"id":2}]} 2024-05-28T06:08:41.021 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 16 2024-05-28T06:08:41.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.022+0000 7f410ac00640 1 -- 172.21.15.96:0/303008148 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f40e406cbf0 msgr2=0x7f40e406f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:41.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.022+0000 7f410ac00640 1 --2- 172.21.15.96:0/303008148 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f40e406cbf0 0x7f40e406f0b0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f40f4009a20 tx=0x7f40f4008040 comp rx=0 tx=0).stop 2024-05-28T06:08:41.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.022+0000 7f410ac00640 1 -- 172.21.15.96:0/303008148 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41040d0ca0 msgr2=0x7f41040b2740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:41.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.022+0000 7f410ac00640 1 --2- 172.21.15.96:0/303008148 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41040d0ca0 0x7f41040b2740 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f40fc009eb0 tx=0x7f40fc004a40 comp rx=0 tx=0).stop 2024-05-28T06:08:41.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.022+0000 7f410ac00640 1 -- 172.21.15.96:0/303008148 shutdown_connections 2024-05-28T06:08:41.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.022+0000 7f410ac00640 1 --2- 172.21.15.96:0/303008148 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f40e406cbf0 0x7f40e406f0b0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:41.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.022+0000 7f410ac00640 1 --2- 172.21.15.96:0/303008148 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41040d0ca0 0x7f41040b2740 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:41.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.022+0000 7f410ac00640 1 --2- 172.21.15.96:0/303008148 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f41040ce370 0x7f41040b2200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:41.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.022+0000 7f410ac00640 1 --2- 172.21.15.96:0/303008148 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41040b4e10 0x7f41041f47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:41.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.022+0000 7f410ac00640 1 -- 172.21.15.96:0/303008148 >> 172.21.15.96:0/303008148 conn(0x7f410406bae0 msgr2=0x7f41040ca840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:41.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.022+0000 7f410ac00640 1 -- 172.21.15.96:0/303008148 shutdown_connections 2024-05-28T06:08:41.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.022+0000 7f410ac00640 1 -- 172.21.15.96:0/303008148 wait complete. 2024-05-28T06:08:41.041 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:08:41.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.239+0000 7f872b200640 1 Processor -- start 2024-05-28T06:08:41.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.239+0000 7f872b200640 1 -- start start 2024-05-28T06:08:41.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.240+0000 7f872b200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f872c156b30 0x7f872c0af6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:41.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.241+0000 7f872b200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f872c06ee40 0x7f872c0ad230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:41.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.241+0000 7f872b200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f872c156f10 con 0x7f872c06f2c0 2024-05-28T06:08:41.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.241+0000 7f872b200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f872c06dda0 con 0x7f872c06ee40 2024-05-28T06:08:41.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.241+0000 7f872b200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f872c06e080 con 0x7f872c156b30 2024-05-28T06:08:41.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.241+0000 7f872a800640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f872c06f2c0 0x7f872c0b32c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:45568/0 (socket says 172.21.15.96:45568) 2024-05-28T06:08:41.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.241+0000 7f872a800640 1 -- 172.21.15.96:0/2039347510 learned_addr learned my addr 172.21.15.96:0/2039347510 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:08:41.245 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.241+0000 7f8728a00640 1 -- 172.21.15.96:0/2039347510 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1204683646 0 0) 0x7f872c156f10 con 0x7f872c06f2c0 2024-05-28T06:08:41.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.241+0000 7f8729e00640 1 --2- 172.21.15.96:0/2039347510 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f872c156b30 0x7f872c0af6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:41.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.241+0000 7f8728a00640 1 -- 172.21.15.96:0/2039347510 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8710003520 con 0x7f872c06f2c0 2024-05-28T06:08:41.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.241+0000 7f8729400640 1 --2- 172.21.15.96:0/2039347510 >> v2:172.21.15.123:3300/0 conn(0x7f872c06ee40 0x7f872c0ad230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:41.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.242+0000 7f8728a00640 1 -- 172.21.15.96:0/2039347510 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3080094656 0 0) 0x7f8710003520 con 0x7f872c06f2c0 2024-05-28T06:08:41.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.242+0000 7f8728a00640 1 -- 172.21.15.96:0/2039347510 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f872c156b30 msgr2=0x7f872c0af6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:41.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.242+0000 7f8728a00640 1 --2- 172.21.15.96:0/2039347510 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f872c156b30 0x7f872c0af6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:41.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.242+0000 7f8728a00640 1 -- 172.21.15.96:0/2039347510 >> v2:172.21.15.123:3300/0 conn(0x7f872c06ee40 msgr2=0x7f872c0ad230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:41.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.242+0000 7f8728a00640 1 --2- 172.21.15.96:0/2039347510 >> v2:172.21.15.123:3300/0 conn(0x7f872c06ee40 0x7f872c0ad230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:41.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.242+0000 7f8728a00640 1 -- 172.21.15.96:0/2039347510 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f872c06e360 con 0x7f872c06f2c0 2024-05-28T06:08:41.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.242+0000 7f8728a00640 1 -- 172.21.15.96:0/2039347510 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f8724001670 con 0x7f872c06f2c0 2024-05-28T06:08:41.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.242+0000 7f8728a00640 1 -- 172.21.15.96:0/2039347510 >> v1:172.21.15.96:6789/0 conn(0x7f872c06f2c0 legacy=0x7f872c0b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:41.247 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.242+0000 7f8728a00640 1 --2- 172.21.15.96:0/2039347510 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f872c156b30 0x7f87100039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:41.247 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.242+0000 7f8728a00640 1 --2- 172.21.15.96:0/2039347510 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8710004030 0x7f8710004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:41.247 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.242+0000 7f8728a00640 1 --2- 172.21.15.96:0/2039347510 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8710004d40 0x7f87100051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:41.247 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.242+0000 7f8728a00640 1 -- 172.21.15.96:0/2039347510 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f872c06dda0 con 0x7f8710004d40 2024-05-28T06:08:41.247 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.242+0000 7f8728a00640 1 -- 172.21.15.96:0/2039347510 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f872c06e080 con 0x7f8710004030 2024-05-28T06:08:41.247 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.242+0000 7f8728a00640 1 -- 172.21.15.96:0/2039347510 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f871000bbd0 con 0x7f872c156b30 2024-05-28T06:08:41.247 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.242+0000 7f872a800640 1 --2- 172.21.15.96:0/2039347510 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8710004d40 0x7f87100051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:41.247 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.242+0000 7f8729e00640 1 --2- 172.21.15.96:0/2039347510 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f872c156b30 0x7f87100039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:41.247 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.243+0000 7f8729400640 1 --2- 172.21.15.96:0/2039347510 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8710004030 0x7f8710004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:41.247 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.243+0000 7f872a800640 1 -- 172.21.15.96:0/2039347510 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f872c156b30 msgr2=0x7f87100039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:41.247 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.243+0000 7f872a800640 1 --2- 172.21.15.96:0/2039347510 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f872c156b30 0x7f87100039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:41.247 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.243+0000 7f872a800640 1 -- 172.21.15.96:0/2039347510 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8710004030 msgr2=0x7f8710004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:41.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.243+0000 7f872a800640 1 --2- 172.21.15.96:0/2039347510 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8710004030 0x7f8710004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:41.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.243+0000 7f872a800640 1 -- 172.21.15.96:0/2039347510 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8710003520 con 0x7f8710004d40 2024-05-28T06:08:41.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.243+0000 7f8729400640 1 --2- 172.21.15.96:0/2039347510 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8710004030 0x7f8710004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:08:41.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.244+0000 7f872a800640 1 --2- 172.21.15.96:0/2039347510 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8710004d40 0x7f87100051c0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f872400c190 tx=0x7f8710005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1e8c003c78cf4f6 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:41.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.244+0000 7f8728a00640 1 -- 172.21.15.96:0/2039347510 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8724007c80 con 0x7f8710004d40 2024-05-28T06:08:41.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.244+0000 7f8728a00640 1 -- 172.21.15.96:0/2039347510 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8724005ad0 con 0x7f8710004d40 2024-05-28T06:08:41.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.244+0000 7f8728a00640 1 -- 172.21.15.96:0/2039347510 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8724007070 con 0x7f8710004d40 2024-05-28T06:08:41.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.244+0000 7f872b200640 1 -- 172.21.15.96:0/2039347510 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8710004d40 msgr2=0x7f87100051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:41.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.244+0000 7f872b200640 1 --2- 172.21.15.96:0/2039347510 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8710004d40 0x7f87100051c0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f872400c190 tx=0x7f8710005830 comp rx=0 tx=0).stop 2024-05-28T06:08:41.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.244+0000 7f872a800640 1 -- 172.21.15.96:0/2039347510 reap_dead start 2024-05-28T06:08:41.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.245+0000 7f872b200640 1 -- 172.21.15.96:0/2039347510 shutdown_connections 2024-05-28T06:08:41.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.245+0000 7f872b200640 1 -- 172.21.15.96:0/2039347510 >> 172.21.15.96:0/2039347510 conn(0x7f872c0b1fc0 msgr2=0x7f872c0b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:41.249 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.245+0000 7f872b200640 1 -- 172.21.15.96:0/2039347510 shutdown_connections 2024-05-28T06:08:41.249 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.245+0000 7f872b200640 1 -- 172.21.15.96:0/2039347510 wait complete. 2024-05-28T06:08:41.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.246+0000 7f872b200640 1 Processor -- start 2024-05-28T06:08:41.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.246+0000 7f872b200640 1 -- start start 2024-05-28T06:08:41.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.247+0000 7f872b200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8710004d40 0x7f872c159130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:41.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.247+0000 7f872b200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f872c1514b0 0x7f872c0b1810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:41.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.247+0000 7f872b200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f872c1538c0 0x7f872c0b4080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:41.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.247+0000 7f872b200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f872c0b2db0 con 0x7f8710004d40 2024-05-28T06:08:41.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.247+0000 7f872b200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f872c0b2c30 con 0x7f872c1538c0 2024-05-28T06:08:41.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.247+0000 7f872b200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f872c0b39d0 con 0x7f872c1514b0 2024-05-28T06:08:41.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.247+0000 7f8729e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8710004d40 0x7f872c159130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:41.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.248+0000 7f8729e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8710004d40 0x7f872c159130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58126/0 (socket says 172.21.15.96:58126) 2024-05-28T06:08:41.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.248+0000 7f8729e00640 1 -- 172.21.15.96:0/2742966446 learned_addr learned my addr 172.21.15.96:0/2742966446 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:08:41.252 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.248+0000 7f8729e00640 1 -- 172.21.15.96:0/2742966446 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f872c1514b0 msgr2=0x7f872c0b1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:41.252 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.248+0000 7f8729e00640 1 --2- 172.21.15.96:0/2742966446 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f872c1514b0 0x7f872c0b1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:41.252 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.248+0000 7f8729e00640 1 -- 172.21.15.96:0/2742966446 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f872c1538c0 msgr2=0x7f872c0b4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:41.252 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.248+0000 7f8729e00640 1 --2- 172.21.15.96:0/2742966446 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f872c1538c0 0x7f872c0b4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:41.252 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.248+0000 7f8729e00640 1 -- 172.21.15.96:0/2742966446 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f872c0b1d50 con 0x7f8710004d40 2024-05-28T06:08:41.252 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.248+0000 7f8729e00640 1 --2- 172.21.15.96:0/2742966446 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8710004d40 0x7f872c159130 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f871c009d10 tx=0x7f871c007700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:41.252 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.248+0000 7f8722000640 1 -- 172.21.15.96:0/2742966446 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f871c014610 con 0x7f8710004d40 2024-05-28T06:08:41.252 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.248+0000 7f8722000640 1 -- 172.21.15.96:0/2742966446 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f871c019070 con 0x7f8710004d40 2024-05-28T06:08:41.252 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.248+0000 7f8722000640 1 -- 172.21.15.96:0/2742966446 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f871c0149d0 con 0x7f8710004d40 2024-05-28T06:08:41.252 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.248+0000 7f872b200640 1 -- 172.21.15.96:0/2742966446 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f872c159730 con 0x7f8710004d40 2024-05-28T06:08:41.252 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.249+0000 7f872b200640 1 -- 172.21.15.96:0/2742966446 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f872c159cf0 con 0x7f8710004d40 2024-05-28T06:08:41.252 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.249+0000 7f872b200640 1 -- 172.21.15.96:0/2742966446 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f872c0689e0 con 0x7f8710004d40 2024-05-28T06:08:41.252 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.250+0000 7f8722000640 1 -- 172.21.15.96:0/2742966446 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 9) ==== 91295+0+0 (secure 0 0 0) 0x7f871c014b70 con 0x7f8710004d40 2024-05-28T06:08:41.253 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.251+0000 7f8722000640 1 --2- 172.21.15.96:0/2742966446 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f870c06cbc0 0x7f870c06f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:41.253 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.251+0000 7f8722000640 1 -- 172.21.15.96:0/2742966446 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5765+0+0 (secure 0 0 0) 0x7f871c08c9c0 con 0x7f8710004d40 2024-05-28T06:08:41.253 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.252+0000 7f8729400640 1 --2- 172.21.15.96:0/2742966446 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f870c06cbc0 0x7f870c06f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:41.253 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.253+0000 7f8729400640 1 --2- 172.21.15.96:0/2742966446 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f870c06cbc0 0x7f870c06f080 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f872c0b4c90 tx=0x7f87180093d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:41.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.256+0000 7f8722000640 1 -- 172.21.15.96:0/2742966446 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f871c05a9a0 con 0x7f8710004d40 2024-05-28T06:08:41.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.444+0000 7f872b200640 1 -- 172.21.15.96:0/2742966446 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f872c1536a0 con 0x7f8710004d40 2024-05-28T06:08:41.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.446+0000 7f8722000640 1 -- 172.21.15.96:0/2742966446 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) ==== 74+0+16789 (secure 0 0 0) 0x7f871c05a2a0 con 0x7f8710004d40 2024-05-28T06:08:41.448 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:08:41.448 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":34,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:08:37.595959+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-05-28T06:05:27.674327+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-05-28T06:05:28.878204+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":34,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[{"pool":5,"snaps":[{"begin":2,"length":2}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:08:41.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.449+0000 7f872b200640 1 -- 172.21.15.96:0/2742966446 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f870c06cbc0 msgr2=0x7f870c06f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:41.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.449+0000 7f872b200640 1 --2- 172.21.15.96:0/2742966446 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f870c06cbc0 0x7f870c06f080 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f872c0b4c90 tx=0x7f87180093d0 comp rx=0 tx=0).stop 2024-05-28T06:08:41.451 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.449+0000 7f872b200640 1 -- 172.21.15.96:0/2742966446 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8710004d40 msgr2=0x7f872c159130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:41.451 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.449+0000 7f872b200640 1 --2- 172.21.15.96:0/2742966446 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8710004d40 0x7f872c159130 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f871c009d10 tx=0x7f871c007700 comp rx=0 tx=0).stop 2024-05-28T06:08:41.451 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.449+0000 7f872b200640 1 -- 172.21.15.96:0/2742966446 shutdown_connections 2024-05-28T06:08:41.451 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.449+0000 7f872b200640 1 --2- 172.21.15.96:0/2742966446 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f870c06cbc0 0x7f870c06f080 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:41.451 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.449+0000 7f872b200640 1 --2- 172.21.15.96:0/2742966446 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f872c1538c0 0x7f872c0b4080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:41.451 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.449+0000 7f872b200640 1 --2- 172.21.15.96:0/2742966446 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f872c1514b0 0x7f872c0b1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:41.451 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.449+0000 7f872b200640 1 --2- 172.21.15.96:0/2742966446 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8710004d40 0x7f872c159130 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:41.451 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.449+0000 7f872b200640 1 -- 172.21.15.96:0/2742966446 >> 172.21.15.96:0/2742966446 conn(0x7f872c0b1fc0 msgr2=0x7f872c15b810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:41.451 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.449+0000 7f872b200640 1 -- 172.21.15.96:0/2742966446 shutdown_connections 2024-05-28T06:08:41.452 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.450+0000 7f872b200640 1 -- 172.21.15.96:0/2742966446 wait complete. 2024-05-28T06:08:41.471 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-05-28T06:08:41.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.665+0000 7faf4b400640 1 Processor -- start 2024-05-28T06:08:41.666 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.665+0000 7faf4b400640 1 -- start start 2024-05-28T06:08:41.667 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.666+0000 7faf4b400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faf440b4e10 0x7faf440cde30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:41.667 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.666+0000 7faf4b400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7faf440d0ca0 0x7faf44099090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:41.667 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.666+0000 7faf4b400640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faf440b51f0 con 0x7faf440ce370 2024-05-28T06:08:41.667 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.666+0000 7faf4b400640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7faf440aef60 con 0x7faf440d0ca0 2024-05-28T06:08:41.667 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.666+0000 7faf4b400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7faf440ad1d0 con 0x7faf440b4e10 2024-05-28T06:08:41.667 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.667+0000 7faf49400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7faf440d0ca0 0x7faf44099090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:41.667 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.667+0000 7faf43e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7faf440ce370 0x7faf440d0590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:45578/0 (socket says 172.21.15.96:45578) 2024-05-28T06:08:41.667 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.667+0000 7faf43e00640 1 -- 172.21.15.96:0/3255598019 learned_addr learned my addr 172.21.15.96:0/3255598019 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:08:41.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.667+0000 7faf48a00640 1 --2- 172.21.15.96:0/3255598019 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faf440b4e10 0x7faf440cde30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:41.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.667+0000 7faf49400640 1 -- 172.21.15.96:0/3255598019 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faf440b4e10 msgr2=0x7faf440cde30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:41.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.667+0000 7faf49400640 1 --2- 172.21.15.96:0/3255598019 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faf440b4e10 0x7faf440cde30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:41.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.667+0000 7faf49400640 1 -- 172.21.15.96:0/3255598019 >> v1:172.21.15.96:6789/0 conn(0x7faf440ce370 legacy=0x7faf440d0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:41.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.667+0000 7faf43400640 1 -- 172.21.15.96:0/3255598019 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2511309424 0 0) 0x7faf440b51f0 con 0x7faf440ce370 2024-05-28T06:08:41.669 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.667+0000 7faf49400640 1 -- 172.21.15.96:0/3255598019 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf440ad470 con 0x7faf440d0ca0 2024-05-28T06:08:41.669 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.668+0000 7faf49400640 1 --2- 172.21.15.96:0/3255598019 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf440d0ca0 0x7faf44099090 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7faf3c009d90 tx=0x7faf3c02a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c328ec5e5adebfe server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:41.670 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.668+0000 7faf43400640 1 -- 172.21.15.96:0/3255598019 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf3c031600 con 0x7faf440d0ca0 2024-05-28T06:08:41.670 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.668+0000 7faf43400640 1 -- 172.21.15.96:0/3255598019 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf3c0317a0 con 0x7faf440d0ca0 2024-05-28T06:08:41.670 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.668+0000 7faf43400640 1 -- 172.21.15.96:0/3255598019 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf3c031b70 con 0x7faf440d0ca0 2024-05-28T06:08:41.670 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.669+0000 7faf4b400640 1 -- 172.21.15.96:0/3255598019 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf440d0ca0 msgr2=0x7faf44099090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:41.670 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.669+0000 7faf4b400640 1 --2- 172.21.15.96:0/3255598019 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf440d0ca0 0x7faf44099090 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7faf3c009d90 tx=0x7faf3c02a5d0 comp rx=0 tx=0).stop 2024-05-28T06:08:41.670 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.669+0000 7faf4b400640 1 -- 172.21.15.96:0/3255598019 shutdown_connections 2024-05-28T06:08:41.670 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.669+0000 7faf4b400640 1 --2- 172.21.15.96:0/3255598019 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf440d0ca0 0x7faf44099090 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:41.670 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.669+0000 7faf4b400640 1 --2- 172.21.15.96:0/3255598019 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faf440b4e10 0x7faf440cde30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:41.670 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.669+0000 7faf4b400640 1 -- 172.21.15.96:0/3255598019 >> 172.21.15.96:0/3255598019 conn(0x7faf4406bae0 msgr2=0x7faf4406bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:41.670 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.669+0000 7faf4b400640 1 -- 172.21.15.96:0/3255598019 shutdown_connections 2024-05-28T06:08:41.670 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.669+0000 7faf4b400640 1 -- 172.21.15.96:0/3255598019 wait complete. 2024-05-28T06:08:41.671 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.671+0000 7faf4b400640 1 Processor -- start 2024-05-28T06:08:41.671 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.671+0000 7faf4b400640 1 -- start start 2024-05-28T06:08:41.673 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.672+0000 7faf4b400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faf440b4e10 0x7faf4415d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:41.673 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.672+0000 7faf4b400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf440ce370 0x7faf44158210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:41.673 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.672+0000 7faf4b400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faf440d0ca0 0x7faf440b18e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:41.673 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.672+0000 7faf4b400640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7faf440cea00 con 0x7faf440ce370 2024-05-28T06:08:41.673 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.672+0000 7faf4b400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7faf440ce880 con 0x7faf440d0ca0 2024-05-28T06:08:41.673 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.672+0000 7faf4b400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7faf440b3150 con 0x7faf440b4e10 2024-05-28T06:08:41.673 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.672+0000 7faf43e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf440ce370 0x7faf44158210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:41.673 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.672+0000 7faf43e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf440ce370 0x7faf44158210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58132/0 (socket says 172.21.15.96:58132) 2024-05-28T06:08:41.673 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.672+0000 7faf43e00640 1 -- 172.21.15.96:0/3377554255 learned_addr learned my addr 172.21.15.96:0/3377554255 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:08:41.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.673+0000 7faf43e00640 1 -- 172.21.15.96:0/3377554255 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faf440b4e10 msgr2=0x7faf4415d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:41.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.673+0000 7faf43e00640 1 --2- 172.21.15.96:0/3377554255 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faf440b4e10 0x7faf4415d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:41.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.673+0000 7faf43e00640 1 -- 172.21.15.96:0/3377554255 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faf440d0ca0 msgr2=0x7faf440b18e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:41.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.673+0000 7faf43e00640 1 --2- 172.21.15.96:0/3377554255 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faf440d0ca0 0x7faf440b18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:41.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.673+0000 7faf43e00640 1 -- 172.21.15.96:0/3377554255 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf4415dc90 con 0x7faf440ce370 2024-05-28T06:08:41.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.673+0000 7faf43e00640 1 --2- 172.21.15.96:0/3377554255 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf440ce370 0x7faf44158210 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7faf3000c410 tx=0x7faf3000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:41.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.673+0000 7faf40c00640 1 -- 172.21.15.96:0/3377554255 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf300074e0 con 0x7faf440ce370 2024-05-28T06:08:41.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.673+0000 7faf40c00640 1 -- 172.21.15.96:0/3377554255 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf300047e0 con 0x7faf440ce370 2024-05-28T06:08:41.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.673+0000 7faf40c00640 1 -- 172.21.15.96:0/3377554255 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf30004bd0 con 0x7faf440ce370 2024-05-28T06:08:41.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.673+0000 7faf4b400640 1 -- 172.21.15.96:0/3377554255 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf440b2330 con 0x7faf440ce370 2024-05-28T06:08:41.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.673+0000 7faf4b400640 1 -- 172.21.15.96:0/3377554255 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf4415cf50 con 0x7faf440ce370 2024-05-28T06:08:41.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.675+0000 7faf4b400640 1 -- 172.21.15.96:0/3377554255 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf440689e0 con 0x7faf440ce370 2024-05-28T06:08:41.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.676+0000 7faf40c00640 1 -- 172.21.15.96:0/3377554255 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 9) ==== 91295+0+0 (secure 0 0 0) 0x7faf30011070 con 0x7faf440ce370 2024-05-28T06:08:41.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.677+0000 7faf40c00640 1 --2- 172.21.15.96:0/3377554255 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7faf2406cbf0 0x7faf2406f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:41.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.677+0000 7faf40c00640 1 -- 172.21.15.96:0/3377554255 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5765+0+0 (secure 0 0 0) 0x7faf3008bc60 con 0x7faf440ce370 2024-05-28T06:08:41.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.678+0000 7faf48a00640 1 --2- 172.21.15.96:0/3377554255 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7faf2406cbf0 0x7faf2406f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:41.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.679+0000 7faf48a00640 1 --2- 172.21.15.96:0/3377554255 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7faf2406cbf0 0x7faf2406f0b0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7faf38006fd0 tx=0x7faf38008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:41.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.682+0000 7faf40c00640 1 -- 172.21.15.96:0/3377554255 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7faf3000e280 con 0x7faf440ce370 2024-05-28T06:08:41.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.908+0000 7faf4b400640 1 -- 172.21.15.96:0/3377554255 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7faf440cd810 con 0x7faf440ce370 2024-05-28T06:08:41.909 INFO:tasks.ceph.mon.a.smithi096.stderr:2024-05-28T06:08:41.909+0000 7fb2fca00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-28T06:08:41.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.912+0000 7faf40c00640 1 -- 172.21.15.96:0/3377554255 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v17) ==== 161+0+0 (secure 0 0 0) 0x7faf30059c70 con 0x7faf440ce370 2024-05-28T06:08:41.913 INFO:teuthology.orchestra.run.smithi096.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-28T06:08:41.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.915+0000 7faf4b400640 1 -- 172.21.15.96:0/3377554255 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7faf2406cbf0 msgr2=0x7faf2406f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:41.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.915+0000 7faf4b400640 1 --2- 172.21.15.96:0/3377554255 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7faf2406cbf0 0x7faf2406f0b0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7faf38006fd0 tx=0x7faf38008040 comp rx=0 tx=0).stop 2024-05-28T06:08:41.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.915+0000 7faf4b400640 1 -- 172.21.15.96:0/3377554255 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf440ce370 msgr2=0x7faf44158210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:41.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.915+0000 7faf4b400640 1 --2- 172.21.15.96:0/3377554255 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf440ce370 0x7faf44158210 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7faf3000c410 tx=0x7faf3000c8d0 comp rx=0 tx=0).stop 2024-05-28T06:08:41.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.915+0000 7faf4b400640 1 -- 172.21.15.96:0/3377554255 shutdown_connections 2024-05-28T06:08:41.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.915+0000 7faf4b400640 1 --2- 172.21.15.96:0/3377554255 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7faf2406cbf0 0x7faf2406f0b0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:41.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.915+0000 7faf4b400640 1 --2- 172.21.15.96:0/3377554255 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faf440d0ca0 0x7faf440b18e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:41.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.915+0000 7faf4b400640 1 --2- 172.21.15.96:0/3377554255 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf440ce370 0x7faf44158210 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:41.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.915+0000 7faf4b400640 1 --2- 172.21.15.96:0/3377554255 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faf440b4e10 0x7faf4415d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:41.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.915+0000 7faf4b400640 1 -- 172.21.15.96:0/3377554255 >> 172.21.15.96:0/3377554255 conn(0x7faf4406bae0 msgr2=0x7faf440d0590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:41.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.915+0000 7faf4b400640 1 -- 172.21.15.96:0/3377554255 shutdown_connections 2024-05-28T06:08:41.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:41.915+0000 7faf4b400640 1 -- 172.21.15.96:0/3377554255 wait complete. 2024-05-28T06:08:41.935 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-05-28T06:08:42.044 INFO:tasks.ceph.mds.d.smithi123.stdout:starting mds.d at 2024-05-28T06:08:42.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:42.136+0000 7f8626800640 1 Processor -- start 2024-05-28T06:08:42.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:42.137+0000 7f8626800640 1 -- start start 2024-05-28T06:08:42.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:42.138+0000 7f8626800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f862015ed80 0x7f86200cff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:42.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:42.138+0000 7f8626800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f86200b68f0 0x7f86200cedb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:42.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:42.138+0000 7f8626800640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f862015f160 con 0x7f86200b7e50 2024-05-28T06:08:42.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:42.138+0000 7f8626800640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f8620163540 con 0x7f86200b68f0 2024-05-28T06:08:42.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:42.138+0000 7f8626800640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f8620162930 con 0x7f862015ed80 2024-05-28T06:08:42.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:42.138+0000 7f861fe00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f86200b68f0 0x7f86200cedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:42.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:42.138+0000 7f861ea00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f86200b7e50 0x7f86200b61e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:45594/0 (socket says 172.21.15.96:45594) 2024-05-28T06:08:42.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:42.138+0000 7f861ea00640 1 -- 172.21.15.96:0/3889551851 learned_addr learned my addr 172.21.15.96:0/3889551851 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:08:42.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:42.139+0000 7f861f400640 1 --2- 172.21.15.96:0/3889551851 >> v2:172.21.15.123:3300/0 conn(0x7f862015ed80 0x7f86200cff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:42.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:42.139+0000 7f861fe00640 1 -- 172.21.15.96:0/3889551851 >> v2:172.21.15.123:3300/0 conn(0x7f862015ed80 msgr2=0x7f86200cff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:42.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:42.139+0000 7f861fe00640 1 --2- 172.21.15.96:0/3889551851 >> v2:172.21.15.123:3300/0 conn(0x7f862015ed80 0x7f86200cff80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:42.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:42.139+0000 7f861fe00640 1 -- 172.21.15.96:0/3889551851 >> v1:172.21.15.96:6789/0 conn(0x7f86200b7e50 legacy=0x7f86200b61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:42.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:42.139+0000 7f861fe00640 1 -- 172.21.15.96:0/3889551851 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8620162c10 con 0x7f86200b68f0 2024-05-28T06:08:42.143 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:42.140+0000 7f861fe00640 1 --2- 172.21.15.96:0/3889551851 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f86200b68f0 0x7f86200cedb0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f862008cc30 tx=0x7f861802a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80ce385142700a14 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:42.143 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:42.140+0000 7f861e000640 1 -- 172.21.15.96:0/3889551851 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8618007740 con 0x7f86200b68f0 2024-05-28T06:08:42.143 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:42.140+0000 7f861e000640 1 -- 172.21.15.96:0/3889551851 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86180078e0 con 0x7f86200b68f0 2024-05-28T06:08:42.143 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:42.140+0000 7f861e000640 1 -- 172.21.15.96:0/3889551851 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8618007cb0 con 0x7f86200b68f0 2024-05-28T06:08:42.143 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:42.141+0000 7f8626800640 1 -- 172.21.15.96:0/3889551851 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f86200b68f0 msgr2=0x7f86200cedb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:42.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:42.141+0000 7f8626800640 1 --2- 172.21.15.96:0/3889551851 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f86200b68f0 0x7f86200cedb0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f862008cc30 tx=0x7f861802a680 comp rx=0 tx=0).stop 2024-05-28T06:08:42.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:42.141+0000 7f8626800640 1 -- 172.21.15.96:0/3889551851 shutdown_connections 2024-05-28T06:08:42.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:42.141+0000 7f8626800640 1 --2- 172.21.15.96:0/3889551851 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f86200b68f0 0x7f86200cedb0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:42.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:42.141+0000 7f8626800640 1 --2- 172.21.15.96:0/3889551851 >> v2:172.21.15.123:3300/0 conn(0x7f862015ed80 0x7f86200cff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:42.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:42.141+0000 7f8626800640 1 -- 172.21.15.96:0/3889551851 >> 172.21.15.96:0/3889551851 conn(0x7f86200d13c0 msgr2=0x7f86200d17d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:42.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:42.141+0000 7f8626800640 1 -- 172.21.15.96:0/3889551851 shutdown_connections 2024-05-28T06:08:42.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:42.141+0000 7f8626800640 1 -- 172.21.15.96:0/3889551851 wait complete. 2024-05-28T06:08:42.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:42.142+0000 7f8626800640 1 Processor -- start 2024-05-28T06:08:42.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:42.143+0000 7f8626800640 1 -- start start 2024-05-28T06:08:42.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:42.143+0000 7f8626800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f86200b68f0 0x7f86201616e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:42.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:42.143+0000 7f8626800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f86200b7e50 0x7f86200c8ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:42.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:42.143+0000 7f8626800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f862015ed80 0x7f86200c91e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:42.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:42.143+0000 7f8626800640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f86200677b0 con 0x7f86200b7e50 2024-05-28T06:08:42.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:42.143+0000 7f8626800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f86201631d0 con 0x7f86200b68f0 2024-05-28T06:08:42.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:42.143+0000 7f8626800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f8620067930 con 0x7f862015ed80 2024-05-28T06:08:42.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:42.144+0000 7f861ea00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f86200b7e50 0x7f86200c8ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:42.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:42.144+0000 7f861ea00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f86200b7e50 0x7f86200c8ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58148/0 (socket says 172.21.15.96:58148) 2024-05-28T06:08:42.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:42.144+0000 7f861ea00640 1 -- 172.21.15.96:0/4230366531 learned_addr learned my addr 172.21.15.96:0/4230366531 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:08:42.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:42.144+0000 7f861f400640 1 --2- 172.21.15.96:0/4230366531 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f86200b68f0 0x7f86201616e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:42.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:42.144+0000 7f861fe00640 1 --2- 172.21.15.96:0/4230366531 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f862015ed80 0x7f86200c91e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:42.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:42.144+0000 7f861ea00640 1 -- 172.21.15.96:0/4230366531 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f862015ed80 msgr2=0x7f86200c91e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:42.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:42.144+0000 7f861ea00640 1 --2- 172.21.15.96:0/4230366531 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f862015ed80 0x7f86200c91e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:42.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:42.144+0000 7f861ea00640 1 -- 172.21.15.96:0/4230366531 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f86200b68f0 msgr2=0x7f86201616e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:42.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:42.144+0000 7f861ea00640 1 --2- 172.21.15.96:0/4230366531 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f86200b68f0 0x7f86201616e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:42.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:42.144+0000 7f861ea00640 1 -- 172.21.15.96:0/4230366531 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8620154520 con 0x7f86200b7e50 2024-05-28T06:08:42.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:42.145+0000 7f861ea00640 1 --2- 172.21.15.96:0/4230366531 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f86200b7e50 0x7f86200c8ca0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f860c00c400 tx=0x7f860c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:42.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:42.145+0000 7f8617400640 1 -- 172.21.15.96:0/4230366531 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f860c017070 con 0x7f86200b7e50 2024-05-28T06:08:42.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:42.145+0000 7f8617400640 1 -- 172.21.15.96:0/4230366531 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f860c005ce0 con 0x7f86200b7e50 2024-05-28T06:08:42.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:42.145+0000 7f8617400640 1 -- 172.21.15.96:0/4230366531 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f860c0072f0 con 0x7f86200b7e50 2024-05-28T06:08:42.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:42.145+0000 7f8626800640 1 -- 172.21.15.96:0/4230366531 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8620154800 con 0x7f86200b7e50 2024-05-28T06:08:42.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:42.145+0000 7f8626800640 1 -- 172.21.15.96:0/4230366531 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8620154dc0 con 0x7f86200b7e50 2024-05-28T06:08:42.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:42.146+0000 7f8626800640 1 -- 172.21.15.96:0/4230366531 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86200689e0 con 0x7f86200b7e50 2024-05-28T06:08:42.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:42.147+0000 7f8617400640 1 -- 172.21.15.96:0/4230366531 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 9) ==== 91295+0+0 (secure 0 0 0) 0x7f860c01a070 con 0x7f86200b7e50 2024-05-28T06:08:42.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:42.148+0000 7f8617400640 1 --2- 172.21.15.96:0/4230366531 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f860006cbc0 0x7f860006f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:42.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:42.148+0000 7f8617400640 1 -- 172.21.15.96:0/4230366531 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5823+0+0 (secure 0 0 0) 0x7f860c08b3a0 con 0x7f86200b7e50 2024-05-28T06:08:42.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:42.149+0000 7f861f400640 1 --2- 172.21.15.96:0/4230366531 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f860006cbc0 0x7f860006f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:42.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:42.150+0000 7f861f400640 1 --2- 172.21.15.96:0/4230366531 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f860006cbc0 0x7f860006f080 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f8610006fd0 tx=0x7f8610008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:42.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:42.152+0000 7f8617400640 1 -- 172.21.15.96:0/4230366531 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f860c059460 con 0x7f86200b7e50 2024-05-28T06:08:42.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:42.393+0000 7f8626800640 1 -- 172.21.15.96:0/4230366531 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f86200b8550 con 0x7f86200b7e50 2024-05-28T06:08:42.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:42.915+0000 7f8617400640 1 -- 172.21.15.96:0/4230366531 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v18) ==== 104+0+0 (secure 0 0 0) 0x7f860c058d60 con 0x7f86200b7e50 2024-05-28T06:08:42.919 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:42.919+0000 7f8626800640 1 -- 172.21.15.96:0/4230366531 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f860006cbc0 msgr2=0x7f860006f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:42.920 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:42.919+0000 7f8626800640 1 --2- 172.21.15.96:0/4230366531 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f860006cbc0 0x7f860006f080 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f8610006fd0 tx=0x7f8610008040 comp rx=0 tx=0).stop 2024-05-28T06:08:42.920 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:42.919+0000 7f8626800640 1 -- 172.21.15.96:0/4230366531 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f86200b7e50 msgr2=0x7f86200c8ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:42.920 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:42.919+0000 7f8626800640 1 --2- 172.21.15.96:0/4230366531 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f86200b7e50 0x7f86200c8ca0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f860c00c400 tx=0x7f860c00c8c0 comp rx=0 tx=0).stop 2024-05-28T06:08:42.920 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:42.919+0000 7f8626800640 1 -- 172.21.15.96:0/4230366531 shutdown_connections 2024-05-28T06:08:42.920 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:42.919+0000 7f8626800640 1 --2- 172.21.15.96:0/4230366531 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f860006cbc0 0x7f860006f080 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:42.920 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:42.919+0000 7f8626800640 1 --2- 172.21.15.96:0/4230366531 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f862015ed80 0x7f86200c91e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:42.920 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:42.919+0000 7f8626800640 1 --2- 172.21.15.96:0/4230366531 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f86200b7e50 0x7f86200c8ca0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:42.920 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:42.919+0000 7f8626800640 1 --2- 172.21.15.96:0/4230366531 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f86200b68f0 0x7f86201616e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:42.920 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:42.919+0000 7f8626800640 1 -- 172.21.15.96:0/4230366531 >> 172.21.15.96:0/4230366531 conn(0x7f86200d13c0 msgr2=0x7f86200b9d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:42.920 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:42.919+0000 7f8626800640 1 -- 172.21.15.96:0/4230366531 shutdown_connections 2024-05-28T06:08:42.920 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:42.919+0000 7f8626800640 1 -- 172.21.15.96:0/4230366531 wait complete. 2024-05-28T06:08:42.942 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-05-28T06:08:43.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:43.143+0000 7f7d8ee00640 1 Processor -- start 2024-05-28T06:08:43.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:43.144+0000 7f7d8ee00640 1 -- start start 2024-05-28T06:08:43.145 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:43.145+0000 7f7d8ee00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f7d8815fa00 0x7f7d8806e970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:43.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:43.145+0000 7f7d8ee00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7d881600f0 0x7f7d880d0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:43.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:43.145+0000 7f7d8ee00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d880d0d90 con 0x7f7d880d0970 2024-05-28T06:08:43.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:43.145+0000 7f7d8ee00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f7d8806fa00 con 0x7f7d8815fa00 2024-05-28T06:08:43.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:43.145+0000 7f7d8ee00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f7d8806eeb0 con 0x7f7d881600f0 2024-05-28T06:08:43.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:43.146+0000 7f7d8d000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7d881600f0 0x7f7d880d0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:43.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:43.146+0000 7f7d8d000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7d881600f0 0x7f7d880d0430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.96:59828/0 (socket says 172.21.15.96:59828) 2024-05-28T06:08:43.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:43.146+0000 7f7d8d000640 1 -- 172.21.15.96:0/1791135931 learned_addr learned my addr 172.21.15.96:0/1791135931 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:08:43.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:43.146+0000 7f7d8da00640 1 --2- 172.21.15.96:0/1791135931 >> v2:172.21.15.123:3300/0 conn(0x7f7d8815fa00 0x7f7d8806e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:43.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:43.147+0000 7f7d8d000640 1 -- 172.21.15.96:0/1791135931 >> v2:172.21.15.123:3300/0 conn(0x7f7d8815fa00 msgr2=0x7f7d8806e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:43.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:43.147+0000 7f7d8d000640 1 --2- 172.21.15.96:0/1791135931 >> v2:172.21.15.123:3300/0 conn(0x7f7d8815fa00 0x7f7d8806e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:43.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:43.147+0000 7f7d8d000640 1 -- 172.21.15.96:0/1791135931 >> v1:172.21.15.123:6789/0 conn(0x7f7d880d0970 legacy=0x7f7d880b3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:43.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:43.147+0000 7f7d8d000640 1 -- 172.21.15.96:0/1791135931 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d8806f160 con 0x7f7d881600f0 2024-05-28T06:08:43.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:43.148+0000 7f7d8d000640 1 --2- 172.21.15.96:0/1791135931 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7d881600f0 0x7f7d880d0430 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7d74009870 tx=0x7f7d7402a1c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf195a74b6a0d608 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:43.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:43.149+0000 7f7d87e00640 1 -- 172.21.15.96:0/1791135931 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d74037070 con 0x7f7d881600f0 2024-05-28T06:08:43.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:43.149+0000 7f7d87e00640 1 -- 172.21.15.96:0/1791135931 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d74032440 con 0x7f7d881600f0 2024-05-28T06:08:43.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:43.149+0000 7f7d87e00640 1 -- 172.21.15.96:0/1791135931 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d74032810 con 0x7f7d881600f0 2024-05-28T06:08:43.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:43.149+0000 7f7d8ee00640 1 -- 172.21.15.96:0/1791135931 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7d881600f0 msgr2=0x7f7d880d0430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:43.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:43.149+0000 7f7d8ee00640 1 --2- 172.21.15.96:0/1791135931 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7d881600f0 0x7f7d880d0430 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7d74009870 tx=0x7f7d7402a1c0 comp rx=0 tx=0).stop 2024-05-28T06:08:43.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:43.149+0000 7f7d8ee00640 1 -- 172.21.15.96:0/1791135931 shutdown_connections 2024-05-28T06:08:43.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:43.149+0000 7f7d8ee00640 1 --2- 172.21.15.96:0/1791135931 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7d881600f0 0x7f7d880d0430 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:43.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:43.149+0000 7f7d8ee00640 1 --2- 172.21.15.96:0/1791135931 >> v2:172.21.15.123:3300/0 conn(0x7f7d8815fa00 0x7f7d8806e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:43.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:43.149+0000 7f7d8ee00640 1 -- 172.21.15.96:0/1791135931 >> 172.21.15.96:0/1791135931 conn(0x7f7d8806bae0 msgr2=0x7f7d8806bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:43.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:43.150+0000 7f7d8ee00640 1 -- 172.21.15.96:0/1791135931 shutdown_connections 2024-05-28T06:08:43.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:43.150+0000 7f7d8ee00640 1 -- 172.21.15.96:0/1791135931 wait complete. 2024-05-28T06:08:43.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:43.151+0000 7f7d8ee00640 1 Processor -- start 2024-05-28T06:08:43.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:43.151+0000 7f7d8ee00640 1 -- start start 2024-05-28T06:08:43.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:43.152+0000 7f7d8ee00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d880d0970 0x7f7d8815aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:43.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:43.152+0000 7f7d8ee00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7d8815fa00 0x7f7d880cf040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:43.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:43.152+0000 7f7d8ee00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7d881600f0 0x7f7d881581a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:43.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:43.152+0000 7f7d8ee00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f7d88071c70 con 0x7f7d881600f0 2024-05-28T06:08:43.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:43.152+0000 7f7d8ee00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f7d8806f720 con 0x7f7d880d0970 2024-05-28T06:08:43.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:43.152+0000 7f7d8ee00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f7d8815fde0 con 0x7f7d8815fa00 2024-05-28T06:08:43.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:43.152+0000 7f7d8e400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7d881600f0 0x7f7d881581a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:43.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:43.153+0000 7f7d8e400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7d881600f0 0x7f7d881581a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58158/0 (socket says 172.21.15.96:58158) 2024-05-28T06:08:43.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:43.153+0000 7f7d8e400640 1 -- 172.21.15.96:0/1417084294 learned_addr learned my addr 172.21.15.96:0/1417084294 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:08:43.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:43.153+0000 7f7d8d000640 1 --2- 172.21.15.96:0/1417084294 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7d8815fa00 0x7f7d880cf040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:43.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:43.153+0000 7f7d8da00640 1 --2- 172.21.15.96:0/1417084294 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d880d0970 0x7f7d8815aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:43.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:43.153+0000 7f7d8e400640 1 -- 172.21.15.96:0/1417084294 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7d8815fa00 msgr2=0x7f7d880cf040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:43.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:43.153+0000 7f7d8e400640 1 --2- 172.21.15.96:0/1417084294 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7d8815fa00 0x7f7d880cf040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:43.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:43.153+0000 7f7d8e400640 1 -- 172.21.15.96:0/1417084294 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d880d0970 msgr2=0x7f7d8815aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:43.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:43.153+0000 7f7d8e400640 1 --2- 172.21.15.96:0/1417084294 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d880d0970 0x7f7d8815aeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:43.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:43.153+0000 7f7d8e400640 1 -- 172.21.15.96:0/1417084294 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d88159ea0 con 0x7f7d881600f0 2024-05-28T06:08:43.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:43.153+0000 7f7d8e400640 1 --2- 172.21.15.96:0/1417084294 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7d881600f0 0x7f7d881581a0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f7d8000c6b0 tx=0x7f7d8000cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:43.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:43.153+0000 7f7d85600640 1 -- 172.21.15.96:0/1417084294 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d80003b70 con 0x7f7d881600f0 2024-05-28T06:08:43.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:43.153+0000 7f7d85600640 1 -- 172.21.15.96:0/1417084294 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d80003d30 con 0x7f7d881600f0 2024-05-28T06:08:43.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:43.153+0000 7f7d85600640 1 -- 172.21.15.96:0/1417084294 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d80005310 con 0x7f7d881600f0 2024-05-28T06:08:43.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:43.154+0000 7f7d8ee00640 1 -- 172.21.15.96:0/1417084294 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d8815a3f0 con 0x7f7d881600f0 2024-05-28T06:08:43.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:43.154+0000 7f7d8ee00640 1 -- 172.21.15.96:0/1417084294 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d8815a9b0 con 0x7f7d881600f0 2024-05-28T06:08:43.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:43.155+0000 7f7d8ee00640 1 -- 172.21.15.96:0/1417084294 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d880689e0 con 0x7f7d881600f0 2024-05-28T06:08:43.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:43.156+0000 7f7d85600640 1 -- 172.21.15.96:0/1417084294 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 9) ==== 91295+0+0 (secure 0 0 0) 0x7f7d800054b0 con 0x7f7d881600f0 2024-05-28T06:08:43.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:43.157+0000 7f7d85600640 1 --2- 172.21.15.96:0/1417084294 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7d6806cbc0 0x7f7d6806f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:43.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:43.157+0000 7f7d85600640 1 -- 172.21.15.96:0/1417084294 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5823+0+0 (secure 0 0 0) 0x7f7d8008bea0 con 0x7f7d881600f0 2024-05-28T06:08:43.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:43.157+0000 7f7d8da00640 1 --2- 172.21.15.96:0/1417084294 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7d6806cbc0 0x7f7d6806f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:43.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:43.158+0000 7f7d8da00640 1 --2- 172.21.15.96:0/1417084294 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7d6806cbc0 0x7f7d6806f080 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f7d7c009970 tx=0x7f7d7c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:43.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:43.161+0000 7f7d85600640 1 -- 172.21.15.96:0/1417084294 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7d80059f10 con 0x7f7d881600f0 2024-05-28T06:08:43.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:43.344+0000 7f7d8ee00640 1 -- 172.21.15.96:0/1417084294 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f7d880af1c0 con 0x7f7d881600f0 2024-05-28T06:08:43.920 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:43.919+0000 7f7d85600640 1 -- 172.21.15.96:0/1417084294 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v36) ==== 181+0+0 (secure 0 0 0) 0x7f7d80059810 con 0x7f7d881600f0 2024-05-28T06:08:43.920 INFO:teuthology.orchestra.run.smithi096.stderr:pool 'cephfs_metadata' removed 2024-05-28T06:08:43.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:43.924+0000 7f7d8ee00640 1 -- 172.21.15.96:0/1417084294 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7d6806cbc0 msgr2=0x7f7d6806f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:43.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:43.924+0000 7f7d8ee00640 1 --2- 172.21.15.96:0/1417084294 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7d6806cbc0 0x7f7d6806f080 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f7d7c009970 tx=0x7f7d7c008040 comp rx=0 tx=0).stop 2024-05-28T06:08:43.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:43.924+0000 7f7d8ee00640 1 -- 172.21.15.96:0/1417084294 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7d881600f0 msgr2=0x7f7d881581a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:43.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:43.924+0000 7f7d8ee00640 1 --2- 172.21.15.96:0/1417084294 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7d881600f0 0x7f7d881581a0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f7d8000c6b0 tx=0x7f7d8000cb70 comp rx=0 tx=0).stop 2024-05-28T06:08:43.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:43.924+0000 7f7d8ee00640 1 -- 172.21.15.96:0/1417084294 shutdown_connections 2024-05-28T06:08:43.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:43.924+0000 7f7d8ee00640 1 --2- 172.21.15.96:0/1417084294 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7d6806cbc0 0x7f7d6806f080 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:43.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:43.924+0000 7f7d8ee00640 1 --2- 172.21.15.96:0/1417084294 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7d881600f0 0x7f7d881581a0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:43.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:43.924+0000 7f7d8ee00640 1 --2- 172.21.15.96:0/1417084294 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7d8815fa00 0x7f7d880cf040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:43.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:43.924+0000 7f7d8ee00640 1 --2- 172.21.15.96:0/1417084294 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d880d0970 0x7f7d8815aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:43.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:43.924+0000 7f7d8ee00640 1 -- 172.21.15.96:0/1417084294 >> 172.21.15.96:0/1417084294 conn(0x7f7d8806bae0 msgr2=0x7f7d880b4680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:43.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:43.925+0000 7f7d8ee00640 1 -- 172.21.15.96:0/1417084294 shutdown_connections 2024-05-28T06:08:43.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:43.925+0000 7f7d8ee00640 1 -- 172.21.15.96:0/1417084294 wait complete. 2024-05-28T06:08:43.947 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-05-28T06:08:44.143 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:44.142+0000 7f0406200640 1 Processor -- start 2024-05-28T06:08:44.143 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:44.143+0000 7f0406200640 1 -- start start 2024-05-28T06:08:44.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:44.144+0000 7f0406200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f04000cf2f0 0x7f04000cf910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:44.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:44.144+0000 7f0406200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f0400140500 0x7f04000ad130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:44.145 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:44.144+0000 7f0406200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f040006dd50 con 0x7f040006edf0 2024-05-28T06:08:44.145 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:44.144+0000 7f0406200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f040006e030 con 0x7f04000cf2f0 2024-05-28T06:08:44.145 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:44.144+0000 7f0406200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f040006e310 con 0x7f0400140500 2024-05-28T06:08:44.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:44.144+0000 7f03ff400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f04000cf2f0 0x7f04000cf910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:44.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:44.145+0000 7f03f6a00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f040006edf0 0x7f040013fdf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:45610/0 (socket says 172.21.15.96:45610) 2024-05-28T06:08:44.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:44.145+0000 7f03f6a00640 1 -- 172.21.15.96:0/3756663632 learned_addr learned my addr 172.21.15.96:0/3756663632 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:08:44.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:44.145+0000 7f03ffe00640 1 --2- 172.21.15.96:0/3756663632 >> v2:172.21.15.123:3300/0 conn(0x7f0400140500 0x7f04000ad130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:44.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:44.145+0000 7f03ff400640 1 -- 172.21.15.96:0/3756663632 >> v2:172.21.15.123:3300/0 conn(0x7f0400140500 msgr2=0x7f04000ad130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:44.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:44.145+0000 7f03ff400640 1 --2- 172.21.15.96:0/3756663632 >> v2:172.21.15.123:3300/0 conn(0x7f0400140500 0x7f04000ad130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:44.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:44.145+0000 7f03ff400640 1 -- 172.21.15.96:0/3756663632 >> v1:172.21.15.96:6789/0 conn(0x7f040006edf0 legacy=0x7f040013fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:44.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:44.145+0000 7f03ff400640 1 -- 172.21.15.96:0/3756663632 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f040006e5f0 con 0x7f04000cf2f0 2024-05-28T06:08:44.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:44.146+0000 7f03ff400640 1 --2- 172.21.15.96:0/3756663632 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f04000cf2f0 0x7f04000cf910 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f04000afd50 tx=0x7f03f002a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=591968f2420dcf97 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:44.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:44.146+0000 7f03fea00640 1 -- 172.21.15.96:0/3756663632 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03f0037070 con 0x7f04000cf2f0 2024-05-28T06:08:44.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:44.146+0000 7f03fea00640 1 -- 172.21.15.96:0/3756663632 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03f0032440 con 0x7f04000cf2f0 2024-05-28T06:08:44.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:44.146+0000 7f03fea00640 1 -- 172.21.15.96:0/3756663632 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03f002ae60 con 0x7f04000cf2f0 2024-05-28T06:08:44.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:44.147+0000 7f0406200640 1 -- 172.21.15.96:0/3756663632 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f04000cf2f0 msgr2=0x7f04000cf910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:44.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:44.147+0000 7f0406200640 1 --2- 172.21.15.96:0/3756663632 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f04000cf2f0 0x7f04000cf910 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f04000afd50 tx=0x7f03f002a110 comp rx=0 tx=0).stop 2024-05-28T06:08:44.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:44.147+0000 7f0406200640 1 -- 172.21.15.96:0/3756663632 shutdown_connections 2024-05-28T06:08:44.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:44.147+0000 7f0406200640 1 --2- 172.21.15.96:0/3756663632 >> v2:172.21.15.123:3300/0 conn(0x7f0400140500 0x7f04000ad130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:44.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:44.147+0000 7f0406200640 1 --2- 172.21.15.96:0/3756663632 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f04000cf2f0 0x7f04000cf910 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:44.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:44.147+0000 7f0406200640 1 -- 172.21.15.96:0/3756663632 >> 172.21.15.96:0/3756663632 conn(0x7f04000b1e50 msgr2=0x7f04000b2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:44.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:44.147+0000 7f0406200640 1 -- 172.21.15.96:0/3756663632 shutdown_connections 2024-05-28T06:08:44.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:44.147+0000 7f0406200640 1 -- 172.21.15.96:0/3756663632 wait complete. 2024-05-28T06:08:44.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:44.149+0000 7f0406200640 1 Processor -- start 2024-05-28T06:08:44.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:44.149+0000 7f0406200640 1 -- start start 2024-05-28T06:08:44.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:44.150+0000 7f0406200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f040006edf0 0x7f04000c9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:44.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:44.150+0000 7f0406200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0400072ab0 0x7f0400099550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:44.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:44.150+0000 7f0406200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f04000b12f0 0x7f0400099a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:44.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:44.150+0000 7f0406200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f04001461a0 con 0x7f0400072ab0 2024-05-28T06:08:44.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:44.150+0000 7f0406200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f0400146020 con 0x7f040006edf0 2024-05-28T06:08:44.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:44.150+0000 7f0406200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f04000cfe50 con 0x7f04000b12f0 2024-05-28T06:08:44.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:44.150+0000 7f03f6a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0400072ab0 0x7f0400099550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:44.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:44.150+0000 7f03f6a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0400072ab0 0x7f0400099550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58180/0 (socket says 172.21.15.96:58180) 2024-05-28T06:08:44.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:44.150+0000 7f03f6a00640 1 -- 172.21.15.96:0/1603067519 learned_addr learned my addr 172.21.15.96:0/1603067519 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:08:44.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:44.151+0000 7f03f6a00640 1 -- 172.21.15.96:0/1603067519 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f04000b12f0 msgr2=0x7f0400099a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:44.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:44.151+0000 7f03f6a00640 1 --2- 172.21.15.96:0/1603067519 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f04000b12f0 0x7f0400099a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:44.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:44.151+0000 7f03f6a00640 1 -- 172.21.15.96:0/1603067519 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f040006edf0 msgr2=0x7f04000c9aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:44.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:44.151+0000 7f03f6a00640 1 --2- 172.21.15.96:0/1603067519 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f040006edf0 0x7f04000c9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:44.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:44.151+0000 7f03f6a00640 1 -- 172.21.15.96:0/1603067519 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0400072fc0 con 0x7f0400072ab0 2024-05-28T06:08:44.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:44.151+0000 7f03f6a00640 1 --2- 172.21.15.96:0/1603067519 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0400072ab0 0x7f0400099550 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f03e800c430 tx=0x7f03e800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:44.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:44.151+0000 7f03f7e00640 1 -- 172.21.15.96:0/1603067519 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03e8017070 con 0x7f0400072ab0 2024-05-28T06:08:44.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:44.151+0000 7f03f7e00640 1 -- 172.21.15.96:0/1603067519 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03e8005ce0 con 0x7f0400072ab0 2024-05-28T06:08:44.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:44.151+0000 7f0406200640 1 -- 172.21.15.96:0/1603067519 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04000732a0 con 0x7f0400072ab0 2024-05-28T06:08:44.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:44.151+0000 7f03f7e00640 1 -- 172.21.15.96:0/1603067519 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03e80072f0 con 0x7f0400072ab0 2024-05-28T06:08:44.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:44.151+0000 7f0406200640 1 -- 172.21.15.96:0/1603067519 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0400140d50 con 0x7f0400072ab0 2024-05-28T06:08:44.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:44.153+0000 7f0406200640 1 -- 172.21.15.96:0/1603067519 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04000b51b0 con 0x7f0400072ab0 2024-05-28T06:08:44.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:44.154+0000 7f03f7e00640 1 -- 172.21.15.96:0/1603067519 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 9) ==== 91295+0+0 (secure 0 0 0) 0x7f03e801a070 con 0x7f0400072ab0 2024-05-28T06:08:44.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:44.154+0000 7f03f7e00640 1 --2- 172.21.15.96:0/1603067519 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f03dc06cbc0 0x7f03dc06f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:44.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:44.155+0000 7f03f7e00640 1 -- 172.21.15.96:0/1603067519 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5385+0+0 (secure 0 0 0) 0x7f03e808b350 con 0x7f0400072ab0 2024-05-28T06:08:44.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:44.155+0000 7f03ff400640 1 --2- 172.21.15.96:0/1603067519 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f03dc06cbc0 0x7f03dc06f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:44.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:44.156+0000 7f03ff400640 1 --2- 172.21.15.96:0/1603067519 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f03dc06cbc0 0x7f03dc06f080 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f03f0034000 tx=0x7f03f002c000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:44.159 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:44.159+0000 7f03f7e00640 1 -- 172.21.15.96:0/1603067519 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f04000b51b0 con 0x7f0400072ab0 2024-05-28T06:08:44.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:44.345+0000 7f0406200640 1 -- 172.21.15.96:0/1603067519 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f04000689e0 con 0x7f0400072ab0 2024-05-28T06:08:44.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:44.933+0000 7f03f7e00640 1 -- 172.21.15.96:0/1603067519 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v37) ==== 169+0+0 (secure 0 0 0) 0x7f03e8059490 con 0x7f0400072ab0 2024-05-28T06:08:44.934 INFO:teuthology.orchestra.run.smithi096.stderr:pool 'cephfs_data' removed 2024-05-28T06:08:44.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:44.937+0000 7f0406200640 1 -- 172.21.15.96:0/1603067519 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f03dc06cbc0 msgr2=0x7f03dc06f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:44.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:44.937+0000 7f0406200640 1 --2- 172.21.15.96:0/1603067519 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f03dc06cbc0 0x7f03dc06f080 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f03f0034000 tx=0x7f03f002c000 comp rx=0 tx=0).stop 2024-05-28T06:08:44.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:44.937+0000 7f0406200640 1 -- 172.21.15.96:0/1603067519 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0400072ab0 msgr2=0x7f0400099550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:44.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:44.937+0000 7f0406200640 1 --2- 172.21.15.96:0/1603067519 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0400072ab0 0x7f0400099550 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f03e800c430 tx=0x7f03e800c8f0 comp rx=0 tx=0).stop 2024-05-28T06:08:44.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:44.937+0000 7f0406200640 1 -- 172.21.15.96:0/1603067519 shutdown_connections 2024-05-28T06:08:44.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:44.937+0000 7f0406200640 1 --2- 172.21.15.96:0/1603067519 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f03dc06cbc0 0x7f03dc06f080 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:44.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:44.937+0000 7f0406200640 1 --2- 172.21.15.96:0/1603067519 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f04000b12f0 0x7f0400099a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:44.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:44.937+0000 7f0406200640 1 --2- 172.21.15.96:0/1603067519 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0400072ab0 0x7f0400099550 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:44.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:44.937+0000 7f0406200640 1 --2- 172.21.15.96:0/1603067519 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f040006edf0 0x7f04000c9aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:44.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:44.937+0000 7f0406200640 1 -- 172.21.15.96:0/1603067519 >> 172.21.15.96:0/1603067519 conn(0x7f04000b1e50 msgr2=0x7f04000ce630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:44.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:44.938+0000 7f0406200640 1 -- 172.21.15.96:0/1603067519 shutdown_connections 2024-05-28T06:08:44.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:44.939+0000 7f0406200640 1 -- 172.21.15.96:0/1603067519 wait complete. 2024-05-28T06:08:44.968 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-05-28T06:08:45.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.162+0000 7f60dc600640 1 Processor -- start 2024-05-28T06:08:45.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.162+0000 7f60dc600640 1 -- start start 2024-05-28T06:08:45.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.164+0000 7f60dc600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f60d40d05d0 0x7f60d40afee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:45.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.164+0000 7f60dc600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f60d415d590 0x7f60d406ca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:45.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.164+0000 7f60dc600640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60d415da00 con 0x7f60d406cf80 2024-05-28T06:08:45.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.164+0000 7f60dc600640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f60d406c5b0 con 0x7f60d415d590 2024-05-28T06:08:45.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.164+0000 7f60dc600640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f60d406b9a0 con 0x7f60d40d05d0 2024-05-28T06:08:45.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.164+0000 7f60da600640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f60d406cf80 0x7f60d40b4240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:45614/0 (socket says 172.21.15.96:45614) 2024-05-28T06:08:45.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.164+0000 7f60da600640 1 -- 172.21.15.96:0/4140862505 learned_addr learned my addr 172.21.15.96:0/4140862505 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:08:45.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.164+0000 7f60d9200640 1 --2- 172.21.15.96:0/4140862505 >> v2:172.21.15.96:3300/0 conn(0x7f60d415d590 0x7f60d406ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:45.166 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.165+0000 7f60d9c00640 1 --2- 172.21.15.96:0/4140862505 >> v2:172.21.15.123:3300/0 conn(0x7f60d40d05d0 0x7f60d40afee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:45.166 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.165+0000 7f60d3e00640 1 -- 172.21.15.96:0/4140862505 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3981272119 0 0) 0x7f60d415da00 con 0x7f60d406cf80 2024-05-28T06:08:45.166 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.165+0000 7f60d3e00640 1 -- 172.21.15.96:0/4140862505 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f60bc003520 con 0x7f60d406cf80 2024-05-28T06:08:45.166 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.165+0000 7f60d9200640 1 -- 172.21.15.96:0/4140862505 >> v2:172.21.15.123:3300/0 conn(0x7f60d40d05d0 msgr2=0x7f60d40afee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:45.166 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.165+0000 7f60d9200640 1 --2- 172.21.15.96:0/4140862505 >> v2:172.21.15.123:3300/0 conn(0x7f60d40d05d0 0x7f60d40afee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:45.166 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.165+0000 7f60d9200640 1 -- 172.21.15.96:0/4140862505 >> v1:172.21.15.96:6789/0 conn(0x7f60d406cf80 legacy=0x7f60d40b4240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:45.166 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.165+0000 7f60d9200640 1 -- 172.21.15.96:0/4140862505 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60d406bc80 con 0x7f60d415d590 2024-05-28T06:08:45.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.167+0000 7f60d9200640 1 --2- 172.21.15.96:0/4140862505 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f60d415d590 0x7f60d406ca40 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f60d4067860 tx=0x7f60c002a150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c13d97916060599e server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:45.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.167+0000 7f60d3e00640 1 -- 172.21.15.96:0/4140862505 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60c0037070 con 0x7f60d415d590 2024-05-28T06:08:45.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.167+0000 7f60d3e00640 1 -- 172.21.15.96:0/4140862505 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60c0032440 con 0x7f60d415d590 2024-05-28T06:08:45.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.167+0000 7f60d3e00640 1 -- 172.21.15.96:0/4140862505 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60c002ae60 con 0x7f60d415d590 2024-05-28T06:08:45.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.168+0000 7f60dc600640 1 -- 172.21.15.96:0/4140862505 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f60d415d590 msgr2=0x7f60d406ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:45.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.168+0000 7f60dc600640 1 --2- 172.21.15.96:0/4140862505 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f60d415d590 0x7f60d406ca40 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f60d4067860 tx=0x7f60c002a150 comp rx=0 tx=0).stop 2024-05-28T06:08:45.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.168+0000 7f60dc600640 1 -- 172.21.15.96:0/4140862505 shutdown_connections 2024-05-28T06:08:45.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.168+0000 7f60dc600640 1 --2- 172.21.15.96:0/4140862505 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f60d415d590 0x7f60d406ca40 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:45.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.168+0000 7f60dc600640 1 --2- 172.21.15.96:0/4140862505 >> v2:172.21.15.123:3300/0 conn(0x7f60d40d05d0 0x7f60d40afee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:45.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.168+0000 7f60dc600640 1 -- 172.21.15.96:0/4140862505 >> 172.21.15.96:0/4140862505 conn(0x7f60d40b3240 msgr2=0x7f60d40d0b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:45.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.168+0000 7f60dc600640 1 -- 172.21.15.96:0/4140862505 shutdown_connections 2024-05-28T06:08:45.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.168+0000 7f60dc600640 1 -- 172.21.15.96:0/4140862505 wait complete. 2024-05-28T06:08:45.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.170+0000 7f60dc600640 1 Processor -- start 2024-05-28T06:08:45.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.170+0000 7f60dc600640 1 -- start start 2024-05-28T06:08:45.171 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.171+0000 7f60dc600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f60d406cf80 0x7f60d415dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:45.171 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.171+0000 7f60dc600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60d40d05d0 0x7f60d409e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:45.171 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.171+0000 7f60dc600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f60d415d590 0x7f60d413cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:45.171 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.171+0000 7f60dc600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f60d40b4950 con 0x7f60d415d590 2024-05-28T06:08:45.172 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.171+0000 7f60dc600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f60d406c240 con 0x7f60d40d05d0 2024-05-28T06:08:45.172 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.171+0000 7f60dc600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f60d40b4ad0 con 0x7f60d406cf80 2024-05-28T06:08:45.172 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.171+0000 7f60da600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f60d415d590 0x7f60d413cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:45.172 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.171+0000 7f60da600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f60d415d590 0x7f60d413cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58208/0 (socket says 172.21.15.96:58208) 2024-05-28T06:08:45.172 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.171+0000 7f60da600640 1 -- 172.21.15.96:0/453150855 learned_addr learned my addr 172.21.15.96:0/453150855 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:08:45.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.171+0000 7f60da600640 1 -- 172.21.15.96:0/453150855 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f60d406cf80 msgr2=0x7f60d415dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:45.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.171+0000 7f60da600640 1 --2- 172.21.15.96:0/453150855 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f60d406cf80 0x7f60d415dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:45.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.171+0000 7f60d9200640 1 --2- 172.21.15.96:0/453150855 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60d40d05d0 0x7f60d409e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:45.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.171+0000 7f60da600640 1 -- 172.21.15.96:0/453150855 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60d40d05d0 msgr2=0x7f60d409e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:45.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.171+0000 7f60da600640 1 --2- 172.21.15.96:0/453150855 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60d40d05d0 0x7f60d409e3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:45.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.171+0000 7f60da600640 1 -- 172.21.15.96:0/453150855 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60d413d420 con 0x7f60d415d590 2024-05-28T06:08:45.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.172+0000 7f60da600640 1 --2- 172.21.15.96:0/453150855 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f60d415d590 0x7f60d413cee0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f60cc00c840 tx=0x7f60cc00cd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:45.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.172+0000 7f60d1600640 1 -- 172.21.15.96:0/453150855 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60cc018070 con 0x7f60d415d590 2024-05-28T06:08:45.174 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.172+0000 7f60d1600640 1 -- 172.21.15.96:0/453150855 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60cc00a640 con 0x7f60d415d590 2024-05-28T06:08:45.174 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.172+0000 7f60dc600640 1 -- 172.21.15.96:0/453150855 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60d415e300 con 0x7f60d415d590 2024-05-28T06:08:45.174 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.172+0000 7f60d1600640 1 -- 172.21.15.96:0/453150855 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60cc00a9f0 con 0x7f60d415d590 2024-05-28T06:08:45.174 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.172+0000 7f60dc600640 1 -- 172.21.15.96:0/453150855 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60d415e8c0 con 0x7f60d415d590 2024-05-28T06:08:45.174 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.173+0000 7f60dc600640 1 -- 172.21.15.96:0/453150855 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60d40b2550 con 0x7f60d415d590 2024-05-28T06:08:45.174 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.174+0000 7f60d1600640 1 -- 172.21.15.96:0/453150855 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 9) ==== 91295+0+0 (secure 0 0 0) 0x7f60cc01b070 con 0x7f60d415d590 2024-05-28T06:08:45.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.175+0000 7f60d1600640 1 --2- 172.21.15.96:0/453150855 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f60b406cbc0 0x7f60b406f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:45.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.175+0000 7f60d1600640 1 -- 172.21.15.96:0/453150855 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 4971+0+0 (secure 0 0 0) 0x7f60cc08b5a0 con 0x7f60d415d590 2024-05-28T06:08:45.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.176+0000 7f60d9c00640 1 --2- 172.21.15.96:0/453150855 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f60b406cbc0 0x7f60b406f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:45.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.177+0000 7f60d9c00640 1 --2- 172.21.15.96:0/453150855 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f60b406cbc0 0x7f60b406f080 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f60d41f37a0 tx=0x7f60c800f040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:45.180 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.180+0000 7f60d1600640 1 -- 172.21.15.96:0/453150855 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f60d40b2550 con 0x7f60d415d590 2024-05-28T06:08:45.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.379+0000 7f60dc600640 1 -- 172.21.15.96:0/453150855 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7f60d40690f0 con 0x7f60d415d590 2024-05-28T06:08:45.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.384+0000 7f60d1600640 1 -- 172.21.15.96:0/453150855 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v11) ==== 104+0+0 (secure 0 0 0) 0x7f60cc059840 con 0x7f60d415d590 2024-05-28T06:08:45.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.387+0000 7f60dc600640 1 -- 172.21.15.96:0/453150855 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f60b406cbc0 msgr2=0x7f60b406f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:45.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.387+0000 7f60dc600640 1 --2- 172.21.15.96:0/453150855 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f60b406cbc0 0x7f60b406f080 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f60d41f37a0 tx=0x7f60c800f040 comp rx=0 tx=0).stop 2024-05-28T06:08:45.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.387+0000 7f60dc600640 1 -- 172.21.15.96:0/453150855 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f60d415d590 msgr2=0x7f60d413cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:45.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.387+0000 7f60dc600640 1 --2- 172.21.15.96:0/453150855 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f60d415d590 0x7f60d413cee0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f60cc00c840 tx=0x7f60cc00cd00 comp rx=0 tx=0).stop 2024-05-28T06:08:45.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.388+0000 7f60dc600640 1 -- 172.21.15.96:0/453150855 shutdown_connections 2024-05-28T06:08:45.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.388+0000 7f60dc600640 1 --2- 172.21.15.96:0/453150855 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f60b406cbc0 0x7f60b406f080 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:45.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.388+0000 7f60dc600640 1 --2- 172.21.15.96:0/453150855 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f60d415d590 0x7f60d413cee0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:45.389 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.388+0000 7f60dc600640 1 --2- 172.21.15.96:0/453150855 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60d40d05d0 0x7f60d409e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:45.389 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.388+0000 7f60dc600640 1 --2- 172.21.15.96:0/453150855 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f60d406cf80 0x7f60d415dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:45.389 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.388+0000 7f60dc600640 1 -- 172.21.15.96:0/453150855 >> 172.21.15.96:0/453150855 conn(0x7f60d40b3240 msgr2=0x7f60d40b4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:45.389 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.388+0000 7f60dc600640 1 -- 172.21.15.96:0/453150855 shutdown_connections 2024-05-28T06:08:45.389 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.388+0000 7f60dc600640 1 -- 172.21.15.96:0/453150855 wait complete. 2024-05-28T06:08:45.407 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestCloneProgressReporter.test_progress_bar_for_1_clone' 2024-05-28T06:08:45.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.618+0000 7f0021200640 1 Processor -- start 2024-05-28T06:08:45.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.618+0000 7f0021200640 1 -- start start 2024-05-28T06:08:45.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.620+0000 7f0021200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f001c15b3a0 0x7f001c0b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:45.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.620+0000 7f0021200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f001c06dc20 0x7f001c15f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:45.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.620+0000 7f0021200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f001c0afd60 con 0x7f001c072c00 2024-05-28T06:08:45.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.620+0000 7f0021200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f001c06f980 con 0x7f001c06dc20 2024-05-28T06:08:45.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.620+0000 7f0021200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f001c0ae0c0 con 0x7f001c15b3a0 2024-05-28T06:08:45.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.620+0000 7f001aa00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f001c06dc20 0x7f001c15f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:45.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.620+0000 7f001aa00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f001c06dc20 0x7f001c15f9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58212/0 (socket says 172.21.15.96:58212) 2024-05-28T06:08:45.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.620+0000 7f001aa00640 1 -- 172.21.15.96:0/3014564505 learned_addr learned my addr 172.21.15.96:0/3014564505 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:08:45.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.620+0000 7f001aa00640 1 -- 172.21.15.96:0/3014564505 >> v2:172.21.15.123:3300/0 conn(0x7f001c15b3a0 msgr2=0x7f001c0b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:45.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.620+0000 7f001aa00640 1 --2- 172.21.15.96:0/3014564505 >> v2:172.21.15.123:3300/0 conn(0x7f001c15b3a0 0x7f001c0b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:45.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.620+0000 7f001aa00640 1 -- 172.21.15.96:0/3014564505 >> v1:172.21.15.123:6789/0 conn(0x7f001c072c00 legacy=0x7f001c073010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:45.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.620+0000 7f001aa00640 1 -- 172.21.15.96:0/3014564505 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f001c0af7a0 con 0x7f001c06dc20 2024-05-28T06:08:45.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.622+0000 7f001aa00640 1 --2- 172.21.15.96:0/3014564505 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f001c06dc20 0x7f001c15f9a0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f001c072bd0 tx=0x7f000802a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41f5607b214e1ecf server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:45.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.622+0000 7f001a000640 1 -- 172.21.15.96:0/3014564505 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0008037070 con 0x7f001c06dc20 2024-05-28T06:08:45.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.622+0000 7f001a000640 1 -- 172.21.15.96:0/3014564505 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0008032440 con 0x7f001c06dc20 2024-05-28T06:08:45.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.622+0000 7f001a000640 1 -- 172.21.15.96:0/3014564505 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0008032810 con 0x7f001c06dc20 2024-05-28T06:08:45.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.623+0000 7f0021200640 1 -- 172.21.15.96:0/3014564505 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f001c06dc20 msgr2=0x7f001c15f9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:45.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.623+0000 7f0021200640 1 --2- 172.21.15.96:0/3014564505 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f001c06dc20 0x7f001c15f9a0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f001c072bd0 tx=0x7f000802a230 comp rx=0 tx=0).stop 2024-05-28T06:08:45.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.623+0000 7f0021200640 1 -- 172.21.15.96:0/3014564505 shutdown_connections 2024-05-28T06:08:45.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.623+0000 7f0021200640 1 --2- 172.21.15.96:0/3014564505 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f001c06dc20 0x7f001c15f9a0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:45.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.623+0000 7f0021200640 1 --2- 172.21.15.96:0/3014564505 >> v2:172.21.15.123:3300/0 conn(0x7f001c15b3a0 0x7f001c0b32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:45.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.623+0000 7f0021200640 1 -- 172.21.15.96:0/3014564505 >> 172.21.15.96:0/3014564505 conn(0x7f001c0b1fc0 msgr2=0x7f001c0b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:45.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.623+0000 7f0021200640 1 -- 172.21.15.96:0/3014564505 shutdown_connections 2024-05-28T06:08:45.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.623+0000 7f0021200640 1 -- 172.21.15.96:0/3014564505 wait complete. 2024-05-28T06:08:45.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.625+0000 7f0021200640 1 Processor -- start 2024-05-28T06:08:45.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.625+0000 7f0021200640 1 -- start start 2024-05-28T06:08:45.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.626+0000 7f0021200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f001c06dc20 0x7f001c15b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:45.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.626+0000 7f0021200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f001c072c00 0x7f001c145690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:45.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.626+0000 7f0021200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f001c15b3a0 0x7f001c0c9d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:45.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.626+0000 7f0021200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f001c15b780 con 0x7f001c072c00 2024-05-28T06:08:45.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.626+0000 7f0021200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f001c15fee0 con 0x7f001c06dc20 2024-05-28T06:08:45.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.626+0000 7f0021200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f001c0ae550 con 0x7f001c15b3a0 2024-05-28T06:08:45.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.626+0000 7f001aa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f001c072c00 0x7f001c145690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:45.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.626+0000 7f001aa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f001c072c00 0x7f001c145690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58224/0 (socket says 172.21.15.96:58224) 2024-05-28T06:08:45.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.626+0000 7f001aa00640 1 -- 172.21.15.96:0/2219195172 learned_addr learned my addr 172.21.15.96:0/2219195172 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:08:45.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.627+0000 7f001b400640 1 --2- 172.21.15.96:0/2219195172 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f001c06dc20 0x7f001c15b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:45.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.627+0000 7f001aa00640 1 -- 172.21.15.96:0/2219195172 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f001c15b3a0 msgr2=0x7f001c0c9d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:45.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.627+0000 7f001be00640 1 --2- 172.21.15.96:0/2219195172 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f001c15b3a0 0x7f001c0c9d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:45.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.627+0000 7f001aa00640 1 --2- 172.21.15.96:0/2219195172 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f001c15b3a0 0x7f001c0c9d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:45.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.627+0000 7f001aa00640 1 -- 172.21.15.96:0/2219195172 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f001c06dc20 msgr2=0x7f001c15b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:45.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.627+0000 7f001aa00640 1 --2- 172.21.15.96:0/2219195172 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f001c06dc20 0x7f001c15b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:45.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.627+0000 7f001aa00640 1 -- 172.21.15.96:0/2219195172 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f001c0cae50 con 0x7f001c072c00 2024-05-28T06:08:45.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.627+0000 7f001b400640 1 --2- 172.21.15.96:0/2219195172 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f001c06dc20 0x7f001c15b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:08:45.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.627+0000 7f001be00640 1 --2- 172.21.15.96:0/2219195172 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f001c15b3a0 0x7f001c0c9d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:08:45.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.627+0000 7f001aa00640 1 --2- 172.21.15.96:0/2219195172 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f001c072c00 0x7f001c145690 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f0008000c00 tx=0x7f00080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:45.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.628+0000 7f0013400640 1 -- 172.21.15.96:0/2219195172 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0008041070 con 0x7f001c072c00 2024-05-28T06:08:45.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.628+0000 7f0013400640 1 -- 172.21.15.96:0/2219195172 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f000803c870 con 0x7f001c072c00 2024-05-28T06:08:45.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.628+0000 7f0013400640 1 -- 172.21.15.96:0/2219195172 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0008037040 con 0x7f001c072c00 2024-05-28T06:08:45.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.628+0000 7f0021200640 1 -- 172.21.15.96:0/2219195172 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f001c0cb130 con 0x7f001c072c00 2024-05-28T06:08:45.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.628+0000 7f0021200640 1 -- 172.21.15.96:0/2219195172 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f001c0c97e0 con 0x7f001c072c00 2024-05-28T06:08:45.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.629+0000 7f0021200640 1 -- 172.21.15.96:0/2219195172 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f001c0cb650 con 0x7f001c072c00 2024-05-28T06:08:45.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.630+0000 7f0013400640 1 -- 172.21.15.96:0/2219195172 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 9) ==== 91295+0+0 (secure 0 0 0) 0x7f0008044070 con 0x7f001c072c00 2024-05-28T06:08:45.631 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.631+0000 7f0013400640 1 --2- 172.21.15.96:0/2219195172 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7efffc06cc40 0x7efffc06f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:45.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.631+0000 7f0013400640 1 -- 172.21.15.96:0/2219195172 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 4971+0+0 (secure 0 0 0) 0x7f00080ac890 con 0x7f001c072c00 2024-05-28T06:08:45.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.632+0000 7f001b400640 1 --2- 172.21.15.96:0/2219195172 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7efffc06cc40 0x7efffc06f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:45.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.633+0000 7f001b400640 1 --2- 172.21.15.96:0/2219195172 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7efffc06cc40 0x7efffc06f100 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f000c00a870 tx=0x7f000c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:45.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.636+0000 7f0013400640 1 -- 172.21.15.96:0/2219195172 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f001c0cb650 con 0x7f001c072c00 2024-05-28T06:08:45.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.910+0000 7f0021200640 1 -- 172.21.15.96:0/2219195172 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgressReporter.test_progress_bar_for_1_clone"]} v 0) -- 0x7f001c0681e0 con 0x7f001c072c00 2024-05-28T06:08:45.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.937+0000 7f0013400640 1 -- 172.21.15.96:0/2219195172 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgressReporter.test_progress_bar_for_1_clone"]}]=0 v219) ==== 160+0+0 (secure 0 0 0) 0x7f001c0681e0 con 0x7f001c072c00 2024-05-28T06:08:45.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.939+0000 7f0021200640 1 -- 172.21.15.96:0/2219195172 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7efffc06cc40 msgr2=0x7efffc06f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:45.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.939+0000 7f0021200640 1 --2- 172.21.15.96:0/2219195172 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7efffc06cc40 0x7efffc06f100 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f000c00a870 tx=0x7f000c008040 comp rx=0 tx=0).stop 2024-05-28T06:08:45.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.939+0000 7f0021200640 1 -- 172.21.15.96:0/2219195172 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f001c072c00 msgr2=0x7f001c145690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:45.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.939+0000 7f0021200640 1 --2- 172.21.15.96:0/2219195172 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f001c072c00 0x7f001c145690 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f0008000c00 tx=0x7f00080046c0 comp rx=0 tx=0).stop 2024-05-28T06:08:45.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.939+0000 7f0021200640 1 -- 172.21.15.96:0/2219195172 shutdown_connections 2024-05-28T06:08:45.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.939+0000 7f0021200640 1 --2- 172.21.15.96:0/2219195172 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7efffc06cc40 0x7efffc06f100 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:45.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.939+0000 7f0021200640 1 --2- 172.21.15.96:0/2219195172 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f001c15b3a0 0x7f001c0c9d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:45.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.939+0000 7f0021200640 1 --2- 172.21.15.96:0/2219195172 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f001c072c00 0x7f001c145690 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:45.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.939+0000 7f0021200640 1 --2- 172.21.15.96:0/2219195172 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f001c06dc20 0x7f001c15b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:45.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.940+0000 7f0021200640 1 -- 172.21.15.96:0/2219195172 >> 172.21.15.96:0/2219195172 conn(0x7f001c0b1fc0 msgr2=0x7f001c14dda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:45.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.940+0000 7f0021200640 1 -- 172.21.15.96:0/2219195172 shutdown_connections 2024-05-28T06:08:45.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:45.940+0000 7f0021200640 1 -- 172.21.15.96:0/2219195172 wait complete. 2024-05-28T06:08:45.958 INFO:tasks.cephfs_test_runner:Test that when one clone is in progress, one progress bar is printed ... ok 2024-05-28T06:08:45.959 INFO:tasks.cephfs_test_runner:Starting test: test_progress_bars_for_4_clones (tasks.cephfs.test_volumes.TestCloneProgressReporter) Test that one progress bar is printed when number of clone jobs is 2024-05-28T06:08:45.959 INFO:tasks.cephfs_test_runner:test_progress_bars_for_4_clones (tasks.cephfs.test_volumes.TestCloneProgressReporter) 2024-05-28T06:08:45.959 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:08:46.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.155+0000 7f42f6200640 1 Processor -- start 2024-05-28T06:08:46.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.155+0000 7f42f6200640 1 -- start start 2024-05-28T06:08:46.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.156+0000 7f42f6200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f42f015b3a0 0x7f42f00b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:46.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.156+0000 7f42f6200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f42f0073310 0x7f42f015f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:46.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.156+0000 7f42f6200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42f00afd60 con 0x7f42f006dc20 2024-05-28T06:08:46.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.157+0000 7f42f6200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f42f006f980 con 0x7f42f0073310 2024-05-28T06:08:46.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.157+0000 7f42f6200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f42f00ae0c0 con 0x7f42f015b3a0 2024-05-28T06:08:46.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.157+0000 7f42f5800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f42f0073310 0x7f42f015f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:46.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.157+0000 7f42e7e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f42f006dc20 0x7f42f0072c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:45616/0 (socket says 172.21.15.96:45616) 2024-05-28T06:08:46.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.157+0000 7f42e7e00640 1 -- 172.21.15.96:0/2407795799 learned_addr learned my addr 172.21.15.96:0/2407795799 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:08:46.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.157+0000 7f42f4e00640 1 --2- 172.21.15.96:0/2407795799 >> v2:172.21.15.123:3300/0 conn(0x7f42f015b3a0 0x7f42f00b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:46.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.157+0000 7f42f5800640 1 -- 172.21.15.96:0/2407795799 >> v2:172.21.15.123:3300/0 conn(0x7f42f015b3a0 msgr2=0x7f42f00b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:46.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.157+0000 7f42f5800640 1 --2- 172.21.15.96:0/2407795799 >> v2:172.21.15.123:3300/0 conn(0x7f42f015b3a0 0x7f42f00b32c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:46.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.157+0000 7f42f5800640 1 -- 172.21.15.96:0/2407795799 >> v1:172.21.15.96:6789/0 conn(0x7f42f006dc20 legacy=0x7f42f0072c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:46.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.157+0000 7f42efe00640 1 -- 172.21.15.96:0/2407795799 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 279092139 0 0) 0x7f42f00afd60 con 0x7f42f006dc20 2024-05-28T06:08:46.159 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.157+0000 7f42f5800640 1 -- 172.21.15.96:0/2407795799 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42f00af7a0 con 0x7f42f0073310 2024-05-28T06:08:46.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.158+0000 7f42f5800640 1 --2- 172.21.15.96:0/2407795799 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f42f0073310 0x7f42f015f9a0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f42f0072bd0 tx=0x7f42e802a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=230fbad32f26a141 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:46.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.159+0000 7f42efe00640 1 -- 172.21.15.96:0/2407795799 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42e8007740 con 0x7f42f0073310 2024-05-28T06:08:46.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.159+0000 7f42efe00640 1 -- 172.21.15.96:0/2407795799 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42e80078e0 con 0x7f42f0073310 2024-05-28T06:08:46.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.159+0000 7f42efe00640 1 -- 172.21.15.96:0/2407795799 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42e8007cb0 con 0x7f42f0073310 2024-05-28T06:08:46.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.159+0000 7f42f6200640 1 -- 172.21.15.96:0/2407795799 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f42f0073310 msgr2=0x7f42f015f9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:46.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.159+0000 7f42f6200640 1 --2- 172.21.15.96:0/2407795799 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f42f0073310 0x7f42f015f9a0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f42f0072bd0 tx=0x7f42e802a680 comp rx=0 tx=0).stop 2024-05-28T06:08:46.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.159+0000 7f42f6200640 1 -- 172.21.15.96:0/2407795799 shutdown_connections 2024-05-28T06:08:46.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.159+0000 7f42f6200640 1 --2- 172.21.15.96:0/2407795799 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f42f0073310 0x7f42f015f9a0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:46.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.159+0000 7f42f6200640 1 --2- 172.21.15.96:0/2407795799 >> v2:172.21.15.123:3300/0 conn(0x7f42f015b3a0 0x7f42f00b32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:46.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.159+0000 7f42f6200640 1 -- 172.21.15.96:0/2407795799 >> 172.21.15.96:0/2407795799 conn(0x7f42f00b1fc0 msgr2=0x7f42f00b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:46.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.159+0000 7f42f6200640 1 -- 172.21.15.96:0/2407795799 shutdown_connections 2024-05-28T06:08:46.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.160+0000 7f42f6200640 1 -- 172.21.15.96:0/2407795799 wait complete. 2024-05-28T06:08:46.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.161+0000 7f42f6200640 1 Processor -- start 2024-05-28T06:08:46.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.161+0000 7f42f6200640 1 -- start start 2024-05-28T06:08:46.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.162+0000 7f42f6200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f42f006dc20 0x7f42f00ce700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:46.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.162+0000 7f42f6200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42f0073310 0x7f42f01f3180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:46.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.162+0000 7f42f6200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f42f015b3a0 0x7f42f01f36c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:46.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.162+0000 7f42f6200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f42f015b780 con 0x7f42f006dc20 2024-05-28T06:08:46.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.162+0000 7f42f6200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f42f015fee0 con 0x7f42f0073310 2024-05-28T06:08:46.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.162+0000 7f42f6200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f42f00ae550 con 0x7f42f015b3a0 2024-05-28T06:08:46.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.163+0000 7f42f4e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f42f006dc20 0x7f42f00ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:46.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.163+0000 7f42f4e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f42f006dc20 0x7f42f00ce700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58236/0 (socket says 172.21.15.96:58236) 2024-05-28T06:08:46.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.163+0000 7f42f4e00640 1 -- 172.21.15.96:0/4057140488 learned_addr learned my addr 172.21.15.96:0/4057140488 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:08:46.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.163+0000 7f42f4e00640 1 -- 172.21.15.96:0/4057140488 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f42f015b3a0 msgr2=0x7f42f01f36c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:46.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.163+0000 7f42e7e00640 1 --2- 172.21.15.96:0/4057140488 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42f0073310 0x7f42f01f3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:46.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.163+0000 7f42f4e00640 1 --2- 172.21.15.96:0/4057140488 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f42f015b3a0 0x7f42f01f36c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:46.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.163+0000 7f42f4e00640 1 -- 172.21.15.96:0/4057140488 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42f0073310 msgr2=0x7f42f01f3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:46.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.163+0000 7f42f4e00640 1 --2- 172.21.15.96:0/4057140488 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42f0073310 0x7f42f01f3180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:46.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.163+0000 7f42f4e00640 1 -- 172.21.15.96:0/4057140488 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42f00cef30 con 0x7f42f006dc20 2024-05-28T06:08:46.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.163+0000 7f42f4e00640 1 --2- 172.21.15.96:0/4057140488 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f42f006dc20 0x7f42f00ce700 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f42e000c820 tx=0x7f42e000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:46.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.164+0000 7f42ed600640 1 -- 172.21.15.96:0/4057140488 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42e0015070 con 0x7f42f006dc20 2024-05-28T06:08:46.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.164+0000 7f42ed600640 1 -- 172.21.15.96:0/4057140488 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42e0004020 con 0x7f42f006dc20 2024-05-28T06:08:46.166 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.164+0000 7f42ed600640 1 -- 172.21.15.96:0/4057140488 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42e00043f0 con 0x7f42f006dc20 2024-05-28T06:08:46.166 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.164+0000 7f42f6200640 1 -- 172.21.15.96:0/4057140488 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42f00cf210 con 0x7f42f006dc20 2024-05-28T06:08:46.166 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.164+0000 7f42f6200640 1 -- 172.21.15.96:0/4057140488 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42f0149200 con 0x7f42f006dc20 2024-05-28T06:08:46.166 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.165+0000 7f42f6200640 1 -- 172.21.15.96:0/4057140488 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42f00689e0 con 0x7f42f006dc20 2024-05-28T06:08:46.166 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.166+0000 7f42ed600640 1 -- 172.21.15.96:0/4057140488 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 9) ==== 91295+0+0 (secure 0 0 0) 0x7f42e0007830 con 0x7f42f006dc20 2024-05-28T06:08:46.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.167+0000 7f42ed600640 1 --2- 172.21.15.96:0/4057140488 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f42cc06cbc0 0x7f42cc06f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:46.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.167+0000 7f42ed600640 1 -- 172.21.15.96:0/4057140488 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 4971+0+0 (secure 0 0 0) 0x7f42e008bc50 con 0x7f42f006dc20 2024-05-28T06:08:46.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.167+0000 7f42e7e00640 1 --2- 172.21.15.96:0/4057140488 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f42cc06cbc0 0x7f42cc06f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:46.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.168+0000 7f42e7e00640 1 --2- 172.21.15.96:0/4057140488 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f42cc06cbc0 0x7f42cc06f080 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f42d8007fd0 tx=0x7f42d8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:46.172 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.172+0000 7f42ed600640 1 -- 172.21.15.96:0/4057140488 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f42e0059ec0 con 0x7f42f006dc20 2024-05-28T06:08:46.374 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.374+0000 7f42f6200640 1 -- 172.21.15.96:0/4057140488 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f42f00b2e90 con 0x7f42f006dc20 2024-05-28T06:08:46.375 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.375+0000 7f42ed600640 1 -- 172.21.15.96:0/4057140488 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) ==== 74+0+13075 (secure 0 0 0) 0x7f42e00597c0 con 0x7f42f006dc20 2024-05-28T06:08:46.376 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:08:46.376 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":37,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:08:44.929209+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.123:6837/458334664":"2024-05-29T06:08:41.909118+0000","172.21.15.123:6835/458334664":"2024-05-29T06:08:41.909118+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:08:46.378 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.377+0000 7f42f6200640 1 -- 172.21.15.96:0/4057140488 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f42cc06cbc0 msgr2=0x7f42cc06f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:46.378 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.377+0000 7f42f6200640 1 --2- 172.21.15.96:0/4057140488 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f42cc06cbc0 0x7f42cc06f080 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f42d8007fd0 tx=0x7f42d8009040 comp rx=0 tx=0).stop 2024-05-28T06:08:46.378 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.377+0000 7f42f6200640 1 -- 172.21.15.96:0/4057140488 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f42f006dc20 msgr2=0x7f42f00ce700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:46.378 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.377+0000 7f42f6200640 1 --2- 172.21.15.96:0/4057140488 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f42f006dc20 0x7f42f00ce700 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f42e000c820 tx=0x7f42e000cce0 comp rx=0 tx=0).stop 2024-05-28T06:08:46.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.378+0000 7f42f6200640 1 -- 172.21.15.96:0/4057140488 shutdown_connections 2024-05-28T06:08:46.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.378+0000 7f42f6200640 1 --2- 172.21.15.96:0/4057140488 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f42cc06cbc0 0x7f42cc06f080 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:46.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.378+0000 7f42f6200640 1 --2- 172.21.15.96:0/4057140488 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f42f015b3a0 0x7f42f01f36c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:46.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.378+0000 7f42f6200640 1 --2- 172.21.15.96:0/4057140488 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f42f0073310 0x7f42f01f3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:46.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.378+0000 7f42f6200640 1 --2- 172.21.15.96:0/4057140488 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f42f006dc20 0x7f42f00ce700 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:46.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.378+0000 7f42f6200640 1 -- 172.21.15.96:0/4057140488 >> 172.21.15.96:0/4057140488 conn(0x7f42f00b1fc0 msgr2=0x7f42f0071bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:46.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.378+0000 7f42f6200640 1 -- 172.21.15.96:0/4057140488 shutdown_connections 2024-05-28T06:08:46.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.378+0000 7f42f6200640 1 -- 172.21.15.96:0/4057140488 wait complete. 2024-05-28T06:08:46.399 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-28T06:03:59.517102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-28T06:08:46.399 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-28T06:08:46.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.599+0000 7f1536000640 1 Processor -- start 2024-05-28T06:08:46.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.599+0000 7f1536000640 1 -- start start 2024-05-28T06:08:46.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.600+0000 7f1536000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f15300cd2a0 0x7f15300b4d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:46.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.601+0000 7f1536000640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f153015d680 con 0x7f1530153510 2024-05-28T06:08:46.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.601+0000 7f1536000640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f153015d900 con 0x7f153015cb60 2024-05-28T06:08:46.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.601+0000 7f1536000640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f15301538f0 con 0x7f15300cd2a0 2024-05-28T06:08:46.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.601+0000 7f1534c00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f1530153510 0x7f15300ccb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:45628/0 (socket says 172.21.15.96:45628) 2024-05-28T06:08:46.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.601+0000 7f1534c00640 1 -- 172.21.15.96:0/342132238 learned_addr learned my addr 172.21.15.96:0/342132238 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:08:46.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.601+0000 7f152f400640 1 -- 172.21.15.96:0/342132238 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3033241816 0 0) 0x7f1524001110 con 0x7f1530153510 2024-05-28T06:08:46.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.601+0000 7f152f400640 1 -- 172.21.15.96:0/342132238 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1518003520 con 0x7f1530153510 2024-05-28T06:08:46.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.602+0000 7f152fe00640 1 --2- 172.21.15.96:0/342132238 >> v2:172.21.15.123:3300/0 conn(0x7f15300cd2a0 0x7f15300b4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:46.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.602+0000 7f152f400640 1 -- 172.21.15.96:0/342132238 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 790005227 0 0) 0x7f1518003520 con 0x7f1530153510 2024-05-28T06:08:46.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.602+0000 7f152f400640 1 -- 172.21.15.96:0/342132238 >> v2:172.21.15.123:3300/0 conn(0x7f15300cd2a0 msgr2=0x7f15300b4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:46.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.602+0000 7f152f400640 1 --2- 172.21.15.96:0/342132238 >> v2:172.21.15.123:3300/0 conn(0x7f15300cd2a0 0x7f15300b4d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:46.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.602+0000 7f152f400640 1 -- 172.21.15.96:0/342132238 >> v1:172.21.15.123:6789/0 conn(0x7f153015cb60 legacy=0x7f153015cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:46.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.602+0000 7f152f400640 1 -- 172.21.15.96:0/342132238 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15300b3f90 con 0x7f1530153510 2024-05-28T06:08:46.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.602+0000 7f152f400640 1 -- 172.21.15.96:0/342132238 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f1524001840 con 0x7f1530153510 2024-05-28T06:08:46.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.602+0000 7f152f400640 1 -- 172.21.15.96:0/342132238 >> v1:172.21.15.96:6789/0 conn(0x7f1530153510 legacy=0x7f15300ccb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:46.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.602+0000 7f152f400640 1 --2- 172.21.15.96:0/342132238 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1518003a00 0x7f1518003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:46.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.602+0000 7f152f400640 1 --2- 172.21.15.96:0/342132238 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1518004800 0x7f1518004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:46.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.602+0000 7f152f400640 1 --2- 172.21.15.96:0/342132238 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1518005290 0x7f151800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:46.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.602+0000 7f152f400640 1 -- 172.21.15.96:0/342132238 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f151800c1b0 con 0x7f1518003a00 2024-05-28T06:08:46.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.602+0000 7f152f400640 1 -- 172.21.15.96:0/342132238 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f151800c360 con 0x7f1518005290 2024-05-28T06:08:46.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.602+0000 7f152f400640 1 -- 172.21.15.96:0/342132238 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f151800c560 con 0x7f1518004800 2024-05-28T06:08:46.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.602+0000 7f1534c00640 1 --2- 172.21.15.96:0/342132238 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1518003a00 0x7f1518003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:46.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.603+0000 7f1535600640 1 --2- 172.21.15.96:0/342132238 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1518005290 0x7f151800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:46.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.603+0000 7f152fe00640 1 --2- 172.21.15.96:0/342132238 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1518004800 0x7f1518004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:46.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.603+0000 7f1534c00640 1 -- 172.21.15.96:0/342132238 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1518004800 msgr2=0x7f1518004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:46.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.603+0000 7f1534c00640 1 --2- 172.21.15.96:0/342132238 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1518004800 0x7f1518004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:46.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.603+0000 7f1534c00640 1 -- 172.21.15.96:0/342132238 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1518005290 msgr2=0x7f151800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:46.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.603+0000 7f1534c00640 1 --2- 172.21.15.96:0/342132238 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1518005290 0x7f151800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:46.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.603+0000 7f1534c00640 1 -- 172.21.15.96:0/342132238 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1518003520 con 0x7f1518003a00 2024-05-28T06:08:46.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.603+0000 7f152fe00640 1 --2- 172.21.15.96:0/342132238 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1518004800 0x7f1518004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:08:46.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.603+0000 7f1535600640 1 --2- 172.21.15.96:0/342132238 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1518005290 0x7f151800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:08:46.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.603+0000 7f1535600640 1 -- 172.21.15.96:0/342132238 reap_dead start 2024-05-28T06:08:46.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.604+0000 7f1534c00640 1 --2- 172.21.15.96:0/342132238 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1518003a00 0x7f1518003e00 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f152400c320 tx=0x7f1518005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d85a733ad32b370 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:46.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.604+0000 7f152f400640 1 -- 172.21.15.96:0/342132238 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1524035070 con 0x7f1518003a00 2024-05-28T06:08:46.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.604+0000 7f152f400640 1 -- 172.21.15.96:0/342132238 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1524005ad0 con 0x7f1518003a00 2024-05-28T06:08:46.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.604+0000 7f152f400640 1 -- 172.21.15.96:0/342132238 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1524007e10 con 0x7f1518003a00 2024-05-28T06:08:46.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.605+0000 7f1536000640 1 -- 172.21.15.96:0/342132238 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1518003a00 msgr2=0x7f1518003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:46.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.605+0000 7f1536000640 1 --2- 172.21.15.96:0/342132238 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1518003a00 0x7f1518003e00 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f152400c320 tx=0x7f1518005830 comp rx=0 tx=0).stop 2024-05-28T06:08:46.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.605+0000 7f1536000640 1 -- 172.21.15.96:0/342132238 shutdown_connections 2024-05-28T06:08:46.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.605+0000 7f1536000640 1 --2- 172.21.15.96:0/342132238 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1518003a00 0x7f1518003e00 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:46.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.605+0000 7f1536000640 1 -- 172.21.15.96:0/342132238 >> 172.21.15.96:0/342132238 conn(0x7f153006b900 msgr2=0x7f153006bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:46.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.605+0000 7f1536000640 1 -- 172.21.15.96:0/342132238 shutdown_connections 2024-05-28T06:08:46.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.605+0000 7f1536000640 1 -- 172.21.15.96:0/342132238 wait complete. 2024-05-28T06:08:46.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.607+0000 7f1536000640 1 Processor -- start 2024-05-28T06:08:46.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.607+0000 7f1536000640 1 -- start start 2024-05-28T06:08:46.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.608+0000 7f1536000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1518003a00 0x7f15300cf760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:46.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.608+0000 7f1536000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1530071a80 0x7f15301f3c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:46.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.608+0000 7f1536000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f15301450a0 0x7f15300b05a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:46.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.608+0000 7f1536000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f15300b4550 con 0x7f1518003a00 2024-05-28T06:08:46.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.608+0000 7f1536000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f15300b48c0 con 0x7f15301450a0 2024-05-28T06:08:46.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.608+0000 7f1536000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f15300b4d50 con 0x7f1530071a80 2024-05-28T06:08:46.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.608+0000 7f1534c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1518003a00 0x7f15300cf760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:46.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.608+0000 7f1534c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1518003a00 0x7f15300cf760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58256/0 (socket says 172.21.15.96:58256) 2024-05-28T06:08:46.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.608+0000 7f1534c00640 1 -- 172.21.15.96:0/3887292136 learned_addr learned my addr 172.21.15.96:0/3887292136 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:08:46.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.608+0000 7f152fe00640 1 --2- 172.21.15.96:0/3887292136 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1530071a80 0x7f15301f3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:46.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.608+0000 7f1534c00640 1 -- 172.21.15.96:0/3887292136 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1530071a80 msgr2=0x7f15301f3c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:46.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.609+0000 7f1534c00640 1 --2- 172.21.15.96:0/3887292136 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1530071a80 0x7f15301f3c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:46.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.609+0000 7f1535600640 1 --2- 172.21.15.96:0/3887292136 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f15301450a0 0x7f15300b05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:46.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.609+0000 7f1534c00640 1 -- 172.21.15.96:0/3887292136 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f15301450a0 msgr2=0x7f15300b05a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:46.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.609+0000 7f1534c00640 1 --2- 172.21.15.96:0/3887292136 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f15301450a0 0x7f15300b05a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:46.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.609+0000 7f1534c00640 1 -- 172.21.15.96:0/3887292136 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1530071ff0 con 0x7f1518003a00 2024-05-28T06:08:46.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.609+0000 7f1534c00640 1 --2- 172.21.15.96:0/3887292136 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1518003a00 0x7f15300cf760 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f152400c320 tx=0x7f15300b4160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:46.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.609+0000 7f152cc00640 1 -- 172.21.15.96:0/3887292136 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1524040070 con 0x7f1518003a00 2024-05-28T06:08:46.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.609+0000 7f152cc00640 1 -- 172.21.15.96:0/3887292136 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1524001110 con 0x7f1518003a00 2024-05-28T06:08:46.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.609+0000 7f152cc00640 1 -- 172.21.15.96:0/3887292136 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1524035030 con 0x7f1518003a00 2024-05-28T06:08:46.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.609+0000 7f1536000640 1 -- 172.21.15.96:0/3887292136 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15300722d0 con 0x7f1518003a00 2024-05-28T06:08:46.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.609+0000 7f1536000640 1 -- 172.21.15.96:0/3887292136 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1530072890 con 0x7f1518003a00 2024-05-28T06:08:46.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.611+0000 7f1536000640 1 -- 172.21.15.96:0/3887292136 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15300689e0 con 0x7f1518003a00 2024-05-28T06:08:46.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.612+0000 7f152cc00640 1 -- 172.21.15.96:0/3887292136 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 9) ==== 91295+0+0 (secure 0 0 0) 0x7f1524007270 con 0x7f1518003a00 2024-05-28T06:08:46.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.612+0000 7f152cc00640 1 --2- 172.21.15.96:0/3887292136 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f151006cbf0 0x7f151006f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:46.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.613+0000 7f152cc00640 1 -- 172.21.15.96:0/3887292136 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 4971+0+0 (secure 0 0 0) 0x7f15240abb50 con 0x7f1518003a00 2024-05-28T06:08:46.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.613+0000 7f152fe00640 1 --2- 172.21.15.96:0/3887292136 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f151006cbf0 0x7f151006f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:46.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.614+0000 7f152fe00640 1 --2- 172.21.15.96:0/3887292136 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f151006cbf0 0x7f151006f0b0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f1530160a50 tx=0x7f151c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:46.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.617+0000 7f152cc00640 1 -- 172.21.15.96:0/3887292136 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f152402d070 con 0x7f1518003a00 2024-05-28T06:08:46.821 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.821+0000 7f1536000640 1 -- 172.21.15.96:0/3887292136 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f15300b1520 con 0x7f1518003a00 2024-05-28T06:08:46.822 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.822+0000 7f152cc00640 1 -- 172.21.15.96:0/3887292136 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v37) ==== 93+0+10 (secure 0 0 0) 0x7f153015d680 con 0x7f1518003a00 2024-05-28T06:08:46.822 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 1 2024-05-28T06:08:46.825 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.824+0000 7f1536000640 1 -- 172.21.15.96:0/3887292136 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f151006cbf0 msgr2=0x7f151006f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:46.825 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.824+0000 7f1536000640 1 --2- 172.21.15.96:0/3887292136 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f151006cbf0 0x7f151006f0b0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f1530160a50 tx=0x7f151c008040 comp rx=0 tx=0).stop 2024-05-28T06:08:46.825 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.824+0000 7f1536000640 1 -- 172.21.15.96:0/3887292136 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1518003a00 msgr2=0x7f15300cf760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:46.825 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.824+0000 7f1536000640 1 --2- 172.21.15.96:0/3887292136 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1518003a00 0x7f15300cf760 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f152400c320 tx=0x7f15300b4160 comp rx=0 tx=0).stop 2024-05-28T06:08:46.825 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.824+0000 7f1536000640 1 -- 172.21.15.96:0/3887292136 shutdown_connections 2024-05-28T06:08:46.826 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.824+0000 7f1536000640 1 --2- 172.21.15.96:0/3887292136 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f151006cbf0 0x7f151006f0b0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:46.826 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.824+0000 7f1536000640 1 --2- 172.21.15.96:0/3887292136 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f15301450a0 0x7f15300b05a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:46.826 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.824+0000 7f1536000640 1 --2- 172.21.15.96:0/3887292136 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1530071a80 0x7f15301f3c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:46.826 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.824+0000 7f1536000640 1 --2- 172.21.15.96:0/3887292136 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1518003a00 0x7f15300cf760 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:46.826 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.824+0000 7f1536000640 1 -- 172.21.15.96:0/3887292136 >> 172.21.15.96:0/3887292136 conn(0x7f153006b900 msgr2=0x7f15300afdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:46.826 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.825+0000 7f1536000640 1 -- 172.21.15.96:0/3887292136 shutdown_connections 2024-05-28T06:08:46.826 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:46.825+0000 7f1536000640 1 -- 172.21.15.96:0/3887292136 wait complete. 2024-05-28T06:08:46.844 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestCloneProgressReporter.test_progress_bars_for_4_clones' 2024-05-28T06:08:47.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:47.038+0000 7f4990c00640 1 Processor -- start 2024-05-28T06:08:47.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:47.038+0000 7f4990c00640 1 -- start start 2024-05-28T06:08:47.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:47.040+0000 7f4990c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f498c0d04a0 0x7f498c0d0900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:47.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:47.040+0000 7f4990c00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f498c0ad130 0x7f498c0ad5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:47.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:47.040+0000 7f4990c00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f498c06ed10 con 0x7f498c06f860 2024-05-28T06:08:47.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:47.040+0000 7f4990c00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f498c0d0e40 con 0x7f498c0ad130 2024-05-28T06:08:47.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:47.040+0000 7f4990c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f498c15cb00 con 0x7f498c0d04a0 2024-05-28T06:08:47.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:47.040+0000 7f498a800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f498c0ad130 0x7f498c0ad5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:47.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:47.040+0000 7f498a800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f498c0ad130 0x7f498c0ad5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58260/0 (socket says 172.21.15.96:58260) 2024-05-28T06:08:47.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:47.040+0000 7f498a800640 1 -- 172.21.15.96:0/1337193021 learned_addr learned my addr 172.21.15.96:0/1337193021 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:08:47.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:47.041+0000 7f4989400640 1 --2- 172.21.15.96:0/1337193021 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f498c0d04a0 0x7f498c0d0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:47.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:47.041+0000 7f498a800640 1 -- 172.21.15.96:0/1337193021 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f498c0d04a0 msgr2=0x7f498c0d0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:47.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:47.041+0000 7f498a800640 1 --2- 172.21.15.96:0/1337193021 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f498c0d04a0 0x7f498c0d0900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:47.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:47.041+0000 7f498a800640 1 -- 172.21.15.96:0/1337193021 >> v1:172.21.15.123:6789/0 conn(0x7f498c06f860 legacy=0x7f498c0b3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:47.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:47.041+0000 7f498a800640 1 -- 172.21.15.96:0/1337193021 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f498c0b3cc0 con 0x7f498c0ad130 2024-05-28T06:08:47.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:47.042+0000 7f498a800640 1 --2- 172.21.15.96:0/1337193021 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f498c0ad130 0x7f498c0ad5a0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f4984009870 tx=0x7f498402a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0871519a6dae822 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:47.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:47.042+0000 7f4988a00640 1 -- 172.21.15.96:0/1337193021 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4984037070 con 0x7f498c0ad130 2024-05-28T06:08:47.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:47.042+0000 7f4988a00640 1 -- 172.21.15.96:0/1337193021 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4984032440 con 0x7f498c0ad130 2024-05-28T06:08:47.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:47.042+0000 7f4988a00640 1 -- 172.21.15.96:0/1337193021 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4984032810 con 0x7f498c0ad130 2024-05-28T06:08:47.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:47.043+0000 7f4990c00640 1 -- 172.21.15.96:0/1337193021 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f498c0ad130 msgr2=0x7f498c0ad5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:47.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:47.043+0000 7f4990c00640 1 --2- 172.21.15.96:0/1337193021 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f498c0ad130 0x7f498c0ad5a0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f4984009870 tx=0x7f498402a210 comp rx=0 tx=0).stop 2024-05-28T06:08:47.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:47.043+0000 7f4990c00640 1 -- 172.21.15.96:0/1337193021 shutdown_connections 2024-05-28T06:08:47.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:47.043+0000 7f4990c00640 1 --2- 172.21.15.96:0/1337193021 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f498c0ad130 0x7f498c0ad5a0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:47.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:47.043+0000 7f4990c00640 1 --2- 172.21.15.96:0/1337193021 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f498c0d04a0 0x7f498c0d0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:47.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:47.043+0000 7f4990c00640 1 -- 172.21.15.96:0/1337193021 >> 172.21.15.96:0/1337193021 conn(0x7f498c06bae0 msgr2=0x7f498c06bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:47.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:47.043+0000 7f4990c00640 1 -- 172.21.15.96:0/1337193021 shutdown_connections 2024-05-28T06:08:47.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:47.043+0000 7f4990c00640 1 -- 172.21.15.96:0/1337193021 wait complete. 2024-05-28T06:08:47.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:47.045+0000 7f4990c00640 1 Processor -- start 2024-05-28T06:08:47.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:47.045+0000 7f4990c00640 1 -- start start 2024-05-28T06:08:47.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:47.046+0000 7f4990c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f498c06f860 0x7f498c0c3a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:47.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:47.046+0000 7f4990c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f498c0ad130 0x7f498c0c7a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:47.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:47.046+0000 7f4990c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f498c0d04a0 0x7f498c1f7290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:47.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:47.046+0000 7f4990c00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f498c0b4560 con 0x7f498c0ad130 2024-05-28T06:08:47.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:47.046+0000 7f4990c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f498c0b4280 con 0x7f498c06f860 2024-05-28T06:08:47.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:47.046+0000 7f4990c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f498c0b4840 con 0x7f498c0d04a0 2024-05-28T06:08:47.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:47.046+0000 7f4989400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f498c0ad130 0x7f498c0c7a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:47.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:47.046+0000 7f4989400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f498c0ad130 0x7f498c0c7a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58272/0 (socket says 172.21.15.96:58272) 2024-05-28T06:08:47.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:47.046+0000 7f4989400640 1 -- 172.21.15.96:0/1913795170 learned_addr learned my addr 172.21.15.96:0/1913795170 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:08:47.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:47.047+0000 7f498a800640 1 --2- 172.21.15.96:0/1913795170 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f498c0d04a0 0x7f498c1f7290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:47.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:47.047+0000 7f4989400640 1 -- 172.21.15.96:0/1913795170 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f498c0d04a0 msgr2=0x7f498c1f7290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:47.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:47.047+0000 7f4989e00640 1 --2- 172.21.15.96:0/1913795170 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f498c06f860 0x7f498c0c3a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:47.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:47.047+0000 7f4989400640 1 --2- 172.21.15.96:0/1913795170 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f498c0d04a0 0x7f498c1f7290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:47.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:47.047+0000 7f4989400640 1 -- 172.21.15.96:0/1913795170 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f498c06f860 msgr2=0x7f498c0c3a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:47.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:47.047+0000 7f4989400640 1 --2- 172.21.15.96:0/1913795170 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f498c06f860 0x7f498c0c3a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:47.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:47.047+0000 7f4989400640 1 -- 172.21.15.96:0/1913795170 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f498c0c8440 con 0x7f498c0ad130 2024-05-28T06:08:47.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:47.047+0000 7f498a800640 1 --2- 172.21.15.96:0/1913795170 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f498c0d04a0 0x7f498c1f7290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:08:47.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:47.047+0000 7f4989e00640 1 --2- 172.21.15.96:0/1913795170 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f498c06f860 0x7f498c0c3a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:08:47.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:47.047+0000 7f4989400640 1 --2- 172.21.15.96:0/1913795170 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f498c0ad130 0x7f498c0c7a50 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f497c00ce10 tx=0x7f497c00a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:47.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:47.047+0000 7f4982000640 1 -- 172.21.15.96:0/1913795170 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f497c003b90 con 0x7f498c0ad130 2024-05-28T06:08:47.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:47.047+0000 7f4982000640 1 -- 172.21.15.96:0/1913795170 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f497c003d30 con 0x7f498c0ad130 2024-05-28T06:08:47.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:47.047+0000 7f4982000640 1 -- 172.21.15.96:0/1913795170 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f497c004240 con 0x7f498c0ad130 2024-05-28T06:08:47.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:47.047+0000 7f4990c00640 1 -- 172.21.15.96:0/1913795170 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f498c0ce970 con 0x7f498c0ad130 2024-05-28T06:08:47.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:47.047+0000 7f4990c00640 1 -- 172.21.15.96:0/1913795170 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f498c0c8720 con 0x7f498c0ad130 2024-05-28T06:08:47.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:47.049+0000 7f4990c00640 1 -- 172.21.15.96:0/1913795170 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f498c0689e0 con 0x7f498c0ad130 2024-05-28T06:08:47.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:47.050+0000 7f4982000640 1 -- 172.21.15.96:0/1913795170 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 9) ==== 91295+0+0 (secure 0 0 0) 0x7f497c0043e0 con 0x7f498c0ad130 2024-05-28T06:08:47.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:47.050+0000 7f4982000640 1 --2- 172.21.15.96:0/1913795170 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f496806cbc0 0x7f496806f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:47.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:47.051+0000 7f4982000640 1 -- 172.21.15.96:0/1913795170 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 4971+0+0 (secure 0 0 0) 0x7f497c08c9c0 con 0x7f498c0ad130 2024-05-28T06:08:47.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:47.051+0000 7f4989e00640 1 --2- 172.21.15.96:0/1913795170 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f496806cbc0 0x7f496806f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:47.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:47.052+0000 7f4989e00640 1 --2- 172.21.15.96:0/1913795170 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f496806cbc0 0x7f496806f080 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f497400a830 tx=0x7f4974008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:47.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:47.055+0000 7f4982000640 1 -- 172.21.15.96:0/1913795170 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f497c01d340 con 0x7f498c0ad130 2024-05-28T06:08:47.327 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:47.326+0000 7f4990c00640 1 -- 172.21.15.96:0/1913795170 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgressReporter.test_progress_bars_for_4_clones"]} v 0) -- 0x7f498c06e2f0 con 0x7f498c0ad130 2024-05-28T06:08:47.943 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:47.941+0000 7f4982000640 1 -- 172.21.15.96:0/1913795170 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgressReporter.test_progress_bars_for_4_clones"]}]=0 v221) ==== 165+0+0 (secure 0 0 0) 0x7f497c05ac20 con 0x7f498c0ad130 2024-05-28T06:08:47.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:47.944+0000 7f4990c00640 1 -- 172.21.15.96:0/1913795170 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f496806cbc0 msgr2=0x7f496806f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:47.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:47.945+0000 7f4990c00640 1 --2- 172.21.15.96:0/1913795170 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f496806cbc0 0x7f496806f080 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f497400a830 tx=0x7f4974008040 comp rx=0 tx=0).stop 2024-05-28T06:08:47.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:47.945+0000 7f4990c00640 1 -- 172.21.15.96:0/1913795170 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f498c0ad130 msgr2=0x7f498c0c7a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:47.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:47.945+0000 7f4990c00640 1 --2- 172.21.15.96:0/1913795170 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f498c0ad130 0x7f498c0c7a50 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f497c00ce10 tx=0x7f497c00a660 comp rx=0 tx=0).stop 2024-05-28T06:08:47.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:47.945+0000 7f4990c00640 1 -- 172.21.15.96:0/1913795170 shutdown_connections 2024-05-28T06:08:47.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:47.945+0000 7f4990c00640 1 --2- 172.21.15.96:0/1913795170 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f496806cbc0 0x7f496806f080 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:47.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:47.945+0000 7f4990c00640 1 --2- 172.21.15.96:0/1913795170 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f498c0d04a0 0x7f498c1f7290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:47.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:47.945+0000 7f4990c00640 1 --2- 172.21.15.96:0/1913795170 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f498c0ad130 0x7f498c0c7a50 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:47.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:47.945+0000 7f4990c00640 1 --2- 172.21.15.96:0/1913795170 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f498c06f860 0x7f498c0c3a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:47.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:47.945+0000 7f4990c00640 1 -- 172.21.15.96:0/1913795170 >> 172.21.15.96:0/1913795170 conn(0x7f498c06bae0 msgr2=0x7f498c071b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:47.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:47.945+0000 7f4990c00640 1 -- 172.21.15.96:0/1913795170 shutdown_connections 2024-05-28T06:08:47.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:47.945+0000 7f4990c00640 1 -- 172.21.15.96:0/1913795170 wait complete. 2024-05-28T06:08:47.968 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-05-28T06:08:48.171 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.169+0000 7f668d800640 1 Processor -- start 2024-05-28T06:08:48.171 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.169+0000 7f668d800640 1 -- start start 2024-05-28T06:08:48.172 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.170+0000 7f668d800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6688162480 0x7f6688160280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:48.172 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.171+0000 7f668d800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f668809b5d0 0x7f668815f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:48.172 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.171+0000 7f668d800640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f668806d1f0 con 0x7f668806ca10 2024-05-28T06:08:48.172 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.171+0000 7f668d800640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f668806d4d0 con 0x7f668809b5d0 2024-05-28T06:08:48.172 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.171+0000 7f668d800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f668806d7b0 con 0x7f6688162480 2024-05-28T06:08:48.174 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.171+0000 7f6686000640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f668806ca10 0x7f6688162860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:45630/0 (socket says 172.21.15.96:45630) 2024-05-28T06:08:48.174 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.171+0000 7f6686000640 1 -- 172.21.15.96:0/457437722 learned_addr learned my addr 172.21.15.96:0/457437722 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:08:48.174 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.172+0000 7f6686a00640 1 --2- 172.21.15.96:0/457437722 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6688162480 0x7f6688160280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:48.174 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.172+0000 7f6687400640 1 --2- 172.21.15.96:0/457437722 >> v2:172.21.15.123:3300/0 conn(0x7f668809b5d0 0x7f668815f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:48.174 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.172+0000 7f6685600640 1 -- 172.21.15.96:0/457437722 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3793520102 0 0) 0x7f668806d1f0 con 0x7f668806ca10 2024-05-28T06:08:48.174 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.172+0000 7f6685600640 1 -- 172.21.15.96:0/457437722 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f666c003520 con 0x7f668806ca10 2024-05-28T06:08:48.174 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.172+0000 7f6685600640 1 -- 172.21.15.96:0/457437722 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3212370840 0 0) 0x7f666c003520 con 0x7f668806ca10 2024-05-28T06:08:48.174 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.172+0000 7f6685600640 1 -- 172.21.15.96:0/457437722 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6688162480 msgr2=0x7f6688160280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:48.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.172+0000 7f6685600640 1 --2- 172.21.15.96:0/457437722 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6688162480 0x7f6688160280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:48.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.172+0000 7f6685600640 1 -- 172.21.15.96:0/457437722 >> v2:172.21.15.123:3300/0 conn(0x7f668809b5d0 msgr2=0x7f668815f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:48.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.172+0000 7f6685600640 1 --2- 172.21.15.96:0/457437722 >> v2:172.21.15.123:3300/0 conn(0x7f668809b5d0 0x7f668815f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:48.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.172+0000 7f6685600640 1 -- 172.21.15.96:0/457437722 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f668806da90 con 0x7f668806ca10 2024-05-28T06:08:48.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.172+0000 7f6685600640 1 -- 172.21.15.96:0/457437722 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f6674001670 con 0x7f668806ca10 2024-05-28T06:08:48.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.172+0000 7f6685600640 1 -- 172.21.15.96:0/457437722 >> v1:172.21.15.96:6789/0 conn(0x7f668806ca10 legacy=0x7f6688162860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:48.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.172+0000 7f6685600640 1 --2- 172.21.15.96:0/457437722 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f666c0039b0 0x7f666c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:48.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.172+0000 7f6685600640 1 --2- 172.21.15.96:0/457437722 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f666c004800 0x7f666c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:48.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.172+0000 7f6685600640 1 --2- 172.21.15.96:0/457437722 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6688162480 0x7f666c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:48.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.172+0000 7f6685600640 1 -- 172.21.15.96:0/457437722 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f668806d4d0 con 0x7f666c0039b0 2024-05-28T06:08:48.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.172+0000 7f6685600640 1 -- 172.21.15.96:0/457437722 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f668806d7b0 con 0x7f666c004800 2024-05-28T06:08:48.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.172+0000 7f6685600640 1 -- 172.21.15.96:0/457437722 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f666c00b9c0 con 0x7f6688162480 2024-05-28T06:08:48.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.173+0000 7f6686a00640 1 --2- 172.21.15.96:0/457437722 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f666c0039b0 0x7f666c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:48.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.173+0000 7f6686000640 1 --2- 172.21.15.96:0/457437722 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f666c004800 0x7f666c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:48.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.173+0000 7f6686a00640 1 -- 172.21.15.96:0/457437722 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6688162480 msgr2=0x7f666c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:48.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.173+0000 7f6686a00640 1 --2- 172.21.15.96:0/457437722 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6688162480 0x7f666c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:48.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.173+0000 7f6686a00640 1 -- 172.21.15.96:0/457437722 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f666c004800 msgr2=0x7f666c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:48.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.173+0000 7f6686a00640 1 --2- 172.21.15.96:0/457437722 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f666c004800 0x7f666c004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:48.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.173+0000 7f6686a00640 1 -- 172.21.15.96:0/457437722 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f666c003520 con 0x7f666c0039b0 2024-05-28T06:08:48.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.174+0000 7f6686a00640 1 --2- 172.21.15.96:0/457437722 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f666c0039b0 0x7f666c003dd0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f6678009ac0 tx=0x7f666c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65d1531d0c0b8e11 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:48.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.174+0000 7f6685600640 1 -- 172.21.15.96:0/457437722 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6678036070 con 0x7f666c0039b0 2024-05-28T06:08:48.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.174+0000 7f6685600640 1 -- 172.21.15.96:0/457437722 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6678031400 con 0x7f666c0039b0 2024-05-28T06:08:48.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.174+0000 7f6685600640 1 -- 172.21.15.96:0/457437722 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6678009690 con 0x7f666c0039b0 2024-05-28T06:08:48.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.175+0000 7f668d800640 1 -- 172.21.15.96:0/457437722 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f666c0039b0 msgr2=0x7f666c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:48.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.175+0000 7f668d800640 1 --2- 172.21.15.96:0/457437722 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f666c0039b0 0x7f666c003dd0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f6678009ac0 tx=0x7f666c005830 comp rx=0 tx=0).stop 2024-05-28T06:08:48.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.175+0000 7f6687400640 1 -- 172.21.15.96:0/457437722 reap_dead start 2024-05-28T06:08:48.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.175+0000 7f668d800640 1 -- 172.21.15.96:0/457437722 shutdown_connections 2024-05-28T06:08:48.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.175+0000 7f668d800640 1 -- 172.21.15.96:0/457437722 >> 172.21.15.96:0/457437722 conn(0x7f66880d2430 msgr2=0x7f66880d2840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:48.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.175+0000 7f668d800640 1 -- 172.21.15.96:0/457437722 shutdown_connections 2024-05-28T06:08:48.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.175+0000 7f668d800640 1 -- 172.21.15.96:0/457437722 wait complete. 2024-05-28T06:08:48.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.177+0000 7f668d800640 1 Processor -- start 2024-05-28T06:08:48.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.177+0000 7f668d800640 1 -- start start 2024-05-28T06:08:48.180 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.178+0000 7f668d800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f666c0039b0 0x7f668809cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:48.180 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.178+0000 7f668d800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f668809efe0 0x7f6688071500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:48.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.178+0000 7f668d800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66881f7530 0x7f66881f7980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:48.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.178+0000 7f668d800640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f668806ce70 con 0x7f666c0039b0 2024-05-28T06:08:48.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.178+0000 7f668d800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f668809ba60 con 0x7f66881f7530 2024-05-28T06:08:48.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.178+0000 7f668d800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f6688160280 con 0x7f668809efe0 2024-05-28T06:08:48.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.178+0000 7f6686a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f666c0039b0 0x7f668809cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:48.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.178+0000 7f6686a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f666c0039b0 0x7f668809cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58280/0 (socket says 172.21.15.96:58280) 2024-05-28T06:08:48.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.178+0000 7f6686a00640 1 -- 172.21.15.96:0/3350862015 learned_addr learned my addr 172.21.15.96:0/3350862015 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:08:48.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.179+0000 7f6686a00640 1 -- 172.21.15.96:0/3350862015 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f668809efe0 msgr2=0x7f6688071500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:48.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.179+0000 7f6686a00640 1 --2- 172.21.15.96:0/3350862015 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f668809efe0 0x7f6688071500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:48.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.179+0000 7f6686a00640 1 -- 172.21.15.96:0/3350862015 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66881f7530 msgr2=0x7f66881f7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:48.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.179+0000 7f6686a00640 1 --2- 172.21.15.96:0/3350862015 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66881f7530 0x7f66881f7980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:48.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.179+0000 7f6686a00640 1 -- 172.21.15.96:0/3350862015 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f668809d420 con 0x7f666c0039b0 2024-05-28T06:08:48.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.179+0000 7f6686a00640 1 --2- 172.21.15.96:0/3350862015 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f666c0039b0 0x7f668809cdd0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f6678009ac0 tx=0x7f6678004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:48.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.179+0000 7f667ea00640 1 -- 172.21.15.96:0/3350862015 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6678040020 con 0x7f666c0039b0 2024-05-28T06:08:48.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.180+0000 7f667ea00640 1 -- 172.21.15.96:0/3350862015 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66780038a0 con 0x7f666c0039b0 2024-05-28T06:08:48.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.180+0000 7f667ea00640 1 -- 172.21.15.96:0/3350862015 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6678036040 con 0x7f666c0039b0 2024-05-28T06:08:48.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.180+0000 7f668d800640 1 -- 172.21.15.96:0/3350862015 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f668809d700 con 0x7f666c0039b0 2024-05-28T06:08:48.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.180+0000 7f668d800640 1 -- 172.21.15.96:0/3350862015 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f668809dcc0 con 0x7f666c0039b0 2024-05-28T06:08:48.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.181+0000 7f668d800640 1 -- 172.21.15.96:0/3350862015 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66880689e0 con 0x7f666c0039b0 2024-05-28T06:08:48.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.182+0000 7f667ea00640 1 -- 172.21.15.96:0/3350862015 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 9) ==== 91295+0+0 (secure 0 0 0) 0x7f6678043020 con 0x7f666c0039b0 2024-05-28T06:08:48.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.183+0000 7f667ea00640 1 --2- 172.21.15.96:0/3350862015 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f666806cbc0 0x7f666806f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:48.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.183+0000 7f667ea00640 1 -- 172.21.15.96:0/3350862015 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 4971+0+0 (secure 0 0 0) 0x7f66780abd60 con 0x7f666c0039b0 2024-05-28T06:08:48.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.183+0000 7f6686000640 1 --2- 172.21.15.96:0/3350862015 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f666806cbc0 0x7f666806f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:48.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.185+0000 7f6686000640 1 --2- 172.21.15.96:0/3350862015 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f666806cbc0 0x7f666806f080 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f66740017e0 tx=0x7f6674009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:48.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.187+0000 7f667ea00640 1 -- 172.21.15.96:0/3350862015 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f667804a070 con 0x7f666c0039b0 2024-05-28T06:08:48.390 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.389+0000 7f668d800640 1 -- 172.21.15.96:0/3350862015 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f66880703a0 con 0x7f666c0039b0 2024-05-28T06:08:48.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.396+0000 7f667ea00640 1 -- 172.21.15.96:0/3350862015 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v12)=0 v12) ==== 122+0+0 (secure 0 0 0) 0x7f667807a120 con 0x7f666c0039b0 2024-05-28T06:08:48.400 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.399+0000 7f668d800640 1 -- 172.21.15.96:0/3350862015 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f666806cbc0 msgr2=0x7f666806f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:48.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.399+0000 7f668d800640 1 --2- 172.21.15.96:0/3350862015 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f666806cbc0 0x7f666806f080 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f66740017e0 tx=0x7f6674009040 comp rx=0 tx=0).stop 2024-05-28T06:08:48.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.399+0000 7f668d800640 1 -- 172.21.15.96:0/3350862015 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f666c0039b0 msgr2=0x7f668809cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:48.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.399+0000 7f668d800640 1 --2- 172.21.15.96:0/3350862015 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f666c0039b0 0x7f668809cdd0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f6678009ac0 tx=0x7f6678004730 comp rx=0 tx=0).stop 2024-05-28T06:08:48.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.399+0000 7f668d800640 1 -- 172.21.15.96:0/3350862015 shutdown_connections 2024-05-28T06:08:48.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.399+0000 7f668d800640 1 --2- 172.21.15.96:0/3350862015 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f666806cbc0 0x7f666806f080 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:48.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.399+0000 7f668d800640 1 --2- 172.21.15.96:0/3350862015 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66881f7530 0x7f66881f7980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:48.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.399+0000 7f668d800640 1 --2- 172.21.15.96:0/3350862015 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f668809efe0 0x7f6688071500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:48.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.399+0000 7f668d800640 1 --2- 172.21.15.96:0/3350862015 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f666c0039b0 0x7f668809cdd0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:48.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.399+0000 7f668d800640 1 -- 172.21.15.96:0/3350862015 >> 172.21.15.96:0/3350862015 conn(0x7f66880d2430 msgr2=0x7f66880c88a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:48.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.399+0000 7f668d800640 1 -- 172.21.15.96:0/3350862015 shutdown_connections 2024-05-28T06:08:48.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:48.399+0000 7f668d800640 1 -- 172.21.15.96:0/3350862015 wait complete. 2024-05-28T06:08:48.420 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-05-28T06:08:48.420 DEBUG:teuthology.orchestra.run.smithi096:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-05-28T06:08:48.525 DEBUG:teuthology.orchestra.run.smithi123:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-05-28T06:08:48.637 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-05-28T06:08:48.638 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:08:48.638 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:08:48.692 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052731,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248996k,nr_inodes=812249,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-28T06:08:48.692 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-05-28T06:08:48.692 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:08:48.744 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052731,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248996k,nr_inodes=812249,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-05-28T06:08:48.744 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:08:48.744 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:08:48.802 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052731,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248996k,nr_inodes=812249,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-28T06:08:48.802 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-05-28T06:08:48.802 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:08:48.860 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052731,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248996k,nr_inodes=812249,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-05-28T06:08:48.860 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-05-28T06:08:48.860 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-28T06:08:49.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.071+0000 7f03ff400640 1 Processor -- start 2024-05-28T06:08:49.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.071+0000 7f03ff400640 1 -- start start 2024-05-28T06:08:49.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.072+0000 7f03ff400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f03f80d0930 0x7f03f80ad840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:49.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.072+0000 7f03ff400640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03f806ed10 con 0x7f03f80d04a0 2024-05-28T06:08:49.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.072+0000 7f03ff400640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03f815cb00 con 0x7f03f806f860 2024-05-28T06:08:49.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.072+0000 7f03ff400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f03f80b3cc0 con 0x7f03f80d0930 2024-05-28T06:08:49.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.073+0000 7f03f7e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f03f80d04a0 0x7f03f80ad130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:56394/0 (socket says 172.21.15.96:56394) 2024-05-28T06:08:49.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.073+0000 7f03f7e00640 1 -- 172.21.15.96:0/1323806597 learned_addr learned my addr 172.21.15.96:0/1323806597 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:08:49.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.073+0000 7f03fd400640 1 --2- 172.21.15.96:0/1323806597 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f03f80d0930 0x7f03f80ad840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:49.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.073+0000 7f03f7400640 1 -- 172.21.15.96:0/1323806597 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1919781841 0 0) 0x7f03e4001110 con 0x7f03f80d04a0 2024-05-28T06:08:49.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.073+0000 7f03f7400640 1 -- 172.21.15.96:0/1323806597 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f03e0003520 con 0x7f03f80d04a0 2024-05-28T06:08:49.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.074+0000 7f03f7400640 1 -- 172.21.15.96:0/1323806597 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1410971711 0 0) 0x7f03e0003520 con 0x7f03f80d04a0 2024-05-28T06:08:49.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.074+0000 7f03f7400640 1 -- 172.21.15.96:0/1323806597 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f03f80d0930 msgr2=0x7f03f80ad840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:49.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.074+0000 7f03f7400640 1 --2- 172.21.15.96:0/1323806597 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f03f80d0930 0x7f03f80ad840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:49.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.074+0000 7f03f7400640 1 -- 172.21.15.96:0/1323806597 >> v1:172.21.15.123:6789/0 conn(0x7f03f806f860 legacy=0x7f03f80b3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:49.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.074+0000 7f03f7400640 1 -- 172.21.15.96:0/1323806597 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03f80b3fa0 con 0x7f03f80d04a0 2024-05-28T06:08:49.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.074+0000 7f03f7400640 1 -- 172.21.15.96:0/1323806597 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f03e4001840 con 0x7f03f80d04a0 2024-05-28T06:08:49.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.074+0000 7f03f7400640 1 -- 172.21.15.96:0/1323806597 >> v1:172.21.15.96:6789/0 conn(0x7f03f80d04a0 legacy=0x7f03f80ad130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:49.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.074+0000 7f03f7400640 1 --2- 172.21.15.96:0/1323806597 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f03e00039e0 0x7f03e0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:49.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.074+0000 7f03f7400640 1 --2- 172.21.15.96:0/1323806597 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f03f80d0930 0x7f03e0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:49.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.074+0000 7f03f7400640 1 --2- 172.21.15.96:0/1323806597 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f03e0004d40 0x7f03e00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:49.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.074+0000 7f03f7400640 1 -- 172.21.15.96:0/1323806597 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f03f80b3cc0 con 0x7f03e00039e0 2024-05-28T06:08:49.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.074+0000 7f03f7400640 1 -- 172.21.15.96:0/1323806597 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f03e000ba70 con 0x7f03e0004d40 2024-05-28T06:08:49.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.074+0000 7f03f7400640 1 -- 172.21.15.96:0/1323806597 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f03e000bc70 con 0x7f03f80d0930 2024-05-28T06:08:49.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.074+0000 7f03fca00640 1 --2- 172.21.15.96:0/1323806597 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f03e00039e0 0x7f03e0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:49.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.075+0000 7f03fca00640 1 -- 172.21.15.96:0/1323806597 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f03f80d0930 msgr2=0x7f03e0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:49.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.075+0000 7f03fca00640 1 --2- 172.21.15.96:0/1323806597 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f03f80d0930 0x7f03e0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:49.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.075+0000 7f03fca00640 1 -- 172.21.15.96:0/1323806597 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f03e0004d40 msgr2=0x7f03e00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:49.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.075+0000 7f03fca00640 1 --2- 172.21.15.96:0/1323806597 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f03e0004d40 0x7f03e00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:49.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.075+0000 7f03fca00640 1 -- 172.21.15.96:0/1323806597 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03e0003520 con 0x7f03e00039e0 2024-05-28T06:08:49.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.076+0000 7f03fca00640 1 --2- 172.21.15.96:0/1323806597 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f03e00039e0 0x7f03e0003e00 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f03ec001720 tx=0x7f03e0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12da6aad37b9f3a5 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:49.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.076+0000 7f03f7400640 1 -- 172.21.15.96:0/1323806597 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03ec001370 con 0x7f03e00039e0 2024-05-28T06:08:49.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.076+0000 7f03f7400640 1 -- 172.21.15.96:0/1323806597 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03ec0052d0 con 0x7f03e00039e0 2024-05-28T06:08:49.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.076+0000 7f03f7400640 1 -- 172.21.15.96:0/1323806597 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03ec02abc0 con 0x7f03e00039e0 2024-05-28T06:08:49.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.076+0000 7f03ff400640 1 -- 172.21.15.96:0/1323806597 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f03e00039e0 msgr2=0x7f03e0003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:49.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.076+0000 7f03ff400640 1 --2- 172.21.15.96:0/1323806597 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f03e00039e0 0x7f03e0003e00 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f03ec001720 tx=0x7f03e0005830 comp rx=0 tx=0).stop 2024-05-28T06:08:49.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.076+0000 7f03fd400640 1 -- 172.21.15.96:0/1323806597 reap_dead start 2024-05-28T06:08:49.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.076+0000 7f03ff400640 1 -- 172.21.15.96:0/1323806597 shutdown_connections 2024-05-28T06:08:49.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.076+0000 7f03ff400640 1 -- 172.21.15.96:0/1323806597 >> 172.21.15.96:0/1323806597 conn(0x7f03f806bae0 msgr2=0x7f03f806bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:49.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.077+0000 7f03ff400640 1 -- 172.21.15.96:0/1323806597 shutdown_connections 2024-05-28T06:08:49.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.077+0000 7f03ff400640 1 -- 172.21.15.96:0/1323806597 wait complete. 2024-05-28T06:08:49.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.078+0000 7f03ff400640 1 Processor -- start 2024-05-28T06:08:49.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.078+0000 7f03ff400640 1 -- start start 2024-05-28T06:08:49.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.079+0000 7f03ff400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f03e00039e0 0x7f03f815c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:49.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.079+0000 7f03ff400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f03f815dd60 0x7f03f815f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:49.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.079+0000 7f03ff400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f03f81f3fa0 0x7f03f80cea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:49.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.079+0000 7f03ff400640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f03f80b4840 con 0x7f03f815dd60 2024-05-28T06:08:49.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.079+0000 7f03ff400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f03f80b4560 con 0x7f03e00039e0 2024-05-28T06:08:49.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.079+0000 7f03ff400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f03f80d0d80 con 0x7f03f81f3fa0 2024-05-28T06:08:49.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.080+0000 7f03f7e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f03f815dd60 0x7f03f815f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:49.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.080+0000 7f03f7e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f03f815dd60 0x7f03f815f500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51154/0 (socket says 172.21.15.96:51154) 2024-05-28T06:08:49.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.080+0000 7f03f7e00640 1 -- 172.21.15.96:0/1967128686 learned_addr learned my addr 172.21.15.96:0/1967128686 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:08:49.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.080+0000 7f03fca00640 1 --2- 172.21.15.96:0/1967128686 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f03e00039e0 0x7f03f815c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:49.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.080+0000 7f03fd400640 1 --2- 172.21.15.96:0/1967128686 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f03f81f3fa0 0x7f03f80cea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:49.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.080+0000 7f03f7e00640 1 -- 172.21.15.96:0/1967128686 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f03f81f3fa0 msgr2=0x7f03f80cea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:49.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.080+0000 7f03f7e00640 1 --2- 172.21.15.96:0/1967128686 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f03f81f3fa0 0x7f03f80cea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:49.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.080+0000 7f03f7e00640 1 -- 172.21.15.96:0/1967128686 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f03e00039e0 msgr2=0x7f03f815c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:49.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.080+0000 7f03f7e00640 1 --2- 172.21.15.96:0/1967128686 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f03e00039e0 0x7f03f815c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:49.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.080+0000 7f03f7e00640 1 -- 172.21.15.96:0/1967128686 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03f815e1c0 con 0x7f03f815dd60 2024-05-28T06:08:49.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.080+0000 7f03f7e00640 1 --2- 172.21.15.96:0/1967128686 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f03f815dd60 0x7f03f815f500 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f03e400c730 tx=0x7f03e400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:49.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.080+0000 7f03f4c00640 1 -- 172.21.15.96:0/1967128686 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03e4018070 con 0x7f03f815dd60 2024-05-28T06:08:49.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.081+0000 7f03f4c00640 1 -- 172.21.15.96:0/1967128686 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03e40072d0 con 0x7f03f815dd60 2024-05-28T06:08:49.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.081+0000 7f03f4c00640 1 -- 172.21.15.96:0/1967128686 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03e40047e0 con 0x7f03f815dd60 2024-05-28T06:08:49.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.081+0000 7f03ff400640 1 -- 172.21.15.96:0/1967128686 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03f815e460 con 0x7f03f815dd60 2024-05-28T06:08:49.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.081+0000 7f03ff400640 1 -- 172.21.15.96:0/1967128686 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03f815ea20 con 0x7f03f815dd60 2024-05-28T06:08:49.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.082+0000 7f03ff400640 1 -- 172.21.15.96:0/1967128686 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03f80689e0 con 0x7f03f815dd60 2024-05-28T06:08:49.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.083+0000 7f03f4c00640 1 -- 172.21.15.96:0/1967128686 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 9) ==== 91295+0+0 (secure 0 0 0) 0x7f03e4004980 con 0x7f03f815dd60 2024-05-28T06:08:49.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.084+0000 7f03f4c00640 1 --2- 172.21.15.96:0/1967128686 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f03d806cbf0 0x7f03d806f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:49.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.084+0000 7f03f4c00640 1 -- 172.21.15.96:0/1967128686 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 4971+0+0 (secure 0 0 0) 0x7f03e408b030 con 0x7f03f815dd60 2024-05-28T06:08:49.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.085+0000 7f03fca00640 1 --2- 172.21.15.96:0/1967128686 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f03d806cbf0 0x7f03d806f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:49.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.086+0000 7f03fca00640 1 --2- 172.21.15.96:0/1967128686 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f03d806cbf0 0x7f03d806f0b0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f03ec032000 tx=0x7f03ec02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:49.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.089+0000 7f03f4c00640 1 -- 172.21.15.96:0/1967128686 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f03f806ed10 con 0x7f03f815dd60 2024-05-28T06:08:49.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.328+0000 7f03ff400640 1 -- 172.21.15.96:0/1967128686 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f03f80cd320 con 0x7f03f815dd60 2024-05-28T06:08:49.331 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.329+0000 7f03f4c00640 1 -- 172.21.15.96:0/1967128686 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+3866 (secure 0 0 0) 0x7f03e4059440 con 0x7f03f815dd60 2024-05-28T06:08:49.331 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:08:49.332 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":18,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4580,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6836/918905879","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":918905879},{"type":"v1","addr":"172.21.15.123:6836","nonce":918905879}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4775,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/902611696","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6835","nonce":902611696},{"type":"v1","addr":"172.21.15.123:6837","nonce":902611696}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":18},{"gid":5223,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6834/697317719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6833","nonce":697317719},{"type":"v1","addr":"172.21.15.96:6834","nonce":697317719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":5226,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/1327552826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":1327552826},{"type":"v1","addr":"172.21.15.96:6835","nonce":1327552826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[]} 2024-05-28T06:08:49.332 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 18 2024-05-28T06:08:49.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.332+0000 7f03ff400640 1 -- 172.21.15.96:0/1967128686 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f03d806cbf0 msgr2=0x7f03d806f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:49.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.332+0000 7f03ff400640 1 --2- 172.21.15.96:0/1967128686 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f03d806cbf0 0x7f03d806f0b0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f03ec032000 tx=0x7f03ec02e000 comp rx=0 tx=0).stop 2024-05-28T06:08:49.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.332+0000 7f03ff400640 1 -- 172.21.15.96:0/1967128686 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f03f815dd60 msgr2=0x7f03f815f500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:49.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.332+0000 7f03ff400640 1 --2- 172.21.15.96:0/1967128686 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f03f815dd60 0x7f03f815f500 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f03e400c730 tx=0x7f03e400cbf0 comp rx=0 tx=0).stop 2024-05-28T06:08:49.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.332+0000 7f03ff400640 1 -- 172.21.15.96:0/1967128686 shutdown_connections 2024-05-28T06:08:49.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.332+0000 7f03ff400640 1 --2- 172.21.15.96:0/1967128686 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f03d806cbf0 0x7f03d806f0b0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:49.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.332+0000 7f03ff400640 1 --2- 172.21.15.96:0/1967128686 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f03f81f3fa0 0x7f03f80cea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:49.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.332+0000 7f03ff400640 1 --2- 172.21.15.96:0/1967128686 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f03f815dd60 0x7f03f815f500 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:49.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.332+0000 7f03ff400640 1 --2- 172.21.15.96:0/1967128686 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f03e00039e0 0x7f03f815c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:49.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.332+0000 7f03ff400640 1 -- 172.21.15.96:0/1967128686 >> 172.21.15.96:0/1967128686 conn(0x7f03f806bae0 msgr2=0x7f03f8071b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:49.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.332+0000 7f03ff400640 1 -- 172.21.15.96:0/1967128686 shutdown_connections 2024-05-28T06:08:49.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.332+0000 7f03ff400640 1 -- 172.21.15.96:0/1967128686 wait complete. 2024-05-28T06:08:49.354 INFO:tasks.ceph.mds.a:Restarting daemon 2024-05-28T06:08:49.354 INFO:tasks.ceph.mds.a:Stopping old one... 2024-05-28T06:08:49.354 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-05-28T06:08:49.354 INFO:teuthology.orchestra.run:waiting for 300 2024-05-28T06:08:49.354 INFO:tasks.ceph.mds.c:Restarting daemon 2024-05-28T06:08:49.355 INFO:tasks.ceph.mds.c:Stopping old one... 2024-05-28T06:08:49.355 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-05-28T06:08:49.355 INFO:teuthology.orchestra.run:waiting for 300 2024-05-28T06:08:49.355 INFO:tasks.ceph.mds.b:Restarting daemon 2024-05-28T06:08:49.355 INFO:tasks.ceph.mds.b:Stopping old one... 2024-05-28T06:08:49.356 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-05-28T06:08:49.356 INFO:teuthology.orchestra.run:waiting for 300 2024-05-28T06:08:49.356 INFO:tasks.ceph.mds.d:Restarting daemon 2024-05-28T06:08:49.356 INFO:tasks.ceph.mds.d:Stopping old one... 2024-05-28T06:08:49.356 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-05-28T06:08:49.357 INFO:teuthology.orchestra.run:waiting for 300 2024-05-28T06:08:49.371 INFO:tasks.ceph.mds.a:Stopped 2024-05-28T06:08:49.372 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-05-28T06:08:49.372 INFO:tasks.ceph.mds.c:Stopped 2024-05-28T06:08:49.373 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-05-28T06:08:49.383 INFO:tasks.ceph.mds.d:Stopped 2024-05-28T06:08:49.383 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-05-28T06:08:49.383 INFO:tasks.ceph.mds.b:Stopped 2024-05-28T06:08:49.384 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-05-28T06:08:49.420 INFO:tasks.ceph.mds.a:Started 2024-05-28T06:08:49.420 DEBUG:teuthology.parallel:result is None 2024-05-28T06:08:49.425 INFO:tasks.ceph.mds.c:Started 2024-05-28T06:08:49.426 DEBUG:teuthology.parallel:result is None 2024-05-28T06:08:49.430 INFO:tasks.ceph.mds.d:Started 2024-05-28T06:08:49.431 DEBUG:teuthology.parallel:result is None 2024-05-28T06:08:49.436 INFO:tasks.ceph.mds.b:Started 2024-05-28T06:08:49.436 DEBUG:teuthology.parallel:result is None 2024-05-28T06:08:49.437 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-05-28T06:08:49.569 INFO:tasks.ceph.mds.c.smithi096.stdout:starting mds.c at 2024-05-28T06:08:49.570 INFO:tasks.ceph.mds.a.smithi096.stdout:starting mds.a at 2024-05-28T06:08:49.586 INFO:tasks.ceph.mds.d.smithi123.stdout:starting mds.d at 2024-05-28T06:08:49.590 INFO:tasks.ceph.mds.b.smithi123.stdout:starting mds.b at 2024-05-28T06:08:49.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.614+0000 7f3663000640 1 Processor -- start 2024-05-28T06:08:49.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.614+0000 7f3663000640 1 -- start start 2024-05-28T06:08:49.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.615+0000 7f3663000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f365c160380 0x7f365c0b5e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:49.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.615+0000 7f3663000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f365c0b5430 0x7f365c0b63a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:49.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.615+0000 7f3663000640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f365c0d3300 con 0x7f365c15fae0 2024-05-28T06:08:49.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.615+0000 7f3663000640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f365c0ced60 con 0x7f365c0b5430 2024-05-28T06:08:49.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.615+0000 7f3663000640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f365c0d35e0 con 0x7f365c160380 2024-05-28T06:08:49.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.615+0000 7f3662600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f365c0b5430 0x7f365c0b63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:49.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.615+0000 7f3662600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f365c0b5430 0x7f365c0b63a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51194/0 (socket says 172.21.15.96:51194) 2024-05-28T06:08:49.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.615+0000 7f3662600640 1 -- 172.21.15.96:0/389396219 learned_addr learned my addr 172.21.15.96:0/389396219 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:08:49.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.615+0000 7f3661200640 1 --2- 172.21.15.96:0/389396219 >> v2:172.21.15.123:3300/0 conn(0x7f365c160380 0x7f365c0b5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:49.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.616+0000 7f3662600640 1 -- 172.21.15.96:0/389396219 >> v2:172.21.15.123:3300/0 conn(0x7f365c160380 msgr2=0x7f365c0b5e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:49.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.616+0000 7f3662600640 1 --2- 172.21.15.96:0/389396219 >> v2:172.21.15.123:3300/0 conn(0x7f365c160380 0x7f365c0b5e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:49.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.616+0000 7f3662600640 1 -- 172.21.15.96:0/389396219 >> v1:172.21.15.123:6789/0 conn(0x7f365c15fae0 legacy=0x7f365c0d4e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:49.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.616+0000 7f3662600640 1 -- 172.21.15.96:0/389396219 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f365c0d38c0 con 0x7f365c0b5430 2024-05-28T06:08:49.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.617+0000 7f3662600640 1 --2- 172.21.15.96:0/389396219 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f365c0b5430 0x7f365c0b63a0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f365c0ceee0 tx=0x7f365402a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad966be626b7b59 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:49.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.617+0000 7f365be00640 1 -- 172.21.15.96:0/389396219 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3654036070 con 0x7f365c0b5430 2024-05-28T06:08:49.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.617+0000 7f365be00640 1 -- 172.21.15.96:0/389396219 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3654031600 con 0x7f365c0b5430 2024-05-28T06:08:49.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.617+0000 7f365be00640 1 -- 172.21.15.96:0/389396219 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36540319d0 con 0x7f365c0b5430 2024-05-28T06:08:49.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.618+0000 7f3663000640 1 -- 172.21.15.96:0/389396219 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f365c0b5430 msgr2=0x7f365c0b63a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:49.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.618+0000 7f3663000640 1 --2- 172.21.15.96:0/389396219 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f365c0b5430 0x7f365c0b63a0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f365c0ceee0 tx=0x7f365402a730 comp rx=0 tx=0).stop 2024-05-28T06:08:49.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.618+0000 7f3663000640 1 -- 172.21.15.96:0/389396219 shutdown_connections 2024-05-28T06:08:49.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.618+0000 7f3663000640 1 --2- 172.21.15.96:0/389396219 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f365c0b5430 0x7f365c0b63a0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:49.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.618+0000 7f3663000640 1 --2- 172.21.15.96:0/389396219 >> v2:172.21.15.123:3300/0 conn(0x7f365c160380 0x7f365c0b5e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:49.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.618+0000 7f3663000640 1 -- 172.21.15.96:0/389396219 >> 172.21.15.96:0/389396219 conn(0x7f365c08bfc0 msgr2=0x7f365c0d4500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:49.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.618+0000 7f3663000640 1 -- 172.21.15.96:0/389396219 shutdown_connections 2024-05-28T06:08:49.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.618+0000 7f3663000640 1 -- 172.21.15.96:0/389396219 wait complete. 2024-05-28T06:08:49.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.620+0000 7f3663000640 1 Processor -- start 2024-05-28T06:08:49.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.620+0000 7f3663000640 1 -- start start 2024-05-28T06:08:49.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.621+0000 7f3663000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f365c0b5430 0x7f365c0b7060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:49.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.621+0000 7f3663000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f365c15fae0 0x7f365c0d1110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:49.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.621+0000 7f3663000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f365c160380 0x7f365c0c6de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:49.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.621+0000 7f3663000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f365c0cf370 con 0x7f365c15fae0 2024-05-28T06:08:49.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.621+0000 7f3663000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f365c0cf1f0 con 0x7f365c160380 2024-05-28T06:08:49.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.621+0000 7f3663000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f365c0cf4f0 con 0x7f365c0b5430 2024-05-28T06:08:49.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.621+0000 7f3661200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f365c15fae0 0x7f365c0d1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:49.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.621+0000 7f3661200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f365c15fae0 0x7f365c0d1110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51202/0 (socket says 172.21.15.96:51202) 2024-05-28T06:08:49.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.621+0000 7f3661200640 1 -- 172.21.15.96:0/3690611329 learned_addr learned my addr 172.21.15.96:0/3690611329 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:08:49.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.621+0000 7f3661c00640 1 --2- 172.21.15.96:0/3690611329 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f365c0b5430 0x7f365c0b7060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:49.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.621+0000 7f3661200640 1 -- 172.21.15.96:0/3690611329 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f365c0b5430 msgr2=0x7f365c0b7060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:49.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.621+0000 7f3662600640 1 --2- 172.21.15.96:0/3690611329 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f365c160380 0x7f365c0c6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:49.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.621+0000 7f3661200640 1 --2- 172.21.15.96:0/3690611329 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f365c0b5430 0x7f365c0b7060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:49.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.621+0000 7f3661200640 1 -- 172.21.15.96:0/3690611329 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f365c160380 msgr2=0x7f365c0c6de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:49.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.621+0000 7f3661200640 1 --2- 172.21.15.96:0/3690611329 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f365c160380 0x7f365c0c6de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:49.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.621+0000 7f3661200640 1 -- 172.21.15.96:0/3690611329 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f365c15ad60 con 0x7f365c15fae0 2024-05-28T06:08:49.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.621+0000 7f3661c00640 1 --2- 172.21.15.96:0/3690611329 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f365c0b5430 0x7f365c0b7060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:08:49.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.621+0000 7f3662600640 1 --2- 172.21.15.96:0/3690611329 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f365c160380 0x7f365c0c6de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:08:49.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.622+0000 7f3661200640 1 --2- 172.21.15.96:0/3690611329 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f365c15fae0 0x7f365c0d1110 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f364800e870 tx=0x7f364800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:49.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.622+0000 7f3659600640 1 -- 172.21.15.96:0/3690611329 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3648017070 con 0x7f365c15fae0 2024-05-28T06:08:49.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.622+0000 7f3659600640 1 -- 172.21.15.96:0/3690611329 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f364800c940 con 0x7f365c15fae0 2024-05-28T06:08:49.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.622+0000 7f3659600640 1 -- 172.21.15.96:0/3690611329 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f364800cd30 con 0x7f365c15fae0 2024-05-28T06:08:49.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.622+0000 7f3663000640 1 -- 172.21.15.96:0/3690611329 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f365c14a070 con 0x7f365c15fae0 2024-05-28T06:08:49.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.622+0000 7f3663000640 1 -- 172.21.15.96:0/3690611329 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f365c0b8dd0 con 0x7f365c15fae0 2024-05-28T06:08:49.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.623+0000 7f3663000640 1 -- 172.21.15.96:0/3690611329 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f365c0689e0 con 0x7f365c15fae0 2024-05-28T06:08:49.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.624+0000 7f3659600640 1 -- 172.21.15.96:0/3690611329 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 9) ==== 91295+0+0 (secure 0 0 0) 0x7f364801a070 con 0x7f365c15fae0 2024-05-28T06:08:49.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.625+0000 7f3659600640 1 --2- 172.21.15.96:0/3690611329 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f363c06cc10 0x7f363c06f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:49.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.625+0000 7f3659600640 1 -- 172.21.15.96:0/3690611329 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 4971+0+0 (secure 0 0 0) 0x7f364808bfb0 con 0x7f365c15fae0 2024-05-28T06:08:49.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.625+0000 7f3661c00640 1 --2- 172.21.15.96:0/3690611329 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f363c06cc10 0x7f363c06f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:49.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.626+0000 7f3661c00640 1 --2- 172.21.15.96:0/3690611329 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f363c06cc10 0x7f363c06f0d0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f3650001680 tx=0x7f3650009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:49.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.629+0000 7f3659600640 1 -- 172.21.15.96:0/3690611329 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f364805a470 con 0x7f365c15fae0 2024-05-28T06:08:49.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.839+0000 7f3663000640 1 -- 172.21.15.96:0/3690611329 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f365c0b6c90 con 0x7f365c15fae0 2024-05-28T06:08:49.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.968+0000 7f3659600640 1 -- 172.21.15.96:0/3690611329 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v39) ==== 97+0+0 (secure 0 0 0) 0x7f3648059d70 con 0x7f365c15fae0 2024-05-28T06:08:49.970 INFO:teuthology.orchestra.run.smithi096.stderr: removed all blocklist entries 2024-05-28T06:08:49.973 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.971+0000 7f3663000640 1 -- 172.21.15.96:0/3690611329 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f363c06cc10 msgr2=0x7f363c06f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:49.973 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.971+0000 7f3663000640 1 --2- 172.21.15.96:0/3690611329 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f363c06cc10 0x7f363c06f0d0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f3650001680 tx=0x7f3650009040 comp rx=0 tx=0).stop 2024-05-28T06:08:49.973 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.971+0000 7f3663000640 1 -- 172.21.15.96:0/3690611329 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f365c15fae0 msgr2=0x7f365c0d1110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:49.973 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.971+0000 7f3663000640 1 --2- 172.21.15.96:0/3690611329 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f365c15fae0 0x7f365c0d1110 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f364800e870 tx=0x7f364800ed30 comp rx=0 tx=0).stop 2024-05-28T06:08:49.974 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.972+0000 7f3663000640 1 -- 172.21.15.96:0/3690611329 shutdown_connections 2024-05-28T06:08:49.974 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.972+0000 7f3663000640 1 --2- 172.21.15.96:0/3690611329 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f363c06cc10 0x7f363c06f0d0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:49.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.972+0000 7f3663000640 1 --2- 172.21.15.96:0/3690611329 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f365c160380 0x7f365c0c6de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:49.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.972+0000 7f3663000640 1 --2- 172.21.15.96:0/3690611329 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f365c15fae0 0x7f365c0d1110 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:49.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.972+0000 7f3663000640 1 --2- 172.21.15.96:0/3690611329 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f365c0b5430 0x7f365c0b7060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:49.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.972+0000 7f3663000640 1 -- 172.21.15.96:0/3690611329 >> 172.21.15.96:0/3690611329 conn(0x7f365c08bfc0 msgr2=0x7f365c1616f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:49.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.972+0000 7f3663000640 1 -- 172.21.15.96:0/3690611329 shutdown_connections 2024-05-28T06:08:49.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:49.972+0000 7f3663000640 1 -- 172.21.15.96:0/3690611329 wait complete. 2024-05-28T06:08:49.999 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-05-28T06:08:50.202 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.201+0000 7f4e9da00640 1 Processor -- start 2024-05-28T06:08:50.202 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.201+0000 7f4e9da00640 1 -- start start 2024-05-28T06:08:50.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.203+0000 7f4e9da00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4e98156b30 0x7f4e980af6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:50.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.203+0000 7f4e9da00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f4e9806ee40 0x7f4e980ad230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:50.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.203+0000 7f4e9da00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e98156f10 con 0x7f4e9806f2c0 2024-05-28T06:08:50.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.203+0000 7f4e9da00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f4e9806dda0 con 0x7f4e9806ee40 2024-05-28T06:08:50.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.203+0000 7f4e9da00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f4e9806e080 con 0x7f4e98156b30 2024-05-28T06:08:50.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.203+0000 7f4e97e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4e98156b30 0x7f4e980af6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:50.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.204+0000 7f4e97e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4e98156b30 0x7f4e980af6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.96:47338/0 (socket says 172.21.15.96:47338) 2024-05-28T06:08:50.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.204+0000 7f4e97e00640 1 -- 172.21.15.96:0/2398749704 learned_addr learned my addr 172.21.15.96:0/2398749704 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:08:50.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.204+0000 7f4e97400640 1 --2- 172.21.15.96:0/2398749704 >> v2:172.21.15.123:3300/0 conn(0x7f4e9806ee40 0x7f4e980ad230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:50.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.204+0000 7f4e97e00640 1 -- 172.21.15.96:0/2398749704 >> v2:172.21.15.123:3300/0 conn(0x7f4e9806ee40 msgr2=0x7f4e980ad230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:50.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.204+0000 7f4e97e00640 1 --2- 172.21.15.96:0/2398749704 >> v2:172.21.15.123:3300/0 conn(0x7f4e9806ee40 0x7f4e980ad230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:50.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.204+0000 7f4e97e00640 1 -- 172.21.15.96:0/2398749704 >> v1:172.21.15.123:6789/0 conn(0x7f4e9806f2c0 legacy=0x7f4e980b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:50.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.204+0000 7f4e97e00640 1 -- 172.21.15.96:0/2398749704 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e9806e360 con 0x7f4e98156b30 2024-05-28T06:08:50.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.205+0000 7f4e97e00640 1 --2- 172.21.15.96:0/2398749704 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4e98156b30 0x7f4e980af6e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4e88009870 tx=0x7f4e8802a1c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f0d6ed47998ff658 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:50.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.206+0000 7f4e96a00640 1 -- 172.21.15.96:0/2398749704 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e88037070 con 0x7f4e98156b30 2024-05-28T06:08:50.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.206+0000 7f4e96a00640 1 -- 172.21.15.96:0/2398749704 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e88032440 con 0x7f4e98156b30 2024-05-28T06:08:50.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.206+0000 7f4e96a00640 1 -- 172.21.15.96:0/2398749704 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e88032810 con 0x7f4e98156b30 2024-05-28T06:08:50.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.207+0000 7f4e9da00640 1 -- 172.21.15.96:0/2398749704 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4e98156b30 msgr2=0x7f4e980af6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:50.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.207+0000 7f4e9da00640 1 --2- 172.21.15.96:0/2398749704 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4e98156b30 0x7f4e980af6e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4e88009870 tx=0x7f4e8802a1c0 comp rx=0 tx=0).stop 2024-05-28T06:08:50.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.207+0000 7f4e9da00640 1 -- 172.21.15.96:0/2398749704 shutdown_connections 2024-05-28T06:08:50.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.207+0000 7f4e9da00640 1 --2- 172.21.15.96:0/2398749704 >> v2:172.21.15.123:3300/0 conn(0x7f4e9806ee40 0x7f4e980ad230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:50.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.207+0000 7f4e9da00640 1 --2- 172.21.15.96:0/2398749704 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4e98156b30 0x7f4e980af6e0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:50.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.207+0000 7f4e9da00640 1 -- 172.21.15.96:0/2398749704 >> 172.21.15.96:0/2398749704 conn(0x7f4e980b1fc0 msgr2=0x7f4e980b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:50.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.207+0000 7f4e9da00640 1 -- 172.21.15.96:0/2398749704 shutdown_connections 2024-05-28T06:08:50.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.207+0000 7f4e9da00640 1 -- 172.21.15.96:0/2398749704 wait complete. 2024-05-28T06:08:50.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.209+0000 7f4e9da00640 1 Processor -- start 2024-05-28T06:08:50.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.209+0000 7f4e9da00640 1 -- start start 2024-05-28T06:08:50.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.210+0000 7f4e9da00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e9806ee40 0x7f4e980cf8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:50.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.210+0000 7f4e9da00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4e9806f2c0 0x7f4e981570e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:50.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.210+0000 7f4e9da00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e98156b30 0x7f4e98149fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:50.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.210+0000 7f4e9da00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4e980afc20 con 0x7f4e98156b30 2024-05-28T06:08:50.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.210+0000 7f4e9da00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f4e980b2c30 con 0x7f4e9806ee40 2024-05-28T06:08:50.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.210+0000 7f4e9da00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f4e98157620 con 0x7f4e9806f2c0 2024-05-28T06:08:50.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.210+0000 7f4e9d000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e98156b30 0x7f4e98149fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:50.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.210+0000 7f4e9d000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e98156b30 0x7f4e98149fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51212/0 (socket says 172.21.15.96:51212) 2024-05-28T06:08:50.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.210+0000 7f4e9d000640 1 -- 172.21.15.96:0/3531541250 learned_addr learned my addr 172.21.15.96:0/3531541250 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:08:50.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.211+0000 7f4e97e00640 1 --2- 172.21.15.96:0/3531541250 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e9806ee40 0x7f4e980cf8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:50.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.211+0000 7f4e9d000640 1 -- 172.21.15.96:0/3531541250 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4e9806f2c0 msgr2=0x7f4e981570e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:50.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.211+0000 7f4e9d000640 1 --2- 172.21.15.96:0/3531541250 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4e9806f2c0 0x7f4e981570e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:50.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.211+0000 7f4e9d000640 1 -- 172.21.15.96:0/3531541250 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e9806ee40 msgr2=0x7f4e980cf8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:50.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.211+0000 7f4e9d000640 1 --2- 172.21.15.96:0/3531541250 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e9806ee40 0x7f4e980cf8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:50.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.211+0000 7f4e9d000640 1 -- 172.21.15.96:0/3531541250 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e9814a510 con 0x7f4e98156b30 2024-05-28T06:08:50.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.211+0000 7f4e9d000640 1 --2- 172.21.15.96:0/3531541250 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e98156b30 0x7f4e98149fd0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f4e9000c730 tx=0x7f4e9000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:50.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.211+0000 7f4e8fe00640 1 -- 172.21.15.96:0/3531541250 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e90003b70 con 0x7f4e98156b30 2024-05-28T06:08:50.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.211+0000 7f4e8fe00640 1 -- 172.21.15.96:0/3531541250 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e90003d30 con 0x7f4e98156b30 2024-05-28T06:08:50.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.211+0000 7f4e8fe00640 1 -- 172.21.15.96:0/3531541250 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e90005310 con 0x7f4e98156b30 2024-05-28T06:08:50.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.211+0000 7f4e9da00640 1 -- 172.21.15.96:0/3531541250 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e98157850 con 0x7f4e98156b30 2024-05-28T06:08:50.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.211+0000 7f4e9da00640 1 -- 172.21.15.96:0/3531541250 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e981580f0 con 0x7f4e98156b30 2024-05-28T06:08:50.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.213+0000 7f4e9da00640 1 -- 172.21.15.96:0/3531541250 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e980689e0 con 0x7f4e98156b30 2024-05-28T06:08:50.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.213+0000 7f4e8fe00640 1 -- 172.21.15.96:0/3531541250 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 9) ==== 91295+0+0 (secure 0 0 0) 0x7f4e900054b0 con 0x7f4e98156b30 2024-05-28T06:08:50.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.214+0000 7f4e8fe00640 1 --2- 172.21.15.96:0/3531541250 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4e7806cbf0 0x7f4e7806f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:50.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.214+0000 7f4e8fe00640 1 -- 172.21.15.96:0/3531541250 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4885+0+0 (secure 0 0 0) 0x7f4e9008bab0 con 0x7f4e98156b30 2024-05-28T06:08:50.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.215+0000 7f4e97e00640 1 --2- 172.21.15.96:0/3531541250 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4e7806cbf0 0x7f4e7806f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:50.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.216+0000 7f4e97e00640 1 --2- 172.21.15.96:0/3531541250 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4e7806cbf0 0x7f4e7806f0b0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f4e88034000 tx=0x7f4e8802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:50.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.218+0000 7f4e8fe00640 1 -- 172.21.15.96:0/3531541250 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4e90059f10 con 0x7f4e98156b30 2024-05-28T06:08:50.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.491+0000 7f4e9da00640 1 -- 172.21.15.96:0/3531541250 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f4e9806f7f0 con 0x7f4e98156b30 2024-05-28T06:08:50.494 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.492+0000 7f4e8fe00640 1 -- 172.21.15.96:0/3531541250 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v13) ==== 80+0+5304 (secure 0 0 0) 0x7f4e90059810 con 0x7f4e98156b30 2024-05-28T06:08:50.494 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:08:50.494 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:08:50.494 INFO:teuthology.orchestra.run.smithi096.stdout: "auth_dump": [ 2024-05-28T06:08:50.494 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:08:50.494 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "mds.a", 2024-05-28T06:08:50.494 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCWc1VmyA9HHBAANJCF8I/RP5ZYviyOzo0JCQ==", 2024-05-28T06:08:50.495 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:08:50.495 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow", 2024-05-28T06:08:50.495 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow *", 2024-05-28T06:08:50.495 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow *", 2024-05-28T06:08:50.495 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:08:50.495 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:08:50.495 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:08:50.495 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:08:50.495 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "mds.b", 2024-05-28T06:08:50.495 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCWc1VmtR0EOBAA296qGHTmB/2sMztDbDJ56g==", 2024-05-28T06:08:50.495 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:08:50.495 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow", 2024-05-28T06:08:50.495 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow *", 2024-05-28T06:08:50.495 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow *", 2024-05-28T06:08:50.496 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:08:50.496 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:08:50.496 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:08:50.496 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:08:50.496 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "mds.c", 2024-05-28T06:08:50.496 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCWc1VmrLxHKhAAhkTRLDKL6RUJQGetB8JQQA==", 2024-05-28T06:08:50.496 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:08:50.496 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow", 2024-05-28T06:08:50.496 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow *", 2024-05-28T06:08:50.496 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow *", 2024-05-28T06:08:50.496 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:08:50.496 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:08:50.496 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:08:50.496 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:08:50.497 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "mds.d", 2024-05-28T06:08:50.497 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCXc1VmW2aQCxAAnkDwQzlgy39vFe2O9Wwf7Q==", 2024-05-28T06:08:50.497 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:08:50.497 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow", 2024-05-28T06:08:50.497 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow *", 2024-05-28T06:08:50.497 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow *", 2024-05-28T06:08:50.497 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:08:50.497 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:08:50.497 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:08:50.497 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:08:50.497 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "osd.0", 2024-05-28T06:08:50.497 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCbc1VmEcNAAxAAiB5AsfY2uRY1kc6h80chuw==", 2024-05-28T06:08:50.497 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:08:50.497 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow profile osd", 2024-05-28T06:08:50.498 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile osd", 2024-05-28T06:08:50.498 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:08:50.498 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:08:50.498 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:08:50.498 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:08:50.498 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "osd.1", 2024-05-28T06:08:50.498 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCec1Vm6CAaDhAA1NkYM2q4fTsgSDSwJZL3Qg==", 2024-05-28T06:08:50.498 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:08:50.498 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow profile osd", 2024-05-28T06:08:50.498 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile osd", 2024-05-28T06:08:50.498 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:08:50.498 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:08:50.498 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:08:50.499 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:08:50.499 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "osd.2", 2024-05-28T06:08:50.499 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQChc1VmKn1jFxAA+dM4gM7FR8NwBpyd833A/Q==", 2024-05-28T06:08:50.499 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:08:50.499 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow profile osd", 2024-05-28T06:08:50.499 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile osd", 2024-05-28T06:08:50.499 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:08:50.499 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:08:50.499 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:08:50.499 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:08:50.499 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "osd.3", 2024-05-28T06:08:50.499 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCkc1VmV07MIRAAHIXQGDVbL9V5QqexQ89IJA==", 2024-05-28T06:08:50.499 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:08:50.500 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow profile osd", 2024-05-28T06:08:50.500 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile osd", 2024-05-28T06:08:50.500 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:08:50.500 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:08:50.500 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:08:50.500 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:08:50.500 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "osd.4", 2024-05-28T06:08:50.500 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCqc1VmobuZMxAAlY8xhuzqlkuFrcbTouF5rA==", 2024-05-28T06:08:50.500 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:08:50.500 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow profile osd", 2024-05-28T06:08:50.500 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile osd", 2024-05-28T06:08:50.500 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:08:50.500 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:08:50.500 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:08:50.501 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:08:50.501 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "osd.5", 2024-05-28T06:08:50.501 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCuc1Vmwt3QAxAAUFj/e/wJvxljgAbcTb32Ug==", 2024-05-28T06:08:50.501 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:08:50.501 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow profile osd", 2024-05-28T06:08:50.501 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile osd", 2024-05-28T06:08:50.501 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:08:50.501 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:08:50.501 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:08:50.501 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:08:50.501 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "osd.6", 2024-05-28T06:08:50.501 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCxc1VmscerDxAAqEDAqy6/OvdfSJ+ALZ7WeQ==", 2024-05-28T06:08:50.502 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:08:50.502 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow profile osd", 2024-05-28T06:08:50.502 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile osd", 2024-05-28T06:08:50.502 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:08:50.502 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:08:50.502 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:08:50.502 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:08:50.502 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "osd.7", 2024-05-28T06:08:50.502 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQC0c1VmgxkbKhAA7dFzDTBKFmKiFFxPsyiT4Q==", 2024-05-28T06:08:50.502 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:08:50.502 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow profile osd", 2024-05-28T06:08:50.502 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile osd", 2024-05-28T06:08:50.502 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:08:50.503 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:08:50.503 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:08:50.503 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:08:50.503 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "client.0", 2024-05-28T06:08:50.503 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCXc1Vm91hbGBAASTL3HQR8XOs7wVQEHbPl/g==", 2024-05-28T06:08:50.503 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:08:50.503 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow", 2024-05-28T06:08:50.503 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow r", 2024-05-28T06:08:50.503 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-05-28T06:08:50.503 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:08:50.503 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:08:50.503 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:08:50.503 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "client.1", 2024-05-28T06:08:50.503 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCXc1VmIc77KhAAw2e0alTmVROhyCIFW4H3Sg==", 2024-05-28T06:08:50.504 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:08:50.504 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow", 2024-05-28T06:08:50.504 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow r", 2024-05-28T06:08:50.504 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow rw", 2024-05-28T06:08:50.504 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow rwx" 2024-05-28T06:08:50.504 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:08:50.504 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:08:50.504 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:08:50.504 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "client.2", 2024-05-28T06:08:50.504 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCXc1VmCS+OIRAAMV3Au6KuchLhA59QTGbv1w==", 2024-05-28T06:08:50.504 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:08:50.504 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow", 2024-05-28T06:08:50.504 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow r", 2024-05-28T06:08:50.504 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow rw", 2024-05-28T06:08:50.505 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow rwx" 2024-05-28T06:08:50.505 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:08:50.505 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:08:50.505 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:08:50.505 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "client.3", 2024-05-28T06:08:50.505 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCXc1Vm5r3pNBAAkVZ3arjywHQkEP5lmS8xqg==", 2024-05-28T06:08:50.505 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:08:50.505 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow", 2024-05-28T06:08:50.505 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow r", 2024-05-28T06:08:50.505 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow rw", 2024-05-28T06:08:50.505 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow rwx" 2024-05-28T06:08:50.505 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:08:50.505 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:08:50.505 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:08:50.505 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "client.admin", 2024-05-28T06:08:50.506 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCVc1Vm5QaXFhAAQFGKgbZ/kpGisNm2lIw0Uw==", 2024-05-28T06:08:50.506 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:08:50.506 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow *", 2024-05-28T06:08:50.506 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow *", 2024-05-28T06:08:50.506 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow *", 2024-05-28T06:08:50.506 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:08:50.506 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:08:50.506 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:08:50.506 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:08:50.506 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "mgr.x", 2024-05-28T06:08:50.506 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCWc1VmVMx/EhAAylvi37aq3hbMOhUSQWLVSg==", 2024-05-28T06:08:50.506 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:08:50.506 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow *", 2024-05-28T06:08:50.507 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile mgr", 2024-05-28T06:08:50.507 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:08:50.507 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:08:50.507 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:08:50.507 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:08:50.507 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "mgr.y", 2024-05-28T06:08:50.507 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCWc1Vmh5r7BxAAA74gn+FozehM+34eFik8kw==", 2024-05-28T06:08:50.507 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:08:50.507 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow *", 2024-05-28T06:08:50.507 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile mgr", 2024-05-28T06:08:50.507 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:08:50.507 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:08:50.507 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:08:50.507 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-05-28T06:08:50.508 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:08:50.509 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.495+0000 7f4e9da00640 1 -- 172.21.15.96:0/3531541250 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4e7806cbf0 msgr2=0x7f4e7806f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:50.509 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.495+0000 7f4e9da00640 1 --2- 172.21.15.96:0/3531541250 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4e7806cbf0 0x7f4e7806f0b0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f4e88034000 tx=0x7f4e8802e000 comp rx=0 tx=0).stop 2024-05-28T06:08:50.509 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.495+0000 7f4e9da00640 1 -- 172.21.15.96:0/3531541250 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e98156b30 msgr2=0x7f4e98149fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:50.509 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.495+0000 7f4e9da00640 1 --2- 172.21.15.96:0/3531541250 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e98156b30 0x7f4e98149fd0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f4e9000c730 tx=0x7f4e9000cbf0 comp rx=0 tx=0).stop 2024-05-28T06:08:50.509 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.496+0000 7f4e9da00640 1 -- 172.21.15.96:0/3531541250 shutdown_connections 2024-05-28T06:08:50.509 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.496+0000 7f4e9da00640 1 --2- 172.21.15.96:0/3531541250 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4e7806cbf0 0x7f4e7806f0b0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:50.509 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.496+0000 7f4e9da00640 1 --2- 172.21.15.96:0/3531541250 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e98156b30 0x7f4e98149fd0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:50.509 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.496+0000 7f4e9da00640 1 --2- 172.21.15.96:0/3531541250 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4e9806f2c0 0x7f4e981570e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:50.509 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.496+0000 7f4e9da00640 1 --2- 172.21.15.96:0/3531541250 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e9806ee40 0x7f4e980cf8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:50.510 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.496+0000 7f4e9da00640 1 -- 172.21.15.96:0/3531541250 >> 172.21.15.96:0/3531541250 conn(0x7f4e980b1fc0 msgr2=0x7f4e980ad230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:50.510 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.496+0000 7f4e9da00640 1 -- 172.21.15.96:0/3531541250 shutdown_connections 2024-05-28T06:08:50.510 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.496+0000 7f4e9da00640 1 -- 172.21.15.96:0/3531541250 wait complete. 2024-05-28T06:08:50.516 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:08:50.698 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.697+0000 7f3dd2400640 1 Processor -- start 2024-05-28T06:08:50.699 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.697+0000 7f3dd2400640 1 -- start start 2024-05-28T06:08:50.700 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.698+0000 7f3dd2400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f3dcc15b3a0 0x7f3dcc0b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:50.700 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.698+0000 7f3dd2400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f3dcc06dc20 0x7f3dcc15f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:50.700 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.698+0000 7f3dd2400640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3dcc0afd60 con 0x7f3dcc072c00 2024-05-28T06:08:50.700 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.698+0000 7f3dd2400640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f3dcc06f980 con 0x7f3dcc15b3a0 2024-05-28T06:08:50.700 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.698+0000 7f3dd2400640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f3dcc0ae0c0 con 0x7f3dcc06dc20 2024-05-28T06:08:50.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.699+0000 7f3dd1000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f3dcc15b3a0 0x7f3dcc0b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:50.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.699+0000 7f3dd1000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f3dcc15b3a0 0x7f3dcc0b32c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51214/0 (socket says 172.21.15.96:51214) 2024-05-28T06:08:50.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.699+0000 7f3dd1000640 1 -- 172.21.15.96:0/1185691751 learned_addr learned my addr 172.21.15.96:0/1185691751 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:08:50.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.699+0000 7f3dd1000640 1 -- 172.21.15.96:0/1185691751 >> v2:172.21.15.123:3300/0 conn(0x7f3dcc06dc20 msgr2=0x7f3dcc15f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:50.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.699+0000 7f3dd1000640 1 --2- 172.21.15.96:0/1185691751 >> v2:172.21.15.123:3300/0 conn(0x7f3dcc06dc20 0x7f3dcc15f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:50.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.699+0000 7f3dd1000640 1 -- 172.21.15.96:0/1185691751 >> v1:172.21.15.123:6789/0 conn(0x7f3dcc072c00 legacy=0x7f3dcc073010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:50.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.699+0000 7f3dd1000640 1 -- 172.21.15.96:0/1185691751 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dcc0af7a0 con 0x7f3dcc15b3a0 2024-05-28T06:08:50.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.700+0000 7f3dd1000640 1 --2- 172.21.15.96:0/1185691751 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3dcc15b3a0 0x7f3dcc0b32c0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f3dcc0711f0 tx=0x7f3dc002c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6896ec92f6b14f05 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:50.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.701+0000 7f3dcb400640 1 -- 172.21.15.96:0/1185691751 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dc0038070 con 0x7f3dcc15b3a0 2024-05-28T06:08:50.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.701+0000 7f3dcb400640 1 -- 172.21.15.96:0/1185691751 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3dc0031710 con 0x7f3dcc15b3a0 2024-05-28T06:08:50.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.701+0000 7f3dcb400640 1 -- 172.21.15.96:0/1185691751 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dc0031ae0 con 0x7f3dcc15b3a0 2024-05-28T06:08:50.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.701+0000 7f3dd2400640 1 -- 172.21.15.96:0/1185691751 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3dcc15b3a0 msgr2=0x7f3dcc0b32c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:50.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.701+0000 7f3dd2400640 1 --2- 172.21.15.96:0/1185691751 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3dcc15b3a0 0x7f3dcc0b32c0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f3dcc0711f0 tx=0x7f3dc002c440 comp rx=0 tx=0).stop 2024-05-28T06:08:50.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.701+0000 7f3dd2400640 1 -- 172.21.15.96:0/1185691751 shutdown_connections 2024-05-28T06:08:50.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.701+0000 7f3dd2400640 1 --2- 172.21.15.96:0/1185691751 >> v2:172.21.15.123:3300/0 conn(0x7f3dcc06dc20 0x7f3dcc15f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:50.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.701+0000 7f3dd2400640 1 --2- 172.21.15.96:0/1185691751 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3dcc15b3a0 0x7f3dcc0b32c0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:50.705 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.701+0000 7f3dd2400640 1 -- 172.21.15.96:0/1185691751 >> 172.21.15.96:0/1185691751 conn(0x7f3dcc0b1fc0 msgr2=0x7f3dcc0b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:50.705 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.701+0000 7f3dd2400640 1 -- 172.21.15.96:0/1185691751 shutdown_connections 2024-05-28T06:08:50.705 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.702+0000 7f3dd2400640 1 -- 172.21.15.96:0/1185691751 wait complete. 2024-05-28T06:08:50.705 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.703+0000 7f3dd2400640 1 Processor -- start 2024-05-28T06:08:50.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.703+0000 7f3dd2400640 1 -- start start 2024-05-28T06:08:50.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.704+0000 7f3dd2400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3dcc06dc20 0x7f3dcc148870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:50.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.704+0000 7f3dd2400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3dcc072c00 0x7f3dcc153700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:50.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.704+0000 7f3dd2400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3dcc15b3a0 0x7f3dcc153c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:50.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.704+0000 7f3dd2400640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f3dcc15b780 con 0x7f3dcc15b3a0 2024-05-28T06:08:50.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.704+0000 7f3dd2400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f3dcc15fee0 con 0x7f3dcc06dc20 2024-05-28T06:08:50.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.704+0000 7f3dd2400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f3dcc0ae550 con 0x7f3dcc072c00 2024-05-28T06:08:50.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.705+0000 7f3dd1a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3dcc15b3a0 0x7f3dcc153c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:50.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.705+0000 7f3dd1a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3dcc15b3a0 0x7f3dcc153c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51224/0 (socket says 172.21.15.96:51224) 2024-05-28T06:08:50.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.705+0000 7f3dd1a00640 1 -- 172.21.15.96:0/4256945865 learned_addr learned my addr 172.21.15.96:0/4256945865 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:08:50.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.705+0000 7f3dcbe00640 1 --2- 172.21.15.96:0/4256945865 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3dcc072c00 0x7f3dcc153700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:50.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.705+0000 7f3dd1000640 1 --2- 172.21.15.96:0/4256945865 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3dcc06dc20 0x7f3dcc148870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:50.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.705+0000 7f3dd1a00640 1 -- 172.21.15.96:0/4256945865 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3dcc072c00 msgr2=0x7f3dcc153700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:50.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.705+0000 7f3dd1a00640 1 --2- 172.21.15.96:0/4256945865 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3dcc072c00 0x7f3dcc153700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:50.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.705+0000 7f3dd1a00640 1 -- 172.21.15.96:0/4256945865 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3dcc06dc20 msgr2=0x7f3dcc148870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:50.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.705+0000 7f3dd1a00640 1 --2- 172.21.15.96:0/4256945865 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3dcc06dc20 0x7f3dcc148870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:50.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.705+0000 7f3dd1a00640 1 -- 172.21.15.96:0/4256945865 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dcc148db0 con 0x7f3dcc15b3a0 2024-05-28T06:08:50.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.705+0000 7f3dcbe00640 1 --2- 172.21.15.96:0/4256945865 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3dcc072c00 0x7f3dcc153700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:08:50.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.705+0000 7f3dd1a00640 1 --2- 172.21.15.96:0/4256945865 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3dcc15b3a0 0x7f3dcc153c40 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f3dc400c780 tx=0x7f3dc400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:50.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.706+0000 7f3dc8c00640 1 -- 172.21.15.96:0/4256945865 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dc4017070 con 0x7f3dcc15b3a0 2024-05-28T06:08:50.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.706+0000 7f3dc8c00640 1 -- 172.21.15.96:0/4256945865 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3dc4003da0 con 0x7f3dcc15b3a0 2024-05-28T06:08:50.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.706+0000 7f3dd2400640 1 -- 172.21.15.96:0/4256945865 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3dcc150da0 con 0x7f3dcc15b3a0 2024-05-28T06:08:50.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.706+0000 7f3dc8c00640 1 -- 172.21.15.96:0/4256945865 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dc40052b0 con 0x7f3dcc15b3a0 2024-05-28T06:08:50.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.706+0000 7f3dd2400640 1 -- 172.21.15.96:0/4256945865 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3dcc151320 con 0x7f3dcc15b3a0 2024-05-28T06:08:50.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.707+0000 7f3dd2400640 1 -- 172.21.15.96:0/4256945865 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3dcc0689e0 con 0x7f3dcc15b3a0 2024-05-28T06:08:50.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.708+0000 7f3dc8c00640 1 -- 172.21.15.96:0/4256945865 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 9) ==== 91295+0+0 (secure 0 0 0) 0x7f3dc4005450 con 0x7f3dcc15b3a0 2024-05-28T06:08:50.710 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.709+0000 7f3dc8c00640 1 --2- 172.21.15.96:0/4256945865 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3dac06cbc0 0x7f3dac06f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:50.710 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.709+0000 7f3dc8c00640 1 -- 172.21.15.96:0/4256945865 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4885+0+0 (secure 0 0 0) 0x7f3dc408b270 con 0x7f3dcc15b3a0 2024-05-28T06:08:50.711 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.709+0000 7f3dd1000640 1 --2- 172.21.15.96:0/4256945865 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3dac06cbc0 0x7f3dac06f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:50.712 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.711+0000 7f3dd1000640 1 --2- 172.21.15.96:0/4256945865 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3dac06cbc0 0x7f3dac06f080 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f3dc0035000 tx=0x7f3dc0008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:50.714 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.713+0000 7f3dc8c00640 1 -- 172.21.15.96:0/4256945865 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3dcc148db0 con 0x7f3dcc15b3a0 2024-05-28T06:08:50.917 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.915+0000 7f3dd2400640 1 -- 172.21.15.96:0/4256945865 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3dcc0b0be0 con 0x7f3dcc15b3a0 2024-05-28T06:08:50.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.917+0000 7f3dc8c00640 1 -- 172.21.15.96:0/4256945865 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) ==== 74+0+12946 (secure 0 0 0) 0x7f3dc40594e0 con 0x7f3dcc15b3a0 2024-05-28T06:08:50.919 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:08:50.919 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":41,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:08:49.981793+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:08:50.922 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.920+0000 7f3dd2400640 1 -- 172.21.15.96:0/4256945865 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3dac06cbc0 msgr2=0x7f3dac06f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:50.922 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.920+0000 7f3dd2400640 1 --2- 172.21.15.96:0/4256945865 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3dac06cbc0 0x7f3dac06f080 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f3dc0035000 tx=0x7f3dc0008030 comp rx=0 tx=0).stop 2024-05-28T06:08:50.922 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.920+0000 7f3dd2400640 1 -- 172.21.15.96:0/4256945865 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3dcc15b3a0 msgr2=0x7f3dcc153c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:50.922 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.920+0000 7f3dd2400640 1 --2- 172.21.15.96:0/4256945865 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3dcc15b3a0 0x7f3dcc153c40 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f3dc400c780 tx=0x7f3dc400cc40 comp rx=0 tx=0).stop 2024-05-28T06:08:50.922 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.920+0000 7f3dd2400640 1 -- 172.21.15.96:0/4256945865 shutdown_connections 2024-05-28T06:08:50.922 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.920+0000 7f3dd2400640 1 --2- 172.21.15.96:0/4256945865 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3dac06cbc0 0x7f3dac06f080 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:50.922 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.920+0000 7f3dd2400640 1 --2- 172.21.15.96:0/4256945865 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3dcc15b3a0 0x7f3dcc153c40 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:50.922 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.920+0000 7f3dd2400640 1 --2- 172.21.15.96:0/4256945865 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3dcc072c00 0x7f3dcc153700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:50.922 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.920+0000 7f3dd2400640 1 --2- 172.21.15.96:0/4256945865 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3dcc06dc20 0x7f3dcc148870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:50.922 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.920+0000 7f3dd2400640 1 -- 172.21.15.96:0/4256945865 >> 172.21.15.96:0/4256945865 conn(0x7f3dcc0b1fc0 msgr2=0x7f3dcc0d0ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:50.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.920+0000 7f3dd2400640 1 -- 172.21.15.96:0/4256945865 shutdown_connections 2024-05-28T06:08:50.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:50.920+0000 7f3dd2400640 1 -- 172.21.15.96:0/4256945865 wait complete. 2024-05-28T06:08:50.940 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-28T06:03:59.517102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-28T06:08:50.941 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-28T06:08:51.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.132+0000 7f3e60a00640 1 Processor -- start 2024-05-28T06:08:51.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.132+0000 7f3e60a00640 1 -- start start 2024-05-28T06:08:51.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.134+0000 7f3e60a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f3e5c15e870 0x7f3e5c0d0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:51.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.134+0000 7f3e60a00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e5c15ec50 con 0x7f3e5c0d14d0 2024-05-28T06:08:51.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.134+0000 7f3e60a00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e5c0cff80 con 0x7f3e5c0cfa40 2024-05-28T06:08:51.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.134+0000 7f3e60a00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f3e5c0d2150 con 0x7f3e5c15e870 2024-05-28T06:08:51.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.134+0000 7f3e5b400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f3e5c15e870 0x7f3e5c0d0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:51.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.134+0000 7f3e52a00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f3e5c0d14d0 0x7f3e5c0cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:56406/0 (socket says 172.21.15.96:56406) 2024-05-28T06:08:51.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.134+0000 7f3e52a00640 1 -- 172.21.15.96:0/131312556 learned_addr learned my addr 172.21.15.96:0/131312556 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:08:51.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.134+0000 7f3e5b400640 1 -- 172.21.15.96:0/131312556 >> v1:172.21.15.123:6789/0 conn(0x7f3e5c0cfa40 legacy=0x7f3e5c0d25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:51.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.134+0000 7f3e5b400640 1 -- 172.21.15.96:0/131312556 >> v1:172.21.15.96:6789/0 conn(0x7f3e5c0d14d0 legacy=0x7f3e5c0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:51.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.134+0000 7f3e5b400640 1 -- 172.21.15.96:0/131312556 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e5c0d0260 con 0x7f3e5c15e870 2024-05-28T06:08:51.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.134+0000 7f3e5aa00640 1 -- 172.21.15.96:0/131312556 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2854856540 0 0) 0x7f3e5c15ec50 con 0x7f3e5c0d14d0 2024-05-28T06:08:51.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.136+0000 7f3e5b400640 1 --2- 172.21.15.96:0/131312556 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3e5c15e870 0x7f3e5c0d0f90 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c0097a0 tx=0x7f3e4c02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=485fee0082865e7b server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:51.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.136+0000 7f3e5aa00640 1 -- 172.21.15.96:0/131312556 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e4c037070 con 0x7f3e5c15e870 2024-05-28T06:08:51.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.136+0000 7f3e5aa00640 1 -- 172.21.15.96:0/131312556 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e4c032440 con 0x7f3e5c15e870 2024-05-28T06:08:51.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.136+0000 7f3e5aa00640 1 -- 172.21.15.96:0/131312556 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e4c032810 con 0x7f3e5c15e870 2024-05-28T06:08:51.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.136+0000 7f3e60a00640 1 -- 172.21.15.96:0/131312556 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3e5c15e870 msgr2=0x7f3e5c0d0f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:51.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.136+0000 7f3e60a00640 1 --2- 172.21.15.96:0/131312556 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3e5c15e870 0x7f3e5c0d0f90 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c0097a0 tx=0x7f3e4c02a1e0 comp rx=0 tx=0).stop 2024-05-28T06:08:51.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.136+0000 7f3e60a00640 1 -- 172.21.15.96:0/131312556 shutdown_connections 2024-05-28T06:08:51.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.136+0000 7f3e60a00640 1 --2- 172.21.15.96:0/131312556 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3e5c15e870 0x7f3e5c0d0f90 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:51.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.136+0000 7f3e60a00640 1 -- 172.21.15.96:0/131312556 >> 172.21.15.96:0/131312556 conn(0x7f3e5c08bfc0 msgr2=0x7f3e5c0d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:51.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.137+0000 7f3e60a00640 1 -- 172.21.15.96:0/131312556 shutdown_connections 2024-05-28T06:08:51.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.137+0000 7f3e60a00640 1 -- 172.21.15.96:0/131312556 wait complete. 2024-05-28T06:08:51.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.138+0000 7f3e60a00640 1 Processor -- start 2024-05-28T06:08:51.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.138+0000 7f3e60a00640 1 -- start start 2024-05-28T06:08:51.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.139+0000 7f3e60a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3e5c0cfa40 0x7f3e5c0c9c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:51.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.139+0000 7f3e60a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3e5c0d14d0 0x7f3e5c159140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:51.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.139+0000 7f3e60a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3e5c15e870 0x7f3e5c15c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:51.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.139+0000 7f3e60a00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f3e5c0d4fb0 con 0x7f3e5c0d14d0 2024-05-28T06:08:51.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.139+0000 7f3e60a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f3e5c0d0820 con 0x7f3e5c15e870 2024-05-28T06:08:51.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.139+0000 7f3e60a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f3e5c0d2cf0 con 0x7f3e5c0cfa40 2024-05-28T06:08:51.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.140+0000 7f3e52a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3e5c0d14d0 0x7f3e5c159140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:51.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.140+0000 7f3e52a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3e5c0d14d0 0x7f3e5c159140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51236/0 (socket says 172.21.15.96:51236) 2024-05-28T06:08:51.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.140+0000 7f3e52a00640 1 -- 172.21.15.96:0/1301716191 learned_addr learned my addr 172.21.15.96:0/1301716191 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:08:51.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.140+0000 7f3e52a00640 1 -- 172.21.15.96:0/1301716191 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3e5c0cfa40 msgr2=0x7f3e5c0c9c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:51.143 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.140+0000 7f3e5b400640 1 --2- 172.21.15.96:0/1301716191 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3e5c0cfa40 0x7f3e5c0c9c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:51.143 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.140+0000 7f3e5be00640 1 --2- 172.21.15.96:0/1301716191 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3e5c15e870 0x7f3e5c15c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:51.143 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.140+0000 7f3e52a00640 1 --2- 172.21.15.96:0/1301716191 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3e5c0cfa40 0x7f3e5c0c9c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:51.143 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.140+0000 7f3e52a00640 1 -- 172.21.15.96:0/1301716191 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3e5c15e870 msgr2=0x7f3e5c15c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:51.143 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.140+0000 7f3e52a00640 1 --2- 172.21.15.96:0/1301716191 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3e5c15e870 0x7f3e5c15c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:51.143 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.140+0000 7f3e52a00640 1 -- 172.21.15.96:0/1301716191 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e5c0c8b60 con 0x7f3e5c0d14d0 2024-05-28T06:08:51.143 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.140+0000 7f3e52a00640 1 --2- 172.21.15.96:0/1301716191 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3e5c0d14d0 0x7f3e5c159140 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f3e4400c4d0 tx=0x7f3e4400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:51.143 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.140+0000 7f3e53e00640 1 -- 172.21.15.96:0/1301716191 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e44007480 con 0x7f3e5c0d14d0 2024-05-28T06:08:51.143 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.141+0000 7f3e53e00640 1 -- 172.21.15.96:0/1301716191 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e440057a0 con 0x7f3e5c0d14d0 2024-05-28T06:08:51.143 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.141+0000 7f3e53e00640 1 -- 172.21.15.96:0/1301716191 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e44007620 con 0x7f3e5c0d14d0 2024-05-28T06:08:51.143 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.141+0000 7f3e60a00640 1 -- 172.21.15.96:0/1301716191 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e5c0c8e40 con 0x7f3e5c0d14d0 2024-05-28T06:08:51.143 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.141+0000 7f3e60a00640 1 -- 172.21.15.96:0/1301716191 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e5c0c9400 con 0x7f3e5c0d14d0 2024-05-28T06:08:51.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.142+0000 7f3e60a00640 1 -- 172.21.15.96:0/1301716191 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e5c06dfb0 con 0x7f3e5c0d14d0 2024-05-28T06:08:51.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.143+0000 7f3e53e00640 1 -- 172.21.15.96:0/1301716191 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 9) ==== 91295+0+0 (secure 0 0 0) 0x7f3e44005b50 con 0x7f3e5c0d14d0 2024-05-28T06:08:51.145 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.143+0000 7f3e53e00640 1 --2- 172.21.15.96:0/1301716191 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3e3806cbc0 0x7f3e3806f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:51.145 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.144+0000 7f3e53e00640 1 -- 172.21.15.96:0/1301716191 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4885+0+0 (secure 0 0 0) 0x7f3e4408ba30 con 0x7f3e5c0d14d0 2024-05-28T06:08:51.145 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.144+0000 7f3e5b400640 1 --2- 172.21.15.96:0/1301716191 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3e3806cbc0 0x7f3e3806f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:51.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.145+0000 7f3e5b400640 1 --2- 172.21.15.96:0/1301716191 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3e3806cbc0 0x7f3e3806f080 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c034000 tx=0x7f3e4c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:51.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.148+0000 7f3e53e00640 1 -- 172.21.15.96:0/1301716191 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3e5c06dfb0 con 0x7f3e5c0d14d0 2024-05-28T06:08:51.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.346+0000 7f3e60a00640 1 -- 172.21.15.96:0/1301716191 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f3e5c0689e0 con 0x7f3e5c0d14d0 2024-05-28T06:08:51.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.347+0000 7f3e53e00640 1 -- 172.21.15.96:0/1301716191 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v41) ==== 93+0+10 (secure 0 0 0) 0x7f3e440182b0 con 0x7f3e5c0d14d0 2024-05-28T06:08:51.348 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 1 2024-05-28T06:08:51.351 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.349+0000 7f3e60a00640 1 -- 172.21.15.96:0/1301716191 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3e3806cbc0 msgr2=0x7f3e3806f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:51.351 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.349+0000 7f3e60a00640 1 --2- 172.21.15.96:0/1301716191 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3e3806cbc0 0x7f3e3806f080 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c034000 tx=0x7f3e4c02e000 comp rx=0 tx=0).stop 2024-05-28T06:08:51.352 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.349+0000 7f3e60a00640 1 -- 172.21.15.96:0/1301716191 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3e5c0d14d0 msgr2=0x7f3e5c159140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:51.352 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.349+0000 7f3e60a00640 1 --2- 172.21.15.96:0/1301716191 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3e5c0d14d0 0x7f3e5c159140 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f3e4400c4d0 tx=0x7f3e4400c990 comp rx=0 tx=0).stop 2024-05-28T06:08:51.352 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.349+0000 7f3e60a00640 1 -- 172.21.15.96:0/1301716191 shutdown_connections 2024-05-28T06:08:51.352 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.349+0000 7f3e60a00640 1 --2- 172.21.15.96:0/1301716191 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3e3806cbc0 0x7f3e3806f080 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:51.352 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.349+0000 7f3e60a00640 1 --2- 172.21.15.96:0/1301716191 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3e5c15e870 0x7f3e5c15c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:51.352 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.349+0000 7f3e60a00640 1 --2- 172.21.15.96:0/1301716191 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3e5c0d14d0 0x7f3e5c159140 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:51.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.349+0000 7f3e60a00640 1 --2- 172.21.15.96:0/1301716191 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3e5c0cfa40 0x7f3e5c0c9c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:51.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.349+0000 7f3e60a00640 1 -- 172.21.15.96:0/1301716191 >> 172.21.15.96:0/1301716191 conn(0x7f3e5c08bfc0 msgr2=0x7f3e5c0cdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:51.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.350+0000 7f3e60a00640 1 -- 172.21.15.96:0/1301716191 shutdown_connections 2024-05-28T06:08:51.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.350+0000 7f3e60a00640 1 -- 172.21.15.96:0/1301716191 wait complete. 2024-05-28T06:08:51.370 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-05-28T06:08:51.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.571+0000 7f53aac00640 1 Processor -- start 2024-05-28T06:08:51.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.572+0000 7f53aac00640 1 -- start start 2024-05-28T06:08:51.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.573+0000 7f53aac00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f53a40b4b60 0x7f53a40cfaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:51.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.573+0000 7f53aac00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f53a40d04a0 0x7f53a40d0930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:51.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.573+0000 7f53aac00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53a40ced40 con 0x7f53a40b3460 2024-05-28T06:08:51.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.573+0000 7f53aac00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f53a40b3f90 con 0x7f53a40b4b60 2024-05-28T06:08:51.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.573+0000 7f53aac00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f53a415fbb0 con 0x7f53a40d04a0 2024-05-28T06:08:51.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.573+0000 7f53a3e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f53a40b4b60 0x7f53a40cfaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:51.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.573+0000 7f53a3e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f53a40b4b60 0x7f53a40cfaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51252/0 (socket says 172.21.15.96:51252) 2024-05-28T06:08:51.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.573+0000 7f53a3e00640 1 -- 172.21.15.96:0/3381646297 learned_addr learned my addr 172.21.15.96:0/3381646297 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:08:51.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.574+0000 7f53a3e00640 1 -- 172.21.15.96:0/3381646297 >> v2:172.21.15.123:3300/0 conn(0x7f53a40d04a0 msgr2=0x7f53a40d0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:51.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.574+0000 7f539b400640 1 --2- 172.21.15.96:0/3381646297 >> v2:172.21.15.123:3300/0 conn(0x7f53a40d04a0 0x7f53a40d0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:51.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.574+0000 7f53a3e00640 1 --2- 172.21.15.96:0/3381646297 >> v2:172.21.15.123:3300/0 conn(0x7f53a40d04a0 0x7f53a40d0930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:51.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.574+0000 7f53a3e00640 1 -- 172.21.15.96:0/3381646297 >> v1:172.21.15.123:6789/0 conn(0x7f53a40b3460 legacy=0x7f53a40b3880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:51.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.574+0000 7f53a3e00640 1 -- 172.21.15.96:0/3381646297 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53a40cefc0 con 0x7f53a40b4b60 2024-05-28T06:08:51.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.575+0000 7f53a3e00640 1 --2- 172.21.15.96:0/3381646297 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f53a40b4b60 0x7f53a40cfaa0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f53a40d1180 tx=0x7f539402a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6284f46c03839885 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:51.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.575+0000 7f53a3400640 1 -- 172.21.15.96:0/3381646297 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5394037070 con 0x7f53a40b4b60 2024-05-28T06:08:51.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.575+0000 7f53a3400640 1 -- 172.21.15.96:0/3381646297 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5394032440 con 0x7f53a40b4b60 2024-05-28T06:08:51.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.575+0000 7f53a3400640 1 -- 172.21.15.96:0/3381646297 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5394032810 con 0x7f53a40b4b60 2024-05-28T06:08:51.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.576+0000 7f53aac00640 1 -- 172.21.15.96:0/3381646297 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f53a40b4b60 msgr2=0x7f53a40cfaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:51.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.576+0000 7f53aac00640 1 --2- 172.21.15.96:0/3381646297 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f53a40b4b60 0x7f53a40cfaa0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f53a40d1180 tx=0x7f539402a230 comp rx=0 tx=0).stop 2024-05-28T06:08:51.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.576+0000 7f53aac00640 1 -- 172.21.15.96:0/3381646297 shutdown_connections 2024-05-28T06:08:51.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.576+0000 7f53aac00640 1 --2- 172.21.15.96:0/3381646297 >> v2:172.21.15.123:3300/0 conn(0x7f53a40d04a0 0x7f53a40d0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:51.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.576+0000 7f53aac00640 1 --2- 172.21.15.96:0/3381646297 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f53a40b4b60 0x7f53a40cfaa0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:51.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.576+0000 7f53aac00640 1 -- 172.21.15.96:0/3381646297 >> 172.21.15.96:0/3381646297 conn(0x7f53a406bae0 msgr2=0x7f53a406bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:51.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.576+0000 7f53aac00640 1 -- 172.21.15.96:0/3381646297 shutdown_connections 2024-05-28T06:08:51.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.576+0000 7f53aac00640 1 -- 172.21.15.96:0/3381646297 wait complete. 2024-05-28T06:08:51.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.578+0000 7f53aac00640 1 Processor -- start 2024-05-28T06:08:51.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.578+0000 7f53aac00640 1 -- start start 2024-05-28T06:08:51.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.579+0000 7f53aac00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f53a40b3460 0x7f53a40c6940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:51.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.579+0000 7f53aac00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f53a40b4b60 0x7f53a40c4a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:51.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.579+0000 7f53aac00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f53a40d04a0 0x7f53a41f7350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:51.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.579+0000 7f53aac00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f53a40cffe0 con 0x7f53a40d04a0 2024-05-28T06:08:51.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.579+0000 7f53aac00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f53a40cf580 con 0x7f53a40b4b60 2024-05-28T06:08:51.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.579+0000 7f53aac00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f53a415ede0 con 0x7f53a40b3460 2024-05-28T06:08:51.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.579+0000 7f53a8c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f53a40d04a0 0x7f53a41f7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:51.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.579+0000 7f53a8c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f53a40d04a0 0x7f53a41f7350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51258/0 (socket says 172.21.15.96:51258) 2024-05-28T06:08:51.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.579+0000 7f53a8c00640 1 -- 172.21.15.96:0/3756987104 learned_addr learned my addr 172.21.15.96:0/3756987104 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:08:51.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.580+0000 7f539b400640 1 --2- 172.21.15.96:0/3756987104 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f53a40b4b60 0x7f53a40c4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:51.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.580+0000 7f53a3e00640 1 --2- 172.21.15.96:0/3756987104 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f53a40b3460 0x7f53a40c6940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:51.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.580+0000 7f53a8c00640 1 -- 172.21.15.96:0/3756987104 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f53a40b3460 msgr2=0x7f53a40c6940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:51.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.580+0000 7f53a8c00640 1 --2- 172.21.15.96:0/3756987104 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f53a40b3460 0x7f53a40c6940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:51.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.580+0000 7f53a8c00640 1 -- 172.21.15.96:0/3756987104 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f53a40b4b60 msgr2=0x7f53a40c4a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:51.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.580+0000 7f53a8c00640 1 --2- 172.21.15.96:0/3756987104 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f53a40b4b60 0x7f53a40c4a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:51.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.580+0000 7f53a8c00640 1 -- 172.21.15.96:0/3756987104 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53a40c5170 con 0x7f53a40d04a0 2024-05-28T06:08:51.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.580+0000 7f53a8c00640 1 --2- 172.21.15.96:0/3756987104 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f53a40d04a0 0x7f53a41f7350 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f539c00c780 tx=0x7f539c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:51.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.580+0000 7f53a0c00640 1 -- 172.21.15.96:0/3756987104 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f539c004b60 con 0x7f53a40d04a0 2024-05-28T06:08:51.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.581+0000 7f53a0c00640 1 -- 172.21.15.96:0/3756987104 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f539c004d00 con 0x7f53a40d04a0 2024-05-28T06:08:51.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.581+0000 7f53a0c00640 1 -- 172.21.15.96:0/3756987104 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f539c00aa90 con 0x7f53a40d04a0 2024-05-28T06:08:51.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.581+0000 7f53aac00640 1 -- 172.21.15.96:0/3756987104 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53a40c5450 con 0x7f53a40d04a0 2024-05-28T06:08:51.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.581+0000 7f53aac00640 1 -- 172.21.15.96:0/3756987104 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53a40c5a10 con 0x7f53a40d04a0 2024-05-28T06:08:51.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.582+0000 7f53aac00640 1 -- 172.21.15.96:0/3756987104 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53a40689e0 con 0x7f53a40d04a0 2024-05-28T06:08:51.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.583+0000 7f53a0c00640 1 -- 172.21.15.96:0/3756987104 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 9) ==== 91295+0+0 (secure 0 0 0) 0x7f539c00ac30 con 0x7f53a40d04a0 2024-05-28T06:08:51.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.584+0000 7f53a0c00640 1 --2- 172.21.15.96:0/3756987104 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f538006cbc0 0x7f538006f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:51.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.584+0000 7f53a0c00640 1 -- 172.21.15.96:0/3756987104 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4885+0+0 (secure 0 0 0) 0x7f539c08bff0 con 0x7f53a40d04a0 2024-05-28T06:08:51.586 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.585+0000 7f53a3e00640 1 --2- 172.21.15.96:0/3756987104 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f538006cbc0 0x7f538006f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:51.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.586+0000 7f53a3e00640 1 --2- 172.21.15.96:0/3756987104 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f538006cbc0 0x7f538006f080 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f5394034000 tx=0x7f539402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:51.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.589+0000 7f53a0c00640 1 -- 172.21.15.96:0/3756987104 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f539c020070 con 0x7f53a40d04a0 2024-05-28T06:08:51.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.790+0000 7f53aac00640 1 -- 172.21.15.96:0/3756987104 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f53a406c0d0 con 0x7f53a40d04a0 2024-05-28T06:08:51.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.791+0000 7f53a0c00640 1 -- 172.21.15.96:0/3756987104 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v41) ==== 84+0+65 (secure 0 0 0) 0x7f539c05a3c0 con 0x7f53a40d04a0 2024-05-28T06:08:51.793 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:08:51.793 INFO:teuthology.orchestra.run.smithi096.stdout:[ 2024-05-28T06:08:51.793 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:08:51.793 INFO:teuthology.orchestra.run.smithi096.stdout: "poolnum": 1, 2024-05-28T06:08:51.793 INFO:teuthology.orchestra.run.smithi096.stdout: "poolname": ".mgr" 2024-05-28T06:08:51.793 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:08:51.793 INFO:teuthology.orchestra.run.smithi096.stdout:] 2024-05-28T06:08:51.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.793+0000 7f53aac00640 1 -- 172.21.15.96:0/3756987104 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f538006cbc0 msgr2=0x7f538006f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:51.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.793+0000 7f53aac00640 1 --2- 172.21.15.96:0/3756987104 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f538006cbc0 0x7f538006f080 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f5394034000 tx=0x7f539402e000 comp rx=0 tx=0).stop 2024-05-28T06:08:51.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.793+0000 7f53aac00640 1 -- 172.21.15.96:0/3756987104 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f53a40d04a0 msgr2=0x7f53a41f7350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:51.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.793+0000 7f53aac00640 1 --2- 172.21.15.96:0/3756987104 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f53a40d04a0 0x7f53a41f7350 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f539c00c780 tx=0x7f539c00cc40 comp rx=0 tx=0).stop 2024-05-28T06:08:51.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.793+0000 7f53aac00640 1 -- 172.21.15.96:0/3756987104 shutdown_connections 2024-05-28T06:08:51.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.793+0000 7f53aac00640 1 --2- 172.21.15.96:0/3756987104 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f538006cbc0 0x7f538006f080 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:51.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.793+0000 7f53aac00640 1 --2- 172.21.15.96:0/3756987104 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f53a40d04a0 0x7f53a41f7350 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:51.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.793+0000 7f53aac00640 1 --2- 172.21.15.96:0/3756987104 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f53a40b4b60 0x7f53a40c4a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:51.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.793+0000 7f53aac00640 1 --2- 172.21.15.96:0/3756987104 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f53a40b3460 0x7f53a40c6940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:51.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.793+0000 7f53aac00640 1 -- 172.21.15.96:0/3756987104 >> 172.21.15.96:0/3756987104 conn(0x7f53a406bae0 msgr2=0x7f53a406ec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:51.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.794+0000 7f53aac00640 1 -- 172.21.15.96:0/3756987104 shutdown_connections 2024-05-28T06:08:51.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:51.794+0000 7f53aac00640 1 -- 172.21.15.96:0/3756987104 wait complete. 2024-05-28T06:08:51.815 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-05-28T06:08:51.815 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-05-28T06:08:52.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:52.023+0000 7f9747400640 1 Processor -- start 2024-05-28T06:08:52.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:52.024+0000 7f9747400640 1 -- start start 2024-05-28T06:08:52.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:52.025+0000 7f9747400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f974015c830 0x7f97400ce230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:52.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:52.025+0000 7f9747400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f97400ce770 0x7f97400b40b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:52.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:52.025+0000 7f9747400640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f974015cc10 con 0x7f97400b45f0 2024-05-28T06:08:52.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:52.025+0000 7f9747400640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f97400b4b90 con 0x7f974015c830 2024-05-28T06:08:52.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:52.025+0000 7f9747400640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f97400b4e70 con 0x7f97400ce770 2024-05-28T06:08:52.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:52.025+0000 7f9744a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f974015c830 0x7f97400ce230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:52.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:52.025+0000 7f9745400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f97400b45f0 0x7f97400d08e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:56414/0 (socket says 172.21.15.96:56414) 2024-05-28T06:08:52.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:52.025+0000 7f9745400640 1 -- 172.21.15.96:0/2310835872 learned_addr learned my addr 172.21.15.96:0/2310835872 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:08:52.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:52.026+0000 7f9737e00640 1 --2- 172.21.15.96:0/2310835872 >> v2:172.21.15.123:3300/0 conn(0x7f97400ce770 0x7f97400b40b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:52.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:52.026+0000 7f9744a00640 1 -- 172.21.15.96:0/2310835872 >> v2:172.21.15.123:3300/0 conn(0x7f97400ce770 msgr2=0x7f97400b40b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:52.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:52.026+0000 7f9744a00640 1 --2- 172.21.15.96:0/2310835872 >> v2:172.21.15.123:3300/0 conn(0x7f97400ce770 0x7f97400b40b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:52.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:52.026+0000 7f9744a00640 1 -- 172.21.15.96:0/2310835872 >> v1:172.21.15.96:6789/0 conn(0x7f97400b45f0 legacy=0x7f97400d08e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:52.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:52.026+0000 7f9744a00640 1 -- 172.21.15.96:0/2310835872 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97400b5150 con 0x7f974015c830 2024-05-28T06:08:52.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:52.026+0000 7f973fe00640 1 -- 172.21.15.96:0/2310835872 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3792815489 0 0) 0x7f974015cc10 con 0x7f97400b45f0 2024-05-28T06:08:52.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:52.027+0000 7f9744a00640 1 --2- 172.21.15.96:0/2310835872 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f974015c830 0x7f97400ce230 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f974006dec0 tx=0x7f973002a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a89cc65b76d4e318 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:52.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:52.027+0000 7f973fe00640 1 -- 172.21.15.96:0/2310835872 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9730037070 con 0x7f974015c830 2024-05-28T06:08:52.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:52.027+0000 7f973fe00640 1 -- 172.21.15.96:0/2310835872 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9730032440 con 0x7f974015c830 2024-05-28T06:08:52.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:52.027+0000 7f973fe00640 1 -- 172.21.15.96:0/2310835872 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9730032810 con 0x7f974015c830 2024-05-28T06:08:52.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:52.028+0000 7f9747400640 1 -- 172.21.15.96:0/2310835872 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f974015c830 msgr2=0x7f97400ce230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:52.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:52.028+0000 7f9747400640 1 --2- 172.21.15.96:0/2310835872 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f974015c830 0x7f97400ce230 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f974006dec0 tx=0x7f973002a1a0 comp rx=0 tx=0).stop 2024-05-28T06:08:52.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:52.028+0000 7f9747400640 1 -- 172.21.15.96:0/2310835872 shutdown_connections 2024-05-28T06:08:52.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:52.028+0000 7f9747400640 1 --2- 172.21.15.96:0/2310835872 >> v2:172.21.15.123:3300/0 conn(0x7f97400ce770 0x7f97400b40b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:52.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:52.028+0000 7f9747400640 1 --2- 172.21.15.96:0/2310835872 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f974015c830 0x7f97400ce230 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:52.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:52.028+0000 7f9747400640 1 -- 172.21.15.96:0/2310835872 >> 172.21.15.96:0/2310835872 conn(0x7f97400b20a0 msgr2=0x7f97400b24b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:52.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:52.028+0000 7f9747400640 1 -- 172.21.15.96:0/2310835872 shutdown_connections 2024-05-28T06:08:52.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:52.028+0000 7f9747400640 1 -- 172.21.15.96:0/2310835872 wait complete. 2024-05-28T06:08:52.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:52.030+0000 7f9747400640 1 Processor -- start 2024-05-28T06:08:52.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:52.030+0000 7f9747400640 1 -- start start 2024-05-28T06:08:52.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:52.031+0000 7f9747400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f97400b45f0 0x7f97400ca980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:52.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:52.031+0000 7f9747400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97400ce770 0x7f97401581a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:52.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:52.031+0000 7f9747400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f974015c830 0x7f97400c6080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:52.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:52.031+0000 7f9747400640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f97400d0160 con 0x7f97400ce770 2024-05-28T06:08:52.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:52.031+0000 7f9747400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f97400cffe0 con 0x7f97400b45f0 2024-05-28T06:08:52.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:52.031+0000 7f9747400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f9740160480 con 0x7f974015c830 2024-05-28T06:08:52.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:52.032+0000 7f9737e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97400ce770 0x7f97401581a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:52.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:52.032+0000 7f9737e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97400ce770 0x7f97401581a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51270/0 (socket says 172.21.15.96:51270) 2024-05-28T06:08:52.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:52.032+0000 7f9737e00640 1 -- 172.21.15.96:0/720662436 learned_addr learned my addr 172.21.15.96:0/720662436 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:08:52.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:52.032+0000 7f9745400640 1 --2- 172.21.15.96:0/720662436 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f974015c830 0x7f97400c6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:52.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:52.032+0000 7f9744a00640 1 --2- 172.21.15.96:0/720662436 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f97400b45f0 0x7f97400ca980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:52.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:52.032+0000 7f9737e00640 1 -- 172.21.15.96:0/720662436 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f974015c830 msgr2=0x7f97400c6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:52.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:52.032+0000 7f9737e00640 1 --2- 172.21.15.96:0/720662436 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f974015c830 0x7f97400c6080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:52.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:52.032+0000 7f9737e00640 1 -- 172.21.15.96:0/720662436 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f97400b45f0 msgr2=0x7f97400ca980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:52.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:52.032+0000 7f9737e00640 1 --2- 172.21.15.96:0/720662436 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f97400b45f0 0x7f97400ca980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:52.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:52.032+0000 7f9737e00640 1 -- 172.21.15.96:0/720662436 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97400cb1b0 con 0x7f97400ce770 2024-05-28T06:08:52.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:52.032+0000 7f9744a00640 1 --2- 172.21.15.96:0/720662436 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f97400b45f0 0x7f97400ca980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:08:52.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:52.033+0000 7f9737e00640 1 --2- 172.21.15.96:0/720662436 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97400ce770 0x7f97401581a0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f972800d800 tx=0x7f972800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:52.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:52.033+0000 7f973d600640 1 -- 172.21.15.96:0/720662436 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f972800b6c0 con 0x7f97400ce770 2024-05-28T06:08:52.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:52.033+0000 7f973d600640 1 -- 172.21.15.96:0/720662436 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9728019070 con 0x7f97400ce770 2024-05-28T06:08:52.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:52.033+0000 7f9747400640 1 -- 172.21.15.96:0/720662436 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97400cb490 con 0x7f97400ce770 2024-05-28T06:08:52.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:52.033+0000 7f973d600640 1 -- 172.21.15.96:0/720662436 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f972800ba80 con 0x7f97400ce770 2024-05-28T06:08:52.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:52.033+0000 7f9747400640 1 -- 172.21.15.96:0/720662436 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97400cba50 con 0x7f97400ce770 2024-05-28T06:08:52.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:52.034+0000 7f9747400640 1 -- 172.21.15.96:0/720662436 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97400757d0 con 0x7f97400ce770 2024-05-28T06:08:52.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:52.035+0000 7f973d600640 1 -- 172.21.15.96:0/720662436 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 9) ==== 91295+0+0 (secure 0 0 0) 0x7f972800bc20 con 0x7f97400ce770 2024-05-28T06:08:52.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:52.036+0000 7f973d600640 1 --2- 172.21.15.96:0/720662436 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f971c06cbc0 0x7f971c06f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:52.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:52.037+0000 7f973d600640 1 -- 172.21.15.96:0/720662436 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4885+0+0 (secure 0 0 0) 0x7f972808c170 con 0x7f97400ce770 2024-05-28T06:08:52.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:52.037+0000 7f9744a00640 1 --2- 172.21.15.96:0/720662436 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f971c06cbc0 0x7f971c06f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:52.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:52.038+0000 7f9744a00640 1 --2- 172.21.15.96:0/720662436 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f971c06cbc0 0x7f971c06f080 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f9730034000 tx=0x7f973002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:52.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:52.041+0000 7f973d600640 1 -- 172.21.15.96:0/720662436 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f972805a540 con 0x7f97400ce770 2024-05-28T06:08:52.243 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:52.241+0000 7f9747400640 1 -- 172.21.15.96:0/720662436 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f97400689e0 con 0x7f97400ce770 2024-05-28T06:08:52.998 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:52.996+0000 7f973d600640 1 -- 172.21.15.96:0/720662436 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v42) ==== 138+0+0 (secure 0 0 0) 0x7f9728059e40 con 0x7f97400ce770 2024-05-28T06:08:52.998 INFO:teuthology.orchestra.run.smithi096.stderr:pool 'cephfs_metadata' created 2024-05-28T06:08:53.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:53.000+0000 7f9747400640 1 -- 172.21.15.96:0/720662436 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f971c06cbc0 msgr2=0x7f971c06f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:53.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:53.000+0000 7f9747400640 1 --2- 172.21.15.96:0/720662436 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f971c06cbc0 0x7f971c06f080 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f9730034000 tx=0x7f973002e000 comp rx=0 tx=0).stop 2024-05-28T06:08:53.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:53.000+0000 7f9747400640 1 -- 172.21.15.96:0/720662436 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97400ce770 msgr2=0x7f97401581a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:53.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:53.000+0000 7f9747400640 1 --2- 172.21.15.96:0/720662436 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97400ce770 0x7f97401581a0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f972800d800 tx=0x7f972800dcc0 comp rx=0 tx=0).stop 2024-05-28T06:08:53.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:53.000+0000 7f9747400640 1 -- 172.21.15.96:0/720662436 shutdown_connections 2024-05-28T06:08:53.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:53.000+0000 7f9747400640 1 --2- 172.21.15.96:0/720662436 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f971c06cbc0 0x7f971c06f080 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:53.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:53.000+0000 7f9747400640 1 --2- 172.21.15.96:0/720662436 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f974015c830 0x7f97400c6080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:53.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:53.000+0000 7f9747400640 1 --2- 172.21.15.96:0/720662436 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97400ce770 0x7f97401581a0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:53.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:53.000+0000 7f9747400640 1 --2- 172.21.15.96:0/720662436 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f97400b45f0 0x7f97400ca980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:53.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:53.000+0000 7f9747400640 1 -- 172.21.15.96:0/720662436 >> 172.21.15.96:0/720662436 conn(0x7f97400b20a0 msgr2=0x7f97400b0330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:53.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:53.000+0000 7f9747400640 1 -- 172.21.15.96:0/720662436 shutdown_connections 2024-05-28T06:08:53.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:53.000+0000 7f9747400640 1 -- 172.21.15.96:0/720662436 wait complete. 2024-05-28T06:08:53.030 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-05-28T06:08:53.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:53.224+0000 7f4036e00640 1 Processor -- start 2024-05-28T06:08:53.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:53.225+0000 7f4036e00640 1 -- start start 2024-05-28T06:08:53.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:53.226+0000 7f4036e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f403015b6f0 0x7f403006e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:53.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:53.226+0000 7f4036e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f40300725e0 0x7f40300ae650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:53.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:53.226+0000 7f4036e00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40300ade90 con 0x7f40300d0640 2024-05-28T06:08:53.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:53.226+0000 7f4036e00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f4030071c40 con 0x7f403015b6f0 2024-05-28T06:08:53.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:53.226+0000 7f4036e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f4030071e70 con 0x7f40300725e0 2024-05-28T06:08:53.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:53.226+0000 7f402f400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f403015b6f0 0x7f403006e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:53.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:53.227+0000 7f402f400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f403015b6f0 0x7f403006e350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51282/0 (socket says 172.21.15.96:51282) 2024-05-28T06:08:53.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:53.227+0000 7f402f400640 1 -- 172.21.15.96:0/3016215246 learned_addr learned my addr 172.21.15.96:0/3016215246 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:08:53.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:53.227+0000 7f402f400640 1 -- 172.21.15.96:0/3016215246 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f40300725e0 msgr2=0x7f40300ae650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:53.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:53.227+0000 7f4034e00640 1 --2- 172.21.15.96:0/3016215246 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f40300725e0 0x7f40300ae650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:53.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:53.227+0000 7f402f400640 1 --2- 172.21.15.96:0/3016215246 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f40300725e0 0x7f40300ae650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:53.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:53.227+0000 7f402f400640 1 -- 172.21.15.96:0/3016215246 >> v1:172.21.15.123:6789/0 conn(0x7f40300d0640 legacy=0x7f40300d0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:53.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:53.227+0000 7f402f400640 1 -- 172.21.15.96:0/3016215246 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4030072130 con 0x7f403015b6f0 2024-05-28T06:08:53.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:53.227+0000 7f4034e00640 1 --2- 172.21.15.96:0/3016215246 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f40300725e0 0x7f40300ae650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:08:53.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:53.228+0000 7f402f400640 1 --2- 172.21.15.96:0/3016215246 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f403015b6f0 0x7f403006e350 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f401c009870 tx=0x7f401c02a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f145ca60f4bfeca9 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:53.231 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:53.228+0000 7f402ea00640 1 -- 172.21.15.96:0/3016215246 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f401c037070 con 0x7f403015b6f0 2024-05-28T06:08:53.231 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:53.228+0000 7f402ea00640 1 -- 172.21.15.96:0/3016215246 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f401c032440 con 0x7f403015b6f0 2024-05-28T06:08:53.231 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:53.228+0000 7f402ea00640 1 -- 172.21.15.96:0/3016215246 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f401c032810 con 0x7f403015b6f0 2024-05-28T06:08:53.231 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:53.229+0000 7f4036e00640 1 -- 172.21.15.96:0/3016215246 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f403015b6f0 msgr2=0x7f403006e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:53.231 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:53.229+0000 7f4036e00640 1 --2- 172.21.15.96:0/3016215246 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f403015b6f0 0x7f403006e350 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f401c009870 tx=0x7f401c02a210 comp rx=0 tx=0).stop 2024-05-28T06:08:53.231 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:53.229+0000 7f4036e00640 1 -- 172.21.15.96:0/3016215246 shutdown_connections 2024-05-28T06:08:53.231 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:53.229+0000 7f4036e00640 1 --2- 172.21.15.96:0/3016215246 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f40300725e0 0x7f40300ae650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:53.231 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:53.229+0000 7f4036e00640 1 --2- 172.21.15.96:0/3016215246 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f403015b6f0 0x7f403006e350 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:53.231 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:53.229+0000 7f4036e00640 1 -- 172.21.15.96:0/3016215246 >> 172.21.15.96:0/3016215246 conn(0x7f403006bd20 msgr2=0x7f403006c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:53.231 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:53.229+0000 7f4036e00640 1 -- 172.21.15.96:0/3016215246 shutdown_connections 2024-05-28T06:08:53.231 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:53.229+0000 7f4036e00640 1 -- 172.21.15.96:0/3016215246 wait complete. 2024-05-28T06:08:53.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:53.231+0000 7f4036e00640 1 Processor -- start 2024-05-28T06:08:53.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:53.231+0000 7f4036e00640 1 -- start start 2024-05-28T06:08:53.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:53.232+0000 7f4036e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40300725e0 0x7f40300999f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:53.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:53.232+0000 7f4036e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40300d0640 0x7f403015e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:53.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:53.232+0000 7f4036e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f403015b6f0 0x7f4030157ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:53.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:53.232+0000 7f4036e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f40300aeb90 con 0x7f40300d0640 2024-05-28T06:08:53.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:53.232+0000 7f4036e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f403006e890 con 0x7f40300725e0 2024-05-28T06:08:53.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:53.232+0000 7f4036e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f40300aed10 con 0x7f403015b6f0 2024-05-28T06:08:53.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:53.232+0000 7f402f400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40300d0640 0x7f403015e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:53.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:53.232+0000 7f402f400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40300d0640 0x7f403015e490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51284/0 (socket says 172.21.15.96:51284) 2024-05-28T06:08:53.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:53.232+0000 7f402f400640 1 -- 172.21.15.96:0/323971714 learned_addr learned my addr 172.21.15.96:0/323971714 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:08:53.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:53.233+0000 7f402f400640 1 -- 172.21.15.96:0/323971714 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f403015b6f0 msgr2=0x7f4030157ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:53.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:53.233+0000 7f402f400640 1 --2- 172.21.15.96:0/323971714 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f403015b6f0 0x7f4030157ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:53.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:53.233+0000 7f402f400640 1 -- 172.21.15.96:0/323971714 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40300725e0 msgr2=0x7f40300999f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:53.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:53.233+0000 7f402f400640 1 --2- 172.21.15.96:0/323971714 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40300725e0 0x7f40300999f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:53.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:53.233+0000 7f402f400640 1 -- 172.21.15.96:0/323971714 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4030156e30 con 0x7f40300d0640 2024-05-28T06:08:53.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:53.233+0000 7f402f400640 1 --2- 172.21.15.96:0/323971714 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40300d0640 0x7f403015e490 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f401c0090b0 tx=0x7f401c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:53.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:53.233+0000 7f4027e00640 1 -- 172.21.15.96:0/323971714 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f401c041020 con 0x7f40300d0640 2024-05-28T06:08:53.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:53.233+0000 7f4027e00640 1 -- 172.21.15.96:0/323971714 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f401c030990 con 0x7f40300d0640 2024-05-28T06:08:53.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:53.233+0000 7f4027e00640 1 -- 172.21.15.96:0/323971714 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f401c037040 con 0x7f40300d0640 2024-05-28T06:08:53.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:53.234+0000 7f4036e00640 1 -- 172.21.15.96:0/323971714 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4030157110 con 0x7f40300d0640 2024-05-28T06:08:53.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:53.234+0000 7f4036e00640 1 -- 172.21.15.96:0/323971714 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40301576d0 con 0x7f40300d0640 2024-05-28T06:08:53.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:53.235+0000 7f4036e00640 1 -- 172.21.15.96:0/323971714 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40300689e0 con 0x7f40300d0640 2024-05-28T06:08:53.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:53.236+0000 7f4027e00640 1 -- 172.21.15.96:0/323971714 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 9) ==== 91295+0+0 (secure 0 0 0) 0x7f401c044020 con 0x7f40300d0640 2024-05-28T06:08:53.238 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:53.237+0000 7f4027e00640 1 --2- 172.21.15.96:0/323971714 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f401006cbc0 0x7f401006f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:53.238 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:53.237+0000 7f4027e00640 1 -- 172.21.15.96:0/323971714 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 5255+0+0 (secure 0 0 0) 0x7f401c0ac850 con 0x7f40300d0640 2024-05-28T06:08:53.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:53.238+0000 7f402fe00640 1 --2- 172.21.15.96:0/323971714 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f401006cbc0 0x7f401006f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:53.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:53.239+0000 7f402fe00640 1 --2- 172.21.15.96:0/323971714 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f401006cbc0 0x7f401006f080 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f403009e530 tx=0x7f40200079b0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:53.244 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:53.243+0000 7f4027e00640 1 -- 172.21.15.96:0/323971714 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f401c04b030 con 0x7f40300d0640 2024-05-28T06:08:53.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:53.444+0000 7f4036e00640 1 -- 172.21.15.96:0/323971714 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f403006c3b0 con 0x7f40300d0640 2024-05-28T06:08:54.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:54.004+0000 7f4027e00640 1 -- 172.21.15.96:0/323971714 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v43) ==== 170+0+0 (secure 0 0 0) 0x7f401c07aab0 con 0x7f40300d0640 2024-05-28T06:08:54.006 INFO:teuthology.orchestra.run.smithi096.stderr:pool 'cephfs_data' created 2024-05-28T06:08:54.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:54.008+0000 7f4036e00640 1 -- 172.21.15.96:0/323971714 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f401006cbc0 msgr2=0x7f401006f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:54.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:54.008+0000 7f4036e00640 1 --2- 172.21.15.96:0/323971714 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f401006cbc0 0x7f401006f080 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f403009e530 tx=0x7f40200079b0 comp rx=0 tx=0).stop 2024-05-28T06:08:54.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:54.008+0000 7f4036e00640 1 -- 172.21.15.96:0/323971714 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40300d0640 msgr2=0x7f403015e490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:54.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:54.008+0000 7f4036e00640 1 --2- 172.21.15.96:0/323971714 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40300d0640 0x7f403015e490 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f401c0090b0 tx=0x7f401c0046c0 comp rx=0 tx=0).stop 2024-05-28T06:08:54.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:54.008+0000 7f4036e00640 1 -- 172.21.15.96:0/323971714 shutdown_connections 2024-05-28T06:08:54.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:54.008+0000 7f4036e00640 1 --2- 172.21.15.96:0/323971714 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f401006cbc0 0x7f401006f080 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:54.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:54.009+0000 7f4036e00640 1 --2- 172.21.15.96:0/323971714 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f403015b6f0 0x7f4030157ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:54.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:54.009+0000 7f4036e00640 1 --2- 172.21.15.96:0/323971714 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40300d0640 0x7f403015e490 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:54.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:54.009+0000 7f4036e00640 1 --2- 172.21.15.96:0/323971714 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40300725e0 0x7f40300999f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:54.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:54.009+0000 7f4036e00640 1 -- 172.21.15.96:0/323971714 >> 172.21.15.96:0/323971714 conn(0x7f403006bd20 msgr2=0x7f40300afea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:54.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:54.009+0000 7f4036e00640 1 -- 172.21.15.96:0/323971714 shutdown_connections 2024-05-28T06:08:54.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:54.009+0000 7f4036e00640 1 -- 172.21.15.96:0/323971714 wait complete. 2024-05-28T06:08:54.042 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-05-28T06:08:54.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:54.217+0000 7f5e18e00640 1 Processor -- start 2024-05-28T06:08:54.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:54.217+0000 7f5e18e00640 1 -- start start 2024-05-28T06:08:54.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:54.218+0000 7f5e18e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5e14162480 0x7f5e14160280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:54.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:54.218+0000 7f5e18e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f5e1409b5d0 0x7f5e1415f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:54.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:54.218+0000 7f5e18e00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e1406d1f0 con 0x7f5e1406ca10 2024-05-28T06:08:54.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:54.219+0000 7f5e18e00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f5e1406d4d0 con 0x7f5e1409b5d0 2024-05-28T06:08:54.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:54.219+0000 7f5e18e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f5e1406d7b0 con 0x7f5e14162480 2024-05-28T06:08:54.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:54.219+0000 7f5e12800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f5e1409b5d0 0x7f5e1415f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:54.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:54.219+0000 7f5e11400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f5e1406ca10 0x7f5e14162860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:56428/0 (socket says 172.21.15.96:56428) 2024-05-28T06:08:54.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:54.219+0000 7f5e11400640 1 -- 172.21.15.96:0/1227313202 learned_addr learned my addr 172.21.15.96:0/1227313202 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:08:54.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:54.219+0000 7f5e11e00640 1 --2- 172.21.15.96:0/1227313202 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5e14162480 0x7f5e14160280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:54.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:54.219+0000 7f5e12800640 1 -- 172.21.15.96:0/1227313202 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5e14162480 msgr2=0x7f5e14160280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:54.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:54.219+0000 7f5e12800640 1 --2- 172.21.15.96:0/1227313202 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5e14162480 0x7f5e14160280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:54.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:54.219+0000 7f5e12800640 1 -- 172.21.15.96:0/1227313202 >> v1:172.21.15.96:6789/0 conn(0x7f5e1406ca10 legacy=0x7f5e14162860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:54.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:54.219+0000 7f5e12800640 1 -- 172.21.15.96:0/1227313202 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e1406da90 con 0x7f5e1409b5d0 2024-05-28T06:08:54.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:54.219+0000 7f5e11e00640 1 --2- 172.21.15.96:0/1227313202 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5e14162480 0x7f5e14160280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:08:54.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:54.220+0000 7f5e12800640 1 --2- 172.21.15.96:0/1227313202 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5e1409b5d0 0x7f5e1415f0b0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f5e0c009dc0 tx=0x7f5e0c02a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb234e0aa926fd54 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:54.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:54.221+0000 7f5e10a00640 1 -- 172.21.15.96:0/1227313202 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e0c02ae60 con 0x7f5e1409b5d0 2024-05-28T06:08:54.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:54.221+0000 7f5e10a00640 1 -- 172.21.15.96:0/1227313202 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e0c007740 con 0x7f5e1409b5d0 2024-05-28T06:08:54.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:54.221+0000 7f5e10a00640 1 -- 172.21.15.96:0/1227313202 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e0c007b10 con 0x7f5e1409b5d0 2024-05-28T06:08:54.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:54.221+0000 7f5e18e00640 1 -- 172.21.15.96:0/1227313202 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5e1409b5d0 msgr2=0x7f5e1415f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:54.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:54.221+0000 7f5e18e00640 1 --2- 172.21.15.96:0/1227313202 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5e1409b5d0 0x7f5e1415f0b0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f5e0c009dc0 tx=0x7f5e0c02a680 comp rx=0 tx=0).stop 2024-05-28T06:08:54.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:54.221+0000 7f5e18e00640 1 -- 172.21.15.96:0/1227313202 shutdown_connections 2024-05-28T06:08:54.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:54.221+0000 7f5e18e00640 1 --2- 172.21.15.96:0/1227313202 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5e1409b5d0 0x7f5e1415f0b0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:54.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:54.221+0000 7f5e18e00640 1 --2- 172.21.15.96:0/1227313202 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5e14162480 0x7f5e14160280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:54.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:54.221+0000 7f5e18e00640 1 -- 172.21.15.96:0/1227313202 >> 172.21.15.96:0/1227313202 conn(0x7f5e140d2430 msgr2=0x7f5e140d2840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:54.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:54.222+0000 7f5e18e00640 1 -- 172.21.15.96:0/1227313202 shutdown_connections 2024-05-28T06:08:54.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:54.222+0000 7f5e18e00640 1 -- 172.21.15.96:0/1227313202 wait complete. 2024-05-28T06:08:54.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:54.223+0000 7f5e18e00640 1 Processor -- start 2024-05-28T06:08:54.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:54.223+0000 7f5e18e00640 1 -- start start 2024-05-28T06:08:54.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:54.224+0000 7f5e18e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5e1406ca10 0x7f5e141509d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:54.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:54.224+0000 7f5e18e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5e1409b5d0 0x7f5e140b7430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:54.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:54.224+0000 7f5e18e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e14162480 0x7f5e1409de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:54.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:54.224+0000 7f5e18e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f5e1406ce70 con 0x7f5e1406ca10 2024-05-28T06:08:54.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:54.224+0000 7f5e18e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5e1409ba60 con 0x7f5e14162480 2024-05-28T06:08:54.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:54.224+0000 7f5e18e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f5e1415f5f0 con 0x7f5e1409b5d0 2024-05-28T06:08:54.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:54.225+0000 7f5e11e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5e1406ca10 0x7f5e141509d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:54.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:54.225+0000 7f5e11e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5e1406ca10 0x7f5e141509d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51294/0 (socket says 172.21.15.96:51294) 2024-05-28T06:08:54.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:54.225+0000 7f5e11e00640 1 -- 172.21.15.96:0/353501434 learned_addr learned my addr 172.21.15.96:0/353501434 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:08:54.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:54.225+0000 7f5e11e00640 1 -- 172.21.15.96:0/353501434 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5e1409b5d0 msgr2=0x7f5e140b7430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:54.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:54.225+0000 7f5e11e00640 1 --2- 172.21.15.96:0/353501434 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5e1409b5d0 0x7f5e140b7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:54.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:54.225+0000 7f5e11e00640 1 -- 172.21.15.96:0/353501434 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e14162480 msgr2=0x7f5e1409de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:54.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:54.225+0000 7f5e11e00640 1 --2- 172.21.15.96:0/353501434 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e14162480 0x7f5e1409de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:54.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:54.225+0000 7f5e11e00640 1 -- 172.21.15.96:0/353501434 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e1409cda0 con 0x7f5e1406ca10 2024-05-28T06:08:54.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:54.225+0000 7f5e11e00640 1 --2- 172.21.15.96:0/353501434 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5e1406ca10 0x7f5e141509d0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f5e0400c820 tx=0x7f5e0400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:54.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:54.226+0000 7f5e0a000640 1 -- 172.21.15.96:0/353501434 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e04015070 con 0x7f5e1406ca10 2024-05-28T06:08:54.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:54.226+0000 7f5e0a000640 1 -- 172.21.15.96:0/353501434 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e04007a30 con 0x7f5e1406ca10 2024-05-28T06:08:54.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:54.226+0000 7f5e0a000640 1 -- 172.21.15.96:0/353501434 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e04007de0 con 0x7f5e1406ca10 2024-05-28T06:08:54.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:54.226+0000 7f5e18e00640 1 -- 172.21.15.96:0/353501434 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e1409d080 con 0x7f5e1406ca10 2024-05-28T06:08:54.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:54.226+0000 7f5e18e00640 1 -- 172.21.15.96:0/353501434 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e1409d640 con 0x7f5e1406ca10 2024-05-28T06:08:54.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:54.227+0000 7f5e18e00640 1 -- 172.21.15.96:0/353501434 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e140689e0 con 0x7f5e1406ca10 2024-05-28T06:08:54.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:54.228+0000 7f5e0a000640 1 -- 172.21.15.96:0/353501434 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 9) ==== 91295+0+0 (secure 0 0 0) 0x7f5e04018070 con 0x7f5e1406ca10 2024-05-28T06:08:54.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:54.229+0000 7f5e0a000640 1 --2- 172.21.15.96:0/353501434 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5df006cbc0 0x7f5df006f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:54.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:54.229+0000 7f5e0a000640 1 -- 172.21.15.96:0/353501434 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 5637+0+0 (secure 0 0 0) 0x7f5e0408bf40 con 0x7f5e1406ca10 2024-05-28T06:08:54.231 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:54.230+0000 7f5e11400640 1 --2- 172.21.15.96:0/353501434 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5df006cbc0 0x7f5df006f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:54.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:54.231+0000 7f5e11400640 1 --2- 172.21.15.96:0/353501434 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5df006cbc0 0x7f5df006f080 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f5dfc000f60 tx=0x7f5dfc007a80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:54.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:54.234+0000 7f5e0a000640 1 -- 172.21.15.96:0/353501434 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5e04059f10 con 0x7f5e1406ca10 2024-05-28T06:08:54.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:54.475+0000 7f5e18e00640 1 -- 172.21.15.96:0/353501434 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f5e140d3020 con 0x7f5e1406ca10 2024-05-28T06:08:55.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.025+0000 7f5e0a000640 1 -- 172.21.15.96:0/353501434 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 6 and data pool 7 v21) ==== 172+0+0 (secure 0 0 0) 0x7f5e04059810 con 0x7f5e1406ca10 2024-05-28T06:08:55.027 INFO:teuthology.orchestra.run.smithi096.stderr:new fs with metadata pool 6 and data pool 7 2024-05-28T06:08:55.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.028+0000 7f5e18e00640 1 -- 172.21.15.96:0/353501434 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5df006cbc0 msgr2=0x7f5df006f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:55.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.028+0000 7f5e18e00640 1 --2- 172.21.15.96:0/353501434 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5df006cbc0 0x7f5df006f080 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f5dfc000f60 tx=0x7f5dfc007a80 comp rx=0 tx=0).stop 2024-05-28T06:08:55.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.028+0000 7f5e18e00640 1 -- 172.21.15.96:0/353501434 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5e1406ca10 msgr2=0x7f5e141509d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:55.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.028+0000 7f5e18e00640 1 --2- 172.21.15.96:0/353501434 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5e1406ca10 0x7f5e141509d0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f5e0400c820 tx=0x7f5e0400cce0 comp rx=0 tx=0).stop 2024-05-28T06:08:55.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.028+0000 7f5e18e00640 1 -- 172.21.15.96:0/353501434 shutdown_connections 2024-05-28T06:08:55.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.028+0000 7f5e18e00640 1 --2- 172.21.15.96:0/353501434 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5df006cbc0 0x7f5df006f080 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:55.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.028+0000 7f5e18e00640 1 --2- 172.21.15.96:0/353501434 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e14162480 0x7f5e1409de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:55.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.028+0000 7f5e18e00640 1 --2- 172.21.15.96:0/353501434 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5e1409b5d0 0x7f5e140b7430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:55.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.028+0000 7f5e18e00640 1 --2- 172.21.15.96:0/353501434 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5e1406ca10 0x7f5e141509d0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:55.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.028+0000 7f5e18e00640 1 -- 172.21.15.96:0/353501434 >> 172.21.15.96:0/353501434 conn(0x7f5e140d2430 msgr2=0x7f5e1415ebd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:55.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.029+0000 7f5e18e00640 1 -- 172.21.15.96:0/353501434 shutdown_connections 2024-05-28T06:08:55.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.029+0000 7f5e18e00640 1 -- 172.21.15.96:0/353501434 wait complete. 2024-05-28T06:08:55.075 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:08:55.243 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.242+0000 7fdf04600640 1 Processor -- start 2024-05-28T06:08:55.244 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.242+0000 7fdf04600640 1 -- start start 2024-05-28T06:08:55.245 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.243+0000 7fdf04600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fdefc0b4e10 0x7fdefc0cde30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:55.245 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.243+0000 7fdf04600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdefc0ce370 0x7fdefc0d0590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:55.245 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.243+0000 7fdf04600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdefc0b51f0 con 0x7fdefc0d0ad0 2024-05-28T06:08:55.245 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.243+0000 7fdf04600640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fdefc0d0f20 con 0x7fdefc0b4e10 2024-05-28T06:08:55.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.243+0000 7fdf04600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fdefc0aef60 con 0x7fdefc0ce370 2024-05-28T06:08:55.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.243+0000 7fdf01c00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fdefc0b4e10 0x7fdefc0cde30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:55.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.243+0000 7fdf01c00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fdefc0b4e10 0x7fdefc0cde30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51298/0 (socket says 172.21.15.96:51298) 2024-05-28T06:08:55.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.244+0000 7fdf01c00640 1 -- 172.21.15.96:0/184189279 learned_addr learned my addr 172.21.15.96:0/184189279 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:08:55.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.244+0000 7fdf01c00640 1 -- 172.21.15.96:0/184189279 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdefc0ce370 msgr2=0x7fdefc0d0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:55.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.244+0000 7fdf01200640 1 --2- 172.21.15.96:0/184189279 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdefc0ce370 0x7fdefc0d0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:55.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.244+0000 7fdf01c00640 1 --2- 172.21.15.96:0/184189279 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdefc0ce370 0x7fdefc0d0590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:55.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.244+0000 7fdf01c00640 1 -- 172.21.15.96:0/184189279 >> v1:172.21.15.123:6789/0 conn(0x7fdefc0d0ad0 legacy=0x7fdefc099090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:55.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.244+0000 7fdf01c00640 1 -- 172.21.15.96:0/184189279 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdefc0ad1d0 con 0x7fdefc0b4e10 2024-05-28T06:08:55.247 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.245+0000 7fdf01c00640 1 --2- 172.21.15.96:0/184189279 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdefc0b4e10 0x7fdefc0cde30 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fdef000bb10 tx=0x7fdef002c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52be98fcf3630b9b server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:55.247 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.245+0000 7fdefbe00640 1 -- 172.21.15.96:0/184189279 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdef003a070 con 0x7fdefc0b4e10 2024-05-28T06:08:55.247 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.245+0000 7fdefbe00640 1 -- 172.21.15.96:0/184189279 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdef00326f0 con 0x7fdefc0b4e10 2024-05-28T06:08:55.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.245+0000 7fdefbe00640 1 -- 172.21.15.96:0/184189279 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdef0032ac0 con 0x7fdefc0b4e10 2024-05-28T06:08:55.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.246+0000 7fdf04600640 1 -- 172.21.15.96:0/184189279 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdefc0b4e10 msgr2=0x7fdefc0cde30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:55.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.246+0000 7fdf04600640 1 --2- 172.21.15.96:0/184189279 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdefc0b4e10 0x7fdefc0cde30 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fdef000bb10 tx=0x7fdef002c830 comp rx=0 tx=0).stop 2024-05-28T06:08:55.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.246+0000 7fdf04600640 1 -- 172.21.15.96:0/184189279 shutdown_connections 2024-05-28T06:08:55.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.246+0000 7fdf04600640 1 --2- 172.21.15.96:0/184189279 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdefc0ce370 0x7fdefc0d0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:55.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.246+0000 7fdf04600640 1 --2- 172.21.15.96:0/184189279 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdefc0b4e10 0x7fdefc0cde30 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:55.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.246+0000 7fdf04600640 1 -- 172.21.15.96:0/184189279 >> 172.21.15.96:0/184189279 conn(0x7fdefc06bae0 msgr2=0x7fdefc06bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:55.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.246+0000 7fdf04600640 1 -- 172.21.15.96:0/184189279 shutdown_connections 2024-05-28T06:08:55.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.246+0000 7fdf04600640 1 -- 172.21.15.96:0/184189279 wait complete. 2024-05-28T06:08:55.249 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.248+0000 7fdf04600640 1 Processor -- start 2024-05-28T06:08:55.249 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.248+0000 7fdf04600640 1 -- start start 2024-05-28T06:08:55.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.249+0000 7fdf04600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdefc0b4e10 0x7fdefc14f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:55.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.249+0000 7fdf04600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdefc0ce370 0x7fdefc1f3e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:55.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.249+0000 7fdf04600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdefc0d0ad0 0x7fdefc1e3b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:55.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.249+0000 7fdf04600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fdefc0ce880 con 0x7fdefc0b4e10 2024-05-28T06:08:55.252 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.249+0000 7fdf04600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fdefc0ad750 con 0x7fdefc0d0ad0 2024-05-28T06:08:55.252 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.249+0000 7fdf04600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fdefc0cea00 con 0x7fdefc0ce370 2024-05-28T06:08:55.252 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.249+0000 7fdf01c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdefc0b4e10 0x7fdefc14f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:55.252 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.249+0000 7fdf01c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdefc0b4e10 0x7fdefc14f9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51302/0 (socket says 172.21.15.96:51302) 2024-05-28T06:08:55.252 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.249+0000 7fdf01c00640 1 -- 172.21.15.96:0/1986833316 learned_addr learned my addr 172.21.15.96:0/1986833316 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:08:55.252 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.249+0000 7fdf01c00640 1 -- 172.21.15.96:0/1986833316 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdefc0ce370 msgr2=0x7fdefc1f3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:55.252 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.249+0000 7fdf01c00640 1 --2- 172.21.15.96:0/1986833316 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdefc0ce370 0x7fdefc1f3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:55.252 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.249+0000 7fdf01c00640 1 -- 172.21.15.96:0/1986833316 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdefc0d0ad0 msgr2=0x7fdefc1e3b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:55.252 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.249+0000 7fdf01c00640 1 --2- 172.21.15.96:0/1986833316 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdefc0d0ad0 0x7fdefc1e3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:55.252 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.249+0000 7fdf01c00640 1 -- 172.21.15.96:0/1986833316 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdefc0b10c0 con 0x7fdefc0b4e10 2024-05-28T06:08:55.252 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.250+0000 7fdf01c00640 1 --2- 172.21.15.96:0/1986833316 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdefc0b4e10 0x7fdefc14f9a0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fdef002c870 tx=0x7fdef0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:55.252 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.250+0000 7fdef9600640 1 -- 172.21.15.96:0/1986833316 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdef0045070 con 0x7fdefc0b4e10 2024-05-28T06:08:55.253 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.250+0000 7fdef9600640 1 -- 172.21.15.96:0/1986833316 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdef0035890 con 0x7fdefc0b4e10 2024-05-28T06:08:55.253 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.250+0000 7fdf04600640 1 -- 172.21.15.96:0/1986833316 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdefc0b13a0 con 0x7fdefc0b4e10 2024-05-28T06:08:55.253 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.250+0000 7fdef9600640 1 -- 172.21.15.96:0/1986833316 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdef003a070 con 0x7fdefc0b4e10 2024-05-28T06:08:55.253 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.250+0000 7fdf04600640 1 -- 172.21.15.96:0/1986833316 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdefc0b1960 con 0x7fdefc0b4e10 2024-05-28T06:08:55.253 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.251+0000 7fdf04600640 1 -- 172.21.15.96:0/1986833316 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdefc0732b0 con 0x7fdefc0b4e10 2024-05-28T06:08:55.253 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.252+0000 7fdef9600640 1 -- 172.21.15.96:0/1986833316 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 9) ==== 91295+0+0 (secure 0 0 0) 0x7fdef0048070 con 0x7fdefc0b4e10 2024-05-28T06:08:55.254 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.253+0000 7fdef9600640 1 --2- 172.21.15.96:0/1986833316 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdedc06cbc0 0x7fdedc06f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:55.254 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.253+0000 7fdef9600640 1 -- 172.21.15.96:0/1986833316 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5737+0+0 (secure 0 0 0) 0x7fdef00ac4a0 con 0x7fdefc0b4e10 2024-05-28T06:08:55.255 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.254+0000 7fdf01200640 1 --2- 172.21.15.96:0/1986833316 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdedc06cbc0 0x7fdedc06f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:55.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.255+0000 7fdf01200640 1 --2- 172.21.15.96:0/1986833316 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdedc06cbc0 0x7fdedc06f080 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fdee8006fd0 tx=0x7fdee8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:55.259 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.258+0000 7fdef9600640 1 -- 172.21.15.96:0/1986833316 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fdefc0732b0 con 0x7fdefc0b4e10 2024-05-28T06:08:55.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.442+0000 7fdf04600640 1 -- 172.21.15.96:0/1986833316 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdefc0689e0 con 0x7fdefc0b4e10 2024-05-28T06:08:55.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.446+0000 7fdef9600640 1 -- 172.21.15.96:0/1986833316 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v44) ==== 74+0+16681 (secure 0 0 0) 0x7fdef007a410 con 0x7fdefc0b4e10 2024-05-28T06:08:55.448 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:08:55.448 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":44,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:08:54.477715+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-05-28T06:08:52.257477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-05-28T06:08:53.459376+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:08:55.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.448+0000 7fdf04600640 1 -- 172.21.15.96:0/1986833316 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdedc06cbc0 msgr2=0x7fdedc06f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:55.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.448+0000 7fdf04600640 1 --2- 172.21.15.96:0/1986833316 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdedc06cbc0 0x7fdedc06f080 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fdee8006fd0 tx=0x7fdee8008040 comp rx=0 tx=0).stop 2024-05-28T06:08:55.451 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.448+0000 7fdf04600640 1 -- 172.21.15.96:0/1986833316 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdefc0b4e10 msgr2=0x7fdefc14f9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:55.451 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.448+0000 7fdf04600640 1 --2- 172.21.15.96:0/1986833316 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdefc0b4e10 0x7fdefc14f9a0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fdef002c870 tx=0x7fdef0004730 comp rx=0 tx=0).stop 2024-05-28T06:08:55.451 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.448+0000 7fdf04600640 1 -- 172.21.15.96:0/1986833316 shutdown_connections 2024-05-28T06:08:55.451 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.448+0000 7fdf04600640 1 --2- 172.21.15.96:0/1986833316 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdedc06cbc0 0x7fdedc06f080 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:55.451 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.448+0000 7fdf04600640 1 --2- 172.21.15.96:0/1986833316 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdefc0d0ad0 0x7fdefc1e3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:55.451 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.448+0000 7fdf04600640 1 --2- 172.21.15.96:0/1986833316 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdefc0ce370 0x7fdefc1f3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:55.452 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.448+0000 7fdf04600640 1 --2- 172.21.15.96:0/1986833316 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdefc0b4e10 0x7fdefc14f9a0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:55.452 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.448+0000 7fdf04600640 1 -- 172.21.15.96:0/1986833316 >> 172.21.15.96:0/1986833316 conn(0x7fdefc06bae0 msgr2=0x7fdefc0b3150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:55.452 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.448+0000 7fdf04600640 1 -- 172.21.15.96:0/1986833316 shutdown_connections 2024-05-28T06:08:55.452 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.449+0000 7fdf04600640 1 -- 172.21.15.96:0/1986833316 wait complete. 2024-05-28T06:08:55.468 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:08:55.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.651+0000 7f88a6c00640 1 Processor -- start 2024-05-28T06:08:55.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.652+0000 7f88a6c00640 1 -- start start 2024-05-28T06:08:55.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.653+0000 7f88a6c00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f88a015fec0 0x7f88a00b4b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:55.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.653+0000 7f88a6c00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f88a00b2800 0x7f88a00b33b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:55.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.653+0000 7f88a6c00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88a00b1d40 con 0x7f88a00b2c90 2024-05-28T06:08:55.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.653+0000 7f88a6c00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f88a00d0fe0 con 0x7f88a015fec0 2024-05-28T06:08:55.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.653+0000 7f88a6c00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f88a00d12c0 con 0x7f88a00b2800 2024-05-28T06:08:55.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.653+0000 7f88a4c00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f88a00b2c90 0x7f88a00cd040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:56438/0 (socket says 172.21.15.96:56438) 2024-05-28T06:08:55.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.653+0000 7f889fe00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f88a015fec0 0x7f88a00b4b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:55.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.653+0000 7f88a4c00640 1 -- 172.21.15.96:0/4276930064 learned_addr learned my addr 172.21.15.96:0/4276930064 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:08:55.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.654+0000 7f889f400640 1 --2- 172.21.15.96:0/4276930064 >> v2:172.21.15.123:3300/0 conn(0x7f88a00b2800 0x7f88a00b33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:55.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.654+0000 7f889ea00640 1 -- 172.21.15.96:0/4276930064 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 892735428 0 0) 0x7f88a00b1d40 con 0x7f88a00b2c90 2024-05-28T06:08:55.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.654+0000 7f889fe00640 1 -- 172.21.15.96:0/4276930064 >> v2:172.21.15.123:3300/0 conn(0x7f88a00b2800 msgr2=0x7f88a00b33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:55.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.654+0000 7f889fe00640 1 --2- 172.21.15.96:0/4276930064 >> v2:172.21.15.123:3300/0 conn(0x7f88a00b2800 0x7f88a00b33b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:55.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.654+0000 7f889fe00640 1 -- 172.21.15.96:0/4276930064 >> v1:172.21.15.96:6789/0 conn(0x7f88a00b2c90 legacy=0x7f88a00cd040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:55.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.654+0000 7f889fe00640 1 -- 172.21.15.96:0/4276930064 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88a00b2000 con 0x7f88a015fec0 2024-05-28T06:08:55.657 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.655+0000 7f889fe00640 1 --2- 172.21.15.96:0/4276930064 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f88a015fec0 0x7f88a00b4b40 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f88a00734c0 tx=0x7f889002a080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6daadf9823d122ac server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:55.657 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.655+0000 7f889ea00640 1 -- 172.21.15.96:0/4276930064 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8890037070 con 0x7f88a015fec0 2024-05-28T06:08:55.657 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.655+0000 7f889ea00640 1 -- 172.21.15.96:0/4276930064 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f889002add0 con 0x7f88a015fec0 2024-05-28T06:08:55.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.655+0000 7f889ea00640 1 -- 172.21.15.96:0/4276930064 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8890032660 con 0x7f88a015fec0 2024-05-28T06:08:55.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.656+0000 7f88a6c00640 1 -- 172.21.15.96:0/4276930064 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f88a015fec0 msgr2=0x7f88a00b4b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:55.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.656+0000 7f88a6c00640 1 --2- 172.21.15.96:0/4276930064 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f88a015fec0 0x7f88a00b4b40 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f88a00734c0 tx=0x7f889002a080 comp rx=0 tx=0).stop 2024-05-28T06:08:55.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.656+0000 7f88a6c00640 1 -- 172.21.15.96:0/4276930064 shutdown_connections 2024-05-28T06:08:55.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.656+0000 7f88a6c00640 1 --2- 172.21.15.96:0/4276930064 >> v2:172.21.15.123:3300/0 conn(0x7f88a00b2800 0x7f88a00b33b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:55.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.656+0000 7f88a6c00640 1 --2- 172.21.15.96:0/4276930064 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f88a015fec0 0x7f88a00b4b40 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:55.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.656+0000 7f88a6c00640 1 -- 172.21.15.96:0/4276930064 >> 172.21.15.96:0/4276930064 conn(0x7f88a00b4020 msgr2=0x7f88a00b4430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:55.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.656+0000 7f88a6c00640 1 -- 172.21.15.96:0/4276930064 shutdown_connections 2024-05-28T06:08:55.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.656+0000 7f88a6c00640 1 -- 172.21.15.96:0/4276930064 wait complete. 2024-05-28T06:08:55.659 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.658+0000 7f88a6c00640 1 Processor -- start 2024-05-28T06:08:55.659 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.658+0000 7f88a6c00640 1 -- start start 2024-05-28T06:08:55.661 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.659+0000 7f88a6c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f88a00b2800 0x7f88a01539e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:55.661 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.659+0000 7f88a6c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f88a00b2c90 0x7f88a01f2280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:55.661 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.659+0000 7f88a6c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f88a015fec0 0x7f88a00af310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:55.661 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.659+0000 7f88a6c00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f88a00cd8d0 con 0x7f88a00b2c90 2024-05-28T06:08:55.661 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.659+0000 7f88a6c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f88a00cd750 con 0x7f88a00b2800 2024-05-28T06:08:55.661 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.659+0000 7f88a6c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f88a00cda50 con 0x7f88a015fec0 2024-05-28T06:08:55.661 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.659+0000 7f889f400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f88a00b2c90 0x7f88a01f2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:55.661 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.659+0000 7f889f400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f88a00b2c90 0x7f88a01f2280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51324/0 (socket says 172.21.15.96:51324) 2024-05-28T06:08:55.661 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.659+0000 7f889f400640 1 -- 172.21.15.96:0/2081988627 learned_addr learned my addr 172.21.15.96:0/2081988627 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:08:55.661 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.659+0000 7f889fe00640 1 --2- 172.21.15.96:0/2081988627 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f88a00b2800 0x7f88a01539e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:55.661 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.659+0000 7f889f400640 1 -- 172.21.15.96:0/2081988627 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f88a015fec0 msgr2=0x7f88a00af310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:55.661 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.659+0000 7f889f400640 1 --2- 172.21.15.96:0/2081988627 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f88a015fec0 0x7f88a00af310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:55.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.659+0000 7f889f400640 1 -- 172.21.15.96:0/2081988627 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f88a00b2800 msgr2=0x7f88a01539e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:55.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.659+0000 7f889f400640 1 --2- 172.21.15.96:0/2081988627 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f88a00b2800 0x7f88a01539e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:55.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.660+0000 7f889f400640 1 -- 172.21.15.96:0/2081988627 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88a01f2b20 con 0x7f88a00b2c90 2024-05-28T06:08:55.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.660+0000 7f889fe00640 1 --2- 172.21.15.96:0/2081988627 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f88a00b2800 0x7f88a01539e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:08:55.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.660+0000 7f889f400640 1 --2- 172.21.15.96:0/2081988627 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f88a00b2c90 0x7f88a01f2280 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f8888007e40 tx=0x7f888800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:55.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.660+0000 7f8897e00640 1 -- 172.21.15.96:0/2081988627 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8888014620 con 0x7f88a00b2c90 2024-05-28T06:08:55.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.660+0000 7f8897e00640 1 -- 172.21.15.96:0/2081988627 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8888019070 con 0x7f88a00b2c90 2024-05-28T06:08:55.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.660+0000 7f8897e00640 1 -- 172.21.15.96:0/2081988627 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8888014a00 con 0x7f88a00b2c90 2024-05-28T06:08:55.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.660+0000 7f88a6c00640 1 -- 172.21.15.96:0/2081988627 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88a01f2e00 con 0x7f88a00b2c90 2024-05-28T06:08:55.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.660+0000 7f88a6c00640 1 -- 172.21.15.96:0/2081988627 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88a009e990 con 0x7f88a00b2c90 2024-05-28T06:08:55.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.661+0000 7f88a6c00640 1 -- 172.21.15.96:0/2081988627 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88a00690f0 con 0x7f88a00b2c90 2024-05-28T06:08:55.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.662+0000 7f8897e00640 1 -- 172.21.15.96:0/2081988627 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 9) ==== 91295+0+0 (secure 0 0 0) 0x7f8888014ba0 con 0x7f88a00b2c90 2024-05-28T06:08:55.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.663+0000 7f8897e00640 1 --2- 172.21.15.96:0/2081988627 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f887c06cbc0 0x7f887c06f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:55.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.663+0000 7f8897e00640 1 -- 172.21.15.96:0/2081988627 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5737+0+0 (secure 0 0 0) 0x7f888808c930 con 0x7f88a00b2c90 2024-05-28T06:08:55.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.664+0000 7f889fe00640 1 --2- 172.21.15.96:0/2081988627 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f887c06cbc0 0x7f887c06f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:55.666 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.665+0000 7f889fe00640 1 --2- 172.21.15.96:0/2081988627 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f887c06cbc0 0x7f887c06f080 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f8890034000 tx=0x7f889002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:55.669 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.668+0000 7f8897e00640 1 -- 172.21.15.96:0/2081988627 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f888805a9b0 con 0x7f88a00b2c90 2024-05-28T06:08:55.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.869+0000 7f88a6c00640 1 -- 172.21.15.96:0/2081988627 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f88a00cfb20 con 0x7f88a00b2c90 2024-05-28T06:08:55.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.870+0000 7f8897e00640 1 -- 172.21.15.96:0/2081988627 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v44) ==== 74+0+16681 (secure 0 0 0) 0x7f888805a2b0 con 0x7f88a00b2c90 2024-05-28T06:08:55.872 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:08:55.872 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":44,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:08:54.477715+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-05-28T06:08:52.257477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-05-28T06:08:53.459376+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:08:55.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.873+0000 7f88a6c00640 1 -- 172.21.15.96:0/2081988627 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f887c06cbc0 msgr2=0x7f887c06f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:55.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.873+0000 7f88a6c00640 1 --2- 172.21.15.96:0/2081988627 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f887c06cbc0 0x7f887c06f080 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f8890034000 tx=0x7f889002e000 comp rx=0 tx=0).stop 2024-05-28T06:08:55.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.873+0000 7f88a6c00640 1 -- 172.21.15.96:0/2081988627 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f88a00b2c90 msgr2=0x7f88a01f2280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:55.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.873+0000 7f88a6c00640 1 --2- 172.21.15.96:0/2081988627 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f88a00b2c90 0x7f88a01f2280 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f8888007e40 tx=0x7f888800e4b0 comp rx=0 tx=0).stop 2024-05-28T06:08:55.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.873+0000 7f88a6c00640 1 -- 172.21.15.96:0/2081988627 shutdown_connections 2024-05-28T06:08:55.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.873+0000 7f88a6c00640 1 --2- 172.21.15.96:0/2081988627 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f887c06cbc0 0x7f887c06f080 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:55.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.873+0000 7f88a6c00640 1 --2- 172.21.15.96:0/2081988627 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f88a015fec0 0x7f88a00af310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:55.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.873+0000 7f88a6c00640 1 --2- 172.21.15.96:0/2081988627 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f88a00b2c90 0x7f88a01f2280 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:55.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.873+0000 7f88a6c00640 1 --2- 172.21.15.96:0/2081988627 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f88a00b2800 0x7f88a01539e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:55.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.873+0000 7f88a6c00640 1 -- 172.21.15.96:0/2081988627 >> 172.21.15.96:0/2081988627 conn(0x7f88a00b4020 msgr2=0x7f88a0070dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:55.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.873+0000 7f88a6c00640 1 -- 172.21.15.96:0/2081988627 shutdown_connections 2024-05-28T06:08:55.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:55.873+0000 7f88a6c00640 1 -- 172.21.15.96:0/2081988627 wait complete. 2024-05-28T06:08:55.893 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-05-28T06:08:56.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:56.096+0000 7f5d32c00640 1 Processor -- start 2024-05-28T06:08:56.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:56.097+0000 7f5d32c00640 1 -- start start 2024-05-28T06:08:56.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:56.098+0000 7f5d32c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5d2c0d0640 0x7f5d2c0d0a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:56.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:56.098+0000 7f5d32c00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f5d2c15b6f0 0x7f5d2c06e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:56.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:56.098+0000 7f5d32c00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d2c0ade90 con 0x7f5d2c0725e0 2024-05-28T06:08:56.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:56.098+0000 7f5d32c00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f5d2c071c40 con 0x7f5d2c15b6f0 2024-05-28T06:08:56.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:56.098+0000 7f5d32c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f5d2c0d0f60 con 0x7f5d2c0d0640 2024-05-28T06:08:56.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:56.098+0000 7f5d23400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f5d2c15b6f0 0x7f5d2c06e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:56.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:56.098+0000 7f5d23400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f5d2c15b6f0 0x7f5d2c06e350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51334/0 (socket says 172.21.15.96:51334) 2024-05-28T06:08:56.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:56.098+0000 7f5d23400640 1 -- 172.21.15.96:0/2870245864 learned_addr learned my addr 172.21.15.96:0/2870245864 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:08:56.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:56.099+0000 7f5d2be00640 1 --2- 172.21.15.96:0/2870245864 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5d2c0d0640 0x7f5d2c0d0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:56.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:56.099+0000 7f5d23400640 1 -- 172.21.15.96:0/2870245864 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5d2c0d0640 msgr2=0x7f5d2c0d0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:56.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:56.099+0000 7f5d23400640 1 --2- 172.21.15.96:0/2870245864 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5d2c0d0640 0x7f5d2c0d0a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:56.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:56.099+0000 7f5d23400640 1 -- 172.21.15.96:0/2870245864 >> v1:172.21.15.123:6789/0 conn(0x7f5d2c0725e0 legacy=0x7f5d2c0b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:56.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:56.099+0000 7f5d23400640 1 -- 172.21.15.96:0/2870245864 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d2c071e70 con 0x7f5d2c15b6f0 2024-05-28T06:08:56.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:56.100+0000 7f5d23400640 1 --2- 172.21.15.96:0/2870245864 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5d2c15b6f0 0x7f5d2c06e350 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f5d1400bb10 tx=0x7f5d1402c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3dd8bdffc9e545e server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:56.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:56.100+0000 7f5d2b400640 1 -- 172.21.15.96:0/2870245864 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d14032790 con 0x7f5d2c15b6f0 2024-05-28T06:08:56.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:56.100+0000 7f5d2b400640 1 -- 172.21.15.96:0/2870245864 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d14038070 con 0x7f5d2c15b6f0 2024-05-28T06:08:56.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:56.100+0000 7f5d2b400640 1 -- 172.21.15.96:0/2870245864 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d14032b50 con 0x7f5d2c15b6f0 2024-05-28T06:08:56.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:56.101+0000 7f5d32c00640 1 -- 172.21.15.96:0/2870245864 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5d2c15b6f0 msgr2=0x7f5d2c06e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:56.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:56.101+0000 7f5d32c00640 1 --2- 172.21.15.96:0/2870245864 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5d2c15b6f0 0x7f5d2c06e350 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f5d1400bb10 tx=0x7f5d1402c8d0 comp rx=0 tx=0).stop 2024-05-28T06:08:56.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:56.101+0000 7f5d32c00640 1 -- 172.21.15.96:0/2870245864 shutdown_connections 2024-05-28T06:08:56.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:56.101+0000 7f5d32c00640 1 --2- 172.21.15.96:0/2870245864 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5d2c15b6f0 0x7f5d2c06e350 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:56.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:56.101+0000 7f5d32c00640 1 --2- 172.21.15.96:0/2870245864 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5d2c0d0640 0x7f5d2c0d0a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:56.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:56.101+0000 7f5d32c00640 1 -- 172.21.15.96:0/2870245864 >> 172.21.15.96:0/2870245864 conn(0x7f5d2c06bd20 msgr2=0x7f5d2c06c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:56.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:56.101+0000 7f5d32c00640 1 -- 172.21.15.96:0/2870245864 shutdown_connections 2024-05-28T06:08:56.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:56.101+0000 7f5d32c00640 1 -- 172.21.15.96:0/2870245864 wait complete. 2024-05-28T06:08:56.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:56.104+0000 7f5d32c00640 1 Processor -- start 2024-05-28T06:08:56.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:56.104+0000 7f5d32c00640 1 -- start start 2024-05-28T06:08:56.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:56.105+0000 7f5d32c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5d2c0725e0 0x7f5d2c1f55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:56.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:56.105+0000 7f5d32c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d2c0d0640 0x7f5d2c155af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:56.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:56.105+0000 7f5d32c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5d2c15b6f0 0x7f5d2c1515e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:56.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:56.105+0000 7f5d32c00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f5d2c06e890 con 0x7f5d2c15b6f0 2024-05-28T06:08:56.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:56.105+0000 7f5d32c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5d2c0ae130 con 0x7f5d2c0d0640 2024-05-28T06:08:56.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:56.105+0000 7f5d32c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f5d2c0af660 con 0x7f5d2c0725e0 2024-05-28T06:08:56.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:56.106+0000 7f5d30c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5d2c15b6f0 0x7f5d2c1515e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:56.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:56.106+0000 7f5d30c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5d2c15b6f0 0x7f5d2c1515e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51350/0 (socket says 172.21.15.96:51350) 2024-05-28T06:08:56.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:56.106+0000 7f5d30c00640 1 -- 172.21.15.96:0/3869419670 learned_addr learned my addr 172.21.15.96:0/3869419670 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:08:56.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:56.106+0000 7f5d23400640 1 --2- 172.21.15.96:0/3869419670 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d2c0d0640 0x7f5d2c155af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:56.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:56.106+0000 7f5d2be00640 1 --2- 172.21.15.96:0/3869419670 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5d2c0725e0 0x7f5d2c1f55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:56.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:56.106+0000 7f5d30c00640 1 -- 172.21.15.96:0/3869419670 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5d2c0725e0 msgr2=0x7f5d2c1f55c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:56.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:56.106+0000 7f5d30c00640 1 --2- 172.21.15.96:0/3869419670 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5d2c0725e0 0x7f5d2c1f55c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:56.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:56.106+0000 7f5d30c00640 1 -- 172.21.15.96:0/3869419670 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d2c0d0640 msgr2=0x7f5d2c155af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:56.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:56.106+0000 7f5d30c00640 1 --2- 172.21.15.96:0/3869419670 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d2c0d0640 0x7f5d2c155af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:56.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:56.106+0000 7f5d30c00640 1 -- 172.21.15.96:0/3869419670 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d2c156030 con 0x7f5d2c15b6f0 2024-05-28T06:08:56.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:56.106+0000 7f5d30c00640 1 --2- 172.21.15.96:0/3869419670 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5d2c15b6f0 0x7f5d2c1515e0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f5d2400c7b0 tx=0x7f5d2400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:56.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:56.107+0000 7f5d28c00640 1 -- 172.21.15.96:0/3869419670 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d24004b60 con 0x7f5d2c15b6f0 2024-05-28T06:08:56.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:56.107+0000 7f5d28c00640 1 -- 172.21.15.96:0/3869419670 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d24004d00 con 0x7f5d2c15b6f0 2024-05-28T06:08:56.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:56.107+0000 7f5d32c00640 1 -- 172.21.15.96:0/3869419670 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d2c15cf40 con 0x7f5d2c15b6f0 2024-05-28T06:08:56.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:56.107+0000 7f5d28c00640 1 -- 172.21.15.96:0/3869419670 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d2400aa90 con 0x7f5d2c15b6f0 2024-05-28T06:08:56.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:56.107+0000 7f5d32c00640 1 -- 172.21.15.96:0/3869419670 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d2c15d500 con 0x7f5d2c15b6f0 2024-05-28T06:08:56.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:56.108+0000 7f5d32c00640 1 -- 172.21.15.96:0/3869419670 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d2c0689e0 con 0x7f5d2c15b6f0 2024-05-28T06:08:56.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:56.109+0000 7f5d28c00640 1 -- 172.21.15.96:0/3869419670 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 9) ==== 91295+0+0 (secure 0 0 0) 0x7f5d2400ac30 con 0x7f5d2c15b6f0 2024-05-28T06:08:56.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:56.110+0000 7f5d28c00640 1 --2- 172.21.15.96:0/3869419670 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5d0806cbf0 0x7f5d0806f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:56.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:56.110+0000 7f5d28c00640 1 -- 172.21.15.96:0/3869419670 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5737+0+0 (secure 0 0 0) 0x7f5d2408c190 con 0x7f5d2c15b6f0 2024-05-28T06:08:56.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:56.111+0000 7f5d2be00640 1 --2- 172.21.15.96:0/3869419670 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5d0806cbf0 0x7f5d0806f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:56.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:56.112+0000 7f5d2be00640 1 --2- 172.21.15.96:0/3869419670 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5d0806cbf0 0x7f5d0806f0b0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c00a970 tx=0x7f5d1c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:56.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:56.114+0000 7f5d28c00640 1 -- 172.21.15.96:0/3869419670 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5d24020070 con 0x7f5d2c15b6f0 2024-05-28T06:08:56.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:56.356+0000 7f5d32c00640 1 -- 172.21.15.96:0/3869419670 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f5d2c15a7c0 con 0x7f5d2c15b6f0 2024-05-28T06:08:57.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:57.076+0000 7f5d28c00640 1 -- 172.21.15.96:0/3869419670 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v24) ==== 118+0+0 (secure 0 0 0) 0x7f5d2405a2a0 con 0x7f5d2c15b6f0 2024-05-28T06:08:57.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:57.079+0000 7f5d32c00640 1 -- 172.21.15.96:0/3869419670 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5d0806cbf0 msgr2=0x7f5d0806f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:57.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:57.079+0000 7f5d32c00640 1 --2- 172.21.15.96:0/3869419670 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5d0806cbf0 0x7f5d0806f0b0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c00a970 tx=0x7f5d1c008040 comp rx=0 tx=0).stop 2024-05-28T06:08:57.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:57.079+0000 7f5d32c00640 1 -- 172.21.15.96:0/3869419670 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5d2c15b6f0 msgr2=0x7f5d2c1515e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:57.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:57.079+0000 7f5d32c00640 1 --2- 172.21.15.96:0/3869419670 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5d2c15b6f0 0x7f5d2c1515e0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f5d2400c7b0 tx=0x7f5d2400cc70 comp rx=0 tx=0).stop 2024-05-28T06:08:57.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:57.079+0000 7f5d32c00640 1 -- 172.21.15.96:0/3869419670 shutdown_connections 2024-05-28T06:08:57.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:57.079+0000 7f5d32c00640 1 --2- 172.21.15.96:0/3869419670 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5d0806cbf0 0x7f5d0806f0b0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:57.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:57.079+0000 7f5d32c00640 1 --2- 172.21.15.96:0/3869419670 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5d2c15b6f0 0x7f5d2c1515e0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:57.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:57.079+0000 7f5d32c00640 1 --2- 172.21.15.96:0/3869419670 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d2c0d0640 0x7f5d2c155af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:57.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:57.079+0000 7f5d32c00640 1 --2- 172.21.15.96:0/3869419670 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5d2c0725e0 0x7f5d2c1f55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:57.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:57.080+0000 7f5d32c00640 1 -- 172.21.15.96:0/3869419670 >> 172.21.15.96:0/3869419670 conn(0x7f5d2c06bd20 msgr2=0x7f5d2c0afe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:57.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:57.080+0000 7f5d32c00640 1 -- 172.21.15.96:0/3869419670 shutdown_connections 2024-05-28T06:08:57.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:57.080+0000 7f5d32c00640 1 -- 172.21.15.96:0/3869419670 wait complete. 2024-05-28T06:08:57.103 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-05-28T06:08:57.104 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-05-28T06:08:57.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:57.308+0000 7f0ce6e00640 1 Processor -- start 2024-05-28T06:08:57.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:57.308+0000 7f0ce6e00640 1 -- start start 2024-05-28T06:08:57.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:57.310+0000 7f0ce6e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f0ce015d5d0 0x7f0ce00cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:57.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:57.310+0000 7f0ce6e00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ce015cb10 con 0x7f0ce00b32c0 2024-05-28T06:08:57.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:57.310+0000 7f0ce6e00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ce00ad5f0 con 0x7f0ce00d0640 2024-05-28T06:08:57.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:57.310+0000 7f0ce6e00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f0ce00adbb0 con 0x7f0ce015d5d0 2024-05-28T06:08:57.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:57.310+0000 7f0ce4e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f0ce00b32c0 0x7f0ce00b3700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:56452/0 (socket says 172.21.15.96:56452) 2024-05-28T06:08:57.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:57.310+0000 7f0ce4e00640 1 -- 172.21.15.96:0/606095462 learned_addr learned my addr 172.21.15.96:0/606095462 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:08:57.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:57.311+0000 7f0cdf400640 1 --2- 172.21.15.96:0/606095462 >> v2:172.21.15.123:3300/0 conn(0x7f0ce015d5d0 0x7f0ce00cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:57.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:57.311+0000 7f0cdea00640 1 -- 172.21.15.96:0/606095462 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1795870106 0 0) 0x7f0ce015cb10 con 0x7f0ce00b32c0 2024-05-28T06:08:57.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:57.311+0000 7f0cdea00640 1 -- 172.21.15.96:0/606095462 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0cc4003520 con 0x7f0ce00b32c0 2024-05-28T06:08:57.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:57.311+0000 7f0cdea00640 1 -- 172.21.15.96:0/606095462 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2314795872 0 0) 0x7f0cc4003520 con 0x7f0ce00b32c0 2024-05-28T06:08:57.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:57.311+0000 7f0cdea00640 1 -- 172.21.15.96:0/606095462 >> v2:172.21.15.123:3300/0 conn(0x7f0ce015d5d0 msgr2=0x7f0ce00cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:57.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:57.311+0000 7f0cdea00640 1 --2- 172.21.15.96:0/606095462 >> v2:172.21.15.123:3300/0 conn(0x7f0ce015d5d0 0x7f0ce00cf950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:57.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:57.311+0000 7f0cdea00640 1 -- 172.21.15.96:0/606095462 >> v1:172.21.15.123:6789/0 conn(0x7f0ce00d0640 legacy=0x7f0ce00d0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:57.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:57.311+0000 7f0cdea00640 1 -- 172.21.15.96:0/606095462 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ce00ad8d0 con 0x7f0ce00b32c0 2024-05-28T06:08:57.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:57.311+0000 7f0cdea00640 1 -- 172.21.15.96:0/606095462 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f0cd80019f0 con 0x7f0ce00b32c0 2024-05-28T06:08:57.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:57.311+0000 7f0cdea00640 1 -- 172.21.15.96:0/606095462 >> v1:172.21.15.96:6789/0 conn(0x7f0ce00b32c0 legacy=0x7f0ce00b3700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:57.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:57.311+0000 7f0cdea00640 1 --2- 172.21.15.96:0/606095462 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0cc4003a00 0x7f0cc4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:57.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:57.311+0000 7f0cdea00640 1 --2- 172.21.15.96:0/606095462 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0cc4004800 0x7f0cc4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:57.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:57.311+0000 7f0cdea00640 1 --2- 172.21.15.96:0/606095462 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0cc4005210 0x7f0cc400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:57.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:57.311+0000 7f0cdea00640 1 -- 172.21.15.96:0/606095462 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f0cc400c130 con 0x7f0cc4003a00 2024-05-28T06:08:57.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:57.311+0000 7f0cdea00640 1 -- 172.21.15.96:0/606095462 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f0cc400c2e0 con 0x7f0cc4005210 2024-05-28T06:08:57.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:57.311+0000 7f0cdea00640 1 -- 172.21.15.96:0/606095462 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f0cc400c4e0 con 0x7f0cc4004800 2024-05-28T06:08:57.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:57.311+0000 7f0cdfe00640 1 --2- 172.21.15.96:0/606095462 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0cc4003a00 0x7f0cc4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:57.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:57.312+0000 7f0cdfe00640 1 -- 172.21.15.96:0/606095462 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0cc4004800 msgr2=0x7f0cc4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:57.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:57.312+0000 7f0cdfe00640 1 --2- 172.21.15.96:0/606095462 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0cc4004800 0x7f0cc4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:57.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:57.312+0000 7f0cdfe00640 1 -- 172.21.15.96:0/606095462 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0cc4005210 msgr2=0x7f0cc400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:57.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:57.312+0000 7f0cdfe00640 1 --2- 172.21.15.96:0/606095462 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0cc4005210 0x7f0cc400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:57.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:57.312+0000 7f0cdfe00640 1 -- 172.21.15.96:0/606095462 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cc4003520 con 0x7f0cc4003a00 2024-05-28T06:08:57.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:57.312+0000 7f0ce4e00640 1 -- 172.21.15.96:0/606095462 reap_dead start 2024-05-28T06:08:57.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:57.313+0000 7f0cdfe00640 1 --2- 172.21.15.96:0/606095462 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0cc4003a00 0x7f0cc4003e20 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f0cd0001720 tx=0x7f0cc4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fb4f8e8a5b04510 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:57.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:57.313+0000 7f0cdea00640 1 -- 172.21.15.96:0/606095462 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cd0001370 con 0x7f0cc4003a00 2024-05-28T06:08:57.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:57.313+0000 7f0cdea00640 1 -- 172.21.15.96:0/606095462 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0cd00052d0 con 0x7f0cc4003a00 2024-05-28T06:08:57.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:57.313+0000 7f0cdea00640 1 -- 172.21.15.96:0/606095462 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cd002ab70 con 0x7f0cc4003a00 2024-05-28T06:08:57.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:57.314+0000 7f0ce6e00640 1 -- 172.21.15.96:0/606095462 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0cc4003a00 msgr2=0x7f0cc4003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:57.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:57.314+0000 7f0ce6e00640 1 --2- 172.21.15.96:0/606095462 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0cc4003a00 0x7f0cc4003e20 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f0cd0001720 tx=0x7f0cc4005830 comp rx=0 tx=0).stop 2024-05-28T06:08:57.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:57.314+0000 7f0ce6e00640 1 -- 172.21.15.96:0/606095462 shutdown_connections 2024-05-28T06:08:57.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:57.314+0000 7f0ce6e00640 1 --2- 172.21.15.96:0/606095462 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0cc4003a00 0x7f0cc4003e20 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:57.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:57.314+0000 7f0ce6e00640 1 -- 172.21.15.96:0/606095462 >> 172.21.15.96:0/606095462 conn(0x7f0ce006bd20 msgr2=0x7f0ce006c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:57.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:57.314+0000 7f0ce6e00640 1 -- 172.21.15.96:0/606095462 shutdown_connections 2024-05-28T06:08:57.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:57.314+0000 7f0ce6e00640 1 -- 172.21.15.96:0/606095462 wait complete. 2024-05-28T06:08:57.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:57.316+0000 7f0ce6e00640 1 Processor -- start 2024-05-28T06:08:57.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:57.316+0000 7f0ce6e00640 1 -- start start 2024-05-28T06:08:57.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:57.317+0000 7f0ce6e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0cc4003a00 0x7f0ce01f7760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:57.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:57.317+0000 7f0ce6e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0ce0072ed0 0x7f0ce00b2470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:57.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:57.317+0000 7f0ce6e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ce0073350 0x7f0ce00b29b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:57.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:57.317+0000 7f0ce6e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f0ce015da50 con 0x7f0cc4003a00 2024-05-28T06:08:57.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:57.317+0000 7f0ce6e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f0ce015d0b0 con 0x7f0ce0073350 2024-05-28T06:08:57.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:57.317+0000 7f0ce6e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f0ce00ad2a0 con 0x7f0ce0072ed0 2024-05-28T06:08:57.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:57.317+0000 7f0cdfe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0cc4003a00 0x7f0ce01f7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:57.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:57.317+0000 7f0cdfe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0cc4003a00 0x7f0ce01f7760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51366/0 (socket says 172.21.15.96:51366) 2024-05-28T06:08:57.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:57.317+0000 7f0cdfe00640 1 -- 172.21.15.96:0/502417126 learned_addr learned my addr 172.21.15.96:0/502417126 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:08:57.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:57.317+0000 7f0cdfe00640 1 -- 172.21.15.96:0/502417126 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0ce0072ed0 msgr2=0x7f0ce00b2470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:57.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:57.317+0000 7f0cdfe00640 1 --2- 172.21.15.96:0/502417126 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0ce0072ed0 0x7f0ce00b2470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:57.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:57.317+0000 7f0cdfe00640 1 -- 172.21.15.96:0/502417126 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ce0073350 msgr2=0x7f0ce00b29b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:57.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:57.317+0000 7f0cdfe00640 1 --2- 172.21.15.96:0/502417126 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ce0073350 0x7f0ce00b29b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:57.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:57.318+0000 7f0cdfe00640 1 -- 172.21.15.96:0/502417126 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ce014ac20 con 0x7f0cc4003a00 2024-05-28T06:08:57.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:57.318+0000 7f0cdfe00640 1 --2- 172.21.15.96:0/502417126 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0cc4003a00 0x7f0ce01f7760 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f0cd0005040 tx=0x7f0cd0001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:57.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:57.318+0000 7f0cd7e00640 1 -- 172.21.15.96:0/502417126 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cd0037040 con 0x7f0cc4003a00 2024-05-28T06:08:57.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:57.318+0000 7f0cd7e00640 1 -- 172.21.15.96:0/502417126 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0cd0040070 con 0x7f0cc4003a00 2024-05-28T06:08:57.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:57.318+0000 7f0cd7e00640 1 -- 172.21.15.96:0/502417126 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cd0007a30 con 0x7f0cc4003a00 2024-05-28T06:08:57.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:57.318+0000 7f0ce6e00640 1 -- 172.21.15.96:0/502417126 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ce014ba40 con 0x7f0cc4003a00 2024-05-28T06:08:57.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:57.318+0000 7f0ce6e00640 1 -- 172.21.15.96:0/502417126 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ce014b1a0 con 0x7f0cc4003a00 2024-05-28T06:08:57.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:57.319+0000 7f0ce6e00640 1 -- 172.21.15.96:0/502417126 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ce00689e0 con 0x7f0cc4003a00 2024-05-28T06:08:57.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:57.320+0000 7f0cd7e00640 1 -- 172.21.15.96:0/502417126 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 9) ==== 91295+0+0 (secure 0 0 0) 0x7f0cd00052d0 con 0x7f0cc4003a00 2024-05-28T06:08:57.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:57.321+0000 7f0cd7e00640 1 --2- 172.21.15.96:0/502417126 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0cbc06cbc0 0x7f0cbc06f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:57.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:57.321+0000 7f0cd7e00640 1 -- 172.21.15.96:0/502417126 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5737+0+0 (secure 0 0 0) 0x7f0cd00abca0 con 0x7f0cc4003a00 2024-05-28T06:08:57.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:57.322+0000 7f0cdf400640 1 --2- 172.21.15.96:0/502417126 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0cbc06cbc0 0x7f0cbc06f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:57.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:57.323+0000 7f0cdf400640 1 --2- 172.21.15.96:0/502417126 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0cbc06cbc0 0x7f0cbc06f080 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f0cc8006e80 tx=0x7f0cc8006dd0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:57.327 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:57.326+0000 7f0cd7e00640 1 -- 172.21.15.96:0/502417126 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0cd0079d60 con 0x7f0cc4003a00 2024-05-28T06:08:57.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:57.571+0000 7f0ce6e00640 1 -- 172.21.15.96:0/502417126 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f0ce0071ef0 con 0x7f0cc4003a00 2024-05-28T06:08:58.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.079+0000 7f0cd7e00640 1 -- 172.21.15.96:0/502417126 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v25) ==== 122+0+0 (secure 0 0 0) 0x7f0cd0079660 con 0x7f0cc4003a00 2024-05-28T06:08:58.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.083+0000 7f0ce6e00640 1 -- 172.21.15.96:0/502417126 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0cbc06cbc0 msgr2=0x7f0cbc06f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:58.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.083+0000 7f0ce6e00640 1 --2- 172.21.15.96:0/502417126 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0cbc06cbc0 0x7f0cbc06f080 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f0cc8006e80 tx=0x7f0cc8006dd0 comp rx=0 tx=0).stop 2024-05-28T06:08:58.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.083+0000 7f0ce6e00640 1 -- 172.21.15.96:0/502417126 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0cc4003a00 msgr2=0x7f0ce01f7760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:58.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.083+0000 7f0ce6e00640 1 --2- 172.21.15.96:0/502417126 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0cc4003a00 0x7f0ce01f7760 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f0cd0005040 tx=0x7f0cd0001120 comp rx=0 tx=0).stop 2024-05-28T06:08:58.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.083+0000 7f0ce6e00640 1 -- 172.21.15.96:0/502417126 shutdown_connections 2024-05-28T06:08:58.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.083+0000 7f0ce6e00640 1 --2- 172.21.15.96:0/502417126 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0cbc06cbc0 0x7f0cbc06f080 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:58.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.083+0000 7f0ce6e00640 1 --2- 172.21.15.96:0/502417126 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ce0073350 0x7f0ce00b29b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:58.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.083+0000 7f0ce6e00640 1 --2- 172.21.15.96:0/502417126 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0ce0072ed0 0x7f0ce00b2470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:58.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.083+0000 7f0ce6e00640 1 --2- 172.21.15.96:0/502417126 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0cc4003a00 0x7f0ce01f7760 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:58.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.083+0000 7f0ce6e00640 1 -- 172.21.15.96:0/502417126 >> 172.21.15.96:0/502417126 conn(0x7f0ce006bd20 msgr2=0x7f0ce0151d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:58.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.083+0000 7f0ce6e00640 1 -- 172.21.15.96:0/502417126 shutdown_connections 2024-05-28T06:08:58.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.083+0000 7f0ce6e00640 1 -- 172.21.15.96:0/502417126 wait complete. 2024-05-28T06:08:58.110 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-05-28T06:08:58.110 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-28T06:08:58.308 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.307+0000 7f49bbe00640 1 Processor -- start 2024-05-28T06:08:58.308 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.307+0000 7f49bbe00640 1 -- start start 2024-05-28T06:08:58.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.308+0000 7f49bbe00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f49b40b4e10 0x7f49b40cde30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:58.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.308+0000 7f49bbe00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f49b40ce370 0x7f49b40d0590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:58.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.308+0000 7f49bbe00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f49b40d0ad0 0x7f49b4099090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:58.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.308+0000 7f49bbe00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f49b40d0f20 con 0x7f49b40ce370 2024-05-28T06:08:58.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.308+0000 7f49bbe00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f49b40b51f0 con 0x7f49b40d0ad0 2024-05-28T06:08:58.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.308+0000 7f49bbe00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f49b40ad1d0 con 0x7f49b40b4e10 2024-05-28T06:08:58.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.309+0000 7f49b8a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f49b40ce370 0x7f49b40d0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:58.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.309+0000 7f49b8a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f49b40ce370 0x7f49b40d0590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51378/0 (socket says 172.21.15.96:51378) 2024-05-28T06:08:58.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.309+0000 7f49b8a00640 1 -- 172.21.15.96:0/1645628589 learned_addr learned my addr 172.21.15.96:0/1645628589 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:08:58.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.309+0000 7f49b9400640 1 --2- 172.21.15.96:0/1645628589 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f49b40b4e10 0x7f49b40cde30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:58.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.309+0000 7f49b8a00640 1 -- 172.21.15.96:0/1645628589 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f49b40b4e10 msgr2=0x7f49b40cde30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:58.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.309+0000 7f49b8a00640 1 --2- 172.21.15.96:0/1645628589 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f49b40b4e10 0x7f49b40cde30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:58.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.309+0000 7f49b8a00640 1 -- 172.21.15.96:0/1645628589 >> v2:172.21.15.123:3300/0 conn(0x7f49b40d0ad0 msgr2=0x7f49b4099090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:58.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.309+0000 7f49b8a00640 1 --2- 172.21.15.96:0/1645628589 >> v2:172.21.15.123:3300/0 conn(0x7f49b40d0ad0 0x7f49b4099090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:58.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.309+0000 7f49b8a00640 1 -- 172.21.15.96:0/1645628589 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49b40ad470 con 0x7f49b40ce370 2024-05-28T06:08:58.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.309+0000 7f49b9400640 1 --2- 172.21.15.96:0/1645628589 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f49b40b4e10 0x7f49b40cde30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:08:58.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.310+0000 7f49b8a00640 1 --2- 172.21.15.96:0/1645628589 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f49b40ce370 0x7f49b40d0590 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f49b40cfb50 tx=0x7f49a002a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9ad0297345a1889 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:58.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.311+0000 7f49b3e00640 1 -- 172.21.15.96:0/1645628589 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49a0037070 con 0x7f49b40ce370 2024-05-28T06:08:58.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.311+0000 7f49b3e00640 1 -- 172.21.15.96:0/1645628589 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49a0032440 con 0x7f49b40ce370 2024-05-28T06:08:58.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.311+0000 7f49b3e00640 1 -- 172.21.15.96:0/1645628589 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49a0032810 con 0x7f49b40ce370 2024-05-28T06:08:58.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.311+0000 7f49bbe00640 1 -- 172.21.15.96:0/1645628589 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f49b40ce370 msgr2=0x7f49b40d0590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:58.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.311+0000 7f49bbe00640 1 --2- 172.21.15.96:0/1645628589 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f49b40ce370 0x7f49b40d0590 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f49b40cfb50 tx=0x7f49a002a140 comp rx=0 tx=0).stop 2024-05-28T06:08:58.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.311+0000 7f49bbe00640 1 -- 172.21.15.96:0/1645628589 shutdown_connections 2024-05-28T06:08:58.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.311+0000 7f49bbe00640 1 --2- 172.21.15.96:0/1645628589 >> v2:172.21.15.123:3300/0 conn(0x7f49b40d0ad0 0x7f49b4099090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:58.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.311+0000 7f49bbe00640 1 --2- 172.21.15.96:0/1645628589 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f49b40ce370 0x7f49b40d0590 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:58.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.311+0000 7f49bbe00640 1 --2- 172.21.15.96:0/1645628589 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f49b40b4e10 0x7f49b40cde30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:58.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.311+0000 7f49bbe00640 1 -- 172.21.15.96:0/1645628589 >> 172.21.15.96:0/1645628589 conn(0x7f49b406bae0 msgr2=0x7f49b406bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:58.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.311+0000 7f49bbe00640 1 -- 172.21.15.96:0/1645628589 shutdown_connections 2024-05-28T06:08:58.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.311+0000 7f49bbe00640 1 -- 172.21.15.96:0/1645628589 wait complete. 2024-05-28T06:08:58.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.313+0000 7f49bbe00640 1 Processor -- start 2024-05-28T06:08:58.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.313+0000 7f49bbe00640 1 -- start start 2024-05-28T06:08:58.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.314+0000 7f49bbe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49b40b4e10 0x7f49b415e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:58.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.314+0000 7f49bbe00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f49b40ce370 0x7f49b4158620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:58.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.314+0000 7f49bbe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f49b40d0ad0 0x7f49b41561f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:58.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.314+0000 7f49bbe00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f49b40cea00 con 0x7f49b40d0ad0 2024-05-28T06:08:58.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.314+0000 7f49bbe00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f49b40ce880 con 0x7f49b40b4e10 2024-05-28T06:08:58.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.314+0000 7f49bbe00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f49b40b3150 con 0x7f49b40ce370 2024-05-28T06:08:58.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.314+0000 7f49b9e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f49b40d0ad0 0x7f49b41561f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:58.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.314+0000 7f49b9e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f49b40d0ad0 0x7f49b41561f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51388/0 (socket says 172.21.15.96:51388) 2024-05-28T06:08:58.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.314+0000 7f49b9e00640 1 -- 172.21.15.96:0/4113431187 learned_addr learned my addr 172.21.15.96:0/4113431187 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:08:58.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.315+0000 7f49b9400640 1 --2- 172.21.15.96:0/4113431187 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49b40b4e10 0x7f49b415e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:58.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.315+0000 7f49b8a00640 1 --2- 172.21.15.96:0/4113431187 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f49b40ce370 0x7f49b4158620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:58.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.315+0000 7f49b9e00640 1 -- 172.21.15.96:0/4113431187 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f49b40ce370 msgr2=0x7f49b4158620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:58.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.315+0000 7f49b9e00640 1 --2- 172.21.15.96:0/4113431187 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f49b40ce370 0x7f49b4158620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:58.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.315+0000 7f49b9e00640 1 -- 172.21.15.96:0/4113431187 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49b40b4e10 msgr2=0x7f49b415e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:58.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.315+0000 7f49b9e00640 1 --2- 172.21.15.96:0/4113431187 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49b40b4e10 0x7f49b415e120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:58.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.315+0000 7f49b9e00640 1 -- 172.21.15.96:0/4113431187 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49b415e800 con 0x7f49b40d0ad0 2024-05-28T06:08:58.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.315+0000 7f49b9400640 1 --2- 172.21.15.96:0/4113431187 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49b40b4e10 0x7f49b415e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:08:58.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.315+0000 7f49b9e00640 1 --2- 172.21.15.96:0/4113431187 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f49b40d0ad0 0x7f49b41561f0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f49ac009fd0 tx=0x7f49ac00b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:58.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.315+0000 7f49b1600640 1 -- 172.21.15.96:0/4113431187 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49ac011830 con 0x7f49b40d0ad0 2024-05-28T06:08:58.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.315+0000 7f49b1600640 1 -- 172.21.15.96:0/4113431187 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49ac016070 con 0x7f49b40d0ad0 2024-05-28T06:08:58.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.315+0000 7f49bbe00640 1 -- 172.21.15.96:0/4113431187 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49b40b16b0 con 0x7f49b40d0ad0 2024-05-28T06:08:58.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.315+0000 7f49b1600640 1 -- 172.21.15.96:0/4113431187 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49ac011bf0 con 0x7f49b40d0ad0 2024-05-28T06:08:58.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.316+0000 7f49bbe00640 1 -- 172.21.15.96:0/4113431187 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49b40b1c70 con 0x7f49b40d0ad0 2024-05-28T06:08:58.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.317+0000 7f49bbe00640 1 -- 172.21.15.96:0/4113431187 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49b40689e0 con 0x7f49b40d0ad0 2024-05-28T06:08:58.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.318+0000 7f49b1600640 1 -- 172.21.15.96:0/4113431187 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 9) ==== 91295+0+0 (secure 0 0 0) 0x7f49ac01b460 con 0x7f49b40d0ad0 2024-05-28T06:08:58.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.319+0000 7f49b1600640 1 --2- 172.21.15.96:0/4113431187 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f499406cbc0 0x7f499406f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:58.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.319+0000 7f49b1600640 1 -- 172.21.15.96:0/4113431187 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5737+0+0 (secure 0 0 0) 0x7f49ac08bd20 con 0x7f49b40d0ad0 2024-05-28T06:08:58.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.320+0000 7f49b9400640 1 --2- 172.21.15.96:0/4113431187 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f499406cbc0 0x7f499406f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:58.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.321+0000 7f49b9400640 1 --2- 172.21.15.96:0/4113431187 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f499406cbc0 0x7f499406f080 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f49b4099600 tx=0x7f49a80074b0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:58.325 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.324+0000 7f49b1600640 1 -- 172.21.15.96:0/4113431187 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f49ac059e30 con 0x7f49b40d0ad0 2024-05-28T06:08:58.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.564+0000 7f49bbe00640 1 -- 172.21.15.96:0/4113431187 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f49b406f540 con 0x7f49b40d0ad0 2024-05-28T06:08:58.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.565+0000 7f49b1600640 1 -- 172.21.15.96:0/4113431187 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+5181 (secure 0 0 0) 0x7f49ac059730 con 0x7f49b40d0ad0 2024-05-28T06:08:58.566 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:08:58.566 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/1693172076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":1693172076},{"type":"v1","addr":"172.21.15.123:6835","nonce":1693172076}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":4805,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/1098611268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":1098611268},{"type":"v1","addr":"172.21.15.123:6837","nonce":1098611268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5856,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6834/673165997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":673165997},{"type":"v1","addr":"172.21.15.96:6834","nonce":673165997}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-28T06:08:54.477674+0000","modified":"2024-05-28T06:08:57.572929+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5859},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5859":{"gid":5859,"name":"a","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.96:6835/935552515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6833","nonce":935552515},{"type":"v1","addr":"172.21.15.96:6835","nonce":935552515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5859,"qdb_cluster":[5859]},"id":3}]} 2024-05-28T06:08:58.567 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 25 2024-05-28T06:08:58.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.567+0000 7f49bbe00640 1 -- 172.21.15.96:0/4113431187 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f499406cbc0 msgr2=0x7f499406f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:58.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.567+0000 7f49bbe00640 1 --2- 172.21.15.96:0/4113431187 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f499406cbc0 0x7f499406f080 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f49b4099600 tx=0x7f49a80074b0 comp rx=0 tx=0).stop 2024-05-28T06:08:58.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.567+0000 7f49bbe00640 1 -- 172.21.15.96:0/4113431187 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f49b40d0ad0 msgr2=0x7f49b41561f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:58.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.567+0000 7f49bbe00640 1 --2- 172.21.15.96:0/4113431187 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f49b40d0ad0 0x7f49b41561f0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f49ac009fd0 tx=0x7f49ac00b8e0 comp rx=0 tx=0).stop 2024-05-28T06:08:58.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.567+0000 7f49bbe00640 1 -- 172.21.15.96:0/4113431187 shutdown_connections 2024-05-28T06:08:58.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.567+0000 7f49bbe00640 1 --2- 172.21.15.96:0/4113431187 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f499406cbc0 0x7f499406f080 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:58.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.567+0000 7f49bbe00640 1 --2- 172.21.15.96:0/4113431187 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f49b40d0ad0 0x7f49b41561f0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:58.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.567+0000 7f49bbe00640 1 --2- 172.21.15.96:0/4113431187 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f49b40ce370 0x7f49b4158620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:58.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.567+0000 7f49bbe00640 1 --2- 172.21.15.96:0/4113431187 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49b40b4e10 0x7f49b415e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:58.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.568+0000 7f49bbe00640 1 -- 172.21.15.96:0/4113431187 >> 172.21.15.96:0/4113431187 conn(0x7f49b406bae0 msgr2=0x7f49b415ac60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:58.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.568+0000 7f49bbe00640 1 -- 172.21.15.96:0/4113431187 shutdown_connections 2024-05-28T06:08:58.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.568+0000 7f49bbe00640 1 -- 172.21.15.96:0/4113431187 wait complete. 2024-05-28T06:08:58.588 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:08:58.798 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.796+0000 7f1e4da00640 1 Processor -- start 2024-05-28T06:08:58.798 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.796+0000 7f1e4da00640 1 -- start start 2024-05-28T06:08:58.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.798+0000 7f1e4da00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f1e480b3b50 0x7f1e480d3190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:58.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.798+0000 7f1e4da00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f1e4815ed20 0x7f1e4815f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:58.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.798+0000 7f1e4da00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e480d36d0 con 0x7f1e480d2db0 2024-05-28T06:08:58.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.798+0000 7f1e4da00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f1e480d5540 con 0x7f1e4815ed20 2024-05-28T06:08:58.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.798+0000 7f1e4da00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f1e48163500 con 0x7f1e480b3b50 2024-05-28T06:08:58.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.798+0000 7f1e47400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f1e4815ed20 0x7f1e4815f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:58.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.798+0000 7f1e47400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f1e4815ed20 0x7f1e4815f1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51400/0 (socket says 172.21.15.96:51400) 2024-05-28T06:08:58.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.798+0000 7f1e47400640 1 -- 172.21.15.96:0/3291139059 learned_addr learned my addr 172.21.15.96:0/3291139059 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:08:58.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.798+0000 7f1e47400640 1 -- 172.21.15.96:0/3291139059 >> v2:172.21.15.123:3300/0 conn(0x7f1e480b3b50 msgr2=0x7f1e480d3190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:58.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.798+0000 7f1e47400640 1 --2- 172.21.15.96:0/3291139059 >> v2:172.21.15.123:3300/0 conn(0x7f1e480b3b50 0x7f1e480d3190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:58.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.798+0000 7f1e47400640 1 -- 172.21.15.96:0/3291139059 >> v1:172.21.15.123:6789/0 conn(0x7f1e480d2db0 legacy=0x7f1e480d4d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:58.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.798+0000 7f1e47400640 1 -- 172.21.15.96:0/3291139059 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e48162740 con 0x7f1e4815ed20 2024-05-28T06:08:58.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.799+0000 7f1e47400640 1 --2- 172.21.15.96:0/3291139059 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1e4815ed20 0x7f1e4815f1b0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f1e480b3b00 tx=0x7f1e4002a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77f7f2590eb15c07 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:58.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.800+0000 7f1e46000640 1 -- 172.21.15.96:0/3291139059 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e40036070 con 0x7f1e4815ed20 2024-05-28T06:08:58.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.800+0000 7f1e46000640 1 -- 172.21.15.96:0/3291139059 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e40031600 con 0x7f1e4815ed20 2024-05-28T06:08:58.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.800+0000 7f1e46000640 1 -- 172.21.15.96:0/3291139059 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e400319d0 con 0x7f1e4815ed20 2024-05-28T06:08:58.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.800+0000 7f1e4da00640 1 -- 172.21.15.96:0/3291139059 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1e4815ed20 msgr2=0x7f1e4815f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:58.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.800+0000 7f1e4da00640 1 --2- 172.21.15.96:0/3291139059 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1e4815ed20 0x7f1e4815f1b0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f1e480b3b00 tx=0x7f1e4002a660 comp rx=0 tx=0).stop 2024-05-28T06:08:58.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.800+0000 7f1e4da00640 1 -- 172.21.15.96:0/3291139059 shutdown_connections 2024-05-28T06:08:58.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.800+0000 7f1e4da00640 1 --2- 172.21.15.96:0/3291139059 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1e4815ed20 0x7f1e4815f1b0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:58.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.800+0000 7f1e4da00640 1 --2- 172.21.15.96:0/3291139059 >> v2:172.21.15.123:3300/0 conn(0x7f1e480b3b50 0x7f1e480d3190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:58.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.800+0000 7f1e4da00640 1 -- 172.21.15.96:0/3291139059 >> 172.21.15.96:0/3291139059 conn(0x7f1e4808bfc0 msgr2=0x7f1e480d4650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:58.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.801+0000 7f1e4da00640 1 -- 172.21.15.96:0/3291139059 shutdown_connections 2024-05-28T06:08:58.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.801+0000 7f1e4da00640 1 -- 172.21.15.96:0/3291139059 wait complete. 2024-05-28T06:08:58.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.802+0000 7f1e4da00640 1 Processor -- start 2024-05-28T06:08:58.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.802+0000 7f1e4da00640 1 -- start start 2024-05-28T06:08:58.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.803+0000 7f1e4da00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1e480b3b50 0x7f1e48155480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:58.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.803+0000 7f1e4da00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1e480d2db0 0x7f1e480b3f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:58.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.803+0000 7f1e4da00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1e4815ed20 0x7f1e4815c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:58.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.803+0000 7f1e4da00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f1e48162fe0 con 0x7f1e480b3b50 2024-05-28T06:08:58.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.803+0000 7f1e4da00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f1e48162d00 con 0x7f1e4815ed20 2024-05-28T06:08:58.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.803+0000 7f1e4da00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f1e480d04a0 con 0x7f1e480d2db0 2024-05-28T06:08:58.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.803+0000 7f1e46a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1e480b3b50 0x7f1e48155480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:58.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.804+0000 7f1e46a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1e480b3b50 0x7f1e48155480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51412/0 (socket says 172.21.15.96:51412) 2024-05-28T06:08:58.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.804+0000 7f1e46a00640 1 -- 172.21.15.96:0/3860182408 learned_addr learned my addr 172.21.15.96:0/3860182408 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:08:58.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.804+0000 7f1e3e000640 1 --2- 172.21.15.96:0/3860182408 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1e480d2db0 0x7f1e480b3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:58.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.804+0000 7f1e46a00640 1 -- 172.21.15.96:0/3860182408 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1e480d2db0 msgr2=0x7f1e480b3f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:58.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.804+0000 7f1e47400640 1 --2- 172.21.15.96:0/3860182408 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1e4815ed20 0x7f1e4815c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:58.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.804+0000 7f1e46a00640 1 --2- 172.21.15.96:0/3860182408 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1e480d2db0 0x7f1e480b3f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:58.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.804+0000 7f1e46a00640 1 -- 172.21.15.96:0/3860182408 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1e4815ed20 msgr2=0x7f1e4815c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:58.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.804+0000 7f1e46a00640 1 --2- 172.21.15.96:0/3860182408 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1e4815ed20 0x7f1e4815c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:58.807 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.804+0000 7f1e46a00640 1 -- 172.21.15.96:0/3860182408 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e480b6a20 con 0x7f1e480b3b50 2024-05-28T06:08:58.807 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.804+0000 7f1e47400640 1 --2- 172.21.15.96:0/3860182408 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1e4815ed20 0x7f1e4815c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:08:58.807 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.804+0000 7f1e46a00640 1 --2- 172.21.15.96:0/3860182408 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1e480b3b50 0x7f1e48155480 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f1e3800c440 tx=0x7f1e3800c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:58.807 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.804+0000 7f1e3f400640 1 -- 172.21.15.96:0/3860182408 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e38017070 con 0x7f1e480b3b50 2024-05-28T06:08:58.808 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.805+0000 7f1e3f400640 1 -- 172.21.15.96:0/3860182408 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e380035f0 con 0x7f1e480b3b50 2024-05-28T06:08:58.808 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.805+0000 7f1e4da00640 1 -- 172.21.15.96:0/3860182408 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e480b6d00 con 0x7f1e480b3b50 2024-05-28T06:08:58.808 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.805+0000 7f1e3f400640 1 -- 172.21.15.96:0/3860182408 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e380039e0 con 0x7f1e480b3b50 2024-05-28T06:08:58.808 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.805+0000 7f1e4da00640 1 -- 172.21.15.96:0/3860182408 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e480c8800 con 0x7f1e480b3b50 2024-05-28T06:08:58.808 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.806+0000 7f1e4da00640 1 -- 172.21.15.96:0/3860182408 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e480689e0 con 0x7f1e480b3b50 2024-05-28T06:08:58.808 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.807+0000 7f1e3f400640 1 -- 172.21.15.96:0/3860182408 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 9) ==== 91295+0+0 (secure 0 0 0) 0x7f1e3801a070 con 0x7f1e480b3b50 2024-05-28T06:08:58.809 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.807+0000 7f1e3f400640 1 --2- 172.21.15.96:0/3860182408 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1e2406cbc0 0x7f1e2406f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:58.809 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.808+0000 7f1e3f400640 1 -- 172.21.15.96:0/3860182408 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5737+0+0 (secure 0 0 0) 0x7f1e3808b4b0 con 0x7f1e480b3b50 2024-05-28T06:08:58.809 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.808+0000 7f1e3e000640 1 --2- 172.21.15.96:0/3860182408 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1e2406cbc0 0x7f1e2406f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:58.811 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.810+0000 7f1e3e000640 1 --2- 172.21.15.96:0/3860182408 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1e2406cbc0 0x7f1e2406f080 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f1e4806e840 tx=0x7f1e30008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:58.813 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:58.812+0000 7f1e3f400640 1 -- 172.21.15.96:0/3860182408 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1e38059450 con 0x7f1e480b3b50 2024-05-28T06:08:59.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.000+0000 7f1e4da00640 1 -- 172.21.15.96:0/3860182408 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1e480d2af0 con 0x7f1e480b3b50 2024-05-28T06:08:59.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.001+0000 7f1e3f400640 1 -- 172.21.15.96:0/3860182408 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v45) ==== 74+0+16668 (secure 0 0 0) 0x7f1e380010c0 con 0x7f1e480b3b50 2024-05-28T06:08:59.003 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:08:59.003 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":45,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:08:56.025778+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-05-28T06:08:52.257477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-05-28T06:08:53.459376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:08:59.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.004+0000 7f1e4da00640 1 -- 172.21.15.96:0/3860182408 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1e2406cbc0 msgr2=0x7f1e2406f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:59.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.004+0000 7f1e4da00640 1 --2- 172.21.15.96:0/3860182408 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1e2406cbc0 0x7f1e2406f080 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f1e4806e840 tx=0x7f1e30008040 comp rx=0 tx=0).stop 2024-05-28T06:08:59.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.004+0000 7f1e4da00640 1 -- 172.21.15.96:0/3860182408 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1e480b3b50 msgr2=0x7f1e48155480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:59.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.004+0000 7f1e4da00640 1 --2- 172.21.15.96:0/3860182408 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1e480b3b50 0x7f1e48155480 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f1e3800c440 tx=0x7f1e3800c900 comp rx=0 tx=0).stop 2024-05-28T06:08:59.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.004+0000 7f1e4da00640 1 -- 172.21.15.96:0/3860182408 shutdown_connections 2024-05-28T06:08:59.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.004+0000 7f1e4da00640 1 --2- 172.21.15.96:0/3860182408 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1e2406cbc0 0x7f1e2406f080 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:59.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.004+0000 7f1e4da00640 1 --2- 172.21.15.96:0/3860182408 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1e4815ed20 0x7f1e4815c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:59.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.004+0000 7f1e4da00640 1 --2- 172.21.15.96:0/3860182408 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1e480d2db0 0x7f1e480b3f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:59.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.004+0000 7f1e4da00640 1 --2- 172.21.15.96:0/3860182408 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1e480b3b50 0x7f1e48155480 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:59.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.004+0000 7f1e4da00640 1 -- 172.21.15.96:0/3860182408 >> 172.21.15.96:0/3860182408 conn(0x7f1e4808bfc0 msgr2=0x7f1e480b3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:59.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.004+0000 7f1e4da00640 1 -- 172.21.15.96:0/3860182408 shutdown_connections 2024-05-28T06:08:59.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.004+0000 7f1e4da00640 1 -- 172.21.15.96:0/3860182408 wait complete. 2024-05-28T06:08:59.025 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-05-28T06:08:59.025 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-28T06:08:59.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.221+0000 7f2956000640 1 Processor -- start 2024-05-28T06:08:59.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.221+0000 7f2956000640 1 -- start start 2024-05-28T06:08:59.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.223+0000 7f2956000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f29500d0bb0 0x7f29500b3580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:59.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.223+0000 7f2956000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f295006f7b0 0x7f29500b3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:59.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.223+0000 7f2956000640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f295015b9d0 con 0x7f295015b5f0 2024-05-28T06:08:59.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.223+0000 7f2956000640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f295015f150 con 0x7f29500d0bb0 2024-05-28T06:08:59.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.223+0000 7f2956000640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f295015f430 con 0x7f295006f7b0 2024-05-28T06:08:59.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.223+0000 7f2946a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f29500d0bb0 0x7f29500b3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:59.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.223+0000 7f294f400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f295015b5f0 0x7f29500d04a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:51180/0 (socket says 172.21.15.96:51180) 2024-05-28T06:08:59.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.223+0000 7f294f400640 1 -- 172.21.15.96:0/3136175069 learned_addr learned my addr 172.21.15.96:0/3136175069 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:08:59.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.223+0000 7f2946a00640 1 -- 172.21.15.96:0/3136175069 >> v2:172.21.15.123:3300/0 conn(0x7f295006f7b0 msgr2=0x7f29500b3ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:59.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.223+0000 7f2946a00640 1 --2- 172.21.15.96:0/3136175069 >> v2:172.21.15.123:3300/0 conn(0x7f295006f7b0 0x7f29500b3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:59.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.223+0000 7f2946a00640 1 -- 172.21.15.96:0/3136175069 >> v1:172.21.15.96:6789/0 conn(0x7f295015b5f0 legacy=0x7f29500d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:59.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.223+0000 7f2946a00640 1 -- 172.21.15.96:0/3136175069 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f295015f710 con 0x7f29500d0bb0 2024-05-28T06:08:59.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.223+0000 7f294ea00640 1 -- 172.21.15.96:0/3136175069 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1393721886 0 0) 0x7f2940001110 con 0x7f295015b5f0 2024-05-28T06:08:59.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.224+0000 7f2946a00640 1 --2- 172.21.15.96:0/3136175069 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f29500d0bb0 0x7f29500b3580 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f295006f040 tx=0x7f293802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1c01c4a1005c517 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:59.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.225+0000 7f294ea00640 1 -- 172.21.15.96:0/3136175069 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2938037070 con 0x7f29500d0bb0 2024-05-28T06:08:59.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.225+0000 7f294ea00640 1 -- 172.21.15.96:0/3136175069 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2938032440 con 0x7f29500d0bb0 2024-05-28T06:08:59.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.225+0000 7f294ea00640 1 -- 172.21.15.96:0/3136175069 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2938032810 con 0x7f29500d0bb0 2024-05-28T06:08:59.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.225+0000 7f2956000640 1 -- 172.21.15.96:0/3136175069 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f29500d0bb0 msgr2=0x7f29500b3580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:59.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.225+0000 7f2956000640 1 --2- 172.21.15.96:0/3136175069 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f29500d0bb0 0x7f29500b3580 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f295006f040 tx=0x7f293802a1a0 comp rx=0 tx=0).stop 2024-05-28T06:08:59.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.225+0000 7f2956000640 1 -- 172.21.15.96:0/3136175069 shutdown_connections 2024-05-28T06:08:59.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.225+0000 7f2956000640 1 --2- 172.21.15.96:0/3136175069 >> v2:172.21.15.123:3300/0 conn(0x7f295006f7b0 0x7f29500b3ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:59.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.225+0000 7f2956000640 1 --2- 172.21.15.96:0/3136175069 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f29500d0bb0 0x7f29500b3580 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:59.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.225+0000 7f2956000640 1 -- 172.21.15.96:0/3136175069 >> 172.21.15.96:0/3136175069 conn(0x7f295006bae0 msgr2=0x7f295006bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:59.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.225+0000 7f2956000640 1 -- 172.21.15.96:0/3136175069 shutdown_connections 2024-05-28T06:08:59.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.226+0000 7f2956000640 1 -- 172.21.15.96:0/3136175069 wait complete. 2024-05-28T06:08:59.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.227+0000 7f2956000640 1 Processor -- start 2024-05-28T06:08:59.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.227+0000 7f2956000640 1 -- start start 2024-05-28T06:08:59.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.228+0000 7f2956000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f295006f7b0 0x7f2950156db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:59.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.228+0000 7f2956000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f29500d0bb0 0x7f295015bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:59.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.228+0000 7f2956000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f295015b5f0 0x7f29501e5f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:59.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.228+0000 7f2956000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f29500b4e50 con 0x7f29500d0bb0 2024-05-28T06:08:59.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.228+0000 7f2956000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f29500b5160 con 0x7f295006f7b0 2024-05-28T06:08:59.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.228+0000 7f2956000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f295015ff10 con 0x7f295015b5f0 2024-05-28T06:08:59.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.228+0000 7f2946a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f29500d0bb0 0x7f295015bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:59.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.228+0000 7f2946a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f29500d0bb0 0x7f295015bbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47462/0 (socket says 172.21.15.96:47462) 2024-05-28T06:08:59.231 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.228+0000 7f2946a00640 1 -- 172.21.15.96:0/1966498755 learned_addr learned my addr 172.21.15.96:0/1966498755 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:08:59.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.229+0000 7f294f400640 1 --2- 172.21.15.96:0/1966498755 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f295006f7b0 0x7f2950156db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:59.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.229+0000 7f2946a00640 1 -- 172.21.15.96:0/1966498755 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f295015b5f0 msgr2=0x7f29501e5f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:59.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.229+0000 7f294fe00640 1 --2- 172.21.15.96:0/1966498755 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f295015b5f0 0x7f29501e5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:59.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.229+0000 7f2946a00640 1 --2- 172.21.15.96:0/1966498755 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f295015b5f0 0x7f29501e5f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:59.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.229+0000 7f2946a00640 1 -- 172.21.15.96:0/1966498755 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f295006f7b0 msgr2=0x7f2950156db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:59.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.229+0000 7f2946a00640 1 --2- 172.21.15.96:0/1966498755 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f295006f7b0 0x7f2950156db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:59.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.229+0000 7f2946a00640 1 -- 172.21.15.96:0/1966498755 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f295015dd70 con 0x7f29500d0bb0 2024-05-28T06:08:59.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.229+0000 7f294f400640 1 --2- 172.21.15.96:0/1966498755 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f295006f7b0 0x7f2950156db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:08:59.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.229+0000 7f2946a00640 1 --2- 172.21.15.96:0/1966498755 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f29500d0bb0 0x7f295015bbb0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f2938000c00 tx=0x7f29380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:59.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.229+0000 7f2947e00640 1 -- 172.21.15.96:0/1966498755 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2938041070 con 0x7f29500d0bb0 2024-05-28T06:08:59.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.230+0000 7f2947e00640 1 -- 172.21.15.96:0/1966498755 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f293803c950 con 0x7f29500d0bb0 2024-05-28T06:08:59.233 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.230+0000 7f2947e00640 1 -- 172.21.15.96:0/1966498755 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2938037040 con 0x7f29500d0bb0 2024-05-28T06:08:59.233 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.230+0000 7f2956000640 1 -- 172.21.15.96:0/1966498755 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f295015e050 con 0x7f29500d0bb0 2024-05-28T06:08:59.233 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.230+0000 7f2956000640 1 -- 172.21.15.96:0/1966498755 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29501572f0 con 0x7f29500d0bb0 2024-05-28T06:08:59.233 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.231+0000 7f2956000640 1 -- 172.21.15.96:0/1966498755 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29500ce640 con 0x7f29500d0bb0 2024-05-28T06:08:59.233 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.232+0000 7f2947e00640 1 -- 172.21.15.96:0/1966498755 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 9) ==== 91295+0+0 (secure 0 0 0) 0x7f2938044070 con 0x7f29500d0bb0 2024-05-28T06:08:59.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.233+0000 7f2947e00640 1 --2- 172.21.15.96:0/1966498755 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f292c06cbc0 0x7f292c06f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:59.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.233+0000 7f2947e00640 1 -- 172.21.15.96:0/1966498755 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5737+0+0 (secure 0 0 0) 0x7f29380acf90 con 0x7f29500d0bb0 2024-05-28T06:08:59.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.233+0000 7f294f400640 1 --2- 172.21.15.96:0/1966498755 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f292c06cbc0 0x7f292c06f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:59.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.234+0000 7f294f400640 1 --2- 172.21.15.96:0/1966498755 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f292c06cbc0 0x7f292c06f080 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f29400013b0 tx=0x7f2940009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:59.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.237+0000 7f2947e00640 1 -- 172.21.15.96:0/1966498755 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f29500ce640 con 0x7f29500d0bb0 2024-05-28T06:08:59.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.433+0000 7f2956000640 1 -- 172.21.15.96:0/1966498755 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f29500689e0 con 0x7f292c06cbc0 2024-05-28T06:08:59.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.454+0000 7f2947e00640 1 -- 172.21.15.96:0/1966498755 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+246639 (secure 0 0 0) 0x7f29500689e0 con 0x7f292c06cbc0 2024-05-28T06:08:59.459 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:08:59.461 INFO:teuthology.orchestra.run.smithi096.stderr:dumped all 2024-05-28T06:08:59.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.459+0000 7f2956000640 1 -- 172.21.15.96:0/1966498755 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f292c06cbc0 msgr2=0x7f292c06f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:59.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.459+0000 7f2956000640 1 --2- 172.21.15.96:0/1966498755 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f292c06cbc0 0x7f292c06f080 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f29400013b0 tx=0x7f2940009040 comp rx=0 tx=0).stop 2024-05-28T06:08:59.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.459+0000 7f2956000640 1 -- 172.21.15.96:0/1966498755 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f29500d0bb0 msgr2=0x7f295015bbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:59.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.459+0000 7f2956000640 1 --2- 172.21.15.96:0/1966498755 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f29500d0bb0 0x7f295015bbb0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f2938000c00 tx=0x7f29380046c0 comp rx=0 tx=0).stop 2024-05-28T06:08:59.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.460+0000 7f2956000640 1 -- 172.21.15.96:0/1966498755 shutdown_connections 2024-05-28T06:08:59.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.460+0000 7f2956000640 1 --2- 172.21.15.96:0/1966498755 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f292c06cbc0 0x7f292c06f080 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:59.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.460+0000 7f2956000640 1 --2- 172.21.15.96:0/1966498755 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f295015b5f0 0x7f29501e5f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:59.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.460+0000 7f2956000640 1 --2- 172.21.15.96:0/1966498755 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f29500d0bb0 0x7f295015bbb0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:59.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.460+0000 7f2956000640 1 --2- 172.21.15.96:0/1966498755 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f295006f7b0 0x7f2950156db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:59.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.460+0000 7f2956000640 1 -- 172.21.15.96:0/1966498755 >> 172.21.15.96:0/1966498755 conn(0x7f295006bae0 msgr2=0x7f29500af410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:59.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.460+0000 7f2956000640 1 -- 172.21.15.96:0/1966498755 shutdown_connections 2024-05-28T06:08:59.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.460+0000 7f2956000640 1 -- 172.21.15.96:0/1966498755 wait complete. 2024-05-28T06:08:59.479 INFO:teuthology.orchestra.run.smithi096.stdout:{"pg_ready":true,"pg_map":{"version":187,"stamp":"2024-05-28T06:08:57.591306+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":28,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":283660,"kb_used_data":5072,"kb_used_omap":12,"kb_used_meta":278451,"kb_avail":754691060,"statfs":{"total":773094113280,"available":772803645440,"internally_reserved":0,"allocated":5193728,"data_stored":2795098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":285134384},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.591040"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.030805+0000","last_change":"2024-05-28T06:08:54.030805+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.030805+0000","last_undegraded":"2024-05-28T06:08:54.030805+0000","last_fullsized":"2024-05-28T06:08:54.030805+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.023029+0000","last_change":"2024-05-28T06:08:54.023029+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.023029+0000","last_undegraded":"2024-05-28T06:08:54.023029+0000","last_fullsized":"2024-05-28T06:08:54.023029+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.019511+0000","last_change":"2024-05-28T06:08:54.019511+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.019511+0000","last_undegraded":"2024-05-28T06:08:54.019511+0000","last_fullsized":"2024-05-28T06:08:54.019511+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.031775+0000","last_change":"2024-05-28T06:08:54.031775+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.031775+0000","last_undegraded":"2024-05-28T06:08:54.031775+0000","last_fullsized":"2024-05-28T06:08:54.031775+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.021863+0000","last_change":"2024-05-28T06:08:54.021863+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.021863+0000","last_undegraded":"2024-05-28T06:08:54.021863+0000","last_fullsized":"2024-05-28T06:08:54.021863+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.023805+0000","last_change":"2024-05-28T06:08:54.023805+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.023805+0000","last_undegraded":"2024-05-28T06:08:54.023805+0000","last_fullsized":"2024-05-28T06:08:54.023805+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.029593+0000","last_change":"2024-05-28T06:08:54.029593+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.029593+0000","last_undegraded":"2024-05-28T06:08:54.029593+0000","last_fullsized":"2024-05-28T06:08:54.029593+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.022144+0000","last_change":"2024-05-28T06:08:54.022144+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.022144+0000","last_undegraded":"2024-05-28T06:08:54.022144+0000","last_fullsized":"2024-05-28T06:08:54.022144+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.030711+0000","last_change":"2024-05-28T06:08:54.030711+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.030711+0000","last_undegraded":"2024-05-28T06:08:54.030711+0000","last_fullsized":"2024-05-28T06:08:54.030711+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.024096+0000","last_change":"2024-05-28T06:08:54.024096+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.024096+0000","last_undegraded":"2024-05-28T06:08:54.024096+0000","last_fullsized":"2024-05-28T06:08:54.024096+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.018302+0000","last_change":"2024-05-28T06:08:54.018302+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.018302+0000","last_undegraded":"2024-05-28T06:08:54.018302+0000","last_fullsized":"2024-05-28T06:08:54.018302+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.024586+0000","last_change":"2024-05-28T06:08:54.024586+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.024586+0000","last_undegraded":"2024-05-28T06:08:54.024586+0000","last_fullsized":"2024-05-28T06:08:54.024586+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.014791+0000","last_change":"2024-05-28T06:08:54.014791+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.014791+0000","last_undegraded":"2024-05-28T06:08:54.014791+0000","last_fullsized":"2024-05-28T06:08:54.014791+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.027464+0000","last_change":"2024-05-28T06:08:54.027464+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.027464+0000","last_undegraded":"2024-05-28T06:08:54.027464+0000","last_fullsized":"2024-05-28T06:08:54.027464+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.025774+0000","last_change":"2024-05-28T06:08:54.025774+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.025774+0000","last_undegraded":"2024-05-28T06:08:54.025774+0000","last_fullsized":"2024-05-28T06:08:54.025774+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.029213+0000","last_change":"2024-05-28T06:08:54.029213+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.029213+0000","last_undegraded":"2024-05-28T06:08:54.029213+0000","last_fullsized":"2024-05-28T06:08:54.029213+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.014182+0000","last_change":"2024-05-28T06:08:54.014182+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.014182+0000","last_undegraded":"2024-05-28T06:08:54.014182+0000","last_fullsized":"2024-05-28T06:08:54.014182+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.016245+0000","last_change":"2024-05-28T06:08:54.016245+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.016245+0000","last_undegraded":"2024-05-28T06:08:54.016245+0000","last_fullsized":"2024-05-28T06:08:54.016245+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.032693+0000","last_change":"2024-05-28T06:08:54.032693+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.032693+0000","last_undegraded":"2024-05-28T06:08:54.032693+0000","last_fullsized":"2024-05-28T06:08:54.032693+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.014442+0000","last_change":"2024-05-28T06:08:54.014442+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.014442+0000","last_undegraded":"2024-05-28T06:08:54.014442+0000","last_fullsized":"2024-05-28T06:08:54.014442+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.025473+0000","last_change":"2024-05-28T06:08:54.025473+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.025473+0000","last_undegraded":"2024-05-28T06:08:54.025473+0000","last_fullsized":"2024-05-28T06:08:54.025473+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.018238+0000","last_change":"2024-05-28T06:08:54.018238+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.018238+0000","last_undegraded":"2024-05-28T06:08:54.018238+0000","last_fullsized":"2024-05-28T06:08:54.018238+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.029393+0000","last_change":"2024-05-28T06:08:54.029393+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.029393+0000","last_undegraded":"2024-05-28T06:08:54.029393+0000","last_fullsized":"2024-05-28T06:08:54.029393+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.028295+0000","last_change":"2024-05-28T06:08:54.028295+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.028295+0000","last_undegraded":"2024-05-28T06:08:54.028295+0000","last_fullsized":"2024-05-28T06:08:54.028295+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.037153+0000","last_change":"2024-05-28T06:08:54.037153+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.037153+0000","last_undegraded":"2024-05-28T06:08:54.037153+0000","last_fullsized":"2024-05-28T06:08:54.037153+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.015960+0000","last_change":"2024-05-28T06:08:54.015960+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.015960+0000","last_undegraded":"2024-05-28T06:08:54.015960+0000","last_fullsized":"2024-05-28T06:08:54.015960+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.014740+0000","last_change":"2024-05-28T06:08:54.014740+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.014740+0000","last_undegraded":"2024-05-28T06:08:54.014740+0000","last_fullsized":"2024-05-28T06:08:54.014740+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.023762+0000","last_change":"2024-05-28T06:08:54.023762+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.023762+0000","last_undegraded":"2024-05-28T06:08:54.023762+0000","last_fullsized":"2024-05-28T06:08:54.023762+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.030708+0000","last_change":"2024-05-28T06:08:54.030708+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.030708+0000","last_undegraded":"2024-05-28T06:08:54.030708+0000","last_fullsized":"2024-05-28T06:08:54.030708+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.017736+0000","last_change":"2024-05-28T06:08:54.017736+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.017736+0000","last_undegraded":"2024-05-28T06:08:54.017736+0000","last_fullsized":"2024-05-28T06:08:54.017736+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.019413+0000","last_change":"2024-05-28T06:08:54.019413+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.019413+0000","last_undegraded":"2024-05-28T06:08:54.019413+0000","last_fullsized":"2024-05-28T06:08:54.019413+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.016151+0000","last_change":"2024-05-28T06:08:54.016151+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.016151+0000","last_undegraded":"2024-05-28T06:08:54.016151+0000","last_fullsized":"2024-05-28T06:08:54.016151+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.028257+0000","last_change":"2024-05-28T06:08:54.028257+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.028257+0000","last_undegraded":"2024-05-28T06:08:54.028257+0000","last_fullsized":"2024-05-28T06:08:54.028257+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.045658+0000","last_change":"2024-05-28T06:08:54.044108+0000","last_active":"2024-05-28T06:08:54.045658+0000","last_peered":"2024-05-28T06:08:54.045658+0000","last_clean":"2024-05-28T06:08:54.045658+0000","last_became_active":"2024-05-28T06:08:54.043637+0000","last_became_peered":"2024-05-28T06:08:54.043637+0000","last_unstale":"2024-05-28T06:08:54.045658+0000","last_undegraded":"2024-05-28T06:08:54.045658+0000","last_fullsized":"2024-05-28T06:08:54.045658+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:11:14.276087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.036622+0000","last_change":"2024-05-28T06:08:54.036622+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.036622+0000","last_undegraded":"2024-05-28T06:08:54.036622+0000","last_fullsized":"2024-05-28T06:08:54.036622+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.039961+0000","last_change":"2024-05-28T06:08:54.036739+0000","last_active":"2024-05-28T06:08:54.039961+0000","last_peered":"2024-05-28T06:08:54.039961+0000","last_clean":"2024-05-28T06:08:54.039961+0000","last_became_active":"2024-05-28T06:08:54.036446+0000","last_became_peered":"2024-05-28T06:08:54.036446+0000","last_unstale":"2024-05-28T06:08:54.039961+0000","last_undegraded":"2024-05-28T06:08:54.039961+0000","last_fullsized":"2024-05-28T06:08:54.039961+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:07:41.209833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.022836+0000","last_change":"2024-05-28T06:08:54.022836+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.022836+0000","last_undegraded":"2024-05-28T06:08:54.022836+0000","last_fullsized":"2024-05-28T06:08:54.022836+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.040347+0000","last_change":"2024-05-28T06:08:54.035269+0000","last_active":"2024-05-28T06:08:54.040347+0000","last_peered":"2024-05-28T06:08:54.040347+0000","last_clean":"2024-05-28T06:08:54.040347+0000","last_became_active":"2024-05-28T06:08:54.034916+0000","last_became_peered":"2024-05-28T06:08:54.034916+0000","last_unstale":"2024-05-28T06:08:54.040347+0000","last_undegraded":"2024-05-28T06:08:54.040347+0000","last_fullsized":"2024-05-28T06:08:54.040347+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:37:34.767912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.026952+0000","last_change":"2024-05-28T06:08:54.026952+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.026952+0000","last_undegraded":"2024-05-28T06:08:54.026952+0000","last_fullsized":"2024-05-28T06:08:54.026952+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.046591+0000","last_change":"2024-05-28T06:08:54.044956+0000","last_active":"2024-05-28T06:08:54.046591+0000","last_peered":"2024-05-28T06:08:54.046591+0000","last_clean":"2024-05-28T06:08:54.046591+0000","last_became_active":"2024-05-28T06:08:54.044646+0000","last_became_peered":"2024-05-28T06:08:54.044646+0000","last_unstale":"2024-05-28T06:08:54.046591+0000","last_undegraded":"2024-05-28T06:08:54.046591+0000","last_fullsized":"2024-05-28T06:08:54.046591+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:34:31.576736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.014512+0000","last_change":"2024-05-28T06:08:54.014512+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.014512+0000","last_undegraded":"2024-05-28T06:08:54.014512+0000","last_fullsized":"2024-05-28T06:08:54.014512+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.045585+0000","last_change":"2024-05-28T06:08:54.042495+0000","last_active":"2024-05-28T06:08:54.045585+0000","last_peered":"2024-05-28T06:08:54.045585+0000","last_clean":"2024-05-28T06:08:54.045585+0000","last_became_active":"2024-05-28T06:08:54.042143+0000","last_became_peered":"2024-05-28T06:08:54.042143+0000","last_unstale":"2024-05-28T06:08:54.045585+0000","last_undegraded":"2024-05-28T06:08:54.045585+0000","last_fullsized":"2024-05-28T06:08:54.045585+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:20:27.131737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.022747+0000","last_change":"2024-05-28T06:08:54.022747+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.022747+0000","last_undegraded":"2024-05-28T06:08:54.022747+0000","last_fullsized":"2024-05-28T06:08:54.022747+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.039866+0000","last_change":"2024-05-28T06:08:54.030180+0000","last_active":"2024-05-28T06:08:54.039866+0000","last_peered":"2024-05-28T06:08:54.039866+0000","last_clean":"2024-05-28T06:08:54.039866+0000","last_became_active":"2024-05-28T06:08:54.029892+0000","last_became_peered":"2024-05-28T06:08:54.029892+0000","last_unstale":"2024-05-28T06:08:54.039866+0000","last_undegraded":"2024-05-28T06:08:54.039866+0000","last_fullsized":"2024-05-28T06:08:54.039866+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:16:17.652829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.021598+0000","last_change":"2024-05-28T06:08:54.021598+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.021598+0000","last_undegraded":"2024-05-28T06:08:54.021598+0000","last_fullsized":"2024-05-28T06:08:54.021598+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.038065+0000","last_change":"2024-05-28T06:08:54.035697+0000","last_active":"2024-05-28T06:08:54.038065+0000","last_peered":"2024-05-28T06:08:54.038065+0000","last_clean":"2024-05-28T06:08:54.038065+0000","last_became_active":"2024-05-28T06:08:54.032931+0000","last_became_peered":"2024-05-28T06:08:54.032931+0000","last_unstale":"2024-05-28T06:08:54.038065+0000","last_undegraded":"2024-05-28T06:08:54.038065+0000","last_fullsized":"2024-05-28T06:08:54.038065+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:29:27.490700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.030054+0000","last_change":"2024-05-28T06:08:54.030054+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.030054+0000","last_undegraded":"2024-05-28T06:08:54.030054+0000","last_fullsized":"2024-05-28T06:08:54.030054+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.036080+0000","last_change":"2024-05-28T06:08:54.032388+0000","last_active":"2024-05-28T06:08:54.036080+0000","last_peered":"2024-05-28T06:08:54.036080+0000","last_clean":"2024-05-28T06:08:54.036080+0000","last_became_active":"2024-05-28T06:08:54.032095+0000","last_became_peered":"2024-05-28T06:08:54.032095+0000","last_unstale":"2024-05-28T06:08:54.036080+0000","last_undegraded":"2024-05-28T06:08:54.036080+0000","last_fullsized":"2024-05-28T06:08:54.036080+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:21:39.812262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.014966+0000","last_change":"2024-05-28T06:08:54.014966+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.014966+0000","last_undegraded":"2024-05-28T06:08:54.014966+0000","last_fullsized":"2024-05-28T06:08:54.014966+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.039945+0000","last_change":"2024-05-28T06:08:54.027262+0000","last_active":"2024-05-28T06:08:54.039945+0000","last_peered":"2024-05-28T06:08:54.039945+0000","last_clean":"2024-05-28T06:08:54.039945+0000","last_became_active":"2024-05-28T06:08:54.026066+0000","last_became_peered":"2024-05-28T06:08:54.026066+0000","last_unstale":"2024-05-28T06:08:54.039945+0000","last_undegraded":"2024-05-28T06:08:54.039945+0000","last_fullsized":"2024-05-28T06:08:54.039945+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:16:01.996383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.030190+0000","last_change":"2024-05-28T06:08:54.030190+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.030190+0000","last_undegraded":"2024-05-28T06:08:54.030190+0000","last_fullsized":"2024-05-28T06:08:54.030190+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.041630+0000","last_change":"2024-05-28T06:08:54.040877+0000","last_active":"2024-05-28T06:08:54.041630+0000","last_peered":"2024-05-28T06:08:54.041630+0000","last_clean":"2024-05-28T06:08:54.041630+0000","last_became_active":"2024-05-28T06:08:54.040615+0000","last_became_peered":"2024-05-28T06:08:54.040615+0000","last_unstale":"2024-05-28T06:08:54.041630+0000","last_undegraded":"2024-05-28T06:08:54.041630+0000","last_fullsized":"2024-05-28T06:08:54.041630+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:40:59.926424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.018762+0000","last_change":"2024-05-28T06:08:54.018762+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.018762+0000","last_undegraded":"2024-05-28T06:08:54.018762+0000","last_fullsized":"2024-05-28T06:08:54.018762+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.048326+0000","last_change":"2024-05-28T06:08:54.046805+0000","last_active":"2024-05-28T06:08:54.048326+0000","last_peered":"2024-05-28T06:08:54.048326+0000","last_clean":"2024-05-28T06:08:54.048326+0000","last_became_active":"2024-05-28T06:08:54.046336+0000","last_became_peered":"2024-05-28T06:08:54.046336+0000","last_unstale":"2024-05-28T06:08:54.048326+0000","last_undegraded":"2024-05-28T06:08:54.048326+0000","last_fullsized":"2024-05-28T06:08:54.048326+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:19:17.873244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.032123+0000","last_change":"2024-05-28T06:08:54.032123+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.032123+0000","last_undegraded":"2024-05-28T06:08:54.032123+0000","last_fullsized":"2024-05-28T06:08:54.032123+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.031479+0000","last_change":"2024-05-28T06:08:54.029622+0000","last_active":"2024-05-28T06:08:54.031479+0000","last_peered":"2024-05-28T06:08:54.031479+0000","last_clean":"2024-05-28T06:08:54.031479+0000","last_became_active":"2024-05-28T06:08:54.026140+0000","last_became_peered":"2024-05-28T06:08:54.026140+0000","last_unstale":"2024-05-28T06:08:54.031479+0000","last_undegraded":"2024-05-28T06:08:54.031479+0000","last_fullsized":"2024-05-28T06:08:54.031479+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:14:10.883135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.028947+0000","last_change":"2024-05-28T06:08:54.028947+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.028947+0000","last_undegraded":"2024-05-28T06:08:54.028947+0000","last_fullsized":"2024-05-28T06:08:54.028947+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.041714+0000","last_change":"2024-05-28T06:08:54.038320+0000","last_active":"2024-05-28T06:08:54.041714+0000","last_peered":"2024-05-28T06:08:54.041714+0000","last_clean":"2024-05-28T06:08:54.041714+0000","last_became_active":"2024-05-28T06:08:54.037824+0000","last_became_peered":"2024-05-28T06:08:54.037824+0000","last_unstale":"2024-05-28T06:08:54.041714+0000","last_undegraded":"2024-05-28T06:08:54.041714+0000","last_fullsized":"2024-05-28T06:08:54.041714+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:47:29.434576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.021818+0000","last_change":"2024-05-28T06:08:54.021818+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.021818+0000","last_undegraded":"2024-05-28T06:08:54.021818+0000","last_fullsized":"2024-05-28T06:08:54.021818+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.048280+0000","last_change":"2024-05-28T06:08:54.046264+0000","last_active":"2024-05-28T06:08:54.048280+0000","last_peered":"2024-05-28T06:08:54.048280+0000","last_clean":"2024-05-28T06:08:54.048280+0000","last_became_active":"2024-05-28T06:08:54.045957+0000","last_became_peered":"2024-05-28T06:08:54.045957+0000","last_unstale":"2024-05-28T06:08:54.048280+0000","last_undegraded":"2024-05-28T06:08:54.048280+0000","last_fullsized":"2024-05-28T06:08:54.048280+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:47:29.434576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.024967+0000","last_change":"2024-05-28T06:08:54.024967+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.024967+0000","last_undegraded":"2024-05-28T06:08:54.024967+0000","last_fullsized":"2024-05-28T06:08:54.024967+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.041658+0000","last_change":"2024-05-28T06:08:54.038131+0000","last_active":"2024-05-28T06:08:54.041658+0000","last_peered":"2024-05-28T06:08:54.041658+0000","last_clean":"2024-05-28T06:08:54.041658+0000","last_became_active":"2024-05-28T06:08:54.037578+0000","last_became_peered":"2024-05-28T06:08:54.037578+0000","last_unstale":"2024-05-28T06:08:54.041658+0000","last_undegraded":"2024-05-28T06:08:54.041658+0000","last_fullsized":"2024-05-28T06:08:54.041658+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:52:34.490485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.029151+0000","last_change":"2024-05-28T06:08:54.029151+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.029151+0000","last_undegraded":"2024-05-28T06:08:54.029151+0000","last_fullsized":"2024-05-28T06:08:54.029151+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.037597+0000","last_change":"2024-05-28T06:08:54.034498+0000","last_active":"2024-05-28T06:08:54.037597+0000","last_peered":"2024-05-28T06:08:54.037597+0000","last_clean":"2024-05-28T06:08:54.037597+0000","last_became_active":"2024-05-28T06:08:54.034152+0000","last_became_peered":"2024-05-28T06:08:54.034152+0000","last_unstale":"2024-05-28T06:08:54.037597+0000","last_undegraded":"2024-05-28T06:08:54.037597+0000","last_fullsized":"2024-05-28T06:08:54.037597+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:57:06.361350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.015569+0000","last_change":"2024-05-28T06:08:54.015569+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.015569+0000","last_undegraded":"2024-05-28T06:08:54.015569+0000","last_fullsized":"2024-05-28T06:08:54.015569+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.040334+0000","last_change":"2024-05-28T06:08:54.035386+0000","last_active":"2024-05-28T06:08:54.040334+0000","last_peered":"2024-05-28T06:08:54.040334+0000","last_clean":"2024-05-28T06:08:54.040334+0000","last_became_active":"2024-05-28T06:08:54.033511+0000","last_became_peered":"2024-05-28T06:08:54.033511+0000","last_unstale":"2024-05-28T06:08:54.040334+0000","last_undegraded":"2024-05-28T06:08:54.040334+0000","last_fullsized":"2024-05-28T06:08:54.040334+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:51:43.694955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.029865+0000","last_change":"2024-05-28T06:08:54.029865+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.029865+0000","last_undegraded":"2024-05-28T06:08:54.029865+0000","last_fullsized":"2024-05-28T06:08:54.029865+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.039938+0000","last_change":"2024-05-28T06:08:54.035942+0000","last_active":"2024-05-28T06:08:54.039938+0000","last_peered":"2024-05-28T06:08:54.039938+0000","last_clean":"2024-05-28T06:08:54.039938+0000","last_became_active":"2024-05-28T06:08:54.033436+0000","last_became_peered":"2024-05-28T06:08:54.033436+0000","last_unstale":"2024-05-28T06:08:54.039938+0000","last_undegraded":"2024-05-28T06:08:54.039938+0000","last_fullsized":"2024-05-28T06:08:54.039938+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:51:33.930123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.031829+0000","last_change":"2024-05-28T06:08:54.031829+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.031829+0000","last_undegraded":"2024-05-28T06:08:54.031829+0000","last_fullsized":"2024-05-28T06:08:54.031829+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.037891+0000","last_change":"2024-05-28T06:08:54.036250+0000","last_active":"2024-05-28T06:08:54.037891+0000","last_peered":"2024-05-28T06:08:54.037891+0000","last_clean":"2024-05-28T06:08:54.037891+0000","last_became_active":"2024-05-28T06:08:54.034306+0000","last_became_peered":"2024-05-28T06:08:54.034306+0000","last_unstale":"2024-05-28T06:08:54.037891+0000","last_undegraded":"2024-05-28T06:08:54.037891+0000","last_fullsized":"2024-05-28T06:08:54.037891+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:59:23.999803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.033234+0000","last_change":"2024-05-28T06:08:54.033234+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.033234+0000","last_undegraded":"2024-05-28T06:08:54.033234+0000","last_fullsized":"2024-05-28T06:08:54.033234+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.045657+0000","last_change":"2024-05-28T06:08:54.043914+0000","last_active":"2024-05-28T06:08:54.045657+0000","last_peered":"2024-05-28T06:08:54.045657+0000","last_clean":"2024-05-28T06:08:54.045657+0000","last_became_active":"2024-05-28T06:08:54.043662+0000","last_became_peered":"2024-05-28T06:08:54.043662+0000","last_unstale":"2024-05-28T06:08:54.045657+0000","last_undegraded":"2024-05-28T06:08:54.045657+0000","last_fullsized":"2024-05-28T06:08:54.045657+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:14:59.855890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.023267+0000","last_change":"2024-05-28T06:08:54.023267+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.023267+0000","last_undegraded":"2024-05-28T06:08:54.023267+0000","last_fullsized":"2024-05-28T06:08:54.023267+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.045619+0000","last_change":"2024-05-28T06:08:54.042952+0000","last_active":"2024-05-28T06:08:54.045619+0000","last_peered":"2024-05-28T06:08:54.045619+0000","last_clean":"2024-05-28T06:08:54.045619+0000","last_became_active":"2024-05-28T06:08:54.042713+0000","last_became_peered":"2024-05-28T06:08:54.042713+0000","last_unstale":"2024-05-28T06:08:54.045619+0000","last_undegraded":"2024-05-28T06:08:54.045619+0000","last_fullsized":"2024-05-28T06:08:54.045619+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:19:17.873244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.028763+0000","last_change":"2024-05-28T06:08:54.028763+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.028763+0000","last_undegraded":"2024-05-28T06:08:54.028763+0000","last_fullsized":"2024-05-28T06:08:54.028763+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.041728+0000","last_change":"2024-05-28T06:08:54.039984+0000","last_active":"2024-05-28T06:08:54.041728+0000","last_peered":"2024-05-28T06:08:54.041728+0000","last_clean":"2024-05-28T06:08:54.041728+0000","last_became_active":"2024-05-28T06:08:54.038849+0000","last_became_peered":"2024-05-28T06:08:54.038849+0000","last_unstale":"2024-05-28T06:08:54.041728+0000","last_undegraded":"2024-05-28T06:08:54.041728+0000","last_fullsized":"2024-05-28T06:08:54.041728+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:52:34.490485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.014980+0000","last_change":"2024-05-28T06:08:54.014980+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.014980+0000","last_undegraded":"2024-05-28T06:08:54.014980+0000","last_fullsized":"2024-05-28T06:08:54.014980+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.044755+0000","last_change":"2024-05-28T06:08:54.043822+0000","last_active":"2024-05-28T06:08:54.044755+0000","last_peered":"2024-05-28T06:08:54.044755+0000","last_clean":"2024-05-28T06:08:54.044755+0000","last_became_active":"2024-05-28T06:08:54.043402+0000","last_became_peered":"2024-05-28T06:08:54.043402+0000","last_unstale":"2024-05-28T06:08:54.044755+0000","last_undegraded":"2024-05-28T06:08:54.044755+0000","last_fullsized":"2024-05-28T06:08:54.044755+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:16:01.996383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.021077+0000","last_change":"2024-05-28T06:08:54.021077+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.021077+0000","last_undegraded":"2024-05-28T06:08:54.021077+0000","last_fullsized":"2024-05-28T06:08:54.021077+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.047595+0000","last_change":"2024-05-28T06:08:54.046229+0000","last_active":"2024-05-28T06:08:54.047595+0000","last_peered":"2024-05-28T06:08:54.047595+0000","last_clean":"2024-05-28T06:08:54.047595+0000","last_became_active":"2024-05-28T06:08:54.045868+0000","last_became_peered":"2024-05-28T06:08:54.045868+0000","last_unstale":"2024-05-28T06:08:54.047595+0000","last_undegraded":"2024-05-28T06:08:54.047595+0000","last_fullsized":"2024-05-28T06:08:54.047595+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:52:34.490485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.034897+0000","last_change":"2024-05-28T06:08:54.034897+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.034897+0000","last_undegraded":"2024-05-28T06:08:54.034897+0000","last_fullsized":"2024-05-28T06:08:54.034897+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.040552+0000","last_change":"2024-05-28T06:08:54.034283+0000","last_active":"2024-05-28T06:08:54.040552+0000","last_peered":"2024-05-28T06:08:54.040552+0000","last_clean":"2024-05-28T06:08:54.040552+0000","last_became_active":"2024-05-28T06:08:54.033985+0000","last_became_peered":"2024-05-28T06:08:54.033985+0000","last_unstale":"2024-05-28T06:08:54.040552+0000","last_undegraded":"2024-05-28T06:08:54.040552+0000","last_fullsized":"2024-05-28T06:08:54.040552+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:34:31.576736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.020069+0000","last_change":"2024-05-28T06:08:54.020069+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.020069+0000","last_undegraded":"2024-05-28T06:08:54.020069+0000","last_fullsized":"2024-05-28T06:08:54.020069+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.048832+0000","last_change":"2024-05-28T06:08:54.047896+0000","last_active":"2024-05-28T06:08:54.048832+0000","last_peered":"2024-05-28T06:08:54.048832+0000","last_clean":"2024-05-28T06:08:54.048832+0000","last_became_active":"2024-05-28T06:08:54.047535+0000","last_became_peered":"2024-05-28T06:08:54.047535+0000","last_unstale":"2024-05-28T06:08:54.048832+0000","last_undegraded":"2024-05-28T06:08:54.048832+0000","last_fullsized":"2024-05-28T06:08:54.048832+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:14:59.855890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.027417+0000","last_change":"2024-05-28T06:08:54.027417+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.027417+0000","last_undegraded":"2024-05-28T06:08:54.027417+0000","last_fullsized":"2024-05-28T06:08:54.027417+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.041547+0000","last_change":"2024-05-28T06:08:54.037383+0000","last_active":"2024-05-28T06:08:54.041547+0000","last_peered":"2024-05-28T06:08:54.041547+0000","last_clean":"2024-05-28T06:08:54.041547+0000","last_became_active":"2024-05-28T06:08:54.036598+0000","last_became_peered":"2024-05-28T06:08:54.036598+0000","last_unstale":"2024-05-28T06:08:54.041547+0000","last_undegraded":"2024-05-28T06:08:54.041547+0000","last_fullsized":"2024-05-28T06:08:54.041547+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:11:06.365855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.022643+0000","last_change":"2024-05-28T06:08:54.022643+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.022643+0000","last_undegraded":"2024-05-28T06:08:54.022643+0000","last_fullsized":"2024-05-28T06:08:54.022643+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.048321+0000","last_change":"2024-05-28T06:08:54.046483+0000","last_active":"2024-05-28T06:08:54.048321+0000","last_peered":"2024-05-28T06:08:54.048321+0000","last_clean":"2024-05-28T06:08:54.048321+0000","last_became_active":"2024-05-28T06:08:54.045605+0000","last_became_peered":"2024-05-28T06:08:54.045605+0000","last_unstale":"2024-05-28T06:08:54.048321+0000","last_undegraded":"2024-05-28T06:08:54.048321+0000","last_fullsized":"2024-05-28T06:08:54.048321+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:20:27.131737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.024821+0000","last_change":"2024-05-28T06:08:54.024821+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.024821+0000","last_undegraded":"2024-05-28T06:08:54.024821+0000","last_fullsized":"2024-05-28T06:08:54.024821+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.040115+0000","last_change":"2024-05-28T06:08:54.037676+0000","last_active":"2024-05-28T06:08:54.040115+0000","last_peered":"2024-05-28T06:08:54.040115+0000","last_clean":"2024-05-28T06:08:54.040115+0000","last_became_active":"2024-05-28T06:08:54.037301+0000","last_became_peered":"2024-05-28T06:08:54.037301+0000","last_unstale":"2024-05-28T06:08:54.040115+0000","last_undegraded":"2024-05-28T06:08:54.040115+0000","last_fullsized":"2024-05-28T06:08:54.040115+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:01:37.817834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.019527+0000","last_change":"2024-05-28T06:08:54.019527+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.019527+0000","last_undegraded":"2024-05-28T06:08:54.019527+0000","last_fullsized":"2024-05-28T06:08:54.019527+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.045431+0000","last_change":"2024-05-28T06:08:54.044188+0000","last_active":"2024-05-28T06:08:54.045431+0000","last_peered":"2024-05-28T06:08:54.045431+0000","last_clean":"2024-05-28T06:08:54.045431+0000","last_became_active":"2024-05-28T06:08:54.043655+0000","last_became_peered":"2024-05-28T06:08:54.043655+0000","last_unstale":"2024-05-28T06:08:54.045431+0000","last_undegraded":"2024-05-28T06:08:54.045431+0000","last_fullsized":"2024-05-28T06:08:54.045431+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:27:40.364767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.013578+0000","last_change":"2024-05-28T06:08:54.013578+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.013578+0000","last_undegraded":"2024-05-28T06:08:54.013578+0000","last_fullsized":"2024-05-28T06:08:54.013578+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.040079+0000","last_change":"2024-05-28T06:08:54.038376+0000","last_active":"2024-05-28T06:08:54.040079+0000","last_peered":"2024-05-28T06:08:54.040079+0000","last_clean":"2024-05-28T06:08:54.040079+0000","last_became_active":"2024-05-28T06:08:54.037952+0000","last_became_peered":"2024-05-28T06:08:54.037952+0000","last_unstale":"2024-05-28T06:08:54.040079+0000","last_undegraded":"2024-05-28T06:08:54.040079+0000","last_fullsized":"2024-05-28T06:08:54.040079+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:52:58.719365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.023494+0000","last_change":"2024-05-28T06:08:54.023494+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.023494+0000","last_undegraded":"2024-05-28T06:08:54.023494+0000","last_fullsized":"2024-05-28T06:08:54.023494+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.039837+0000","last_change":"2024-05-28T06:08:54.036597+0000","last_active":"2024-05-28T06:08:54.039837+0000","last_peered":"2024-05-28T06:08:54.039837+0000","last_clean":"2024-05-28T06:08:54.039837+0000","last_became_active":"2024-05-28T06:08:54.034833+0000","last_became_peered":"2024-05-28T06:08:54.034833+0000","last_unstale":"2024-05-28T06:08:54.039837+0000","last_undegraded":"2024-05-28T06:08:54.039837+0000","last_fullsized":"2024-05-28T06:08:54.039837+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:34:31.576736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":213,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.037334+0000","last_change":"2024-05-28T06:04:01.039129+0000","last_active":"2024-05-28T06:08:54.037334+0000","last_peered":"2024-05-28T06:08:54.037334+0000","last_clean":"2024-05-28T06:08:54.037334+0000","last_became_active":"2024-05-28T06:04:01.033906+0000","last_became_peered":"2024-05-28T06:04:01.033906+0000","last_unstale":"2024-05-28T06:08:54.037334+0000","last_undegraded":"2024-05-28T06:08:54.037334+0000","last_fullsized":"2024-05-28T06:08:54.037334+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:03:59.864237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:03:59.864237+0000","last_clean_scrub_stamp":"2024-05-28T06:03:59.864237+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:17:36.440245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574910,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36400,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335440,"statfs":{"total":96636764160,"available":96599490560,"internally_reserved":0,"allocated":1085440,"data_stored":787442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574912,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31680,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340160,"statfs":{"total":96636764160,"available":96604323840,"internally_reserved":0,"allocated":503808,"data_stored":203369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574910,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35836,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94336004,"statfs":{"total":96636764160,"available":96600068096,"internally_reserved":0,"allocated":503808,"data_stored":203369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574910,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35836,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94336004,"statfs":{"total":96636764160,"available":96600068096,"internally_reserved":0,"allocated":503808,"data_stored":203369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574912,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35820,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94336020,"statfs":{"total":96636764160,"available":96600084480,"internally_reserved":0,"allocated":491520,"data_stored":197074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574911,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35840,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94336000,"statfs":{"total":96636764160,"available":96600064000,"internally_reserved":0,"allocated":503808,"data_stored":203369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574911,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35832,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94336008,"statfs":{"total":96636764160,"available":96600072192,"internally_reserved":0,"allocated":503808,"data_stored":203369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574910,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36416,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335424,"statfs":{"total":96636764160,"available":96599474176,"internally_reserved":0,"allocated":1097728,"data_stored":793737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-28T06:08:59.485 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-28T06:08:59.730 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.729+0000 7f5563600640 1 Processor -- start 2024-05-28T06:08:59.730 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.729+0000 7f5563600640 1 -- start start 2024-05-28T06:08:59.732 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.730+0000 7f5563600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f555c15b850 0x7f555c0cb430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:59.732 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.730+0000 7f5563600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f555c1600d0 0x7f555c0b0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:59.733 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.730+0000 7f5563600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f555c0ce1a0 con 0x7f555c0c3930 2024-05-28T06:08:59.733 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.730+0000 7f5563600640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f555c0cdf50 con 0x7f555c15b850 2024-05-28T06:08:59.733 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.730+0000 7f5563600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f555c15f530 con 0x7f555c1600d0 2024-05-28T06:08:59.733 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.731+0000 7f5560c00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f555c15b850 0x7f555c0cb430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:59.733 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.731+0000 7f5560c00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f555c15b850 0x7f555c0cb430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47474/0 (socket says 172.21.15.96:47474) 2024-05-28T06:08:59.733 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.731+0000 7f5560c00640 1 -- 172.21.15.96:0/1109200642 learned_addr learned my addr 172.21.15.96:0/1109200642 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:08:59.733 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.731+0000 7f5560c00640 1 -- 172.21.15.96:0/1109200642 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f555c1600d0 msgr2=0x7f555c0b0ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:59.733 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.731+0000 7f5560c00640 1 --2- 172.21.15.96:0/1109200642 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f555c1600d0 0x7f555c0b0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:59.733 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.731+0000 7f5560c00640 1 -- 172.21.15.96:0/1109200642 >> v1:172.21.15.123:6789/0 conn(0x7f555c0c3930 legacy=0x7f555c15f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:08:59.733 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.731+0000 7f5560c00640 1 -- 172.21.15.96:0/1109200642 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f555c0ce480 con 0x7f555c15b850 2024-05-28T06:08:59.734 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.732+0000 7f5560c00640 1 --2- 172.21.15.96:0/1109200642 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f555c15b850 0x7f555c0cb430 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f5550009870 tx=0x7f555002a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adb5d0adc059da06 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:59.734 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.733+0000 7f555b400640 1 -- 172.21.15.96:0/1109200642 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5550037070 con 0x7f555c15b850 2024-05-28T06:08:59.734 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.733+0000 7f555b400640 1 -- 172.21.15.96:0/1109200642 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5550032440 con 0x7f555c15b850 2024-05-28T06:08:59.734 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.733+0000 7f555b400640 1 -- 172.21.15.96:0/1109200642 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5550032810 con 0x7f555c15b850 2024-05-28T06:08:59.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.733+0000 7f5563600640 1 -- 172.21.15.96:0/1109200642 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f555c15b850 msgr2=0x7f555c0cb430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:59.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.733+0000 7f5563600640 1 --2- 172.21.15.96:0/1109200642 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f555c15b850 0x7f555c0cb430 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f5550009870 tx=0x7f555002a180 comp rx=0 tx=0).stop 2024-05-28T06:08:59.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.733+0000 7f5563600640 1 -- 172.21.15.96:0/1109200642 shutdown_connections 2024-05-28T06:08:59.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.733+0000 7f5563600640 1 --2- 172.21.15.96:0/1109200642 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f555c1600d0 0x7f555c0b0ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:59.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.733+0000 7f5563600640 1 --2- 172.21.15.96:0/1109200642 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f555c15b850 0x7f555c0cb430 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:59.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.733+0000 7f5563600640 1 -- 172.21.15.96:0/1109200642 >> 172.21.15.96:0/1109200642 conn(0x7f555c0d0260 msgr2=0x7f555c0d0670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:08:59.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.733+0000 7f5563600640 1 -- 172.21.15.96:0/1109200642 shutdown_connections 2024-05-28T06:08:59.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.733+0000 7f5563600640 1 -- 172.21.15.96:0/1109200642 wait complete. 2024-05-28T06:08:59.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.735+0000 7f5563600640 1 Processor -- start 2024-05-28T06:08:59.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.735+0000 7f5563600640 1 -- start start 2024-05-28T06:08:59.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.736+0000 7f5563600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f555c0c3930 0x7f555c1f4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:59.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.736+0000 7f5563600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f555c15b850 0x7f555c0b5820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:59.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.736+0000 7f5563600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f555c1600d0 0x7f555c0c7330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:59.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.736+0000 7f5563600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f555c0cf400 con 0x7f555c15b850 2024-05-28T06:08:59.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.736+0000 7f5563600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f555c15bc30 con 0x7f555c1600d0 2024-05-28T06:08:59.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.736+0000 7f5563600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f555c0cf580 con 0x7f555c0c3930 2024-05-28T06:08:59.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.737+0000 7f555be00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f555c15b850 0x7f555c0b5820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:59.739 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.737+0000 7f555be00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f555c15b850 0x7f555c0b5820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47484/0 (socket says 172.21.15.96:47484) 2024-05-28T06:08:59.739 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.737+0000 7f555be00640 1 -- 172.21.15.96:0/452113478 learned_addr learned my addr 172.21.15.96:0/452113478 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:08:59.739 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.737+0000 7f5560c00640 1 --2- 172.21.15.96:0/452113478 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f555c0c3930 0x7f555c1f4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:59.739 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.737+0000 7f5561600640 1 --2- 172.21.15.96:0/452113478 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f555c1600d0 0x7f555c0c7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:59.740 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.737+0000 7f555be00640 1 -- 172.21.15.96:0/452113478 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f555c0c3930 msgr2=0x7f555c1f4770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:59.740 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.737+0000 7f555be00640 1 --2- 172.21.15.96:0/452113478 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f555c0c3930 0x7f555c1f4770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:59.740 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.737+0000 7f555be00640 1 -- 172.21.15.96:0/452113478 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f555c1600d0 msgr2=0x7f555c0c7330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:08:59.740 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.737+0000 7f555be00640 1 --2- 172.21.15.96:0/452113478 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f555c1600d0 0x7f555c0c7330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:08:59.740 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.737+0000 7f555be00640 1 -- 172.21.15.96:0/452113478 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f555c0b5d60 con 0x7f555c15b850 2024-05-28T06:08:59.740 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.737+0000 7f555be00640 1 --2- 172.21.15.96:0/452113478 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f555c15b850 0x7f555c0b5820 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f554800b3c0 tx=0x7f554800b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:59.740 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.738+0000 7f5558c00640 1 -- 172.21.15.96:0/452113478 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5548011830 con 0x7f555c15b850 2024-05-28T06:08:59.740 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.738+0000 7f5558c00640 1 -- 172.21.15.96:0/452113478 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5548016070 con 0x7f555c15b850 2024-05-28T06:08:59.740 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.738+0000 7f5558c00640 1 -- 172.21.15.96:0/452113478 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5548011c50 con 0x7f555c15b850 2024-05-28T06:08:59.740 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.738+0000 7f5563600640 1 -- 172.21.15.96:0/452113478 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f555c0b30c0 con 0x7f555c15b850 2024-05-28T06:08:59.741 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.738+0000 7f5563600640 1 -- 172.21.15.96:0/452113478 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f555c0b3680 con 0x7f555c15b850 2024-05-28T06:08:59.741 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.739+0000 7f5563600640 1 -- 172.21.15.96:0/452113478 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f555c0689e0 con 0x7f555c15b850 2024-05-28T06:08:59.741 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.740+0000 7f5558c00640 1 -- 172.21.15.96:0/452113478 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 9) ==== 91295+0+0 (secure 0 0 0) 0x7f554801b460 con 0x7f555c15b850 2024-05-28T06:08:59.742 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.741+0000 7f5558c00640 1 --2- 172.21.15.96:0/452113478 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f553c06cbc0 0x7f553c06f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:08:59.742 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.741+0000 7f5558c00640 1 -- 172.21.15.96:0/452113478 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5737+0+0 (secure 0 0 0) 0x7f554808ba80 con 0x7f555c15b850 2024-05-28T06:08:59.743 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.742+0000 7f5560c00640 1 --2- 172.21.15.96:0/452113478 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f553c06cbc0 0x7f553c06f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:08:59.744 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.743+0000 7f5560c00640 1 --2- 172.21.15.96:0/452113478 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f553c06cbc0 0x7f553c06f080 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f5550034000 tx=0x7f555002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:08:59.746 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.745+0000 7f5558c00640 1 -- 172.21.15.96:0/452113478 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f554808de20 con 0x7f555c15b850 2024-05-28T06:08:59.922 DEBUG:teuthology.orchestra.run.smithi096:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:08:59.930 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:08:59.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.945+0000 7f5563600640 1 -- 172.21.15.96:0/452113478 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f555c1f4cb0 con 0x7f553c06cbc0 2024-05-28T06:09:00.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.965+0000 7f5558c00640 1 -- 172.21.15.96:0/452113478 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+246639 (secure 0 0 0) 0x7f55500327d0 con 0x7f553c06cbc0 2024-05-28T06:09:00.343 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:09:00.345 INFO:teuthology.orchestra.run.smithi096.stderr:dumped all 2024-05-28T06:09:00.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.970+0000 7f5563600640 1 -- 172.21.15.96:0/452113478 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f553c06cbc0 msgr2=0x7f553c06f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:09:00.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.970+0000 7f5563600640 1 --2- 172.21.15.96:0/452113478 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f553c06cbc0 0x7f553c06f080 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f5550034000 tx=0x7f555002e000 comp rx=0 tx=0).stop 2024-05-28T06:09:00.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.970+0000 7f5563600640 1 -- 172.21.15.96:0/452113478 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f555c15b850 msgr2=0x7f555c0b5820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:09:00.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.970+0000 7f5563600640 1 --2- 172.21.15.96:0/452113478 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f555c15b850 0x7f555c0b5820 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f554800b3c0 tx=0x7f554800b880 comp rx=0 tx=0).stop 2024-05-28T06:09:00.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.970+0000 7f5563600640 1 -- 172.21.15.96:0/452113478 shutdown_connections 2024-05-28T06:09:00.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.970+0000 7f5563600640 1 --2- 172.21.15.96:0/452113478 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f553c06cbc0 0x7f553c06f080 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:00.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.970+0000 7f5563600640 1 --2- 172.21.15.96:0/452113478 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f555c1600d0 0x7f555c0c7330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:00.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.970+0000 7f5563600640 1 --2- 172.21.15.96:0/452113478 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f555c15b850 0x7f555c0b5820 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:00.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.970+0000 7f5563600640 1 --2- 172.21.15.96:0/452113478 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f555c0c3930 0x7f555c1f4770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:00.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.970+0000 7f5563600640 1 -- 172.21.15.96:0/452113478 >> 172.21.15.96:0/452113478 conn(0x7f555c0d0260 msgr2=0x7f555c0cbc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:09:00.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.970+0000 7f5563600640 1 -- 172.21.15.96:0/452113478 shutdown_connections 2024-05-28T06:09:00.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:08:59.970+0000 7f5563600640 1 -- 172.21.15.96:0/452113478 wait complete. 2024-05-28T06:09:00.347 INFO:teuthology.orchestra.run.smithi096.stdout:{"pg_ready":true,"pg_map":{"version":188,"stamp":"2024-05-28T06:08:59.591908+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":28,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":283660,"kb_used_data":5072,"kb_used_omap":12,"kb_used_meta":278451,"kb_avail":754691060,"statfs":{"total":773094113280,"available":772803645440,"internally_reserved":0,"allocated":5193728,"data_stored":2795098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":285134384},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001854"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.030805+0000","last_change":"2024-05-28T06:08:54.030805+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.030805+0000","last_undegraded":"2024-05-28T06:08:54.030805+0000","last_fullsized":"2024-05-28T06:08:54.030805+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.023029+0000","last_change":"2024-05-28T06:08:54.023029+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.023029+0000","last_undegraded":"2024-05-28T06:08:54.023029+0000","last_fullsized":"2024-05-28T06:08:54.023029+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.019511+0000","last_change":"2024-05-28T06:08:54.019511+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.019511+0000","last_undegraded":"2024-05-28T06:08:54.019511+0000","last_fullsized":"2024-05-28T06:08:54.019511+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.031775+0000","last_change":"2024-05-28T06:08:54.031775+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.031775+0000","last_undegraded":"2024-05-28T06:08:54.031775+0000","last_fullsized":"2024-05-28T06:08:54.031775+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.021863+0000","last_change":"2024-05-28T06:08:54.021863+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.021863+0000","last_undegraded":"2024-05-28T06:08:54.021863+0000","last_fullsized":"2024-05-28T06:08:54.021863+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.023805+0000","last_change":"2024-05-28T06:08:54.023805+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.023805+0000","last_undegraded":"2024-05-28T06:08:54.023805+0000","last_fullsized":"2024-05-28T06:08:54.023805+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.029593+0000","last_change":"2024-05-28T06:08:54.029593+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.029593+0000","last_undegraded":"2024-05-28T06:08:54.029593+0000","last_fullsized":"2024-05-28T06:08:54.029593+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.022144+0000","last_change":"2024-05-28T06:08:54.022144+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.022144+0000","last_undegraded":"2024-05-28T06:08:54.022144+0000","last_fullsized":"2024-05-28T06:08:54.022144+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.030711+0000","last_change":"2024-05-28T06:08:54.030711+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.030711+0000","last_undegraded":"2024-05-28T06:08:54.030711+0000","last_fullsized":"2024-05-28T06:08:54.030711+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.024096+0000","last_change":"2024-05-28T06:08:54.024096+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.024096+0000","last_undegraded":"2024-05-28T06:08:54.024096+0000","last_fullsized":"2024-05-28T06:08:54.024096+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.018302+0000","last_change":"2024-05-28T06:08:54.018302+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.018302+0000","last_undegraded":"2024-05-28T06:08:54.018302+0000","last_fullsized":"2024-05-28T06:08:54.018302+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.024586+0000","last_change":"2024-05-28T06:08:54.024586+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.024586+0000","last_undegraded":"2024-05-28T06:08:54.024586+0000","last_fullsized":"2024-05-28T06:08:54.024586+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.014791+0000","last_change":"2024-05-28T06:08:54.014791+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.014791+0000","last_undegraded":"2024-05-28T06:08:54.014791+0000","last_fullsized":"2024-05-28T06:08:54.014791+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.027464+0000","last_change":"2024-05-28T06:08:54.027464+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.027464+0000","last_undegraded":"2024-05-28T06:08:54.027464+0000","last_fullsized":"2024-05-28T06:08:54.027464+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.025774+0000","last_change":"2024-05-28T06:08:54.025774+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.025774+0000","last_undegraded":"2024-05-28T06:08:54.025774+0000","last_fullsized":"2024-05-28T06:08:54.025774+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.029213+0000","last_change":"2024-05-28T06:08:54.029213+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.029213+0000","last_undegraded":"2024-05-28T06:08:54.029213+0000","last_fullsized":"2024-05-28T06:08:54.029213+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.014182+0000","last_change":"2024-05-28T06:08:54.014182+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.014182+0000","last_undegraded":"2024-05-28T06:08:54.014182+0000","last_fullsized":"2024-05-28T06:08:54.014182+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.016245+0000","last_change":"2024-05-28T06:08:54.016245+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.016245+0000","last_undegraded":"2024-05-28T06:08:54.016245+0000","last_fullsized":"2024-05-28T06:08:54.016245+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.032693+0000","last_change":"2024-05-28T06:08:54.032693+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.032693+0000","last_undegraded":"2024-05-28T06:08:54.032693+0000","last_fullsized":"2024-05-28T06:08:54.032693+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.014442+0000","last_change":"2024-05-28T06:08:54.014442+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.014442+0000","last_undegraded":"2024-05-28T06:08:54.014442+0000","last_fullsized":"2024-05-28T06:08:54.014442+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.025473+0000","last_change":"2024-05-28T06:08:54.025473+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.025473+0000","last_undegraded":"2024-05-28T06:08:54.025473+0000","last_fullsized":"2024-05-28T06:08:54.025473+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.018238+0000","last_change":"2024-05-28T06:08:54.018238+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.018238+0000","last_undegraded":"2024-05-28T06:08:54.018238+0000","last_fullsized":"2024-05-28T06:08:54.018238+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.029393+0000","last_change":"2024-05-28T06:08:54.029393+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.029393+0000","last_undegraded":"2024-05-28T06:08:54.029393+0000","last_fullsized":"2024-05-28T06:08:54.029393+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.028295+0000","last_change":"2024-05-28T06:08:54.028295+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.028295+0000","last_undegraded":"2024-05-28T06:08:54.028295+0000","last_fullsized":"2024-05-28T06:08:54.028295+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.037153+0000","last_change":"2024-05-28T06:08:54.037153+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.037153+0000","last_undegraded":"2024-05-28T06:08:54.037153+0000","last_fullsized":"2024-05-28T06:08:54.037153+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.015960+0000","last_change":"2024-05-28T06:08:54.015960+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.015960+0000","last_undegraded":"2024-05-28T06:08:54.015960+0000","last_fullsized":"2024-05-28T06:08:54.015960+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.014740+0000","last_change":"2024-05-28T06:08:54.014740+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.014740+0000","last_undegraded":"2024-05-28T06:08:54.014740+0000","last_fullsized":"2024-05-28T06:08:54.014740+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.023762+0000","last_change":"2024-05-28T06:08:54.023762+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.023762+0000","last_undegraded":"2024-05-28T06:08:54.023762+0000","last_fullsized":"2024-05-28T06:08:54.023762+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.030708+0000","last_change":"2024-05-28T06:08:54.030708+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.030708+0000","last_undegraded":"2024-05-28T06:08:54.030708+0000","last_fullsized":"2024-05-28T06:08:54.030708+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.017736+0000","last_change":"2024-05-28T06:08:54.017736+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.017736+0000","last_undegraded":"2024-05-28T06:08:54.017736+0000","last_fullsized":"2024-05-28T06:08:54.017736+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.019413+0000","last_change":"2024-05-28T06:08:54.019413+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.019413+0000","last_undegraded":"2024-05-28T06:08:54.019413+0000","last_fullsized":"2024-05-28T06:08:54.019413+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.016151+0000","last_change":"2024-05-28T06:08:54.016151+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.016151+0000","last_undegraded":"2024-05-28T06:08:54.016151+0000","last_fullsized":"2024-05-28T06:08:54.016151+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.028257+0000","last_change":"2024-05-28T06:08:54.028257+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.028257+0000","last_undegraded":"2024-05-28T06:08:54.028257+0000","last_fullsized":"2024-05-28T06:08:54.028257+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.045658+0000","last_change":"2024-05-28T06:08:54.044108+0000","last_active":"2024-05-28T06:08:54.045658+0000","last_peered":"2024-05-28T06:08:54.045658+0000","last_clean":"2024-05-28T06:08:54.045658+0000","last_became_active":"2024-05-28T06:08:54.043637+0000","last_became_peered":"2024-05-28T06:08:54.043637+0000","last_unstale":"2024-05-28T06:08:54.045658+0000","last_undegraded":"2024-05-28T06:08:54.045658+0000","last_fullsized":"2024-05-28T06:08:54.045658+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:11:14.276087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.036622+0000","last_change":"2024-05-28T06:08:54.036622+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.036622+0000","last_undegraded":"2024-05-28T06:08:54.036622+0000","last_fullsized":"2024-05-28T06:08:54.036622+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.039961+0000","last_change":"2024-05-28T06:08:54.036739+0000","last_active":"2024-05-28T06:08:54.039961+0000","last_peered":"2024-05-28T06:08:54.039961+0000","last_clean":"2024-05-28T06:08:54.039961+0000","last_became_active":"2024-05-28T06:08:54.036446+0000","last_became_peered":"2024-05-28T06:08:54.036446+0000","last_unstale":"2024-05-28T06:08:54.039961+0000","last_undegraded":"2024-05-28T06:08:54.039961+0000","last_fullsized":"2024-05-28T06:08:54.039961+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:07:41.209833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.022836+0000","last_change":"2024-05-28T06:08:54.022836+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.022836+0000","last_undegraded":"2024-05-28T06:08:54.022836+0000","last_fullsized":"2024-05-28T06:08:54.022836+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.040347+0000","last_change":"2024-05-28T06:08:54.035269+0000","last_active":"2024-05-28T06:08:54.040347+0000","last_peered":"2024-05-28T06:08:54.040347+0000","last_clean":"2024-05-28T06:08:54.040347+0000","last_became_active":"2024-05-28T06:08:54.034916+0000","last_became_peered":"2024-05-28T06:08:54.034916+0000","last_unstale":"2024-05-28T06:08:54.040347+0000","last_undegraded":"2024-05-28T06:08:54.040347+0000","last_fullsized":"2024-05-28T06:08:54.040347+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:37:34.767912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.026952+0000","last_change":"2024-05-28T06:08:54.026952+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.026952+0000","last_undegraded":"2024-05-28T06:08:54.026952+0000","last_fullsized":"2024-05-28T06:08:54.026952+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.046591+0000","last_change":"2024-05-28T06:08:54.044956+0000","last_active":"2024-05-28T06:08:54.046591+0000","last_peered":"2024-05-28T06:08:54.046591+0000","last_clean":"2024-05-28T06:08:54.046591+0000","last_became_active":"2024-05-28T06:08:54.044646+0000","last_became_peered":"2024-05-28T06:08:54.044646+0000","last_unstale":"2024-05-28T06:08:54.046591+0000","last_undegraded":"2024-05-28T06:08:54.046591+0000","last_fullsized":"2024-05-28T06:08:54.046591+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:34:31.576736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.014512+0000","last_change":"2024-05-28T06:08:54.014512+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.014512+0000","last_undegraded":"2024-05-28T06:08:54.014512+0000","last_fullsized":"2024-05-28T06:08:54.014512+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.045585+0000","last_change":"2024-05-28T06:08:54.042495+0000","last_active":"2024-05-28T06:08:54.045585+0000","last_peered":"2024-05-28T06:08:54.045585+0000","last_clean":"2024-05-28T06:08:54.045585+0000","last_became_active":"2024-05-28T06:08:54.042143+0000","last_became_peered":"2024-05-28T06:08:54.042143+0000","last_unstale":"2024-05-28T06:08:54.045585+0000","last_undegraded":"2024-05-28T06:08:54.045585+0000","last_fullsized":"2024-05-28T06:08:54.045585+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:20:27.131737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.022747+0000","last_change":"2024-05-28T06:08:54.022747+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.022747+0000","last_undegraded":"2024-05-28T06:08:54.022747+0000","last_fullsized":"2024-05-28T06:08:54.022747+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.039866+0000","last_change":"2024-05-28T06:08:54.030180+0000","last_active":"2024-05-28T06:08:54.039866+0000","last_peered":"2024-05-28T06:08:54.039866+0000","last_clean":"2024-05-28T06:08:54.039866+0000","last_became_active":"2024-05-28T06:08:54.029892+0000","last_became_peered":"2024-05-28T06:08:54.029892+0000","last_unstale":"2024-05-28T06:08:54.039866+0000","last_undegraded":"2024-05-28T06:08:54.039866+0000","last_fullsized":"2024-05-28T06:08:54.039866+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:16:17.652829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.021598+0000","last_change":"2024-05-28T06:08:54.021598+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.021598+0000","last_undegraded":"2024-05-28T06:08:54.021598+0000","last_fullsized":"2024-05-28T06:08:54.021598+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.038065+0000","last_change":"2024-05-28T06:08:54.035697+0000","last_active":"2024-05-28T06:08:54.038065+0000","last_peered":"2024-05-28T06:08:54.038065+0000","last_clean":"2024-05-28T06:08:54.038065+0000","last_became_active":"2024-05-28T06:08:54.032931+0000","last_became_peered":"2024-05-28T06:08:54.032931+0000","last_unstale":"2024-05-28T06:08:54.038065+0000","last_undegraded":"2024-05-28T06:08:54.038065+0000","last_fullsized":"2024-05-28T06:08:54.038065+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:29:27.490700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.030054+0000","last_change":"2024-05-28T06:08:54.030054+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.030054+0000","last_undegraded":"2024-05-28T06:08:54.030054+0000","last_fullsized":"2024-05-28T06:08:54.030054+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.036080+0000","last_change":"2024-05-28T06:08:54.032388+0000","last_active":"2024-05-28T06:08:54.036080+0000","last_peered":"2024-05-28T06:08:54.036080+0000","last_clean":"2024-05-28T06:08:54.036080+0000","last_became_active":"2024-05-28T06:08:54.032095+0000","last_became_peered":"2024-05-28T06:08:54.032095+0000","last_unstale":"2024-05-28T06:08:54.036080+0000","last_undegraded":"2024-05-28T06:08:54.036080+0000","last_fullsized":"2024-05-28T06:08:54.036080+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:21:39.812262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.014966+0000","last_change":"2024-05-28T06:08:54.014966+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.014966+0000","last_undegraded":"2024-05-28T06:08:54.014966+0000","last_fullsized":"2024-05-28T06:08:54.014966+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.039945+0000","last_change":"2024-05-28T06:08:54.027262+0000","last_active":"2024-05-28T06:08:54.039945+0000","last_peered":"2024-05-28T06:08:54.039945+0000","last_clean":"2024-05-28T06:08:54.039945+0000","last_became_active":"2024-05-28T06:08:54.026066+0000","last_became_peered":"2024-05-28T06:08:54.026066+0000","last_unstale":"2024-05-28T06:08:54.039945+0000","last_undegraded":"2024-05-28T06:08:54.039945+0000","last_fullsized":"2024-05-28T06:08:54.039945+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:16:01.996383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.030190+0000","last_change":"2024-05-28T06:08:54.030190+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.030190+0000","last_undegraded":"2024-05-28T06:08:54.030190+0000","last_fullsized":"2024-05-28T06:08:54.030190+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.041630+0000","last_change":"2024-05-28T06:08:54.040877+0000","last_active":"2024-05-28T06:08:54.041630+0000","last_peered":"2024-05-28T06:08:54.041630+0000","last_clean":"2024-05-28T06:08:54.041630+0000","last_became_active":"2024-05-28T06:08:54.040615+0000","last_became_peered":"2024-05-28T06:08:54.040615+0000","last_unstale":"2024-05-28T06:08:54.041630+0000","last_undegraded":"2024-05-28T06:08:54.041630+0000","last_fullsized":"2024-05-28T06:08:54.041630+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:40:59.926424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.018762+0000","last_change":"2024-05-28T06:08:54.018762+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.018762+0000","last_undegraded":"2024-05-28T06:08:54.018762+0000","last_fullsized":"2024-05-28T06:08:54.018762+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.048326+0000","last_change":"2024-05-28T06:08:54.046805+0000","last_active":"2024-05-28T06:08:54.048326+0000","last_peered":"2024-05-28T06:08:54.048326+0000","last_clean":"2024-05-28T06:08:54.048326+0000","last_became_active":"2024-05-28T06:08:54.046336+0000","last_became_peered":"2024-05-28T06:08:54.046336+0000","last_unstale":"2024-05-28T06:08:54.048326+0000","last_undegraded":"2024-05-28T06:08:54.048326+0000","last_fullsized":"2024-05-28T06:08:54.048326+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:19:17.873244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.032123+0000","last_change":"2024-05-28T06:08:54.032123+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.032123+0000","last_undegraded":"2024-05-28T06:08:54.032123+0000","last_fullsized":"2024-05-28T06:08:54.032123+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.031479+0000","last_change":"2024-05-28T06:08:54.029622+0000","last_active":"2024-05-28T06:08:54.031479+0000","last_peered":"2024-05-28T06:08:54.031479+0000","last_clean":"2024-05-28T06:08:54.031479+0000","last_became_active":"2024-05-28T06:08:54.026140+0000","last_became_peered":"2024-05-28T06:08:54.026140+0000","last_unstale":"2024-05-28T06:08:54.031479+0000","last_undegraded":"2024-05-28T06:08:54.031479+0000","last_fullsized":"2024-05-28T06:08:54.031479+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:14:10.883135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.028947+0000","last_change":"2024-05-28T06:08:54.028947+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.028947+0000","last_undegraded":"2024-05-28T06:08:54.028947+0000","last_fullsized":"2024-05-28T06:08:54.028947+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.041714+0000","last_change":"2024-05-28T06:08:54.038320+0000","last_active":"2024-05-28T06:08:54.041714+0000","last_peered":"2024-05-28T06:08:54.041714+0000","last_clean":"2024-05-28T06:08:54.041714+0000","last_became_active":"2024-05-28T06:08:54.037824+0000","last_became_peered":"2024-05-28T06:08:54.037824+0000","last_unstale":"2024-05-28T06:08:54.041714+0000","last_undegraded":"2024-05-28T06:08:54.041714+0000","last_fullsized":"2024-05-28T06:08:54.041714+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:47:29.434576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.021818+0000","last_change":"2024-05-28T06:08:54.021818+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.021818+0000","last_undegraded":"2024-05-28T06:08:54.021818+0000","last_fullsized":"2024-05-28T06:08:54.021818+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.048280+0000","last_change":"2024-05-28T06:08:54.046264+0000","last_active":"2024-05-28T06:08:54.048280+0000","last_peered":"2024-05-28T06:08:54.048280+0000","last_clean":"2024-05-28T06:08:54.048280+0000","last_became_active":"2024-05-28T06:08:54.045957+0000","last_became_peered":"2024-05-28T06:08:54.045957+0000","last_unstale":"2024-05-28T06:08:54.048280+0000","last_undegraded":"2024-05-28T06:08:54.048280+0000","last_fullsized":"2024-05-28T06:08:54.048280+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:47:29.434576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.024967+0000","last_change":"2024-05-28T06:08:54.024967+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.024967+0000","last_undegraded":"2024-05-28T06:08:54.024967+0000","last_fullsized":"2024-05-28T06:08:54.024967+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.041658+0000","last_change":"2024-05-28T06:08:54.038131+0000","last_active":"2024-05-28T06:08:54.041658+0000","last_peered":"2024-05-28T06:08:54.041658+0000","last_clean":"2024-05-28T06:08:54.041658+0000","last_became_active":"2024-05-28T06:08:54.037578+0000","last_became_peered":"2024-05-28T06:08:54.037578+0000","last_unstale":"2024-05-28T06:08:54.041658+0000","last_undegraded":"2024-05-28T06:08:54.041658+0000","last_fullsized":"2024-05-28T06:08:54.041658+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:52:34.490485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.029151+0000","last_change":"2024-05-28T06:08:54.029151+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.029151+0000","last_undegraded":"2024-05-28T06:08:54.029151+0000","last_fullsized":"2024-05-28T06:08:54.029151+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.037597+0000","last_change":"2024-05-28T06:08:54.034498+0000","last_active":"2024-05-28T06:08:54.037597+0000","last_peered":"2024-05-28T06:08:54.037597+0000","last_clean":"2024-05-28T06:08:54.037597+0000","last_became_active":"2024-05-28T06:08:54.034152+0000","last_became_peered":"2024-05-28T06:08:54.034152+0000","last_unstale":"2024-05-28T06:08:54.037597+0000","last_undegraded":"2024-05-28T06:08:54.037597+0000","last_fullsized":"2024-05-28T06:08:54.037597+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:57:06.361350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.015569+0000","last_change":"2024-05-28T06:08:54.015569+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.015569+0000","last_undegraded":"2024-05-28T06:08:54.015569+0000","last_fullsized":"2024-05-28T06:08:54.015569+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.040334+0000","last_change":"2024-05-28T06:08:54.035386+0000","last_active":"2024-05-28T06:08:54.040334+0000","last_peered":"2024-05-28T06:08:54.040334+0000","last_clean":"2024-05-28T06:08:54.040334+0000","last_became_active":"2024-05-28T06:08:54.033511+0000","last_became_peered":"2024-05-28T06:08:54.033511+0000","last_unstale":"2024-05-28T06:08:54.040334+0000","last_undegraded":"2024-05-28T06:08:54.040334+0000","last_fullsized":"2024-05-28T06:08:54.040334+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:51:43.694955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.029865+0000","last_change":"2024-05-28T06:08:54.029865+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.029865+0000","last_undegraded":"2024-05-28T06:08:54.029865+0000","last_fullsized":"2024-05-28T06:08:54.029865+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.039938+0000","last_change":"2024-05-28T06:08:54.035942+0000","last_active":"2024-05-28T06:08:54.039938+0000","last_peered":"2024-05-28T06:08:54.039938+0000","last_clean":"2024-05-28T06:08:54.039938+0000","last_became_active":"2024-05-28T06:08:54.033436+0000","last_became_peered":"2024-05-28T06:08:54.033436+0000","last_unstale":"2024-05-28T06:08:54.039938+0000","last_undegraded":"2024-05-28T06:08:54.039938+0000","last_fullsized":"2024-05-28T06:08:54.039938+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:51:33.930123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.031829+0000","last_change":"2024-05-28T06:08:54.031829+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.031829+0000","last_undegraded":"2024-05-28T06:08:54.031829+0000","last_fullsized":"2024-05-28T06:08:54.031829+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.037891+0000","last_change":"2024-05-28T06:08:54.036250+0000","last_active":"2024-05-28T06:08:54.037891+0000","last_peered":"2024-05-28T06:08:54.037891+0000","last_clean":"2024-05-28T06:08:54.037891+0000","last_became_active":"2024-05-28T06:08:54.034306+0000","last_became_peered":"2024-05-28T06:08:54.034306+0000","last_unstale":"2024-05-28T06:08:54.037891+0000","last_undegraded":"2024-05-28T06:08:54.037891+0000","last_fullsized":"2024-05-28T06:08:54.037891+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:59:23.999803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.033234+0000","last_change":"2024-05-28T06:08:54.033234+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.033234+0000","last_undegraded":"2024-05-28T06:08:54.033234+0000","last_fullsized":"2024-05-28T06:08:54.033234+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.045657+0000","last_change":"2024-05-28T06:08:54.043914+0000","last_active":"2024-05-28T06:08:54.045657+0000","last_peered":"2024-05-28T06:08:54.045657+0000","last_clean":"2024-05-28T06:08:54.045657+0000","last_became_active":"2024-05-28T06:08:54.043662+0000","last_became_peered":"2024-05-28T06:08:54.043662+0000","last_unstale":"2024-05-28T06:08:54.045657+0000","last_undegraded":"2024-05-28T06:08:54.045657+0000","last_fullsized":"2024-05-28T06:08:54.045657+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:14:59.855890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.023267+0000","last_change":"2024-05-28T06:08:54.023267+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.023267+0000","last_undegraded":"2024-05-28T06:08:54.023267+0000","last_fullsized":"2024-05-28T06:08:54.023267+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.045619+0000","last_change":"2024-05-28T06:08:54.042952+0000","last_active":"2024-05-28T06:08:54.045619+0000","last_peered":"2024-05-28T06:08:54.045619+0000","last_clean":"2024-05-28T06:08:54.045619+0000","last_became_active":"2024-05-28T06:08:54.042713+0000","last_became_peered":"2024-05-28T06:08:54.042713+0000","last_unstale":"2024-05-28T06:08:54.045619+0000","last_undegraded":"2024-05-28T06:08:54.045619+0000","last_fullsized":"2024-05-28T06:08:54.045619+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:19:17.873244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.028763+0000","last_change":"2024-05-28T06:08:54.028763+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.028763+0000","last_undegraded":"2024-05-28T06:08:54.028763+0000","last_fullsized":"2024-05-28T06:08:54.028763+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.041728+0000","last_change":"2024-05-28T06:08:54.039984+0000","last_active":"2024-05-28T06:08:54.041728+0000","last_peered":"2024-05-28T06:08:54.041728+0000","last_clean":"2024-05-28T06:08:54.041728+0000","last_became_active":"2024-05-28T06:08:54.038849+0000","last_became_peered":"2024-05-28T06:08:54.038849+0000","last_unstale":"2024-05-28T06:08:54.041728+0000","last_undegraded":"2024-05-28T06:08:54.041728+0000","last_fullsized":"2024-05-28T06:08:54.041728+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:52:34.490485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.014980+0000","last_change":"2024-05-28T06:08:54.014980+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.014980+0000","last_undegraded":"2024-05-28T06:08:54.014980+0000","last_fullsized":"2024-05-28T06:08:54.014980+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.044755+0000","last_change":"2024-05-28T06:08:54.043822+0000","last_active":"2024-05-28T06:08:54.044755+0000","last_peered":"2024-05-28T06:08:54.044755+0000","last_clean":"2024-05-28T06:08:54.044755+0000","last_became_active":"2024-05-28T06:08:54.043402+0000","last_became_peered":"2024-05-28T06:08:54.043402+0000","last_unstale":"2024-05-28T06:08:54.044755+0000","last_undegraded":"2024-05-28T06:08:54.044755+0000","last_fullsized":"2024-05-28T06:08:54.044755+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:16:01.996383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.021077+0000","last_change":"2024-05-28T06:08:54.021077+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.021077+0000","last_undegraded":"2024-05-28T06:08:54.021077+0000","last_fullsized":"2024-05-28T06:08:54.021077+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.047595+0000","last_change":"2024-05-28T06:08:54.046229+0000","last_active":"2024-05-28T06:08:54.047595+0000","last_peered":"2024-05-28T06:08:54.047595+0000","last_clean":"2024-05-28T06:08:54.047595+0000","last_became_active":"2024-05-28T06:08:54.045868+0000","last_became_peered":"2024-05-28T06:08:54.045868+0000","last_unstale":"2024-05-28T06:08:54.047595+0000","last_undegraded":"2024-05-28T06:08:54.047595+0000","last_fullsized":"2024-05-28T06:08:54.047595+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:52:34.490485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.034897+0000","last_change":"2024-05-28T06:08:54.034897+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.034897+0000","last_undegraded":"2024-05-28T06:08:54.034897+0000","last_fullsized":"2024-05-28T06:08:54.034897+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.040552+0000","last_change":"2024-05-28T06:08:54.034283+0000","last_active":"2024-05-28T06:08:54.040552+0000","last_peered":"2024-05-28T06:08:54.040552+0000","last_clean":"2024-05-28T06:08:54.040552+0000","last_became_active":"2024-05-28T06:08:54.033985+0000","last_became_peered":"2024-05-28T06:08:54.033985+0000","last_unstale":"2024-05-28T06:08:54.040552+0000","last_undegraded":"2024-05-28T06:08:54.040552+0000","last_fullsized":"2024-05-28T06:08:54.040552+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:34:31.576736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.020069+0000","last_change":"2024-05-28T06:08:54.020069+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.020069+0000","last_undegraded":"2024-05-28T06:08:54.020069+0000","last_fullsized":"2024-05-28T06:08:54.020069+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.048832+0000","last_change":"2024-05-28T06:08:54.047896+0000","last_active":"2024-05-28T06:08:54.048832+0000","last_peered":"2024-05-28T06:08:54.048832+0000","last_clean":"2024-05-28T06:08:54.048832+0000","last_became_active":"2024-05-28T06:08:54.047535+0000","last_became_peered":"2024-05-28T06:08:54.047535+0000","last_unstale":"2024-05-28T06:08:54.048832+0000","last_undegraded":"2024-05-28T06:08:54.048832+0000","last_fullsized":"2024-05-28T06:08:54.048832+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:14:59.855890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.027417+0000","last_change":"2024-05-28T06:08:54.027417+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.027417+0000","last_undegraded":"2024-05-28T06:08:54.027417+0000","last_fullsized":"2024-05-28T06:08:54.027417+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.041547+0000","last_change":"2024-05-28T06:08:54.037383+0000","last_active":"2024-05-28T06:08:54.041547+0000","last_peered":"2024-05-28T06:08:54.041547+0000","last_clean":"2024-05-28T06:08:54.041547+0000","last_became_active":"2024-05-28T06:08:54.036598+0000","last_became_peered":"2024-05-28T06:08:54.036598+0000","last_unstale":"2024-05-28T06:08:54.041547+0000","last_undegraded":"2024-05-28T06:08:54.041547+0000","last_fullsized":"2024-05-28T06:08:54.041547+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:11:06.365855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.022643+0000","last_change":"2024-05-28T06:08:54.022643+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.022643+0000","last_undegraded":"2024-05-28T06:08:54.022643+0000","last_fullsized":"2024-05-28T06:08:54.022643+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.048321+0000","last_change":"2024-05-28T06:08:54.046483+0000","last_active":"2024-05-28T06:08:54.048321+0000","last_peered":"2024-05-28T06:08:54.048321+0000","last_clean":"2024-05-28T06:08:54.048321+0000","last_became_active":"2024-05-28T06:08:54.045605+0000","last_became_peered":"2024-05-28T06:08:54.045605+0000","last_unstale":"2024-05-28T06:08:54.048321+0000","last_undegraded":"2024-05-28T06:08:54.048321+0000","last_fullsized":"2024-05-28T06:08:54.048321+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:20:27.131737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.024821+0000","last_change":"2024-05-28T06:08:54.024821+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.024821+0000","last_undegraded":"2024-05-28T06:08:54.024821+0000","last_fullsized":"2024-05-28T06:08:54.024821+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.040115+0000","last_change":"2024-05-28T06:08:54.037676+0000","last_active":"2024-05-28T06:08:54.040115+0000","last_peered":"2024-05-28T06:08:54.040115+0000","last_clean":"2024-05-28T06:08:54.040115+0000","last_became_active":"2024-05-28T06:08:54.037301+0000","last_became_peered":"2024-05-28T06:08:54.037301+0000","last_unstale":"2024-05-28T06:08:54.040115+0000","last_undegraded":"2024-05-28T06:08:54.040115+0000","last_fullsized":"2024-05-28T06:08:54.040115+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:01:37.817834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.019527+0000","last_change":"2024-05-28T06:08:54.019527+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.019527+0000","last_undegraded":"2024-05-28T06:08:54.019527+0000","last_fullsized":"2024-05-28T06:08:54.019527+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.045431+0000","last_change":"2024-05-28T06:08:54.044188+0000","last_active":"2024-05-28T06:08:54.045431+0000","last_peered":"2024-05-28T06:08:54.045431+0000","last_clean":"2024-05-28T06:08:54.045431+0000","last_became_active":"2024-05-28T06:08:54.043655+0000","last_became_peered":"2024-05-28T06:08:54.043655+0000","last_unstale":"2024-05-28T06:08:54.045431+0000","last_undegraded":"2024-05-28T06:08:54.045431+0000","last_fullsized":"2024-05-28T06:08:54.045431+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:27:40.364767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.013578+0000","last_change":"2024-05-28T06:08:54.013578+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.013578+0000","last_undegraded":"2024-05-28T06:08:54.013578+0000","last_fullsized":"2024-05-28T06:08:54.013578+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.040079+0000","last_change":"2024-05-28T06:08:54.038376+0000","last_active":"2024-05-28T06:08:54.040079+0000","last_peered":"2024-05-28T06:08:54.040079+0000","last_clean":"2024-05-28T06:08:54.040079+0000","last_became_active":"2024-05-28T06:08:54.037952+0000","last_became_peered":"2024-05-28T06:08:54.037952+0000","last_unstale":"2024-05-28T06:08:54.040079+0000","last_undegraded":"2024-05-28T06:08:54.040079+0000","last_fullsized":"2024-05-28T06:08:54.040079+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:52:58.719365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.023494+0000","last_change":"2024-05-28T06:08:54.023494+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.023494+0000","last_undegraded":"2024-05-28T06:08:54.023494+0000","last_fullsized":"2024-05-28T06:08:54.023494+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.039837+0000","last_change":"2024-05-28T06:08:54.036597+0000","last_active":"2024-05-28T06:08:54.039837+0000","last_peered":"2024-05-28T06:08:54.039837+0000","last_clean":"2024-05-28T06:08:54.039837+0000","last_became_active":"2024-05-28T06:08:54.034833+0000","last_became_peered":"2024-05-28T06:08:54.034833+0000","last_unstale":"2024-05-28T06:08:54.039837+0000","last_undegraded":"2024-05-28T06:08:54.039837+0000","last_fullsized":"2024-05-28T06:08:54.039837+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:34:31.576736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":213,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.037334+0000","last_change":"2024-05-28T06:04:01.039129+0000","last_active":"2024-05-28T06:08:54.037334+0000","last_peered":"2024-05-28T06:08:54.037334+0000","last_clean":"2024-05-28T06:08:54.037334+0000","last_became_active":"2024-05-28T06:04:01.033906+0000","last_became_peered":"2024-05-28T06:04:01.033906+0000","last_unstale":"2024-05-28T06:08:54.037334+0000","last_undegraded":"2024-05-28T06:08:54.037334+0000","last_fullsized":"2024-05-28T06:08:54.037334+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:03:59.864237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:03:59.864237+0000","last_clean_scrub_stamp":"2024-05-28T06:03:59.864237+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:17:36.440245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574910,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36400,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335440,"statfs":{"total":96636764160,"available":96599490560,"internally_reserved":0,"allocated":1085440,"data_stored":787442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574912,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31680,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340160,"statfs":{"total":96636764160,"available":96604323840,"internally_reserved":0,"allocated":503808,"data_stored":203369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574910,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35836,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94336004,"statfs":{"total":96636764160,"available":96600068096,"internally_reserved":0,"allocated":503808,"data_stored":203369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574910,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35836,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94336004,"statfs":{"total":96636764160,"available":96600068096,"internally_reserved":0,"allocated":503808,"data_stored":203369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574912,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35820,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94336020,"statfs":{"total":96636764160,"available":96600084480,"internally_reserved":0,"allocated":491520,"data_stored":197074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574911,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35840,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94336000,"statfs":{"total":96636764160,"available":96600064000,"internally_reserved":0,"allocated":503808,"data_stored":203369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574911,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35832,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94336008,"statfs":{"total":96636764160,"available":96600072192,"internally_reserved":0,"allocated":503808,"data_stored":203369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574910,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36416,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335424,"statfs":{"total":96636764160,"available":96599474176,"internally_reserved":0,"allocated":1097728,"data_stored":793737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-28T06:09:00.351 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-28T06:09:00.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:00.602+0000 7f7c14000640 1 Processor -- start 2024-05-28T06:09:00.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:00.602+0000 7f7c14000640 1 -- start start 2024-05-28T06:09:00.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:00.604+0000 7f7c14000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f7c0c0d0730 0x7f7c0c0d0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:00.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:00.604+0000 7f7c14000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f7c0c0b1ab0 0x7f7c0c0b1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:00.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:00.604+0000 7f7c14000640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c0c06c010 con 0x7f7c0c160000 2024-05-28T06:09:00.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:00.604+0000 7f7c14000640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f7c0c0b1620 con 0x7f7c0c0b1ab0 2024-05-28T06:09:00.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:00.604+0000 7f7c14000640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f7c0c06c2e0 con 0x7f7c0c0d0730 2024-05-28T06:09:00.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:00.604+0000 7f7c12000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f7c0c0b1ab0 0x7f7c0c0b1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:00.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:00.604+0000 7f7c12000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f7c0c0b1ab0 0x7f7c0c0b1ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47496/0 (socket says 172.21.15.96:47496) 2024-05-28T06:09:00.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:00.604+0000 7f7c12000640 1 -- 172.21.15.96:0/1592595546 learned_addr learned my addr 172.21.15.96:0/1592595546 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:09:00.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:00.605+0000 7f7c12000640 1 -- 172.21.15.96:0/1592595546 >> v2:172.21.15.123:3300/0 conn(0x7f7c0c0d0730 msgr2=0x7f7c0c0d0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:09:00.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:00.605+0000 7f7c12000640 1 --2- 172.21.15.96:0/1592595546 >> v2:172.21.15.123:3300/0 conn(0x7f7c0c0d0730 0x7f7c0c0d0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:00.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:00.605+0000 7f7c12000640 1 -- 172.21.15.96:0/1592595546 >> v1:172.21.15.123:6789/0 conn(0x7f7c0c160000 legacy=0x7f7c0c06b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:09:00.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:00.605+0000 7f7c12000640 1 -- 172.21.15.96:0/1592595546 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c0c0b0cf0 con 0x7f7c0c0b1ab0 2024-05-28T06:09:00.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:00.606+0000 7f7c12000640 1 --2- 172.21.15.96:0/1592595546 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c0c0b1ab0 0x7f7c0c0b1ec0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c0cfbe0 tx=0x7f7c0402a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcb5634ede42f098 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:09:00.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:00.606+0000 7f7c0be00640 1 -- 172.21.15.96:0/1592595546 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c04036070 con 0x7f7c0c0b1ab0 2024-05-28T06:09:00.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:00.606+0000 7f7c0be00640 1 -- 172.21.15.96:0/1592595546 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c04031600 con 0x7f7c0c0b1ab0 2024-05-28T06:09:00.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:00.606+0000 7f7c0be00640 1 -- 172.21.15.96:0/1592595546 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c040319d0 con 0x7f7c0c0b1ab0 2024-05-28T06:09:00.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:00.607+0000 7f7c14000640 1 -- 172.21.15.96:0/1592595546 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c0c0b1ab0 msgr2=0x7f7c0c0b1ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:09:00.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:00.607+0000 7f7c14000640 1 --2- 172.21.15.96:0/1592595546 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c0c0b1ab0 0x7f7c0c0b1ec0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c0cfbe0 tx=0x7f7c0402a6f0 comp rx=0 tx=0).stop 2024-05-28T06:09:00.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:00.607+0000 7f7c14000640 1 -- 172.21.15.96:0/1592595546 shutdown_connections 2024-05-28T06:09:00.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:00.607+0000 7f7c14000640 1 --2- 172.21.15.96:0/1592595546 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c0c0b1ab0 0x7f7c0c0b1ec0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:00.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:00.607+0000 7f7c14000640 1 --2- 172.21.15.96:0/1592595546 >> v2:172.21.15.123:3300/0 conn(0x7f7c0c0d0730 0x7f7c0c0d0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:00.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:00.607+0000 7f7c14000640 1 -- 172.21.15.96:0/1592595546 >> 172.21.15.96:0/1592595546 conn(0x7f7c0c09aa30 msgr2=0x7f7c0c0b2f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:09:00.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:00.607+0000 7f7c14000640 1 -- 172.21.15.96:0/1592595546 shutdown_connections 2024-05-28T06:09:00.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:00.607+0000 7f7c14000640 1 -- 172.21.15.96:0/1592595546 wait complete. 2024-05-28T06:09:00.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:00.608+0000 7f7c14000640 1 Processor -- start 2024-05-28T06:09:00.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:00.609+0000 7f7c14000640 1 -- start start 2024-05-28T06:09:00.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:00.610+0000 7f7c14000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c0c0b1ab0 0x7f7c0c15e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:00.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:00.610+0000 7f7c14000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7c0c0d0730 0x7f7c0c0c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:00.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:00.610+0000 7f7c14000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c0c160000 0x7f7c0c0aebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:00.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:00.610+0000 7f7c14000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f7c0c0d1100 con 0x7f7c0c160000 2024-05-28T06:09:00.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:00.610+0000 7f7c14000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f7c0c0b12b0 con 0x7f7c0c0b1ab0 2024-05-28T06:09:00.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:00.610+0000 7f7c14000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f7c0c1612c0 con 0x7f7c0c0d0730 2024-05-28T06:09:00.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:00.610+0000 7f7c12000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c0c160000 0x7f7c0c0aebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:00.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:00.610+0000 7f7c12000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c0c160000 0x7f7c0c0aebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47512/0 (socket says 172.21.15.96:47512) 2024-05-28T06:09:00.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:00.610+0000 7f7c12000640 1 -- 172.21.15.96:0/275034519 learned_addr learned my addr 172.21.15.96:0/275034519 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:09:00.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:00.610+0000 7f7c11600640 1 --2- 172.21.15.96:0/275034519 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c0c0b1ab0 0x7f7c0c15e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:00.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:00.610+0000 7f7c12000640 1 -- 172.21.15.96:0/275034519 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7c0c0d0730 msgr2=0x7f7c0c0c5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:09:00.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:00.610+0000 7f7c12000640 1 --2- 172.21.15.96:0/275034519 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7c0c0d0730 0x7f7c0c0c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:00.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:00.610+0000 7f7c12000640 1 -- 172.21.15.96:0/275034519 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c0c0b1ab0 msgr2=0x7f7c0c15e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:09:00.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:00.610+0000 7f7c12000640 1 --2- 172.21.15.96:0/275034519 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c0c0b1ab0 0x7f7c0c15e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:00.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:00.610+0000 7f7c12000640 1 -- 172.21.15.96:0/275034519 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c0c15ed30 con 0x7f7c0c160000 2024-05-28T06:09:00.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:00.610+0000 7f7c11600640 1 --2- 172.21.15.96:0/275034519 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c0c0b1ab0 0x7f7c0c15e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:09:00.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:00.611+0000 7f7c12000640 1 --2- 172.21.15.96:0/275034519 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c0c160000 0x7f7c0c0aebc0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f7c04034040 tx=0x7f7c04004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:09:00.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:00.611+0000 7f7c09600640 1 -- 172.21.15.96:0/275034519 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c04041020 con 0x7f7c0c160000 2024-05-28T06:09:00.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:00.611+0000 7f7c09600640 1 -- 172.21.15.96:0/275034519 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c040077a0 con 0x7f7c0c160000 2024-05-28T06:09:00.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:00.611+0000 7f7c09600640 1 -- 172.21.15.96:0/275034519 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c04036040 con 0x7f7c0c160000 2024-05-28T06:09:00.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:00.611+0000 7f7c14000640 1 -- 172.21.15.96:0/275034519 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c0c0c6760 con 0x7f7c0c160000 2024-05-28T06:09:00.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:00.611+0000 7f7c14000640 1 -- 172.21.15.96:0/275034519 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c0c0c7000 con 0x7f7c0c160000 2024-05-28T06:09:00.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:00.612+0000 7f7c14000640 1 -- 172.21.15.96:0/275034519 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c0c0690f0 con 0x7f7c0c160000 2024-05-28T06:09:00.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:00.613+0000 7f7c09600640 1 -- 172.21.15.96:0/275034519 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 9) ==== 91295+0+0 (secure 0 0 0) 0x7f7c04044020 con 0x7f7c0c160000 2024-05-28T06:09:00.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:00.615+0000 7f7c09600640 1 --2- 172.21.15.96:0/275034519 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7bec06cbf0 0x7f7bec06f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:00.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:00.615+0000 7f7c09600640 1 -- 172.21.15.96:0/275034519 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5737+0+0 (secure 0 0 0) 0x7f7c040ad0d0 con 0x7f7c0c160000 2024-05-28T06:09:00.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:00.615+0000 7f7c11600640 1 --2- 172.21.15.96:0/275034519 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7bec06cbf0 0x7f7bec06f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:00.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:00.616+0000 7f7c11600640 1 --2- 172.21.15.96:0/275034519 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7bec06cbf0 0x7f7bec06f0b0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f7c00003eb0 tx=0x7f7c00009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:09:00.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:00.619+0000 7f7c09600640 1 -- 172.21.15.96:0/275034519 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7c0404b030 con 0x7f7c0c160000 2024-05-28T06:09:00.819 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:00.818+0000 7f7c14000640 1 -- 172.21.15.96:0/275034519 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f7c0c0c6af0 con 0x7f7bec06cbf0 2024-05-28T06:09:00.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:00.836+0000 7f7c09600640 1 -- 172.21.15.96:0/275034519 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+246639 (secure 0 0 0) 0x7f7c00054380 con 0x7f7bec06cbf0 2024-05-28T06:09:00.839 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:09:00.851 INFO:teuthology.orchestra.run.smithi096.stderr:dumped all 2024-05-28T06:09:00.852 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:00.842+0000 7f7c14000640 1 -- 172.21.15.96:0/275034519 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7bec06cbf0 msgr2=0x7f7bec06f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:09:00.852 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:00.842+0000 7f7c14000640 1 --2- 172.21.15.96:0/275034519 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7bec06cbf0 0x7f7bec06f0b0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f7c00003eb0 tx=0x7f7c00009040 comp rx=0 tx=0).stop 2024-05-28T06:09:00.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:00.842+0000 7f7c14000640 1 -- 172.21.15.96:0/275034519 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c0c160000 msgr2=0x7f7c0c0aebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:09:00.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:00.842+0000 7f7c14000640 1 --2- 172.21.15.96:0/275034519 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c0c160000 0x7f7c0c0aebc0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f7c04034040 tx=0x7f7c04004a40 comp rx=0 tx=0).stop 2024-05-28T06:09:00.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:00.842+0000 7f7c14000640 1 -- 172.21.15.96:0/275034519 shutdown_connections 2024-05-28T06:09:00.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:00.842+0000 7f7c14000640 1 --2- 172.21.15.96:0/275034519 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7bec06cbf0 0x7f7bec06f0b0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:00.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:00.842+0000 7f7c14000640 1 --2- 172.21.15.96:0/275034519 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c0c160000 0x7f7c0c0aebc0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:00.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:00.842+0000 7f7c14000640 1 --2- 172.21.15.96:0/275034519 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7c0c0d0730 0x7f7c0c0c5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:00.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:00.842+0000 7f7c14000640 1 --2- 172.21.15.96:0/275034519 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c0c0b1ab0 0x7f7c0c15e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:00.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:00.842+0000 7f7c14000640 1 -- 172.21.15.96:0/275034519 >> 172.21.15.96:0/275034519 conn(0x7f7c0c09aa30 msgr2=0x7f7c0c1e6320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:09:00.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:00.842+0000 7f7c14000640 1 -- 172.21.15.96:0/275034519 shutdown_connections 2024-05-28T06:09:00.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:00.842+0000 7f7c14000640 1 -- 172.21.15.96:0/275034519 wait complete. 2024-05-28T06:09:00.863 INFO:teuthology.orchestra.run.smithi096.stdout:{"pg_ready":true,"pg_map":{"version":188,"stamp":"2024-05-28T06:08:59.591908+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":28,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":283660,"kb_used_data":5072,"kb_used_omap":12,"kb_used_meta":278451,"kb_avail":754691060,"statfs":{"total":773094113280,"available":772803645440,"internally_reserved":0,"allocated":5193728,"data_stored":2795098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":285134384},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001854"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.030805+0000","last_change":"2024-05-28T06:08:54.030805+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.030805+0000","last_undegraded":"2024-05-28T06:08:54.030805+0000","last_fullsized":"2024-05-28T06:08:54.030805+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.023029+0000","last_change":"2024-05-28T06:08:54.023029+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.023029+0000","last_undegraded":"2024-05-28T06:08:54.023029+0000","last_fullsized":"2024-05-28T06:08:54.023029+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.019511+0000","last_change":"2024-05-28T06:08:54.019511+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.019511+0000","last_undegraded":"2024-05-28T06:08:54.019511+0000","last_fullsized":"2024-05-28T06:08:54.019511+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.031775+0000","last_change":"2024-05-28T06:08:54.031775+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.031775+0000","last_undegraded":"2024-05-28T06:08:54.031775+0000","last_fullsized":"2024-05-28T06:08:54.031775+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.021863+0000","last_change":"2024-05-28T06:08:54.021863+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.021863+0000","last_undegraded":"2024-05-28T06:08:54.021863+0000","last_fullsized":"2024-05-28T06:08:54.021863+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.023805+0000","last_change":"2024-05-28T06:08:54.023805+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.023805+0000","last_undegraded":"2024-05-28T06:08:54.023805+0000","last_fullsized":"2024-05-28T06:08:54.023805+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.029593+0000","last_change":"2024-05-28T06:08:54.029593+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.029593+0000","last_undegraded":"2024-05-28T06:08:54.029593+0000","last_fullsized":"2024-05-28T06:08:54.029593+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.022144+0000","last_change":"2024-05-28T06:08:54.022144+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.022144+0000","last_undegraded":"2024-05-28T06:08:54.022144+0000","last_fullsized":"2024-05-28T06:08:54.022144+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.030711+0000","last_change":"2024-05-28T06:08:54.030711+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.030711+0000","last_undegraded":"2024-05-28T06:08:54.030711+0000","last_fullsized":"2024-05-28T06:08:54.030711+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.024096+0000","last_change":"2024-05-28T06:08:54.024096+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.024096+0000","last_undegraded":"2024-05-28T06:08:54.024096+0000","last_fullsized":"2024-05-28T06:08:54.024096+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.018302+0000","last_change":"2024-05-28T06:08:54.018302+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.018302+0000","last_undegraded":"2024-05-28T06:08:54.018302+0000","last_fullsized":"2024-05-28T06:08:54.018302+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.024586+0000","last_change":"2024-05-28T06:08:54.024586+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.024586+0000","last_undegraded":"2024-05-28T06:08:54.024586+0000","last_fullsized":"2024-05-28T06:08:54.024586+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.014791+0000","last_change":"2024-05-28T06:08:54.014791+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.014791+0000","last_undegraded":"2024-05-28T06:08:54.014791+0000","last_fullsized":"2024-05-28T06:08:54.014791+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.027464+0000","last_change":"2024-05-28T06:08:54.027464+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.027464+0000","last_undegraded":"2024-05-28T06:08:54.027464+0000","last_fullsized":"2024-05-28T06:08:54.027464+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.025774+0000","last_change":"2024-05-28T06:08:54.025774+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.025774+0000","last_undegraded":"2024-05-28T06:08:54.025774+0000","last_fullsized":"2024-05-28T06:08:54.025774+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.029213+0000","last_change":"2024-05-28T06:08:54.029213+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.029213+0000","last_undegraded":"2024-05-28T06:08:54.029213+0000","last_fullsized":"2024-05-28T06:08:54.029213+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.014182+0000","last_change":"2024-05-28T06:08:54.014182+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.014182+0000","last_undegraded":"2024-05-28T06:08:54.014182+0000","last_fullsized":"2024-05-28T06:08:54.014182+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.016245+0000","last_change":"2024-05-28T06:08:54.016245+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.016245+0000","last_undegraded":"2024-05-28T06:08:54.016245+0000","last_fullsized":"2024-05-28T06:08:54.016245+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.032693+0000","last_change":"2024-05-28T06:08:54.032693+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.032693+0000","last_undegraded":"2024-05-28T06:08:54.032693+0000","last_fullsized":"2024-05-28T06:08:54.032693+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.014442+0000","last_change":"2024-05-28T06:08:54.014442+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.014442+0000","last_undegraded":"2024-05-28T06:08:54.014442+0000","last_fullsized":"2024-05-28T06:08:54.014442+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.025473+0000","last_change":"2024-05-28T06:08:54.025473+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.025473+0000","last_undegraded":"2024-05-28T06:08:54.025473+0000","last_fullsized":"2024-05-28T06:08:54.025473+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.018238+0000","last_change":"2024-05-28T06:08:54.018238+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.018238+0000","last_undegraded":"2024-05-28T06:08:54.018238+0000","last_fullsized":"2024-05-28T06:08:54.018238+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.029393+0000","last_change":"2024-05-28T06:08:54.029393+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.029393+0000","last_undegraded":"2024-05-28T06:08:54.029393+0000","last_fullsized":"2024-05-28T06:08:54.029393+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.028295+0000","last_change":"2024-05-28T06:08:54.028295+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.028295+0000","last_undegraded":"2024-05-28T06:08:54.028295+0000","last_fullsized":"2024-05-28T06:08:54.028295+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.037153+0000","last_change":"2024-05-28T06:08:54.037153+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.037153+0000","last_undegraded":"2024-05-28T06:08:54.037153+0000","last_fullsized":"2024-05-28T06:08:54.037153+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.015960+0000","last_change":"2024-05-28T06:08:54.015960+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.015960+0000","last_undegraded":"2024-05-28T06:08:54.015960+0000","last_fullsized":"2024-05-28T06:08:54.015960+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.014740+0000","last_change":"2024-05-28T06:08:54.014740+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.014740+0000","last_undegraded":"2024-05-28T06:08:54.014740+0000","last_fullsized":"2024-05-28T06:08:54.014740+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.023762+0000","last_change":"2024-05-28T06:08:54.023762+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.023762+0000","last_undegraded":"2024-05-28T06:08:54.023762+0000","last_fullsized":"2024-05-28T06:08:54.023762+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.030708+0000","last_change":"2024-05-28T06:08:54.030708+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.030708+0000","last_undegraded":"2024-05-28T06:08:54.030708+0000","last_fullsized":"2024-05-28T06:08:54.030708+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.017736+0000","last_change":"2024-05-28T06:08:54.017736+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.017736+0000","last_undegraded":"2024-05-28T06:08:54.017736+0000","last_fullsized":"2024-05-28T06:08:54.017736+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.019413+0000","last_change":"2024-05-28T06:08:54.019413+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.019413+0000","last_undegraded":"2024-05-28T06:08:54.019413+0000","last_fullsized":"2024-05-28T06:08:54.019413+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.016151+0000","last_change":"2024-05-28T06:08:54.016151+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.016151+0000","last_undegraded":"2024-05-28T06:08:54.016151+0000","last_fullsized":"2024-05-28T06:08:54.016151+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.028257+0000","last_change":"2024-05-28T06:08:54.028257+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.028257+0000","last_undegraded":"2024-05-28T06:08:54.028257+0000","last_fullsized":"2024-05-28T06:08:54.028257+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.045658+0000","last_change":"2024-05-28T06:08:54.044108+0000","last_active":"2024-05-28T06:08:54.045658+0000","last_peered":"2024-05-28T06:08:54.045658+0000","last_clean":"2024-05-28T06:08:54.045658+0000","last_became_active":"2024-05-28T06:08:54.043637+0000","last_became_peered":"2024-05-28T06:08:54.043637+0000","last_unstale":"2024-05-28T06:08:54.045658+0000","last_undegraded":"2024-05-28T06:08:54.045658+0000","last_fullsized":"2024-05-28T06:08:54.045658+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:11:14.276087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.036622+0000","last_change":"2024-05-28T06:08:54.036622+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.036622+0000","last_undegraded":"2024-05-28T06:08:54.036622+0000","last_fullsized":"2024-05-28T06:08:54.036622+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.039961+0000","last_change":"2024-05-28T06:08:54.036739+0000","last_active":"2024-05-28T06:08:54.039961+0000","last_peered":"2024-05-28T06:08:54.039961+0000","last_clean":"2024-05-28T06:08:54.039961+0000","last_became_active":"2024-05-28T06:08:54.036446+0000","last_became_peered":"2024-05-28T06:08:54.036446+0000","last_unstale":"2024-05-28T06:08:54.039961+0000","last_undegraded":"2024-05-28T06:08:54.039961+0000","last_fullsized":"2024-05-28T06:08:54.039961+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:07:41.209833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.022836+0000","last_change":"2024-05-28T06:08:54.022836+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.022836+0000","last_undegraded":"2024-05-28T06:08:54.022836+0000","last_fullsized":"2024-05-28T06:08:54.022836+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.040347+0000","last_change":"2024-05-28T06:08:54.035269+0000","last_active":"2024-05-28T06:08:54.040347+0000","last_peered":"2024-05-28T06:08:54.040347+0000","last_clean":"2024-05-28T06:08:54.040347+0000","last_became_active":"2024-05-28T06:08:54.034916+0000","last_became_peered":"2024-05-28T06:08:54.034916+0000","last_unstale":"2024-05-28T06:08:54.040347+0000","last_undegraded":"2024-05-28T06:08:54.040347+0000","last_fullsized":"2024-05-28T06:08:54.040347+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:37:34.767912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.026952+0000","last_change":"2024-05-28T06:08:54.026952+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.026952+0000","last_undegraded":"2024-05-28T06:08:54.026952+0000","last_fullsized":"2024-05-28T06:08:54.026952+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.046591+0000","last_change":"2024-05-28T06:08:54.044956+0000","last_active":"2024-05-28T06:08:54.046591+0000","last_peered":"2024-05-28T06:08:54.046591+0000","last_clean":"2024-05-28T06:08:54.046591+0000","last_became_active":"2024-05-28T06:08:54.044646+0000","last_became_peered":"2024-05-28T06:08:54.044646+0000","last_unstale":"2024-05-28T06:08:54.046591+0000","last_undegraded":"2024-05-28T06:08:54.046591+0000","last_fullsized":"2024-05-28T06:08:54.046591+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:34:31.576736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.014512+0000","last_change":"2024-05-28T06:08:54.014512+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.014512+0000","last_undegraded":"2024-05-28T06:08:54.014512+0000","last_fullsized":"2024-05-28T06:08:54.014512+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.045585+0000","last_change":"2024-05-28T06:08:54.042495+0000","last_active":"2024-05-28T06:08:54.045585+0000","last_peered":"2024-05-28T06:08:54.045585+0000","last_clean":"2024-05-28T06:08:54.045585+0000","last_became_active":"2024-05-28T06:08:54.042143+0000","last_became_peered":"2024-05-28T06:08:54.042143+0000","last_unstale":"2024-05-28T06:08:54.045585+0000","last_undegraded":"2024-05-28T06:08:54.045585+0000","last_fullsized":"2024-05-28T06:08:54.045585+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:20:27.131737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.022747+0000","last_change":"2024-05-28T06:08:54.022747+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.022747+0000","last_undegraded":"2024-05-28T06:08:54.022747+0000","last_fullsized":"2024-05-28T06:08:54.022747+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.039866+0000","last_change":"2024-05-28T06:08:54.030180+0000","last_active":"2024-05-28T06:08:54.039866+0000","last_peered":"2024-05-28T06:08:54.039866+0000","last_clean":"2024-05-28T06:08:54.039866+0000","last_became_active":"2024-05-28T06:08:54.029892+0000","last_became_peered":"2024-05-28T06:08:54.029892+0000","last_unstale":"2024-05-28T06:08:54.039866+0000","last_undegraded":"2024-05-28T06:08:54.039866+0000","last_fullsized":"2024-05-28T06:08:54.039866+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:16:17.652829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.021598+0000","last_change":"2024-05-28T06:08:54.021598+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.021598+0000","last_undegraded":"2024-05-28T06:08:54.021598+0000","last_fullsized":"2024-05-28T06:08:54.021598+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.038065+0000","last_change":"2024-05-28T06:08:54.035697+0000","last_active":"2024-05-28T06:08:54.038065+0000","last_peered":"2024-05-28T06:08:54.038065+0000","last_clean":"2024-05-28T06:08:54.038065+0000","last_became_active":"2024-05-28T06:08:54.032931+0000","last_became_peered":"2024-05-28T06:08:54.032931+0000","last_unstale":"2024-05-28T06:08:54.038065+0000","last_undegraded":"2024-05-28T06:08:54.038065+0000","last_fullsized":"2024-05-28T06:08:54.038065+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:29:27.490700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.030054+0000","last_change":"2024-05-28T06:08:54.030054+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.030054+0000","last_undegraded":"2024-05-28T06:08:54.030054+0000","last_fullsized":"2024-05-28T06:08:54.030054+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.036080+0000","last_change":"2024-05-28T06:08:54.032388+0000","last_active":"2024-05-28T06:08:54.036080+0000","last_peered":"2024-05-28T06:08:54.036080+0000","last_clean":"2024-05-28T06:08:54.036080+0000","last_became_active":"2024-05-28T06:08:54.032095+0000","last_became_peered":"2024-05-28T06:08:54.032095+0000","last_unstale":"2024-05-28T06:08:54.036080+0000","last_undegraded":"2024-05-28T06:08:54.036080+0000","last_fullsized":"2024-05-28T06:08:54.036080+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:21:39.812262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.014966+0000","last_change":"2024-05-28T06:08:54.014966+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.014966+0000","last_undegraded":"2024-05-28T06:08:54.014966+0000","last_fullsized":"2024-05-28T06:08:54.014966+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.039945+0000","last_change":"2024-05-28T06:08:54.027262+0000","last_active":"2024-05-28T06:08:54.039945+0000","last_peered":"2024-05-28T06:08:54.039945+0000","last_clean":"2024-05-28T06:08:54.039945+0000","last_became_active":"2024-05-28T06:08:54.026066+0000","last_became_peered":"2024-05-28T06:08:54.026066+0000","last_unstale":"2024-05-28T06:08:54.039945+0000","last_undegraded":"2024-05-28T06:08:54.039945+0000","last_fullsized":"2024-05-28T06:08:54.039945+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:16:01.996383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.030190+0000","last_change":"2024-05-28T06:08:54.030190+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.030190+0000","last_undegraded":"2024-05-28T06:08:54.030190+0000","last_fullsized":"2024-05-28T06:08:54.030190+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.041630+0000","last_change":"2024-05-28T06:08:54.040877+0000","last_active":"2024-05-28T06:08:54.041630+0000","last_peered":"2024-05-28T06:08:54.041630+0000","last_clean":"2024-05-28T06:08:54.041630+0000","last_became_active":"2024-05-28T06:08:54.040615+0000","last_became_peered":"2024-05-28T06:08:54.040615+0000","last_unstale":"2024-05-28T06:08:54.041630+0000","last_undegraded":"2024-05-28T06:08:54.041630+0000","last_fullsized":"2024-05-28T06:08:54.041630+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:40:59.926424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.018762+0000","last_change":"2024-05-28T06:08:54.018762+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.018762+0000","last_undegraded":"2024-05-28T06:08:54.018762+0000","last_fullsized":"2024-05-28T06:08:54.018762+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.048326+0000","last_change":"2024-05-28T06:08:54.046805+0000","last_active":"2024-05-28T06:08:54.048326+0000","last_peered":"2024-05-28T06:08:54.048326+0000","last_clean":"2024-05-28T06:08:54.048326+0000","last_became_active":"2024-05-28T06:08:54.046336+0000","last_became_peered":"2024-05-28T06:08:54.046336+0000","last_unstale":"2024-05-28T06:08:54.048326+0000","last_undegraded":"2024-05-28T06:08:54.048326+0000","last_fullsized":"2024-05-28T06:08:54.048326+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:19:17.873244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.032123+0000","last_change":"2024-05-28T06:08:54.032123+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.032123+0000","last_undegraded":"2024-05-28T06:08:54.032123+0000","last_fullsized":"2024-05-28T06:08:54.032123+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.031479+0000","last_change":"2024-05-28T06:08:54.029622+0000","last_active":"2024-05-28T06:08:54.031479+0000","last_peered":"2024-05-28T06:08:54.031479+0000","last_clean":"2024-05-28T06:08:54.031479+0000","last_became_active":"2024-05-28T06:08:54.026140+0000","last_became_peered":"2024-05-28T06:08:54.026140+0000","last_unstale":"2024-05-28T06:08:54.031479+0000","last_undegraded":"2024-05-28T06:08:54.031479+0000","last_fullsized":"2024-05-28T06:08:54.031479+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:14:10.883135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.028947+0000","last_change":"2024-05-28T06:08:54.028947+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.028947+0000","last_undegraded":"2024-05-28T06:08:54.028947+0000","last_fullsized":"2024-05-28T06:08:54.028947+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.041714+0000","last_change":"2024-05-28T06:08:54.038320+0000","last_active":"2024-05-28T06:08:54.041714+0000","last_peered":"2024-05-28T06:08:54.041714+0000","last_clean":"2024-05-28T06:08:54.041714+0000","last_became_active":"2024-05-28T06:08:54.037824+0000","last_became_peered":"2024-05-28T06:08:54.037824+0000","last_unstale":"2024-05-28T06:08:54.041714+0000","last_undegraded":"2024-05-28T06:08:54.041714+0000","last_fullsized":"2024-05-28T06:08:54.041714+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:47:29.434576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.021818+0000","last_change":"2024-05-28T06:08:54.021818+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.021818+0000","last_undegraded":"2024-05-28T06:08:54.021818+0000","last_fullsized":"2024-05-28T06:08:54.021818+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.048280+0000","last_change":"2024-05-28T06:08:54.046264+0000","last_active":"2024-05-28T06:08:54.048280+0000","last_peered":"2024-05-28T06:08:54.048280+0000","last_clean":"2024-05-28T06:08:54.048280+0000","last_became_active":"2024-05-28T06:08:54.045957+0000","last_became_peered":"2024-05-28T06:08:54.045957+0000","last_unstale":"2024-05-28T06:08:54.048280+0000","last_undegraded":"2024-05-28T06:08:54.048280+0000","last_fullsized":"2024-05-28T06:08:54.048280+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:47:29.434576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.024967+0000","last_change":"2024-05-28T06:08:54.024967+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.024967+0000","last_undegraded":"2024-05-28T06:08:54.024967+0000","last_fullsized":"2024-05-28T06:08:54.024967+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.041658+0000","last_change":"2024-05-28T06:08:54.038131+0000","last_active":"2024-05-28T06:08:54.041658+0000","last_peered":"2024-05-28T06:08:54.041658+0000","last_clean":"2024-05-28T06:08:54.041658+0000","last_became_active":"2024-05-28T06:08:54.037578+0000","last_became_peered":"2024-05-28T06:08:54.037578+0000","last_unstale":"2024-05-28T06:08:54.041658+0000","last_undegraded":"2024-05-28T06:08:54.041658+0000","last_fullsized":"2024-05-28T06:08:54.041658+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:52:34.490485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.029151+0000","last_change":"2024-05-28T06:08:54.029151+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.029151+0000","last_undegraded":"2024-05-28T06:08:54.029151+0000","last_fullsized":"2024-05-28T06:08:54.029151+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.037597+0000","last_change":"2024-05-28T06:08:54.034498+0000","last_active":"2024-05-28T06:08:54.037597+0000","last_peered":"2024-05-28T06:08:54.037597+0000","last_clean":"2024-05-28T06:08:54.037597+0000","last_became_active":"2024-05-28T06:08:54.034152+0000","last_became_peered":"2024-05-28T06:08:54.034152+0000","last_unstale":"2024-05-28T06:08:54.037597+0000","last_undegraded":"2024-05-28T06:08:54.037597+0000","last_fullsized":"2024-05-28T06:08:54.037597+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:57:06.361350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.015569+0000","last_change":"2024-05-28T06:08:54.015569+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.015569+0000","last_undegraded":"2024-05-28T06:08:54.015569+0000","last_fullsized":"2024-05-28T06:08:54.015569+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.040334+0000","last_change":"2024-05-28T06:08:54.035386+0000","last_active":"2024-05-28T06:08:54.040334+0000","last_peered":"2024-05-28T06:08:54.040334+0000","last_clean":"2024-05-28T06:08:54.040334+0000","last_became_active":"2024-05-28T06:08:54.033511+0000","last_became_peered":"2024-05-28T06:08:54.033511+0000","last_unstale":"2024-05-28T06:08:54.040334+0000","last_undegraded":"2024-05-28T06:08:54.040334+0000","last_fullsized":"2024-05-28T06:08:54.040334+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:51:43.694955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.029865+0000","last_change":"2024-05-28T06:08:54.029865+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.029865+0000","last_undegraded":"2024-05-28T06:08:54.029865+0000","last_fullsized":"2024-05-28T06:08:54.029865+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.039938+0000","last_change":"2024-05-28T06:08:54.035942+0000","last_active":"2024-05-28T06:08:54.039938+0000","last_peered":"2024-05-28T06:08:54.039938+0000","last_clean":"2024-05-28T06:08:54.039938+0000","last_became_active":"2024-05-28T06:08:54.033436+0000","last_became_peered":"2024-05-28T06:08:54.033436+0000","last_unstale":"2024-05-28T06:08:54.039938+0000","last_undegraded":"2024-05-28T06:08:54.039938+0000","last_fullsized":"2024-05-28T06:08:54.039938+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:51:33.930123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.031829+0000","last_change":"2024-05-28T06:08:54.031829+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.031829+0000","last_undegraded":"2024-05-28T06:08:54.031829+0000","last_fullsized":"2024-05-28T06:08:54.031829+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.037891+0000","last_change":"2024-05-28T06:08:54.036250+0000","last_active":"2024-05-28T06:08:54.037891+0000","last_peered":"2024-05-28T06:08:54.037891+0000","last_clean":"2024-05-28T06:08:54.037891+0000","last_became_active":"2024-05-28T06:08:54.034306+0000","last_became_peered":"2024-05-28T06:08:54.034306+0000","last_unstale":"2024-05-28T06:08:54.037891+0000","last_undegraded":"2024-05-28T06:08:54.037891+0000","last_fullsized":"2024-05-28T06:08:54.037891+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:59:23.999803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.033234+0000","last_change":"2024-05-28T06:08:54.033234+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.033234+0000","last_undegraded":"2024-05-28T06:08:54.033234+0000","last_fullsized":"2024-05-28T06:08:54.033234+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.045657+0000","last_change":"2024-05-28T06:08:54.043914+0000","last_active":"2024-05-28T06:08:54.045657+0000","last_peered":"2024-05-28T06:08:54.045657+0000","last_clean":"2024-05-28T06:08:54.045657+0000","last_became_active":"2024-05-28T06:08:54.043662+0000","last_became_peered":"2024-05-28T06:08:54.043662+0000","last_unstale":"2024-05-28T06:08:54.045657+0000","last_undegraded":"2024-05-28T06:08:54.045657+0000","last_fullsized":"2024-05-28T06:08:54.045657+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:14:59.855890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.023267+0000","last_change":"2024-05-28T06:08:54.023267+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.023267+0000","last_undegraded":"2024-05-28T06:08:54.023267+0000","last_fullsized":"2024-05-28T06:08:54.023267+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.045619+0000","last_change":"2024-05-28T06:08:54.042952+0000","last_active":"2024-05-28T06:08:54.045619+0000","last_peered":"2024-05-28T06:08:54.045619+0000","last_clean":"2024-05-28T06:08:54.045619+0000","last_became_active":"2024-05-28T06:08:54.042713+0000","last_became_peered":"2024-05-28T06:08:54.042713+0000","last_unstale":"2024-05-28T06:08:54.045619+0000","last_undegraded":"2024-05-28T06:08:54.045619+0000","last_fullsized":"2024-05-28T06:08:54.045619+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:19:17.873244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.028763+0000","last_change":"2024-05-28T06:08:54.028763+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.028763+0000","last_undegraded":"2024-05-28T06:08:54.028763+0000","last_fullsized":"2024-05-28T06:08:54.028763+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.041728+0000","last_change":"2024-05-28T06:08:54.039984+0000","last_active":"2024-05-28T06:08:54.041728+0000","last_peered":"2024-05-28T06:08:54.041728+0000","last_clean":"2024-05-28T06:08:54.041728+0000","last_became_active":"2024-05-28T06:08:54.038849+0000","last_became_peered":"2024-05-28T06:08:54.038849+0000","last_unstale":"2024-05-28T06:08:54.041728+0000","last_undegraded":"2024-05-28T06:08:54.041728+0000","last_fullsized":"2024-05-28T06:08:54.041728+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:52:34.490485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.014980+0000","last_change":"2024-05-28T06:08:54.014980+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.014980+0000","last_undegraded":"2024-05-28T06:08:54.014980+0000","last_fullsized":"2024-05-28T06:08:54.014980+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.044755+0000","last_change":"2024-05-28T06:08:54.043822+0000","last_active":"2024-05-28T06:08:54.044755+0000","last_peered":"2024-05-28T06:08:54.044755+0000","last_clean":"2024-05-28T06:08:54.044755+0000","last_became_active":"2024-05-28T06:08:54.043402+0000","last_became_peered":"2024-05-28T06:08:54.043402+0000","last_unstale":"2024-05-28T06:08:54.044755+0000","last_undegraded":"2024-05-28T06:08:54.044755+0000","last_fullsized":"2024-05-28T06:08:54.044755+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:16:01.996383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.021077+0000","last_change":"2024-05-28T06:08:54.021077+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.021077+0000","last_undegraded":"2024-05-28T06:08:54.021077+0000","last_fullsized":"2024-05-28T06:08:54.021077+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.047595+0000","last_change":"2024-05-28T06:08:54.046229+0000","last_active":"2024-05-28T06:08:54.047595+0000","last_peered":"2024-05-28T06:08:54.047595+0000","last_clean":"2024-05-28T06:08:54.047595+0000","last_became_active":"2024-05-28T06:08:54.045868+0000","last_became_peered":"2024-05-28T06:08:54.045868+0000","last_unstale":"2024-05-28T06:08:54.047595+0000","last_undegraded":"2024-05-28T06:08:54.047595+0000","last_fullsized":"2024-05-28T06:08:54.047595+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:52:34.490485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.034897+0000","last_change":"2024-05-28T06:08:54.034897+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.034897+0000","last_undegraded":"2024-05-28T06:08:54.034897+0000","last_fullsized":"2024-05-28T06:08:54.034897+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.040552+0000","last_change":"2024-05-28T06:08:54.034283+0000","last_active":"2024-05-28T06:08:54.040552+0000","last_peered":"2024-05-28T06:08:54.040552+0000","last_clean":"2024-05-28T06:08:54.040552+0000","last_became_active":"2024-05-28T06:08:54.033985+0000","last_became_peered":"2024-05-28T06:08:54.033985+0000","last_unstale":"2024-05-28T06:08:54.040552+0000","last_undegraded":"2024-05-28T06:08:54.040552+0000","last_fullsized":"2024-05-28T06:08:54.040552+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:34:31.576736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.020069+0000","last_change":"2024-05-28T06:08:54.020069+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.020069+0000","last_undegraded":"2024-05-28T06:08:54.020069+0000","last_fullsized":"2024-05-28T06:08:54.020069+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.048832+0000","last_change":"2024-05-28T06:08:54.047896+0000","last_active":"2024-05-28T06:08:54.048832+0000","last_peered":"2024-05-28T06:08:54.048832+0000","last_clean":"2024-05-28T06:08:54.048832+0000","last_became_active":"2024-05-28T06:08:54.047535+0000","last_became_peered":"2024-05-28T06:08:54.047535+0000","last_unstale":"2024-05-28T06:08:54.048832+0000","last_undegraded":"2024-05-28T06:08:54.048832+0000","last_fullsized":"2024-05-28T06:08:54.048832+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:14:59.855890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.027417+0000","last_change":"2024-05-28T06:08:54.027417+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.027417+0000","last_undegraded":"2024-05-28T06:08:54.027417+0000","last_fullsized":"2024-05-28T06:08:54.027417+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.041547+0000","last_change":"2024-05-28T06:08:54.037383+0000","last_active":"2024-05-28T06:08:54.041547+0000","last_peered":"2024-05-28T06:08:54.041547+0000","last_clean":"2024-05-28T06:08:54.041547+0000","last_became_active":"2024-05-28T06:08:54.036598+0000","last_became_peered":"2024-05-28T06:08:54.036598+0000","last_unstale":"2024-05-28T06:08:54.041547+0000","last_undegraded":"2024-05-28T06:08:54.041547+0000","last_fullsized":"2024-05-28T06:08:54.041547+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:11:06.365855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.022643+0000","last_change":"2024-05-28T06:08:54.022643+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.022643+0000","last_undegraded":"2024-05-28T06:08:54.022643+0000","last_fullsized":"2024-05-28T06:08:54.022643+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.048321+0000","last_change":"2024-05-28T06:08:54.046483+0000","last_active":"2024-05-28T06:08:54.048321+0000","last_peered":"2024-05-28T06:08:54.048321+0000","last_clean":"2024-05-28T06:08:54.048321+0000","last_became_active":"2024-05-28T06:08:54.045605+0000","last_became_peered":"2024-05-28T06:08:54.045605+0000","last_unstale":"2024-05-28T06:08:54.048321+0000","last_undegraded":"2024-05-28T06:08:54.048321+0000","last_fullsized":"2024-05-28T06:08:54.048321+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:20:27.131737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.024821+0000","last_change":"2024-05-28T06:08:54.024821+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.024821+0000","last_undegraded":"2024-05-28T06:08:54.024821+0000","last_fullsized":"2024-05-28T06:08:54.024821+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.040115+0000","last_change":"2024-05-28T06:08:54.037676+0000","last_active":"2024-05-28T06:08:54.040115+0000","last_peered":"2024-05-28T06:08:54.040115+0000","last_clean":"2024-05-28T06:08:54.040115+0000","last_became_active":"2024-05-28T06:08:54.037301+0000","last_became_peered":"2024-05-28T06:08:54.037301+0000","last_unstale":"2024-05-28T06:08:54.040115+0000","last_undegraded":"2024-05-28T06:08:54.040115+0000","last_fullsized":"2024-05-28T06:08:54.040115+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:01:37.817834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.019527+0000","last_change":"2024-05-28T06:08:54.019527+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.019527+0000","last_undegraded":"2024-05-28T06:08:54.019527+0000","last_fullsized":"2024-05-28T06:08:54.019527+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.045431+0000","last_change":"2024-05-28T06:08:54.044188+0000","last_active":"2024-05-28T06:08:54.045431+0000","last_peered":"2024-05-28T06:08:54.045431+0000","last_clean":"2024-05-28T06:08:54.045431+0000","last_became_active":"2024-05-28T06:08:54.043655+0000","last_became_peered":"2024-05-28T06:08:54.043655+0000","last_unstale":"2024-05-28T06:08:54.045431+0000","last_undegraded":"2024-05-28T06:08:54.045431+0000","last_fullsized":"2024-05-28T06:08:54.045431+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:27:40.364767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.013578+0000","last_change":"2024-05-28T06:08:54.013578+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.013578+0000","last_undegraded":"2024-05-28T06:08:54.013578+0000","last_fullsized":"2024-05-28T06:08:54.013578+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.040079+0000","last_change":"2024-05-28T06:08:54.038376+0000","last_active":"2024-05-28T06:08:54.040079+0000","last_peered":"2024-05-28T06:08:54.040079+0000","last_clean":"2024-05-28T06:08:54.040079+0000","last_became_active":"2024-05-28T06:08:54.037952+0000","last_became_peered":"2024-05-28T06:08:54.037952+0000","last_unstale":"2024-05-28T06:08:54.040079+0000","last_undegraded":"2024-05-28T06:08:54.040079+0000","last_fullsized":"2024-05-28T06:08:54.040079+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:52:58.719365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-05-28T06:08:54.023494+0000","last_change":"2024-05-28T06:08:54.023494+0000","last_active":"2024-05-28T06:08:53.997522+0000","last_peered":"2024-05-28T06:08:53.997522+0000","last_clean":"2024-05-28T06:08:53.997522+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:08:54.023494+0000","last_undegraded":"2024-05-28T06:08:54.023494+0000","last_fullsized":"2024-05-28T06:08:54.023494+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.039837+0000","last_change":"2024-05-28T06:08:54.036597+0000","last_active":"2024-05-28T06:08:54.039837+0000","last_peered":"2024-05-28T06:08:54.039837+0000","last_clean":"2024-05-28T06:08:54.039837+0000","last_became_active":"2024-05-28T06:08:54.034833+0000","last_became_peered":"2024-05-28T06:08:54.034833+0000","last_unstale":"2024-05-28T06:08:54.039837+0000","last_undegraded":"2024-05-28T06:08:54.039837+0000","last_fullsized":"2024-05-28T06:08:54.039837+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:34:31.576736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":213,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-28T06:08:54.037334+0000","last_change":"2024-05-28T06:04:01.039129+0000","last_active":"2024-05-28T06:08:54.037334+0000","last_peered":"2024-05-28T06:08:54.037334+0000","last_clean":"2024-05-28T06:08:54.037334+0000","last_became_active":"2024-05-28T06:04:01.033906+0000","last_became_peered":"2024-05-28T06:04:01.033906+0000","last_unstale":"2024-05-28T06:08:54.037334+0000","last_undegraded":"2024-05-28T06:08:54.037334+0000","last_fullsized":"2024-05-28T06:08:54.037334+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:03:59.864237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:03:59.864237+0000","last_clean_scrub_stamp":"2024-05-28T06:03:59.864237+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:17:36.440245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574910,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36400,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335440,"statfs":{"total":96636764160,"available":96599490560,"internally_reserved":0,"allocated":1085440,"data_stored":787442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574912,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31680,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340160,"statfs":{"total":96636764160,"available":96604323840,"internally_reserved":0,"allocated":503808,"data_stored":203369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574910,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35836,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94336004,"statfs":{"total":96636764160,"available":96600068096,"internally_reserved":0,"allocated":503808,"data_stored":203369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574910,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35836,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94336004,"statfs":{"total":96636764160,"available":96600068096,"internally_reserved":0,"allocated":503808,"data_stored":203369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574912,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35820,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94336020,"statfs":{"total":96636764160,"available":96600084480,"internally_reserved":0,"allocated":491520,"data_stored":197074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574911,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35840,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94336000,"statfs":{"total":96636764160,"available":96600064000,"internally_reserved":0,"allocated":503808,"data_stored":203369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574911,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35832,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94336008,"statfs":{"total":96636764160,"available":96600072192,"internally_reserved":0,"allocated":503808,"data_stored":203369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574910,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36416,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335424,"statfs":{"total":96636764160,"available":96599474176,"internally_reserved":0,"allocated":1097728,"data_stored":793737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-28T06:09:00.868 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.3f is not active+clean 2024-05-28T06:09:00.868 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.3f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-05-28T06:08:54.030805+0000', 'last_change': '2024-05-28T06:08:54.030805+0000', 'last_active': '2024-05-28T06:08:53.997522+0000', 'last_peered': '2024-05-28T06:08:53.997522+0000', 'last_clean': '2024-05-28T06:08:53.997522+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-28T06:08:54.030805+0000', 'last_undegraded': '2024-05-28T06:08:54.030805+0000', 'last_fullsized': '2024-05-28T06:08:54.030805+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_clean_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-05-28T06:09:00.868 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.3e is not active+clean 2024-05-28T06:09:00.869 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.3e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-05-28T06:08:54.023029+0000', 'last_change': '2024-05-28T06:08:54.023029+0000', 'last_active': '2024-05-28T06:08:53.997522+0000', 'last_peered': '2024-05-28T06:08:53.997522+0000', 'last_clean': '2024-05-28T06:08:53.997522+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-28T06:08:54.023029+0000', 'last_undegraded': '2024-05-28T06:08:54.023029+0000', 'last_fullsized': '2024-05-28T06:08:54.023029+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_clean_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-05-28T06:09:00.869 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.3d is not active+clean 2024-05-28T06:09:00.869 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.3d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-05-28T06:08:54.019511+0000', 'last_change': '2024-05-28T06:08:54.019511+0000', 'last_active': '2024-05-28T06:08:53.997522+0000', 'last_peered': '2024-05-28T06:08:53.997522+0000', 'last_clean': '2024-05-28T06:08:53.997522+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-28T06:08:54.019511+0000', 'last_undegraded': '2024-05-28T06:08:54.019511+0000', 'last_fullsized': '2024-05-28T06:08:54.019511+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_clean_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-05-28T06:09:00.869 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.3c is not active+clean 2024-05-28T06:09:00.869 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.3c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-05-28T06:08:54.031775+0000', 'last_change': '2024-05-28T06:08:54.031775+0000', 'last_active': '2024-05-28T06:08:53.997522+0000', 'last_peered': '2024-05-28T06:08:53.997522+0000', 'last_clean': '2024-05-28T06:08:53.997522+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-28T06:08:54.031775+0000', 'last_undegraded': '2024-05-28T06:08:54.031775+0000', 'last_fullsized': '2024-05-28T06:08:54.031775+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_clean_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-05-28T06:09:00.869 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.3b is not active+clean 2024-05-28T06:09:00.869 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.3b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-05-28T06:08:54.021863+0000', 'last_change': '2024-05-28T06:08:54.021863+0000', 'last_active': '2024-05-28T06:08:53.997522+0000', 'last_peered': '2024-05-28T06:08:53.997522+0000', 'last_clean': '2024-05-28T06:08:53.997522+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-28T06:08:54.021863+0000', 'last_undegraded': '2024-05-28T06:08:54.021863+0000', 'last_fullsized': '2024-05-28T06:08:54.021863+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_clean_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-05-28T06:09:00.869 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.3a is not active+clean 2024-05-28T06:09:00.869 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-05-28T06:08:54.023805+0000', 'last_change': '2024-05-28T06:08:54.023805+0000', 'last_active': '2024-05-28T06:08:53.997522+0000', 'last_peered': '2024-05-28T06:08:53.997522+0000', 'last_clean': '2024-05-28T06:08:53.997522+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-28T06:08:54.023805+0000', 'last_undegraded': '2024-05-28T06:08:54.023805+0000', 'last_fullsized': '2024-05-28T06:08:54.023805+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_clean_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-05-28T06:09:00.869 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.39 is not active+clean 2024-05-28T06:09:00.869 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.39', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-05-28T06:08:54.029593+0000', 'last_change': '2024-05-28T06:08:54.029593+0000', 'last_active': '2024-05-28T06:08:53.997522+0000', 'last_peered': '2024-05-28T06:08:53.997522+0000', 'last_clean': '2024-05-28T06:08:53.997522+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-28T06:08:54.029593+0000', 'last_undegraded': '2024-05-28T06:08:54.029593+0000', 'last_fullsized': '2024-05-28T06:08:54.029593+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_clean_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-05-28T06:09:00.870 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.38 is not active+clean 2024-05-28T06:09:00.870 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.38', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-05-28T06:08:54.022144+0000', 'last_change': '2024-05-28T06:08:54.022144+0000', 'last_active': '2024-05-28T06:08:53.997522+0000', 'last_peered': '2024-05-28T06:08:53.997522+0000', 'last_clean': '2024-05-28T06:08:53.997522+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-28T06:08:54.022144+0000', 'last_undegraded': '2024-05-28T06:08:54.022144+0000', 'last_fullsized': '2024-05-28T06:08:54.022144+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_clean_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-05-28T06:09:00.870 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.37 is not active+clean 2024-05-28T06:09:00.870 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.37', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-05-28T06:08:54.030711+0000', 'last_change': '2024-05-28T06:08:54.030711+0000', 'last_active': '2024-05-28T06:08:53.997522+0000', 'last_peered': '2024-05-28T06:08:53.997522+0000', 'last_clean': '2024-05-28T06:08:53.997522+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-28T06:08:54.030711+0000', 'last_undegraded': '2024-05-28T06:08:54.030711+0000', 'last_fullsized': '2024-05-28T06:08:54.030711+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_clean_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-05-28T06:09:00.870 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.36 is not active+clean 2024-05-28T06:09:00.870 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.36', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-05-28T06:08:54.024096+0000', 'last_change': '2024-05-28T06:08:54.024096+0000', 'last_active': '2024-05-28T06:08:53.997522+0000', 'last_peered': '2024-05-28T06:08:53.997522+0000', 'last_clean': '2024-05-28T06:08:53.997522+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-28T06:08:54.024096+0000', 'last_undegraded': '2024-05-28T06:08:54.024096+0000', 'last_fullsized': '2024-05-28T06:08:54.024096+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_clean_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-05-28T06:09:00.870 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.35 is not active+clean 2024-05-28T06:09:00.870 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.35', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-05-28T06:08:54.018302+0000', 'last_change': '2024-05-28T06:08:54.018302+0000', 'last_active': '2024-05-28T06:08:53.997522+0000', 'last_peered': '2024-05-28T06:08:53.997522+0000', 'last_clean': '2024-05-28T06:08:53.997522+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-28T06:08:54.018302+0000', 'last_undegraded': '2024-05-28T06:08:54.018302+0000', 'last_fullsized': '2024-05-28T06:08:54.018302+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_clean_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-05-28T06:09:00.870 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.34 is not active+clean 2024-05-28T06:09:00.870 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.34', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-05-28T06:08:54.024586+0000', 'last_change': '2024-05-28T06:08:54.024586+0000', 'last_active': '2024-05-28T06:08:53.997522+0000', 'last_peered': '2024-05-28T06:08:53.997522+0000', 'last_clean': '2024-05-28T06:08:53.997522+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-28T06:08:54.024586+0000', 'last_undegraded': '2024-05-28T06:08:54.024586+0000', 'last_fullsized': '2024-05-28T06:08:54.024586+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_clean_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-05-28T06:09:00.871 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.33 is not active+clean 2024-05-28T06:09:00.871 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.33', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-05-28T06:08:54.014791+0000', 'last_change': '2024-05-28T06:08:54.014791+0000', 'last_active': '2024-05-28T06:08:53.997522+0000', 'last_peered': '2024-05-28T06:08:53.997522+0000', 'last_clean': '2024-05-28T06:08:53.997522+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-28T06:08:54.014791+0000', 'last_undegraded': '2024-05-28T06:08:54.014791+0000', 'last_fullsized': '2024-05-28T06:08:54.014791+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_clean_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-05-28T06:09:00.871 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.32 is not active+clean 2024-05-28T06:09:00.871 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.32', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-05-28T06:08:54.027464+0000', 'last_change': '2024-05-28T06:08:54.027464+0000', 'last_active': '2024-05-28T06:08:53.997522+0000', 'last_peered': '2024-05-28T06:08:53.997522+0000', 'last_clean': '2024-05-28T06:08:53.997522+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-28T06:08:54.027464+0000', 'last_undegraded': '2024-05-28T06:08:54.027464+0000', 'last_fullsized': '2024-05-28T06:08:54.027464+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_clean_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-05-28T06:09:00.871 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.31 is not active+clean 2024-05-28T06:09:00.871 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.31', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-05-28T06:08:54.025774+0000', 'last_change': '2024-05-28T06:08:54.025774+0000', 'last_active': '2024-05-28T06:08:53.997522+0000', 'last_peered': '2024-05-28T06:08:53.997522+0000', 'last_clean': '2024-05-28T06:08:53.997522+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-28T06:08:54.025774+0000', 'last_undegraded': '2024-05-28T06:08:54.025774+0000', 'last_fullsized': '2024-05-28T06:08:54.025774+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_clean_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-05-28T06:09:00.871 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.30 is not active+clean 2024-05-28T06:09:00.871 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.30', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-05-28T06:08:54.029213+0000', 'last_change': '2024-05-28T06:08:54.029213+0000', 'last_active': '2024-05-28T06:08:53.997522+0000', 'last_peered': '2024-05-28T06:08:53.997522+0000', 'last_clean': '2024-05-28T06:08:53.997522+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-28T06:08:54.029213+0000', 'last_undegraded': '2024-05-28T06:08:54.029213+0000', 'last_fullsized': '2024-05-28T06:08:54.029213+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_clean_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-05-28T06:09:00.871 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.2f is not active+clean 2024-05-28T06:09:00.871 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.2f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-05-28T06:08:54.014182+0000', 'last_change': '2024-05-28T06:08:54.014182+0000', 'last_active': '2024-05-28T06:08:53.997522+0000', 'last_peered': '2024-05-28T06:08:53.997522+0000', 'last_clean': '2024-05-28T06:08:53.997522+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-28T06:08:54.014182+0000', 'last_undegraded': '2024-05-28T06:08:54.014182+0000', 'last_fullsized': '2024-05-28T06:08:54.014182+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_clean_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-05-28T06:09:00.871 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.2e is not active+clean 2024-05-28T06:09:00.871 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.2e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-05-28T06:08:54.016245+0000', 'last_change': '2024-05-28T06:08:54.016245+0000', 'last_active': '2024-05-28T06:08:53.997522+0000', 'last_peered': '2024-05-28T06:08:53.997522+0000', 'last_clean': '2024-05-28T06:08:53.997522+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-28T06:08:54.016245+0000', 'last_undegraded': '2024-05-28T06:08:54.016245+0000', 'last_fullsized': '2024-05-28T06:08:54.016245+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_clean_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-05-28T06:09:00.872 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.2d is not active+clean 2024-05-28T06:09:00.872 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.2d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-05-28T06:08:54.032693+0000', 'last_change': '2024-05-28T06:08:54.032693+0000', 'last_active': '2024-05-28T06:08:53.997522+0000', 'last_peered': '2024-05-28T06:08:53.997522+0000', 'last_clean': '2024-05-28T06:08:53.997522+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-28T06:08:54.032693+0000', 'last_undegraded': '2024-05-28T06:08:54.032693+0000', 'last_fullsized': '2024-05-28T06:08:54.032693+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_clean_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-05-28T06:09:00.872 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.2c is not active+clean 2024-05-28T06:09:00.872 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.2c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-05-28T06:08:54.014442+0000', 'last_change': '2024-05-28T06:08:54.014442+0000', 'last_active': '2024-05-28T06:08:53.997522+0000', 'last_peered': '2024-05-28T06:08:53.997522+0000', 'last_clean': '2024-05-28T06:08:53.997522+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-28T06:08:54.014442+0000', 'last_undegraded': '2024-05-28T06:08:54.014442+0000', 'last_fullsized': '2024-05-28T06:08:54.014442+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_clean_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-05-28T06:09:00.872 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.2b is not active+clean 2024-05-28T06:09:00.872 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.2b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-05-28T06:08:54.025473+0000', 'last_change': '2024-05-28T06:08:54.025473+0000', 'last_active': '2024-05-28T06:08:53.997522+0000', 'last_peered': '2024-05-28T06:08:53.997522+0000', 'last_clean': '2024-05-28T06:08:53.997522+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-28T06:08:54.025473+0000', 'last_undegraded': '2024-05-28T06:08:54.025473+0000', 'last_fullsized': '2024-05-28T06:08:54.025473+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_clean_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-05-28T06:09:00.872 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.2a is not active+clean 2024-05-28T06:09:00.872 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.2a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-05-28T06:08:54.018238+0000', 'last_change': '2024-05-28T06:08:54.018238+0000', 'last_active': '2024-05-28T06:08:53.997522+0000', 'last_peered': '2024-05-28T06:08:53.997522+0000', 'last_clean': '2024-05-28T06:08:53.997522+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-28T06:08:54.018238+0000', 'last_undegraded': '2024-05-28T06:08:54.018238+0000', 'last_fullsized': '2024-05-28T06:08:54.018238+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_clean_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-05-28T06:09:00.872 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.29 is not active+clean 2024-05-28T06:09:00.873 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.29', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-05-28T06:08:54.029393+0000', 'last_change': '2024-05-28T06:08:54.029393+0000', 'last_active': '2024-05-28T06:08:53.997522+0000', 'last_peered': '2024-05-28T06:08:53.997522+0000', 'last_clean': '2024-05-28T06:08:53.997522+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-28T06:08:54.029393+0000', 'last_undegraded': '2024-05-28T06:08:54.029393+0000', 'last_fullsized': '2024-05-28T06:08:54.029393+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_clean_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-05-28T06:09:00.873 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.28 is not active+clean 2024-05-28T06:09:00.873 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.28', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-05-28T06:08:54.028295+0000', 'last_change': '2024-05-28T06:08:54.028295+0000', 'last_active': '2024-05-28T06:08:53.997522+0000', 'last_peered': '2024-05-28T06:08:53.997522+0000', 'last_clean': '2024-05-28T06:08:53.997522+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-28T06:08:54.028295+0000', 'last_undegraded': '2024-05-28T06:08:54.028295+0000', 'last_fullsized': '2024-05-28T06:08:54.028295+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_clean_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-05-28T06:09:00.873 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.27 is not active+clean 2024-05-28T06:09:00.873 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.27', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-05-28T06:08:54.037153+0000', 'last_change': '2024-05-28T06:08:54.037153+0000', 'last_active': '2024-05-28T06:08:53.997522+0000', 'last_peered': '2024-05-28T06:08:53.997522+0000', 'last_clean': '2024-05-28T06:08:53.997522+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-28T06:08:54.037153+0000', 'last_undegraded': '2024-05-28T06:08:54.037153+0000', 'last_fullsized': '2024-05-28T06:08:54.037153+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_clean_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-05-28T06:09:00.873 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.26 is not active+clean 2024-05-28T06:09:00.873 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.26', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-05-28T06:08:54.015960+0000', 'last_change': '2024-05-28T06:08:54.015960+0000', 'last_active': '2024-05-28T06:08:53.997522+0000', 'last_peered': '2024-05-28T06:08:53.997522+0000', 'last_clean': '2024-05-28T06:08:53.997522+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-28T06:08:54.015960+0000', 'last_undegraded': '2024-05-28T06:08:54.015960+0000', 'last_fullsized': '2024-05-28T06:08:54.015960+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_clean_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-05-28T06:09:00.873 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.25 is not active+clean 2024-05-28T06:09:00.873 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.25', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-05-28T06:08:54.014740+0000', 'last_change': '2024-05-28T06:08:54.014740+0000', 'last_active': '2024-05-28T06:08:53.997522+0000', 'last_peered': '2024-05-28T06:08:53.997522+0000', 'last_clean': '2024-05-28T06:08:53.997522+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-28T06:08:54.014740+0000', 'last_undegraded': '2024-05-28T06:08:54.014740+0000', 'last_fullsized': '2024-05-28T06:08:54.014740+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_clean_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-05-28T06:09:00.873 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.24 is not active+clean 2024-05-28T06:09:00.873 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.24', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-05-28T06:08:54.023762+0000', 'last_change': '2024-05-28T06:08:54.023762+0000', 'last_active': '2024-05-28T06:08:53.997522+0000', 'last_peered': '2024-05-28T06:08:53.997522+0000', 'last_clean': '2024-05-28T06:08:53.997522+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-28T06:08:54.023762+0000', 'last_undegraded': '2024-05-28T06:08:54.023762+0000', 'last_fullsized': '2024-05-28T06:08:54.023762+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_clean_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-05-28T06:09:00.874 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.23 is not active+clean 2024-05-28T06:09:00.874 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.23', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-05-28T06:08:54.030708+0000', 'last_change': '2024-05-28T06:08:54.030708+0000', 'last_active': '2024-05-28T06:08:53.997522+0000', 'last_peered': '2024-05-28T06:08:53.997522+0000', 'last_clean': '2024-05-28T06:08:53.997522+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-28T06:08:54.030708+0000', 'last_undegraded': '2024-05-28T06:08:54.030708+0000', 'last_fullsized': '2024-05-28T06:08:54.030708+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_clean_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-05-28T06:09:00.874 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.22 is not active+clean 2024-05-28T06:09:00.874 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.22', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-05-28T06:08:54.017736+0000', 'last_change': '2024-05-28T06:08:54.017736+0000', 'last_active': '2024-05-28T06:08:53.997522+0000', 'last_peered': '2024-05-28T06:08:53.997522+0000', 'last_clean': '2024-05-28T06:08:53.997522+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-28T06:08:54.017736+0000', 'last_undegraded': '2024-05-28T06:08:54.017736+0000', 'last_fullsized': '2024-05-28T06:08:54.017736+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_clean_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-05-28T06:09:00.874 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.21 is not active+clean 2024-05-28T06:09:00.874 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.21', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-05-28T06:08:54.019413+0000', 'last_change': '2024-05-28T06:08:54.019413+0000', 'last_active': '2024-05-28T06:08:53.997522+0000', 'last_peered': '2024-05-28T06:08:53.997522+0000', 'last_clean': '2024-05-28T06:08:53.997522+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-28T06:08:54.019413+0000', 'last_undegraded': '2024-05-28T06:08:54.019413+0000', 'last_fullsized': '2024-05-28T06:08:54.019413+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_clean_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-05-28T06:09:00.874 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.20 is not active+clean 2024-05-28T06:09:00.874 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.20', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-05-28T06:08:54.016151+0000', 'last_change': '2024-05-28T06:08:54.016151+0000', 'last_active': '2024-05-28T06:08:53.997522+0000', 'last_peered': '2024-05-28T06:08:53.997522+0000', 'last_clean': '2024-05-28T06:08:53.997522+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-28T06:08:54.016151+0000', 'last_undegraded': '2024-05-28T06:08:54.016151+0000', 'last_fullsized': '2024-05-28T06:08:54.016151+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_clean_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-05-28T06:09:00.874 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1e is not active+clean 2024-05-28T06:09:00.874 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-05-28T06:08:54.028257+0000', 'last_change': '2024-05-28T06:08:54.028257+0000', 'last_active': '2024-05-28T06:08:53.997522+0000', 'last_peered': '2024-05-28T06:08:53.997522+0000', 'last_clean': '2024-05-28T06:08:53.997522+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-28T06:08:54.028257+0000', 'last_undegraded': '2024-05-28T06:08:54.028257+0000', 'last_fullsized': '2024-05-28T06:08:54.028257+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_clean_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-05-28T06:09:00.875 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1f is not active+clean 2024-05-28T06:09:00.875 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-05-28T06:08:54.036622+0000', 'last_change': '2024-05-28T06:08:54.036622+0000', 'last_active': '2024-05-28T06:08:53.997522+0000', 'last_peered': '2024-05-28T06:08:53.997522+0000', 'last_clean': '2024-05-28T06:08:53.997522+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-28T06:08:54.036622+0000', 'last_undegraded': '2024-05-28T06:08:54.036622+0000', 'last_fullsized': '2024-05-28T06:08:54.036622+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_clean_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-05-28T06:09:00.875 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1c is not active+clean 2024-05-28T06:09:00.875 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-05-28T06:08:54.022836+0000', 'last_change': '2024-05-28T06:08:54.022836+0000', 'last_active': '2024-05-28T06:08:53.997522+0000', 'last_peered': '2024-05-28T06:08:53.997522+0000', 'last_clean': '2024-05-28T06:08:53.997522+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-28T06:08:54.022836+0000', 'last_undegraded': '2024-05-28T06:08:54.022836+0000', 'last_fullsized': '2024-05-28T06:08:54.022836+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_clean_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-05-28T06:09:00.875 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1d is not active+clean 2024-05-28T06:09:00.875 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-05-28T06:08:54.026952+0000', 'last_change': '2024-05-28T06:08:54.026952+0000', 'last_active': '2024-05-28T06:08:53.997522+0000', 'last_peered': '2024-05-28T06:08:53.997522+0000', 'last_clean': '2024-05-28T06:08:53.997522+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-28T06:08:54.026952+0000', 'last_undegraded': '2024-05-28T06:08:54.026952+0000', 'last_fullsized': '2024-05-28T06:08:54.026952+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_clean_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-05-28T06:09:00.875 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1a is not active+clean 2024-05-28T06:09:00.875 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-05-28T06:08:54.014512+0000', 'last_change': '2024-05-28T06:08:54.014512+0000', 'last_active': '2024-05-28T06:08:53.997522+0000', 'last_peered': '2024-05-28T06:08:53.997522+0000', 'last_clean': '2024-05-28T06:08:53.997522+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-28T06:08:54.014512+0000', 'last_undegraded': '2024-05-28T06:08:54.014512+0000', 'last_fullsized': '2024-05-28T06:08:54.014512+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_clean_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-05-28T06:09:00.875 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1b is not active+clean 2024-05-28T06:09:00.875 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-05-28T06:08:54.022747+0000', 'last_change': '2024-05-28T06:08:54.022747+0000', 'last_active': '2024-05-28T06:08:53.997522+0000', 'last_peered': '2024-05-28T06:08:53.997522+0000', 'last_clean': '2024-05-28T06:08:53.997522+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-28T06:08:54.022747+0000', 'last_undegraded': '2024-05-28T06:08:54.022747+0000', 'last_fullsized': '2024-05-28T06:08:54.022747+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_clean_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-05-28T06:09:00.875 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.18 is not active+clean 2024-05-28T06:09:00.876 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-05-28T06:08:54.021598+0000', 'last_change': '2024-05-28T06:08:54.021598+0000', 'last_active': '2024-05-28T06:08:53.997522+0000', 'last_peered': '2024-05-28T06:08:53.997522+0000', 'last_clean': '2024-05-28T06:08:53.997522+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-28T06:08:54.021598+0000', 'last_undegraded': '2024-05-28T06:08:54.021598+0000', 'last_fullsized': '2024-05-28T06:08:54.021598+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_clean_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-05-28T06:09:00.876 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.19 is not active+clean 2024-05-28T06:09:00.876 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-05-28T06:08:54.030054+0000', 'last_change': '2024-05-28T06:08:54.030054+0000', 'last_active': '2024-05-28T06:08:53.997522+0000', 'last_peered': '2024-05-28T06:08:53.997522+0000', 'last_clean': '2024-05-28T06:08:53.997522+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-28T06:08:54.030054+0000', 'last_undegraded': '2024-05-28T06:08:54.030054+0000', 'last_fullsized': '2024-05-28T06:08:54.030054+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_clean_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-05-28T06:09:00.876 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.16 is not active+clean 2024-05-28T06:09:00.876 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-05-28T06:08:54.014966+0000', 'last_change': '2024-05-28T06:08:54.014966+0000', 'last_active': '2024-05-28T06:08:53.997522+0000', 'last_peered': '2024-05-28T06:08:53.997522+0000', 'last_clean': '2024-05-28T06:08:53.997522+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-28T06:08:54.014966+0000', 'last_undegraded': '2024-05-28T06:08:54.014966+0000', 'last_fullsized': '2024-05-28T06:08:54.014966+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_clean_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-05-28T06:09:00.876 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.17 is not active+clean 2024-05-28T06:09:00.876 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-05-28T06:08:54.030190+0000', 'last_change': '2024-05-28T06:08:54.030190+0000', 'last_active': '2024-05-28T06:08:53.997522+0000', 'last_peered': '2024-05-28T06:08:53.997522+0000', 'last_clean': '2024-05-28T06:08:53.997522+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-28T06:08:54.030190+0000', 'last_undegraded': '2024-05-28T06:08:54.030190+0000', 'last_fullsized': '2024-05-28T06:08:54.030190+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_clean_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-05-28T06:09:00.876 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.14 is not active+clean 2024-05-28T06:09:00.876 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-05-28T06:08:54.018762+0000', 'last_change': '2024-05-28T06:08:54.018762+0000', 'last_active': '2024-05-28T06:08:53.997522+0000', 'last_peered': '2024-05-28T06:08:53.997522+0000', 'last_clean': '2024-05-28T06:08:53.997522+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-28T06:08:54.018762+0000', 'last_undegraded': '2024-05-28T06:08:54.018762+0000', 'last_fullsized': '2024-05-28T06:08:54.018762+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_clean_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-05-28T06:09:00.876 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.15 is not active+clean 2024-05-28T06:09:00.876 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-05-28T06:08:54.032123+0000', 'last_change': '2024-05-28T06:08:54.032123+0000', 'last_active': '2024-05-28T06:08:53.997522+0000', 'last_peered': '2024-05-28T06:08:53.997522+0000', 'last_clean': '2024-05-28T06:08:53.997522+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-28T06:08:54.032123+0000', 'last_undegraded': '2024-05-28T06:08:54.032123+0000', 'last_fullsized': '2024-05-28T06:08:54.032123+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_clean_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-05-28T06:09:00.877 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.12 is not active+clean 2024-05-28T06:09:00.877 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-05-28T06:08:54.028947+0000', 'last_change': '2024-05-28T06:08:54.028947+0000', 'last_active': '2024-05-28T06:08:53.997522+0000', 'last_peered': '2024-05-28T06:08:53.997522+0000', 'last_clean': '2024-05-28T06:08:53.997522+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-28T06:08:54.028947+0000', 'last_undegraded': '2024-05-28T06:08:54.028947+0000', 'last_fullsized': '2024-05-28T06:08:54.028947+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_clean_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-05-28T06:09:00.877 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.13 is not active+clean 2024-05-28T06:09:00.877 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-05-28T06:08:54.021818+0000', 'last_change': '2024-05-28T06:08:54.021818+0000', 'last_active': '2024-05-28T06:08:53.997522+0000', 'last_peered': '2024-05-28T06:08:53.997522+0000', 'last_clean': '2024-05-28T06:08:53.997522+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-28T06:08:54.021818+0000', 'last_undegraded': '2024-05-28T06:08:54.021818+0000', 'last_fullsized': '2024-05-28T06:08:54.021818+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_clean_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-05-28T06:09:00.877 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.10 is not active+clean 2024-05-28T06:09:00.877 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-05-28T06:08:54.024967+0000', 'last_change': '2024-05-28T06:08:54.024967+0000', 'last_active': '2024-05-28T06:08:53.997522+0000', 'last_peered': '2024-05-28T06:08:53.997522+0000', 'last_clean': '2024-05-28T06:08:53.997522+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-28T06:08:54.024967+0000', 'last_undegraded': '2024-05-28T06:08:54.024967+0000', 'last_fullsized': '2024-05-28T06:08:54.024967+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_clean_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-05-28T06:09:00.877 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.11 is not active+clean 2024-05-28T06:09:00.877 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-05-28T06:08:54.029151+0000', 'last_change': '2024-05-28T06:08:54.029151+0000', 'last_active': '2024-05-28T06:08:53.997522+0000', 'last_peered': '2024-05-28T06:08:53.997522+0000', 'last_clean': '2024-05-28T06:08:53.997522+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-28T06:08:54.029151+0000', 'last_undegraded': '2024-05-28T06:08:54.029151+0000', 'last_fullsized': '2024-05-28T06:08:54.029151+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_clean_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-05-28T06:09:00.877 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.e is not active+clean 2024-05-28T06:09:00.877 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-05-28T06:08:54.015569+0000', 'last_change': '2024-05-28T06:08:54.015569+0000', 'last_active': '2024-05-28T06:08:53.997522+0000', 'last_peered': '2024-05-28T06:08:53.997522+0000', 'last_clean': '2024-05-28T06:08:53.997522+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-28T06:08:54.015569+0000', 'last_undegraded': '2024-05-28T06:08:54.015569+0000', 'last_fullsized': '2024-05-28T06:08:54.015569+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_clean_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-05-28T06:09:00.878 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.f is not active+clean 2024-05-28T06:09:00.878 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-05-28T06:08:54.029865+0000', 'last_change': '2024-05-28T06:08:54.029865+0000', 'last_active': '2024-05-28T06:08:53.997522+0000', 'last_peered': '2024-05-28T06:08:53.997522+0000', 'last_clean': '2024-05-28T06:08:53.997522+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-28T06:08:54.029865+0000', 'last_undegraded': '2024-05-28T06:08:54.029865+0000', 'last_fullsized': '2024-05-28T06:08:54.029865+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_clean_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-05-28T06:09:00.878 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.c is not active+clean 2024-05-28T06:09:00.878 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-05-28T06:08:54.031829+0000', 'last_change': '2024-05-28T06:08:54.031829+0000', 'last_active': '2024-05-28T06:08:53.997522+0000', 'last_peered': '2024-05-28T06:08:53.997522+0000', 'last_clean': '2024-05-28T06:08:53.997522+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-28T06:08:54.031829+0000', 'last_undegraded': '2024-05-28T06:08:54.031829+0000', 'last_fullsized': '2024-05-28T06:08:54.031829+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_clean_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-05-28T06:09:00.878 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.d is not active+clean 2024-05-28T06:09:00.878 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-05-28T06:08:54.033234+0000', 'last_change': '2024-05-28T06:08:54.033234+0000', 'last_active': '2024-05-28T06:08:53.997522+0000', 'last_peered': '2024-05-28T06:08:53.997522+0000', 'last_clean': '2024-05-28T06:08:53.997522+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-28T06:08:54.033234+0000', 'last_undegraded': '2024-05-28T06:08:54.033234+0000', 'last_fullsized': '2024-05-28T06:08:54.033234+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_clean_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-05-28T06:09:00.878 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.a is not active+clean 2024-05-28T06:09:00.878 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-05-28T06:08:54.023267+0000', 'last_change': '2024-05-28T06:08:54.023267+0000', 'last_active': '2024-05-28T06:08:53.997522+0000', 'last_peered': '2024-05-28T06:08:53.997522+0000', 'last_clean': '2024-05-28T06:08:53.997522+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-28T06:08:54.023267+0000', 'last_undegraded': '2024-05-28T06:08:54.023267+0000', 'last_fullsized': '2024-05-28T06:08:54.023267+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_clean_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-05-28T06:09:00.878 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.b is not active+clean 2024-05-28T06:09:00.878 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-05-28T06:08:54.028763+0000', 'last_change': '2024-05-28T06:08:54.028763+0000', 'last_active': '2024-05-28T06:08:53.997522+0000', 'last_peered': '2024-05-28T06:08:53.997522+0000', 'last_clean': '2024-05-28T06:08:53.997522+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-28T06:08:54.028763+0000', 'last_undegraded': '2024-05-28T06:08:54.028763+0000', 'last_fullsized': '2024-05-28T06:08:54.028763+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_clean_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-05-28T06:09:00.878 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.8 is not active+clean 2024-05-28T06:09:00.879 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-05-28T06:08:54.014980+0000', 'last_change': '2024-05-28T06:08:54.014980+0000', 'last_active': '2024-05-28T06:08:53.997522+0000', 'last_peered': '2024-05-28T06:08:53.997522+0000', 'last_clean': '2024-05-28T06:08:53.997522+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-28T06:08:54.014980+0000', 'last_undegraded': '2024-05-28T06:08:54.014980+0000', 'last_fullsized': '2024-05-28T06:08:54.014980+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_clean_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-05-28T06:09:00.879 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.9 is not active+clean 2024-05-28T06:09:00.879 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-05-28T06:08:54.021077+0000', 'last_change': '2024-05-28T06:08:54.021077+0000', 'last_active': '2024-05-28T06:08:53.997522+0000', 'last_peered': '2024-05-28T06:08:53.997522+0000', 'last_clean': '2024-05-28T06:08:53.997522+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-28T06:08:54.021077+0000', 'last_undegraded': '2024-05-28T06:08:54.021077+0000', 'last_fullsized': '2024-05-28T06:08:54.021077+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_clean_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-05-28T06:09:00.879 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.7 is not active+clean 2024-05-28T06:09:00.879 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-05-28T06:08:54.034897+0000', 'last_change': '2024-05-28T06:08:54.034897+0000', 'last_active': '2024-05-28T06:08:53.997522+0000', 'last_peered': '2024-05-28T06:08:53.997522+0000', 'last_clean': '2024-05-28T06:08:53.997522+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-28T06:08:54.034897+0000', 'last_undegraded': '2024-05-28T06:08:54.034897+0000', 'last_fullsized': '2024-05-28T06:08:54.034897+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_clean_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-05-28T06:09:00.879 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.4 is not active+clean 2024-05-28T06:09:00.879 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-05-28T06:08:54.020069+0000', 'last_change': '2024-05-28T06:08:54.020069+0000', 'last_active': '2024-05-28T06:08:53.997522+0000', 'last_peered': '2024-05-28T06:08:53.997522+0000', 'last_clean': '2024-05-28T06:08:53.997522+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-28T06:08:54.020069+0000', 'last_undegraded': '2024-05-28T06:08:54.020069+0000', 'last_fullsized': '2024-05-28T06:08:54.020069+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_clean_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-05-28T06:09:00.879 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.5 is not active+clean 2024-05-28T06:09:00.879 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-05-28T06:08:54.027417+0000', 'last_change': '2024-05-28T06:08:54.027417+0000', 'last_active': '2024-05-28T06:08:53.997522+0000', 'last_peered': '2024-05-28T06:08:53.997522+0000', 'last_clean': '2024-05-28T06:08:53.997522+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-28T06:08:54.027417+0000', 'last_undegraded': '2024-05-28T06:08:54.027417+0000', 'last_fullsized': '2024-05-28T06:08:54.027417+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_clean_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-05-28T06:09:00.879 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.2 is not active+clean 2024-05-28T06:09:00.879 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-05-28T06:08:54.022643+0000', 'last_change': '2024-05-28T06:08:54.022643+0000', 'last_active': '2024-05-28T06:08:53.997522+0000', 'last_peered': '2024-05-28T06:08:53.997522+0000', 'last_clean': '2024-05-28T06:08:53.997522+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-28T06:08:54.022643+0000', 'last_undegraded': '2024-05-28T06:08:54.022643+0000', 'last_fullsized': '2024-05-28T06:08:54.022643+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_clean_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-05-28T06:09:00.880 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.3 is not active+clean 2024-05-28T06:09:00.880 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-05-28T06:08:54.024821+0000', 'last_change': '2024-05-28T06:08:54.024821+0000', 'last_active': '2024-05-28T06:08:53.997522+0000', 'last_peered': '2024-05-28T06:08:53.997522+0000', 'last_clean': '2024-05-28T06:08:53.997522+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-28T06:08:54.024821+0000', 'last_undegraded': '2024-05-28T06:08:54.024821+0000', 'last_fullsized': '2024-05-28T06:08:54.024821+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_clean_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-05-28T06:09:00.880 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.0 is not active+clean 2024-05-28T06:09:00.880 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-05-28T06:08:54.019527+0000', 'last_change': '2024-05-28T06:08:54.019527+0000', 'last_active': '2024-05-28T06:08:53.997522+0000', 'last_peered': '2024-05-28T06:08:53.997522+0000', 'last_clean': '2024-05-28T06:08:53.997522+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-28T06:08:54.019527+0000', 'last_undegraded': '2024-05-28T06:08:54.019527+0000', 'last_fullsized': '2024-05-28T06:08:54.019527+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_clean_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-05-28T06:09:00.880 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1 is not active+clean 2024-05-28T06:09:00.880 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-05-28T06:08:54.013578+0000', 'last_change': '2024-05-28T06:08:54.013578+0000', 'last_active': '2024-05-28T06:08:53.997522+0000', 'last_peered': '2024-05-28T06:08:53.997522+0000', 'last_clean': '2024-05-28T06:08:53.997522+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-28T06:08:54.013578+0000', 'last_undegraded': '2024-05-28T06:08:54.013578+0000', 'last_fullsized': '2024-05-28T06:08:54.013578+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_clean_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-05-28T06:09:00.880 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.6 is not active+clean 2024-05-28T06:09:00.880 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-05-28T06:08:54.023494+0000', 'last_change': '2024-05-28T06:08:54.023494+0000', 'last_active': '2024-05-28T06:08:53.997522+0000', 'last_peered': '2024-05-28T06:08:53.997522+0000', 'last_clean': '2024-05-28T06:08:53.997522+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-28T06:08:54.023494+0000', 'last_undegraded': '2024-05-28T06:08:54.023494+0000', 'last_fullsized': '2024-05-28T06:08:54.023494+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'last_clean_scrub_stamp': '2024-05-28T06:08:53.997522+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-05-28T06:09:00.881 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-05-28T06:09:00.980 INFO:tasks.ceph.mds.a.smithi096.stderr:2024-05-28T06:09:00.979+0000 7fed0e400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-28T06:09:01.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.108+0000 7fbe6a200640 1 Processor -- start 2024-05-28T06:09:01.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.109+0000 7fbe6a200640 1 -- start start 2024-05-28T06:09:01.120 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.110+0000 7fbe6a200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbe6415ea40 0x7fbe640d4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:01.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.110+0000 7fbe6a200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fbe640d1040 0x7fbe640d1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:01.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.110+0000 7fbe6a200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe640d19a0 con 0x7fbe64162520 2024-05-28T06:09:01.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.110+0000 7fbe6a200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fbe6415ee20 con 0x7fbe640d1040 2024-05-28T06:09:01.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.110+0000 7fbe6a200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fbe640cf7e0 con 0x7fbe6415ea40 2024-05-28T06:09:01.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.110+0000 7fbe5be00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fbe640d1040 0x7fbe640d1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:01.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.110+0000 7fbe63e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fbe64162520 0x7fbe64162970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:51182/0 (socket says 172.21.15.96:51182) 2024-05-28T06:09:01.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.110+0000 7fbe63e00640 1 -- 172.21.15.96:0/4229788261 learned_addr learned my addr 172.21.15.96:0/4229788261 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:09:01.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.111+0000 7fbe63400640 1 --2- 172.21.15.96:0/4229788261 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbe6415ea40 0x7fbe640d4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:01.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.111+0000 7fbe5be00640 1 -- 172.21.15.96:0/4229788261 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbe6415ea40 msgr2=0x7fbe640d4f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:09:01.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.111+0000 7fbe5be00640 1 --2- 172.21.15.96:0/4229788261 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbe6415ea40 0x7fbe640d4f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:01.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.111+0000 7fbe5be00640 1 -- 172.21.15.96:0/4229788261 >> v1:172.21.15.96:6789/0 conn(0x7fbe64162520 legacy=0x7fbe64162970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:09:01.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.111+0000 7fbe5be00640 1 -- 172.21.15.96:0/4229788261 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe6415fe50 con 0x7fbe640d1040 2024-05-28T06:09:01.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.111+0000 7fbe63400640 1 --2- 172.21.15.96:0/4229788261 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbe6415ea40 0x7fbe640d4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:09:01.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.112+0000 7fbe5be00640 1 --2- 172.21.15.96:0/4229788261 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe640d1040 0x7fbe640d1460 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fbe4c0097a0 tx=0x7fbe4c02a0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc8792f1df66251f server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:09:01.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.112+0000 7fbe62a00640 1 -- 172.21.15.96:0/4229788261 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe4c037070 con 0x7fbe640d1040 2024-05-28T06:09:01.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.112+0000 7fbe62a00640 1 -- 172.21.15.96:0/4229788261 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe4c02ae40 con 0x7fbe640d1040 2024-05-28T06:09:01.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.112+0000 7fbe62a00640 1 -- 172.21.15.96:0/4229788261 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe4c032660 con 0x7fbe640d1040 2024-05-28T06:09:01.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.113+0000 7fbe6a200640 1 -- 172.21.15.96:0/4229788261 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe640d1040 msgr2=0x7fbe640d1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:09:01.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.113+0000 7fbe6a200640 1 --2- 172.21.15.96:0/4229788261 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe640d1040 0x7fbe640d1460 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fbe4c0097a0 tx=0x7fbe4c02a0f0 comp rx=0 tx=0).stop 2024-05-28T06:09:01.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.113+0000 7fbe6a200640 1 -- 172.21.15.96:0/4229788261 shutdown_connections 2024-05-28T06:09:01.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.113+0000 7fbe6a200640 1 --2- 172.21.15.96:0/4229788261 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe640d1040 0x7fbe640d1460 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:01.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.113+0000 7fbe6a200640 1 --2- 172.21.15.96:0/4229788261 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbe6415ea40 0x7fbe640d4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:01.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.113+0000 7fbe6a200640 1 -- 172.21.15.96:0/4229788261 >> 172.21.15.96:0/4229788261 conn(0x7fbe6408bfc0 msgr2=0x7fbe640d4630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:09:01.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.113+0000 7fbe6a200640 1 -- 172.21.15.96:0/4229788261 shutdown_connections 2024-05-28T06:09:01.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.113+0000 7fbe6a200640 1 -- 172.21.15.96:0/4229788261 wait complete. 2024-05-28T06:09:01.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.115+0000 7fbe6a200640 1 Processor -- start 2024-05-28T06:09:01.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.115+0000 7fbe6a200640 1 -- start start 2024-05-28T06:09:01.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.116+0000 7fbe6a200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe640d1040 0x7fbe640b9d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:01.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.116+0000 7fbe6a200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbe6415ea40 0x7fbe640b6710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:01.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.116+0000 7fbe6a200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbe64162520 0x7fbe64152680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:01.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.116+0000 7fbe6a200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fbe640d0040 con 0x7fbe640d1040 2024-05-28T06:09:01.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.116+0000 7fbe6a200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fbe640cfd60 con 0x7fbe6415ea40 2024-05-28T06:09:01.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.116+0000 7fbe6a200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fbe640d0320 con 0x7fbe64162520 2024-05-28T06:09:01.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.116+0000 7fbe63400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe640d1040 0x7fbe640b9d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:01.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.116+0000 7fbe63400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe640d1040 0x7fbe640b9d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47530/0 (socket says 172.21.15.96:47530) 2024-05-28T06:09:01.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.116+0000 7fbe63400640 1 -- 172.21.15.96:0/3179214340 learned_addr learned my addr 172.21.15.96:0/3179214340 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:09:01.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.116+0000 7fbe63400640 1 -- 172.21.15.96:0/3179214340 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbe64162520 msgr2=0x7fbe64152680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:09:01.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.116+0000 7fbe63400640 1 --2- 172.21.15.96:0/3179214340 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbe64162520 0x7fbe64152680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:01.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.116+0000 7fbe63400640 1 -- 172.21.15.96:0/3179214340 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbe6415ea40 msgr2=0x7fbe640b6710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:09:01.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.116+0000 7fbe63400640 1 --2- 172.21.15.96:0/3179214340 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbe6415ea40 0x7fbe640b6710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:01.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.116+0000 7fbe63400640 1 -- 172.21.15.96:0/3179214340 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe640cd1e0 con 0x7fbe640d1040 2024-05-28T06:09:01.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.117+0000 7fbe63400640 1 --2- 172.21.15.96:0/3179214340 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe640d1040 0x7fbe640b9d00 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fbe5400c7f0 tx=0x7fbe5400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:09:01.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.117+0000 7fbe5b400640 1 -- 172.21.15.96:0/3179214340 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe54015070 con 0x7fbe640d1040 2024-05-28T06:09:01.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.117+0000 7fbe5b400640 1 -- 172.21.15.96:0/3179214340 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe54007a30 con 0x7fbe640d1040 2024-05-28T06:09:01.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.117+0000 7fbe5b400640 1 -- 172.21.15.96:0/3179214340 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe54007e00 con 0x7fbe640d1040 2024-05-28T06:09:01.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.117+0000 7fbe6a200640 1 -- 172.21.15.96:0/3179214340 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe640cbae0 con 0x7fbe640d1040 2024-05-28T06:09:01.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.117+0000 7fbe6a200640 1 -- 172.21.15.96:0/3179214340 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe640cc0a0 con 0x7fbe640d1040 2024-05-28T06:09:01.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.118+0000 7fbe6a200640 1 -- 172.21.15.96:0/3179214340 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe640689e0 con 0x7fbe640d1040 2024-05-28T06:09:01.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.119+0000 7fbe5b400640 1 -- 172.21.15.96:0/3179214340 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 9) ==== 91295+0+0 (secure 0 0 0) 0x7fbe54018070 con 0x7fbe640d1040 2024-05-28T06:09:01.125 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.120+0000 7fbe5b400640 1 --2- 172.21.15.96:0/3179214340 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbe4006cbc0 0x7fbe4006f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:01.125 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.120+0000 7fbe5b400640 1 -- 172.21.15.96:0/3179214340 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5737+0+0 (secure 0 0 0) 0x7fbe5408c090 con 0x7fbe640d1040 2024-05-28T06:09:01.125 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.121+0000 7fbe5be00640 1 --2- 172.21.15.96:0/3179214340 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbe4006cbc0 0x7fbe4006f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:01.125 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.122+0000 7fbe5be00640 1 --2- 172.21.15.96:0/3179214340 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbe4006cbc0 0x7fbe4006f080 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fbe4c034000 tx=0x7fbe4c02c000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:09:01.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.125+0000 7fbe5b400640 1 -- 172.21.15.96:0/3179214340 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbe5405a090 con 0x7fbe640d1040 2024-05-28T06:09:01.378 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.376+0000 7fbe6a200640 1 -- 172.21.15.96:0/3179214340 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7fbe640d1c70 con 0x7fbe640d1040 2024-05-28T06:09:01.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.377+0000 7fbe5b400640 1 -- 172.21.15.96:0/3179214340 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1034 (secure 0 0 0) 0x7fbe54059990 con 0x7fbe640d1040 2024-05-28T06:09:01.379 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:09:01.379 INFO:teuthology.orchestra.run.smithi096.stdout:{"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":314,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":45,"num_osds":8,"num_up_osds":8,"osd_up_since":1716876238,"num_in_osds":8,"osd_in_since":1716876229,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"creating+peering","count":64},{"state_name":"active+clean","count":33}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":290467840,"bytes_avail":772803645440,"bytes_total":773094113280,"inactive_pgs_ratio":0.65979379415512085},"fsmap":{"epoch":25,"id":3,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":3,"rank":0,"name":"a","status":"up:active","gid":5859}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-05-28T06:06:33.535436+0000","services":{}},"progress_events":{}} 2024-05-28T06:09:01.381 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.380+0000 7fbe6a200640 1 -- 172.21.15.96:0/3179214340 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbe4006cbc0 msgr2=0x7fbe4006f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:09:01.381 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.380+0000 7fbe6a200640 1 --2- 172.21.15.96:0/3179214340 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbe4006cbc0 0x7fbe4006f080 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fbe4c034000 tx=0x7fbe4c02c000 comp rx=0 tx=0).stop 2024-05-28T06:09:01.382 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.380+0000 7fbe6a200640 1 -- 172.21.15.96:0/3179214340 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe640d1040 msgr2=0x7fbe640b9d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:09:01.382 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.380+0000 7fbe6a200640 1 --2- 172.21.15.96:0/3179214340 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe640d1040 0x7fbe640b9d00 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fbe5400c7f0 tx=0x7fbe5400cbf0 comp rx=0 tx=0).stop 2024-05-28T06:09:01.382 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.380+0000 7fbe6a200640 1 -- 172.21.15.96:0/3179214340 shutdown_connections 2024-05-28T06:09:01.382 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.380+0000 7fbe6a200640 1 --2- 172.21.15.96:0/3179214340 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbe4006cbc0 0x7fbe4006f080 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:01.382 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.380+0000 7fbe6a200640 1 --2- 172.21.15.96:0/3179214340 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbe64162520 0x7fbe64152680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:01.382 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.380+0000 7fbe6a200640 1 --2- 172.21.15.96:0/3179214340 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbe6415ea40 0x7fbe640b6710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:01.382 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.380+0000 7fbe6a200640 1 --2- 172.21.15.96:0/3179214340 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe640d1040 0x7fbe640b9d00 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:01.382 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.380+0000 7fbe6a200640 1 -- 172.21.15.96:0/3179214340 >> 172.21.15.96:0/3179214340 conn(0x7fbe6408bfc0 msgr2=0x7fbe640b31f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:09:01.382 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.380+0000 7fbe6a200640 1 -- 172.21.15.96:0/3179214340 shutdown_connections 2024-05-28T06:09:01.382 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.380+0000 7fbe6a200640 1 -- 172.21.15.96:0/3179214340 wait complete. 2024-05-28T06:09:01.399 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-05-28T06:09:01.400 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-28T06:09:01.594 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.592+0000 7f6c70400640 1 Processor -- start 2024-05-28T06:09:01.594 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.593+0000 7f6c70400640 1 -- start start 2024-05-28T06:09:01.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.594+0000 7f6c70400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f6c680b49c0 0x7f6c68165d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:01.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.594+0000 7f6c70400640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c6806e9e0 con 0x7f6c6815ae80 2024-05-28T06:09:01.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.594+0000 7f6c70400640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c680b4e70 con 0x7f6c680b3fb0 2024-05-28T06:09:01.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.594+0000 7f6c70400640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f6c680b31f0 con 0x7f6c680b49c0 2024-05-28T06:09:01.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.594+0000 7f6c6da00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f6c6815ae80 0x7f6c6806dbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:51186/0 (socket says 172.21.15.96:51186) 2024-05-28T06:09:01.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.594+0000 7f6c6da00640 1 -- 172.21.15.96:0/171517692 learned_addr learned my addr 172.21.15.96:0/171517692 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:09:01.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.595+0000 7f6c67e00640 1 -- 172.21.15.96:0/171517692 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4163085244 0 0) 0x7f6c6806e9e0 con 0x7f6c6815ae80 2024-05-28T06:09:01.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.595+0000 7f6c67e00640 1 -- 172.21.15.96:0/171517692 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6c50003440 con 0x7f6c6815ae80 2024-05-28T06:09:01.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.595+0000 7f6c6e400640 1 --2- 172.21.15.96:0/171517692 >> v2:172.21.15.123:3300/0 conn(0x7f6c680b49c0 0x7f6c68165d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:01.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.595+0000 7f6c67e00640 1 -- 172.21.15.96:0/171517692 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2836050414 0 0) 0x7f6c50003440 con 0x7f6c6815ae80 2024-05-28T06:09:01.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.595+0000 7f6c67e00640 1 -- 172.21.15.96:0/171517692 >> v2:172.21.15.123:3300/0 conn(0x7f6c680b49c0 msgr2=0x7f6c68165d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:09:01.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.595+0000 7f6c67e00640 1 --2- 172.21.15.96:0/171517692 >> v2:172.21.15.123:3300/0 conn(0x7f6c680b49c0 0x7f6c68165d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:01.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.595+0000 7f6c67e00640 1 -- 172.21.15.96:0/171517692 >> v1:172.21.15.123:6789/0 conn(0x7f6c680b3fb0 legacy=0x7f6c6806e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:09:01.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.595+0000 7f6c67e00640 1 -- 172.21.15.96:0/171517692 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c680b34d0 con 0x7f6c6815ae80 2024-05-28T06:09:01.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.595+0000 7f6c67e00640 1 -- 172.21.15.96:0/171517692 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f6c5c001670 con 0x7f6c6815ae80 2024-05-28T06:09:01.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.595+0000 7f6c67e00640 1 -- 172.21.15.96:0/171517692 >> v1:172.21.15.96:6789/0 conn(0x7f6c6815ae80 legacy=0x7f6c6806dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:09:01.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.595+0000 7f6c67e00640 1 --2- 172.21.15.96:0/171517692 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c50003af0 0x7f6c50003f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:01.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.595+0000 7f6c67e00640 1 --2- 172.21.15.96:0/171517692 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c50008950 0x7f6c5000adc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:01.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.595+0000 7f6c67e00640 1 --2- 172.21.15.96:0/171517692 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6c5000b5c0 0x7f6c5000da80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:01.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.595+0000 7f6c67e00640 1 -- 172.21.15.96:0/171517692 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f6c680b31f0 con 0x7f6c50003af0 2024-05-28T06:09:01.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.595+0000 7f6c67e00640 1 -- 172.21.15.96:0/171517692 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f6c5000e360 con 0x7f6c50008950 2024-05-28T06:09:01.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.595+0000 7f6c67e00640 1 -- 172.21.15.96:0/171517692 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f6c5000e560 con 0x7f6c5000b5c0 2024-05-28T06:09:01.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.596+0000 7f6c6da00640 1 --2- 172.21.15.96:0/171517692 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c50003af0 0x7f6c50003f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:01.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.596+0000 7f6c6d000640 1 --2- 172.21.15.96:0/171517692 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c50008950 0x7f6c5000adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:01.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.596+0000 7f6c6e400640 1 --2- 172.21.15.96:0/171517692 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6c5000b5c0 0x7f6c5000da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:01.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.596+0000 7f6c6da00640 1 -- 172.21.15.96:0/171517692 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6c5000b5c0 msgr2=0x7f6c5000da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:09:01.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.596+0000 7f6c6da00640 1 --2- 172.21.15.96:0/171517692 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6c5000b5c0 0x7f6c5000da80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:01.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.596+0000 7f6c6da00640 1 -- 172.21.15.96:0/171517692 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c50008950 msgr2=0x7f6c5000adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:09:01.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.596+0000 7f6c6da00640 1 --2- 172.21.15.96:0/171517692 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c50008950 0x7f6c5000adc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:01.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.596+0000 7f6c6da00640 1 -- 172.21.15.96:0/171517692 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c50003440 con 0x7f6c50003af0 2024-05-28T06:09:01.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.596+0000 7f6c6e400640 1 -- 172.21.15.96:0/171517692 reap_dead start 2024-05-28T06:09:01.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.597+0000 7f6c6da00640 1 --2- 172.21.15.96:0/171517692 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c50003af0 0x7f6c50003f10 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f6c5c00c190 tx=0x7f6c50003730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6cd34ceb959a77a server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:09:01.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.597+0000 7f6c67e00640 1 -- 172.21.15.96:0/171517692 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c5c007c60 con 0x7f6c50003af0 2024-05-28T06:09:01.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.597+0000 7f6c67e00640 1 -- 172.21.15.96:0/171517692 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c5c005af0 con 0x7f6c50003af0 2024-05-28T06:09:01.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.597+0000 7f6c67e00640 1 -- 172.21.15.96:0/171517692 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c5c007070 con 0x7f6c50003af0 2024-05-28T06:09:01.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.598+0000 7f6c70400640 1 -- 172.21.15.96:0/171517692 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c50003af0 msgr2=0x7f6c50003f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:09:01.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.598+0000 7f6c70400640 1 --2- 172.21.15.96:0/171517692 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c50003af0 0x7f6c50003f10 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f6c5c00c190 tx=0x7f6c50003730 comp rx=0 tx=0).stop 2024-05-28T06:09:01.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.598+0000 7f6c70400640 1 -- 172.21.15.96:0/171517692 shutdown_connections 2024-05-28T06:09:01.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.598+0000 7f6c70400640 1 --2- 172.21.15.96:0/171517692 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c50003af0 0x7f6c50003f10 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:01.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.598+0000 7f6c70400640 1 -- 172.21.15.96:0/171517692 >> 172.21.15.96:0/171517692 conn(0x7f6c6806bae0 msgr2=0x7f6c6806bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:09:01.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.598+0000 7f6c70400640 1 -- 172.21.15.96:0/171517692 shutdown_connections 2024-05-28T06:09:01.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.598+0000 7f6c70400640 1 -- 172.21.15.96:0/171517692 wait complete. 2024-05-28T06:09:01.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.600+0000 7f6c70400640 1 Processor -- start 2024-05-28T06:09:01.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.600+0000 7f6c70400640 1 -- start start 2024-05-28T06:09:01.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.601+0000 7f6c70400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6c50003af0 0x7f6c6809dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:01.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.601+0000 7f6c70400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c6815bc60 0x7f6c6815c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:01.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.601+0000 7f6c70400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c680cc020 0x7f6c680cc4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:01.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.601+0000 7f6c70400640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f6c6815b260 con 0x7f6c6815bc60 2024-05-28T06:09:01.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.601+0000 7f6c70400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f6c680b3a90 con 0x7f6c680cc020 2024-05-28T06:09:01.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.601+0000 7f6c70400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f6c680b5130 con 0x7f6c50003af0 2024-05-28T06:09:01.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.601+0000 7f6c6d000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c6815bc60 0x7f6c6815c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:01.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.601+0000 7f6c6d000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c6815bc60 0x7f6c6815c0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47556/0 (socket says 172.21.15.96:47556) 2024-05-28T06:09:01.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.601+0000 7f6c6d000640 1 -- 172.21.15.96:0/2941527013 learned_addr learned my addr 172.21.15.96:0/2941527013 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:09:01.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.601+0000 7f6c6d000640 1 -- 172.21.15.96:0/2941527013 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6c50003af0 msgr2=0x7f6c6809dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:09:01.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.601+0000 7f6c6d000640 1 --2- 172.21.15.96:0/2941527013 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6c50003af0 0x7f6c6809dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:01.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.601+0000 7f6c6d000640 1 -- 172.21.15.96:0/2941527013 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c680cc020 msgr2=0x7f6c680cc4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:09:01.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.601+0000 7f6c6e400640 1 --2- 172.21.15.96:0/2941527013 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c680cc020 0x7f6c680cc4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:01.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.602+0000 7f6c6d000640 1 --2- 172.21.15.96:0/2941527013 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c680cc020 0x7f6c680cc4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:01.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.602+0000 7f6c6d000640 1 -- 172.21.15.96:0/2941527013 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c680cc9e0 con 0x7f6c6815bc60 2024-05-28T06:09:01.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.602+0000 7f6c6d000640 1 --2- 172.21.15.96:0/2941527013 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c6815bc60 0x7f6c6815c0e0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f6c5400d540 tx=0x7f6c5400da00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:09:01.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.602+0000 7f6c65600640 1 -- 172.21.15.96:0/2941527013 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c54005480 con 0x7f6c6815bc60 2024-05-28T06:09:01.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.602+0000 7f6c65600640 1 -- 172.21.15.96:0/2941527013 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c54017070 con 0x7f6c6815bc60 2024-05-28T06:09:01.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.602+0000 7f6c70400640 1 -- 172.21.15.96:0/2941527013 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c680ccc50 con 0x7f6c6815bc60 2024-05-28T06:09:01.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.602+0000 7f6c65600640 1 -- 172.21.15.96:0/2941527013 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c54003810 con 0x7f6c6815bc60 2024-05-28T06:09:01.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.602+0000 7f6c70400640 1 -- 172.21.15.96:0/2941527013 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c680cd210 con 0x7f6c6815bc60 2024-05-28T06:09:01.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.603+0000 7f6c70400640 1 -- 172.21.15.96:0/2941527013 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c680689e0 con 0x7f6c6815bc60 2024-05-28T06:09:01.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.604+0000 7f6c65600640 1 -- 172.21.15.96:0/2941527013 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 9) ==== 91295+0+0 (secure 0 0 0) 0x7f6c54003a70 con 0x7f6c6815bc60 2024-05-28T06:09:01.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.605+0000 7f6c65600640 1 --2- 172.21.15.96:0/2941527013 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6c4806cbc0 0x7f6c4806f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:01.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.605+0000 7f6c65600640 1 -- 172.21.15.96:0/2941527013 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5737+0+0 (secure 0 0 0) 0x7f6c5408bf30 con 0x7f6c6815bc60 2024-05-28T06:09:01.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.606+0000 7f6c6da00640 1 --2- 172.21.15.96:0/2941527013 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6c4806cbc0 0x7f6c4806f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:01.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.607+0000 7f6c6da00640 1 --2- 172.21.15.96:0/2941527013 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6c4806cbc0 0x7f6c4806f080 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f6c5c00c160 tx=0x7f6c5c009030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:09:01.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.610+0000 7f6c65600640 1 -- 172.21.15.96:0/2941527013 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6c54019310 con 0x7f6c6815bc60 2024-05-28T06:09:01.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.789+0000 7f6c70400640 1 -- 172.21.15.96:0/2941527013 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f6c680b1070 con 0x7f6c4806cbc0 2024-05-28T06:09:01.809 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.808+0000 7f6c65600640 1 -- 172.21.15.96:0/2941527013 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251960 (secure 0 0 0) 0x7f6c680b1070 con 0x7f6c4806cbc0 2024-05-28T06:09:01.813 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:09:01.814 INFO:teuthology.orchestra.run.smithi096.stderr:dumped all 2024-05-28T06:09:01.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.814+0000 7f6c70400640 1 -- 172.21.15.96:0/2941527013 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6c4806cbc0 msgr2=0x7f6c4806f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:09:01.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.814+0000 7f6c70400640 1 --2- 172.21.15.96:0/2941527013 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6c4806cbc0 0x7f6c4806f080 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f6c5c00c160 tx=0x7f6c5c009030 comp rx=0 tx=0).stop 2024-05-28T06:09:01.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.814+0000 7f6c70400640 1 -- 172.21.15.96:0/2941527013 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c6815bc60 msgr2=0x7f6c6815c0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:09:01.816 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.814+0000 7f6c70400640 1 --2- 172.21.15.96:0/2941527013 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c6815bc60 0x7f6c6815c0e0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f6c5400d540 tx=0x7f6c5400da00 comp rx=0 tx=0).stop 2024-05-28T06:09:01.816 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.814+0000 7f6c70400640 1 -- 172.21.15.96:0/2941527013 shutdown_connections 2024-05-28T06:09:01.816 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.814+0000 7f6c70400640 1 --2- 172.21.15.96:0/2941527013 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6c4806cbc0 0x7f6c4806f080 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:01.816 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.814+0000 7f6c70400640 1 --2- 172.21.15.96:0/2941527013 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c680cc020 0x7f6c680cc4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:01.816 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.814+0000 7f6c70400640 1 --2- 172.21.15.96:0/2941527013 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c6815bc60 0x7f6c6815c0e0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:01.816 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.814+0000 7f6c70400640 1 --2- 172.21.15.96:0/2941527013 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6c50003af0 0x7f6c6809dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:01.816 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.814+0000 7f6c70400640 1 -- 172.21.15.96:0/2941527013 >> 172.21.15.96:0/2941527013 conn(0x7f6c6806bae0 msgr2=0x7f6c68071b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:09:01.816 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.814+0000 7f6c70400640 1 -- 172.21.15.96:0/2941527013 shutdown_connections 2024-05-28T06:09:01.816 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:01.814+0000 7f6c70400640 1 -- 172.21.15.96:0/2941527013 wait complete. 2024-05-28T06:09:01.834 INFO:teuthology.orchestra.run.smithi096.stdout:{"pg_ready":true,"pg_map":{"version":189,"stamp":"2024-05-28T06:09:01.592571+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":288092,"kb_used_data":5352,"kb_used_omap":12,"kb_used_meta":282611,"kb_avail":754686628,"statfs":{"total":773094113280,"available":772799107072,"internally_reserved":0,"allocated":5480448,"data_stored":2917556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":289394224},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.584583"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.605956+0000","last_change":"2024-05-28T06:08:55.066371+0000","last_active":"2024-05-28T06:08:56.605956+0000","last_peered":"2024-05-28T06:08:56.605956+0000","last_clean":"2024-05-28T06:08:56.605956+0000","last_became_active":"2024-05-28T06:08:55.066047+0000","last_became_peered":"2024-05-28T06:08:55.066047+0000","last_unstale":"2024-05-28T06:08:56.605956+0000","last_undegraded":"2024-05-28T06:08:56.605956+0000","last_fullsized":"2024-05-28T06:08:56.605956+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:02:39.055643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.604302+0000","last_change":"2024-05-28T06:08:55.076582+0000","last_active":"2024-05-28T06:08:56.604302+0000","last_peered":"2024-05-28T06:08:56.604302+0000","last_clean":"2024-05-28T06:08:56.604302+0000","last_became_active":"2024-05-28T06:08:55.076064+0000","last_became_peered":"2024-05-28T06:08:55.076064+0000","last_unstale":"2024-05-28T06:08:56.604302+0000","last_undegraded":"2024-05-28T06:08:56.604302+0000","last_fullsized":"2024-05-28T06:08:56.604302+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:05:25.905405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.038037+0000","last_change":"2024-05-28T06:08:55.064457+0000","last_active":"2024-05-28T06:08:56.038037+0000","last_peered":"2024-05-28T06:08:56.038037+0000","last_clean":"2024-05-28T06:08:56.038037+0000","last_became_active":"2024-05-28T06:08:55.064156+0000","last_became_peered":"2024-05-28T06:08:55.064156+0000","last_unstale":"2024-05-28T06:08:56.038037+0000","last_undegraded":"2024-05-28T06:08:56.038037+0000","last_fullsized":"2024-05-28T06:08:56.038037+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:47:30.448166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.604178+0000","last_change":"2024-05-28T06:08:55.076802+0000","last_active":"2024-05-28T06:08:56.604178+0000","last_peered":"2024-05-28T06:08:56.604178+0000","last_clean":"2024-05-28T06:08:56.604178+0000","last_became_active":"2024-05-28T06:08:55.076486+0000","last_became_peered":"2024-05-28T06:08:55.076486+0000","last_unstale":"2024-05-28T06:08:56.604178+0000","last_undegraded":"2024-05-28T06:08:56.604178+0000","last_fullsized":"2024-05-28T06:08:56.604178+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:45:57.789170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.604023+0000","last_change":"2024-05-28T06:08:55.077749+0000","last_active":"2024-05-28T06:08:56.604023+0000","last_peered":"2024-05-28T06:08:56.604023+0000","last_clean":"2024-05-28T06:08:56.604023+0000","last_became_active":"2024-05-28T06:08:55.077000+0000","last_became_peered":"2024-05-28T06:08:55.077000+0000","last_unstale":"2024-05-28T06:08:56.604023+0000","last_undegraded":"2024-05-28T06:08:56.604023+0000","last_fullsized":"2024-05-28T06:08:56.604023+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:42:35.586471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.605482+0000","last_change":"2024-05-28T06:08:55.060394+0000","last_active":"2024-05-28T06:08:56.605482+0000","last_peered":"2024-05-28T06:08:56.605482+0000","last_clean":"2024-05-28T06:08:56.605482+0000","last_became_active":"2024-05-28T06:08:55.060049+0000","last_became_peered":"2024-05-28T06:08:55.060049+0000","last_unstale":"2024-05-28T06:08:56.605482+0000","last_undegraded":"2024-05-28T06:08:56.605482+0000","last_fullsized":"2024-05-28T06:08:56.605482+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:21:27.696390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.604828+0000","last_change":"2024-05-28T06:08:55.062884+0000","last_active":"2024-05-28T06:08:56.604828+0000","last_peered":"2024-05-28T06:08:56.604828+0000","last_clean":"2024-05-28T06:08:56.604828+0000","last_became_active":"2024-05-28T06:08:55.062316+0000","last_became_peered":"2024-05-28T06:08:55.062316+0000","last_unstale":"2024-05-28T06:08:56.604828+0000","last_undegraded":"2024-05-28T06:08:56.604828+0000","last_fullsized":"2024-05-28T06:08:56.604828+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:28:06.436338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.602146+0000","last_change":"2024-05-28T06:08:55.075905+0000","last_active":"2024-05-28T06:08:56.602146+0000","last_peered":"2024-05-28T06:08:56.602146+0000","last_clean":"2024-05-28T06:08:56.602146+0000","last_became_active":"2024-05-28T06:08:55.075585+0000","last_became_peered":"2024-05-28T06:08:55.075585+0000","last_unstale":"2024-05-28T06:08:56.602146+0000","last_undegraded":"2024-05-28T06:08:56.602146+0000","last_fullsized":"2024-05-28T06:08:56.602146+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:52:35.504075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.604619+0000","last_change":"2024-05-28T06:08:55.049828+0000","last_active":"2024-05-28T06:08:56.604619+0000","last_peered":"2024-05-28T06:08:56.604619+0000","last_clean":"2024-05-28T06:08:56.604619+0000","last_became_active":"2024-05-28T06:08:55.049541+0000","last_became_peered":"2024-05-28T06:08:55.049541+0000","last_unstale":"2024-05-28T06:08:56.604619+0000","last_undegraded":"2024-05-28T06:08:56.604619+0000","last_fullsized":"2024-05-28T06:08:56.604619+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:53:22.458046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.603342+0000","last_change":"2024-05-28T06:08:55.076779+0000","last_active":"2024-05-28T06:08:56.603342+0000","last_peered":"2024-05-28T06:08:56.603342+0000","last_clean":"2024-05-28T06:08:56.603342+0000","last_became_active":"2024-05-28T06:08:55.076505+0000","last_became_peered":"2024-05-28T06:08:55.076505+0000","last_unstale":"2024-05-28T06:08:56.603342+0000","last_undegraded":"2024-05-28T06:08:56.603342+0000","last_fullsized":"2024-05-28T06:08:56.603342+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:11:15.289677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.602597+0000","last_change":"2024-05-28T06:08:55.051639+0000","last_active":"2024-05-28T06:08:56.602597+0000","last_peered":"2024-05-28T06:08:56.602597+0000","last_clean":"2024-05-28T06:08:56.602597+0000","last_became_active":"2024-05-28T06:08:55.049516+0000","last_became_peered":"2024-05-28T06:08:55.049516+0000","last_unstale":"2024-05-28T06:08:56.602597+0000","last_undegraded":"2024-05-28T06:08:56.602597+0000","last_fullsized":"2024-05-28T06:08:56.602597+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:20:28.145327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.604976+0000","last_change":"2024-05-28T06:08:55.069606+0000","last_active":"2024-05-28T06:08:56.604976+0000","last_peered":"2024-05-28T06:08:56.604976+0000","last_clean":"2024-05-28T06:08:56.604976+0000","last_became_active":"2024-05-28T06:08:55.065520+0000","last_became_peered":"2024-05-28T06:08:55.065520+0000","last_unstale":"2024-05-28T06:08:56.604976+0000","last_undegraded":"2024-05-28T06:08:56.604976+0000","last_fullsized":"2024-05-28T06:08:56.604976+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:51:56.422690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.033328+0000","last_change":"2024-05-28T06:08:55.046096+0000","last_active":"2024-05-28T06:08:56.033328+0000","last_peered":"2024-05-28T06:08:56.033328+0000","last_clean":"2024-05-28T06:08:56.033328+0000","last_became_active":"2024-05-28T06:08:55.045807+0000","last_became_peered":"2024-05-28T06:08:55.045807+0000","last_unstale":"2024-05-28T06:08:56.033328+0000","last_undegraded":"2024-05-28T06:08:56.033328+0000","last_fullsized":"2024-05-28T06:08:56.033328+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:57:07.374940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.602759+0000","last_change":"2024-05-28T06:08:55.057317+0000","last_active":"2024-05-28T06:08:56.602759+0000","last_peered":"2024-05-28T06:08:56.602759+0000","last_clean":"2024-05-28T06:08:56.602759+0000","last_became_active":"2024-05-28T06:08:55.057002+0000","last_became_peered":"2024-05-28T06:08:55.057002+0000","last_unstale":"2024-05-28T06:08:56.602759+0000","last_undegraded":"2024-05-28T06:08:56.602759+0000","last_fullsized":"2024-05-28T06:08:56.602759+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:19:18.886834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.605336+0000","last_change":"2024-05-28T06:08:55.053668+0000","last_active":"2024-05-28T06:08:56.605336+0000","last_peered":"2024-05-28T06:08:56.605336+0000","last_clean":"2024-05-28T06:08:56.605336+0000","last_became_active":"2024-05-28T06:08:55.053380+0000","last_became_peered":"2024-05-28T06:08:55.053380+0000","last_unstale":"2024-05-28T06:08:56.605336+0000","last_undegraded":"2024-05-28T06:08:56.605336+0000","last_fullsized":"2024-05-28T06:08:56.605336+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:55:01.463179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.602983+0000","last_change":"2024-05-28T06:08:55.058030+0000","last_active":"2024-05-28T06:08:56.602983+0000","last_peered":"2024-05-28T06:08:56.602983+0000","last_clean":"2024-05-28T06:08:56.602983+0000","last_became_active":"2024-05-28T06:08:55.057745+0000","last_became_peered":"2024-05-28T06:08:55.057745+0000","last_unstale":"2024-05-28T06:08:56.602983+0000","last_undegraded":"2024-05-28T06:08:56.602983+0000","last_fullsized":"2024-05-28T06:08:56.602983+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:11:07.379445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.038663+0000","last_change":"2024-05-28T06:08:55.068139+0000","last_active":"2024-05-28T06:08:56.038663+0000","last_peered":"2024-05-28T06:08:56.038663+0000","last_clean":"2024-05-28T06:08:56.038663+0000","last_became_active":"2024-05-28T06:08:55.067770+0000","last_became_peered":"2024-05-28T06:08:55.067770+0000","last_unstale":"2024-05-28T06:08:56.038663+0000","last_undegraded":"2024-05-28T06:08:56.038663+0000","last_fullsized":"2024-05-28T06:08:56.038663+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:11:15.289677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.038242+0000","last_change":"2024-05-28T06:08:55.076477+0000","last_active":"2024-05-28T06:08:56.038242+0000","last_peered":"2024-05-28T06:08:56.038242+0000","last_clean":"2024-05-28T06:08:56.038242+0000","last_became_active":"2024-05-28T06:08:55.076071+0000","last_became_peered":"2024-05-28T06:08:55.076071+0000","last_unstale":"2024-05-28T06:08:56.038242+0000","last_undegraded":"2024-05-28T06:08:56.038242+0000","last_fullsized":"2024-05-28T06:08:56.038242+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:20:28.145327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.603955+0000","last_change":"2024-05-28T06:08:55.052221+0000","last_active":"2024-05-28T06:08:56.603955+0000","last_peered":"2024-05-28T06:08:56.603955+0000","last_clean":"2024-05-28T06:08:56.603955+0000","last_became_active":"2024-05-28T06:08:55.051913+0000","last_became_peered":"2024-05-28T06:08:55.051913+0000","last_unstale":"2024-05-28T06:08:56.603955+0000","last_undegraded":"2024-05-28T06:08:56.603955+0000","last_fullsized":"2024-05-28T06:08:56.603955+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:54:12.186812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.033789+0000","last_change":"2024-05-28T06:08:55.047906+0000","last_active":"2024-05-28T06:08:56.033789+0000","last_peered":"2024-05-28T06:08:56.033789+0000","last_clean":"2024-05-28T06:08:56.033789+0000","last_became_active":"2024-05-28T06:08:55.046685+0000","last_became_peered":"2024-05-28T06:08:55.046685+0000","last_unstale":"2024-05-28T06:08:56.033789+0000","last_undegraded":"2024-05-28T06:08:56.033789+0000","last_fullsized":"2024-05-28T06:08:56.033789+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:16:03.009973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.606351+0000","last_change":"2024-05-28T06:08:55.063895+0000","last_active":"2024-05-28T06:08:56.606351+0000","last_peered":"2024-05-28T06:08:56.606351+0000","last_clean":"2024-05-28T06:08:56.606351+0000","last_became_active":"2024-05-28T06:08:55.063571+0000","last_became_peered":"2024-05-28T06:08:55.063571+0000","last_unstale":"2024-05-28T06:08:56.606351+0000","last_undegraded":"2024-05-28T06:08:56.606351+0000","last_fullsized":"2024-05-28T06:08:56.606351+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:53:12.783877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.603160+0000","last_change":"2024-05-28T06:08:55.076056+0000","last_active":"2024-05-28T06:08:56.603160+0000","last_peered":"2024-05-28T06:08:56.603160+0000","last_clean":"2024-05-28T06:08:56.603160+0000","last_became_active":"2024-05-28T06:08:55.075548+0000","last_became_peered":"2024-05-28T06:08:55.075548+0000","last_unstale":"2024-05-28T06:08:56.603160+0000","last_undegraded":"2024-05-28T06:08:56.603160+0000","last_fullsized":"2024-05-28T06:08:56.603160+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:15:00.869480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.603581+0000","last_change":"2024-05-28T06:08:55.066981+0000","last_active":"2024-05-28T06:08:56.603581+0000","last_peered":"2024-05-28T06:08:56.603581+0000","last_clean":"2024-05-28T06:08:56.603581+0000","last_became_active":"2024-05-28T06:08:55.066660+0000","last_became_peered":"2024-05-28T06:08:55.066660+0000","last_unstale":"2024-05-28T06:08:56.603581+0000","last_undegraded":"2024-05-28T06:08:56.603581+0000","last_fullsized":"2024-05-28T06:08:56.603581+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:24:10.605107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.603426+0000","last_change":"2024-05-28T06:08:55.061728+0000","last_active":"2024-05-28T06:08:56.603426+0000","last_peered":"2024-05-28T06:08:56.603426+0000","last_clean":"2024-05-28T06:08:56.603426+0000","last_became_active":"2024-05-28T06:08:55.060468+0000","last_became_peered":"2024-05-28T06:08:55.060468+0000","last_unstale":"2024-05-28T06:08:56.603426+0000","last_undegraded":"2024-05-28T06:08:56.603426+0000","last_fullsized":"2024-05-28T06:08:56.603426+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:00:01.875672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.605528+0000","last_change":"2024-05-28T06:08:55.056744+0000","last_active":"2024-05-28T06:08:56.605528+0000","last_peered":"2024-05-28T06:08:56.605528+0000","last_clean":"2024-05-28T06:08:56.605528+0000","last_became_active":"2024-05-28T06:08:55.056453+0000","last_became_peered":"2024-05-28T06:08:55.056453+0000","last_unstale":"2024-05-28T06:08:56.605528+0000","last_undegraded":"2024-05-28T06:08:56.605528+0000","last_fullsized":"2024-05-28T06:08:56.605528+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:59:25.013392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.033711+0000","last_change":"2024-05-28T06:08:55.044404+0000","last_active":"2024-05-28T06:08:56.033711+0000","last_peered":"2024-05-28T06:08:56.033711+0000","last_clean":"2024-05-28T06:08:56.033711+0000","last_became_active":"2024-05-28T06:08:55.044132+0000","last_became_peered":"2024-05-28T06:08:55.044132+0000","last_unstale":"2024-05-28T06:08:56.033711+0000","last_undegraded":"2024-05-28T06:08:56.033711+0000","last_fullsized":"2024-05-28T06:08:56.033711+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:41:00.940014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.038455+0000","last_change":"2024-05-28T06:08:55.067112+0000","last_active":"2024-05-28T06:08:56.038455+0000","last_peered":"2024-05-28T06:08:56.038455+0000","last_clean":"2024-05-28T06:08:56.038455+0000","last_became_active":"2024-05-28T06:08:55.066818+0000","last_became_peered":"2024-05-28T06:08:55.066818+0000","last_unstale":"2024-05-28T06:08:56.038455+0000","last_undegraded":"2024-05-28T06:08:56.038455+0000","last_fullsized":"2024-05-28T06:08:56.038455+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:15:00.869480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.604973+0000","last_change":"2024-05-28T06:08:55.067044+0000","last_active":"2024-05-28T06:08:56.604973+0000","last_peered":"2024-05-28T06:08:56.604973+0000","last_clean":"2024-05-28T06:08:56.604973+0000","last_became_active":"2024-05-28T06:08:55.062614+0000","last_became_peered":"2024-05-28T06:08:55.062614+0000","last_unstale":"2024-05-28T06:08:56.604973+0000","last_undegraded":"2024-05-28T06:08:56.604973+0000","last_fullsized":"2024-05-28T06:08:56.604973+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:46:07.291928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.603899+0000","last_change":"2024-05-28T06:08:55.055182+0000","last_active":"2024-05-28T06:08:56.603899+0000","last_peered":"2024-05-28T06:08:56.603899+0000","last_clean":"2024-05-28T06:08:56.603899+0000","last_became_active":"2024-05-28T06:08:55.053384+0000","last_became_peered":"2024-05-28T06:08:55.053384+0000","last_unstale":"2024-05-28T06:08:56.603899+0000","last_undegraded":"2024-05-28T06:08:56.603899+0000","last_fullsized":"2024-05-28T06:08:56.603899+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:16:34.302370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.037405+0000","last_change":"2024-05-28T06:08:55.063836+0000","last_active":"2024-05-28T06:08:56.037405+0000","last_peered":"2024-05-28T06:08:56.037405+0000","last_clean":"2024-05-28T06:08:56.037405+0000","last_became_active":"2024-05-28T06:08:55.063404+0000","last_became_peered":"2024-05-28T06:08:55.063404+0000","last_unstale":"2024-05-28T06:08:56.037405+0000","last_undegraded":"2024-05-28T06:08:56.037405+0000","last_fullsized":"2024-05-28T06:08:56.037405+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:52:35.504075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.606010+0000","last_change":"2024-05-28T06:08:55.067468+0000","last_active":"2024-05-28T06:08:56.606010+0000","last_peered":"2024-05-28T06:08:56.606010+0000","last_clean":"2024-05-28T06:08:56.606010+0000","last_became_active":"2024-05-28T06:08:55.067187+0000","last_became_peered":"2024-05-28T06:08:55.067187+0000","last_unstale":"2024-05-28T06:08:56.606010+0000","last_undegraded":"2024-05-28T06:08:56.606010+0000","last_fullsized":"2024-05-28T06:08:56.606010+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:45:51.613393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.033553+0000","last_change":"2024-05-28T06:08:55.049528+0000","last_active":"2024-05-28T06:08:56.033553+0000","last_peered":"2024-05-28T06:08:56.033553+0000","last_clean":"2024-05-28T06:08:56.033553+0000","last_became_active":"2024-05-28T06:08:55.049228+0000","last_became_peered":"2024-05-28T06:08:55.049228+0000","last_unstale":"2024-05-28T06:08:56.033553+0000","last_undegraded":"2024-05-28T06:08:56.033553+0000","last_fullsized":"2024-05-28T06:08:56.033553+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:52:59.732955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.601983+0000","last_change":"2024-05-28T06:08:55.054821+0000","last_active":"2024-05-28T06:08:56.601983+0000","last_peered":"2024-05-28T06:08:56.601983+0000","last_clean":"2024-05-28T06:08:56.601983+0000","last_became_active":"2024-05-28T06:08:55.054506+0000","last_became_peered":"2024-05-28T06:08:55.054506+0000","last_unstale":"2024-05-28T06:08:56.601983+0000","last_undegraded":"2024-05-28T06:08:56.601983+0000","last_fullsized":"2024-05-28T06:08:56.601983+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:34:32.590326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"44'4","reported_seq":18,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.606385+0000","last_change":"2024-05-28T06:08:54.044108+0000","last_active":"2024-05-28T06:08:56.606385+0000","last_peered":"2024-05-28T06:08:56.606385+0000","last_clean":"2024-05-28T06:08:56.606385+0000","last_became_active":"2024-05-28T06:08:54.043637+0000","last_became_peered":"2024-05-28T06:08:54.043637+0000","last_unstale":"2024-05-28T06:08:56.606385+0000","last_undegraded":"2024-05-28T06:08:56.606385+0000","last_fullsized":"2024-05-28T06:08:56.606385+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:59:53.626690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.605924+0000","last_change":"2024-05-28T06:08:55.066359+0000","last_active":"2024-05-28T06:08:56.605924+0000","last_peered":"2024-05-28T06:08:56.605924+0000","last_clean":"2024-05-28T06:08:56.605924+0000","last_became_active":"2024-05-28T06:08:55.066019+0000","last_became_peered":"2024-05-28T06:08:55.066019+0000","last_unstale":"2024-05-28T06:08:56.605924+0000","last_undegraded":"2024-05-28T06:08:56.605924+0000","last_fullsized":"2024-05-28T06:08:56.605924+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:34:32.590326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"44'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.038760+0000","last_change":"2024-05-28T06:08:54.036739+0000","last_active":"2024-05-28T06:08:56.038760+0000","last_peered":"2024-05-28T06:08:56.038760+0000","last_clean":"2024-05-28T06:08:56.038760+0000","last_became_active":"2024-05-28T06:08:54.036446+0000","last_became_peered":"2024-05-28T06:08:54.036446+0000","last_unstale":"2024-05-28T06:08:56.038760+0000","last_undegraded":"2024-05-28T06:08:56.038760+0000","last_fullsized":"2024-05-28T06:08:56.038760+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:08:09.679468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.605330+0000","last_change":"2024-05-28T06:08:55.066245+0000","last_active":"2024-05-28T06:08:56.605330+0000","last_peered":"2024-05-28T06:08:56.605330+0000","last_clean":"2024-05-28T06:08:56.605330+0000","last_became_active":"2024-05-28T06:08:55.063953+0000","last_became_peered":"2024-05-28T06:08:55.063953+0000","last_unstale":"2024-05-28T06:08:56.605330+0000","last_undegraded":"2024-05-28T06:08:56.605330+0000","last_fullsized":"2024-05-28T06:08:56.605330+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:33:57.271643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.603441+0000","last_change":"2024-05-28T06:08:54.035269+0000","last_active":"2024-05-28T06:08:56.603441+0000","last_peered":"2024-05-28T06:08:56.603441+0000","last_clean":"2024-05-28T06:08:56.603441+0000","last_became_active":"2024-05-28T06:08:54.034916+0000","last_became_peered":"2024-05-28T06:08:54.034916+0000","last_unstale":"2024-05-28T06:08:56.603441+0000","last_undegraded":"2024-05-28T06:08:56.603441+0000","last_fullsized":"2024-05-28T06:08:56.603441+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:16:33.288780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.602784+0000","last_change":"2024-05-28T06:08:55.070511+0000","last_active":"2024-05-28T06:08:56.602784+0000","last_peered":"2024-05-28T06:08:56.602784+0000","last_clean":"2024-05-28T06:08:56.602784+0000","last_became_active":"2024-05-28T06:08:55.070201+0000","last_became_peered":"2024-05-28T06:08:55.070201+0000","last_unstale":"2024-05-28T06:08:56.602784+0000","last_undegraded":"2024-05-28T06:08:56.602784+0000","last_fullsized":"2024-05-28T06:08:56.602784+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:47:30.448166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.605662+0000","last_change":"2024-05-28T06:08:54.044956+0000","last_active":"2024-05-28T06:08:56.605662+0000","last_peered":"2024-05-28T06:08:56.605662+0000","last_clean":"2024-05-28T06:08:56.605662+0000","last_became_active":"2024-05-28T06:08:54.044646+0000","last_became_peered":"2024-05-28T06:08:54.044646+0000","last_unstale":"2024-05-28T06:08:56.605662+0000","last_undegraded":"2024-05-28T06:08:56.605662+0000","last_fullsized":"2024-05-28T06:08:56.605662+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:45:56.775580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.037409+0000","last_change":"2024-05-28T06:08:55.073311+0000","last_active":"2024-05-28T06:08:56.037409+0000","last_peered":"2024-05-28T06:08:56.037409+0000","last_clean":"2024-05-28T06:08:56.037409+0000","last_became_active":"2024-05-28T06:08:55.072761+0000","last_became_peered":"2024-05-28T06:08:55.072761+0000","last_unstale":"2024-05-28T06:08:56.037409+0000","last_undegraded":"2024-05-28T06:08:56.037409+0000","last_fullsized":"2024-05-28T06:08:56.037409+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:34:32.590326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"44'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.606218+0000","last_change":"2024-05-28T06:08:54.042495+0000","last_active":"2024-05-28T06:08:56.606218+0000","last_peered":"2024-05-28T06:08:56.606218+0000","last_clean":"2024-05-28T06:08:56.606218+0000","last_became_active":"2024-05-28T06:08:54.042143+0000","last_became_peered":"2024-05-28T06:08:54.042143+0000","last_unstale":"2024-05-28T06:08:56.606218+0000","last_undegraded":"2024-05-28T06:08:56.606218+0000","last_fullsized":"2024-05-28T06:08:56.606218+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:01:51.040147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.604208+0000","last_change":"2024-05-28T06:08:55.055134+0000","last_active":"2024-05-28T06:08:56.604208+0000","last_peered":"2024-05-28T06:08:56.604208+0000","last_clean":"2024-05-28T06:08:56.604208+0000","last_became_active":"2024-05-28T06:08:55.054881+0000","last_became_peered":"2024-05-28T06:08:55.054881+0000","last_unstale":"2024-05-28T06:08:56.604208+0000","last_undegraded":"2024-05-28T06:08:56.604208+0000","last_fullsized":"2024-05-28T06:08:56.604208+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:59:54.640280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"44'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.038029+0000","last_change":"2024-05-28T06:08:54.030180+0000","last_active":"2024-05-28T06:08:56.038029+0000","last_peered":"2024-05-28T06:08:56.038029+0000","last_clean":"2024-05-28T06:08:56.038029+0000","last_became_active":"2024-05-28T06:08:54.029892+0000","last_became_peered":"2024-05-28T06:08:54.029892+0000","last_unstale":"2024-05-28T06:08:56.038029+0000","last_undegraded":"2024-05-28T06:08:56.038029+0000","last_fullsized":"2024-05-28T06:08:56.038029+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:18:13.820152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.602565+0000","last_change":"2024-05-28T06:08:55.076181+0000","last_active":"2024-05-28T06:08:56.602565+0000","last_peered":"2024-05-28T06:08:56.602565+0000","last_clean":"2024-05-28T06:08:56.602565+0000","last_became_active":"2024-05-28T06:08:55.075570+0000","last_became_peered":"2024-05-28T06:08:55.075570+0000","last_unstale":"2024-05-28T06:08:56.602565+0000","last_undegraded":"2024-05-28T06:08:56.602565+0000","last_fullsized":"2024-05-28T06:08:56.602565+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:47:30.448166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.604320+0000","last_change":"2024-05-28T06:08:54.035697+0000","last_active":"2024-05-28T06:08:56.604320+0000","last_peered":"2024-05-28T06:08:56.604320+0000","last_clean":"2024-05-28T06:08:56.604320+0000","last_became_active":"2024-05-28T06:08:54.032931+0000","last_became_peered":"2024-05-28T06:08:54.032931+0000","last_unstale":"2024-05-28T06:08:56.604320+0000","last_undegraded":"2024-05-28T06:08:56.604320+0000","last_fullsized":"2024-05-28T06:08:56.604320+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:14:59.855890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.606117+0000","last_change":"2024-05-28T06:08:55.066914+0000","last_active":"2024-05-28T06:08:56.606117+0000","last_peered":"2024-05-28T06:08:56.606117+0000","last_clean":"2024-05-28T06:08:56.606117+0000","last_became_active":"2024-05-28T06:08:55.066242+0000","last_became_peered":"2024-05-28T06:08:55.066242+0000","last_unstale":"2024-05-28T06:08:56.606117+0000","last_undegraded":"2024-05-28T06:08:56.606117+0000","last_fullsized":"2024-05-28T06:08:56.606117+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:52:35.504075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.602798+0000","last_change":"2024-05-28T06:08:54.032388+0000","last_active":"2024-05-28T06:08:56.602798+0000","last_peered":"2024-05-28T06:08:56.602798+0000","last_clean":"2024-05-28T06:08:56.602798+0000","last_became_active":"2024-05-28T06:08:54.032095+0000","last_became_peered":"2024-05-28T06:08:54.032095+0000","last_unstale":"2024-05-28T06:08:56.602798+0000","last_undegraded":"2024-05-28T06:08:56.602798+0000","last_fullsized":"2024-05-28T06:08:56.602798+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:16:17.652829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.038435+0000","last_change":"2024-05-28T06:08:55.071079+0000","last_active":"2024-05-28T06:08:56.038435+0000","last_peered":"2024-05-28T06:08:56.038435+0000","last_clean":"2024-05-28T06:08:56.038435+0000","last_became_active":"2024-05-28T06:08:55.070804+0000","last_became_peered":"2024-05-28T06:08:55.070804+0000","last_unstale":"2024-05-28T06:08:56.038435+0000","last_undegraded":"2024-05-28T06:08:56.038435+0000","last_fullsized":"2024-05-28T06:08:56.038435+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:19:18.886834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.038778+0000","last_change":"2024-05-28T06:08:54.027262+0000","last_active":"2024-05-28T06:08:56.038778+0000","last_peered":"2024-05-28T06:08:56.038778+0000","last_clean":"2024-05-28T06:08:56.038778+0000","last_became_active":"2024-05-28T06:08:54.026066+0000","last_became_peered":"2024-05-28T06:08:54.026066+0000","last_unstale":"2024-05-28T06:08:56.038778+0000","last_undegraded":"2024-05-28T06:08:56.038778+0000","last_fullsized":"2024-05-28T06:08:56.038778+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:37:34.767912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.606324+0000","last_change":"2024-05-28T06:08:55.071950+0000","last_active":"2024-05-28T06:08:56.606324+0000","last_peered":"2024-05-28T06:08:56.606324+0000","last_clean":"2024-05-28T06:08:56.606324+0000","last_became_active":"2024-05-28T06:08:55.071671+0000","last_became_peered":"2024-05-28T06:08:55.071671+0000","last_unstale":"2024-05-28T06:08:56.606324+0000","last_undegraded":"2024-05-28T06:08:56.606324+0000","last_fullsized":"2024-05-28T06:08:56.606324+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:30:19.818261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.603530+0000","last_change":"2024-05-28T06:08:54.040877+0000","last_active":"2024-05-28T06:08:56.603530+0000","last_peered":"2024-05-28T06:08:56.603530+0000","last_clean":"2024-05-28T06:08:56.603530+0000","last_became_active":"2024-05-28T06:08:54.040615+0000","last_became_peered":"2024-05-28T06:08:54.040615+0000","last_unstale":"2024-05-28T06:08:56.603530+0000","last_undegraded":"2024-05-28T06:08:56.603530+0000","last_fullsized":"2024-05-28T06:08:56.603530+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:18:13.820152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.606380+0000","last_change":"2024-05-28T06:08:55.059781+0000","last_active":"2024-05-28T06:08:56.606380+0000","last_peered":"2024-05-28T06:08:56.606380+0000","last_clean":"2024-05-28T06:08:56.606380+0000","last_became_active":"2024-05-28T06:08:55.059497+0000","last_became_peered":"2024-05-28T06:08:55.059497+0000","last_unstale":"2024-05-28T06:08:56.606380+0000","last_undegraded":"2024-05-28T06:08:56.606380+0000","last_fullsized":"2024-05-28T06:08:56.606380+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:32:24.478463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"44'3","reported_seq":17,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.606779+0000","last_change":"2024-05-28T06:08:54.046805+0000","last_active":"2024-05-28T06:08:56.606779+0000","last_peered":"2024-05-28T06:08:56.606779+0000","last_clean":"2024-05-28T06:08:56.606779+0000","last_became_active":"2024-05-28T06:08:54.046336+0000","last_became_peered":"2024-05-28T06:08:54.046336+0000","last_unstale":"2024-05-28T06:08:56.606779+0000","last_undegraded":"2024-05-28T06:08:56.606779+0000","last_fullsized":"2024-05-28T06:08:56.606779+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:59:53.626690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.606518+0000","last_change":"2024-05-28T06:08:55.073615+0000","last_active":"2024-05-28T06:08:56.606518+0000","last_peered":"2024-05-28T06:08:56.606518+0000","last_clean":"2024-05-28T06:08:56.606518+0000","last_became_active":"2024-05-28T06:08:55.071441+0000","last_became_peered":"2024-05-28T06:08:55.071441+0000","last_unstale":"2024-05-28T06:08:56.606518+0000","last_undegraded":"2024-05-28T06:08:56.606518+0000","last_fullsized":"2024-05-28T06:08:56.606518+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:10:00.499041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"44'6","reported_seq":20,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.033710+0000","last_change":"2024-05-28T06:08:54.029622+0000","last_active":"2024-05-28T06:08:56.033710+0000","last_peered":"2024-05-28T06:08:56.033710+0000","last_clean":"2024-05-28T06:08:56.033710+0000","last_became_active":"2024-05-28T06:08:54.026140+0000","last_became_peered":"2024-05-28T06:08:54.026140+0000","last_unstale":"2024-05-28T06:08:56.033710+0000","last_undegraded":"2024-05-28T06:08:56.033710+0000","last_fullsized":"2024-05-28T06:08:56.033710+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:56:58.418726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.602994+0000","last_change":"2024-05-28T06:08:55.053056+0000","last_active":"2024-05-28T06:08:56.602994+0000","last_peered":"2024-05-28T06:08:56.602994+0000","last_clean":"2024-05-28T06:08:56.602994+0000","last_became_active":"2024-05-28T06:08:55.050989+0000","last_became_peered":"2024-05-28T06:08:55.050989+0000","last_unstale":"2024-05-28T06:08:56.602994+0000","last_undegraded":"2024-05-28T06:08:56.602994+0000","last_fullsized":"2024-05-28T06:08:56.602994+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:27:41.378357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.606515+0000","last_change":"2024-05-28T06:08:54.038320+0000","last_active":"2024-05-28T06:08:56.606515+0000","last_peered":"2024-05-28T06:08:56.606515+0000","last_clean":"2024-05-28T06:08:56.606515+0000","last_became_active":"2024-05-28T06:08:54.037824+0000","last_became_peered":"2024-05-28T06:08:54.037824+0000","last_unstale":"2024-05-28T06:08:56.606515+0000","last_undegraded":"2024-05-28T06:08:56.606515+0000","last_fullsized":"2024-05-28T06:08:56.606515+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:59:27.653504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.604164+0000","last_change":"2024-05-28T06:08:55.075823+0000","last_active":"2024-05-28T06:08:56.604164+0000","last_peered":"2024-05-28T06:08:56.604164+0000","last_clean":"2024-05-28T06:08:56.604164+0000","last_became_active":"2024-05-28T06:08:55.069901+0000","last_became_peered":"2024-05-28T06:08:55.069901+0000","last_unstale":"2024-05-28T06:08:56.604164+0000","last_undegraded":"2024-05-28T06:08:56.604164+0000","last_fullsized":"2024-05-28T06:08:56.604164+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:59:28.667094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.605527+0000","last_change":"2024-05-28T06:08:54.046264+0000","last_active":"2024-05-28T06:08:56.605527+0000","last_peered":"2024-05-28T06:08:56.605527+0000","last_clean":"2024-05-28T06:08:56.605527+0000","last_became_active":"2024-05-28T06:08:54.045957+0000","last_became_peered":"2024-05-28T06:08:54.045957+0000","last_unstale":"2024-05-28T06:08:56.605527+0000","last_undegraded":"2024-05-28T06:08:56.605527+0000","last_fullsized":"2024-05-28T06:08:56.605527+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:59:27.653504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.605332+0000","last_change":"2024-05-28T06:08:55.047136+0000","last_active":"2024-05-28T06:08:56.605332+0000","last_peered":"2024-05-28T06:08:56.605332+0000","last_clean":"2024-05-28T06:08:56.605332+0000","last_became_active":"2024-05-28T06:08:55.045959+0000","last_became_peered":"2024-05-28T06:08:55.045959+0000","last_unstale":"2024-05-28T06:08:56.605332+0000","last_undegraded":"2024-05-28T06:08:56.605332+0000","last_fullsized":"2024-05-28T06:08:56.605332+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:10:00.499041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.605624+0000","last_change":"2024-05-28T06:08:54.038131+0000","last_active":"2024-05-28T06:08:56.605624+0000","last_peered":"2024-05-28T06:08:56.605624+0000","last_clean":"2024-05-28T06:08:56.605624+0000","last_became_active":"2024-05-28T06:08:54.037578+0000","last_became_peered":"2024-05-28T06:08:54.037578+0000","last_unstale":"2024-05-28T06:08:56.605624+0000","last_undegraded":"2024-05-28T06:08:56.605624+0000","last_fullsized":"2024-05-28T06:08:56.605624+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:06:36.665935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.604966+0000","last_change":"2024-05-28T06:08:55.072863+0000","last_active":"2024-05-28T06:08:56.604966+0000","last_peered":"2024-05-28T06:08:56.604966+0000","last_clean":"2024-05-28T06:08:56.604966+0000","last_became_active":"2024-05-28T06:08:55.072388+0000","last_became_peered":"2024-05-28T06:08:55.072388+0000","last_unstale":"2024-05-28T06:08:56.604966+0000","last_undegraded":"2024-05-28T06:08:56.604966+0000","last_fullsized":"2024-05-28T06:08:56.604966+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:51:13.118226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"44'8","reported_seq":22,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.603088+0000","last_change":"2024-05-28T06:08:54.034498+0000","last_active":"2024-05-28T06:08:56.603088+0000","last_peered":"2024-05-28T06:08:56.603088+0000","last_clean":"2024-05-28T06:08:56.603088+0000","last_became_active":"2024-05-28T06:08:54.034152+0000","last_became_peered":"2024-05-28T06:08:54.034152+0000","last_unstale":"2024-05-28T06:08:56.603088+0000","last_undegraded":"2024-05-28T06:08:56.603088+0000","last_fullsized":"2024-05-28T06:08:56.603088+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:51:33.930123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.034026+0000","last_change":"2024-05-28T06:08:55.066857+0000","last_active":"2024-05-28T06:08:56.034026+0000","last_peered":"2024-05-28T06:08:56.034026+0000","last_clean":"2024-05-28T06:08:56.034026+0000","last_became_active":"2024-05-28T06:08:55.066592+0000","last_became_peered":"2024-05-28T06:08:55.066592+0000","last_unstale":"2024-05-28T06:08:56.034026+0000","last_undegraded":"2024-05-28T06:08:56.034026+0000","last_fullsized":"2024-05-28T06:08:56.034026+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:51:34.943713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"44'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.034196+0000","last_change":"2024-05-28T06:08:54.035386+0000","last_active":"2024-05-28T06:08:56.034196+0000","last_peered":"2024-05-28T06:08:56.034196+0000","last_clean":"2024-05-28T06:08:56.034196+0000","last_became_active":"2024-05-28T06:08:54.033511+0000","last_became_peered":"2024-05-28T06:08:54.033511+0000","last_unstale":"2024-05-28T06:08:56.034196+0000","last_undegraded":"2024-05-28T06:08:56.034196+0000","last_fullsized":"2024-05-28T06:08:56.034196+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:31:59.983717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.604783+0000","last_change":"2024-05-28T06:08:55.052937+0000","last_active":"2024-05-28T06:08:56.604783+0000","last_peered":"2024-05-28T06:08:56.604783+0000","last_clean":"2024-05-28T06:08:56.604783+0000","last_became_active":"2024-05-28T06:08:55.052639+0000","last_became_peered":"2024-05-28T06:08:55.052639+0000","last_unstale":"2024-05-28T06:08:56.604783+0000","last_undegraded":"2024-05-28T06:08:56.604783+0000","last_fullsized":"2024-05-28T06:08:56.604783+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:42:35.586471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.038514+0000","last_change":"2024-05-28T06:08:54.035942+0000","last_active":"2024-05-28T06:08:56.038514+0000","last_peered":"2024-05-28T06:08:56.038514+0000","last_clean":"2024-05-28T06:08:56.038514+0000","last_became_active":"2024-05-28T06:08:54.033436+0000","last_became_peered":"2024-05-28T06:08:54.033436+0000","last_unstale":"2024-05-28T06:08:56.038514+0000","last_undegraded":"2024-05-28T06:08:56.038514+0000","last_fullsized":"2024-05-28T06:08:56.038514+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:50:52.128343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.604363+0000","last_change":"2024-05-28T06:08:55.057536+0000","last_active":"2024-05-28T06:08:56.604363+0000","last_peered":"2024-05-28T06:08:56.604363+0000","last_clean":"2024-05-28T06:08:56.604363+0000","last_became_active":"2024-05-28T06:08:55.057206+0000","last_became_peered":"2024-05-28T06:08:55.057206+0000","last_unstale":"2024-05-28T06:08:56.604363+0000","last_undegraded":"2024-05-28T06:08:56.604363+0000","last_fullsized":"2024-05-28T06:08:56.604363+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:29:28.504290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"44'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.605231+0000","last_change":"2024-05-28T06:08:54.036250+0000","last_active":"2024-05-28T06:08:56.605231+0000","last_peered":"2024-05-28T06:08:56.605231+0000","last_clean":"2024-05-28T06:08:56.605231+0000","last_became_active":"2024-05-28T06:08:54.034306+0000","last_became_peered":"2024-05-28T06:08:54.034306+0000","last_unstale":"2024-05-28T06:08:56.605231+0000","last_undegraded":"2024-05-28T06:08:56.605231+0000","last_fullsized":"2024-05-28T06:08:56.605231+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:11:14.276087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.603354+0000","last_change":"2024-05-28T06:08:55.053434+0000","last_active":"2024-05-28T06:08:56.603354+0000","last_peered":"2024-05-28T06:08:56.603354+0000","last_clean":"2024-05-28T06:08:56.603354+0000","last_became_active":"2024-05-28T06:08:55.052617+0000","last_became_peered":"2024-05-28T06:08:55.052617+0000","last_unstale":"2024-05-28T06:08:56.603354+0000","last_undegraded":"2024-05-28T06:08:56.603354+0000","last_fullsized":"2024-05-28T06:08:56.603354+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:01:16.405059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.606407+0000","last_change":"2024-05-28T06:08:54.043914+0000","last_active":"2024-05-28T06:08:56.606407+0000","last_peered":"2024-05-28T06:08:56.606407+0000","last_clean":"2024-05-28T06:08:56.606407+0000","last_became_active":"2024-05-28T06:08:54.043662+0000","last_became_peered":"2024-05-28T06:08:54.043662+0000","last_unstale":"2024-05-28T06:08:56.606407+0000","last_undegraded":"2024-05-28T06:08:56.606407+0000","last_fullsized":"2024-05-28T06:08:56.606407+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:53:21.444456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.605566+0000","last_change":"2024-05-28T06:08:55.051810+0000","last_active":"2024-05-28T06:08:56.605566+0000","last_peered":"2024-05-28T06:08:56.605566+0000","last_clean":"2024-05-28T06:08:56.605566+0000","last_became_active":"2024-05-28T06:08:55.050715+0000","last_became_peered":"2024-05-28T06:08:55.050715+0000","last_unstale":"2024-05-28T06:08:56.605566+0000","last_undegraded":"2024-05-28T06:08:56.605566+0000","last_fullsized":"2024-05-28T06:08:56.605566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:25:17.106626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.606671+0000","last_change":"2024-05-28T06:08:54.042952+0000","last_active":"2024-05-28T06:08:56.606671+0000","last_peered":"2024-05-28T06:08:56.606671+0000","last_clean":"2024-05-28T06:08:56.606671+0000","last_became_active":"2024-05-28T06:08:54.042713+0000","last_became_peered":"2024-05-28T06:08:54.042713+0000","last_unstale":"2024-05-28T06:08:56.606671+0000","last_undegraded":"2024-05-28T06:08:56.606671+0000","last_fullsized":"2024-05-28T06:08:56.606671+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:45:56.775580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.602347+0000","last_change":"2024-05-28T06:08:55.066822+0000","last_active":"2024-05-28T06:08:56.602347+0000","last_peered":"2024-05-28T06:08:56.602347+0000","last_clean":"2024-05-28T06:08:56.602347+0000","last_became_active":"2024-05-28T06:08:55.066494+0000","last_became_peered":"2024-05-28T06:08:55.066494+0000","last_unstale":"2024-05-28T06:08:56.602347+0000","last_undegraded":"2024-05-28T06:08:56.602347+0000","last_fullsized":"2024-05-28T06:08:56.602347+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:52:35.504075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.603292+0000","last_change":"2024-05-28T06:08:54.039984+0000","last_active":"2024-05-28T06:08:56.603292+0000","last_peered":"2024-05-28T06:08:56.603292+0000","last_clean":"2024-05-28T06:08:56.603292+0000","last_became_active":"2024-05-28T06:08:54.038849+0000","last_became_peered":"2024-05-28T06:08:54.038849+0000","last_unstale":"2024-05-28T06:08:56.603292+0000","last_undegraded":"2024-05-28T06:08:56.603292+0000","last_fullsized":"2024-05-28T06:08:56.603292+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:19:17.873244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.033956+0000","last_change":"2024-05-28T06:08:55.057248+0000","last_active":"2024-05-28T06:08:56.033956+0000","last_peered":"2024-05-28T06:08:56.033956+0000","last_clean":"2024-05-28T06:08:56.033956+0000","last_became_active":"2024-05-28T06:08:55.056958+0000","last_became_peered":"2024-05-28T06:08:55.056958+0000","last_unstale":"2024-05-28T06:08:56.033956+0000","last_undegraded":"2024-05-28T06:08:56.033956+0000","last_fullsized":"2024-05-28T06:08:56.033956+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:16:18.666419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.603963+0000","last_change":"2024-05-28T06:08:54.043822+0000","last_active":"2024-05-28T06:08:56.603963+0000","last_peered":"2024-05-28T06:08:56.603963+0000","last_clean":"2024-05-28T06:08:56.603963+0000","last_became_active":"2024-05-28T06:08:54.043402+0000","last_became_peered":"2024-05-28T06:08:54.043402+0000","last_unstale":"2024-05-28T06:08:56.603963+0000","last_undegraded":"2024-05-28T06:08:56.603963+0000","last_fullsized":"2024-05-28T06:08:56.603963+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:07:41.209833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.604001+0000","last_change":"2024-05-28T06:08:55.059492+0000","last_active":"2024-05-28T06:08:56.604001+0000","last_peered":"2024-05-28T06:08:56.604001+0000","last_clean":"2024-05-28T06:08:56.604001+0000","last_became_active":"2024-05-28T06:08:55.057314+0000","last_became_peered":"2024-05-28T06:08:55.057314+0000","last_unstale":"2024-05-28T06:08:56.604001+0000","last_undegraded":"2024-05-28T06:08:56.604001+0000","last_fullsized":"2024-05-28T06:08:56.604001+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:27:47.697809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.602543+0000","last_change":"2024-05-28T06:08:54.046229+0000","last_active":"2024-05-28T06:08:56.602543+0000","last_peered":"2024-05-28T06:08:56.602543+0000","last_clean":"2024-05-28T06:08:56.602543+0000","last_became_active":"2024-05-28T06:08:54.045868+0000","last_became_peered":"2024-05-28T06:08:54.045868+0000","last_unstale":"2024-05-28T06:08:56.602543+0000","last_undegraded":"2024-05-28T06:08:56.602543+0000","last_fullsized":"2024-05-28T06:08:56.602543+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:11:14.276087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.606295+0000","last_change":"2024-05-28T06:08:55.059577+0000","last_active":"2024-05-28T06:08:56.606295+0000","last_peered":"2024-05-28T06:08:56.606295+0000","last_clean":"2024-05-28T06:08:56.606295+0000","last_became_active":"2024-05-28T06:08:55.058682+0000","last_became_peered":"2024-05-28T06:08:55.058682+0000","last_unstale":"2024-05-28T06:08:56.606295+0000","last_undegraded":"2024-05-28T06:08:56.606295+0000","last_fullsized":"2024-05-28T06:08:56.606295+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:47:30.448166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"45'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.606231+0000","last_change":"2024-05-28T06:08:54.034283+0000","last_active":"2024-05-28T06:08:56.606231+0000","last_peered":"2024-05-28T06:08:56.606231+0000","last_clean":"2024-05-28T06:08:56.606231+0000","last_became_active":"2024-05-28T06:08:54.033985+0000","last_became_peered":"2024-05-28T06:08:54.033985+0000","last_unstale":"2024-05-28T06:08:56.606231+0000","last_undegraded":"2024-05-28T06:08:56.606231+0000","last_fullsized":"2024-05-28T06:08:56.606231+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:27:46.684219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.604602+0000","last_change":"2024-05-28T06:08:55.065596+0000","last_active":"2024-05-28T06:08:56.604602+0000","last_peered":"2024-05-28T06:08:56.604602+0000","last_clean":"2024-05-28T06:08:56.604602+0000","last_became_active":"2024-05-28T06:08:55.065284+0000","last_became_peered":"2024-05-28T06:08:55.065284+0000","last_unstale":"2024-05-28T06:08:56.604602+0000","last_undegraded":"2024-05-28T06:08:56.604602+0000","last_fullsized":"2024-05-28T06:08:56.604602+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:06:37.679525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.606888+0000","last_change":"2024-05-28T06:08:54.047896+0000","last_active":"2024-05-28T06:08:56.606888+0000","last_peered":"2024-05-28T06:08:56.606888+0000","last_clean":"2024-05-28T06:08:56.606888+0000","last_became_active":"2024-05-28T06:08:54.047535+0000","last_became_peered":"2024-05-28T06:08:54.047535+0000","last_unstale":"2024-05-28T06:08:56.606888+0000","last_undegraded":"2024-05-28T06:08:56.606888+0000","last_fullsized":"2024-05-28T06:08:56.606888+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:27:46.684219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.606644+0000","last_change":"2024-05-28T06:08:55.064486+0000","last_active":"2024-05-28T06:08:56.606644+0000","last_peered":"2024-05-28T06:08:56.606644+0000","last_clean":"2024-05-28T06:08:56.606644+0000","last_became_active":"2024-05-28T06:08:55.062008+0000","last_became_peered":"2024-05-28T06:08:55.062008+0000","last_unstale":"2024-05-28T06:08:56.606644+0000","last_undegraded":"2024-05-28T06:08:56.606644+0000","last_fullsized":"2024-05-28T06:08:56.606644+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:55:01.463179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.601727+0000","last_change":"2024-05-28T06:08:54.037383+0000","last_active":"2024-05-28T06:08:56.601727+0000","last_peered":"2024-05-28T06:08:56.601727+0000","last_clean":"2024-05-28T06:08:56.601727+0000","last_became_active":"2024-05-28T06:08:54.036598+0000","last_became_peered":"2024-05-28T06:08:54.036598+0000","last_unstale":"2024-05-28T06:08:56.601727+0000","last_undegraded":"2024-05-28T06:08:56.601727+0000","last_fullsized":"2024-05-28T06:08:56.601727+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:00:00.862082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.605922+0000","last_change":"2024-05-28T06:08:55.051492+0000","last_active":"2024-05-28T06:08:56.605922+0000","last_peered":"2024-05-28T06:08:56.605922+0000","last_clean":"2024-05-28T06:08:56.605922+0000","last_became_active":"2024-05-28T06:08:55.051094+0000","last_became_peered":"2024-05-28T06:08:55.051094+0000","last_unstale":"2024-05-28T06:08:56.605922+0000","last_undegraded":"2024-05-28T06:08:56.605922+0000","last_fullsized":"2024-05-28T06:08:56.605922+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:02:41.768007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.605220+0000","last_change":"2024-05-28T06:08:54.046483+0000","last_active":"2024-05-28T06:08:56.605220+0000","last_peered":"2024-05-28T06:08:56.605220+0000","last_clean":"2024-05-28T06:08:56.605220+0000","last_became_active":"2024-05-28T06:08:54.045605+0000","last_became_peered":"2024-05-28T06:08:54.045605+0000","last_unstale":"2024-05-28T06:08:56.605220+0000","last_undegraded":"2024-05-28T06:08:56.605220+0000","last_fullsized":"2024-05-28T06:08:56.605220+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:01:51.040147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.605884+0000","last_change":"2024-05-28T06:08:55.050859+0000","last_active":"2024-05-28T06:08:56.605884+0000","last_peered":"2024-05-28T06:08:56.605884+0000","last_clean":"2024-05-28T06:08:56.605884+0000","last_became_active":"2024-05-28T06:08:55.050572+0000","last_became_peered":"2024-05-28T06:08:55.050572+0000","last_unstale":"2024-05-28T06:08:56.605884+0000","last_undegraded":"2024-05-28T06:08:56.605884+0000","last_fullsized":"2024-05-28T06:08:56.605884+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:06:45.885680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.037820+0000","last_change":"2024-05-28T06:08:54.037676+0000","last_active":"2024-05-28T06:08:56.037820+0000","last_peered":"2024-05-28T06:08:56.037820+0000","last_clean":"2024-05-28T06:08:56.037820+0000","last_became_active":"2024-05-28T06:08:54.037301+0000","last_became_peered":"2024-05-28T06:08:54.037301+0000","last_unstale":"2024-05-28T06:08:56.037820+0000","last_undegraded":"2024-05-28T06:08:56.037820+0000","last_fullsized":"2024-05-28T06:08:56.037820+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:37:24.350191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.602195+0000","last_change":"2024-05-28T06:08:55.062795+0000","last_active":"2024-05-28T06:08:56.602195+0000","last_peered":"2024-05-28T06:08:56.602195+0000","last_clean":"2024-05-28T06:08:56.602195+0000","last_became_active":"2024-05-28T06:08:55.059485+0000","last_became_peered":"2024-05-28T06:08:55.059485+0000","last_unstale":"2024-05-28T06:08:56.602195+0000","last_undegraded":"2024-05-28T06:08:56.602195+0000","last_fullsized":"2024-05-28T06:08:56.602195+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:53:22.458046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.601577+0000","last_change":"2024-05-28T06:08:54.044188+0000","last_active":"2024-05-28T06:08:56.601577+0000","last_peered":"2024-05-28T06:08:56.601577+0000","last_clean":"2024-05-28T06:08:56.601577+0000","last_became_active":"2024-05-28T06:08:54.043655+0000","last_became_peered":"2024-05-28T06:08:54.043655+0000","last_unstale":"2024-05-28T06:08:56.601577+0000","last_undegraded":"2024-05-28T06:08:56.601577+0000","last_fullsized":"2024-05-28T06:08:56.601577+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:24:09.591517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.037385+0000","last_change":"2024-05-28T06:08:55.070064+0000","last_active":"2024-05-28T06:08:56.037385+0000","last_peered":"2024-05-28T06:08:56.037385+0000","last_clean":"2024-05-28T06:08:56.037385+0000","last_became_active":"2024-05-28T06:08:55.069804+0000","last_became_peered":"2024-05-28T06:08:55.069804+0000","last_unstale":"2024-05-28T06:08:56.037385+0000","last_undegraded":"2024-05-28T06:08:56.037385+0000","last_fullsized":"2024-05-28T06:08:56.037385+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:59:25.013392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.603247+0000","last_change":"2024-05-28T06:08:54.038376+0000","last_active":"2024-05-28T06:08:56.603247+0000","last_peered":"2024-05-28T06:08:56.603247+0000","last_clean":"2024-05-28T06:08:56.603247+0000","last_became_active":"2024-05-28T06:08:54.037952+0000","last_became_peered":"2024-05-28T06:08:54.037952+0000","last_unstale":"2024-05-28T06:08:56.603247+0000","last_undegraded":"2024-05-28T06:08:56.603247+0000","last_fullsized":"2024-05-28T06:08:56.603247+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:01:37.817834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.603480+0000","last_change":"2024-05-28T06:08:55.071657+0000","last_active":"2024-05-28T06:08:56.603480+0000","last_peered":"2024-05-28T06:08:56.603480+0000","last_clean":"2024-05-28T06:08:56.603480+0000","last_became_active":"2024-05-28T06:08:55.071416+0000","last_became_peered":"2024-05-28T06:08:55.071416+0000","last_unstale":"2024-05-28T06:08:56.603480+0000","last_undegraded":"2024-05-28T06:08:56.603480+0000","last_fullsized":"2024-05-28T06:08:56.603480+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:01:52.053737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.604754+0000","last_change":"2024-05-28T06:08:54.036597+0000","last_active":"2024-05-28T06:08:56.604754+0000","last_peered":"2024-05-28T06:08:56.604754+0000","last_clean":"2024-05-28T06:08:56.604754+0000","last_became_active":"2024-05-28T06:08:54.034833+0000","last_became_peered":"2024-05-28T06:08:54.034833+0000","last_unstale":"2024-05-28T06:08:56.604754+0000","last_undegraded":"2024-05-28T06:08:56.604754+0000","last_fullsized":"2024-05-28T06:08:56.604754+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:20:27.131737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":217,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.602825+0000","last_change":"2024-05-28T06:04:01.039129+0000","last_active":"2024-05-28T06:08:56.602825+0000","last_peered":"2024-05-28T06:08:56.602825+0000","last_clean":"2024-05-28T06:08:56.602825+0000","last_became_active":"2024-05-28T06:04:01.033906+0000","last_became_peered":"2024-05-28T06:04:01.033906+0000","last_unstale":"2024-05-28T06:08:56.602825+0000","last_undegraded":"2024-05-28T06:08:56.602825+0000","last_fullsized":"2024-05-28T06:08:56.602825+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:03:59.864237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:03:59.864237+0000","last_clean_scrub_stamp":"2024-05-28T06:03:59.864237+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:17:36.440245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574911,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36440,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335400,"statfs":{"total":96636764160,"available":96599449600,"internally_reserved":0,"allocated":1126400,"data_stored":806958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574913,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35864,"kb_used_data":524,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335976,"statfs":{"total":96636764160,"available":96600039424,"internally_reserved":0,"allocated":536576,"data_stored":217172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574911,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35864,"kb_used_data":520,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335976,"statfs":{"total":96636764160,"available":96600039424,"internally_reserved":0,"allocated":532480,"data_stored":216570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574911,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35872,"kb_used_data":528,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335968,"statfs":{"total":96636764160,"available":96600031232,"internally_reserved":0,"allocated":540672,"data_stored":218112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574913,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35868,"kb_used_data":528,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335972,"statfs":{"total":96636764160,"available":96600035328,"internally_reserved":0,"allocated":540672,"data_stored":217138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574912,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35876,"kb_used_data":532,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335964,"statfs":{"total":96636764160,"available":96600027136,"internally_reserved":0,"allocated":544768,"data_stored":218134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574912,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35868,"kb_used_data":520,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335972,"statfs":{"total":96636764160,"available":96600035328,"internally_reserved":0,"allocated":532480,"data_stored":216570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574911,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36440,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335400,"statfs":{"total":96636764160,"available":96599449600,"internally_reserved":0,"allocated":1126400,"data_stored":806902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-28T06:09:04.838 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-28T06:09:05.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.046+0000 7efda9400640 1 Processor -- start 2024-05-28T06:09:05.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.046+0000 7efda9400640 1 -- start start 2024-05-28T06:09:05.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.047+0000 7efda9400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7efda4073310 0x7efda40b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:05.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.047+0000 7efda9400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efda406dc20 0x7efda415f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:05.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.047+0000 7efda9400640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efda40afd60 con 0x7efda415b3a0 2024-05-28T06:09:05.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.047+0000 7efda9400640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7efda406f980 con 0x7efda4073310 2024-05-28T06:09:05.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.047+0000 7efda9400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7efda40ae0c0 con 0x7efda406dc20 2024-05-28T06:09:05.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.048+0000 7efda3e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7efda415b3a0 0x7efda4072c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:51190/0 (socket says 172.21.15.96:51190) 2024-05-28T06:09:05.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.048+0000 7efda3e00640 1 -- 172.21.15.96:0/3831974123 learned_addr learned my addr 172.21.15.96:0/3831974123 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:09:05.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.048+0000 7efda3400640 1 --2- 172.21.15.96:0/3831974123 >> v2:172.21.15.123:3300/0 conn(0x7efda4073310 0x7efda40b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:05.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.048+0000 7efda2a00640 1 -- 172.21.15.96:0/3831974123 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1288234382 0 0) 0x7efda40afd60 con 0x7efda415b3a0 2024-05-28T06:09:05.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.048+0000 7efda8a00640 1 --2- 172.21.15.96:0/3831974123 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efda406dc20 0x7efda415f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:05.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.048+0000 7efda2a00640 1 -- 172.21.15.96:0/3831974123 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd88005730 con 0x7efda415b3a0 2024-05-28T06:09:05.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.048+0000 7efda2a00640 1 -- 172.21.15.96:0/3831974123 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4197862780 0 0) 0x7efd88005730 con 0x7efda415b3a0 2024-05-28T06:09:05.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.049+0000 7efda2a00640 1 -- 172.21.15.96:0/3831974123 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efda406dc20 msgr2=0x7efda415f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:09:05.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.049+0000 7efda2a00640 1 --2- 172.21.15.96:0/3831974123 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efda406dc20 0x7efda415f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:05.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.049+0000 7efda2a00640 1 -- 172.21.15.96:0/3831974123 >> v2:172.21.15.123:3300/0 conn(0x7efda4073310 msgr2=0x7efda40b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:09:05.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.049+0000 7efda2a00640 1 --2- 172.21.15.96:0/3831974123 >> v2:172.21.15.123:3300/0 conn(0x7efda4073310 0x7efda40b32c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:05.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.049+0000 7efda2a00640 1 -- 172.21.15.96:0/3831974123 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efda40af7a0 con 0x7efda415b3a0 2024-05-28T06:09:05.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.049+0000 7efda2a00640 1 -- 172.21.15.96:0/3831974123 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7efd94001670 con 0x7efda415b3a0 2024-05-28T06:09:05.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.049+0000 7efda2a00640 1 -- 172.21.15.96:0/3831974123 >> v1:172.21.15.96:6789/0 conn(0x7efda415b3a0 legacy=0x7efda4072c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:09:05.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.049+0000 7efda2a00640 1 --2- 172.21.15.96:0/3831974123 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efd88005bc0 0x7efd88005fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:05.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.049+0000 7efda2a00640 1 --2- 172.21.15.96:0/3831974123 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd880065b0 0x7efd88006a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:05.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.049+0000 7efda2a00640 1 --2- 172.21.15.96:0/3831974123 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efd88007250 0x7efd8800dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:05.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.049+0000 7efda2a00640 1 -- 172.21.15.96:0/3831974123 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7efda406f980 con 0x7efd88007250 2024-05-28T06:09:05.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.049+0000 7efda2a00640 1 -- 172.21.15.96:0/3831974123 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7efda40ae0c0 con 0x7efd880065b0 2024-05-28T06:09:05.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.049+0000 7efda2a00640 1 -- 172.21.15.96:0/3831974123 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7efd8800e1f0 con 0x7efd88005bc0 2024-05-28T06:09:05.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.049+0000 7efda8a00640 1 --2- 172.21.15.96:0/3831974123 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efda406dc20 0x7efda415f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-28T06:09:05.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.049+0000 7efda8a00640 1 --2- 172.21.15.96:0/3831974123 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efd88007250 0x7efd8800dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:05.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.049+0000 7efda3e00640 1 --2- 172.21.15.96:0/3831974123 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efd88005bc0 0x7efd88005fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:05.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.049+0000 7efda3400640 1 --2- 172.21.15.96:0/3831974123 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd880065b0 0x7efd88006a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:05.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.050+0000 7efda8a00640 1 -- 172.21.15.96:0/3831974123 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efd88005bc0 msgr2=0x7efd88005fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:09:05.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.050+0000 7efda8a00640 1 --2- 172.21.15.96:0/3831974123 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efd88005bc0 0x7efd88005fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:05.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.050+0000 7efda8a00640 1 -- 172.21.15.96:0/3831974123 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd880065b0 msgr2=0x7efd88006a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:09:05.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.050+0000 7efda8a00640 1 --2- 172.21.15.96:0/3831974123 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd880065b0 0x7efd88006a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:05.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.050+0000 7efda8a00640 1 -- 172.21.15.96:0/3831974123 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd88005730 con 0x7efd88007250 2024-05-28T06:09:05.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.050+0000 7efda3400640 1 --2- 172.21.15.96:0/3831974123 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd880065b0 0x7efd88006a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:09:05.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.051+0000 7efda8a00640 1 --2- 172.21.15.96:0/3831974123 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efd88007250 0x7efd8800dbd0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7efda4071b70 tx=0x7efd88007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7c86ca82f8c6bea server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:09:05.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.051+0000 7efda2a00640 1 -- 172.21.15.96:0/3831974123 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd9c009b50 con 0x7efd88007250 2024-05-28T06:09:05.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.051+0000 7efda2a00640 1 -- 172.21.15.96:0/3831974123 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd9c003990 con 0x7efd88007250 2024-05-28T06:09:05.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.051+0000 7efda2a00640 1 -- 172.21.15.96:0/3831974123 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd9c003d60 con 0x7efd88007250 2024-05-28T06:09:05.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.051+0000 7efda9400640 1 -- 172.21.15.96:0/3831974123 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efd88007250 msgr2=0x7efd8800dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:09:05.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.051+0000 7efda9400640 1 --2- 172.21.15.96:0/3831974123 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efd88007250 0x7efd8800dbd0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7efda4071b70 tx=0x7efd88007a40 comp rx=0 tx=0).stop 2024-05-28T06:09:05.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.051+0000 7efda8a00640 1 -- 172.21.15.96:0/3831974123 reap_dead start 2024-05-28T06:09:05.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.051+0000 7efda9400640 1 -- 172.21.15.96:0/3831974123 shutdown_connections 2024-05-28T06:09:05.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.051+0000 7efda9400640 1 -- 172.21.15.96:0/3831974123 >> 172.21.15.96:0/3831974123 conn(0x7efda40b1fc0 msgr2=0x7efda40b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:09:05.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.052+0000 7efda9400640 1 -- 172.21.15.96:0/3831974123 shutdown_connections 2024-05-28T06:09:05.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.052+0000 7efda9400640 1 -- 172.21.15.96:0/3831974123 wait complete. 2024-05-28T06:09:05.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.053+0000 7efda9400640 1 Processor -- start 2024-05-28T06:09:05.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.053+0000 7efda9400640 1 -- start start 2024-05-28T06:09:05.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.054+0000 7efda9400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd88007250 0x7efda415c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:05.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.054+0000 7efda9400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efda41576a0 0x7efda4149150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:05.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.054+0000 7efda9400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efda4157b20 0x7efda4149690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:05.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.054+0000 7efda9400640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7efda415f9a0 con 0x7efda41576a0 2024-05-28T06:09:05.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.054+0000 7efda9400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7efda415b780 con 0x7efd88007250 2024-05-28T06:09:05.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.054+0000 7efda9400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7efda415fb20 con 0x7efda4157b20 2024-05-28T06:09:05.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.055+0000 7efda3400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efda41576a0 0x7efda4149150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:05.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.055+0000 7efda3400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efda41576a0 0x7efda4149150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47566/0 (socket says 172.21.15.96:47566) 2024-05-28T06:09:05.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.055+0000 7efda3400640 1 -- 172.21.15.96:0/3833647058 learned_addr learned my addr 172.21.15.96:0/3833647058 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:09:05.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.055+0000 7efda3e00640 1 --2- 172.21.15.96:0/3833647058 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd88007250 0x7efda415c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:05.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.055+0000 7efda8a00640 1 --2- 172.21.15.96:0/3833647058 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efda4157b20 0x7efda4149690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:05.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.055+0000 7efda3400640 1 -- 172.21.15.96:0/3833647058 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efda4157b20 msgr2=0x7efda4149690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:09:05.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.055+0000 7efda3400640 1 --2- 172.21.15.96:0/3833647058 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efda4157b20 0x7efda4149690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:05.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.055+0000 7efda3400640 1 -- 172.21.15.96:0/3833647058 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd88007250 msgr2=0x7efda415c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:09:05.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.055+0000 7efda3400640 1 --2- 172.21.15.96:0/3833647058 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd88007250 0x7efda415c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:05.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.055+0000 7efda3400640 1 -- 172.21.15.96:0/3833647058 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efda415ce90 con 0x7efda41576a0 2024-05-28T06:09:05.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.055+0000 7efda3e00640 1 --2- 172.21.15.96:0/3833647058 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd88007250 0x7efda415c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:09:05.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.055+0000 7efda3400640 1 --2- 172.21.15.96:0/3833647058 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efda41576a0 0x7efda4149150 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7efd900109d0 tx=0x7efd90010e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:09:05.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.056+0000 7efd9be00640 1 -- 172.21.15.96:0/3833647058 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd9000f040 con 0x7efda41576a0 2024-05-28T06:09:05.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.056+0000 7efd9be00640 1 -- 172.21.15.96:0/3833647058 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd9000ec20 con 0x7efda41576a0 2024-05-28T06:09:05.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.056+0000 7efd9be00640 1 -- 172.21.15.96:0/3833647058 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd9001c400 con 0x7efda41576a0 2024-05-28T06:09:05.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.056+0000 7efda9400640 1 -- 172.21.15.96:0/3833647058 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efda415d130 con 0x7efda41576a0 2024-05-28T06:09:05.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.056+0000 7efda9400640 1 -- 172.21.15.96:0/3833647058 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efda415d6f0 con 0x7efda41576a0 2024-05-28T06:09:05.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.057+0000 7efda9400640 1 -- 172.21.15.96:0/3833647058 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efda40689e0 con 0x7efda41576a0 2024-05-28T06:09:05.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.058+0000 7efd9be00640 1 -- 172.21.15.96:0/3833647058 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 9) ==== 91295+0+0 (secure 0 0 0) 0x7efd9001c5a0 con 0x7efda41576a0 2024-05-28T06:09:05.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.059+0000 7efd9be00640 1 --2- 172.21.15.96:0/3833647058 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7efd8406cbf0 0x7efd8406f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:05.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.059+0000 7efd9be00640 1 -- 172.21.15.96:0/3833647058 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5737+0+0 (secure 0 0 0) 0x7efd9008c710 con 0x7efda41576a0 2024-05-28T06:09:05.061 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.060+0000 7efda3e00640 1 --2- 172.21.15.96:0/3833647058 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7efd8406cbf0 0x7efd8406f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:05.062 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.061+0000 7efda3e00640 1 --2- 172.21.15.96:0/3833647058 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7efd8406cbf0 0x7efd8406f0b0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7efd94005990 tx=0x7efd94000f90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:09:05.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.063+0000 7efd9be00640 1 -- 172.21.15.96:0/3833647058 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7efd90018030 con 0x7efda41576a0 2024-05-28T06:09:05.259 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.257+0000 7efda9400640 1 -- 172.21.15.96:0/3833647058 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7efda40b2e90 con 0x7efd8406cbf0 2024-05-28T06:09:05.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.275+0000 7efd9be00640 1 -- 172.21.15.96:0/3833647058 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251960 (secure 0 0 0) 0x7efda40b2e90 con 0x7efd8406cbf0 2024-05-28T06:09:05.280 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:09:05.281 INFO:teuthology.orchestra.run.smithi096.stderr:dumped all 2024-05-28T06:09:05.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.280+0000 7efda9400640 1 -- 172.21.15.96:0/3833647058 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7efd8406cbf0 msgr2=0x7efd8406f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:09:05.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.280+0000 7efda9400640 1 --2- 172.21.15.96:0/3833647058 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7efd8406cbf0 0x7efd8406f0b0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7efd94005990 tx=0x7efd94000f90 comp rx=0 tx=0).stop 2024-05-28T06:09:05.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.280+0000 7efda9400640 1 -- 172.21.15.96:0/3833647058 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efda41576a0 msgr2=0x7efda4149150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:09:05.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.280+0000 7efda9400640 1 --2- 172.21.15.96:0/3833647058 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efda41576a0 0x7efda4149150 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7efd900109d0 tx=0x7efd90010e90 comp rx=0 tx=0).stop 2024-05-28T06:09:05.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.280+0000 7efda9400640 1 -- 172.21.15.96:0/3833647058 shutdown_connections 2024-05-28T06:09:05.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.280+0000 7efda9400640 1 --2- 172.21.15.96:0/3833647058 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7efd8406cbf0 0x7efd8406f0b0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:05.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.280+0000 7efda9400640 1 --2- 172.21.15.96:0/3833647058 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efda4157b20 0x7efda4149690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:05.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.280+0000 7efda9400640 1 --2- 172.21.15.96:0/3833647058 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efda41576a0 0x7efda4149150 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:05.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.280+0000 7efda9400640 1 --2- 172.21.15.96:0/3833647058 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd88007250 0x7efda415c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:05.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.280+0000 7efda9400640 1 -- 172.21.15.96:0/3833647058 >> 172.21.15.96:0/3833647058 conn(0x7efda40b1fc0 msgr2=0x7efda40b37e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:09:05.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.281+0000 7efda9400640 1 -- 172.21.15.96:0/3833647058 shutdown_connections 2024-05-28T06:09:05.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.281+0000 7efda9400640 1 -- 172.21.15.96:0/3833647058 wait complete. 2024-05-28T06:09:05.302 INFO:teuthology.orchestra.run.smithi096.stdout:{"pg_ready":true,"pg_map":{"version":190,"stamp":"2024-05-28T06:09:03.593647+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":288092,"kb_used_data":5352,"kb_used_omap":12,"kb_used_meta":282611,"kb_avail":754686628,"statfs":{"total":773094113280,"available":772799107072,"internally_reserved":0,"allocated":5480448,"data_stored":2917556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":289394224},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.564511"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.605956+0000","last_change":"2024-05-28T06:08:55.066371+0000","last_active":"2024-05-28T06:08:56.605956+0000","last_peered":"2024-05-28T06:08:56.605956+0000","last_clean":"2024-05-28T06:08:56.605956+0000","last_became_active":"2024-05-28T06:08:55.066047+0000","last_became_peered":"2024-05-28T06:08:55.066047+0000","last_unstale":"2024-05-28T06:08:56.605956+0000","last_undegraded":"2024-05-28T06:08:56.605956+0000","last_fullsized":"2024-05-28T06:08:56.605956+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:02:39.055643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.604302+0000","last_change":"2024-05-28T06:08:55.076582+0000","last_active":"2024-05-28T06:08:56.604302+0000","last_peered":"2024-05-28T06:08:56.604302+0000","last_clean":"2024-05-28T06:08:56.604302+0000","last_became_active":"2024-05-28T06:08:55.076064+0000","last_became_peered":"2024-05-28T06:08:55.076064+0000","last_unstale":"2024-05-28T06:08:56.604302+0000","last_undegraded":"2024-05-28T06:08:56.604302+0000","last_fullsized":"2024-05-28T06:08:56.604302+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:05:25.905405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.038037+0000","last_change":"2024-05-28T06:08:55.064457+0000","last_active":"2024-05-28T06:08:56.038037+0000","last_peered":"2024-05-28T06:08:56.038037+0000","last_clean":"2024-05-28T06:08:56.038037+0000","last_became_active":"2024-05-28T06:08:55.064156+0000","last_became_peered":"2024-05-28T06:08:55.064156+0000","last_unstale":"2024-05-28T06:08:56.038037+0000","last_undegraded":"2024-05-28T06:08:56.038037+0000","last_fullsized":"2024-05-28T06:08:56.038037+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:47:30.448166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.604178+0000","last_change":"2024-05-28T06:08:55.076802+0000","last_active":"2024-05-28T06:08:56.604178+0000","last_peered":"2024-05-28T06:08:56.604178+0000","last_clean":"2024-05-28T06:08:56.604178+0000","last_became_active":"2024-05-28T06:08:55.076486+0000","last_became_peered":"2024-05-28T06:08:55.076486+0000","last_unstale":"2024-05-28T06:08:56.604178+0000","last_undegraded":"2024-05-28T06:08:56.604178+0000","last_fullsized":"2024-05-28T06:08:56.604178+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:45:57.789170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.604023+0000","last_change":"2024-05-28T06:08:55.077749+0000","last_active":"2024-05-28T06:08:56.604023+0000","last_peered":"2024-05-28T06:08:56.604023+0000","last_clean":"2024-05-28T06:08:56.604023+0000","last_became_active":"2024-05-28T06:08:55.077000+0000","last_became_peered":"2024-05-28T06:08:55.077000+0000","last_unstale":"2024-05-28T06:08:56.604023+0000","last_undegraded":"2024-05-28T06:08:56.604023+0000","last_fullsized":"2024-05-28T06:08:56.604023+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:42:35.586471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.605482+0000","last_change":"2024-05-28T06:08:55.060394+0000","last_active":"2024-05-28T06:08:56.605482+0000","last_peered":"2024-05-28T06:08:56.605482+0000","last_clean":"2024-05-28T06:08:56.605482+0000","last_became_active":"2024-05-28T06:08:55.060049+0000","last_became_peered":"2024-05-28T06:08:55.060049+0000","last_unstale":"2024-05-28T06:08:56.605482+0000","last_undegraded":"2024-05-28T06:08:56.605482+0000","last_fullsized":"2024-05-28T06:08:56.605482+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:21:27.696390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.604828+0000","last_change":"2024-05-28T06:08:55.062884+0000","last_active":"2024-05-28T06:08:56.604828+0000","last_peered":"2024-05-28T06:08:56.604828+0000","last_clean":"2024-05-28T06:08:56.604828+0000","last_became_active":"2024-05-28T06:08:55.062316+0000","last_became_peered":"2024-05-28T06:08:55.062316+0000","last_unstale":"2024-05-28T06:08:56.604828+0000","last_undegraded":"2024-05-28T06:08:56.604828+0000","last_fullsized":"2024-05-28T06:08:56.604828+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:28:06.436338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.602146+0000","last_change":"2024-05-28T06:08:55.075905+0000","last_active":"2024-05-28T06:08:56.602146+0000","last_peered":"2024-05-28T06:08:56.602146+0000","last_clean":"2024-05-28T06:08:56.602146+0000","last_became_active":"2024-05-28T06:08:55.075585+0000","last_became_peered":"2024-05-28T06:08:55.075585+0000","last_unstale":"2024-05-28T06:08:56.602146+0000","last_undegraded":"2024-05-28T06:08:56.602146+0000","last_fullsized":"2024-05-28T06:08:56.602146+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:52:35.504075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.604619+0000","last_change":"2024-05-28T06:08:55.049828+0000","last_active":"2024-05-28T06:08:56.604619+0000","last_peered":"2024-05-28T06:08:56.604619+0000","last_clean":"2024-05-28T06:08:56.604619+0000","last_became_active":"2024-05-28T06:08:55.049541+0000","last_became_peered":"2024-05-28T06:08:55.049541+0000","last_unstale":"2024-05-28T06:08:56.604619+0000","last_undegraded":"2024-05-28T06:08:56.604619+0000","last_fullsized":"2024-05-28T06:08:56.604619+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:53:22.458046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.603342+0000","last_change":"2024-05-28T06:08:55.076779+0000","last_active":"2024-05-28T06:08:56.603342+0000","last_peered":"2024-05-28T06:08:56.603342+0000","last_clean":"2024-05-28T06:08:56.603342+0000","last_became_active":"2024-05-28T06:08:55.076505+0000","last_became_peered":"2024-05-28T06:08:55.076505+0000","last_unstale":"2024-05-28T06:08:56.603342+0000","last_undegraded":"2024-05-28T06:08:56.603342+0000","last_fullsized":"2024-05-28T06:08:56.603342+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:11:15.289677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.602597+0000","last_change":"2024-05-28T06:08:55.051639+0000","last_active":"2024-05-28T06:08:56.602597+0000","last_peered":"2024-05-28T06:08:56.602597+0000","last_clean":"2024-05-28T06:08:56.602597+0000","last_became_active":"2024-05-28T06:08:55.049516+0000","last_became_peered":"2024-05-28T06:08:55.049516+0000","last_unstale":"2024-05-28T06:08:56.602597+0000","last_undegraded":"2024-05-28T06:08:56.602597+0000","last_fullsized":"2024-05-28T06:08:56.602597+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:20:28.145327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.604976+0000","last_change":"2024-05-28T06:08:55.069606+0000","last_active":"2024-05-28T06:08:56.604976+0000","last_peered":"2024-05-28T06:08:56.604976+0000","last_clean":"2024-05-28T06:08:56.604976+0000","last_became_active":"2024-05-28T06:08:55.065520+0000","last_became_peered":"2024-05-28T06:08:55.065520+0000","last_unstale":"2024-05-28T06:08:56.604976+0000","last_undegraded":"2024-05-28T06:08:56.604976+0000","last_fullsized":"2024-05-28T06:08:56.604976+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:51:56.422690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.033328+0000","last_change":"2024-05-28T06:08:55.046096+0000","last_active":"2024-05-28T06:08:56.033328+0000","last_peered":"2024-05-28T06:08:56.033328+0000","last_clean":"2024-05-28T06:08:56.033328+0000","last_became_active":"2024-05-28T06:08:55.045807+0000","last_became_peered":"2024-05-28T06:08:55.045807+0000","last_unstale":"2024-05-28T06:08:56.033328+0000","last_undegraded":"2024-05-28T06:08:56.033328+0000","last_fullsized":"2024-05-28T06:08:56.033328+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:57:07.374940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.602759+0000","last_change":"2024-05-28T06:08:55.057317+0000","last_active":"2024-05-28T06:08:56.602759+0000","last_peered":"2024-05-28T06:08:56.602759+0000","last_clean":"2024-05-28T06:08:56.602759+0000","last_became_active":"2024-05-28T06:08:55.057002+0000","last_became_peered":"2024-05-28T06:08:55.057002+0000","last_unstale":"2024-05-28T06:08:56.602759+0000","last_undegraded":"2024-05-28T06:08:56.602759+0000","last_fullsized":"2024-05-28T06:08:56.602759+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:19:18.886834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.605336+0000","last_change":"2024-05-28T06:08:55.053668+0000","last_active":"2024-05-28T06:08:56.605336+0000","last_peered":"2024-05-28T06:08:56.605336+0000","last_clean":"2024-05-28T06:08:56.605336+0000","last_became_active":"2024-05-28T06:08:55.053380+0000","last_became_peered":"2024-05-28T06:08:55.053380+0000","last_unstale":"2024-05-28T06:08:56.605336+0000","last_undegraded":"2024-05-28T06:08:56.605336+0000","last_fullsized":"2024-05-28T06:08:56.605336+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:55:01.463179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.602983+0000","last_change":"2024-05-28T06:08:55.058030+0000","last_active":"2024-05-28T06:08:56.602983+0000","last_peered":"2024-05-28T06:08:56.602983+0000","last_clean":"2024-05-28T06:08:56.602983+0000","last_became_active":"2024-05-28T06:08:55.057745+0000","last_became_peered":"2024-05-28T06:08:55.057745+0000","last_unstale":"2024-05-28T06:08:56.602983+0000","last_undegraded":"2024-05-28T06:08:56.602983+0000","last_fullsized":"2024-05-28T06:08:56.602983+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:11:07.379445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.038663+0000","last_change":"2024-05-28T06:08:55.068139+0000","last_active":"2024-05-28T06:08:56.038663+0000","last_peered":"2024-05-28T06:08:56.038663+0000","last_clean":"2024-05-28T06:08:56.038663+0000","last_became_active":"2024-05-28T06:08:55.067770+0000","last_became_peered":"2024-05-28T06:08:55.067770+0000","last_unstale":"2024-05-28T06:08:56.038663+0000","last_undegraded":"2024-05-28T06:08:56.038663+0000","last_fullsized":"2024-05-28T06:08:56.038663+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:11:15.289677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.038242+0000","last_change":"2024-05-28T06:08:55.076477+0000","last_active":"2024-05-28T06:08:56.038242+0000","last_peered":"2024-05-28T06:08:56.038242+0000","last_clean":"2024-05-28T06:08:56.038242+0000","last_became_active":"2024-05-28T06:08:55.076071+0000","last_became_peered":"2024-05-28T06:08:55.076071+0000","last_unstale":"2024-05-28T06:08:56.038242+0000","last_undegraded":"2024-05-28T06:08:56.038242+0000","last_fullsized":"2024-05-28T06:08:56.038242+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:20:28.145327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.603955+0000","last_change":"2024-05-28T06:08:55.052221+0000","last_active":"2024-05-28T06:08:56.603955+0000","last_peered":"2024-05-28T06:08:56.603955+0000","last_clean":"2024-05-28T06:08:56.603955+0000","last_became_active":"2024-05-28T06:08:55.051913+0000","last_became_peered":"2024-05-28T06:08:55.051913+0000","last_unstale":"2024-05-28T06:08:56.603955+0000","last_undegraded":"2024-05-28T06:08:56.603955+0000","last_fullsized":"2024-05-28T06:08:56.603955+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:54:12.186812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.033789+0000","last_change":"2024-05-28T06:08:55.047906+0000","last_active":"2024-05-28T06:08:56.033789+0000","last_peered":"2024-05-28T06:08:56.033789+0000","last_clean":"2024-05-28T06:08:56.033789+0000","last_became_active":"2024-05-28T06:08:55.046685+0000","last_became_peered":"2024-05-28T06:08:55.046685+0000","last_unstale":"2024-05-28T06:08:56.033789+0000","last_undegraded":"2024-05-28T06:08:56.033789+0000","last_fullsized":"2024-05-28T06:08:56.033789+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:16:03.009973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.606351+0000","last_change":"2024-05-28T06:08:55.063895+0000","last_active":"2024-05-28T06:08:56.606351+0000","last_peered":"2024-05-28T06:08:56.606351+0000","last_clean":"2024-05-28T06:08:56.606351+0000","last_became_active":"2024-05-28T06:08:55.063571+0000","last_became_peered":"2024-05-28T06:08:55.063571+0000","last_unstale":"2024-05-28T06:08:56.606351+0000","last_undegraded":"2024-05-28T06:08:56.606351+0000","last_fullsized":"2024-05-28T06:08:56.606351+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:53:12.783877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.603160+0000","last_change":"2024-05-28T06:08:55.076056+0000","last_active":"2024-05-28T06:08:56.603160+0000","last_peered":"2024-05-28T06:08:56.603160+0000","last_clean":"2024-05-28T06:08:56.603160+0000","last_became_active":"2024-05-28T06:08:55.075548+0000","last_became_peered":"2024-05-28T06:08:55.075548+0000","last_unstale":"2024-05-28T06:08:56.603160+0000","last_undegraded":"2024-05-28T06:08:56.603160+0000","last_fullsized":"2024-05-28T06:08:56.603160+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:15:00.869480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.603581+0000","last_change":"2024-05-28T06:08:55.066981+0000","last_active":"2024-05-28T06:08:56.603581+0000","last_peered":"2024-05-28T06:08:56.603581+0000","last_clean":"2024-05-28T06:08:56.603581+0000","last_became_active":"2024-05-28T06:08:55.066660+0000","last_became_peered":"2024-05-28T06:08:55.066660+0000","last_unstale":"2024-05-28T06:08:56.603581+0000","last_undegraded":"2024-05-28T06:08:56.603581+0000","last_fullsized":"2024-05-28T06:08:56.603581+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:24:10.605107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.603426+0000","last_change":"2024-05-28T06:08:55.061728+0000","last_active":"2024-05-28T06:08:56.603426+0000","last_peered":"2024-05-28T06:08:56.603426+0000","last_clean":"2024-05-28T06:08:56.603426+0000","last_became_active":"2024-05-28T06:08:55.060468+0000","last_became_peered":"2024-05-28T06:08:55.060468+0000","last_unstale":"2024-05-28T06:08:56.603426+0000","last_undegraded":"2024-05-28T06:08:56.603426+0000","last_fullsized":"2024-05-28T06:08:56.603426+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:00:01.875672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.605528+0000","last_change":"2024-05-28T06:08:55.056744+0000","last_active":"2024-05-28T06:08:56.605528+0000","last_peered":"2024-05-28T06:08:56.605528+0000","last_clean":"2024-05-28T06:08:56.605528+0000","last_became_active":"2024-05-28T06:08:55.056453+0000","last_became_peered":"2024-05-28T06:08:55.056453+0000","last_unstale":"2024-05-28T06:08:56.605528+0000","last_undegraded":"2024-05-28T06:08:56.605528+0000","last_fullsized":"2024-05-28T06:08:56.605528+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:59:25.013392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.033711+0000","last_change":"2024-05-28T06:08:55.044404+0000","last_active":"2024-05-28T06:08:56.033711+0000","last_peered":"2024-05-28T06:08:56.033711+0000","last_clean":"2024-05-28T06:08:56.033711+0000","last_became_active":"2024-05-28T06:08:55.044132+0000","last_became_peered":"2024-05-28T06:08:55.044132+0000","last_unstale":"2024-05-28T06:08:56.033711+0000","last_undegraded":"2024-05-28T06:08:56.033711+0000","last_fullsized":"2024-05-28T06:08:56.033711+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:41:00.940014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.038455+0000","last_change":"2024-05-28T06:08:55.067112+0000","last_active":"2024-05-28T06:08:56.038455+0000","last_peered":"2024-05-28T06:08:56.038455+0000","last_clean":"2024-05-28T06:08:56.038455+0000","last_became_active":"2024-05-28T06:08:55.066818+0000","last_became_peered":"2024-05-28T06:08:55.066818+0000","last_unstale":"2024-05-28T06:08:56.038455+0000","last_undegraded":"2024-05-28T06:08:56.038455+0000","last_fullsized":"2024-05-28T06:08:56.038455+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:15:00.869480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.604973+0000","last_change":"2024-05-28T06:08:55.067044+0000","last_active":"2024-05-28T06:08:56.604973+0000","last_peered":"2024-05-28T06:08:56.604973+0000","last_clean":"2024-05-28T06:08:56.604973+0000","last_became_active":"2024-05-28T06:08:55.062614+0000","last_became_peered":"2024-05-28T06:08:55.062614+0000","last_unstale":"2024-05-28T06:08:56.604973+0000","last_undegraded":"2024-05-28T06:08:56.604973+0000","last_fullsized":"2024-05-28T06:08:56.604973+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:46:07.291928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.603899+0000","last_change":"2024-05-28T06:08:55.055182+0000","last_active":"2024-05-28T06:08:56.603899+0000","last_peered":"2024-05-28T06:08:56.603899+0000","last_clean":"2024-05-28T06:08:56.603899+0000","last_became_active":"2024-05-28T06:08:55.053384+0000","last_became_peered":"2024-05-28T06:08:55.053384+0000","last_unstale":"2024-05-28T06:08:56.603899+0000","last_undegraded":"2024-05-28T06:08:56.603899+0000","last_fullsized":"2024-05-28T06:08:56.603899+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:16:34.302370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.037405+0000","last_change":"2024-05-28T06:08:55.063836+0000","last_active":"2024-05-28T06:08:56.037405+0000","last_peered":"2024-05-28T06:08:56.037405+0000","last_clean":"2024-05-28T06:08:56.037405+0000","last_became_active":"2024-05-28T06:08:55.063404+0000","last_became_peered":"2024-05-28T06:08:55.063404+0000","last_unstale":"2024-05-28T06:08:56.037405+0000","last_undegraded":"2024-05-28T06:08:56.037405+0000","last_fullsized":"2024-05-28T06:08:56.037405+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:52:35.504075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.606010+0000","last_change":"2024-05-28T06:08:55.067468+0000","last_active":"2024-05-28T06:08:56.606010+0000","last_peered":"2024-05-28T06:08:56.606010+0000","last_clean":"2024-05-28T06:08:56.606010+0000","last_became_active":"2024-05-28T06:08:55.067187+0000","last_became_peered":"2024-05-28T06:08:55.067187+0000","last_unstale":"2024-05-28T06:08:56.606010+0000","last_undegraded":"2024-05-28T06:08:56.606010+0000","last_fullsized":"2024-05-28T06:08:56.606010+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:45:51.613393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.033553+0000","last_change":"2024-05-28T06:08:55.049528+0000","last_active":"2024-05-28T06:08:56.033553+0000","last_peered":"2024-05-28T06:08:56.033553+0000","last_clean":"2024-05-28T06:08:56.033553+0000","last_became_active":"2024-05-28T06:08:55.049228+0000","last_became_peered":"2024-05-28T06:08:55.049228+0000","last_unstale":"2024-05-28T06:08:56.033553+0000","last_undegraded":"2024-05-28T06:08:56.033553+0000","last_fullsized":"2024-05-28T06:08:56.033553+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:52:59.732955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.601983+0000","last_change":"2024-05-28T06:08:55.054821+0000","last_active":"2024-05-28T06:08:56.601983+0000","last_peered":"2024-05-28T06:08:56.601983+0000","last_clean":"2024-05-28T06:08:56.601983+0000","last_became_active":"2024-05-28T06:08:55.054506+0000","last_became_peered":"2024-05-28T06:08:55.054506+0000","last_unstale":"2024-05-28T06:08:56.601983+0000","last_undegraded":"2024-05-28T06:08:56.601983+0000","last_fullsized":"2024-05-28T06:08:56.601983+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:34:32.590326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"44'4","reported_seq":18,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.606385+0000","last_change":"2024-05-28T06:08:54.044108+0000","last_active":"2024-05-28T06:08:56.606385+0000","last_peered":"2024-05-28T06:08:56.606385+0000","last_clean":"2024-05-28T06:08:56.606385+0000","last_became_active":"2024-05-28T06:08:54.043637+0000","last_became_peered":"2024-05-28T06:08:54.043637+0000","last_unstale":"2024-05-28T06:08:56.606385+0000","last_undegraded":"2024-05-28T06:08:56.606385+0000","last_fullsized":"2024-05-28T06:08:56.606385+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:59:53.626690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.605924+0000","last_change":"2024-05-28T06:08:55.066359+0000","last_active":"2024-05-28T06:08:56.605924+0000","last_peered":"2024-05-28T06:08:56.605924+0000","last_clean":"2024-05-28T06:08:56.605924+0000","last_became_active":"2024-05-28T06:08:55.066019+0000","last_became_peered":"2024-05-28T06:08:55.066019+0000","last_unstale":"2024-05-28T06:08:56.605924+0000","last_undegraded":"2024-05-28T06:08:56.605924+0000","last_fullsized":"2024-05-28T06:08:56.605924+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:34:32.590326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"44'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.038760+0000","last_change":"2024-05-28T06:08:54.036739+0000","last_active":"2024-05-28T06:08:56.038760+0000","last_peered":"2024-05-28T06:08:56.038760+0000","last_clean":"2024-05-28T06:08:56.038760+0000","last_became_active":"2024-05-28T06:08:54.036446+0000","last_became_peered":"2024-05-28T06:08:54.036446+0000","last_unstale":"2024-05-28T06:08:56.038760+0000","last_undegraded":"2024-05-28T06:08:56.038760+0000","last_fullsized":"2024-05-28T06:08:56.038760+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:08:09.679468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.605330+0000","last_change":"2024-05-28T06:08:55.066245+0000","last_active":"2024-05-28T06:08:56.605330+0000","last_peered":"2024-05-28T06:08:56.605330+0000","last_clean":"2024-05-28T06:08:56.605330+0000","last_became_active":"2024-05-28T06:08:55.063953+0000","last_became_peered":"2024-05-28T06:08:55.063953+0000","last_unstale":"2024-05-28T06:08:56.605330+0000","last_undegraded":"2024-05-28T06:08:56.605330+0000","last_fullsized":"2024-05-28T06:08:56.605330+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:33:57.271643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.603441+0000","last_change":"2024-05-28T06:08:54.035269+0000","last_active":"2024-05-28T06:08:56.603441+0000","last_peered":"2024-05-28T06:08:56.603441+0000","last_clean":"2024-05-28T06:08:56.603441+0000","last_became_active":"2024-05-28T06:08:54.034916+0000","last_became_peered":"2024-05-28T06:08:54.034916+0000","last_unstale":"2024-05-28T06:08:56.603441+0000","last_undegraded":"2024-05-28T06:08:56.603441+0000","last_fullsized":"2024-05-28T06:08:56.603441+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:16:33.288780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.602784+0000","last_change":"2024-05-28T06:08:55.070511+0000","last_active":"2024-05-28T06:08:56.602784+0000","last_peered":"2024-05-28T06:08:56.602784+0000","last_clean":"2024-05-28T06:08:56.602784+0000","last_became_active":"2024-05-28T06:08:55.070201+0000","last_became_peered":"2024-05-28T06:08:55.070201+0000","last_unstale":"2024-05-28T06:08:56.602784+0000","last_undegraded":"2024-05-28T06:08:56.602784+0000","last_fullsized":"2024-05-28T06:08:56.602784+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:47:30.448166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.605662+0000","last_change":"2024-05-28T06:08:54.044956+0000","last_active":"2024-05-28T06:08:56.605662+0000","last_peered":"2024-05-28T06:08:56.605662+0000","last_clean":"2024-05-28T06:08:56.605662+0000","last_became_active":"2024-05-28T06:08:54.044646+0000","last_became_peered":"2024-05-28T06:08:54.044646+0000","last_unstale":"2024-05-28T06:08:56.605662+0000","last_undegraded":"2024-05-28T06:08:56.605662+0000","last_fullsized":"2024-05-28T06:08:56.605662+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:45:56.775580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.037409+0000","last_change":"2024-05-28T06:08:55.073311+0000","last_active":"2024-05-28T06:08:56.037409+0000","last_peered":"2024-05-28T06:08:56.037409+0000","last_clean":"2024-05-28T06:08:56.037409+0000","last_became_active":"2024-05-28T06:08:55.072761+0000","last_became_peered":"2024-05-28T06:08:55.072761+0000","last_unstale":"2024-05-28T06:08:56.037409+0000","last_undegraded":"2024-05-28T06:08:56.037409+0000","last_fullsized":"2024-05-28T06:08:56.037409+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:34:32.590326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"44'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.606218+0000","last_change":"2024-05-28T06:08:54.042495+0000","last_active":"2024-05-28T06:08:56.606218+0000","last_peered":"2024-05-28T06:08:56.606218+0000","last_clean":"2024-05-28T06:08:56.606218+0000","last_became_active":"2024-05-28T06:08:54.042143+0000","last_became_peered":"2024-05-28T06:08:54.042143+0000","last_unstale":"2024-05-28T06:08:56.606218+0000","last_undegraded":"2024-05-28T06:08:56.606218+0000","last_fullsized":"2024-05-28T06:08:56.606218+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:01:51.040147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.604208+0000","last_change":"2024-05-28T06:08:55.055134+0000","last_active":"2024-05-28T06:08:56.604208+0000","last_peered":"2024-05-28T06:08:56.604208+0000","last_clean":"2024-05-28T06:08:56.604208+0000","last_became_active":"2024-05-28T06:08:55.054881+0000","last_became_peered":"2024-05-28T06:08:55.054881+0000","last_unstale":"2024-05-28T06:08:56.604208+0000","last_undegraded":"2024-05-28T06:08:56.604208+0000","last_fullsized":"2024-05-28T06:08:56.604208+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:59:54.640280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"44'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.038029+0000","last_change":"2024-05-28T06:08:54.030180+0000","last_active":"2024-05-28T06:08:56.038029+0000","last_peered":"2024-05-28T06:08:56.038029+0000","last_clean":"2024-05-28T06:08:56.038029+0000","last_became_active":"2024-05-28T06:08:54.029892+0000","last_became_peered":"2024-05-28T06:08:54.029892+0000","last_unstale":"2024-05-28T06:08:56.038029+0000","last_undegraded":"2024-05-28T06:08:56.038029+0000","last_fullsized":"2024-05-28T06:08:56.038029+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:18:13.820152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.602565+0000","last_change":"2024-05-28T06:08:55.076181+0000","last_active":"2024-05-28T06:08:56.602565+0000","last_peered":"2024-05-28T06:08:56.602565+0000","last_clean":"2024-05-28T06:08:56.602565+0000","last_became_active":"2024-05-28T06:08:55.075570+0000","last_became_peered":"2024-05-28T06:08:55.075570+0000","last_unstale":"2024-05-28T06:08:56.602565+0000","last_undegraded":"2024-05-28T06:08:56.602565+0000","last_fullsized":"2024-05-28T06:08:56.602565+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:47:30.448166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.604320+0000","last_change":"2024-05-28T06:08:54.035697+0000","last_active":"2024-05-28T06:08:56.604320+0000","last_peered":"2024-05-28T06:08:56.604320+0000","last_clean":"2024-05-28T06:08:56.604320+0000","last_became_active":"2024-05-28T06:08:54.032931+0000","last_became_peered":"2024-05-28T06:08:54.032931+0000","last_unstale":"2024-05-28T06:08:56.604320+0000","last_undegraded":"2024-05-28T06:08:56.604320+0000","last_fullsized":"2024-05-28T06:08:56.604320+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:14:59.855890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.606117+0000","last_change":"2024-05-28T06:08:55.066914+0000","last_active":"2024-05-28T06:08:56.606117+0000","last_peered":"2024-05-28T06:08:56.606117+0000","last_clean":"2024-05-28T06:08:56.606117+0000","last_became_active":"2024-05-28T06:08:55.066242+0000","last_became_peered":"2024-05-28T06:08:55.066242+0000","last_unstale":"2024-05-28T06:08:56.606117+0000","last_undegraded":"2024-05-28T06:08:56.606117+0000","last_fullsized":"2024-05-28T06:08:56.606117+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:52:35.504075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.602798+0000","last_change":"2024-05-28T06:08:54.032388+0000","last_active":"2024-05-28T06:08:56.602798+0000","last_peered":"2024-05-28T06:08:56.602798+0000","last_clean":"2024-05-28T06:08:56.602798+0000","last_became_active":"2024-05-28T06:08:54.032095+0000","last_became_peered":"2024-05-28T06:08:54.032095+0000","last_unstale":"2024-05-28T06:08:56.602798+0000","last_undegraded":"2024-05-28T06:08:56.602798+0000","last_fullsized":"2024-05-28T06:08:56.602798+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:16:17.652829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.038435+0000","last_change":"2024-05-28T06:08:55.071079+0000","last_active":"2024-05-28T06:08:56.038435+0000","last_peered":"2024-05-28T06:08:56.038435+0000","last_clean":"2024-05-28T06:08:56.038435+0000","last_became_active":"2024-05-28T06:08:55.070804+0000","last_became_peered":"2024-05-28T06:08:55.070804+0000","last_unstale":"2024-05-28T06:08:56.038435+0000","last_undegraded":"2024-05-28T06:08:56.038435+0000","last_fullsized":"2024-05-28T06:08:56.038435+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:19:18.886834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.038778+0000","last_change":"2024-05-28T06:08:54.027262+0000","last_active":"2024-05-28T06:08:56.038778+0000","last_peered":"2024-05-28T06:08:56.038778+0000","last_clean":"2024-05-28T06:08:56.038778+0000","last_became_active":"2024-05-28T06:08:54.026066+0000","last_became_peered":"2024-05-28T06:08:54.026066+0000","last_unstale":"2024-05-28T06:08:56.038778+0000","last_undegraded":"2024-05-28T06:08:56.038778+0000","last_fullsized":"2024-05-28T06:08:56.038778+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:37:34.767912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.606324+0000","last_change":"2024-05-28T06:08:55.071950+0000","last_active":"2024-05-28T06:08:56.606324+0000","last_peered":"2024-05-28T06:08:56.606324+0000","last_clean":"2024-05-28T06:08:56.606324+0000","last_became_active":"2024-05-28T06:08:55.071671+0000","last_became_peered":"2024-05-28T06:08:55.071671+0000","last_unstale":"2024-05-28T06:08:56.606324+0000","last_undegraded":"2024-05-28T06:08:56.606324+0000","last_fullsized":"2024-05-28T06:08:56.606324+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:30:19.818261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.603530+0000","last_change":"2024-05-28T06:08:54.040877+0000","last_active":"2024-05-28T06:08:56.603530+0000","last_peered":"2024-05-28T06:08:56.603530+0000","last_clean":"2024-05-28T06:08:56.603530+0000","last_became_active":"2024-05-28T06:08:54.040615+0000","last_became_peered":"2024-05-28T06:08:54.040615+0000","last_unstale":"2024-05-28T06:08:56.603530+0000","last_undegraded":"2024-05-28T06:08:56.603530+0000","last_fullsized":"2024-05-28T06:08:56.603530+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:18:13.820152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.606380+0000","last_change":"2024-05-28T06:08:55.059781+0000","last_active":"2024-05-28T06:08:56.606380+0000","last_peered":"2024-05-28T06:08:56.606380+0000","last_clean":"2024-05-28T06:08:56.606380+0000","last_became_active":"2024-05-28T06:08:55.059497+0000","last_became_peered":"2024-05-28T06:08:55.059497+0000","last_unstale":"2024-05-28T06:08:56.606380+0000","last_undegraded":"2024-05-28T06:08:56.606380+0000","last_fullsized":"2024-05-28T06:08:56.606380+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:32:24.478463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"44'3","reported_seq":17,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.606779+0000","last_change":"2024-05-28T06:08:54.046805+0000","last_active":"2024-05-28T06:08:56.606779+0000","last_peered":"2024-05-28T06:08:56.606779+0000","last_clean":"2024-05-28T06:08:56.606779+0000","last_became_active":"2024-05-28T06:08:54.046336+0000","last_became_peered":"2024-05-28T06:08:54.046336+0000","last_unstale":"2024-05-28T06:08:56.606779+0000","last_undegraded":"2024-05-28T06:08:56.606779+0000","last_fullsized":"2024-05-28T06:08:56.606779+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:59:53.626690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.606518+0000","last_change":"2024-05-28T06:08:55.073615+0000","last_active":"2024-05-28T06:08:56.606518+0000","last_peered":"2024-05-28T06:08:56.606518+0000","last_clean":"2024-05-28T06:08:56.606518+0000","last_became_active":"2024-05-28T06:08:55.071441+0000","last_became_peered":"2024-05-28T06:08:55.071441+0000","last_unstale":"2024-05-28T06:08:56.606518+0000","last_undegraded":"2024-05-28T06:08:56.606518+0000","last_fullsized":"2024-05-28T06:08:56.606518+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:10:00.499041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"44'6","reported_seq":20,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.033710+0000","last_change":"2024-05-28T06:08:54.029622+0000","last_active":"2024-05-28T06:08:56.033710+0000","last_peered":"2024-05-28T06:08:56.033710+0000","last_clean":"2024-05-28T06:08:56.033710+0000","last_became_active":"2024-05-28T06:08:54.026140+0000","last_became_peered":"2024-05-28T06:08:54.026140+0000","last_unstale":"2024-05-28T06:08:56.033710+0000","last_undegraded":"2024-05-28T06:08:56.033710+0000","last_fullsized":"2024-05-28T06:08:56.033710+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:56:58.418726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.602994+0000","last_change":"2024-05-28T06:08:55.053056+0000","last_active":"2024-05-28T06:08:56.602994+0000","last_peered":"2024-05-28T06:08:56.602994+0000","last_clean":"2024-05-28T06:08:56.602994+0000","last_became_active":"2024-05-28T06:08:55.050989+0000","last_became_peered":"2024-05-28T06:08:55.050989+0000","last_unstale":"2024-05-28T06:08:56.602994+0000","last_undegraded":"2024-05-28T06:08:56.602994+0000","last_fullsized":"2024-05-28T06:08:56.602994+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:27:41.378357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.606515+0000","last_change":"2024-05-28T06:08:54.038320+0000","last_active":"2024-05-28T06:08:56.606515+0000","last_peered":"2024-05-28T06:08:56.606515+0000","last_clean":"2024-05-28T06:08:56.606515+0000","last_became_active":"2024-05-28T06:08:54.037824+0000","last_became_peered":"2024-05-28T06:08:54.037824+0000","last_unstale":"2024-05-28T06:08:56.606515+0000","last_undegraded":"2024-05-28T06:08:56.606515+0000","last_fullsized":"2024-05-28T06:08:56.606515+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:59:27.653504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.604164+0000","last_change":"2024-05-28T06:08:55.075823+0000","last_active":"2024-05-28T06:08:56.604164+0000","last_peered":"2024-05-28T06:08:56.604164+0000","last_clean":"2024-05-28T06:08:56.604164+0000","last_became_active":"2024-05-28T06:08:55.069901+0000","last_became_peered":"2024-05-28T06:08:55.069901+0000","last_unstale":"2024-05-28T06:08:56.604164+0000","last_undegraded":"2024-05-28T06:08:56.604164+0000","last_fullsized":"2024-05-28T06:08:56.604164+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:59:28.667094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.605527+0000","last_change":"2024-05-28T06:08:54.046264+0000","last_active":"2024-05-28T06:08:56.605527+0000","last_peered":"2024-05-28T06:08:56.605527+0000","last_clean":"2024-05-28T06:08:56.605527+0000","last_became_active":"2024-05-28T06:08:54.045957+0000","last_became_peered":"2024-05-28T06:08:54.045957+0000","last_unstale":"2024-05-28T06:08:56.605527+0000","last_undegraded":"2024-05-28T06:08:56.605527+0000","last_fullsized":"2024-05-28T06:08:56.605527+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:59:27.653504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.605332+0000","last_change":"2024-05-28T06:08:55.047136+0000","last_active":"2024-05-28T06:08:56.605332+0000","last_peered":"2024-05-28T06:08:56.605332+0000","last_clean":"2024-05-28T06:08:56.605332+0000","last_became_active":"2024-05-28T06:08:55.045959+0000","last_became_peered":"2024-05-28T06:08:55.045959+0000","last_unstale":"2024-05-28T06:08:56.605332+0000","last_undegraded":"2024-05-28T06:08:56.605332+0000","last_fullsized":"2024-05-28T06:08:56.605332+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:10:00.499041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.605624+0000","last_change":"2024-05-28T06:08:54.038131+0000","last_active":"2024-05-28T06:08:56.605624+0000","last_peered":"2024-05-28T06:08:56.605624+0000","last_clean":"2024-05-28T06:08:56.605624+0000","last_became_active":"2024-05-28T06:08:54.037578+0000","last_became_peered":"2024-05-28T06:08:54.037578+0000","last_unstale":"2024-05-28T06:08:56.605624+0000","last_undegraded":"2024-05-28T06:08:56.605624+0000","last_fullsized":"2024-05-28T06:08:56.605624+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:06:36.665935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.604966+0000","last_change":"2024-05-28T06:08:55.072863+0000","last_active":"2024-05-28T06:08:56.604966+0000","last_peered":"2024-05-28T06:08:56.604966+0000","last_clean":"2024-05-28T06:08:56.604966+0000","last_became_active":"2024-05-28T06:08:55.072388+0000","last_became_peered":"2024-05-28T06:08:55.072388+0000","last_unstale":"2024-05-28T06:08:56.604966+0000","last_undegraded":"2024-05-28T06:08:56.604966+0000","last_fullsized":"2024-05-28T06:08:56.604966+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:51:13.118226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"44'8","reported_seq":22,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.603088+0000","last_change":"2024-05-28T06:08:54.034498+0000","last_active":"2024-05-28T06:08:56.603088+0000","last_peered":"2024-05-28T06:08:56.603088+0000","last_clean":"2024-05-28T06:08:56.603088+0000","last_became_active":"2024-05-28T06:08:54.034152+0000","last_became_peered":"2024-05-28T06:08:54.034152+0000","last_unstale":"2024-05-28T06:08:56.603088+0000","last_undegraded":"2024-05-28T06:08:56.603088+0000","last_fullsized":"2024-05-28T06:08:56.603088+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:51:33.930123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.034026+0000","last_change":"2024-05-28T06:08:55.066857+0000","last_active":"2024-05-28T06:08:56.034026+0000","last_peered":"2024-05-28T06:08:56.034026+0000","last_clean":"2024-05-28T06:08:56.034026+0000","last_became_active":"2024-05-28T06:08:55.066592+0000","last_became_peered":"2024-05-28T06:08:55.066592+0000","last_unstale":"2024-05-28T06:08:56.034026+0000","last_undegraded":"2024-05-28T06:08:56.034026+0000","last_fullsized":"2024-05-28T06:08:56.034026+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:51:34.943713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"44'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.034196+0000","last_change":"2024-05-28T06:08:54.035386+0000","last_active":"2024-05-28T06:08:56.034196+0000","last_peered":"2024-05-28T06:08:56.034196+0000","last_clean":"2024-05-28T06:08:56.034196+0000","last_became_active":"2024-05-28T06:08:54.033511+0000","last_became_peered":"2024-05-28T06:08:54.033511+0000","last_unstale":"2024-05-28T06:08:56.034196+0000","last_undegraded":"2024-05-28T06:08:56.034196+0000","last_fullsized":"2024-05-28T06:08:56.034196+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:31:59.983717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.604783+0000","last_change":"2024-05-28T06:08:55.052937+0000","last_active":"2024-05-28T06:08:56.604783+0000","last_peered":"2024-05-28T06:08:56.604783+0000","last_clean":"2024-05-28T06:08:56.604783+0000","last_became_active":"2024-05-28T06:08:55.052639+0000","last_became_peered":"2024-05-28T06:08:55.052639+0000","last_unstale":"2024-05-28T06:08:56.604783+0000","last_undegraded":"2024-05-28T06:08:56.604783+0000","last_fullsized":"2024-05-28T06:08:56.604783+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:42:35.586471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.038514+0000","last_change":"2024-05-28T06:08:54.035942+0000","last_active":"2024-05-28T06:08:56.038514+0000","last_peered":"2024-05-28T06:08:56.038514+0000","last_clean":"2024-05-28T06:08:56.038514+0000","last_became_active":"2024-05-28T06:08:54.033436+0000","last_became_peered":"2024-05-28T06:08:54.033436+0000","last_unstale":"2024-05-28T06:08:56.038514+0000","last_undegraded":"2024-05-28T06:08:56.038514+0000","last_fullsized":"2024-05-28T06:08:56.038514+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:50:52.128343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.604363+0000","last_change":"2024-05-28T06:08:55.057536+0000","last_active":"2024-05-28T06:08:56.604363+0000","last_peered":"2024-05-28T06:08:56.604363+0000","last_clean":"2024-05-28T06:08:56.604363+0000","last_became_active":"2024-05-28T06:08:55.057206+0000","last_became_peered":"2024-05-28T06:08:55.057206+0000","last_unstale":"2024-05-28T06:08:56.604363+0000","last_undegraded":"2024-05-28T06:08:56.604363+0000","last_fullsized":"2024-05-28T06:08:56.604363+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:29:28.504290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"44'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.605231+0000","last_change":"2024-05-28T06:08:54.036250+0000","last_active":"2024-05-28T06:08:56.605231+0000","last_peered":"2024-05-28T06:08:56.605231+0000","last_clean":"2024-05-28T06:08:56.605231+0000","last_became_active":"2024-05-28T06:08:54.034306+0000","last_became_peered":"2024-05-28T06:08:54.034306+0000","last_unstale":"2024-05-28T06:08:56.605231+0000","last_undegraded":"2024-05-28T06:08:56.605231+0000","last_fullsized":"2024-05-28T06:08:56.605231+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:11:14.276087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.603354+0000","last_change":"2024-05-28T06:08:55.053434+0000","last_active":"2024-05-28T06:08:56.603354+0000","last_peered":"2024-05-28T06:08:56.603354+0000","last_clean":"2024-05-28T06:08:56.603354+0000","last_became_active":"2024-05-28T06:08:55.052617+0000","last_became_peered":"2024-05-28T06:08:55.052617+0000","last_unstale":"2024-05-28T06:08:56.603354+0000","last_undegraded":"2024-05-28T06:08:56.603354+0000","last_fullsized":"2024-05-28T06:08:56.603354+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:01:16.405059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.606407+0000","last_change":"2024-05-28T06:08:54.043914+0000","last_active":"2024-05-28T06:08:56.606407+0000","last_peered":"2024-05-28T06:08:56.606407+0000","last_clean":"2024-05-28T06:08:56.606407+0000","last_became_active":"2024-05-28T06:08:54.043662+0000","last_became_peered":"2024-05-28T06:08:54.043662+0000","last_unstale":"2024-05-28T06:08:56.606407+0000","last_undegraded":"2024-05-28T06:08:56.606407+0000","last_fullsized":"2024-05-28T06:08:56.606407+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:53:21.444456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.605566+0000","last_change":"2024-05-28T06:08:55.051810+0000","last_active":"2024-05-28T06:08:56.605566+0000","last_peered":"2024-05-28T06:08:56.605566+0000","last_clean":"2024-05-28T06:08:56.605566+0000","last_became_active":"2024-05-28T06:08:55.050715+0000","last_became_peered":"2024-05-28T06:08:55.050715+0000","last_unstale":"2024-05-28T06:08:56.605566+0000","last_undegraded":"2024-05-28T06:08:56.605566+0000","last_fullsized":"2024-05-28T06:08:56.605566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:25:17.106626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.606671+0000","last_change":"2024-05-28T06:08:54.042952+0000","last_active":"2024-05-28T06:08:56.606671+0000","last_peered":"2024-05-28T06:08:56.606671+0000","last_clean":"2024-05-28T06:08:56.606671+0000","last_became_active":"2024-05-28T06:08:54.042713+0000","last_became_peered":"2024-05-28T06:08:54.042713+0000","last_unstale":"2024-05-28T06:08:56.606671+0000","last_undegraded":"2024-05-28T06:08:56.606671+0000","last_fullsized":"2024-05-28T06:08:56.606671+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:45:56.775580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.602347+0000","last_change":"2024-05-28T06:08:55.066822+0000","last_active":"2024-05-28T06:08:56.602347+0000","last_peered":"2024-05-28T06:08:56.602347+0000","last_clean":"2024-05-28T06:08:56.602347+0000","last_became_active":"2024-05-28T06:08:55.066494+0000","last_became_peered":"2024-05-28T06:08:55.066494+0000","last_unstale":"2024-05-28T06:08:56.602347+0000","last_undegraded":"2024-05-28T06:08:56.602347+0000","last_fullsized":"2024-05-28T06:08:56.602347+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:52:35.504075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.603292+0000","last_change":"2024-05-28T06:08:54.039984+0000","last_active":"2024-05-28T06:08:56.603292+0000","last_peered":"2024-05-28T06:08:56.603292+0000","last_clean":"2024-05-28T06:08:56.603292+0000","last_became_active":"2024-05-28T06:08:54.038849+0000","last_became_peered":"2024-05-28T06:08:54.038849+0000","last_unstale":"2024-05-28T06:08:56.603292+0000","last_undegraded":"2024-05-28T06:08:56.603292+0000","last_fullsized":"2024-05-28T06:08:56.603292+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:19:17.873244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.033956+0000","last_change":"2024-05-28T06:08:55.057248+0000","last_active":"2024-05-28T06:08:56.033956+0000","last_peered":"2024-05-28T06:08:56.033956+0000","last_clean":"2024-05-28T06:08:56.033956+0000","last_became_active":"2024-05-28T06:08:55.056958+0000","last_became_peered":"2024-05-28T06:08:55.056958+0000","last_unstale":"2024-05-28T06:08:56.033956+0000","last_undegraded":"2024-05-28T06:08:56.033956+0000","last_fullsized":"2024-05-28T06:08:56.033956+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:16:18.666419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.603963+0000","last_change":"2024-05-28T06:08:54.043822+0000","last_active":"2024-05-28T06:08:56.603963+0000","last_peered":"2024-05-28T06:08:56.603963+0000","last_clean":"2024-05-28T06:08:56.603963+0000","last_became_active":"2024-05-28T06:08:54.043402+0000","last_became_peered":"2024-05-28T06:08:54.043402+0000","last_unstale":"2024-05-28T06:08:56.603963+0000","last_undegraded":"2024-05-28T06:08:56.603963+0000","last_fullsized":"2024-05-28T06:08:56.603963+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:07:41.209833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.604001+0000","last_change":"2024-05-28T06:08:55.059492+0000","last_active":"2024-05-28T06:08:56.604001+0000","last_peered":"2024-05-28T06:08:56.604001+0000","last_clean":"2024-05-28T06:08:56.604001+0000","last_became_active":"2024-05-28T06:08:55.057314+0000","last_became_peered":"2024-05-28T06:08:55.057314+0000","last_unstale":"2024-05-28T06:08:56.604001+0000","last_undegraded":"2024-05-28T06:08:56.604001+0000","last_fullsized":"2024-05-28T06:08:56.604001+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:27:47.697809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.602543+0000","last_change":"2024-05-28T06:08:54.046229+0000","last_active":"2024-05-28T06:08:56.602543+0000","last_peered":"2024-05-28T06:08:56.602543+0000","last_clean":"2024-05-28T06:08:56.602543+0000","last_became_active":"2024-05-28T06:08:54.045868+0000","last_became_peered":"2024-05-28T06:08:54.045868+0000","last_unstale":"2024-05-28T06:08:56.602543+0000","last_undegraded":"2024-05-28T06:08:56.602543+0000","last_fullsized":"2024-05-28T06:08:56.602543+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:11:14.276087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.606295+0000","last_change":"2024-05-28T06:08:55.059577+0000","last_active":"2024-05-28T06:08:56.606295+0000","last_peered":"2024-05-28T06:08:56.606295+0000","last_clean":"2024-05-28T06:08:56.606295+0000","last_became_active":"2024-05-28T06:08:55.058682+0000","last_became_peered":"2024-05-28T06:08:55.058682+0000","last_unstale":"2024-05-28T06:08:56.606295+0000","last_undegraded":"2024-05-28T06:08:56.606295+0000","last_fullsized":"2024-05-28T06:08:56.606295+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:47:30.448166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"45'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.606231+0000","last_change":"2024-05-28T06:08:54.034283+0000","last_active":"2024-05-28T06:08:56.606231+0000","last_peered":"2024-05-28T06:08:56.606231+0000","last_clean":"2024-05-28T06:08:56.606231+0000","last_became_active":"2024-05-28T06:08:54.033985+0000","last_became_peered":"2024-05-28T06:08:54.033985+0000","last_unstale":"2024-05-28T06:08:56.606231+0000","last_undegraded":"2024-05-28T06:08:56.606231+0000","last_fullsized":"2024-05-28T06:08:56.606231+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:27:46.684219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.604602+0000","last_change":"2024-05-28T06:08:55.065596+0000","last_active":"2024-05-28T06:08:56.604602+0000","last_peered":"2024-05-28T06:08:56.604602+0000","last_clean":"2024-05-28T06:08:56.604602+0000","last_became_active":"2024-05-28T06:08:55.065284+0000","last_became_peered":"2024-05-28T06:08:55.065284+0000","last_unstale":"2024-05-28T06:08:56.604602+0000","last_undegraded":"2024-05-28T06:08:56.604602+0000","last_fullsized":"2024-05-28T06:08:56.604602+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:06:37.679525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.606888+0000","last_change":"2024-05-28T06:08:54.047896+0000","last_active":"2024-05-28T06:08:56.606888+0000","last_peered":"2024-05-28T06:08:56.606888+0000","last_clean":"2024-05-28T06:08:56.606888+0000","last_became_active":"2024-05-28T06:08:54.047535+0000","last_became_peered":"2024-05-28T06:08:54.047535+0000","last_unstale":"2024-05-28T06:08:56.606888+0000","last_undegraded":"2024-05-28T06:08:56.606888+0000","last_fullsized":"2024-05-28T06:08:56.606888+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:27:46.684219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.606644+0000","last_change":"2024-05-28T06:08:55.064486+0000","last_active":"2024-05-28T06:08:56.606644+0000","last_peered":"2024-05-28T06:08:56.606644+0000","last_clean":"2024-05-28T06:08:56.606644+0000","last_became_active":"2024-05-28T06:08:55.062008+0000","last_became_peered":"2024-05-28T06:08:55.062008+0000","last_unstale":"2024-05-28T06:08:56.606644+0000","last_undegraded":"2024-05-28T06:08:56.606644+0000","last_fullsized":"2024-05-28T06:08:56.606644+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:55:01.463179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.601727+0000","last_change":"2024-05-28T06:08:54.037383+0000","last_active":"2024-05-28T06:08:56.601727+0000","last_peered":"2024-05-28T06:08:56.601727+0000","last_clean":"2024-05-28T06:08:56.601727+0000","last_became_active":"2024-05-28T06:08:54.036598+0000","last_became_peered":"2024-05-28T06:08:54.036598+0000","last_unstale":"2024-05-28T06:08:56.601727+0000","last_undegraded":"2024-05-28T06:08:56.601727+0000","last_fullsized":"2024-05-28T06:08:56.601727+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:00:00.862082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.605922+0000","last_change":"2024-05-28T06:08:55.051492+0000","last_active":"2024-05-28T06:08:56.605922+0000","last_peered":"2024-05-28T06:08:56.605922+0000","last_clean":"2024-05-28T06:08:56.605922+0000","last_became_active":"2024-05-28T06:08:55.051094+0000","last_became_peered":"2024-05-28T06:08:55.051094+0000","last_unstale":"2024-05-28T06:08:56.605922+0000","last_undegraded":"2024-05-28T06:08:56.605922+0000","last_fullsized":"2024-05-28T06:08:56.605922+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:02:41.768007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.605220+0000","last_change":"2024-05-28T06:08:54.046483+0000","last_active":"2024-05-28T06:08:56.605220+0000","last_peered":"2024-05-28T06:08:56.605220+0000","last_clean":"2024-05-28T06:08:56.605220+0000","last_became_active":"2024-05-28T06:08:54.045605+0000","last_became_peered":"2024-05-28T06:08:54.045605+0000","last_unstale":"2024-05-28T06:08:56.605220+0000","last_undegraded":"2024-05-28T06:08:56.605220+0000","last_fullsized":"2024-05-28T06:08:56.605220+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:01:51.040147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.605884+0000","last_change":"2024-05-28T06:08:55.050859+0000","last_active":"2024-05-28T06:08:56.605884+0000","last_peered":"2024-05-28T06:08:56.605884+0000","last_clean":"2024-05-28T06:08:56.605884+0000","last_became_active":"2024-05-28T06:08:55.050572+0000","last_became_peered":"2024-05-28T06:08:55.050572+0000","last_unstale":"2024-05-28T06:08:56.605884+0000","last_undegraded":"2024-05-28T06:08:56.605884+0000","last_fullsized":"2024-05-28T06:08:56.605884+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:06:45.885680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.037820+0000","last_change":"2024-05-28T06:08:54.037676+0000","last_active":"2024-05-28T06:08:56.037820+0000","last_peered":"2024-05-28T06:08:56.037820+0000","last_clean":"2024-05-28T06:08:56.037820+0000","last_became_active":"2024-05-28T06:08:54.037301+0000","last_became_peered":"2024-05-28T06:08:54.037301+0000","last_unstale":"2024-05-28T06:08:56.037820+0000","last_undegraded":"2024-05-28T06:08:56.037820+0000","last_fullsized":"2024-05-28T06:08:56.037820+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:37:24.350191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.602195+0000","last_change":"2024-05-28T06:08:55.062795+0000","last_active":"2024-05-28T06:08:56.602195+0000","last_peered":"2024-05-28T06:08:56.602195+0000","last_clean":"2024-05-28T06:08:56.602195+0000","last_became_active":"2024-05-28T06:08:55.059485+0000","last_became_peered":"2024-05-28T06:08:55.059485+0000","last_unstale":"2024-05-28T06:08:56.602195+0000","last_undegraded":"2024-05-28T06:08:56.602195+0000","last_fullsized":"2024-05-28T06:08:56.602195+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:53:22.458046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.601577+0000","last_change":"2024-05-28T06:08:54.044188+0000","last_active":"2024-05-28T06:08:56.601577+0000","last_peered":"2024-05-28T06:08:56.601577+0000","last_clean":"2024-05-28T06:08:56.601577+0000","last_became_active":"2024-05-28T06:08:54.043655+0000","last_became_peered":"2024-05-28T06:08:54.043655+0000","last_unstale":"2024-05-28T06:08:56.601577+0000","last_undegraded":"2024-05-28T06:08:56.601577+0000","last_fullsized":"2024-05-28T06:08:56.601577+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:24:09.591517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.037385+0000","last_change":"2024-05-28T06:08:55.070064+0000","last_active":"2024-05-28T06:08:56.037385+0000","last_peered":"2024-05-28T06:08:56.037385+0000","last_clean":"2024-05-28T06:08:56.037385+0000","last_became_active":"2024-05-28T06:08:55.069804+0000","last_became_peered":"2024-05-28T06:08:55.069804+0000","last_unstale":"2024-05-28T06:08:56.037385+0000","last_undegraded":"2024-05-28T06:08:56.037385+0000","last_fullsized":"2024-05-28T06:08:56.037385+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:59:25.013392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.603247+0000","last_change":"2024-05-28T06:08:54.038376+0000","last_active":"2024-05-28T06:08:56.603247+0000","last_peered":"2024-05-28T06:08:56.603247+0000","last_clean":"2024-05-28T06:08:56.603247+0000","last_became_active":"2024-05-28T06:08:54.037952+0000","last_became_peered":"2024-05-28T06:08:54.037952+0000","last_unstale":"2024-05-28T06:08:56.603247+0000","last_undegraded":"2024-05-28T06:08:56.603247+0000","last_fullsized":"2024-05-28T06:08:56.603247+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:01:37.817834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.603480+0000","last_change":"2024-05-28T06:08:55.071657+0000","last_active":"2024-05-28T06:08:56.603480+0000","last_peered":"2024-05-28T06:08:56.603480+0000","last_clean":"2024-05-28T06:08:56.603480+0000","last_became_active":"2024-05-28T06:08:55.071416+0000","last_became_peered":"2024-05-28T06:08:55.071416+0000","last_unstale":"2024-05-28T06:08:56.603480+0000","last_undegraded":"2024-05-28T06:08:56.603480+0000","last_fullsized":"2024-05-28T06:08:56.603480+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:53.997522+0000","last_clean_scrub_stamp":"2024-05-28T06:08:53.997522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:01:52.053737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.604754+0000","last_change":"2024-05-28T06:08:54.036597+0000","last_active":"2024-05-28T06:08:56.604754+0000","last_peered":"2024-05-28T06:08:56.604754+0000","last_clean":"2024-05-28T06:08:56.604754+0000","last_became_active":"2024-05-28T06:08:54.034833+0000","last_became_peered":"2024-05-28T06:08:54.034833+0000","last_unstale":"2024-05-28T06:08:56.604754+0000","last_undegraded":"2024-05-28T06:08:56.604754+0000","last_fullsized":"2024-05-28T06:08:56.604754+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:08:52.983932+0000","last_clean_scrub_stamp":"2024-05-28T06:08:52.983932+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:20:27.131737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":217,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-28T06:08:56.602825+0000","last_change":"2024-05-28T06:04:01.039129+0000","last_active":"2024-05-28T06:08:56.602825+0000","last_peered":"2024-05-28T06:08:56.602825+0000","last_clean":"2024-05-28T06:08:56.602825+0000","last_became_active":"2024-05-28T06:04:01.033906+0000","last_became_peered":"2024-05-28T06:04:01.033906+0000","last_unstale":"2024-05-28T06:08:56.602825+0000","last_undegraded":"2024-05-28T06:08:56.602825+0000","last_fullsized":"2024-05-28T06:08:56.602825+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:03:59.864237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:03:59.864237+0000","last_clean_scrub_stamp":"2024-05-28T06:03:59.864237+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:17:36.440245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574911,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36440,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335400,"statfs":{"total":96636764160,"available":96599449600,"internally_reserved":0,"allocated":1126400,"data_stored":806958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574913,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35864,"kb_used_data":524,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335976,"statfs":{"total":96636764160,"available":96600039424,"internally_reserved":0,"allocated":536576,"data_stored":217172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574911,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35864,"kb_used_data":520,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335976,"statfs":{"total":96636764160,"available":96600039424,"internally_reserved":0,"allocated":532480,"data_stored":216570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574911,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35872,"kb_used_data":528,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335968,"statfs":{"total":96636764160,"available":96600031232,"internally_reserved":0,"allocated":540672,"data_stored":218112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574913,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35868,"kb_used_data":528,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335972,"statfs":{"total":96636764160,"available":96600035328,"internally_reserved":0,"allocated":540672,"data_stored":217138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574912,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35876,"kb_used_data":532,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335964,"statfs":{"total":96636764160,"available":96600027136,"internally_reserved":0,"allocated":544768,"data_stored":218134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574912,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35868,"kb_used_data":520,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335972,"statfs":{"total":96636764160,"available":96600035328,"internally_reserved":0,"allocated":532480,"data_stored":216570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574911,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36440,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335400,"statfs":{"total":96636764160,"available":96599449600,"internally_reserved":0,"allocated":1126400,"data_stored":806902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-28T06:09:05.306 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-05-28T06:09:05.306 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-05-28T06:09:05.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.556+0000 7f4681e00640 1 Processor -- start 2024-05-28T06:09:05.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.557+0000 7f4681e00640 1 -- start start 2024-05-28T06:09:05.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.558+0000 7f4681e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f467c0b32c0 0x7f467c0b3700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:05.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.558+0000 7f4681e00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f467c0d0060 con 0x7f467c15d5d0 2024-05-28T06:09:05.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.558+0000 7f4681e00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f467c15cb10 con 0x7f467c0d0640 2024-05-28T06:09:05.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.558+0000 7f4681e00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f467c0adbb0 con 0x7f467c0b32c0 2024-05-28T06:09:05.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.558+0000 7f4673e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f467c15d5d0 0x7f467c0cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:51202/0 (socket says 172.21.15.96:51202) 2024-05-28T06:09:05.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.558+0000 7f4673e00640 1 -- 172.21.15.96:0/712048721 learned_addr learned my addr 172.21.15.96:0/712048721 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:09:05.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.559+0000 7f467b400640 1 --2- 172.21.15.96:0/712048721 >> v2:172.21.15.123:3300/0 conn(0x7f467c0b32c0 0x7f467c0b3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:05.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.559+0000 7f467a000640 1 -- 172.21.15.96:0/712048721 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2397276880 0 0) 0x7f467c0d0060 con 0x7f467c15d5d0 2024-05-28T06:09:05.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.559+0000 7f467a000640 1 -- 172.21.15.96:0/712048721 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4660003520 con 0x7f467c15d5d0 2024-05-28T06:09:05.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.559+0000 7f467a000640 1 -- 172.21.15.96:0/712048721 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 23068615 0 0) 0x7f4660003520 con 0x7f467c15d5d0 2024-05-28T06:09:05.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.559+0000 7f467a000640 1 -- 172.21.15.96:0/712048721 >> v2:172.21.15.123:3300/0 conn(0x7f467c0b32c0 msgr2=0x7f467c0b3700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:09:05.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.559+0000 7f467a000640 1 --2- 172.21.15.96:0/712048721 >> v2:172.21.15.123:3300/0 conn(0x7f467c0b32c0 0x7f467c0b3700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:05.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.559+0000 7f467a000640 1 -- 172.21.15.96:0/712048721 >> v1:172.21.15.123:6789/0 conn(0x7f467c0d0640 legacy=0x7f467c0d0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:09:05.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.559+0000 7f467a000640 1 -- 172.21.15.96:0/712048721 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f467c0ad5f0 con 0x7f467c15d5d0 2024-05-28T06:09:05.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.559+0000 7f467a000640 1 -- 172.21.15.96:0/712048721 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f4664001670 con 0x7f467c15d5d0 2024-05-28T06:09:05.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.559+0000 7f467a000640 1 -- 172.21.15.96:0/712048721 >> v1:172.21.15.96:6789/0 conn(0x7f467c15d5d0 legacy=0x7f467c0cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:09:05.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.560+0000 7f467a000640 1 --2- 172.21.15.96:0/712048721 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f46600039b0 0x7f4660003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:05.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.560+0000 7f467a000640 1 --2- 172.21.15.96:0/712048721 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4660004800 0x7f4660004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:05.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.560+0000 7f467a000640 1 --2- 172.21.15.96:0/712048721 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f46600051c0 0x7f466000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:05.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.560+0000 7f467a000640 1 -- 172.21.15.96:0/712048721 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f466000c130 con 0x7f4660004800 2024-05-28T06:09:05.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.560+0000 7f467a000640 1 -- 172.21.15.96:0/712048721 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f466000c2e0 con 0x7f46600039b0 2024-05-28T06:09:05.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.560+0000 7f467a000640 1 -- 172.21.15.96:0/712048721 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f466000c4e0 con 0x7f46600051c0 2024-05-28T06:09:05.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.560+0000 7f4673e00640 1 --2- 172.21.15.96:0/712048721 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4660004800 0x7f4660004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:05.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.560+0000 7f467b400640 1 --2- 172.21.15.96:0/712048721 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f46600051c0 0x7f466000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:05.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.560+0000 7f467aa00640 1 --2- 172.21.15.96:0/712048721 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f46600039b0 0x7f4660003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:05.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.560+0000 7f4673e00640 1 -- 172.21.15.96:0/712048721 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f46600051c0 msgr2=0x7f466000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:09:05.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.560+0000 7f4673e00640 1 --2- 172.21.15.96:0/712048721 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f46600051c0 0x7f466000b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:05.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.560+0000 7f4673e00640 1 -- 172.21.15.96:0/712048721 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f46600039b0 msgr2=0x7f4660003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:09:05.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.560+0000 7f4673e00640 1 --2- 172.21.15.96:0/712048721 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f46600039b0 0x7f4660003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:05.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.560+0000 7f4673e00640 1 -- 172.21.15.96:0/712048721 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4660003520 con 0x7f4660004800 2024-05-28T06:09:05.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.560+0000 7f467aa00640 1 --2- 172.21.15.96:0/712048721 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f46600039b0 0x7f4660003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:09:05.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.560+0000 7f467b400640 1 -- 172.21.15.96:0/712048721 reap_dead start 2024-05-28T06:09:05.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.561+0000 7f4673e00640 1 --2- 172.21.15.96:0/712048721 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4660004800 0x7f4660004c80 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f466400c240 tx=0x7f4660005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c6d357279ec5f97 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:09:05.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.562+0000 7f467a000640 1 -- 172.21.15.96:0/712048721 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4664007d10 con 0x7f4660004800 2024-05-28T06:09:05.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.562+0000 7f467a000640 1 -- 172.21.15.96:0/712048721 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4664005ad0 con 0x7f4660004800 2024-05-28T06:09:05.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.562+0000 7f467a000640 1 -- 172.21.15.96:0/712048721 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4664007070 con 0x7f4660004800 2024-05-28T06:09:05.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.562+0000 7f4681e00640 1 -- 172.21.15.96:0/712048721 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4660004800 msgr2=0x7f4660004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:09:05.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.562+0000 7f4681e00640 1 --2- 172.21.15.96:0/712048721 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4660004800 0x7f4660004c80 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f466400c240 tx=0x7f4660005830 comp rx=0 tx=0).stop 2024-05-28T06:09:05.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.562+0000 7f4681e00640 1 -- 172.21.15.96:0/712048721 shutdown_connections 2024-05-28T06:09:05.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.562+0000 7f4681e00640 1 --2- 172.21.15.96:0/712048721 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4660004800 0x7f4660004c80 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:05.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.562+0000 7f4681e00640 1 -- 172.21.15.96:0/712048721 >> 172.21.15.96:0/712048721 conn(0x7f467c06bd20 msgr2=0x7f467c06c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:09:05.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.562+0000 7f4681e00640 1 -- 172.21.15.96:0/712048721 shutdown_connections 2024-05-28T06:09:05.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.562+0000 7f4681e00640 1 -- 172.21.15.96:0/712048721 wait complete. 2024-05-28T06:09:05.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.564+0000 7f4681e00640 1 Processor -- start 2024-05-28T06:09:05.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.564+0000 7f4681e00640 1 -- start start 2024-05-28T06:09:05.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.565+0000 7f4681e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4660004800 0x7f467c0b1b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:05.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.565+0000 7f4681e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f467c14c020 0x7f467c0b04b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:05.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.565+0000 7f4681e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f467c0b09f0 0x7f467c0b0e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:05.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.565+0000 7f4681e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f467c15d0b0 con 0x7f4660004800 2024-05-28T06:09:05.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.565+0000 7f4681e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f467c15cdd0 con 0x7f467c0b09f0 2024-05-28T06:09:05.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.565+0000 7f4681e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f467c15da50 con 0x7f467c14c020 2024-05-28T06:09:05.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.566+0000 7f467aa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4660004800 0x7f467c0b1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:05.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.566+0000 7f467aa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4660004800 0x7f467c0b1b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47590/0 (socket says 172.21.15.96:47590) 2024-05-28T06:09:05.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.566+0000 7f467aa00640 1 -- 172.21.15.96:0/1061202679 learned_addr learned my addr 172.21.15.96:0/1061202679 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:09:05.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.566+0000 7f4673e00640 1 --2- 172.21.15.96:0/1061202679 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f467c14c020 0x7f467c0b04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:05.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.566+0000 7f467aa00640 1 -- 172.21.15.96:0/1061202679 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f467c14c020 msgr2=0x7f467c0b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:09:05.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.566+0000 7f467aa00640 1 --2- 172.21.15.96:0/1061202679 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f467c14c020 0x7f467c0b04b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:05.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.566+0000 7f467aa00640 1 -- 172.21.15.96:0/1061202679 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f467c0b09f0 msgr2=0x7f467c0b0e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:09:05.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.566+0000 7f467aa00640 1 --2- 172.21.15.96:0/1061202679 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f467c0b09f0 0x7f467c0b0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:05.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.566+0000 7f467aa00640 1 -- 172.21.15.96:0/1061202679 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f467c14d0b0 con 0x7f4660004800 2024-05-28T06:09:05.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.566+0000 7f467aa00640 1 --2- 172.21.15.96:0/1061202679 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4660004800 0x7f467c0b1b00 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f466c00d480 tx=0x7f466c00d940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:09:05.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.567+0000 7f4673400640 1 -- 172.21.15.96:0/1061202679 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f466c018070 con 0x7f4660004800 2024-05-28T06:09:05.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.567+0000 7f4673400640 1 -- 172.21.15.96:0/1061202679 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f466c003610 con 0x7f4660004800 2024-05-28T06:09:05.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.567+0000 7f4681e00640 1 -- 172.21.15.96:0/1061202679 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f467c14b560 con 0x7f4660004800 2024-05-28T06:09:05.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.567+0000 7f4673400640 1 -- 172.21.15.96:0/1061202679 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f466c003a40 con 0x7f4660004800 2024-05-28T06:09:05.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.567+0000 7f4681e00640 1 -- 172.21.15.96:0/1061202679 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f467c14da30 con 0x7f4660004800 2024-05-28T06:09:05.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.568+0000 7f4681e00640 1 -- 172.21.15.96:0/1061202679 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f467c0689e0 con 0x7f4660004800 2024-05-28T06:09:05.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.569+0000 7f4673400640 1 -- 172.21.15.96:0/1061202679 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 9) ==== 91295+0+0 (secure 0 0 0) 0x7f466c01b070 con 0x7f4660004800 2024-05-28T06:09:05.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.570+0000 7f4673400640 1 --2- 172.21.15.96:0/1061202679 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f465806cbc0 0x7f465806f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:05.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.570+0000 7f4673400640 1 -- 172.21.15.96:0/1061202679 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5737+0+0 (secure 0 0 0) 0x7f466c08b800 con 0x7f4660004800 2024-05-28T06:09:05.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.571+0000 7f4673e00640 1 --2- 172.21.15.96:0/1061202679 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f465806cbc0 0x7f465806f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:05.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.572+0000 7f4673e00640 1 --2- 172.21.15.96:0/1061202679 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f465806cbc0 0x7f465806f080 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f4664032000 tx=0x7f4664034040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:09:05.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.575+0000 7f4673400640 1 -- 172.21.15.96:0/1061202679 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f466c059270 con 0x7f4660004800 2024-05-28T06:09:05.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.854+0000 7f4681e00640 1 -- 172.21.15.96:0/1061202679 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f467c071ef0 con 0x7f4660004800 2024-05-28T06:09:05.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.859+0000 7f4673400640 1 -- 172.21.15.96:0/1061202679 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v14) ==== 190+0+147 (secure 0 0 0) 0x7f466c05d570 con 0x7f4660004800 2024-05-28T06:09:05.861 INFO:teuthology.orchestra.run.smithi096.stdout:[client.0] 2024-05-28T06:09:05.861 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQCXc1Vm91hbGBAASTL3HQR8XOs7wVQEHbPl/g== 2024-05-28T06:09:05.861 INFO:teuthology.orchestra.run.smithi096.stdout: caps mds = "allow" 2024-05-28T06:09:05.861 INFO:teuthology.orchestra.run.smithi096.stdout: caps mon = "allow r" 2024-05-28T06:09:05.861 INFO:teuthology.orchestra.run.smithi096.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-05-28T06:09:05.862 INFO:teuthology.orchestra.run.smithi096.stderr:updated caps for client.0 2024-05-28T06:09:05.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.862+0000 7f4681e00640 1 -- 172.21.15.96:0/1061202679 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f465806cbc0 msgr2=0x7f465806f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:09:05.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.862+0000 7f4681e00640 1 --2- 172.21.15.96:0/1061202679 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f465806cbc0 0x7f465806f080 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f4664032000 tx=0x7f4664034040 comp rx=0 tx=0).stop 2024-05-28T06:09:05.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.862+0000 7f4681e00640 1 -- 172.21.15.96:0/1061202679 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4660004800 msgr2=0x7f467c0b1b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:09:05.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.862+0000 7f4681e00640 1 --2- 172.21.15.96:0/1061202679 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4660004800 0x7f467c0b1b00 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f466c00d480 tx=0x7f466c00d940 comp rx=0 tx=0).stop 2024-05-28T06:09:05.865 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.863+0000 7f4681e00640 1 -- 172.21.15.96:0/1061202679 shutdown_connections 2024-05-28T06:09:05.865 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.863+0000 7f4681e00640 1 --2- 172.21.15.96:0/1061202679 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f465806cbc0 0x7f465806f080 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:05.865 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.863+0000 7f4681e00640 1 --2- 172.21.15.96:0/1061202679 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f467c0b09f0 0x7f467c0b0e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:05.865 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.863+0000 7f4681e00640 1 --2- 172.21.15.96:0/1061202679 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f467c14c020 0x7f467c0b04b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:05.865 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.863+0000 7f4681e00640 1 --2- 172.21.15.96:0/1061202679 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4660004800 0x7f467c0b1b00 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:05.865 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.863+0000 7f4681e00640 1 -- 172.21.15.96:0/1061202679 >> 172.21.15.96:0/1061202679 conn(0x7f467c06bd20 msgr2=0x7f467c0af250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:09:05.865 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.863+0000 7f4681e00640 1 -- 172.21.15.96:0/1061202679 shutdown_connections 2024-05-28T06:09:05.865 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:05.863+0000 7f4681e00640 1 -- 172.21.15.96:0/1061202679 wait complete. 2024-05-28T06:09:05.883 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-28T06:09:06.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.072+0000 7f1ac4e00640 1 Processor -- start 2024-05-28T06:09:06.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.072+0000 7f1ac4e00640 1 -- start start 2024-05-28T06:09:06.075 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.074+0000 7f1ac4e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1ac00d04a0 0x7f1ac00d0900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:06.075 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.074+0000 7f1ac4e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f1ac00ad130 0x7f1ac00ad5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:06.075 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.074+0000 7f1ac4e00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ac006ed10 con 0x7f1ac006f860 2024-05-28T06:09:06.075 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.074+0000 7f1ac4e00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f1ac00d0e40 con 0x7f1ac00ad130 2024-05-28T06:09:06.075 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.074+0000 7f1ac4e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f1ac015cb00 con 0x7f1ac00d04a0 2024-05-28T06:09:06.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.074+0000 7f1abe800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f1ac00ad130 0x7f1ac00ad5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:06.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.074+0000 7f1abe800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f1ac00ad130 0x7f1ac00ad5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47592/0 (socket says 172.21.15.96:47592) 2024-05-28T06:09:06.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.074+0000 7f1abe800640 1 -- 172.21.15.96:0/1752488608 learned_addr learned my addr 172.21.15.96:0/1752488608 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:09:06.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.075+0000 7f1ab7e00640 1 --2- 172.21.15.96:0/1752488608 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1ac00d04a0 0x7f1ac00d0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:06.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.075+0000 7f1abe800640 1 -- 172.21.15.96:0/1752488608 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1ac00d04a0 msgr2=0x7f1ac00d0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:09:06.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.075+0000 7f1abe800640 1 --2- 172.21.15.96:0/1752488608 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1ac00d04a0 0x7f1ac00d0900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:06.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.075+0000 7f1abe800640 1 -- 172.21.15.96:0/1752488608 >> v1:172.21.15.123:6789/0 conn(0x7f1ac006f860 legacy=0x7f1ac00b3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:09:06.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.075+0000 7f1abe800640 1 -- 172.21.15.96:0/1752488608 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ac00b3cc0 con 0x7f1ac00ad130 2024-05-28T06:09:06.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.075+0000 7f1ab7e00640 1 --2- 172.21.15.96:0/1752488608 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1ac00d04a0 0x7f1ac00d0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:09:06.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.076+0000 7f1abe800640 1 --2- 172.21.15.96:0/1752488608 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1ac00ad130 0x7f1ac00ad5a0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f1ab8009e50 tx=0x7f1ab802a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3484287ccf29eff server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:09:06.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.076+0000 7f1abd400640 1 -- 172.21.15.96:0/1752488608 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ab8036070 con 0x7f1ac00ad130 2024-05-28T06:09:06.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.076+0000 7f1abd400640 1 -- 172.21.15.96:0/1752488608 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ab8031600 con 0x7f1ac00ad130 2024-05-28T06:09:06.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.076+0000 7f1abd400640 1 -- 172.21.15.96:0/1752488608 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ab80319d0 con 0x7f1ac00ad130 2024-05-28T06:09:06.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.077+0000 7f1ac4e00640 1 -- 172.21.15.96:0/1752488608 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1ac00ad130 msgr2=0x7f1ac00ad5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:09:06.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.077+0000 7f1ac4e00640 1 --2- 172.21.15.96:0/1752488608 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1ac00ad130 0x7f1ac00ad5a0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f1ab8009e50 tx=0x7f1ab802a6c0 comp rx=0 tx=0).stop 2024-05-28T06:09:06.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.077+0000 7f1ac4e00640 1 -- 172.21.15.96:0/1752488608 shutdown_connections 2024-05-28T06:09:06.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.077+0000 7f1ac4e00640 1 --2- 172.21.15.96:0/1752488608 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1ac00ad130 0x7f1ac00ad5a0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:06.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.077+0000 7f1ac4e00640 1 --2- 172.21.15.96:0/1752488608 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1ac00d04a0 0x7f1ac00d0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:06.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.077+0000 7f1ac4e00640 1 -- 172.21.15.96:0/1752488608 >> 172.21.15.96:0/1752488608 conn(0x7f1ac006bae0 msgr2=0x7f1ac006bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:09:06.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.077+0000 7f1ac4e00640 1 -- 172.21.15.96:0/1752488608 shutdown_connections 2024-05-28T06:09:06.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.077+0000 7f1ac4e00640 1 -- 172.21.15.96:0/1752488608 wait complete. 2024-05-28T06:09:06.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.079+0000 7f1ac4e00640 1 Processor -- start 2024-05-28T06:09:06.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.079+0000 7f1ac4e00640 1 -- start start 2024-05-28T06:09:06.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.080+0000 7f1ac4e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1ac006f860 0x7f1ac015f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:06.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.080+0000 7f1ac4e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1ac00ad130 0x7f1ac01f3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:06.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.080+0000 7f1ac4e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ac00d04a0 0x7f1ac015e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:06.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.080+0000 7f1ac4e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f1ac00b4560 con 0x7f1ac00ad130 2024-05-28T06:09:06.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.080+0000 7f1ac4e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f1ac00b4280 con 0x7f1ac00d04a0 2024-05-28T06:09:06.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.080+0000 7f1ac4e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f1ac00b4840 con 0x7f1ac006f860 2024-05-28T06:09:06.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.080+0000 7f1ab7e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1ac00ad130 0x7f1ac01f3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:06.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.080+0000 7f1ab7e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1ac00ad130 0x7f1ac01f3ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47606/0 (socket says 172.21.15.96:47606) 2024-05-28T06:09:06.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.080+0000 7f1ab7e00640 1 -- 172.21.15.96:0/2199343915 learned_addr learned my addr 172.21.15.96:0/2199343915 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:09:06.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.081+0000 7f1abe800640 1 --2- 172.21.15.96:0/2199343915 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ac00d04a0 0x7f1ac015e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:06.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.081+0000 7f1ab7e00640 1 -- 172.21.15.96:0/2199343915 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1ac006f860 msgr2=0x7f1ac015f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:09:06.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.081+0000 7f1abde00640 1 --2- 172.21.15.96:0/2199343915 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1ac006f860 0x7f1ac015f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:06.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.081+0000 7f1ab7e00640 1 --2- 172.21.15.96:0/2199343915 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1ac006f860 0x7f1ac015f580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:06.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.081+0000 7f1ab7e00640 1 -- 172.21.15.96:0/2199343915 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ac00d04a0 msgr2=0x7f1ac015e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:09:06.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.081+0000 7f1ab7e00640 1 --2- 172.21.15.96:0/2199343915 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ac00d04a0 0x7f1ac015e810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:06.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.081+0000 7f1ab7e00640 1 -- 172.21.15.96:0/2199343915 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ac00c73d0 con 0x7f1ac00ad130 2024-05-28T06:09:06.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.081+0000 7f1abe800640 1 --2- 172.21.15.96:0/2199343915 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ac00d04a0 0x7f1ac015e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:09:06.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.081+0000 7f1ab7e00640 1 --2- 172.21.15.96:0/2199343915 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1ac00ad130 0x7f1ac01f3ff0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f1aa800c7f0 tx=0x7f1aa800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:09:06.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.081+0000 7f1ab6000640 1 -- 172.21.15.96:0/2199343915 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aa8015070 con 0x7f1ac00ad130 2024-05-28T06:09:06.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.082+0000 7f1ab6000640 1 -- 172.21.15.96:0/2199343915 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1aa8007a30 con 0x7f1ac00ad130 2024-05-28T06:09:06.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.082+0000 7f1ac4e00640 1 -- 172.21.15.96:0/2199343915 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ac00c9e20 con 0x7f1ac00ad130 2024-05-28T06:09:06.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.082+0000 7f1ab6000640 1 -- 172.21.15.96:0/2199343915 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aa8007e00 con 0x7f1ac00ad130 2024-05-28T06:09:06.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.082+0000 7f1ac4e00640 1 -- 172.21.15.96:0/2199343915 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ac015fac0 con 0x7f1ac00ad130 2024-05-28T06:09:06.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.083+0000 7f1ac4e00640 1 -- 172.21.15.96:0/2199343915 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ac00c7a50 con 0x7f1ac00ad130 2024-05-28T06:09:06.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.084+0000 7f1ab6000640 1 -- 172.21.15.96:0/2199343915 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 9) ==== 91295+0+0 (secure 0 0 0) 0x7f1aa8018070 con 0x7f1ac00ad130 2024-05-28T06:09:06.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.085+0000 7f1ab6000640 1 --2- 172.21.15.96:0/2199343915 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1a9c06cbc0 0x7f1a9c06f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:06.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.085+0000 7f1ab6000640 1 -- 172.21.15.96:0/2199343915 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5737+0+0 (secure 0 0 0) 0x7f1aa808c060 con 0x7f1ac00ad130 2024-05-28T06:09:06.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.085+0000 7f1abde00640 1 --2- 172.21.15.96:0/2199343915 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1a9c06cbc0 0x7f1a9c06f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:06.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.087+0000 7f1abde00640 1 --2- 172.21.15.96:0/2199343915 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1a9c06cbc0 0x7f1a9c06f080 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f1ab000a750 tx=0x7f1ab0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:09:06.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.089+0000 7f1ab6000640 1 -- 172.21.15.96:0/2199343915 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1ac00c7a50 con 0x7f1ac00ad130 2024-05-28T06:09:06.331 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.329+0000 7f1ac4e00640 1 -- 172.21.15.96:0/2199343915 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1ac006e2f0 con 0x7f1ac00ad130 2024-05-28T06:09:06.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.331+0000 7f1ab6000640 1 -- 172.21.15.96:0/2199343915 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+5181 (secure 0 0 0) 0x7f1aa805a060 con 0x7f1ac00ad130 2024-05-28T06:09:06.333 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:09:06.333 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/1693172076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":1693172076},{"type":"v1","addr":"172.21.15.123:6835","nonce":1693172076}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":4805,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/1098611268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":1098611268},{"type":"v1","addr":"172.21.15.123:6837","nonce":1098611268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5856,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6834/673165997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":673165997},{"type":"v1","addr":"172.21.15.96:6834","nonce":673165997}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-28T06:08:54.477674+0000","modified":"2024-05-28T06:08:57.572929+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5859},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5859":{"gid":5859,"name":"a","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.96:6835/935552515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6833","nonce":935552515},{"type":"v1","addr":"172.21.15.96:6835","nonce":935552515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5859,"qdb_cluster":[5859]},"id":3}]} 2024-05-28T06:09:06.333 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 25 2024-05-28T06:09:06.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.333+0000 7f1ac4e00640 1 -- 172.21.15.96:0/2199343915 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1a9c06cbc0 msgr2=0x7f1a9c06f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:09:06.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.333+0000 7f1ac4e00640 1 --2- 172.21.15.96:0/2199343915 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1a9c06cbc0 0x7f1a9c06f080 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f1ab000a750 tx=0x7f1ab0008040 comp rx=0 tx=0).stop 2024-05-28T06:09:06.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.333+0000 7f1ac4e00640 1 -- 172.21.15.96:0/2199343915 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1ac00ad130 msgr2=0x7f1ac01f3ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:09:06.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.333+0000 7f1ac4e00640 1 --2- 172.21.15.96:0/2199343915 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1ac00ad130 0x7f1ac01f3ff0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f1aa800c7f0 tx=0x7f1aa800cbf0 comp rx=0 tx=0).stop 2024-05-28T06:09:06.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.333+0000 7f1ac4e00640 1 -- 172.21.15.96:0/2199343915 shutdown_connections 2024-05-28T06:09:06.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.333+0000 7f1ac4e00640 1 --2- 172.21.15.96:0/2199343915 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1a9c06cbc0 0x7f1a9c06f080 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:06.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.333+0000 7f1ac4e00640 1 --2- 172.21.15.96:0/2199343915 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ac00d04a0 0x7f1ac015e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:06.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.333+0000 7f1ac4e00640 1 --2- 172.21.15.96:0/2199343915 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1ac00ad130 0x7f1ac01f3ff0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:06.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.333+0000 7f1ac4e00640 1 --2- 172.21.15.96:0/2199343915 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1ac006f860 0x7f1ac015f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:06.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.333+0000 7f1ac4e00640 1 -- 172.21.15.96:0/2199343915 >> 172.21.15.96:0/2199343915 conn(0x7f1ac006bae0 msgr2=0x7f1ac0071b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:09:06.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.334+0000 7f1ac4e00640 1 -- 172.21.15.96:0/2199343915 shutdown_connections 2024-05-28T06:09:06.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.334+0000 7f1ac4e00640 1 -- 172.21.15.96:0/2199343915 wait complete. 2024-05-28T06:09:06.756 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-05-28T06:08:54.477674+0000', 'modified': '2024-05-28T06:08:57.572929+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5859}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5859': {'gid': 5859, 'name': 'a', 'rank': 0, 'incarnation': 21, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.96:6835/935552515', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.96:6833', 'nonce': 935552515}, {'type': 'v1', 'addr': '172.21.15.96:6835', 'nonce': 935552515}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5859, 'qdb_cluster': [5859]} 2024-05-28T06:09:06.757 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-05-28T06:09:06.757 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-05-28T06:09:06.967 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.966+0000 7f9b10a00640 1 Processor -- start 2024-05-28T06:09:06.968 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.966+0000 7f9b10a00640 1 -- start start 2024-05-28T06:09:06.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.967+0000 7f9b10a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f9b0c15e870 0x7f9b0c0d0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:06.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.967+0000 7f9b10a00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b0c15ec50 con 0x7f9b0c0d14d0 2024-05-28T06:09:06.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.967+0000 7f9b10a00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b0c0cff80 con 0x7f9b0c0cfa40 2024-05-28T06:09:06.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.967+0000 7f9b10a00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f9b0c0d2150 con 0x7f9b0c15e870 2024-05-28T06:09:06.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.968+0000 7f9b0b400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f9b0c15e870 0x7f9b0c0d0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:06.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.968+0000 7f9b03e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f9b0c0d14d0 0x7f9b0c0cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:51204/0 (socket says 172.21.15.96:51204) 2024-05-28T06:09:06.970 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.968+0000 7f9b03e00640 1 -- 172.21.15.96:0/438758413 learned_addr learned my addr 172.21.15.96:0/438758413 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:09:06.970 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.968+0000 7f9b0b400640 1 -- 172.21.15.96:0/438758413 >> v1:172.21.15.123:6789/0 conn(0x7f9b0c0cfa40 legacy=0x7f9b0c0d25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:09:06.970 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.968+0000 7f9b0b400640 1 -- 172.21.15.96:0/438758413 >> v1:172.21.15.96:6789/0 conn(0x7f9b0c0d14d0 legacy=0x7f9b0c0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:09:06.970 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.968+0000 7f9b0b400640 1 -- 172.21.15.96:0/438758413 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b0c0d0260 con 0x7f9b0c15e870 2024-05-28T06:09:06.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.969+0000 7f9b0b400640 1 --2- 172.21.15.96:0/438758413 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9b0c15e870 0x7f9b0c0d0f90 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f9afc0097a0 tx=0x7f9afc02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=195a7ba7216ed094 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:09:06.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.970+0000 7f9b0aa00640 1 -- 172.21.15.96:0/438758413 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9afc037070 con 0x7f9b0c15e870 2024-05-28T06:09:06.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.970+0000 7f9b0aa00640 1 -- 172.21.15.96:0/438758413 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9afc032440 con 0x7f9b0c15e870 2024-05-28T06:09:06.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.970+0000 7f9b0aa00640 1 -- 172.21.15.96:0/438758413 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9afc032810 con 0x7f9b0c15e870 2024-05-28T06:09:06.972 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.970+0000 7f9b10a00640 1 -- 172.21.15.96:0/438758413 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9b0c15e870 msgr2=0x7f9b0c0d0f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:09:06.972 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.970+0000 7f9b10a00640 1 --2- 172.21.15.96:0/438758413 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9b0c15e870 0x7f9b0c0d0f90 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f9afc0097a0 tx=0x7f9afc02a1e0 comp rx=0 tx=0).stop 2024-05-28T06:09:06.972 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.970+0000 7f9b10a00640 1 -- 172.21.15.96:0/438758413 shutdown_connections 2024-05-28T06:09:06.972 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.970+0000 7f9b10a00640 1 --2- 172.21.15.96:0/438758413 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9b0c15e870 0x7f9b0c0d0f90 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:06.972 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.970+0000 7f9b10a00640 1 -- 172.21.15.96:0/438758413 >> 172.21.15.96:0/438758413 conn(0x7f9b0c08bfc0 msgr2=0x7f9b0c0d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:09:06.972 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.970+0000 7f9b10a00640 1 -- 172.21.15.96:0/438758413 shutdown_connections 2024-05-28T06:09:06.972 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.971+0000 7f9b10a00640 1 -- 172.21.15.96:0/438758413 wait complete. 2024-05-28T06:09:06.973 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.972+0000 7f9b10a00640 1 Processor -- start 2024-05-28T06:09:06.973 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.972+0000 7f9b10a00640 1 -- start start 2024-05-28T06:09:06.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.973+0000 7f9b10a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9b0c0cfa40 0x7f9b0c0ca500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:06.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.973+0000 7f9b10a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9b0c0d14d0 0x7f9b0c0cd100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:06.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.973+0000 7f9b10a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9b0c15e870 0x7f9b0c0cd640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:06.976 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.973+0000 7f9b10a00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f9b0c0d4fb0 con 0x7f9b0c15e870 2024-05-28T06:09:06.976 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.973+0000 7f9b10a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f9b0c0d0820 con 0x7f9b0c0cfa40 2024-05-28T06:09:06.976 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.973+0000 7f9b10a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f9b0c0d2cf0 con 0x7f9b0c0d14d0 2024-05-28T06:09:06.976 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.974+0000 7f9b0be00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9b0c15e870 0x7f9b0c0cd640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:06.976 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.974+0000 7f9b0be00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9b0c15e870 0x7f9b0c0cd640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47618/0 (socket says 172.21.15.96:47618) 2024-05-28T06:09:06.976 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.974+0000 7f9b0be00640 1 -- 172.21.15.96:0/2716655156 learned_addr learned my addr 172.21.15.96:0/2716655156 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:09:06.976 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.974+0000 7f9b0b400640 1 --2- 172.21.15.96:0/2716655156 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9b0c0cfa40 0x7f9b0c0ca500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:06.976 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.974+0000 7f9b03e00640 1 --2- 172.21.15.96:0/2716655156 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9b0c0d14d0 0x7f9b0c0cd100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:06.976 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.974+0000 7f9b0be00640 1 -- 172.21.15.96:0/2716655156 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9b0c0d14d0 msgr2=0x7f9b0c0cd100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:09:06.976 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.974+0000 7f9b0be00640 1 --2- 172.21.15.96:0/2716655156 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9b0c0d14d0 0x7f9b0c0cd100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:06.976 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.974+0000 7f9b0be00640 1 -- 172.21.15.96:0/2716655156 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9b0c0cfa40 msgr2=0x7f9b0c0ca500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:09:06.976 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.974+0000 7f9b0be00640 1 --2- 172.21.15.96:0/2716655156 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9b0c0cfa40 0x7f9b0c0ca500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:06.976 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.974+0000 7f9b0be00640 1 -- 172.21.15.96:0/2716655156 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b0c0cb650 con 0x7f9b0c15e870 2024-05-28T06:09:06.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.974+0000 7f9b03e00640 1 --2- 172.21.15.96:0/2716655156 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9b0c0d14d0 0x7f9b0c0cd100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:09:06.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.974+0000 7f9b0b400640 1 --2- 172.21.15.96:0/2716655156 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9b0c0cfa40 0x7f9b0c0ca500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-28T06:09:06.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.974+0000 7f9b0be00640 1 --2- 172.21.15.96:0/2716655156 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9b0c15e870 0x7f9b0c0cd640 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f9b0400c700 tx=0x7f9b0400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:09:06.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.975+0000 7f9b03400640 1 -- 172.21.15.96:0/2716655156 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b04016020 con 0x7f9b0c15e870 2024-05-28T06:09:06.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.975+0000 7f9b03400640 1 -- 172.21.15.96:0/2716655156 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b0400aa50 con 0x7f9b0c15e870 2024-05-28T06:09:06.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.975+0000 7f9b03400640 1 -- 172.21.15.96:0/2716655156 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b0400ce10 con 0x7f9b0c15e870 2024-05-28T06:09:06.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.975+0000 7f9b10a00640 1 -- 172.21.15.96:0/2716655156 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b0c0cb8f0 con 0x7f9b0c15e870 2024-05-28T06:09:06.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.975+0000 7f9b10a00640 1 -- 172.21.15.96:0/2716655156 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b0c0cbeb0 con 0x7f9b0c15e870 2024-05-28T06:09:06.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.977+0000 7f9b03400640 1 -- 172.21.15.96:0/2716655156 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 9) ==== 91295+0+0 (secure 0 0 0) 0x7f9b04019020 con 0x7f9b0c15e870 2024-05-28T06:09:06.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.978+0000 7f9b03400640 1 --2- 172.21.15.96:0/2716655156 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9ae806cbf0 0x7f9ae806f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:06.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.978+0000 7f9b03400640 1 -- 172.21.15.96:0/2716655156 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5737+0+0 (secure 0 0 0) 0x7f9b0408c2b0 con 0x7f9b0c15e870 2024-05-28T06:09:06.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.978+0000 7f9b0b400640 1 --2- 172.21.15.96:0/2716655156 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9ae806cbf0 0x7f9ae806f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:06.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.979+0000 7f9b14deb740 1 Processor -- start 2024-05-28T06:09:06.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.979+0000 7f9b14deb740 1 -- start start 2024-05-28T06:09:06.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.980+0000 7f9b14deb740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56143508a490 0x56143508dc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:06.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.980+0000 7f9b14deb740 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x56143508e150 0x5614350aa090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:06.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.980+0000 7f9b14deb740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5614350aa5d0 0x5614350aaa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:06.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.980+0000 7f9b14deb740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x561434fc7b10 con 0x56143508a490 2024-05-28T06:09:06.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.980+0000 7f9b14deb740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x561434f6e2b0 con 0x5614350aa5d0 2024-05-28T06:09:06.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.980+0000 7f9b14deb740 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x561434fc3ad0 con 0x56143508e150 2024-05-28T06:09:06.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.980+0000 7f9b0b400640 1 --2- 172.21.15.96:0/2716655156 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9ae806cbf0 0x7f9ae806f0b0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f9afc034000 tx=0x7f9afc02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:09:06.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.980+0000 7f9b0b400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56143508a490 0x56143508dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:06.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.980+0000 7f9b0b400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56143508a490 0x56143508dc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47632/0 (socket says 172.21.15.96:47632) 2024-05-28T06:09:06.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.980+0000 7f9b0b400640 1 -- 172.21.15.96:0/1501707485 learned_addr learned my addr 172.21.15.96:0/1501707485 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:09:06.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.980+0000 7f9b0be00640 1 --2- 172.21.15.96:0/1501707485 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5614350aa5d0 0x5614350aaa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:06.986 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.980+0000 7f9b03e00640 1 --2- 172.21.15.96:0/1501707485 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x56143508e150 0x5614350aa090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:06.986 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.980+0000 7f9b0b400640 1 -- 172.21.15.96:0/1501707485 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x56143508e150 msgr2=0x5614350aa090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:09:06.986 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.980+0000 7f9b0b400640 1 --2- 172.21.15.96:0/1501707485 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x56143508e150 0x5614350aa090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:06.986 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.980+0000 7f9b0b400640 1 -- 172.21.15.96:0/1501707485 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5614350aa5d0 msgr2=0x5614350aaa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:09:06.986 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.980+0000 7f9b0b400640 1 --2- 172.21.15.96:0/1501707485 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5614350aa5d0 0x5614350aaa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:06.986 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.980+0000 7f9b0b400640 1 -- 172.21.15.96:0/1501707485 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561434cda2d0 con 0x56143508a490 2024-05-28T06:09:06.986 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.981+0000 7f9b0b400640 1 --2- 172.21.15.96:0/1501707485 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56143508a490 0x56143508dc10 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f9afc02a7d0 tx=0x7f9afc02c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f49dc1996faf4fec server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:09:06.986 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.981+0000 7f9afb400640 1 -- 172.21.15.96:0/1501707485 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9afc037040 con 0x56143508a490 2024-05-28T06:09:06.986 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.981+0000 7f9afb400640 1 -- 172.21.15.96:0/1501707485 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9afc03e460 con 0x56143508a490 2024-05-28T06:09:06.986 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.981+0000 7f9afb400640 1 -- 172.21.15.96:0/1501707485 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9afc03e810 con 0x56143508a490 2024-05-28T06:09:06.986 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.981+0000 7f9b14deb740 1 -- 172.21.15.96:0/1501707485 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56143508a490 msgr2=0x56143508dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:09:06.986 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.981+0000 7f9b14deb740 1 --2- 172.21.15.96:0/1501707485 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56143508a490 0x56143508dc10 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f9afc02a7d0 tx=0x7f9afc02c000 comp rx=0 tx=0).stop 2024-05-28T06:09:06.986 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.982+0000 7f9b14deb740 1 -- 172.21.15.96:0/1501707485 shutdown_connections 2024-05-28T06:09:06.987 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.982+0000 7f9b14deb740 1 --2- 172.21.15.96:0/1501707485 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5614350aa5d0 0x5614350aaa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:06.987 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.982+0000 7f9b14deb740 1 --2- 172.21.15.96:0/1501707485 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x56143508e150 0x5614350aa090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:06.987 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.982+0000 7f9b14deb740 1 --2- 172.21.15.96:0/1501707485 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56143508a490 0x56143508dc10 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:06.987 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.982+0000 7f9b14deb740 1 -- 172.21.15.96:0/1501707485 >> 172.21.15.96:0/1501707485 conn(0x561435060470 msgr2=0x561435061470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:09:06.987 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.982+0000 7f9b14deb740 1 -- 172.21.15.96:0/1501707485 shutdown_connections 2024-05-28T06:09:06.987 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.982+0000 7f9b14deb740 1 -- 172.21.15.96:0/1501707485 wait complete. 2024-05-28T06:09:06.987 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.982+0000 7f9b14deb740 1 -- start start 2024-05-28T06:09:06.987 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.983+0000 7f9b14deb740 1 Processor -- start 2024-05-28T06:09:06.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.984+0000 7f9b14deb740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56143508e150 0x5614350b92b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:06.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.984+0000 7f9b14deb740 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x5614350aa5d0 0x56143505d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:06.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.984+0000 7f9b14deb740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5614350860c0 0x56143505d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:06.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.984+0000 7f9b14deb740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x56143505e2a0 con 0x56143508e150 2024-05-28T06:09:06.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.984+0000 7f9b14deb740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x561434f81570 con 0x5614350860c0 2024-05-28T06:09:06.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.984+0000 7f9b14deb740 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x56143505e580 con 0x5614350aa5d0 2024-05-28T06:09:06.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.984+0000 7f9b14deb740 10 client.0 _ll_register_callbacks cb 0x561434f4ffc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-28T06:09:06.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.984+0000 7f9b0b400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56143508e150 0x5614350b92b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:06.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.984+0000 7f9b0b400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56143508e150 0x5614350b92b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47634/0 (socket says 172.21.15.96:47634) 2024-05-28T06:09:06.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.984+0000 7f9b0b400640 1 -- 172.21.15.96:0/3248039156 learned_addr learned my addr 172.21.15.96:0/3248039156 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:09:06.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.984+0000 7f9b0b400640 1 -- 172.21.15.96:0/3248039156 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x5614350aa5d0 msgr2=0x56143505d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:09:06.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.984+0000 7f9b0b400640 1 --2- 172.21.15.96:0/3248039156 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x5614350aa5d0 0x56143505d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:06.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.984+0000 7f9b0b400640 1 -- 172.21.15.96:0/3248039156 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5614350860c0 msgr2=0x56143505d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:09:06.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.984+0000 7f9b0b400640 1 --2- 172.21.15.96:0/3248039156 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5614350860c0 0x56143505d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:06.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.984+0000 7f9b0b400640 1 -- 172.21.15.96:0/3248039156 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561434da46b0 con 0x56143508e150 2024-05-28T06:09:06.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.985+0000 7f9b0b400640 1 --2- 172.21.15.96:0/3248039156 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56143508e150 0x5614350b92b0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f9afc02a7d0 tx=0x7f9afc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=156a461292941077 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:09:06.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.985+0000 7f9aefe00640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-05-28T06:09:06.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.985+0000 7f9aefe00640 1 -- 172.21.15.96:0/3248039156 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9afc03ea10 con 0x56143508e150 2024-05-28T06:09:06.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.985+0000 7f9aefe00640 1 -- 172.21.15.96:0/3248039156 <== mon.0 v2:172.21.15.96:3300/0 2 ==== osd_map(45..45 src has 1..45) ==== 5737+0+0 (secure 0 0 0) 0x7f9afc03ebb0 con 0x56143508e150 2024-05-28T06:09:06.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.985+0000 7f9b14deb740 1 -- 172.21.15.96:0/3248039156 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9afc03e460 con 0x56143508e150 2024-05-28T06:09:06.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.985+0000 7f9aefe00640 1 -- 172.21.15.96:0/3248039156 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f9afc03edb0 con 0x56143508e150 2024-05-28T06:09:06.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.985+0000 7f9b14deb740 10 client.6036 fetch_fsmap learned FSMap version 25 2024-05-28T06:09:06.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.985+0000 7f9b14deb740 1 -- 172.21.15.96:0/3248039156 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) -- 0x5614350b97f0 con 0x56143508e150 2024-05-28T06:09:06.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.985+0000 7f9aefe00640 1 -- 172.21.15.96:0/3248039156 <== mon.0 v2:172.21.15.96:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9afc03e460 con 0x56143508e150 2024-05-28T06:09:06.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.985+0000 7f9aefe00640 1 -- 172.21.15.96:0/3248039156 <== mon.0 v2:172.21.15.96:3300/0 5 ==== fsmap(e 25) ==== 3494+0+0 (secure 0 0 0) 0x7f9afc051070 con 0x56143508e150 2024-05-28T06:09:06.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.985+0000 7f9b14deb740 10 client.6036 fetch_fsmap finished waiting for FSMap version 25 2024-05-28T06:09:06.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.985+0000 7f9b14deb740 10 client.6036 resolve_mds: resolved name 'a' to daemon mds.a 2024-05-28T06:09:06.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.985+0000 7f9b14deb740 20 client.6036 populate_metadata read hostname 'smithi096' 2024-05-28T06:09:06.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.985+0000 7f9b14deb740 1 --2- 172.21.15.96:0/3248039156 >> [v2:172.21.15.96:6833/935552515,v1:172.21.15.96:6835/935552515] conn(0x5614350bc5f0 0x5614350c72b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:06.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.986+0000 7f9b14deb740 4 client.6036 mds_command: new command op to 5859 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-28T06:09:06.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.986+0000 7f9b14deb740 1 -- 172.21.15.96:0/3248039156 --> [v2:172.21.15.96:6833/935552515,v1:172.21.15.96:6835/935552515] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561434fd5530 con 0x5614350bc5f0 2024-05-28T06:09:06.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.986+0000 7f9b03e00640 1 --2- 172.21.15.96:0/3248039156 >> [v2:172.21.15.96:6833/935552515,v1:172.21.15.96:6835/935552515] conn(0x5614350bc5f0 0x5614350c72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:06.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.986+0000 7f9b03e00640 1 --2- 172.21.15.96:0/3248039156 >> [v2:172.21.15.96:6833/935552515,v1:172.21.15.96:6835/935552515] conn(0x5614350bc5f0 0x5614350c72b0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f2322a50ae2b3756 server_cookie=ba65a32c63e3bcc0 in_seq=0 out_seq=0 2024-05-28T06:09:06.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.986+0000 7f9aefe00640 10 client.6036 ms_handle_connect on v2:172.21.15.96:6833/935552515 2024-05-28T06:09:06.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.987+0000 7f9aefe00640 1 -- 172.21.15.96:0/3248039156 <== mds.0 v2:172.21.15.96:6833/935552515 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561434fd5530 con 0x5614350bc5f0 2024-05-28T06:09:06.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.987+0000 7f9aefe00640 10 client.6036 handle_command_reply: tid=0 2024-05-28T06:09:06.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.987+0000 7f9b14deb740 1 client.6036 shutdown 2024-05-28T06:09:06.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.987+0000 7f9b14deb740 1 -- 172.21.15.96:0/3248039156 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56143508e150 msgr2=0x5614350b92b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:09:06.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.987+0000 7f9b14deb740 1 --2- 172.21.15.96:0/3248039156 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56143508e150 0x5614350b92b0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f9afc02a7d0 tx=0x7f9afc004730 comp rx=0 tx=0).stop 2024-05-28T06:09:06.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.987+0000 7f9b14deb740 1 -- 172.21.15.96:0/3248039156 shutdown_connections 2024-05-28T06:09:06.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.987+0000 7f9b14deb740 1 --2- 172.21.15.96:0/3248039156 >> [v2:172.21.15.96:6833/935552515,v1:172.21.15.96:6835/935552515] conn(0x5614350bc5f0 0x5614350c72b0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:06.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.988+0000 7f9b14deb740 1 --2- 172.21.15.96:0/3248039156 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5614350860c0 0x56143505d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:06.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.988+0000 7f9b14deb740 1 --2- 172.21.15.96:0/3248039156 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x5614350aa5d0 0x56143505d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:06.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.988+0000 7f9b14deb740 1 --2- 172.21.15.96:0/3248039156 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56143508e150 0x5614350b92b0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:06.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.988+0000 7f9aefe00640 0 client.6036 ms_handle_reset on v2:172.21.15.96:6833/935552515 2024-05-28T06:09:06.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.988+0000 7f9b14deb740 1 -- 172.21.15.96:0/3248039156 >> 172.21.15.96:0/3248039156 conn(0x56143508a490 msgr2=0x5614350a8be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:09:06.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.988+0000 7f9b14deb740 1 -- 172.21.15.96:0/3248039156 shutdown_connections 2024-05-28T06:09:06.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.988+0000 7f9b14deb740 1 -- 172.21.15.96:0/3248039156 wait complete. 2024-05-28T06:09:06.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:06.988+0000 7f9b14deb740 20 client.6036 trim_cache size 0 max 16384 2024-05-28T06:09:07.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.007+0000 7f9b14deb740 1 Processor -- start 2024-05-28T06:09:07.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.007+0000 7f9b14deb740 1 -- start start 2024-05-28T06:09:07.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.008+0000 7f9b14deb740 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x5614350860c0 0x5614350631a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:07.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.008+0000 7f9b14deb740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56143508e150 0x5614350636e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:07.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.008+0000 7f9b14deb740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5614350aa5d0 0x5614350a8be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:07.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.008+0000 7f9b14deb740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f9b0c200de0 con 0x5614350aa5d0 2024-05-28T06:09:07.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.008+0000 7f9b14deb740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f9b0c1ffe50 con 0x56143508e150 2024-05-28T06:09:07.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.008+0000 7f9b14deb740 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f9b0c15f940 con 0x5614350860c0 2024-05-28T06:09:07.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.008+0000 7f9b0be00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5614350aa5d0 0x5614350a8be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:07.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.009+0000 7f9b0be00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5614350aa5d0 0x5614350a8be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47636/0 (socket says 172.21.15.96:47636) 2024-05-28T06:09:07.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.009+0000 7f9b0be00640 1 -- 172.21.15.96:0/1513219693 learned_addr learned my addr 172.21.15.96:0/1513219693 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:09:07.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.009+0000 7f9b0b400640 1 --2- 172.21.15.96:0/1513219693 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x5614350860c0 0x5614350631a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:07.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.009+0000 7f9b03e00640 1 --2- 172.21.15.96:0/1513219693 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56143508e150 0x5614350636e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:07.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.009+0000 7f9b0be00640 1 -- 172.21.15.96:0/1513219693 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x5614350860c0 msgr2=0x5614350631a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:09:07.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.009+0000 7f9b0be00640 1 --2- 172.21.15.96:0/1513219693 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x5614350860c0 0x5614350631a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:07.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.009+0000 7f9b0be00640 1 -- 172.21.15.96:0/1513219693 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56143508e150 msgr2=0x5614350636e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:09:07.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.009+0000 7f9b0be00640 1 --2- 172.21.15.96:0/1513219693 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56143508e150 0x5614350636e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:07.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.009+0000 7f9b0be00640 1 -- 172.21.15.96:0/1513219693 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5614350bbc10 con 0x5614350aa5d0 2024-05-28T06:09:07.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.009+0000 7f9b0be00640 1 --2- 172.21.15.96:0/1513219693 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5614350aa5d0 0x5614350a8be0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f9b04059930 tx=0x7f9b04059cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97f7bea0342ba352 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:09:07.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.010+0000 7f9afaa00640 1 -- 172.21.15.96:0/1513219693 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b04060050 con 0x5614350aa5d0 2024-05-28T06:09:07.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.010+0000 7f9afaa00640 1 -- 172.21.15.96:0/1513219693 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b04072180 con 0x5614350aa5d0 2024-05-28T06:09:07.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.010+0000 7f9afaa00640 1 -- 172.21.15.96:0/1513219693 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b04057f30 con 0x5614350aa5d0 2024-05-28T06:09:07.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.010+0000 7f9b14deb740 1 -- 172.21.15.96:0/1513219693 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5614350aa5d0 msgr2=0x5614350a8be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:09:07.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.010+0000 7f9b14deb740 1 --2- 172.21.15.96:0/1513219693 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5614350aa5d0 0x5614350a8be0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f9b04059930 tx=0x7f9b04059cf0 comp rx=0 tx=0).stop 2024-05-28T06:09:07.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.010+0000 7f9b14deb740 1 -- 172.21.15.96:0/1513219693 shutdown_connections 2024-05-28T06:09:07.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.010+0000 7f9b14deb740 1 --2- 172.21.15.96:0/1513219693 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5614350aa5d0 0x5614350a8be0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:07.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.010+0000 7f9b14deb740 1 --2- 172.21.15.96:0/1513219693 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x56143508e150 0x5614350636e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:07.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.010+0000 7f9b14deb740 1 --2- 172.21.15.96:0/1513219693 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x5614350860c0 0x5614350631a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:07.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.010+0000 7f9b14deb740 1 -- 172.21.15.96:0/1513219693 >> 172.21.15.96:0/1513219693 conn(0x56143508a490 msgr2=0x5614350a25c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:09:07.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.010+0000 7f9b14deb740 1 -- 172.21.15.96:0/1513219693 shutdown_connections 2024-05-28T06:09:07.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.010+0000 7f9b14deb740 1 -- 172.21.15.96:0/1513219693 wait complete. 2024-05-28T06:09:07.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.011+0000 7f9b14deb740 1 -- start start 2024-05-28T06:09:07.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.011+0000 7f9b14deb740 1 Processor -- start 2024-05-28T06:09:07.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.012+0000 7f9b14deb740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5614350860c0 0x5614350ac190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:07.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.012+0000 7f9b14deb740 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x56143508e150 0x5614350ccd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:07.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.012+0000 7f9b14deb740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5614350aa5d0 0x56143508fe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:07.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.012+0000 7f9b14deb740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x561435088370 con 0x5614350aa5d0 2024-05-28T06:09:07.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.012+0000 7f9b14deb740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f9b0c1ff830 con 0x5614350860c0 2024-05-28T06:09:07.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.012+0000 7f9b14deb740 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x5614350889d0 con 0x56143508e150 2024-05-28T06:09:07.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.012+0000 7f9b14deb740 10 client.0 _ll_register_callbacks cb 0x7f9afc03e610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-28T06:09:07.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.012+0000 7f9b0be00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5614350aa5d0 0x56143508fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:07.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.012+0000 7f9b0be00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5614350aa5d0 0x56143508fe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47642/0 (socket says 172.21.15.96:47642) 2024-05-28T06:09:07.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.012+0000 7f9b0be00640 1 -- 172.21.15.96:0/220460151 learned_addr learned my addr 172.21.15.96:0/220460151 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:09:07.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.013+0000 7f9b0b400640 1 --2- 172.21.15.96:0/220460151 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5614350860c0 0x5614350ac190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:07.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.013+0000 7f9b0be00640 1 -- 172.21.15.96:0/220460151 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x56143508e150 msgr2=0x5614350ccd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:09:07.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.013+0000 7f9b03e00640 1 --2- 172.21.15.96:0/220460151 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x56143508e150 0x5614350ccd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:07.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.013+0000 7f9b0be00640 1 --2- 172.21.15.96:0/220460151 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x56143508e150 0x5614350ccd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:07.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.013+0000 7f9b0be00640 1 -- 172.21.15.96:0/220460151 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5614350860c0 msgr2=0x5614350ac190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:09:07.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.013+0000 7f9b0be00640 1 --2- 172.21.15.96:0/220460151 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5614350860c0 0x5614350ac190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:07.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.013+0000 7f9b0be00640 1 -- 172.21.15.96:0/220460151 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561435063820 con 0x5614350aa5d0 2024-05-28T06:09:07.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.013+0000 7f9b0b400640 1 --2- 172.21.15.96:0/220460151 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5614350860c0 0x5614350ac190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:09:07.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.013+0000 7f9b0be00640 1 --2- 172.21.15.96:0/220460151 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5614350aa5d0 0x56143508fe50 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f9b0401d040 tx=0x7f9b0400ae80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb02ed2b0e385c72 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:09:07.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.013+0000 7f9aefe00640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-05-28T06:09:07.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.013+0000 7f9aefe00640 1 -- 172.21.15.96:0/220460151 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b0408c0b0 con 0x5614350aa5d0 2024-05-28T06:09:07.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.013+0000 7f9aefe00640 1 -- 172.21.15.96:0/220460151 <== mon.0 v2:172.21.15.96:3300/0 2 ==== osd_map(45..45 src has 1..45) ==== 5737+0+0 (secure 0 0 0) 0x7f9b04079170 con 0x5614350aa5d0 2024-05-28T06:09:07.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.013+0000 7f9b14deb740 1 -- 172.21.15.96:0/220460151 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9b04072180 con 0x5614350aa5d0 2024-05-28T06:09:07.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.014+0000 7f9aefe00640 1 -- 172.21.15.96:0/220460151 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f9b04060020 con 0x5614350aa5d0 2024-05-28T06:09:07.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.014+0000 7f9b14deb740 10 client.6042 fetch_fsmap learned FSMap version 25 2024-05-28T06:09:07.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.014+0000 7f9b14deb740 1 -- 172.21.15.96:0/220460151 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) -- 0x561435090490 con 0x5614350aa5d0 2024-05-28T06:09:07.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.014+0000 7f9aefe00640 1 -- 172.21.15.96:0/220460151 <== mon.0 v2:172.21.15.96:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b04072180 con 0x5614350aa5d0 2024-05-28T06:09:07.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.014+0000 7f9aefe00640 1 -- 172.21.15.96:0/220460151 <== mon.0 v2:172.21.15.96:3300/0 5 ==== fsmap(e 25) ==== 3494+0+0 (secure 0 0 0) 0x7f9b04062380 con 0x5614350aa5d0 2024-05-28T06:09:07.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.014+0000 7f9b14deb740 10 client.6042 fetch_fsmap finished waiting for FSMap version 25 2024-05-28T06:09:07.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.014+0000 7f9b14deb740 10 client.6042 resolve_mds: resolved name 'a' to daemon mds.a 2024-05-28T06:09:07.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.014+0000 7f9b14deb740 20 client.6042 populate_metadata read hostname 'smithi096' 2024-05-28T06:09:07.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.014+0000 7f9b14deb740 1 --2- 172.21.15.96:0/220460151 >> [v2:172.21.15.96:6833/935552515,v1:172.21.15.96:6835/935552515] conn(0x5614350a9230 0x5614350b7240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:07.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.014+0000 7f9b14deb740 4 client.6042 mds_command: new command op to 5859 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-28T06:09:07.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.014+0000 7f9b14deb740 1 -- 172.21.15.96:0/220460151 --> [v2:172.21.15.96:6833/935552515,v1:172.21.15.96:6835/935552515] -- command(tid 0: {"prefix": "status"}) -- 0x561434fcf3a0 con 0x5614350a9230 2024-05-28T06:09:07.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.014+0000 7f9b0b400640 1 --2- 172.21.15.96:0/220460151 >> [v2:172.21.15.96:6833/935552515,v1:172.21.15.96:6835/935552515] conn(0x5614350a9230 0x5614350b7240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:07.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.015+0000 7f9b0b400640 1 --2- 172.21.15.96:0/220460151 >> [v2:172.21.15.96:6833/935552515,v1:172.21.15.96:6835/935552515] conn(0x5614350a9230 0x5614350b7240 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=15ed9983ec3d444d server_cookie=2e4f1d845efa089a in_seq=0 out_seq=0 2024-05-28T06:09:07.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.015+0000 7f9aefe00640 10 client.6042 ms_handle_connect on v2:172.21.15.96:6833/935552515 2024-05-28T06:09:07.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.015+0000 7f9aefe00640 1 -- 172.21.15.96:0/220460151 <== mds.0 v2:172.21.15.96:6833/935552515 1 ==== command_reply(tid 0: 0 ) ==== 8+0+325 (crc 0 0 0) 0x561434fcf3a0 con 0x5614350a9230 2024-05-28T06:09:07.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.015+0000 7f9aefe00640 10 client.6042 handle_command_reply: tid=0 2024-05-28T06:09:07.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.015+0000 7f9b14deb740 1 client.6042 shutdown 2024-05-28T06:09:07.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.015+0000 7f9b14deb740 1 -- 172.21.15.96:0/220460151 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5614350aa5d0 msgr2=0x56143508fe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:09:07.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.015+0000 7f9b14deb740 1 --2- 172.21.15.96:0/220460151 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5614350aa5d0 0x56143508fe50 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f9b0401d040 tx=0x7f9b0400ae80 comp rx=0 tx=0).stop 2024-05-28T06:09:07.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.015+0000 7f9b14deb740 1 -- 172.21.15.96:0/220460151 shutdown_connections 2024-05-28T06:09:07.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.015+0000 7f9b14deb740 1 --2- 172.21.15.96:0/220460151 >> [v2:172.21.15.96:6833/935552515,v1:172.21.15.96:6835/935552515] conn(0x5614350a9230 0x5614350b7240 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:07.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.015+0000 7f9b14deb740 1 --2- 172.21.15.96:0/220460151 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5614350aa5d0 0x56143508fe50 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:07.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.015+0000 7f9b14deb740 1 --2- 172.21.15.96:0/220460151 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x56143508e150 0x5614350ccd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:07.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.015+0000 7f9b14deb740 1 --2- 172.21.15.96:0/220460151 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5614350860c0 0x5614350ac190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:07.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.015+0000 7f9b14deb740 1 -- 172.21.15.96:0/220460151 >> 172.21.15.96:0/220460151 conn(0x56143508a490 msgr2=0x5614350877f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:09:07.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.015+0000 7f9aefe00640 0 client.6042 ms_handle_reset on v2:172.21.15.96:6833/935552515 2024-05-28T06:09:07.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.015+0000 7f9b14deb740 1 -- 172.21.15.96:0/220460151 shutdown_connections 2024-05-28T06:09:07.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.015+0000 7f9b14deb740 1 -- 172.21.15.96:0/220460151 wait complete. 2024-05-28T06:09:07.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.016+0000 7f9b14deb740 20 client.6042 trim_cache size 0 max 16384 2024-05-28T06:09:07.022 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:09:07.022 INFO:teuthology.orchestra.run.smithi096.stdout: "cluster_fsid": "e4a831b5-e6d1-47a7-96cb-bf85154ee0d2", 2024-05-28T06:09:07.022 INFO:teuthology.orchestra.run.smithi096.stdout: "whoami": 0, 2024-05-28T06:09:07.022 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 5859, 2024-05-28T06:09:07.022 INFO:teuthology.orchestra.run.smithi096.stdout: "want_state": "up:active", 2024-05-28T06:09:07.022 INFO:teuthology.orchestra.run.smithi096.stdout: "state": "up:active", 2024-05-28T06:09:07.022 INFO:teuthology.orchestra.run.smithi096.stdout: "fs_name": "cephfs", 2024-05-28T06:09:07.022 INFO:teuthology.orchestra.run.smithi096.stdout: "rank_uptime": 11.989151029, 2024-05-28T06:09:07.022 INFO:teuthology.orchestra.run.smithi096.stdout: "mdsmap_epoch": 25, 2024-05-28T06:09:07.023 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap_epoch": 45, 2024-05-28T06:09:07.023 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap_epoch_barrier": 45, 2024-05-28T06:09:07.023 INFO:teuthology.orchestra.run.smithi096.stdout: "uptime": 17.440725696000001 2024-05-28T06:09:07.023 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:09:07.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.016+0000 7f9b10a00640 1 -- 172.21.15.96:0/2716655156 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9ae806cbf0 msgr2=0x7f9ae806f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:09:07.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.016+0000 7f9b10a00640 1 --2- 172.21.15.96:0/2716655156 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9ae806cbf0 0x7f9ae806f0b0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f9afc034000 tx=0x7f9afc02e000 comp rx=0 tx=0).stop 2024-05-28T06:09:07.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.016+0000 7f9b10a00640 1 -- 172.21.15.96:0/2716655156 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9b0c15e870 msgr2=0x7f9b0c0cd640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:09:07.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.016+0000 7f9b10a00640 1 --2- 172.21.15.96:0/2716655156 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9b0c15e870 0x7f9b0c0cd640 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f9b0400c700 tx=0x7f9b0400cbc0 comp rx=0 tx=0).stop 2024-05-28T06:09:07.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.016+0000 7f9b10a00640 1 -- 172.21.15.96:0/2716655156 shutdown_connections 2024-05-28T06:09:07.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.016+0000 7f9b10a00640 1 --2- 172.21.15.96:0/2716655156 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9ae806cbf0 0x7f9ae806f0b0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:07.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.016+0000 7f9b10a00640 1 --2- 172.21.15.96:0/2716655156 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9b0c15e870 0x7f9b0c0cd640 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:07.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.016+0000 7f9b10a00640 1 --2- 172.21.15.96:0/2716655156 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9b0c0d14d0 0x7f9b0c0cd100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:07.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.016+0000 7f9b10a00640 1 --2- 172.21.15.96:0/2716655156 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9b0c0cfa40 0x7f9b0c0ca500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:07.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.016+0000 7f9b10a00640 1 -- 172.21.15.96:0/2716655156 >> 172.21.15.96:0/2716655156 conn(0x7f9b0c08bfc0 msgr2=0x7f9b0c0b6360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:09:07.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.016+0000 7f9b10a00640 1 -- 172.21.15.96:0/2716655156 shutdown_connections 2024-05-28T06:09:07.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.016+0000 7f9b10a00640 1 -- 172.21.15.96:0/2716655156 wait complete. 2024-05-28T06:09:07.037 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-05-28T06:09:07.233 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.232+0000 7fd30d200640 1 Processor -- start 2024-05-28T06:09:07.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.232+0000 7fd30d200640 1 -- start start 2024-05-28T06:09:07.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.234+0000 7fd30d200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd308073310 0x7fd3080b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:07.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.234+0000 7fd30d200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3080afd60 con 0x7fd30815b3a0 2024-05-28T06:09:07.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.234+0000 7fd30d200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3080af7a0 con 0x7fd30806dc20 2024-05-28T06:09:07.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.234+0000 7fd30d200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fd30806f980 con 0x7fd308073310 2024-05-28T06:09:07.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.234+0000 7fd307400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fd30815b3a0 0x7fd308072c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:51212/0 (socket says 172.21.15.96:51212) 2024-05-28T06:09:07.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.234+0000 7fd307400640 1 -- 172.21.15.96:0/722400469 learned_addr learned my addr 172.21.15.96:0/722400469 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:09:07.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.234+0000 7fd306000640 1 -- 172.21.15.96:0/722400469 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3181114633 0 0) 0x7fd3080afd60 con 0x7fd30815b3a0 2024-05-28T06:09:07.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.234+0000 7fd306a00640 1 --2- 172.21.15.96:0/722400469 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd308073310 0x7fd3080b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:07.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.234+0000 7fd306000640 1 -- 172.21.15.96:0/722400469 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd2ec003520 con 0x7fd30815b3a0 2024-05-28T06:09:07.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.235+0000 7fd306000640 1 -- 172.21.15.96:0/722400469 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 574123352 0 0) 0x7fd2ec003520 con 0x7fd30815b3a0 2024-05-28T06:09:07.238 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.235+0000 7fd306000640 1 -- 172.21.15.96:0/722400469 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd308073310 msgr2=0x7fd3080b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:09:07.238 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.235+0000 7fd306000640 1 --2- 172.21.15.96:0/722400469 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd308073310 0x7fd3080b32c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:07.238 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.235+0000 7fd306000640 1 -- 172.21.15.96:0/722400469 >> v1:172.21.15.123:6789/0 conn(0x7fd30806dc20 legacy=0x7fd3080ae550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:09:07.238 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.235+0000 7fd306000640 1 -- 172.21.15.96:0/722400469 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3080afa80 con 0x7fd30815b3a0 2024-05-28T06:09:07.238 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.235+0000 7fd306000640 1 -- 172.21.15.96:0/722400469 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7fd2f8001670 con 0x7fd30815b3a0 2024-05-28T06:09:07.238 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.235+0000 7fd306000640 1 -- 172.21.15.96:0/722400469 >> v1:172.21.15.96:6789/0 conn(0x7fd30815b3a0 legacy=0x7fd308072c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:09:07.238 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.235+0000 7fd306000640 1 --2- 172.21.15.96:0/722400469 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd308073310 0x7fd2ec0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:07.238 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.235+0000 7fd306000640 1 --2- 172.21.15.96:0/722400469 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2ec004010 0x7fd2ec004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:07.238 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.235+0000 7fd306000640 1 --2- 172.21.15.96:0/722400469 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd2ec004d40 0x7fd2ec0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:07.238 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.235+0000 7fd306000640 1 -- 172.21.15.96:0/722400469 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd30806f980 con 0x7fd2ec004d40 2024-05-28T06:09:07.238 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.235+0000 7fd306000640 1 -- 172.21.15.96:0/722400469 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd2ec00bb80 con 0x7fd2ec004010 2024-05-28T06:09:07.238 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.235+0000 7fd306000640 1 -- 172.21.15.96:0/722400469 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fd2ec00bd80 con 0x7fd308073310 2024-05-28T06:09:07.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.235+0000 7fd307e00640 1 --2- 172.21.15.96:0/722400469 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd2ec004d40 0x7fd2ec0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:07.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.236+0000 7fd306a00640 1 --2- 172.21.15.96:0/722400469 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2ec004010 0x7fd2ec004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:07.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.236+0000 7fd307400640 1 --2- 172.21.15.96:0/722400469 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd308073310 0x7fd2ec0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:07.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.236+0000 7fd307e00640 1 -- 172.21.15.96:0/722400469 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd308073310 msgr2=0x7fd2ec0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:09:07.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.236+0000 7fd307e00640 1 --2- 172.21.15.96:0/722400469 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd308073310 0x7fd2ec0039e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:07.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.236+0000 7fd307e00640 1 -- 172.21.15.96:0/722400469 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2ec004010 msgr2=0x7fd2ec004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:09:07.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.236+0000 7fd307e00640 1 --2- 172.21.15.96:0/722400469 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2ec004010 0x7fd2ec004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:07.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.236+0000 7fd307e00640 1 -- 172.21.15.96:0/722400469 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2ec003520 con 0x7fd2ec004d40 2024-05-28T06:09:07.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.237+0000 7fd307e00640 1 --2- 172.21.15.96:0/722400469 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd2ec004d40 0x7fd2ec0051c0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fd300001330 tx=0x7fd2ec005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f279235e1429947 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:09:07.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.237+0000 7fd306000640 1 -- 172.21.15.96:0/722400469 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd300001370 con 0x7fd2ec004d40 2024-05-28T06:09:07.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.237+0000 7fd306000640 1 -- 172.21.15.96:0/722400469 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd30002cdb0 con 0x7fd2ec004d40 2024-05-28T06:09:07.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.237+0000 7fd306000640 1 -- 172.21.15.96:0/722400469 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd300005310 con 0x7fd2ec004d40 2024-05-28T06:09:07.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.238+0000 7fd30d200640 1 -- 172.21.15.96:0/722400469 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd2ec004d40 msgr2=0x7fd2ec0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:09:07.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.238+0000 7fd30d200640 1 --2- 172.21.15.96:0/722400469 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd2ec004d40 0x7fd2ec0051c0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fd300001330 tx=0x7fd2ec005830 comp rx=0 tx=0).stop 2024-05-28T06:09:07.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.238+0000 7fd307e00640 1 -- 172.21.15.96:0/722400469 reap_dead start 2024-05-28T06:09:07.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.238+0000 7fd30d200640 1 -- 172.21.15.96:0/722400469 shutdown_connections 2024-05-28T06:09:07.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.238+0000 7fd30d200640 1 -- 172.21.15.96:0/722400469 >> 172.21.15.96:0/722400469 conn(0x7fd3080b1fc0 msgr2=0x7fd3080b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:09:07.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.238+0000 7fd30d200640 1 -- 172.21.15.96:0/722400469 shutdown_connections 2024-05-28T06:09:07.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.238+0000 7fd30d200640 1 -- 172.21.15.96:0/722400469 wait complete. 2024-05-28T06:09:07.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.240+0000 7fd30d200640 1 Processor -- start 2024-05-28T06:09:07.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.240+0000 7fd30d200640 1 -- start start 2024-05-28T06:09:07.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.241+0000 7fd30d200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2ec004d40 0x7fd3080cced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:07.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.241+0000 7fd30d200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd3080cf130 0x7fd30806e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:07.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.241+0000 7fd30d200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd3080cf590 0x7fd3081f3c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:07.243 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.241+0000 7fd30d200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd3080b32c0 con 0x7fd3080cf130 2024-05-28T06:09:07.243 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.241+0000 7fd30d200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd3080ae0c0 con 0x7fd2ec004d40 2024-05-28T06:09:07.243 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.241+0000 7fd30d200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fd3080b3440 con 0x7fd3080cf590 2024-05-28T06:09:07.243 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.241+0000 7fd306a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd3080cf130 0x7fd30806e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:07.243 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.241+0000 7fd306a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd3080cf130 0x7fd30806e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47660/0 (socket says 172.21.15.96:47660) 2024-05-28T06:09:07.243 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.241+0000 7fd306a00640 1 -- 172.21.15.96:0/1286248607 learned_addr learned my addr 172.21.15.96:0/1286248607 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:09:07.244 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.242+0000 7fd306a00640 1 -- 172.21.15.96:0/1286248607 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd3080cf590 msgr2=0x7fd3081f3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:09:07.244 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.242+0000 7fd306a00640 1 --2- 172.21.15.96:0/1286248607 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd3080cf590 0x7fd3081f3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:07.244 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.242+0000 7fd306a00640 1 -- 172.21.15.96:0/1286248607 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2ec004d40 msgr2=0x7fd3080cced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:09:07.244 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.242+0000 7fd307400640 1 --2- 172.21.15.96:0/1286248607 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2ec004d40 0x7fd3080cced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:07.244 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.242+0000 7fd306a00640 1 --2- 172.21.15.96:0/1286248607 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2ec004d40 0x7fd3080cced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:07.244 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.242+0000 7fd306a00640 1 -- 172.21.15.96:0/1286248607 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3080cfa80 con 0x7fd3080cf130 2024-05-28T06:09:07.244 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.242+0000 7fd306a00640 1 --2- 172.21.15.96:0/1286248607 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd3080cf130 0x7fd30806e820 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fd2f0009bb0 tx=0x7fd2f0007700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:09:07.245 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.242+0000 7fd2ff400640 1 -- 172.21.15.96:0/1286248607 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2f0014610 con 0x7fd3080cf130 2024-05-28T06:09:07.245 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.242+0000 7fd2ff400640 1 -- 172.21.15.96:0/1286248607 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2f0019070 con 0x7fd3080cf130 2024-05-28T06:09:07.245 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.242+0000 7fd2ff400640 1 -- 172.21.15.96:0/1286248607 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2f00149d0 con 0x7fd3080cf130 2024-05-28T06:09:07.245 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.242+0000 7fd30d200640 1 -- 172.21.15.96:0/1286248607 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3080c97c0 con 0x7fd3080cf130 2024-05-28T06:09:07.245 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.242+0000 7fd30d200640 1 -- 172.21.15.96:0/1286248607 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3080c9d80 con 0x7fd3080cf130 2024-05-28T06:09:07.245 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.244+0000 7fd30d200640 1 -- 172.21.15.96:0/1286248607 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3080689e0 con 0x7fd3080cf130 2024-05-28T06:09:07.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.244+0000 7fd2ff400640 1 -- 172.21.15.96:0/1286248607 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 9) ==== 91295+0+0 (secure 0 0 0) 0x7fd2f0014b70 con 0x7fd3080cf130 2024-05-28T06:09:07.247 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.245+0000 7fd2ff400640 1 --2- 172.21.15.96:0/1286248607 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd2e406cbc0 0x7fd2e406f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:07.247 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.245+0000 7fd2ff400640 1 -- 172.21.15.96:0/1286248607 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5737+0+0 (secure 0 0 0) 0x7fd2f008ca20 con 0x7fd3080cf130 2024-05-28T06:09:07.247 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.246+0000 7fd307400640 1 --2- 172.21.15.96:0/1286248607 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd2e406cbc0 0x7fd2e406f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:07.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.247+0000 7fd307400640 1 --2- 172.21.15.96:0/1286248607 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd2e406cbc0 0x7fd2e406f080 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fd2f8007950 tx=0x7fd2f8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:09:07.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.250+0000 7fd2ff400640 1 -- 172.21.15.96:0/1286248607 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd2f005a9d0 con 0x7fd3080cf130 2024-05-28T06:09:07.473 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.471+0000 7fd30d200640 1 -- 172.21.15.96:0/1286248607 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fd3080b2e90 con 0x7fd3080cf130 2024-05-28T06:09:07.473 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.472+0000 7fd2ff400640 1 -- 172.21.15.96:0/1286248607 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v25) ==== 53+0+73 (secure 0 0 0) 0x7fd2f0003ea0 con 0x7fd3080cf130 2024-05-28T06:09:07.473 INFO:teuthology.orchestra.run.smithi096.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-05-28T06:09:07.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.474+0000 7fd30d200640 1 -- 172.21.15.96:0/1286248607 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd2e406cbc0 msgr2=0x7fd2e406f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:09:07.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.474+0000 7fd30d200640 1 --2- 172.21.15.96:0/1286248607 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd2e406cbc0 0x7fd2e406f080 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fd2f8007950 tx=0x7fd2f8009040 comp rx=0 tx=0).stop 2024-05-28T06:09:07.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.474+0000 7fd30d200640 1 -- 172.21.15.96:0/1286248607 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd3080cf130 msgr2=0x7fd30806e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:09:07.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.474+0000 7fd30d200640 1 --2- 172.21.15.96:0/1286248607 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd3080cf130 0x7fd30806e820 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fd2f0009bb0 tx=0x7fd2f0007700 comp rx=0 tx=0).stop 2024-05-28T06:09:07.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.474+0000 7fd30d200640 1 -- 172.21.15.96:0/1286248607 shutdown_connections 2024-05-28T06:09:07.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.474+0000 7fd30d200640 1 --2- 172.21.15.96:0/1286248607 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd2e406cbc0 0x7fd2e406f080 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:07.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.474+0000 7fd30d200640 1 --2- 172.21.15.96:0/1286248607 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd3080cf590 0x7fd3081f3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:07.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.474+0000 7fd30d200640 1 --2- 172.21.15.96:0/1286248607 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd3080cf130 0x7fd30806e820 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:07.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.474+0000 7fd30d200640 1 --2- 172.21.15.96:0/1286248607 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2ec004d40 0x7fd3080cced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:07.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.474+0000 7fd30d200640 1 -- 172.21.15.96:0/1286248607 >> 172.21.15.96:0/1286248607 conn(0x7fd3080b1fc0 msgr2=0x7fd308155700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:09:07.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.474+0000 7fd30d200640 1 -- 172.21.15.96:0/1286248607 shutdown_connections 2024-05-28T06:09:07.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07.474+0000 7fd30d200640 1 -- 172.21.15.96:0/1286248607 wait complete. 2024-05-28T06:09:07.494 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-05-28T06:09:07.494 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-28T06:09:07.494 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi096.front.sepia.ceph.com 2024-05-28T06:09:07.494 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-05-28T06:09:07.494 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-28T06:09:07.494 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-05-28T06:09:07.495 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-05-28T06:09:07.495 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-28T06:09:07.495 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-28T06:09:07.495 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:09:07.495 DEBUG:teuthology.orchestra.run.smithi096:> ip addr 2024-05-28T06:09:07.541 INFO:teuthology.orchestra.run.smithi096.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-28T06:09:07.541 INFO:teuthology.orchestra.run.smithi096.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-28T06:09:07.542 INFO:teuthology.orchestra.run.smithi096.stdout: inet 127.0.0.1/8 scope host lo 2024-05-28T06:09:07.542 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:09:07.542 INFO:teuthology.orchestra.run.smithi096.stdout: inet6 ::1/128 scope host 2024-05-28T06:09:07.542 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:09:07.542 INFO:teuthology.orchestra.run.smithi096.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-28T06:09:07.542 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether 0c:c4:7a:d8:d9:bc brd ff:ff:ff:ff:ff:ff 2024-05-28T06:09:07.542 INFO:teuthology.orchestra.run.smithi096.stdout: altname enp7s0f0 2024-05-28T06:09:07.542 INFO:teuthology.orchestra.run.smithi096.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-28T06:09:07.542 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether 0c:c4:7a:d8:d9:bd brd ff:ff:ff:ff:ff:ff 2024-05-28T06:09:07.542 INFO:teuthology.orchestra.run.smithi096.stdout: altname enp7s0f1 2024-05-28T06:09:07.542 INFO:teuthology.orchestra.run.smithi096.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-28T06:09:07.542 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether 0c:c4:7a:8f:cc:0c brd ff:ff:ff:ff:ff:ff 2024-05-28T06:09:07.543 INFO:teuthology.orchestra.run.smithi096.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-28T06:09:07.543 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether 0c:c4:7a:8f:cc:0d brd ff:ff:ff:ff:ff:ff 2024-05-28T06:09:07.543 INFO:teuthology.orchestra.run.smithi096.stdout: inet 172.21.15.96/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-05-28T06:09:07.543 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft 42289sec preferred_lft 42289sec 2024-05-28T06:09:07.543 INFO:teuthology.orchestra.run.smithi096.stdout: inet6 fe80::ec4:7aff:fe8f:cc0d/64 scope link 2024-05-28T06:09:07.543 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:09:07.543 INFO:teuthology.orchestra.run.smithi096.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-05-28T06:09:07.543 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether ae:d9:f5:b0:4f:f3 brd ff:ff:ff:ff:ff:ff 2024-05-28T06:09:07.543 INFO:teuthology.orchestra.run.smithi096.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-05-28T06:09:07.543 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:09:07.543 INFO:teuthology.orchestra.run.smithi096.stdout: inet6 fe80::604e:fcff:fe60:7892/64 scope link 2024-05-28T06:09:07.543 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:09:07.543 INFO:teuthology.orchestra.run.smithi096.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-28T06:09:07.543 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether ce:6f:19:93:44:f9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-28T06:09:07.544 INFO:teuthology.orchestra.run.smithi096.stdout: inet6 fe80::cc6f:19ff:fe93:44f9/64 scope link 2024-05-28T06:09:07.544 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:09:07.544 INFO:teuthology.orchestra.run.smithi096.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-28T06:09:07.544 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether ae:d9:f5:b0:4f:f3 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-28T06:09:07.544 INFO:teuthology.orchestra.run.smithi096.stdout: inet6 fe80::acd9:f5ff:feb0:4ff3/64 scope link 2024-05-28T06:09:07.544 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:09:07.544 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:09:07.544 DEBUG:teuthology.orchestra.run.smithi096:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-28T06:09:07.660 INFO:teuthology.orchestra.run.smithi096.stdout:1 2024-05-28T06:09:07.664 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:09:07.664 DEBUG:teuthology.orchestra.run.smithi096:> route 2024-05-28T06:09:07.718 INFO:teuthology.orchestra.run.smithi096.stdout:Kernel IP routing table 2024-05-28T06:09:07.719 INFO:teuthology.orchestra.run.smithi096.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-28T06:09:07.719 INFO:teuthology.orchestra.run.smithi096.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-05-28T06:09:07.719 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-05-28T06:09:07.719 INFO:teuthology.orchestra.run.smithi096.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-28T06:09:07.719 DEBUG:teuthology.orchestra.run.smithi096:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-28T06:09:07.719 DEBUG:teuthology.orchestra.run.smithi096:> set -e 2024-05-28T06:09:07.719 DEBUG:teuthology.orchestra.run.smithi096:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-05-28T06:09:07.720 DEBUG:teuthology.orchestra.run.smithi096:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-05-28T06:09:07.720 DEBUG:teuthology.orchestra.run.smithi096:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-05-28T06:09:07.720 DEBUG:teuthology.orchestra.run.smithi096:> ') 2024-05-28T06:09:07.823 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-28T06:09:07.919 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:07 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-28T06:09:07.929 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:09:07.930 DEBUG:teuthology.orchestra.run.smithi096:> ip netns list 2024-05-28T06:09:07.980 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-05-28T06:09:07.980 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-05-28T06:09:07.981 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-28T06:09:07.981 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:09:07.981 DEBUG:teuthology.orchestra.run.smithi096:> sudo ip link set brx.0 up 2024-05-28T06:09:08.064 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-05-28T06:09:08.065 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-28T06:09:08.065 DEBUG:teuthology.orchestra.run.smithi096:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-05-28T06:09:08.119 INFO:teuthology.orchestra.run.smithi096.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-05-28T06:09:08.119 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-28T06:09:08.119 DEBUG:teuthology.orchestra.run.smithi096:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-05-28T06:09:08.169 DEBUG:teuthology.orchestra.run.smithi096:> sudo modprobe fuse 2024-05-28T06:09:08.251 DEBUG:teuthology.orchestra.run.smithi096:> cat /proc/self/mounts | awk '{print $2}' 2024-05-28T06:09:08.307 INFO:teuthology.orchestra.run.smithi096.stdout:/proc 2024-05-28T06:09:08.307 INFO:teuthology.orchestra.run.smithi096.stdout:/sys 2024-05-28T06:09:08.307 INFO:teuthology.orchestra.run.smithi096.stdout:/dev 2024-05-28T06:09:08.307 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/security 2024-05-28T06:09:08.308 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/shm 2024-05-28T06:09:08.308 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/pts 2024-05-28T06:09:08.308 INFO:teuthology.orchestra.run.smithi096.stdout:/run 2024-05-28T06:09:08.308 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/cgroup 2024-05-28T06:09:08.308 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/pstore 2024-05-28T06:09:08.308 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/bpf 2024-05-28T06:09:08.308 INFO:teuthology.orchestra.run.smithi096.stdout:/ 2024-05-28T06:09:08.308 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/selinux 2024-05-28T06:09:08.308 INFO:teuthology.orchestra.run.smithi096.stdout:/proc/sys/fs/binfmt_misc 2024-05-28T06:09:08.308 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/mqueue 2024-05-28T06:09:08.308 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/debug 2024-05-28T06:09:08.308 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/hugepages 2024-05-28T06:09:08.308 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/tracing 2024-05-28T06:09:08.308 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/fuse/connections 2024-05-28T06:09:08.309 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/config 2024-05-28T06:09:08.309 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-sysctl.service 2024-05-28T06:09:08.309 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-28T06:09:08.309 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-28T06:09:08.309 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/nfs/rpc_pipefs 2024-05-28T06:09:08.309 INFO:teuthology.orchestra.run.smithi096.stdout:/run/user/1000 2024-05-28T06:09:08.309 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph 2024-05-28T06:09:08.309 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-0 2024-05-28T06:09:08.309 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-1 2024-05-28T06:09:08.309 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-2 2024-05-28T06:09:08.309 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-3 2024-05-28T06:09:08.309 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns 2024-05-28T06:09:08.309 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-28T06:09:08.309 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-28T06:09:08.310 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-28T06:09:08.310 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-28T06:09:08.311 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:09:08.311 DEBUG:teuthology.orchestra.run.smithi096:> ls /sys/fs/fuse/connections 2024-05-28T06:09:08.362 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-05-28T06:09:08.362 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-05-28T06:09:08.367 DEBUG:teuthology.orchestra.run.smithi096:> sudo modprobe fuse 2024-05-28T06:09:08.433 DEBUG:teuthology.orchestra.run.smithi096:> cat /proc/self/mounts | awk '{print $2}' 2024-05-28T06:09:08.476 INFO:teuthology.orchestra.run.smithi096.stdout:/proc 2024-05-28T06:09:08.476 INFO:teuthology.orchestra.run.smithi096.stdout:/sys 2024-05-28T06:09:08.476 INFO:teuthology.orchestra.run.smithi096.stdout:/dev 2024-05-28T06:09:08.476 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/security 2024-05-28T06:09:08.476 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/shm 2024-05-28T06:09:08.476 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/pts 2024-05-28T06:09:08.477 INFO:teuthology.orchestra.run.smithi096.stdout:/run 2024-05-28T06:09:08.477 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/cgroup 2024-05-28T06:09:08.477 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/pstore 2024-05-28T06:09:08.477 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/bpf 2024-05-28T06:09:08.477 INFO:teuthology.orchestra.run.smithi096.stdout:/ 2024-05-28T06:09:08.477 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/selinux 2024-05-28T06:09:08.477 INFO:teuthology.orchestra.run.smithi096.stdout:/proc/sys/fs/binfmt_misc 2024-05-28T06:09:08.477 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/mqueue 2024-05-28T06:09:08.477 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/debug 2024-05-28T06:09:08.477 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/hugepages 2024-05-28T06:09:08.477 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/tracing 2024-05-28T06:09:08.477 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/fuse/connections 2024-05-28T06:09:08.477 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/config 2024-05-28T06:09:08.477 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-sysctl.service 2024-05-28T06:09:08.478 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-28T06:09:08.478 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-28T06:09:08.478 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/nfs/rpc_pipefs 2024-05-28T06:09:08.478 INFO:teuthology.orchestra.run.smithi096.stdout:/run/user/1000 2024-05-28T06:09:08.478 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph 2024-05-28T06:09:08.478 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-0 2024-05-28T06:09:08.478 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-1 2024-05-28T06:09:08.478 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-2 2024-05-28T06:09:08.478 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-3 2024-05-28T06:09:08.478 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns 2024-05-28T06:09:08.478 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-28T06:09:08.478 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-28T06:09:08.479 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-28T06:09:08.479 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-28T06:09:08.479 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:09:08.480 DEBUG:teuthology.orchestra.run.smithi096:> ls /sys/fs/fuse/connections 2024-05-28T06:09:08.531 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi096.stderr:2024-05-28T06:09:08.530+0000 7f915b4e7480 -1 init, newargv = 0x55fb2033ba70 newargc=15 2024-05-28T06:09:08.532 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi096.stderr:ceph-fuse[55615]: starting ceph client 2024-05-28T06:09:08.541 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi096.stderr:ceph-fuse[55615]: starting fuse 2024-05-28T06:09:08.561 INFO:teuthology.orchestra.run.smithi096.stdout:40 2024-05-28T06:09:08.561 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40] 2024-05-28T06:09:08.562 DEBUG:teuthology.orchestra.run.smithi096:> sudo stdin-killer -- python3 -c ' 2024-05-28T06:09:08.562 DEBUG:teuthology.orchestra.run.smithi096:> import glob 2024-05-28T06:09:08.562 DEBUG:teuthology.orchestra.run.smithi096:> import re 2024-05-28T06:09:08.562 DEBUG:teuthology.orchestra.run.smithi096:> import os 2024-05-28T06:09:08.562 DEBUG:teuthology.orchestra.run.smithi096:> import subprocess 2024-05-28T06:09:08.562 DEBUG:teuthology.orchestra.run.smithi096:> 2024-05-28T06:09:08.562 DEBUG:teuthology.orchestra.run.smithi096:> def _find_admin_socket(client_name): 2024-05-28T06:09:08.562 DEBUG:teuthology.orchestra.run.smithi096:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-05-28T06:09:08.562 DEBUG:teuthology.orchestra.run.smithi096:> files = glob.glob(asok_path) 2024-05-28T06:09:08.562 DEBUG:teuthology.orchestra.run.smithi096:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-05-28T06:09:08.562 DEBUG:teuthology.orchestra.run.smithi096:> 2024-05-28T06:09:08.562 DEBUG:teuthology.orchestra.run.smithi096:> # Given a non-glob path, it better be there 2024-05-28T06:09:08.563 DEBUG:teuthology.orchestra.run.smithi096:> if "*" not in asok_path: 2024-05-28T06:09:08.563 DEBUG:teuthology.orchestra.run.smithi096:> assert(len(files) == 1) 2024-05-28T06:09:08.563 DEBUG:teuthology.orchestra.run.smithi096:> return files[0] 2024-05-28T06:09:08.563 DEBUG:teuthology.orchestra.run.smithi096:> 2024-05-28T06:09:08.563 DEBUG:teuthology.orchestra.run.smithi096:> for f in files: 2024-05-28T06:09:08.563 DEBUG:teuthology.orchestra.run.smithi096:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-05-28T06:09:08.563 DEBUG:teuthology.orchestra.run.smithi096:> if os.path.exists("/proc/{0}".format(pid)): 2024-05-28T06:09:08.563 DEBUG:teuthology.orchestra.run.smithi096:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-05-28T06:09:08.563 DEBUG:teuthology.orchestra.run.smithi096:> contents = proc_f.read() 2024-05-28T06:09:08.563 DEBUG:teuthology.orchestra.run.smithi096:> if mountpoint in contents: 2024-05-28T06:09:08.563 DEBUG:teuthology.orchestra.run.smithi096:> return f 2024-05-28T06:09:08.563 DEBUG:teuthology.orchestra.run.smithi096:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-05-28T06:09:08.563 DEBUG:teuthology.orchestra.run.smithi096:> 2024-05-28T06:09:08.563 DEBUG:teuthology.orchestra.run.smithi096:> print(_find_admin_socket("client.0")) 2024-05-28T06:09:08.563 DEBUG:teuthology.orchestra.run.smithi096:> ' 2024-05-28T06:09:08.753 INFO:teuthology.orchestra.run.smithi096.stdout:/var/run/ceph/ceph-client.0.55615.asok 2024-05-28T06:09:08.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:08 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-28T06:09:08.770 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.55615.asok 2024-05-28T06:09:08.771 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:09:08.771 DEBUG:teuthology.orchestra.run.smithi096:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.55615.asok status 2024-05-28T06:09:08.998 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:09:08.998 INFO:teuthology.orchestra.run.smithi096.stdout: "metadata": { 2024-05-28T06:09:08.998 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph_sha1": "9ed025fbf4406fb54980a0647506b15895b6d863", 2024-05-28T06:09:08.998 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph_version": "ceph version 19.0.0-3154-g9ed025fb (9ed025fbf4406fb54980a0647506b15895b6d863) squid (dev)", 2024-05-28T06:09:08.998 INFO:teuthology.orchestra.run.smithi096.stdout: "entity_id": "0", 2024-05-28T06:09:08.998 INFO:teuthology.orchestra.run.smithi096.stdout: "hostname": "smithi096", 2024-05-28T06:09:08.998 INFO:teuthology.orchestra.run.smithi096.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-05-28T06:09:08.998 INFO:teuthology.orchestra.run.smithi096.stdout: "pid": "55615", 2024-05-28T06:09:08.998 INFO:teuthology.orchestra.run.smithi096.stdout: "root": "/" 2024-05-28T06:09:08.998 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:09:08.999 INFO:teuthology.orchestra.run.smithi096.stdout: "dentry_count": 0, 2024-05-28T06:09:08.999 INFO:teuthology.orchestra.run.smithi096.stdout: "dentry_pinned_count": 0, 2024-05-28T06:09:08.999 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 6057, 2024-05-28T06:09:08.999 INFO:teuthology.orchestra.run.smithi096.stdout: "inst": { 2024-05-28T06:09:08.999 INFO:teuthology.orchestra.run.smithi096.stdout: "name": { 2024-05-28T06:09:08.999 INFO:teuthology.orchestra.run.smithi096.stdout: "type": "client", 2024-05-28T06:09:08.999 INFO:teuthology.orchestra.run.smithi096.stdout: "num": 6057 2024-05-28T06:09:08.999 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:09:08.999 INFO:teuthology.orchestra.run.smithi096.stdout: "addr": { 2024-05-28T06:09:08.999 INFO:teuthology.orchestra.run.smithi096.stdout: "type": "v1", 2024-05-28T06:09:08.999 INFO:teuthology.orchestra.run.smithi096.stdout: "addr": "192.168.0.1:0", 2024-05-28T06:09:08.999 INFO:teuthology.orchestra.run.smithi096.stdout: "nonce": 2931765512 2024-05-28T06:09:08.999 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:09:09.000 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:09:09.000 INFO:teuthology.orchestra.run.smithi096.stdout: "addr": { 2024-05-28T06:09:09.000 INFO:teuthology.orchestra.run.smithi096.stdout: "type": "v1", 2024-05-28T06:09:09.000 INFO:teuthology.orchestra.run.smithi096.stdout: "addr": "192.168.0.1:0", 2024-05-28T06:09:09.000 INFO:teuthology.orchestra.run.smithi096.stdout: "nonce": 2931765512 2024-05-28T06:09:09.000 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:09:09.000 INFO:teuthology.orchestra.run.smithi096.stdout: "inst_str": "client.6057 192.168.0.1:0/2931765512", 2024-05-28T06:09:09.000 INFO:teuthology.orchestra.run.smithi096.stdout: "addr_str": "192.168.0.1:0/2931765512", 2024-05-28T06:09:09.000 INFO:teuthology.orchestra.run.smithi096.stdout: "inode_count": 1, 2024-05-28T06:09:09.000 INFO:teuthology.orchestra.run.smithi096.stdout: "mds_epoch": 25, 2024-05-28T06:09:09.000 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_epoch": 45, 2024-05-28T06:09:09.000 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_epoch_barrier": 0, 2024-05-28T06:09:09.000 INFO:teuthology.orchestra.run.smithi096.stdout: "blocklisted": false, 2024-05-28T06:09:09.000 INFO:teuthology.orchestra.run.smithi096.stdout: "fs_name": "cephfs" 2024-05-28T06:09:09.001 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:09:09.013 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:09:09.013 DEBUG:teuthology.orchestra.run.smithi096:> stat --file-system '--printf=%T 2024-05-28T06:09:09.013 DEBUG:teuthology.orchestra.run.smithi096:> ' -- /home/ubuntu/cephtest/mnt.0 2024-05-28T06:09:09.067 INFO:teuthology.orchestra.run.smithi096.stdout:fuseblk 2024-05-28T06:09:09.067 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-05-28T06:09:09.067 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:09:09.067 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-05-28T06:09:09.148 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-05-28T06:09:09.355 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:09.353+0000 7f1b96200640 1 Processor -- start 2024-05-28T06:09:09.355 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:09.354+0000 7f1b96200640 1 -- start start 2024-05-28T06:09:09.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:09.355+0000 7f1b96200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1b900d1420 0x7f1b9006c740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:09.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:09.355+0000 7f1b96200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b900d2200 con 0x7f1b9015e5a0 2024-05-28T06:09:09.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:09.356+0000 7f1b96200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b900d24e0 con 0x7f1b900cfb40 2024-05-28T06:09:09.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:09.356+0000 7f1b96200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f1b900d2b30 con 0x7f1b900d1420 2024-05-28T06:09:09.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:09.356+0000 7f1b94e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f1b9015e5a0 0x7f1b900d2fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:45768/0 (socket says 172.21.15.96:45768) 2024-05-28T06:09:09.358 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:09.356+0000 7f1b94e00640 1 -- 172.21.15.96:0/2765778925 learned_addr learned my addr 172.21.15.96:0/2765778925 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:09:09.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:09.356+0000 7f1b8f400640 1 -- 172.21.15.96:0/2765778925 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1406307775 0 0) 0x7f1b900d2200 con 0x7f1b9015e5a0 2024-05-28T06:09:09.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:09.356+0000 7f1b8f400640 1 -- 172.21.15.96:0/2765778925 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1b70003520 con 0x7f1b9015e5a0 2024-05-28T06:09:09.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:09.356+0000 7f1b95800640 1 --2- 172.21.15.96:0/2765778925 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1b900d1420 0x7f1b9006c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:09.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:09.356+0000 7f1b8f400640 1 -- 172.21.15.96:0/2765778925 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 768159385 0 0) 0x7f1b70003520 con 0x7f1b9015e5a0 2024-05-28T06:09:09.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:09.356+0000 7f1b8f400640 1 -- 172.21.15.96:0/2765778925 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1b900d1420 msgr2=0x7f1b9006c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:09:09.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:09.357+0000 7f1b8f400640 1 --2- 172.21.15.96:0/2765778925 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1b900d1420 0x7f1b9006c740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:09.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:09.357+0000 7f1b8f400640 1 -- 172.21.15.96:0/2765778925 >> v1:172.21.15.123:6789/0 conn(0x7f1b900cfb40 legacy=0x7f1b900d0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:09:09.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:09.357+0000 7f1b8f400640 1 -- 172.21.15.96:0/2765778925 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b900d27c0 con 0x7f1b9015e5a0 2024-05-28T06:09:09.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:09.357+0000 7f1b8f400640 1 -- 172.21.15.96:0/2765778925 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f1b80001670 con 0x7f1b9015e5a0 2024-05-28T06:09:09.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:09.357+0000 7f1b8f400640 1 -- 172.21.15.96:0/2765778925 >> v1:172.21.15.96:6789/0 conn(0x7f1b9015e5a0 legacy=0x7f1b900d2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:09:09.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:09.357+0000 7f1b8f400640 1 --2- 172.21.15.96:0/2765778925 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1b700039e0 0x7f1b70003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:09.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:09.357+0000 7f1b8f400640 1 --2- 172.21.15.96:0/2765778925 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1b900d1420 0x7f1b70004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:09.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:09.357+0000 7f1b8f400640 1 --2- 172.21.15.96:0/2765778925 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b70004e60 0x7f1b7000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:09.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:09.357+0000 7f1b8f400640 1 -- 172.21.15.96:0/2765778925 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f1b700054e0 con 0x7f1b700039e0 2024-05-28T06:09:09.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:09.357+0000 7f1b8f400640 1 -- 172.21.15.96:0/2765778925 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f1b70005690 con 0x7f1b70004e60 2024-05-28T06:09:09.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:09.357+0000 7f1b8f400640 1 -- 172.21.15.96:0/2765778925 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f1b7000bf80 con 0x7f1b900d1420 2024-05-28T06:09:09.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:09.357+0000 7f1b94e00640 1 --2- 172.21.15.96:0/2765778925 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1b700039e0 0x7f1b70003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:09.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:09.357+0000 7f1b94e00640 1 -- 172.21.15.96:0/2765778925 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1b900d1420 msgr2=0x7f1b70004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:09:09.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:09.357+0000 7f1b94e00640 1 --2- 172.21.15.96:0/2765778925 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1b900d1420 0x7f1b70004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:09.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:09.357+0000 7f1b94e00640 1 -- 172.21.15.96:0/2765778925 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b70004e60 msgr2=0x7f1b7000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:09:09.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:09.357+0000 7f1b94e00640 1 --2- 172.21.15.96:0/2765778925 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b70004e60 0x7f1b7000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:09.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:09.357+0000 7f1b94e00640 1 -- 172.21.15.96:0/2765778925 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b70003520 con 0x7f1b700039e0 2024-05-28T06:09:09.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:09.358+0000 7f1b94e00640 1 --2- 172.21.15.96:0/2765778925 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1b700039e0 0x7f1b70003e00 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f1b8000c210 tx=0x7f1b70005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dff27da9d269c758 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:09:09.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:09.359+0000 7f1b8f400640 1 -- 172.21.15.96:0/2765778925 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b80007d30 con 0x7f1b700039e0 2024-05-28T06:09:09.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:09.359+0000 7f1b8f400640 1 -- 172.21.15.96:0/2765778925 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b80005ad0 con 0x7f1b700039e0 2024-05-28T06:09:09.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:09.359+0000 7f1b8f400640 1 -- 172.21.15.96:0/2765778925 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b80007070 con 0x7f1b700039e0 2024-05-28T06:09:09.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:09.359+0000 7f1b96200640 1 -- 172.21.15.96:0/2765778925 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1b700039e0 msgr2=0x7f1b70003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:09:09.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:09.359+0000 7f1b96200640 1 --2- 172.21.15.96:0/2765778925 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1b700039e0 0x7f1b70003e00 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f1b8000c210 tx=0x7f1b70005830 comp rx=0 tx=0).stop 2024-05-28T06:09:09.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:09.359+0000 7f1b95800640 1 -- 172.21.15.96:0/2765778925 reap_dead start 2024-05-28T06:09:09.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:09.359+0000 7f1b96200640 1 -- 172.21.15.96:0/2765778925 shutdown_connections 2024-05-28T06:09:09.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:09.359+0000 7f1b96200640 1 -- 172.21.15.96:0/2765778925 >> 172.21.15.96:0/2765778925 conn(0x7f1b9008bfc0 msgr2=0x7f1b900d44b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:09:09.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:09.360+0000 7f1b96200640 1 -- 172.21.15.96:0/2765778925 shutdown_connections 2024-05-28T06:09:09.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:09.360+0000 7f1b96200640 1 -- 172.21.15.96:0/2765778925 wait complete. 2024-05-28T06:09:09.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:09.361+0000 7f1b96200640 1 Processor -- start 2024-05-28T06:09:09.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:09.361+0000 7f1b96200640 1 -- start start 2024-05-28T06:09:09.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:09.362+0000 7f1b96200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1b700039e0 0x7f1b9015a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:09.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:09.362+0000 7f1b96200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1b900b6730 0x7f1b900b8ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:09.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:09.362+0000 7f1b96200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b900b9400 0x7f1b900b70b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:09.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:09.362+0000 7f1b96200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f1b900d53a0 con 0x7f1b900b6730 2024-05-28T06:09:09.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:09.362+0000 7f1b96200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f1b900d5220 con 0x7f1b900b9400 2024-05-28T06:09:09.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:09.362+0000 7f1b96200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f1b900cffd0 con 0x7f1b700039e0 2024-05-28T06:09:09.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:09.363+0000 7f1b8fe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1b900b6730 0x7f1b900b8ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:09.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:09.363+0000 7f1b8fe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1b900b6730 0x7f1b900b8ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54998/0 (socket says 172.21.15.96:54998) 2024-05-28T06:09:09.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:09.363+0000 7f1b8fe00640 1 -- 172.21.15.96:0/978437331 learned_addr learned my addr 172.21.15.96:0/978437331 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:09:09.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:09.363+0000 7f1b94e00640 1 --2- 172.21.15.96:0/978437331 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1b700039e0 0x7f1b9015a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:09.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:09.363+0000 7f1b8fe00640 1 -- 172.21.15.96:0/978437331 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1b700039e0 msgr2=0x7f1b9015a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:09:09.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:09.363+0000 7f1b8fe00640 1 --2- 172.21.15.96:0/978437331 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1b700039e0 0x7f1b9015a1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:09.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:09.363+0000 7f1b8fe00640 1 -- 172.21.15.96:0/978437331 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b900b9400 msgr2=0x7f1b900b70b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:09:09.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:09.363+0000 7f1b8fe00640 1 --2- 172.21.15.96:0/978437331 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b900b9400 0x7f1b900b70b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:09.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:09.363+0000 7f1b8fe00640 1 -- 172.21.15.96:0/978437331 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b900cd180 con 0x7f1b900b6730 2024-05-28T06:09:09.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:09.363+0000 7f1b8fe00640 1 --2- 172.21.15.96:0/978437331 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1b900b6730 0x7f1b900b8ec0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f1b7800c7a0 tx=0x7f1b7800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:09:09.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:09.364+0000 7f1b8cc00640 1 -- 172.21.15.96:0/978437331 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b78003b70 con 0x7f1b900b6730 2024-05-28T06:09:09.367 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:09.364+0000 7f1b8cc00640 1 -- 172.21.15.96:0/978437331 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b78003d30 con 0x7f1b900b6730 2024-05-28T06:09:09.367 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:09.364+0000 7f1b96200640 1 -- 172.21.15.96:0/978437331 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b900cd460 con 0x7f1b900b6730 2024-05-28T06:09:09.367 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:09.364+0000 7f1b8cc00640 1 -- 172.21.15.96:0/978437331 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b7800aa30 con 0x7f1b900b6730 2024-05-28T06:09:09.367 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:09.364+0000 7f1b96200640 1 -- 172.21.15.96:0/978437331 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b900cda20 con 0x7f1b900b6730 2024-05-28T06:09:09.367 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:09.365+0000 7f1b96200640 1 -- 172.21.15.96:0/978437331 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b900689e0 con 0x7f1b900b6730 2024-05-28T06:09:09.367 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:09.366+0000 7f1b8cc00640 1 -- 172.21.15.96:0/978437331 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 9) ==== 91295+0+0 (secure 0 0 0) 0x7f1b7800abd0 con 0x7f1b900b6730 2024-05-28T06:09:09.368 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:09.367+0000 7f1b8cc00640 1 --2- 172.21.15.96:0/978437331 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1b6c06cbf0 0x7f1b6c06f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:09.368 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:09.367+0000 7f1b8cc00640 1 -- 172.21.15.96:0/978437331 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5737+0+0 (secure 0 0 0) 0x7f1b7808bf30 con 0x7f1b900b6730 2024-05-28T06:09:09.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:09.368+0000 7f1b94e00640 1 --2- 172.21.15.96:0/978437331 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1b6c06cbf0 0x7f1b6c06f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:09.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:09.369+0000 7f1b94e00640 1 --2- 172.21.15.96:0/978437331 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1b6c06cbf0 0x7f1b6c06f0b0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f1b80033000 tx=0x7f1b8002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:09:09.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:09.371+0000 7f1b8cc00640 1 -- 172.21.15.96:0/978437331 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1b78059fb0 con 0x7f1b900b6730 2024-05-28T06:09:09.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:09.615+0000 7f1b96200640 1 -- 172.21.15.96:0/978437331 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) -- 0x7f1b900d1d30 con 0x7f1b900b6730 2024-05-28T06:09:10.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.164+0000 7f1b8cc00640 1 -- 172.21.15.96:0/978437331 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v26) ==== 136+0+0 (secure 0 0 0) 0x7f1b780598b0 con 0x7f1b900b6730 2024-05-28T06:09:10.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.167+0000 7f1b96200640 1 -- 172.21.15.96:0/978437331 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1b6c06cbf0 msgr2=0x7f1b6c06f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:09:10.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.167+0000 7f1b96200640 1 --2- 172.21.15.96:0/978437331 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1b6c06cbf0 0x7f1b6c06f0b0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f1b80033000 tx=0x7f1b8002e000 comp rx=0 tx=0).stop 2024-05-28T06:09:10.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.168+0000 7f1b96200640 1 -- 172.21.15.96:0/978437331 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1b900b6730 msgr2=0x7f1b900b8ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:09:10.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.168+0000 7f1b96200640 1 --2- 172.21.15.96:0/978437331 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1b900b6730 0x7f1b900b8ec0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f1b7800c7a0 tx=0x7f1b7800cc60 comp rx=0 tx=0).stop 2024-05-28T06:09:10.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.168+0000 7f1b96200640 1 -- 172.21.15.96:0/978437331 shutdown_connections 2024-05-28T06:09:10.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.168+0000 7f1b96200640 1 --2- 172.21.15.96:0/978437331 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1b6c06cbf0 0x7f1b6c06f0b0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:10.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.168+0000 7f1b96200640 1 --2- 172.21.15.96:0/978437331 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b900b9400 0x7f1b900b70b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:10.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.168+0000 7f1b96200640 1 --2- 172.21.15.96:0/978437331 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1b900b6730 0x7f1b900b8ec0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:10.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.168+0000 7f1b96200640 1 --2- 172.21.15.96:0/978437331 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1b700039e0 0x7f1b9015a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:10.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.168+0000 7f1b96200640 1 -- 172.21.15.96:0/978437331 >> 172.21.15.96:0/978437331 conn(0x7f1b9008bfc0 msgr2=0x7f1b900d44b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:09:10.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.168+0000 7f1b96200640 1 -- 172.21.15.96:0/978437331 shutdown_connections 2024-05-28T06:09:10.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.168+0000 7f1b96200640 1 -- 172.21.15.96:0/978437331 wait complete. 2024-05-28T06:09:10.192 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-05-28T06:09:10.400 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.398+0000 7f06bac00640 1 Processor -- start 2024-05-28T06:09:10.400 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.398+0000 7f06bac00640 1 -- start start 2024-05-28T06:09:10.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.399+0000 7f06bac00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f06b40d0590 0x7f06b40d0a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:10.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.399+0000 7f06bac00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f06b4099090 0x7f06b40994e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:10.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.400+0000 7f06bac00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06b40b51f0 con 0x7f06b40b4e10 2024-05-28T06:09:10.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.400+0000 7f06bac00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f06b40aef60 con 0x7f06b40d0590 2024-05-28T06:09:10.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.400+0000 7f06bac00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f06b40ad1d0 con 0x7f06b4099090 2024-05-28T06:09:10.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.400+0000 7f06b8c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f06b4099090 0x7f06b40994e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:10.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.400+0000 7f06b3e00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f06b40b4e10 0x7f06b40cde30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.96:60080/0 (socket says 172.21.15.96:60080) 2024-05-28T06:09:10.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.400+0000 7f06b3e00640 1 -- 172.21.15.96:0/1532568542 learned_addr learned my addr 172.21.15.96:0/1532568542 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:09:10.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.401+0000 7f06b3400640 1 --2- 172.21.15.96:0/1532568542 >> v2:172.21.15.123:3300/0 conn(0x7f06b40d0590 0x7f06b40d0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:10.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.401+0000 7f06b2a00640 1 -- 172.21.15.96:0/1532568542 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2122874265 0 0) 0x7f06b40b51f0 con 0x7f06b40b4e10 2024-05-28T06:09:10.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.401+0000 7f06b2a00640 1 -- 172.21.15.96:0/1532568542 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0698005730 con 0x7f06b40b4e10 2024-05-28T06:09:10.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.401+0000 7f06b8c00640 1 -- 172.21.15.96:0/1532568542 >> v2:172.21.15.123:3300/0 conn(0x7f06b40d0590 msgr2=0x7f06b40d0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:09:10.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.401+0000 7f06b8c00640 1 --2- 172.21.15.96:0/1532568542 >> v2:172.21.15.123:3300/0 conn(0x7f06b40d0590 0x7f06b40d0a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:10.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.401+0000 7f06b8c00640 1 -- 172.21.15.96:0/1532568542 >> v1:172.21.15.123:6789/0 conn(0x7f06b40b4e10 legacy=0x7f06b40cde30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:09:10.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.401+0000 7f06b8c00640 1 -- 172.21.15.96:0/1532568542 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06b40ad470 con 0x7f06b4099090 2024-05-28T06:09:10.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.403+0000 7f06b8c00640 1 --2- 172.21.15.96:0/1532568542 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f06b4099090 0x7f06b40994e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f06ac009d90 tx=0x7f06ac02a5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=31f7665b0ae4dddb server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:09:10.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.404+0000 7f06b2a00640 1 -- 172.21.15.96:0/1532568542 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06ac031600 con 0x7f06b4099090 2024-05-28T06:09:10.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.404+0000 7f06b2a00640 1 -- 172.21.15.96:0/1532568542 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06ac0317a0 con 0x7f06b4099090 2024-05-28T06:09:10.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.404+0000 7f06b2a00640 1 -- 172.21.15.96:0/1532568542 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06ac031b70 con 0x7f06b4099090 2024-05-28T06:09:10.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.404+0000 7f06bac00640 1 -- 172.21.15.96:0/1532568542 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f06b4099090 msgr2=0x7f06b40994e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:09:10.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.404+0000 7f06bac00640 1 --2- 172.21.15.96:0/1532568542 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f06b4099090 0x7f06b40994e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f06ac009d90 tx=0x7f06ac02a5d0 comp rx=0 tx=0).stop 2024-05-28T06:09:10.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.404+0000 7f06bac00640 1 -- 172.21.15.96:0/1532568542 shutdown_connections 2024-05-28T06:09:10.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.404+0000 7f06bac00640 1 --2- 172.21.15.96:0/1532568542 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f06b4099090 0x7f06b40994e0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:10.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.404+0000 7f06bac00640 1 --2- 172.21.15.96:0/1532568542 >> v2:172.21.15.123:3300/0 conn(0x7f06b40d0590 0x7f06b40d0a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:10.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.404+0000 7f06bac00640 1 -- 172.21.15.96:0/1532568542 >> 172.21.15.96:0/1532568542 conn(0x7f06b406bae0 msgr2=0x7f06b406bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:09:10.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.404+0000 7f06bac00640 1 -- 172.21.15.96:0/1532568542 shutdown_connections 2024-05-28T06:09:10.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.405+0000 7f06bac00640 1 -- 172.21.15.96:0/1532568542 wait complete. 2024-05-28T06:09:10.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.407+0000 7f06bac00640 1 Processor -- start 2024-05-28T06:09:10.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.407+0000 7f06bac00640 1 -- start start 2024-05-28T06:09:10.410 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.408+0000 7f06bac00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f06b4099090 0x7f06b415e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:10.410 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.408+0000 7f06bac00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f06b40b4e10 0x7f06b415bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:10.410 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.408+0000 7f06bac00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f06b40d0590 0x7f06b41f3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:10.410 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.408+0000 7f06bac00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f06b40ce880 con 0x7f06b4099090 2024-05-28T06:09:10.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.408+0000 7f06bac00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f06b40ce540 con 0x7f06b40b4e10 2024-05-28T06:09:10.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.408+0000 7f06bac00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f06b40cea00 con 0x7f06b40d0590 2024-05-28T06:09:10.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.408+0000 7f06b3e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f06b4099090 0x7f06b415e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:10.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.408+0000 7f06b3e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f06b4099090 0x7f06b415e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:55004/0 (socket says 172.21.15.96:55004) 2024-05-28T06:09:10.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.408+0000 7f06b3e00640 1 -- 172.21.15.96:0/923955664 learned_addr learned my addr 172.21.15.96:0/923955664 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:09:10.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.409+0000 7f06b8c00640 1 --2- 172.21.15.96:0/923955664 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f06b40d0590 0x7f06b41f3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:10.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.409+0000 7f06b3400640 1 --2- 172.21.15.96:0/923955664 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f06b40b4e10 0x7f06b415bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:10.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.409+0000 7f06b3e00640 1 -- 172.21.15.96:0/923955664 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f06b40d0590 msgr2=0x7f06b41f3bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:09:10.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.409+0000 7f06b3e00640 1 --2- 172.21.15.96:0/923955664 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f06b40d0590 0x7f06b41f3bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:10.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.409+0000 7f06b3e00640 1 -- 172.21.15.96:0/923955664 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f06b40b4e10 msgr2=0x7f06b415bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:09:10.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.409+0000 7f06b3e00640 1 --2- 172.21.15.96:0/923955664 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f06b40b4e10 0x7f06b415bb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:10.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.409+0000 7f06b3e00640 1 -- 172.21.15.96:0/923955664 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06b415c2d0 con 0x7f06b4099090 2024-05-28T06:09:10.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.409+0000 7f06b3e00640 1 --2- 172.21.15.96:0/923955664 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f06b4099090 0x7f06b415e590 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f06a400c500 tx=0x7f06a400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:09:10.412 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.409+0000 7f06abe00640 1 -- 172.21.15.96:0/923955664 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06a4017070 con 0x7f06b4099090 2024-05-28T06:09:10.412 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.409+0000 7f06abe00640 1 -- 172.21.15.96:0/923955664 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06a4005ce0 con 0x7f06b4099090 2024-05-28T06:09:10.412 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.409+0000 7f06abe00640 1 -- 172.21.15.96:0/923955664 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06a4007330 con 0x7f06b4099090 2024-05-28T06:09:10.412 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.409+0000 7f06bac00640 1 -- 172.21.15.96:0/923955664 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06b415c5b0 con 0x7f06b4099090 2024-05-28T06:09:10.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.410+0000 7f06bac00640 1 -- 172.21.15.96:0/923955664 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06b415cb70 con 0x7f06b4099090 2024-05-28T06:09:10.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.411+0000 7f06bac00640 1 -- 172.21.15.96:0/923955664 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06b40689e0 con 0x7f06b4099090 2024-05-28T06:09:10.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.412+0000 7f06abe00640 1 -- 172.21.15.96:0/923955664 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 9) ==== 91295+0+0 (secure 0 0 0) 0x7f06a401a070 con 0x7f06b4099090 2024-05-28T06:09:10.414 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.412+0000 7f06abe00640 1 --2- 172.21.15.96:0/923955664 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f069406cbc0 0x7f069406f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:10.414 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.413+0000 7f06abe00640 1 -- 172.21.15.96:0/923955664 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5737+0+0 (secure 0 0 0) 0x7f06a408b530 con 0x7f06b4099090 2024-05-28T06:09:10.414 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.413+0000 7f06b3400640 1 --2- 172.21.15.96:0/923955664 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f069406cbc0 0x7f069406f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:10.415 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.414+0000 7f06b3400640 1 --2- 172.21.15.96:0/923955664 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f069406cbc0 0x7f069406f080 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f06b41f78b0 tx=0x7f06a0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:09:10.418 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.417+0000 7f06abe00640 1 -- 172.21.15.96:0/923955664 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f06a4059620 con 0x7f06b4099090 2024-05-28T06:09:10.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.644+0000 7f06bac00640 1 -- 172.21.15.96:0/923955664 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7f06b406f540 con 0x7f069406cbc0 2024-05-28T06:09:10.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.649+0000 7f06abe00640 1 -- 172.21.15.96:0/923955664 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7f06b406f540 con 0x7f069406cbc0 2024-05-28T06:09:10.650 INFO:teuthology.orchestra.run.smithi096.stdout:[ 2024-05-28T06:09:10.650 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:09:10.650 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "cephfs" 2024-05-28T06:09:10.650 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:09:10.650 INFO:teuthology.orchestra.run.smithi096.stdout:] 2024-05-28T06:09:10.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.651+0000 7f06bac00640 1 -- 172.21.15.96:0/923955664 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f069406cbc0 msgr2=0x7f069406f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:09:10.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.651+0000 7f06bac00640 1 --2- 172.21.15.96:0/923955664 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f069406cbc0 0x7f069406f080 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f06b41f78b0 tx=0x7f06a0008040 comp rx=0 tx=0).stop 2024-05-28T06:09:10.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.651+0000 7f06bac00640 1 -- 172.21.15.96:0/923955664 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f06b4099090 msgr2=0x7f06b415e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:09:10.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.651+0000 7f06bac00640 1 --2- 172.21.15.96:0/923955664 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f06b4099090 0x7f06b415e590 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f06a400c500 tx=0x7f06a400c9c0 comp rx=0 tx=0).stop 2024-05-28T06:09:10.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.651+0000 7f06bac00640 1 -- 172.21.15.96:0/923955664 shutdown_connections 2024-05-28T06:09:10.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.651+0000 7f06bac00640 1 --2- 172.21.15.96:0/923955664 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f069406cbc0 0x7f069406f080 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:10.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.651+0000 7f06bac00640 1 --2- 172.21.15.96:0/923955664 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f06b40d0590 0x7f06b41f3bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:10.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.651+0000 7f06bac00640 1 --2- 172.21.15.96:0/923955664 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f06b40b4e10 0x7f06b415bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:10.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.651+0000 7f06bac00640 1 --2- 172.21.15.96:0/923955664 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f06b4099090 0x7f06b415e590 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:10.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.651+0000 7f06bac00640 1 -- 172.21.15.96:0/923955664 >> 172.21.15.96:0/923955664 conn(0x7f06b406bae0 msgr2=0x7f06b40ca940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:09:10.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.651+0000 7f06bac00640 1 -- 172.21.15.96:0/923955664 shutdown_connections 2024-05-28T06:09:10.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.651+0000 7f06bac00640 1 -- 172.21.15.96:0/923955664 wait complete. 2024-05-28T06:09:10.676 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-05-28T06:09:10.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.881+0000 7f97cea00640 1 Processor -- start 2024-05-28T06:09:10.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.882+0000 7f97cea00640 1 -- start start 2024-05-28T06:09:10.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.883+0000 7f97cea00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f97c80d16a0 0x7f97c80cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:10.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.883+0000 7f97cea00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f97c80cf870 0x7f97c80d25e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:10.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.883+0000 7f97cea00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97c815ec50 con 0x7f97c815e870 2024-05-28T06:09:10.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.883+0000 7f97cea00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f97c80d2150 con 0x7f97c80cf870 2024-05-28T06:09:10.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.883+0000 7f97cea00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f97c80cfd20 con 0x7f97c80d16a0 2024-05-28T06:09:10.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.883+0000 7f97cd600640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f97c815e870 0x7f97c80d0f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:45772/0 (socket says 172.21.15.96:45772) 2024-05-28T06:09:10.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.883+0000 7f97cd600640 1 -- 172.21.15.96:0/4124781152 learned_addr learned my addr 172.21.15.96:0/4124781152 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:09:10.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.883+0000 7f97ce000640 1 --2- 172.21.15.96:0/4124781152 >> v2:172.21.15.96:3300/0 conn(0x7f97c80cf870 0x7f97c80d25e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:10.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.884+0000 7f97ccc00640 1 --2- 172.21.15.96:0/4124781152 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f97c80d16a0 0x7f97c80cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:10.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.884+0000 7f97ce000640 1 -- 172.21.15.96:0/4124781152 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f97c80d16a0 msgr2=0x7f97c80cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:09:10.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.884+0000 7f97ce000640 1 --2- 172.21.15.96:0/4124781152 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f97c80d16a0 0x7f97c80cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:10.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.884+0000 7f97ce000640 1 -- 172.21.15.96:0/4124781152 >> v1:172.21.15.96:6789/0 conn(0x7f97c815e870 legacy=0x7f97c80d0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:09:10.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.884+0000 7f97ce000640 1 -- 172.21.15.96:0/4124781152 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97c80cff80 con 0x7f97c80cf870 2024-05-28T06:09:10.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.885+0000 7f97ce000640 1 --2- 172.21.15.96:0/4124781152 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97c80cf870 0x7f97c80d25e0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f97c0009d90 tx=0x7f97c002a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d325fc6d793362f server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:09:10.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.885+0000 7f97c7e00640 1 -- 172.21.15.96:0/4124781152 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97c0007740 con 0x7f97c80cf870 2024-05-28T06:09:10.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.885+0000 7f97c7e00640 1 -- 172.21.15.96:0/4124781152 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97c00078e0 con 0x7f97c80cf870 2024-05-28T06:09:10.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.885+0000 7f97c7e00640 1 -- 172.21.15.96:0/4124781152 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97c0007cb0 con 0x7f97c80cf870 2024-05-28T06:09:10.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.886+0000 7f97cea00640 1 -- 172.21.15.96:0/4124781152 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97c80cf870 msgr2=0x7f97c80d25e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:09:10.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.886+0000 7f97cea00640 1 --2- 172.21.15.96:0/4124781152 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97c80cf870 0x7f97c80d25e0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f97c0009d90 tx=0x7f97c002a660 comp rx=0 tx=0).stop 2024-05-28T06:09:10.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.886+0000 7f97cea00640 1 -- 172.21.15.96:0/4124781152 shutdown_connections 2024-05-28T06:09:10.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.886+0000 7f97cea00640 1 --2- 172.21.15.96:0/4124781152 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97c80cf870 0x7f97c80d25e0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:10.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.886+0000 7f97cea00640 1 --2- 172.21.15.96:0/4124781152 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f97c80d16a0 0x7f97c80cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:10.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.886+0000 7f97cea00640 1 -- 172.21.15.96:0/4124781152 >> 172.21.15.96:0/4124781152 conn(0x7f97c808bfc0 msgr2=0x7f97c80d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:09:10.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.886+0000 7f97cea00640 1 -- 172.21.15.96:0/4124781152 shutdown_connections 2024-05-28T06:09:10.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.886+0000 7f97cea00640 1 -- 172.21.15.96:0/4124781152 wait complete. 2024-05-28T06:09:10.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.888+0000 7f97cea00640 1 Processor -- start 2024-05-28T06:09:10.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.888+0000 7f97cea00640 1 -- start start 2024-05-28T06:09:10.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.889+0000 7f97cea00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97c80cf870 0x7f97c80cd450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:10.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.889+0000 7f97cea00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f97c80d16a0 0x7f97c81599e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:10.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.889+0000 7f97cea00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f97c815e870 0x7f97c80b9370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:10.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.889+0000 7f97cea00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f97c80d0820 con 0x7f97c80cf870 2024-05-28T06:09:10.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.889+0000 7f97cea00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f97c80d0540 con 0x7f97c80d16a0 2024-05-28T06:09:10.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.889+0000 7f97cea00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f97c80d4fb0 con 0x7f97c815e870 2024-05-28T06:09:10.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.889+0000 7f97cd600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97c80cf870 0x7f97c80cd450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:10.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.889+0000 7f97cd600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97c80cf870 0x7f97c80cd450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:55018/0 (socket says 172.21.15.96:55018) 2024-05-28T06:09:10.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.889+0000 7f97cd600640 1 -- 172.21.15.96:0/321598371 learned_addr learned my addr 172.21.15.96:0/321598371 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:09:10.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.890+0000 7f97cd600640 1 -- 172.21.15.96:0/321598371 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f97c815e870 msgr2=0x7f97c80b9370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:09:10.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.890+0000 7f97cd600640 1 --2- 172.21.15.96:0/321598371 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f97c815e870 0x7f97c80b9370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:10.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.890+0000 7f97cd600640 1 -- 172.21.15.96:0/321598371 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f97c80d16a0 msgr2=0x7f97c81599e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:09:10.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.890+0000 7f97cd600640 1 --2- 172.21.15.96:0/321598371 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f97c80d16a0 0x7f97c81599e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:10.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.890+0000 7f97cd600640 1 -- 172.21.15.96:0/321598371 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97c80cb650 con 0x7f97c80cf870 2024-05-28T06:09:10.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.890+0000 7f97cd600640 1 --2- 172.21.15.96:0/321598371 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97c80cf870 0x7f97c80cd450 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f97bc00c3c0 tx=0x7f97bc00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:09:10.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.890+0000 7f97c5600640 1 -- 172.21.15.96:0/321598371 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97bc017070 con 0x7f97c80cf870 2024-05-28T06:09:10.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.890+0000 7f97c5600640 1 -- 172.21.15.96:0/321598371 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97bc005ce0 con 0x7f97c80cf870 2024-05-28T06:09:10.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.890+0000 7f97cea00640 1 -- 172.21.15.96:0/321598371 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97c80cb8f0 con 0x7f97c80cf870 2024-05-28T06:09:10.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.890+0000 7f97c5600640 1 -- 172.21.15.96:0/321598371 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97bc0072f0 con 0x7f97c80cf870 2024-05-28T06:09:10.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.890+0000 7f97cea00640 1 -- 172.21.15.96:0/321598371 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97c80cbeb0 con 0x7f97c80cf870 2024-05-28T06:09:10.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.891+0000 7f97cea00640 1 -- 172.21.15.96:0/321598371 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97c80689e0 con 0x7f97c80cf870 2024-05-28T06:09:10.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.893+0000 7f97c5600640 1 -- 172.21.15.96:0/321598371 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 9) ==== 91295+0+0 (secure 0 0 0) 0x7f97bc01a070 con 0x7f97c80cf870 2024-05-28T06:09:10.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.894+0000 7f97c5600640 1 --2- 172.21.15.96:0/321598371 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f97ac06cbc0 0x7f97ac06f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:10.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.894+0000 7f97c5600640 1 -- 172.21.15.96:0/321598371 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5737+0+0 (secure 0 0 0) 0x7f97bc08b3e0 con 0x7f97c80cf870 2024-05-28T06:09:10.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.895+0000 7f97ccc00640 1 --2- 172.21.15.96:0/321598371 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f97ac06cbc0 0x7f97ac06f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:10.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.896+0000 7f97ccc00640 1 --2- 172.21.15.96:0/321598371 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f97ac06cbc0 0x7f97ac06f080 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f97b8006e20 tx=0x7f97b8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:09:10.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:10.898+0000 7f97c5600640 1 -- 172.21.15.96:0/321598371 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f97bc059450 con 0x7f97c80cf870 2024-05-28T06:09:11.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.107+0000 7f97cea00640 1 -- 172.21.15.96:0/321598371 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f97c80b1340 con 0x7f97c80cf870 2024-05-28T06:09:11.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.108+0000 7f97c5600640 1 -- 172.21.15.96:0/321598371 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v12)=0 v12) ==== 122+0+0 (secure 0 0 0) 0x7f97bc058d50 con 0x7f97c80cf870 2024-05-28T06:09:11.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.110+0000 7f97cea00640 1 -- 172.21.15.96:0/321598371 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f97ac06cbc0 msgr2=0x7f97ac06f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:09:11.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.111+0000 7f97cea00640 1 --2- 172.21.15.96:0/321598371 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f97ac06cbc0 0x7f97ac06f080 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f97b8006e20 tx=0x7f97b8008040 comp rx=0 tx=0).stop 2024-05-28T06:09:11.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.111+0000 7f97cea00640 1 -- 172.21.15.96:0/321598371 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97c80cf870 msgr2=0x7f97c80cd450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:09:11.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.111+0000 7f97cea00640 1 --2- 172.21.15.96:0/321598371 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97c80cf870 0x7f97c80cd450 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f97bc00c3c0 tx=0x7f97bc00c880 comp rx=0 tx=0).stop 2024-05-28T06:09:11.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.111+0000 7f97cea00640 1 -- 172.21.15.96:0/321598371 shutdown_connections 2024-05-28T06:09:11.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.111+0000 7f97cea00640 1 --2- 172.21.15.96:0/321598371 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f97ac06cbc0 0x7f97ac06f080 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:11.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.111+0000 7f97cea00640 1 --2- 172.21.15.96:0/321598371 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f97c815e870 0x7f97c80b9370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:11.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.111+0000 7f97cea00640 1 --2- 172.21.15.96:0/321598371 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f97c80d16a0 0x7f97c81599e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:11.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.111+0000 7f97cea00640 1 --2- 172.21.15.96:0/321598371 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97c80cf870 0x7f97c80cd450 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:11.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.111+0000 7f97cea00640 1 -- 172.21.15.96:0/321598371 >> 172.21.15.96:0/321598371 conn(0x7f97c808bfc0 msgr2=0x7f97c80b2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:09:11.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.111+0000 7f97cea00640 1 -- 172.21.15.96:0/321598371 shutdown_connections 2024-05-28T06:09:11.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.111+0000 7f97cea00640 1 -- 172.21.15.96:0/321598371 wait complete. 2024-05-28T06:09:11.130 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv1 --mode=777 2024-05-28T06:09:11.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.337+0000 7f6fa3000640 1 Processor -- start 2024-05-28T06:09:11.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.337+0000 7f6fa3000640 1 -- start start 2024-05-28T06:09:11.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.338+0000 7f6fa3000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f6f9c0b3a90 0x7f6f9c0d0320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:11.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.339+0000 7f6fa3000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f6f9c0723e0 0x7f6f9c072830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:11.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.339+0000 7f6fa3000640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f9c072d70 con 0x7f6f9c155180 2024-05-28T06:09:11.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.339+0000 7f6fa3000640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f6f9c15c000 con 0x7f6f9c0b3a90 2024-05-28T06:09:11.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.339+0000 7f6fa3000640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f6f9c0b4c20 con 0x7f6f9c0723e0 2024-05-28T06:09:11.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.339+0000 7f6fa1c00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f6f9c0b3a90 0x7f6f9c0d0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:11.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.339+0000 7f6fa1c00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f6f9c0b3a90 0x7f6f9c0d0320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:55028/0 (socket says 172.21.15.96:55028) 2024-05-28T06:09:11.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.339+0000 7f6fa1c00640 1 -- 172.21.15.96:0/1640066176 learned_addr learned my addr 172.21.15.96:0/1640066176 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:09:11.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.339+0000 7f6fa2600640 1 --2- 172.21.15.96:0/1640066176 >> v2:172.21.15.123:3300/0 conn(0x7f6f9c0723e0 0x7f6f9c072830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:11.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.340+0000 7f6fa1c00640 1 -- 172.21.15.96:0/1640066176 >> v2:172.21.15.123:3300/0 conn(0x7f6f9c0723e0 msgr2=0x7f6f9c072830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:09:11.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.340+0000 7f6fa1c00640 1 --2- 172.21.15.96:0/1640066176 >> v2:172.21.15.123:3300/0 conn(0x7f6f9c0723e0 0x7f6f9c072830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:11.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.340+0000 7f6fa1c00640 1 -- 172.21.15.96:0/1640066176 >> v1:172.21.15.96:6789/0 conn(0x7f6f9c155180 legacy=0x7f6f9c15b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:09:11.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.340+0000 7f6fa1c00640 1 -- 172.21.15.96:0/1640066176 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f9c0b4010 con 0x7f6f9c0b3a90 2024-05-28T06:09:11.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.340+0000 7f6f9be00640 1 -- 172.21.15.96:0/1640066176 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1363741681 0 0) 0x7f6f88001110 con 0x7f6f9c155180 2024-05-28T06:09:11.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.340+0000 7f6fa2600640 1 --2- 172.21.15.96:0/1640066176 >> v2:172.21.15.123:3300/0 conn(0x7f6f9c0723e0 0x7f6f9c072830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:09:11.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.341+0000 7f6fa1c00640 1 --2- 172.21.15.96:0/1640066176 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6f9c0b3a90 0x7f6f9c0d0320 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c0cc700 tx=0x7f6f9002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d7fcb456bf6b918 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:09:11.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.341+0000 7f6f9be00640 1 -- 172.21.15.96:0/1640066176 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f90037070 con 0x7f6f9c0b3a90 2024-05-28T06:09:11.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.341+0000 7f6f9be00640 1 -- 172.21.15.96:0/1640066176 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f90032440 con 0x7f6f9c0b3a90 2024-05-28T06:09:11.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.341+0000 7f6f9be00640 1 -- 172.21.15.96:0/1640066176 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f90032810 con 0x7f6f9c0b3a90 2024-05-28T06:09:11.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.342+0000 7f6fa3000640 1 -- 172.21.15.96:0/1640066176 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6f9c0b3a90 msgr2=0x7f6f9c0d0320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:09:11.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.342+0000 7f6fa3000640 1 --2- 172.21.15.96:0/1640066176 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6f9c0b3a90 0x7f6f9c0d0320 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c0cc700 tx=0x7f6f9002a270 comp rx=0 tx=0).stop 2024-05-28T06:09:11.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.342+0000 7f6fa3000640 1 -- 172.21.15.96:0/1640066176 shutdown_connections 2024-05-28T06:09:11.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.342+0000 7f6fa3000640 1 --2- 172.21.15.96:0/1640066176 >> v2:172.21.15.123:3300/0 conn(0x7f6f9c0723e0 0x7f6f9c072830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:11.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.342+0000 7f6fa3000640 1 --2- 172.21.15.96:0/1640066176 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6f9c0b3a90 0x7f6f9c0d0320 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:11.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.342+0000 7f6fa3000640 1 -- 172.21.15.96:0/1640066176 >> 172.21.15.96:0/1640066176 conn(0x7f6f9c06bae0 msgr2=0x7f6f9c06bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:09:11.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.342+0000 7f6fa3000640 1 -- 172.21.15.96:0/1640066176 shutdown_connections 2024-05-28T06:09:11.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.342+0000 7f6fa3000640 1 -- 172.21.15.96:0/1640066176 wait complete. 2024-05-28T06:09:11.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.344+0000 7f6fa3000640 1 Processor -- start 2024-05-28T06:09:11.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.344+0000 7f6fa3000640 1 -- start start 2024-05-28T06:09:11.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.345+0000 7f6fa3000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f9c0723e0 0x7f6f9c14e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:11.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.345+0000 7f6fa3000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6f9c0b3a90 0x7f6f9c13b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:11.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.345+0000 7f6fa3000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6f9c155180 0x7f6f9c151b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:11.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.345+0000 7f6fa3000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f6f9c0b48b0 con 0x7f6f9c155180 2024-05-28T06:09:11.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.345+0000 7f6fa3000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f6f9c0b45d0 con 0x7f6f9c0723e0 2024-05-28T06:09:11.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.345+0000 7f6fa3000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f6f9c0b50b0 con 0x7f6f9c0b3a90 2024-05-28T06:09:11.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.345+0000 7f6fa2600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6f9c155180 0x7f6f9c151b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:11.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.345+0000 7f6fa2600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6f9c155180 0x7f6f9c151b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:55044/0 (socket says 172.21.15.96:55044) 2024-05-28T06:09:11.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.345+0000 7f6fa2600640 1 -- 172.21.15.96:0/7679919 learned_addr learned my addr 172.21.15.96:0/7679919 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:09:11.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.345+0000 7f6fa1c00640 1 --2- 172.21.15.96:0/7679919 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f9c0723e0 0x7f6f9c14e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:11.349 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.345+0000 7f6fa2600640 1 -- 172.21.15.96:0/7679919 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6f9c0b3a90 msgr2=0x7f6f9c13b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:09:11.349 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.345+0000 7f6fa1200640 1 --2- 172.21.15.96:0/7679919 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6f9c0b3a90 0x7f6f9c13b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:11.349 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.346+0000 7f6fa2600640 1 --2- 172.21.15.96:0/7679919 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6f9c0b3a90 0x7f6f9c13b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:11.349 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.346+0000 7f6fa2600640 1 -- 172.21.15.96:0/7679919 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f9c0723e0 msgr2=0x7f6f9c14e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:09:11.349 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.346+0000 7f6fa2600640 1 --2- 172.21.15.96:0/7679919 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f9c0723e0 0x7f6f9c14e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:11.349 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.346+0000 7f6fa2600640 1 -- 172.21.15.96:0/7679919 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f9c0b07a0 con 0x7f6f9c155180 2024-05-28T06:09:11.349 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.346+0000 7f6fa1c00640 1 --2- 172.21.15.96:0/7679919 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f9c0723e0 0x7f6f9c14e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:09:11.349 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.346+0000 7f6fa1200640 1 --2- 172.21.15.96:0/7679919 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6f9c0b3a90 0x7f6f9c13b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:09:11.349 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.346+0000 7f6fa2600640 1 --2- 172.21.15.96:0/7679919 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6f9c155180 0x7f6f9c151b30 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f6f9400cef0 tx=0x7f6f9400a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:09:11.349 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.346+0000 7f6f99600640 1 -- 172.21.15.96:0/7679919 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f9401a070 con 0x7f6f9c155180 2024-05-28T06:09:11.349 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.346+0000 7f6f99600640 1 -- 172.21.15.96:0/7679919 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f9400d860 con 0x7f6f9c155180 2024-05-28T06:09:11.349 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.346+0000 7f6f99600640 1 -- 172.21.15.96:0/7679919 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f9400dc50 con 0x7f6f9c155180 2024-05-28T06:09:11.350 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.346+0000 7f6fa3000640 1 -- 172.21.15.96:0/7679919 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f9c0b0a80 con 0x7f6f9c155180 2024-05-28T06:09:11.350 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.346+0000 7f6fa3000640 1 -- 172.21.15.96:0/7679919 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f9c0b1040 con 0x7f6f9c155180 2024-05-28T06:09:11.350 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.347+0000 7f6fa3000640 1 -- 172.21.15.96:0/7679919 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f9c0689e0 con 0x7f6f9c155180 2024-05-28T06:09:11.350 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.348+0000 7f6f99600640 1 -- 172.21.15.96:0/7679919 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 9) ==== 91295+0+0 (secure 0 0 0) 0x7f6f9401d070 con 0x7f6f9c155180 2024-05-28T06:09:11.351 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.349+0000 7f6f99600640 1 --2- 172.21.15.96:0/7679919 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6f7c06cbc0 0x7f6f7c06f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:11.351 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.349+0000 7f6f99600640 1 -- 172.21.15.96:0/7679919 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5737+0+0 (secure 0 0 0) 0x7f6f9408c920 con 0x7f6f9c155180 2024-05-28T06:09:11.351 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.350+0000 7f6fa1c00640 1 --2- 172.21.15.96:0/7679919 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6f7c06cbc0 0x7f6f7c06f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:11.352 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.351+0000 7f6fa1c00640 1 --2- 172.21.15.96:0/7679919 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6f7c06cbc0 0x7f6f7c06f080 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f6f90034000 tx=0x7f6f9002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:09:11.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.354+0000 7f6f99600640 1 -- 172.21.15.96:0/7679919 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6f9405aa10 con 0x7f6f9c155180 2024-05-28T06:09:11.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.586+0000 7f6fa3000640 1 -- 172.21.15.96:0/7679919 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv1", "mode": "777", "target": ["mon-mgr", ""]}) -- 0x7f6f9c0b34a0 con 0x7f6f7c06cbc0 2024-05-28T06:09:11.603 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:09:11.603+0000 7fea59e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:09:11.603 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:09:11.603+0000 7fea59e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:09:11.603 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:09:11.603+0000 7fea59e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:09:11.604 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:09:11.603+0000 7fea59e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:09:11.604 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:09:11.603+0000 7fea59e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:09:11.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.675+0000 7f6f99600640 1 -- 172.21.15.96:0/7679919 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f6f9c0b34a0 con 0x7f6f7c06cbc0 2024-05-28T06:09:11.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.678+0000 7f6fa3000640 1 -- 172.21.15.96:0/7679919 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6f7c06cbc0 msgr2=0x7f6f7c06f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:09:11.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.678+0000 7f6fa3000640 1 --2- 172.21.15.96:0/7679919 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6f7c06cbc0 0x7f6f7c06f080 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f6f90034000 tx=0x7f6f9002e000 comp rx=0 tx=0).stop 2024-05-28T06:09:11.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.678+0000 7f6fa3000640 1 -- 172.21.15.96:0/7679919 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6f9c155180 msgr2=0x7f6f9c151b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:09:11.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.678+0000 7f6fa3000640 1 --2- 172.21.15.96:0/7679919 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6f9c155180 0x7f6f9c151b30 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f6f9400cef0 tx=0x7f6f9400a710 comp rx=0 tx=0).stop 2024-05-28T06:09:11.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.678+0000 7f6fa3000640 1 -- 172.21.15.96:0/7679919 shutdown_connections 2024-05-28T06:09:11.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.678+0000 7f6fa3000640 1 --2- 172.21.15.96:0/7679919 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6f7c06cbc0 0x7f6f7c06f080 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:11.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.678+0000 7f6fa3000640 1 --2- 172.21.15.96:0/7679919 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6f9c155180 0x7f6f9c151b30 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:11.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.678+0000 7f6fa3000640 1 --2- 172.21.15.96:0/7679919 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6f9c0b3a90 0x7f6f9c13b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:11.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.678+0000 7f6fa3000640 1 --2- 172.21.15.96:0/7679919 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f9c0723e0 0x7f6f9c14e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:11.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.678+0000 7f6fa3000640 1 -- 172.21.15.96:0/7679919 >> 172.21.15.96:0/7679919 conn(0x7f6f9c06bae0 msgr2=0x7f6f9c0af460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:09:11.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.678+0000 7f6fa3000640 1 -- 172.21.15.96:0/7679919 shutdown_connections 2024-05-28T06:09:11.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.679+0000 7f6fa3000640 1 -- 172.21.15.96:0/7679919 wait complete. 2024-05-28T06:09:11.703 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-05-28T06:09:11.905 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.899+0000 7fcbc7e00640 1 Processor -- start 2024-05-28T06:09:11.905 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.899+0000 7fcbc7e00640 1 -- start start 2024-05-28T06:09:11.905 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.901+0000 7fcbc7e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcbc815d270 0x7fcbc80b4d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:11.905 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.901+0000 7fcbc7e00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcbc815d900 con 0x7fcbc80cd2a0 2024-05-28T06:09:11.905 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.901+0000 7fcbc7e00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcbc80b3f90 con 0x7fcbc8153510 2024-05-28T06:09:11.905 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.901+0000 7fcbc7e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fcbc81538f0 con 0x7fcbc815d270 2024-05-28T06:09:11.905 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.901+0000 7fcbc6000640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fcbc80cd2a0 0x7fcbc815cb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:45794/0 (socket says 172.21.15.96:45794) 2024-05-28T06:09:11.905 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.901+0000 7fcbc6000640 1 -- 172.21.15.96:0/1097598942 learned_addr learned my addr 172.21.15.96:0/1097598942 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:09:11.906 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.901+0000 7fcbc7400640 1 --2- 172.21.15.96:0/1097598942 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcbc815d270 0x7fcbc80b4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:11.906 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.902+0000 7fcbc5600640 1 -- 172.21.15.96:0/1097598942 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 6251577 0 0) 0x7fcbc815d900 con 0x7fcbc80cd2a0 2024-05-28T06:09:11.906 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.902+0000 7fcbc5600640 1 -- 172.21.15.96:0/1097598942 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcbac003520 con 0x7fcbc80cd2a0 2024-05-28T06:09:11.906 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.902+0000 7fcbc5600640 1 -- 172.21.15.96:0/1097598942 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2827876906 0 0) 0x7fcbac003520 con 0x7fcbc80cd2a0 2024-05-28T06:09:11.906 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.902+0000 7fcbc5600640 1 -- 172.21.15.96:0/1097598942 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcbc815d270 msgr2=0x7fcbc80b4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:09:11.906 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.902+0000 7fcbc5600640 1 --2- 172.21.15.96:0/1097598942 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcbc815d270 0x7fcbc80b4d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:11.906 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.902+0000 7fcbc5600640 1 -- 172.21.15.96:0/1097598942 >> v1:172.21.15.123:6789/0 conn(0x7fcbc8153510 legacy=0x7fcbc80ccb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:09:11.906 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.902+0000 7fcbc5600640 1 -- 172.21.15.96:0/1097598942 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbc80b4270 con 0x7fcbc80cd2a0 2024-05-28T06:09:11.906 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.902+0000 7fcbc5600640 1 -- 172.21.15.96:0/1097598942 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7fcbb4001670 con 0x7fcbc80cd2a0 2024-05-28T06:09:11.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.902+0000 7fcbc5600640 1 -- 172.21.15.96:0/1097598942 >> v1:172.21.15.96:6789/0 conn(0x7fcbc80cd2a0 legacy=0x7fcbc815cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:09:11.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.902+0000 7fcbc5600640 1 --2- 172.21.15.96:0/1097598942 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcbc815d270 0x7fcbac0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:11.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.902+0000 7fcbc5600640 1 --2- 172.21.15.96:0/1097598942 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcbac003f20 0x7fcbac004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:11.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.902+0000 7fcbc5600640 1 --2- 172.21.15.96:0/1097598942 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcbac004d40 0x7fcbac0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:11.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.902+0000 7fcbc5600640 1 -- 172.21.15.96:0/1097598942 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fcbac00baa0 con 0x7fcbac004d40 2024-05-28T06:09:11.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.902+0000 7fcbc5600640 1 -- 172.21.15.96:0/1097598942 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fcbac00bc50 con 0x7fcbac003f20 2024-05-28T06:09:11.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.902+0000 7fcbc5600640 1 -- 172.21.15.96:0/1097598942 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fcbac00be50 con 0x7fcbc815d270 2024-05-28T06:09:11.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.903+0000 7fcbc7400640 1 --2- 172.21.15.96:0/1097598942 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcbac004d40 0x7fcbac0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:11.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.903+0000 7fcbc6000640 1 --2- 172.21.15.96:0/1097598942 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcbac003f20 0x7fcbac004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:11.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.903+0000 7fcbc6a00640 1 --2- 172.21.15.96:0/1097598942 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcbc815d270 0x7fcbac0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:11.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.903+0000 7fcbc7400640 1 -- 172.21.15.96:0/1097598942 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcbc815d270 msgr2=0x7fcbac0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:09:11.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.903+0000 7fcbc7400640 1 --2- 172.21.15.96:0/1097598942 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcbc815d270 0x7fcbac0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:11.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.903+0000 7fcbc7400640 1 -- 172.21.15.96:0/1097598942 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcbac003f20 msgr2=0x7fcbac004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:09:11.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.903+0000 7fcbc7400640 1 --2- 172.21.15.96:0/1097598942 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcbac003f20 0x7fcbac004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:11.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.903+0000 7fcbc7400640 1 -- 172.21.15.96:0/1097598942 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbac003520 con 0x7fcbac004d40 2024-05-28T06:09:11.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.904+0000 7fcbc7400640 1 --2- 172.21.15.96:0/1097598942 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcbac004d40 0x7fcbac0051c0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fcbc0009fc0 tx=0x7fcbac005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f0c2ff49612588 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:09:11.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.904+0000 7fcbc5600640 1 -- 172.21.15.96:0/1097598942 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbc0009c10 con 0x7fcbac004d40 2024-05-28T06:09:11.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.904+0000 7fcbc5600640 1 -- 172.21.15.96:0/1097598942 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbc0036070 con 0x7fcbac004d40 2024-05-28T06:09:11.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.904+0000 7fcbc5600640 1 -- 172.21.15.96:0/1097598942 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbc0031880 con 0x7fcbac004d40 2024-05-28T06:09:11.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.905+0000 7fcbc7e00640 1 -- 172.21.15.96:0/1097598942 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcbac004d40 msgr2=0x7fcbac0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:09:11.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.905+0000 7fcbc7e00640 1 --2- 172.21.15.96:0/1097598942 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcbac004d40 0x7fcbac0051c0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fcbc0009fc0 tx=0x7fcbac005830 comp rx=0 tx=0).stop 2024-05-28T06:09:11.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.905+0000 7fcbc7400640 1 -- 172.21.15.96:0/1097598942 reap_dead start 2024-05-28T06:09:11.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.905+0000 7fcbc7e00640 1 -- 172.21.15.96:0/1097598942 shutdown_connections 2024-05-28T06:09:11.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.905+0000 7fcbc7e00640 1 -- 172.21.15.96:0/1097598942 >> 172.21.15.96:0/1097598942 conn(0x7fcbc806b900 msgr2=0x7fcbc806bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:09:11.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.905+0000 7fcbc7e00640 1 -- 172.21.15.96:0/1097598942 shutdown_connections 2024-05-28T06:09:11.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.905+0000 7fcbc7e00640 1 -- 172.21.15.96:0/1097598942 wait complete. 2024-05-28T06:09:11.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.907+0000 7fcbc7e00640 1 Processor -- start 2024-05-28T06:09:11.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.907+0000 7fcbc7e00640 1 -- start start 2024-05-28T06:09:11.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.908+0000 7fcbc7e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcbac004d40 0x7fcbc81f2750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:11.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.908+0000 7fcbc7e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcbc81564f0 0x7fcbc81f2e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:11.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.908+0000 7fcbc7e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcbc80d00a0 0x7fcbc80d0530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:11.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.908+0000 7fcbc7e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fcbc80b48c0 con 0x7fcbac004d40 2024-05-28T06:09:11.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.908+0000 7fcbc7e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fcbc815d6c0 con 0x7fcbc81564f0 2024-05-28T06:09:11.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.908+0000 7fcbc7e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fcbc80b4d50 con 0x7fcbc80d00a0 2024-05-28T06:09:11.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.908+0000 7fcbc6a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcbac004d40 0x7fcbc81f2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:11.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.908+0000 7fcbc6a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcbac004d40 0x7fcbc81f2750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:55052/0 (socket says 172.21.15.96:55052) 2024-05-28T06:09:11.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.908+0000 7fcbc6a00640 1 -- 172.21.15.96:0/3957129830 learned_addr learned my addr 172.21.15.96:0/3957129830 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:09:11.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.908+0000 7fcbc6a00640 1 -- 172.21.15.96:0/3957129830 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcbc80d00a0 msgr2=0x7fcbc80d0530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:09:11.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.908+0000 7fcbc6a00640 1 --2- 172.21.15.96:0/3957129830 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcbc80d00a0 0x7fcbc80d0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:11.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.908+0000 7fcbc6a00640 1 -- 172.21.15.96:0/3957129830 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcbc81564f0 msgr2=0x7fcbc81f2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:09:11.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.908+0000 7fcbc6a00640 1 --2- 172.21.15.96:0/3957129830 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcbc81564f0 0x7fcbc81f2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:11.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.909+0000 7fcbc6a00640 1 -- 172.21.15.96:0/3957129830 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbc81f2c90 con 0x7fcbac004d40 2024-05-28T06:09:11.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.909+0000 7fcbc6a00640 1 --2- 172.21.15.96:0/3957129830 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcbac004d40 0x7fcbc81f2750 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fcbb800e830 tx=0x7fcbb800ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:09:11.914 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.909+0000 7fcbbf400640 1 -- 172.21.15.96:0/3957129830 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbb80052e0 con 0x7fcbac004d40 2024-05-28T06:09:11.914 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.909+0000 7fcbbf400640 1 -- 172.21.15.96:0/3957129830 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbb8007020 con 0x7fcbac004d40 2024-05-28T06:09:11.914 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.909+0000 7fcbbf400640 1 -- 172.21.15.96:0/3957129830 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbb8007210 con 0x7fcbac004d40 2024-05-28T06:09:11.914 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.909+0000 7fcbc7e00640 1 -- 172.21.15.96:0/3957129830 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbc815dd50 con 0x7fcbac004d40 2024-05-28T06:09:11.914 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.909+0000 7fcbc7e00640 1 -- 172.21.15.96:0/3957129830 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbc815e310 con 0x7fcbac004d40 2024-05-28T06:09:11.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.910+0000 7fcbc7e00640 1 -- 172.21.15.96:0/3957129830 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcbc80689e0 con 0x7fcbac004d40 2024-05-28T06:09:11.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.911+0000 7fcbbf400640 1 -- 172.21.15.96:0/3957129830 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 9) ==== 91295+0+0 (secure 0 0 0) 0x7fcbb801a020 con 0x7fcbac004d40 2024-05-28T06:09:11.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.912+0000 7fcbbf400640 1 --2- 172.21.15.96:0/3957129830 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fcba806cbc0 0x7fcba806f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:09:11.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.912+0000 7fcbbf400640 1 -- 172.21.15.96:0/3957129830 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5737+0+0 (secure 0 0 0) 0x7fcbb808b8b0 con 0x7fcbac004d40 2024-05-28T06:09:11.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.913+0000 7fcbc6000640 1 --2- 172.21.15.96:0/3957129830 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fcba806cbc0 0x7fcba806f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:09:11.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.914+0000 7fcbc6000640 1 --2- 172.21.15.96:0/3957129830 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fcba806cbc0 0x7fcba806f080 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fcbb4001eb0 tx=0x7fcbb4001d20 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:09:11.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:11.917+0000 7fcbbf400640 1 -- 172.21.15.96:0/3957129830 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fcbc81f2c90 con 0x7fcbac004d40 2024-05-28T06:09:12.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:12.148+0000 7fcbc7e00640 1 -- 172.21.15.96:0/3957129830 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) -- 0x7fcbc80b3c40 con 0x7fcba806cbc0 2024-05-28T06:09:12.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:12.160+0000 7fcbbf400640 1 -- 172.21.15.96:0/3957129830 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+59 (secure 0 0 0) 0x7fcbc80b3c40 con 0x7fcba806cbc0 2024-05-28T06:09:12.161 INFO:teuthology.orchestra.run.smithi096.stdout:/volumes/_nogroup/sv1/3761c493-2f24-4ca4-89b1-4aca4cd20f2a 2024-05-28T06:09:12.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:12.161+0000 7fcbc7e00640 1 -- 172.21.15.96:0/3957129830 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fcba806cbc0 msgr2=0x7fcba806f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:09:12.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:12.161+0000 7fcbc7e00640 1 --2- 172.21.15.96:0/3957129830 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fcba806cbc0 0x7fcba806f080 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fcbb4001eb0 tx=0x7fcbb4001d20 comp rx=0 tx=0).stop 2024-05-28T06:09:12.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:12.162+0000 7fcbc7e00640 1 -- 172.21.15.96:0/3957129830 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcbac004d40 msgr2=0x7fcbc81f2750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:09:12.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:12.162+0000 7fcbc7e00640 1 --2- 172.21.15.96:0/3957129830 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcbac004d40 0x7fcbc81f2750 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fcbb800e830 tx=0x7fcbb800ecf0 comp rx=0 tx=0).stop 2024-05-28T06:09:12.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:12.162+0000 7fcbc7e00640 1 -- 172.21.15.96:0/3957129830 shutdown_connections 2024-05-28T06:09:12.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:12.162+0000 7fcbc7e00640 1 --2- 172.21.15.96:0/3957129830 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fcba806cbc0 0x7fcba806f080 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:12.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:12.162+0000 7fcbc7e00640 1 --2- 172.21.15.96:0/3957129830 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcbc80d00a0 0x7fcbc80d0530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:12.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:12.162+0000 7fcbc7e00640 1 --2- 172.21.15.96:0/3957129830 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcbc81564f0 0x7fcbc81f2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:12.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:12.162+0000 7fcbc7e00640 1 --2- 172.21.15.96:0/3957129830 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcbac004d40 0x7fcbc81f2750 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:09:12.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:12.162+0000 7fcbc7e00640 1 -- 172.21.15.96:0/3957129830 >> 172.21.15.96:0/3957129830 conn(0x7fcbc806b900 msgr2=0x7fcbc8152490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:09:12.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:12.162+0000 7fcbc7e00640 1 -- 172.21.15.96:0/3957129830 shutdown_connections 2024-05-28T06:09:12.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:09:12.162+0000 7fcbc7e00640 1 -- 172.21.15.96:0/3957129830 wait complete. 2024-05-28T06:09:12.184 DEBUG:tasks.cephfs.test_volumes:filling subvolume sv1 with 10 files each 1024MB size under directory volumes/_nogroup/sv1/3761c493-2f24-4ca4-89b1-4aca4cd20f2a 2024-05-28T06:09:12.184 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:09:12.185 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:09:12.229 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:09:12.229 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/3761c493-2f24-4ca4-89b1-4aca4cd20f2a/subvolume_file.0 bs=1M conv=fdatasync count=1024 seek=0) 2024-05-28T06:09:27.304 INFO:teuthology.orchestra.run.smithi096.stderr:1024+0 records in 2024-05-28T06:09:27.304 INFO:teuthology.orchestra.run.smithi096.stderr:1024+0 records out 2024-05-28T06:09:27.304 INFO:teuthology.orchestra.run.smithi096.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 14.9573 s, 71.8 MB/s 2024-05-28T06:09:27.306 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:09:27.306 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:09:27.358 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:09:27.358 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/3761c493-2f24-4ca4-89b1-4aca4cd20f2a/subvolume_file.1 bs=1M conv=fdatasync count=1024 seek=0) 2024-05-28T06:09:30.347 DEBUG:teuthology.orchestra.run.smithi096:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:09:30.356 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:09:38.356 INFO:teuthology.orchestra.run.smithi096.stderr:1024+0 records in 2024-05-28T06:09:38.356 INFO:teuthology.orchestra.run.smithi096.stderr:1024+0 records out 2024-05-28T06:09:38.356 INFO:teuthology.orchestra.run.smithi096.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 10.8944 s, 98.6 MB/s 2024-05-28T06:09:38.358 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:09:38.358 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:09:38.413 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:09:38.413 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/3761c493-2f24-4ca4-89b1-4aca4cd20f2a/subvolume_file.2 bs=1M conv=fdatasync count=1024 seek=0) 2024-05-28T06:09:49.926 INFO:teuthology.orchestra.run.smithi096.stderr:1024+0 records in 2024-05-28T06:09:49.926 INFO:teuthology.orchestra.run.smithi096.stderr:1024+0 records out 2024-05-28T06:09:49.927 INFO:teuthology.orchestra.run.smithi096.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 11.4569 s, 93.7 MB/s 2024-05-28T06:09:49.928 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:09:49.928 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:09:49.981 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:09:49.981 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/3761c493-2f24-4ca4-89b1-4aca4cd20f2a/subvolume_file.3 bs=1M conv=fdatasync count=1024 seek=0) 2024-05-28T06:10:00.477 DEBUG:teuthology.orchestra.run.smithi096:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:10:00.487 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:10:01.426 INFO:teuthology.orchestra.run.smithi096.stderr:1024+0 records in 2024-05-28T06:10:01.426 INFO:teuthology.orchestra.run.smithi096.stderr:1024+0 records out 2024-05-28T06:10:01.426 INFO:teuthology.orchestra.run.smithi096.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 11.3905 s, 94.3 MB/s 2024-05-28T06:10:01.427 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:10:01.428 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:10:01.478 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:10:01.478 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/3761c493-2f24-4ca4-89b1-4aca4cd20f2a/subvolume_file.4 bs=1M conv=fdatasync count=1024 seek=0) 2024-05-28T06:10:12.895 INFO:teuthology.orchestra.run.smithi096.stderr:1024+0 records in 2024-05-28T06:10:12.895 INFO:teuthology.orchestra.run.smithi096.stderr:1024+0 records out 2024-05-28T06:10:12.895 INFO:teuthology.orchestra.run.smithi096.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 11.3633 s, 94.5 MB/s 2024-05-28T06:10:12.897 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:10:12.897 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:10:12.951 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:10:12.951 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/3761c493-2f24-4ca4-89b1-4aca4cd20f2a/subvolume_file.5 bs=1M conv=fdatasync count=1024 seek=0) 2024-05-28T06:10:24.210 INFO:teuthology.orchestra.run.smithi096.stderr:1024+0 records in 2024-05-28T06:10:24.211 INFO:teuthology.orchestra.run.smithi096.stderr:1024+0 records out 2024-05-28T06:10:24.211 INFO:teuthology.orchestra.run.smithi096.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 11.1994 s, 95.9 MB/s 2024-05-28T06:10:24.212 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:10:24.212 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:10:24.265 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:10:24.265 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/3761c493-2f24-4ca4-89b1-4aca4cd20f2a/subvolume_file.6 bs=1M conv=fdatasync count=1024 seek=0) 2024-05-28T06:10:30.595 DEBUG:teuthology.orchestra.run.smithi096:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:10:30.605 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:10:35.390 INFO:teuthology.orchestra.run.smithi096.stderr:1024+0 records in 2024-05-28T06:10:35.391 INFO:teuthology.orchestra.run.smithi096.stderr:1024+0 records out 2024-05-28T06:10:35.391 INFO:teuthology.orchestra.run.smithi096.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 11.0695 s, 97.0 MB/s 2024-05-28T06:10:35.392 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:10:35.392 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:10:35.445 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:10:35.446 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/3761c493-2f24-4ca4-89b1-4aca4cd20f2a/subvolume_file.7 bs=1M conv=fdatasync count=1024 seek=0) 2024-05-28T06:10:47.631 INFO:teuthology.orchestra.run.smithi096.stderr:1024+0 records in 2024-05-28T06:10:47.632 INFO:teuthology.orchestra.run.smithi096.stderr:1024+0 records out 2024-05-28T06:10:47.632 INFO:teuthology.orchestra.run.smithi096.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 12.1288 s, 88.5 MB/s 2024-05-28T06:10:47.633 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:10:47.633 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:10:47.687 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:10:47.687 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/3761c493-2f24-4ca4-89b1-4aca4cd20f2a/subvolume_file.8 bs=1M conv=fdatasync count=1024 seek=0) 2024-05-28T06:10:59.003 INFO:teuthology.orchestra.run.smithi096.stderr:1024+0 records in 2024-05-28T06:10:59.003 INFO:teuthology.orchestra.run.smithi096.stderr:1024+0 records out 2024-05-28T06:10:59.003 INFO:teuthology.orchestra.run.smithi096.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 11.2558 s, 95.4 MB/s 2024-05-28T06:10:59.005 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:10:59.005 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:10:59.063 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:10:59.063 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/3761c493-2f24-4ca4-89b1-4aca4cd20f2a/subvolume_file.9 bs=1M conv=fdatasync count=1024 seek=0) 2024-05-28T06:11:00.727 DEBUG:teuthology.orchestra.run.smithi096:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:11:00.734 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:11:10.256 INFO:teuthology.orchestra.run.smithi096.stderr:1024+0 records in 2024-05-28T06:11:10.257 INFO:teuthology.orchestra.run.smithi096.stderr:1024+0 records out 2024-05-28T06:11:10.257 INFO:teuthology.orchestra.run.smithi096.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 11.135 s, 96.4 MB/s 2024-05-28T06:11:10.259 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot create cephfs sv1 ss1 2024-05-28T06:11:10.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:10.460+0000 7faac7000640 1 Processor -- start 2024-05-28T06:11:10.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:10.460+0000 7faac7000640 1 -- start start 2024-05-28T06:11:10.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:10.461+0000 7faac7000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7faac006edf0 0x7faac013fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:10.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:10.461+0000 7faac7000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7faac006f250 0x7faac0140330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:10.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:10.462+0000 7faac7000640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faac00d0020 con 0x7faac00cf2f0 2024-05-28T06:11:10.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:10.462+0000 7faac7000640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7faac006dd50 con 0x7faac006f250 2024-05-28T06:11:10.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:10.462+0000 7faac7000640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7faac006e030 con 0x7faac006edf0 2024-05-28T06:11:10.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:10.462+0000 7faac5000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7faac006f250 0x7faac0140330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:10.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:10.462+0000 7faac5000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7faac006f250 0x7faac0140330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47272/0 (socket says 172.21.15.96:47272) 2024-05-28T06:11:10.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:10.462+0000 7faac5000640 1 -- 172.21.15.96:0/328360759 learned_addr learned my addr 172.21.15.96:0/328360759 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:11:10.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:10.462+0000 7faab7400640 1 --2- 172.21.15.96:0/328360759 >> v2:172.21.15.123:3300/0 conn(0x7faac006edf0 0x7faac013fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:10.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:10.463+0000 7faac5000640 1 -- 172.21.15.96:0/328360759 >> v2:172.21.15.123:3300/0 conn(0x7faac006edf0 msgr2=0x7faac013fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:11:10.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:10.463+0000 7faac5000640 1 --2- 172.21.15.96:0/328360759 >> v2:172.21.15.123:3300/0 conn(0x7faac006edf0 0x7faac013fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:10.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:10.463+0000 7faac5000640 1 -- 172.21.15.96:0/328360759 >> v1:172.21.15.123:6789/0 conn(0x7faac00cf2f0 legacy=0x7faac00cf910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:11:10.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:10.463+0000 7faac5000640 1 -- 172.21.15.96:0/328360759 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faac006e310 con 0x7faac006f250 2024-05-28T06:11:10.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:10.464+0000 7faac5000640 1 --2- 172.21.15.96:0/328360759 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faac006f250 0x7faac0140330 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7faac0071330 tx=0x7faab802a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=970a5b7a75d08762 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:11:10.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:10.464+0000 7faabf400640 1 -- 172.21.15.96:0/328360759 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faab8036070 con 0x7faac006f250 2024-05-28T06:11:10.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:10.464+0000 7faabf400640 1 -- 172.21.15.96:0/328360759 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faab8031600 con 0x7faac006f250 2024-05-28T06:11:10.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:10.464+0000 7faabf400640 1 -- 172.21.15.96:0/328360759 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faab80319d0 con 0x7faac006f250 2024-05-28T06:11:10.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:10.465+0000 7faac7000640 1 -- 172.21.15.96:0/328360759 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faac006f250 msgr2=0x7faac0140330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:10.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:10.465+0000 7faac7000640 1 --2- 172.21.15.96:0/328360759 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faac006f250 0x7faac0140330 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7faac0071330 tx=0x7faab802a660 comp rx=0 tx=0).stop 2024-05-28T06:11:10.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:10.465+0000 7faac7000640 1 -- 172.21.15.96:0/328360759 shutdown_connections 2024-05-28T06:11:10.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:10.465+0000 7faac7000640 1 --2- 172.21.15.96:0/328360759 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faac006f250 0x7faac0140330 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:10.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:10.465+0000 7faac7000640 1 --2- 172.21.15.96:0/328360759 >> v2:172.21.15.123:3300/0 conn(0x7faac006edf0 0x7faac013fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:10.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:10.465+0000 7faac7000640 1 -- 172.21.15.96:0/328360759 >> 172.21.15.96:0/328360759 conn(0x7faac00b1e50 msgr2=0x7faac00b2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:11:10.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:10.465+0000 7faac7000640 1 -- 172.21.15.96:0/328360759 shutdown_connections 2024-05-28T06:11:10.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:10.465+0000 7faac7000640 1 -- 172.21.15.96:0/328360759 wait complete. 2024-05-28T06:11:10.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:10.467+0000 7faac7000640 1 Processor -- start 2024-05-28T06:11:10.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:10.467+0000 7faac7000640 1 -- start start 2024-05-28T06:11:10.469 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:10.468+0000 7faac7000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faac006edf0 0x7faac0141600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:10.470 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:10.468+0000 7faac7000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faac006f250 0x7faac0159e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:10.470 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:10.468+0000 7faac7000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faac00cf2f0 0x7faac00ccc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:10.470 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:10.468+0000 7faac7000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7faac0146020 con 0x7faac00cf2f0 2024-05-28T06:11:10.470 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:10.468+0000 7faac7000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7faac006e8d0 con 0x7faac006f250 2024-05-28T06:11:10.470 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:10.468+0000 7faac7000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7faac01461a0 con 0x7faac006edf0 2024-05-28T06:11:10.470 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:10.468+0000 7faac5000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faac00cf2f0 0x7faac00ccc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:10.470 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:10.468+0000 7faac5000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faac00cf2f0 0x7faac00ccc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47286/0 (socket says 172.21.15.96:47286) 2024-05-28T06:11:10.470 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:10.468+0000 7faac5000640 1 -- 172.21.15.96:0/1168977224 learned_addr learned my addr 172.21.15.96:0/1168977224 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:11:10.471 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:10.468+0000 7faabfe00640 1 --2- 172.21.15.96:0/1168977224 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faac006edf0 0x7faac0141600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:10.471 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:10.469+0000 7faab7400640 1 --2- 172.21.15.96:0/1168977224 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faac006f250 0x7faac0159e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:10.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:10.469+0000 7faac5000640 1 -- 172.21.15.96:0/1168977224 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faac006edf0 msgr2=0x7faac0141600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:10.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:10.469+0000 7faac5000640 1 --2- 172.21.15.96:0/1168977224 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faac006edf0 0x7faac0141600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:10.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:10.469+0000 7faac5000640 1 -- 172.21.15.96:0/1168977224 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faac006f250 msgr2=0x7faac0159e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:10.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:10.469+0000 7faac5000640 1 --2- 172.21.15.96:0/1168977224 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faac006f250 0x7faac0159e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:10.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:10.469+0000 7faac5000640 1 -- 172.21.15.96:0/1168977224 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faac015a3d0 con 0x7faac00cf2f0 2024-05-28T06:11:10.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:10.469+0000 7faabfe00640 1 --2- 172.21.15.96:0/1168977224 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faac006edf0 0x7faac0141600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:11:10.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:10.469+0000 7faab7400640 1 --2- 172.21.15.96:0/1168977224 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faac006f250 0x7faac0159e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:11:10.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:10.469+0000 7faac5000640 1 --2- 172.21.15.96:0/1168977224 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faac00cf2f0 0x7faac00ccc50 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7faab8034040 tx=0x7faab8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:11:10.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:10.469+0000 7faabcc00640 1 -- 172.21.15.96:0/1168977224 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faab8036050 con 0x7faac00cf2f0 2024-05-28T06:11:10.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:10.469+0000 7faabcc00640 1 -- 172.21.15.96:0/1168977224 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faab8042070 con 0x7faac00cf2f0 2024-05-28T06:11:10.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:10.469+0000 7faabcc00640 1 -- 172.21.15.96:0/1168977224 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faab8009510 con 0x7faac00cf2f0 2024-05-28T06:11:10.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:10.469+0000 7faac7000640 1 -- 172.21.15.96:0/1168977224 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faac015f920 con 0x7faac00cf2f0 2024-05-28T06:11:10.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:10.469+0000 7faac7000640 1 -- 172.21.15.96:0/1168977224 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faac0140e20 con 0x7faac00cf2f0 2024-05-28T06:11:10.473 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:10.471+0000 7faac7000640 1 -- 172.21.15.96:0/1168977224 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faac00689e0 con 0x7faac00cf2f0 2024-05-28T06:11:10.473 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:10.471+0000 7faabcc00640 1 -- 172.21.15.96:0/1168977224 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 11) ==== 91295+0+0 (secure 0 0 0) 0x7faab8007bd0 con 0x7faac00cf2f0 2024-05-28T06:11:10.474 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:10.472+0000 7faabcc00640 1 --2- 172.21.15.96:0/1168977224 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7faa9c06cbc0 0x7faa9c06f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:10.474 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:10.472+0000 7faabcc00640 1 -- 172.21.15.96:0/1168977224 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5795+0+0 (secure 0 0 0) 0x7faab80acfb0 con 0x7faac00cf2f0 2024-05-28T06:11:10.474 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:10.473+0000 7faabfe00640 1 --2- 172.21.15.96:0/1168977224 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7faa9c06cbc0 0x7faa9c06f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:10.475 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:10.474+0000 7faabfe00640 1 --2- 172.21.15.96:0/1168977224 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7faa9c06cbc0 0x7faa9c06f080 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7faab0003eb0 tx=0x7faab0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:11:10.478 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:10.477+0000 7faabcc00640 1 -- 172.21.15.96:0/1168977224 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7faab807af70 con 0x7faac00cf2f0 2024-05-28T06:11:10.716 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:10.715+0000 7faac7000640 1 -- 172.21.15.96:0/1168977224 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""]}) -- 0x7faac0073f30 con 0x7faa9c06cbc0 2024-05-28T06:11:10.728 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:11:10.726+0000 7fea59e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:11:10.728 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:11:10.726+0000 7fea59e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:11:10.728 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:11:10.726+0000 7fea59e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:11:10.728 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:11:10.726+0000 7fea59e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:11:10.728 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:11:10.726+0000 7fea59e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:11:10.777 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:10.775+0000 7faabcc00640 1 -- 172.21.15.96:0/1168977224 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7faac0073f30 con 0x7faa9c06cbc0 2024-05-28T06:11:10.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:10.778+0000 7faac7000640 1 -- 172.21.15.96:0/1168977224 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7faa9c06cbc0 msgr2=0x7faa9c06f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:10.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:10.778+0000 7faac7000640 1 --2- 172.21.15.96:0/1168977224 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7faa9c06cbc0 0x7faa9c06f080 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7faab0003eb0 tx=0x7faab0009040 comp rx=0 tx=0).stop 2024-05-28T06:11:10.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:10.778+0000 7faac7000640 1 -- 172.21.15.96:0/1168977224 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faac00cf2f0 msgr2=0x7faac00ccc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:10.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:10.778+0000 7faac7000640 1 --2- 172.21.15.96:0/1168977224 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faac00cf2f0 0x7faac00ccc50 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7faab8034040 tx=0x7faab8004a40 comp rx=0 tx=0).stop 2024-05-28T06:11:10.781 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:10.778+0000 7faac7000640 1 -- 172.21.15.96:0/1168977224 shutdown_connections 2024-05-28T06:11:10.781 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:10.778+0000 7faac7000640 1 --2- 172.21.15.96:0/1168977224 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7faa9c06cbc0 0x7faa9c06f080 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:10.781 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:10.778+0000 7faac7000640 1 --2- 172.21.15.96:0/1168977224 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faac00cf2f0 0x7faac00ccc50 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:10.781 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:10.778+0000 7faac7000640 1 --2- 172.21.15.96:0/1168977224 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faac006f250 0x7faac0159e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:10.781 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:10.778+0000 7faac7000640 1 --2- 172.21.15.96:0/1168977224 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faac006edf0 0x7faac0141600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:10.781 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:10.778+0000 7faac7000640 1 -- 172.21.15.96:0/1168977224 >> 172.21.15.96:0/1168977224 conn(0x7faac00b1e50 msgr2=0x7faac00ce3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:11:10.781 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:10.779+0000 7faac7000640 1 -- 172.21.15.96:0/1168977224 shutdown_connections 2024-05-28T06:11:10.781 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:10.779+0000 7faac7000640 1 -- 172.21.15.96:0/1168977224 wait complete. 2024-05-28T06:11:10.799 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-05-28T06:11:11.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.001+0000 7f4c52c00640 1 Processor -- start 2024-05-28T06:11:11.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.001+0000 7f4c52c00640 1 -- start start 2024-05-28T06:11:11.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.003+0000 7f4c52c00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f4c4c15b6f0 0x7f4c4c06e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:11.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.003+0000 7f4c52c00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c4c0ade90 con 0x7f4c4c0d0640 2024-05-28T06:11:11.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.003+0000 7f4c52c00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c4c071e70 con 0x7f4c4c0725e0 2024-05-28T06:11:11.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.003+0000 7f4c52c00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f4c4c071c40 con 0x7f4c4c15b6f0 2024-05-28T06:11:11.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.003+0000 7f4c4be00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f4c4c0d0640 0x7f4c4c0d0a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:57724/0 (socket says 172.21.15.96:57724) 2024-05-28T06:11:11.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.003+0000 7f4c4be00640 1 -- 172.21.15.96:0/1158020186 learned_addr learned my addr 172.21.15.96:0/1158020186 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:11:11.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.004+0000 7f4c4b400640 1 -- 172.21.15.96:0/1158020186 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3776485634 0 0) 0x7f4c4c0ade90 con 0x7f4c4c0d0640 2024-05-28T06:11:11.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.004+0000 7f4c4b400640 1 -- 172.21.15.96:0/1158020186 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4c30005730 con 0x7f4c4c0d0640 2024-05-28T06:11:11.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.004+0000 7f4c43400640 1 --2- 172.21.15.96:0/1158020186 >> v2:172.21.15.123:3300/0 conn(0x7f4c4c15b6f0 0x7f4c4c06e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:11.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.004+0000 7f4c4b400640 1 -- 172.21.15.96:0/1158020186 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2380419247 0 0) 0x7f4c30005730 con 0x7f4c4c0d0640 2024-05-28T06:11:11.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.004+0000 7f4c4b400640 1 -- 172.21.15.96:0/1158020186 >> v2:172.21.15.123:3300/0 conn(0x7f4c4c15b6f0 msgr2=0x7f4c4c06e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:11:11.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.004+0000 7f4c4b400640 1 --2- 172.21.15.96:0/1158020186 >> v2:172.21.15.123:3300/0 conn(0x7f4c4c15b6f0 0x7f4c4c06e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:11.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.004+0000 7f4c4b400640 1 -- 172.21.15.96:0/1158020186 >> v1:172.21.15.123:6789/0 conn(0x7f4c4c0725e0 legacy=0x7f4c4c0ae650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:11.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.004+0000 7f4c4b400640 1 -- 172.21.15.96:0/1158020186 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c4c072130 con 0x7f4c4c0d0640 2024-05-28T06:11:11.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.004+0000 7f4c4b400640 1 -- 172.21.15.96:0/1158020186 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f4c3c001670 con 0x7f4c4c0d0640 2024-05-28T06:11:11.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.004+0000 7f4c4b400640 1 -- 172.21.15.96:0/1158020186 >> v1:172.21.15.96:6789/0 conn(0x7f4c4c0d0640 legacy=0x7f4c4c0d0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:11.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.004+0000 7f4c4b400640 1 --2- 172.21.15.96:0/1158020186 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4c30005bc0 0x7f4c30005fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:11.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.005+0000 7f4c4b400640 1 --2- 172.21.15.96:0/1158020186 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4c30006a10 0x7f4c30006e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:11.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.005+0000 7f4c4b400640 1 --2- 172.21.15.96:0/1158020186 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4c300073d0 0x7f4c3000dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:11.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.005+0000 7f4c4b400640 1 -- 172.21.15.96:0/1158020186 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4c3000e240 con 0x7f4c300073d0 2024-05-28T06:11:11.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.005+0000 7f4c4b400640 1 -- 172.21.15.96:0/1158020186 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f4c3000e3f0 con 0x7f4c30005bc0 2024-05-28T06:11:11.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.005+0000 7f4c4b400640 1 -- 172.21.15.96:0/1158020186 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f4c3000e5f0 con 0x7f4c30006a10 2024-05-28T06:11:11.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.005+0000 7f4c50c00640 1 --2- 172.21.15.96:0/1158020186 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4c300073d0 0x7f4c3000dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:11.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.005+0000 7f4c50c00640 1 -- 172.21.15.96:0/1158020186 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4c30006a10 msgr2=0x7f4c30006e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:11:11.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.005+0000 7f4c43400640 1 --2- 172.21.15.96:0/1158020186 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4c30006a10 0x7f4c30006e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:11.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.005+0000 7f4c50c00640 1 --2- 172.21.15.96:0/1158020186 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4c30006a10 0x7f4c30006e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:11.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.005+0000 7f4c50c00640 1 -- 172.21.15.96:0/1158020186 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4c30005bc0 msgr2=0x7f4c30005fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:11:11.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.005+0000 7f4c50c00640 1 --2- 172.21.15.96:0/1158020186 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4c30005bc0 0x7f4c30005fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:11.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.005+0000 7f4c50c00640 1 -- 172.21.15.96:0/1158020186 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c30005730 con 0x7f4c300073d0 2024-05-28T06:11:11.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.006+0000 7f4c50c00640 1 -- 172.21.15.96:0/1158020186 reap_dead start 2024-05-28T06:11:11.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.006+0000 7f4c50c00640 1 --2- 172.21.15.96:0/1158020186 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4c300073d0 0x7f4c3000dbd0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f4c44001e50 tx=0x7f4c30007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca1d64fd77304996 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:11:11.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.006+0000 7f4c4b400640 1 -- 172.21.15.96:0/1158020186 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c44001a70 con 0x7f4c300073d0 2024-05-28T06:11:11.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.006+0000 7f4c4b400640 1 -- 172.21.15.96:0/1158020186 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c44003970 con 0x7f4c300073d0 2024-05-28T06:11:11.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.006+0000 7f4c4b400640 1 -- 172.21.15.96:0/1158020186 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c44003d60 con 0x7f4c300073d0 2024-05-28T06:11:11.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.007+0000 7f4c52c00640 1 -- 172.21.15.96:0/1158020186 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4c300073d0 msgr2=0x7f4c3000dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:11.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.007+0000 7f4c52c00640 1 --2- 172.21.15.96:0/1158020186 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4c300073d0 0x7f4c3000dbd0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f4c44001e50 tx=0x7f4c30007a40 comp rx=0 tx=0).stop 2024-05-28T06:11:11.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.007+0000 7f4c52c00640 1 -- 172.21.15.96:0/1158020186 shutdown_connections 2024-05-28T06:11:11.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.007+0000 7f4c52c00640 1 --2- 172.21.15.96:0/1158020186 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4c300073d0 0x7f4c3000dbd0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:11.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.007+0000 7f4c52c00640 1 -- 172.21.15.96:0/1158020186 >> 172.21.15.96:0/1158020186 conn(0x7f4c4c06bd20 msgr2=0x7f4c4c06c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:11:11.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.007+0000 7f4c52c00640 1 -- 172.21.15.96:0/1158020186 shutdown_connections 2024-05-28T06:11:11.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.007+0000 7f4c52c00640 1 -- 172.21.15.96:0/1158020186 wait complete. 2024-05-28T06:11:11.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.009+0000 7f4c52c00640 1 Processor -- start 2024-05-28T06:11:11.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.009+0000 7f4c52c00640 1 -- start start 2024-05-28T06:11:11.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.010+0000 7f4c52c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4c300073d0 0x7f4c4c0cc740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:11.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.010+0000 7f4c52c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4c4c1477f0 0x7f4c4c0ce530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:11.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.010+0000 7f4c52c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4c4c15bd40 0x7f4c4c0b4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:11.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.010+0000 7f4c52c00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4c4c0d0ba0 con 0x7f4c4c1477f0 2024-05-28T06:11:11.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.010+0000 7f4c52c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f4c4c0d0a20 con 0x7f4c4c15bd40 2024-05-28T06:11:11.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.010+0000 7f4c52c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f4c4c0d0d20 con 0x7f4c300073d0 2024-05-28T06:11:11.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.010+0000 7f4c43400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4c4c1477f0 0x7f4c4c0ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:11.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.010+0000 7f4c43400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4c4c1477f0 0x7f4c4c0ce530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47300/0 (socket says 172.21.15.96:47300) 2024-05-28T06:11:11.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.010+0000 7f4c43400640 1 -- 172.21.15.96:0/418392287 learned_addr learned my addr 172.21.15.96:0/418392287 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:11:11.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.011+0000 7f4c4be00640 1 --2- 172.21.15.96:0/418392287 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4c300073d0 0x7f4c4c0cc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:11.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.011+0000 7f4c43400640 1 -- 172.21.15.96:0/418392287 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4c300073d0 msgr2=0x7f4c4c0cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:11.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.011+0000 7f4c43400640 1 --2- 172.21.15.96:0/418392287 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4c300073d0 0x7f4c4c0cc740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:11.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.011+0000 7f4c43400640 1 -- 172.21.15.96:0/418392287 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4c4c15bd40 msgr2=0x7f4c4c0b4430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:11.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.011+0000 7f4c50c00640 1 --2- 172.21.15.96:0/418392287 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4c4c15bd40 0x7f4c4c0b4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:11.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.011+0000 7f4c43400640 1 --2- 172.21.15.96:0/418392287 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4c4c15bd40 0x7f4c4c0b4430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:11.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.011+0000 7f4c43400640 1 -- 172.21.15.96:0/418392287 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c4c0cea70 con 0x7f4c4c1477f0 2024-05-28T06:11:11.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.011+0000 7f4c4be00640 1 --2- 172.21.15.96:0/418392287 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4c300073d0 0x7f4c4c0cc740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:11:11.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.011+0000 7f4c43400640 1 --2- 172.21.15.96:0/418392287 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4c4c1477f0 0x7f4c4c0ce530 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f4c34010890 tx=0x7f4c34010d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:11:11.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.011+0000 7f4c48c00640 1 -- 172.21.15.96:0/418392287 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c3400f040 con 0x7f4c4c1477f0 2024-05-28T06:11:11.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.011+0000 7f4c48c00640 1 -- 172.21.15.96:0/418392287 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c3400eb60 con 0x7f4c4c1477f0 2024-05-28T06:11:11.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.011+0000 7f4c48c00640 1 -- 172.21.15.96:0/418392287 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c3401c400 con 0x7f4c4c1477f0 2024-05-28T06:11:11.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.011+0000 7f4c52c00640 1 -- 172.21.15.96:0/418392287 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c4c0ccdd0 con 0x7f4c4c1477f0 2024-05-28T06:11:11.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.012+0000 7f4c52c00640 1 -- 172.21.15.96:0/418392287 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c4c0cd390 con 0x7f4c4c1477f0 2024-05-28T06:11:11.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.013+0000 7f4c52c00640 1 -- 172.21.15.96:0/418392287 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c4c0689e0 con 0x7f4c4c1477f0 2024-05-28T06:11:11.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.014+0000 7f4c48c00640 1 -- 172.21.15.96:0/418392287 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 11) ==== 91295+0+0 (secure 0 0 0) 0x7f4c3401c5a0 con 0x7f4c4c1477f0 2024-05-28T06:11:11.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.015+0000 7f4c48c00640 1 --2- 172.21.15.96:0/418392287 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4c2806cbc0 0x7f4c2806f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:11.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.015+0000 7f4c48c00640 1 -- 172.21.15.96:0/418392287 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5795+0+0 (secure 0 0 0) 0x7f4c3408c730 con 0x7f4c4c1477f0 2024-05-28T06:11:11.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.015+0000 7f4c4be00640 1 --2- 172.21.15.96:0/418392287 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4c2806cbc0 0x7f4c2806f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:11.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.017+0000 7f4c4be00640 1 --2- 172.21.15.96:0/418392287 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4c2806cbc0 0x7f4c2806f080 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f4c3c000f60 tx=0x7f4c3c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:11:11.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.019+0000 7f4c48c00640 1 -- 172.21.15.96:0/418392287 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4c34018030 con 0x7f4c4c1477f0 2024-05-28T06:11:11.260 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.259+0000 7f4c52c00640 1 -- 172.21.15.96:0/418392287 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) -- 0x7f4c4c0b1910 con 0x7f4c2806cbc0 2024-05-28T06:11:11.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.273+0000 7f4c48c00640 1 -- 172.21.15.96:0/418392287 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+59 (secure 0 0 0) 0x7f4c4c0b1910 con 0x7f4c2806cbc0 2024-05-28T06:11:11.275 INFO:teuthology.orchestra.run.smithi096.stdout:/volumes/_nogroup/sv1/3761c493-2f24-4ca4-89b1-4aca4cd20f2a 2024-05-28T06:11:11.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.275+0000 7f4c52c00640 1 -- 172.21.15.96:0/418392287 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4c2806cbc0 msgr2=0x7f4c2806f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:11.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.275+0000 7f4c52c00640 1 --2- 172.21.15.96:0/418392287 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4c2806cbc0 0x7f4c2806f080 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f4c3c000f60 tx=0x7f4c3c009040 comp rx=0 tx=0).stop 2024-05-28T06:11:11.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.275+0000 7f4c52c00640 1 -- 172.21.15.96:0/418392287 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4c4c1477f0 msgr2=0x7f4c4c0ce530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:11.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.275+0000 7f4c52c00640 1 --2- 172.21.15.96:0/418392287 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4c4c1477f0 0x7f4c4c0ce530 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f4c34010890 tx=0x7f4c34010d50 comp rx=0 tx=0).stop 2024-05-28T06:11:11.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.276+0000 7f4c52c00640 1 -- 172.21.15.96:0/418392287 shutdown_connections 2024-05-28T06:11:11.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.276+0000 7f4c52c00640 1 --2- 172.21.15.96:0/418392287 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4c2806cbc0 0x7f4c2806f080 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:11.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.276+0000 7f4c52c00640 1 --2- 172.21.15.96:0/418392287 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4c4c15bd40 0x7f4c4c0b4430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:11.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.276+0000 7f4c52c00640 1 --2- 172.21.15.96:0/418392287 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4c4c1477f0 0x7f4c4c0ce530 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:11.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.276+0000 7f4c52c00640 1 --2- 172.21.15.96:0/418392287 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4c300073d0 0x7f4c4c0cc740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:11.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.276+0000 7f4c52c00640 1 -- 172.21.15.96:0/418392287 >> 172.21.15.96:0/418392287 conn(0x7f4c4c06bd20 msgr2=0x7f4c4c151340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:11:11.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.276+0000 7f4c52c00640 1 -- 172.21.15.96:0/418392287 shutdown_connections 2024-05-28T06:11:11.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:11.276+0000 7f4c52c00640 1 -- 172.21.15.96:0/418392287 wait complete. 2024-05-28T06:11:11.295 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:11:11.296 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/3761c493-2f24-4ca4-89b1-4aca4cd20f2a 2024-05-28T06:11:11.296 DEBUG:teuthology.orchestra.run.smithi096:> ) 2024-05-28T06:11:11.353 INFO:teuthology.orchestra.run.smithi096.stdout:# file: volumes/_nogroup/sv1/3761c493-2f24-4ca4-89b1-4aca4cd20f2a 2024-05-28T06:11:11.354 INFO:teuthology.orchestra.run.smithi096.stdout:ceph.dir.rbytes="10654449664" 2024-05-28T06:11:11.354 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:11:11.355 INFO:tasks.cephfs.test_volumes:r_size = 10654449664 exp_size = 10737418240 2024-05-28T06:11:13.356 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:11:13.356 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/3761c493-2f24-4ca4-89b1-4aca4cd20f2a 2024-05-28T06:11:13.356 DEBUG:teuthology.orchestra.run.smithi096:> ) 2024-05-28T06:11:13.423 INFO:teuthology.orchestra.run.smithi096.stdout:# file: volumes/_nogroup/sv1/3761c493-2f24-4ca4-89b1-4aca4cd20f2a 2024-05-28T06:11:13.423 INFO:teuthology.orchestra.run.smithi096.stdout:ceph.dir.rbytes="10654449664" 2024-05-28T06:11:13.423 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:11:13.424 INFO:tasks.cephfs.test_volumes:r_size = 10654449664 exp_size = 10737418240 2024-05-28T06:11:15.426 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:11:15.426 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/3761c493-2f24-4ca4-89b1-4aca4cd20f2a 2024-05-28T06:11:15.426 DEBUG:teuthology.orchestra.run.smithi096:> ) 2024-05-28T06:11:15.491 INFO:teuthology.orchestra.run.smithi096.stdout:# file: volumes/_nogroup/sv1/3761c493-2f24-4ca4-89b1-4aca4cd20f2a 2024-05-28T06:11:15.491 INFO:teuthology.orchestra.run.smithi096.stdout:ceph.dir.rbytes="10737418240" 2024-05-28T06:11:15.491 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:11:15.492 INFO:tasks.cephfs.test_volumes:r_size = 10737418240 exp_size = 10737418240 2024-05-28T06:11:15.493 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone26281 2024-05-28T06:11:15.710 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:15.708+0000 7f83f2000640 1 Processor -- start 2024-05-28T06:11:15.710 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:15.709+0000 7f83f2000640 1 -- start start 2024-05-28T06:11:15.712 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:15.710+0000 7f83f2000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f83ec0d06d0 0x7f83ec0d0b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:15.712 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:15.710+0000 7f83f2000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f83ec0ad630 0x7f83ec0ada60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:15.712 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:15.710+0000 7f83f2000640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83ec06dbc0 con 0x7f83ec15e810 2024-05-28T06:11:15.712 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:15.710+0000 7f83f2000640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f83ec0d1070 con 0x7f83ec0ad630 2024-05-28T06:11:15.712 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:15.710+0000 7f83f2000640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f83ec0ad1a0 con 0x7f83ec0d06d0 2024-05-28T06:11:15.712 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:15.710+0000 7f83ebe00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f83ec0ad630 0x7f83ec0ada60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:15.712 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:15.711+0000 7f83eb400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f83ec15e810 0x7f83ec072740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:57726/0 (socket says 172.21.15.96:57726) 2024-05-28T06:11:15.712 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:15.711+0000 7f83eb400640 1 -- 172.21.15.96:0/2804265951 learned_addr learned my addr 172.21.15.96:0/2804265951 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:11:15.713 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:15.711+0000 7f83e3e00640 1 --2- 172.21.15.96:0/2804265951 >> v2:172.21.15.123:3300/0 conn(0x7f83ec0d06d0 0x7f83ec0d0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:15.713 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:15.711+0000 7f83eaa00640 1 -- 172.21.15.96:0/2804265951 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2412351379 0 0) 0x7f83ec06dbc0 con 0x7f83ec15e810 2024-05-28T06:11:15.713 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:15.711+0000 7f83eaa00640 1 -- 172.21.15.96:0/2804265951 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f83d0003520 con 0x7f83ec15e810 2024-05-28T06:11:15.713 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:15.711+0000 7f83ebe00640 1 -- 172.21.15.96:0/2804265951 >> v2:172.21.15.123:3300/0 conn(0x7f83ec0d06d0 msgr2=0x7f83ec0d0b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:11:15.713 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:15.711+0000 7f83ebe00640 1 --2- 172.21.15.96:0/2804265951 >> v2:172.21.15.123:3300/0 conn(0x7f83ec0d06d0 0x7f83ec0d0b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:15.713 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:15.711+0000 7f83ebe00640 1 -- 172.21.15.96:0/2804265951 >> v1:172.21.15.96:6789/0 conn(0x7f83ec15e810 legacy=0x7f83ec072740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:15.713 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:15.711+0000 7f83ebe00640 1 -- 172.21.15.96:0/2804265951 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83ec06dea0 con 0x7f83ec0ad630 2024-05-28T06:11:15.714 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:15.712+0000 7f83ebe00640 1 --2- 172.21.15.96:0/2804265951 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83ec0ad630 0x7f83ec0ada60 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f83ec0b49b0 tx=0x7f83e402a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b6219531d537791 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:11:15.714 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:15.712+0000 7f83eaa00640 1 -- 172.21.15.96:0/2804265951 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83e4007740 con 0x7f83ec0ad630 2024-05-28T06:11:15.714 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:15.712+0000 7f83eaa00640 1 -- 172.21.15.96:0/2804265951 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83e40078e0 con 0x7f83ec0ad630 2024-05-28T06:11:15.714 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:15.712+0000 7f83eaa00640 1 -- 172.21.15.96:0/2804265951 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83e4007cb0 con 0x7f83ec0ad630 2024-05-28T06:11:15.715 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:15.713+0000 7f83f2000640 1 -- 172.21.15.96:0/2804265951 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83ec0ad630 msgr2=0x7f83ec0ada60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:15.715 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:15.713+0000 7f83f2000640 1 --2- 172.21.15.96:0/2804265951 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83ec0ad630 0x7f83ec0ada60 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f83ec0b49b0 tx=0x7f83e402a680 comp rx=0 tx=0).stop 2024-05-28T06:11:15.715 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:15.713+0000 7f83f2000640 1 -- 172.21.15.96:0/2804265951 shutdown_connections 2024-05-28T06:11:15.715 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:15.713+0000 7f83f2000640 1 --2- 172.21.15.96:0/2804265951 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83ec0ad630 0x7f83ec0ada60 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:15.715 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:15.713+0000 7f83f2000640 1 --2- 172.21.15.96:0/2804265951 >> v2:172.21.15.123:3300/0 conn(0x7f83ec0d06d0 0x7f83ec0d0b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:15.715 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:15.713+0000 7f83f2000640 1 -- 172.21.15.96:0/2804265951 >> 172.21.15.96:0/2804265951 conn(0x7f83ec06bd20 msgr2=0x7f83ec06c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:11:15.716 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:15.713+0000 7f83f2000640 1 -- 172.21.15.96:0/2804265951 shutdown_connections 2024-05-28T06:11:15.716 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:15.713+0000 7f83f2000640 1 -- 172.21.15.96:0/2804265951 wait complete. 2024-05-28T06:11:15.716 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:15.715+0000 7f83f2000640 1 Processor -- start 2024-05-28T06:11:15.717 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:15.715+0000 7f83f2000640 1 -- start start 2024-05-28T06:11:15.718 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:15.716+0000 7f83f2000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f83ec0ad630 0x7f83ec09a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:15.719 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:15.716+0000 7f83f2000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f83ec0d06d0 0x7f83ec0998f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:15.719 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:15.716+0000 7f83f2000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83ec15e810 0x7f83ec09bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:15.719 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:15.716+0000 7f83f2000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f83ec06e460 con 0x7f83ec15e810 2024-05-28T06:11:15.719 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:15.716+0000 7f83f2000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f83ec15ebf0 con 0x7f83ec0d06d0 2024-05-28T06:11:15.719 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:15.716+0000 7f83f2000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f83ec06e740 con 0x7f83ec0ad630 2024-05-28T06:11:15.719 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:15.717+0000 7f83ebe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83ec15e810 0x7f83ec09bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:15.719 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:15.717+0000 7f83ebe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83ec15e810 0x7f83ec09bbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47322/0 (socket says 172.21.15.96:47322) 2024-05-28T06:11:15.719 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:15.717+0000 7f83ebe00640 1 -- 172.21.15.96:0/293515938 learned_addr learned my addr 172.21.15.96:0/293515938 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:11:15.719 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:15.717+0000 7f83eb400640 1 --2- 172.21.15.96:0/293515938 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f83ec0ad630 0x7f83ec09a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:15.719 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:15.717+0000 7f83e3e00640 1 --2- 172.21.15.96:0/293515938 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f83ec0d06d0 0x7f83ec0998f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:15.719 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:15.717+0000 7f83ebe00640 1 -- 172.21.15.96:0/293515938 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f83ec0ad630 msgr2=0x7f83ec09a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:15.719 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:15.717+0000 7f83ebe00640 1 --2- 172.21.15.96:0/293515938 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f83ec0ad630 0x7f83ec09a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:15.719 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:15.717+0000 7f83ebe00640 1 -- 172.21.15.96:0/293515938 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f83ec0d06d0 msgr2=0x7f83ec0998f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:15.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:15.717+0000 7f83ebe00640 1 --2- 172.21.15.96:0/293515938 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f83ec0d06d0 0x7f83ec0998f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:15.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:15.717+0000 7f83ebe00640 1 -- 172.21.15.96:0/293515938 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83ec142700 con 0x7f83ec15e810 2024-05-28T06:11:15.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:15.717+0000 7f83e3e00640 1 --2- 172.21.15.96:0/293515938 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f83ec0d06d0 0x7f83ec0998f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:11:15.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:15.718+0000 7f83ebe00640 1 --2- 172.21.15.96:0/293515938 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83ec15e810 0x7f83ec09bbd0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f83e402ac70 tx=0x7f83e4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:11:15.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:15.718+0000 7f83e3400640 1 -- 172.21.15.96:0/293515938 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83e4040070 con 0x7f83ec15e810 2024-05-28T06:11:15.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:15.718+0000 7f83e3400640 1 -- 172.21.15.96:0/293515938 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83e4031bc0 con 0x7f83ec15e810 2024-05-28T06:11:15.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:15.718+0000 7f83f2000640 1 -- 172.21.15.96:0/293515938 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83ec1429e0 con 0x7f83ec15e810 2024-05-28T06:11:15.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:15.718+0000 7f83e3400640 1 -- 172.21.15.96:0/293515938 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83e4009450 con 0x7f83ec15e810 2024-05-28T06:11:15.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:15.718+0000 7f83f2000640 1 -- 172.21.15.96:0/293515938 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83ec142fa0 con 0x7f83ec15e810 2024-05-28T06:11:15.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:15.719+0000 7f83f2000640 1 -- 172.21.15.96:0/293515938 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83ec0689e0 con 0x7f83ec15e810 2024-05-28T06:11:15.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:15.720+0000 7f83e3400640 1 -- 172.21.15.96:0/293515938 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 12) ==== 91346+0+0 (secure 0 0 0) 0x7f83e4039040 con 0x7f83ec15e810 2024-05-28T06:11:15.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:15.721+0000 7f83e3400640 1 --2- 172.21.15.96:0/293515938 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f83c806cc50 0x7f83c806f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:15.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:15.721+0000 7f83e3400640 1 -- 172.21.15.96:0/293515938 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5795+0+0 (secure 0 0 0) 0x7f83e40adab0 con 0x7f83ec15e810 2024-05-28T06:11:15.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:15.722+0000 7f83eb400640 1 --2- 172.21.15.96:0/293515938 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f83c806cc50 0x7f83c806f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:15.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:15.723+0000 7f83eb400640 1 --2- 172.21.15.96:0/293515938 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f83c806cc50 0x7f83c806f110 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f83dc005880 tx=0x7f83dc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:11:15.727 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:15.726+0000 7f83e3400640 1 -- 172.21.15.96:0/293515938 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f83e407b920 con 0x7f83ec15e810 2024-05-28T06:11:15.962 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:15.961+0000 7f83f2000640 1 -- 172.21.15.96:0/293515938 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone26281", "target": ["mon-mgr", ""]}) -- 0x7f83ec15c870 con 0x7f83c806cc50 2024-05-28T06:11:16.166 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:16.164+0000 7f83e3400640 1 -- 172.21.15.96:0/293515938 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f83ec15c870 con 0x7f83c806cc50 2024-05-28T06:11:16.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:16.167+0000 7f83f2000640 1 -- 172.21.15.96:0/293515938 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f83c806cc50 msgr2=0x7f83c806f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:16.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:16.167+0000 7f83f2000640 1 --2- 172.21.15.96:0/293515938 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f83c806cc50 0x7f83c806f110 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f83dc005880 tx=0x7f83dc009040 comp rx=0 tx=0).stop 2024-05-28T06:11:16.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:16.167+0000 7f83f2000640 1 -- 172.21.15.96:0/293515938 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83ec15e810 msgr2=0x7f83ec09bbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:16.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:16.167+0000 7f83f2000640 1 --2- 172.21.15.96:0/293515938 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83ec15e810 0x7f83ec09bbd0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f83e402ac70 tx=0x7f83e4004a40 comp rx=0 tx=0).stop 2024-05-28T06:11:16.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:16.168+0000 7f83f2000640 1 -- 172.21.15.96:0/293515938 shutdown_connections 2024-05-28T06:11:16.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:16.168+0000 7f83f2000640 1 --2- 172.21.15.96:0/293515938 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f83c806cc50 0x7f83c806f110 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:16.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:16.168+0000 7f83f2000640 1 --2- 172.21.15.96:0/293515938 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83ec15e810 0x7f83ec09bbd0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:16.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:16.168+0000 7f83f2000640 1 --2- 172.21.15.96:0/293515938 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f83ec0d06d0 0x7f83ec0998f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:16.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:16.168+0000 7f83f2000640 1 --2- 172.21.15.96:0/293515938 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f83ec0ad630 0x7f83ec09a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:16.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:16.168+0000 7f83f2000640 1 -- 172.21.15.96:0/293515938 >> 172.21.15.96:0/293515938 conn(0x7f83ec06bd20 msgr2=0x7f83ec15b560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:11:16.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:16.168+0000 7f83f2000640 1 -- 172.21.15.96:0/293515938 shutdown_connections 2024-05-28T06:11:16.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:16.168+0000 7f83f2000640 1 -- 172.21.15.96:0/293515938 wait complete. 2024-05-28T06:11:16.171 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:11:16.169+0000 7fea61600640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:11:16.171 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:11:16.169+0000 7fea61600640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:11:16.171 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:11:16.169+0000 7fea61600640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:11:16.171 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:11:16.169+0000 7fea61600640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:11:16.171 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:11:16.169+0000 7fea61600640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:11:16.193 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone26282 2024-05-28T06:11:16.201 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:11:16.200+0000 7fea60200640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:11:16.201 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:11:16.200+0000 7fea60200640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:11:16.201 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:11:16.200+0000 7fea60200640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:11:16.202 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:11:16.200+0000 7fea60200640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:11:16.202 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:11:16.200+0000 7fea60200640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:11:16.389 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:16.387+0000 7fb074a00640 1 Processor -- start 2024-05-28T06:11:16.389 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:16.387+0000 7fb074a00640 1 -- start start 2024-05-28T06:11:16.390 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:16.389+0000 7fb074a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fb07015b180 0x7fb0700b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:16.390 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:16.389+0000 7fb074a00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fb07015f240 0x7fb0700ae660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:16.390 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:16.389+0000 7fb074a00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0700af200 con 0x7fb07015edd0 2024-05-28T06:11:16.390 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:16.389+0000 7fb074a00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fb07015fc30 con 0x7fb07015b180 2024-05-28T06:11:16.390 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:16.389+0000 7fb074a00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fb0700b4a40 con 0x7fb07015f240 2024-05-28T06:11:16.391 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:16.389+0000 7fb06e800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fb07015b180 0x7fb0700b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:16.391 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:16.389+0000 7fb06e800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fb07015b180 0x7fb0700b4f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47330/0 (socket says 172.21.15.96:47330) 2024-05-28T06:11:16.391 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:16.389+0000 7fb06e800640 1 -- 172.21.15.96:0/1298435136 learned_addr learned my addr 172.21.15.96:0/1298435136 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:11:16.391 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:16.390+0000 7fb06f200640 1 --2- 172.21.15.96:0/1298435136 >> v2:172.21.15.123:3300/0 conn(0x7fb07015f240 0x7fb0700ae660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:16.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:16.390+0000 7fb06e800640 1 -- 172.21.15.96:0/1298435136 >> v2:172.21.15.123:3300/0 conn(0x7fb07015f240 msgr2=0x7fb0700ae660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:11:16.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:16.390+0000 7fb06e800640 1 --2- 172.21.15.96:0/1298435136 >> v2:172.21.15.123:3300/0 conn(0x7fb07015f240 0x7fb0700ae660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:16.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:16.390+0000 7fb06e800640 1 -- 172.21.15.96:0/1298435136 >> v1:172.21.15.123:6789/0 conn(0x7fb07015edd0 legacy=0x7fb0700adf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:11:16.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:16.390+0000 7fb06e800640 1 -- 172.21.15.96:0/1298435136 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0700b3e30 con 0x7fb07015b180 2024-05-28T06:11:16.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:16.391+0000 7fb06e800640 1 --2- 172.21.15.96:0/1298435136 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb07015b180 0x7fb0700b4f00 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fb07006ffb0 tx=0x7fb06002a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a559c739bce8b9eb server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:11:16.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:16.391+0000 7fb06de00640 1 -- 172.21.15.96:0/1298435136 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb060036070 con 0x7fb07015b180 2024-05-28T06:11:16.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:16.391+0000 7fb06de00640 1 -- 172.21.15.96:0/1298435136 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb060031600 con 0x7fb07015b180 2024-05-28T06:11:16.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:16.391+0000 7fb06de00640 1 -- 172.21.15.96:0/1298435136 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0600319d0 con 0x7fb07015b180 2024-05-28T06:11:16.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:16.391+0000 7fb074a00640 1 -- 172.21.15.96:0/1298435136 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb07015b180 msgr2=0x7fb0700b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:16.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:16.391+0000 7fb074a00640 1 --2- 172.21.15.96:0/1298435136 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb07015b180 0x7fb0700b4f00 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fb07006ffb0 tx=0x7fb06002a6f0 comp rx=0 tx=0).stop 2024-05-28T06:11:16.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:16.392+0000 7fb074a00640 1 -- 172.21.15.96:0/1298435136 shutdown_connections 2024-05-28T06:11:16.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:16.392+0000 7fb074a00640 1 --2- 172.21.15.96:0/1298435136 >> v2:172.21.15.123:3300/0 conn(0x7fb07015f240 0x7fb0700ae660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:16.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:16.392+0000 7fb074a00640 1 --2- 172.21.15.96:0/1298435136 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb07015b180 0x7fb0700b4f00 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:16.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:16.392+0000 7fb074a00640 1 -- 172.21.15.96:0/1298435136 >> 172.21.15.96:0/1298435136 conn(0x7fb0700b1db0 msgr2=0x7fb0700b21c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:11:16.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:16.392+0000 7fb074a00640 1 -- 172.21.15.96:0/1298435136 shutdown_connections 2024-05-28T06:11:16.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:16.392+0000 7fb074a00640 1 -- 172.21.15.96:0/1298435136 wait complete. 2024-05-28T06:11:16.395 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:16.393+0000 7fb074a00640 1 Processor -- start 2024-05-28T06:11:16.395 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:16.393+0000 7fb074a00640 1 -- start start 2024-05-28T06:11:16.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:16.394+0000 7fb074a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb07015b180 0x7fb0700999d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:16.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:16.394+0000 7fb074a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb07015edd0 0x7fb070072540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:16.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:16.394+0000 7fb074a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb07015f240 0x7fb070072a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:16.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:16.394+0000 7fb074a00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb0700b46d0 con 0x7fb07015b180 2024-05-28T06:11:16.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:16.394+0000 7fb074a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fb0700b43f0 con 0x7fb07015edd0 2024-05-28T06:11:16.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:16.394+0000 7fb074a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fb07015f6b0 con 0x7fb07015f240 2024-05-28T06:11:16.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:16.395+0000 7fb06e800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb07015b180 0x7fb0700999d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:16.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:16.395+0000 7fb06e800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb07015b180 0x7fb0700999d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47334/0 (socket says 172.21.15.96:47334) 2024-05-28T06:11:16.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:16.395+0000 7fb06e800640 1 -- 172.21.15.96:0/1520756172 learned_addr learned my addr 172.21.15.96:0/1520756172 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:11:16.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:16.395+0000 7fb06e800640 1 -- 172.21.15.96:0/1520756172 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb07015f240 msgr2=0x7fb070072a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:16.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:16.395+0000 7fb06e800640 1 --2- 172.21.15.96:0/1520756172 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb07015f240 0x7fb070072a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:16.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:16.395+0000 7fb06e800640 1 -- 172.21.15.96:0/1520756172 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb07015edd0 msgr2=0x7fb070072540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:16.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:16.395+0000 7fb06e800640 1 --2- 172.21.15.96:0/1520756172 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb07015edd0 0x7fb070072540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:16.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:16.395+0000 7fb06e800640 1 -- 172.21.15.96:0/1520756172 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb070151aa0 con 0x7fb07015b180 2024-05-28T06:11:16.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:16.395+0000 7fb06e800640 1 --2- 172.21.15.96:0/1520756172 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb07015b180 0x7fb0700999d0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fb060000f80 tx=0x7fb060004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:11:16.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:16.396+0000 7fb067400640 1 -- 172.21.15.96:0/1520756172 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb060036050 con 0x7fb07015b180 2024-05-28T06:11:16.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:16.396+0000 7fb067400640 1 -- 172.21.15.96:0/1520756172 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb060042070 con 0x7fb07015b180 2024-05-28T06:11:16.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:16.396+0000 7fb074a00640 1 -- 172.21.15.96:0/1520756172 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb070151d80 con 0x7fb07015b180 2024-05-28T06:11:16.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:16.396+0000 7fb067400640 1 -- 172.21.15.96:0/1520756172 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb060009510 con 0x7fb07015b180 2024-05-28T06:11:16.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:16.396+0000 7fb074a00640 1 -- 172.21.15.96:0/1520756172 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb070152340 con 0x7fb07015b180 2024-05-28T06:11:16.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:16.397+0000 7fb074a00640 1 -- 172.21.15.96:0/1520756172 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0700689e0 con 0x7fb07015b180 2024-05-28T06:11:16.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:16.398+0000 7fb067400640 1 -- 172.21.15.96:0/1520756172 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 12) ==== 91346+0+0 (secure 0 0 0) 0x7fb060007c70 con 0x7fb07015b180 2024-05-28T06:11:16.400 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:16.399+0000 7fb067400640 1 --2- 172.21.15.96:0/1520756172 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb04c06cc50 0x7fb04c06f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:16.400 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:16.399+0000 7fb067400640 1 -- 172.21.15.96:0/1520756172 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5795+0+0 (secure 0 0 0) 0x7fb0600ad1f0 con 0x7fb07015b180 2024-05-28T06:11:16.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:16.400+0000 7fb065e00640 1 --2- 172.21.15.96:0/1520756172 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb04c06cc50 0x7fb04c06f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:16.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:16.401+0000 7fb065e00640 1 --2- 172.21.15.96:0/1520756172 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb04c06cc50 0x7fb04c06f110 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fb0701531e0 tx=0x7fb0580079b0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:11:16.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:16.404+0000 7fb067400640 1 -- 172.21.15.96:0/1520756172 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb06007b060 con 0x7fb07015b180 2024-05-28T06:11:16.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:16.622+0000 7fb074a00640 1 -- 172.21.15.96:0/1520756172 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone26282", "target": ["mon-mgr", ""]}) -- 0x7fb0700ad4d0 con 0x7fb04c06cc50 2024-05-28T06:11:17.174 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:11:17.173+0000 7fea4d600640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:11:17.174 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:11:17.173+0000 7fea4d600640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:11:17.174 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:11:17.173+0000 7fea4d600640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:11:17.175 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:11:17.173+0000 7fea4d600640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:11:17.175 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:11:17.173+0000 7fea4d600640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:11:17.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:17.431+0000 7fb067400640 1 -- 172.21.15.96:0/1520756172 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fb0700ad4d0 con 0x7fb04c06cc50 2024-05-28T06:11:17.439 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:17.437+0000 7fb074a00640 1 -- 172.21.15.96:0/1520756172 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb04c06cc50 msgr2=0x7fb04c06f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:17.439 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:17.437+0000 7fb074a00640 1 --2- 172.21.15.96:0/1520756172 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb04c06cc50 0x7fb04c06f110 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fb0701531e0 tx=0x7fb0580079b0 comp rx=0 tx=0).stop 2024-05-28T06:11:17.439 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:17.437+0000 7fb074a00640 1 -- 172.21.15.96:0/1520756172 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb07015b180 msgr2=0x7fb0700999d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:17.439 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:17.437+0000 7fb074a00640 1 --2- 172.21.15.96:0/1520756172 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb07015b180 0x7fb0700999d0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fb060000f80 tx=0x7fb060004a40 comp rx=0 tx=0).stop 2024-05-28T06:11:17.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:17.440+0000 7fb074a00640 1 -- 172.21.15.96:0/1520756172 shutdown_connections 2024-05-28T06:11:17.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:17.440+0000 7fb074a00640 1 --2- 172.21.15.96:0/1520756172 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb04c06cc50 0x7fb04c06f110 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:17.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:17.440+0000 7fb074a00640 1 --2- 172.21.15.96:0/1520756172 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb07015f240 0x7fb070072a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:17.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:17.440+0000 7fb074a00640 1 --2- 172.21.15.96:0/1520756172 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb07015edd0 0x7fb070072540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:17.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:17.440+0000 7fb074a00640 1 --2- 172.21.15.96:0/1520756172 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb07015b180 0x7fb0700999d0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:17.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:17.440+0000 7fb074a00640 1 -- 172.21.15.96:0/1520756172 >> 172.21.15.96:0/1520756172 conn(0x7fb0700b1db0 msgr2=0x7fb0700b0170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:11:17.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:17.440+0000 7fb074a00640 1 -- 172.21.15.96:0/1520756172 shutdown_connections 2024-05-28T06:11:17.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:17.440+0000 7fb074a00640 1 -- 172.21.15.96:0/1520756172 wait complete. 2024-05-28T06:11:17.458 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:11:17.458+0000 7fea60c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:11:17.458 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:11:17.458+0000 7fea60c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:11:17.458 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:11:17.458+0000 7fea60c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:11:17.458 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:11:17.458+0000 7fea60c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:11:17.459 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:11:17.458+0000 7fea60c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:11:17.471 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone26283 2024-05-28T06:11:17.755 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:17.753+0000 7fea57000640 1 Processor -- start 2024-05-28T06:11:17.755 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:17.753+0000 7fea57000640 1 -- start start 2024-05-28T06:11:17.757 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:17.755+0000 7fea57000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fea50162170 0x7fea500d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:17.757 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:17.755+0000 7fea57000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fea500d02d0 0x7fea500b4c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:17.757 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:17.756+0000 7fea57000640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea500d2d60 con 0x7fea500d15e0 2024-05-28T06:11:17.757 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:17.756+0000 7fea57000640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fea500d1a20 con 0x7fea500d02d0 2024-05-28T06:11:17.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:17.756+0000 7fea57000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fea500d3040 con 0x7fea50162170 2024-05-28T06:11:17.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:17.756+0000 7fea55000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fea500d02d0 0x7fea500b4c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:17.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:17.756+0000 7fea55000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fea500d02d0 0x7fea500b4c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47340/0 (socket says 172.21.15.96:47340) 2024-05-28T06:11:17.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:17.756+0000 7fea55000640 1 -- 172.21.15.96:0/2823115015 learned_addr learned my addr 172.21.15.96:0/2823115015 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:11:17.759 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:17.756+0000 7fea55000640 1 -- 172.21.15.96:0/2823115015 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fea50162170 msgr2=0x7fea500d10a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-28T06:11:17.759 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:17.756+0000 7fea4fe00640 1 --2- 172.21.15.96:0/2823115015 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fea50162170 0x7fea500d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:17.759 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:17.757+0000 7fea55000640 1 --2- 172.21.15.96:0/2823115015 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fea50162170 0x7fea500d10a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:17.759 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:17.757+0000 7fea55000640 1 -- 172.21.15.96:0/2823115015 >> v1:172.21.15.96:6789/0 conn(0x7fea500d15e0 legacy=0x7fea500b4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:17.759 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:17.757+0000 7fea55000640 1 -- 172.21.15.96:0/2823115015 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea500d3320 con 0x7fea500d02d0 2024-05-28T06:11:17.760 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:17.759+0000 7fea55000640 1 --2- 172.21.15.96:0/2823115015 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fea500d02d0 0x7fea500b4c40 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fea48009e50 tx=0x7fea4802a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91f489c953345201 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:11:17.761 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:17.759+0000 7fea4ea00640 1 -- 172.21.15.96:0/2823115015 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea48036070 con 0x7fea500d02d0 2024-05-28T06:11:17.761 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:17.759+0000 7fea4ea00640 1 -- 172.21.15.96:0/2823115015 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea48031600 con 0x7fea500d02d0 2024-05-28T06:11:17.761 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:17.759+0000 7fea4ea00640 1 -- 172.21.15.96:0/2823115015 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea480319d0 con 0x7fea500d02d0 2024-05-28T06:11:17.761 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:17.760+0000 7fea57000640 1 -- 172.21.15.96:0/2823115015 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fea500d02d0 msgr2=0x7fea500b4c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:17.761 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:17.760+0000 7fea57000640 1 --2- 172.21.15.96:0/2823115015 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fea500d02d0 0x7fea500b4c40 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fea48009e50 tx=0x7fea4802a710 comp rx=0 tx=0).stop 2024-05-28T06:11:17.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:17.760+0000 7fea57000640 1 -- 172.21.15.96:0/2823115015 shutdown_connections 2024-05-28T06:11:17.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:17.760+0000 7fea57000640 1 --2- 172.21.15.96:0/2823115015 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fea500d02d0 0x7fea500b4c40 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:17.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:17.760+0000 7fea57000640 1 --2- 172.21.15.96:0/2823115015 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fea50162170 0x7fea500d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:17.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:17.760+0000 7fea57000640 1 -- 172.21.15.96:0/2823115015 >> 172.21.15.96:0/2823115015 conn(0x7fea5008bfc0 msgr2=0x7fea500d46b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:11:17.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:17.761+0000 7fea57000640 1 -- 172.21.15.96:0/2823115015 shutdown_connections 2024-05-28T06:11:17.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:17.761+0000 7fea57000640 1 -- 172.21.15.96:0/2823115015 wait complete. 2024-05-28T06:11:17.764 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:17.763+0000 7fea57000640 1 Processor -- start 2024-05-28T06:11:17.764 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:17.763+0000 7fea57000640 1 -- start start 2024-05-28T06:11:17.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:17.764+0000 7fea57000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fea500d02d0 0x7fea5014eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:17.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:17.764+0000 7fea57000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fea500d15e0 0x7fea5006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:17.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:17.764+0000 7fea57000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fea50162170 0x7fea5006d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:17.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:17.764+0000 7fea57000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fea500d29e0 con 0x7fea50162170 2024-05-28T06:11:17.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:17.764+0000 7fea57000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fea500b4010 con 0x7fea500d02d0 2024-05-28T06:11:17.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:17.764+0000 7fea57000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fea500b37a0 con 0x7fea500d15e0 2024-05-28T06:11:17.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:17.764+0000 7fea55000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fea50162170 0x7fea5006d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:17.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:17.764+0000 7fea55000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fea50162170 0x7fea5006d870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47346/0 (socket says 172.21.15.96:47346) 2024-05-28T06:11:17.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:17.764+0000 7fea55000640 1 -- 172.21.15.96:0/4132266542 learned_addr learned my addr 172.21.15.96:0/4132266542 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:11:17.767 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:17.764+0000 7fea4fe00640 1 --2- 172.21.15.96:0/4132266542 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fea500d02d0 0x7fea5014eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:17.767 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:17.765+0000 7fea4f400640 1 --2- 172.21.15.96:0/4132266542 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fea500d15e0 0x7fea5006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:17.768 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:17.765+0000 7fea55000640 1 -- 172.21.15.96:0/4132266542 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fea500d15e0 msgr2=0x7fea5006d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:17.768 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:17.765+0000 7fea55000640 1 --2- 172.21.15.96:0/4132266542 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fea500d15e0 0x7fea5006d330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:17.768 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:17.765+0000 7fea55000640 1 -- 172.21.15.96:0/4132266542 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fea500d02d0 msgr2=0x7fea5014eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:17.768 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:17.765+0000 7fea55000640 1 --2- 172.21.15.96:0/4132266542 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fea500d02d0 0x7fea5014eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:17.768 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:17.765+0000 7fea55000640 1 -- 172.21.15.96:0/4132266542 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea50152910 con 0x7fea50162170 2024-05-28T06:11:17.768 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:17.765+0000 7fea55000640 1 --2- 172.21.15.96:0/4132266542 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fea50162170 0x7fea5006d870 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fea48009f70 tx=0x7fea48004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:11:17.768 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:17.766+0000 7fea47e00640 1 -- 172.21.15.96:0/4132266542 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea4803f020 con 0x7fea50162170 2024-05-28T06:11:17.768 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:17.766+0000 7fea47e00640 1 -- 172.21.15.96:0/4132266542 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea48037040 con 0x7fea50162170 2024-05-28T06:11:17.768 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:17.766+0000 7fea47e00640 1 -- 172.21.15.96:0/4132266542 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea48036070 con 0x7fea50162170 2024-05-28T06:11:17.768 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:17.766+0000 7fea57000640 1 -- 172.21.15.96:0/4132266542 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea500cd250 con 0x7fea50162170 2024-05-28T06:11:17.768 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:17.766+0000 7fea57000640 1 -- 172.21.15.96:0/4132266542 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea500cd810 con 0x7fea50162170 2024-05-28T06:11:17.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:17.767+0000 7fea57000640 1 -- 172.21.15.96:0/4132266542 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea500689e0 con 0x7fea50162170 2024-05-28T06:11:17.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:17.768+0000 7fea47e00640 1 -- 172.21.15.96:0/4132266542 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 13) ==== 91550+0+0 (secure 0 0 0) 0x7fea48042020 con 0x7fea50162170 2024-05-28T06:11:17.770 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:17.769+0000 7fea47e00640 1 --2- 172.21.15.96:0/4132266542 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fea3006ce90 0x7fea3006f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:17.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:17.769+0000 7fea47e00640 1 -- 172.21.15.96:0/4132266542 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5795+0+0 (secure 0 0 0) 0x7fea480acfa0 con 0x7fea50162170 2024-05-28T06:11:17.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:17.770+0000 7fea4fe00640 1 --2- 172.21.15.96:0/4132266542 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fea3006ce90 0x7fea3006f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:17.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:17.771+0000 7fea4fe00640 1 --2- 172.21.15.96:0/4132266542 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fea3006ce90 0x7fea3006f350 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fea40006fd0 tx=0x7fea40008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:11:17.777 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:17.775+0000 7fea47e00640 1 -- 172.21.15.96:0/4132266542 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fea48049020 con 0x7fea50162170 2024-05-28T06:11:18.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:18.167+0000 7fea57000640 1 -- 172.21.15.96:0/4132266542 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone26283", "target": ["mon-mgr", ""]}) -- 0x7fea500b6870 con 0x7fea3006ce90 2024-05-28T06:11:18.717 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:18.715+0000 7fea47e00640 1 -- 172.21.15.96:0/4132266542 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fea500b6870 con 0x7fea3006ce90 2024-05-28T06:11:18.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:18.719+0000 7fea57000640 1 -- 172.21.15.96:0/4132266542 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fea3006ce90 msgr2=0x7fea3006f350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:18.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:18.719+0000 7fea57000640 1 --2- 172.21.15.96:0/4132266542 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fea3006ce90 0x7fea3006f350 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fea40006fd0 tx=0x7fea40008040 comp rx=0 tx=0).stop 2024-05-28T06:11:18.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:18.719+0000 7fea57000640 1 -- 172.21.15.96:0/4132266542 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fea50162170 msgr2=0x7fea5006d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:18.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:18.719+0000 7fea57000640 1 --2- 172.21.15.96:0/4132266542 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fea50162170 0x7fea5006d870 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fea48009f70 tx=0x7fea48004a40 comp rx=0 tx=0).stop 2024-05-28T06:11:18.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:18.720+0000 7fea57000640 1 -- 172.21.15.96:0/4132266542 shutdown_connections 2024-05-28T06:11:18.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:18.720+0000 7fea57000640 1 --2- 172.21.15.96:0/4132266542 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fea3006ce90 0x7fea3006f350 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:18.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:18.720+0000 7fea57000640 1 --2- 172.21.15.96:0/4132266542 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fea50162170 0x7fea5006d870 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:18.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:18.720+0000 7fea57000640 1 --2- 172.21.15.96:0/4132266542 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fea500d15e0 0x7fea5006d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:18.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:18.720+0000 7fea57000640 1 --2- 172.21.15.96:0/4132266542 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fea500d02d0 0x7fea5014eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:18.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:18.720+0000 7fea57000640 1 -- 172.21.15.96:0/4132266542 >> 172.21.15.96:0/4132266542 conn(0x7fea5008bfc0 msgr2=0x7fea500b51f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:11:18.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:18.721+0000 7fea57000640 1 -- 172.21.15.96:0/4132266542 shutdown_connections 2024-05-28T06:11:18.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:18.721+0000 7fea57000640 1 -- 172.21.15.96:0/4132266542 wait complete. 2024-05-28T06:11:18.756 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone26284 2024-05-28T06:11:18.826 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:11:18.826+0000 7fea60c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:11:18.827 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:11:18.826+0000 7fea60c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:11:18.827 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:11:18.826+0000 7fea60c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:11:18.827 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:11:18.826+0000 7fea60c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:11:18.827 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:11:18.826+0000 7fea60c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:11:19.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:19.027+0000 7faa8a000640 1 Processor -- start 2024-05-28T06:11:19.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:19.027+0000 7faa8a000640 1 -- start start 2024-05-28T06:11:19.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:19.028+0000 7faa8a000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faa8415c830 0x7faa840b4a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:19.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:19.028+0000 7faa8a000640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa840cda90 con 0x7faa840d0010 2024-05-28T06:11:19.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:19.028+0000 7faa8a000640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa840d0cc0 con 0x7faa8409aa30 2024-05-28T06:11:19.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:19.028+0000 7faa8a000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7faa8415cc10 con 0x7faa8415c830 2024-05-28T06:11:19.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:19.028+0000 7faa82a00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7faa840d0010 0x7faa840d0450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:46920/0 (socket says 172.21.15.96:46920) 2024-05-28T06:11:19.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:19.028+0000 7faa82a00640 1 -- 172.21.15.96:0/2122862181 learned_addr learned my addr 172.21.15.96:0/2122862181 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:11:19.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:19.029+0000 7faa82000640 1 -- 172.21.15.96:0/2122862181 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1150918130 0 0) 0x7faa840cda90 con 0x7faa840d0010 2024-05-28T06:11:19.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:19.029+0000 7faa82000640 1 -- 172.21.15.96:0/2122862181 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa68003520 con 0x7faa840d0010 2024-05-28T06:11:19.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:19.029+0000 7faa83400640 1 --2- 172.21.15.96:0/2122862181 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faa8415c830 0x7faa840b4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:19.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:19.029+0000 7faa82000640 1 -- 172.21.15.96:0/2122862181 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3037391858 0 0) 0x7faa68003520 con 0x7faa840d0010 2024-05-28T06:11:19.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:19.029+0000 7faa82000640 1 -- 172.21.15.96:0/2122862181 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faa8415c830 msgr2=0x7faa840b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:11:19.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:19.029+0000 7faa82000640 1 --2- 172.21.15.96:0/2122862181 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faa8415c830 0x7faa840b4a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:19.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:19.029+0000 7faa82000640 1 -- 172.21.15.96:0/2122862181 >> v1:172.21.15.123:6789/0 conn(0x7faa8409aa30 legacy=0x7faa8409aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:19.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:19.029+0000 7faa82000640 1 -- 172.21.15.96:0/2122862181 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa840d0fa0 con 0x7faa840d0010 2024-05-28T06:11:19.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:19.029+0000 7faa82000640 1 -- 172.21.15.96:0/2122862181 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7faa70001670 con 0x7faa840d0010 2024-05-28T06:11:19.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:19.029+0000 7faa82000640 1 -- 172.21.15.96:0/2122862181 >> v1:172.21.15.96:6789/0 conn(0x7faa840d0010 legacy=0x7faa840d0450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:19.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:19.029+0000 7faa82000640 1 --2- 172.21.15.96:0/2122862181 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa680039e0 0x7faa68003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:19.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:19.029+0000 7faa82000640 1 --2- 172.21.15.96:0/2122862181 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa68004800 0x7faa68004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:19.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:19.029+0000 7faa82000640 1 --2- 172.21.15.96:0/2122862181 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faa8415c830 0x7faa680051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:19.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:19.029+0000 7faa82000640 1 -- 172.21.15.96:0/2122862181 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7faa8415cc10 con 0x7faa680039e0 2024-05-28T06:11:19.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:19.029+0000 7faa82000640 1 -- 172.21.15.96:0/2122862181 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7faa6800ba70 con 0x7faa68004800 2024-05-28T06:11:19.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:19.029+0000 7faa82000640 1 -- 172.21.15.96:0/2122862181 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7faa6800bc70 con 0x7faa8415c830 2024-05-28T06:11:19.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:19.030+0000 7faa83400640 1 --2- 172.21.15.96:0/2122862181 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa680039e0 0x7faa68003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:19.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:19.030+0000 7faa83400640 1 -- 172.21.15.96:0/2122862181 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faa8415c830 msgr2=0x7faa680051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:11:19.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:19.030+0000 7faa83400640 1 --2- 172.21.15.96:0/2122862181 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faa8415c830 0x7faa680051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:19.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:19.030+0000 7faa83400640 1 -- 172.21.15.96:0/2122862181 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa68004800 msgr2=0x7faa68004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:11:19.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:19.030+0000 7faa83400640 1 --2- 172.21.15.96:0/2122862181 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa68004800 0x7faa68004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:19.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:19.030+0000 7faa83400640 1 -- 172.21.15.96:0/2122862181 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa68003520 con 0x7faa680039e0 2024-05-28T06:11:19.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:19.031+0000 7faa83400640 1 --2- 172.21.15.96:0/2122862181 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa680039e0 0x7faa68003dc0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7faa7400ba00 tx=0x7faa68005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21c759544656743a server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:11:19.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:19.031+0000 7faa82000640 1 -- 172.21.15.96:0/2122862181 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa74030070 con 0x7faa680039e0 2024-05-28T06:11:19.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:19.031+0000 7faa82000640 1 -- 172.21.15.96:0/2122862181 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa7402adb0 con 0x7faa680039e0 2024-05-28T06:11:19.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:19.031+0000 7faa82000640 1 -- 172.21.15.96:0/2122862181 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa7400b600 con 0x7faa680039e0 2024-05-28T06:11:19.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:19.032+0000 7faa8a000640 1 -- 172.21.15.96:0/2122862181 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa680039e0 msgr2=0x7faa68003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:19.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:19.032+0000 7faa8a000640 1 --2- 172.21.15.96:0/2122862181 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa680039e0 0x7faa68003dc0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7faa7400ba00 tx=0x7faa68005830 comp rx=0 tx=0).stop 2024-05-28T06:11:19.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:19.032+0000 7faa83e00640 1 -- 172.21.15.96:0/2122862181 reap_dead start 2024-05-28T06:11:19.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:19.032+0000 7faa8a000640 1 -- 172.21.15.96:0/2122862181 shutdown_connections 2024-05-28T06:11:19.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:19.032+0000 7faa8a000640 1 -- 172.21.15.96:0/2122862181 >> 172.21.15.96:0/2122862181 conn(0x7faa840b3f30 msgr2=0x7faa840b4340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:11:19.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:19.033+0000 7faa8a000640 1 -- 172.21.15.96:0/2122862181 shutdown_connections 2024-05-28T06:11:19.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:19.033+0000 7faa8a000640 1 -- 172.21.15.96:0/2122862181 wait complete. 2024-05-28T06:11:19.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:19.035+0000 7faa8a000640 1 Processor -- start 2024-05-28T06:11:19.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:19.035+0000 7faa8a000640 1 -- start start 2024-05-28T06:11:19.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:19.037+0000 7faa8a000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa680039e0 0x7faa840c7d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:19.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:19.037+0000 7faa8a000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa840c9510 0x7faa84154680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:19.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:19.037+0000 7faa8a000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faa84154bc0 0x7faa840b05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:19.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:19.037+0000 7faa8a000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7faa840d1400 con 0x7faa680039e0 2024-05-28T06:11:19.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:19.037+0000 7faa8a000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7faa840d1280 con 0x7faa840c9510 2024-05-28T06:11:19.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:19.037+0000 7faa8a000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7faa840ce250 con 0x7faa84154bc0 2024-05-28T06:11:19.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:19.039+0000 7faa82a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa840c9510 0x7faa84154680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:19.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:19.039+0000 7faa83e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faa84154bc0 0x7faa840b05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:19.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:19.040+0000 7faa83e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faa84154bc0 0x7faa840b05e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.96:53142/0 (socket says 172.21.15.96:53142) 2024-05-28T06:11:19.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:19.040+0000 7faa83e00640 1 -- 172.21.15.96:0/4114742127 learned_addr learned my addr 172.21.15.96:0/4114742127 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:11:19.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:19.040+0000 7faa83400640 1 --2- 172.21.15.96:0/4114742127 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa680039e0 0x7faa840c7d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:19.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:19.042+0000 7faa83400640 1 -- 172.21.15.96:0/4114742127 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faa84154bc0 msgr2=0x7faa840b05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:19.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:19.042+0000 7faa83400640 1 --2- 172.21.15.96:0/4114742127 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faa84154bc0 0x7faa840b05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:19.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:19.042+0000 7faa83400640 1 -- 172.21.15.96:0/4114742127 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa840c9510 msgr2=0x7faa84154680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:19.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:19.042+0000 7faa83400640 1 --2- 172.21.15.96:0/4114742127 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa840c9510 0x7faa84154680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:19.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:19.042+0000 7faa83400640 1 -- 172.21.15.96:0/4114742127 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa840c9990 con 0x7faa680039e0 2024-05-28T06:11:19.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:19.044+0000 7faa83400640 1 --2- 172.21.15.96:0/4114742127 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa680039e0 0x7faa840c7d90 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7faa680057e0 tx=0x7faa74004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:11:19.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:19.045+0000 7faa7b400640 1 -- 172.21.15.96:0/4114742127 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa74043020 con 0x7faa680039e0 2024-05-28T06:11:19.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:19.045+0000 7faa7b400640 1 -- 172.21.15.96:0/4114742127 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa74035eb0 con 0x7faa680039e0 2024-05-28T06:11:19.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:19.045+0000 7faa7b400640 1 -- 172.21.15.96:0/4114742127 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa7403f8c0 con 0x7faa680039e0 2024-05-28T06:11:19.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:19.045+0000 7faa8a000640 1 -- 172.21.15.96:0/4114742127 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa8414ef90 con 0x7faa680039e0 2024-05-28T06:11:19.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:19.046+0000 7faa8a000640 1 -- 172.21.15.96:0/4114742127 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa840b1be0 con 0x7faa680039e0 2024-05-28T06:11:19.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:19.048+0000 7faa8a000640 1 -- 172.21.15.96:0/4114742127 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa8406be20 con 0x7faa680039e0 2024-05-28T06:11:19.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:19.050+0000 7faa7b400640 1 -- 172.21.15.96:0/4114742127 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 13) ==== 91550+0+0 (secure 0 0 0) 0x7faa74046020 con 0x7faa680039e0 2024-05-28T06:11:19.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:19.052+0000 7faa7b400640 1 --2- 172.21.15.96:0/4114742127 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7faa6406cee0 0x7faa6406f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:19.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:19.052+0000 7faa7b400640 1 -- 172.21.15.96:0/4114742127 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5795+0+0 (secure 0 0 0) 0x7faa740aced0 con 0x7faa680039e0 2024-05-28T06:11:19.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:19.058+0000 7faa82a00640 1 --2- 172.21.15.96:0/4114742127 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7faa6406cee0 0x7faa6406f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:19.065 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:19.063+0000 7faa82a00640 1 --2- 172.21.15.96:0/4114742127 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7faa6406cee0 0x7faa6406f3a0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7faa8406de10 tx=0x7faa70001af0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:11:19.065 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:19.063+0000 7faa7b400640 1 -- 172.21.15.96:0/4114742127 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7faa8406be20 con 0x7faa680039e0 2024-05-28T06:11:19.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:19.448+0000 7faa8a000640 1 -- 172.21.15.96:0/4114742127 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone26284", "target": ["mon-mgr", ""]}) -- 0x7faa840690f0 con 0x7faa6406cee0 2024-05-28T06:11:19.851 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:19.848+0000 7faa7b400640 1 -- 172.21.15.96:0/4114742127 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mgrmap(e 14) ==== 91601+0+0 (secure 0 0 0) 0x7faa74076a00 con 0x7faa680039e0 2024-05-28T06:11:20.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:20.330+0000 7faa7b400640 1 -- 172.21.15.96:0/4114742127 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7faa840690f0 con 0x7faa6406cee0 2024-05-28T06:11:20.337 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:20.335+0000 7faa8a000640 1 -- 172.21.15.96:0/4114742127 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7faa6406cee0 msgr2=0x7faa6406f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:20.337 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:20.335+0000 7faa8a000640 1 --2- 172.21.15.96:0/4114742127 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7faa6406cee0 0x7faa6406f3a0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7faa8406de10 tx=0x7faa70001af0 comp rx=0 tx=0).stop 2024-05-28T06:11:20.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:20.335+0000 7faa8a000640 1 -- 172.21.15.96:0/4114742127 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa680039e0 msgr2=0x7faa840c7d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:20.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:20.335+0000 7faa8a000640 1 --2- 172.21.15.96:0/4114742127 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa680039e0 0x7faa840c7d90 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7faa680057e0 tx=0x7faa74004730 comp rx=0 tx=0).stop 2024-05-28T06:11:20.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:20.335+0000 7faa8a000640 1 -- 172.21.15.96:0/4114742127 shutdown_connections 2024-05-28T06:11:20.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:20.336+0000 7faa8a000640 1 --2- 172.21.15.96:0/4114742127 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7faa6406cee0 0x7faa6406f3a0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:20.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:20.336+0000 7faa8a000640 1 --2- 172.21.15.96:0/4114742127 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faa84154bc0 0x7faa840b05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:20.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:20.336+0000 7faa8a000640 1 --2- 172.21.15.96:0/4114742127 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa840c9510 0x7faa84154680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:20.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:20.336+0000 7faa8a000640 1 --2- 172.21.15.96:0/4114742127 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa680039e0 0x7faa840c7d90 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:20.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:20.336+0000 7faa8a000640 1 -- 172.21.15.96:0/4114742127 >> 172.21.15.96:0/4114742127 conn(0x7faa840b3f30 msgr2=0x7faa840cf4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:11:20.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:20.336+0000 7faa8a000640 1 -- 172.21.15.96:0/4114742127 shutdown_connections 2024-05-28T06:11:20.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:20.336+0000 7faa8a000640 1 -- 172.21.15.96:0/4114742127 wait complete. 2024-05-28T06:11:20.365 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-05-28T06:11:20.667 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:20.665+0000 7f9f03400640 1 Processor -- start 2024-05-28T06:11:20.667 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:20.665+0000 7f9f03400640 1 -- start start 2024-05-28T06:11:20.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:20.666+0000 7f9f03400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9efc15ea40 0x7f9efc0d4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:20.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:20.666+0000 7f9f03400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f9efc06c950 0x7f9efc06cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:20.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:20.666+0000 7f9f03400640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9efc0cf7e0 con 0x7f9efc0d1040 2024-05-28T06:11:20.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:20.666+0000 7f9f03400640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f9efc15ee20 con 0x7f9efc06c950 2024-05-28T06:11:20.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:20.666+0000 7f9f03400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f9efc15fe50 con 0x7f9efc15ea40 2024-05-28T06:11:20.669 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:20.667+0000 7f9f01400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f9efc06c950 0x7f9efc06cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:20.669 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:20.667+0000 7f9f01400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f9efc06c950 0x7f9efc06cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:56404/0 (socket says 172.21.15.96:56404) 2024-05-28T06:11:20.669 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:20.667+0000 7f9f01400640 1 -- 172.21.15.96:0/3933295610 learned_addr learned my addr 172.21.15.96:0/3933295610 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:11:20.669 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:20.668+0000 7f9f01400640 1 -- 172.21.15.96:0/3933295610 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9efc15ea40 msgr2=0x7f9efc0d4f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:11:20.670 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:20.668+0000 7f9f01400640 1 --2- 172.21.15.96:0/3933295610 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9efc15ea40 0x7f9efc0d4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:20.670 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:20.668+0000 7f9f01400640 1 -- 172.21.15.96:0/3933295610 >> v1:172.21.15.96:6789/0 conn(0x7f9efc0d1040 legacy=0x7f9efc0d1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:11:20.671 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:20.669+0000 7f9f01400640 1 -- 172.21.15.96:0/3933295610 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9efc0cfa80 con 0x7f9efc06c950 2024-05-28T06:11:20.672 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:20.671+0000 7f9f01400640 1 --2- 172.21.15.96:0/3933295610 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9efc06c950 0x7f9efc06cda0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f9ef4009e50 tx=0x7f9ef402a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d56568da13fb4865 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:11:20.672 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:20.671+0000 7f9efb400640 1 -- 172.21.15.96:0/3933295610 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ef4036070 con 0x7f9efc06c950 2024-05-28T06:11:20.672 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:20.671+0000 7f9efb400640 1 -- 172.21.15.96:0/3933295610 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ef4031600 con 0x7f9efc06c950 2024-05-28T06:11:20.673 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:20.672+0000 7f9efb400640 1 -- 172.21.15.96:0/3933295610 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ef40319d0 con 0x7f9efc06c950 2024-05-28T06:11:20.673 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:20.672+0000 7f9f03400640 1 -- 172.21.15.96:0/3933295610 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9efc06c950 msgr2=0x7f9efc06cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:20.673 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:20.672+0000 7f9f03400640 1 --2- 172.21.15.96:0/3933295610 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9efc06c950 0x7f9efc06cda0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f9ef4009e50 tx=0x7f9ef402a680 comp rx=0 tx=0).stop 2024-05-28T06:11:20.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:20.672+0000 7f9f03400640 1 -- 172.21.15.96:0/3933295610 shutdown_connections 2024-05-28T06:11:20.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:20.673+0000 7f9f03400640 1 --2- 172.21.15.96:0/3933295610 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9efc06c950 0x7f9efc06cda0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:20.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:20.673+0000 7f9f03400640 1 --2- 172.21.15.96:0/3933295610 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9efc15ea40 0x7f9efc0d4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:20.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:20.673+0000 7f9f03400640 1 -- 172.21.15.96:0/3933295610 >> 172.21.15.96:0/3933295610 conn(0x7f9efc08bfc0 msgr2=0x7f9efc0d4630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:11:20.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:20.673+0000 7f9f03400640 1 -- 172.21.15.96:0/3933295610 shutdown_connections 2024-05-28T06:11:20.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:20.673+0000 7f9f03400640 1 -- 172.21.15.96:0/3933295610 wait complete. 2024-05-28T06:11:20.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:20.675+0000 7f9f03400640 1 Processor -- start 2024-05-28T06:11:20.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:20.675+0000 7f9f03400640 1 -- start start 2024-05-28T06:11:20.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:20.676+0000 7f9f03400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9efc06c950 0x7f9efc15d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:20.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:20.676+0000 7f9f03400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9efc0d1040 0x7f9efc0b96b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:20.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:20.676+0000 7f9f03400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9efc15ea40 0x7f9efc0b7250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:20.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:20.676+0000 7f9f03400640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f9efc0d0320 con 0x7f9efc06c950 2024-05-28T06:11:20.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:20.676+0000 7f9f03400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f9efc0d0040 con 0x7f9efc15ea40 2024-05-28T06:11:20.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:20.676+0000 7f9f03400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f9efc0d5440 con 0x7f9efc0d1040 2024-05-28T06:11:20.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:20.677+0000 7f9f00a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9efc06c950 0x7f9efc15d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:20.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:20.677+0000 7f9f00a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9efc06c950 0x7f9efc15d1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:56414/0 (socket says 172.21.15.96:56414) 2024-05-28T06:11:20.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:20.677+0000 7f9f00a00640 1 -- 172.21.15.96:0/223091871 learned_addr learned my addr 172.21.15.96:0/223091871 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:11:20.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:20.677+0000 7f9efbe00640 1 --2- 172.21.15.96:0/223091871 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9efc0d1040 0x7f9efc0b96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:20.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:20.677+0000 7f9f01400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9efc15ea40 0x7f9efc0b7250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:20.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:20.680+0000 7f9f00a00640 1 -- 172.21.15.96:0/223091871 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9efc0d1040 msgr2=0x7f9efc0b96b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:20.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:20.680+0000 7f9f00a00640 1 --2- 172.21.15.96:0/223091871 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9efc0d1040 0x7f9efc0b96b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:20.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:20.680+0000 7f9f00a00640 1 -- 172.21.15.96:0/223091871 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9efc15ea40 msgr2=0x7f9efc0b7250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:20.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:20.680+0000 7f9f00a00640 1 --2- 172.21.15.96:0/223091871 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9efc15ea40 0x7f9efc0b7250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:20.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:20.680+0000 7f9f00a00640 1 -- 172.21.15.96:0/223091871 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9efc0b8610 con 0x7f9efc06c950 2024-05-28T06:11:20.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:20.680+0000 7f9f01400640 1 --2- 172.21.15.96:0/223091871 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9efc15ea40 0x7f9efc0b7250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-28T06:11:20.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:20.680+0000 7f9f00a00640 1 --2- 172.21.15.96:0/223091871 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9efc06c950 0x7f9efc15d1a0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f9ef000b410 tx=0x7f9ef000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:11:20.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:20.681+0000 7f9ef8c00640 1 -- 172.21.15.96:0/223091871 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ef0015020 con 0x7f9efc06c950 2024-05-28T06:11:20.683 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:20.681+0000 7f9ef8c00640 1 -- 172.21.15.96:0/223091871 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ef0011850 con 0x7f9efc06c950 2024-05-28T06:11:20.683 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:20.681+0000 7f9ef8c00640 1 -- 172.21.15.96:0/223091871 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ef0011c40 con 0x7f9efc06c950 2024-05-28T06:11:20.683 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:20.681+0000 7f9f03400640 1 -- 172.21.15.96:0/223091871 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9efc0b88f0 con 0x7f9efc06c950 2024-05-28T06:11:20.683 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:20.681+0000 7f9f03400640 1 -- 172.21.15.96:0/223091871 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9efc0b8eb0 con 0x7f9efc06c950 2024-05-28T06:11:20.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:20.684+0000 7f9f03400640 1 -- 172.21.15.96:0/223091871 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9efc0b92e0 con 0x7f9efc06c950 2024-05-28T06:11:20.687 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:20.686+0000 7f9ef8c00640 1 -- 172.21.15.96:0/223091871 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 14) ==== 91601+0+0 (secure 0 0 0) 0x7f9ef0018020 con 0x7f9efc06c950 2024-05-28T06:11:20.690 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:20.688+0000 7f9ef8c00640 1 --2- 172.21.15.96:0/223091871 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9edc06cf20 0x7f9edc06f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:20.690 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:20.688+0000 7f9ef8c00640 1 -- 172.21.15.96:0/223091871 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5795+0+0 (secure 0 0 0) 0x7f9ef008c5b0 con 0x7f9efc06c950 2024-05-28T06:11:20.691 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:20.690+0000 7f9efbe00640 1 --2- 172.21.15.96:0/223091871 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9edc06cf20 0x7f9edc06f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:20.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:20.701+0000 7f9ef8c00640 1 -- 172.21.15.96:0/223091871 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9ef001f020 con 0x7f9efc06c950 2024-05-28T06:11:20.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:20.703+0000 7f9efbe00640 1 --2- 172.21.15.96:0/223091871 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9edc06cf20 0x7f9edc06f3e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f9efc0cb6b0 tx=0x7f9ee8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:11:21.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:21.166+0000 7f9f03400640 1 -- 172.21.15.96:0/223091871 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f9efc0d1c70 con 0x7f9efc06c950 2024-05-28T06:11:21.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:21.168+0000 7f9ef8c00640 1 -- 172.21.15.96:0/223091871 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+2102 (secure 0 0 0) 0x7f9ef00599f0 con 0x7f9efc06c950 2024-05-28T06:11:21.169 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:11:21.170 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:11:21.170 INFO:teuthology.orchestra.run.smithi096.stdout: "fsid": "e4a831b5-e6d1-47a7-96cb-bf85154ee0d2", 2024-05-28T06:11:21.170 INFO:teuthology.orchestra.run.smithi096.stdout: "health": { 2024-05-28T06:11:21.170 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "HEALTH_OK", 2024-05-28T06:11:21.170 INFO:teuthology.orchestra.run.smithi096.stdout: "checks": {}, 2024-05-28T06:11:21.170 INFO:teuthology.orchestra.run.smithi096.stdout: "mutes": [] 2024-05-28T06:11:21.170 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:11:21.170 INFO:teuthology.orchestra.run.smithi096.stdout: "election_epoch": 8, 2024-05-28T06:11:21.170 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum": [ 2024-05-28T06:11:21.170 INFO:teuthology.orchestra.run.smithi096.stdout: 0, 2024-05-28T06:11:21.170 INFO:teuthology.orchestra.run.smithi096.stdout: 1, 2024-05-28T06:11:21.170 INFO:teuthology.orchestra.run.smithi096.stdout: 2 2024-05-28T06:11:21.170 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:11:21.171 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_names": [ 2024-05-28T06:11:21.171 INFO:teuthology.orchestra.run.smithi096.stdout: "a", 2024-05-28T06:11:21.171 INFO:teuthology.orchestra.run.smithi096.stdout: "b", 2024-05-28T06:11:21.171 INFO:teuthology.orchestra.run.smithi096.stdout: "c" 2024-05-28T06:11:21.171 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:11:21.171 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_age": 454, 2024-05-28T06:11:21.171 INFO:teuthology.orchestra.run.smithi096.stdout: "monmap": { 2024-05-28T06:11:21.171 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 1, 2024-05-28T06:11:21.171 INFO:teuthology.orchestra.run.smithi096.stdout: "min_mon_release_name": "squid", 2024-05-28T06:11:21.171 INFO:teuthology.orchestra.run.smithi096.stdout: "num_mons": 3 2024-05-28T06:11:21.171 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:11:21.171 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap": { 2024-05-28T06:11:21.171 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 47, 2024-05-28T06:11:21.171 INFO:teuthology.orchestra.run.smithi096.stdout: "num_osds": 8, 2024-05-28T06:11:21.172 INFO:teuthology.orchestra.run.smithi096.stdout: "num_up_osds": 8, 2024-05-28T06:11:21.172 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_up_since": 1716876238, 2024-05-28T06:11:21.172 INFO:teuthology.orchestra.run.smithi096.stdout: "num_in_osds": 8, 2024-05-28T06:11:21.172 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_in_since": 1716876229, 2024-05-28T06:11:21.172 INFO:teuthology.orchestra.run.smithi096.stdout: "num_remapped_pgs": 0 2024-05-28T06:11:21.172 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:11:21.172 INFO:teuthology.orchestra.run.smithi096.stdout: "pgmap": { 2024-05-28T06:11:21.172 INFO:teuthology.orchestra.run.smithi096.stdout: "pgs_by_state": [ 2024-05-28T06:11:21.172 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:11:21.172 INFO:teuthology.orchestra.run.smithi096.stdout: "state_name": "active+clean", 2024-05-28T06:11:21.172 INFO:teuthology.orchestra.run.smithi096.stdout: "count": 97 2024-05-28T06:11:21.172 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:11:21.172 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:11:21.172 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pgs": 97, 2024-05-28T06:11:21.173 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pools": 3, 2024-05-28T06:11:21.173 INFO:teuthology.orchestra.run.smithi096.stdout: "num_objects": 2744, 2024-05-28T06:11:21.173 INFO:teuthology.orchestra.run.smithi096.stdout: "data_bytes": 11397371036, 2024-05-28T06:11:21.173 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_used": 22837448704, 2024-05-28T06:11:21.173 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_avail": 750256664576, 2024-05-28T06:11:21.173 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_total": 773094113280, 2024-05-28T06:11:21.173 INFO:teuthology.orchestra.run.smithi096.stdout: "read_bytes_sec": 67386450, 2024-05-28T06:11:21.173 INFO:teuthology.orchestra.run.smithi096.stdout: "write_bytes_sec": 100231352, 2024-05-28T06:11:21.173 INFO:teuthology.orchestra.run.smithi096.stdout: "read_op_per_sec": 16, 2024-05-28T06:11:21.173 INFO:teuthology.orchestra.run.smithi096.stdout: "write_op_per_sec": 30 2024-05-28T06:11:21.173 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:11:21.173 INFO:teuthology.orchestra.run.smithi096.stdout: "fsmap": { 2024-05-28T06:11:21.173 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 26, 2024-05-28T06:11:21.173 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 3, 2024-05-28T06:11:21.174 INFO:teuthology.orchestra.run.smithi096.stdout: "up": 1, 2024-05-28T06:11:21.174 INFO:teuthology.orchestra.run.smithi096.stdout: "in": 1, 2024-05-28T06:11:21.174 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 1, 2024-05-28T06:11:21.174 INFO:teuthology.orchestra.run.smithi096.stdout: "by_rank": [ 2024-05-28T06:11:21.174 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:11:21.174 INFO:teuthology.orchestra.run.smithi096.stdout: "filesystem_id": 3, 2024-05-28T06:11:21.174 INFO:teuthology.orchestra.run.smithi096.stdout: "rank": 0, 2024-05-28T06:11:21.174 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "a", 2024-05-28T06:11:21.174 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "up:active", 2024-05-28T06:11:21.174 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 5859 2024-05-28T06:11:21.174 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:11:21.174 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:11:21.174 INFO:teuthology.orchestra.run.smithi096.stdout: "up:standby": 3 2024-05-28T06:11:21.175 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:11:21.175 INFO:teuthology.orchestra.run.smithi096.stdout: "mgrmap": { 2024-05-28T06:11:21.175 INFO:teuthology.orchestra.run.smithi096.stdout: "available": true, 2024-05-28T06:11:21.175 INFO:teuthology.orchestra.run.smithi096.stdout: "num_standbys": 1, 2024-05-28T06:11:21.175 INFO:teuthology.orchestra.run.smithi096.stdout: "modules": [ 2024-05-28T06:11:21.175 INFO:teuthology.orchestra.run.smithi096.stdout: "iostat", 2024-05-28T06:11:21.175 INFO:teuthology.orchestra.run.smithi096.stdout: "nfs", 2024-05-28T06:11:21.175 INFO:teuthology.orchestra.run.smithi096.stdout: "restful" 2024-05-28T06:11:21.175 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:11:21.175 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:11:21.175 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:11:21.175 INFO:teuthology.orchestra.run.smithi096.stdout: "servicemap": { 2024-05-28T06:11:21.175 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 9, 2024-05-28T06:11:21.175 INFO:teuthology.orchestra.run.smithi096.stdout: "modified": "2024-05-28T06:10:01.618312+0000", 2024-05-28T06:11:21.176 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:11:21.176 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:11:21.176 INFO:teuthology.orchestra.run.smithi096.stdout: "progress_events": { 2024-05-28T06:11:21.176 INFO:teuthology.orchestra.run.smithi096.stdout: "72116a8d-650b-4cfc-99af-ee8db5167322": { 2024-05-28T06:11:21.176 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "2 ongoing clones; Avg progress is 0.0% (1s)\n [............................] ", 2024-05-28T06:11:21.176 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": 0, 2024-05-28T06:11:21.176 INFO:teuthology.orchestra.run.smithi096.stdout: "add_to_ceph_s": true 2024-05-28T06:11:21.176 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:11:21.176 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:11:21.176 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:11:21.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:21.173+0000 7f9f03400640 1 -- 172.21.15.96:0/223091871 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9edc06cf20 msgr2=0x7f9edc06f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:21.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:21.173+0000 7f9f03400640 1 --2- 172.21.15.96:0/223091871 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9edc06cf20 0x7f9edc06f3e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f9efc0cb6b0 tx=0x7f9ee8009040 comp rx=0 tx=0).stop 2024-05-28T06:11:21.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:21.173+0000 7f9f03400640 1 -- 172.21.15.96:0/223091871 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9efc06c950 msgr2=0x7f9efc15d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:21.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:21.173+0000 7f9f03400640 1 --2- 172.21.15.96:0/223091871 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9efc06c950 0x7f9efc15d1a0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f9ef000b410 tx=0x7f9ef000b8d0 comp rx=0 tx=0).stop 2024-05-28T06:11:21.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:21.173+0000 7f9f03400640 1 -- 172.21.15.96:0/223091871 shutdown_connections 2024-05-28T06:11:21.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:21.173+0000 7f9f03400640 1 --2- 172.21.15.96:0/223091871 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9edc06cf20 0x7f9edc06f3e0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:21.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:21.173+0000 7f9f03400640 1 --2- 172.21.15.96:0/223091871 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9efc15ea40 0x7f9efc0b7250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:21.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:21.173+0000 7f9f03400640 1 --2- 172.21.15.96:0/223091871 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9efc0d1040 0x7f9efc0b96b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:21.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:21.173+0000 7f9f03400640 1 --2- 172.21.15.96:0/223091871 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9efc06c950 0x7f9efc15d1a0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:21.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:21.173+0000 7f9f03400640 1 -- 172.21.15.96:0/223091871 >> 172.21.15.96:0/223091871 conn(0x7f9efc08bfc0 msgr2=0x7f9efc0b2880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:11:21.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:21.174+0000 7f9f03400640 1 -- 172.21.15.96:0/223091871 shutdown_connections 2024-05-28T06:11:21.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:21.174+0000 7f9f03400640 1 -- 172.21.15.96:0/223091871 wait complete. 2024-05-28T06:11:21.209 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone26281 2024-05-28T06:11:21.489 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:21.487+0000 7fb723e00640 1 Processor -- start 2024-05-28T06:11:21.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:21.489+0000 7fb723e00640 1 -- start start 2024-05-28T06:11:21.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:21.492+0000 7fb723e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fb71c15e9e0 0x7fb71c162dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:21.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:21.492+0000 7fb723e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fb71c0d4e00 0x7fb71c0d33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:21.495 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:21.492+0000 7fb723e00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb71c0b3990 con 0x7fb71c0cf7a0 2024-05-28T06:11:21.495 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:21.492+0000 7fb723e00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fb71c163310 con 0x7fb71c0d4e00 2024-05-28T06:11:21.495 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:21.492+0000 7fb723e00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fb71c0b3c70 con 0x7fb71c15e9e0 2024-05-28T06:11:21.495 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:21.493+0000 7fb721400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fb71c15e9e0 0x7fb71c162dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:21.495 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:21.493+0000 7fb721400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fb71c15e9e0 0x7fb71c162dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.96:59908/0 (socket says 172.21.15.96:59908) 2024-05-28T06:11:21.495 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:21.493+0000 7fb721400640 1 -- 172.21.15.96:0/977034953 learned_addr learned my addr 172.21.15.96:0/977034953 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:11:21.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:21.494+0000 7fb721400640 1 -- 172.21.15.96:0/977034953 >> v2:172.21.15.96:3300/0 conn(0x7fb71c0d4e00 msgr2=0x7fb71c0d33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:11:21.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:21.494+0000 7fb721400640 1 --2- 172.21.15.96:0/977034953 >> v2:172.21.15.96:3300/0 conn(0x7fb71c0d4e00 0x7fb71c0d33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:21.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:21.494+0000 7fb721400640 1 -- 172.21.15.96:0/977034953 >> v1:172.21.15.96:6789/0 conn(0x7fb71c0cf7a0 legacy=0x7fb71c0d46f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:11:21.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:21.494+0000 7fb721400640 1 -- 172.21.15.96:0/977034953 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb71c0b3f50 con 0x7fb71c15e9e0 2024-05-28T06:11:21.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:21.496+0000 7fb721400640 1 --2- 172.21.15.96:0/977034953 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb71c15e9e0 0x7fb71c162dd0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fb71c06dbb0 tx=0x7fb70802a1c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dfad56185b2ce282 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:11:21.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:21.501+0000 7fb71be00640 1 -- 172.21.15.96:0/977034953 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb708037070 con 0x7fb71c15e9e0 2024-05-28T06:11:21.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:21.501+0000 7fb71be00640 1 -- 172.21.15.96:0/977034953 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb708032440 con 0x7fb71c15e9e0 2024-05-28T06:11:21.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:21.501+0000 7fb71be00640 1 -- 172.21.15.96:0/977034953 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb708032810 con 0x7fb71c15e9e0 2024-05-28T06:11:21.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:21.502+0000 7fb723e00640 1 -- 172.21.15.96:0/977034953 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb71c15e9e0 msgr2=0x7fb71c162dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:21.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:21.502+0000 7fb723e00640 1 --2- 172.21.15.96:0/977034953 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb71c15e9e0 0x7fb71c162dd0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fb71c06dbb0 tx=0x7fb70802a1c0 comp rx=0 tx=0).stop 2024-05-28T06:11:21.505 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:21.503+0000 7fb723e00640 1 -- 172.21.15.96:0/977034953 shutdown_connections 2024-05-28T06:11:21.506 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:21.503+0000 7fb723e00640 1 --2- 172.21.15.96:0/977034953 >> v2:172.21.15.96:3300/0 conn(0x7fb71c0d4e00 0x7fb71c0d33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:21.506 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:21.503+0000 7fb723e00640 1 --2- 172.21.15.96:0/977034953 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb71c15e9e0 0x7fb71c162dd0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:21.506 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:21.503+0000 7fb723e00640 1 -- 172.21.15.96:0/977034953 >> 172.21.15.96:0/977034953 conn(0x7fb71c067aa0 msgr2=0x7fb71c067eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:11:21.506 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:21.503+0000 7fb723e00640 1 -- 172.21.15.96:0/977034953 shutdown_connections 2024-05-28T06:11:21.506 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:21.503+0000 7fb723e00640 1 -- 172.21.15.96:0/977034953 wait complete. 2024-05-28T06:11:21.508 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:21.507+0000 7fb723e00640 1 Processor -- start 2024-05-28T06:11:21.508 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:21.507+0000 7fb723e00640 1 -- start start 2024-05-28T06:11:21.509 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:21.508+0000 7fb723e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb71c0cf7a0 0x7fb71c1577e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:21.509 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:21.508+0000 7fb723e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb71c0d4e00 0x7fb71c15aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:21.510 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:21.508+0000 7fb723e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb71c15e9e0 0x7fb71c15b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:21.511 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:21.508+0000 7fb723e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb71c0b7890 con 0x7fb71c0cf7a0 2024-05-28T06:11:21.511 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:21.508+0000 7fb723e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fb71c0d2eb0 con 0x7fb71c15e9e0 2024-05-28T06:11:21.511 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:21.508+0000 7fb723e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fb71c0b7a10 con 0x7fb71c0d4e00 2024-05-28T06:11:21.511 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:21.509+0000 7fb721400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb71c0cf7a0 0x7fb71c1577e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:21.511 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:21.509+0000 7fb721400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb71c0cf7a0 0x7fb71c1577e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:56422/0 (socket says 172.21.15.96:56422) 2024-05-28T06:11:21.511 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:21.509+0000 7fb721400640 1 -- 172.21.15.96:0/939022071 learned_addr learned my addr 172.21.15.96:0/939022071 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:11:21.511 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:21.509+0000 7fb721e00640 1 --2- 172.21.15.96:0/939022071 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb71c15e9e0 0x7fb71c15b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:21.511 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:21.509+0000 7fb720a00640 1 --2- 172.21.15.96:0/939022071 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb71c0d4e00 0x7fb71c15aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:21.512 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:21.510+0000 7fb721400640 1 -- 172.21.15.96:0/939022071 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb71c0d4e00 msgr2=0x7fb71c15aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:21.512 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:21.510+0000 7fb721400640 1 --2- 172.21.15.96:0/939022071 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb71c0d4e00 0x7fb71c15aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:21.512 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:21.510+0000 7fb721400640 1 -- 172.21.15.96:0/939022071 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb71c15e9e0 msgr2=0x7fb71c15b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:21.512 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:21.510+0000 7fb721400640 1 --2- 172.21.15.96:0/939022071 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb71c15e9e0 0x7fb71c15b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:21.512 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:21.510+0000 7fb721400640 1 -- 172.21.15.96:0/939022071 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb71c0cdda0 con 0x7fb71c0cf7a0 2024-05-28T06:11:21.512 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:21.511+0000 7fb721400640 1 --2- 172.21.15.96:0/939022071 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb71c0cf7a0 0x7fb71c1577e0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fb7080090b0 tx=0x7fb7080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:11:21.513 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:21.512+0000 7fb719600640 1 -- 172.21.15.96:0/939022071 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb708041070 con 0x7fb71c0cf7a0 2024-05-28T06:11:21.513 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:21.512+0000 7fb719600640 1 -- 172.21.15.96:0/939022071 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb70803c870 con 0x7fb71c0cf7a0 2024-05-28T06:11:21.513 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:21.512+0000 7fb719600640 1 -- 172.21.15.96:0/939022071 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb708037040 con 0x7fb71c0cf7a0 2024-05-28T06:11:21.514 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:21.512+0000 7fb723e00640 1 -- 172.21.15.96:0/939022071 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb71c0ce080 con 0x7fb71c0cf7a0 2024-05-28T06:11:21.516 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:21.515+0000 7fb723e00640 1 -- 172.21.15.96:0/939022071 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb71c0cd9e0 con 0x7fb71c0cf7a0 2024-05-28T06:11:21.518 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:21.517+0000 7fb723e00640 1 -- 172.21.15.96:0/939022071 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb71c0689e0 con 0x7fb71c0cf7a0 2024-05-28T06:11:21.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:21.517+0000 7fb719600640 1 -- 172.21.15.96:0/939022071 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 14) ==== 91601+0+0 (secure 0 0 0) 0x7fb708044070 con 0x7fb71c0cf7a0 2024-05-28T06:11:21.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:21.518+0000 7fb719600640 1 --2- 172.21.15.96:0/939022071 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb6fc06cf70 0x7fb6fc06f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:21.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:21.519+0000 7fb719600640 1 -- 172.21.15.96:0/939022071 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5795+0+0 (secure 0 0 0) 0x7fb7080ac5b0 con 0x7fb71c0cf7a0 2024-05-28T06:11:21.530 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:21.529+0000 7fb719600640 1 -- 172.21.15.96:0/939022071 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb708079a70 con 0x7fb71c0cf7a0 2024-05-28T06:11:21.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:21.589+0000 7fb720a00640 1 --2- 172.21.15.96:0/939022071 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb6fc06cf70 0x7fb6fc06f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:21.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:21.595+0000 7fb720a00640 1 --2- 172.21.15.96:0/939022071 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb6fc06cf70 0x7fb6fc06f430 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fb71c157d20 tx=0x7fb710009060 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:11:21.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:21.907+0000 7fb723e00640 1 -- 172.21.15.96:0/939022071 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone26281", "target": ["mon-mgr", ""]}) -- 0x7fb71c0d49b0 con 0x7fb6fc06cf70 2024-05-28T06:11:21.982 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:11:21.980+0000 7fea59e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:11:21.982 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:11:21.980+0000 7fea59e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:11:21.983 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:11:21.980+0000 7fea59e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:11:21.983 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:11:21.980+0000 7fea59e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:11:21.983 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:11:21.980+0000 7fea59e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:11:23.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:23.616+0000 7fb719600640 1 -- 172.21.15.96:0/939022071 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fb71c0d49b0 con 0x7fb6fc06cf70 2024-05-28T06:11:23.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:23.624+0000 7fb723e00640 1 -- 172.21.15.96:0/939022071 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb6fc06cf70 msgr2=0x7fb6fc06f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:23.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:23.624+0000 7fb723e00640 1 --2- 172.21.15.96:0/939022071 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb6fc06cf70 0x7fb6fc06f430 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fb71c157d20 tx=0x7fb710009060 comp rx=0 tx=0).stop 2024-05-28T06:11:23.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:23.624+0000 7fb723e00640 1 -- 172.21.15.96:0/939022071 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb71c0cf7a0 msgr2=0x7fb71c1577e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:23.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:23.624+0000 7fb723e00640 1 --2- 172.21.15.96:0/939022071 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb71c0cf7a0 0x7fb71c1577e0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fb7080090b0 tx=0x7fb7080046c0 comp rx=0 tx=0).stop 2024-05-28T06:11:23.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:23.625+0000 7fb723e00640 1 -- 172.21.15.96:0/939022071 shutdown_connections 2024-05-28T06:11:23.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:23.625+0000 7fb723e00640 1 --2- 172.21.15.96:0/939022071 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb6fc06cf70 0x7fb6fc06f430 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:23.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:23.625+0000 7fb723e00640 1 --2- 172.21.15.96:0/939022071 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb71c15e9e0 0x7fb71c15b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:23.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:23.625+0000 7fb723e00640 1 --2- 172.21.15.96:0/939022071 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb71c0d4e00 0x7fb71c15aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:23.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:23.625+0000 7fb723e00640 1 --2- 172.21.15.96:0/939022071 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb71c0cf7a0 0x7fb71c1577e0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:23.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:23.625+0000 7fb723e00640 1 -- 172.21.15.96:0/939022071 >> 172.21.15.96:0/939022071 conn(0x7fb71c067aa0 msgr2=0x7fb71c0b9740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:11:23.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:23.625+0000 7fb723e00640 1 -- 172.21.15.96:0/939022071 shutdown_connections 2024-05-28T06:11:23.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:23.626+0000 7fb723e00640 1 -- 172.21.15.96:0/939022071 wait complete. 2024-05-28T06:11:23.664 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone26282 2024-05-28T06:11:23.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:23.944+0000 7ffa09000640 1 Processor -- start 2024-05-28T06:11:23.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:23.945+0000 7ffa09000640 1 -- start start 2024-05-28T06:11:23.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:23.946+0000 7ffa09000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7ffa040b45d0 0x7ffa040b4a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:23.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:23.947+0000 7ffa09000640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa0406e2e0 con 0x7ffa0415abe0 2024-05-28T06:11:23.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:23.947+0000 7ffa09000640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa0406e5c0 con 0x7ffa040725c0 2024-05-28T06:11:23.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:23.947+0000 7ffa09000640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7ffa0406e8a0 con 0x7ffa040b45d0 2024-05-28T06:11:23.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:23.948+0000 7ffa02800640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7ffa040725c0 0x7ffa040ad680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.96:41272/0 (socket says 172.21.15.96:41272) 2024-05-28T06:11:23.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:23.948+0000 7ffa02800640 1 -- 172.21.15.96:0/1937157275 learned_addr learned my addr 172.21.15.96:0/1937157275 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:11:23.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:23.948+0000 7ff9fbe00640 1 --2- 172.21.15.96:0/1937157275 >> v2:172.21.15.123:3300/0 conn(0x7ffa040b45d0 0x7ffa040b4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:23.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:23.949+0000 7ff9fbe00640 1 -- 172.21.15.96:0/1937157275 >> v1:172.21.15.123:6789/0 conn(0x7ffa040725c0 legacy=0x7ffa040ad680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:23.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:23.950+0000 7ff9fbe00640 1 -- 172.21.15.96:0/1937157275 >> v1:172.21.15.96:6789/0 conn(0x7ffa0415abe0 legacy=0x7ffa0406f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:23.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:23.950+0000 7ff9fbe00640 1 -- 172.21.15.96:0/1937157275 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa0406eb80 con 0x7ffa040b45d0 2024-05-28T06:11:23.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:23.950+0000 7ffa01400640 1 -- 172.21.15.96:0/1937157275 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1129741057 0 0) 0x7ffa0406e2e0 con 0x7ffa0415abe0 2024-05-28T06:11:23.956 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:23.955+0000 7ff9fbe00640 1 --2- 172.21.15.96:0/1937157275 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa040b45d0 0x7ffa040b4a30 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7ff9ec00a8c0 tx=0x7ff9ec02b350 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9c5a612df1dd0d38 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:11:23.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:23.968+0000 7ffa01400640 1 -- 172.21.15.96:0/1937157275 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9ec038070 con 0x7ffa040b45d0 2024-05-28T06:11:23.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:23.968+0000 7ffa01400640 1 -- 172.21.15.96:0/1937157275 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9ec033640 con 0x7ffa040b45d0 2024-05-28T06:11:23.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:23.968+0000 7ffa01400640 1 -- 172.21.15.96:0/1937157275 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9ec033a10 con 0x7ffa040b45d0 2024-05-28T06:11:23.970 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:23.969+0000 7ffa09000640 1 -- 172.21.15.96:0/1937157275 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa040b45d0 msgr2=0x7ffa040b4a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:23.970 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:23.969+0000 7ffa09000640 1 --2- 172.21.15.96:0/1937157275 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa040b45d0 0x7ffa040b4a30 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7ff9ec00a8c0 tx=0x7ff9ec02b350 comp rx=0 tx=0).stop 2024-05-28T06:11:23.970 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:23.969+0000 7ffa09000640 1 -- 172.21.15.96:0/1937157275 shutdown_connections 2024-05-28T06:11:23.970 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:23.969+0000 7ffa09000640 1 --2- 172.21.15.96:0/1937157275 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa040b45d0 0x7ffa040b4a30 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:23.970 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:23.969+0000 7ffa09000640 1 -- 172.21.15.96:0/1937157275 >> 172.21.15.96:0/1937157275 conn(0x7ffa0406bd20 msgr2=0x7ffa0406c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:11:23.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:23.970+0000 7ffa09000640 1 -- 172.21.15.96:0/1937157275 shutdown_connections 2024-05-28T06:11:23.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:23.970+0000 7ffa09000640 1 -- 172.21.15.96:0/1937157275 wait complete. 2024-05-28T06:11:23.973 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:23.972+0000 7ffa09000640 1 Processor -- start 2024-05-28T06:11:23.974 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:23.972+0000 7ffa09000640 1 -- start start 2024-05-28T06:11:23.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:23.973+0000 7ffa09000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffa040725c0 0x7ffa041fbad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:23.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:23.973+0000 7ffa09000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa040b45d0 0x7ffa0414d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:23.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:23.974+0000 7ffa09000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa0415abe0 0x7ffa0414d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:23.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:23.974+0000 7ffa09000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ffa0406fa90 con 0x7ffa0415abe0 2024-05-28T06:11:23.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:23.974+0000 7ffa09000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ffa0415afc0 con 0x7ffa040b45d0 2024-05-28T06:11:23.976 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:23.974+0000 7ffa09000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ffa0406df60 con 0x7ffa040725c0 2024-05-28T06:11:23.976 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:23.974+0000 7ffa02800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa0415abe0 0x7ffa0414d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:23.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:23.974+0000 7ffa02800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa0415abe0 0x7ffa0414d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:56426/0 (socket says 172.21.15.96:56426) 2024-05-28T06:11:23.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:23.974+0000 7ffa02800640 1 -- 172.21.15.96:0/3547667942 learned_addr learned my addr 172.21.15.96:0/3547667942 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:11:23.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:23.975+0000 7ff9fbe00640 1 --2- 172.21.15.96:0/3547667942 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa040b45d0 0x7ffa0414d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:23.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:23.975+0000 7ffa02800640 1 -- 172.21.15.96:0/3547667942 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffa040725c0 msgr2=0x7ffa041fbad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:23.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:23.975+0000 7ffa02800640 1 --2- 172.21.15.96:0/3547667942 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffa040725c0 0x7ffa041fbad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:23.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:23.975+0000 7ffa02800640 1 -- 172.21.15.96:0/3547667942 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa040b45d0 msgr2=0x7ffa0414d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:23.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:23.975+0000 7ffa02800640 1 --2- 172.21.15.96:0/3547667942 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa040b45d0 0x7ffa0414d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:23.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:23.975+0000 7ffa02800640 1 -- 172.21.15.96:0/3547667942 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa0409dd60 con 0x7ffa0415abe0 2024-05-28T06:11:23.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:23.975+0000 7ffa02800640 1 --2- 172.21.15.96:0/3547667942 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa0415abe0 0x7ffa0414d930 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7ff9fc00cbc0 tx=0x7ff9fc00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:11:23.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:23.976+0000 7ff9fa000640 1 -- 172.21.15.96:0/3547667942 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9fc017070 con 0x7ffa0415abe0 2024-05-28T06:11:23.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:23.976+0000 7ff9fa000640 1 -- 172.21.15.96:0/3547667942 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9fc00ad40 con 0x7ffa0415abe0 2024-05-28T06:11:23.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:23.976+0000 7ff9fa000640 1 -- 172.21.15.96:0/3547667942 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9fc012600 con 0x7ffa0415abe0 2024-05-28T06:11:23.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:23.977+0000 7ffa09000640 1 -- 172.21.15.96:0/3547667942 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa0409e020 con 0x7ffa0415abe0 2024-05-28T06:11:23.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:23.977+0000 7ffa09000640 1 -- 172.21.15.96:0/3547667942 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa041504a0 con 0x7ffa0415abe0 2024-05-28T06:11:23.982 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:23.979+0000 7ffa09000640 1 -- 172.21.15.96:0/3547667942 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa0406c330 con 0x7ffa0415abe0 2024-05-28T06:11:23.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:23.981+0000 7ff9fa000640 1 -- 172.21.15.96:0/3547667942 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 15) ==== 91652+0+0 (secure 0 0 0) 0x7ff9fc01a070 con 0x7ffa0415abe0 2024-05-28T06:11:23.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:23.981+0000 7ff9fa000640 1 --2- 172.21.15.96:0/3547667942 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff9e006d030 0x7ff9e006f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:23.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:23.982+0000 7ff9fa000640 1 -- 172.21.15.96:0/3547667942 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5795+0+0 (secure 0 0 0) 0x7ff9fc08c330 con 0x7ffa0415abe0 2024-05-28T06:11:23.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:23.989+0000 7ff9fa000640 1 -- 172.21.15.96:0/3547667942 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ffa0406c330 con 0x7ffa0415abe0 2024-05-28T06:11:24.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:24.006+0000 7ffa01e00640 1 --2- 172.21.15.96:0/3547667942 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff9e006d030 0x7ff9e006f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:24.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:24.008+0000 7ffa01e00640 1 --2- 172.21.15.96:0/3547667942 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff9e006d030 0x7ff9e006f4f0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7ff9f4001510 tx=0x7ff9f40079b0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:11:24.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:24.392+0000 7ffa09000640 1 -- 172.21.15.96:0/3547667942 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone26282", "target": ["mon-mgr", ""]}) -- 0x7ffa0415ed90 con 0x7ff9e006d030 2024-05-28T06:11:25.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:25.127+0000 7ff9fa000640 1 -- 172.21.15.96:0/3547667942 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7ffa0415ed90 con 0x7ff9e006d030 2024-05-28T06:11:25.133 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:25.131+0000 7ffa09000640 1 -- 172.21.15.96:0/3547667942 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff9e006d030 msgr2=0x7ff9e006f4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:25.133 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:25.131+0000 7ffa09000640 1 --2- 172.21.15.96:0/3547667942 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff9e006d030 0x7ff9e006f4f0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7ff9f4001510 tx=0x7ff9f40079b0 comp rx=0 tx=0).stop 2024-05-28T06:11:25.133 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:25.131+0000 7ffa09000640 1 -- 172.21.15.96:0/3547667942 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa0415abe0 msgr2=0x7ffa0414d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:25.133 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:25.131+0000 7ffa09000640 1 --2- 172.21.15.96:0/3547667942 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa0415abe0 0x7ffa0414d930 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7ff9fc00cbc0 tx=0x7ff9fc00a430 comp rx=0 tx=0).stop 2024-05-28T06:11:25.133 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:25.132+0000 7ffa09000640 1 -- 172.21.15.96:0/3547667942 shutdown_connections 2024-05-28T06:11:25.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:25.132+0000 7ffa09000640 1 --2- 172.21.15.96:0/3547667942 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff9e006d030 0x7ff9e006f4f0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:25.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:25.132+0000 7ffa09000640 1 --2- 172.21.15.96:0/3547667942 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa0415abe0 0x7ffa0414d930 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:25.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:25.132+0000 7ffa09000640 1 --2- 172.21.15.96:0/3547667942 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa040b45d0 0x7ffa0414d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:25.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:25.132+0000 7ffa09000640 1 --2- 172.21.15.96:0/3547667942 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffa040725c0 0x7ffa041fbad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:25.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:25.132+0000 7ffa09000640 1 -- 172.21.15.96:0/3547667942 >> 172.21.15.96:0/3547667942 conn(0x7ffa0406bd20 msgr2=0x7ffa0415efd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:11:25.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:25.132+0000 7ffa09000640 1 -- 172.21.15.96:0/3547667942 shutdown_connections 2024-05-28T06:11:25.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:25.132+0000 7ffa09000640 1 -- 172.21.15.96:0/3547667942 wait complete. 2024-05-28T06:11:25.157 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone26283 2024-05-28T06:11:25.417 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:25.414+0000 7f30ab200640 1 Processor -- start 2024-05-28T06:11:25.418 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:25.414+0000 7f30ab200640 1 -- start start 2024-05-28T06:11:25.418 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:25.415+0000 7f30ab200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f30a40d0640 0x7f30a415d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:25.418 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:25.415+0000 7f30ab200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f30a40d0a20 0x7f30a40cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:25.418 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:25.415+0000 7f30ab200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f30a415cb10 con 0x7f30a40b32c0 2024-05-28T06:11:25.418 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:25.416+0000 7f30ab200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f30a40adbb0 con 0x7f30a40d0640 2024-05-28T06:11:25.418 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:25.416+0000 7f30ab200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f30a415db10 con 0x7f30a40d0a20 2024-05-28T06:11:25.418 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:25.416+0000 7f30a3e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f30a40d0640 0x7f30a415d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:25.418 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:25.416+0000 7f30a9200640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f30a40b32c0 0x7f30a40b3730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:46960/0 (socket says 172.21.15.96:46960) 2024-05-28T06:11:25.418 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:25.416+0000 7f30a9200640 1 -- 172.21.15.96:0/513091291 learned_addr learned my addr 172.21.15.96:0/513091291 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:11:25.419 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:25.417+0000 7f30a3400640 1 --2- 172.21.15.96:0/513091291 >> v2:172.21.15.123:3300/0 conn(0x7f30a40d0a20 0x7f30a40cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:25.419 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:25.417+0000 7f30a3e00640 1 -- 172.21.15.96:0/513091291 >> v2:172.21.15.123:3300/0 conn(0x7f30a40d0a20 msgr2=0x7f30a40cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:11:25.419 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:25.417+0000 7f30a3e00640 1 --2- 172.21.15.96:0/513091291 >> v2:172.21.15.123:3300/0 conn(0x7f30a40d0a20 0x7f30a40cf950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:25.419 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:25.417+0000 7f30a3e00640 1 -- 172.21.15.96:0/513091291 >> v1:172.21.15.96:6789/0 conn(0x7f30a40b32c0 legacy=0x7f30a40b3730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:25.419 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:25.417+0000 7f30a3e00640 1 -- 172.21.15.96:0/513091291 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30a40ad5f0 con 0x7f30a40d0640 2024-05-28T06:11:25.419 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:25.417+0000 7f30a2a00640 1 -- 172.21.15.96:0/513091291 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1796031075 0 0) 0x7f30a415cb10 con 0x7f30a40b32c0 2024-05-28T06:11:25.420 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:25.417+0000 7f30a3400640 1 --2- 172.21.15.96:0/513091291 >> v2:172.21.15.123:3300/0 conn(0x7f30a40d0a20 0x7f30a40cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:11:25.420 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:25.418+0000 7f30a3e00640 1 --2- 172.21.15.96:0/513091291 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f30a40d0640 0x7f30a415d5d0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f30a4074120 tx=0x7f309002a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=798c50ed6ffab213 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:11:25.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:25.418+0000 7f30a2a00640 1 -- 172.21.15.96:0/513091291 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3090037070 con 0x7f30a40d0640 2024-05-28T06:11:25.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:25.418+0000 7f30a2a00640 1 -- 172.21.15.96:0/513091291 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f309002ae50 con 0x7f30a40d0640 2024-05-28T06:11:25.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:25.418+0000 7f30a2a00640 1 -- 172.21.15.96:0/513091291 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3090032660 con 0x7f30a40d0640 2024-05-28T06:11:25.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:25.419+0000 7f30ab200640 1 -- 172.21.15.96:0/513091291 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f30a40d0640 msgr2=0x7f30a415d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:25.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:25.419+0000 7f30ab200640 1 --2- 172.21.15.96:0/513091291 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f30a40d0640 0x7f30a415d5d0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f30a4074120 tx=0x7f309002a140 comp rx=0 tx=0).stop 2024-05-28T06:11:25.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:25.419+0000 7f30ab200640 1 -- 172.21.15.96:0/513091291 shutdown_connections 2024-05-28T06:11:25.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:25.419+0000 7f30ab200640 1 --2- 172.21.15.96:0/513091291 >> v2:172.21.15.123:3300/0 conn(0x7f30a40d0a20 0x7f30a40cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:25.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:25.419+0000 7f30ab200640 1 --2- 172.21.15.96:0/513091291 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f30a40d0640 0x7f30a415d5d0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:25.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:25.419+0000 7f30ab200640 1 -- 172.21.15.96:0/513091291 >> 172.21.15.96:0/513091291 conn(0x7f30a406bd20 msgr2=0x7f30a406c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:11:25.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:25.419+0000 7f30ab200640 1 -- 172.21.15.96:0/513091291 shutdown_connections 2024-05-28T06:11:25.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:25.419+0000 7f30ab200640 1 -- 172.21.15.96:0/513091291 wait complete. 2024-05-28T06:11:25.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:25.421+0000 7f30ab200640 1 Processor -- start 2024-05-28T06:11:25.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:25.421+0000 7f30ab200640 1 -- start start 2024-05-28T06:11:25.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:25.422+0000 7f30ab200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f30a40b32c0 0x7f30a409c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:25.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:25.422+0000 7f30ab200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f30a40d0640 0x7f30a409b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:25.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:25.422+0000 7f30ab200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f30a40d0a20 0x7f30a40aff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:25.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:25.422+0000 7f30ab200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f30a415d0b0 con 0x7f30a40b32c0 2024-05-28T06:11:25.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:25.422+0000 7f30ab200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f30a415cdd0 con 0x7f30a40d0640 2024-05-28T06:11:25.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:25.422+0000 7f30ab200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f30a40b3e40 con 0x7f30a40d0a20 2024-05-28T06:11:25.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:25.422+0000 7f30a3e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f30a40b32c0 0x7f30a409c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:25.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:25.423+0000 7f30a3e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f30a40b32c0 0x7f30a409c0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:56446/0 (socket says 172.21.15.96:56446) 2024-05-28T06:11:25.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:25.423+0000 7f30a3e00640 1 -- 172.21.15.96:0/920521800 learned_addr learned my addr 172.21.15.96:0/920521800 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:11:25.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:25.423+0000 7f30a3e00640 1 -- 172.21.15.96:0/920521800 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f30a40d0a20 msgr2=0x7f30a40aff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:25.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:25.423+0000 7f30a3e00640 1 --2- 172.21.15.96:0/920521800 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f30a40d0a20 0x7f30a40aff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:25.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:25.423+0000 7f30a3e00640 1 -- 172.21.15.96:0/920521800 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f30a40d0640 msgr2=0x7f30a409b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:25.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:25.423+0000 7f30a3e00640 1 --2- 172.21.15.96:0/920521800 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f30a40d0640 0x7f30a409b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:25.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:25.423+0000 7f30a3e00640 1 -- 172.21.15.96:0/920521800 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30a409a550 con 0x7f30a40b32c0 2024-05-28T06:11:25.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:25.423+0000 7f30a3e00640 1 --2- 172.21.15.96:0/920521800 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f30a40b32c0 0x7f30a409c0c0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f3090035040 tx=0x7f30900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:11:25.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:25.424+0000 7f309be00640 1 -- 172.21.15.96:0/920521800 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3090041070 con 0x7f30a40b32c0 2024-05-28T06:11:25.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:25.424+0000 7f309be00640 1 -- 172.21.15.96:0/920521800 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f309003c670 con 0x7f30a40b32c0 2024-05-28T06:11:25.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:25.424+0000 7f30ab200640 1 -- 172.21.15.96:0/920521800 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30a409a830 con 0x7f30a40b32c0 2024-05-28T06:11:25.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:25.424+0000 7f309be00640 1 -- 172.21.15.96:0/920521800 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3090037040 con 0x7f30a40b32c0 2024-05-28T06:11:25.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:25.424+0000 7f30ab200640 1 -- 172.21.15.96:0/920521800 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30a409c600 con 0x7f30a40b32c0 2024-05-28T06:11:25.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:25.425+0000 7f30ab200640 1 -- 172.21.15.96:0/920521800 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30a40689e0 con 0x7f30a40b32c0 2024-05-28T06:11:25.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:25.426+0000 7f309be00640 1 -- 172.21.15.96:0/920521800 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 15) ==== 91652+0+0 (secure 0 0 0) 0x7f309003a070 con 0x7f30a40b32c0 2024-05-28T06:11:25.430 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:25.428+0000 7f309be00640 1 --2- 172.21.15.96:0/920521800 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f308406cfe0 0x7f308406f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:25.430 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:25.428+0000 7f309be00640 1 -- 172.21.15.96:0/920521800 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5795+0+0 (secure 0 0 0) 0x7f30900ac960 con 0x7f30a40b32c0 2024-05-28T06:11:25.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:25.429+0000 7f30a3400640 1 --2- 172.21.15.96:0/920521800 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f308406cfe0 0x7f308406f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:25.433 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:25.431+0000 7f30a3400640 1 --2- 172.21.15.96:0/920521800 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f308406cfe0 0x7f308406f4a0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f3094006fd0 tx=0x7f3094008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:11:25.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:25.436+0000 7f309be00640 1 -- 172.21.15.96:0/920521800 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f309007a460 con 0x7f30a40b32c0 2024-05-28T06:11:25.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:25.813+0000 7f30ab200640 1 -- 172.21.15.96:0/920521800 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone26283", "target": ["mon-mgr", ""]}) -- 0x7f30a4071ef0 con 0x7f308406cfe0 2024-05-28T06:11:26.297 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.295+0000 7f309be00640 1 -- 172.21.15.96:0/920521800 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f30a4071ef0 con 0x7f308406cfe0 2024-05-28T06:11:26.302 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.300+0000 7f30ab200640 1 -- 172.21.15.96:0/920521800 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f308406cfe0 msgr2=0x7f308406f4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:26.302 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.300+0000 7f30ab200640 1 --2- 172.21.15.96:0/920521800 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f308406cfe0 0x7f308406f4a0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f3094006fd0 tx=0x7f3094008040 comp rx=0 tx=0).stop 2024-05-28T06:11:26.302 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.300+0000 7f30ab200640 1 -- 172.21.15.96:0/920521800 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f30a40b32c0 msgr2=0x7f30a409c0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:26.302 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.300+0000 7f30ab200640 1 --2- 172.21.15.96:0/920521800 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f30a40b32c0 0x7f30a409c0c0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f3090035040 tx=0x7f30900046c0 comp rx=0 tx=0).stop 2024-05-28T06:11:26.303 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.301+0000 7f30ab200640 1 -- 172.21.15.96:0/920521800 shutdown_connections 2024-05-28T06:11:26.303 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.301+0000 7f30ab200640 1 --2- 172.21.15.96:0/920521800 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f308406cfe0 0x7f308406f4a0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:26.303 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.301+0000 7f30ab200640 1 --2- 172.21.15.96:0/920521800 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f30a40d0a20 0x7f30a40aff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:26.303 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.301+0000 7f30ab200640 1 --2- 172.21.15.96:0/920521800 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f30a40d0640 0x7f30a409b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:26.303 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.301+0000 7f30ab200640 1 --2- 172.21.15.96:0/920521800 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f30a40b32c0 0x7f30a409c0c0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:26.303 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.301+0000 7f30ab200640 1 -- 172.21.15.96:0/920521800 >> 172.21.15.96:0/920521800 conn(0x7f30a406bd20 msgr2=0x7f30a406c100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:11:26.303 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.301+0000 7f30ab200640 1 -- 172.21.15.96:0/920521800 shutdown_connections 2024-05-28T06:11:26.305 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.303+0000 7f30ab200640 1 -- 172.21.15.96:0/920521800 wait complete. 2024-05-28T06:11:26.335 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone26284 2024-05-28T06:11:26.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.548+0000 7f38e5600640 1 Processor -- start 2024-05-28T06:11:26.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.549+0000 7f38e5600640 1 -- start start 2024-05-28T06:11:26.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.552+0000 7f38e5600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f38e00caa40 0x7f38e00ce960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:26.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.552+0000 7f38e5600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f38e01524c0 0x7f38e00cf710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:26.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.552+0000 7f38e5600640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f38e00b3e80 con 0x7f38e00cae20 2024-05-28T06:11:26.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.552+0000 7f38e5600640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f38e00d0b30 con 0x7f38e01524c0 2024-05-28T06:11:26.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.552+0000 7f38e5600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f38e00b4160 con 0x7f38e00caa40 2024-05-28T06:11:26.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.552+0000 7f38d7e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f38e00cae20 0x7f38e0151db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:46972/0 (socket says 172.21.15.96:46972) 2024-05-28T06:11:26.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.552+0000 7f38d7e00640 1 -- 172.21.15.96:0/477273109 learned_addr learned my addr 172.21.15.96:0/477273109 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:11:26.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.552+0000 7f38e4c00640 1 --2- 172.21.15.96:0/477273109 >> v2:172.21.15.123:3300/0 conn(0x7f38e01524c0 0x7f38e00cf710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:26.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.552+0000 7f38dfe00640 1 --2- 172.21.15.96:0/477273109 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f38e00caa40 0x7f38e00ce960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:26.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.552+0000 7f38df400640 1 -- 172.21.15.96:0/477273109 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1194659774 0 0) 0x7f38e00b3e80 con 0x7f38e00cae20 2024-05-28T06:11:26.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.553+0000 7f38df400640 1 -- 172.21.15.96:0/477273109 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f38c0005730 con 0x7f38e00cae20 2024-05-28T06:11:26.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.553+0000 7f38df400640 1 -- 172.21.15.96:0/477273109 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1978787890 0 0) 0x7f38c0005730 con 0x7f38e00cae20 2024-05-28T06:11:26.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.553+0000 7f38df400640 1 -- 172.21.15.96:0/477273109 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f38e00caa40 msgr2=0x7f38e00ce960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:11:26.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.553+0000 7f38df400640 1 --2- 172.21.15.96:0/477273109 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f38e00caa40 0x7f38e00ce960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:26.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.553+0000 7f38df400640 1 -- 172.21.15.96:0/477273109 >> v2:172.21.15.123:3300/0 conn(0x7f38e01524c0 msgr2=0x7f38e00cf710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:11:26.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.553+0000 7f38df400640 1 --2- 172.21.15.96:0/477273109 >> v2:172.21.15.123:3300/0 conn(0x7f38e01524c0 0x7f38e00cf710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:26.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.553+0000 7f38df400640 1 -- 172.21.15.96:0/477273109 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38e00cde80 con 0x7f38e00cae20 2024-05-28T06:11:26.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.553+0000 7f38df400640 1 -- 172.21.15.96:0/477273109 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f38c8001670 con 0x7f38e00cae20 2024-05-28T06:11:26.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.553+0000 7f38e4c00640 1 --2- 172.21.15.96:0/477273109 >> v2:172.21.15.123:3300/0 conn(0x7f38e01524c0 0x7f38e00cf710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-28T06:11:26.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.553+0000 7f38df400640 1 -- 172.21.15.96:0/477273109 >> v1:172.21.15.96:6789/0 conn(0x7f38e00cae20 legacy=0x7f38e0151db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:26.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.553+0000 7f38df400640 1 --2- 172.21.15.96:0/477273109 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f38e00caa40 0x7f38c0005bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:26.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.553+0000 7f38df400640 1 --2- 172.21.15.96:0/477273109 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f38c0006140 0x7f38c0006a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:26.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.553+0000 7f38df400640 1 --2- 172.21.15.96:0/477273109 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38c0006f50 0x7f38c00073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:26.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.553+0000 7f38df400640 1 -- 172.21.15.96:0/477273109 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f38e00d0b30 con 0x7f38c0006140 2024-05-28T06:11:26.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.553+0000 7f38df400640 1 -- 172.21.15.96:0/477273109 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f38e00b4160 con 0x7f38c0006f50 2024-05-28T06:11:26.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.553+0000 7f38df400640 1 -- 172.21.15.96:0/477273109 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f38c000dc20 con 0x7f38e00caa40 2024-05-28T06:11:26.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.553+0000 7f38d7e00640 1 --2- 172.21.15.96:0/477273109 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f38c0006140 0x7f38c0006a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:26.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.553+0000 7f38dfe00640 1 --2- 172.21.15.96:0/477273109 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f38e00caa40 0x7f38c0005bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:26.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.554+0000 7f38e4c00640 1 --2- 172.21.15.96:0/477273109 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38c0006f50 0x7f38c00073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:26.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.554+0000 7f38d7e00640 1 -- 172.21.15.96:0/477273109 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f38e00caa40 msgr2=0x7f38c0005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:11:26.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.554+0000 7f38d7e00640 1 --2- 172.21.15.96:0/477273109 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f38e00caa40 0x7f38c0005bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:26.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.554+0000 7f38d7e00640 1 -- 172.21.15.96:0/477273109 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38c0006f50 msgr2=0x7f38c00073d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:11:26.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.554+0000 7f38d7e00640 1 --2- 172.21.15.96:0/477273109 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38c0006f50 0x7f38c00073d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:26.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.554+0000 7f38d7e00640 1 -- 172.21.15.96:0/477273109 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38c0005730 con 0x7f38c0006140 2024-05-28T06:11:26.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.555+0000 7f38d7e00640 1 --2- 172.21.15.96:0/477273109 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f38c0006140 0x7f38c0006a10 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f38c800c270 tx=0x7f38c0007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d17867b729e71b1 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:11:26.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.555+0000 7f38df400640 1 -- 172.21.15.96:0/477273109 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38c8007d30 con 0x7f38c0006140 2024-05-28T06:11:26.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.555+0000 7f38df400640 1 -- 172.21.15.96:0/477273109 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38c8005ad0 con 0x7f38c0006140 2024-05-28T06:11:26.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.555+0000 7f38df400640 1 -- 172.21.15.96:0/477273109 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38c8007070 con 0x7f38c0006140 2024-05-28T06:11:26.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.556+0000 7f38e5600640 1 -- 172.21.15.96:0/477273109 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f38c0006140 msgr2=0x7f38c0006a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:26.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.556+0000 7f38e5600640 1 --2- 172.21.15.96:0/477273109 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f38c0006140 0x7f38c0006a10 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f38c800c270 tx=0x7f38c0007a40 comp rx=0 tx=0).stop 2024-05-28T06:11:26.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.556+0000 7f38e4c00640 1 -- 172.21.15.96:0/477273109 reap_dead start 2024-05-28T06:11:26.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.556+0000 7f38e5600640 1 -- 172.21.15.96:0/477273109 shutdown_connections 2024-05-28T06:11:26.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.556+0000 7f38e5600640 1 -- 172.21.15.96:0/477273109 >> 172.21.15.96:0/477273109 conn(0x7f38e006b900 msgr2=0x7f38e006bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:11:26.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.556+0000 7f38e5600640 1 -- 172.21.15.96:0/477273109 shutdown_connections 2024-05-28T06:11:26.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.556+0000 7f38e5600640 1 -- 172.21.15.96:0/477273109 wait complete. 2024-05-28T06:11:26.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.558+0000 7f38e5600640 1 Processor -- start 2024-05-28T06:11:26.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.558+0000 7f38e5600640 1 -- start start 2024-05-28T06:11:26.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.559+0000 7f38e5600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f38c0006140 0x7f38e01432f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:26.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.559+0000 7f38e5600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38e00b0f20 0x7f38e0156d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:26.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.559+0000 7f38e5600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f38e00b13a0 0x7f38e0141d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:26.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.559+0000 7f38e5600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f38e00cfc50 con 0x7f38c0006140 2024-05-28T06:11:26.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.559+0000 7f38e5600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f38e00ce440 con 0x7f38e00b0f20 2024-05-28T06:11:26.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.559+0000 7f38e5600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f38e00cfdd0 con 0x7f38e00b13a0 2024-05-28T06:11:26.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.559+0000 7f38dfe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f38c0006140 0x7f38e01432f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:26.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.559+0000 7f38dfe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f38c0006140 0x7f38e01432f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:56466/0 (socket says 172.21.15.96:56466) 2024-05-28T06:11:26.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.559+0000 7f38dfe00640 1 -- 172.21.15.96:0/1171162195 learned_addr learned my addr 172.21.15.96:0/1171162195 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:11:26.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.559+0000 7f38e4c00640 1 --2- 172.21.15.96:0/1171162195 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f38e00b13a0 0x7f38e0141d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:26.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.559+0000 7f38d7e00640 1 --2- 172.21.15.96:0/1171162195 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38e00b0f20 0x7f38e0156d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:26.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.560+0000 7f38dfe00640 1 -- 172.21.15.96:0/1171162195 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f38e00b13a0 msgr2=0x7f38e0141d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:26.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.560+0000 7f38dfe00640 1 --2- 172.21.15.96:0/1171162195 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f38e00b13a0 0x7f38e0141d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:26.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.560+0000 7f38dfe00640 1 -- 172.21.15.96:0/1171162195 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38e00b0f20 msgr2=0x7f38e0156d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:26.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.560+0000 7f38dfe00640 1 --2- 172.21.15.96:0/1171162195 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38e00b0f20 0x7f38e0156d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:26.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.560+0000 7f38dfe00640 1 -- 172.21.15.96:0/1171162195 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38e01f43f0 con 0x7f38c0006140 2024-05-28T06:11:26.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.560+0000 7f38e4c00640 1 --2- 172.21.15.96:0/1171162195 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f38e00b13a0 0x7f38e0141d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-28T06:11:26.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.560+0000 7f38d7e00640 1 --2- 172.21.15.96:0/1171162195 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38e00b0f20 0x7f38e0156d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:11:26.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.560+0000 7f38dfe00640 1 --2- 172.21.15.96:0/1171162195 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f38c0006140 0x7f38e01432f0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f38d0011570 tx=0x7f38d0011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:11:26.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.560+0000 7f38dcc00640 1 -- 172.21.15.96:0/1171162195 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38d0015930 con 0x7f38c0006140 2024-05-28T06:11:26.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.560+0000 7f38e5600640 1 -- 172.21.15.96:0/1171162195 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38e0143830 con 0x7f38c0006140 2024-05-28T06:11:26.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.560+0000 7f38dcc00640 1 -- 172.21.15.96:0/1171162195 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38d0015ad0 con 0x7f38c0006140 2024-05-28T06:11:26.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.560+0000 7f38dcc00640 1 -- 172.21.15.96:0/1171162195 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38d000f460 con 0x7f38c0006140 2024-05-28T06:11:26.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.560+0000 7f38e5600640 1 -- 172.21.15.96:0/1171162195 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38e0141260 con 0x7f38c0006140 2024-05-28T06:11:26.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.562+0000 7f38e5600640 1 -- 172.21.15.96:0/1171162195 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38e00689e0 con 0x7f38c0006140 2024-05-28T06:11:26.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.564+0000 7f38dcc00640 1 -- 172.21.15.96:0/1171162195 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 15) ==== 91652+0+0 (secure 0 0 0) 0x7f38d000f600 con 0x7f38c0006140 2024-05-28T06:11:26.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.565+0000 7f38dcc00640 1 --2- 172.21.15.96:0/1171162195 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f38bc06cfb0 0x7f38bc06f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:26.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.565+0000 7f38dcc00640 1 -- 172.21.15.96:0/1171162195 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5795+0+0 (secure 0 0 0) 0x7f38d008d2c0 con 0x7f38c0006140 2024-05-28T06:11:26.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.568+0000 7f38dcc00640 1 -- 172.21.15.96:0/1171162195 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f38e0143830 con 0x7f38c0006140 2024-05-28T06:11:26.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.568+0000 7f38d7e00640 1 --2- 172.21.15.96:0/1171162195 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f38bc06cfb0 0x7f38bc06f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:26.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.574+0000 7f38d7e00640 1 --2- 172.21.15.96:0/1171162195 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f38bc06cfb0 0x7f38bc06f470 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f38c8033000 tx=0x7f38c8009030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:11:26.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:26.885+0000 7f38e5600640 1 -- 172.21.15.96:0/1171162195 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone26284", "target": ["mon-mgr", ""]}) -- 0x7f38e00b4f50 con 0x7f38bc06cfb0 2024-05-28T06:11:27.062 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.061+0000 7f38dcc00640 1 -- 172.21.15.96:0/1171162195 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f38e00b4f50 con 0x7f38bc06cfb0 2024-05-28T06:11:27.065 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.064+0000 7f38e5600640 1 -- 172.21.15.96:0/1171162195 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f38bc06cfb0 msgr2=0x7f38bc06f470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:27.065 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.064+0000 7f38e5600640 1 --2- 172.21.15.96:0/1171162195 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f38bc06cfb0 0x7f38bc06f470 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f38c8033000 tx=0x7f38c8009030 comp rx=0 tx=0).stop 2024-05-28T06:11:27.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.064+0000 7f38e5600640 1 -- 172.21.15.96:0/1171162195 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f38c0006140 msgr2=0x7f38e01432f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:27.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.064+0000 7f38e5600640 1 --2- 172.21.15.96:0/1171162195 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f38c0006140 0x7f38e01432f0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f38d0011570 tx=0x7f38d0011a30 comp rx=0 tx=0).stop 2024-05-28T06:11:27.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.064+0000 7f38e5600640 1 -- 172.21.15.96:0/1171162195 shutdown_connections 2024-05-28T06:11:27.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.064+0000 7f38e5600640 1 --2- 172.21.15.96:0/1171162195 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f38bc06cfb0 0x7f38bc06f470 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:27.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.064+0000 7f38e5600640 1 --2- 172.21.15.96:0/1171162195 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f38e00b13a0 0x7f38e0141d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:27.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.064+0000 7f38e5600640 1 --2- 172.21.15.96:0/1171162195 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38e00b0f20 0x7f38e0156d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:27.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.064+0000 7f38e5600640 1 --2- 172.21.15.96:0/1171162195 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f38c0006140 0x7f38e01432f0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:27.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.064+0000 7f38e5600640 1 -- 172.21.15.96:0/1171162195 >> 172.21.15.96:0/1171162195 conn(0x7f38e006b900 msgr2=0x7f38e00cec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:11:27.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.064+0000 7f38e5600640 1 -- 172.21.15.96:0/1171162195 shutdown_connections 2024-05-28T06:11:27.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.064+0000 7f38e5600640 1 -- 172.21.15.96:0/1171162195 wait complete. 2024-05-28T06:11:27.086 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-05-28T06:11:27.265 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.264+0000 7fb819000640 1 Processor -- start 2024-05-28T06:11:27.266 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.264+0000 7fb819000640 1 -- start start 2024-05-28T06:11:27.268 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.265+0000 7fb819000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fb81415c6b0 0x7fb81406b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:27.268 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.266+0000 7fb819000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fb8140731b0 0x7fb81406be40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:27.268 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.266+0000 7fb819000640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb81409ae10 con 0x7fb8140b2a60 2024-05-28T06:11:27.268 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.266+0000 7fb819000640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fb8140b3ee0 con 0x7fb8140731b0 2024-05-28T06:11:27.268 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.266+0000 7fb819000640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fb8140b2420 con 0x7fb81415c6b0 2024-05-28T06:11:27.268 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.266+0000 7fb813e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fb8140731b0 0x7fb81406be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:27.268 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.266+0000 7fb813e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fb8140731b0 0x7fb81406be40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:56482/0 (socket says 172.21.15.96:56482) 2024-05-28T06:11:27.268 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.266+0000 7fb813e00640 1 -- 172.21.15.96:0/3016463554 learned_addr learned my addr 172.21.15.96:0/3016463554 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:11:27.269 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.266+0000 7fb813400640 1 --2- 172.21.15.96:0/3016463554 >> v2:172.21.15.123:3300/0 conn(0x7fb81415c6b0 0x7fb81406b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:27.269 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.266+0000 7fb813e00640 1 -- 172.21.15.96:0/3016463554 >> v2:172.21.15.123:3300/0 conn(0x7fb81415c6b0 msgr2=0x7fb81406b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:11:27.269 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.266+0000 7fb813e00640 1 --2- 172.21.15.96:0/3016463554 >> v2:172.21.15.123:3300/0 conn(0x7fb81415c6b0 0x7fb81406b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:27.269 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.267+0000 7fb813e00640 1 -- 172.21.15.96:0/3016463554 >> v1:172.21.15.123:6789/0 conn(0x7fb8140b2a60 legacy=0x7fb8141602c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:11:27.269 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.267+0000 7fb813e00640 1 -- 172.21.15.96:0/3016463554 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8140b2700 con 0x7fb8140731b0 2024-05-28T06:11:27.270 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.268+0000 7fb813e00640 1 --2- 172.21.15.96:0/3016463554 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb8140731b0 0x7fb81406be40 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fb8140b69c0 tx=0x7fb80c02a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f671f22b0bdfd05b server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:11:27.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.268+0000 7fb812000640 1 -- 172.21.15.96:0/3016463554 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb80c036070 con 0x7fb8140731b0 2024-05-28T06:11:27.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.268+0000 7fb812000640 1 -- 172.21.15.96:0/3016463554 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb80c031600 con 0x7fb8140731b0 2024-05-28T06:11:27.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.268+0000 7fb812000640 1 -- 172.21.15.96:0/3016463554 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb80c0319d0 con 0x7fb8140731b0 2024-05-28T06:11:27.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.269+0000 7fb819000640 1 -- 172.21.15.96:0/3016463554 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb8140731b0 msgr2=0x7fb81406be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:27.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.269+0000 7fb819000640 1 --2- 172.21.15.96:0/3016463554 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb8140731b0 0x7fb81406be40 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fb8140b69c0 tx=0x7fb80c02a6f0 comp rx=0 tx=0).stop 2024-05-28T06:11:27.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.269+0000 7fb819000640 1 -- 172.21.15.96:0/3016463554 shutdown_connections 2024-05-28T06:11:27.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.269+0000 7fb819000640 1 --2- 172.21.15.96:0/3016463554 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb8140731b0 0x7fb81406be40 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:27.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.269+0000 7fb819000640 1 --2- 172.21.15.96:0/3016463554 >> v2:172.21.15.123:3300/0 conn(0x7fb81415c6b0 0x7fb81406b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:27.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.269+0000 7fb819000640 1 -- 172.21.15.96:0/3016463554 >> 172.21.15.96:0/3016463554 conn(0x7fb8140b6a10 msgr2=0x7fb8140b2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:11:27.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.269+0000 7fb819000640 1 -- 172.21.15.96:0/3016463554 shutdown_connections 2024-05-28T06:11:27.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.269+0000 7fb819000640 1 -- 172.21.15.96:0/3016463554 wait complete. 2024-05-28T06:11:27.274 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.270+0000 7fb819000640 1 Processor -- start 2024-05-28T06:11:27.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.271+0000 7fb819000640 1 -- start start 2024-05-28T06:11:27.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.271+0000 7fb819000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb8140731b0 0x7fb81414bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:27.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.271+0000 7fb819000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb8140b2a60 0x7fb8140b0b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:27.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.271+0000 7fb819000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb81415c6b0 0x7fb8140b1090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:27.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.271+0000 7fb819000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb81415ca90 con 0x7fb8140731b0 2024-05-28T06:11:27.276 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.271+0000 7fb819000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fb8140b39c0 con 0x7fb8140b2a60 2024-05-28T06:11:27.276 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.271+0000 7fb819000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fb81409aa30 con 0x7fb81415c6b0 2024-05-28T06:11:27.276 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.272+0000 7fb813400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb8140731b0 0x7fb81414bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:27.276 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.272+0000 7fb813400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb8140731b0 0x7fb81414bf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:56486/0 (socket says 172.21.15.96:56486) 2024-05-28T06:11:27.276 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.272+0000 7fb813400640 1 -- 172.21.15.96:0/2835768810 learned_addr learned my addr 172.21.15.96:0/2835768810 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:11:27.276 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.272+0000 7fb813400640 1 -- 172.21.15.96:0/2835768810 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb81415c6b0 msgr2=0x7fb8140b1090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:27.276 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.272+0000 7fb813400640 1 --2- 172.21.15.96:0/2835768810 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb81415c6b0 0x7fb8140b1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:27.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.272+0000 7fb813400640 1 -- 172.21.15.96:0/2835768810 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb8140b2a60 msgr2=0x7fb8140b0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:27.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.272+0000 7fb813400640 1 --2- 172.21.15.96:0/2835768810 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb8140b2a60 0x7fb8140b0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:27.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.272+0000 7fb813400640 1 -- 172.21.15.96:0/2835768810 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb81414c4c0 con 0x7fb8140731b0 2024-05-28T06:11:27.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.272+0000 7fb813400640 1 --2- 172.21.15.96:0/2835768810 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb8140731b0 0x7fb81414bf80 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fb80400d800 tx=0x7fb80400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:11:27.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.272+0000 7fb80b400640 1 -- 172.21.15.96:0/2835768810 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb80400b6c0 con 0x7fb8140731b0 2024-05-28T06:11:27.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.273+0000 7fb80b400640 1 -- 172.21.15.96:0/2835768810 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb804019070 con 0x7fb8140731b0 2024-05-28T06:11:27.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.273+0000 7fb819000640 1 -- 172.21.15.96:0/2835768810 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8141573c0 con 0x7fb8140731b0 2024-05-28T06:11:27.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.273+0000 7fb80b400640 1 -- 172.21.15.96:0/2835768810 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb80400ba80 con 0x7fb8140731b0 2024-05-28T06:11:27.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.273+0000 7fb819000640 1 -- 172.21.15.96:0/2835768810 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb814157980 con 0x7fb8140731b0 2024-05-28T06:11:27.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.274+0000 7fb819000640 1 -- 172.21.15.96:0/2835768810 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8140690f0 con 0x7fb8140731b0 2024-05-28T06:11:27.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.275+0000 7fb80b400640 1 -- 172.21.15.96:0/2835768810 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 15) ==== 91652+0+0 (secure 0 0 0) 0x7fb80400bc20 con 0x7fb8140731b0 2024-05-28T06:11:27.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.276+0000 7fb80b400640 1 --2- 172.21.15.96:0/2835768810 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb7f006cfe0 0x7fb7f006f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:27.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.276+0000 7fb80b400640 1 -- 172.21.15.96:0/2835768810 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5795+0+0 (secure 0 0 0) 0x7fb80408cb80 con 0x7fb8140731b0 2024-05-28T06:11:27.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.276+0000 7fb812a00640 1 --2- 172.21.15.96:0/2835768810 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb7f006cfe0 0x7fb7f006f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:27.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.277+0000 7fb812a00640 1 --2- 172.21.15.96:0/2835768810 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb7f006cfe0 0x7fb7f006f4a0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc006fd0 tx=0x7fb7fc006bc0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:11:27.281 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.280+0000 7fb80b400640 1 -- 172.21.15.96:0/2835768810 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb80401c360 con 0x7fb8140731b0 2024-05-28T06:11:27.508 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.507+0000 7fb819000640 1 -- 172.21.15.96:0/2835768810 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7fb8140b6640 con 0x7fb7f006cfe0 2024-05-28T06:11:27.511 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.510+0000 7fb80b400640 1 -- 172.21.15.96:0/2835768810 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7fb8140b6640 con 0x7fb7f006cfe0 2024-05-28T06:11:27.512 INFO:teuthology.orchestra.run.smithi096.stdout:[ 2024-05-28T06:11:27.512 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:11:27.512 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "cephfs" 2024-05-28T06:11:27.512 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:11:27.512 INFO:teuthology.orchestra.run.smithi096.stdout:] 2024-05-28T06:11:27.514 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.513+0000 7fb819000640 1 -- 172.21.15.96:0/2835768810 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb7f006cfe0 msgr2=0x7fb7f006f4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:27.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.513+0000 7fb819000640 1 --2- 172.21.15.96:0/2835768810 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb7f006cfe0 0x7fb7f006f4a0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc006fd0 tx=0x7fb7fc006bc0 comp rx=0 tx=0).stop 2024-05-28T06:11:27.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.513+0000 7fb819000640 1 -- 172.21.15.96:0/2835768810 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb8140731b0 msgr2=0x7fb81414bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:27.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.513+0000 7fb819000640 1 --2- 172.21.15.96:0/2835768810 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb8140731b0 0x7fb81414bf80 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fb80400d800 tx=0x7fb80400dcc0 comp rx=0 tx=0).stop 2024-05-28T06:11:27.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.513+0000 7fb819000640 1 -- 172.21.15.96:0/2835768810 shutdown_connections 2024-05-28T06:11:27.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.513+0000 7fb819000640 1 --2- 172.21.15.96:0/2835768810 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb7f006cfe0 0x7fb7f006f4a0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:27.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.513+0000 7fb819000640 1 --2- 172.21.15.96:0/2835768810 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb81415c6b0 0x7fb8140b1090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:27.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.513+0000 7fb819000640 1 --2- 172.21.15.96:0/2835768810 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb8140b2a60 0x7fb8140b0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:27.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.513+0000 7fb819000640 1 --2- 172.21.15.96:0/2835768810 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb8140731b0 0x7fb81414bf80 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:27.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.513+0000 7fb819000640 1 -- 172.21.15.96:0/2835768810 >> 172.21.15.96:0/2835768810 conn(0x7fb8140b6a10 msgr2=0x7fb8140b48f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:11:27.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.513+0000 7fb819000640 1 -- 172.21.15.96:0/2835768810 shutdown_connections 2024-05-28T06:11:27.516 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.513+0000 7fb819000640 1 -- 172.21.15.96:0/2835768810 wait complete. 2024-05-28T06:11:27.535 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-05-28T06:11:27.717 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.715+0000 7f5fdb200640 1 Processor -- start 2024-05-28T06:11:27.717 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.715+0000 7f5fdb200640 1 -- start start 2024-05-28T06:11:27.718 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.716+0000 7f5fdb200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5fd415ea50 0x7f5fd40d1c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:27.719 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.716+0000 7f5fdb200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f5fd40d3290 0x7f5fd406cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:27.719 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.716+0000 7f5fdb200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5fd40d4f90 con 0x7f5fd40b3570 2024-05-28T06:11:27.719 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.717+0000 7f5fdb200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f5fd40d5530 con 0x7f5fd40d3290 2024-05-28T06:11:27.719 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.717+0000 7f5fdb200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f5fd40d5250 con 0x7f5fd415ea50 2024-05-28T06:11:27.719 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.717+0000 7f5fd9400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f5fd40b3570 0x7f5fd40d2b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:46976/0 (socket says 172.21.15.96:46976) 2024-05-28T06:11:27.719 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.717+0000 7f5fd9400640 1 -- 172.21.15.96:0/772292870 learned_addr learned my addr 172.21.15.96:0/772292870 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:11:27.719 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.717+0000 7f5fda800640 1 --2- 172.21.15.96:0/772292870 >> v2:172.21.15.96:3300/0 conn(0x7f5fd40d3290 0x7f5fd406cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:27.719 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.717+0000 7f5fd9e00640 1 --2- 172.21.15.96:0/772292870 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5fd415ea50 0x7f5fd40d1c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:27.719 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.717+0000 7f5fd8a00640 1 -- 172.21.15.96:0/772292870 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3066004927 0 0) 0x7f5fd40d4f90 con 0x7f5fd40b3570 2024-05-28T06:11:27.719 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.717+0000 7f5fd8a00640 1 -- 172.21.15.96:0/772292870 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5fc0003520 con 0x7f5fd40b3570 2024-05-28T06:11:27.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.718+0000 7f5fda800640 1 -- 172.21.15.96:0/772292870 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5fd415ea50 msgr2=0x7f5fd40d1c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:11:27.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.718+0000 7f5fda800640 1 --2- 172.21.15.96:0/772292870 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5fd415ea50 0x7f5fd40d1c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:27.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.718+0000 7f5fda800640 1 -- 172.21.15.96:0/772292870 >> v1:172.21.15.96:6789/0 conn(0x7f5fd40b3570 legacy=0x7f5fd40d2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:27.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.718+0000 7f5fda800640 1 -- 172.21.15.96:0/772292870 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fd40d2520 con 0x7f5fd40d3290 2024-05-28T06:11:27.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.719+0000 7f5fda800640 1 --2- 172.21.15.96:0/772292870 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5fd40d3290 0x7f5fd406cc40 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f5fd00097a0 tx=0x7f5fd002a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3f4e31fd506d3a0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:11:27.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.719+0000 7f5fd8a00640 1 -- 172.21.15.96:0/772292870 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fd0037070 con 0x7f5fd40d3290 2024-05-28T06:11:27.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.719+0000 7f5fd8a00640 1 -- 172.21.15.96:0/772292870 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5fd002aaf0 con 0x7f5fd40d3290 2024-05-28T06:11:27.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.719+0000 7f5fd8a00640 1 -- 172.21.15.96:0/772292870 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fc4001670 con 0x7f5fd40d3290 2024-05-28T06:11:27.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.720+0000 7f5fdb200640 1 -- 172.21.15.96:0/772292870 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5fd40d3290 msgr2=0x7f5fd406cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:27.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.720+0000 7f5fdb200640 1 --2- 172.21.15.96:0/772292870 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5fd40d3290 0x7f5fd406cc40 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f5fd00097a0 tx=0x7f5fd002a060 comp rx=0 tx=0).stop 2024-05-28T06:11:27.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.720+0000 7f5fdb200640 1 -- 172.21.15.96:0/772292870 shutdown_connections 2024-05-28T06:11:27.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.720+0000 7f5fdb200640 1 --2- 172.21.15.96:0/772292870 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5fd40d3290 0x7f5fd406cc40 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:27.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.720+0000 7f5fdb200640 1 --2- 172.21.15.96:0/772292870 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5fd415ea50 0x7f5fd40d1c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:27.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.720+0000 7f5fdb200640 1 -- 172.21.15.96:0/772292870 >> 172.21.15.96:0/772292870 conn(0x7f5fd408bfc0 msgr2=0x7f5fd40d45a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:11:27.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.720+0000 7f5fdb200640 1 -- 172.21.15.96:0/772292870 shutdown_connections 2024-05-28T06:11:27.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.720+0000 7f5fdb200640 1 -- 172.21.15.96:0/772292870 wait complete. 2024-05-28T06:11:27.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.721+0000 7f5fdb200640 1 Processor -- start 2024-05-28T06:11:27.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.722+0000 7f5fdb200640 1 -- start start 2024-05-28T06:11:27.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.722+0000 7f5fdb200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5fd40b3570 0x7f5fd415aed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:27.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.722+0000 7f5fdb200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5fd40d3290 0x7f5fd40b8400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:27.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.722+0000 7f5fdb200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5fd415ea50 0x7f5fd40b8940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:27.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.722+0000 7f5fdb200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f5fd406d180 con 0x7f5fd40b3570 2024-05-28T06:11:27.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.722+0000 7f5fdb200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5fd415ee30 con 0x7f5fd40d3290 2024-05-28T06:11:27.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.722+0000 7f5fdb200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f5fd406d300 con 0x7f5fd415ea50 2024-05-28T06:11:27.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.723+0000 7f5fd9e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5fd40b3570 0x7f5fd415aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:27.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.723+0000 7f5fd9e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5fd40b3570 0x7f5fd415aed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:56500/0 (socket says 172.21.15.96:56500) 2024-05-28T06:11:27.725 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.723+0000 7f5fd9e00640 1 -- 172.21.15.96:0/782272928 learned_addr learned my addr 172.21.15.96:0/782272928 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:11:27.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.723+0000 7f5fd9400640 1 --2- 172.21.15.96:0/782272928 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5fd40d3290 0x7f5fd40b8400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:27.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.723+0000 7f5fd9e00640 1 -- 172.21.15.96:0/782272928 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5fd415ea50 msgr2=0x7f5fd40b8940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:27.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.723+0000 7f5fd9e00640 1 --2- 172.21.15.96:0/782272928 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5fd415ea50 0x7f5fd40b8940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:27.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.723+0000 7f5fd9e00640 1 -- 172.21.15.96:0/782272928 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5fd40d3290 msgr2=0x7f5fd40b8400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:27.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.723+0000 7f5fd9e00640 1 --2- 172.21.15.96:0/782272928 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5fd40d3290 0x7f5fd40b8400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:27.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.723+0000 7f5fd9e00640 1 -- 172.21.15.96:0/782272928 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fd415b410 con 0x7f5fd40b3570 2024-05-28T06:11:27.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.723+0000 7f5fd9400640 1 --2- 172.21.15.96:0/782272928 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5fd40d3290 0x7f5fd40b8400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:11:27.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.723+0000 7f5fd9e00640 1 --2- 172.21.15.96:0/782272928 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5fd40b3570 0x7f5fd415aed0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f5fcc007e70 tx=0x7f5fcc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:11:27.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.724+0000 7f5fca000640 1 -- 172.21.15.96:0/782272928 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fcc014620 con 0x7f5fd40b3570 2024-05-28T06:11:27.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.724+0000 7f5fca000640 1 -- 172.21.15.96:0/782272928 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5fcc019070 con 0x7f5fd40b3570 2024-05-28T06:11:27.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.724+0000 7f5fca000640 1 -- 172.21.15.96:0/782272928 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fcc014a00 con 0x7f5fd40b3570 2024-05-28T06:11:27.727 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.724+0000 7f5fdb200640 1 -- 172.21.15.96:0/782272928 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fd40b7360 con 0x7f5fd40b3570 2024-05-28T06:11:27.727 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.724+0000 7f5fdb200640 1 -- 172.21.15.96:0/782272928 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fd40b7920 con 0x7f5fd40b3570 2024-05-28T06:11:27.727 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.725+0000 7f5fdb200640 1 -- 172.21.15.96:0/782272928 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5fd40689e0 con 0x7f5fd40b3570 2024-05-28T06:11:27.727 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.726+0000 7f5fca000640 1 -- 172.21.15.96:0/782272928 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 15) ==== 91652+0+0 (secure 0 0 0) 0x7f5fcc014ba0 con 0x7f5fd40b3570 2024-05-28T06:11:27.729 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.727+0000 7f5fca000640 1 --2- 172.21.15.96:0/782272928 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5fb806cfb0 0x7f5fb806f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:27.729 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.727+0000 7f5fca000640 1 -- 172.21.15.96:0/782272928 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5795+0+0 (secure 0 0 0) 0x7f5fcc08cf30 con 0x7f5fd40b3570 2024-05-28T06:11:27.729 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.728+0000 7f5fd9400640 1 --2- 172.21.15.96:0/782272928 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5fb806cfb0 0x7f5fb806f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:27.730 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.729+0000 7f5fd9400640 1 --2- 172.21.15.96:0/782272928 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5fb806cfb0 0x7f5fb806f470 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f5fc40017e0 tx=0x7f5fc4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:11:27.733 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.732+0000 7f5fca000640 1 -- 172.21.15.96:0/782272928 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5fcc090030 con 0x7f5fd40b3570 2024-05-28T06:11:27.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.990+0000 7f5fdb200640 1 -- 172.21.15.96:0/782272928 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f5fd40d49d0 con 0x7f5fd40b3570 2024-05-28T06:11:27.995 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.993+0000 7f5fca000640 1 -- 172.21.15.96:0/782272928 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+2142 (secure 0 0 0) 0x7f5fd415b410 con 0x7f5fd40b3570 2024-05-28T06:11:27.995 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:11:27.995 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:11:27.995 INFO:teuthology.orchestra.run.smithi096.stdout: "fsid": "e4a831b5-e6d1-47a7-96cb-bf85154ee0d2", 2024-05-28T06:11:27.995 INFO:teuthology.orchestra.run.smithi096.stdout: "health": { 2024-05-28T06:11:27.995 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "HEALTH_OK", 2024-05-28T06:11:27.995 INFO:teuthology.orchestra.run.smithi096.stdout: "checks": {}, 2024-05-28T06:11:27.995 INFO:teuthology.orchestra.run.smithi096.stdout: "mutes": [] 2024-05-28T06:11:27.996 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:11:27.996 INFO:teuthology.orchestra.run.smithi096.stdout: "election_epoch": 8, 2024-05-28T06:11:27.996 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum": [ 2024-05-28T06:11:27.996 INFO:teuthology.orchestra.run.smithi096.stdout: 0, 2024-05-28T06:11:27.996 INFO:teuthology.orchestra.run.smithi096.stdout: 1, 2024-05-28T06:11:27.996 INFO:teuthology.orchestra.run.smithi096.stdout: 2 2024-05-28T06:11:27.996 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:11:27.996 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_names": [ 2024-05-28T06:11:27.996 INFO:teuthology.orchestra.run.smithi096.stdout: "a", 2024-05-28T06:11:27.996 INFO:teuthology.orchestra.run.smithi096.stdout: "b", 2024-05-28T06:11:27.996 INFO:teuthology.orchestra.run.smithi096.stdout: "c" 2024-05-28T06:11:27.996 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:11:27.996 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_age": 461, 2024-05-28T06:11:27.997 INFO:teuthology.orchestra.run.smithi096.stdout: "monmap": { 2024-05-28T06:11:27.997 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 1, 2024-05-28T06:11:27.997 INFO:teuthology.orchestra.run.smithi096.stdout: "min_mon_release_name": "squid", 2024-05-28T06:11:27.997 INFO:teuthology.orchestra.run.smithi096.stdout: "num_mons": 3 2024-05-28T06:11:27.997 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:11:27.997 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap": { 2024-05-28T06:11:27.997 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 47, 2024-05-28T06:11:27.997 INFO:teuthology.orchestra.run.smithi096.stdout: "num_osds": 8, 2024-05-28T06:11:27.997 INFO:teuthology.orchestra.run.smithi096.stdout: "num_up_osds": 8, 2024-05-28T06:11:27.997 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_up_since": 1716876238, 2024-05-28T06:11:27.997 INFO:teuthology.orchestra.run.smithi096.stdout: "num_in_osds": 8, 2024-05-28T06:11:27.997 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_in_since": 1716876229, 2024-05-28T06:11:27.997 INFO:teuthology.orchestra.run.smithi096.stdout: "num_remapped_pgs": 0 2024-05-28T06:11:27.998 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:11:27.998 INFO:teuthology.orchestra.run.smithi096.stdout: "pgmap": { 2024-05-28T06:11:27.998 INFO:teuthology.orchestra.run.smithi096.stdout: "pgs_by_state": [ 2024-05-28T06:11:27.998 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:11:27.998 INFO:teuthology.orchestra.run.smithi096.stdout: "state_name": "active+clean", 2024-05-28T06:11:27.998 INFO:teuthology.orchestra.run.smithi096.stdout: "count": 97 2024-05-28T06:11:27.998 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:11:27.998 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:11:27.998 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pgs": 97, 2024-05-28T06:11:27.998 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pools": 3, 2024-05-28T06:11:27.998 INFO:teuthology.orchestra.run.smithi096.stdout: "num_objects": 3814, 2024-05-28T06:11:27.998 INFO:teuthology.orchestra.run.smithi096.stdout: "data_bytes": 15851964532, 2024-05-28T06:11:27.998 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_used": 31238504448, 2024-05-28T06:11:27.998 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_avail": 741855608832, 2024-05-28T06:11:27.999 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_total": 773094113280, 2024-05-28T06:11:27.999 INFO:teuthology.orchestra.run.smithi096.stdout: "read_bytes_sec": 436782936, 2024-05-28T06:11:27.999 INFO:teuthology.orchestra.run.smithi096.stdout: "write_bytes_sec": 431242138, 2024-05-28T06:11:27.999 INFO:teuthology.orchestra.run.smithi096.stdout: "read_op_per_sec": 106, 2024-05-28T06:11:27.999 INFO:teuthology.orchestra.run.smithi096.stdout: "write_op_per_sec": 118 2024-05-28T06:11:27.999 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:11:27.999 INFO:teuthology.orchestra.run.smithi096.stdout: "fsmap": { 2024-05-28T06:11:27.999 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 26, 2024-05-28T06:11:27.999 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 3, 2024-05-28T06:11:27.999 INFO:teuthology.orchestra.run.smithi096.stdout: "up": 1, 2024-05-28T06:11:27.999 INFO:teuthology.orchestra.run.smithi096.stdout: "in": 1, 2024-05-28T06:11:27.999 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 1, 2024-05-28T06:11:27.999 INFO:teuthology.orchestra.run.smithi096.stdout: "by_rank": [ 2024-05-28T06:11:28.000 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:11:28.000 INFO:teuthology.orchestra.run.smithi096.stdout: "filesystem_id": 3, 2024-05-28T06:11:28.000 INFO:teuthology.orchestra.run.smithi096.stdout: "rank": 0, 2024-05-28T06:11:28.000 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "a", 2024-05-28T06:11:28.000 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "up:active", 2024-05-28T06:11:28.000 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 5859 2024-05-28T06:11:28.000 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:11:28.000 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:11:28.000 INFO:teuthology.orchestra.run.smithi096.stdout: "up:standby": 3 2024-05-28T06:11:28.000 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:11:28.000 INFO:teuthology.orchestra.run.smithi096.stdout: "mgrmap": { 2024-05-28T06:11:28.000 INFO:teuthology.orchestra.run.smithi096.stdout: "available": true, 2024-05-28T06:11:28.000 INFO:teuthology.orchestra.run.smithi096.stdout: "num_standbys": 1, 2024-05-28T06:11:28.000 INFO:teuthology.orchestra.run.smithi096.stdout: "modules": [ 2024-05-28T06:11:28.001 INFO:teuthology.orchestra.run.smithi096.stdout: "iostat", 2024-05-28T06:11:28.001 INFO:teuthology.orchestra.run.smithi096.stdout: "nfs", 2024-05-28T06:11:28.001 INFO:teuthology.orchestra.run.smithi096.stdout: "restful" 2024-05-28T06:11:28.001 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:11:28.001 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:11:28.001 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:11:28.001 INFO:teuthology.orchestra.run.smithi096.stdout: "servicemap": { 2024-05-28T06:11:28.001 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 9, 2024-05-28T06:11:28.001 INFO:teuthology.orchestra.run.smithi096.stdout: "modified": "2024-05-28T06:10:01.618312+0000", 2024-05-28T06:11:28.001 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:11:28.001 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:11:28.001 INFO:teuthology.orchestra.run.smithi096.stdout: "progress_events": { 2024-05-28T06:11:28.001 INFO:teuthology.orchestra.run.smithi096.stdout: "72116a8d-650b-4cfc-99af-ee8db5167322": { 2024-05-28T06:11:28.002 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "2 ongoing clones; Avg progress is 10.742% (8s)\n [===.........................] (remaining: 66s)", 2024-05-28T06:11:28.002 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": 0.10741999745368958, 2024-05-28T06:11:28.002 INFO:teuthology.orchestra.run.smithi096.stdout: "add_to_ceph_s": true 2024-05-28T06:11:28.002 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:11:28.002 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:11:28.002 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:11:28.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.996+0000 7f5fdb200640 1 -- 172.21.15.96:0/782272928 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5fb806cfb0 msgr2=0x7f5fb806f470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:28.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.996+0000 7f5fdb200640 1 --2- 172.21.15.96:0/782272928 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5fb806cfb0 0x7f5fb806f470 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f5fc40017e0 tx=0x7f5fc4009040 comp rx=0 tx=0).stop 2024-05-28T06:11:28.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.996+0000 7f5fdb200640 1 -- 172.21.15.96:0/782272928 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5fd40b3570 msgr2=0x7f5fd415aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:28.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.996+0000 7f5fdb200640 1 --2- 172.21.15.96:0/782272928 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5fd40b3570 0x7f5fd415aed0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f5fcc007e70 tx=0x7f5fcc00e4b0 comp rx=0 tx=0).stop 2024-05-28T06:11:28.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.996+0000 7f5fdb200640 1 -- 172.21.15.96:0/782272928 shutdown_connections 2024-05-28T06:11:28.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.996+0000 7f5fdb200640 1 --2- 172.21.15.96:0/782272928 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5fb806cfb0 0x7f5fb806f470 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:28.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.996+0000 7f5fdb200640 1 --2- 172.21.15.96:0/782272928 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5fd415ea50 0x7f5fd40b8940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:28.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.996+0000 7f5fdb200640 1 --2- 172.21.15.96:0/782272928 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5fd40d3290 0x7f5fd40b8400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:28.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.996+0000 7f5fdb200640 1 --2- 172.21.15.96:0/782272928 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5fd40b3570 0x7f5fd415aed0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:28.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.996+0000 7f5fdb200640 1 -- 172.21.15.96:0/782272928 >> 172.21.15.96:0/782272928 conn(0x7f5fd408bfc0 msgr2=0x7f5fd40d39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:11:28.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.996+0000 7f5fdb200640 1 -- 172.21.15.96:0/782272928 shutdown_connections 2024-05-28T06:11:28.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:27.996+0000 7f5fdb200640 1 -- 172.21.15.96:0/782272928 wait complete. 2024-05-28T06:11:30.856 DEBUG:teuthology.orchestra.run.smithi096:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:11:30.865 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:11:33.017 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-05-28T06:11:33.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:33.223+0000 7f9864000640 1 Processor -- start 2024-05-28T06:11:33.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:33.223+0000 7f9864000640 1 -- start start 2024-05-28T06:11:33.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:33.223+0000 7f9864000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f985c15ea40 0x7f985c0d4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:33.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:33.223+0000 7f9864000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f985c06c950 0x7f985c06cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:33.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:33.223+0000 7f9864000640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f985c0cf7e0 con 0x7f985c0d1040 2024-05-28T06:11:33.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:33.223+0000 7f9864000640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f985c15ee20 con 0x7f985c06c950 2024-05-28T06:11:33.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:33.224+0000 7f9864000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f985c15fe50 con 0x7f985c15ea40 2024-05-28T06:11:33.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:33.224+0000 7f9862000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f985c06c950 0x7f985c06cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:33.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:33.224+0000 7f9861600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f985c15ea40 0x7f985c0d4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:33.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:33.224+0000 7f9862000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f985c06c950 0x7f985c06cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.96:59516/0 (socket says 172.21.15.96:59516) 2024-05-28T06:11:33.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:33.224+0000 7f9862000640 1 -- 172.21.15.96:0/2295707419 learned_addr learned my addr 172.21.15.96:0/2295707419 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:11:33.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:33.225+0000 7f9862000640 1 -- 172.21.15.96:0/2295707419 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f985c15ea40 msgr2=0x7f985c0d4f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:11:33.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:33.225+0000 7f9862000640 1 --2- 172.21.15.96:0/2295707419 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f985c15ea40 0x7f985c0d4f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:33.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:33.225+0000 7f9862000640 1 -- 172.21.15.96:0/2295707419 >> v1:172.21.15.123:6789/0 conn(0x7f985c0d1040 legacy=0x7f985c0d1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:33.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:33.225+0000 7f9862000640 1 -- 172.21.15.96:0/2295707419 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f985c0cfa80 con 0x7f985c06c950 2024-05-28T06:11:33.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:33.226+0000 7f9862000640 1 --2- 172.21.15.96:0/2295707419 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f985c06c950 0x7f985c06cda0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f9854009e50 tx=0x7f985402a680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=66396881ed146cae server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:11:33.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:33.227+0000 7f985be00640 1 -- 172.21.15.96:0/2295707419 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9854036070 con 0x7f985c06c950 2024-05-28T06:11:33.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:33.227+0000 7f985be00640 1 -- 172.21.15.96:0/2295707419 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9854031600 con 0x7f985c06c950 2024-05-28T06:11:33.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:33.227+0000 7f985be00640 1 -- 172.21.15.96:0/2295707419 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98540319d0 con 0x7f985c06c950 2024-05-28T06:11:33.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:33.227+0000 7f9864000640 1 -- 172.21.15.96:0/2295707419 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f985c06c950 msgr2=0x7f985c06cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:33.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:33.227+0000 7f9864000640 1 --2- 172.21.15.96:0/2295707419 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f985c06c950 0x7f985c06cda0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f9854009e50 tx=0x7f985402a680 comp rx=0 tx=0).stop 2024-05-28T06:11:33.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:33.227+0000 7f9864000640 1 -- 172.21.15.96:0/2295707419 shutdown_connections 2024-05-28T06:11:33.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:33.227+0000 7f9864000640 1 --2- 172.21.15.96:0/2295707419 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f985c06c950 0x7f985c06cda0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:33.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:33.227+0000 7f9864000640 1 --2- 172.21.15.96:0/2295707419 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f985c15ea40 0x7f985c0d4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:33.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:33.227+0000 7f9864000640 1 -- 172.21.15.96:0/2295707419 >> 172.21.15.96:0/2295707419 conn(0x7f985c08bfc0 msgr2=0x7f985c0d4630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:11:33.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:33.227+0000 7f9864000640 1 -- 172.21.15.96:0/2295707419 shutdown_connections 2024-05-28T06:11:33.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:33.228+0000 7f9864000640 1 -- 172.21.15.96:0/2295707419 wait complete. 2024-05-28T06:11:33.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:33.229+0000 7f9864000640 1 Processor -- start 2024-05-28T06:11:33.231 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:33.229+0000 7f9864000640 1 -- start start 2024-05-28T06:11:33.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:33.230+0000 7f9864000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f985c06c950 0x7f985c153a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:33.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:33.230+0000 7f9864000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f985c0d1040 0x7f985c1612c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:33.233 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:33.230+0000 7f9864000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f985c15ea40 0x7f985c0cd880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:33.233 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:33.230+0000 7f9864000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f985c0d0320 con 0x7f985c15ea40 2024-05-28T06:11:33.233 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:33.230+0000 7f9864000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f985c0d0040 con 0x7f985c06c950 2024-05-28T06:11:33.233 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:33.230+0000 7f9864000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f985c0d5440 con 0x7f985c0d1040 2024-05-28T06:11:33.233 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:33.231+0000 7f9862000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f985c15ea40 0x7f985c0cd880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:33.233 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:33.231+0000 7f9862000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f985c15ea40 0x7f985c0cd880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48720/0 (socket says 172.21.15.96:48720) 2024-05-28T06:11:33.233 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:33.231+0000 7f9862000640 1 -- 172.21.15.96:0/296978064 learned_addr learned my addr 172.21.15.96:0/296978064 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:11:33.233 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:33.231+0000 7f9861600640 1 --2- 172.21.15.96:0/296978064 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f985c06c950 0x7f985c153a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:33.233 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:33.231+0000 7f9860c00640 1 --2- 172.21.15.96:0/296978064 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f985c0d1040 0x7f985c1612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:33.233 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:33.231+0000 7f9862000640 1 -- 172.21.15.96:0/296978064 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f985c0d1040 msgr2=0x7f985c1612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:33.233 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:33.231+0000 7f9862000640 1 --2- 172.21.15.96:0/296978064 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f985c0d1040 0x7f985c1612c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:33.233 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:33.231+0000 7f9862000640 1 -- 172.21.15.96:0/296978064 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f985c06c950 msgr2=0x7f985c153a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:33.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:33.231+0000 7f9862000640 1 --2- 172.21.15.96:0/296978064 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f985c06c950 0x7f985c153a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:33.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:33.231+0000 7f9862000640 1 -- 172.21.15.96:0/296978064 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f985c0ce760 con 0x7f985c15ea40 2024-05-28T06:11:33.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:33.231+0000 7f9860c00640 1 --2- 172.21.15.96:0/296978064 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f985c0d1040 0x7f985c1612c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:11:33.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:33.231+0000 7f9862000640 1 --2- 172.21.15.96:0/296978064 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f985c15ea40 0x7f985c0cd880 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f9854009f70 tx=0x7f9854004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:11:33.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:33.232+0000 7f9859600640 1 -- 172.21.15.96:0/296978064 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f985403f020 con 0x7f985c15ea40 2024-05-28T06:11:33.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:33.232+0000 7f9859600640 1 -- 172.21.15.96:0/296978064 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9854037040 con 0x7f985c15ea40 2024-05-28T06:11:33.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:33.232+0000 7f9859600640 1 -- 172.21.15.96:0/296978064 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9854036070 con 0x7f985c15ea40 2024-05-28T06:11:33.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:33.232+0000 7f9864000640 1 -- 172.21.15.96:0/296978064 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f985c160220 con 0x7f985c15ea40 2024-05-28T06:11:33.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:33.232+0000 7f9864000640 1 -- 172.21.15.96:0/296978064 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f985c1607e0 con 0x7f985c15ea40 2024-05-28T06:11:33.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:33.233+0000 7f9864000640 1 -- 172.21.15.96:0/296978064 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f985c0689e0 con 0x7f985c15ea40 2024-05-28T06:11:33.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:33.234+0000 7f9859600640 1 -- 172.21.15.96:0/296978064 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 15) ==== 91652+0+0 (secure 0 0 0) 0x7f9854042020 con 0x7f985c15ea40 2024-05-28T06:11:33.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:33.235+0000 7f9859600640 1 --2- 172.21.15.96:0/296978064 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f983c06cfe0 0x7f983c06f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:33.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:33.235+0000 7f9859600640 1 -- 172.21.15.96:0/296978064 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5795+0+0 (secure 0 0 0) 0x7f98540acd70 con 0x7f985c15ea40 2024-05-28T06:11:33.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:33.236+0000 7f9861600640 1 --2- 172.21.15.96:0/296978064 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f983c06cfe0 0x7f983c06f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:33.238 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:33.237+0000 7f9861600640 1 --2- 172.21.15.96:0/296978064 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f983c06cfe0 0x7f983c06f4a0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f9850009a30 tx=0x7f9850008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:11:33.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:33.240+0000 7f9859600640 1 -- 172.21.15.96:0/296978064 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9854049030 con 0x7f985c15ea40 2024-05-28T06:11:33.511 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:33.510+0000 7f9864000640 1 -- 172.21.15.96:0/296978064 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f985c0ba5b0 con 0x7f985c15ea40 2024-05-28T06:11:33.512 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:33.511+0000 7f9859600640 1 -- 172.21.15.96:0/296978064 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+2125 (secure 0 0 0) 0x7f985407a660 con 0x7f985c15ea40 2024-05-28T06:11:33.512 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:11:33.512 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:11:33.513 INFO:teuthology.orchestra.run.smithi096.stdout: "fsid": "e4a831b5-e6d1-47a7-96cb-bf85154ee0d2", 2024-05-28T06:11:33.513 INFO:teuthology.orchestra.run.smithi096.stdout: "health": { 2024-05-28T06:11:33.513 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "HEALTH_OK", 2024-05-28T06:11:33.513 INFO:teuthology.orchestra.run.smithi096.stdout: "checks": {}, 2024-05-28T06:11:33.513 INFO:teuthology.orchestra.run.smithi096.stdout: "mutes": [] 2024-05-28T06:11:33.513 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:11:33.513 INFO:teuthology.orchestra.run.smithi096.stdout: "election_epoch": 8, 2024-05-28T06:11:33.513 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum": [ 2024-05-28T06:11:33.513 INFO:teuthology.orchestra.run.smithi096.stdout: 0, 2024-05-28T06:11:33.513 INFO:teuthology.orchestra.run.smithi096.stdout: 1, 2024-05-28T06:11:33.513 INFO:teuthology.orchestra.run.smithi096.stdout: 2 2024-05-28T06:11:33.513 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:11:33.513 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_names": [ 2024-05-28T06:11:33.514 INFO:teuthology.orchestra.run.smithi096.stdout: "a", 2024-05-28T06:11:33.514 INFO:teuthology.orchestra.run.smithi096.stdout: "b", 2024-05-28T06:11:33.514 INFO:teuthology.orchestra.run.smithi096.stdout: "c" 2024-05-28T06:11:33.514 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:11:33.514 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_age": 467, 2024-05-28T06:11:33.514 INFO:teuthology.orchestra.run.smithi096.stdout: "monmap": { 2024-05-28T06:11:33.514 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 1, 2024-05-28T06:11:33.514 INFO:teuthology.orchestra.run.smithi096.stdout: "min_mon_release_name": "squid", 2024-05-28T06:11:33.514 INFO:teuthology.orchestra.run.smithi096.stdout: "num_mons": 3 2024-05-28T06:11:33.514 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:11:33.514 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap": { 2024-05-28T06:11:33.514 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 47, 2024-05-28T06:11:33.514 INFO:teuthology.orchestra.run.smithi096.stdout: "num_osds": 8, 2024-05-28T06:11:33.515 INFO:teuthology.orchestra.run.smithi096.stdout: "num_up_osds": 8, 2024-05-28T06:11:33.515 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_up_since": 1716876238, 2024-05-28T06:11:33.515 INFO:teuthology.orchestra.run.smithi096.stdout: "num_in_osds": 8, 2024-05-28T06:11:33.515 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_in_since": 1716876229, 2024-05-28T06:11:33.515 INFO:teuthology.orchestra.run.smithi096.stdout: "num_remapped_pgs": 0 2024-05-28T06:11:33.515 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:11:33.515 INFO:teuthology.orchestra.run.smithi096.stdout: "pgmap": { 2024-05-28T06:11:33.515 INFO:teuthology.orchestra.run.smithi096.stdout: "pgs_by_state": [ 2024-05-28T06:11:33.515 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:11:33.515 INFO:teuthology.orchestra.run.smithi096.stdout: "state_name": "active+clean", 2024-05-28T06:11:33.515 INFO:teuthology.orchestra.run.smithi096.stdout: "count": 97 2024-05-28T06:11:33.515 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:11:33.515 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:11:33.516 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pgs": 97, 2024-05-28T06:11:33.516 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pools": 3, 2024-05-28T06:11:33.516 INFO:teuthology.orchestra.run.smithi096.stdout: "num_objects": 4177, 2024-05-28T06:11:33.516 INFO:teuthology.orchestra.run.smithi096.stdout: "data_bytes": 17383116873, 2024-05-28T06:11:33.516 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_used": 35149709312, 2024-05-28T06:11:33.516 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_avail": 737944403968, 2024-05-28T06:11:33.516 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_total": 773094113280, 2024-05-28T06:11:33.516 INFO:teuthology.orchestra.run.smithi096.stdout: "read_bytes_sec": 494391076, 2024-05-28T06:11:33.516 INFO:teuthology.orchestra.run.smithi096.stdout: "write_bytes_sec": 498626924, 2024-05-28T06:11:33.516 INFO:teuthology.orchestra.run.smithi096.stdout: "read_op_per_sec": 121, 2024-05-28T06:11:33.516 INFO:teuthology.orchestra.run.smithi096.stdout: "write_op_per_sec": 134 2024-05-28T06:11:33.516 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:11:33.516 INFO:teuthology.orchestra.run.smithi096.stdout: "fsmap": { 2024-05-28T06:11:33.516 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 26, 2024-05-28T06:11:33.516 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 3, 2024-05-28T06:11:33.517 INFO:teuthology.orchestra.run.smithi096.stdout: "up": 1, 2024-05-28T06:11:33.517 INFO:teuthology.orchestra.run.smithi096.stdout: "in": 1, 2024-05-28T06:11:33.517 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 1, 2024-05-28T06:11:33.517 INFO:teuthology.orchestra.run.smithi096.stdout: "by_rank": [ 2024-05-28T06:11:33.517 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:11:33.517 INFO:teuthology.orchestra.run.smithi096.stdout: "filesystem_id": 3, 2024-05-28T06:11:33.517 INFO:teuthology.orchestra.run.smithi096.stdout: "rank": 0, 2024-05-28T06:11:33.517 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "a", 2024-05-28T06:11:33.517 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "up:active", 2024-05-28T06:11:33.517 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 5859 2024-05-28T06:11:33.517 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:11:33.517 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:11:33.517 INFO:teuthology.orchestra.run.smithi096.stdout: "up:standby": 3 2024-05-28T06:11:33.517 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:11:33.517 INFO:teuthology.orchestra.run.smithi096.stdout: "mgrmap": { 2024-05-28T06:11:33.518 INFO:teuthology.orchestra.run.smithi096.stdout: "available": true, 2024-05-28T06:11:33.518 INFO:teuthology.orchestra.run.smithi096.stdout: "num_standbys": 1, 2024-05-28T06:11:33.518 INFO:teuthology.orchestra.run.smithi096.stdout: "modules": [ 2024-05-28T06:11:33.518 INFO:teuthology.orchestra.run.smithi096.stdout: "iostat", 2024-05-28T06:11:33.518 INFO:teuthology.orchestra.run.smithi096.stdout: "nfs", 2024-05-28T06:11:33.518 INFO:teuthology.orchestra.run.smithi096.stdout: "restful" 2024-05-28T06:11:33.518 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:11:33.518 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:11:33.518 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:11:33.518 INFO:teuthology.orchestra.run.smithi096.stdout: "servicemap": { 2024-05-28T06:11:33.518 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 9, 2024-05-28T06:11:33.518 INFO:teuthology.orchestra.run.smithi096.stdout: "modified": "2024-05-28T06:10:01.618312+0000", 2024-05-28T06:11:33.518 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:11:33.519 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:11:33.519 INFO:teuthology.orchestra.run.smithi096.stdout: "progress_events": { 2024-05-28T06:11:33.519 INFO:teuthology.orchestra.run.smithi096.stdout: "72116a8d-650b-4cfc-99af-ee8db5167322": { 2024-05-28T06:11:33.519 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "All cloning finished, waiting 26.0 seconds before exiting (14s)\n [============================] ", 2024-05-28T06:11:33.519 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": 1, 2024-05-28T06:11:33.519 INFO:teuthology.orchestra.run.smithi096.stdout: "add_to_ceph_s": true 2024-05-28T06:11:33.519 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:11:33.519 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:11:33.519 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:11:33.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:33.513+0000 7f9864000640 1 -- 172.21.15.96:0/296978064 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f983c06cfe0 msgr2=0x7f983c06f4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:33.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:33.513+0000 7f9864000640 1 --2- 172.21.15.96:0/296978064 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f983c06cfe0 0x7f983c06f4a0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f9850009a30 tx=0x7f9850008040 comp rx=0 tx=0).stop 2024-05-28T06:11:33.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:33.513+0000 7f9864000640 1 -- 172.21.15.96:0/296978064 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f985c15ea40 msgr2=0x7f985c0cd880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:33.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:33.513+0000 7f9864000640 1 --2- 172.21.15.96:0/296978064 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f985c15ea40 0x7f985c0cd880 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f9854009f70 tx=0x7f9854004a40 comp rx=0 tx=0).stop 2024-05-28T06:11:33.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:33.513+0000 7f9864000640 1 -- 172.21.15.96:0/296978064 shutdown_connections 2024-05-28T06:11:33.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:33.513+0000 7f9864000640 1 --2- 172.21.15.96:0/296978064 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f983c06cfe0 0x7f983c06f4a0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:33.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:33.513+0000 7f9864000640 1 --2- 172.21.15.96:0/296978064 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f985c15ea40 0x7f985c0cd880 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:33.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:33.513+0000 7f9864000640 1 --2- 172.21.15.96:0/296978064 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f985c0d1040 0x7f985c1612c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:33.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:33.513+0000 7f9864000640 1 --2- 172.21.15.96:0/296978064 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f985c06c950 0x7f985c153a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:33.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:33.513+0000 7f9864000640 1 -- 172.21.15.96:0/296978064 >> 172.21.15.96:0/296978064 conn(0x7f985c08bfc0 msgr2=0x7f985c0b2be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:11:33.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:33.513+0000 7f9864000640 1 -- 172.21.15.96:0/296978064 shutdown_connections 2024-05-28T06:11:33.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:33.513+0000 7f9864000640 1 -- 172.21.15.96:0/296978064 wait complete. 2024-05-28T06:11:38.533 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-05-28T06:11:38.743 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:38.742+0000 7f034a400640 1 Processor -- start 2024-05-28T06:11:38.744 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:38.743+0000 7f034a400640 1 -- start start 2024-05-28T06:11:38.745 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:38.744+0000 7f034a400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f03440ccc30 0x7f034406dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:38.745 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:38.744+0000 7f034a400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f03440d0460 0x7f03440d08f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:38.745 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:38.744+0000 7f034a400640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03440cd1a0 con 0x7f034415c6b0 2024-05-28T06:11:38.745 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:38.744+0000 7f034a400640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f03440cd480 con 0x7f03440ccc30 2024-05-28T06:11:38.746 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:38.744+0000 7f034a400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f03440cd760 con 0x7f03440d0460 2024-05-28T06:11:38.746 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:38.745+0000 7f0349a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f03440d0460 0x7f03440d08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:38.746 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:38.745+0000 7f033be00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f03440ccc30 0x7f034406dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:38.746 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:38.745+0000 7f033be00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f03440ccc30 0x7f034406dc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.96:59526/0 (socket says 172.21.15.96:59526) 2024-05-28T06:11:38.746 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:38.745+0000 7f033be00640 1 -- 172.21.15.96:0/1967862077 learned_addr learned my addr 172.21.15.96:0/1967862077 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:11:38.746 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:38.745+0000 7f0349a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f03440d0460 0x7f03440d08f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.96:49392/0 (socket says 172.21.15.96:49392) 2024-05-28T06:11:38.747 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:38.746+0000 7f033be00640 1 -- 172.21.15.96:0/1967862077 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f03440d0460 msgr2=0x7f03440d08f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:11:38.747 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:38.746+0000 7f033be00640 1 --2- 172.21.15.96:0/1967862077 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f03440d0460 0x7f03440d08f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:38.747 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:38.746+0000 7f033be00640 1 -- 172.21.15.96:0/1967862077 >> v1:172.21.15.123:6789/0 conn(0x7f034415c6b0 legacy=0x7f03440b3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:38.747 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:38.746+0000 7f033be00640 1 -- 172.21.15.96:0/1967862077 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03440cda40 con 0x7f03440ccc30 2024-05-28T06:11:38.747 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:38.746+0000 7f0349a00640 1 --2- 172.21.15.96:0/1967862077 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f03440d0460 0x7f03440d08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-28T06:11:38.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:38.747+0000 7f033be00640 1 --2- 172.21.15.96:0/1967862077 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f03440ccc30 0x7f034406dc90 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f032c0098a0 tx=0x7f032c02a2a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ee8519468e6c28ad server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:11:38.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:38.747+0000 7f0343e00640 1 -- 172.21.15.96:0/1967862077 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f032c037070 con 0x7f03440ccc30 2024-05-28T06:11:38.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:38.747+0000 7f0343e00640 1 -- 172.21.15.96:0/1967862077 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f032c032440 con 0x7f03440ccc30 2024-05-28T06:11:38.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:38.747+0000 7f0343e00640 1 -- 172.21.15.96:0/1967862077 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f032c032810 con 0x7f03440ccc30 2024-05-28T06:11:38.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:38.748+0000 7f034a400640 1 -- 172.21.15.96:0/1967862077 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f03440ccc30 msgr2=0x7f034406dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:38.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:38.748+0000 7f034a400640 1 --2- 172.21.15.96:0/1967862077 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f03440ccc30 0x7f034406dc90 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f032c0098a0 tx=0x7f032c02a2a0 comp rx=0 tx=0).stop 2024-05-28T06:11:38.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:38.748+0000 7f034a400640 1 -- 172.21.15.96:0/1967862077 shutdown_connections 2024-05-28T06:11:38.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:38.748+0000 7f034a400640 1 --2- 172.21.15.96:0/1967862077 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f03440d0460 0x7f03440d08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:38.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:38.748+0000 7f034a400640 1 --2- 172.21.15.96:0/1967862077 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f03440ccc30 0x7f034406dc90 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:38.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:38.748+0000 7f034a400640 1 -- 172.21.15.96:0/1967862077 >> 172.21.15.96:0/1967862077 conn(0x7f034406bae0 msgr2=0x7f034406bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:11:38.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:38.748+0000 7f034a400640 1 -- 172.21.15.96:0/1967862077 shutdown_connections 2024-05-28T06:11:38.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:38.748+0000 7f034a400640 1 -- 172.21.15.96:0/1967862077 wait complete. 2024-05-28T06:11:38.750 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:38.750+0000 7f034a400640 1 Processor -- start 2024-05-28T06:11:38.750 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:38.750+0000 7f034a400640 1 -- start start 2024-05-28T06:11:38.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:38.751+0000 7f034a400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f03440ccc30 0x7f03441e5d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:38.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:38.751+0000 7f034a400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f03440d0460 0x7f03440b1820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:38.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:38.751+0000 7f034a400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f034415c6b0 0x7f0344142130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:38.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:38.751+0000 7f034a400640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f034406e350 con 0x7f034415c6b0 2024-05-28T06:11:38.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:38.751+0000 7f034a400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f034406e1d0 con 0x7f03440d0460 2024-05-28T06:11:38.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:38.751+0000 7f034a400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f03440cbf30 con 0x7f03440ccc30 2024-05-28T06:11:38.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:38.751+0000 7f0349a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f034415c6b0 0x7f0344142130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:38.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:38.751+0000 7f0349a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f034415c6b0 0x7f0344142130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48734/0 (socket says 172.21.15.96:48734) 2024-05-28T06:11:38.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:38.751+0000 7f0349a00640 1 -- 172.21.15.96:0/3039363131 learned_addr learned my addr 172.21.15.96:0/3039363131 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:11:38.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:38.752+0000 7f0349000640 1 --2- 172.21.15.96:0/3039363131 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f03440ccc30 0x7f03441e5d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:38.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:38.752+0000 7f033be00640 1 --2- 172.21.15.96:0/3039363131 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f03440d0460 0x7f03440b1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:38.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:38.752+0000 7f0349a00640 1 -- 172.21.15.96:0/3039363131 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f03440ccc30 msgr2=0x7f03441e5d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:38.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:38.752+0000 7f0349a00640 1 --2- 172.21.15.96:0/3039363131 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f03440ccc30 0x7f03441e5d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:38.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:38.752+0000 7f0349a00640 1 -- 172.21.15.96:0/3039363131 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f03440d0460 msgr2=0x7f03440b1820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:38.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:38.752+0000 7f0349a00640 1 --2- 172.21.15.96:0/3039363131 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f03440d0460 0x7f03440b1820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:38.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:38.752+0000 7f0349a00640 1 -- 172.21.15.96:0/3039363131 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03440b22e0 con 0x7f034415c6b0 2024-05-28T06:11:38.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:38.752+0000 7f033be00640 1 --2- 172.21.15.96:0/3039363131 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f03440d0460 0x7f03440b1820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:11:38.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:38.752+0000 7f0349a00640 1 --2- 172.21.15.96:0/3039363131 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f034415c6b0 0x7f0344142130 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f033c00efc0 tx=0x7f033c009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:11:38.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:38.752+0000 7f0341600640 1 -- 172.21.15.96:0/3039363131 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f033c018030 con 0x7f034415c6b0 2024-05-28T06:11:38.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:38.752+0000 7f0341600640 1 -- 172.21.15.96:0/3039363131 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f033c00cde0 con 0x7f034415c6b0 2024-05-28T06:11:38.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:38.752+0000 7f0341600640 1 -- 172.21.15.96:0/3039363131 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f033c003bb0 con 0x7f034415c6b0 2024-05-28T06:11:38.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:38.752+0000 7f034a400640 1 -- 172.21.15.96:0/3039363131 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03440b25c0 con 0x7f034415c6b0 2024-05-28T06:11:38.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:38.752+0000 7f034a400640 1 -- 172.21.15.96:0/3039363131 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0344143250 con 0x7f034415c6b0 2024-05-28T06:11:38.755 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:38.753+0000 7f034a400640 1 -- 172.21.15.96:0/3039363131 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0344142d60 con 0x7f034415c6b0 2024-05-28T06:11:38.755 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:38.755+0000 7f0341600640 1 -- 172.21.15.96:0/3039363131 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 15) ==== 91652+0+0 (secure 0 0 0) 0x7f033c005d10 con 0x7f034415c6b0 2024-05-28T06:11:38.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:38.756+0000 7f0341600640 1 --2- 172.21.15.96:0/3039363131 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f032006cfb0 0x7f032006f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:38.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:38.756+0000 7f0341600640 1 -- 172.21.15.96:0/3039363131 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5795+0+0 (secure 0 0 0) 0x7f033c08d470 con 0x7f034415c6b0 2024-05-28T06:11:38.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:38.756+0000 7f0349000640 1 --2- 172.21.15.96:0/3039363131 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f032006cfb0 0x7f032006f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:38.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:38.757+0000 7f0349000640 1 --2- 172.21.15.96:0/3039363131 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f032006cfb0 0x7f032006f470 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f0334007fd0 tx=0x7f0334009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:11:38.761 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:38.760+0000 7f0341600640 1 -- 172.21.15.96:0/3039363131 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f033c022070 con 0x7f034415c6b0 2024-05-28T06:11:39.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:39.030+0000 7f034a400640 1 -- 172.21.15.96:0/3039363131 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f03440689e0 con 0x7f034415c6b0 2024-05-28T06:11:39.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:39.036+0000 7f0341600640 1 -- 172.21.15.96:0/3039363131 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+2123 (secure 0 0 0) 0x7f033c010040 con 0x7f034415c6b0 2024-05-28T06:11:39.036 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:11:39.036 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:11:39.037 INFO:teuthology.orchestra.run.smithi096.stdout: "fsid": "e4a831b5-e6d1-47a7-96cb-bf85154ee0d2", 2024-05-28T06:11:39.037 INFO:teuthology.orchestra.run.smithi096.stdout: "health": { 2024-05-28T06:11:39.037 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "HEALTH_OK", 2024-05-28T06:11:39.037 INFO:teuthology.orchestra.run.smithi096.stdout: "checks": {}, 2024-05-28T06:11:39.037 INFO:teuthology.orchestra.run.smithi096.stdout: "mutes": [] 2024-05-28T06:11:39.037 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:11:39.037 INFO:teuthology.orchestra.run.smithi096.stdout: "election_epoch": 8, 2024-05-28T06:11:39.037 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum": [ 2024-05-28T06:11:39.037 INFO:teuthology.orchestra.run.smithi096.stdout: 0, 2024-05-28T06:11:39.037 INFO:teuthology.orchestra.run.smithi096.stdout: 1, 2024-05-28T06:11:39.037 INFO:teuthology.orchestra.run.smithi096.stdout: 2 2024-05-28T06:11:39.037 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:11:39.037 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_names": [ 2024-05-28T06:11:39.037 INFO:teuthology.orchestra.run.smithi096.stdout: "a", 2024-05-28T06:11:39.038 INFO:teuthology.orchestra.run.smithi096.stdout: "b", 2024-05-28T06:11:39.038 INFO:teuthology.orchestra.run.smithi096.stdout: "c" 2024-05-28T06:11:39.038 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:11:39.038 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_age": 472, 2024-05-28T06:11:39.038 INFO:teuthology.orchestra.run.smithi096.stdout: "monmap": { 2024-05-28T06:11:39.038 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 1, 2024-05-28T06:11:39.038 INFO:teuthology.orchestra.run.smithi096.stdout: "min_mon_release_name": "squid", 2024-05-28T06:11:39.038 INFO:teuthology.orchestra.run.smithi096.stdout: "num_mons": 3 2024-05-28T06:11:39.038 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:11:39.038 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap": { 2024-05-28T06:11:39.038 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 47, 2024-05-28T06:11:39.038 INFO:teuthology.orchestra.run.smithi096.stdout: "num_osds": 8, 2024-05-28T06:11:39.039 INFO:teuthology.orchestra.run.smithi096.stdout: "num_up_osds": 8, 2024-05-28T06:11:39.039 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_up_since": 1716876238, 2024-05-28T06:11:39.039 INFO:teuthology.orchestra.run.smithi096.stdout: "num_in_osds": 8, 2024-05-28T06:11:39.039 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_in_since": 1716876229, 2024-05-28T06:11:39.039 INFO:teuthology.orchestra.run.smithi096.stdout: "num_remapped_pgs": 0 2024-05-28T06:11:39.039 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:11:39.039 INFO:teuthology.orchestra.run.smithi096.stdout: "pgmap": { 2024-05-28T06:11:39.039 INFO:teuthology.orchestra.run.smithi096.stdout: "pgs_by_state": [ 2024-05-28T06:11:39.039 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:11:39.039 INFO:teuthology.orchestra.run.smithi096.stdout: "state_name": "active+clean", 2024-05-28T06:11:39.039 INFO:teuthology.orchestra.run.smithi096.stdout: "count": 97 2024-05-28T06:11:39.039 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:11:39.039 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:11:39.040 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pgs": 97, 2024-05-28T06:11:39.040 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pools": 3, 2024-05-28T06:11:39.040 INFO:teuthology.orchestra.run.smithi096.stdout: "num_objects": 4176, 2024-05-28T06:11:39.040 INFO:teuthology.orchestra.run.smithi096.stdout: "data_bytes": 17383157191, 2024-05-28T06:11:39.040 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_used": 35149766656, 2024-05-28T06:11:39.040 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_avail": 737944346624, 2024-05-28T06:11:39.040 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_total": 773094113280, 2024-05-28T06:11:39.040 INFO:teuthology.orchestra.run.smithi096.stdout: "read_bytes_sec": 124733104, 2024-05-28T06:11:39.040 INFO:teuthology.orchestra.run.smithi096.stdout: "write_bytes_sec": 127553630, 2024-05-28T06:11:39.040 INFO:teuthology.orchestra.run.smithi096.stdout: "read_op_per_sec": 31, 2024-05-28T06:11:39.040 INFO:teuthology.orchestra.run.smithi096.stdout: "write_op_per_sec": 39 2024-05-28T06:11:39.040 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:11:39.040 INFO:teuthology.orchestra.run.smithi096.stdout: "fsmap": { 2024-05-28T06:11:39.040 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 26, 2024-05-28T06:11:39.041 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 3, 2024-05-28T06:11:39.041 INFO:teuthology.orchestra.run.smithi096.stdout: "up": 1, 2024-05-28T06:11:39.041 INFO:teuthology.orchestra.run.smithi096.stdout: "in": 1, 2024-05-28T06:11:39.041 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 1, 2024-05-28T06:11:39.041 INFO:teuthology.orchestra.run.smithi096.stdout: "by_rank": [ 2024-05-28T06:11:39.041 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:11:39.041 INFO:teuthology.orchestra.run.smithi096.stdout: "filesystem_id": 3, 2024-05-28T06:11:39.041 INFO:teuthology.orchestra.run.smithi096.stdout: "rank": 0, 2024-05-28T06:11:39.041 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "a", 2024-05-28T06:11:39.041 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "up:active", 2024-05-28T06:11:39.041 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 5859 2024-05-28T06:11:39.041 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:11:39.041 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:11:39.041 INFO:teuthology.orchestra.run.smithi096.stdout: "up:standby": 3 2024-05-28T06:11:39.042 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:11:39.042 INFO:teuthology.orchestra.run.smithi096.stdout: "mgrmap": { 2024-05-28T06:11:39.042 INFO:teuthology.orchestra.run.smithi096.stdout: "available": true, 2024-05-28T06:11:39.042 INFO:teuthology.orchestra.run.smithi096.stdout: "num_standbys": 1, 2024-05-28T06:11:39.042 INFO:teuthology.orchestra.run.smithi096.stdout: "modules": [ 2024-05-28T06:11:39.042 INFO:teuthology.orchestra.run.smithi096.stdout: "iostat", 2024-05-28T06:11:39.042 INFO:teuthology.orchestra.run.smithi096.stdout: "nfs", 2024-05-28T06:11:39.042 INFO:teuthology.orchestra.run.smithi096.stdout: "restful" 2024-05-28T06:11:39.042 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:11:39.042 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:11:39.042 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:11:39.042 INFO:teuthology.orchestra.run.smithi096.stdout: "servicemap": { 2024-05-28T06:11:39.042 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 9, 2024-05-28T06:11:39.042 INFO:teuthology.orchestra.run.smithi096.stdout: "modified": "2024-05-28T06:10:01.618312+0000", 2024-05-28T06:11:39.043 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:11:39.043 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:11:39.043 INFO:teuthology.orchestra.run.smithi096.stdout: "progress_events": { 2024-05-28T06:11:39.043 INFO:teuthology.orchestra.run.smithi096.stdout: "72116a8d-650b-4cfc-99af-ee8db5167322": { 2024-05-28T06:11:39.043 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "All cloning finished, waiting 20.0 seconds before exiting (20s)\n [============================] ", 2024-05-28T06:11:39.043 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": 1, 2024-05-28T06:11:39.043 INFO:teuthology.orchestra.run.smithi096.stdout: "add_to_ceph_s": true 2024-05-28T06:11:39.043 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:11:39.043 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:11:39.043 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:11:39.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:39.038+0000 7f034a400640 1 -- 172.21.15.96:0/3039363131 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f032006cfb0 msgr2=0x7f032006f470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:39.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:39.038+0000 7f034a400640 1 --2- 172.21.15.96:0/3039363131 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f032006cfb0 0x7f032006f470 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f0334007fd0 tx=0x7f0334009040 comp rx=0 tx=0).stop 2024-05-28T06:11:39.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:39.038+0000 7f034a400640 1 -- 172.21.15.96:0/3039363131 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f034415c6b0 msgr2=0x7f0344142130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:39.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:39.038+0000 7f034a400640 1 --2- 172.21.15.96:0/3039363131 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f034415c6b0 0x7f0344142130 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f033c00efc0 tx=0x7f033c009f90 comp rx=0 tx=0).stop 2024-05-28T06:11:39.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:39.038+0000 7f034a400640 1 -- 172.21.15.96:0/3039363131 shutdown_connections 2024-05-28T06:11:39.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:39.038+0000 7f034a400640 1 --2- 172.21.15.96:0/3039363131 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f032006cfb0 0x7f032006f470 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:39.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:39.038+0000 7f034a400640 1 --2- 172.21.15.96:0/3039363131 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f034415c6b0 0x7f0344142130 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:39.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:39.038+0000 7f034a400640 1 --2- 172.21.15.96:0/3039363131 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f03440d0460 0x7f03440b1820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:39.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:39.038+0000 7f034a400640 1 --2- 172.21.15.96:0/3039363131 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f03440ccc30 0x7f03441e5d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:39.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:39.038+0000 7f034a400640 1 -- 172.21.15.96:0/3039363131 >> 172.21.15.96:0/3039363131 conn(0x7f034406bae0 msgr2=0x7f034406bec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:11:39.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:39.039+0000 7f034a400640 1 -- 172.21.15.96:0/3039363131 shutdown_connections 2024-05-28T06:11:39.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:39.039+0000 7f034a400640 1 -- 172.21.15.96:0/3039363131 wait complete. 2024-05-28T06:11:44.057 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-05-28T06:11:44.268 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:44.267+0000 7f0785600640 1 Processor -- start 2024-05-28T06:11:44.268 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:44.268+0000 7f0785600640 1 -- start start 2024-05-28T06:11:44.269 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:44.269+0000 7f0785600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f078006b900 0x7f078006bce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:44.269 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:44.269+0000 7f0785600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f07800cba20 0x7f078006f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:44.270 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:44.269+0000 7f0785600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07800d05f0 con 0x7f078015ed60 2024-05-28T06:11:44.270 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:44.269+0000 7f0785600640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f078006c220 con 0x7f07800cba20 2024-05-28T06:11:44.270 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:44.269+0000 7f0785600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f07800d0890 con 0x7f078006b900 2024-05-28T06:11:44.270 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:44.269+0000 7f077f400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f07800cba20 0x7f078006f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:44.270 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:44.269+0000 7f077f400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f07800cba20 0x7f078006f7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:42836/0 (socket says 172.21.15.96:42836) 2024-05-28T06:11:44.270 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:44.269+0000 7f077f400640 1 -- 172.21.15.96:0/211596441 learned_addr learned my addr 172.21.15.96:0/211596441 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:11:44.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:44.269+0000 7f077ea00640 1 --2- 172.21.15.96:0/211596441 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f078006b900 0x7f078006bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:44.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:44.270+0000 7f077f400640 1 -- 172.21.15.96:0/211596441 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f078006b900 msgr2=0x7f078006bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:11:44.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:44.270+0000 7f077f400640 1 --2- 172.21.15.96:0/211596441 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f078006b900 0x7f078006bce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:44.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:44.270+0000 7f077f400640 1 -- 172.21.15.96:0/211596441 >> v1:172.21.15.123:6789/0 conn(0x7f078015ed60 legacy=0x7f078006f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:11:44.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:44.270+0000 7f077f400640 1 -- 172.21.15.96:0/211596441 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07800d0b70 con 0x7f07800cba20 2024-05-28T06:11:44.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:44.271+0000 7f077f400640 1 --2- 172.21.15.96:0/211596441 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f07800cba20 0x7f078006f7f0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f0778009870 tx=0x7f077802a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2a01f6df260d227 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:11:44.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:44.271+0000 7f077d600640 1 -- 172.21.15.96:0/211596441 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0778037070 con 0x7f07800cba20 2024-05-28T06:11:44.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:44.271+0000 7f077d600640 1 -- 172.21.15.96:0/211596441 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0778032440 con 0x7f07800cba20 2024-05-28T06:11:44.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:44.271+0000 7f077d600640 1 -- 172.21.15.96:0/211596441 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0778032810 con 0x7f07800cba20 2024-05-28T06:11:44.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:44.271+0000 7f0785600640 1 -- 172.21.15.96:0/211596441 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f07800cba20 msgr2=0x7f078006f7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:44.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:44.271+0000 7f0785600640 1 --2- 172.21.15.96:0/211596441 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f07800cba20 0x7f078006f7f0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f0778009870 tx=0x7f077802a180 comp rx=0 tx=0).stop 2024-05-28T06:11:44.273 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:44.272+0000 7f0785600640 1 -- 172.21.15.96:0/211596441 shutdown_connections 2024-05-28T06:11:44.273 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:44.272+0000 7f0785600640 1 --2- 172.21.15.96:0/211596441 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f07800cba20 0x7f078006f7f0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:44.273 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:44.272+0000 7f0785600640 1 --2- 172.21.15.96:0/211596441 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f078006b900 0x7f078006bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:44.273 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:44.272+0000 7f0785600640 1 -- 172.21.15.96:0/211596441 >> 172.21.15.96:0/211596441 conn(0x7f078009aa30 msgr2=0x7f07800b2f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:11:44.273 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:44.272+0000 7f0785600640 1 -- 172.21.15.96:0/211596441 shutdown_connections 2024-05-28T06:11:44.273 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:44.272+0000 7f0785600640 1 -- 172.21.15.96:0/211596441 wait complete. 2024-05-28T06:11:44.274 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:44.273+0000 7f0785600640 1 Processor -- start 2024-05-28T06:11:44.274 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:44.273+0000 7f0785600640 1 -- start start 2024-05-28T06:11:44.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:44.274+0000 7f0785600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f078006b900 0x7f078014c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:44.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:44.274+0000 7f0785600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f07800cba20 0x7f0780161640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:44.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:44.274+0000 7f0785600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f078015ed60 0x7f07800c7440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:44.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:44.274+0000 7f0785600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f07800d1130 con 0x7f07800cba20 2024-05-28T06:11:44.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:44.274+0000 7f0785600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f07800b3810 con 0x7f078006b900 2024-05-28T06:11:44.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:44.274+0000 7f0785600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f078006fd30 con 0x7f078015ed60 2024-05-28T06:11:44.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:44.275+0000 7f077e000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f07800cba20 0x7f0780161640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:44.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:44.275+0000 7f077e000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f07800cba20 0x7f0780161640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:42838/0 (socket says 172.21.15.96:42838) 2024-05-28T06:11:44.276 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:44.275+0000 7f077e000640 1 -- 172.21.15.96:0/1525382063 learned_addr learned my addr 172.21.15.96:0/1525382063 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:11:44.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:44.275+0000 7f077ea00640 1 --2- 172.21.15.96:0/1525382063 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f078006b900 0x7f078014c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:44.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:44.275+0000 7f077f400640 1 --2- 172.21.15.96:0/1525382063 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f078015ed60 0x7f07800c7440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:44.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:44.275+0000 7f077e000640 1 -- 172.21.15.96:0/1525382063 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f078015ed60 msgr2=0x7f07800c7440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:44.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:44.275+0000 7f077e000640 1 --2- 172.21.15.96:0/1525382063 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f078015ed60 0x7f07800c7440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:44.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:44.275+0000 7f077e000640 1 -- 172.21.15.96:0/1525382063 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f078006b900 msgr2=0x7f078014c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:44.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:44.275+0000 7f077e000640 1 --2- 172.21.15.96:0/1525382063 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f078006b900 0x7f078014c5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:44.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:44.275+0000 7f077e000640 1 -- 172.21.15.96:0/1525382063 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07800b6a60 con 0x7f07800cba20 2024-05-28T06:11:44.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:44.275+0000 7f077e000640 1 --2- 172.21.15.96:0/1525382063 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f07800cba20 0x7f0780161640 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f076800c7c0 tx=0x7f076800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:11:44.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:44.276+0000 7f0776a00640 1 -- 172.21.15.96:0/1525382063 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0768016020 con 0x7f07800cba20 2024-05-28T06:11:44.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:44.276+0000 7f0776a00640 1 -- 172.21.15.96:0/1525382063 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f076800aa50 con 0x7f07800cba20 2024-05-28T06:11:44.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:44.276+0000 7f0785600640 1 -- 172.21.15.96:0/1525382063 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07800b6fb0 con 0x7f07800cba20 2024-05-28T06:11:44.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:44.276+0000 7f0776a00640 1 -- 172.21.15.96:0/1525382063 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f076800ae40 con 0x7f07800cba20 2024-05-28T06:11:44.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:44.276+0000 7f0785600640 1 -- 172.21.15.96:0/1525382063 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07800c6960 con 0x7f07800cba20 2024-05-28T06:11:44.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:44.277+0000 7f0785600640 1 -- 172.21.15.96:0/1525382063 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07800689e0 con 0x7f07800cba20 2024-05-28T06:11:44.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:44.278+0000 7f0776a00640 1 -- 172.21.15.96:0/1525382063 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 15) ==== 91652+0+0 (secure 0 0 0) 0x7f0768019020 con 0x7f07800cba20 2024-05-28T06:11:44.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:44.279+0000 7f0776a00640 1 --2- 172.21.15.96:0/1525382063 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f075c06cfb0 0x7f075c06f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:44.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:44.279+0000 7f0776a00640 1 -- 172.21.15.96:0/1525382063 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5795+0+0 (secure 0 0 0) 0x7f076808c970 con 0x7f07800cba20 2024-05-28T06:11:44.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:44.279+0000 7f077ea00640 1 --2- 172.21.15.96:0/1525382063 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f075c06cfb0 0x7f075c06f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:44.281 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:44.281+0000 7f077ea00640 1 --2- 172.21.15.96:0/1525382063 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f075c06cfb0 0x7f075c06f470 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f077000b7c0 tx=0x7f0770008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:11:44.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:44.283+0000 7f0776a00640 1 -- 172.21.15.96:0/1525382063 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0768020030 con 0x7f07800cba20 2024-05-28T06:11:44.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:44.556+0000 7f0785600640 1 -- 172.21.15.96:0/1525382063 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f07800d15e0 con 0x7f07800cba20 2024-05-28T06:11:44.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:44.557+0000 7f0776a00640 1 -- 172.21.15.96:0/1525382063 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+2109 (secure 0 0 0) 0x7f076805a250 con 0x7f07800cba20 2024-05-28T06:11:44.558 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:11:44.558 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:11:44.558 INFO:teuthology.orchestra.run.smithi096.stdout: "fsid": "e4a831b5-e6d1-47a7-96cb-bf85154ee0d2", 2024-05-28T06:11:44.558 INFO:teuthology.orchestra.run.smithi096.stdout: "health": { 2024-05-28T06:11:44.558 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "HEALTH_OK", 2024-05-28T06:11:44.559 INFO:teuthology.orchestra.run.smithi096.stdout: "checks": {}, 2024-05-28T06:11:44.559 INFO:teuthology.orchestra.run.smithi096.stdout: "mutes": [] 2024-05-28T06:11:44.559 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:11:44.559 INFO:teuthology.orchestra.run.smithi096.stdout: "election_epoch": 8, 2024-05-28T06:11:44.559 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum": [ 2024-05-28T06:11:44.559 INFO:teuthology.orchestra.run.smithi096.stdout: 0, 2024-05-28T06:11:44.559 INFO:teuthology.orchestra.run.smithi096.stdout: 1, 2024-05-28T06:11:44.559 INFO:teuthology.orchestra.run.smithi096.stdout: 2 2024-05-28T06:11:44.559 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:11:44.559 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_names": [ 2024-05-28T06:11:44.559 INFO:teuthology.orchestra.run.smithi096.stdout: "a", 2024-05-28T06:11:44.559 INFO:teuthology.orchestra.run.smithi096.stdout: "b", 2024-05-28T06:11:44.559 INFO:teuthology.orchestra.run.smithi096.stdout: "c" 2024-05-28T06:11:44.560 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:11:44.560 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_age": 478, 2024-05-28T06:11:44.560 INFO:teuthology.orchestra.run.smithi096.stdout: "monmap": { 2024-05-28T06:11:44.560 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 1, 2024-05-28T06:11:44.560 INFO:teuthology.orchestra.run.smithi096.stdout: "min_mon_release_name": "squid", 2024-05-28T06:11:44.560 INFO:teuthology.orchestra.run.smithi096.stdout: "num_mons": 3 2024-05-28T06:11:44.560 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:11:44.560 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap": { 2024-05-28T06:11:44.560 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 47, 2024-05-28T06:11:44.560 INFO:teuthology.orchestra.run.smithi096.stdout: "num_osds": 8, 2024-05-28T06:11:44.560 INFO:teuthology.orchestra.run.smithi096.stdout: "num_up_osds": 8, 2024-05-28T06:11:44.561 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_up_since": 1716876238, 2024-05-28T06:11:44.561 INFO:teuthology.orchestra.run.smithi096.stdout: "num_in_osds": 8, 2024-05-28T06:11:44.561 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_in_since": 1716876229, 2024-05-28T06:11:44.561 INFO:teuthology.orchestra.run.smithi096.stdout: "num_remapped_pgs": 0 2024-05-28T06:11:44.561 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:11:44.561 INFO:teuthology.orchestra.run.smithi096.stdout: "pgmap": { 2024-05-28T06:11:44.561 INFO:teuthology.orchestra.run.smithi096.stdout: "pgs_by_state": [ 2024-05-28T06:11:44.561 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:11:44.561 INFO:teuthology.orchestra.run.smithi096.stdout: "state_name": "active+clean", 2024-05-28T06:11:44.561 INFO:teuthology.orchestra.run.smithi096.stdout: "count": 97 2024-05-28T06:11:44.561 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:11:44.561 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:11:44.561 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pgs": 97, 2024-05-28T06:11:44.561 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pools": 3, 2024-05-28T06:11:44.562 INFO:teuthology.orchestra.run.smithi096.stdout: "num_objects": 4176, 2024-05-28T06:11:44.562 INFO:teuthology.orchestra.run.smithi096.stdout: "data_bytes": 17383166762, 2024-05-28T06:11:44.562 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_used": 35149856768, 2024-05-28T06:11:44.562 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_avail": 737944256512, 2024-05-28T06:11:44.562 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_total": 773094113280, 2024-05-28T06:11:44.562 INFO:teuthology.orchestra.run.smithi096.stdout: "read_bytes_sec": 85, 2024-05-28T06:11:44.562 INFO:teuthology.orchestra.run.smithi096.stdout: "write_bytes_sec": 4947, 2024-05-28T06:11:44.562 INFO:teuthology.orchestra.run.smithi096.stdout: "read_op_per_sec": 0, 2024-05-28T06:11:44.562 INFO:teuthology.orchestra.run.smithi096.stdout: "write_op_per_sec": 1 2024-05-28T06:11:44.562 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:11:44.562 INFO:teuthology.orchestra.run.smithi096.stdout: "fsmap": { 2024-05-28T06:11:44.562 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 26, 2024-05-28T06:11:44.562 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 3, 2024-05-28T06:11:44.563 INFO:teuthology.orchestra.run.smithi096.stdout: "up": 1, 2024-05-28T06:11:44.563 INFO:teuthology.orchestra.run.smithi096.stdout: "in": 1, 2024-05-28T06:11:44.563 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 1, 2024-05-28T06:11:44.563 INFO:teuthology.orchestra.run.smithi096.stdout: "by_rank": [ 2024-05-28T06:11:44.563 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:11:44.563 INFO:teuthology.orchestra.run.smithi096.stdout: "filesystem_id": 3, 2024-05-28T06:11:44.563 INFO:teuthology.orchestra.run.smithi096.stdout: "rank": 0, 2024-05-28T06:11:44.563 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "a", 2024-05-28T06:11:44.563 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "up:active", 2024-05-28T06:11:44.563 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 5859 2024-05-28T06:11:44.563 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:11:44.563 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:11:44.563 INFO:teuthology.orchestra.run.smithi096.stdout: "up:standby": 3 2024-05-28T06:11:44.563 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:11:44.564 INFO:teuthology.orchestra.run.smithi096.stdout: "mgrmap": { 2024-05-28T06:11:44.564 INFO:teuthology.orchestra.run.smithi096.stdout: "available": true, 2024-05-28T06:11:44.564 INFO:teuthology.orchestra.run.smithi096.stdout: "num_standbys": 1, 2024-05-28T06:11:44.564 INFO:teuthology.orchestra.run.smithi096.stdout: "modules": [ 2024-05-28T06:11:44.564 INFO:teuthology.orchestra.run.smithi096.stdout: "iostat", 2024-05-28T06:11:44.564 INFO:teuthology.orchestra.run.smithi096.stdout: "nfs", 2024-05-28T06:11:44.564 INFO:teuthology.orchestra.run.smithi096.stdout: "restful" 2024-05-28T06:11:44.564 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:11:44.564 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:11:44.564 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:11:44.564 INFO:teuthology.orchestra.run.smithi096.stdout: "servicemap": { 2024-05-28T06:11:44.564 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 9, 2024-05-28T06:11:44.564 INFO:teuthology.orchestra.run.smithi096.stdout: "modified": "2024-05-28T06:10:01.618312+0000", 2024-05-28T06:11:44.564 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:11:44.564 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:11:44.565 INFO:teuthology.orchestra.run.smithi096.stdout: "progress_events": { 2024-05-28T06:11:44.565 INFO:teuthology.orchestra.run.smithi096.stdout: "72116a8d-650b-4cfc-99af-ee8db5167322": { 2024-05-28T06:11:44.565 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "All cloning finished, waiting 14.0 seconds before exiting (26s)\n [============================] ", 2024-05-28T06:11:44.565 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": 1, 2024-05-28T06:11:44.565 INFO:teuthology.orchestra.run.smithi096.stdout: "add_to_ceph_s": true 2024-05-28T06:11:44.565 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:11:44.565 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:11:44.565 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:11:44.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:44.560+0000 7f0785600640 1 -- 172.21.15.96:0/1525382063 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f075c06cfb0 msgr2=0x7f075c06f470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:44.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:44.560+0000 7f0785600640 1 --2- 172.21.15.96:0/1525382063 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f075c06cfb0 0x7f075c06f470 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f077000b7c0 tx=0x7f0770008040 comp rx=0 tx=0).stop 2024-05-28T06:11:44.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:44.560+0000 7f0785600640 1 -- 172.21.15.96:0/1525382063 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f07800cba20 msgr2=0x7f0780161640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:44.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:44.560+0000 7f0785600640 1 --2- 172.21.15.96:0/1525382063 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f07800cba20 0x7f0780161640 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f076800c7c0 tx=0x7f076800cc80 comp rx=0 tx=0).stop 2024-05-28T06:11:44.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:44.560+0000 7f0785600640 1 -- 172.21.15.96:0/1525382063 shutdown_connections 2024-05-28T06:11:44.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:44.560+0000 7f0785600640 1 --2- 172.21.15.96:0/1525382063 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f075c06cfb0 0x7f075c06f470 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:44.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:44.560+0000 7f0785600640 1 --2- 172.21.15.96:0/1525382063 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f078015ed60 0x7f07800c7440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:44.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:44.560+0000 7f0785600640 1 --2- 172.21.15.96:0/1525382063 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f07800cba20 0x7f0780161640 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:44.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:44.560+0000 7f0785600640 1 --2- 172.21.15.96:0/1525382063 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f078006b900 0x7f078014c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:44.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:44.560+0000 7f0785600640 1 -- 172.21.15.96:0/1525382063 >> 172.21.15.96:0/1525382063 conn(0x7f078009aa30 msgr2=0x7f07800b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:11:44.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:44.560+0000 7f0785600640 1 -- 172.21.15.96:0/1525382063 shutdown_connections 2024-05-28T06:11:44.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:44.560+0000 7f0785600640 1 -- 172.21.15.96:0/1525382063 wait complete. 2024-05-28T06:11:49.580 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-05-28T06:11:49.787 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:49.785+0000 7fa7d6e00640 1 Processor -- start 2024-05-28T06:11:49.787 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:49.786+0000 7fa7d6e00640 1 -- start start 2024-05-28T06:11:49.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:49.787+0000 7fa7d6e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fa7d015baa0 0x7fa7d00d0970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:49.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:49.787+0000 7fa7d6e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fa7d006f390 0x7fa7d006f800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:49.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:49.787+0000 7fa7d6e00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7d006e3d0 con 0x7fa7d0155a00 2024-05-28T06:11:49.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:49.787+0000 7fa7d6e00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fa7d006fd40 con 0x7fa7d006f390 2024-05-28T06:11:49.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:49.787+0000 7fa7d6e00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fa7d00ccf20 con 0x7fa7d015baa0 2024-05-28T06:11:49.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:49.787+0000 7fa7d4e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fa7d006f390 0x7fa7d006f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:49.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:49.788+0000 7fa7d4e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fa7d006f390 0x7fa7d006f800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35354/0 (socket says 172.21.15.96:35354) 2024-05-28T06:11:49.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:49.788+0000 7fa7d4e00640 1 -- 172.21.15.96:0/1261994701 learned_addr learned my addr 172.21.15.96:0/1261994701 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:11:49.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:49.788+0000 7fa7c7400640 1 --2- 172.21.15.96:0/1261994701 >> v2:172.21.15.123:3300/0 conn(0x7fa7d015baa0 0x7fa7d00d0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:49.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:49.788+0000 7fa7d4e00640 1 -- 172.21.15.96:0/1261994701 >> v2:172.21.15.123:3300/0 conn(0x7fa7d015baa0 msgr2=0x7fa7d00d0970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:11:49.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:49.788+0000 7fa7d4e00640 1 --2- 172.21.15.96:0/1261994701 >> v2:172.21.15.123:3300/0 conn(0x7fa7d015baa0 0x7fa7d00d0970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:49.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:49.788+0000 7fa7d4e00640 1 -- 172.21.15.96:0/1261994701 >> v1:172.21.15.123:6789/0 conn(0x7fa7d0155a00 legacy=0x7fa7d006dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:49.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:49.788+0000 7fa7d4e00640 1 -- 172.21.15.96:0/1261994701 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7d015c0b0 con 0x7fa7d006f390 2024-05-28T06:11:49.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:49.790+0000 7fa7d4e00640 1 --2- 172.21.15.96:0/1261994701 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa7d006f390 0x7fa7d006f800 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fa7d00cfee0 tx=0x7fa7c802a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c78347c659b06e5f server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:11:49.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:49.790+0000 7fa7cf400640 1 -- 172.21.15.96:0/1261994701 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7c8036070 con 0x7fa7d006f390 2024-05-28T06:11:49.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:49.790+0000 7fa7cf400640 1 -- 172.21.15.96:0/1261994701 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7c8031600 con 0x7fa7d006f390 2024-05-28T06:11:49.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:49.790+0000 7fa7cf400640 1 -- 172.21.15.96:0/1261994701 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7c80319d0 con 0x7fa7d006f390 2024-05-28T06:11:49.793 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:49.791+0000 7fa7d6e00640 1 -- 172.21.15.96:0/1261994701 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa7d006f390 msgr2=0x7fa7d006f800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:49.793 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:49.791+0000 7fa7d6e00640 1 --2- 172.21.15.96:0/1261994701 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa7d006f390 0x7fa7d006f800 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fa7d00cfee0 tx=0x7fa7c802a730 comp rx=0 tx=0).stop 2024-05-28T06:11:49.793 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:49.791+0000 7fa7d6e00640 1 -- 172.21.15.96:0/1261994701 shutdown_connections 2024-05-28T06:11:49.793 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:49.791+0000 7fa7d6e00640 1 --2- 172.21.15.96:0/1261994701 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa7d006f390 0x7fa7d006f800 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:49.793 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:49.791+0000 7fa7d6e00640 1 --2- 172.21.15.96:0/1261994701 >> v2:172.21.15.123:3300/0 conn(0x7fa7d015baa0 0x7fa7d00d0970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:49.793 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:49.791+0000 7fa7d6e00640 1 -- 172.21.15.96:0/1261994701 >> 172.21.15.96:0/1261994701 conn(0x7fa7d0067ce0 msgr2=0x7fa7d006b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:11:49.793 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:49.791+0000 7fa7d6e00640 1 -- 172.21.15.96:0/1261994701 shutdown_connections 2024-05-28T06:11:49.793 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:49.791+0000 7fa7d6e00640 1 -- 172.21.15.96:0/1261994701 wait complete. 2024-05-28T06:11:49.794 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:49.793+0000 7fa7d6e00640 1 Processor -- start 2024-05-28T06:11:49.794 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:49.793+0000 7fa7d6e00640 1 -- start start 2024-05-28T06:11:49.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:49.794+0000 7fa7d6e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa7d006f390 0x7fa7d0150660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:49.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:49.794+0000 7fa7d6e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa7d0155a00 0x7fa7d01542a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:49.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:49.794+0000 7fa7d6e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa7d015baa0 0x7fa7d014f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:49.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:49.794+0000 7fa7d6e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa7d015c670 con 0x7fa7d0155a00 2024-05-28T06:11:49.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:49.794+0000 7fa7d6e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa7d015c950 con 0x7fa7d015baa0 2024-05-28T06:11:49.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:49.794+0000 7fa7d6e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fa7d015f090 con 0x7fa7d006f390 2024-05-28T06:11:49.797 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:49.794+0000 7fa7c7400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa7d0155a00 0x7fa7d01542a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:49.797 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:49.794+0000 7fa7c7400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa7d0155a00 0x7fa7d01542a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35366/0 (socket says 172.21.15.96:35366) 2024-05-28T06:11:49.797 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:49.794+0000 7fa7c7400640 1 -- 172.21.15.96:0/2365402561 learned_addr learned my addr 172.21.15.96:0/2365402561 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:11:49.797 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:49.794+0000 7fa7cfe00640 1 --2- 172.21.15.96:0/2365402561 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa7d006f390 0x7fa7d0150660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:49.797 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:49.794+0000 7fa7c7400640 1 -- 172.21.15.96:0/2365402561 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa7d006f390 msgr2=0x7fa7d0150660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:49.797 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:49.794+0000 7fa7c7400640 1 --2- 172.21.15.96:0/2365402561 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa7d006f390 0x7fa7d0150660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:49.797 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:49.794+0000 7fa7c7400640 1 -- 172.21.15.96:0/2365402561 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa7d015baa0 msgr2=0x7fa7d014f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:49.797 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:49.794+0000 7fa7d4e00640 1 --2- 172.21.15.96:0/2365402561 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa7d015baa0 0x7fa7d014f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:49.797 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:49.795+0000 7fa7c7400640 1 --2- 172.21.15.96:0/2365402561 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa7d015baa0 0x7fa7d014f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:49.797 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:49.795+0000 7fa7c7400640 1 -- 172.21.15.96:0/2365402561 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7d014b930 con 0x7fa7d0155a00 2024-05-28T06:11:49.797 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:49.795+0000 7fa7cfe00640 1 --2- 172.21.15.96:0/2365402561 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa7d006f390 0x7fa7d0150660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:11:49.797 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:49.795+0000 7fa7d4e00640 1 --2- 172.21.15.96:0/2365402561 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa7d015baa0 0x7fa7d014f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:11:49.797 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:49.795+0000 7fa7c7400640 1 --2- 172.21.15.96:0/2365402561 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa7d0155a00 0x7fa7d01542a0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fa7b800d5f0 tx=0x7fa7b800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:11:49.798 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:49.795+0000 7fa7ccc00640 1 -- 172.21.15.96:0/2365402561 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7b80125f0 con 0x7fa7d0155a00 2024-05-28T06:11:49.798 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:49.795+0000 7fa7ccc00640 1 -- 172.21.15.96:0/2365402561 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7b8019070 con 0x7fa7d0155a00 2024-05-28T06:11:49.798 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:49.795+0000 7fa7ccc00640 1 -- 172.21.15.96:0/2365402561 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7b80129d0 con 0x7fa7d0155a00 2024-05-28T06:11:49.798 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:49.795+0000 7fa7d6e00640 1 -- 172.21.15.96:0/2365402561 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7d014ebd0 con 0x7fa7d0155a00 2024-05-28T06:11:49.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:49.795+0000 7fa7d6e00640 1 -- 172.21.15.96:0/2365402561 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7d014bc10 con 0x7fa7d0155a00 2024-05-28T06:11:49.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:49.796+0000 7fa7d6e00640 1 -- 172.21.15.96:0/2365402561 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7d00689e0 con 0x7fa7d0155a00 2024-05-28T06:11:49.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:49.797+0000 7fa7ccc00640 1 -- 172.21.15.96:0/2365402561 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 15) ==== 91652+0+0 (secure 0 0 0) 0x7fa7b8012b70 con 0x7fa7d0155a00 2024-05-28T06:11:49.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:49.798+0000 7fa7ccc00640 1 --2- 172.21.15.96:0/2365402561 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa7ac06cfb0 0x7fa7ac06f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:49.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:49.798+0000 7fa7ccc00640 1 -- 172.21.15.96:0/2365402561 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5795+0+0 (secure 0 0 0) 0x7fa7b808cab0 con 0x7fa7d0155a00 2024-05-28T06:11:49.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:49.799+0000 7fa7cfe00640 1 --2- 172.21.15.96:0/2365402561 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa7ac06cfb0 0x7fa7ac06f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:49.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:49.800+0000 7fa7cfe00640 1 --2- 172.21.15.96:0/2365402561 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa7ac06cfb0 0x7fa7ac06f470 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fa7c0001680 tx=0x7fa7c0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:11:49.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:49.804+0000 7fa7ccc00640 1 -- 172.21.15.96:0/2365402561 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa7d014ebd0 con 0x7fa7d0155a00 2024-05-28T06:11:50.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:50.083+0000 7fa7d6e00640 1 -- 172.21.15.96:0/2365402561 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fa7d006f170 con 0x7fa7d0155a00 2024-05-28T06:11:50.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:50.084+0000 7fa7ccc00640 1 -- 172.21.15.96:0/2365402561 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+2078 (secure 0 0 0) 0x7fa7b805a640 con 0x7fa7d0155a00 2024-05-28T06:11:50.086 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:11:50.086 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:11:50.086 INFO:teuthology.orchestra.run.smithi096.stdout: "fsid": "e4a831b5-e6d1-47a7-96cb-bf85154ee0d2", 2024-05-28T06:11:50.086 INFO:teuthology.orchestra.run.smithi096.stdout: "health": { 2024-05-28T06:11:50.086 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "HEALTH_OK", 2024-05-28T06:11:50.086 INFO:teuthology.orchestra.run.smithi096.stdout: "checks": {}, 2024-05-28T06:11:50.086 INFO:teuthology.orchestra.run.smithi096.stdout: "mutes": [] 2024-05-28T06:11:50.086 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:11:50.087 INFO:teuthology.orchestra.run.smithi096.stdout: "election_epoch": 8, 2024-05-28T06:11:50.087 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum": [ 2024-05-28T06:11:50.087 INFO:teuthology.orchestra.run.smithi096.stdout: 0, 2024-05-28T06:11:50.087 INFO:teuthology.orchestra.run.smithi096.stdout: 1, 2024-05-28T06:11:50.087 INFO:teuthology.orchestra.run.smithi096.stdout: 2 2024-05-28T06:11:50.087 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:11:50.087 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_names": [ 2024-05-28T06:11:50.087 INFO:teuthology.orchestra.run.smithi096.stdout: "a", 2024-05-28T06:11:50.087 INFO:teuthology.orchestra.run.smithi096.stdout: "b", 2024-05-28T06:11:50.087 INFO:teuthology.orchestra.run.smithi096.stdout: "c" 2024-05-28T06:11:50.087 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:11:50.087 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_age": 483, 2024-05-28T06:11:50.087 INFO:teuthology.orchestra.run.smithi096.stdout: "monmap": { 2024-05-28T06:11:50.087 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 1, 2024-05-28T06:11:50.088 INFO:teuthology.orchestra.run.smithi096.stdout: "min_mon_release_name": "squid", 2024-05-28T06:11:50.088 INFO:teuthology.orchestra.run.smithi096.stdout: "num_mons": 3 2024-05-28T06:11:50.088 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:11:50.088 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap": { 2024-05-28T06:11:50.088 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 47, 2024-05-28T06:11:50.088 INFO:teuthology.orchestra.run.smithi096.stdout: "num_osds": 8, 2024-05-28T06:11:50.088 INFO:teuthology.orchestra.run.smithi096.stdout: "num_up_osds": 8, 2024-05-28T06:11:50.088 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_up_since": 1716876238, 2024-05-28T06:11:50.088 INFO:teuthology.orchestra.run.smithi096.stdout: "num_in_osds": 8, 2024-05-28T06:11:50.088 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_in_since": 1716876229, 2024-05-28T06:11:50.088 INFO:teuthology.orchestra.run.smithi096.stdout: "num_remapped_pgs": 0 2024-05-28T06:11:50.088 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:11:50.089 INFO:teuthology.orchestra.run.smithi096.stdout: "pgmap": { 2024-05-28T06:11:50.089 INFO:teuthology.orchestra.run.smithi096.stdout: "pgs_by_state": [ 2024-05-28T06:11:50.089 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:11:50.089 INFO:teuthology.orchestra.run.smithi096.stdout: "state_name": "active+clean", 2024-05-28T06:11:50.089 INFO:teuthology.orchestra.run.smithi096.stdout: "count": 97 2024-05-28T06:11:50.089 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:11:50.089 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:11:50.089 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pgs": 97, 2024-05-28T06:11:50.089 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pools": 3, 2024-05-28T06:11:50.089 INFO:teuthology.orchestra.run.smithi096.stdout: "num_objects": 4176, 2024-05-28T06:11:50.089 INFO:teuthology.orchestra.run.smithi096.stdout: "data_bytes": 17383168377, 2024-05-28T06:11:50.090 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_used": 35149873152, 2024-05-28T06:11:50.090 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_avail": 737944240128, 2024-05-28T06:11:50.090 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_total": 773094113280, 2024-05-28T06:11:50.090 INFO:teuthology.orchestra.run.smithi096.stdout: "write_bytes_sec": 1108, 2024-05-28T06:11:50.090 INFO:teuthology.orchestra.run.smithi096.stdout: "read_op_per_sec": 0, 2024-05-28T06:11:50.090 INFO:teuthology.orchestra.run.smithi096.stdout: "write_op_per_sec": 0 2024-05-28T06:11:50.090 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:11:50.090 INFO:teuthology.orchestra.run.smithi096.stdout: "fsmap": { 2024-05-28T06:11:50.090 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 26, 2024-05-28T06:11:50.090 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 3, 2024-05-28T06:11:50.090 INFO:teuthology.orchestra.run.smithi096.stdout: "up": 1, 2024-05-28T06:11:50.090 INFO:teuthology.orchestra.run.smithi096.stdout: "in": 1, 2024-05-28T06:11:50.091 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 1, 2024-05-28T06:11:50.091 INFO:teuthology.orchestra.run.smithi096.stdout: "by_rank": [ 2024-05-28T06:11:50.091 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:11:50.091 INFO:teuthology.orchestra.run.smithi096.stdout: "filesystem_id": 3, 2024-05-28T06:11:50.091 INFO:teuthology.orchestra.run.smithi096.stdout: "rank": 0, 2024-05-28T06:11:50.091 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "a", 2024-05-28T06:11:50.091 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "up:active", 2024-05-28T06:11:50.091 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 5859 2024-05-28T06:11:50.091 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:11:50.091 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:11:50.091 INFO:teuthology.orchestra.run.smithi096.stdout: "up:standby": 3 2024-05-28T06:11:50.091 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:11:50.091 INFO:teuthology.orchestra.run.smithi096.stdout: "mgrmap": { 2024-05-28T06:11:50.091 INFO:teuthology.orchestra.run.smithi096.stdout: "available": true, 2024-05-28T06:11:50.092 INFO:teuthology.orchestra.run.smithi096.stdout: "num_standbys": 1, 2024-05-28T06:11:50.092 INFO:teuthology.orchestra.run.smithi096.stdout: "modules": [ 2024-05-28T06:11:50.092 INFO:teuthology.orchestra.run.smithi096.stdout: "iostat", 2024-05-28T06:11:50.092 INFO:teuthology.orchestra.run.smithi096.stdout: "nfs", 2024-05-28T06:11:50.092 INFO:teuthology.orchestra.run.smithi096.stdout: "restful" 2024-05-28T06:11:50.092 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:11:50.092 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:11:50.092 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:11:50.092 INFO:teuthology.orchestra.run.smithi096.stdout: "servicemap": { 2024-05-28T06:11:50.092 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 9, 2024-05-28T06:11:50.092 INFO:teuthology.orchestra.run.smithi096.stdout: "modified": "2024-05-28T06:10:01.618312+0000", 2024-05-28T06:11:50.092 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:11:50.092 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:11:50.092 INFO:teuthology.orchestra.run.smithi096.stdout: "progress_events": { 2024-05-28T06:11:50.093 INFO:teuthology.orchestra.run.smithi096.stdout: "72116a8d-650b-4cfc-99af-ee8db5167322": { 2024-05-28T06:11:50.093 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "All cloning finished, waiting 8.0 seconds before exiting (32s)\n [============================] ", 2024-05-28T06:11:50.093 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": 1, 2024-05-28T06:11:50.093 INFO:teuthology.orchestra.run.smithi096.stdout: "add_to_ceph_s": true 2024-05-28T06:11:50.093 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:11:50.093 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:11:50.093 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:11:50.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:50.087+0000 7fa7d6e00640 1 -- 172.21.15.96:0/2365402561 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa7ac06cfb0 msgr2=0x7fa7ac06f470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:50.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:50.087+0000 7fa7d6e00640 1 --2- 172.21.15.96:0/2365402561 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa7ac06cfb0 0x7fa7ac06f470 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fa7c0001680 tx=0x7fa7c0009040 comp rx=0 tx=0).stop 2024-05-28T06:11:50.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:50.087+0000 7fa7d6e00640 1 -- 172.21.15.96:0/2365402561 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa7d0155a00 msgr2=0x7fa7d01542a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:50.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:50.087+0000 7fa7d6e00640 1 --2- 172.21.15.96:0/2365402561 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa7d0155a00 0x7fa7d01542a0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fa7b800d5f0 tx=0x7fa7b800dab0 comp rx=0 tx=0).stop 2024-05-28T06:11:50.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:50.087+0000 7fa7d6e00640 1 -- 172.21.15.96:0/2365402561 shutdown_connections 2024-05-28T06:11:50.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:50.087+0000 7fa7d6e00640 1 --2- 172.21.15.96:0/2365402561 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa7ac06cfb0 0x7fa7ac06f470 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:50.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:50.087+0000 7fa7d6e00640 1 --2- 172.21.15.96:0/2365402561 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa7d015baa0 0x7fa7d014f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:50.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:50.087+0000 7fa7d6e00640 1 --2- 172.21.15.96:0/2365402561 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa7d0155a00 0x7fa7d01542a0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:50.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:50.087+0000 7fa7d6e00640 1 --2- 172.21.15.96:0/2365402561 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa7d006f390 0x7fa7d0150660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:50.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:50.087+0000 7fa7d6e00640 1 -- 172.21.15.96:0/2365402561 >> 172.21.15.96:0/2365402561 conn(0x7fa7d0067ce0 msgr2=0x7fa7d00b3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:11:50.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:50.087+0000 7fa7d6e00640 1 -- 172.21.15.96:0/2365402561 shutdown_connections 2024-05-28T06:11:50.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:50.087+0000 7fa7d6e00640 1 -- 172.21.15.96:0/2365402561 wait complete. 2024-05-28T06:11:55.107 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-05-28T06:11:55.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.315+0000 7f6f50400640 1 Processor -- start 2024-05-28T06:11:55.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.315+0000 7f6f50400640 1 -- start start 2024-05-28T06:11:55.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.316+0000 7f6f50400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f6f48145320 0x7f6f48071cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:55.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.316+0000 7f6f50400640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f480cf0c0 con 0x7f6f48072200 2024-05-28T06:11:55.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.316+0000 7f6f50400640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f480b4a20 con 0x7f6f480b43b0 2024-05-28T06:11:55.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.316+0000 7f6f50400640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f6f480b4d00 con 0x7f6f48145320 2024-05-28T06:11:55.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.317+0000 7f6f4d000640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f6f48072200 0x7f6f480b3ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:42450/0 (socket says 172.21.15.96:42450) 2024-05-28T06:11:55.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.317+0000 7f6f4d000640 1 -- 172.21.15.96:0/3798703208 learned_addr learned my addr 172.21.15.96:0/3798703208 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:11:55.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.317+0000 7f6f4da00640 1 --2- 172.21.15.96:0/3798703208 >> v2:172.21.15.123:3300/0 conn(0x7f6f48145320 0x7f6f48071cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:55.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.317+0000 7f6f47e00640 1 -- 172.21.15.96:0/3798703208 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2243427297 0 0) 0x7f6f480cf0c0 con 0x7f6f48072200 2024-05-28T06:11:55.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.317+0000 7f6f47e00640 1 -- 172.21.15.96:0/3798703208 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6f30003520 con 0x7f6f48072200 2024-05-28T06:11:55.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.318+0000 7f6f47e00640 1 -- 172.21.15.96:0/3798703208 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4119529592 0 0) 0x7f6f30003520 con 0x7f6f48072200 2024-05-28T06:11:55.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.318+0000 7f6f47e00640 1 -- 172.21.15.96:0/3798703208 >> v2:172.21.15.123:3300/0 conn(0x7f6f48145320 msgr2=0x7f6f48071cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:11:55.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.318+0000 7f6f47e00640 1 --2- 172.21.15.96:0/3798703208 >> v2:172.21.15.123:3300/0 conn(0x7f6f48145320 0x7f6f48071cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:55.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.318+0000 7f6f47e00640 1 -- 172.21.15.96:0/3798703208 >> v1:172.21.15.123:6789/0 conn(0x7f6f480b43b0 legacy=0x7f6f480cf880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:55.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.318+0000 7f6f47e00640 1 -- 172.21.15.96:0/3798703208 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f480b4fe0 con 0x7f6f48072200 2024-05-28T06:11:55.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.318+0000 7f6f47e00640 1 -- 172.21.15.96:0/3798703208 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f6f34001670 con 0x7f6f48072200 2024-05-28T06:11:55.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.318+0000 7f6f47e00640 1 -- 172.21.15.96:0/3798703208 >> v1:172.21.15.96:6789/0 conn(0x7f6f48072200 legacy=0x7f6f480b3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:55.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.318+0000 7f6f47e00640 1 --2- 172.21.15.96:0/3798703208 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6f30003a00 0x7f6f30003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:55.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.318+0000 7f6f47e00640 1 --2- 172.21.15.96:0/3798703208 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6f30004800 0x7f6f30004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:55.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.318+0000 7f6f47e00640 1 --2- 172.21.15.96:0/3798703208 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f300051c0 0x7f6f3000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:55.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.318+0000 7f6f47e00640 1 -- 172.21.15.96:0/3798703208 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f6f480b4d00 con 0x7f6f30004800 2024-05-28T06:11:55.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.318+0000 7f6f47e00640 1 -- 172.21.15.96:0/3798703208 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f6f3000c0c0 con 0x7f6f300051c0 2024-05-28T06:11:55.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.318+0000 7f6f47e00640 1 -- 172.21.15.96:0/3798703208 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f6f3000c2c0 con 0x7f6f30003a00 2024-05-28T06:11:55.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.318+0000 7f6f4d000640 1 --2- 172.21.15.96:0/3798703208 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6f30004800 0x7f6f30004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:55.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.319+0000 7f6f4da00640 1 --2- 172.21.15.96:0/3798703208 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6f30003a00 0x7f6f30003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:55.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.319+0000 7f6f4e400640 1 --2- 172.21.15.96:0/3798703208 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f300051c0 0x7f6f3000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:55.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.319+0000 7f6f4d000640 1 -- 172.21.15.96:0/3798703208 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6f30003a00 msgr2=0x7f6f30003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:11:55.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.319+0000 7f6f4d000640 1 --2- 172.21.15.96:0/3798703208 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6f30003a00 0x7f6f30003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:55.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.319+0000 7f6f4d000640 1 -- 172.21.15.96:0/3798703208 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f300051c0 msgr2=0x7f6f3000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:11:55.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.319+0000 7f6f4d000640 1 --2- 172.21.15.96:0/3798703208 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f300051c0 0x7f6f3000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:55.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.319+0000 7f6f4d000640 1 -- 172.21.15.96:0/3798703208 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f30003520 con 0x7f6f30004800 2024-05-28T06:11:55.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.319+0000 7f6f4e400640 1 --2- 172.21.15.96:0/3798703208 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f300051c0 0x7f6f3000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:11:55.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.319+0000 7f6f4e400640 1 -- 172.21.15.96:0/3798703208 reap_dead start 2024-05-28T06:11:55.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.320+0000 7f6f4d000640 1 --2- 172.21.15.96:0/3798703208 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6f30004800 0x7f6f30004c80 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f6f3400c0e0 tx=0x7f6f30005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=341d06c06ad92de6 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:11:55.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.320+0000 7f6f47e00640 1 -- 172.21.15.96:0/3798703208 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f34007c20 con 0x7f6f30004800 2024-05-28T06:11:55.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.320+0000 7f6f47e00640 1 -- 172.21.15.96:0/3798703208 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f34005af0 con 0x7f6f30004800 2024-05-28T06:11:55.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.320+0000 7f6f47e00640 1 -- 172.21.15.96:0/3798703208 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f34007070 con 0x7f6f30004800 2024-05-28T06:11:55.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.321+0000 7f6f50400640 1 -- 172.21.15.96:0/3798703208 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6f30004800 msgr2=0x7f6f30004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:55.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.321+0000 7f6f50400640 1 --2- 172.21.15.96:0/3798703208 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6f30004800 0x7f6f30004c80 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f6f3400c0e0 tx=0x7f6f30005830 comp rx=0 tx=0).stop 2024-05-28T06:11:55.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.321+0000 7f6f50400640 1 -- 172.21.15.96:0/3798703208 shutdown_connections 2024-05-28T06:11:55.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.321+0000 7f6f50400640 1 --2- 172.21.15.96:0/3798703208 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6f30004800 0x7f6f30004c80 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:55.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.321+0000 7f6f50400640 1 -- 172.21.15.96:0/3798703208 >> 172.21.15.96:0/3798703208 conn(0x7f6f48067ce0 msgr2=0x7f6f4806b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:11:55.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.321+0000 7f6f50400640 1 -- 172.21.15.96:0/3798703208 shutdown_connections 2024-05-28T06:11:55.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.321+0000 7f6f50400640 1 -- 172.21.15.96:0/3798703208 wait complete. 2024-05-28T06:11:55.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.323+0000 7f6f50400640 1 Processor -- start 2024-05-28T06:11:55.325 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.323+0000 7f6f50400640 1 -- start start 2024-05-28T06:11:55.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.324+0000 7f6f50400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6f30004800 0x7f6f4813c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:55.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.324+0000 7f6f50400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6f481e6a90 0x7f6f481f5b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:55.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.324+0000 7f6f50400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f481f60d0 0x7f6f481f6560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:55.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.324+0000 7f6f50400640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f6f48071cc0 con 0x7f6f481e6a90 2024-05-28T06:11:55.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.324+0000 7f6f50400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f6f48145700 con 0x7f6f481f60d0 2024-05-28T06:11:55.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.324+0000 7f6f50400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f6f48071e40 con 0x7f6f30004800 2024-05-28T06:11:55.327 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.324+0000 7f6f4d000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6f481e6a90 0x7f6f481f5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:55.327 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.324+0000 7f6f4d000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6f481e6a90 0x7f6f481f5b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35386/0 (socket says 172.21.15.96:35386) 2024-05-28T06:11:55.327 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.324+0000 7f6f4d000640 1 -- 172.21.15.96:0/675300216 learned_addr learned my addr 172.21.15.96:0/675300216 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:11:55.327 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.324+0000 7f6f4e400640 1 --2- 172.21.15.96:0/675300216 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f481f60d0 0x7f6f481f6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:55.327 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.324+0000 7f6f4da00640 1 --2- 172.21.15.96:0/675300216 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6f30004800 0x7f6f4813c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:55.327 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.324+0000 7f6f4d000640 1 -- 172.21.15.96:0/675300216 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6f30004800 msgr2=0x7f6f4813c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:55.327 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.324+0000 7f6f4d000640 1 --2- 172.21.15.96:0/675300216 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6f30004800 0x7f6f4813c3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:55.327 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.324+0000 7f6f4d000640 1 -- 172.21.15.96:0/675300216 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f481f60d0 msgr2=0x7f6f481f6560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:55.327 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.324+0000 7f6f4d000640 1 --2- 172.21.15.96:0/675300216 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f481f60d0 0x7f6f481f6560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:55.327 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.324+0000 7f6f4d000640 1 -- 172.21.15.96:0/675300216 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f4813c910 con 0x7f6f481e6a90 2024-05-28T06:11:55.327 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.325+0000 7f6f4d000640 1 --2- 172.21.15.96:0/675300216 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6f481e6a90 0x7f6f481f5b90 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f6f3400c200 tx=0x7f6f480b4bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:11:55.328 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.325+0000 7f6f45600640 1 -- 172.21.15.96:0/675300216 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f34042070 con 0x7f6f481e6a90 2024-05-28T06:11:55.328 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.325+0000 7f6f45600640 1 -- 172.21.15.96:0/675300216 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f340047e0 con 0x7f6f481e6a90 2024-05-28T06:11:55.328 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.325+0000 7f6f45600640 1 -- 172.21.15.96:0/675300216 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f34037040 con 0x7f6f481e6a90 2024-05-28T06:11:55.328 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.325+0000 7f6f50400640 1 -- 172.21.15.96:0/675300216 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f481e6f70 con 0x7f6f481e6a90 2024-05-28T06:11:55.328 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.325+0000 7f6f50400640 1 -- 172.21.15.96:0/675300216 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f481e7530 con 0x7f6f481e6a90 2024-05-28T06:11:55.328 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.326+0000 7f6f50400640 1 -- 172.21.15.96:0/675300216 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f480689e0 con 0x7f6f481e6a90 2024-05-28T06:11:55.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.327+0000 7f6f45600640 1 -- 172.21.15.96:0/675300216 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 15) ==== 91652+0+0 (secure 0 0 0) 0x7f6f34045070 con 0x7f6f481e6a90 2024-05-28T06:11:55.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.328+0000 7f6f45600640 1 --2- 172.21.15.96:0/675300216 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6f2806cfe0 0x7f6f2806f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:11:55.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.329+0000 7f6f45600640 1 -- 172.21.15.96:0/675300216 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5795+0+0 (secure 0 0 0) 0x7f6f340abe50 con 0x7f6f481e6a90 2024-05-28T06:11:55.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.329+0000 7f6f4da00640 1 --2- 172.21.15.96:0/675300216 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6f2806cfe0 0x7f6f2806f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:11:55.331 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.330+0000 7f6f4da00640 1 --2- 172.21.15.96:0/675300216 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6f2806cfe0 0x7f6f2806f4a0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f6f3c0099d0 tx=0x7f6f3c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:11:55.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.333+0000 7f6f45600640 1 -- 172.21.15.96:0/675300216 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6f34079300 con 0x7f6f481e6a90 2024-05-28T06:11:55.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.605+0000 7f6f50400640 1 -- 172.21.15.96:0/675300216 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f6f480b34e0 con 0x7f6f481e6a90 2024-05-28T06:11:55.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.605+0000 7f6f45600640 1 -- 172.21.15.96:0/675300216 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+2077 (secure 0 0 0) 0x7f6f3407d600 con 0x7f6f481e6a90 2024-05-28T06:11:55.607 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:11:55.607 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:11:55.607 INFO:teuthology.orchestra.run.smithi096.stdout: "fsid": "e4a831b5-e6d1-47a7-96cb-bf85154ee0d2", 2024-05-28T06:11:55.607 INFO:teuthology.orchestra.run.smithi096.stdout: "health": { 2024-05-28T06:11:55.607 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "HEALTH_OK", 2024-05-28T06:11:55.607 INFO:teuthology.orchestra.run.smithi096.stdout: "checks": {}, 2024-05-28T06:11:55.608 INFO:teuthology.orchestra.run.smithi096.stdout: "mutes": [] 2024-05-28T06:11:55.608 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:11:55.608 INFO:teuthology.orchestra.run.smithi096.stdout: "election_epoch": 8, 2024-05-28T06:11:55.608 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum": [ 2024-05-28T06:11:55.608 INFO:teuthology.orchestra.run.smithi096.stdout: 0, 2024-05-28T06:11:55.608 INFO:teuthology.orchestra.run.smithi096.stdout: 1, 2024-05-28T06:11:55.608 INFO:teuthology.orchestra.run.smithi096.stdout: 2 2024-05-28T06:11:55.608 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:11:55.608 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_names": [ 2024-05-28T06:11:55.608 INFO:teuthology.orchestra.run.smithi096.stdout: "a", 2024-05-28T06:11:55.608 INFO:teuthology.orchestra.run.smithi096.stdout: "b", 2024-05-28T06:11:55.608 INFO:teuthology.orchestra.run.smithi096.stdout: "c" 2024-05-28T06:11:55.608 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:11:55.608 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_age": 489, 2024-05-28T06:11:55.608 INFO:teuthology.orchestra.run.smithi096.stdout: "monmap": { 2024-05-28T06:11:55.609 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 1, 2024-05-28T06:11:55.609 INFO:teuthology.orchestra.run.smithi096.stdout: "min_mon_release_name": "squid", 2024-05-28T06:11:55.609 INFO:teuthology.orchestra.run.smithi096.stdout: "num_mons": 3 2024-05-28T06:11:55.609 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:11:55.609 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap": { 2024-05-28T06:11:55.609 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 47, 2024-05-28T06:11:55.609 INFO:teuthology.orchestra.run.smithi096.stdout: "num_osds": 8, 2024-05-28T06:11:55.609 INFO:teuthology.orchestra.run.smithi096.stdout: "num_up_osds": 8, 2024-05-28T06:11:55.609 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_up_since": 1716876238, 2024-05-28T06:11:55.609 INFO:teuthology.orchestra.run.smithi096.stdout: "num_in_osds": 8, 2024-05-28T06:11:55.609 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_in_since": 1716876229, 2024-05-28T06:11:55.609 INFO:teuthology.orchestra.run.smithi096.stdout: "num_remapped_pgs": 0 2024-05-28T06:11:55.609 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:11:55.610 INFO:teuthology.orchestra.run.smithi096.stdout: "pgmap": { 2024-05-28T06:11:55.610 INFO:teuthology.orchestra.run.smithi096.stdout: "pgs_by_state": [ 2024-05-28T06:11:55.610 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:11:55.610 INFO:teuthology.orchestra.run.smithi096.stdout: "state_name": "active+clean", 2024-05-28T06:11:55.610 INFO:teuthology.orchestra.run.smithi096.stdout: "count": 97 2024-05-28T06:11:55.610 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:11:55.610 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:11:55.610 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pgs": 97, 2024-05-28T06:11:55.610 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pools": 3, 2024-05-28T06:11:55.610 INFO:teuthology.orchestra.run.smithi096.stdout: "num_objects": 4176, 2024-05-28T06:11:55.610 INFO:teuthology.orchestra.run.smithi096.stdout: "data_bytes": 17383168377, 2024-05-28T06:11:55.610 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_used": 35149873152, 2024-05-28T06:11:55.610 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_avail": 737944240128, 2024-05-28T06:11:55.611 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_total": 773094113280, 2024-05-28T06:11:55.611 INFO:teuthology.orchestra.run.smithi096.stdout: "write_bytes_sec": 170, 2024-05-28T06:11:55.611 INFO:teuthology.orchestra.run.smithi096.stdout: "read_op_per_sec": 0, 2024-05-28T06:11:55.611 INFO:teuthology.orchestra.run.smithi096.stdout: "write_op_per_sec": 0 2024-05-28T06:11:55.611 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:11:55.611 INFO:teuthology.orchestra.run.smithi096.stdout: "fsmap": { 2024-05-28T06:11:55.611 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 26, 2024-05-28T06:11:55.611 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 3, 2024-05-28T06:11:55.611 INFO:teuthology.orchestra.run.smithi096.stdout: "up": 1, 2024-05-28T06:11:55.611 INFO:teuthology.orchestra.run.smithi096.stdout: "in": 1, 2024-05-28T06:11:55.611 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 1, 2024-05-28T06:11:55.611 INFO:teuthology.orchestra.run.smithi096.stdout: "by_rank": [ 2024-05-28T06:11:55.611 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:11:55.611 INFO:teuthology.orchestra.run.smithi096.stdout: "filesystem_id": 3, 2024-05-28T06:11:55.612 INFO:teuthology.orchestra.run.smithi096.stdout: "rank": 0, 2024-05-28T06:11:55.612 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "a", 2024-05-28T06:11:55.612 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "up:active", 2024-05-28T06:11:55.612 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 5859 2024-05-28T06:11:55.612 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:11:55.612 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:11:55.612 INFO:teuthology.orchestra.run.smithi096.stdout: "up:standby": 3 2024-05-28T06:11:55.612 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:11:55.612 INFO:teuthology.orchestra.run.smithi096.stdout: "mgrmap": { 2024-05-28T06:11:55.612 INFO:teuthology.orchestra.run.smithi096.stdout: "available": true, 2024-05-28T06:11:55.612 INFO:teuthology.orchestra.run.smithi096.stdout: "num_standbys": 1, 2024-05-28T06:11:55.612 INFO:teuthology.orchestra.run.smithi096.stdout: "modules": [ 2024-05-28T06:11:55.612 INFO:teuthology.orchestra.run.smithi096.stdout: "iostat", 2024-05-28T06:11:55.612 INFO:teuthology.orchestra.run.smithi096.stdout: "nfs", 2024-05-28T06:11:55.612 INFO:teuthology.orchestra.run.smithi096.stdout: "restful" 2024-05-28T06:11:55.613 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:11:55.613 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:11:55.613 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:11:55.613 INFO:teuthology.orchestra.run.smithi096.stdout: "servicemap": { 2024-05-28T06:11:55.613 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 9, 2024-05-28T06:11:55.613 INFO:teuthology.orchestra.run.smithi096.stdout: "modified": "2024-05-28T06:10:01.618312+0000", 2024-05-28T06:11:55.613 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:11:55.613 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:11:55.613 INFO:teuthology.orchestra.run.smithi096.stdout: "progress_events": { 2024-05-28T06:11:55.613 INFO:teuthology.orchestra.run.smithi096.stdout: "72116a8d-650b-4cfc-99af-ee8db5167322": { 2024-05-28T06:11:55.613 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "All cloning finished, waiting 4.0 seconds before exiting (36s)\n [============================] ", 2024-05-28T06:11:55.613 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": 1, 2024-05-28T06:11:55.613 INFO:teuthology.orchestra.run.smithi096.stdout: "add_to_ceph_s": true 2024-05-28T06:11:55.613 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:11:55.614 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:11:55.614 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:11:55.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.608+0000 7f6f50400640 1 -- 172.21.15.96:0/675300216 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6f2806cfe0 msgr2=0x7f6f2806f4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:55.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.608+0000 7f6f50400640 1 --2- 172.21.15.96:0/675300216 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6f2806cfe0 0x7f6f2806f4a0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f6f3c0099d0 tx=0x7f6f3c008040 comp rx=0 tx=0).stop 2024-05-28T06:11:55.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.608+0000 7f6f50400640 1 -- 172.21.15.96:0/675300216 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6f481e6a90 msgr2=0x7f6f481f5b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:11:55.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.608+0000 7f6f50400640 1 --2- 172.21.15.96:0/675300216 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6f481e6a90 0x7f6f481f5b90 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f6f3400c200 tx=0x7f6f480b4bf0 comp rx=0 tx=0).stop 2024-05-28T06:11:55.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.608+0000 7f6f50400640 1 -- 172.21.15.96:0/675300216 shutdown_connections 2024-05-28T06:11:55.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.608+0000 7f6f50400640 1 --2- 172.21.15.96:0/675300216 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6f2806cfe0 0x7f6f2806f4a0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:55.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.608+0000 7f6f50400640 1 --2- 172.21.15.96:0/675300216 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f481f60d0 0x7f6f481f6560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:55.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.608+0000 7f6f50400640 1 --2- 172.21.15.96:0/675300216 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6f481e6a90 0x7f6f481f5b90 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:55.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.608+0000 7f6f50400640 1 --2- 172.21.15.96:0/675300216 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6f30004800 0x7f6f4813c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:11:55.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.608+0000 7f6f50400640 1 -- 172.21.15.96:0/675300216 >> 172.21.15.96:0/675300216 conn(0x7f6f48067ce0 msgr2=0x7f6f480afb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:11:55.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.608+0000 7f6f50400640 1 -- 172.21.15.96:0/675300216 shutdown_connections 2024-05-28T06:11:55.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:11:55.608+0000 7f6f50400640 1 -- 172.21.15.96:0/675300216 wait complete. 2024-05-28T06:12:00.630 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-05-28T06:12:00.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:00.836+0000 7fb72f200640 1 Processor -- start 2024-05-28T06:12:00.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:00.837+0000 7fb72f200640 1 -- start start 2024-05-28T06:12:00.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:00.838+0000 7fb72f200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb7300d0460 0x7fb7300d08f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:00.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:00.838+0000 7fb72f200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7300cd1a0 con 0x7fb73015c6b0 2024-05-28T06:12:00.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:00.838+0000 7fb72f200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7300cd480 con 0x7fb7300ccc30 2024-05-28T06:12:00.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:00.838+0000 7fb72f200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fb73006e3a0 con 0x7fb7300d0460 2024-05-28T06:12:00.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:00.838+0000 7fb72de00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fb73015c6b0 0x7fb7300b3db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:59680/0 (socket says 172.21.15.96:59680) 2024-05-28T06:12:00.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:00.838+0000 7fb72de00640 1 -- 172.21.15.96:0/1659603753 learned_addr learned my addr 172.21.15.96:0/1659603753 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:12:00.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:00.839+0000 7fb72e800640 1 --2- 172.21.15.96:0/1659603753 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb7300d0460 0x7fb7300d08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:00.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:00.839+0000 7fb72d400640 1 -- 172.21.15.96:0/1659603753 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2572824073 0 0) 0x7fb7300cd1a0 con 0x7fb73015c6b0 2024-05-28T06:12:00.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:00.839+0000 7fb72d400640 1 -- 172.21.15.96:0/1659603753 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb714003520 con 0x7fb73015c6b0 2024-05-28T06:12:00.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:00.839+0000 7fb72d400640 1 -- 172.21.15.96:0/1659603753 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1274259225 0 0) 0x7fb714003520 con 0x7fb73015c6b0 2024-05-28T06:12:00.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:00.839+0000 7fb72d400640 1 -- 172.21.15.96:0/1659603753 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb7300d0460 msgr2=0x7fb7300d08f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:00.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:00.839+0000 7fb72d400640 1 --2- 172.21.15.96:0/1659603753 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb7300d0460 0x7fb7300d08f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:00.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:00.839+0000 7fb72d400640 1 -- 172.21.15.96:0/1659603753 >> v1:172.21.15.123:6789/0 conn(0x7fb7300ccc30 legacy=0x7fb73006dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:00.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:00.839+0000 7fb72d400640 1 -- 172.21.15.96:0/1659603753 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7300cd760 con 0x7fb73015c6b0 2024-05-28T06:12:00.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:00.839+0000 7fb72d400640 1 -- 172.21.15.96:0/1659603753 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7fb720001670 con 0x7fb73015c6b0 2024-05-28T06:12:00.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:00.840+0000 7fb72d400640 1 -- 172.21.15.96:0/1659603753 >> v1:172.21.15.96:6789/0 conn(0x7fb73015c6b0 legacy=0x7fb7300b3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:00.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:00.840+0000 7fb72d400640 1 --2- 172.21.15.96:0/1659603753 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb7300d0460 0x7fb7140039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:00.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:00.840+0000 7fb72d400640 1 --2- 172.21.15.96:0/1659603753 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb714003f60 0x7fb714004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:00.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:00.840+0000 7fb72d400640 1 --2- 172.21.15.96:0/1659603753 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb714004d40 0x7fb7140051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:00.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:00.840+0000 7fb72d400640 1 -- 172.21.15.96:0/1659603753 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb73006e3a0 con 0x7fb714004d40 2024-05-28T06:12:00.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:00.840+0000 7fb72d400640 1 -- 172.21.15.96:0/1659603753 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fb71400baa0 con 0x7fb714003f60 2024-05-28T06:12:00.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:00.840+0000 7fb72d400640 1 -- 172.21.15.96:0/1659603753 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fb71400bca0 con 0x7fb7300d0460 2024-05-28T06:12:00.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:00.840+0000 7fb72e800640 1 --2- 172.21.15.96:0/1659603753 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb714004d40 0x7fb7140051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:00.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:00.840+0000 7fb72de00640 1 --2- 172.21.15.96:0/1659603753 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb7300d0460 0x7fb7140039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:00.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:00.840+0000 7fb727e00640 1 --2- 172.21.15.96:0/1659603753 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb714003f60 0x7fb714004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:00.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:00.841+0000 7fb72e800640 1 -- 172.21.15.96:0/1659603753 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb7300d0460 msgr2=0x7fb7140039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:00.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:00.841+0000 7fb72e800640 1 --2- 172.21.15.96:0/1659603753 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb7300d0460 0x7fb7140039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:00.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:00.841+0000 7fb72e800640 1 -- 172.21.15.96:0/1659603753 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb714003f60 msgr2=0x7fb714004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:00.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:00.841+0000 7fb72e800640 1 --2- 172.21.15.96:0/1659603753 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb714003f60 0x7fb714004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:00.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:00.841+0000 7fb72e800640 1 -- 172.21.15.96:0/1659603753 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb714003520 con 0x7fb714004d40 2024-05-28T06:12:00.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:00.841+0000 7fb727e00640 1 --2- 172.21.15.96:0/1659603753 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb714003f60 0x7fb714004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:12:00.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:00.842+0000 7fb72e800640 1 --2- 172.21.15.96:0/1659603753 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb714004d40 0x7fb7140051c0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fb728009fc0 tx=0x7fb714005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa0c22a55d42c57 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:00.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:00.842+0000 7fb72d400640 1 -- 172.21.15.96:0/1659603753 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb728009c10 con 0x7fb714004d40 2024-05-28T06:12:00.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:00.842+0000 7fb72d400640 1 -- 172.21.15.96:0/1659603753 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb728036070 con 0x7fb714004d40 2024-05-28T06:12:00.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:00.842+0000 7fb72d400640 1 -- 172.21.15.96:0/1659603753 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb728031880 con 0x7fb714004d40 2024-05-28T06:12:00.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:00.842+0000 7fb72f200640 1 -- 172.21.15.96:0/1659603753 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb714004d40 msgr2=0x7fb7140051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:00.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:00.842+0000 7fb72f200640 1 --2- 172.21.15.96:0/1659603753 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb714004d40 0x7fb7140051c0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fb728009fc0 tx=0x7fb714005830 comp rx=0 tx=0).stop 2024-05-28T06:12:00.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:00.842+0000 7fb72e800640 1 -- 172.21.15.96:0/1659603753 reap_dead start 2024-05-28T06:12:00.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:00.843+0000 7fb72f200640 1 -- 172.21.15.96:0/1659603753 shutdown_connections 2024-05-28T06:12:00.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:00.843+0000 7fb72f200640 1 -- 172.21.15.96:0/1659603753 >> 172.21.15.96:0/1659603753 conn(0x7fb73006bae0 msgr2=0x7fb73006bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:00.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:00.843+0000 7fb72f200640 1 -- 172.21.15.96:0/1659603753 shutdown_connections 2024-05-28T06:12:00.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:00.843+0000 7fb72f200640 1 -- 172.21.15.96:0/1659603753 wait complete. 2024-05-28T06:12:00.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:00.844+0000 7fb72f200640 1 Processor -- start 2024-05-28T06:12:00.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:00.845+0000 7fb72f200640 1 -- start start 2024-05-28T06:12:00.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:00.845+0000 7fb72f200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb714004d40 0x7fb730099a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:00.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:00.846+0000 7fb72f200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb730072280 0x7fb730072700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:00.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:00.846+0000 7fb72f200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb73014a110 0x7fb73014a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:00.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:00.846+0000 7fb72f200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb7300d08f0 con 0x7fb73014a110 2024-05-28T06:12:00.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:00.846+0000 7fb72f200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fb7300cba10 con 0x7fb730072280 2024-05-28T06:12:00.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:00.846+0000 7fb72f200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fb7300d0a70 con 0x7fb714004d40 2024-05-28T06:12:00.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:00.846+0000 7fb72e800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb73014a110 0x7fb73014a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:00.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:00.846+0000 7fb72e800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb73014a110 0x7fb73014a590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:34710/0 (socket says 172.21.15.96:34710) 2024-05-28T06:12:00.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:00.846+0000 7fb72e800640 1 -- 172.21.15.96:0/456155787 learned_addr learned my addr 172.21.15.96:0/456155787 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:00.849 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:00.846+0000 7fb727e00640 1 --2- 172.21.15.96:0/456155787 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb730072280 0x7fb730072700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:00.849 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:00.846+0000 7fb72e800640 1 -- 172.21.15.96:0/456155787 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb714004d40 msgr2=0x7fb730099a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:00.849 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:00.846+0000 7fb72de00640 1 --2- 172.21.15.96:0/456155787 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb714004d40 0x7fb730099a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:00.849 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:00.846+0000 7fb72e800640 1 --2- 172.21.15.96:0/456155787 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb714004d40 0x7fb730099a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:00.849 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:00.846+0000 7fb72e800640 1 -- 172.21.15.96:0/456155787 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb730072280 msgr2=0x7fb730072700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:00.849 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:00.846+0000 7fb72e800640 1 --2- 172.21.15.96:0/456155787 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb730072280 0x7fb730072700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:00.849 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:00.846+0000 7fb72e800640 1 -- 172.21.15.96:0/456155787 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb73014ae80 con 0x7fb73014a110 2024-05-28T06:12:00.849 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:00.847+0000 7fb72e800640 1 --2- 172.21.15.96:0/456155787 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb73014a110 0x7fb73014a590 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fb72802aee0 tx=0x7fb728009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:00.850 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:00.847+0000 7fb726000640 1 -- 172.21.15.96:0/456155787 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb728009c10 con 0x7fb73014a110 2024-05-28T06:12:00.850 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:00.847+0000 7fb726000640 1 -- 172.21.15.96:0/456155787 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb728031dd0 con 0x7fb73014a110 2024-05-28T06:12:00.850 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:00.847+0000 7fb726000640 1 -- 172.21.15.96:0/456155787 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb728003bd0 con 0x7fb73014a110 2024-05-28T06:12:00.850 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:00.847+0000 7fb72f200640 1 -- 172.21.15.96:0/456155787 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb730148280 con 0x7fb73014a110 2024-05-28T06:12:00.850 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:00.847+0000 7fb72f200640 1 -- 172.21.15.96:0/456155787 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb73014aad0 con 0x7fb73014a110 2024-05-28T06:12:00.850 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:00.848+0000 7fb72f200640 1 -- 172.21.15.96:0/456155787 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb730072c40 con 0x7fb73014a110 2024-05-28T06:12:00.851 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:00.849+0000 7fb726000640 1 -- 172.21.15.96:0/456155787 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 15) ==== 91652+0+0 (secure 0 0 0) 0x7fb728005d10 con 0x7fb73014a110 2024-05-28T06:12:00.852 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:00.850+0000 7fb726000640 1 --2- 172.21.15.96:0/456155787 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb70c06cfe0 0x7fb70c06f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:00.852 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:00.851+0000 7fb726000640 1 -- 172.21.15.96:0/456155787 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5795+0+0 (secure 0 0 0) 0x7fb7280ad610 con 0x7fb73014a110 2024-05-28T06:12:00.852 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:00.851+0000 7fb72de00640 1 --2- 172.21.15.96:0/456155787 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb70c06cfe0 0x7fb70c06f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:00.854 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:00.852+0000 7fb72de00640 1 --2- 172.21.15.96:0/456155787 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb70c06cfe0 0x7fb70c06f4a0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fb720001050 tx=0x7fb720001470 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:00.857 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:00.856+0000 7fb726000640 1 -- 172.21.15.96:0/456155787 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb72807b010 con 0x7fb73014a110 2024-05-28T06:12:01.021 DEBUG:teuthology.orchestra.run.smithi096:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:12:01.029 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:12:01.131 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.130+0000 7fb72f200640 1 -- 172.21.15.96:0/456155787 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fb7300681e0 con 0x7fb73014a110 2024-05-28T06:12:01.132 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.130+0000 7fb726000640 1 -- 172.21.15.96:0/456155787 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1730 (secure 0 0 0) 0x7fb7300681e0 con 0x7fb73014a110 2024-05-28T06:12:01.132 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:12:01.132 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:12:01.132 INFO:teuthology.orchestra.run.smithi096.stdout: "fsid": "e4a831b5-e6d1-47a7-96cb-bf85154ee0d2", 2024-05-28T06:12:01.132 INFO:teuthology.orchestra.run.smithi096.stdout: "health": { 2024-05-28T06:12:01.132 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "HEALTH_OK", 2024-05-28T06:12:01.132 INFO:teuthology.orchestra.run.smithi096.stdout: "checks": {}, 2024-05-28T06:12:01.133 INFO:teuthology.orchestra.run.smithi096.stdout: "mutes": [] 2024-05-28T06:12:01.133 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:12:01.133 INFO:teuthology.orchestra.run.smithi096.stdout: "election_epoch": 8, 2024-05-28T06:12:01.133 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum": [ 2024-05-28T06:12:01.133 INFO:teuthology.orchestra.run.smithi096.stdout: 0, 2024-05-28T06:12:01.133 INFO:teuthology.orchestra.run.smithi096.stdout: 1, 2024-05-28T06:12:01.133 INFO:teuthology.orchestra.run.smithi096.stdout: 2 2024-05-28T06:12:01.133 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:12:01.133 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_names": [ 2024-05-28T06:12:01.133 INFO:teuthology.orchestra.run.smithi096.stdout: "a", 2024-05-28T06:12:01.133 INFO:teuthology.orchestra.run.smithi096.stdout: "b", 2024-05-28T06:12:01.133 INFO:teuthology.orchestra.run.smithi096.stdout: "c" 2024-05-28T06:12:01.133 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:12:01.133 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_age": 494, 2024-05-28T06:12:01.134 INFO:teuthology.orchestra.run.smithi096.stdout: "monmap": { 2024-05-28T06:12:01.134 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 1, 2024-05-28T06:12:01.134 INFO:teuthology.orchestra.run.smithi096.stdout: "min_mon_release_name": "squid", 2024-05-28T06:12:01.134 INFO:teuthology.orchestra.run.smithi096.stdout: "num_mons": 3 2024-05-28T06:12:01.134 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:12:01.134 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap": { 2024-05-28T06:12:01.134 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 47, 2024-05-28T06:12:01.134 INFO:teuthology.orchestra.run.smithi096.stdout: "num_osds": 8, 2024-05-28T06:12:01.134 INFO:teuthology.orchestra.run.smithi096.stdout: "num_up_osds": 8, 2024-05-28T06:12:01.134 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_up_since": 1716876238, 2024-05-28T06:12:01.134 INFO:teuthology.orchestra.run.smithi096.stdout: "num_in_osds": 8, 2024-05-28T06:12:01.134 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_in_since": 1716876229, 2024-05-28T06:12:01.134 INFO:teuthology.orchestra.run.smithi096.stdout: "num_remapped_pgs": 0 2024-05-28T06:12:01.135 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:12:01.135 INFO:teuthology.orchestra.run.smithi096.stdout: "pgmap": { 2024-05-28T06:12:01.135 INFO:teuthology.orchestra.run.smithi096.stdout: "pgs_by_state": [ 2024-05-28T06:12:01.135 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:12:01.135 INFO:teuthology.orchestra.run.smithi096.stdout: "state_name": "active+clean", 2024-05-28T06:12:01.135 INFO:teuthology.orchestra.run.smithi096.stdout: "count": 97 2024-05-28T06:12:01.135 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:12:01.135 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:12:01.135 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pgs": 97, 2024-05-28T06:12:01.135 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pools": 3, 2024-05-28T06:12:01.135 INFO:teuthology.orchestra.run.smithi096.stdout: "num_objects": 4176, 2024-05-28T06:12:01.135 INFO:teuthology.orchestra.run.smithi096.stdout: "data_bytes": 17383168377, 2024-05-28T06:12:01.135 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_used": 35149873152, 2024-05-28T06:12:01.135 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_avail": 737944240128, 2024-05-28T06:12:01.136 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_total": 773094113280 2024-05-28T06:12:01.136 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:12:01.136 INFO:teuthology.orchestra.run.smithi096.stdout: "fsmap": { 2024-05-28T06:12:01.136 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 26, 2024-05-28T06:12:01.136 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 3, 2024-05-28T06:12:01.136 INFO:teuthology.orchestra.run.smithi096.stdout: "up": 1, 2024-05-28T06:12:01.136 INFO:teuthology.orchestra.run.smithi096.stdout: "in": 1, 2024-05-28T06:12:01.136 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 1, 2024-05-28T06:12:01.136 INFO:teuthology.orchestra.run.smithi096.stdout: "by_rank": [ 2024-05-28T06:12:01.136 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:12:01.136 INFO:teuthology.orchestra.run.smithi096.stdout: "filesystem_id": 3, 2024-05-28T06:12:01.136 INFO:teuthology.orchestra.run.smithi096.stdout: "rank": 0, 2024-05-28T06:12:01.136 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "a", 2024-05-28T06:12:01.136 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "up:active", 2024-05-28T06:12:01.137 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 5859 2024-05-28T06:12:01.137 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:12:01.137 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:12:01.137 INFO:teuthology.orchestra.run.smithi096.stdout: "up:standby": 3 2024-05-28T06:12:01.137 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:12:01.137 INFO:teuthology.orchestra.run.smithi096.stdout: "mgrmap": { 2024-05-28T06:12:01.137 INFO:teuthology.orchestra.run.smithi096.stdout: "available": true, 2024-05-28T06:12:01.137 INFO:teuthology.orchestra.run.smithi096.stdout: "num_standbys": 1, 2024-05-28T06:12:01.137 INFO:teuthology.orchestra.run.smithi096.stdout: "modules": [ 2024-05-28T06:12:01.137 INFO:teuthology.orchestra.run.smithi096.stdout: "iostat", 2024-05-28T06:12:01.137 INFO:teuthology.orchestra.run.smithi096.stdout: "nfs", 2024-05-28T06:12:01.137 INFO:teuthology.orchestra.run.smithi096.stdout: "restful" 2024-05-28T06:12:01.137 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:12:01.138 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:12:01.138 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:12:01.138 INFO:teuthology.orchestra.run.smithi096.stdout: "servicemap": { 2024-05-28T06:12:01.138 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 9, 2024-05-28T06:12:01.138 INFO:teuthology.orchestra.run.smithi096.stdout: "modified": "2024-05-28T06:10:01.618312+0000", 2024-05-28T06:12:01.138 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:12:01.138 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:12:01.138 INFO:teuthology.orchestra.run.smithi096.stdout: "progress_events": {} 2024-05-28T06:12:01.138 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:12:01.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.133+0000 7fb72f200640 1 -- 172.21.15.96:0/456155787 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb70c06cfe0 msgr2=0x7fb70c06f4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:01.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.133+0000 7fb72f200640 1 --2- 172.21.15.96:0/456155787 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb70c06cfe0 0x7fb70c06f4a0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fb720001050 tx=0x7fb720001470 comp rx=0 tx=0).stop 2024-05-28T06:12:01.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.133+0000 7fb72f200640 1 -- 172.21.15.96:0/456155787 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb73014a110 msgr2=0x7fb73014a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:01.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.133+0000 7fb72f200640 1 --2- 172.21.15.96:0/456155787 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb73014a110 0x7fb73014a590 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fb72802aee0 tx=0x7fb728009750 comp rx=0 tx=0).stop 2024-05-28T06:12:01.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.133+0000 7fb72f200640 1 -- 172.21.15.96:0/456155787 shutdown_connections 2024-05-28T06:12:01.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.133+0000 7fb72f200640 1 --2- 172.21.15.96:0/456155787 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb70c06cfe0 0x7fb70c06f4a0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:01.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.133+0000 7fb72f200640 1 --2- 172.21.15.96:0/456155787 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb73014a110 0x7fb73014a590 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:01.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.133+0000 7fb72f200640 1 --2- 172.21.15.96:0/456155787 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb730072280 0x7fb730072700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:01.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.133+0000 7fb72f200640 1 --2- 172.21.15.96:0/456155787 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb714004d40 0x7fb730099a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:01.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.133+0000 7fb72f200640 1 -- 172.21.15.96:0/456155787 >> 172.21.15.96:0/456155787 conn(0x7fb73006bae0 msgr2=0x7fb7300738a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:01.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.133+0000 7fb72f200640 1 -- 172.21.15.96:0/456155787 shutdown_connections 2024-05-28T06:12:01.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.133+0000 7fb72f200640 1 -- 172.21.15.96:0/456155787 wait complete. 2024-05-28T06:12:01.151 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume ls cephfs --format json 2024-05-28T06:12:01.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.395+0000 7f4e25e00640 1 Processor -- start 2024-05-28T06:12:01.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.395+0000 7f4e25e00640 1 -- start start 2024-05-28T06:12:01.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.396+0000 7f4e25e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f4e2015d5d0 0x7f4e200cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:01.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.396+0000 7f4e25e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4e200b32c0 0x7f4e200b3750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:01.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.396+0000 7f4e25e00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e2015cb10 con 0x7f4e200d0640 2024-05-28T06:12:01.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.397+0000 7f4e25e00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f4e200adbb0 con 0x7f4e2015d5d0 2024-05-28T06:12:01.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.397+0000 7f4e25e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f4e200ad5f0 con 0x7f4e200b32c0 2024-05-28T06:12:01.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.397+0000 7f4e1ea00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f4e200d0640 0x7f4e200d0a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:59696/0 (socket says 172.21.15.96:59696) 2024-05-28T06:12:01.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.397+0000 7f4e1ea00640 1 -- 172.21.15.96:0/3852248305 learned_addr learned my addr 172.21.15.96:0/3852248305 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:12:01.400 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.397+0000 7f4e1e000640 1 --2- 172.21.15.96:0/3852248305 >> v2:172.21.15.123:3300/0 conn(0x7f4e2015d5d0 0x7f4e200cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:01.400 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.397+0000 7f4e1f400640 1 --2- 172.21.15.96:0/3852248305 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4e200b32c0 0x7f4e200b3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:01.400 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.397+0000 7f4e1d600640 1 -- 172.21.15.96:0/3852248305 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1337131970 0 0) 0x7f4e2015cb10 con 0x7f4e200d0640 2024-05-28T06:12:01.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.398+0000 7f4e1d600640 1 -- 172.21.15.96:0/3852248305 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4e04003520 con 0x7f4e200d0640 2024-05-28T06:12:01.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.398+0000 7f4e1d600640 1 -- 172.21.15.96:0/3852248305 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 385842240 0 0) 0x7f4e04003520 con 0x7f4e200d0640 2024-05-28T06:12:01.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.398+0000 7f4e1d600640 1 -- 172.21.15.96:0/3852248305 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4e200b32c0 msgr2=0x7f4e200b3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:01.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.398+0000 7f4e1d600640 1 --2- 172.21.15.96:0/3852248305 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4e200b32c0 0x7f4e200b3750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:01.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.398+0000 7f4e1d600640 1 -- 172.21.15.96:0/3852248305 >> v2:172.21.15.123:3300/0 conn(0x7f4e2015d5d0 msgr2=0x7f4e200cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:01.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.398+0000 7f4e1d600640 1 --2- 172.21.15.96:0/3852248305 >> v2:172.21.15.123:3300/0 conn(0x7f4e2015d5d0 0x7f4e200cf950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:01.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.398+0000 7f4e1d600640 1 -- 172.21.15.96:0/3852248305 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e200ad8d0 con 0x7f4e200d0640 2024-05-28T06:12:01.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.398+0000 7f4e1d600640 1 -- 172.21.15.96:0/3852248305 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f4e10001690 con 0x7f4e200d0640 2024-05-28T06:12:01.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.398+0000 7f4e1d600640 1 -- 172.21.15.96:0/3852248305 >> v1:172.21.15.96:6789/0 conn(0x7f4e200d0640 legacy=0x7f4e200d0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:01.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.398+0000 7f4e1d600640 1 --2- 172.21.15.96:0/3852248305 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e040039b0 0x7f4e04003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:01.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.398+0000 7f4e1d600640 1 --2- 172.21.15.96:0/3852248305 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e04004800 0x7f4e04004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:01.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.398+0000 7f4e1d600640 1 --2- 172.21.15.96:0/3852248305 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4e200b32c0 0x7f4e040051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:01.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.398+0000 7f4e1d600640 1 -- 172.21.15.96:0/3852248305 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4e200ad5f0 con 0x7f4e04004800 2024-05-28T06:12:01.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.398+0000 7f4e1d600640 1 -- 172.21.15.96:0/3852248305 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f4e0400b9c0 con 0x7f4e040039b0 2024-05-28T06:12:01.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.398+0000 7f4e1d600640 1 -- 172.21.15.96:0/3852248305 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f4e0400bb40 con 0x7f4e200b32c0 2024-05-28T06:12:01.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.398+0000 7f4e1e000640 1 --2- 172.21.15.96:0/3852248305 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e04004800 0x7f4e04004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:01.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.399+0000 7f4e1e000640 1 -- 172.21.15.96:0/3852248305 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4e200b32c0 msgr2=0x7f4e040051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:01.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.399+0000 7f4e1e000640 1 --2- 172.21.15.96:0/3852248305 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4e200b32c0 0x7f4e040051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:01.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.399+0000 7f4e1e000640 1 -- 172.21.15.96:0/3852248305 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e040039b0 msgr2=0x7f4e04003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:01.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.399+0000 7f4e1e000640 1 --2- 172.21.15.96:0/3852248305 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e040039b0 0x7f4e04003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:01.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.399+0000 7f4e1e000640 1 -- 172.21.15.96:0/3852248305 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e04003520 con 0x7f4e04004800 2024-05-28T06:12:01.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.400+0000 7f4e1e000640 1 --2- 172.21.15.96:0/3852248305 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e04004800 0x7f4e04004c80 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f4e0c009ac0 tx=0x7f4e04005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a2c3a8512bc4c20 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:01.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.400+0000 7f4e1d600640 1 -- 172.21.15.96:0/3852248305 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e0c036070 con 0x7f4e04004800 2024-05-28T06:12:01.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.400+0000 7f4e1d600640 1 -- 172.21.15.96:0/3852248305 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e0c031400 con 0x7f4e04004800 2024-05-28T06:12:01.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.400+0000 7f4e1d600640 1 -- 172.21.15.96:0/3852248305 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e0c009690 con 0x7f4e04004800 2024-05-28T06:12:01.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.401+0000 7f4e25e00640 1 -- 172.21.15.96:0/3852248305 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e04004800 msgr2=0x7f4e04004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:01.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.401+0000 7f4e25e00640 1 --2- 172.21.15.96:0/3852248305 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e04004800 0x7f4e04004c80 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f4e0c009ac0 tx=0x7f4e04005830 comp rx=0 tx=0).stop 2024-05-28T06:12:01.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.401+0000 7f4e1f400640 1 -- 172.21.15.96:0/3852248305 reap_dead start 2024-05-28T06:12:01.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.401+0000 7f4e25e00640 1 -- 172.21.15.96:0/3852248305 shutdown_connections 2024-05-28T06:12:01.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.401+0000 7f4e25e00640 1 -- 172.21.15.96:0/3852248305 >> 172.21.15.96:0/3852248305 conn(0x7f4e2006bd20 msgr2=0x7f4e2006c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:01.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.401+0000 7f4e25e00640 1 -- 172.21.15.96:0/3852248305 shutdown_connections 2024-05-28T06:12:01.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.401+0000 7f4e25e00640 1 -- 172.21.15.96:0/3852248305 wait complete. 2024-05-28T06:12:01.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.403+0000 7f4e25e00640 1 Processor -- start 2024-05-28T06:12:01.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.403+0000 7f4e25e00640 1 -- start start 2024-05-28T06:12:01.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.404+0000 7f4e25e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e04004800 0x7f4e200b2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:01.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.404+0000 7f4e25e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e2006d9c0 0x7f4e2006f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:01.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.404+0000 7f4e25e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4e201f5030 0x7f4e20155470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:01.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.404+0000 7f4e25e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4e2015da50 con 0x7f4e2006d9c0 2024-05-28T06:12:01.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.404+0000 7f4e25e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f4e2015d0b0 con 0x7f4e04004800 2024-05-28T06:12:01.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.404+0000 7f4e25e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f4e200ad2a0 con 0x7f4e201f5030 2024-05-28T06:12:01.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.404+0000 7f4e1e000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e2006d9c0 0x7f4e2006f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:01.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.404+0000 7f4e1e000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e2006d9c0 0x7f4e2006f390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:34726/0 (socket says 172.21.15.96:34726) 2024-05-28T06:12:01.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.404+0000 7f4e1e000640 1 -- 172.21.15.96:0/1256515176 learned_addr learned my addr 172.21.15.96:0/1256515176 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:01.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.404+0000 7f4e1ea00640 1 --2- 172.21.15.96:0/1256515176 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e04004800 0x7f4e200b2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:01.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.404+0000 7f4e1e000640 1 -- 172.21.15.96:0/1256515176 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4e201f5030 msgr2=0x7f4e20155470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:01.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.404+0000 7f4e1f400640 1 --2- 172.21.15.96:0/1256515176 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4e201f5030 0x7f4e20155470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:01.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.404+0000 7f4e1e000640 1 --2- 172.21.15.96:0/1256515176 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4e201f5030 0x7f4e20155470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:01.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.405+0000 7f4e1e000640 1 -- 172.21.15.96:0/1256515176 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e04004800 msgr2=0x7f4e200b2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:01.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.405+0000 7f4e1e000640 1 --2- 172.21.15.96:0/1256515176 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e04004800 0x7f4e200b2100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:01.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.405+0000 7f4e1e000640 1 -- 172.21.15.96:0/1256515176 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e2006ef90 con 0x7f4e2006d9c0 2024-05-28T06:12:01.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.405+0000 7f4e1ea00640 1 --2- 172.21.15.96:0/1256515176 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e04004800 0x7f4e200b2100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:12:01.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.405+0000 7f4e1e000640 1 --2- 172.21.15.96:0/1256515176 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e2006d9c0 0x7f4e2006f390 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f4e0c034040 tx=0x7f4e0c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:01.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.405+0000 7f4e17400640 1 -- 172.21.15.96:0/1256515176 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e0c040020 con 0x7f4e2006d9c0 2024-05-28T06:12:01.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.405+0000 7f4e17400640 1 -- 172.21.15.96:0/1256515176 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e0c0038d0 con 0x7f4e2006d9c0 2024-05-28T06:12:01.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.405+0000 7f4e17400640 1 -- 172.21.15.96:0/1256515176 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e0c036040 con 0x7f4e2006d9c0 2024-05-28T06:12:01.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.405+0000 7f4e25e00640 1 -- 172.21.15.96:0/1256515176 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e2006f8d0 con 0x7f4e2006d9c0 2024-05-28T06:12:01.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.405+0000 7f4e25e00640 1 -- 172.21.15.96:0/1256515176 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e2006de50 con 0x7f4e2006d9c0 2024-05-28T06:12:01.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.407+0000 7f4e25e00640 1 -- 172.21.15.96:0/1256515176 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e200689e0 con 0x7f4e2006d9c0 2024-05-28T06:12:01.410 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.407+0000 7f4e17400640 1 -- 172.21.15.96:0/1256515176 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 15) ==== 91652+0+0 (secure 0 0 0) 0x7f4e0c043020 con 0x7f4e2006d9c0 2024-05-28T06:12:01.410 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.408+0000 7f4e17400640 1 --2- 172.21.15.96:0/1256515176 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4e0006cfb0 0x7f4e0006f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:01.410 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.408+0000 7f4e17400640 1 -- 172.21.15.96:0/1256515176 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5795+0+0 (secure 0 0 0) 0x7f4e0c0ac9c0 con 0x7f4e2006d9c0 2024-05-28T06:12:01.410 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.409+0000 7f4e1ea00640 1 --2- 172.21.15.96:0/1256515176 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4e0006cfb0 0x7f4e0006f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:01.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.410+0000 7f4e1ea00640 1 --2- 172.21.15.96:0/1256515176 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4e0006cfb0 0x7f4e0006f470 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f4e10001eb0 tx=0x7f4e10009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:01.414 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.413+0000 7f4e17400640 1 -- 172.21.15.96:0/1256515176 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4e0c04a070 con 0x7f4e2006d9c0 2024-05-28T06:12:01.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.641+0000 7f4e25e00640 1 -- 172.21.15.96:0/1256515176 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume ls", "vol_name": "cephfs", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4e20071ef0 con 0x7f4e0006cfb0 2024-05-28T06:12:01.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.657+0000 7f4e17400640 1 -- 172.21.15.96:0/1256515176 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+234 (secure 0 0 0) 0x7f4e20071ef0 con 0x7f4e0006cfb0 2024-05-28T06:12:01.659 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:12:01.659 INFO:teuthology.orchestra.run.smithi096.stdout:[ 2024-05-28T06:12:01.659 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:12:01.659 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "sv1" 2024-05-28T06:12:01.659 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:12:01.659 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:12:01.659 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "subvol_clone26284" 2024-05-28T06:12:01.659 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:12:01.660 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:12:01.660 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "subvol_clone26282" 2024-05-28T06:12:01.660 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:12:01.660 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:12:01.660 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "subvol_clone26283" 2024-05-28T06:12:01.660 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:12:01.660 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:12:01.660 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "subvol_clone26281" 2024-05-28T06:12:01.660 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:12:01.660 INFO:teuthology.orchestra.run.smithi096.stdout:] 2024-05-28T06:12:01.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.660+0000 7f4e25e00640 1 -- 172.21.15.96:0/1256515176 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4e0006cfb0 msgr2=0x7f4e0006f470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:01.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.660+0000 7f4e25e00640 1 --2- 172.21.15.96:0/1256515176 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4e0006cfb0 0x7f4e0006f470 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f4e10001eb0 tx=0x7f4e10009040 comp rx=0 tx=0).stop 2024-05-28T06:12:01.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.660+0000 7f4e25e00640 1 -- 172.21.15.96:0/1256515176 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e2006d9c0 msgr2=0x7f4e2006f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:01.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.660+0000 7f4e25e00640 1 --2- 172.21.15.96:0/1256515176 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e2006d9c0 0x7f4e2006f390 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f4e0c034040 tx=0x7f4e0c004730 comp rx=0 tx=0).stop 2024-05-28T06:12:01.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.660+0000 7f4e25e00640 1 -- 172.21.15.96:0/1256515176 shutdown_connections 2024-05-28T06:12:01.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.660+0000 7f4e25e00640 1 --2- 172.21.15.96:0/1256515176 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4e0006cfb0 0x7f4e0006f470 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:01.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.660+0000 7f4e25e00640 1 --2- 172.21.15.96:0/1256515176 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4e201f5030 0x7f4e20155470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:01.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.660+0000 7f4e25e00640 1 --2- 172.21.15.96:0/1256515176 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e2006d9c0 0x7f4e2006f390 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:01.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.660+0000 7f4e25e00640 1 --2- 172.21.15.96:0/1256515176 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e04004800 0x7f4e200b2100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:01.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.660+0000 7f4e25e00640 1 -- 172.21.15.96:0/1256515176 >> 172.21.15.96:0/1256515176 conn(0x7f4e2006bd20 msgr2=0x7f4e200721a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:01.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.660+0000 7f4e25e00640 1 -- 172.21.15.96:0/1256515176 shutdown_connections 2024-05-28T06:12:01.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.660+0000 7f4e25e00640 1 -- 172.21.15.96:0/1256515176 wait complete. 2024-05-28T06:12:01.682 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot ls cephfs sv1 --format json 2024-05-28T06:12:01.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.883+0000 7f1a1a400640 1 Processor -- start 2024-05-28T06:12:01.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.884+0000 7f1a1a400640 1 -- start start 2024-05-28T06:12:01.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.885+0000 7f1a1a400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f1a140cf2f0 0x7f1a140cf910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:01.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.885+0000 7f1a1a400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f1a14140500 0x7f1a140ad130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:01.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.885+0000 7f1a1a400640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a1406dd50 con 0x7f1a1406edf0 2024-05-28T06:12:01.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.885+0000 7f1a1a400640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f1a1406e030 con 0x7f1a14140500 2024-05-28T06:12:01.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.885+0000 7f1a1a400640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f1a1406e310 con 0x7f1a140cf2f0 2024-05-28T06:12:01.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.885+0000 7f1a13e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f1a14140500 0x7f1a140ad130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:01.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.885+0000 7f1a13e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f1a14140500 0x7f1a140ad130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:34734/0 (socket says 172.21.15.96:34734) 2024-05-28T06:12:01.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.885+0000 7f1a13e00640 1 -- 172.21.15.96:0/2611854085 learned_addr learned my addr 172.21.15.96:0/2611854085 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:01.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.886+0000 7f1a13e00640 1 -- 172.21.15.96:0/2611854085 >> v2:172.21.15.123:3300/0 conn(0x7f1a140cf2f0 msgr2=0x7f1a140cf910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:01.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.886+0000 7f1a13400640 1 --2- 172.21.15.96:0/2611854085 >> v2:172.21.15.123:3300/0 conn(0x7f1a140cf2f0 0x7f1a140cf910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:01.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.886+0000 7f1a13e00640 1 --2- 172.21.15.96:0/2611854085 >> v2:172.21.15.123:3300/0 conn(0x7f1a140cf2f0 0x7f1a140cf910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:01.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.886+0000 7f1a13e00640 1 -- 172.21.15.96:0/2611854085 >> v1:172.21.15.123:6789/0 conn(0x7f1a1406edf0 legacy=0x7f1a1413fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:01.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.886+0000 7f1a13e00640 1 -- 172.21.15.96:0/2611854085 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a1406e5f0 con 0x7f1a14140500 2024-05-28T06:12:01.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.887+0000 7f1a13e00640 1 --2- 172.21.15.96:0/2611854085 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a14140500 0x7f1a140ad130 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f1a140b6170 tx=0x7f1a0c02a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=964e7f771ded231e server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:01.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.887+0000 7f1a12a00640 1 -- 172.21.15.96:0/2611854085 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a0c036070 con 0x7f1a14140500 2024-05-28T06:12:01.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.887+0000 7f1a12a00640 1 -- 172.21.15.96:0/2611854085 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a0c031600 con 0x7f1a14140500 2024-05-28T06:12:01.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.887+0000 7f1a12a00640 1 -- 172.21.15.96:0/2611854085 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a0c0319d0 con 0x7f1a14140500 2024-05-28T06:12:01.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.888+0000 7f1a1a400640 1 -- 172.21.15.96:0/2611854085 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a14140500 msgr2=0x7f1a140ad130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:01.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.888+0000 7f1a1a400640 1 --2- 172.21.15.96:0/2611854085 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a14140500 0x7f1a140ad130 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f1a140b6170 tx=0x7f1a0c02a660 comp rx=0 tx=0).stop 2024-05-28T06:12:01.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.888+0000 7f1a1a400640 1 -- 172.21.15.96:0/2611854085 shutdown_connections 2024-05-28T06:12:01.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.888+0000 7f1a1a400640 1 --2- 172.21.15.96:0/2611854085 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a14140500 0x7f1a140ad130 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:01.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.888+0000 7f1a1a400640 1 --2- 172.21.15.96:0/2611854085 >> v2:172.21.15.123:3300/0 conn(0x7f1a140cf2f0 0x7f1a140cf910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:01.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.888+0000 7f1a1a400640 1 -- 172.21.15.96:0/2611854085 >> 172.21.15.96:0/2611854085 conn(0x7f1a140b1e50 msgr2=0x7f1a140b2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:01.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.888+0000 7f1a1a400640 1 -- 172.21.15.96:0/2611854085 shutdown_connections 2024-05-28T06:12:01.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.889+0000 7f1a1a400640 1 -- 172.21.15.96:0/2611854085 wait complete. 2024-05-28T06:12:01.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.890+0000 7f1a1a400640 1 Processor -- start 2024-05-28T06:12:01.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.890+0000 7f1a1a400640 1 -- start start 2024-05-28T06:12:01.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.891+0000 7f1a1a400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1a1406edf0 0x7f1a14144c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:01.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.891+0000 7f1a1a400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a140cf2f0 0x7f1a14159af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:01.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.891+0000 7f1a1a400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a14143100 0x7f1a141435b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:01.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.891+0000 7f1a1a400640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f1a141461a0 con 0x7f1a14143100 2024-05-28T06:12:01.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.891+0000 7f1a1a400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f1a14146020 con 0x7f1a140cf2f0 2024-05-28T06:12:01.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.891+0000 7f1a1a400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f1a140cfe50 con 0x7f1a1406edf0 2024-05-28T06:12:01.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.891+0000 7f1a13e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a14143100 0x7f1a141435b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:01.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.892+0000 7f1a13e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a14143100 0x7f1a141435b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:34744/0 (socket says 172.21.15.96:34744) 2024-05-28T06:12:01.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.892+0000 7f1a13e00640 1 -- 172.21.15.96:0/1861407656 learned_addr learned my addr 172.21.15.96:0/1861407656 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:01.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.892+0000 7f1a0be00640 1 --2- 172.21.15.96:0/1861407656 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a140cf2f0 0x7f1a14159af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:01.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.892+0000 7f1a13e00640 1 -- 172.21.15.96:0/1861407656 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1a1406edf0 msgr2=0x7f1a14144c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:01.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.892+0000 7f1a13e00640 1 --2- 172.21.15.96:0/1861407656 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1a1406edf0 0x7f1a14144c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:01.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.892+0000 7f1a13e00640 1 -- 172.21.15.96:0/1861407656 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a140cf2f0 msgr2=0x7f1a14159af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:01.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.892+0000 7f1a13e00640 1 --2- 172.21.15.96:0/1861407656 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a140cf2f0 0x7f1a14159af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:01.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.892+0000 7f1a13e00640 1 -- 172.21.15.96:0/1861407656 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a14143bc0 con 0x7f1a14143100 2024-05-28T06:12:01.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.892+0000 7f1a13e00640 1 --2- 172.21.15.96:0/1861407656 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a14143100 0x7f1a141435b0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f1a0c009f40 tx=0x7f1a0c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:01.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.893+0000 7f1a0b400640 1 -- 172.21.15.96:0/1861407656 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a0c036050 con 0x7f1a14143100 2024-05-28T06:12:01.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.893+0000 7f1a0b400640 1 -- 172.21.15.96:0/1861407656 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a0c042070 con 0x7f1a14143100 2024-05-28T06:12:01.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.893+0000 7f1a0b400640 1 -- 172.21.15.96:0/1861407656 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a0c009510 con 0x7f1a14143100 2024-05-28T06:12:01.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.893+0000 7f1a1a400640 1 -- 172.21.15.96:0/1861407656 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a14143ea0 con 0x7f1a14143100 2024-05-28T06:12:01.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.893+0000 7f1a1a400640 1 -- 172.21.15.96:0/1861407656 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a14144460 con 0x7f1a14143100 2024-05-28T06:12:01.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.894+0000 7f1a1a400640 1 -- 172.21.15.96:0/1861407656 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a140689e0 con 0x7f1a14143100 2024-05-28T06:12:01.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.895+0000 7f1a0b400640 1 -- 172.21.15.96:0/1861407656 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 15) ==== 91652+0+0 (secure 0 0 0) 0x7f1a0c007bd0 con 0x7f1a14143100 2024-05-28T06:12:01.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.896+0000 7f1a0b400640 1 --2- 172.21.15.96:0/1861407656 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f19f006cfb0 0x7f19f006f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:01.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.896+0000 7f1a0b400640 1 -- 172.21.15.96:0/1861407656 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5795+0+0 (secure 0 0 0) 0x7f1a0c0ad700 con 0x7f1a14143100 2024-05-28T06:12:01.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.896+0000 7f1a13400640 1 --2- 172.21.15.96:0/1861407656 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f19f006cfb0 0x7f19f006f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:01.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.897+0000 7f1a13400640 1 --2- 172.21.15.96:0/1861407656 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f19f006cfb0 0x7f19f006f470 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f1a04006fd0 tx=0x7f1a04008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:01.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:01.900+0000 7f1a0b400640 1 -- 172.21.15.96:0/1861407656 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1a0c07b1f0 con 0x7f1a14143100 2024-05-28T06:12:02.133 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.132+0000 7f1a1a400640 1 -- 172.21.15.96:0/1861407656 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot ls", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1a1415c6a0 con 0x7f19f006cfb0 2024-05-28T06:12:02.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.160+0000 7f1a0b400640 1 -- 172.21.15.96:0/1861407656 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f1a1415c6a0 con 0x7f19f006cfb0 2024-05-28T06:12:02.161 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:12:02.162 INFO:teuthology.orchestra.run.smithi096.stdout:[ 2024-05-28T06:12:02.162 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:12:02.162 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "ss1" 2024-05-28T06:12:02.162 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:12:02.162 INFO:teuthology.orchestra.run.smithi096.stdout:] 2024-05-28T06:12:02.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.162+0000 7f1a1a400640 1 -- 172.21.15.96:0/1861407656 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f19f006cfb0 msgr2=0x7f19f006f470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:02.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.162+0000 7f1a1a400640 1 --2- 172.21.15.96:0/1861407656 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f19f006cfb0 0x7f19f006f470 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f1a04006fd0 tx=0x7f1a04008040 comp rx=0 tx=0).stop 2024-05-28T06:12:02.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.162+0000 7f1a1a400640 1 -- 172.21.15.96:0/1861407656 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a14143100 msgr2=0x7f1a141435b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:02.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.162+0000 7f1a1a400640 1 --2- 172.21.15.96:0/1861407656 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a14143100 0x7f1a141435b0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f1a0c009f40 tx=0x7f1a0c004a40 comp rx=0 tx=0).stop 2024-05-28T06:12:02.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.163+0000 7f1a1a400640 1 -- 172.21.15.96:0/1861407656 shutdown_connections 2024-05-28T06:12:02.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.163+0000 7f1a1a400640 1 --2- 172.21.15.96:0/1861407656 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f19f006cfb0 0x7f19f006f470 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:02.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.163+0000 7f1a1a400640 1 --2- 172.21.15.96:0/1861407656 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a14143100 0x7f1a141435b0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:02.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.163+0000 7f1a1a400640 1 --2- 172.21.15.96:0/1861407656 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a140cf2f0 0x7f1a14159af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:02.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.163+0000 7f1a1a400640 1 --2- 172.21.15.96:0/1861407656 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1a1406edf0 0x7f1a14144c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:02.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.163+0000 7f1a1a400640 1 -- 172.21.15.96:0/1861407656 >> 172.21.15.96:0/1861407656 conn(0x7f1a140b1e50 msgr2=0x7f1a1415bc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:02.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.163+0000 7f1a1a400640 1 -- 172.21.15.96:0/1861407656 shutdown_connections 2024-05-28T06:12:02.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.163+0000 7f1a1a400640 1 -- 172.21.15.96:0/1861407656 wait complete. 2024-05-28T06:12:02.184 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-05-28T06:12:02.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.384+0000 7fccd7000640 1 Processor -- start 2024-05-28T06:12:02.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.384+0000 7fccd7000640 1 -- start start 2024-05-28T06:12:02.386 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.385+0000 7fccd7000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fccd015d5d0 0x7fccd00cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:02.387 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.385+0000 7fccd7000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fccd00b32c0 0x7fccd00b3700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:02.387 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.385+0000 7fccd7000640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fccd015cb10 con 0x7fccd00d0640 2024-05-28T06:12:02.387 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.385+0000 7fccd7000640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fccd00adbb0 con 0x7fccd015d5d0 2024-05-28T06:12:02.387 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.385+0000 7fccd7000640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fccd00ad5f0 con 0x7fccd00b32c0 2024-05-28T06:12:02.387 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.385+0000 7fccc7400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fccd015d5d0 0x7fccd00cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:02.387 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.385+0000 7fcccfe00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fccd00d0640 0x7fccd00d0a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:59710/0 (socket says 172.21.15.96:59710) 2024-05-28T06:12:02.387 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.385+0000 7fccc7400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fccd015d5d0 0x7fccd00cf950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:34746/0 (socket says 172.21.15.96:34746) 2024-05-28T06:12:02.387 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.385+0000 7fcccfe00640 1 -- 172.21.15.96:0/4033900706 learned_addr learned my addr 172.21.15.96:0/4033900706 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:12:02.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.386+0000 7fccd5000640 1 --2- 172.21.15.96:0/4033900706 >> v2:172.21.15.123:3300/0 conn(0x7fccd00b32c0 0x7fccd00b3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:02.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.386+0000 7fccc7400640 1 -- 172.21.15.96:0/4033900706 >> v2:172.21.15.123:3300/0 conn(0x7fccd00b32c0 msgr2=0x7fccd00b3700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:02.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.386+0000 7fccc7400640 1 --2- 172.21.15.96:0/4033900706 >> v2:172.21.15.123:3300/0 conn(0x7fccd00b32c0 0x7fccd00b3700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:02.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.386+0000 7fccc7400640 1 -- 172.21.15.96:0/4033900706 >> v1:172.21.15.96:6789/0 conn(0x7fccd00d0640 legacy=0x7fccd00d0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:02.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.386+0000 7fccc7400640 1 -- 172.21.15.96:0/4033900706 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccd00ad8d0 con 0x7fccd015d5d0 2024-05-28T06:12:02.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.386+0000 7fccd5000640 1 --2- 172.21.15.96:0/4033900706 >> v2:172.21.15.123:3300/0 conn(0x7fccd00b32c0 0x7fccd00b3700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:12:02.389 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.387+0000 7fccc7400640 1 --2- 172.21.15.96:0/4033900706 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fccd015d5d0 0x7fccd00cf950 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fccd00d1200 tx=0x7fccb802a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=406d407e8796a8c0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:02.389 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.387+0000 7fcccf400640 1 -- 172.21.15.96:0/4033900706 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccb8037070 con 0x7fccd015d5d0 2024-05-28T06:12:02.389 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.388+0000 7fcccf400640 1 -- 172.21.15.96:0/4033900706 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fccb8032440 con 0x7fccd015d5d0 2024-05-28T06:12:02.389 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.388+0000 7fcccf400640 1 -- 172.21.15.96:0/4033900706 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccb802ae60 con 0x7fccd015d5d0 2024-05-28T06:12:02.390 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.388+0000 7fccd7000640 1 -- 172.21.15.96:0/4033900706 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fccd015d5d0 msgr2=0x7fccd00cf950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:02.390 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.388+0000 7fccd7000640 1 --2- 172.21.15.96:0/4033900706 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fccd015d5d0 0x7fccd00cf950 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fccd00d1200 tx=0x7fccb802a110 comp rx=0 tx=0).stop 2024-05-28T06:12:02.390 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.388+0000 7fccd7000640 1 -- 172.21.15.96:0/4033900706 shutdown_connections 2024-05-28T06:12:02.390 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.388+0000 7fccd7000640 1 --2- 172.21.15.96:0/4033900706 >> v2:172.21.15.123:3300/0 conn(0x7fccd00b32c0 0x7fccd00b3700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:02.390 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.388+0000 7fccd7000640 1 --2- 172.21.15.96:0/4033900706 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fccd015d5d0 0x7fccd00cf950 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:02.390 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.388+0000 7fccd7000640 1 -- 172.21.15.96:0/4033900706 >> 172.21.15.96:0/4033900706 conn(0x7fccd006bd20 msgr2=0x7fccd006c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:02.390 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.388+0000 7fccd7000640 1 -- 172.21.15.96:0/4033900706 shutdown_connections 2024-05-28T06:12:02.390 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.388+0000 7fccd7000640 1 -- 172.21.15.96:0/4033900706 wait complete. 2024-05-28T06:12:02.391 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.390+0000 7fccd7000640 1 Processor -- start 2024-05-28T06:12:02.391 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.390+0000 7fccd7000640 1 -- start start 2024-05-28T06:12:02.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.391+0000 7fccd7000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fccd00b32c0 0x7fccd00b0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:02.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.391+0000 7fccd7000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fccd00d0640 0x7fccd01f3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:02.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.391+0000 7fccd7000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fccd015d5d0 0x7fccd01f4200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:02.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.391+0000 7fccd7000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fccd015da50 con 0x7fccd00d0640 2024-05-28T06:12:02.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.391+0000 7fccd7000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fccd015d0b0 con 0x7fccd00b32c0 2024-05-28T06:12:02.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.391+0000 7fccd7000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fccd00ad2a0 con 0x7fccd015d5d0 2024-05-28T06:12:02.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.391+0000 7fccc7400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fccd00d0640 0x7fccd01f3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:02.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.391+0000 7fccc7400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fccd00d0640 0x7fccd01f3cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:34760/0 (socket says 172.21.15.96:34760) 2024-05-28T06:12:02.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.391+0000 7fccc7400640 1 -- 172.21.15.96:0/2991629489 learned_addr learned my addr 172.21.15.96:0/2991629489 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:02.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.392+0000 7fcccfe00640 1 --2- 172.21.15.96:0/2991629489 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fccd00b32c0 0x7fccd00b0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:02.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.392+0000 7fccc7400640 1 -- 172.21.15.96:0/2991629489 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fccd015d5d0 msgr2=0x7fccd01f4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:02.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.392+0000 7fccc7400640 1 --2- 172.21.15.96:0/2991629489 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fccd015d5d0 0x7fccd01f4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:02.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.392+0000 7fccc7400640 1 -- 172.21.15.96:0/2991629489 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fccd00b32c0 msgr2=0x7fccd00b0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:02.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.392+0000 7fccc7400640 1 --2- 172.21.15.96:0/2991629489 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fccd00b32c0 0x7fccd00b0bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:02.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.392+0000 7fccc7400640 1 -- 172.21.15.96:0/2991629489 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccd0155380 con 0x7fccd00d0640 2024-05-28T06:12:02.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.392+0000 7fcccfe00640 1 --2- 172.21.15.96:0/2991629489 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fccd00b32c0 0x7fccd00b0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:12:02.395 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.392+0000 7fccc7400640 1 --2- 172.21.15.96:0/2991629489 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fccd00d0640 0x7fccd01f3cc0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fccb8000c00 tx=0x7fccb80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:02.395 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.393+0000 7fccccc00640 1 -- 172.21.15.96:0/2991629489 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccb8037040 con 0x7fccd00d0640 2024-05-28T06:12:02.395 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.393+0000 7fccccc00640 1 -- 172.21.15.96:0/2991629489 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fccb8042070 con 0x7fccd00d0640 2024-05-28T06:12:02.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.393+0000 7fccccc00640 1 -- 172.21.15.96:0/2991629489 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccb8030bb0 con 0x7fccd00d0640 2024-05-28T06:12:02.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.393+0000 7fccd7000640 1 -- 172.21.15.96:0/2991629489 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fccd0155660 con 0x7fccd00d0640 2024-05-28T06:12:02.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.393+0000 7fccd7000640 1 -- 172.21.15.96:0/2991629489 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fccd0155c20 con 0x7fccd00d0640 2024-05-28T06:12:02.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.394+0000 7fccd7000640 1 -- 172.21.15.96:0/2991629489 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fccd00689e0 con 0x7fccd00d0640 2024-05-28T06:12:02.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.395+0000 7fccccc00640 1 -- 172.21.15.96:0/2991629489 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 15) ==== 91652+0+0 (secure 0 0 0) 0x7fccb8047460 con 0x7fccd00d0640 2024-05-28T06:12:02.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.396+0000 7fccccc00640 1 --2- 172.21.15.96:0/2991629489 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fccac06cfe0 0x7fccac06f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:02.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.396+0000 7fccccc00640 1 -- 172.21.15.96:0/2991629489 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5795+0+0 (secure 0 0 0) 0x7fccb80ad370 con 0x7fccd00d0640 2024-05-28T06:12:02.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.396+0000 7fcccfe00640 1 --2- 172.21.15.96:0/2991629489 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fccac06cfe0 0x7fccac06f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:02.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.398+0000 7fcccfe00640 1 --2- 172.21.15.96:0/2991629489 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fccac06cfe0 0x7fccac06f4a0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fccc00012e0 tx=0x7fccc0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:02.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.400+0000 7fccccc00640 1 -- 172.21.15.96:0/2991629489 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fccb8045360 con 0x7fccd00d0640 2024-05-28T06:12:02.638 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.636+0000 7fccd7000640 1 -- 172.21.15.96:0/2991629489 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fccd0071ef0 con 0x7fccac06cfe0 2024-05-28T06:12:02.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.703+0000 7fccccc00640 1 -- 172.21.15.96:0/2991629489 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fccd0071ef0 con 0x7fccac06cfe0 2024-05-28T06:12:02.705 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:12:02.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.706+0000 7fccd7000640 1 -- 172.21.15.96:0/2991629489 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fccac06cfe0 msgr2=0x7fccac06f4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:02.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.706+0000 7fccd7000640 1 --2- 172.21.15.96:0/2991629489 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fccac06cfe0 0x7fccac06f4a0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fccc00012e0 tx=0x7fccc0009040 comp rx=0 tx=0).stop 2024-05-28T06:12:02.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.706+0000 7fccd7000640 1 -- 172.21.15.96:0/2991629489 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fccd00d0640 msgr2=0x7fccd01f3cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:02.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.706+0000 7fccd7000640 1 --2- 172.21.15.96:0/2991629489 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fccd00d0640 0x7fccd01f3cc0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fccb8000c00 tx=0x7fccb80046c0 comp rx=0 tx=0).stop 2024-05-28T06:12:02.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.706+0000 7fccd7000640 1 -- 172.21.15.96:0/2991629489 shutdown_connections 2024-05-28T06:12:02.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.706+0000 7fccd7000640 1 --2- 172.21.15.96:0/2991629489 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fccac06cfe0 0x7fccac06f4a0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:02.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.706+0000 7fccd7000640 1 --2- 172.21.15.96:0/2991629489 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fccd015d5d0 0x7fccd01f4200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:02.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.706+0000 7fccd7000640 1 --2- 172.21.15.96:0/2991629489 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fccd00d0640 0x7fccd01f3cc0 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:02.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.706+0000 7fccd7000640 1 --2- 172.21.15.96:0/2991629489 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fccd00b32c0 0x7fccd00b0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:02.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.706+0000 7fccd7000640 1 -- 172.21.15.96:0/2991629489 >> 172.21.15.96:0/2991629489 conn(0x7fccd006bd20 msgr2=0x7fccd00b3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:02.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.706+0000 7fccd7000640 1 -- 172.21.15.96:0/2991629489 shutdown_connections 2024-05-28T06:12:02.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.706+0000 7fccd7000640 1 -- 172.21.15.96:0/2991629489 wait complete. 2024-05-28T06:12:02.728 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs sv1 2024-05-28T06:12:02.922 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.920+0000 7fb547200640 1 Processor -- start 2024-05-28T06:12:02.922 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.920+0000 7fb547200640 1 -- start start 2024-05-28T06:12:02.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.922+0000 7fb547200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb548153510 0x7fb5480b4d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:02.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.922+0000 7fb547200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fb5480ccb90 0x7fb5480ccff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:02.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.922+0000 7fb547200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb54815d900 con 0x7fb54815cb60 2024-05-28T06:12:02.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.922+0000 7fb547200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fb5481538f0 con 0x7fb5480ccb90 2024-05-28T06:12:02.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.922+0000 7fb547200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fb5480cd530 con 0x7fb548153510 2024-05-28T06:12:02.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.922+0000 7fb53fe00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fb5480ccb90 0x7fb5480ccff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:02.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.922+0000 7fb53fe00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fb5480ccb90 0x7fb5480ccff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:34770/0 (socket says 172.21.15.96:34770) 2024-05-28T06:12:02.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.922+0000 7fb53fe00640 1 -- 172.21.15.96:0/284353254 learned_addr learned my addr 172.21.15.96:0/284353254 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:02.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.923+0000 7fb545e00640 1 --2- 172.21.15.96:0/284353254 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb548153510 0x7fb5480b4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:02.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.923+0000 7fb53fe00640 1 -- 172.21.15.96:0/284353254 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb548153510 msgr2=0x7fb5480b4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:02.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.923+0000 7fb53fe00640 1 --2- 172.21.15.96:0/284353254 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb548153510 0x7fb5480b4d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:02.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.923+0000 7fb53fe00640 1 -- 172.21.15.96:0/284353254 >> v1:172.21.15.123:6789/0 conn(0x7fb54815cb60 legacy=0x7fb54815cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:02.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.923+0000 7fb53fe00640 1 -- 172.21.15.96:0/284353254 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5480b3f90 con 0x7fb5480ccb90 2024-05-28T06:12:02.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.924+0000 7fb53fe00640 1 --2- 172.21.15.96:0/284353254 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb5480ccb90 0x7fb5480ccff0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fb530009870 tx=0x7fb53002a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4987600687466766 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:02.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.924+0000 7fb545400640 1 -- 172.21.15.96:0/284353254 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb530037070 con 0x7fb5480ccb90 2024-05-28T06:12:02.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.924+0000 7fb545400640 1 -- 172.21.15.96:0/284353254 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb530032440 con 0x7fb5480ccb90 2024-05-28T06:12:02.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.924+0000 7fb545400640 1 -- 172.21.15.96:0/284353254 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb530032810 con 0x7fb5480ccb90 2024-05-28T06:12:02.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.925+0000 7fb547200640 1 -- 172.21.15.96:0/284353254 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb5480ccb90 msgr2=0x7fb5480ccff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:02.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.925+0000 7fb547200640 1 --2- 172.21.15.96:0/284353254 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb5480ccb90 0x7fb5480ccff0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fb530009870 tx=0x7fb53002a1c0 comp rx=0 tx=0).stop 2024-05-28T06:12:02.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.925+0000 7fb547200640 1 -- 172.21.15.96:0/284353254 shutdown_connections 2024-05-28T06:12:02.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.925+0000 7fb547200640 1 --2- 172.21.15.96:0/284353254 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb5480ccb90 0x7fb5480ccff0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:02.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.925+0000 7fb547200640 1 --2- 172.21.15.96:0/284353254 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb548153510 0x7fb5480b4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:02.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.925+0000 7fb547200640 1 -- 172.21.15.96:0/284353254 >> 172.21.15.96:0/284353254 conn(0x7fb54806b900 msgr2=0x7fb54806bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:02.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.925+0000 7fb547200640 1 -- 172.21.15.96:0/284353254 shutdown_connections 2024-05-28T06:12:02.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.925+0000 7fb547200640 1 -- 172.21.15.96:0/284353254 wait complete. 2024-05-28T06:12:02.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.927+0000 7fb547200640 1 Processor -- start 2024-05-28T06:12:02.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.927+0000 7fb547200640 1 -- start start 2024-05-28T06:12:02.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.928+0000 7fb547200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb5480ccb90 0x7fb54815db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:02.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.928+0000 7fb547200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb548153510 0x7fb5480ce820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:02.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.928+0000 7fb547200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb54815cb60 0x7fb5480ced60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:02.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.928+0000 7fb547200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb5480b4550 con 0x7fb5480ccb90 2024-05-28T06:12:02.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.928+0000 7fb547200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fb5480b48c0 con 0x7fb548153510 2024-05-28T06:12:02.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.928+0000 7fb547200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fb54806e940 con 0x7fb54815cb60 2024-05-28T06:12:02.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.928+0000 7fb545e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb5480ccb90 0x7fb54815db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:02.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.928+0000 7fb545e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb5480ccb90 0x7fb54815db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:34772/0 (socket says 172.21.15.96:34772) 2024-05-28T06:12:02.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.928+0000 7fb545e00640 1 -- 172.21.15.96:0/3328219335 learned_addr learned my addr 172.21.15.96:0/3328219335 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:02.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.929+0000 7fb53fe00640 1 --2- 172.21.15.96:0/3328219335 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb548153510 0x7fb5480ce820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:02.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.929+0000 7fb546800640 1 --2- 172.21.15.96:0/3328219335 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb54815cb60 0x7fb5480ced60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:02.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.929+0000 7fb545e00640 1 -- 172.21.15.96:0/3328219335 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb54815cb60 msgr2=0x7fb5480ced60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:02.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.929+0000 7fb545e00640 1 --2- 172.21.15.96:0/3328219335 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb54815cb60 0x7fb5480ced60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:02.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.929+0000 7fb545e00640 1 -- 172.21.15.96:0/3328219335 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb548153510 msgr2=0x7fb5480ce820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:02.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.929+0000 7fb545e00640 1 --2- 172.21.15.96:0/3328219335 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb548153510 0x7fb5480ce820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:02.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.929+0000 7fb545e00640 1 -- 172.21.15.96:0/3328219335 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb54815e550 con 0x7fb5480ccb90 2024-05-28T06:12:02.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.929+0000 7fb53fe00640 1 --2- 172.21.15.96:0/3328219335 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb548153510 0x7fb5480ce820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:12:02.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.929+0000 7fb545e00640 1 --2- 172.21.15.96:0/3328219335 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb5480ccb90 0x7fb54815db00 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fb53800d800 tx=0x7fb53800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:02.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.929+0000 7fb53e000640 1 -- 172.21.15.96:0/3328219335 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb53800b6c0 con 0x7fb5480ccb90 2024-05-28T06:12:02.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.929+0000 7fb53e000640 1 -- 172.21.15.96:0/3328219335 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb538019070 con 0x7fb5480ccb90 2024-05-28T06:12:02.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.929+0000 7fb53e000640 1 -- 172.21.15.96:0/3328219335 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb53800bac0 con 0x7fb5480ccb90 2024-05-28T06:12:02.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.929+0000 7fb547200640 1 -- 172.21.15.96:0/3328219335 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb54815e830 con 0x7fb5480ccb90 2024-05-28T06:12:02.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.930+0000 7fb547200640 1 -- 172.21.15.96:0/3328219335 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb54815edf0 con 0x7fb5480ccb90 2024-05-28T06:12:02.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.931+0000 7fb547200640 1 -- 172.21.15.96:0/3328219335 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5480689e0 con 0x7fb5480ccb90 2024-05-28T06:12:02.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.932+0000 7fb53e000640 1 -- 172.21.15.96:0/3328219335 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 15) ==== 91652+0+0 (secure 0 0 0) 0x7fb53800bc60 con 0x7fb5480ccb90 2024-05-28T06:12:02.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.933+0000 7fb53e000640 1 --2- 172.21.15.96:0/3328219335 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb52406cfb0 0x7fb52406f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:02.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.933+0000 7fb53e000640 1 -- 172.21.15.96:0/3328219335 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5795+0+0 (secure 0 0 0) 0x7fb53808cc30 con 0x7fb5480ccb90 2024-05-28T06:12:02.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.933+0000 7fb53fe00640 1 --2- 172.21.15.96:0/3328219335 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb52406cfb0 0x7fb52406f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:02.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.935+0000 7fb53fe00640 1 --2- 172.21.15.96:0/3328219335 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb52406cfb0 0x7fb52406f470 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fb530034000 tx=0x7fb53002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:02.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:02.937+0000 7fb53e000640 1 -- 172.21.15.96:0/3328219335 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb53805a730 con 0x7fb5480ccb90 2024-05-28T06:12:03.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.168+0000 7fb547200640 1 -- 172.21.15.96:0/3328219335 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) -- 0x7fb5480b3c40 con 0x7fb52406cfb0 2024-05-28T06:12:03.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.217+0000 7fb53e000640 1 -- 172.21.15.96:0/3328219335 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fb5480b3c40 con 0x7fb52406cfb0 2024-05-28T06:12:03.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.220+0000 7fb547200640 1 -- 172.21.15.96:0/3328219335 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb52406cfb0 msgr2=0x7fb52406f470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:03.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.220+0000 7fb547200640 1 --2- 172.21.15.96:0/3328219335 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb52406cfb0 0x7fb52406f470 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fb530034000 tx=0x7fb53002e000 comp rx=0 tx=0).stop 2024-05-28T06:12:03.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.220+0000 7fb547200640 1 -- 172.21.15.96:0/3328219335 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb5480ccb90 msgr2=0x7fb54815db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:03.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.220+0000 7fb547200640 1 --2- 172.21.15.96:0/3328219335 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb5480ccb90 0x7fb54815db00 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fb53800d800 tx=0x7fb53800dcc0 comp rx=0 tx=0).stop 2024-05-28T06:12:03.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.220+0000 7fb547200640 1 -- 172.21.15.96:0/3328219335 shutdown_connections 2024-05-28T06:12:03.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.220+0000 7fb547200640 1 --2- 172.21.15.96:0/3328219335 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb52406cfb0 0x7fb52406f470 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:03.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.221+0000 7fb547200640 1 --2- 172.21.15.96:0/3328219335 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb54815cb60 0x7fb5480ced60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:03.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.221+0000 7fb547200640 1 --2- 172.21.15.96:0/3328219335 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb548153510 0x7fb5480ce820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:03.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.221+0000 7fb547200640 1 --2- 172.21.15.96:0/3328219335 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb5480ccb90 0x7fb54815db00 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:03.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.221+0000 7fb547200640 1 -- 172.21.15.96:0/3328219335 >> 172.21.15.96:0/3328219335 conn(0x7fb54806b900 msgr2=0x7fb54806ed70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:03.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.221+0000 7fb547200640 1 -- 172.21.15.96:0/3328219335 shutdown_connections 2024-05-28T06:12:03.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.221+0000 7fb547200640 1 -- 172.21.15.96:0/3328219335 wait complete. 2024-05-28T06:12:03.228 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:12:03.227+0000 7fea5bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:12:03.228 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:12:03.227+0000 7fea5bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:12:03.228 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:12:03.227+0000 7fea5bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:12:03.229 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:12:03.227+0000 7fea5bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:12:03.229 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:12:03.227+0000 7fea5bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:12:03.246 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone26284 2024-05-28T06:12:03.258 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:12:03.257+0000 7fea5da00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:12:03.258 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:12:03.257+0000 7fea5da00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:12:03.258 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:12:03.257+0000 7fea5da00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:12:03.258 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:12:03.257+0000 7fea5da00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:12:03.258 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:12:03.257+0000 7fea5da00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:12:03.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.455+0000 7f8d6ec00640 1 Processor -- start 2024-05-28T06:12:03.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.455+0000 7f8d6ec00640 1 -- start start 2024-05-28T06:12:03.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.456+0000 7f8d6ec00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f8d6815e5a0 0x7f8d680d2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:03.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.456+0000 7f8d6ec00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8d680cfb40 0x7f8d680d0d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:03.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.456+0000 7f8d6ec00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f8d680cffa0 0x7f8d680d1250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:03.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.456+0000 7f8d6ec00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f8d680d2b30 con 0x7f8d680cffa0 2024-05-28T06:12:03.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.456+0000 7f8d6ec00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f8d680d3500 con 0x7f8d6815e5a0 2024-05-28T06:12:03.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.456+0000 7f8d6ec00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f8d680d2200 con 0x7f8d680cfb40 2024-05-28T06:12:03.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.457+0000 7f8d6e200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f8d680cffa0 0x7f8d680d1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:03.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.457+0000 7f8d6e200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f8d680cffa0 0x7f8d680d1250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:34786/0 (socket says 172.21.15.96:34786) 2024-05-28T06:12:03.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.457+0000 7f8d6e200640 1 -- 172.21.15.96:0/1220896114 learned_addr learned my addr 172.21.15.96:0/1220896114 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:03.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.457+0000 7f8d6ce00640 1 --2- 172.21.15.96:0/1220896114 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8d680cfb40 0x7f8d680d0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:03.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.457+0000 7f8d6d800640 1 --2- 172.21.15.96:0/1220896114 >> v2:172.21.15.123:3300/0 conn(0x7f8d6815e5a0 0x7f8d680d2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:03.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.457+0000 7f8d6e200640 1 -- 172.21.15.96:0/1220896114 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8d680cfb40 msgr2=0x7f8d680d0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:03.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.457+0000 7f8d6e200640 1 --2- 172.21.15.96:0/1220896114 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8d680cfb40 0x7f8d680d0d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:03.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.457+0000 7f8d6e200640 1 -- 172.21.15.96:0/1220896114 >> v2:172.21.15.123:3300/0 conn(0x7f8d6815e5a0 msgr2=0x7f8d680d2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:03.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.457+0000 7f8d6e200640 1 --2- 172.21.15.96:0/1220896114 >> v2:172.21.15.123:3300/0 conn(0x7f8d6815e5a0 0x7f8d680d2fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:03.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.457+0000 7f8d6e200640 1 -- 172.21.15.96:0/1220896114 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d680d24e0 con 0x7f8d680cffa0 2024-05-28T06:12:03.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.457+0000 7f8d6d800640 1 --2- 172.21.15.96:0/1220896114 >> v2:172.21.15.123:3300/0 conn(0x7f8d6815e5a0 0x7f8d680d2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:12:03.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.458+0000 7f8d6e200640 1 --2- 172.21.15.96:0/1220896114 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d680cffa0 0x7f8d680d1250 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f8d6808e450 tx=0x7f8d6002a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e26d86ffffc1c065 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:03.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.459+0000 7f8d67e00640 1 -- 172.21.15.96:0/1220896114 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d60036070 con 0x7f8d680cffa0 2024-05-28T06:12:03.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.459+0000 7f8d67e00640 1 -- 172.21.15.96:0/1220896114 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d60031600 con 0x7f8d680cffa0 2024-05-28T06:12:03.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.459+0000 7f8d67e00640 1 -- 172.21.15.96:0/1220896114 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d600319d0 con 0x7f8d680cffa0 2024-05-28T06:12:03.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.459+0000 7f8d6ec00640 1 -- 172.21.15.96:0/1220896114 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d680cffa0 msgr2=0x7f8d680d1250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:03.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.459+0000 7f8d6ec00640 1 --2- 172.21.15.96:0/1220896114 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d680cffa0 0x7f8d680d1250 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f8d6808e450 tx=0x7f8d6002a6a0 comp rx=0 tx=0).stop 2024-05-28T06:12:03.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.459+0000 7f8d6ec00640 1 -- 172.21.15.96:0/1220896114 shutdown_connections 2024-05-28T06:12:03.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.459+0000 7f8d6ec00640 1 --2- 172.21.15.96:0/1220896114 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d680cffa0 0x7f8d680d1250 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:03.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.459+0000 7f8d6ec00640 1 --2- 172.21.15.96:0/1220896114 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8d680cfb40 0x7f8d680d0d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:03.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.459+0000 7f8d6ec00640 1 --2- 172.21.15.96:0/1220896114 >> v2:172.21.15.123:3300/0 conn(0x7f8d6815e5a0 0x7f8d680d2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:03.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.459+0000 7f8d6ec00640 1 -- 172.21.15.96:0/1220896114 >> 172.21.15.96:0/1220896114 conn(0x7f8d6808bfc0 msgr2=0x7f8d680d44b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:03.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.460+0000 7f8d6ec00640 1 -- 172.21.15.96:0/1220896114 shutdown_connections 2024-05-28T06:12:03.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.460+0000 7f8d6ec00640 1 -- 172.21.15.96:0/1220896114 wait complete. 2024-05-28T06:12:03.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.461+0000 7f8d6ec00640 1 Processor -- start 2024-05-28T06:12:03.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.461+0000 7f8d6ec00640 1 -- start start 2024-05-28T06:12:03.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.462+0000 7f8d6ec00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8d680cfb40 0x7f8d680cbf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:03.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.462+0000 7f8d6ec00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d680cffa0 0x7f8d680c9f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:03.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.462+0000 7f8d6ec00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8d6815e5a0 0x7f8d680cd880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:03.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.462+0000 7f8d6ec00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f8d6815eb00 con 0x7f8d680cffa0 2024-05-28T06:12:03.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.462+0000 7f8d6ec00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f8d6815e980 con 0x7f8d6815e5a0 2024-05-28T06:12:03.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.462+0000 7f8d6ec00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f8d680d5220 con 0x7f8d680cfb40 2024-05-28T06:12:03.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.462+0000 7f8d6ce00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d680cffa0 0x7f8d680c9f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:03.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.463+0000 7f8d6ce00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d680cffa0 0x7f8d680c9f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:34792/0 (socket says 172.21.15.96:34792) 2024-05-28T06:12:03.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.463+0000 7f8d6ce00640 1 -- 172.21.15.96:0/1686174349 learned_addr learned my addr 172.21.15.96:0/1686174349 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:03.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.463+0000 7f8d6e200640 1 --2- 172.21.15.96:0/1686174349 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8d6815e5a0 0x7f8d680cd880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:03.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.463+0000 7f8d6d800640 1 --2- 172.21.15.96:0/1686174349 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8d680cfb40 0x7f8d680cbf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:03.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.463+0000 7f8d6ce00640 1 -- 172.21.15.96:0/1686174349 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8d680cfb40 msgr2=0x7f8d680cbf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:03.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.463+0000 7f8d6ce00640 1 --2- 172.21.15.96:0/1686174349 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8d680cfb40 0x7f8d680cbf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:03.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.463+0000 7f8d6ce00640 1 -- 172.21.15.96:0/1686174349 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8d6815e5a0 msgr2=0x7f8d680cd880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:03.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.463+0000 7f8d6ce00640 1 --2- 172.21.15.96:0/1686174349 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8d6815e5a0 0x7f8d680cd880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:03.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.463+0000 7f8d6ce00640 1 -- 172.21.15.96:0/1686174349 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d681f4cb0 con 0x7f8d680cffa0 2024-05-28T06:12:03.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.463+0000 7f8d6ce00640 1 --2- 172.21.15.96:0/1686174349 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d680cffa0 0x7f8d680c9f30 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f8d5400d5f0 tx=0x7f8d5400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:03.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.464+0000 7f8d65600640 1 -- 172.21.15.96:0/1686174349 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d540125f0 con 0x7f8d680cffa0 2024-05-28T06:12:03.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.464+0000 7f8d65600640 1 -- 172.21.15.96:0/1686174349 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d54019070 con 0x7f8d680cffa0 2024-05-28T06:12:03.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.464+0000 7f8d65600640 1 -- 172.21.15.96:0/1686174349 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d540129d0 con 0x7f8d680cffa0 2024-05-28T06:12:03.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.464+0000 7f8d6ec00640 1 -- 172.21.15.96:0/1686174349 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d681f4f90 con 0x7f8d680cffa0 2024-05-28T06:12:03.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.464+0000 7f8d6ec00640 1 -- 172.21.15.96:0/1686174349 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d681f5550 con 0x7f8d680cffa0 2024-05-28T06:12:03.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.465+0000 7f8d6ec00640 1 -- 172.21.15.96:0/1686174349 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d680ca530 con 0x7f8d680cffa0 2024-05-28T06:12:03.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.466+0000 7f8d65600640 1 -- 172.21.15.96:0/1686174349 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 15) ==== 91652+0+0 (secure 0 0 0) 0x7f8d54012b70 con 0x7f8d680cffa0 2024-05-28T06:12:03.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.467+0000 7f8d65600640 1 --2- 172.21.15.96:0/1686174349 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8d4806cfb0 0x7f8d4806f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:03.469 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.467+0000 7f8d65600640 1 -- 172.21.15.96:0/1686174349 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5795+0+0 (secure 0 0 0) 0x7f8d5408ca20 con 0x7f8d680cffa0 2024-05-28T06:12:03.469 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.468+0000 7f8d6d800640 1 --2- 172.21.15.96:0/1686174349 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8d4806cfb0 0x7f8d4806f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:03.470 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.469+0000 7f8d6d800640 1 --2- 172.21.15.96:0/1686174349 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8d4806cfb0 0x7f8d4806f470 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c00a910 tx=0x7f8d5c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:03.473 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.472+0000 7f8d65600640 1 -- 172.21.15.96:0/1686174349 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8d5405a5f0 con 0x7f8d680cffa0 2024-05-28T06:12:03.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.707+0000 7f8d6ec00640 1 -- 172.21.15.96:0/1686174349 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone26284", "target": ["mon-mgr", ""]}) -- 0x7f8d680d15a0 con 0x7f8d4806cfb0 2024-05-28T06:12:03.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.751+0000 7f8d65600640 1 -- 172.21.15.96:0/1686174349 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f8d680d15a0 con 0x7f8d4806cfb0 2024-05-28T06:12:03.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.754+0000 7f8d6ec00640 1 -- 172.21.15.96:0/1686174349 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8d4806cfb0 msgr2=0x7f8d4806f470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:03.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.754+0000 7f8d6ec00640 1 --2- 172.21.15.96:0/1686174349 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8d4806cfb0 0x7f8d4806f470 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c00a910 tx=0x7f8d5c008040 comp rx=0 tx=0).stop 2024-05-28T06:12:03.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.754+0000 7f8d6ec00640 1 -- 172.21.15.96:0/1686174349 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d680cffa0 msgr2=0x7f8d680c9f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:03.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.754+0000 7f8d6ec00640 1 --2- 172.21.15.96:0/1686174349 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d680cffa0 0x7f8d680c9f30 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f8d5400d5f0 tx=0x7f8d5400dab0 comp rx=0 tx=0).stop 2024-05-28T06:12:03.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.754+0000 7f8d6ec00640 1 -- 172.21.15.96:0/1686174349 shutdown_connections 2024-05-28T06:12:03.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.754+0000 7f8d6ec00640 1 --2- 172.21.15.96:0/1686174349 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8d4806cfb0 0x7f8d4806f470 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:03.757 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.754+0000 7f8d6ec00640 1 --2- 172.21.15.96:0/1686174349 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8d6815e5a0 0x7f8d680cd880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:03.757 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.754+0000 7f8d6ec00640 1 --2- 172.21.15.96:0/1686174349 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d680cffa0 0x7f8d680c9f30 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:03.757 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.754+0000 7f8d6ec00640 1 --2- 172.21.15.96:0/1686174349 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8d680cfb40 0x7f8d680cbf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:03.757 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.754+0000 7f8d6ec00640 1 -- 172.21.15.96:0/1686174349 >> 172.21.15.96:0/1686174349 conn(0x7f8d6808bfc0 msgr2=0x7f8d680b2ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:03.757 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.755+0000 7f8d6ec00640 1 -- 172.21.15.96:0/1686174349 shutdown_connections 2024-05-28T06:12:03.757 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.755+0000 7f8d6ec00640 1 -- 172.21.15.96:0/1686174349 wait complete. 2024-05-28T06:12:03.775 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone26282 2024-05-28T06:12:03.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.955+0000 7fc30c200640 1 Processor -- start 2024-05-28T06:12:03.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.956+0000 7fc30c200640 1 -- start start 2024-05-28T06:12:03.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.957+0000 7fc30c200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc3040b5c80 0x7fc3040d4e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:03.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.957+0000 7fc30c200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3040d5500 con 0x7fc3040d34e0 2024-05-28T06:12:03.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.957+0000 7fc30c200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3040b4f50 con 0x7fc3041622b0 2024-05-28T06:12:03.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.957+0000 7fc30c200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fc3040d2720 con 0x7fc3040b5c80 2024-05-28T06:12:03.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.957+0000 7fc308e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fc3040d34e0 0x7fc3040b5570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:59726/0 (socket says 172.21.15.96:59726) 2024-05-28T06:12:03.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.957+0000 7fc308e00640 1 -- 172.21.15.96:0/731713299 learned_addr learned my addr 172.21.15.96:0/731713299 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:12:03.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.958+0000 7fc30a200640 1 --2- 172.21.15.96:0/731713299 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc3040b5c80 0x7fc3040d4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:03.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.958+0000 7fc303e00640 1 -- 172.21.15.96:0/731713299 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 659101324 0 0) 0x7fc3040d5500 con 0x7fc3040d34e0 2024-05-28T06:12:03.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.958+0000 7fc303e00640 1 -- 172.21.15.96:0/731713299 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc2ec003520 con 0x7fc3040d34e0 2024-05-28T06:12:03.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.958+0000 7fc303e00640 1 -- 172.21.15.96:0/731713299 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3523256861 0 0) 0x7fc2ec003520 con 0x7fc3040d34e0 2024-05-28T06:12:03.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.958+0000 7fc303e00640 1 -- 172.21.15.96:0/731713299 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc3040b5c80 msgr2=0x7fc3040d4e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:03.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.958+0000 7fc303e00640 1 --2- 172.21.15.96:0/731713299 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc3040b5c80 0x7fc3040d4e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:03.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.958+0000 7fc303e00640 1 -- 172.21.15.96:0/731713299 >> v1:172.21.15.123:6789/0 conn(0x7fc3041622b0 legacy=0x7fc3040d1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:03.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.958+0000 7fc303e00640 1 -- 172.21.15.96:0/731713299 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3040d2a00 con 0x7fc3040d34e0 2024-05-28T06:12:03.962 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.958+0000 7fc303e00640 1 -- 172.21.15.96:0/731713299 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7fc2f0001670 con 0x7fc3040d34e0 2024-05-28T06:12:03.962 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.958+0000 7fc303e00640 1 -- 172.21.15.96:0/731713299 >> v1:172.21.15.96:6789/0 conn(0x7fc3040d34e0 legacy=0x7fc3040b5570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:03.962 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.958+0000 7fc303e00640 1 --2- 172.21.15.96:0/731713299 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc2ec0039e0 0x7fc2ec003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:03.962 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.958+0000 7fc303e00640 1 --2- 172.21.15.96:0/731713299 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc2ec004800 0x7fc2ec004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:03.962 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.958+0000 7fc303e00640 1 --2- 172.21.15.96:0/731713299 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc3040b5c80 0x7fc2ec005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:03.962 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.959+0000 7fc303e00640 1 -- 172.21.15.96:0/731713299 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc3040d2720 con 0x7fc2ec0039e0 2024-05-28T06:12:03.962 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.959+0000 7fc303e00640 1 -- 172.21.15.96:0/731713299 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc2ec00bb20 con 0x7fc2ec004800 2024-05-28T06:12:03.962 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.959+0000 7fc303e00640 1 -- 172.21.15.96:0/731713299 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fc2ec00bd20 con 0x7fc3040b5c80 2024-05-28T06:12:03.962 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.959+0000 7fc309800640 1 --2- 172.21.15.96:0/731713299 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc2ec0039e0 0x7fc2ec003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:03.962 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.959+0000 7fc308e00640 1 --2- 172.21.15.96:0/731713299 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc2ec004800 0x7fc2ec004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:03.964 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.959+0000 7fc30a200640 1 --2- 172.21.15.96:0/731713299 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc3040b5c80 0x7fc2ec005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:03.964 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.959+0000 7fc309800640 1 -- 172.21.15.96:0/731713299 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc3040b5c80 msgr2=0x7fc2ec005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:03.964 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.959+0000 7fc309800640 1 --2- 172.21.15.96:0/731713299 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc3040b5c80 0x7fc2ec005200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:03.964 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.959+0000 7fc309800640 1 -- 172.21.15.96:0/731713299 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc2ec004800 msgr2=0x7fc2ec004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:03.964 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.959+0000 7fc309800640 1 --2- 172.21.15.96:0/731713299 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc2ec004800 0x7fc2ec004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:03.964 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.959+0000 7fc309800640 1 -- 172.21.15.96:0/731713299 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2ec003520 con 0x7fc2ec0039e0 2024-05-28T06:12:03.964 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.960+0000 7fc309800640 1 --2- 172.21.15.96:0/731713299 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc2ec0039e0 0x7fc2ec003e00 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fc2f8001750 tx=0x7fc2ec005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39d2ca03944eee1f server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:03.964 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.961+0000 7fc303e00640 1 -- 172.21.15.96:0/731713299 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f8001370 con 0x7fc2ec0039e0 2024-05-28T06:12:03.964 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.961+0000 7fc303e00640 1 -- 172.21.15.96:0/731713299 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2f80052d0 con 0x7fc2ec0039e0 2024-05-28T06:12:03.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.961+0000 7fc303e00640 1 -- 172.21.15.96:0/731713299 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f802abe0 con 0x7fc2ec0039e0 2024-05-28T06:12:03.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.961+0000 7fc30c200640 1 -- 172.21.15.96:0/731713299 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc2ec0039e0 msgr2=0x7fc2ec003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:03.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.961+0000 7fc30c200640 1 --2- 172.21.15.96:0/731713299 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc2ec0039e0 0x7fc2ec003e00 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fc2f8001750 tx=0x7fc2ec005830 comp rx=0 tx=0).stop 2024-05-28T06:12:03.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.961+0000 7fc30a200640 1 -- 172.21.15.96:0/731713299 reap_dead start 2024-05-28T06:12:03.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.961+0000 7fc30c200640 1 -- 172.21.15.96:0/731713299 shutdown_connections 2024-05-28T06:12:03.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.961+0000 7fc30c200640 1 -- 172.21.15.96:0/731713299 >> 172.21.15.96:0/731713299 conn(0x7fc30408bfc0 msgr2=0x7fc3040d46f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:03.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.961+0000 7fc30c200640 1 -- 172.21.15.96:0/731713299 shutdown_connections 2024-05-28T06:12:03.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.961+0000 7fc30c200640 1 -- 172.21.15.96:0/731713299 wait complete. 2024-05-28T06:12:03.966 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.964+0000 7fc30c200640 1 Processor -- start 2024-05-28T06:12:03.966 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.964+0000 7fc30c200640 1 -- start start 2024-05-28T06:12:03.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.966+0000 7fc30c200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc2ec0039e0 0x7fc3040b8100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:03.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.966+0000 7fc30c200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3040b8640 0x7fc3040c8220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:03.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.966+0000 7fc30c200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc30415ae80 0x7fc3040c8760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:03.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.966+0000 7fc30c200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc3040d4f80 con 0x7fc30415ae80 2024-05-28T06:12:03.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.966+0000 7fc30c200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc3040d4e00 con 0x7fc3040b8640 2024-05-28T06:12:03.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.966+0000 7fc30c200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fc3040d5100 con 0x7fc2ec0039e0 2024-05-28T06:12:03.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.966+0000 7fc30a200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc30415ae80 0x7fc3040c8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:03.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.966+0000 7fc30a200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc30415ae80 0x7fc3040c8760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:34816/0 (socket says 172.21.15.96:34816) 2024-05-28T06:12:03.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.966+0000 7fc30a200640 1 -- 172.21.15.96:0/1050618727 learned_addr learned my addr 172.21.15.96:0/1050618727 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:03.970 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.966+0000 7fc309800640 1 --2- 172.21.15.96:0/1050618727 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc2ec0039e0 0x7fc3040b8100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:03.970 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.966+0000 7fc308e00640 1 --2- 172.21.15.96:0/1050618727 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3040b8640 0x7fc3040c8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:03.970 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.966+0000 7fc30a200640 1 -- 172.21.15.96:0/1050618727 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc2ec0039e0 msgr2=0x7fc3040b8100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:03.970 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.966+0000 7fc30a200640 1 --2- 172.21.15.96:0/1050618727 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc2ec0039e0 0x7fc3040b8100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:03.970 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.966+0000 7fc30a200640 1 -- 172.21.15.96:0/1050618727 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3040b8640 msgr2=0x7fc3040c8220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:03.970 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.966+0000 7fc30a200640 1 --2- 172.21.15.96:0/1050618727 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3040b8640 0x7fc3040c8220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:03.970 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.966+0000 7fc30a200640 1 -- 172.21.15.96:0/1050618727 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3040b9b80 con 0x7fc30415ae80 2024-05-28T06:12:03.970 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.967+0000 7fc30a200640 1 --2- 172.21.15.96:0/1050618727 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc30415ae80 0x7fc3040c8760 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fc3040d0d40 tx=0x7fc2fc00f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:03.970 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.967+0000 7fc301600640 1 -- 172.21.15.96:0/1050618727 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2fc01c070 con 0x7fc30415ae80 2024-05-28T06:12:03.970 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.967+0000 7fc301600640 1 -- 172.21.15.96:0/1050618727 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2fc017400 con 0x7fc30415ae80 2024-05-28T06:12:03.970 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.967+0000 7fc30c200640 1 -- 172.21.15.96:0/1050618727 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3040b8ae0 con 0x7fc30415ae80 2024-05-28T06:12:03.970 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.967+0000 7fc301600640 1 -- 172.21.15.96:0/1050618727 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2fc0177f0 con 0x7fc30415ae80 2024-05-28T06:12:03.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.967+0000 7fc30c200640 1 -- 172.21.15.96:0/1050618727 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3040b90a0 con 0x7fc30415ae80 2024-05-28T06:12:03.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.968+0000 7fc30c200640 1 -- 172.21.15.96:0/1050618727 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3040689e0 con 0x7fc30415ae80 2024-05-28T06:12:03.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.969+0000 7fc301600640 1 -- 172.21.15.96:0/1050618727 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 15) ==== 91652+0+0 (secure 0 0 0) 0x7fc2fc005d10 con 0x7fc30415ae80 2024-05-28T06:12:03.972 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.970+0000 7fc301600640 1 --2- 172.21.15.96:0/1050618727 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc2e406cfb0 0x7fc2e406f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:03.972 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.970+0000 7fc301600640 1 -- 172.21.15.96:0/1050618727 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5795+0+0 (secure 0 0 0) 0x7fc2fc08cd00 con 0x7fc30415ae80 2024-05-28T06:12:03.972 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.971+0000 7fc309800640 1 --2- 172.21.15.96:0/1050618727 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc2e406cfb0 0x7fc2e406f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:03.973 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.972+0000 7fc309800640 1 --2- 172.21.15.96:0/1050618727 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc2e406cfb0 0x7fc2e406f470 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fc2f8032000 tx=0x7fc2f802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:03.976 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:03.975+0000 7fc301600640 1 -- 172.21.15.96:0/1050618727 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc2fc012070 con 0x7fc30415ae80 2024-05-28T06:12:04.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.201+0000 7fc30c200640 1 -- 172.21.15.96:0/1050618727 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone26282", "target": ["mon-mgr", ""]}) -- 0x7fc3040c79f0 con 0x7fc2e406cfb0 2024-05-28T06:12:04.252 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.250+0000 7fc301600640 1 -- 172.21.15.96:0/1050618727 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fc3040c79f0 con 0x7fc2e406cfb0 2024-05-28T06:12:04.255 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.253+0000 7fc30c200640 1 -- 172.21.15.96:0/1050618727 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc2e406cfb0 msgr2=0x7fc2e406f470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:04.255 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.253+0000 7fc30c200640 1 --2- 172.21.15.96:0/1050618727 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc2e406cfb0 0x7fc2e406f470 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fc2f8032000 tx=0x7fc2f802e000 comp rx=0 tx=0).stop 2024-05-28T06:12:04.255 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.253+0000 7fc30c200640 1 -- 172.21.15.96:0/1050618727 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc30415ae80 msgr2=0x7fc3040c8760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:04.255 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.253+0000 7fc30c200640 1 --2- 172.21.15.96:0/1050618727 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc30415ae80 0x7fc3040c8760 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fc3040d0d40 tx=0x7fc2fc00f4a0 comp rx=0 tx=0).stop 2024-05-28T06:12:04.255 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.253+0000 7fc30c200640 1 -- 172.21.15.96:0/1050618727 shutdown_connections 2024-05-28T06:12:04.255 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.253+0000 7fc30c200640 1 --2- 172.21.15.96:0/1050618727 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc2e406cfb0 0x7fc2e406f470 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:04.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.253+0000 7fc30c200640 1 --2- 172.21.15.96:0/1050618727 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc30415ae80 0x7fc3040c8760 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:04.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.253+0000 7fc30c200640 1 --2- 172.21.15.96:0/1050618727 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3040b8640 0x7fc3040c8220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:04.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.253+0000 7fc30c200640 1 --2- 172.21.15.96:0/1050618727 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc2ec0039e0 0x7fc3040b8100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:04.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.253+0000 7fc30c200640 1 -- 172.21.15.96:0/1050618727 >> 172.21.15.96:0/1050618727 conn(0x7fc30408bfc0 msgr2=0x7fc3040b3b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:04.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.254+0000 7fc30c200640 1 -- 172.21.15.96:0/1050618727 shutdown_connections 2024-05-28T06:12:04.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.254+0000 7fc30c200640 1 -- 172.21.15.96:0/1050618727 wait complete. 2024-05-28T06:12:04.274 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone26283 2024-05-28T06:12:04.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.553+0000 7f88fea00640 1 Processor -- start 2024-05-28T06:12:04.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.554+0000 7f88fea00640 1 -- start start 2024-05-28T06:12:04.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.555+0000 7f88fea00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f88f815e5a0 0x7f88f80d2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:04.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.555+0000 7f88fea00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f88f80cfb40 0x7f88f80d0d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:04.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.555+0000 7f88fea00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88f80d3500 con 0x7f88f80cffa0 2024-05-28T06:12:04.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.555+0000 7f88fea00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f88f80d2b30 con 0x7f88f80cfb40 2024-05-28T06:12:04.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.555+0000 7f88fea00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f88f80d2200 con 0x7f88f815e5a0 2024-05-28T06:12:04.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.555+0000 7f88fcc00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f88f80cfb40 0x7f88f80d0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:04.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.555+0000 7f88fcc00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f88f80cfb40 0x7f88f80d0d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:34820/0 (socket says 172.21.15.96:34820) 2024-05-28T06:12:04.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.555+0000 7f88fcc00640 1 -- 172.21.15.96:0/2849726990 learned_addr learned my addr 172.21.15.96:0/2849726990 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:04.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.555+0000 7f88fe000640 1 --1- 172.21.15.96:0/2849726990 >> v1:172.21.15.96:6789/0 conn(0x7f88f80cffa0 0x7f88f806c740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:59738/0 (socket says 172.21.15.96:59738) 2024-05-28T06:12:04.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.556+0000 7f88fd600640 1 --2- 172.21.15.96:0/2849726990 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f88f815e5a0 0x7f88f80d2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:04.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.556+0000 7f88fcc00640 1 -- 172.21.15.96:0/2849726990 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f88f815e5a0 msgr2=0x7f88f80d2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:04.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.556+0000 7f88fcc00640 1 --2- 172.21.15.96:0/2849726990 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f88f815e5a0 0x7f88f80d2fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:04.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.556+0000 7f88fcc00640 1 -- 172.21.15.96:0/2849726990 >> v1:172.21.15.96:6789/0 conn(0x7f88f80cffa0 legacy=0x7f88f806c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:04.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.556+0000 7f88fcc00640 1 -- 172.21.15.96:0/2849726990 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88f80d24e0 con 0x7f88f80cfb40 2024-05-28T06:12:04.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.556+0000 7f88fd600640 1 --2- 172.21.15.96:0/2849726990 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f88f815e5a0 0x7f88f80d2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:12:04.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.557+0000 7f88fcc00640 1 --2- 172.21.15.96:0/2849726990 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f88f80cfb40 0x7f88f80d0d10 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f88e4009870 tx=0x7f88e402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3d14b49b409b918 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:04.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.557+0000 7f88f7e00640 1 -- 172.21.15.96:0/2849726990 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88e4037070 con 0x7f88f80cfb40 2024-05-28T06:12:04.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.557+0000 7f88f7e00640 1 -- 172.21.15.96:0/2849726990 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88e4032440 con 0x7f88f80cfb40 2024-05-28T06:12:04.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.557+0000 7f88f7e00640 1 -- 172.21.15.96:0/2849726990 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88e4032810 con 0x7f88f80cfb40 2024-05-28T06:12:04.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.558+0000 7f88fea00640 1 -- 172.21.15.96:0/2849726990 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f88f80cfb40 msgr2=0x7f88f80d0d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:04.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.558+0000 7f88fea00640 1 --2- 172.21.15.96:0/2849726990 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f88f80cfb40 0x7f88f80d0d10 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f88e4009870 tx=0x7f88e402a270 comp rx=0 tx=0).stop 2024-05-28T06:12:04.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.558+0000 7f88fea00640 1 -- 172.21.15.96:0/2849726990 shutdown_connections 2024-05-28T06:12:04.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.558+0000 7f88fea00640 1 --2- 172.21.15.96:0/2849726990 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f88f80cfb40 0x7f88f80d0d10 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:04.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.558+0000 7f88fea00640 1 --2- 172.21.15.96:0/2849726990 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f88f815e5a0 0x7f88f80d2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:04.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.558+0000 7f88fea00640 1 -- 172.21.15.96:0/2849726990 >> 172.21.15.96:0/2849726990 conn(0x7f88f808bfc0 msgr2=0x7f88f80d44b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:04.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.558+0000 7f88fea00640 1 -- 172.21.15.96:0/2849726990 shutdown_connections 2024-05-28T06:12:04.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.558+0000 7f88fea00640 1 -- 172.21.15.96:0/2849726990 wait complete. 2024-05-28T06:12:04.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.560+0000 7f88fea00640 1 Processor -- start 2024-05-28T06:12:04.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.560+0000 7f88fea00640 1 -- start start 2024-05-28T06:12:04.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.560+0000 7f88fea00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f88f80cfb40 0x7f88f814e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:04.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.561+0000 7f88fea00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f88f80cffa0 0x7f88f814bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:04.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.561+0000 7f88fea00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f88f815e5a0 0x7f88f814b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:04.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.561+0000 7f88fea00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f88f815eb00 con 0x7f88f80cffa0 2024-05-28T06:12:04.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.561+0000 7f88fea00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f88f815e980 con 0x7f88f815e5a0 2024-05-28T06:12:04.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.561+0000 7f88fea00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f88f80d5220 con 0x7f88f80cfb40 2024-05-28T06:12:04.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.561+0000 7f88fcc00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f88f80cffa0 0x7f88f814bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:04.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.561+0000 7f88fcc00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f88f80cffa0 0x7f88f814bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:34824/0 (socket says 172.21.15.96:34824) 2024-05-28T06:12:04.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.561+0000 7f88fcc00640 1 -- 172.21.15.96:0/290283288 learned_addr learned my addr 172.21.15.96:0/290283288 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:04.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.561+0000 7f88fd600640 1 --2- 172.21.15.96:0/290283288 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f88f80cfb40 0x7f88f814e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:04.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.561+0000 7f88fcc00640 1 -- 172.21.15.96:0/290283288 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f88f80cfb40 msgr2=0x7f88f814e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:04.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.561+0000 7f88fcc00640 1 --2- 172.21.15.96:0/290283288 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f88f80cfb40 0x7f88f814e980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:04.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.561+0000 7f88fcc00640 1 -- 172.21.15.96:0/290283288 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f88f815e5a0 msgr2=0x7f88f814b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:04.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.561+0000 7f88fcc00640 1 --2- 172.21.15.96:0/290283288 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f88f815e5a0 0x7f88f814b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:04.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.561+0000 7f88fcc00640 1 -- 172.21.15.96:0/290283288 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88f814eec0 con 0x7f88f80cffa0 2024-05-28T06:12:04.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.562+0000 7f88fcc00640 1 --2- 172.21.15.96:0/290283288 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f88f80cffa0 0x7f88f814bee0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f88e4000c00 tx=0x7f88e40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:04.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.562+0000 7f88f5600640 1 -- 172.21.15.96:0/290283288 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88e4037040 con 0x7f88f80cffa0 2024-05-28T06:12:04.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.562+0000 7f88f5600640 1 -- 172.21.15.96:0/290283288 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88e4042070 con 0x7f88f80cffa0 2024-05-28T06:12:04.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.562+0000 7f88f5600640 1 -- 172.21.15.96:0/290283288 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88e4030bd0 con 0x7f88f80cffa0 2024-05-28T06:12:04.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.562+0000 7f88fea00640 1 -- 172.21.15.96:0/290283288 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88f80cdd60 con 0x7f88f80cffa0 2024-05-28T06:12:04.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.562+0000 7f88fea00640 1 -- 172.21.15.96:0/290283288 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88f80ce2e0 con 0x7f88f80cffa0 2024-05-28T06:12:04.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.563+0000 7f88fea00640 1 -- 172.21.15.96:0/290283288 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88f80689e0 con 0x7f88f80cffa0 2024-05-28T06:12:04.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.564+0000 7f88f5600640 1 -- 172.21.15.96:0/290283288 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 15) ==== 91652+0+0 (secure 0 0 0) 0x7f88e4047460 con 0x7f88f80cffa0 2024-05-28T06:12:04.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.565+0000 7f88f5600640 1 --2- 172.21.15.96:0/290283288 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f88d806cfb0 0x7f88d806f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:04.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.565+0000 7f88f5600640 1 -- 172.21.15.96:0/290283288 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5795+0+0 (secure 0 0 0) 0x7f88e40ad2f0 con 0x7f88f80cffa0 2024-05-28T06:12:04.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.566+0000 7f88fd600640 1 --2- 172.21.15.96:0/290283288 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f88d806cfb0 0x7f88d806f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:04.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.567+0000 7f88fd600640 1 --2- 172.21.15.96:0/290283288 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f88d806cfb0 0x7f88d806f470 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f88ec00b840 tx=0x7f88ec008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:04.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.574+0000 7f88f5600640 1 -- 172.21.15.96:0/290283288 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f88f814eec0 con 0x7f88f80cffa0 2024-05-28T06:12:04.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.612+0000 7f88f5600640 1 -- 172.21.15.96:0/290283288 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mgrmap(e 16) ==== 91754+0+0 (secure 0 0 0) 0x7f88e4076d60 con 0x7f88f80cffa0 2024-05-28T06:12:04.813 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.811+0000 7f88fea00640 1 -- 172.21.15.96:0/290283288 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone26283", "target": ["mon-mgr", ""]}) -- 0x7f88f80ba2c0 con 0x7f88d806cfb0 2024-05-28T06:12:04.858 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.857+0000 7f88f5600640 1 -- 172.21.15.96:0/290283288 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f88f80ba2c0 con 0x7f88d806cfb0 2024-05-28T06:12:04.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.859+0000 7f88fea00640 1 -- 172.21.15.96:0/290283288 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f88d806cfb0 msgr2=0x7f88d806f470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:04.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.859+0000 7f88fea00640 1 --2- 172.21.15.96:0/290283288 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f88d806cfb0 0x7f88d806f470 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f88ec00b840 tx=0x7f88ec008040 comp rx=0 tx=0).stop 2024-05-28T06:12:04.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.859+0000 7f88fea00640 1 -- 172.21.15.96:0/290283288 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f88f80cffa0 msgr2=0x7f88f814bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:04.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.859+0000 7f88fea00640 1 --2- 172.21.15.96:0/290283288 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f88f80cffa0 0x7f88f814bee0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f88e4000c00 tx=0x7f88e40046c0 comp rx=0 tx=0).stop 2024-05-28T06:12:04.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.860+0000 7f88fea00640 1 -- 172.21.15.96:0/290283288 shutdown_connections 2024-05-28T06:12:04.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.860+0000 7f88fea00640 1 --2- 172.21.15.96:0/290283288 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f88d806cfb0 0x7f88d806f470 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:04.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.860+0000 7f88fea00640 1 --2- 172.21.15.96:0/290283288 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f88f815e5a0 0x7f88f814b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:04.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.860+0000 7f88fea00640 1 --2- 172.21.15.96:0/290283288 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f88f80cffa0 0x7f88f814bee0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:04.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.860+0000 7f88fea00640 1 --2- 172.21.15.96:0/290283288 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f88f80cfb40 0x7f88f814e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:04.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.860+0000 7f88fea00640 1 -- 172.21.15.96:0/290283288 >> 172.21.15.96:0/290283288 conn(0x7f88f808bfc0 msgr2=0x7f88f80b2ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:04.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.860+0000 7f88fea00640 1 -- 172.21.15.96:0/290283288 shutdown_connections 2024-05-28T06:12:04.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:04.860+0000 7f88fea00640 1 -- 172.21.15.96:0/290283288 wait complete. 2024-05-28T06:12:04.880 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone26281 2024-05-28T06:12:05.061 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:05.059+0000 7f9783200640 1 Processor -- start 2024-05-28T06:12:05.061 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:05.059+0000 7f9783200640 1 -- start start 2024-05-28T06:12:05.062 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:05.061+0000 7f9783200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f978415e5a0 0x7f97840d2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:05.062 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:05.061+0000 7f9783200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f97840d1420 0x7f978406c740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:05.062 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:05.061+0000 7f9783200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97840d3500 con 0x7f97840cfb40 2024-05-28T06:12:05.062 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:05.061+0000 7f9783200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f97840d2b30 con 0x7f97840d1420 2024-05-28T06:12:05.062 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:05.061+0000 7f9783200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f97840d2200 con 0x7f978415e5a0 2024-05-28T06:12:05.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:05.061+0000 7f9781e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f978415e5a0 0x7f97840d2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:05.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:05.062+0000 7f9781e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f978415e5a0 0x7f97840d2fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.96:48830/0 (socket says 172.21.15.96:48830) 2024-05-28T06:12:05.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:05.062+0000 7f9781e00640 1 -- 172.21.15.96:0/209993188 learned_addr learned my addr 172.21.15.96:0/209993188 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:05.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:05.062+0000 7f9782800640 1 --2- 172.21.15.96:0/209993188 >> v2:172.21.15.123:3300/0 conn(0x7f97840d1420 0x7f978406c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:05.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:05.062+0000 7f9781e00640 1 -- 172.21.15.96:0/209993188 >> v2:172.21.15.123:3300/0 conn(0x7f97840d1420 msgr2=0x7f978406c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:05.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:05.062+0000 7f9781e00640 1 --2- 172.21.15.96:0/209993188 >> v2:172.21.15.123:3300/0 conn(0x7f97840d1420 0x7f978406c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:05.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:05.062+0000 7f9781e00640 1 -- 172.21.15.96:0/209993188 >> v1:172.21.15.123:6789/0 conn(0x7f97840cfb40 legacy=0x7f97840d0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:05.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:05.062+0000 7f9781e00640 1 -- 172.21.15.96:0/209993188 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97840d24e0 con 0x7f978415e5a0 2024-05-28T06:12:05.065 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:05.064+0000 7f9781e00640 1 --2- 172.21.15.96:0/209993188 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f978415e5a0 0x7f97840d2fc0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f9774009870 tx=0x7f977402a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5697d7f23b66e305 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:05.065 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:05.064+0000 7f9780a00640 1 -- 172.21.15.96:0/209993188 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9774037070 con 0x7f978415e5a0 2024-05-28T06:12:05.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:05.064+0000 7f9780a00640 1 -- 172.21.15.96:0/209993188 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9774032440 con 0x7f978415e5a0 2024-05-28T06:12:05.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:05.064+0000 7f9780a00640 1 -- 172.21.15.96:0/209993188 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9774032810 con 0x7f978415e5a0 2024-05-28T06:12:05.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:05.064+0000 7f9783200640 1 -- 172.21.15.96:0/209993188 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f978415e5a0 msgr2=0x7f97840d2fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:05.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:05.064+0000 7f9783200640 1 --2- 172.21.15.96:0/209993188 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f978415e5a0 0x7f97840d2fc0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f9774009870 tx=0x7f977402a270 comp rx=0 tx=0).stop 2024-05-28T06:12:05.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:05.065+0000 7f9783200640 1 -- 172.21.15.96:0/209993188 shutdown_connections 2024-05-28T06:12:05.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:05.065+0000 7f9783200640 1 --2- 172.21.15.96:0/209993188 >> v2:172.21.15.123:3300/0 conn(0x7f97840d1420 0x7f978406c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:05.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:05.065+0000 7f9783200640 1 --2- 172.21.15.96:0/209993188 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f978415e5a0 0x7f97840d2fc0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:05.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:05.065+0000 7f9783200640 1 -- 172.21.15.96:0/209993188 >> 172.21.15.96:0/209993188 conn(0x7f978408bfc0 msgr2=0x7f97840d44b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:05.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:05.065+0000 7f9783200640 1 -- 172.21.15.96:0/209993188 shutdown_connections 2024-05-28T06:12:05.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:05.065+0000 7f9783200640 1 -- 172.21.15.96:0/209993188 wait complete. 2024-05-28T06:12:05.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:05.066+0000 7f9783200640 1 Processor -- start 2024-05-28T06:12:05.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:05.066+0000 7f9783200640 1 -- start start 2024-05-28T06:12:05.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:05.067+0000 7f9783200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97840cfb40 0x7f97840b9900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:05.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:05.067+0000 7f9783200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f97840d1420 0x7f97841f86a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:05.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:05.067+0000 7f9783200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f978415e5a0 0x7f978414e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:05.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:05.067+0000 7f9783200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f978415eb00 con 0x7f97840cfb40 2024-05-28T06:12:05.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:05.067+0000 7f9783200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f978415e980 con 0x7f978415e5a0 2024-05-28T06:12:05.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:05.067+0000 7f9783200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f97840d5220 con 0x7f97840d1420 2024-05-28T06:12:05.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:05.068+0000 7f9781e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97840cfb40 0x7f97840b9900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:05.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:05.068+0000 7f9781e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97840cfb40 0x7f97840b9900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:34834/0 (socket says 172.21.15.96:34834) 2024-05-28T06:12:05.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:05.068+0000 7f9781e00640 1 -- 172.21.15.96:0/1930647232 learned_addr learned my addr 172.21.15.96:0/1930647232 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:05.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:05.068+0000 7f9781400640 1 --2- 172.21.15.96:0/1930647232 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f97840d1420 0x7f97841f86a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:05.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:05.068+0000 7f9781e00640 1 -- 172.21.15.96:0/1930647232 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f97840d1420 msgr2=0x7f97841f86a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:05.071 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:05.068+0000 7f9781e00640 1 --2- 172.21.15.96:0/1930647232 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f97840d1420 0x7f97841f86a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:05.071 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:05.068+0000 7f9781e00640 1 -- 172.21.15.96:0/1930647232 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f978415e5a0 msgr2=0x7f978414e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:05.071 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:05.068+0000 7f9781e00640 1 --2- 172.21.15.96:0/1930647232 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f978415e5a0 0x7f978414e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:05.071 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:05.068+0000 7f9781e00640 1 -- 172.21.15.96:0/1930647232 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97840cd6c0 con 0x7f97840cfb40 2024-05-28T06:12:05.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:05.069+0000 7f9781e00640 1 --2- 172.21.15.96:0/1930647232 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97840cfb40 0x7f97840b9900 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f977402a2b0 tx=0x7f97740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:05.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:05.069+0000 7f977a000640 1 -- 172.21.15.96:0/1930647232 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9774041030 con 0x7f97840cfb40 2024-05-28T06:12:05.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:05.069+0000 7f977a000640 1 -- 172.21.15.96:0/1930647232 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f977403d420 con 0x7f97840cfb40 2024-05-28T06:12:05.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:05.069+0000 7f9783200640 1 -- 172.21.15.96:0/1930647232 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97840cda00 con 0x7f97840cfb40 2024-05-28T06:12:05.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:05.069+0000 7f977a000640 1 -- 172.21.15.96:0/1930647232 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9774037040 con 0x7f97840cfb40 2024-05-28T06:12:05.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:05.069+0000 7f9783200640 1 -- 172.21.15.96:0/1930647232 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97840b4df0 con 0x7f97840cfb40 2024-05-28T06:12:05.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:05.071+0000 7f9783200640 1 -- 172.21.15.96:0/1930647232 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97840689e0 con 0x7f97840cfb40 2024-05-28T06:12:05.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:05.072+0000 7f977a000640 1 -- 172.21.15.96:0/1930647232 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 16) ==== 91754+0+0 (secure 0 0 0) 0x7f9774044070 con 0x7f97840cfb40 2024-05-28T06:12:05.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:05.072+0000 7f977a000640 1 --2- 172.21.15.96:0/1930647232 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f976406d0d0 0x7f976406f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:05.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:05.072+0000 7f977a000640 1 -- 172.21.15.96:0/1930647232 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5851+0+0 (secure 0 0 0) 0x7f97740ad4b0 con 0x7f97840cfb40 2024-05-28T06:12:05.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:05.073+0000 7f9781400640 1 --2- 172.21.15.96:0/1930647232 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f976406d0d0 0x7f976406f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:05.075 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:05.074+0000 7f9781400640 1 --2- 172.21.15.96:0/1930647232 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f976406d0d0 0x7f976406f590 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f97840ce380 tx=0x7f9770009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:05.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:05.077+0000 7f977a000640 1 -- 172.21.15.96:0/1930647232 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f97840cda00 con 0x7f97840cfb40 2024-05-28T06:12:05.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:05.310+0000 7f9783200640 1 -- 172.21.15.96:0/1930647232 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone26281", "target": ["mon-mgr", ""]}) -- 0x7f97840cffa0 con 0x7f976406d0d0 2024-05-28T06:12:05.374 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:05.373+0000 7f977a000640 1 -- 172.21.15.96:0/1930647232 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f97840cffa0 con 0x7f976406d0d0 2024-05-28T06:12:05.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:05.377+0000 7f9783200640 1 -- 172.21.15.96:0/1930647232 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f976406d0d0 msgr2=0x7f976406f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:05.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:05.377+0000 7f9783200640 1 --2- 172.21.15.96:0/1930647232 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f976406d0d0 0x7f976406f590 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f97840ce380 tx=0x7f9770009040 comp rx=0 tx=0).stop 2024-05-28T06:12:05.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:05.377+0000 7f9783200640 1 -- 172.21.15.96:0/1930647232 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97840cfb40 msgr2=0x7f97840b9900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:05.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:05.377+0000 7f9783200640 1 --2- 172.21.15.96:0/1930647232 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97840cfb40 0x7f97840b9900 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f977402a2b0 tx=0x7f97740046c0 comp rx=0 tx=0).stop 2024-05-28T06:12:05.380 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:05.377+0000 7f9783200640 1 -- 172.21.15.96:0/1930647232 shutdown_connections 2024-05-28T06:12:05.380 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:05.378+0000 7f9783200640 1 --2- 172.21.15.96:0/1930647232 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f976406d0d0 0x7f976406f590 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:05.380 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:05.378+0000 7f9783200640 1 --2- 172.21.15.96:0/1930647232 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f978415e5a0 0x7f978414e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:05.380 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:05.378+0000 7f9783200640 1 --2- 172.21.15.96:0/1930647232 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f97840d1420 0x7f97841f86a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:05.380 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:05.378+0000 7f9783200640 1 --2- 172.21.15.96:0/1930647232 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97840cfb40 0x7f97840b9900 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:05.380 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:05.378+0000 7f9783200640 1 -- 172.21.15.96:0/1930647232 >> 172.21.15.96:0/1930647232 conn(0x7f978408bfc0 msgr2=0x7f97840d44b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:05.380 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:05.378+0000 7f9783200640 1 -- 172.21.15.96:0/1930647232 shutdown_connections 2024-05-28T06:12:05.380 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:05.378+0000 7f9783200640 1 -- 172.21.15.96:0/1930647232 wait complete. 2024-05-28T06:12:05.406 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2024-05-28T06:12:05.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:05 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-28T06:12:05.557 INFO:teuthology.orchestra.run.smithi096.stdout:2 2024-05-28T06:12:05.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:05 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-28T06:12:05.572 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-05-28T06:12:05.572 DEBUG:teuthology.orchestra.run.smithi096:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-05-28T06:12:05.689 DEBUG:teuthology.orchestra.run.smithi123:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-05-28T06:12:05.795 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-05-28T06:12:05.795 INFO:tasks.cephfs.mount:Terminating background process 2024-05-28T06:12:05.795 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:12:05.795 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:12:05.833 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:12:06.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.053+0000 7fb8cbe00640 1 Processor -- start 2024-05-28T06:12:06.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.053+0000 7fb8cbe00640 1 -- start start 2024-05-28T06:12:06.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.054+0000 7fb8cbe00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb8cc15b3a0 0x7fb8cc0b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:06.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.054+0000 7fb8cbe00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fb8cc06dc20 0x7fb8cc15f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:06.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.054+0000 7fb8cbe00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8cc0afd60 con 0x7fb8cc072c00 2024-05-28T06:12:06.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.054+0000 7fb8cbe00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fb8cc06f980 con 0x7fb8cc06dc20 2024-05-28T06:12:06.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.054+0000 7fb8cbe00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fb8cc0ae0c0 con 0x7fb8cc15b3a0 2024-05-28T06:12:06.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.054+0000 7fb8ca000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fb8cc06dc20 0x7fb8cc15f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:06.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.054+0000 7fb8ca000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fb8cc06dc20 0x7fb8cc15f9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:34850/0 (socket says 172.21.15.96:34850) 2024-05-28T06:12:06.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.054+0000 7fb8ca000640 1 -- 172.21.15.96:0/3185451454 learned_addr learned my addr 172.21.15.96:0/3185451454 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:06.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.054+0000 7fb8cb400640 1 --1- 172.21.15.96:0/3185451454 >> v1:172.21.15.96:6789/0 conn(0x7fb8cc072c00 0x7fb8cc0ae550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:59740/0 (socket says 172.21.15.96:59740) 2024-05-28T06:12:06.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.055+0000 7fb8ca000640 1 -- 172.21.15.96:0/3185451454 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb8cc15b3a0 msgr2=0x7fb8cc0b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:06.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.055+0000 7fb8ca000640 1 --2- 172.21.15.96:0/3185451454 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb8cc15b3a0 0x7fb8cc0b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:06.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.055+0000 7fb8ca000640 1 -- 172.21.15.96:0/3185451454 >> v1:172.21.15.96:6789/0 conn(0x7fb8cc072c00 legacy=0x7fb8cc0ae550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:06.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.055+0000 7fb8ca000640 1 -- 172.21.15.96:0/3185451454 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8cc0af7a0 con 0x7fb8cc06dc20 2024-05-28T06:12:06.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.056+0000 7fb8ca000640 1 --2- 172.21.15.96:0/3185451454 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb8cc06dc20 0x7fb8cc15f9a0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fb8b8009870 tx=0x7fb8b802a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbcb034b4b61b635 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:06.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.056+0000 7fb8c9600640 1 -- 172.21.15.96:0/3185451454 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8b8037070 con 0x7fb8cc06dc20 2024-05-28T06:12:06.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.056+0000 7fb8c9600640 1 -- 172.21.15.96:0/3185451454 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8b8032440 con 0x7fb8cc06dc20 2024-05-28T06:12:06.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.056+0000 7fb8c9600640 1 -- 172.21.15.96:0/3185451454 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8b8032810 con 0x7fb8cc06dc20 2024-05-28T06:12:06.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.057+0000 7fb8cbe00640 1 -- 172.21.15.96:0/3185451454 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb8cc06dc20 msgr2=0x7fb8cc15f9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:06.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.057+0000 7fb8cbe00640 1 --2- 172.21.15.96:0/3185451454 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb8cc06dc20 0x7fb8cc15f9a0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fb8b8009870 tx=0x7fb8b802a250 comp rx=0 tx=0).stop 2024-05-28T06:12:06.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.057+0000 7fb8cbe00640 1 -- 172.21.15.96:0/3185451454 shutdown_connections 2024-05-28T06:12:06.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.057+0000 7fb8cbe00640 1 --2- 172.21.15.96:0/3185451454 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb8cc06dc20 0x7fb8cc15f9a0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:06.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.057+0000 7fb8cbe00640 1 --2- 172.21.15.96:0/3185451454 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb8cc15b3a0 0x7fb8cc0b32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:06.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.057+0000 7fb8cbe00640 1 -- 172.21.15.96:0/3185451454 >> 172.21.15.96:0/3185451454 conn(0x7fb8cc0b1fc0 msgr2=0x7fb8cc0b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:06.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.057+0000 7fb8cbe00640 1 -- 172.21.15.96:0/3185451454 shutdown_connections 2024-05-28T06:12:06.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.057+0000 7fb8cbe00640 1 -- 172.21.15.96:0/3185451454 wait complete. 2024-05-28T06:12:06.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.058+0000 7fb8cbe00640 1 Processor -- start 2024-05-28T06:12:06.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.059+0000 7fb8cbe00640 1 -- start start 2024-05-28T06:12:06.061 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.060+0000 7fb8cbe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb8cc06dc20 0x7fb8cc149210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:06.061 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.060+0000 7fb8cbe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb8cc072c00 0x7fb8cc1574c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:06.061 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.060+0000 7fb8cbe00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb8cc15b3a0 0x7fb8cc157a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:06.061 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.060+0000 7fb8cbe00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb8cc15b780 con 0x7fb8cc072c00 2024-05-28T06:12:06.061 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.060+0000 7fb8cbe00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fb8cc15fee0 con 0x7fb8cc06dc20 2024-05-28T06:12:06.062 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.060+0000 7fb8cbe00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fb8cc073050 con 0x7fb8cc15b3a0 2024-05-28T06:12:06.062 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.060+0000 7fb8ca000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb8cc072c00 0x7fb8cc1574c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:06.062 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.060+0000 7fb8ca000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb8cc072c00 0x7fb8cc1574c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:34860/0 (socket says 172.21.15.96:34860) 2024-05-28T06:12:06.062 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.060+0000 7fb8ca000640 1 -- 172.21.15.96:0/363017506 learned_addr learned my addr 172.21.15.96:0/363017506 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:06.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.060+0000 7fb8ca000640 1 -- 172.21.15.96:0/363017506 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb8cc15b3a0 msgr2=0x7fb8cc157a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:06.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.060+0000 7fb8cb400640 1 --2- 172.21.15.96:0/363017506 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb8cc15b3a0 0x7fb8cc157a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:06.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.060+0000 7fb8caa00640 1 --2- 172.21.15.96:0/363017506 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb8cc06dc20 0x7fb8cc149210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:06.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.060+0000 7fb8ca000640 1 --2- 172.21.15.96:0/363017506 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb8cc15b3a0 0x7fb8cc157a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:06.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.060+0000 7fb8ca000640 1 -- 172.21.15.96:0/363017506 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb8cc06dc20 msgr2=0x7fb8cc149210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:06.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.060+0000 7fb8ca000640 1 --2- 172.21.15.96:0/363017506 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb8cc06dc20 0x7fb8cc149210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:06.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.060+0000 7fb8ca000640 1 -- 172.21.15.96:0/363017506 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8cc15a3e0 con 0x7fb8cc072c00 2024-05-28T06:12:06.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.061+0000 7fb8ca000640 1 --2- 172.21.15.96:0/363017506 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb8cc072c00 0x7fb8cc1574c0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fb8b8009870 tx=0x7fb8b80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:06.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.061+0000 7fb8c2a00640 1 -- 172.21.15.96:0/363017506 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8b8041020 con 0x7fb8cc072c00 2024-05-28T06:12:06.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.061+0000 7fb8c2a00640 1 -- 172.21.15.96:0/363017506 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8b8030970 con 0x7fb8cc072c00 2024-05-28T06:12:06.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.061+0000 7fb8c2a00640 1 -- 172.21.15.96:0/363017506 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8b8037040 con 0x7fb8cc072c00 2024-05-28T06:12:06.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.061+0000 7fb8cbe00640 1 -- 172.21.15.96:0/363017506 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8cc15a6c0 con 0x7fb8cc072c00 2024-05-28T06:12:06.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.061+0000 7fb8cbe00640 1 -- 172.21.15.96:0/363017506 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8cc149750 con 0x7fb8cc072c00 2024-05-28T06:12:06.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.062+0000 7fb8cbe00640 1 -- 172.21.15.96:0/363017506 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8cc15d0e0 con 0x7fb8cc072c00 2024-05-28T06:12:06.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.063+0000 7fb8c2a00640 1 -- 172.21.15.96:0/363017506 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 16) ==== 91754+0+0 (secure 0 0 0) 0x7fb8b8044020 con 0x7fb8cc072c00 2024-05-28T06:12:06.065 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.064+0000 7fb8c2a00640 1 --2- 172.21.15.96:0/363017506 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb8ac06d0d0 0x7fb8ac06f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:06.065 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.064+0000 7fb8c2a00640 1 -- 172.21.15.96:0/363017506 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5851+0+0 (secure 0 0 0) 0x7fb8b80ad8c0 con 0x7fb8cc072c00 2024-05-28T06:12:06.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.065+0000 7fb8caa00640 1 --2- 172.21.15.96:0/363017506 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb8ac06d0d0 0x7fb8ac06f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:06.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.066+0000 7fb8caa00640 1 --2- 172.21.15.96:0/363017506 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb8ac06d0d0 0x7fb8ac06f590 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fb8bc00a770 tx=0x7fb8bc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:06.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.068+0000 7fb8c2a00640 1 -- 172.21.15.96:0/363017506 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb8b804b030 con 0x7fb8cc072c00 2024-05-28T06:12:06.267 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.265+0000 7fb8cbe00640 1 -- 172.21.15.96:0/363017506 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb8cc15abe0 con 0x7fb8cc072c00 2024-05-28T06:12:06.269 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.267+0000 7fb8c2a00640 1 -- 172.21.15.96:0/363017506 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v48) ==== 74+0+16868 (secure 0 0 0) 0x7fb8b807ac40 con 0x7fb8cc072c00 2024-05-28T06:12:06.269 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:12:06.269 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":48,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:12:04.592827+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-05-28T06:08:52.257477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-05-28T06:08:53.459376+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"48","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":48,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.9","mappings":[{"from":0,"to":5}]},{"pgid":"7.37","mappings":[{"from":0,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":7,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[{"pool":7,"snaps":[{"begin":2,"length":2}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:12:06.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.269+0000 7fb8cbe00640 1 -- 172.21.15.96:0/363017506 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb8ac06d0d0 msgr2=0x7fb8ac06f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:06.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.269+0000 7fb8cbe00640 1 --2- 172.21.15.96:0/363017506 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb8ac06d0d0 0x7fb8ac06f590 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fb8bc00a770 tx=0x7fb8bc008040 comp rx=0 tx=0).stop 2024-05-28T06:12:06.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.269+0000 7fb8cbe00640 1 -- 172.21.15.96:0/363017506 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb8cc072c00 msgr2=0x7fb8cc1574c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:06.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.269+0000 7fb8cbe00640 1 --2- 172.21.15.96:0/363017506 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb8cc072c00 0x7fb8cc1574c0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fb8b8009870 tx=0x7fb8b80046c0 comp rx=0 tx=0).stop 2024-05-28T06:12:06.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.270+0000 7fb8cbe00640 1 -- 172.21.15.96:0/363017506 shutdown_connections 2024-05-28T06:12:06.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.270+0000 7fb8cbe00640 1 --2- 172.21.15.96:0/363017506 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb8ac06d0d0 0x7fb8ac06f590 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:06.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.270+0000 7fb8cbe00640 1 --2- 172.21.15.96:0/363017506 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb8cc15b3a0 0x7fb8cc157a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:06.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.270+0000 7fb8cbe00640 1 --2- 172.21.15.96:0/363017506 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb8cc072c00 0x7fb8cc1574c0 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:06.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.270+0000 7fb8cbe00640 1 --2- 172.21.15.96:0/363017506 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb8cc06dc20 0x7fb8cc149210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:06.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.270+0000 7fb8cbe00640 1 -- 172.21.15.96:0/363017506 >> 172.21.15.96:0/363017506 conn(0x7fb8cc0b1fc0 msgr2=0x7fb8cc0ae550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:06.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.270+0000 7fb8cbe00640 1 -- 172.21.15.96:0/363017506 shutdown_connections 2024-05-28T06:12:06.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.270+0000 7fb8cbe00640 1 -- 172.21.15.96:0/363017506 wait complete. 2024-05-28T06:12:06.288 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-28T06:03:59.517102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-28T06:08:52.257477+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2024-05-28T06:08:53.459376+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '48', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 48, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-28T06:12:06.289 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-28T06:12:06.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.538+0000 7f1433200640 1 Processor -- start 2024-05-28T06:12:06.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.538+0000 7f1433200640 1 -- start start 2024-05-28T06:12:06.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.540+0000 7f1433200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f142c0cef30 0x7f142c0d1640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:06.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.540+0000 7f1433200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f142c0d1b80 0x7f142c0b90a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:06.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.540+0000 7f1433200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f142c0d1f90 con 0x7f142c15e850 2024-05-28T06:12:06.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.540+0000 7f1433200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f142c15ec30 con 0x7f142c0cef30 2024-05-28T06:12:06.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.540+0000 7f1433200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f142c0b8000 con 0x7f142c0d1b80 2024-05-28T06:12:06.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.540+0000 7f1431200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f142c0d1b80 0x7f142c0b90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:06.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.541+0000 7f1431200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f142c0d1b80 0x7f142c0b90a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.96:48848/0 (socket says 172.21.15.96:48848) 2024-05-28T06:12:06.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.541+0000 7f1431200640 1 -- 172.21.15.96:0/1005269242 learned_addr learned my addr 172.21.15.96:0/1005269242 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:06.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.541+0000 7f142b400640 1 --2- 172.21.15.96:0/1005269242 >> v2:172.21.15.123:3300/0 conn(0x7f142c0cef30 0x7f142c0d1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:06.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.542+0000 7f1431200640 1 -- 172.21.15.96:0/1005269242 >> v2:172.21.15.123:3300/0 conn(0x7f142c0cef30 msgr2=0x7f142c0d1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:06.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.542+0000 7f1431200640 1 --2- 172.21.15.96:0/1005269242 >> v2:172.21.15.123:3300/0 conn(0x7f142c0cef30 0x7f142c0d1640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:06.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.542+0000 7f1431200640 1 -- 172.21.15.96:0/1005269242 >> v1:172.21.15.96:6789/0 conn(0x7f142c15e850 legacy=0x7f142c0b77b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:06.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.542+0000 7f1431200640 1 -- 172.21.15.96:0/1005269242 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f142c0b82e0 con 0x7f142c0d1b80 2024-05-28T06:12:06.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.543+0000 7f1431200640 1 --2- 172.21.15.96:0/1005269242 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f142c0d1b80 0x7f142c0b90a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f142c0cb730 tx=0x7f142402a610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=134bc31bd8be0f8a server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:06.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.543+0000 7f142aa00640 1 -- 172.21.15.96:0/1005269242 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1424036070 con 0x7f142c0d1b80 2024-05-28T06:12:06.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.544+0000 7f142aa00640 1 -- 172.21.15.96:0/1005269242 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1424031400 con 0x7f142c0d1b80 2024-05-28T06:12:06.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.544+0000 7f142aa00640 1 -- 172.21.15.96:0/1005269242 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14240317d0 con 0x7f142c0d1b80 2024-05-28T06:12:06.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.544+0000 7f1433200640 1 -- 172.21.15.96:0/1005269242 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f142c0d1b80 msgr2=0x7f142c0b90a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:06.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.544+0000 7f1433200640 1 --2- 172.21.15.96:0/1005269242 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f142c0d1b80 0x7f142c0b90a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f142c0cb730 tx=0x7f142402a610 comp rx=0 tx=0).stop 2024-05-28T06:12:06.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.546+0000 7f1433200640 1 -- 172.21.15.96:0/1005269242 shutdown_connections 2024-05-28T06:12:06.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.546+0000 7f1433200640 1 --2- 172.21.15.96:0/1005269242 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f142c0d1b80 0x7f142c0b90a0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:06.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.546+0000 7f1433200640 1 --2- 172.21.15.96:0/1005269242 >> v2:172.21.15.123:3300/0 conn(0x7f142c0cef30 0x7f142c0d1640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:06.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.546+0000 7f1433200640 1 -- 172.21.15.96:0/1005269242 >> 172.21.15.96:0/1005269242 conn(0x7f142c067aa0 msgr2=0x7f142c067eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:06.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.547+0000 7f1433200640 1 -- 172.21.15.96:0/1005269242 shutdown_connections 2024-05-28T06:12:06.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.547+0000 7f1433200640 1 -- 172.21.15.96:0/1005269242 wait complete. 2024-05-28T06:12:06.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.550+0000 7f1433200640 1 Processor -- start 2024-05-28T06:12:06.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.550+0000 7f1433200640 1 -- start start 2024-05-28T06:12:06.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.551+0000 7f1433200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f142c0cef30 0x7f142c0cc9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:06.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.551+0000 7f1433200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f142c0d1b80 0x7f142c0ccf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:06.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.552+0000 7f1433200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f142c15e850 0x7f142c15edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:06.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.552+0000 7f1433200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f142c0b95e0 con 0x7f142c0cef30 2024-05-28T06:12:06.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.552+0000 7f1433200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f142c0b88a0 con 0x7f142c0d1b80 2024-05-28T06:12:06.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.552+0000 7f1433200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f142c0b9760 con 0x7f142c15e850 2024-05-28T06:12:06.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.552+0000 7f142be00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f142c0cef30 0x7f142c0cc9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:06.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.552+0000 7f142b400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f142c0d1b80 0x7f142c0ccf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:06.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.552+0000 7f142b400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f142c0d1b80 0x7f142c0ccf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.96:32896/0 (socket says 172.21.15.96:32896) 2024-05-28T06:12:06.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.552+0000 7f142b400640 1 -- 172.21.15.96:0/4258365989 learned_addr learned my addr 172.21.15.96:0/4258365989 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:06.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.552+0000 7f1431200640 1 --2- 172.21.15.96:0/4258365989 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f142c15e850 0x7f142c15edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:06.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.553+0000 7f142be00640 1 -- 172.21.15.96:0/4258365989 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f142c15e850 msgr2=0x7f142c15edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:06.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.553+0000 7f142be00640 1 --2- 172.21.15.96:0/4258365989 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f142c15e850 0x7f142c15edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:06.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.553+0000 7f142be00640 1 -- 172.21.15.96:0/4258365989 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f142c0d1b80 msgr2=0x7f142c0ccf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:06.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.553+0000 7f142be00640 1 --2- 172.21.15.96:0/4258365989 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f142c0d1b80 0x7f142c0ccf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:06.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.553+0000 7f142be00640 1 -- 172.21.15.96:0/4258365989 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f142c15f9f0 con 0x7f142c0cef30 2024-05-28T06:12:06.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.553+0000 7f142b400640 1 --2- 172.21.15.96:0/4258365989 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f142c0d1b80 0x7f142c0ccf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-28T06:12:06.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.553+0000 7f142be00640 1 --2- 172.21.15.96:0/4258365989 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f142c0cef30 0x7f142c0cc9c0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f141c00b3e0 tx=0x7f141c00b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:06.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.554+0000 7f1423e00640 1 -- 172.21.15.96:0/4258365989 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f141c015030 con 0x7f142c0cef30 2024-05-28T06:12:06.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.554+0000 7f1423e00640 1 -- 172.21.15.96:0/4258365989 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f141c004020 con 0x7f142c0cef30 2024-05-28T06:12:06.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.554+0000 7f1423e00640 1 -- 172.21.15.96:0/4258365989 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f141c004410 con 0x7f142c0cef30 2024-05-28T06:12:06.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.554+0000 7f1433200640 1 -- 172.21.15.96:0/4258365989 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f142c15fcd0 con 0x7f142c0cef30 2024-05-28T06:12:06.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.554+0000 7f1433200640 1 -- 172.21.15.96:0/4258365989 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f142c160290 con 0x7f142c0cef30 2024-05-28T06:12:06.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.558+0000 7f1423e00640 1 -- 172.21.15.96:0/4258365989 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 16) ==== 91754+0+0 (secure 0 0 0) 0x7f141c018070 con 0x7f142c0cef30 2024-05-28T06:12:06.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.559+0000 7f1433200640 1 -- 172.21.15.96:0/4258365989 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f142c0689e0 con 0x7f142c0cef30 2024-05-28T06:12:06.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.559+0000 7f1423e00640 1 --2- 172.21.15.96:0/4258365989 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f140c06d0d0 0x7f140c06f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:06.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.559+0000 7f1423e00640 1 -- 172.21.15.96:0/4258365989 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5851+0+0 (secure 0 0 0) 0x7f141c089ee0 con 0x7f142c0cef30 2024-05-28T06:12:06.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.562+0000 7f142b400640 1 --2- 172.21.15.96:0/4258365989 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f140c06d0d0 0x7f140c06f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:06.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.563+0000 7f142b400640 1 --2- 172.21.15.96:0/4258365989 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f140c06d0d0 0x7f140c06f590 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f142c1fbc70 tx=0x7f1418009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:06.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.568+0000 7f1423e00640 1 -- 172.21.15.96:0/4258365989 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f141c0595b0 con 0x7f142c0cef30 2024-05-28T06:12:06.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.925+0000 7f1433200640 1 -- 172.21.15.96:0/4258365989 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f142c0cd560 con 0x7f142c0cef30 2024-05-28T06:12:06.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.926+0000 7f1423e00640 1 -- 172.21.15.96:0/4258365989 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v48) ==== 93+0+10 (secure 0 0 0) 0x7f141c05d8b0 con 0x7f142c0cef30 2024-05-28T06:12:06.927 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 1 2024-05-28T06:12:06.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.929+0000 7f1433200640 1 -- 172.21.15.96:0/4258365989 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f140c06d0d0 msgr2=0x7f140c06f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:06.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.929+0000 7f1433200640 1 --2- 172.21.15.96:0/4258365989 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f140c06d0d0 0x7f140c06f590 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f142c1fbc70 tx=0x7f1418009190 comp rx=0 tx=0).stop 2024-05-28T06:12:06.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.929+0000 7f1433200640 1 -- 172.21.15.96:0/4258365989 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f142c0cef30 msgr2=0x7f142c0cc9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:06.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.929+0000 7f1433200640 1 --2- 172.21.15.96:0/4258365989 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f142c0cef30 0x7f142c0cc9c0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f141c00b3e0 tx=0x7f141c00b8a0 comp rx=0 tx=0).stop 2024-05-28T06:12:06.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.929+0000 7f1433200640 1 -- 172.21.15.96:0/4258365989 shutdown_connections 2024-05-28T06:12:06.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.929+0000 7f1433200640 1 --2- 172.21.15.96:0/4258365989 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f140c06d0d0 0x7f140c06f590 secure :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f142c1fbc70 tx=0x7f1418009190 comp rx=0 tx=0).stop 2024-05-28T06:12:06.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.929+0000 7f1433200640 1 --2- 172.21.15.96:0/4258365989 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f142c15e850 0x7f142c15edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:06.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.929+0000 7f1433200640 1 --2- 172.21.15.96:0/4258365989 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f142c0d1b80 0x7f142c0ccf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:06.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.929+0000 7f1433200640 1 --2- 172.21.15.96:0/4258365989 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f142c0cef30 0x7f142c0cc9c0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:06.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.929+0000 7f1433200640 1 -- 172.21.15.96:0/4258365989 >> 172.21.15.96:0/4258365989 conn(0x7f142c067aa0 msgr2=0x7f142c074990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:06.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.930+0000 7f1433200640 1 -- 172.21.15.96:0/4258365989 shutdown_connections 2024-05-28T06:12:06.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:06.930+0000 7f1433200640 1 -- 172.21.15.96:0/4258365989 wait complete. 2024-05-28T06:12:06.956 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-28T06:12:07.199 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.198+0000 7ff94fc00640 1 Processor -- start 2024-05-28T06:12:07.200 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.198+0000 7ff94fc00640 1 -- start start 2024-05-28T06:12:07.202 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.201+0000 7ff94fc00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7ff94815e850 0x7ff9480b77b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:07.202 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.201+0000 7ff94fc00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7ff9480cef30 0x7ff9480d1640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:07.202 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.201+0000 7ff94fc00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9480d1f60 con 0x7ff9480d1b80 2024-05-28T06:12:07.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.201+0000 7ff94fc00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7ff94815ec30 con 0x7ff9480cef30 2024-05-28T06:12:07.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.201+0000 7ff94fc00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7ff9480b8000 con 0x7ff94815e850 2024-05-28T06:12:07.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.201+0000 7ff94d200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7ff94815e850 0x7ff9480b77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:07.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.201+0000 7ff94d200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7ff94815e850 0x7ff9480b77b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.96:32902/0 (socket says 172.21.15.96:32902) 2024-05-28T06:12:07.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.202+0000 7ff94d200640 1 -- 172.21.15.96:0/1683488638 learned_addr learned my addr 172.21.15.96:0/1683488638 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:07.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.202+0000 7ff947e00640 1 --2- 172.21.15.96:0/1683488638 >> v2:172.21.15.96:3300/0 conn(0x7ff9480cef30 0x7ff9480d1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:07.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.202+0000 7ff94d200640 1 -- 172.21.15.96:0/1683488638 >> v2:172.21.15.96:3300/0 conn(0x7ff9480cef30 msgr2=0x7ff9480d1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:07.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.202+0000 7ff94d200640 1 --2- 172.21.15.96:0/1683488638 >> v2:172.21.15.96:3300/0 conn(0x7ff9480cef30 0x7ff9480d1640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:07.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.202+0000 7ff94d200640 1 -- 172.21.15.96:0/1683488638 >> v1:172.21.15.123:6789/0 conn(0x7ff9480d1b80 legacy=0x7ff948162340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:07.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.202+0000 7ff94d200640 1 -- 172.21.15.96:0/1683488638 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9480b82e0 con 0x7ff94815e850 2024-05-28T06:12:07.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.205+0000 7ff94d200640 1 --2- 172.21.15.96:0/1683488638 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff94815e850 0x7ff9480b77b0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7ff94806dba0 tx=0x7ff94002a6b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=146964d455103984 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:07.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.206+0000 7ff947400640 1 -- 172.21.15.96:0/1683488638 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff940036070 con 0x7ff94815e850 2024-05-28T06:12:07.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.206+0000 7ff947400640 1 -- 172.21.15.96:0/1683488638 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff940031600 con 0x7ff94815e850 2024-05-28T06:12:07.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.206+0000 7ff947400640 1 -- 172.21.15.96:0/1683488638 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9400319d0 con 0x7ff94815e850 2024-05-28T06:12:07.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.207+0000 7ff94fc00640 1 -- 172.21.15.96:0/1683488638 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff94815e850 msgr2=0x7ff9480b77b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:07.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.207+0000 7ff94fc00640 1 --2- 172.21.15.96:0/1683488638 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff94815e850 0x7ff9480b77b0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7ff94806dba0 tx=0x7ff94002a6b0 comp rx=0 tx=0).stop 2024-05-28T06:12:07.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.207+0000 7ff94fc00640 1 -- 172.21.15.96:0/1683488638 shutdown_connections 2024-05-28T06:12:07.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.207+0000 7ff94fc00640 1 --2- 172.21.15.96:0/1683488638 >> v2:172.21.15.96:3300/0 conn(0x7ff9480cef30 0x7ff9480d1640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:07.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.207+0000 7ff94fc00640 1 --2- 172.21.15.96:0/1683488638 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff94815e850 0x7ff9480b77b0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:07.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.207+0000 7ff94fc00640 1 -- 172.21.15.96:0/1683488638 >> 172.21.15.96:0/1683488638 conn(0x7ff948067aa0 msgr2=0x7ff948067eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:07.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.208+0000 7ff94fc00640 1 -- 172.21.15.96:0/1683488638 shutdown_connections 2024-05-28T06:12:07.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.208+0000 7ff94fc00640 1 -- 172.21.15.96:0/1683488638 wait complete. 2024-05-28T06:12:07.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.210+0000 7ff94fc00640 1 Processor -- start 2024-05-28T06:12:07.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.210+0000 7ff94fc00640 1 -- start start 2024-05-28T06:12:07.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.211+0000 7ff94fc00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff9480cef30 0x7ff9480d2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:07.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.211+0000 7ff94fc00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff9480d1b80 0x7ff94815b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:07.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.211+0000 7ff94fc00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff94815e850 0x7ff94815c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:07.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.211+0000 7ff94fc00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff9480b7cf0 con 0x7ff94815e850 2024-05-28T06:12:07.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.211+0000 7ff94fc00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff9480b88a0 con 0x7ff9480cef30 2024-05-28T06:12:07.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.211+0000 7ff94fc00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ff9480ba0b0 con 0x7ff9480d1b80 2024-05-28T06:12:07.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.212+0000 7ff94d200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff9480cef30 0x7ff9480d2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:07.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.212+0000 7ff94d200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff9480cef30 0x7ff9480d2ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.96:32916/0 (socket says 172.21.15.96:32916) 2024-05-28T06:12:07.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.212+0000 7ff94d200640 1 -- 172.21.15.96:0/2467873493 learned_addr learned my addr 172.21.15.96:0/2467873493 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:07.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.212+0000 7ff947e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff9480d1b80 0x7ff94815b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:07.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.212+0000 7ff94d200640 1 -- 172.21.15.96:0/2467873493 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff9480d1b80 msgr2=0x7ff94815b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:07.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.212+0000 7ff94d200640 1 --2- 172.21.15.96:0/2467873493 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff9480d1b80 0x7ff94815b5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:07.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.212+0000 7ff94d200640 1 -- 172.21.15.96:0/2467873493 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff94815e850 msgr2=0x7ff94815c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:07.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.212+0000 7ff94d200640 1 --2- 172.21.15.96:0/2467873493 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff94815e850 0x7ff94815c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:07.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.212+0000 7ff94d200640 1 -- 172.21.15.96:0/2467873493 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9480cc600 con 0x7ff9480cef30 2024-05-28T06:12:07.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.213+0000 7ff94d200640 1 --2- 172.21.15.96:0/2467873493 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff9480cef30 0x7ff9480d2ac0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ff940034040 tx=0x7ff940004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:07.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.214+0000 7ff944c00640 1 -- 172.21.15.96:0/2467873493 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff940036050 con 0x7ff9480cef30 2024-05-28T06:12:07.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.214+0000 7ff944c00640 1 -- 172.21.15.96:0/2467873493 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff940042070 con 0x7ff9480cef30 2024-05-28T06:12:07.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.214+0000 7ff944c00640 1 -- 172.21.15.96:0/2467873493 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff940009510 con 0x7ff9480cef30 2024-05-28T06:12:07.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.214+0000 7ff94fc00640 1 -- 172.21.15.96:0/2467873493 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9480cc8a0 con 0x7ff9480cef30 2024-05-28T06:12:07.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.214+0000 7ff94fc00640 1 -- 172.21.15.96:0/2467873493 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9480cce60 con 0x7ff9480cef30 2024-05-28T06:12:07.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.219+0000 7ff944c00640 1 -- 172.21.15.96:0/2467873493 <== mon.1 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 91754+0+0 (secure 0 0 0) 0x7ff940007c30 con 0x7ff9480cef30 2024-05-28T06:12:07.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.219+0000 7ff94fc00640 1 -- 172.21.15.96:0/2467873493 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9480cd060 con 0x7ff9480cef30 2024-05-28T06:12:07.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.220+0000 7ff944c00640 1 --2- 172.21.15.96:0/2467873493 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff92406d1a0 0x7ff92406f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:07.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.220+0000 7ff944c00640 1 -- 172.21.15.96:0/2467873493 <== mon.1 v2:172.21.15.123:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5851+0+0 (secure 0 0 0) 0x7ff9400adde0 con 0x7ff9480cef30 2024-05-28T06:12:07.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.221+0000 7ff947e00640 1 --2- 172.21.15.96:0/2467873493 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff92406d1a0 0x7ff92406f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:07.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.223+0000 7ff947e00640 1 --2- 172.21.15.96:0/2467873493 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff92406d1a0 0x7ff92406f660 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7ff94815c340 tx=0x7ff930017040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:07.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.228+0000 7ff944c00640 1 -- 172.21.15.96:0/2467873493 <== mon.1 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff9480cd060 con 0x7ff9480cef30 2024-05-28T06:12:07.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.585+0000 7ff94fc00640 1 -- 172.21.15.96:0/2467873493 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ff9480689e0 con 0x7ff9480cef30 2024-05-28T06:12:07.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.587+0000 7ff944c00640 1 -- 172.21.15.96:0/2467873493 <== mon.1 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v48) ==== 104+0+11 (secure 0 0 0) 0x7ff94007b160 con 0x7ff9480cef30 2024-05-28T06:12:07.589 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 32 2024-05-28T06:12:07.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.592+0000 7ff93e000640 1 -- 172.21.15.96:0/2467873493 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff92406d1a0 msgr2=0x7ff92406f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:07.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.592+0000 7ff93e000640 1 --2- 172.21.15.96:0/2467873493 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff92406d1a0 0x7ff92406f660 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7ff94815c340 tx=0x7ff930017040 comp rx=0 tx=0).stop 2024-05-28T06:12:07.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.592+0000 7ff93e000640 1 -- 172.21.15.96:0/2467873493 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff9480cef30 msgr2=0x7ff9480d2ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:07.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.592+0000 7ff93e000640 1 --2- 172.21.15.96:0/2467873493 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff9480cef30 0x7ff9480d2ac0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ff940034040 tx=0x7ff940004a40 comp rx=0 tx=0).stop 2024-05-28T06:12:07.594 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.593+0000 7ff93e000640 1 -- 172.21.15.96:0/2467873493 shutdown_connections 2024-05-28T06:12:07.594 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.593+0000 7ff93e000640 1 --2- 172.21.15.96:0/2467873493 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff92406d1a0 0x7ff92406f660 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:07.594 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.593+0000 7ff93e000640 1 --2- 172.21.15.96:0/2467873493 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff94815e850 0x7ff94815c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:07.595 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.593+0000 7ff93e000640 1 --2- 172.21.15.96:0/2467873493 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff9480d1b80 0x7ff94815b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:07.595 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.593+0000 7ff93e000640 1 --2- 172.21.15.96:0/2467873493 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff9480cef30 0x7ff9480d2ac0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:07.595 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.593+0000 7ff93e000640 1 -- 172.21.15.96:0/2467873493 >> 172.21.15.96:0/2467873493 conn(0x7ff948067aa0 msgr2=0x7ff9480c8710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:07.595 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.593+0000 7ff93e000640 1 -- 172.21.15.96:0/2467873493 shutdown_connections 2024-05-28T06:12:07.595 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.593+0000 7ff93e000640 1 -- 172.21.15.96:0/2467873493 wait complete. 2024-05-28T06:12:07.625 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-28T06:12:07.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.882+0000 7ff29b200640 1 Processor -- start 2024-05-28T06:12:07.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.882+0000 7ff29b200640 1 -- start start 2024-05-28T06:12:07.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.884+0000 7ff29b200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff29c15e870 0x7ff29c0d0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:07.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.884+0000 7ff29b200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7ff29c0d14d0 0x7ff29c0cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:07.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.884+0000 7ff29b200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff29c0cfd00 con 0x7ff29c0cf870 2024-05-28T06:12:07.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.884+0000 7ff29b200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7ff29c0d2150 con 0x7ff29c0d14d0 2024-05-28T06:12:07.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.884+0000 7ff29b200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ff29c15ec50 con 0x7ff29c15e870 2024-05-28T06:12:07.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.885+0000 7ff299400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7ff29c0d14d0 0x7ff29c0cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:07.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.885+0000 7ff29a800640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7ff29c0cf870 0x7ff29c162360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.96:52808/0 (socket says 172.21.15.96:52808) 2024-05-28T06:12:07.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.885+0000 7ff29a800640 1 -- 172.21.15.96:0/2655965065 learned_addr learned my addr 172.21.15.96:0/2655965065 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:12:07.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.886+0000 7ff299e00640 1 --2- 172.21.15.96:0/2655965065 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff29c15e870 0x7ff29c0d0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:07.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.886+0000 7ff299400640 1 -- 172.21.15.96:0/2655965065 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff29c15e870 msgr2=0x7ff29c0d0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:07.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.886+0000 7ff298a00640 1 -- 172.21.15.96:0/2655965065 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3337648426 0 0) 0x7ff294001490 con 0x7ff29c0cf870 2024-05-28T06:12:07.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.886+0000 7ff299400640 1 --2- 172.21.15.96:0/2655965065 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff29c15e870 0x7ff29c0d0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:07.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.886+0000 7ff299400640 1 -- 172.21.15.96:0/2655965065 >> v1:172.21.15.123:6789/0 conn(0x7ff29c0cf870 legacy=0x7ff29c162360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:07.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.886+0000 7ff299400640 1 -- 172.21.15.96:0/2655965065 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff29c0cff80 con 0x7ff29c0d14d0 2024-05-28T06:12:07.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.886+0000 7ff299e00640 1 --2- 172.21.15.96:0/2655965065 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff29c15e870 0x7ff29c0d0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:12:07.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.887+0000 7ff299400640 1 --2- 172.21.15.96:0/2655965065 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff29c0d14d0 0x7ff29c0cf330 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7ff288009790 tx=0x7ff28802a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=984ae5ead9501eb0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:07.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.888+0000 7ff298a00640 1 -- 172.21.15.96:0/2655965065 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff288037070 con 0x7ff29c0d14d0 2024-05-28T06:12:07.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.888+0000 7ff298a00640 1 -- 172.21.15.96:0/2655965065 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff288032440 con 0x7ff29c0d14d0 2024-05-28T06:12:07.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.889+0000 7ff298a00640 1 -- 172.21.15.96:0/2655965065 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff288037070 con 0x7ff29c0d14d0 2024-05-28T06:12:07.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.889+0000 7ff29b200640 1 -- 172.21.15.96:0/2655965065 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff29c0d14d0 msgr2=0x7ff29c0cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:07.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.889+0000 7ff29b200640 1 --2- 172.21.15.96:0/2655965065 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff29c0d14d0 0x7ff29c0cf330 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7ff288009790 tx=0x7ff28802a180 comp rx=0 tx=0).stop 2024-05-28T06:12:07.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.890+0000 7ff29b200640 1 -- 172.21.15.96:0/2655965065 shutdown_connections 2024-05-28T06:12:07.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.890+0000 7ff29b200640 1 --2- 172.21.15.96:0/2655965065 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff29c0d14d0 0x7ff29c0cf330 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:07.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.890+0000 7ff29b200640 1 --2- 172.21.15.96:0/2655965065 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff29c15e870 0x7ff29c0d0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:07.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.890+0000 7ff29b200640 1 -- 172.21.15.96:0/2655965065 >> 172.21.15.96:0/2655965065 conn(0x7ff29c08bfc0 msgr2=0x7ff29c0d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:07.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.891+0000 7ff29b200640 1 -- 172.21.15.96:0/2655965065 shutdown_connections 2024-05-28T06:12:07.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.892+0000 7ff29b200640 1 -- 172.21.15.96:0/2655965065 wait complete. 2024-05-28T06:12:07.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.895+0000 7ff29b200640 1 Processor -- start 2024-05-28T06:12:07.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.895+0000 7ff29b200640 1 -- start start 2024-05-28T06:12:07.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.897+0000 7ff29b200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff29c0cf870 0x7ff29c0cdf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:07.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.897+0000 7ff29b200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff29c0d14d0 0x7ff29c0cc710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:07.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.897+0000 7ff29b200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff29c15e870 0x7ff29c200760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:07.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.897+0000 7ff29b200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff29c0d0820 con 0x7ff29c15e870 2024-05-28T06:12:07.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.897+0000 7ff29b200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff29c0d0540 con 0x7ff29c0d14d0 2024-05-28T06:12:07.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.897+0000 7ff29b200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ff29c0d4fb0 con 0x7ff29c0cf870 2024-05-28T06:12:07.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.897+0000 7ff29a800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff29c15e870 0x7ff29c200760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:07.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.897+0000 7ff299e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff29c0cf870 0x7ff29c0cdf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:07.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.897+0000 7ff29a800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff29c15e870 0x7ff29c200760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:34886/0 (socket says 172.21.15.96:34886) 2024-05-28T06:12:07.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.897+0000 7ff29a800640 1 -- 172.21.15.96:0/4036710919 learned_addr learned my addr 172.21.15.96:0/4036710919 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:07.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.897+0000 7ff299400640 1 --2- 172.21.15.96:0/4036710919 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff29c0d14d0 0x7ff29c0cc710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:07.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.898+0000 7ff299e00640 1 -- 172.21.15.96:0/4036710919 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff29c0d14d0 msgr2=0x7ff29c0cc710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:07.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.898+0000 7ff299e00640 1 --2- 172.21.15.96:0/4036710919 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff29c0d14d0 0x7ff29c0cc710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:07.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.898+0000 7ff299e00640 1 -- 172.21.15.96:0/4036710919 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff29c15e870 msgr2=0x7ff29c200760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:07.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.898+0000 7ff299e00640 1 --2- 172.21.15.96:0/4036710919 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff29c15e870 0x7ff29c200760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:07.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.898+0000 7ff299e00640 1 -- 172.21.15.96:0/4036710919 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff29c0ccc50 con 0x7ff29c0cf870 2024-05-28T06:12:07.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.898+0000 7ff299e00640 1 --2- 172.21.15.96:0/4036710919 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff29c0cf870 0x7ff29c0cdf70 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ff28c00c820 tx=0x7ff28c00cce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:07.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.899+0000 7ff292000640 1 -- 172.21.15.96:0/4036710919 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff28c015070 con 0x7ff29c0cf870 2024-05-28T06:12:07.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.899+0000 7ff292000640 1 -- 172.21.15.96:0/4036710919 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff28c007a30 con 0x7ff29c0cf870 2024-05-28T06:12:07.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.899+0000 7ff292000640 1 -- 172.21.15.96:0/4036710919 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff28c007de0 con 0x7ff29c0cf870 2024-05-28T06:12:07.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.899+0000 7ff29b200640 1 -- 172.21.15.96:0/4036710919 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff29c200ca0 con 0x7ff29c0cf870 2024-05-28T06:12:07.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.899+0000 7ff29b200640 1 -- 172.21.15.96:0/4036710919 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff29c0cd1b0 con 0x7ff29c0cf870 2024-05-28T06:12:07.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.901+0000 7ff29b200640 1 -- 172.21.15.96:0/4036710919 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff29c0689e0 con 0x7ff29c0cf870 2024-05-28T06:12:07.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.907+0000 7ff292000640 1 -- 172.21.15.96:0/4036710919 <== mon.2 v2:172.21.15.123:3301/0 4 ==== mgrmap(e 16) ==== 91754+0+0 (secure 0 0 0) 0x7ff28c018070 con 0x7ff29c0cf870 2024-05-28T06:12:07.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.909+0000 7ff292000640 1 --2- 172.21.15.96:0/4036710919 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff27c06d1a0 0x7ff27c06f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:07.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.909+0000 7ff292000640 1 -- 172.21.15.96:0/4036710919 <== mon.2 v2:172.21.15.123:3301/0 5 ==== osd_map(48..48 src has 1..48) ==== 5851+0+0 (secure 0 0 0) 0x7ff28c08cdd0 con 0x7ff29c0cf870 2024-05-28T06:12:07.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.909+0000 7ff299400640 1 --2- 172.21.15.96:0/4036710919 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff27c06d1a0 0x7ff27c06f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:07.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.912+0000 7ff299400640 1 --2- 172.21.15.96:0/4036710919 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff27c06d1a0 0x7ff27c06f660 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7ff288034000 tx=0x7ff28802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:07.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:07.913+0000 7ff292000640 1 -- 172.21.15.96:0/4036710919 <== mon.2 v2:172.21.15.123:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff29c0ccc50 con 0x7ff29c0cf870 2024-05-28T06:12:08.282 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:08.280+0000 7ff29b200640 1 -- 172.21.15.96:0/4036710919 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7ff29c0ce940 con 0x7ff29c0cf870 2024-05-28T06:12:08.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:08.281+0000 7ff292000640 1 -- 172.21.15.96:0/4036710919 <== mon.2 v2:172.21.15.123:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v48) ==== 100+0+11 (secure 0 0 0) 0x7ff28c059f10 con 0x7ff29c0cf870 2024-05-28T06:12:08.283 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 64 2024-05-28T06:12:08.290 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:08.289+0000 7ff29b200640 1 -- 172.21.15.96:0/4036710919 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff27c06d1a0 msgr2=0x7ff27c06f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:08.290 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:08.289+0000 7ff29b200640 1 --2- 172.21.15.96:0/4036710919 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff27c06d1a0 0x7ff27c06f660 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7ff288034000 tx=0x7ff28802e000 comp rx=0 tx=0).stop 2024-05-28T06:12:08.290 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:08.289+0000 7ff29b200640 1 -- 172.21.15.96:0/4036710919 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff29c0cf870 msgr2=0x7ff29c0cdf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:08.290 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:08.289+0000 7ff29b200640 1 --2- 172.21.15.96:0/4036710919 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff29c0cf870 0x7ff29c0cdf70 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ff28c00c820 tx=0x7ff28c00cce0 comp rx=0 tx=0).stop 2024-05-28T06:12:08.291 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:08.289+0000 7ff29b200640 1 -- 172.21.15.96:0/4036710919 shutdown_connections 2024-05-28T06:12:08.291 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:08.289+0000 7ff29b200640 1 --2- 172.21.15.96:0/4036710919 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff27c06d1a0 0x7ff27c06f660 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:08.291 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:08.289+0000 7ff29b200640 1 --2- 172.21.15.96:0/4036710919 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff29c15e870 0x7ff29c200760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:08.291 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:08.289+0000 7ff29b200640 1 --2- 172.21.15.96:0/4036710919 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff29c0d14d0 0x7ff29c0cc710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:08.291 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:08.289+0000 7ff29b200640 1 --2- 172.21.15.96:0/4036710919 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff29c0cf870 0x7ff29c0cdf70 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:08.292 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:08.289+0000 7ff29b200640 1 -- 172.21.15.96:0/4036710919 >> 172.21.15.96:0/4036710919 conn(0x7ff29c08bfc0 msgr2=0x7ff29c0b6310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:08.292 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:08.290+0000 7ff29b200640 1 -- 172.21.15.96:0/4036710919 shutdown_connections 2024-05-28T06:12:08.292 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:08.290+0000 7ff29b200640 1 -- 172.21.15.96:0/4036710919 wait complete. 2024-05-28T06:12:08.316 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-28T06:12:08.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:08.496+0000 7fbd1f600640 1 Processor -- start 2024-05-28T06:12:08.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:08.496+0000 7fbd1f600640 1 -- start start 2024-05-28T06:12:08.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:08.497+0000 7fbd1f600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fbd180d0590 0x7fbd180d0a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:08.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:08.497+0000 7fbd1f600640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd180b51f0 con 0x7fbd18099090 2024-05-28T06:12:08.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:08.497+0000 7fbd1f600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd180aef60 con 0x7fbd180b4e10 2024-05-28T06:12:08.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:08.497+0000 7fbd1f600640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fbd180ad1d0 con 0x7fbd180d0590 2024-05-28T06:12:08.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:08.497+0000 7fbd17e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fbd180d0590 0x7fbd180d0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:08.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:08.497+0000 7fbd17e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fbd180d0590 0x7fbd180d0a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:34888/0 (socket says 172.21.15.96:34888) 2024-05-28T06:12:08.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:08.497+0000 7fbd1d600640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fbd18099090 0x7fbd180994a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:59752/0 (socket says 172.21.15.96:59752) 2024-05-28T06:12:08.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:08.497+0000 7fbd17e00640 1 -- 172.21.15.96:0/1021748980 learned_addr learned my addr 172.21.15.96:0/1021748980 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:08.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:08.497+0000 7fbd17e00640 1 -- 172.21.15.96:0/1021748980 >> v1:172.21.15.123:6789/0 conn(0x7fbd180b4e10 legacy=0x7fbd180cde30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:08.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:08.498+0000 7fbd17e00640 1 -- 172.21.15.96:0/1021748980 >> v1:172.21.15.96:6789/0 conn(0x7fbd18099090 legacy=0x7fbd180994a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:08.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:08.498+0000 7fbd17e00640 1 -- 172.21.15.96:0/1021748980 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd180ad470 con 0x7fbd180d0590 2024-05-28T06:12:08.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:08.498+0000 7fbd17400640 1 -- 172.21.15.96:0/1021748980 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 999420722 0 0) 0x7fbd180b51f0 con 0x7fbd18099090 2024-05-28T06:12:08.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:08.499+0000 7fbd17e00640 1 --2- 172.21.15.96:0/1021748980 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbd180d0590 0x7fbd180d0a20 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fbd04009870 tx=0x7fbd0402a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5f05192713949d8 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:08.501 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:08.499+0000 7fbd17400640 1 -- 172.21.15.96:0/1021748980 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd04037070 con 0x7fbd180d0590 2024-05-28T06:12:08.501 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:08.499+0000 7fbd17400640 1 -- 172.21.15.96:0/1021748980 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd04032440 con 0x7fbd180d0590 2024-05-28T06:12:08.501 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:08.499+0000 7fbd17400640 1 -- 172.21.15.96:0/1021748980 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd04032810 con 0x7fbd180d0590 2024-05-28T06:12:08.501 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:08.499+0000 7fbd1f600640 1 -- 172.21.15.96:0/1021748980 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbd180d0590 msgr2=0x7fbd180d0a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:08.501 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:08.499+0000 7fbd1f600640 1 --2- 172.21.15.96:0/1021748980 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbd180d0590 0x7fbd180d0a20 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fbd04009870 tx=0x7fbd0402a210 comp rx=0 tx=0).stop 2024-05-28T06:12:08.501 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:08.499+0000 7fbd1f600640 1 -- 172.21.15.96:0/1021748980 shutdown_connections 2024-05-28T06:12:08.501 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:08.499+0000 7fbd1f600640 1 --2- 172.21.15.96:0/1021748980 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbd180d0590 0x7fbd180d0a20 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:08.502 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:08.499+0000 7fbd1f600640 1 -- 172.21.15.96:0/1021748980 >> 172.21.15.96:0/1021748980 conn(0x7fbd1806bae0 msgr2=0x7fbd1806bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:08.502 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:08.499+0000 7fbd1f600640 1 -- 172.21.15.96:0/1021748980 shutdown_connections 2024-05-28T06:12:08.502 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:08.499+0000 7fbd1f600640 1 -- 172.21.15.96:0/1021748980 wait complete. 2024-05-28T06:12:08.502 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:08.501+0000 7fbd1f600640 1 Processor -- start 2024-05-28T06:12:08.502 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:08.501+0000 7fbd1f600640 1 -- start start 2024-05-28T06:12:08.505 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:08.502+0000 7fbd1f600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbd18099090 0x7fbd1815b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:08.505 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:08.502+0000 7fbd1f600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbd180b4e10 0x7fbd180c6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:08.505 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:08.502+0000 7fbd1f600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbd180d0590 0x7fbd180b14e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:08.505 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:08.502+0000 7fbd1f600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fbd180ce880 con 0x7fbd18099090 2024-05-28T06:12:08.505 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:08.502+0000 7fbd1f600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fbd180ce540 con 0x7fbd180d0590 2024-05-28T06:12:08.505 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:08.502+0000 7fbd1f600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fbd180cea00 con 0x7fbd180b4e10 2024-05-28T06:12:08.505 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:08.502+0000 7fbd1cc00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbd18099090 0x7fbd1815b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:08.505 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:08.502+0000 7fbd1cc00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbd18099090 0x7fbd1815b240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:34902/0 (socket says 172.21.15.96:34902) 2024-05-28T06:12:08.505 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:08.502+0000 7fbd1cc00640 1 -- 172.21.15.96:0/2079903620 learned_addr learned my addr 172.21.15.96:0/2079903620 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:08.506 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:08.503+0000 7fbd1cc00640 1 -- 172.21.15.96:0/2079903620 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbd180b4e10 msgr2=0x7fbd180c6ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:08.506 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:08.503+0000 7fbd1cc00640 1 --2- 172.21.15.96:0/2079903620 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbd180b4e10 0x7fbd180c6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:08.506 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:08.503+0000 7fbd1cc00640 1 -- 172.21.15.96:0/2079903620 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbd180d0590 msgr2=0x7fbd180b14e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:08.506 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:08.503+0000 7fbd1cc00640 1 --2- 172.21.15.96:0/2079903620 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbd180d0590 0x7fbd180b14e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:08.506 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:08.503+0000 7fbd1cc00640 1 -- 172.21.15.96:0/2079903620 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd180b0760 con 0x7fbd18099090 2024-05-28T06:12:08.506 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:08.503+0000 7fbd1cc00640 1 --2- 172.21.15.96:0/2079903620 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbd18099090 0x7fbd1815b240 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fbd0800c430 tx=0x7fbd0800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:08.506 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:08.503+0000 7fbd14c00640 1 -- 172.21.15.96:0/2079903620 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd08011790 con 0x7fbd18099090 2024-05-28T06:12:08.506 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:08.503+0000 7fbd14c00640 1 -- 172.21.15.96:0/2079903620 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd08017070 con 0x7fbd18099090 2024-05-28T06:12:08.506 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:08.503+0000 7fbd14c00640 1 -- 172.21.15.96:0/2079903620 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd08011bb0 con 0x7fbd18099090 2024-05-28T06:12:08.506 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:08.503+0000 7fbd1f600640 1 -- 172.21.15.96:0/2079903620 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd180c9180 con 0x7fbd18099090 2024-05-28T06:12:08.506 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:08.503+0000 7fbd1f600640 1 -- 172.21.15.96:0/2079903620 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd180b0ce0 con 0x7fbd18099090 2024-05-28T06:12:08.507 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:08.505+0000 7fbd1f600640 1 -- 172.21.15.96:0/2079903620 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd180689e0 con 0x7fbd18099090 2024-05-28T06:12:08.507 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:08.505+0000 7fbd14c00640 1 -- 172.21.15.96:0/2079903620 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 16) ==== 91754+0+0 (secure 0 0 0) 0x7fbd0801c460 con 0x7fbd18099090 2024-05-28T06:12:08.508 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:08.506+0000 7fbd14c00640 1 --2- 172.21.15.96:0/2079903620 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbcf806d0d0 0x7fbcf806f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:08.508 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:08.506+0000 7fbd14c00640 1 -- 172.21.15.96:0/2079903620 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5851+0+0 (secure 0 0 0) 0x7fbd0808cdf0 con 0x7fbd18099090 2024-05-28T06:12:08.508 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:08.507+0000 7fbd17e00640 1 --2- 172.21.15.96:0/2079903620 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbcf806d0d0 0x7fbcf806f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:08.509 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:08.508+0000 7fbd17e00640 1 --2- 172.21.15.96:0/2079903620 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbcf806d0d0 0x7fbcf806f590 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fbd04034000 tx=0x7fbd0402c000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:08.512 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:08.511+0000 7fbd14c00640 1 -- 172.21.15.96:0/2079903620 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbd0805a170 con 0x7fbd18099090 2024-05-28T06:12:08.712 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:08.711+0000 7fbd1f600640 1 -- 172.21.15.96:0/2079903620 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fbd180cf100 con 0x7fbd18099090 2024-05-28T06:12:08.713 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:08.711+0000 7fbd14c00640 1 -- 172.21.15.96:0/2079903620 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v48) ==== 80+0+0 (secure 0 0 0) 0x7fbd08059a70 con 0x7fbd18099090 2024-05-28T06:12:08.713 INFO:teuthology.orchestra.run.smithi096.stderr:listed 0 entries 2024-05-28T06:12:08.717 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:08.715+0000 7fbd1f600640 1 -- 172.21.15.96:0/2079903620 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbcf806d0d0 msgr2=0x7fbcf806f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:08.717 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:08.715+0000 7fbd1f600640 1 --2- 172.21.15.96:0/2079903620 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbcf806d0d0 0x7fbcf806f590 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fbd04034000 tx=0x7fbd0402c000 comp rx=0 tx=0).stop 2024-05-28T06:12:08.718 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:08.715+0000 7fbd1f600640 1 -- 172.21.15.96:0/2079903620 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbd18099090 msgr2=0x7fbd1815b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:08.718 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:08.715+0000 7fbd1f600640 1 --2- 172.21.15.96:0/2079903620 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbd18099090 0x7fbd1815b240 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fbd0800c430 tx=0x7fbd0800c8f0 comp rx=0 tx=0).stop 2024-05-28T06:12:08.718 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:08.716+0000 7fbd1f600640 1 -- 172.21.15.96:0/2079903620 shutdown_connections 2024-05-28T06:12:08.718 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:08.716+0000 7fbd1f600640 1 --2- 172.21.15.96:0/2079903620 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbcf806d0d0 0x7fbcf806f590 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:08.718 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:08.716+0000 7fbd1f600640 1 --2- 172.21.15.96:0/2079903620 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbd180d0590 0x7fbd180b14e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:08.718 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:08.716+0000 7fbd1f600640 1 --2- 172.21.15.96:0/2079903620 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbd180b4e10 0x7fbd180c6ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:08.718 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:08.716+0000 7fbd1f600640 1 --2- 172.21.15.96:0/2079903620 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbd18099090 0x7fbd1815b240 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:08.718 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:08.716+0000 7fbd1f600640 1 -- 172.21.15.96:0/2079903620 >> 172.21.15.96:0/2079903620 conn(0x7fbd1806bae0 msgr2=0x7fbd180cb8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:08.718 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:08.716+0000 7fbd1f600640 1 -- 172.21.15.96:0/2079903620 shutdown_connections 2024-05-28T06:12:08.718 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:08.716+0000 7fbd1f600640 1 -- 172.21.15.96:0/2079903620 wait complete. 2024-05-28T06:12:08.735 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi096.front.sepia.ceph.com... 2024-05-28T06:12:08.735 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:12:08.735 DEBUG:teuthology.orchestra.run.smithi096:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-05-28T06:12:08.814 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi096.front.sepia.ceph.com 2024-05-28T06:12:08.814 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:12:08.814 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-28T06:12:08.826 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi096.stderr:ceph-fuse[55615]: fuse finished with error 0 and tester_r 0 2024-05-28T06:12:08.979 INFO:tasks.cephfs.mount:Terminating background process 2024-05-28T06:12:08.979 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-05-28T06:12:08.979 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:12:09.021 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052731,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248996k,nr_inodes=812249,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-05-28T06:12:09.021 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi123.front.sepia.ceph.com 2024-05-28T06:12:09.021 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:12:09.022 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-05-28T06:12:09.111 INFO:teuthology.orchestra.run.smithi123.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-05-28T06:12:09.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-28T06:12:09.113 INFO:tasks.cephfs.mount:Terminating background process 2024-05-28T06:12:09.113 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:12:09.113 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:12:09.167 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052731,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248996k,nr_inodes=812249,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-28T06:12:09.167 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi096.front.sepia.ceph.com 2024-05-28T06:12:09.167 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:12:09.167 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-05-28T06:12:09.215 INFO:teuthology.orchestra.run.smithi096.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-05-28T06:12:09.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-28T06:12:09.216 INFO:tasks.cephfs.mount:Terminating background process 2024-05-28T06:12:09.216 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-05-28T06:12:09.216 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:12:09.272 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052731,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248996k,nr_inodes=812249,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-05-28T06:12:09.272 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi123.front.sepia.ceph.com 2024-05-28T06:12:09.272 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:12:09.272 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-05-28T06:12:09.324 INFO:teuthology.orchestra.run.smithi123.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-05-28T06:12:09.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-28T06:12:09.326 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-28T06:12:09.530 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.528+0000 7f0a5b200640 1 Processor -- start 2024-05-28T06:12:09.530 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.529+0000 7f0a5b200640 1 -- start start 2024-05-28T06:12:09.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.530+0000 7f0a5b200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f0a5415e870 0x7f0a540d0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:09.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.530+0000 7f0a5b200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f0a540d14d0 0x7f0a540cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:09.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.530+0000 7f0a5b200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a540cfce0 con 0x7f0a540cf870 2024-05-28T06:12:09.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.530+0000 7f0a5b200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f0a540d2150 con 0x7f0a5415e870 2024-05-28T06:12:09.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.530+0000 7f0a5b200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f0a5415ec50 con 0x7f0a540d14d0 2024-05-28T06:12:09.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.530+0000 7f0a59e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f0a5415e870 0x7f0a540d0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:09.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.530+0000 7f0a59e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f0a5415e870 0x7f0a540d0f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:60806/0 (socket says 172.21.15.96:60806) 2024-05-28T06:12:09.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.530+0000 7f0a59e00640 1 -- 172.21.15.96:0/3077747697 learned_addr learned my addr 172.21.15.96:0/3077747697 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:09.533 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.531+0000 7f0a59e00640 1 -- 172.21.15.96:0/3077747697 >> v2:172.21.15.123:3300/0 conn(0x7f0a540d14d0 msgr2=0x7f0a540cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:09.533 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.531+0000 7f0a59e00640 1 --2- 172.21.15.96:0/3077747697 >> v2:172.21.15.123:3300/0 conn(0x7f0a540d14d0 0x7f0a540cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:09.533 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.531+0000 7f0a59e00640 1 -- 172.21.15.96:0/3077747697 >> v1:172.21.15.123:6789/0 conn(0x7f0a540cf870 legacy=0x7f0a54162360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:09.533 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.531+0000 7f0a59e00640 1 -- 172.21.15.96:0/3077747697 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a540cff80 con 0x7f0a5415e870 2024-05-28T06:12:09.533 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.532+0000 7f0a59e00640 1 --2- 172.21.15.96:0/3077747697 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a5415e870 0x7f0a540d0f90 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f0a5408e520 tx=0x7f0a4802a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e22b886b2767ea0f server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:09.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.532+0000 7f0a58a00640 1 -- 172.21.15.96:0/3077747697 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a48037070 con 0x7f0a5415e870 2024-05-28T06:12:09.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.532+0000 7f0a58a00640 1 -- 172.21.15.96:0/3077747697 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a48032440 con 0x7f0a5415e870 2024-05-28T06:12:09.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.532+0000 7f0a58a00640 1 -- 172.21.15.96:0/3077747697 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a48032810 con 0x7f0a5415e870 2024-05-28T06:12:09.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.533+0000 7f0a5b200640 1 -- 172.21.15.96:0/3077747697 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a5415e870 msgr2=0x7f0a540d0f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:09.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.533+0000 7f0a5b200640 1 --2- 172.21.15.96:0/3077747697 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a5415e870 0x7f0a540d0f90 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f0a5408e520 tx=0x7f0a4802a230 comp rx=0 tx=0).stop 2024-05-28T06:12:09.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.533+0000 7f0a5b200640 1 -- 172.21.15.96:0/3077747697 shutdown_connections 2024-05-28T06:12:09.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.533+0000 7f0a5b200640 1 --2- 172.21.15.96:0/3077747697 >> v2:172.21.15.123:3300/0 conn(0x7f0a540d14d0 0x7f0a540cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:09.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.533+0000 7f0a5b200640 1 --2- 172.21.15.96:0/3077747697 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a5415e870 0x7f0a540d0f90 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:09.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.533+0000 7f0a5b200640 1 -- 172.21.15.96:0/3077747697 >> 172.21.15.96:0/3077747697 conn(0x7f0a5408bfc0 msgr2=0x7f0a540d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:09.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.533+0000 7f0a5b200640 1 -- 172.21.15.96:0/3077747697 shutdown_connections 2024-05-28T06:12:09.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.533+0000 7f0a5b200640 1 -- 172.21.15.96:0/3077747697 wait complete. 2024-05-28T06:12:09.536 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.534+0000 7f0a5b200640 1 Processor -- start 2024-05-28T06:12:09.536 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.535+0000 7f0a5b200640 1 -- start start 2024-05-28T06:12:09.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.535+0000 7f0a5b200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0a540cf870 0x7f0a54158170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:09.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.535+0000 7f0a5b200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a540d14d0 0x7f0a540b7b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:09.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.535+0000 7f0a5b200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0a5415e870 0x7f0a540b6010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:09.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.535+0000 7f0a5b200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f0a540d0820 con 0x7f0a540d14d0 2024-05-28T06:12:09.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.535+0000 7f0a5b200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f0a540d0540 con 0x7f0a540cf870 2024-05-28T06:12:09.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.535+0000 7f0a5b200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f0a540d4fb0 con 0x7f0a5415e870 2024-05-28T06:12:09.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.536+0000 7f0a59400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a540d14d0 0x7f0a540b7b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:09.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.536+0000 7f0a59400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a540d14d0 0x7f0a540b7b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:60814/0 (socket says 172.21.15.96:60814) 2024-05-28T06:12:09.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.536+0000 7f0a59400640 1 -- 172.21.15.96:0/1488366312 learned_addr learned my addr 172.21.15.96:0/1488366312 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:09.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.536+0000 7f0a59e00640 1 --2- 172.21.15.96:0/1488366312 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0a540cf870 0x7f0a54158170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:09.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.536+0000 7f0a5a800640 1 --2- 172.21.15.96:0/1488366312 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0a5415e870 0x7f0a540b6010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:09.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.536+0000 7f0a59400640 1 -- 172.21.15.96:0/1488366312 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0a5415e870 msgr2=0x7f0a540b6010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:09.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.536+0000 7f0a59400640 1 --2- 172.21.15.96:0/1488366312 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0a5415e870 0x7f0a540b6010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:09.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.536+0000 7f0a59400640 1 -- 172.21.15.96:0/1488366312 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0a540cf870 msgr2=0x7f0a54158170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:09.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.536+0000 7f0a59400640 1 --2- 172.21.15.96:0/1488366312 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0a540cf870 0x7f0a54158170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:09.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.536+0000 7f0a59400640 1 -- 172.21.15.96:0/1488366312 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a541586b0 con 0x7f0a540d14d0 2024-05-28T06:12:09.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.536+0000 7f0a5a800640 1 --2- 172.21.15.96:0/1488366312 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0a5415e870 0x7f0a540b6010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:12:09.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.537+0000 7f0a59400640 1 --2- 172.21.15.96:0/1488366312 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a540d14d0 0x7f0a540b7b60 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f0a4400b4d0 tx=0x7f0a4400b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:09.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.537+0000 7f0a4e000640 1 -- 172.21.15.96:0/1488366312 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a44014070 con 0x7f0a540d14d0 2024-05-28T06:12:09.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.537+0000 7f0a5b200640 1 -- 172.21.15.96:0/1488366312 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a540b80a0 con 0x7f0a540d14d0 2024-05-28T06:12:09.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.537+0000 7f0a4e000640 1 -- 172.21.15.96:0/1488366312 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a44005c10 con 0x7f0a540d14d0 2024-05-28T06:12:09.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.537+0000 7f0a4e000640 1 -- 172.21.15.96:0/1488366312 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a44010670 con 0x7f0a540d14d0 2024-05-28T06:12:09.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.537+0000 7f0a5b200640 1 -- 172.21.15.96:0/1488366312 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a5415f4f0 con 0x7f0a540d14d0 2024-05-28T06:12:09.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.538+0000 7f0a5b200640 1 -- 172.21.15.96:0/1488366312 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a540689e0 con 0x7f0a540d14d0 2024-05-28T06:12:09.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.539+0000 7f0a4e000640 1 -- 172.21.15.96:0/1488366312 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 16) ==== 91754+0+0 (secure 0 0 0) 0x7f0a44010810 con 0x7f0a540d14d0 2024-05-28T06:12:09.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.540+0000 7f0a4e000640 1 --2- 172.21.15.96:0/1488366312 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0a3806d0d0 0x7f0a3806f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:09.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.541+0000 7f0a4e000640 1 -- 172.21.15.96:0/1488366312 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5851+0+0 (secure 0 0 0) 0x7f0a4408cc10 con 0x7f0a540d14d0 2024-05-28T06:12:09.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.541+0000 7f0a59e00640 1 --2- 172.21.15.96:0/1488366312 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0a3806d0d0 0x7f0a3806f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:09.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.542+0000 7f0a59e00640 1 --2- 172.21.15.96:0/1488366312 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0a3806d0d0 0x7f0a3806f590 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f0a48034000 tx=0x7f0a4802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:09.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.545+0000 7f0a4e000640 1 -- 172.21.15.96:0/1488366312 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0a44059fe0 con 0x7f0a540d14d0 2024-05-28T06:12:09.775 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.773+0000 7f0a5b200640 1 -- 172.21.15.96:0/1488366312 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0a540d3770 con 0x7f0a540d14d0 2024-05-28T06:12:09.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.775+0000 7f0a4e000640 1 -- 172.21.15.96:0/1488366312 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+5181 (secure 0 0 0) 0x7f0a4405ee20 con 0x7f0a540d14d0 2024-05-28T06:12:09.777 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:12:09.777 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":26,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/1693172076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":1693172076},{"type":"v1","addr":"172.21.15.123:6835","nonce":1693172076}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":4805,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/1098611268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":1098611268},{"type":"v1","addr":"172.21.15.123:6837","nonce":1098611268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5856,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6834/673165997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":673165997},{"type":"v1","addr":"172.21.15.96:6834","nonce":673165997}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-28T06:08:54.477674+0000","modified":"2024-05-28T06:08:57.572929+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5859},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5859":{"gid":5859,"name":"a","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.96:6835/935552515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6833","nonce":935552515},{"type":"v1","addr":"172.21.15.96:6835","nonce":935552515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5859,"qdb_cluster":[5859]},"id":3}]} 2024-05-28T06:12:09.777 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 26 2024-05-28T06:12:09.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.777+0000 7f0a5b200640 1 -- 172.21.15.96:0/1488366312 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0a3806d0d0 msgr2=0x7f0a3806f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:09.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.777+0000 7f0a5b200640 1 --2- 172.21.15.96:0/1488366312 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0a3806d0d0 0x7f0a3806f590 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f0a48034000 tx=0x7f0a4802e000 comp rx=0 tx=0).stop 2024-05-28T06:12:09.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.777+0000 7f0a5b200640 1 -- 172.21.15.96:0/1488366312 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a540d14d0 msgr2=0x7f0a540b7b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:09.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.777+0000 7f0a5b200640 1 --2- 172.21.15.96:0/1488366312 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a540d14d0 0x7f0a540b7b60 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f0a4400b4d0 tx=0x7f0a4400b990 comp rx=0 tx=0).stop 2024-05-28T06:12:09.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.777+0000 7f0a5b200640 1 -- 172.21.15.96:0/1488366312 shutdown_connections 2024-05-28T06:12:09.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.777+0000 7f0a5b200640 1 --2- 172.21.15.96:0/1488366312 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0a3806d0d0 0x7f0a3806f590 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:09.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.777+0000 7f0a5b200640 1 --2- 172.21.15.96:0/1488366312 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0a5415e870 0x7f0a540b6010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:09.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.777+0000 7f0a5b200640 1 --2- 172.21.15.96:0/1488366312 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a540d14d0 0x7f0a540b7b60 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:09.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.777+0000 7f0a5b200640 1 --2- 172.21.15.96:0/1488366312 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0a540cf870 0x7f0a54158170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:09.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.777+0000 7f0a5b200640 1 -- 172.21.15.96:0/1488366312 >> 172.21.15.96:0/1488366312 conn(0x7f0a5408bfc0 msgr2=0x7f0a540b2ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:09.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.778+0000 7f0a5b200640 1 -- 172.21.15.96:0/1488366312 shutdown_connections 2024-05-28T06:12:09.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.778+0000 7f0a5b200640 1 -- 172.21.15.96:0/1488366312 wait complete. 2024-05-28T06:12:09.798 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:12:09.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.993+0000 7f13e1600640 1 Processor -- start 2024-05-28T06:12:09.995 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.993+0000 7f13e1600640 1 -- start start 2024-05-28T06:12:09.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.994+0000 7f13e1600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f13dc071bc0 0x7f13dc072070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:09.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.994+0000 7f13e1600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f13dc0b32c0 0x7f13dc0b3710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:09.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.995+0000 7f13e1600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13dc06e6e0 con 0x7f13dc158b80 2024-05-28T06:12:09.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.995+0000 7f13e1600640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f13dc06e9c0 con 0x7f13dc0b32c0 2024-05-28T06:12:09.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.995+0000 7f13e1600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f13dc06eca0 con 0x7f13dc071bc0 2024-05-28T06:12:09.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.995+0000 7f13e0c00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f13dc0b32c0 0x7f13dc0b3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:09.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.995+0000 7f13e0c00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f13dc0b32c0 0x7f13dc0b3710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:60818/0 (socket says 172.21.15.96:60818) 2024-05-28T06:12:09.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.995+0000 7f13e0c00640 1 -- 172.21.15.96:0/1860145692 learned_addr learned my addr 172.21.15.96:0/1860145692 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:09.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.995+0000 7f13db400640 1 --2- 172.21.15.96:0/1860145692 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f13dc071bc0 0x7f13dc072070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:09.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.995+0000 7f13e0c00640 1 -- 172.21.15.96:0/1860145692 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f13dc071bc0 msgr2=0x7f13dc072070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:09.998 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.995+0000 7f13e0c00640 1 --2- 172.21.15.96:0/1860145692 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f13dc071bc0 0x7f13dc072070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:09.998 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.995+0000 7f13e0c00640 1 -- 172.21.15.96:0/1860145692 >> v1:172.21.15.123:6789/0 conn(0x7f13dc158b80 legacy=0x7f13dc06dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:09.998 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.995+0000 7f13e0c00640 1 -- 172.21.15.96:0/1860145692 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13dc06ef80 con 0x7f13dc0b32c0 2024-05-28T06:12:09.998 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.996+0000 7f13e0c00640 1 --2- 172.21.15.96:0/1860145692 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13dc0b32c0 0x7f13dc0b3710 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f13d4009870 tx=0x7f13d402a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aeb5743cdaf19108 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:09.998 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.997+0000 7f13daa00640 1 -- 172.21.15.96:0/1860145692 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d4037070 con 0x7f13dc0b32c0 2024-05-28T06:12:09.999 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.997+0000 7f13daa00640 1 -- 172.21.15.96:0/1860145692 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13d4032440 con 0x7f13dc0b32c0 2024-05-28T06:12:09.999 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.997+0000 7f13daa00640 1 -- 172.21.15.96:0/1860145692 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d4032810 con 0x7f13dc0b32c0 2024-05-28T06:12:09.999 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.997+0000 7f13e1600640 1 -- 172.21.15.96:0/1860145692 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13dc0b32c0 msgr2=0x7f13dc0b3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:09.999 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.997+0000 7f13e1600640 1 --2- 172.21.15.96:0/1860145692 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13dc0b32c0 0x7f13dc0b3710 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f13d4009870 tx=0x7f13d402a210 comp rx=0 tx=0).stop 2024-05-28T06:12:09.999 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.998+0000 7f13e1600640 1 -- 172.21.15.96:0/1860145692 shutdown_connections 2024-05-28T06:12:10.000 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.998+0000 7f13e1600640 1 --2- 172.21.15.96:0/1860145692 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13dc0b32c0 0x7f13dc0b3710 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:10.000 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.998+0000 7f13e1600640 1 --2- 172.21.15.96:0/1860145692 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f13dc071bc0 0x7f13dc072070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:10.000 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.998+0000 7f13e1600640 1 -- 172.21.15.96:0/1860145692 >> 172.21.15.96:0/1860145692 conn(0x7f13dc0b1fc0 msgr2=0x7f13dc0b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:10.000 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.998+0000 7f13e1600640 1 -- 172.21.15.96:0/1860145692 shutdown_connections 2024-05-28T06:12:10.000 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.998+0000 7f13e1600640 1 -- 172.21.15.96:0/1860145692 wait complete. 2024-05-28T06:12:10.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:09.999+0000 7f13e1600640 1 Processor -- start 2024-05-28T06:12:10.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.000+0000 7f13e1600640 1 -- start start 2024-05-28T06:12:10.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.000+0000 7f13e1600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f13dc071bc0 0x7f13dc099a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:10.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.000+0000 7f13e1600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f13dc0b32c0 0x7f13dc1f80b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:10.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.001+0000 7f13e1600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13dc158b80 0x7f13dc09d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:10.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.001+0000 7f13e1600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f13dc06f780 con 0x7f13dc158b80 2024-05-28T06:12:10.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.001+0000 7f13e1600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f13dc0b2d30 con 0x7f13dc0b32c0 2024-05-28T06:12:10.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.001+0000 7f13e1600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f13dc1f85f0 con 0x7f13dc071bc0 2024-05-28T06:12:10.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.001+0000 7f13e0c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13dc158b80 0x7f13dc09d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:10.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.001+0000 7f13e0c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13dc158b80 0x7f13dc09d330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:60834/0 (socket says 172.21.15.96:60834) 2024-05-28T06:12:10.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.001+0000 7f13e0c00640 1 -- 172.21.15.96:0/2957750325 learned_addr learned my addr 172.21.15.96:0/2957750325 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:10.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.001+0000 7f13db400640 1 --2- 172.21.15.96:0/2957750325 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f13dc0b32c0 0x7f13dc1f80b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:10.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.001+0000 7f13dbe00640 1 --2- 172.21.15.96:0/2957750325 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f13dc071bc0 0x7f13dc099a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:10.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.001+0000 7f13e0c00640 1 -- 172.21.15.96:0/2957750325 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f13dc071bc0 msgr2=0x7f13dc099a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:10.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.001+0000 7f13e0c00640 1 --2- 172.21.15.96:0/2957750325 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f13dc071bc0 0x7f13dc099a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:10.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.001+0000 7f13e0c00640 1 -- 172.21.15.96:0/2957750325 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f13dc0b32c0 msgr2=0x7f13dc1f80b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:10.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.001+0000 7f13e0c00640 1 --2- 172.21.15.96:0/2957750325 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f13dc0b32c0 0x7f13dc1f80b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:10.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.001+0000 7f13e0c00640 1 -- 172.21.15.96:0/2957750325 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13dc09ded0 con 0x7f13dc158b80 2024-05-28T06:12:10.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.002+0000 7f13e0c00640 1 --2- 172.21.15.96:0/2957750325 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13dc158b80 0x7f13dc09d330 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f13d402a250 tx=0x7f13d40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:10.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.002+0000 7f13d3e00640 1 -- 172.21.15.96:0/2957750325 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d4041020 con 0x7f13dc158b80 2024-05-28T06:12:10.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.002+0000 7f13d3e00640 1 -- 172.21.15.96:0/2957750325 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13d4030820 con 0x7f13dc158b80 2024-05-28T06:12:10.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.002+0000 7f13d3e00640 1 -- 172.21.15.96:0/2957750325 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d4037040 con 0x7f13dc158b80 2024-05-28T06:12:10.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.002+0000 7f13e1600640 1 -- 172.21.15.96:0/2957750325 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13dc09e1b0 con 0x7f13dc158b80 2024-05-28T06:12:10.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.002+0000 7f13e1600640 1 -- 172.21.15.96:0/2957750325 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13dc1f7b90 con 0x7f13dc158b80 2024-05-28T06:12:10.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.003+0000 7f13e1600640 1 -- 172.21.15.96:0/2957750325 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13dc0689e0 con 0x7f13dc158b80 2024-05-28T06:12:10.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.004+0000 7f13d3e00640 1 -- 172.21.15.96:0/2957750325 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 16) ==== 91754+0+0 (secure 0 0 0) 0x7f13d4044020 con 0x7f13dc158b80 2024-05-28T06:12:10.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.005+0000 7f13d3e00640 1 --2- 172.21.15.96:0/2957750325 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f13bc06d0d0 0x7f13bc06f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:10.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.006+0000 7f13d3e00640 1 -- 172.21.15.96:0/2957750325 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5851+0+0 (secure 0 0 0) 0x7f13d40ad710 con 0x7f13dc158b80 2024-05-28T06:12:10.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.006+0000 7f13dbe00640 1 --2- 172.21.15.96:0/2957750325 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f13bc06d0d0 0x7f13bc06f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:10.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.008+0000 7f13dbe00640 1 --2- 172.21.15.96:0/2957750325 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f13bc06d0d0 0x7f13bc06f590 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f13c8006fd0 tx=0x7f13c8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:10.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.010+0000 7f13d3e00640 1 -- 172.21.15.96:0/2957750325 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f13d404b030 con 0x7f13dc158b80 2024-05-28T06:12:10.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.213+0000 7f13e1600640 1 -- 172.21.15.96:0/2957750325 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f13dc09b120 con 0x7f13dc158b80 2024-05-28T06:12:10.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.215+0000 7f13d3e00640 1 -- 172.21.15.96:0/2957750325 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v48) ==== 74+0+16868 (secure 0 0 0) 0x7f13d407ab50 con 0x7f13dc158b80 2024-05-28T06:12:10.217 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:12:10.217 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":48,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:12:04.592827+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-05-28T06:08:52.257477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-05-28T06:08:53.459376+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"48","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":48,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.9","mappings":[{"from":0,"to":5}]},{"pgid":"7.37","mappings":[{"from":0,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":7,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[{"pool":7,"snaps":[{"begin":2,"length":2}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:12:10.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.218+0000 7f13e1600640 1 -- 172.21.15.96:0/2957750325 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f13bc06d0d0 msgr2=0x7f13bc06f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:10.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.218+0000 7f13e1600640 1 --2- 172.21.15.96:0/2957750325 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f13bc06d0d0 0x7f13bc06f590 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f13c8006fd0 tx=0x7f13c8008040 comp rx=0 tx=0).stop 2024-05-28T06:12:10.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.218+0000 7f13e1600640 1 -- 172.21.15.96:0/2957750325 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13dc158b80 msgr2=0x7f13dc09d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:10.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.218+0000 7f13e1600640 1 --2- 172.21.15.96:0/2957750325 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13dc158b80 0x7f13dc09d330 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f13d402a250 tx=0x7f13d40046c0 comp rx=0 tx=0).stop 2024-05-28T06:12:10.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.218+0000 7f13e1600640 1 -- 172.21.15.96:0/2957750325 shutdown_connections 2024-05-28T06:12:10.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.218+0000 7f13e1600640 1 --2- 172.21.15.96:0/2957750325 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f13bc06d0d0 0x7f13bc06f590 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:10.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.218+0000 7f13e1600640 1 --2- 172.21.15.96:0/2957750325 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13dc158b80 0x7f13dc09d330 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:10.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.218+0000 7f13e1600640 1 --2- 172.21.15.96:0/2957750325 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f13dc0b32c0 0x7f13dc1f80b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:10.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.218+0000 7f13e1600640 1 --2- 172.21.15.96:0/2957750325 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f13dc071bc0 0x7f13dc099a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:10.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.218+0000 7f13e1600640 1 -- 172.21.15.96:0/2957750325 >> 172.21.15.96:0/2957750325 conn(0x7f13dc0b1fc0 msgr2=0x7f13dc162740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:10.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.219+0000 7f13e1600640 1 -- 172.21.15.96:0/2957750325 shutdown_connections 2024-05-28T06:12:10.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.219+0000 7f13e1600640 1 -- 172.21.15.96:0/2957750325 wait complete. 2024-05-28T06:12:10.239 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-28T06:03:59.517102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-28T06:08:52.257477+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2024-05-28T06:08:53.459376+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '48', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 48, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-28T06:12:10.239 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-28T06:12:10.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.438+0000 7fd448800640 1 Processor -- start 2024-05-28T06:12:10.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.439+0000 7fd448800640 1 -- start start 2024-05-28T06:12:10.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.440+0000 7fd448800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fd4400d0640 0x7fd4400d0a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:10.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.440+0000 7fd448800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fd44015b6f0 0x7fd44006e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:10.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.440+0000 7fd448800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd4400725e0 0x7fd4400b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:10.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.440+0000 7fd448800640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fd440071c40 con 0x7fd4400d0640 2024-05-28T06:12:10.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.440+0000 7fd448800640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fd4400ade90 con 0x7fd44015b6f0 2024-05-28T06:12:10.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.440+0000 7fd448800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fd4400d0f60 con 0x7fd4400725e0 2024-05-28T06:12:10.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.440+0000 7fd445e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fd4400d0640 0x7fd4400d0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:10.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.440+0000 7fd445e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fd4400d0640 0x7fd4400d0a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:60838/0 (socket says 172.21.15.96:60838) 2024-05-28T06:12:10.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.440+0000 7fd445e00640 1 -- 172.21.15.96:0/2927821965 learned_addr learned my addr 172.21.15.96:0/2927821965 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:10.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.441+0000 7fd445400640 1 --2- 172.21.15.96:0/2927821965 >> v2:172.21.15.123:3300/0 conn(0x7fd44015b6f0 0x7fd44006e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:10.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.441+0000 7fd445e00640 1 -- 172.21.15.96:0/2927821965 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd4400725e0 msgr2=0x7fd4400b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:10.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.441+0000 7fd445e00640 1 --2- 172.21.15.96:0/2927821965 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd4400725e0 0x7fd4400b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:10.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.441+0000 7fd445e00640 1 -- 172.21.15.96:0/2927821965 >> v2:172.21.15.123:3300/0 conn(0x7fd44015b6f0 msgr2=0x7fd44006e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:10.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.441+0000 7fd445e00640 1 --2- 172.21.15.96:0/2927821965 >> v2:172.21.15.123:3300/0 conn(0x7fd44015b6f0 0x7fd44006e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:10.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.441+0000 7fd445e00640 1 -- 172.21.15.96:0/2927821965 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd440071e70 con 0x7fd4400d0640 2024-05-28T06:12:10.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.441+0000 7fd445400640 1 --2- 172.21.15.96:0/2927821965 >> v2:172.21.15.123:3300/0 conn(0x7fd44015b6f0 0x7fd44006e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:12:10.444 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.442+0000 7fd445e00640 1 --2- 172.21.15.96:0/2927821965 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd4400d0640 0x7fd4400d0a20 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fd4400d0220 tx=0x7fd43402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df2f93e8dc7471c6 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:10.444 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.442+0000 7fd444a00640 1 -- 172.21.15.96:0/2927821965 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd434037070 con 0x7fd4400d0640 2024-05-28T06:12:10.444 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.442+0000 7fd444a00640 1 -- 172.21.15.96:0/2927821965 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd434032440 con 0x7fd4400d0640 2024-05-28T06:12:10.444 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.442+0000 7fd444a00640 1 -- 172.21.15.96:0/2927821965 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd434032810 con 0x7fd4400d0640 2024-05-28T06:12:10.444 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.443+0000 7fd448800640 1 -- 172.21.15.96:0/2927821965 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd4400d0640 msgr2=0x7fd4400d0a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:10.444 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.443+0000 7fd448800640 1 --2- 172.21.15.96:0/2927821965 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd4400d0640 0x7fd4400d0a20 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fd4400d0220 tx=0x7fd43402a1e0 comp rx=0 tx=0).stop 2024-05-28T06:12:10.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.443+0000 7fd448800640 1 -- 172.21.15.96:0/2927821965 shutdown_connections 2024-05-28T06:12:10.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.443+0000 7fd448800640 1 --2- 172.21.15.96:0/2927821965 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd4400725e0 0x7fd4400b44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:10.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.443+0000 7fd448800640 1 --2- 172.21.15.96:0/2927821965 >> v2:172.21.15.123:3300/0 conn(0x7fd44015b6f0 0x7fd44006e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:10.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.443+0000 7fd448800640 1 --2- 172.21.15.96:0/2927821965 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd4400d0640 0x7fd4400d0a20 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:10.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.443+0000 7fd448800640 1 -- 172.21.15.96:0/2927821965 >> 172.21.15.96:0/2927821965 conn(0x7fd44006bd20 msgr2=0x7fd44006c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:10.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.443+0000 7fd448800640 1 -- 172.21.15.96:0/2927821965 shutdown_connections 2024-05-28T06:12:10.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.443+0000 7fd448800640 1 -- 172.21.15.96:0/2927821965 wait complete. 2024-05-28T06:12:10.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.445+0000 7fd448800640 1 Processor -- start 2024-05-28T06:12:10.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.446+0000 7fd448800640 1 -- start start 2024-05-28T06:12:10.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.447+0000 7fd448800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd4400725e0 0x7fd4401f4520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:10.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.447+0000 7fd448800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd4400d0640 0x7fd440157e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:10.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.447+0000 7fd448800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd44015b6f0 0x7fd4400cf330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:10.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.447+0000 7fd448800640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd44006e890 con 0x7fd4400d0640 2024-05-28T06:12:10.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.447+0000 7fd448800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd4400ae130 con 0x7fd44015b6f0 2024-05-28T06:12:10.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.447+0000 7fd448800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fd4400b4a10 con 0x7fd4400725e0 2024-05-28T06:12:10.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.447+0000 7fd445400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd4400d0640 0x7fd440157e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:10.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.447+0000 7fd445400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd4400d0640 0x7fd440157e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:60854/0 (socket says 172.21.15.96:60854) 2024-05-28T06:12:10.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.447+0000 7fd445400640 1 -- 172.21.15.96:0/1990082401 learned_addr learned my addr 172.21.15.96:0/1990082401 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:10.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.448+0000 7fd445e00640 1 --2- 172.21.15.96:0/1990082401 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd4400725e0 0x7fd4401f4520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:10.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.448+0000 7fd445400640 1 -- 172.21.15.96:0/1990082401 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd4400725e0 msgr2=0x7fd4401f4520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:10.451 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.448+0000 7fd445400640 1 --2- 172.21.15.96:0/1990082401 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd4400725e0 0x7fd4401f4520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:10.451 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.448+0000 7fd446800640 1 --2- 172.21.15.96:0/1990082401 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd44015b6f0 0x7fd4400cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:10.451 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.448+0000 7fd445400640 1 -- 172.21.15.96:0/1990082401 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd44015b6f0 msgr2=0x7fd4400cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:10.451 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.448+0000 7fd445400640 1 --2- 172.21.15.96:0/1990082401 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd44015b6f0 0x7fd4400cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:10.451 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.448+0000 7fd445400640 1 -- 172.21.15.96:0/1990082401 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd440158c50 con 0x7fd4400d0640 2024-05-28T06:12:10.451 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.448+0000 7fd446800640 1 --2- 172.21.15.96:0/1990082401 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd44015b6f0 0x7fd4400cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:12:10.451 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.448+0000 7fd445400640 1 --2- 172.21.15.96:0/1990082401 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd4400d0640 0x7fd440157e30 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fd43000c7f0 tx=0x7fd43000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:10.451 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.448+0000 7fd43a000640 1 -- 172.21.15.96:0/1990082401 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd430015070 con 0x7fd4400d0640 2024-05-28T06:12:10.451 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.448+0000 7fd43a000640 1 -- 172.21.15.96:0/1990082401 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd430007820 con 0x7fd4400d0640 2024-05-28T06:12:10.451 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.448+0000 7fd43a000640 1 -- 172.21.15.96:0/1990082401 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd430007bd0 con 0x7fd4400d0640 2024-05-28T06:12:10.451 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.448+0000 7fd448800640 1 -- 172.21.15.96:0/1990082401 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd440158f30 con 0x7fd4400d0640 2024-05-28T06:12:10.451 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.449+0000 7fd448800640 1 -- 172.21.15.96:0/1990082401 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4401594f0 con 0x7fd4400d0640 2024-05-28T06:12:10.452 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.450+0000 7fd448800640 1 -- 172.21.15.96:0/1990082401 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4400689e0 con 0x7fd4400d0640 2024-05-28T06:12:10.452 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.451+0000 7fd43a000640 1 -- 172.21.15.96:0/1990082401 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 16) ==== 91754+0+0 (secure 0 0 0) 0x7fd430018070 con 0x7fd4400d0640 2024-05-28T06:12:10.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.452+0000 7fd43a000640 1 --2- 172.21.15.96:0/1990082401 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd42406d0d0 0x7fd42406f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:10.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.452+0000 7fd43a000640 1 -- 172.21.15.96:0/1990082401 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5851+0+0 (secure 0 0 0) 0x7fd43008d180 con 0x7fd4400d0640 2024-05-28T06:12:10.454 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.452+0000 7fd445e00640 1 --2- 172.21.15.96:0/1990082401 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd42406d0d0 0x7fd42406f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:10.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.453+0000 7fd445e00640 1 --2- 172.21.15.96:0/1990082401 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd42406d0d0 0x7fd42406f590 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fd434034000 tx=0x7fd43402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:10.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.456+0000 7fd43a000640 1 -- 172.21.15.96:0/1990082401 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd43005a360 con 0x7fd4400d0640 2024-05-28T06:12:10.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.656+0000 7fd448800640 1 -- 172.21.15.96:0/1990082401 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd4400afb40 con 0x7fd4400d0640 2024-05-28T06:12:10.659 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.657+0000 7fd43a000640 1 -- 172.21.15.96:0/1990082401 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v48) ==== 93+0+10 (secure 0 0 0) 0x7fd430059c60 con 0x7fd4400d0640 2024-05-28T06:12:10.659 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 1 2024-05-28T06:12:10.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.660+0000 7fd448800640 1 -- 172.21.15.96:0/1990082401 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd42406d0d0 msgr2=0x7fd42406f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:10.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.660+0000 7fd448800640 1 --2- 172.21.15.96:0/1990082401 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd42406d0d0 0x7fd42406f590 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fd434034000 tx=0x7fd43402e000 comp rx=0 tx=0).stop 2024-05-28T06:12:10.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.660+0000 7fd448800640 1 -- 172.21.15.96:0/1990082401 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd4400d0640 msgr2=0x7fd440157e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:10.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.660+0000 7fd448800640 1 --2- 172.21.15.96:0/1990082401 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd4400d0640 0x7fd440157e30 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fd43000c7f0 tx=0x7fd43000ccb0 comp rx=0 tx=0).stop 2024-05-28T06:12:10.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.660+0000 7fd448800640 1 -- 172.21.15.96:0/1990082401 shutdown_connections 2024-05-28T06:12:10.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.660+0000 7fd448800640 1 --2- 172.21.15.96:0/1990082401 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd42406d0d0 0x7fd42406f590 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:10.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.660+0000 7fd448800640 1 --2- 172.21.15.96:0/1990082401 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd44015b6f0 0x7fd4400cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:10.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.660+0000 7fd448800640 1 --2- 172.21.15.96:0/1990082401 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd4400d0640 0x7fd440157e30 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:10.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.660+0000 7fd448800640 1 --2- 172.21.15.96:0/1990082401 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd4400725e0 0x7fd4401f4520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:10.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.660+0000 7fd448800640 1 -- 172.21.15.96:0/1990082401 >> 172.21.15.96:0/1990082401 conn(0x7fd44006bd20 msgr2=0x7fd4400b09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:10.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.660+0000 7fd448800640 1 -- 172.21.15.96:0/1990082401 shutdown_connections 2024-05-28T06:12:10.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.660+0000 7fd448800640 1 -- 172.21.15.96:0/1990082401 wait complete. 2024-05-28T06:12:10.681 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-28T06:12:10.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.874+0000 7efeb8c00640 1 Processor -- start 2024-05-28T06:12:10.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.874+0000 7efeb8c00640 1 -- start start 2024-05-28T06:12:10.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.876+0000 7efeb8c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efeb40ad2a0 0x7efeb40ad730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:10.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.876+0000 7efeb8c00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efeb40b3d40 con 0x7efeb4161a90 2024-05-28T06:12:10.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.876+0000 7efeb8c00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efeb40b4020 con 0x7efeb4161e70 2024-05-28T06:12:10.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.876+0000 7efeb8c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7efeb40b4300 con 0x7efeb40ad2a0 2024-05-28T06:12:10.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.876+0000 7efeb1e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7efeb4161a90 0x7efeb406e470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:59234/0 (socket says 172.21.15.96:59234) 2024-05-28T06:12:10.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.876+0000 7efeb1e00640 1 -- 172.21.15.96:0/2869736512 learned_addr learned my addr 172.21.15.96:0/2869736512 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:12:10.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.877+0000 7efeb2800640 1 --2- 172.21.15.96:0/2869736512 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efeb40ad2a0 0x7efeb40ad730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:10.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.877+0000 7efeb1400640 1 -- 172.21.15.96:0/2869736512 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 296581670 0 0) 0x7efeb40b3d40 con 0x7efeb4161a90 2024-05-28T06:12:10.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.877+0000 7efeb1400640 1 -- 172.21.15.96:0/2869736512 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efe98003520 con 0x7efeb4161a90 2024-05-28T06:12:10.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.877+0000 7efeb1400640 1 -- 172.21.15.96:0/2869736512 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 185914271 0 0) 0x7efe98003520 con 0x7efeb4161a90 2024-05-28T06:12:10.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.877+0000 7efeb1400640 1 -- 172.21.15.96:0/2869736512 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efeb40ad2a0 msgr2=0x7efeb40ad730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:10.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.877+0000 7efeb1400640 1 --2- 172.21.15.96:0/2869736512 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efeb40ad2a0 0x7efeb40ad730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:10.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.877+0000 7efeb1400640 1 -- 172.21.15.96:0/2869736512 >> v1:172.21.15.123:6789/0 conn(0x7efeb4161e70 legacy=0x7efeb40b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:10.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.877+0000 7efeb1400640 1 -- 172.21.15.96:0/2869736512 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efeb406dc70 con 0x7efeb4161a90 2024-05-28T06:12:10.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.877+0000 7efeb1400640 1 -- 172.21.15.96:0/2869736512 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7efea4001670 con 0x7efeb4161a90 2024-05-28T06:12:10.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.877+0000 7efeb1400640 1 -- 172.21.15.96:0/2869736512 >> v1:172.21.15.96:6789/0 conn(0x7efeb4161a90 legacy=0x7efeb406e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:10.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.877+0000 7efeb1400640 1 --2- 172.21.15.96:0/2869736512 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efe980039e0 0x7efe98003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:10.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.877+0000 7efeb1400640 1 --2- 172.21.15.96:0/2869736512 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efeb40ad2a0 0x7efe98004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:10.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.877+0000 7efeb1400640 1 --2- 172.21.15.96:0/2869736512 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efe98004d40 0x7efe980051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:10.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.877+0000 7efeb1400640 1 -- 172.21.15.96:0/2869736512 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7efeb40b4300 con 0x7efe980039e0 2024-05-28T06:12:10.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.877+0000 7efeb1400640 1 -- 172.21.15.96:0/2869736512 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7efe9800baa0 con 0x7efe98004d40 2024-05-28T06:12:10.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.877+0000 7efeb1400640 1 -- 172.21.15.96:0/2869736512 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7efe9800bca0 con 0x7efeb40ad2a0 2024-05-28T06:12:10.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.878+0000 7efeb1e00640 1 --2- 172.21.15.96:0/2869736512 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efe980039e0 0x7efe98003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:10.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.878+0000 7efeb2800640 1 --2- 172.21.15.96:0/2869736512 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efe98004d40 0x7efe980051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:10.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.878+0000 7efeb1e00640 1 -- 172.21.15.96:0/2869736512 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efeb40ad2a0 msgr2=0x7efe98004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:10.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.878+0000 7efea9400640 1 --2- 172.21.15.96:0/2869736512 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efeb40ad2a0 0x7efe98004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:10.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.878+0000 7efeb1e00640 1 --2- 172.21.15.96:0/2869736512 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efeb40ad2a0 0x7efe98004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:10.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.878+0000 7efeb1e00640 1 -- 172.21.15.96:0/2869736512 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efe98004d40 msgr2=0x7efe980051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:10.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.878+0000 7efeb1e00640 1 --2- 172.21.15.96:0/2869736512 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efe98004d40 0x7efe980051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:10.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.878+0000 7efeb1e00640 1 -- 172.21.15.96:0/2869736512 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe98003520 con 0x7efe980039e0 2024-05-28T06:12:10.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.878+0000 7efea9400640 1 --2- 172.21.15.96:0/2869736512 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efeb40ad2a0 0x7efe98004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:12:10.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.878+0000 7efeb2800640 1 --2- 172.21.15.96:0/2869736512 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efe98004d40 0x7efe980051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:12:10.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.879+0000 7efeb1e00640 1 --2- 172.21.15.96:0/2869736512 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efe980039e0 0x7efe98003e00 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7efea4007fd0 tx=0x7efe98005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ee54afac3a88c16 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:10.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.879+0000 7efeb1400640 1 -- 172.21.15.96:0/2869736512 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efea4007c20 con 0x7efe980039e0 2024-05-28T06:12:10.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.879+0000 7efeb1400640 1 -- 172.21.15.96:0/2869736512 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efea4005af0 con 0x7efe980039e0 2024-05-28T06:12:10.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.879+0000 7efeb1400640 1 -- 172.21.15.96:0/2869736512 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efea4007070 con 0x7efe980039e0 2024-05-28T06:12:10.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.880+0000 7efeb8c00640 1 -- 172.21.15.96:0/2869736512 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efe980039e0 msgr2=0x7efe98003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:10.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.880+0000 7efeb8c00640 1 --2- 172.21.15.96:0/2869736512 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efe980039e0 0x7efe98003e00 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7efea4007fd0 tx=0x7efe98005830 comp rx=0 tx=0).stop 2024-05-28T06:12:10.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.880+0000 7efeb2800640 1 -- 172.21.15.96:0/2869736512 reap_dead start 2024-05-28T06:12:10.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.880+0000 7efeb8c00640 1 -- 172.21.15.96:0/2869736512 shutdown_connections 2024-05-28T06:12:10.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.880+0000 7efeb8c00640 1 -- 172.21.15.96:0/2869736512 >> 172.21.15.96:0/2869736512 conn(0x7efeb406bd20 msgr2=0x7efeb406c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:10.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.880+0000 7efeb8c00640 1 -- 172.21.15.96:0/2869736512 shutdown_connections 2024-05-28T06:12:10.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.880+0000 7efeb8c00640 1 -- 172.21.15.96:0/2869736512 wait complete. 2024-05-28T06:12:10.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.882+0000 7efeb8c00640 1 Processor -- start 2024-05-28T06:12:10.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.882+0000 7efeb8c00640 1 -- start start 2024-05-28T06:12:10.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.883+0000 7efeb8c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efe980039e0 0x7efeb40b0370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:10.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.883+0000 7efeb8c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efeb409b5a0 0x7efeb414e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:10.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.883+0000 7efeb8c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efeb40aec00 0x7efeb40b0990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:10.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.883+0000 7efeb8c00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7efeb406eb80 con 0x7efeb409b5a0 2024-05-28T06:12:10.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.883+0000 7efeb8c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7efeb40b5020 con 0x7efe980039e0 2024-05-28T06:12:10.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.883+0000 7efeb8c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7efeb40b39d0 con 0x7efeb40aec00 2024-05-28T06:12:10.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.883+0000 7efea9400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efeb409b5a0 0x7efeb414e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:10.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.883+0000 7efea9400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efeb409b5a0 0x7efeb414e8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:60886/0 (socket says 172.21.15.96:60886) 2024-05-28T06:12:10.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.883+0000 7efea9400640 1 -- 172.21.15.96:0/544172771 learned_addr learned my addr 172.21.15.96:0/544172771 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:10.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.883+0000 7efea9400640 1 -- 172.21.15.96:0/544172771 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efeb40aec00 msgr2=0x7efeb40b0990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:10.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.883+0000 7efea9400640 1 --2- 172.21.15.96:0/544172771 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efeb40aec00 0x7efeb40b0990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:10.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.883+0000 7efeb1e00640 1 --2- 172.21.15.96:0/544172771 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efe980039e0 0x7efeb40b0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:10.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.883+0000 7efea9400640 1 -- 172.21.15.96:0/544172771 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efe980039e0 msgr2=0x7efeb40b0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:10.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.883+0000 7efea9400640 1 --2- 172.21.15.96:0/544172771 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efe980039e0 0x7efeb40b0370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:10.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.884+0000 7efea9400640 1 -- 172.21.15.96:0/544172771 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efeb4150200 con 0x7efeb409b5a0 2024-05-28T06:12:10.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.884+0000 7efea9400640 1 --2- 172.21.15.96:0/544172771 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efeb409b5a0 0x7efeb414e8f0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7efe9c00d4c0 tx=0x7efe9c00d980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:10.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.884+0000 7efeaaa00640 1 -- 172.21.15.96:0/544172771 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe9c005480 con 0x7efeb409b5a0 2024-05-28T06:12:10.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.884+0000 7efeaaa00640 1 -- 172.21.15.96:0/544172771 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe9c017070 con 0x7efeb409b5a0 2024-05-28T06:12:10.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.884+0000 7efeaaa00640 1 -- 172.21.15.96:0/544172771 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe9c003850 con 0x7efeb409b5a0 2024-05-28T06:12:10.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.884+0000 7efeb8c00640 1 -- 172.21.15.96:0/544172771 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efeb40af3e0 con 0x7efeb409b5a0 2024-05-28T06:12:10.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.884+0000 7efeb8c00640 1 -- 172.21.15.96:0/544172771 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efeb41504c0 con 0x7efeb409b5a0 2024-05-28T06:12:10.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.885+0000 7efeb8c00640 1 -- 172.21.15.96:0/544172771 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efeb40689e0 con 0x7efeb409b5a0 2024-05-28T06:12:10.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.886+0000 7efeaaa00640 1 -- 172.21.15.96:0/544172771 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 16) ==== 91754+0+0 (secure 0 0 0) 0x7efe9c0039f0 con 0x7efeb409b5a0 2024-05-28T06:12:10.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.887+0000 7efeaaa00640 1 --2- 172.21.15.96:0/544172771 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7efe9006d0d0 0x7efe9006f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:10.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.887+0000 7efeaaa00640 1 -- 172.21.15.96:0/544172771 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5851+0+0 (secure 0 0 0) 0x7efe9c08a2e0 con 0x7efeb409b5a0 2024-05-28T06:12:10.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.888+0000 7efeb1e00640 1 --2- 172.21.15.96:0/544172771 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7efe9006d0d0 0x7efe9006f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:10.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.889+0000 7efeb1e00640 1 --2- 172.21.15.96:0/544172771 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7efe9006d0d0 0x7efe9006f590 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7efea4032000 tx=0x7efea4009030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:10.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:10.891+0000 7efeaaa00640 1 -- 172.21.15.96:0/544172771 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7efe9c05a000 con 0x7efeb409b5a0 2024-05-28T06:12:11.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.092+0000 7efeb8c00640 1 -- 172.21.15.96:0/544172771 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7efeb4072330 con 0x7efeb409b5a0 2024-05-28T06:12:11.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.092+0000 7efeaaa00640 1 -- 172.21.15.96:0/544172771 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v48) ==== 104+0+11 (secure 0 0 0) 0x7efe9c059900 con 0x7efeb409b5a0 2024-05-28T06:12:11.094 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 32 2024-05-28T06:12:11.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.095+0000 7efeb8c00640 1 -- 172.21.15.96:0/544172771 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7efe9006d0d0 msgr2=0x7efe9006f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:11.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.095+0000 7efeb8c00640 1 --2- 172.21.15.96:0/544172771 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7efe9006d0d0 0x7efe9006f590 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7efea4032000 tx=0x7efea4009030 comp rx=0 tx=0).stop 2024-05-28T06:12:11.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.095+0000 7efeb8c00640 1 -- 172.21.15.96:0/544172771 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efeb409b5a0 msgr2=0x7efeb414e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:11.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.095+0000 7efeb8c00640 1 --2- 172.21.15.96:0/544172771 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efeb409b5a0 0x7efeb414e8f0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7efe9c00d4c0 tx=0x7efe9c00d980 comp rx=0 tx=0).stop 2024-05-28T06:12:11.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.095+0000 7efeb8c00640 1 -- 172.21.15.96:0/544172771 shutdown_connections 2024-05-28T06:12:11.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.095+0000 7efeb8c00640 1 --2- 172.21.15.96:0/544172771 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7efe9006d0d0 0x7efe9006f590 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:11.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.095+0000 7efeb8c00640 1 --2- 172.21.15.96:0/544172771 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efeb40aec00 0x7efeb40b0990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:11.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.095+0000 7efeb8c00640 1 --2- 172.21.15.96:0/544172771 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efeb409b5a0 0x7efeb414e8f0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:11.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.095+0000 7efeb8c00640 1 --2- 172.21.15.96:0/544172771 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efe980039e0 0x7efeb40b0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:11.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.095+0000 7efeb8c00640 1 -- 172.21.15.96:0/544172771 >> 172.21.15.96:0/544172771 conn(0x7efeb406bd20 msgr2=0x7efeb41548c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:11.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.095+0000 7efeb8c00640 1 -- 172.21.15.96:0/544172771 shutdown_connections 2024-05-28T06:12:11.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.095+0000 7efeb8c00640 1 -- 172.21.15.96:0/544172771 wait complete. 2024-05-28T06:12:11.115 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-28T06:12:11.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.315+0000 7f4f58600640 1 Processor -- start 2024-05-28T06:12:11.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.316+0000 7f4f58600640 1 -- start start 2024-05-28T06:12:11.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.317+0000 7f4f58600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f4f5015e850 0x7f4f500b77b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:11.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.317+0000 7f4f58600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f4f500d1d50 0x7f4f500b90a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:11.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.317+0000 7f4f58600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f500b8000 con 0x7f4f500cef30 2024-05-28T06:12:11.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.317+0000 7f4f58600640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f4f5015ec30 con 0x7f4f5015e850 2024-05-28T06:12:11.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.317+0000 7f4f58600640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f4f500b82e0 con 0x7f4f500d1d50 2024-05-28T06:12:11.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.318+0000 7f4f55c00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f4f5015e850 0x7f4f500b77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:11.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.318+0000 7f4f55c00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f4f5015e850 0x7f4f500b77b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:60898/0 (socket says 172.21.15.96:60898) 2024-05-28T06:12:11.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.318+0000 7f4f55c00640 1 -- 172.21.15.96:0/3498579865 learned_addr learned my addr 172.21.15.96:0/3498579865 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:11.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.318+0000 7f4f56600640 1 --2- 172.21.15.96:0/3498579865 >> v2:172.21.15.123:3300/0 conn(0x7f4f500d1d50 0x7f4f500b90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:11.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.318+0000 7f4f55c00640 1 -- 172.21.15.96:0/3498579865 >> v2:172.21.15.123:3300/0 conn(0x7f4f500d1d50 msgr2=0x7f4f500b90a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:11.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.318+0000 7f4f55c00640 1 --2- 172.21.15.96:0/3498579865 >> v2:172.21.15.123:3300/0 conn(0x7f4f500d1d50 0x7f4f500b90a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:11.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.318+0000 7f4f55c00640 1 -- 172.21.15.96:0/3498579865 >> v1:172.21.15.123:6789/0 conn(0x7f4f500cef30 legacy=0x7f4f500d1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:11.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.318+0000 7f4f55c00640 1 -- 172.21.15.96:0/3498579865 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f500b85c0 con 0x7f4f5015e850 2024-05-28T06:12:11.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.319+0000 7f4f55c00640 1 --2- 172.21.15.96:0/3498579865 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f5015e850 0x7f4f500b77b0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f4f500cb280 tx=0x7f4f4402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c51bd3ab375f0b1a server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:11.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.319+0000 7f4f4fe00640 1 -- 172.21.15.96:0/3498579865 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f44037070 con 0x7f4f5015e850 2024-05-28T06:12:11.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.320+0000 7f4f4fe00640 1 -- 172.21.15.96:0/3498579865 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f44032440 con 0x7f4f5015e850 2024-05-28T06:12:11.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.320+0000 7f4f4fe00640 1 -- 172.21.15.96:0/3498579865 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f44032810 con 0x7f4f5015e850 2024-05-28T06:12:11.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.320+0000 7f4f58600640 1 -- 172.21.15.96:0/3498579865 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f5015e850 msgr2=0x7f4f500b77b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:11.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.320+0000 7f4f58600640 1 --2- 172.21.15.96:0/3498579865 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f5015e850 0x7f4f500b77b0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f4f500cb280 tx=0x7f4f4402a1a0 comp rx=0 tx=0).stop 2024-05-28T06:12:11.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.320+0000 7f4f58600640 1 -- 172.21.15.96:0/3498579865 shutdown_connections 2024-05-28T06:12:11.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.320+0000 7f4f58600640 1 --2- 172.21.15.96:0/3498579865 >> v2:172.21.15.123:3300/0 conn(0x7f4f500d1d50 0x7f4f500b90a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:11.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.320+0000 7f4f58600640 1 --2- 172.21.15.96:0/3498579865 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f5015e850 0x7f4f500b77b0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:11.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.320+0000 7f4f58600640 1 -- 172.21.15.96:0/3498579865 >> 172.21.15.96:0/3498579865 conn(0x7f4f50067aa0 msgr2=0x7f4f50067eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:11.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.321+0000 7f4f58600640 1 -- 172.21.15.96:0/3498579865 shutdown_connections 2024-05-28T06:12:11.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.321+0000 7f4f58600640 1 -- 172.21.15.96:0/3498579865 wait complete. 2024-05-28T06:12:11.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.322+0000 7f4f58600640 1 Processor -- start 2024-05-28T06:12:11.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.322+0000 7f4f58600640 1 -- start start 2024-05-28T06:12:11.325 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.323+0000 7f4f58600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4f500cef30 0x7f4f5015edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:11.325 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.323+0000 7f4f58600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f500d1d50 0x7f4f50150ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:11.325 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.323+0000 7f4f58600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4f5015e850 0x7f4f50162190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:11.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.323+0000 7f4f58600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4f500b95e0 con 0x7f4f500d1d50 2024-05-28T06:12:11.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.323+0000 7f4f58600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f4f500b7cf0 con 0x7f4f500cef30 2024-05-28T06:12:11.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.323+0000 7f4f58600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f4f500b9760 con 0x7f4f5015e850 2024-05-28T06:12:11.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.323+0000 7f4f55200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f500d1d50 0x7f4f50150ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:11.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.324+0000 7f4f55200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f500d1d50 0x7f4f50150ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:60914/0 (socket says 172.21.15.96:60914) 2024-05-28T06:12:11.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.324+0000 7f4f55200640 1 -- 172.21.15.96:0/3928055976 learned_addr learned my addr 172.21.15.96:0/3928055976 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:11.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.324+0000 7f4f55c00640 1 --2- 172.21.15.96:0/3928055976 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4f500cef30 0x7f4f5015edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:11.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.324+0000 7f4f56600640 1 --2- 172.21.15.96:0/3928055976 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4f5015e850 0x7f4f50162190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:11.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.324+0000 7f4f55200640 1 -- 172.21.15.96:0/3928055976 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4f5015e850 msgr2=0x7f4f50162190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:11.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.324+0000 7f4f55200640 1 --2- 172.21.15.96:0/3928055976 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4f5015e850 0x7f4f50162190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:11.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.324+0000 7f4f55200640 1 -- 172.21.15.96:0/3928055976 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4f500cef30 msgr2=0x7f4f5015edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:11.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.324+0000 7f4f55200640 1 --2- 172.21.15.96:0/3928055976 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4f500cef30 0x7f4f5015edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:11.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.324+0000 7f4f55200640 1 -- 172.21.15.96:0/3928055976 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f5015fe50 con 0x7f4f500d1d50 2024-05-28T06:12:11.327 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.324+0000 7f4f55200640 1 --2- 172.21.15.96:0/3928055976 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f500d1d50 0x7f4f50150ac0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f4f3c00c430 tx=0x7f4f3c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:11.327 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.325+0000 7f4f4d600640 1 -- 172.21.15.96:0/3928055976 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f3c011790 con 0x7f4f500d1d50 2024-05-28T06:12:11.328 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.325+0000 7f4f4d600640 1 -- 172.21.15.96:0/3928055976 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f3c017070 con 0x7f4f500d1d50 2024-05-28T06:12:11.328 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.325+0000 7f4f4d600640 1 -- 172.21.15.96:0/3928055976 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f3c011bb0 con 0x7f4f500d1d50 2024-05-28T06:12:11.328 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.325+0000 7f4f58600640 1 -- 172.21.15.96:0/3928055976 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f5015f300 con 0x7f4f500d1d50 2024-05-28T06:12:11.328 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.325+0000 7f4f58600640 1 -- 172.21.15.96:0/3928055976 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f5015f890 con 0x7f4f500d1d50 2024-05-28T06:12:11.328 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.326+0000 7f4f58600640 1 -- 172.21.15.96:0/3928055976 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f500689e0 con 0x7f4f500d1d50 2024-05-28T06:12:11.328 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.327+0000 7f4f4d600640 1 -- 172.21.15.96:0/3928055976 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 16) ==== 91754+0+0 (secure 0 0 0) 0x7f4f3c01c460 con 0x7f4f500d1d50 2024-05-28T06:12:11.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.328+0000 7f4f4d600640 1 --2- 172.21.15.96:0/3928055976 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4f3006d0d0 0x7f4f3006f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:11.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.328+0000 7f4f4d600640 1 -- 172.21.15.96:0/3928055976 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5851+0+0 (secure 0 0 0) 0x7f4f3c08cd40 con 0x7f4f500d1d50 2024-05-28T06:12:11.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.329+0000 7f4f55c00640 1 --2- 172.21.15.96:0/3928055976 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4f3006d0d0 0x7f4f3006f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:11.331 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.330+0000 7f4f55c00640 1 --2- 172.21.15.96:0/3928055976 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4f3006d0d0 0x7f4f3006f590 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f4f44034000 tx=0x7f4f4402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:11.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.332+0000 7f4f4d600640 1 -- 172.21.15.96:0/3928055976 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4f3c05a150 con 0x7f4f500d1d50 2024-05-28T06:12:11.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.529+0000 7f4f58600640 1 -- 172.21.15.96:0/3928055976 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f4f5006bb00 con 0x7f4f500d1d50 2024-05-28T06:12:11.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.530+0000 7f4f4d600640 1 -- 172.21.15.96:0/3928055976 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v48) ==== 100+0+11 (secure 0 0 0) 0x7f4f3c059a50 con 0x7f4f500d1d50 2024-05-28T06:12:11.532 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 64 2024-05-28T06:12:11.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.533+0000 7f4f58600640 1 -- 172.21.15.96:0/3928055976 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4f3006d0d0 msgr2=0x7f4f3006f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:11.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.533+0000 7f4f58600640 1 --2- 172.21.15.96:0/3928055976 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4f3006d0d0 0x7f4f3006f590 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f4f44034000 tx=0x7f4f4402e000 comp rx=0 tx=0).stop 2024-05-28T06:12:11.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.533+0000 7f4f58600640 1 -- 172.21.15.96:0/3928055976 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f500d1d50 msgr2=0x7f4f50150ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:11.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.533+0000 7f4f58600640 1 --2- 172.21.15.96:0/3928055976 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f500d1d50 0x7f4f50150ac0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f4f3c00c430 tx=0x7f4f3c00c8f0 comp rx=0 tx=0).stop 2024-05-28T06:12:11.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.533+0000 7f4f58600640 1 -- 172.21.15.96:0/3928055976 shutdown_connections 2024-05-28T06:12:11.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.533+0000 7f4f58600640 1 --2- 172.21.15.96:0/3928055976 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4f3006d0d0 0x7f4f3006f590 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:11.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.533+0000 7f4f58600640 1 --2- 172.21.15.96:0/3928055976 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4f5015e850 0x7f4f50162190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:11.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.533+0000 7f4f58600640 1 --2- 172.21.15.96:0/3928055976 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f500d1d50 0x7f4f50150ac0 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:11.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.533+0000 7f4f58600640 1 --2- 172.21.15.96:0/3928055976 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4f500cef30 0x7f4f5015edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:11.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.533+0000 7f4f58600640 1 -- 172.21.15.96:0/3928055976 >> 172.21.15.96:0/3928055976 conn(0x7f4f50067aa0 msgr2=0x7f4f5015dc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:11.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.533+0000 7f4f58600640 1 -- 172.21.15.96:0/3928055976 shutdown_connections 2024-05-28T06:12:11.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.533+0000 7f4f58600640 1 -- 172.21.15.96:0/3928055976 wait complete. 2024-05-28T06:12:11.554 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-28T06:12:11.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.752+0000 7f9f91c00640 1 Processor -- start 2024-05-28T06:12:11.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.752+0000 7f9f91c00640 1 -- start start 2024-05-28T06:12:11.755 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.754+0000 7f9f91c00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f9f8c072070 0x7f9f8c0b39d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:11.755 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.754+0000 7f9f91c00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f8c06e6e0 con 0x7f9f8c158b80 2024-05-28T06:12:11.755 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.754+0000 7f9f91c00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f8c06e9c0 con 0x7f9f8c071bc0 2024-05-28T06:12:11.755 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.754+0000 7f9f91c00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f9f8c06eca0 con 0x7f9f8c072070 2024-05-28T06:12:11.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.754+0000 7f9f91200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f9f8c072070 0x7f9f8c0b39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:11.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.754+0000 7f9f8be00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f9f8c158b80 0x7f9f8c06dc20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:59238/0 (socket says 172.21.15.96:59238) 2024-05-28T06:12:11.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.754+0000 7f9f8be00640 1 -- 172.21.15.96:0/1308797017 learned_addr learned my addr 172.21.15.96:0/1308797017 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:12:11.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.755+0000 7f9f91200640 1 -- 172.21.15.96:0/1308797017 >> v1:172.21.15.123:6789/0 conn(0x7f9f8c071bc0 legacy=0x7f9f8c0b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:11.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.755+0000 7f9f91200640 1 -- 172.21.15.96:0/1308797017 >> v1:172.21.15.96:6789/0 conn(0x7f9f8c158b80 legacy=0x7f9f8c06dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:11.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.755+0000 7f9f91200640 1 -- 172.21.15.96:0/1308797017 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f8c06ef80 con 0x7f9f8c072070 2024-05-28T06:12:11.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.755+0000 7f9f8aa00640 1 -- 172.21.15.96:0/1308797017 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3298942652 0 0) 0x7f9f8c06e6e0 con 0x7f9f8c158b80 2024-05-28T06:12:11.757 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.756+0000 7f9f91200640 1 --2- 172.21.15.96:0/1308797017 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f8c072070 0x7f9f8c0b39d0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f9f84009d90 tx=0x7f9f8402a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb2ad7af06c3b2c3 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:11.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.756+0000 7f9f8aa00640 1 -- 172.21.15.96:0/1308797017 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f84007740 con 0x7f9f8c072070 2024-05-28T06:12:11.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.756+0000 7f9f8aa00640 1 -- 172.21.15.96:0/1308797017 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f840078e0 con 0x7f9f8c072070 2024-05-28T06:12:11.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.756+0000 7f9f8aa00640 1 -- 172.21.15.96:0/1308797017 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f84007cb0 con 0x7f9f8c072070 2024-05-28T06:12:11.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.757+0000 7f9f91c00640 1 -- 172.21.15.96:0/1308797017 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f8c072070 msgr2=0x7f9f8c0b39d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:11.759 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.757+0000 7f9f91c00640 1 --2- 172.21.15.96:0/1308797017 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f8c072070 0x7f9f8c0b39d0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f9f84009d90 tx=0x7f9f8402a6f0 comp rx=0 tx=0).stop 2024-05-28T06:12:11.759 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.757+0000 7f9f91c00640 1 -- 172.21.15.96:0/1308797017 shutdown_connections 2024-05-28T06:12:11.759 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.757+0000 7f9f91c00640 1 --2- 172.21.15.96:0/1308797017 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f8c072070 0x7f9f8c0b39d0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:11.759 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.757+0000 7f9f91c00640 1 -- 172.21.15.96:0/1308797017 >> 172.21.15.96:0/1308797017 conn(0x7f9f8c0b1fc0 msgr2=0x7f9f8c0b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:11.759 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.757+0000 7f9f91c00640 1 -- 172.21.15.96:0/1308797017 shutdown_connections 2024-05-28T06:12:11.759 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.757+0000 7f9f91c00640 1 -- 172.21.15.96:0/1308797017 wait complete. 2024-05-28T06:12:11.760 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.759+0000 7f9f91c00640 1 Processor -- start 2024-05-28T06:12:11.760 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.759+0000 7f9f91c00640 1 -- start start 2024-05-28T06:12:11.761 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.760+0000 7f9f91c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9f8c071bc0 0x7f9f8c0b0d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:11.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.760+0000 7f9f91c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f8c072070 0x7f9f8c09d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:11.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.760+0000 7f9f91c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f8c158b80 0x7f9f8c1470a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:11.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.760+0000 7f9f91c00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f9f8c072450 con 0x7f9f8c072070 2024-05-28T06:12:11.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.760+0000 7f9f91c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f9f8c0b2d30 con 0x7f9f8c158b80 2024-05-28T06:12:11.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.760+0000 7f9f91c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f9f8c06f780 con 0x7f9f8c071bc0 2024-05-28T06:12:11.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.760+0000 7f9f8b400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f8c072070 0x7f9f8c09d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:11.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.760+0000 7f9f8b400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f8c072070 0x7f9f8c09d530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:60938/0 (socket says 172.21.15.96:60938) 2024-05-28T06:12:11.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.760+0000 7f9f8b400640 1 -- 172.21.15.96:0/348171865 learned_addr learned my addr 172.21.15.96:0/348171865 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:11.764 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.761+0000 7f9f91200640 1 --2- 172.21.15.96:0/348171865 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f8c158b80 0x7f9f8c1470a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:11.764 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.761+0000 7f9f8b400640 1 -- 172.21.15.96:0/348171865 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9f8c071bc0 msgr2=0x7f9f8c0b0d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:11.764 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.761+0000 7f9f8be00640 1 --2- 172.21.15.96:0/348171865 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9f8c071bc0 0x7f9f8c0b0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:11.764 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.761+0000 7f9f8b400640 1 --2- 172.21.15.96:0/348171865 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9f8c071bc0 0x7f9f8c0b0d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:11.764 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.761+0000 7f9f8b400640 1 -- 172.21.15.96:0/348171865 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f8c158b80 msgr2=0x7f9f8c1470a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:11.764 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.761+0000 7f9f8b400640 1 --2- 172.21.15.96:0/348171865 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f8c158b80 0x7f9f8c1470a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:11.764 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.761+0000 7f9f8b400640 1 -- 172.21.15.96:0/348171865 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f8c09b970 con 0x7f9f8c072070 2024-05-28T06:12:11.764 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.761+0000 7f9f91200640 1 --2- 172.21.15.96:0/348171865 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f8c158b80 0x7f9f8c1470a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:12:11.764 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.761+0000 7f9f8b400640 1 --2- 172.21.15.96:0/348171865 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f8c072070 0x7f9f8c09d530 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f9f7400c400 tx=0x7f9f7400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:11.764 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.761+0000 7f9f83e00640 1 -- 172.21.15.96:0/348171865 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f74017070 con 0x7f9f8c072070 2024-05-28T06:12:11.764 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.761+0000 7f9f83e00640 1 -- 172.21.15.96:0/348171865 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f740052d0 con 0x7f9f8c072070 2024-05-28T06:12:11.764 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.761+0000 7f9f83e00640 1 -- 172.21.15.96:0/348171865 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f74003830 con 0x7f9f8c072070 2024-05-28T06:12:11.764 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.761+0000 7f9f91c00640 1 -- 172.21.15.96:0/348171865 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f8c09bc50 con 0x7f9f8c072070 2024-05-28T06:12:11.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.761+0000 7f9f91c00640 1 -- 172.21.15.96:0/348171865 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f8c09c210 con 0x7f9f8c072070 2024-05-28T06:12:11.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.763+0000 7f9f91c00640 1 -- 172.21.15.96:0/348171865 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f8c0689e0 con 0x7f9f8c072070 2024-05-28T06:12:11.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.764+0000 7f9f83e00640 1 -- 172.21.15.96:0/348171865 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 16) ==== 91754+0+0 (secure 0 0 0) 0x7f9f7401a070 con 0x7f9f8c072070 2024-05-28T06:12:11.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.765+0000 7f9f83e00640 1 --2- 172.21.15.96:0/348171865 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9f6806d100 0x7f9f6806f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:11.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.765+0000 7f9f83e00640 1 -- 172.21.15.96:0/348171865 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5851+0+0 (secure 0 0 0) 0x7f9f74089860 con 0x7f9f8c072070 2024-05-28T06:12:11.767 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.766+0000 7f9f8be00640 1 --2- 172.21.15.96:0/348171865 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9f6806d100 0x7f9f6806f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:11.768 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.767+0000 7f9f8be00640 1 --2- 172.21.15.96:0/348171865 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9f6806d100 0x7f9f6806f5c0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f9f7c007fd0 tx=0x7f9f7c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:11.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:11.770+0000 7f9f83e00640 1 -- 172.21.15.96:0/348171865 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9f74059580 con 0x7f9f8c072070 2024-05-28T06:12:12.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.006+0000 7f9f91c00640 1 -- 172.21.15.96:0/348171865 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9f8c157ec0 con 0x7f9f8c072070 2024-05-28T06:12:12.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.007+0000 7f9f83e00640 1 -- 172.21.15.96:0/348171865 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+5181 (secure 0 0 0) 0x7f9f74058e80 con 0x7f9f8c072070 2024-05-28T06:12:12.009 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:12:12.009 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":26,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/1693172076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":1693172076},{"type":"v1","addr":"172.21.15.123:6835","nonce":1693172076}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":4805,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/1098611268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":1098611268},{"type":"v1","addr":"172.21.15.123:6837","nonce":1098611268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5856,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6834/673165997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":673165997},{"type":"v1","addr":"172.21.15.96:6834","nonce":673165997}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-28T06:08:54.477674+0000","modified":"2024-05-28T06:08:57.572929+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5859},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5859":{"gid":5859,"name":"a","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.96:6835/935552515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6833","nonce":935552515},{"type":"v1","addr":"172.21.15.96:6835","nonce":935552515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5859,"qdb_cluster":[5859]},"id":3}]} 2024-05-28T06:12:12.009 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 26 2024-05-28T06:12:12.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.009+0000 7f9f91c00640 1 -- 172.21.15.96:0/348171865 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9f6806d100 msgr2=0x7f9f6806f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:12.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.009+0000 7f9f91c00640 1 --2- 172.21.15.96:0/348171865 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9f6806d100 0x7f9f6806f5c0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f9f7c007fd0 tx=0x7f9f7c009040 comp rx=0 tx=0).stop 2024-05-28T06:12:12.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.010+0000 7f9f91c00640 1 -- 172.21.15.96:0/348171865 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f8c072070 msgr2=0x7f9f8c09d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:12.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.010+0000 7f9f91c00640 1 --2- 172.21.15.96:0/348171865 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f8c072070 0x7f9f8c09d530 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f9f7400c400 tx=0x7f9f7400c8c0 comp rx=0 tx=0).stop 2024-05-28T06:12:12.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.010+0000 7f9f91c00640 1 -- 172.21.15.96:0/348171865 shutdown_connections 2024-05-28T06:12:12.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.010+0000 7f9f91c00640 1 --2- 172.21.15.96:0/348171865 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9f6806d100 0x7f9f6806f5c0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:12.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.010+0000 7f9f91c00640 1 --2- 172.21.15.96:0/348171865 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f8c158b80 0x7f9f8c1470a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:12.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.010+0000 7f9f91c00640 1 --2- 172.21.15.96:0/348171865 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f8c072070 0x7f9f8c09d530 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:12.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.010+0000 7f9f91c00640 1 --2- 172.21.15.96:0/348171865 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9f8c071bc0 0x7f9f8c0b0d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:12.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.010+0000 7f9f91c00640 1 -- 172.21.15.96:0/348171865 >> 172.21.15.96:0/348171865 conn(0x7f9f8c0b1fc0 msgr2=0x7f9f8c0d0310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:12.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.010+0000 7f9f91c00640 1 -- 172.21.15.96:0/348171865 shutdown_connections 2024-05-28T06:12:12.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.010+0000 7f9f91c00640 1 -- 172.21.15.96:0/348171865 wait complete. 2024-05-28T06:12:12.031 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:12:12.231 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.229+0000 7fdffee00640 1 Processor -- start 2024-05-28T06:12:12.231 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.229+0000 7fdffee00640 1 -- start start 2024-05-28T06:12:12.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.231+0000 7fdffee00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdff80d04a0 0x7fdff80d0900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:12.233 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.231+0000 7fdffee00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fdff80ad130 0x7fdff80ad5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:12.233 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.231+0000 7fdffee00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdff806ed10 con 0x7fdff806f860 2024-05-28T06:12:12.233 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.231+0000 7fdffee00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fdff80d0e40 con 0x7fdff80ad130 2024-05-28T06:12:12.233 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.231+0000 7fdffee00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fdff815cb00 con 0x7fdff80d04a0 2024-05-28T06:12:12.233 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.231+0000 7fdff7e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fdff806f860 0x7fdff80b3150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:59248/0 (socket says 172.21.15.96:59248) 2024-05-28T06:12:12.233 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.231+0000 7fdff7e00640 1 -- 172.21.15.96:0/1122415213 learned_addr learned my addr 172.21.15.96:0/1122415213 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:12:12.233 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.231+0000 7fdffce00640 1 --2- 172.21.15.96:0/1122415213 >> v2:172.21.15.96:3300/0 conn(0x7fdff80ad130 0x7fdff80ad5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:12.233 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.231+0000 7fdff7400640 1 --2- 172.21.15.96:0/1122415213 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdff80d04a0 0x7fdff80d0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:12.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.232+0000 7fdff6a00640 1 -- 172.21.15.96:0/1122415213 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3254560097 0 0) 0x7fdfe8001110 con 0x7fdff806f860 2024-05-28T06:12:12.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.232+0000 7fdffce00640 1 -- 172.21.15.96:0/1122415213 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdff80d04a0 msgr2=0x7fdff80d0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:12.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.232+0000 7fdffce00640 1 --2- 172.21.15.96:0/1122415213 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdff80d04a0 0x7fdff80d0900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:12.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.232+0000 7fdffce00640 1 -- 172.21.15.96:0/1122415213 >> v1:172.21.15.96:6789/0 conn(0x7fdff806f860 legacy=0x7fdff80b3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:12.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.232+0000 7fdffce00640 1 -- 172.21.15.96:0/1122415213 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdff80b3cc0 con 0x7fdff80ad130 2024-05-28T06:12:12.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.233+0000 7fdffce00640 1 --2- 172.21.15.96:0/1122415213 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdff80ad130 0x7fdff80ad5a0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fdff0009d90 tx=0x7fdff002a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16868f285e2773f7 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:12.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.233+0000 7fdff6a00640 1 -- 172.21.15.96:0/1122415213 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdff0007740 con 0x7fdff80ad130 2024-05-28T06:12:12.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.233+0000 7fdff6a00640 1 -- 172.21.15.96:0/1122415213 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdff00078e0 con 0x7fdff80ad130 2024-05-28T06:12:12.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.233+0000 7fdff6a00640 1 -- 172.21.15.96:0/1122415213 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdff0007cb0 con 0x7fdff80ad130 2024-05-28T06:12:12.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.233+0000 7fdffee00640 1 -- 172.21.15.96:0/1122415213 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdff80ad130 msgr2=0x7fdff80ad5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:12.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.233+0000 7fdffee00640 1 --2- 172.21.15.96:0/1122415213 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdff80ad130 0x7fdff80ad5a0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fdff0009d90 tx=0x7fdff002a660 comp rx=0 tx=0).stop 2024-05-28T06:12:12.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.234+0000 7fdffee00640 1 -- 172.21.15.96:0/1122415213 shutdown_connections 2024-05-28T06:12:12.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.234+0000 7fdffee00640 1 --2- 172.21.15.96:0/1122415213 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdff80ad130 0x7fdff80ad5a0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:12.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.234+0000 7fdffee00640 1 --2- 172.21.15.96:0/1122415213 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdff80d04a0 0x7fdff80d0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:12.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.234+0000 7fdffee00640 1 -- 172.21.15.96:0/1122415213 >> 172.21.15.96:0/1122415213 conn(0x7fdff806bae0 msgr2=0x7fdff806bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:12.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.234+0000 7fdffee00640 1 -- 172.21.15.96:0/1122415213 shutdown_connections 2024-05-28T06:12:12.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.234+0000 7fdffee00640 1 -- 172.21.15.96:0/1122415213 wait complete. 2024-05-28T06:12:12.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.235+0000 7fdffee00640 1 Processor -- start 2024-05-28T06:12:12.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.236+0000 7fdffee00640 1 -- start start 2024-05-28T06:12:12.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.236+0000 7fdffee00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdff806f860 0x7fdff8155370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:12.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.236+0000 7fdffee00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdff80ad130 0x7fdff815a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:12.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.236+0000 7fdffee00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdff80d04a0 0x7fdff815ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:12.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.236+0000 7fdffee00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fdff80b4560 con 0x7fdff80ad130 2024-05-28T06:12:12.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.236+0000 7fdffee00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fdff80b4280 con 0x7fdff806f860 2024-05-28T06:12:12.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.236+0000 7fdffee00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fdff80b4840 con 0x7fdff80d04a0 2024-05-28T06:12:12.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.237+0000 7fdff7400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdff80ad130 0x7fdff815a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:12.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.237+0000 7fdff7400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdff80ad130 0x7fdff815a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:60958/0 (socket says 172.21.15.96:60958) 2024-05-28T06:12:12.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.237+0000 7fdff7400640 1 -- 172.21.15.96:0/3983633595 learned_addr learned my addr 172.21.15.96:0/3983633595 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:12.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.237+0000 7fdff7e00640 1 --2- 172.21.15.96:0/3983633595 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdff806f860 0x7fdff8155370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:12.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.237+0000 7fdff7400640 1 -- 172.21.15.96:0/3983633595 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdff80d04a0 msgr2=0x7fdff815ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:12.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.237+0000 7fdffce00640 1 --2- 172.21.15.96:0/3983633595 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdff80d04a0 0x7fdff815ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:12.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.237+0000 7fdff7400640 1 --2- 172.21.15.96:0/3983633595 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdff80d04a0 0x7fdff815ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:12.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.237+0000 7fdff7400640 1 -- 172.21.15.96:0/3983633595 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdff806f860 msgr2=0x7fdff8155370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:12.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.237+0000 7fdff7400640 1 --2- 172.21.15.96:0/3983633595 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdff806f860 0x7fdff8155370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:12.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.237+0000 7fdff7400640 1 -- 172.21.15.96:0/3983633595 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdff81558b0 con 0x7fdff80ad130 2024-05-28T06:12:12.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.237+0000 7fdff7400640 1 --2- 172.21.15.96:0/3983633595 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdff80ad130 0x7fdff815a6c0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fdfe400d800 tx=0x7fdfe400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:12.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.238+0000 7fdfefe00640 1 -- 172.21.15.96:0/3983633595 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfe400b6c0 con 0x7fdff80ad130 2024-05-28T06:12:12.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.238+0000 7fdfefe00640 1 -- 172.21.15.96:0/3983633595 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdfe4019070 con 0x7fdff80ad130 2024-05-28T06:12:12.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.238+0000 7fdfefe00640 1 -- 172.21.15.96:0/3983633595 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfe400ba80 con 0x7fdff80ad130 2024-05-28T06:12:12.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.238+0000 7fdffee00640 1 -- 172.21.15.96:0/3983633595 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdff815ee80 con 0x7fdff80ad130 2024-05-28T06:12:12.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.238+0000 7fdffee00640 1 -- 172.21.15.96:0/3983633595 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdff80c3ef0 con 0x7fdff80ad130 2024-05-28T06:12:12.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.239+0000 7fdffee00640 1 -- 172.21.15.96:0/3983633595 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdff80689e0 con 0x7fdff80ad130 2024-05-28T06:12:12.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.240+0000 7fdfefe00640 1 -- 172.21.15.96:0/3983633595 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 16) ==== 91754+0+0 (secure 0 0 0) 0x7fdfe400bc20 con 0x7fdff80ad130 2024-05-28T06:12:12.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.241+0000 7fdfefe00640 1 --2- 172.21.15.96:0/3983633595 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdfd806d0d0 0x7fdfd806f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:12.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.241+0000 7fdfefe00640 1 -- 172.21.15.96:0/3983633595 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5851+0+0 (secure 0 0 0) 0x7fdfe408d3d0 con 0x7fdff80ad130 2024-05-28T06:12:12.243 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.242+0000 7fdff7e00640 1 --2- 172.21.15.96:0/3983633595 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdfd806d0d0 0x7fdfd806f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:12.244 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.243+0000 7fdff7e00640 1 --2- 172.21.15.96:0/3983633595 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdfd806d0d0 0x7fdfd806f590 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fdfe8000f60 tx=0x7fdfe8007c00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:12.247 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.245+0000 7fdfefe00640 1 -- 172.21.15.96:0/3983633595 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fdff81558b0 con 0x7fdff80ad130 2024-05-28T06:12:12.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.445+0000 7fdffee00640 1 -- 172.21.15.96:0/3983633595 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdff806e2f0 con 0x7fdff80ad130 2024-05-28T06:12:12.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.447+0000 7fdfefe00640 1 -- 172.21.15.96:0/3983633595 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v48) ==== 74+0+16868 (secure 0 0 0) 0x7fdfe4089200 con 0x7fdff80ad130 2024-05-28T06:12:12.449 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:12:12.450 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":48,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:12:04.592827+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-05-28T06:08:52.257477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-05-28T06:08:53.459376+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"48","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":48,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.9","mappings":[{"from":0,"to":5}]},{"pgid":"7.37","mappings":[{"from":0,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":7,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[{"pool":7,"snaps":[{"begin":2,"length":2}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:12:12.452 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.450+0000 7fdffee00640 1 -- 172.21.15.96:0/3983633595 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdfd806d0d0 msgr2=0x7fdfd806f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:12.452 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.450+0000 7fdffee00640 1 --2- 172.21.15.96:0/3983633595 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdfd806d0d0 0x7fdfd806f590 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fdfe8000f60 tx=0x7fdfe8007c00 comp rx=0 tx=0).stop 2024-05-28T06:12:12.452 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.450+0000 7fdffee00640 1 -- 172.21.15.96:0/3983633595 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdff80ad130 msgr2=0x7fdff815a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:12.452 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.450+0000 7fdffee00640 1 --2- 172.21.15.96:0/3983633595 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdff80ad130 0x7fdff815a6c0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fdfe400d800 tx=0x7fdfe400dcc0 comp rx=0 tx=0).stop 2024-05-28T06:12:12.452 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.450+0000 7fdffee00640 1 -- 172.21.15.96:0/3983633595 shutdown_connections 2024-05-28T06:12:12.452 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.450+0000 7fdffee00640 1 --2- 172.21.15.96:0/3983633595 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdfd806d0d0 0x7fdfd806f590 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:12.452 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.450+0000 7fdffee00640 1 --2- 172.21.15.96:0/3983633595 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdff80d04a0 0x7fdff815ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:12.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.450+0000 7fdffee00640 1 --2- 172.21.15.96:0/3983633595 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdff80ad130 0x7fdff815a6c0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:12.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.450+0000 7fdffee00640 1 --2- 172.21.15.96:0/3983633595 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdff806f860 0x7fdff8155370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:12.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.450+0000 7fdffee00640 1 -- 172.21.15.96:0/3983633595 >> 172.21.15.96:0/3983633595 conn(0x7fdff806bae0 msgr2=0x7fdff80729c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:12.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.451+0000 7fdffee00640 1 -- 172.21.15.96:0/3983633595 shutdown_connections 2024-05-28T06:12:12.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.451+0000 7fdffee00640 1 -- 172.21.15.96:0/3983633595 wait complete. 2024-05-28T06:12:12.473 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-05-28T06:12:12.473 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-28T06:12:12.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.676+0000 7fc58d400640 1 Processor -- start 2024-05-28T06:12:12.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.676+0000 7fc58d400640 1 -- start start 2024-05-28T06:12:12.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.677+0000 7fc58d400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fc5880d0590 0x7fc5880d0a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:12.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.677+0000 7fc58d400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fc588099090 0x7fc5880994a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:12.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.677+0000 7fc58d400640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5880b51f0 con 0x7fc5880b4e10 2024-05-28T06:12:12.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.677+0000 7fc58d400640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fc5880aef60 con 0x7fc5880d0590 2024-05-28T06:12:12.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.677+0000 7fc58d400640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fc5880ad1d0 con 0x7fc588099090 2024-05-28T06:12:12.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.677+0000 7fc585e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fc5880d0590 0x7fc5880d0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:12.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.678+0000 7fc585e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fc5880d0590 0x7fc5880d0a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:60964/0 (socket says 172.21.15.96:60964) 2024-05-28T06:12:12.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.678+0000 7fc585e00640 1 -- 172.21.15.96:0/3066799397 learned_addr learned my addr 172.21.15.96:0/3066799397 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:12.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.678+0000 7fc585e00640 1 -- 172.21.15.96:0/3066799397 >> v2:172.21.15.123:3300/0 conn(0x7fc588099090 msgr2=0x7fc5880994a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:12.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.678+0000 7fc587200640 1 --2- 172.21.15.96:0/3066799397 >> v2:172.21.15.123:3300/0 conn(0x7fc588099090 0x7fc5880994a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:12.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.678+0000 7fc585e00640 1 --2- 172.21.15.96:0/3066799397 >> v2:172.21.15.123:3300/0 conn(0x7fc588099090 0x7fc5880994a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:12.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.678+0000 7fc585e00640 1 -- 172.21.15.96:0/3066799397 >> v1:172.21.15.123:6789/0 conn(0x7fc5880b4e10 legacy=0x7fc5880cde30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:12.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.678+0000 7fc585e00640 1 -- 172.21.15.96:0/3066799397 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5880ad470 con 0x7fc5880d0590 2024-05-28T06:12:12.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.679+0000 7fc585e00640 1 --2- 172.21.15.96:0/3066799397 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc5880d0590 0x7fc5880d0a20 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fc5880afd70 tx=0x7fc57402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9be5fca7c0f0b57 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:12.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.679+0000 7fc585400640 1 -- 172.21.15.96:0/3066799397 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc574037070 con 0x7fc5880d0590 2024-05-28T06:12:12.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.679+0000 7fc585400640 1 -- 172.21.15.96:0/3066799397 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc574032440 con 0x7fc5880d0590 2024-05-28T06:12:12.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.679+0000 7fc585400640 1 -- 172.21.15.96:0/3066799397 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc574032810 con 0x7fc5880d0590 2024-05-28T06:12:12.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.680+0000 7fc58d400640 1 -- 172.21.15.96:0/3066799397 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc5880d0590 msgr2=0x7fc5880d0a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:12.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.680+0000 7fc58d400640 1 --2- 172.21.15.96:0/3066799397 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc5880d0590 0x7fc5880d0a20 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fc5880afd70 tx=0x7fc57402a1a0 comp rx=0 tx=0).stop 2024-05-28T06:12:12.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.680+0000 7fc58d400640 1 -- 172.21.15.96:0/3066799397 shutdown_connections 2024-05-28T06:12:12.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.680+0000 7fc58d400640 1 --2- 172.21.15.96:0/3066799397 >> v2:172.21.15.123:3300/0 conn(0x7fc588099090 0x7fc5880994a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:12.683 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.680+0000 7fc58d400640 1 --2- 172.21.15.96:0/3066799397 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc5880d0590 0x7fc5880d0a20 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:12.683 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.680+0000 7fc58d400640 1 -- 172.21.15.96:0/3066799397 >> 172.21.15.96:0/3066799397 conn(0x7fc58806bae0 msgr2=0x7fc58806bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:12.683 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.680+0000 7fc58d400640 1 -- 172.21.15.96:0/3066799397 shutdown_connections 2024-05-28T06:12:12.683 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.680+0000 7fc58d400640 1 -- 172.21.15.96:0/3066799397 wait complete. 2024-05-28T06:12:12.683 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.682+0000 7fc58d400640 1 Processor -- start 2024-05-28T06:12:12.683 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.682+0000 7fc58d400640 1 -- start start 2024-05-28T06:12:12.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.683+0000 7fc58d400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc588099090 0x7fc588155830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:12.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.683+0000 7fc58d400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc5880b4e10 0x7fc5880b1040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:12.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.683+0000 7fc58d400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5880d0590 0x7fc5880ca8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:12.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.683+0000 7fc58d400640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc5880ce880 con 0x7fc588099090 2024-05-28T06:12:12.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.683+0000 7fc58d400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc5880ce540 con 0x7fc5880d0590 2024-05-28T06:12:12.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.683+0000 7fc58d400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fc5880cea00 con 0x7fc5880b4e10 2024-05-28T06:12:12.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.683+0000 7fc586800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc588099090 0x7fc588155830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:12.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.683+0000 7fc586800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc588099090 0x7fc588155830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:60970/0 (socket says 172.21.15.96:60970) 2024-05-28T06:12:12.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.683+0000 7fc586800640 1 -- 172.21.15.96:0/404760649 learned_addr learned my addr 172.21.15.96:0/404760649 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:12.687 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.684+0000 7fc585e00640 1 --2- 172.21.15.96:0/404760649 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc5880b4e10 0x7fc5880b1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:12.687 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.684+0000 7fc587200640 1 --2- 172.21.15.96:0/404760649 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5880d0590 0x7fc5880ca8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:12.687 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.684+0000 7fc586800640 1 -- 172.21.15.96:0/404760649 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc5880b4e10 msgr2=0x7fc5880b1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:12.687 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.684+0000 7fc586800640 1 --2- 172.21.15.96:0/404760649 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc5880b4e10 0x7fc5880b1040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:12.687 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.684+0000 7fc586800640 1 -- 172.21.15.96:0/404760649 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5880d0590 msgr2=0x7fc5880ca8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:12.687 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.684+0000 7fc586800640 1 --2- 172.21.15.96:0/404760649 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5880d0590 0x7fc5880ca8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:12.687 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.684+0000 7fc586800640 1 -- 172.21.15.96:0/404760649 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5880c9820 con 0x7fc588099090 2024-05-28T06:12:12.687 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.684+0000 7fc585e00640 1 --2- 172.21.15.96:0/404760649 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc5880b4e10 0x7fc5880b1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:12:12.687 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.684+0000 7fc586800640 1 --2- 172.21.15.96:0/404760649 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc588099090 0x7fc588155830 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fc57800c440 tx=0x7fc57800c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:12.687 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.684+0000 7fc57ea00640 1 -- 172.21.15.96:0/404760649 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc578017070 con 0x7fc588099090 2024-05-28T06:12:12.687 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.684+0000 7fc57ea00640 1 -- 172.21.15.96:0/404760649 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5780035f0 con 0x7fc588099090 2024-05-28T06:12:12.687 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.684+0000 7fc57ea00640 1 -- 172.21.15.96:0/404760649 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5780039e0 con 0x7fc588099090 2024-05-28T06:12:12.688 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.684+0000 7fc58d400640 1 -- 172.21.15.96:0/404760649 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5880c9b00 con 0x7fc588099090 2024-05-28T06:12:12.688 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.685+0000 7fc58d400640 1 -- 172.21.15.96:0/404760649 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc588155d70 con 0x7fc588099090 2024-05-28T06:12:12.688 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.686+0000 7fc58d400640 1 -- 172.21.15.96:0/404760649 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5880689e0 con 0x7fc588099090 2024-05-28T06:12:12.688 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.687+0000 7fc57ea00640 1 -- 172.21.15.96:0/404760649 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 16) ==== 91754+0+0 (secure 0 0 0) 0x7fc57801a070 con 0x7fc588099090 2024-05-28T06:12:12.689 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.688+0000 7fc57ea00640 1 --2- 172.21.15.96:0/404760649 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc56806d0d0 0x7fc56806f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:12.689 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.688+0000 7fc57ea00640 1 -- 172.21.15.96:0/404760649 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5851+0+0 (secure 0 0 0) 0x7fc578089950 con 0x7fc588099090 2024-05-28T06:12:12.691 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.689+0000 7fc585e00640 1 --2- 172.21.15.96:0/404760649 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc56806d0d0 0x7fc56806f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:12.691 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.690+0000 7fc585e00640 1 --2- 172.21.15.96:0/404760649 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc56806d0d0 0x7fc56806f590 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fc574034000 tx=0x7fc57402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:12.694 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.692+0000 7fc57ea00640 1 -- 172.21.15.96:0/404760649 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc578059670 con 0x7fc588099090 2024-05-28T06:12:12.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.932+0000 7fc58d400640 1 -- 172.21.15.96:0/404760649 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc5880cf3e0 con 0x7fc588099090 2024-05-28T06:12:12.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.933+0000 7fc57ea00640 1 -- 172.21.15.96:0/404760649 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+5181 (secure 0 0 0) 0x7fc5780010c0 con 0x7fc588099090 2024-05-28T06:12:12.935 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:12:12.935 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":26,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/1693172076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":1693172076},{"type":"v1","addr":"172.21.15.123:6835","nonce":1693172076}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":4805,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/1098611268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":1098611268},{"type":"v1","addr":"172.21.15.123:6837","nonce":1098611268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5856,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6834/673165997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":673165997},{"type":"v1","addr":"172.21.15.96:6834","nonce":673165997}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-28T06:08:54.477674+0000","modified":"2024-05-28T06:08:57.572929+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5859},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5859":{"gid":5859,"name":"a","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.96:6835/935552515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6833","nonce":935552515},{"type":"v1","addr":"172.21.15.96:6835","nonce":935552515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5859,"qdb_cluster":[5859]},"id":3}]} 2024-05-28T06:12:12.936 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 26 2024-05-28T06:12:12.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.936+0000 7fc58d400640 1 -- 172.21.15.96:0/404760649 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc56806d0d0 msgr2=0x7fc56806f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:12.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.936+0000 7fc58d400640 1 --2- 172.21.15.96:0/404760649 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc56806d0d0 0x7fc56806f590 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fc574034000 tx=0x7fc57402e000 comp rx=0 tx=0).stop 2024-05-28T06:12:12.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.936+0000 7fc58d400640 1 -- 172.21.15.96:0/404760649 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc588099090 msgr2=0x7fc588155830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:12.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.936+0000 7fc58d400640 1 --2- 172.21.15.96:0/404760649 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc588099090 0x7fc588155830 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fc57800c440 tx=0x7fc57800c900 comp rx=0 tx=0).stop 2024-05-28T06:12:12.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.936+0000 7fc58d400640 1 -- 172.21.15.96:0/404760649 shutdown_connections 2024-05-28T06:12:12.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.936+0000 7fc58d400640 1 --2- 172.21.15.96:0/404760649 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc56806d0d0 0x7fc56806f590 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:12.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.936+0000 7fc58d400640 1 --2- 172.21.15.96:0/404760649 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5880d0590 0x7fc5880ca8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:12.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.936+0000 7fc58d400640 1 --2- 172.21.15.96:0/404760649 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc5880b4e10 0x7fc5880b1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:12.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.936+0000 7fc58d400640 1 --2- 172.21.15.96:0/404760649 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc588099090 0x7fc588155830 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:12.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.936+0000 7fc58d400640 1 -- 172.21.15.96:0/404760649 >> 172.21.15.96:0/404760649 conn(0x7fc58806bae0 msgr2=0x7fc5880cb700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:12.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.937+0000 7fc58d400640 1 -- 172.21.15.96:0/404760649 shutdown_connections 2024-05-28T06:12:12.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:12.937+0000 7fc58d400640 1 -- 172.21.15.96:0/404760649 wait complete. 2024-05-28T06:12:12.958 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-28T06:12:13.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.149+0000 7f0bed600640 1 Processor -- start 2024-05-28T06:12:13.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.150+0000 7f0bed600640 1 -- start start 2024-05-28T06:12:13.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.151+0000 7f0bed600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0be806f860 0x7f0be80b3150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:13.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.151+0000 7f0bed600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f0be80b3690 0x7f0be80d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:13.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.151+0000 7f0bed600640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0be806ed10 con 0x7f0be80d09e0 2024-05-28T06:12:13.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.151+0000 7f0bed600640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f0be80d0e50 con 0x7f0be80b3690 2024-05-28T06:12:13.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.151+0000 7f0bed600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f0be815cb00 con 0x7f0be806f860 2024-05-28T06:12:13.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.151+0000 7f0be7400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f0be80d09e0 0x7f0be8160210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:59262/0 (socket says 172.21.15.96:59262) 2024-05-28T06:12:13.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.151+0000 7f0be7400640 1 -- 172.21.15.96:0/1322401302 learned_addr learned my addr 172.21.15.96:0/1322401302 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:12:13.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.151+0000 7f0bde000640 1 --2- 172.21.15.96:0/1322401302 >> v2:172.21.15.96:3300/0 conn(0x7f0be80b3690 0x7f0be80d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:13.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.151+0000 7f0be6a00640 1 --2- 172.21.15.96:0/1322401302 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0be806f860 0x7f0be80b3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:13.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.152+0000 7f0bde000640 1 -- 172.21.15.96:0/1322401302 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0be806f860 msgr2=0x7f0be80b3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:13.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.152+0000 7f0bde000640 1 --2- 172.21.15.96:0/1322401302 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0be806f860 0x7f0be80b3150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:13.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.152+0000 7f0bde000640 1 -- 172.21.15.96:0/1322401302 >> v1:172.21.15.96:6789/0 conn(0x7f0be80d09e0 legacy=0x7f0be8160210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:13.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.152+0000 7f0bde000640 1 -- 172.21.15.96:0/1322401302 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0be80b3cc0 con 0x7f0be80b3690 2024-05-28T06:12:13.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.153+0000 7f0bde000640 1 --2- 172.21.15.96:0/1322401302 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0be80b3690 0x7f0be80d04a0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f0bd00097a0 tx=0x7f0bd002a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbc12b8edf24363d server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:13.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.153+0000 7f0be6000640 1 -- 172.21.15.96:0/1322401302 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bd0037070 con 0x7f0be80b3690 2024-05-28T06:12:13.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.153+0000 7f0be6000640 1 -- 172.21.15.96:0/1322401302 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0bd0032440 con 0x7f0be80b3690 2024-05-28T06:12:13.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.153+0000 7f0be6000640 1 -- 172.21.15.96:0/1322401302 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bd0032810 con 0x7f0be80b3690 2024-05-28T06:12:13.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.153+0000 7f0bed600640 1 -- 172.21.15.96:0/1322401302 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0be80b3690 msgr2=0x7f0be80d04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:13.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.153+0000 7f0bed600640 1 --2- 172.21.15.96:0/1322401302 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0be80b3690 0x7f0be80d04a0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f0bd00097a0 tx=0x7f0bd002a180 comp rx=0 tx=0).stop 2024-05-28T06:12:13.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.154+0000 7f0bed600640 1 -- 172.21.15.96:0/1322401302 shutdown_connections 2024-05-28T06:12:13.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.154+0000 7f0bed600640 1 --2- 172.21.15.96:0/1322401302 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0be80b3690 0x7f0be80d04a0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:13.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.154+0000 7f0bed600640 1 --2- 172.21.15.96:0/1322401302 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0be806f860 0x7f0be80b3150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:13.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.154+0000 7f0bed600640 1 -- 172.21.15.96:0/1322401302 >> 172.21.15.96:0/1322401302 conn(0x7f0be806bae0 msgr2=0x7f0be806bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:13.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.154+0000 7f0bed600640 1 -- 172.21.15.96:0/1322401302 shutdown_connections 2024-05-28T06:12:13.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.154+0000 7f0bed600640 1 -- 172.21.15.96:0/1322401302 wait complete. 2024-05-28T06:12:13.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.156+0000 7f0bed600640 1 Processor -- start 2024-05-28T06:12:13.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.156+0000 7f0bed600640 1 -- start start 2024-05-28T06:12:13.159 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.157+0000 7f0bed600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0be806f860 0x7f0be80c4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:13.159 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.157+0000 7f0bed600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0be80b3690 0x7f0be815e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:13.159 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.157+0000 7f0bed600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0be80d09e0 0x7f0be80cefb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:13.159 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.157+0000 7f0bed600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f0be80b4560 con 0x7f0be80b3690 2024-05-28T06:12:13.159 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.157+0000 7f0bed600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f0be80b4280 con 0x7f0be80d09e0 2024-05-28T06:12:13.159 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.157+0000 7f0bed600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f0be80b4840 con 0x7f0be806f860 2024-05-28T06:12:13.159 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.157+0000 7f0bde000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0be80b3690 0x7f0be815e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:13.159 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.157+0000 7f0bde000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0be80b3690 0x7f0be815e770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:60988/0 (socket says 172.21.15.96:60988) 2024-05-28T06:12:13.159 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.157+0000 7f0bde000640 1 -- 172.21.15.96:0/2246208306 learned_addr learned my addr 172.21.15.96:0/2246208306 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:13.159 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.157+0000 7f0bde000640 1 -- 172.21.15.96:0/2246208306 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0be806f860 msgr2=0x7f0be80c4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:13.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.157+0000 7f0bde000640 1 --2- 172.21.15.96:0/2246208306 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0be806f860 0x7f0be80c4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:13.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.157+0000 7f0bde000640 1 -- 172.21.15.96:0/2246208306 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0be80d09e0 msgr2=0x7f0be80cefb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:13.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.157+0000 7f0bde000640 1 --2- 172.21.15.96:0/2246208306 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0be80d09e0 0x7f0be80cefb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:13.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.157+0000 7f0bde000640 1 -- 172.21.15.96:0/2246208306 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0be81559e0 con 0x7f0be80b3690 2024-05-28T06:12:13.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.158+0000 7f0bde000640 1 --2- 172.21.15.96:0/2246208306 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0be80b3690 0x7f0be815e770 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f0bd0035040 tx=0x7f0bd00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:13.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.158+0000 7f0bdf400640 1 -- 172.21.15.96:0/2246208306 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bd0037040 con 0x7f0be80b3690 2024-05-28T06:12:13.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.158+0000 7f0bdf400640 1 -- 172.21.15.96:0/2246208306 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0bd0030770 con 0x7f0be80b3690 2024-05-28T06:12:13.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.158+0000 7f0bdf400640 1 -- 172.21.15.96:0/2246208306 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bd0030b80 con 0x7f0be80b3690 2024-05-28T06:12:13.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.158+0000 7f0bed600640 1 -- 172.21.15.96:0/2246208306 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0be80c5500 con 0x7f0be80b3690 2024-05-28T06:12:13.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.158+0000 7f0bed600640 1 -- 172.21.15.96:0/2246208306 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0be814fd40 con 0x7f0be80b3690 2024-05-28T06:12:13.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.159+0000 7f0bed600640 1 -- 172.21.15.96:0/2246208306 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0be80689e0 con 0x7f0be80b3690 2024-05-28T06:12:13.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.160+0000 7f0bdf400640 1 -- 172.21.15.96:0/2246208306 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 16) ==== 91754+0+0 (secure 0 0 0) 0x7f0bd0044020 con 0x7f0be80b3690 2024-05-28T06:12:13.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.161+0000 7f0bdf400640 1 --2- 172.21.15.96:0/2246208306 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0bc406d0d0 0x7f0bc406f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:13.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.161+0000 7f0bdf400640 1 -- 172.21.15.96:0/2246208306 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5851+0+0 (secure 0 0 0) 0x7f0bd00ad390 con 0x7f0be80b3690 2024-05-28T06:12:13.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.162+0000 7f0be6a00640 1 --2- 172.21.15.96:0/2246208306 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0bc406d0d0 0x7f0bc406f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:13.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.163+0000 7f0be6a00640 1 --2- 172.21.15.96:0/2246208306 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0bc406d0d0 0x7f0bc406f590 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f0bd8006fd0 tx=0x7f0bd8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:13.166 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.165+0000 7f0bdf400640 1 -- 172.21.15.96:0/2246208306 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0be81559e0 con 0x7f0be80b3690 2024-05-28T06:12:13.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.403+0000 7f0bed600640 1 -- 172.21.15.96:0/2246208306 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0be80c5210 con 0x7f0be80b3690 2024-05-28T06:12:13.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.404+0000 7f0bdf400640 1 -- 172.21.15.96:0/2246208306 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+5181 (secure 0 0 0) 0x7f0bd0032e20 con 0x7f0be80b3690 2024-05-28T06:12:13.406 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:12:13.406 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":26,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/1693172076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":1693172076},{"type":"v1","addr":"172.21.15.123:6835","nonce":1693172076}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":4805,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/1098611268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":1098611268},{"type":"v1","addr":"172.21.15.123:6837","nonce":1098611268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5856,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6834/673165997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":673165997},{"type":"v1","addr":"172.21.15.96:6834","nonce":673165997}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-28T06:08:54.477674+0000","modified":"2024-05-28T06:08:57.572929+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5859},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5859":{"gid":5859,"name":"a","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.96:6835/935552515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6833","nonce":935552515},{"type":"v1","addr":"172.21.15.96:6835","nonce":935552515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5859,"qdb_cluster":[5859]},"id":3}]} 2024-05-28T06:12:13.406 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 26 2024-05-28T06:12:13.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.407+0000 7f0bed600640 1 -- 172.21.15.96:0/2246208306 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0bc406d0d0 msgr2=0x7f0bc406f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:13.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.407+0000 7f0bed600640 1 --2- 172.21.15.96:0/2246208306 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0bc406d0d0 0x7f0bc406f590 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f0bd8006fd0 tx=0x7f0bd8008040 comp rx=0 tx=0).stop 2024-05-28T06:12:13.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.407+0000 7f0bed600640 1 -- 172.21.15.96:0/2246208306 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0be80b3690 msgr2=0x7f0be815e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:13.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.407+0000 7f0bed600640 1 --2- 172.21.15.96:0/2246208306 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0be80b3690 0x7f0be815e770 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f0bd0035040 tx=0x7f0bd00046c0 comp rx=0 tx=0).stop 2024-05-28T06:12:13.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.407+0000 7f0bed600640 1 -- 172.21.15.96:0/2246208306 shutdown_connections 2024-05-28T06:12:13.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.407+0000 7f0bed600640 1 --2- 172.21.15.96:0/2246208306 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0bc406d0d0 0x7f0bc406f590 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:13.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.407+0000 7f0bed600640 1 --2- 172.21.15.96:0/2246208306 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0be80d09e0 0x7f0be80cefb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:13.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.407+0000 7f0bed600640 1 --2- 172.21.15.96:0/2246208306 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0be80b3690 0x7f0be815e770 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:13.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.407+0000 7f0bed600640 1 --2- 172.21.15.96:0/2246208306 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0be806f860 0x7f0be80c4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:13.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.407+0000 7f0bed600640 1 -- 172.21.15.96:0/2246208306 >> 172.21.15.96:0/2246208306 conn(0x7f0be806bae0 msgr2=0x7f0be80cb9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:13.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.407+0000 7f0bed600640 1 -- 172.21.15.96:0/2246208306 shutdown_connections 2024-05-28T06:12:13.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.407+0000 7f0bed600640 1 -- 172.21.15.96:0/2246208306 wait complete. 2024-05-28T06:12:13.428 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:12:13.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.627+0000 7fdbf3600640 1 Processor -- start 2024-05-28T06:12:13.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.627+0000 7fdbf3600640 1 -- start start 2024-05-28T06:12:13.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.629+0000 7fdbf3600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fdbec0d0590 0x7fdbec0d0a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:13.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.629+0000 7fdbf3600640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdbec0b51f0 con 0x7fdbec099090 2024-05-28T06:12:13.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.629+0000 7fdbf3600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdbec0aef60 con 0x7fdbec0b4e10 2024-05-28T06:12:13.631 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.629+0000 7fdbf3600640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fdbec0ad1d0 con 0x7fdbec0d0590 2024-05-28T06:12:13.631 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.629+0000 7fdbf1600640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fdbec099090 0x7fdbec0994a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:59274/0 (socket says 172.21.15.96:59274) 2024-05-28T06:12:13.631 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.629+0000 7fdbf1600640 1 -- 172.21.15.96:0/1546841561 learned_addr learned my addr 172.21.15.96:0/1546841561 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:12:13.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.629+0000 7fdbebe00640 1 --2- 172.21.15.96:0/1546841561 >> v2:172.21.15.123:3300/0 conn(0x7fdbec0d0590 0x7fdbec0d0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:13.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.629+0000 7fdbeb400640 1 -- 172.21.15.96:0/1546841561 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4224740115 0 0) 0x7fdbec0b51f0 con 0x7fdbec099090 2024-05-28T06:12:13.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.629+0000 7fdbeb400640 1 -- 172.21.15.96:0/1546841561 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdbd4003520 con 0x7fdbec099090 2024-05-28T06:12:13.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.630+0000 7fdbeb400640 1 -- 172.21.15.96:0/1546841561 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1822712170 0 0) 0x7fdbd4003520 con 0x7fdbec099090 2024-05-28T06:12:13.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.630+0000 7fdbeb400640 1 -- 172.21.15.96:0/1546841561 >> v2:172.21.15.123:3300/0 conn(0x7fdbec0d0590 msgr2=0x7fdbec0d0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:13.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.630+0000 7fdbeb400640 1 --2- 172.21.15.96:0/1546841561 >> v2:172.21.15.123:3300/0 conn(0x7fdbec0d0590 0x7fdbec0d0a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:13.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.630+0000 7fdbeb400640 1 -- 172.21.15.96:0/1546841561 >> v1:172.21.15.123:6789/0 conn(0x7fdbec0b4e10 legacy=0x7fdbec0cde30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:13.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.630+0000 7fdbeb400640 1 -- 172.21.15.96:0/1546841561 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbec0ad470 con 0x7fdbec099090 2024-05-28T06:12:13.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.630+0000 7fdbeb400640 1 -- 172.21.15.96:0/1546841561 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7fdbe40019f0 con 0x7fdbec099090 2024-05-28T06:12:13.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.630+0000 7fdbeb400640 1 -- 172.21.15.96:0/1546841561 >> v1:172.21.15.96:6789/0 conn(0x7fdbec099090 legacy=0x7fdbec0994a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:13.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.630+0000 7fdbeb400640 1 --2- 172.21.15.96:0/1546841561 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdbd4003a00 0x7fdbd4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:13.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.630+0000 7fdbeb400640 1 --2- 172.21.15.96:0/1546841561 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdbd4004800 0x7fdbd4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:13.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.630+0000 7fdbeb400640 1 --2- 172.21.15.96:0/1546841561 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdbd4005210 0x7fdbd400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:13.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.630+0000 7fdbeb400640 1 -- 172.21.15.96:0/1546841561 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fdbec0ad1d0 con 0x7fdbd4004800 2024-05-28T06:12:13.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.630+0000 7fdbeb400640 1 -- 172.21.15.96:0/1546841561 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fdbd400c160 con 0x7fdbd4003a00 2024-05-28T06:12:13.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.630+0000 7fdbeb400640 1 -- 172.21.15.96:0/1546841561 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fdbd400c360 con 0x7fdbd4005210 2024-05-28T06:12:13.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.630+0000 7fdbebe00640 1 --2- 172.21.15.96:0/1546841561 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdbd4004800 0x7fdbd4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:13.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.630+0000 7fdbebe00640 1 -- 172.21.15.96:0/1546841561 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdbd4005210 msgr2=0x7fdbd400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:13.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.630+0000 7fdbebe00640 1 --2- 172.21.15.96:0/1546841561 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdbd4005210 0x7fdbd400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:13.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.630+0000 7fdbebe00640 1 -- 172.21.15.96:0/1546841561 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdbd4003a00 msgr2=0x7fdbd4003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:13.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.630+0000 7fdbebe00640 1 --2- 172.21.15.96:0/1546841561 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdbd4003a00 0x7fdbd4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:13.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.631+0000 7fdbebe00640 1 -- 172.21.15.96:0/1546841561 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbd4003520 con 0x7fdbd4004800 2024-05-28T06:12:13.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.631+0000 7fdbf1600640 1 -- 172.21.15.96:0/1546841561 reap_dead start 2024-05-28T06:12:13.635 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.631+0000 7fdbebe00640 1 --2- 172.21.15.96:0/1546841561 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdbd4004800 0x7fdbd4004c80 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fdbd8009650 tx=0x7fdbd4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a8d33172df68aa7 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:13.635 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.632+0000 7fdbeb400640 1 -- 172.21.15.96:0/1546841561 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbd8038070 con 0x7fdbd4004800 2024-05-28T06:12:13.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.632+0000 7fdbeb400640 1 -- 172.21.15.96:0/1546841561 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdbd8033400 con 0x7fdbd4004800 2024-05-28T06:12:13.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.632+0000 7fdbeb400640 1 -- 172.21.15.96:0/1546841561 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbd8009690 con 0x7fdbd4004800 2024-05-28T06:12:13.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.632+0000 7fdbf3600640 1 -- 172.21.15.96:0/1546841561 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdbd4004800 msgr2=0x7fdbd4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:13.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.632+0000 7fdbf3600640 1 --2- 172.21.15.96:0/1546841561 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdbd4004800 0x7fdbd4004c80 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fdbd8009650 tx=0x7fdbd4005830 comp rx=0 tx=0).stop 2024-05-28T06:12:13.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.632+0000 7fdbf3600640 1 -- 172.21.15.96:0/1546841561 shutdown_connections 2024-05-28T06:12:13.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.632+0000 7fdbf3600640 1 --2- 172.21.15.96:0/1546841561 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdbd4004800 0x7fdbd4004c80 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:13.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.632+0000 7fdbf3600640 1 -- 172.21.15.96:0/1546841561 >> 172.21.15.96:0/1546841561 conn(0x7fdbec06bae0 msgr2=0x7fdbec06bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:13.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.632+0000 7fdbf3600640 1 -- 172.21.15.96:0/1546841561 shutdown_connections 2024-05-28T06:12:13.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.633+0000 7fdbf3600640 1 -- 172.21.15.96:0/1546841561 wait complete. 2024-05-28T06:12:13.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.634+0000 7fdbf3600640 1 Processor -- start 2024-05-28T06:12:13.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.634+0000 7fdbf3600640 1 -- start start 2024-05-28T06:12:13.638 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.635+0000 7fdbf3600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdbd4004800 0x7fdbec155ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:13.638 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.635+0000 7fdbf3600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdbec0c6650 0x7fdbec15d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:13.638 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.635+0000 7fdbf3600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdbec1f3d00 0x7fdbec1f41b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:13.638 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.635+0000 7fdbf3600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fdbec0cea00 con 0x7fdbd4004800 2024-05-28T06:12:13.639 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.635+0000 7fdbf3600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fdbec0ce880 con 0x7fdbec1f3d00 2024-05-28T06:12:13.639 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.635+0000 7fdbf3600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fdbec0b3150 con 0x7fdbec0c6650 2024-05-28T06:12:13.639 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.636+0000 7fdbf0c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdbd4004800 0x7fdbec155ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:13.639 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.636+0000 7fdbf0c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdbd4004800 0x7fdbec155ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:32768/0 (socket says 172.21.15.96:32768) 2024-05-28T06:12:13.639 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.636+0000 7fdbf0c00640 1 -- 172.21.15.96:0/705379289 learned_addr learned my addr 172.21.15.96:0/705379289 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:13.639 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.636+0000 7fdbf0c00640 1 -- 172.21.15.96:0/705379289 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdbec0c6650 msgr2=0x7fdbec15d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:13.639 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.636+0000 7fdbebe00640 1 --2- 172.21.15.96:0/705379289 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdbec0c6650 0x7fdbec15d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:13.639 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.636+0000 7fdbf1600640 1 --2- 172.21.15.96:0/705379289 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdbec1f3d00 0x7fdbec1f41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:13.639 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.636+0000 7fdbf0c00640 1 --2- 172.21.15.96:0/705379289 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdbec0c6650 0x7fdbec15d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:13.639 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.636+0000 7fdbf0c00640 1 -- 172.21.15.96:0/705379289 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdbec1f3d00 msgr2=0x7fdbec1f41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:13.639 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.636+0000 7fdbf0c00640 1 --2- 172.21.15.96:0/705379289 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdbec1f3d00 0x7fdbec1f41b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:13.639 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.636+0000 7fdbf0c00640 1 -- 172.21.15.96:0/705379289 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbec15dd00 con 0x7fdbd4004800 2024-05-28T06:12:13.639 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.636+0000 7fdbebe00640 1 --2- 172.21.15.96:0/705379289 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdbec0c6650 0x7fdbec15d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:12:13.640 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.636+0000 7fdbf0c00640 1 --2- 172.21.15.96:0/705379289 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdbd4004800 0x7fdbec155ef0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fdbe000c3c0 tx=0x7fdbe000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:13.640 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.637+0000 7fdbe8c00640 1 -- 172.21.15.96:0/705379289 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbe0017070 con 0x7fdbd4004800 2024-05-28T06:12:13.640 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.637+0000 7fdbe8c00640 1 -- 172.21.15.96:0/705379289 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdbe00052f0 con 0x7fdbd4004800 2024-05-28T06:12:13.640 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.637+0000 7fdbf3600640 1 -- 172.21.15.96:0/705379289 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdbec15dfa0 con 0x7fdbd4004800 2024-05-28T06:12:13.640 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.637+0000 7fdbe8c00640 1 -- 172.21.15.96:0/705379289 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbe0003830 con 0x7fdbd4004800 2024-05-28T06:12:13.640 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.637+0000 7fdbf3600640 1 -- 172.21.15.96:0/705379289 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbec15e560 con 0x7fdbd4004800 2024-05-28T06:12:13.640 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.638+0000 7fdbf3600640 1 -- 172.21.15.96:0/705379289 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdbec15e7a0 con 0x7fdbd4004800 2024-05-28T06:12:13.640 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.639+0000 7fdbe8c00640 1 -- 172.21.15.96:0/705379289 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 16) ==== 91754+0+0 (secure 0 0 0) 0x7fdbe001a070 con 0x7fdbd4004800 2024-05-28T06:12:13.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.640+0000 7fdbe8c00640 1 --2- 172.21.15.96:0/705379289 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdbcc06d100 0x7fdbcc06f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:13.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.640+0000 7fdbe8c00640 1 -- 172.21.15.96:0/705379289 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5851+0+0 (secure 0 0 0) 0x7fdbe008c1b0 con 0x7fdbd4004800 2024-05-28T06:12:13.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.640+0000 7fdbebe00640 1 --2- 172.21.15.96:0/705379289 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdbcc06d100 0x7fdbcc06f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:13.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.641+0000 7fdbebe00640 1 --2- 172.21.15.96:0/705379289 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdbcc06d100 0x7fdbcc06f5c0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fdbd8035000 tx=0x7fdbd8008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:13.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.644+0000 7fdbe8c00640 1 -- 172.21.15.96:0/705379289 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fdbec15e7a0 con 0x7fdbd4004800 2024-05-28T06:12:13.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.830+0000 7fdbf3600640 1 -- 172.21.15.96:0/705379289 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdbec0689e0 con 0x7fdbd4004800 2024-05-28T06:12:13.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.832+0000 7fdbe8c00640 1 -- 172.21.15.96:0/705379289 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v48) ==== 74+0+16868 (secure 0 0 0) 0x7fdbe00595f0 con 0x7fdbd4004800 2024-05-28T06:12:13.835 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:12:13.835 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":48,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:12:04.592827+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-05-28T06:08:52.257477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-05-28T06:08:53.459376+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"48","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":48,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.9","mappings":[{"from":0,"to":5}]},{"pgid":"7.37","mappings":[{"from":0,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":7,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[{"pool":7,"snaps":[{"begin":2,"length":2}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:12:13.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.835+0000 7fdbf3600640 1 -- 172.21.15.96:0/705379289 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdbcc06d100 msgr2=0x7fdbcc06f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:13.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.835+0000 7fdbf3600640 1 --2- 172.21.15.96:0/705379289 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdbcc06d100 0x7fdbcc06f5c0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fdbd8035000 tx=0x7fdbd8008030 comp rx=0 tx=0).stop 2024-05-28T06:12:13.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.835+0000 7fdbf3600640 1 -- 172.21.15.96:0/705379289 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdbd4004800 msgr2=0x7fdbec155ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:13.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.835+0000 7fdbf3600640 1 --2- 172.21.15.96:0/705379289 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdbd4004800 0x7fdbec155ef0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fdbe000c3c0 tx=0x7fdbe000c880 comp rx=0 tx=0).stop 2024-05-28T06:12:13.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.835+0000 7fdbf3600640 1 -- 172.21.15.96:0/705379289 shutdown_connections 2024-05-28T06:12:13.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.835+0000 7fdbf3600640 1 --2- 172.21.15.96:0/705379289 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdbcc06d100 0x7fdbcc06f5c0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:13.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.835+0000 7fdbf3600640 1 --2- 172.21.15.96:0/705379289 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdbec1f3d00 0x7fdbec1f41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:13.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.835+0000 7fdbf3600640 1 --2- 172.21.15.96:0/705379289 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdbec0c6650 0x7fdbec15d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:13.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.835+0000 7fdbf3600640 1 --2- 172.21.15.96:0/705379289 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdbd4004800 0x7fdbec155ef0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:13.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.835+0000 7fdbf3600640 1 -- 172.21.15.96:0/705379289 >> 172.21.15.96:0/705379289 conn(0x7fdbec06bae0 msgr2=0x7fdbec06bec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:13.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.835+0000 7fdbf3600640 1 -- 172.21.15.96:0/705379289 shutdown_connections 2024-05-28T06:12:13.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:13.835+0000 7fdbf3600640 1 -- 172.21.15.96:0/705379289 wait complete. 2024-05-28T06:12:13.856 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-05-28T06:12:14.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.042+0000 7f517ec00640 1 Processor -- start 2024-05-28T06:12:14.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.043+0000 7f517ec00640 1 -- start start 2024-05-28T06:12:14.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.044+0000 7f517ec00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f5178162cd0 0x7f51780d0090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:14.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.044+0000 7f517ec00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5178163330 con 0x7f51780d3280 2024-05-28T06:12:14.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.044+0000 7f517ec00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51780cf2d0 con 0x7f517815e8a0 2024-05-28T06:12:14.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.044+0000 7f517ec00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f51780cf5b0 con 0x7f5178162cd0 2024-05-28T06:12:14.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.044+0000 7f517ce00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f51780d3280 0x7f51781625c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:59290/0 (socket says 172.21.15.96:59290) 2024-05-28T06:12:14.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.044+0000 7f517ce00640 1 -- 172.21.15.96:0/1386062513 learned_addr learned my addr 172.21.15.96:0/1386062513 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:12:14.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.045+0000 7f5177e00640 1 -- 172.21.15.96:0/1386062513 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1150601941 0 0) 0x7f5178163330 con 0x7f51780d3280 2024-05-28T06:12:14.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.045+0000 7f5177e00640 1 -- 172.21.15.96:0/1386062513 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5160003520 con 0x7f51780d3280 2024-05-28T06:12:14.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.045+0000 7f517e200640 1 --2- 172.21.15.96:0/1386062513 >> v2:172.21.15.123:3300/0 conn(0x7f5178162cd0 0x7f51780d0090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:14.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.045+0000 7f5177e00640 1 -- 172.21.15.96:0/1386062513 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3562164584 0 0) 0x7f5160003520 con 0x7f51780d3280 2024-05-28T06:12:14.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.045+0000 7f5177e00640 1 -- 172.21.15.96:0/1386062513 >> v2:172.21.15.123:3300/0 conn(0x7f5178162cd0 msgr2=0x7f51780d0090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:14.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.045+0000 7f5177e00640 1 --2- 172.21.15.96:0/1386062513 >> v2:172.21.15.123:3300/0 conn(0x7f5178162cd0 0x7f51780d0090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:14.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.045+0000 7f5177e00640 1 -- 172.21.15.96:0/1386062513 >> v1:172.21.15.123:6789/0 conn(0x7f517815e8a0 legacy=0x7f51780d2b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:14.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.045+0000 7f5177e00640 1 -- 172.21.15.96:0/1386062513 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51780cf890 con 0x7f51780d3280 2024-05-28T06:12:14.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.045+0000 7f5177e00640 1 -- 172.21.15.96:0/1386062513 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f5168001670 con 0x7f51780d3280 2024-05-28T06:12:14.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.045+0000 7f5177e00640 1 -- 172.21.15.96:0/1386062513 >> v1:172.21.15.96:6789/0 conn(0x7f51780d3280 legacy=0x7f51781625c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:14.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.045+0000 7f5177e00640 1 --2- 172.21.15.96:0/1386062513 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f51600039b0 0x7f5160003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:14.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.045+0000 7f5177e00640 1 --2- 172.21.15.96:0/1386062513 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5160004800 0x7f5160004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:14.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.045+0000 7f5177e00640 1 --2- 172.21.15.96:0/1386062513 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f51600051c0 0x7f516000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:14.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.045+0000 7f5177e00640 1 -- 172.21.15.96:0/1386062513 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f51780cf5b0 con 0x7f51600039b0 2024-05-28T06:12:14.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.045+0000 7f5177e00640 1 -- 172.21.15.96:0/1386062513 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f516000c0f0 con 0x7f5160004800 2024-05-28T06:12:14.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.045+0000 7f5177e00640 1 -- 172.21.15.96:0/1386062513 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f516000c2f0 con 0x7f51600051c0 2024-05-28T06:12:14.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.046+0000 7f517d800640 1 --2- 172.21.15.96:0/1386062513 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f51600039b0 0x7f5160003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:14.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.046+0000 7f517ce00640 1 --2- 172.21.15.96:0/1386062513 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5160004800 0x7f5160004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:14.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.046+0000 7f517e200640 1 --2- 172.21.15.96:0/1386062513 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f51600051c0 0x7f516000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:14.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.046+0000 7f517d800640 1 -- 172.21.15.96:0/1386062513 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f51600051c0 msgr2=0x7f516000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:14.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.046+0000 7f517d800640 1 --2- 172.21.15.96:0/1386062513 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f51600051c0 0x7f516000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:14.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.046+0000 7f517d800640 1 -- 172.21.15.96:0/1386062513 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5160004800 msgr2=0x7f5160004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:14.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.046+0000 7f517d800640 1 --2- 172.21.15.96:0/1386062513 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5160004800 0x7f5160004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:14.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.046+0000 7f517d800640 1 -- 172.21.15.96:0/1386062513 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5160003520 con 0x7f51600039b0 2024-05-28T06:12:14.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.046+0000 7f517e200640 1 -- 172.21.15.96:0/1386062513 reap_dead start 2024-05-28T06:12:14.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.047+0000 7f517d800640 1 --2- 172.21.15.96:0/1386062513 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f51600039b0 0x7f5160003db0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f516c003da0 tx=0x7f5160005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=413d05c41598402c server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:14.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.048+0000 7f5177e00640 1 -- 172.21.15.96:0/1386062513 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f516c037070 con 0x7f51600039b0 2024-05-28T06:12:14.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.048+0000 7f5177e00640 1 -- 172.21.15.96:0/1386062513 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f516c02a9a0 con 0x7f51600039b0 2024-05-28T06:12:14.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.048+0000 7f5177e00640 1 -- 172.21.15.96:0/1386062513 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f516c0039a0 con 0x7f51600039b0 2024-05-28T06:12:14.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.048+0000 7f517ec00640 1 -- 172.21.15.96:0/1386062513 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f51600039b0 msgr2=0x7f5160003db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:14.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.048+0000 7f517ec00640 1 --2- 172.21.15.96:0/1386062513 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f51600039b0 0x7f5160003db0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f516c003da0 tx=0x7f5160005830 comp rx=0 tx=0).stop 2024-05-28T06:12:14.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.049+0000 7f517ec00640 1 -- 172.21.15.96:0/1386062513 shutdown_connections 2024-05-28T06:12:14.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.049+0000 7f517ec00640 1 --2- 172.21.15.96:0/1386062513 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f51600039b0 0x7f5160003db0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:14.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.049+0000 7f517ec00640 1 -- 172.21.15.96:0/1386062513 >> 172.21.15.96:0/1386062513 conn(0x7f517808bfc0 msgr2=0x7f51780d4590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:14.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.049+0000 7f517ec00640 1 -- 172.21.15.96:0/1386062513 shutdown_connections 2024-05-28T06:12:14.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.049+0000 7f517ec00640 1 -- 172.21.15.96:0/1386062513 wait complete. 2024-05-28T06:12:14.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.051+0000 7f517ec00640 1 Processor -- start 2024-05-28T06:12:14.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.051+0000 7f517ec00640 1 -- start start 2024-05-28T06:12:14.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.052+0000 7f517ec00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f51600039b0 0x7f51780cdaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:14.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.052+0000 7f517ec00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f517815d690 0x7f517815db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:14.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.052+0000 7f517ec00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f517809f070 0x7f517809f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:14.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.052+0000 7f517ec00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f517815ee00 con 0x7f51600039b0 2024-05-28T06:12:14.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.052+0000 7f517ec00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f517815ec80 con 0x7f517815d690 2024-05-28T06:12:14.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.052+0000 7f517ec00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f51780d0090 con 0x7f517809f070 2024-05-28T06:12:14.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.052+0000 7f517d800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f51600039b0 0x7f51780cdaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:14.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.052+0000 7f517d800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f51600039b0 0x7f51780cdaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:32774/0 (socket says 172.21.15.96:32774) 2024-05-28T06:12:14.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.052+0000 7f517d800640 1 -- 172.21.15.96:0/347396736 learned_addr learned my addr 172.21.15.96:0/347396736 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:14.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.052+0000 7f517d800640 1 -- 172.21.15.96:0/347396736 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f517809f070 msgr2=0x7f517809f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:14.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.052+0000 7f517d800640 1 --2- 172.21.15.96:0/347396736 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f517809f070 0x7f517809f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:14.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.052+0000 7f517d800640 1 -- 172.21.15.96:0/347396736 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f517815d690 msgr2=0x7f517815db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:14.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.052+0000 7f517d800640 1 --2- 172.21.15.96:0/347396736 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f517815d690 0x7f517815db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:14.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.052+0000 7f517d800640 1 -- 172.21.15.96:0/347396736 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51781f88c0 con 0x7f51600039b0 2024-05-28T06:12:14.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.053+0000 7f517d800640 1 --2- 172.21.15.96:0/347396736 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f51600039b0 0x7f51780cdaa0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f516c003da0 tx=0x7f516c0036f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:14.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.053+0000 7f5175600640 1 -- 172.21.15.96:0/347396736 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f516c041030 con 0x7f51600039b0 2024-05-28T06:12:14.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.053+0000 7f5175600640 1 -- 172.21.15.96:0/347396736 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f516c02a7a0 con 0x7f51600039b0 2024-05-28T06:12:14.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.053+0000 7f517ec00640 1 -- 172.21.15.96:0/347396736 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51781f7b00 con 0x7f51600039b0 2024-05-28T06:12:14.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.053+0000 7f5175600640 1 -- 172.21.15.96:0/347396736 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f516c037040 con 0x7f51600039b0 2024-05-28T06:12:14.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.053+0000 7f517ec00640 1 -- 172.21.15.96:0/347396736 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51781f80c0 con 0x7f51600039b0 2024-05-28T06:12:14.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.054+0000 7f517ec00640 1 -- 172.21.15.96:0/347396736 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51780689e0 con 0x7f51600039b0 2024-05-28T06:12:14.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.055+0000 7f5175600640 1 -- 172.21.15.96:0/347396736 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 16) ==== 91754+0+0 (secure 0 0 0) 0x7f516c044070 con 0x7f51600039b0 2024-05-28T06:12:14.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.056+0000 7f5175600640 1 --2- 172.21.15.96:0/347396736 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f515c06d0d0 0x7f515c06f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:14.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.056+0000 7f5175600640 1 -- 172.21.15.96:0/347396736 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5851+0+0 (secure 0 0 0) 0x7f516c0aca60 con 0x7f51600039b0 2024-05-28T06:12:14.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.057+0000 7f517ce00640 1 --2- 172.21.15.96:0/347396736 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f515c06d0d0 0x7f515c06f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:14.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.058+0000 7f517ce00640 1 --2- 172.21.15.96:0/347396736 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f515c06d0d0 0x7f515c06f590 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f5168000f60 tx=0x7f5168000ed0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:14.062 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.060+0000 7f5175600640 1 -- 172.21.15.96:0/347396736 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f516c04b070 con 0x7f51600039b0 2024-05-28T06:12:14.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.281+0000 7f517ec00640 1 -- 172.21.15.96:0/347396736 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f51780d2950 con 0x7f51600039b0 2024-05-28T06:12:14.284 INFO:tasks.ceph.mon.a.smithi096.stderr:2024-05-28T06:12:14.282+0000 7fb2fca00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-28T06:12:14.288 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.286+0000 7f5175600640 1 -- 172.21.15.96:0/347396736 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v27) ==== 161+0+0 (secure 0 0 0) 0x7f516c079e70 con 0x7f51600039b0 2024-05-28T06:12:14.288 INFO:teuthology.orchestra.run.smithi096.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-28T06:12:14.291 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.289+0000 7f517ec00640 1 -- 172.21.15.96:0/347396736 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f515c06d0d0 msgr2=0x7f515c06f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:14.291 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.289+0000 7f517ec00640 1 --2- 172.21.15.96:0/347396736 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f515c06d0d0 0x7f515c06f590 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f5168000f60 tx=0x7f5168000ed0 comp rx=0 tx=0).stop 2024-05-28T06:12:14.291 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.289+0000 7f517ec00640 1 -- 172.21.15.96:0/347396736 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f51600039b0 msgr2=0x7f51780cdaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:14.291 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.289+0000 7f517ec00640 1 --2- 172.21.15.96:0/347396736 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f51600039b0 0x7f51780cdaa0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f516c003da0 tx=0x7f516c0036f0 comp rx=0 tx=0).stop 2024-05-28T06:12:14.291 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.289+0000 7f517ec00640 1 -- 172.21.15.96:0/347396736 shutdown_connections 2024-05-28T06:12:14.291 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.289+0000 7f517ec00640 1 --2- 172.21.15.96:0/347396736 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f515c06d0d0 0x7f515c06f590 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:14.291 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.289+0000 7f517ec00640 1 --2- 172.21.15.96:0/347396736 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f517809f070 0x7f517809f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:14.291 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.289+0000 7f517ec00640 1 --2- 172.21.15.96:0/347396736 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f517815d690 0x7f517815db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:14.291 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.289+0000 7f517ec00640 1 --2- 172.21.15.96:0/347396736 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f51600039b0 0x7f51780cdaa0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:14.291 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.289+0000 7f517ec00640 1 -- 172.21.15.96:0/347396736 >> 172.21.15.96:0/347396736 conn(0x7f517808bfc0 msgr2=0x7f51780d4620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:14.291 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.289+0000 7f517ec00640 1 -- 172.21.15.96:0/347396736 shutdown_connections 2024-05-28T06:12:14.292 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.290+0000 7f517ec00640 1 -- 172.21.15.96:0/347396736 wait complete. 2024-05-28T06:12:14.311 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-05-28T06:12:14.375 INFO:tasks.ceph.mds.a.smithi096.stdout:starting mds.a at 2024-05-28T06:12:14.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.491+0000 7f2845600640 1 Processor -- start 2024-05-28T06:12:14.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.491+0000 7f2845600640 1 -- start start 2024-05-28T06:12:14.494 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.492+0000 7f2845600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f284015d5d0 0x7f28400cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:14.494 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.492+0000 7f2845600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f28400b32c0 0x7f28400b3750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:14.494 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.492+0000 7f2845600640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f284015cb10 con 0x7f28400d0640 2024-05-28T06:12:14.494 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.492+0000 7f2845600640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f28400adbb0 con 0x7f284015d5d0 2024-05-28T06:12:14.494 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.492+0000 7f2845600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f28400ad5f0 con 0x7f28400b32c0 2024-05-28T06:12:14.494 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.492+0000 7f283ea00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f28400d0640 0x7f28400d0a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:59310/0 (socket says 172.21.15.96:59310) 2024-05-28T06:12:14.494 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.492+0000 7f283ea00640 1 -- 172.21.15.96:0/699468295 learned_addr learned my addr 172.21.15.96:0/699468295 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:12:14.496 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.493+0000 7f2836000640 1 --2- 172.21.15.96:0/699468295 >> v2:172.21.15.123:3300/0 conn(0x7f284015d5d0 0x7f28400cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:14.496 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.493+0000 7f283f400640 1 --2- 172.21.15.96:0/699468295 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f28400b32c0 0x7f28400b3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:14.496 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.493+0000 7f283e000640 1 -- 172.21.15.96:0/699468295 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2726409017 0 0) 0x7f284015cb10 con 0x7f28400d0640 2024-05-28T06:12:14.496 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.493+0000 7f283e000640 1 -- 172.21.15.96:0/699468295 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2824003520 con 0x7f28400d0640 2024-05-28T06:12:14.496 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.493+0000 7f283e000640 1 -- 172.21.15.96:0/699468295 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 614625062 0 0) 0x7f2824003520 con 0x7f28400d0640 2024-05-28T06:12:14.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.493+0000 7f283e000640 1 -- 172.21.15.96:0/699468295 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f28400b32c0 msgr2=0x7f28400b3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:14.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.493+0000 7f283e000640 1 --2- 172.21.15.96:0/699468295 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f28400b32c0 0x7f28400b3750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:14.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.493+0000 7f283e000640 1 -- 172.21.15.96:0/699468295 >> v2:172.21.15.123:3300/0 conn(0x7f284015d5d0 msgr2=0x7f28400cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:14.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.493+0000 7f283e000640 1 --2- 172.21.15.96:0/699468295 >> v2:172.21.15.123:3300/0 conn(0x7f284015d5d0 0x7f28400cf950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:14.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.493+0000 7f283e000640 1 -- 172.21.15.96:0/699468295 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28400ad8d0 con 0x7f28400d0640 2024-05-28T06:12:14.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.493+0000 7f283e000640 1 -- 172.21.15.96:0/699468295 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f2830001690 con 0x7f28400d0640 2024-05-28T06:12:14.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.493+0000 7f283f400640 1 --2- 172.21.15.96:0/699468295 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f28400b32c0 0x7f28400b3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-28T06:12:14.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.493+0000 7f283e000640 1 -- 172.21.15.96:0/699468295 >> v1:172.21.15.96:6789/0 conn(0x7f28400d0640 legacy=0x7f28400d0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:14.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.493+0000 7f283e000640 1 --2- 172.21.15.96:0/699468295 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f28240039b0 0x7f2824003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:14.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.493+0000 7f283e000640 1 --2- 172.21.15.96:0/699468295 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f28400b32c0 0x7f2824004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:14.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.493+0000 7f283e000640 1 --2- 172.21.15.96:0/699468295 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2824004d40 0x7f28240051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:14.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.494+0000 7f283e000640 1 -- 172.21.15.96:0/699468295 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f28400ad5f0 con 0x7f28240039b0 2024-05-28T06:12:14.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.494+0000 7f283e000640 1 -- 172.21.15.96:0/699468295 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f282400b9c0 con 0x7f2824004d40 2024-05-28T06:12:14.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.494+0000 7f283e000640 1 -- 172.21.15.96:0/699468295 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f282400bb40 con 0x7f28400b32c0 2024-05-28T06:12:14.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.494+0000 7f283ea00640 1 --2- 172.21.15.96:0/699468295 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f28240039b0 0x7f2824003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:14.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.494+0000 7f2836000640 1 --2- 172.21.15.96:0/699468295 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f28400b32c0 0x7f2824004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:14.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.494+0000 7f283ea00640 1 -- 172.21.15.96:0/699468295 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f28400b32c0 msgr2=0x7f2824004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:14.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.494+0000 7f283ea00640 1 --2- 172.21.15.96:0/699468295 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f28400b32c0 0x7f2824004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:14.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.494+0000 7f283ea00640 1 -- 172.21.15.96:0/699468295 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2824004d40 msgr2=0x7f28240051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:14.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.494+0000 7f283ea00640 1 --2- 172.21.15.96:0/699468295 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2824004d40 0x7f28240051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:14.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.494+0000 7f283ea00640 1 -- 172.21.15.96:0/699468295 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2824003520 con 0x7f28240039b0 2024-05-28T06:12:14.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.495+0000 7f283ea00640 1 --2- 172.21.15.96:0/699468295 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f28240039b0 0x7f2824003dd0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f283000c210 tx=0x7f2824005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=817868810a4a88a3 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:14.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.495+0000 7f283e000640 1 -- 172.21.15.96:0/699468295 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2830007d60 con 0x7f28240039b0 2024-05-28T06:12:14.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.495+0000 7f283e000640 1 -- 172.21.15.96:0/699468295 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2830005af0 con 0x7f28240039b0 2024-05-28T06:12:14.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.495+0000 7f283e000640 1 -- 172.21.15.96:0/699468295 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2830007070 con 0x7f28240039b0 2024-05-28T06:12:14.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.496+0000 7f2845600640 1 -- 172.21.15.96:0/699468295 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f28240039b0 msgr2=0x7f2824003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:14.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.496+0000 7f2845600640 1 --2- 172.21.15.96:0/699468295 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f28240039b0 0x7f2824003dd0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f283000c210 tx=0x7f2824005830 comp rx=0 tx=0).stop 2024-05-28T06:12:14.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.496+0000 7f283f400640 1 -- 172.21.15.96:0/699468295 reap_dead start 2024-05-28T06:12:14.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.496+0000 7f2845600640 1 -- 172.21.15.96:0/699468295 shutdown_connections 2024-05-28T06:12:14.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.496+0000 7f2845600640 1 -- 172.21.15.96:0/699468295 >> 172.21.15.96:0/699468295 conn(0x7f284006bd20 msgr2=0x7f284006c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:14.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.496+0000 7f2845600640 1 -- 172.21.15.96:0/699468295 shutdown_connections 2024-05-28T06:12:14.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.496+0000 7f2845600640 1 -- 172.21.15.96:0/699468295 wait complete. 2024-05-28T06:12:14.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.498+0000 7f2845600640 1 Processor -- start 2024-05-28T06:12:14.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.498+0000 7f2845600640 1 -- start start 2024-05-28T06:12:14.502 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.499+0000 7f2845600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f28240039b0 0x7f284014d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:14.502 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.499+0000 7f2845600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2840073240 0x7f28400b2250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:14.502 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.499+0000 7f2845600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f28400b2790 0x7f28401f79b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:14.502 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.499+0000 7f2845600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f284015da50 con 0x7f2840073240 2024-05-28T06:12:14.502 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.499+0000 7f2845600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f284015d0b0 con 0x7f28400b2790 2024-05-28T06:12:14.502 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.499+0000 7f2845600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f28400ad2a0 con 0x7f28240039b0 2024-05-28T06:12:14.502 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.499+0000 7f2836000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2840073240 0x7f28400b2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:14.502 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.499+0000 7f2836000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2840073240 0x7f28400b2250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:32824/0 (socket says 172.21.15.96:32824) 2024-05-28T06:12:14.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.499+0000 7f2836000640 1 -- 172.21.15.96:0/2403996914 learned_addr learned my addr 172.21.15.96:0/2403996914 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:14.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.499+0000 7f2836000640 1 -- 172.21.15.96:0/2403996914 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f28240039b0 msgr2=0x7f284014d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:14.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.499+0000 7f2836000640 1 --2- 172.21.15.96:0/2403996914 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f28240039b0 0x7f284014d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:14.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.499+0000 7f2836000640 1 -- 172.21.15.96:0/2403996914 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f28400b2790 msgr2=0x7f28401f79b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:14.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.499+0000 7f2836000640 1 --2- 172.21.15.96:0/2403996914 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f28400b2790 0x7f28401f79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:14.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.499+0000 7f2836000640 1 -- 172.21.15.96:0/2403996914 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f284014da50 con 0x7f2840073240 2024-05-28T06:12:14.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.500+0000 7f2836000640 1 --2- 172.21.15.96:0/2403996914 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2840073240 0x7f28400b2250 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f2828010910 tx=0x7f2828010dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:14.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.500+0000 7f2837400640 1 -- 172.21.15.96:0/2403996914 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f282800f040 con 0x7f2840073240 2024-05-28T06:12:14.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.500+0000 7f2837400640 1 -- 172.21.15.96:0/2403996914 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f282800ec20 con 0x7f2840073240 2024-05-28T06:12:14.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.500+0000 7f2837400640 1 -- 172.21.15.96:0/2403996914 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f282801c400 con 0x7f2840073240 2024-05-28T06:12:14.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.500+0000 7f2845600640 1 -- 172.21.15.96:0/2403996914 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28400aff30 con 0x7f2840073240 2024-05-28T06:12:14.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.500+0000 7f2845600640 1 -- 172.21.15.96:0/2403996914 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28400b04f0 con 0x7f2840073240 2024-05-28T06:12:14.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.501+0000 7f2845600640 1 -- 172.21.15.96:0/2403996914 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f284006fa50 con 0x7f2840073240 2024-05-28T06:12:14.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.502+0000 7f2837400640 1 -- 172.21.15.96:0/2403996914 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 16) ==== 91754+0+0 (secure 0 0 0) 0x7f282801c5a0 con 0x7f2840073240 2024-05-28T06:12:14.505 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.503+0000 7f2837400640 1 --2- 172.21.15.96:0/2403996914 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f281c06d0d0 0x7f281c06f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:14.505 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.503+0000 7f2837400640 1 -- 172.21.15.96:0/2403996914 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 5909+0+0 (secure 0 0 0) 0x7f282808d550 con 0x7f2840073240 2024-05-28T06:12:14.505 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.504+0000 7f283ea00640 1 --2- 172.21.15.96:0/2403996914 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f281c06d0d0 0x7f281c06f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:14.506 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.505+0000 7f283ea00640 1 --2- 172.21.15.96:0/2403996914 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f281c06d0d0 0x7f281c06f590 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f2830033000 tx=0x7f2830007000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:14.509 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.508+0000 7f2837400640 1 -- 172.21.15.96:0/2403996914 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2828018030 con 0x7f2840073240 2024-05-28T06:12:14.755 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:14.753+0000 7f2845600640 1 -- 172.21.15.96:0/2403996914 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f28400689e0 con 0x7f2840073240 2024-05-28T06:12:15.293 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.290+0000 7f2837400640 1 -- 172.21.15.96:0/2403996914 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v28) ==== 104+0+0 (secure 0 0 0) 0x7f282805a240 con 0x7f2840073240 2024-05-28T06:12:15.297 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.295+0000 7f2845600640 1 -- 172.21.15.96:0/2403996914 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f281c06d0d0 msgr2=0x7f281c06f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:15.297 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.295+0000 7f2845600640 1 --2- 172.21.15.96:0/2403996914 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f281c06d0d0 0x7f281c06f590 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f2830033000 tx=0x7f2830007000 comp rx=0 tx=0).stop 2024-05-28T06:12:15.297 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.295+0000 7f2845600640 1 -- 172.21.15.96:0/2403996914 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2840073240 msgr2=0x7f28400b2250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:15.297 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.295+0000 7f2845600640 1 --2- 172.21.15.96:0/2403996914 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2840073240 0x7f28400b2250 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f2828010910 tx=0x7f2828010dd0 comp rx=0 tx=0).stop 2024-05-28T06:12:15.297 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.295+0000 7f2845600640 1 -- 172.21.15.96:0/2403996914 shutdown_connections 2024-05-28T06:12:15.297 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.295+0000 7f2845600640 1 --2- 172.21.15.96:0/2403996914 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f281c06d0d0 0x7f281c06f590 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:15.297 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.295+0000 7f2845600640 1 --2- 172.21.15.96:0/2403996914 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f28400b2790 0x7f28401f79b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:15.297 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.295+0000 7f2845600640 1 --2- 172.21.15.96:0/2403996914 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2840073240 0x7f28400b2250 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:15.297 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.295+0000 7f2845600640 1 --2- 172.21.15.96:0/2403996914 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f28240039b0 0x7f284014d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:15.298 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.295+0000 7f2845600640 1 -- 172.21.15.96:0/2403996914 >> 172.21.15.96:0/2403996914 conn(0x7f284006bd20 msgr2=0x7f2840155c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:15.298 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.295+0000 7f2845600640 1 -- 172.21.15.96:0/2403996914 shutdown_connections 2024-05-28T06:12:15.298 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.295+0000 7f2845600640 1 -- 172.21.15.96:0/2403996914 wait complete. 2024-05-28T06:12:15.320 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-05-28T06:12:15.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.519+0000 7f7885c00640 1 Processor -- start 2024-05-28T06:12:15.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.519+0000 7f7885c00640 1 -- start start 2024-05-28T06:12:15.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.521+0000 7f7885c00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f78800afde0 0x7f788015ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:15.523 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.521+0000 7f7885c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f788015db50 0x7f788015dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:15.523 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.521+0000 7f7885c00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f788015c9b0 con 0x7f788006fa90 2024-05-28T06:12:15.523 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.521+0000 7f7885c00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f78801604e0 con 0x7f78800afde0 2024-05-28T06:12:15.523 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.521+0000 7f7885c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f78800b01c0 con 0x7f788015db50 2024-05-28T06:12:15.523 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.521+0000 7f787f400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f788006fa90 0x7f788015d150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:59326/0 (socket says 172.21.15.96:59326) 2024-05-28T06:12:15.523 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.521+0000 7f787f400640 1 -- 172.21.15.96:0/627678002 learned_addr learned my addr 172.21.15.96:0/627678002 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:12:15.525 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.522+0000 7f7885200640 1 --2- 172.21.15.96:0/627678002 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f788015db50 0x7f788015dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:15.525 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.522+0000 7f787fe00640 1 --2- 172.21.15.96:0/627678002 >> v2:172.21.15.123:3300/0 conn(0x7f78800afde0 0x7f788015ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:15.525 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.522+0000 7f787ea00640 1 -- 172.21.15.96:0/627678002 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2317964288 0 0) 0x7f786c001110 con 0x7f788006fa90 2024-05-28T06:12:15.525 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.522+0000 7f787ea00640 1 -- 172.21.15.96:0/627678002 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7864003520 con 0x7f788006fa90 2024-05-28T06:12:15.525 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.522+0000 7f787ea00640 1 -- 172.21.15.96:0/627678002 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2887415183 0 0) 0x7f7864003520 con 0x7f788006fa90 2024-05-28T06:12:15.525 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.522+0000 7f787ea00640 1 -- 172.21.15.96:0/627678002 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f788015db50 msgr2=0x7f788015dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:15.525 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.522+0000 7f787ea00640 1 --2- 172.21.15.96:0/627678002 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f788015db50 0x7f788015dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:15.525 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.522+0000 7f787ea00640 1 -- 172.21.15.96:0/627678002 >> v2:172.21.15.123:3300/0 conn(0x7f78800afde0 msgr2=0x7f788015ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:15.525 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.522+0000 7f787ea00640 1 --2- 172.21.15.96:0/627678002 >> v2:172.21.15.123:3300/0 conn(0x7f78800afde0 0x7f788015ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:15.525 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.522+0000 7f787ea00640 1 -- 172.21.15.96:0/627678002 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7880160710 con 0x7f788006fa90 2024-05-28T06:12:15.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.522+0000 7f787ea00640 1 -- 172.21.15.96:0/627678002 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f786c001840 con 0x7f788006fa90 2024-05-28T06:12:15.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.522+0000 7f787ea00640 1 -- 172.21.15.96:0/627678002 >> v1:172.21.15.96:6789/0 conn(0x7f788006fa90 legacy=0x7f788015d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:15.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.522+0000 7f787ea00640 1 --2- 172.21.15.96:0/627678002 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f788015db50 0x7f78640039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:15.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.522+0000 7f787ea00640 1 --2- 172.21.15.96:0/627678002 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7864003f30 0x7f7864004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:15.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.522+0000 7f787ea00640 1 --2- 172.21.15.96:0/627678002 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7864004d40 0x7f78640051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:15.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.522+0000 7f787ea00640 1 -- 172.21.15.96:0/627678002 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f78800b01c0 con 0x7f7864004d40 2024-05-28T06:12:15.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.522+0000 7f787ea00640 1 -- 172.21.15.96:0/627678002 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f786400ba70 con 0x7f7864003f30 2024-05-28T06:12:15.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.522+0000 7f787ea00640 1 -- 172.21.15.96:0/627678002 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f786400bc40 con 0x7f788015db50 2024-05-28T06:12:15.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.522+0000 7f787fe00640 1 --2- 172.21.15.96:0/627678002 >> v2:172.21.15.123:3300/0 conn(0x7f78800afde0 0x7f788015ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-28T06:12:15.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.522+0000 7f7885200640 1 --2- 172.21.15.96:0/627678002 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7864004d40 0x7f78640051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:15.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.523+0000 7f787f400640 1 --2- 172.21.15.96:0/627678002 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7864003f30 0x7f7864004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:15.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.523+0000 7f7885200640 1 -- 172.21.15.96:0/627678002 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f788015db50 msgr2=0x7f78640039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:15.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.523+0000 7f7885200640 1 --2- 172.21.15.96:0/627678002 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f788015db50 0x7f78640039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:15.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.523+0000 7f7885200640 1 -- 172.21.15.96:0/627678002 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7864003f30 msgr2=0x7f7864004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:15.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.523+0000 7f7885200640 1 --2- 172.21.15.96:0/627678002 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7864003f30 0x7f7864004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:15.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.523+0000 7f7885200640 1 -- 172.21.15.96:0/627678002 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7864003520 con 0x7f7864004d40 2024-05-28T06:12:15.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.523+0000 7f787f400640 1 --2- 172.21.15.96:0/627678002 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7864003f30 0x7f7864004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:12:15.528 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.524+0000 7f7885200640 1 --2- 172.21.15.96:0/627678002 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7864004d40 0x7f78640051c0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f787800e170 tx=0x7f7864005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d0fd5038096ca29 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:15.528 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.524+0000 7f787ea00640 1 -- 172.21.15.96:0/627678002 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7878034020 con 0x7f7864004d40 2024-05-28T06:12:15.528 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.524+0000 7f787ea00640 1 -- 172.21.15.96:0/627678002 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7878030430 con 0x7f7864004d40 2024-05-28T06:12:15.528 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.524+0000 7f787ea00640 1 -- 172.21.15.96:0/627678002 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7878009c10 con 0x7f7864004d40 2024-05-28T06:12:15.528 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.525+0000 7f7885c00640 1 -- 172.21.15.96:0/627678002 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7864004d40 msgr2=0x7f78640051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:15.529 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.525+0000 7f7885c00640 1 --2- 172.21.15.96:0/627678002 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7864004d40 0x7f78640051c0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f787800e170 tx=0x7f7864005830 comp rx=0 tx=0).stop 2024-05-28T06:12:15.529 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.525+0000 7f7885200640 1 -- 172.21.15.96:0/627678002 reap_dead start 2024-05-28T06:12:15.529 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.525+0000 7f7885c00640 1 -- 172.21.15.96:0/627678002 shutdown_connections 2024-05-28T06:12:15.529 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.525+0000 7f7885c00640 1 -- 172.21.15.96:0/627678002 >> 172.21.15.96:0/627678002 conn(0x7f78800723b0 msgr2=0x7f7880072790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:15.529 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.525+0000 7f7885c00640 1 -- 172.21.15.96:0/627678002 shutdown_connections 2024-05-28T06:12:15.529 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.525+0000 7f7885c00640 1 -- 172.21.15.96:0/627678002 wait complete. 2024-05-28T06:12:15.529 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.527+0000 7f7885c00640 1 Processor -- start 2024-05-28T06:12:15.529 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.527+0000 7f7885c00640 1 -- start start 2024-05-28T06:12:15.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.528+0000 7f7885c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f788013d2d0 0x7f788014cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:15.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.528+0000 7f7885c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f788014aa10 0x7f788014ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:15.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.528+0000 7f7885c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f78801f0080 0x7f78801f04f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:15.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.528+0000 7f7885c00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f7880160cd0 con 0x7f788014aa10 2024-05-28T06:12:15.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.528+0000 7f7885c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f78800cf470 con 0x7f788013d2d0 2024-05-28T06:12:15.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.528+0000 7f7885c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f788015cc30 con 0x7f78801f0080 2024-05-28T06:12:15.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.528+0000 7f787f400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f788014aa10 0x7f788014ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:15.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.528+0000 7f787f400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f788014aa10 0x7f788014ae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:32832/0 (socket says 172.21.15.96:32832) 2024-05-28T06:12:15.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.528+0000 7f787f400640 1 -- 172.21.15.96:0/3011443767 learned_addr learned my addr 172.21.15.96:0/3011443767 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:15.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.528+0000 7f787fe00640 1 --2- 172.21.15.96:0/3011443767 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f788013d2d0 0x7f788014cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:15.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.528+0000 7f787f400640 1 -- 172.21.15.96:0/3011443767 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f78801f0080 msgr2=0x7f78801f04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:15.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.528+0000 7f787f400640 1 --2- 172.21.15.96:0/3011443767 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f78801f0080 0x7f78801f04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:15.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.528+0000 7f787f400640 1 -- 172.21.15.96:0/3011443767 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f788013d2d0 msgr2=0x7f788014cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:15.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.528+0000 7f787f400640 1 --2- 172.21.15.96:0/3011443767 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f788013d2d0 0x7f788014cdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:15.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.528+0000 7f787f400640 1 -- 172.21.15.96:0/3011443767 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f788015eb90 con 0x7f788014aa10 2024-05-28T06:12:15.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.529+0000 7f787f400640 1 --2- 172.21.15.96:0/3011443767 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f788014aa10 0x7f788014ae70 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f786c00dae0 tx=0x7f786c007e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:15.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.529+0000 7f7877e00640 1 -- 172.21.15.96:0/3011443767 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f786c019070 con 0x7f788014aa10 2024-05-28T06:12:15.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.529+0000 7f7877e00640 1 -- 172.21.15.96:0/3011443767 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f786c004cb0 con 0x7f788014aa10 2024-05-28T06:12:15.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.529+0000 7f7877e00640 1 -- 172.21.15.96:0/3011443767 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f786c004e60 con 0x7f788014aa10 2024-05-28T06:12:15.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.529+0000 7f7885c00640 1 -- 172.21.15.96:0/3011443767 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f788015ee70 con 0x7f788014aa10 2024-05-28T06:12:15.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.529+0000 7f7885c00640 1 -- 172.21.15.96:0/3011443767 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78800b4350 con 0x7f788014aa10 2024-05-28T06:12:15.533 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.530+0000 7f7885c00640 1 -- 172.21.15.96:0/3011443767 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78800689e0 con 0x7f788014aa10 2024-05-28T06:12:15.533 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.531+0000 7f7877e00640 1 -- 172.21.15.96:0/3011443767 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 16) ==== 91754+0+0 (secure 0 0 0) 0x7f786c007310 con 0x7f788014aa10 2024-05-28T06:12:15.533 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.532+0000 7f7877e00640 1 --2- 172.21.15.96:0/3011443767 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f786006d0a0 0x7f786006f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:15.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.532+0000 7f7877e00640 1 -- 172.21.15.96:0/3011443767 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 5909+0+0 (secure 0 0 0) 0x7f786c08cae0 con 0x7f788014aa10 2024-05-28T06:12:15.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.533+0000 7f787fe00640 1 --2- 172.21.15.96:0/3011443767 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f786006d0a0 0x7f786006f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:15.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.534+0000 7f787fe00640 1 --2- 172.21.15.96:0/3011443767 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f786006d0a0 0x7f786006f560 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f787000d8d0 tx=0x7f7870008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:15.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.537+0000 7f7877e00640 1 -- 172.21.15.96:0/3011443767 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f788015c9b0 con 0x7f788014aa10 2024-05-28T06:12:15.739 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:15.738+0000 7f7885c00640 1 -- 172.21.15.96:0/3011443767 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f788006c060 con 0x7f788014aa10 2024-05-28T06:12:16.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:16.324+0000 7f7877e00640 1 -- 172.21.15.96:0/3011443767 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v50) ==== 181+0+0 (secure 0 0 0) 0x7f786c001800 con 0x7f788014aa10 2024-05-28T06:12:16.326 INFO:teuthology.orchestra.run.smithi096.stderr:pool 'cephfs_metadata' removed 2024-05-28T06:12:16.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:16.327+0000 7f7885c00640 1 -- 172.21.15.96:0/3011443767 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f786006d0a0 msgr2=0x7f786006f560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:16.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:16.327+0000 7f7885c00640 1 --2- 172.21.15.96:0/3011443767 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f786006d0a0 0x7f786006f560 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f787000d8d0 tx=0x7f7870008040 comp rx=0 tx=0).stop 2024-05-28T06:12:16.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:16.327+0000 7f7885c00640 1 -- 172.21.15.96:0/3011443767 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f788014aa10 msgr2=0x7f788014ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:16.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:16.327+0000 7f7885c00640 1 --2- 172.21.15.96:0/3011443767 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f788014aa10 0x7f788014ae70 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f786c00dae0 tx=0x7f786c007e50 comp rx=0 tx=0).stop 2024-05-28T06:12:16.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:16.328+0000 7f7885c00640 1 -- 172.21.15.96:0/3011443767 shutdown_connections 2024-05-28T06:12:16.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:16.328+0000 7f7885c00640 1 --2- 172.21.15.96:0/3011443767 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f786006d0a0 0x7f786006f560 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:16.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:16.328+0000 7f7885c00640 1 --2- 172.21.15.96:0/3011443767 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f78801f0080 0x7f78801f04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:16.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:16.328+0000 7f7885c00640 1 --2- 172.21.15.96:0/3011443767 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f788014aa10 0x7f788014ae70 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:16.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:16.328+0000 7f7885c00640 1 --2- 172.21.15.96:0/3011443767 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f788013d2d0 0x7f788014cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:16.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:16.328+0000 7f7885c00640 1 -- 172.21.15.96:0/3011443767 >> 172.21.15.96:0/3011443767 conn(0x7f78800723b0 msgr2=0x7f788006f240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:16.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:16.330+0000 7f7885c00640 1 -- 172.21.15.96:0/3011443767 shutdown_connections 2024-05-28T06:12:16.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:16.331+0000 7f7885c00640 1 -- 172.21.15.96:0/3011443767 wait complete. 2024-05-28T06:12:16.355 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-05-28T06:12:16.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:16.541+0000 7fa284800640 1 Processor -- start 2024-05-28T06:12:16.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:16.541+0000 7fa284800640 1 -- start start 2024-05-28T06:12:16.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:16.543+0000 7fa284800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fa27c06edf0 0x7fa27c13fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:16.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:16.543+0000 7fa284800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fa27c06f250 0x7fa27c140330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:16.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:16.543+0000 7fa284800640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa27c0d0020 con 0x7fa27c0cf2f0 2024-05-28T06:12:16.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:16.543+0000 7fa284800640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fa27c06dd50 con 0x7fa27c06f250 2024-05-28T06:12:16.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:16.543+0000 7fa284800640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fa27c06e030 con 0x7fa27c06edf0 2024-05-28T06:12:16.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:16.543+0000 7fa282800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fa27c06f250 0x7fa27c140330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:16.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:16.543+0000 7fa281e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fa27c0cf2f0 0x7fa27c0cf910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:59332/0 (socket says 172.21.15.96:59332) 2024-05-28T06:12:16.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:16.543+0000 7fa281e00640 1 -- 172.21.15.96:0/3538166084 learned_addr learned my addr 172.21.15.96:0/3538166084 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:12:16.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:16.544+0000 7fa282800640 1 -- 172.21.15.96:0/3538166084 >> v2:172.21.15.123:3300/0 conn(0x7fa27c06edf0 msgr2=0x7fa27c13fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:16.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:16.544+0000 7fa281400640 1 --2- 172.21.15.96:0/3538166084 >> v2:172.21.15.123:3300/0 conn(0x7fa27c06edf0 0x7fa27c13fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:16.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:16.544+0000 7fa282800640 1 --2- 172.21.15.96:0/3538166084 >> v2:172.21.15.123:3300/0 conn(0x7fa27c06edf0 0x7fa27c13fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:16.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:16.544+0000 7fa282800640 1 -- 172.21.15.96:0/3538166084 >> v1:172.21.15.96:6789/0 conn(0x7fa27c0cf2f0 legacy=0x7fa27c0cf910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:16.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:16.544+0000 7fa280a00640 1 -- 172.21.15.96:0/3538166084 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2324593803 0 0) 0x7fa27c0d0020 con 0x7fa27c0cf2f0 2024-05-28T06:12:16.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:16.544+0000 7fa282800640 1 -- 172.21.15.96:0/3538166084 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa27c06e310 con 0x7fa27c06f250 2024-05-28T06:12:16.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:16.544+0000 7fa281400640 1 --2- 172.21.15.96:0/3538166084 >> v2:172.21.15.123:3300/0 conn(0x7fa27c06edf0 0x7fa27c13fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:12:16.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:16.545+0000 7fa282800640 1 --2- 172.21.15.96:0/3538166084 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa27c06f250 0x7fa27c140330 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fa27c071330 tx=0x7fa27402a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=284cc0566c9c8924 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:16.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:16.545+0000 7fa280a00640 1 -- 172.21.15.96:0/3538166084 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa274037070 con 0x7fa27c06f250 2024-05-28T06:12:16.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:16.545+0000 7fa280a00640 1 -- 172.21.15.96:0/3538166084 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa274032440 con 0x7fa27c06f250 2024-05-28T06:12:16.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:16.545+0000 7fa280a00640 1 -- 172.21.15.96:0/3538166084 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa27402ae60 con 0x7fa27c06f250 2024-05-28T06:12:16.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:16.546+0000 7fa284800640 1 -- 172.21.15.96:0/3538166084 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa27c06f250 msgr2=0x7fa27c140330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:16.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:16.546+0000 7fa284800640 1 --2- 172.21.15.96:0/3538166084 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa27c06f250 0x7fa27c140330 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fa27c071330 tx=0x7fa27402a110 comp rx=0 tx=0).stop 2024-05-28T06:12:16.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:16.546+0000 7fa284800640 1 -- 172.21.15.96:0/3538166084 shutdown_connections 2024-05-28T06:12:16.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:16.546+0000 7fa284800640 1 --2- 172.21.15.96:0/3538166084 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa27c06f250 0x7fa27c140330 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:16.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:16.546+0000 7fa284800640 1 --2- 172.21.15.96:0/3538166084 >> v2:172.21.15.123:3300/0 conn(0x7fa27c06edf0 0x7fa27c13fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:16.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:16.546+0000 7fa284800640 1 -- 172.21.15.96:0/3538166084 >> 172.21.15.96:0/3538166084 conn(0x7fa27c0b1e50 msgr2=0x7fa27c0b2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:16.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:16.546+0000 7fa284800640 1 -- 172.21.15.96:0/3538166084 shutdown_connections 2024-05-28T06:12:16.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:16.546+0000 7fa284800640 1 -- 172.21.15.96:0/3538166084 wait complete. 2024-05-28T06:12:16.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:16.548+0000 7fa284800640 1 Processor -- start 2024-05-28T06:12:16.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:16.548+0000 7fa284800640 1 -- start start 2024-05-28T06:12:16.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:16.549+0000 7fa284800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa27c06edf0 0x7fa27c099a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:16.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:16.549+0000 7fa284800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa27c06f250 0x7fa27c09cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:16.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:16.549+0000 7fa284800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa27c0cf2f0 0x7fa27c09a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:16.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:16.549+0000 7fa284800640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa27c146020 con 0x7fa27c06f250 2024-05-28T06:12:16.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:16.549+0000 7fa284800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa27c06e8d0 con 0x7fa27c0cf2f0 2024-05-28T06:12:16.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:16.549+0000 7fa284800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fa27c1461a0 con 0x7fa27c06edf0 2024-05-28T06:12:16.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:16.549+0000 7fa281400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa27c06f250 0x7fa27c09cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:16.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:16.549+0000 7fa281400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa27c06f250 0x7fa27c09cc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:32850/0 (socket says 172.21.15.96:32850) 2024-05-28T06:12:16.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:16.549+0000 7fa281400640 1 -- 172.21.15.96:0/1842407202 learned_addr learned my addr 172.21.15.96:0/1842407202 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:16.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:16.549+0000 7fa281400640 1 -- 172.21.15.96:0/1842407202 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa27c06edf0 msgr2=0x7fa27c099a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:16.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:16.549+0000 7fa281400640 1 --2- 172.21.15.96:0/1842407202 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa27c06edf0 0x7fa27c099a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:16.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:16.549+0000 7fa281400640 1 -- 172.21.15.96:0/1842407202 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa27c0cf2f0 msgr2=0x7fa27c09a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:16.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:16.549+0000 7fa281400640 1 --2- 172.21.15.96:0/1842407202 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa27c0cf2f0 0x7fa27c09a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:16.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:16.549+0000 7fa281400640 1 -- 172.21.15.96:0/1842407202 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa27c09d6f0 con 0x7fa27c06f250 2024-05-28T06:12:16.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:16.550+0000 7fa281400640 1 --2- 172.21.15.96:0/1842407202 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa27c06f250 0x7fa27c09cc30 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fa26c00c7f0 tx=0x7fa26c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:16.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:16.550+0000 7fa272000640 1 -- 172.21.15.96:0/1842407202 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa26c015070 con 0x7fa27c06f250 2024-05-28T06:12:16.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:16.550+0000 7fa272000640 1 -- 172.21.15.96:0/1842407202 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa26c007820 con 0x7fa27c06f250 2024-05-28T06:12:16.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:16.550+0000 7fa284800640 1 -- 172.21.15.96:0/1842407202 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa27c09d9d0 con 0x7fa27c06f250 2024-05-28T06:12:16.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:16.550+0000 7fa272000640 1 -- 172.21.15.96:0/1842407202 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa26c007bd0 con 0x7fa27c06f250 2024-05-28T06:12:16.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:16.550+0000 7fa284800640 1 -- 172.21.15.96:0/1842407202 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa27c15d200 con 0x7fa27c06f250 2024-05-28T06:12:16.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:16.551+0000 7fa284800640 1 -- 172.21.15.96:0/1842407202 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa27c09def0 con 0x7fa27c06f250 2024-05-28T06:12:16.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:16.552+0000 7fa272000640 1 -- 172.21.15.96:0/1842407202 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 16) ==== 91754+0+0 (secure 0 0 0) 0x7fa26c018070 con 0x7fa27c06f250 2024-05-28T06:12:16.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:16.553+0000 7fa272000640 1 --2- 172.21.15.96:0/1842407202 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa26006d100 0x7fa26006f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:16.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:16.553+0000 7fa272000640 1 -- 172.21.15.96:0/1842407202 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5471+0+0 (secure 0 0 0) 0x7fa26c08cee0 con 0x7fa27c06f250 2024-05-28T06:12:16.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:16.554+0000 7fa281e00640 1 --2- 172.21.15.96:0/1842407202 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa26006d100 0x7fa26006f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:16.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:16.555+0000 7fa281e00640 1 --2- 172.21.15.96:0/1842407202 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa26006d100 0x7fa26006f5c0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fa278001560 tx=0x7fa278007ca0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:16.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:16.558+0000 7fa272000640 1 -- 172.21.15.96:0/1842407202 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa27c09def0 con 0x7fa27c06f250 2024-05-28T06:12:16.774 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:16.773+0000 7fa284800640 1 -- 172.21.15.96:0/1842407202 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fa27c0689e0 con 0x7fa27c06f250 2024-05-28T06:12:17.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.699+0000 7fa272000640 1 -- 172.21.15.96:0/1842407202 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v51) ==== 169+0+0 (secure 0 0 0) 0x7fa26c05a360 con 0x7fa27c06f250 2024-05-28T06:12:17.701 INFO:teuthology.orchestra.run.smithi096.stderr:pool 'cephfs_data' removed 2024-05-28T06:12:17.705 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.703+0000 7fa284800640 1 -- 172.21.15.96:0/1842407202 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa26006d100 msgr2=0x7fa26006f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:17.705 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.703+0000 7fa284800640 1 --2- 172.21.15.96:0/1842407202 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa26006d100 0x7fa26006f5c0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fa278001560 tx=0x7fa278007ca0 comp rx=0 tx=0).stop 2024-05-28T06:12:17.705 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.703+0000 7fa284800640 1 -- 172.21.15.96:0/1842407202 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa27c06f250 msgr2=0x7fa27c09cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:17.705 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.703+0000 7fa284800640 1 --2- 172.21.15.96:0/1842407202 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa27c06f250 0x7fa27c09cc30 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fa26c00c7f0 tx=0x7fa26c00ccb0 comp rx=0 tx=0).stop 2024-05-28T06:12:17.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.704+0000 7fa284800640 1 -- 172.21.15.96:0/1842407202 shutdown_connections 2024-05-28T06:12:17.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.704+0000 7fa284800640 1 --2- 172.21.15.96:0/1842407202 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa26006d100 0x7fa26006f5c0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:17.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.704+0000 7fa284800640 1 --2- 172.21.15.96:0/1842407202 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa27c0cf2f0 0x7fa27c09a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:17.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.704+0000 7fa284800640 1 --2- 172.21.15.96:0/1842407202 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa27c06f250 0x7fa27c09cc30 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:17.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.704+0000 7fa284800640 1 --2- 172.21.15.96:0/1842407202 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa27c06edf0 0x7fa27c099a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:17.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.704+0000 7fa284800640 1 -- 172.21.15.96:0/1842407202 >> 172.21.15.96:0/1842407202 conn(0x7fa27c0b1e50 msgr2=0x7fa27c15c1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:17.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.705+0000 7fa284800640 1 -- 172.21.15.96:0/1842407202 shutdown_connections 2024-05-28T06:12:17.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.706+0000 7fa284800640 1 -- 172.21.15.96:0/1842407202 wait complete. 2024-05-28T06:12:17.733 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-05-28T06:12:17.922 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.920+0000 7f5382600640 1 Processor -- start 2024-05-28T06:12:17.922 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.922+0000 7f5382600640 1 -- start start 2024-05-28T06:12:17.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.923+0000 7f5382600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f537c153510 0x7f537c0b4d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:17.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.923+0000 7f5382600640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f537c15d900 con 0x7f537c0ccb90 2024-05-28T06:12:17.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.923+0000 7f5382600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f537c0b3f90 con 0x7f537c15d270 2024-05-28T06:12:17.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.923+0000 7f5382600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f537c1538f0 con 0x7f537c153510 2024-05-28T06:12:17.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.923+0000 7f537be00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f537c0ccb90 0x7f537c15cb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:59334/0 (socket says 172.21.15.96:59334) 2024-05-28T06:12:17.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.923+0000 7f537be00640 1 -- 172.21.15.96:0/740254407 learned_addr learned my addr 172.21.15.96:0/740254407 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:12:17.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.923+0000 7f5381200640 1 --2- 172.21.15.96:0/740254407 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f537c153510 0x7f537c0b4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:17.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.923+0000 7f537b400640 1 -- 172.21.15.96:0/740254407 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2219679776 0 0) 0x7f537c15d900 con 0x7f537c0ccb90 2024-05-28T06:12:17.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.923+0000 7f537b400640 1 -- 172.21.15.96:0/740254407 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5364003520 con 0x7f537c0ccb90 2024-05-28T06:12:17.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.923+0000 7f537b400640 1 -- 172.21.15.96:0/740254407 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2301335538 0 0) 0x7f5364003520 con 0x7f537c0ccb90 2024-05-28T06:12:17.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.923+0000 7f537b400640 1 -- 172.21.15.96:0/740254407 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f537c153510 msgr2=0x7f537c0b4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:17.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.923+0000 7f537b400640 1 --2- 172.21.15.96:0/740254407 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f537c153510 0x7f537c0b4d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:17.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.923+0000 7f537b400640 1 -- 172.21.15.96:0/740254407 >> v1:172.21.15.123:6789/0 conn(0x7f537c15d270 legacy=0x7f537c06e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:17.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.923+0000 7f537b400640 1 -- 172.21.15.96:0/740254407 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f537c0b4270 con 0x7f537c0ccb90 2024-05-28T06:12:17.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.923+0000 7f537b400640 1 -- 172.21.15.96:0/740254407 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f536c001670 con 0x7f537c0ccb90 2024-05-28T06:12:17.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.923+0000 7f537b400640 1 -- 172.21.15.96:0/740254407 >> v1:172.21.15.96:6789/0 conn(0x7f537c0ccb90 legacy=0x7f537c15cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:17.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.924+0000 7f537b400640 1 --2- 172.21.15.96:0/740254407 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f537c153510 0x7f53640039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:17.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.924+0000 7f537b400640 1 --2- 172.21.15.96:0/740254407 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5364003f60 0x7f5364004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:17.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.924+0000 7f537b400640 1 --2- 172.21.15.96:0/740254407 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5364004d40 0x7f53640051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:17.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.924+0000 7f537b400640 1 -- 172.21.15.96:0/740254407 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f536400bb50 con 0x7f5364003f60 2024-05-28T06:12:17.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.924+0000 7f537b400640 1 -- 172.21.15.96:0/740254407 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f536400bcd0 con 0x7f5364004d40 2024-05-28T06:12:17.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.924+0000 7f537b400640 1 -- 172.21.15.96:0/740254407 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f536400bed0 con 0x7f537c153510 2024-05-28T06:12:17.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.924+0000 7f537be00640 1 --2- 172.21.15.96:0/740254407 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5364003f60 0x7f5364004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:17.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.924+0000 7f5381200640 1 --2- 172.21.15.96:0/740254407 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f537c153510 0x7f53640039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:17.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.924+0000 7f5381c00640 1 --2- 172.21.15.96:0/740254407 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5364004d40 0x7f53640051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:17.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.924+0000 7f537be00640 1 -- 172.21.15.96:0/740254407 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f537c153510 msgr2=0x7f53640039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:17.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.924+0000 7f537be00640 1 --2- 172.21.15.96:0/740254407 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f537c153510 0x7f53640039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:17.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.924+0000 7f537be00640 1 -- 172.21.15.96:0/740254407 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5364004d40 msgr2=0x7f53640051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:17.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.924+0000 7f537be00640 1 --2- 172.21.15.96:0/740254407 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5364004d40 0x7f53640051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:17.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.924+0000 7f537be00640 1 -- 172.21.15.96:0/740254407 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5364003520 con 0x7f5364003f60 2024-05-28T06:12:17.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.925+0000 7f537be00640 1 --2- 172.21.15.96:0/740254407 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5364003f60 0x7f5364004800 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f536c00c240 tx=0x7f5364005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a09e2b795dfcbe7 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:17.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.926+0000 7f537b400640 1 -- 172.21.15.96:0/740254407 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f536c007d10 con 0x7f5364003f60 2024-05-28T06:12:17.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.926+0000 7f537b400640 1 -- 172.21.15.96:0/740254407 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f536c005ad0 con 0x7f5364003f60 2024-05-28T06:12:17.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.926+0000 7f537b400640 1 -- 172.21.15.96:0/740254407 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f536c007070 con 0x7f5364003f60 2024-05-28T06:12:17.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.926+0000 7f5382600640 1 -- 172.21.15.96:0/740254407 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5364003f60 msgr2=0x7f5364004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:17.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.926+0000 7f5382600640 1 --2- 172.21.15.96:0/740254407 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5364003f60 0x7f5364004800 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f536c00c240 tx=0x7f5364005830 comp rx=0 tx=0).stop 2024-05-28T06:12:17.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.926+0000 7f5381c00640 1 -- 172.21.15.96:0/740254407 reap_dead start 2024-05-28T06:12:17.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.926+0000 7f5382600640 1 -- 172.21.15.96:0/740254407 shutdown_connections 2024-05-28T06:12:17.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.926+0000 7f5382600640 1 -- 172.21.15.96:0/740254407 >> 172.21.15.96:0/740254407 conn(0x7f537c06b900 msgr2=0x7f537c06bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:17.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.926+0000 7f5382600640 1 -- 172.21.15.96:0/740254407 shutdown_connections 2024-05-28T06:12:17.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.926+0000 7f5382600640 1 -- 172.21.15.96:0/740254407 wait complete. 2024-05-28T06:12:17.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.928+0000 7f5382600640 1 Processor -- start 2024-05-28T06:12:17.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.928+0000 7f5382600640 1 -- start start 2024-05-28T06:12:17.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.929+0000 7f5382600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5364003f60 0x7f537c0cf090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:17.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.929+0000 7f5382600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f537c151b50 0x7f537c072ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:17.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.929+0000 7f5382600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f537c151fd0 0x7f537c073010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:17.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.929+0000 7f5382600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f537c0b4d50 con 0x7f537c151fd0 2024-05-28T06:12:17.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.929+0000 7f5382600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f537c0b48c0 con 0x7f5364003f60 2024-05-28T06:12:17.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.929+0000 7f5382600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f537c0b4ed0 con 0x7f537c151b50 2024-05-28T06:12:17.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.930+0000 7f5381c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f537c151fd0 0x7f537c073010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:17.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.930+0000 7f5381c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f537c151fd0 0x7f537c073010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:32868/0 (socket says 172.21.15.96:32868) 2024-05-28T06:12:17.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.930+0000 7f5381c00640 1 -- 172.21.15.96:0/1760447801 learned_addr learned my addr 172.21.15.96:0/1760447801 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:17.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.930+0000 7f537be00640 1 --2- 172.21.15.96:0/1760447801 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f537c151b50 0x7f537c072ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:17.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.930+0000 7f5381200640 1 --2- 172.21.15.96:0/1760447801 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5364003f60 0x7f537c0cf090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:17.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.930+0000 7f5381c00640 1 -- 172.21.15.96:0/1760447801 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f537c151b50 msgr2=0x7f537c072ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:17.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.930+0000 7f5381c00640 1 --2- 172.21.15.96:0/1760447801 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f537c151b50 0x7f537c072ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:17.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.930+0000 7f5381c00640 1 -- 172.21.15.96:0/1760447801 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5364003f60 msgr2=0x7f537c0cf090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:17.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.930+0000 7f5381c00640 1 --2- 172.21.15.96:0/1760447801 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5364003f60 0x7f537c0cf090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:17.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.930+0000 7f5381c00640 1 -- 172.21.15.96:0/1760447801 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f537c0cfc40 con 0x7f537c151fd0 2024-05-28T06:12:17.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.930+0000 7f537be00640 1 --2- 172.21.15.96:0/1760447801 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f537c151b50 0x7f537c072ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:12:17.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.930+0000 7f5381c00640 1 --2- 172.21.15.96:0/1760447801 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f537c151fd0 0x7f537c073010 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f537c15d650 tx=0x7f537400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:17.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.931+0000 7f5378c00640 1 -- 172.21.15.96:0/1760447801 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5374018070 con 0x7f537c151fd0 2024-05-28T06:12:17.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.931+0000 7f5378c00640 1 -- 172.21.15.96:0/1760447801 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5374003b70 con 0x7f537c151fd0 2024-05-28T06:12:17.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.931+0000 7f5378c00640 1 -- 172.21.15.96:0/1760447801 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5374005070 con 0x7f537c151fd0 2024-05-28T06:12:17.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.931+0000 7f5382600640 1 -- 172.21.15.96:0/1760447801 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f537c0ad130 con 0x7f537c151fd0 2024-05-28T06:12:17.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.931+0000 7f5382600640 1 -- 172.21.15.96:0/1760447801 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f537c0ad6f0 con 0x7f537c151fd0 2024-05-28T06:12:17.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.932+0000 7f5382600640 1 -- 172.21.15.96:0/1760447801 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f537c0689e0 con 0x7f537c151fd0 2024-05-28T06:12:17.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.933+0000 7f5378c00640 1 -- 172.21.15.96:0/1760447801 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 16) ==== 91754+0+0 (secure 0 0 0) 0x7f537401b070 con 0x7f537c151fd0 2024-05-28T06:12:17.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.934+0000 7f5378c00640 1 --2- 172.21.15.96:0/1760447801 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f536006d100 0x7f536006f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:17.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.934+0000 7f5378c00640 1 -- 172.21.15.96:0/1760447801 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 4971+0+0 (secure 0 0 0) 0x7f5374089e60 con 0x7f537c151fd0 2024-05-28T06:12:17.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.935+0000 7f5381200640 1 --2- 172.21.15.96:0/1760447801 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f536006d100 0x7f536006f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:17.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.936+0000 7f5381200640 1 --2- 172.21.15.96:0/1760447801 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f536006d100 0x7f536006f5c0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f5370009a10 tx=0x7f5370008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:17.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:17.939+0000 7f5378c00640 1 -- 172.21.15.96:0/1760447801 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5374059b80 con 0x7f537c151fd0 2024-05-28T06:12:18.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.150+0000 7f5382600640 1 -- 172.21.15.96:0/1760447801 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7f537c0b3c40 con 0x7f537c151fd0 2024-05-28T06:12:18.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.156+0000 7f5378c00640 1 -- 172.21.15.96:0/1760447801 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v13) ==== 104+0+0 (secure 0 0 0) 0x7f5374059480 con 0x7f537c151fd0 2024-05-28T06:12:18.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.158+0000 7f5382600640 1 -- 172.21.15.96:0/1760447801 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f536006d100 msgr2=0x7f536006f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:18.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.158+0000 7f5382600640 1 --2- 172.21.15.96:0/1760447801 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f536006d100 0x7f536006f5c0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f5370009a10 tx=0x7f5370008040 comp rx=0 tx=0).stop 2024-05-28T06:12:18.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.158+0000 7f5382600640 1 -- 172.21.15.96:0/1760447801 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f537c151fd0 msgr2=0x7f537c073010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:18.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.158+0000 7f5382600640 1 --2- 172.21.15.96:0/1760447801 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f537c151fd0 0x7f537c073010 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f537c15d650 tx=0x7f537400dc00 comp rx=0 tx=0).stop 2024-05-28T06:12:18.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.158+0000 7f5382600640 1 -- 172.21.15.96:0/1760447801 shutdown_connections 2024-05-28T06:12:18.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.158+0000 7f5382600640 1 --2- 172.21.15.96:0/1760447801 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f536006d100 0x7f536006f5c0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:18.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.158+0000 7f5382600640 1 --2- 172.21.15.96:0/1760447801 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f537c151fd0 0x7f537c073010 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:18.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.158+0000 7f5382600640 1 --2- 172.21.15.96:0/1760447801 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f537c151b50 0x7f537c072ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:18.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.158+0000 7f5382600640 1 --2- 172.21.15.96:0/1760447801 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5364003f60 0x7f537c0cf090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:18.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.158+0000 7f5382600640 1 -- 172.21.15.96:0/1760447801 >> 172.21.15.96:0/1760447801 conn(0x7f537c06b900 msgr2=0x7f537c06ee20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:18.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.158+0000 7f5382600640 1 -- 172.21.15.96:0/1760447801 shutdown_connections 2024-05-28T06:12:18.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.158+0000 7f5382600640 1 -- 172.21.15.96:0/1760447801 wait complete. 2024-05-28T06:12:18.179 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestCloneProgressReporter.test_progress_bars_for_4_clones' 2024-05-28T06:12:18.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.371+0000 7fa065e00640 1 Processor -- start 2024-05-28T06:12:18.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.371+0000 7fa065e00640 1 -- start start 2024-05-28T06:12:18.373 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.372+0000 7fa065e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fa06006dc20 0x7fa0600b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:18.373 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.372+0000 7fa065e00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0600afd60 con 0x7fa060073310 2024-05-28T06:12:18.374 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.372+0000 7fa065e00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0600af7a0 con 0x7fa06015b3a0 2024-05-28T06:12:18.374 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.372+0000 7fa065e00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fa06006f980 con 0x7fa06006dc20 2024-05-28T06:12:18.374 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.372+0000 7fa065400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fa06006dc20 0x7fa0600b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:18.374 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.372+0000 7fa057e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fa060073310 0x7fa0600ae550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:59348/0 (socket says 172.21.15.96:59348) 2024-05-28T06:12:18.374 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.372+0000 7fa057e00640 1 -- 172.21.15.96:0/3249410920 learned_addr learned my addr 172.21.15.96:0/3249410920 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:12:18.375 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.373+0000 7fa065400640 1 -- 172.21.15.96:0/3249410920 >> v1:172.21.15.123:6789/0 conn(0x7fa06015b3a0 legacy=0x7fa060072c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:18.375 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.373+0000 7fa065400640 1 -- 172.21.15.96:0/3249410920 >> v1:172.21.15.96:6789/0 conn(0x7fa060073310 legacy=0x7fa0600ae550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:18.375 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.373+0000 7fa065400640 1 -- 172.21.15.96:0/3249410920 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0600afa80 con 0x7fa06006dc20 2024-05-28T06:12:18.375 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.373+0000 7fa05fe00640 1 -- 172.21.15.96:0/3249410920 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 330643419 0 0) 0x7fa0600afd60 con 0x7fa060073310 2024-05-28T06:12:18.375 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.374+0000 7fa065400640 1 --2- 172.21.15.96:0/3249410920 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa06006dc20 0x7fa0600b32c0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fa058009d90 tx=0x7fa05802a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ba777ab50a7d5c server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:18.375 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.374+0000 7fa05fe00640 1 -- 172.21.15.96:0/3249410920 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa058007740 con 0x7fa06006dc20 2024-05-28T06:12:18.376 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.374+0000 7fa05fe00640 1 -- 172.21.15.96:0/3249410920 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0580078e0 con 0x7fa06006dc20 2024-05-28T06:12:18.376 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.374+0000 7fa05fe00640 1 -- 172.21.15.96:0/3249410920 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa058007cb0 con 0x7fa06006dc20 2024-05-28T06:12:18.376 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.375+0000 7fa065e00640 1 -- 172.21.15.96:0/3249410920 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa06006dc20 msgr2=0x7fa0600b32c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:18.376 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.375+0000 7fa065e00640 1 --2- 172.21.15.96:0/3249410920 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa06006dc20 0x7fa0600b32c0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fa058009d90 tx=0x7fa05802a6a0 comp rx=0 tx=0).stop 2024-05-28T06:12:18.377 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.375+0000 7fa065e00640 1 -- 172.21.15.96:0/3249410920 shutdown_connections 2024-05-28T06:12:18.377 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.375+0000 7fa065e00640 1 --2- 172.21.15.96:0/3249410920 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa06006dc20 0x7fa0600b32c0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:18.377 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.375+0000 7fa065e00640 1 -- 172.21.15.96:0/3249410920 >> 172.21.15.96:0/3249410920 conn(0x7fa0600b1fc0 msgr2=0x7fa0600b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:18.377 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.375+0000 7fa065e00640 1 -- 172.21.15.96:0/3249410920 shutdown_connections 2024-05-28T06:12:18.377 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.375+0000 7fa065e00640 1 -- 172.21.15.96:0/3249410920 wait complete. 2024-05-28T06:12:18.378 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.377+0000 7fa065e00640 1 Processor -- start 2024-05-28T06:12:18.378 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.377+0000 7fa065e00640 1 -- start start 2024-05-28T06:12:18.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.378+0000 7fa065e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa06006dc20 0x7fa060137c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:18.380 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.378+0000 7fa065e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa060073310 0x7fa0600b1440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:18.380 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.378+0000 7fa065e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa06015b3a0 0x7fa0600cab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:18.380 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.378+0000 7fa065e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa06015f9a0 con 0x7fa06015b3a0 2024-05-28T06:12:18.380 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.378+0000 7fa065e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa0600ae0c0 con 0x7fa06006dc20 2024-05-28T06:12:18.380 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.378+0000 7fa065e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fa06015fb20 con 0x7fa060073310 2024-05-28T06:12:18.380 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.378+0000 7fa065400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa06015b3a0 0x7fa0600cab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:18.380 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.378+0000 7fa065400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa06015b3a0 0x7fa0600cab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:32882/0 (socket says 172.21.15.96:32882) 2024-05-28T06:12:18.380 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.378+0000 7fa065400640 1 -- 172.21.15.96:0/3949224139 learned_addr learned my addr 172.21.15.96:0/3949224139 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:18.380 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.378+0000 7fa064a00640 1 --2- 172.21.15.96:0/3949224139 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa06006dc20 0x7fa060137c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:18.380 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.378+0000 7fa057e00640 1 --2- 172.21.15.96:0/3949224139 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa060073310 0x7fa0600b1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:18.381 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.379+0000 7fa065400640 1 -- 172.21.15.96:0/3949224139 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa060073310 msgr2=0x7fa0600b1440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:18.381 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.379+0000 7fa065400640 1 --2- 172.21.15.96:0/3949224139 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa060073310 0x7fa0600b1440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:18.381 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.379+0000 7fa065400640 1 -- 172.21.15.96:0/3949224139 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa06006dc20 msgr2=0x7fa060137c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:18.381 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.379+0000 7fa065400640 1 --2- 172.21.15.96:0/3949224139 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa06006dc20 0x7fa060137c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:18.382 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.379+0000 7fa065400640 1 -- 172.21.15.96:0/3949224139 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0600cb500 con 0x7fa06015b3a0 2024-05-28T06:12:18.382 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.379+0000 7fa057e00640 1 --2- 172.21.15.96:0/3949224139 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa060073310 0x7fa0600b1440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:12:18.382 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.379+0000 7fa065400640 1 --2- 172.21.15.96:0/3949224139 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa06015b3a0 0x7fa0600cab20 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fa058009d90 tx=0x7fa058004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:18.382 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.379+0000 7fa05d600640 1 -- 172.21.15.96:0/3949224139 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa058040070 con 0x7fa06015b3a0 2024-05-28T06:12:18.382 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.379+0000 7fa05d600640 1 -- 172.21.15.96:0/3949224139 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa058031c40 con 0x7fa06015b3a0 2024-05-28T06:12:18.382 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.379+0000 7fa05d600640 1 -- 172.21.15.96:0/3949224139 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa058009450 con 0x7fa06015b3a0 2024-05-28T06:12:18.382 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.379+0000 7fa065e00640 1 -- 172.21.15.96:0/3949224139 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa06015ba70 con 0x7fa06015b3a0 2024-05-28T06:12:18.382 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.379+0000 7fa065e00640 1 -- 172.21.15.96:0/3949224139 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0600cb7e0 con 0x7fa06015b3a0 2024-05-28T06:12:18.383 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.380+0000 7fa065e00640 1 -- 172.21.15.96:0/3949224139 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0600689e0 con 0x7fa06015b3a0 2024-05-28T06:12:18.383 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.381+0000 7fa05d600640 1 -- 172.21.15.96:0/3949224139 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 16) ==== 91754+0+0 (secure 0 0 0) 0x7fa058039040 con 0x7fa06015b3a0 2024-05-28T06:12:18.384 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.382+0000 7fa05d600640 1 --2- 172.21.15.96:0/3949224139 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa03c06d0d0 0x7fa03c06f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:18.384 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.382+0000 7fa05d600640 1 -- 172.21.15.96:0/3949224139 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 4971+0+0 (secure 0 0 0) 0x7fa0580adc00 con 0x7fa06015b3a0 2024-05-28T06:12:18.384 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.383+0000 7fa064a00640 1 --2- 172.21.15.96:0/3949224139 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa03c06d0d0 0x7fa03c06f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:18.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.384+0000 7fa064a00640 1 --2- 172.21.15.96:0/3949224139 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa03c06d0d0 0x7fa03c06f590 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fa048006fd0 tx=0x7fa048008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:18.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.387+0000 7fa05d600640 1 -- 172.21.15.96:0/3949224139 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa0580afe20 con 0x7fa06015b3a0 2024-05-28T06:12:18.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.649+0000 7fa065e00640 1 -- 172.21.15.96:0/3949224139 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgressReporter.test_progress_bars_for_4_clones"]} v 0) -- 0x7fa0600737b0 con 0x7fa06015b3a0 2024-05-28T06:12:18.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.703+0000 7fa05d600640 1 -- 172.21.15.96:0/3949224139 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgressReporter.test_progress_bars_for_4_clones"]}]=0 v358) ==== 162+0+0 (secure 0 0 0) 0x7fa05807b1e0 con 0x7fa06015b3a0 2024-05-28T06:12:18.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.707+0000 7fa065e00640 1 -- 172.21.15.96:0/3949224139 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa03c06d0d0 msgr2=0x7fa03c06f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:18.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.707+0000 7fa065e00640 1 --2- 172.21.15.96:0/3949224139 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa03c06d0d0 0x7fa03c06f590 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fa048006fd0 tx=0x7fa048008040 comp rx=0 tx=0).stop 2024-05-28T06:12:18.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.707+0000 7fa065e00640 1 -- 172.21.15.96:0/3949224139 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa06015b3a0 msgr2=0x7fa0600cab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:18.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.707+0000 7fa065e00640 1 --2- 172.21.15.96:0/3949224139 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa06015b3a0 0x7fa0600cab20 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fa058009d90 tx=0x7fa058004a40 comp rx=0 tx=0).stop 2024-05-28T06:12:18.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.707+0000 7fa065e00640 1 -- 172.21.15.96:0/3949224139 shutdown_connections 2024-05-28T06:12:18.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.707+0000 7fa065e00640 1 --2- 172.21.15.96:0/3949224139 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa03c06d0d0 0x7fa03c06f590 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:18.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.707+0000 7fa065e00640 1 --2- 172.21.15.96:0/3949224139 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa06015b3a0 0x7fa0600cab20 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:18.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.707+0000 7fa065e00640 1 --2- 172.21.15.96:0/3949224139 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa060073310 0x7fa0600b1440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:18.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.707+0000 7fa065e00640 1 --2- 172.21.15.96:0/3949224139 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa06006dc20 0x7fa060137c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:18.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.707+0000 7fa065e00640 1 -- 172.21.15.96:0/3949224139 >> 172.21.15.96:0/3949224139 conn(0x7fa0600b1fc0 msgr2=0x7fa0600725f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:18.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.707+0000 7fa065e00640 1 -- 172.21.15.96:0/3949224139 shutdown_connections 2024-05-28T06:12:18.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.707+0000 7fa065e00640 1 -- 172.21.15.96:0/3949224139 wait complete. 2024-05-28T06:12:18.727 INFO:tasks.cephfs_test_runner:Test that one progress bar is printed when number of clone jobs is ... ok 2024-05-28T06:12:18.727 INFO:tasks.cephfs_test_runner:Starting test: test_progress_bars_for_7_clones (tasks.cephfs.test_volumes.TestCloneProgressReporter) Test that 2 progress bars are printed when number of clone jobs is 2024-05-28T06:12:18.727 INFO:tasks.cephfs_test_runner:test_progress_bars_for_7_clones (tasks.cephfs.test_volumes.TestCloneProgressReporter) 2024-05-28T06:12:18.727 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:12:18.922 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.922+0000 7f0deb200640 1 Processor -- start 2024-05-28T06:12:18.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.922+0000 7f0deb200640 1 -- start start 2024-05-28T06:12:18.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.923+0000 7f0deb200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0de4156b30 0x7f0de40af6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:18.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.923+0000 7f0deb200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0de4156f10 con 0x7f0de40ad940 2024-05-28T06:12:18.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.923+0000 7f0deb200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0de406dda0 con 0x7f0de406ee40 2024-05-28T06:12:18.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.923+0000 7f0deb200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f0de406e080 con 0x7f0de4156b30 2024-05-28T06:12:18.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.923+0000 7f0dea800640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f0de40ad940 0x7f0de40b32c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:60062/0 (socket says 172.21.15.96:60062) 2024-05-28T06:12:18.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.923+0000 7f0dea800640 1 -- 172.21.15.96:0/2751798762 learned_addr learned my addr 172.21.15.96:0/2751798762 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:12:18.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.924+0000 7f0de8a00640 1 -- 172.21.15.96:0/2751798762 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3803558169 0 0) 0x7f0de4156f10 con 0x7f0de40ad940 2024-05-28T06:12:18.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.924+0000 7f0de8a00640 1 -- 172.21.15.96:0/2751798762 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0dcc003520 con 0x7f0de40ad940 2024-05-28T06:12:18.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.924+0000 7f0de9e00640 1 --2- 172.21.15.96:0/2751798762 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0de4156b30 0x7f0de40af6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:18.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.924+0000 7f0de8a00640 1 -- 172.21.15.96:0/2751798762 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3754148892 0 0) 0x7f0dcc003520 con 0x7f0de40ad940 2024-05-28T06:12:18.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.924+0000 7f0de8a00640 1 -- 172.21.15.96:0/2751798762 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0de4156b30 msgr2=0x7f0de40af6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:18.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.924+0000 7f0de8a00640 1 --2- 172.21.15.96:0/2751798762 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0de4156b30 0x7f0de40af6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:18.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.924+0000 7f0de8a00640 1 -- 172.21.15.96:0/2751798762 >> v1:172.21.15.123:6789/0 conn(0x7f0de406ee40 legacy=0x7f0de40ad230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:18.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.924+0000 7f0de8a00640 1 -- 172.21.15.96:0/2751798762 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0de406e360 con 0x7f0de40ad940 2024-05-28T06:12:18.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.924+0000 7f0de8a00640 1 -- 172.21.15.96:0/2751798762 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f0de00019f0 con 0x7f0de40ad940 2024-05-28T06:12:18.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.924+0000 7f0de8a00640 1 -- 172.21.15.96:0/2751798762 >> v1:172.21.15.96:6789/0 conn(0x7f0de40ad940 legacy=0x7f0de40b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:18.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.924+0000 7f0de8a00640 1 --2- 172.21.15.96:0/2751798762 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0dcc0039e0 0x7f0dcc003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:18.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.924+0000 7f0de8a00640 1 --2- 172.21.15.96:0/2751798762 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0de4156b30 0x7f0dcc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:18.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.924+0000 7f0de8a00640 1 --2- 172.21.15.96:0/2751798762 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0dcc004d40 0x7f0dcc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:18.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.924+0000 7f0de8a00640 1 -- 172.21.15.96:0/2751798762 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f0de406e080 con 0x7f0dcc0039e0 2024-05-28T06:12:18.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.924+0000 7f0de8a00640 1 -- 172.21.15.96:0/2751798762 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f0dcc00bbd0 con 0x7f0dcc004d40 2024-05-28T06:12:18.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.924+0000 7f0de8a00640 1 -- 172.21.15.96:0/2751798762 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f0dcc00bdd0 con 0x7f0de4156b30 2024-05-28T06:12:18.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.924+0000 7f0de9e00640 1 --2- 172.21.15.96:0/2751798762 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0dcc0039e0 0x7f0dcc003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:18.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.925+0000 7f0de9e00640 1 -- 172.21.15.96:0/2751798762 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0de4156b30 msgr2=0x7f0dcc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:18.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.925+0000 7f0de9e00640 1 --2- 172.21.15.96:0/2751798762 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0de4156b30 0x7f0dcc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:18.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.925+0000 7f0de9e00640 1 -- 172.21.15.96:0/2751798762 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0dcc004d40 msgr2=0x7f0dcc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:18.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.925+0000 7f0de9e00640 1 --2- 172.21.15.96:0/2751798762 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0dcc004d40 0x7f0dcc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:18.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.925+0000 7f0de9e00640 1 -- 172.21.15.96:0/2751798762 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dcc003520 con 0x7f0dcc0039e0 2024-05-28T06:12:18.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.926+0000 7f0de9e00640 1 --2- 172.21.15.96:0/2751798762 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0dcc0039e0 0x7f0dcc003de0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f0dd800ba00 tx=0x7f0dcc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=387711694054c237 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:18.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.926+0000 7f0de8a00640 1 -- 172.21.15.96:0/2751798762 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dd8030070 con 0x7f0dcc0039e0 2024-05-28T06:12:18.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.926+0000 7f0de8a00640 1 -- 172.21.15.96:0/2751798762 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0dd802ade0 con 0x7f0dcc0039e0 2024-05-28T06:12:18.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.926+0000 7f0de8a00640 1 -- 172.21.15.96:0/2751798762 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dd800b600 con 0x7f0dcc0039e0 2024-05-28T06:12:18.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.926+0000 7f0deb200640 1 -- 172.21.15.96:0/2751798762 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0dcc0039e0 msgr2=0x7f0dcc003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:18.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.926+0000 7f0deb200640 1 --2- 172.21.15.96:0/2751798762 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0dcc0039e0 0x7f0dcc003de0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f0dd800ba00 tx=0x7f0dcc005830 comp rx=0 tx=0).stop 2024-05-28T06:12:18.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.927+0000 7f0dea800640 1 -- 172.21.15.96:0/2751798762 reap_dead start 2024-05-28T06:12:18.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.927+0000 7f0deb200640 1 -- 172.21.15.96:0/2751798762 shutdown_connections 2024-05-28T06:12:18.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.927+0000 7f0deb200640 1 -- 172.21.15.96:0/2751798762 >> 172.21.15.96:0/2751798762 conn(0x7f0de40b1fc0 msgr2=0x7f0de40b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:18.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.927+0000 7f0deb200640 1 -- 172.21.15.96:0/2751798762 shutdown_connections 2024-05-28T06:12:18.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.927+0000 7f0deb200640 1 -- 172.21.15.96:0/2751798762 wait complete. 2024-05-28T06:12:18.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.928+0000 7f0deb200640 1 Processor -- start 2024-05-28T06:12:18.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.929+0000 7f0deb200640 1 -- start start 2024-05-28T06:12:18.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.929+0000 7f0deb200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0dcc0039e0 0x7f0de414d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:18.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.929+0000 7f0deb200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0de409a4d0 0x7f0de4158470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:18.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.929+0000 7f0deb200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0de40b00f0 0x7f0de40b0570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:18.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.930+0000 7f0deb200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f0de40b2db0 con 0x7f0de409a4d0 2024-05-28T06:12:18.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.930+0000 7f0deb200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f0de40b2c30 con 0x7f0dcc0039e0 2024-05-28T06:12:18.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.930+0000 7f0deb200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f0de40b39d0 con 0x7f0de40b00f0 2024-05-28T06:12:18.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.930+0000 7f0de9400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0de409a4d0 0x7f0de4158470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:18.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.930+0000 7f0de9400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0de409a4d0 0x7f0de4158470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44000/0 (socket says 172.21.15.96:44000) 2024-05-28T06:12:18.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.930+0000 7f0de9400640 1 -- 172.21.15.96:0/3528898952 learned_addr learned my addr 172.21.15.96:0/3528898952 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:18.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.930+0000 7f0de9400640 1 -- 172.21.15.96:0/3528898952 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0de40b00f0 msgr2=0x7f0de40b0570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:18.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.930+0000 7f0de9400640 1 --2- 172.21.15.96:0/3528898952 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0de40b00f0 0x7f0de40b0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:18.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.930+0000 7f0de9400640 1 -- 172.21.15.96:0/3528898952 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0dcc0039e0 msgr2=0x7f0de414d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:18.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.930+0000 7f0de9e00640 1 --2- 172.21.15.96:0/3528898952 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0dcc0039e0 0x7f0de414d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:18.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.930+0000 7f0de9400640 1 --2- 172.21.15.96:0/3528898952 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0dcc0039e0 0x7f0de414d680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:18.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.930+0000 7f0de9400640 1 -- 172.21.15.96:0/3528898952 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0de414dbc0 con 0x7f0de409a4d0 2024-05-28T06:12:18.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.930+0000 7f0de9400640 1 --2- 172.21.15.96:0/3528898952 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0de409a4d0 0x7f0de4158470 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f0dd400c480 tx=0x7f0dd400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:18.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.931+0000 7f0dde000640 1 -- 172.21.15.96:0/3528898952 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dd4016020 con 0x7f0de409a4d0 2024-05-28T06:12:18.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.931+0000 7f0dde000640 1 -- 172.21.15.96:0/3528898952 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0dd4012830 con 0x7f0de409a4d0 2024-05-28T06:12:18.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.931+0000 7f0dde000640 1 -- 172.21.15.96:0/3528898952 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dd4012c60 con 0x7f0de409a4d0 2024-05-28T06:12:18.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.931+0000 7f0deb200640 1 -- 172.21.15.96:0/3528898952 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0de4158d00 con 0x7f0de409a4d0 2024-05-28T06:12:18.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.931+0000 7f0deb200640 1 -- 172.21.15.96:0/3528898952 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0de41592c0 con 0x7f0de409a4d0 2024-05-28T06:12:18.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.932+0000 7f0deb200640 1 -- 172.21.15.96:0/3528898952 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0de40689e0 con 0x7f0de409a4d0 2024-05-28T06:12:18.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.933+0000 7f0dde000640 1 -- 172.21.15.96:0/3528898952 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 16) ==== 91754+0+0 (secure 0 0 0) 0x7f0dd4019020 con 0x7f0de409a4d0 2024-05-28T06:12:18.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.934+0000 7f0dde000640 1 --2- 172.21.15.96:0/3528898952 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0dc806d100 0x7f0dc806f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:18.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.934+0000 7f0dde000640 1 -- 172.21.15.96:0/3528898952 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 4971+0+0 (secure 0 0 0) 0x7f0dd408c810 con 0x7f0de409a4d0 2024-05-28T06:12:18.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.935+0000 7f0de9e00640 1 --2- 172.21.15.96:0/3528898952 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0dc806d100 0x7f0dc806f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:18.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.936+0000 7f0de9e00640 1 --2- 172.21.15.96:0/3528898952 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0dc806d100 0x7f0dc806f5c0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f0dcc004780 tx=0x7f0dd8030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:18.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:18.939+0000 7f0dde000640 1 -- 172.21.15.96:0/3528898952 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0dd4020030 con 0x7f0de409a4d0 2024-05-28T06:12:19.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.139+0000 7f0deb200640 1 -- 172.21.15.96:0/3528898952 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0de406f7f0 con 0x7f0de409a4d0 2024-05-28T06:12:19.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.140+0000 7f0dde000640 1 -- 172.21.15.96:0/3528898952 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v51) ==== 74+0+13073 (secure 0 0 0) 0x7f0dd40182c0 con 0x7f0de409a4d0 2024-05-28T06:12:19.141 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:12:19.141 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":51,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:12:17.689620+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.96:6835/935552515":"2024-05-29T06:12:14.283885+0000","172.21.15.96:6833/935552515":"2024-05-29T06:12:14.283885+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:12:19.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.143+0000 7f0deb200640 1 -- 172.21.15.96:0/3528898952 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0dc806d100 msgr2=0x7f0dc806f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:19.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.143+0000 7f0deb200640 1 --2- 172.21.15.96:0/3528898952 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0dc806d100 0x7f0dc806f5c0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f0dcc004780 tx=0x7f0dd8030000 comp rx=0 tx=0).stop 2024-05-28T06:12:19.145 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.143+0000 7f0deb200640 1 -- 172.21.15.96:0/3528898952 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0de409a4d0 msgr2=0x7f0de4158470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:19.145 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.143+0000 7f0deb200640 1 --2- 172.21.15.96:0/3528898952 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0de409a4d0 0x7f0de4158470 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f0dd400c480 tx=0x7f0dd400c940 comp rx=0 tx=0).stop 2024-05-28T06:12:19.145 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.143+0000 7f0deb200640 1 -- 172.21.15.96:0/3528898952 shutdown_connections 2024-05-28T06:12:19.145 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.143+0000 7f0deb200640 1 --2- 172.21.15.96:0/3528898952 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0dc806d100 0x7f0dc806f5c0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:19.145 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.143+0000 7f0deb200640 1 --2- 172.21.15.96:0/3528898952 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0de40b00f0 0x7f0de40b0570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:19.145 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.143+0000 7f0deb200640 1 --2- 172.21.15.96:0/3528898952 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0de409a4d0 0x7f0de4158470 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:19.145 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.143+0000 7f0deb200640 1 --2- 172.21.15.96:0/3528898952 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0dcc0039e0 0x7f0de414d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:19.145 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.143+0000 7f0deb200640 1 -- 172.21.15.96:0/3528898952 >> 172.21.15.96:0/3528898952 conn(0x7f0de40b1fc0 msgr2=0x7f0de40b4a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:19.145 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.143+0000 7f0deb200640 1 -- 172.21.15.96:0/3528898952 shutdown_connections 2024-05-28T06:12:19.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.144+0000 7f0deb200640 1 -- 172.21.15.96:0/3528898952 wait complete. 2024-05-28T06:12:19.164 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-28T06:03:59.517102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-28T06:12:19.165 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-28T06:12:19.354 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.353+0000 7f3265600640 1 Processor -- start 2024-05-28T06:12:19.354 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.353+0000 7f3265600640 1 -- start start 2024-05-28T06:12:19.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.355+0000 7f3265600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f32600d0640 0x7f326015d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:19.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.355+0000 7f3265600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f32600b32c0 0x7f32600b3750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:19.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.355+0000 7f3265600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32600d0060 con 0x7f32600d0a20 2024-05-28T06:12:19.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.355+0000 7f3265600640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f32600adbb0 con 0x7f32600b32c0 2024-05-28T06:12:19.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.355+0000 7f3265600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f326015db10 con 0x7f32600d0640 2024-05-28T06:12:19.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.355+0000 7f325f400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f32600b32c0 0x7f32600b3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:19.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.355+0000 7f325f400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f32600b32c0 0x7f32600b3750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44016/0 (socket says 172.21.15.96:44016) 2024-05-28T06:12:19.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.355+0000 7f325f400640 1 -- 172.21.15.96:0/2248095997 learned_addr learned my addr 172.21.15.96:0/2248095997 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:19.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.356+0000 7f325ea00640 1 --2- 172.21.15.96:0/2248095997 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f32600d0640 0x7f326015d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:19.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.356+0000 7f325f400640 1 -- 172.21.15.96:0/2248095997 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f32600d0640 msgr2=0x7f326015d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:19.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.356+0000 7f325f400640 1 --2- 172.21.15.96:0/2248095997 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f32600d0640 0x7f326015d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:19.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.356+0000 7f325f400640 1 -- 172.21.15.96:0/2248095997 >> v1:172.21.15.123:6789/0 conn(0x7f32600d0a20 legacy=0x7f32600cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:19.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.356+0000 7f325f400640 1 -- 172.21.15.96:0/2248095997 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f326015cb10 con 0x7f32600b32c0 2024-05-28T06:12:19.358 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.357+0000 7f325f400640 1 --2- 172.21.15.96:0/2248095997 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f32600b32c0 0x7f32600b3750 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f3258009870 tx=0x7f325802a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99d9ed76fd87c4f server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:19.358 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.357+0000 7f325e000640 1 -- 172.21.15.96:0/2248095997 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3258037070 con 0x7f32600b32c0 2024-05-28T06:12:19.358 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.358+0000 7f325e000640 1 -- 172.21.15.96:0/2248095997 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3258032440 con 0x7f32600b32c0 2024-05-28T06:12:19.358 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.358+0000 7f325e000640 1 -- 172.21.15.96:0/2248095997 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3258032810 con 0x7f32600b32c0 2024-05-28T06:12:19.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.358+0000 7f3265600640 1 -- 172.21.15.96:0/2248095997 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f32600b32c0 msgr2=0x7f32600b3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:19.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.358+0000 7f3265600640 1 --2- 172.21.15.96:0/2248095997 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f32600b32c0 0x7f32600b3750 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f3258009870 tx=0x7f325802a210 comp rx=0 tx=0).stop 2024-05-28T06:12:19.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.358+0000 7f3265600640 1 -- 172.21.15.96:0/2248095997 shutdown_connections 2024-05-28T06:12:19.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.358+0000 7f3265600640 1 --2- 172.21.15.96:0/2248095997 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f32600b32c0 0x7f32600b3750 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:19.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.358+0000 7f3265600640 1 --2- 172.21.15.96:0/2248095997 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f32600d0640 0x7f326015d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:19.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.358+0000 7f3265600640 1 -- 172.21.15.96:0/2248095997 >> 172.21.15.96:0/2248095997 conn(0x7f326006bd20 msgr2=0x7f326006c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:19.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.358+0000 7f3265600640 1 -- 172.21.15.96:0/2248095997 shutdown_connections 2024-05-28T06:12:19.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.359+0000 7f3265600640 1 -- 172.21.15.96:0/2248095997 wait complete. 2024-05-28T06:12:19.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.360+0000 7f3265600640 1 Processor -- start 2024-05-28T06:12:19.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.360+0000 7f3265600640 1 -- start start 2024-05-28T06:12:19.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.361+0000 7f3265600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32600b32c0 0x7f326009a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:19.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.361+0000 7f3265600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f32600d0640 0x7f326014c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:19.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.361+0000 7f3265600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f32600d0a20 0x7f3260072c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:19.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.361+0000 7f3265600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f326015cdd0 con 0x7f32600d0a20 2024-05-28T06:12:19.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.361+0000 7f3265600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f32600ad8d0 con 0x7f32600b32c0 2024-05-28T06:12:19.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.361+0000 7f3265600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f326015d0b0 con 0x7f32600d0640 2024-05-28T06:12:19.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.361+0000 7f325f400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f32600d0a20 0x7f3260072c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:19.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.362+0000 7f325f400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f32600d0a20 0x7f3260072c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44026/0 (socket says 172.21.15.96:44026) 2024-05-28T06:12:19.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.362+0000 7f325f400640 1 -- 172.21.15.96:0/2792383602 learned_addr learned my addr 172.21.15.96:0/2792383602 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:19.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.362+0000 7f325ea00640 1 --2- 172.21.15.96:0/2792383602 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32600b32c0 0x7f326009a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:19.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.362+0000 7f3256000640 1 --2- 172.21.15.96:0/2792383602 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f32600d0640 0x7f326014c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:19.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.362+0000 7f325f400640 1 -- 172.21.15.96:0/2792383602 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f32600d0640 msgr2=0x7f326014c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:19.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.362+0000 7f325f400640 1 --2- 172.21.15.96:0/2792383602 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f32600d0640 0x7f326014c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:19.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.362+0000 7f325f400640 1 -- 172.21.15.96:0/2792383602 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32600b32c0 msgr2=0x7f326009a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:19.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.362+0000 7f325f400640 1 --2- 172.21.15.96:0/2792383602 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32600b32c0 0x7f326009a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:19.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.362+0000 7f325f400640 1 -- 172.21.15.96:0/2792383602 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f326009ae10 con 0x7f32600d0a20 2024-05-28T06:12:19.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.362+0000 7f3256000640 1 --2- 172.21.15.96:0/2792383602 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f32600d0640 0x7f326014c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:12:19.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.362+0000 7f325f400640 1 --2- 172.21.15.96:0/2792383602 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f32600d0a20 0x7f3260072c20 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f325802a250 tx=0x7f32580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:19.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.363+0000 7f3257400640 1 -- 172.21.15.96:0/2792383602 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3258041020 con 0x7f32600d0a20 2024-05-28T06:12:19.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.363+0000 7f3257400640 1 -- 172.21.15.96:0/2792383602 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32580306c0 con 0x7f32600d0a20 2024-05-28T06:12:19.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.363+0000 7f3265600640 1 -- 172.21.15.96:0/2792383602 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f326009b0f0 con 0x7f32600d0a20 2024-05-28T06:12:19.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.363+0000 7f3257400640 1 -- 172.21.15.96:0/2792383602 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3258037040 con 0x7f32600d0a20 2024-05-28T06:12:19.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.363+0000 7f3265600640 1 -- 172.21.15.96:0/2792383602 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32600cfdf0 con 0x7f32600d0a20 2024-05-28T06:12:19.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.364+0000 7f3265600640 1 -- 172.21.15.96:0/2792383602 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32600ad2a0 con 0x7f32600d0a20 2024-05-28T06:12:19.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.365+0000 7f3257400640 1 -- 172.21.15.96:0/2792383602 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 16) ==== 91754+0+0 (secure 0 0 0) 0x7f3258044020 con 0x7f32600d0a20 2024-05-28T06:12:19.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.366+0000 7f3257400640 1 --2- 172.21.15.96:0/2792383602 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f323c06d0d0 0x7f323c06f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:19.367 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.366+0000 7f3257400640 1 -- 172.21.15.96:0/2792383602 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 4971+0+0 (secure 0 0 0) 0x7f32580ad300 con 0x7f32600d0a20 2024-05-28T06:12:19.367 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.367+0000 7f325ea00640 1 --2- 172.21.15.96:0/2792383602 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f323c06d0d0 0x7f323c06f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:19.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.369+0000 7f325ea00640 1 --2- 172.21.15.96:0/2792383602 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f323c06d0d0 0x7f323c06f590 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f3250009a60 tx=0x7f3250003990 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:19.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.370+0000 7f3257400640 1 -- 172.21.15.96:0/2792383602 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f32600ad2a0 con 0x7f32600d0a20 2024-05-28T06:12:19.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.569+0000 7f3265600640 1 -- 172.21.15.96:0/2792383602 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f3260071ef0 con 0x7f32600d0a20 2024-05-28T06:12:19.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.570+0000 7f3257400640 1 -- 172.21.15.96:0/2792383602 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v51) ==== 93+0+10 (secure 0 0 0) 0x7f325804b030 con 0x7f32600d0a20 2024-05-28T06:12:19.571 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 1 2024-05-28T06:12:19.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.572+0000 7f3265600640 1 -- 172.21.15.96:0/2792383602 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f323c06d0d0 msgr2=0x7f323c06f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:19.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.572+0000 7f3265600640 1 --2- 172.21.15.96:0/2792383602 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f323c06d0d0 0x7f323c06f590 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f3250009a60 tx=0x7f3250003990 comp rx=0 tx=0).stop 2024-05-28T06:12:19.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.572+0000 7f3265600640 1 -- 172.21.15.96:0/2792383602 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f32600d0a20 msgr2=0x7f3260072c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:19.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.572+0000 7f3265600640 1 --2- 172.21.15.96:0/2792383602 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f32600d0a20 0x7f3260072c20 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f325802a250 tx=0x7f32580046c0 comp rx=0 tx=0).stop 2024-05-28T06:12:19.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.573+0000 7f3265600640 1 -- 172.21.15.96:0/2792383602 shutdown_connections 2024-05-28T06:12:19.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.573+0000 7f3265600640 1 --2- 172.21.15.96:0/2792383602 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f323c06d0d0 0x7f323c06f590 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:19.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.573+0000 7f3265600640 1 --2- 172.21.15.96:0/2792383602 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f32600d0a20 0x7f3260072c20 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:19.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.573+0000 7f3265600640 1 --2- 172.21.15.96:0/2792383602 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f32600d0640 0x7f326014c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:19.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.573+0000 7f3265600640 1 --2- 172.21.15.96:0/2792383602 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32600b32c0 0x7f326009a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:19.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.573+0000 7f3265600640 1 -- 172.21.15.96:0/2792383602 >> 172.21.15.96:0/2792383602 conn(0x7f326006bd20 msgr2=0x7f32601505e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:19.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.573+0000 7f3265600640 1 -- 172.21.15.96:0/2792383602 shutdown_connections 2024-05-28T06:12:19.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.573+0000 7f3265600640 1 -- 172.21.15.96:0/2792383602 wait complete. 2024-05-28T06:12:19.592 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestCloneProgressReporter.test_progress_bars_for_7_clones' 2024-05-28T06:12:19.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.790+0000 7f8170c00640 1 Processor -- start 2024-05-28T06:12:19.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.791+0000 7f8170c00640 1 -- start start 2024-05-28T06:12:19.793 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.792+0000 7f8170c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f816c156b30 0x7f816c0af6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:19.793 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.792+0000 7f8170c00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f816c0ad940 0x7f816c0b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:19.793 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.792+0000 7f8170c00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f816c156f10 con 0x7f816c06ee40 2024-05-28T06:12:19.793 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.792+0000 7f8170c00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f816c06dda0 con 0x7f816c0ad940 2024-05-28T06:12:19.793 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.792+0000 7f8170c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f816c06e080 con 0x7f816c156b30 2024-05-28T06:12:19.793 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.792+0000 7f816aa00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f816c06ee40 0x7f816c0ad230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:60076/0 (socket says 172.21.15.96:60076) 2024-05-28T06:12:19.793 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.792+0000 7f816aa00640 1 -- 172.21.15.96:0/1853903035 learned_addr learned my addr 172.21.15.96:0/1853903035 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:12:19.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.793+0000 7f816be00640 1 --2- 172.21.15.96:0/1853903035 >> v2:172.21.15.123:3300/0 conn(0x7f816c0ad940 0x7f816c0b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:19.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.793+0000 7f816b400640 1 --2- 172.21.15.96:0/1853903035 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f816c156b30 0x7f816c0af6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:19.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.793+0000 7f816a000640 1 -- 172.21.15.96:0/1853903035 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3532615732 0 0) 0x7f816c156f10 con 0x7f816c06ee40 2024-05-28T06:12:19.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.793+0000 7f816a000640 1 -- 172.21.15.96:0/1853903035 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8150003520 con 0x7f816c06ee40 2024-05-28T06:12:19.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.793+0000 7f816a000640 1 -- 172.21.15.96:0/1853903035 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2534275036 0 0) 0x7f8150003520 con 0x7f816c06ee40 2024-05-28T06:12:19.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.793+0000 7f816a000640 1 -- 172.21.15.96:0/1853903035 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f816c156b30 msgr2=0x7f816c0af6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:19.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.793+0000 7f816a000640 1 --2- 172.21.15.96:0/1853903035 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f816c156b30 0x7f816c0af6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:19.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.793+0000 7f816a000640 1 -- 172.21.15.96:0/1853903035 >> v2:172.21.15.123:3300/0 conn(0x7f816c0ad940 msgr2=0x7f816c0b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:19.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.793+0000 7f816a000640 1 --2- 172.21.15.96:0/1853903035 >> v2:172.21.15.123:3300/0 conn(0x7f816c0ad940 0x7f816c0b32c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:19.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.793+0000 7f816a000640 1 -- 172.21.15.96:0/1853903035 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f816c06e360 con 0x7f816c06ee40 2024-05-28T06:12:19.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.793+0000 7f816a000640 1 -- 172.21.15.96:0/1853903035 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f8158001670 con 0x7f816c06ee40 2024-05-28T06:12:19.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.793+0000 7f816be00640 1 --2- 172.21.15.96:0/1853903035 >> v2:172.21.15.123:3300/0 conn(0x7f816c0ad940 0x7f816c0b32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-28T06:12:19.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.793+0000 7f816a000640 1 -- 172.21.15.96:0/1853903035 >> v1:172.21.15.96:6789/0 conn(0x7f816c06ee40 legacy=0x7f816c0ad230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:19.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.793+0000 7f816a000640 1 --2- 172.21.15.96:0/1853903035 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f81500039b0 0x7f8150003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:19.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.793+0000 7f816a000640 1 --2- 172.21.15.96:0/1853903035 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f816c156b30 0x7f8150004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:19.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.793+0000 7f816a000640 1 --2- 172.21.15.96:0/1853903035 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8150004d40 0x7f81500051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:19.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.793+0000 7f816a000640 1 -- 172.21.15.96:0/1853903035 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f816c06dda0 con 0x7f8150004d40 2024-05-28T06:12:19.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.793+0000 7f816a000640 1 -- 172.21.15.96:0/1853903035 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f816c06e080 con 0x7f81500039b0 2024-05-28T06:12:19.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.793+0000 7f816a000640 1 -- 172.21.15.96:0/1853903035 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f815000baa0 con 0x7f816c156b30 2024-05-28T06:12:19.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.794+0000 7f816be00640 1 --2- 172.21.15.96:0/1853903035 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8150004d40 0x7f81500051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:19.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.794+0000 7f816b400640 1 --2- 172.21.15.96:0/1853903035 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f81500039b0 0x7f8150003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:19.797 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.794+0000 7f816aa00640 1 --2- 172.21.15.96:0/1853903035 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f816c156b30 0x7f8150004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:19.797 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.794+0000 7f816be00640 1 -- 172.21.15.96:0/1853903035 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f816c156b30 msgr2=0x7f8150004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:19.797 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.794+0000 7f816be00640 1 --2- 172.21.15.96:0/1853903035 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f816c156b30 0x7f8150004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:19.797 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.794+0000 7f816be00640 1 -- 172.21.15.96:0/1853903035 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f81500039b0 msgr2=0x7f8150003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:19.797 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.794+0000 7f816be00640 1 --2- 172.21.15.96:0/1853903035 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f81500039b0 0x7f8150003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:19.797 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.794+0000 7f816be00640 1 -- 172.21.15.96:0/1853903035 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8150003520 con 0x7f8150004d40 2024-05-28T06:12:19.798 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.795+0000 7f816be00640 1 --2- 172.21.15.96:0/1853903035 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8150004d40 0x7f81500051c0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f81640073b0 tx=0x7f8150005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5940675bc1b078e server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:19.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.795+0000 7f816a000640 1 -- 172.21.15.96:0/1853903035 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8164009c80 con 0x7f8150004d40 2024-05-28T06:12:19.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.795+0000 7f816a000640 1 -- 172.21.15.96:0/1853903035 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f816402c650 con 0x7f8150004d40 2024-05-28T06:12:19.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.795+0000 7f816a000640 1 -- 172.21.15.96:0/1853903035 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f816402ca00 con 0x7f8150004d40 2024-05-28T06:12:19.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.796+0000 7f8170c00640 1 -- 172.21.15.96:0/1853903035 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8150004d40 msgr2=0x7f81500051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:19.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.796+0000 7f8170c00640 1 --2- 172.21.15.96:0/1853903035 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8150004d40 0x7f81500051c0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f81640073b0 tx=0x7f8150005830 comp rx=0 tx=0).stop 2024-05-28T06:12:19.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.796+0000 7f816be00640 1 -- 172.21.15.96:0/1853903035 reap_dead start 2024-05-28T06:12:19.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.796+0000 7f8170c00640 1 -- 172.21.15.96:0/1853903035 shutdown_connections 2024-05-28T06:12:19.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.796+0000 7f8170c00640 1 -- 172.21.15.96:0/1853903035 >> 172.21.15.96:0/1853903035 conn(0x7f816c0b1fc0 msgr2=0x7f816c0b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:19.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.796+0000 7f8170c00640 1 -- 172.21.15.96:0/1853903035 shutdown_connections 2024-05-28T06:12:19.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.796+0000 7f8170c00640 1 -- 172.21.15.96:0/1853903035 wait complete. 2024-05-28T06:12:19.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.798+0000 7f8170c00640 1 Processor -- start 2024-05-28T06:12:19.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.798+0000 7f8170c00640 1 -- start start 2024-05-28T06:12:19.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.799+0000 7f8170c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f816c099910 0x7f816c15a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:19.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.799+0000 7f8170c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f816c153000 0x7f816c1549e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:19.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.799+0000 7f8170c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f816c152460 0x7f816c1528e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:19.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.799+0000 7f8170c00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f816c0b2db0 con 0x7f816c153000 2024-05-28T06:12:19.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.799+0000 7f8170c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f816c0b2c30 con 0x7f816c152460 2024-05-28T06:12:19.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.799+0000 7f8170c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f816c06f2c0 con 0x7f816c099910 2024-05-28T06:12:19.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.799+0000 7f816aa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f816c153000 0x7f816c1549e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:19.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.799+0000 7f816aa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f816c153000 0x7f816c1549e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44038/0 (socket says 172.21.15.96:44038) 2024-05-28T06:12:19.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.799+0000 7f816aa00640 1 -- 172.21.15.96:0/1697129929 learned_addr learned my addr 172.21.15.96:0/1697129929 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:19.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.799+0000 7f816aa00640 1 -- 172.21.15.96:0/1697129929 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f816c099910 msgr2=0x7f816c15a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:19.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.799+0000 7f816aa00640 1 --2- 172.21.15.96:0/1697129929 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f816c099910 0x7f816c15a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:19.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.799+0000 7f816aa00640 1 -- 172.21.15.96:0/1697129929 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f816c152460 msgr2=0x7f816c1528e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:19.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.799+0000 7f816aa00640 1 --2- 172.21.15.96:0/1697129929 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f816c152460 0x7f816c1528e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:19.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.799+0000 7f816aa00640 1 -- 172.21.15.96:0/1697129929 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f816c153940 con 0x7f816c153000 2024-05-28T06:12:19.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.800+0000 7f816aa00640 1 --2- 172.21.15.96:0/1697129929 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f816c153000 0x7f816c1549e0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f8158001050 tx=0x7f815800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:19.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.800+0000 7f8163400640 1 -- 172.21.15.96:0/1697129929 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f815800c850 con 0x7f816c153000 2024-05-28T06:12:19.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.800+0000 7f8163400640 1 -- 172.21.15.96:0/1697129929 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8158018070 con 0x7f816c153000 2024-05-28T06:12:19.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.800+0000 7f8163400640 1 -- 172.21.15.96:0/1697129929 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f815800cc30 con 0x7f816c153000 2024-05-28T06:12:19.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.800+0000 7f8170c00640 1 -- 172.21.15.96:0/1697129929 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f816c153c20 con 0x7f816c153000 2024-05-28T06:12:19.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.800+0000 7f8170c00640 1 -- 172.21.15.96:0/1697129929 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f816c1541e0 con 0x7f816c153000 2024-05-28T06:12:19.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.801+0000 7f8170c00640 1 -- 172.21.15.96:0/1697129929 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f816c0689e0 con 0x7f816c153000 2024-05-28T06:12:19.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.802+0000 7f8163400640 1 -- 172.21.15.96:0/1697129929 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 16) ==== 91754+0+0 (secure 0 0 0) 0x7f815801d460 con 0x7f816c153000 2024-05-28T06:12:19.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.803+0000 7f8163400640 1 --2- 172.21.15.96:0/1697129929 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f814c06d0d0 0x7f814c06f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:19.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.803+0000 7f8163400640 1 -- 172.21.15.96:0/1697129929 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 4971+0+0 (secure 0 0 0) 0x7f815808a500 con 0x7f816c153000 2024-05-28T06:12:19.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.804+0000 7f816b400640 1 --2- 172.21.15.96:0/1697129929 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f814c06d0d0 0x7f814c06f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:19.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.805+0000 7f816b400640 1 --2- 172.21.15.96:0/1697129929 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f814c06d0d0 0x7f814c06f590 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f815c009890 tx=0x7f815c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:19.810 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:19.810+0000 7f8163400640 1 -- 172.21.15.96:0/1697129929 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f815805a220 con 0x7f816c153000 2024-05-28T06:12:20.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.073+0000 7f8170c00640 1 -- 172.21.15.96:0/1697129929 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgressReporter.test_progress_bars_for_7_clones"]} v 0) -- 0x7f816c06f7f0 con 0x7f816c153000 2024-05-28T06:12:20.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.721+0000 7f8163400640 1 -- 172.21.15.96:0/1697129929 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgressReporter.test_progress_bars_for_7_clones"]}]=0 v360) ==== 165+0+0 (secure 0 0 0) 0x7f8158059b20 con 0x7f816c153000 2024-05-28T06:12:20.725 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.725+0000 7f8170c00640 1 -- 172.21.15.96:0/1697129929 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f814c06d0d0 msgr2=0x7f814c06f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:20.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.725+0000 7f8170c00640 1 --2- 172.21.15.96:0/1697129929 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f814c06d0d0 0x7f814c06f590 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f815c009890 tx=0x7f815c008040 comp rx=0 tx=0).stop 2024-05-28T06:12:20.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.725+0000 7f8170c00640 1 -- 172.21.15.96:0/1697129929 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f816c153000 msgr2=0x7f816c1549e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:20.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.725+0000 7f8170c00640 1 --2- 172.21.15.96:0/1697129929 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f816c153000 0x7f816c1549e0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f8158001050 tx=0x7f815800ecd0 comp rx=0 tx=0).stop 2024-05-28T06:12:20.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.725+0000 7f8170c00640 1 -- 172.21.15.96:0/1697129929 shutdown_connections 2024-05-28T06:12:20.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.725+0000 7f8170c00640 1 --2- 172.21.15.96:0/1697129929 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f814c06d0d0 0x7f814c06f590 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:20.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.725+0000 7f8170c00640 1 --2- 172.21.15.96:0/1697129929 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f816c152460 0x7f816c1528e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:20.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.725+0000 7f8170c00640 1 --2- 172.21.15.96:0/1697129929 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f816c153000 0x7f816c1549e0 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:20.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.725+0000 7f8170c00640 1 --2- 172.21.15.96:0/1697129929 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f816c099910 0x7f816c15a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:20.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.725+0000 7f8170c00640 1 -- 172.21.15.96:0/1697129929 >> 172.21.15.96:0/1697129929 conn(0x7f816c0b1fc0 msgr2=0x7f816c160010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:20.727 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.726+0000 7f8170c00640 1 -- 172.21.15.96:0/1697129929 shutdown_connections 2024-05-28T06:12:20.727 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.726+0000 7f8170c00640 1 -- 172.21.15.96:0/1697129929 wait complete. 2024-05-28T06:12:20.747 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-05-28T06:12:20.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.949+0000 7f9359a00640 1 Processor -- start 2024-05-28T06:12:20.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.949+0000 7f9359a00640 1 -- start start 2024-05-28T06:12:20.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.951+0000 7f9359a00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f93540cfa40 0x7f9354169690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:20.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.951+0000 7f9359a00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f935415ec50 con 0x7f935415e870 2024-05-28T06:12:20.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.951+0000 7f9359a00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93540cff80 con 0x7f93540d16a0 2024-05-28T06:12:20.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.951+0000 7f9359a00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f93540d2150 con 0x7f93540cfa40 2024-05-28T06:12:20.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.951+0000 7f9353e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f935415e870 0x7f93540d0f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:60082/0 (socket says 172.21.15.96:60082) 2024-05-28T06:12:20.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.952+0000 7f9353e00640 1 -- 172.21.15.96:0/3918658566 learned_addr learned my addr 172.21.15.96:0/3918658566 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:12:20.954 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.952+0000 7f9353400640 1 -- 172.21.15.96:0/3918658566 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1606630057 0 0) 0x7f935415ec50 con 0x7f935415e870 2024-05-28T06:12:20.954 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.952+0000 7f9353400640 1 -- 172.21.15.96:0/3918658566 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9338003520 con 0x7f935415e870 2024-05-28T06:12:20.954 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.952+0000 7f9359000640 1 --2- 172.21.15.96:0/3918658566 >> v2:172.21.15.123:3300/0 conn(0x7f93540cfa40 0x7f9354169690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:20.954 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.952+0000 7f9353400640 1 -- 172.21.15.96:0/3918658566 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2179139598 0 0) 0x7f9338003520 con 0x7f935415e870 2024-05-28T06:12:20.954 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.952+0000 7f9353400640 1 -- 172.21.15.96:0/3918658566 >> v2:172.21.15.123:3300/0 conn(0x7f93540cfa40 msgr2=0x7f9354169690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:20.954 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.952+0000 7f9353400640 1 --2- 172.21.15.96:0/3918658566 >> v2:172.21.15.123:3300/0 conn(0x7f93540cfa40 0x7f9354169690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:20.954 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.952+0000 7f9353400640 1 -- 172.21.15.96:0/3918658566 >> v1:172.21.15.123:6789/0 conn(0x7f93540d16a0 legacy=0x7f93540cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:20.954 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.952+0000 7f9353400640 1 -- 172.21.15.96:0/3918658566 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93540d0260 con 0x7f935415e870 2024-05-28T06:12:20.954 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.952+0000 7f9353400640 1 -- 172.21.15.96:0/3918658566 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f9344001670 con 0x7f935415e870 2024-05-28T06:12:20.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.952+0000 7f9353400640 1 -- 172.21.15.96:0/3918658566 >> v1:172.21.15.96:6789/0 conn(0x7f935415e870 legacy=0x7f93540d0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:20.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.953+0000 7f9353400640 1 --2- 172.21.15.96:0/3918658566 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f93380039b0 0x7f9338003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:20.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.953+0000 7f9353400640 1 --2- 172.21.15.96:0/3918658566 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9338004800 0x7f9338004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:20.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.953+0000 7f9353400640 1 --2- 172.21.15.96:0/3918658566 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f93380051c0 0x7f933800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:20.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.953+0000 7f9353400640 1 -- 172.21.15.96:0/3918658566 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f93540d2150 con 0x7f93380051c0 2024-05-28T06:12:20.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.953+0000 7f9353400640 1 -- 172.21.15.96:0/3918658566 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f933800c0f0 con 0x7f93380039b0 2024-05-28T06:12:20.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.953+0000 7f9353400640 1 -- 172.21.15.96:0/3918658566 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f933800c2f0 con 0x7f9338004800 2024-05-28T06:12:20.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.953+0000 7f9359000640 1 --2- 172.21.15.96:0/3918658566 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f93380051c0 0x7f933800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:20.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.953+0000 7f9353e00640 1 --2- 172.21.15.96:0/3918658566 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f93380039b0 0x7f9338003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:20.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.953+0000 7f9359000640 1 -- 172.21.15.96:0/3918658566 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9338004800 msgr2=0x7f9338004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:20.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.953+0000 7f9359000640 1 --2- 172.21.15.96:0/3918658566 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9338004800 0x7f9338004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:20.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.953+0000 7f9359000640 1 -- 172.21.15.96:0/3918658566 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f93380039b0 msgr2=0x7f9338003db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:20.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.953+0000 7f9359000640 1 --2- 172.21.15.96:0/3918658566 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f93380039b0 0x7f9338003db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:20.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.953+0000 7f9359000640 1 -- 172.21.15.96:0/3918658566 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9338003520 con 0x7f93380051c0 2024-05-28T06:12:20.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.953+0000 7f9353e00640 1 --2- 172.21.15.96:0/3918658566 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f93380039b0 0x7f9338003db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:12:20.956 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.954+0000 7f9359000640 1 -- 172.21.15.96:0/3918658566 reap_dead start 2024-05-28T06:12:20.956 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.954+0000 7f9359000640 1 --2- 172.21.15.96:0/3918658566 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f93380051c0 0x7f933800b9c0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f93540cfeb0 tx=0x7f9338005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cff5a34f4f964949 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:20.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.954+0000 7f9353400640 1 -- 172.21.15.96:0/3918658566 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f934c033070 con 0x7f93380051c0 2024-05-28T06:12:20.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.955+0000 7f9353400640 1 -- 172.21.15.96:0/3918658566 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f934c02f650 con 0x7f93380051c0 2024-05-28T06:12:20.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.955+0000 7f9353400640 1 -- 172.21.15.96:0/3918658566 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f934c009be0 con 0x7f93380051c0 2024-05-28T06:12:20.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.955+0000 7f9359a00640 1 -- 172.21.15.96:0/3918658566 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f93380051c0 msgr2=0x7f933800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:20.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.955+0000 7f9359a00640 1 --2- 172.21.15.96:0/3918658566 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f93380051c0 0x7f933800b9c0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f93540cfeb0 tx=0x7f9338005830 comp rx=0 tx=0).stop 2024-05-28T06:12:20.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.955+0000 7f9359a00640 1 -- 172.21.15.96:0/3918658566 shutdown_connections 2024-05-28T06:12:20.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.955+0000 7f9359a00640 1 --2- 172.21.15.96:0/3918658566 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f93380051c0 0x7f933800b9c0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:20.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.955+0000 7f9359a00640 1 -- 172.21.15.96:0/3918658566 >> 172.21.15.96:0/3918658566 conn(0x7f935408bfc0 msgr2=0x7f93540d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:20.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.955+0000 7f9359a00640 1 -- 172.21.15.96:0/3918658566 shutdown_connections 2024-05-28T06:12:20.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.955+0000 7f9359a00640 1 -- 172.21.15.96:0/3918658566 wait complete. 2024-05-28T06:12:20.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.957+0000 7f9359a00640 1 Processor -- start 2024-05-28T06:12:20.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.957+0000 7f9359a00640 1 -- start start 2024-05-28T06:12:20.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.958+0000 7f9359a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f93380051c0 0x7f93540cd5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:20.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.958+0000 7f9359a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9354152a20 0x7f9354153ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:20.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.958+0000 7f9359a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9354152110 0x7f935415d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:20.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.958+0000 7f9359a00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f93540d4fb0 con 0x7f9354152110 2024-05-28T06:12:20.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.958+0000 7f9359a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f93540d0820 con 0x7f93380051c0 2024-05-28T06:12:20.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.958+0000 7f9359a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f93540cf330 con 0x7f9354152a20 2024-05-28T06:12:20.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.958+0000 7f9359000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9354152110 0x7f935415d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:20.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.958+0000 7f9359000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9354152110 0x7f935415d190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44050/0 (socket says 172.21.15.96:44050) 2024-05-28T06:12:20.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.958+0000 7f9359000640 1 -- 172.21.15.96:0/2293385655 learned_addr learned my addr 172.21.15.96:0/2293385655 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:20.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.959+0000 7f934be00640 1 --2- 172.21.15.96:0/2293385655 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9354152a20 0x7f9354153ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:20.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.959+0000 7f9353e00640 1 --2- 172.21.15.96:0/2293385655 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f93380051c0 0x7f93540cd5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:20.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.959+0000 7f9359000640 1 -- 172.21.15.96:0/2293385655 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9354152a20 msgr2=0x7f9354153ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:20.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.959+0000 7f9359000640 1 --2- 172.21.15.96:0/2293385655 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9354152a20 0x7f9354153ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:20.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.959+0000 7f9359000640 1 -- 172.21.15.96:0/2293385655 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f93380051c0 msgr2=0x7f93540cd5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:20.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.959+0000 7f9359000640 1 --2- 172.21.15.96:0/2293385655 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f93380051c0 0x7f93540cd5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:20.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.959+0000 7f9359000640 1 -- 172.21.15.96:0/2293385655 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93540cddf0 con 0x7f9354152110 2024-05-28T06:12:20.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.959+0000 7f9353e00640 1 --2- 172.21.15.96:0/2293385655 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f93380051c0 0x7f93540cd5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:12:20.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.959+0000 7f934be00640 1 --2- 172.21.15.96:0/2293385655 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9354152a20 0x7f9354153ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:12:20.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.959+0000 7f9359000640 1 --2- 172.21.15.96:0/2293385655 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9354152110 0x7f935415d190 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f934c0073f0 tx=0x7f934c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:20.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.959+0000 7f9350c00640 1 -- 172.21.15.96:0/2293385655 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f934c048070 con 0x7f9354152110 2024-05-28T06:12:20.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.959+0000 7f9350c00640 1 -- 172.21.15.96:0/2293385655 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f934c039840 con 0x7f9354152110 2024-05-28T06:12:20.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.959+0000 7f9359a00640 1 -- 172.21.15.96:0/2293385655 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93540ce0d0 con 0x7f9354152110 2024-05-28T06:12:20.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.959+0000 7f9350c00640 1 -- 172.21.15.96:0/2293385655 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f934c033040 con 0x7f9354152110 2024-05-28T06:12:20.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.960+0000 7f9359a00640 1 -- 172.21.15.96:0/2293385655 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93540cdb00 con 0x7f9354152110 2024-05-28T06:12:20.962 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.961+0000 7f9359a00640 1 -- 172.21.15.96:0/2293385655 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93540689e0 con 0x7f9354152110 2024-05-28T06:12:20.962 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.962+0000 7f9350c00640 1 -- 172.21.15.96:0/2293385655 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 16) ==== 91754+0+0 (secure 0 0 0) 0x7f934c04b070 con 0x7f9354152110 2024-05-28T06:12:20.963 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.962+0000 7f9350c00640 1 --2- 172.21.15.96:0/2293385655 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f933006d0d0 0x7f933006f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:20.963 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.963+0000 7f9350c00640 1 -- 172.21.15.96:0/2293385655 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 4971+0+0 (secure 0 0 0) 0x7f934c0ad360 con 0x7f9354152110 2024-05-28T06:12:20.963 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.963+0000 7f9353e00640 1 --2- 172.21.15.96:0/2293385655 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f933006d0d0 0x7f933006f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:20.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.964+0000 7f9353e00640 1 --2- 172.21.15.96:0/2293385655 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f933006d0d0 0x7f933006f590 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f93440079d0 tx=0x7f9344001c10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:20.967 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:20.967+0000 7f9350c00640 1 -- 172.21.15.96:0/2293385655 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f934c051060 con 0x7f9354152110 2024-05-28T06:12:21.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.151+0000 7f9359a00640 1 -- 172.21.15.96:0/2293385655 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f93540c7e40 con 0x7f9354152110 2024-05-28T06:12:21.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.158+0000 7f9350c00640 1 -- 172.21.15.96:0/2293385655 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v14)=0 v14) ==== 122+0+0 (secure 0 0 0) 0x7f934c03e070 con 0x7f9354152110 2024-05-28T06:12:21.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.160+0000 7f9359a00640 1 -- 172.21.15.96:0/2293385655 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f933006d0d0 msgr2=0x7f933006f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:21.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.160+0000 7f9359a00640 1 --2- 172.21.15.96:0/2293385655 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f933006d0d0 0x7f933006f590 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f93440079d0 tx=0x7f9344001c10 comp rx=0 tx=0).stop 2024-05-28T06:12:21.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.161+0000 7f9359a00640 1 -- 172.21.15.96:0/2293385655 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9354152110 msgr2=0x7f935415d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:21.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.161+0000 7f9359a00640 1 --2- 172.21.15.96:0/2293385655 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9354152110 0x7f935415d190 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f934c0073f0 tx=0x7f934c004ab0 comp rx=0 tx=0).stop 2024-05-28T06:12:21.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.161+0000 7f9359a00640 1 -- 172.21.15.96:0/2293385655 shutdown_connections 2024-05-28T06:12:21.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.161+0000 7f9359a00640 1 --2- 172.21.15.96:0/2293385655 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f933006d0d0 0x7f933006f590 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:21.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.161+0000 7f9359a00640 1 --2- 172.21.15.96:0/2293385655 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9354152110 0x7f935415d190 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:21.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.161+0000 7f9359a00640 1 --2- 172.21.15.96:0/2293385655 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9354152a20 0x7f9354153ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:21.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.161+0000 7f9359a00640 1 --2- 172.21.15.96:0/2293385655 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f93380051c0 0x7f93540cd5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:21.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.161+0000 7f9359a00640 1 -- 172.21.15.96:0/2293385655 >> 172.21.15.96:0/2293385655 conn(0x7f935408bfc0 msgr2=0x7f93540b3490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:21.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.161+0000 7f9359a00640 1 -- 172.21.15.96:0/2293385655 shutdown_connections 2024-05-28T06:12:21.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.161+0000 7f9359a00640 1 -- 172.21.15.96:0/2293385655 wait complete. 2024-05-28T06:12:21.181 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-05-28T06:12:21.181 DEBUG:teuthology.orchestra.run.smithi096:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-05-28T06:12:21.275 DEBUG:teuthology.orchestra.run.smithi123:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-05-28T06:12:21.389 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-05-28T06:12:21.389 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:12:21.389 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:12:21.446 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052731,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248996k,nr_inodes=812249,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-28T06:12:21.447 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-05-28T06:12:21.447 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:12:21.505 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052731,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248996k,nr_inodes=812249,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-05-28T06:12:21.506 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:12:21.506 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:12:21.563 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052731,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248996k,nr_inodes=812249,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-28T06:12:21.563 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-05-28T06:12:21.563 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:12:21.621 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052731,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248996k,nr_inodes=812249,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-05-28T06:12:21.622 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-05-28T06:12:21.622 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-28T06:12:21.827 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.827+0000 7f0cbac00640 1 Processor -- start 2024-05-28T06:12:21.827 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.827+0000 7f0cbac00640 1 -- start start 2024-05-28T06:12:21.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.828+0000 7f0cbac00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0cb4073310 0x7f0cb40b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:21.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.828+0000 7f0cbac00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0cb40afd60 con 0x7f0cb415b3a0 2024-05-28T06:12:21.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.828+0000 7f0cbac00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0cb40af7a0 con 0x7f0cb406dc20 2024-05-28T06:12:21.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.828+0000 7f0cbac00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f0cb406f980 con 0x7f0cb4073310 2024-05-28T06:12:21.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.829+0000 7f0cb9800640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f0cb415b3a0 0x7f0cb4072c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:60096/0 (socket says 172.21.15.96:60096) 2024-05-28T06:12:21.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.829+0000 7f0cb9800640 1 -- 172.21.15.96:0/3231273664 learned_addr learned my addr 172.21.15.96:0/3231273664 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:12:21.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.829+0000 7f0cb3e00640 1 -- 172.21.15.96:0/3231273664 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2404053111 0 0) 0x7f0cb40afd60 con 0x7f0cb415b3a0 2024-05-28T06:12:21.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.829+0000 7f0cb8e00640 1 --2- 172.21.15.96:0/3231273664 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0cb4073310 0x7f0cb40b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:21.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.829+0000 7f0cb3e00640 1 -- 172.21.15.96:0/3231273664 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0c9c003520 con 0x7f0cb415b3a0 2024-05-28T06:12:21.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.829+0000 7f0cb3e00640 1 -- 172.21.15.96:0/3231273664 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 825690084 0 0) 0x7f0c9c003520 con 0x7f0cb415b3a0 2024-05-28T06:12:21.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.829+0000 7f0cb3e00640 1 -- 172.21.15.96:0/3231273664 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0cb4073310 msgr2=0x7f0cb40b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:21.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.829+0000 7f0cb3e00640 1 --2- 172.21.15.96:0/3231273664 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0cb4073310 0x7f0cb40b32c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:21.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.829+0000 7f0cb3e00640 1 -- 172.21.15.96:0/3231273664 >> v1:172.21.15.123:6789/0 conn(0x7f0cb406dc20 legacy=0x7f0cb40ae550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:21.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.830+0000 7f0cb3e00640 1 -- 172.21.15.96:0/3231273664 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cb40afa80 con 0x7f0cb415b3a0 2024-05-28T06:12:21.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.830+0000 7f0cb3e00640 1 -- 172.21.15.96:0/3231273664 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f0ca8001670 con 0x7f0cb415b3a0 2024-05-28T06:12:21.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.830+0000 7f0cb3e00640 1 -- 172.21.15.96:0/3231273664 >> v1:172.21.15.96:6789/0 conn(0x7f0cb415b3a0 legacy=0x7f0cb4072c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:21.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.830+0000 7f0cb3e00640 1 --2- 172.21.15.96:0/3231273664 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0c9c0039e0 0x7f0c9c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:21.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.830+0000 7f0cb3e00640 1 --2- 172.21.15.96:0/3231273664 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0cb4073310 0x7f0c9c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:21.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.830+0000 7f0cb3e00640 1 --2- 172.21.15.96:0/3231273664 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c9c004d40 0x7f0c9c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:21.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.830+0000 7f0cb3e00640 1 -- 172.21.15.96:0/3231273664 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f0cb406f980 con 0x7f0c9c004d40 2024-05-28T06:12:21.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.830+0000 7f0cb3e00640 1 -- 172.21.15.96:0/3231273664 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f0c9c00bb50 con 0x7f0c9c0039e0 2024-05-28T06:12:21.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.830+0000 7f0cb3e00640 1 -- 172.21.15.96:0/3231273664 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f0c9c00bd20 con 0x7f0cb4073310 2024-05-28T06:12:21.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.830+0000 7f0cba200640 1 --2- 172.21.15.96:0/3231273664 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c9c004d40 0x7f0c9c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:21.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.830+0000 7f0cb8e00640 1 --2- 172.21.15.96:0/3231273664 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0cb4073310 0x7f0c9c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:21.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.830+0000 7f0cb9800640 1 --2- 172.21.15.96:0/3231273664 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0c9c0039e0 0x7f0c9c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:21.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.830+0000 7f0cba200640 1 -- 172.21.15.96:0/3231273664 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0cb4073310 msgr2=0x7f0c9c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:21.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.830+0000 7f0cba200640 1 --2- 172.21.15.96:0/3231273664 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0cb4073310 0x7f0c9c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:21.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.830+0000 7f0cba200640 1 -- 172.21.15.96:0/3231273664 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0c9c0039e0 msgr2=0x7f0c9c003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:21.833 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.830+0000 7f0cba200640 1 --2- 172.21.15.96:0/3231273664 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0c9c0039e0 0x7f0c9c003e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:21.833 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.830+0000 7f0cba200640 1 -- 172.21.15.96:0/3231273664 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c9c003520 con 0x7f0c9c004d40 2024-05-28T06:12:21.833 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.830+0000 7f0cb8e00640 1 --2- 172.21.15.96:0/3231273664 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0cb4073310 0x7f0c9c004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:12:21.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.831+0000 7f0cba200640 1 --2- 172.21.15.96:0/3231273664 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c9c004d40 0x7f0c9c0051c0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f0cac001e80 tx=0x7f0c9c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd90bd875cd6f5e6 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:21.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.832+0000 7f0cb3e00640 1 -- 172.21.15.96:0/3231273664 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cac001a70 con 0x7f0c9c004d40 2024-05-28T06:12:21.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.832+0000 7f0cb3e00640 1 -- 172.21.15.96:0/3231273664 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0cac02ae50 con 0x7f0c9c004d40 2024-05-28T06:12:21.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.832+0000 7f0cb3e00640 1 -- 172.21.15.96:0/3231273664 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cac003bb0 con 0x7f0c9c004d40 2024-05-28T06:12:21.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.832+0000 7f0cbac00640 1 -- 172.21.15.96:0/3231273664 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c9c004d40 msgr2=0x7f0c9c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:21.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.832+0000 7f0cbac00640 1 --2- 172.21.15.96:0/3231273664 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c9c004d40 0x7f0c9c0051c0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f0cac001e80 tx=0x7f0c9c005830 comp rx=0 tx=0).stop 2024-05-28T06:12:21.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.832+0000 7f0cba200640 1 -- 172.21.15.96:0/3231273664 reap_dead start 2024-05-28T06:12:21.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.832+0000 7f0cbac00640 1 -- 172.21.15.96:0/3231273664 shutdown_connections 2024-05-28T06:12:21.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.832+0000 7f0cbac00640 1 -- 172.21.15.96:0/3231273664 >> 172.21.15.96:0/3231273664 conn(0x7f0cb40b1fc0 msgr2=0x7f0cb40b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:21.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.832+0000 7f0cbac00640 1 -- 172.21.15.96:0/3231273664 shutdown_connections 2024-05-28T06:12:21.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.832+0000 7f0cbac00640 1 -- 172.21.15.96:0/3231273664 wait complete. 2024-05-28T06:12:21.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.834+0000 7f0cbac00640 1 Processor -- start 2024-05-28T06:12:21.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.834+0000 7f0cbac00640 1 -- start start 2024-05-28T06:12:21.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.835+0000 7f0cbac00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c9c004d40 0x7f0cb414e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:21.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.835+0000 7f0cbac00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0cb414eb10 0x7f0cb40998f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:21.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.835+0000 7f0cbac00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0cb4150610 0x7f0cb4150a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:21.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.835+0000 7f0cbac00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f0cb40b32c0 con 0x7f0c9c004d40 2024-05-28T06:12:21.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.835+0000 7f0cbac00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f0cb40ae0c0 con 0x7f0cb414eb10 2024-05-28T06:12:21.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.835+0000 7f0cbac00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f0cb40b3440 con 0x7f0cb4150610 2024-05-28T06:12:21.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.835+0000 7f0cb9800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c9c004d40 0x7f0cb414e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:21.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.835+0000 7f0cb9800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c9c004d40 0x7f0cb414e1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44058/0 (socket says 172.21.15.96:44058) 2024-05-28T06:12:21.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.835+0000 7f0cb9800640 1 -- 172.21.15.96:0/1402233188 learned_addr learned my addr 172.21.15.96:0/1402233188 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:21.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.836+0000 7f0cb9800640 1 -- 172.21.15.96:0/1402233188 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0cb4150610 msgr2=0x7f0cb4150a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:21.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.836+0000 7f0cb9800640 1 --2- 172.21.15.96:0/1402233188 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0cb4150610 0x7f0cb4150a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:21.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.836+0000 7f0cb9800640 1 -- 172.21.15.96:0/1402233188 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0cb414eb10 msgr2=0x7f0cb40998f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:21.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.836+0000 7f0cb9800640 1 --2- 172.21.15.96:0/1402233188 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0cb414eb10 0x7f0cb40998f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:21.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.836+0000 7f0cb9800640 1 -- 172.21.15.96:0/1402233188 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cb40cdca0 con 0x7f0c9c004d40 2024-05-28T06:12:21.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.836+0000 7f0cb9800640 1 --2- 172.21.15.96:0/1402233188 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c9c004d40 0x7f0cb414e1a0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f0ca800d840 tx=0x7f0ca800dd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:21.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.836+0000 7f0cb1600640 1 -- 172.21.15.96:0/1402233188 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ca8019070 con 0x7f0c9c004d40 2024-05-28T06:12:21.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.836+0000 7f0cb1600640 1 -- 172.21.15.96:0/1402233188 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ca80074e0 con 0x7f0c9c004d40 2024-05-28T06:12:21.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.836+0000 7f0cb1600640 1 -- 172.21.15.96:0/1402233188 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ca80049e0 con 0x7f0c9c004d40 2024-05-28T06:12:21.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.836+0000 7f0cbac00640 1 -- 172.21.15.96:0/1402233188 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0cb40cdf80 con 0x7f0c9c004d40 2024-05-28T06:12:21.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.837+0000 7f0cbac00640 1 -- 172.21.15.96:0/1402233188 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0cb40ce540 con 0x7f0c9c004d40 2024-05-28T06:12:21.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.838+0000 7f0cbac00640 1 -- 172.21.15.96:0/1402233188 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0cb40689e0 con 0x7f0c9c004d40 2024-05-28T06:12:21.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.839+0000 7f0cb1600640 1 -- 172.21.15.96:0/1402233188 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 16) ==== 91754+0+0 (secure 0 0 0) 0x7f0ca8004b80 con 0x7f0c9c004d40 2024-05-28T06:12:21.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.839+0000 7f0cb1600640 1 --2- 172.21.15.96:0/1402233188 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0c9406d0d0 0x7f0c9406f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:21.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.840+0000 7f0cb1600640 1 -- 172.21.15.96:0/1402233188 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 4971+0+0 (secure 0 0 0) 0x7f0ca808c610 con 0x7f0c9c004d40 2024-05-28T06:12:21.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.840+0000 7f0cb8e00640 1 --2- 172.21.15.96:0/1402233188 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0c9406d0d0 0x7f0c9406f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:21.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.842+0000 7f0cb8e00640 1 --2- 172.21.15.96:0/1402233188 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0c9406d0d0 0x7f0c9406f590 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f0cb4154c00 tx=0x7f0ca0009350 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:21.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:21.844+0000 7f0cb1600640 1 -- 172.21.15.96:0/1402233188 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0ca8059bf0 con 0x7f0c9c004d40 2024-05-28T06:12:22.065 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:22.065+0000 7f0cbac00640 1 -- 172.21.15.96:0/1402233188 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0cb40b4c40 con 0x7f0c9c004d40 2024-05-28T06:12:22.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:22.066+0000 7f0cb1600640 1 -- 172.21.15.96:0/1402233188 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+3872 (secure 0 0 0) 0x7f0ca80594f0 con 0x7f0c9c004d40 2024-05-28T06:12:22.067 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:12:22.067 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":28,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/1693172076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":1693172076},{"type":"v1","addr":"172.21.15.123:6835","nonce":1693172076}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":4805,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/1098611268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":1098611268},{"type":"v1","addr":"172.21.15.123:6837","nonce":1098611268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5856,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6834/673165997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":673165997},{"type":"v1","addr":"172.21.15.96:6834","nonce":673165997}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19},{"gid":6408,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/1966973064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6833","nonce":1966973064},{"type":"v1","addr":"172.21.15.96:6835","nonce":1966973064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[]} 2024-05-28T06:12:22.067 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 28 2024-05-28T06:12:22.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:22.068+0000 7f0cbac00640 1 -- 172.21.15.96:0/1402233188 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0c9406d0d0 msgr2=0x7f0c9406f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:22.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:22.068+0000 7f0cbac00640 1 --2- 172.21.15.96:0/1402233188 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0c9406d0d0 0x7f0c9406f590 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f0cb4154c00 tx=0x7f0ca0009350 comp rx=0 tx=0).stop 2024-05-28T06:12:22.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:22.068+0000 7f0cbac00640 1 -- 172.21.15.96:0/1402233188 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c9c004d40 msgr2=0x7f0cb414e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:22.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:22.068+0000 7f0cbac00640 1 --2- 172.21.15.96:0/1402233188 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c9c004d40 0x7f0cb414e1a0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f0ca800d840 tx=0x7f0ca800dd00 comp rx=0 tx=0).stop 2024-05-28T06:12:22.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:22.068+0000 7f0cbac00640 1 -- 172.21.15.96:0/1402233188 shutdown_connections 2024-05-28T06:12:22.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:22.068+0000 7f0cbac00640 1 --2- 172.21.15.96:0/1402233188 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0c9406d0d0 0x7f0c9406f590 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:22.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:22.068+0000 7f0cbac00640 1 --2- 172.21.15.96:0/1402233188 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0cb4150610 0x7f0cb4150a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:22.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:22.068+0000 7f0cbac00640 1 --2- 172.21.15.96:0/1402233188 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0cb414eb10 0x7f0cb40998f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:22.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:22.068+0000 7f0cbac00640 1 --2- 172.21.15.96:0/1402233188 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c9c004d40 0x7f0cb414e1a0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:22.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:22.068+0000 7f0cbac00640 1 -- 172.21.15.96:0/1402233188 >> 172.21.15.96:0/1402233188 conn(0x7f0cb40b1fc0 msgr2=0x7f0cb40b3f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:22.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:22.069+0000 7f0cbac00640 1 -- 172.21.15.96:0/1402233188 shutdown_connections 2024-05-28T06:12:22.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:22.069+0000 7f0cbac00640 1 -- 172.21.15.96:0/1402233188 wait complete. 2024-05-28T06:12:22.087 INFO:tasks.ceph.mds.a:Restarting daemon 2024-05-28T06:12:22.087 INFO:tasks.ceph.mds.a:Stopping old one... 2024-05-28T06:12:22.088 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-05-28T06:12:22.088 INFO:teuthology.orchestra.run:waiting for 300 2024-05-28T06:12:22.088 INFO:tasks.ceph.mds.c:Restarting daemon 2024-05-28T06:12:22.088 INFO:tasks.ceph.mds.c:Stopping old one... 2024-05-28T06:12:22.088 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-05-28T06:12:22.088 INFO:teuthology.orchestra.run:waiting for 300 2024-05-28T06:12:22.088 INFO:tasks.ceph.mds.b:Restarting daemon 2024-05-28T06:12:22.089 INFO:tasks.ceph.mds.b:Stopping old one... 2024-05-28T06:12:22.089 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-05-28T06:12:22.089 INFO:teuthology.orchestra.run:waiting for 300 2024-05-28T06:12:22.089 INFO:tasks.ceph.mds.d:Restarting daemon 2024-05-28T06:12:22.089 INFO:tasks.ceph.mds.d:Stopping old one... 2024-05-28T06:12:22.089 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-05-28T06:12:22.089 INFO:teuthology.orchestra.run:waiting for 300 2024-05-28T06:12:22.107 INFO:tasks.ceph.mds.a:Stopped 2024-05-28T06:12:22.107 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-05-28T06:12:22.109 INFO:tasks.ceph.mds.c:Stopped 2024-05-28T06:12:22.109 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-05-28T06:12:22.121 INFO:tasks.ceph.mds.d:Stopped 2024-05-28T06:12:22.121 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-05-28T06:12:22.124 INFO:tasks.ceph.mds.b:Stopped 2024-05-28T06:12:22.124 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-05-28T06:12:22.155 INFO:tasks.ceph.mds.a:Started 2024-05-28T06:12:22.155 DEBUG:teuthology.parallel:result is None 2024-05-28T06:12:22.159 INFO:tasks.ceph.mds.c:Started 2024-05-28T06:12:22.160 DEBUG:teuthology.parallel:result is None 2024-05-28T06:12:22.171 INFO:tasks.ceph.mds.d:Started 2024-05-28T06:12:22.171 DEBUG:teuthology.parallel:result is None 2024-05-28T06:12:22.177 INFO:tasks.ceph.mds.b:Started 2024-05-28T06:12:22.177 DEBUG:teuthology.parallel:result is None 2024-05-28T06:12:22.177 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-05-28T06:12:22.294 INFO:tasks.ceph.mds.c.smithi096.stdout:starting mds.c at 2024-05-28T06:12:22.295 INFO:tasks.ceph.mds.a.smithi096.stdout:starting mds.a at 2024-05-28T06:12:22.322 INFO:tasks.ceph.mds.b.smithi123.stdout:starting mds.b at 2024-05-28T06:12:22.324 INFO:tasks.ceph.mds.d.smithi123.stdout:starting mds.d at 2024-05-28T06:12:22.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:22.342+0000 7f5ecd200640 1 Processor -- start 2024-05-28T06:12:22.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:22.342+0000 7f5ecd200640 1 -- start start 2024-05-28T06:12:22.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:22.343+0000 7f5ecd200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f5ec80b7f80 0x7f5ec80d1280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:22.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:22.343+0000 7f5ecd200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5ec80d17c0 0x7f5ec809aa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:22.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:22.343+0000 7f5ecd200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f5ec809af70 0x7f5ec815e160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:22.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:22.343+0000 7f5ecd200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f5ec80ccd00 con 0x7f5ec809af70 2024-05-28T06:12:22.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:22.343+0000 7f5ecd200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f5ec80ccfe0 con 0x7f5ec80b7f80 2024-05-28T06:12:22.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:22.343+0000 7f5ecd200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f5ec80cd2c0 con 0x7f5ec80d17c0 2024-05-28T06:12:22.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:22.343+0000 7f5ec6800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f5ec809af70 0x7f5ec815e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:22.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:22.343+0000 7f5ec6800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f5ec809af70 0x7f5ec815e160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44090/0 (socket says 172.21.15.96:44090) 2024-05-28T06:12:22.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:22.343+0000 7f5ec6800640 1 -- 172.21.15.96:0/861378008 learned_addr learned my addr 172.21.15.96:0/861378008 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:22.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:22.343+0000 7f5ec5e00640 1 --2- 172.21.15.96:0/861378008 >> v2:172.21.15.123:3300/0 conn(0x7f5ec80b7f80 0x7f5ec80d1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:22.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:22.343+0000 7f5ec5400640 1 --2- 172.21.15.96:0/861378008 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5ec80d17c0 0x7f5ec809aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:22.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:22.344+0000 7f5ec6800640 1 -- 172.21.15.96:0/861378008 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5ec80d17c0 msgr2=0x7f5ec809aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:22.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:22.344+0000 7f5ec6800640 1 --2- 172.21.15.96:0/861378008 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5ec80d17c0 0x7f5ec809aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:22.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:22.344+0000 7f5ec6800640 1 -- 172.21.15.96:0/861378008 >> v2:172.21.15.123:3300/0 conn(0x7f5ec80b7f80 msgr2=0x7f5ec80d1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:22.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:22.344+0000 7f5ec6800640 1 --2- 172.21.15.96:0/861378008 >> v2:172.21.15.123:3300/0 conn(0x7f5ec80b7f80 0x7f5ec80d1280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:22.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:22.344+0000 7f5ec6800640 1 -- 172.21.15.96:0/861378008 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ec80cd5a0 con 0x7f5ec809af70 2024-05-28T06:12:22.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:22.345+0000 7f5ec6800640 1 --2- 172.21.15.96:0/861378008 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5ec809af70 0x7f5ec815e160 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f5ec80730a0 tx=0x7f5ec002a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cea460eed921b68a server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:22.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:22.345+0000 7f5ec4a00640 1 -- 172.21.15.96:0/861378008 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ec0036070 con 0x7f5ec809af70 2024-05-28T06:12:22.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:22.345+0000 7f5ec4a00640 1 -- 172.21.15.96:0/861378008 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ec0031600 con 0x7f5ec809af70 2024-05-28T06:12:22.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:22.345+0000 7f5ec4a00640 1 -- 172.21.15.96:0/861378008 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ec00319d0 con 0x7f5ec809af70 2024-05-28T06:12:22.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:22.345+0000 7f5ecd200640 1 -- 172.21.15.96:0/861378008 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5ec809af70 msgr2=0x7f5ec815e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:22.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:22.345+0000 7f5ecd200640 1 --2- 172.21.15.96:0/861378008 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5ec809af70 0x7f5ec815e160 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f5ec80730a0 tx=0x7f5ec002a6a0 comp rx=0 tx=0).stop 2024-05-28T06:12:22.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:22.346+0000 7f5ecd200640 1 -- 172.21.15.96:0/861378008 shutdown_connections 2024-05-28T06:12:22.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:22.346+0000 7f5ecd200640 1 --2- 172.21.15.96:0/861378008 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5ec809af70 0x7f5ec815e160 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:22.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:22.346+0000 7f5ecd200640 1 --2- 172.21.15.96:0/861378008 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5ec80d17c0 0x7f5ec809aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:22.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:22.346+0000 7f5ecd200640 1 --2- 172.21.15.96:0/861378008 >> v2:172.21.15.123:3300/0 conn(0x7f5ec80b7f80 0x7f5ec80d1280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:22.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:22.346+0000 7f5ecd200640 1 -- 172.21.15.96:0/861378008 >> 172.21.15.96:0/861378008 conn(0x7f5ec80b3fe0 msgr2=0x7f5ec80b43f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:22.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:22.346+0000 7f5ecd200640 1 -- 172.21.15.96:0/861378008 shutdown_connections 2024-05-28T06:12:22.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:22.346+0000 7f5ecd200640 1 -- 172.21.15.96:0/861378008 wait complete. 2024-05-28T06:12:22.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:22.347+0000 7f5ecd200640 1 Processor -- start 2024-05-28T06:12:22.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:22.347+0000 7f5ecd200640 1 -- start start 2024-05-28T06:12:22.350 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:22.348+0000 7f5ecd200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5ec809af70 0x7f5ec80c9fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:22.350 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:22.348+0000 7f5ecd200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5ec80b7f80 0x7f5ec8160cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:22.350 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:22.348+0000 7f5ecd200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5ec80d17c0 0x7f5ec815b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:22.350 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:22.348+0000 7f5ecd200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f5ec80b5340 con 0x7f5ec80b7f80 2024-05-28T06:12:22.351 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:22.348+0000 7f5ecd200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5ec80b8360 con 0x7f5ec809af70 2024-05-28T06:12:22.351 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:22.348+0000 7f5ecd200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f5ec80b54c0 con 0x7f5ec80d17c0 2024-05-28T06:12:22.351 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:22.349+0000 7f5ec5400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5ec80b7f80 0x7f5ec8160cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:22.351 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:22.349+0000 7f5ec5400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5ec80b7f80 0x7f5ec8160cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44100/0 (socket says 172.21.15.96:44100) 2024-05-28T06:12:22.351 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:22.349+0000 7f5ec5400640 1 -- 172.21.15.96:0/3244741687 learned_addr learned my addr 172.21.15.96:0/3244741687 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:22.351 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:22.349+0000 7f5ec5e00640 1 --2- 172.21.15.96:0/3244741687 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5ec809af70 0x7f5ec80c9fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:22.351 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:22.349+0000 7f5ec6800640 1 --2- 172.21.15.96:0/3244741687 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5ec80d17c0 0x7f5ec815b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:22.351 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:22.349+0000 7f5ec5400640 1 -- 172.21.15.96:0/3244741687 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5ec80d17c0 msgr2=0x7f5ec815b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:22.351 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:22.349+0000 7f5ec5400640 1 --2- 172.21.15.96:0/3244741687 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5ec80d17c0 0x7f5ec815b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:22.351 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:22.349+0000 7f5ec5400640 1 -- 172.21.15.96:0/3244741687 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5ec809af70 msgr2=0x7f5ec80c9fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:22.351 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:22.349+0000 7f5ec5400640 1 --2- 172.21.15.96:0/3244741687 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5ec809af70 0x7f5ec80c9fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:22.351 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:22.349+0000 7f5ec5400640 1 -- 172.21.15.96:0/3244741687 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ec815a720 con 0x7f5ec80b7f80 2024-05-28T06:12:22.351 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:22.349+0000 7f5ec6800640 1 --2- 172.21.15.96:0/3244741687 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5ec80d17c0 0x7f5ec815b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-28T06:12:22.352 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:22.349+0000 7f5ec5400640 1 --2- 172.21.15.96:0/3244741687 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5ec80b7f80 0x7f5ec8160cc0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f5eb400e9b0 tx=0x7f5eb400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:22.352 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:22.350+0000 7f5ebe000640 1 -- 172.21.15.96:0/3244741687 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5eb4017070 con 0x7f5ec80b7f80 2024-05-28T06:12:22.352 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:22.350+0000 7f5ebe000640 1 -- 172.21.15.96:0/3244741687 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5eb400c9a0 con 0x7f5ec80b7f80 2024-05-28T06:12:22.352 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:22.350+0000 7f5ebe000640 1 -- 172.21.15.96:0/3244741687 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5eb400cd90 con 0x7f5ec80b7f80 2024-05-28T06:12:22.352 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:22.350+0000 7f5ecd200640 1 -- 172.21.15.96:0/3244741687 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ec815aa00 con 0x7f5ec80b7f80 2024-05-28T06:12:22.352 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:22.350+0000 7f5ecd200640 1 -- 172.21.15.96:0/3244741687 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ec815afc0 con 0x7f5ec80b7f80 2024-05-28T06:12:22.352 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:22.351+0000 7f5ecd200640 1 -- 172.21.15.96:0/3244741687 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ec806e680 con 0x7f5ec80b7f80 2024-05-28T06:12:22.352 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:22.352+0000 7f5ebe000640 1 -- 172.21.15.96:0/3244741687 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 16) ==== 91754+0+0 (secure 0 0 0) 0x7f5eb401a070 con 0x7f5ec80b7f80 2024-05-28T06:12:22.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:22.353+0000 7f5ebe000640 1 --2- 172.21.15.96:0/3244741687 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5ea806d0d0 0x7f5ea806f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:22.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:22.353+0000 7f5ebe000640 1 -- 172.21.15.96:0/3244741687 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 4971+0+0 (secure 0 0 0) 0x7f5eb408ce60 con 0x7f5ec80b7f80 2024-05-28T06:12:22.354 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:22.353+0000 7f5ec5e00640 1 --2- 172.21.15.96:0/3244741687 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5ea806d0d0 0x7f5ea806f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:22.354 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:22.354+0000 7f5ec5e00640 1 --2- 172.21.15.96:0/3244741687 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5ea806d0d0 0x7f5ea806f590 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f5eb8009a20 tx=0x7f5eb8009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:22.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:22.357+0000 7f5ebe000640 1 -- 172.21.15.96:0/3244741687 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5ec806e680 con 0x7f5ec80b7f80 2024-05-28T06:12:22.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:22.553+0000 7f5ecd200640 1 -- 172.21.15.96:0/3244741687 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f5ec80689e0 con 0x7f5ec80b7f80 2024-05-28T06:12:23.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:23.761+0000 7f5ebe000640 1 -- 172.21.15.96:0/3244741687 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v56) ==== 97+0+0 (secure 0 0 0) 0x7f5eb4059f00 con 0x7f5ec80b7f80 2024-05-28T06:12:23.762 INFO:teuthology.orchestra.run.smithi096.stderr: removed all blocklist entries 2024-05-28T06:12:23.764 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:23.764+0000 7f5ecd200640 1 -- 172.21.15.96:0/3244741687 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5ea806d0d0 msgr2=0x7f5ea806f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:23.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:23.764+0000 7f5ecd200640 1 --2- 172.21.15.96:0/3244741687 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5ea806d0d0 0x7f5ea806f590 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f5eb8009a20 tx=0x7f5eb8009190 comp rx=0 tx=0).stop 2024-05-28T06:12:23.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:23.764+0000 7f5ecd200640 1 -- 172.21.15.96:0/3244741687 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5ec80b7f80 msgr2=0x7f5ec8160cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:23.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:23.764+0000 7f5ecd200640 1 --2- 172.21.15.96:0/3244741687 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5ec80b7f80 0x7f5ec8160cc0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f5eb400e9b0 tx=0x7f5eb400ee70 comp rx=0 tx=0).stop 2024-05-28T06:12:23.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:23.764+0000 7f5ecd200640 1 -- 172.21.15.96:0/3244741687 shutdown_connections 2024-05-28T06:12:23.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:23.764+0000 7f5ecd200640 1 --2- 172.21.15.96:0/3244741687 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5ea806d0d0 0x7f5ea806f590 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:23.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:23.764+0000 7f5ecd200640 1 --2- 172.21.15.96:0/3244741687 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5ec80d17c0 0x7f5ec815b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:23.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:23.764+0000 7f5ecd200640 1 --2- 172.21.15.96:0/3244741687 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5ec80b7f80 0x7f5ec8160cc0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:23.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:23.764+0000 7f5ecd200640 1 --2- 172.21.15.96:0/3244741687 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5ec809af70 0x7f5ec80c9fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:23.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:23.764+0000 7f5ecd200640 1 -- 172.21.15.96:0/3244741687 >> 172.21.15.96:0/3244741687 conn(0x7f5ec80b3fe0 msgr2=0x7f5ec80b71e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:23.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:23.764+0000 7f5ecd200640 1 -- 172.21.15.96:0/3244741687 shutdown_connections 2024-05-28T06:12:23.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:23.764+0000 7f5ecd200640 1 -- 172.21.15.96:0/3244741687 wait complete. 2024-05-28T06:12:23.786 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-05-28T06:12:23.988 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:23.987+0000 7fdbbea00640 1 Processor -- start 2024-05-28T06:12:23.988 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:23.988+0000 7fdbbea00640 1 -- start start 2024-05-28T06:12:23.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:23.989+0000 7fdbbea00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fdbb815e870 0x7fdbb80d0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:23.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:23.989+0000 7fdbbea00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fdbb80d14d0 0x7fdbb80cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:23.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:23.989+0000 7fdbbea00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdbb80cf870 0x7fdbb8162360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:23.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:23.989+0000 7fdbbea00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fdbb80d2150 con 0x7fdbb80d14d0 2024-05-28T06:12:23.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:23.989+0000 7fdbbea00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fdbb80cfd00 con 0x7fdbb815e870 2024-05-28T06:12:23.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:23.989+0000 7fdbbea00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fdbb815ec50 con 0x7fdbb80cf870 2024-05-28T06:12:23.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:23.989+0000 7fdbbcc00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fdbb80d14d0 0x7fdbb80cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:23.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:23.989+0000 7fdbbcc00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fdbb80d14d0 0x7fdbb80cf330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44106/0 (socket says 172.21.15.96:44106) 2024-05-28T06:12:23.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:23.989+0000 7fdbbcc00640 1 -- 172.21.15.96:0/2697647405 learned_addr learned my addr 172.21.15.96:0/2697647405 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:23.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:23.990+0000 7fdbbd600640 1 --2- 172.21.15.96:0/2697647405 >> v2:172.21.15.123:3300/0 conn(0x7fdbb815e870 0x7fdbb80d0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:23.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:23.990+0000 7fdbbe000640 1 --2- 172.21.15.96:0/2697647405 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdbb80cf870 0x7fdbb8162360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:23.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:23.990+0000 7fdbbcc00640 1 -- 172.21.15.96:0/2697647405 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdbb80cf870 msgr2=0x7fdbb8162360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:23.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:23.990+0000 7fdbbcc00640 1 --2- 172.21.15.96:0/2697647405 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdbb80cf870 0x7fdbb8162360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:23.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:23.990+0000 7fdbbcc00640 1 -- 172.21.15.96:0/2697647405 >> v2:172.21.15.123:3300/0 conn(0x7fdbb815e870 msgr2=0x7fdbb80d0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:23.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:23.990+0000 7fdbbcc00640 1 --2- 172.21.15.96:0/2697647405 >> v2:172.21.15.123:3300/0 conn(0x7fdbb815e870 0x7fdbb80d0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:23.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:23.990+0000 7fdbbcc00640 1 -- 172.21.15.96:0/2697647405 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbb80cff80 con 0x7fdbb80d14d0 2024-05-28T06:12:23.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:23.990+0000 7fdbbe000640 1 --2- 172.21.15.96:0/2697647405 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdbb80cf870 0x7fdbb8162360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:12:23.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:23.991+0000 7fdbbcc00640 1 --2- 172.21.15.96:0/2697647405 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdbb80d14d0 0x7fdbb80cf330 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fdbb808e720 tx=0x7fdba802a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a31f47b128bd00ad server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:23.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:23.991+0000 7fdbb7e00640 1 -- 172.21.15.96:0/2697647405 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdba8004030 con 0x7fdbb80d14d0 2024-05-28T06:12:23.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:23.991+0000 7fdbb7e00640 1 -- 172.21.15.96:0/2697647405 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdba80041d0 con 0x7fdbb80d14d0 2024-05-28T06:12:23.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:23.991+0000 7fdbb7e00640 1 -- 172.21.15.96:0/2697647405 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdba8003610 con 0x7fdbb80d14d0 2024-05-28T06:12:23.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:23.992+0000 7fdbbea00640 1 -- 172.21.15.96:0/2697647405 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdbb80d14d0 msgr2=0x7fdbb80cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:23.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:23.992+0000 7fdbbea00640 1 --2- 172.21.15.96:0/2697647405 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdbb80d14d0 0x7fdbb80cf330 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fdbb808e720 tx=0x7fdba802a1c0 comp rx=0 tx=0).stop 2024-05-28T06:12:23.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:23.992+0000 7fdbbea00640 1 -- 172.21.15.96:0/2697647405 shutdown_connections 2024-05-28T06:12:23.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:23.992+0000 7fdbbea00640 1 --2- 172.21.15.96:0/2697647405 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdbb80cf870 0x7fdbb8162360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:23.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:23.992+0000 7fdbbea00640 1 --2- 172.21.15.96:0/2697647405 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdbb80d14d0 0x7fdbb80cf330 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:23.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:23.992+0000 7fdbbea00640 1 --2- 172.21.15.96:0/2697647405 >> v2:172.21.15.123:3300/0 conn(0x7fdbb815e870 0x7fdbb80d0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:23.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:23.992+0000 7fdbbea00640 1 -- 172.21.15.96:0/2697647405 >> 172.21.15.96:0/2697647405 conn(0x7fdbb808bfc0 msgr2=0x7fdbb80d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:23.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:23.992+0000 7fdbbea00640 1 -- 172.21.15.96:0/2697647405 shutdown_connections 2024-05-28T06:12:23.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:23.992+0000 7fdbbea00640 1 -- 172.21.15.96:0/2697647405 wait complete. 2024-05-28T06:12:23.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:23.994+0000 7fdbbea00640 1 Processor -- start 2024-05-28T06:12:23.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:23.994+0000 7fdbbea00640 1 -- start start 2024-05-28T06:12:23.995 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:23.995+0000 7fdbbea00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdbb80cf870 0x7fdbb8157c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:23.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:23.995+0000 7fdbbea00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdbb80d14d0 0x7fdbb81573b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:23.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:23.995+0000 7fdbbea00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdbb815e870 0x7fdbb80b9740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:23.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:23.995+0000 7fdbbea00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fdbb80d0820 con 0x7fdbb80cf870 2024-05-28T06:12:23.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:23.995+0000 7fdbbea00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fdbb80d0540 con 0x7fdbb80d14d0 2024-05-28T06:12:23.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:23.995+0000 7fdbbea00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fdbb80d4fb0 con 0x7fdbb815e870 2024-05-28T06:12:23.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:23.995+0000 7fdbbd600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdbb80cf870 0x7fdbb8157c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:23.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:23.995+0000 7fdbbd600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdbb80cf870 0x7fdbb8157c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44122/0 (socket says 172.21.15.96:44122) 2024-05-28T06:12:23.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:23.995+0000 7fdbbd600640 1 -- 172.21.15.96:0/2340732180 learned_addr learned my addr 172.21.15.96:0/2340732180 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:23.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:23.996+0000 7fdbbd600640 1 -- 172.21.15.96:0/2340732180 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdbb815e870 msgr2=0x7fdbb80b9740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:23.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:23.996+0000 7fdbbd600640 1 --2- 172.21.15.96:0/2340732180 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdbb815e870 0x7fdbb80b9740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:23.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:23.996+0000 7fdbbd600640 1 -- 172.21.15.96:0/2340732180 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdbb80d14d0 msgr2=0x7fdbb81573b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:23.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:23.996+0000 7fdbbd600640 1 --2- 172.21.15.96:0/2340732180 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdbb80d14d0 0x7fdbb81573b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:23.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:23.996+0000 7fdbbd600640 1 -- 172.21.15.96:0/2340732180 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbb8161770 con 0x7fdbb80cf870 2024-05-28T06:12:23.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:23.996+0000 7fdbbd600640 1 --2- 172.21.15.96:0/2340732180 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdbb80cf870 0x7fdbb8157c70 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fdbac00d800 tx=0x7fdbac00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:23.998 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:23.996+0000 7fdbb5600640 1 -- 172.21.15.96:0/2340732180 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbac00b6c0 con 0x7fdbb80cf870 2024-05-28T06:12:23.998 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:23.996+0000 7fdbb5600640 1 -- 172.21.15.96:0/2340732180 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdbac019070 con 0x7fdbb80cf870 2024-05-28T06:12:23.998 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:23.996+0000 7fdbb5600640 1 -- 172.21.15.96:0/2340732180 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbac00ba80 con 0x7fdbb80cf870 2024-05-28T06:12:23.998 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:23.996+0000 7fdbbea00640 1 -- 172.21.15.96:0/2340732180 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdbb80b7340 con 0x7fdbb80cf870 2024-05-28T06:12:23.998 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:23.996+0000 7fdbbea00640 1 -- 172.21.15.96:0/2340732180 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbb80b7900 con 0x7fdbb80cf870 2024-05-28T06:12:23.998 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:23.997+0000 7fdbbea00640 1 -- 172.21.15.96:0/2340732180 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdbb80689e0 con 0x7fdbb80cf870 2024-05-28T06:12:23.998 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:23.998+0000 7fdbb5600640 1 -- 172.21.15.96:0/2340732180 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 16) ==== 91754+0+0 (secure 0 0 0) 0x7fdbac00bc20 con 0x7fdbb80cf870 2024-05-28T06:12:23.999 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:23.999+0000 7fdbb5600640 1 --2- 172.21.15.96:0/2340732180 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdb9c06d100 0x7fdb9c06f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:24.000 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:23.999+0000 7fdbb5600640 1 -- 172.21.15.96:0/2340732180 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 4885+0+0 (secure 0 0 0) 0x7fdbac08ceb0 con 0x7fdbb80cf870 2024-05-28T06:12:24.000 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.000+0000 7fdbbcc00640 1 --2- 172.21.15.96:0/2340732180 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdb9c06d100 0x7fdb9c06f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:24.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.001+0000 7fdbbcc00640 1 --2- 172.21.15.96:0/2340732180 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdb9c06d100 0x7fdb9c06f5c0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fdba8033000 tx=0x7fdba802c000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:24.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.003+0000 7fdbb5600640 1 -- 172.21.15.96:0/2340732180 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fdbac05a680 con 0x7fdbb80cf870 2024-05-28T06:12:24.274 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.274+0000 7fdbbea00640 1 -- 172.21.15.96:0/2340732180 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7fdbb80ba500 con 0x7fdbb80cf870 2024-05-28T06:12:24.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.274+0000 7fdbb5600640 1 -- 172.21.15.96:0/2340732180 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v15) ==== 80+0+5304 (secure 0 0 0) 0x7fdbac059f80 con 0x7fdbb80cf870 2024-05-28T06:12:24.275 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:12:24.275 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:12:24.275 INFO:teuthology.orchestra.run.smithi096.stdout: "auth_dump": [ 2024-05-28T06:12:24.275 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:12:24.276 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "mds.a", 2024-05-28T06:12:24.276 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCWc1VmyA9HHBAANJCF8I/RP5ZYviyOzo0JCQ==", 2024-05-28T06:12:24.276 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:12:24.276 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow", 2024-05-28T06:12:24.276 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow *", 2024-05-28T06:12:24.276 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow *", 2024-05-28T06:12:24.276 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:12:24.276 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:12:24.276 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:12:24.276 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:12:24.276 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "mds.b", 2024-05-28T06:12:24.276 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCWc1VmtR0EOBAA296qGHTmB/2sMztDbDJ56g==", 2024-05-28T06:12:24.276 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:12:24.276 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow", 2024-05-28T06:12:24.277 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow *", 2024-05-28T06:12:24.277 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow *", 2024-05-28T06:12:24.277 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:12:24.277 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:12:24.277 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:12:24.277 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:12:24.277 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "mds.c", 2024-05-28T06:12:24.277 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCWc1VmrLxHKhAAhkTRLDKL6RUJQGetB8JQQA==", 2024-05-28T06:12:24.277 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:12:24.277 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow", 2024-05-28T06:12:24.277 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow *", 2024-05-28T06:12:24.277 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow *", 2024-05-28T06:12:24.277 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:12:24.278 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:12:24.278 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:12:24.278 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:12:24.278 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "mds.d", 2024-05-28T06:12:24.278 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCXc1VmW2aQCxAAnkDwQzlgy39vFe2O9Wwf7Q==", 2024-05-28T06:12:24.278 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:12:24.278 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow", 2024-05-28T06:12:24.278 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow *", 2024-05-28T06:12:24.278 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow *", 2024-05-28T06:12:24.278 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:12:24.278 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:12:24.278 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:12:24.278 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:12:24.278 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "osd.0", 2024-05-28T06:12:24.279 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCbc1VmEcNAAxAAiB5AsfY2uRY1kc6h80chuw==", 2024-05-28T06:12:24.279 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:12:24.279 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow profile osd", 2024-05-28T06:12:24.279 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile osd", 2024-05-28T06:12:24.279 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:12:24.279 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:12:24.279 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:12:24.279 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:12:24.279 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "osd.1", 2024-05-28T06:12:24.279 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCec1Vm6CAaDhAA1NkYM2q4fTsgSDSwJZL3Qg==", 2024-05-28T06:12:24.279 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:12:24.279 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow profile osd", 2024-05-28T06:12:24.279 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile osd", 2024-05-28T06:12:24.280 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:12:24.280 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:12:24.280 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:12:24.280 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:12:24.280 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "osd.2", 2024-05-28T06:12:24.280 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQChc1VmKn1jFxAA+dM4gM7FR8NwBpyd833A/Q==", 2024-05-28T06:12:24.280 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:12:24.280 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow profile osd", 2024-05-28T06:12:24.280 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile osd", 2024-05-28T06:12:24.280 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:12:24.280 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:12:24.280 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:12:24.280 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:12:24.280 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "osd.3", 2024-05-28T06:12:24.281 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCkc1VmV07MIRAAHIXQGDVbL9V5QqexQ89IJA==", 2024-05-28T06:12:24.281 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:12:24.281 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow profile osd", 2024-05-28T06:12:24.281 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile osd", 2024-05-28T06:12:24.281 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:12:24.281 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:12:24.281 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:12:24.281 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:12:24.281 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "osd.4", 2024-05-28T06:12:24.281 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCqc1VmobuZMxAAlY8xhuzqlkuFrcbTouF5rA==", 2024-05-28T06:12:24.281 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:12:24.281 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow profile osd", 2024-05-28T06:12:24.281 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile osd", 2024-05-28T06:12:24.281 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:12:24.282 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:12:24.282 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:12:24.282 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:12:24.282 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "osd.5", 2024-05-28T06:12:24.282 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCuc1Vmwt3QAxAAUFj/e/wJvxljgAbcTb32Ug==", 2024-05-28T06:12:24.282 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:12:24.282 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow profile osd", 2024-05-28T06:12:24.282 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile osd", 2024-05-28T06:12:24.282 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:12:24.282 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:12:24.282 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:12:24.282 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:12:24.282 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "osd.6", 2024-05-28T06:12:24.283 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCxc1VmscerDxAAqEDAqy6/OvdfSJ+ALZ7WeQ==", 2024-05-28T06:12:24.283 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:12:24.283 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow profile osd", 2024-05-28T06:12:24.283 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile osd", 2024-05-28T06:12:24.283 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:12:24.283 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:12:24.283 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:12:24.283 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:12:24.283 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "osd.7", 2024-05-28T06:12:24.283 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQC0c1VmgxkbKhAA7dFzDTBKFmKiFFxPsyiT4Q==", 2024-05-28T06:12:24.283 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:12:24.283 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow profile osd", 2024-05-28T06:12:24.283 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile osd", 2024-05-28T06:12:24.284 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:12:24.284 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:12:24.284 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:12:24.284 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:12:24.284 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "client.0", 2024-05-28T06:12:24.284 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCXc1Vm91hbGBAASTL3HQR8XOs7wVQEHbPl/g==", 2024-05-28T06:12:24.284 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:12:24.284 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow", 2024-05-28T06:12:24.284 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow r", 2024-05-28T06:12:24.284 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-05-28T06:12:24.284 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:12:24.284 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:12:24.284 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:12:24.284 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "client.1", 2024-05-28T06:12:24.285 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCXc1VmIc77KhAAw2e0alTmVROhyCIFW4H3Sg==", 2024-05-28T06:12:24.285 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:12:24.285 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow", 2024-05-28T06:12:24.285 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow r", 2024-05-28T06:12:24.285 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow rw", 2024-05-28T06:12:24.285 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow rwx" 2024-05-28T06:12:24.285 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:12:24.285 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:12:24.285 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:12:24.285 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "client.2", 2024-05-28T06:12:24.285 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCXc1VmCS+OIRAAMV3Au6KuchLhA59QTGbv1w==", 2024-05-28T06:12:24.285 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:12:24.285 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow", 2024-05-28T06:12:24.285 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow r", 2024-05-28T06:12:24.286 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow rw", 2024-05-28T06:12:24.286 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow rwx" 2024-05-28T06:12:24.286 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:12:24.286 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:12:24.286 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:12:24.286 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "client.3", 2024-05-28T06:12:24.286 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCXc1Vm5r3pNBAAkVZ3arjywHQkEP5lmS8xqg==", 2024-05-28T06:12:24.286 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:12:24.286 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow", 2024-05-28T06:12:24.286 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow r", 2024-05-28T06:12:24.286 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow rw", 2024-05-28T06:12:24.286 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow rwx" 2024-05-28T06:12:24.286 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:12:24.287 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:12:24.287 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:12:24.287 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "client.admin", 2024-05-28T06:12:24.287 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCVc1Vm5QaXFhAAQFGKgbZ/kpGisNm2lIw0Uw==", 2024-05-28T06:12:24.287 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:12:24.287 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow *", 2024-05-28T06:12:24.287 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow *", 2024-05-28T06:12:24.287 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow *", 2024-05-28T06:12:24.287 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:12:24.287 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:12:24.287 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:12:24.287 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:12:24.287 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "mgr.x", 2024-05-28T06:12:24.287 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCWc1VmVMx/EhAAylvi37aq3hbMOhUSQWLVSg==", 2024-05-28T06:12:24.288 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:12:24.288 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow *", 2024-05-28T06:12:24.288 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile mgr", 2024-05-28T06:12:24.288 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:12:24.288 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:12:24.288 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:12:24.288 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:12:24.288 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "mgr.y", 2024-05-28T06:12:24.288 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCWc1Vmh5r7BxAAA74gn+FozehM+34eFik8kw==", 2024-05-28T06:12:24.288 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:12:24.288 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow *", 2024-05-28T06:12:24.288 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile mgr", 2024-05-28T06:12:24.288 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:12:24.288 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:12:24.289 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:12:24.289 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-05-28T06:12:24.289 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:12:24.289 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.277+0000 7fdbbea00640 1 -- 172.21.15.96:0/2340732180 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdb9c06d100 msgr2=0x7fdb9c06f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:24.290 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.277+0000 7fdbbea00640 1 --2- 172.21.15.96:0/2340732180 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdb9c06d100 0x7fdb9c06f5c0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fdba8033000 tx=0x7fdba802c000 comp rx=0 tx=0).stop 2024-05-28T06:12:24.290 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.277+0000 7fdbbea00640 1 -- 172.21.15.96:0/2340732180 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdbb80cf870 msgr2=0x7fdbb8157c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:24.290 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.277+0000 7fdbbea00640 1 --2- 172.21.15.96:0/2340732180 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdbb80cf870 0x7fdbb8157c70 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fdbac00d800 tx=0x7fdbac00dcc0 comp rx=0 tx=0).stop 2024-05-28T06:12:24.290 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.277+0000 7fdbbea00640 1 -- 172.21.15.96:0/2340732180 shutdown_connections 2024-05-28T06:12:24.290 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.277+0000 7fdbbea00640 1 --2- 172.21.15.96:0/2340732180 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdb9c06d100 0x7fdb9c06f5c0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:24.290 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.277+0000 7fdbbea00640 1 --2- 172.21.15.96:0/2340732180 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdbb815e870 0x7fdbb80b9740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:24.290 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.277+0000 7fdbbea00640 1 --2- 172.21.15.96:0/2340732180 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdbb80d14d0 0x7fdbb81573b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:24.290 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.277+0000 7fdbbea00640 1 --2- 172.21.15.96:0/2340732180 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdbb80cf870 0x7fdbb8157c70 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:24.290 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.277+0000 7fdbbea00640 1 -- 172.21.15.96:0/2340732180 >> 172.21.15.96:0/2340732180 conn(0x7fdbb808bfc0 msgr2=0x7fdbb806d290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:24.290 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.277+0000 7fdbbea00640 1 -- 172.21.15.96:0/2340732180 shutdown_connections 2024-05-28T06:12:24.290 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.277+0000 7fdbbea00640 1 -- 172.21.15.96:0/2340732180 wait complete. 2024-05-28T06:12:24.298 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:12:24.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.476+0000 7fc409600640 1 Processor -- start 2024-05-28T06:12:24.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.477+0000 7fc409600640 1 -- start start 2024-05-28T06:12:24.479 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.478+0000 7fc409600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fc40406f860 0x7fc4040b3150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:24.479 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.478+0000 7fc409600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fc4040b3690 0x7fc4040d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:24.479 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.478+0000 7fc409600640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc40406ed10 con 0x7fc4040d09e0 2024-05-28T06:12:24.479 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.478+0000 7fc409600640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fc4040d0df0 con 0x7fc40406f860 2024-05-28T06:12:24.479 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.478+0000 7fc409600640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fc40415cb00 con 0x7fc4040b3690 2024-05-28T06:12:24.479 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.478+0000 7fc402a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fc40406f860 0x7fc4040b3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:24.479 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.478+0000 7fc403400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fc4040d09e0 0x7fc404160210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:60122/0 (socket says 172.21.15.96:60122) 2024-05-28T06:12:24.479 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.478+0000 7fc403400640 1 -- 172.21.15.96:0/2889590353 learned_addr learned my addr 172.21.15.96:0/2889590353 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:12:24.480 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.479+0000 7fc3fa000640 1 --2- 172.21.15.96:0/2889590353 >> v2:172.21.15.123:3300/0 conn(0x7fc4040b3690 0x7fc4040d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:24.481 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.479+0000 7fc402a00640 1 -- 172.21.15.96:0/2889590353 >> v2:172.21.15.123:3300/0 conn(0x7fc4040b3690 msgr2=0x7fc4040d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:24.481 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.479+0000 7fc402000640 1 -- 172.21.15.96:0/2889590353 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1121318635 0 0) 0x7fc3fc001490 con 0x7fc4040d09e0 2024-05-28T06:12:24.481 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.479+0000 7fc402a00640 1 --2- 172.21.15.96:0/2889590353 >> v2:172.21.15.123:3300/0 conn(0x7fc4040b3690 0x7fc4040d04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:24.481 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.479+0000 7fc402a00640 1 -- 172.21.15.96:0/2889590353 >> v1:172.21.15.96:6789/0 conn(0x7fc4040d09e0 legacy=0x7fc404160210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:24.481 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.479+0000 7fc402a00640 1 -- 172.21.15.96:0/2889590353 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4040b3cc0 con 0x7fc40406f860 2024-05-28T06:12:24.481 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.479+0000 7fc3fa000640 1 --2- 172.21.15.96:0/2889590353 >> v2:172.21.15.123:3300/0 conn(0x7fc4040b3690 0x7fc4040d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:12:24.481 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.480+0000 7fc402a00640 1 --2- 172.21.15.96:0/2889590353 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc40406f860 0x7fc4040b3150 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fc404070120 tx=0x7fc3f402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed8ac9172f08ab71 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:24.481 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.480+0000 7fc402000640 1 -- 172.21.15.96:0/2889590353 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3f4037070 con 0x7fc40406f860 2024-05-28T06:12:24.481 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.480+0000 7fc402000640 1 -- 172.21.15.96:0/2889590353 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3f4032440 con 0x7fc40406f860 2024-05-28T06:12:24.481 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.480+0000 7fc402000640 1 -- 172.21.15.96:0/2889590353 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3f4032810 con 0x7fc40406f860 2024-05-28T06:12:24.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.481+0000 7fc409600640 1 -- 172.21.15.96:0/2889590353 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc40406f860 msgr2=0x7fc4040b3150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:24.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.481+0000 7fc409600640 1 --2- 172.21.15.96:0/2889590353 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc40406f860 0x7fc4040b3150 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fc404070120 tx=0x7fc3f402a1a0 comp rx=0 tx=0).stop 2024-05-28T06:12:24.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.481+0000 7fc409600640 1 -- 172.21.15.96:0/2889590353 shutdown_connections 2024-05-28T06:12:24.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.481+0000 7fc409600640 1 --2- 172.21.15.96:0/2889590353 >> v2:172.21.15.123:3300/0 conn(0x7fc4040b3690 0x7fc4040d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:24.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.481+0000 7fc409600640 1 --2- 172.21.15.96:0/2889590353 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc40406f860 0x7fc4040b3150 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:24.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.481+0000 7fc409600640 1 -- 172.21.15.96:0/2889590353 >> 172.21.15.96:0/2889590353 conn(0x7fc40406bae0 msgr2=0x7fc40406bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:24.483 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.481+0000 7fc409600640 1 -- 172.21.15.96:0/2889590353 shutdown_connections 2024-05-28T06:12:24.483 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.481+0000 7fc409600640 1 -- 172.21.15.96:0/2889590353 wait complete. 2024-05-28T06:12:24.483 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.483+0000 7fc409600640 1 Processor -- start 2024-05-28T06:12:24.483 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.483+0000 7fc409600640 1 -- start start 2024-05-28T06:12:24.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.484+0000 7fc409600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc40406f860 0x7fc4041f40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:24.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.484+0000 7fc409600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4040b3690 0x7fc40415e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:24.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.484+0000 7fc409600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc4040d09e0 0x7fc40415f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:24.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.484+0000 7fc409600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc4040b4560 con 0x7fc40406f860 2024-05-28T06:12:24.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.484+0000 7fc409600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc4040b4280 con 0x7fc4040b3690 2024-05-28T06:12:24.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.484+0000 7fc409600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fc4040b4840 con 0x7fc4040d09e0 2024-05-28T06:12:24.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.484+0000 7fc402a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc40406f860 0x7fc4041f40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:24.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.484+0000 7fc402a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc40406f860 0x7fc4041f40e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44136/0 (socket says 172.21.15.96:44136) 2024-05-28T06:12:24.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.484+0000 7fc402a00640 1 -- 172.21.15.96:0/3008597844 learned_addr learned my addr 172.21.15.96:0/3008597844 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:24.486 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.485+0000 7fc402a00640 1 -- 172.21.15.96:0/3008597844 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc4040d09e0 msgr2=0x7fc40415f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:24.486 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.485+0000 7fc402a00640 1 --2- 172.21.15.96:0/3008597844 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc4040d09e0 0x7fc40415f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:24.486 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.485+0000 7fc402a00640 1 -- 172.21.15.96:0/3008597844 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4040b3690 msgr2=0x7fc40415e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:24.486 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.485+0000 7fc402a00640 1 --2- 172.21.15.96:0/3008597844 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4040b3690 0x7fc40415e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:24.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.485+0000 7fc402a00640 1 -- 172.21.15.96:0/3008597844 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc40415cf60 con 0x7fc40406f860 2024-05-28T06:12:24.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.485+0000 7fc402a00640 1 --2- 172.21.15.96:0/3008597844 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc40406f860 0x7fc4041f40e0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fc3f4000c00 tx=0x7fc3f40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:24.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.485+0000 7fc3fb400640 1 -- 172.21.15.96:0/3008597844 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3f4037040 con 0x7fc40406f860 2024-05-28T06:12:24.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.485+0000 7fc3fb400640 1 -- 172.21.15.96:0/3008597844 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3f4042070 con 0x7fc40406f860 2024-05-28T06:12:24.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.485+0000 7fc3fb400640 1 -- 172.21.15.96:0/3008597844 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3f4030a50 con 0x7fc40406f860 2024-05-28T06:12:24.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.485+0000 7fc409600640 1 -- 172.21.15.96:0/3008597844 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc40415d240 con 0x7fc40406f860 2024-05-28T06:12:24.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.486+0000 7fc409600640 1 -- 172.21.15.96:0/3008597844 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc40415d800 con 0x7fc40406f860 2024-05-28T06:12:24.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.487+0000 7fc409600640 1 -- 172.21.15.96:0/3008597844 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4040689e0 con 0x7fc40406f860 2024-05-28T06:12:24.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.488+0000 7fc3fb400640 1 -- 172.21.15.96:0/3008597844 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 16) ==== 91754+0+0 (secure 0 0 0) 0x7fc3f4030bf0 con 0x7fc40406f860 2024-05-28T06:12:24.489 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.489+0000 7fc3fb400640 1 --2- 172.21.15.96:0/3008597844 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc3e006d0d0 0x7fc3e006f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:24.489 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.489+0000 7fc3fb400640 1 -- 172.21.15.96:0/3008597844 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 4885+0+0 (secure 0 0 0) 0x7fc3f40ad680 con 0x7fc40406f860 2024-05-28T06:12:24.490 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.490+0000 7fc3fa000640 1 --2- 172.21.15.96:0/3008597844 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc3e006d0d0 0x7fc3e006f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:24.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.491+0000 7fc3fa000640 1 --2- 172.21.15.96:0/3008597844 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc3e006d0d0 0x7fc3e006f590 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fc3ec006fd0 tx=0x7fc3ec008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:24.494 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.493+0000 7fc3fb400640 1 -- 172.21.15.96:0/3008597844 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc3f4045360 con 0x7fc40406f860 2024-05-28T06:12:24.696 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.695+0000 7fc409600640 1 -- 172.21.15.96:0/3008597844 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc40406e2f0 con 0x7fc40406f860 2024-05-28T06:12:24.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.697+0000 7fc3fb400640 1 -- 172.21.15.96:0/3008597844 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) ==== 74+0+12946 (secure 0 0 0) 0x7fc3f407ad30 con 0x7fc40406f860 2024-05-28T06:12:24.697 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:12:24.698 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":56,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:12:23.757512+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:12:24.700 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.699+0000 7fc409600640 1 -- 172.21.15.96:0/3008597844 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc3e006d0d0 msgr2=0x7fc3e006f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:24.700 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.699+0000 7fc409600640 1 --2- 172.21.15.96:0/3008597844 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc3e006d0d0 0x7fc3e006f590 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fc3ec006fd0 tx=0x7fc3ec008040 comp rx=0 tx=0).stop 2024-05-28T06:12:24.700 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.699+0000 7fc409600640 1 -- 172.21.15.96:0/3008597844 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc40406f860 msgr2=0x7fc4041f40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:24.700 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.699+0000 7fc409600640 1 --2- 172.21.15.96:0/3008597844 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc40406f860 0x7fc4041f40e0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fc3f4000c00 tx=0x7fc3f40046c0 comp rx=0 tx=0).stop 2024-05-28T06:12:24.700 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.699+0000 7fc409600640 1 -- 172.21.15.96:0/3008597844 shutdown_connections 2024-05-28T06:12:24.700 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.699+0000 7fc409600640 1 --2- 172.21.15.96:0/3008597844 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc3e006d0d0 0x7fc3e006f590 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:24.700 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.699+0000 7fc409600640 1 --2- 172.21.15.96:0/3008597844 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc4040d09e0 0x7fc40415f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:24.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.699+0000 7fc409600640 1 --2- 172.21.15.96:0/3008597844 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4040b3690 0x7fc40415e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:24.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.699+0000 7fc409600640 1 --2- 172.21.15.96:0/3008597844 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc40406f860 0x7fc4041f40e0 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:24.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.699+0000 7fc409600640 1 -- 172.21.15.96:0/3008597844 >> 172.21.15.96:0/3008597844 conn(0x7fc40406bae0 msgr2=0x7fc404072040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:24.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.700+0000 7fc409600640 1 -- 172.21.15.96:0/3008597844 shutdown_connections 2024-05-28T06:12:24.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.700+0000 7fc409600640 1 -- 172.21.15.96:0/3008597844 wait complete. 2024-05-28T06:12:24.718 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-28T06:03:59.517102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-28T06:12:24.718 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-28T06:12:24.919 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.918+0000 7fae83200640 1 Processor -- start 2024-05-28T06:12:24.919 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.918+0000 7fae83200640 1 -- start start 2024-05-28T06:12:24.920 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.920+0000 7fae83200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fae8415e870 0x7fae840d0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:24.921 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.920+0000 7fae83200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae8415ec50 con 0x7fae840cfa40 2024-05-28T06:12:24.921 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.920+0000 7fae83200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae840cff80 con 0x7fae840d14d0 2024-05-28T06:12:24.921 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.920+0000 7fae83200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fae840d2150 con 0x7fae8415e870 2024-05-28T06:12:24.921 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.920+0000 7fae82800640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fae840cfa40 0x7fae840d25e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:60132/0 (socket says 172.21.15.96:60132) 2024-05-28T06:12:24.921 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.920+0000 7fae82800640 1 -- 172.21.15.96:0/2798722241 learned_addr learned my addr 172.21.15.96:0/2798722241 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:12:24.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.921+0000 7fae81e00640 1 --2- 172.21.15.96:0/2798722241 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fae8415e870 0x7fae840d0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:24.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.921+0000 7fae80a00640 1 -- 172.21.15.96:0/2798722241 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 847559487 0 0) 0x7fae8415ec50 con 0x7fae840cfa40 2024-05-28T06:12:24.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.921+0000 7fae80a00640 1 -- 172.21.15.96:0/2798722241 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fae64003520 con 0x7fae840cfa40 2024-05-28T06:12:24.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.921+0000 7fae80a00640 1 -- 172.21.15.96:0/2798722241 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3124731478 0 0) 0x7fae64003520 con 0x7fae840cfa40 2024-05-28T06:12:24.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.921+0000 7fae80a00640 1 -- 172.21.15.96:0/2798722241 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fae8415e870 msgr2=0x7fae840d0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:24.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.921+0000 7fae80a00640 1 --2- 172.21.15.96:0/2798722241 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fae8415e870 0x7fae840d0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:24.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.921+0000 7fae80a00640 1 -- 172.21.15.96:0/2798722241 >> v1:172.21.15.123:6789/0 conn(0x7fae840d14d0 legacy=0x7fae840cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:24.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.921+0000 7fae80a00640 1 -- 172.21.15.96:0/2798722241 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae840d0260 con 0x7fae840cfa40 2024-05-28T06:12:24.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.921+0000 7fae80a00640 1 -- 172.21.15.96:0/2798722241 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7fae7c001670 con 0x7fae840cfa40 2024-05-28T06:12:24.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.921+0000 7fae80a00640 1 -- 172.21.15.96:0/2798722241 >> v1:172.21.15.96:6789/0 conn(0x7fae840cfa40 legacy=0x7fae840d25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:24.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.921+0000 7fae80a00640 1 --2- 172.21.15.96:0/2798722241 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fae640039e0 0x7fae64003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:24.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.921+0000 7fae80a00640 1 --2- 172.21.15.96:0/2798722241 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fae64004800 0x7fae64004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:24.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.921+0000 7fae80a00640 1 --2- 172.21.15.96:0/2798722241 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fae8415e870 0x7fae640051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:24.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.921+0000 7fae80a00640 1 -- 172.21.15.96:0/2798722241 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fae840d2150 con 0x7fae640039e0 2024-05-28T06:12:24.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.921+0000 7fae80a00640 1 -- 172.21.15.96:0/2798722241 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fae6400ba70 con 0x7fae64004800 2024-05-28T06:12:24.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.921+0000 7fae80a00640 1 -- 172.21.15.96:0/2798722241 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fae6400bc70 con 0x7fae8415e870 2024-05-28T06:12:24.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.922+0000 7fae81e00640 1 --2- 172.21.15.96:0/2798722241 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fae640039e0 0x7fae64003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:24.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.922+0000 7fae82800640 1 --2- 172.21.15.96:0/2798722241 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fae8415e870 0x7fae640051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:24.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.922+0000 7fae81400640 1 --2- 172.21.15.96:0/2798722241 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fae64004800 0x7fae64004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:24.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.922+0000 7fae81e00640 1 -- 172.21.15.96:0/2798722241 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fae8415e870 msgr2=0x7fae640051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:24.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.922+0000 7fae81e00640 1 --2- 172.21.15.96:0/2798722241 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fae8415e870 0x7fae640051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:24.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.922+0000 7fae81e00640 1 -- 172.21.15.96:0/2798722241 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fae64004800 msgr2=0x7fae64004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:24.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.922+0000 7fae81e00640 1 --2- 172.21.15.96:0/2798722241 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fae64004800 0x7fae64004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:24.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.922+0000 7fae81e00640 1 -- 172.21.15.96:0/2798722241 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae64003520 con 0x7fae640039e0 2024-05-28T06:12:24.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.922+0000 7fae81400640 1 --2- 172.21.15.96:0/2798722241 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fae64004800 0x7fae64004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:12:24.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.923+0000 7fae81e00640 1 --2- 172.21.15.96:0/2798722241 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fae640039e0 0x7fae64003e00 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fae74009a90 tx=0x7fae64005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15bc40cd9d943bd9 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:24.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.924+0000 7fae80a00640 1 -- 172.21.15.96:0/2798722241 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae74036070 con 0x7fae640039e0 2024-05-28T06:12:24.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.924+0000 7fae80a00640 1 -- 172.21.15.96:0/2798722241 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae74031400 con 0x7fae640039e0 2024-05-28T06:12:24.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.924+0000 7fae80a00640 1 -- 172.21.15.96:0/2798722241 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae74009690 con 0x7fae640039e0 2024-05-28T06:12:24.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.924+0000 7fae83200640 1 -- 172.21.15.96:0/2798722241 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fae640039e0 msgr2=0x7fae64003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:24.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.924+0000 7fae83200640 1 --2- 172.21.15.96:0/2798722241 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fae640039e0 0x7fae64003e00 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fae74009a90 tx=0x7fae64005830 comp rx=0 tx=0).stop 2024-05-28T06:12:24.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.924+0000 7fae82800640 1 -- 172.21.15.96:0/2798722241 reap_dead start 2024-05-28T06:12:24.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.924+0000 7fae83200640 1 -- 172.21.15.96:0/2798722241 shutdown_connections 2024-05-28T06:12:24.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.924+0000 7fae83200640 1 -- 172.21.15.96:0/2798722241 >> 172.21.15.96:0/2798722241 conn(0x7fae8408bfc0 msgr2=0x7fae840d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:24.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.924+0000 7fae83200640 1 -- 172.21.15.96:0/2798722241 shutdown_connections 2024-05-28T06:12:24.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.924+0000 7fae83200640 1 -- 172.21.15.96:0/2798722241 wait complete. 2024-05-28T06:12:24.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.926+0000 7fae83200640 1 Processor -- start 2024-05-28T06:12:24.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.926+0000 7fae83200640 1 -- start start 2024-05-28T06:12:24.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.928+0000 7fae83200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fae640039e0 0x7fae84159c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:24.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.928+0000 7fae83200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fae8415a770 0x7fae8414f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:24.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.928+0000 7fae83200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fae841f4a50 0x7fae840cd820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:24.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.928+0000 7fae83200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fae840d4fb0 con 0x7fae640039e0 2024-05-28T06:12:24.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.928+0000 7fae83200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fae840d0820 con 0x7fae841f4a50 2024-05-28T06:12:24.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.928+0000 7fae83200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fae840d0f90 con 0x7fae8415a770 2024-05-28T06:12:24.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.928+0000 7fae81e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fae640039e0 0x7fae84159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:24.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.928+0000 7fae81e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fae640039e0 0x7fae84159c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44156/0 (socket says 172.21.15.96:44156) 2024-05-28T06:12:24.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.928+0000 7fae81e00640 1 -- 172.21.15.96:0/143314310 learned_addr learned my addr 172.21.15.96:0/143314310 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:24.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.928+0000 7fae81e00640 1 -- 172.21.15.96:0/143314310 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fae8415a770 msgr2=0x7fae8414f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:24.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.928+0000 7fae81e00640 1 --2- 172.21.15.96:0/143314310 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fae8415a770 0x7fae8414f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:24.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.928+0000 7fae81e00640 1 -- 172.21.15.96:0/143314310 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fae841f4a50 msgr2=0x7fae840cd820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:24.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.928+0000 7fae81e00640 1 --2- 172.21.15.96:0/143314310 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fae841f4a50 0x7fae840cd820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:24.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.928+0000 7fae81e00640 1 -- 172.21.15.96:0/143314310 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae8415ac50 con 0x7fae640039e0 2024-05-28T06:12:24.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.929+0000 7fae81e00640 1 --2- 172.21.15.96:0/143314310 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fae640039e0 0x7fae84159c30 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fae74009a90 tx=0x7fae74004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:24.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.929+0000 7fae7a000640 1 -- 172.21.15.96:0/143314310 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae74040020 con 0x7fae640039e0 2024-05-28T06:12:24.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.929+0000 7fae7a000640 1 -- 172.21.15.96:0/143314310 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae74003bc0 con 0x7fae640039e0 2024-05-28T06:12:24.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.929+0000 7fae7a000640 1 -- 172.21.15.96:0/143314310 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae74036040 con 0x7fae640039e0 2024-05-28T06:12:24.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.929+0000 7fae83200640 1 -- 172.21.15.96:0/143314310 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae84160a30 con 0x7fae640039e0 2024-05-28T06:12:24.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.929+0000 7fae83200640 1 -- 172.21.15.96:0/143314310 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae84160ff0 con 0x7fae640039e0 2024-05-28T06:12:24.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.930+0000 7fae83200640 1 -- 172.21.15.96:0/143314310 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae840689e0 con 0x7fae640039e0 2024-05-28T06:12:24.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.932+0000 7fae7a000640 1 -- 172.21.15.96:0/143314310 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 16) ==== 91754+0+0 (secure 0 0 0) 0x7fae74043020 con 0x7fae640039e0 2024-05-28T06:12:24.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.932+0000 7fae7a000640 1 --2- 172.21.15.96:0/143314310 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fae6006d100 0x7fae6006f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:24.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.933+0000 7fae7a000640 1 -- 172.21.15.96:0/143314310 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 4885+0+0 (secure 0 0 0) 0x7fae740ace20 con 0x7fae640039e0 2024-05-28T06:12:24.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.933+0000 7fae81400640 1 --2- 172.21.15.96:0/143314310 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fae6006d100 0x7fae6006f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:24.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.934+0000 7fae81400640 1 --2- 172.21.15.96:0/143314310 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fae6006d100 0x7fae6006f5c0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fae840ce490 tx=0x7fae6c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:24.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:24.937+0000 7fae7a000640 1 -- 172.21.15.96:0/143314310 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fae7404a030 con 0x7fae640039e0 2024-05-28T06:12:25.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.124+0000 7fae83200640 1 -- 172.21.15.96:0/143314310 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fae840ba500 con 0x7fae640039e0 2024-05-28T06:12:25.125 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.125+0000 7fae7a000640 1 -- 172.21.15.96:0/143314310 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v56) ==== 93+0+10 (secure 0 0 0) 0x7fae740044e0 con 0x7fae640039e0 2024-05-28T06:12:25.125 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 1 2024-05-28T06:12:25.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.127+0000 7fae83200640 1 -- 172.21.15.96:0/143314310 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fae6006d100 msgr2=0x7fae6006f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:25.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.127+0000 7fae83200640 1 --2- 172.21.15.96:0/143314310 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fae6006d100 0x7fae6006f5c0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fae840ce490 tx=0x7fae6c009040 comp rx=0 tx=0).stop 2024-05-28T06:12:25.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.127+0000 7fae83200640 1 -- 172.21.15.96:0/143314310 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fae640039e0 msgr2=0x7fae84159c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:25.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.127+0000 7fae83200640 1 --2- 172.21.15.96:0/143314310 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fae640039e0 0x7fae84159c30 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fae74009a90 tx=0x7fae74004730 comp rx=0 tx=0).stop 2024-05-28T06:12:25.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.127+0000 7fae83200640 1 -- 172.21.15.96:0/143314310 shutdown_connections 2024-05-28T06:12:25.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.127+0000 7fae83200640 1 --2- 172.21.15.96:0/143314310 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fae6006d100 0x7fae6006f5c0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:25.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.127+0000 7fae83200640 1 --2- 172.21.15.96:0/143314310 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fae841f4a50 0x7fae840cd820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:25.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.127+0000 7fae83200640 1 --2- 172.21.15.96:0/143314310 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fae8415a770 0x7fae8414f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:25.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.127+0000 7fae83200640 1 --2- 172.21.15.96:0/143314310 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fae640039e0 0x7fae84159c30 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:25.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.127+0000 7fae83200640 1 -- 172.21.15.96:0/143314310 >> 172.21.15.96:0/143314310 conn(0x7fae8408bfc0 msgr2=0x7fae840d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:25.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.127+0000 7fae83200640 1 -- 172.21.15.96:0/143314310 shutdown_connections 2024-05-28T06:12:25.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.127+0000 7fae83200640 1 -- 172.21.15.96:0/143314310 wait complete. 2024-05-28T06:12:25.145 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-05-28T06:12:25.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.337+0000 7f5c8a000640 1 Processor -- start 2024-05-28T06:12:25.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.337+0000 7f5c8a000640 1 -- start start 2024-05-28T06:12:25.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.338+0000 7f5c8a000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5c8406f860 0x7f5c840b3150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:25.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.338+0000 7f5c8a000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f5c840b3690 0x7f5c840d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:25.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.338+0000 7f5c8a000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f5c840d09e0 0x7f5c84160210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:25.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.338+0000 7f5c8a000640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f5c840d0e50 con 0x7f5c840b3690 2024-05-28T06:12:25.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.338+0000 7f5c8a000640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f5c8406ed10 con 0x7f5c840d09e0 2024-05-28T06:12:25.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.338+0000 7f5c8a000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f5c8415cb00 con 0x7f5c8406f860 2024-05-28T06:12:25.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.339+0000 7f5c7aa00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f5c840b3690 0x7f5c840d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:25.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.339+0000 7f5c7aa00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f5c840b3690 0x7f5c840d04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44168/0 (socket says 172.21.15.96:44168) 2024-05-28T06:12:25.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.339+0000 7f5c7aa00640 1 -- 172.21.15.96:0/4002961627 learned_addr learned my addr 172.21.15.96:0/4002961627 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:25.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.339+0000 7f5c83400640 1 --2- 172.21.15.96:0/4002961627 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5c8406f860 0x7f5c840b3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:25.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.339+0000 7f5c83e00640 1 --2- 172.21.15.96:0/4002961627 >> v2:172.21.15.123:3300/0 conn(0x7f5c840d09e0 0x7f5c84160210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:25.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.339+0000 7f5c7aa00640 1 -- 172.21.15.96:0/4002961627 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5c8406f860 msgr2=0x7f5c840b3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:25.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.339+0000 7f5c7aa00640 1 --2- 172.21.15.96:0/4002961627 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5c8406f860 0x7f5c840b3150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:25.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.339+0000 7f5c7aa00640 1 -- 172.21.15.96:0/4002961627 >> v2:172.21.15.123:3300/0 conn(0x7f5c840d09e0 msgr2=0x7f5c84160210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:25.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.339+0000 7f5c7aa00640 1 --2- 172.21.15.96:0/4002961627 >> v2:172.21.15.123:3300/0 conn(0x7f5c840d09e0 0x7f5c84160210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:25.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.339+0000 7f5c7aa00640 1 -- 172.21.15.96:0/4002961627 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c840b3cc0 con 0x7f5c840b3690 2024-05-28T06:12:25.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.340+0000 7f5c7aa00640 1 --2- 172.21.15.96:0/4002961627 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5c840b3690 0x7f5c840d04a0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f5c8406c900 tx=0x7f5c6c02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5dfd09aca719ba04 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:25.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.341+0000 7f5c82a00640 1 -- 172.21.15.96:0/4002961627 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c6c037070 con 0x7f5c840b3690 2024-05-28T06:12:25.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.341+0000 7f5c82a00640 1 -- 172.21.15.96:0/4002961627 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c6c032440 con 0x7f5c840b3690 2024-05-28T06:12:25.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.341+0000 7f5c82a00640 1 -- 172.21.15.96:0/4002961627 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c6c032810 con 0x7f5c840b3690 2024-05-28T06:12:25.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.341+0000 7f5c8a000640 1 -- 172.21.15.96:0/4002961627 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5c840b3690 msgr2=0x7f5c840d04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:25.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.341+0000 7f5c8a000640 1 --2- 172.21.15.96:0/4002961627 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5c840b3690 0x7f5c840d04a0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f5c8406c900 tx=0x7f5c6c02a1e0 comp rx=0 tx=0).stop 2024-05-28T06:12:25.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.342+0000 7f5c8a000640 1 -- 172.21.15.96:0/4002961627 shutdown_connections 2024-05-28T06:12:25.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.342+0000 7f5c8a000640 1 --2- 172.21.15.96:0/4002961627 >> v2:172.21.15.123:3300/0 conn(0x7f5c840d09e0 0x7f5c84160210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:25.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.342+0000 7f5c8a000640 1 --2- 172.21.15.96:0/4002961627 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5c840b3690 0x7f5c840d04a0 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:25.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.342+0000 7f5c8a000640 1 --2- 172.21.15.96:0/4002961627 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5c8406f860 0x7f5c840b3150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:25.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.342+0000 7f5c8a000640 1 -- 172.21.15.96:0/4002961627 >> 172.21.15.96:0/4002961627 conn(0x7f5c8406bae0 msgr2=0x7f5c8406bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:25.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.342+0000 7f5c8a000640 1 -- 172.21.15.96:0/4002961627 shutdown_connections 2024-05-28T06:12:25.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.342+0000 7f5c8a000640 1 -- 172.21.15.96:0/4002961627 wait complete. 2024-05-28T06:12:25.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.343+0000 7f5c8a000640 1 Processor -- start 2024-05-28T06:12:25.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.343+0000 7f5c8a000640 1 -- start start 2024-05-28T06:12:25.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.344+0000 7f5c8a000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5c8406f860 0x7f5c84098fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:25.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.344+0000 7f5c8a000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c840b3690 0x7f5c840c9b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:25.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.344+0000 7f5c8a000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5c840d09e0 0x7f5c84151e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:25.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.344+0000 7f5c8a000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f5c840b4560 con 0x7f5c8406f860 2024-05-28T06:12:25.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.344+0000 7f5c8a000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5c840b4280 con 0x7f5c840b3690 2024-05-28T06:12:25.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.344+0000 7f5c8a000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f5c840b4840 con 0x7f5c840d09e0 2024-05-28T06:12:25.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.345+0000 7f5c83400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5c8406f860 0x7f5c84098fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:25.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.345+0000 7f5c83400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5c8406f860 0x7f5c84098fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44184/0 (socket says 172.21.15.96:44184) 2024-05-28T06:12:25.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.345+0000 7f5c83400640 1 -- 172.21.15.96:0/194358708 learned_addr learned my addr 172.21.15.96:0/194358708 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:25.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.345+0000 7f5c83e00640 1 --2- 172.21.15.96:0/194358708 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5c840d09e0 0x7f5c84151e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:25.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.345+0000 7f5c7aa00640 1 --2- 172.21.15.96:0/194358708 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c840b3690 0x7f5c840c9b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:25.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.345+0000 7f5c83400640 1 -- 172.21.15.96:0/194358708 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5c840d09e0 msgr2=0x7f5c84151e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:25.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.345+0000 7f5c83400640 1 --2- 172.21.15.96:0/194358708 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5c840d09e0 0x7f5c84151e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:25.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.345+0000 7f5c83400640 1 -- 172.21.15.96:0/194358708 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c840b3690 msgr2=0x7f5c840c9b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:25.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.345+0000 7f5c83400640 1 --2- 172.21.15.96:0/194358708 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c840b3690 0x7f5c840c9b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:25.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.345+0000 7f5c83400640 1 -- 172.21.15.96:0/194358708 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c84152a80 con 0x7f5c8406f860 2024-05-28T06:12:25.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.345+0000 7f5c7aa00640 1 --2- 172.21.15.96:0/194358708 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c840b3690 0x7f5c840c9b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:12:25.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.345+0000 7f5c83400640 1 --2- 172.21.15.96:0/194358708 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5c8406f860 0x7f5c84098fe0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f5c7400ccd0 tx=0x7f5c7400a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:25.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.346+0000 7f5c7be00640 1 -- 172.21.15.96:0/194358708 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c74003990 con 0x7f5c8406f860 2024-05-28T06:12:25.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.346+0000 7f5c7be00640 1 -- 172.21.15.96:0/194358708 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c74003b30 con 0x7f5c8406f860 2024-05-28T06:12:25.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.346+0000 7f5c7be00640 1 -- 172.21.15.96:0/194358708 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c74004020 con 0x7f5c8406f860 2024-05-28T06:12:25.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.346+0000 7f5c8a000640 1 -- 172.21.15.96:0/194358708 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c840c74d0 con 0x7f5c8406f860 2024-05-28T06:12:25.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.346+0000 7f5c8a000640 1 -- 172.21.15.96:0/194358708 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c840c7770 con 0x7f5c8406f860 2024-05-28T06:12:25.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.347+0000 7f5c8a000640 1 -- 172.21.15.96:0/194358708 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c840689e0 con 0x7f5c8406f860 2024-05-28T06:12:25.349 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.348+0000 7f5c7be00640 1 -- 172.21.15.96:0/194358708 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 16) ==== 91754+0+0 (secure 0 0 0) 0x7f5c740041c0 con 0x7f5c8406f860 2024-05-28T06:12:25.349 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.349+0000 7f5c7be00640 1 --2- 172.21.15.96:0/194358708 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5c6006d0d0 0x7f5c6006f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:25.349 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.349+0000 7f5c7be00640 1 -- 172.21.15.96:0/194358708 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 4885+0+0 (secure 0 0 0) 0x7f5c7408d020 con 0x7f5c8406f860 2024-05-28T06:12:25.350 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.350+0000 7f5c7aa00640 1 --2- 172.21.15.96:0/194358708 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5c6006d0d0 0x7f5c6006f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:25.351 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.351+0000 7f5c7aa00640 1 --2- 172.21.15.96:0/194358708 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5c6006d0d0 0x7f5c6006f590 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f5c6c034000 tx=0x7f5c6c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:25.354 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.353+0000 7f5c7be00640 1 -- 172.21.15.96:0/194358708 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5c74014030 con 0x7f5c8406f860 2024-05-28T06:12:25.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.558+0000 7f5c8a000640 1 -- 172.21.15.96:0/194358708 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f5c84073550 con 0x7f5c8406f860 2024-05-28T06:12:25.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.559+0000 7f5c7be00640 1 -- 172.21.15.96:0/194358708 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v56) ==== 84+0+65 (secure 0 0 0) 0x7f5c7405a720 con 0x7f5c8406f860 2024-05-28T06:12:25.560 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:12:25.560 INFO:teuthology.orchestra.run.smithi096.stdout:[ 2024-05-28T06:12:25.560 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:12:25.560 INFO:teuthology.orchestra.run.smithi096.stdout: "poolnum": 1, 2024-05-28T06:12:25.560 INFO:teuthology.orchestra.run.smithi096.stdout: "poolname": ".mgr" 2024-05-28T06:12:25.561 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:12:25.561 INFO:teuthology.orchestra.run.smithi096.stdout:] 2024-05-28T06:12:25.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.562+0000 7f5c8a000640 1 -- 172.21.15.96:0/194358708 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5c6006d0d0 msgr2=0x7f5c6006f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:25.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.562+0000 7f5c8a000640 1 --2- 172.21.15.96:0/194358708 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5c6006d0d0 0x7f5c6006f590 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f5c6c034000 tx=0x7f5c6c02e000 comp rx=0 tx=0).stop 2024-05-28T06:12:25.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.562+0000 7f5c8a000640 1 -- 172.21.15.96:0/194358708 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5c8406f860 msgr2=0x7f5c84098fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:25.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.562+0000 7f5c8a000640 1 --2- 172.21.15.96:0/194358708 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5c8406f860 0x7f5c84098fe0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f5c7400ccd0 tx=0x7f5c7400a5c0 comp rx=0 tx=0).stop 2024-05-28T06:12:25.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.562+0000 7f5c8a000640 1 -- 172.21.15.96:0/194358708 shutdown_connections 2024-05-28T06:12:25.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.562+0000 7f5c8a000640 1 --2- 172.21.15.96:0/194358708 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5c6006d0d0 0x7f5c6006f590 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:25.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.562+0000 7f5c8a000640 1 --2- 172.21.15.96:0/194358708 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5c840d09e0 0x7f5c84151e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:25.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.562+0000 7f5c8a000640 1 --2- 172.21.15.96:0/194358708 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c840b3690 0x7f5c840c9b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:25.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.562+0000 7f5c8a000640 1 --2- 172.21.15.96:0/194358708 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5c8406f860 0x7f5c84098fe0 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:25.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.562+0000 7f5c8a000640 1 -- 172.21.15.96:0/194358708 >> 172.21.15.96:0/194358708 conn(0x7f5c8406bae0 msgr2=0x7f5c840ce5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:25.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.562+0000 7f5c8a000640 1 -- 172.21.15.96:0/194358708 shutdown_connections 2024-05-28T06:12:25.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.562+0000 7f5c8a000640 1 -- 172.21.15.96:0/194358708 wait complete. 2024-05-28T06:12:25.581 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-05-28T06:12:25.581 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-05-28T06:12:25.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.787+0000 7f4d5d600640 1 Processor -- start 2024-05-28T06:12:25.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.787+0000 7f4d5d600640 1 -- start start 2024-05-28T06:12:25.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.789+0000 7f4d5d600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f4d580cfff0 0x7f4d580b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:25.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.789+0000 7f4d5d600640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d580d0500 con 0x7f4d5815e9b0 2024-05-28T06:12:25.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.789+0000 7f4d5d600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d580d07e0 con 0x7f4d580cd2e0 2024-05-28T06:12:25.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.789+0000 7f4d5d600640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f4d580d0ac0 con 0x7f4d580cfff0 2024-05-28T06:12:25.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.789+0000 7f4d56a00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f4d5815e9b0 0x7f4d580ccbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:60148/0 (socket says 172.21.15.96:60148) 2024-05-28T06:12:25.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.789+0000 7f4d56a00640 1 -- 172.21.15.96:0/3603183723 learned_addr learned my addr 172.21.15.96:0/3603183723 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:12:25.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.789+0000 7f4d57400640 1 --2- 172.21.15.96:0/3603183723 >> v2:172.21.15.123:3300/0 conn(0x7f4d580cfff0 0x7f4d580b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:25.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.789+0000 7f4d56000640 1 -- 172.21.15.96:0/3603183723 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2647741336 0 0) 0x7f4d580d0500 con 0x7f4d5815e9b0 2024-05-28T06:12:25.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.789+0000 7f4d56000640 1 -- 172.21.15.96:0/3603183723 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4d3c003520 con 0x7f4d5815e9b0 2024-05-28T06:12:25.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.790+0000 7f4d56000640 1 -- 172.21.15.96:0/3603183723 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3647693805 0 0) 0x7f4d3c003520 con 0x7f4d5815e9b0 2024-05-28T06:12:25.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.790+0000 7f4d56000640 1 -- 172.21.15.96:0/3603183723 >> v2:172.21.15.123:3300/0 conn(0x7f4d580cfff0 msgr2=0x7f4d580b4dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:25.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.790+0000 7f4d56000640 1 --2- 172.21.15.96:0/3603183723 >> v2:172.21.15.123:3300/0 conn(0x7f4d580cfff0 0x7f4d580b4dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:25.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.790+0000 7f4d56000640 1 -- 172.21.15.96:0/3603183723 >> v1:172.21.15.123:6789/0 conn(0x7f4d580cd2e0 legacy=0x7f4d5806e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:25.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.790+0000 7f4d56000640 1 -- 172.21.15.96:0/3603183723 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d580d0da0 con 0x7f4d5815e9b0 2024-05-28T06:12:25.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.790+0000 7f4d56000640 1 -- 172.21.15.96:0/3603183723 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f4d48001670 con 0x7f4d5815e9b0 2024-05-28T06:12:25.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.790+0000 7f4d56000640 1 -- 172.21.15.96:0/3603183723 >> v1:172.21.15.96:6789/0 conn(0x7f4d5815e9b0 legacy=0x7f4d580ccbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:25.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.790+0000 7f4d56000640 1 --2- 172.21.15.96:0/3603183723 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d3c0039b0 0x7f4d3c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:25.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.790+0000 7f4d56000640 1 --2- 172.21.15.96:0/3603183723 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4d3c004800 0x7f4d3c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:25.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.790+0000 7f4d56000640 1 --2- 172.21.15.96:0/3603183723 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4d3c0051c0 0x7f4d3c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:25.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.790+0000 7f4d56000640 1 -- 172.21.15.96:0/3603183723 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4d580d0ac0 con 0x7f4d3c0039b0 2024-05-28T06:12:25.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.790+0000 7f4d56000640 1 -- 172.21.15.96:0/3603183723 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f4d3c00c060 con 0x7f4d3c004800 2024-05-28T06:12:25.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.790+0000 7f4d56000640 1 -- 172.21.15.96:0/3603183723 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f4d3c00c260 con 0x7f4d3c0051c0 2024-05-28T06:12:25.793 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.790+0000 7f4d56a00640 1 --2- 172.21.15.96:0/3603183723 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d3c0039b0 0x7f4d3c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:25.793 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.791+0000 7f4d4fe00640 1 --2- 172.21.15.96:0/3603183723 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4d3c004800 0x7f4d3c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:25.793 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.791+0000 7f4d57400640 1 --2- 172.21.15.96:0/3603183723 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4d3c0051c0 0x7f4d3c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:25.793 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.791+0000 7f4d56a00640 1 -- 172.21.15.96:0/3603183723 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4d3c0051c0 msgr2=0x7f4d3c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:25.793 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.791+0000 7f4d56a00640 1 --2- 172.21.15.96:0/3603183723 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4d3c0051c0 0x7f4d3c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:25.793 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.791+0000 7f4d56a00640 1 -- 172.21.15.96:0/3603183723 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4d3c004800 msgr2=0x7f4d3c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:25.793 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.791+0000 7f4d56a00640 1 --2- 172.21.15.96:0/3603183723 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4d3c004800 0x7f4d3c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:25.793 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.791+0000 7f4d56a00640 1 -- 172.21.15.96:0/3603183723 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d3c003520 con 0x7f4d3c0039b0 2024-05-28T06:12:25.793 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.791+0000 7f4d57400640 1 --2- 172.21.15.96:0/3603183723 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4d3c0051c0 0x7f4d3c00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:12:25.793 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.791+0000 7f4d4fe00640 1 --2- 172.21.15.96:0/3603183723 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4d3c004800 0x7f4d3c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-28T06:12:25.793 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.791+0000 7f4d57400640 1 -- 172.21.15.96:0/3603183723 reap_dead start 2024-05-28T06:12:25.794 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.792+0000 7f4d56a00640 1 --2- 172.21.15.96:0/3603183723 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d3c0039b0 0x7f4d3c003dd0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f4d4800c210 tx=0x7f4d3c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81e43134512625cf server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:25.794 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.793+0000 7f4d56000640 1 -- 172.21.15.96:0/3603183723 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d48007d30 con 0x7f4d3c0039b0 2024-05-28T06:12:25.794 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.793+0000 7f4d56000640 1 -- 172.21.15.96:0/3603183723 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d48005af0 con 0x7f4d3c0039b0 2024-05-28T06:12:25.794 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.793+0000 7f4d56000640 1 -- 172.21.15.96:0/3603183723 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d48007070 con 0x7f4d3c0039b0 2024-05-28T06:12:25.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.793+0000 7f4d5d600640 1 -- 172.21.15.96:0/3603183723 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d3c0039b0 msgr2=0x7f4d3c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:25.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.793+0000 7f4d5d600640 1 --2- 172.21.15.96:0/3603183723 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d3c0039b0 0x7f4d3c003dd0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f4d4800c210 tx=0x7f4d3c005830 comp rx=0 tx=0).stop 2024-05-28T06:12:25.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.793+0000 7f4d5d600640 1 -- 172.21.15.96:0/3603183723 shutdown_connections 2024-05-28T06:12:25.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.793+0000 7f4d5d600640 1 --2- 172.21.15.96:0/3603183723 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d3c0039b0 0x7f4d3c003dd0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:25.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.793+0000 7f4d5d600640 1 -- 172.21.15.96:0/3603183723 >> 172.21.15.96:0/3603183723 conn(0x7f4d580b20a0 msgr2=0x7f4d580b24b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:25.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.793+0000 7f4d5d600640 1 -- 172.21.15.96:0/3603183723 shutdown_connections 2024-05-28T06:12:25.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.793+0000 7f4d5d600640 1 -- 172.21.15.96:0/3603183723 wait complete. 2024-05-28T06:12:25.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.795+0000 7f4d5d600640 1 Processor -- start 2024-05-28T06:12:25.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.795+0000 7f4d5d600640 1 -- start start 2024-05-28T06:12:25.797 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.796+0000 7f4d5d600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4d3c0039b0 0x7f4d58161a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:25.797 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.796+0000 7f4d5d600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d580aed40 0x7f4d580af1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:25.797 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.796+0000 7f4d5d600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4d580c6440 0x7f4d580c68c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:25.797 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.796+0000 7f4d5d600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4d5815ed90 con 0x7f4d580aed40 2024-05-28T06:12:25.797 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.796+0000 7f4d5d600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f4d580cc8f0 con 0x7f4d3c0039b0 2024-05-28T06:12:25.797 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.796+0000 7f4d5d600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f4d5815ef10 con 0x7f4d580c6440 2024-05-28T06:12:25.797 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.796+0000 7f4d4fe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d580aed40 0x7f4d580af1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:25.797 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.797+0000 7f4d4fe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d580aed40 0x7f4d580af1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44206/0 (socket says 172.21.15.96:44206) 2024-05-28T06:12:25.797 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.797+0000 7f4d4fe00640 1 -- 172.21.15.96:0/1360765499 learned_addr learned my addr 172.21.15.96:0/1360765499 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:25.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.797+0000 7f4d4fe00640 1 -- 172.21.15.96:0/1360765499 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4d580c6440 msgr2=0x7f4d580c68c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:25.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.797+0000 7f4d4fe00640 1 --2- 172.21.15.96:0/1360765499 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4d580c6440 0x7f4d580c68c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:25.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.797+0000 7f4d4fe00640 1 -- 172.21.15.96:0/1360765499 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4d3c0039b0 msgr2=0x7f4d58161a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:25.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.797+0000 7f4d4fe00640 1 --2- 172.21.15.96:0/1360765499 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4d3c0039b0 0x7f4d58161a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:25.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.797+0000 7f4d4fe00640 1 -- 172.21.15.96:0/1360765499 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d58161f40 con 0x7f4d580aed40 2024-05-28T06:12:25.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.797+0000 7f4d4fe00640 1 --2- 172.21.15.96:0/1360765499 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d580aed40 0x7f4d580af1c0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f4d4000e3f0 tx=0x7f4d4000e8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:25.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.797+0000 7f4d4ea00640 1 -- 172.21.15.96:0/1360765499 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d40019070 con 0x7f4d580aed40 2024-05-28T06:12:25.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.797+0000 7f4d4ea00640 1 -- 172.21.15.96:0/1360765499 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d400052e0 con 0x7f4d580aed40 2024-05-28T06:12:25.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.797+0000 7f4d4ea00640 1 -- 172.21.15.96:0/1360765499 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d40003830 con 0x7f4d580aed40 2024-05-28T06:12:25.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.798+0000 7f4d5d600640 1 -- 172.21.15.96:0/1360765499 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d5815f090 con 0x7f4d580aed40 2024-05-28T06:12:25.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.798+0000 7f4d5d600640 1 -- 172.21.15.96:0/1360765499 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d580c8200 con 0x7f4d580aed40 2024-05-28T06:12:25.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.799+0000 7f4d5d600640 1 -- 172.21.15.96:0/1360765499 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d580757d0 con 0x7f4d580aed40 2024-05-28T06:12:25.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.800+0000 7f4d4ea00640 1 -- 172.21.15.96:0/1360765499 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 16) ==== 91754+0+0 (secure 0 0 0) 0x7f4d4001c070 con 0x7f4d580aed40 2024-05-28T06:12:25.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.800+0000 7f4d4ea00640 1 --2- 172.21.15.96:0/1360765499 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4d3406d0d0 0x7f4d3406f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:25.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.801+0000 7f4d4ea00640 1 -- 172.21.15.96:0/1360765499 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 4885+0+0 (secure 0 0 0) 0x7f4d4008bb30 con 0x7f4d580aed40 2024-05-28T06:12:25.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.801+0000 7f4d56a00640 1 --2- 172.21.15.96:0/1360765499 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4d3406d0d0 0x7f4d3406f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:25.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.802+0000 7f4d56a00640 1 --2- 172.21.15.96:0/1360765499 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4d3406d0d0 0x7f4d3406f590 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f4d48032000 tx=0x7f4d48009030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:25.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:25.805+0000 7f4d4ea00640 1 -- 172.21.15.96:0/1360765499 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4d40059030 con 0x7f4d580aed40 2024-05-28T06:12:26.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:26.006+0000 7f4d5d600640 1 -- 172.21.15.96:0/1360765499 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f4d5806e440 con 0x7f4d580aed40 2024-05-28T06:12:26.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:26.787+0000 7f4d4ea00640 1 -- 172.21.15.96:0/1360765499 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v57) ==== 138+0+0 (secure 0 0 0) 0x7f4d4005d330 con 0x7f4d580aed40 2024-05-28T06:12:26.789 INFO:teuthology.orchestra.run.smithi096.stderr:pool 'cephfs_metadata' created 2024-05-28T06:12:26.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:26.791+0000 7f4d5d600640 1 -- 172.21.15.96:0/1360765499 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4d3406d0d0 msgr2=0x7f4d3406f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:26.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:26.791+0000 7f4d5d600640 1 --2- 172.21.15.96:0/1360765499 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4d3406d0d0 0x7f4d3406f590 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f4d48032000 tx=0x7f4d48009030 comp rx=0 tx=0).stop 2024-05-28T06:12:26.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:26.791+0000 7f4d5d600640 1 -- 172.21.15.96:0/1360765499 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d580aed40 msgr2=0x7f4d580af1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:26.793 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:26.791+0000 7f4d5d600640 1 --2- 172.21.15.96:0/1360765499 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d580aed40 0x7f4d580af1c0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f4d4000e3f0 tx=0x7f4d4000e8b0 comp rx=0 tx=0).stop 2024-05-28T06:12:26.793 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:26.791+0000 7f4d5d600640 1 -- 172.21.15.96:0/1360765499 shutdown_connections 2024-05-28T06:12:26.793 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:26.791+0000 7f4d5d600640 1 --2- 172.21.15.96:0/1360765499 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4d3406d0d0 0x7f4d3406f590 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:26.793 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:26.791+0000 7f4d5d600640 1 --2- 172.21.15.96:0/1360765499 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4d580c6440 0x7f4d580c68c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:26.793 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:26.791+0000 7f4d5d600640 1 --2- 172.21.15.96:0/1360765499 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d580aed40 0x7f4d580af1c0 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:26.793 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:26.791+0000 7f4d5d600640 1 --2- 172.21.15.96:0/1360765499 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4d3c0039b0 0x7f4d58161a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:26.793 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:26.791+0000 7f4d5d600640 1 -- 172.21.15.96:0/1360765499 >> 172.21.15.96:0/1360765499 conn(0x7f4d580b20a0 msgr2=0x7f4d5815de20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:26.793 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:26.791+0000 7f4d5d600640 1 -- 172.21.15.96:0/1360765499 shutdown_connections 2024-05-28T06:12:26.794 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:26.792+0000 7f4d5d600640 1 -- 172.21.15.96:0/1360765499 wait complete. 2024-05-28T06:12:26.818 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-05-28T06:12:27.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:27.012+0000 7f17f5200640 1 Processor -- start 2024-05-28T06:12:27.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:27.012+0000 7f17f5200640 1 -- start start 2024-05-28T06:12:27.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:27.013+0000 7f17f5200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f17f0161a90 0x7f17f006e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:27.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:27.013+0000 7f17f5200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f17f0161e70 0x7f17f00b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:27.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:27.013+0000 7f17f5200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17f00b3d40 con 0x7f17f006e9b0 2024-05-28T06:12:27.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:27.013+0000 7f17f5200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f17f00b4020 con 0x7f17f0161e70 2024-05-28T06:12:27.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:27.013+0000 7f17f5200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f17f00b4300 con 0x7f17f0161a90 2024-05-28T06:12:27.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:27.014+0000 7f17e5400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f17f0161e70 0x7f17f00b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:27.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:27.014+0000 7f17e5400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f17f0161e70 0x7f17f00b32c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44210/0 (socket says 172.21.15.96:44210) 2024-05-28T06:12:27.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:27.014+0000 7f17e5400640 1 -- 172.21.15.96:0/1149695166 learned_addr learned my addr 172.21.15.96:0/1149695166 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:27.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:27.014+0000 7f17ede00640 1 --2- 172.21.15.96:0/1149695166 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f17f0161a90 0x7f17f006e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:27.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:27.014+0000 7f17e5400640 1 -- 172.21.15.96:0/1149695166 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f17f0161a90 msgr2=0x7f17f006e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:27.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:27.014+0000 7f17e5400640 1 --2- 172.21.15.96:0/1149695166 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f17f0161a90 0x7f17f006e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:27.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:27.014+0000 7f17e5400640 1 -- 172.21.15.96:0/1149695166 >> v1:172.21.15.123:6789/0 conn(0x7f17f006e9b0 legacy=0x7f17f00ad2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:27.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:27.014+0000 7f17e5400640 1 -- 172.21.15.96:0/1149695166 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17f006dc70 con 0x7f17f0161e70 2024-05-28T06:12:27.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:27.015+0000 7f17e5400640 1 --2- 172.21.15.96:0/1149695166 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f17f0161e70 0x7f17f00b32c0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f17d8009870 tx=0x7f17d802a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfcc404635f46cd server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:27.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:27.016+0000 7f17ed400640 1 -- 172.21.15.96:0/1149695166 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17d8037070 con 0x7f17f0161e70 2024-05-28T06:12:27.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:27.016+0000 7f17ed400640 1 -- 172.21.15.96:0/1149695166 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17d8032440 con 0x7f17f0161e70 2024-05-28T06:12:27.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:27.016+0000 7f17ed400640 1 -- 172.21.15.96:0/1149695166 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17d8032810 con 0x7f17f0161e70 2024-05-28T06:12:27.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:27.016+0000 7f17f5200640 1 -- 172.21.15.96:0/1149695166 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f17f0161e70 msgr2=0x7f17f00b32c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:27.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:27.016+0000 7f17f5200640 1 --2- 172.21.15.96:0/1149695166 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f17f0161e70 0x7f17f00b32c0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f17d8009870 tx=0x7f17d802a210 comp rx=0 tx=0).stop 2024-05-28T06:12:27.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:27.016+0000 7f17f5200640 1 -- 172.21.15.96:0/1149695166 shutdown_connections 2024-05-28T06:12:27.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:27.016+0000 7f17f5200640 1 --2- 172.21.15.96:0/1149695166 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f17f0161e70 0x7f17f00b32c0 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:27.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:27.016+0000 7f17f5200640 1 --2- 172.21.15.96:0/1149695166 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f17f0161a90 0x7f17f006e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:27.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:27.016+0000 7f17f5200640 1 -- 172.21.15.96:0/1149695166 >> 172.21.15.96:0/1149695166 conn(0x7f17f006bd20 msgr2=0x7f17f006c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:27.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:27.016+0000 7f17f5200640 1 -- 172.21.15.96:0/1149695166 shutdown_connections 2024-05-28T06:12:27.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:27.016+0000 7f17f5200640 1 -- 172.21.15.96:0/1149695166 wait complete. 2024-05-28T06:12:27.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:27.018+0000 7f17f5200640 1 Processor -- start 2024-05-28T06:12:27.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:27.019+0000 7f17f5200640 1 -- start start 2024-05-28T06:12:27.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:27.020+0000 7f17f5200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f17f006e9b0 0x7f17f0155000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:27.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:27.020+0000 7f17f5200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f17f0161a90 0x7f17f014c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:27.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:27.020+0000 7f17f5200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f17f0161e70 0x7f17f00b0f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:27.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:27.020+0000 7f17f5200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f17f00ad9b0 con 0x7f17f0161a90 2024-05-28T06:12:27.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:27.020+0000 7f17f5200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f17f00b5020 con 0x7f17f0161e70 2024-05-28T06:12:27.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:27.020+0000 7f17f5200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f17f00adb30 con 0x7f17f006e9b0 2024-05-28T06:12:27.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:27.020+0000 7f17e5400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f17f0161a90 0x7f17f014c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:27.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:27.020+0000 7f17e5400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f17f0161a90 0x7f17f014c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44222/0 (socket says 172.21.15.96:44222) 2024-05-28T06:12:27.022 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:27.020+0000 7f17e5400640 1 -- 172.21.15.96:0/2238246954 learned_addr learned my addr 172.21.15.96:0/2238246954 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:27.022 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:27.020+0000 7f17ede00640 1 --2- 172.21.15.96:0/2238246954 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f17f006e9b0 0x7f17f0155000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:27.022 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:27.020+0000 7f17ee800640 1 --2- 172.21.15.96:0/2238246954 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f17f0161e70 0x7f17f00b0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:27.022 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:27.021+0000 7f17e5400640 1 -- 172.21.15.96:0/2238246954 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f17f006e9b0 msgr2=0x7f17f0155000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:27.022 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:27.021+0000 7f17e5400640 1 --2- 172.21.15.96:0/2238246954 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f17f006e9b0 0x7f17f0155000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:27.022 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:27.021+0000 7f17e5400640 1 -- 172.21.15.96:0/2238246954 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f17f0161e70 msgr2=0x7f17f00b0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:27.022 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:27.021+0000 7f17e5400640 1 --2- 172.21.15.96:0/2238246954 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f17f0161e70 0x7f17f00b0f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:27.022 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:27.021+0000 7f17e5400640 1 -- 172.21.15.96:0/2238246954 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17f0159c90 con 0x7f17f0161a90 2024-05-28T06:12:27.022 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:27.021+0000 7f17ede00640 1 --2- 172.21.15.96:0/2238246954 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f17f006e9b0 0x7f17f0155000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:12:27.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:27.021+0000 7f17e5400640 1 --2- 172.21.15.96:0/2238246954 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f17f0161a90 0x7f17f014c8d0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f17d80090b0 tx=0x7f17d80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:27.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:27.021+0000 7f17e6a00640 1 -- 172.21.15.96:0/2238246954 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17d8041020 con 0x7f17f0161a90 2024-05-28T06:12:27.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:27.021+0000 7f17e6a00640 1 -- 172.21.15.96:0/2238246954 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17d80306c0 con 0x7f17f0161a90 2024-05-28T06:12:27.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:27.021+0000 7f17e6a00640 1 -- 172.21.15.96:0/2238246954 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17d8037040 con 0x7f17f0161a90 2024-05-28T06:12:27.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:27.021+0000 7f17f5200640 1 -- 172.21.15.96:0/2238246954 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17f015f7f0 con 0x7f17f0161a90 2024-05-28T06:12:27.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:27.022+0000 7f17f5200640 1 -- 172.21.15.96:0/2238246954 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17f0160160 con 0x7f17f0161a90 2024-05-28T06:12:27.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:27.023+0000 7f17f5200640 1 -- 172.21.15.96:0/2238246954 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17f00689e0 con 0x7f17f0161a90 2024-05-28T06:12:27.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:27.025+0000 7f17e6a00640 1 -- 172.21.15.96:0/2238246954 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 16) ==== 91754+0+0 (secure 0 0 0) 0x7f17d8044020 con 0x7f17f0161a90 2024-05-28T06:12:27.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:27.026+0000 7f17e6a00640 1 --2- 172.21.15.96:0/2238246954 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f17cc06d100 0x7f17cc06f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:27.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:27.026+0000 7f17e6a00640 1 -- 172.21.15.96:0/2238246954 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5255+0+0 (secure 0 0 0) 0x7f17d80ad720 con 0x7f17f0161a90 2024-05-28T06:12:27.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:27.027+0000 7f17ede00640 1 --2- 172.21.15.96:0/2238246954 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f17cc06d100 0x7f17cc06f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:27.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:27.028+0000 7f17ede00640 1 --2- 172.21.15.96:0/2238246954 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f17cc06d100 0x7f17cc06f5c0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f17e0007330 tx=0x7f17e000b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:27.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:27.030+0000 7f17e6a00640 1 -- 172.21.15.96:0/2238246954 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f17d804b030 con 0x7f17f0161a90 2024-05-28T06:12:27.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:27.237+0000 7f17f5200640 1 -- 172.21.15.96:0/2238246954 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f17f006cc20 con 0x7f17f0161a90 2024-05-28T06:12:27.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:27.794+0000 7f17e6a00640 1 -- 172.21.15.96:0/2238246954 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v58) ==== 170+0+0 (secure 0 0 0) 0x7f17d807aec0 con 0x7f17f0161a90 2024-05-28T06:12:27.796 INFO:teuthology.orchestra.run.smithi096.stderr:pool 'cephfs_data' created 2024-05-28T06:12:27.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:27.798+0000 7f17f5200640 1 -- 172.21.15.96:0/2238246954 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f17cc06d100 msgr2=0x7f17cc06f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:27.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:27.798+0000 7f17f5200640 1 --2- 172.21.15.96:0/2238246954 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f17cc06d100 0x7f17cc06f5c0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f17e0007330 tx=0x7f17e000b040 comp rx=0 tx=0).stop 2024-05-28T06:12:27.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:27.798+0000 7f17f5200640 1 -- 172.21.15.96:0/2238246954 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f17f0161a90 msgr2=0x7f17f014c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:27.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:27.798+0000 7f17f5200640 1 --2- 172.21.15.96:0/2238246954 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f17f0161a90 0x7f17f014c8d0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f17d80090b0 tx=0x7f17d80046c0 comp rx=0 tx=0).stop 2024-05-28T06:12:27.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:27.798+0000 7f17f5200640 1 -- 172.21.15.96:0/2238246954 shutdown_connections 2024-05-28T06:12:27.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:27.798+0000 7f17f5200640 1 --2- 172.21.15.96:0/2238246954 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f17cc06d100 0x7f17cc06f5c0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:27.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:27.798+0000 7f17f5200640 1 --2- 172.21.15.96:0/2238246954 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f17f0161e70 0x7f17f00b0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:27.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:27.798+0000 7f17f5200640 1 --2- 172.21.15.96:0/2238246954 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f17f0161a90 0x7f17f014c8d0 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:27.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:27.798+0000 7f17f5200640 1 --2- 172.21.15.96:0/2238246954 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f17f006e9b0 0x7f17f0155000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:27.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:27.798+0000 7f17f5200640 1 -- 172.21.15.96:0/2238246954 >> 172.21.15.96:0/2238246954 conn(0x7f17f006bd20 msgr2=0x7f17f00aeef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:27.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:27.798+0000 7f17f5200640 1 -- 172.21.15.96:0/2238246954 shutdown_connections 2024-05-28T06:12:27.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:27.799+0000 7f17f5200640 1 -- 172.21.15.96:0/2238246954 wait complete. 2024-05-28T06:12:27.831 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-05-28T06:12:28.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.008+0000 7fb619000640 1 Processor -- start 2024-05-28T06:12:28.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.009+0000 7fb619000640 1 -- start start 2024-05-28T06:12:28.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.010+0000 7fb619000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb61415d270 0x7fb6140b4d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:28.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.010+0000 7fb619000640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb61415d900 con 0x7fb6140cd2a0 2024-05-28T06:12:28.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.010+0000 7fb619000640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6140b3f90 con 0x7fb614153510 2024-05-28T06:12:28.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.010+0000 7fb619000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fb6141538f0 con 0x7fb61415d270 2024-05-28T06:12:28.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.010+0000 7fb60be00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fb6140cd2a0 0x7fb61415cb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:60164/0 (socket says 172.21.15.96:60164) 2024-05-28T06:12:28.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.010+0000 7fb60be00640 1 -- 172.21.15.96:0/4033201019 learned_addr learned my addr 172.21.15.96:0/4033201019 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:12:28.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.011+0000 7fb612a00640 1 -- 172.21.15.96:0/4033201019 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2556693079 0 0) 0x7fb61415d900 con 0x7fb6140cd2a0 2024-05-28T06:12:28.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.011+0000 7fb612a00640 1 -- 172.21.15.96:0/4033201019 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb5f8003520 con 0x7fb6140cd2a0 2024-05-28T06:12:28.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.011+0000 7fb613e00640 1 --2- 172.21.15.96:0/4033201019 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb61415d270 0x7fb6140b4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:28.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.011+0000 7fb612a00640 1 -- 172.21.15.96:0/4033201019 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1520020192 0 0) 0x7fb5f8003520 con 0x7fb6140cd2a0 2024-05-28T06:12:28.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.011+0000 7fb612a00640 1 -- 172.21.15.96:0/4033201019 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb61415d270 msgr2=0x7fb6140b4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:28.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.011+0000 7fb612a00640 1 --2- 172.21.15.96:0/4033201019 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb61415d270 0x7fb6140b4d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:28.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.011+0000 7fb612a00640 1 -- 172.21.15.96:0/4033201019 >> v1:172.21.15.123:6789/0 conn(0x7fb614153510 legacy=0x7fb6140ccb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:28.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.011+0000 7fb612a00640 1 -- 172.21.15.96:0/4033201019 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6140b4270 con 0x7fb6140cd2a0 2024-05-28T06:12:28.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.011+0000 7fb612a00640 1 -- 172.21.15.96:0/4033201019 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7fb5fc001670 con 0x7fb6140cd2a0 2024-05-28T06:12:28.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.011+0000 7fb612a00640 1 -- 172.21.15.96:0/4033201019 >> v1:172.21.15.96:6789/0 conn(0x7fb6140cd2a0 legacy=0x7fb61415cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:28.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.011+0000 7fb612a00640 1 --2- 172.21.15.96:0/4033201019 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb61415d270 0x7fb5f80039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:28.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.011+0000 7fb612a00640 1 --2- 172.21.15.96:0/4033201019 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb5f8004010 0x7fb5f8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:28.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.011+0000 7fb612a00640 1 --2- 172.21.15.96:0/4033201019 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb5f8004d40 0x7fb5f80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:28.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.011+0000 7fb612a00640 1 -- 172.21.15.96:0/4033201019 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb5f800bbd0 con 0x7fb5f8004010 2024-05-28T06:12:28.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.011+0000 7fb612a00640 1 -- 172.21.15.96:0/4033201019 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fb5f800bd80 con 0x7fb5f8004d40 2024-05-28T06:12:28.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.012+0000 7fb612a00640 1 -- 172.21.15.96:0/4033201019 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fb5f800bf80 con 0x7fb61415d270 2024-05-28T06:12:28.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.012+0000 7fb60be00640 1 --2- 172.21.15.96:0/4033201019 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb5f8004010 0x7fb5f8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:28.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.012+0000 7fb613400640 1 --2- 172.21.15.96:0/4033201019 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb61415d270 0x7fb5f80039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:28.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.012+0000 7fb60be00640 1 -- 172.21.15.96:0/4033201019 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb61415d270 msgr2=0x7fb5f80039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:28.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.012+0000 7fb60be00640 1 --2- 172.21.15.96:0/4033201019 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb61415d270 0x7fb5f80039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:28.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.012+0000 7fb60be00640 1 -- 172.21.15.96:0/4033201019 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb5f8004d40 msgr2=0x7fb5f80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:28.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.012+0000 7fb60be00640 1 --2- 172.21.15.96:0/4033201019 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb5f8004d40 0x7fb5f80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:28.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.012+0000 7fb60be00640 1 -- 172.21.15.96:0/4033201019 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5f8003520 con 0x7fb5f8004010 2024-05-28T06:12:28.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.013+0000 7fb60be00640 1 --2- 172.21.15.96:0/4033201019 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb5f8004010 0x7fb5f8004800 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc00c210 tx=0x7fb5f8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26c80ef4157bd99c server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:28.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.013+0000 7fb612a00640 1 -- 172.21.15.96:0/4033201019 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5fc007d30 con 0x7fb5f8004010 2024-05-28T06:12:28.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.013+0000 7fb612a00640 1 -- 172.21.15.96:0/4033201019 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5fc005ad0 con 0x7fb5f8004010 2024-05-28T06:12:28.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.013+0000 7fb612a00640 1 -- 172.21.15.96:0/4033201019 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5fc007070 con 0x7fb5f8004010 2024-05-28T06:12:28.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.014+0000 7fb619000640 1 -- 172.21.15.96:0/4033201019 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb5f8004010 msgr2=0x7fb5f8004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:28.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.014+0000 7fb619000640 1 --2- 172.21.15.96:0/4033201019 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb5f8004010 0x7fb5f8004800 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc00c210 tx=0x7fb5f8005830 comp rx=0 tx=0).stop 2024-05-28T06:12:28.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.014+0000 7fb613e00640 1 -- 172.21.15.96:0/4033201019 reap_dead start 2024-05-28T06:12:28.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.014+0000 7fb619000640 1 -- 172.21.15.96:0/4033201019 shutdown_connections 2024-05-28T06:12:28.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.014+0000 7fb619000640 1 -- 172.21.15.96:0/4033201019 >> 172.21.15.96:0/4033201019 conn(0x7fb61406b900 msgr2=0x7fb61406bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:28.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.014+0000 7fb619000640 1 -- 172.21.15.96:0/4033201019 shutdown_connections 2024-05-28T06:12:28.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.014+0000 7fb619000640 1 -- 172.21.15.96:0/4033201019 wait complete. 2024-05-28T06:12:28.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.016+0000 7fb619000640 1 Processor -- start 2024-05-28T06:12:28.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.016+0000 7fb619000640 1 -- start start 2024-05-28T06:12:28.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.017+0000 7fb619000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb5f8004010 0x7fb614150180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:28.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.017+0000 7fb619000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb6141512f0 0x7fb614151770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:28.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.017+0000 7fb619000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb6140cdd50 0x7fb6140ce1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:28.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.017+0000 7fb619000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb6140b48c0 con 0x7fb6140cdd50 2024-05-28T06:12:28.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.017+0000 7fb619000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fb61415d6c0 con 0x7fb5f8004010 2024-05-28T06:12:28.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.017+0000 7fb619000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fb6140b4d50 con 0x7fb6141512f0 2024-05-28T06:12:28.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.017+0000 7fb613e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb6140cdd50 0x7fb6140ce1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:28.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.017+0000 7fb613e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb6140cdd50 0x7fb6140ce1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44238/0 (socket says 172.21.15.96:44238) 2024-05-28T06:12:28.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.017+0000 7fb613e00640 1 -- 172.21.15.96:0/3927892039 learned_addr learned my addr 172.21.15.96:0/3927892039 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:28.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.018+0000 7fb613400640 1 --2- 172.21.15.96:0/3927892039 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb5f8004010 0x7fb614150180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:28.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.018+0000 7fb60be00640 1 --2- 172.21.15.96:0/3927892039 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb6141512f0 0x7fb614151770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:28.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.018+0000 7fb613e00640 1 -- 172.21.15.96:0/3927892039 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb6141512f0 msgr2=0x7fb614151770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:28.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.018+0000 7fb613e00640 1 --2- 172.21.15.96:0/3927892039 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb6141512f0 0x7fb614151770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:28.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.018+0000 7fb613e00640 1 -- 172.21.15.96:0/3927892039 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb5f8004010 msgr2=0x7fb614150180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:28.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.018+0000 7fb613e00640 1 --2- 172.21.15.96:0/3927892039 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb5f8004010 0x7fb614150180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:28.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.018+0000 7fb613e00640 1 -- 172.21.15.96:0/3927892039 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6140c9f60 con 0x7fb6140cdd50 2024-05-28T06:12:28.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.018+0000 7fb613e00640 1 --2- 172.21.15.96:0/3927892039 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb6140cdd50 0x7fb6140ce1d0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fb6140b3c10 tx=0x7fb60c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:28.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.018+0000 7fb60b400640 1 -- 172.21.15.96:0/3927892039 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb60c00eb90 con 0x7fb6140cdd50 2024-05-28T06:12:28.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.018+0000 7fb60b400640 1 -- 172.21.15.96:0/3927892039 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb60c00ed50 con 0x7fb6140cdd50 2024-05-28T06:12:28.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.018+0000 7fb619000640 1 -- 172.21.15.96:0/3927892039 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6141f80c0 con 0x7fb6140cdd50 2024-05-28T06:12:28.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.019+0000 7fb60b400640 1 -- 172.21.15.96:0/3927892039 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb60c003bd0 con 0x7fb6140cdd50 2024-05-28T06:12:28.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.019+0000 7fb619000640 1 -- 172.21.15.96:0/3927892039 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb614150810 con 0x7fb6140cdd50 2024-05-28T06:12:28.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.020+0000 7fb619000640 1 -- 172.21.15.96:0/3927892039 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6140689e0 con 0x7fb6140cdd50 2024-05-28T06:12:28.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.021+0000 7fb60b400640 1 -- 172.21.15.96:0/3927892039 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 16) ==== 91754+0+0 (secure 0 0 0) 0x7fb60c01b020 con 0x7fb6140cdd50 2024-05-28T06:12:28.022 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.022+0000 7fb60b400640 1 --2- 172.21.15.96:0/3927892039 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb5f006d100 0x7fb5f006f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:28.022 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.022+0000 7fb60b400640 1 -- 172.21.15.96:0/3927892039 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5637+0+0 (secure 0 0 0) 0x7fb60c08cbe0 con 0x7fb6140cdd50 2024-05-28T06:12:28.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.023+0000 7fb613400640 1 --2- 172.21.15.96:0/3927892039 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb5f006d100 0x7fb5f006f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:28.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.024+0000 7fb613400640 1 --2- 172.21.15.96:0/3927892039 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb5f006d100 0x7fb5f006f5c0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fb604007fd0 tx=0x7fb604009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:28.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.027+0000 7fb60b400640 1 -- 172.21.15.96:0/3927892039 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb60c05a0c0 con 0x7fb6140cdd50 2024-05-28T06:12:28.267 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.267+0000 7fb619000640 1 -- 172.21.15.96:0/3927892039 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fb6140b3c40 con 0x7fb6140cdd50 2024-05-28T06:12:28.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.837+0000 7fb60b400640 1 -- 172.21.15.96:0/3927892039 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 8 and data pool 9 v30) ==== 172+0+0 (secure 0 0 0) 0x7fb60c0599c0 con 0x7fb6140cdd50 2024-05-28T06:12:28.840 INFO:teuthology.orchestra.run.smithi096.stderr:new fs with metadata pool 8 and data pool 9 2024-05-28T06:12:28.850 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.849+0000 7fb619000640 1 -- 172.21.15.96:0/3927892039 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb5f006d100 msgr2=0x7fb5f006f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:28.850 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.849+0000 7fb619000640 1 --2- 172.21.15.96:0/3927892039 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb5f006d100 0x7fb5f006f5c0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fb604007fd0 tx=0x7fb604009040 comp rx=0 tx=0).stop 2024-05-28T06:12:28.850 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.849+0000 7fb619000640 1 -- 172.21.15.96:0/3927892039 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb6140cdd50 msgr2=0x7fb6140ce1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:28.850 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.849+0000 7fb619000640 1 --2- 172.21.15.96:0/3927892039 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb6140cdd50 0x7fb6140ce1d0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fb6140b3c10 tx=0x7fb60c00e4b0 comp rx=0 tx=0).stop 2024-05-28T06:12:28.851 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.850+0000 7fb619000640 1 -- 172.21.15.96:0/3927892039 shutdown_connections 2024-05-28T06:12:28.851 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.850+0000 7fb619000640 1 --2- 172.21.15.96:0/3927892039 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb5f006d100 0x7fb5f006f5c0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:28.851 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.850+0000 7fb619000640 1 --2- 172.21.15.96:0/3927892039 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb6140cdd50 0x7fb6140ce1d0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:28.851 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.850+0000 7fb619000640 1 --2- 172.21.15.96:0/3927892039 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb6141512f0 0x7fb614151770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:28.851 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.850+0000 7fb619000640 1 --2- 172.21.15.96:0/3927892039 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb5f8004010 0x7fb614150180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:28.851 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.850+0000 7fb619000640 1 -- 172.21.15.96:0/3927892039 >> 172.21.15.96:0/3927892039 conn(0x7fb61406b900 msgr2=0x7fb6141527e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:28.851 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.850+0000 7fb619000640 1 -- 172.21.15.96:0/3927892039 shutdown_connections 2024-05-28T06:12:28.852 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:28.851+0000 7fb619000640 1 -- 172.21.15.96:0/3927892039 wait complete. 2024-05-28T06:12:28.884 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:12:29.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.054+0000 7f1c94600640 1 Processor -- start 2024-05-28T06:12:29.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.054+0000 7f1c94600640 1 -- start start 2024-05-28T06:12:29.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.055+0000 7f1c94600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1c8c0d04a0 0x7f1c8c0d0900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:29.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.055+0000 7f1c94600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f1c8c0ad130 0x7f1c8c0ad5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:29.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.055+0000 7f1c94600640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c8c06ed10 con 0x7f1c8c06f860 2024-05-28T06:12:29.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.055+0000 7f1c94600640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f1c8c0d0e40 con 0x7f1c8c0ad130 2024-05-28T06:12:29.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.055+0000 7f1c94600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f1c8c15cb00 con 0x7f1c8c0d04a0 2024-05-28T06:12:29.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.055+0000 7f1c92600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f1c8c0ad130 0x7f1c8c0ad5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:29.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.055+0000 7f1c91c00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f1c8c06f860 0x7f1c8c0b3150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:44560/0 (socket says 172.21.15.96:44560) 2024-05-28T06:12:29.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.055+0000 7f1c91c00640 1 -- 172.21.15.96:0/105264214 learned_addr learned my addr 172.21.15.96:0/105264214 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:12:29.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.056+0000 7f1c91200640 1 --2- 172.21.15.96:0/105264214 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1c8c0d04a0 0x7f1c8c0d0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:29.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.056+0000 7f1c8be00640 1 -- 172.21.15.96:0/105264214 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1133104422 0 0) 0x7f1c80001490 con 0x7f1c8c06f860 2024-05-28T06:12:29.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.056+0000 7f1c8be00640 1 -- 172.21.15.96:0/105264214 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1c74003520 con 0x7f1c8c06f860 2024-05-28T06:12:29.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.056+0000 7f1c92600640 1 -- 172.21.15.96:0/105264214 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1c8c0d04a0 msgr2=0x7f1c8c0d0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:29.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.056+0000 7f1c92600640 1 --2- 172.21.15.96:0/105264214 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1c8c0d04a0 0x7f1c8c0d0900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:29.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.056+0000 7f1c92600640 1 -- 172.21.15.96:0/105264214 >> v1:172.21.15.96:6789/0 conn(0x7f1c8c06f860 legacy=0x7f1c8c0b3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:29.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.056+0000 7f1c92600640 1 -- 172.21.15.96:0/105264214 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c8c0b3cc0 con 0x7f1c8c0ad130 2024-05-28T06:12:29.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.057+0000 7f1c92600640 1 --2- 172.21.15.96:0/105264214 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1c8c0ad130 0x7f1c8c0ad5a0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f1c840097a0 tx=0x7f1c8402a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9446779a6b9505cc server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:29.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.057+0000 7f1c8be00640 1 -- 172.21.15.96:0/105264214 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c84037070 con 0x7f1c8c0ad130 2024-05-28T06:12:29.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.057+0000 7f1c8be00640 1 -- 172.21.15.96:0/105264214 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c84032440 con 0x7f1c8c0ad130 2024-05-28T06:12:29.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.057+0000 7f1c8be00640 1 -- 172.21.15.96:0/105264214 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c84032810 con 0x7f1c8c0ad130 2024-05-28T06:12:29.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.058+0000 7f1c94600640 1 -- 172.21.15.96:0/105264214 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1c8c0ad130 msgr2=0x7f1c8c0ad5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:29.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.058+0000 7f1c94600640 1 --2- 172.21.15.96:0/105264214 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1c8c0ad130 0x7f1c8c0ad5a0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f1c840097a0 tx=0x7f1c8402a180 comp rx=0 tx=0).stop 2024-05-28T06:12:29.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.058+0000 7f1c94600640 1 -- 172.21.15.96:0/105264214 shutdown_connections 2024-05-28T06:12:29.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.058+0000 7f1c94600640 1 --2- 172.21.15.96:0/105264214 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1c8c0ad130 0x7f1c8c0ad5a0 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:29.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.058+0000 7f1c94600640 1 --2- 172.21.15.96:0/105264214 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1c8c0d04a0 0x7f1c8c0d0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:29.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.058+0000 7f1c94600640 1 -- 172.21.15.96:0/105264214 >> 172.21.15.96:0/105264214 conn(0x7f1c8c06bae0 msgr2=0x7f1c8c06bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:29.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.058+0000 7f1c94600640 1 -- 172.21.15.96:0/105264214 shutdown_connections 2024-05-28T06:12:29.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.058+0000 7f1c94600640 1 -- 172.21.15.96:0/105264214 wait complete. 2024-05-28T06:12:29.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.060+0000 7f1c94600640 1 Processor -- start 2024-05-28T06:12:29.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.060+0000 7f1c94600640 1 -- start start 2024-05-28T06:12:29.062 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.061+0000 7f1c94600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1c8c06f860 0x7f1c8c15e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:29.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.061+0000 7f1c94600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c8c0ad130 0x7f1c8c0c8e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:29.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.061+0000 7f1c94600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1c8c0d04a0 0x7f1c8c0c93a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:29.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.061+0000 7f1c94600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f1c8c0b4560 con 0x7f1c8c06f860 2024-05-28T06:12:29.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.061+0000 7f1c94600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f1c8c0b4280 con 0x7f1c8c0ad130 2024-05-28T06:12:29.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.061+0000 7f1c94600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f1c8c0b4840 con 0x7f1c8c0d04a0 2024-05-28T06:12:29.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.061+0000 7f1c91c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1c8c06f860 0x7f1c8c15e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:29.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.061+0000 7f1c91c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1c8c06f860 0x7f1c8c15e2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44714/0 (socket says 172.21.15.96:44714) 2024-05-28T06:12:29.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.061+0000 7f1c91c00640 1 -- 172.21.15.96:0/1624791319 learned_addr learned my addr 172.21.15.96:0/1624791319 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:29.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.061+0000 7f1c92600640 1 --2- 172.21.15.96:0/1624791319 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1c8c0d04a0 0x7f1c8c0c93a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:29.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.061+0000 7f1c91c00640 1 -- 172.21.15.96:0/1624791319 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1c8c0d04a0 msgr2=0x7f1c8c0c93a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:29.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.062+0000 7f1c91c00640 1 --2- 172.21.15.96:0/1624791319 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1c8c0d04a0 0x7f1c8c0c93a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:29.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.062+0000 7f1c91200640 1 --2- 172.21.15.96:0/1624791319 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c8c0ad130 0x7f1c8c0c8e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:29.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.062+0000 7f1c91c00640 1 -- 172.21.15.96:0/1624791319 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c8c0ad130 msgr2=0x7f1c8c0c8e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:29.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.062+0000 7f1c91c00640 1 --2- 172.21.15.96:0/1624791319 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c8c0ad130 0x7f1c8c0c8e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:29.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.062+0000 7f1c91c00640 1 -- 172.21.15.96:0/1624791319 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c8c1f41c0 con 0x7f1c8c06f860 2024-05-28T06:12:29.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.062+0000 7f1c91c00640 1 --2- 172.21.15.96:0/1624791319 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1c8c06f860 0x7f1c8c15e2b0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f1c8000cb40 tx=0x7f1c8000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:29.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.062+0000 7f1c89600640 1 -- 172.21.15.96:0/1624791319 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c80017070 con 0x7f1c8c06f860 2024-05-28T06:12:29.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.062+0000 7f1c89600640 1 -- 172.21.15.96:0/1624791319 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c8000ac40 con 0x7f1c8c06f860 2024-05-28T06:12:29.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.062+0000 7f1c89600640 1 -- 172.21.15.96:0/1624791319 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c80012400 con 0x7f1c8c06f860 2024-05-28T06:12:29.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.062+0000 7f1c94600640 1 -- 172.21.15.96:0/1624791319 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c8c15d210 con 0x7f1c8c06f860 2024-05-28T06:12:29.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.062+0000 7f1c94600640 1 -- 172.21.15.96:0/1624791319 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c8c15d7d0 con 0x7f1c8c06f860 2024-05-28T06:12:29.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.063+0000 7f1c94600640 1 -- 172.21.15.96:0/1624791319 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c8c0689e0 con 0x7f1c8c06f860 2024-05-28T06:12:29.065 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.064+0000 7f1c89600640 1 -- 172.21.15.96:0/1624791319 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 16) ==== 91754+0+0 (secure 0 0 0) 0x7f1c8001a070 con 0x7f1c8c06f860 2024-05-28T06:12:29.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.065+0000 7f1c89600640 1 --2- 172.21.15.96:0/1624791319 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1c6c06d0d0 0x7f1c6c06f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:29.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.065+0000 7f1c89600640 1 -- 172.21.15.96:0/1624791319 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 5737+0+0 (secure 0 0 0) 0x7f1c8008cd20 con 0x7f1c8c06f860 2024-05-28T06:12:29.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.066+0000 7f1c91200640 1 --2- 172.21.15.96:0/1624791319 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1c6c06d0d0 0x7f1c6c06f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:29.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.067+0000 7f1c91200640 1 --2- 172.21.15.96:0/1624791319 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1c6c06d0d0 0x7f1c6c06f590 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f1c8c0c8690 tx=0x7f1c78008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:29.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.070+0000 7f1c89600640 1 -- 172.21.15.96:0/1624791319 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1c8c06ed10 con 0x7f1c8c06f860 2024-05-28T06:12:29.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.270+0000 7f1c94600640 1 -- 172.21.15.96:0/1624791319 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1c8c06cc30 con 0x7f1c8c06f860 2024-05-28T06:12:29.273 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.273+0000 7f1c89600640 1 -- 172.21.15.96:0/1624791319 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v59) ==== 74+0+16733 (secure 0 0 0) 0x7f1c8005a080 con 0x7f1c8c06f860 2024-05-28T06:12:29.274 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:12:29.274 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":59,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:12:28.268502+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-05-28T06:12:26.024060+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-05-28T06:12:27.254401+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:12:29.276 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.276+0000 7f1c94600640 1 -- 172.21.15.96:0/1624791319 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1c6c06d0d0 msgr2=0x7f1c6c06f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:29.276 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.276+0000 7f1c94600640 1 --2- 172.21.15.96:0/1624791319 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1c6c06d0d0 0x7f1c6c06f590 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f1c8c0c8690 tx=0x7f1c78008040 comp rx=0 tx=0).stop 2024-05-28T06:12:29.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.276+0000 7f1c94600640 1 -- 172.21.15.96:0/1624791319 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1c8c06f860 msgr2=0x7f1c8c15e2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:29.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.276+0000 7f1c94600640 1 --2- 172.21.15.96:0/1624791319 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1c8c06f860 0x7f1c8c15e2b0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f1c8000cb40 tx=0x7f1c8000a060 comp rx=0 tx=0).stop 2024-05-28T06:12:29.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.276+0000 7f1c94600640 1 -- 172.21.15.96:0/1624791319 shutdown_connections 2024-05-28T06:12:29.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.276+0000 7f1c94600640 1 --2- 172.21.15.96:0/1624791319 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1c6c06d0d0 0x7f1c6c06f590 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:29.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.276+0000 7f1c94600640 1 --2- 172.21.15.96:0/1624791319 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1c8c0d04a0 0x7f1c8c0c93a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:29.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.276+0000 7f1c94600640 1 --2- 172.21.15.96:0/1624791319 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c8c0ad130 0x7f1c8c0c8e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:29.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.276+0000 7f1c94600640 1 --2- 172.21.15.96:0/1624791319 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1c8c06f860 0x7f1c8c15e2b0 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:29.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.276+0000 7f1c94600640 1 -- 172.21.15.96:0/1624791319 >> 172.21.15.96:0/1624791319 conn(0x7f1c8c06bae0 msgr2=0x7f1c8c0cbbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:29.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.276+0000 7f1c94600640 1 -- 172.21.15.96:0/1624791319 shutdown_connections 2024-05-28T06:12:29.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.276+0000 7f1c94600640 1 -- 172.21.15.96:0/1624791319 wait complete. 2024-05-28T06:12:29.296 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:12:29.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.471+0000 7f7d87e00640 1 Processor -- start 2024-05-28T06:12:29.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.471+0000 7f7d87e00640 1 -- start start 2024-05-28T06:12:29.473 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.472+0000 7f7d87e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7d88071bc0 0x7f7d88072070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:29.473 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.473+0000 7f7d87e00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d8806e6e0 con 0x7f7d880b32c0 2024-05-28T06:12:29.473 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.473+0000 7f7d87e00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d8806e9c0 con 0x7f7d88158b80 2024-05-28T06:12:29.473 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.473+0000 7f7d87e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f7d8806eca0 con 0x7f7d88071bc0 2024-05-28T06:12:29.473 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.473+0000 7f7d87400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f7d880b32c0 0x7f7d880b3710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:44572/0 (socket says 172.21.15.96:44572) 2024-05-28T06:12:29.474 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.473+0000 7f7d87400640 1 -- 172.21.15.96:0/2174048699 learned_addr learned my addr 172.21.15.96:0/2174048699 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:12:29.475 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.473+0000 7f7d86000640 1 --2- 172.21.15.96:0/2174048699 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7d88071bc0 0x7f7d88072070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:29.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.473+0000 7f7d85600640 1 -- 172.21.15.96:0/2174048699 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3955061934 0 0) 0x7f7d8806e6e0 con 0x7f7d880b32c0 2024-05-28T06:12:29.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.473+0000 7f7d85600640 1 -- 172.21.15.96:0/2174048699 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7d6c003520 con 0x7f7d880b32c0 2024-05-28T06:12:29.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.474+0000 7f7d85600640 1 -- 172.21.15.96:0/2174048699 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1566384532 0 0) 0x7f7d6c003520 con 0x7f7d880b32c0 2024-05-28T06:12:29.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.474+0000 7f7d85600640 1 -- 172.21.15.96:0/2174048699 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7d88071bc0 msgr2=0x7f7d88072070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:29.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.474+0000 7f7d85600640 1 --2- 172.21.15.96:0/2174048699 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7d88071bc0 0x7f7d88072070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:29.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.474+0000 7f7d85600640 1 -- 172.21.15.96:0/2174048699 >> v1:172.21.15.123:6789/0 conn(0x7f7d88158b80 legacy=0x7f7d8806dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:29.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.474+0000 7f7d85600640 1 -- 172.21.15.96:0/2174048699 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d8806ef80 con 0x7f7d880b32c0 2024-05-28T06:12:29.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.474+0000 7f7d85600640 1 -- 172.21.15.96:0/2174048699 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f7d800019f0 con 0x7f7d880b32c0 2024-05-28T06:12:29.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.474+0000 7f7d85600640 1 -- 172.21.15.96:0/2174048699 >> v1:172.21.15.96:6789/0 conn(0x7f7d880b32c0 legacy=0x7f7d880b3710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:29.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.474+0000 7f7d85600640 1 --2- 172.21.15.96:0/2174048699 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7d88071bc0 0x7f7d6c0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:29.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.474+0000 7f7d85600640 1 --2- 172.21.15.96:0/2174048699 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7d6c003f60 0x7f7d6c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:29.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.474+0000 7f7d85600640 1 --2- 172.21.15.96:0/2174048699 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d6c004d40 0x7f7d6c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:29.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.474+0000 7f7d85600640 1 -- 172.21.15.96:0/2174048699 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f7d8806eca0 con 0x7f7d6c003f60 2024-05-28T06:12:29.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.474+0000 7f7d85600640 1 -- 172.21.15.96:0/2174048699 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f7d6c00baa0 con 0x7f7d6c004d40 2024-05-28T06:12:29.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.474+0000 7f7d85600640 1 -- 172.21.15.96:0/2174048699 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f7d6c00bca0 con 0x7f7d88071bc0 2024-05-28T06:12:29.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.474+0000 7f7d86000640 1 --2- 172.21.15.96:0/2174048699 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7d6c003f60 0x7f7d6c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:29.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.475+0000 7f7d86000640 1 -- 172.21.15.96:0/2174048699 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7d88071bc0 msgr2=0x7f7d6c0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:29.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.475+0000 7f7d86000640 1 --2- 172.21.15.96:0/2174048699 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7d88071bc0 0x7f7d6c0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:29.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.475+0000 7f7d86000640 1 -- 172.21.15.96:0/2174048699 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d6c004d40 msgr2=0x7f7d6c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:29.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.475+0000 7f7d86000640 1 --2- 172.21.15.96:0/2174048699 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d6c004d40 0x7f7d6c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:29.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.475+0000 7f7d86000640 1 -- 172.21.15.96:0/2174048699 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d6c003520 con 0x7f7d6c003f60 2024-05-28T06:12:29.478 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.476+0000 7f7d86000640 1 --2- 172.21.15.96:0/2174048699 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7d6c003f60 0x7f7d6c004800 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f7d74009ac0 tx=0x7f7d6c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4b3a76e6a9f5c1c server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:29.478 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.476+0000 7f7d85600640 1 -- 172.21.15.96:0/2174048699 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d74036070 con 0x7f7d6c003f60 2024-05-28T06:12:29.479 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.476+0000 7f7d85600640 1 -- 172.21.15.96:0/2174048699 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d74031400 con 0x7f7d6c003f60 2024-05-28T06:12:29.479 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.476+0000 7f7d85600640 1 -- 172.21.15.96:0/2174048699 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d74009690 con 0x7f7d6c003f60 2024-05-28T06:12:29.479 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.476+0000 7f7d87e00640 1 -- 172.21.15.96:0/2174048699 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7d6c003f60 msgr2=0x7f7d6c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:29.479 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.476+0000 7f7d87e00640 1 --2- 172.21.15.96:0/2174048699 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7d6c003f60 0x7f7d6c004800 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f7d74009ac0 tx=0x7f7d6c005830 comp rx=0 tx=0).stop 2024-05-28T06:12:29.479 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.476+0000 7f7d87400640 1 -- 172.21.15.96:0/2174048699 reap_dead start 2024-05-28T06:12:29.479 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.476+0000 7f7d87e00640 1 -- 172.21.15.96:0/2174048699 shutdown_connections 2024-05-28T06:12:29.479 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.476+0000 7f7d87e00640 1 -- 172.21.15.96:0/2174048699 >> 172.21.15.96:0/2174048699 conn(0x7f7d880b1fc0 msgr2=0x7f7d880b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:29.479 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.476+0000 7f7d87e00640 1 -- 172.21.15.96:0/2174048699 shutdown_connections 2024-05-28T06:12:29.479 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.477+0000 7f7d87e00640 1 -- 172.21.15.96:0/2174048699 wait complete. 2024-05-28T06:12:29.479 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.478+0000 7f7d87e00640 1 Processor -- start 2024-05-28T06:12:29.479 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.478+0000 7f7d87e00640 1 -- start start 2024-05-28T06:12:29.481 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.479+0000 7f7d87e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7d6c003f60 0x7f7d88161770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:29.481 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.479+0000 7f7d87e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d881465a0 0x7f7d8809c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:29.481 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.479+0000 7f7d87e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7d8809ca40 0x7f7d881f7660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:29.481 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.479+0000 7f7d87e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f7d8806f780 con 0x7f7d8809ca40 2024-05-28T06:12:29.481 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.479+0000 7f7d87e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f7d880b2d30 con 0x7f7d881465a0 2024-05-28T06:12:29.481 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.479+0000 7f7d87e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f7d8815ec70 con 0x7f7d6c003f60 2024-05-28T06:12:29.481 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.479+0000 7f7d87400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7d8809ca40 0x7f7d881f7660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:29.481 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.479+0000 7f7d87400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7d8809ca40 0x7f7d881f7660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44738/0 (socket says 172.21.15.96:44738) 2024-05-28T06:12:29.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.479+0000 7f7d87400640 1 -- 172.21.15.96:0/304847942 learned_addr learned my addr 172.21.15.96:0/304847942 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:29.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.480+0000 7f7d86000640 1 --2- 172.21.15.96:0/304847942 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d881465a0 0x7f7d8809c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:29.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.480+0000 7f7d86a00640 1 --2- 172.21.15.96:0/304847942 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7d6c003f60 0x7f7d88161770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:29.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.480+0000 7f7d87400640 1 -- 172.21.15.96:0/304847942 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7d6c003f60 msgr2=0x7f7d88161770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:29.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.480+0000 7f7d87400640 1 --2- 172.21.15.96:0/304847942 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7d6c003f60 0x7f7d88161770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:29.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.480+0000 7f7d87400640 1 -- 172.21.15.96:0/304847942 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d881465a0 msgr2=0x7f7d8809c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:29.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.480+0000 7f7d87400640 1 --2- 172.21.15.96:0/304847942 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d881465a0 0x7f7d8809c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:29.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.480+0000 7f7d87400640 1 -- 172.21.15.96:0/304847942 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d8809b240 con 0x7f7d8809ca40 2024-05-28T06:12:29.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.480+0000 7f7d87400640 1 --2- 172.21.15.96:0/304847942 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7d8809ca40 0x7f7d881f7660 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f7d80000f80 tx=0x7f7d8000ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:29.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.480+0000 7f7d7ea00640 1 -- 172.21.15.96:0/304847942 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d80007110 con 0x7f7d8809ca40 2024-05-28T06:12:29.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.481+0000 7f7d7ea00640 1 -- 172.21.15.96:0/304847942 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d800072d0 con 0x7f7d8809ca40 2024-05-28T06:12:29.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.481+0000 7f7d7ea00640 1 -- 172.21.15.96:0/304847942 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d800076c0 con 0x7f7d8809ca40 2024-05-28T06:12:29.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.481+0000 7f7d87e00640 1 -- 172.21.15.96:0/304847942 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d8809b510 con 0x7f7d8809ca40 2024-05-28T06:12:29.483 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.481+0000 7f7d87e00640 1 -- 172.21.15.96:0/304847942 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d8809bad0 con 0x7f7d8809ca40 2024-05-28T06:12:29.483 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.482+0000 7f7d87e00640 1 -- 172.21.15.96:0/304847942 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d880689e0 con 0x7f7d8809ca40 2024-05-28T06:12:29.483 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.483+0000 7f7d7ea00640 1 -- 172.21.15.96:0/304847942 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 16) ==== 91754+0+0 (secure 0 0 0) 0x7f7d80011070 con 0x7f7d8809ca40 2024-05-28T06:12:29.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.483+0000 7f7d7ea00640 1 --2- 172.21.15.96:0/304847942 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7d6806d100 0x7f7d6806f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:29.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.484+0000 7f7d7ea00640 1 -- 172.21.15.96:0/304847942 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 5737+0+0 (secure 0 0 0) 0x7f7d8008c400 con 0x7f7d8809ca40 2024-05-28T06:12:29.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.484+0000 7f7d86a00640 1 --2- 172.21.15.96:0/304847942 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7d6806d100 0x7f7d6806f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:29.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.485+0000 7f7d86a00640 1 --2- 172.21.15.96:0/304847942 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7d6806d100 0x7f7d6806f5c0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f7d78001690 tx=0x7f7d78009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:29.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.488+0000 7f7d7ea00640 1 -- 172.21.15.96:0/304847942 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7d800597f0 con 0x7f7d8809ca40 2024-05-28T06:12:29.690 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.690+0000 7f7d87e00640 1 -- 172.21.15.96:0/304847942 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7d8815edf0 con 0x7f7d8809ca40 2024-05-28T06:12:29.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.692+0000 7f7d7ea00640 1 -- 172.21.15.96:0/304847942 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v59) ==== 74+0+16733 (secure 0 0 0) 0x7f7d800590f0 con 0x7f7d8809ca40 2024-05-28T06:12:29.693 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:12:29.693 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":59,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:12:28.268502+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-05-28T06:12:26.024060+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-05-28T06:12:27.254401+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:12:29.695 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.694+0000 7f7d87e00640 1 -- 172.21.15.96:0/304847942 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7d6806d100 msgr2=0x7f7d6806f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:29.695 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.694+0000 7f7d87e00640 1 --2- 172.21.15.96:0/304847942 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7d6806d100 0x7f7d6806f5c0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f7d78001690 tx=0x7f7d78009040 comp rx=0 tx=0).stop 2024-05-28T06:12:29.695 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.694+0000 7f7d87e00640 1 -- 172.21.15.96:0/304847942 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7d8809ca40 msgr2=0x7f7d881f7660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:29.695 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.694+0000 7f7d87e00640 1 --2- 172.21.15.96:0/304847942 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7d8809ca40 0x7f7d881f7660 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f7d80000f80 tx=0x7f7d8000ccd0 comp rx=0 tx=0).stop 2024-05-28T06:12:29.695 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.694+0000 7f7d87e00640 1 -- 172.21.15.96:0/304847942 shutdown_connections 2024-05-28T06:12:29.695 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.694+0000 7f7d87e00640 1 --2- 172.21.15.96:0/304847942 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7d6806d100 0x7f7d6806f5c0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:29.695 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.694+0000 7f7d87e00640 1 --2- 172.21.15.96:0/304847942 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7d8809ca40 0x7f7d881f7660 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:29.695 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.694+0000 7f7d87e00640 1 --2- 172.21.15.96:0/304847942 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d881465a0 0x7f7d8809c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:29.696 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.694+0000 7f7d87e00640 1 --2- 172.21.15.96:0/304847942 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7d6c003f60 0x7f7d88161770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:29.696 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.694+0000 7f7d87e00640 1 -- 172.21.15.96:0/304847942 >> 172.21.15.96:0/304847942 conn(0x7f7d880b1fc0 msgr2=0x7f7d88162a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:29.696 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.695+0000 7f7d87e00640 1 -- 172.21.15.96:0/304847942 shutdown_connections 2024-05-28T06:12:29.696 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.695+0000 7f7d87e00640 1 -- 172.21.15.96:0/304847942 wait complete. 2024-05-28T06:12:29.714 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-05-28T06:12:29.919 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.919+0000 7fe40fe00640 1 Processor -- start 2024-05-28T06:12:29.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.919+0000 7fe40fe00640 1 -- start start 2024-05-28T06:12:29.943 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.920+0000 7fe40fe00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fe410153510 0x7fe4100b4d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:29.943 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.920+0000 7fe40fe00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe41015d270 0x7fe4100ccff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:29.943 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.920+0000 7fe40fe00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe41015d900 con 0x7fe4100ccb90 2024-05-28T06:12:29.943 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.920+0000 7fe40fe00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fe41015d6e0 con 0x7fe410153510 2024-05-28T06:12:29.943 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.920+0000 7fe40fe00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fe4101538f0 con 0x7fe41015d270 2024-05-28T06:12:29.943 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.921+0000 7fe40ea00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fe410153510 0x7fe4100b4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:29.943 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.921+0000 7fe40ea00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fe410153510 0x7fe4100b4d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44754/0 (socket says 172.21.15.96:44754) 2024-05-28T06:12:29.943 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.921+0000 7fe40ea00640 1 -- 172.21.15.96:0/3595542541 learned_addr learned my addr 172.21.15.96:0/3595542541 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:29.943 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.921+0000 7fe40f400640 1 --2- 172.21.15.96:0/3595542541 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe41015d270 0x7fe4100ccff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:29.943 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.921+0000 7fe40ea00640 1 -- 172.21.15.96:0/3595542541 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe41015d270 msgr2=0x7fe4100ccff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:29.943 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.921+0000 7fe40ea00640 1 --2- 172.21.15.96:0/3595542541 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe41015d270 0x7fe4100ccff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:29.943 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.921+0000 7fe40ea00640 1 -- 172.21.15.96:0/3595542541 >> v1:172.21.15.123:6789/0 conn(0x7fe4100ccb90 legacy=0x7fe41015cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:29.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.921+0000 7fe40ea00640 1 -- 172.21.15.96:0/3595542541 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4100b3f90 con 0x7fe410153510 2024-05-28T06:12:29.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.922+0000 7fe40ea00640 1 --2- 172.21.15.96:0/3595542541 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe410153510 0x7fe4100b4d50 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fe400009870 tx=0x7fe40002a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70a8ebd10181d938 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:29.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.922+0000 7fe40e000640 1 -- 172.21.15.96:0/3595542541 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe400037070 con 0x7fe410153510 2024-05-28T06:12:29.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.923+0000 7fe40e000640 1 -- 172.21.15.96:0/3595542541 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe400032440 con 0x7fe410153510 2024-05-28T06:12:29.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.923+0000 7fe40e000640 1 -- 172.21.15.96:0/3595542541 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe400032810 con 0x7fe410153510 2024-05-28T06:12:29.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.923+0000 7fe40fe00640 1 -- 172.21.15.96:0/3595542541 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe410153510 msgr2=0x7fe4100b4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:29.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.923+0000 7fe40fe00640 1 --2- 172.21.15.96:0/3595542541 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe410153510 0x7fe4100b4d50 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fe400009870 tx=0x7fe40002a180 comp rx=0 tx=0).stop 2024-05-28T06:12:29.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.923+0000 7fe40fe00640 1 -- 172.21.15.96:0/3595542541 shutdown_connections 2024-05-28T06:12:29.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.923+0000 7fe40fe00640 1 --2- 172.21.15.96:0/3595542541 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe41015d270 0x7fe4100ccff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:29.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.923+0000 7fe40fe00640 1 --2- 172.21.15.96:0/3595542541 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe410153510 0x7fe4100b4d50 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:29.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.923+0000 7fe40fe00640 1 -- 172.21.15.96:0/3595542541 >> 172.21.15.96:0/3595542541 conn(0x7fe41006b900 msgr2=0x7fe41006bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:29.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.923+0000 7fe40fe00640 1 -- 172.21.15.96:0/3595542541 shutdown_connections 2024-05-28T06:12:29.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.924+0000 7fe40fe00640 1 -- 172.21.15.96:0/3595542541 wait complete. 2024-05-28T06:12:29.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.925+0000 7fe40fe00640 1 Processor -- start 2024-05-28T06:12:29.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.926+0000 7fe40fe00640 1 -- start start 2024-05-28T06:12:29.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.927+0000 7fe40fe00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe4100ccb90 0x7fe410151be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:29.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.927+0000 7fe40fe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe410153510 0x7fe410072890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:29.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.927+0000 7fe40fe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe41015d270 0x7fe4100cb3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:29.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.927+0000 7fe40fe00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe4100b48c0 con 0x7fe410153510 2024-05-28T06:12:29.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.927+0000 7fe40fe00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe4100b4550 con 0x7fe41015d270 2024-05-28T06:12:29.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.927+0000 7fe40fe00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fe41006e940 con 0x7fe4100ccb90 2024-05-28T06:12:29.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.927+0000 7fe406000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe410153510 0x7fe410072890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:29.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.927+0000 7fe406000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe410153510 0x7fe410072890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44756/0 (socket says 172.21.15.96:44756) 2024-05-28T06:12:29.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.927+0000 7fe406000640 1 -- 172.21.15.96:0/3440698840 learned_addr learned my addr 172.21.15.96:0/3440698840 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:29.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.927+0000 7fe40ea00640 1 --2- 172.21.15.96:0/3440698840 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe4100ccb90 0x7fe410151be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:29.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.927+0000 7fe406000640 1 -- 172.21.15.96:0/3440698840 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe4100ccb90 msgr2=0x7fe410151be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:29.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.927+0000 7fe406000640 1 --2- 172.21.15.96:0/3440698840 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe4100ccb90 0x7fe410151be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:29.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.927+0000 7fe406000640 1 -- 172.21.15.96:0/3440698840 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe41015d270 msgr2=0x7fe4100cb3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:29.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.927+0000 7fe406000640 1 --2- 172.21.15.96:0/3440698840 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe41015d270 0x7fe4100cb3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:29.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.927+0000 7fe406000640 1 -- 172.21.15.96:0/3440698840 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe410155510 con 0x7fe410153510 2024-05-28T06:12:29.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.928+0000 7fe406000640 1 --2- 172.21.15.96:0/3440698840 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe410153510 0x7fe410072890 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fe3f800c430 tx=0x7fe3f800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:29.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.928+0000 7fe407400640 1 -- 172.21.15.96:0/3440698840 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3f8011790 con 0x7fe410153510 2024-05-28T06:12:29.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.928+0000 7fe407400640 1 -- 172.21.15.96:0/3440698840 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3f8017070 con 0x7fe410153510 2024-05-28T06:12:29.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.928+0000 7fe407400640 1 -- 172.21.15.96:0/3440698840 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3f8011bb0 con 0x7fe410153510 2024-05-28T06:12:29.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.928+0000 7fe40fe00640 1 -- 172.21.15.96:0/3440698840 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4101557b0 con 0x7fe410153510 2024-05-28T06:12:29.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.928+0000 7fe40fe00640 1 -- 172.21.15.96:0/3440698840 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe410155d70 con 0x7fe410153510 2024-05-28T06:12:29.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.929+0000 7fe40fe00640 1 -- 172.21.15.96:0/3440698840 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4100689e0 con 0x7fe410153510 2024-05-28T06:12:29.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.930+0000 7fe407400640 1 -- 172.21.15.96:0/3440698840 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 16) ==== 91754+0+0 (secure 0 0 0) 0x7fe3f801c460 con 0x7fe410153510 2024-05-28T06:12:29.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.931+0000 7fe407400640 1 --2- 172.21.15.96:0/3440698840 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe3ec06d0d0 0x7fe3ec06f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:29.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.931+0000 7fe407400640 1 -- 172.21.15.96:0/3440698840 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5737+0+0 (secure 0 0 0) 0x7fe3f808a450 con 0x7fe410153510 2024-05-28T06:12:29.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.932+0000 7fe40ea00640 1 --2- 172.21.15.96:0/3440698840 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe3ec06d0d0 0x7fe3ec06f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:29.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.933+0000 7fe40ea00640 1 --2- 172.21.15.96:0/3440698840 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe3ec06d0d0 0x7fe3ec06f590 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fe400034000 tx=0x7fe40002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:29.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:29.935+0000 7fe407400640 1 -- 172.21.15.96:0/3440698840 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe3f805a170 con 0x7fe410153510 2024-05-28T06:12:30.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:30.177+0000 7fe40fe00640 1 -- 172.21.15.96:0/3440698840 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fe4100b3c40 con 0x7fe410153510 2024-05-28T06:12:30.850 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:30.848+0000 7fe407400640 1 -- 172.21.15.96:0/3440698840 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v33) ==== 118+0+0 (secure 0 0 0) 0x7fe3f8059a70 con 0x7fe410153510 2024-05-28T06:12:30.854 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:30.852+0000 7fe40fe00640 1 -- 172.21.15.96:0/3440698840 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe3ec06d0d0 msgr2=0x7fe3ec06f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:30.854 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:30.852+0000 7fe40fe00640 1 --2- 172.21.15.96:0/3440698840 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe3ec06d0d0 0x7fe3ec06f590 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fe400034000 tx=0x7fe40002e000 comp rx=0 tx=0).stop 2024-05-28T06:12:30.854 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:30.852+0000 7fe40fe00640 1 -- 172.21.15.96:0/3440698840 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe410153510 msgr2=0x7fe410072890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:30.854 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:30.852+0000 7fe40fe00640 1 --2- 172.21.15.96:0/3440698840 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe410153510 0x7fe410072890 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fe3f800c430 tx=0x7fe3f800c8f0 comp rx=0 tx=0).stop 2024-05-28T06:12:30.854 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:30.852+0000 7fe40fe00640 1 -- 172.21.15.96:0/3440698840 shutdown_connections 2024-05-28T06:12:30.854 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:30.852+0000 7fe40fe00640 1 --2- 172.21.15.96:0/3440698840 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe3ec06d0d0 0x7fe3ec06f590 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:30.854 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:30.852+0000 7fe40fe00640 1 --2- 172.21.15.96:0/3440698840 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe41015d270 0x7fe4100cb3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:30.855 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:30.852+0000 7fe40fe00640 1 --2- 172.21.15.96:0/3440698840 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe410153510 0x7fe410072890 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:30.855 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:30.852+0000 7fe40fe00640 1 --2- 172.21.15.96:0/3440698840 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe4100ccb90 0x7fe410151be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:30.855 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:30.852+0000 7fe40fe00640 1 -- 172.21.15.96:0/3440698840 >> 172.21.15.96:0/3440698840 conn(0x7fe41006b900 msgr2=0x7fe41015fee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:30.855 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:30.853+0000 7fe40fe00640 1 -- 172.21.15.96:0/3440698840 shutdown_connections 2024-05-28T06:12:30.855 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:30.853+0000 7fe40fe00640 1 -- 172.21.15.96:0/3440698840 wait complete. 2024-05-28T06:12:30.877 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-05-28T06:12:30.877 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-05-28T06:12:31.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:31.083+0000 7f6cb2400640 1 Processor -- start 2024-05-28T06:12:31.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:31.083+0000 7f6cb2400640 1 -- start start 2024-05-28T06:12:31.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:31.085+0000 7f6cb2400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f6cac0d0640 0x7f6cac0d0a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:31.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:31.085+0000 7f6cb2400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6cac0725e0 0x7f6cac0ae650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:31.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:31.085+0000 7f6cb2400640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6cac0ade90 con 0x7f6cac15b6f0 2024-05-28T06:12:31.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:31.085+0000 7f6cb2400640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f6cac071c40 con 0x7f6cac0d0640 2024-05-28T06:12:31.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:31.085+0000 7f6cb2400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f6cac0d0f60 con 0x7f6cac0725e0 2024-05-28T06:12:31.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:31.085+0000 7f6cab400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f6cac0d0640 0x7f6cac0d0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:31.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:31.085+0000 7f6cab400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f6cac0d0640 0x7f6cac0d0a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44764/0 (socket says 172.21.15.96:44764) 2024-05-28T06:12:31.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:31.085+0000 7f6cab400640 1 -- 172.21.15.96:0/742677485 learned_addr learned my addr 172.21.15.96:0/742677485 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:31.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:31.086+0000 7f6cab400640 1 -- 172.21.15.96:0/742677485 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6cac0725e0 msgr2=0x7f6cac0ae650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:31.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:31.086+0000 7f6cab400640 1 --2- 172.21.15.96:0/742677485 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6cac0725e0 0x7f6cac0ae650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:31.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:31.086+0000 7f6cab400640 1 -- 172.21.15.96:0/742677485 >> v1:172.21.15.123:6789/0 conn(0x7f6cac15b6f0 legacy=0x7f6cac06e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:31.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:31.086+0000 7f6cab400640 1 -- 172.21.15.96:0/742677485 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cac071e70 con 0x7f6cac0d0640 2024-05-28T06:12:31.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:31.087+0000 7f6cab400640 1 --2- 172.21.15.96:0/742677485 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6cac0d0640 0x7f6cac0d0a20 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f6c9c009870 tx=0x7f6c9c02a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3199651bd5154445 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:31.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:31.087+0000 7f6caa000640 1 -- 172.21.15.96:0/742677485 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c9c037070 con 0x7f6cac0d0640 2024-05-28T06:12:31.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:31.087+0000 7f6caa000640 1 -- 172.21.15.96:0/742677485 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c9c032440 con 0x7f6cac0d0640 2024-05-28T06:12:31.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:31.087+0000 7f6caa000640 1 -- 172.21.15.96:0/742677485 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c9c032810 con 0x7f6cac0d0640 2024-05-28T06:12:31.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:31.087+0000 7f6cb2400640 1 -- 172.21.15.96:0/742677485 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6cac0d0640 msgr2=0x7f6cac0d0a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:31.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:31.087+0000 7f6cb2400640 1 --2- 172.21.15.96:0/742677485 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6cac0d0640 0x7f6cac0d0a20 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f6c9c009870 tx=0x7f6c9c02a210 comp rx=0 tx=0).stop 2024-05-28T06:12:31.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:31.088+0000 7f6cb2400640 1 -- 172.21.15.96:0/742677485 shutdown_connections 2024-05-28T06:12:31.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:31.088+0000 7f6cb2400640 1 --2- 172.21.15.96:0/742677485 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6cac0725e0 0x7f6cac0ae650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:31.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:31.088+0000 7f6cb2400640 1 --2- 172.21.15.96:0/742677485 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6cac0d0640 0x7f6cac0d0a20 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:31.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:31.088+0000 7f6cb2400640 1 -- 172.21.15.96:0/742677485 >> 172.21.15.96:0/742677485 conn(0x7f6cac06bd20 msgr2=0x7f6cac06c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:31.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:31.088+0000 7f6cb2400640 1 -- 172.21.15.96:0/742677485 shutdown_connections 2024-05-28T06:12:31.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:31.088+0000 7f6cb2400640 1 -- 172.21.15.96:0/742677485 wait complete. 2024-05-28T06:12:31.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:31.089+0000 7f6cb2400640 1 Processor -- start 2024-05-28T06:12:31.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:31.089+0000 7f6cb2400640 1 -- start start 2024-05-28T06:12:31.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:31.090+0000 7f6cb2400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6cac0725e0 0x7f6cac0ce930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:31.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:31.090+0000 7f6cb2400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6cac0d0640 0x7f6cac157600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:31.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:31.090+0000 7f6cb2400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6cac15b6f0 0x7f6cac157b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:31.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:31.090+0000 7f6cb2400640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f6cac0aeb90 con 0x7f6cac0d0640 2024-05-28T06:12:31.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:31.090+0000 7f6cb2400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f6cac0ae130 con 0x7f6cac15b6f0 2024-05-28T06:12:31.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:31.090+0000 7f6cb2400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f6cac0aed10 con 0x7f6cac0725e0 2024-05-28T06:12:31.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:31.091+0000 7f6caaa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6cac0d0640 0x7f6cac157600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:31.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:31.091+0000 7f6caaa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6cac0d0640 0x7f6cac157600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44768/0 (socket says 172.21.15.96:44768) 2024-05-28T06:12:31.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:31.091+0000 7f6caaa00640 1 -- 172.21.15.96:0/1664437960 learned_addr learned my addr 172.21.15.96:0/1664437960 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:31.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:31.091+0000 7f6cabe00640 1 --2- 172.21.15.96:0/1664437960 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6cac15b6f0 0x7f6cac157b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:31.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:31.091+0000 7f6cab400640 1 --2- 172.21.15.96:0/1664437960 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6cac0725e0 0x7f6cac0ce930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:31.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:31.091+0000 7f6caaa00640 1 -- 172.21.15.96:0/1664437960 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6cac0725e0 msgr2=0x7f6cac0ce930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:31.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:31.091+0000 7f6caaa00640 1 --2- 172.21.15.96:0/1664437960 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6cac0725e0 0x7f6cac0ce930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:31.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:31.091+0000 7f6caaa00640 1 -- 172.21.15.96:0/1664437960 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6cac15b6f0 msgr2=0x7f6cac157b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:31.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:31.091+0000 7f6caaa00640 1 --2- 172.21.15.96:0/1664437960 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6cac15b6f0 0x7f6cac157b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:31.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:31.091+0000 7f6caaa00640 1 -- 172.21.15.96:0/1664437960 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cac0cca50 con 0x7f6cac0d0640 2024-05-28T06:12:31.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:31.092+0000 7f6caaa00640 1 --2- 172.21.15.96:0/1664437960 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6cac0d0640 0x7f6cac157600 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f6c9800b3e0 tx=0x7f6c9800b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:31.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:31.092+0000 7f6ca3400640 1 -- 172.21.15.96:0/1664437960 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c98015020 con 0x7f6cac0d0640 2024-05-28T06:12:31.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:31.092+0000 7f6ca3400640 1 -- 172.21.15.96:0/1664437960 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c98011850 con 0x7f6cac0d0640 2024-05-28T06:12:31.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:31.092+0000 7f6cb2400640 1 -- 172.21.15.96:0/1664437960 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cac0ccd30 con 0x7f6cac0d0640 2024-05-28T06:12:31.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:31.092+0000 7f6ca3400640 1 -- 172.21.15.96:0/1664437960 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c98011c40 con 0x7f6cac0d0640 2024-05-28T06:12:31.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:31.092+0000 7f6cb2400640 1 -- 172.21.15.96:0/1664437960 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cac0cd2f0 con 0x7f6cac0d0640 2024-05-28T06:12:31.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:31.093+0000 7f6cb2400640 1 -- 172.21.15.96:0/1664437960 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6cac0ae950 con 0x7f6cac0d0640 2024-05-28T06:12:31.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:31.094+0000 7f6ca3400640 1 -- 172.21.15.96:0/1664437960 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 16) ==== 91754+0+0 (secure 0 0 0) 0x7f6c98018020 con 0x7f6cac0d0640 2024-05-28T06:12:31.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:31.095+0000 7f6ca3400640 1 --2- 172.21.15.96:0/1664437960 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6c8c06d0d0 0x7f6c8c06f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:31.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:31.095+0000 7f6ca3400640 1 -- 172.21.15.96:0/1664437960 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5737+0+0 (secure 0 0 0) 0x7f6c9808cda0 con 0x7f6cac0d0640 2024-05-28T06:12:31.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:31.096+0000 7f6cab400640 1 --2- 172.21.15.96:0/1664437960 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6c8c06d0d0 0x7f6c8c06f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:31.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:31.097+0000 7f6cab400640 1 --2- 172.21.15.96:0/1664437960 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6c8c06d0d0 0x7f6c8c06f590 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f6c9c034000 tx=0x7f6c9c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:31.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:31.101+0000 7f6ca3400640 1 -- 172.21.15.96:0/1664437960 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6cac0ae950 con 0x7f6cac0d0640 2024-05-28T06:12:31.151 DEBUG:teuthology.orchestra.run.smithi096:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:12:31.158 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:12:31.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:31.361+0000 7f6cb2400640 1 -- 172.21.15.96:0/1664437960 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f6cac0689e0 con 0x7f6cac0d0640 2024-05-28T06:12:32.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.338+0000 7f6ca3400640 1 -- 172.21.15.96:0/1664437960 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v34) ==== 122+0+0 (secure 0 0 0) 0x7f6c9801f020 con 0x7f6cac0d0640 2024-05-28T06:12:32.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.341+0000 7f6cb2400640 1 -- 172.21.15.96:0/1664437960 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6c8c06d0d0 msgr2=0x7f6c8c06f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:32.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.341+0000 7f6cb2400640 1 --2- 172.21.15.96:0/1664437960 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6c8c06d0d0 0x7f6c8c06f590 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f6c9c034000 tx=0x7f6c9c02e000 comp rx=0 tx=0).stop 2024-05-28T06:12:32.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.341+0000 7f6cb2400640 1 -- 172.21.15.96:0/1664437960 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6cac0d0640 msgr2=0x7f6cac157600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:32.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.341+0000 7f6cb2400640 1 --2- 172.21.15.96:0/1664437960 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6cac0d0640 0x7f6cac157600 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f6c9800b3e0 tx=0x7f6c9800b8a0 comp rx=0 tx=0).stop 2024-05-28T06:12:32.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.342+0000 7f6cb2400640 1 -- 172.21.15.96:0/1664437960 shutdown_connections 2024-05-28T06:12:32.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.342+0000 7f6cb2400640 1 --2- 172.21.15.96:0/1664437960 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6c8c06d0d0 0x7f6c8c06f590 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:32.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.342+0000 7f6cb2400640 1 --2- 172.21.15.96:0/1664437960 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6cac15b6f0 0x7f6cac157b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:32.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.342+0000 7f6cb2400640 1 --2- 172.21.15.96:0/1664437960 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6cac0d0640 0x7f6cac157600 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:32.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.342+0000 7f6cb2400640 1 --2- 172.21.15.96:0/1664437960 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6cac0725e0 0x7f6cac0ce930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:32.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.342+0000 7f6cb2400640 1 -- 172.21.15.96:0/1664437960 >> 172.21.15.96:0/1664437960 conn(0x7f6cac06bd20 msgr2=0x7f6cac0afe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:32.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.342+0000 7f6cb2400640 1 -- 172.21.15.96:0/1664437960 shutdown_connections 2024-05-28T06:12:32.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.342+0000 7f6cb2400640 1 -- 172.21.15.96:0/1664437960 wait complete. 2024-05-28T06:12:32.367 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-05-28T06:12:32.367 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-28T06:12:32.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.570+0000 7f5004e00640 1 Processor -- start 2024-05-28T06:12:32.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.570+0000 7f5004e00640 1 -- start start 2024-05-28T06:12:32.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.571+0000 7f5004e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f500006f860 0x7f50000b3150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:32.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.571+0000 7f5004e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f50000b3690 0x7f50000d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:32.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.571+0000 7f5004e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f50000d09e0 0x7f5000160210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:32.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.571+0000 7f5004e00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f50000d0e30 con 0x7f500006f860 2024-05-28T06:12:32.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.571+0000 7f5004e00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f500006ed10 con 0x7f50000b3690 2024-05-28T06:12:32.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.571+0000 7f5004e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f500015cb00 con 0x7f50000d09e0 2024-05-28T06:12:32.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.572+0000 7f4ffde00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f500006f860 0x7f50000b3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:32.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.572+0000 7f4ffde00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f500006f860 0x7f50000b3150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44776/0 (socket says 172.21.15.96:44776) 2024-05-28T06:12:32.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.572+0000 7f4ffde00640 1 -- 172.21.15.96:0/1260563382 learned_addr learned my addr 172.21.15.96:0/1260563382 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:32.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.572+0000 7f4ffe800640 1 --2- 172.21.15.96:0/1260563382 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f50000d09e0 0x7f5000160210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:32.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.572+0000 7f4ffd400640 1 --2- 172.21.15.96:0/1260563382 >> v2:172.21.15.123:3300/0 conn(0x7f50000b3690 0x7f50000d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:32.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.572+0000 7f4ffde00640 1 -- 172.21.15.96:0/1260563382 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f50000d09e0 msgr2=0x7f5000160210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:32.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.572+0000 7f4ffde00640 1 --2- 172.21.15.96:0/1260563382 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f50000d09e0 0x7f5000160210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:32.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.572+0000 7f4ffde00640 1 -- 172.21.15.96:0/1260563382 >> v2:172.21.15.123:3300/0 conn(0x7f50000b3690 msgr2=0x7f50000d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:32.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.572+0000 7f4ffde00640 1 --2- 172.21.15.96:0/1260563382 >> v2:172.21.15.123:3300/0 conn(0x7f50000b3690 0x7f50000d04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:32.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.572+0000 7f4ffde00640 1 -- 172.21.15.96:0/1260563382 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50000b3cc0 con 0x7f500006f860 2024-05-28T06:12:32.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.572+0000 7f4ffe800640 1 --2- 172.21.15.96:0/1260563382 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f50000d09e0 0x7f5000160210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-28T06:12:32.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.574+0000 7f4ffde00640 1 --2- 172.21.15.96:0/1260563382 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f500006f860 0x7f50000b3150 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f5000070120 tx=0x7f4ff002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71fa538e0b1f8e34 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:32.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.574+0000 7f4ffca00640 1 -- 172.21.15.96:0/1260563382 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ff0037070 con 0x7f500006f860 2024-05-28T06:12:32.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.574+0000 7f4ffca00640 1 -- 172.21.15.96:0/1260563382 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ff0032440 con 0x7f500006f860 2024-05-28T06:12:32.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.574+0000 7f4ffca00640 1 -- 172.21.15.96:0/1260563382 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ff0032810 con 0x7f500006f860 2024-05-28T06:12:32.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.575+0000 7f5004e00640 1 -- 172.21.15.96:0/1260563382 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f500006f860 msgr2=0x7f50000b3150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:32.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.575+0000 7f5004e00640 1 --2- 172.21.15.96:0/1260563382 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f500006f860 0x7f50000b3150 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f5000070120 tx=0x7f4ff002a1e0 comp rx=0 tx=0).stop 2024-05-28T06:12:32.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.575+0000 7f5004e00640 1 -- 172.21.15.96:0/1260563382 shutdown_connections 2024-05-28T06:12:32.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.575+0000 7f5004e00640 1 --2- 172.21.15.96:0/1260563382 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f50000d09e0 0x7f5000160210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:32.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.575+0000 7f5004e00640 1 --2- 172.21.15.96:0/1260563382 >> v2:172.21.15.123:3300/0 conn(0x7f50000b3690 0x7f50000d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:32.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.575+0000 7f5004e00640 1 --2- 172.21.15.96:0/1260563382 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f500006f860 0x7f50000b3150 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:32.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.575+0000 7f5004e00640 1 -- 172.21.15.96:0/1260563382 >> 172.21.15.96:0/1260563382 conn(0x7f500006bae0 msgr2=0x7f500006bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:32.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.575+0000 7f5004e00640 1 -- 172.21.15.96:0/1260563382 shutdown_connections 2024-05-28T06:12:32.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.575+0000 7f5004e00640 1 -- 172.21.15.96:0/1260563382 wait complete. 2024-05-28T06:12:32.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.577+0000 7f5004e00640 1 Processor -- start 2024-05-28T06:12:32.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.577+0000 7f5004e00640 1 -- start start 2024-05-28T06:12:32.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.578+0000 7f5004e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f500006f860 0x7f50000adfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:32.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.578+0000 7f5004e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f50000b3690 0x7f5000159290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:32.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.578+0000 7f5004e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f50000d09e0 0x7f50000c8e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:32.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.578+0000 7f5004e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f50000b4560 con 0x7f50000b3690 2024-05-28T06:12:32.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.578+0000 7f5004e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f50000b4280 con 0x7f500006f860 2024-05-28T06:12:32.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.578+0000 7f5004e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f50000b4840 con 0x7f50000d09e0 2024-05-28T06:12:32.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.578+0000 7f4ffd400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f50000b3690 0x7f5000159290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:32.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.578+0000 7f4ffd400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f50000b3690 0x7f5000159290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44782/0 (socket says 172.21.15.96:44782) 2024-05-28T06:12:32.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.578+0000 7f4ffd400640 1 -- 172.21.15.96:0/2440732810 learned_addr learned my addr 172.21.15.96:0/2440732810 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:32.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.578+0000 7f4ffde00640 1 --2- 172.21.15.96:0/2440732810 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f500006f860 0x7f50000adfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:32.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.578+0000 7f4ffe800640 1 --2- 172.21.15.96:0/2440732810 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f50000d09e0 0x7f50000c8e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:32.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.579+0000 7f4ffd400640 1 -- 172.21.15.96:0/2440732810 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f50000d09e0 msgr2=0x7f50000c8e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:32.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.579+0000 7f4ffd400640 1 --2- 172.21.15.96:0/2440732810 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f50000d09e0 0x7f50000c8e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:32.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.579+0000 7f4ffd400640 1 -- 172.21.15.96:0/2440732810 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f500006f860 msgr2=0x7f50000adfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:32.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.579+0000 7f4ffd400640 1 --2- 172.21.15.96:0/2440732810 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f500006f860 0x7f50000adfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:32.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.579+0000 7f4ffd400640 1 -- 172.21.15.96:0/2440732810 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50000c43a0 con 0x7f50000b3690 2024-05-28T06:12:32.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.579+0000 7f4ffde00640 1 --2- 172.21.15.96:0/2440732810 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f500006f860 0x7f50000adfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:12:32.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.579+0000 7f4ffd400640 1 --2- 172.21.15.96:0/2440732810 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f50000b3690 0x7f5000159290 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f4fec00d830 tx=0x7f4fec00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:32.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.579+0000 7f4ff6000640 1 -- 172.21.15.96:0/2440732810 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fec018020 con 0x7f50000b3690 2024-05-28T06:12:32.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.579+0000 7f4ff6000640 1 -- 172.21.15.96:0/2440732810 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fec0096a0 con 0x7f50000b3690 2024-05-28T06:12:32.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.579+0000 7f5004e00640 1 -- 172.21.15.96:0/2440732810 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f500015ec00 con 0x7f50000b3690 2024-05-28T06:12:32.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.579+0000 7f4ff6000640 1 -- 172.21.15.96:0/2440732810 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fec009a70 con 0x7f50000b3690 2024-05-28T06:12:32.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.579+0000 7f5004e00640 1 -- 172.21.15.96:0/2440732810 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f500015f1c0 con 0x7f50000b3690 2024-05-28T06:12:32.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.580+0000 7f5004e00640 1 -- 172.21.15.96:0/2440732810 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50000689e0 con 0x7f50000b3690 2024-05-28T06:12:32.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.582+0000 7f4ff6000640 1 -- 172.21.15.96:0/2440732810 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 16) ==== 91754+0+0 (secure 0 0 0) 0x7f4fec01b020 con 0x7f50000b3690 2024-05-28T06:12:32.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.584+0000 7f4ff6000640 1 --2- 172.21.15.96:0/2440732810 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4fe006d0d0 0x7f4fe006f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:32.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.584+0000 7f4ff6000640 1 -- 172.21.15.96:0/2440732810 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5737+0+0 (secure 0 0 0) 0x7f4fec08ac70 con 0x7f50000b3690 2024-05-28T06:12:32.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.584+0000 7f4ffde00640 1 --2- 172.21.15.96:0/2440732810 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4fe006d0d0 0x7f4fe006f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:32.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.585+0000 7f4ffde00640 1 --2- 172.21.15.96:0/2440732810 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4fe006d0d0 0x7f4fe006f590 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f4ff0034000 tx=0x7f4ff002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:32.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.588+0000 7f4ff6000640 1 -- 172.21.15.96:0/2440732810 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4fec022020 con 0x7f50000b3690 2024-05-28T06:12:32.828 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.826+0000 7f5004e00640 1 -- 172.21.15.96:0/2440732810 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f500006e2f0 con 0x7f50000b3690 2024-05-28T06:12:32.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.828+0000 7f4ff6000640 1 -- 172.21.15.96:0/2440732810 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+5184 (secure 0 0 0) 0x7f50000c43a0 con 0x7f50000b3690 2024-05-28T06:12:32.829 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:12:32.829 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":34,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5015,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/1869890774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":1869890774},{"type":"v1","addr":"172.21.15.123:6837","nonce":1869890774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":5386,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/873923319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":873923319},{"type":"v1","addr":"172.21.15.123:6835","nonce":873923319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":6486,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6834/1041571401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":1041571401},{"type":"v1","addr":"172.21.15.96:6834","nonce":1041571401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-28T06:12:28.268409+0000","modified":"2024-05-28T06:12:31.363489+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6489},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6489":{"gid":6489,"name":"a","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.96:6835/1590650245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6833","nonce":1590650245},{"type":"v1","addr":"172.21.15.96:6835","nonce":1590650245}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6489,"qdb_cluster":[6489]},"id":4}]} 2024-05-28T06:12:32.830 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 34 2024-05-28T06:12:32.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.830+0000 7f5004e00640 1 -- 172.21.15.96:0/2440732810 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4fe006d0d0 msgr2=0x7f4fe006f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:32.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.830+0000 7f5004e00640 1 --2- 172.21.15.96:0/2440732810 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4fe006d0d0 0x7f4fe006f590 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f4ff0034000 tx=0x7f4ff002e000 comp rx=0 tx=0).stop 2024-05-28T06:12:32.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.830+0000 7f5004e00640 1 -- 172.21.15.96:0/2440732810 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f50000b3690 msgr2=0x7f5000159290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:32.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.830+0000 7f5004e00640 1 --2- 172.21.15.96:0/2440732810 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f50000b3690 0x7f5000159290 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f4fec00d830 tx=0x7f4fec00dcf0 comp rx=0 tx=0).stop 2024-05-28T06:12:32.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.830+0000 7f5004e00640 1 -- 172.21.15.96:0/2440732810 shutdown_connections 2024-05-28T06:12:32.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.830+0000 7f5004e00640 1 --2- 172.21.15.96:0/2440732810 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4fe006d0d0 0x7f4fe006f590 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:32.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.830+0000 7f5004e00640 1 --2- 172.21.15.96:0/2440732810 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f50000d09e0 0x7f50000c8e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:32.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.830+0000 7f5004e00640 1 --2- 172.21.15.96:0/2440732810 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f50000b3690 0x7f5000159290 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:32.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.830+0000 7f5004e00640 1 --2- 172.21.15.96:0/2440732810 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f500006f860 0x7f50000adfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:32.833 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.830+0000 7f5004e00640 1 -- 172.21.15.96:0/2440732810 >> 172.21.15.96:0/2440732810 conn(0x7f500006bae0 msgr2=0x7f50000b0b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:32.833 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.830+0000 7f5004e00640 1 -- 172.21.15.96:0/2440732810 shutdown_connections 2024-05-28T06:12:32.833 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:32.830+0000 7f5004e00640 1 -- 172.21.15.96:0/2440732810 wait complete. 2024-05-28T06:12:32.850 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:12:33.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.046+0000 7f460fe00640 1 Processor -- start 2024-05-28T06:12:33.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.046+0000 7f460fe00640 1 -- start start 2024-05-28T06:12:33.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.048+0000 7f460fe00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f4610156b30 0x7f46100af6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:33.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.048+0000 7f460fe00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f461006ee40 0x7f46100ad230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:33.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.048+0000 7f460fe00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f461006f2f0 0x7f46100ad770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:33.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.048+0000 7f460fe00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f4610156f10 con 0x7f4610156b30 2024-05-28T06:12:33.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.048+0000 7f460fe00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f461006dda0 con 0x7f461006ee40 2024-05-28T06:12:33.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.048+0000 7f460fe00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f461006e080 con 0x7f461006f2f0 2024-05-28T06:12:33.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.048+0000 7f460ea00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f4610156b30 0x7f46100af6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:33.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.048+0000 7f460ea00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f4610156b30 0x7f46100af6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44794/0 (socket says 172.21.15.96:44794) 2024-05-28T06:12:33.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.048+0000 7f460ea00640 1 -- 172.21.15.96:0/2749662597 learned_addr learned my addr 172.21.15.96:0/2749662597 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:33.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.048+0000 7f460e000640 1 --2- 172.21.15.96:0/2749662597 >> v2:172.21.15.123:3300/0 conn(0x7f461006ee40 0x7f46100ad230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:33.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.049+0000 7f460f400640 1 --2- 172.21.15.96:0/2749662597 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f461006f2f0 0x7f46100ad770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:33.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.049+0000 7f460ea00640 1 -- 172.21.15.96:0/2749662597 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f461006f2f0 msgr2=0x7f46100ad770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:33.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.049+0000 7f460ea00640 1 --2- 172.21.15.96:0/2749662597 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f461006f2f0 0x7f46100ad770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:33.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.049+0000 7f460ea00640 1 -- 172.21.15.96:0/2749662597 >> v2:172.21.15.123:3300/0 conn(0x7f461006ee40 msgr2=0x7f46100ad230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:33.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.049+0000 7f460ea00640 1 --2- 172.21.15.96:0/2749662597 >> v2:172.21.15.123:3300/0 conn(0x7f461006ee40 0x7f46100ad230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:33.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.049+0000 7f460ea00640 1 -- 172.21.15.96:0/2749662597 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f461006e360 con 0x7f4610156b30 2024-05-28T06:12:33.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.049+0000 7f460e000640 1 --2- 172.21.15.96:0/2749662597 >> v2:172.21.15.123:3300/0 conn(0x7f461006ee40 0x7f46100ad230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-28T06:12:33.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.050+0000 7f460ea00640 1 --2- 172.21.15.96:0/2749662597 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4610156b30 0x7f46100af6e0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f461006fdf0 tx=0x7f460002a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ff495b0854e519a server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:33.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.051+0000 7f460d600640 1 -- 172.21.15.96:0/2749662597 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4600037070 con 0x7f4610156b30 2024-05-28T06:12:33.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.051+0000 7f460d600640 1 -- 172.21.15.96:0/2749662597 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4600032440 con 0x7f4610156b30 2024-05-28T06:12:33.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.051+0000 7f460d600640 1 -- 172.21.15.96:0/2749662597 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4600032810 con 0x7f4610156b30 2024-05-28T06:12:33.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.051+0000 7f460fe00640 1 -- 172.21.15.96:0/2749662597 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4610156b30 msgr2=0x7f46100af6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:33.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.051+0000 7f460fe00640 1 --2- 172.21.15.96:0/2749662597 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4610156b30 0x7f46100af6e0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f461006fdf0 tx=0x7f460002a140 comp rx=0 tx=0).stop 2024-05-28T06:12:33.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.052+0000 7f460fe00640 1 -- 172.21.15.96:0/2749662597 shutdown_connections 2024-05-28T06:12:33.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.052+0000 7f460fe00640 1 --2- 172.21.15.96:0/2749662597 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f461006f2f0 0x7f46100ad770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:33.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.052+0000 7f460fe00640 1 --2- 172.21.15.96:0/2749662597 >> v2:172.21.15.123:3300/0 conn(0x7f461006ee40 0x7f46100ad230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:33.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.052+0000 7f460fe00640 1 --2- 172.21.15.96:0/2749662597 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4610156b30 0x7f46100af6e0 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:33.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.052+0000 7f460fe00640 1 -- 172.21.15.96:0/2749662597 >> 172.21.15.96:0/2749662597 conn(0x7f46100b1fc0 msgr2=0x7f46100b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:33.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.052+0000 7f460fe00640 1 -- 172.21.15.96:0/2749662597 shutdown_connections 2024-05-28T06:12:33.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.052+0000 7f460fe00640 1 -- 172.21.15.96:0/2749662597 wait complete. 2024-05-28T06:12:33.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.053+0000 7f460fe00640 1 Processor -- start 2024-05-28T06:12:33.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.054+0000 7f460fe00640 1 -- start start 2024-05-28T06:12:33.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.054+0000 7f460fe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f461006ee40 0x7f4610144710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:33.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.054+0000 7f460fe00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f461006f2f0 0x7f461015e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:33.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.055+0000 7f460fe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4610156b30 0x7f461015e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:33.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.055+0000 7f460fe00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f46100afc20 con 0x7f4610156b30 2024-05-28T06:12:33.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.055+0000 7f460fe00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f46100b2c30 con 0x7f461006ee40 2024-05-28T06:12:33.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.055+0000 7f460fe00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f46101f41c0 con 0x7f461006f2f0 2024-05-28T06:12:33.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.055+0000 7f460f400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4610156b30 0x7f461015e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:33.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.055+0000 7f460f400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4610156b30 0x7f461015e8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44806/0 (socket says 172.21.15.96:44806) 2024-05-28T06:12:33.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.055+0000 7f460f400640 1 -- 172.21.15.96:0/2099731305 learned_addr learned my addr 172.21.15.96:0/2099731305 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:33.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.055+0000 7f460ea00640 1 --2- 172.21.15.96:0/2099731305 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f461006ee40 0x7f4610144710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:33.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.055+0000 7f460e000640 1 --2- 172.21.15.96:0/2099731305 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f461006f2f0 0x7f461015e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:33.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.055+0000 7f460f400640 1 -- 172.21.15.96:0/2099731305 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f461006f2f0 msgr2=0x7f461015e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:33.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.055+0000 7f460f400640 1 --2- 172.21.15.96:0/2099731305 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f461006f2f0 0x7f461015e360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:33.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.055+0000 7f460f400640 1 -- 172.21.15.96:0/2099731305 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f461006ee40 msgr2=0x7f4610144710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:33.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.055+0000 7f460f400640 1 --2- 172.21.15.96:0/2099731305 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f461006ee40 0x7f4610144710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:33.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.055+0000 7f460f400640 1 -- 172.21.15.96:0/2099731305 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f461009d700 con 0x7f4610156b30 2024-05-28T06:12:33.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.056+0000 7f460f400640 1 --2- 172.21.15.96:0/2099731305 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4610156b30 0x7f461015e8a0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f4608007cf0 tx=0x7f460800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:33.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.056+0000 7f4606a00640 1 -- 172.21.15.96:0/2099731305 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4608014420 con 0x7f4610156b30 2024-05-28T06:12:33.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.056+0000 7f4606a00640 1 -- 172.21.15.96:0/2099731305 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4608019070 con 0x7f4610156b30 2024-05-28T06:12:33.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.056+0000 7f4606a00640 1 -- 172.21.15.96:0/2099731305 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4608014820 con 0x7f4610156b30 2024-05-28T06:12:33.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.056+0000 7f460fe00640 1 -- 172.21.15.96:0/2099731305 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f461015eed0 con 0x7f4610156b30 2024-05-28T06:12:33.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.056+0000 7f460fe00640 1 -- 172.21.15.96:0/2099731305 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4610157760 con 0x7f4610156b30 2024-05-28T06:12:33.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.057+0000 7f460fe00640 1 -- 172.21.15.96:0/2099731305 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46100689e0 con 0x7f4610156b30 2024-05-28T06:12:33.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.058+0000 7f4606a00640 1 -- 172.21.15.96:0/2099731305 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 16) ==== 91754+0+0 (secure 0 0 0) 0x7f46080149c0 con 0x7f4610156b30 2024-05-28T06:12:33.061 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.059+0000 7f4606a00640 1 --2- 172.21.15.96:0/2099731305 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f45f006d0d0 0x7f45f006f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:33.061 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.059+0000 7f4606a00640 1 -- 172.21.15.96:0/2099731305 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5737+0+0 (secure 0 0 0) 0x7f460808d640 con 0x7f4610156b30 2024-05-28T06:12:33.061 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.060+0000 7f460ea00640 1 --2- 172.21.15.96:0/2099731305 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f45f006d0d0 0x7f45f006f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:33.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.061+0000 7f460ea00640 1 --2- 172.21.15.96:0/2099731305 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f45f006d0d0 0x7f45f006f590 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f4600034000 tx=0x7f460002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:33.065 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.064+0000 7f4606a00640 1 -- 172.21.15.96:0/2099731305 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f460805aa30 con 0x7f4610156b30 2024-05-28T06:12:33.173 INFO:tasks.ceph.mds.a.smithi096.stderr:2024-05-28T06:12:33.172+0000 7f3a9b400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-28T06:12:33.270 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.269+0000 7f460fe00640 1 -- 172.21.15.96:0/2099731305 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f461006f7f0 con 0x7f4610156b30 2024-05-28T06:12:33.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.271+0000 7f4606a00640 1 -- 172.21.15.96:0/2099731305 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v60) ==== 74+0+16720 (secure 0 0 0) 0x7f460805a330 con 0x7f4610156b30 2024-05-28T06:12:33.273 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:12:33.273 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":60,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:12:29.836221+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-05-28T06:12:26.024060+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-05-28T06:12:27.254401+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:12:33.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.273+0000 7f460fe00640 1 -- 172.21.15.96:0/2099731305 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f45f006d0d0 msgr2=0x7f45f006f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:33.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.273+0000 7f460fe00640 1 --2- 172.21.15.96:0/2099731305 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f45f006d0d0 0x7f45f006f590 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f4600034000 tx=0x7f460002e000 comp rx=0 tx=0).stop 2024-05-28T06:12:33.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.273+0000 7f460fe00640 1 -- 172.21.15.96:0/2099731305 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4610156b30 msgr2=0x7f461015e8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:33.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.273+0000 7f460fe00640 1 --2- 172.21.15.96:0/2099731305 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4610156b30 0x7f461015e8a0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f4608007cf0 tx=0x7f460800e4b0 comp rx=0 tx=0).stop 2024-05-28T06:12:33.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.273+0000 7f460fe00640 1 -- 172.21.15.96:0/2099731305 shutdown_connections 2024-05-28T06:12:33.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.273+0000 7f460fe00640 1 --2- 172.21.15.96:0/2099731305 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f45f006d0d0 0x7f45f006f590 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:33.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.273+0000 7f460fe00640 1 --2- 172.21.15.96:0/2099731305 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4610156b30 0x7f461015e8a0 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:33.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.273+0000 7f460fe00640 1 --2- 172.21.15.96:0/2099731305 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f461006f2f0 0x7f461015e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:33.276 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.273+0000 7f460fe00640 1 --2- 172.21.15.96:0/2099731305 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f461006ee40 0x7f4610144710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:33.276 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.273+0000 7f460fe00640 1 -- 172.21.15.96:0/2099731305 >> 172.21.15.96:0/2099731305 conn(0x7f46100b1fc0 msgr2=0x7f46100ad230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:33.276 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.274+0000 7f460fe00640 1 -- 172.21.15.96:0/2099731305 shutdown_connections 2024-05-28T06:12:33.276 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.274+0000 7f460fe00640 1 -- 172.21.15.96:0/2099731305 wait complete. 2024-05-28T06:12:33.294 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-05-28T06:12:33.294 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-28T06:12:33.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.492+0000 7fccc2c00640 1 Processor -- start 2024-05-28T06:12:33.494 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.492+0000 7fccc2c00640 1 -- start start 2024-05-28T06:12:33.495 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.493+0000 7fccc2c00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fccbc156b30 0x7fccbc0af6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:33.495 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.493+0000 7fccc2c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fccbc06ee40 0x7fccbc0ad230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:33.495 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.493+0000 7fccc2c00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fccbc06f2c0 0x7fccbc0ad770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:33.495 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.493+0000 7fccc2c00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fccbc156f10 con 0x7fccbc156b30 2024-05-28T06:12:33.495 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.493+0000 7fccc2c00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fccbc06dda0 con 0x7fccbc06f2c0 2024-05-28T06:12:33.496 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.493+0000 7fccc2c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fccbc06e080 con 0x7fccbc06ee40 2024-05-28T06:12:33.496 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.494+0000 7fccc1800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fccbc156b30 0x7fccbc0af6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:33.496 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.494+0000 7fccc1800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fccbc156b30 0x7fccbc0af6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44820/0 (socket says 172.21.15.96:44820) 2024-05-28T06:12:33.496 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.494+0000 7fccc1800640 1 -- 172.21.15.96:0/3954752323 learned_addr learned my addr 172.21.15.96:0/3954752323 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:33.496 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.494+0000 7fccc0e00640 1 --2- 172.21.15.96:0/3954752323 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fccbc06ee40 0x7fccbc0ad230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:33.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.494+0000 7fccc2200640 1 --2- 172.21.15.96:0/3954752323 >> v2:172.21.15.123:3300/0 conn(0x7fccbc06f2c0 0x7fccbc0ad770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:33.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.495+0000 7fccc1800640 1 -- 172.21.15.96:0/3954752323 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fccbc06ee40 msgr2=0x7fccbc0ad230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:33.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.495+0000 7fccc1800640 1 --2- 172.21.15.96:0/3954752323 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fccbc06ee40 0x7fccbc0ad230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:33.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.495+0000 7fccc1800640 1 -- 172.21.15.96:0/3954752323 >> v2:172.21.15.123:3300/0 conn(0x7fccbc06f2c0 msgr2=0x7fccbc0ad770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:33.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.495+0000 7fccc1800640 1 --2- 172.21.15.96:0/3954752323 >> v2:172.21.15.123:3300/0 conn(0x7fccbc06f2c0 0x7fccbc0ad770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:33.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.495+0000 7fccc1800640 1 -- 172.21.15.96:0/3954752323 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccbc06e360 con 0x7fccbc156b30 2024-05-28T06:12:33.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.495+0000 7fccc2200640 1 --2- 172.21.15.96:0/3954752323 >> v2:172.21.15.123:3300/0 conn(0x7fccbc06f2c0 0x7fccbc0ad770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-28T06:12:33.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.496+0000 7fccc1800640 1 --2- 172.21.15.96:0/3954752323 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fccbc156b30 0x7fccbc0af6e0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fccbc071410 tx=0x7fccb002a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1b8f6e765d34d71 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:33.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.497+0000 7fccbbe00640 1 -- 172.21.15.96:0/3954752323 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccb0036070 con 0x7fccbc156b30 2024-05-28T06:12:33.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.497+0000 7fccbbe00640 1 -- 172.21.15.96:0/3954752323 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fccb0031600 con 0x7fccbc156b30 2024-05-28T06:12:33.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.497+0000 7fccbbe00640 1 -- 172.21.15.96:0/3954752323 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccb00319d0 con 0x7fccbc156b30 2024-05-28T06:12:33.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.497+0000 7fccc2c00640 1 -- 172.21.15.96:0/3954752323 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fccbc156b30 msgr2=0x7fccbc0af6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:33.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.497+0000 7fccc2c00640 1 --2- 172.21.15.96:0/3954752323 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fccbc156b30 0x7fccbc0af6e0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fccbc071410 tx=0x7fccb002a610 comp rx=0 tx=0).stop 2024-05-28T06:12:33.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.497+0000 7fccc2c00640 1 -- 172.21.15.96:0/3954752323 shutdown_connections 2024-05-28T06:12:33.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.497+0000 7fccc2c00640 1 --2- 172.21.15.96:0/3954752323 >> v2:172.21.15.123:3300/0 conn(0x7fccbc06f2c0 0x7fccbc0ad770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:33.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.498+0000 7fccc2c00640 1 --2- 172.21.15.96:0/3954752323 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fccbc06ee40 0x7fccbc0ad230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:33.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.498+0000 7fccc2c00640 1 --2- 172.21.15.96:0/3954752323 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fccbc156b30 0x7fccbc0af6e0 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:33.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.498+0000 7fccc2c00640 1 -- 172.21.15.96:0/3954752323 >> 172.21.15.96:0/3954752323 conn(0x7fccbc0b1fc0 msgr2=0x7fccbc0b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:33.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.498+0000 7fccc2c00640 1 -- 172.21.15.96:0/3954752323 shutdown_connections 2024-05-28T06:12:33.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.498+0000 7fccc2c00640 1 -- 172.21.15.96:0/3954752323 wait complete. 2024-05-28T06:12:33.501 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.499+0000 7fccc2c00640 1 Processor -- start 2024-05-28T06:12:33.501 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.500+0000 7fccc2c00640 1 -- start start 2024-05-28T06:12:33.502 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.501+0000 7fccc2c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fccbc06ee40 0x7fccbc071dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:33.502 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.501+0000 7fccc2c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fccbc06f2c0 0x7fccbc159a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:33.502 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.501+0000 7fccc2c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fccbc156b30 0x7fccbc1f45d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:33.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.501+0000 7fccc2c00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fccbc0afc20 con 0x7fccbc06f2c0 2024-05-28T06:12:33.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.501+0000 7fccc2c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fccbc0b2c30 con 0x7fccbc06ee40 2024-05-28T06:12:33.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.501+0000 7fccc2c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fccbc159fb0 con 0x7fccbc156b30 2024-05-28T06:12:33.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.501+0000 7fccc0e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fccbc06f2c0 0x7fccbc159a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:33.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.501+0000 7fccc0e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fccbc06f2c0 0x7fccbc159a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44824/0 (socket says 172.21.15.96:44824) 2024-05-28T06:12:33.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.501+0000 7fccc0e00640 1 -- 172.21.15.96:0/380380650 learned_addr learned my addr 172.21.15.96:0/380380650 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:33.506 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.501+0000 7fccc1800640 1 --2- 172.21.15.96:0/380380650 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fccbc06ee40 0x7fccbc071dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:33.506 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.501+0000 7fccc2200640 1 --2- 172.21.15.96:0/380380650 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fccbc156b30 0x7fccbc1f45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:33.506 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.501+0000 7fccc0e00640 1 -- 172.21.15.96:0/380380650 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fccbc156b30 msgr2=0x7fccbc1f45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:33.506 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.501+0000 7fccc0e00640 1 --2- 172.21.15.96:0/380380650 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fccbc156b30 0x7fccbc1f45d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:33.506 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.501+0000 7fccc0e00640 1 -- 172.21.15.96:0/380380650 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fccbc06ee40 msgr2=0x7fccbc071dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:33.507 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.501+0000 7fccc0e00640 1 --2- 172.21.15.96:0/380380650 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fccbc06ee40 0x7fccbc071dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:33.507 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.501+0000 7fccc0e00640 1 -- 172.21.15.96:0/380380650 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccbc0b42d0 con 0x7fccbc06f2c0 2024-05-28T06:12:33.507 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.502+0000 7fccc0e00640 1 --2- 172.21.15.96:0/380380650 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fccbc06f2c0 0x7fccbc159a70 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fcca800d800 tx=0x7fcca800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:33.507 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.502+0000 7fccb9600640 1 -- 172.21.15.96:0/380380650 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcca8018030 con 0x7fccbc06f2c0 2024-05-28T06:12:33.507 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.502+0000 7fccb9600640 1 -- 172.21.15.96:0/380380650 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcca80096a0 con 0x7fccbc06f2c0 2024-05-28T06:12:33.507 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.502+0000 7fccb9600640 1 -- 172.21.15.96:0/380380650 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcca8009870 con 0x7fccbc06f2c0 2024-05-28T06:12:33.507 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.502+0000 7fccc2c00640 1 -- 172.21.15.96:0/380380650 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fccbc0b45b0 con 0x7fccbc06f2c0 2024-05-28T06:12:33.507 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.502+0000 7fccc2c00640 1 -- 172.21.15.96:0/380380650 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fccbc1f4b10 con 0x7fccbc06f2c0 2024-05-28T06:12:33.508 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.503+0000 7fccc2c00640 1 -- 172.21.15.96:0/380380650 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fccbc0689e0 con 0x7fccbc06f2c0 2024-05-28T06:12:33.508 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.504+0000 7fccb9600640 1 -- 172.21.15.96:0/380380650 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 16) ==== 91754+0+0 (secure 0 0 0) 0x7fcca801b070 con 0x7fccbc06f2c0 2024-05-28T06:12:33.509 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.506+0000 7fccb9600640 1 --2- 172.21.15.96:0/380380650 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fcc9c06d100 0x7fcc9c06f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:33.509 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.506+0000 7fccb9600640 1 -- 172.21.15.96:0/380380650 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5737+0+0 (secure 0 0 0) 0x7fcca808d000 con 0x7fccbc06f2c0 2024-05-28T06:12:33.509 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.506+0000 7fccc1800640 1 --2- 172.21.15.96:0/380380650 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fcc9c06d100 0x7fcc9c06f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:33.510 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.508+0000 7fccc1800640 1 --2- 172.21.15.96:0/380380650 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fcc9c06d100 0x7fcc9c06f5c0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fccb0033000 tx=0x7fccb0009000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:33.512 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.510+0000 7fccb9600640 1 -- 172.21.15.96:0/380380650 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fcca805a480 con 0x7fccbc06f2c0 2024-05-28T06:12:33.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.705+0000 7fccc2c00640 1 -- 172.21.15.96:0/380380650 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fccbc1562e0 con 0x7fcc9c06d100 2024-05-28T06:12:33.729 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.727+0000 7fccb9600640 1 -- 172.21.15.96:0/380380650 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251884 (secure 0 0 0) 0x7fccbc1562e0 con 0x7fcc9c06d100 2024-05-28T06:12:33.733 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:12:33.734 INFO:teuthology.orchestra.run.smithi096.stderr:dumped all 2024-05-28T06:12:33.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.732+0000 7fccc2c00640 1 -- 172.21.15.96:0/380380650 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fcc9c06d100 msgr2=0x7fcc9c06f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:33.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.732+0000 7fccc2c00640 1 --2- 172.21.15.96:0/380380650 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fcc9c06d100 0x7fcc9c06f5c0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fccb0033000 tx=0x7fccb0009000 comp rx=0 tx=0).stop 2024-05-28T06:12:33.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.732+0000 7fccc2c00640 1 -- 172.21.15.96:0/380380650 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fccbc06f2c0 msgr2=0x7fccbc159a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:33.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.732+0000 7fccc2c00640 1 --2- 172.21.15.96:0/380380650 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fccbc06f2c0 0x7fccbc159a70 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fcca800d800 tx=0x7fcca800dcc0 comp rx=0 tx=0).stop 2024-05-28T06:12:33.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.733+0000 7fccc2c00640 1 -- 172.21.15.96:0/380380650 shutdown_connections 2024-05-28T06:12:33.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.733+0000 7fccc2c00640 1 --2- 172.21.15.96:0/380380650 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fcc9c06d100 0x7fcc9c06f5c0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:33.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.733+0000 7fccc2c00640 1 --2- 172.21.15.96:0/380380650 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fccbc156b30 0x7fccbc1f45d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:33.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.733+0000 7fccc2c00640 1 --2- 172.21.15.96:0/380380650 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fccbc06f2c0 0x7fccbc159a70 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:33.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.733+0000 7fccc2c00640 1 --2- 172.21.15.96:0/380380650 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fccbc06ee40 0x7fccbc071dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:33.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.733+0000 7fccc2c00640 1 -- 172.21.15.96:0/380380650 >> 172.21.15.96:0/380380650 conn(0x7fccbc0b1fc0 msgr2=0x7fccbc15bd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:33.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.733+0000 7fccc2c00640 1 -- 172.21.15.96:0/380380650 shutdown_connections 2024-05-28T06:12:33.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:33.733+0000 7fccc2c00640 1 -- 172.21.15.96:0/380380650 wait complete. 2024-05-28T06:12:33.754 INFO:teuthology.orchestra.run.smithi096.stdout:{"pg_ready":true,"pg_map":{"version":310,"stamp":"2024-05-28T06:12:33.685854+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":83,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":353424,"kb_used_data":6740,"kb_used_omap":12,"kb_used_meta":346547,"kb_avail":754621296,"statfs":{"total":773094113280,"available":772732207104,"internally_reserved":0,"allocated":6901760,"data_stored":3560961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":354864688},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002196"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.860311+0000","last_change":"2024-05-28T06:12:28.853943+0000","last_active":"2024-05-28T06:12:28.860311+0000","last_peered":"2024-05-28T06:12:28.860311+0000","last_clean":"2024-05-28T06:12:28.860311+0000","last_became_active":"2024-05-28T06:12:28.853633+0000","last_became_peered":"2024-05-28T06:12:28.853633+0000","last_unstale":"2024-05-28T06:12:28.860311+0000","last_undegraded":"2024-05-28T06:12:28.860311+0000","last_fullsized":"2024-05-28T06:12:28.860311+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:28:27.360488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.869676+0000","last_change":"2024-05-28T06:12:28.866724+0000","last_active":"2024-05-28T06:12:28.869676+0000","last_peered":"2024-05-28T06:12:28.869676+0000","last_clean":"2024-05-28T06:12:28.869676+0000","last_became_active":"2024-05-28T06:12:28.866427+0000","last_became_peered":"2024-05-28T06:12:28.866427+0000","last_unstale":"2024-05-28T06:12:28.869676+0000","last_undegraded":"2024-05-28T06:12:28.869676+0000","last_fullsized":"2024-05-28T06:12:28.869676+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:37:20.555345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.861314+0000","last_change":"2024-05-28T06:12:28.855620+0000","last_active":"2024-05-28T06:12:28.861314+0000","last_peered":"2024-05-28T06:12:28.861314+0000","last_clean":"2024-05-28T06:12:28.861314+0000","last_became_active":"2024-05-28T06:12:28.854897+0000","last_became_peered":"2024-05-28T06:12:28.854897+0000","last_unstale":"2024-05-28T06:12:28.861314+0000","last_undegraded":"2024-05-28T06:12:28.861314+0000","last_fullsized":"2024-05-28T06:12:28.861314+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:15:05.036715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.856899+0000","last_change":"2024-05-28T06:12:28.852854+0000","last_active":"2024-05-28T06:12:28.856899+0000","last_peered":"2024-05-28T06:12:28.856899+0000","last_clean":"2024-05-28T06:12:28.856899+0000","last_became_active":"2024-05-28T06:12:28.852546+0000","last_became_peered":"2024-05-28T06:12:28.852546+0000","last_unstale":"2024-05-28T06:12:28.856899+0000","last_undegraded":"2024-05-28T06:12:28.856899+0000","last_fullsized":"2024-05-28T06:12:28.856899+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:24:41.895529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.863971+0000","last_change":"2024-05-28T06:12:28.860354+0000","last_active":"2024-05-28T06:12:28.863971+0000","last_peered":"2024-05-28T06:12:28.863971+0000","last_clean":"2024-05-28T06:12:28.863971+0000","last_became_active":"2024-05-28T06:12:28.860078+0000","last_became_peered":"2024-05-28T06:12:28.860078+0000","last_unstale":"2024-05-28T06:12:28.863971+0000","last_undegraded":"2024-05-28T06:12:28.863971+0000","last_fullsized":"2024-05-28T06:12:28.863971+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:17:46.322652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.865243+0000","last_change":"2024-05-28T06:12:28.853205+0000","last_active":"2024-05-28T06:12:28.865243+0000","last_peered":"2024-05-28T06:12:28.865243+0000","last_clean":"2024-05-28T06:12:28.865243+0000","last_became_active":"2024-05-28T06:12:28.852876+0000","last_became_peered":"2024-05-28T06:12:28.852876+0000","last_unstale":"2024-05-28T06:12:28.865243+0000","last_undegraded":"2024-05-28T06:12:28.865243+0000","last_fullsized":"2024-05-28T06:12:28.865243+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:57:13.899914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.856849+0000","last_change":"2024-05-28T06:12:28.853316+0000","last_active":"2024-05-28T06:12:28.856849+0000","last_peered":"2024-05-28T06:12:28.856849+0000","last_clean":"2024-05-28T06:12:28.856849+0000","last_became_active":"2024-05-28T06:12:28.853033+0000","last_became_peered":"2024-05-28T06:12:28.853033+0000","last_unstale":"2024-05-28T06:12:28.856849+0000","last_undegraded":"2024-05-28T06:12:28.856849+0000","last_fullsized":"2024-05-28T06:12:28.856849+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:08:29.323479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.867643+0000","last_change":"2024-05-28T06:12:28.858832+0000","last_active":"2024-05-28T06:12:28.867643+0000","last_peered":"2024-05-28T06:12:28.867643+0000","last_clean":"2024-05-28T06:12:28.867643+0000","last_became_active":"2024-05-28T06:12:28.858538+0000","last_became_peered":"2024-05-28T06:12:28.858538+0000","last_unstale":"2024-05-28T06:12:28.867643+0000","last_undegraded":"2024-05-28T06:12:28.867643+0000","last_fullsized":"2024-05-28T06:12:28.867643+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:32:01.579691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.842424+0000","last_change":"2024-05-28T06:12:28.836889+0000","last_active":"2024-05-28T06:12:28.842424+0000","last_peered":"2024-05-28T06:12:28.842424+0000","last_clean":"2024-05-28T06:12:28.842424+0000","last_became_active":"2024-05-28T06:12:28.836070+0000","last_became_peered":"2024-05-28T06:12:28.836070+0000","last_unstale":"2024-05-28T06:12:28.842424+0000","last_undegraded":"2024-05-28T06:12:28.842424+0000","last_fullsized":"2024-05-28T06:12:28.842424+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:25:24.558283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.862524+0000","last_change":"2024-05-28T06:12:28.857691+0000","last_active":"2024-05-28T06:12:28.862524+0000","last_peered":"2024-05-28T06:12:28.862524+0000","last_clean":"2024-05-28T06:12:28.862524+0000","last_became_active":"2024-05-28T06:12:28.857139+0000","last_became_peered":"2024-05-28T06:12:28.857139+0000","last_unstale":"2024-05-28T06:12:28.862524+0000","last_undegraded":"2024-05-28T06:12:28.862524+0000","last_fullsized":"2024-05-28T06:12:28.862524+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:22:54.023449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.867568+0000","last_change":"2024-05-28T06:12:28.866471+0000","last_active":"2024-05-28T06:12:28.867568+0000","last_peered":"2024-05-28T06:12:28.867568+0000","last_clean":"2024-05-28T06:12:28.867568+0000","last_became_active":"2024-05-28T06:12:28.866196+0000","last_became_peered":"2024-05-28T06:12:28.866196+0000","last_unstale":"2024-05-28T06:12:28.867568+0000","last_undegraded":"2024-05-28T06:12:28.867568+0000","last_fullsized":"2024-05-28T06:12:28.867568+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:02:00.111727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.861018+0000","last_change":"2024-05-28T06:12:28.856465+0000","last_active":"2024-05-28T06:12:28.861018+0000","last_peered":"2024-05-28T06:12:28.861018+0000","last_clean":"2024-05-28T06:12:28.861018+0000","last_became_active":"2024-05-28T06:12:28.856136+0000","last_became_peered":"2024-05-28T06:12:28.856136+0000","last_unstale":"2024-05-28T06:12:28.861018+0000","last_undegraded":"2024-05-28T06:12:28.861018+0000","last_fullsized":"2024-05-28T06:12:28.861018+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:51:51.219749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.870688+0000","last_change":"2024-05-28T06:12:28.866498+0000","last_active":"2024-05-28T06:12:28.870688+0000","last_peered":"2024-05-28T06:12:28.870688+0000","last_clean":"2024-05-28T06:12:28.870688+0000","last_became_active":"2024-05-28T06:12:28.866157+0000","last_became_peered":"2024-05-28T06:12:28.866157+0000","last_unstale":"2024-05-28T06:12:28.870688+0000","last_undegraded":"2024-05-28T06:12:28.870688+0000","last_fullsized":"2024-05-28T06:12:28.870688+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:02:00.111727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.856812+0000","last_change":"2024-05-28T06:12:28.852507+0000","last_active":"2024-05-28T06:12:28.856812+0000","last_peered":"2024-05-28T06:12:28.856812+0000","last_clean":"2024-05-28T06:12:28.856812+0000","last_became_active":"2024-05-28T06:12:28.852207+0000","last_became_peered":"2024-05-28T06:12:28.852207+0000","last_unstale":"2024-05-28T06:12:28.856812+0000","last_undegraded":"2024-05-28T06:12:28.856812+0000","last_fullsized":"2024-05-28T06:12:28.856812+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:23:02.057978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.853522+0000","last_change":"2024-05-28T06:12:28.846121+0000","last_active":"2024-05-28T06:12:28.853522+0000","last_peered":"2024-05-28T06:12:28.853522+0000","last_clean":"2024-05-28T06:12:28.853522+0000","last_became_active":"2024-05-28T06:12:28.845778+0000","last_became_peered":"2024-05-28T06:12:28.845778+0000","last_unstale":"2024-05-28T06:12:28.853522+0000","last_undegraded":"2024-05-28T06:12:28.853522+0000","last_fullsized":"2024-05-28T06:12:28.853522+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:29:56.122322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.866341+0000","last_change":"2024-05-28T06:12:28.860149+0000","last_active":"2024-05-28T06:12:28.866341+0000","last_peered":"2024-05-28T06:12:28.866341+0000","last_clean":"2024-05-28T06:12:28.866341+0000","last_became_active":"2024-05-28T06:12:28.853247+0000","last_became_peered":"2024-05-28T06:12:28.853247+0000","last_unstale":"2024-05-28T06:12:28.866341+0000","last_undegraded":"2024-05-28T06:12:28.866341+0000","last_fullsized":"2024-05-28T06:12:28.866341+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:36:53.762642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.860213+0000","last_change":"2024-05-28T06:12:28.854475+0000","last_active":"2024-05-28T06:12:28.860213+0000","last_peered":"2024-05-28T06:12:28.860213+0000","last_clean":"2024-05-28T06:12:28.860213+0000","last_became_active":"2024-05-28T06:12:28.854140+0000","last_became_peered":"2024-05-28T06:12:28.854140+0000","last_unstale":"2024-05-28T06:12:28.860213+0000","last_undegraded":"2024-05-28T06:12:28.860213+0000","last_fullsized":"2024-05-28T06:12:28.860213+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:19:59.499371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.871610+0000","last_change":"2024-05-28T06:12:28.869033+0000","last_active":"2024-05-28T06:12:28.871610+0000","last_peered":"2024-05-28T06:12:28.871610+0000","last_clean":"2024-05-28T06:12:28.871610+0000","last_became_active":"2024-05-28T06:12:28.868747+0000","last_became_peered":"2024-05-28T06:12:28.868747+0000","last_unstale":"2024-05-28T06:12:28.871610+0000","last_undegraded":"2024-05-28T06:12:28.871610+0000","last_fullsized":"2024-05-28T06:12:28.871610+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:34:44.104688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.861855+0000","last_change":"2024-05-28T06:12:28.855732+0000","last_active":"2024-05-28T06:12:28.861855+0000","last_peered":"2024-05-28T06:12:28.861855+0000","last_clean":"2024-05-28T06:12:28.861855+0000","last_became_active":"2024-05-28T06:12:28.854783+0000","last_became_peered":"2024-05-28T06:12:28.854783+0000","last_unstale":"2024-05-28T06:12:28.861855+0000","last_undegraded":"2024-05-28T06:12:28.861855+0000","last_fullsized":"2024-05-28T06:12:28.861855+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:07:56.984961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.871440+0000","last_change":"2024-05-28T06:12:28.866965+0000","last_active":"2024-05-28T06:12:28.871440+0000","last_peered":"2024-05-28T06:12:28.871440+0000","last_clean":"2024-05-28T06:12:28.871440+0000","last_became_active":"2024-05-28T06:12:28.866713+0000","last_became_peered":"2024-05-28T06:12:28.866713+0000","last_unstale":"2024-05-28T06:12:28.871440+0000","last_undegraded":"2024-05-28T06:12:28.871440+0000","last_fullsized":"2024-05-28T06:12:28.871440+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:27:48.034035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.865386+0000","last_change":"2024-05-28T06:12:28.861724+0000","last_active":"2024-05-28T06:12:28.865386+0000","last_peered":"2024-05-28T06:12:28.865386+0000","last_clean":"2024-05-28T06:12:28.865386+0000","last_became_active":"2024-05-28T06:12:28.861435+0000","last_became_peered":"2024-05-28T06:12:28.861435+0000","last_unstale":"2024-05-28T06:12:28.865386+0000","last_undegraded":"2024-05-28T06:12:28.865386+0000","last_fullsized":"2024-05-28T06:12:28.865386+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:05:42.523486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.868419+0000","last_change":"2024-05-28T06:12:28.861993+0000","last_active":"2024-05-28T06:12:28.868419+0000","last_peered":"2024-05-28T06:12:28.868419+0000","last_clean":"2024-05-28T06:12:28.868419+0000","last_became_active":"2024-05-28T06:12:28.860499+0000","last_became_peered":"2024-05-28T06:12:28.860499+0000","last_unstale":"2024-05-28T06:12:28.868419+0000","last_undegraded":"2024-05-28T06:12:28.868419+0000","last_fullsized":"2024-05-28T06:12:28.868419+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:31:46.498176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.863233+0000","last_change":"2024-05-28T06:12:28.858521+0000","last_active":"2024-05-28T06:12:28.863233+0000","last_peered":"2024-05-28T06:12:28.863233+0000","last_clean":"2024-05-28T06:12:28.863233+0000","last_became_active":"2024-05-28T06:12:28.858275+0000","last_became_peered":"2024-05-28T06:12:28.858275+0000","last_unstale":"2024-05-28T06:12:28.863233+0000","last_undegraded":"2024-05-28T06:12:28.863233+0000","last_fullsized":"2024-05-28T06:12:28.863233+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:59:34.090488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.870810+0000","last_change":"2024-05-28T06:12:28.866588+0000","last_active":"2024-05-28T06:12:28.870810+0000","last_peered":"2024-05-28T06:12:28.870810+0000","last_clean":"2024-05-28T06:12:28.870810+0000","last_became_active":"2024-05-28T06:12:28.866248+0000","last_became_peered":"2024-05-28T06:12:28.866248+0000","last_unstale":"2024-05-28T06:12:28.870810+0000","last_undegraded":"2024-05-28T06:12:28.870810+0000","last_fullsized":"2024-05-28T06:12:28.870810+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:13:57.267719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.862617+0000","last_change":"2024-05-28T06:12:28.859005+0000","last_active":"2024-05-28T06:12:28.862617+0000","last_peered":"2024-05-28T06:12:28.862617+0000","last_clean":"2024-05-28T06:12:28.862617+0000","last_became_active":"2024-05-28T06:12:28.858729+0000","last_became_peered":"2024-05-28T06:12:28.858729+0000","last_unstale":"2024-05-28T06:12:28.862617+0000","last_undegraded":"2024-05-28T06:12:28.862617+0000","last_fullsized":"2024-05-28T06:12:28.862617+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:44:54.600832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.862965+0000","last_change":"2024-05-28T06:12:28.857248+0000","last_active":"2024-05-28T06:12:28.862965+0000","last_peered":"2024-05-28T06:12:28.862965+0000","last_clean":"2024-05-28T06:12:28.862965+0000","last_became_active":"2024-05-28T06:12:28.856644+0000","last_became_peered":"2024-05-28T06:12:28.856644+0000","last_unstale":"2024-05-28T06:12:28.862965+0000","last_undegraded":"2024-05-28T06:12:28.862965+0000","last_fullsized":"2024-05-28T06:12:28.862965+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:12:29.534159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.864476+0000","last_change":"2024-05-28T06:12:28.853169+0000","last_active":"2024-05-28T06:12:28.864476+0000","last_peered":"2024-05-28T06:12:28.864476+0000","last_clean":"2024-05-28T06:12:28.864476+0000","last_became_active":"2024-05-28T06:12:28.852864+0000","last_became_peered":"2024-05-28T06:12:28.852864+0000","last_unstale":"2024-05-28T06:12:28.864476+0000","last_undegraded":"2024-05-28T06:12:28.864476+0000","last_fullsized":"2024-05-28T06:12:28.864476+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:12:29.534159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.863013+0000","last_change":"2024-05-28T06:12:28.858942+0000","last_active":"2024-05-28T06:12:28.863013+0000","last_peered":"2024-05-28T06:12:28.863013+0000","last_clean":"2024-05-28T06:12:28.863013+0000","last_became_active":"2024-05-28T06:12:28.858638+0000","last_became_peered":"2024-05-28T06:12:28.858638+0000","last_unstale":"2024-05-28T06:12:28.863013+0000","last_undegraded":"2024-05-28T06:12:28.863013+0000","last_fullsized":"2024-05-28T06:12:28.863013+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:26:19.248645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.867525+0000","last_change":"2024-05-28T06:12:28.858607+0000","last_active":"2024-05-28T06:12:28.867525+0000","last_peered":"2024-05-28T06:12:28.867525+0000","last_clean":"2024-05-28T06:12:28.867525+0000","last_became_active":"2024-05-28T06:12:28.858045+0000","last_became_peered":"2024-05-28T06:12:28.858045+0000","last_unstale":"2024-05-28T06:12:28.867525+0000","last_undegraded":"2024-05-28T06:12:28.867525+0000","last_fullsized":"2024-05-28T06:12:28.867525+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:03:58.734327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.865415+0000","last_change":"2024-05-28T06:12:28.859109+0000","last_active":"2024-05-28T06:12:28.865415+0000","last_peered":"2024-05-28T06:12:28.865415+0000","last_clean":"2024-05-28T06:12:28.865415+0000","last_became_active":"2024-05-28T06:12:28.858826+0000","last_became_peered":"2024-05-28T06:12:28.858826+0000","last_unstale":"2024-05-28T06:12:28.865415+0000","last_undegraded":"2024-05-28T06:12:28.865415+0000","last_fullsized":"2024-05-28T06:12:28.865415+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:51:51.219749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.863015+0000","last_change":"2024-05-28T06:12:28.856581+0000","last_active":"2024-05-28T06:12:28.863015+0000","last_peered":"2024-05-28T06:12:28.863015+0000","last_clean":"2024-05-28T06:12:28.863015+0000","last_became_active":"2024-05-28T06:12:28.854786+0000","last_became_peered":"2024-05-28T06:12:28.854786+0000","last_unstale":"2024-05-28T06:12:28.863015+0000","last_undegraded":"2024-05-28T06:12:28.863015+0000","last_fullsized":"2024-05-28T06:12:28.863015+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:33:27.536298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.866321+0000","last_change":"2024-05-28T06:12:28.858459+0000","last_active":"2024-05-28T06:12:28.866321+0000","last_peered":"2024-05-28T06:12:28.866321+0000","last_clean":"2024-05-28T06:12:28.866321+0000","last_became_active":"2024-05-28T06:12:28.857641+0000","last_became_peered":"2024-05-28T06:12:28.857641+0000","last_unstale":"2024-05-28T06:12:28.866321+0000","last_undegraded":"2024-05-28T06:12:28.866321+0000","last_fullsized":"2024-05-28T06:12:28.866321+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:05:42.523486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.871085+0000","last_change":"2024-05-28T06:12:28.865662+0000","last_active":"2024-05-28T06:12:28.871085+0000","last_peered":"2024-05-28T06:12:28.871085+0000","last_clean":"2024-05-28T06:12:28.871085+0000","last_became_active":"2024-05-28T06:12:28.865366+0000","last_became_peered":"2024-05-28T06:12:28.865366+0000","last_unstale":"2024-05-28T06:12:28.871085+0000","last_undegraded":"2024-05-28T06:12:28.871085+0000","last_fullsized":"2024-05-28T06:12:28.871085+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:24:16.428796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"59'4","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.876516+0000","last_change":"2024-05-28T06:12:27.835309+0000","last_active":"2024-05-28T06:12:28.876516+0000","last_peered":"2024-05-28T06:12:28.876516+0000","last_clean":"2024-05-28T06:12:28.876516+0000","last_became_active":"2024-05-28T06:12:27.834961+0000","last_became_peered":"2024-05-28T06:12:27.834961+0000","last_unstale":"2024-05-28T06:12:28.876516+0000","last_undegraded":"2024-05-28T06:12:28.876516+0000","last_fullsized":"2024-05-28T06:12:28.876516+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_clean_scrub_stamp":"2024-05-28T06:12:26.773418+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:22:57.630551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.865526+0000","last_change":"2024-05-28T06:12:28.863719+0000","last_active":"2024-05-28T06:12:28.865526+0000","last_peered":"2024-05-28T06:12:28.865526+0000","last_clean":"2024-05-28T06:12:28.865526+0000","last_became_active":"2024-05-28T06:12:28.863356+0000","last_became_peered":"2024-05-28T06:12:28.863356+0000","last_unstale":"2024-05-28T06:12:28.865526+0000","last_undegraded":"2024-05-28T06:12:28.865526+0000","last_fullsized":"2024-05-28T06:12:28.865526+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:55:33.524348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"59'1","reported_seq":13,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.873864+0000","last_change":"2024-05-28T06:12:27.829116+0000","last_active":"2024-05-28T06:12:28.873864+0000","last_peered":"2024-05-28T06:12:28.873864+0000","last_clean":"2024-05-28T06:12:28.873864+0000","last_became_active":"2024-05-28T06:12:27.828658+0000","last_became_peered":"2024-05-28T06:12:27.828658+0000","last_unstale":"2024-05-28T06:12:28.873864+0000","last_undegraded":"2024-05-28T06:12:28.873864+0000","last_fullsized":"2024-05-28T06:12:28.873864+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_clean_scrub_stamp":"2024-05-28T06:12:26.773418+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:57:12.885732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.857626+0000","last_change":"2024-05-28T06:12:28.844525+0000","last_active":"2024-05-28T06:12:28.857626+0000","last_peered":"2024-05-28T06:12:28.857626+0000","last_clean":"2024-05-28T06:12:28.857626+0000","last_became_active":"2024-05-28T06:12:28.844215+0000","last_became_peered":"2024-05-28T06:12:28.844215+0000","last_unstale":"2024-05-28T06:12:28.857626+0000","last_undegraded":"2024-05-28T06:12:28.857626+0000","last_fullsized":"2024-05-28T06:12:28.857626+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:51:51.219749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"59'2","reported_seq":14,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.875673+0000","last_change":"2024-05-28T06:12:27.823594+0000","last_active":"2024-05-28T06:12:28.875673+0000","last_peered":"2024-05-28T06:12:28.875673+0000","last_clean":"2024-05-28T06:12:28.875673+0000","last_became_active":"2024-05-28T06:12:27.823218+0000","last_became_peered":"2024-05-28T06:12:27.823218+0000","last_unstale":"2024-05-28T06:12:28.875673+0000","last_undegraded":"2024-05-28T06:12:28.875673+0000","last_fullsized":"2024-05-28T06:12:28.875673+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_clean_scrub_stamp":"2024-05-28T06:12:26.773418+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:59:33.076306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.866120+0000","last_change":"2024-05-28T06:12:28.861071+0000","last_active":"2024-05-28T06:12:28.866120+0000","last_peered":"2024-05-28T06:12:28.866120+0000","last_clean":"2024-05-28T06:12:28.866120+0000","last_became_active":"2024-05-28T06:12:28.860757+0000","last_became_peered":"2024-05-28T06:12:28.860757+0000","last_unstale":"2024-05-28T06:12:28.866120+0000","last_undegraded":"2024-05-28T06:12:28.866120+0000","last_fullsized":"2024-05-28T06:12:28.866120+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:40:00.025391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"59'2","reported_seq":14,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.869319+0000","last_change":"2024-05-28T06:12:27.835180+0000","last_active":"2024-05-28T06:12:28.869319+0000","last_peered":"2024-05-28T06:12:28.869319+0000","last_clean":"2024-05-28T06:12:28.869319+0000","last_became_active":"2024-05-28T06:12:27.834843+0000","last_became_peered":"2024-05-28T06:12:27.834843+0000","last_unstale":"2024-05-28T06:12:28.869319+0000","last_undegraded":"2024-05-28T06:12:28.869319+0000","last_fullsized":"2024-05-28T06:12:28.869319+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_clean_scrub_stamp":"2024-05-28T06:12:26.773418+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:50:55.880407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.861886+0000","last_change":"2024-05-28T06:12:28.856326+0000","last_active":"2024-05-28T06:12:28.861886+0000","last_peered":"2024-05-28T06:12:28.861886+0000","last_clean":"2024-05-28T06:12:28.861886+0000","last_became_active":"2024-05-28T06:12:28.853829+0000","last_became_peered":"2024-05-28T06:12:28.853829+0000","last_unstale":"2024-05-28T06:12:28.861886+0000","last_undegraded":"2024-05-28T06:12:28.861886+0000","last_fullsized":"2024-05-28T06:12:28.861886+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:41:29.505510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"59'1","reported_seq":13,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.868362+0000","last_change":"2024-05-28T06:12:27.830954+0000","last_active":"2024-05-28T06:12:28.868362+0000","last_peered":"2024-05-28T06:12:28.868362+0000","last_clean":"2024-05-28T06:12:28.868362+0000","last_became_active":"2024-05-28T06:12:27.827069+0000","last_became_peered":"2024-05-28T06:12:27.827069+0000","last_unstale":"2024-05-28T06:12:28.868362+0000","last_undegraded":"2024-05-28T06:12:28.868362+0000","last_fullsized":"2024-05-28T06:12:28.868362+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_clean_scrub_stamp":"2024-05-28T06:12:26.773418+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:57:28.294907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.872060+0000","last_change":"2024-05-28T06:12:28.869123+0000","last_active":"2024-05-28T06:12:28.872060+0000","last_peered":"2024-05-28T06:12:28.872060+0000","last_clean":"2024-05-28T06:12:28.872060+0000","last_became_active":"2024-05-28T06:12:28.868733+0000","last_became_peered":"2024-05-28T06:12:28.868733+0000","last_unstale":"2024-05-28T06:12:28.872060+0000","last_undegraded":"2024-05-28T06:12:28.872060+0000","last_fullsized":"2024-05-28T06:12:28.872060+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:25:05.575741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"59'1","reported_seq":13,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.875677+0000","last_change":"2024-05-28T06:12:27.820911+0000","last_active":"2024-05-28T06:12:28.875677+0000","last_peered":"2024-05-28T06:12:28.875677+0000","last_clean":"2024-05-28T06:12:28.875677+0000","last_became_active":"2024-05-28T06:12:27.820643+0000","last_became_peered":"2024-05-28T06:12:27.820643+0000","last_unstale":"2024-05-28T06:12:28.875677+0000","last_undegraded":"2024-05-28T06:12:28.875677+0000","last_fullsized":"2024-05-28T06:12:28.875677+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_clean_scrub_stamp":"2024-05-28T06:12:26.773418+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:50:55.880407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.866623+0000","last_change":"2024-05-28T06:12:28.865742+0000","last_active":"2024-05-28T06:12:28.866623+0000","last_peered":"2024-05-28T06:12:28.866623+0000","last_clean":"2024-05-28T06:12:28.866623+0000","last_became_active":"2024-05-28T06:12:28.864073+0000","last_became_peered":"2024-05-28T06:12:28.864073+0000","last_unstale":"2024-05-28T06:12:28.866623+0000","last_undegraded":"2024-05-28T06:12:28.866623+0000","last_fullsized":"2024-05-28T06:12:28.866623+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:55:33.524348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.853080+0000","last_change":"2024-05-28T06:12:27.835577+0000","last_active":"2024-05-28T06:12:28.853080+0000","last_peered":"2024-05-28T06:12:28.853080+0000","last_clean":"2024-05-28T06:12:28.853080+0000","last_became_active":"2024-05-28T06:12:27.834658+0000","last_became_peered":"2024-05-28T06:12:27.834658+0000","last_unstale":"2024-05-28T06:12:28.853080+0000","last_undegraded":"2024-05-28T06:12:28.853080+0000","last_fullsized":"2024-05-28T06:12:28.853080+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_clean_scrub_stamp":"2024-05-28T06:12:26.773418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:15:04.022533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.860642+0000","last_change":"2024-05-28T06:12:28.856402+0000","last_active":"2024-05-28T06:12:28.860642+0000","last_peered":"2024-05-28T06:12:28.860642+0000","last_clean":"2024-05-28T06:12:28.860642+0000","last_became_active":"2024-05-28T06:12:28.856117+0000","last_became_peered":"2024-05-28T06:12:28.856117+0000","last_unstale":"2024-05-28T06:12:28.860642+0000","last_undegraded":"2024-05-28T06:12:28.860642+0000","last_fullsized":"2024-05-28T06:12:28.860642+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:19:04.809008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.828455+0000","last_change":"2024-05-28T06:12:27.818865+0000","last_active":"2024-05-28T06:12:28.828455+0000","last_peered":"2024-05-28T06:12:28.828455+0000","last_clean":"2024-05-28T06:12:28.828455+0000","last_became_active":"2024-05-28T06:12:27.818068+0000","last_became_peered":"2024-05-28T06:12:27.818068+0000","last_unstale":"2024-05-28T06:12:28.828455+0000","last_undegraded":"2024-05-28T06:12:28.828455+0000","last_fullsized":"2024-05-28T06:12:28.828455+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_clean_scrub_stamp":"2024-05-28T06:12:26.773418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:13:56.253537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.856342+0000","last_change":"2024-05-28T06:12:28.850877+0000","last_active":"2024-05-28T06:12:28.856342+0000","last_peered":"2024-05-28T06:12:28.856342+0000","last_clean":"2024-05-28T06:12:28.856342+0000","last_became_active":"2024-05-28T06:12:28.850573+0000","last_became_peered":"2024-05-28T06:12:28.850573+0000","last_unstale":"2024-05-28T06:12:28.856342+0000","last_undegraded":"2024-05-28T06:12:28.856342+0000","last_fullsized":"2024-05-28T06:12:28.856342+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:40:15.551469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.844564+0000","last_change":"2024-05-28T06:12:27.831952+0000","last_active":"2024-05-28T06:12:28.844564+0000","last_peered":"2024-05-28T06:12:28.844564+0000","last_clean":"2024-05-28T06:12:28.844564+0000","last_became_active":"2024-05-28T06:12:27.831636+0000","last_became_peered":"2024-05-28T06:12:27.831636+0000","last_unstale":"2024-05-28T06:12:28.844564+0000","last_undegraded":"2024-05-28T06:12:28.844564+0000","last_fullsized":"2024-05-28T06:12:28.844564+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_clean_scrub_stamp":"2024-05-28T06:12:26.773418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:35:23.006469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.861288+0000","last_change":"2024-05-28T06:12:28.855376+0000","last_active":"2024-05-28T06:12:28.861288+0000","last_peered":"2024-05-28T06:12:28.861288+0000","last_clean":"2024-05-28T06:12:28.861288+0000","last_became_active":"2024-05-28T06:12:28.855037+0000","last_became_peered":"2024-05-28T06:12:28.855037+0000","last_unstale":"2024-05-28T06:12:28.861288+0000","last_undegraded":"2024-05-28T06:12:28.861288+0000","last_fullsized":"2024-05-28T06:12:28.861288+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:24:41.895529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.844538+0000","last_change":"2024-05-28T06:12:27.828220+0000","last_active":"2024-05-28T06:12:28.844538+0000","last_peered":"2024-05-28T06:12:28.844538+0000","last_clean":"2024-05-28T06:12:28.844538+0000","last_became_active":"2024-05-28T06:12:27.827226+0000","last_became_peered":"2024-05-28T06:12:27.827226+0000","last_unstale":"2024-05-28T06:12:28.844538+0000","last_undegraded":"2024-05-28T06:12:28.844538+0000","last_fullsized":"2024-05-28T06:12:28.844538+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_clean_scrub_stamp":"2024-05-28T06:12:26.773418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:05:41.509304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.864711+0000","last_change":"2024-05-28T06:12:28.858882+0000","last_active":"2024-05-28T06:12:28.864711+0000","last_peered":"2024-05-28T06:12:28.864711+0000","last_clean":"2024-05-28T06:12:28.864711+0000","last_became_active":"2024-05-28T06:12:28.858566+0000","last_became_peered":"2024-05-28T06:12:28.858566+0000","last_unstale":"2024-05-28T06:12:28.864711+0000","last_undegraded":"2024-05-28T06:12:28.864711+0000","last_fullsized":"2024-05-28T06:12:28.864711+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:46:45.289505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"59'3","reported_seq":15,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.872401+0000","last_change":"2024-05-28T06:12:27.832677+0000","last_active":"2024-05-28T06:12:28.872401+0000","last_peered":"2024-05-28T06:12:28.872401+0000","last_clean":"2024-05-28T06:12:28.872401+0000","last_became_active":"2024-05-28T06:12:27.832305+0000","last_became_peered":"2024-05-28T06:12:27.832305+0000","last_unstale":"2024-05-28T06:12:28.872401+0000","last_undegraded":"2024-05-28T06:12:28.872401+0000","last_fullsized":"2024-05-28T06:12:28.872401+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_clean_scrub_stamp":"2024-05-28T06:12:26.773418+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:50:55.880407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.867338+0000","last_change":"2024-05-28T06:12:28.860396+0000","last_active":"2024-05-28T06:12:28.867338+0000","last_peered":"2024-05-28T06:12:28.867338+0000","last_clean":"2024-05-28T06:12:28.867338+0000","last_became_active":"2024-05-28T06:12:28.860074+0000","last_became_peered":"2024-05-28T06:12:28.860074+0000","last_unstale":"2024-05-28T06:12:28.867338+0000","last_undegraded":"2024-05-28T06:12:28.867338+0000","last_fullsized":"2024-05-28T06:12:28.867338+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:12:29.534159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"59'6","reported_seq":18,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.876723+0000","last_change":"2024-05-28T06:12:27.824327+0000","last_active":"2024-05-28T06:12:28.876723+0000","last_peered":"2024-05-28T06:12:28.876723+0000","last_clean":"2024-05-28T06:12:28.876723+0000","last_became_active":"2024-05-28T06:12:27.824025+0000","last_became_peered":"2024-05-28T06:12:27.824025+0000","last_unstale":"2024-05-28T06:12:28.876723+0000","last_undegraded":"2024-05-28T06:12:28.876723+0000","last_fullsized":"2024-05-28T06:12:28.876723+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_clean_scrub_stamp":"2024-05-28T06:12:26.773418+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:34:56.002235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.856382+0000","last_change":"2024-05-28T06:12:28.851495+0000","last_active":"2024-05-28T06:12:28.856382+0000","last_peered":"2024-05-28T06:12:28.856382+0000","last_clean":"2024-05-28T06:12:28.856382+0000","last_became_active":"2024-05-28T06:12:28.851212+0000","last_became_peered":"2024-05-28T06:12:28.851212+0000","last_unstale":"2024-05-28T06:12:28.856382+0000","last_undegraded":"2024-05-28T06:12:28.856382+0000","last_fullsized":"2024-05-28T06:12:28.856382+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:35:24.020651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"59'2","reported_seq":14,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.873607+0000","last_change":"2024-05-28T06:12:27.828771+0000","last_active":"2024-05-28T06:12:28.873607+0000","last_peered":"2024-05-28T06:12:28.873607+0000","last_clean":"2024-05-28T06:12:28.873607+0000","last_became_active":"2024-05-28T06:12:27.828430+0000","last_became_peered":"2024-05-28T06:12:27.828430+0000","last_unstale":"2024-05-28T06:12:28.873607+0000","last_undegraded":"2024-05-28T06:12:28.873607+0000","last_fullsized":"2024-05-28T06:12:28.873607+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_clean_scrub_stamp":"2024-05-28T06:12:26.773418+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:34:43.090506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.861960+0000","last_change":"2024-05-28T06:12:28.857191+0000","last_active":"2024-05-28T06:12:28.861960+0000","last_peered":"2024-05-28T06:12:28.861960+0000","last_clean":"2024-05-28T06:12:28.861960+0000","last_became_active":"2024-05-28T06:12:28.856922+0000","last_became_peered":"2024-05-28T06:12:28.856922+0000","last_unstale":"2024-05-28T06:12:28.861960+0000","last_undegraded":"2024-05-28T06:12:28.861960+0000","last_fullsized":"2024-05-28T06:12:28.861960+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:06:35.999288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.842322+0000","last_change":"2024-05-28T06:12:27.830088+0000","last_active":"2024-05-28T06:12:28.842322+0000","last_peered":"2024-05-28T06:12:28.842322+0000","last_clean":"2024-05-28T06:12:28.842322+0000","last_became_active":"2024-05-28T06:12:27.829762+0000","last_became_peered":"2024-05-28T06:12:27.829762+0000","last_unstale":"2024-05-28T06:12:28.842322+0000","last_undegraded":"2024-05-28T06:12:28.842322+0000","last_fullsized":"2024-05-28T06:12:28.842322+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_clean_scrub_stamp":"2024-05-28T06:12:26.773418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:22:53.009267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.856211+0000","last_change":"2024-05-28T06:12:28.851575+0000","last_active":"2024-05-28T06:12:28.856211+0000","last_peered":"2024-05-28T06:12:28.856211+0000","last_clean":"2024-05-28T06:12:28.856211+0000","last_became_active":"2024-05-28T06:12:28.851196+0000","last_became_peered":"2024-05-28T06:12:28.851196+0000","last_unstale":"2024-05-28T06:12:28.856211+0000","last_undegraded":"2024-05-28T06:12:28.856211+0000","last_fullsized":"2024-05-28T06:12:28.856211+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:02:46.500269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.850295+0000","last_change":"2024-05-28T06:12:27.837113+0000","last_active":"2024-05-28T06:12:28.850295+0000","last_peered":"2024-05-28T06:12:28.850295+0000","last_clean":"2024-05-28T06:12:28.850295+0000","last_became_active":"2024-05-28T06:12:27.835015+0000","last_became_peered":"2024-05-28T06:12:27.835015+0000","last_unstale":"2024-05-28T06:12:28.850295+0000","last_undegraded":"2024-05-28T06:12:28.850295+0000","last_fullsized":"2024-05-28T06:12:28.850295+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_clean_scrub_stamp":"2024-05-28T06:12:26.773418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:05:04.548469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.867206+0000","last_change":"2024-05-28T06:12:28.861743+0000","last_active":"2024-05-28T06:12:28.867206+0000","last_peered":"2024-05-28T06:12:28.867206+0000","last_clean":"2024-05-28T06:12:28.867206+0000","last_became_active":"2024-05-28T06:12:28.861385+0000","last_became_peered":"2024-05-28T06:12:28.861385+0000","last_unstale":"2024-05-28T06:12:28.867206+0000","last_undegraded":"2024-05-28T06:12:28.867206+0000","last_fullsized":"2024-05-28T06:12:28.867206+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:36:53.762642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"59'8","reported_seq":20,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.873474+0000","last_change":"2024-05-28T06:12:27.836110+0000","last_active":"2024-05-28T06:12:28.873474+0000","last_peered":"2024-05-28T06:12:28.873474+0000","last_clean":"2024-05-28T06:12:28.873474+0000","last_became_active":"2024-05-28T06:12:27.835369+0000","last_became_peered":"2024-05-28T06:12:27.835369+0000","last_unstale":"2024-05-28T06:12:28.873474+0000","last_undegraded":"2024-05-28T06:12:28.873474+0000","last_fullsized":"2024-05-28T06:12:28.873474+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_clean_scrub_stamp":"2024-05-28T06:12:26.773418+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:05:04.548469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.864813+0000","last_change":"2024-05-28T06:12:28.860521+0000","last_active":"2024-05-28T06:12:28.864813+0000","last_peered":"2024-05-28T06:12:28.864813+0000","last_clean":"2024-05-28T06:12:28.864813+0000","last_became_active":"2024-05-28T06:12:28.860205+0000","last_became_peered":"2024-05-28T06:12:28.860205+0000","last_unstale":"2024-05-28T06:12:28.864813+0000","last_undegraded":"2024-05-28T06:12:28.864813+0000","last_fullsized":"2024-05-28T06:12:28.864813+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:57:13.899914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"59'1","reported_seq":13,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.873941+0000","last_change":"2024-05-28T06:12:27.828997+0000","last_active":"2024-05-28T06:12:28.873941+0000","last_peered":"2024-05-28T06:12:28.873941+0000","last_clean":"2024-05-28T06:12:28.873941+0000","last_became_active":"2024-05-28T06:12:27.828503+0000","last_became_peered":"2024-05-28T06:12:27.828503+0000","last_unstale":"2024-05-28T06:12:28.873941+0000","last_undegraded":"2024-05-28T06:12:28.873941+0000","last_fullsized":"2024-05-28T06:12:28.873941+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_clean_scrub_stamp":"2024-05-28T06:12:26.773418+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:36:52.748460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.867879+0000","last_change":"2024-05-28T06:12:28.861541+0000","last_active":"2024-05-28T06:12:28.867879+0000","last_peered":"2024-05-28T06:12:28.867879+0000","last_clean":"2024-05-28T06:12:28.867879+0000","last_became_active":"2024-05-28T06:12:28.859138+0000","last_became_peered":"2024-05-28T06:12:28.859138+0000","last_unstale":"2024-05-28T06:12:28.867879+0000","last_undegraded":"2024-05-28T06:12:28.867879+0000","last_fullsized":"2024-05-28T06:12:28.867879+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:57:13.899914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"59'2","reported_seq":14,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.876015+0000","last_change":"2024-05-28T06:12:27.818803+0000","last_active":"2024-05-28T06:12:28.876015+0000","last_peered":"2024-05-28T06:12:28.876015+0000","last_clean":"2024-05-28T06:12:28.876015+0000","last_became_active":"2024-05-28T06:12:27.818533+0000","last_became_peered":"2024-05-28T06:12:27.818533+0000","last_unstale":"2024-05-28T06:12:28.876015+0000","last_undegraded":"2024-05-28T06:12:28.876015+0000","last_fullsized":"2024-05-28T06:12:28.876015+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_clean_scrub_stamp":"2024-05-28T06:12:26.773418+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:24:40.881347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.860868+0000","last_change":"2024-05-28T06:12:28.856658+0000","last_active":"2024-05-28T06:12:28.860868+0000","last_peered":"2024-05-28T06:12:28.860868+0000","last_clean":"2024-05-28T06:12:28.860868+0000","last_became_active":"2024-05-28T06:12:28.856343+0000","last_became_peered":"2024-05-28T06:12:28.856343+0000","last_unstale":"2024-05-28T06:12:28.860868+0000","last_undegraded":"2024-05-28T06:12:28.860868+0000","last_fullsized":"2024-05-28T06:12:28.860868+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:09:17.950265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"59'1","reported_seq":13,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.869003+0000","last_change":"2024-05-28T06:12:27.836306+0000","last_active":"2024-05-28T06:12:28.869003+0000","last_peered":"2024-05-28T06:12:28.869003+0000","last_clean":"2024-05-28T06:12:28.869003+0000","last_became_active":"2024-05-28T06:12:27.835994+0000","last_became_peered":"2024-05-28T06:12:27.835994+0000","last_unstale":"2024-05-28T06:12:28.869003+0000","last_undegraded":"2024-05-28T06:12:28.869003+0000","last_fullsized":"2024-05-28T06:12:28.869003+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_clean_scrub_stamp":"2024-05-28T06:12:26.773418+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:06:34.985106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.867366+0000","last_change":"2024-05-28T06:12:28.861034+0000","last_active":"2024-05-28T06:12:28.867366+0000","last_peered":"2024-05-28T06:12:28.867366+0000","last_clean":"2024-05-28T06:12:28.867366+0000","last_became_active":"2024-05-28T06:12:28.860706+0000","last_became_peered":"2024-05-28T06:12:28.860706+0000","last_unstale":"2024-05-28T06:12:28.867366+0000","last_undegraded":"2024-05-28T06:12:28.867366+0000","last_fullsized":"2024-05-28T06:12:28.867366+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:24:17.868839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.828182+0000","last_change":"2024-05-28T06:12:27.830795+0000","last_active":"2024-05-28T06:12:28.828182+0000","last_peered":"2024-05-28T06:12:28.828182+0000","last_clean":"2024-05-28T06:12:28.828182+0000","last_became_active":"2024-05-28T06:12:27.830512+0000","last_became_peered":"2024-05-28T06:12:27.830512+0000","last_unstale":"2024-05-28T06:12:28.828182+0000","last_undegraded":"2024-05-28T06:12:28.828182+0000","last_fullsized":"2024-05-28T06:12:28.828182+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_clean_scrub_stamp":"2024-05-28T06:12:26.773418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:06:34.985106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.857570+0000","last_change":"2024-05-28T06:12:28.855265+0000","last_active":"2024-05-28T06:12:28.857570+0000","last_peered":"2024-05-28T06:12:28.857570+0000","last_clean":"2024-05-28T06:12:28.857570+0000","last_became_active":"2024-05-28T06:12:28.852846+0000","last_became_peered":"2024-05-28T06:12:28.852846+0000","last_unstale":"2024-05-28T06:12:28.857570+0000","last_undegraded":"2024-05-28T06:12:28.857570+0000","last_fullsized":"2024-05-28T06:12:28.857570+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:59:34.090488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"59'2","reported_seq":14,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.874352+0000","last_change":"2024-05-28T06:12:27.823106+0000","last_active":"2024-05-28T06:12:28.874352+0000","last_peered":"2024-05-28T06:12:28.874352+0000","last_clean":"2024-05-28T06:12:28.874352+0000","last_became_active":"2024-05-28T06:12:27.822811+0000","last_became_peered":"2024-05-28T06:12:27.822811+0000","last_unstale":"2024-05-28T06:12:28.874352+0000","last_undegraded":"2024-05-28T06:12:28.874352+0000","last_fullsized":"2024-05-28T06:12:28.874352+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_clean_scrub_stamp":"2024-05-28T06:12:26.773418+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:39:59.011209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.866021+0000","last_change":"2024-05-28T06:12:28.860981+0000","last_active":"2024-05-28T06:12:28.866021+0000","last_peered":"2024-05-28T06:12:28.866021+0000","last_clean":"2024-05-28T06:12:28.866021+0000","last_became_active":"2024-05-28T06:12:28.860696+0000","last_became_peered":"2024-05-28T06:12:28.860696+0000","last_unstale":"2024-05-28T06:12:28.866021+0000","last_undegraded":"2024-05-28T06:12:28.866021+0000","last_fullsized":"2024-05-28T06:12:28.866021+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:34:44.104688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.837672+0000","last_change":"2024-05-28T06:12:27.821952+0000","last_active":"2024-05-28T06:12:28.837672+0000","last_peered":"2024-05-28T06:12:28.837672+0000","last_clean":"2024-05-28T06:12:28.837672+0000","last_became_active":"2024-05-28T06:12:27.821678+0000","last_became_peered":"2024-05-28T06:12:27.821678+0000","last_unstale":"2024-05-28T06:12:28.837672+0000","last_undegraded":"2024-05-28T06:12:28.837672+0000","last_fullsized":"2024-05-28T06:12:28.837672+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_clean_scrub_stamp":"2024-05-28T06:12:26.773418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:10:06.582863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.871000+0000","last_change":"2024-05-28T06:12:28.865984+0000","last_active":"2024-05-28T06:12:28.871000+0000","last_peered":"2024-05-28T06:12:28.871000+0000","last_clean":"2024-05-28T06:12:28.871000+0000","last_became_active":"2024-05-28T06:12:28.865678+0000","last_became_peered":"2024-05-28T06:12:28.865678+0000","last_unstale":"2024-05-28T06:12:28.871000+0000","last_undegraded":"2024-05-28T06:12:28.871000+0000","last_fullsized":"2024-05-28T06:12:28.871000+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:19:59.499371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.848149+0000","last_change":"2024-05-28T06:12:27.832941+0000","last_active":"2024-05-28T06:12:28.848149+0000","last_peered":"2024-05-28T06:12:28.848149+0000","last_clean":"2024-05-28T06:12:28.848149+0000","last_became_active":"2024-05-28T06:12:27.832625+0000","last_became_peered":"2024-05-28T06:12:27.832625+0000","last_unstale":"2024-05-28T06:12:28.848149+0000","last_undegraded":"2024-05-28T06:12:28.848149+0000","last_fullsized":"2024-05-28T06:12:28.848149+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_clean_scrub_stamp":"2024-05-28T06:12:26.773418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:05:44.190261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.867014+0000","last_change":"2024-05-28T06:12:28.862277+0000","last_active":"2024-05-28T06:12:28.867014+0000","last_peered":"2024-05-28T06:12:28.867014+0000","last_clean":"2024-05-28T06:12:28.867014+0000","last_became_active":"2024-05-28T06:12:28.861497+0000","last_became_peered":"2024-05-28T06:12:28.861497+0000","last_unstale":"2024-05-28T06:12:28.867014+0000","last_undegraded":"2024-05-28T06:12:28.867014+0000","last_fullsized":"2024-05-28T06:12:28.867014+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:24:17.868839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"59'2","reported_seq":14,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.871256+0000","last_change":"2024-05-28T06:12:27.819445+0000","last_active":"2024-05-28T06:12:28.871256+0000","last_peered":"2024-05-28T06:12:28.871256+0000","last_clean":"2024-05-28T06:12:28.871256+0000","last_became_active":"2024-05-28T06:12:27.819086+0000","last_became_peered":"2024-05-28T06:12:27.819086+0000","last_unstale":"2024-05-28T06:12:28.871256+0000","last_undegraded":"2024-05-28T06:12:28.871256+0000","last_fullsized":"2024-05-28T06:12:28.871256+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_clean_scrub_stamp":"2024-05-28T06:12:26.773418+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:18:20.308309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.868148+0000","last_change":"2024-05-28T06:12:28.865397+0000","last_active":"2024-05-28T06:12:28.868148+0000","last_peered":"2024-05-28T06:12:28.868148+0000","last_clean":"2024-05-28T06:12:28.868148+0000","last_became_active":"2024-05-28T06:12:28.865080+0000","last_became_peered":"2024-05-28T06:12:28.865080+0000","last_unstale":"2024-05-28T06:12:28.868148+0000","last_undegraded":"2024-05-28T06:12:28.868148+0000","last_fullsized":"2024-05-28T06:12:28.868148+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:05:42.523486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.828298+0000","last_change":"2024-05-28T06:12:27.824567+0000","last_active":"2024-05-28T06:12:28.828298+0000","last_peered":"2024-05-28T06:12:28.828298+0000","last_clean":"2024-05-28T06:12:28.828298+0000","last_became_active":"2024-05-28T06:12:27.824298+0000","last_became_peered":"2024-05-28T06:12:27.824298+0000","last_unstale":"2024-05-28T06:12:28.828298+0000","last_undegraded":"2024-05-28T06:12:28.828298+0000","last_fullsized":"2024-05-28T06:12:28.828298+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_clean_scrub_stamp":"2024-05-28T06:12:26.773418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:15:04.022533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.858739+0000","last_change":"2024-05-28T06:12:28.852690+0000","last_active":"2024-05-28T06:12:28.858739+0000","last_peered":"2024-05-28T06:12:28.858739+0000","last_clean":"2024-05-28T06:12:28.858739+0000","last_became_active":"2024-05-28T06:12:28.852423+0000","last_became_peered":"2024-05-28T06:12:28.852423+0000","last_unstale":"2024-05-28T06:12:28.858739+0000","last_undegraded":"2024-05-28T06:12:28.858739+0000","last_fullsized":"2024-05-28T06:12:28.858739+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:52:07.650328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.845921+0000","last_change":"2024-05-28T06:12:27.824162+0000","last_active":"2024-05-28T06:12:28.845921+0000","last_peered":"2024-05-28T06:12:28.845921+0000","last_clean":"2024-05-28T06:12:28.845921+0000","last_became_active":"2024-05-28T06:12:27.823785+0000","last_became_peered":"2024-05-28T06:12:27.823785+0000","last_unstale":"2024-05-28T06:12:28.845921+0000","last_undegraded":"2024-05-28T06:12:28.845921+0000","last_fullsized":"2024-05-28T06:12:28.845921+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_clean_scrub_stamp":"2024-05-28T06:12:26.773418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:46:44.275323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.865378+0000","last_change":"2024-05-28T06:12:28.857754+0000","last_active":"2024-05-28T06:12:28.865378+0000","last_peered":"2024-05-28T06:12:28.865378+0000","last_clean":"2024-05-28T06:12:28.865378+0000","last_became_active":"2024-05-28T06:12:28.851582+0000","last_became_peered":"2024-05-28T06:12:28.851582+0000","last_unstale":"2024-05-28T06:12:28.865378+0000","last_undegraded":"2024-05-28T06:12:28.865378+0000","last_fullsized":"2024-05-28T06:12:28.865378+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:24:17.868839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.852358+0000","last_change":"2024-05-28T06:12:27.833854+0000","last_active":"2024-05-28T06:12:28.852358+0000","last_peered":"2024-05-28T06:12:28.852358+0000","last_clean":"2024-05-28T06:12:28.852358+0000","last_became_active":"2024-05-28T06:12:27.831590+0000","last_became_peered":"2024-05-28T06:12:27.831590+0000","last_unstale":"2024-05-28T06:12:28.852358+0000","last_undegraded":"2024-05-28T06:12:28.852358+0000","last_fullsized":"2024-05-28T06:12:28.852358+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_clean_scrub_stamp":"2024-05-28T06:12:26.773418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:22:57.630551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.867774+0000","last_change":"2024-05-28T06:12:28.859312+0000","last_active":"2024-05-28T06:12:28.867774+0000","last_peered":"2024-05-28T06:12:28.867774+0000","last_clean":"2024-05-28T06:12:28.867774+0000","last_became_active":"2024-05-28T06:12:28.854245+0000","last_became_peered":"2024-05-28T06:12:28.854245+0000","last_unstale":"2024-05-28T06:12:28.867774+0000","last_undegraded":"2024-05-28T06:12:28.867774+0000","last_fullsized":"2024-05-28T06:12:28.867774+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:26:46.284987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.836388+0000","last_change":"2024-05-28T06:12:27.835553+0000","last_active":"2024-05-28T06:12:28.836388+0000","last_peered":"2024-05-28T06:12:28.836388+0000","last_clean":"2024-05-28T06:12:28.836388+0000","last_became_active":"2024-05-28T06:12:27.835062+0000","last_became_peered":"2024-05-28T06:12:27.835062+0000","last_unstale":"2024-05-28T06:12:28.836388+0000","last_undegraded":"2024-05-28T06:12:28.836388+0000","last_fullsized":"2024-05-28T06:12:28.836388+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_clean_scrub_stamp":"2024-05-28T06:12:26.773418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:49:12.956452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.861940+0000","last_change":"2024-05-28T06:12:28.854431+0000","last_active":"2024-05-28T06:12:28.861940+0000","last_peered":"2024-05-28T06:12:28.861940+0000","last_clean":"2024-05-28T06:12:28.861940+0000","last_became_active":"2024-05-28T06:12:28.854153+0000","last_became_peered":"2024-05-28T06:12:28.854153+0000","last_unstale":"2024-05-28T06:12:28.861940+0000","last_undegraded":"2024-05-28T06:12:28.861940+0000","last_fullsized":"2024-05-28T06:12:28.861940+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:22:58.644733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.828541+0000","last_change":"2024-05-28T06:12:27.825489+0000","last_active":"2024-05-28T06:12:28.828541+0000","last_peered":"2024-05-28T06:12:28.828541+0000","last_clean":"2024-05-28T06:12:28.828541+0000","last_became_active":"2024-05-28T06:12:27.825137+0000","last_became_peered":"2024-05-28T06:12:27.825137+0000","last_unstale":"2024-05-28T06:12:28.828541+0000","last_undegraded":"2024-05-28T06:12:28.828541+0000","last_fullsized":"2024-05-28T06:12:28.828541+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_clean_scrub_stamp":"2024-05-28T06:12:26.773418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:11:32.083055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.855546+0000","last_change":"2024-05-28T06:12:28.851925+0000","last_active":"2024-05-28T06:12:28.855546+0000","last_peered":"2024-05-28T06:12:28.855546+0000","last_clean":"2024-05-28T06:12:28.855546+0000","last_became_active":"2024-05-28T06:12:28.851651+0000","last_became_peered":"2024-05-28T06:12:28.851651+0000","last_unstale":"2024-05-28T06:12:28.855546+0000","last_undegraded":"2024-05-28T06:12:28.855546+0000","last_fullsized":"2024-05-28T06:12:28.855546+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:05:40.776948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.838657+0000","last_change":"2024-05-28T06:12:27.818691+0000","last_active":"2024-05-28T06:12:28.838657+0000","last_peered":"2024-05-28T06:12:28.838657+0000","last_clean":"2024-05-28T06:12:28.838657+0000","last_became_active":"2024-05-28T06:12:27.818319+0000","last_became_peered":"2024-05-28T06:12:27.818319+0000","last_unstale":"2024-05-28T06:12:28.838657+0000","last_undegraded":"2024-05-28T06:12:28.838657+0000","last_fullsized":"2024-05-28T06:12:28.838657+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_clean_scrub_stamp":"2024-05-28T06:12:26.773418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:44:53.422507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.860621+0000","last_change":"2024-05-28T06:12:28.855778+0000","last_active":"2024-05-28T06:12:28.860621+0000","last_peered":"2024-05-28T06:12:28.860621+0000","last_clean":"2024-05-28T06:12:28.860621+0000","last_became_active":"2024-05-28T06:12:28.855205+0000","last_became_peered":"2024-05-28T06:12:28.855205+0000","last_unstale":"2024-05-28T06:12:28.860621+0000","last_undegraded":"2024-05-28T06:12:28.860621+0000","last_fullsized":"2024-05-28T06:12:28.860621+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:24:16.428796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.844521+0000","last_change":"2024-05-28T06:12:27.829596+0000","last_active":"2024-05-28T06:12:28.844521+0000","last_peered":"2024-05-28T06:12:28.844521+0000","last_clean":"2024-05-28T06:12:28.844521+0000","last_became_active":"2024-05-28T06:12:27.829368+0000","last_became_peered":"2024-05-28T06:12:27.829368+0000","last_unstale":"2024-05-28T06:12:28.844521+0000","last_undegraded":"2024-05-28T06:12:28.844521+0000","last_fullsized":"2024-05-28T06:12:28.844521+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_clean_scrub_stamp":"2024-05-28T06:12:26.773418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:24:16.854657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.869003+0000","last_change":"2024-05-28T06:12:28.865677+0000","last_active":"2024-05-28T06:12:28.869003+0000","last_peered":"2024-05-28T06:12:28.869003+0000","last_clean":"2024-05-28T06:12:28.869003+0000","last_became_active":"2024-05-28T06:12:28.864490+0000","last_became_peered":"2024-05-28T06:12:28.864490+0000","last_unstale":"2024-05-28T06:12:28.869003+0000","last_undegraded":"2024-05-28T06:12:28.869003+0000","last_fullsized":"2024-05-28T06:12:28.869003+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:22:20.292341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.839986+0000","last_change":"2024-05-28T06:12:27.832907+0000","last_active":"2024-05-28T06:12:28.839986+0000","last_peered":"2024-05-28T06:12:28.839986+0000","last_clean":"2024-05-28T06:12:28.839986+0000","last_became_active":"2024-05-28T06:12:27.832384+0000","last_became_peered":"2024-05-28T06:12:27.832384+0000","last_unstale":"2024-05-28T06:12:28.839986+0000","last_undegraded":"2024-05-28T06:12:28.839986+0000","last_fullsized":"2024-05-28T06:12:28.839986+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_clean_scrub_stamp":"2024-05-28T06:12:26.773418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:52:06.636146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":244,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.838241+0000","last_change":"2024-05-28T06:04:01.039129+0000","last_active":"2024-05-28T06:12:28.838241+0000","last_peered":"2024-05-28T06:12:28.838241+0000","last_clean":"2024-05-28T06:12:28.838241+0000","last_became_active":"2024-05-28T06:04:01.033906+0000","last_became_peered":"2024-05-28T06:04:01.033906+0000","last_unstale":"2024-05-28T06:12:28.838241+0000","last_undegraded":"2024-05-28T06:12:28.838241+0000","last_fullsized":"2024-05-28T06:12:28.838241+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:03:59.864237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:03:59.864237+0000","last_clean_scrub_stamp":"2024-05-28T06:03:59.864237+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:17:36.440245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574953,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46352,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325488,"statfs":{"total":96636764160,"available":96589299712,"internally_reserved":0,"allocated":1310720,"data_stored":888311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":46135754},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574955,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45764,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94326076,"statfs":{"total":96636764160,"available":96589901824,"internally_reserved":0,"allocated":708608,"data_stored":297783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574953,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27728,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344112,"statfs":{"total":96636764160,"available":96608370688,"internally_reserved":0,"allocated":716800,"data_stored":298435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574953,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45772,"kb_used_data":704,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94326068,"statfs":{"total":96636764160,"available":96589893632,"internally_reserved":0,"allocated":720896,"data_stored":299395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574955,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45744,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94326096,"statfs":{"total":96636764160,"available":96589922304,"internally_reserved":0,"allocated":696320,"data_stored":290980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574954,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45776,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94326064,"statfs":{"total":96636764160,"available":96589889536,"internally_reserved":0,"allocated":716800,"data_stored":297839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574954,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49924,"kb_used_data":696,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321916,"statfs":{"total":96636764160,"available":96585641984,"internally_reserved":0,"allocated":712704,"data_stored":297873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574953,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46364,"kb_used_data":1288,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325476,"statfs":{"total":96636764160,"available":96589287424,"internally_reserved":0,"allocated":1318912,"data_stored":890345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-28T06:12:33.759 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-28T06:12:34.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.001+0000 7f908be00640 1 Processor -- start 2024-05-28T06:12:34.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.001+0000 7f908be00640 1 -- start start 2024-05-28T06:12:34.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.003+0000 7f908be00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f908c0d16a0 0x7f908c0cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:34.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.003+0000 7f908be00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f908c0cfce0 con 0x7f908c0cf870 2024-05-28T06:12:34.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.003+0000 7f908be00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f908c15ec50 con 0x7f908c15e870 2024-05-28T06:12:34.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.003+0000 7f908be00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f908c0d2150 con 0x7f908c0d16a0 2024-05-28T06:12:34.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.003+0000 7f908b400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f908c0cf870 0x7f908c0d25e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:44584/0 (socket says 172.21.15.96:44584) 2024-05-28T06:12:34.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.003+0000 7f908b400640 1 -- 172.21.15.96:0/4015690419 learned_addr learned my addr 172.21.15.96:0/4015690419 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:12:34.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.004+0000 7f9082000640 1 --2- 172.21.15.96:0/4015690419 >> v2:172.21.15.123:3300/0 conn(0x7f908c0d16a0 0x7f908c0cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:34.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.004+0000 7f908a000640 1 -- 172.21.15.96:0/4015690419 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 624658154 0 0) 0x7f908c0cfce0 con 0x7f908c0cf870 2024-05-28T06:12:34.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.004+0000 7f908a000640 1 -- 172.21.15.96:0/4015690419 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9070003520 con 0x7f908c0cf870 2024-05-28T06:12:34.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.004+0000 7f908a000640 1 -- 172.21.15.96:0/4015690419 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 499825898 0 0) 0x7f9070003520 con 0x7f908c0cf870 2024-05-28T06:12:34.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.004+0000 7f908a000640 1 -- 172.21.15.96:0/4015690419 >> v2:172.21.15.123:3300/0 conn(0x7f908c0d16a0 msgr2=0x7f908c0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:34.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.004+0000 7f908a000640 1 --2- 172.21.15.96:0/4015690419 >> v2:172.21.15.123:3300/0 conn(0x7f908c0d16a0 0x7f908c0cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:34.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.004+0000 7f908a000640 1 -- 172.21.15.96:0/4015690419 >> v1:172.21.15.123:6789/0 conn(0x7f908c15e870 legacy=0x7f908c0d0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:34.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.004+0000 7f908a000640 1 -- 172.21.15.96:0/4015690419 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f908c0cff80 con 0x7f908c0cf870 2024-05-28T06:12:34.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.004+0000 7f908a000640 1 -- 172.21.15.96:0/4015690419 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f9084001670 con 0x7f908c0cf870 2024-05-28T06:12:34.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.004+0000 7f908a000640 1 -- 172.21.15.96:0/4015690419 >> v1:172.21.15.96:6789/0 conn(0x7f908c0cf870 legacy=0x7f908c0d25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:34.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.004+0000 7f908a000640 1 --2- 172.21.15.96:0/4015690419 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f90700039b0 0x7f9070003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:34.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.004+0000 7f908a000640 1 --2- 172.21.15.96:0/4015690419 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9070004800 0x7f9070004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:34.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.004+0000 7f908a000640 1 --2- 172.21.15.96:0/4015690419 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f90700051c0 0x7f907000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:34.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.004+0000 7f908a000640 1 -- 172.21.15.96:0/4015690419 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f908c0d2150 con 0x7f90700039b0 2024-05-28T06:12:34.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.004+0000 7f908a000640 1 -- 172.21.15.96:0/4015690419 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f907000c060 con 0x7f9070004800 2024-05-28T06:12:34.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.004+0000 7f908a000640 1 -- 172.21.15.96:0/4015690419 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f907000c260 con 0x7f90700051c0 2024-05-28T06:12:34.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.005+0000 7f908aa00640 1 --2- 172.21.15.96:0/4015690419 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f90700039b0 0x7f9070003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:34.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.005+0000 7f908aa00640 1 -- 172.21.15.96:0/4015690419 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f90700051c0 msgr2=0x7f907000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:34.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.005+0000 7f908aa00640 1 --2- 172.21.15.96:0/4015690419 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f90700051c0 0x7f907000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:34.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.005+0000 7f908aa00640 1 -- 172.21.15.96:0/4015690419 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9070004800 msgr2=0x7f9070004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:34.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.005+0000 7f908aa00640 1 --2- 172.21.15.96:0/4015690419 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9070004800 0x7f9070004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:34.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.005+0000 7f908aa00640 1 -- 172.21.15.96:0/4015690419 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9070003520 con 0x7f90700039b0 2024-05-28T06:12:34.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.005+0000 7f908b400640 1 -- 172.21.15.96:0/4015690419 reap_dead start 2024-05-28T06:12:34.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.006+0000 7f908aa00640 1 --2- 172.21.15.96:0/4015690419 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f90700039b0 0x7f9070003dd0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f907c001e50 tx=0x7f9070005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=859fde09eed7a180 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:34.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.006+0000 7f908a000640 1 -- 172.21.15.96:0/4015690419 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f907c001a70 con 0x7f90700039b0 2024-05-28T06:12:34.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.006+0000 7f908a000640 1 -- 172.21.15.96:0/4015690419 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f907c02ae40 con 0x7f90700039b0 2024-05-28T06:12:34.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.006+0000 7f908a000640 1 -- 172.21.15.96:0/4015690419 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f907c003bb0 con 0x7f90700039b0 2024-05-28T06:12:34.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.007+0000 7f908be00640 1 -- 172.21.15.96:0/4015690419 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f90700039b0 msgr2=0x7f9070003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:34.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.007+0000 7f908be00640 1 --2- 172.21.15.96:0/4015690419 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f90700039b0 0x7f9070003dd0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f907c001e50 tx=0x7f9070005830 comp rx=0 tx=0).stop 2024-05-28T06:12:34.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.007+0000 7f908be00640 1 -- 172.21.15.96:0/4015690419 shutdown_connections 2024-05-28T06:12:34.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.007+0000 7f908be00640 1 --2- 172.21.15.96:0/4015690419 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f90700039b0 0x7f9070003dd0 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:34.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.007+0000 7f908be00640 1 -- 172.21.15.96:0/4015690419 >> 172.21.15.96:0/4015690419 conn(0x7f908c08bfc0 msgr2=0x7f908c0d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:34.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.007+0000 7f908be00640 1 -- 172.21.15.96:0/4015690419 shutdown_connections 2024-05-28T06:12:34.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.007+0000 7f908be00640 1 -- 172.21.15.96:0/4015690419 wait complete. 2024-05-28T06:12:34.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.009+0000 7f908be00640 1 Processor -- start 2024-05-28T06:12:34.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.009+0000 7f908be00640 1 -- start start 2024-05-28T06:12:34.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.010+0000 7f908be00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f90700039b0 0x7f908c1565f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:34.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.010+0000 7f908be00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f908c15cae0 0x7f908c0b7b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:34.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.010+0000 7f908be00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f908c0b9890 0x7f908c0b9d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:34.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.010+0000 7f908be00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f908c0d0820 con 0x7f90700039b0 2024-05-28T06:12:34.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.010+0000 7f908be00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f908c0d0540 con 0x7f908c0b9890 2024-05-28T06:12:34.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.010+0000 7f908be00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f908c0d4fb0 con 0x7f908c15cae0 2024-05-28T06:12:34.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.010+0000 7f908aa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f90700039b0 0x7f908c1565f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:34.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.010+0000 7f908aa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f90700039b0 0x7f908c1565f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44836/0 (socket says 172.21.15.96:44836) 2024-05-28T06:12:34.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.010+0000 7f908aa00640 1 -- 172.21.15.96:0/3376583965 learned_addr learned my addr 172.21.15.96:0/3376583965 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:34.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.010+0000 7f9082000640 1 --2- 172.21.15.96:0/3376583965 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f908c15cae0 0x7f908c0b7b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:34.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.010+0000 7f908aa00640 1 -- 172.21.15.96:0/3376583965 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f908c15cae0 msgr2=0x7f908c0b7b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:34.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.010+0000 7f908aa00640 1 --2- 172.21.15.96:0/3376583965 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f908c15cae0 0x7f908c0b7b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:34.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.010+0000 7f908aa00640 1 -- 172.21.15.96:0/3376583965 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f908c0b9890 msgr2=0x7f908c0b9d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:34.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.010+0000 7f908b400640 1 --2- 172.21.15.96:0/3376583965 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f908c0b9890 0x7f908c0b9d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:34.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.010+0000 7f908aa00640 1 --2- 172.21.15.96:0/3376583965 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f908c0b9890 0x7f908c0b9d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:34.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.010+0000 7f908aa00640 1 -- 172.21.15.96:0/3376583965 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f908c0b83c0 con 0x7f90700039b0 2024-05-28T06:12:34.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.011+0000 7f908aa00640 1 --2- 172.21.15.96:0/3376583965 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f90700039b0 0x7f908c1565f0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f907c005040 tx=0x7f907c0014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:34.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.011+0000 7f9083e00640 1 -- 172.21.15.96:0/3376583965 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f907c007db0 con 0x7f90700039b0 2024-05-28T06:12:34.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.011+0000 7f9083e00640 1 -- 172.21.15.96:0/3376583965 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f907c02ac40 con 0x7f90700039b0 2024-05-28T06:12:34.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.011+0000 7f9083e00640 1 -- 172.21.15.96:0/3376583965 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f907c0050d0 con 0x7f90700039b0 2024-05-28T06:12:34.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.011+0000 7f908be00640 1 -- 172.21.15.96:0/3376583965 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f908c0b86a0 con 0x7f90700039b0 2024-05-28T06:12:34.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.011+0000 7f908be00640 1 -- 172.21.15.96:0/3376583965 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f908c0b80a0 con 0x7f90700039b0 2024-05-28T06:12:34.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.012+0000 7f908be00640 1 -- 172.21.15.96:0/3376583965 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f908c0689e0 con 0x7f90700039b0 2024-05-28T06:12:34.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.013+0000 7f9083e00640 1 -- 172.21.15.96:0/3376583965 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 16) ==== 91754+0+0 (secure 0 0 0) 0x7f907c044020 con 0x7f90700039b0 2024-05-28T06:12:34.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.014+0000 7f9083e00640 1 --2- 172.21.15.96:0/3376583965 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f906806d0d0 0x7f906806f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:34.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.014+0000 7f9083e00640 1 -- 172.21.15.96:0/3376583965 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5737+0+0 (secure 0 0 0) 0x7f907c0acc10 con 0x7f90700039b0 2024-05-28T06:12:34.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.015+0000 7f9082000640 1 --2- 172.21.15.96:0/3376583965 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f906806d0d0 0x7f906806f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:34.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.016+0000 7f9082000640 1 --2- 172.21.15.96:0/3376583965 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f906806d0d0 0x7f906806f590 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f908c06e0b0 tx=0x7f9074008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:34.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.018+0000 7f9083e00640 1 -- 172.21.15.96:0/3376583965 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f907c031e20 con 0x7f90700039b0 2024-05-28T06:12:34.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.219+0000 7f908be00640 1 -- 172.21.15.96:0/3376583965 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f908c0ba500 con 0x7f906806d0d0 2024-05-28T06:12:34.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.241+0000 7f9083e00640 1 -- 172.21.15.96:0/3376583965 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251884 (secure 0 0 0) 0x7f908c0ba500 con 0x7f906806d0d0 2024-05-28T06:12:34.244 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:12:34.247 INFO:teuthology.orchestra.run.smithi096.stderr:dumped all 2024-05-28T06:12:34.249 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.247+0000 7f908be00640 1 -- 172.21.15.96:0/3376583965 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f906806d0d0 msgr2=0x7f906806f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:34.249 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.247+0000 7f908be00640 1 --2- 172.21.15.96:0/3376583965 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f906806d0d0 0x7f906806f590 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f908c06e0b0 tx=0x7f9074008040 comp rx=0 tx=0).stop 2024-05-28T06:12:34.249 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.247+0000 7f908be00640 1 -- 172.21.15.96:0/3376583965 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f90700039b0 msgr2=0x7f908c1565f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:34.249 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.247+0000 7f908be00640 1 --2- 172.21.15.96:0/3376583965 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f90700039b0 0x7f908c1565f0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f907c005040 tx=0x7f907c0014a0 comp rx=0 tx=0).stop 2024-05-28T06:12:34.249 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.247+0000 7f908be00640 1 -- 172.21.15.96:0/3376583965 shutdown_connections 2024-05-28T06:12:34.249 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.247+0000 7f908be00640 1 --2- 172.21.15.96:0/3376583965 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f906806d0d0 0x7f906806f590 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:34.249 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.247+0000 7f908be00640 1 --2- 172.21.15.96:0/3376583965 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f908c0b9890 0x7f908c0b9d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:34.249 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.247+0000 7f908be00640 1 --2- 172.21.15.96:0/3376583965 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f908c15cae0 0x7f908c0b7b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:34.250 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.247+0000 7f908be00640 1 --2- 172.21.15.96:0/3376583965 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f90700039b0 0x7f908c1565f0 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:34.250 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.247+0000 7f908be00640 1 -- 172.21.15.96:0/3376583965 >> 172.21.15.96:0/3376583965 conn(0x7f908c08bfc0 msgr2=0x7f908c15d960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:34.250 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.247+0000 7f908be00640 1 -- 172.21.15.96:0/3376583965 shutdown_connections 2024-05-28T06:12:34.250 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.247+0000 7f908be00640 1 -- 172.21.15.96:0/3376583965 wait complete. 2024-05-28T06:12:34.270 INFO:teuthology.orchestra.run.smithi096.stdout:{"pg_ready":true,"pg_map":{"version":310,"stamp":"2024-05-28T06:12:33.685854+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":83,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":353424,"kb_used_data":6740,"kb_used_omap":12,"kb_used_meta":346547,"kb_avail":754621296,"statfs":{"total":773094113280,"available":772732207104,"internally_reserved":0,"allocated":6901760,"data_stored":3560961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":354864688},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002196"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.860311+0000","last_change":"2024-05-28T06:12:28.853943+0000","last_active":"2024-05-28T06:12:28.860311+0000","last_peered":"2024-05-28T06:12:28.860311+0000","last_clean":"2024-05-28T06:12:28.860311+0000","last_became_active":"2024-05-28T06:12:28.853633+0000","last_became_peered":"2024-05-28T06:12:28.853633+0000","last_unstale":"2024-05-28T06:12:28.860311+0000","last_undegraded":"2024-05-28T06:12:28.860311+0000","last_fullsized":"2024-05-28T06:12:28.860311+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:28:27.360488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.869676+0000","last_change":"2024-05-28T06:12:28.866724+0000","last_active":"2024-05-28T06:12:28.869676+0000","last_peered":"2024-05-28T06:12:28.869676+0000","last_clean":"2024-05-28T06:12:28.869676+0000","last_became_active":"2024-05-28T06:12:28.866427+0000","last_became_peered":"2024-05-28T06:12:28.866427+0000","last_unstale":"2024-05-28T06:12:28.869676+0000","last_undegraded":"2024-05-28T06:12:28.869676+0000","last_fullsized":"2024-05-28T06:12:28.869676+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:37:20.555345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.861314+0000","last_change":"2024-05-28T06:12:28.855620+0000","last_active":"2024-05-28T06:12:28.861314+0000","last_peered":"2024-05-28T06:12:28.861314+0000","last_clean":"2024-05-28T06:12:28.861314+0000","last_became_active":"2024-05-28T06:12:28.854897+0000","last_became_peered":"2024-05-28T06:12:28.854897+0000","last_unstale":"2024-05-28T06:12:28.861314+0000","last_undegraded":"2024-05-28T06:12:28.861314+0000","last_fullsized":"2024-05-28T06:12:28.861314+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:15:05.036715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.856899+0000","last_change":"2024-05-28T06:12:28.852854+0000","last_active":"2024-05-28T06:12:28.856899+0000","last_peered":"2024-05-28T06:12:28.856899+0000","last_clean":"2024-05-28T06:12:28.856899+0000","last_became_active":"2024-05-28T06:12:28.852546+0000","last_became_peered":"2024-05-28T06:12:28.852546+0000","last_unstale":"2024-05-28T06:12:28.856899+0000","last_undegraded":"2024-05-28T06:12:28.856899+0000","last_fullsized":"2024-05-28T06:12:28.856899+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:24:41.895529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.863971+0000","last_change":"2024-05-28T06:12:28.860354+0000","last_active":"2024-05-28T06:12:28.863971+0000","last_peered":"2024-05-28T06:12:28.863971+0000","last_clean":"2024-05-28T06:12:28.863971+0000","last_became_active":"2024-05-28T06:12:28.860078+0000","last_became_peered":"2024-05-28T06:12:28.860078+0000","last_unstale":"2024-05-28T06:12:28.863971+0000","last_undegraded":"2024-05-28T06:12:28.863971+0000","last_fullsized":"2024-05-28T06:12:28.863971+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:17:46.322652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.865243+0000","last_change":"2024-05-28T06:12:28.853205+0000","last_active":"2024-05-28T06:12:28.865243+0000","last_peered":"2024-05-28T06:12:28.865243+0000","last_clean":"2024-05-28T06:12:28.865243+0000","last_became_active":"2024-05-28T06:12:28.852876+0000","last_became_peered":"2024-05-28T06:12:28.852876+0000","last_unstale":"2024-05-28T06:12:28.865243+0000","last_undegraded":"2024-05-28T06:12:28.865243+0000","last_fullsized":"2024-05-28T06:12:28.865243+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:57:13.899914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.856849+0000","last_change":"2024-05-28T06:12:28.853316+0000","last_active":"2024-05-28T06:12:28.856849+0000","last_peered":"2024-05-28T06:12:28.856849+0000","last_clean":"2024-05-28T06:12:28.856849+0000","last_became_active":"2024-05-28T06:12:28.853033+0000","last_became_peered":"2024-05-28T06:12:28.853033+0000","last_unstale":"2024-05-28T06:12:28.856849+0000","last_undegraded":"2024-05-28T06:12:28.856849+0000","last_fullsized":"2024-05-28T06:12:28.856849+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:08:29.323479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.867643+0000","last_change":"2024-05-28T06:12:28.858832+0000","last_active":"2024-05-28T06:12:28.867643+0000","last_peered":"2024-05-28T06:12:28.867643+0000","last_clean":"2024-05-28T06:12:28.867643+0000","last_became_active":"2024-05-28T06:12:28.858538+0000","last_became_peered":"2024-05-28T06:12:28.858538+0000","last_unstale":"2024-05-28T06:12:28.867643+0000","last_undegraded":"2024-05-28T06:12:28.867643+0000","last_fullsized":"2024-05-28T06:12:28.867643+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:32:01.579691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.842424+0000","last_change":"2024-05-28T06:12:28.836889+0000","last_active":"2024-05-28T06:12:28.842424+0000","last_peered":"2024-05-28T06:12:28.842424+0000","last_clean":"2024-05-28T06:12:28.842424+0000","last_became_active":"2024-05-28T06:12:28.836070+0000","last_became_peered":"2024-05-28T06:12:28.836070+0000","last_unstale":"2024-05-28T06:12:28.842424+0000","last_undegraded":"2024-05-28T06:12:28.842424+0000","last_fullsized":"2024-05-28T06:12:28.842424+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:25:24.558283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.862524+0000","last_change":"2024-05-28T06:12:28.857691+0000","last_active":"2024-05-28T06:12:28.862524+0000","last_peered":"2024-05-28T06:12:28.862524+0000","last_clean":"2024-05-28T06:12:28.862524+0000","last_became_active":"2024-05-28T06:12:28.857139+0000","last_became_peered":"2024-05-28T06:12:28.857139+0000","last_unstale":"2024-05-28T06:12:28.862524+0000","last_undegraded":"2024-05-28T06:12:28.862524+0000","last_fullsized":"2024-05-28T06:12:28.862524+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:22:54.023449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.867568+0000","last_change":"2024-05-28T06:12:28.866471+0000","last_active":"2024-05-28T06:12:28.867568+0000","last_peered":"2024-05-28T06:12:28.867568+0000","last_clean":"2024-05-28T06:12:28.867568+0000","last_became_active":"2024-05-28T06:12:28.866196+0000","last_became_peered":"2024-05-28T06:12:28.866196+0000","last_unstale":"2024-05-28T06:12:28.867568+0000","last_undegraded":"2024-05-28T06:12:28.867568+0000","last_fullsized":"2024-05-28T06:12:28.867568+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:02:00.111727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.861018+0000","last_change":"2024-05-28T06:12:28.856465+0000","last_active":"2024-05-28T06:12:28.861018+0000","last_peered":"2024-05-28T06:12:28.861018+0000","last_clean":"2024-05-28T06:12:28.861018+0000","last_became_active":"2024-05-28T06:12:28.856136+0000","last_became_peered":"2024-05-28T06:12:28.856136+0000","last_unstale":"2024-05-28T06:12:28.861018+0000","last_undegraded":"2024-05-28T06:12:28.861018+0000","last_fullsized":"2024-05-28T06:12:28.861018+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:51:51.219749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.870688+0000","last_change":"2024-05-28T06:12:28.866498+0000","last_active":"2024-05-28T06:12:28.870688+0000","last_peered":"2024-05-28T06:12:28.870688+0000","last_clean":"2024-05-28T06:12:28.870688+0000","last_became_active":"2024-05-28T06:12:28.866157+0000","last_became_peered":"2024-05-28T06:12:28.866157+0000","last_unstale":"2024-05-28T06:12:28.870688+0000","last_undegraded":"2024-05-28T06:12:28.870688+0000","last_fullsized":"2024-05-28T06:12:28.870688+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:02:00.111727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.856812+0000","last_change":"2024-05-28T06:12:28.852507+0000","last_active":"2024-05-28T06:12:28.856812+0000","last_peered":"2024-05-28T06:12:28.856812+0000","last_clean":"2024-05-28T06:12:28.856812+0000","last_became_active":"2024-05-28T06:12:28.852207+0000","last_became_peered":"2024-05-28T06:12:28.852207+0000","last_unstale":"2024-05-28T06:12:28.856812+0000","last_undegraded":"2024-05-28T06:12:28.856812+0000","last_fullsized":"2024-05-28T06:12:28.856812+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:23:02.057978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.853522+0000","last_change":"2024-05-28T06:12:28.846121+0000","last_active":"2024-05-28T06:12:28.853522+0000","last_peered":"2024-05-28T06:12:28.853522+0000","last_clean":"2024-05-28T06:12:28.853522+0000","last_became_active":"2024-05-28T06:12:28.845778+0000","last_became_peered":"2024-05-28T06:12:28.845778+0000","last_unstale":"2024-05-28T06:12:28.853522+0000","last_undegraded":"2024-05-28T06:12:28.853522+0000","last_fullsized":"2024-05-28T06:12:28.853522+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:29:56.122322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.866341+0000","last_change":"2024-05-28T06:12:28.860149+0000","last_active":"2024-05-28T06:12:28.866341+0000","last_peered":"2024-05-28T06:12:28.866341+0000","last_clean":"2024-05-28T06:12:28.866341+0000","last_became_active":"2024-05-28T06:12:28.853247+0000","last_became_peered":"2024-05-28T06:12:28.853247+0000","last_unstale":"2024-05-28T06:12:28.866341+0000","last_undegraded":"2024-05-28T06:12:28.866341+0000","last_fullsized":"2024-05-28T06:12:28.866341+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:36:53.762642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.860213+0000","last_change":"2024-05-28T06:12:28.854475+0000","last_active":"2024-05-28T06:12:28.860213+0000","last_peered":"2024-05-28T06:12:28.860213+0000","last_clean":"2024-05-28T06:12:28.860213+0000","last_became_active":"2024-05-28T06:12:28.854140+0000","last_became_peered":"2024-05-28T06:12:28.854140+0000","last_unstale":"2024-05-28T06:12:28.860213+0000","last_undegraded":"2024-05-28T06:12:28.860213+0000","last_fullsized":"2024-05-28T06:12:28.860213+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:19:59.499371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.871610+0000","last_change":"2024-05-28T06:12:28.869033+0000","last_active":"2024-05-28T06:12:28.871610+0000","last_peered":"2024-05-28T06:12:28.871610+0000","last_clean":"2024-05-28T06:12:28.871610+0000","last_became_active":"2024-05-28T06:12:28.868747+0000","last_became_peered":"2024-05-28T06:12:28.868747+0000","last_unstale":"2024-05-28T06:12:28.871610+0000","last_undegraded":"2024-05-28T06:12:28.871610+0000","last_fullsized":"2024-05-28T06:12:28.871610+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:34:44.104688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.861855+0000","last_change":"2024-05-28T06:12:28.855732+0000","last_active":"2024-05-28T06:12:28.861855+0000","last_peered":"2024-05-28T06:12:28.861855+0000","last_clean":"2024-05-28T06:12:28.861855+0000","last_became_active":"2024-05-28T06:12:28.854783+0000","last_became_peered":"2024-05-28T06:12:28.854783+0000","last_unstale":"2024-05-28T06:12:28.861855+0000","last_undegraded":"2024-05-28T06:12:28.861855+0000","last_fullsized":"2024-05-28T06:12:28.861855+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:07:56.984961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.871440+0000","last_change":"2024-05-28T06:12:28.866965+0000","last_active":"2024-05-28T06:12:28.871440+0000","last_peered":"2024-05-28T06:12:28.871440+0000","last_clean":"2024-05-28T06:12:28.871440+0000","last_became_active":"2024-05-28T06:12:28.866713+0000","last_became_peered":"2024-05-28T06:12:28.866713+0000","last_unstale":"2024-05-28T06:12:28.871440+0000","last_undegraded":"2024-05-28T06:12:28.871440+0000","last_fullsized":"2024-05-28T06:12:28.871440+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:27:48.034035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.865386+0000","last_change":"2024-05-28T06:12:28.861724+0000","last_active":"2024-05-28T06:12:28.865386+0000","last_peered":"2024-05-28T06:12:28.865386+0000","last_clean":"2024-05-28T06:12:28.865386+0000","last_became_active":"2024-05-28T06:12:28.861435+0000","last_became_peered":"2024-05-28T06:12:28.861435+0000","last_unstale":"2024-05-28T06:12:28.865386+0000","last_undegraded":"2024-05-28T06:12:28.865386+0000","last_fullsized":"2024-05-28T06:12:28.865386+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:05:42.523486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.868419+0000","last_change":"2024-05-28T06:12:28.861993+0000","last_active":"2024-05-28T06:12:28.868419+0000","last_peered":"2024-05-28T06:12:28.868419+0000","last_clean":"2024-05-28T06:12:28.868419+0000","last_became_active":"2024-05-28T06:12:28.860499+0000","last_became_peered":"2024-05-28T06:12:28.860499+0000","last_unstale":"2024-05-28T06:12:28.868419+0000","last_undegraded":"2024-05-28T06:12:28.868419+0000","last_fullsized":"2024-05-28T06:12:28.868419+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:31:46.498176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.863233+0000","last_change":"2024-05-28T06:12:28.858521+0000","last_active":"2024-05-28T06:12:28.863233+0000","last_peered":"2024-05-28T06:12:28.863233+0000","last_clean":"2024-05-28T06:12:28.863233+0000","last_became_active":"2024-05-28T06:12:28.858275+0000","last_became_peered":"2024-05-28T06:12:28.858275+0000","last_unstale":"2024-05-28T06:12:28.863233+0000","last_undegraded":"2024-05-28T06:12:28.863233+0000","last_fullsized":"2024-05-28T06:12:28.863233+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:59:34.090488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.870810+0000","last_change":"2024-05-28T06:12:28.866588+0000","last_active":"2024-05-28T06:12:28.870810+0000","last_peered":"2024-05-28T06:12:28.870810+0000","last_clean":"2024-05-28T06:12:28.870810+0000","last_became_active":"2024-05-28T06:12:28.866248+0000","last_became_peered":"2024-05-28T06:12:28.866248+0000","last_unstale":"2024-05-28T06:12:28.870810+0000","last_undegraded":"2024-05-28T06:12:28.870810+0000","last_fullsized":"2024-05-28T06:12:28.870810+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:13:57.267719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.862617+0000","last_change":"2024-05-28T06:12:28.859005+0000","last_active":"2024-05-28T06:12:28.862617+0000","last_peered":"2024-05-28T06:12:28.862617+0000","last_clean":"2024-05-28T06:12:28.862617+0000","last_became_active":"2024-05-28T06:12:28.858729+0000","last_became_peered":"2024-05-28T06:12:28.858729+0000","last_unstale":"2024-05-28T06:12:28.862617+0000","last_undegraded":"2024-05-28T06:12:28.862617+0000","last_fullsized":"2024-05-28T06:12:28.862617+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:44:54.600832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.862965+0000","last_change":"2024-05-28T06:12:28.857248+0000","last_active":"2024-05-28T06:12:28.862965+0000","last_peered":"2024-05-28T06:12:28.862965+0000","last_clean":"2024-05-28T06:12:28.862965+0000","last_became_active":"2024-05-28T06:12:28.856644+0000","last_became_peered":"2024-05-28T06:12:28.856644+0000","last_unstale":"2024-05-28T06:12:28.862965+0000","last_undegraded":"2024-05-28T06:12:28.862965+0000","last_fullsized":"2024-05-28T06:12:28.862965+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:12:29.534159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.864476+0000","last_change":"2024-05-28T06:12:28.853169+0000","last_active":"2024-05-28T06:12:28.864476+0000","last_peered":"2024-05-28T06:12:28.864476+0000","last_clean":"2024-05-28T06:12:28.864476+0000","last_became_active":"2024-05-28T06:12:28.852864+0000","last_became_peered":"2024-05-28T06:12:28.852864+0000","last_unstale":"2024-05-28T06:12:28.864476+0000","last_undegraded":"2024-05-28T06:12:28.864476+0000","last_fullsized":"2024-05-28T06:12:28.864476+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:12:29.534159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.863013+0000","last_change":"2024-05-28T06:12:28.858942+0000","last_active":"2024-05-28T06:12:28.863013+0000","last_peered":"2024-05-28T06:12:28.863013+0000","last_clean":"2024-05-28T06:12:28.863013+0000","last_became_active":"2024-05-28T06:12:28.858638+0000","last_became_peered":"2024-05-28T06:12:28.858638+0000","last_unstale":"2024-05-28T06:12:28.863013+0000","last_undegraded":"2024-05-28T06:12:28.863013+0000","last_fullsized":"2024-05-28T06:12:28.863013+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:26:19.248645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.867525+0000","last_change":"2024-05-28T06:12:28.858607+0000","last_active":"2024-05-28T06:12:28.867525+0000","last_peered":"2024-05-28T06:12:28.867525+0000","last_clean":"2024-05-28T06:12:28.867525+0000","last_became_active":"2024-05-28T06:12:28.858045+0000","last_became_peered":"2024-05-28T06:12:28.858045+0000","last_unstale":"2024-05-28T06:12:28.867525+0000","last_undegraded":"2024-05-28T06:12:28.867525+0000","last_fullsized":"2024-05-28T06:12:28.867525+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:03:58.734327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.865415+0000","last_change":"2024-05-28T06:12:28.859109+0000","last_active":"2024-05-28T06:12:28.865415+0000","last_peered":"2024-05-28T06:12:28.865415+0000","last_clean":"2024-05-28T06:12:28.865415+0000","last_became_active":"2024-05-28T06:12:28.858826+0000","last_became_peered":"2024-05-28T06:12:28.858826+0000","last_unstale":"2024-05-28T06:12:28.865415+0000","last_undegraded":"2024-05-28T06:12:28.865415+0000","last_fullsized":"2024-05-28T06:12:28.865415+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:51:51.219749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.863015+0000","last_change":"2024-05-28T06:12:28.856581+0000","last_active":"2024-05-28T06:12:28.863015+0000","last_peered":"2024-05-28T06:12:28.863015+0000","last_clean":"2024-05-28T06:12:28.863015+0000","last_became_active":"2024-05-28T06:12:28.854786+0000","last_became_peered":"2024-05-28T06:12:28.854786+0000","last_unstale":"2024-05-28T06:12:28.863015+0000","last_undegraded":"2024-05-28T06:12:28.863015+0000","last_fullsized":"2024-05-28T06:12:28.863015+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:33:27.536298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.866321+0000","last_change":"2024-05-28T06:12:28.858459+0000","last_active":"2024-05-28T06:12:28.866321+0000","last_peered":"2024-05-28T06:12:28.866321+0000","last_clean":"2024-05-28T06:12:28.866321+0000","last_became_active":"2024-05-28T06:12:28.857641+0000","last_became_peered":"2024-05-28T06:12:28.857641+0000","last_unstale":"2024-05-28T06:12:28.866321+0000","last_undegraded":"2024-05-28T06:12:28.866321+0000","last_fullsized":"2024-05-28T06:12:28.866321+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:05:42.523486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.871085+0000","last_change":"2024-05-28T06:12:28.865662+0000","last_active":"2024-05-28T06:12:28.871085+0000","last_peered":"2024-05-28T06:12:28.871085+0000","last_clean":"2024-05-28T06:12:28.871085+0000","last_became_active":"2024-05-28T06:12:28.865366+0000","last_became_peered":"2024-05-28T06:12:28.865366+0000","last_unstale":"2024-05-28T06:12:28.871085+0000","last_undegraded":"2024-05-28T06:12:28.871085+0000","last_fullsized":"2024-05-28T06:12:28.871085+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:24:16.428796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"59'4","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.876516+0000","last_change":"2024-05-28T06:12:27.835309+0000","last_active":"2024-05-28T06:12:28.876516+0000","last_peered":"2024-05-28T06:12:28.876516+0000","last_clean":"2024-05-28T06:12:28.876516+0000","last_became_active":"2024-05-28T06:12:27.834961+0000","last_became_peered":"2024-05-28T06:12:27.834961+0000","last_unstale":"2024-05-28T06:12:28.876516+0000","last_undegraded":"2024-05-28T06:12:28.876516+0000","last_fullsized":"2024-05-28T06:12:28.876516+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_clean_scrub_stamp":"2024-05-28T06:12:26.773418+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:22:57.630551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.865526+0000","last_change":"2024-05-28T06:12:28.863719+0000","last_active":"2024-05-28T06:12:28.865526+0000","last_peered":"2024-05-28T06:12:28.865526+0000","last_clean":"2024-05-28T06:12:28.865526+0000","last_became_active":"2024-05-28T06:12:28.863356+0000","last_became_peered":"2024-05-28T06:12:28.863356+0000","last_unstale":"2024-05-28T06:12:28.865526+0000","last_undegraded":"2024-05-28T06:12:28.865526+0000","last_fullsized":"2024-05-28T06:12:28.865526+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:55:33.524348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"59'1","reported_seq":13,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.873864+0000","last_change":"2024-05-28T06:12:27.829116+0000","last_active":"2024-05-28T06:12:28.873864+0000","last_peered":"2024-05-28T06:12:28.873864+0000","last_clean":"2024-05-28T06:12:28.873864+0000","last_became_active":"2024-05-28T06:12:27.828658+0000","last_became_peered":"2024-05-28T06:12:27.828658+0000","last_unstale":"2024-05-28T06:12:28.873864+0000","last_undegraded":"2024-05-28T06:12:28.873864+0000","last_fullsized":"2024-05-28T06:12:28.873864+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_clean_scrub_stamp":"2024-05-28T06:12:26.773418+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:57:12.885732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.857626+0000","last_change":"2024-05-28T06:12:28.844525+0000","last_active":"2024-05-28T06:12:28.857626+0000","last_peered":"2024-05-28T06:12:28.857626+0000","last_clean":"2024-05-28T06:12:28.857626+0000","last_became_active":"2024-05-28T06:12:28.844215+0000","last_became_peered":"2024-05-28T06:12:28.844215+0000","last_unstale":"2024-05-28T06:12:28.857626+0000","last_undegraded":"2024-05-28T06:12:28.857626+0000","last_fullsized":"2024-05-28T06:12:28.857626+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:51:51.219749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"59'2","reported_seq":14,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.875673+0000","last_change":"2024-05-28T06:12:27.823594+0000","last_active":"2024-05-28T06:12:28.875673+0000","last_peered":"2024-05-28T06:12:28.875673+0000","last_clean":"2024-05-28T06:12:28.875673+0000","last_became_active":"2024-05-28T06:12:27.823218+0000","last_became_peered":"2024-05-28T06:12:27.823218+0000","last_unstale":"2024-05-28T06:12:28.875673+0000","last_undegraded":"2024-05-28T06:12:28.875673+0000","last_fullsized":"2024-05-28T06:12:28.875673+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_clean_scrub_stamp":"2024-05-28T06:12:26.773418+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:59:33.076306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.866120+0000","last_change":"2024-05-28T06:12:28.861071+0000","last_active":"2024-05-28T06:12:28.866120+0000","last_peered":"2024-05-28T06:12:28.866120+0000","last_clean":"2024-05-28T06:12:28.866120+0000","last_became_active":"2024-05-28T06:12:28.860757+0000","last_became_peered":"2024-05-28T06:12:28.860757+0000","last_unstale":"2024-05-28T06:12:28.866120+0000","last_undegraded":"2024-05-28T06:12:28.866120+0000","last_fullsized":"2024-05-28T06:12:28.866120+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:40:00.025391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"59'2","reported_seq":14,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.869319+0000","last_change":"2024-05-28T06:12:27.835180+0000","last_active":"2024-05-28T06:12:28.869319+0000","last_peered":"2024-05-28T06:12:28.869319+0000","last_clean":"2024-05-28T06:12:28.869319+0000","last_became_active":"2024-05-28T06:12:27.834843+0000","last_became_peered":"2024-05-28T06:12:27.834843+0000","last_unstale":"2024-05-28T06:12:28.869319+0000","last_undegraded":"2024-05-28T06:12:28.869319+0000","last_fullsized":"2024-05-28T06:12:28.869319+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_clean_scrub_stamp":"2024-05-28T06:12:26.773418+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:50:55.880407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.861886+0000","last_change":"2024-05-28T06:12:28.856326+0000","last_active":"2024-05-28T06:12:28.861886+0000","last_peered":"2024-05-28T06:12:28.861886+0000","last_clean":"2024-05-28T06:12:28.861886+0000","last_became_active":"2024-05-28T06:12:28.853829+0000","last_became_peered":"2024-05-28T06:12:28.853829+0000","last_unstale":"2024-05-28T06:12:28.861886+0000","last_undegraded":"2024-05-28T06:12:28.861886+0000","last_fullsized":"2024-05-28T06:12:28.861886+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:41:29.505510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"59'1","reported_seq":13,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.868362+0000","last_change":"2024-05-28T06:12:27.830954+0000","last_active":"2024-05-28T06:12:28.868362+0000","last_peered":"2024-05-28T06:12:28.868362+0000","last_clean":"2024-05-28T06:12:28.868362+0000","last_became_active":"2024-05-28T06:12:27.827069+0000","last_became_peered":"2024-05-28T06:12:27.827069+0000","last_unstale":"2024-05-28T06:12:28.868362+0000","last_undegraded":"2024-05-28T06:12:28.868362+0000","last_fullsized":"2024-05-28T06:12:28.868362+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_clean_scrub_stamp":"2024-05-28T06:12:26.773418+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:57:28.294907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.872060+0000","last_change":"2024-05-28T06:12:28.869123+0000","last_active":"2024-05-28T06:12:28.872060+0000","last_peered":"2024-05-28T06:12:28.872060+0000","last_clean":"2024-05-28T06:12:28.872060+0000","last_became_active":"2024-05-28T06:12:28.868733+0000","last_became_peered":"2024-05-28T06:12:28.868733+0000","last_unstale":"2024-05-28T06:12:28.872060+0000","last_undegraded":"2024-05-28T06:12:28.872060+0000","last_fullsized":"2024-05-28T06:12:28.872060+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:25:05.575741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"59'1","reported_seq":13,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.875677+0000","last_change":"2024-05-28T06:12:27.820911+0000","last_active":"2024-05-28T06:12:28.875677+0000","last_peered":"2024-05-28T06:12:28.875677+0000","last_clean":"2024-05-28T06:12:28.875677+0000","last_became_active":"2024-05-28T06:12:27.820643+0000","last_became_peered":"2024-05-28T06:12:27.820643+0000","last_unstale":"2024-05-28T06:12:28.875677+0000","last_undegraded":"2024-05-28T06:12:28.875677+0000","last_fullsized":"2024-05-28T06:12:28.875677+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_clean_scrub_stamp":"2024-05-28T06:12:26.773418+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:50:55.880407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.866623+0000","last_change":"2024-05-28T06:12:28.865742+0000","last_active":"2024-05-28T06:12:28.866623+0000","last_peered":"2024-05-28T06:12:28.866623+0000","last_clean":"2024-05-28T06:12:28.866623+0000","last_became_active":"2024-05-28T06:12:28.864073+0000","last_became_peered":"2024-05-28T06:12:28.864073+0000","last_unstale":"2024-05-28T06:12:28.866623+0000","last_undegraded":"2024-05-28T06:12:28.866623+0000","last_fullsized":"2024-05-28T06:12:28.866623+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:55:33.524348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.853080+0000","last_change":"2024-05-28T06:12:27.835577+0000","last_active":"2024-05-28T06:12:28.853080+0000","last_peered":"2024-05-28T06:12:28.853080+0000","last_clean":"2024-05-28T06:12:28.853080+0000","last_became_active":"2024-05-28T06:12:27.834658+0000","last_became_peered":"2024-05-28T06:12:27.834658+0000","last_unstale":"2024-05-28T06:12:28.853080+0000","last_undegraded":"2024-05-28T06:12:28.853080+0000","last_fullsized":"2024-05-28T06:12:28.853080+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_clean_scrub_stamp":"2024-05-28T06:12:26.773418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:15:04.022533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.860642+0000","last_change":"2024-05-28T06:12:28.856402+0000","last_active":"2024-05-28T06:12:28.860642+0000","last_peered":"2024-05-28T06:12:28.860642+0000","last_clean":"2024-05-28T06:12:28.860642+0000","last_became_active":"2024-05-28T06:12:28.856117+0000","last_became_peered":"2024-05-28T06:12:28.856117+0000","last_unstale":"2024-05-28T06:12:28.860642+0000","last_undegraded":"2024-05-28T06:12:28.860642+0000","last_fullsized":"2024-05-28T06:12:28.860642+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:19:04.809008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.828455+0000","last_change":"2024-05-28T06:12:27.818865+0000","last_active":"2024-05-28T06:12:28.828455+0000","last_peered":"2024-05-28T06:12:28.828455+0000","last_clean":"2024-05-28T06:12:28.828455+0000","last_became_active":"2024-05-28T06:12:27.818068+0000","last_became_peered":"2024-05-28T06:12:27.818068+0000","last_unstale":"2024-05-28T06:12:28.828455+0000","last_undegraded":"2024-05-28T06:12:28.828455+0000","last_fullsized":"2024-05-28T06:12:28.828455+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_clean_scrub_stamp":"2024-05-28T06:12:26.773418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:13:56.253537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.856342+0000","last_change":"2024-05-28T06:12:28.850877+0000","last_active":"2024-05-28T06:12:28.856342+0000","last_peered":"2024-05-28T06:12:28.856342+0000","last_clean":"2024-05-28T06:12:28.856342+0000","last_became_active":"2024-05-28T06:12:28.850573+0000","last_became_peered":"2024-05-28T06:12:28.850573+0000","last_unstale":"2024-05-28T06:12:28.856342+0000","last_undegraded":"2024-05-28T06:12:28.856342+0000","last_fullsized":"2024-05-28T06:12:28.856342+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:40:15.551469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.844564+0000","last_change":"2024-05-28T06:12:27.831952+0000","last_active":"2024-05-28T06:12:28.844564+0000","last_peered":"2024-05-28T06:12:28.844564+0000","last_clean":"2024-05-28T06:12:28.844564+0000","last_became_active":"2024-05-28T06:12:27.831636+0000","last_became_peered":"2024-05-28T06:12:27.831636+0000","last_unstale":"2024-05-28T06:12:28.844564+0000","last_undegraded":"2024-05-28T06:12:28.844564+0000","last_fullsized":"2024-05-28T06:12:28.844564+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_clean_scrub_stamp":"2024-05-28T06:12:26.773418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:35:23.006469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.861288+0000","last_change":"2024-05-28T06:12:28.855376+0000","last_active":"2024-05-28T06:12:28.861288+0000","last_peered":"2024-05-28T06:12:28.861288+0000","last_clean":"2024-05-28T06:12:28.861288+0000","last_became_active":"2024-05-28T06:12:28.855037+0000","last_became_peered":"2024-05-28T06:12:28.855037+0000","last_unstale":"2024-05-28T06:12:28.861288+0000","last_undegraded":"2024-05-28T06:12:28.861288+0000","last_fullsized":"2024-05-28T06:12:28.861288+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:24:41.895529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.844538+0000","last_change":"2024-05-28T06:12:27.828220+0000","last_active":"2024-05-28T06:12:28.844538+0000","last_peered":"2024-05-28T06:12:28.844538+0000","last_clean":"2024-05-28T06:12:28.844538+0000","last_became_active":"2024-05-28T06:12:27.827226+0000","last_became_peered":"2024-05-28T06:12:27.827226+0000","last_unstale":"2024-05-28T06:12:28.844538+0000","last_undegraded":"2024-05-28T06:12:28.844538+0000","last_fullsized":"2024-05-28T06:12:28.844538+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_clean_scrub_stamp":"2024-05-28T06:12:26.773418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:05:41.509304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.864711+0000","last_change":"2024-05-28T06:12:28.858882+0000","last_active":"2024-05-28T06:12:28.864711+0000","last_peered":"2024-05-28T06:12:28.864711+0000","last_clean":"2024-05-28T06:12:28.864711+0000","last_became_active":"2024-05-28T06:12:28.858566+0000","last_became_peered":"2024-05-28T06:12:28.858566+0000","last_unstale":"2024-05-28T06:12:28.864711+0000","last_undegraded":"2024-05-28T06:12:28.864711+0000","last_fullsized":"2024-05-28T06:12:28.864711+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:46:45.289505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"59'3","reported_seq":15,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.872401+0000","last_change":"2024-05-28T06:12:27.832677+0000","last_active":"2024-05-28T06:12:28.872401+0000","last_peered":"2024-05-28T06:12:28.872401+0000","last_clean":"2024-05-28T06:12:28.872401+0000","last_became_active":"2024-05-28T06:12:27.832305+0000","last_became_peered":"2024-05-28T06:12:27.832305+0000","last_unstale":"2024-05-28T06:12:28.872401+0000","last_undegraded":"2024-05-28T06:12:28.872401+0000","last_fullsized":"2024-05-28T06:12:28.872401+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_clean_scrub_stamp":"2024-05-28T06:12:26.773418+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:50:55.880407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.867338+0000","last_change":"2024-05-28T06:12:28.860396+0000","last_active":"2024-05-28T06:12:28.867338+0000","last_peered":"2024-05-28T06:12:28.867338+0000","last_clean":"2024-05-28T06:12:28.867338+0000","last_became_active":"2024-05-28T06:12:28.860074+0000","last_became_peered":"2024-05-28T06:12:28.860074+0000","last_unstale":"2024-05-28T06:12:28.867338+0000","last_undegraded":"2024-05-28T06:12:28.867338+0000","last_fullsized":"2024-05-28T06:12:28.867338+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:12:29.534159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"59'6","reported_seq":18,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.876723+0000","last_change":"2024-05-28T06:12:27.824327+0000","last_active":"2024-05-28T06:12:28.876723+0000","last_peered":"2024-05-28T06:12:28.876723+0000","last_clean":"2024-05-28T06:12:28.876723+0000","last_became_active":"2024-05-28T06:12:27.824025+0000","last_became_peered":"2024-05-28T06:12:27.824025+0000","last_unstale":"2024-05-28T06:12:28.876723+0000","last_undegraded":"2024-05-28T06:12:28.876723+0000","last_fullsized":"2024-05-28T06:12:28.876723+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_clean_scrub_stamp":"2024-05-28T06:12:26.773418+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:34:56.002235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.856382+0000","last_change":"2024-05-28T06:12:28.851495+0000","last_active":"2024-05-28T06:12:28.856382+0000","last_peered":"2024-05-28T06:12:28.856382+0000","last_clean":"2024-05-28T06:12:28.856382+0000","last_became_active":"2024-05-28T06:12:28.851212+0000","last_became_peered":"2024-05-28T06:12:28.851212+0000","last_unstale":"2024-05-28T06:12:28.856382+0000","last_undegraded":"2024-05-28T06:12:28.856382+0000","last_fullsized":"2024-05-28T06:12:28.856382+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:35:24.020651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"59'2","reported_seq":14,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.873607+0000","last_change":"2024-05-28T06:12:27.828771+0000","last_active":"2024-05-28T06:12:28.873607+0000","last_peered":"2024-05-28T06:12:28.873607+0000","last_clean":"2024-05-28T06:12:28.873607+0000","last_became_active":"2024-05-28T06:12:27.828430+0000","last_became_peered":"2024-05-28T06:12:27.828430+0000","last_unstale":"2024-05-28T06:12:28.873607+0000","last_undegraded":"2024-05-28T06:12:28.873607+0000","last_fullsized":"2024-05-28T06:12:28.873607+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_clean_scrub_stamp":"2024-05-28T06:12:26.773418+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:34:43.090506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.861960+0000","last_change":"2024-05-28T06:12:28.857191+0000","last_active":"2024-05-28T06:12:28.861960+0000","last_peered":"2024-05-28T06:12:28.861960+0000","last_clean":"2024-05-28T06:12:28.861960+0000","last_became_active":"2024-05-28T06:12:28.856922+0000","last_became_peered":"2024-05-28T06:12:28.856922+0000","last_unstale":"2024-05-28T06:12:28.861960+0000","last_undegraded":"2024-05-28T06:12:28.861960+0000","last_fullsized":"2024-05-28T06:12:28.861960+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:06:35.999288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.842322+0000","last_change":"2024-05-28T06:12:27.830088+0000","last_active":"2024-05-28T06:12:28.842322+0000","last_peered":"2024-05-28T06:12:28.842322+0000","last_clean":"2024-05-28T06:12:28.842322+0000","last_became_active":"2024-05-28T06:12:27.829762+0000","last_became_peered":"2024-05-28T06:12:27.829762+0000","last_unstale":"2024-05-28T06:12:28.842322+0000","last_undegraded":"2024-05-28T06:12:28.842322+0000","last_fullsized":"2024-05-28T06:12:28.842322+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_clean_scrub_stamp":"2024-05-28T06:12:26.773418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:22:53.009267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.856211+0000","last_change":"2024-05-28T06:12:28.851575+0000","last_active":"2024-05-28T06:12:28.856211+0000","last_peered":"2024-05-28T06:12:28.856211+0000","last_clean":"2024-05-28T06:12:28.856211+0000","last_became_active":"2024-05-28T06:12:28.851196+0000","last_became_peered":"2024-05-28T06:12:28.851196+0000","last_unstale":"2024-05-28T06:12:28.856211+0000","last_undegraded":"2024-05-28T06:12:28.856211+0000","last_fullsized":"2024-05-28T06:12:28.856211+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:02:46.500269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.850295+0000","last_change":"2024-05-28T06:12:27.837113+0000","last_active":"2024-05-28T06:12:28.850295+0000","last_peered":"2024-05-28T06:12:28.850295+0000","last_clean":"2024-05-28T06:12:28.850295+0000","last_became_active":"2024-05-28T06:12:27.835015+0000","last_became_peered":"2024-05-28T06:12:27.835015+0000","last_unstale":"2024-05-28T06:12:28.850295+0000","last_undegraded":"2024-05-28T06:12:28.850295+0000","last_fullsized":"2024-05-28T06:12:28.850295+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_clean_scrub_stamp":"2024-05-28T06:12:26.773418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:05:04.548469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.867206+0000","last_change":"2024-05-28T06:12:28.861743+0000","last_active":"2024-05-28T06:12:28.867206+0000","last_peered":"2024-05-28T06:12:28.867206+0000","last_clean":"2024-05-28T06:12:28.867206+0000","last_became_active":"2024-05-28T06:12:28.861385+0000","last_became_peered":"2024-05-28T06:12:28.861385+0000","last_unstale":"2024-05-28T06:12:28.867206+0000","last_undegraded":"2024-05-28T06:12:28.867206+0000","last_fullsized":"2024-05-28T06:12:28.867206+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:36:53.762642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"59'8","reported_seq":20,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.873474+0000","last_change":"2024-05-28T06:12:27.836110+0000","last_active":"2024-05-28T06:12:28.873474+0000","last_peered":"2024-05-28T06:12:28.873474+0000","last_clean":"2024-05-28T06:12:28.873474+0000","last_became_active":"2024-05-28T06:12:27.835369+0000","last_became_peered":"2024-05-28T06:12:27.835369+0000","last_unstale":"2024-05-28T06:12:28.873474+0000","last_undegraded":"2024-05-28T06:12:28.873474+0000","last_fullsized":"2024-05-28T06:12:28.873474+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_clean_scrub_stamp":"2024-05-28T06:12:26.773418+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:05:04.548469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.864813+0000","last_change":"2024-05-28T06:12:28.860521+0000","last_active":"2024-05-28T06:12:28.864813+0000","last_peered":"2024-05-28T06:12:28.864813+0000","last_clean":"2024-05-28T06:12:28.864813+0000","last_became_active":"2024-05-28T06:12:28.860205+0000","last_became_peered":"2024-05-28T06:12:28.860205+0000","last_unstale":"2024-05-28T06:12:28.864813+0000","last_undegraded":"2024-05-28T06:12:28.864813+0000","last_fullsized":"2024-05-28T06:12:28.864813+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:57:13.899914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"59'1","reported_seq":13,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.873941+0000","last_change":"2024-05-28T06:12:27.828997+0000","last_active":"2024-05-28T06:12:28.873941+0000","last_peered":"2024-05-28T06:12:28.873941+0000","last_clean":"2024-05-28T06:12:28.873941+0000","last_became_active":"2024-05-28T06:12:27.828503+0000","last_became_peered":"2024-05-28T06:12:27.828503+0000","last_unstale":"2024-05-28T06:12:28.873941+0000","last_undegraded":"2024-05-28T06:12:28.873941+0000","last_fullsized":"2024-05-28T06:12:28.873941+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_clean_scrub_stamp":"2024-05-28T06:12:26.773418+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:36:52.748460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.867879+0000","last_change":"2024-05-28T06:12:28.861541+0000","last_active":"2024-05-28T06:12:28.867879+0000","last_peered":"2024-05-28T06:12:28.867879+0000","last_clean":"2024-05-28T06:12:28.867879+0000","last_became_active":"2024-05-28T06:12:28.859138+0000","last_became_peered":"2024-05-28T06:12:28.859138+0000","last_unstale":"2024-05-28T06:12:28.867879+0000","last_undegraded":"2024-05-28T06:12:28.867879+0000","last_fullsized":"2024-05-28T06:12:28.867879+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:57:13.899914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"59'2","reported_seq":14,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.876015+0000","last_change":"2024-05-28T06:12:27.818803+0000","last_active":"2024-05-28T06:12:28.876015+0000","last_peered":"2024-05-28T06:12:28.876015+0000","last_clean":"2024-05-28T06:12:28.876015+0000","last_became_active":"2024-05-28T06:12:27.818533+0000","last_became_peered":"2024-05-28T06:12:27.818533+0000","last_unstale":"2024-05-28T06:12:28.876015+0000","last_undegraded":"2024-05-28T06:12:28.876015+0000","last_fullsized":"2024-05-28T06:12:28.876015+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_clean_scrub_stamp":"2024-05-28T06:12:26.773418+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:24:40.881347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.860868+0000","last_change":"2024-05-28T06:12:28.856658+0000","last_active":"2024-05-28T06:12:28.860868+0000","last_peered":"2024-05-28T06:12:28.860868+0000","last_clean":"2024-05-28T06:12:28.860868+0000","last_became_active":"2024-05-28T06:12:28.856343+0000","last_became_peered":"2024-05-28T06:12:28.856343+0000","last_unstale":"2024-05-28T06:12:28.860868+0000","last_undegraded":"2024-05-28T06:12:28.860868+0000","last_fullsized":"2024-05-28T06:12:28.860868+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:09:17.950265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"59'1","reported_seq":13,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.869003+0000","last_change":"2024-05-28T06:12:27.836306+0000","last_active":"2024-05-28T06:12:28.869003+0000","last_peered":"2024-05-28T06:12:28.869003+0000","last_clean":"2024-05-28T06:12:28.869003+0000","last_became_active":"2024-05-28T06:12:27.835994+0000","last_became_peered":"2024-05-28T06:12:27.835994+0000","last_unstale":"2024-05-28T06:12:28.869003+0000","last_undegraded":"2024-05-28T06:12:28.869003+0000","last_fullsized":"2024-05-28T06:12:28.869003+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_clean_scrub_stamp":"2024-05-28T06:12:26.773418+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:06:34.985106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.867366+0000","last_change":"2024-05-28T06:12:28.861034+0000","last_active":"2024-05-28T06:12:28.867366+0000","last_peered":"2024-05-28T06:12:28.867366+0000","last_clean":"2024-05-28T06:12:28.867366+0000","last_became_active":"2024-05-28T06:12:28.860706+0000","last_became_peered":"2024-05-28T06:12:28.860706+0000","last_unstale":"2024-05-28T06:12:28.867366+0000","last_undegraded":"2024-05-28T06:12:28.867366+0000","last_fullsized":"2024-05-28T06:12:28.867366+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:24:17.868839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.828182+0000","last_change":"2024-05-28T06:12:27.830795+0000","last_active":"2024-05-28T06:12:28.828182+0000","last_peered":"2024-05-28T06:12:28.828182+0000","last_clean":"2024-05-28T06:12:28.828182+0000","last_became_active":"2024-05-28T06:12:27.830512+0000","last_became_peered":"2024-05-28T06:12:27.830512+0000","last_unstale":"2024-05-28T06:12:28.828182+0000","last_undegraded":"2024-05-28T06:12:28.828182+0000","last_fullsized":"2024-05-28T06:12:28.828182+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_clean_scrub_stamp":"2024-05-28T06:12:26.773418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:06:34.985106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.857570+0000","last_change":"2024-05-28T06:12:28.855265+0000","last_active":"2024-05-28T06:12:28.857570+0000","last_peered":"2024-05-28T06:12:28.857570+0000","last_clean":"2024-05-28T06:12:28.857570+0000","last_became_active":"2024-05-28T06:12:28.852846+0000","last_became_peered":"2024-05-28T06:12:28.852846+0000","last_unstale":"2024-05-28T06:12:28.857570+0000","last_undegraded":"2024-05-28T06:12:28.857570+0000","last_fullsized":"2024-05-28T06:12:28.857570+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:59:34.090488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"59'2","reported_seq":14,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.874352+0000","last_change":"2024-05-28T06:12:27.823106+0000","last_active":"2024-05-28T06:12:28.874352+0000","last_peered":"2024-05-28T06:12:28.874352+0000","last_clean":"2024-05-28T06:12:28.874352+0000","last_became_active":"2024-05-28T06:12:27.822811+0000","last_became_peered":"2024-05-28T06:12:27.822811+0000","last_unstale":"2024-05-28T06:12:28.874352+0000","last_undegraded":"2024-05-28T06:12:28.874352+0000","last_fullsized":"2024-05-28T06:12:28.874352+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_clean_scrub_stamp":"2024-05-28T06:12:26.773418+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:39:59.011209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.866021+0000","last_change":"2024-05-28T06:12:28.860981+0000","last_active":"2024-05-28T06:12:28.866021+0000","last_peered":"2024-05-28T06:12:28.866021+0000","last_clean":"2024-05-28T06:12:28.866021+0000","last_became_active":"2024-05-28T06:12:28.860696+0000","last_became_peered":"2024-05-28T06:12:28.860696+0000","last_unstale":"2024-05-28T06:12:28.866021+0000","last_undegraded":"2024-05-28T06:12:28.866021+0000","last_fullsized":"2024-05-28T06:12:28.866021+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:34:44.104688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.837672+0000","last_change":"2024-05-28T06:12:27.821952+0000","last_active":"2024-05-28T06:12:28.837672+0000","last_peered":"2024-05-28T06:12:28.837672+0000","last_clean":"2024-05-28T06:12:28.837672+0000","last_became_active":"2024-05-28T06:12:27.821678+0000","last_became_peered":"2024-05-28T06:12:27.821678+0000","last_unstale":"2024-05-28T06:12:28.837672+0000","last_undegraded":"2024-05-28T06:12:28.837672+0000","last_fullsized":"2024-05-28T06:12:28.837672+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_clean_scrub_stamp":"2024-05-28T06:12:26.773418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:10:06.582863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.871000+0000","last_change":"2024-05-28T06:12:28.865984+0000","last_active":"2024-05-28T06:12:28.871000+0000","last_peered":"2024-05-28T06:12:28.871000+0000","last_clean":"2024-05-28T06:12:28.871000+0000","last_became_active":"2024-05-28T06:12:28.865678+0000","last_became_peered":"2024-05-28T06:12:28.865678+0000","last_unstale":"2024-05-28T06:12:28.871000+0000","last_undegraded":"2024-05-28T06:12:28.871000+0000","last_fullsized":"2024-05-28T06:12:28.871000+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:19:59.499371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.848149+0000","last_change":"2024-05-28T06:12:27.832941+0000","last_active":"2024-05-28T06:12:28.848149+0000","last_peered":"2024-05-28T06:12:28.848149+0000","last_clean":"2024-05-28T06:12:28.848149+0000","last_became_active":"2024-05-28T06:12:27.832625+0000","last_became_peered":"2024-05-28T06:12:27.832625+0000","last_unstale":"2024-05-28T06:12:28.848149+0000","last_undegraded":"2024-05-28T06:12:28.848149+0000","last_fullsized":"2024-05-28T06:12:28.848149+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_clean_scrub_stamp":"2024-05-28T06:12:26.773418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:05:44.190261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.867014+0000","last_change":"2024-05-28T06:12:28.862277+0000","last_active":"2024-05-28T06:12:28.867014+0000","last_peered":"2024-05-28T06:12:28.867014+0000","last_clean":"2024-05-28T06:12:28.867014+0000","last_became_active":"2024-05-28T06:12:28.861497+0000","last_became_peered":"2024-05-28T06:12:28.861497+0000","last_unstale":"2024-05-28T06:12:28.867014+0000","last_undegraded":"2024-05-28T06:12:28.867014+0000","last_fullsized":"2024-05-28T06:12:28.867014+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:24:17.868839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"59'2","reported_seq":14,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.871256+0000","last_change":"2024-05-28T06:12:27.819445+0000","last_active":"2024-05-28T06:12:28.871256+0000","last_peered":"2024-05-28T06:12:28.871256+0000","last_clean":"2024-05-28T06:12:28.871256+0000","last_became_active":"2024-05-28T06:12:27.819086+0000","last_became_peered":"2024-05-28T06:12:27.819086+0000","last_unstale":"2024-05-28T06:12:28.871256+0000","last_undegraded":"2024-05-28T06:12:28.871256+0000","last_fullsized":"2024-05-28T06:12:28.871256+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_clean_scrub_stamp":"2024-05-28T06:12:26.773418+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:18:20.308309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.868148+0000","last_change":"2024-05-28T06:12:28.865397+0000","last_active":"2024-05-28T06:12:28.868148+0000","last_peered":"2024-05-28T06:12:28.868148+0000","last_clean":"2024-05-28T06:12:28.868148+0000","last_became_active":"2024-05-28T06:12:28.865080+0000","last_became_peered":"2024-05-28T06:12:28.865080+0000","last_unstale":"2024-05-28T06:12:28.868148+0000","last_undegraded":"2024-05-28T06:12:28.868148+0000","last_fullsized":"2024-05-28T06:12:28.868148+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:05:42.523486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.828298+0000","last_change":"2024-05-28T06:12:27.824567+0000","last_active":"2024-05-28T06:12:28.828298+0000","last_peered":"2024-05-28T06:12:28.828298+0000","last_clean":"2024-05-28T06:12:28.828298+0000","last_became_active":"2024-05-28T06:12:27.824298+0000","last_became_peered":"2024-05-28T06:12:27.824298+0000","last_unstale":"2024-05-28T06:12:28.828298+0000","last_undegraded":"2024-05-28T06:12:28.828298+0000","last_fullsized":"2024-05-28T06:12:28.828298+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_clean_scrub_stamp":"2024-05-28T06:12:26.773418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:15:04.022533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.858739+0000","last_change":"2024-05-28T06:12:28.852690+0000","last_active":"2024-05-28T06:12:28.858739+0000","last_peered":"2024-05-28T06:12:28.858739+0000","last_clean":"2024-05-28T06:12:28.858739+0000","last_became_active":"2024-05-28T06:12:28.852423+0000","last_became_peered":"2024-05-28T06:12:28.852423+0000","last_unstale":"2024-05-28T06:12:28.858739+0000","last_undegraded":"2024-05-28T06:12:28.858739+0000","last_fullsized":"2024-05-28T06:12:28.858739+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:52:07.650328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.845921+0000","last_change":"2024-05-28T06:12:27.824162+0000","last_active":"2024-05-28T06:12:28.845921+0000","last_peered":"2024-05-28T06:12:28.845921+0000","last_clean":"2024-05-28T06:12:28.845921+0000","last_became_active":"2024-05-28T06:12:27.823785+0000","last_became_peered":"2024-05-28T06:12:27.823785+0000","last_unstale":"2024-05-28T06:12:28.845921+0000","last_undegraded":"2024-05-28T06:12:28.845921+0000","last_fullsized":"2024-05-28T06:12:28.845921+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_clean_scrub_stamp":"2024-05-28T06:12:26.773418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:46:44.275323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.865378+0000","last_change":"2024-05-28T06:12:28.857754+0000","last_active":"2024-05-28T06:12:28.865378+0000","last_peered":"2024-05-28T06:12:28.865378+0000","last_clean":"2024-05-28T06:12:28.865378+0000","last_became_active":"2024-05-28T06:12:28.851582+0000","last_became_peered":"2024-05-28T06:12:28.851582+0000","last_unstale":"2024-05-28T06:12:28.865378+0000","last_undegraded":"2024-05-28T06:12:28.865378+0000","last_fullsized":"2024-05-28T06:12:28.865378+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:24:17.868839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.852358+0000","last_change":"2024-05-28T06:12:27.833854+0000","last_active":"2024-05-28T06:12:28.852358+0000","last_peered":"2024-05-28T06:12:28.852358+0000","last_clean":"2024-05-28T06:12:28.852358+0000","last_became_active":"2024-05-28T06:12:27.831590+0000","last_became_peered":"2024-05-28T06:12:27.831590+0000","last_unstale":"2024-05-28T06:12:28.852358+0000","last_undegraded":"2024-05-28T06:12:28.852358+0000","last_fullsized":"2024-05-28T06:12:28.852358+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_clean_scrub_stamp":"2024-05-28T06:12:26.773418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:22:57.630551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.867774+0000","last_change":"2024-05-28T06:12:28.859312+0000","last_active":"2024-05-28T06:12:28.867774+0000","last_peered":"2024-05-28T06:12:28.867774+0000","last_clean":"2024-05-28T06:12:28.867774+0000","last_became_active":"2024-05-28T06:12:28.854245+0000","last_became_peered":"2024-05-28T06:12:28.854245+0000","last_unstale":"2024-05-28T06:12:28.867774+0000","last_undegraded":"2024-05-28T06:12:28.867774+0000","last_fullsized":"2024-05-28T06:12:28.867774+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:26:46.284987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.836388+0000","last_change":"2024-05-28T06:12:27.835553+0000","last_active":"2024-05-28T06:12:28.836388+0000","last_peered":"2024-05-28T06:12:28.836388+0000","last_clean":"2024-05-28T06:12:28.836388+0000","last_became_active":"2024-05-28T06:12:27.835062+0000","last_became_peered":"2024-05-28T06:12:27.835062+0000","last_unstale":"2024-05-28T06:12:28.836388+0000","last_undegraded":"2024-05-28T06:12:28.836388+0000","last_fullsized":"2024-05-28T06:12:28.836388+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_clean_scrub_stamp":"2024-05-28T06:12:26.773418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:49:12.956452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.861940+0000","last_change":"2024-05-28T06:12:28.854431+0000","last_active":"2024-05-28T06:12:28.861940+0000","last_peered":"2024-05-28T06:12:28.861940+0000","last_clean":"2024-05-28T06:12:28.861940+0000","last_became_active":"2024-05-28T06:12:28.854153+0000","last_became_peered":"2024-05-28T06:12:28.854153+0000","last_unstale":"2024-05-28T06:12:28.861940+0000","last_undegraded":"2024-05-28T06:12:28.861940+0000","last_fullsized":"2024-05-28T06:12:28.861940+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:22:58.644733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.828541+0000","last_change":"2024-05-28T06:12:27.825489+0000","last_active":"2024-05-28T06:12:28.828541+0000","last_peered":"2024-05-28T06:12:28.828541+0000","last_clean":"2024-05-28T06:12:28.828541+0000","last_became_active":"2024-05-28T06:12:27.825137+0000","last_became_peered":"2024-05-28T06:12:27.825137+0000","last_unstale":"2024-05-28T06:12:28.828541+0000","last_undegraded":"2024-05-28T06:12:28.828541+0000","last_fullsized":"2024-05-28T06:12:28.828541+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_clean_scrub_stamp":"2024-05-28T06:12:26.773418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:11:32.083055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.855546+0000","last_change":"2024-05-28T06:12:28.851925+0000","last_active":"2024-05-28T06:12:28.855546+0000","last_peered":"2024-05-28T06:12:28.855546+0000","last_clean":"2024-05-28T06:12:28.855546+0000","last_became_active":"2024-05-28T06:12:28.851651+0000","last_became_peered":"2024-05-28T06:12:28.851651+0000","last_unstale":"2024-05-28T06:12:28.855546+0000","last_undegraded":"2024-05-28T06:12:28.855546+0000","last_fullsized":"2024-05-28T06:12:28.855546+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:05:40.776948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.838657+0000","last_change":"2024-05-28T06:12:27.818691+0000","last_active":"2024-05-28T06:12:28.838657+0000","last_peered":"2024-05-28T06:12:28.838657+0000","last_clean":"2024-05-28T06:12:28.838657+0000","last_became_active":"2024-05-28T06:12:27.818319+0000","last_became_peered":"2024-05-28T06:12:27.818319+0000","last_unstale":"2024-05-28T06:12:28.838657+0000","last_undegraded":"2024-05-28T06:12:28.838657+0000","last_fullsized":"2024-05-28T06:12:28.838657+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_clean_scrub_stamp":"2024-05-28T06:12:26.773418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:44:53.422507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.860621+0000","last_change":"2024-05-28T06:12:28.855778+0000","last_active":"2024-05-28T06:12:28.860621+0000","last_peered":"2024-05-28T06:12:28.860621+0000","last_clean":"2024-05-28T06:12:28.860621+0000","last_became_active":"2024-05-28T06:12:28.855205+0000","last_became_peered":"2024-05-28T06:12:28.855205+0000","last_unstale":"2024-05-28T06:12:28.860621+0000","last_undegraded":"2024-05-28T06:12:28.860621+0000","last_fullsized":"2024-05-28T06:12:28.860621+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:24:16.428796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.844521+0000","last_change":"2024-05-28T06:12:27.829596+0000","last_active":"2024-05-28T06:12:28.844521+0000","last_peered":"2024-05-28T06:12:28.844521+0000","last_clean":"2024-05-28T06:12:28.844521+0000","last_became_active":"2024-05-28T06:12:27.829368+0000","last_became_peered":"2024-05-28T06:12:27.829368+0000","last_unstale":"2024-05-28T06:12:28.844521+0000","last_undegraded":"2024-05-28T06:12:28.844521+0000","last_fullsized":"2024-05-28T06:12:28.844521+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_clean_scrub_stamp":"2024-05-28T06:12:26.773418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:24:16.854657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.869003+0000","last_change":"2024-05-28T06:12:28.865677+0000","last_active":"2024-05-28T06:12:28.869003+0000","last_peered":"2024-05-28T06:12:28.869003+0000","last_clean":"2024-05-28T06:12:28.869003+0000","last_became_active":"2024-05-28T06:12:28.864490+0000","last_became_peered":"2024-05-28T06:12:28.864490+0000","last_unstale":"2024-05-28T06:12:28.869003+0000","last_undegraded":"2024-05-28T06:12:28.869003+0000","last_fullsized":"2024-05-28T06:12:28.869003+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:27.787600+0000","last_clean_scrub_stamp":"2024-05-28T06:12:27.787600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:22:20.292341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.839986+0000","last_change":"2024-05-28T06:12:27.832907+0000","last_active":"2024-05-28T06:12:28.839986+0000","last_peered":"2024-05-28T06:12:28.839986+0000","last_clean":"2024-05-28T06:12:28.839986+0000","last_became_active":"2024-05-28T06:12:27.832384+0000","last_became_peered":"2024-05-28T06:12:27.832384+0000","last_unstale":"2024-05-28T06:12:28.839986+0000","last_undegraded":"2024-05-28T06:12:28.839986+0000","last_fullsized":"2024-05-28T06:12:28.839986+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:12:26.773418+0000","last_clean_scrub_stamp":"2024-05-28T06:12:26.773418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:52:06.636146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":244,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-28T06:12:28.838241+0000","last_change":"2024-05-28T06:04:01.039129+0000","last_active":"2024-05-28T06:12:28.838241+0000","last_peered":"2024-05-28T06:12:28.838241+0000","last_clean":"2024-05-28T06:12:28.838241+0000","last_became_active":"2024-05-28T06:04:01.033906+0000","last_became_peered":"2024-05-28T06:04:01.033906+0000","last_unstale":"2024-05-28T06:12:28.838241+0000","last_undegraded":"2024-05-28T06:12:28.838241+0000","last_fullsized":"2024-05-28T06:12:28.838241+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:03:59.864237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:03:59.864237+0000","last_clean_scrub_stamp":"2024-05-28T06:03:59.864237+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:17:36.440245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574953,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46352,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325488,"statfs":{"total":96636764160,"available":96589299712,"internally_reserved":0,"allocated":1310720,"data_stored":888311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":46135754},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574955,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45764,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94326076,"statfs":{"total":96636764160,"available":96589901824,"internally_reserved":0,"allocated":708608,"data_stored":297783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574953,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27728,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344112,"statfs":{"total":96636764160,"available":96608370688,"internally_reserved":0,"allocated":716800,"data_stored":298435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574953,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45772,"kb_used_data":704,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94326068,"statfs":{"total":96636764160,"available":96589893632,"internally_reserved":0,"allocated":720896,"data_stored":299395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574955,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45744,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94326096,"statfs":{"total":96636764160,"available":96589922304,"internally_reserved":0,"allocated":696320,"data_stored":290980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574954,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45776,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94326064,"statfs":{"total":96636764160,"available":96589889536,"internally_reserved":0,"allocated":716800,"data_stored":297839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574954,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49924,"kb_used_data":696,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321916,"statfs":{"total":96636764160,"available":96585641984,"internally_reserved":0,"allocated":712704,"data_stored":297873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574953,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46364,"kb_used_data":1288,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325476,"statfs":{"total":96636764160,"available":96589287424,"internally_reserved":0,"allocated":1318912,"data_stored":890345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-28T06:12:34.273 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-05-28T06:12:34.273 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-05-28T06:12:34.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.525+0000 7f8a74e00640 1 Processor -- start 2024-05-28T06:12:34.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.525+0000 7f8a74e00640 1 -- start start 2024-05-28T06:12:34.529 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.526+0000 7f8a74e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f8a700d4680 0x7f8a700d2d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:34.530 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.526+0000 7f8a74e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f8a700d4a60 0x7f8a700d3280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:34.530 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.527+0000 7f8a74e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8a700b46a0 0x7f8a700b4b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:34.530 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.527+0000 7f8a74e00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f8a700d28b0 con 0x7f8a700d4680 2024-05-28T06:12:34.530 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.527+0000 7f8a74e00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f8a700d0150 con 0x7f8a700d4a60 2024-05-28T06:12:34.530 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.527+0000 7f8a74e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f8a700d0430 con 0x7f8a700b46a0 2024-05-28T06:12:34.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.527+0000 7f8a6f400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f8a700d4680 0x7f8a700d2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:34.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.527+0000 7f8a6f400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f8a700d4680 0x7f8a700d2d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44852/0 (socket says 172.21.15.96:44852) 2024-05-28T06:12:34.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.527+0000 7f8a6f400640 1 -- 172.21.15.96:0/2905560805 learned_addr learned my addr 172.21.15.96:0/2905560805 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:34.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.527+0000 7f8a6fe00640 1 --2- 172.21.15.96:0/2905560805 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8a700b46a0 0x7f8a700b4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:34.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.527+0000 7f8a6ea00640 1 --2- 172.21.15.96:0/2905560805 >> v2:172.21.15.123:3300/0 conn(0x7f8a700d4a60 0x7f8a700d3280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:34.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.527+0000 7f8a6f400640 1 -- 172.21.15.96:0/2905560805 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8a700b46a0 msgr2=0x7f8a700b4b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:34.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.527+0000 7f8a6f400640 1 --2- 172.21.15.96:0/2905560805 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8a700b46a0 0x7f8a700b4b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:34.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.527+0000 7f8a6f400640 1 -- 172.21.15.96:0/2905560805 >> v2:172.21.15.123:3300/0 conn(0x7f8a700d4a60 msgr2=0x7f8a700d3280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:34.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.527+0000 7f8a6f400640 1 --2- 172.21.15.96:0/2905560805 >> v2:172.21.15.123:3300/0 conn(0x7f8a700d4a60 0x7f8a700d3280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:34.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.527+0000 7f8a6f400640 1 -- 172.21.15.96:0/2905560805 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a700d2260 con 0x7f8a700d4680 2024-05-28T06:12:34.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.528+0000 7f8a6f400640 1 --2- 172.21.15.96:0/2905560805 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8a700d4680 0x7f8a700d2d40 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f8a700d0640 tx=0x7f8a6002a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b588e92288e2d2df server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:34.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.529+0000 7f8a6e000640 1 -- 172.21.15.96:0/2905560805 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a60037070 con 0x7f8a700d4680 2024-05-28T06:12:34.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.529+0000 7f8a6e000640 1 -- 172.21.15.96:0/2905560805 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a60032440 con 0x7f8a700d4680 2024-05-28T06:12:34.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.529+0000 7f8a6e000640 1 -- 172.21.15.96:0/2905560805 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a60032810 con 0x7f8a700d4680 2024-05-28T06:12:34.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.530+0000 7f8a74e00640 1 -- 172.21.15.96:0/2905560805 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8a700d4680 msgr2=0x7f8a700d2d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:34.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.530+0000 7f8a74e00640 1 --2- 172.21.15.96:0/2905560805 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8a700d4680 0x7f8a700d2d40 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f8a700d0640 tx=0x7f8a6002a1c0 comp rx=0 tx=0).stop 2024-05-28T06:12:34.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.530+0000 7f8a74e00640 1 -- 172.21.15.96:0/2905560805 shutdown_connections 2024-05-28T06:12:34.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.530+0000 7f8a74e00640 1 --2- 172.21.15.96:0/2905560805 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8a700b46a0 0x7f8a700b4b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:34.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.530+0000 7f8a74e00640 1 --2- 172.21.15.96:0/2905560805 >> v2:172.21.15.123:3300/0 conn(0x7f8a700d4a60 0x7f8a700d3280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:34.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.530+0000 7f8a74e00640 1 --2- 172.21.15.96:0/2905560805 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8a700d4680 0x7f8a700d2d40 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:34.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.530+0000 7f8a74e00640 1 -- 172.21.15.96:0/2905560805 >> 172.21.15.96:0/2905560805 conn(0x7f8a7009d810 msgr2=0x7f8a7008bfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:34.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.530+0000 7f8a74e00640 1 -- 172.21.15.96:0/2905560805 shutdown_connections 2024-05-28T06:12:34.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.530+0000 7f8a74e00640 1 -- 172.21.15.96:0/2905560805 wait complete. 2024-05-28T06:12:34.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.532+0000 7f8a74e00640 1 Processor -- start 2024-05-28T06:12:34.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.532+0000 7f8a74e00640 1 -- start start 2024-05-28T06:12:34.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.534+0000 7f8a74e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8a700b46a0 0x7f8a700b6460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:34.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.534+0000 7f8a74e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8a700d4680 0x7f8a700c7fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:34.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.534+0000 7f8a74e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8a700d4a60 0x7f8a700c9af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:34.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.534+0000 7f8a74e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f8a7015e460 con 0x7f8a700d4680 2024-05-28T06:12:34.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.534+0000 7f8a74e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f8a7015e2e0 con 0x7f8a700d4a60 2024-05-28T06:12:34.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.534+0000 7f8a74e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f8a7015e5e0 con 0x7f8a700b46a0 2024-05-28T06:12:34.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.534+0000 7f8a6ea00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8a700d4680 0x7f8a700c7fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:34.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.534+0000 7f8a6ea00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8a700d4680 0x7f8a700c7fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44856/0 (socket says 172.21.15.96:44856) 2024-05-28T06:12:34.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.534+0000 7f8a6ea00640 1 -- 172.21.15.96:0/3523613935 learned_addr learned my addr 172.21.15.96:0/3523613935 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:34.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.534+0000 7f8a6f400640 1 --2- 172.21.15.96:0/3523613935 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8a700b46a0 0x7f8a700b6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:34.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.534+0000 7f8a6fe00640 1 --2- 172.21.15.96:0/3523613935 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8a700d4a60 0x7f8a700c9af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:34.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.534+0000 7f8a6ea00640 1 -- 172.21.15.96:0/3523613935 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8a700b46a0 msgr2=0x7f8a700b6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:34.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.534+0000 7f8a6ea00640 1 --2- 172.21.15.96:0/3523613935 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8a700b46a0 0x7f8a700b6460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:34.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.534+0000 7f8a6ea00640 1 -- 172.21.15.96:0/3523613935 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8a700d4a60 msgr2=0x7f8a700c9af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:34.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.534+0000 7f8a6ea00640 1 --2- 172.21.15.96:0/3523613935 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8a700d4a60 0x7f8a700c9af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:34.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.534+0000 7f8a6ea00640 1 -- 172.21.15.96:0/3523613935 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a700c8a50 con 0x7f8a700d4680 2024-05-28T06:12:34.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.535+0000 7f8a6ea00640 1 --2- 172.21.15.96:0/3523613935 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8a700d4680 0x7f8a700c7fb0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f8a5c00c5e0 tx=0x7f8a5c00caa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:34.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.535+0000 7f8a67400640 1 -- 172.21.15.96:0/3523613935 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a5c004230 con 0x7f8a700d4680 2024-05-28T06:12:34.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.535+0000 7f8a67400640 1 -- 172.21.15.96:0/3523613935 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a5c0043f0 con 0x7f8a700d4680 2024-05-28T06:12:34.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.535+0000 7f8a74e00640 1 -- 172.21.15.96:0/3523613935 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a700c8d90 con 0x7f8a700d4680 2024-05-28T06:12:34.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.535+0000 7f8a67400640 1 -- 172.21.15.96:0/3523613935 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a5c003850 con 0x7f8a700d4680 2024-05-28T06:12:34.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.535+0000 7f8a74e00640 1 -- 172.21.15.96:0/3523613935 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a700c95d0 con 0x7f8a700d4680 2024-05-28T06:12:34.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.536+0000 7f8a74e00640 1 -- 172.21.15.96:0/3523613935 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a700cd7d0 con 0x7f8a700d4680 2024-05-28T06:12:34.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.537+0000 7f8a67400640 1 -- 172.21.15.96:0/3523613935 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 16) ==== 91754+0+0 (secure 0 0 0) 0x7f8a5c0039f0 con 0x7f8a700d4680 2024-05-28T06:12:34.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.538+0000 7f8a67400640 1 --2- 172.21.15.96:0/3523613935 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8a5006d0d0 0x7f8a5006f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:34.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.538+0000 7f8a67400640 1 -- 172.21.15.96:0/3523613935 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5737+0+0 (secure 0 0 0) 0x7f8a5c08ce90 con 0x7f8a700d4680 2024-05-28T06:12:34.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.539+0000 7f8a6f400640 1 --2- 172.21.15.96:0/3523613935 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8a5006d0d0 0x7f8a5006f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:34.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.540+0000 7f8a6f400640 1 --2- 172.21.15.96:0/3523613935 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8a5006d0d0 0x7f8a5006f590 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f8a60034000 tx=0x7f8a6002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:34.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.543+0000 7f8a67400640 1 -- 172.21.15.96:0/3523613935 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8a5c092020 con 0x7f8a700d4680 2024-05-28T06:12:34.811 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.809+0000 7f8a74e00640 1 -- 172.21.15.96:0/3523613935 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f8a700689e0 con 0x7f8a700d4680 2024-05-28T06:12:34.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.814+0000 7f8a67400640 1 -- 172.21.15.96:0/3523613935 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v16) ==== 190+0+147 (secure 0 0 0) 0x7f8a700c8d90 con 0x7f8a700d4680 2024-05-28T06:12:34.816 INFO:teuthology.orchestra.run.smithi096.stdout:[client.0] 2024-05-28T06:12:34.816 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQCXc1Vm91hbGBAASTL3HQR8XOs7wVQEHbPl/g== 2024-05-28T06:12:34.816 INFO:teuthology.orchestra.run.smithi096.stdout: caps mds = "allow" 2024-05-28T06:12:34.816 INFO:teuthology.orchestra.run.smithi096.stdout: caps mon = "allow r" 2024-05-28T06:12:34.816 INFO:teuthology.orchestra.run.smithi096.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-05-28T06:12:34.816 INFO:teuthology.orchestra.run.smithi096.stderr:updated caps for client.0 2024-05-28T06:12:34.818 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.816+0000 7f8a74e00640 1 -- 172.21.15.96:0/3523613935 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8a5006d0d0 msgr2=0x7f8a5006f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:34.818 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.816+0000 7f8a74e00640 1 --2- 172.21.15.96:0/3523613935 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8a5006d0d0 0x7f8a5006f590 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f8a60034000 tx=0x7f8a6002e000 comp rx=0 tx=0).stop 2024-05-28T06:12:34.818 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.816+0000 7f8a74e00640 1 -- 172.21.15.96:0/3523613935 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8a700d4680 msgr2=0x7f8a700c7fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:34.818 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.816+0000 7f8a74e00640 1 --2- 172.21.15.96:0/3523613935 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8a700d4680 0x7f8a700c7fb0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f8a5c00c5e0 tx=0x7f8a5c00caa0 comp rx=0 tx=0).stop 2024-05-28T06:12:34.818 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.816+0000 7f8a74e00640 1 -- 172.21.15.96:0/3523613935 shutdown_connections 2024-05-28T06:12:34.818 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.816+0000 7f8a74e00640 1 --2- 172.21.15.96:0/3523613935 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8a5006d0d0 0x7f8a5006f590 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:34.818 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.816+0000 7f8a74e00640 1 --2- 172.21.15.96:0/3523613935 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8a700d4a60 0x7f8a700c9af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:34.818 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.816+0000 7f8a74e00640 1 --2- 172.21.15.96:0/3523613935 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8a700d4680 0x7f8a700c7fb0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:34.818 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.816+0000 7f8a74e00640 1 --2- 172.21.15.96:0/3523613935 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8a700b46a0 0x7f8a700b6460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:34.819 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.816+0000 7f8a74e00640 1 -- 172.21.15.96:0/3523613935 >> 172.21.15.96:0/3523613935 conn(0x7f8a7009d810 msgr2=0x7f8a700d0710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:34.819 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.816+0000 7f8a74e00640 1 -- 172.21.15.96:0/3523613935 shutdown_connections 2024-05-28T06:12:34.819 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:34.816+0000 7f8a74e00640 1 -- 172.21.15.96:0/3523613935 wait complete. 2024-05-28T06:12:34.835 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-28T06:12:35.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.029+0000 7fe0d2200640 1 Processor -- start 2024-05-28T06:12:35.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.029+0000 7fe0d2200640 1 -- start start 2024-05-28T06:12:35.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.031+0000 7fe0d2200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fe0cc0b28c0 0x7fe0cc0b2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:35.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.031+0000 7fe0d2200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe0cc0d1140 0x7fe0cc0b52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:35.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.031+0000 7fe0d2200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0cc0cdcf0 con 0x7fe0cc0d0230 2024-05-28T06:12:35.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.031+0000 7fe0d2200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fe0cc09aa30 con 0x7fe0cc0b28c0 2024-05-28T06:12:35.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.031+0000 7fe0d2200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fe0cc0cdfd0 con 0x7fe0cc0d1140 2024-05-28T06:12:35.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.032+0000 7fe0cbe00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe0cc0d1140 0x7fe0cc0b52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:35.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.032+0000 7fe0cb400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fe0cc0b28c0 0x7fe0cc0b2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:35.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.032+0000 7fe0cb400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fe0cc0b28c0 0x7fe0cc0b2fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.96:39724/0 (socket says 172.21.15.96:39724) 2024-05-28T06:12:35.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.032+0000 7fe0cb400640 1 -- 172.21.15.96:0/3074228961 learned_addr learned my addr 172.21.15.96:0/3074228961 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:35.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.033+0000 7fe0cb400640 1 -- 172.21.15.96:0/3074228961 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe0cc0d1140 msgr2=0x7fe0cc0b52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:35.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.033+0000 7fe0cb400640 1 --2- 172.21.15.96:0/3074228961 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe0cc0d1140 0x7fe0cc0b52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:35.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.033+0000 7fe0cb400640 1 -- 172.21.15.96:0/3074228961 >> v1:172.21.15.123:6789/0 conn(0x7fe0cc0d0230 legacy=0x7fe0cc0d0a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:35.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.033+0000 7fe0cb400640 1 -- 172.21.15.96:0/3074228961 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0cc0ce2b0 con 0x7fe0cc0b28c0 2024-05-28T06:12:35.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.034+0000 7fe0cb400640 1 --2- 172.21.15.96:0/3074228961 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe0cc0b28c0 0x7fe0cc0b2fe0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fe0bc009870 tx=0x7fe0bc02a250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6e9ad480b380e857 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:35.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.035+0000 7fe0ca000640 1 -- 172.21.15.96:0/3074228961 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0bc037070 con 0x7fe0cc0b28c0 2024-05-28T06:12:35.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.035+0000 7fe0ca000640 1 -- 172.21.15.96:0/3074228961 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0bc032440 con 0x7fe0cc0b28c0 2024-05-28T06:12:35.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.035+0000 7fe0ca000640 1 -- 172.21.15.96:0/3074228961 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0bc032810 con 0x7fe0cc0b28c0 2024-05-28T06:12:35.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.036+0000 7fe0d2200640 1 -- 172.21.15.96:0/3074228961 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe0cc0b28c0 msgr2=0x7fe0cc0b2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:35.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.036+0000 7fe0d2200640 1 --2- 172.21.15.96:0/3074228961 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe0cc0b28c0 0x7fe0cc0b2fe0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fe0bc009870 tx=0x7fe0bc02a250 comp rx=0 tx=0).stop 2024-05-28T06:12:35.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.036+0000 7fe0d2200640 1 -- 172.21.15.96:0/3074228961 shutdown_connections 2024-05-28T06:12:35.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.036+0000 7fe0d2200640 1 --2- 172.21.15.96:0/3074228961 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe0cc0d1140 0x7fe0cc0b52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:35.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.036+0000 7fe0d2200640 1 --2- 172.21.15.96:0/3074228961 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe0cc0b28c0 0x7fe0cc0b2fe0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:35.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.036+0000 7fe0d2200640 1 -- 172.21.15.96:0/3074228961 >> 172.21.15.96:0/3074228961 conn(0x7fe0cc0b3f40 msgr2=0x7fe0cc0b4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:35.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.036+0000 7fe0d2200640 1 -- 172.21.15.96:0/3074228961 shutdown_connections 2024-05-28T06:12:35.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.036+0000 7fe0d2200640 1 -- 172.21.15.96:0/3074228961 wait complete. 2024-05-28T06:12:35.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.038+0000 7fe0d2200640 1 Processor -- start 2024-05-28T06:12:35.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.038+0000 7fe0d2200640 1 -- start start 2024-05-28T06:12:35.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.039+0000 7fe0d2200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe0cc0b28c0 0x7fe0cc154980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:35.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.039+0000 7fe0d2200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe0cc0d0230 0x7fe0cc155240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:35.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.039+0000 7fe0d2200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe0cc0d1140 0x7fe0cc0c8b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:35.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.039+0000 7fe0d2200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe0cc0b2e20 con 0x7fe0cc0d0230 2024-05-28T06:12:35.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.039+0000 7fe0d2200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe0cc0b2ca0 con 0x7fe0cc0b28c0 2024-05-28T06:12:35.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.039+0000 7fe0d2200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fe0cc09b1a0 con 0x7fe0cc0d1140 2024-05-28T06:12:35.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.040+0000 7fe0caa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe0cc0d0230 0x7fe0cc155240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:35.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.040+0000 7fe0caa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe0cc0d0230 0x7fe0cc155240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44870/0 (socket says 172.21.15.96:44870) 2024-05-28T06:12:35.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.040+0000 7fe0caa00640 1 -- 172.21.15.96:0/2815117570 learned_addr learned my addr 172.21.15.96:0/2815117570 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:35.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.040+0000 7fe0cb400640 1 --2- 172.21.15.96:0/2815117570 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe0cc0b28c0 0x7fe0cc154980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:35.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.040+0000 7fe0cbe00640 1 --2- 172.21.15.96:0/2815117570 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe0cc0d1140 0x7fe0cc0c8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:35.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.040+0000 7fe0caa00640 1 -- 172.21.15.96:0/2815117570 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe0cc0d1140 msgr2=0x7fe0cc0c8b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:35.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.040+0000 7fe0caa00640 1 --2- 172.21.15.96:0/2815117570 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe0cc0d1140 0x7fe0cc0c8b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:35.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.040+0000 7fe0caa00640 1 -- 172.21.15.96:0/2815117570 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe0cc0b28c0 msgr2=0x7fe0cc154980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:35.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.040+0000 7fe0caa00640 1 --2- 172.21.15.96:0/2815117570 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe0cc0b28c0 0x7fe0cc154980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:35.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.040+0000 7fe0caa00640 1 -- 172.21.15.96:0/2815117570 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0cc0b04d0 con 0x7fe0cc0d0230 2024-05-28T06:12:35.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.040+0000 7fe0caa00640 1 --2- 172.21.15.96:0/2815117570 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe0cc0d0230 0x7fe0cc155240 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fe0b800c790 tx=0x7fe0b800cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:35.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.041+0000 7fe0c3400640 1 -- 172.21.15.96:0/2815117570 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0b8016020 con 0x7fe0cc0d0230 2024-05-28T06:12:35.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.041+0000 7fe0c3400640 1 -- 172.21.15.96:0/2815117570 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0b8006d50 con 0x7fe0cc0d0230 2024-05-28T06:12:35.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.041+0000 7fe0c3400640 1 -- 172.21.15.96:0/2815117570 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0b8003bb0 con 0x7fe0cc0d0230 2024-05-28T06:12:35.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.041+0000 7fe0d2200640 1 -- 172.21.15.96:0/2815117570 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0cc0b07b0 con 0x7fe0cc0d0230 2024-05-28T06:12:35.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.041+0000 7fe0d2200640 1 -- 172.21.15.96:0/2815117570 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0cc0b0d70 con 0x7fe0cc0d0230 2024-05-28T06:12:35.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.042+0000 7fe0d2200640 1 -- 172.21.15.96:0/2815117570 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0cc0689e0 con 0x7fe0cc0d0230 2024-05-28T06:12:35.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.043+0000 7fe0c3400640 1 -- 172.21.15.96:0/2815117570 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 16) ==== 91754+0+0 (secure 0 0 0) 0x7fe0b800a7f0 con 0x7fe0cc0d0230 2024-05-28T06:12:35.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.044+0000 7fe0c3400640 1 --2- 172.21.15.96:0/2815117570 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe0ac06d0d0 0x7fe0ac06f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:35.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.044+0000 7fe0c3400640 1 -- 172.21.15.96:0/2815117570 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5737+0+0 (secure 0 0 0) 0x7fe0b808cb00 con 0x7fe0cc0d0230 2024-05-28T06:12:35.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.045+0000 7fe0cb400640 1 --2- 172.21.15.96:0/2815117570 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe0ac06d0d0 0x7fe0ac06f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:35.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.046+0000 7fe0cb400640 1 --2- 172.21.15.96:0/2815117570 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe0ac06d0d0 0x7fe0ac06f590 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fe0bc034000 tx=0x7fe0bc02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:35.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.048+0000 7fe0c3400640 1 -- 172.21.15.96:0/2815117570 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe0b8020020 con 0x7fe0cc0d0230 2024-05-28T06:12:35.274 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.273+0000 7fe0d2200640 1 -- 172.21.15.96:0/2815117570 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe0cc0d0810 con 0x7fe0cc0d0230 2024-05-28T06:12:35.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.274+0000 7fe0c3400640 1 -- 172.21.15.96:0/2815117570 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+5184 (secure 0 0 0) 0x7fe0b8059f80 con 0x7fe0cc0d0230 2024-05-28T06:12:35.276 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:12:35.276 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":34,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5015,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/1869890774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":1869890774},{"type":"v1","addr":"172.21.15.123:6837","nonce":1869890774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":5386,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/873923319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":873923319},{"type":"v1","addr":"172.21.15.123:6835","nonce":873923319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":6486,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6834/1041571401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":1041571401},{"type":"v1","addr":"172.21.15.96:6834","nonce":1041571401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-28T06:12:28.268409+0000","modified":"2024-05-28T06:12:31.363489+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6489},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6489":{"gid":6489,"name":"a","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.96:6835/1590650245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6833","nonce":1590650245},{"type":"v1","addr":"172.21.15.96:6835","nonce":1590650245}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6489,"qdb_cluster":[6489]},"id":4}]} 2024-05-28T06:12:35.276 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 34 2024-05-28T06:12:35.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.276+0000 7fe0d2200640 1 -- 172.21.15.96:0/2815117570 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe0ac06d0d0 msgr2=0x7fe0ac06f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:35.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.276+0000 7fe0d2200640 1 --2- 172.21.15.96:0/2815117570 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe0ac06d0d0 0x7fe0ac06f590 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fe0bc034000 tx=0x7fe0bc02e000 comp rx=0 tx=0).stop 2024-05-28T06:12:35.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.276+0000 7fe0d2200640 1 -- 172.21.15.96:0/2815117570 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe0cc0d0230 msgr2=0x7fe0cc155240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:35.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.276+0000 7fe0d2200640 1 --2- 172.21.15.96:0/2815117570 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe0cc0d0230 0x7fe0cc155240 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fe0b800c790 tx=0x7fe0b800cc50 comp rx=0 tx=0).stop 2024-05-28T06:12:35.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.276+0000 7fe0d2200640 1 -- 172.21.15.96:0/2815117570 shutdown_connections 2024-05-28T06:12:35.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.276+0000 7fe0d2200640 1 --2- 172.21.15.96:0/2815117570 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe0ac06d0d0 0x7fe0ac06f590 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:35.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.276+0000 7fe0d2200640 1 --2- 172.21.15.96:0/2815117570 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe0cc0d1140 0x7fe0cc0c8b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:35.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.276+0000 7fe0d2200640 1 --2- 172.21.15.96:0/2815117570 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe0cc0d0230 0x7fe0cc155240 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:35.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.276+0000 7fe0d2200640 1 --2- 172.21.15.96:0/2815117570 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe0cc0b28c0 0x7fe0cc154980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:35.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.276+0000 7fe0d2200640 1 -- 172.21.15.96:0/2815117570 >> 172.21.15.96:0/2815117570 conn(0x7fe0cc0b3f40 msgr2=0x7fe0cc0b53a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:35.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.277+0000 7fe0d2200640 1 -- 172.21.15.96:0/2815117570 shutdown_connections 2024-05-28T06:12:35.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.277+0000 7fe0d2200640 1 -- 172.21.15.96:0/2815117570 wait complete. 2024-05-28T06:12:35.699 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 34, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-05-28T06:12:28.268409+0000', 'modified': '2024-05-28T06:12:31.363489+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6489}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6489': {'gid': 6489, 'name': 'a', 'rank': 0, 'incarnation': 30, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.96:6835/1590650245', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.96:6833', 'nonce': 1590650245}, {'type': 'v1', 'addr': '172.21.15.96:6835', 'nonce': 1590650245}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6489, 'qdb_cluster': [6489]} 2024-05-28T06:12:35.699 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-05-28T06:12:35.699 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-05-28T06:12:35.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.911+0000 7fb583c00640 1 Processor -- start 2024-05-28T06:12:35.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.911+0000 7fb583c00640 1 -- start start 2024-05-28T06:12:35.914 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.913+0000 7fb583c00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fb57c0b3b70 0x7fb57c0d0930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:35.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.913+0000 7fb583c00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb57c0ced40 con 0x7fb57c0b4b60 2024-05-28T06:12:35.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.913+0000 7fb583c00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb57c0b3f90 con 0x7fb57c0d04a0 2024-05-28T06:12:35.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.913+0000 7fb583c00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fb57c15fbb0 con 0x7fb57c0b3b70 2024-05-28T06:12:35.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.913+0000 7fb581200640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fb57c0b4b60 0x7fb57c0cfaa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:44598/0 (socket says 172.21.15.96:44598) 2024-05-28T06:12:35.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.913+0000 7fb581200640 1 -- 172.21.15.96:0/3341021262 learned_addr learned my addr 172.21.15.96:0/3341021262 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:12:35.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.913+0000 7fb581c00640 1 --2- 172.21.15.96:0/3341021262 >> v2:172.21.15.96:3300/0 conn(0x7fb57c0b3b70 0x7fb57c0d0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:35.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.914+0000 7fb57be00640 1 -- 172.21.15.96:0/3341021262 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 25149597 0 0) 0x7fb564001110 con 0x7fb57c0b4b60 2024-05-28T06:12:35.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.914+0000 7fb57be00640 1 -- 172.21.15.96:0/3341021262 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb55c003520 con 0x7fb57c0b4b60 2024-05-28T06:12:35.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.914+0000 7fb581c00640 1 -- 172.21.15.96:0/3341021262 >> v1:172.21.15.123:6789/0 conn(0x7fb57c0d04a0 legacy=0x7fb57c0b3460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:35.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.914+0000 7fb581c00640 1 -- 172.21.15.96:0/3341021262 >> v1:172.21.15.96:6789/0 conn(0x7fb57c0b4b60 legacy=0x7fb57c0cfaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:35.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.914+0000 7fb581c00640 1 -- 172.21.15.96:0/3341021262 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb57c0cefc0 con 0x7fb57c0b3b70 2024-05-28T06:12:35.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.915+0000 7fb581c00640 1 --2- 172.21.15.96:0/3341021262 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb57c0b3b70 0x7fb57c0d0930 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fb574009d90 tx=0x7fb57402a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77976dbb0eee2bf8 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:35.917 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.915+0000 7fb57be00640 1 -- 172.21.15.96:0/3341021262 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb574007740 con 0x7fb57c0b3b70 2024-05-28T06:12:35.917 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.915+0000 7fb57be00640 1 -- 172.21.15.96:0/3341021262 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5740078e0 con 0x7fb57c0b3b70 2024-05-28T06:12:35.917 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.915+0000 7fb57be00640 1 -- 172.21.15.96:0/3341021262 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb574007cb0 con 0x7fb57c0b3b70 2024-05-28T06:12:35.917 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.916+0000 7fb583c00640 1 -- 172.21.15.96:0/3341021262 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb57c0b3b70 msgr2=0x7fb57c0d0930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:35.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.916+0000 7fb583c00640 1 --2- 172.21.15.96:0/3341021262 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb57c0b3b70 0x7fb57c0d0930 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fb574009d90 tx=0x7fb57402a6a0 comp rx=0 tx=0).stop 2024-05-28T06:12:35.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.916+0000 7fb583c00640 1 -- 172.21.15.96:0/3341021262 shutdown_connections 2024-05-28T06:12:35.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.916+0000 7fb583c00640 1 --2- 172.21.15.96:0/3341021262 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb57c0b3b70 0x7fb57c0d0930 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:35.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.916+0000 7fb583c00640 1 -- 172.21.15.96:0/3341021262 >> 172.21.15.96:0/3341021262 conn(0x7fb57c06bae0 msgr2=0x7fb57c06bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:35.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.916+0000 7fb583c00640 1 -- 172.21.15.96:0/3341021262 shutdown_connections 2024-05-28T06:12:35.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.916+0000 7fb583c00640 1 -- 172.21.15.96:0/3341021262 wait complete. 2024-05-28T06:12:35.919 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.918+0000 7fb583c00640 1 Processor -- start 2024-05-28T06:12:35.919 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.918+0000 7fb583c00640 1 -- start start 2024-05-28T06:12:35.922 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.919+0000 7fb583c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb57c0b3b70 0x7fb57c099850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:35.922 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.919+0000 7fb583c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb57c0b4b60 0x7fb57c160dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:35.922 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.919+0000 7fb583c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb57c0d04a0 0x7fb57c161300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:35.922 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.919+0000 7fb583c00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb57c15ede0 con 0x7fb57c0d04a0 2024-05-28T06:12:35.922 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.919+0000 7fb583c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fb57c0cf580 con 0x7fb57c0b3b70 2024-05-28T06:12:35.922 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.919+0000 7fb583c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fb57c15ef60 con 0x7fb57c0b4b60 2024-05-28T06:12:35.922 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.919+0000 7fb581c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb57c0d04a0 0x7fb57c161300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:35.922 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.919+0000 7fb581c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb57c0d04a0 0x7fb57c161300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44888/0 (socket says 172.21.15.96:44888) 2024-05-28T06:12:35.922 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.919+0000 7fb581c00640 1 -- 172.21.15.96:0/1862191649 learned_addr learned my addr 172.21.15.96:0/1862191649 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:35.922 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.919+0000 7fb581200640 1 --2- 172.21.15.96:0/1862191649 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb57c0b3b70 0x7fb57c099850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:35.922 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.919+0000 7fb573e00640 1 --2- 172.21.15.96:0/1862191649 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb57c0b4b60 0x7fb57c160dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:35.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.919+0000 7fb581c00640 1 -- 172.21.15.96:0/1862191649 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb57c0b4b60 msgr2=0x7fb57c160dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:35.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.919+0000 7fb581c00640 1 --2- 172.21.15.96:0/1862191649 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb57c0b4b60 0x7fb57c160dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:35.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.919+0000 7fb581c00640 1 -- 172.21.15.96:0/1862191649 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb57c0b3b70 msgr2=0x7fb57c099850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:35.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.919+0000 7fb581c00640 1 --2- 172.21.15.96:0/1862191649 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb57c0b3b70 0x7fb57c099850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:35.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.919+0000 7fb581c00640 1 -- 172.21.15.96:0/1862191649 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb57c09d2a0 con 0x7fb57c0d04a0 2024-05-28T06:12:35.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.920+0000 7fb581c00640 1 --2- 172.21.15.96:0/1862191649 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb57c0d04a0 0x7fb57c161300 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fb574009d90 tx=0x7fb574004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:35.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.920+0000 7fb579600640 1 -- 172.21.15.96:0/1862191649 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb574040070 con 0x7fb57c0d04a0 2024-05-28T06:12:35.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.920+0000 7fb579600640 1 -- 172.21.15.96:0/1862191649 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb574031bc0 con 0x7fb57c0d04a0 2024-05-28T06:12:35.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.920+0000 7fb579600640 1 -- 172.21.15.96:0/1862191649 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb574009450 con 0x7fb57c0d04a0 2024-05-28T06:12:35.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.920+0000 7fb583c00640 1 -- 172.21.15.96:0/1862191649 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb57c09d580 con 0x7fb57c0d04a0 2024-05-28T06:12:35.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.920+0000 7fb583c00640 1 -- 172.21.15.96:0/1862191649 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb57c09db40 con 0x7fb57c0d04a0 2024-05-28T06:12:35.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.923+0000 7fb579600640 1 -- 172.21.15.96:0/1862191649 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 16) ==== 91754+0+0 (secure 0 0 0) 0x7fb574039040 con 0x7fb57c0d04a0 2024-05-28T06:12:35.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.923+0000 7fb579600640 1 --2- 172.21.15.96:0/1862191649 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb55806d0d0 0x7fb55806f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:35.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.924+0000 7fb579600640 1 -- 172.21.15.96:0/1862191649 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5737+0+0 (secure 0 0 0) 0x7fb5740ae4e0 con 0x7fb57c0d04a0 2024-05-28T06:12:35.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.924+0000 7fb586a5c740 1 Processor -- start 2024-05-28T06:12:35.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.924+0000 7fb581200640 1 --2- 172.21.15.96:0/1862191649 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb55806d0d0 0x7fb55806f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:35.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.924+0000 7fb586a5c740 1 -- start start 2024-05-28T06:12:35.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.925+0000 7fb586a5c740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560aa21eb2a0 0x560aa21eea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:35.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.925+0000 7fb586a5c740 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x560aa21eef60 0x560aa220aea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:35.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.925+0000 7fb586a5c740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560aa220b3e0 0x560aa220b840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:35.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.925+0000 7fb586a5c740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x560aa2128b10 con 0x560aa220b3e0 2024-05-28T06:12:35.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.925+0000 7fb586a5c740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x560aa20cf2b0 con 0x560aa21eb2a0 2024-05-28T06:12:35.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.925+0000 7fb586a5c740 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x560aa2124ad0 con 0x560aa21eef60 2024-05-28T06:12:35.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.925+0000 7fb581c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560aa220b3e0 0x560aa220b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:35.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.925+0000 7fb581c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560aa220b3e0 0x560aa220b840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44900/0 (socket says 172.21.15.96:44900) 2024-05-28T06:12:35.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.925+0000 7fb581c00640 1 -- 172.21.15.96:0/3661820156 learned_addr learned my addr 172.21.15.96:0/3661820156 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:35.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.925+0000 7fb581200640 1 --2- 172.21.15.96:0/1862191649 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb55806d0d0 0x7fb55806f590 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fb564005880 tx=0x7fb564009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:35.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.926+0000 7fb581c00640 1 -- 172.21.15.96:0/3661820156 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x560aa21eef60 msgr2=0x560aa220aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:35.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.926+0000 7fb581c00640 1 --2- 172.21.15.96:0/3661820156 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x560aa21eef60 0x560aa220aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:35.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.926+0000 7fb581c00640 1 -- 172.21.15.96:0/3661820156 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560aa21eb2a0 msgr2=0x560aa21eea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:35.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.926+0000 7fb581c00640 1 --2- 172.21.15.96:0/3661820156 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560aa21eb2a0 0x560aa21eea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:35.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.926+0000 7fb581c00640 1 -- 172.21.15.96:0/3661820156 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560aa1e3b2d0 con 0x560aa220b3e0 2024-05-28T06:12:35.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.926+0000 7fb581c00640 1 --2- 172.21.15.96:0/3661820156 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560aa220b3e0 0x560aa220b840 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fb574077520 tx=0x7fb57404b340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bee4f671584e224f server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:35.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.926+0000 7fb571600640 1 -- 172.21.15.96:0/3661820156 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb57407b0c0 con 0x560aa220b3e0 2024-05-28T06:12:35.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.926+0000 7fb571600640 1 -- 172.21.15.96:0/3661820156 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb57407f470 con 0x560aa220b3e0 2024-05-28T06:12:35.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.926+0000 7fb571600640 1 -- 172.21.15.96:0/3661820156 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb574083ca0 con 0x560aa220b3e0 2024-05-28T06:12:35.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.927+0000 7fb586a5c740 1 -- 172.21.15.96:0/3661820156 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560aa220b3e0 msgr2=0x560aa220b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:35.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.927+0000 7fb586a5c740 1 --2- 172.21.15.96:0/3661820156 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560aa220b3e0 0x560aa220b840 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fb574077520 tx=0x7fb57404b340 comp rx=0 tx=0).stop 2024-05-28T06:12:35.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.927+0000 7fb586a5c740 1 -- 172.21.15.96:0/3661820156 shutdown_connections 2024-05-28T06:12:35.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.927+0000 7fb586a5c740 1 --2- 172.21.15.96:0/3661820156 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560aa220b3e0 0x560aa220b840 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:35.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.927+0000 7fb586a5c740 1 --2- 172.21.15.96:0/3661820156 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x560aa21eef60 0x560aa220aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:35.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.927+0000 7fb586a5c740 1 --2- 172.21.15.96:0/3661820156 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560aa21eb2a0 0x560aa21eea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:35.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.927+0000 7fb586a5c740 1 -- 172.21.15.96:0/3661820156 >> 172.21.15.96:0/3661820156 conn(0x560aa21c1280 msgr2=0x560aa21c2280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:35.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.927+0000 7fb586a5c740 1 -- 172.21.15.96:0/3661820156 shutdown_connections 2024-05-28T06:12:35.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.927+0000 7fb586a5c740 1 -- 172.21.15.96:0/3661820156 wait complete. 2024-05-28T06:12:35.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.928+0000 7fb586a5c740 1 -- start start 2024-05-28T06:12:35.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.928+0000 7fb586a5c740 1 Processor -- start 2024-05-28T06:12:35.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.929+0000 7fb586a5c740 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x560aa21eef60 0x560aa21e6b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:35.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.929+0000 7fb586a5c740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560aa220b3e0 0x560aa221ca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:35.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.929+0000 7fb586a5c740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560aa21e9900 0x560aa221cf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:35.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.929+0000 7fb586a5c740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x560aa2128b10 con 0x560aa220b3e0 2024-05-28T06:12:35.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.929+0000 7fb586a5c740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x560aa20e2570 con 0x560aa21e9900 2024-05-28T06:12:35.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.929+0000 7fb586a5c740 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x560aa21bf0b0 con 0x560aa21eef60 2024-05-28T06:12:35.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.929+0000 7fb586a5c740 10 client.0 _ll_register_callbacks cb 0x560aa20b0fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-28T06:12:35.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.929+0000 7fb573e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560aa220b3e0 0x560aa221ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:35.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.929+0000 7fb573e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560aa220b3e0 0x560aa221ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44902/0 (socket says 172.21.15.96:44902) 2024-05-28T06:12:35.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.929+0000 7fb573e00640 1 -- 172.21.15.96:0/3661377340 learned_addr learned my addr 172.21.15.96:0/3661377340 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:35.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.929+0000 7fb581c00640 1 --2- 172.21.15.96:0/3661377340 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560aa21e9900 0x560aa221cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:35.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.930+0000 7fb581200640 1 --2- 172.21.15.96:0/3661377340 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x560aa21eef60 0x560aa21e6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:35.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.930+0000 7fb573e00640 1 -- 172.21.15.96:0/3661377340 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x560aa21eef60 msgr2=0x560aa21e6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:35.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.930+0000 7fb573e00640 1 --2- 172.21.15.96:0/3661377340 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x560aa21eef60 0x560aa21e6b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:35.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.930+0000 7fb573e00640 1 -- 172.21.15.96:0/3661377340 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560aa21e9900 msgr2=0x560aa221cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:35.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.930+0000 7fb573e00640 1 --2- 172.21.15.96:0/3661377340 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560aa21e9900 0x560aa221cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:35.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.930+0000 7fb573e00640 1 -- 172.21.15.96:0/3661377340 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560aa1f056b0 con 0x560aa220b3e0 2024-05-28T06:12:35.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.930+0000 7fb581200640 1 --2- 172.21.15.96:0/3661377340 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x560aa21eef60 0x560aa21e6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:12:35.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.930+0000 7fb573e00640 1 --2- 172.21.15.96:0/3661377340 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560aa220b3e0 0x560aa221ca40 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7fb56c00ae20 tx=0x7fb56c00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=972682e76620b401 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:35.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.930+0000 7fb56a000640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-05-28T06:12:35.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.930+0000 7fb56a000640 1 -- 172.21.15.96:0/3661377340 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb56c015070 con 0x560aa220b3e0 2024-05-28T06:12:35.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.930+0000 7fb56a000640 1 -- 172.21.15.96:0/3661377340 <== mon.0 v2:172.21.15.96:3300/0 2 ==== osd_map(60..60 src has 1..60) ==== 5737+0+0 (secure 0 0 0) 0x7fb56c0115f0 con 0x560aa220b3e0 2024-05-28T06:12:35.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.930+0000 7fb586a5c740 1 -- 172.21.15.96:0/3661377340 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb57407f470 con 0x560aa220b3e0 2024-05-28T06:12:35.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.930+0000 7fb56a000640 1 -- 172.21.15.96:0/3661377340 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fb56c020050 con 0x560aa220b3e0 2024-05-28T06:12:35.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.931+0000 7fb586a5c740 10 client.6627 fetch_fsmap learned FSMap version 34 2024-05-28T06:12:35.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.931+0000 7fb586a5c740 1 -- 172.21.15.96:0/3661377340 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=2+}) -- 0x560aa221d8e0 con 0x560aa220b3e0 2024-05-28T06:12:35.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.931+0000 7fb56a000640 1 -- 172.21.15.96:0/3661377340 <== mon.0 v2:172.21.15.96:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb57407f470 con 0x560aa220b3e0 2024-05-28T06:12:35.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.931+0000 7fb56a000640 1 -- 172.21.15.96:0/3661377340 <== mon.0 v2:172.21.15.96:3300/0 5 ==== fsmap(e 34) ==== 3494+0+0 (secure 0 0 0) 0x7fb56c023020 con 0x560aa220b3e0 2024-05-28T06:12:35.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.931+0000 7fb586a5c740 10 client.6627 fetch_fsmap finished waiting for FSMap version 34 2024-05-28T06:12:35.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.931+0000 7fb586a5c740 10 client.6627 resolve_mds: resolved name 'a' to daemon mds.a 2024-05-28T06:12:35.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.931+0000 7fb586a5c740 20 client.6627 populate_metadata read hostname 'smithi096' 2024-05-28T06:12:35.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.931+0000 7fb586a5c740 1 --2- 172.21.15.96:0/3661377340 >> [v2:172.21.15.96:6833/1590650245,v1:172.21.15.96:6835/1590650245] conn(0x560aa21eff20 0x560aa2205f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:35.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.931+0000 7fb586a5c740 4 client.6627 mds_command: new command op to 6489 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-28T06:12:35.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.931+0000 7fb586a5c740 1 -- 172.21.15.96:0/3661377340 --> [v2:172.21.15.96:6833/1590650245,v1:172.21.15.96:6835/1590650245] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560aa2136530 con 0x560aa21eff20 2024-05-28T06:12:35.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.931+0000 7fb581200640 1 --2- 172.21.15.96:0/3661377340 >> [v2:172.21.15.96:6833/1590650245,v1:172.21.15.96:6835/1590650245] conn(0x560aa21eff20 0x560aa2205f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:35.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.932+0000 7fb581200640 1 --2- 172.21.15.96:0/3661377340 >> [v2:172.21.15.96:6833/1590650245,v1:172.21.15.96:6835/1590650245] conn(0x560aa21eff20 0x560aa2205f20 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=819efc1bd2f942f3 server_cookie=5acfa13bff87b317 in_seq=0 out_seq=0 2024-05-28T06:12:35.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.932+0000 7fb56a000640 10 client.6627 ms_handle_connect on v2:172.21.15.96:6833/1590650245 2024-05-28T06:12:35.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.933+0000 7fb56a000640 1 -- 172.21.15.96:0/3661377340 <== mds.0 v2:172.21.15.96:6833/1590650245 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560aa2136530 con 0x560aa21eff20 2024-05-28T06:12:35.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.933+0000 7fb56a000640 10 client.6627 handle_command_reply: tid=0 2024-05-28T06:12:35.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.933+0000 7fb586a5c740 1 client.6627 shutdown 2024-05-28T06:12:35.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.933+0000 7fb586a5c740 1 -- 172.21.15.96:0/3661377340 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560aa220b3e0 msgr2=0x560aa221ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:35.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.933+0000 7fb586a5c740 1 --2- 172.21.15.96:0/3661377340 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560aa220b3e0 0x560aa221ca40 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7fb56c00ae20 tx=0x7fb56c00c520 comp rx=0 tx=0).stop 2024-05-28T06:12:35.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.933+0000 7fb586a5c740 1 -- 172.21.15.96:0/3661377340 shutdown_connections 2024-05-28T06:12:35.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.933+0000 7fb586a5c740 1 --2- 172.21.15.96:0/3661377340 >> [v2:172.21.15.96:6833/1590650245,v1:172.21.15.96:6835/1590650245] conn(0x560aa21eff20 0x560aa2205f20 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:35.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.933+0000 7fb586a5c740 1 --2- 172.21.15.96:0/3661377340 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560aa21e9900 0x560aa221cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:35.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.933+0000 7fb586a5c740 1 --2- 172.21.15.96:0/3661377340 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560aa220b3e0 0x560aa221ca40 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:35.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.933+0000 7fb586a5c740 1 --2- 172.21.15.96:0/3661377340 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x560aa21eef60 0x560aa21e6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:35.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.933+0000 7fb56a000640 0 client.6627 ms_handle_reset on v2:172.21.15.96:6833/1590650245 2024-05-28T06:12:35.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.933+0000 7fb586a5c740 1 -- 172.21.15.96:0/3661377340 >> 172.21.15.96:0/3661377340 conn(0x560aa21eb2a0 msgr2=0x560aa220cfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:35.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.933+0000 7fb586a5c740 1 -- 172.21.15.96:0/3661377340 shutdown_connections 2024-05-28T06:12:35.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.934+0000 7fb586a5c740 1 -- 172.21.15.96:0/3661377340 wait complete. 2024-05-28T06:12:35.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.934+0000 7fb586a5c740 20 client.6627 trim_cache size 0 max 16384 2024-05-28T06:12:35.954 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.953+0000 7fb586a5c740 1 Processor -- start 2024-05-28T06:12:35.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.953+0000 7fb586a5c740 1 -- start start 2024-05-28T06:12:35.956 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.954+0000 7fb586a5c740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560aa21e9900 0x560aa21c0040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:35.956 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.954+0000 7fb586a5c740 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x560aa21eef60 0x560aa22099f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:35.956 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.954+0000 7fb586a5c740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560aa220b3e0 0x560aa2209f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:35.956 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.954+0000 7fb586a5c740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb57c200000 con 0x560aa220b3e0 2024-05-28T06:12:35.956 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.954+0000 7fb586a5c740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fb57c0c5d70 con 0x560aa21e9900 2024-05-28T06:12:35.956 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.954+0000 7fb586a5c740 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fb57c0c63c0 con 0x560aa21eef60 2024-05-28T06:12:35.956 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.954+0000 7fb581c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560aa220b3e0 0x560aa2209f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:35.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.954+0000 7fb581c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560aa220b3e0 0x560aa2209f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44906/0 (socket says 172.21.15.96:44906) 2024-05-28T06:12:35.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.954+0000 7fb581c00640 1 -- 172.21.15.96:0/717718860 learned_addr learned my addr 172.21.15.96:0/717718860 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:35.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.955+0000 7fb581c00640 1 -- 172.21.15.96:0/717718860 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x560aa21eef60 msgr2=0x560aa22099f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:35.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.955+0000 7fb581c00640 1 --2- 172.21.15.96:0/717718860 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x560aa21eef60 0x560aa22099f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:35.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.955+0000 7fb581c00640 1 -- 172.21.15.96:0/717718860 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560aa21e9900 msgr2=0x560aa21c0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:35.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.955+0000 7fb581c00640 1 --2- 172.21.15.96:0/717718860 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560aa21e9900 0x560aa21c0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:35.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.955+0000 7fb581c00640 1 -- 172.21.15.96:0/717718860 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560aa21e4050 con 0x560aa220b3e0 2024-05-28T06:12:35.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.955+0000 7fb581c00640 1 --2- 172.21.15.96:0/717718860 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560aa220b3e0 0x560aa2209f30 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fb574077520 tx=0x7fb5740ae050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a74de9b4bb1fa47a server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:35.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.955+0000 7fb570c00640 1 -- 172.21.15.96:0/717718860 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb57407f470 con 0x560aa220b3e0 2024-05-28T06:12:35.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.955+0000 7fb570c00640 1 -- 172.21.15.96:0/717718860 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5740793c0 con 0x560aa220b3e0 2024-05-28T06:12:35.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.955+0000 7fb570c00640 1 -- 172.21.15.96:0/717718860 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb574077730 con 0x560aa220b3e0 2024-05-28T06:12:35.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.956+0000 7fb586a5c740 1 -- 172.21.15.96:0/717718860 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560aa220b3e0 msgr2=0x560aa2209f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:35.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.956+0000 7fb586a5c740 1 --2- 172.21.15.96:0/717718860 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560aa220b3e0 0x560aa2209f30 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fb574077520 tx=0x7fb5740ae050 comp rx=0 tx=0).stop 2024-05-28T06:12:35.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.956+0000 7fb586a5c740 1 -- 172.21.15.96:0/717718860 shutdown_connections 2024-05-28T06:12:35.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.956+0000 7fb586a5c740 1 --2- 172.21.15.96:0/717718860 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560aa220b3e0 0x560aa2209f30 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:35.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.956+0000 7fb586a5c740 1 --2- 172.21.15.96:0/717718860 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x560aa21eef60 0x560aa22099f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:35.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.956+0000 7fb586a5c740 1 --2- 172.21.15.96:0/717718860 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560aa21e9900 0x560aa21c0040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:35.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.956+0000 7fb586a5c740 1 -- 172.21.15.96:0/717718860 >> 172.21.15.96:0/717718860 conn(0x560aa21eb2a0 msgr2=0x560aa21e9fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:35.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.956+0000 7fb586a5c740 1 -- 172.21.15.96:0/717718860 shutdown_connections 2024-05-28T06:12:35.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.956+0000 7fb586a5c740 1 -- 172.21.15.96:0/717718860 wait complete. 2024-05-28T06:12:35.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.957+0000 7fb586a5c740 1 -- start start 2024-05-28T06:12:35.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.957+0000 7fb586a5c740 1 Processor -- start 2024-05-28T06:12:35.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.958+0000 7fb586a5c740 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x560aa21e9900 0x560aa21e7b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:35.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.958+0000 7fb586a5c740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560aa21eef60 0x560aa2283020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:35.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.958+0000 7fb586a5c740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560aa220b3e0 0x560aa2284190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:35.962 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.958+0000 7fb586a5c740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x560aa21efda0 con 0x560aa220b3e0 2024-05-28T06:12:35.962 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.958+0000 7fb586a5c740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x560aa21f0820 con 0x560aa21eef60 2024-05-28T06:12:35.962 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.958+0000 7fb586a5c740 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x560aa21f0330 con 0x560aa21e9900 2024-05-28T06:12:35.962 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.958+0000 7fb586a5c740 10 client.0 _ll_register_callbacks cb 0x7fb56c00a9b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-28T06:12:35.962 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.958+0000 7fb581c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560aa220b3e0 0x560aa2284190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:35.962 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.958+0000 7fb581c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560aa220b3e0 0x560aa2284190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44908/0 (socket says 172.21.15.96:44908) 2024-05-28T06:12:35.962 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.958+0000 7fb581c00640 1 -- 172.21.15.96:0/4127721301 learned_addr learned my addr 172.21.15.96:0/4127721301 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:35.962 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.958+0000 7fb581c00640 1 -- 172.21.15.96:0/4127721301 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x560aa21e9900 msgr2=0x560aa21e7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:35.962 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.958+0000 7fb581c00640 1 --2- 172.21.15.96:0/4127721301 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x560aa21e9900 0x560aa21e7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:35.962 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.958+0000 7fb581c00640 1 -- 172.21.15.96:0/4127721301 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560aa21eef60 msgr2=0x560aa2283020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:35.963 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.958+0000 7fb581c00640 1 --2- 172.21.15.96:0/4127721301 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560aa21eef60 0x560aa2283020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:35.963 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.958+0000 7fb581c00640 1 -- 172.21.15.96:0/4127721301 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560aa2204af0 con 0x560aa220b3e0 2024-05-28T06:12:35.963 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.959+0000 7fb581c00640 1 --2- 172.21.15.96:0/4127721301 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560aa220b3e0 0x560aa2284190 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fb57407adc0 tx=0x7fb574046070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b239907cf9b1ade server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:35.963 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.959+0000 7fb56a000640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-05-28T06:12:35.963 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.959+0000 7fb56a000640 1 -- 172.21.15.96:0/4127721301 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5740ae2b0 con 0x560aa220b3e0 2024-05-28T06:12:35.963 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.959+0000 7fb56a000640 1 -- 172.21.15.96:0/4127721301 <== mon.0 v2:172.21.15.96:3300/0 2 ==== osd_map(60..60 src has 1..60) ==== 5737+0+0 (secure 0 0 0) 0x7fb574083810 con 0x560aa220b3e0 2024-05-28T06:12:35.963 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.959+0000 7fb586a5c740 1 -- 172.21.15.96:0/4127721301 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb5740793c0 con 0x560aa220b3e0 2024-05-28T06:12:35.963 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.959+0000 7fb56a000640 1 -- 172.21.15.96:0/4127721301 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fb57407f470 con 0x560aa220b3e0 2024-05-28T06:12:35.963 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.959+0000 7fb586a5c740 10 client.6633 fetch_fsmap learned FSMap version 34 2024-05-28T06:12:35.963 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.959+0000 7fb586a5c740 1 -- 172.21.15.96:0/4127721301 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=2+}) -- 0x560aa22846d0 con 0x560aa220b3e0 2024-05-28T06:12:35.963 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.959+0000 7fb56a000640 1 -- 172.21.15.96:0/4127721301 <== mon.0 v2:172.21.15.96:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5740793c0 con 0x560aa220b3e0 2024-05-28T06:12:35.963 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.959+0000 7fb56a000640 1 -- 172.21.15.96:0/4127721301 <== mon.0 v2:172.21.15.96:3300/0 5 ==== fsmap(e 34) ==== 3494+0+0 (secure 0 0 0) 0x7fb57407a510 con 0x560aa220b3e0 2024-05-28T06:12:35.963 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.960+0000 7fb586a5c740 10 client.6633 fetch_fsmap finished waiting for FSMap version 34 2024-05-28T06:12:35.964 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.960+0000 7fb586a5c740 10 client.6633 resolve_mds: resolved name 'a' to daemon mds.a 2024-05-28T06:12:35.964 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.960+0000 7fb586a5c740 20 client.6633 populate_metadata read hostname 'smithi096' 2024-05-28T06:12:35.964 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.960+0000 7fb586a5c740 1 --2- 172.21.15.96:0/4127721301 >> [v2:172.21.15.96:6833/1590650245,v1:172.21.15.96:6835/1590650245] conn(0x560aa220d220 0x560aa222bef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:35.964 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.960+0000 7fb586a5c740 4 client.6633 mds_command: new command op to 6489 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-28T06:12:35.964 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.960+0000 7fb586a5c740 1 -- 172.21.15.96:0/4127721301 --> [v2:172.21.15.96:6833/1590650245,v1:172.21.15.96:6835/1590650245] -- command(tid 0: {"prefix": "status"}) -- 0x560aa21303a0 con 0x560aa220d220 2024-05-28T06:12:35.964 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.960+0000 7fb581200640 1 --2- 172.21.15.96:0/4127721301 >> [v2:172.21.15.96:6833/1590650245,v1:172.21.15.96:6835/1590650245] conn(0x560aa220d220 0x560aa222bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:35.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.960+0000 7fb56a000640 10 client.6633 ms_handle_connect on v2:172.21.15.96:6833/1590650245 2024-05-28T06:12:35.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.960+0000 7fb581200640 1 --2- 172.21.15.96:0/4127721301 >> [v2:172.21.15.96:6833/1590650245,v1:172.21.15.96:6835/1590650245] conn(0x560aa220d220 0x560aa222bef0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6d42c7b1c17f4620 server_cookie=4ef2c6cf8deafe0f in_seq=0 out_seq=0 2024-05-28T06:12:35.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.961+0000 7fb56a000640 1 -- 172.21.15.96:0/4127721301 <== mds.0 v2:172.21.15.96:6833/1590650245 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x560aa21303a0 con 0x560aa220d220 2024-05-28T06:12:35.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.961+0000 7fb56a000640 10 client.6633 handle_command_reply: tid=0 2024-05-28T06:12:35.966 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.961+0000 7fb586a5c740 1 client.6633 shutdown 2024-05-28T06:12:35.966 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.961+0000 7fb586a5c740 1 -- 172.21.15.96:0/4127721301 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560aa220b3e0 msgr2=0x560aa2284190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:35.966 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.961+0000 7fb586a5c740 1 --2- 172.21.15.96:0/4127721301 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560aa220b3e0 0x560aa2284190 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fb57407adc0 tx=0x7fb574046070 comp rx=0 tx=0).stop 2024-05-28T06:12:35.966 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.961+0000 7fb586a5c740 1 -- 172.21.15.96:0/4127721301 shutdown_connections 2024-05-28T06:12:35.966 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.961+0000 7fb586a5c740 1 --2- 172.21.15.96:0/4127721301 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560aa220b3e0 0x560aa2284190 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:35.966 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.961+0000 7fb586a5c740 1 --2- 172.21.15.96:0/4127721301 >> [v2:172.21.15.96:6833/1590650245,v1:172.21.15.96:6835/1590650245] conn(0x560aa220d220 0x560aa222bef0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:35.966 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.961+0000 7fb586a5c740 1 --2- 172.21.15.96:0/4127721301 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560aa21eef60 0x560aa2283020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:35.966 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.961+0000 7fb586a5c740 1 --2- 172.21.15.96:0/4127721301 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x560aa21e9900 0x560aa21e7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:35.966 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.961+0000 7fb586a5c740 1 -- 172.21.15.96:0/4127721301 >> 172.21.15.96:0/4127721301 conn(0x560aa21eb2a0 msgr2=0x560aa2203670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:35.966 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.961+0000 7fb56a000640 0 client.6633 ms_handle_reset on v2:172.21.15.96:6833/1590650245 2024-05-28T06:12:35.966 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.961+0000 7fb586a5c740 1 -- 172.21.15.96:0/4127721301 shutdown_connections 2024-05-28T06:12:35.966 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.961+0000 7fb586a5c740 1 -- 172.21.15.96:0/4127721301 wait complete. 2024-05-28T06:12:35.967 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.961+0000 7fb586a5c740 20 client.6633 trim_cache size 0 max 16384 2024-05-28T06:12:35.967 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:12:35.967 INFO:teuthology.orchestra.run.smithi096.stdout: "cluster_fsid": "e4a831b5-e6d1-47a7-96cb-bf85154ee0d2", 2024-05-28T06:12:35.967 INFO:teuthology.orchestra.run.smithi096.stdout: "whoami": 0, 2024-05-28T06:12:35.967 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 6489, 2024-05-28T06:12:35.967 INFO:teuthology.orchestra.run.smithi096.stdout: "want_state": "up:active", 2024-05-28T06:12:35.967 INFO:teuthology.orchestra.run.smithi096.stdout: "state": "up:active", 2024-05-28T06:12:35.967 INFO:teuthology.orchestra.run.smithi096.stdout: "fs_name": "cephfs", 2024-05-28T06:12:35.967 INFO:teuthology.orchestra.run.smithi096.stdout: "rank_uptime": 7.1204278140000001, 2024-05-28T06:12:35.967 INFO:teuthology.orchestra.run.smithi096.stdout: "mdsmap_epoch": 34, 2024-05-28T06:12:35.968 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap_epoch": 60, 2024-05-28T06:12:35.968 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap_epoch_barrier": 60, 2024-05-28T06:12:35.968 INFO:teuthology.orchestra.run.smithi096.stdout: "uptime": 13.662414888000001 2024-05-28T06:12:35.968 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:12:35.968 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.962+0000 7fb583c00640 1 -- 172.21.15.96:0/1862191649 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb55806d0d0 msgr2=0x7fb55806f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:35.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.962+0000 7fb583c00640 1 --2- 172.21.15.96:0/1862191649 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb55806d0d0 0x7fb55806f590 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fb564005880 tx=0x7fb564009040 comp rx=0 tx=0).stop 2024-05-28T06:12:35.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.962+0000 7fb583c00640 1 -- 172.21.15.96:0/1862191649 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb57c0d04a0 msgr2=0x7fb57c161300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:35.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.962+0000 7fb583c00640 1 --2- 172.21.15.96:0/1862191649 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb57c0d04a0 0x7fb57c161300 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fb574009d90 tx=0x7fb574004a40 comp rx=0 tx=0).stop 2024-05-28T06:12:35.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.962+0000 7fb583c00640 1 -- 172.21.15.96:0/1862191649 shutdown_connections 2024-05-28T06:12:35.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.962+0000 7fb583c00640 1 --2- 172.21.15.96:0/1862191649 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb55806d0d0 0x7fb55806f590 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:35.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.962+0000 7fb583c00640 1 --2- 172.21.15.96:0/1862191649 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb57c0d04a0 0x7fb57c161300 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:35.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.962+0000 7fb583c00640 1 --2- 172.21.15.96:0/1862191649 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb57c0b4b60 0x7fb57c160dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:35.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.962+0000 7fb583c00640 1 --2- 172.21.15.96:0/1862191649 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb57c0b3b70 0x7fb57c099850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:35.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.962+0000 7fb583c00640 1 -- 172.21.15.96:0/1862191649 >> 172.21.15.96:0/1862191649 conn(0x7fb57c06bae0 msgr2=0x7fb57c0b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:35.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.962+0000 7fb583c00640 1 -- 172.21.15.96:0/1862191649 shutdown_connections 2024-05-28T06:12:35.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:35.962+0000 7fb583c00640 1 -- 172.21.15.96:0/1862191649 wait complete. 2024-05-28T06:12:35.981 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-05-28T06:12:36.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:36.177+0000 7f6856000640 1 Processor -- start 2024-05-28T06:12:36.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:36.177+0000 7f6856000640 1 -- start start 2024-05-28T06:12:36.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:36.178+0000 7f6856000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f685006ee40 0x7f68500ad230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:36.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:36.179+0000 7f6856000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f685006f2c0 0x7f68500ad770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:36.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:36.179+0000 7f6856000640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6850156f10 con 0x7f6850156b30 2024-05-28T06:12:36.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:36.179+0000 7f6856000640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f685006dda0 con 0x7f685006f2c0 2024-05-28T06:12:36.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:36.179+0000 7f6856000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f685006e080 con 0x7f685006ee40 2024-05-28T06:12:36.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:36.179+0000 7f6855600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f685006f2c0 0x7f68500ad770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:36.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:36.179+0000 7f6855600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f685006f2c0 0x7f68500ad770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44916/0 (socket says 172.21.15.96:44916) 2024-05-28T06:12:36.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:36.179+0000 7f6855600640 1 -- 172.21.15.96:0/2002343209 learned_addr learned my addr 172.21.15.96:0/2002343209 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:36.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:36.179+0000 7f684fe00640 1 --2- 172.21.15.96:0/2002343209 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f685006ee40 0x7f68500ad230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:36.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:36.179+0000 7f6855600640 1 -- 172.21.15.96:0/2002343209 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f685006ee40 msgr2=0x7f68500ad230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:36.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:36.179+0000 7f6855600640 1 --2- 172.21.15.96:0/2002343209 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f685006ee40 0x7f68500ad230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:36.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:36.179+0000 7f6855600640 1 -- 172.21.15.96:0/2002343209 >> v1:172.21.15.123:6789/0 conn(0x7f6850156b30 legacy=0x7f68500af6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:36.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:36.179+0000 7f6855600640 1 -- 172.21.15.96:0/2002343209 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f685006e360 con 0x7f685006f2c0 2024-05-28T06:12:36.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:36.180+0000 7f684fe00640 1 --2- 172.21.15.96:0/2002343209 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f685006ee40 0x7f68500ad230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:12:36.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:36.181+0000 7f6855600640 1 --2- 172.21.15.96:0/2002343209 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f685006f2c0 0x7f68500ad770 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f6848009e50 tx=0x7f684802a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b5aff9f791eed81 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:36.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:36.181+0000 7f684f400640 1 -- 172.21.15.96:0/2002343209 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6848036070 con 0x7f685006f2c0 2024-05-28T06:12:36.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:36.181+0000 7f684f400640 1 -- 172.21.15.96:0/2002343209 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6848031600 con 0x7f685006f2c0 2024-05-28T06:12:36.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:36.181+0000 7f684f400640 1 -- 172.21.15.96:0/2002343209 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68480319d0 con 0x7f685006f2c0 2024-05-28T06:12:36.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:36.181+0000 7f6856000640 1 -- 172.21.15.96:0/2002343209 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f685006f2c0 msgr2=0x7f68500ad770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:36.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:36.181+0000 7f6856000640 1 --2- 172.21.15.96:0/2002343209 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f685006f2c0 0x7f68500ad770 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f6848009e50 tx=0x7f684802a6a0 comp rx=0 tx=0).stop 2024-05-28T06:12:36.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:36.182+0000 7f6856000640 1 -- 172.21.15.96:0/2002343209 shutdown_connections 2024-05-28T06:12:36.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:36.182+0000 7f6856000640 1 --2- 172.21.15.96:0/2002343209 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f685006f2c0 0x7f68500ad770 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:36.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:36.182+0000 7f6856000640 1 --2- 172.21.15.96:0/2002343209 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f685006ee40 0x7f68500ad230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:36.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:36.182+0000 7f6856000640 1 -- 172.21.15.96:0/2002343209 >> 172.21.15.96:0/2002343209 conn(0x7f68500b1fc0 msgr2=0x7f68500b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:36.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:36.182+0000 7f6856000640 1 -- 172.21.15.96:0/2002343209 shutdown_connections 2024-05-28T06:12:36.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:36.182+0000 7f6856000640 1 -- 172.21.15.96:0/2002343209 wait complete. 2024-05-28T06:12:36.185 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:36.183+0000 7f6856000640 1 Processor -- start 2024-05-28T06:12:36.185 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:36.184+0000 7f6856000640 1 -- start start 2024-05-28T06:12:36.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:36.185+0000 7f6856000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f685006ee40 0x7f68501535e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:36.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:36.185+0000 7f6856000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f685006f2c0 0x7f6850151ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:36.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:36.185+0000 7f6856000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6850156b30 0x7f68501f3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:36.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:36.185+0000 7f6856000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f68500b2db0 con 0x7f685006ee40 2024-05-28T06:12:36.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:36.185+0000 7f6856000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f68500b2c30 con 0x7f685006f2c0 2024-05-28T06:12:36.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:36.185+0000 7f6856000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f68500b32c0 con 0x7f6850156b30 2024-05-28T06:12:36.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:36.185+0000 7f6854c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f685006ee40 0x7f68501535e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:36.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:36.185+0000 7f6854c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f685006ee40 0x7f68501535e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44930/0 (socket says 172.21.15.96:44930) 2024-05-28T06:12:36.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:36.185+0000 7f6854c00640 1 -- 172.21.15.96:0/2250240369 learned_addr learned my addr 172.21.15.96:0/2250240369 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:36.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:36.185+0000 7f684fe00640 1 --2- 172.21.15.96:0/2250240369 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f685006f2c0 0x7f6850151ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:36.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:36.185+0000 7f6854c00640 1 -- 172.21.15.96:0/2250240369 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6850156b30 msgr2=0x7f68501f3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:36.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:36.185+0000 7f6854c00640 1 --2- 172.21.15.96:0/2250240369 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6850156b30 0x7f68501f3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:36.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:36.185+0000 7f6854c00640 1 -- 172.21.15.96:0/2250240369 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f685006f2c0 msgr2=0x7f6850151ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:36.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:36.185+0000 7f6854c00640 1 --2- 172.21.15.96:0/2250240369 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f685006f2c0 0x7f6850151ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:36.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:36.185+0000 7f6854c00640 1 -- 172.21.15.96:0/2250240369 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68501520e0 con 0x7f685006ee40 2024-05-28T06:12:36.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:36.186+0000 7f6854c00640 1 --2- 172.21.15.96:0/2250240369 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f685006ee40 0x7f68501535e0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f684000c3c0 tx=0x7f684000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:36.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:36.186+0000 7f684cc00640 1 -- 172.21.15.96:0/2250240369 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68400052d0 con 0x7f685006ee40 2024-05-28T06:12:36.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:36.186+0000 7f684cc00640 1 -- 172.21.15.96:0/2250240369 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6840005490 con 0x7f685006ee40 2024-05-28T06:12:36.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:36.186+0000 7f684cc00640 1 -- 172.21.15.96:0/2250240369 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6840007260 con 0x7f685006ee40 2024-05-28T06:12:36.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:36.186+0000 7f6856000640 1 -- 172.21.15.96:0/2250240369 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6850154f30 con 0x7f685006ee40 2024-05-28T06:12:36.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:36.186+0000 7f6856000640 1 -- 172.21.15.96:0/2250240369 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68501554f0 con 0x7f685006ee40 2024-05-28T06:12:36.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:36.187+0000 7f6856000640 1 -- 172.21.15.96:0/2250240369 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68500689e0 con 0x7f685006ee40 2024-05-28T06:12:36.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:36.188+0000 7f684cc00640 1 -- 172.21.15.96:0/2250240369 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 16) ==== 91754+0+0 (secure 0 0 0) 0x7f6840007400 con 0x7f685006ee40 2024-05-28T06:12:36.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:36.189+0000 7f684cc00640 1 --2- 172.21.15.96:0/2250240369 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f683006d100 0x7f683006f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:36.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:36.189+0000 7f684cc00640 1 -- 172.21.15.96:0/2250240369 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5737+0+0 (secure 0 0 0) 0x7f684008c280 con 0x7f685006ee40 2024-05-28T06:12:36.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:36.190+0000 7f684fe00640 1 --2- 172.21.15.96:0/2250240369 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f683006d100 0x7f683006f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:36.192 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:36.191+0000 7f684fe00640 1 --2- 172.21.15.96:0/2250240369 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f683006d100 0x7f683006f5c0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f68501f7b50 tx=0x7f683c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:36.194 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:36.193+0000 7f684cc00640 1 -- 172.21.15.96:0/2250240369 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f68501520e0 con 0x7f685006ee40 2024-05-28T06:12:36.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:36.426+0000 7f6856000640 1 -- 172.21.15.96:0/2250240369 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f685006f7f0 con 0x7f685006ee40 2024-05-28T06:12:36.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:36.427+0000 7f684cc00640 1 -- 172.21.15.96:0/2250240369 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v34) ==== 53+0+73 (secure 0 0 0) 0x7f6840059700 con 0x7f685006ee40 2024-05-28T06:12:36.428 INFO:teuthology.orchestra.run.smithi096.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-05-28T06:12:36.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:36.429+0000 7f6856000640 1 -- 172.21.15.96:0/2250240369 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f683006d100 msgr2=0x7f683006f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:36.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:36.429+0000 7f6856000640 1 --2- 172.21.15.96:0/2250240369 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f683006d100 0x7f683006f5c0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f68501f7b50 tx=0x7f683c008040 comp rx=0 tx=0).stop 2024-05-28T06:12:36.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:36.429+0000 7f6856000640 1 -- 172.21.15.96:0/2250240369 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f685006ee40 msgr2=0x7f68501535e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:36.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:36.429+0000 7f6856000640 1 --2- 172.21.15.96:0/2250240369 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f685006ee40 0x7f68501535e0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f684000c3c0 tx=0x7f684000c880 comp rx=0 tx=0).stop 2024-05-28T06:12:36.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:36.429+0000 7f6856000640 1 -- 172.21.15.96:0/2250240369 shutdown_connections 2024-05-28T06:12:36.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:36.429+0000 7f6856000640 1 --2- 172.21.15.96:0/2250240369 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f683006d100 0x7f683006f5c0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:36.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:36.429+0000 7f6856000640 1 --2- 172.21.15.96:0/2250240369 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6850156b30 0x7f68501f3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:36.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:36.429+0000 7f6856000640 1 --2- 172.21.15.96:0/2250240369 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f685006f2c0 0x7f6850151ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:36.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:36.429+0000 7f6856000640 1 --2- 172.21.15.96:0/2250240369 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f685006ee40 0x7f68501535e0 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:36.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:36.429+0000 7f6856000640 1 -- 172.21.15.96:0/2250240369 >> 172.21.15.96:0/2250240369 conn(0x7f68500b1fc0 msgr2=0x7f6850072460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:36.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:36.430+0000 7f6856000640 1 -- 172.21.15.96:0/2250240369 shutdown_connections 2024-05-28T06:12:36.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:36.430+0000 7f6856000640 1 -- 172.21.15.96:0/2250240369 wait complete. 2024-05-28T06:12:36.450 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-05-28T06:12:36.450 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-28T06:12:36.450 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi096.front.sepia.ceph.com 2024-05-28T06:12:36.450 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-05-28T06:12:36.450 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-28T06:12:36.450 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-05-28T06:12:36.450 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-05-28T06:12:36.451 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-28T06:12:36.451 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-28T06:12:36.451 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:12:36.451 DEBUG:teuthology.orchestra.run.smithi096:> ip addr 2024-05-28T06:12:36.500 INFO:teuthology.orchestra.run.smithi096.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-28T06:12:36.500 INFO:teuthology.orchestra.run.smithi096.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-28T06:12:36.500 INFO:teuthology.orchestra.run.smithi096.stdout: inet 127.0.0.1/8 scope host lo 2024-05-28T06:12:36.500 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:12:36.500 INFO:teuthology.orchestra.run.smithi096.stdout: inet6 ::1/128 scope host 2024-05-28T06:12:36.501 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:12:36.501 INFO:teuthology.orchestra.run.smithi096.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-28T06:12:36.501 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether 0c:c4:7a:d8:d9:bc brd ff:ff:ff:ff:ff:ff 2024-05-28T06:12:36.501 INFO:teuthology.orchestra.run.smithi096.stdout: altname enp7s0f0 2024-05-28T06:12:36.501 INFO:teuthology.orchestra.run.smithi096.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-28T06:12:36.501 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether 0c:c4:7a:d8:d9:bd brd ff:ff:ff:ff:ff:ff 2024-05-28T06:12:36.501 INFO:teuthology.orchestra.run.smithi096.stdout: altname enp7s0f1 2024-05-28T06:12:36.501 INFO:teuthology.orchestra.run.smithi096.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-28T06:12:36.501 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether 0c:c4:7a:8f:cc:0c brd ff:ff:ff:ff:ff:ff 2024-05-28T06:12:36.501 INFO:teuthology.orchestra.run.smithi096.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-28T06:12:36.501 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether 0c:c4:7a:8f:cc:0d brd ff:ff:ff:ff:ff:ff 2024-05-28T06:12:36.501 INFO:teuthology.orchestra.run.smithi096.stdout: inet 172.21.15.96/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-05-28T06:12:36.501 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft 42081sec preferred_lft 42081sec 2024-05-28T06:12:36.502 INFO:teuthology.orchestra.run.smithi096.stdout: inet6 fe80::ec4:7aff:fe8f:cc0d/64 scope link 2024-05-28T06:12:36.502 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:12:36.502 INFO:teuthology.orchestra.run.smithi096.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-05-28T06:12:36.502 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether ae:d9:f5:b0:4f:f3 brd ff:ff:ff:ff:ff:ff 2024-05-28T06:12:36.502 INFO:teuthology.orchestra.run.smithi096.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-05-28T06:12:36.502 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:12:36.502 INFO:teuthology.orchestra.run.smithi096.stdout: inet6 fe80::604e:fcff:fe60:7892/64 scope link 2024-05-28T06:12:36.502 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:12:36.502 INFO:teuthology.orchestra.run.smithi096.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-28T06:12:36.502 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether ce:6f:19:93:44:f9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-28T06:12:36.502 INFO:teuthology.orchestra.run.smithi096.stdout: inet6 fe80::cc6f:19ff:fe93:44f9/64 scope link 2024-05-28T06:12:36.502 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:12:36.502 INFO:teuthology.orchestra.run.smithi096.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-28T06:12:36.502 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether ae:d9:f5:b0:4f:f3 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-28T06:12:36.503 INFO:teuthology.orchestra.run.smithi096.stdout: inet6 fe80::acd9:f5ff:feb0:4ff3/64 scope link 2024-05-28T06:12:36.503 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:12:36.503 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:12:36.504 DEBUG:teuthology.orchestra.run.smithi096:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-28T06:12:36.585 INFO:teuthology.orchestra.run.smithi096.stdout:1 2024-05-28T06:12:36.589 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:12:36.589 DEBUG:teuthology.orchestra.run.smithi096:> route 2024-05-28T06:12:36.635 INFO:teuthology.orchestra.run.smithi096.stdout:Kernel IP routing table 2024-05-28T06:12:36.635 INFO:teuthology.orchestra.run.smithi096.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-28T06:12:36.636 INFO:teuthology.orchestra.run.smithi096.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-05-28T06:12:36.636 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-05-28T06:12:36.636 INFO:teuthology.orchestra.run.smithi096.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-28T06:12:36.637 DEBUG:teuthology.orchestra.run.smithi096:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-28T06:12:36.637 DEBUG:teuthology.orchestra.run.smithi096:> set -e 2024-05-28T06:12:36.637 DEBUG:teuthology.orchestra.run.smithi096:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-05-28T06:12:36.637 DEBUG:teuthology.orchestra.run.smithi096:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-05-28T06:12:36.637 DEBUG:teuthology.orchestra.run.smithi096:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-05-28T06:12:36.637 DEBUG:teuthology.orchestra.run.smithi096:> ') 2024-05-28T06:12:36.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:36 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-28T06:12:36.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:36 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-28T06:12:36.898 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:12:36.898 DEBUG:teuthology.orchestra.run.smithi096:> ip netns list 2024-05-28T06:12:36.943 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-05-28T06:12:36.943 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-05-28T06:12:36.943 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-28T06:12:36.944 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:12:36.944 DEBUG:teuthology.orchestra.run.smithi096:> sudo ip link set brx.0 up 2024-05-28T06:12:37.067 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-05-28T06:12:37.067 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-28T06:12:37.067 DEBUG:teuthology.orchestra.run.smithi096:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-05-28T06:12:37.121 INFO:teuthology.orchestra.run.smithi096.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-05-28T06:12:37.121 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-28T06:12:37.122 DEBUG:teuthology.orchestra.run.smithi096:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-05-28T06:12:37.172 DEBUG:teuthology.orchestra.run.smithi096:> sudo modprobe fuse 2024-05-28T06:12:37.255 DEBUG:teuthology.orchestra.run.smithi096:> cat /proc/self/mounts | awk '{print $2}' 2024-05-28T06:12:37.308 INFO:teuthology.orchestra.run.smithi096.stdout:/proc 2024-05-28T06:12:37.309 INFO:teuthology.orchestra.run.smithi096.stdout:/sys 2024-05-28T06:12:37.309 INFO:teuthology.orchestra.run.smithi096.stdout:/dev 2024-05-28T06:12:37.309 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/security 2024-05-28T06:12:37.309 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/shm 2024-05-28T06:12:37.309 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/pts 2024-05-28T06:12:37.309 INFO:teuthology.orchestra.run.smithi096.stdout:/run 2024-05-28T06:12:37.309 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/cgroup 2024-05-28T06:12:37.309 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/pstore 2024-05-28T06:12:37.309 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/bpf 2024-05-28T06:12:37.309 INFO:teuthology.orchestra.run.smithi096.stdout:/ 2024-05-28T06:12:37.310 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/selinux 2024-05-28T06:12:37.310 INFO:teuthology.orchestra.run.smithi096.stdout:/proc/sys/fs/binfmt_misc 2024-05-28T06:12:37.310 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/mqueue 2024-05-28T06:12:37.310 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/debug 2024-05-28T06:12:37.310 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/hugepages 2024-05-28T06:12:37.310 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/tracing 2024-05-28T06:12:37.310 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/fuse/connections 2024-05-28T06:12:37.310 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/config 2024-05-28T06:12:37.310 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-sysctl.service 2024-05-28T06:12:37.310 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-28T06:12:37.310 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-28T06:12:37.310 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/nfs/rpc_pipefs 2024-05-28T06:12:37.311 INFO:teuthology.orchestra.run.smithi096.stdout:/run/user/1000 2024-05-28T06:12:37.311 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph 2024-05-28T06:12:37.311 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-0 2024-05-28T06:12:37.311 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-1 2024-05-28T06:12:37.311 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-2 2024-05-28T06:12:37.311 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-3 2024-05-28T06:12:37.311 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns 2024-05-28T06:12:37.311 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-28T06:12:37.311 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-28T06:12:37.311 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-28T06:12:37.311 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-28T06:12:37.312 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:12:37.312 DEBUG:teuthology.orchestra.run.smithi096:> ls /sys/fs/fuse/connections 2024-05-28T06:12:37.364 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-05-28T06:12:37.364 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-05-28T06:12:37.369 DEBUG:teuthology.orchestra.run.smithi096:> sudo modprobe fuse 2024-05-28T06:12:37.434 DEBUG:teuthology.orchestra.run.smithi096:> cat /proc/self/mounts | awk '{print $2}' 2024-05-28T06:12:37.475 INFO:teuthology.orchestra.run.smithi096.stdout:/proc 2024-05-28T06:12:37.475 INFO:teuthology.orchestra.run.smithi096.stdout:/sys 2024-05-28T06:12:37.475 INFO:teuthology.orchestra.run.smithi096.stdout:/dev 2024-05-28T06:12:37.475 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/security 2024-05-28T06:12:37.476 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/shm 2024-05-28T06:12:37.476 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/pts 2024-05-28T06:12:37.476 INFO:teuthology.orchestra.run.smithi096.stdout:/run 2024-05-28T06:12:37.476 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/cgroup 2024-05-28T06:12:37.476 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/pstore 2024-05-28T06:12:37.476 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/bpf 2024-05-28T06:12:37.476 INFO:teuthology.orchestra.run.smithi096.stdout:/ 2024-05-28T06:12:37.476 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/selinux 2024-05-28T06:12:37.476 INFO:teuthology.orchestra.run.smithi096.stdout:/proc/sys/fs/binfmt_misc 2024-05-28T06:12:37.476 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/mqueue 2024-05-28T06:12:37.476 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/debug 2024-05-28T06:12:37.476 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/hugepages 2024-05-28T06:12:37.476 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/tracing 2024-05-28T06:12:37.477 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/fuse/connections 2024-05-28T06:12:37.477 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/config 2024-05-28T06:12:37.477 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-sysctl.service 2024-05-28T06:12:37.477 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-28T06:12:37.477 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-28T06:12:37.477 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/nfs/rpc_pipefs 2024-05-28T06:12:37.477 INFO:teuthology.orchestra.run.smithi096.stdout:/run/user/1000 2024-05-28T06:12:37.477 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph 2024-05-28T06:12:37.477 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-0 2024-05-28T06:12:37.477 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-1 2024-05-28T06:12:37.477 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-2 2024-05-28T06:12:37.477 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-3 2024-05-28T06:12:37.478 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns 2024-05-28T06:12:37.478 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-28T06:12:37.478 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-28T06:12:37.478 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-28T06:12:37.478 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-28T06:12:37.479 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:12:37.479 DEBUG:teuthology.orchestra.run.smithi096:> ls /sys/fs/fuse/connections 2024-05-28T06:12:37.536 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi096.stderr:2024-05-28T06:12:37.534+0000 7fc721321480 -1 init, newargv = 0x55ecf0bc9d10 newargc=15 2024-05-28T06:12:37.536 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi096.stderr:ceph-fuse[60297]: starting ceph client 2024-05-28T06:12:37.545 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi096.stderr:ceph-fuse[60297]: starting fuse 2024-05-28T06:12:37.565 INFO:teuthology.orchestra.run.smithi096.stdout:40 2024-05-28T06:12:37.565 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40] 2024-05-28T06:12:37.565 DEBUG:teuthology.orchestra.run.smithi096:> sudo stdin-killer -- python3 -c ' 2024-05-28T06:12:37.565 DEBUG:teuthology.orchestra.run.smithi096:> import glob 2024-05-28T06:12:37.565 DEBUG:teuthology.orchestra.run.smithi096:> import re 2024-05-28T06:12:37.565 DEBUG:teuthology.orchestra.run.smithi096:> import os 2024-05-28T06:12:37.565 DEBUG:teuthology.orchestra.run.smithi096:> import subprocess 2024-05-28T06:12:37.566 DEBUG:teuthology.orchestra.run.smithi096:> 2024-05-28T06:12:37.566 DEBUG:teuthology.orchestra.run.smithi096:> def _find_admin_socket(client_name): 2024-05-28T06:12:37.566 DEBUG:teuthology.orchestra.run.smithi096:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-05-28T06:12:37.566 DEBUG:teuthology.orchestra.run.smithi096:> files = glob.glob(asok_path) 2024-05-28T06:12:37.566 DEBUG:teuthology.orchestra.run.smithi096:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-05-28T06:12:37.566 DEBUG:teuthology.orchestra.run.smithi096:> 2024-05-28T06:12:37.566 DEBUG:teuthology.orchestra.run.smithi096:> # Given a non-glob path, it better be there 2024-05-28T06:12:37.566 DEBUG:teuthology.orchestra.run.smithi096:> if "*" not in asok_path: 2024-05-28T06:12:37.566 DEBUG:teuthology.orchestra.run.smithi096:> assert(len(files) == 1) 2024-05-28T06:12:37.566 DEBUG:teuthology.orchestra.run.smithi096:> return files[0] 2024-05-28T06:12:37.566 DEBUG:teuthology.orchestra.run.smithi096:> 2024-05-28T06:12:37.566 DEBUG:teuthology.orchestra.run.smithi096:> for f in files: 2024-05-28T06:12:37.566 DEBUG:teuthology.orchestra.run.smithi096:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-05-28T06:12:37.566 DEBUG:teuthology.orchestra.run.smithi096:> if os.path.exists("/proc/{0}".format(pid)): 2024-05-28T06:12:37.567 DEBUG:teuthology.orchestra.run.smithi096:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-05-28T06:12:37.567 DEBUG:teuthology.orchestra.run.smithi096:> contents = proc_f.read() 2024-05-28T06:12:37.567 DEBUG:teuthology.orchestra.run.smithi096:> if mountpoint in contents: 2024-05-28T06:12:37.567 DEBUG:teuthology.orchestra.run.smithi096:> return f 2024-05-28T06:12:37.567 DEBUG:teuthology.orchestra.run.smithi096:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-05-28T06:12:37.567 DEBUG:teuthology.orchestra.run.smithi096:> 2024-05-28T06:12:37.567 DEBUG:teuthology.orchestra.run.smithi096:> print(_find_admin_socket("client.0")) 2024-05-28T06:12:37.567 DEBUG:teuthology.orchestra.run.smithi096:> ' 2024-05-28T06:12:37.750 INFO:teuthology.orchestra.run.smithi096.stdout:/var/run/ceph/ceph-client.0.60297.asok 2024-05-28T06:12:37.755 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:37 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-28T06:12:37.768 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.60297.asok 2024-05-28T06:12:37.768 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:12:37.768 DEBUG:teuthology.orchestra.run.smithi096:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.60297.asok status 2024-05-28T06:12:38.000 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:12:38.000 INFO:teuthology.orchestra.run.smithi096.stdout: "metadata": { 2024-05-28T06:12:38.000 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph_sha1": "9ed025fbf4406fb54980a0647506b15895b6d863", 2024-05-28T06:12:38.000 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph_version": "ceph version 19.0.0-3154-g9ed025fb (9ed025fbf4406fb54980a0647506b15895b6d863) squid (dev)", 2024-05-28T06:12:38.000 INFO:teuthology.orchestra.run.smithi096.stdout: "entity_id": "0", 2024-05-28T06:12:38.000 INFO:teuthology.orchestra.run.smithi096.stdout: "hostname": "smithi096", 2024-05-28T06:12:38.000 INFO:teuthology.orchestra.run.smithi096.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-05-28T06:12:38.000 INFO:teuthology.orchestra.run.smithi096.stdout: "pid": "60297", 2024-05-28T06:12:38.000 INFO:teuthology.orchestra.run.smithi096.stdout: "root": "/" 2024-05-28T06:12:38.001 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:12:38.001 INFO:teuthology.orchestra.run.smithi096.stdout: "dentry_count": 0, 2024-05-28T06:12:38.001 INFO:teuthology.orchestra.run.smithi096.stdout: "dentry_pinned_count": 0, 2024-05-28T06:12:38.001 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 6642, 2024-05-28T06:12:38.001 INFO:teuthology.orchestra.run.smithi096.stdout: "inst": { 2024-05-28T06:12:38.001 INFO:teuthology.orchestra.run.smithi096.stdout: "name": { 2024-05-28T06:12:38.001 INFO:teuthology.orchestra.run.smithi096.stdout: "type": "client", 2024-05-28T06:12:38.001 INFO:teuthology.orchestra.run.smithi096.stdout: "num": 6642 2024-05-28T06:12:38.001 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:12:38.001 INFO:teuthology.orchestra.run.smithi096.stdout: "addr": { 2024-05-28T06:12:38.001 INFO:teuthology.orchestra.run.smithi096.stdout: "type": "v1", 2024-05-28T06:12:38.001 INFO:teuthology.orchestra.run.smithi096.stdout: "addr": "192.168.0.1:0", 2024-05-28T06:12:38.001 INFO:teuthology.orchestra.run.smithi096.stdout: "nonce": 2782968185 2024-05-28T06:12:38.001 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:12:38.002 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:12:38.002 INFO:teuthology.orchestra.run.smithi096.stdout: "addr": { 2024-05-28T06:12:38.002 INFO:teuthology.orchestra.run.smithi096.stdout: "type": "v1", 2024-05-28T06:12:38.002 INFO:teuthology.orchestra.run.smithi096.stdout: "addr": "192.168.0.1:0", 2024-05-28T06:12:38.002 INFO:teuthology.orchestra.run.smithi096.stdout: "nonce": 2782968185 2024-05-28T06:12:38.002 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:12:38.002 INFO:teuthology.orchestra.run.smithi096.stdout: "inst_str": "client.6642 192.168.0.1:0/2782968185", 2024-05-28T06:12:38.002 INFO:teuthology.orchestra.run.smithi096.stdout: "addr_str": "192.168.0.1:0/2782968185", 2024-05-28T06:12:38.002 INFO:teuthology.orchestra.run.smithi096.stdout: "inode_count": 1, 2024-05-28T06:12:38.002 INFO:teuthology.orchestra.run.smithi096.stdout: "mds_epoch": 34, 2024-05-28T06:12:38.002 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_epoch": 60, 2024-05-28T06:12:38.002 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_epoch_barrier": 0, 2024-05-28T06:12:38.002 INFO:teuthology.orchestra.run.smithi096.stdout: "blocklisted": false, 2024-05-28T06:12:38.002 INFO:teuthology.orchestra.run.smithi096.stdout: "fs_name": "cephfs" 2024-05-28T06:12:38.003 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:12:38.015 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:12:38.015 DEBUG:teuthology.orchestra.run.smithi096:> stat --file-system '--printf=%T 2024-05-28T06:12:38.015 DEBUG:teuthology.orchestra.run.smithi096:> ' -- /home/ubuntu/cephtest/mnt.0 2024-05-28T06:12:38.069 INFO:teuthology.orchestra.run.smithi096.stdout:fuseblk 2024-05-28T06:12:38.069 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-05-28T06:12:38.069 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:12:38.070 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-05-28T06:12:38.156 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-05-28T06:12:38.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:38.357+0000 7fdd62a00640 1 Processor -- start 2024-05-28T06:12:38.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:38.357+0000 7fdd62a00640 1 -- start start 2024-05-28T06:12:38.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:38.359+0000 7fdd62a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fdd5c15fa00 0x7fdd5c06e970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:38.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:38.359+0000 7fdd62a00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fdd5c1600f0 0x7fdd5c0d0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:38.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:38.359+0000 7fdd62a00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd5c06fa00 con 0x7fdd5c0d0970 2024-05-28T06:12:38.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:38.359+0000 7fdd62a00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fdd5c0d0dc0 con 0x7fdd5c15fa00 2024-05-28T06:12:38.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:38.359+0000 7fdd62a00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fdd5c06eeb0 con 0x7fdd5c1600f0 2024-05-28T06:12:38.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:38.360+0000 7fdd61600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fdd5c15fa00 0x7fdd5c06e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:38.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:38.360+0000 7fdd61600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fdd5c15fa00 0x7fdd5c06e970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44932/0 (socket says 172.21.15.96:44932) 2024-05-28T06:12:38.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:38.360+0000 7fdd61600640 1 -- 172.21.15.96:0/3309003703 learned_addr learned my addr 172.21.15.96:0/3309003703 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:38.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:38.360+0000 7fdd60c00640 1 --2- 172.21.15.96:0/3309003703 >> v2:172.21.15.123:3300/0 conn(0x7fdd5c1600f0 0x7fdd5c0d0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:38.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:38.360+0000 7fdd61600640 1 -- 172.21.15.96:0/3309003703 >> v2:172.21.15.123:3300/0 conn(0x7fdd5c1600f0 msgr2=0x7fdd5c0d0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:38.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:38.360+0000 7fdd61600640 1 --2- 172.21.15.96:0/3309003703 >> v2:172.21.15.123:3300/0 conn(0x7fdd5c1600f0 0x7fdd5c0d0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:38.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:38.360+0000 7fdd61600640 1 -- 172.21.15.96:0/3309003703 >> v1:172.21.15.123:6789/0 conn(0x7fdd5c0d0970 legacy=0x7fdd5c0b3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:38.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:38.360+0000 7fdd61600640 1 -- 172.21.15.96:0/3309003703 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd5c06f160 con 0x7fdd5c15fa00 2024-05-28T06:12:38.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:38.361+0000 7fdd61600640 1 --2- 172.21.15.96:0/3309003703 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdd5c15fa00 0x7fdd5c06e970 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fdd5c0d0130 tx=0x7fdd5002a160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=940467b68a5ac1a0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:38.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:38.361+0000 7fdd5be00640 1 -- 172.21.15.96:0/3309003703 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd50037070 con 0x7fdd5c15fa00 2024-05-28T06:12:38.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:38.362+0000 7fdd5be00640 1 -- 172.21.15.96:0/3309003703 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd50032440 con 0x7fdd5c15fa00 2024-05-28T06:12:38.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:38.362+0000 7fdd5be00640 1 -- 172.21.15.96:0/3309003703 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd50032810 con 0x7fdd5c15fa00 2024-05-28T06:12:38.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:38.362+0000 7fdd62a00640 1 -- 172.21.15.96:0/3309003703 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdd5c15fa00 msgr2=0x7fdd5c06e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:38.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:38.362+0000 7fdd62a00640 1 --2- 172.21.15.96:0/3309003703 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdd5c15fa00 0x7fdd5c06e970 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fdd5c0d0130 tx=0x7fdd5002a160 comp rx=0 tx=0).stop 2024-05-28T06:12:38.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:38.362+0000 7fdd62a00640 1 -- 172.21.15.96:0/3309003703 shutdown_connections 2024-05-28T06:12:38.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:38.362+0000 7fdd62a00640 1 --2- 172.21.15.96:0/3309003703 >> v2:172.21.15.123:3300/0 conn(0x7fdd5c1600f0 0x7fdd5c0d0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:38.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:38.362+0000 7fdd62a00640 1 --2- 172.21.15.96:0/3309003703 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdd5c15fa00 0x7fdd5c06e970 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:38.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:38.362+0000 7fdd62a00640 1 -- 172.21.15.96:0/3309003703 >> 172.21.15.96:0/3309003703 conn(0x7fdd5c06bae0 msgr2=0x7fdd5c06bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:38.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:38.363+0000 7fdd62a00640 1 -- 172.21.15.96:0/3309003703 shutdown_connections 2024-05-28T06:12:38.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:38.363+0000 7fdd62a00640 1 -- 172.21.15.96:0/3309003703 wait complete. 2024-05-28T06:12:38.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:38.365+0000 7fdd62a00640 1 Processor -- start 2024-05-28T06:12:38.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:38.365+0000 7fdd62a00640 1 -- start start 2024-05-28T06:12:38.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:38.366+0000 7fdd62a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdd5c0d0970 0x7fdd5c14f440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:38.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:38.366+0000 7fdd62a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdd5c15fa00 0x7fdd5c09a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:38.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:38.366+0000 7fdd62a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdd5c1600f0 0x7fdd5c09a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:38.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:38.366+0000 7fdd62a00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fdd5c071c70 con 0x7fdd5c0d0970 2024-05-28T06:12:38.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:38.366+0000 7fdd62a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fdd5c06f720 con 0x7fdd5c15fa00 2024-05-28T06:12:38.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:38.366+0000 7fdd62a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fdd5c15fde0 con 0x7fdd5c1600f0 2024-05-28T06:12:38.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:38.366+0000 7fdd61600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdd5c0d0970 0x7fdd5c14f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:38.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:38.366+0000 7fdd61600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdd5c0d0970 0x7fdd5c14f440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44938/0 (socket says 172.21.15.96:44938) 2024-05-28T06:12:38.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:38.366+0000 7fdd61600640 1 -- 172.21.15.96:0/3995684917 learned_addr learned my addr 172.21.15.96:0/3995684917 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:38.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:38.367+0000 7fdd61600640 1 -- 172.21.15.96:0/3995684917 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdd5c1600f0 msgr2=0x7fdd5c09a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:38.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:38.367+0000 7fdd62000640 1 --2- 172.21.15.96:0/3995684917 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdd5c1600f0 0x7fdd5c09a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:38.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:38.367+0000 7fdd60c00640 1 --2- 172.21.15.96:0/3995684917 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdd5c15fa00 0x7fdd5c09a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:38.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:38.367+0000 7fdd61600640 1 --2- 172.21.15.96:0/3995684917 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdd5c1600f0 0x7fdd5c09a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:38.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:38.367+0000 7fdd61600640 1 -- 172.21.15.96:0/3995684917 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdd5c15fa00 msgr2=0x7fdd5c09a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:38.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:38.367+0000 7fdd61600640 1 --2- 172.21.15.96:0/3995684917 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdd5c15fa00 0x7fdd5c09a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:38.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:38.367+0000 7fdd61600640 1 -- 172.21.15.96:0/3995684917 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd5c0cec00 con 0x7fdd5c0d0970 2024-05-28T06:12:38.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:38.367+0000 7fdd62000640 1 --2- 172.21.15.96:0/3995684917 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdd5c1600f0 0x7fdd5c09a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:12:38.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:38.367+0000 7fdd60c00640 1 --2- 172.21.15.96:0/3995684917 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdd5c15fa00 0x7fdd5c09a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:12:38.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:38.367+0000 7fdd61600640 1 --2- 172.21.15.96:0/3995684917 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdd5c0d0970 0x7fdd5c14f440 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fdd50000c00 tx=0x7fdd500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:38.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:38.368+0000 7fdd59600640 1 -- 172.21.15.96:0/3995684917 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd50041020 con 0x7fdd5c0d0970 2024-05-28T06:12:38.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:38.368+0000 7fdd59600640 1 -- 172.21.15.96:0/3995684917 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd500306c0 con 0x7fdd5c0d0970 2024-05-28T06:12:38.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:38.368+0000 7fdd59600640 1 -- 172.21.15.96:0/3995684917 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd50037040 con 0x7fdd5c0d0970 2024-05-28T06:12:38.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:38.368+0000 7fdd62a00640 1 -- 172.21.15.96:0/3995684917 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd5c0ceee0 con 0x7fdd5c0d0970 2024-05-28T06:12:38.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:38.368+0000 7fdd62a00640 1 -- 172.21.15.96:0/3995684917 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd5c0cf4a0 con 0x7fdd5c0d0970 2024-05-28T06:12:38.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:38.369+0000 7fdd62a00640 1 -- 172.21.15.96:0/3995684917 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd5c0689e0 con 0x7fdd5c0d0970 2024-05-28T06:12:38.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:38.370+0000 7fdd59600640 1 -- 172.21.15.96:0/3995684917 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 16) ==== 91754+0+0 (secure 0 0 0) 0x7fdd50044020 con 0x7fdd5c0d0970 2024-05-28T06:12:38.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:38.371+0000 7fdd59600640 1 --2- 172.21.15.96:0/3995684917 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdd3c06d0d0 0x7fdd3c06f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:38.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:38.371+0000 7fdd59600640 1 -- 172.21.15.96:0/3995684917 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5737+0+0 (secure 0 0 0) 0x7fdd500ad6c0 con 0x7fdd5c0d0970 2024-05-28T06:12:38.373 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:38.372+0000 7fdd60c00640 1 --2- 172.21.15.96:0/3995684917 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdd3c06d0d0 0x7fdd3c06f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:38.374 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:38.373+0000 7fdd60c00640 1 --2- 172.21.15.96:0/3995684917 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdd3c06d0d0 0x7fdd3c06f590 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fdd48006fd0 tx=0x7fdd48008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:38.377 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:38.375+0000 7fdd59600640 1 -- 172.21.15.96:0/3995684917 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fdd5004b030 con 0x7fdd5c0d0970 2024-05-28T06:12:38.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:38.624+0000 7fdd62a00640 1 -- 172.21.15.96:0/3995684917 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) -- 0x7fdd5c0cda60 con 0x7fdd5c0d0970 2024-05-28T06:12:39.367 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:39.365+0000 7fdd59600640 1 -- 172.21.15.96:0/3995684917 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v35) ==== 136+0+0 (secure 0 0 0) 0x7fdd5007aab0 con 0x7fdd5c0d0970 2024-05-28T06:12:39.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:39.368+0000 7fdd62a00640 1 -- 172.21.15.96:0/3995684917 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdd3c06d0d0 msgr2=0x7fdd3c06f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:39.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:39.368+0000 7fdd62a00640 1 --2- 172.21.15.96:0/3995684917 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdd3c06d0d0 0x7fdd3c06f590 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fdd48006fd0 tx=0x7fdd48008040 comp rx=0 tx=0).stop 2024-05-28T06:12:39.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:39.368+0000 7fdd62a00640 1 -- 172.21.15.96:0/3995684917 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdd5c0d0970 msgr2=0x7fdd5c14f440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:39.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:39.368+0000 7fdd62a00640 1 --2- 172.21.15.96:0/3995684917 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdd5c0d0970 0x7fdd5c14f440 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fdd50000c00 tx=0x7fdd500046c0 comp rx=0 tx=0).stop 2024-05-28T06:12:39.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:39.368+0000 7fdd62a00640 1 -- 172.21.15.96:0/3995684917 shutdown_connections 2024-05-28T06:12:39.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:39.368+0000 7fdd62a00640 1 --2- 172.21.15.96:0/3995684917 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdd3c06d0d0 0x7fdd3c06f590 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:39.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:39.368+0000 7fdd62a00640 1 --2- 172.21.15.96:0/3995684917 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdd5c1600f0 0x7fdd5c09a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:39.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:39.368+0000 7fdd62a00640 1 --2- 172.21.15.96:0/3995684917 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdd5c15fa00 0x7fdd5c09a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:39.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:39.368+0000 7fdd62a00640 1 --2- 172.21.15.96:0/3995684917 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdd5c0d0970 0x7fdd5c14f440 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:39.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:39.368+0000 7fdd62a00640 1 -- 172.21.15.96:0/3995684917 >> 172.21.15.96:0/3995684917 conn(0x7fdd5c06bae0 msgr2=0x7fdd5c0b0a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:39.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:39.369+0000 7fdd62a00640 1 -- 172.21.15.96:0/3995684917 shutdown_connections 2024-05-28T06:12:39.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:39.369+0000 7fdd62a00640 1 -- 172.21.15.96:0/3995684917 wait complete. 2024-05-28T06:12:39.391 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-05-28T06:12:39.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:39.582+0000 7fc3e2e00640 1 Processor -- start 2024-05-28T06:12:39.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:39.582+0000 7fc3e2e00640 1 -- start start 2024-05-28T06:12:39.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:39.584+0000 7fc3e2e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc3dc0d04a0 0x7fc3dc0d0900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:39.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:39.584+0000 7fc3e2e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fc3dc0ad130 0x7fc3dc0ad5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:39.586 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:39.584+0000 7fc3e2e00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3dc06ed10 con 0x7fc3dc06f860 2024-05-28T06:12:39.586 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:39.584+0000 7fc3e2e00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fc3dc0d0e40 con 0x7fc3dc0ad130 2024-05-28T06:12:39.586 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:39.584+0000 7fc3e2e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fc3dc15cb00 con 0x7fc3dc0d04a0 2024-05-28T06:12:39.586 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:39.585+0000 7fc3d3400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc3dc0d04a0 0x7fc3dc0d0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:39.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:39.585+0000 7fc3d3400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc3dc0d04a0 0x7fc3dc0d0900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.96:41448/0 (socket says 172.21.15.96:41448) 2024-05-28T06:12:39.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:39.585+0000 7fc3d3400640 1 -- 172.21.15.96:0/3522112970 learned_addr learned my addr 172.21.15.96:0/3522112970 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:39.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:39.585+0000 7fc3e0e00640 1 --2- 172.21.15.96:0/3522112970 >> v2:172.21.15.123:3300/0 conn(0x7fc3dc0ad130 0x7fc3dc0ad5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:39.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:39.586+0000 7fc3d3400640 1 -- 172.21.15.96:0/3522112970 >> v2:172.21.15.123:3300/0 conn(0x7fc3dc0ad130 msgr2=0x7fc3dc0ad5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:39.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:39.586+0000 7fc3d3400640 1 --2- 172.21.15.96:0/3522112970 >> v2:172.21.15.123:3300/0 conn(0x7fc3dc0ad130 0x7fc3dc0ad5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:39.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:39.586+0000 7fc3d3400640 1 -- 172.21.15.96:0/3522112970 >> v1:172.21.15.123:6789/0 conn(0x7fc3dc06f860 legacy=0x7fc3dc0b3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:39.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:39.586+0000 7fc3d3400640 1 -- 172.21.15.96:0/3522112970 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3dc0b3cc0 con 0x7fc3dc0d04a0 2024-05-28T06:12:39.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:39.588+0000 7fc3d3400640 1 --2- 172.21.15.96:0/3522112970 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc3dc0d04a0 0x7fc3dc0d0900 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc3c4009870 tx=0x7fc3c402a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f88151ae8231b8fb server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:39.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:39.588+0000 7fc3db400640 1 -- 172.21.15.96:0/3522112970 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3c4037070 con 0x7fc3dc0d04a0 2024-05-28T06:12:39.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:39.588+0000 7fc3db400640 1 -- 172.21.15.96:0/3522112970 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3c4032440 con 0x7fc3dc0d04a0 2024-05-28T06:12:39.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:39.588+0000 7fc3db400640 1 -- 172.21.15.96:0/3522112970 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3c4032810 con 0x7fc3dc0d04a0 2024-05-28T06:12:39.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:39.589+0000 7fc3e2e00640 1 -- 172.21.15.96:0/3522112970 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc3dc0d04a0 msgr2=0x7fc3dc0d0900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:39.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:39.589+0000 7fc3e2e00640 1 --2- 172.21.15.96:0/3522112970 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc3dc0d04a0 0x7fc3dc0d0900 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc3c4009870 tx=0x7fc3c402a270 comp rx=0 tx=0).stop 2024-05-28T06:12:39.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:39.589+0000 7fc3e2e00640 1 -- 172.21.15.96:0/3522112970 shutdown_connections 2024-05-28T06:12:39.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:39.589+0000 7fc3e2e00640 1 --2- 172.21.15.96:0/3522112970 >> v2:172.21.15.123:3300/0 conn(0x7fc3dc0ad130 0x7fc3dc0ad5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:39.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:39.589+0000 7fc3e2e00640 1 --2- 172.21.15.96:0/3522112970 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc3dc0d04a0 0x7fc3dc0d0900 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:39.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:39.589+0000 7fc3e2e00640 1 -- 172.21.15.96:0/3522112970 >> 172.21.15.96:0/3522112970 conn(0x7fc3dc06bae0 msgr2=0x7fc3dc06bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:39.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:39.589+0000 7fc3e2e00640 1 -- 172.21.15.96:0/3522112970 shutdown_connections 2024-05-28T06:12:39.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:39.589+0000 7fc3e2e00640 1 -- 172.21.15.96:0/3522112970 wait complete. 2024-05-28T06:12:39.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:39.591+0000 7fc3e2e00640 1 Processor -- start 2024-05-28T06:12:39.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:39.591+0000 7fc3e2e00640 1 -- start start 2024-05-28T06:12:39.594 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:39.592+0000 7fc3e2e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3dc06f860 0x7fc3dc099a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:39.594 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:39.592+0000 7fc3e2e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc3dc0ad130 0x7fc3dc0c4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:39.594 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:39.592+0000 7fc3e2e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc3dc0d04a0 0x7fc3dc0c5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:39.594 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:39.592+0000 7fc3e2e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc3dc0b4560 con 0x7fc3dc0d04a0 2024-05-28T06:12:39.594 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:39.592+0000 7fc3e2e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc3dc0b4280 con 0x7fc3dc06f860 2024-05-28T06:12:39.594 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:39.592+0000 7fc3e2e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fc3dc0b4840 con 0x7fc3dc0ad130 2024-05-28T06:12:39.595 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:39.592+0000 7fc3e0e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc3dc0d04a0 0x7fc3dc0c5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:39.595 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:39.592+0000 7fc3e0e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc3dc0d04a0 0x7fc3dc0c5290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47100/0 (socket says 172.21.15.96:47100) 2024-05-28T06:12:39.595 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:39.592+0000 7fc3e0e00640 1 -- 172.21.15.96:0/365234512 learned_addr learned my addr 172.21.15.96:0/365234512 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:39.595 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:39.593+0000 7fc3dbe00640 1 --2- 172.21.15.96:0/365234512 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3dc06f860 0x7fc3dc099a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:39.595 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:39.593+0000 7fc3d3400640 1 --2- 172.21.15.96:0/365234512 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc3dc0ad130 0x7fc3dc0c4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:39.595 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:39.593+0000 7fc3e0e00640 1 -- 172.21.15.96:0/365234512 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc3dc0ad130 msgr2=0x7fc3dc0c4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:39.595 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:39.593+0000 7fc3e0e00640 1 --2- 172.21.15.96:0/365234512 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc3dc0ad130 0x7fc3dc0c4d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:39.595 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:39.593+0000 7fc3e0e00640 1 -- 172.21.15.96:0/365234512 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3dc06f860 msgr2=0x7fc3dc099a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:39.595 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:39.593+0000 7fc3e0e00640 1 --2- 172.21.15.96:0/365234512 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3dc06f860 0x7fc3dc099a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:39.595 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:39.593+0000 7fc3e0e00640 1 -- 172.21.15.96:0/365234512 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3dc0c76c0 con 0x7fc3dc0d04a0 2024-05-28T06:12:39.596 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:39.593+0000 7fc3e0e00640 1 --2- 172.21.15.96:0/365234512 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc3dc0d04a0 0x7fc3dc0c5290 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fc3d4009dd0 tx=0x7fc3d400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:39.596 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:39.593+0000 7fc3d8c00640 1 -- 172.21.15.96:0/365234512 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3d4018020 con 0x7fc3dc0d04a0 2024-05-28T06:12:39.596 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:39.593+0000 7fc3d8c00640 1 -- 172.21.15.96:0/365234512 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3d4014480 con 0x7fc3dc0d04a0 2024-05-28T06:12:39.596 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:39.593+0000 7fc3e2e00640 1 -- 172.21.15.96:0/365234512 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3dc0c4460 con 0x7fc3dc0d04a0 2024-05-28T06:12:39.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:39.594+0000 7fc3d8c00640 1 -- 172.21.15.96:0/365234512 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3d4014870 con 0x7fc3dc0d04a0 2024-05-28T06:12:39.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:39.594+0000 7fc3e2e00640 1 -- 172.21.15.96:0/365234512 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3dc0c4a20 con 0x7fc3dc0d04a0 2024-05-28T06:12:39.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:39.595+0000 7fc3e2e00640 1 -- 172.21.15.96:0/365234512 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3dc0689e0 con 0x7fc3dc0d04a0 2024-05-28T06:12:39.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:39.596+0000 7fc3d8c00640 1 -- 172.21.15.96:0/365234512 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 16) ==== 91754+0+0 (secure 0 0 0) 0x7fc3d4005d10 con 0x7fc3dc0d04a0 2024-05-28T06:12:39.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:39.597+0000 7fc3d8c00640 1 --2- 172.21.15.96:0/365234512 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc3b806d100 0x7fc3b806f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:39.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:39.597+0000 7fc3d8c00640 1 -- 172.21.15.96:0/365234512 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5737+0+0 (secure 0 0 0) 0x7fc3d408d6a0 con 0x7fc3dc0d04a0 2024-05-28T06:12:39.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:39.598+0000 7fc3dbe00640 1 --2- 172.21.15.96:0/365234512 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc3b806d100 0x7fc3b806f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:39.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:39.599+0000 7fc3dbe00640 1 --2- 172.21.15.96:0/365234512 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc3b806d100 0x7fc3b806f5c0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fc3cc007fd0 tx=0x7fc3cc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:39.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:39.602+0000 7fc3d8c00640 1 -- 172.21.15.96:0/365234512 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc3d4022070 con 0x7fc3dc0d04a0 2024-05-28T06:12:39.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:39.833+0000 7fc3e2e00640 1 -- 172.21.15.96:0/365234512 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7fc3dc06cc30 con 0x7fc3b806d100 2024-05-28T06:12:39.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:39.837+0000 7fc3d8c00640 1 -- 172.21.15.96:0/365234512 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7fc3dc06cc30 con 0x7fc3b806d100 2024-05-28T06:12:39.839 INFO:teuthology.orchestra.run.smithi096.stdout:[ 2024-05-28T06:12:39.839 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:12:39.839 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "cephfs" 2024-05-28T06:12:39.839 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:12:39.839 INFO:teuthology.orchestra.run.smithi096.stdout:] 2024-05-28T06:12:39.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:39.840+0000 7fc3e2e00640 1 -- 172.21.15.96:0/365234512 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc3b806d100 msgr2=0x7fc3b806f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:39.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:39.840+0000 7fc3e2e00640 1 --2- 172.21.15.96:0/365234512 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc3b806d100 0x7fc3b806f5c0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fc3cc007fd0 tx=0x7fc3cc009040 comp rx=0 tx=0).stop 2024-05-28T06:12:39.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:39.840+0000 7fc3e2e00640 1 -- 172.21.15.96:0/365234512 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc3dc0d04a0 msgr2=0x7fc3dc0c5290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:39.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:39.840+0000 7fc3e2e00640 1 --2- 172.21.15.96:0/365234512 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc3dc0d04a0 0x7fc3dc0c5290 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fc3d4009dd0 tx=0x7fc3d400c490 comp rx=0 tx=0).stop 2024-05-28T06:12:39.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:39.840+0000 7fc3e2e00640 1 -- 172.21.15.96:0/365234512 shutdown_connections 2024-05-28T06:12:39.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:39.840+0000 7fc3e2e00640 1 --2- 172.21.15.96:0/365234512 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc3b806d100 0x7fc3b806f5c0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:39.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:39.840+0000 7fc3e2e00640 1 --2- 172.21.15.96:0/365234512 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc3dc0d04a0 0x7fc3dc0c5290 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:39.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:39.840+0000 7fc3e2e00640 1 --2- 172.21.15.96:0/365234512 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc3dc0ad130 0x7fc3dc0c4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:39.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:39.840+0000 7fc3e2e00640 1 --2- 172.21.15.96:0/365234512 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3dc06f860 0x7fc3dc099a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:39.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:39.840+0000 7fc3e2e00640 1 -- 172.21.15.96:0/365234512 >> 172.21.15.96:0/365234512 conn(0x7fc3dc06bae0 msgr2=0x7fc3dc0b2cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:39.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:39.840+0000 7fc3e2e00640 1 -- 172.21.15.96:0/365234512 shutdown_connections 2024-05-28T06:12:39.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:39.840+0000 7fc3e2e00640 1 -- 172.21.15.96:0/365234512 wait complete. 2024-05-28T06:12:39.862 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-05-28T06:12:40.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.064+0000 7f2392000640 1 Processor -- start 2024-05-28T06:12:40.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.065+0000 7f2392000640 1 -- start start 2024-05-28T06:12:40.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.066+0000 7f2392000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f238c0caa40 0x7f238c0ce960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:40.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.066+0000 7f2392000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f238c0cae20 0x7f238c151db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:40.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.066+0000 7f2392000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f238c1522f0 0x7f238c0cf710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:40.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.066+0000 7f2392000640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f238c0d0b30 con 0x7f238c0cae20 2024-05-28T06:12:40.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.066+0000 7f2392000640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f238c0b4160 con 0x7f238c1522f0 2024-05-28T06:12:40.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.066+0000 7f2392000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f238c152780 con 0x7f238c0caa40 2024-05-28T06:12:40.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.067+0000 7f238be00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f238c0cae20 0x7f238c151db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:40.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.067+0000 7f238be00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f238c0cae20 0x7f238c151db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47102/0 (socket says 172.21.15.96:47102) 2024-05-28T06:12:40.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.067+0000 7f238be00640 1 -- 172.21.15.96:0/3654089821 learned_addr learned my addr 172.21.15.96:0/3654089821 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:40.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.067+0000 7f238be00640 1 -- 172.21.15.96:0/3654089821 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f238c0caa40 msgr2=0x7f238c0ce960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:40.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.067+0000 7f2390c00640 1 --2- 172.21.15.96:0/3654089821 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f238c0caa40 0x7f238c0ce960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:40.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.067+0000 7f2391600640 1 --2- 172.21.15.96:0/3654089821 >> v2:172.21.15.123:3300/0 conn(0x7f238c1522f0 0x7f238c0cf710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:40.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.067+0000 7f238be00640 1 --2- 172.21.15.96:0/3654089821 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f238c0caa40 0x7f238c0ce960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:40.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.067+0000 7f238be00640 1 -- 172.21.15.96:0/3654089821 >> v2:172.21.15.123:3300/0 conn(0x7f238c1522f0 msgr2=0x7f238c0cf710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:40.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.067+0000 7f238be00640 1 --2- 172.21.15.96:0/3654089821 >> v2:172.21.15.123:3300/0 conn(0x7f238c1522f0 0x7f238c0cf710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:40.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.067+0000 7f238be00640 1 -- 172.21.15.96:0/3654089821 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f238c0b3e80 con 0x7f238c0cae20 2024-05-28T06:12:40.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.067+0000 7f2390c00640 1 --2- 172.21.15.96:0/3654089821 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f238c0caa40 0x7f238c0ce960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:12:40.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.067+0000 7f2391600640 1 --2- 172.21.15.96:0/3654089821 >> v2:172.21.15.123:3300/0 conn(0x7f238c1522f0 0x7f238c0cf710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:12:40.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.068+0000 7f238be00640 1 --2- 172.21.15.96:0/3654089821 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f238c0cae20 0x7f238c151db0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f238c06c8e0 tx=0x7f237402a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3869d43276679e3 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:40.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.069+0000 7f238b400640 1 -- 172.21.15.96:0/3654089821 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2374036070 con 0x7f238c0cae20 2024-05-28T06:12:40.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.069+0000 7f238b400640 1 -- 172.21.15.96:0/3654089821 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2374031600 con 0x7f238c0cae20 2024-05-28T06:12:40.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.069+0000 7f238b400640 1 -- 172.21.15.96:0/3654089821 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23740319d0 con 0x7f238c0cae20 2024-05-28T06:12:40.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.069+0000 7f2392000640 1 -- 172.21.15.96:0/3654089821 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f238c0cae20 msgr2=0x7f238c151db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:40.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.069+0000 7f2392000640 1 --2- 172.21.15.96:0/3654089821 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f238c0cae20 0x7f238c151db0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f238c06c8e0 tx=0x7f237402a610 comp rx=0 tx=0).stop 2024-05-28T06:12:40.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.069+0000 7f2392000640 1 -- 172.21.15.96:0/3654089821 shutdown_connections 2024-05-28T06:12:40.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.069+0000 7f2392000640 1 --2- 172.21.15.96:0/3654089821 >> v2:172.21.15.123:3300/0 conn(0x7f238c1522f0 0x7f238c0cf710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:40.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.069+0000 7f2392000640 1 --2- 172.21.15.96:0/3654089821 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f238c0cae20 0x7f238c151db0 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:40.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.069+0000 7f2392000640 1 --2- 172.21.15.96:0/3654089821 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f238c0caa40 0x7f238c0ce960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:40.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.069+0000 7f2392000640 1 -- 172.21.15.96:0/3654089821 >> 172.21.15.96:0/3654089821 conn(0x7f238c06b900 msgr2=0x7f238c06bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:40.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.070+0000 7f2392000640 1 -- 172.21.15.96:0/3654089821 shutdown_connections 2024-05-28T06:12:40.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.070+0000 7f2392000640 1 -- 172.21.15.96:0/3654089821 wait complete. 2024-05-28T06:12:40.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.072+0000 7f2392000640 1 Processor -- start 2024-05-28T06:12:40.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.072+0000 7f2392000640 1 -- start start 2024-05-28T06:12:40.075 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.073+0000 7f2392000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f238c0caa40 0x7f238c1f7460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:40.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.073+0000 7f2392000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f238c0cae20 0x7f238c15b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:40.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.073+0000 7f2392000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f238c1522f0 0x7f238c15cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:40.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.073+0000 7f2392000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f238c0ce440 con 0x7f238c0caa40 2024-05-28T06:12:40.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.073+0000 7f2392000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f238c0ce160 con 0x7f238c1522f0 2024-05-28T06:12:40.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.073+0000 7f2392000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f238c0ceea0 con 0x7f238c0cae20 2024-05-28T06:12:40.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.073+0000 7f2390c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f238c0caa40 0x7f238c1f7460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:40.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.074+0000 7f2390c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f238c0caa40 0x7f238c1f7460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47112/0 (socket says 172.21.15.96:47112) 2024-05-28T06:12:40.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.074+0000 7f2390c00640 1 -- 172.21.15.96:0/2573951086 learned_addr learned my addr 172.21.15.96:0/2573951086 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:40.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.074+0000 7f2390c00640 1 -- 172.21.15.96:0/2573951086 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f238c0cae20 msgr2=0x7f238c15b6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:40.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.074+0000 7f238be00640 1 --2- 172.21.15.96:0/2573951086 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f238c0cae20 0x7f238c15b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:40.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.074+0000 7f2390c00640 1 --2- 172.21.15.96:0/2573951086 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f238c0cae20 0x7f238c15b6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:40.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.074+0000 7f2390c00640 1 -- 172.21.15.96:0/2573951086 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f238c1522f0 msgr2=0x7f238c15cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:40.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.074+0000 7f2391600640 1 --2- 172.21.15.96:0/2573951086 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f238c1522f0 0x7f238c15cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:40.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.074+0000 7f2390c00640 1 --2- 172.21.15.96:0/2573951086 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f238c1522f0 0x7f238c15cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:40.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.074+0000 7f2390c00640 1 -- 172.21.15.96:0/2573951086 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f238c15bf30 con 0x7f238c0caa40 2024-05-28T06:12:40.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.074+0000 7f2390c00640 1 --2- 172.21.15.96:0/2573951086 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f238c0caa40 0x7f238c1f7460 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f237c00c7f0 tx=0x7f237c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:40.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.075+0000 7f2388c00640 1 -- 172.21.15.96:0/2573951086 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f237c015070 con 0x7f238c0caa40 2024-05-28T06:12:40.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.075+0000 7f2388c00640 1 -- 172.21.15.96:0/2573951086 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f237c0077c0 con 0x7f238c0caa40 2024-05-28T06:12:40.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.075+0000 7f2388c00640 1 -- 172.21.15.96:0/2573951086 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f237c007bb0 con 0x7f238c0caa40 2024-05-28T06:12:40.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.075+0000 7f2392000640 1 -- 172.21.15.96:0/2573951086 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f238c15c210 con 0x7f238c0caa40 2024-05-28T06:12:40.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.075+0000 7f2392000640 1 -- 172.21.15.96:0/2573951086 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f238c15c7d0 con 0x7f238c0caa40 2024-05-28T06:12:40.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.076+0000 7f2392000640 1 -- 172.21.15.96:0/2573951086 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f238c0689e0 con 0x7f238c0caa40 2024-05-28T06:12:40.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.077+0000 7f2388c00640 1 -- 172.21.15.96:0/2573951086 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 16) ==== 91754+0+0 (secure 0 0 0) 0x7f237c018070 con 0x7f238c0caa40 2024-05-28T06:12:40.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.078+0000 7f2388c00640 1 --2- 172.21.15.96:0/2573951086 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f236806d0d0 0x7f236806f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:40.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.078+0000 7f2388c00640 1 -- 172.21.15.96:0/2573951086 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5737+0+0 (secure 0 0 0) 0x7f237c08d0b0 con 0x7f238c0caa40 2024-05-28T06:12:40.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.079+0000 7f238be00640 1 --2- 172.21.15.96:0/2573951086 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f236806d0d0 0x7f236806f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:40.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.080+0000 7f238be00640 1 --2- 172.21.15.96:0/2573951086 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f236806d0d0 0x7f236806f590 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f2374033000 tx=0x7f2374009000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:40.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.083+0000 7f2388c00640 1 -- 172.21.15.96:0/2573951086 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f237c05a310 con 0x7f238c0caa40 2024-05-28T06:12:40.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.282+0000 7f2392000640 1 -- 172.21.15.96:0/2573951086 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f238c0aeaa0 con 0x7f238c0caa40 2024-05-28T06:12:40.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.282+0000 7f2388c00640 1 -- 172.21.15.96:0/2573951086 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v14)=0 v14) ==== 122+0+0 (secure 0 0 0) 0x7f237c059c10 con 0x7f238c0caa40 2024-05-28T06:12:40.287 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.285+0000 7f2392000640 1 -- 172.21.15.96:0/2573951086 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f236806d0d0 msgr2=0x7f236806f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:40.287 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.285+0000 7f2392000640 1 --2- 172.21.15.96:0/2573951086 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f236806d0d0 0x7f236806f590 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f2374033000 tx=0x7f2374009000 comp rx=0 tx=0).stop 2024-05-28T06:12:40.287 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.285+0000 7f2392000640 1 -- 172.21.15.96:0/2573951086 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f238c0caa40 msgr2=0x7f238c1f7460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:40.287 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.285+0000 7f2392000640 1 --2- 172.21.15.96:0/2573951086 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f238c0caa40 0x7f238c1f7460 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f237c00c7f0 tx=0x7f237c00ccb0 comp rx=0 tx=0).stop 2024-05-28T06:12:40.287 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.285+0000 7f2392000640 1 -- 172.21.15.96:0/2573951086 shutdown_connections 2024-05-28T06:12:40.287 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.285+0000 7f2392000640 1 --2- 172.21.15.96:0/2573951086 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f236806d0d0 0x7f236806f590 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:40.287 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.285+0000 7f2392000640 1 --2- 172.21.15.96:0/2573951086 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f238c1522f0 0x7f238c15cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:40.288 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.285+0000 7f2392000640 1 --2- 172.21.15.96:0/2573951086 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f238c0cae20 0x7f238c15b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:40.288 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.285+0000 7f2392000640 1 --2- 172.21.15.96:0/2573951086 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f238c0caa40 0x7f238c1f7460 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:40.288 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.285+0000 7f2392000640 1 -- 172.21.15.96:0/2573951086 >> 172.21.15.96:0/2573951086 conn(0x7f238c06b900 msgr2=0x7f238c06e000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:40.288 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.285+0000 7f2392000640 1 -- 172.21.15.96:0/2573951086 shutdown_connections 2024-05-28T06:12:40.288 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.285+0000 7f2392000640 1 -- 172.21.15.96:0/2573951086 wait complete. 2024-05-28T06:12:40.307 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/snapshot_clone_no_wait false 2024-05-28T06:12:40.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.513+0000 7fcfafe00640 1 Processor -- start 2024-05-28T06:12:40.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.513+0000 7fcfafe00640 1 -- start start 2024-05-28T06:12:40.516 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.515+0000 7fcfafe00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcfa8156d40 0x7fcfa80b3c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:40.516 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.515+0000 7fcfafe00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fcfa80d0de0 0x7fcfa815cf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:40.517 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.515+0000 7fcfafe00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcfa80b31c0 con 0x7fcfa815caf0 2024-05-28T06:12:40.517 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.515+0000 7fcfafe00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fcfa815d490 con 0x7fcfa80d0de0 2024-05-28T06:12:40.517 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.515+0000 7fcfafe00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fcfa80ad440 con 0x7fcfa8156d40 2024-05-28T06:12:40.518 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.515+0000 7fcfade00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fcfa80d0de0 0x7fcfa815cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:40.518 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.515+0000 7fcfad400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcfa8156d40 0x7fcfa80b3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:40.518 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.515+0000 7fcfade00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fcfa80d0de0 0x7fcfa815cf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.96:51248/0 (socket says 172.21.15.96:51248) 2024-05-28T06:12:40.518 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.515+0000 7fcfad400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcfa8156d40 0x7fcfa80b3c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.96:41488/0 (socket says 172.21.15.96:41488) 2024-05-28T06:12:40.518 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.515+0000 7fcfade00640 1 -- 172.21.15.96:0/1487963124 learned_addr learned my addr 172.21.15.96:0/1487963124 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:40.518 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.516+0000 7fcfad400640 1 -- 172.21.15.96:0/1487963124 >> v2:172.21.15.123:3300/0 conn(0x7fcfa80d0de0 msgr2=0x7fcfa815cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:40.518 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.516+0000 7fcfad400640 1 --2- 172.21.15.96:0/1487963124 >> v2:172.21.15.123:3300/0 conn(0x7fcfa80d0de0 0x7fcfa815cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:40.518 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.516+0000 7fcfad400640 1 -- 172.21.15.96:0/1487963124 >> v1:172.21.15.123:6789/0 conn(0x7fcfa815caf0 legacy=0x7fcfa80d06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:40.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.516+0000 7fcfad400640 1 -- 172.21.15.96:0/1487963124 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfa815d6b0 con 0x7fcfa8156d40 2024-05-28T06:12:40.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.516+0000 7fcfade00640 1 --2- 172.21.15.96:0/1487963124 >> v2:172.21.15.123:3300/0 conn(0x7fcfa80d0de0 0x7fcfa815cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-28T06:12:40.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.518+0000 7fcfad400640 1 --2- 172.21.15.96:0/1487963124 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcfa8156d40 0x7fcfa80b3c80 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c0097a0 tx=0x7fcf9c02a0f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=45f5d48f84d62e57 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:40.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.518+0000 7fcfa7e00640 1 -- 172.21.15.96:0/1487963124 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf9c037070 con 0x7fcfa8156d40 2024-05-28T06:12:40.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.518+0000 7fcfa7e00640 1 -- 172.21.15.96:0/1487963124 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf9c02ae40 con 0x7fcfa8156d40 2024-05-28T06:12:40.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.518+0000 7fcfa7e00640 1 -- 172.21.15.96:0/1487963124 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf9c032660 con 0x7fcfa8156d40 2024-05-28T06:12:40.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.519+0000 7fcfafe00640 1 -- 172.21.15.96:0/1487963124 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcfa8156d40 msgr2=0x7fcfa80b3c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:40.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.519+0000 7fcfafe00640 1 --2- 172.21.15.96:0/1487963124 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcfa8156d40 0x7fcfa80b3c80 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c0097a0 tx=0x7fcf9c02a0f0 comp rx=0 tx=0).stop 2024-05-28T06:12:40.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.519+0000 7fcfafe00640 1 -- 172.21.15.96:0/1487963124 shutdown_connections 2024-05-28T06:12:40.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.519+0000 7fcfafe00640 1 --2- 172.21.15.96:0/1487963124 >> v2:172.21.15.123:3300/0 conn(0x7fcfa80d0de0 0x7fcfa815cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:40.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.519+0000 7fcfafe00640 1 --2- 172.21.15.96:0/1487963124 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcfa8156d40 0x7fcfa80b3c80 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:40.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.519+0000 7fcfafe00640 1 -- 172.21.15.96:0/1487963124 >> 172.21.15.96:0/1487963124 conn(0x7fcfa80b22e0 msgr2=0x7fcfa80b26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:40.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.519+0000 7fcfafe00640 1 -- 172.21.15.96:0/1487963124 shutdown_connections 2024-05-28T06:12:40.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.519+0000 7fcfafe00640 1 -- 172.21.15.96:0/1487963124 wait complete. 2024-05-28T06:12:40.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.521+0000 7fcfafe00640 1 Processor -- start 2024-05-28T06:12:40.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.521+0000 7fcfafe00640 1 -- start start 2024-05-28T06:12:40.524 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.522+0000 7fcfafe00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcfa80d0de0 0x7fcfa806f910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:40.524 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.522+0000 7fcfafe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcfa8156d40 0x7fcfa81f87b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:40.524 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.522+0000 7fcfafe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcfa815caf0 0x7fcfa814e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:40.524 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.522+0000 7fcfafe00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fcfa80b3760 con 0x7fcfa815caf0 2024-05-28T06:12:40.524 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.522+0000 7fcfafe00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fcfa80b3480 con 0x7fcfa8156d40 2024-05-28T06:12:40.524 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.522+0000 7fcfafe00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fcfa80b41c0 con 0x7fcfa80d0de0 2024-05-28T06:12:40.524 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.522+0000 7fcfade00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcfa815caf0 0x7fcfa814e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:40.524 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.522+0000 7fcfade00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcfa815caf0 0x7fcfa814e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47120/0 (socket says 172.21.15.96:47120) 2024-05-28T06:12:40.524 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.522+0000 7fcfade00640 1 -- 172.21.15.96:0/2508894327 learned_addr learned my addr 172.21.15.96:0/2508894327 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:40.525 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.523+0000 7fcfad400640 1 --2- 172.21.15.96:0/2508894327 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcfa80d0de0 0x7fcfa806f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:40.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.523+0000 7fcfaca00640 1 --2- 172.21.15.96:0/2508894327 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcfa8156d40 0x7fcfa81f87b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:40.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.523+0000 7fcfade00640 1 -- 172.21.15.96:0/2508894327 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcfa80d0de0 msgr2=0x7fcfa806f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:40.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.523+0000 7fcfade00640 1 --2- 172.21.15.96:0/2508894327 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcfa80d0de0 0x7fcfa806f910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:40.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.523+0000 7fcfade00640 1 -- 172.21.15.96:0/2508894327 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcfa8156d40 msgr2=0x7fcfa81f87b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:40.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.523+0000 7fcfade00640 1 --2- 172.21.15.96:0/2508894327 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcfa8156d40 0x7fcfa81f87b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:40.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.523+0000 7fcfade00640 1 -- 172.21.15.96:0/2508894327 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfa815b990 con 0x7fcfa815caf0 2024-05-28T06:12:40.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.523+0000 7fcfaca00640 1 --2- 172.21.15.96:0/2508894327 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcfa8156d40 0x7fcfa81f87b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:12:40.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.523+0000 7fcfade00640 1 --2- 172.21.15.96:0/2508894327 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcfa815caf0 0x7fcfa814e000 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fcfa000efc0 tx=0x7fcfa000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:40.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.523+0000 7fcfa5600640 1 -- 172.21.15.96:0/2508894327 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfa000cbe0 con 0x7fcfa815caf0 2024-05-28T06:12:40.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.523+0000 7fcfa5600640 1 -- 172.21.15.96:0/2508894327 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcfa000cd80 con 0x7fcfa815caf0 2024-05-28T06:12:40.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.523+0000 7fcfa5600640 1 -- 172.21.15.96:0/2508894327 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfa0003bd0 con 0x7fcfa815caf0 2024-05-28T06:12:40.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.523+0000 7fcfafe00640 1 -- 172.21.15.96:0/2508894327 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfa815bc70 con 0x7fcfa815caf0 2024-05-28T06:12:40.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.524+0000 7fcfafe00640 1 -- 172.21.15.96:0/2508894327 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfa814e540 con 0x7fcfa815caf0 2024-05-28T06:12:40.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.525+0000 7fcfafe00640 1 -- 172.21.15.96:0/2508894327 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcfa81584d0 con 0x7fcfa815caf0 2024-05-28T06:12:40.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.526+0000 7fcfa5600640 1 -- 172.21.15.96:0/2508894327 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 16) ==== 91754+0+0 (secure 0 0 0) 0x7fcfa001b030 con 0x7fcfa815caf0 2024-05-28T06:12:40.528 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.527+0000 7fcfa5600640 1 --2- 172.21.15.96:0/2508894327 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fcf8806d0d0 0x7fcf8806f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:40.528 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.527+0000 7fcfa5600640 1 -- 172.21.15.96:0/2508894327 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5737+0+0 (secure 0 0 0) 0x7fcfa008ce90 con 0x7fcfa815caf0 2024-05-28T06:12:40.529 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.527+0000 7fcfad400640 1 --2- 172.21.15.96:0/2508894327 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fcf8806d0d0 0x7fcf8806f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:40.530 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.529+0000 7fcfad400640 1 --2- 172.21.15.96:0/2508894327 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fcf8806d0d0 0x7fcf8806f590 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c034000 tx=0x7fcf9c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:40.533 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.531+0000 7fcfa5600640 1 -- 172.21.15.96:0/2508894327 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fcfa0092020 con 0x7fcfa815caf0 2024-05-28T06:12:40.739 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.737+0000 7fcfafe00640 1 -- 172.21.15.96:0/2508894327 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/snapshot_clone_no_wait}] v 0) -- 0x7fcfa80689e0 con 0x7fcfa815caf0 2024-05-28T06:12:40.744 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.742+0000 7fcfa5600640 1 -- 172.21.15.96:0/2508894327 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/snapshot_clone_no_wait}]=0 v15)=0 v15) ==== 136+0+0 (secure 0 0 0) 0x7fcfa0022020 con 0x7fcfa815caf0 2024-05-28T06:12:40.747 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.745+0000 7fcfafe00640 1 -- 172.21.15.96:0/2508894327 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fcf8806d0d0 msgr2=0x7fcf8806f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:40.748 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.745+0000 7fcfafe00640 1 --2- 172.21.15.96:0/2508894327 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fcf8806d0d0 0x7fcf8806f590 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c034000 tx=0x7fcf9c02e000 comp rx=0 tx=0).stop 2024-05-28T06:12:40.748 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.745+0000 7fcfafe00640 1 -- 172.21.15.96:0/2508894327 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcfa815caf0 msgr2=0x7fcfa814e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:40.748 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.745+0000 7fcfafe00640 1 --2- 172.21.15.96:0/2508894327 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcfa815caf0 0x7fcfa814e000 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fcfa000efc0 tx=0x7fcfa000c490 comp rx=0 tx=0).stop 2024-05-28T06:12:40.748 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.745+0000 7fcfafe00640 1 -- 172.21.15.96:0/2508894327 shutdown_connections 2024-05-28T06:12:40.748 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.745+0000 7fcfafe00640 1 --2- 172.21.15.96:0/2508894327 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fcf8806d0d0 0x7fcf8806f590 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:40.748 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.745+0000 7fcfafe00640 1 --2- 172.21.15.96:0/2508894327 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcfa815caf0 0x7fcfa814e000 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:40.748 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.745+0000 7fcfafe00640 1 --2- 172.21.15.96:0/2508894327 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcfa8156d40 0x7fcfa81f87b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:40.748 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.745+0000 7fcfafe00640 1 --2- 172.21.15.96:0/2508894327 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcfa80d0de0 0x7fcfa806f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:40.748 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.745+0000 7fcfafe00640 1 -- 172.21.15.96:0/2508894327 >> 172.21.15.96:0/2508894327 conn(0x7fcfa80b22e0 msgr2=0x7fcfa815ef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:40.748 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.746+0000 7fcfafe00640 1 -- 172.21.15.96:0/2508894327 shutdown_connections 2024-05-28T06:12:40.748 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.746+0000 7fcfafe00640 1 -- 172.21.15.96:0/2508894327 wait complete. 2024-05-28T06:12:40.767 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv1 --mode=777 2024-05-28T06:12:40.785 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:12:40.784+0000 7fea5d000640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:12:40.785 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:12:40.784+0000 7fea5d000640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:12:40.785 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:12:40.784+0000 7fea5d000640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:12:40.785 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:12:40.784+0000 7fea5d000640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:12:40.785 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:12:40.784+0000 7fea5d000640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:12:40.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.967+0000 7f8582a00640 1 Processor -- start 2024-05-28T06:12:40.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.968+0000 7f8582a00640 1 -- start start 2024-05-28T06:12:40.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.969+0000 7f8582a00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f857c06ef00 0x7f857c0d0320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:40.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.969+0000 7f8582a00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f857c15c7d0 con 0x7f857c0b4790 2024-05-28T06:12:40.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.969+0000 7f8582a00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f857c0ad9a0 con 0x7f857c15c3f0 2024-05-28T06:12:40.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.969+0000 7f8582a00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f857c0ad3e0 con 0x7f857c06ef00 2024-05-28T06:12:40.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.970+0000 7f8582000640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f857c0b4790 0x7f857c0b30b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:42344/0 (socket says 172.21.15.96:42344) 2024-05-28T06:12:40.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.970+0000 7f8582000640 1 -- 172.21.15.96:0/3340686386 learned_addr learned my addr 172.21.15.96:0/3340686386 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:12:40.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.970+0000 7f8580c00640 1 --2- 172.21.15.96:0/3340686386 >> v2:172.21.15.123:3300/0 conn(0x7f857c06ef00 0x7f857c0d0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:40.973 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.970+0000 7f857be00640 1 -- 172.21.15.96:0/3340686386 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1496169007 0 0) 0x7f857c15c7d0 con 0x7f857c0b4790 2024-05-28T06:12:40.973 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.970+0000 7f857be00640 1 -- 172.21.15.96:0/3340686386 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8564003520 con 0x7f857c0b4790 2024-05-28T06:12:40.973 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.971+0000 7f857be00640 1 -- 172.21.15.96:0/3340686386 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2408068233 0 0) 0x7f8564003520 con 0x7f857c0b4790 2024-05-28T06:12:40.973 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.971+0000 7f857be00640 1 -- 172.21.15.96:0/3340686386 >> v2:172.21.15.123:3300/0 conn(0x7f857c06ef00 msgr2=0x7f857c0d0320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:40.973 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.971+0000 7f857be00640 1 --2- 172.21.15.96:0/3340686386 >> v2:172.21.15.123:3300/0 conn(0x7f857c06ef00 0x7f857c0d0320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:40.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.971+0000 7f857be00640 1 -- 172.21.15.96:0/3340686386 >> v1:172.21.15.123:6789/0 conn(0x7f857c15c3f0 legacy=0x7f857c1608d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:40.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.971+0000 7f857be00640 1 -- 172.21.15.96:0/3340686386 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f857c0ad6c0 con 0x7f857c0b4790 2024-05-28T06:12:40.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.971+0000 7f857be00640 1 -- 172.21.15.96:0/3340686386 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f85740019f0 con 0x7f857c0b4790 2024-05-28T06:12:40.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.971+0000 7f857be00640 1 -- 172.21.15.96:0/3340686386 >> v1:172.21.15.96:6789/0 conn(0x7f857c0b4790 legacy=0x7f857c0b30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:40.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.971+0000 7f857be00640 1 --2- 172.21.15.96:0/3340686386 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8564003a00 0x7f8564003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:40.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.971+0000 7f857be00640 1 --2- 172.21.15.96:0/3340686386 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8564004800 0x7f8564004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:40.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.971+0000 7f857be00640 1 --2- 172.21.15.96:0/3340686386 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f85640051c0 0x7f856400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:40.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.971+0000 7f857be00640 1 -- 172.21.15.96:0/3340686386 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f857c0ad3e0 con 0x7f8564003a00 2024-05-28T06:12:40.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.971+0000 7f857be00640 1 -- 172.21.15.96:0/3340686386 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f856400c0f0 con 0x7f85640051c0 2024-05-28T06:12:40.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.971+0000 7f857be00640 1 -- 172.21.15.96:0/3340686386 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f856400c2f0 con 0x7f8564004800 2024-05-28T06:12:40.976 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.971+0000 7f8581600640 1 --2- 172.21.15.96:0/3340686386 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8564003a00 0x7f8564003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:40.976 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.971+0000 7f8582000640 1 --2- 172.21.15.96:0/3340686386 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f85640051c0 0x7f856400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:40.976 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.971+0000 7f8581600640 1 -- 172.21.15.96:0/3340686386 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8564004800 msgr2=0x7f8564004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:40.976 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.971+0000 7f8581600640 1 --2- 172.21.15.96:0/3340686386 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8564004800 0x7f8564004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:40.976 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.971+0000 7f8581600640 1 -- 172.21.15.96:0/3340686386 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f85640051c0 msgr2=0x7f856400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:40.976 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.972+0000 7f8581600640 1 --2- 172.21.15.96:0/3340686386 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f85640051c0 0x7f856400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:40.976 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.972+0000 7f8581600640 1 -- 172.21.15.96:0/3340686386 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8564003520 con 0x7f8564003a00 2024-05-28T06:12:40.976 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.972+0000 7f8582000640 1 --2- 172.21.15.96:0/3340686386 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f85640051c0 0x7f856400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:12:40.976 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.972+0000 7f8582000640 1 -- 172.21.15.96:0/3340686386 reap_dead start 2024-05-28T06:12:40.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.973+0000 7f8581600640 1 --2- 172.21.15.96:0/3340686386 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8564003a00 0x7f8564003e40 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f8570001750 tx=0x7f8564005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe5c0d9b2196587f server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:40.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.973+0000 7f857be00640 1 -- 172.21.15.96:0/3340686386 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8570001370 con 0x7f8564003a00 2024-05-28T06:12:40.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.973+0000 7f857be00640 1 -- 172.21.15.96:0/3340686386 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85700052d0 con 0x7f8564003a00 2024-05-28T06:12:40.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.973+0000 7f857be00640 1 -- 172.21.15.96:0/3340686386 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f857002abc0 con 0x7f8564003a00 2024-05-28T06:12:40.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.974+0000 7f8582a00640 1 -- 172.21.15.96:0/3340686386 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8564003a00 msgr2=0x7f8564003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:40.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.974+0000 7f8582a00640 1 --2- 172.21.15.96:0/3340686386 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8564003a00 0x7f8564003e40 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f8570001750 tx=0x7f8564005830 comp rx=0 tx=0).stop 2024-05-28T06:12:40.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.974+0000 7f8582a00640 1 -- 172.21.15.96:0/3340686386 shutdown_connections 2024-05-28T06:12:40.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.974+0000 7f8582a00640 1 --2- 172.21.15.96:0/3340686386 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8564003a00 0x7f8564003e40 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:40.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.974+0000 7f8582a00640 1 -- 172.21.15.96:0/3340686386 >> 172.21.15.96:0/3340686386 conn(0x7f857c06bae0 msgr2=0x7f857c06bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:40.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.974+0000 7f8582a00640 1 -- 172.21.15.96:0/3340686386 shutdown_connections 2024-05-28T06:12:40.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.974+0000 7f8582a00640 1 -- 172.21.15.96:0/3340686386 wait complete. 2024-05-28T06:12:40.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.976+0000 7f8582a00640 1 Processor -- start 2024-05-28T06:12:40.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.976+0000 7f8582a00640 1 -- start start 2024-05-28T06:12:40.980 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.977+0000 7f8582a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8564003a00 0x7f857c09d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:40.980 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.977+0000 7f8582a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f857c09a1a0 0x7f857c09a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:40.980 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.977+0000 7f8582a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f857c1f56d0 0x7f857c1f5b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:40.980 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.977+0000 7f8582a00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f857c0ad090 con 0x7f857c09a1a0 2024-05-28T06:12:40.980 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.977+0000 7f8582a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f857c1603b0 con 0x7f857c1f56d0 2024-05-28T06:12:40.980 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.977+0000 7f8582a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f857c1608d0 con 0x7f8564003a00 2024-05-28T06:12:40.980 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.977+0000 7f8580c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f857c09a1a0 0x7f857c09a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:40.980 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.977+0000 7f8580c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f857c09a1a0 0x7f857c09a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47138/0 (socket says 172.21.15.96:47138) 2024-05-28T06:12:40.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.977+0000 7f8580c00640 1 -- 172.21.15.96:0/1577181993 learned_addr learned my addr 172.21.15.96:0/1577181993 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:40.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.977+0000 7f8582000640 1 --2- 172.21.15.96:0/1577181993 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f857c1f56d0 0x7f857c1f5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:40.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.977+0000 7f8581600640 1 --2- 172.21.15.96:0/1577181993 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8564003a00 0x7f857c09d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:40.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.977+0000 7f8580c00640 1 -- 172.21.15.96:0/1577181993 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8564003a00 msgr2=0x7f857c09d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:40.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.977+0000 7f8580c00640 1 --2- 172.21.15.96:0/1577181993 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8564003a00 0x7f857c09d500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:40.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.977+0000 7f8580c00640 1 -- 172.21.15.96:0/1577181993 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f857c1f56d0 msgr2=0x7f857c1f5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:40.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.977+0000 7f8580c00640 1 --2- 172.21.15.96:0/1577181993 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f857c1f56d0 0x7f857c1f5b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:40.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.977+0000 7f8580c00640 1 -- 172.21.15.96:0/1577181993 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f857c09da40 con 0x7f857c09a1a0 2024-05-28T06:12:40.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.978+0000 7f8580c00640 1 --2- 172.21.15.96:0/1577181993 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f857c09a1a0 0x7f857c09a620 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f856800e7a0 tx=0x7f856800ec60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:40.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.978+0000 7f8579600640 1 -- 172.21.15.96:0/1577181993 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8568017070 con 0x7f857c09a1a0 2024-05-28T06:12:40.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.978+0000 7f8579600640 1 -- 172.21.15.96:0/1577181993 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f856800ee50 con 0x7f857c09a1a0 2024-05-28T06:12:40.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.978+0000 7f8579600640 1 -- 172.21.15.96:0/1577181993 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f856800cb20 con 0x7f857c09a1a0 2024-05-28T06:12:40.982 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.978+0000 7f8582a00640 1 -- 172.21.15.96:0/1577181993 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f857c09c480 con 0x7f857c09a1a0 2024-05-28T06:12:40.982 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.978+0000 7f8582a00640 1 -- 172.21.15.96:0/1577181993 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f857c09ca40 con 0x7f857c09a1a0 2024-05-28T06:12:40.982 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.979+0000 7f8582a00640 1 -- 172.21.15.96:0/1577181993 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f857c0addd0 con 0x7f857c09a1a0 2024-05-28T06:12:40.982 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.980+0000 7f8579600640 1 -- 172.21.15.96:0/1577181993 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 16) ==== 91754+0+0 (secure 0 0 0) 0x7f856801a070 con 0x7f857c09a1a0 2024-05-28T06:12:40.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.982+0000 7f8579600640 1 --2- 172.21.15.96:0/1577181993 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f855c06d100 0x7f855c06f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:40.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.982+0000 7f8579600640 1 -- 172.21.15.96:0/1577181993 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5737+0+0 (secure 0 0 0) 0x7f856808a560 con 0x7f857c09a1a0 2024-05-28T06:12:40.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.982+0000 7f8581600640 1 --2- 172.21.15.96:0/1577181993 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f855c06d100 0x7f855c06f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:40.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.983+0000 7f8581600640 1 --2- 172.21.15.96:0/1577181993 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f855c06d100 0x7f855c06f5c0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f8570032000 tx=0x7f857002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:40.988 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:40.986+0000 7f8579600640 1 -- 172.21.15.96:0/1577181993 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8568092020 con 0x7f857c09a1a0 2024-05-28T06:12:41.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.213+0000 7f8582a00640 1 -- 172.21.15.96:0/1577181993 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv1", "mode": "777", "target": ["mon-mgr", ""]}) -- 0x7f857c068a30 con 0x7f855c06d100 2024-05-28T06:12:41.228 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:12:41.227+0000 7fea59e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:12:41.228 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:12:41.227+0000 7fea59e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:12:41.229 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:12:41.227+0000 7fea59e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:12:41.229 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:12:41.227+0000 7fea59e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:12:41.229 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:12:41.227+0000 7fea59e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:12:41.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.310+0000 7f8579600640 1 -- 172.21.15.96:0/1577181993 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f8570005250 con 0x7f855c06d100 2024-05-28T06:12:41.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.314+0000 7f8582a00640 1 -- 172.21.15.96:0/1577181993 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f855c06d100 msgr2=0x7f855c06f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:41.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.314+0000 7f8582a00640 1 --2- 172.21.15.96:0/1577181993 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f855c06d100 0x7f855c06f5c0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f8570032000 tx=0x7f857002e000 comp rx=0 tx=0).stop 2024-05-28T06:12:41.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.314+0000 7f8582a00640 1 -- 172.21.15.96:0/1577181993 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f857c09a1a0 msgr2=0x7f857c09a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:41.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.314+0000 7f8582a00640 1 --2- 172.21.15.96:0/1577181993 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f857c09a1a0 0x7f857c09a620 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f856800e7a0 tx=0x7f856800ec60 comp rx=0 tx=0).stop 2024-05-28T06:12:41.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.314+0000 7f8582a00640 1 -- 172.21.15.96:0/1577181993 shutdown_connections 2024-05-28T06:12:41.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.314+0000 7f8582a00640 1 --2- 172.21.15.96:0/1577181993 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f855c06d100 0x7f855c06f5c0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:41.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.314+0000 7f8582a00640 1 --2- 172.21.15.96:0/1577181993 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f857c1f56d0 0x7f857c1f5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:41.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.314+0000 7f8582a00640 1 --2- 172.21.15.96:0/1577181993 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f857c09a1a0 0x7f857c09a620 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:41.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.314+0000 7f8582a00640 1 --2- 172.21.15.96:0/1577181993 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8564003a00 0x7f857c09d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:41.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.314+0000 7f8582a00640 1 -- 172.21.15.96:0/1577181993 >> 172.21.15.96:0/1577181993 conn(0x7f857c06bae0 msgr2=0x7f857c0b2910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:41.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.314+0000 7f8582a00640 1 -- 172.21.15.96:0/1577181993 shutdown_connections 2024-05-28T06:12:41.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.314+0000 7f8582a00640 1 -- 172.21.15.96:0/1577181993 wait complete. 2024-05-28T06:12:41.338 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-05-28T06:12:41.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.536+0000 7f1f5fe00640 1 Processor -- start 2024-05-28T06:12:41.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.536+0000 7f1f5fe00640 1 -- start start 2024-05-28T06:12:41.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.538+0000 7f1f5fe00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1f58162ac0 0x7f1f580d0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:41.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.538+0000 7f1f5fe00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f1f580b4120 0x7f1f580d0b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:41.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.538+0000 7f1f5fe00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f581640e0 con 0x7f1f581633b0 2024-05-28T06:12:41.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.538+0000 7f1f5fe00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f1f580b3c90 con 0x7f1f580b4120 2024-05-28T06:12:41.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.538+0000 7f1f5fe00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f1f580b3360 con 0x7f1f58162ac0 2024-05-28T06:12:41.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.538+0000 7f1f5de00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f1f581633b0 0x7f1f58163820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:42354/0 (socket says 172.21.15.96:42354) 2024-05-28T06:12:41.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.538+0000 7f1f5de00640 1 -- 172.21.15.96:0/3020230241 learned_addr learned my addr 172.21.15.96:0/3020230241 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:12:41.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.539+0000 7f1f5ca00640 1 --2- 172.21.15.96:0/3020230241 >> v2:172.21.15.123:3300/0 conn(0x7f1f580b4120 0x7f1f580d0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:41.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.539+0000 7f1f5d400640 1 --2- 172.21.15.96:0/3020230241 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1f58162ac0 0x7f1f580d0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:41.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.539+0000 7f1f57e00640 1 -- 172.21.15.96:0/3020230241 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1087446675 0 0) 0x7f1f581640e0 con 0x7f1f581633b0 2024-05-28T06:12:41.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.539+0000 7f1f57e00640 1 -- 172.21.15.96:0/3020230241 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1f40003520 con 0x7f1f581633b0 2024-05-28T06:12:41.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.539+0000 7f1f57e00640 1 -- 172.21.15.96:0/3020230241 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3044905264 0 0) 0x7f1f40003520 con 0x7f1f581633b0 2024-05-28T06:12:41.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.539+0000 7f1f57e00640 1 -- 172.21.15.96:0/3020230241 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1f58162ac0 msgr2=0x7f1f580d0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:41.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.539+0000 7f1f57e00640 1 --2- 172.21.15.96:0/3020230241 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1f58162ac0 0x7f1f580d0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:41.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.539+0000 7f1f57e00640 1 -- 172.21.15.96:0/3020230241 >> v2:172.21.15.123:3300/0 conn(0x7f1f580b4120 msgr2=0x7f1f580d0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:41.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.539+0000 7f1f57e00640 1 --2- 172.21.15.96:0/3020230241 >> v2:172.21.15.123:3300/0 conn(0x7f1f580b4120 0x7f1f580d0b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:41.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.539+0000 7f1f57e00640 1 -- 172.21.15.96:0/3020230241 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f580b3640 con 0x7f1f581633b0 2024-05-28T06:12:41.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.539+0000 7f1f57e00640 1 -- 172.21.15.96:0/3020230241 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f1f50001670 con 0x7f1f581633b0 2024-05-28T06:12:41.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.539+0000 7f1f57e00640 1 -- 172.21.15.96:0/3020230241 >> v1:172.21.15.96:6789/0 conn(0x7f1f581633b0 legacy=0x7f1f58163820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:41.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.540+0000 7f1f57e00640 1 --2- 172.21.15.96:0/3020230241 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1f400039b0 0x7f1f40003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:41.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.540+0000 7f1f57e00640 1 --2- 172.21.15.96:0/3020230241 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1f40004800 0x7f1f40004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:41.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.540+0000 7f1f57e00640 1 --2- 172.21.15.96:0/3020230241 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1f58162ac0 0x7f1f400051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:41.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.540+0000 7f1f57e00640 1 -- 172.21.15.96:0/3020230241 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f1f580b3c90 con 0x7f1f40004800 2024-05-28T06:12:41.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.540+0000 7f1f57e00640 1 -- 172.21.15.96:0/3020230241 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f1f580b3360 con 0x7f1f400039b0 2024-05-28T06:12:41.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.540+0000 7f1f57e00640 1 -- 172.21.15.96:0/3020230241 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f1f4000b9c0 con 0x7f1f58162ac0 2024-05-28T06:12:41.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.540+0000 7f1f5ca00640 1 --2- 172.21.15.96:0/3020230241 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1f40004800 0x7f1f40004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:41.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.540+0000 7f1f5ca00640 1 -- 172.21.15.96:0/3020230241 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1f58162ac0 msgr2=0x7f1f400051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:41.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.540+0000 7f1f5ca00640 1 --2- 172.21.15.96:0/3020230241 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1f58162ac0 0x7f1f400051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:41.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.540+0000 7f1f5ca00640 1 -- 172.21.15.96:0/3020230241 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1f400039b0 msgr2=0x7f1f40003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:12:41.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.540+0000 7f1f5ca00640 1 --2- 172.21.15.96:0/3020230241 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1f400039b0 0x7f1f40003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:41.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.540+0000 7f1f5ca00640 1 -- 172.21.15.96:0/3020230241 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f40003520 con 0x7f1f40004800 2024-05-28T06:12:41.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.541+0000 7f1f5ca00640 1 --2- 172.21.15.96:0/3020230241 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1f40004800 0x7f1f40004c80 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f1f44009ac0 tx=0x7f1f40005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b8d05de80c5e1e9 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:41.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.542+0000 7f1f57e00640 1 -- 172.21.15.96:0/3020230241 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f44036070 con 0x7f1f40004800 2024-05-28T06:12:41.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.542+0000 7f1f57e00640 1 -- 172.21.15.96:0/3020230241 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f44031400 con 0x7f1f40004800 2024-05-28T06:12:41.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.542+0000 7f1f57e00640 1 -- 172.21.15.96:0/3020230241 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f44009690 con 0x7f1f40004800 2024-05-28T06:12:41.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.542+0000 7f1f5fe00640 1 -- 172.21.15.96:0/3020230241 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1f40004800 msgr2=0x7f1f40004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:41.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.542+0000 7f1f5fe00640 1 --2- 172.21.15.96:0/3020230241 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1f40004800 0x7f1f40004c80 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f1f44009ac0 tx=0x7f1f40005830 comp rx=0 tx=0).stop 2024-05-28T06:12:41.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.542+0000 7f1f5de00640 1 -- 172.21.15.96:0/3020230241 reap_dead start 2024-05-28T06:12:41.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.542+0000 7f1f5fe00640 1 -- 172.21.15.96:0/3020230241 shutdown_connections 2024-05-28T06:12:41.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.542+0000 7f1f5fe00640 1 -- 172.21.15.96:0/3020230241 >> 172.21.15.96:0/3020230241 conn(0x7f1f5806bd20 msgr2=0x7f1f5806c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:41.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.542+0000 7f1f5fe00640 1 -- 172.21.15.96:0/3020230241 shutdown_connections 2024-05-28T06:12:41.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.542+0000 7f1f5fe00640 1 -- 172.21.15.96:0/3020230241 wait complete. 2024-05-28T06:12:41.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.544+0000 7f1f5fe00640 1 Processor -- start 2024-05-28T06:12:41.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.545+0000 7f1f5fe00640 1 -- start start 2024-05-28T06:12:41.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.545+0000 7f1f5fe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1f40004800 0x7f1f5815ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:41.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.545+0000 7f1f5fe00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1f580b1ad0 0x7f1f5815f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:41.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.545+0000 7f1f5fe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1f581fb910 0x7f1f581f8a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:41.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.546+0000 7f1f5fe00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f1f58163020 con 0x7f1f40004800 2024-05-28T06:12:41.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.546+0000 7f1f5fe00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f1f58162ea0 con 0x7f1f581fb910 2024-05-28T06:12:41.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.546+0000 7f1f5fe00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f1f580d0640 con 0x7f1f580b1ad0 2024-05-28T06:12:41.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.546+0000 7f1f5d400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1f40004800 0x7f1f5815ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:41.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.546+0000 7f1f5d400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1f40004800 0x7f1f5815ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47150/0 (socket says 172.21.15.96:47150) 2024-05-28T06:12:41.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.546+0000 7f1f5d400640 1 -- 172.21.15.96:0/251018088 learned_addr learned my addr 172.21.15.96:0/251018088 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:12:41.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.546+0000 7f1f5ca00640 1 --2- 172.21.15.96:0/251018088 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1f580b1ad0 0x7f1f5815f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:41.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.546+0000 7f1f5de00640 1 --2- 172.21.15.96:0/251018088 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1f581fb910 0x7f1f581f8a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:41.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.546+0000 7f1f5d400640 1 -- 172.21.15.96:0/251018088 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1f580b1ad0 msgr2=0x7f1f5815f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:41.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.546+0000 7f1f5d400640 1 --2- 172.21.15.96:0/251018088 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1f580b1ad0 0x7f1f5815f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:41.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.546+0000 7f1f5d400640 1 -- 172.21.15.96:0/251018088 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1f581fb910 msgr2=0x7f1f581f8a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:41.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.546+0000 7f1f5d400640 1 --2- 172.21.15.96:0/251018088 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1f581fb910 0x7f1f581f8a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:41.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.546+0000 7f1f5d400640 1 -- 172.21.15.96:0/251018088 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f5815d080 con 0x7f1f40004800 2024-05-28T06:12:41.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.546+0000 7f1f5de00640 1 --2- 172.21.15.96:0/251018088 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1f581fb910 0x7f1f581f8a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:12:41.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.547+0000 7f1f5d400640 1 --2- 172.21.15.96:0/251018088 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1f40004800 0x7f1f5815ef90 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f1f4c00ef40 tx=0x7f1f4c00b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:41.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.547+0000 7f1f55600640 1 -- 172.21.15.96:0/251018088 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f4c018020 con 0x7f1f40004800 2024-05-28T06:12:41.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.547+0000 7f1f55600640 1 -- 172.21.15.96:0/251018088 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f4c014400 con 0x7f1f40004800 2024-05-28T06:12:41.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.547+0000 7f1f55600640 1 -- 172.21.15.96:0/251018088 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f4c0147f0 con 0x7f1f40004800 2024-05-28T06:12:41.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.547+0000 7f1f5fe00640 1 -- 172.21.15.96:0/251018088 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f580ce260 con 0x7f1f40004800 2024-05-28T06:12:41.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.547+0000 7f1f5fe00640 1 -- 172.21.15.96:0/251018088 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f580ce820 con 0x7f1f40004800 2024-05-28T06:12:41.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.548+0000 7f1f5fe00640 1 -- 172.21.15.96:0/251018088 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f580689e0 con 0x7f1f40004800 2024-05-28T06:12:41.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.549+0000 7f1f55600640 1 -- 172.21.15.96:0/251018088 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 16) ==== 91754+0+0 (secure 0 0 0) 0x7f1f4c005d10 con 0x7f1f40004800 2024-05-28T06:12:41.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.550+0000 7f1f55600640 1 --2- 172.21.15.96:0/251018088 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1f3806d100 0x7f1f3806f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:12:41.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.550+0000 7f1f55600640 1 -- 172.21.15.96:0/251018088 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5737+0+0 (secure 0 0 0) 0x7f1f4c08d6c0 con 0x7f1f40004800 2024-05-28T06:12:41.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.551+0000 7f1f5ca00640 1 --2- 172.21.15.96:0/251018088 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1f3806d100 0x7f1f3806f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:12:41.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.552+0000 7f1f5ca00640 1 --2- 172.21.15.96:0/251018088 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1f3806d100 0x7f1f3806f5c0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f1f44033000 tx=0x7f1f44008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:12:41.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.555+0000 7f1f55600640 1 -- 172.21.15.96:0/251018088 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1f4c022030 con 0x7f1f40004800 2024-05-28T06:12:41.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.787+0000 7f1f5fe00640 1 -- 172.21.15.96:0/251018088 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) -- 0x7f1f580b12f0 con 0x7f1f3806d100 2024-05-28T06:12:41.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.802+0000 7f1f55600640 1 -- 172.21.15.96:0/251018088 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+59 (secure 0 0 0) 0x7f1f580b12f0 con 0x7f1f3806d100 2024-05-28T06:12:41.803 INFO:teuthology.orchestra.run.smithi096.stdout:/volumes/_nogroup/sv1/a9a192ad-d1f9-4019-bd8b-1d4f4917cd9a 2024-05-28T06:12:41.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.804+0000 7f1f5fe00640 1 -- 172.21.15.96:0/251018088 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1f3806d100 msgr2=0x7f1f3806f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:41.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.804+0000 7f1f5fe00640 1 --2- 172.21.15.96:0/251018088 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1f3806d100 0x7f1f3806f5c0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f1f44033000 tx=0x7f1f44008030 comp rx=0 tx=0).stop 2024-05-28T06:12:41.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.804+0000 7f1f5fe00640 1 -- 172.21.15.96:0/251018088 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1f40004800 msgr2=0x7f1f5815ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:12:41.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.804+0000 7f1f5fe00640 1 --2- 172.21.15.96:0/251018088 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1f40004800 0x7f1f5815ef90 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f1f4c00ef40 tx=0x7f1f4c00b480 comp rx=0 tx=0).stop 2024-05-28T06:12:41.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.804+0000 7f1f5fe00640 1 -- 172.21.15.96:0/251018088 shutdown_connections 2024-05-28T06:12:41.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.804+0000 7f1f5fe00640 1 --2- 172.21.15.96:0/251018088 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1f3806d100 0x7f1f3806f5c0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:41.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.804+0000 7f1f5fe00640 1 --2- 172.21.15.96:0/251018088 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1f581fb910 0x7f1f581f8a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:41.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.804+0000 7f1f5fe00640 1 --2- 172.21.15.96:0/251018088 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1f580b1ad0 0x7f1f5815f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:41.807 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.804+0000 7f1f5fe00640 1 --2- 172.21.15.96:0/251018088 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1f40004800 0x7f1f5815ef90 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:12:41.807 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.804+0000 7f1f5fe00640 1 -- 172.21.15.96:0/251018088 >> 172.21.15.96:0/251018088 conn(0x7f1f5806bd20 msgr2=0x7f1f58160830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:12:41.807 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.804+0000 7f1f5fe00640 1 -- 172.21.15.96:0/251018088 shutdown_connections 2024-05-28T06:12:41.807 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:12:41.805+0000 7f1f5fe00640 1 -- 172.21.15.96:0/251018088 wait complete. 2024-05-28T06:12:41.823 DEBUG:tasks.cephfs.test_volumes:filling subvolume sv1 with 3 files each 1024MB size under directory volumes/_nogroup/sv1/a9a192ad-d1f9-4019-bd8b-1d4f4917cd9a 2024-05-28T06:12:41.824 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:12:41.824 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:12:41.875 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:12:41.876 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/a9a192ad-d1f9-4019-bd8b-1d4f4917cd9a/subvolume_file.0 bs=1M conv=fdatasync count=1024 seek=0) 2024-05-28T06:12:55.371 INFO:teuthology.orchestra.run.smithi096.stderr:1024+0 records in 2024-05-28T06:12:55.371 INFO:teuthology.orchestra.run.smithi096.stderr:1024+0 records out 2024-05-28T06:12:55.371 INFO:teuthology.orchestra.run.smithi096.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 13.3744 s, 80.3 MB/s 2024-05-28T06:12:55.372 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:12:55.373 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:12:55.426 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:12:55.426 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/a9a192ad-d1f9-4019-bd8b-1d4f4917cd9a/subvolume_file.1 bs=1M conv=fdatasync count=1024 seek=0) 2024-05-28T06:13:01.280 DEBUG:teuthology.orchestra.run.smithi096:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:13:01.289 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:13:07.247 INFO:teuthology.orchestra.run.smithi096.stderr:1024+0 records in 2024-05-28T06:13:07.247 INFO:teuthology.orchestra.run.smithi096.stderr:1024+0 records out 2024-05-28T06:13:07.248 INFO:teuthology.orchestra.run.smithi096.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 11.7655 s, 91.3 MB/s 2024-05-28T06:13:07.249 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:13:07.249 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:13:07.304 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:13:07.304 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/a9a192ad-d1f9-4019-bd8b-1d4f4917cd9a/subvolume_file.2 bs=1M conv=fdatasync count=1024 seek=0) 2024-05-28T06:13:18.867 INFO:teuthology.orchestra.run.smithi096.stderr:1024+0 records in 2024-05-28T06:13:18.868 INFO:teuthology.orchestra.run.smithi096.stderr:1024+0 records out 2024-05-28T06:13:18.868 INFO:teuthology.orchestra.run.smithi096.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 11.5063 s, 93.3 MB/s 2024-05-28T06:13:18.869 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot create cephfs sv1 ss1 2024-05-28T06:13:19.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.075+0000 7ff12e200640 1 Processor -- start 2024-05-28T06:13:19.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.075+0000 7ff12e200640 1 -- start start 2024-05-28T06:13:19.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.077+0000 7ff12e200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff12815fa00 0x7ff12806e970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:19.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.077+0000 7ff12e200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff12806fa00 con 0x7ff1281600f0 2024-05-28T06:13:19.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.077+0000 7ff12e200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff12806eeb0 con 0x7ff1280d0b40 2024-05-28T06:13:19.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.077+0000 7ff12e200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ff12806f160 con 0x7ff12815fa00 2024-05-28T06:13:19.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.077+0000 7ff127e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7ff1281600f0 0x7ff1280d0430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:47940/0 (socket says 172.21.15.96:47940) 2024-05-28T06:13:19.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.077+0000 7ff127e00640 1 -- 172.21.15.96:0/1084777635 learned_addr learned my addr 172.21.15.96:0/1084777635 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:13:19.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.078+0000 7ff127400640 1 -- 172.21.15.96:0/1084777635 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3759118616 0 0) 0x7ff12806fa00 con 0x7ff1281600f0 2024-05-28T06:13:19.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.078+0000 7ff12ce00640 1 --2- 172.21.15.96:0/1084777635 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff12815fa00 0x7ff12806e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:19.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.078+0000 7ff127400640 1 -- 172.21.15.96:0/1084777635 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff110003520 con 0x7ff1281600f0 2024-05-28T06:13:19.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.078+0000 7ff127400640 1 -- 172.21.15.96:0/1084777635 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 29979107 0 0) 0x7ff110003520 con 0x7ff1281600f0 2024-05-28T06:13:19.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.078+0000 7ff127400640 1 -- 172.21.15.96:0/1084777635 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff12815fa00 msgr2=0x7ff12806e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:13:19.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.078+0000 7ff127400640 1 --2- 172.21.15.96:0/1084777635 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff12815fa00 0x7ff12806e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:19.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.078+0000 7ff127400640 1 -- 172.21.15.96:0/1084777635 >> v1:172.21.15.123:6789/0 conn(0x7ff1280d0b40 legacy=0x7ff128072190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:19.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.078+0000 7ff127400640 1 -- 172.21.15.96:0/1084777635 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff12806f440 con 0x7ff1281600f0 2024-05-28T06:13:19.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.078+0000 7ff127400640 1 -- 172.21.15.96:0/1084777635 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7ff114001690 con 0x7ff1281600f0 2024-05-28T06:13:19.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.078+0000 7ff127400640 1 -- 172.21.15.96:0/1084777635 >> v1:172.21.15.96:6789/0 conn(0x7ff1281600f0 legacy=0x7ff1280d0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:19.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.078+0000 7ff127400640 1 --2- 172.21.15.96:0/1084777635 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff1100039e0 0x7ff110003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:19.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.078+0000 7ff127400640 1 --2- 172.21.15.96:0/1084777635 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff110004800 0x7ff110004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:19.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.078+0000 7ff127400640 1 --2- 172.21.15.96:0/1084777635 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff12815fa00 0x7ff11000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:19.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.078+0000 7ff127400640 1 -- 172.21.15.96:0/1084777635 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff12806f160 con 0x7ff110004800 2024-05-28T06:13:19.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.078+0000 7ff127400640 1 -- 172.21.15.96:0/1084777635 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff110005600 con 0x7ff1100039e0 2024-05-28T06:13:19.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.078+0000 7ff127400640 1 -- 172.21.15.96:0/1084777635 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ff11000bf00 con 0x7ff12815fa00 2024-05-28T06:13:19.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.078+0000 7ff127e00640 1 --2- 172.21.15.96:0/1084777635 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff110004800 0x7ff110004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:19.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.079+0000 7ff12ce00640 1 --2- 172.21.15.96:0/1084777635 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff1100039e0 0x7ff110003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:19.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.079+0000 7ff127e00640 1 -- 172.21.15.96:0/1084777635 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff12815fa00 msgr2=0x7ff11000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:13:19.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.079+0000 7ff127e00640 1 --2- 172.21.15.96:0/1084777635 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff12815fa00 0x7ff11000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:19.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.079+0000 7ff127e00640 1 -- 172.21.15.96:0/1084777635 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff1100039e0 msgr2=0x7ff110003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:13:19.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.079+0000 7ff127e00640 1 --2- 172.21.15.96:0/1084777635 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff1100039e0 0x7ff110003e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:19.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.079+0000 7ff127e00640 1 -- 172.21.15.96:0/1084777635 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff110003520 con 0x7ff110004800 2024-05-28T06:13:19.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.080+0000 7ff127e00640 1 --2- 172.21.15.96:0/1084777635 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff110004800 0x7ff110004c80 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7ff11400c190 tx=0x7ff110005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9de68c3a1e68e5ec server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:13:19.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.080+0000 7ff127400640 1 -- 172.21.15.96:0/1084777635 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff114007c60 con 0x7ff110004800 2024-05-28T06:13:19.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.080+0000 7ff127400640 1 -- 172.21.15.96:0/1084777635 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff114005ad0 con 0x7ff110004800 2024-05-28T06:13:19.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.080+0000 7ff127400640 1 -- 172.21.15.96:0/1084777635 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff114007070 con 0x7ff110004800 2024-05-28T06:13:19.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.081+0000 7ff12e200640 1 -- 172.21.15.96:0/1084777635 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff110004800 msgr2=0x7ff110004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:19.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.081+0000 7ff12e200640 1 --2- 172.21.15.96:0/1084777635 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff110004800 0x7ff110004c80 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7ff11400c190 tx=0x7ff110005830 comp rx=0 tx=0).stop 2024-05-28T06:13:19.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.081+0000 7ff12d800640 1 -- 172.21.15.96:0/1084777635 reap_dead start 2024-05-28T06:13:19.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.081+0000 7ff12e200640 1 -- 172.21.15.96:0/1084777635 shutdown_connections 2024-05-28T06:13:19.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.081+0000 7ff12e200640 1 -- 172.21.15.96:0/1084777635 >> 172.21.15.96:0/1084777635 conn(0x7ff12806bae0 msgr2=0x7ff12806bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:13:19.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.081+0000 7ff12e200640 1 -- 172.21.15.96:0/1084777635 shutdown_connections 2024-05-28T06:13:19.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.081+0000 7ff12e200640 1 -- 172.21.15.96:0/1084777635 wait complete. 2024-05-28T06:13:19.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.083+0000 7ff12e200640 1 Processor -- start 2024-05-28T06:13:19.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.083+0000 7ff12e200640 1 -- start start 2024-05-28T06:13:19.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.084+0000 7ff12e200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff110004800 0x7ff1280b1340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:19.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.084+0000 7ff12e200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff1280afe10 0x7ff128153790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:19.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.084+0000 7ff12e200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff1280b26e0 0x7ff1281e8000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:19.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.084+0000 7ff12e200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff12815fde0 con 0x7ff1280afe10 2024-05-28T06:13:19.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.084+0000 7ff12e200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff128071c70 con 0x7ff110004800 2024-05-28T06:13:19.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.084+0000 7ff12e200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ff12806e970 con 0x7ff1280b26e0 2024-05-28T06:13:19.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.084+0000 7ff127e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff1280afe10 0x7ff128153790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:19.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.084+0000 7ff127e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff1280afe10 0x7ff128153790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58934/0 (socket says 172.21.15.96:58934) 2024-05-28T06:13:19.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.084+0000 7ff127e00640 1 -- 172.21.15.96:0/4016887968 learned_addr learned my addr 172.21.15.96:0/4016887968 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:13:19.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.085+0000 7ff12ce00640 1 --2- 172.21.15.96:0/4016887968 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff110004800 0x7ff1280b1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:19.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.085+0000 7ff127e00640 1 -- 172.21.15.96:0/4016887968 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff1280b26e0 msgr2=0x7ff1281e8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:19.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.085+0000 7ff127e00640 1 --2- 172.21.15.96:0/4016887968 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff1280b26e0 0x7ff1281e8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:19.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.085+0000 7ff127e00640 1 -- 172.21.15.96:0/4016887968 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff110004800 msgr2=0x7ff1280b1340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:19.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.085+0000 7ff127e00640 1 --2- 172.21.15.96:0/4016887968 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff110004800 0x7ff1280b1340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:19.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.085+0000 7ff127e00640 1 -- 172.21.15.96:0/4016887968 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1280b02a0 con 0x7ff1280afe10 2024-05-28T06:13:19.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.085+0000 7ff127e00640 1 --2- 172.21.15.96:0/4016887968 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff1280afe10 0x7ff128153790 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7ff11400c2b0 tx=0x7ff12806dfe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:13:19.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.085+0000 7ff124c00640 1 -- 172.21.15.96:0/4016887968 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff114040020 con 0x7ff1280afe10 2024-05-28T06:13:19.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.085+0000 7ff124c00640 1 -- 172.21.15.96:0/4016887968 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1140013d0 con 0x7ff1280afe10 2024-05-28T06:13:19.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.085+0000 7ff124c00640 1 -- 172.21.15.96:0/4016887968 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff114037040 con 0x7ff1280afe10 2024-05-28T06:13:19.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.085+0000 7ff12e200640 1 -- 172.21.15.96:0/4016887968 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1280b0580 con 0x7ff1280afe10 2024-05-28T06:13:19.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.085+0000 7ff12e200640 1 -- 172.21.15.96:0/4016887968 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1280b0b40 con 0x7ff1280afe10 2024-05-28T06:13:19.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.087+0000 7ff12e200640 1 -- 172.21.15.96:0/4016887968 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1280689e0 con 0x7ff1280afe10 2024-05-28T06:13:19.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.088+0000 7ff124c00640 1 -- 172.21.15.96:0/4016887968 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 17) ==== 91856+0+0 (secure 0 0 0) 0x7ff114043020 con 0x7ff1280afe10 2024-05-28T06:13:19.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.088+0000 7ff124c00640 1 --2- 172.21.15.96:0/4016887968 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff10806d1f0 0x7ff10806f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:19.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.089+0000 7ff124c00640 1 -- 172.21.15.96:0/4016887968 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5737+0+0 (secure 0 0 0) 0x7ff1140a97f0 con 0x7ff1280afe10 2024-05-28T06:13:19.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.089+0000 7ff12ce00640 1 --2- 172.21.15.96:0/4016887968 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff10806d1f0 0x7ff10806f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:19.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.090+0000 7ff12ce00640 1 --2- 172.21.15.96:0/4016887968 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff10806d1f0 0x7ff10806f6b0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7ff1280b2670 tx=0x7ff11c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:13:19.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.093+0000 7ff124c00640 1 -- 172.21.15.96:0/4016887968 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff1140796d0 con 0x7ff1280afe10 2024-05-28T06:13:19.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.325+0000 7ff12e200640 1 -- 172.21.15.96:0/4016887968 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""]}) -- 0x7ff12806e200 con 0x7ff10806d1f0 2024-05-28T06:13:19.358 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.356+0000 7ff124c00640 1 -- 172.21.15.96:0/4016887968 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7ff12806e200 con 0x7ff10806d1f0 2024-05-28T06:13:19.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.359+0000 7ff12e200640 1 -- 172.21.15.96:0/4016887968 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff10806d1f0 msgr2=0x7ff10806f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:19.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.359+0000 7ff12e200640 1 --2- 172.21.15.96:0/4016887968 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff10806d1f0 0x7ff10806f6b0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7ff1280b2670 tx=0x7ff11c008040 comp rx=0 tx=0).stop 2024-05-28T06:13:19.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.359+0000 7ff12e200640 1 -- 172.21.15.96:0/4016887968 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff1280afe10 msgr2=0x7ff128153790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:19.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.359+0000 7ff12e200640 1 --2- 172.21.15.96:0/4016887968 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff1280afe10 0x7ff128153790 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7ff11400c2b0 tx=0x7ff12806dfe0 comp rx=0 tx=0).stop 2024-05-28T06:13:19.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.359+0000 7ff12e200640 1 -- 172.21.15.96:0/4016887968 shutdown_connections 2024-05-28T06:13:19.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.359+0000 7ff12e200640 1 --2- 172.21.15.96:0/4016887968 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff10806d1f0 0x7ff10806f6b0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:19.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.359+0000 7ff12e200640 1 --2- 172.21.15.96:0/4016887968 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff1280b26e0 0x7ff1281e8000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:19.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.359+0000 7ff12e200640 1 --2- 172.21.15.96:0/4016887968 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff1280afe10 0x7ff128153790 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:19.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.359+0000 7ff12e200640 1 --2- 172.21.15.96:0/4016887968 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff110004800 0x7ff1280b1340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:19.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.359+0000 7ff12e200640 1 -- 172.21.15.96:0/4016887968 >> 172.21.15.96:0/4016887968 conn(0x7ff12806bae0 msgr2=0x7ff12815e740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:13:19.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.360+0000 7ff12e200640 1 -- 172.21.15.96:0/4016887968 shutdown_connections 2024-05-28T06:13:19.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.360+0000 7ff12e200640 1 -- 172.21.15.96:0/4016887968 wait complete. 2024-05-28T06:13:19.380 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-05-28T06:13:19.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.581+0000 7f0f33000640 1 Processor -- start 2024-05-28T06:13:19.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.581+0000 7f0f33000640 1 -- start start 2024-05-28T06:13:19.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.583+0000 7f0f33000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0f2c0cd2a0 0x7f0f2c0b4d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:19.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.583+0000 7f0f33000640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f2c15d900 con 0x7f0f2c15cb60 2024-05-28T06:13:19.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.583+0000 7f0f33000640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f2c0b3f90 con 0x7f0f2c153510 2024-05-28T06:13:19.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.583+0000 7f0f33000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f0f2c15d6e0 con 0x7f0f2c0cd2a0 2024-05-28T06:13:19.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.583+0000 7f0f32600640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f0f2c15cb60 0x7f0f2c15cfd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:47944/0 (socket says 172.21.15.96:47944) 2024-05-28T06:13:19.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.583+0000 7f0f32600640 1 -- 172.21.15.96:0/2509529533 learned_addr learned my addr 172.21.15.96:0/2509529533 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:13:19.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.584+0000 7f0f31200640 1 --2- 172.21.15.96:0/2509529533 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0f2c0cd2a0 0x7f0f2c0b4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:19.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.584+0000 7f0f2be00640 1 -- 172.21.15.96:0/2509529533 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3147359799 0 0) 0x7f0f2c15d900 con 0x7f0f2c15cb60 2024-05-28T06:13:19.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.584+0000 7f0f2be00640 1 -- 172.21.15.96:0/2509529533 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0f14003520 con 0x7f0f2c15cb60 2024-05-28T06:13:19.586 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.584+0000 7f0f2be00640 1 -- 172.21.15.96:0/2509529533 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3054091310 0 0) 0x7f0f14003520 con 0x7f0f2c15cb60 2024-05-28T06:13:19.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.584+0000 7f0f2be00640 1 -- 172.21.15.96:0/2509529533 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0f2c0cd2a0 msgr2=0x7f0f2c0b4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:13:19.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.584+0000 7f0f2be00640 1 --2- 172.21.15.96:0/2509529533 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0f2c0cd2a0 0x7f0f2c0b4d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:19.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.584+0000 7f0f2be00640 1 -- 172.21.15.96:0/2509529533 >> v1:172.21.15.123:6789/0 conn(0x7f0f2c153510 legacy=0x7f0f2c0ccb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:19.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.584+0000 7f0f2be00640 1 -- 172.21.15.96:0/2509529533 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f2c0b4270 con 0x7f0f2c15cb60 2024-05-28T06:13:19.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.584+0000 7f0f2be00640 1 -- 172.21.15.96:0/2509529533 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f0f240019f0 con 0x7f0f2c15cb60 2024-05-28T06:13:19.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.584+0000 7f0f2be00640 1 -- 172.21.15.96:0/2509529533 >> v1:172.21.15.96:6789/0 conn(0x7f0f2c15cb60 legacy=0x7f0f2c15cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:19.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.584+0000 7f0f2be00640 1 --2- 172.21.15.96:0/2509529533 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0f140039e0 0x7f0f14003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:19.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.584+0000 7f0f2be00640 1 --2- 172.21.15.96:0/2509529533 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0f2c0cd2a0 0x7f0f14004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:19.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.584+0000 7f0f2be00640 1 --2- 172.21.15.96:0/2509529533 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0f14004d40 0x7f0f140051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:19.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.584+0000 7f0f2be00640 1 -- 172.21.15.96:0/2509529533 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f0f2c15d6e0 con 0x7f0f140039e0 2024-05-28T06:13:19.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.584+0000 7f0f2be00640 1 -- 172.21.15.96:0/2509529533 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f0f1400bb50 con 0x7f0f14004d40 2024-05-28T06:13:19.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.584+0000 7f0f2be00640 1 -- 172.21.15.96:0/2509529533 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f0f1400bd50 con 0x7f0f2c0cd2a0 2024-05-28T06:13:19.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.584+0000 7f0f31c00640 1 --2- 172.21.15.96:0/2509529533 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0f140039e0 0x7f0f14003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:19.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.585+0000 7f0f32600640 1 --2- 172.21.15.96:0/2509529533 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0f14004d40 0x7f0f140051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:19.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.585+0000 7f0f31c00640 1 -- 172.21.15.96:0/2509529533 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0f2c0cd2a0 msgr2=0x7f0f14004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:13:19.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.585+0000 7f0f31c00640 1 --2- 172.21.15.96:0/2509529533 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0f2c0cd2a0 0x7f0f14004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:19.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.585+0000 7f0f31c00640 1 -- 172.21.15.96:0/2509529533 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0f14004d40 msgr2=0x7f0f140051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:13:19.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.585+0000 7f0f31c00640 1 --2- 172.21.15.96:0/2509529533 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0f14004d40 0x7f0f140051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:19.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.585+0000 7f0f31c00640 1 -- 172.21.15.96:0/2509529533 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f14003520 con 0x7f0f140039e0 2024-05-28T06:13:19.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.586+0000 7f0f31c00640 1 --2- 172.21.15.96:0/2509529533 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0f140039e0 0x7f0f14003e00 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f0f20001750 tx=0x7f0f14005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8719d3ffc4f55c7a server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:13:19.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.586+0000 7f0f2be00640 1 -- 172.21.15.96:0/2509529533 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f20001370 con 0x7f0f140039e0 2024-05-28T06:13:19.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.586+0000 7f0f2be00640 1 -- 172.21.15.96:0/2509529533 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f200052d0 con 0x7f0f140039e0 2024-05-28T06:13:19.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.586+0000 7f0f2be00640 1 -- 172.21.15.96:0/2509529533 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f2002abe0 con 0x7f0f140039e0 2024-05-28T06:13:19.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.587+0000 7f0f33000640 1 -- 172.21.15.96:0/2509529533 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0f140039e0 msgr2=0x7f0f14003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:19.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.587+0000 7f0f33000640 1 --2- 172.21.15.96:0/2509529533 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0f140039e0 0x7f0f14003e00 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f0f20001750 tx=0x7f0f14005830 comp rx=0 tx=0).stop 2024-05-28T06:13:19.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.587+0000 7f0f32600640 1 -- 172.21.15.96:0/2509529533 reap_dead start 2024-05-28T06:13:19.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.587+0000 7f0f33000640 1 -- 172.21.15.96:0/2509529533 shutdown_connections 2024-05-28T06:13:19.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.587+0000 7f0f33000640 1 -- 172.21.15.96:0/2509529533 >> 172.21.15.96:0/2509529533 conn(0x7f0f2c06b900 msgr2=0x7f0f2c06bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:13:19.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.587+0000 7f0f33000640 1 -- 172.21.15.96:0/2509529533 shutdown_connections 2024-05-28T06:13:19.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.587+0000 7f0f33000640 1 -- 172.21.15.96:0/2509529533 wait complete. 2024-05-28T06:13:19.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.589+0000 7f0f33000640 1 Processor -- start 2024-05-28T06:13:19.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.589+0000 7f0f33000640 1 -- start start 2024-05-28T06:13:19.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.590+0000 7f0f33000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0f140039e0 0x7f0f2c073680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:19.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.590+0000 7f0f33000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0f2c1f76e0 0x7f0f2c1f7df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:19.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.590+0000 7f0f33000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0f2c142b30 0x7f0f2c142fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:19.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.590+0000 7f0f33000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f0f2c0b48c0 con 0x7f0f2c142b30 2024-05-28T06:13:19.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.590+0000 7f0f33000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f0f2c0b4d50 con 0x7f0f140039e0 2024-05-28T06:13:19.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.590+0000 7f0f33000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f0f2c0b4ed0 con 0x7f0f2c1f76e0 2024-05-28T06:13:19.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.590+0000 7f0f32600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0f2c142b30 0x7f0f2c142fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:19.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.590+0000 7f0f32600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0f2c142b30 0x7f0f2c142fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58942/0 (socket says 172.21.15.96:58942) 2024-05-28T06:13:19.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.590+0000 7f0f32600640 1 -- 172.21.15.96:0/1684406824 learned_addr learned my addr 172.21.15.96:0/1684406824 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:13:19.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.591+0000 7f0f32600640 1 -- 172.21.15.96:0/1684406824 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0f2c1f76e0 msgr2=0x7f0f2c1f7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:19.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.591+0000 7f0f32600640 1 --2- 172.21.15.96:0/1684406824 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0f2c1f76e0 0x7f0f2c1f7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:19.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.591+0000 7f0f32600640 1 -- 172.21.15.96:0/1684406824 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0f140039e0 msgr2=0x7f0f2c073680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:19.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.591+0000 7f0f32600640 1 --2- 172.21.15.96:0/1684406824 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0f140039e0 0x7f0f2c073680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:19.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.591+0000 7f0f32600640 1 -- 172.21.15.96:0/1684406824 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f2c1f7b60 con 0x7f0f2c142b30 2024-05-28T06:13:19.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.591+0000 7f0f32600640 1 --2- 172.21.15.96:0/1684406824 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0f2c142b30 0x7f0f2c142fe0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f0f2c15cf40 tx=0x7f0f2400b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:13:19.594 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.591+0000 7f0f29600640 1 -- 172.21.15.96:0/1684406824 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f24004d60 con 0x7f0f2c142b30 2024-05-28T06:13:19.594 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.591+0000 7f0f29600640 1 -- 172.21.15.96:0/1684406824 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f2400bb80 con 0x7f0f2c142b30 2024-05-28T06:13:19.594 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.591+0000 7f0f33000640 1 -- 172.21.15.96:0/1684406824 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f2c1f8330 con 0x7f0f2c142b30 2024-05-28T06:13:19.594 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.591+0000 7f0f29600640 1 -- 172.21.15.96:0/1684406824 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f24007070 con 0x7f0f2c142b30 2024-05-28T06:13:19.594 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.591+0000 7f0f33000640 1 -- 172.21.15.96:0/1684406824 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f2c1f88e0 con 0x7f0f2c142b30 2024-05-28T06:13:19.594 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.593+0000 7f0f33000640 1 -- 172.21.15.96:0/1684406824 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f2c0689e0 con 0x7f0f2c142b30 2024-05-28T06:13:19.595 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.594+0000 7f0f29600640 1 -- 172.21.15.96:0/1684406824 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 17) ==== 91856+0+0 (secure 0 0 0) 0x7f0f2401b020 con 0x7f0f2c142b30 2024-05-28T06:13:19.596 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.595+0000 7f0f29600640 1 --2- 172.21.15.96:0/1684406824 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0f0c06d220 0x7f0f0c06f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:19.596 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.595+0000 7f0f29600640 1 -- 172.21.15.96:0/1684406824 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5737+0+0 (secure 0 0 0) 0x7f0f2408a5a0 con 0x7f0f2c142b30 2024-05-28T06:13:19.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.595+0000 7f0f31c00640 1 --2- 172.21.15.96:0/1684406824 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0f0c06d220 0x7f0f0c06f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:19.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.597+0000 7f0f31c00640 1 --2- 172.21.15.96:0/1684406824 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0f0c06d220 0x7f0f0c06f6e0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f0f20032000 tx=0x7f0f2002c000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:13:19.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.600+0000 7f0f29600640 1 -- 172.21.15.96:0/1684406824 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0f2405a4e0 con 0x7f0f2c142b30 2024-05-28T06:13:19.830 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.829+0000 7f0f33000640 1 -- 172.21.15.96:0/1684406824 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) -- 0x7f0f2c1520d0 con 0x7f0f0c06d220 2024-05-28T06:13:19.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.841+0000 7f0f29600640 1 -- 172.21.15.96:0/1684406824 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+59 (secure 0 0 0) 0x7f0f2c1520d0 con 0x7f0f0c06d220 2024-05-28T06:13:19.842 INFO:teuthology.orchestra.run.smithi096.stdout:/volumes/_nogroup/sv1/a9a192ad-d1f9-4019-bd8b-1d4f4917cd9a 2024-05-28T06:13:19.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.843+0000 7f0f33000640 1 -- 172.21.15.96:0/1684406824 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0f0c06d220 msgr2=0x7f0f0c06f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:19.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.843+0000 7f0f33000640 1 --2- 172.21.15.96:0/1684406824 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0f0c06d220 0x7f0f0c06f6e0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f0f20032000 tx=0x7f0f2002c000 comp rx=0 tx=0).stop 2024-05-28T06:13:19.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.843+0000 7f0f33000640 1 -- 172.21.15.96:0/1684406824 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0f2c142b30 msgr2=0x7f0f2c142fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:19.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.843+0000 7f0f33000640 1 --2- 172.21.15.96:0/1684406824 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0f2c142b30 0x7f0f2c142fe0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f0f2c15cf40 tx=0x7f0f2400b4a0 comp rx=0 tx=0).stop 2024-05-28T06:13:19.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.843+0000 7f0f33000640 1 -- 172.21.15.96:0/1684406824 shutdown_connections 2024-05-28T06:13:19.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.843+0000 7f0f33000640 1 --2- 172.21.15.96:0/1684406824 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0f0c06d220 0x7f0f0c06f6e0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:19.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.843+0000 7f0f33000640 1 --2- 172.21.15.96:0/1684406824 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0f2c142b30 0x7f0f2c142fe0 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:19.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.843+0000 7f0f33000640 1 --2- 172.21.15.96:0/1684406824 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0f2c1f76e0 0x7f0f2c1f7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:19.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.843+0000 7f0f33000640 1 --2- 172.21.15.96:0/1684406824 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0f140039e0 0x7f0f2c073680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:19.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.843+0000 7f0f33000640 1 -- 172.21.15.96:0/1684406824 >> 172.21.15.96:0/1684406824 conn(0x7f0f2c06b900 msgr2=0x7f0f2c0cb630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:13:19.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.844+0000 7f0f33000640 1 -- 172.21.15.96:0/1684406824 shutdown_connections 2024-05-28T06:13:19.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:19.844+0000 7f0f33000640 1 -- 172.21.15.96:0/1684406824 wait complete. 2024-05-28T06:13:19.865 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:13:19.865 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/a9a192ad-d1f9-4019-bd8b-1d4f4917cd9a 2024-05-28T06:13:19.865 DEBUG:teuthology.orchestra.run.smithi096:> ) 2024-05-28T06:13:19.925 INFO:teuthology.orchestra.run.smithi096.stdout:# file: volumes/_nogroup/sv1/a9a192ad-d1f9-4019-bd8b-1d4f4917cd9a 2024-05-28T06:13:19.925 INFO:teuthology.orchestra.run.smithi096.stdout:ceph.dir.rbytes="3138256896" 2024-05-28T06:13:19.925 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:13:19.926 INFO:tasks.cephfs.test_volumes:r_size = 3138256896 exp_size = 3221225472 2024-05-28T06:13:21.927 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:13:21.927 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/a9a192ad-d1f9-4019-bd8b-1d4f4917cd9a 2024-05-28T06:13:21.927 DEBUG:teuthology.orchestra.run.smithi096:> ) 2024-05-28T06:13:21.994 INFO:teuthology.orchestra.run.smithi096.stdout:# file: volumes/_nogroup/sv1/a9a192ad-d1f9-4019-bd8b-1d4f4917cd9a 2024-05-28T06:13:21.994 INFO:teuthology.orchestra.run.smithi096.stdout:ceph.dir.rbytes="3138256896" 2024-05-28T06:13:21.994 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:13:21.995 INFO:tasks.cephfs.test_volumes:r_size = 3138256896 exp_size = 3221225472 2024-05-28T06:13:23.995 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:13:23.996 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/a9a192ad-d1f9-4019-bd8b-1d4f4917cd9a 2024-05-28T06:13:23.996 DEBUG:teuthology.orchestra.run.smithi096:> ) 2024-05-28T06:13:24.059 INFO:teuthology.orchestra.run.smithi096.stdout:# file: volumes/_nogroup/sv1/a9a192ad-d1f9-4019-bd8b-1d4f4917cd9a 2024-05-28T06:13:24.059 INFO:teuthology.orchestra.run.smithi096.stdout:ceph.dir.rbytes="3221225472" 2024-05-28T06:13:24.059 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:13:24.060 INFO:tasks.cephfs.test_volumes:r_size = 3221225472 exp_size = 3221225472 2024-05-28T06:13:24.060 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone81711 2024-05-28T06:13:24.266 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.264+0000 7f1ade800640 1 Processor -- start 2024-05-28T06:13:24.266 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.264+0000 7f1ade800640 1 -- start start 2024-05-28T06:13:24.268 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.266+0000 7f1ade800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f1ad8156bd0 0x7f1ad80b3af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:24.268 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.266+0000 7f1ade800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f1ad80b4e30 0x7f1ad815caf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:24.268 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.266+0000 7f1ade800640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ad8156fb0 con 0x7f1ad815d030 2024-05-28T06:13:24.268 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.266+0000 7f1ade800640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f1ad815dae0 con 0x7f1ad80b4e30 2024-05-28T06:13:24.268 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.266+0000 7f1ade800640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f1ad815d520 con 0x7f1ad8156bd0 2024-05-28T06:13:24.268 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.266+0000 7f1acfe00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f1ad80b4e30 0x7f1ad815caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:24.268 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.266+0000 7f1ad7e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f1ad815d030 0x7f1ad80d06d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:47958/0 (socket says 172.21.15.96:47958) 2024-05-28T06:13:24.268 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.266+0000 7f1ad7e00640 1 -- 172.21.15.96:0/2901553389 learned_addr learned my addr 172.21.15.96:0/2901553389 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:13:24.269 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.267+0000 7f1ad7400640 1 --2- 172.21.15.96:0/2901553389 >> v2:172.21.15.123:3300/0 conn(0x7f1ad8156bd0 0x7f1ad80b3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:24.269 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.267+0000 7f1acfe00640 1 -- 172.21.15.96:0/2901553389 >> v2:172.21.15.123:3300/0 conn(0x7f1ad8156bd0 msgr2=0x7f1ad80b3af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:13:24.269 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.267+0000 7f1acfe00640 1 --2- 172.21.15.96:0/2901553389 >> v2:172.21.15.123:3300/0 conn(0x7f1ad8156bd0 0x7f1ad80b3af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:24.269 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.267+0000 7f1acfe00640 1 -- 172.21.15.96:0/2901553389 >> v1:172.21.15.96:6789/0 conn(0x7f1ad815d030 legacy=0x7f1ad80d06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:24.269 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.267+0000 7f1acfe00640 1 -- 172.21.15.96:0/2901553389 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ad815d800 con 0x7f1ad80b4e30 2024-05-28T06:13:24.270 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.268+0000 7f1acfe00640 1 --2- 172.21.15.96:0/2901553389 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1ad80b4e30 0x7f1ad815caf0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f1ad815dc60 tx=0x7f1ac002a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32f0f3c48664bbda server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:13:24.270 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.268+0000 7f1ad6a00640 1 -- 172.21.15.96:0/2901553389 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ac0037070 con 0x7f1ad80b4e30 2024-05-28T06:13:24.270 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.268+0000 7f1ad6a00640 1 -- 172.21.15.96:0/2901553389 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ac0032440 con 0x7f1ad80b4e30 2024-05-28T06:13:24.270 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.268+0000 7f1ad6a00640 1 -- 172.21.15.96:0/2901553389 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ac002ae60 con 0x7f1ad80b4e30 2024-05-28T06:13:24.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.269+0000 7f1ade800640 1 -- 172.21.15.96:0/2901553389 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1ad80b4e30 msgr2=0x7f1ad815caf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:24.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.269+0000 7f1ade800640 1 --2- 172.21.15.96:0/2901553389 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1ad80b4e30 0x7f1ad815caf0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f1ad815dc60 tx=0x7f1ac002a110 comp rx=0 tx=0).stop 2024-05-28T06:13:24.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.269+0000 7f1ade800640 1 -- 172.21.15.96:0/2901553389 shutdown_connections 2024-05-28T06:13:24.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.269+0000 7f1ade800640 1 --2- 172.21.15.96:0/2901553389 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1ad80b4e30 0x7f1ad815caf0 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:24.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.269+0000 7f1ade800640 1 --2- 172.21.15.96:0/2901553389 >> v2:172.21.15.123:3300/0 conn(0x7f1ad8156bd0 0x7f1ad80b3af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:24.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.269+0000 7f1ade800640 1 -- 172.21.15.96:0/2901553389 >> 172.21.15.96:0/2901553389 conn(0x7f1ad806bd20 msgr2=0x7f1ad806c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:13:24.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.269+0000 7f1ade800640 1 -- 172.21.15.96:0/2901553389 shutdown_connections 2024-05-28T06:13:24.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.269+0000 7f1ade800640 1 -- 172.21.15.96:0/2901553389 wait complete. 2024-05-28T06:13:24.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.271+0000 7f1ade800640 1 Processor -- start 2024-05-28T06:13:24.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.271+0000 7f1ade800640 1 -- start start 2024-05-28T06:13:24.274 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.272+0000 7f1ade800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ad80b4e30 0x7f1ad81571b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:24.274 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.272+0000 7f1ade800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1ad8156bd0 0x7f1ad81f40a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:24.274 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.272+0000 7f1ade800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1ad815d030 0x7f1ad80cb710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:24.274 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.272+0000 7f1ade800640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f1ad80b4030 con 0x7f1ad8156bd0 2024-05-28T06:13:24.274 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.272+0000 7f1ade800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f1ad80b35d0 con 0x7f1ad80b4e30 2024-05-28T06:13:24.274 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.272+0000 7f1ade800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f1ad80b41b0 con 0x7f1ad815d030 2024-05-28T06:13:24.274 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.272+0000 7f1acfe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1ad8156bd0 0x7f1ad81f40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:24.274 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.272+0000 7f1acfe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1ad8156bd0 0x7f1ad81f40a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58956/0 (socket says 172.21.15.96:58956) 2024-05-28T06:13:24.274 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.272+0000 7f1acfe00640 1 -- 172.21.15.96:0/3730547165 learned_addr learned my addr 172.21.15.96:0/3730547165 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:13:24.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.272+0000 7f1acfe00640 1 -- 172.21.15.96:0/3730547165 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1ad815d030 msgr2=0x7f1ad80cb710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:24.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.272+0000 7f1acfe00640 1 --2- 172.21.15.96:0/3730547165 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1ad815d030 0x7f1ad80cb710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:24.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.272+0000 7f1acfe00640 1 -- 172.21.15.96:0/3730547165 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ad80b4e30 msgr2=0x7f1ad81571b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:24.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.272+0000 7f1acfe00640 1 --2- 172.21.15.96:0/3730547165 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ad80b4e30 0x7f1ad81571b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:24.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.272+0000 7f1acfe00640 1 -- 172.21.15.96:0/3730547165 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ad8157890 con 0x7f1ad8156bd0 2024-05-28T06:13:24.276 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.273+0000 7f1acfe00640 1 --2- 172.21.15.96:0/3730547165 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1ad8156bd0 0x7f1ad81f40a0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f1ac0000c00 tx=0x7f1ac00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:13:24.276 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.273+0000 7f1acf400640 1 -- 172.21.15.96:0/3730547165 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ac0037040 con 0x7f1ad8156bd0 2024-05-28T06:13:24.276 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.273+0000 7f1acf400640 1 -- 172.21.15.96:0/3730547165 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ac0042070 con 0x7f1ad8156bd0 2024-05-28T06:13:24.276 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.273+0000 7f1acf400640 1 -- 172.21.15.96:0/3730547165 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ac0030bd0 con 0x7f1ad8156bd0 2024-05-28T06:13:24.276 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.273+0000 7f1ade800640 1 -- 172.21.15.96:0/3730547165 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ad8157b70 con 0x7f1ad8156bd0 2024-05-28T06:13:24.276 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.273+0000 7f1ade800640 1 -- 172.21.15.96:0/3730547165 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ad8158130 con 0x7f1ad8156bd0 2024-05-28T06:13:24.276 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.274+0000 7f1ade800640 1 -- 172.21.15.96:0/3730547165 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ad80689e0 con 0x7f1ad8156bd0 2024-05-28T06:13:24.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.275+0000 7f1acf400640 1 -- 172.21.15.96:0/3730547165 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 17) ==== 91856+0+0 (secure 0 0 0) 0x7f1ac0047460 con 0x7f1ad8156bd0 2024-05-28T06:13:24.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.276+0000 7f1acf400640 1 --2- 172.21.15.96:0/3730547165 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1ab406d1f0 0x7f1ab406f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:24.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.276+0000 7f1acf400640 1 -- 172.21.15.96:0/3730547165 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5737+0+0 (secure 0 0 0) 0x7f1ac00ab240 con 0x7f1ad8156bd0 2024-05-28T06:13:24.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.277+0000 7f1ad7400640 1 --2- 172.21.15.96:0/3730547165 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1ab406d1f0 0x7f1ab406f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:24.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.278+0000 7f1ad7400640 1 --2- 172.21.15.96:0/3730547165 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1ab406d1f0 0x7f1ab406f6b0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f1ac80098f0 tx=0x7f1ac8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:13:24.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.281+0000 7f1acf400640 1 -- 172.21.15.96:0/3730547165 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1ac0032e20 con 0x7f1ad8156bd0 2024-05-28T06:13:24.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.525+0000 7f1ade800640 1 -- 172.21.15.96:0/3730547165 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone81711", "target": ["mon-mgr", ""]}) -- 0x7f1ad80b1180 con 0x7f1ab406d1f0 2024-05-28T06:13:24.689 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.687+0000 7f1acf400640 1 -- 172.21.15.96:0/3730547165 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f1ad80b1180 con 0x7f1ab406d1f0 2024-05-28T06:13:24.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.690+0000 7f1ade800640 1 -- 172.21.15.96:0/3730547165 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1ab406d1f0 msgr2=0x7f1ab406f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:24.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.690+0000 7f1ade800640 1 --2- 172.21.15.96:0/3730547165 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1ab406d1f0 0x7f1ab406f6b0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f1ac80098f0 tx=0x7f1ac8008040 comp rx=0 tx=0).stop 2024-05-28T06:13:24.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.690+0000 7f1ade800640 1 -- 172.21.15.96:0/3730547165 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1ad8156bd0 msgr2=0x7f1ad81f40a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:24.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.690+0000 7f1ade800640 1 --2- 172.21.15.96:0/3730547165 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1ad8156bd0 0x7f1ad81f40a0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f1ac0000c00 tx=0x7f1ac00046c0 comp rx=0 tx=0).stop 2024-05-28T06:13:24.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.690+0000 7f1ade800640 1 -- 172.21.15.96:0/3730547165 shutdown_connections 2024-05-28T06:13:24.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.690+0000 7f1ade800640 1 --2- 172.21.15.96:0/3730547165 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1ab406d1f0 0x7f1ab406f6b0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:24.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.690+0000 7f1ade800640 1 --2- 172.21.15.96:0/3730547165 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1ad815d030 0x7f1ad80cb710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:24.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.690+0000 7f1ade800640 1 --2- 172.21.15.96:0/3730547165 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1ad8156bd0 0x7f1ad81f40a0 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:24.693 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.690+0000 7f1ade800640 1 --2- 172.21.15.96:0/3730547165 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ad80b4e30 0x7f1ad81571b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:24.693 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.690+0000 7f1ade800640 1 -- 172.21.15.96:0/3730547165 >> 172.21.15.96:0/3730547165 conn(0x7f1ad806bd20 msgr2=0x7f1ad815a710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:13:24.693 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.690+0000 7f1ade800640 1 -- 172.21.15.96:0/3730547165 shutdown_connections 2024-05-28T06:13:24.693 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.691+0000 7f1ade800640 1 -- 172.21.15.96:0/3730547165 wait complete. 2024-05-28T06:13:24.693 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:13:24.693+0000 7fea60c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:13:24.693 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:13:24.693+0000 7fea60c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:13:24.694 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:13:24.693+0000 7fea60c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:13:24.694 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:13:24.693+0000 7fea60c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:13:24.694 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:13:24.693+0000 7fea60c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:13:24.713 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone81712 2024-05-28T06:13:24.723 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:13:24.723+0000 7fea5f800640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:13:24.723 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:13:24.723+0000 7fea5f800640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:13:24.723 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:13:24.723+0000 7fea5f800640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:13:24.723 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:13:24.723+0000 7fea5f800640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:13:24.724 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:13:24.723+0000 7fea5f800640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:13:24.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.914+0000 7f0f02200640 1 Processor -- start 2024-05-28T06:13:24.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.914+0000 7f0f02200640 1 -- start start 2024-05-28T06:13:24.917 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.915+0000 7f0f02200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f0efc0af330 0x7f0efc073ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:24.917 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.915+0000 7f0f02200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f0efc15d3b0 0x7f0efc071f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:24.917 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.915+0000 7f0f02200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0efc15a2e0 con 0x7f0efc159f00 2024-05-28T06:13:24.917 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.915+0000 7f0f02200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f0efc15d7e0 con 0x7f0efc15d3b0 2024-05-28T06:13:24.917 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.915+0000 7f0f02200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f0efc0ad230 con 0x7f0efc0af330 2024-05-28T06:13:24.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.916+0000 7f0f01800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f0efc15d3b0 0x7f0efc071f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:24.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.916+0000 7f0f01800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f0efc15d3b0 0x7f0efc071f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58968/0 (socket says 172.21.15.96:58968) 2024-05-28T06:13:24.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.916+0000 7f0f01800640 1 -- 172.21.15.96:0/430259679 learned_addr learned my addr 172.21.15.96:0/430259679 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:13:24.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.916+0000 7f0f00e00640 1 --1- 172.21.15.96:0/430259679 >> v1:172.21.15.96:6789/0 conn(0x7f0efc159f00 0x7f0efc0aec20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:47972/0 (socket says 172.21.15.96:47972) 2024-05-28T06:13:24.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.916+0000 7f0ef3e00640 1 --2- 172.21.15.96:0/430259679 >> v2:172.21.15.123:3300/0 conn(0x7f0efc0af330 0x7f0efc073ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:24.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.916+0000 7f0f01800640 1 -- 172.21.15.96:0/430259679 >> v2:172.21.15.123:3300/0 conn(0x7f0efc0af330 msgr2=0x7f0efc073ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:13:24.919 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.916+0000 7f0f01800640 1 --2- 172.21.15.96:0/430259679 >> v2:172.21.15.123:3300/0 conn(0x7f0efc0af330 0x7f0efc073ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:24.919 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.916+0000 7f0f01800640 1 -- 172.21.15.96:0/430259679 >> v1:172.21.15.96:6789/0 conn(0x7f0efc159f00 legacy=0x7f0efc0aec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:24.919 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.916+0000 7f0f01800640 1 -- 172.21.15.96:0/430259679 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0efc0ad510 con 0x7f0efc15d3b0 2024-05-28T06:13:24.919 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.916+0000 7f0efbe00640 1 -- 172.21.15.96:0/430259679 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 874165442 0 0) 0x7f0eec001110 con 0x7f0efc159f00 2024-05-28T06:13:24.920 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.917+0000 7f0f01800640 1 --2- 172.21.15.96:0/430259679 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0efc15d3b0 0x7f0efc071f00 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f0efc0714d0 tx=0x7f0ef402a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=946fd4eca24af5aa server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:13:24.920 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.917+0000 7f0efbe00640 1 -- 172.21.15.96:0/430259679 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ef4036070 con 0x7f0efc15d3b0 2024-05-28T06:13:24.920 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.918+0000 7f0efbe00640 1 -- 172.21.15.96:0/430259679 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ef4031600 con 0x7f0efc15d3b0 2024-05-28T06:13:24.920 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.918+0000 7f0efbe00640 1 -- 172.21.15.96:0/430259679 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ef40319d0 con 0x7f0efc15d3b0 2024-05-28T06:13:24.920 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.918+0000 7f0f02200640 1 -- 172.21.15.96:0/430259679 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0efc15d3b0 msgr2=0x7f0efc071f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:24.920 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.918+0000 7f0f02200640 1 --2- 172.21.15.96:0/430259679 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0efc15d3b0 0x7f0efc071f00 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f0efc0714d0 tx=0x7f0ef402a6a0 comp rx=0 tx=0).stop 2024-05-28T06:13:24.920 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.918+0000 7f0f02200640 1 -- 172.21.15.96:0/430259679 shutdown_connections 2024-05-28T06:13:24.920 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.918+0000 7f0f02200640 1 --2- 172.21.15.96:0/430259679 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0efc15d3b0 0x7f0efc071f00 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:24.920 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.918+0000 7f0f02200640 1 --2- 172.21.15.96:0/430259679 >> v2:172.21.15.123:3300/0 conn(0x7f0efc0af330 0x7f0efc073ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:24.921 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.918+0000 7f0f02200640 1 -- 172.21.15.96:0/430259679 >> 172.21.15.96:0/430259679 conn(0x7f0efc0b1db0 msgr2=0x7f0efc0b21c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:13:24.921 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.918+0000 7f0f02200640 1 -- 172.21.15.96:0/430259679 shutdown_connections 2024-05-28T06:13:24.921 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.918+0000 7f0f02200640 1 -- 172.21.15.96:0/430259679 wait complete. 2024-05-28T06:13:24.921 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.920+0000 7f0f02200640 1 Processor -- start 2024-05-28T06:13:24.921 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.920+0000 7f0f02200640 1 -- start start 2024-05-28T06:13:24.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.921+0000 7f0f02200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0efc0af330 0x7f0efc0b3190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:24.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.921+0000 7f0f02200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0efc159f00 0x7f0efc153f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:24.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.921+0000 7f0f02200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0efc15d3b0 0x7f0efc14b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:24.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.921+0000 7f0f02200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f0efc072440 con 0x7f0efc15d3b0 2024-05-28T06:13:24.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.921+0000 7f0f02200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f0efc0cfbc0 con 0x7f0efc0af330 2024-05-28T06:13:24.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.921+0000 7f0f02200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f0efc0725c0 con 0x7f0efc159f00 2024-05-28T06:13:24.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.921+0000 7f0f01800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0efc15d3b0 0x7f0efc14b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:24.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.921+0000 7f0f01800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0efc15d3b0 0x7f0efc14b580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58984/0 (socket says 172.21.15.96:58984) 2024-05-28T06:13:24.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.921+0000 7f0f01800640 1 -- 172.21.15.96:0/3770490773 learned_addr learned my addr 172.21.15.96:0/3770490773 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:13:24.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.922+0000 7f0f01800640 1 -- 172.21.15.96:0/3770490773 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0efc159f00 msgr2=0x7f0efc153f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:24.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.922+0000 7f0f01800640 1 --2- 172.21.15.96:0/3770490773 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0efc159f00 0x7f0efc153f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:24.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.922+0000 7f0f00e00640 1 --2- 172.21.15.96:0/3770490773 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0efc0af330 0x7f0efc0b3190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:24.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.922+0000 7f0f01800640 1 -- 172.21.15.96:0/3770490773 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0efc0af330 msgr2=0x7f0efc0b3190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:24.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.922+0000 7f0f01800640 1 --2- 172.21.15.96:0/3770490773 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0efc0af330 0x7f0efc0b3190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:24.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.922+0000 7f0f01800640 1 -- 172.21.15.96:0/3770490773 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0efc0b4f00 con 0x7f0efc15d3b0 2024-05-28T06:13:24.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.922+0000 7f0f00e00640 1 --2- 172.21.15.96:0/3770490773 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0efc0af330 0x7f0efc0b3190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:13:24.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.922+0000 7f0f01800640 1 --2- 172.21.15.96:0/3770490773 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0efc15d3b0 0x7f0efc14b580 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f0ef4009f40 tx=0x7f0ef4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:13:24.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.922+0000 7f0ef9600640 1 -- 172.21.15.96:0/3770490773 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ef4041020 con 0x7f0efc15d3b0 2024-05-28T06:13:24.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.922+0000 7f0ef9600640 1 -- 172.21.15.96:0/3770490773 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ef4037040 con 0x7f0efc15d3b0 2024-05-28T06:13:24.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.922+0000 7f0ef9600640 1 -- 172.21.15.96:0/3770490773 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ef4036050 con 0x7f0efc15d3b0 2024-05-28T06:13:24.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.922+0000 7f0f02200640 1 -- 172.21.15.96:0/3770490773 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0efc155b00 con 0x7f0efc15d3b0 2024-05-28T06:13:24.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.923+0000 7f0f02200640 1 -- 172.21.15.96:0/3770490773 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0efc14bac0 con 0x7f0efc15d3b0 2024-05-28T06:13:24.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.924+0000 7f0f02200640 1 -- 172.21.15.96:0/3770490773 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0efc1561c0 con 0x7f0efc15d3b0 2024-05-28T06:13:24.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.925+0000 7f0ef9600640 1 -- 172.21.15.96:0/3770490773 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 17) ==== 91856+0+0 (secure 0 0 0) 0x7f0ef4007c30 con 0x7f0efc15d3b0 2024-05-28T06:13:24.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.926+0000 7f0ef9600640 1 --2- 172.21.15.96:0/3770490773 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0ed806d1f0 0x7f0ed806f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:24.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.926+0000 7f0ef9600640 1 -- 172.21.15.96:0/3770490773 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5737+0+0 (secure 0 0 0) 0x7f0ef40ad6a0 con 0x7f0efc15d3b0 2024-05-28T06:13:24.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.927+0000 7f0f00e00640 1 --2- 172.21.15.96:0/3770490773 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0ed806d1f0 0x7f0ed806f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:24.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.928+0000 7f0f00e00640 1 --2- 172.21.15.96:0/3770490773 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0ed806d1f0 0x7f0ed806f6b0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f0eec0013b0 tx=0x7f0eec009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:13:24.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:24.931+0000 7f0ef9600640 1 -- 172.21.15.96:0/3770490773 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0ef404b060 con 0x7f0efc15d3b0 2024-05-28T06:13:25.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:25.167+0000 7f0f02200640 1 -- 172.21.15.96:0/3770490773 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone81712", "target": ["mon-mgr", ""]}) -- 0x7f0efc0b51e0 con 0x7f0ed806d1f0 2024-05-28T06:13:25.695 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:13:25.695+0000 7fea4c200640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:13:25.695 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:13:25.695+0000 7fea4c200640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:13:25.695 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:13:25.695+0000 7fea4c200640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:13:25.696 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:13:25.695+0000 7fea4c200640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:13:25.696 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:13:25.695+0000 7fea4c200640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:13:25.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:25.900+0000 7f0ef9600640 1 -- 172.21.15.96:0/3770490773 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f0efc0b51e0 con 0x7f0ed806d1f0 2024-05-28T06:13:25.906 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:25.904+0000 7f0f02200640 1 -- 172.21.15.96:0/3770490773 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0ed806d1f0 msgr2=0x7f0ed806f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:25.906 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:25.904+0000 7f0f02200640 1 --2- 172.21.15.96:0/3770490773 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0ed806d1f0 0x7f0ed806f6b0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f0eec0013b0 tx=0x7f0eec009040 comp rx=0 tx=0).stop 2024-05-28T06:13:25.906 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:25.904+0000 7f0f02200640 1 -- 172.21.15.96:0/3770490773 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0efc15d3b0 msgr2=0x7f0efc14b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:25.906 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:25.904+0000 7f0f02200640 1 --2- 172.21.15.96:0/3770490773 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0efc15d3b0 0x7f0efc14b580 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f0ef4009f40 tx=0x7f0ef4004a40 comp rx=0 tx=0).stop 2024-05-28T06:13:25.906 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:25.904+0000 7f0f02200640 1 -- 172.21.15.96:0/3770490773 shutdown_connections 2024-05-28T06:13:25.906 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:25.904+0000 7f0f02200640 1 --2- 172.21.15.96:0/3770490773 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0ed806d1f0 0x7f0ed806f6b0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:25.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:25.904+0000 7f0f02200640 1 --2- 172.21.15.96:0/3770490773 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0efc15d3b0 0x7f0efc14b580 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:25.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:25.904+0000 7f0f02200640 1 --2- 172.21.15.96:0/3770490773 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0efc159f00 0x7f0efc153f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:25.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:25.904+0000 7f0f02200640 1 --2- 172.21.15.96:0/3770490773 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0efc0af330 0x7f0efc0b3190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:25.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:25.904+0000 7f0f02200640 1 -- 172.21.15.96:0/3770490773 >> 172.21.15.96:0/3770490773 conn(0x7f0efc0b1db0 msgr2=0x7f0efc06e230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:13:25.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:25.904+0000 7f0f02200640 1 -- 172.21.15.96:0/3770490773 shutdown_connections 2024-05-28T06:13:25.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:25.904+0000 7f0f02200640 1 -- 172.21.15.96:0/3770490773 wait complete. 2024-05-28T06:13:25.934 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone81713 2024-05-28T06:13:25.938 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:13:25.938+0000 7fea61600640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:13:25.938 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:13:25.938+0000 7fea61600640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:13:25.938 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:13:25.938+0000 7fea61600640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:13:25.938 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:13:25.938+0000 7fea61600640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:13:25.939 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:13:25.938+0000 7fea61600640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:13:26.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:26.219+0000 7f380d200640 1 Processor -- start 2024-05-28T06:13:26.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:26.220+0000 7f380d200640 1 -- start start 2024-05-28T06:13:26.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:26.221+0000 7f380d200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f3808161fa0 0x7f3808162600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:26.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:26.222+0000 7f380d200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f38080b52a0 0x7f3808162b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:26.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:26.222+0000 7f380d200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3808162380 con 0x7f38080b56d0 2024-05-28T06:13:26.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:26.222+0000 7f380d200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f38080d33e0 con 0x7f3808161fa0 2024-05-28T06:13:26.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:26.222+0000 7f380d200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f38080b5bd0 con 0x7f38080b52a0 2024-05-28T06:13:26.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:26.222+0000 7f3807400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f3808161fa0 0x7f3808162600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:26.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:26.222+0000 7f3807400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f3808161fa0 0x7f3808162600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58992/0 (socket says 172.21.15.96:58992) 2024-05-28T06:13:26.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:26.222+0000 7f3807400640 1 -- 172.21.15.96:0/1311591107 learned_addr learned my addr 172.21.15.96:0/1311591107 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:13:26.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:26.223+0000 7f3807400640 1 -- 172.21.15.96:0/1311591107 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f38080b52a0 msgr2=0x7f3808162b40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-28T06:13:26.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:26.223+0000 7f3807400640 1 --2- 172.21.15.96:0/1311591107 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f38080b52a0 0x7f3808162b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:26.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:26.223+0000 7f3807400640 1 -- 172.21.15.96:0/1311591107 >> v1:172.21.15.96:6789/0 conn(0x7f38080b56d0 legacy=0x7f38080d0e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:13:26.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:26.223+0000 7f3807400640 1 -- 172.21.15.96:0/1311591107 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38080d2e20 con 0x7f3808161fa0 2024-05-28T06:13:26.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:26.225+0000 7f3807400640 1 --2- 172.21.15.96:0/1311591107 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3808161fa0 0x7f3808162600 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f37f8009870 tx=0x7f37f802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ff6fff3e2609d71 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:13:26.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:26.225+0000 7f3806a00640 1 -- 172.21.15.96:0/1311591107 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37f8037070 con 0x7f3808161fa0 2024-05-28T06:13:26.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:26.225+0000 7f3806a00640 1 -- 172.21.15.96:0/1311591107 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37f8032440 con 0x7f3808161fa0 2024-05-28T06:13:26.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:26.225+0000 7f3806a00640 1 -- 172.21.15.96:0/1311591107 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37f8032810 con 0x7f3808161fa0 2024-05-28T06:13:26.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:26.226+0000 7f380d200640 1 -- 172.21.15.96:0/1311591107 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3808161fa0 msgr2=0x7f3808162600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:26.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:26.226+0000 7f380d200640 1 --2- 172.21.15.96:0/1311591107 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3808161fa0 0x7f3808162600 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f37f8009870 tx=0x7f37f802a1a0 comp rx=0 tx=0).stop 2024-05-28T06:13:26.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:26.226+0000 7f380d200640 1 -- 172.21.15.96:0/1311591107 shutdown_connections 2024-05-28T06:13:26.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:26.226+0000 7f380d200640 1 --2- 172.21.15.96:0/1311591107 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f38080b52a0 0x7f3808162b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:26.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:26.226+0000 7f380d200640 1 --2- 172.21.15.96:0/1311591107 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3808161fa0 0x7f3808162600 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:26.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:26.226+0000 7f380d200640 1 -- 172.21.15.96:0/1311591107 >> 172.21.15.96:0/1311591107 conn(0x7f380808bfc0 msgr2=0x7f38080d4470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:13:26.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:26.227+0000 7f380d200640 1 -- 172.21.15.96:0/1311591107 shutdown_connections 2024-05-28T06:13:26.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:26.227+0000 7f380d200640 1 -- 172.21.15.96:0/1311591107 wait complete. 2024-05-28T06:13:26.231 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:26.230+0000 7f380d200640 1 Processor -- start 2024-05-28T06:13:26.231 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:26.230+0000 7f380d200640 1 -- start start 2024-05-28T06:13:26.233 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:26.231+0000 7f380d200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38080b52a0 0x7f38080c81f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:26.233 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:26.231+0000 7f380d200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f38080b56d0 0x7f38081e9450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:26.233 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:26.231+0000 7f380d200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3808161fa0 0x7f38080b92b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:26.233 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:26.231+0000 7f380d200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f38080d51d0 con 0x7f3808161fa0 2024-05-28T06:13:26.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:26.231+0000 7f380d200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f38080d4cb0 con 0x7f38080b52a0 2024-05-28T06:13:26.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:26.231+0000 7f380d200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f38080d5350 con 0x7f38080b56d0 2024-05-28T06:13:26.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:26.232+0000 7f3807e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3808161fa0 0x7f38080b92b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:26.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:26.232+0000 7f3807e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3808161fa0 0x7f38080b92b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:59004/0 (socket says 172.21.15.96:59004) 2024-05-28T06:13:26.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:26.232+0000 7f3807e00640 1 -- 172.21.15.96:0/938068281 learned_addr learned my addr 172.21.15.96:0/938068281 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:13:26.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:26.232+0000 7f37ffe00640 1 --2- 172.21.15.96:0/938068281 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f38080b56d0 0x7f38081e9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:26.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:26.233+0000 7f3807e00640 1 -- 172.21.15.96:0/938068281 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f38080b56d0 msgr2=0x7f38081e9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:26.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:26.233+0000 7f3807e00640 1 --2- 172.21.15.96:0/938068281 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f38080b56d0 0x7f38081e9450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:26.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:26.233+0000 7f3807e00640 1 -- 172.21.15.96:0/938068281 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38080b52a0 msgr2=0x7f38080c81f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:26.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:26.233+0000 7f3807e00640 1 --2- 172.21.15.96:0/938068281 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38080b52a0 0x7f38080c81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:26.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:26.233+0000 7f3807e00640 1 -- 172.21.15.96:0/938068281 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38081e9dc0 con 0x7f3808161fa0 2024-05-28T06:13:26.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:26.233+0000 7f3807e00640 1 --2- 172.21.15.96:0/938068281 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3808161fa0 0x7f38080b92b0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f380000b9e0 tx=0x7f380000bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:13:26.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:26.233+0000 7f37ff400640 1 -- 172.21.15.96:0/938068281 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3800003b70 con 0x7f3808161fa0 2024-05-28T06:13:26.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:26.234+0000 7f37ff400640 1 -- 172.21.15.96:0/938068281 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3800003d10 con 0x7f3808161fa0 2024-05-28T06:13:26.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:26.234+0000 7f37ff400640 1 -- 172.21.15.96:0/938068281 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f380000cb60 con 0x7f3808161fa0 2024-05-28T06:13:26.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:26.234+0000 7f380d200640 1 -- 172.21.15.96:0/938068281 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38081ea0a0 con 0x7f3808161fa0 2024-05-28T06:13:26.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:26.234+0000 7f380d200640 1 -- 172.21.15.96:0/938068281 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38081ea660 con 0x7f3808161fa0 2024-05-28T06:13:26.238 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:26.237+0000 7f380d200640 1 -- 172.21.15.96:0/938068281 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38080690f0 con 0x7f3808161fa0 2024-05-28T06:13:26.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:26.239+0000 7f37ff400640 1 -- 172.21.15.96:0/938068281 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 17) ==== 91856+0+0 (secure 0 0 0) 0x7f380000cd00 con 0x7f3808161fa0 2024-05-28T06:13:26.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:26.240+0000 7f37ff400640 1 --2- 172.21.15.96:0/938068281 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f37e406d2c0 0x7f37e406f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:26.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:26.241+0000 7f37ff400640 1 -- 172.21.15.96:0/938068281 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5737+0+0 (secure 0 0 0) 0x7f380008a270 con 0x7f3808161fa0 2024-05-28T06:13:26.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:26.249+0000 7f37ff400640 1 -- 172.21.15.96:0/938068281 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f380005a0f0 con 0x7f3808161fa0 2024-05-28T06:13:26.259 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:26.257+0000 7f3807400640 1 --2- 172.21.15.96:0/938068281 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f37e406d2c0 0x7f37e406f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:26.282 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:26.280+0000 7f3807400640 1 --2- 172.21.15.96:0/938068281 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f37e406d2c0 0x7f37e406f780 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f37f8034000 tx=0x7f37f802c000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:13:26.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:26.673+0000 7f380d200640 1 -- 172.21.15.96:0/938068281 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone81713", "target": ["mon-mgr", ""]}) -- 0x7f38080ba190 con 0x7f37e406d2c0 2024-05-28T06:13:26.690 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:26.688+0000 7f37ff400640 1 -- 172.21.15.96:0/938068281 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mgrmap(e 18) ==== 92009+0+0 (secure 0 0 0) 0x7f3800056060 con 0x7f3808161fa0 2024-05-28T06:13:27.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:27.187+0000 7f37ff400640 1 -- 172.21.15.96:0/938068281 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f38080ba190 con 0x7f37e406d2c0 2024-05-28T06:13:27.194 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:27.193+0000 7f380d200640 1 -- 172.21.15.96:0/938068281 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f37e406d2c0 msgr2=0x7f37e406f780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:27.194 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:27.193+0000 7f380d200640 1 --2- 172.21.15.96:0/938068281 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f37e406d2c0 0x7f37e406f780 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f37f8034000 tx=0x7f37f802c000 comp rx=0 tx=0).stop 2024-05-28T06:13:27.194 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:27.193+0000 7f380d200640 1 -- 172.21.15.96:0/938068281 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3808161fa0 msgr2=0x7f38080b92b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:27.195 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:27.193+0000 7f380d200640 1 --2- 172.21.15.96:0/938068281 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3808161fa0 0x7f38080b92b0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f380000b9e0 tx=0x7f380000bea0 comp rx=0 tx=0).stop 2024-05-28T06:13:27.195 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:27.193+0000 7f380d200640 1 -- 172.21.15.96:0/938068281 shutdown_connections 2024-05-28T06:13:27.195 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:27.193+0000 7f380d200640 1 --2- 172.21.15.96:0/938068281 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f37e406d2c0 0x7f37e406f780 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:27.195 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:27.193+0000 7f380d200640 1 --2- 172.21.15.96:0/938068281 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3808161fa0 0x7f38080b92b0 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:27.195 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:27.193+0000 7f380d200640 1 --2- 172.21.15.96:0/938068281 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f38080b56d0 0x7f38081e9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:27.195 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:27.193+0000 7f380d200640 1 --2- 172.21.15.96:0/938068281 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38080b52a0 0x7f38080c81f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:27.195 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:27.193+0000 7f380d200640 1 -- 172.21.15.96:0/938068281 >> 172.21.15.96:0/938068281 conn(0x7f380808bfc0 msgr2=0x7f38080b2140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:13:27.196 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:27.194+0000 7f380d200640 1 -- 172.21.15.96:0/938068281 shutdown_connections 2024-05-28T06:13:27.196 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:27.194+0000 7f380d200640 1 -- 172.21.15.96:0/938068281 wait complete. 2024-05-28T06:13:27.228 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone81714 2024-05-28T06:13:27.251 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:13:27.250+0000 7fea61600640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:13:27.251 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:13:27.250+0000 7fea61600640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:13:27.251 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:13:27.250+0000 7fea61600640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:13:27.251 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:13:27.250+0000 7fea61600640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:13:27.251 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:13:27.250+0000 7fea61600640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:13:27.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:27.533+0000 7f71a2400640 1 Processor -- start 2024-05-28T06:13:27.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:27.533+0000 7f71a2400640 1 -- start start 2024-05-28T06:13:27.536 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:27.535+0000 7f71a2400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f719c0b3a50 0x7f719c169980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:27.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:27.535+0000 7f71a2400640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f719c0b5310 con 0x7f719c0b3670 2024-05-28T06:13:27.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:27.535+0000 7f71a2400640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f719c0d2d30 con 0x7f719c0b4e80 2024-05-28T06:13:27.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:27.535+0000 7f71a2400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f719c162450 con 0x7f719c0b3a50 2024-05-28T06:13:27.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:27.535+0000 7f719b400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f719c0b3670 0x7f719c162650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:47998/0 (socket says 172.21.15.96:47998) 2024-05-28T06:13:27.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:27.535+0000 7f719b400640 1 -- 172.21.15.96:0/1631749530 learned_addr learned my addr 172.21.15.96:0/1631749530 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:13:27.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:27.536+0000 7f719a000640 1 -- 172.21.15.96:0/1631749530 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2533811282 0 0) 0x7f719c0b5310 con 0x7f719c0b3670 2024-05-28T06:13:27.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:27.536+0000 7f719a000640 1 -- 172.21.15.96:0/1631749530 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7180003520 con 0x7f719c0b3670 2024-05-28T06:13:27.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:27.536+0000 7f719a000640 1 -- 172.21.15.96:0/1631749530 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1627964092 0 0) 0x7f7180003520 con 0x7f719c0b3670 2024-05-28T06:13:27.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:27.536+0000 7f719a000640 1 -- 172.21.15.96:0/1631749530 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f719c0b3a50 msgr2=0x7f719c169980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:13:27.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:27.536+0000 7f719a000640 1 --2- 172.21.15.96:0/1631749530 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f719c0b3a50 0x7f719c169980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:27.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:27.536+0000 7f719a000640 1 -- 172.21.15.96:0/1631749530 >> v1:172.21.15.123:6789/0 conn(0x7f719c0b4e80 legacy=0x7f719c162d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:13:27.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:27.536+0000 7f719a000640 1 -- 172.21.15.96:0/1631749530 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f719c0d3010 con 0x7f719c0b3670 2024-05-28T06:13:27.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:27.536+0000 7f719a000640 1 -- 172.21.15.96:0/1631749530 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f718c0017f0 con 0x7f719c0b3670 2024-05-28T06:13:27.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:27.536+0000 7f719a000640 1 -- 172.21.15.96:0/1631749530 >> v1:172.21.15.96:6789/0 conn(0x7f719c0b3670 legacy=0x7f719c162650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:27.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:27.536+0000 7f719a000640 1 --2- 172.21.15.96:0/1631749530 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f719c0b3a50 0x7f71800039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:27.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:27.536+0000 7f719a000640 1 --2- 172.21.15.96:0/1631749530 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f71800040d0 0x7f7180004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:27.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:27.536+0000 7f719a000640 1 --2- 172.21.15.96:0/1631749530 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7180004db0 0x7f7180005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:27.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:27.537+0000 7f719a000640 1 -- 172.21.15.96:0/1631749530 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f718000bc90 con 0x7f7180004db0 2024-05-28T06:13:27.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:27.537+0000 7f719a000640 1 -- 172.21.15.96:0/1631749530 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f718000be40 con 0x7f71800040d0 2024-05-28T06:13:27.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:27.537+0000 7f719a000640 1 -- 172.21.15.96:0/1631749530 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f718000c040 con 0x7f719c0b3a50 2024-05-28T06:13:27.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:27.537+0000 7f719be00640 1 --2- 172.21.15.96:0/1631749530 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7180004db0 0x7f7180005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:27.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:27.537+0000 7f719be00640 1 -- 172.21.15.96:0/1631749530 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f719c0b3a50 msgr2=0x7f71800039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:13:27.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:27.537+0000 7f719be00640 1 --2- 172.21.15.96:0/1631749530 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f719c0b3a50 0x7f71800039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:27.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:27.537+0000 7f719be00640 1 -- 172.21.15.96:0/1631749530 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f71800040d0 msgr2=0x7f7180004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:13:27.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:27.537+0000 7f719be00640 1 --2- 172.21.15.96:0/1631749530 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f71800040d0 0x7f7180004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:27.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:27.537+0000 7f719be00640 1 -- 172.21.15.96:0/1631749530 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7180003520 con 0x7f7180004db0 2024-05-28T06:13:27.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:27.539+0000 7f719be00640 1 --2- 172.21.15.96:0/1631749530 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7180004db0 0x7f7180005230 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f7194009e10 tx=0x7f7180005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a10ed60325f68f9 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:13:27.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:27.540+0000 7f719a000640 1 -- 172.21.15.96:0/1631749530 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7194009a10 con 0x7f7180004db0 2024-05-28T06:13:27.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:27.540+0000 7f719a000640 1 -- 172.21.15.96:0/1631749530 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7194036070 con 0x7f7180004db0 2024-05-28T06:13:27.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:27.540+0000 7f719a000640 1 -- 172.21.15.96:0/1631749530 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7194031900 con 0x7f7180004db0 2024-05-28T06:13:27.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:27.541+0000 7f71a2400640 1 -- 172.21.15.96:0/1631749530 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7180004db0 msgr2=0x7f7180005230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:27.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:27.541+0000 7f71a2400640 1 --2- 172.21.15.96:0/1631749530 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7180004db0 0x7f7180005230 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f7194009e10 tx=0x7f7180005830 comp rx=0 tx=0).stop 2024-05-28T06:13:27.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:27.541+0000 7f719be00640 1 -- 172.21.15.96:0/1631749530 reap_dead start 2024-05-28T06:13:27.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:27.541+0000 7f71a2400640 1 -- 172.21.15.96:0/1631749530 shutdown_connections 2024-05-28T06:13:27.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:27.541+0000 7f71a2400640 1 -- 172.21.15.96:0/1631749530 >> 172.21.15.96:0/1631749530 conn(0x7f719c08bfc0 msgr2=0x7f719c0d4640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:13:27.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:27.542+0000 7f71a2400640 1 -- 172.21.15.96:0/1631749530 shutdown_connections 2024-05-28T06:13:27.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:27.542+0000 7f71a2400640 1 -- 172.21.15.96:0/1631749530 wait complete. 2024-05-28T06:13:27.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:27.544+0000 7f71a2400640 1 Processor -- start 2024-05-28T06:13:27.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:27.544+0000 7f71a2400640 1 -- start start 2024-05-28T06:13:27.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:27.546+0000 7f71a2400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f719c1579e0 0x7f719c155a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:27.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:27.546+0000 7f71a2400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f719c0b8be0 0x7f719c1f8260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:27.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:27.547+0000 7f719b400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f719c1579e0 0x7f719c155a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:27.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:27.547+0000 7f719b400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f719c1579e0 0x7f719c155a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:59010/0 (socket says 172.21.15.96:59010) 2024-05-28T06:13:27.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:27.547+0000 7f71a2400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f719c157dc0 0x7f719c1f87a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:27.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:27.547+0000 7f719b400640 1 -- 172.21.15.96:0/1816842881 learned_addr learned my addr 172.21.15.96:0/1816842881 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:13:27.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:27.547+0000 7f71a2400640 1 -- 172.21.15.96:0/1816842881 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f719c163470 con 0x7f719c1579e0 2024-05-28T06:13:27.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:27.547+0000 7f71a2400640 1 -- 172.21.15.96:0/1816842881 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f719c0b56e0 con 0x7f719c0b8be0 2024-05-28T06:13:27.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:27.547+0000 7f71a2400640 1 -- 172.21.15.96:0/1816842881 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f719c161560 con 0x7f719c157dc0 2024-05-28T06:13:27.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:27.547+0000 7f719b400640 1 -- 172.21.15.96:0/1816842881 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f719c157dc0 msgr2=0x7f719c1f87a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-28T06:13:27.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:27.547+0000 7f719b400640 1 --2- 172.21.15.96:0/1816842881 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f719c157dc0 0x7f719c1f87a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:27.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:27.547+0000 7f719b400640 1 -- 172.21.15.96:0/1816842881 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f719c0b8be0 msgr2=0x7f719c1f8260 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-28T06:13:27.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:27.547+0000 7f719b400640 1 --2- 172.21.15.96:0/1816842881 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f719c0b8be0 0x7f719c1f8260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:27.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:27.547+0000 7f719b400640 1 -- 172.21.15.96:0/1816842881 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f719c0b9ba0 con 0x7f719c1579e0 2024-05-28T06:13:27.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:27.548+0000 7f719b400640 1 --2- 172.21.15.96:0/1816842881 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f719c1579e0 0x7f719c155a70 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f718c00c7f0 tx=0x7f718c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:13:27.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:27.549+0000 7f7193400640 1 -- 172.21.15.96:0/1816842881 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f718c017070 con 0x7f719c1579e0 2024-05-28T06:13:27.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:27.549+0000 7f7193400640 1 -- 172.21.15.96:0/1816842881 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f718c0074a0 con 0x7f719c1579e0 2024-05-28T06:13:27.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:27.549+0000 7f7193400640 1 -- 172.21.15.96:0/1816842881 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f718c0059e0 con 0x7f719c1579e0 2024-05-28T06:13:27.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:27.549+0000 7f71a2400640 1 -- 172.21.15.96:0/1816842881 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f719c0b9e80 con 0x7f719c1579e0 2024-05-28T06:13:27.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:27.549+0000 7f71a2400640 1 -- 172.21.15.96:0/1816842881 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f719c155fb0 con 0x7f719c1579e0 2024-05-28T06:13:27.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:27.552+0000 7f71a2400640 1 -- 172.21.15.96:0/1816842881 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f719c0ba3a0 con 0x7f719c1579e0 2024-05-28T06:13:27.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:27.553+0000 7f7193400640 1 -- 172.21.15.96:0/1816842881 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 18) ==== 92009+0+0 (secure 0 0 0) 0x7f718c01a070 con 0x7f719c1579e0 2024-05-28T06:13:27.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:27.555+0000 7f7193400640 1 --2- 172.21.15.96:0/1816842881 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f717c06d470 0x7f717c06f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:27.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:27.555+0000 7f7193400640 1 -- 172.21.15.96:0/1816842881 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5737+0+0 (secure 0 0 0) 0x7f718c089d90 con 0x7f719c1579e0 2024-05-28T06:13:27.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:27.560+0000 7f7193400640 1 -- 172.21.15.96:0/1816842881 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f719c0ba3a0 con 0x7f719c1579e0 2024-05-28T06:13:27.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:27.568+0000 7f719aa00640 1 --2- 172.21.15.96:0/1816842881 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f717c06d470 0x7f717c06f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:27.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:27.608+0000 7f719aa00640 1 --2- 172.21.15.96:0/1816842881 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f717c06d470 0x7f717c06f930 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f719c1f80e0 tx=0x7f71880079b0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:13:27.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:27.957+0000 7f71a2400640 1 -- 172.21.15.96:0/1816842881 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone81714", "target": ["mon-mgr", ""]}) -- 0x7f719c0cf430 con 0x7f717c06d470 2024-05-28T06:13:28.483 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:28.482+0000 7f7193400640 1 -- 172.21.15.96:0/1816842881 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f719c0cf430 con 0x7f717c06d470 2024-05-28T06:13:28.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:28.485+0000 7f71a2400640 1 -- 172.21.15.96:0/1816842881 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f717c06d470 msgr2=0x7f717c06f930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:28.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:28.485+0000 7f71a2400640 1 --2- 172.21.15.96:0/1816842881 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f717c06d470 0x7f717c06f930 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f719c1f80e0 tx=0x7f71880079b0 comp rx=0 tx=0).stop 2024-05-28T06:13:28.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:28.485+0000 7f71a2400640 1 -- 172.21.15.96:0/1816842881 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f719c1579e0 msgr2=0x7f719c155a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:28.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:28.485+0000 7f71a2400640 1 --2- 172.21.15.96:0/1816842881 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f719c1579e0 0x7f719c155a70 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f718c00c7f0 tx=0x7f718c00ccb0 comp rx=0 tx=0).stop 2024-05-28T06:13:28.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:28.486+0000 7f71a2400640 1 -- 172.21.15.96:0/1816842881 shutdown_connections 2024-05-28T06:13:28.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:28.486+0000 7f71a2400640 1 --2- 172.21.15.96:0/1816842881 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f717c06d470 0x7f717c06f930 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:28.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:28.486+0000 7f71a2400640 1 --2- 172.21.15.96:0/1816842881 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f719c157dc0 0x7f719c1f87a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:28.489 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:28.486+0000 7f71a2400640 1 --2- 172.21.15.96:0/1816842881 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f719c0b8be0 0x7f719c1f8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:28.489 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:28.486+0000 7f71a2400640 1 --2- 172.21.15.96:0/1816842881 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f719c1579e0 0x7f719c155a70 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:28.489 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:28.486+0000 7f71a2400640 1 -- 172.21.15.96:0/1816842881 >> 172.21.15.96:0/1816842881 conn(0x7f719c08bfc0 msgr2=0x7f719c06e110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:13:28.489 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:28.486+0000 7f71a2400640 1 -- 172.21.15.96:0/1816842881 shutdown_connections 2024-05-28T06:13:28.489 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:28.486+0000 7f71a2400640 1 -- 172.21.15.96:0/1816842881 wait complete. 2024-05-28T06:13:28.521 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone81715 2024-05-28T06:13:28.793 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:28.792+0000 7fd1c2600640 1 Processor -- start 2024-05-28T06:13:28.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:28.792+0000 7fd1c2600640 1 -- start start 2024-05-28T06:13:28.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:28.794+0000 7fd1c2600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fd1bc161d00 0x7fd1bc0b3d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:28.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:28.794+0000 7fd1c2600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd1bc0d4d90 0x7fd1bc0d0e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:28.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:28.794+0000 7fd1c2600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fd1bc0d1380 0x7fd1bc0d27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:28.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:28.794+0000 7fd1c2600640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fd1bc0d17e0 con 0x7fd1bc161d00 2024-05-28T06:13:28.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:28.794+0000 7fd1c2600640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fd1bc0d2040 con 0x7fd1bc0d1380 2024-05-28T06:13:28.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:28.794+0000 7fd1c2600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fd1bc0cfb40 con 0x7fd1bc0d4d90 2024-05-28T06:13:28.798 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:28.796+0000 7fd1c1c00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fd1bc0d1380 0x7fd1bc0d27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:28.798 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:28.796+0000 7fd1bbe00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd1bc0d4d90 0x7fd1bc0d0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:28.798 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:28.796+0000 7fd1bbe00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd1bc0d4d90 0x7fd1bc0d0e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.96:60540/0 (socket says 172.21.15.96:60540) 2024-05-28T06:13:28.798 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:28.796+0000 7fd1bbe00640 1 -- 172.21.15.96:0/1055238731 learned_addr learned my addr 172.21.15.96:0/1055238731 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:13:28.798 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:28.797+0000 7fd1c1200640 1 --2- 172.21.15.96:0/1055238731 >> v2:172.21.15.96:3300/0 conn(0x7fd1bc161d00 0x7fd1bc0b3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:28.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:28.797+0000 7fd1c1200640 1 -- 172.21.15.96:0/1055238731 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd1bc0d4d90 msgr2=0x7fd1bc0d0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:13:28.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:28.797+0000 7fd1c1200640 1 --2- 172.21.15.96:0/1055238731 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd1bc0d4d90 0x7fd1bc0d0e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:28.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:28.797+0000 7fd1c1200640 1 -- 172.21.15.96:0/1055238731 >> v2:172.21.15.123:3300/0 conn(0x7fd1bc0d1380 msgr2=0x7fd1bc0d27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:13:28.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:28.797+0000 7fd1c1200640 1 --2- 172.21.15.96:0/1055238731 >> v2:172.21.15.123:3300/0 conn(0x7fd1bc0d1380 0x7fd1bc0d27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:28.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:28.797+0000 7fd1c1200640 1 -- 172.21.15.96:0/1055238731 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1bc0cfdc0 con 0x7fd1bc161d00 2024-05-28T06:13:28.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:28.797+0000 7fd1c1c00640 1 --2- 172.21.15.96:0/1055238731 >> v2:172.21.15.123:3300/0 conn(0x7fd1bc0d1380 0x7fd1bc0d27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-28T06:13:28.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:28.800+0000 7fd1c1200640 1 --2- 172.21.15.96:0/1055238731 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd1bc161d00 0x7fd1bc0b3d00 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fd1bc0c7d30 tx=0x7fd1b002a130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3559d5dc6469a99 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:13:28.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:28.800+0000 7fd1bb400640 1 -- 172.21.15.96:0/1055238731 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1b0037070 con 0x7fd1bc161d00 2024-05-28T06:13:28.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:28.801+0000 7fd1bb400640 1 -- 172.21.15.96:0/1055238731 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1b002ae40 con 0x7fd1bc161d00 2024-05-28T06:13:28.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:28.801+0000 7fd1bb400640 1 -- 172.21.15.96:0/1055238731 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1b0032660 con 0x7fd1bc161d00 2024-05-28T06:13:28.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:28.802+0000 7fd1c2600640 1 -- 172.21.15.96:0/1055238731 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd1bc161d00 msgr2=0x7fd1bc0b3d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:28.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:28.802+0000 7fd1c2600640 1 --2- 172.21.15.96:0/1055238731 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd1bc161d00 0x7fd1bc0b3d00 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fd1bc0c7d30 tx=0x7fd1b002a130 comp rx=0 tx=0).stop 2024-05-28T06:13:28.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:28.802+0000 7fd1c2600640 1 -- 172.21.15.96:0/1055238731 shutdown_connections 2024-05-28T06:13:28.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:28.802+0000 7fd1c2600640 1 --2- 172.21.15.96:0/1055238731 >> v2:172.21.15.123:3300/0 conn(0x7fd1bc0d1380 0x7fd1bc0d27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:28.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:28.802+0000 7fd1c2600640 1 --2- 172.21.15.96:0/1055238731 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd1bc0d4d90 0x7fd1bc0d0e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:28.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:28.802+0000 7fd1c2600640 1 --2- 172.21.15.96:0/1055238731 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd1bc161d00 0x7fd1bc0b3d00 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:28.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:28.802+0000 7fd1c2600640 1 -- 172.21.15.96:0/1055238731 >> 172.21.15.96:0/1055238731 conn(0x7fd1bc08bfc0 msgr2=0x7fd1bc0d4470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:13:28.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:28.803+0000 7fd1c2600640 1 -- 172.21.15.96:0/1055238731 shutdown_connections 2024-05-28T06:13:28.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:28.803+0000 7fd1c2600640 1 -- 172.21.15.96:0/1055238731 wait complete. 2024-05-28T06:13:28.808 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:28.806+0000 7fd1c2600640 1 Processor -- start 2024-05-28T06:13:28.808 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:28.806+0000 7fd1c2600640 1 -- start start 2024-05-28T06:13:28.809 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:28.808+0000 7fd1c2600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd1bc0d1380 0x7fd1bc0c6f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:28.810 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:28.809+0000 7fd1c2600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd1bc0d4d90 0x7fd1bc0b3230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:28.810 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:28.809+0000 7fd1c2600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd1bc161d00 0x7fd1bc0b3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:28.810 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:28.809+0000 7fd1c2600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd1bc0d2290 con 0x7fd1bc161d00 2024-05-28T06:13:28.810 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:28.809+0000 7fd1c2600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd1bc0d0380 con 0x7fd1bc0d1380 2024-05-28T06:13:28.811 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:28.809+0000 7fd1c2600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fd1bc0b5560 con 0x7fd1bc0d4d90 2024-05-28T06:13:28.811 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:28.809+0000 7fd1c1c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd1bc161d00 0x7fd1bc0b3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:28.812 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:28.810+0000 7fd1c1c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd1bc161d00 0x7fd1bc0b3770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49076/0 (socket says 172.21.15.96:49076) 2024-05-28T06:13:28.812 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:28.810+0000 7fd1c1c00640 1 -- 172.21.15.96:0/1137060598 learned_addr learned my addr 172.21.15.96:0/1137060598 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:13:28.812 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:28.810+0000 7fd1c1200640 1 --2- 172.21.15.96:0/1137060598 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd1bc0d1380 0x7fd1bc0c6f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:28.812 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:28.810+0000 7fd1c1c00640 1 -- 172.21.15.96:0/1137060598 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd1bc0d4d90 msgr2=0x7fd1bc0b3230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:28.812 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:28.810+0000 7fd1c1c00640 1 --2- 172.21.15.96:0/1137060598 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd1bc0d4d90 0x7fd1bc0b3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:28.813 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:28.810+0000 7fd1c1c00640 1 -- 172.21.15.96:0/1137060598 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd1bc0d1380 msgr2=0x7fd1bc0c6f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:28.813 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:28.810+0000 7fd1c1c00640 1 --2- 172.21.15.96:0/1137060598 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd1bc0d1380 0x7fd1bc0c6f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:28.813 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:28.810+0000 7fd1c1c00640 1 -- 172.21.15.96:0/1137060598 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1bc1534b0 con 0x7fd1bc161d00 2024-05-28T06:13:28.813 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:28.811+0000 7fd1c1c00640 1 --2- 172.21.15.96:0/1137060598 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd1bc161d00 0x7fd1bc0b3770 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fd1b400eed0 tx=0x7fd1b400b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:13:28.813 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:28.811+0000 7fd1b8c00640 1 -- 172.21.15.96:0/1137060598 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1b4018020 con 0x7fd1bc161d00 2024-05-28T06:13:28.814 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:28.812+0000 7fd1c2600640 1 -- 172.21.15.96:0/1137060598 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1bc153790 con 0x7fd1bc161d00 2024-05-28T06:13:28.814 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:28.812+0000 7fd1b8c00640 1 -- 172.21.15.96:0/1137060598 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1b4014400 con 0x7fd1bc161d00 2024-05-28T06:13:28.814 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:28.812+0000 7fd1c2600640 1 -- 172.21.15.96:0/1137060598 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1bc153d50 con 0x7fd1bc161d00 2024-05-28T06:13:28.818 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:28.814+0000 7fd1b8c00640 1 -- 172.21.15.96:0/1137060598 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1b40147b0 con 0x7fd1bc161d00 2024-05-28T06:13:28.820 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:28.817+0000 7fd1c2600640 1 -- 172.21.15.96:0/1137060598 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd198005290 con 0x7fd1bc161d00 2024-05-28T06:13:28.822 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:28.820+0000 7fd1b8c00640 1 -- 172.21.15.96:0/1137060598 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 19) ==== 92111+0+0 (secure 0 0 0) 0x7fd1b4005d10 con 0x7fd1bc161d00 2024-05-28T06:13:28.822 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:28.821+0000 7fd1b8c00640 1 --2- 172.21.15.96:0/1137060598 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd19c06d4c0 0x7fd19c06f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:28.823 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:28.821+0000 7fd1b8c00640 1 -- 172.21.15.96:0/1137060598 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5737+0+0 (secure 0 0 0) 0x7fd1b408aef0 con 0x7fd1bc161d00 2024-05-28T06:13:28.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:28.828+0000 7fd1c1200640 1 --2- 172.21.15.96:0/1137060598 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd19c06d4c0 0x7fd19c06f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:28.830 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:28.828+0000 7fd1b8c00640 1 -- 172.21.15.96:0/1137060598 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd1b4022030 con 0x7fd1bc161d00 2024-05-28T06:13:28.857 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:28.856+0000 7fd1c1200640 1 --2- 172.21.15.96:0/1137060598 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd19c06d4c0 0x7fd19c06f980 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fd1b0034000 tx=0x7fd1b002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:13:29.185 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:29.183+0000 7fd1c2600640 1 -- 172.21.15.96:0/1137060598 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone81715", "target": ["mon-mgr", ""]}) -- 0x7fd198000c80 con 0x7fd19c06d4c0 2024-05-28T06:13:30.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:30.308+0000 7fd1b8c00640 1 -- 172.21.15.96:0/1137060598 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fd198000c80 con 0x7fd19c06d4c0 2024-05-28T06:13:30.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:30.315+0000 7fd1c2600640 1 -- 172.21.15.96:0/1137060598 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd19c06d4c0 msgr2=0x7fd19c06f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:30.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:30.315+0000 7fd1c2600640 1 --2- 172.21.15.96:0/1137060598 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd19c06d4c0 0x7fd19c06f980 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fd1b0034000 tx=0x7fd1b002e000 comp rx=0 tx=0).stop 2024-05-28T06:13:30.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:30.315+0000 7fd1c2600640 1 -- 172.21.15.96:0/1137060598 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd1bc161d00 msgr2=0x7fd1bc0b3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:30.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:30.315+0000 7fd1c2600640 1 --2- 172.21.15.96:0/1137060598 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd1bc161d00 0x7fd1bc0b3770 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fd1b400eed0 tx=0x7fd1b400b480 comp rx=0 tx=0).stop 2024-05-28T06:13:30.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:30.315+0000 7fd1c2600640 1 -- 172.21.15.96:0/1137060598 shutdown_connections 2024-05-28T06:13:30.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:30.315+0000 7fd1c2600640 1 --2- 172.21.15.96:0/1137060598 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd19c06d4c0 0x7fd19c06f980 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:30.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:30.315+0000 7fd1c2600640 1 --2- 172.21.15.96:0/1137060598 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd1bc161d00 0x7fd1bc0b3770 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:30.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:30.315+0000 7fd1c2600640 1 --2- 172.21.15.96:0/1137060598 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd1bc0d4d90 0x7fd1bc0b3230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:30.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:30.315+0000 7fd1c2600640 1 --2- 172.21.15.96:0/1137060598 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd1bc0d1380 0x7fd1bc0c6f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:30.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:30.315+0000 7fd1c2600640 1 -- 172.21.15.96:0/1137060598 >> 172.21.15.96:0/1137060598 conn(0x7fd1bc08bfc0 msgr2=0x7fd1bc0d41c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:13:30.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:30.315+0000 7fd1c2600640 1 -- 172.21.15.96:0/1137060598 shutdown_connections 2024-05-28T06:13:30.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:30.315+0000 7fd1c2600640 1 -- 172.21.15.96:0/1137060598 wait complete. 2024-05-28T06:13:30.343 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone81716 2024-05-28T06:13:30.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:30.679+0000 7f8e61e00640 1 Processor -- start 2024-05-28T06:13:30.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:30.679+0000 7f8e61e00640 1 -- start start 2024-05-28T06:13:30.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:30.680+0000 7f8e61e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f8e5c0b3670 0x7f8e5c162650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:30.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:30.680+0000 7f8e61e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8e5c0b3a50 0x7f8e5c162b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:30.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:30.680+0000 7f8e61e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f8e5c1630d0 0x7f8e5c0b3e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:30.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:30.680+0000 7f8e61e00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f8e5c162450 con 0x7f8e5c1630d0 2024-05-28T06:13:30.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:30.680+0000 7f8e61e00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f8e5c163580 con 0x7f8e5c0b3670 2024-05-28T06:13:30.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:30.680+0000 7f8e61e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f8e5c0b5310 con 0x7f8e5c0b3a50 2024-05-28T06:13:30.683 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:30.681+0000 7f8e52000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8e5c0b3a50 0x7f8e5c162b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:30.683 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:30.681+0000 7f8e52000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8e5c0b3a50 0x7f8e5c162b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.96:34916/0 (socket says 172.21.15.96:34916) 2024-05-28T06:13:30.683 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:30.681+0000 7f8e52000640 1 -- 172.21.15.96:0/2616071493 learned_addr learned my addr 172.21.15.96:0/2616071493 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:13:30.683 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:30.681+0000 7f8e5aa00640 1 --2- 172.21.15.96:0/2616071493 >> v2:172.21.15.123:3300/0 conn(0x7f8e5c0b3670 0x7f8e5c162650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:30.683 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:30.681+0000 7f8e52000640 1 -- 172.21.15.96:0/2616071493 >> v2:172.21.15.123:3300/0 conn(0x7f8e5c0b3670 msgr2=0x7f8e5c162650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:13:30.684 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:30.681+0000 7f8e52000640 1 --2- 172.21.15.96:0/2616071493 >> v2:172.21.15.123:3300/0 conn(0x7f8e5c0b3670 0x7f8e5c162650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:30.684 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:30.681+0000 7f8e52000640 1 -- 172.21.15.96:0/2616071493 >> v2:172.21.15.96:3300/0 conn(0x7f8e5c1630d0 msgr2=0x7f8e5c0b3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:13:30.684 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:30.681+0000 7f8e52000640 1 --2- 172.21.15.96:0/2616071493 >> v2:172.21.15.96:3300/0 conn(0x7f8e5c1630d0 0x7f8e5c0b3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:30.684 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:30.681+0000 7f8e52000640 1 -- 172.21.15.96:0/2616071493 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e5c0d2d30 con 0x7f8e5c0b3a50 2024-05-28T06:13:30.684 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:30.683+0000 7f8e52000640 1 --2- 172.21.15.96:0/2616071493 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8e5c0b3a50 0x7f8e5c162b90 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f8e5c073f70 tx=0x7f8e4402a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2e07207373ab2e66 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:13:30.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:30.684+0000 7f8e5a000640 1 -- 172.21.15.96:0/2616071493 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e44037070 con 0x7f8e5c0b3a50 2024-05-28T06:13:30.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:30.684+0000 7f8e5a000640 1 -- 172.21.15.96:0/2616071493 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e44032440 con 0x7f8e5c0b3a50 2024-05-28T06:13:30.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:30.684+0000 7f8e5a000640 1 -- 172.21.15.96:0/2616071493 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e44032810 con 0x7f8e5c0b3a50 2024-05-28T06:13:30.686 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:30.684+0000 7f8e61e00640 1 -- 172.21.15.96:0/2616071493 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8e5c0b3a50 msgr2=0x7f8e5c162b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:30.686 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:30.684+0000 7f8e61e00640 1 --2- 172.21.15.96:0/2616071493 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8e5c0b3a50 0x7f8e5c162b90 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f8e5c073f70 tx=0x7f8e4402a180 comp rx=0 tx=0).stop 2024-05-28T06:13:30.686 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:30.685+0000 7f8e61e00640 1 -- 172.21.15.96:0/2616071493 shutdown_connections 2024-05-28T06:13:30.686 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:30.685+0000 7f8e61e00640 1 --2- 172.21.15.96:0/2616071493 >> v2:172.21.15.96:3300/0 conn(0x7f8e5c1630d0 0x7f8e5c0b3e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:30.686 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:30.685+0000 7f8e61e00640 1 --2- 172.21.15.96:0/2616071493 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8e5c0b3a50 0x7f8e5c162b90 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:30.687 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:30.685+0000 7f8e61e00640 1 --2- 172.21.15.96:0/2616071493 >> v2:172.21.15.123:3300/0 conn(0x7f8e5c0b3670 0x7f8e5c162650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:30.687 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:30.685+0000 7f8e61e00640 1 -- 172.21.15.96:0/2616071493 >> 172.21.15.96:0/2616071493 conn(0x7f8e5c08bfc0 msgr2=0x7f8e5c0d4640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:13:30.687 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:30.685+0000 7f8e61e00640 1 -- 172.21.15.96:0/2616071493 shutdown_connections 2024-05-28T06:13:30.687 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:30.685+0000 7f8e61e00640 1 -- 172.21.15.96:0/2616071493 wait complete. 2024-05-28T06:13:30.688 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:30.687+0000 7f8e61e00640 1 Processor -- start 2024-05-28T06:13:30.688 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:30.687+0000 7f8e61e00640 1 -- start start 2024-05-28T06:13:30.691 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:30.688+0000 7f8e61e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e5c0b3670 0x7f8e5c0cab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:30.691 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:30.688+0000 7f8e61e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8e5c0b3a50 0x7f8e5c1584e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:30.691 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:30.688+0000 7f8e61e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8e5c1630d0 0x7f8e5c158a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:30.691 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:30.688+0000 7f8e61e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f8e5c0b56e0 con 0x7f8e5c1630d0 2024-05-28T06:13:30.691 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:30.688+0000 7f8e61e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f8e5c0d32f0 con 0x7f8e5c0b3670 2024-05-28T06:13:30.691 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:30.688+0000 7f8e61e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f8e5c0b43b0 con 0x7f8e5c0b3a50 2024-05-28T06:13:30.691 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:30.689+0000 7f8e5b400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8e5c1630d0 0x7f8e5c158a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:30.691 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:30.689+0000 7f8e5b400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8e5c1630d0 0x7f8e5c158a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49092/0 (socket says 172.21.15.96:49092) 2024-05-28T06:13:30.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:30.689+0000 7f8e5b400640 1 -- 172.21.15.96:0/4017225972 learned_addr learned my addr 172.21.15.96:0/4017225972 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:13:30.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:30.689+0000 7f8e5b400640 1 -- 172.21.15.96:0/4017225972 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8e5c0b3a50 msgr2=0x7f8e5c1584e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:30.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:30.689+0000 7f8e5b400640 1 --2- 172.21.15.96:0/4017225972 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8e5c0b3a50 0x7f8e5c1584e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:30.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:30.689+0000 7f8e5b400640 1 -- 172.21.15.96:0/4017225972 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e5c0b3670 msgr2=0x7f8e5c0cab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:30.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:30.689+0000 7f8e5b400640 1 --2- 172.21.15.96:0/4017225972 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e5c0b3670 0x7f8e5c0cab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:30.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:30.689+0000 7f8e5b400640 1 -- 172.21.15.96:0/4017225972 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e5c0cdb80 con 0x7f8e5c1630d0 2024-05-28T06:13:30.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:30.689+0000 7f8e5b400640 1 --2- 172.21.15.96:0/4017225972 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8e5c1630d0 0x7f8e5c158a20 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f8e5400b440 tx=0x7f8e5400b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:13:30.693 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:30.690+0000 7f8e53e00640 1 -- 172.21.15.96:0/4017225972 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e54015020 con 0x7f8e5c1630d0 2024-05-28T06:13:30.693 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:30.690+0000 7f8e53e00640 1 -- 172.21.15.96:0/4017225972 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e54011830 con 0x7f8e5c1630d0 2024-05-28T06:13:30.693 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:30.690+0000 7f8e53e00640 1 -- 172.21.15.96:0/4017225972 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e54011c20 con 0x7f8e5c1630d0 2024-05-28T06:13:30.693 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:30.690+0000 7f8e61e00640 1 -- 172.21.15.96:0/4017225972 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e5c0cde00 con 0x7f8e5c1630d0 2024-05-28T06:13:30.693 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:30.690+0000 7f8e61e00640 1 -- 172.21.15.96:0/4017225972 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e5c0ce3c0 con 0x7f8e5c1630d0 2024-05-28T06:13:30.694 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:30.692+0000 7f8e61e00640 1 -- 172.21.15.96:0/4017225972 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e5c0689e0 con 0x7f8e5c1630d0 2024-05-28T06:13:30.694 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:30.693+0000 7f8e53e00640 1 -- 172.21.15.96:0/4017225972 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 19) ==== 92111+0+0 (secure 0 0 0) 0x7f8e54018020 con 0x7f8e5c1630d0 2024-05-28T06:13:30.696 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:30.695+0000 7f8e53e00640 1 --2- 172.21.15.96:0/4017225972 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8e3806d510 0x7f8e3806f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:30.696 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:30.695+0000 7f8e53e00640 1 -- 172.21.15.96:0/4017225972 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5737+0+0 (secure 0 0 0) 0x7f8e5408a6b0 con 0x7f8e5c1630d0 2024-05-28T06:13:30.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:30.699+0000 7f8e53e00640 1 -- 172.21.15.96:0/4017225972 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8e5401f020 con 0x7f8e5c1630d0 2024-05-28T06:13:30.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:30.721+0000 7f8e5aa00640 1 --2- 172.21.15.96:0/4017225972 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8e3806d510 0x7f8e3806f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:30.744 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:30.743+0000 7f8e5aa00640 1 --2- 172.21.15.96:0/4017225972 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8e3806d510 0x7f8e3806f9d0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f8e5c0ceb60 tx=0x7f8e4c0073e0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:13:31.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:31.132+0000 7f8e61e00640 1 -- 172.21.15.96:0/4017225972 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone81716", "target": ["mon-mgr", ""]}) -- 0x7f8e5c0ba5c0 con 0x7f8e3806d510 2024-05-28T06:13:31.401 DEBUG:teuthology.orchestra.run.smithi096:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:13:31.406 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:13:32.970 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:32.968+0000 7f8e53e00640 1 -- 172.21.15.96:0/4017225972 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f8e5c0ba5c0 con 0x7f8e3806d510 2024-05-28T06:13:32.974 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:32.972+0000 7f8e61e00640 1 -- 172.21.15.96:0/4017225972 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8e3806d510 msgr2=0x7f8e3806f9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:32.974 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:32.972+0000 7f8e61e00640 1 --2- 172.21.15.96:0/4017225972 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8e3806d510 0x7f8e3806f9d0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f8e5c0ceb60 tx=0x7f8e4c0073e0 comp rx=0 tx=0).stop 2024-05-28T06:13:32.974 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:32.972+0000 7f8e61e00640 1 -- 172.21.15.96:0/4017225972 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8e5c1630d0 msgr2=0x7f8e5c158a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:32.974 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:32.972+0000 7f8e61e00640 1 --2- 172.21.15.96:0/4017225972 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8e5c1630d0 0x7f8e5c158a20 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f8e5400b440 tx=0x7f8e5400b900 comp rx=0 tx=0).stop 2024-05-28T06:13:32.974 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:32.972+0000 7f8e61e00640 1 -- 172.21.15.96:0/4017225972 shutdown_connections 2024-05-28T06:13:32.974 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:32.972+0000 7f8e61e00640 1 --2- 172.21.15.96:0/4017225972 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8e3806d510 0x7f8e3806f9d0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:32.974 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:32.972+0000 7f8e61e00640 1 --2- 172.21.15.96:0/4017225972 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8e5c1630d0 0x7f8e5c158a20 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:32.974 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:32.972+0000 7f8e61e00640 1 --2- 172.21.15.96:0/4017225972 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8e5c0b3a50 0x7f8e5c1584e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:32.974 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:32.972+0000 7f8e61e00640 1 --2- 172.21.15.96:0/4017225972 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e5c0b3670 0x7f8e5c0cab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:32.974 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:32.972+0000 7f8e61e00640 1 -- 172.21.15.96:0/4017225972 >> 172.21.15.96:0/4017225972 conn(0x7f8e5c08bfc0 msgr2=0x7f8e5c0b24f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:13:32.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:32.972+0000 7f8e61e00640 1 -- 172.21.15.96:0/4017225972 shutdown_connections 2024-05-28T06:13:32.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:32.972+0000 7f8e61e00640 1 -- 172.21.15.96:0/4017225972 wait complete. 2024-05-28T06:13:32.996 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone81717 2024-05-28T06:13:33.308 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:33.307+0000 7f2484e00640 1 Processor -- start 2024-05-28T06:13:33.308 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:33.307+0000 7f2484e00640 1 -- start start 2024-05-28T06:13:33.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:33.308+0000 7f2484e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f24801525d0 0x7f2480073910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:33.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:33.308+0000 7f2484e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f24800d06d0 0x7f24800d0b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:33.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:33.308+0000 7f2484e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f24800b43d0 0x7f24800b4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:33.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:33.308+0000 7f2484e00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f248015f050 con 0x7f24800b43d0 2024-05-28T06:13:33.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:33.308+0000 7f2484e00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f24800d1050 con 0x7f24800d06d0 2024-05-28T06:13:33.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:33.308+0000 7f2484e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f24801529b0 con 0x7f24801525d0 2024-05-28T06:13:33.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:33.308+0000 7f247e800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f24800b43d0 0x7f24800b4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:33.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:33.309+0000 7f247e800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f24800b43d0 0x7f24800b4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49094/0 (socket says 172.21.15.96:49094) 2024-05-28T06:13:33.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:33.309+0000 7f247e800640 1 -- 172.21.15.96:0/860077665 learned_addr learned my addr 172.21.15.96:0/860077665 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:13:33.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:33.309+0000 7f247e800640 1 -- 172.21.15.96:0/860077665 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f24801525d0 msgr2=0x7f2480073910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:13:33.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:33.309+0000 7f247e800640 1 --2- 172.21.15.96:0/860077665 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f24801525d0 0x7f2480073910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:33.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:33.309+0000 7f247e800640 1 -- 172.21.15.96:0/860077665 >> v2:172.21.15.123:3300/0 conn(0x7f24800d06d0 msgr2=0x7f24800d0b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:13:33.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:33.309+0000 7f247e800640 1 --2- 172.21.15.96:0/860077665 >> v2:172.21.15.123:3300/0 conn(0x7f24800d06d0 0x7f24800d0b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:33.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:33.309+0000 7f247e800640 1 -- 172.21.15.96:0/860077665 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f248015f2a0 con 0x7f24800b43d0 2024-05-28T06:13:33.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:33.310+0000 7f247e800640 1 --2- 172.21.15.96:0/860077665 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f24800b43d0 0x7f24800b4860 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f248006c8c0 tx=0x7f247802c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6af1dd05ae056a08 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:13:33.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:33.310+0000 7f247ca00640 1 -- 172.21.15.96:0/860077665 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2478033400 con 0x7f24800b43d0 2024-05-28T06:13:33.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:33.310+0000 7f247ca00640 1 -- 172.21.15.96:0/860077665 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24780335a0 con 0x7f24800b43d0 2024-05-28T06:13:33.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:33.310+0000 7f247ca00640 1 -- 172.21.15.96:0/860077665 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2478033970 con 0x7f24800b43d0 2024-05-28T06:13:33.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:33.311+0000 7f2484e00640 1 -- 172.21.15.96:0/860077665 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f24800b43d0 msgr2=0x7f24800b4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:33.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:33.311+0000 7f2484e00640 1 --2- 172.21.15.96:0/860077665 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f24800b43d0 0x7f24800b4860 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f248006c8c0 tx=0x7f247802c8b0 comp rx=0 tx=0).stop 2024-05-28T06:13:33.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:33.311+0000 7f2484e00640 1 -- 172.21.15.96:0/860077665 shutdown_connections 2024-05-28T06:13:33.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:33.311+0000 7f2484e00640 1 --2- 172.21.15.96:0/860077665 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f24800b43d0 0x7f24800b4860 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:33.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:33.311+0000 7f2484e00640 1 --2- 172.21.15.96:0/860077665 >> v2:172.21.15.123:3300/0 conn(0x7f24800d06d0 0x7f24800d0b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:33.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:33.311+0000 7f2484e00640 1 --2- 172.21.15.96:0/860077665 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f24801525d0 0x7f2480073910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:33.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:33.311+0000 7f2484e00640 1 -- 172.21.15.96:0/860077665 >> 172.21.15.96:0/860077665 conn(0x7f248006bd20 msgr2=0x7f248006c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:13:33.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:33.311+0000 7f2484e00640 1 -- 172.21.15.96:0/860077665 shutdown_connections 2024-05-28T06:13:33.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:33.311+0000 7f2484e00640 1 -- 172.21.15.96:0/860077665 wait complete. 2024-05-28T06:13:33.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:33.314+0000 7f2484e00640 1 Processor -- start 2024-05-28T06:13:33.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:33.314+0000 7f2484e00640 1 -- start start 2024-05-28T06:13:33.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:33.315+0000 7f2484e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f24800b43d0 0x7f24800ae220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:33.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:33.315+0000 7f2484e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f24800d06d0 0x7f24800ce2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:33.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:33.315+0000 7f2484e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f24801525d0 0x7f24800ce7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:33.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:33.315+0000 7f2484e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f248015fb40 con 0x7f24800d06d0 2024-05-28T06:13:33.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:33.315+0000 7f2484e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f248015f860 con 0x7f24800b43d0 2024-05-28T06:13:33.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:33.315+0000 7f2484e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f248015fe20 con 0x7f24801525d0 2024-05-28T06:13:33.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:33.316+0000 7f247d400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f24800d06d0 0x7f24800ce2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:33.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:33.316+0000 7f247d400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f24800d06d0 0x7f24800ce2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49102/0 (socket says 172.21.15.96:49102) 2024-05-28T06:13:33.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:33.316+0000 7f247d400640 1 -- 172.21.15.96:0/2665402633 learned_addr learned my addr 172.21.15.96:0/2665402633 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:13:33.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:33.316+0000 7f247d400640 1 -- 172.21.15.96:0/2665402633 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f24801525d0 msgr2=0x7f24800ce7f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-28T06:13:33.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:33.316+0000 7f247d400640 1 --2- 172.21.15.96:0/2665402633 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f24801525d0 0x7f24800ce7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:33.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:33.316+0000 7f247d400640 1 -- 172.21.15.96:0/2665402633 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f24800b43d0 msgr2=0x7f24800ae220 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-28T06:13:33.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:33.316+0000 7f247d400640 1 --2- 172.21.15.96:0/2665402633 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f24800b43d0 0x7f24800ae220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:33.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:33.316+0000 7f247d400640 1 -- 172.21.15.96:0/2665402633 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24800ae760 con 0x7f24800d06d0 2024-05-28T06:13:33.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:33.317+0000 7f247d400640 1 --2- 172.21.15.96:0/2665402633 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f24800d06d0 0x7f24800ce2b0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f246c00b4d0 tx=0x7f246c00b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:13:33.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:33.317+0000 7f2476000640 1 -- 172.21.15.96:0/2665402633 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f246c014070 con 0x7f24800d06d0 2024-05-28T06:13:33.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:33.317+0000 7f2476000640 1 -- 172.21.15.96:0/2665402633 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f246c010870 con 0x7f24800d06d0 2024-05-28T06:13:33.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:33.317+0000 7f2476000640 1 -- 172.21.15.96:0/2665402633 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f246c010c60 con 0x7f24800d06d0 2024-05-28T06:13:33.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:33.317+0000 7f2484e00640 1 -- 172.21.15.96:0/2665402633 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f248014f750 con 0x7f24800d06d0 2024-05-28T06:13:33.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:33.317+0000 7f2484e00640 1 -- 172.21.15.96:0/2665402633 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f248014fd10 con 0x7f24800d06d0 2024-05-28T06:13:33.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:33.319+0000 7f2484e00640 1 -- 172.21.15.96:0/2665402633 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24800689e0 con 0x7f24800d06d0 2024-05-28T06:13:33.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:33.322+0000 7f2476000640 1 -- 172.21.15.96:0/2665402633 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 19) ==== 92111+0+0 (secure 0 0 0) 0x7f246c017070 con 0x7f24800d06d0 2024-05-28T06:13:33.328 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:33.327+0000 7f2476000640 1 --2- 172.21.15.96:0/2665402633 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f246006d510 0x7f246006f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:33.328 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:33.327+0000 7f2476000640 1 -- 172.21.15.96:0/2665402633 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5737+0+0 (secure 0 0 0) 0x7f246c08a700 con 0x7f24800d06d0 2024-05-28T06:13:33.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:33.332+0000 7f2476000640 1 -- 172.21.15.96:0/2665402633 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f246c05a5c0 con 0x7f24800d06d0 2024-05-28T06:13:33.349 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:33.348+0000 7f247de00640 1 --2- 172.21.15.96:0/2665402633 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f246006d510 0x7f246006f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:33.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:33.367+0000 7f247de00640 1 --2- 172.21.15.96:0/2665402633 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f246006d510 0x7f246006f9d0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f24700096b0 tx=0x7f2470006c20 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:13:33.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:33.681+0000 7f2484e00640 1 -- 172.21.15.96:0/2665402633 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone81717", "target": ["mon-mgr", ""]}) -- 0x7f2480072c00 con 0x7f246006d510 2024-05-28T06:13:34.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:34.707+0000 7f2476000640 1 -- 172.21.15.96:0/2665402633 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f2480072c00 con 0x7f246006d510 2024-05-28T06:13:34.713 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:34.711+0000 7f2484e00640 1 -- 172.21.15.96:0/2665402633 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f246006d510 msgr2=0x7f246006f9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:34.713 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:34.711+0000 7f2484e00640 1 --2- 172.21.15.96:0/2665402633 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f246006d510 0x7f246006f9d0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f24700096b0 tx=0x7f2470006c20 comp rx=0 tx=0).stop 2024-05-28T06:13:34.713 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:34.711+0000 7f2484e00640 1 -- 172.21.15.96:0/2665402633 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f24800d06d0 msgr2=0x7f24800ce2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:34.713 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:34.711+0000 7f2484e00640 1 --2- 172.21.15.96:0/2665402633 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f24800d06d0 0x7f24800ce2b0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f246c00b4d0 tx=0x7f246c00b990 comp rx=0 tx=0).stop 2024-05-28T06:13:34.714 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:34.712+0000 7f2484e00640 1 -- 172.21.15.96:0/2665402633 shutdown_connections 2024-05-28T06:13:34.714 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:34.712+0000 7f2484e00640 1 --2- 172.21.15.96:0/2665402633 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f246006d510 0x7f246006f9d0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:34.714 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:34.712+0000 7f2484e00640 1 --2- 172.21.15.96:0/2665402633 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f24801525d0 0x7f24800ce7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:34.714 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:34.712+0000 7f2484e00640 1 --2- 172.21.15.96:0/2665402633 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f24800d06d0 0x7f24800ce2b0 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:34.714 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:34.712+0000 7f2484e00640 1 --2- 172.21.15.96:0/2665402633 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f24800b43d0 0x7f24800ae220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:34.714 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:34.712+0000 7f2484e00640 1 -- 172.21.15.96:0/2665402633 >> 172.21.15.96:0/2665402633 conn(0x7f248006bd20 msgr2=0x7f24800b3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:13:34.714 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:34.712+0000 7f2484e00640 1 -- 172.21.15.96:0/2665402633 shutdown_connections 2024-05-28T06:13:34.714 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:34.712+0000 7f2484e00640 1 -- 172.21.15.96:0/2665402633 wait complete. 2024-05-28T06:13:34.751 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-05-28T06:13:35.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.026+0000 7f7f62600640 1 Processor -- start 2024-05-28T06:13:35.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.027+0000 7f7f62600640 1 -- start start 2024-05-28T06:13:35.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.029+0000 7f7f62600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7f5c09ba40 0x7f7f5c0b1480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:35.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.029+0000 7f7f62600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f7f5c06daa0 0x7f7f5c06df50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:35.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.029+0000 7f7f62600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f5c1634d0 con 0x7f7f5c15ed50 2024-05-28T06:13:35.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.029+0000 7f7f62600640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f7f5c0b1d10 con 0x7f7f5c06daa0 2024-05-28T06:13:35.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.029+0000 7f7f62600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f7f5c0b38d0 con 0x7f7f5c09ba40 2024-05-28T06:13:35.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.029+0000 7f7f61200640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f7f5c15ed50 0x7f7f5c0d0f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.96:44670/0 (socket says 172.21.15.96:44670) 2024-05-28T06:13:35.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.029+0000 7f7f61200640 1 -- 172.21.15.96:0/1508572036 learned_addr learned my addr 172.21.15.96:0/1508572036 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:13:35.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.029+0000 7f7f61c00640 1 --2- 172.21.15.96:0/1508572036 >> v2:172.21.15.123:3300/0 conn(0x7f7f5c06daa0 0x7f7f5c06df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:35.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.029+0000 7f7f5be00640 1 --2- 172.21.15.96:0/1508572036 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7f5c09ba40 0x7f7f5c0b1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:35.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.030+0000 7f7f5b400640 1 -- 172.21.15.96:0/1508572036 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4043974536 0 0) 0x7f7f5c1634d0 con 0x7f7f5c15ed50 2024-05-28T06:13:35.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.030+0000 7f7f5b400640 1 -- 172.21.15.96:0/1508572036 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7f44003520 con 0x7f7f5c15ed50 2024-05-28T06:13:35.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.030+0000 7f7f61c00640 1 -- 172.21.15.96:0/1508572036 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7f5c09ba40 msgr2=0x7f7f5c0b1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:13:35.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.030+0000 7f7f61c00640 1 --2- 172.21.15.96:0/1508572036 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7f5c09ba40 0x7f7f5c0b1480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:35.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.030+0000 7f7f61c00640 1 -- 172.21.15.96:0/1508572036 >> v1:172.21.15.123:6789/0 conn(0x7f7f5c15ed50 legacy=0x7f7f5c0d0f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:35.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.030+0000 7f7f61c00640 1 -- 172.21.15.96:0/1508572036 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f5c0b3bb0 con 0x7f7f5c06daa0 2024-05-28T06:13:35.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.032+0000 7f7f61c00640 1 --2- 172.21.15.96:0/1508572036 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f5c06daa0 0x7f7f5c06df50 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f7f54009d90 tx=0x7f7f5402a5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=25b5663639bdd796 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:13:35.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.032+0000 7f7f5b400640 1 -- 172.21.15.96:0/1508572036 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f54031600 con 0x7f7f5c06daa0 2024-05-28T06:13:35.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.033+0000 7f7f5b400640 1 -- 172.21.15.96:0/1508572036 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f540317a0 con 0x7f7f5c06daa0 2024-05-28T06:13:35.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.033+0000 7f7f5b400640 1 -- 172.21.15.96:0/1508572036 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f54031b50 con 0x7f7f5c06daa0 2024-05-28T06:13:35.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.034+0000 7f7f62600640 1 -- 172.21.15.96:0/1508572036 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f5c06daa0 msgr2=0x7f7f5c06df50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:35.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.034+0000 7f7f62600640 1 --2- 172.21.15.96:0/1508572036 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f5c06daa0 0x7f7f5c06df50 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f7f54009d90 tx=0x7f7f5402a5d0 comp rx=0 tx=0).stop 2024-05-28T06:13:35.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.034+0000 7f7f62600640 1 -- 172.21.15.96:0/1508572036 shutdown_connections 2024-05-28T06:13:35.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.034+0000 7f7f62600640 1 --2- 172.21.15.96:0/1508572036 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f5c06daa0 0x7f7f5c06df50 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:35.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.034+0000 7f7f62600640 1 --2- 172.21.15.96:0/1508572036 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7f5c09ba40 0x7f7f5c0b1480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:35.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.034+0000 7f7f62600640 1 -- 172.21.15.96:0/1508572036 >> 172.21.15.96:0/1508572036 conn(0x7f7f5c0b3ec0 msgr2=0x7f7f5c0b42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:13:35.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.034+0000 7f7f62600640 1 -- 172.21.15.96:0/1508572036 shutdown_connections 2024-05-28T06:13:35.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.034+0000 7f7f62600640 1 -- 172.21.15.96:0/1508572036 wait complete. 2024-05-28T06:13:35.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.037+0000 7f7f62600640 1 Processor -- start 2024-05-28T06:13:35.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.038+0000 7f7f62600640 1 -- start start 2024-05-28T06:13:35.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.040+0000 7f7f62600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f5c06daa0 0x7f7f5c1f9ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:35.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.040+0000 7f7f62600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f5c09ba40 0x7f7f5c0c7dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:35.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.040+0000 7f7f62600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7f5c15ed50 0x7f7f5c1598e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:35.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.040+0000 7f7f62600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f7f5c06e490 con 0x7f7f5c09ba40 2024-05-28T06:13:35.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.040+0000 7f7f62600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f7f5c0b4cc0 con 0x7f7f5c06daa0 2024-05-28T06:13:35.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.040+0000 7f7f62600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f7f5c06e610 con 0x7f7f5c15ed50 2024-05-28T06:13:35.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.041+0000 7f7f5be00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f5c09ba40 0x7f7f5c0c7dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:35.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.041+0000 7f7f5be00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f5c09ba40 0x7f7f5c0c7dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49108/0 (socket says 172.21.15.96:49108) 2024-05-28T06:13:35.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.041+0000 7f7f5be00640 1 -- 172.21.15.96:0/3713641584 learned_addr learned my addr 172.21.15.96:0/3713641584 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:13:35.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.041+0000 7f7f61200640 1 --2- 172.21.15.96:0/3713641584 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f5c06daa0 0x7f7f5c1f9ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:35.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.042+0000 7f7f5be00640 1 -- 172.21.15.96:0/3713641584 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7f5c15ed50 msgr2=0x7f7f5c1598e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-28T06:13:35.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.042+0000 7f7f5be00640 1 --2- 172.21.15.96:0/3713641584 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7f5c15ed50 0x7f7f5c1598e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:35.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.042+0000 7f7f5be00640 1 -- 172.21.15.96:0/3713641584 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f5c06daa0 msgr2=0x7f7f5c1f9ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:35.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.042+0000 7f7f5be00640 1 --2- 172.21.15.96:0/3713641584 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f5c06daa0 0x7f7f5c1f9ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:35.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.042+0000 7f7f5be00640 1 -- 172.21.15.96:0/3713641584 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f5c0c8300 con 0x7f7f5c09ba40 2024-05-28T06:13:35.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.042+0000 7f7f61200640 1 --2- 172.21.15.96:0/3713641584 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f5c06daa0 0x7f7f5c1f9ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-28T06:13:35.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.042+0000 7f7f5be00640 1 --2- 172.21.15.96:0/3713641584 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f5c09ba40 0x7f7f5c0c7dc0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f7f4800e870 tx=0x7f7f4800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:13:35.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.043+0000 7f7f58c00640 1 -- 172.21.15.96:0/3713641584 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f48017070 con 0x7f7f5c09ba40 2024-05-28T06:13:35.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.043+0000 7f7f58c00640 1 -- 172.21.15.96:0/3713641584 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f4800c940 con 0x7f7f5c09ba40 2024-05-28T06:13:35.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.043+0000 7f7f58c00640 1 -- 172.21.15.96:0/3713641584 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f4800cd10 con 0x7f7f5c09ba40 2024-05-28T06:13:35.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.043+0000 7f7f62600640 1 -- 172.21.15.96:0/3713641584 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f5c0c85a0 con 0x7f7f5c09ba40 2024-05-28T06:13:35.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.043+0000 7f7f62600640 1 -- 172.21.15.96:0/3713641584 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f5c0c8b60 con 0x7f7f5c09ba40 2024-05-28T06:13:35.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.046+0000 7f7f62600640 1 -- 172.21.15.96:0/3713641584 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f38005290 con 0x7f7f5c09ba40 2024-05-28T06:13:35.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.050+0000 7f7f58c00640 1 -- 172.21.15.96:0/3713641584 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 19) ==== 92111+0+0 (secure 0 0 0) 0x7f7f4801a070 con 0x7f7f5c09ba40 2024-05-28T06:13:35.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.054+0000 7f7f58c00640 1 --2- 172.21.15.96:0/3713641584 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7f3c06d590 0x7f7f3c06fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:35.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.054+0000 7f7f58c00640 1 -- 172.21.15.96:0/3713641584 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5737+0+0 (secure 0 0 0) 0x7f7f4808a6a0 con 0x7f7f5c09ba40 2024-05-28T06:13:35.065 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.063+0000 7f7f58c00640 1 -- 172.21.15.96:0/3713641584 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7f4805a2c0 con 0x7f7f5c09ba40 2024-05-28T06:13:35.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.083+0000 7f7f61200640 1 --2- 172.21.15.96:0/3713641584 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7f3c06d590 0x7f7f3c06fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:35.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.104+0000 7f7f61200640 1 --2- 172.21.15.96:0/3713641584 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7f3c06d590 0x7f7f3c06fa50 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f7f500011e0 tx=0x7f7f50009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:13:35.594 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.592+0000 7f7f62600640 1 -- 172.21.15.96:0/3713641584 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f7f38005850 con 0x7f7f5c09ba40 2024-05-28T06:13:35.594 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.593+0000 7f7f58c00640 1 -- 172.21.15.96:0/3713641584 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+2400 (secure 0 0 0) 0x7f7f48059bc0 con 0x7f7f5c09ba40 2024-05-28T06:13:35.595 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:13:35.595 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:13:35.595 INFO:teuthology.orchestra.run.smithi096.stdout: "fsid": "e4a831b5-e6d1-47a7-96cb-bf85154ee0d2", 2024-05-28T06:13:35.595 INFO:teuthology.orchestra.run.smithi096.stdout: "health": { 2024-05-28T06:13:35.595 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "HEALTH_OK", 2024-05-28T06:13:35.595 INFO:teuthology.orchestra.run.smithi096.stdout: "checks": {}, 2024-05-28T06:13:35.595 INFO:teuthology.orchestra.run.smithi096.stdout: "mutes": [] 2024-05-28T06:13:35.595 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:13:35.595 INFO:teuthology.orchestra.run.smithi096.stdout: "election_epoch": 8, 2024-05-28T06:13:35.595 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum": [ 2024-05-28T06:13:35.595 INFO:teuthology.orchestra.run.smithi096.stdout: 0, 2024-05-28T06:13:35.596 INFO:teuthology.orchestra.run.smithi096.stdout: 1, 2024-05-28T06:13:35.596 INFO:teuthology.orchestra.run.smithi096.stdout: 2 2024-05-28T06:13:35.596 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:13:35.596 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_names": [ 2024-05-28T06:13:35.596 INFO:teuthology.orchestra.run.smithi096.stdout: "a", 2024-05-28T06:13:35.596 INFO:teuthology.orchestra.run.smithi096.stdout: "b", 2024-05-28T06:13:35.596 INFO:teuthology.orchestra.run.smithi096.stdout: "c" 2024-05-28T06:13:35.596 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:13:35.596 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_age": 589, 2024-05-28T06:13:35.596 INFO:teuthology.orchestra.run.smithi096.stdout: "monmap": { 2024-05-28T06:13:35.596 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 1, 2024-05-28T06:13:35.596 INFO:teuthology.orchestra.run.smithi096.stdout: "min_mon_release_name": "squid", 2024-05-28T06:13:35.596 INFO:teuthology.orchestra.run.smithi096.stdout: "num_mons": 3 2024-05-28T06:13:35.596 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:13:35.597 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap": { 2024-05-28T06:13:35.597 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 60, 2024-05-28T06:13:35.597 INFO:teuthology.orchestra.run.smithi096.stdout: "num_osds": 8, 2024-05-28T06:13:35.597 INFO:teuthology.orchestra.run.smithi096.stdout: "num_up_osds": 8, 2024-05-28T06:13:35.597 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_up_since": 1716876238, 2024-05-28T06:13:35.597 INFO:teuthology.orchestra.run.smithi096.stdout: "num_in_osds": 8, 2024-05-28T06:13:35.597 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_in_since": 1716876229, 2024-05-28T06:13:35.597 INFO:teuthology.orchestra.run.smithi096.stdout: "num_remapped_pgs": 0 2024-05-28T06:13:35.597 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:13:35.597 INFO:teuthology.orchestra.run.smithi096.stdout: "pgmap": { 2024-05-28T06:13:35.597 INFO:teuthology.orchestra.run.smithi096.stdout: "pgs_by_state": [ 2024-05-28T06:13:35.597 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:13:35.597 INFO:teuthology.orchestra.run.smithi096.stdout: "state_name": "active+clean", 2024-05-28T06:13:35.597 INFO:teuthology.orchestra.run.smithi096.stdout: "count": 97 2024-05-28T06:13:35.598 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:13:35.598 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:13:35.598 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pgs": 97, 2024-05-28T06:13:35.598 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pools": 3, 2024-05-28T06:13:35.598 INFO:teuthology.orchestra.run.smithi096.stdout: "num_objects": 1456, 2024-05-28T06:13:35.598 INFO:teuthology.orchestra.run.smithi096.stdout: "data_bytes": 5957100635, 2024-05-28T06:13:35.598 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_used": 11530256384, 2024-05-28T06:13:35.598 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_avail": 761563856896, 2024-05-28T06:13:35.598 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_total": 773094113280, 2024-05-28T06:13:35.598 INFO:teuthology.orchestra.run.smithi096.stdout: "read_bytes_sec": 243433519, 2024-05-28T06:13:35.598 INFO:teuthology.orchestra.run.smithi096.stdout: "write_bytes_sec": 227751598, 2024-05-28T06:13:35.598 INFO:teuthology.orchestra.run.smithi096.stdout: "read_op_per_sec": 59, 2024-05-28T06:13:35.598 INFO:teuthology.orchestra.run.smithi096.stdout: "write_op_per_sec": 65 2024-05-28T06:13:35.598 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:13:35.599 INFO:teuthology.orchestra.run.smithi096.stdout: "fsmap": { 2024-05-28T06:13:35.599 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 35, 2024-05-28T06:13:35.599 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 4, 2024-05-28T06:13:35.599 INFO:teuthology.orchestra.run.smithi096.stdout: "up": 1, 2024-05-28T06:13:35.599 INFO:teuthology.orchestra.run.smithi096.stdout: "in": 1, 2024-05-28T06:13:35.599 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 1, 2024-05-28T06:13:35.599 INFO:teuthology.orchestra.run.smithi096.stdout: "by_rank": [ 2024-05-28T06:13:35.599 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:13:35.599 INFO:teuthology.orchestra.run.smithi096.stdout: "filesystem_id": 4, 2024-05-28T06:13:35.599 INFO:teuthology.orchestra.run.smithi096.stdout: "rank": 0, 2024-05-28T06:13:35.599 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "a", 2024-05-28T06:13:35.599 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "up:active", 2024-05-28T06:13:35.600 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 6489 2024-05-28T06:13:35.600 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:13:35.600 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:13:35.600 INFO:teuthology.orchestra.run.smithi096.stdout: "up:standby": 3 2024-05-28T06:13:35.600 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:13:35.600 INFO:teuthology.orchestra.run.smithi096.stdout: "mgrmap": { 2024-05-28T06:13:35.600 INFO:teuthology.orchestra.run.smithi096.stdout: "available": true, 2024-05-28T06:13:35.600 INFO:teuthology.orchestra.run.smithi096.stdout: "num_standbys": 1, 2024-05-28T06:13:35.600 INFO:teuthology.orchestra.run.smithi096.stdout: "modules": [ 2024-05-28T06:13:35.600 INFO:teuthology.orchestra.run.smithi096.stdout: "iostat", 2024-05-28T06:13:35.600 INFO:teuthology.orchestra.run.smithi096.stdout: "nfs", 2024-05-28T06:13:35.600 INFO:teuthology.orchestra.run.smithi096.stdout: "restful" 2024-05-28T06:13:35.601 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:13:35.601 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:13:35.601 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:13:35.601 INFO:teuthology.orchestra.run.smithi096.stdout: "servicemap": { 2024-05-28T06:13:35.601 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 11, 2024-05-28T06:13:35.601 INFO:teuthology.orchestra.run.smithi096.stdout: "modified": "2024-05-28T06:13:33.716527+0000", 2024-05-28T06:13:35.601 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:13:35.601 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:13:35.601 INFO:teuthology.orchestra.run.smithi096.stdout: "progress_events": { 2024-05-28T06:13:35.601 INFO:teuthology.orchestra.run.smithi096.stdout: "58c36a82-8e7f-41c9-9e90-890fce489f3b": { 2024-05-28T06:13:35.601 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "4 ongoing clones; Avg progress is 21.354% (6s)\n [=====.......................] (remaining: 22s)", 2024-05-28T06:13:35.601 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": 0.21354000270366669, 2024-05-28T06:13:35.601 INFO:teuthology.orchestra.run.smithi096.stdout: "add_to_ceph_s": true 2024-05-28T06:13:35.601 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:13:35.602 INFO:teuthology.orchestra.run.smithi096.stdout: "6a73bfaa-3b9e-443d-83e8-82fc1faa8b7f": { 2024-05-28T06:13:35.602 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "5 ongoing+pending clones; Avg progress is 26.927% (0s)\n [=======.....................] ", 2024-05-28T06:13:35.602 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": 0.26927000284194946, 2024-05-28T06:13:35.602 INFO:teuthology.orchestra.run.smithi096.stdout: "add_to_ceph_s": true 2024-05-28T06:13:35.602 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:13:35.602 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:13:35.602 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:13:35.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.596+0000 7f7f62600640 1 -- 172.21.15.96:0/3713641584 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7f3c06d590 msgr2=0x7f7f3c06fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:35.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.596+0000 7f7f62600640 1 --2- 172.21.15.96:0/3713641584 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7f3c06d590 0x7f7f3c06fa50 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f7f500011e0 tx=0x7f7f50009040 comp rx=0 tx=0).stop 2024-05-28T06:13:35.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.596+0000 7f7f62600640 1 -- 172.21.15.96:0/3713641584 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f5c09ba40 msgr2=0x7f7f5c0c7dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:35.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.596+0000 7f7f62600640 1 --2- 172.21.15.96:0/3713641584 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f5c09ba40 0x7f7f5c0c7dc0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f7f4800e870 tx=0x7f7f4800ed30 comp rx=0 tx=0).stop 2024-05-28T06:13:35.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.596+0000 7f7f62600640 1 -- 172.21.15.96:0/3713641584 shutdown_connections 2024-05-28T06:13:35.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.596+0000 7f7f62600640 1 --2- 172.21.15.96:0/3713641584 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7f3c06d590 0x7f7f3c06fa50 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:35.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.596+0000 7f7f62600640 1 --2- 172.21.15.96:0/3713641584 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7f5c15ed50 0x7f7f5c1598e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:35.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.596+0000 7f7f62600640 1 --2- 172.21.15.96:0/3713641584 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f5c09ba40 0x7f7f5c0c7dc0 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:35.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.596+0000 7f7f62600640 1 --2- 172.21.15.96:0/3713641584 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f5c06daa0 0x7f7f5c1f9ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:35.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.596+0000 7f7f62600640 1 -- 172.21.15.96:0/3713641584 >> 172.21.15.96:0/3713641584 conn(0x7f7f5c0b3ec0 msgr2=0x7f7f5c0cf6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:13:35.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.596+0000 7f7f62600640 1 -- 172.21.15.96:0/3713641584 shutdown_connections 2024-05-28T06:13:35.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.596+0000 7f7f62600640 1 -- 172.21.15.96:0/3713641584 wait complete. 2024-05-28T06:13:35.632 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone81711 2024-05-28T06:13:35.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.951+0000 7fadb9e00640 1 Processor -- start 2024-05-28T06:13:35.953 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.951+0000 7fadb9e00640 1 -- start start 2024-05-28T06:13:35.956 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.955+0000 7fadb9e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fadb40cef30 0x7fadb40d1640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:35.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.955+0000 7fadb9e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fadb40d1b80 0x7fadb40b90a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:35.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.955+0000 7fadb9e00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fadb40b8000 con 0x7fadb415e850 2024-05-28T06:13:35.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.955+0000 7fadb9e00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fadb40d1ff0 con 0x7fadb40d1b80 2024-05-28T06:13:35.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.955+0000 7fadb9e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fadb415ec30 con 0x7fadb40cef30 2024-05-28T06:13:35.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.955+0000 7fadb2a00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fadb415e850 0x7fadb40b77b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:54334/0 (socket says 172.21.15.96:54334) 2024-05-28T06:13:35.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.955+0000 7fadb2a00640 1 -- 172.21.15.96:0/2253847691 learned_addr learned my addr 172.21.15.96:0/2253847691 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:13:35.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.956+0000 7fadb1600640 1 -- 172.21.15.96:0/2253847691 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3721682760 0 0) 0x7fadb40b8000 con 0x7fadb415e850 2024-05-28T06:13:35.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.956+0000 7fadb1600640 1 -- 172.21.15.96:0/2253847691 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fad98003520 con 0x7fadb415e850 2024-05-28T06:13:35.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.957+0000 7fadb2000640 1 --2- 172.21.15.96:0/2253847691 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fadb40cef30 0x7fadb40d1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:35.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.957+0000 7fadb1600640 1 -- 172.21.15.96:0/2253847691 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 202055720 0 0) 0x7fad98003520 con 0x7fadb415e850 2024-05-28T06:13:35.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.957+0000 7fadb1600640 1 -- 172.21.15.96:0/2253847691 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fadb40cef30 msgr2=0x7fadb40d1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:13:35.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.957+0000 7fadb3400640 1 --2- 172.21.15.96:0/2253847691 >> v2:172.21.15.123:3300/0 conn(0x7fadb40d1b80 0x7fadb40b90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:35.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.957+0000 7fadb1600640 1 --2- 172.21.15.96:0/2253847691 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fadb40cef30 0x7fadb40d1640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:35.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.957+0000 7fadb1600640 1 -- 172.21.15.96:0/2253847691 >> v2:172.21.15.123:3300/0 conn(0x7fadb40d1b80 msgr2=0x7fadb40b90a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:13:35.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.957+0000 7fadb1600640 1 --2- 172.21.15.96:0/2253847691 >> v2:172.21.15.123:3300/0 conn(0x7fadb40d1b80 0x7fadb40b90a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:35.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.957+0000 7fadb1600640 1 -- 172.21.15.96:0/2253847691 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadb40b82e0 con 0x7fadb415e850 2024-05-28T06:13:35.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.957+0000 7fadb1600640 1 -- 172.21.15.96:0/2253847691 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7fada4001670 con 0x7fadb415e850 2024-05-28T06:13:35.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.957+0000 7fadb1600640 1 -- 172.21.15.96:0/2253847691 >> v1:172.21.15.96:6789/0 conn(0x7fadb415e850 legacy=0x7fadb40b77b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:35.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.958+0000 7fadb1600640 1 --2- 172.21.15.96:0/2253847691 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fadb40cef30 0x7fad980039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:35.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.958+0000 7fadb1600640 1 --2- 172.21.15.96:0/2253847691 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fad98004030 0x7fad98004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:35.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.958+0000 7fadb1600640 1 --2- 172.21.15.96:0/2253847691 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad98004d40 0x7fad980051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:35.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.958+0000 7fadb1600640 1 -- 172.21.15.96:0/2253847691 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fadb40d1ff0 con 0x7fad98004d40 2024-05-28T06:13:35.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.958+0000 7fadb1600640 1 -- 172.21.15.96:0/2253847691 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fad9800bc20 con 0x7fad98004030 2024-05-28T06:13:35.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.958+0000 7fadb1600640 1 -- 172.21.15.96:0/2253847691 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fad9800be20 con 0x7fadb40cef30 2024-05-28T06:13:35.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.958+0000 7fadb3400640 1 --2- 172.21.15.96:0/2253847691 >> v2:172.21.15.123:3300/0 conn(0x7fadb40d1b80 0x7fadb40b90a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:13:35.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.959+0000 7fadb2000640 1 --2- 172.21.15.96:0/2253847691 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fad98004030 0x7fad98004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:35.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.959+0000 7fadb3400640 1 --2- 172.21.15.96:0/2253847691 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad98004d40 0x7fad980051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:35.962 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.960+0000 7fadb3400640 1 -- 172.21.15.96:0/2253847691 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fadb40cef30 msgr2=0x7fad980039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:13:35.963 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.960+0000 7fadb3400640 1 --2- 172.21.15.96:0/2253847691 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fadb40cef30 0x7fad980039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:35.963 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.960+0000 7fadb3400640 1 -- 172.21.15.96:0/2253847691 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fad98004030 msgr2=0x7fad98004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:13:35.963 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.960+0000 7fadb3400640 1 --2- 172.21.15.96:0/2253847691 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fad98004030 0x7fad98004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:35.963 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.960+0000 7fadb3400640 1 -- 172.21.15.96:0/2253847691 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad98003520 con 0x7fad98004d40 2024-05-28T06:13:35.963 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.962+0000 7fadb3400640 1 --2- 172.21.15.96:0/2253847691 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad98004d40 0x7fad980051c0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fadac00c410 tx=0x7fad98005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a767e1bf8c8ef8c3 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:13:35.964 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.962+0000 7fadb1600640 1 -- 172.21.15.96:0/2253847691 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadac007c00 con 0x7fad98004d40 2024-05-28T06:13:35.964 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.962+0000 7fadb1600640 1 -- 172.21.15.96:0/2253847691 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fadac003b90 con 0x7fad98004d40 2024-05-28T06:13:35.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.962+0000 7fadb1600640 1 -- 172.21.15.96:0/2253847691 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadac02b970 con 0x7fad98004d40 2024-05-28T06:13:35.967 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.965+0000 7fadb9e00640 1 -- 172.21.15.96:0/2253847691 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad98004d40 msgr2=0x7fad980051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:35.967 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.965+0000 7fadb9e00640 1 --2- 172.21.15.96:0/2253847691 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad98004d40 0x7fad980051c0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fadac00c410 tx=0x7fad98005830 comp rx=0 tx=0).stop 2024-05-28T06:13:35.968 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.965+0000 7fadb3400640 1 -- 172.21.15.96:0/2253847691 reap_dead start 2024-05-28T06:13:35.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.966+0000 7fadb9e00640 1 -- 172.21.15.96:0/2253847691 shutdown_connections 2024-05-28T06:13:35.970 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.966+0000 7fadb9e00640 1 -- 172.21.15.96:0/2253847691 >> 172.21.15.96:0/2253847691 conn(0x7fadb4067aa0 msgr2=0x7fadb4067eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:13:35.970 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.967+0000 7fadb9e00640 1 -- 172.21.15.96:0/2253847691 shutdown_connections 2024-05-28T06:13:35.972 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.968+0000 7fadb9e00640 1 -- 172.21.15.96:0/2253847691 wait complete. 2024-05-28T06:13:35.974 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.972+0000 7fadb9e00640 1 Processor -- start 2024-05-28T06:13:35.974 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.973+0000 7fadb9e00640 1 -- start start 2024-05-28T06:13:35.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.975+0000 7fadb9e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad98004d40 0x7fadb4159e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:35.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.975+0000 7fadb9e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fadb415ca70 0x7fadb4158720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:35.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.976+0000 7fadb9e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fadb4159480 0x7fadb415b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:35.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.976+0000 7fadb9e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fadb40d1640 con 0x7fad98004d40 2024-05-28T06:13:35.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.976+0000 7fadb9e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fadb40b88a0 con 0x7fadb4159480 2024-05-28T06:13:35.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.976+0000 7fadb9e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fadb40d17c0 con 0x7fadb415ca70 2024-05-28T06:13:35.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.977+0000 7fadb3400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fadb4159480 0x7fadb415b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:35.979 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.977+0000 7fadb2a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad98004d40 0x7fadb4159e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:35.979 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.977+0000 7fadb3400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fadb4159480 0x7fadb415b660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.96:45604/0 (socket says 172.21.15.96:45604) 2024-05-28T06:13:35.979 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.977+0000 7fadb3400640 1 -- 172.21.15.96:0/960675611 learned_addr learned my addr 172.21.15.96:0/960675611 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:13:35.979 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.978+0000 7fadb2000640 1 --2- 172.21.15.96:0/960675611 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fadb415ca70 0x7fadb4158720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:35.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.979+0000 7fadb2a00640 1 -- 172.21.15.96:0/960675611 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fadb415ca70 msgr2=0x7fadb4158720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:35.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.979+0000 7fadb2a00640 1 --2- 172.21.15.96:0/960675611 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fadb415ca70 0x7fadb4158720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:35.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.979+0000 7fadb2a00640 1 -- 172.21.15.96:0/960675611 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fadb4159480 msgr2=0x7fadb415b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:35.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.979+0000 7fadb2a00640 1 --2- 172.21.15.96:0/960675611 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fadb4159480 0x7fadb415b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:35.982 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.979+0000 7fadb2a00640 1 -- 172.21.15.96:0/960675611 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadb40c6db0 con 0x7fad98004d40 2024-05-28T06:13:35.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.982+0000 7fadb2a00640 1 --2- 172.21.15.96:0/960675611 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad98004d40 0x7fadb4159e90 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fada400c730 tx=0x7fada400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:13:35.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.983+0000 7fadab400640 1 -- 172.21.15.96:0/960675611 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fada4018070 con 0x7fad98004d40 2024-05-28T06:13:35.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.983+0000 7fadab400640 1 -- 172.21.15.96:0/960675611 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fada40074e0 con 0x7fad98004d40 2024-05-28T06:13:35.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.983+0000 7fadab400640 1 -- 172.21.15.96:0/960675611 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fada4004a20 con 0x7fad98004d40 2024-05-28T06:13:35.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.983+0000 7fadb9e00640 1 -- 172.21.15.96:0/960675611 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fadb40c7090 con 0x7fad98004d40 2024-05-28T06:13:35.986 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.984+0000 7fadb9e00640 1 -- 172.21.15.96:0/960675611 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fadb4158c60 con 0x7fad98004d40 2024-05-28T06:13:35.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.988+0000 7fadb9e00640 1 -- 172.21.15.96:0/960675611 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fadb40689e0 con 0x7fad98004d40 2024-05-28T06:13:35.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.990+0000 7fadab400640 1 -- 172.21.15.96:0/960675611 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 19) ==== 92111+0+0 (secure 0 0 0) 0x7fada401b070 con 0x7fad98004d40 2024-05-28T06:13:35.998 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.992+0000 7fadab400640 1 --2- 172.21.15.96:0/960675611 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fad9406d510 0x7fad9406f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:35.998 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:35.993+0000 7fadab400640 1 -- 172.21.15.96:0/960675611 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5737+0+0 (secure 0 0 0) 0x7fada4089d30 con 0x7fad98004d40 2024-05-28T06:13:36.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:36.003+0000 7fadab400640 1 -- 172.21.15.96:0/960675611 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fada4059340 con 0x7fad98004d40 2024-05-28T06:13:36.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:36.036+0000 7fadb2000640 1 --2- 172.21.15.96:0/960675611 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fad9406d510 0x7fad9406f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:36.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:36.040+0000 7fadb2000640 1 --2- 172.21.15.96:0/960675611 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fad9406d510 0x7fad9406f9d0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fadb41fbe30 tx=0x7fada0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:13:36.410 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:36.408+0000 7fadb9e00640 1 -- 172.21.15.96:0/960675611 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone81711", "target": ["mon-mgr", ""]}) -- 0x7fadb40b4ff0 con 0x7fad9406d510 2024-05-28T06:13:36.549 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:13:36.546+0000 7fea59e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:13:36.549 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:13:36.546+0000 7fea59e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:13:36.549 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:13:36.546+0000 7fea59e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:13:36.549 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:13:36.546+0000 7fea59e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:13:36.549 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:13:36.546+0000 7fea59e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:13:38.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:38.442+0000 7fadab400640 1 -- 172.21.15.96:0/960675611 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fadb40b4ff0 con 0x7fad9406d510 2024-05-28T06:13:38.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:38.445+0000 7fadb9e00640 1 -- 172.21.15.96:0/960675611 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fad9406d510 msgr2=0x7fad9406f9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:38.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:38.445+0000 7fadb9e00640 1 --2- 172.21.15.96:0/960675611 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fad9406d510 0x7fad9406f9d0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fadb41fbe30 tx=0x7fada0008040 comp rx=0 tx=0).stop 2024-05-28T06:13:38.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:38.445+0000 7fadb9e00640 1 -- 172.21.15.96:0/960675611 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad98004d40 msgr2=0x7fadb4159e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:38.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:38.445+0000 7fadb9e00640 1 --2- 172.21.15.96:0/960675611 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad98004d40 0x7fadb4159e90 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fada400c730 tx=0x7fada400cbf0 comp rx=0 tx=0).stop 2024-05-28T06:13:38.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:38.445+0000 7fadb9e00640 1 -- 172.21.15.96:0/960675611 shutdown_connections 2024-05-28T06:13:38.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:38.445+0000 7fadb9e00640 1 --2- 172.21.15.96:0/960675611 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fad9406d510 0x7fad9406f9d0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:38.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:38.445+0000 7fadb9e00640 1 --2- 172.21.15.96:0/960675611 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fadb4159480 0x7fadb415b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:38.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:38.445+0000 7fadb9e00640 1 --2- 172.21.15.96:0/960675611 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fadb415ca70 0x7fadb4158720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:38.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:38.445+0000 7fadb9e00640 1 --2- 172.21.15.96:0/960675611 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad98004d40 0x7fadb4159e90 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:38.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:38.445+0000 7fadb9e00640 1 -- 172.21.15.96:0/960675611 >> 172.21.15.96:0/960675611 conn(0x7fadb4067aa0 msgr2=0x7fadb40b5640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:13:38.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:38.446+0000 7fadb9e00640 1 -- 172.21.15.96:0/960675611 shutdown_connections 2024-05-28T06:13:38.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:38.446+0000 7fadb9e00640 1 -- 172.21.15.96:0/960675611 wait complete. 2024-05-28T06:13:38.484 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone81712 2024-05-28T06:13:38.742 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:38.740+0000 7ff81aa00640 1 Processor -- start 2024-05-28T06:13:38.743 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:38.741+0000 7ff81aa00640 1 -- start start 2024-05-28T06:13:38.744 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:38.742+0000 7ff81aa00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7ff81415e850 0x7ff8140b77b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:38.745 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:38.742+0000 7ff81aa00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7ff8140cef30 0x7ff8140d1640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:38.745 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:38.742+0000 7ff81aa00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8140d1f60 con 0x7ff8140d1b80 2024-05-28T06:13:38.745 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:38.742+0000 7ff81aa00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7ff81415ec30 con 0x7ff8140cef30 2024-05-28T06:13:38.745 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:38.742+0000 7ff81aa00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7ff8140b8000 con 0x7ff81415e850 2024-05-28T06:13:38.745 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:38.743+0000 7ff813400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7ff8140cef30 0x7ff8140d1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:38.745 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:38.743+0000 7ff818a00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7ff8140d1b80 0x7ff814162340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:54346/0 (socket says 172.21.15.96:54346) 2024-05-28T06:13:38.745 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:38.743+0000 7ff813400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7ff8140cef30 0x7ff8140d1640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49150/0 (socket says 172.21.15.96:49150) 2024-05-28T06:13:38.745 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:38.743+0000 7ff818a00640 1 -- 172.21.15.96:0/2427947851 learned_addr learned my addr 172.21.15.96:0/2427947851 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:13:38.745 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:38.743+0000 7ff813400640 1 -- 172.21.15.96:0/2427947851 >> v2:172.21.15.123:3300/0 conn(0x7ff81415e850 msgr2=0x7ff8140b77b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:13:38.745 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:38.743+0000 7ff813400640 1 --2- 172.21.15.96:0/2427947851 >> v2:172.21.15.123:3300/0 conn(0x7ff81415e850 0x7ff8140b77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:38.745 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:38.743+0000 7ff813400640 1 -- 172.21.15.96:0/2427947851 >> v1:172.21.15.96:6789/0 conn(0x7ff8140d1b80 legacy=0x7ff814162340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:38.746 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:38.743+0000 7ff812a00640 1 -- 172.21.15.96:0/2427947851 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2030510064 0 0) 0x7ff8140d1f60 con 0x7ff8140d1b80 2024-05-28T06:13:38.746 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:38.743+0000 7ff813400640 1 -- 172.21.15.96:0/2427947851 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8140b82e0 con 0x7ff8140cef30 2024-05-28T06:13:38.746 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:38.745+0000 7ff813400640 1 --2- 172.21.15.96:0/2427947851 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff8140cef30 0x7ff8140d1640 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7ff81406dba0 tx=0x7ff80002a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46d6e4d77139bc0e server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:13:38.747 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:38.745+0000 7ff812a00640 1 -- 172.21.15.96:0/2427947851 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff800037070 con 0x7ff8140cef30 2024-05-28T06:13:38.747 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:38.745+0000 7ff812a00640 1 -- 172.21.15.96:0/2427947851 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff800032440 con 0x7ff8140cef30 2024-05-28T06:13:38.747 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:38.745+0000 7ff812a00640 1 -- 172.21.15.96:0/2427947851 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff80002ae60 con 0x7ff8140cef30 2024-05-28T06:13:38.748 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:38.746+0000 7ff81aa00640 1 -- 172.21.15.96:0/2427947851 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff8140cef30 msgr2=0x7ff8140d1640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:38.748 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:38.746+0000 7ff81aa00640 1 --2- 172.21.15.96:0/2427947851 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff8140cef30 0x7ff8140d1640 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7ff81406dba0 tx=0x7ff80002a110 comp rx=0 tx=0).stop 2024-05-28T06:13:38.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:38.747+0000 7ff81aa00640 1 -- 172.21.15.96:0/2427947851 shutdown_connections 2024-05-28T06:13:38.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:38.747+0000 7ff81aa00640 1 --2- 172.21.15.96:0/2427947851 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff8140cef30 0x7ff8140d1640 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:38.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:38.747+0000 7ff81aa00640 1 --2- 172.21.15.96:0/2427947851 >> v2:172.21.15.123:3300/0 conn(0x7ff81415e850 0x7ff8140b77b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:38.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:38.747+0000 7ff81aa00640 1 -- 172.21.15.96:0/2427947851 >> 172.21.15.96:0/2427947851 conn(0x7ff814067aa0 msgr2=0x7ff814067eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:13:38.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:38.747+0000 7ff81aa00640 1 -- 172.21.15.96:0/2427947851 shutdown_connections 2024-05-28T06:13:38.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:38.747+0000 7ff81aa00640 1 -- 172.21.15.96:0/2427947851 wait complete. 2024-05-28T06:13:38.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:38.749+0000 7ff81aa00640 1 Processor -- start 2024-05-28T06:13:38.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:38.749+0000 7ff81aa00640 1 -- start start 2024-05-28T06:13:38.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:38.750+0000 7ff81aa00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8140cef30 0x7ff8140cd490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:38.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:38.750+0000 7ff81aa00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff8140d1b80 0x7ff81414ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:38.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:38.750+0000 7ff81aa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff81415e850 0x7ff8140ab0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:38.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:38.750+0000 7ff81aa00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff8140b7cf0 con 0x7ff81415e850 2024-05-28T06:13:38.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:38.750+0000 7ff81aa00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff8140b88a0 con 0x7ff8140cef30 2024-05-28T06:13:38.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:38.750+0000 7ff81aa00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ff8140ba0b0 con 0x7ff8140d1b80 2024-05-28T06:13:38.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:38.751+0000 7ff818a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff81415e850 0x7ff8140ab0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:38.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:38.751+0000 7ff818a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff81415e850 0x7ff8140ab0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49156/0 (socket says 172.21.15.96:49156) 2024-05-28T06:13:38.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:38.751+0000 7ff818a00640 1 -- 172.21.15.96:0/4162588398 learned_addr learned my addr 172.21.15.96:0/4162588398 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:13:38.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:38.751+0000 7ff813e00640 1 --2- 172.21.15.96:0/4162588398 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8140cef30 0x7ff8140cd490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:38.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:38.751+0000 7ff813400640 1 --2- 172.21.15.96:0/4162588398 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff8140d1b80 0x7ff81414ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:38.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:38.751+0000 7ff818a00640 1 -- 172.21.15.96:0/4162588398 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff8140d1b80 msgr2=0x7ff81414ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:38.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:38.751+0000 7ff818a00640 1 --2- 172.21.15.96:0/4162588398 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff8140d1b80 0x7ff81414ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:38.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:38.751+0000 7ff818a00640 1 -- 172.21.15.96:0/4162588398 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8140cef30 msgr2=0x7ff8140cd490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:38.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:38.751+0000 7ff818a00640 1 --2- 172.21.15.96:0/4162588398 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8140cef30 0x7ff8140cd490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:38.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:38.752+0000 7ff818a00640 1 -- 172.21.15.96:0/4162588398 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8140cdce0 con 0x7ff81415e850 2024-05-28T06:13:38.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:38.752+0000 7ff813e00640 1 --2- 172.21.15.96:0/4162588398 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8140cef30 0x7ff8140cd490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-28T06:13:38.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:38.752+0000 7ff818a00640 1 --2- 172.21.15.96:0/4162588398 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff81415e850 0x7ff8140ab0c0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7ff80c00c7a0 tx=0x7ff80c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:13:38.755 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:38.752+0000 7ff80be00640 1 -- 172.21.15.96:0/4162588398 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff80c018070 con 0x7ff81415e850 2024-05-28T06:13:38.755 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:38.752+0000 7ff81aa00640 1 -- 172.21.15.96:0/4162588398 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8140cdfc0 con 0x7ff81415e850 2024-05-28T06:13:38.755 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:38.752+0000 7ff80be00640 1 -- 172.21.15.96:0/4162588398 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff80c00a640 con 0x7ff81415e850 2024-05-28T06:13:38.755 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:38.752+0000 7ff81aa00640 1 -- 172.21.15.96:0/4162588398 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8140ce580 con 0x7ff81415e850 2024-05-28T06:13:38.755 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:38.752+0000 7ff80be00640 1 -- 172.21.15.96:0/4162588398 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff80c00aab0 con 0x7ff81415e850 2024-05-28T06:13:38.755 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:38.753+0000 7ff81aa00640 1 -- 172.21.15.96:0/4162588398 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7ec005290 con 0x7ff81415e850 2024-05-28T06:13:38.757 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:38.755+0000 7ff80be00640 1 -- 172.21.15.96:0/4162588398 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 20) ==== 92162+0+0 (secure 0 0 0) 0x7ff80c01b070 con 0x7ff81415e850 2024-05-28T06:13:38.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:38.756+0000 7ff80be00640 1 --2- 172.21.15.96:0/4162588398 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff7f406d550 0x7ff7f406fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:38.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:38.757+0000 7ff80be00640 1 -- 172.21.15.96:0/4162588398 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5737+0+0 (secure 0 0 0) 0x7ff80c089d40 con 0x7ff81415e850 2024-05-28T06:13:38.759 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:38.758+0000 7ff813e00640 1 --2- 172.21.15.96:0/4162588398 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff7f406d550 0x7ff7f406fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:38.760 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:38.759+0000 7ff813e00640 1 --2- 172.21.15.96:0/4162588398 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff7f406d550 0x7ff7f406fa10 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7ff804009cb0 tx=0x7ff804009060 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:13:38.763 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:38.761+0000 7ff80be00640 1 -- 172.21.15.96:0/4162588398 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff80c0592f0 con 0x7ff81415e850 2024-05-28T06:13:39.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:39.126+0000 7ff81aa00640 1 -- 172.21.15.96:0/4162588398 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone81712", "target": ["mon-mgr", ""]}) -- 0x7ff7ec000bc0 con 0x7ff7f406d550 2024-05-28T06:13:39.228 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:13:39.224+0000 7fea59e00640 -1 mgr.server reply reply (22) Invalid argument cannot cancel -- clone finished (check clone status) 2024-05-28T06:13:39.231 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:39.229+0000 7ff80be00640 1 -- 172.21.15.96:0/4162588398 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: -22 cannot cancel -- clone finished (check clone status)) ==== 60+0+0 (secure 0 0 0) 0x7ff7ec000bc0 con 0x7ff7f406d550 2024-05-28T06:13:39.231 INFO:teuthology.orchestra.run.smithi096.stderr:Error EINVAL: cannot cancel -- clone finished (check clone status) 2024-05-28T06:13:39.244 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-05-28T06:13:39.245 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone81713 2024-05-28T06:13:39.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:39.502+0000 7f1c13e00640 1 Processor -- start 2024-05-28T06:13:39.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:39.503+0000 7f1c13e00640 1 -- start start 2024-05-28T06:13:39.506 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:39.504+0000 7f1c13e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1c0c15d5d0 0x7f1c0c0cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:39.506 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:39.504+0000 7f1c13e00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c0c15cb10 con 0x7f1c0c0b32c0 2024-05-28T06:13:39.506 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:39.504+0000 7f1c13e00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c0c0ad5f0 con 0x7f1c0c0d0640 2024-05-28T06:13:39.506 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:39.504+0000 7f1c13e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f1c0c0adbb0 con 0x7f1c0c15d5d0 2024-05-28T06:13:39.506 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:39.505+0000 7f1c11e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f1c0c0b32c0 0x7f1c0c0b3750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:53462/0 (socket says 172.21.15.96:53462) 2024-05-28T06:13:39.506 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:39.505+0000 7f1c11e00640 1 -- 172.21.15.96:0/1450017178 learned_addr learned my addr 172.21.15.96:0/1450017178 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:13:39.508 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:39.505+0000 7f1c0be00640 1 -- 172.21.15.96:0/1450017178 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3285066430 0 0) 0x7f1c0c15cb10 con 0x7f1c0c0b32c0 2024-05-28T06:13:39.508 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:39.505+0000 7f1c10a00640 1 --2- 172.21.15.96:0/1450017178 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1c0c15d5d0 0x7f1c0c0cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:39.508 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:39.505+0000 7f1c0be00640 1 -- 172.21.15.96:0/1450017178 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1bf4003520 con 0x7f1c0c0b32c0 2024-05-28T06:13:39.508 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:39.506+0000 7f1c0be00640 1 -- 172.21.15.96:0/1450017178 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3523271797 0 0) 0x7f1bf4003520 con 0x7f1c0c0b32c0 2024-05-28T06:13:39.508 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:39.506+0000 7f1c0be00640 1 -- 172.21.15.96:0/1450017178 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1c0c15d5d0 msgr2=0x7f1c0c0cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:13:39.508 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:39.506+0000 7f1c0be00640 1 --2- 172.21.15.96:0/1450017178 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1c0c15d5d0 0x7f1c0c0cf950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:39.508 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:39.506+0000 7f1c0be00640 1 -- 172.21.15.96:0/1450017178 >> v1:172.21.15.123:6789/0 conn(0x7f1c0c0d0640 legacy=0x7f1c0c0d0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:39.508 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:39.506+0000 7f1c0be00640 1 -- 172.21.15.96:0/1450017178 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c0c0ad8d0 con 0x7f1c0c0b32c0 2024-05-28T06:13:39.508 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:39.506+0000 7f1c0be00640 1 -- 172.21.15.96:0/1450017178 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f1c040019f0 con 0x7f1c0c0b32c0 2024-05-28T06:13:39.508 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:39.506+0000 7f1c0be00640 1 -- 172.21.15.96:0/1450017178 >> v1:172.21.15.96:6789/0 conn(0x7f1c0c0b32c0 legacy=0x7f1c0c0b3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:39.509 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:39.506+0000 7f1c0be00640 1 --2- 172.21.15.96:0/1450017178 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1bf40039e0 0x7f1bf4003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:39.509 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:39.506+0000 7f1c0be00640 1 --2- 172.21.15.96:0/1450017178 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1bf4004800 0x7f1bf4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:39.509 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:39.506+0000 7f1c0be00640 1 --2- 172.21.15.96:0/1450017178 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1c0c15d5d0 0x7f1bf4005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:39.509 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:39.506+0000 7f1c0be00640 1 -- 172.21.15.96:0/1450017178 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f1bf400fed0 con 0x7f1bf40039e0 2024-05-28T06:13:39.509 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:39.506+0000 7f1c0be00640 1 -- 172.21.15.96:0/1450017178 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f1bf4010080 con 0x7f1bf4004800 2024-05-28T06:13:39.509 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:39.506+0000 7f1c0be00640 1 -- 172.21.15.96:0/1450017178 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f1bf4010280 con 0x7f1c0c15d5d0 2024-05-28T06:13:39.509 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:39.506+0000 7f1c11400640 1 --2- 172.21.15.96:0/1450017178 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1bf40039e0 0x7f1bf4003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:39.509 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:39.507+0000 7f1c11e00640 1 --2- 172.21.15.96:0/1450017178 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1c0c15d5d0 0x7f1bf4005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:39.510 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:39.508+0000 7f1c10a00640 1 --2- 172.21.15.96:0/1450017178 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1bf4004800 0x7f1bf4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:39.511 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:39.509+0000 7f1c11400640 1 -- 172.21.15.96:0/1450017178 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1c0c15d5d0 msgr2=0x7f1bf4005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:13:39.511 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:39.509+0000 7f1c11400640 1 --2- 172.21.15.96:0/1450017178 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1c0c15d5d0 0x7f1bf4005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:39.511 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:39.509+0000 7f1c11400640 1 -- 172.21.15.96:0/1450017178 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1bf4004800 msgr2=0x7f1bf4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:13:39.511 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:39.509+0000 7f1c11400640 1 --2- 172.21.15.96:0/1450017178 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1bf4004800 0x7f1bf4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:39.511 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:39.509+0000 7f1c11400640 1 -- 172.21.15.96:0/1450017178 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c0c0ad8d0 con 0x7f1bf40039e0 2024-05-28T06:13:39.511 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:39.510+0000 7f1c11e00640 1 --2- 172.21.15.96:0/1450017178 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1c0c15d5d0 0x7f1bf4005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-28T06:13:39.513 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:39.511+0000 7f1c11400640 1 --2- 172.21.15.96:0/1450017178 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1bf40039e0 0x7f1bf4003e00 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f1c00001730 tx=0x7f1bf4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12293560d63a6552 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:13:39.513 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:39.512+0000 7f1c0be00640 1 -- 172.21.15.96:0/1450017178 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c00001350 con 0x7f1bf40039e0 2024-05-28T06:13:39.513 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:39.512+0000 7f1c0be00640 1 -- 172.21.15.96:0/1450017178 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c000052f0 con 0x7f1bf40039e0 2024-05-28T06:13:39.514 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:39.512+0000 7f1c0be00640 1 -- 172.21.15.96:0/1450017178 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c0002abc0 con 0x7f1bf40039e0 2024-05-28T06:13:39.514 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:39.513+0000 7f1c13e00640 1 -- 172.21.15.96:0/1450017178 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1bf40039e0 msgr2=0x7f1bf4003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:39.514 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:39.513+0000 7f1c13e00640 1 --2- 172.21.15.96:0/1450017178 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1bf40039e0 0x7f1bf4003e00 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f1c00001730 tx=0x7f1bf4005830 comp rx=0 tx=0).stop 2024-05-28T06:13:39.514 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:39.513+0000 7f1c11e00640 1 -- 172.21.15.96:0/1450017178 reap_dead start 2024-05-28T06:13:39.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:39.513+0000 7f1c13e00640 1 -- 172.21.15.96:0/1450017178 shutdown_connections 2024-05-28T06:13:39.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:39.513+0000 7f1c13e00640 1 -- 172.21.15.96:0/1450017178 >> 172.21.15.96:0/1450017178 conn(0x7f1c0c06bd20 msgr2=0x7f1c0c06c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:13:39.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:39.514+0000 7f1c13e00640 1 -- 172.21.15.96:0/1450017178 shutdown_connections 2024-05-28T06:13:39.516 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:39.514+0000 7f1c13e00640 1 -- 172.21.15.96:0/1450017178 wait complete. 2024-05-28T06:13:39.518 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:39.516+0000 7f1c13e00640 1 Processor -- start 2024-05-28T06:13:39.518 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:39.516+0000 7f1c13e00640 1 -- start start 2024-05-28T06:13:39.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:39.517+0000 7f1c13e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1c0c1f8140 0x7f1c0c0ce800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:39.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:39.517+0000 7f1c13e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c0c0ced40 0x7f1c0c154870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:39.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:39.517+0000 7f1c13e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1c0c154db0 0x7f1c0c146a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:39.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:39.517+0000 7f1c13e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f1c0c0b3e60 con 0x7f1c0c154db0 2024-05-28T06:13:39.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:39.517+0000 7f1c13e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f1c0c15d0b0 con 0x7f1c0c0ced40 2024-05-28T06:13:39.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:39.517+0000 7f1c13e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f1c0c15da50 con 0x7f1c0c1f8140 2024-05-28T06:13:39.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:39.518+0000 7f1c11e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1c0c154db0 0x7f1c0c146a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:39.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:39.518+0000 7f1c11e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1c0c154db0 0x7f1c0c146a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:56320/0 (socket says 172.21.15.96:56320) 2024-05-28T06:13:39.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:39.518+0000 7f1c11e00640 1 -- 172.21.15.96:0/842991135 learned_addr learned my addr 172.21.15.96:0/842991135 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:13:39.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:39.518+0000 7f1c10a00640 1 --2- 172.21.15.96:0/842991135 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c0c0ced40 0x7f1c0c154870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:39.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:39.519+0000 7f1c11e00640 1 -- 172.21.15.96:0/842991135 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1c0c1f8140 msgr2=0x7f1c0c0ce800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:39.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:39.519+0000 7f1c11e00640 1 --2- 172.21.15.96:0/842991135 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1c0c1f8140 0x7f1c0c0ce800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:39.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:39.519+0000 7f1c11e00640 1 -- 172.21.15.96:0/842991135 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c0c0ced40 msgr2=0x7f1c0c154870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:39.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:39.519+0000 7f1c11e00640 1 --2- 172.21.15.96:0/842991135 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c0c0ced40 0x7f1c0c154870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:39.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:39.519+0000 7f1c11e00640 1 -- 172.21.15.96:0/842991135 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c0c155630 con 0x7f1c0c154db0 2024-05-28T06:13:39.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:39.520+0000 7f1c11e00640 1 --2- 172.21.15.96:0/842991135 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1c0c154db0 0x7f1c0c146a70 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c0d1230 tx=0x7f1c04007720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:13:39.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:39.520+0000 7f1c09600640 1 -- 172.21.15.96:0/842991135 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c0400a480 con 0x7f1c0c154db0 2024-05-28T06:13:39.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:39.520+0000 7f1c09600640 1 -- 172.21.15.96:0/842991135 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c04018070 con 0x7f1c0c154db0 2024-05-28T06:13:39.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:39.520+0000 7f1c09600640 1 -- 172.21.15.96:0/842991135 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c040138e0 con 0x7f1c0c154db0 2024-05-28T06:13:39.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:39.520+0000 7f1c13e00640 1 -- 172.21.15.96:0/842991135 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c0c155910 con 0x7f1c0c154db0 2024-05-28T06:13:39.523 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:39.520+0000 7f1c13e00640 1 -- 172.21.15.96:0/842991135 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c0c1f8520 con 0x7f1c0c154db0 2024-05-28T06:13:39.524 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:39.523+0000 7f1c09600640 1 -- 172.21.15.96:0/842991135 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 20) ==== 92162+0+0 (secure 0 0 0) 0x7f1c04013a80 con 0x7f1c0c154db0 2024-05-28T06:13:39.525 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:39.523+0000 7f1c13e00640 1 -- 172.21.15.96:0/842991135 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c0c0689e0 con 0x7f1c0c154db0 2024-05-28T06:13:39.525 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:39.524+0000 7f1c09600640 1 --2- 172.21.15.96:0/842991135 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1bec06d580 0x7f1bec06fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:39.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:39.524+0000 7f1c09600640 1 -- 172.21.15.96:0/842991135 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5737+0+0 (secure 0 0 0) 0x7f1c0408af20 con 0x7f1c0c154db0 2024-05-28T06:13:39.530 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:39.526+0000 7f1c11400640 1 --2- 172.21.15.96:0/842991135 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1bec06d580 0x7f1bec06fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:39.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:39.528+0000 7f1c11400640 1 --2- 172.21.15.96:0/842991135 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1bec06d580 0x7f1bec06fa40 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f1c00032000 tx=0x7f1c0002c000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:13:39.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:39.533+0000 7f1c09600640 1 -- 172.21.15.96:0/842991135 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1c0405ae00 con 0x7f1c0c154db0 2024-05-28T06:13:39.954 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:39.952+0000 7f1c13e00640 1 -- 172.21.15.96:0/842991135 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone81713", "target": ["mon-mgr", ""]}) -- 0x7f1c0c0ce610 con 0x7f1bec06d580 2024-05-28T06:13:40.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:40.934+0000 7f1c09600640 1 -- 172.21.15.96:0/842991135 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f1c0c0ce610 con 0x7f1bec06d580 2024-05-28T06:13:40.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:40.938+0000 7f1c13e00640 1 -- 172.21.15.96:0/842991135 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1bec06d580 msgr2=0x7f1bec06fa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:40.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:40.938+0000 7f1c13e00640 1 --2- 172.21.15.96:0/842991135 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1bec06d580 0x7f1bec06fa40 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f1c00032000 tx=0x7f1c0002c000 comp rx=0 tx=0).stop 2024-05-28T06:13:40.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:40.938+0000 7f1c13e00640 1 -- 172.21.15.96:0/842991135 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1c0c154db0 msgr2=0x7f1c0c146a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:40.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:40.938+0000 7f1c13e00640 1 --2- 172.21.15.96:0/842991135 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1c0c154db0 0x7f1c0c146a70 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c0d1230 tx=0x7f1c04007720 comp rx=0 tx=0).stop 2024-05-28T06:13:40.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:40.939+0000 7f1c13e00640 1 -- 172.21.15.96:0/842991135 shutdown_connections 2024-05-28T06:13:40.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:40.939+0000 7f1c13e00640 1 --2- 172.21.15.96:0/842991135 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1bec06d580 0x7f1bec06fa40 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:40.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:40.939+0000 7f1c13e00640 1 --2- 172.21.15.96:0/842991135 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1c0c154db0 0x7f1c0c146a70 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:40.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:40.939+0000 7f1c13e00640 1 --2- 172.21.15.96:0/842991135 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c0c0ced40 0x7f1c0c154870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:40.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:40.939+0000 7f1c13e00640 1 --2- 172.21.15.96:0/842991135 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1c0c1f8140 0x7f1c0c0ce800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:40.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:40.939+0000 7f1c13e00640 1 -- 172.21.15.96:0/842991135 >> 172.21.15.96:0/842991135 conn(0x7f1c0c06bd20 msgr2=0x7f1c0c0d0a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:13:40.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:40.939+0000 7f1c13e00640 1 -- 172.21.15.96:0/842991135 shutdown_connections 2024-05-28T06:13:40.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:40.939+0000 7f1c13e00640 1 -- 172.21.15.96:0/842991135 wait complete. 2024-05-28T06:13:40.972 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone81714 2024-05-28T06:13:41.274 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:41.273+0000 7f003e200640 1 Processor -- start 2024-05-28T06:13:41.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:41.273+0000 7f003e200640 1 -- start start 2024-05-28T06:13:41.276 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:41.274+0000 7f003e200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f00380d0d50 0x7f00381633b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:41.276 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:41.274+0000 7f003e200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f00380b4120 0x7f00381638f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:41.276 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:41.274+0000 7f003e200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0038163e30 con 0x7f0038162ac0 2024-05-28T06:13:41.276 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:41.274+0000 7f003e200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f00380b3c90 con 0x7f00380d0d50 2024-05-28T06:13:41.276 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:41.274+0000 7f003e200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f00381640e0 con 0x7f00380b4120 2024-05-28T06:13:41.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:41.275+0000 7f0036a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f00380d0d50 0x7f00381633b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:41.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:41.275+0000 7f0036a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f00380d0d50 0x7f00381633b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:56328/0 (socket says 172.21.15.96:56328) 2024-05-28T06:13:41.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:41.275+0000 7f0036a00640 1 -- 172.21.15.96:0/4227698994 learned_addr learned my addr 172.21.15.96:0/4227698994 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:13:41.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:41.275+0000 7f0037e00640 1 --2- 172.21.15.96:0/4227698994 >> v2:172.21.15.123:3300/0 conn(0x7f00380b4120 0x7f00381638f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:41.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:41.275+0000 7f0036a00640 1 -- 172.21.15.96:0/4227698994 >> v2:172.21.15.123:3300/0 conn(0x7f00380b4120 msgr2=0x7f00381638f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:13:41.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:41.275+0000 7f0036a00640 1 --2- 172.21.15.96:0/4227698994 >> v2:172.21.15.123:3300/0 conn(0x7f00380b4120 0x7f00381638f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:41.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:41.275+0000 7f0036a00640 1 -- 172.21.15.96:0/4227698994 >> v1:172.21.15.123:6789/0 conn(0x7f0038162ac0 legacy=0x7f00380d0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:13:41.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:41.275+0000 7f0036a00640 1 -- 172.21.15.96:0/4227698994 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00380b3360 con 0x7f00380d0d50 2024-05-28T06:13:41.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:41.275+0000 7f0037e00640 1 --2- 172.21.15.96:0/4227698994 >> v2:172.21.15.123:3300/0 conn(0x7f00380b4120 0x7f00381638f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:13:41.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:41.277+0000 7f0036a00640 1 --2- 172.21.15.96:0/4227698994 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00380d0d50 0x7f00381633b0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f00380740c0 tx=0x7f002002a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc36d061347d930f server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:13:41.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:41.278+0000 7f0036000640 1 -- 172.21.15.96:0/4227698994 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0020037070 con 0x7f00380d0d50 2024-05-28T06:13:41.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:41.278+0000 7f0036000640 1 -- 172.21.15.96:0/4227698994 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0020032440 con 0x7f00380d0d50 2024-05-28T06:13:41.280 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:41.278+0000 7f0036000640 1 -- 172.21.15.96:0/4227698994 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0020032810 con 0x7f00380d0d50 2024-05-28T06:13:41.280 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:41.279+0000 7f003e200640 1 -- 172.21.15.96:0/4227698994 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00380d0d50 msgr2=0x7f00381633b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:41.281 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:41.279+0000 7f003e200640 1 --2- 172.21.15.96:0/4227698994 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00380d0d50 0x7f00381633b0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f00380740c0 tx=0x7f002002a1a0 comp rx=0 tx=0).stop 2024-05-28T06:13:41.281 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:41.280+0000 7f003e200640 1 -- 172.21.15.96:0/4227698994 shutdown_connections 2024-05-28T06:13:41.282 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:41.280+0000 7f003e200640 1 --2- 172.21.15.96:0/4227698994 >> v2:172.21.15.123:3300/0 conn(0x7f00380b4120 0x7f00381638f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:41.282 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:41.280+0000 7f003e200640 1 --2- 172.21.15.96:0/4227698994 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00380d0d50 0x7f00381633b0 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:41.282 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:41.280+0000 7f003e200640 1 -- 172.21.15.96:0/4227698994 >> 172.21.15.96:0/4227698994 conn(0x7f003806bd20 msgr2=0x7f003806c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:13:41.282 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:41.280+0000 7f003e200640 1 -- 172.21.15.96:0/4227698994 shutdown_connections 2024-05-28T06:13:41.282 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:41.280+0000 7f003e200640 1 -- 172.21.15.96:0/4227698994 wait complete. 2024-05-28T06:13:41.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:41.282+0000 7f003e200640 1 Processor -- start 2024-05-28T06:13:41.285 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:41.283+0000 7f003e200640 1 -- start start 2024-05-28T06:13:41.286 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:41.284+0000 7f003e200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f00380b4120 0x7f00380cfbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:41.286 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:41.284+0000 7f003e200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00380d0d50 0x7f0038158e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:41.286 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:41.284+0000 7f003e200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0038162ac0 0x7f003809a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:41.286 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:41.284+0000 7f003e200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f0038162ea0 con 0x7f00380d0d50 2024-05-28T06:13:41.286 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:41.284+0000 7f003e200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f00380b3920 con 0x7f0038162ac0 2024-05-28T06:13:41.286 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:41.284+0000 7f003e200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f0038163020 con 0x7f00380b4120 2024-05-28T06:13:41.287 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:41.285+0000 7f0036a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00380d0d50 0x7f0038158e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:41.287 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:41.285+0000 7f0036a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00380d0d50 0x7f0038158e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:56336/0 (socket says 172.21.15.96:56336) 2024-05-28T06:13:41.287 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:41.285+0000 7f0036a00640 1 -- 172.21.15.96:0/2610804524 learned_addr learned my addr 172.21.15.96:0/2610804524 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:13:41.288 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:41.285+0000 7f0037e00640 1 --2- 172.21.15.96:0/2610804524 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0038162ac0 0x7f003809a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:41.288 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:41.286+0000 7f0036a00640 1 -- 172.21.15.96:0/2610804524 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f00380b4120 msgr2=0x7f00380cfbe0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-28T06:13:41.288 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:41.286+0000 7f0036a00640 1 --2- 172.21.15.96:0/2610804524 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f00380b4120 0x7f00380cfbe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:41.288 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:41.286+0000 7f0036a00640 1 -- 172.21.15.96:0/2610804524 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0038162ac0 msgr2=0x7f003809a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:41.288 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:41.286+0000 7f0036a00640 1 --2- 172.21.15.96:0/2610804524 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0038162ac0 0x7f003809a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:41.289 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:41.286+0000 7f0036a00640 1 -- 172.21.15.96:0/2610804524 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f003809a5c0 con 0x7f00380d0d50 2024-05-28T06:13:41.289 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:41.287+0000 7f0036a00640 1 --2- 172.21.15.96:0/2610804524 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00380d0d50 0x7f0038158e90 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f0020000c00 tx=0x7f00200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:13:41.290 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:41.288+0000 7f002f400640 1 -- 172.21.15.96:0/2610804524 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0020041070 con 0x7f00380d0d50 2024-05-28T06:13:41.290 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:41.288+0000 7f002f400640 1 -- 172.21.15.96:0/2610804524 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f002003c870 con 0x7f00380d0d50 2024-05-28T06:13:41.290 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:41.289+0000 7f002f400640 1 -- 172.21.15.96:0/2610804524 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0020037040 con 0x7f00380d0d50 2024-05-28T06:13:41.290 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:41.289+0000 7f003e200640 1 -- 172.21.15.96:0/2610804524 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f003809c9e0 con 0x7f00380d0d50 2024-05-28T06:13:41.292 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:41.289+0000 7f003e200640 1 -- 172.21.15.96:0/2610804524 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f003809cfa0 con 0x7f00380d0d50 2024-05-28T06:13:41.295 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:41.293+0000 7f002cc00640 1 -- 172.21.15.96:0/2610804524 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00380689e0 con 0x7f00380d0d50 2024-05-28T06:13:41.295 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:41.293+0000 7f002f400640 1 -- 172.21.15.96:0/2610804524 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 20) ==== 92162+0+0 (secure 0 0 0) 0x7f0020044070 con 0x7f00380d0d50 2024-05-28T06:13:41.298 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:41.295+0000 7f002f400640 1 --2- 172.21.15.96:0/2610804524 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f001406d550 0x7f001406fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:41.298 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:41.296+0000 7f002f400640 1 -- 172.21.15.96:0/2610804524 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5737+0+0 (secure 0 0 0) 0x7f00200ab260 con 0x7f00380d0d50 2024-05-28T06:13:41.306 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:41.305+0000 7f002f400640 1 -- 172.21.15.96:0/2610804524 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f002007b610 con 0x7f00380d0d50 2024-05-28T06:13:41.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:41.309+0000 7f0037400640 1 --2- 172.21.15.96:0/2610804524 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f001406d550 0x7f001406fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:41.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:41.336+0000 7f0037400640 1 --2- 172.21.15.96:0/2610804524 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f001406d550 0x7f001406fa10 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f00280095f0 tx=0x7f0028006b50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:13:41.690 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:41.688+0000 7f002cc00640 1 -- 172.21.15.96:0/2610804524 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone81714", "target": ["mon-mgr", ""]}) -- 0x7f00380d0920 con 0x7f001406d550 2024-05-28T06:13:42.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:42.338+0000 7f002f400640 1 -- 172.21.15.96:0/2610804524 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f00380d0920 con 0x7f001406d550 2024-05-28T06:13:42.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:42.341+0000 7f002cc00640 1 -- 172.21.15.96:0/2610804524 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f001406d550 msgr2=0x7f001406fa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:42.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:42.341+0000 7f002cc00640 1 --2- 172.21.15.96:0/2610804524 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f001406d550 0x7f001406fa10 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f00280095f0 tx=0x7f0028006b50 comp rx=0 tx=0).stop 2024-05-28T06:13:42.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:42.341+0000 7f002cc00640 1 -- 172.21.15.96:0/2610804524 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00380d0d50 msgr2=0x7f0038158e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:42.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:42.341+0000 7f002cc00640 1 --2- 172.21.15.96:0/2610804524 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00380d0d50 0x7f0038158e90 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f0020000c00 tx=0x7f00200046c0 comp rx=0 tx=0).stop 2024-05-28T06:13:42.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:42.341+0000 7f002cc00640 1 -- 172.21.15.96:0/2610804524 shutdown_connections 2024-05-28T06:13:42.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:42.341+0000 7f002cc00640 1 --2- 172.21.15.96:0/2610804524 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f001406d550 0x7f001406fa10 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:42.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:42.341+0000 7f002cc00640 1 --2- 172.21.15.96:0/2610804524 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0038162ac0 0x7f003809a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:42.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:42.341+0000 7f002cc00640 1 --2- 172.21.15.96:0/2610804524 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00380d0d50 0x7f0038158e90 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:42.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:42.341+0000 7f002cc00640 1 --2- 172.21.15.96:0/2610804524 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f00380b4120 0x7f00380cfbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:42.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:42.341+0000 7f002cc00640 1 -- 172.21.15.96:0/2610804524 >> 172.21.15.96:0/2610804524 conn(0x7f003806bd20 msgr2=0x7f00380af2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:13:42.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:42.341+0000 7f002cc00640 1 -- 172.21.15.96:0/2610804524 shutdown_connections 2024-05-28T06:13:42.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:42.341+0000 7f002cc00640 1 -- 172.21.15.96:0/2610804524 wait complete. 2024-05-28T06:13:42.371 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone81715 2024-05-28T06:13:42.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:42.683+0000 7fbf87e00640 1 Processor -- start 2024-05-28T06:13:42.686 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:42.684+0000 7fbf87e00640 1 -- start start 2024-05-28T06:13:42.688 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:42.686+0000 7fbf87e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbf880ce960 0x7fbf880caa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:42.688 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:42.686+0000 7fbf87e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fbf881524c0 0x7fbf880cf710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:42.688 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:42.686+0000 7fbf87e00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf880b3e80 con 0x7fbf880ced40 2024-05-28T06:13:42.688 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:42.686+0000 7fbf87e00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fbf880d0b30 con 0x7fbf881524c0 2024-05-28T06:13:42.689 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:42.686+0000 7fbf87e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fbf880b4160 con 0x7fbf880ce960 2024-05-28T06:13:42.689 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:42.688+0000 7fbf87400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fbf881524c0 0x7fbf880cf710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:42.690 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:42.688+0000 7fbf87400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fbf881524c0 0x7fbf880cf710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:56352/0 (socket says 172.21.15.96:56352) 2024-05-28T06:13:42.690 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:42.688+0000 7fbf87400640 1 -- 172.21.15.96:0/1284910100 learned_addr learned my addr 172.21.15.96:0/1284910100 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:13:42.690 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:42.688+0000 7fbf86a00640 1 --2- 172.21.15.96:0/1284910100 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbf880ce960 0x7fbf880caa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:42.690 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:42.689+0000 7fbf87400640 1 -- 172.21.15.96:0/1284910100 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbf880ce960 msgr2=0x7fbf880caa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:13:42.691 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:42.689+0000 7fbf87400640 1 --2- 172.21.15.96:0/1284910100 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbf880ce960 0x7fbf880caa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:42.691 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:42.689+0000 7fbf87400640 1 -- 172.21.15.96:0/1284910100 >> v1:172.21.15.123:6789/0 conn(0x7fbf880ced40 legacy=0x7fbf88151db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:13:42.691 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:42.689+0000 7fbf87400640 1 -- 172.21.15.96:0/1284910100 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf880cde80 con 0x7fbf881524c0 2024-05-28T06:13:42.693 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:42.691+0000 7fbf87400640 1 --2- 172.21.15.96:0/1284910100 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf881524c0 0x7fbf880cf710 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7fbf80009e50 tx=0x7fbf8002a630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=369bfe289c26ec85 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:13:42.693 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:42.691+0000 7fbf85600640 1 -- 172.21.15.96:0/1284910100 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf80036070 con 0x7fbf881524c0 2024-05-28T06:13:42.694 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:42.693+0000 7fbf85600640 1 -- 172.21.15.96:0/1284910100 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf80031600 con 0x7fbf881524c0 2024-05-28T06:13:42.694 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:42.693+0000 7fbf85600640 1 -- 172.21.15.96:0/1284910100 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf800319d0 con 0x7fbf881524c0 2024-05-28T06:13:42.698 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:42.694+0000 7fbf87e00640 1 -- 172.21.15.96:0/1284910100 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf881524c0 msgr2=0x7fbf880cf710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:42.698 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:42.694+0000 7fbf87e00640 1 --2- 172.21.15.96:0/1284910100 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf881524c0 0x7fbf880cf710 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7fbf80009e50 tx=0x7fbf8002a630 comp rx=0 tx=0).stop 2024-05-28T06:13:42.698 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:42.695+0000 7fbf87e00640 1 -- 172.21.15.96:0/1284910100 shutdown_connections 2024-05-28T06:13:42.698 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:42.695+0000 7fbf87e00640 1 --2- 172.21.15.96:0/1284910100 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf881524c0 0x7fbf880cf710 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:42.698 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:42.695+0000 7fbf87e00640 1 --2- 172.21.15.96:0/1284910100 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbf880ce960 0x7fbf880caa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:42.699 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:42.695+0000 7fbf87e00640 1 -- 172.21.15.96:0/1284910100 >> 172.21.15.96:0/1284910100 conn(0x7fbf8806b900 msgr2=0x7fbf8806bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:13:42.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:42.697+0000 7fbf87e00640 1 -- 172.21.15.96:0/1284910100 shutdown_connections 2024-05-28T06:13:42.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:42.697+0000 7fbf87e00640 1 -- 172.21.15.96:0/1284910100 wait complete. 2024-05-28T06:13:42.705 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:42.700+0000 7fbf87e00640 1 Processor -- start 2024-05-28T06:13:42.705 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:42.700+0000 7fbf87e00640 1 -- start start 2024-05-28T06:13:42.705 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:42.702+0000 7fbf87e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbf880ce960 0x7fbf8816cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:42.705 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:42.702+0000 7fbf87e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbf880ced40 0x7fbf88143a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:42.705 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:42.702+0000 7fbf87e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf881524c0 0x7fbf88143fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:42.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:42.703+0000 7fbf87e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fbf880caf80 con 0x7fbf881524c0 2024-05-28T06:13:42.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:42.703+0000 7fbf87e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fbf880ce440 con 0x7fbf880ce960 2024-05-28T06:13:42.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:42.703+0000 7fbf87e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fbf880cb100 con 0x7fbf880ced40 2024-05-28T06:13:42.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:42.703+0000 7fbf87400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf881524c0 0x7fbf88143fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:42.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:42.703+0000 7fbf87400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf881524c0 0x7fbf88143fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:56368/0 (socket says 172.21.15.96:56368) 2024-05-28T06:13:42.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:42.703+0000 7fbf87400640 1 -- 172.21.15.96:0/3372888153 learned_addr learned my addr 172.21.15.96:0/3372888153 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:13:42.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:42.704+0000 7fbf87400640 1 -- 172.21.15.96:0/3372888153 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbf880ced40 msgr2=0x7fbf88143a70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-28T06:13:42.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:42.704+0000 7fbf87400640 1 --2- 172.21.15.96:0/3372888153 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbf880ced40 0x7fbf88143a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:42.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:42.704+0000 7fbf87400640 1 -- 172.21.15.96:0/3372888153 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbf880ce960 msgr2=0x7fbf8816cb40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-28T06:13:42.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:42.704+0000 7fbf87400640 1 --2- 172.21.15.96:0/3372888153 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbf880ce960 0x7fbf8816cb40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:42.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:42.704+0000 7fbf87400640 1 -- 172.21.15.96:0/3372888153 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf881453d0 con 0x7fbf881524c0 2024-05-28T06:13:42.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:42.704+0000 7fbf87400640 1 --2- 172.21.15.96:0/3372888153 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf881524c0 0x7fbf88143fb0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fbf80009f70 tx=0x7fbf80004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:13:42.710 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:42.708+0000 7fbf7ea00640 1 -- 172.21.15.96:0/3372888153 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf8003f020 con 0x7fbf881524c0 2024-05-28T06:13:42.711 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:42.708+0000 7fbf7ea00640 1 -- 172.21.15.96:0/3372888153 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf80037040 con 0x7fbf881524c0 2024-05-28T06:13:42.711 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:42.708+0000 7fbf7ea00640 1 -- 172.21.15.96:0/3372888153 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf80036070 con 0x7fbf881524c0 2024-05-28T06:13:42.711 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:42.708+0000 7fbf87e00640 1 -- 172.21.15.96:0/3372888153 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf88141080 con 0x7fbf881524c0 2024-05-28T06:13:42.711 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:42.708+0000 7fbf87e00640 1 -- 172.21.15.96:0/3372888153 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf88097f90 con 0x7fbf881524c0 2024-05-28T06:13:42.714 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:42.712+0000 7fbf87e00640 1 -- 172.21.15.96:0/3372888153 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf880689e0 con 0x7fbf881524c0 2024-05-28T06:13:42.715 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:42.713+0000 7fbf7ea00640 1 -- 172.21.15.96:0/3372888153 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 20) ==== 92162+0+0 (secure 0 0 0) 0x7fbf80042020 con 0x7fbf881524c0 2024-05-28T06:13:42.715 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:42.714+0000 7fbf7ea00640 1 --2- 172.21.15.96:0/3372888153 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbf6806d5a0 0x7fbf6806fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:42.715 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:42.714+0000 7fbf7ea00640 1 -- 172.21.15.96:0/3372888153 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5737+0+0 (secure 0 0 0) 0x7fbf800aad60 con 0x7fbf881524c0 2024-05-28T06:13:42.725 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:42.724+0000 7fbf7ea00640 1 -- 172.21.15.96:0/3372888153 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbf80049030 con 0x7fbf881524c0 2024-05-28T06:13:42.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:42.736+0000 7fbf86a00640 1 --2- 172.21.15.96:0/3372888153 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbf6806d5a0 0x7fbf6806fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:42.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:42.762+0000 7fbf86a00640 1 --2- 172.21.15.96:0/3372888153 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbf6806d5a0 0x7fbf6806fa60 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fbf78009870 tx=0x7fbf78008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:13:43.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:43.138+0000 7fbf87e00640 1 -- 172.21.15.96:0/3372888153 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone81715", "target": ["mon-mgr", ""]}) -- 0x7fbf88158f40 con 0x7fbf6806d5a0 2024-05-28T06:13:43.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:43.656+0000 7fbf7ea00640 1 -- 172.21.15.96:0/3372888153 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fbf88158f40 con 0x7fbf6806d5a0 2024-05-28T06:13:43.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:43.661+0000 7fbf87e00640 1 -- 172.21.15.96:0/3372888153 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbf6806d5a0 msgr2=0x7fbf6806fa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:43.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:43.661+0000 7fbf87e00640 1 --2- 172.21.15.96:0/3372888153 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbf6806d5a0 0x7fbf6806fa60 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fbf78009870 tx=0x7fbf78008040 comp rx=0 tx=0).stop 2024-05-28T06:13:43.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:43.661+0000 7fbf87e00640 1 -- 172.21.15.96:0/3372888153 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf881524c0 msgr2=0x7fbf88143fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:43.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:43.661+0000 7fbf87e00640 1 --2- 172.21.15.96:0/3372888153 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf881524c0 0x7fbf88143fb0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fbf80009f70 tx=0x7fbf80004a40 comp rx=0 tx=0).stop 2024-05-28T06:13:43.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:43.662+0000 7fbf87e00640 1 -- 172.21.15.96:0/3372888153 shutdown_connections 2024-05-28T06:13:43.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:43.662+0000 7fbf87e00640 1 --2- 172.21.15.96:0/3372888153 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbf6806d5a0 0x7fbf6806fa60 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:43.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:43.662+0000 7fbf87e00640 1 --2- 172.21.15.96:0/3372888153 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf881524c0 0x7fbf88143fb0 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:43.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:43.662+0000 7fbf87e00640 1 --2- 172.21.15.96:0/3372888153 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbf880ced40 0x7fbf88143a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:43.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:43.662+0000 7fbf87e00640 1 --2- 172.21.15.96:0/3372888153 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbf880ce960 0x7fbf8816cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:43.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:43.662+0000 7fbf87e00640 1 -- 172.21.15.96:0/3372888153 >> 172.21.15.96:0/3372888153 conn(0x7fbf8806b900 msgr2=0x7fbf8806f490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:13:43.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:43.662+0000 7fbf87e00640 1 -- 172.21.15.96:0/3372888153 shutdown_connections 2024-05-28T06:13:43.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:43.662+0000 7fbf87e00640 1 -- 172.21.15.96:0/3372888153 wait complete. 2024-05-28T06:13:43.695 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone81716 2024-05-28T06:13:44.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:44.013+0000 7f0792000640 1 Processor -- start 2024-05-28T06:13:44.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:44.014+0000 7f0792000640 1 -- start start 2024-05-28T06:13:44.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:44.015+0000 7f0792000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f078c161a90 0x7f078c06e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:44.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:44.015+0000 7f0792000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f078c161e70 0x7f078c0b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:44.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:44.016+0000 7f0792000640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f078c0b3d40 con 0x7f078c06e9b0 2024-05-28T06:13:44.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:44.016+0000 7f0792000640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f078c0b4020 con 0x7f078c161e70 2024-05-28T06:13:44.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:44.016+0000 7f0792000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f078c0b4300 con 0x7f078c161a90 2024-05-28T06:13:44.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:44.017+0000 7f078aa00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f078c161e70 0x7f078c0b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:44.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:44.017+0000 7f078be00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f078c06e9b0 0x7f078c0ad2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:53476/0 (socket says 172.21.15.96:53476) 2024-05-28T06:13:44.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:44.017+0000 7f078be00640 1 -- 172.21.15.96:0/573775945 learned_addr learned my addr 172.21.15.96:0/573775945 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:13:44.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:44.017+0000 7f078b400640 1 --2- 172.21.15.96:0/573775945 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f078c161a90 0x7f078c06e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:44.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:44.018+0000 7f078aa00640 1 -- 172.21.15.96:0/573775945 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f078c161a90 msgr2=0x7f078c06e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:13:44.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:44.018+0000 7f078aa00640 1 --2- 172.21.15.96:0/573775945 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f078c161a90 0x7f078c06e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:44.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:44.018+0000 7f078aa00640 1 -- 172.21.15.96:0/573775945 >> v1:172.21.15.96:6789/0 conn(0x7f078c06e9b0 legacy=0x7f078c0ad2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:44.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:44.018+0000 7f078aa00640 1 -- 172.21.15.96:0/573775945 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f078c06dc70 con 0x7f078c161e70 2024-05-28T06:13:44.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:44.019+0000 7f078aa00640 1 --2- 172.21.15.96:0/573775945 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f078c161e70 0x7f078c0b32c0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f077c0097a0 tx=0x7f077c02a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71a31de55174d1a6 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:13:44.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:44.020+0000 7f078a000640 1 -- 172.21.15.96:0/573775945 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f077c037070 con 0x7f078c161e70 2024-05-28T06:13:44.022 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:44.020+0000 7f078a000640 1 -- 172.21.15.96:0/573775945 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f077c032440 con 0x7f078c161e70 2024-05-28T06:13:44.022 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:44.020+0000 7f078a000640 1 -- 172.21.15.96:0/573775945 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f077c032810 con 0x7f078c161e70 2024-05-28T06:13:44.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:44.021+0000 7f0792000640 1 -- 172.21.15.96:0/573775945 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f078c161e70 msgr2=0x7f078c0b32c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:44.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:44.021+0000 7f0792000640 1 --2- 172.21.15.96:0/573775945 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f078c161e70 0x7f078c0b32c0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f077c0097a0 tx=0x7f077c02a180 comp rx=0 tx=0).stop 2024-05-28T06:13:44.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:44.021+0000 7f0792000640 1 -- 172.21.15.96:0/573775945 shutdown_connections 2024-05-28T06:13:44.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:44.021+0000 7f0792000640 1 --2- 172.21.15.96:0/573775945 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f078c161e70 0x7f078c0b32c0 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:44.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:44.021+0000 7f0792000640 1 --2- 172.21.15.96:0/573775945 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f078c161a90 0x7f078c06e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:44.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:44.021+0000 7f0792000640 1 -- 172.21.15.96:0/573775945 >> 172.21.15.96:0/573775945 conn(0x7f078c06bd20 msgr2=0x7f078c06c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:13:44.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:44.021+0000 7f0792000640 1 -- 172.21.15.96:0/573775945 shutdown_connections 2024-05-28T06:13:44.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:44.021+0000 7f0792000640 1 -- 172.21.15.96:0/573775945 wait complete. 2024-05-28T06:13:44.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:44.023+0000 7f0792000640 1 Processor -- start 2024-05-28T06:13:44.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:44.024+0000 7f0792000640 1 -- start start 2024-05-28T06:13:44.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:44.025+0000 7f0792000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f078c06e9b0 0x7f078c1f3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:44.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:44.025+0000 7f0792000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f078c161a90 0x7f078c14f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:44.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:44.025+0000 7f0792000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f078c161e70 0x7f078c1f4a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:44.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:44.025+0000 7f0792000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f078c0ad9b0 con 0x7f078c161e70 2024-05-28T06:13:44.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:44.025+0000 7f0792000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f078c0b5020 con 0x7f078c06e9b0 2024-05-28T06:13:44.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:44.025+0000 7f0792000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f078c0adb30 con 0x7f078c161a90 2024-05-28T06:13:44.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:44.026+0000 7f078be00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f078c161e70 0x7f078c1f4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:44.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:44.026+0000 7f078be00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f078c161e70 0x7f078c1f4a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:56382/0 (socket says 172.21.15.96:56382) 2024-05-28T06:13:44.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:44.026+0000 7f078be00640 1 -- 172.21.15.96:0/4223274577 learned_addr learned my addr 172.21.15.96:0/4223274577 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:13:44.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:44.026+0000 7f078be00640 1 -- 172.21.15.96:0/4223274577 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f078c161a90 msgr2=0x7f078c14f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:44.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:44.026+0000 7f078be00640 1 --2- 172.21.15.96:0/4223274577 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f078c161a90 0x7f078c14f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:44.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:44.026+0000 7f078be00640 1 -- 172.21.15.96:0/4223274577 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f078c06e9b0 msgr2=0x7f078c1f3aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:44.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:44.026+0000 7f078be00640 1 --2- 172.21.15.96:0/4223274577 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f078c06e9b0 0x7f078c1f3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:44.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:44.026+0000 7f078be00640 1 -- 172.21.15.96:0/4223274577 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f078c14ff90 con 0x7f078c161e70 2024-05-28T06:13:44.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:44.027+0000 7f078be00640 1 --2- 172.21.15.96:0/4223274577 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f078c161e70 0x7f078c1f4a10 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f078400caa0 tx=0x7f078400cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:13:44.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:44.027+0000 7f0783400640 1 -- 172.21.15.96:0/4223274577 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0784017070 con 0x7f078c161e70 2024-05-28T06:13:44.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:44.027+0000 7f0783400640 1 -- 172.21.15.96:0/4223274577 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f078400ac00 con 0x7f078c161e70 2024-05-28T06:13:44.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:44.027+0000 7f0783400640 1 -- 172.21.15.96:0/4223274577 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0784012620 con 0x7f078c161e70 2024-05-28T06:13:44.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:44.027+0000 7f0792000640 1 -- 172.21.15.96:0/4223274577 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f078c150270 con 0x7f078c161e70 2024-05-28T06:13:44.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:44.028+0000 7f0792000640 1 -- 172.21.15.96:0/4223274577 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f078c150830 con 0x7f078c161e70 2024-05-28T06:13:44.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:44.030+0000 7f0792000640 1 -- 172.21.15.96:0/4223274577 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f078c0689e0 con 0x7f078c161e70 2024-05-28T06:13:44.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:44.030+0000 7f0783400640 1 -- 172.21.15.96:0/4223274577 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 20) ==== 92162+0+0 (secure 0 0 0) 0x7f078401a070 con 0x7f078c161e70 2024-05-28T06:13:44.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:44.033+0000 7f0783400640 1 --2- 172.21.15.96:0/4223274577 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f076c06d550 0x7f076c06fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:44.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:44.033+0000 7f0783400640 1 -- 172.21.15.96:0/4223274577 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5737+0+0 (secure 0 0 0) 0x7f078408a580 con 0x7f078c161e70 2024-05-28T06:13:44.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:44.035+0000 7f078b400640 1 --2- 172.21.15.96:0/4223274577 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f076c06d550 0x7f076c06fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:44.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:44.042+0000 7f0783400640 1 -- 172.21.15.96:0/4223274577 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f078405a180 con 0x7f078c161e70 2024-05-28T06:13:44.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:44.043+0000 7f078b400640 1 --2- 172.21.15.96:0/4223274577 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f076c06d550 0x7f076c06fa10 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f07780098f0 tx=0x7f0778008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:13:44.416 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:44.414+0000 7f0792000640 1 -- 172.21.15.96:0/4223274577 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone81716", "target": ["mon-mgr", ""]}) -- 0x7f078c160cd0 con 0x7f076c06d550 2024-05-28T06:13:44.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:44.764+0000 7f0783400640 1 -- 172.21.15.96:0/4223274577 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f078c160cd0 con 0x7f076c06d550 2024-05-28T06:13:44.770 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:44.768+0000 7f0792000640 1 -- 172.21.15.96:0/4223274577 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f076c06d550 msgr2=0x7f076c06fa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:44.770 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:44.768+0000 7f0792000640 1 --2- 172.21.15.96:0/4223274577 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f076c06d550 0x7f076c06fa10 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f07780098f0 tx=0x7f0778008040 comp rx=0 tx=0).stop 2024-05-28T06:13:44.770 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:44.768+0000 7f0792000640 1 -- 172.21.15.96:0/4223274577 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f078c161e70 msgr2=0x7f078c1f4a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:44.770 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:44.768+0000 7f0792000640 1 --2- 172.21.15.96:0/4223274577 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f078c161e70 0x7f078c1f4a10 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f078400caa0 tx=0x7f078400cf60 comp rx=0 tx=0).stop 2024-05-28T06:13:44.770 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:44.768+0000 7f0792000640 1 -- 172.21.15.96:0/4223274577 shutdown_connections 2024-05-28T06:13:44.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:44.768+0000 7f0792000640 1 --2- 172.21.15.96:0/4223274577 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f076c06d550 0x7f076c06fa10 secure :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f07780098f0 tx=0x7f0778008040 comp rx=0 tx=0).stop 2024-05-28T06:13:44.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:44.768+0000 7f0792000640 1 --2- 172.21.15.96:0/4223274577 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f078c161e70 0x7f078c1f4a10 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:44.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:44.768+0000 7f0792000640 1 --2- 172.21.15.96:0/4223274577 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f078c161a90 0x7f078c14f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:44.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:44.768+0000 7f0792000640 1 --2- 172.21.15.96:0/4223274577 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f078c06e9b0 0x7f078c1f3aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:44.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:44.768+0000 7f0792000640 1 -- 172.21.15.96:0/4223274577 >> 172.21.15.96:0/4223274577 conn(0x7f078c06bd20 msgr2=0x7f078c0afe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:13:44.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:44.768+0000 7f0792000640 1 -- 172.21.15.96:0/4223274577 shutdown_connections 2024-05-28T06:13:44.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:44.768+0000 7f0792000640 1 -- 172.21.15.96:0/4223274577 wait complete. 2024-05-28T06:13:44.790 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone81717 2024-05-28T06:13:45.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.056+0000 7ff883000640 1 Processor -- start 2024-05-28T06:13:45.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.056+0000 7ff883000640 1 -- start start 2024-05-28T06:13:45.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.058+0000 7ff883000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff87c0d5190 0x7ff87c0d30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:45.062 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.059+0000 7ff881c00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7ff87c15ea40 0x7ff87c0cfab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:53488/0 (socket says 172.21.15.96:53488) 2024-05-28T06:13:45.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.059+0000 7ff881c00640 1 -- 172.21.15.96:0/1934584961 learned_addr learned my addr 172.21.15.96:0/1934584961 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:13:45.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.059+0000 7ff883000640 1 -- 172.21.15.96:0/1934584961 --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff87c0cecf0 con 0x7ff87c15ea40 2024-05-28T06:13:45.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.060+0000 7ff883000640 1 -- 172.21.15.96:0/1934584961 --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff87c0cefd0 con 0x7ff87c15ee20 2024-05-28T06:13:45.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.060+0000 7ff883000640 1 -- 172.21.15.96:0/1934584961 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ff87c0cf2b0 con 0x7ff87c0d5190 2024-05-28T06:13:45.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.060+0000 7ff881200640 1 --2- 172.21.15.96:0/1934584961 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff87c0d5190 0x7ff87c0d30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:45.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.060+0000 7ff87be00640 1 -- 172.21.15.96:0/1934584961 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4232140457 0 0) 0x7ff87c0cecf0 con 0x7ff87c15ea40 2024-05-28T06:13:45.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.060+0000 7ff87be00640 1 -- 172.21.15.96:0/1934584961 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff864003520 con 0x7ff87c15ea40 2024-05-28T06:13:45.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.060+0000 7ff87be00640 1 -- 172.21.15.96:0/1934584961 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1521758338 0 0) 0x7ff864003520 con 0x7ff87c15ea40 2024-05-28T06:13:45.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.060+0000 7ff87be00640 1 -- 172.21.15.96:0/1934584961 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff87c0d5190 msgr2=0x7ff87c0d30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:13:45.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.060+0000 7ff87be00640 1 --2- 172.21.15.96:0/1934584961 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff87c0d5190 0x7ff87c0d30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:45.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.060+0000 7ff87be00640 1 -- 172.21.15.96:0/1934584961 >> v1:172.21.15.123:6789/0 conn(0x7ff87c15ee20 legacy=0x7ff87c15fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:13:45.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.060+0000 7ff87be00640 1 -- 172.21.15.96:0/1934584961 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff87c0cf590 con 0x7ff87c15ea40 2024-05-28T06:13:45.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.060+0000 7ff87be00640 1 -- 172.21.15.96:0/1934584961 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7ff8700019f0 con 0x7ff87c15ea40 2024-05-28T06:13:45.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.061+0000 7ff87be00640 1 -- 172.21.15.96:0/1934584961 >> v1:172.21.15.96:6789/0 conn(0x7ff87c15ea40 legacy=0x7ff87c0cfab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:45.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.061+0000 7ff87be00640 1 --2- 172.21.15.96:0/1934584961 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8640039e0 0x7ff864003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:45.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.061+0000 7ff87be00640 1 --2- 172.21.15.96:0/1934584961 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff864004800 0x7ff864004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:45.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.061+0000 7ff87be00640 1 --2- 172.21.15.96:0/1934584961 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff87c0d5190 0x7ff8640051e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:45.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.061+0000 7ff87be00640 1 -- 172.21.15.96:0/1934584961 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff87c0cf2b0 con 0x7ff864004800 2024-05-28T06:13:45.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.061+0000 7ff87be00640 1 -- 172.21.15.96:0/1934584961 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff86400bad0 con 0x7ff8640039e0 2024-05-28T06:13:45.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.061+0000 7ff87be00640 1 -- 172.21.15.96:0/1934584961 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ff86400bcd0 con 0x7ff87c0d5190 2024-05-28T06:13:45.065 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.061+0000 7ff881200640 1 --2- 172.21.15.96:0/1934584961 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff864004800 0x7ff864004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:45.065 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.061+0000 7ff881200640 1 -- 172.21.15.96:0/1934584961 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff87c0d5190 msgr2=0x7ff8640051e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:13:45.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.061+0000 7ff881200640 1 --2- 172.21.15.96:0/1934584961 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff87c0d5190 0x7ff8640051e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:45.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.061+0000 7ff881200640 1 -- 172.21.15.96:0/1934584961 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8640039e0 msgr2=0x7ff864003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:13:45.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.061+0000 7ff881200640 1 --2- 172.21.15.96:0/1934584961 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8640039e0 0x7ff864003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:45.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.061+0000 7ff881200640 1 -- 172.21.15.96:0/1934584961 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff864003520 con 0x7ff864004800 2024-05-28T06:13:45.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.063+0000 7ff881200640 1 --2- 172.21.15.96:0/1934584961 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff864004800 0x7ff864004c80 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7ff86800b9e0 tx=0x7ff864005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8694cd5e0c1cccc3 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:13:45.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.064+0000 7ff87be00640 1 -- 172.21.15.96:0/1934584961 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff86800b5e0 con 0x7ff864004800 2024-05-28T06:13:45.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.065+0000 7ff87be00640 1 -- 172.21.15.96:0/1934584961 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff86802adb0 con 0x7ff864004800 2024-05-28T06:13:45.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.065+0000 7ff87be00640 1 -- 172.21.15.96:0/1934584961 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff868005bb0 con 0x7ff864004800 2024-05-28T06:13:45.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.066+0000 7ff883000640 1 -- 172.21.15.96:0/1934584961 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff864004800 msgr2=0x7ff864004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:45.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.066+0000 7ff883000640 1 --2- 172.21.15.96:0/1934584961 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff864004800 0x7ff864004c80 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7ff86800b9e0 tx=0x7ff864005830 comp rx=0 tx=0).stop 2024-05-28T06:13:45.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.066+0000 7ff882600640 1 -- 172.21.15.96:0/1934584961 reap_dead start 2024-05-28T06:13:45.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.066+0000 7ff883000640 1 -- 172.21.15.96:0/1934584961 shutdown_connections 2024-05-28T06:13:45.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.066+0000 7ff883000640 1 -- 172.21.15.96:0/1934584961 >> 172.21.15.96:0/1934584961 conn(0x7ff87c08bfc0 msgr2=0x7ff87c0d4500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:13:45.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.066+0000 7ff883000640 1 -- 172.21.15.96:0/1934584961 shutdown_connections 2024-05-28T06:13:45.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.066+0000 7ff883000640 1 -- 172.21.15.96:0/1934584961 wait complete. 2024-05-28T06:13:45.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.070+0000 7ff883000640 1 Processor -- start 2024-05-28T06:13:45.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.071+0000 7ff883000640 1 -- start start 2024-05-28T06:13:45.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.073+0000 7ff883000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff864004800 0x7ff87c0c94f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:45.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.073+0000 7ff883000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff87c150580 0x7ff87c0ca5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:45.075 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.073+0000 7ff883000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff87c1e8eb0 0x7ff87c151bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:45.075 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.073+0000 7ff883000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff87c15f660 con 0x7ff87c1e8eb0 2024-05-28T06:13:45.075 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.073+0000 7ff883000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff87c15f4e0 con 0x7ff87c150580 2024-05-28T06:13:45.075 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.073+0000 7ff883000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ff87c160700 con 0x7ff864004800 2024-05-28T06:13:45.075 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.073+0000 7ff882600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff87c1e8eb0 0x7ff87c151bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:45.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.073+0000 7ff882600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff87c1e8eb0 0x7ff87c151bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:56388/0 (socket says 172.21.15.96:56388) 2024-05-28T06:13:45.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.073+0000 7ff882600640 1 -- 172.21.15.96:0/459599230 learned_addr learned my addr 172.21.15.96:0/459599230 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:13:45.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.073+0000 7ff881200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff87c150580 0x7ff87c0ca5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:45.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.074+0000 7ff881c00640 1 --2- 172.21.15.96:0/459599230 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff864004800 0x7ff87c0c94f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:45.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.074+0000 7ff882600640 1 -- 172.21.15.96:0/459599230 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff864004800 msgr2=0x7ff87c0c94f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:45.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.074+0000 7ff882600640 1 --2- 172.21.15.96:0/459599230 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff864004800 0x7ff87c0c94f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:45.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.074+0000 7ff882600640 1 -- 172.21.15.96:0/459599230 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff87c150580 msgr2=0x7ff87c0ca5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:45.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.074+0000 7ff882600640 1 --2- 172.21.15.96:0/459599230 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff87c150580 0x7ff87c0ca5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:45.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.074+0000 7ff882600640 1 -- 172.21.15.96:0/459599230 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff87c0cad40 con 0x7ff87c1e8eb0 2024-05-28T06:13:45.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.074+0000 7ff882600640 1 --2- 172.21.15.96:0/459599230 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff87c1e8eb0 0x7ff87c151bf0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7ff87400c390 tx=0x7ff87400c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:13:45.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.075+0000 7ff879600640 1 -- 172.21.15.96:0/459599230 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8740044a0 con 0x7ff87c1e8eb0 2024-05-28T06:13:45.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.075+0000 7ff879600640 1 -- 172.21.15.96:0/459599230 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff874007480 con 0x7ff87c1e8eb0 2024-05-28T06:13:45.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.075+0000 7ff883000640 1 -- 172.21.15.96:0/459599230 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff87c1e80f0 con 0x7ff87c1e8eb0 2024-05-28T06:13:45.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.075+0000 7ff879600640 1 -- 172.21.15.96:0/459599230 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff874005a00 con 0x7ff87c1e8eb0 2024-05-28T06:13:45.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.075+0000 7ff883000640 1 -- 172.21.15.96:0/459599230 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff87c1e86b0 con 0x7ff87c1e8eb0 2024-05-28T06:13:45.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.076+0000 7ff883000640 1 -- 172.21.15.96:0/459599230 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff87c0690f0 con 0x7ff87c1e8eb0 2024-05-28T06:13:45.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.077+0000 7ff879600640 1 -- 172.21.15.96:0/459599230 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 20) ==== 92162+0+0 (secure 0 0 0) 0x7ff874011070 con 0x7ff87c1e8eb0 2024-05-28T06:13:45.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.078+0000 7ff879600640 1 --2- 172.21.15.96:0/459599230 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff85c06d550 0x7ff85c06fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:45.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.078+0000 7ff879600640 1 -- 172.21.15.96:0/459599230 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5737+0+0 (secure 0 0 0) 0x7ff874089bd0 con 0x7ff87c1e8eb0 2024-05-28T06:13:45.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.082+0000 7ff881c00640 1 --2- 172.21.15.96:0/459599230 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff85c06d550 0x7ff85c06fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:45.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.084+0000 7ff881c00640 1 --2- 172.21.15.96:0/459599230 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff85c06d550 0x7ff85c06fa10 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7ff8700012e0 tx=0x7ff870001a50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:13:45.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.086+0000 7ff879600640 1 -- 172.21.15.96:0/459599230 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff874019060 con 0x7ff87c1e8eb0 2024-05-28T06:13:45.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.424+0000 7ff883000640 1 -- 172.21.15.96:0/459599230 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone81717", "target": ["mon-mgr", ""]}) -- 0x7ff87c0c9a30 con 0x7ff85c06d550 2024-05-28T06:13:45.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.645+0000 7ff879600640 1 -- 172.21.15.96:0/459599230 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7ff87c0c9a30 con 0x7ff85c06d550 2024-05-28T06:13:45.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.648+0000 7ff883000640 1 -- 172.21.15.96:0/459599230 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff85c06d550 msgr2=0x7ff85c06fa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:45.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.648+0000 7ff883000640 1 --2- 172.21.15.96:0/459599230 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff85c06d550 0x7ff85c06fa10 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7ff8700012e0 tx=0x7ff870001a50 comp rx=0 tx=0).stop 2024-05-28T06:13:45.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.648+0000 7ff883000640 1 -- 172.21.15.96:0/459599230 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff87c1e8eb0 msgr2=0x7ff87c151bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:45.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.648+0000 7ff883000640 1 --2- 172.21.15.96:0/459599230 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff87c1e8eb0 0x7ff87c151bf0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7ff87400c390 tx=0x7ff87400c850 comp rx=0 tx=0).stop 2024-05-28T06:13:45.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.648+0000 7ff883000640 1 -- 172.21.15.96:0/459599230 shutdown_connections 2024-05-28T06:13:45.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.648+0000 7ff883000640 1 --2- 172.21.15.96:0/459599230 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff85c06d550 0x7ff85c06fa10 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:45.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.648+0000 7ff883000640 1 --2- 172.21.15.96:0/459599230 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff87c1e8eb0 0x7ff87c151bf0 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:45.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.648+0000 7ff883000640 1 --2- 172.21.15.96:0/459599230 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff87c150580 0x7ff87c0ca5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:45.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.648+0000 7ff883000640 1 --2- 172.21.15.96:0/459599230 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff864004800 0x7ff87c0c94f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:45.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.648+0000 7ff883000640 1 -- 172.21.15.96:0/459599230 >> 172.21.15.96:0/459599230 conn(0x7ff87c08bfc0 msgr2=0x7ff87c0cf9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:13:45.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.648+0000 7ff883000640 1 -- 172.21.15.96:0/459599230 shutdown_connections 2024-05-28T06:13:45.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.648+0000 7ff883000640 1 -- 172.21.15.96:0/459599230 wait complete. 2024-05-28T06:13:45.668 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-05-28T06:13:45.851 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.849+0000 7fb03b400640 1 Processor -- start 2024-05-28T06:13:45.851 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.850+0000 7fb03b400640 1 -- start start 2024-05-28T06:13:45.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.851+0000 7fb03b400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fb03406f860 0x7fb0340b3150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:45.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.851+0000 7fb03b400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fb0340b3690 0x7fb0340d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:45.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.851+0000 7fb03b400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb0340d09e0 0x7fb034160210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:45.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.851+0000 7fb03b400640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fb0340d0e30 con 0x7fb03406f860 2024-05-28T06:13:45.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.851+0000 7fb03b400640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fb03406ed10 con 0x7fb0340b3690 2024-05-28T06:13:45.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.851+0000 7fb03b400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fb03415cb00 con 0x7fb0340d09e0 2024-05-28T06:13:45.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.851+0000 7fb038a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fb03406f860 0x7fb0340b3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:45.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.851+0000 7fb038a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fb03406f860 0x7fb0340b3150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:56392/0 (socket says 172.21.15.96:56392) 2024-05-28T06:13:45.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.851+0000 7fb038a00640 1 -- 172.21.15.96:0/53837173 learned_addr learned my addr 172.21.15.96:0/53837173 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:13:45.854 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.852+0000 7fb038a00640 1 -- 172.21.15.96:0/53837173 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb0340d09e0 msgr2=0x7fb034160210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:13:45.854 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.852+0000 7fb038a00640 1 --2- 172.21.15.96:0/53837173 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb0340d09e0 0x7fb034160210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:45.854 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.852+0000 7fb038a00640 1 -- 172.21.15.96:0/53837173 >> v2:172.21.15.123:3300/0 conn(0x7fb0340b3690 msgr2=0x7fb0340d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:13:45.854 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.852+0000 7fb038a00640 1 --2- 172.21.15.96:0/53837173 >> v2:172.21.15.123:3300/0 conn(0x7fb0340b3690 0x7fb0340d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:45.854 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.852+0000 7fb038a00640 1 -- 172.21.15.96:0/53837173 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0340b3cc0 con 0x7fb03406f860 2024-05-28T06:13:45.855 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.853+0000 7fb038a00640 1 --2- 172.21.15.96:0/53837173 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb03406f860 0x7fb0340b3150 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fb034070120 tx=0x7fb02402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b87f4458f4c6172a server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:13:45.855 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.853+0000 7fb033e00640 1 -- 172.21.15.96:0/53837173 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb024037070 con 0x7fb03406f860 2024-05-28T06:13:45.855 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.853+0000 7fb033e00640 1 -- 172.21.15.96:0/53837173 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb024032440 con 0x7fb03406f860 2024-05-28T06:13:45.855 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.853+0000 7fb033e00640 1 -- 172.21.15.96:0/53837173 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb024032810 con 0x7fb03406f860 2024-05-28T06:13:45.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.854+0000 7fb03b400640 1 -- 172.21.15.96:0/53837173 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb03406f860 msgr2=0x7fb0340b3150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:45.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.854+0000 7fb03b400640 1 --2- 172.21.15.96:0/53837173 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb03406f860 0x7fb0340b3150 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fb034070120 tx=0x7fb02402a1e0 comp rx=0 tx=0).stop 2024-05-28T06:13:45.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.854+0000 7fb03b400640 1 -- 172.21.15.96:0/53837173 shutdown_connections 2024-05-28T06:13:45.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.854+0000 7fb03b400640 1 --2- 172.21.15.96:0/53837173 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb0340d09e0 0x7fb034160210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:45.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.854+0000 7fb03b400640 1 --2- 172.21.15.96:0/53837173 >> v2:172.21.15.123:3300/0 conn(0x7fb0340b3690 0x7fb0340d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:45.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.854+0000 7fb03b400640 1 --2- 172.21.15.96:0/53837173 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb03406f860 0x7fb0340b3150 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:45.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.854+0000 7fb03b400640 1 -- 172.21.15.96:0/53837173 >> 172.21.15.96:0/53837173 conn(0x7fb03406bae0 msgr2=0x7fb03406bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:13:45.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.854+0000 7fb03b400640 1 -- 172.21.15.96:0/53837173 shutdown_connections 2024-05-28T06:13:45.857 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.854+0000 7fb03b400640 1 -- 172.21.15.96:0/53837173 wait complete. 2024-05-28T06:13:45.857 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.856+0000 7fb03b400640 1 Processor -- start 2024-05-28T06:13:45.857 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.856+0000 7fb03b400640 1 -- start start 2024-05-28T06:13:45.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.857+0000 7fb03b400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb03406f860 0x7fb0341588f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:45.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.857+0000 7fb03b400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb0340b3690 0x7fb0340c4460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:45.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.857+0000 7fb03b400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb0340d09e0 0x7fb0340c54b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:45.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.857+0000 7fb03b400640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb0340b4560 con 0x7fb03406f860 2024-05-28T06:13:45.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.857+0000 7fb03b400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fb0340b4280 con 0x7fb0340b3690 2024-05-28T06:13:45.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.857+0000 7fb03b400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fb0340b4840 con 0x7fb0340d09e0 2024-05-28T06:13:45.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.857+0000 7fb038a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb03406f860 0x7fb0341588f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:45.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.857+0000 7fb038a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb03406f860 0x7fb0341588f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:56394/0 (socket says 172.21.15.96:56394) 2024-05-28T06:13:45.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.857+0000 7fb038a00640 1 -- 172.21.15.96:0/4165096292 learned_addr learned my addr 172.21.15.96:0/4165096292 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:13:45.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.858+0000 7fb038a00640 1 -- 172.21.15.96:0/4165096292 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb0340d09e0 msgr2=0x7fb0340c54b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:45.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.858+0000 7fb039400640 1 --2- 172.21.15.96:0/4165096292 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb0340d09e0 0x7fb0340c54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:45.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.858+0000 7fb02be00640 1 --2- 172.21.15.96:0/4165096292 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb0340b3690 0x7fb0340c4460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:45.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.858+0000 7fb038a00640 1 --2- 172.21.15.96:0/4165096292 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb0340d09e0 0x7fb0340c54b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:45.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.858+0000 7fb038a00640 1 -- 172.21.15.96:0/4165096292 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb0340b3690 msgr2=0x7fb0340c4460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:45.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.858+0000 7fb038a00640 1 --2- 172.21.15.96:0/4165096292 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb0340b3690 0x7fb0340c4460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:45.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.858+0000 7fb038a00640 1 -- 172.21.15.96:0/4165096292 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb03415db70 con 0x7fb03406f860 2024-05-28T06:13:45.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.858+0000 7fb038a00640 1 --2- 172.21.15.96:0/4165096292 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb03406f860 0x7fb0341588f0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7fb02402a220 tx=0x7fb0240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:13:45.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.858+0000 7fb031600640 1 -- 172.21.15.96:0/4165096292 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb024041070 con 0x7fb03406f860 2024-05-28T06:13:45.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.858+0000 7fb031600640 1 -- 172.21.15.96:0/4165096292 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb02403c870 con 0x7fb03406f860 2024-05-28T06:13:45.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.858+0000 7fb031600640 1 -- 172.21.15.96:0/4165096292 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb024037040 con 0x7fb03406f860 2024-05-28T06:13:45.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.858+0000 7fb03b400640 1 -- 172.21.15.96:0/4165096292 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb034158e30 con 0x7fb03406f860 2024-05-28T06:13:45.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.859+0000 7fb03b400640 1 -- 172.21.15.96:0/4165096292 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0340c5040 con 0x7fb03406f860 2024-05-28T06:13:45.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.860+0000 7fb03b400640 1 -- 172.21.15.96:0/4165096292 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0340689e0 con 0x7fb03406f860 2024-05-28T06:13:45.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.861+0000 7fb031600640 1 -- 172.21.15.96:0/4165096292 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 20) ==== 92162+0+0 (secure 0 0 0) 0x7fb024044070 con 0x7fb03406f860 2024-05-28T06:13:45.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.861+0000 7fb031600640 1 --2- 172.21.15.96:0/4165096292 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb01006d550 0x7fb01006fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:45.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.862+0000 7fb031600640 1 -- 172.21.15.96:0/4165096292 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5737+0+0 (secure 0 0 0) 0x7fb0240ab160 con 0x7fb03406f860 2024-05-28T06:13:45.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.862+0000 7fb02be00640 1 --2- 172.21.15.96:0/4165096292 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb01006d550 0x7fb01006fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:45.865 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.863+0000 7fb02be00640 1 --2- 172.21.15.96:0/4165096292 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb01006d550 0x7fb01006fa10 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fb03409e480 tx=0x7fb01c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:13:45.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:45.867+0000 7fb031600640 1 -- 172.21.15.96:0/4165096292 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb02407b020 con 0x7fb03406f860 2024-05-28T06:13:46.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.082+0000 7fb03b400640 1 -- 172.21.15.96:0/4165096292 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7fb034069210 con 0x7fb01006d550 2024-05-28T06:13:46.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.086+0000 7fb031600640 1 -- 172.21.15.96:0/4165096292 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7fb034069210 con 0x7fb01006d550 2024-05-28T06:13:46.088 INFO:teuthology.orchestra.run.smithi096.stdout:[ 2024-05-28T06:13:46.088 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:13:46.088 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "cephfs" 2024-05-28T06:13:46.088 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:13:46.088 INFO:teuthology.orchestra.run.smithi096.stdout:] 2024-05-28T06:13:46.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.088+0000 7fb03b400640 1 -- 172.21.15.96:0/4165096292 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb01006d550 msgr2=0x7fb01006fa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:46.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.088+0000 7fb03b400640 1 --2- 172.21.15.96:0/4165096292 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb01006d550 0x7fb01006fa10 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fb03409e480 tx=0x7fb01c008040 comp rx=0 tx=0).stop 2024-05-28T06:13:46.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.088+0000 7fb03b400640 1 -- 172.21.15.96:0/4165096292 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb03406f860 msgr2=0x7fb0341588f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:46.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.088+0000 7fb03b400640 1 --2- 172.21.15.96:0/4165096292 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb03406f860 0x7fb0341588f0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7fb02402a220 tx=0x7fb0240046c0 comp rx=0 tx=0).stop 2024-05-28T06:13:46.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.089+0000 7fb03b400640 1 -- 172.21.15.96:0/4165096292 shutdown_connections 2024-05-28T06:13:46.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.089+0000 7fb03b400640 1 --2- 172.21.15.96:0/4165096292 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb01006d550 0x7fb01006fa10 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:46.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.089+0000 7fb03b400640 1 --2- 172.21.15.96:0/4165096292 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb0340d09e0 0x7fb0340c54b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:46.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.089+0000 7fb03b400640 1 --2- 172.21.15.96:0/4165096292 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb0340b3690 0x7fb0340c4460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:46.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.089+0000 7fb03b400640 1 --2- 172.21.15.96:0/4165096292 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb03406f860 0x7fb0341588f0 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:46.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.089+0000 7fb03b400640 1 -- 172.21.15.96:0/4165096292 >> 172.21.15.96:0/4165096292 conn(0x7fb03406bae0 msgr2=0x7fb0340afe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:13:46.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.089+0000 7fb03b400640 1 -- 172.21.15.96:0/4165096292 shutdown_connections 2024-05-28T06:13:46.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.089+0000 7fb03b400640 1 -- 172.21.15.96:0/4165096292 wait complete. 2024-05-28T06:13:46.108 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-05-28T06:13:46.300 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.298+0000 7f8999600640 1 Processor -- start 2024-05-28T06:13:46.301 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.299+0000 7f8999600640 1 -- start start 2024-05-28T06:13:46.302 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.300+0000 7f8999600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f8994161df0 0x7f8994099800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:46.303 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.300+0000 7f8999600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f89940d0b70 0x7f89940cefa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:46.303 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.300+0000 7f8999600640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89941621d0 con 0x7f89940cc2b0 2024-05-28T06:13:46.303 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.300+0000 7f8999600640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f89940cd840 con 0x7f8994161df0 2024-05-28T06:13:46.303 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.300+0000 7f8999600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f89940ccf10 con 0x7f89940d0b70 2024-05-28T06:13:46.303 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.301+0000 7f8993e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f8994161df0 0x7f8994099800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:46.303 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.301+0000 7f898be00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f89940cc2b0 0x7f89940d0460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:53492/0 (socket says 172.21.15.96:53492) 2024-05-28T06:13:46.303 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.301+0000 7f898be00640 1 -- 172.21.15.96:0/2831219592 learned_addr learned my addr 172.21.15.96:0/2831219592 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:13:46.303 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.301+0000 7f8993e00640 1 -- 172.21.15.96:0/2831219592 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f89940d0b70 msgr2=0x7f89940cefa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:13:46.303 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.301+0000 7f8993e00640 1 --2- 172.21.15.96:0/2831219592 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f89940d0b70 0x7f89940cefa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:46.303 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.301+0000 7f8993e00640 1 -- 172.21.15.96:0/2831219592 >> v1:172.21.15.96:6789/0 conn(0x7f89940cc2b0 legacy=0x7f89940d0460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:46.303 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.301+0000 7f8993e00640 1 -- 172.21.15.96:0/2831219592 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89940cd1f0 con 0x7f8994161df0 2024-05-28T06:13:46.303 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.301+0000 7f8993400640 1 -- 172.21.15.96:0/2831219592 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4020584008 0 0) 0x7f897c001110 con 0x7f89940cc2b0 2024-05-28T06:13:46.304 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.302+0000 7f8993e00640 1 --2- 172.21.15.96:0/2831219592 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8994161df0 0x7f8994099800 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f89840097a0 tx=0x7f898402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf49bee1a801f6cc server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:13:46.305 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.302+0000 7f8993400640 1 -- 172.21.15.96:0/2831219592 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8984037070 con 0x7f8994161df0 2024-05-28T06:13:46.305 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.302+0000 7f8993400640 1 -- 172.21.15.96:0/2831219592 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8984032440 con 0x7f8994161df0 2024-05-28T06:13:46.305 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.302+0000 7f8993400640 1 -- 172.21.15.96:0/2831219592 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8984032810 con 0x7f8994161df0 2024-05-28T06:13:46.305 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.303+0000 7f8999600640 1 -- 172.21.15.96:0/2831219592 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8994161df0 msgr2=0x7f8994099800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:46.305 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.303+0000 7f8999600640 1 --2- 172.21.15.96:0/2831219592 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8994161df0 0x7f8994099800 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f89840097a0 tx=0x7f898402a1a0 comp rx=0 tx=0).stop 2024-05-28T06:13:46.305 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.303+0000 7f8999600640 1 -- 172.21.15.96:0/2831219592 shutdown_connections 2024-05-28T06:13:46.305 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.303+0000 7f8999600640 1 --2- 172.21.15.96:0/2831219592 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f89940d0b70 0x7f89940cefa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:46.305 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.303+0000 7f8999600640 1 --2- 172.21.15.96:0/2831219592 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8994161df0 0x7f8994099800 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:46.305 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.303+0000 7f8999600640 1 -- 172.21.15.96:0/2831219592 >> 172.21.15.96:0/2831219592 conn(0x7f899406bae0 msgr2=0x7f899406bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:13:46.305 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.303+0000 7f8999600640 1 -- 172.21.15.96:0/2831219592 shutdown_connections 2024-05-28T06:13:46.305 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.303+0000 7f8999600640 1 -- 172.21.15.96:0/2831219592 wait complete. 2024-05-28T06:13:46.306 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.305+0000 7f8999600640 1 Processor -- start 2024-05-28T06:13:46.306 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.305+0000 7f8999600640 1 -- start start 2024-05-28T06:13:46.308 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.306+0000 7f8999600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f89940cc2b0 0x7f899414c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:46.308 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.306+0000 7f8999600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f89940d0b70 0x7f89940c8860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:46.308 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.306+0000 7f8999600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8994161df0 0x7f89940c72f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:46.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.306+0000 7f8999600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f89940ccbc0 con 0x7f89940d0b70 2024-05-28T06:13:46.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.306+0000 7f8999600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f8994099d40 con 0x7f89940cc2b0 2024-05-28T06:13:46.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.306+0000 7f8999600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f89940ca870 con 0x7f8994161df0 2024-05-28T06:13:46.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.306+0000 7f898be00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f89940d0b70 0x7f89940c8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:46.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.306+0000 7f898be00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f89940d0b70 0x7f89940c8860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:56418/0 (socket says 172.21.15.96:56418) 2024-05-28T06:13:46.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.306+0000 7f898be00640 1 -- 172.21.15.96:0/4141133089 learned_addr learned my addr 172.21.15.96:0/4141133089 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:13:46.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.306+0000 7f898be00640 1 -- 172.21.15.96:0/4141133089 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8994161df0 msgr2=0x7f89940c72f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:46.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.306+0000 7f8993e00640 1 --2- 172.21.15.96:0/4141133089 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f89940cc2b0 0x7f899414c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:46.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.306+0000 7f8998c00640 1 --2- 172.21.15.96:0/4141133089 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8994161df0 0x7f89940c72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:46.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.306+0000 7f898be00640 1 --2- 172.21.15.96:0/4141133089 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8994161df0 0x7f89940c72f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:46.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.306+0000 7f898be00640 1 -- 172.21.15.96:0/4141133089 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f89940cc2b0 msgr2=0x7f899414c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:46.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.306+0000 7f898be00640 1 --2- 172.21.15.96:0/4141133089 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f89940cc2b0 0x7f899414c0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:46.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.306+0000 7f898be00640 1 -- 172.21.15.96:0/4141133089 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89940c7db0 con 0x7f89940d0b70 2024-05-28T06:13:46.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.307+0000 7f898be00640 1 --2- 172.21.15.96:0/4141133089 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f89940d0b70 0x7f89940c8860 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f897c00c7b0 tx=0x7f897c00cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:13:46.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.307+0000 7f8990c00640 1 -- 172.21.15.96:0/4141133089 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f897c018070 con 0x7f89940d0b70 2024-05-28T06:13:46.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.307+0000 7f8990c00640 1 -- 172.21.15.96:0/4141133089 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f897c0072d0 con 0x7f89940d0b70 2024-05-28T06:13:46.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.307+0000 7f8990c00640 1 -- 172.21.15.96:0/4141133089 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f897c0047e0 con 0x7f89940d0b70 2024-05-28T06:13:46.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.307+0000 7f8999600640 1 -- 172.21.15.96:0/4141133089 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89940c8090 con 0x7f89940d0b70 2024-05-28T06:13:46.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.307+0000 7f8999600640 1 -- 172.21.15.96:0/4141133089 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f899414b8d0 con 0x7f89940d0b70 2024-05-28T06:13:46.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.308+0000 7f8999600640 1 -- 172.21.15.96:0/4141133089 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89940689e0 con 0x7f89940d0b70 2024-05-28T06:13:46.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.309+0000 7f8990c00640 1 -- 172.21.15.96:0/4141133089 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 20) ==== 92162+0+0 (secure 0 0 0) 0x7f897c01b070 con 0x7f89940d0b70 2024-05-28T06:13:46.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.310+0000 7f8990c00640 1 --2- 172.21.15.96:0/4141133089 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f897006d550 0x7f897006fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:46.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.310+0000 7f8990c00640 1 -- 172.21.15.96:0/4141133089 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5737+0+0 (secure 0 0 0) 0x7f897c089ac0 con 0x7f89940d0b70 2024-05-28T06:13:46.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.311+0000 7f8993e00640 1 --2- 172.21.15.96:0/4141133089 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f897006d550 0x7f897006fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:46.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.312+0000 7f8993e00640 1 --2- 172.21.15.96:0/4141133089 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f897006d550 0x7f897006fa10 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f8984034000 tx=0x7f898402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:13:46.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.314+0000 7f8990c00640 1 -- 172.21.15.96:0/4141133089 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f89941621d0 con 0x7f89940d0b70 2024-05-28T06:13:46.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.588+0000 7f8999600640 1 -- 172.21.15.96:0/4141133089 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f89940b4c90 con 0x7f89940d0b70 2024-05-28T06:13:46.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.588+0000 7f8990c00640 1 -- 172.21.15.96:0/4141133089 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+2421 (secure 0 0 0) 0x7f897c059720 con 0x7f89940d0b70 2024-05-28T06:13:46.590 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:13:46.590 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:13:46.591 INFO:teuthology.orchestra.run.smithi096.stdout: "fsid": "e4a831b5-e6d1-47a7-96cb-bf85154ee0d2", 2024-05-28T06:13:46.591 INFO:teuthology.orchestra.run.smithi096.stdout: "health": { 2024-05-28T06:13:46.591 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "HEALTH_OK", 2024-05-28T06:13:46.591 INFO:teuthology.orchestra.run.smithi096.stdout: "checks": {}, 2024-05-28T06:13:46.591 INFO:teuthology.orchestra.run.smithi096.stdout: "mutes": [] 2024-05-28T06:13:46.591 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:13:46.591 INFO:teuthology.orchestra.run.smithi096.stdout: "election_epoch": 8, 2024-05-28T06:13:46.591 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum": [ 2024-05-28T06:13:46.592 INFO:teuthology.orchestra.run.smithi096.stdout: 0, 2024-05-28T06:13:46.592 INFO:teuthology.orchestra.run.smithi096.stdout: 1, 2024-05-28T06:13:46.592 INFO:teuthology.orchestra.run.smithi096.stdout: 2 2024-05-28T06:13:46.592 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:13:46.592 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_names": [ 2024-05-28T06:13:46.592 INFO:teuthology.orchestra.run.smithi096.stdout: "a", 2024-05-28T06:13:46.592 INFO:teuthology.orchestra.run.smithi096.stdout: "b", 2024-05-28T06:13:46.593 INFO:teuthology.orchestra.run.smithi096.stdout: "c" 2024-05-28T06:13:46.593 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:13:46.593 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_age": 600, 2024-05-28T06:13:46.593 INFO:teuthology.orchestra.run.smithi096.stdout: "monmap": { 2024-05-28T06:13:46.593 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 1, 2024-05-28T06:13:46.593 INFO:teuthology.orchestra.run.smithi096.stdout: "min_mon_release_name": "squid", 2024-05-28T06:13:46.593 INFO:teuthology.orchestra.run.smithi096.stdout: "num_mons": 3 2024-05-28T06:13:46.593 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:13:46.594 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap": { 2024-05-28T06:13:46.594 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 60, 2024-05-28T06:13:46.594 INFO:teuthology.orchestra.run.smithi096.stdout: "num_osds": 8, 2024-05-28T06:13:46.594 INFO:teuthology.orchestra.run.smithi096.stdout: "num_up_osds": 8, 2024-05-28T06:13:46.594 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_up_since": 1716876238, 2024-05-28T06:13:46.594 INFO:teuthology.orchestra.run.smithi096.stdout: "num_in_osds": 8, 2024-05-28T06:13:46.594 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_in_since": 1716876229, 2024-05-28T06:13:46.595 INFO:teuthology.orchestra.run.smithi096.stdout: "num_remapped_pgs": 0 2024-05-28T06:13:46.595 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:13:46.595 INFO:teuthology.orchestra.run.smithi096.stdout: "pgmap": { 2024-05-28T06:13:46.595 INFO:teuthology.orchestra.run.smithi096.stdout: "pgs_by_state": [ 2024-05-28T06:13:46.595 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:13:46.595 INFO:teuthology.orchestra.run.smithi096.stdout: "state_name": "active+clean", 2024-05-28T06:13:46.595 INFO:teuthology.orchestra.run.smithi096.stdout: "count": 97 2024-05-28T06:13:46.595 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:13:46.595 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:13:46.596 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pgs": 97, 2024-05-28T06:13:46.596 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pools": 3, 2024-05-28T06:13:46.596 INFO:teuthology.orchestra.run.smithi096.stdout: "num_objects": 4211, 2024-05-28T06:13:46.596 INFO:teuthology.orchestra.run.smithi096.stdout: "data_bytes": 17479660722, 2024-05-28T06:13:46.596 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_used": 34596646912, 2024-05-28T06:13:46.596 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_avail": 738497466368, 2024-05-28T06:13:46.596 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_total": 773094113280, 2024-05-28T06:13:46.596 INFO:teuthology.orchestra.run.smithi096.stdout: "read_bytes_sec": 957282963, 2024-05-28T06:13:46.596 INFO:teuthology.orchestra.run.smithi096.stdout: "write_bytes_sec": 959801920, 2024-05-28T06:13:46.596 INFO:teuthology.orchestra.run.smithi096.stdout: "read_op_per_sec": 232, 2024-05-28T06:13:46.597 INFO:teuthology.orchestra.run.smithi096.stdout: "write_op_per_sec": 253 2024-05-28T06:13:46.597 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:13:46.597 INFO:teuthology.orchestra.run.smithi096.stdout: "fsmap": { 2024-05-28T06:13:46.597 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 35, 2024-05-28T06:13:46.597 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 4, 2024-05-28T06:13:46.597 INFO:teuthology.orchestra.run.smithi096.stdout: "up": 1, 2024-05-28T06:13:46.597 INFO:teuthology.orchestra.run.smithi096.stdout: "in": 1, 2024-05-28T06:13:46.597 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 1, 2024-05-28T06:13:46.597 INFO:teuthology.orchestra.run.smithi096.stdout: "by_rank": [ 2024-05-28T06:13:46.597 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:13:46.597 INFO:teuthology.orchestra.run.smithi096.stdout: "filesystem_id": 4, 2024-05-28T06:13:46.597 INFO:teuthology.orchestra.run.smithi096.stdout: "rank": 0, 2024-05-28T06:13:46.598 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "a", 2024-05-28T06:13:46.598 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "up:active", 2024-05-28T06:13:46.598 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 6489 2024-05-28T06:13:46.598 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:13:46.598 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:13:46.598 INFO:teuthology.orchestra.run.smithi096.stdout: "up:standby": 3 2024-05-28T06:13:46.598 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:13:46.598 INFO:teuthology.orchestra.run.smithi096.stdout: "mgrmap": { 2024-05-28T06:13:46.598 INFO:teuthology.orchestra.run.smithi096.stdout: "available": true, 2024-05-28T06:13:46.598 INFO:teuthology.orchestra.run.smithi096.stdout: "num_standbys": 1, 2024-05-28T06:13:46.598 INFO:teuthology.orchestra.run.smithi096.stdout: "modules": [ 2024-05-28T06:13:46.599 INFO:teuthology.orchestra.run.smithi096.stdout: "iostat", 2024-05-28T06:13:46.599 INFO:teuthology.orchestra.run.smithi096.stdout: "nfs", 2024-05-28T06:13:46.599 INFO:teuthology.orchestra.run.smithi096.stdout: "restful" 2024-05-28T06:13:46.599 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:13:46.599 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:13:46.599 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:13:46.599 INFO:teuthology.orchestra.run.smithi096.stdout: "servicemap": { 2024-05-28T06:13:46.599 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 11, 2024-05-28T06:13:46.599 INFO:teuthology.orchestra.run.smithi096.stdout: "modified": "2024-05-28T06:13:33.716527+0000", 2024-05-28T06:13:46.599 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:13:46.599 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:13:46.600 INFO:teuthology.orchestra.run.smithi096.stdout: "progress_events": { 2024-05-28T06:13:46.600 INFO:teuthology.orchestra.run.smithi096.stdout: "58c36a82-8e7f-41c9-9e90-890fce489f3b": { 2024-05-28T06:13:46.600 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "1 ongoing clones; Avg progress is 23.958% (19s)\n [======......................] (remaining: 61s)", 2024-05-28T06:13:46.600 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": 0.23958000540733337, 2024-05-28T06:13:46.600 INFO:teuthology.orchestra.run.smithi096.stdout: "add_to_ceph_s": true 2024-05-28T06:13:46.600 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:13:46.600 INFO:teuthology.orchestra.run.smithi096.stdout: "6a73bfaa-3b9e-443d-83e8-82fc1faa8b7f": { 2024-05-28T06:13:46.600 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "1 ongoing+pending clones; Avg progress is 23.958% (13s)\n [======......................] (remaining: 42s)", 2024-05-28T06:13:46.600 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": 0.23958000540733337, 2024-05-28T06:13:46.600 INFO:teuthology.orchestra.run.smithi096.stdout: "add_to_ceph_s": true 2024-05-28T06:13:46.600 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:13:46.600 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:13:46.601 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:13:46.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.592+0000 7f8999600640 1 -- 172.21.15.96:0/4141133089 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f897006d550 msgr2=0x7f897006fa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:46.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.592+0000 7f8999600640 1 --2- 172.21.15.96:0/4141133089 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f897006d550 0x7f897006fa10 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f8984034000 tx=0x7f898402e000 comp rx=0 tx=0).stop 2024-05-28T06:13:46.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.592+0000 7f8999600640 1 -- 172.21.15.96:0/4141133089 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f89940d0b70 msgr2=0x7f89940c8860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:46.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.592+0000 7f8999600640 1 --2- 172.21.15.96:0/4141133089 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f89940d0b70 0x7f89940c8860 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f897c00c7b0 tx=0x7f897c00cc70 comp rx=0 tx=0).stop 2024-05-28T06:13:46.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.592+0000 7f8999600640 1 -- 172.21.15.96:0/4141133089 shutdown_connections 2024-05-28T06:13:46.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.592+0000 7f8999600640 1 --2- 172.21.15.96:0/4141133089 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f897006d550 0x7f897006fa10 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:46.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.592+0000 7f8999600640 1 --2- 172.21.15.96:0/4141133089 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8994161df0 0x7f89940c72f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:46.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.592+0000 7f8999600640 1 --2- 172.21.15.96:0/4141133089 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f89940d0b70 0x7f89940c8860 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:46.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.592+0000 7f8999600640 1 --2- 172.21.15.96:0/4141133089 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f89940cc2b0 0x7f899414c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:46.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.592+0000 7f8999600640 1 -- 172.21.15.96:0/4141133089 >> 172.21.15.96:0/4141133089 conn(0x7f899406bae0 msgr2=0x7f89940b25f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:13:46.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.592+0000 7f8999600640 1 -- 172.21.15.96:0/4141133089 shutdown_connections 2024-05-28T06:13:46.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:46.592+0000 7f8999600640 1 -- 172.21.15.96:0/4141133089 wait complete. 2024-05-28T06:13:51.614 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-05-28T06:13:51.822 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:51.822+0000 7f87e3200640 1 Processor -- start 2024-05-28T06:13:51.823 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:51.822+0000 7f87e3200640 1 -- start start 2024-05-28T06:13:51.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:51.823+0000 7f87e3200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f87e415c6b0 0x7f87e40b3db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:51.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:51.823+0000 7f87e3200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f87e40ccc30 0x7f87e406dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:51.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:51.823+0000 7f87e3200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87e40cd1a0 con 0x7f87e40d0460 2024-05-28T06:13:51.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:51.823+0000 7f87e3200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f87e40cd480 con 0x7f87e40ccc30 2024-05-28T06:13:51.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:51.823+0000 7f87e3200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f87e40cd760 con 0x7f87e415c6b0 2024-05-28T06:13:51.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:51.823+0000 7f87d9400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f87e40ccc30 0x7f87e406dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:51.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:51.823+0000 7f87d9400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f87e40ccc30 0x7f87e406dc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47018/0 (socket says 172.21.15.96:47018) 2024-05-28T06:13:51.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:51.823+0000 7f87d9400640 1 -- 172.21.15.96:0/2128522330 learned_addr learned my addr 172.21.15.96:0/2128522330 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:13:51.825 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:51.824+0000 7f87e1e00640 1 --2- 172.21.15.96:0/2128522330 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f87e415c6b0 0x7f87e40b3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:51.825 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:51.824+0000 7f87d9400640 1 -- 172.21.15.96:0/2128522330 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f87e415c6b0 msgr2=0x7f87e40b3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:13:51.826 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:51.824+0000 7f87d9400640 1 --2- 172.21.15.96:0/2128522330 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f87e415c6b0 0x7f87e40b3db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:51.826 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:51.824+0000 7f87d9400640 1 -- 172.21.15.96:0/2128522330 >> v1:172.21.15.123:6789/0 conn(0x7f87e40d0460 legacy=0x7f87e40cbf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:13:51.826 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:51.824+0000 7f87d9400640 1 -- 172.21.15.96:0/2128522330 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87e40cda40 con 0x7f87e40ccc30 2024-05-28T06:13:51.826 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:51.825+0000 7f87d9400640 1 --2- 172.21.15.96:0/2128522330 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f87e40ccc30 0x7f87e406dc90 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f87cc009870 tx=0x7f87cc02a200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f211465f5005008b server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:13:51.826 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:51.825+0000 7f87e1400640 1 -- 172.21.15.96:0/2128522330 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87cc037070 con 0x7f87e40ccc30 2024-05-28T06:13:51.826 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:51.825+0000 7f87e1400640 1 -- 172.21.15.96:0/2128522330 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87cc032440 con 0x7f87e40ccc30 2024-05-28T06:13:51.826 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:51.825+0000 7f87e1400640 1 -- 172.21.15.96:0/2128522330 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87cc032810 con 0x7f87e40ccc30 2024-05-28T06:13:51.827 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:51.826+0000 7f87e3200640 1 -- 172.21.15.96:0/2128522330 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f87e40ccc30 msgr2=0x7f87e406dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:51.827 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:51.826+0000 7f87e3200640 1 --2- 172.21.15.96:0/2128522330 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f87e40ccc30 0x7f87e406dc90 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f87cc009870 tx=0x7f87cc02a200 comp rx=0 tx=0).stop 2024-05-28T06:13:51.827 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:51.826+0000 7f87e3200640 1 -- 172.21.15.96:0/2128522330 shutdown_connections 2024-05-28T06:13:51.827 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:51.826+0000 7f87e3200640 1 --2- 172.21.15.96:0/2128522330 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f87e40ccc30 0x7f87e406dc90 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:51.827 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:51.826+0000 7f87e3200640 1 --2- 172.21.15.96:0/2128522330 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f87e415c6b0 0x7f87e40b3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:51.827 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:51.826+0000 7f87e3200640 1 -- 172.21.15.96:0/2128522330 >> 172.21.15.96:0/2128522330 conn(0x7f87e406bae0 msgr2=0x7f87e406bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:13:51.827 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:51.826+0000 7f87e3200640 1 -- 172.21.15.96:0/2128522330 shutdown_connections 2024-05-28T06:13:51.828 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:51.826+0000 7f87e3200640 1 -- 172.21.15.96:0/2128522330 wait complete. 2024-05-28T06:13:51.828 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:51.828+0000 7f87e3200640 1 Processor -- start 2024-05-28T06:13:51.828 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:51.828+0000 7f87e3200640 1 -- start start 2024-05-28T06:13:51.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:51.829+0000 7f87e3200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87e40ccc30 0x7f87e4156dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:51.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:51.829+0000 7f87e3200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f87e40d0460 0x7f87e40c6740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:51.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:51.829+0000 7f87e3200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f87e415c6b0 0x7f87e40c50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:51.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:51.829+0000 7f87e3200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f87e40b4470 con 0x7f87e40d0460 2024-05-28T06:13:51.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:51.829+0000 7f87e3200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f87e40b42f0 con 0x7f87e40ccc30 2024-05-28T06:13:51.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:51.829+0000 7f87e3200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f87e406e1d0 con 0x7f87e415c6b0 2024-05-28T06:13:51.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:51.829+0000 7f87d9400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f87e40d0460 0x7f87e40c6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:51.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:51.829+0000 7f87d9400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f87e40d0460 0x7f87e40c6740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47028/0 (socket says 172.21.15.96:47028) 2024-05-28T06:13:51.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:51.829+0000 7f87d9400640 1 -- 172.21.15.96:0/913412645 learned_addr learned my addr 172.21.15.96:0/913412645 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:13:51.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:51.829+0000 7f87d9400640 1 -- 172.21.15.96:0/913412645 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f87e415c6b0 msgr2=0x7f87e40c50c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:51.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:51.829+0000 7f87d9400640 1 --2- 172.21.15.96:0/913412645 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f87e415c6b0 0x7f87e40c50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:51.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:51.829+0000 7f87d9400640 1 -- 172.21.15.96:0/913412645 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87e40ccc30 msgr2=0x7f87e4156dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:51.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:51.829+0000 7f87d9400640 1 --2- 172.21.15.96:0/913412645 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87e40ccc30 0x7f87e4156dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:51.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:51.829+0000 7f87d9400640 1 -- 172.21.15.96:0/913412645 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87e40c5810 con 0x7f87e40d0460 2024-05-28T06:13:51.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:51.830+0000 7f87d9400640 1 --2- 172.21.15.96:0/913412645 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f87e40d0460 0x7f87e40c6740 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f87cc009870 tx=0x7f87cc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:13:51.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:51.830+0000 7f87daa00640 1 -- 172.21.15.96:0/913412645 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87cc041020 con 0x7f87e40d0460 2024-05-28T06:13:51.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:51.830+0000 7f87daa00640 1 -- 172.21.15.96:0/913412645 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87cc030970 con 0x7f87e40d0460 2024-05-28T06:13:51.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:51.830+0000 7f87e3200640 1 -- 172.21.15.96:0/913412645 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87e40c5af0 con 0x7f87e40d0460 2024-05-28T06:13:51.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:51.830+0000 7f87daa00640 1 -- 172.21.15.96:0/913412645 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87cc037040 con 0x7f87e40d0460 2024-05-28T06:13:51.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:51.830+0000 7f87e3200640 1 -- 172.21.15.96:0/913412645 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87e40c60b0 con 0x7f87e40d0460 2024-05-28T06:13:51.833 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:51.832+0000 7f87e3200640 1 -- 172.21.15.96:0/913412645 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87e40afdc0 con 0x7f87e40d0460 2024-05-28T06:13:51.833 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:51.832+0000 7f87daa00640 1 -- 172.21.15.96:0/913412645 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 20) ==== 92162+0+0 (secure 0 0 0) 0x7f87cc044020 con 0x7f87e40d0460 2024-05-28T06:13:51.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:51.833+0000 7f87daa00640 1 --2- 172.21.15.96:0/913412645 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f87c006d550 0x7f87c006fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:51.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:51.833+0000 7f87daa00640 1 -- 172.21.15.96:0/913412645 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5737+0+0 (secure 0 0 0) 0x7f87cc0aabb0 con 0x7f87e40d0460 2024-05-28T06:13:51.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:51.834+0000 7f87e1e00640 1 --2- 172.21.15.96:0/913412645 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f87c006d550 0x7f87c006fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:51.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:51.835+0000 7f87e1e00640 1 --2- 172.21.15.96:0/913412645 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f87c006d550 0x7f87c006fa10 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f87e40c7270 tx=0x7f87d4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:13:51.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:51.838+0000 7f87daa00640 1 -- 172.21.15.96:0/913412645 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f87cc04b070 con 0x7f87e40d0460 2024-05-28T06:13:52.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:52.116+0000 7f87e3200640 1 -- 172.21.15.96:0/913412645 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f87e40689e0 con 0x7f87e40d0460 2024-05-28T06:13:52.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:52.117+0000 7f87daa00640 1 -- 172.21.15.96:0/913412645 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+2379 (secure 0 0 0) 0x7f87cc07aa90 con 0x7f87e40d0460 2024-05-28T06:13:52.118 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:13:52.118 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:13:52.118 INFO:teuthology.orchestra.run.smithi096.stdout: "fsid": "e4a831b5-e6d1-47a7-96cb-bf85154ee0d2", 2024-05-28T06:13:52.118 INFO:teuthology.orchestra.run.smithi096.stdout: "health": { 2024-05-28T06:13:52.118 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "HEALTH_OK", 2024-05-28T06:13:52.118 INFO:teuthology.orchestra.run.smithi096.stdout: "checks": {}, 2024-05-28T06:13:52.118 INFO:teuthology.orchestra.run.smithi096.stdout: "mutes": [] 2024-05-28T06:13:52.119 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:13:52.119 INFO:teuthology.orchestra.run.smithi096.stdout: "election_epoch": 8, 2024-05-28T06:13:52.119 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum": [ 2024-05-28T06:13:52.119 INFO:teuthology.orchestra.run.smithi096.stdout: 0, 2024-05-28T06:13:52.119 INFO:teuthology.orchestra.run.smithi096.stdout: 1, 2024-05-28T06:13:52.119 INFO:teuthology.orchestra.run.smithi096.stdout: 2 2024-05-28T06:13:52.119 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:13:52.119 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_names": [ 2024-05-28T06:13:52.119 INFO:teuthology.orchestra.run.smithi096.stdout: "a", 2024-05-28T06:13:52.119 INFO:teuthology.orchestra.run.smithi096.stdout: "b", 2024-05-28T06:13:52.119 INFO:teuthology.orchestra.run.smithi096.stdout: "c" 2024-05-28T06:13:52.119 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:13:52.119 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_age": 605, 2024-05-28T06:13:52.119 INFO:teuthology.orchestra.run.smithi096.stdout: "monmap": { 2024-05-28T06:13:52.120 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 1, 2024-05-28T06:13:52.120 INFO:teuthology.orchestra.run.smithi096.stdout: "min_mon_release_name": "squid", 2024-05-28T06:13:52.120 INFO:teuthology.orchestra.run.smithi096.stdout: "num_mons": 3 2024-05-28T06:13:52.120 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:13:52.120 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap": { 2024-05-28T06:13:52.120 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 60, 2024-05-28T06:13:52.120 INFO:teuthology.orchestra.run.smithi096.stdout: "num_osds": 8, 2024-05-28T06:13:52.120 INFO:teuthology.orchestra.run.smithi096.stdout: "num_up_osds": 8, 2024-05-28T06:13:52.120 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_up_since": 1716876238, 2024-05-28T06:13:52.120 INFO:teuthology.orchestra.run.smithi096.stdout: "num_in_osds": 8, 2024-05-28T06:13:52.120 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_in_since": 1716876229, 2024-05-28T06:13:52.120 INFO:teuthology.orchestra.run.smithi096.stdout: "num_remapped_pgs": 0 2024-05-28T06:13:52.120 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:13:52.121 INFO:teuthology.orchestra.run.smithi096.stdout: "pgmap": { 2024-05-28T06:13:52.121 INFO:teuthology.orchestra.run.smithi096.stdout: "pgs_by_state": [ 2024-05-28T06:13:52.121 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:13:52.121 INFO:teuthology.orchestra.run.smithi096.stdout: "state_name": "active+clean", 2024-05-28T06:13:52.121 INFO:teuthology.orchestra.run.smithi096.stdout: "count": 97 2024-05-28T06:13:52.121 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:13:52.121 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:13:52.121 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pgs": 97, 2024-05-28T06:13:52.121 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pools": 3, 2024-05-28T06:13:52.121 INFO:teuthology.orchestra.run.smithi096.stdout: "num_objects": 4322, 2024-05-28T06:13:52.121 INFO:teuthology.orchestra.run.smithi096.stdout: "data_bytes": 17978918560, 2024-05-28T06:13:52.121 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_used": 36366999552, 2024-05-28T06:13:52.121 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_avail": 736727113728, 2024-05-28T06:13:52.122 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_total": 773094113280, 2024-05-28T06:13:52.122 INFO:teuthology.orchestra.run.smithi096.stdout: "read_bytes_sec": 513257112, 2024-05-28T06:13:52.122 INFO:teuthology.orchestra.run.smithi096.stdout: "write_bytes_sec": 524829189, 2024-05-28T06:13:52.122 INFO:teuthology.orchestra.run.smithi096.stdout: "read_op_per_sec": 126, 2024-05-28T06:13:52.122 INFO:teuthology.orchestra.run.smithi096.stdout: "write_op_per_sec": 142 2024-05-28T06:13:52.122 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:13:52.122 INFO:teuthology.orchestra.run.smithi096.stdout: "fsmap": { 2024-05-28T06:13:52.122 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 35, 2024-05-28T06:13:52.122 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 4, 2024-05-28T06:13:52.122 INFO:teuthology.orchestra.run.smithi096.stdout: "up": 1, 2024-05-28T06:13:52.122 INFO:teuthology.orchestra.run.smithi096.stdout: "in": 1, 2024-05-28T06:13:52.122 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 1, 2024-05-28T06:13:52.123 INFO:teuthology.orchestra.run.smithi096.stdout: "by_rank": [ 2024-05-28T06:13:52.123 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:13:52.123 INFO:teuthology.orchestra.run.smithi096.stdout: "filesystem_id": 4, 2024-05-28T06:13:52.123 INFO:teuthology.orchestra.run.smithi096.stdout: "rank": 0, 2024-05-28T06:13:52.123 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "a", 2024-05-28T06:13:52.123 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "up:active", 2024-05-28T06:13:52.123 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 6489 2024-05-28T06:13:52.123 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:13:52.123 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:13:52.123 INFO:teuthology.orchestra.run.smithi096.stdout: "up:standby": 3 2024-05-28T06:13:52.124 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:13:52.124 INFO:teuthology.orchestra.run.smithi096.stdout: "mgrmap": { 2024-05-28T06:13:52.124 INFO:teuthology.orchestra.run.smithi096.stdout: "available": true, 2024-05-28T06:13:52.124 INFO:teuthology.orchestra.run.smithi096.stdout: "num_standbys": 1, 2024-05-28T06:13:52.124 INFO:teuthology.orchestra.run.smithi096.stdout: "modules": [ 2024-05-28T06:13:52.124 INFO:teuthology.orchestra.run.smithi096.stdout: "iostat", 2024-05-28T06:13:52.124 INFO:teuthology.orchestra.run.smithi096.stdout: "nfs", 2024-05-28T06:13:52.124 INFO:teuthology.orchestra.run.smithi096.stdout: "restful" 2024-05-28T06:13:52.124 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:13:52.124 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:13:52.125 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:13:52.125 INFO:teuthology.orchestra.run.smithi096.stdout: "servicemap": { 2024-05-28T06:13:52.125 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 11, 2024-05-28T06:13:52.125 INFO:teuthology.orchestra.run.smithi096.stdout: "modified": "2024-05-28T06:13:33.716527+0000", 2024-05-28T06:13:52.125 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:13:52.125 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:13:52.125 INFO:teuthology.orchestra.run.smithi096.stdout: "progress_events": { 2024-05-28T06:13:52.125 INFO:teuthology.orchestra.run.smithi096.stdout: "58c36a82-8e7f-41c9-9e90-890fce489f3b": { 2024-05-28T06:13:52.125 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "All cloning finished, waiting 25.0 seconds before exiting (25s)\n [============================] ", 2024-05-28T06:13:52.126 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": 1, 2024-05-28T06:13:52.126 INFO:teuthology.orchestra.run.smithi096.stdout: "add_to_ceph_s": true 2024-05-28T06:13:52.126 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:13:52.126 INFO:teuthology.orchestra.run.smithi096.stdout: "6a73bfaa-3b9e-443d-83e8-82fc1faa8b7f": { 2024-05-28T06:13:52.126 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "No pending clones left, waiting 25.0 seconds before exiting (19s)\n [============================] ", 2024-05-28T06:13:52.126 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": 1, 2024-05-28T06:13:52.126 INFO:teuthology.orchestra.run.smithi096.stdout: "add_to_ceph_s": true 2024-05-28T06:13:52.126 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:13:52.126 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:13:52.126 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:13:52.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:52.120+0000 7f87e3200640 1 -- 172.21.15.96:0/913412645 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f87c006d550 msgr2=0x7f87c006fa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:52.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:52.120+0000 7f87e3200640 1 --2- 172.21.15.96:0/913412645 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f87c006d550 0x7f87c006fa10 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f87e40c7270 tx=0x7f87d4008040 comp rx=0 tx=0).stop 2024-05-28T06:13:52.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:52.120+0000 7f87e3200640 1 -- 172.21.15.96:0/913412645 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f87e40d0460 msgr2=0x7f87e40c6740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:52.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:52.120+0000 7f87e3200640 1 --2- 172.21.15.96:0/913412645 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f87e40d0460 0x7f87e40c6740 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f87cc009870 tx=0x7f87cc0046c0 comp rx=0 tx=0).stop 2024-05-28T06:13:52.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:52.120+0000 7f87e3200640 1 -- 172.21.15.96:0/913412645 shutdown_connections 2024-05-28T06:13:52.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:52.120+0000 7f87e3200640 1 --2- 172.21.15.96:0/913412645 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f87c006d550 0x7f87c006fa10 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:52.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:52.120+0000 7f87e3200640 1 --2- 172.21.15.96:0/913412645 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f87e415c6b0 0x7f87e40c50c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:52.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:52.120+0000 7f87e3200640 1 --2- 172.21.15.96:0/913412645 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f87e40d0460 0x7f87e40c6740 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:52.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:52.120+0000 7f87e3200640 1 --2- 172.21.15.96:0/913412645 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87e40ccc30 0x7f87e4156dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:52.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:52.120+0000 7f87e3200640 1 -- 172.21.15.96:0/913412645 >> 172.21.15.96:0/913412645 conn(0x7f87e406bae0 msgr2=0x7f87e4072a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:13:52.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:52.120+0000 7f87e3200640 1 -- 172.21.15.96:0/913412645 shutdown_connections 2024-05-28T06:13:52.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:52.120+0000 7f87e3200640 1 -- 172.21.15.96:0/913412645 wait complete. 2024-05-28T06:13:57.140 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-05-28T06:13:57.352 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:57.351+0000 7fcbba600640 1 Processor -- start 2024-05-28T06:13:57.352 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:57.351+0000 7fcbba600640 1 -- start start 2024-05-28T06:13:57.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:57.352+0000 7fcbba600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fcbb415c6b0 0x7fcbb40b3db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:57.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:57.353+0000 7fcbba600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fcbb40ccc30 0x7fcbb406dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:57.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:57.353+0000 7fcbba600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcbb40cd1a0 con 0x7fcbb40d0460 2024-05-28T06:13:57.354 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:57.353+0000 7fcbba600640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fcbb40cd480 con 0x7fcbb40ccc30 2024-05-28T06:13:57.354 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:57.353+0000 7fcbba600640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fcbb40cd760 con 0x7fcbb415c6b0 2024-05-28T06:13:57.354 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:57.353+0000 7fcbb3e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fcbb40ccc30 0x7fcbb406dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:57.354 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:57.353+0000 7fcbb3e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fcbb40ccc30 0x7fcbb406dc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47032/0 (socket says 172.21.15.96:47032) 2024-05-28T06:13:57.354 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:57.353+0000 7fcbb3e00640 1 -- 172.21.15.96:0/3199071861 learned_addr learned my addr 172.21.15.96:0/3199071861 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:13:57.354 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:57.353+0000 7fcbb3e00640 1 -- 172.21.15.96:0/3199071861 >> v2:172.21.15.123:3300/0 conn(0x7fcbb415c6b0 msgr2=0x7fcbb40b3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:13:57.355 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:57.353+0000 7fcbb3e00640 1 --2- 172.21.15.96:0/3199071861 >> v2:172.21.15.123:3300/0 conn(0x7fcbb415c6b0 0x7fcbb40b3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:57.355 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:57.353+0000 7fcbb3e00640 1 -- 172.21.15.96:0/3199071861 >> v1:172.21.15.123:6789/0 conn(0x7fcbb40d0460 legacy=0x7fcbb40cbf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:13:57.355 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:57.353+0000 7fcbb3e00640 1 -- 172.21.15.96:0/3199071861 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbb40cda40 con 0x7fcbb40ccc30 2024-05-28T06:13:57.355 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:57.355+0000 7fcbb3e00640 1 --2- 172.21.15.96:0/3199071861 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcbb40ccc30 0x7fcbb406dc90 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fcba0009e50 tx=0x7fcba002a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c3f4634eb39842 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:13:57.355 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:57.355+0000 7fcbb3400640 1 -- 172.21.15.96:0/3199071861 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcba0036070 con 0x7fcbb40ccc30 2024-05-28T06:13:57.355 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:57.355+0000 7fcbb3400640 1 -- 172.21.15.96:0/3199071861 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcba0031600 con 0x7fcbb40ccc30 2024-05-28T06:13:57.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:57.355+0000 7fcbb3400640 1 -- 172.21.15.96:0/3199071861 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcba00319d0 con 0x7fcbb40ccc30 2024-05-28T06:13:57.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:57.355+0000 7fcbba600640 1 -- 172.21.15.96:0/3199071861 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcbb40ccc30 msgr2=0x7fcbb406dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:57.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:57.355+0000 7fcbba600640 1 --2- 172.21.15.96:0/3199071861 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcbb40ccc30 0x7fcbb406dc90 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fcba0009e50 tx=0x7fcba002a710 comp rx=0 tx=0).stop 2024-05-28T06:13:57.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:57.356+0000 7fcbba600640 1 -- 172.21.15.96:0/3199071861 shutdown_connections 2024-05-28T06:13:57.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:57.356+0000 7fcbba600640 1 --2- 172.21.15.96:0/3199071861 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcbb40ccc30 0x7fcbb406dc90 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:57.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:57.356+0000 7fcbba600640 1 --2- 172.21.15.96:0/3199071861 >> v2:172.21.15.123:3300/0 conn(0x7fcbb415c6b0 0x7fcbb40b3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:57.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:57.356+0000 7fcbba600640 1 -- 172.21.15.96:0/3199071861 >> 172.21.15.96:0/3199071861 conn(0x7fcbb406bae0 msgr2=0x7fcbb406bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:13:57.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:57.356+0000 7fcbba600640 1 -- 172.21.15.96:0/3199071861 shutdown_connections 2024-05-28T06:13:57.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:57.356+0000 7fcbba600640 1 -- 172.21.15.96:0/3199071861 wait complete. 2024-05-28T06:13:57.358 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:57.357+0000 7fcbba600640 1 Processor -- start 2024-05-28T06:13:57.358 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:57.358+0000 7fcbba600640 1 -- start start 2024-05-28T06:13:57.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:57.359+0000 7fcbba600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcbb40ccc30 0x7fcbb40c6d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:57.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:57.359+0000 7fcbba600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcbb40d0460 0x7fcbb40ae900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:57.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:57.359+0000 7fcbba600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcbb415c6b0 0x7fcbb4073580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:57.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:57.359+0000 7fcbba600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fcbb40b4470 con 0x7fcbb40d0460 2024-05-28T06:13:57.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:57.359+0000 7fcbba600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fcbb40b42f0 con 0x7fcbb415c6b0 2024-05-28T06:13:57.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:57.359+0000 7fcbba600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fcbb406e1d0 con 0x7fcbb40ccc30 2024-05-28T06:13:57.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:57.359+0000 7fcbb3e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcbb40d0460 0x7fcbb40ae900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:57.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:57.359+0000 7fcbb3e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcbb40d0460 0x7fcbb40ae900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47042/0 (socket says 172.21.15.96:47042) 2024-05-28T06:13:57.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:57.359+0000 7fcbb3e00640 1 -- 172.21.15.96:0/1734173365 learned_addr learned my addr 172.21.15.96:0/1734173365 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:13:57.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:57.359+0000 7fcbb3e00640 1 -- 172.21.15.96:0/1734173365 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcbb40ccc30 msgr2=0x7fcbb40c6d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:57.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:57.359+0000 7fcbb3e00640 1 --2- 172.21.15.96:0/1734173365 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcbb40ccc30 0x7fcbb40c6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:57.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:57.359+0000 7fcbb3e00640 1 -- 172.21.15.96:0/1734173365 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcbb415c6b0 msgr2=0x7fcbb4073580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:57.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:57.359+0000 7fcbb9c00640 1 --2- 172.21.15.96:0/1734173365 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcbb415c6b0 0x7fcbb4073580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:57.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:57.359+0000 7fcbb3e00640 1 --2- 172.21.15.96:0/1734173365 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcbb415c6b0 0x7fcbb4073580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:57.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:57.359+0000 7fcbb3e00640 1 -- 172.21.15.96:0/1734173365 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbb41f8060 con 0x7fcbb40d0460 2024-05-28T06:13:57.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:57.360+0000 7fcbb3e00640 1 --2- 172.21.15.96:0/1734173365 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcbb40d0460 0x7fcbb40ae900 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fcba0009e50 tx=0x7fcba0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:13:57.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:57.360+0000 7fcbb0c00640 1 -- 172.21.15.96:0/1734173365 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcba0036040 con 0x7fcbb40d0460 2024-05-28T06:13:57.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:57.360+0000 7fcbb0c00640 1 -- 172.21.15.96:0/1734173365 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcba0003990 con 0x7fcbb40d0460 2024-05-28T06:13:57.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:57.360+0000 7fcbba600640 1 -- 172.21.15.96:0/1734173365 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbb41f85c0 con 0x7fcbb40d0460 2024-05-28T06:13:57.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:57.360+0000 7fcbb0c00640 1 -- 172.21.15.96:0/1734173365 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcba0009510 con 0x7fcbb40d0460 2024-05-28T06:13:57.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:57.360+0000 7fcbba600640 1 -- 172.21.15.96:0/1734173365 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbb41f82e0 con 0x7fcbb40d0460 2024-05-28T06:13:57.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:57.361+0000 7fcbba600640 1 -- 172.21.15.96:0/1734173365 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcbb40689e0 con 0x7fcbb40d0460 2024-05-28T06:13:57.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:57.362+0000 7fcbb0c00640 1 -- 172.21.15.96:0/1734173365 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 21) ==== 92111+0+0 (secure 0 0 0) 0x7fcba0007820 con 0x7fcbb40d0460 2024-05-28T06:13:57.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:57.363+0000 7fcbb0c00640 1 --2- 172.21.15.96:0/1734173365 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fcb9406d4c0 0x7fcb9406f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:13:57.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:57.364+0000 7fcbb0c00640 1 -- 172.21.15.96:0/1734173365 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5737+0+0 (secure 0 0 0) 0x7fcba00abb10 con 0x7fcbb40d0460 2024-05-28T06:13:57.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:57.364+0000 7fcbb9200640 1 --2- 172.21.15.96:0/1734173365 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fcb9406d4c0 0x7fcb9406f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:13:57.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:57.365+0000 7fcbb9200640 1 --2- 172.21.15.96:0/1734173365 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fcb9406d4c0 0x7fcb9406f980 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fcba4009710 tx=0x7fcba4006c90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:13:57.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:57.369+0000 7fcbb0c00640 1 -- 172.21.15.96:0/1734173365 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fcbb41f8060 con 0x7fcbb40d0460 2024-05-28T06:13:57.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:57.645+0000 7fcbba600640 1 -- 172.21.15.96:0/1734173365 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fcbb40c63e0 con 0x7fcbb40d0460 2024-05-28T06:13:57.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:57.646+0000 7fcbb0c00640 1 -- 172.21.15.96:0/1734173365 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+2377 (secure 0 0 0) 0x7fcba007b890 con 0x7fcbb40d0460 2024-05-28T06:13:57.647 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:13:57.647 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:13:57.647 INFO:teuthology.orchestra.run.smithi096.stdout: "fsid": "e4a831b5-e6d1-47a7-96cb-bf85154ee0d2", 2024-05-28T06:13:57.647 INFO:teuthology.orchestra.run.smithi096.stdout: "health": { 2024-05-28T06:13:57.647 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "HEALTH_OK", 2024-05-28T06:13:57.647 INFO:teuthology.orchestra.run.smithi096.stdout: "checks": {}, 2024-05-28T06:13:57.647 INFO:teuthology.orchestra.run.smithi096.stdout: "mutes": [] 2024-05-28T06:13:57.648 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:13:57.648 INFO:teuthology.orchestra.run.smithi096.stdout: "election_epoch": 8, 2024-05-28T06:13:57.648 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum": [ 2024-05-28T06:13:57.648 INFO:teuthology.orchestra.run.smithi096.stdout: 0, 2024-05-28T06:13:57.648 INFO:teuthology.orchestra.run.smithi096.stdout: 1, 2024-05-28T06:13:57.648 INFO:teuthology.orchestra.run.smithi096.stdout: 2 2024-05-28T06:13:57.648 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:13:57.648 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_names": [ 2024-05-28T06:13:57.648 INFO:teuthology.orchestra.run.smithi096.stdout: "a", 2024-05-28T06:13:57.648 INFO:teuthology.orchestra.run.smithi096.stdout: "b", 2024-05-28T06:13:57.648 INFO:teuthology.orchestra.run.smithi096.stdout: "c" 2024-05-28T06:13:57.648 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:13:57.648 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_age": 611, 2024-05-28T06:13:57.649 INFO:teuthology.orchestra.run.smithi096.stdout: "monmap": { 2024-05-28T06:13:57.649 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 1, 2024-05-28T06:13:57.649 INFO:teuthology.orchestra.run.smithi096.stdout: "min_mon_release_name": "squid", 2024-05-28T06:13:57.649 INFO:teuthology.orchestra.run.smithi096.stdout: "num_mons": 3 2024-05-28T06:13:57.649 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:13:57.649 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap": { 2024-05-28T06:13:57.649 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 60, 2024-05-28T06:13:57.649 INFO:teuthology.orchestra.run.smithi096.stdout: "num_osds": 8, 2024-05-28T06:13:57.649 INFO:teuthology.orchestra.run.smithi096.stdout: "num_up_osds": 8, 2024-05-28T06:13:57.649 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_up_since": 1716876238, 2024-05-28T06:13:57.649 INFO:teuthology.orchestra.run.smithi096.stdout: "num_in_osds": 8, 2024-05-28T06:13:57.649 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_in_since": 1716876229, 2024-05-28T06:13:57.649 INFO:teuthology.orchestra.run.smithi096.stdout: "num_remapped_pgs": 0 2024-05-28T06:13:57.649 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:13:57.650 INFO:teuthology.orchestra.run.smithi096.stdout: "pgmap": { 2024-05-28T06:13:57.650 INFO:teuthology.orchestra.run.smithi096.stdout: "pgs_by_state": [ 2024-05-28T06:13:57.650 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:13:57.650 INFO:teuthology.orchestra.run.smithi096.stdout: "state_name": "active+clean", 2024-05-28T06:13:57.650 INFO:teuthology.orchestra.run.smithi096.stdout: "count": 97 2024-05-28T06:13:57.650 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:13:57.650 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:13:57.650 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pgs": 97, 2024-05-28T06:13:57.650 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pools": 3, 2024-05-28T06:13:57.650 INFO:teuthology.orchestra.run.smithi096.stdout: "num_objects": 4321, 2024-05-28T06:13:57.650 INFO:teuthology.orchestra.run.smithi096.stdout: "data_bytes": 17978952492, 2024-05-28T06:13:57.650 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_used": 36367056896, 2024-05-28T06:13:57.650 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_avail": 736727056384, 2024-05-28T06:13:57.651 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_total": 773094113280, 2024-05-28T06:13:57.651 INFO:teuthology.orchestra.run.smithi096.stdout: "read_bytes_sec": 340313559, 2024-05-28T06:13:57.651 INFO:teuthology.orchestra.run.smithi096.stdout: "write_bytes_sec": 340358171, 2024-05-28T06:13:57.651 INFO:teuthology.orchestra.run.smithi096.stdout: "read_op_per_sec": 84, 2024-05-28T06:13:57.651 INFO:teuthology.orchestra.run.smithi096.stdout: "write_op_per_sec": 97 2024-05-28T06:13:57.651 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:13:57.651 INFO:teuthology.orchestra.run.smithi096.stdout: "fsmap": { 2024-05-28T06:13:57.651 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 35, 2024-05-28T06:13:57.651 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 4, 2024-05-28T06:13:57.651 INFO:teuthology.orchestra.run.smithi096.stdout: "up": 1, 2024-05-28T06:13:57.651 INFO:teuthology.orchestra.run.smithi096.stdout: "in": 1, 2024-05-28T06:13:57.651 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 1, 2024-05-28T06:13:57.651 INFO:teuthology.orchestra.run.smithi096.stdout: "by_rank": [ 2024-05-28T06:13:57.652 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:13:57.652 INFO:teuthology.orchestra.run.smithi096.stdout: "filesystem_id": 4, 2024-05-28T06:13:57.652 INFO:teuthology.orchestra.run.smithi096.stdout: "rank": 0, 2024-05-28T06:13:57.652 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "a", 2024-05-28T06:13:57.652 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "up:active", 2024-05-28T06:13:57.652 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 6489 2024-05-28T06:13:57.652 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:13:57.652 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:13:57.652 INFO:teuthology.orchestra.run.smithi096.stdout: "up:standby": 3 2024-05-28T06:13:57.652 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:13:57.652 INFO:teuthology.orchestra.run.smithi096.stdout: "mgrmap": { 2024-05-28T06:13:57.652 INFO:teuthology.orchestra.run.smithi096.stdout: "available": true, 2024-05-28T06:13:57.652 INFO:teuthology.orchestra.run.smithi096.stdout: "num_standbys": 1, 2024-05-28T06:13:57.652 INFO:teuthology.orchestra.run.smithi096.stdout: "modules": [ 2024-05-28T06:13:57.653 INFO:teuthology.orchestra.run.smithi096.stdout: "iostat", 2024-05-28T06:13:57.653 INFO:teuthology.orchestra.run.smithi096.stdout: "nfs", 2024-05-28T06:13:57.653 INFO:teuthology.orchestra.run.smithi096.stdout: "restful" 2024-05-28T06:13:57.653 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:13:57.653 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:13:57.653 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:13:57.653 INFO:teuthology.orchestra.run.smithi096.stdout: "servicemap": { 2024-05-28T06:13:57.653 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 11, 2024-05-28T06:13:57.653 INFO:teuthology.orchestra.run.smithi096.stdout: "modified": "2024-05-28T06:13:33.716527+0000", 2024-05-28T06:13:57.653 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:13:57.653 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:13:57.653 INFO:teuthology.orchestra.run.smithi096.stdout: "progress_events": { 2024-05-28T06:13:57.653 INFO:teuthology.orchestra.run.smithi096.stdout: "58c36a82-8e7f-41c9-9e90-890fce489f3b": { 2024-05-28T06:13:57.653 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "All cloning finished, waiting 21.0 seconds before exiting (29s)\n [============================] ", 2024-05-28T06:13:57.654 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": 1, 2024-05-28T06:13:57.654 INFO:teuthology.orchestra.run.smithi096.stdout: "add_to_ceph_s": true 2024-05-28T06:13:57.654 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:13:57.654 INFO:teuthology.orchestra.run.smithi096.stdout: "6a73bfaa-3b9e-443d-83e8-82fc1faa8b7f": { 2024-05-28T06:13:57.654 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "No pending clones left, waiting 21.0 seconds before exiting (23s)\n [============================] ", 2024-05-28T06:13:57.654 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": 1, 2024-05-28T06:13:57.654 INFO:teuthology.orchestra.run.smithi096.stdout: "add_to_ceph_s": true 2024-05-28T06:13:57.654 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:13:57.654 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:13:57.654 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:13:57.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:57.649+0000 7fcbba600640 1 -- 172.21.15.96:0/1734173365 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fcb9406d4c0 msgr2=0x7fcb9406f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:57.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:57.649+0000 7fcbba600640 1 --2- 172.21.15.96:0/1734173365 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fcb9406d4c0 0x7fcb9406f980 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fcba4009710 tx=0x7fcba4006c90 comp rx=0 tx=0).stop 2024-05-28T06:13:57.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:57.649+0000 7fcbba600640 1 -- 172.21.15.96:0/1734173365 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcbb40d0460 msgr2=0x7fcbb40ae900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:13:57.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:57.649+0000 7fcbba600640 1 --2- 172.21.15.96:0/1734173365 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcbb40d0460 0x7fcbb40ae900 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fcba0009e50 tx=0x7fcba0004a40 comp rx=0 tx=0).stop 2024-05-28T06:13:57.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:57.649+0000 7fcbba600640 1 -- 172.21.15.96:0/1734173365 shutdown_connections 2024-05-28T06:13:57.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:57.649+0000 7fcbba600640 1 --2- 172.21.15.96:0/1734173365 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fcb9406d4c0 0x7fcb9406f980 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:57.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:57.649+0000 7fcbba600640 1 --2- 172.21.15.96:0/1734173365 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcbb415c6b0 0x7fcbb4073580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:57.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:57.649+0000 7fcbba600640 1 --2- 172.21.15.96:0/1734173365 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcbb40d0460 0x7fcbb40ae900 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:57.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:57.649+0000 7fcbba600640 1 --2- 172.21.15.96:0/1734173365 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcbb40ccc30 0x7fcbb40c6d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:13:57.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:57.649+0000 7fcbba600640 1 -- 172.21.15.96:0/1734173365 >> 172.21.15.96:0/1734173365 conn(0x7fcbb406bae0 msgr2=0x7fcbb406bec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:13:57.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:57.649+0000 7fcbba600640 1 -- 172.21.15.96:0/1734173365 shutdown_connections 2024-05-28T06:13:57.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:13:57.649+0000 7fcbba600640 1 -- 172.21.15.96:0/1734173365 wait complete. 2024-05-28T06:14:01.607 DEBUG:teuthology.orchestra.run.smithi096:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:14:01.615 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:14:02.671 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-05-28T06:14:02.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:02.880+0000 7fd298a00640 1 Processor -- start 2024-05-28T06:14:02.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:02.881+0000 7fd298a00640 1 -- start start 2024-05-28T06:14:02.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:02.882+0000 7fd298a00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fd29415e690 0x7fd2940d4530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:02.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:02.882+0000 7fd298a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fd2940d1ca0 0x7fd2940d4a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:02.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:02.882+0000 7fd298a00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2940d3060 con 0x7fd29415ea70 2024-05-28T06:14:02.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:02.882+0000 7fd298a00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fd2940d3340 con 0x7fd2940d1ca0 2024-05-28T06:14:02.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:02.882+0000 7fd298a00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fd2940d3620 con 0x7fd29415e690 2024-05-28T06:14:02.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:02.882+0000 7fd291e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fd2940d1ca0 0x7fd2940d4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:02.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:02.882+0000 7fd291e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fd2940d1ca0 0x7fd2940d4a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:38100/0 (socket says 172.21.15.96:38100) 2024-05-28T06:14:02.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:02.882+0000 7fd291e00640 1 -- 172.21.15.96:0/2876234583 learned_addr learned my addr 172.21.15.96:0/2876234583 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:02.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:02.883+0000 7fd291e00640 1 -- 172.21.15.96:0/2876234583 >> v2:172.21.15.123:3300/0 conn(0x7fd29415e690 msgr2=0x7fd2940d4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:02.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:02.883+0000 7fd291e00640 1 --2- 172.21.15.96:0/2876234583 >> v2:172.21.15.123:3300/0 conn(0x7fd29415e690 0x7fd2940d4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:02.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:02.883+0000 7fd291e00640 1 -- 172.21.15.96:0/2876234583 >> v1:172.21.15.123:6789/0 conn(0x7fd29415ea70 legacy=0x7fd2940d2330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:02.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:02.883+0000 7fd291e00640 1 -- 172.21.15.96:0/2876234583 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2940cf770 con 0x7fd2940d1ca0 2024-05-28T06:14:02.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:02.884+0000 7fd291e00640 1 --2- 172.21.15.96:0/2876234583 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd2940d1ca0 0x7fd2940d4a70 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fd284009ea0 tx=0x7fd28402a7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2db69ebf2a065266 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:02.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:02.884+0000 7fd291400640 1 -- 172.21.15.96:0/2876234583 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd284036070 con 0x7fd2940d1ca0 2024-05-28T06:14:02.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:02.884+0000 7fd291400640 1 -- 172.21.15.96:0/2876234583 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd284031600 con 0x7fd2940d1ca0 2024-05-28T06:14:02.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:02.884+0000 7fd291400640 1 -- 172.21.15.96:0/2876234583 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2840319d0 con 0x7fd2940d1ca0 2024-05-28T06:14:02.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:02.884+0000 7fd298a00640 1 -- 172.21.15.96:0/2876234583 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd2940d1ca0 msgr2=0x7fd2940d4a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:02.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:02.884+0000 7fd298a00640 1 --2- 172.21.15.96:0/2876234583 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd2940d1ca0 0x7fd2940d4a70 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fd284009ea0 tx=0x7fd28402a7b0 comp rx=0 tx=0).stop 2024-05-28T06:14:02.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:02.885+0000 7fd298a00640 1 -- 172.21.15.96:0/2876234583 shutdown_connections 2024-05-28T06:14:02.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:02.885+0000 7fd298a00640 1 --2- 172.21.15.96:0/2876234583 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd2940d1ca0 0x7fd2940d4a70 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:02.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:02.885+0000 7fd298a00640 1 --2- 172.21.15.96:0/2876234583 >> v2:172.21.15.123:3300/0 conn(0x7fd29415e690 0x7fd2940d4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:02.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:02.885+0000 7fd298a00640 1 -- 172.21.15.96:0/2876234583 >> 172.21.15.96:0/2876234583 conn(0x7fd29409d910 msgr2=0x7fd29408bfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:02.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:02.885+0000 7fd298a00640 1 -- 172.21.15.96:0/2876234583 shutdown_connections 2024-05-28T06:14:02.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:02.885+0000 7fd298a00640 1 -- 172.21.15.96:0/2876234583 wait complete. 2024-05-28T06:14:02.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:02.886+0000 7fd298a00640 1 Processor -- start 2024-05-28T06:14:02.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:02.887+0000 7fd298a00640 1 -- start start 2024-05-28T06:14:02.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:02.887+0000 7fd298a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2940d1ca0 0x7fd2940cd0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:02.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:02.887+0000 7fd298a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd29415e690 0x7fd2940cbc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:02.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:02.887+0000 7fd298a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd29415ea70 0x7fd2940cc180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:02.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:02.887+0000 7fd298a00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd2940d2bc0 con 0x7fd29415e690 2024-05-28T06:14:02.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:02.887+0000 7fd298a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd2940d2a40 con 0x7fd2940d1ca0 2024-05-28T06:14:02.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:02.887+0000 7fd298a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fd2940d2d40 con 0x7fd29415ea70 2024-05-28T06:14:02.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:02.888+0000 7fd291e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd29415e690 0x7fd2940cbc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:02.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:02.888+0000 7fd291e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd29415e690 0x7fd2940cbc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:38112/0 (socket says 172.21.15.96:38112) 2024-05-28T06:14:02.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:02.888+0000 7fd291e00640 1 -- 172.21.15.96:0/3284353062 learned_addr learned my addr 172.21.15.96:0/3284353062 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:02.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:02.888+0000 7fd291e00640 1 -- 172.21.15.96:0/3284353062 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd29415ea70 msgr2=0x7fd2940cc180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:02.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:02.888+0000 7fd293200640 1 --2- 172.21.15.96:0/3284353062 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd29415ea70 0x7fd2940cc180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:02.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:02.888+0000 7fd292800640 1 --2- 172.21.15.96:0/3284353062 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2940d1ca0 0x7fd2940cd0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:02.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:02.888+0000 7fd291e00640 1 --2- 172.21.15.96:0/3284353062 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd29415ea70 0x7fd2940cc180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:02.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:02.888+0000 7fd291e00640 1 -- 172.21.15.96:0/3284353062 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2940d1ca0 msgr2=0x7fd2940cd0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:02.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:02.888+0000 7fd291e00640 1 --2- 172.21.15.96:0/3284353062 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2940d1ca0 0x7fd2940cd0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:02.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:02.888+0000 7fd291e00640 1 -- 172.21.15.96:0/3284353062 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2940ba3b0 con 0x7fd29415e690 2024-05-28T06:14:02.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:02.888+0000 7fd293200640 1 --2- 172.21.15.96:0/3284353062 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd29415ea70 0x7fd2940cc180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:14:02.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:02.889+0000 7fd291e00640 1 --2- 172.21.15.96:0/3284353062 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd29415e690 0x7fd2940cbc40 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fd284009ea0 tx=0x7fd284004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:02.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:02.889+0000 7fd28aa00640 1 -- 172.21.15.96:0/3284353062 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd284036040 con 0x7fd29415e690 2024-05-28T06:14:02.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:02.889+0000 7fd28aa00640 1 -- 172.21.15.96:0/3284353062 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd284003990 con 0x7fd29415e690 2024-05-28T06:14:02.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:02.889+0000 7fd298a00640 1 -- 172.21.15.96:0/3284353062 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd29415f570 con 0x7fd29415e690 2024-05-28T06:14:02.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:02.889+0000 7fd28aa00640 1 -- 172.21.15.96:0/3284353062 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd284009510 con 0x7fd29415e690 2024-05-28T06:14:02.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:02.889+0000 7fd298a00640 1 -- 172.21.15.96:0/3284353062 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd29415fb30 con 0x7fd29415e690 2024-05-28T06:14:02.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:02.890+0000 7fd298a00640 1 -- 172.21.15.96:0/3284353062 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd29409dde0 con 0x7fd29415e690 2024-05-28T06:14:02.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:02.891+0000 7fd28aa00640 1 -- 172.21.15.96:0/3284353062 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 21) ==== 92111+0+0 (secure 0 0 0) 0x7fd284007820 con 0x7fd29415e690 2024-05-28T06:14:02.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:02.892+0000 7fd28aa00640 1 --2- 172.21.15.96:0/3284353062 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd27006d4f0 0x7fd27006f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:02.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:02.892+0000 7fd28aa00640 1 -- 172.21.15.96:0/3284353062 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5737+0+0 (secure 0 0 0) 0x7fd2840abb90 con 0x7fd29415e690 2024-05-28T06:14:02.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:02.893+0000 7fd292800640 1 --2- 172.21.15.96:0/3284353062 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd27006d4f0 0x7fd27006f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:02.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:02.894+0000 7fd292800640 1 --2- 172.21.15.96:0/3284353062 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd27006d4f0 0x7fd27006f9b0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fd27c00a770 tx=0x7fd27c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:02.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:02.897+0000 7fd28aa00640 1 -- 172.21.15.96:0/3284353062 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd29409dde0 con 0x7fd29415e690 2024-05-28T06:14:03.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:03.171+0000 7fd298a00640 1 -- 172.21.15.96:0/3284353062 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fd2940689e0 con 0x7fd29415e690 2024-05-28T06:14:03.174 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:03.172+0000 7fd28aa00640 1 -- 172.21.15.96:0/3284353062 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+2373 (secure 0 0 0) 0x7fd28407b7b0 con 0x7fd29415e690 2024-05-28T06:14:03.174 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:14:03.174 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:14:03.174 INFO:teuthology.orchestra.run.smithi096.stdout: "fsid": "e4a831b5-e6d1-47a7-96cb-bf85154ee0d2", 2024-05-28T06:14:03.174 INFO:teuthology.orchestra.run.smithi096.stdout: "health": { 2024-05-28T06:14:03.174 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "HEALTH_OK", 2024-05-28T06:14:03.175 INFO:teuthology.orchestra.run.smithi096.stdout: "checks": {}, 2024-05-28T06:14:03.175 INFO:teuthology.orchestra.run.smithi096.stdout: "mutes": [] 2024-05-28T06:14:03.175 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:14:03.175 INFO:teuthology.orchestra.run.smithi096.stdout: "election_epoch": 8, 2024-05-28T06:14:03.175 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum": [ 2024-05-28T06:14:03.175 INFO:teuthology.orchestra.run.smithi096.stdout: 0, 2024-05-28T06:14:03.175 INFO:teuthology.orchestra.run.smithi096.stdout: 1, 2024-05-28T06:14:03.175 INFO:teuthology.orchestra.run.smithi096.stdout: 2 2024-05-28T06:14:03.175 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:14:03.175 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_names": [ 2024-05-28T06:14:03.175 INFO:teuthology.orchestra.run.smithi096.stdout: "a", 2024-05-28T06:14:03.175 INFO:teuthology.orchestra.run.smithi096.stdout: "b", 2024-05-28T06:14:03.175 INFO:teuthology.orchestra.run.smithi096.stdout: "c" 2024-05-28T06:14:03.175 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:14:03.176 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_age": 616, 2024-05-28T06:14:03.176 INFO:teuthology.orchestra.run.smithi096.stdout: "monmap": { 2024-05-28T06:14:03.176 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 1, 2024-05-28T06:14:03.176 INFO:teuthology.orchestra.run.smithi096.stdout: "min_mon_release_name": "squid", 2024-05-28T06:14:03.176 INFO:teuthology.orchestra.run.smithi096.stdout: "num_mons": 3 2024-05-28T06:14:03.176 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:14:03.176 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap": { 2024-05-28T06:14:03.176 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 60, 2024-05-28T06:14:03.176 INFO:teuthology.orchestra.run.smithi096.stdout: "num_osds": 8, 2024-05-28T06:14:03.176 INFO:teuthology.orchestra.run.smithi096.stdout: "num_up_osds": 8, 2024-05-28T06:14:03.176 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_up_since": 1716876238, 2024-05-28T06:14:03.176 INFO:teuthology.orchestra.run.smithi096.stdout: "num_in_osds": 8, 2024-05-28T06:14:03.176 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_in_since": 1716876229, 2024-05-28T06:14:03.177 INFO:teuthology.orchestra.run.smithi096.stdout: "num_remapped_pgs": 0 2024-05-28T06:14:03.177 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:14:03.177 INFO:teuthology.orchestra.run.smithi096.stdout: "pgmap": { 2024-05-28T06:14:03.177 INFO:teuthology.orchestra.run.smithi096.stdout: "pgs_by_state": [ 2024-05-28T06:14:03.177 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:14:03.177 INFO:teuthology.orchestra.run.smithi096.stdout: "state_name": "active+clean", 2024-05-28T06:14:03.177 INFO:teuthology.orchestra.run.smithi096.stdout: "count": 97 2024-05-28T06:14:03.177 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:14:03.177 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:14:03.177 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pgs": 97, 2024-05-28T06:14:03.177 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pools": 3, 2024-05-28T06:14:03.177 INFO:teuthology.orchestra.run.smithi096.stdout: "num_objects": 4321, 2024-05-28T06:14:03.177 INFO:teuthology.orchestra.run.smithi096.stdout: "data_bytes": 17978963346, 2024-05-28T06:14:03.177 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_used": 36367081472, 2024-05-28T06:14:03.178 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_avail": 736727031808, 2024-05-28T06:14:03.178 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_total": 773094113280, 2024-05-28T06:14:03.178 INFO:teuthology.orchestra.run.smithi096.stdout: "read_bytes_sec": 15372884, 2024-05-28T06:14:03.178 INFO:teuthology.orchestra.run.smithi096.stdout: "write_bytes_sec": 11534333, 2024-05-28T06:14:03.178 INFO:teuthology.orchestra.run.smithi096.stdout: "read_op_per_sec": 3, 2024-05-28T06:14:03.178 INFO:teuthology.orchestra.run.smithi096.stdout: "write_op_per_sec": 5 2024-05-28T06:14:03.178 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:14:03.178 INFO:teuthology.orchestra.run.smithi096.stdout: "fsmap": { 2024-05-28T06:14:03.178 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 35, 2024-05-28T06:14:03.178 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 4, 2024-05-28T06:14:03.178 INFO:teuthology.orchestra.run.smithi096.stdout: "up": 1, 2024-05-28T06:14:03.178 INFO:teuthology.orchestra.run.smithi096.stdout: "in": 1, 2024-05-28T06:14:03.179 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 1, 2024-05-28T06:14:03.179 INFO:teuthology.orchestra.run.smithi096.stdout: "by_rank": [ 2024-05-28T06:14:03.179 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:14:03.179 INFO:teuthology.orchestra.run.smithi096.stdout: "filesystem_id": 4, 2024-05-28T06:14:03.179 INFO:teuthology.orchestra.run.smithi096.stdout: "rank": 0, 2024-05-28T06:14:03.179 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "a", 2024-05-28T06:14:03.179 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "up:active", 2024-05-28T06:14:03.179 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 6489 2024-05-28T06:14:03.179 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:14:03.179 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:14:03.179 INFO:teuthology.orchestra.run.smithi096.stdout: "up:standby": 3 2024-05-28T06:14:03.179 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:14:03.179 INFO:teuthology.orchestra.run.smithi096.stdout: "mgrmap": { 2024-05-28T06:14:03.179 INFO:teuthology.orchestra.run.smithi096.stdout: "available": true, 2024-05-28T06:14:03.179 INFO:teuthology.orchestra.run.smithi096.stdout: "num_standbys": 1, 2024-05-28T06:14:03.180 INFO:teuthology.orchestra.run.smithi096.stdout: "modules": [ 2024-05-28T06:14:03.180 INFO:teuthology.orchestra.run.smithi096.stdout: "iostat", 2024-05-28T06:14:03.180 INFO:teuthology.orchestra.run.smithi096.stdout: "nfs", 2024-05-28T06:14:03.180 INFO:teuthology.orchestra.run.smithi096.stdout: "restful" 2024-05-28T06:14:03.180 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:14:03.180 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:14:03.180 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:14:03.180 INFO:teuthology.orchestra.run.smithi096.stdout: "servicemap": { 2024-05-28T06:14:03.180 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 11, 2024-05-28T06:14:03.180 INFO:teuthology.orchestra.run.smithi096.stdout: "modified": "2024-05-28T06:13:33.716527+0000", 2024-05-28T06:14:03.180 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:14:03.180 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:14:03.180 INFO:teuthology.orchestra.run.smithi096.stdout: "progress_events": { 2024-05-28T06:14:03.180 INFO:teuthology.orchestra.run.smithi096.stdout: "58c36a82-8e7f-41c9-9e90-890fce489f3b": { 2024-05-28T06:14:03.180 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "All cloning finished, waiting 15.0 seconds before exiting (35s)\n [============================] ", 2024-05-28T06:14:03.181 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": 1, 2024-05-28T06:14:03.181 INFO:teuthology.orchestra.run.smithi096.stdout: "add_to_ceph_s": true 2024-05-28T06:14:03.181 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:14:03.181 INFO:teuthology.orchestra.run.smithi096.stdout: "6a73bfaa-3b9e-443d-83e8-82fc1faa8b7f": { 2024-05-28T06:14:03.181 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "No pending clones left, waiting 15.0 seconds before exiting (29s)\n [============================] ", 2024-05-28T06:14:03.181 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": 1, 2024-05-28T06:14:03.181 INFO:teuthology.orchestra.run.smithi096.stdout: "add_to_ceph_s": true 2024-05-28T06:14:03.181 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:14:03.181 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:14:03.181 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:14:03.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:03.175+0000 7fd298a00640 1 -- 172.21.15.96:0/3284353062 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd27006d4f0 msgr2=0x7fd27006f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:03.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:03.175+0000 7fd298a00640 1 --2- 172.21.15.96:0/3284353062 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd27006d4f0 0x7fd27006f9b0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fd27c00a770 tx=0x7fd27c008040 comp rx=0 tx=0).stop 2024-05-28T06:14:03.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:03.175+0000 7fd298a00640 1 -- 172.21.15.96:0/3284353062 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd29415e690 msgr2=0x7fd2940cbc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:03.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:03.175+0000 7fd298a00640 1 --2- 172.21.15.96:0/3284353062 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd29415e690 0x7fd2940cbc40 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fd284009ea0 tx=0x7fd284004a40 comp rx=0 tx=0).stop 2024-05-28T06:14:03.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:03.175+0000 7fd298a00640 1 -- 172.21.15.96:0/3284353062 shutdown_connections 2024-05-28T06:14:03.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:03.175+0000 7fd298a00640 1 --2- 172.21.15.96:0/3284353062 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd27006d4f0 0x7fd27006f9b0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:03.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:03.175+0000 7fd298a00640 1 --2- 172.21.15.96:0/3284353062 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd29415ea70 0x7fd2940cc180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:03.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:03.175+0000 7fd298a00640 1 --2- 172.21.15.96:0/3284353062 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd29415e690 0x7fd2940cbc40 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:03.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:03.175+0000 7fd298a00640 1 --2- 172.21.15.96:0/3284353062 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2940d1ca0 0x7fd2940cd0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:03.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:03.175+0000 7fd298a00640 1 -- 172.21.15.96:0/3284353062 >> 172.21.15.96:0/3284353062 conn(0x7fd29409d910 msgr2=0x7fd2940b44a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:03.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:03.175+0000 7fd298a00640 1 -- 172.21.15.96:0/3284353062 shutdown_connections 2024-05-28T06:14:03.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:03.175+0000 7fd298a00640 1 -- 172.21.15.96:0/3284353062 wait complete. 2024-05-28T06:14:08.197 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-05-28T06:14:08.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:08.406+0000 7f5c00a00640 1 Processor -- start 2024-05-28T06:14:08.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:08.406+0000 7f5c00a00640 1 -- start start 2024-05-28T06:14:08.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:08.408+0000 7f5c00a00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f5bfc0723d0 0x7f5bfc0b3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:08.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:08.408+0000 7f5c00a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f5bfc0b41e0 0x7f5bfc0cf880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:08.410 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:08.408+0000 7f5c00a00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5bfc0cf0c0 con 0x7f5bfc145320 2024-05-28T06:14:08.410 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:08.408+0000 7f5c00a00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f5bfc0b4a20 con 0x7f5bfc0b41e0 2024-05-28T06:14:08.410 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:08.408+0000 7f5c00a00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f5bfc0b4d00 con 0x7f5bfc0723d0 2024-05-28T06:14:08.410 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:08.408+0000 7f5bfa800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f5bfc0b41e0 0x7f5bfc0cf880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:08.410 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:08.408+0000 7f5bfa800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f5bfc0b41e0 0x7f5bfc0cf880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:38122/0 (socket says 172.21.15.96:38122) 2024-05-28T06:14:08.410 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:08.408+0000 7f5bfa800640 1 -- 172.21.15.96:0/3516716039 learned_addr learned my addr 172.21.15.96:0/3516716039 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:08.410 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:08.408+0000 7f5bf3e00640 1 --2- 172.21.15.96:0/3516716039 >> v2:172.21.15.123:3300/0 conn(0x7f5bfc0723d0 0x7f5bfc0b3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:08.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:08.409+0000 7f5bfa800640 1 -- 172.21.15.96:0/3516716039 >> v2:172.21.15.123:3300/0 conn(0x7f5bfc0723d0 msgr2=0x7f5bfc0b3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:08.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:08.409+0000 7f5bfa800640 1 --2- 172.21.15.96:0/3516716039 >> v2:172.21.15.123:3300/0 conn(0x7f5bfc0723d0 0x7f5bfc0b3ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:08.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:08.409+0000 7f5bfa800640 1 -- 172.21.15.96:0/3516716039 >> v1:172.21.15.123:6789/0 conn(0x7f5bfc145320 legacy=0x7f5bfc071cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:08.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:08.409+0000 7f5bfa800640 1 -- 172.21.15.96:0/3516716039 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bfc0b4fe0 con 0x7f5bfc0b41e0 2024-05-28T06:14:08.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:08.410+0000 7f5bfa800640 1 --2- 172.21.15.96:0/3516716039 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5bfc0b41e0 0x7f5bfc0cf880 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f5bfc0ce950 tx=0x7f5bf402a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3becfc8797f6534 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:08.412 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:08.410+0000 7f5bf9400640 1 -- 172.21.15.96:0/3516716039 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bf4036070 con 0x7f5bfc0b41e0 2024-05-28T06:14:08.412 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:08.410+0000 7f5bf9400640 1 -- 172.21.15.96:0/3516716039 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5bf4031600 con 0x7f5bfc0b41e0 2024-05-28T06:14:08.412 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:08.410+0000 7f5bf9400640 1 -- 172.21.15.96:0/3516716039 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bf40319d0 con 0x7f5bfc0b41e0 2024-05-28T06:14:08.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:08.411+0000 7f5c00a00640 1 -- 172.21.15.96:0/3516716039 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5bfc0b41e0 msgr2=0x7f5bfc0cf880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:08.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:08.411+0000 7f5c00a00640 1 --2- 172.21.15.96:0/3516716039 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5bfc0b41e0 0x7f5bfc0cf880 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f5bfc0ce950 tx=0x7f5bf402a6c0 comp rx=0 tx=0).stop 2024-05-28T06:14:08.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:08.411+0000 7f5c00a00640 1 -- 172.21.15.96:0/3516716039 shutdown_connections 2024-05-28T06:14:08.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:08.411+0000 7f5c00a00640 1 --2- 172.21.15.96:0/3516716039 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5bfc0b41e0 0x7f5bfc0cf880 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:08.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:08.411+0000 7f5c00a00640 1 --2- 172.21.15.96:0/3516716039 >> v2:172.21.15.123:3300/0 conn(0x7f5bfc0723d0 0x7f5bfc0b3ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:08.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:08.411+0000 7f5c00a00640 1 -- 172.21.15.96:0/3516716039 >> 172.21.15.96:0/3516716039 conn(0x7f5bfc067ce0 msgr2=0x7f5bfc06b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:08.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:08.411+0000 7f5c00a00640 1 -- 172.21.15.96:0/3516716039 shutdown_connections 2024-05-28T06:14:08.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:08.411+0000 7f5c00a00640 1 -- 172.21.15.96:0/3516716039 wait complete. 2024-05-28T06:14:08.414 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:08.413+0000 7f5c00a00640 1 Processor -- start 2024-05-28T06:14:08.414 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:08.413+0000 7f5c00a00640 1 -- start start 2024-05-28T06:14:08.415 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:08.414+0000 7f5c00a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5bfc0723d0 0x7f5bfc15d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:08.416 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:08.414+0000 7f5c00a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5bfc0b41e0 0x7f5bfc0b1070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:08.416 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:08.414+0000 7f5c00a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5bfc145320 0x7f5bfc0b15b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:08.416 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:08.414+0000 7f5c00a00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f5bfc0b4650 con 0x7f5bfc0723d0 2024-05-28T06:14:08.416 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:08.414+0000 7f5c00a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5bfc145700 con 0x7f5bfc145320 2024-05-28T06:14:08.416 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:08.414+0000 7f5c00a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f5bfc0b47d0 con 0x7f5bfc0b41e0 2024-05-28T06:14:08.416 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:08.414+0000 7f5bf9e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5bfc0723d0 0x7f5bfc15d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:08.416 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:08.414+0000 7f5bf9e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5bfc0723d0 0x7f5bfc15d5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:38130/0 (socket says 172.21.15.96:38130) 2024-05-28T06:14:08.416 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:08.414+0000 7f5bf9e00640 1 -- 172.21.15.96:0/1347225760 learned_addr learned my addr 172.21.15.96:0/1347225760 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:08.417 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:08.414+0000 7f5bf9e00640 1 -- 172.21.15.96:0/1347225760 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5bfc0b41e0 msgr2=0x7f5bfc0b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:08.417 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:08.414+0000 7f5bf9e00640 1 --2- 172.21.15.96:0/1347225760 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5bfc0b41e0 0x7f5bfc0b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:08.417 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:08.414+0000 7f5bfa800640 1 --2- 172.21.15.96:0/1347225760 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5bfc145320 0x7f5bfc0b15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:08.417 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:08.414+0000 7f5bf9e00640 1 -- 172.21.15.96:0/1347225760 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5bfc145320 msgr2=0x7f5bfc0b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:08.418 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:08.414+0000 7f5bf9e00640 1 --2- 172.21.15.96:0/1347225760 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5bfc145320 0x7f5bfc0b15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:08.418 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:08.414+0000 7f5bf9e00640 1 -- 172.21.15.96:0/1347225760 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bfc15f3f0 con 0x7f5bfc0723d0 2024-05-28T06:14:08.418 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:08.415+0000 7f5bfa800640 1 --2- 172.21.15.96:0/1347225760 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5bfc145320 0x7f5bfc0b15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:14:08.418 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:08.415+0000 7f5bf9e00640 1 --2- 172.21.15.96:0/1347225760 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5bfc0723d0 0x7f5bfc15d5a0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f5bec00b3f0 tx=0x7f5bec00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:08.418 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:08.415+0000 7f5bf2000640 1 -- 172.21.15.96:0/1347225760 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bec015020 con 0x7f5bfc0723d0 2024-05-28T06:14:08.418 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:08.415+0000 7f5bf2000640 1 -- 172.21.15.96:0/1347225760 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5bec011830 con 0x7f5bfc0723d0 2024-05-28T06:14:08.418 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:08.415+0000 7f5bf2000640 1 -- 172.21.15.96:0/1347225760 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bec011c20 con 0x7f5bfc0723d0 2024-05-28T06:14:08.418 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:08.415+0000 7f5c00a00640 1 -- 172.21.15.96:0/1347225760 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bfc15f6d0 con 0x7f5bfc0723d0 2024-05-28T06:14:08.418 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:08.415+0000 7f5c00a00640 1 -- 172.21.15.96:0/1347225760 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bfc15dca0 con 0x7f5bfc0723d0 2024-05-28T06:14:08.419 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:08.416+0000 7f5c00a00640 1 -- 172.21.15.96:0/1347225760 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5bfc0689e0 con 0x7f5bfc0723d0 2024-05-28T06:14:08.419 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:08.417+0000 7f5bf2000640 1 -- 172.21.15.96:0/1347225760 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 21) ==== 92111+0+0 (secure 0 0 0) 0x7f5bec018020 con 0x7f5bfc0723d0 2024-05-28T06:14:08.420 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:08.419+0000 7f5bf2000640 1 --2- 172.21.15.96:0/1347225760 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5bd806d4f0 0x7f5bd806f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:08.420 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:08.419+0000 7f5bf2000640 1 -- 172.21.15.96:0/1347225760 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5737+0+0 (secure 0 0 0) 0x7f5bec08a3f0 con 0x7f5bfc0723d0 2024-05-28T06:14:08.420 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:08.419+0000 7f5bf3e00640 1 --2- 172.21.15.96:0/1347225760 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5bd806d4f0 0x7f5bd806f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:08.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:08.420+0000 7f5bf3e00640 1 --2- 172.21.15.96:0/1347225760 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5bd806d4f0 0x7f5bd806f9b0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f5bfc0b1c10 tx=0x7f5be4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:08.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:08.424+0000 7f5bf2000640 1 -- 172.21.15.96:0/1347225760 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5bec01f030 con 0x7f5bfc0723d0 2024-05-28T06:14:08.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:08.690+0000 7f5c00a00640 1 -- 172.21.15.96:0/1347225760 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f5bfc0b34e0 con 0x7f5bfc0723d0 2024-05-28T06:14:08.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:08.691+0000 7f5bf2000640 1 -- 172.21.15.96:0/1347225760 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+2331 (secure 0 0 0) 0x7f5bec0172c0 con 0x7f5bfc0723d0 2024-05-28T06:14:08.693 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:14:08.693 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:14:08.693 INFO:teuthology.orchestra.run.smithi096.stdout: "fsid": "e4a831b5-e6d1-47a7-96cb-bf85154ee0d2", 2024-05-28T06:14:08.693 INFO:teuthology.orchestra.run.smithi096.stdout: "health": { 2024-05-28T06:14:08.693 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "HEALTH_OK", 2024-05-28T06:14:08.693 INFO:teuthology.orchestra.run.smithi096.stdout: "checks": {}, 2024-05-28T06:14:08.693 INFO:teuthology.orchestra.run.smithi096.stdout: "mutes": [] 2024-05-28T06:14:08.693 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:14:08.693 INFO:teuthology.orchestra.run.smithi096.stdout: "election_epoch": 8, 2024-05-28T06:14:08.693 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum": [ 2024-05-28T06:14:08.693 INFO:teuthology.orchestra.run.smithi096.stdout: 0, 2024-05-28T06:14:08.693 INFO:teuthology.orchestra.run.smithi096.stdout: 1, 2024-05-28T06:14:08.693 INFO:teuthology.orchestra.run.smithi096.stdout: 2 2024-05-28T06:14:08.694 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:14:08.694 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_names": [ 2024-05-28T06:14:08.694 INFO:teuthology.orchestra.run.smithi096.stdout: "a", 2024-05-28T06:14:08.694 INFO:teuthology.orchestra.run.smithi096.stdout: "b", 2024-05-28T06:14:08.694 INFO:teuthology.orchestra.run.smithi096.stdout: "c" 2024-05-28T06:14:08.694 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:14:08.694 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_age": 622, 2024-05-28T06:14:08.694 INFO:teuthology.orchestra.run.smithi096.stdout: "monmap": { 2024-05-28T06:14:08.694 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 1, 2024-05-28T06:14:08.694 INFO:teuthology.orchestra.run.smithi096.stdout: "min_mon_release_name": "squid", 2024-05-28T06:14:08.694 INFO:teuthology.orchestra.run.smithi096.stdout: "num_mons": 3 2024-05-28T06:14:08.694 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:14:08.694 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap": { 2024-05-28T06:14:08.695 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 60, 2024-05-28T06:14:08.695 INFO:teuthology.orchestra.run.smithi096.stdout: "num_osds": 8, 2024-05-28T06:14:08.695 INFO:teuthology.orchestra.run.smithi096.stdout: "num_up_osds": 8, 2024-05-28T06:14:08.695 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_up_since": 1716876238, 2024-05-28T06:14:08.695 INFO:teuthology.orchestra.run.smithi096.stdout: "num_in_osds": 8, 2024-05-28T06:14:08.695 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_in_since": 1716876229, 2024-05-28T06:14:08.695 INFO:teuthology.orchestra.run.smithi096.stdout: "num_remapped_pgs": 0 2024-05-28T06:14:08.695 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:14:08.695 INFO:teuthology.orchestra.run.smithi096.stdout: "pgmap": { 2024-05-28T06:14:08.695 INFO:teuthology.orchestra.run.smithi096.stdout: "pgs_by_state": [ 2024-05-28T06:14:08.695 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:14:08.695 INFO:teuthology.orchestra.run.smithi096.stdout: "state_name": "active+clean", 2024-05-28T06:14:08.695 INFO:teuthology.orchestra.run.smithi096.stdout: "count": 97 2024-05-28T06:14:08.695 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:14:08.696 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:14:08.696 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pgs": 97, 2024-05-28T06:14:08.696 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pools": 3, 2024-05-28T06:14:08.696 INFO:teuthology.orchestra.run.smithi096.stdout: "num_objects": 4321, 2024-05-28T06:14:08.696 INFO:teuthology.orchestra.run.smithi096.stdout: "data_bytes": 17978963346, 2024-05-28T06:14:08.696 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_used": 36367081472, 2024-05-28T06:14:08.696 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_avail": 736727031808, 2024-05-28T06:14:08.696 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_total": 773094113280, 2024-05-28T06:14:08.696 INFO:teuthology.orchestra.run.smithi096.stdout: "write_bytes_sec": 1023, 2024-05-28T06:14:08.696 INFO:teuthology.orchestra.run.smithi096.stdout: "read_op_per_sec": 0, 2024-05-28T06:14:08.696 INFO:teuthology.orchestra.run.smithi096.stdout: "write_op_per_sec": 0 2024-05-28T06:14:08.696 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:14:08.696 INFO:teuthology.orchestra.run.smithi096.stdout: "fsmap": { 2024-05-28T06:14:08.696 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 35, 2024-05-28T06:14:08.697 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 4, 2024-05-28T06:14:08.697 INFO:teuthology.orchestra.run.smithi096.stdout: "up": 1, 2024-05-28T06:14:08.697 INFO:teuthology.orchestra.run.smithi096.stdout: "in": 1, 2024-05-28T06:14:08.697 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 1, 2024-05-28T06:14:08.697 INFO:teuthology.orchestra.run.smithi096.stdout: "by_rank": [ 2024-05-28T06:14:08.697 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:14:08.697 INFO:teuthology.orchestra.run.smithi096.stdout: "filesystem_id": 4, 2024-05-28T06:14:08.697 INFO:teuthology.orchestra.run.smithi096.stdout: "rank": 0, 2024-05-28T06:14:08.697 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "a", 2024-05-28T06:14:08.697 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "up:active", 2024-05-28T06:14:08.697 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 6489 2024-05-28T06:14:08.697 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:14:08.697 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:14:08.697 INFO:teuthology.orchestra.run.smithi096.stdout: "up:standby": 3 2024-05-28T06:14:08.698 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:14:08.698 INFO:teuthology.orchestra.run.smithi096.stdout: "mgrmap": { 2024-05-28T06:14:08.698 INFO:teuthology.orchestra.run.smithi096.stdout: "available": true, 2024-05-28T06:14:08.698 INFO:teuthology.orchestra.run.smithi096.stdout: "num_standbys": 1, 2024-05-28T06:14:08.698 INFO:teuthology.orchestra.run.smithi096.stdout: "modules": [ 2024-05-28T06:14:08.698 INFO:teuthology.orchestra.run.smithi096.stdout: "iostat", 2024-05-28T06:14:08.698 INFO:teuthology.orchestra.run.smithi096.stdout: "nfs", 2024-05-28T06:14:08.698 INFO:teuthology.orchestra.run.smithi096.stdout: "restful" 2024-05-28T06:14:08.698 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:14:08.698 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:14:08.698 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:14:08.698 INFO:teuthology.orchestra.run.smithi096.stdout: "servicemap": { 2024-05-28T06:14:08.698 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 11, 2024-05-28T06:14:08.698 INFO:teuthology.orchestra.run.smithi096.stdout: "modified": "2024-05-28T06:13:33.716527+0000", 2024-05-28T06:14:08.699 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:14:08.699 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:14:08.699 INFO:teuthology.orchestra.run.smithi096.stdout: "progress_events": { 2024-05-28T06:14:08.699 INFO:teuthology.orchestra.run.smithi096.stdout: "58c36a82-8e7f-41c9-9e90-890fce489f3b": { 2024-05-28T06:14:08.699 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "All cloning finished, waiting 9.0 seconds before exiting (41s)\n [============================] ", 2024-05-28T06:14:08.699 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": 1, 2024-05-28T06:14:08.699 INFO:teuthology.orchestra.run.smithi096.stdout: "add_to_ceph_s": true 2024-05-28T06:14:08.699 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:14:08.699 INFO:teuthology.orchestra.run.smithi096.stdout: "6a73bfaa-3b9e-443d-83e8-82fc1faa8b7f": { 2024-05-28T06:14:08.699 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "No pending clones left, waiting 9.0 seconds before exiting (35s)\n [============================] ", 2024-05-28T06:14:08.699 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": 1, 2024-05-28T06:14:08.699 INFO:teuthology.orchestra.run.smithi096.stdout: "add_to_ceph_s": true 2024-05-28T06:14:08.699 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:14:08.699 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:14:08.700 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:14:08.700 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:08.693+0000 7f5c00a00640 1 -- 172.21.15.96:0/1347225760 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5bd806d4f0 msgr2=0x7f5bd806f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:08.700 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:08.693+0000 7f5c00a00640 1 --2- 172.21.15.96:0/1347225760 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5bd806d4f0 0x7f5bd806f9b0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f5bfc0b1c10 tx=0x7f5be4008040 comp rx=0 tx=0).stop 2024-05-28T06:14:08.700 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:08.693+0000 7f5c00a00640 1 -- 172.21.15.96:0/1347225760 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5bfc0723d0 msgr2=0x7f5bfc15d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:08.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:08.693+0000 7f5c00a00640 1 --2- 172.21.15.96:0/1347225760 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5bfc0723d0 0x7f5bfc15d5a0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f5bec00b3f0 tx=0x7f5bec00b8b0 comp rx=0 tx=0).stop 2024-05-28T06:14:08.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:08.694+0000 7f5c00a00640 1 -- 172.21.15.96:0/1347225760 shutdown_connections 2024-05-28T06:14:08.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:08.694+0000 7f5c00a00640 1 --2- 172.21.15.96:0/1347225760 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5bd806d4f0 0x7f5bd806f9b0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:08.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:08.694+0000 7f5c00a00640 1 --2- 172.21.15.96:0/1347225760 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5bfc145320 0x7f5bfc0b15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:08.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:08.694+0000 7f5c00a00640 1 --2- 172.21.15.96:0/1347225760 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5bfc0b41e0 0x7f5bfc0b1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:08.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:08.694+0000 7f5c00a00640 1 --2- 172.21.15.96:0/1347225760 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5bfc0723d0 0x7f5bfc15d5a0 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:08.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:08.694+0000 7f5c00a00640 1 -- 172.21.15.96:0/1347225760 >> 172.21.15.96:0/1347225760 conn(0x7f5bfc067ce0 msgr2=0x7f5bfc071cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:08.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:08.694+0000 7f5c00a00640 1 -- 172.21.15.96:0/1347225760 shutdown_connections 2024-05-28T06:14:08.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:08.694+0000 7f5c00a00640 1 -- 172.21.15.96:0/1347225760 wait complete. 2024-05-28T06:14:13.716 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-05-28T06:14:13.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:13.927+0000 7f11d2000640 1 Processor -- start 2024-05-28T06:14:13.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:13.927+0000 7f11d2000640 1 -- start start 2024-05-28T06:14:13.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:13.929+0000 7f11d2000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f11cc06b900 0x7f11cc06bce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:13.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:13.929+0000 7f11d2000640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11cc06ff00 con 0x7f11cc15ed60 2024-05-28T06:14:13.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:13.929+0000 7f11d2000640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11cc0d05f0 con 0x7f11cc0cba20 2024-05-28T06:14:13.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:13.929+0000 7f11d2000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f11cc06c220 con 0x7f11cc06b900 2024-05-28T06:14:13.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:13.929+0000 7f11caa00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f11cc15ed60 0x7f11cc06f0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:39336/0 (socket says 172.21.15.96:39336) 2024-05-28T06:14:13.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:13.929+0000 7f11caa00640 1 -- 172.21.15.96:0/1661596483 learned_addr learned my addr 172.21.15.96:0/1661596483 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:14:13.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:13.930+0000 7f11cb400640 1 --2- 172.21.15.96:0/1661596483 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f11cc06b900 0x7f11cc06bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:13.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:13.930+0000 7f11ca000640 1 -- 172.21.15.96:0/1661596483 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3399181835 0 0) 0x7f11cc06ff00 con 0x7f11cc15ed60 2024-05-28T06:14:13.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:13.930+0000 7f11ca000640 1 -- 172.21.15.96:0/1661596483 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f11b0003520 con 0x7f11cc15ed60 2024-05-28T06:14:13.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:13.930+0000 7f11ca000640 1 -- 172.21.15.96:0/1661596483 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2330275703 0 0) 0x7f11b0003520 con 0x7f11cc15ed60 2024-05-28T06:14:13.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:13.930+0000 7f11ca000640 1 -- 172.21.15.96:0/1661596483 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f11cc06b900 msgr2=0x7f11cc06bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:13.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:13.930+0000 7f11ca000640 1 --2- 172.21.15.96:0/1661596483 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f11cc06b900 0x7f11cc06bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:13.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:13.930+0000 7f11ca000640 1 -- 172.21.15.96:0/1661596483 >> v1:172.21.15.123:6789/0 conn(0x7f11cc0cba20 legacy=0x7f11cc06f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:13.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:13.930+0000 7f11ca000640 1 -- 172.21.15.96:0/1661596483 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11cc0d0890 con 0x7f11cc15ed60 2024-05-28T06:14:13.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:13.930+0000 7f11ca000640 1 -- 172.21.15.96:0/1661596483 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f11b8001670 con 0x7f11cc15ed60 2024-05-28T06:14:13.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:13.930+0000 7f11ca000640 1 -- 172.21.15.96:0/1661596483 >> v1:172.21.15.96:6789/0 conn(0x7f11cc15ed60 legacy=0x7f11cc06f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:13.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:13.930+0000 7f11ca000640 1 --2- 172.21.15.96:0/1661596483 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f11cc06b900 0x7f11b00039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:13.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:13.930+0000 7f11ca000640 1 --2- 172.21.15.96:0/1661596483 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11b0003fb0 0x7f11b0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:13.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:13.930+0000 7f11ca000640 1 --2- 172.21.15.96:0/1661596483 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f11b0004d40 0x7f11b00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:13.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:13.930+0000 7f11ca000640 1 -- 172.21.15.96:0/1661596483 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f11cc06c220 con 0x7f11b0003fb0 2024-05-28T06:14:13.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:13.930+0000 7f11ca000640 1 -- 172.21.15.96:0/1661596483 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f11b000bbb0 con 0x7f11b0004d40 2024-05-28T06:14:13.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:13.930+0000 7f11ca000640 1 -- 172.21.15.96:0/1661596483 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f11b000bdb0 con 0x7f11cc06b900 2024-05-28T06:14:13.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:13.931+0000 7f11caa00640 1 --2- 172.21.15.96:0/1661596483 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11b0003fb0 0x7f11b0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:13.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:13.931+0000 7f11cbe00640 1 --2- 172.21.15.96:0/1661596483 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f11b0004d40 0x7f11b00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:13.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:13.931+0000 7f11caa00640 1 -- 172.21.15.96:0/1661596483 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f11cc06b900 msgr2=0x7f11b00039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:13.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:13.931+0000 7f11caa00640 1 --2- 172.21.15.96:0/1661596483 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f11cc06b900 0x7f11b00039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:13.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:13.931+0000 7f11caa00640 1 -- 172.21.15.96:0/1661596483 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f11b0004d40 msgr2=0x7f11b00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:13.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:13.931+0000 7f11caa00640 1 --2- 172.21.15.96:0/1661596483 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f11b0004d40 0x7f11b00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:13.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:13.931+0000 7f11caa00640 1 -- 172.21.15.96:0/1661596483 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11b0003520 con 0x7f11b0003fb0 2024-05-28T06:14:13.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:13.931+0000 7f11cbe00640 1 --2- 172.21.15.96:0/1661596483 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f11b0004d40 0x7f11b00051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:14:13.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:13.932+0000 7f11caa00640 1 --2- 172.21.15.96:0/1661596483 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11b0003fb0 0x7f11b0004800 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f11b800c190 tx=0x7f11b0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=736fbf420d2dcdbb server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:13.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:13.932+0000 7f11ca000640 1 -- 172.21.15.96:0/1661596483 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11b8007c60 con 0x7f11b0003fb0 2024-05-28T06:14:13.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:13.932+0000 7f11ca000640 1 -- 172.21.15.96:0/1661596483 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11b8005ad0 con 0x7f11b0003fb0 2024-05-28T06:14:13.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:13.932+0000 7f11ca000640 1 -- 172.21.15.96:0/1661596483 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11b8007070 con 0x7f11b0003fb0 2024-05-28T06:14:13.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:13.933+0000 7f11d2000640 1 -- 172.21.15.96:0/1661596483 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11b0003fb0 msgr2=0x7f11b0004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:13.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:13.933+0000 7f11d2000640 1 --2- 172.21.15.96:0/1661596483 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11b0003fb0 0x7f11b0004800 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f11b800c190 tx=0x7f11b0005830 comp rx=0 tx=0).stop 2024-05-28T06:14:13.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:13.933+0000 7f11cbe00640 1 -- 172.21.15.96:0/1661596483 reap_dead start 2024-05-28T06:14:13.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:13.933+0000 7f11d2000640 1 -- 172.21.15.96:0/1661596483 shutdown_connections 2024-05-28T06:14:13.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:13.933+0000 7f11d2000640 1 -- 172.21.15.96:0/1661596483 >> 172.21.15.96:0/1661596483 conn(0x7f11cc09aa30 msgr2=0x7f11cc0b2f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:13.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:13.933+0000 7f11d2000640 1 -- 172.21.15.96:0/1661596483 shutdown_connections 2024-05-28T06:14:13.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:13.933+0000 7f11d2000640 1 -- 172.21.15.96:0/1661596483 wait complete. 2024-05-28T06:14:13.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:13.935+0000 7f11d2000640 1 Processor -- start 2024-05-28T06:14:13.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:13.935+0000 7f11d2000640 1 -- start start 2024-05-28T06:14:13.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:13.936+0000 7f11d2000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f11b0003fb0 0x7f11cc0b5f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:13.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:13.936+0000 7f11d2000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11cc0b4460 0x7f11cc1f8070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:13.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:13.936+0000 7f11d2000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f11cc1f85b0 0x7f11cc1f8a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:13.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:13.936+0000 7f11d2000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f11cc0b3810 con 0x7f11cc0b4460 2024-05-28T06:14:13.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:13.936+0000 7f11d2000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f11cc0d0e50 con 0x7f11b0003fb0 2024-05-28T06:14:13.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:13.936+0000 7f11d2000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f11cc0d1130 con 0x7f11cc1f85b0 2024-05-28T06:14:13.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:13.936+0000 7f11caa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11cc0b4460 0x7f11cc1f8070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:13.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:13.936+0000 7f11caa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11cc0b4460 0x7f11cc1f8070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:45696/0 (socket says 172.21.15.96:45696) 2024-05-28T06:14:13.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:13.936+0000 7f11caa00640 1 -- 172.21.15.96:0/3497248168 learned_addr learned my addr 172.21.15.96:0/3497248168 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:13.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:13.937+0000 7f11cbe00640 1 --2- 172.21.15.96:0/3497248168 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f11cc1f85b0 0x7f11cc1f8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:13.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:13.937+0000 7f11caa00640 1 -- 172.21.15.96:0/3497248168 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f11cc1f85b0 msgr2=0x7f11cc1f8a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:13.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:13.937+0000 7f11caa00640 1 --2- 172.21.15.96:0/3497248168 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f11cc1f85b0 0x7f11cc1f8a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:13.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:13.937+0000 7f11cb400640 1 --2- 172.21.15.96:0/3497248168 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f11b0003fb0 0x7f11cc0b5f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:13.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:13.937+0000 7f11caa00640 1 -- 172.21.15.96:0/3497248168 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f11b0003fb0 msgr2=0x7f11cc0b5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:13.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:13.937+0000 7f11caa00640 1 --2- 172.21.15.96:0/3497248168 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f11b0003fb0 0x7f11cc0b5f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:13.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:13.937+0000 7f11caa00640 1 -- 172.21.15.96:0/3497248168 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11cc15f3f0 con 0x7f11cc0b4460 2024-05-28T06:14:13.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:13.937+0000 7f11cb400640 1 --2- 172.21.15.96:0/3497248168 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f11b0003fb0 0x7f11cc0b5f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:14:13.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:13.937+0000 7f11caa00640 1 --2- 172.21.15.96:0/3497248168 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11cc0b4460 0x7f11cc1f8070 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f11b800c2b0 tx=0x7f11b8001970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:13.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:13.937+0000 7f11c3400640 1 -- 172.21.15.96:0/3497248168 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11b80013b0 con 0x7f11cc0b4460 2024-05-28T06:14:13.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:13.937+0000 7f11c3400640 1 -- 172.21.15.96:0/3497248168 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11b8043070 con 0x7f11cc0b4460 2024-05-28T06:14:13.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:13.937+0000 7f11d2000640 1 -- 172.21.15.96:0/3497248168 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11cc15f690 con 0x7f11cc0b4460 2024-05-28T06:14:13.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:13.937+0000 7f11c3400640 1 -- 172.21.15.96:0/3497248168 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11b8007070 con 0x7f11cc0b4460 2024-05-28T06:14:13.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:13.938+0000 7f11d2000640 1 -- 172.21.15.96:0/3497248168 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11cc0b2830 con 0x7f11cc0b4460 2024-05-28T06:14:13.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:13.939+0000 7f11d2000640 1 -- 172.21.15.96:0/3497248168 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11cc0cd5f0 con 0x7f11cc0b4460 2024-05-28T06:14:13.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:13.940+0000 7f11c3400640 1 -- 172.21.15.96:0/3497248168 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 21) ==== 92111+0+0 (secure 0 0 0) 0x7f11b8007210 con 0x7f11cc0b4460 2024-05-28T06:14:13.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:13.940+0000 7f11c3400640 1 --2- 172.21.15.96:0/3497248168 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f11ac06d4c0 0x7f11ac06f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:13.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:13.941+0000 7f11c3400640 1 -- 172.21.15.96:0/3497248168 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5737+0+0 (secure 0 0 0) 0x7f11b80aa830 con 0x7f11cc0b4460 2024-05-28T06:14:13.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:13.941+0000 7f11cb400640 1 --2- 172.21.15.96:0/3497248168 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f11ac06d4c0 0x7f11ac06f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:13.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:13.943+0000 7f11cb400640 1 --2- 172.21.15.96:0/3497248168 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f11ac06d4c0 0x7f11ac06f980 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f11bc009a40 tx=0x7f11bc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:13.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:13.945+0000 7f11c3400640 1 -- 172.21.15.96:0/3497248168 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f11cc0cd5f0 con 0x7f11cc0b4460 2024-05-28T06:14:14.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:14.219+0000 7f11d2000640 1 -- 172.21.15.96:0/3497248168 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f11cc0b7480 con 0x7f11cc0b4460 2024-05-28T06:14:14.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:14.220+0000 7f11c3400640 1 -- 172.21.15.96:0/3497248168 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+2330 (secure 0 0 0) 0x7f11cc0b7480 con 0x7f11cc0b4460 2024-05-28T06:14:14.222 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:14:14.222 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:14:14.222 INFO:teuthology.orchestra.run.smithi096.stdout: "fsid": "e4a831b5-e6d1-47a7-96cb-bf85154ee0d2", 2024-05-28T06:14:14.222 INFO:teuthology.orchestra.run.smithi096.stdout: "health": { 2024-05-28T06:14:14.222 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "HEALTH_OK", 2024-05-28T06:14:14.222 INFO:teuthology.orchestra.run.smithi096.stdout: "checks": {}, 2024-05-28T06:14:14.222 INFO:teuthology.orchestra.run.smithi096.stdout: "mutes": [] 2024-05-28T06:14:14.222 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:14:14.222 INFO:teuthology.orchestra.run.smithi096.stdout: "election_epoch": 8, 2024-05-28T06:14:14.222 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum": [ 2024-05-28T06:14:14.222 INFO:teuthology.orchestra.run.smithi096.stdout: 0, 2024-05-28T06:14:14.223 INFO:teuthology.orchestra.run.smithi096.stdout: 1, 2024-05-28T06:14:14.223 INFO:teuthology.orchestra.run.smithi096.stdout: 2 2024-05-28T06:14:14.223 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:14:14.223 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_names": [ 2024-05-28T06:14:14.223 INFO:teuthology.orchestra.run.smithi096.stdout: "a", 2024-05-28T06:14:14.223 INFO:teuthology.orchestra.run.smithi096.stdout: "b", 2024-05-28T06:14:14.223 INFO:teuthology.orchestra.run.smithi096.stdout: "c" 2024-05-28T06:14:14.223 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:14:14.223 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_age": 627, 2024-05-28T06:14:14.223 INFO:teuthology.orchestra.run.smithi096.stdout: "monmap": { 2024-05-28T06:14:14.223 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 1, 2024-05-28T06:14:14.223 INFO:teuthology.orchestra.run.smithi096.stdout: "min_mon_release_name": "squid", 2024-05-28T06:14:14.223 INFO:teuthology.orchestra.run.smithi096.stdout: "num_mons": 3 2024-05-28T06:14:14.223 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:14:14.223 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap": { 2024-05-28T06:14:14.223 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 60, 2024-05-28T06:14:14.224 INFO:teuthology.orchestra.run.smithi096.stdout: "num_osds": 8, 2024-05-28T06:14:14.224 INFO:teuthology.orchestra.run.smithi096.stdout: "num_up_osds": 8, 2024-05-28T06:14:14.224 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_up_since": 1716876238, 2024-05-28T06:14:14.224 INFO:teuthology.orchestra.run.smithi096.stdout: "num_in_osds": 8, 2024-05-28T06:14:14.224 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_in_since": 1716876229, 2024-05-28T06:14:14.224 INFO:teuthology.orchestra.run.smithi096.stdout: "num_remapped_pgs": 0 2024-05-28T06:14:14.224 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:14:14.224 INFO:teuthology.orchestra.run.smithi096.stdout: "pgmap": { 2024-05-28T06:14:14.224 INFO:teuthology.orchestra.run.smithi096.stdout: "pgs_by_state": [ 2024-05-28T06:14:14.224 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:14:14.224 INFO:teuthology.orchestra.run.smithi096.stdout: "state_name": "active+clean", 2024-05-28T06:14:14.224 INFO:teuthology.orchestra.run.smithi096.stdout: "count": 97 2024-05-28T06:14:14.224 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:14:14.224 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:14:14.224 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pgs": 97, 2024-05-28T06:14:14.225 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pools": 3, 2024-05-28T06:14:14.225 INFO:teuthology.orchestra.run.smithi096.stdout: "num_objects": 4321, 2024-05-28T06:14:14.225 INFO:teuthology.orchestra.run.smithi096.stdout: "data_bytes": 17978965952, 2024-05-28T06:14:14.225 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_used": 36367081472, 2024-05-28T06:14:14.225 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_avail": 736727031808, 2024-05-28T06:14:14.225 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_total": 773094113280, 2024-05-28T06:14:14.225 INFO:teuthology.orchestra.run.smithi096.stdout: "write_bytes_sec": 341, 2024-05-28T06:14:14.225 INFO:teuthology.orchestra.run.smithi096.stdout: "read_op_per_sec": 0, 2024-05-28T06:14:14.225 INFO:teuthology.orchestra.run.smithi096.stdout: "write_op_per_sec": 0 2024-05-28T06:14:14.225 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:14:14.225 INFO:teuthology.orchestra.run.smithi096.stdout: "fsmap": { 2024-05-28T06:14:14.225 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 35, 2024-05-28T06:14:14.225 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 4, 2024-05-28T06:14:14.225 INFO:teuthology.orchestra.run.smithi096.stdout: "up": 1, 2024-05-28T06:14:14.225 INFO:teuthology.orchestra.run.smithi096.stdout: "in": 1, 2024-05-28T06:14:14.226 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 1, 2024-05-28T06:14:14.226 INFO:teuthology.orchestra.run.smithi096.stdout: "by_rank": [ 2024-05-28T06:14:14.226 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:14:14.226 INFO:teuthology.orchestra.run.smithi096.stdout: "filesystem_id": 4, 2024-05-28T06:14:14.226 INFO:teuthology.orchestra.run.smithi096.stdout: "rank": 0, 2024-05-28T06:14:14.226 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "a", 2024-05-28T06:14:14.226 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "up:active", 2024-05-28T06:14:14.226 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 6489 2024-05-28T06:14:14.226 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:14:14.226 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:14:14.226 INFO:teuthology.orchestra.run.smithi096.stdout: "up:standby": 3 2024-05-28T06:14:14.226 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:14:14.226 INFO:teuthology.orchestra.run.smithi096.stdout: "mgrmap": { 2024-05-28T06:14:14.226 INFO:teuthology.orchestra.run.smithi096.stdout: "available": true, 2024-05-28T06:14:14.227 INFO:teuthology.orchestra.run.smithi096.stdout: "num_standbys": 1, 2024-05-28T06:14:14.227 INFO:teuthology.orchestra.run.smithi096.stdout: "modules": [ 2024-05-28T06:14:14.227 INFO:teuthology.orchestra.run.smithi096.stdout: "iostat", 2024-05-28T06:14:14.227 INFO:teuthology.orchestra.run.smithi096.stdout: "nfs", 2024-05-28T06:14:14.227 INFO:teuthology.orchestra.run.smithi096.stdout: "restful" 2024-05-28T06:14:14.227 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:14:14.227 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:14:14.227 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:14:14.227 INFO:teuthology.orchestra.run.smithi096.stdout: "servicemap": { 2024-05-28T06:14:14.227 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 11, 2024-05-28T06:14:14.227 INFO:teuthology.orchestra.run.smithi096.stdout: "modified": "2024-05-28T06:13:33.716527+0000", 2024-05-28T06:14:14.227 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:14:14.227 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:14:14.227 INFO:teuthology.orchestra.run.smithi096.stdout: "progress_events": { 2024-05-28T06:14:14.227 INFO:teuthology.orchestra.run.smithi096.stdout: "58c36a82-8e7f-41c9-9e90-890fce489f3b": { 2024-05-28T06:14:14.228 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "All cloning finished, waiting 3.0 seconds before exiting (47s)\n [============================] ", 2024-05-28T06:14:14.228 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": 1, 2024-05-28T06:14:14.228 INFO:teuthology.orchestra.run.smithi096.stdout: "add_to_ceph_s": true 2024-05-28T06:14:14.228 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:14:14.228 INFO:teuthology.orchestra.run.smithi096.stdout: "6a73bfaa-3b9e-443d-83e8-82fc1faa8b7f": { 2024-05-28T06:14:14.228 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "No pending clones left, waiting 3.0 seconds before exiting (41s)\n [============================] ", 2024-05-28T06:14:14.228 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": 1, 2024-05-28T06:14:14.228 INFO:teuthology.orchestra.run.smithi096.stdout: "add_to_ceph_s": true 2024-05-28T06:14:14.228 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:14:14.228 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:14:14.228 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:14:14.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:14.223+0000 7f11d2000640 1 -- 172.21.15.96:0/3497248168 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f11ac06d4c0 msgr2=0x7f11ac06f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:14.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:14.223+0000 7f11d2000640 1 --2- 172.21.15.96:0/3497248168 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f11ac06d4c0 0x7f11ac06f980 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f11bc009a40 tx=0x7f11bc008040 comp rx=0 tx=0).stop 2024-05-28T06:14:14.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:14.223+0000 7f11d2000640 1 -- 172.21.15.96:0/3497248168 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11cc0b4460 msgr2=0x7f11cc1f8070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:14.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:14.223+0000 7f11d2000640 1 --2- 172.21.15.96:0/3497248168 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11cc0b4460 0x7f11cc1f8070 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f11b800c2b0 tx=0x7f11b8001970 comp rx=0 tx=0).stop 2024-05-28T06:14:14.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:14.223+0000 7f11d2000640 1 -- 172.21.15.96:0/3497248168 shutdown_connections 2024-05-28T06:14:14.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:14.223+0000 7f11d2000640 1 --2- 172.21.15.96:0/3497248168 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f11ac06d4c0 0x7f11ac06f980 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:14.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:14.223+0000 7f11d2000640 1 --2- 172.21.15.96:0/3497248168 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f11cc1f85b0 0x7f11cc1f8a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:14.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:14.223+0000 7f11d2000640 1 --2- 172.21.15.96:0/3497248168 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11cc0b4460 0x7f11cc1f8070 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:14.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:14.223+0000 7f11d2000640 1 --2- 172.21.15.96:0/3497248168 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f11b0003fb0 0x7f11cc0b5f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:14.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:14.223+0000 7f11d2000640 1 -- 172.21.15.96:0/3497248168 >> 172.21.15.96:0/3497248168 conn(0x7f11cc09aa30 msgr2=0x7f11cc0aeb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:14.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:14.223+0000 7f11d2000640 1 -- 172.21.15.96:0/3497248168 shutdown_connections 2024-05-28T06:14:14.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:14.223+0000 7f11d2000640 1 -- 172.21.15.96:0/3497248168 wait complete. 2024-05-28T06:14:19.243 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-05-28T06:14:19.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.448+0000 7f5b19a00640 1 Processor -- start 2024-05-28T06:14:19.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.448+0000 7f5b19a00640 1 -- start start 2024-05-28T06:14:19.452 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.450+0000 7f5b19a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5b140ccf50 0x7f5b140cd330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:19.452 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.450+0000 7f5b19a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f5b140ce530 0x7f5b1415cc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:19.452 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.450+0000 7f5b19a00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b140cd870 con 0x7f5b140ce9b0 2024-05-28T06:14:19.452 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.450+0000 7f5b19a00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f5b1406c540 con 0x7f5b140ce530 2024-05-28T06:14:19.452 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.450+0000 7f5b19a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f5b14157180 con 0x7f5b140ccf50 2024-05-28T06:14:19.452 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.450+0000 7f5b12000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f5b140ce530 0x7f5b1415cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:19.452 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.450+0000 7f5b13400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f5b140ce9b0 0x7f5b140d0790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:54794/0 (socket says 172.21.15.96:54794) 2024-05-28T06:14:19.452 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.450+0000 7f5b12000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f5b140ce530 0x7f5b1415cc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48520/0 (socket says 172.21.15.96:48520) 2024-05-28T06:14:19.452 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.450+0000 7f5b13400640 1 -- 172.21.15.96:0/501276956 learned_addr learned my addr 172.21.15.96:0/501276956 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:14:19.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.451+0000 7f5b12a00640 1 --2- 172.21.15.96:0/501276956 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5b140ccf50 0x7f5b140cd330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:19.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.451+0000 7f5b12000640 1 -- 172.21.15.96:0/501276956 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5b140ccf50 msgr2=0x7f5b140cd330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:19.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.451+0000 7f5b12000640 1 --2- 172.21.15.96:0/501276956 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5b140ccf50 0x7f5b140cd330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:19.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.451+0000 7f5b12000640 1 -- 172.21.15.96:0/501276956 >> v1:172.21.15.96:6789/0 conn(0x7f5b140ce9b0 legacy=0x7f5b140d0790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:19.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.451+0000 7f5b12000640 1 -- 172.21.15.96:0/501276956 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b1406bca0 con 0x7f5b140ce530 2024-05-28T06:14:19.454 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.451+0000 7f5b11600640 1 -- 172.21.15.96:0/501276956 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2495756214 0 0) 0x7f5b140cd870 con 0x7f5b140ce9b0 2024-05-28T06:14:19.454 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.452+0000 7f5b12000640 1 --2- 172.21.15.96:0/501276956 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5b140ce530 0x7f5b1415cc80 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f5b000097a0 tx=0x7f5b0002a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac4daad30ce017ec server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:19.454 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.453+0000 7f5b11600640 1 -- 172.21.15.96:0/501276956 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b00037070 con 0x7f5b140ce530 2024-05-28T06:14:19.454 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.453+0000 7f5b11600640 1 -- 172.21.15.96:0/501276956 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b0002ae50 con 0x7f5b140ce530 2024-05-28T06:14:19.454 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.453+0000 7f5b11600640 1 -- 172.21.15.96:0/501276956 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b00032660 con 0x7f5b140ce530 2024-05-28T06:14:19.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.453+0000 7f5b19a00640 1 -- 172.21.15.96:0/501276956 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5b140ce530 msgr2=0x7f5b1415cc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:19.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.453+0000 7f5b19a00640 1 --2- 172.21.15.96:0/501276956 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5b140ce530 0x7f5b1415cc80 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f5b000097a0 tx=0x7f5b0002a100 comp rx=0 tx=0).stop 2024-05-28T06:14:19.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.454+0000 7f5b19a00640 1 -- 172.21.15.96:0/501276956 shutdown_connections 2024-05-28T06:14:19.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.454+0000 7f5b19a00640 1 --2- 172.21.15.96:0/501276956 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5b140ce530 0x7f5b1415cc80 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:19.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.454+0000 7f5b19a00640 1 --2- 172.21.15.96:0/501276956 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5b140ccf50 0x7f5b140cd330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:19.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.454+0000 7f5b19a00640 1 -- 172.21.15.96:0/501276956 >> 172.21.15.96:0/501276956 conn(0x7f5b140b2f40 msgr2=0x7f5b140b3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:19.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.454+0000 7f5b19a00640 1 -- 172.21.15.96:0/501276956 shutdown_connections 2024-05-28T06:14:19.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.454+0000 7f5b19a00640 1 -- 172.21.15.96:0/501276956 wait complete. 2024-05-28T06:14:19.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.455+0000 7f5b19a00640 1 Processor -- start 2024-05-28T06:14:19.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.456+0000 7f5b19a00640 1 -- start start 2024-05-28T06:14:19.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.456+0000 7f5b19a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5b140ccf50 0x7f5b1414b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:19.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.456+0000 7f5b19a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b140ce530 0x7f5b140b47f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:19.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.456+0000 7f5b19a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5b140ce9b0 0x7f5b1409b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:19.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.456+0000 7f5b19a00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f5b140d0270 con 0x7f5b140ccf50 2024-05-28T06:14:19.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.456+0000 7f5b19a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5b1406c260 con 0x7f5b140ce530 2024-05-28T06:14:19.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.456+0000 7f5b19a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f5b1406b900 con 0x7f5b140ce9b0 2024-05-28T06:14:19.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.457+0000 7f5b12a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5b140ccf50 0x7f5b1414b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:19.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.457+0000 7f5b12a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5b140ccf50 0x7f5b1414b890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48528/0 (socket says 172.21.15.96:48528) 2024-05-28T06:14:19.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.457+0000 7f5b12a00640 1 -- 172.21.15.96:0/2344451224 learned_addr learned my addr 172.21.15.96:0/2344451224 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:19.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.457+0000 7f5b13400640 1 --2- 172.21.15.96:0/2344451224 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5b140ce9b0 0x7f5b1409b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:19.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.457+0000 7f5b12a00640 1 -- 172.21.15.96:0/2344451224 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5b140ce9b0 msgr2=0x7f5b1409b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:19.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.457+0000 7f5b12a00640 1 --2- 172.21.15.96:0/2344451224 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5b140ce9b0 0x7f5b1409b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:19.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.457+0000 7f5b12a00640 1 -- 172.21.15.96:0/2344451224 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b140ce530 msgr2=0x7f5b140b47f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:19.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.457+0000 7f5b12a00640 1 --2- 172.21.15.96:0/2344451224 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b140ce530 0x7f5b140b47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:19.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.457+0000 7f5b12a00640 1 -- 172.21.15.96:0/2344451224 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b140c7780 con 0x7f5b140ccf50 2024-05-28T06:14:19.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.457+0000 7f5b12a00640 1 --2- 172.21.15.96:0/2344451224 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5b140ccf50 0x7f5b1414b890 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f5b0400d800 tx=0x7f5b0400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:19.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.458+0000 7f5b0aa00640 1 -- 172.21.15.96:0/2344451224 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b0400b6c0 con 0x7f5b140ccf50 2024-05-28T06:14:19.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.458+0000 7f5b0aa00640 1 -- 172.21.15.96:0/2344451224 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b04019070 con 0x7f5b140ccf50 2024-05-28T06:14:19.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.458+0000 7f5b19a00640 1 -- 172.21.15.96:0/2344451224 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b1409d4f0 con 0x7f5b140ccf50 2024-05-28T06:14:19.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.458+0000 7f5b0aa00640 1 -- 172.21.15.96:0/2344451224 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b0400ba60 con 0x7f5b140ccf50 2024-05-28T06:14:19.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.458+0000 7f5b19a00640 1 -- 172.21.15.96:0/2344451224 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b1409dab0 con 0x7f5b140ccf50 2024-05-28T06:14:19.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.459+0000 7f5b19a00640 1 -- 172.21.15.96:0/2344451224 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b140690f0 con 0x7f5b140ccf50 2024-05-28T06:14:19.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.460+0000 7f5b0aa00640 1 -- 172.21.15.96:0/2344451224 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 21) ==== 92111+0+0 (secure 0 0 0) 0x7f5b0400bc00 con 0x7f5b140ccf50 2024-05-28T06:14:19.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.461+0000 7f5b0aa00640 1 --2- 172.21.15.96:0/2344451224 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5af406d4c0 0x7f5af406f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:19.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.461+0000 7f5b0aa00640 1 -- 172.21.15.96:0/2344451224 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5737+0+0 (secure 0 0 0) 0x7f5b0408acb0 con 0x7f5b140ccf50 2024-05-28T06:14:19.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.462+0000 7f5b12000640 1 --2- 172.21.15.96:0/2344451224 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5af406d4c0 0x7f5af406f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:19.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.463+0000 7f5b12000640 1 --2- 172.21.15.96:0/2344451224 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5af406d4c0 0x7f5af406f980 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f5b00034000 tx=0x7f5b0002c000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:19.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.465+0000 7f5b0aa00640 1 -- 172.21.15.96:0/2344451224 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5b0405a8d0 con 0x7f5b140ccf50 2024-05-28T06:14:19.737 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.736+0000 7f5b19a00640 1 -- 172.21.15.96:0/2344451224 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f5b140b2290 con 0x7f5b140ccf50 2024-05-28T06:14:19.739 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.737+0000 7f5b0aa00640 1 -- 172.21.15.96:0/2344451224 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1824 (secure 0 0 0) 0x7f5b0405a1d0 con 0x7f5b140ccf50 2024-05-28T06:14:19.739 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:14:19.739 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:14:19.739 INFO:teuthology.orchestra.run.smithi096.stdout: "fsid": "e4a831b5-e6d1-47a7-96cb-bf85154ee0d2", 2024-05-28T06:14:19.739 INFO:teuthology.orchestra.run.smithi096.stdout: "health": { 2024-05-28T06:14:19.739 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "HEALTH_OK", 2024-05-28T06:14:19.739 INFO:teuthology.orchestra.run.smithi096.stdout: "checks": {}, 2024-05-28T06:14:19.739 INFO:teuthology.orchestra.run.smithi096.stdout: "mutes": [] 2024-05-28T06:14:19.739 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:14:19.740 INFO:teuthology.orchestra.run.smithi096.stdout: "election_epoch": 8, 2024-05-28T06:14:19.740 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum": [ 2024-05-28T06:14:19.740 INFO:teuthology.orchestra.run.smithi096.stdout: 0, 2024-05-28T06:14:19.740 INFO:teuthology.orchestra.run.smithi096.stdout: 1, 2024-05-28T06:14:19.740 INFO:teuthology.orchestra.run.smithi096.stdout: 2 2024-05-28T06:14:19.740 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:14:19.740 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_names": [ 2024-05-28T06:14:19.740 INFO:teuthology.orchestra.run.smithi096.stdout: "a", 2024-05-28T06:14:19.740 INFO:teuthology.orchestra.run.smithi096.stdout: "b", 2024-05-28T06:14:19.740 INFO:teuthology.orchestra.run.smithi096.stdout: "c" 2024-05-28T06:14:19.740 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:14:19.740 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_age": 633, 2024-05-28T06:14:19.740 INFO:teuthology.orchestra.run.smithi096.stdout: "monmap": { 2024-05-28T06:14:19.740 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 1, 2024-05-28T06:14:19.741 INFO:teuthology.orchestra.run.smithi096.stdout: "min_mon_release_name": "squid", 2024-05-28T06:14:19.741 INFO:teuthology.orchestra.run.smithi096.stdout: "num_mons": 3 2024-05-28T06:14:19.741 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:14:19.741 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap": { 2024-05-28T06:14:19.741 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 60, 2024-05-28T06:14:19.741 INFO:teuthology.orchestra.run.smithi096.stdout: "num_osds": 8, 2024-05-28T06:14:19.741 INFO:teuthology.orchestra.run.smithi096.stdout: "num_up_osds": 8, 2024-05-28T06:14:19.741 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_up_since": 1716876238, 2024-05-28T06:14:19.741 INFO:teuthology.orchestra.run.smithi096.stdout: "num_in_osds": 8, 2024-05-28T06:14:19.741 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_in_since": 1716876229, 2024-05-28T06:14:19.741 INFO:teuthology.orchestra.run.smithi096.stdout: "num_remapped_pgs": 0 2024-05-28T06:14:19.741 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:14:19.741 INFO:teuthology.orchestra.run.smithi096.stdout: "pgmap": { 2024-05-28T06:14:19.741 INFO:teuthology.orchestra.run.smithi096.stdout: "pgs_by_state": [ 2024-05-28T06:14:19.742 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:14:19.742 INFO:teuthology.orchestra.run.smithi096.stdout: "state_name": "active+clean", 2024-05-28T06:14:19.742 INFO:teuthology.orchestra.run.smithi096.stdout: "count": 97 2024-05-28T06:14:19.742 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:14:19.742 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:14:19.742 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pgs": 97, 2024-05-28T06:14:19.742 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pools": 3, 2024-05-28T06:14:19.742 INFO:teuthology.orchestra.run.smithi096.stdout: "num_objects": 4321, 2024-05-28T06:14:19.742 INFO:teuthology.orchestra.run.smithi096.stdout: "data_bytes": 17978965952, 2024-05-28T06:14:19.742 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_used": 36367081472, 2024-05-28T06:14:19.742 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_avail": 736727031808, 2024-05-28T06:14:19.742 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_total": 773094113280, 2024-05-28T06:14:19.743 INFO:teuthology.orchestra.run.smithi096.stdout: "write_bytes_sec": 341, 2024-05-28T06:14:19.743 INFO:teuthology.orchestra.run.smithi096.stdout: "read_op_per_sec": 0, 2024-05-28T06:14:19.743 INFO:teuthology.orchestra.run.smithi096.stdout: "write_op_per_sec": 0 2024-05-28T06:14:19.743 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:14:19.743 INFO:teuthology.orchestra.run.smithi096.stdout: "fsmap": { 2024-05-28T06:14:19.743 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 35, 2024-05-28T06:14:19.743 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 4, 2024-05-28T06:14:19.743 INFO:teuthology.orchestra.run.smithi096.stdout: "up": 1, 2024-05-28T06:14:19.743 INFO:teuthology.orchestra.run.smithi096.stdout: "in": 1, 2024-05-28T06:14:19.743 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 1, 2024-05-28T06:14:19.743 INFO:teuthology.orchestra.run.smithi096.stdout: "by_rank": [ 2024-05-28T06:14:19.743 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:14:19.743 INFO:teuthology.orchestra.run.smithi096.stdout: "filesystem_id": 4, 2024-05-28T06:14:19.743 INFO:teuthology.orchestra.run.smithi096.stdout: "rank": 0, 2024-05-28T06:14:19.744 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "a", 2024-05-28T06:14:19.744 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "up:active", 2024-05-28T06:14:19.744 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 6489 2024-05-28T06:14:19.744 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:14:19.744 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:14:19.744 INFO:teuthology.orchestra.run.smithi096.stdout: "up:standby": 3 2024-05-28T06:14:19.744 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:14:19.744 INFO:teuthology.orchestra.run.smithi096.stdout: "mgrmap": { 2024-05-28T06:14:19.744 INFO:teuthology.orchestra.run.smithi096.stdout: "available": true, 2024-05-28T06:14:19.744 INFO:teuthology.orchestra.run.smithi096.stdout: "num_standbys": 1, 2024-05-28T06:14:19.744 INFO:teuthology.orchestra.run.smithi096.stdout: "modules": [ 2024-05-28T06:14:19.744 INFO:teuthology.orchestra.run.smithi096.stdout: "iostat", 2024-05-28T06:14:19.744 INFO:teuthology.orchestra.run.smithi096.stdout: "nfs", 2024-05-28T06:14:19.744 INFO:teuthology.orchestra.run.smithi096.stdout: "restful" 2024-05-28T06:14:19.744 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:14:19.745 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:14:19.745 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:14:19.745 INFO:teuthology.orchestra.run.smithi096.stdout: "servicemap": { 2024-05-28T06:14:19.745 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 11, 2024-05-28T06:14:19.745 INFO:teuthology.orchestra.run.smithi096.stdout: "modified": "2024-05-28T06:13:33.716527+0000", 2024-05-28T06:14:19.745 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:14:19.745 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:14:19.745 INFO:teuthology.orchestra.run.smithi096.stdout: "progress_events": {} 2024-05-28T06:14:19.745 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:14:19.746 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.740+0000 7f5b19a00640 1 -- 172.21.15.96:0/2344451224 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5af406d4c0 msgr2=0x7f5af406f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:19.746 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.740+0000 7f5b19a00640 1 --2- 172.21.15.96:0/2344451224 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5af406d4c0 0x7f5af406f980 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f5b00034000 tx=0x7f5b0002c000 comp rx=0 tx=0).stop 2024-05-28T06:14:19.746 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.740+0000 7f5b19a00640 1 -- 172.21.15.96:0/2344451224 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5b140ccf50 msgr2=0x7f5b1414b890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:19.746 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.740+0000 7f5b19a00640 1 --2- 172.21.15.96:0/2344451224 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5b140ccf50 0x7f5b1414b890 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f5b0400d800 tx=0x7f5b0400dcc0 comp rx=0 tx=0).stop 2024-05-28T06:14:19.746 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.740+0000 7f5b19a00640 1 -- 172.21.15.96:0/2344451224 shutdown_connections 2024-05-28T06:14:19.746 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.740+0000 7f5b19a00640 1 --2- 172.21.15.96:0/2344451224 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5af406d4c0 0x7f5af406f980 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:19.746 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.740+0000 7f5b19a00640 1 --2- 172.21.15.96:0/2344451224 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5b140ce9b0 0x7f5b1409b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:19.746 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.740+0000 7f5b19a00640 1 --2- 172.21.15.96:0/2344451224 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b140ce530 0x7f5b140b47f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:19.746 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.740+0000 7f5b19a00640 1 --2- 172.21.15.96:0/2344451224 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5b140ccf50 0x7f5b1414b890 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:19.746 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.740+0000 7f5b19a00640 1 -- 172.21.15.96:0/2344451224 >> 172.21.15.96:0/2344451224 conn(0x7f5b140b2f40 msgr2=0x7f5b140ae630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:19.747 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.740+0000 7f5b19a00640 1 -- 172.21.15.96:0/2344451224 shutdown_connections 2024-05-28T06:14:19.747 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.740+0000 7f5b19a00640 1 -- 172.21.15.96:0/2344451224 wait complete. 2024-05-28T06:14:19.761 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume ls cephfs --format json 2024-05-28T06:14:19.956 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.954+0000 7fde0a000640 1 Processor -- start 2024-05-28T06:14:19.956 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.955+0000 7fde0a000640 1 -- start start 2024-05-28T06:14:19.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.956+0000 7fde0a000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fde04153510 0x7fde040b4d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:19.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.956+0000 7fde0a000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fde0415d270 0x7fde040ccff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:19.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.956+0000 7fde0a000640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde0415d900 con 0x7fde040ccb90 2024-05-28T06:14:19.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.956+0000 7fde0a000640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fde0415d6e0 con 0x7fde04153510 2024-05-28T06:14:19.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.956+0000 7fde0a000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fde041538f0 con 0x7fde0415d270 2024-05-28T06:14:19.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.957+0000 7fde09600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fde0415d270 0x7fde040ccff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:19.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.957+0000 7fde09600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fde0415d270 0x7fde040ccff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.96:53096/0 (socket says 172.21.15.96:53096) 2024-05-28T06:14:19.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.957+0000 7fde03e00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7fde040ccb90 0x7fde0415cb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.96:37912/0 (socket says 172.21.15.96:37912) 2024-05-28T06:14:19.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.957+0000 7fde09600640 1 -- 172.21.15.96:0/2465705890 learned_addr learned my addr 172.21.15.96:0/2465705890 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:19.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.957+0000 7fde08c00640 1 --2- 172.21.15.96:0/2465705890 >> v2:172.21.15.123:3300/0 conn(0x7fde04153510 0x7fde040b4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:19.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.958+0000 7fde09600640 1 -- 172.21.15.96:0/2465705890 >> v2:172.21.15.123:3300/0 conn(0x7fde04153510 msgr2=0x7fde040b4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:19.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.958+0000 7fde09600640 1 --2- 172.21.15.96:0/2465705890 >> v2:172.21.15.123:3300/0 conn(0x7fde04153510 0x7fde040b4d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:19.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.958+0000 7fde09600640 1 -- 172.21.15.96:0/2465705890 >> v1:172.21.15.123:6789/0 conn(0x7fde040ccb90 legacy=0x7fde0415cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:19.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.958+0000 7fde09600640 1 -- 172.21.15.96:0/2465705890 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde040b3f90 con 0x7fde0415d270 2024-05-28T06:14:19.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.958+0000 7fde03400640 1 -- 172.21.15.96:0/2465705890 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1638747818 0 0) 0x7fde0415d900 con 0x7fde040ccb90 2024-05-28T06:14:19.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.958+0000 7fde08c00640 1 --2- 172.21.15.96:0/2465705890 >> v2:172.21.15.123:3300/0 conn(0x7fde04153510 0x7fde040b4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-28T06:14:19.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.959+0000 7fde09600640 1 --2- 172.21.15.96:0/2465705890 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fde0415d270 0x7fde040ccff0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fddfc009e50 tx=0x7fddfc02a680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=13fc8d9283733a00 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:19.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.960+0000 7fde03400640 1 -- 172.21.15.96:0/2465705890 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddfc036070 con 0x7fde0415d270 2024-05-28T06:14:19.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.960+0000 7fde03400640 1 -- 172.21.15.96:0/2465705890 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fddfc031600 con 0x7fde0415d270 2024-05-28T06:14:19.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.960+0000 7fde03400640 1 -- 172.21.15.96:0/2465705890 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddfc0319d0 con 0x7fde0415d270 2024-05-28T06:14:19.962 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.960+0000 7fde0a000640 1 -- 172.21.15.96:0/2465705890 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fde0415d270 msgr2=0x7fde040ccff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:19.962 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.960+0000 7fde0a000640 1 --2- 172.21.15.96:0/2465705890 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fde0415d270 0x7fde040ccff0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fddfc009e50 tx=0x7fddfc02a680 comp rx=0 tx=0).stop 2024-05-28T06:14:19.962 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.960+0000 7fde0a000640 1 -- 172.21.15.96:0/2465705890 shutdown_connections 2024-05-28T06:14:19.962 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.960+0000 7fde0a000640 1 --2- 172.21.15.96:0/2465705890 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fde0415d270 0x7fde040ccff0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:19.962 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.960+0000 7fde0a000640 1 --2- 172.21.15.96:0/2465705890 >> v2:172.21.15.123:3300/0 conn(0x7fde04153510 0x7fde040b4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:19.962 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.960+0000 7fde0a000640 1 -- 172.21.15.96:0/2465705890 >> 172.21.15.96:0/2465705890 conn(0x7fde0406b900 msgr2=0x7fde0406bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:19.962 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.960+0000 7fde0a000640 1 -- 172.21.15.96:0/2465705890 shutdown_connections 2024-05-28T06:14:19.963 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.960+0000 7fde0a000640 1 -- 172.21.15.96:0/2465705890 wait complete. 2024-05-28T06:14:19.963 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.962+0000 7fde0a000640 1 Processor -- start 2024-05-28T06:14:19.963 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.962+0000 7fde0a000640 1 -- start start 2024-05-28T06:14:19.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.963+0000 7fde0a000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fde040ccb90 0x7fde040cefa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:19.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.963+0000 7fde0a000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde04153510 0x7fde041514f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:19.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.963+0000 7fde0a000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fde0415d270 0x7fde040ce510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:19.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.963+0000 7fde0a000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fde040b48c0 con 0x7fde0415d270 2024-05-28T06:14:19.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.963+0000 7fde0a000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fde040b4550 con 0x7fde04153510 2024-05-28T06:14:19.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.963+0000 7fde0a000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fde0406e940 con 0x7fde040ccb90 2024-05-28T06:14:19.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.963+0000 7fde09600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fde0415d270 0x7fde040ce510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:19.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.963+0000 7fde09600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fde0415d270 0x7fde040ce510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48532/0 (socket says 172.21.15.96:48532) 2024-05-28T06:14:19.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.963+0000 7fde09600640 1 -- 172.21.15.96:0/2924778256 learned_addr learned my addr 172.21.15.96:0/2924778256 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:19.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.964+0000 7fde08c00640 1 --2- 172.21.15.96:0/2924778256 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fde040ccb90 0x7fde040cefa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:19.966 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.964+0000 7fde03e00640 1 --2- 172.21.15.96:0/2924778256 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde04153510 0x7fde041514f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:19.967 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.964+0000 7fde09600640 1 -- 172.21.15.96:0/2924778256 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fde040ccb90 msgr2=0x7fde040cefa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:19.967 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.964+0000 7fde09600640 1 --2- 172.21.15.96:0/2924778256 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fde040ccb90 0x7fde040cefa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:19.967 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.964+0000 7fde09600640 1 -- 172.21.15.96:0/2924778256 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde04153510 msgr2=0x7fde041514f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:19.967 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.964+0000 7fde09600640 1 --2- 172.21.15.96:0/2924778256 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde04153510 0x7fde041514f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:19.967 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.964+0000 7fde09600640 1 -- 172.21.15.96:0/2924778256 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde040cdd50 con 0x7fde0415d270 2024-05-28T06:14:19.967 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.964+0000 7fde03e00640 1 --2- 172.21.15.96:0/2924778256 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde04153510 0x7fde041514f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:14:19.967 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.964+0000 7fde09600640 1 --2- 172.21.15.96:0/2924778256 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fde0415d270 0x7fde040ce510 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7fddfc034040 tx=0x7fddfc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:19.967 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.964+0000 7fde00c00640 1 -- 172.21.15.96:0/2924778256 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddfc041020 con 0x7fde0415d270 2024-05-28T06:14:19.967 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.965+0000 7fde00c00640 1 -- 172.21.15.96:0/2924778256 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fddfc037040 con 0x7fde0415d270 2024-05-28T06:14:19.967 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.965+0000 7fde00c00640 1 -- 172.21.15.96:0/2924778256 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddfc036050 con 0x7fde0415d270 2024-05-28T06:14:19.967 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.965+0000 7fde0a000640 1 -- 172.21.15.96:0/2924778256 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde040cf680 con 0x7fde0415d270 2024-05-28T06:14:19.967 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.965+0000 7fde0a000640 1 -- 172.21.15.96:0/2924778256 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde040cfc40 con 0x7fde0415d270 2024-05-28T06:14:19.968 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.966+0000 7fde0a000640 1 -- 172.21.15.96:0/2924778256 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde040689e0 con 0x7fde0415d270 2024-05-28T06:14:19.968 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.967+0000 7fde00c00640 1 -- 172.21.15.96:0/2924778256 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 21) ==== 92111+0+0 (secure 0 0 0) 0x7fddfc007bd0 con 0x7fde0415d270 2024-05-28T06:14:19.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.968+0000 7fde00c00640 1 --2- 172.21.15.96:0/2924778256 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdde406d4c0 0x7fdde406f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:19.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.968+0000 7fde00c00640 1 -- 172.21.15.96:0/2924778256 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5737+0+0 (secure 0 0 0) 0x7fddfc0aac30 con 0x7fde0415d270 2024-05-28T06:14:19.970 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.969+0000 7fde08c00640 1 --2- 172.21.15.96:0/2924778256 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdde406d4c0 0x7fdde406f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:19.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.970+0000 7fde08c00640 1 --2- 172.21.15.96:0/2924778256 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdde406d4c0 0x7fdde406f980 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fddf8009560 tx=0x7fddf8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:19.974 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:19.972+0000 7fde00c00640 1 -- 172.21.15.96:0/2924778256 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fddfc04b060 con 0x7fde0415d270 2024-05-28T06:14:20.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.186+0000 7fde0a000640 1 -- 172.21.15.96:0/2924778256 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume ls", "vol_name": "cephfs", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fde040b3c40 con 0x7fdde406d4c0 2024-05-28T06:14:20.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.204+0000 7fde00c00640 1 -- 172.21.15.96:0/2924778256 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+381 (secure 0 0 0) 0x7fde040b3c40 con 0x7fdde406d4c0 2024-05-28T06:14:20.205 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:14:20.205 INFO:teuthology.orchestra.run.smithi096.stdout:[ 2024-05-28T06:14:20.206 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:14:20.206 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "subvol_clone81713" 2024-05-28T06:14:20.206 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:14:20.206 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:14:20.206 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "sv1" 2024-05-28T06:14:20.206 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:14:20.206 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:14:20.206 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "subvol_clone81711" 2024-05-28T06:14:20.206 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:14:20.206 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:14:20.206 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "subvol_clone81715" 2024-05-28T06:14:20.206 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:14:20.206 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:14:20.206 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "subvol_clone81717" 2024-05-28T06:14:20.207 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:14:20.207 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:14:20.207 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "subvol_clone81714" 2024-05-28T06:14:20.207 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:14:20.207 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:14:20.207 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "subvol_clone81712" 2024-05-28T06:14:20.207 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:14:20.207 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:14:20.207 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "subvol_clone81716" 2024-05-28T06:14:20.207 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:14:20.207 INFO:teuthology.orchestra.run.smithi096.stdout:] 2024-05-28T06:14:20.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.206+0000 7fde0a000640 1 -- 172.21.15.96:0/2924778256 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdde406d4c0 msgr2=0x7fdde406f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:20.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.206+0000 7fde0a000640 1 --2- 172.21.15.96:0/2924778256 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdde406d4c0 0x7fdde406f980 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fddf8009560 tx=0x7fddf8008040 comp rx=0 tx=0).stop 2024-05-28T06:14:20.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.206+0000 7fde0a000640 1 -- 172.21.15.96:0/2924778256 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fde0415d270 msgr2=0x7fde040ce510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:20.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.206+0000 7fde0a000640 1 --2- 172.21.15.96:0/2924778256 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fde0415d270 0x7fde040ce510 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7fddfc034040 tx=0x7fddfc004a40 comp rx=0 tx=0).stop 2024-05-28T06:14:20.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.206+0000 7fde0a000640 1 -- 172.21.15.96:0/2924778256 shutdown_connections 2024-05-28T06:14:20.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.206+0000 7fde0a000640 1 --2- 172.21.15.96:0/2924778256 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdde406d4c0 0x7fdde406f980 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:20.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.206+0000 7fde0a000640 1 --2- 172.21.15.96:0/2924778256 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fde0415d270 0x7fde040ce510 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:20.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.206+0000 7fde0a000640 1 --2- 172.21.15.96:0/2924778256 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde04153510 0x7fde041514f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:20.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.206+0000 7fde0a000640 1 --2- 172.21.15.96:0/2924778256 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fde040ccb90 0x7fde040cefa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:20.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.206+0000 7fde0a000640 1 -- 172.21.15.96:0/2924778256 >> 172.21.15.96:0/2924778256 conn(0x7fde0406b900 msgr2=0x7fde040cb2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:20.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.206+0000 7fde0a000640 1 -- 172.21.15.96:0/2924778256 shutdown_connections 2024-05-28T06:14:20.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.206+0000 7fde0a000640 1 -- 172.21.15.96:0/2924778256 wait complete. 2024-05-28T06:14:20.225 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone81713 2024-05-28T06:14:20.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.425+0000 7fea5b400640 1 Processor -- start 2024-05-28T06:14:20.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.425+0000 7fea5b400640 1 -- start start 2024-05-28T06:14:20.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.426+0000 7fea5b400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fea54072310 0x7fea5406dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:20.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.426+0000 7fea5b400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fea540b3510 0x7fea5406f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:20.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.426+0000 7fea5b400640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea5406e190 con 0x7fea5415e830 2024-05-28T06:14:20.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.426+0000 7fea5b400640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fea5406f770 con 0x7fea54072310 2024-05-28T06:14:20.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.426+0000 7fea5b400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fea5406e470 con 0x7fea540b3510 2024-05-28T06:14:20.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.427+0000 7fea59400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fea540b3510 0x7fea5406f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:20.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.427+0000 7fea59400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fea540b3510 0x7fea5406f230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.96:53114/0 (socket says 172.21.15.96:53114) 2024-05-28T06:14:20.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.427+0000 7fea59400640 1 -- 172.21.15.96:0/1263224826 learned_addr learned my addr 172.21.15.96:0/1263224826 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:20.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.428+0000 7fea53e00640 1 --2- 172.21.15.96:0/1263224826 >> v2:172.21.15.123:3300/0 conn(0x7fea54072310 0x7fea5406dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:20.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.428+0000 7fea59400640 1 -- 172.21.15.96:0/1263224826 >> v2:172.21.15.123:3300/0 conn(0x7fea54072310 msgr2=0x7fea5406dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:20.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.428+0000 7fea59400640 1 --2- 172.21.15.96:0/1263224826 >> v2:172.21.15.123:3300/0 conn(0x7fea54072310 0x7fea5406dbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:20.430 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.428+0000 7fea59400640 1 -- 172.21.15.96:0/1263224826 >> v1:172.21.15.123:6789/0 conn(0x7fea5415e830 legacy=0x7fea54071c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:20.430 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.428+0000 7fea59400640 1 -- 172.21.15.96:0/1263224826 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea5406e750 con 0x7fea540b3510 2024-05-28T06:14:20.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.429+0000 7fea59400640 1 --2- 172.21.15.96:0/1263224826 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fea540b3510 0x7fea5406f230 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fea4c009e50 tx=0x7fea4c02a730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b24e2e890c6c52b0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:20.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.430+0000 7fea53400640 1 -- 172.21.15.96:0/1263224826 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea4c036070 con 0x7fea540b3510 2024-05-28T06:14:20.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.430+0000 7fea53400640 1 -- 172.21.15.96:0/1263224826 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea4c031600 con 0x7fea540b3510 2024-05-28T06:14:20.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.430+0000 7fea53400640 1 -- 172.21.15.96:0/1263224826 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea4c0319d0 con 0x7fea540b3510 2024-05-28T06:14:20.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.431+0000 7fea5b400640 1 -- 172.21.15.96:0/1263224826 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fea540b3510 msgr2=0x7fea5406f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:20.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.431+0000 7fea5b400640 1 --2- 172.21.15.96:0/1263224826 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fea540b3510 0x7fea5406f230 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fea4c009e50 tx=0x7fea4c02a730 comp rx=0 tx=0).stop 2024-05-28T06:14:20.433 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.431+0000 7fea5b400640 1 -- 172.21.15.96:0/1263224826 shutdown_connections 2024-05-28T06:14:20.433 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.431+0000 7fea5b400640 1 --2- 172.21.15.96:0/1263224826 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fea540b3510 0x7fea5406f230 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:20.433 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.431+0000 7fea5b400640 1 --2- 172.21.15.96:0/1263224826 >> v2:172.21.15.123:3300/0 conn(0x7fea54072310 0x7fea5406dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:20.433 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.431+0000 7fea5b400640 1 -- 172.21.15.96:0/1263224826 >> 172.21.15.96:0/1263224826 conn(0x7fea540b1e50 msgr2=0x7fea540b2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:20.433 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.431+0000 7fea5b400640 1 -- 172.21.15.96:0/1263224826 shutdown_connections 2024-05-28T06:14:20.433 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.431+0000 7fea5b400640 1 -- 172.21.15.96:0/1263224826 wait complete. 2024-05-28T06:14:20.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.433+0000 7fea5b400640 1 Processor -- start 2024-05-28T06:14:20.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.433+0000 7fea5b400640 1 -- start start 2024-05-28T06:14:20.436 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.434+0000 7fea5b400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fea54072310 0x7fea5409a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:20.436 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.434+0000 7fea5b400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fea540b3510 0x7fea540aee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:20.436 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.434+0000 7fea5b400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fea5415e830 0x7fea54148580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:20.436 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.434+0000 7fea5b400640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fea5415ec10 con 0x7fea5415e830 2024-05-28T06:14:20.436 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.434+0000 7fea5b400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fea5406ed10 con 0x7fea54072310 2024-05-28T06:14:20.436 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.434+0000 7fea5b400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fea540728c0 con 0x7fea540b3510 2024-05-28T06:14:20.436 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.434+0000 7fea59400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fea5415e830 0x7fea54148580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:20.436 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.434+0000 7fea59400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fea5415e830 0x7fea54148580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48540/0 (socket says 172.21.15.96:48540) 2024-05-28T06:14:20.436 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.434+0000 7fea59400640 1 -- 172.21.15.96:0/259551006 learned_addr learned my addr 172.21.15.96:0/259551006 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:20.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.434+0000 7fea53e00640 1 --2- 172.21.15.96:0/259551006 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fea540b3510 0x7fea540aee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:20.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.434+0000 7fea59400640 1 -- 172.21.15.96:0/259551006 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fea540b3510 msgr2=0x7fea540aee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:20.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.434+0000 7fea59400640 1 --2- 172.21.15.96:0/259551006 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fea540b3510 0x7fea540aee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:20.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.434+0000 7fea59400640 1 -- 172.21.15.96:0/259551006 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fea54072310 msgr2=0x7fea5409a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:20.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.434+0000 7fea58a00640 1 --2- 172.21.15.96:0/259551006 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fea54072310 0x7fea5409a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:20.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.435+0000 7fea59400640 1 --2- 172.21.15.96:0/259551006 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fea54072310 0x7fea5409a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:20.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.435+0000 7fea59400640 1 -- 172.21.15.96:0/259551006 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea5414b740 con 0x7fea5415e830 2024-05-28T06:14:20.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.435+0000 7fea59400640 1 --2- 172.21.15.96:0/259551006 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fea5415e830 0x7fea54148580 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fea4c009f70 tx=0x7fea4c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:20.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.435+0000 7fea50c00640 1 -- 172.21.15.96:0/259551006 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea4c041020 con 0x7fea5415e830 2024-05-28T06:14:20.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.435+0000 7fea50c00640 1 -- 172.21.15.96:0/259551006 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea4c008730 con 0x7fea5415e830 2024-05-28T06:14:20.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.435+0000 7fea5b400640 1 -- 172.21.15.96:0/259551006 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea5414ba20 con 0x7fea5415e830 2024-05-28T06:14:20.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.435+0000 7fea50c00640 1 -- 172.21.15.96:0/259551006 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea4c036040 con 0x7fea5415e830 2024-05-28T06:14:20.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.435+0000 7fea5b400640 1 -- 172.21.15.96:0/259551006 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea5414bfe0 con 0x7fea5415e830 2024-05-28T06:14:20.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.436+0000 7fea5b400640 1 -- 172.21.15.96:0/259551006 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea540689e0 con 0x7fea5415e830 2024-05-28T06:14:20.439 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.437+0000 7fea50c00640 1 -- 172.21.15.96:0/259551006 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 21) ==== 92111+0+0 (secure 0 0 0) 0x7fea4c044020 con 0x7fea5415e830 2024-05-28T06:14:20.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.438+0000 7fea50c00640 1 --2- 172.21.15.96:0/259551006 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fea3406d4c0 0x7fea3406f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:20.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.438+0000 7fea50c00640 1 -- 172.21.15.96:0/259551006 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5737+0+0 (secure 0 0 0) 0x7fea4c0ab0f0 con 0x7fea5415e830 2024-05-28T06:14:20.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.439+0000 7fea58a00640 1 --2- 172.21.15.96:0/259551006 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fea3406d4c0 0x7fea3406f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:20.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.440+0000 7fea58a00640 1 --2- 172.21.15.96:0/259551006 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fea3406d4c0 0x7fea3406f980 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fea48000f60 tx=0x7fea48009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:20.444 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.443+0000 7fea50c00640 1 -- 172.21.15.96:0/259551006 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fea4c0b3020 con 0x7fea5415e830 2024-05-28T06:14:20.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.673+0000 7fea5b400640 1 -- 172.21.15.96:0/259551006 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone81713", "target": ["mon-mgr", ""]}) -- 0x7fea5409a7f0 con 0x7fea3406d4c0 2024-05-28T06:14:20.714 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.712+0000 7fea50c00640 1 -- 172.21.15.96:0/259551006 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fea5409a7f0 con 0x7fea3406d4c0 2024-05-28T06:14:20.717 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.715+0000 7fea5b400640 1 -- 172.21.15.96:0/259551006 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fea3406d4c0 msgr2=0x7fea3406f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:20.717 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.715+0000 7fea5b400640 1 --2- 172.21.15.96:0/259551006 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fea3406d4c0 0x7fea3406f980 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fea48000f60 tx=0x7fea48009040 comp rx=0 tx=0).stop 2024-05-28T06:14:20.717 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.715+0000 7fea5b400640 1 -- 172.21.15.96:0/259551006 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fea5415e830 msgr2=0x7fea54148580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:20.717 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.715+0000 7fea5b400640 1 --2- 172.21.15.96:0/259551006 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fea5415e830 0x7fea54148580 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fea4c009f70 tx=0x7fea4c004a40 comp rx=0 tx=0).stop 2024-05-28T06:14:20.717 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.715+0000 7fea5b400640 1 -- 172.21.15.96:0/259551006 shutdown_connections 2024-05-28T06:14:20.717 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.715+0000 7fea5b400640 1 --2- 172.21.15.96:0/259551006 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fea3406d4c0 0x7fea3406f980 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:20.718 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.715+0000 7fea5b400640 1 --2- 172.21.15.96:0/259551006 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fea5415e830 0x7fea54148580 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:20.718 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.715+0000 7fea5b400640 1 --2- 172.21.15.96:0/259551006 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fea540b3510 0x7fea540aee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:20.718 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.715+0000 7fea5b400640 1 --2- 172.21.15.96:0/259551006 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fea54072310 0x7fea5409a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:20.718 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.715+0000 7fea5b400640 1 -- 172.21.15.96:0/259551006 >> 172.21.15.96:0/259551006 conn(0x7fea540b1e50 msgr2=0x7fea54162410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:20.718 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.716+0000 7fea5b400640 1 -- 172.21.15.96:0/259551006 shutdown_connections 2024-05-28T06:14:20.718 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.716+0000 7fea5b400640 1 -- 172.21.15.96:0/259551006 wait complete. 2024-05-28T06:14:20.723 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:14:20.722+0000 7fea5da00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:14:20.723 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:14:20.722+0000 7fea5da00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:14:20.724 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:14:20.722+0000 7fea5da00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:14:20.724 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:14:20.722+0000 7fea5da00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:14:20.724 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:14:20.722+0000 7fea5da00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:14:20.737 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot ls cephfs sv1 --format json 2024-05-28T06:14:20.754 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:14:20.753+0000 7fea5d000640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:14:20.754 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:14:20.753+0000 7fea5d000640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:14:20.755 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:14:20.753+0000 7fea5d000640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:14:20.755 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:14:20.753+0000 7fea5d000640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:14:20.755 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:14:20.753+0000 7fea5d000640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:14:20.922 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.920+0000 7f00e3200640 1 Processor -- start 2024-05-28T06:14:20.922 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.921+0000 7f00e3200640 1 -- start start 2024-05-28T06:14:20.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.922+0000 7f00e3200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f00e4162940 0x7f00e406e560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:20.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.922+0000 7f00e3200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f00e40cf800 0x7f00e40cfc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:20.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.922+0000 7f00e3200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00e40b49f0 con 0x7f00e4162d20 2024-05-28T06:14:20.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.922+0000 7f00e3200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f00e40b4cd0 con 0x7f00e4162940 2024-05-28T06:14:20.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.922+0000 7f00e3200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f00e40cf020 con 0x7f00e40cf800 2024-05-28T06:14:20.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.922+0000 7f00e1400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f00e4162d20 0x7f00e40b4210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:54810/0 (socket says 172.21.15.96:54810) 2024-05-28T06:14:20.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.922+0000 7f00e1400640 1 -- 172.21.15.96:0/1772450040 learned_addr learned my addr 172.21.15.96:0/1772450040 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:14:20.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.923+0000 7f00e1e00640 1 --2- 172.21.15.96:0/1772450040 >> v2:172.21.15.123:3300/0 conn(0x7f00e4162940 0x7f00e406e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:20.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.923+0000 7f00e0a00640 1 -- 172.21.15.96:0/1772450040 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1835351790 0 0) 0x7f00e40b49f0 con 0x7f00e4162d20 2024-05-28T06:14:20.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.923+0000 7f00e0a00640 1 -- 172.21.15.96:0/1772450040 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f00c8003520 con 0x7f00e4162d20 2024-05-28T06:14:20.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.923+0000 7f00e2800640 1 --2- 172.21.15.96:0/1772450040 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f00e40cf800 0x7f00e40cfc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:20.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.923+0000 7f00e0a00640 1 -- 172.21.15.96:0/1772450040 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1697746469 0 0) 0x7f00c8003520 con 0x7f00e4162d20 2024-05-28T06:14:20.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.923+0000 7f00e0a00640 1 -- 172.21.15.96:0/1772450040 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f00e40cf800 msgr2=0x7f00e40cfc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:20.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.923+0000 7f00e0a00640 1 --2- 172.21.15.96:0/1772450040 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f00e40cf800 0x7f00e40cfc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:20.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.923+0000 7f00e0a00640 1 -- 172.21.15.96:0/1772450040 >> v2:172.21.15.123:3300/0 conn(0x7f00e4162940 msgr2=0x7f00e406e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:20.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.923+0000 7f00e0a00640 1 --2- 172.21.15.96:0/1772450040 >> v2:172.21.15.123:3300/0 conn(0x7f00e4162940 0x7f00e406e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:20.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.923+0000 7f00e0a00640 1 -- 172.21.15.96:0/1772450040 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00e40cecd0 con 0x7f00e4162d20 2024-05-28T06:14:20.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.923+0000 7f00e0a00640 1 -- 172.21.15.96:0/1772450040 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f00d40019f0 con 0x7f00e4162d20 2024-05-28T06:14:20.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.923+0000 7f00e0a00640 1 -- 172.21.15.96:0/1772450040 >> v1:172.21.15.96:6789/0 conn(0x7f00e4162d20 legacy=0x7f00e40b4210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:20.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.923+0000 7f00e0a00640 1 --2- 172.21.15.96:0/1772450040 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f00e40cf800 0x7f00c80039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:20.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.923+0000 7f00e0a00640 1 --2- 172.21.15.96:0/1772450040 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00c8003f80 0x7f00c8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:20.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.923+0000 7f00e0a00640 1 --2- 172.21.15.96:0/1772450040 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f00c8004d40 0x7f00c80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:20.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.923+0000 7f00e0a00640 1 -- 172.21.15.96:0/1772450040 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f00e40b4cd0 con 0x7f00c8003f80 2024-05-28T06:14:20.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.923+0000 7f00e0a00640 1 -- 172.21.15.96:0/1772450040 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f00e40cf020 con 0x7f00c8004d40 2024-05-28T06:14:20.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.923+0000 7f00e0a00640 1 -- 172.21.15.96:0/1772450040 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f00c800bba0 con 0x7f00e40cf800 2024-05-28T06:14:20.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.924+0000 7f00e1400640 1 --2- 172.21.15.96:0/1772450040 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00c8003f80 0x7f00c8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:20.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.924+0000 7f00e1400640 1 -- 172.21.15.96:0/1772450040 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f00e40cf800 msgr2=0x7f00c80039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:20.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.924+0000 7f00e1e00640 1 --2- 172.21.15.96:0/1772450040 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f00e40cf800 0x7f00c80039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:20.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.924+0000 7f00e2800640 1 --2- 172.21.15.96:0/1772450040 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f00c8004d40 0x7f00c80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:20.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.924+0000 7f00e1400640 1 --2- 172.21.15.96:0/1772450040 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f00e40cf800 0x7f00c80039b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:20.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.924+0000 7f00e1400640 1 -- 172.21.15.96:0/1772450040 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f00c8004d40 msgr2=0x7f00c80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:20.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.924+0000 7f00e1400640 1 --2- 172.21.15.96:0/1772450040 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f00c8004d40 0x7f00c80051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:20.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.924+0000 7f00e1400640 1 -- 172.21.15.96:0/1772450040 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00c8003520 con 0x7f00c8003f80 2024-05-28T06:14:20.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.925+0000 7f00e1400640 1 --2- 172.21.15.96:0/1772450040 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00c8003f80 0x7f00c8004800 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f00d40013d0 tx=0x7f00c8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7816d55265c4a4c6 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:20.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.925+0000 7f00e0a00640 1 -- 172.21.15.96:0/1772450040 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00d400a410 con 0x7f00c8003f80 2024-05-28T06:14:20.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.925+0000 7f00e0a00640 1 -- 172.21.15.96:0/1772450040 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00d402f070 con 0x7f00c8003f80 2024-05-28T06:14:20.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.925+0000 7f00e0a00640 1 -- 172.21.15.96:0/1772450040 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00d402bba0 con 0x7f00c8003f80 2024-05-28T06:14:20.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.926+0000 7f00e3200640 1 -- 172.21.15.96:0/1772450040 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00c8003f80 msgr2=0x7f00c8004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:20.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.926+0000 7f00e3200640 1 --2- 172.21.15.96:0/1772450040 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00c8003f80 0x7f00c8004800 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f00d40013d0 tx=0x7f00c8005830 comp rx=0 tx=0).stop 2024-05-28T06:14:20.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.926+0000 7f00e2800640 1 -- 172.21.15.96:0/1772450040 reap_dead start 2024-05-28T06:14:20.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.926+0000 7f00e3200640 1 -- 172.21.15.96:0/1772450040 shutdown_connections 2024-05-28T06:14:20.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.926+0000 7f00e3200640 1 -- 172.21.15.96:0/1772450040 >> 172.21.15.96:0/1772450040 conn(0x7f00e406bae0 msgr2=0x7f00e406bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:20.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.926+0000 7f00e3200640 1 -- 172.21.15.96:0/1772450040 shutdown_connections 2024-05-28T06:14:20.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.926+0000 7f00e3200640 1 -- 172.21.15.96:0/1772450040 wait complete. 2024-05-28T06:14:20.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.928+0000 7f00e3200640 1 Processor -- start 2024-05-28T06:14:20.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.928+0000 7f00e3200640 1 -- start start 2024-05-28T06:14:20.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.929+0000 7f00e3200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f00c8003f80 0x7f00e409da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:20.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.929+0000 7f00e3200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00e41f8e20 0x7f00e41f7ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:20.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.929+0000 7f00e3200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f00e4160120 0x7f00e4160580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:20.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.929+0000 7f00e3200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f00e40d0020 con 0x7f00e41f8e20 2024-05-28T06:14:20.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.929+0000 7f00e3200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f00e40cfea0 con 0x7f00c8003f80 2024-05-28T06:14:20.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.929+0000 7f00e3200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f00e4071b40 con 0x7f00e4160120 2024-05-28T06:14:20.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.929+0000 7f00e1400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00e41f8e20 0x7f00e41f7ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:20.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.929+0000 7f00e1400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00e41f8e20 0x7f00e41f7ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48570/0 (socket says 172.21.15.96:48570) 2024-05-28T06:14:20.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.929+0000 7f00e1400640 1 -- 172.21.15.96:0/3150323502 learned_addr learned my addr 172.21.15.96:0/3150323502 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:20.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.930+0000 7f00e2800640 1 --2- 172.21.15.96:0/3150323502 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f00e4160120 0x7f00e4160580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:20.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.930+0000 7f00e1e00640 1 --2- 172.21.15.96:0/3150323502 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f00c8003f80 0x7f00e409da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:20.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.930+0000 7f00e1400640 1 -- 172.21.15.96:0/3150323502 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f00e4160120 msgr2=0x7f00e4160580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:20.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.930+0000 7f00e1400640 1 --2- 172.21.15.96:0/3150323502 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f00e4160120 0x7f00e4160580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:20.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.930+0000 7f00e1400640 1 -- 172.21.15.96:0/3150323502 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f00c8003f80 msgr2=0x7f00e409da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:20.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.930+0000 7f00e1400640 1 --2- 172.21.15.96:0/3150323502 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f00c8003f80 0x7f00e409da10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:20.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.930+0000 7f00e1400640 1 -- 172.21.15.96:0/3150323502 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00e41f92a0 con 0x7f00e41f8e20 2024-05-28T06:14:20.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.930+0000 7f00e1400640 1 --2- 172.21.15.96:0/3150323502 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00e41f8e20 0x7f00e41f7ab0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f00d4001330 tx=0x7f00d4001360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:20.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.930+0000 7f00da000640 1 -- 172.21.15.96:0/3150323502 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00d402bb70 con 0x7f00e41f8e20 2024-05-28T06:14:20.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.930+0000 7f00da000640 1 -- 172.21.15.96:0/3150323502 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00d402bd30 con 0x7f00e41f8e20 2024-05-28T06:14:20.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.930+0000 7f00da000640 1 -- 172.21.15.96:0/3150323502 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00d40346b0 con 0x7f00e41f8e20 2024-05-28T06:14:20.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.930+0000 7f00e3200640 1 -- 172.21.15.96:0/3150323502 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00e409e140 con 0x7f00e41f8e20 2024-05-28T06:14:20.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.931+0000 7f00e3200640 1 -- 172.21.15.96:0/3150323502 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00e41f99c0 con 0x7f00e41f8e20 2024-05-28T06:14:20.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.931+0000 7f00e3200640 1 -- 172.21.15.96:0/3150323502 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00e41f94d0 con 0x7f00e41f8e20 2024-05-28T06:14:20.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.932+0000 7f00da000640 1 -- 172.21.15.96:0/3150323502 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 21) ==== 92111+0+0 (secure 0 0 0) 0x7f00d4034850 con 0x7f00e41f8e20 2024-05-28T06:14:20.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.934+0000 7f00da000640 1 --2- 172.21.15.96:0/3150323502 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f00c006d4f0 0x7f00c006f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:20.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.934+0000 7f00da000640 1 -- 172.21.15.96:0/3150323502 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5737+0+0 (secure 0 0 0) 0x7f00d40aabb0 con 0x7f00e41f8e20 2024-05-28T06:14:20.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.935+0000 7f00e1e00640 1 --2- 172.21.15.96:0/3150323502 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f00c006d4f0 0x7f00c006f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:20.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.936+0000 7f00e1e00640 1 --2- 172.21.15.96:0/3150323502 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f00c006d4f0 0x7f00c006f9b0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f00cc009a10 tx=0x7f00cc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:20.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:20.938+0000 7f00da000640 1 -- 172.21.15.96:0/3150323502 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f00d407a7d0 con 0x7f00e41f8e20 2024-05-28T06:14:21.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.153+0000 7f00e3200640 1 -- 172.21.15.96:0/3150323502 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot ls", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f00e40689e0 con 0x7f00c006d4f0 2024-05-28T06:14:21.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.180+0000 7f00da000640 1 -- 172.21.15.96:0/3150323502 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f00e40689e0 con 0x7f00c006d4f0 2024-05-28T06:14:21.181 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:14:21.182 INFO:teuthology.orchestra.run.smithi096.stdout:[ 2024-05-28T06:14:21.182 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:14:21.182 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "ss1" 2024-05-28T06:14:21.182 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:14:21.182 INFO:teuthology.orchestra.run.smithi096.stdout:] 2024-05-28T06:14:21.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.182+0000 7f00e3200640 1 -- 172.21.15.96:0/3150323502 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f00c006d4f0 msgr2=0x7f00c006f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:21.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.182+0000 7f00e3200640 1 --2- 172.21.15.96:0/3150323502 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f00c006d4f0 0x7f00c006f9b0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f00cc009a10 tx=0x7f00cc008040 comp rx=0 tx=0).stop 2024-05-28T06:14:21.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.182+0000 7f00e3200640 1 -- 172.21.15.96:0/3150323502 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00e41f8e20 msgr2=0x7f00e41f7ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:21.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.182+0000 7f00e3200640 1 --2- 172.21.15.96:0/3150323502 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00e41f8e20 0x7f00e41f7ab0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f00d4001330 tx=0x7f00d4001360 comp rx=0 tx=0).stop 2024-05-28T06:14:21.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.182+0000 7f00e3200640 1 -- 172.21.15.96:0/3150323502 shutdown_connections 2024-05-28T06:14:21.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.182+0000 7f00e3200640 1 --2- 172.21.15.96:0/3150323502 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f00c006d4f0 0x7f00c006f9b0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:21.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.182+0000 7f00e3200640 1 --2- 172.21.15.96:0/3150323502 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f00e4160120 0x7f00e4160580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:21.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.182+0000 7f00e3200640 1 --2- 172.21.15.96:0/3150323502 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00e41f8e20 0x7f00e41f7ab0 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:21.185 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.182+0000 7f00e3200640 1 --2- 172.21.15.96:0/3150323502 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f00c8003f80 0x7f00e409da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:21.185 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.182+0000 7f00e3200640 1 -- 172.21.15.96:0/3150323502 >> 172.21.15.96:0/3150323502 conn(0x7f00e406bae0 msgr2=0x7f00e40b37e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:21.185 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.183+0000 7f00e3200640 1 -- 172.21.15.96:0/3150323502 shutdown_connections 2024-05-28T06:14:21.185 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.183+0000 7f00e3200640 1 -- 172.21.15.96:0/3150323502 wait complete. 2024-05-28T06:14:21.201 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-05-28T06:14:21.395 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.394+0000 7ff23b000640 1 Processor -- start 2024-05-28T06:14:21.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.394+0000 7ff23b000640 1 -- start start 2024-05-28T06:14:21.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.395+0000 7ff23b000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff2340d0640 0x7ff2340d0a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:21.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.395+0000 7ff23b000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7ff2340725e0 0x7ff2340ae650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:21.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.395+0000 7ff23b000640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2340ade90 con 0x7ff23415b6f0 2024-05-28T06:14:21.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.395+0000 7ff23b000640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7ff234071c40 con 0x7ff2340725e0 2024-05-28T06:14:21.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.395+0000 7ff23b000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ff2340d0f60 con 0x7ff2340d0640 2024-05-28T06:14:21.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.396+0000 7ff239000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7ff2340725e0 0x7ff2340ae650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:21.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.396+0000 7ff233400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7ff23415b6f0 0x7ff23406e350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:54818/0 (socket says 172.21.15.96:54818) 2024-05-28T06:14:21.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.396+0000 7ff239000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7ff2340725e0 0x7ff2340ae650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48582/0 (socket says 172.21.15.96:48582) 2024-05-28T06:14:21.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.396+0000 7ff233400640 1 -- 172.21.15.96:0/248739841 learned_addr learned my addr 172.21.15.96:0/248739841 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:14:21.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.396+0000 7ff233e00640 1 --2- 172.21.15.96:0/248739841 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff2340d0640 0x7ff2340d0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:21.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.396+0000 7ff239000640 1 -- 172.21.15.96:0/248739841 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff2340d0640 msgr2=0x7ff2340d0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:21.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.396+0000 7ff239000640 1 --2- 172.21.15.96:0/248739841 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff2340d0640 0x7ff2340d0a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:21.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.396+0000 7ff239000640 1 -- 172.21.15.96:0/248739841 >> v1:172.21.15.96:6789/0 conn(0x7ff23415b6f0 legacy=0x7ff23406e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:21.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.396+0000 7ff239000640 1 -- 172.21.15.96:0/248739841 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff234071e70 con 0x7ff2340725e0 2024-05-28T06:14:21.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.396+0000 7ff232a00640 1 -- 172.21.15.96:0/248739841 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4092212236 0 0) 0x7ff2340ade90 con 0x7ff23415b6f0 2024-05-28T06:14:21.400 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.397+0000 7ff239000640 1 --2- 172.21.15.96:0/248739841 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff2340725e0 0x7ff2340ae650 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7ff22c009d90 tx=0x7ff22c02a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a50c69760f909499 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:21.400 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.398+0000 7ff232a00640 1 -- 172.21.15.96:0/248739841 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff22c007740 con 0x7ff2340725e0 2024-05-28T06:14:21.400 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.398+0000 7ff232a00640 1 -- 172.21.15.96:0/248739841 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff22c0078e0 con 0x7ff2340725e0 2024-05-28T06:14:21.400 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.398+0000 7ff232a00640 1 -- 172.21.15.96:0/248739841 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff22c007cb0 con 0x7ff2340725e0 2024-05-28T06:14:21.400 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.398+0000 7ff23b000640 1 -- 172.21.15.96:0/248739841 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff2340725e0 msgr2=0x7ff2340ae650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:21.400 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.398+0000 7ff23b000640 1 --2- 172.21.15.96:0/248739841 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff2340725e0 0x7ff2340ae650 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7ff22c009d90 tx=0x7ff22c02a660 comp rx=0 tx=0).stop 2024-05-28T06:14:21.400 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.398+0000 7ff23b000640 1 -- 172.21.15.96:0/248739841 shutdown_connections 2024-05-28T06:14:21.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.398+0000 7ff23b000640 1 --2- 172.21.15.96:0/248739841 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff2340725e0 0x7ff2340ae650 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:21.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.398+0000 7ff23b000640 1 --2- 172.21.15.96:0/248739841 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff2340d0640 0x7ff2340d0a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:21.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.398+0000 7ff23b000640 1 -- 172.21.15.96:0/248739841 >> 172.21.15.96:0/248739841 conn(0x7ff23406bd20 msgr2=0x7ff23406c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:21.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.398+0000 7ff23b000640 1 -- 172.21.15.96:0/248739841 shutdown_connections 2024-05-28T06:14:21.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.399+0000 7ff23b000640 1 -- 172.21.15.96:0/248739841 wait complete. 2024-05-28T06:14:21.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.400+0000 7ff23b000640 1 Processor -- start 2024-05-28T06:14:21.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.400+0000 7ff23b000640 1 -- start start 2024-05-28T06:14:21.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.401+0000 7ff23b000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff2340725e0 0x7ff234151650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:21.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.401+0000 7ff23b000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff2340d0640 0x7ff2340b4410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:21.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.401+0000 7ff23b000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff23415b6f0 0x7ff2340b4950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:21.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.401+0000 7ff23b000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff2340aeb90 con 0x7ff2340d0640 2024-05-28T06:14:21.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.401+0000 7ff23b000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff2340ae130 con 0x7ff2340725e0 2024-05-28T06:14:21.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.401+0000 7ff23b000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ff2340aed10 con 0x7ff23415b6f0 2024-05-28T06:14:21.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.402+0000 7ff233400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff2340d0640 0x7ff2340b4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:21.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.402+0000 7ff233400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff2340d0640 0x7ff2340b4410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48596/0 (socket says 172.21.15.96:48596) 2024-05-28T06:14:21.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.402+0000 7ff233400640 1 -- 172.21.15.96:0/3709367410 learned_addr learned my addr 172.21.15.96:0/3709367410 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:21.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.402+0000 7ff239000640 1 --2- 172.21.15.96:0/3709367410 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff23415b6f0 0x7ff2340b4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:21.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.402+0000 7ff233e00640 1 --2- 172.21.15.96:0/3709367410 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff2340725e0 0x7ff234151650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:21.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.402+0000 7ff233400640 1 -- 172.21.15.96:0/3709367410 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff23415b6f0 msgr2=0x7ff2340b4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:21.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.402+0000 7ff233400640 1 --2- 172.21.15.96:0/3709367410 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff23415b6f0 0x7ff2340b4950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:21.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.402+0000 7ff233400640 1 -- 172.21.15.96:0/3709367410 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff2340725e0 msgr2=0x7ff234151650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:21.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.402+0000 7ff233400640 1 --2- 172.21.15.96:0/3709367410 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff2340725e0 0x7ff234151650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:21.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.402+0000 7ff233400640 1 -- 172.21.15.96:0/3709367410 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff234152a90 con 0x7ff2340d0640 2024-05-28T06:14:21.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.402+0000 7ff239000640 1 --2- 172.21.15.96:0/3709367410 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff23415b6f0 0x7ff2340b4950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-28T06:14:21.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.403+0000 7ff233400640 1 --2- 172.21.15.96:0/3709367410 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff2340d0640 0x7ff2340b4410 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7ff22000c430 tx=0x7ff22000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:21.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.403+0000 7ff22be00640 1 -- 172.21.15.96:0/3709367410 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2200074e0 con 0x7ff2340d0640 2024-05-28T06:14:21.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.403+0000 7ff22be00640 1 -- 172.21.15.96:0/3709367410 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2200047e0 con 0x7ff2340d0640 2024-05-28T06:14:21.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.403+0000 7ff22be00640 1 -- 172.21.15.96:0/3709367410 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff220004bd0 con 0x7ff2340d0640 2024-05-28T06:14:21.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.403+0000 7ff23b000640 1 -- 172.21.15.96:0/3709367410 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff234152d70 con 0x7ff2340d0640 2024-05-28T06:14:21.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.403+0000 7ff23b000640 1 -- 172.21.15.96:0/3709367410 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2340ce380 con 0x7ff2340d0640 2024-05-28T06:14:21.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.404+0000 7ff23b000640 1 -- 172.21.15.96:0/3709367410 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2340689e0 con 0x7ff2340d0640 2024-05-28T06:14:21.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.405+0000 7ff22be00640 1 -- 172.21.15.96:0/3709367410 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 21) ==== 92111+0+0 (secure 0 0 0) 0x7ff220011070 con 0x7ff2340d0640 2024-05-28T06:14:21.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.406+0000 7ff22be00640 1 --2- 172.21.15.96:0/3709367410 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff21406d4c0 0x7ff21406f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:21.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.406+0000 7ff22be00640 1 -- 172.21.15.96:0/3709367410 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5737+0+0 (secure 0 0 0) 0x7ff22008a4e0 con 0x7ff2340d0640 2024-05-28T06:14:21.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.407+0000 7ff233e00640 1 --2- 172.21.15.96:0/3709367410 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff21406d4c0 0x7ff21406f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:21.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.408+0000 7ff233e00640 1 --2- 172.21.15.96:0/3709367410 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff21406d4c0 0x7ff21406f980 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7ff224006fd0 tx=0x7ff224008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:21.412 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.411+0000 7ff22be00640 1 -- 172.21.15.96:0/3709367410 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff22005a070 con 0x7ff2340d0640 2024-05-28T06:14:21.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.646+0000 7ff23b000640 1 -- 172.21.15.96:0/3709367410 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff2340afb90 con 0x7ff21406d4c0 2024-05-28T06:14:21.712 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.710+0000 7ff22be00640 1 -- 172.21.15.96:0/3709367410 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7ff2340afb90 con 0x7ff21406d4c0 2024-05-28T06:14:21.712 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:14:21.714 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.713+0000 7ff23b000640 1 -- 172.21.15.96:0/3709367410 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff21406d4c0 msgr2=0x7ff21406f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:21.714 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.713+0000 7ff23b000640 1 --2- 172.21.15.96:0/3709367410 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff21406d4c0 0x7ff21406f980 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7ff224006fd0 tx=0x7ff224008040 comp rx=0 tx=0).stop 2024-05-28T06:14:21.715 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.713+0000 7ff23b000640 1 -- 172.21.15.96:0/3709367410 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff2340d0640 msgr2=0x7ff2340b4410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:21.715 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.713+0000 7ff23b000640 1 --2- 172.21.15.96:0/3709367410 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff2340d0640 0x7ff2340b4410 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7ff22000c430 tx=0x7ff22000c8f0 comp rx=0 tx=0).stop 2024-05-28T06:14:21.715 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.713+0000 7ff23b000640 1 -- 172.21.15.96:0/3709367410 shutdown_connections 2024-05-28T06:14:21.715 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.713+0000 7ff23b000640 1 --2- 172.21.15.96:0/3709367410 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff21406d4c0 0x7ff21406f980 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:21.716 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.713+0000 7ff23b000640 1 --2- 172.21.15.96:0/3709367410 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff23415b6f0 0x7ff2340b4950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:21.716 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.713+0000 7ff23b000640 1 --2- 172.21.15.96:0/3709367410 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff2340d0640 0x7ff2340b4410 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:21.716 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.713+0000 7ff23b000640 1 --2- 172.21.15.96:0/3709367410 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff2340725e0 0x7ff234151650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:21.716 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.713+0000 7ff23b000640 1 -- 172.21.15.96:0/3709367410 >> 172.21.15.96:0/3709367410 conn(0x7ff23406bd20 msgr2=0x7ff234072c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:21.716 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.713+0000 7ff23b000640 1 -- 172.21.15.96:0/3709367410 shutdown_connections 2024-05-28T06:14:21.716 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.713+0000 7ff23b000640 1 -- 172.21.15.96:0/3709367410 wait complete. 2024-05-28T06:14:21.734 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs sv1 2024-05-28T06:14:21.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.933+0000 7f4055a00640 1 Processor -- start 2024-05-28T06:14:21.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.933+0000 7f4055a00640 1 -- start start 2024-05-28T06:14:21.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.934+0000 7f4055a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f40500ce1f0 0x7f40500b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:21.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.934+0000 7f4055a00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f40500b3d60 0x7f40500b4210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:21.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.935+0000 7f4055a00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40500b3580 con 0x7f40500ce5d0 2024-05-28T06:14:21.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.935+0000 7f4055a00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f405015f490 con 0x7f40500ce1f0 2024-05-28T06:14:21.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.935+0000 7f4055a00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f405015f770 con 0x7f40500b3d60 2024-05-28T06:14:21.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.935+0000 7f404fe00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f40500ce1f0 0x7f40500b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:21.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.935+0000 7f404fe00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f40500ce1f0 0x7f40500b4dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48608/0 (socket says 172.21.15.96:48608) 2024-05-28T06:14:21.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.935+0000 7f404fe00640 1 -- 172.21.15.96:0/169239336 learned_addr learned my addr 172.21.15.96:0/169239336 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:21.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.935+0000 7f4055000640 1 --2- 172.21.15.96:0/169239336 >> v2:172.21.15.123:3300/0 conn(0x7f40500b3d60 0x7f40500b4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:21.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.935+0000 7f404fe00640 1 -- 172.21.15.96:0/169239336 >> v2:172.21.15.123:3300/0 conn(0x7f40500b3d60 msgr2=0x7f40500b4210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:21.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.935+0000 7f404fe00640 1 --2- 172.21.15.96:0/169239336 >> v2:172.21.15.123:3300/0 conn(0x7f40500b3d60 0x7f40500b4210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:21.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.935+0000 7f404fe00640 1 -- 172.21.15.96:0/169239336 >> v1:172.21.15.123:6789/0 conn(0x7f40500ce5d0 legacy=0x7f405015ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:21.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.935+0000 7f404fe00640 1 -- 172.21.15.96:0/169239336 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f405015fa50 con 0x7f40500ce1f0 2024-05-28T06:14:21.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.936+0000 7f404fe00640 1 --2- 172.21.15.96:0/169239336 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40500ce1f0 0x7f40500b4dd0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f40500cf860 tx=0x7f404002a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c303baca47acb4ea server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:21.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.937+0000 7f404f400640 1 -- 172.21.15.96:0/169239336 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4040037070 con 0x7f40500ce1f0 2024-05-28T06:14:21.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.937+0000 7f404f400640 1 -- 172.21.15.96:0/169239336 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f404002ae20 con 0x7f40500ce1f0 2024-05-28T06:14:21.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.937+0000 7f404f400640 1 -- 172.21.15.96:0/169239336 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4040032660 con 0x7f40500ce1f0 2024-05-28T06:14:21.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.937+0000 7f4055a00640 1 -- 172.21.15.96:0/169239336 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40500ce1f0 msgr2=0x7f40500b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:21.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.937+0000 7f4055a00640 1 --2- 172.21.15.96:0/169239336 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40500ce1f0 0x7f40500b4dd0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f40500cf860 tx=0x7f404002a110 comp rx=0 tx=0).stop 2024-05-28T06:14:21.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.937+0000 7f4055a00640 1 -- 172.21.15.96:0/169239336 shutdown_connections 2024-05-28T06:14:21.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.937+0000 7f4055a00640 1 --2- 172.21.15.96:0/169239336 >> v2:172.21.15.123:3300/0 conn(0x7f40500b3d60 0x7f40500b4210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:21.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.937+0000 7f4055a00640 1 --2- 172.21.15.96:0/169239336 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40500ce1f0 0x7f40500b4dd0 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:21.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.937+0000 7f4055a00640 1 -- 172.21.15.96:0/169239336 >> 172.21.15.96:0/169239336 conn(0x7f405006b900 msgr2=0x7f405006bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:21.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.937+0000 7f4055a00640 1 -- 172.21.15.96:0/169239336 shutdown_connections 2024-05-28T06:14:21.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.937+0000 7f4055a00640 1 -- 172.21.15.96:0/169239336 wait complete. 2024-05-28T06:14:21.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.939+0000 7f4055a00640 1 Processor -- start 2024-05-28T06:14:21.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.939+0000 7f4055a00640 1 -- start start 2024-05-28T06:14:21.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.940+0000 7f4055a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40500b3d60 0x7f40500afb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:21.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.940+0000 7f4055a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f40500ce1f0 0x7f4050155b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:21.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.940+0000 7f4055a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40500ce5d0 0x7f40501e9fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:21.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.940+0000 7f4055a00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f40500ccd10 con 0x7f40500ce5d0 2024-05-28T06:14:21.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.940+0000 7f4055a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f40500ccb90 con 0x7f40500b3d60 2024-05-28T06:14:21.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.940+0000 7f4055a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f40500cce90 con 0x7f40500ce1f0 2024-05-28T06:14:21.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.940+0000 7f4055000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40500ce5d0 0x7f40501e9fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:21.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.940+0000 7f4055000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40500ce5d0 0x7f40501e9fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48610/0 (socket says 172.21.15.96:48610) 2024-05-28T06:14:21.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.940+0000 7f4055000640 1 -- 172.21.15.96:0/2240788715 learned_addr learned my addr 172.21.15.96:0/2240788715 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:21.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.941+0000 7f4047400640 1 --2- 172.21.15.96:0/2240788715 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f40500ce1f0 0x7f4050155b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:21.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.941+0000 7f404fe00640 1 --2- 172.21.15.96:0/2240788715 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40500b3d60 0x7f40500afb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:21.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.941+0000 7f4055000640 1 -- 172.21.15.96:0/2240788715 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f40500ce1f0 msgr2=0x7f4050155b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:21.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.941+0000 7f4055000640 1 --2- 172.21.15.96:0/2240788715 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f40500ce1f0 0x7f4050155b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:21.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.941+0000 7f4055000640 1 -- 172.21.15.96:0/2240788715 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40500b3d60 msgr2=0x7f40500afb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:21.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.941+0000 7f4055000640 1 --2- 172.21.15.96:0/2240788715 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40500b3d60 0x7f40500afb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:21.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.941+0000 7f4055000640 1 -- 172.21.15.96:0/2240788715 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40500b0220 con 0x7f40500ce5d0 2024-05-28T06:14:21.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.941+0000 7f4055000640 1 --2- 172.21.15.96:0/2240788715 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40500ce5d0 0x7f40501e9fa0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f404800cd70 tx=0x7f404800a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:21.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.941+0000 7f404cc00640 1 -- 172.21.15.96:0/2240788715 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4048003b90 con 0x7f40500ce5d0 2024-05-28T06:14:21.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.941+0000 7f404cc00640 1 -- 172.21.15.96:0/2240788715 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4048003d30 con 0x7f40500ce5d0 2024-05-28T06:14:21.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.941+0000 7f4055a00640 1 -- 172.21.15.96:0/2240788715 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40500b0500 con 0x7f40500ce5d0 2024-05-28T06:14:21.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.942+0000 7f404cc00640 1 -- 172.21.15.96:0/2240788715 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4048004240 con 0x7f40500ce5d0 2024-05-28T06:14:21.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.942+0000 7f4055a00640 1 -- 172.21.15.96:0/2240788715 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40500b0ac0 con 0x7f40500ce5d0 2024-05-28T06:14:21.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.943+0000 7f4055a00640 1 -- 172.21.15.96:0/2240788715 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40500689e0 con 0x7f40500ce5d0 2024-05-28T06:14:21.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.944+0000 7f404cc00640 1 -- 172.21.15.96:0/2240788715 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 21) ==== 92111+0+0 (secure 0 0 0) 0x7f40480043e0 con 0x7f40500ce5d0 2024-05-28T06:14:21.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.945+0000 7f404cc00640 1 --2- 172.21.15.96:0/2240788715 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f402c06d4c0 0x7f402c06f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:21.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.945+0000 7f404cc00640 1 -- 172.21.15.96:0/2240788715 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5737+0+0 (secure 0 0 0) 0x7f404808b090 con 0x7f40500ce5d0 2024-05-28T06:14:21.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.945+0000 7f404fe00640 1 --2- 172.21.15.96:0/2240788715 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f402c06d4c0 0x7f402c06f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:21.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.946+0000 7f404fe00640 1 --2- 172.21.15.96:0/2240788715 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f402c06d4c0 0x7f402c06f980 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f4040034000 tx=0x7f404002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:21.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:21.949+0000 7f404cc00640 1 -- 172.21.15.96:0/2240788715 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f404801d340 con 0x7f40500ce5d0 2024-05-28T06:14:22.180 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.178+0000 7f4055a00640 1 -- 172.21.15.96:0/2240788715 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) -- 0x7f40500ca070 con 0x7f402c06d4c0 2024-05-28T06:14:22.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.219+0000 7f404cc00640 1 -- 172.21.15.96:0/2240788715 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f40500ca070 con 0x7f402c06d4c0 2024-05-28T06:14:22.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.222+0000 7f4055a00640 1 -- 172.21.15.96:0/2240788715 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f402c06d4c0 msgr2=0x7f402c06f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:22.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.222+0000 7f4055a00640 1 --2- 172.21.15.96:0/2240788715 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f402c06d4c0 0x7f402c06f980 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f4040034000 tx=0x7f404002e000 comp rx=0 tx=0).stop 2024-05-28T06:14:22.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.222+0000 7f4055a00640 1 -- 172.21.15.96:0/2240788715 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40500ce5d0 msgr2=0x7f40501e9fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:22.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.222+0000 7f4055a00640 1 --2- 172.21.15.96:0/2240788715 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40500ce5d0 0x7f40501e9fa0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f404800cd70 tx=0x7f404800a660 comp rx=0 tx=0).stop 2024-05-28T06:14:22.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.222+0000 7f4055a00640 1 -- 172.21.15.96:0/2240788715 shutdown_connections 2024-05-28T06:14:22.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.222+0000 7f4055a00640 1 --2- 172.21.15.96:0/2240788715 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f402c06d4c0 0x7f402c06f980 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:22.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.222+0000 7f4055a00640 1 --2- 172.21.15.96:0/2240788715 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40500ce5d0 0x7f40501e9fa0 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:22.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.222+0000 7f4055a00640 1 --2- 172.21.15.96:0/2240788715 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f40500ce1f0 0x7f4050155b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:22.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.222+0000 7f4055a00640 1 --2- 172.21.15.96:0/2240788715 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40500b3d60 0x7f40500afb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:22.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.222+0000 7f4055a00640 1 -- 172.21.15.96:0/2240788715 >> 172.21.15.96:0/2240788715 conn(0x7f405006b900 msgr2=0x7f40500ca7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:22.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.222+0000 7f4055a00640 1 -- 172.21.15.96:0/2240788715 shutdown_connections 2024-05-28T06:14:22.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.222+0000 7f4055a00640 1 -- 172.21.15.96:0/2240788715 wait complete. 2024-05-28T06:14:22.246 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone81711 2024-05-28T06:14:22.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.432+0000 7f885e800640 1 Processor -- start 2024-05-28T06:14:22.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.432+0000 7f885e800640 1 -- start start 2024-05-28T06:14:22.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.433+0000 7f885e800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f885815e5a0 0x7f88580d2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:22.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.434+0000 7f885e800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f88580d1420 0x7f885806c740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:22.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.434+0000 7f885e800640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88580d3500 con 0x7f88580cfb40 2024-05-28T06:14:22.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.434+0000 7f885e800640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f88580d2b30 con 0x7f885815e5a0 2024-05-28T06:14:22.436 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.434+0000 7f885e800640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f88580d2200 con 0x7f88580d1420 2024-05-28T06:14:22.436 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.434+0000 7f885d400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f885815e5a0 0x7f88580d2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:22.436 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.434+0000 7f885d400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f885815e5a0 0x7f88580d2fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48616/0 (socket says 172.21.15.96:48616) 2024-05-28T06:14:22.436 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.434+0000 7f885d400640 1 -- 172.21.15.96:0/771734377 learned_addr learned my addr 172.21.15.96:0/771734377 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:22.436 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.434+0000 7f885d400640 1 -- 172.21.15.96:0/771734377 >> v2:172.21.15.123:3300/0 conn(0x7f88580d1420 msgr2=0x7f885806c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:22.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.434+0000 7f885d400640 1 --2- 172.21.15.96:0/771734377 >> v2:172.21.15.123:3300/0 conn(0x7f88580d1420 0x7f885806c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:22.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.434+0000 7f885d400640 1 -- 172.21.15.96:0/771734377 >> v1:172.21.15.123:6789/0 conn(0x7f88580cfb40 legacy=0x7f88580d0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:22.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.434+0000 7f885d400640 1 -- 172.21.15.96:0/771734377 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88580d24e0 con 0x7f885815e5a0 2024-05-28T06:14:22.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.435+0000 7f885d400640 1 --2- 172.21.15.96:0/771734377 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f885815e5a0 0x7f88580d2fc0 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f885808e740 tx=0x7f884c02a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98fc8a6673d4dd99 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:22.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.436+0000 7f8857e00640 1 -- 172.21.15.96:0/771734377 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f884c037070 con 0x7f885815e5a0 2024-05-28T06:14:22.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.436+0000 7f8857e00640 1 -- 172.21.15.96:0/771734377 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f884c032440 con 0x7f885815e5a0 2024-05-28T06:14:22.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.436+0000 7f8857e00640 1 -- 172.21.15.96:0/771734377 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f884c032810 con 0x7f885815e5a0 2024-05-28T06:14:22.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.436+0000 7f885e800640 1 -- 172.21.15.96:0/771734377 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f885815e5a0 msgr2=0x7f88580d2fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:22.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.436+0000 7f885e800640 1 --2- 172.21.15.96:0/771734377 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f885815e5a0 0x7f88580d2fc0 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f885808e740 tx=0x7f884c02a230 comp rx=0 tx=0).stop 2024-05-28T06:14:22.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.436+0000 7f885e800640 1 -- 172.21.15.96:0/771734377 shutdown_connections 2024-05-28T06:14:22.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.436+0000 7f885e800640 1 --2- 172.21.15.96:0/771734377 >> v2:172.21.15.123:3300/0 conn(0x7f88580d1420 0x7f885806c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:22.439 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.436+0000 7f885e800640 1 --2- 172.21.15.96:0/771734377 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f885815e5a0 0x7f88580d2fc0 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:22.439 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.436+0000 7f885e800640 1 -- 172.21.15.96:0/771734377 >> 172.21.15.96:0/771734377 conn(0x7f885808bfc0 msgr2=0x7f88580d44b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:22.439 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.436+0000 7f885e800640 1 -- 172.21.15.96:0/771734377 shutdown_connections 2024-05-28T06:14:22.439 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.437+0000 7f885e800640 1 -- 172.21.15.96:0/771734377 wait complete. 2024-05-28T06:14:22.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.438+0000 7f885e800640 1 Processor -- start 2024-05-28T06:14:22.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.438+0000 7f885e800640 1 -- start start 2024-05-28T06:14:22.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.439+0000 7f885e800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f88580cfb40 0x7f8858155070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:22.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.439+0000 7f885e800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f88580d1420 0x7f88580cd590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:22.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.439+0000 7f885e800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f885815e5a0 0x7f88580cdad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:22.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.439+0000 7f885e800640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f885815eb00 con 0x7f885815e5a0 2024-05-28T06:14:22.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.439+0000 7f885e800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f885815e980 con 0x7f88580cfb40 2024-05-28T06:14:22.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.439+0000 7f885e800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f88580d5220 con 0x7f88580d1420 2024-05-28T06:14:22.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.439+0000 7f885de00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f885815e5a0 0x7f88580cdad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:22.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.439+0000 7f885de00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f885815e5a0 0x7f88580cdad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48628/0 (socket says 172.21.15.96:48628) 2024-05-28T06:14:22.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.439+0000 7f885de00640 1 -- 172.21.15.96:0/2203350487 learned_addr learned my addr 172.21.15.96:0/2203350487 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:22.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.440+0000 7f885d400640 1 --2- 172.21.15.96:0/2203350487 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f88580cfb40 0x7f8858155070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:22.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.440+0000 7f885ca00640 1 --2- 172.21.15.96:0/2203350487 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f88580d1420 0x7f88580cd590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:22.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.440+0000 7f885de00640 1 -- 172.21.15.96:0/2203350487 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f88580d1420 msgr2=0x7f88580cd590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:22.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.440+0000 7f885de00640 1 --2- 172.21.15.96:0/2203350487 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f88580d1420 0x7f88580cd590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:22.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.440+0000 7f885de00640 1 -- 172.21.15.96:0/2203350487 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f88580cfb40 msgr2=0x7f8858155070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:22.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.440+0000 7f885de00640 1 --2- 172.21.15.96:0/2203350487 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f88580cfb40 0x7f8858155070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:22.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.440+0000 7f885de00640 1 -- 172.21.15.96:0/2203350487 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88580c9d90 con 0x7f885815e5a0 2024-05-28T06:14:22.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.440+0000 7f885d400640 1 --2- 172.21.15.96:0/2203350487 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f88580cfb40 0x7f8858155070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:14:22.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.440+0000 7f885de00640 1 --2- 172.21.15.96:0/2203350487 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f885815e5a0 0x7f88580cdad0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f885000b9e0 tx=0x7f885000bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:22.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.441+0000 7f8855600640 1 -- 172.21.15.96:0/2203350487 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8850019070 con 0x7f885815e5a0 2024-05-28T06:14:22.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.441+0000 7f8855600640 1 -- 172.21.15.96:0/2203350487 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f885000cb40 con 0x7f885815e5a0 2024-05-28T06:14:22.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.441+0000 7f8855600640 1 -- 172.21.15.96:0/2203350487 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8850014440 con 0x7f885815e5a0 2024-05-28T06:14:22.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.441+0000 7f885e800640 1 -- 172.21.15.96:0/2203350487 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88580ca070 con 0x7f885815e5a0 2024-05-28T06:14:22.444 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.441+0000 7f885e800640 1 -- 172.21.15.96:0/2203350487 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88580ca630 con 0x7f885815e5a0 2024-05-28T06:14:22.444 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.442+0000 7f885e800640 1 -- 172.21.15.96:0/2203350487 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88580689e0 con 0x7f885815e5a0 2024-05-28T06:14:22.444 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.443+0000 7f8855600640 1 -- 172.21.15.96:0/2203350487 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 21) ==== 92111+0+0 (secure 0 0 0) 0x7f885001c070 con 0x7f885815e5a0 2024-05-28T06:14:22.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.444+0000 7f8855600640 1 --2- 172.21.15.96:0/2203350487 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f883806d4c0 0x7f883806f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:22.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.444+0000 7f8855600640 1 -- 172.21.15.96:0/2203350487 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5737+0+0 (secure 0 0 0) 0x7f885008a500 con 0x7f885815e5a0 2024-05-28T06:14:22.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.444+0000 7f885d400640 1 --2- 172.21.15.96:0/2203350487 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f883806d4c0 0x7f883806f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:22.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.446+0000 7f885d400640 1 --2- 172.21.15.96:0/2203350487 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f883806d4c0 0x7f883806f980 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f884c034000 tx=0x7f884c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:22.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.448+0000 7f8855600640 1 -- 172.21.15.96:0/2203350487 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f885005a120 con 0x7f885815e5a0 2024-05-28T06:14:22.659 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.657+0000 7f8855600640 1 -- 172.21.15.96:0/2203350487 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mgrmap(e 22) ==== 92213+0+0 (secure 0 0 0) 0x7f8850056090 con 0x7f885815e5a0 2024-05-28T06:14:22.719 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.718+0000 7f885e800640 1 -- 172.21.15.96:0/2203350487 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone81711", "target": ["mon-mgr", ""]}) -- 0x7f88580d1d30 con 0x7f883806d4c0 2024-05-28T06:14:22.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.754+0000 7f8855600640 1 -- 172.21.15.96:0/2203350487 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f88580d1d30 con 0x7f883806d4c0 2024-05-28T06:14:22.759 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.758+0000 7f885e800640 1 -- 172.21.15.96:0/2203350487 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f883806d4c0 msgr2=0x7f883806f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:22.759 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.758+0000 7f885e800640 1 --2- 172.21.15.96:0/2203350487 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f883806d4c0 0x7f883806f980 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f884c034000 tx=0x7f884c02e000 comp rx=0 tx=0).stop 2024-05-28T06:14:22.760 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.758+0000 7f885e800640 1 -- 172.21.15.96:0/2203350487 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f885815e5a0 msgr2=0x7f88580cdad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:22.760 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.758+0000 7f885e800640 1 --2- 172.21.15.96:0/2203350487 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f885815e5a0 0x7f88580cdad0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f885000b9e0 tx=0x7f885000bea0 comp rx=0 tx=0).stop 2024-05-28T06:14:22.760 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.758+0000 7f885e800640 1 -- 172.21.15.96:0/2203350487 shutdown_connections 2024-05-28T06:14:22.760 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.758+0000 7f885e800640 1 --2- 172.21.15.96:0/2203350487 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f883806d4c0 0x7f883806f980 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:22.760 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.758+0000 7f885e800640 1 --2- 172.21.15.96:0/2203350487 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f885815e5a0 0x7f88580cdad0 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:22.760 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.758+0000 7f885e800640 1 --2- 172.21.15.96:0/2203350487 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f88580d1420 0x7f88580cd590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:22.760 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.758+0000 7f885e800640 1 --2- 172.21.15.96:0/2203350487 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f88580cfb40 0x7f8858155070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:22.760 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.758+0000 7f885e800640 1 -- 172.21.15.96:0/2203350487 >> 172.21.15.96:0/2203350487 conn(0x7f885808bfc0 msgr2=0x7f88580d4480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:22.761 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.759+0000 7f885e800640 1 -- 172.21.15.96:0/2203350487 shutdown_connections 2024-05-28T06:14:22.761 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.759+0000 7f885e800640 1 -- 172.21.15.96:0/2203350487 wait complete. 2024-05-28T06:14:22.786 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone81715 2024-05-28T06:14:22.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.958+0000 7f2577600640 1 Processor -- start 2024-05-28T06:14:22.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.958+0000 7f2577600640 1 -- start start 2024-05-28T06:14:22.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.959+0000 7f2577600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f257015e830 0x7f257006dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:22.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.959+0000 7f2577600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f25700b3510 0x7f257006f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:22.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.959+0000 7f2577600640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f257006e190 con 0x7f2570071c00 2024-05-28T06:14:22.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.959+0000 7f2577600640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f257006f770 con 0x7f257015e830 2024-05-28T06:14:22.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.959+0000 7f2577600640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f257006e470 con 0x7f25700b3510 2024-05-28T06:14:22.962 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.960+0000 7f2574c00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f257015e830 0x7f257006dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:22.962 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.960+0000 7f2575600640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f2570071c00 0x7f25700ad130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:54828/0 (socket says 172.21.15.96:54828) 2024-05-28T06:14:22.962 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.960+0000 7f2575600640 1 -- 172.21.15.96:0/3494190392 learned_addr learned my addr 172.21.15.96:0/3494190392 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:14:22.962 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.960+0000 7f2574c00640 1 -- 172.21.15.96:0/3494190392 >> v2:172.21.15.123:3300/0 conn(0x7f25700b3510 msgr2=0x7f257006f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:22.962 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.960+0000 7f2574c00640 1 --2- 172.21.15.96:0/3494190392 >> v2:172.21.15.123:3300/0 conn(0x7f25700b3510 0x7f257006f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:22.962 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.960+0000 7f2574c00640 1 -- 172.21.15.96:0/3494190392 >> v1:172.21.15.96:6789/0 conn(0x7f2570071c00 legacy=0x7f25700ad130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:22.962 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.960+0000 7f2574c00640 1 -- 172.21.15.96:0/3494190392 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f257006e750 con 0x7f257015e830 2024-05-28T06:14:22.963 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.961+0000 7f2574c00640 1 --2- 172.21.15.96:0/3494190392 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f257015e830 0x7f257006dbb0 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f25700b47b0 tx=0x7f256402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecd03f9384bea02f server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:22.963 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.962+0000 7f256f400640 1 -- 172.21.15.96:0/3494190392 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2564037070 con 0x7f257015e830 2024-05-28T06:14:22.963 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.962+0000 7f256f400640 1 -- 172.21.15.96:0/3494190392 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2564032440 con 0x7f257015e830 2024-05-28T06:14:22.963 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.962+0000 7f256f400640 1 -- 172.21.15.96:0/3494190392 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2564032810 con 0x7f257015e830 2024-05-28T06:14:22.964 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.962+0000 7f2577600640 1 -- 172.21.15.96:0/3494190392 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f257015e830 msgr2=0x7f257006dbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:22.964 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.962+0000 7f2577600640 1 --2- 172.21.15.96:0/3494190392 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f257015e830 0x7f257006dbb0 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f25700b47b0 tx=0x7f256402a1a0 comp rx=0 tx=0).stop 2024-05-28T06:14:22.964 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.962+0000 7f2577600640 1 -- 172.21.15.96:0/3494190392 shutdown_connections 2024-05-28T06:14:22.964 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.962+0000 7f2577600640 1 --2- 172.21.15.96:0/3494190392 >> v2:172.21.15.123:3300/0 conn(0x7f25700b3510 0x7f257006f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:22.964 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.962+0000 7f2577600640 1 --2- 172.21.15.96:0/3494190392 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f257015e830 0x7f257006dbb0 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:22.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.962+0000 7f2577600640 1 -- 172.21.15.96:0/3494190392 >> 172.21.15.96:0/3494190392 conn(0x7f25700b1e50 msgr2=0x7f25700b2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:22.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.963+0000 7f2577600640 1 -- 172.21.15.96:0/3494190392 shutdown_connections 2024-05-28T06:14:22.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.963+0000 7f2577600640 1 -- 172.21.15.96:0/3494190392 wait complete. 2024-05-28T06:14:22.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.964+0000 7f2577600640 1 Processor -- start 2024-05-28T06:14:22.966 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.964+0000 7f2577600640 1 -- start start 2024-05-28T06:14:22.967 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.965+0000 7f2577600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2570071c00 0x7f25701f4530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:22.967 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.965+0000 7f2577600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f25700b3510 0x7f2570159980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:22.967 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.965+0000 7f2577600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f257015e830 0x7f2570154be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:22.967 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.965+0000 7f2577600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f257015ec10 con 0x7f2570071c00 2024-05-28T06:14:22.967 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.965+0000 7f2577600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f257006ed10 con 0x7f25700b3510 2024-05-28T06:14:22.967 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.965+0000 7f2577600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f25700728c0 con 0x7f257015e830 2024-05-28T06:14:22.967 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.966+0000 7f2574c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2570071c00 0x7f25701f4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:22.967 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.966+0000 7f2574c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2570071c00 0x7f25701f4530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48640/0 (socket says 172.21.15.96:48640) 2024-05-28T06:14:22.968 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.966+0000 7f2574c00640 1 -- 172.21.15.96:0/1158960264 learned_addr learned my addr 172.21.15.96:0/1158960264 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:22.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.966+0000 7f2574c00640 1 -- 172.21.15.96:0/1158960264 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f257015e830 msgr2=0x7f2570154be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:22.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.966+0000 7f2575600640 1 --2- 172.21.15.96:0/1158960264 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f257015e830 0x7f2570154be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:22.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.966+0000 7f256fe00640 1 --2- 172.21.15.96:0/1158960264 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f25700b3510 0x7f2570159980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:22.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.966+0000 7f2574c00640 1 --2- 172.21.15.96:0/1158960264 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f257015e830 0x7f2570154be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:22.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.966+0000 7f2574c00640 1 -- 172.21.15.96:0/1158960264 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f25700b3510 msgr2=0x7f2570159980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:22.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.966+0000 7f2574c00640 1 --2- 172.21.15.96:0/1158960264 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f25700b3510 0x7f2570159980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:22.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.966+0000 7f2574c00640 1 -- 172.21.15.96:0/1158960264 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25701f4b60 con 0x7f2570071c00 2024-05-28T06:14:22.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.966+0000 7f2574c00640 1 --2- 172.21.15.96:0/1158960264 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2570071c00 0x7f25701f4530 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f2564000c00 tx=0x7f25640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:22.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.967+0000 7f256cc00640 1 -- 172.21.15.96:0/1158960264 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2564037040 con 0x7f2570071c00 2024-05-28T06:14:22.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.967+0000 7f256cc00640 1 -- 172.21.15.96:0/1158960264 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2564042070 con 0x7f2570071c00 2024-05-28T06:14:22.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.967+0000 7f256cc00640 1 -- 172.21.15.96:0/1158960264 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2564030bd0 con 0x7f2570071c00 2024-05-28T06:14:22.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.967+0000 7f2577600640 1 -- 172.21.15.96:0/1158960264 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25701f4e40 con 0x7f2570071c00 2024-05-28T06:14:22.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.967+0000 7f2577600640 1 -- 172.21.15.96:0/1158960264 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2570157ed0 con 0x7f2570071c00 2024-05-28T06:14:22.970 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.968+0000 7f2577600640 1 -- 172.21.15.96:0/1158960264 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25701551e0 con 0x7f2570071c00 2024-05-28T06:14:22.970 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.969+0000 7f256cc00640 1 -- 172.21.15.96:0/1158960264 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 22) ==== 92213+0+0 (secure 0 0 0) 0x7f2564047460 con 0x7f2570071c00 2024-05-28T06:14:22.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.970+0000 7f256cc00640 1 --2- 172.21.15.96:0/1158960264 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f255006d5e0 0x7f255006faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:22.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.970+0000 7f256cc00640 1 -- 172.21.15.96:0/1158960264 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5793+0+0 (secure 0 0 0) 0x7f25640ab4d0 con 0x7f2570071c00 2024-05-28T06:14:22.972 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.970+0000 7f256fe00640 1 --2- 172.21.15.96:0/1158960264 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f255006d5e0 0x7f255006faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:22.973 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.971+0000 7f256fe00640 1 --2- 172.21.15.96:0/1158960264 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f255006d5e0 0x7f255006faa0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f25700b1cf0 tx=0x7f255c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:22.976 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:22.974+0000 7f256cc00640 1 -- 172.21.15.96:0/1158960264 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2564032e20 con 0x7f2570071c00 2024-05-28T06:14:23.355 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:23.354+0000 7f2577600640 1 -- 172.21.15.96:0/1158960264 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone81715", "target": ["mon-mgr", ""]}) -- 0x7f25701580d0 con 0x7f255006d5e0 2024-05-28T06:14:23.389 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:23.388+0000 7f256cc00640 1 -- 172.21.15.96:0/1158960264 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f25701580d0 con 0x7f255006d5e0 2024-05-28T06:14:23.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:23.390+0000 7f2577600640 1 -- 172.21.15.96:0/1158960264 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f255006d5e0 msgr2=0x7f255006faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:23.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:23.390+0000 7f2577600640 1 --2- 172.21.15.96:0/1158960264 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f255006d5e0 0x7f255006faa0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f25700b1cf0 tx=0x7f255c008040 comp rx=0 tx=0).stop 2024-05-28T06:14:23.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:23.390+0000 7f2577600640 1 -- 172.21.15.96:0/1158960264 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2570071c00 msgr2=0x7f25701f4530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:23.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:23.390+0000 7f2577600640 1 --2- 172.21.15.96:0/1158960264 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2570071c00 0x7f25701f4530 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f2564000c00 tx=0x7f25640046c0 comp rx=0 tx=0).stop 2024-05-28T06:14:23.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:23.390+0000 7f2577600640 1 -- 172.21.15.96:0/1158960264 shutdown_connections 2024-05-28T06:14:23.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:23.390+0000 7f2577600640 1 --2- 172.21.15.96:0/1158960264 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f255006d5e0 0x7f255006faa0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:23.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:23.390+0000 7f2577600640 1 --2- 172.21.15.96:0/1158960264 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f257015e830 0x7f2570154be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:23.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:23.390+0000 7f2577600640 1 --2- 172.21.15.96:0/1158960264 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f25700b3510 0x7f2570159980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:23.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:23.390+0000 7f2577600640 1 --2- 172.21.15.96:0/1158960264 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2570071c00 0x7f25701f4530 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:23.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:23.390+0000 7f2577600640 1 -- 172.21.15.96:0/1158960264 >> 172.21.15.96:0/1158960264 conn(0x7f25700b1e50 msgr2=0x7f257009a8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:23.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:23.390+0000 7f2577600640 1 -- 172.21.15.96:0/1158960264 shutdown_connections 2024-05-28T06:14:23.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:23.391+0000 7f2577600640 1 -- 172.21.15.96:0/1158960264 wait complete. 2024-05-28T06:14:23.409 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone81717 2024-05-28T06:14:23.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:23.573+0000 7ff998a00640 1 Processor -- start 2024-05-28T06:14:23.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:23.573+0000 7ff998a00640 1 -- start start 2024-05-28T06:14:23.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:23.574+0000 7ff998a00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7ff99415e830 0x7ff99406dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:23.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:23.574+0000 7ff998a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7ff9940b3510 0x7ff99406f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:23.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:23.574+0000 7ff998a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff994071c00 0x7ff994072090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:23.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:23.574+0000 7ff998a00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7ff99406f770 con 0x7ff9940b3510 2024-05-28T06:14:23.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:23.574+0000 7ff998a00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7ff99406e190 con 0x7ff99415e830 2024-05-28T06:14:23.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:23.574+0000 7ff998a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ff99406e470 con 0x7ff994071c00 2024-05-28T06:14:23.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:23.574+0000 7ff991400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7ff9940b3510 0x7ff99406f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:23.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:23.574+0000 7ff991400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7ff9940b3510 0x7ff99406f230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48648/0 (socket says 172.21.15.96:48648) 2024-05-28T06:14:23.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:23.574+0000 7ff991400640 1 -- 172.21.15.96:0/2689446294 learned_addr learned my addr 172.21.15.96:0/2689446294 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:23.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:23.575+0000 7ff991e00640 1 --2- 172.21.15.96:0/2689446294 >> v2:172.21.15.123:3300/0 conn(0x7ff99415e830 0x7ff99406dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:23.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:23.575+0000 7ff992800640 1 --2- 172.21.15.96:0/2689446294 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff994071c00 0x7ff994072090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:23.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:23.575+0000 7ff991400640 1 -- 172.21.15.96:0/2689446294 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff994071c00 msgr2=0x7ff994072090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:23.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:23.575+0000 7ff991400640 1 --2- 172.21.15.96:0/2689446294 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff994071c00 0x7ff994072090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:23.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:23.575+0000 7ff991400640 1 -- 172.21.15.96:0/2689446294 >> v2:172.21.15.123:3300/0 conn(0x7ff99415e830 msgr2=0x7ff99406dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:23.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:23.575+0000 7ff991400640 1 --2- 172.21.15.96:0/2689446294 >> v2:172.21.15.123:3300/0 conn(0x7ff99415e830 0x7ff99406dbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:23.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:23.575+0000 7ff991400640 1 -- 172.21.15.96:0/2689446294 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff99406e750 con 0x7ff9940b3510 2024-05-28T06:14:23.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:23.576+0000 7ff991400640 1 --2- 172.21.15.96:0/2689446294 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff9940b3510 0x7ff99406f230 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7ff9940b61d0 tx=0x7ff98002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6805f39b1d626ec0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:23.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:23.576+0000 7ff990a00640 1 -- 172.21.15.96:0/2689446294 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff980037070 con 0x7ff9940b3510 2024-05-28T06:14:23.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:23.576+0000 7ff990a00640 1 -- 172.21.15.96:0/2689446294 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff980032440 con 0x7ff9940b3510 2024-05-28T06:14:23.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:23.576+0000 7ff990a00640 1 -- 172.21.15.96:0/2689446294 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff980032810 con 0x7ff9940b3510 2024-05-28T06:14:23.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:23.577+0000 7ff998a00640 1 -- 172.21.15.96:0/2689446294 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff9940b3510 msgr2=0x7ff99406f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:23.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:23.577+0000 7ff998a00640 1 --2- 172.21.15.96:0/2689446294 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff9940b3510 0x7ff99406f230 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7ff9940b61d0 tx=0x7ff98002a1e0 comp rx=0 tx=0).stop 2024-05-28T06:14:23.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:23.577+0000 7ff998a00640 1 -- 172.21.15.96:0/2689446294 shutdown_connections 2024-05-28T06:14:23.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:23.577+0000 7ff998a00640 1 --2- 172.21.15.96:0/2689446294 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff994071c00 0x7ff994072090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:23.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:23.577+0000 7ff998a00640 1 --2- 172.21.15.96:0/2689446294 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff9940b3510 0x7ff99406f230 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:23.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:23.577+0000 7ff998a00640 1 --2- 172.21.15.96:0/2689446294 >> v2:172.21.15.123:3300/0 conn(0x7ff99415e830 0x7ff99406dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:23.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:23.577+0000 7ff998a00640 1 -- 172.21.15.96:0/2689446294 >> 172.21.15.96:0/2689446294 conn(0x7ff9940b1e50 msgr2=0x7ff9940b2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:23.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:23.577+0000 7ff998a00640 1 -- 172.21.15.96:0/2689446294 shutdown_connections 2024-05-28T06:14:23.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:23.577+0000 7ff998a00640 1 -- 172.21.15.96:0/2689446294 wait complete. 2024-05-28T06:14:23.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:23.579+0000 7ff998a00640 1 Processor -- start 2024-05-28T06:14:23.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:23.579+0000 7ff998a00640 1 -- start start 2024-05-28T06:14:23.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:23.580+0000 7ff998a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff994071c00 0x7ff99414a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:23.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:23.580+0000 7ff998a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff9940b3510 0x7ff994159fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:23.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:23.580+0000 7ff998a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff99415e830 0x7ff99415a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:23.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:23.580+0000 7ff998a00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff99415ec10 con 0x7ff994071c00 2024-05-28T06:14:23.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:23.580+0000 7ff998a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff99406ed10 con 0x7ff9940b3510 2024-05-28T06:14:23.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:23.580+0000 7ff998a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ff9940728c0 con 0x7ff99415e830 2024-05-28T06:14:23.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:23.580+0000 7ff991e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff994071c00 0x7ff99414a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:23.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:23.580+0000 7ff991e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff994071c00 0x7ff99414a450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48662/0 (socket says 172.21.15.96:48662) 2024-05-28T06:14:23.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:23.580+0000 7ff991e00640 1 -- 172.21.15.96:0/3928971260 learned_addr learned my addr 172.21.15.96:0/3928971260 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:23.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:23.580+0000 7ff991e00640 1 -- 172.21.15.96:0/3928971260 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff99415e830 msgr2=0x7ff99415a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:23.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:23.580+0000 7ff991e00640 1 --2- 172.21.15.96:0/3928971260 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff99415e830 0x7ff99415a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:23.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:23.580+0000 7ff991400640 1 --2- 172.21.15.96:0/3928971260 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff9940b3510 0x7ff994159fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:23.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:23.580+0000 7ff991e00640 1 -- 172.21.15.96:0/3928971260 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff9940b3510 msgr2=0x7ff994159fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:23.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:23.580+0000 7ff991e00640 1 --2- 172.21.15.96:0/3928971260 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff9940b3510 0x7ff994159fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:23.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:23.580+0000 7ff991e00640 1 -- 172.21.15.96:0/3928971260 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff994160450 con 0x7ff994071c00 2024-05-28T06:14:23.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:23.581+0000 7ff991e00640 1 --2- 172.21.15.96:0/3928971260 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff994071c00 0x7ff99414a450 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7ff98400c7f0 tx=0x7ff98400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:23.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:23.581+0000 7ff98a000640 1 -- 172.21.15.96:0/3928971260 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff984015070 con 0x7ff994071c00 2024-05-28T06:14:23.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:23.581+0000 7ff98a000640 1 -- 172.21.15.96:0/3928971260 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff984004220 con 0x7ff994071c00 2024-05-28T06:14:23.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:23.581+0000 7ff98a000640 1 -- 172.21.15.96:0/3928971260 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff984007620 con 0x7ff994071c00 2024-05-28T06:14:23.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:23.581+0000 7ff998a00640 1 -- 172.21.15.96:0/3928971260 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9941e9ec0 con 0x7ff994071c00 2024-05-28T06:14:23.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:23.581+0000 7ff998a00640 1 -- 172.21.15.96:0/3928971260 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff99414a990 con 0x7ff994071c00 2024-05-28T06:14:23.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:23.582+0000 7ff998a00640 1 -- 172.21.15.96:0/3928971260 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9941e9cf0 con 0x7ff994071c00 2024-05-28T06:14:23.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:23.583+0000 7ff98a000640 1 -- 172.21.15.96:0/3928971260 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 22) ==== 92213+0+0 (secure 0 0 0) 0x7ff9840079d0 con 0x7ff994071c00 2024-05-28T06:14:23.586 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:23.584+0000 7ff98a000640 1 --2- 172.21.15.96:0/3928971260 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff97406d5e0 0x7ff97406faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:23.586 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:23.584+0000 7ff98a000640 1 -- 172.21.15.96:0/3928971260 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5793+0+0 (secure 0 0 0) 0x7ff98408a6d0 con 0x7ff994071c00 2024-05-28T06:14:23.586 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:23.585+0000 7ff991400640 1 --2- 172.21.15.96:0/3928971260 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff97406d5e0 0x7ff97406faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:23.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:23.586+0000 7ff991400640 1 --2- 172.21.15.96:0/3928971260 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff97406d5e0 0x7ff97406faa0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7ff980034000 tx=0x7ff98002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:23.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:23.589+0000 7ff98a000640 1 -- 172.21.15.96:0/3928971260 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff98405a480 con 0x7ff994071c00 2024-05-28T06:14:23.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:23.802+0000 7ff998a00640 1 -- 172.21.15.96:0/3928971260 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone81717", "target": ["mon-mgr", ""]}) -- 0x7ff9940689e0 con 0x7ff97406d5e0 2024-05-28T06:14:27.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.320+0000 7ff98a000640 1 -- 172.21.15.96:0/3928971260 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7ff9940689e0 con 0x7ff97406d5e0 2024-05-28T06:14:27.327 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.324+0000 7ff998a00640 1 -- 172.21.15.96:0/3928971260 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff97406d5e0 msgr2=0x7ff97406faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:27.327 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.324+0000 7ff998a00640 1 --2- 172.21.15.96:0/3928971260 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff97406d5e0 0x7ff97406faa0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7ff980034000 tx=0x7ff98002e000 comp rx=0 tx=0).stop 2024-05-28T06:14:27.327 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.324+0000 7ff998a00640 1 -- 172.21.15.96:0/3928971260 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff994071c00 msgr2=0x7ff99414a450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:27.327 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.324+0000 7ff998a00640 1 --2- 172.21.15.96:0/3928971260 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff994071c00 0x7ff99414a450 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7ff98400c7f0 tx=0x7ff98400ccb0 comp rx=0 tx=0).stop 2024-05-28T06:14:27.327 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.325+0000 7ff998a00640 1 -- 172.21.15.96:0/3928971260 shutdown_connections 2024-05-28T06:14:27.327 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.325+0000 7ff998a00640 1 --2- 172.21.15.96:0/3928971260 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff97406d5e0 0x7ff97406faa0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:27.327 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.325+0000 7ff998a00640 1 --2- 172.21.15.96:0/3928971260 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff99415e830 0x7ff99415a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:27.327 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.325+0000 7ff998a00640 1 --2- 172.21.15.96:0/3928971260 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff9940b3510 0x7ff994159fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:27.327 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.325+0000 7ff998a00640 1 --2- 172.21.15.96:0/3928971260 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff994071c00 0x7ff99414a450 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:27.327 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.325+0000 7ff998a00640 1 -- 172.21.15.96:0/3928971260 >> 172.21.15.96:0/3928971260 conn(0x7ff9940b1e50 msgr2=0x7ff994074a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:27.327 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.325+0000 7ff998a00640 1 -- 172.21.15.96:0/3928971260 shutdown_connections 2024-05-28T06:14:27.328 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.325+0000 7ff998a00640 1 -- 172.21.15.96:0/3928971260 wait complete. 2024-05-28T06:14:27.355 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone81714 2024-05-28T06:14:27.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.554+0000 7f58f8a00640 1 Processor -- start 2024-05-28T06:14:27.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.554+0000 7f58f8a00640 1 -- start start 2024-05-28T06:14:27.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.555+0000 7f58f8a00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f58f40cf2f0 0x7f58f40cf910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:27.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.555+0000 7f58f8a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f58f4140500 0x7f58f40ad130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:27.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.555+0000 7f58f8a00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58f406dd50 con 0x7f58f406edf0 2024-05-28T06:14:27.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.555+0000 7f58f8a00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f58f406e030 con 0x7f58f40cf2f0 2024-05-28T06:14:27.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.555+0000 7f58f8a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f58f406e310 con 0x7f58f4140500 2024-05-28T06:14:27.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.556+0000 7f58f1e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f58f40cf2f0 0x7f58f40cf910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:27.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.556+0000 7f58f2800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f58f4140500 0x7f58f40ad130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:27.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.556+0000 7f58f1e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f58f40cf2f0 0x7f58f40cf910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.96:47890/0 (socket says 172.21.15.96:47890) 2024-05-28T06:14:27.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.556+0000 7f58f1e00640 1 -- 172.21.15.96:0/2161344660 learned_addr learned my addr 172.21.15.96:0/2161344660 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:27.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.557+0000 7f58f1e00640 1 -- 172.21.15.96:0/2161344660 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f58f4140500 msgr2=0x7f58f40ad130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:27.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.557+0000 7f58f1e00640 1 --2- 172.21.15.96:0/2161344660 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f58f4140500 0x7f58f40ad130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:27.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.557+0000 7f58f1e00640 1 -- 172.21.15.96:0/2161344660 >> v1:172.21.15.123:6789/0 conn(0x7f58f406edf0 legacy=0x7f58f413fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:27.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.557+0000 7f58f1e00640 1 -- 172.21.15.96:0/2161344660 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58f406e5f0 con 0x7f58f40cf2f0 2024-05-28T06:14:27.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.557+0000 7f58f2800640 1 --2- 172.21.15.96:0/2161344660 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f58f4140500 0x7f58f40ad130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-28T06:14:27.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.558+0000 7f58f1e00640 1 --2- 172.21.15.96:0/2161344660 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58f40cf2f0 0x7f58f40cf910 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f58e4009870 tx=0x7f58e402a1e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8cf1e89907f5e358 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:27.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.559+0000 7f58f0a00640 1 -- 172.21.15.96:0/2161344660 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58e4037070 con 0x7f58f40cf2f0 2024-05-28T06:14:27.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.559+0000 7f58f0a00640 1 -- 172.21.15.96:0/2161344660 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58e4032440 con 0x7f58f40cf2f0 2024-05-28T06:14:27.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.559+0000 7f58f0a00640 1 -- 172.21.15.96:0/2161344660 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58e4032810 con 0x7f58f40cf2f0 2024-05-28T06:14:27.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.559+0000 7f58f8a00640 1 -- 172.21.15.96:0/2161344660 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58f40cf2f0 msgr2=0x7f58f40cf910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:27.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.559+0000 7f58f8a00640 1 --2- 172.21.15.96:0/2161344660 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58f40cf2f0 0x7f58f40cf910 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f58e4009870 tx=0x7f58e402a1e0 comp rx=0 tx=0).stop 2024-05-28T06:14:27.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.559+0000 7f58f8a00640 1 -- 172.21.15.96:0/2161344660 shutdown_connections 2024-05-28T06:14:27.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.559+0000 7f58f8a00640 1 --2- 172.21.15.96:0/2161344660 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f58f4140500 0x7f58f40ad130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:27.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.559+0000 7f58f8a00640 1 --2- 172.21.15.96:0/2161344660 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58f40cf2f0 0x7f58f40cf910 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:27.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.559+0000 7f58f8a00640 1 -- 172.21.15.96:0/2161344660 >> 172.21.15.96:0/2161344660 conn(0x7f58f40b1e50 msgr2=0x7f58f40b2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:27.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.560+0000 7f58f8a00640 1 -- 172.21.15.96:0/2161344660 shutdown_connections 2024-05-28T06:14:27.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.560+0000 7f58f8a00640 1 -- 172.21.15.96:0/2161344660 wait complete. 2024-05-28T06:14:27.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.561+0000 7f58f8a00640 1 Processor -- start 2024-05-28T06:14:27.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.561+0000 7f58f8a00640 1 -- start start 2024-05-28T06:14:27.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.562+0000 7f58f8a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58f406edf0 0x7f58f4141300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:27.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.562+0000 7f58f8a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f58f40cf2f0 0x7f58f41f45c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:27.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.562+0000 7f58f8a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f58f40ca150 0x7f58f40b3f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:27.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.562+0000 7f58f8a00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f58f41461a0 con 0x7f58f40ca150 2024-05-28T06:14:27.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.562+0000 7f58f8a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f58f4146020 con 0x7f58f406edf0 2024-05-28T06:14:27.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.562+0000 7f58f8a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f58f40cfe50 con 0x7f58f40cf2f0 2024-05-28T06:14:27.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.563+0000 7f58f2800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f58f40ca150 0x7f58f40b3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:27.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.563+0000 7f58f2800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f58f40ca150 0x7f58f40b3f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48676/0 (socket says 172.21.15.96:48676) 2024-05-28T06:14:27.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.563+0000 7f58f2800640 1 -- 172.21.15.96:0/11687589 learned_addr learned my addr 172.21.15.96:0/11687589 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:27.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.563+0000 7f58f1e00640 1 --2- 172.21.15.96:0/11687589 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58f406edf0 0x7f58f4141300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:27.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.563+0000 7f58f1400640 1 --2- 172.21.15.96:0/11687589 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f58f40cf2f0 0x7f58f41f45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:27.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.563+0000 7f58f2800640 1 -- 172.21.15.96:0/11687589 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f58f40cf2f0 msgr2=0x7f58f41f45c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:27.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.563+0000 7f58f2800640 1 --2- 172.21.15.96:0/11687589 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f58f40cf2f0 0x7f58f41f45c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:27.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.563+0000 7f58f2800640 1 -- 172.21.15.96:0/11687589 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58f406edf0 msgr2=0x7f58f4141300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:27.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.563+0000 7f58f2800640 1 --2- 172.21.15.96:0/11687589 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58f406edf0 0x7f58f4141300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:27.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.563+0000 7f58f2800640 1 -- 172.21.15.96:0/11687589 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58f415e810 con 0x7f58f40ca150 2024-05-28T06:14:27.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.563+0000 7f58f1400640 1 --2- 172.21.15.96:0/11687589 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f58f40cf2f0 0x7f58f41f45c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:14:27.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.563+0000 7f58f1e00640 1 --2- 172.21.15.96:0/11687589 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58f406edf0 0x7f58f4141300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:14:27.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.564+0000 7f58f2800640 1 --2- 172.21.15.96:0/11687589 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f58f40ca150 0x7f58f40b3f90 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f58ec009ea0 tx=0x7f58ec009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:27.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.564+0000 7f58ea000640 1 -- 172.21.15.96:0/11687589 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58ec018020 con 0x7f58f40ca150 2024-05-28T06:14:27.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.564+0000 7f58ea000640 1 -- 172.21.15.96:0/11687589 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58ec014480 con 0x7f58f40ca150 2024-05-28T06:14:27.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.564+0000 7f58ea000640 1 -- 172.21.15.96:0/11687589 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58ec014870 con 0x7f58f40ca150 2024-05-28T06:14:27.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.564+0000 7f58f8a00640 1 -- 172.21.15.96:0/11687589 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58f415eaf0 con 0x7f58f40ca150 2024-05-28T06:14:27.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.564+0000 7f58f8a00640 1 -- 172.21.15.96:0/11687589 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58f415f0b0 con 0x7f58f40ca150 2024-05-28T06:14:27.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.565+0000 7f58f8a00640 1 -- 172.21.15.96:0/11687589 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58f40689e0 con 0x7f58f40ca150 2024-05-28T06:14:27.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.566+0000 7f58ea000640 1 -- 172.21.15.96:0/11687589 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 22) ==== 92213+0+0 (secure 0 0 0) 0x7f58ec01b020 con 0x7f58f40ca150 2024-05-28T06:14:27.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.567+0000 7f58ea000640 1 --2- 172.21.15.96:0/11687589 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f58d406d5e0 0x7f58d406faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:27.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.567+0000 7f58ea000640 1 -- 172.21.15.96:0/11687589 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5793+0+0 (secure 0 0 0) 0x7f58ec08b400 con 0x7f58f40ca150 2024-05-28T06:14:27.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.568+0000 7f58f1e00640 1 --2- 172.21.15.96:0/11687589 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f58d406d5e0 0x7f58d406faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:27.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.569+0000 7f58f1e00640 1 --2- 172.21.15.96:0/11687589 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f58d406d5e0 0x7f58d406faa0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f58e4034000 tx=0x7f58e402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:27.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.572+0000 7f58ea000640 1 -- 172.21.15.96:0/11687589 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f58ec022020 con 0x7f58f40ca150 2024-05-28T06:14:27.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.803+0000 7f58f8a00640 1 -- 172.21.15.96:0/11687589 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone81714", "target": ["mon-mgr", ""]}) -- 0x7f58f40ce5e0 con 0x7f58d406d5e0 2024-05-28T06:14:27.854 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.852+0000 7f58ea000640 1 -- 172.21.15.96:0/11687589 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f58f40ce5e0 con 0x7f58d406d5e0 2024-05-28T06:14:27.857 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.855+0000 7f58f8a00640 1 -- 172.21.15.96:0/11687589 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f58d406d5e0 msgr2=0x7f58d406faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:27.857 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.855+0000 7f58f8a00640 1 --2- 172.21.15.96:0/11687589 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f58d406d5e0 0x7f58d406faa0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f58e4034000 tx=0x7f58e402e000 comp rx=0 tx=0).stop 2024-05-28T06:14:27.857 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.855+0000 7f58f8a00640 1 -- 172.21.15.96:0/11687589 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f58f40ca150 msgr2=0x7f58f40b3f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:27.857 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.855+0000 7f58f8a00640 1 --2- 172.21.15.96:0/11687589 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f58f40ca150 0x7f58f40b3f90 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f58ec009ea0 tx=0x7f58ec009f90 comp rx=0 tx=0).stop 2024-05-28T06:14:27.857 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.855+0000 7f58f8a00640 1 -- 172.21.15.96:0/11687589 shutdown_connections 2024-05-28T06:14:27.857 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.855+0000 7f58f8a00640 1 --2- 172.21.15.96:0/11687589 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f58d406d5e0 0x7f58d406faa0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:27.857 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.855+0000 7f58f8a00640 1 --2- 172.21.15.96:0/11687589 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f58f40ca150 0x7f58f40b3f90 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:27.857 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.855+0000 7f58f8a00640 1 --2- 172.21.15.96:0/11687589 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f58f40cf2f0 0x7f58f41f45c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:27.857 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.855+0000 7f58f8a00640 1 --2- 172.21.15.96:0/11687589 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58f406edf0 0x7f58f4141300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:27.858 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.855+0000 7f58f8a00640 1 -- 172.21.15.96:0/11687589 >> 172.21.15.96:0/11687589 conn(0x7f58f40b1e50 msgr2=0x7f58f4073530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:27.858 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.855+0000 7f58f8a00640 1 -- 172.21.15.96:0/11687589 shutdown_connections 2024-05-28T06:14:27.858 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:27.855+0000 7f58f8a00640 1 -- 172.21.15.96:0/11687589 wait complete. 2024-05-28T06:14:27.879 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone81712 2024-05-28T06:14:28.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.062+0000 7fc68f200640 1 Processor -- start 2024-05-28T06:14:28.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.063+0000 7fc68f200640 1 -- start start 2024-05-28T06:14:28.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.064+0000 7fc68f200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fc6900cfb40 0x7fc6900d0d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:28.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.064+0000 7fc68f200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6900d2200 con 0x7fc69015e5a0 2024-05-28T06:14:28.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.064+0000 7fc68f200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6900d24e0 con 0x7fc6900cffd0 2024-05-28T06:14:28.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.064+0000 7fc68f200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fc6900d2b30 con 0x7fc6900cfb40 2024-05-28T06:14:28.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.064+0000 7fc68de00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fc69015e5a0 0x7fc6900d2fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:54836/0 (socket says 172.21.15.96:54836) 2024-05-28T06:14:28.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.064+0000 7fc68de00640 1 -- 172.21.15.96:0/1718309958 learned_addr learned my addr 172.21.15.96:0/1718309958 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:14:28.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.064+0000 7fc68d400640 1 --2- 172.21.15.96:0/1718309958 >> v2:172.21.15.96:3300/0 conn(0x7fc6900cfb40 0x7fc6900d0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:28.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.065+0000 7fc68ca00640 1 -- 172.21.15.96:0/1718309958 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1127117861 0 0) 0x7fc6900d2200 con 0x7fc69015e5a0 2024-05-28T06:14:28.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.065+0000 7fc68d400640 1 -- 172.21.15.96:0/1718309958 >> v1:172.21.15.123:6789/0 conn(0x7fc6900cffd0 legacy=0x7fc69006c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:28.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.065+0000 7fc68d400640 1 -- 172.21.15.96:0/1718309958 >> v1:172.21.15.96:6789/0 conn(0x7fc69015e5a0 legacy=0x7fc6900d2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:28.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.065+0000 7fc68d400640 1 -- 172.21.15.96:0/1718309958 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6900d27c0 con 0x7fc6900cfb40 2024-05-28T06:14:28.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.066+0000 7fc68d400640 1 --2- 172.21.15.96:0/1718309958 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc6900cfb40 0x7fc6900d0d10 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7fc6780097a0 tx=0x7fc67802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44df4223a7629601 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:28.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.066+0000 7fc68ca00640 1 -- 172.21.15.96:0/1718309958 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc678037070 con 0x7fc6900cfb40 2024-05-28T06:14:28.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.066+0000 7fc68ca00640 1 -- 172.21.15.96:0/1718309958 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc678032440 con 0x7fc6900cfb40 2024-05-28T06:14:28.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.066+0000 7fc68ca00640 1 -- 172.21.15.96:0/1718309958 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc678032810 con 0x7fc6900cfb40 2024-05-28T06:14:28.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.067+0000 7fc68f200640 1 -- 172.21.15.96:0/1718309958 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc6900cfb40 msgr2=0x7fc6900d0d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:28.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.067+0000 7fc68f200640 1 --2- 172.21.15.96:0/1718309958 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc6900cfb40 0x7fc6900d0d10 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7fc6780097a0 tx=0x7fc67802a1e0 comp rx=0 tx=0).stop 2024-05-28T06:14:28.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.067+0000 7fc68f200640 1 -- 172.21.15.96:0/1718309958 shutdown_connections 2024-05-28T06:14:28.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.067+0000 7fc68f200640 1 --2- 172.21.15.96:0/1718309958 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc6900cfb40 0x7fc6900d0d10 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:28.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.067+0000 7fc68f200640 1 -- 172.21.15.96:0/1718309958 >> 172.21.15.96:0/1718309958 conn(0x7fc69008bfc0 msgr2=0x7fc6900d44b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:28.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.067+0000 7fc68f200640 1 -- 172.21.15.96:0/1718309958 shutdown_connections 2024-05-28T06:14:28.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.067+0000 7fc68f200640 1 -- 172.21.15.96:0/1718309958 wait complete. 2024-05-28T06:14:28.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.069+0000 7fc68f200640 1 Processor -- start 2024-05-28T06:14:28.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.069+0000 7fc68f200640 1 -- start start 2024-05-28T06:14:28.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.070+0000 7fc68f200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc6900cfb40 0x7fc6900b80d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:28.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.070+0000 7fc68f200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc6900cffd0 0x7fc6900ce100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:28.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.070+0000 7fc68f200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc69015e5a0 0x7fc6900cb7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:28.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.070+0000 7fc68f200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc6900d53a0 con 0x7fc6900cffd0 2024-05-28T06:14:28.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.070+0000 7fc68f200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc6900d5220 con 0x7fc6900cfb40 2024-05-28T06:14:28.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.070+0000 7fc68f200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fc6900d1250 con 0x7fc69015e5a0 2024-05-28T06:14:28.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.070+0000 7fc68d400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc6900cffd0 0x7fc6900ce100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:28.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.070+0000 7fc68d400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc6900cffd0 0x7fc6900ce100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48704/0 (socket says 172.21.15.96:48704) 2024-05-28T06:14:28.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.070+0000 7fc68d400640 1 -- 172.21.15.96:0/4153906329 learned_addr learned my addr 172.21.15.96:0/4153906329 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:28.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.070+0000 7fc68d400640 1 -- 172.21.15.96:0/4153906329 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc69015e5a0 msgr2=0x7fc6900cb7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:28.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.070+0000 7fc68d400640 1 --2- 172.21.15.96:0/4153906329 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc69015e5a0 0x7fc6900cb7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:28.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.070+0000 7fc68d400640 1 -- 172.21.15.96:0/4153906329 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc6900cfb40 msgr2=0x7fc6900b80d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:28.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.070+0000 7fc68d400640 1 --2- 172.21.15.96:0/4153906329 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc6900cfb40 0x7fc6900b80d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:28.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.070+0000 7fc68d400640 1 -- 172.21.15.96:0/4153906329 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6900b87d0 con 0x7fc6900cffd0 2024-05-28T06:14:28.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.071+0000 7fc68d400640 1 --2- 172.21.15.96:0/4153906329 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc6900cffd0 0x7fc6900ce100 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fc678035040 tx=0x7fc6780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:28.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.071+0000 7fc686000640 1 -- 172.21.15.96:0/4153906329 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc678037040 con 0x7fc6900cffd0 2024-05-28T06:14:28.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.071+0000 7fc686000640 1 -- 172.21.15.96:0/4153906329 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc678004250 con 0x7fc6900cffd0 2024-05-28T06:14:28.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.071+0000 7fc686000640 1 -- 172.21.15.96:0/4153906329 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc678032e20 con 0x7fc6900cffd0 2024-05-28T06:14:28.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.071+0000 7fc68f200640 1 -- 172.21.15.96:0/4153906329 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6900b8ab0 con 0x7fc6900cffd0 2024-05-28T06:14:28.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.071+0000 7fc68f200640 1 -- 172.21.15.96:0/4153906329 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6900b9070 con 0x7fc6900cffd0 2024-05-28T06:14:28.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.072+0000 7fc68f200640 1 -- 172.21.15.96:0/4153906329 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6900689e0 con 0x7fc6900cffd0 2024-05-28T06:14:28.075 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.073+0000 7fc686000640 1 -- 172.21.15.96:0/4153906329 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 22) ==== 92213+0+0 (secure 0 0 0) 0x7fc678030770 con 0x7fc6900cffd0 2024-05-28T06:14:28.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.074+0000 7fc686000640 1 --2- 172.21.15.96:0/4153906329 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc66c06d5e0 0x7fc66c06faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:28.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.074+0000 7fc686000640 1 -- 172.21.15.96:0/4153906329 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5793+0+0 (secure 0 0 0) 0x7fc6780aba00 con 0x7fc6900cffd0 2024-05-28T06:14:28.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.075+0000 7fc68de00640 1 --2- 172.21.15.96:0/4153906329 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc66c06d5e0 0x7fc66c06faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:28.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.076+0000 7fc68de00640 1 --2- 172.21.15.96:0/4153906329 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc66c06d5e0 0x7fc66c06faa0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fc680007fd0 tx=0x7fc680007960 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:28.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.079+0000 7fc686000640 1 -- 172.21.15.96:0/4153906329 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc67807b9a0 con 0x7fc6900cffd0 2024-05-28T06:14:28.308 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.306+0000 7fc68f200640 1 -- 172.21.15.96:0/4153906329 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone81712", "target": ["mon-mgr", ""]}) -- 0x7fc6900d1d30 con 0x7fc66c06d5e0 2024-05-28T06:14:28.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.454+0000 7fc686000640 1 -- 172.21.15.96:0/4153906329 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fc6900d1d30 con 0x7fc66c06d5e0 2024-05-28T06:14:28.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.457+0000 7fc68f200640 1 -- 172.21.15.96:0/4153906329 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc66c06d5e0 msgr2=0x7fc66c06faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:28.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.457+0000 7fc68f200640 1 --2- 172.21.15.96:0/4153906329 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc66c06d5e0 0x7fc66c06faa0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fc680007fd0 tx=0x7fc680007960 comp rx=0 tx=0).stop 2024-05-28T06:14:28.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.457+0000 7fc68f200640 1 -- 172.21.15.96:0/4153906329 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc6900cffd0 msgr2=0x7fc6900ce100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:28.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.457+0000 7fc68f200640 1 --2- 172.21.15.96:0/4153906329 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc6900cffd0 0x7fc6900ce100 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fc678035040 tx=0x7fc6780046c0 comp rx=0 tx=0).stop 2024-05-28T06:14:28.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.457+0000 7fc68f200640 1 -- 172.21.15.96:0/4153906329 shutdown_connections 2024-05-28T06:14:28.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.457+0000 7fc68f200640 1 --2- 172.21.15.96:0/4153906329 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc66c06d5e0 0x7fc66c06faa0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:28.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.457+0000 7fc68f200640 1 --2- 172.21.15.96:0/4153906329 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc69015e5a0 0x7fc6900cb7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:28.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.457+0000 7fc68f200640 1 --2- 172.21.15.96:0/4153906329 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc6900cffd0 0x7fc6900ce100 secure :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fc678035040 tx=0x7fc6780046c0 comp rx=0 tx=0).stop 2024-05-28T06:14:28.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.457+0000 7fc68f200640 1 --2- 172.21.15.96:0/4153906329 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc6900cfb40 0x7fc6900b80d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:28.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.457+0000 7fc68f200640 1 -- 172.21.15.96:0/4153906329 >> 172.21.15.96:0/4153906329 conn(0x7fc69008bfc0 msgr2=0x7fc6900b6880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:28.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.458+0000 7fc68f200640 1 -- 172.21.15.96:0/4153906329 shutdown_connections 2024-05-28T06:14:28.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.458+0000 7fc68f200640 1 -- 172.21.15.96:0/4153906329 wait complete. 2024-05-28T06:14:28.486 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone81716 2024-05-28T06:14:28.693 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.691+0000 7fd073200640 1 Processor -- start 2024-05-28T06:14:28.693 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.692+0000 7fd073200640 1 -- start start 2024-05-28T06:14:28.694 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.693+0000 7fd073200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd06c072310 0x7fd06c06dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:28.694 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.693+0000 7fd073200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd06c06e190 con 0x7fd06c15e830 2024-05-28T06:14:28.696 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.693+0000 7fd073200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd06c06e470 con 0x7fd06c0b3510 2024-05-28T06:14:28.696 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.693+0000 7fd073200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fd06c06e750 con 0x7fd06c072310 2024-05-28T06:14:28.696 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.693+0000 7fd06be00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fd06c15e830 0x7fd06c071c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:54852/0 (socket says 172.21.15.96:54852) 2024-05-28T06:14:28.696 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.693+0000 7fd06be00640 1 -- 172.21.15.96:0/1368737 learned_addr learned my addr 172.21.15.96:0/1368737 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:14:28.698 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.694+0000 7fd063400640 1 --2- 172.21.15.96:0/1368737 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd06c072310 0x7fd06c06dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:28.698 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.694+0000 7fd06b400640 1 -- 172.21.15.96:0/1368737 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3606301931 0 0) 0x7fd06c06e190 con 0x7fd06c15e830 2024-05-28T06:14:28.698 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.694+0000 7fd06b400640 1 -- 172.21.15.96:0/1368737 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd04c003520 con 0x7fd06c15e830 2024-05-28T06:14:28.698 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.694+0000 7fd06b400640 1 -- 172.21.15.96:0/1368737 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3921838606 0 0) 0x7fd04c003520 con 0x7fd06c15e830 2024-05-28T06:14:28.698 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.694+0000 7fd06b400640 1 -- 172.21.15.96:0/1368737 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd06c072310 msgr2=0x7fd06c06dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:28.698 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.694+0000 7fd06b400640 1 --2- 172.21.15.96:0/1368737 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd06c072310 0x7fd06c06dbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:28.698 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.694+0000 7fd06b400640 1 -- 172.21.15.96:0/1368737 >> v1:172.21.15.123:6789/0 conn(0x7fd06c0b3510 legacy=0x7fd06c0ad130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:28.698 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.694+0000 7fd06b400640 1 -- 172.21.15.96:0/1368737 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd06c06ea30 con 0x7fd06c15e830 2024-05-28T06:14:28.698 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.694+0000 7fd06b400640 1 -- 172.21.15.96:0/1368737 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7fd05c001670 con 0x7fd06c15e830 2024-05-28T06:14:28.698 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.694+0000 7fd06b400640 1 -- 172.21.15.96:0/1368737 >> v1:172.21.15.96:6789/0 conn(0x7fd06c15e830 legacy=0x7fd06c071c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:28.698 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.694+0000 7fd06b400640 1 --2- 172.21.15.96:0/1368737 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd06c072310 0x7fd04c0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:28.699 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.694+0000 7fd06b400640 1 --2- 172.21.15.96:0/1368737 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd04c003f60 0x7fd04c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:28.699 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.695+0000 7fd06b400640 1 --2- 172.21.15.96:0/1368737 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd04c004d40 0x7fd04c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:28.699 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.695+0000 7fd06b400640 1 -- 172.21.15.96:0/1368737 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd06c06e750 con 0x7fd04c003f60 2024-05-28T06:14:28.699 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.695+0000 7fd06b400640 1 -- 172.21.15.96:0/1368737 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd04c00baa0 con 0x7fd04c004d40 2024-05-28T06:14:28.699 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.695+0000 7fd06b400640 1 -- 172.21.15.96:0/1368737 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fd04c00bca0 con 0x7fd06c072310 2024-05-28T06:14:28.699 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.695+0000 7fd063400640 1 --2- 172.21.15.96:0/1368737 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd04c003f60 0x7fd04c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:28.699 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.695+0000 7fd071200640 1 --2- 172.21.15.96:0/1368737 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd04c004d40 0x7fd04c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:28.699 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.695+0000 7fd06be00640 1 --2- 172.21.15.96:0/1368737 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd06c072310 0x7fd04c0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:28.699 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.695+0000 7fd063400640 1 -- 172.21.15.96:0/1368737 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd06c072310 msgr2=0x7fd04c0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:28.700 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.695+0000 7fd063400640 1 --2- 172.21.15.96:0/1368737 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd06c072310 0x7fd04c0039e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:28.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.695+0000 7fd063400640 1 -- 172.21.15.96:0/1368737 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd04c004d40 msgr2=0x7fd04c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:28.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.695+0000 7fd063400640 1 --2- 172.21.15.96:0/1368737 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd04c004d40 0x7fd04c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:28.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.695+0000 7fd063400640 1 -- 172.21.15.96:0/1368737 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd04c003520 con 0x7fd04c003f60 2024-05-28T06:14:28.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.695+0000 7fd071200640 1 --2- 172.21.15.96:0/1368737 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd04c004d40 0x7fd04c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:14:28.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.696+0000 7fd063400640 1 --2- 172.21.15.96:0/1368737 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd04c003f60 0x7fd04c004800 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7fd054009ac0 tx=0x7fd04c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=316bf13205a10472 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:28.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.697+0000 7fd06b400640 1 -- 172.21.15.96:0/1368737 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd054036070 con 0x7fd04c003f60 2024-05-28T06:14:28.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.697+0000 7fd06b400640 1 -- 172.21.15.96:0/1368737 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd054031400 con 0x7fd04c003f60 2024-05-28T06:14:28.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.697+0000 7fd06b400640 1 -- 172.21.15.96:0/1368737 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd054009690 con 0x7fd04c003f60 2024-05-28T06:14:28.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.697+0000 7fd073200640 1 -- 172.21.15.96:0/1368737 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd04c003f60 msgr2=0x7fd04c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:28.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.697+0000 7fd073200640 1 --2- 172.21.15.96:0/1368737 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd04c003f60 0x7fd04c004800 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7fd054009ac0 tx=0x7fd04c005830 comp rx=0 tx=0).stop 2024-05-28T06:14:28.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.697+0000 7fd071200640 1 -- 172.21.15.96:0/1368737 reap_dead start 2024-05-28T06:14:28.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.697+0000 7fd073200640 1 -- 172.21.15.96:0/1368737 shutdown_connections 2024-05-28T06:14:28.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.697+0000 7fd073200640 1 -- 172.21.15.96:0/1368737 >> 172.21.15.96:0/1368737 conn(0x7fd06c0b1e50 msgr2=0x7fd06c0b2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:28.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.697+0000 7fd073200640 1 -- 172.21.15.96:0/1368737 shutdown_connections 2024-05-28T06:14:28.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.697+0000 7fd073200640 1 -- 172.21.15.96:0/1368737 wait complete. 2024-05-28T06:14:28.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.699+0000 7fd073200640 1 Processor -- start 2024-05-28T06:14:28.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.699+0000 7fd073200640 1 -- start start 2024-05-28T06:14:28.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.700+0000 7fd073200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd04c003f60 0x7fd06c0b0f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:28.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.700+0000 7fd073200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd06c15c3a0 0x7fd06c0cb030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:28.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.700+0000 7fd073200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd06c15c800 0x7fd06c0cb570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:28.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.700+0000 7fd073200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd06c0728c0 con 0x7fd06c15c800 2024-05-28T06:14:28.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.700+0000 7fd073200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd06c15ec10 con 0x7fd04c003f60 2024-05-28T06:14:28.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.700+0000 7fd073200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fd06c072a40 con 0x7fd06c15c3a0 2024-05-28T06:14:28.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.700+0000 7fd071200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd06c15c800 0x7fd06c0cb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:28.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.700+0000 7fd071200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd06c15c800 0x7fd06c0cb570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48718/0 (socket says 172.21.15.96:48718) 2024-05-28T06:14:28.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.700+0000 7fd071200640 1 -- 172.21.15.96:0/72220727 learned_addr learned my addr 172.21.15.96:0/72220727 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:28.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.701+0000 7fd071200640 1 -- 172.21.15.96:0/72220727 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd06c15c3a0 msgr2=0x7fd06c0cb030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:28.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.701+0000 7fd063400640 1 --2- 172.21.15.96:0/72220727 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd06c15c3a0 0x7fd06c0cb030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:28.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.701+0000 7fd06be00640 1 --2- 172.21.15.96:0/72220727 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd04c003f60 0x7fd06c0b0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:28.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.701+0000 7fd071200640 1 --2- 172.21.15.96:0/72220727 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd06c15c3a0 0x7fd06c0cb030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:28.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.701+0000 7fd071200640 1 -- 172.21.15.96:0/72220727 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd04c003f60 msgr2=0x7fd06c0b0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:28.705 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.701+0000 7fd071200640 1 --2- 172.21.15.96:0/72220727 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd04c003f60 0x7fd06c0b0f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:28.705 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.701+0000 7fd071200640 1 -- 172.21.15.96:0/72220727 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd06c0b15e0 con 0x7fd06c15c800 2024-05-28T06:14:28.705 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.701+0000 7fd071200640 1 --2- 172.21.15.96:0/72220727 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd06c15c800 0x7fd06c0cb570 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fd06c0b61a0 tx=0x7fd06400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:28.705 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.701+0000 7fd068c00640 1 -- 172.21.15.96:0/72220727 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd064006d50 con 0x7fd06c15c800 2024-05-28T06:14:28.705 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.701+0000 7fd068c00640 1 -- 172.21.15.96:0/72220727 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd064003970 con 0x7fd06c15c800 2024-05-28T06:14:28.705 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.701+0000 7fd073200640 1 -- 172.21.15.96:0/72220727 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd06c0b18c0 con 0x7fd06c15c800 2024-05-28T06:14:28.705 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.701+0000 7fd068c00640 1 -- 172.21.15.96:0/72220727 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd064003da0 con 0x7fd06c15c800 2024-05-28T06:14:28.705 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.702+0000 7fd073200640 1 -- 172.21.15.96:0/72220727 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd06c0c9570 con 0x7fd06c15c800 2024-05-28T06:14:28.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.703+0000 7fd073200640 1 -- 172.21.15.96:0/72220727 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd06c0689e0 con 0x7fd06c15c800 2024-05-28T06:14:28.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.704+0000 7fd068c00640 1 -- 172.21.15.96:0/72220727 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 22) ==== 92213+0+0 (secure 0 0 0) 0x7fd06401b020 con 0x7fd06c15c800 2024-05-28T06:14:28.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.705+0000 7fd068c00640 1 --2- 172.21.15.96:0/72220727 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd04806d610 0x7fd04806fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:28.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.705+0000 7fd068c00640 1 -- 172.21.15.96:0/72220727 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5793+0+0 (secure 0 0 0) 0x7fd06408a510 con 0x7fd06c15c800 2024-05-28T06:14:28.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.705+0000 7fd06be00640 1 --2- 172.21.15.96:0/72220727 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd04806d610 0x7fd04806fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:28.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.706+0000 7fd06be00640 1 --2- 172.21.15.96:0/72220727 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd04806d610 0x7fd04806fad0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fd05c007950 tx=0x7fd05c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:28.710 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.709+0000 7fd068c00640 1 -- 172.21.15.96:0/72220727 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd06401ce40 con 0x7fd06c15c800 2024-05-28T06:14:28.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.942+0000 7fd073200640 1 -- 172.21.15.96:0/72220727 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone81716", "target": ["mon-mgr", ""]}) -- 0x7fd06c0b4970 con 0x7fd04806d610 2024-05-28T06:14:28.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.994+0000 7fd068c00640 1 -- 172.21.15.96:0/72220727 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fd06c0b4970 con 0x7fd04806d610 2024-05-28T06:14:29.000 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.998+0000 7fd073200640 1 -- 172.21.15.96:0/72220727 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd04806d610 msgr2=0x7fd04806fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:29.000 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.998+0000 7fd073200640 1 --2- 172.21.15.96:0/72220727 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd04806d610 0x7fd04806fad0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fd05c007950 tx=0x7fd05c009040 comp rx=0 tx=0).stop 2024-05-28T06:14:29.000 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.998+0000 7fd073200640 1 -- 172.21.15.96:0/72220727 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd06c15c800 msgr2=0x7fd06c0cb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:29.000 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.998+0000 7fd073200640 1 --2- 172.21.15.96:0/72220727 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd06c15c800 0x7fd06c0cb570 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fd06c0b61a0 tx=0x7fd06400dc00 comp rx=0 tx=0).stop 2024-05-28T06:14:29.000 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.998+0000 7fd073200640 1 -- 172.21.15.96:0/72220727 shutdown_connections 2024-05-28T06:14:29.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.998+0000 7fd073200640 1 --2- 172.21.15.96:0/72220727 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd04806d610 0x7fd04806fad0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:29.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.998+0000 7fd073200640 1 --2- 172.21.15.96:0/72220727 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd06c15c800 0x7fd06c0cb570 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:29.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.998+0000 7fd073200640 1 --2- 172.21.15.96:0/72220727 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd06c15c3a0 0x7fd06c0cb030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:29.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.998+0000 7fd073200640 1 --2- 172.21.15.96:0/72220727 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd04c003f60 0x7fd06c0b0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:29.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.998+0000 7fd073200640 1 -- 172.21.15.96:0/72220727 >> 172.21.15.96:0/72220727 conn(0x7fd06c0b1e50 msgr2=0x7fd06c162410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:29.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.998+0000 7fd073200640 1 -- 172.21.15.96:0/72220727 shutdown_connections 2024-05-28T06:14:29.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:28.999+0000 7fd073200640 1 -- 172.21.15.96:0/72220727 wait complete. 2024-05-28T06:14:29.024 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2024-05-28T06:14:29.120 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:29 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-28T06:14:29.133 INFO:teuthology.orchestra.run.smithi096.stdout:2 2024-05-28T06:14:29.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-28T06:14:29.144 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-05-28T06:14:29.144 DEBUG:teuthology.orchestra.run.smithi096:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-05-28T06:14:29.247 DEBUG:teuthology.orchestra.run.smithi123:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-05-28T06:14:29.366 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-05-28T06:14:29.367 INFO:tasks.cephfs.mount:Terminating background process 2024-05-28T06:14:29.367 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:14:29.367 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:14:29.422 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:14:29.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:29.625+0000 7f78f3200640 1 Processor -- start 2024-05-28T06:14:29.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:29.625+0000 7f78f3200640 1 -- start start 2024-05-28T06:14:29.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:29.626+0000 7f78f3200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f78ec0d04a0 0x7f78ec0d0900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:29.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:29.626+0000 7f78f3200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f78ec0ad130 0x7f78ec0ad5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:29.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:29.626+0000 7f78f3200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78ec06ed10 con 0x7f78ec06f860 2024-05-28T06:14:29.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:29.626+0000 7f78f3200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f78ec0d0e40 con 0x7f78ec0ad130 2024-05-28T06:14:29.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:29.626+0000 7f78f3200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f78ec15cb00 con 0x7f78ec0d04a0 2024-05-28T06:14:29.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:29.627+0000 7f78f1200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f78ec0ad130 0x7f78ec0ad5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:29.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:29.627+0000 7f78f1200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f78ec0ad130 0x7f78ec0ad5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:43118/0 (socket says 172.21.15.96:43118) 2024-05-28T06:14:29.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:29.627+0000 7f78f1200640 1 -- 172.21.15.96:0/2679913910 learned_addr learned my addr 172.21.15.96:0/2679913910 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:29.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:29.627+0000 7f78eb400640 1 --2- 172.21.15.96:0/2679913910 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f78ec0d04a0 0x7f78ec0d0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:29.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:29.627+0000 7f78f1200640 1 -- 172.21.15.96:0/2679913910 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f78ec0d04a0 msgr2=0x7f78ec0d0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:29.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:29.627+0000 7f78f1200640 1 --2- 172.21.15.96:0/2679913910 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f78ec0d04a0 0x7f78ec0d0900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:29.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:29.627+0000 7f78f1200640 1 -- 172.21.15.96:0/2679913910 >> v1:172.21.15.123:6789/0 conn(0x7f78ec06f860 legacy=0x7f78ec0b3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:29.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:29.627+0000 7f78f1200640 1 -- 172.21.15.96:0/2679913910 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78ec0b3cc0 con 0x7f78ec0ad130 2024-05-28T06:14:29.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:29.629+0000 7f78f1200640 1 --2- 172.21.15.96:0/2679913910 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f78ec0ad130 0x7f78ec0ad5a0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f78e4009e50 tx=0x7f78e402a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72702aaffa462481 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:29.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:29.629+0000 7f78eaa00640 1 -- 172.21.15.96:0/2679913910 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78e4036070 con 0x7f78ec0ad130 2024-05-28T06:14:29.631 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:29.629+0000 7f78eaa00640 1 -- 172.21.15.96:0/2679913910 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78e4031600 con 0x7f78ec0ad130 2024-05-28T06:14:29.631 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:29.629+0000 7f78eaa00640 1 -- 172.21.15.96:0/2679913910 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78e40319d0 con 0x7f78ec0ad130 2024-05-28T06:14:29.631 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:29.630+0000 7f78f3200640 1 -- 172.21.15.96:0/2679913910 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f78ec0ad130 msgr2=0x7f78ec0ad5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:29.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:29.630+0000 7f78f3200640 1 --2- 172.21.15.96:0/2679913910 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f78ec0ad130 0x7f78ec0ad5a0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f78e4009e50 tx=0x7f78e402a6c0 comp rx=0 tx=0).stop 2024-05-28T06:14:29.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:29.630+0000 7f78f3200640 1 -- 172.21.15.96:0/2679913910 shutdown_connections 2024-05-28T06:14:29.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:29.630+0000 7f78f3200640 1 --2- 172.21.15.96:0/2679913910 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f78ec0ad130 0x7f78ec0ad5a0 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:29.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:29.630+0000 7f78f3200640 1 --2- 172.21.15.96:0/2679913910 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f78ec0d04a0 0x7f78ec0d0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:29.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:29.630+0000 7f78f3200640 1 -- 172.21.15.96:0/2679913910 >> 172.21.15.96:0/2679913910 conn(0x7f78ec06bae0 msgr2=0x7f78ec06bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:29.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:29.630+0000 7f78f3200640 1 -- 172.21.15.96:0/2679913910 shutdown_connections 2024-05-28T06:14:29.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:29.630+0000 7f78f3200640 1 -- 172.21.15.96:0/2679913910 wait complete. 2024-05-28T06:14:29.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:29.632+0000 7f78f3200640 1 Processor -- start 2024-05-28T06:14:29.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:29.632+0000 7f78f3200640 1 -- start start 2024-05-28T06:14:29.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:29.634+0000 7f78f3200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f78ec06f860 0x7f78ec0997e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:29.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:29.634+0000 7f78f3200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f78ec0ad130 0x7f78ec15e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:29.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:29.634+0000 7f78f3200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f78ec0d04a0 0x7f78ec15ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:29.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:29.634+0000 7f78f3200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f78ec0b4560 con 0x7f78ec0d04a0 2024-05-28T06:14:29.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:29.634+0000 7f78f3200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f78ec0b4280 con 0x7f78ec06f860 2024-05-28T06:14:29.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:29.634+0000 7f78f3200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f78ec0b4840 con 0x7f78ec0ad130 2024-05-28T06:14:29.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:29.634+0000 7f78f1200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f78ec0d04a0 0x7f78ec15ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:29.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:29.634+0000 7f78f1200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f78ec0d04a0 0x7f78ec15ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:43130/0 (socket says 172.21.15.96:43130) 2024-05-28T06:14:29.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:29.634+0000 7f78f1200640 1 -- 172.21.15.96:0/2954204571 learned_addr learned my addr 172.21.15.96:0/2954204571 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:29.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:29.634+0000 7f78ebe00640 1 --2- 172.21.15.96:0/2954204571 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f78ec06f860 0x7f78ec0997e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:29.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:29.634+0000 7f78eb400640 1 --2- 172.21.15.96:0/2954204571 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f78ec0ad130 0x7f78ec15e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:29.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:29.634+0000 7f78f1200640 1 -- 172.21.15.96:0/2954204571 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f78ec0ad130 msgr2=0x7f78ec15e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:29.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:29.634+0000 7f78f1200640 1 --2- 172.21.15.96:0/2954204571 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f78ec0ad130 0x7f78ec15e810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:29.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:29.634+0000 7f78f1200640 1 -- 172.21.15.96:0/2954204571 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f78ec06f860 msgr2=0x7f78ec0997e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:29.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:29.634+0000 7f78f1200640 1 --2- 172.21.15.96:0/2954204571 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f78ec06f860 0x7f78ec0997e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:29.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:29.634+0000 7f78f1200640 1 -- 172.21.15.96:0/2954204571 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78ec072a10 con 0x7f78ec0d04a0 2024-05-28T06:14:29.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:29.635+0000 7f78f1200640 1 --2- 172.21.15.96:0/2954204571 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f78ec0d04a0 0x7f78ec15ed50 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f78e4009e50 tx=0x7f78e4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:29.638 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:29.635+0000 7f78e3e00640 1 -- 172.21.15.96:0/2954204571 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78e4036050 con 0x7f78ec0d04a0 2024-05-28T06:14:29.638 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:29.635+0000 7f78e3e00640 1 -- 172.21.15.96:0/2954204571 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78e4042070 con 0x7f78ec0d04a0 2024-05-28T06:14:29.638 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:29.635+0000 7f78e3e00640 1 -- 172.21.15.96:0/2954204571 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78e4009510 con 0x7f78ec0d04a0 2024-05-28T06:14:29.638 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:29.635+0000 7f78f3200640 1 -- 172.21.15.96:0/2954204571 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78ec15a3a0 con 0x7f78ec0d04a0 2024-05-28T06:14:29.638 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:29.635+0000 7f78f3200640 1 -- 172.21.15.96:0/2954204571 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78ec15f900 con 0x7f78ec0d04a0 2024-05-28T06:14:29.639 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:29.636+0000 7f78f3200640 1 -- 172.21.15.96:0/2954204571 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78ec0689e0 con 0x7f78ec0d04a0 2024-05-28T06:14:29.639 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:29.637+0000 7f78e3e00640 1 -- 172.21.15.96:0/2954204571 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 22) ==== 92213+0+0 (secure 0 0 0) 0x7f78e4007bd0 con 0x7f78ec0d04a0 2024-05-28T06:14:29.640 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:29.638+0000 7f78e3e00640 1 --2- 172.21.15.96:0/2954204571 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f78cc06d5e0 0x7f78cc06faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:29.640 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:29.638+0000 7f78e3e00640 1 -- 172.21.15.96:0/2954204571 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5793+0+0 (secure 0 0 0) 0x7f78e40ab4a0 con 0x7f78ec0d04a0 2024-05-28T06:14:29.640 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:29.639+0000 7f78ebe00640 1 --2- 172.21.15.96:0/2954204571 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f78cc06d5e0 0x7f78cc06faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:29.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:29.640+0000 7f78ebe00640 1 --2- 172.21.15.96:0/2954204571 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f78cc06d5e0 0x7f78cc06faa0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f78dc00a750 tx=0x7f78dc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:29.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:29.643+0000 7f78e3e00640 1 -- 172.21.15.96:0/2954204571 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f78ec06ed10 con 0x7f78ec0d04a0 2024-05-28T06:14:29.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:29.846+0000 7f78f3200640 1 -- 172.21.15.96:0/2954204571 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f78ec0c3520 con 0x7f78ec0d04a0 2024-05-28T06:14:29.850 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:29.848+0000 7f78e3e00640 1 -- 172.21.15.96:0/2954204571 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v61) ==== 74+0+16829 (secure 0 0 0) 0x7f78e407b290 con 0x7f78ec0d04a0 2024-05-28T06:14:29.851 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:14:29.851 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":61,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:14:22.629235+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-05-28T06:12:26.024060+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-05-28T06:12:27.254401+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":61,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":9,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[{"pool":9,"snaps":[{"begin":2,"length":2}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:14:29.852 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:29.851+0000 7f78f3200640 1 -- 172.21.15.96:0/2954204571 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f78cc06d5e0 msgr2=0x7f78cc06faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:29.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:29.851+0000 7f78f3200640 1 --2- 172.21.15.96:0/2954204571 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f78cc06d5e0 0x7f78cc06faa0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f78dc00a750 tx=0x7f78dc008040 comp rx=0 tx=0).stop 2024-05-28T06:14:29.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:29.851+0000 7f78f3200640 1 -- 172.21.15.96:0/2954204571 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f78ec0d04a0 msgr2=0x7f78ec15ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:29.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:29.851+0000 7f78f3200640 1 --2- 172.21.15.96:0/2954204571 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f78ec0d04a0 0x7f78ec15ed50 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f78e4009e50 tx=0x7f78e4004a40 comp rx=0 tx=0).stop 2024-05-28T06:14:29.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:29.851+0000 7f78f3200640 1 -- 172.21.15.96:0/2954204571 shutdown_connections 2024-05-28T06:14:29.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:29.851+0000 7f78f3200640 1 --2- 172.21.15.96:0/2954204571 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f78cc06d5e0 0x7f78cc06faa0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:29.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:29.851+0000 7f78f3200640 1 --2- 172.21.15.96:0/2954204571 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f78ec0d04a0 0x7f78ec15ed50 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:29.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:29.851+0000 7f78f3200640 1 --2- 172.21.15.96:0/2954204571 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f78ec0ad130 0x7f78ec15e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:29.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:29.851+0000 7f78f3200640 1 --2- 172.21.15.96:0/2954204571 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f78ec06f860 0x7f78ec0997e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:29.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:29.851+0000 7f78f3200640 1 -- 172.21.15.96:0/2954204571 >> 172.21.15.96:0/2954204571 conn(0x7f78ec06bae0 msgr2=0x7f78ec071b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:29.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:29.851+0000 7f78f3200640 1 -- 172.21.15.96:0/2954204571 shutdown_connections 2024-05-28T06:14:29.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:29.851+0000 7f78f3200640 1 -- 172.21.15.96:0/2954204571 wait complete. 2024-05-28T06:14:29.872 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-28T06:03:59.517102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-28T06:12:26.024060+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '59', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-05-28T06:12:27.254401+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '61', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 61, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-28T06:14:29.873 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-28T06:14:30.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.087+0000 7f907cc00640 1 Processor -- start 2024-05-28T06:14:30.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.087+0000 7f907cc00640 1 -- start start 2024-05-28T06:14:30.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.089+0000 7f907cc00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f90780d0640 0x7f907815d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:30.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.089+0000 7f907cc00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f90780d0a20 0x7f90780cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:30.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.089+0000 7f907cc00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f90780b32c0 0x7f90780b3750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:30.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.089+0000 7f907cc00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f90780adbb0 con 0x7f90780b32c0 2024-05-28T06:14:30.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.089+0000 7f907cc00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f907815db10 con 0x7f90780d0a20 2024-05-28T06:14:30.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.089+0000 7f907cc00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f907815cb10 con 0x7f90780d0640 2024-05-28T06:14:30.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.089+0000 7f9076800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f90780b32c0 0x7f90780b3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:30.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.089+0000 7f9075e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f90780d0640 0x7f907815d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:30.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.089+0000 7f906fe00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f90780d0a20 0x7f90780cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:30.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.089+0000 7f9076800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f90780b32c0 0x7f90780b3750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:43144/0 (socket says 172.21.15.96:43144) 2024-05-28T06:14:30.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.089+0000 7f9076800640 1 -- 172.21.15.96:0/2497882561 learned_addr learned my addr 172.21.15.96:0/2497882561 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:30.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.090+0000 7f9076800640 1 -- 172.21.15.96:0/2497882561 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f90780d0640 msgr2=0x7f907815d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:30.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.090+0000 7f9076800640 1 --2- 172.21.15.96:0/2497882561 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f90780d0640 0x7f907815d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:30.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.090+0000 7f9076800640 1 -- 172.21.15.96:0/2497882561 >> v2:172.21.15.123:3300/0 conn(0x7f90780d0a20 msgr2=0x7f90780cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:30.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.090+0000 7f9076800640 1 --2- 172.21.15.96:0/2497882561 >> v2:172.21.15.123:3300/0 conn(0x7f90780d0a20 0x7f90780cf950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:30.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.090+0000 7f9076800640 1 -- 172.21.15.96:0/2497882561 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90780ad5f0 con 0x7f90780b32c0 2024-05-28T06:14:30.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.092+0000 7f9076800640 1 --2- 172.21.15.96:0/2497882561 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f90780b32c0 0x7f90780b3750 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f90780b6580 tx=0x7f907002a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fd77ef9fa4711c4 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:30.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.092+0000 7f9075400640 1 -- 172.21.15.96:0/2497882561 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9070036070 con 0x7f90780b32c0 2024-05-28T06:14:30.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.092+0000 7f9075400640 1 -- 172.21.15.96:0/2497882561 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9070031600 con 0x7f90780b32c0 2024-05-28T06:14:30.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.092+0000 7f9075400640 1 -- 172.21.15.96:0/2497882561 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90700319d0 con 0x7f90780b32c0 2024-05-28T06:14:30.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.093+0000 7f907cc00640 1 -- 172.21.15.96:0/2497882561 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f90780b32c0 msgr2=0x7f90780b3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:30.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.093+0000 7f907cc00640 1 --2- 172.21.15.96:0/2497882561 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f90780b32c0 0x7f90780b3750 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f90780b6580 tx=0x7f907002a6a0 comp rx=0 tx=0).stop 2024-05-28T06:14:30.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.094+0000 7f907cc00640 1 -- 172.21.15.96:0/2497882561 shutdown_connections 2024-05-28T06:14:30.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.094+0000 7f907cc00640 1 --2- 172.21.15.96:0/2497882561 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f90780b32c0 0x7f90780b3750 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:30.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.094+0000 7f907cc00640 1 --2- 172.21.15.96:0/2497882561 >> v2:172.21.15.123:3300/0 conn(0x7f90780d0a20 0x7f90780cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:30.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.094+0000 7f907cc00640 1 --2- 172.21.15.96:0/2497882561 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f90780d0640 0x7f907815d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:30.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.094+0000 7f907cc00640 1 -- 172.21.15.96:0/2497882561 >> 172.21.15.96:0/2497882561 conn(0x7f907806bd20 msgr2=0x7f907806c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:30.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.095+0000 7f907cc00640 1 -- 172.21.15.96:0/2497882561 shutdown_connections 2024-05-28T06:14:30.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.095+0000 7f907cc00640 1 -- 172.21.15.96:0/2497882561 wait complete. 2024-05-28T06:14:30.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.098+0000 7f907cc00640 1 Processor -- start 2024-05-28T06:14:30.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.098+0000 7f907cc00640 1 -- start start 2024-05-28T06:14:30.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.100+0000 7f907cc00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f90780b32c0 0x7f907809dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:30.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.100+0000 7f907cc00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f90780d0640 0x7f907814cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:30.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.100+0000 7f907cc00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f90780d0a20 0x7f907809bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:30.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.100+0000 7f907cc00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f907815d0b0 con 0x7f90780d0a20 2024-05-28T06:14:30.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.100+0000 7f907cc00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f907815cdd0 con 0x7f90780b32c0 2024-05-28T06:14:30.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.100+0000 7f907cc00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f90780ad2a0 con 0x7f90780d0640 2024-05-28T06:14:30.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.100+0000 7f9075e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f90780b32c0 0x7f907809dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:30.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.100+0000 7f9076800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f90780d0a20 0x7f907809bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:30.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.100+0000 7f9075e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f90780b32c0 0x7f907809dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.96:43252/0 (socket says 172.21.15.96:43252) 2024-05-28T06:14:30.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.100+0000 7f9075e00640 1 -- 172.21.15.96:0/1570423071 learned_addr learned my addr 172.21.15.96:0/1570423071 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:30.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.100+0000 7f906fe00640 1 --2- 172.21.15.96:0/1570423071 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f90780d0640 0x7f907814cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:30.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.103+0000 7f906fe00640 1 -- 172.21.15.96:0/1570423071 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f90780b32c0 msgr2=0x7f907809dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:30.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.103+0000 7f906fe00640 1 --2- 172.21.15.96:0/1570423071 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f90780b32c0 0x7f907809dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:30.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.103+0000 7f906fe00640 1 -- 172.21.15.96:0/1570423071 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f90780d0a20 msgr2=0x7f907809bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:30.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.103+0000 7f906fe00640 1 --2- 172.21.15.96:0/1570423071 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f90780d0a20 0x7f907809bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:30.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.103+0000 7f906fe00640 1 -- 172.21.15.96:0/1570423071 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f907814d150 con 0x7f90780d0640 2024-05-28T06:14:30.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.103+0000 7f9076800640 1 --2- 172.21.15.96:0/1570423071 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f90780d0a20 0x7f907809bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-28T06:14:30.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.104+0000 7f906fe00640 1 --2- 172.21.15.96:0/1570423071 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f90780d0640 0x7f907814cc10 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f906000e870 tx=0x7f906000ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:30.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.105+0000 7f906e000640 1 -- 172.21.15.96:0/1570423071 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9060017070 con 0x7f90780d0640 2024-05-28T06:14:30.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.105+0000 7f906e000640 1 -- 172.21.15.96:0/1570423071 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f906000ee60 con 0x7f90780d0640 2024-05-28T06:14:30.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.105+0000 7f906e000640 1 -- 172.21.15.96:0/1570423071 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f906000cb40 con 0x7f90780d0640 2024-05-28T06:14:30.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.105+0000 7f907cc00640 1 -- 172.21.15.96:0/1570423071 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f907814a840 con 0x7f90780d0640 2024-05-28T06:14:30.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.105+0000 7f907cc00640 1 -- 172.21.15.96:0/1570423071 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f907814ae00 con 0x7f90780d0640 2024-05-28T06:14:30.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.107+0000 7f907cc00640 1 -- 172.21.15.96:0/1570423071 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90780689e0 con 0x7f90780d0640 2024-05-28T06:14:30.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.109+0000 7f906e000640 1 -- 172.21.15.96:0/1570423071 <== mon.2 v2:172.21.15.123:3301/0 4 ==== mgrmap(e 22) ==== 92213+0+0 (secure 0 0 0) 0x7f906001a070 con 0x7f90780d0640 2024-05-28T06:14:30.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.111+0000 7f906e000640 1 --2- 172.21.15.96:0/1570423071 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f905406d660 0x7f905406fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:30.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.111+0000 7f906e000640 1 -- 172.21.15.96:0/1570423071 <== mon.2 v2:172.21.15.123:3301/0 5 ==== osd_map(61..61 src has 1..61) ==== 5793+0+0 (secure 0 0 0) 0x7f906008a280 con 0x7f90780d0640 2024-05-28T06:14:30.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.112+0000 7f9075e00640 1 --2- 172.21.15.96:0/1570423071 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f905406d660 0x7f905406fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:30.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.114+0000 7f9075e00640 1 --2- 172.21.15.96:0/1570423071 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f905406d660 0x7f905406fb20 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f907815ea10 tx=0x7f9068009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:30.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.121+0000 7f906e000640 1 -- 172.21.15.96:0/1570423071 <== mon.2 v2:172.21.15.123:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9060059af0 con 0x7f90780d0640 2024-05-28T06:14:30.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.498+0000 7f907cc00640 1 -- 172.21.15.96:0/1570423071 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f9078073070 con 0x7f90780d0640 2024-05-28T06:14:30.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.499+0000 7f906e000640 1 -- 172.21.15.96:0/1570423071 <== mon.2 v2:172.21.15.123:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v61) ==== 93+0+10 (secure 0 0 0) 0x7f906005ddf0 con 0x7f90780d0640 2024-05-28T06:14:30.502 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 1 2024-05-28T06:14:30.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.503+0000 7f907cc00640 1 -- 172.21.15.96:0/1570423071 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f905406d660 msgr2=0x7f905406fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:30.505 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.503+0000 7f907cc00640 1 --2- 172.21.15.96:0/1570423071 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f905406d660 0x7f905406fb20 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f907815ea10 tx=0x7f9068009190 comp rx=0 tx=0).stop 2024-05-28T06:14:30.505 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.503+0000 7f907cc00640 1 -- 172.21.15.96:0/1570423071 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f90780d0640 msgr2=0x7f907814cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:30.505 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.503+0000 7f907cc00640 1 --2- 172.21.15.96:0/1570423071 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f90780d0640 0x7f907814cc10 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f906000e870 tx=0x7f906000ed30 comp rx=0 tx=0).stop 2024-05-28T06:14:30.505 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.503+0000 7f907cc00640 1 -- 172.21.15.96:0/1570423071 shutdown_connections 2024-05-28T06:14:30.505 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.503+0000 7f907cc00640 1 --2- 172.21.15.96:0/1570423071 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f905406d660 0x7f905406fb20 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:30.506 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.503+0000 7f907cc00640 1 --2- 172.21.15.96:0/1570423071 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f90780d0a20 0x7f907809bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:30.506 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.503+0000 7f907cc00640 1 --2- 172.21.15.96:0/1570423071 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f90780d0640 0x7f907814cc10 secure :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f906000e870 tx=0x7f906000ed30 comp rx=0 tx=0).stop 2024-05-28T06:14:30.506 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.503+0000 7f907cc00640 1 --2- 172.21.15.96:0/1570423071 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f90780b32c0 0x7f907809dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:30.506 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.503+0000 7f907cc00640 1 -- 172.21.15.96:0/1570423071 >> 172.21.15.96:0/1570423071 conn(0x7f907806bd20 msgr2=0x7f907806e710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:30.506 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.504+0000 7f907cc00640 1 -- 172.21.15.96:0/1570423071 shutdown_connections 2024-05-28T06:14:30.506 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.504+0000 7f907cc00640 1 -- 172.21.15.96:0/1570423071 wait complete. 2024-05-28T06:14:30.533 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-28T06:14:30.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.798+0000 7f79aa600640 1 Processor -- start 2024-05-28T06:14:30.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.799+0000 7f79aa600640 1 -- start start 2024-05-28T06:14:30.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.801+0000 7f79aa600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f79a415b760 0x7f79a40ae220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:30.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.801+0000 7f79aa600640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79a40aea70 con 0x7f79a415bb40 2024-05-28T06:14:30.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.801+0000 7f79aa600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79a40aed50 con 0x7f79a40cfeb0 2024-05-28T06:14:30.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.801+0000 7f79aa600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f79a40af680 con 0x7f79a415b760 2024-05-28T06:14:30.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.802+0000 7f79a2a00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f79a415bb40 0x7f79a40cf7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:44144/0 (socket says 172.21.15.96:44144) 2024-05-28T06:14:30.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.802+0000 7f79a3400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f79a415b760 0x7f79a40ae220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:30.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.803+0000 7f79a2a00640 1 -- 172.21.15.96:0/2754773078 learned_addr learned my addr 172.21.15.96:0/2754773078 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:14:30.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.802+0000 7f79a3e00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f79a40cfeb0 0x7f79a406ede0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.96:60022/0 (socket says 172.21.15.96:60022) 2024-05-28T06:14:30.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.804+0000 7f79a3400640 1 -- 172.21.15.96:0/2754773078 >> v1:172.21.15.123:6789/0 conn(0x7f79a40cfeb0 legacy=0x7f79a406ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:30.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.804+0000 7f79a3400640 1 -- 172.21.15.96:0/2754773078 >> v1:172.21.15.96:6789/0 conn(0x7f79a415bb40 legacy=0x7f79a40cf7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:30.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.804+0000 7f79a3400640 1 -- 172.21.15.96:0/2754773078 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79a40af030 con 0x7f79a415b760 2024-05-28T06:14:30.807 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.806+0000 7f79a3400640 1 --2- 172.21.15.96:0/2754773078 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f79a415b760 0x7f79a40ae220 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f7994009750 tx=0x7f799402a0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=23ba7c6c4193e32a server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:30.809 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.807+0000 7f79a2000640 1 -- 172.21.15.96:0/2754773078 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7994037070 con 0x7f79a415b760 2024-05-28T06:14:30.809 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.807+0000 7f79a2000640 1 -- 172.21.15.96:0/2754773078 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f799402adb0 con 0x7f79a415b760 2024-05-28T06:14:30.809 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.807+0000 7f79a2000640 1 -- 172.21.15.96:0/2754773078 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7994032660 con 0x7f79a415b760 2024-05-28T06:14:30.810 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.809+0000 7f79aa600640 1 -- 172.21.15.96:0/2754773078 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f79a415b760 msgr2=0x7f79a40ae220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:30.810 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.809+0000 7f79aa600640 1 --2- 172.21.15.96:0/2754773078 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f79a415b760 0x7f79a40ae220 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f7994009750 tx=0x7f799402a0a0 comp rx=0 tx=0).stop 2024-05-28T06:14:30.811 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.809+0000 7f79aa600640 1 -- 172.21.15.96:0/2754773078 shutdown_connections 2024-05-28T06:14:30.811 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.809+0000 7f79aa600640 1 --2- 172.21.15.96:0/2754773078 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f79a415b760 0x7f79a40ae220 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:30.811 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.809+0000 7f79aa600640 1 -- 172.21.15.96:0/2754773078 >> 172.21.15.96:0/2754773078 conn(0x7f79a40b2f40 msgr2=0x7f79a40b3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:30.811 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.810+0000 7f79aa600640 1 -- 172.21.15.96:0/2754773078 shutdown_connections 2024-05-28T06:14:30.812 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.810+0000 7f79aa600640 1 -- 172.21.15.96:0/2754773078 wait complete. 2024-05-28T06:14:30.814 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.813+0000 7f79aa600640 1 Processor -- start 2024-05-28T06:14:30.814 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.813+0000 7f79aa600640 1 -- start start 2024-05-28T06:14:30.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.814+0000 7f79aa600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f79a40cfeb0 0x7f79a414f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:30.816 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.814+0000 7f79aa600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f79a415b760 0x7f79a40c8420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:30.816 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.814+0000 7f79aa600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f79a415bb40 0x7f79a406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:30.816 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.814+0000 7f79aa600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f79a40b0b20 con 0x7f79a415b760 2024-05-28T06:14:30.816 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.814+0000 7f79aa600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f79a40ae760 con 0x7f79a40cfeb0 2024-05-28T06:14:30.816 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.814+0000 7f79aa600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f79a40cbcf0 con 0x7f79a415bb40 2024-05-28T06:14:30.817 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.815+0000 7f79a3400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f79a40cfeb0 0x7f79a414f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:30.817 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.815+0000 7f79a3e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f79a415bb40 0x7f79a406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:30.817 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.815+0000 7f79a3e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f79a415bb40 0x7f79a406ede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.96:33242/0 (socket says 172.21.15.96:33242) 2024-05-28T06:14:30.817 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.815+0000 7f79a3400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f79a40cfeb0 0x7f79a414f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.96:43268/0 (socket says 172.21.15.96:43268) 2024-05-28T06:14:30.817 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.815+0000 7f79a3e00640 1 -- 172.21.15.96:0/1857003968 learned_addr learned my addr 172.21.15.96:0/1857003968 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:30.818 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.815+0000 7f79a2a00640 1 --2- 172.21.15.96:0/1857003968 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f79a415b760 0x7f79a40c8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:30.818 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.816+0000 7f79a3e00640 1 -- 172.21.15.96:0/1857003968 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f79a40cfeb0 msgr2=0x7f79a414f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:30.818 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.816+0000 7f79a3e00640 1 --2- 172.21.15.96:0/1857003968 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f79a40cfeb0 0x7f79a414f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:30.818 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.816+0000 7f79a3e00640 1 -- 172.21.15.96:0/1857003968 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f79a415b760 msgr2=0x7f79a40c8420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:30.818 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.816+0000 7f79a3e00640 1 --2- 172.21.15.96:0/1857003968 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f79a415b760 0x7f79a40c8420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:30.818 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.816+0000 7f79a3e00640 1 -- 172.21.15.96:0/1857003968 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79a40c8d50 con 0x7f79a415bb40 2024-05-28T06:14:30.818 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.816+0000 7f79a2a00640 1 --2- 172.21.15.96:0/1857003968 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f79a415b760 0x7f79a40c8420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:14:30.818 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.817+0000 7f79a3e00640 1 --2- 172.21.15.96:0/1857003968 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f79a415bb40 0x7f79a406ede0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f799c00c910 tx=0x7f799c00cdd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:30.819 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.817+0000 7f799b400640 1 -- 172.21.15.96:0/1857003968 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f799c018070 con 0x7f79a415bb40 2024-05-28T06:14:30.819 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.818+0000 7f799b400640 1 -- 172.21.15.96:0/1857003968 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f799c00a700 con 0x7f79a415bb40 2024-05-28T06:14:30.820 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.818+0000 7f79aa600640 1 -- 172.21.15.96:0/1857003968 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79a40c9030 con 0x7f79a415bb40 2024-05-28T06:14:30.820 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.818+0000 7f79aa600640 1 -- 172.21.15.96:0/1857003968 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f79a40c9e10 con 0x7f79a415bb40 2024-05-28T06:14:30.820 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.818+0000 7f799b400640 1 -- 172.21.15.96:0/1857003968 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f799c018070 con 0x7f79a415bb40 2024-05-28T06:14:30.821 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.820+0000 7f79aa600640 1 -- 172.21.15.96:0/1857003968 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79a406d780 con 0x7f79a415bb40 2024-05-28T06:14:30.823 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.821+0000 7f799b400640 1 -- 172.21.15.96:0/1857003968 <== mon.2 v2:172.21.15.123:3301/0 4 ==== mgrmap(e 22) ==== 92213+0+0 (secure 0 0 0) 0x7f799c00a8b0 con 0x7f79a415bb40 2024-05-28T06:14:30.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.822+0000 7f799b400640 1 --2- 172.21.15.96:0/1857003968 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f798406d9e0 0x7f798406fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:30.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.823+0000 7f799b400640 1 -- 172.21.15.96:0/1857003968 <== mon.2 v2:172.21.15.123:3301/0 5 ==== osd_map(61..61 src has 1..61) ==== 5793+0+0 (secure 0 0 0) 0x7f799c004d70 con 0x7f79a415bb40 2024-05-28T06:14:30.825 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.824+0000 7f79a3400640 1 --2- 172.21.15.96:0/1857003968 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f798406d9e0 0x7f798406fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:30.828 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.826+0000 7f79a3400640 1 --2- 172.21.15.96:0/1857003968 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f798406d9e0 0x7f798406fea0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f7994034000 tx=0x7f799402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:30.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:30.830+0000 7f799b400640 1 -- 172.21.15.96:0/1857003968 <== mon.2 v2:172.21.15.123:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f79a406d780 con 0x7f79a415bb40 2024-05-28T06:14:31.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.171+0000 7f79aa600640 1 -- 172.21.15.96:0/1857003968 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f79a40ca010 con 0x7f79a415bb40 2024-05-28T06:14:31.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.172+0000 7f799b400640 1 -- 172.21.15.96:0/1857003968 <== mon.2 v2:172.21.15.123:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v61) ==== 104+0+11 (secure 0 0 0) 0x7f799c058f00 con 0x7f79a415bb40 2024-05-28T06:14:31.174 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 32 2024-05-28T06:14:31.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.174+0000 7f79aa600640 1 -- 172.21.15.96:0/1857003968 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f798406d9e0 msgr2=0x7f798406fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:31.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.174+0000 7f79aa600640 1 --2- 172.21.15.96:0/1857003968 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f798406d9e0 0x7f798406fea0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f7994034000 tx=0x7f799402e000 comp rx=0 tx=0).stop 2024-05-28T06:14:31.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.174+0000 7f79aa600640 1 -- 172.21.15.96:0/1857003968 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f79a415bb40 msgr2=0x7f79a406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:31.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.174+0000 7f79aa600640 1 --2- 172.21.15.96:0/1857003968 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f79a415bb40 0x7f79a406ede0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f799c00c910 tx=0x7f799c00cdd0 comp rx=0 tx=0).stop 2024-05-28T06:14:31.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.174+0000 7f79aa600640 1 -- 172.21.15.96:0/1857003968 shutdown_connections 2024-05-28T06:14:31.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.174+0000 7f79aa600640 1 --2- 172.21.15.96:0/1857003968 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f798406d9e0 0x7f798406fea0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:31.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.174+0000 7f79aa600640 1 --2- 172.21.15.96:0/1857003968 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f79a415bb40 0x7f79a406ede0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:31.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.174+0000 7f79aa600640 1 --2- 172.21.15.96:0/1857003968 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f79a415b760 0x7f79a40c8420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:31.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.174+0000 7f79aa600640 1 --2- 172.21.15.96:0/1857003968 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f79a40cfeb0 0x7f79a414f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:31.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.175+0000 7f79aa600640 1 -- 172.21.15.96:0/1857003968 >> 172.21.15.96:0/1857003968 conn(0x7f79a40b2f40 msgr2=0x7f79a4159ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:31.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.175+0000 7f79aa600640 1 -- 172.21.15.96:0/1857003968 shutdown_connections 2024-05-28T06:14:31.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.175+0000 7f79aa600640 1 -- 172.21.15.96:0/1857003968 wait complete. 2024-05-28T06:14:31.193 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-28T06:14:31.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.355+0000 7f58ffa00640 1 Processor -- start 2024-05-28T06:14:31.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.355+0000 7f58ffa00640 1 -- start start 2024-05-28T06:14:31.358 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.357+0000 7f58ffa00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f58f80cd370 0x7f58f80cfd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:31.358 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.357+0000 7f58ffa00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58f80b25d0 con 0x7f58f80b27f0 2024-05-28T06:14:31.358 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.357+0000 7f58ffa00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58f80b2c50 con 0x7f58f80cde00 2024-05-28T06:14:31.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.357+0000 7f58ffa00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f58f80d05c0 con 0x7f58f80cd370 2024-05-28T06:14:31.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.357+0000 7f58f7e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f58f80b27f0 0x7f58f80b1860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:44148/0 (socket says 172.21.15.96:44148) 2024-05-28T06:14:31.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.357+0000 7f58f7e00640 1 -- 172.21.15.96:0/3664563747 learned_addr learned my addr 172.21.15.96:0/3664563747 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:14:31.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.357+0000 7f58f7400640 1 -- 172.21.15.96:0/3664563747 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 443015103 0 0) 0x7f58e0001110 con 0x7f58f80b27f0 2024-05-28T06:14:31.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.357+0000 7f58f7400640 1 -- 172.21.15.96:0/3664563747 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f58dc003520 con 0x7f58f80b27f0 2024-05-28T06:14:31.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.357+0000 7f58fd000640 1 --2- 172.21.15.96:0/3664563747 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f58f80cd370 0x7f58f80cfd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:31.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.358+0000 7f58f7400640 1 -- 172.21.15.96:0/3664563747 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1547355818 0 0) 0x7f58dc003520 con 0x7f58f80b27f0 2024-05-28T06:14:31.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.358+0000 7f58f7400640 1 -- 172.21.15.96:0/3664563747 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f58f80cd370 msgr2=0x7f58f80cfd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:31.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.358+0000 7f58f7400640 1 --2- 172.21.15.96:0/3664563747 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f58f80cd370 0x7f58f80cfd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:31.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.358+0000 7f58f7400640 1 -- 172.21.15.96:0/3664563747 >> v1:172.21.15.123:6789/0 conn(0x7f58f80cde00 legacy=0x7f58f80cc0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:31.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.358+0000 7f58f7400640 1 -- 172.21.15.96:0/3664563747 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58f80d02e0 con 0x7f58f80b27f0 2024-05-28T06:14:31.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.358+0000 7f58f7400640 1 -- 172.21.15.96:0/3664563747 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f58e0001840 con 0x7f58f80b27f0 2024-05-28T06:14:31.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.358+0000 7f58f7400640 1 -- 172.21.15.96:0/3664563747 >> v1:172.21.15.96:6789/0 conn(0x7f58f80b27f0 legacy=0x7f58f80b1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:31.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.358+0000 7f58f7400640 1 --2- 172.21.15.96:0/3664563747 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f58f80cd370 0x7f58dc0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:31.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.358+0000 7f58f7400640 1 --2- 172.21.15.96:0/3664563747 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f58dc004040 0x7f58dc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:31.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.358+0000 7f58f7400640 1 --2- 172.21.15.96:0/3664563747 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58dc004d40 0x7f58dc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:31.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.358+0000 7f58f7400640 1 -- 172.21.15.96:0/3664563747 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f58f80d05c0 con 0x7f58dc004040 2024-05-28T06:14:31.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.358+0000 7f58f7400640 1 -- 172.21.15.96:0/3664563747 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f58dc00bbb0 con 0x7f58dc004d40 2024-05-28T06:14:31.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.358+0000 7f58f7400640 1 -- 172.21.15.96:0/3664563747 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f58dc00bdb0 con 0x7f58f80cd370 2024-05-28T06:14:31.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.358+0000 7f58f7e00640 1 --2- 172.21.15.96:0/3664563747 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f58dc004040 0x7f58dc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:31.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.358+0000 7f58fd000640 1 --2- 172.21.15.96:0/3664563747 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f58f80cd370 0x7f58dc0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:31.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.358+0000 7f58f7e00640 1 -- 172.21.15.96:0/3664563747 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f58f80cd370 msgr2=0x7f58dc0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:31.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.358+0000 7f58f7e00640 1 --2- 172.21.15.96:0/3664563747 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f58f80cd370 0x7f58dc0039e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:31.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.359+0000 7f58f7e00640 1 -- 172.21.15.96:0/3664563747 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58dc004d40 msgr2=0x7f58dc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:31.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.359+0000 7f58f7e00640 1 --2- 172.21.15.96:0/3664563747 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58dc004d40 0x7f58dc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:31.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.359+0000 7f58f7e00640 1 -- 172.21.15.96:0/3664563747 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58dc003520 con 0x7f58dc004040 2024-05-28T06:14:31.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.360+0000 7f58f7e00640 1 --2- 172.21.15.96:0/3664563747 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f58dc004040 0x7f58dc004800 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f58e000c320 tx=0x7f58dc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed25591529654475 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:31.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.360+0000 7f58f7400640 1 -- 172.21.15.96:0/3664563747 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58e0005ad0 con 0x7f58dc004040 2024-05-28T06:14:31.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.360+0000 7f58f7400640 1 -- 172.21.15.96:0/3664563747 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58e0007e50 con 0x7f58dc004040 2024-05-28T06:14:31.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.360+0000 7f58f7400640 1 -- 172.21.15.96:0/3664563747 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58e0007070 con 0x7f58dc004040 2024-05-28T06:14:31.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.360+0000 7f58ffa00640 1 -- 172.21.15.96:0/3664563747 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f58dc004040 msgr2=0x7f58dc004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:31.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.360+0000 7f58ffa00640 1 --2- 172.21.15.96:0/3664563747 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f58dc004040 0x7f58dc004800 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f58e000c320 tx=0x7f58dc005830 comp rx=0 tx=0).stop 2024-05-28T06:14:31.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.360+0000 7f58fda00640 1 -- 172.21.15.96:0/3664563747 reap_dead start 2024-05-28T06:14:31.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.360+0000 7f58ffa00640 1 -- 172.21.15.96:0/3664563747 shutdown_connections 2024-05-28T06:14:31.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.360+0000 7f58ffa00640 1 -- 172.21.15.96:0/3664563747 >> 172.21.15.96:0/3664563747 conn(0x7f58f809aa30 msgr2=0x7f58f80b2f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:31.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.360+0000 7f58ffa00640 1 -- 172.21.15.96:0/3664563747 shutdown_connections 2024-05-28T06:14:31.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.361+0000 7f58ffa00640 1 -- 172.21.15.96:0/3664563747 wait complete. 2024-05-28T06:14:31.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.362+0000 7f58ffa00640 1 Processor -- start 2024-05-28T06:14:31.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.362+0000 7f58ffa00640 1 -- start start 2024-05-28T06:14:31.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.363+0000 7f58ffa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f58dc004040 0x7f58f81584b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:31.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.363+0000 7f58ffa00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f58f815a640 0x7f58f815aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:31.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.363+0000 7f58ffa00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58f8158d40 0x7f58f81591c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:31.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.363+0000 7f58ffa00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f58f80b1340 con 0x7f58dc004040 2024-05-28T06:14:31.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.363+0000 7f58ffa00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f58f80b1060 con 0x7f58f8158d40 2024-05-28T06:14:31.367 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.363+0000 7f58ffa00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f58f80b1f70 con 0x7f58f815a640 2024-05-28T06:14:31.367 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.363+0000 7f58fd000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f58dc004040 0x7f58f81584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:31.367 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.363+0000 7f58fd000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f58dc004040 0x7f58f81584b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:43196/0 (socket says 172.21.15.96:43196) 2024-05-28T06:14:31.367 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.363+0000 7f58fd000640 1 -- 172.21.15.96:0/3102071676 learned_addr learned my addr 172.21.15.96:0/3102071676 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:31.367 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.364+0000 7f58fd000640 1 -- 172.21.15.96:0/3102071676 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f58f815a640 msgr2=0x7f58f815aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:31.367 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.364+0000 7f58fd000640 1 --2- 172.21.15.96:0/3102071676 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f58f815a640 0x7f58f815aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:31.367 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.364+0000 7f58fd000640 1 -- 172.21.15.96:0/3102071676 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58f8158d40 msgr2=0x7f58f81591c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:31.367 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.364+0000 7f58fd000640 1 --2- 172.21.15.96:0/3102071676 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58f8158d40 0x7f58f81591c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:31.367 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.364+0000 7f58fd000640 1 -- 172.21.15.96:0/3102071676 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58f8159ab0 con 0x7f58dc004040 2024-05-28T06:14:31.367 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.364+0000 7f58fd000640 1 --2- 172.21.15.96:0/3102071676 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f58dc004040 0x7f58f81584b0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f58e8007e90 tx=0x7f58e800f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:31.367 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.364+0000 7f58f4c00640 1 -- 172.21.15.96:0/3102071676 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58e8018020 con 0x7f58dc004040 2024-05-28T06:14:31.367 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.364+0000 7f58f4c00640 1 -- 172.21.15.96:0/3102071676 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58e80145f0 con 0x7f58dc004040 2024-05-28T06:14:31.368 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.364+0000 7f58f4c00640 1 -- 172.21.15.96:0/3102071676 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58e80149c0 con 0x7f58dc004040 2024-05-28T06:14:31.368 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.364+0000 7f58ffa00640 1 -- 172.21.15.96:0/3102071676 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58f8159d90 con 0x7f58dc004040 2024-05-28T06:14:31.368 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.365+0000 7f58ffa00640 1 -- 172.21.15.96:0/3102071676 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58f80c87a0 con 0x7f58dc004040 2024-05-28T06:14:31.368 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.365+0000 7f58ffa00640 1 -- 172.21.15.96:0/3102071676 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58f815b000 con 0x7f58dc004040 2024-05-28T06:14:31.368 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.366+0000 7f58f4c00640 1 -- 172.21.15.96:0/3102071676 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 22) ==== 92213+0+0 (secure 0 0 0) 0x7f58e801b020 con 0x7f58dc004040 2024-05-28T06:14:31.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.367+0000 7f58f4c00640 1 --2- 172.21.15.96:0/3102071676 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f58d406d610 0x7f58d406fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:31.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.367+0000 7f58f4c00640 1 -- 172.21.15.96:0/3102071676 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5793+0+0 (secure 0 0 0) 0x7f58e808b0f0 con 0x7f58dc004040 2024-05-28T06:14:31.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.368+0000 7f58f7e00640 1 --2- 172.21.15.96:0/3102071676 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f58d406d610 0x7f58d406fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:31.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.369+0000 7f58f7e00640 1 --2- 172.21.15.96:0/3102071676 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f58d406d610 0x7f58d406fad0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f58e0033000 tx=0x7f58e0009030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:31.373 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.371+0000 7f58f4c00640 1 -- 172.21.15.96:0/3102071676 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f58e8022020 con 0x7f58dc004040 2024-05-28T06:14:31.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.560+0000 7f58ffa00640 1 -- 172.21.15.96:0/3102071676 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f58f80b7370 con 0x7f58dc004040 2024-05-28T06:14:31.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.561+0000 7f58f4c00640 1 -- 172.21.15.96:0/3102071676 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v62) ==== 100+0+11 (secure 0 0 0) 0x7f58e805af70 con 0x7f58dc004040 2024-05-28T06:14:31.563 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 64 2024-05-28T06:14:31.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.563+0000 7f58ffa00640 1 -- 172.21.15.96:0/3102071676 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f58d406d610 msgr2=0x7f58d406fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:31.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.563+0000 7f58ffa00640 1 --2- 172.21.15.96:0/3102071676 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f58d406d610 0x7f58d406fad0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f58e0033000 tx=0x7f58e0009030 comp rx=0 tx=0).stop 2024-05-28T06:14:31.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.563+0000 7f58ffa00640 1 -- 172.21.15.96:0/3102071676 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f58dc004040 msgr2=0x7f58f81584b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:31.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.563+0000 7f58ffa00640 1 --2- 172.21.15.96:0/3102071676 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f58dc004040 0x7f58f81584b0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f58e8007e90 tx=0x7f58e800f520 comp rx=0 tx=0).stop 2024-05-28T06:14:31.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.564+0000 7f58ffa00640 1 -- 172.21.15.96:0/3102071676 shutdown_connections 2024-05-28T06:14:31.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.564+0000 7f58ffa00640 1 --2- 172.21.15.96:0/3102071676 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f58d406d610 0x7f58d406fad0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:31.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.564+0000 7f58ffa00640 1 --2- 172.21.15.96:0/3102071676 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58f8158d40 0x7f58f81591c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:31.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.564+0000 7f58ffa00640 1 --2- 172.21.15.96:0/3102071676 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f58f815a640 0x7f58f815aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:31.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.564+0000 7f58ffa00640 1 --2- 172.21.15.96:0/3102071676 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f58dc004040 0x7f58f81584b0 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:31.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.564+0000 7f58ffa00640 1 -- 172.21.15.96:0/3102071676 >> 172.21.15.96:0/3102071676 conn(0x7f58f809aa30 msgr2=0x7f58f806fe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:31.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.564+0000 7f58ffa00640 1 -- 172.21.15.96:0/3102071676 shutdown_connections 2024-05-28T06:14:31.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.564+0000 7f58ffa00640 1 -- 172.21.15.96:0/3102071676 wait complete. 2024-05-28T06:14:31.583 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-28T06:14:31.756 DEBUG:teuthology.orchestra.run.smithi096:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:14:31.764 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:14:31.777 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.776+0000 7fd606e00640 1 Processor -- start 2024-05-28T06:14:31.777 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.776+0000 7fd606e00640 1 -- start start 2024-05-28T06:14:31.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.777+0000 7fd606e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd6000d3280 0x7fd6000d0090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:31.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.777+0000 7fd606e00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd600163330 con 0x7fd6001625c0 2024-05-28T06:14:31.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.777+0000 7fd606e00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6000cf2d0 con 0x7fd60015e8a0 2024-05-28T06:14:31.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.777+0000 7fd606e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fd6001630f0 con 0x7fd6000d3280 2024-05-28T06:14:31.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.777+0000 7fd606400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fd6001625c0 0x7fd6001629e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:44152/0 (socket says 172.21.15.96:44152) 2024-05-28T06:14:31.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.777+0000 7fd606400640 1 -- 172.21.15.96:0/3170687323 learned_addr learned my addr 172.21.15.96:0/3170687323 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:14:31.781 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.778+0000 7fd605000640 1 --2- 172.21.15.96:0/3170687323 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd6000d3280 0x7fd6000d0090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:31.781 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.778+0000 7fd5ffe00640 1 -- 172.21.15.96:0/3170687323 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 952961981 0 0) 0x7fd600163330 con 0x7fd6001625c0 2024-05-28T06:14:31.781 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.778+0000 7fd5ffe00640 1 -- 172.21.15.96:0/3170687323 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd5e8003520 con 0x7fd6001625c0 2024-05-28T06:14:31.781 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.778+0000 7fd5ffe00640 1 -- 172.21.15.96:0/3170687323 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3790757255 0 0) 0x7fd5e8003520 con 0x7fd6001625c0 2024-05-28T06:14:31.781 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.778+0000 7fd5ffe00640 1 -- 172.21.15.96:0/3170687323 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd6000d3280 msgr2=0x7fd6000d0090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:31.781 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.778+0000 7fd5ffe00640 1 --2- 172.21.15.96:0/3170687323 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd6000d3280 0x7fd6000d0090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:31.781 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.778+0000 7fd5ffe00640 1 -- 172.21.15.96:0/3170687323 >> v1:172.21.15.123:6789/0 conn(0x7fd60015e8a0 legacy=0x7fd6000d2b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:31.781 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.778+0000 7fd5ffe00640 1 -- 172.21.15.96:0/3170687323 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6000cf5b0 con 0x7fd6001625c0 2024-05-28T06:14:31.781 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.778+0000 7fd5ffe00640 1 -- 172.21.15.96:0/3170687323 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7fd5f80019f0 con 0x7fd6001625c0 2024-05-28T06:14:31.781 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.778+0000 7fd5ffe00640 1 -- 172.21.15.96:0/3170687323 >> v1:172.21.15.96:6789/0 conn(0x7fd6001625c0 legacy=0x7fd6001629e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:31.781 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.778+0000 7fd5ffe00640 1 --2- 172.21.15.96:0/3170687323 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd5e80039e0 0x7fd5e8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:31.781 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.778+0000 7fd5ffe00640 1 --2- 172.21.15.96:0/3170687323 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd6000d3280 0x7fd5e8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:31.781 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.778+0000 7fd5ffe00640 1 --2- 172.21.15.96:0/3170687323 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd5e8004d40 0x7fd5e80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:31.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.778+0000 7fd5ffe00640 1 -- 172.21.15.96:0/3170687323 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd6001630f0 con 0x7fd5e80039e0 2024-05-28T06:14:31.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.778+0000 7fd5ffe00640 1 -- 172.21.15.96:0/3170687323 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd5e800bc00 con 0x7fd5e8004d40 2024-05-28T06:14:31.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.778+0000 7fd5ffe00640 1 -- 172.21.15.96:0/3170687323 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fd5e800be00 con 0x7fd6000d3280 2024-05-28T06:14:31.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.779+0000 7fd605a00640 1 --2- 172.21.15.96:0/3170687323 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd5e80039e0 0x7fd5e8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:31.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.779+0000 7fd605000640 1 --2- 172.21.15.96:0/3170687323 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd6000d3280 0x7fd5e8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:31.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.779+0000 7fd606400640 1 --2- 172.21.15.96:0/3170687323 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd5e8004d40 0x7fd5e80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:31.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.779+0000 7fd605a00640 1 -- 172.21.15.96:0/3170687323 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd6000d3280 msgr2=0x7fd5e8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:31.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.779+0000 7fd605a00640 1 --2- 172.21.15.96:0/3170687323 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd6000d3280 0x7fd5e8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:31.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.779+0000 7fd605a00640 1 -- 172.21.15.96:0/3170687323 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd5e8004d40 msgr2=0x7fd5e80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:31.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.779+0000 7fd605a00640 1 --2- 172.21.15.96:0/3170687323 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd5e8004d40 0x7fd5e80051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:31.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.779+0000 7fd605a00640 1 -- 172.21.15.96:0/3170687323 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5e8003520 con 0x7fd5e80039e0 2024-05-28T06:14:31.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.780+0000 7fd605a00640 1 --2- 172.21.15.96:0/3170687323 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd5e80039e0 0x7fd5e8003e00 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7fd5f4001750 tx=0x7fd5e8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecf7bbb9c4aff553 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:31.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.780+0000 7fd5ffe00640 1 -- 172.21.15.96:0/3170687323 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5f4001370 con 0x7fd5e80039e0 2024-05-28T06:14:31.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.780+0000 7fd5ffe00640 1 -- 172.21.15.96:0/3170687323 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5f40052d0 con 0x7fd5e80039e0 2024-05-28T06:14:31.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.780+0000 7fd5ffe00640 1 -- 172.21.15.96:0/3170687323 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5f402abe0 con 0x7fd5e80039e0 2024-05-28T06:14:31.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.781+0000 7fd606e00640 1 -- 172.21.15.96:0/3170687323 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd5e80039e0 msgr2=0x7fd5e8003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:31.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.781+0000 7fd606e00640 1 --2- 172.21.15.96:0/3170687323 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd5e80039e0 0x7fd5e8003e00 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7fd5f4001750 tx=0x7fd5e8005830 comp rx=0 tx=0).stop 2024-05-28T06:14:31.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.781+0000 7fd606400640 1 -- 172.21.15.96:0/3170687323 reap_dead start 2024-05-28T06:14:31.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.781+0000 7fd606e00640 1 -- 172.21.15.96:0/3170687323 shutdown_connections 2024-05-28T06:14:31.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.781+0000 7fd606e00640 1 -- 172.21.15.96:0/3170687323 >> 172.21.15.96:0/3170687323 conn(0x7fd60008bfc0 msgr2=0x7fd6000d4590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:31.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.781+0000 7fd606e00640 1 -- 172.21.15.96:0/3170687323 shutdown_connections 2024-05-28T06:14:31.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.781+0000 7fd606e00640 1 -- 172.21.15.96:0/3170687323 wait complete. 2024-05-28T06:14:31.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.783+0000 7fd606e00640 1 Processor -- start 2024-05-28T06:14:31.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.783+0000 7fd606e00640 1 -- start start 2024-05-28T06:14:31.786 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.784+0000 7fd606e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd5e80039e0 0x7fd60015da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:31.786 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.784+0000 7fd606e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd60006db60 0x7fd60006dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:31.786 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.784+0000 7fd606e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd6000cd5a0 0x7fd6000cda50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:31.786 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.784+0000 7fd606e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd60015ec80 con 0x7fd6000cd5a0 2024-05-28T06:14:31.786 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.784+0000 7fd606e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd6000cfb70 con 0x7fd5e80039e0 2024-05-28T06:14:31.786 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.784+0000 7fd606e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fd60015ee00 con 0x7fd60006db60 2024-05-28T06:14:31.786 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.785+0000 7fd606400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd6000cd5a0 0x7fd6000cda50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:31.787 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.785+0000 7fd605a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd5e80039e0 0x7fd60015da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:31.787 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.785+0000 7fd605000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd60006db60 0x7fd60006dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:31.787 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.785+0000 7fd605a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd5e80039e0 0x7fd60015da30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.96:43308/0 (socket says 172.21.15.96:43308) 2024-05-28T06:14:31.787 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.785+0000 7fd605a00640 1 -- 172.21.15.96:0/92611559 learned_addr learned my addr 172.21.15.96:0/92611559 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:31.787 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.785+0000 7fd606400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd6000cd5a0 0x7fd6000cda50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:43214/0 (socket says 172.21.15.96:43214) 2024-05-28T06:14:31.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.785+0000 7fd606400640 1 -- 172.21.15.96:0/92611559 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd60006db60 msgr2=0x7fd60006dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:31.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.785+0000 7fd606400640 1 --2- 172.21.15.96:0/92611559 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd60006db60 0x7fd60006dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:31.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.785+0000 7fd606400640 1 -- 172.21.15.96:0/92611559 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd5e80039e0 msgr2=0x7fd60015da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:31.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.785+0000 7fd606400640 1 --2- 172.21.15.96:0/92611559 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd5e80039e0 0x7fd60015da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:31.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.785+0000 7fd606400640 1 -- 172.21.15.96:0/92611559 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6001fc3f0 con 0x7fd6000cd5a0 2024-05-28T06:14:31.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.785+0000 7fd606400640 1 --2- 172.21.15.96:0/92611559 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd6000cd5a0 0x7fd6000cda50 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7fd60008df10 tx=0x7fd5f800b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:31.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.786+0000 7fd5fd600640 1 -- 172.21.15.96:0/92611559 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5f8004d60 con 0x7fd6000cd5a0 2024-05-28T06:14:31.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.786+0000 7fd5fd600640 1 -- 172.21.15.96:0/92611559 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5f800bb20 con 0x7fd6000cd5a0 2024-05-28T06:14:31.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.786+0000 7fd5fd600640 1 -- 172.21.15.96:0/92611559 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5f8007110 con 0x7fd6000cd5a0 2024-05-28T06:14:31.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.786+0000 7fd606e00640 1 -- 172.21.15.96:0/92611559 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6001fc6d0 con 0x7fd6000cd5a0 2024-05-28T06:14:31.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.786+0000 7fd606e00640 1 -- 172.21.15.96:0/92611559 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6001fcc90 con 0x7fd6000cd5a0 2024-05-28T06:14:31.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.787+0000 7fd606e00640 1 -- 172.21.15.96:0/92611559 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd60006e520 con 0x7fd6000cd5a0 2024-05-28T06:14:31.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.788+0000 7fd5fd600640 1 -- 172.21.15.96:0/92611559 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 22) ==== 92213+0+0 (secure 0 0 0) 0x7fd5f801b020 con 0x7fd6000cd5a0 2024-05-28T06:14:31.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.789+0000 7fd5fd600640 1 --2- 172.21.15.96:0/92611559 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd5e006d610 0x7fd5e006fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:31.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.789+0000 7fd5fd600640 1 -- 172.21.15.96:0/92611559 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5765+0+0 (secure 0 0 0) 0x7fd5f808a830 con 0x7fd6000cd5a0 2024-05-28T06:14:31.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.790+0000 7fd605a00640 1 --2- 172.21.15.96:0/92611559 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd5e006d610 0x7fd5e006fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:31.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.790+0000 7fd605a00640 1 --2- 172.21.15.96:0/92611559 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd5e006d610 0x7fd5e006fad0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fd5f4001750 tx=0x7fd5f402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:31.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:31.794+0000 7fd5fd600640 1 -- 172.21.15.96:0/92611559 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd5f8059dd0 con 0x7fd6000cd5a0 2024-05-28T06:14:32.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:32.000+0000 7fd606e00640 1 -- 172.21.15.96:0/92611559 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fd6000d2950 con 0x7fd6000cd5a0 2024-05-28T06:14:32.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:32.001+0000 7fd5fd600640 1 -- 172.21.15.96:0/92611559 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v62) ==== 80+0+0 (secure 0 0 0) 0x7fd5f805e0d0 con 0x7fd6000cd5a0 2024-05-28T06:14:32.006 INFO:teuthology.orchestra.run.smithi096.stderr:listed 0 entries 2024-05-28T06:14:32.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:32.003+0000 7fd606e00640 1 -- 172.21.15.96:0/92611559 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd5e006d610 msgr2=0x7fd5e006fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:32.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:32.003+0000 7fd606e00640 1 --2- 172.21.15.96:0/92611559 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd5e006d610 0x7fd5e006fad0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fd5f4001750 tx=0x7fd5f402e000 comp rx=0 tx=0).stop 2024-05-28T06:14:32.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:32.003+0000 7fd606e00640 1 -- 172.21.15.96:0/92611559 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd6000cd5a0 msgr2=0x7fd6000cda50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:32.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:32.003+0000 7fd606e00640 1 --2- 172.21.15.96:0/92611559 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd6000cd5a0 0x7fd6000cda50 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7fd60008df10 tx=0x7fd5f800b4a0 comp rx=0 tx=0).stop 2024-05-28T06:14:32.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:32.004+0000 7fd606e00640 1 -- 172.21.15.96:0/92611559 shutdown_connections 2024-05-28T06:14:32.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:32.004+0000 7fd606e00640 1 --2- 172.21.15.96:0/92611559 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd5e006d610 0x7fd5e006fad0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:32.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:32.004+0000 7fd606e00640 1 --2- 172.21.15.96:0/92611559 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd6000cd5a0 0x7fd6000cda50 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:32.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:32.004+0000 7fd606e00640 1 --2- 172.21.15.96:0/92611559 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd60006db60 0x7fd60006dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:32.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:32.004+0000 7fd606e00640 1 --2- 172.21.15.96:0/92611559 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd5e80039e0 0x7fd60015da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:32.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:32.004+0000 7fd606e00640 1 -- 172.21.15.96:0/92611559 >> 172.21.15.96:0/92611559 conn(0x7fd60008bfc0 msgr2=0x7fd6000b2ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:32.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:32.004+0000 7fd606e00640 1 -- 172.21.15.96:0/92611559 shutdown_connections 2024-05-28T06:14:32.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:32.004+0000 7fd606e00640 1 -- 172.21.15.96:0/92611559 wait complete. 2024-05-28T06:14:32.023 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi096.front.sepia.ceph.com... 2024-05-28T06:14:32.023 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:14:32.023 DEBUG:teuthology.orchestra.run.smithi096:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-05-28T06:14:32.093 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi096.front.sepia.ceph.com 2024-05-28T06:14:32.093 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:14:32.093 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-28T06:14:32.106 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi096.stderr:ceph-fuse[60297]: fuse finished with error 0 and tester_r 0 2024-05-28T06:14:32.287 INFO:tasks.cephfs.mount:Terminating background process 2024-05-28T06:14:32.287 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-05-28T06:14:32.288 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:14:32.337 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052731,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248996k,nr_inodes=812249,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-05-28T06:14:32.337 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi123.front.sepia.ceph.com 2024-05-28T06:14:32.337 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:14:32.338 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-05-28T06:14:32.428 INFO:teuthology.orchestra.run.smithi123.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-05-28T06:14:32.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-28T06:14:32.429 INFO:tasks.cephfs.mount:Terminating background process 2024-05-28T06:14:32.429 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:14:32.429 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:14:32.483 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052731,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248996k,nr_inodes=812249,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-28T06:14:32.483 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi096.front.sepia.ceph.com 2024-05-28T06:14:32.484 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:14:32.484 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-05-28T06:14:32.533 INFO:teuthology.orchestra.run.smithi096.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-05-28T06:14:32.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-28T06:14:32.535 INFO:tasks.cephfs.mount:Terminating background process 2024-05-28T06:14:32.535 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-05-28T06:14:32.535 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:14:32.590 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052731,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248996k,nr_inodes=812249,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-05-28T06:14:32.590 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi123.front.sepia.ceph.com 2024-05-28T06:14:32.590 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:14:32.591 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-05-28T06:14:32.638 INFO:teuthology.orchestra.run.smithi123.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-05-28T06:14:32.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-28T06:14:32.639 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-28T06:14:32.850 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:32.850+0000 7fbc0d000640 1 Processor -- start 2024-05-28T06:14:32.851 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:32.850+0000 7fbc0d000640 1 -- start start 2024-05-28T06:14:32.852 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:32.851+0000 7fbc0d000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fbc0815b3a0 0x7fbc080b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:32.852 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:32.851+0000 7fbc0d000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbc08073310 0x7fbc0815f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:32.852 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:32.851+0000 7fbc0d000640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc080afd60 con 0x7fbc0806dc20 2024-05-28T06:14:32.852 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:32.851+0000 7fbc0d000640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fbc0806f980 con 0x7fbc0815b3a0 2024-05-28T06:14:32.852 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:32.851+0000 7fbc0d000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fbc080ae0c0 con 0x7fbc08073310 2024-05-28T06:14:32.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:32.852+0000 7fbc07e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbc08073310 0x7fbc0815f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:32.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:32.852+0000 7fbc07400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fbc0815b3a0 0x7fbc080b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:32.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:32.852+0000 7fbc07e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbc08073310 0x7fbc0815f9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.96:33312/0 (socket says 172.21.15.96:33312) 2024-05-28T06:14:32.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:32.852+0000 7fbc07e00640 1 -- 172.21.15.96:0/259221822 learned_addr learned my addr 172.21.15.96:0/259221822 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:32.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:32.853+0000 7fbc07e00640 1 -- 172.21.15.96:0/259221822 >> v2:172.21.15.123:3300/0 conn(0x7fbc0815b3a0 msgr2=0x7fbc080b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:32.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:32.853+0000 7fbc07e00640 1 --2- 172.21.15.96:0/259221822 >> v2:172.21.15.123:3300/0 conn(0x7fbc0815b3a0 0x7fbc080b32c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:32.854 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:32.853+0000 7fbc07e00640 1 -- 172.21.15.96:0/259221822 >> v1:172.21.15.123:6789/0 conn(0x7fbc0806dc20 legacy=0x7fbc08072c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:32.854 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:32.853+0000 7fbc07e00640 1 -- 172.21.15.96:0/259221822 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc080af7a0 con 0x7fbc08073310 2024-05-28T06:14:32.855 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:32.854+0000 7fbc07e00640 1 --2- 172.21.15.96:0/259221822 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbc08073310 0x7fbc0815f9a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fbc00009e50 tx=0x7fbc0002a730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b69480a1c4c2dd4d server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:32.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:32.855+0000 7fbc06a00640 1 -- 172.21.15.96:0/259221822 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc00036070 con 0x7fbc08073310 2024-05-28T06:14:32.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:32.855+0000 7fbc06a00640 1 -- 172.21.15.96:0/259221822 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc00031600 con 0x7fbc08073310 2024-05-28T06:14:32.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:32.855+0000 7fbc06a00640 1 -- 172.21.15.96:0/259221822 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc000319d0 con 0x7fbc08073310 2024-05-28T06:14:32.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:32.855+0000 7fbc0d000640 1 -- 172.21.15.96:0/259221822 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbc08073310 msgr2=0x7fbc0815f9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:32.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:32.855+0000 7fbc0d000640 1 --2- 172.21.15.96:0/259221822 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbc08073310 0x7fbc0815f9a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fbc00009e50 tx=0x7fbc0002a730 comp rx=0 tx=0).stop 2024-05-28T06:14:32.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:32.855+0000 7fbc0d000640 1 -- 172.21.15.96:0/259221822 shutdown_connections 2024-05-28T06:14:32.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:32.855+0000 7fbc0d000640 1 --2- 172.21.15.96:0/259221822 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbc08073310 0x7fbc0815f9a0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:32.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:32.855+0000 7fbc0d000640 1 --2- 172.21.15.96:0/259221822 >> v2:172.21.15.123:3300/0 conn(0x7fbc0815b3a0 0x7fbc080b32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:32.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:32.855+0000 7fbc0d000640 1 -- 172.21.15.96:0/259221822 >> 172.21.15.96:0/259221822 conn(0x7fbc080b1fc0 msgr2=0x7fbc080b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:32.857 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:32.855+0000 7fbc0d000640 1 -- 172.21.15.96:0/259221822 shutdown_connections 2024-05-28T06:14:32.857 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:32.855+0000 7fbc0d000640 1 -- 172.21.15.96:0/259221822 wait complete. 2024-05-28T06:14:32.858 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:32.857+0000 7fbc0d000640 1 Processor -- start 2024-05-28T06:14:32.858 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:32.857+0000 7fbc0d000640 1 -- start start 2024-05-28T06:14:32.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:32.858+0000 7fbc0d000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc0806dc20 0x7fbc080ccca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:32.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:32.858+0000 7fbc0d000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc08073310 0x7fbc080cd9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:32.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:32.858+0000 7fbc0d000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbc0815b3a0 0x7fbc080ce300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:32.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:32.858+0000 7fbc0d000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fbc0815b780 con 0x7fbc08073310 2024-05-28T06:14:32.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:32.858+0000 7fbc0d000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fbc0815fee0 con 0x7fbc0806dc20 2024-05-28T06:14:32.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:32.858+0000 7fbc0d000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fbc080ae550 con 0x7fbc0815b3a0 2024-05-28T06:14:32.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:32.859+0000 7fbbfea00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc08073310 0x7fbc080cd9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:32.860 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:32.859+0000 7fbbfea00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc08073310 0x7fbc080cd9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:43220/0 (socket says 172.21.15.96:43220) 2024-05-28T06:14:32.860 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:32.859+0000 7fbbfea00640 1 -- 172.21.15.96:0/1238428546 learned_addr learned my addr 172.21.15.96:0/1238428546 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:32.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:32.859+0000 7fbc07400640 1 --2- 172.21.15.96:0/1238428546 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc0806dc20 0x7fbc080ccca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:32.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:32.859+0000 7fbc07e00640 1 --2- 172.21.15.96:0/1238428546 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbc0815b3a0 0x7fbc080ce300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:32.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:32.859+0000 7fbbfea00640 1 -- 172.21.15.96:0/1238428546 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbc0815b3a0 msgr2=0x7fbc080ce300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:32.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:32.859+0000 7fbbfea00640 1 --2- 172.21.15.96:0/1238428546 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbc0815b3a0 0x7fbc080ce300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:32.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:32.859+0000 7fbbfea00640 1 -- 172.21.15.96:0/1238428546 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc0806dc20 msgr2=0x7fbc080ccca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:32.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:32.859+0000 7fbbfea00640 1 --2- 172.21.15.96:0/1238428546 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc0806dc20 0x7fbc080ccca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:32.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:32.859+0000 7fbbfea00640 1 -- 172.21.15.96:0/1238428546 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc081f4c00 con 0x7fbc08073310 2024-05-28T06:14:32.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:32.860+0000 7fbbfea00640 1 --2- 172.21.15.96:0/1238428546 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc08073310 0x7fbc080cd9e0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7fbbf000c400 tx=0x7fbbf000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:32.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:32.860+0000 7fbbffe00640 1 -- 172.21.15.96:0/1238428546 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbf00052d0 con 0x7fbc08073310 2024-05-28T06:14:32.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:32.860+0000 7fbbffe00640 1 -- 172.21.15.96:0/1238428546 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbf0005490 con 0x7fbc08073310 2024-05-28T06:14:32.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:32.860+0000 7fbbffe00640 1 -- 172.21.15.96:0/1238428546 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbf0007260 con 0x7fbc08073310 2024-05-28T06:14:32.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:32.860+0000 7fbc0d000640 1 -- 172.21.15.96:0/1238428546 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc081f4ee0 con 0x7fbc08073310 2024-05-28T06:14:32.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:32.860+0000 7fbc0d000640 1 -- 172.21.15.96:0/1238428546 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc081f54a0 con 0x7fbc08073310 2024-05-28T06:14:32.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:32.861+0000 7fbc0d000640 1 -- 172.21.15.96:0/1238428546 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc080689e0 con 0x7fbc08073310 2024-05-28T06:14:32.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:32.862+0000 7fbbffe00640 1 -- 172.21.15.96:0/1238428546 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 22) ==== 92213+0+0 (secure 0 0 0) 0x7fbbf0007400 con 0x7fbc08073310 2024-05-28T06:14:32.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:32.863+0000 7fbbffe00640 1 --2- 172.21.15.96:0/1238428546 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbbe406d5e0 0x7fbbe406faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:32.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:32.863+0000 7fbbffe00640 1 -- 172.21.15.96:0/1238428546 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5765+0+0 (secure 0 0 0) 0x7fbbf0089b00 con 0x7fbc08073310 2024-05-28T06:14:32.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:32.864+0000 7fbc07400640 1 --2- 172.21.15.96:0/1238428546 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbbe406d5e0 0x7fbbe406faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:32.865 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:32.865+0000 7fbc07400640 1 --2- 172.21.15.96:0/1238428546 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbbe406d5e0 0x7fbbe406faa0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fbbf8009a30 tx=0x7fbbf8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:32.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:32.867+0000 7fbbffe00640 1 -- 172.21.15.96:0/1238428546 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbbf0059980 con 0x7fbc08073310 2024-05-28T06:14:33.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.106+0000 7fbc0d000640 1 -- 172.21.15.96:0/1238428546 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbc080cb420 con 0x7fbc08073310 2024-05-28T06:14:33.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.107+0000 7fbbffe00640 1 -- 172.21.15.96:0/1238428546 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+5184 (secure 0 0 0) 0x7fbbf0059280 con 0x7fbc08073310 2024-05-28T06:14:33.108 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:14:33.108 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":35,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5015,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/1869890774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":1869890774},{"type":"v1","addr":"172.21.15.123:6837","nonce":1869890774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":5386,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/873923319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":873923319},{"type":"v1","addr":"172.21.15.123:6835","nonce":873923319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":6486,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6834/1041571401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":1041571401},{"type":"v1","addr":"172.21.15.96:6834","nonce":1041571401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-28T06:12:28.268409+0000","modified":"2024-05-28T06:12:31.363489+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6489},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6489":{"gid":6489,"name":"a","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.96:6835/1590650245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6833","nonce":1590650245},{"type":"v1","addr":"172.21.15.96:6835","nonce":1590650245}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6489,"qdb_cluster":[6489]},"id":4}]} 2024-05-28T06:14:33.109 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 35 2024-05-28T06:14:33.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.110+0000 7fbc0d000640 1 -- 172.21.15.96:0/1238428546 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbbe406d5e0 msgr2=0x7fbbe406faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:33.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.110+0000 7fbc0d000640 1 --2- 172.21.15.96:0/1238428546 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbbe406d5e0 0x7fbbe406faa0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fbbf8009a30 tx=0x7fbbf8008040 comp rx=0 tx=0).stop 2024-05-28T06:14:33.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.110+0000 7fbc0d000640 1 -- 172.21.15.96:0/1238428546 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc08073310 msgr2=0x7fbc080cd9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:33.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.110+0000 7fbc0d000640 1 --2- 172.21.15.96:0/1238428546 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc08073310 0x7fbc080cd9e0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7fbbf000c400 tx=0x7fbbf000c8c0 comp rx=0 tx=0).stop 2024-05-28T06:14:33.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.110+0000 7fbc0d000640 1 -- 172.21.15.96:0/1238428546 shutdown_connections 2024-05-28T06:14:33.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.110+0000 7fbc0d000640 1 --2- 172.21.15.96:0/1238428546 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbbe406d5e0 0x7fbbe406faa0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:33.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.110+0000 7fbc0d000640 1 --2- 172.21.15.96:0/1238428546 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbc0815b3a0 0x7fbc080ce300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:33.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.110+0000 7fbc0d000640 1 --2- 172.21.15.96:0/1238428546 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc08073310 0x7fbc080cd9e0 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:33.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.110+0000 7fbc0d000640 1 --2- 172.21.15.96:0/1238428546 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc0806dc20 0x7fbc080ccca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:33.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.110+0000 7fbc0d000640 1 -- 172.21.15.96:0/1238428546 >> 172.21.15.96:0/1238428546 conn(0x7fbc080b1fc0 msgr2=0x7fbc0806ee40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:33.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.110+0000 7fbc0d000640 1 -- 172.21.15.96:0/1238428546 shutdown_connections 2024-05-28T06:14:33.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.110+0000 7fbc0d000640 1 -- 172.21.15.96:0/1238428546 wait complete. 2024-05-28T06:14:33.130 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:14:33.328 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.327+0000 7f1514200640 1 Processor -- start 2024-05-28T06:14:33.328 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.327+0000 7f1514200640 1 -- start start 2024-05-28T06:14:33.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.329+0000 7f1514200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f150c0b4e10 0x7f150c0cde30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:33.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.329+0000 7f1514200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f150c0d0ca0 0x7f150c099090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:33.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.329+0000 7f1514200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f150c0b51f0 con 0x7f150c0ce370 2024-05-28T06:14:33.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.329+0000 7f1514200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f150c0aef60 con 0x7f150c0b4e10 2024-05-28T06:14:33.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.329+0000 7f1514200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f150c0ad1d0 con 0x7f150c0d0ca0 2024-05-28T06:14:33.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.329+0000 7f1510e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f150c0ce370 0x7f150c0d0590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:44166/0 (socket says 172.21.15.96:44166) 2024-05-28T06:14:33.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.329+0000 7f1510e00640 1 -- 172.21.15.96:0/741593865 learned_addr learned my addr 172.21.15.96:0/741593865 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:14:33.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.330+0000 7f1511800640 1 --2- 172.21.15.96:0/741593865 >> v2:172.21.15.123:3300/0 conn(0x7f150c0b4e10 0x7f150c0cde30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:33.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.330+0000 7f1512200640 1 --2- 172.21.15.96:0/741593865 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f150c0d0ca0 0x7f150c099090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:33.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.330+0000 7f150be00640 1 -- 172.21.15.96:0/741593865 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1765842739 0 0) 0x7f150c0b51f0 con 0x7f150c0ce370 2024-05-28T06:14:33.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.330+0000 7f150be00640 1 -- 172.21.15.96:0/741593865 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f14f4003520 con 0x7f150c0ce370 2024-05-28T06:14:33.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.330+0000 7f150be00640 1 -- 172.21.15.96:0/741593865 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2891347946 0 0) 0x7f14f4003520 con 0x7f150c0ce370 2024-05-28T06:14:33.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.330+0000 7f150be00640 1 -- 172.21.15.96:0/741593865 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f150c0d0ca0 msgr2=0x7f150c099090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:33.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.330+0000 7f150be00640 1 --2- 172.21.15.96:0/741593865 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f150c0d0ca0 0x7f150c099090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:33.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.330+0000 7f150be00640 1 -- 172.21.15.96:0/741593865 >> v2:172.21.15.123:3300/0 conn(0x7f150c0b4e10 msgr2=0x7f150c0cde30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:33.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.330+0000 7f150be00640 1 --2- 172.21.15.96:0/741593865 >> v2:172.21.15.123:3300/0 conn(0x7f150c0b4e10 0x7f150c0cde30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:33.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.330+0000 7f150be00640 1 -- 172.21.15.96:0/741593865 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f150c0ad470 con 0x7f150c0ce370 2024-05-28T06:14:33.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.330+0000 7f150be00640 1 -- 172.21.15.96:0/741593865 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f14f8001670 con 0x7f150c0ce370 2024-05-28T06:14:33.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.330+0000 7f150be00640 1 -- 172.21.15.96:0/741593865 >> v1:172.21.15.96:6789/0 conn(0x7f150c0ce370 legacy=0x7f150c0d0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:33.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.330+0000 7f150be00640 1 --2- 172.21.15.96:0/741593865 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f14f40039b0 0x7f14f4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:33.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.330+0000 7f150be00640 1 --2- 172.21.15.96:0/741593865 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f150c0d0ca0 0x7f14f4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:33.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.330+0000 7f150be00640 1 --2- 172.21.15.96:0/741593865 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f14f4004d40 0x7f14f40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:33.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.330+0000 7f150be00640 1 -- 172.21.15.96:0/741593865 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f150c0aef60 con 0x7f14f4004d40 2024-05-28T06:14:33.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.330+0000 7f150be00640 1 -- 172.21.15.96:0/741593865 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f150c0ad1d0 con 0x7f14f40039b0 2024-05-28T06:14:33.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.330+0000 7f150be00640 1 -- 172.21.15.96:0/741593865 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f14f400b9f0 con 0x7f150c0d0ca0 2024-05-28T06:14:33.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.331+0000 7f1512200640 1 --2- 172.21.15.96:0/741593865 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f14f4004d40 0x7f14f40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:33.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.331+0000 7f1511800640 1 --2- 172.21.15.96:0/741593865 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f14f40039b0 0x7f14f4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:33.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.331+0000 7f1510e00640 1 --2- 172.21.15.96:0/741593865 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f150c0d0ca0 0x7f14f4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:33.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.331+0000 7f1512200640 1 -- 172.21.15.96:0/741593865 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f150c0d0ca0 msgr2=0x7f14f4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:33.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.331+0000 7f1512200640 1 --2- 172.21.15.96:0/741593865 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f150c0d0ca0 0x7f14f4004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:33.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.331+0000 7f1512200640 1 -- 172.21.15.96:0/741593865 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f14f40039b0 msgr2=0x7f14f4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:33.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.331+0000 7f1512200640 1 --2- 172.21.15.96:0/741593865 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f14f40039b0 0x7f14f4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:33.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.331+0000 7f1512200640 1 -- 172.21.15.96:0/741593865 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14f4003520 con 0x7f14f4004d40 2024-05-28T06:14:33.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.331+0000 7f1511800640 1 --2- 172.21.15.96:0/741593865 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f14f40039b0 0x7f14f4003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:14:33.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.332+0000 7f1512200640 1 --2- 172.21.15.96:0/741593865 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f14f4004d40 0x7f14f40051c0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f150c0afd90 tx=0x7f14f4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50a70cf34bebb237 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:33.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.332+0000 7f150be00640 1 -- 172.21.15.96:0/741593865 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1504034070 con 0x7f14f4004d40 2024-05-28T06:14:33.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.332+0000 7f150be00640 1 -- 172.21.15.96:0/741593865 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f150400b2d0 con 0x7f14f4004d40 2024-05-28T06:14:33.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.332+0000 7f150be00640 1 -- 172.21.15.96:0/741593865 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f150400be20 con 0x7f14f4004d40 2024-05-28T06:14:33.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.333+0000 7f1514200640 1 -- 172.21.15.96:0/741593865 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f14f4004d40 msgr2=0x7f14f40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:33.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.333+0000 7f1514200640 1 --2- 172.21.15.96:0/741593865 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f14f4004d40 0x7f14f40051c0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f150c0afd90 tx=0x7f14f4005830 comp rx=0 tx=0).stop 2024-05-28T06:14:33.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.333+0000 7f1512200640 1 -- 172.21.15.96:0/741593865 reap_dead start 2024-05-28T06:14:33.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.333+0000 7f1514200640 1 -- 172.21.15.96:0/741593865 shutdown_connections 2024-05-28T06:14:33.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.333+0000 7f1514200640 1 -- 172.21.15.96:0/741593865 >> 172.21.15.96:0/741593865 conn(0x7f150c06bae0 msgr2=0x7f150c06bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:33.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.333+0000 7f1514200640 1 -- 172.21.15.96:0/741593865 shutdown_connections 2024-05-28T06:14:33.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.333+0000 7f1514200640 1 -- 172.21.15.96:0/741593865 wait complete. 2024-05-28T06:14:33.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.335+0000 7f1514200640 1 Processor -- start 2024-05-28T06:14:33.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.335+0000 7f1514200640 1 -- start start 2024-05-28T06:14:33.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.336+0000 7f1514200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f14f4004d40 0x7f150c14d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:33.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.336+0000 7f1514200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f150c158540 0x7f150c0b1aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:33.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.336+0000 7f1514200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f150c157ff0 0x7f150c0b1fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:33.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.336+0000 7f1514200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f150c0cea00 con 0x7f150c157ff0 2024-05-28T06:14:33.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.336+0000 7f1514200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f150c0ce880 con 0x7f150c158540 2024-05-28T06:14:33.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.336+0000 7f1514200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f150c0b3150 con 0x7f14f4004d40 2024-05-28T06:14:33.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.336+0000 7f1512200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f150c157ff0 0x7f150c0b1fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:33.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.336+0000 7f1512200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f150c157ff0 0x7f150c0b1fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:43242/0 (socket says 172.21.15.96:43242) 2024-05-28T06:14:33.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.336+0000 7f1512200640 1 -- 172.21.15.96:0/4272475478 learned_addr learned my addr 172.21.15.96:0/4272475478 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:33.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.336+0000 7f1511800640 1 --2- 172.21.15.96:0/4272475478 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f14f4004d40 0x7f150c14d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:33.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.336+0000 7f1510e00640 1 --2- 172.21.15.96:0/4272475478 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f150c158540 0x7f150c0b1aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:33.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.337+0000 7f1512200640 1 -- 172.21.15.96:0/4272475478 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f14f4004d40 msgr2=0x7f150c14d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:33.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.337+0000 7f1512200640 1 --2- 172.21.15.96:0/4272475478 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f14f4004d40 0x7f150c14d1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:33.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.337+0000 7f1512200640 1 -- 172.21.15.96:0/4272475478 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f150c158540 msgr2=0x7f150c0b1aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:33.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.337+0000 7f1512200640 1 --2- 172.21.15.96:0/4272475478 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f150c158540 0x7f150c0b1aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:33.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.337+0000 7f1512200640 1 -- 172.21.15.96:0/4272475478 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f150c0c8d90 con 0x7f150c157ff0 2024-05-28T06:14:33.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.337+0000 7f1512200640 1 --2- 172.21.15.96:0/4272475478 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f150c157ff0 0x7f150c0b1fe0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f15040103e0 tx=0x7f150400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:33.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.337+0000 7f1509600640 1 -- 172.21.15.96:0/4272475478 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1504034070 con 0x7f150c157ff0 2024-05-28T06:14:33.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.337+0000 7f1509600640 1 -- 172.21.15.96:0/4272475478 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1504009e50 con 0x7f150c157ff0 2024-05-28T06:14:33.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.337+0000 7f1514200640 1 -- 172.21.15.96:0/4272475478 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f150c0b0e80 con 0x7f150c157ff0 2024-05-28T06:14:33.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.337+0000 7f1509600640 1 -- 172.21.15.96:0/4272475478 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f150400be20 con 0x7f150c157ff0 2024-05-28T06:14:33.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.337+0000 7f1514200640 1 -- 172.21.15.96:0/4272475478 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f150c0c9050 con 0x7f150c157ff0 2024-05-28T06:14:33.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.339+0000 7f1514200640 1 -- 172.21.15.96:0/4272475478 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f150c0689e0 con 0x7f150c157ff0 2024-05-28T06:14:33.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.340+0000 7f1509600640 1 -- 172.21.15.96:0/4272475478 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 22) ==== 92213+0+0 (secure 0 0 0) 0x7f1504005d10 con 0x7f150c157ff0 2024-05-28T06:14:33.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.340+0000 7f1509600640 1 --2- 172.21.15.96:0/4272475478 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f14ec06d5e0 0x7f14ec06faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:33.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.341+0000 7f1509600640 1 -- 172.21.15.96:0/4272475478 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5765+0+0 (secure 0 0 0) 0x7f15040acfc0 con 0x7f150c157ff0 2024-05-28T06:14:33.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.341+0000 7f1511800640 1 --2- 172.21.15.96:0/4272475478 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f14ec06d5e0 0x7f14ec06faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:33.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.342+0000 7f1511800640 1 --2- 172.21.15.96:0/4272475478 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f14ec06d5e0 0x7f14ec06faa0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f1500009b20 tx=0x7f1500008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:33.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.345+0000 7f1509600640 1 -- 172.21.15.96:0/4272475478 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1504049060 con 0x7f150c157ff0 2024-05-28T06:14:33.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.545+0000 7f1514200640 1 -- 172.21.15.96:0/4272475478 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f150c1594b0 con 0x7f150c157ff0 2024-05-28T06:14:33.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.547+0000 7f1509600640 1 -- 172.21.15.96:0/4272475478 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v62) ==== 74+0+16785 (secure 0 0 0) 0x7f150407ce40 con 0x7f150c157ff0 2024-05-28T06:14:33.548 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:14:33.548 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":62,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:14:31.398967+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-05-28T06:12:26.024060+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-05-28T06:12:27.254401+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":61,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[{"pool":9,"snaps":[{"begin":2,"length":2}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:14:33.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.550+0000 7f1514200640 1 -- 172.21.15.96:0/4272475478 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f14ec06d5e0 msgr2=0x7f14ec06faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:33.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.550+0000 7f1514200640 1 --2- 172.21.15.96:0/4272475478 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f14ec06d5e0 0x7f14ec06faa0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f1500009b20 tx=0x7f1500008040 comp rx=0 tx=0).stop 2024-05-28T06:14:33.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.550+0000 7f1514200640 1 -- 172.21.15.96:0/4272475478 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f150c157ff0 msgr2=0x7f150c0b1fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:33.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.550+0000 7f1514200640 1 --2- 172.21.15.96:0/4272475478 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f150c157ff0 0x7f150c0b1fe0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f15040103e0 tx=0x7f150400b960 comp rx=0 tx=0).stop 2024-05-28T06:14:33.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.550+0000 7f1514200640 1 -- 172.21.15.96:0/4272475478 shutdown_connections 2024-05-28T06:14:33.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.550+0000 7f1514200640 1 --2- 172.21.15.96:0/4272475478 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f14ec06d5e0 0x7f14ec06faa0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:33.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.550+0000 7f1514200640 1 --2- 172.21.15.96:0/4272475478 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f150c157ff0 0x7f150c0b1fe0 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:33.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.550+0000 7f1514200640 1 --2- 172.21.15.96:0/4272475478 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f150c158540 0x7f150c0b1aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:33.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.550+0000 7f1514200640 1 --2- 172.21.15.96:0/4272475478 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f14f4004d40 0x7f150c14d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:33.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.550+0000 7f1514200640 1 -- 172.21.15.96:0/4272475478 >> 172.21.15.96:0/4272475478 conn(0x7f150c06bae0 msgr2=0x7f150c0ca8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:33.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.550+0000 7f1514200640 1 -- 172.21.15.96:0/4272475478 shutdown_connections 2024-05-28T06:14:33.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.550+0000 7f1514200640 1 -- 172.21.15.96:0/4272475478 wait complete. 2024-05-28T06:14:33.569 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-28T06:03:59.517102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-28T06:12:26.024060+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '59', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-05-28T06:12:27.254401+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '61', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 61, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-28T06:14:33.569 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-28T06:14:33.760 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.759+0000 7ff6a7e00640 1 Processor -- start 2024-05-28T06:14:33.761 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.759+0000 7ff6a7e00640 1 -- start start 2024-05-28T06:14:33.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.761+0000 7ff6a7e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff6a00ccfd0 0x7ff6a015d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:33.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.761+0000 7ff6a7e00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6a00cd640 con 0x7ff6a009ab30 2024-05-28T06:14:33.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.761+0000 7ff6a7e00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6a00cd920 con 0x7ff6a00d09b0 2024-05-28T06:14:33.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.761+0000 7ff6a7e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ff6a015dd60 con 0x7ff6a00ccfd0 2024-05-28T06:14:33.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.761+0000 7ff6a5e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7ff6a009ab30 0x7ff6a009afc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:44182/0 (socket says 172.21.15.96:44182) 2024-05-28T06:14:33.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.761+0000 7ff6a5e00640 1 -- 172.21.15.96:0/34856309 learned_addr learned my addr 172.21.15.96:0/34856309 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:14:33.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.762+0000 7ff6a4a00640 1 --2- 172.21.15.96:0/34856309 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff6a00ccfd0 0x7ff6a015d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:33.767 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.762+0000 7ff69fe00640 1 -- 172.21.15.96:0/34856309 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3863108369 0 0) 0x7ff6a00cd640 con 0x7ff6a009ab30 2024-05-28T06:14:33.767 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.762+0000 7ff69fe00640 1 -- 172.21.15.96:0/34856309 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff684003520 con 0x7ff6a009ab30 2024-05-28T06:14:33.767 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.762+0000 7ff69fe00640 1 -- 172.21.15.96:0/34856309 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1608431121 0 0) 0x7ff684003520 con 0x7ff6a009ab30 2024-05-28T06:14:33.767 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.762+0000 7ff69fe00640 1 -- 172.21.15.96:0/34856309 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff6a00ccfd0 msgr2=0x7ff6a015d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:33.767 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.762+0000 7ff69fe00640 1 --2- 172.21.15.96:0/34856309 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff6a00ccfd0 0x7ff6a015d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:33.767 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.762+0000 7ff69fe00640 1 -- 172.21.15.96:0/34856309 >> v1:172.21.15.123:6789/0 conn(0x7ff6a00d09b0 legacy=0x7ff6a00d0d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:33.767 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.762+0000 7ff69fe00640 1 -- 172.21.15.96:0/34856309 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6a00cdc00 con 0x7ff6a009ab30 2024-05-28T06:14:33.767 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.762+0000 7ff69fe00640 1 -- 172.21.15.96:0/34856309 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7ff698001670 con 0x7ff6a009ab30 2024-05-28T06:14:33.767 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.762+0000 7ff69fe00640 1 -- 172.21.15.96:0/34856309 >> v1:172.21.15.96:6789/0 conn(0x7ff6a009ab30 legacy=0x7ff6a009afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:33.767 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.762+0000 7ff69fe00640 1 --2- 172.21.15.96:0/34856309 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff6840039e0 0x7ff684003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:33.767 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.762+0000 7ff69fe00640 1 --2- 172.21.15.96:0/34856309 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff684004800 0x7ff684004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:33.768 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.762+0000 7ff69fe00640 1 --2- 172.21.15.96:0/34856309 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff6a00ccfd0 0x7ff6840051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:33.768 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.762+0000 7ff69fe00640 1 -- 172.21.15.96:0/34856309 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff6a015dd60 con 0x7ff684004800 2024-05-28T06:14:33.768 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.762+0000 7ff69fe00640 1 -- 172.21.15.96:0/34856309 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff68400baa0 con 0x7ff6840039e0 2024-05-28T06:14:33.768 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.762+0000 7ff69fe00640 1 -- 172.21.15.96:0/34856309 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ff68400bca0 con 0x7ff6a00ccfd0 2024-05-28T06:14:33.768 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.763+0000 7ff6a4a00640 1 --2- 172.21.15.96:0/34856309 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff684004800 0x7ff684004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:33.768 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.763+0000 7ff6a5400640 1 --2- 172.21.15.96:0/34856309 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff6840039e0 0x7ff684003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:33.768 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.763+0000 7ff6a5e00640 1 --2- 172.21.15.96:0/34856309 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff6a00ccfd0 0x7ff6840051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:33.768 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.763+0000 7ff6a4a00640 1 -- 172.21.15.96:0/34856309 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff6a00ccfd0 msgr2=0x7ff6840051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:33.768 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.763+0000 7ff6a4a00640 1 --2- 172.21.15.96:0/34856309 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff6a00ccfd0 0x7ff6840051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:33.768 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.763+0000 7ff6a4a00640 1 -- 172.21.15.96:0/34856309 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff6840039e0 msgr2=0x7ff684003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:33.768 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.763+0000 7ff6a4a00640 1 --2- 172.21.15.96:0/34856309 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff6840039e0 0x7ff684003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:33.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.763+0000 7ff6a4a00640 1 -- 172.21.15.96:0/34856309 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff684003520 con 0x7ff684004800 2024-05-28T06:14:33.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.763+0000 7ff6a5e00640 1 --2- 172.21.15.96:0/34856309 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff6a00ccfd0 0x7ff6840051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:14:33.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.763+0000 7ff6a5400640 1 --2- 172.21.15.96:0/34856309 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff6840039e0 0x7ff684003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:14:33.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.764+0000 7ff6a4a00640 1 --2- 172.21.15.96:0/34856309 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff684004800 0x7ff684004c80 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7ff68c009a90 tx=0x7ff684005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ea8025c55c87286 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:33.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.764+0000 7ff69fe00640 1 -- 172.21.15.96:0/34856309 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff68c036070 con 0x7ff684004800 2024-05-28T06:14:33.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.764+0000 7ff69fe00640 1 -- 172.21.15.96:0/34856309 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff68c031400 con 0x7ff684004800 2024-05-28T06:14:33.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.764+0000 7ff69fe00640 1 -- 172.21.15.96:0/34856309 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff68c009690 con 0x7ff684004800 2024-05-28T06:14:33.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.765+0000 7ff6a7e00640 1 -- 172.21.15.96:0/34856309 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff684004800 msgr2=0x7ff684004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:33.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.765+0000 7ff6a7e00640 1 --2- 172.21.15.96:0/34856309 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff684004800 0x7ff684004c80 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7ff68c009a90 tx=0x7ff684005830 comp rx=0 tx=0).stop 2024-05-28T06:14:33.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.765+0000 7ff6a5e00640 1 -- 172.21.15.96:0/34856309 reap_dead start 2024-05-28T06:14:33.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.765+0000 7ff6a7e00640 1 -- 172.21.15.96:0/34856309 shutdown_connections 2024-05-28T06:14:33.770 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.765+0000 7ff6a7e00640 1 -- 172.21.15.96:0/34856309 >> 172.21.15.96:0/34856309 conn(0x7ff6a00d2430 msgr2=0x7ff6a00d2840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:33.770 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.765+0000 7ff6a7e00640 1 -- 172.21.15.96:0/34856309 shutdown_connections 2024-05-28T06:14:33.770 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.765+0000 7ff6a7e00640 1 -- 172.21.15.96:0/34856309 wait complete. 2024-05-28T06:14:33.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.767+0000 7ff6a7e00640 1 Processor -- start 2024-05-28T06:14:33.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.767+0000 7ff6a7e00640 1 -- start start 2024-05-28T06:14:33.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.768+0000 7ff6a7e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff684004800 0x7ff6a00c7b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:33.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.768+0000 7ff6a7e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff6a0157cd0 0x7ff6a00b3500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:33.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.768+0000 7ff6a7e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff6a00b3a40 0x7ff6a015f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:33.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.768+0000 7ff6a7e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff6a015cb90 con 0x7ff684004800 2024-05-28T06:14:33.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.768+0000 7ff6a7e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff6a00ce1c0 con 0x7ff6a00b3a40 2024-05-28T06:14:33.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.768+0000 7ff6a7e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ff6a015d820 con 0x7ff6a0157cd0 2024-05-28T06:14:33.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.768+0000 7ff6a5400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff684004800 0x7ff6a00c7b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:33.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.768+0000 7ff6a5400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff684004800 0x7ff6a00c7b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:43258/0 (socket says 172.21.15.96:43258) 2024-05-28T06:14:33.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.768+0000 7ff6a5400640 1 -- 172.21.15.96:0/1700864476 learned_addr learned my addr 172.21.15.96:0/1700864476 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:33.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.769+0000 7ff6a5e00640 1 --2- 172.21.15.96:0/1700864476 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff6a00b3a40 0x7ff6a015f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:33.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.769+0000 7ff6a5400640 1 -- 172.21.15.96:0/1700864476 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff6a0157cd0 msgr2=0x7ff6a00b3500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:33.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.769+0000 7ff6a4a00640 1 --2- 172.21.15.96:0/1700864476 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff6a0157cd0 0x7ff6a00b3500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:33.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.769+0000 7ff6a5400640 1 --2- 172.21.15.96:0/1700864476 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff6a0157cd0 0x7ff6a00b3500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:33.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.769+0000 7ff6a5400640 1 -- 172.21.15.96:0/1700864476 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff6a00b3a40 msgr2=0x7ff6a015f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:33.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.769+0000 7ff6a5400640 1 --2- 172.21.15.96:0/1700864476 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff6a00b3a40 0x7ff6a015f470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:33.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.769+0000 7ff6a5400640 1 -- 172.21.15.96:0/1700864476 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6a00c6af0 con 0x7ff684004800 2024-05-28T06:14:33.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.769+0000 7ff6a5400640 1 --2- 172.21.15.96:0/1700864476 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff684004800 0x7ff6a00c7b90 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7ff69000db10 tx=0x7ff6900020c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:33.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.769+0000 7ff69d600640 1 -- 172.21.15.96:0/1700864476 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff690003ba0 con 0x7ff684004800 2024-05-28T06:14:33.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.769+0000 7ff69d600640 1 -- 172.21.15.96:0/1700864476 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff690003d40 con 0x7ff684004800 2024-05-28T06:14:33.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.769+0000 7ff69d600640 1 -- 172.21.15.96:0/1700864476 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff690006d70 con 0x7ff684004800 2024-05-28T06:14:33.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.769+0000 7ff6a7e00640 1 -- 172.21.15.96:0/1700864476 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6a00c6dd0 con 0x7ff684004800 2024-05-28T06:14:33.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.769+0000 7ff6a7e00640 1 -- 172.21.15.96:0/1700864476 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6a00c7390 con 0x7ff684004800 2024-05-28T06:14:33.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.771+0000 7ff6a7e00640 1 -- 172.21.15.96:0/1700864476 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6a00689e0 con 0x7ff684004800 2024-05-28T06:14:33.774 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.772+0000 7ff69d600640 1 -- 172.21.15.96:0/1700864476 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 22) ==== 92213+0+0 (secure 0 0 0) 0x7ff69001a020 con 0x7ff684004800 2024-05-28T06:14:33.774 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.773+0000 7ff69d600640 1 --2- 172.21.15.96:0/1700864476 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff68006d5e0 0x7ff68006faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:33.774 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.773+0000 7ff69d600640 1 -- 172.21.15.96:0/1700864476 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5765+0+0 (secure 0 0 0) 0x7ff69008a3c0 con 0x7ff684004800 2024-05-28T06:14:33.774 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.774+0000 7ff6a4a00640 1 --2- 172.21.15.96:0/1700864476 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff68006d5e0 0x7ff68006faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:33.775 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.775+0000 7ff6a4a00640 1 --2- 172.21.15.96:0/1700864476 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff68006d5e0 0x7ff68006faa0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7ff68c033000 tx=0x7ff68c008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:33.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.777+0000 7ff69d600640 1 -- 172.21.15.96:0/1700864476 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff690059ff0 con 0x7ff684004800 2024-05-28T06:14:33.979 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.978+0000 7ff6a7e00640 1 -- 172.21.15.96:0/1700864476 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff6a015bcb0 con 0x7ff684004800 2024-05-28T06:14:33.980 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.979+0000 7ff69d600640 1 -- 172.21.15.96:0/1700864476 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v62) ==== 93+0+10 (secure 0 0 0) 0x7ff6900598f0 con 0x7ff684004800 2024-05-28T06:14:33.980 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 1 2024-05-28T06:14:33.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.982+0000 7ff6a7e00640 1 -- 172.21.15.96:0/1700864476 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff68006d5e0 msgr2=0x7ff68006faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:33.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.982+0000 7ff6a7e00640 1 --2- 172.21.15.96:0/1700864476 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff68006d5e0 0x7ff68006faa0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7ff68c033000 tx=0x7ff68c008030 comp rx=0 tx=0).stop 2024-05-28T06:14:33.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.982+0000 7ff6a7e00640 1 -- 172.21.15.96:0/1700864476 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff684004800 msgr2=0x7ff6a00c7b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:33.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.982+0000 7ff6a7e00640 1 --2- 172.21.15.96:0/1700864476 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff684004800 0x7ff6a00c7b90 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7ff69000db10 tx=0x7ff6900020c0 comp rx=0 tx=0).stop 2024-05-28T06:14:33.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.982+0000 7ff6a7e00640 1 -- 172.21.15.96:0/1700864476 shutdown_connections 2024-05-28T06:14:33.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.982+0000 7ff6a7e00640 1 --2- 172.21.15.96:0/1700864476 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff68006d5e0 0x7ff68006faa0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:33.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.982+0000 7ff6a7e00640 1 --2- 172.21.15.96:0/1700864476 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff6a00b3a40 0x7ff6a015f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:33.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.982+0000 7ff6a7e00640 1 --2- 172.21.15.96:0/1700864476 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff6a0157cd0 0x7ff6a00b3500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:33.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.982+0000 7ff6a7e00640 1 --2- 172.21.15.96:0/1700864476 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff684004800 0x7ff6a00c7b90 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:33.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.982+0000 7ff6a7e00640 1 -- 172.21.15.96:0/1700864476 >> 172.21.15.96:0/1700864476 conn(0x7ff6a00d2430 msgr2=0x7ff6a006c9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:33.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.983+0000 7ff6a7e00640 1 -- 172.21.15.96:0/1700864476 shutdown_connections 2024-05-28T06:14:33.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:33.983+0000 7ff6a7e00640 1 -- 172.21.15.96:0/1700864476 wait complete. 2024-05-28T06:14:34.003 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-28T06:14:34.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.208+0000 7f4edf200640 1 Processor -- start 2024-05-28T06:14:34.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.208+0000 7f4edf200640 1 -- start start 2024-05-28T06:14:34.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.210+0000 7f4edf200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f4ed80d0640 0x7f4ed815d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:34.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.210+0000 7f4edf200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4ed80d0a20 0x7f4ed80cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:34.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.210+0000 7f4edf200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ed815cb10 con 0x7f4ed80b32c0 2024-05-28T06:14:34.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.210+0000 7f4edf200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f4ed80adbb0 con 0x7f4ed80d0640 2024-05-28T06:14:34.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.210+0000 7f4edf200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f4ed815db10 con 0x7f4ed80d0a20 2024-05-28T06:14:34.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.210+0000 7f4ed7e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f4ed80d0640 0x7f4ed815d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:34.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.210+0000 7f4ed7e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f4ed80d0640 0x7f4ed815d5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:43262/0 (socket says 172.21.15.96:43262) 2024-05-28T06:14:34.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.210+0000 7f4ed7e00640 1 -- 172.21.15.96:0/2713871495 learned_addr learned my addr 172.21.15.96:0/2713871495 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:34.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.210+0000 7f4ed7400640 1 --2- 172.21.15.96:0/2713871495 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4ed80d0a20 0x7f4ed80cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:34.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.211+0000 7f4ed7e00640 1 -- 172.21.15.96:0/2713871495 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4ed80d0a20 msgr2=0x7f4ed80cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:34.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.211+0000 7f4ed7e00640 1 --2- 172.21.15.96:0/2713871495 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4ed80d0a20 0x7f4ed80cf950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:34.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.211+0000 7f4ed7e00640 1 -- 172.21.15.96:0/2713871495 >> v1:172.21.15.123:6789/0 conn(0x7f4ed80b32c0 legacy=0x7f4ed80b3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:34.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.211+0000 7f4ed7e00640 1 -- 172.21.15.96:0/2713871495 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ed80ad5f0 con 0x7f4ed80d0640 2024-05-28T06:14:34.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.212+0000 7f4ed7e00640 1 --2- 172.21.15.96:0/2713871495 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4ed80d0640 0x7f4ed815d5d0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f4ec8009870 tx=0x7f4ec802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c7cba03cf3f9918 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:34.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.213+0000 7f4ed6a00640 1 -- 172.21.15.96:0/2713871495 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ec8037070 con 0x7f4ed80d0640 2024-05-28T06:14:34.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.213+0000 7f4ed6a00640 1 -- 172.21.15.96:0/2713871495 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ec8032440 con 0x7f4ed80d0640 2024-05-28T06:14:34.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.213+0000 7f4ed6a00640 1 -- 172.21.15.96:0/2713871495 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ec8032810 con 0x7f4ed80d0640 2024-05-28T06:14:34.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.213+0000 7f4edf200640 1 -- 172.21.15.96:0/2713871495 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4ed80d0640 msgr2=0x7f4ed815d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:34.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.213+0000 7f4edf200640 1 --2- 172.21.15.96:0/2713871495 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4ed80d0640 0x7f4ed815d5d0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f4ec8009870 tx=0x7f4ec802a270 comp rx=0 tx=0).stop 2024-05-28T06:14:34.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.213+0000 7f4edf200640 1 -- 172.21.15.96:0/2713871495 shutdown_connections 2024-05-28T06:14:34.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.213+0000 7f4edf200640 1 --2- 172.21.15.96:0/2713871495 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4ed80d0a20 0x7f4ed80cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:34.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.213+0000 7f4edf200640 1 --2- 172.21.15.96:0/2713871495 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4ed80d0640 0x7f4ed815d5d0 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:34.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.213+0000 7f4edf200640 1 -- 172.21.15.96:0/2713871495 >> 172.21.15.96:0/2713871495 conn(0x7f4ed806bd20 msgr2=0x7f4ed806c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:34.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.214+0000 7f4edf200640 1 -- 172.21.15.96:0/2713871495 shutdown_connections 2024-05-28T06:14:34.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.214+0000 7f4edf200640 1 -- 172.21.15.96:0/2713871495 wait complete. 2024-05-28T06:14:34.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.216+0000 7f4edf200640 1 Processor -- start 2024-05-28T06:14:34.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.216+0000 7f4edf200640 1 -- start start 2024-05-28T06:14:34.218 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.217+0000 7f4edf200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4ed80b32c0 0x7f4ed8155ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:34.218 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.217+0000 7f4edf200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4ed80d0640 0x7f4ed81f4980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:34.218 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.217+0000 7f4edf200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4ed80d0a20 0x7f4ed814bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:34.218 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.217+0000 7f4edf200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4ed815d0b0 con 0x7f4ed80d0a20 2024-05-28T06:14:34.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.217+0000 7f4edf200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f4ed815cdd0 con 0x7f4ed80d0640 2024-05-28T06:14:34.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.217+0000 7f4edf200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f4ed80b3e60 con 0x7f4ed80b32c0 2024-05-28T06:14:34.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.217+0000 7f4edd200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4ed80d0a20 0x7f4ed814bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:34.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.217+0000 7f4edd200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4ed80d0a20 0x7f4ed814bfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:43272/0 (socket says 172.21.15.96:43272) 2024-05-28T06:14:34.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.217+0000 7f4edd200640 1 -- 172.21.15.96:0/3858849869 learned_addr learned my addr 172.21.15.96:0/3858849869 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:34.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.218+0000 7f4ed7400640 1 --2- 172.21.15.96:0/3858849869 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4ed80d0640 0x7f4ed81f4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:34.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.218+0000 7f4ed7e00640 1 --2- 172.21.15.96:0/3858849869 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4ed80b32c0 0x7f4ed8155ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:34.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.218+0000 7f4edd200640 1 -- 172.21.15.96:0/3858849869 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4ed80b32c0 msgr2=0x7f4ed8155ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:34.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.218+0000 7f4edd200640 1 --2- 172.21.15.96:0/3858849869 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4ed80b32c0 0x7f4ed8155ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:34.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.218+0000 7f4edd200640 1 -- 172.21.15.96:0/3858849869 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4ed80d0640 msgr2=0x7f4ed81f4980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:34.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.218+0000 7f4edd200640 1 --2- 172.21.15.96:0/3858849869 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4ed80d0640 0x7f4ed81f4980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:34.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.218+0000 7f4edd200640 1 -- 172.21.15.96:0/3858849869 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ed81f4ec0 con 0x7f4ed80d0a20 2024-05-28T06:14:34.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.218+0000 7f4edd200640 1 --2- 172.21.15.96:0/3858849869 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4ed80d0a20 0x7f4ed814bfe0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f4ed000c7a0 tx=0x7f4ed000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:34.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.218+0000 7f4ecfe00640 1 -- 172.21.15.96:0/3858849869 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ed0003b70 con 0x7f4ed80d0a20 2024-05-28T06:14:34.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.218+0000 7f4ecfe00640 1 -- 172.21.15.96:0/3858849869 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ed0003d30 con 0x7f4ed80d0a20 2024-05-28T06:14:34.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.218+0000 7f4ecfe00640 1 -- 172.21.15.96:0/3858849869 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ed000aa90 con 0x7f4ed80d0a20 2024-05-28T06:14:34.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.218+0000 7f4edf200640 1 -- 172.21.15.96:0/3858849869 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ed81f5160 con 0x7f4ed80d0a20 2024-05-28T06:14:34.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.219+0000 7f4edf200640 1 -- 172.21.15.96:0/3858849869 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ed81f5720 con 0x7f4ed80d0a20 2024-05-28T06:14:34.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.220+0000 7f4edf200640 1 -- 172.21.15.96:0/3858849869 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ed80b3bd0 con 0x7f4ed80d0a20 2024-05-28T06:14:34.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.221+0000 7f4ecfe00640 1 -- 172.21.15.96:0/3858849869 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 22) ==== 92213+0+0 (secure 0 0 0) 0x7f4ed000ac30 con 0x7f4ed80d0a20 2024-05-28T06:14:34.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.222+0000 7f4ecfe00640 1 --2- 172.21.15.96:0/3858849869 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4eb806d5e0 0x7f4eb806faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:34.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.222+0000 7f4ecfe00640 1 -- 172.21.15.96:0/3858849869 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5765+0+0 (secure 0 0 0) 0x7f4ed008a550 con 0x7f4ed80d0a20 2024-05-28T06:14:34.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.222+0000 7f4ed7e00640 1 --2- 172.21.15.96:0/3858849869 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4eb806d5e0 0x7f4eb806faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:34.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.223+0000 7f4ed7e00640 1 --2- 172.21.15.96:0/3858849869 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4eb806d5e0 0x7f4eb806faa0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f4ec8034000 tx=0x7f4ec802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:34.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.226+0000 7f4ecfe00640 1 -- 172.21.15.96:0/3858849869 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4ed80b3bd0 con 0x7f4ed80d0a20 2024-05-28T06:14:34.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.409+0000 7f4edf200640 1 -- 172.21.15.96:0/3858849869 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f4ed80689e0 con 0x7f4ed80d0a20 2024-05-28T06:14:34.410 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.409+0000 7f4ecfe00640 1 -- 172.21.15.96:0/3858849869 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v62) ==== 104+0+11 (secure 0 0 0) 0x7f4ed005a3d0 con 0x7f4ed80d0a20 2024-05-28T06:14:34.410 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 32 2024-05-28T06:14:34.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.412+0000 7f4edf200640 1 -- 172.21.15.96:0/3858849869 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4eb806d5e0 msgr2=0x7f4eb806faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:34.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.412+0000 7f4edf200640 1 --2- 172.21.15.96:0/3858849869 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4eb806d5e0 0x7f4eb806faa0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f4ec8034000 tx=0x7f4ec802e000 comp rx=0 tx=0).stop 2024-05-28T06:14:34.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.412+0000 7f4edf200640 1 -- 172.21.15.96:0/3858849869 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4ed80d0a20 msgr2=0x7f4ed814bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:34.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.412+0000 7f4edf200640 1 --2- 172.21.15.96:0/3858849869 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4ed80d0a20 0x7f4ed814bfe0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f4ed000c7a0 tx=0x7f4ed000cc60 comp rx=0 tx=0).stop 2024-05-28T06:14:34.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.412+0000 7f4edf200640 1 -- 172.21.15.96:0/3858849869 shutdown_connections 2024-05-28T06:14:34.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.412+0000 7f4edf200640 1 --2- 172.21.15.96:0/3858849869 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4eb806d5e0 0x7f4eb806faa0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:34.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.412+0000 7f4edf200640 1 --2- 172.21.15.96:0/3858849869 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4ed80d0a20 0x7f4ed814bfe0 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:34.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.412+0000 7f4edf200640 1 --2- 172.21.15.96:0/3858849869 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4ed80d0640 0x7f4ed81f4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:34.414 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.412+0000 7f4edf200640 1 --2- 172.21.15.96:0/3858849869 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4ed80b32c0 0x7f4ed8155ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:34.414 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.412+0000 7f4edf200640 1 -- 172.21.15.96:0/3858849869 >> 172.21.15.96:0/3858849869 conn(0x7f4ed806bd20 msgr2=0x7f4ed806d9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:34.414 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.412+0000 7f4edf200640 1 -- 172.21.15.96:0/3858849869 shutdown_connections 2024-05-28T06:14:34.414 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.412+0000 7f4edf200640 1 -- 172.21.15.96:0/3858849869 wait complete. 2024-05-28T06:14:34.431 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-28T06:14:34.631 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.630+0000 7f4713000640 1 Processor -- start 2024-05-28T06:14:34.631 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.630+0000 7f4713000640 1 -- start start 2024-05-28T06:14:34.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.631+0000 7f4713000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f470c0d0640 0x7f470c15d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:34.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.631+0000 7f4713000640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f470c0d0060 con 0x7f470c0b32c0 2024-05-28T06:14:34.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.631+0000 7f4713000640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f470c15cb10 con 0x7f470c0d0a20 2024-05-28T06:14:34.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.631+0000 7f4713000640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f470c0adbb0 con 0x7f470c0d0640 2024-05-28T06:14:34.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.632+0000 7f4711000640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f470c0b32c0 0x7f470c0b3730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:44192/0 (socket says 172.21.15.96:44192) 2024-05-28T06:14:34.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.632+0000 7f4711000640 1 -- 172.21.15.96:0/2459566896 learned_addr learned my addr 172.21.15.96:0/2459566896 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:14:34.635 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.632+0000 7f470be00640 1 --2- 172.21.15.96:0/2459566896 >> v2:172.21.15.123:3300/0 conn(0x7f470c0d0640 0x7f470c15d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:34.635 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.632+0000 7f470aa00640 1 -- 172.21.15.96:0/2459566896 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2827639729 0 0) 0x7f470c0d0060 con 0x7f470c0b32c0 2024-05-28T06:14:34.635 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.632+0000 7f470aa00640 1 -- 172.21.15.96:0/2459566896 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f46f0003520 con 0x7f470c0b32c0 2024-05-28T06:14:34.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.633+0000 7f470aa00640 1 -- 172.21.15.96:0/2459566896 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4264232293 0 0) 0x7f46f0003520 con 0x7f470c0b32c0 2024-05-28T06:14:34.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.633+0000 7f470aa00640 1 -- 172.21.15.96:0/2459566896 >> v2:172.21.15.123:3300/0 conn(0x7f470c0d0640 msgr2=0x7f470c15d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:34.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.633+0000 7f470aa00640 1 --2- 172.21.15.96:0/2459566896 >> v2:172.21.15.123:3300/0 conn(0x7f470c0d0640 0x7f470c15d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:34.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.633+0000 7f470aa00640 1 -- 172.21.15.96:0/2459566896 >> v1:172.21.15.123:6789/0 conn(0x7f470c0d0a20 legacy=0x7f470c0cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:34.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.633+0000 7f470aa00640 1 -- 172.21.15.96:0/2459566896 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f470c0ad5f0 con 0x7f470c0b32c0 2024-05-28T06:14:34.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.633+0000 7f470aa00640 1 -- 172.21.15.96:0/2459566896 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f47040019f0 con 0x7f470c0b32c0 2024-05-28T06:14:34.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.633+0000 7f470aa00640 1 -- 172.21.15.96:0/2459566896 >> v1:172.21.15.96:6789/0 conn(0x7f470c0b32c0 legacy=0x7f470c0b3730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:34.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.633+0000 7f470aa00640 1 --2- 172.21.15.96:0/2459566896 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f46f0003a00 0x7f46f0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:34.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.633+0000 7f470aa00640 1 --2- 172.21.15.96:0/2459566896 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f46f0004800 0x7f46f0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:34.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.633+0000 7f470aa00640 1 --2- 172.21.15.96:0/2459566896 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f46f0005210 0x7f46f000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:34.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.633+0000 7f470aa00640 1 -- 172.21.15.96:0/2459566896 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f46f000c130 con 0x7f46f0003a00 2024-05-28T06:14:34.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.633+0000 7f470aa00640 1 -- 172.21.15.96:0/2459566896 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f46f000c2e0 con 0x7f46f0004800 2024-05-28T06:14:34.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.633+0000 7f470aa00640 1 -- 172.21.15.96:0/2459566896 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f46f000c4e0 con 0x7f46f0005210 2024-05-28T06:14:34.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.633+0000 7f470be00640 1 --2- 172.21.15.96:0/2459566896 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f46f0003a00 0x7f46f0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:34.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.633+0000 7f470be00640 1 -- 172.21.15.96:0/2459566896 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f46f0005210 msgr2=0x7f46f000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:34.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.633+0000 7f470be00640 1 --2- 172.21.15.96:0/2459566896 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f46f0005210 0x7f46f000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:34.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.633+0000 7f470be00640 1 -- 172.21.15.96:0/2459566896 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f46f0004800 msgr2=0x7f46f0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:34.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.633+0000 7f470be00640 1 --2- 172.21.15.96:0/2459566896 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f46f0004800 0x7f46f0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:34.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.633+0000 7f470be00640 1 -- 172.21.15.96:0/2459566896 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46f0003520 con 0x7f46f0003a00 2024-05-28T06:14:34.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.633+0000 7f4711000640 1 -- 172.21.15.96:0/2459566896 reap_dead start 2024-05-28T06:14:34.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.635+0000 7f470be00640 1 --2- 172.21.15.96:0/2459566896 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f46f0003a00 0x7f46f0003e20 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f46fc009a90 tx=0x7f46f0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7ec711c683cf0d server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:34.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.635+0000 7f470aa00640 1 -- 172.21.15.96:0/2459566896 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46fc036070 con 0x7f46f0003a00 2024-05-28T06:14:34.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.635+0000 7f470aa00640 1 -- 172.21.15.96:0/2459566896 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46fc031400 con 0x7f46f0003a00 2024-05-28T06:14:34.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.635+0000 7f470aa00640 1 -- 172.21.15.96:0/2459566896 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46fc009690 con 0x7f46f0003a00 2024-05-28T06:14:34.639 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.635+0000 7f4713000640 1 -- 172.21.15.96:0/2459566896 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f46f0003a00 msgr2=0x7f46f0003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:34.640 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.635+0000 7f4713000640 1 --2- 172.21.15.96:0/2459566896 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f46f0003a00 0x7f46f0003e20 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f46fc009a90 tx=0x7f46f0005830 comp rx=0 tx=0).stop 2024-05-28T06:14:34.640 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.635+0000 7f4713000640 1 -- 172.21.15.96:0/2459566896 shutdown_connections 2024-05-28T06:14:34.640 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.635+0000 7f4713000640 1 --2- 172.21.15.96:0/2459566896 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f46f0003a00 0x7f46f0003e20 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:34.640 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.635+0000 7f4713000640 1 -- 172.21.15.96:0/2459566896 >> 172.21.15.96:0/2459566896 conn(0x7f470c06bd20 msgr2=0x7f470c06c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:34.640 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.635+0000 7f4713000640 1 -- 172.21.15.96:0/2459566896 shutdown_connections 2024-05-28T06:14:34.640 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.635+0000 7f4713000640 1 -- 172.21.15.96:0/2459566896 wait complete. 2024-05-28T06:14:34.640 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.637+0000 7f4713000640 1 Processor -- start 2024-05-28T06:14:34.640 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.637+0000 7f4713000640 1 -- start start 2024-05-28T06:14:34.640 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.638+0000 7f4713000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f46f0003a00 0x7f470c149130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:34.640 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.638+0000 7f4713000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f470c1558e0 0x7f470c15d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:34.640 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.638+0000 7f4713000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f470c147150 0x7f470c15d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:34.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.638+0000 7f4713000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f470c15d0b0 con 0x7f470c1558e0 2024-05-28T06:14:34.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.638+0000 7f4713000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f470c15cdd0 con 0x7f46f0003a00 2024-05-28T06:14:34.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.638+0000 7f4713000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f470c0ad2a0 con 0x7f470c147150 2024-05-28T06:14:34.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.638+0000 7f470b400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f470c1558e0 0x7f470c15d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:34.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.639+0000 7f470b400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f470c1558e0 0x7f470c15d390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:43290/0 (socket says 172.21.15.96:43290) 2024-05-28T06:14:34.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.639+0000 7f470b400640 1 -- 172.21.15.96:0/1386204597 learned_addr learned my addr 172.21.15.96:0/1386204597 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:34.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.639+0000 7f4711000640 1 --2- 172.21.15.96:0/1386204597 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f470c147150 0x7f470c15d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:34.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.639+0000 7f470be00640 1 --2- 172.21.15.96:0/1386204597 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f46f0003a00 0x7f470c149130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:34.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.639+0000 7f470b400640 1 -- 172.21.15.96:0/1386204597 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f470c147150 msgr2=0x7f470c15d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:34.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.639+0000 7f470b400640 1 --2- 172.21.15.96:0/1386204597 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f470c147150 0x7f470c15d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:34.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.639+0000 7f470b400640 1 -- 172.21.15.96:0/1386204597 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f46f0003a00 msgr2=0x7f470c149130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:34.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.639+0000 7f470b400640 1 --2- 172.21.15.96:0/1386204597 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f46f0003a00 0x7f470c149130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:34.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.639+0000 7f470b400640 1 -- 172.21.15.96:0/1386204597 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f470c148690 con 0x7f470c1558e0 2024-05-28T06:14:34.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.639+0000 7f470b400640 1 --2- 172.21.15.96:0/1386204597 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f470c1558e0 0x7f470c15d390 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f46f800c490 tx=0x7f46f800c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:34.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.639+0000 7f4703e00640 1 -- 172.21.15.96:0/1386204597 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46f8017070 con 0x7f470c1558e0 2024-05-28T06:14:34.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.640+0000 7f4703e00640 1 -- 172.21.15.96:0/1386204597 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46f80052f0 con 0x7f470c1558e0 2024-05-28T06:14:34.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.640+0000 7f4703e00640 1 -- 172.21.15.96:0/1386204597 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46f8003830 con 0x7f470c1558e0 2024-05-28T06:14:34.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.640+0000 7f4713000640 1 -- 172.21.15.96:0/1386204597 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f470c06f5d0 con 0x7f470c1558e0 2024-05-28T06:14:34.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.640+0000 7f4713000640 1 -- 172.21.15.96:0/1386204597 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f470c148930 con 0x7f470c1558e0 2024-05-28T06:14:34.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.641+0000 7f4713000640 1 -- 172.21.15.96:0/1386204597 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f470c0689e0 con 0x7f470c1558e0 2024-05-28T06:14:34.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.642+0000 7f4703e00640 1 -- 172.21.15.96:0/1386204597 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 22) ==== 92213+0+0 (secure 0 0 0) 0x7f46f801a070 con 0x7f470c1558e0 2024-05-28T06:14:34.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.643+0000 7f4703e00640 1 --2- 172.21.15.96:0/1386204597 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f46ec06d610 0x7f46ec06fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:34.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.643+0000 7f4703e00640 1 -- 172.21.15.96:0/1386204597 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5765+0+0 (secure 0 0 0) 0x7f46f8089780 con 0x7f470c1558e0 2024-05-28T06:14:34.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.643+0000 7f470be00640 1 --2- 172.21.15.96:0/1386204597 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f46ec06d610 0x7f46ec06fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:34.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.644+0000 7f470be00640 1 --2- 172.21.15.96:0/1386204597 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f46ec06d610 0x7f46ec06fad0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f46fc033000 tx=0x7f46fc008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:34.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.647+0000 7f4703e00640 1 -- 172.21.15.96:0/1386204597 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f46f8059600 con 0x7f470c1558e0 2024-05-28T06:14:34.849 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.848+0000 7f4713000640 1 -- 172.21.15.96:0/1386204597 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f470c073070 con 0x7f470c1558e0 2024-05-28T06:14:34.849 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.849+0000 7f4703e00640 1 -- 172.21.15.96:0/1386204597 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v62) ==== 100+0+11 (secure 0 0 0) 0x7f46f8058f00 con 0x7f470c1558e0 2024-05-28T06:14:34.850 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 64 2024-05-28T06:14:34.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.852+0000 7f4713000640 1 -- 172.21.15.96:0/1386204597 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f46ec06d610 msgr2=0x7f46ec06fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:34.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.852+0000 7f4713000640 1 --2- 172.21.15.96:0/1386204597 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f46ec06d610 0x7f46ec06fad0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f46fc033000 tx=0x7f46fc008030 comp rx=0 tx=0).stop 2024-05-28T06:14:34.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.852+0000 7f4713000640 1 -- 172.21.15.96:0/1386204597 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f470c1558e0 msgr2=0x7f470c15d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:34.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.852+0000 7f4713000640 1 --2- 172.21.15.96:0/1386204597 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f470c1558e0 0x7f470c15d390 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f46f800c490 tx=0x7f46f800c950 comp rx=0 tx=0).stop 2024-05-28T06:14:34.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.852+0000 7f4713000640 1 -- 172.21.15.96:0/1386204597 shutdown_connections 2024-05-28T06:14:34.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.852+0000 7f4713000640 1 --2- 172.21.15.96:0/1386204597 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f46ec06d610 0x7f46ec06fad0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:34.854 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.852+0000 7f4713000640 1 --2- 172.21.15.96:0/1386204597 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f470c147150 0x7f470c15d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:34.854 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.852+0000 7f4713000640 1 --2- 172.21.15.96:0/1386204597 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f470c1558e0 0x7f470c15d390 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:34.854 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.852+0000 7f4713000640 1 --2- 172.21.15.96:0/1386204597 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f46f0003a00 0x7f470c149130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:34.854 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.852+0000 7f4713000640 1 -- 172.21.15.96:0/1386204597 >> 172.21.15.96:0/1386204597 conn(0x7f470c06bd20 msgr2=0x7f470c151e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:34.854 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.852+0000 7f4713000640 1 -- 172.21.15.96:0/1386204597 shutdown_connections 2024-05-28T06:14:34.854 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:34.852+0000 7f4713000640 1 -- 172.21.15.96:0/1386204597 wait complete. 2024-05-28T06:14:34.873 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-28T06:14:35.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.073+0000 7fec56a00640 1 Processor -- start 2024-05-28T06:14:35.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.074+0000 7fec56a00640 1 -- start start 2024-05-28T06:14:35.075 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.075+0000 7fec56a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fec5015fd30 0x7fec501699a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:35.075 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.075+0000 7fec56a00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec50163500 con 0x7fec500d2db0 2024-05-28T06:14:35.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.075+0000 7fec56a00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec50162740 con 0x7fec500b3b50 2024-05-28T06:14:35.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.075+0000 7fec56a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fec500d5540 con 0x7fec5015fd30 2024-05-28T06:14:35.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.075+0000 7fec4fe00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fec500d2db0 0x7fec500d4d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:44204/0 (socket says 172.21.15.96:44204) 2024-05-28T06:14:35.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.075+0000 7fec4fe00640 1 -- 172.21.15.96:0/1620537396 learned_addr learned my addr 172.21.15.96:0/1620537396 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:14:35.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.076+0000 7fec54a00640 1 --2- 172.21.15.96:0/1620537396 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fec5015fd30 0x7fec501699a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:35.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.076+0000 7fec4f400640 1 -- 172.21.15.96:0/1620537396 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1591908978 0 0) 0x7fec50163500 con 0x7fec500d2db0 2024-05-28T06:14:35.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.076+0000 7fec4f400640 1 -- 172.21.15.96:0/1620537396 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fec34003520 con 0x7fec500d2db0 2024-05-28T06:14:35.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.076+0000 7fec4f400640 1 -- 172.21.15.96:0/1620537396 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 5544892 0 0) 0x7fec34003520 con 0x7fec500d2db0 2024-05-28T06:14:35.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.076+0000 7fec4f400640 1 -- 172.21.15.96:0/1620537396 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fec5015fd30 msgr2=0x7fec501699a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:35.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.076+0000 7fec4f400640 1 --2- 172.21.15.96:0/1620537396 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fec5015fd30 0x7fec501699a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:35.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.076+0000 7fec4f400640 1 -- 172.21.15.96:0/1620537396 >> v1:172.21.15.123:6789/0 conn(0x7fec500b3b50 legacy=0x7fec500d3190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:35.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.076+0000 7fec4f400640 1 -- 172.21.15.96:0/1620537396 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec50162a20 con 0x7fec500d2db0 2024-05-28T06:14:35.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.076+0000 7fec4f400640 1 -- 172.21.15.96:0/1620537396 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7fec40001670 con 0x7fec500d2db0 2024-05-28T06:14:35.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.076+0000 7fec4f400640 1 -- 172.21.15.96:0/1620537396 >> v1:172.21.15.96:6789/0 conn(0x7fec500d2db0 legacy=0x7fec500d4d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:35.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.076+0000 7fec4f400640 1 --2- 172.21.15.96:0/1620537396 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fec340039e0 0x7fec34003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:35.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.076+0000 7fec4f400640 1 --2- 172.21.15.96:0/1620537396 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fec34004800 0x7fec34004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:35.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.076+0000 7fec4f400640 1 --2- 172.21.15.96:0/1620537396 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fec34005290 0x7fec3400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:35.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.076+0000 7fec4f400640 1 -- 172.21.15.96:0/1620537396 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fec500d5540 con 0x7fec34004800 2024-05-28T06:14:35.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.076+0000 7fec4f400640 1 -- 172.21.15.96:0/1620537396 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fec3400c010 con 0x7fec340039e0 2024-05-28T06:14:35.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.076+0000 7fec4f400640 1 -- 172.21.15.96:0/1620537396 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fec3400c210 con 0x7fec34005290 2024-05-28T06:14:35.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.077+0000 7fec47400640 1 --2- 172.21.15.96:0/1620537396 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fec34004800 0x7fec34004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:35.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.077+0000 7fec4fe00640 1 --2- 172.21.15.96:0/1620537396 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fec340039e0 0x7fec34003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:35.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.077+0000 7fec54a00640 1 --2- 172.21.15.96:0/1620537396 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fec34005290 0x7fec3400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:35.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.077+0000 7fec47400640 1 -- 172.21.15.96:0/1620537396 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fec34005290 msgr2=0x7fec3400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:35.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.077+0000 7fec47400640 1 --2- 172.21.15.96:0/1620537396 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fec34005290 0x7fec3400b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:35.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.077+0000 7fec47400640 1 -- 172.21.15.96:0/1620537396 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fec340039e0 msgr2=0x7fec34003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:35.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.077+0000 7fec47400640 1 --2- 172.21.15.96:0/1620537396 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fec340039e0 0x7fec34003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:35.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.077+0000 7fec47400640 1 -- 172.21.15.96:0/1620537396 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec34003520 con 0x7fec34004800 2024-05-28T06:14:35.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.078+0000 7fec47400640 1 --2- 172.21.15.96:0/1620537396 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fec34004800 0x7fec34004c80 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7fec38001750 tx=0x7fec34005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b34f6fb57b47189 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:35.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.078+0000 7fec4f400640 1 -- 172.21.15.96:0/1620537396 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec38001370 con 0x7fec34004800 2024-05-28T06:14:35.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.078+0000 7fec4f400640 1 -- 172.21.15.96:0/1620537396 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec380052d0 con 0x7fec34004800 2024-05-28T06:14:35.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.078+0000 7fec4f400640 1 -- 172.21.15.96:0/1620537396 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec3802abe0 con 0x7fec34004800 2024-05-28T06:14:35.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.079+0000 7fec56a00640 1 -- 172.21.15.96:0/1620537396 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fec34004800 msgr2=0x7fec34004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:35.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.079+0000 7fec56a00640 1 --2- 172.21.15.96:0/1620537396 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fec34004800 0x7fec34004c80 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7fec38001750 tx=0x7fec34005830 comp rx=0 tx=0).stop 2024-05-28T06:14:35.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.079+0000 7fec54a00640 1 -- 172.21.15.96:0/1620537396 reap_dead start 2024-05-28T06:14:35.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.079+0000 7fec56a00640 1 -- 172.21.15.96:0/1620537396 shutdown_connections 2024-05-28T06:14:35.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.079+0000 7fec56a00640 1 -- 172.21.15.96:0/1620537396 >> 172.21.15.96:0/1620537396 conn(0x7fec5008bfc0 msgr2=0x7fec500d4650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:35.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.079+0000 7fec56a00640 1 -- 172.21.15.96:0/1620537396 shutdown_connections 2024-05-28T06:14:35.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.079+0000 7fec56a00640 1 -- 172.21.15.96:0/1620537396 wait complete. 2024-05-28T06:14:35.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.081+0000 7fec56a00640 1 Processor -- start 2024-05-28T06:14:35.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.081+0000 7fec56a00640 1 -- start start 2024-05-28T06:14:35.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.082+0000 7fec56a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fec34004800 0x7fec50155720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:35.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.082+0000 7fec56a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fec500ca790 0x7fec500cc770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:35.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.082+0000 7fec56a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fec500cccb0 0x7fec500cd160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:35.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.082+0000 7fec56a00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fec500d04a0 con 0x7fec34004800 2024-05-28T06:14:35.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.082+0000 7fec56a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fec50162fe0 con 0x7fec500ca790 2024-05-28T06:14:35.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.082+0000 7fec56a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fec500d0620 con 0x7fec500cccb0 2024-05-28T06:14:35.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.082+0000 7fec4fe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fec34004800 0x7fec50155720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:35.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.082+0000 7fec4fe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fec34004800 0x7fec50155720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:43310/0 (socket says 172.21.15.96:43310) 2024-05-28T06:14:35.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.082+0000 7fec4fe00640 1 -- 172.21.15.96:0/978508538 learned_addr learned my addr 172.21.15.96:0/978508538 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:35.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.082+0000 7fec4fe00640 1 -- 172.21.15.96:0/978508538 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fec500cccb0 msgr2=0x7fec500cd160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:35.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.082+0000 7fec4fe00640 1 --2- 172.21.15.96:0/978508538 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fec500cccb0 0x7fec500cd160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:35.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.082+0000 7fec4fe00640 1 -- 172.21.15.96:0/978508538 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fec500ca790 msgr2=0x7fec500cc770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:35.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.082+0000 7fec4fe00640 1 --2- 172.21.15.96:0/978508538 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fec500ca790 0x7fec500cc770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:35.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.082+0000 7fec4fe00640 1 -- 172.21.15.96:0/978508538 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec500b7e90 con 0x7fec34004800 2024-05-28T06:14:35.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.083+0000 7fec4fe00640 1 --2- 172.21.15.96:0/978508538 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fec34004800 0x7fec50155720 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7fec4000e800 tx=0x7fec4000ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:35.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.083+0000 7fec4cc00640 1 -- 172.21.15.96:0/978508538 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec4000c870 con 0x7fec34004800 2024-05-28T06:14:35.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.083+0000 7fec4cc00640 1 -- 172.21.15.96:0/978508538 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec40018070 con 0x7fec34004800 2024-05-28T06:14:35.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.083+0000 7fec56a00640 1 -- 172.21.15.96:0/978508538 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec501561c0 con 0x7fec34004800 2024-05-28T06:14:35.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.083+0000 7fec4cc00640 1 -- 172.21.15.96:0/978508538 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec4000cc70 con 0x7fec34004800 2024-05-28T06:14:35.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.083+0000 7fec56a00640 1 -- 172.21.15.96:0/978508538 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec500cac70 con 0x7fec34004800 2024-05-28T06:14:35.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.084+0000 7fec56a00640 1 -- 172.21.15.96:0/978508538 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec5015d190 con 0x7fec34004800 2024-05-28T06:14:35.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.085+0000 7fec4cc00640 1 -- 172.21.15.96:0/978508538 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 22) ==== 92213+0+0 (secure 0 0 0) 0x7fec4001d460 con 0x7fec34004800 2024-05-28T06:14:35.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.086+0000 7fec4cc00640 1 --2- 172.21.15.96:0/978508538 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fec2c06d610 0x7fec2c06fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:35.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.087+0000 7fec4cc00640 1 -- 172.21.15.96:0/978508538 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5765+0+0 (secure 0 0 0) 0x7fec4008a720 con 0x7fec34004800 2024-05-28T06:14:35.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.087+0000 7fec47400640 1 --2- 172.21.15.96:0/978508538 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fec2c06d610 0x7fec2c06fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:35.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.088+0000 7fec47400640 1 --2- 172.21.15.96:0/978508538 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fec2c06d610 0x7fec2c06fad0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fec38032000 tx=0x7fec3802c000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:35.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.092+0000 7fec4cc00640 1 -- 172.21.15.96:0/978508538 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fec5015d190 con 0x7fec34004800 2024-05-28T06:14:35.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.331+0000 7fec56a00640 1 -- 172.21.15.96:0/978508538 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fec500d0bf0 con 0x7fec34004800 2024-05-28T06:14:35.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.333+0000 7fec4cc00640 1 -- 172.21.15.96:0/978508538 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+5184 (secure 0 0 0) 0x7fec40059cc0 con 0x7fec34004800 2024-05-28T06:14:35.333 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:14:35.334 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":35,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5015,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/1869890774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":1869890774},{"type":"v1","addr":"172.21.15.123:6837","nonce":1869890774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":5386,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/873923319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":873923319},{"type":"v1","addr":"172.21.15.123:6835","nonce":873923319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":6486,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6834/1041571401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":1041571401},{"type":"v1","addr":"172.21.15.96:6834","nonce":1041571401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-28T06:12:28.268409+0000","modified":"2024-05-28T06:12:31.363489+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6489},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6489":{"gid":6489,"name":"a","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.96:6835/1590650245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6833","nonce":1590650245},{"type":"v1","addr":"172.21.15.96:6835","nonce":1590650245}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6489,"qdb_cluster":[6489]},"id":4}]} 2024-05-28T06:14:35.334 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 35 2024-05-28T06:14:35.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.335+0000 7fec56a00640 1 -- 172.21.15.96:0/978508538 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fec2c06d610 msgr2=0x7fec2c06fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:35.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.335+0000 7fec56a00640 1 --2- 172.21.15.96:0/978508538 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fec2c06d610 0x7fec2c06fad0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fec38032000 tx=0x7fec3802c000 comp rx=0 tx=0).stop 2024-05-28T06:14:35.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.335+0000 7fec56a00640 1 -- 172.21.15.96:0/978508538 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fec34004800 msgr2=0x7fec50155720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:35.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.335+0000 7fec56a00640 1 --2- 172.21.15.96:0/978508538 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fec34004800 0x7fec50155720 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7fec4000e800 tx=0x7fec4000ecc0 comp rx=0 tx=0).stop 2024-05-28T06:14:35.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.335+0000 7fec56a00640 1 -- 172.21.15.96:0/978508538 shutdown_connections 2024-05-28T06:14:35.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.335+0000 7fec56a00640 1 --2- 172.21.15.96:0/978508538 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fec2c06d610 0x7fec2c06fad0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:35.337 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.335+0000 7fec56a00640 1 --2- 172.21.15.96:0/978508538 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fec500cccb0 0x7fec500cd160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:35.337 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.335+0000 7fec56a00640 1 --2- 172.21.15.96:0/978508538 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fec500ca790 0x7fec500cc770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:35.337 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.335+0000 7fec56a00640 1 --2- 172.21.15.96:0/978508538 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fec34004800 0x7fec50155720 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:35.337 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.335+0000 7fec56a00640 1 -- 172.21.15.96:0/978508538 >> 172.21.15.96:0/978508538 conn(0x7fec5008bfc0 msgr2=0x7fec500d3190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:35.337 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.336+0000 7fec56a00640 1 -- 172.21.15.96:0/978508538 shutdown_connections 2024-05-28T06:14:35.337 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.336+0000 7fec56a00640 1 -- 172.21.15.96:0/978508538 wait complete. 2024-05-28T06:14:35.355 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:14:35.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.552+0000 7f0ee1600640 1 Processor -- start 2024-05-28T06:14:35.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.553+0000 7f0ee1600640 1 -- start start 2024-05-28T06:14:35.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.554+0000 7f0ee1600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f0edc0d0a20 0x7f0edc0997a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:35.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.554+0000 7f0ee1600640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0edc0b51f0 con 0x7f0edc0d0590 2024-05-28T06:14:35.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.554+0000 7f0ee1600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0edc0aef60 con 0x7f0edc0b4e10 2024-05-28T06:14:35.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.554+0000 7f0ee1600640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f0edc0ad1d0 con 0x7f0edc0d0a20 2024-05-28T06:14:35.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.554+0000 7f0ed3e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f0edc0d0590 0x7f0edc099090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:44208/0 (socket says 172.21.15.96:44208) 2024-05-28T06:14:35.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.554+0000 7f0ed3e00640 1 -- 172.21.15.96:0/1489901590 learned_addr learned my addr 172.21.15.96:0/1489901590 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:14:35.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.554+0000 7f0edb400640 1 --2- 172.21.15.96:0/1489901590 >> v2:172.21.15.123:3300/0 conn(0x7f0edc0d0a20 0x7f0edc0997a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:35.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.554+0000 7f0eda000640 1 -- 172.21.15.96:0/1489901590 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2585844748 0 0) 0x7f0edc0b51f0 con 0x7f0edc0d0590 2024-05-28T06:14:35.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.554+0000 7f0eda000640 1 -- 172.21.15.96:0/1489901590 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0ec0003520 con 0x7f0edc0d0590 2024-05-28T06:14:35.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.555+0000 7f0eda000640 1 -- 172.21.15.96:0/1489901590 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1641247419 0 0) 0x7f0ec0003520 con 0x7f0edc0d0590 2024-05-28T06:14:35.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.555+0000 7f0eda000640 1 -- 172.21.15.96:0/1489901590 >> v2:172.21.15.123:3300/0 conn(0x7f0edc0d0a20 msgr2=0x7f0edc0997a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:35.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.555+0000 7f0eda000640 1 --2- 172.21.15.96:0/1489901590 >> v2:172.21.15.123:3300/0 conn(0x7f0edc0d0a20 0x7f0edc0997a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:35.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.555+0000 7f0eda000640 1 -- 172.21.15.96:0/1489901590 >> v1:172.21.15.123:6789/0 conn(0x7f0edc0b4e10 legacy=0x7f0edc0cde30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:35.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.555+0000 7f0eda000640 1 -- 172.21.15.96:0/1489901590 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0edc0ad470 con 0x7f0edc0d0590 2024-05-28T06:14:35.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.555+0000 7f0eda000640 1 -- 172.21.15.96:0/1489901590 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f0ec4001670 con 0x7f0edc0d0590 2024-05-28T06:14:35.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.555+0000 7f0eda000640 1 -- 172.21.15.96:0/1489901590 >> v1:172.21.15.96:6789/0 conn(0x7f0edc0d0590 legacy=0x7f0edc099090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:35.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.555+0000 7f0eda000640 1 --2- 172.21.15.96:0/1489901590 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0ec0003a00 0x7f0ec0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:35.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.555+0000 7f0eda000640 1 --2- 172.21.15.96:0/1489901590 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0ec0004800 0x7f0ec0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:35.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.555+0000 7f0eda000640 1 --2- 172.21.15.96:0/1489901590 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ec00052e0 0x7f0ec000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:35.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.555+0000 7f0eda000640 1 -- 172.21.15.96:0/1489901590 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f0edc0ad1d0 con 0x7f0ec0004800 2024-05-28T06:14:35.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.555+0000 7f0eda000640 1 -- 172.21.15.96:0/1489901590 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f0ec000c200 con 0x7f0ec00052e0 2024-05-28T06:14:35.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.555+0000 7f0eda000640 1 -- 172.21.15.96:0/1489901590 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f0ec000c400 con 0x7f0ec0003a00 2024-05-28T06:14:35.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.555+0000 7f0ed3e00640 1 --2- 172.21.15.96:0/1489901590 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0ec0004800 0x7f0ec0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:35.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.556+0000 7f0edb400640 1 --2- 172.21.15.96:0/1489901590 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ec00052e0 0x7f0ec000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:35.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.556+0000 7f0edaa00640 1 --2- 172.21.15.96:0/1489901590 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0ec0003a00 0x7f0ec0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:35.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.556+0000 7f0ed3e00640 1 -- 172.21.15.96:0/1489901590 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0ec0003a00 msgr2=0x7f0ec0003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:35.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.556+0000 7f0ed3e00640 1 --2- 172.21.15.96:0/1489901590 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0ec0003a00 0x7f0ec0003e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:35.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.556+0000 7f0ed3e00640 1 -- 172.21.15.96:0/1489901590 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ec00052e0 msgr2=0x7f0ec000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:35.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.556+0000 7f0ed3e00640 1 --2- 172.21.15.96:0/1489901590 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ec00052e0 0x7f0ec000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:35.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.556+0000 7f0ed3e00640 1 -- 172.21.15.96:0/1489901590 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ec0003520 con 0x7f0ec0004800 2024-05-28T06:14:35.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.556+0000 7f0edb400640 1 --2- 172.21.15.96:0/1489901590 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ec00052e0 0x7f0ec000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:14:35.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.556+0000 7f0edb400640 1 -- 172.21.15.96:0/1489901590 reap_dead start 2024-05-28T06:14:35.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.557+0000 7f0ed3e00640 1 --2- 172.21.15.96:0/1489901590 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0ec0004800 0x7f0ec0004c80 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f0ec400c110 tx=0x7f0ec0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ac78b2d79e8396d server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:35.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.557+0000 7f0eda000640 1 -- 172.21.15.96:0/1489901590 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ec4007c40 con 0x7f0ec0004800 2024-05-28T06:14:35.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.557+0000 7f0eda000640 1 -- 172.21.15.96:0/1489901590 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ec4005af0 con 0x7f0ec0004800 2024-05-28T06:14:35.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.557+0000 7f0eda000640 1 -- 172.21.15.96:0/1489901590 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ec4007070 con 0x7f0ec0004800 2024-05-28T06:14:35.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.558+0000 7f0ee1600640 1 -- 172.21.15.96:0/1489901590 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0ec0004800 msgr2=0x7f0ec0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:35.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.558+0000 7f0ee1600640 1 --2- 172.21.15.96:0/1489901590 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0ec0004800 0x7f0ec0004c80 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f0ec400c110 tx=0x7f0ec0005830 comp rx=0 tx=0).stop 2024-05-28T06:14:35.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.558+0000 7f0ee1600640 1 -- 172.21.15.96:0/1489901590 shutdown_connections 2024-05-28T06:14:35.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.558+0000 7f0ee1600640 1 --2- 172.21.15.96:0/1489901590 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0ec0004800 0x7f0ec0004c80 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:35.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.558+0000 7f0ee1600640 1 -- 172.21.15.96:0/1489901590 >> 172.21.15.96:0/1489901590 conn(0x7f0edc06bae0 msgr2=0x7f0edc06bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:35.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.558+0000 7f0ee1600640 1 -- 172.21.15.96:0/1489901590 shutdown_connections 2024-05-28T06:14:35.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.558+0000 7f0ee1600640 1 -- 172.21.15.96:0/1489901590 wait complete. 2024-05-28T06:14:35.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.560+0000 7f0ee1600640 1 Processor -- start 2024-05-28T06:14:35.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.560+0000 7f0ee1600640 1 -- start start 2024-05-28T06:14:35.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.561+0000 7f0ee1600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0ec0004800 0x7f0edc1593b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:35.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.561+0000 7f0ee1600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0edc0afda0 0x7f0edc0c7330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:35.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.561+0000 7f0ee1600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0edc0c7870 0x7f0edc1e7f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:35.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.561+0000 7f0ee1600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f0edc0ce880 con 0x7f0edc0c7870 2024-05-28T06:14:35.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.561+0000 7f0ee1600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f0edc0d0e30 con 0x7f0edc0afda0 2024-05-28T06:14:35.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.561+0000 7f0ee1600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f0edc0cea00 con 0x7f0ec0004800 2024-05-28T06:14:35.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.561+0000 7f0edb400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0edc0c7870 0x7f0edc1e7f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:35.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.561+0000 7f0edb400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0edc0c7870 0x7f0edc1e7f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:43334/0 (socket says 172.21.15.96:43334) 2024-05-28T06:14:35.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.561+0000 7f0edb400640 1 -- 172.21.15.96:0/895952441 learned_addr learned my addr 172.21.15.96:0/895952441 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:35.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.561+0000 7f0ed3e00640 1 --2- 172.21.15.96:0/895952441 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0edc0afda0 0x7f0edc0c7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:35.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.561+0000 7f0edaa00640 1 --2- 172.21.15.96:0/895952441 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0ec0004800 0x7f0edc1593b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:35.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.561+0000 7f0edb400640 1 -- 172.21.15.96:0/895952441 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0ec0004800 msgr2=0x7f0edc1593b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:35.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.561+0000 7f0edb400640 1 --2- 172.21.15.96:0/895952441 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0ec0004800 0x7f0edc1593b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:35.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.561+0000 7f0edb400640 1 -- 172.21.15.96:0/895952441 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0edc0afda0 msgr2=0x7f0edc0c7330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:35.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.561+0000 7f0edb400640 1 --2- 172.21.15.96:0/895952441 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0edc0afda0 0x7f0edc0c7330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:35.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.561+0000 7f0edb400640 1 -- 172.21.15.96:0/895952441 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0edc0b0310 con 0x7f0edc0c7870 2024-05-28T06:14:35.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.562+0000 7f0edb400640 1 --2- 172.21.15.96:0/895952441 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0edc0c7870 0x7f0edc1e7f10 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f0edc0cfb50 tx=0x7f0ed4009d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:35.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.562+0000 7f0ed2a00640 1 -- 172.21.15.96:0/895952441 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ed4019070 con 0x7f0edc0c7870 2024-05-28T06:14:35.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.562+0000 7f0ed2a00640 1 -- 172.21.15.96:0/895952441 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ed4007c90 con 0x7f0edc0c7870 2024-05-28T06:14:35.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.562+0000 7f0ed2a00640 1 -- 172.21.15.96:0/895952441 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ed4014640 con 0x7f0edc0c7870 2024-05-28T06:14:35.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.562+0000 7f0ee1600640 1 -- 172.21.15.96:0/895952441 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0edc0b05f0 con 0x7f0edc0c7870 2024-05-28T06:14:35.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.562+0000 7f0ee1600640 1 -- 172.21.15.96:0/895952441 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0edc1598f0 con 0x7f0edc0c7870 2024-05-28T06:14:35.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.563+0000 7f0ee1600640 1 -- 172.21.15.96:0/895952441 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0edc0689e0 con 0x7f0edc0c7870 2024-05-28T06:14:35.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.565+0000 7f0ed2a00640 1 -- 172.21.15.96:0/895952441 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 22) ==== 92213+0+0 (secure 0 0 0) 0x7f0ed401c070 con 0x7f0edc0c7870 2024-05-28T06:14:35.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.565+0000 7f0ed2a00640 1 --2- 172.21.15.96:0/895952441 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0eb806d5e0 0x7f0eb806faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:35.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.566+0000 7f0ed2a00640 1 -- 172.21.15.96:0/895952441 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5765+0+0 (secure 0 0 0) 0x7f0ed408a080 con 0x7f0edc0c7870 2024-05-28T06:14:35.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.566+0000 7f0edaa00640 1 --2- 172.21.15.96:0/895952441 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0eb806d5e0 0x7f0eb806faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:35.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.567+0000 7f0edaa00640 1 --2- 172.21.15.96:0/895952441 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0eb806d5e0 0x7f0eb806faa0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc000c00 tx=0x7f0ecc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:35.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.570+0000 7f0ed2a00640 1 -- 172.21.15.96:0/895952441 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0ed4059f00 con 0x7f0edc0c7870 2024-05-28T06:14:35.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.772+0000 7f0ee1600640 1 -- 172.21.15.96:0/895952441 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0edc0cc070 con 0x7f0edc0c7870 2024-05-28T06:14:35.774 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.774+0000 7f0ed2a00640 1 -- 172.21.15.96:0/895952441 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v62) ==== 74+0+16785 (secure 0 0 0) 0x7f0ed4059800 con 0x7f0edc0c7870 2024-05-28T06:14:35.775 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:14:35.775 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":62,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:14:31.398967+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-05-28T06:12:26.024060+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-05-28T06:12:27.254401+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":61,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[{"pool":9,"snaps":[{"begin":2,"length":2}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:14:35.777 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.776+0000 7f0ee1600640 1 -- 172.21.15.96:0/895952441 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0eb806d5e0 msgr2=0x7f0eb806faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:35.777 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.776+0000 7f0ee1600640 1 --2- 172.21.15.96:0/895952441 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0eb806d5e0 0x7f0eb806faa0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc000c00 tx=0x7f0ecc009040 comp rx=0 tx=0).stop 2024-05-28T06:14:35.777 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.776+0000 7f0ee1600640 1 -- 172.21.15.96:0/895952441 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0edc0c7870 msgr2=0x7f0edc1e7f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:35.777 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.776+0000 7f0ee1600640 1 --2- 172.21.15.96:0/895952441 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0edc0c7870 0x7f0edc1e7f10 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f0edc0cfb50 tx=0x7f0ed4009d50 comp rx=0 tx=0).stop 2024-05-28T06:14:35.777 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.776+0000 7f0ee1600640 1 -- 172.21.15.96:0/895952441 shutdown_connections 2024-05-28T06:14:35.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.776+0000 7f0ee1600640 1 --2- 172.21.15.96:0/895952441 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0eb806d5e0 0x7f0eb806faa0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:35.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.776+0000 7f0ee1600640 1 --2- 172.21.15.96:0/895952441 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0edc0c7870 0x7f0edc1e7f10 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:35.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.776+0000 7f0ee1600640 1 --2- 172.21.15.96:0/895952441 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0edc0afda0 0x7f0edc0c7330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:35.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.776+0000 7f0ee1600640 1 --2- 172.21.15.96:0/895952441 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0ec0004800 0x7f0edc1593b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:35.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.776+0000 7f0ee1600640 1 -- 172.21.15.96:0/895952441 >> 172.21.15.96:0/895952441 conn(0x7f0edc06bae0 msgr2=0x7f0edc15a9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:35.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.777+0000 7f0ee1600640 1 -- 172.21.15.96:0/895952441 shutdown_connections 2024-05-28T06:14:35.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.777+0000 7f0ee1600640 1 -- 172.21.15.96:0/895952441 wait complete. 2024-05-28T06:14:35.796 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-05-28T06:14:35.796 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-28T06:14:35.995 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.995+0000 7f7796e00640 1 Processor -- start 2024-05-28T06:14:35.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.995+0000 7f7796e00640 1 -- start start 2024-05-28T06:14:35.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.996+0000 7f7796e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f779015b3a0 0x7f77900b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:35.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.996+0000 7f7796e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f779006dc20 0x7f779015f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:35.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.996+0000 7f7796e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f7790072c00 0x7f7790073050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:35.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.996+0000 7f7796e00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f779006f980 con 0x7f779006dc20 2024-05-28T06:14:35.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.996+0000 7f7796e00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f7790073590 con 0x7f7790072c00 2024-05-28T06:14:35.998 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.996+0000 7f7796e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f77900ae0c0 con 0x7f779015b3a0 2024-05-28T06:14:35.998 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.997+0000 7f7795000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f779006dc20 0x7f779015f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:35.998 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.997+0000 7f7795000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f779006dc20 0x7f779015f9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:43342/0 (socket says 172.21.15.96:43342) 2024-05-28T06:14:35.998 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.997+0000 7f7795000640 1 -- 172.21.15.96:0/2773331082 learned_addr learned my addr 172.21.15.96:0/2773331082 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:35.998 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.997+0000 7f7796400640 1 --2- 172.21.15.96:0/2773331082 >> v2:172.21.15.123:3300/0 conn(0x7f7790072c00 0x7f7790073050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:35.999 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.997+0000 7f7795a00640 1 --2- 172.21.15.96:0/2773331082 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f779015b3a0 0x7f77900b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:35.999 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.997+0000 7f7795000640 1 -- 172.21.15.96:0/2773331082 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f779015b3a0 msgr2=0x7f77900b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:35.999 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.997+0000 7f7795000640 1 --2- 172.21.15.96:0/2773331082 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f779015b3a0 0x7f77900b32c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:35.999 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.997+0000 7f7795000640 1 -- 172.21.15.96:0/2773331082 >> v2:172.21.15.123:3300/0 conn(0x7f7790072c00 msgr2=0x7f7790073050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:35.999 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.997+0000 7f7795000640 1 --2- 172.21.15.96:0/2773331082 >> v2:172.21.15.123:3300/0 conn(0x7f7790072c00 0x7f7790073050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:35.999 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.997+0000 7f7795000640 1 -- 172.21.15.96:0/2773331082 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77900afd60 con 0x7f779006dc20 2024-05-28T06:14:35.999 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.997+0000 7f7795a00640 1 --2- 172.21.15.96:0/2773331082 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f779015b3a0 0x7f77900b32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:14:36.000 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.998+0000 7f7795000640 1 --2- 172.21.15.96:0/2773331082 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f779006dc20 0x7f779015f9a0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f779006fd40 tx=0x7f777c02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fb8233725415fcc server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:36.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.999+0000 7f778fe00640 1 -- 172.21.15.96:0/2773331082 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f777c037070 con 0x7f779006dc20 2024-05-28T06:14:36.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.999+0000 7f778fe00640 1 -- 172.21.15.96:0/2773331082 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f777c032440 con 0x7f779006dc20 2024-05-28T06:14:36.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.999+0000 7f778fe00640 1 -- 172.21.15.96:0/2773331082 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f777c032810 con 0x7f779006dc20 2024-05-28T06:14:36.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.999+0000 7f7796e00640 1 -- 172.21.15.96:0/2773331082 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f779006dc20 msgr2=0x7f779015f9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:36.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:35.999+0000 7f7796e00640 1 --2- 172.21.15.96:0/2773331082 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f779006dc20 0x7f779015f9a0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f779006fd40 tx=0x7f777c02a1e0 comp rx=0 tx=0).stop 2024-05-28T06:14:36.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.000+0000 7f7796e00640 1 -- 172.21.15.96:0/2773331082 shutdown_connections 2024-05-28T06:14:36.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.000+0000 7f7796e00640 1 --2- 172.21.15.96:0/2773331082 >> v2:172.21.15.123:3300/0 conn(0x7f7790072c00 0x7f7790073050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:36.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.000+0000 7f7796e00640 1 --2- 172.21.15.96:0/2773331082 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f779006dc20 0x7f779015f9a0 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:36.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.000+0000 7f7796e00640 1 --2- 172.21.15.96:0/2773331082 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f779015b3a0 0x7f77900b32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:36.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.000+0000 7f7796e00640 1 -- 172.21.15.96:0/2773331082 >> 172.21.15.96:0/2773331082 conn(0x7f77900b1fc0 msgr2=0x7f77900b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:36.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.000+0000 7f7796e00640 1 -- 172.21.15.96:0/2773331082 shutdown_connections 2024-05-28T06:14:36.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.000+0000 7f7796e00640 1 -- 172.21.15.96:0/2773331082 wait complete. 2024-05-28T06:14:36.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.001+0000 7f7796e00640 1 Processor -- start 2024-05-28T06:14:36.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.001+0000 7f7796e00640 1 -- start start 2024-05-28T06:14:36.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.002+0000 7f7796e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f779006dc20 0x7f779013c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:36.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.002+0000 7f7796e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7790072c00 0x7f7790157490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:36.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.002+0000 7f7796e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f779015b3a0 0x7f77901579d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:36.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.002+0000 7f7796e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f77900afa80 con 0x7f779015b3a0 2024-05-28T06:14:36.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.002+0000 7f7796e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f779015fee0 con 0x7f779006dc20 2024-05-28T06:14:36.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.002+0000 7f7796e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f779015b780 con 0x7f7790072c00 2024-05-28T06:14:36.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.003+0000 7f7796400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f779015b3a0 0x7f77901579d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:36.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.003+0000 7f7796400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f779015b3a0 0x7f77901579d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:43346/0 (socket says 172.21.15.96:43346) 2024-05-28T06:14:36.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.003+0000 7f7796400640 1 -- 172.21.15.96:0/3127705933 learned_addr learned my addr 172.21.15.96:0/3127705933 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:36.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.003+0000 7f7795000640 1 --2- 172.21.15.96:0/3127705933 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7790072c00 0x7f7790157490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:36.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.003+0000 7f7795a00640 1 --2- 172.21.15.96:0/3127705933 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f779006dc20 0x7f779013c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:36.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.003+0000 7f7796400640 1 -- 172.21.15.96:0/3127705933 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7790072c00 msgr2=0x7f7790157490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:36.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.003+0000 7f7796400640 1 --2- 172.21.15.96:0/3127705933 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7790072c00 0x7f7790157490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:36.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.003+0000 7f7796400640 1 -- 172.21.15.96:0/3127705933 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f779006dc20 msgr2=0x7f779013c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:36.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.003+0000 7f7796400640 1 --2- 172.21.15.96:0/3127705933 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f779006dc20 0x7f779013c5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:36.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.003+0000 7f7796400640 1 -- 172.21.15.96:0/3127705933 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f779013caf0 con 0x7f779015b3a0 2024-05-28T06:14:36.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.003+0000 7f7796400640 1 --2- 172.21.15.96:0/3127705933 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f779015b3a0 0x7f77901579d0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f7788007d20 tx=0x7f778800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:36.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.004+0000 7f778d600640 1 -- 172.21.15.96:0/3127705933 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7788014420 con 0x7f779015b3a0 2024-05-28T06:14:36.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.004+0000 7f778d600640 1 -- 172.21.15.96:0/3127705933 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7788019070 con 0x7f779015b3a0 2024-05-28T06:14:36.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.004+0000 7f7796e00640 1 -- 172.21.15.96:0/3127705933 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f779015a980 con 0x7f779015b3a0 2024-05-28T06:14:36.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.004+0000 7f778d600640 1 -- 172.21.15.96:0/3127705933 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7788014820 con 0x7f779015b3a0 2024-05-28T06:14:36.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.004+0000 7f7796e00640 1 -- 172.21.15.96:0/3127705933 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77900cd1d0 con 0x7f779015b3a0 2024-05-28T06:14:36.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.005+0000 7f7796e00640 1 -- 172.21.15.96:0/3127705933 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77900689e0 con 0x7f779015b3a0 2024-05-28T06:14:36.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.006+0000 7f778d600640 1 -- 172.21.15.96:0/3127705933 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 22) ==== 92213+0+0 (secure 0 0 0) 0x7f77880149c0 con 0x7f779015b3a0 2024-05-28T06:14:36.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.007+0000 7f778d600640 1 --2- 172.21.15.96:0/3127705933 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f777006d610 0x7f777006fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:36.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.007+0000 7f778d600640 1 -- 172.21.15.96:0/3127705933 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5765+0+0 (secure 0 0 0) 0x7f778808ae30 con 0x7f779015b3a0 2024-05-28T06:14:36.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.007+0000 7f7795a00640 1 --2- 172.21.15.96:0/3127705933 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f777006d610 0x7f777006fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:36.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.008+0000 7f7795a00640 1 --2- 172.21.15.96:0/3127705933 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f777006d610 0x7f777006fad0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f7784006fd0 tx=0x7f7784008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:36.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.011+0000 7f778d600640 1 -- 172.21.15.96:0/3127705933 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7788011260 con 0x7f779015b3a0 2024-05-28T06:14:36.233 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.233+0000 7f7796e00640 1 -- 172.21.15.96:0/3127705933 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f77900730e0 con 0x7f779015b3a0 2024-05-28T06:14:36.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.234+0000 7f778d600640 1 -- 172.21.15.96:0/3127705933 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+5184 (secure 0 0 0) 0x7f778805acb0 con 0x7f779015b3a0 2024-05-28T06:14:36.235 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:14:36.235 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":35,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5015,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/1869890774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":1869890774},{"type":"v1","addr":"172.21.15.123:6837","nonce":1869890774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":5386,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/873923319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":873923319},{"type":"v1","addr":"172.21.15.123:6835","nonce":873923319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":6486,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6834/1041571401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":1041571401},{"type":"v1","addr":"172.21.15.96:6834","nonce":1041571401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-28T06:12:28.268409+0000","modified":"2024-05-28T06:12:31.363489+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6489},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6489":{"gid":6489,"name":"a","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.96:6835/1590650245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6833","nonce":1590650245},{"type":"v1","addr":"172.21.15.96:6835","nonce":1590650245}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6489,"qdb_cluster":[6489]},"id":4}]} 2024-05-28T06:14:36.235 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 35 2024-05-28T06:14:36.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.236+0000 7f7796e00640 1 -- 172.21.15.96:0/3127705933 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f777006d610 msgr2=0x7f777006fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:36.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.236+0000 7f7796e00640 1 --2- 172.21.15.96:0/3127705933 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f777006d610 0x7f777006fad0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f7784006fd0 tx=0x7f7784008040 comp rx=0 tx=0).stop 2024-05-28T06:14:36.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.236+0000 7f7796e00640 1 -- 172.21.15.96:0/3127705933 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f779015b3a0 msgr2=0x7f77901579d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:36.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.236+0000 7f7796e00640 1 --2- 172.21.15.96:0/3127705933 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f779015b3a0 0x7f77901579d0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f7788007d20 tx=0x7f778800e4b0 comp rx=0 tx=0).stop 2024-05-28T06:14:36.238 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.237+0000 7f7796e00640 1 -- 172.21.15.96:0/3127705933 shutdown_connections 2024-05-28T06:14:36.238 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.237+0000 7f7796e00640 1 --2- 172.21.15.96:0/3127705933 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f777006d610 0x7f777006fad0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:36.238 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.237+0000 7f7796e00640 1 --2- 172.21.15.96:0/3127705933 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f779015b3a0 0x7f77901579d0 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:36.238 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.237+0000 7f7796e00640 1 --2- 172.21.15.96:0/3127705933 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7790072c00 0x7f7790157490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:36.238 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.237+0000 7f7796e00640 1 --2- 172.21.15.96:0/3127705933 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f779006dc20 0x7f779013c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:36.238 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.237+0000 7f7796e00640 1 -- 172.21.15.96:0/3127705933 >> 172.21.15.96:0/3127705933 conn(0x7f77900b1fc0 msgr2=0x7f779006e1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:36.238 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.237+0000 7f7796e00640 1 -- 172.21.15.96:0/3127705933 shutdown_connections 2024-05-28T06:14:36.238 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.237+0000 7f7796e00640 1 -- 172.21.15.96:0/3127705933 wait complete. 2024-05-28T06:14:36.255 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-28T06:14:36.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.445+0000 7f65eba00640 1 Processor -- start 2024-05-28T06:14:36.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.445+0000 7f65eba00640 1 -- start start 2024-05-28T06:14:36.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.446+0000 7f65eba00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f65e40b4e10 0x7f65e40cde30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:36.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.446+0000 7f65eba00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f65e40ce370 0x7f65e40d0590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:36.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.446+0000 7f65eba00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65e40b51f0 con 0x7f65e40d0ad0 2024-05-28T06:14:36.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.446+0000 7f65eba00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f65e40d0f20 con 0x7f65e40ce370 2024-05-28T06:14:36.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.446+0000 7f65eba00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f65e40aef60 con 0x7f65e40b4e10 2024-05-28T06:14:36.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.446+0000 7f65e9a00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f65e40d0ad0 0x7f65e4099090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:44210/0 (socket says 172.21.15.96:44210) 2024-05-28T06:14:36.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.446+0000 7f65e9a00640 1 -- 172.21.15.96:0/2261546014 learned_addr learned my addr 172.21.15.96:0/2261546014 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:14:36.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.447+0000 7f65e3e00640 1 --2- 172.21.15.96:0/2261546014 >> v2:172.21.15.123:3300/0 conn(0x7f65e40ce370 0x7f65e40d0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:36.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.447+0000 7f65e9000640 1 --2- 172.21.15.96:0/2261546014 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f65e40b4e10 0x7f65e40cde30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:36.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.447+0000 7f65e3400640 1 -- 172.21.15.96:0/2261546014 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2524674368 0 0) 0x7f65e40b51f0 con 0x7f65e40d0ad0 2024-05-28T06:14:36.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.447+0000 7f65e3400640 1 -- 172.21.15.96:0/2261546014 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f65c8003520 con 0x7f65e40d0ad0 2024-05-28T06:14:36.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.447+0000 7f65e3400640 1 -- 172.21.15.96:0/2261546014 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1724481364 0 0) 0x7f65c8003520 con 0x7f65e40d0ad0 2024-05-28T06:14:36.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.447+0000 7f65e3400640 1 -- 172.21.15.96:0/2261546014 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f65e40b4e10 msgr2=0x7f65e40cde30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:36.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.447+0000 7f65e3400640 1 --2- 172.21.15.96:0/2261546014 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f65e40b4e10 0x7f65e40cde30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:36.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.447+0000 7f65e3400640 1 -- 172.21.15.96:0/2261546014 >> v2:172.21.15.123:3300/0 conn(0x7f65e40ce370 msgr2=0x7f65e40d0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:36.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.447+0000 7f65e3400640 1 --2- 172.21.15.96:0/2261546014 >> v2:172.21.15.123:3300/0 conn(0x7f65e40ce370 0x7f65e40d0590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:36.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.447+0000 7f65e3400640 1 -- 172.21.15.96:0/2261546014 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65e40ad1d0 con 0x7f65e40d0ad0 2024-05-28T06:14:36.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.447+0000 7f65e3400640 1 -- 172.21.15.96:0/2261546014 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f65dc0019f0 con 0x7f65e40d0ad0 2024-05-28T06:14:36.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.447+0000 7f65e3e00640 1 --2- 172.21.15.96:0/2261546014 >> v2:172.21.15.123:3300/0 conn(0x7f65e40ce370 0x7f65e40d0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-28T06:14:36.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.447+0000 7f65e3400640 1 -- 172.21.15.96:0/2261546014 >> v1:172.21.15.96:6789/0 conn(0x7f65e40d0ad0 legacy=0x7f65e4099090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:36.451 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.447+0000 7f65e3400640 1 --2- 172.21.15.96:0/2261546014 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f65c80039b0 0x7f65c8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:36.451 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.447+0000 7f65e3400640 1 --2- 172.21.15.96:0/2261546014 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f65e40b4e10 0x7f65c8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:36.451 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.447+0000 7f65e3400640 1 --2- 172.21.15.96:0/2261546014 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65c8004d40 0x7f65c80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:36.451 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.447+0000 7f65e3400640 1 -- 172.21.15.96:0/2261546014 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f65e40d0f20 con 0x7f65c80039b0 2024-05-28T06:14:36.451 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.447+0000 7f65e3400640 1 -- 172.21.15.96:0/2261546014 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f65e40aef60 con 0x7f65c8004d40 2024-05-28T06:14:36.451 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.447+0000 7f65e3400640 1 -- 172.21.15.96:0/2261546014 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f65c800baa0 con 0x7f65e40b4e10 2024-05-28T06:14:36.451 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.448+0000 7f65e9000640 1 --2- 172.21.15.96:0/2261546014 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f65c80039b0 0x7f65c8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:36.451 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.448+0000 7f65e3e00640 1 --2- 172.21.15.96:0/2261546014 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f65e40b4e10 0x7f65c8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:36.451 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.448+0000 7f65e9a00640 1 --2- 172.21.15.96:0/2261546014 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65c8004d40 0x7f65c80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:36.451 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.448+0000 7f65e9000640 1 -- 172.21.15.96:0/2261546014 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f65e40b4e10 msgr2=0x7f65c8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:36.451 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.448+0000 7f65e9000640 1 --2- 172.21.15.96:0/2261546014 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f65e40b4e10 0x7f65c8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:36.451 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.448+0000 7f65e9000640 1 -- 172.21.15.96:0/2261546014 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65c8004d40 msgr2=0x7f65c80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:36.452 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.448+0000 7f65e9000640 1 --2- 172.21.15.96:0/2261546014 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65c8004d40 0x7f65c80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:36.452 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.448+0000 7f65e9000640 1 -- 172.21.15.96:0/2261546014 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65c8003520 con 0x7f65c80039b0 2024-05-28T06:14:36.452 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.449+0000 7f65e9000640 1 --2- 172.21.15.96:0/2261546014 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f65c80039b0 0x7f65c8003dd0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f65cc009ac0 tx=0x7f65c8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b3a566659141cc5 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:36.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.450+0000 7f65e3400640 1 -- 172.21.15.96:0/2261546014 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65cc036070 con 0x7f65c80039b0 2024-05-28T06:14:36.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.450+0000 7f65e3400640 1 -- 172.21.15.96:0/2261546014 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65cc031400 con 0x7f65c80039b0 2024-05-28T06:14:36.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.450+0000 7f65e3400640 1 -- 172.21.15.96:0/2261546014 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65cc009690 con 0x7f65c80039b0 2024-05-28T06:14:36.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.450+0000 7f65eba00640 1 -- 172.21.15.96:0/2261546014 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f65c80039b0 msgr2=0x7f65c8003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:36.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.450+0000 7f65eba00640 1 --2- 172.21.15.96:0/2261546014 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f65c80039b0 0x7f65c8003dd0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f65cc009ac0 tx=0x7f65c8005830 comp rx=0 tx=0).stop 2024-05-28T06:14:36.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.450+0000 7f65e9a00640 1 -- 172.21.15.96:0/2261546014 reap_dead start 2024-05-28T06:14:36.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.450+0000 7f65eba00640 1 -- 172.21.15.96:0/2261546014 shutdown_connections 2024-05-28T06:14:36.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.450+0000 7f65eba00640 1 -- 172.21.15.96:0/2261546014 >> 172.21.15.96:0/2261546014 conn(0x7f65e406bae0 msgr2=0x7f65e406bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:36.454 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.450+0000 7f65eba00640 1 -- 172.21.15.96:0/2261546014 shutdown_connections 2024-05-28T06:14:36.454 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.450+0000 7f65eba00640 1 -- 172.21.15.96:0/2261546014 wait complete. 2024-05-28T06:14:36.454 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.452+0000 7f65eba00640 1 Processor -- start 2024-05-28T06:14:36.454 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.452+0000 7f65eba00640 1 -- start start 2024-05-28T06:14:36.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.453+0000 7f65eba00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f65c80039b0 0x7f65e415dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:36.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.453+0000 7f65eba00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65e4099a30 0x7f65e415d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:36.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.453+0000 7f65eba00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f65e40b37b0 0x7f65e40c7eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:36.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.453+0000 7f65eba00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f65e40ce880 con 0x7f65c80039b0 2024-05-28T06:14:36.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.453+0000 7f65eba00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f65e40ad750 con 0x7f65e4099a30 2024-05-28T06:14:36.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.453+0000 7f65eba00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f65e40cea00 con 0x7f65e40b37b0 2024-05-28T06:14:36.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.453+0000 7f65e9000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f65c80039b0 0x7f65e415dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:36.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.453+0000 7f65e9000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f65c80039b0 0x7f65e415dd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:43370/0 (socket says 172.21.15.96:43370) 2024-05-28T06:14:36.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.453+0000 7f65e9000640 1 -- 172.21.15.96:0/2219920837 learned_addr learned my addr 172.21.15.96:0/2219920837 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:36.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.454+0000 7f65e3e00640 1 --2- 172.21.15.96:0/2219920837 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65e4099a30 0x7f65e415d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:36.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.454+0000 7f65e9000640 1 -- 172.21.15.96:0/2219920837 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f65e40b37b0 msgr2=0x7f65e40c7eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:36.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.454+0000 7f65e9a00640 1 --2- 172.21.15.96:0/2219920837 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f65e40b37b0 0x7f65e40c7eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:36.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.454+0000 7f65e9000640 1 --2- 172.21.15.96:0/2219920837 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f65e40b37b0 0x7f65e40c7eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:36.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.454+0000 7f65e9000640 1 -- 172.21.15.96:0/2219920837 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65e4099a30 msgr2=0x7f65e415d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:36.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.454+0000 7f65e9000640 1 --2- 172.21.15.96:0/2219920837 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65e4099a30 0x7f65e415d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:36.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.454+0000 7f65e9000640 1 -- 172.21.15.96:0/2219920837 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65e415cc70 con 0x7f65c80039b0 2024-05-28T06:14:36.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.454+0000 7f65e3e00640 1 --2- 172.21.15.96:0/2219920837 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65e4099a30 0x7f65e415d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:14:36.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.454+0000 7f65e9000640 1 --2- 172.21.15.96:0/2219920837 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f65c80039b0 0x7f65e415dd70 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f65cc009ac0 tx=0x7f65cc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:36.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.454+0000 7f65e0c00640 1 -- 172.21.15.96:0/2219920837 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65cc040020 con 0x7f65c80039b0 2024-05-28T06:14:36.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.454+0000 7f65e0c00640 1 -- 172.21.15.96:0/2219920837 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65cc003bc0 con 0x7f65c80039b0 2024-05-28T06:14:36.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.454+0000 7f65eba00640 1 -- 172.21.15.96:0/2219920837 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65e415e350 con 0x7f65c80039b0 2024-05-28T06:14:36.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.455+0000 7f65e0c00640 1 -- 172.21.15.96:0/2219920837 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65cc036040 con 0x7f65c80039b0 2024-05-28T06:14:36.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.455+0000 7f65eba00640 1 -- 172.21.15.96:0/2219920837 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65e415e630 con 0x7f65c80039b0 2024-05-28T06:14:36.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.456+0000 7f65eba00640 1 -- 172.21.15.96:0/2219920837 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65e40689e0 con 0x7f65c80039b0 2024-05-28T06:14:36.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.457+0000 7f65e0c00640 1 -- 172.21.15.96:0/2219920837 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 22) ==== 92213+0+0 (secure 0 0 0) 0x7f65cc043020 con 0x7f65c80039b0 2024-05-28T06:14:36.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.458+0000 7f65e0c00640 1 --2- 172.21.15.96:0/2219920837 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f65c006d5e0 0x7f65c006faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:36.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.458+0000 7f65e0c00640 1 -- 172.21.15.96:0/2219920837 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5765+0+0 (secure 0 0 0) 0x7f65cc0aa9c0 con 0x7f65c80039b0 2024-05-28T06:14:36.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.458+0000 7f65e3e00640 1 --2- 172.21.15.96:0/2219920837 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f65c006d5e0 0x7f65c006faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:36.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.459+0000 7f65e3e00640 1 --2- 172.21.15.96:0/2219920837 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f65c006d5e0 0x7f65c006faa0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f65e415b5d0 tx=0x7f65d4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:36.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.462+0000 7f65e0c00640 1 -- 172.21.15.96:0/2219920837 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f65cc04a030 con 0x7f65c80039b0 2024-05-28T06:14:36.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.705+0000 7f65eba00640 1 -- 172.21.15.96:0/2219920837 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f65e40cb360 con 0x7f65c80039b0 2024-05-28T06:14:36.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.706+0000 7f65e0c00640 1 -- 172.21.15.96:0/2219920837 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+5184 (secure 0 0 0) 0x7f65cc0044e0 con 0x7f65c80039b0 2024-05-28T06:14:36.707 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:14:36.707 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":35,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5015,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/1869890774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":1869890774},{"type":"v1","addr":"172.21.15.123:6837","nonce":1869890774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":5386,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/873923319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":873923319},{"type":"v1","addr":"172.21.15.123:6835","nonce":873923319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":6486,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6834/1041571401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":1041571401},{"type":"v1","addr":"172.21.15.96:6834","nonce":1041571401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-28T06:12:28.268409+0000","modified":"2024-05-28T06:12:31.363489+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6489},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6489":{"gid":6489,"name":"a","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.96:6835/1590650245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6833","nonce":1590650245},{"type":"v1","addr":"172.21.15.96:6835","nonce":1590650245}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6489,"qdb_cluster":[6489]},"id":4}]} 2024-05-28T06:14:36.707 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 35 2024-05-28T06:14:36.710 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.709+0000 7f65eba00640 1 -- 172.21.15.96:0/2219920837 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f65c006d5e0 msgr2=0x7f65c006faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:36.710 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.709+0000 7f65eba00640 1 --2- 172.21.15.96:0/2219920837 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f65c006d5e0 0x7f65c006faa0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f65e415b5d0 tx=0x7f65d4008040 comp rx=0 tx=0).stop 2024-05-28T06:14:36.710 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.709+0000 7f65eba00640 1 -- 172.21.15.96:0/2219920837 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f65c80039b0 msgr2=0x7f65e415dd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:36.710 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.709+0000 7f65eba00640 1 --2- 172.21.15.96:0/2219920837 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f65c80039b0 0x7f65e415dd70 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f65cc009ac0 tx=0x7f65cc004730 comp rx=0 tx=0).stop 2024-05-28T06:14:36.710 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.709+0000 7f65eba00640 1 -- 172.21.15.96:0/2219920837 shutdown_connections 2024-05-28T06:14:36.710 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.709+0000 7f65eba00640 1 --2- 172.21.15.96:0/2219920837 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f65c006d5e0 0x7f65c006faa0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:36.710 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.709+0000 7f65eba00640 1 --2- 172.21.15.96:0/2219920837 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f65e40b37b0 0x7f65e40c7eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:36.710 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.709+0000 7f65eba00640 1 --2- 172.21.15.96:0/2219920837 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65e4099a30 0x7f65e415d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:36.710 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.709+0000 7f65eba00640 1 --2- 172.21.15.96:0/2219920837 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f65c80039b0 0x7f65e415dd70 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:36.710 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.709+0000 7f65eba00640 1 -- 172.21.15.96:0/2219920837 >> 172.21.15.96:0/2219920837 conn(0x7f65e406bae0 msgr2=0x7f65e40cbb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:36.710 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.709+0000 7f65eba00640 1 -- 172.21.15.96:0/2219920837 shutdown_connections 2024-05-28T06:14:36.710 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.709+0000 7f65eba00640 1 -- 172.21.15.96:0/2219920837 wait complete. 2024-05-28T06:14:36.730 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:14:36.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.923+0000 7fc743e00640 1 Processor -- start 2024-05-28T06:14:36.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.923+0000 7fc743e00640 1 -- start start 2024-05-28T06:14:36.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.924+0000 7fc743e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fc73c0d0590 0x7fc73c0d0a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:36.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.924+0000 7fc743e00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc73c0b51f0 con 0x7fc73c099090 2024-05-28T06:14:36.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.924+0000 7fc743e00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc73c0aef60 con 0x7fc73c0b4e10 2024-05-28T06:14:36.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.924+0000 7fc743e00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fc73c0ad1d0 con 0x7fc73c0d0590 2024-05-28T06:14:36.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.924+0000 7fc740a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fc73c0d0590 0x7fc73c0d0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:36.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.924+0000 7fc741e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fc73c099090 0x7fc73c0994a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:44214/0 (socket says 172.21.15.96:44214) 2024-05-28T06:14:36.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.924+0000 7fc741e00640 1 -- 172.21.15.96:0/3596742750 learned_addr learned my addr 172.21.15.96:0/3596742750 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:14:36.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.924+0000 7fc740a00640 1 -- 172.21.15.96:0/3596742750 >> v1:172.21.15.123:6789/0 conn(0x7fc73c0b4e10 legacy=0x7fc73c0cde30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:36.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.925+0000 7fc740a00640 1 -- 172.21.15.96:0/3596742750 >> v1:172.21.15.96:6789/0 conn(0x7fc73c099090 legacy=0x7fc73c0994a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:36.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.925+0000 7fc740a00640 1 -- 172.21.15.96:0/3596742750 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc73c0ad470 con 0x7fc73c0d0590 2024-05-28T06:14:36.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.925+0000 7fc73be00640 1 -- 172.21.15.96:0/3596742750 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4133098771 0 0) 0x7fc73c0b51f0 con 0x7fc73c099090 2024-05-28T06:14:36.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.926+0000 7fc740a00640 1 --2- 172.21.15.96:0/3596742750 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc73c0d0590 0x7fc73c0d0a20 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fc7280097a0 tx=0x7fc72802a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de34cf5741612a36 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:36.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.926+0000 7fc73be00640 1 -- 172.21.15.96:0/3596742750 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc728037070 con 0x7fc73c0d0590 2024-05-28T06:14:36.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.926+0000 7fc73be00640 1 -- 172.21.15.96:0/3596742750 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc72802ae50 con 0x7fc73c0d0590 2024-05-28T06:14:36.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.926+0000 7fc73be00640 1 -- 172.21.15.96:0/3596742750 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc728032660 con 0x7fc73c0d0590 2024-05-28T06:14:36.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.926+0000 7fc743e00640 1 -- 172.21.15.96:0/3596742750 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc73c0d0590 msgr2=0x7fc73c0d0a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:36.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.926+0000 7fc743e00640 1 --2- 172.21.15.96:0/3596742750 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc73c0d0590 0x7fc73c0d0a20 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fc7280097a0 tx=0x7fc72802a140 comp rx=0 tx=0).stop 2024-05-28T06:14:36.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.926+0000 7fc743e00640 1 -- 172.21.15.96:0/3596742750 shutdown_connections 2024-05-28T06:14:36.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.926+0000 7fc743e00640 1 --2- 172.21.15.96:0/3596742750 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc73c0d0590 0x7fc73c0d0a20 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:36.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.926+0000 7fc743e00640 1 -- 172.21.15.96:0/3596742750 >> 172.21.15.96:0/3596742750 conn(0x7fc73c06bae0 msgr2=0x7fc73c06bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:36.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.926+0000 7fc743e00640 1 -- 172.21.15.96:0/3596742750 shutdown_connections 2024-05-28T06:14:36.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.926+0000 7fc743e00640 1 -- 172.21.15.96:0/3596742750 wait complete. 2024-05-28T06:14:36.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.928+0000 7fc743e00640 1 Processor -- start 2024-05-28T06:14:36.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.928+0000 7fc743e00640 1 -- start start 2024-05-28T06:14:36.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.929+0000 7fc743e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc73c099090 0x7fc73c155ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:36.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.929+0000 7fc743e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc73c0b4e10 0x7fc73c0c91c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:36.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.929+0000 7fc743e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc73c0d0590 0x7fc73c0c9700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:36.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.929+0000 7fc743e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc73c0ce880 con 0x7fc73c0d0590 2024-05-28T06:14:36.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.929+0000 7fc743e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc73c0ce540 con 0x7fc73c0b4e10 2024-05-28T06:14:36.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.929+0000 7fc743e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fc73c0cea00 con 0x7fc73c099090 2024-05-28T06:14:36.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.929+0000 7fc741e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc73c0d0590 0x7fc73c0c9700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:36.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.930+0000 7fc741e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc73c0d0590 0x7fc73c0c9700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:43388/0 (socket says 172.21.15.96:43388) 2024-05-28T06:14:36.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.930+0000 7fc741e00640 1 -- 172.21.15.96:0/2309559840 learned_addr learned my addr 172.21.15.96:0/2309559840 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:36.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.930+0000 7fc741e00640 1 -- 172.21.15.96:0/2309559840 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc73c099090 msgr2=0x7fc73c155ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:36.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.930+0000 7fc741e00640 1 --2- 172.21.15.96:0/2309559840 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc73c099090 0x7fc73c155ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:36.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.930+0000 7fc741e00640 1 -- 172.21.15.96:0/2309559840 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc73c0b4e10 msgr2=0x7fc73c0c91c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:36.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.930+0000 7fc741e00640 1 --2- 172.21.15.96:0/2309559840 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc73c0b4e10 0x7fc73c0c91c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:36.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.930+0000 7fc741e00640 1 -- 172.21.15.96:0/2309559840 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc73c156020 con 0x7fc73c0d0590 2024-05-28T06:14:36.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.930+0000 7fc741e00640 1 --2- 172.21.15.96:0/2309559840 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc73c0d0590 0x7fc73c0c9700 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7fc73400c7a0 tx=0x7fc73400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:36.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.930+0000 7fc739600640 1 -- 172.21.15.96:0/2309559840 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc734018070 con 0x7fc73c0d0590 2024-05-28T06:14:36.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.931+0000 7fc739600640 1 -- 172.21.15.96:0/2309559840 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc73400a640 con 0x7fc73c0d0590 2024-05-28T06:14:36.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.931+0000 7fc743e00640 1 -- 172.21.15.96:0/2309559840 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc73c1f8e80 con 0x7fc73c0d0590 2024-05-28T06:14:36.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.931+0000 7fc739600640 1 -- 172.21.15.96:0/2309559840 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc73400aab0 con 0x7fc73c0d0590 2024-05-28T06:14:36.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.931+0000 7fc743e00640 1 -- 172.21.15.96:0/2309559840 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc73c1f80c0 con 0x7fc73c0d0590 2024-05-28T06:14:36.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.932+0000 7fc743e00640 1 -- 172.21.15.96:0/2309559840 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc73c0689e0 con 0x7fc73c0d0590 2024-05-28T06:14:36.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.933+0000 7fc739600640 1 -- 172.21.15.96:0/2309559840 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 22) ==== 92213+0+0 (secure 0 0 0) 0x7fc73401b070 con 0x7fc73c0d0590 2024-05-28T06:14:36.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.933+0000 7fc739600640 1 --2- 172.21.15.96:0/2309559840 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc71c06d610 0x7fc71c06fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:36.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.934+0000 7fc739600640 1 -- 172.21.15.96:0/2309559840 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5765+0+0 (secure 0 0 0) 0x7fc734089b70 con 0x7fc73c0d0590 2024-05-28T06:14:36.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.934+0000 7fc741400640 1 --2- 172.21.15.96:0/2309559840 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc71c06d610 0x7fc71c06fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:36.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.935+0000 7fc741400640 1 --2- 172.21.15.96:0/2309559840 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc71c06d610 0x7fc71c06fad0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fc730001560 tx=0x7fc730007990 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:36.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:36.938+0000 7fc739600640 1 -- 172.21.15.96:0/2309559840 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc7340599f0 con 0x7fc73c0d0590 2024-05-28T06:14:37.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.124+0000 7fc743e00640 1 -- 172.21.15.96:0/2309559840 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc73c156760 con 0x7fc73c0d0590 2024-05-28T06:14:37.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.126+0000 7fc739600640 1 -- 172.21.15.96:0/2309559840 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v62) ==== 74+0+16785 (secure 0 0 0) 0x7fc7340592f0 con 0x7fc73c0d0590 2024-05-28T06:14:37.128 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:14:37.128 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":62,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:14:31.398967+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-05-28T06:12:26.024060+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-05-28T06:12:27.254401+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":61,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[{"pool":9,"snaps":[{"begin":2,"length":2}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:14:37.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.129+0000 7fc743e00640 1 -- 172.21.15.96:0/2309559840 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc71c06d610 msgr2=0x7fc71c06fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:37.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.129+0000 7fc743e00640 1 --2- 172.21.15.96:0/2309559840 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc71c06d610 0x7fc71c06fad0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fc730001560 tx=0x7fc730007990 comp rx=0 tx=0).stop 2024-05-28T06:14:37.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.129+0000 7fc743e00640 1 -- 172.21.15.96:0/2309559840 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc73c0d0590 msgr2=0x7fc73c0c9700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:37.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.129+0000 7fc743e00640 1 --2- 172.21.15.96:0/2309559840 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc73c0d0590 0x7fc73c0c9700 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7fc73400c7a0 tx=0x7fc73400cc60 comp rx=0 tx=0).stop 2024-05-28T06:14:37.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.129+0000 7fc743e00640 1 -- 172.21.15.96:0/2309559840 shutdown_connections 2024-05-28T06:14:37.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.129+0000 7fc743e00640 1 --2- 172.21.15.96:0/2309559840 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc71c06d610 0x7fc71c06fad0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:37.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.129+0000 7fc743e00640 1 --2- 172.21.15.96:0/2309559840 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc73c0d0590 0x7fc73c0c9700 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:37.131 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.129+0000 7fc743e00640 1 --2- 172.21.15.96:0/2309559840 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc73c0b4e10 0x7fc73c0c91c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:37.131 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.129+0000 7fc743e00640 1 --2- 172.21.15.96:0/2309559840 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc73c099090 0x7fc73c155ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:37.131 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.129+0000 7fc743e00640 1 -- 172.21.15.96:0/2309559840 >> 172.21.15.96:0/2309559840 conn(0x7fc73c06bae0 msgr2=0x7fc73c0cc260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:37.131 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.129+0000 7fc743e00640 1 -- 172.21.15.96:0/2309559840 shutdown_connections 2024-05-28T06:14:37.131 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.129+0000 7fc743e00640 1 -- 172.21.15.96:0/2309559840 wait complete. 2024-05-28T06:14:37.150 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-05-28T06:14:37.354 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.354+0000 7ff8aaa00640 1 Processor -- start 2024-05-28T06:14:37.355 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.354+0000 7ff8aaa00640 1 -- start start 2024-05-28T06:14:37.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.355+0000 7ff8aaa00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7ff8a40d0590 0x7ff8a40d0a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:37.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.355+0000 7ff8aaa00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff8a4099090 0x7ff8a40994e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:37.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.355+0000 7ff8aaa00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8a40b51f0 con 0x7ff8a40b4e10 2024-05-28T06:14:37.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.355+0000 7ff8aaa00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7ff8a40aef60 con 0x7ff8a40d0590 2024-05-28T06:14:37.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.355+0000 7ff8aaa00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ff8a40ad1d0 con 0x7ff8a4099090 2024-05-28T06:14:37.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.355+0000 7ff8a3e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7ff8a40b4e10 0x7ff8a40cde30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:44224/0 (socket says 172.21.15.96:44224) 2024-05-28T06:14:37.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.355+0000 7ff8a3e00640 1 -- 172.21.15.96:0/1932415963 learned_addr learned my addr 172.21.15.96:0/1932415963 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:14:37.358 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.356+0000 7ff8a2a00640 1 -- 172.21.15.96:0/1932415963 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 716198230 0 0) 0x7ff8a40b51f0 con 0x7ff8a40b4e10 2024-05-28T06:14:37.358 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.356+0000 7ff8a8a00640 1 --2- 172.21.15.96:0/1932415963 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff8a4099090 0x7ff8a40994e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:37.358 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.356+0000 7ff8a2a00640 1 -- 172.21.15.96:0/1932415963 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff888003520 con 0x7ff8a40b4e10 2024-05-28T06:14:37.358 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.356+0000 7ff8a3400640 1 --2- 172.21.15.96:0/1932415963 >> v2:172.21.15.123:3300/0 conn(0x7ff8a40d0590 0x7ff8a40d0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:37.358 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.356+0000 7ff8a2a00640 1 -- 172.21.15.96:0/1932415963 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2702491590 0 0) 0x7ff888003520 con 0x7ff8a40b4e10 2024-05-28T06:14:37.358 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.356+0000 7ff8a2a00640 1 -- 172.21.15.96:0/1932415963 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff8a4099090 msgr2=0x7ff8a40994e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:37.358 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.356+0000 7ff8a2a00640 1 --2- 172.21.15.96:0/1932415963 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff8a4099090 0x7ff8a40994e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:37.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.356+0000 7ff8a2a00640 1 -- 172.21.15.96:0/1932415963 >> v2:172.21.15.123:3300/0 conn(0x7ff8a40d0590 msgr2=0x7ff8a40d0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:37.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.356+0000 7ff8a2a00640 1 --2- 172.21.15.96:0/1932415963 >> v2:172.21.15.123:3300/0 conn(0x7ff8a40d0590 0x7ff8a40d0a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:37.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.356+0000 7ff8a2a00640 1 -- 172.21.15.96:0/1932415963 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8a40ad470 con 0x7ff8a40b4e10 2024-05-28T06:14:37.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.356+0000 7ff8a2a00640 1 -- 172.21.15.96:0/1932415963 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7ff894001670 con 0x7ff8a40b4e10 2024-05-28T06:14:37.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.356+0000 7ff8a2a00640 1 -- 172.21.15.96:0/1932415963 >> v1:172.21.15.96:6789/0 conn(0x7ff8a40b4e10 legacy=0x7ff8a40cde30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:37.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.356+0000 7ff8a2a00640 1 --2- 172.21.15.96:0/1932415963 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff8880039b0 0x7ff888003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:37.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.356+0000 7ff8a2a00640 1 --2- 172.21.15.96:0/1932415963 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff888004800 0x7ff888004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:37.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.356+0000 7ff8a2a00640 1 --2- 172.21.15.96:0/1932415963 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8880051c0 0x7ff88800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:37.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.357+0000 7ff8a2a00640 1 -- 172.21.15.96:0/1932415963 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff8a40aef60 con 0x7ff8880039b0 2024-05-28T06:14:37.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.357+0000 7ff8a2a00640 1 -- 172.21.15.96:0/1932415963 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff8a40ad1d0 con 0x7ff8880051c0 2024-05-28T06:14:37.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.357+0000 7ff8a2a00640 1 -- 172.21.15.96:0/1932415963 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ff88800e2a0 con 0x7ff888004800 2024-05-28T06:14:37.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.357+0000 7ff8a8a00640 1 --2- 172.21.15.96:0/1932415963 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff8a4099090 0x7ff8a40994e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-28T06:14:37.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.357+0000 7ff8a3e00640 1 --2- 172.21.15.96:0/1932415963 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff8880039b0 0x7ff888003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:37.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.357+0000 7ff8a3e00640 1 -- 172.21.15.96:0/1932415963 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff888004800 msgr2=0x7ff888004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:37.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.357+0000 7ff8a3e00640 1 --2- 172.21.15.96:0/1932415963 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff888004800 0x7ff888004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:37.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.357+0000 7ff8a3e00640 1 -- 172.21.15.96:0/1932415963 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8880051c0 msgr2=0x7ff88800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:37.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.357+0000 7ff8a3e00640 1 --2- 172.21.15.96:0/1932415963 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8880051c0 0x7ff88800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:37.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.357+0000 7ff8a3e00640 1 -- 172.21.15.96:0/1932415963 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff888003520 con 0x7ff8880039b0 2024-05-28T06:14:37.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.358+0000 7ff8a3e00640 1 --2- 172.21.15.96:0/1932415963 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff8880039b0 0x7ff888003dd0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7ff89400c290 tx=0x7ff888005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=336ba58df2d63d17 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:37.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.358+0000 7ff8a2a00640 1 -- 172.21.15.96:0/1932415963 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff894007d50 con 0x7ff8880039b0 2024-05-28T06:14:37.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.358+0000 7ff8a2a00640 1 -- 172.21.15.96:0/1932415963 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff89402e070 con 0x7ff8880039b0 2024-05-28T06:14:37.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.358+0000 7ff8a2a00640 1 -- 172.21.15.96:0/1932415963 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff894005d30 con 0x7ff8880039b0 2024-05-28T06:14:37.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.359+0000 7ff8aaa00640 1 -- 172.21.15.96:0/1932415963 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff8880039b0 msgr2=0x7ff888003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:37.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.359+0000 7ff8aaa00640 1 --2- 172.21.15.96:0/1932415963 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff8880039b0 0x7ff888003dd0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7ff89400c290 tx=0x7ff888005830 comp rx=0 tx=0).stop 2024-05-28T06:14:37.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.359+0000 7ff8a8a00640 1 -- 172.21.15.96:0/1932415963 reap_dead start 2024-05-28T06:14:37.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.359+0000 7ff8aaa00640 1 -- 172.21.15.96:0/1932415963 shutdown_connections 2024-05-28T06:14:37.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.359+0000 7ff8aaa00640 1 -- 172.21.15.96:0/1932415963 >> 172.21.15.96:0/1932415963 conn(0x7ff8a406bae0 msgr2=0x7ff8a406bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:37.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.359+0000 7ff8aaa00640 1 -- 172.21.15.96:0/1932415963 shutdown_connections 2024-05-28T06:14:37.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.359+0000 7ff8aaa00640 1 -- 172.21.15.96:0/1932415963 wait complete. 2024-05-28T06:14:37.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.361+0000 7ff8aaa00640 1 Processor -- start 2024-05-28T06:14:37.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.361+0000 7ff8aaa00640 1 -- start start 2024-05-28T06:14:37.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.362+0000 7ff8aaa00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff8880039b0 0x7ff8a41f70d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:37.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.362+0000 7ff8aaa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff8a40c3d70 0x7ff8a40c41f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:37.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.362+0000 7ff8aaa00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8a41541f0 0x7ff8a41546a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:37.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.362+0000 7ff8aaa00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff8a40ce880 con 0x7ff8a40c3d70 2024-05-28T06:14:37.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.362+0000 7ff8aaa00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff8a40ce540 con 0x7ff8a41541f0 2024-05-28T06:14:37.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.362+0000 7ff8aaa00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ff8a40cea00 con 0x7ff8880039b0 2024-05-28T06:14:37.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.362+0000 7ff8a3400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff8a40c3d70 0x7ff8a40c41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:37.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.362+0000 7ff8a3400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff8a40c3d70 0x7ff8a40c41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:43396/0 (socket says 172.21.15.96:43396) 2024-05-28T06:14:37.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.362+0000 7ff8a3400640 1 -- 172.21.15.96:0/2202849792 learned_addr learned my addr 172.21.15.96:0/2202849792 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:37.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.363+0000 7ff8a3400640 1 -- 172.21.15.96:0/2202849792 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff8880039b0 msgr2=0x7ff8a41f70d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:37.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.363+0000 7ff8a3400640 1 --2- 172.21.15.96:0/2202849792 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff8880039b0 0x7ff8a41f70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:37.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.363+0000 7ff8a8a00640 1 --2- 172.21.15.96:0/2202849792 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8a41541f0 0x7ff8a41546a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:37.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.363+0000 7ff8a3400640 1 -- 172.21.15.96:0/2202849792 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8a41541f0 msgr2=0x7ff8a41546a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:37.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.363+0000 7ff8a3400640 1 --2- 172.21.15.96:0/2202849792 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8a41541f0 0x7ff8a41546a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:37.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.363+0000 7ff8a3400640 1 -- 172.21.15.96:0/2202849792 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8a415bb40 con 0x7ff8a40c3d70 2024-05-28T06:14:37.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.363+0000 7ff8a3400640 1 --2- 172.21.15.96:0/2202849792 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff8a40c3d70 0x7ff8a40c41f0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7ff89000d830 tx=0x7ff89000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:37.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.363+0000 7ff89be00640 1 -- 172.21.15.96:0/2202849792 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff89000b6c0 con 0x7ff8a40c3d70 2024-05-28T06:14:37.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.363+0000 7ff89be00640 1 -- 172.21.15.96:0/2202849792 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff890019070 con 0x7ff8a40c3d70 2024-05-28T06:14:37.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.363+0000 7ff89be00640 1 -- 172.21.15.96:0/2202849792 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff89000baa0 con 0x7ff8a40c3d70 2024-05-28T06:14:37.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.363+0000 7ff8aaa00640 1 -- 172.21.15.96:0/2202849792 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8a4154fb0 con 0x7ff8a40c3d70 2024-05-28T06:14:37.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.363+0000 7ff8aaa00640 1 -- 172.21.15.96:0/2202849792 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8a4152320 con 0x7ff8a40c3d70 2024-05-28T06:14:37.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.365+0000 7ff8aaa00640 1 -- 172.21.15.96:0/2202849792 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8a40689e0 con 0x7ff8a40c3d70 2024-05-28T06:14:37.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.366+0000 7ff89be00640 1 -- 172.21.15.96:0/2202849792 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 22) ==== 92213+0+0 (secure 0 0 0) 0x7ff89001e460 con 0x7ff8a40c3d70 2024-05-28T06:14:37.367 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.366+0000 7ff89be00640 1 --2- 172.21.15.96:0/2202849792 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff88406d610 0x7ff88406fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:37.367 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.367+0000 7ff89be00640 1 -- 172.21.15.96:0/2202849792 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5765+0+0 (secure 0 0 0) 0x7ff89008ad90 con 0x7ff8a40c3d70 2024-05-28T06:14:37.367 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.367+0000 7ff8a3e00640 1 --2- 172.21.15.96:0/2202849792 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff88406d610 0x7ff88406fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:37.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.368+0000 7ff8a3e00640 1 --2- 172.21.15.96:0/2202849792 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff88406d610 0x7ff88406fad0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7ff894033000 tx=0x7ff894007000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:37.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.371+0000 7ff89be00640 1 -- 172.21.15.96:0/2202849792 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff8a415bb40 con 0x7ff8a40c3d70 2024-05-28T06:14:37.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.610+0000 7ff8aaa00640 1 -- 172.21.15.96:0/2202849792 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7ff8a406f540 con 0x7ff8a40c3d70 2024-05-28T06:14:37.611 INFO:tasks.ceph.mon.a.smithi096.stderr:2024-05-28T06:14:37.611+0000 7fb2fca00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-28T06:14:37.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.614+0000 7ff89be00640 1 -- 172.21.15.96:0/2202849792 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v36) ==== 161+0+0 (secure 0 0 0) 0x7ff89005a9e0 con 0x7ff8a40c3d70 2024-05-28T06:14:37.615 INFO:teuthology.orchestra.run.smithi096.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-28T06:14:37.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.617+0000 7ff8aaa00640 1 -- 172.21.15.96:0/2202849792 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff88406d610 msgr2=0x7ff88406fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:37.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.617+0000 7ff8aaa00640 1 --2- 172.21.15.96:0/2202849792 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff88406d610 0x7ff88406fad0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7ff894033000 tx=0x7ff894007000 comp rx=0 tx=0).stop 2024-05-28T06:14:37.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.617+0000 7ff8aaa00640 1 -- 172.21.15.96:0/2202849792 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff8a40c3d70 msgr2=0x7ff8a40c41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:37.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.617+0000 7ff8aaa00640 1 --2- 172.21.15.96:0/2202849792 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff8a40c3d70 0x7ff8a40c41f0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7ff89000d830 tx=0x7ff89000dcf0 comp rx=0 tx=0).stop 2024-05-28T06:14:37.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.617+0000 7ff8aaa00640 1 -- 172.21.15.96:0/2202849792 shutdown_connections 2024-05-28T06:14:37.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.617+0000 7ff8aaa00640 1 --2- 172.21.15.96:0/2202849792 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff88406d610 0x7ff88406fad0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:37.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.617+0000 7ff8aaa00640 1 --2- 172.21.15.96:0/2202849792 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff8a41541f0 0x7ff8a41546a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:37.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.617+0000 7ff8aaa00640 1 --2- 172.21.15.96:0/2202849792 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff8a40c3d70 0x7ff8a40c41f0 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:37.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.617+0000 7ff8aaa00640 1 --2- 172.21.15.96:0/2202849792 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff8880039b0 0x7ff8a41f70d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:37.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.617+0000 7ff8aaa00640 1 -- 172.21.15.96:0/2202849792 >> 172.21.15.96:0/2202849792 conn(0x7ff8a406bae0 msgr2=0x7ff8a40d09a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:37.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.618+0000 7ff8aaa00640 1 -- 172.21.15.96:0/2202849792 shutdown_connections 2024-05-28T06:14:37.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.618+0000 7ff8aaa00640 1 -- 172.21.15.96:0/2202849792 wait complete. 2024-05-28T06:14:37.640 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-05-28T06:14:37.704 INFO:tasks.ceph.mds.a.smithi096.stdout:starting mds.a at 2024-05-28T06:14:37.820 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.819+0000 7f2955c00640 1 Processor -- start 2024-05-28T06:14:37.820 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.819+0000 7f2955c00640 1 -- start start 2024-05-28T06:14:37.821 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.820+0000 7f2955c00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f2950161a90 0x7f295006e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:37.821 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.820+0000 7f2955c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2950161e70 0x7f29500b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:37.821 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.820+0000 7f2955c00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f295006e9b0 0x7f29500ad2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:37.821 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.820+0000 7f2955c00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f29500b3d40 con 0x7f295006e9b0 2024-05-28T06:14:37.821 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.820+0000 7f2955c00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f29500b4020 con 0x7f2950161a90 2024-05-28T06:14:37.821 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.820+0000 7f2955c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f29500b4300 con 0x7f2950161e70 2024-05-28T06:14:37.822 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.821+0000 7f294f400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f295006e9b0 0x7f29500ad2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:37.822 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.821+0000 7f294f400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f295006e9b0 0x7f29500ad2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:43428/0 (socket says 172.21.15.96:43428) 2024-05-28T06:14:37.822 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.821+0000 7f294f400640 1 -- 172.21.15.96:0/4247484649 learned_addr learned my addr 172.21.15.96:0/4247484649 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:37.822 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.821+0000 7f294e000640 1 --2- 172.21.15.96:0/4247484649 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2950161e70 0x7f29500b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:37.822 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.821+0000 7f294ea00640 1 --2- 172.21.15.96:0/4247484649 >> v2:172.21.15.123:3300/0 conn(0x7f2950161a90 0x7f295006e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:37.823 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.821+0000 7f294f400640 1 -- 172.21.15.96:0/4247484649 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2950161e70 msgr2=0x7f29500b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:37.823 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.821+0000 7f294f400640 1 --2- 172.21.15.96:0/4247484649 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2950161e70 0x7f29500b32c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:37.823 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.821+0000 7f294f400640 1 -- 172.21.15.96:0/4247484649 >> v2:172.21.15.123:3300/0 conn(0x7f2950161a90 msgr2=0x7f295006e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:37.823 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.821+0000 7f294f400640 1 --2- 172.21.15.96:0/4247484649 >> v2:172.21.15.123:3300/0 conn(0x7f2950161a90 0x7f295006e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:37.823 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.821+0000 7f294f400640 1 -- 172.21.15.96:0/4247484649 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f295006dc70 con 0x7f295006e9b0 2024-05-28T06:14:37.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.822+0000 7f294f400640 1 --2- 172.21.15.96:0/4247484649 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f295006e9b0 0x7f29500ad2a0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f295006fe90 tx=0x7f294802a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af4894d4cf26425 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:37.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.823+0000 7f294d600640 1 -- 172.21.15.96:0/4247484649 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2948036070 con 0x7f295006e9b0 2024-05-28T06:14:37.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.823+0000 7f294d600640 1 -- 172.21.15.96:0/4247484649 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2948031600 con 0x7f295006e9b0 2024-05-28T06:14:37.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.823+0000 7f294d600640 1 -- 172.21.15.96:0/4247484649 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29480319d0 con 0x7f295006e9b0 2024-05-28T06:14:37.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.823+0000 7f2955c00640 1 -- 172.21.15.96:0/4247484649 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f295006e9b0 msgr2=0x7f29500ad2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:37.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.823+0000 7f2955c00640 1 --2- 172.21.15.96:0/4247484649 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f295006e9b0 0x7f29500ad2a0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f295006fe90 tx=0x7f294802a6a0 comp rx=0 tx=0).stop 2024-05-28T06:14:37.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.823+0000 7f2955c00640 1 -- 172.21.15.96:0/4247484649 shutdown_connections 2024-05-28T06:14:37.825 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.823+0000 7f2955c00640 1 --2- 172.21.15.96:0/4247484649 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f295006e9b0 0x7f29500ad2a0 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:37.825 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.823+0000 7f2955c00640 1 --2- 172.21.15.96:0/4247484649 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2950161e70 0x7f29500b32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:37.825 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.823+0000 7f2955c00640 1 --2- 172.21.15.96:0/4247484649 >> v2:172.21.15.123:3300/0 conn(0x7f2950161a90 0x7f295006e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:37.825 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.823+0000 7f2955c00640 1 -- 172.21.15.96:0/4247484649 >> 172.21.15.96:0/4247484649 conn(0x7f295006bd20 msgr2=0x7f295006c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:37.825 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.823+0000 7f2955c00640 1 -- 172.21.15.96:0/4247484649 shutdown_connections 2024-05-28T06:14:37.825 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.824+0000 7f2955c00640 1 -- 172.21.15.96:0/4247484649 wait complete. 2024-05-28T06:14:37.826 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.825+0000 7f2955c00640 1 Processor -- start 2024-05-28T06:14:37.826 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.826+0000 7f2955c00640 1 -- start start 2024-05-28T06:14:37.828 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.826+0000 7f2955c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f295006e9b0 0x7f295014f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:37.828 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.827+0000 7f2955c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2950161a90 0x7f295014f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:37.828 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.827+0000 7f2955c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2950161e70 0x7f295014fe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:37.828 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.827+0000 7f2955c00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f29500b3800 con 0x7f2950161a90 2024-05-28T06:14:37.828 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.827+0000 7f2955c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f29500b5020 con 0x7f2950161e70 2024-05-28T06:14:37.828 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.827+0000 7f2955c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f29500b3980 con 0x7f295006e9b0 2024-05-28T06:14:37.828 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.827+0000 7f294e000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2950161a90 0x7f295014f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:37.828 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.827+0000 7f294e000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2950161a90 0x7f295014f8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:43434/0 (socket says 172.21.15.96:43434) 2024-05-28T06:14:37.828 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.827+0000 7f294e000640 1 -- 172.21.15.96:0/890011870 learned_addr learned my addr 172.21.15.96:0/890011870 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:37.828 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.827+0000 7f294ea00640 1 --2- 172.21.15.96:0/890011870 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f295006e9b0 0x7f295014f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:37.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.827+0000 7f294e000640 1 -- 172.21.15.96:0/890011870 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f295006e9b0 msgr2=0x7f295014f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:37.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.827+0000 7f294e000640 1 --2- 172.21.15.96:0/890011870 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f295006e9b0 0x7f295014f390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:37.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.827+0000 7f294e000640 1 -- 172.21.15.96:0/890011870 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2950161e70 msgr2=0x7f295014fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:37.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.827+0000 7f294e000640 1 --2- 172.21.15.96:0/890011870 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2950161e70 0x7f295014fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:37.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.827+0000 7f294e000640 1 -- 172.21.15.96:0/890011870 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2950151b60 con 0x7f2950161a90 2024-05-28T06:14:37.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.828+0000 7f294e000640 1 --2- 172.21.15.96:0/890011870 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2950161a90 0x7f295014f8d0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f293c00d7d0 tx=0x7f293c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:37.830 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.828+0000 7f2946a00640 1 -- 172.21.15.96:0/890011870 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f293c00b660 con 0x7f2950161a90 2024-05-28T06:14:37.830 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.828+0000 7f2946a00640 1 -- 172.21.15.96:0/890011870 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f293c019070 con 0x7f2950161a90 2024-05-28T06:14:37.830 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.828+0000 7f2946a00640 1 -- 172.21.15.96:0/890011870 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f293c00ba20 con 0x7f2950161a90 2024-05-28T06:14:37.830 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.828+0000 7f2955c00640 1 -- 172.21.15.96:0/890011870 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2950151e40 con 0x7f2950161a90 2024-05-28T06:14:37.830 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.828+0000 7f2955c00640 1 -- 172.21.15.96:0/890011870 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2950158720 con 0x7f2950161a90 2024-05-28T06:14:37.830 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.829+0000 7f2955c00640 1 -- 172.21.15.96:0/890011870 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2950152360 con 0x7f2950161a90 2024-05-28T06:14:37.830 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.830+0000 7f2946a00640 1 -- 172.21.15.96:0/890011870 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 22) ==== 92213+0+0 (secure 0 0 0) 0x7f293c00bbc0 con 0x7f2950161a90 2024-05-28T06:14:37.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.831+0000 7f2946a00640 1 --2- 172.21.15.96:0/890011870 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f293006d5e0 0x7f293006faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:37.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.831+0000 7f2946a00640 1 -- 172.21.15.96:0/890011870 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 5823+0+0 (secure 0 0 0) 0x7f293c08ac80 con 0x7f2950161a90 2024-05-28T06:14:37.833 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.832+0000 7f294ea00640 1 --2- 172.21.15.96:0/890011870 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f293006d5e0 0x7f293006faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:37.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.834+0000 7f294ea00640 1 --2- 172.21.15.96:0/890011870 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f293006d5e0 0x7f293006faa0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f2940006fd0 tx=0x7f2940008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:37.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:37.836+0000 7f2946a00640 1 -- 172.21.15.96:0/890011870 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2950152360 con 0x7f2950161a90 2024-05-28T06:14:38.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:38.063+0000 7f2955c00640 1 -- 172.21.15.96:0/890011870 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f29500689e0 con 0x7f2950161a90 2024-05-28T06:14:38.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:38.618+0000 7f2946a00640 1 -- 172.21.15.96:0/890011870 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v37) ==== 104+0+0 (secure 0 0 0) 0x7f293c05a870 con 0x7f2950161a90 2024-05-28T06:14:38.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:38.621+0000 7f2955c00640 1 -- 172.21.15.96:0/890011870 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f293006d5e0 msgr2=0x7f293006faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:38.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:38.621+0000 7f2955c00640 1 --2- 172.21.15.96:0/890011870 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f293006d5e0 0x7f293006faa0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f2940006fd0 tx=0x7f2940008040 comp rx=0 tx=0).stop 2024-05-28T06:14:38.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:38.622+0000 7f2955c00640 1 -- 172.21.15.96:0/890011870 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2950161a90 msgr2=0x7f295014f8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:38.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:38.622+0000 7f2955c00640 1 --2- 172.21.15.96:0/890011870 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2950161a90 0x7f295014f8d0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f293c00d7d0 tx=0x7f293c00dc90 comp rx=0 tx=0).stop 2024-05-28T06:14:38.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:38.622+0000 7f2955c00640 1 -- 172.21.15.96:0/890011870 shutdown_connections 2024-05-28T06:14:38.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:38.622+0000 7f2955c00640 1 --2- 172.21.15.96:0/890011870 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f293006d5e0 0x7f293006faa0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:38.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:38.622+0000 7f2955c00640 1 --2- 172.21.15.96:0/890011870 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2950161e70 0x7f295014fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:38.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:38.622+0000 7f2955c00640 1 --2- 172.21.15.96:0/890011870 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2950161a90 0x7f295014f8d0 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:38.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:38.622+0000 7f2955c00640 1 --2- 172.21.15.96:0/890011870 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f295006e9b0 0x7f295014f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:38.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:38.622+0000 7f2955c00640 1 -- 172.21.15.96:0/890011870 >> 172.21.15.96:0/890011870 conn(0x7f295006bd20 msgr2=0x7f29500afe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:38.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:38.622+0000 7f2955c00640 1 -- 172.21.15.96:0/890011870 shutdown_connections 2024-05-28T06:14:38.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:38.622+0000 7f2955c00640 1 -- 172.21.15.96:0/890011870 wait complete. 2024-05-28T06:14:38.647 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-05-28T06:14:38.850 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:38.849+0000 7f4335e00640 1 Processor -- start 2024-05-28T06:14:38.850 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:38.850+0000 7f4335e00640 1 -- start start 2024-05-28T06:14:38.852 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:38.851+0000 7f4335e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f433015fa00 0x7f433006e970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:38.852 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:38.851+0000 7f4335e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f43301600f0 0x7f43300d0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:38.852 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:38.851+0000 7f4335e00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f433006fa00 con 0x7f43300d0970 2024-05-28T06:14:38.852 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:38.851+0000 7f4335e00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f43300d0dc0 con 0x7f433015fa00 2024-05-28T06:14:38.852 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:38.851+0000 7f4335e00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f433006eeb0 con 0x7f43301600f0 2024-05-28T06:14:38.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:38.852+0000 7f4334a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f433015fa00 0x7f433006e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:38.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:38.852+0000 7f4335400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f43300d0970 0x7f43300b3d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:56456/0 (socket says 172.21.15.96:56456) 2024-05-28T06:14:38.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:38.852+0000 7f4335400640 1 -- 172.21.15.96:0/3730393048 learned_addr learned my addr 172.21.15.96:0/3730393048 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:14:38.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:38.852+0000 7f432fe00640 1 --2- 172.21.15.96:0/3730393048 >> v2:172.21.15.123:3300/0 conn(0x7f43301600f0 0x7f43300d0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:38.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:38.852+0000 7f4334a00640 1 -- 172.21.15.96:0/3730393048 >> v2:172.21.15.123:3300/0 conn(0x7f43301600f0 msgr2=0x7f43300d0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:38.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:38.852+0000 7f4334a00640 1 --2- 172.21.15.96:0/3730393048 >> v2:172.21.15.123:3300/0 conn(0x7f43301600f0 0x7f43300d0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:38.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:38.852+0000 7f4334a00640 1 -- 172.21.15.96:0/3730393048 >> v1:172.21.15.96:6789/0 conn(0x7f43300d0970 legacy=0x7f43300b3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:38.854 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:38.852+0000 7f432f400640 1 -- 172.21.15.96:0/3730393048 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1824273656 0 0) 0x7f433006fa00 con 0x7f43300d0970 2024-05-28T06:14:38.854 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:38.852+0000 7f4334a00640 1 -- 172.21.15.96:0/3730393048 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f433006f160 con 0x7f433015fa00 2024-05-28T06:14:38.854 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:38.854+0000 7f4334a00640 1 --2- 172.21.15.96:0/3730393048 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f433015fa00 0x7f433006e970 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f433006d980 tx=0x7f432402a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25de5511b7cd35b3 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:38.855 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:38.854+0000 7f432f400640 1 -- 172.21.15.96:0/3730393048 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4324037070 con 0x7f433015fa00 2024-05-28T06:14:38.855 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:38.854+0000 7f432f400640 1 -- 172.21.15.96:0/3730393048 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4324032440 con 0x7f433015fa00 2024-05-28T06:14:38.855 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:38.854+0000 7f432f400640 1 -- 172.21.15.96:0/3730393048 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f432402ae60 con 0x7f433015fa00 2024-05-28T06:14:38.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:38.855+0000 7f4335e00640 1 -- 172.21.15.96:0/3730393048 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f433015fa00 msgr2=0x7f433006e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:38.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:38.855+0000 7f4335e00640 1 --2- 172.21.15.96:0/3730393048 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f433015fa00 0x7f433006e970 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f433006d980 tx=0x7f432402a110 comp rx=0 tx=0).stop 2024-05-28T06:14:38.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:38.855+0000 7f4335e00640 1 -- 172.21.15.96:0/3730393048 shutdown_connections 2024-05-28T06:14:38.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:38.855+0000 7f4335e00640 1 --2- 172.21.15.96:0/3730393048 >> v2:172.21.15.123:3300/0 conn(0x7f43301600f0 0x7f43300d0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:38.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:38.855+0000 7f4335e00640 1 --2- 172.21.15.96:0/3730393048 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f433015fa00 0x7f433006e970 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:38.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:38.855+0000 7f4335e00640 1 -- 172.21.15.96:0/3730393048 >> 172.21.15.96:0/3730393048 conn(0x7f433006bae0 msgr2=0x7f433006bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:38.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:38.855+0000 7f4335e00640 1 -- 172.21.15.96:0/3730393048 shutdown_connections 2024-05-28T06:14:38.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:38.855+0000 7f4335e00640 1 -- 172.21.15.96:0/3730393048 wait complete. 2024-05-28T06:14:38.858 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:38.857+0000 7f4335e00640 1 Processor -- start 2024-05-28T06:14:38.858 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:38.857+0000 7f4335e00640 1 -- start start 2024-05-28T06:14:38.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:38.859+0000 7f4335e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f43300d0970 0x7f4330099850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:38.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:38.859+0000 7f4335e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f433015fa00 0x7f4330145110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:38.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:38.859+0000 7f4335e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f43301600f0 0x7f43300b1040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:38.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:38.859+0000 7f4335e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4330071c70 con 0x7f433015fa00 2024-05-28T06:14:38.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:38.859+0000 7f4335e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f433006f720 con 0x7f43300d0970 2024-05-28T06:14:38.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:38.859+0000 7f4335e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f433015fde0 con 0x7f43301600f0 2024-05-28T06:14:38.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:38.859+0000 7f432fe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f433015fa00 0x7f4330145110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:38.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:38.859+0000 7f432fe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f433015fa00 0x7f4330145110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35422/0 (socket says 172.21.15.96:35422) 2024-05-28T06:14:38.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:38.859+0000 7f432fe00640 1 -- 172.21.15.96:0/3562492406 learned_addr learned my addr 172.21.15.96:0/3562492406 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:38.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:38.859+0000 7f4334a00640 1 --2- 172.21.15.96:0/3562492406 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f43300d0970 0x7f4330099850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:38.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:38.859+0000 7f432fe00640 1 -- 172.21.15.96:0/3562492406 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f43301600f0 msgr2=0x7f43300b1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:38.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:38.860+0000 7f432fe00640 1 --2- 172.21.15.96:0/3562492406 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f43301600f0 0x7f43300b1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:38.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:38.860+0000 7f432fe00640 1 -- 172.21.15.96:0/3562492406 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f43300d0970 msgr2=0x7f4330099850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:38.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:38.860+0000 7f432fe00640 1 --2- 172.21.15.96:0/3562492406 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f43300d0970 0x7f4330099850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:38.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:38.860+0000 7f432fe00640 1 -- 172.21.15.96:0/3562492406 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43301e63a0 con 0x7f433015fa00 2024-05-28T06:14:38.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:38.860+0000 7f4334a00640 1 --2- 172.21.15.96:0/3562492406 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f43300d0970 0x7f4330099850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:14:38.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:38.860+0000 7f432fe00640 1 --2- 172.21.15.96:0/3562492406 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f433015fa00 0x7f4330145110 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f431c00e870 tx=0x7f431c00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:38.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:38.860+0000 7f432cc00640 1 -- 172.21.15.96:0/3562492406 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f431c017070 con 0x7f433015fa00 2024-05-28T06:14:38.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:38.860+0000 7f432cc00640 1 -- 172.21.15.96:0/3562492406 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f431c00c940 con 0x7f433015fa00 2024-05-28T06:14:38.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:38.860+0000 7f432cc00640 1 -- 172.21.15.96:0/3562492406 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f431c00cd30 con 0x7f433015fa00 2024-05-28T06:14:38.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:38.860+0000 7f4335e00640 1 -- 172.21.15.96:0/3562492406 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43301e6680 con 0x7f433015fa00 2024-05-28T06:14:38.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:38.860+0000 7f4335e00640 1 -- 172.21.15.96:0/3562492406 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43301595b0 con 0x7f433015fa00 2024-05-28T06:14:38.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:38.861+0000 7f4335e00640 1 -- 172.21.15.96:0/3562492406 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43300689e0 con 0x7f433015fa00 2024-05-28T06:14:38.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:38.862+0000 7f432cc00640 1 -- 172.21.15.96:0/3562492406 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 22) ==== 92213+0+0 (secure 0 0 0) 0x7f431c01a070 con 0x7f433015fa00 2024-05-28T06:14:38.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:38.864+0000 7f432cc00640 1 --2- 172.21.15.96:0/3562492406 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f431006d5e0 0x7f431006faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:38.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:38.864+0000 7f432cc00640 1 -- 172.21.15.96:0/3562492406 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 5823+0+0 (secure 0 0 0) 0x7f431c08a890 con 0x7f433015fa00 2024-05-28T06:14:38.865 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:38.865+0000 7f4334a00640 1 --2- 172.21.15.96:0/3562492406 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f431006d5e0 0x7f431006faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:38.866 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:38.866+0000 7f4334a00640 1 --2- 172.21.15.96:0/3562492406 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f431006d5e0 0x7f431006faa0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f4324034000 tx=0x7f432402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:38.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:38.868+0000 7f432cc00640 1 -- 172.21.15.96:0/3562492406 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f431c05a650 con 0x7f433015fa00 2024-05-28T06:14:39.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.052+0000 7f4335e00640 1 -- 172.21.15.96:0/3562492406 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f43300cda60 con 0x7f433015fa00 2024-05-28T06:14:39.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.750+0000 7f432cc00640 1 -- 172.21.15.96:0/3562492406 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v64) ==== 181+0+0 (secure 0 0 0) 0x7f431c059f50 con 0x7f433015fa00 2024-05-28T06:14:39.751 INFO:teuthology.orchestra.run.smithi096.stderr:pool 'cephfs_metadata' removed 2024-05-28T06:14:39.755 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.754+0000 7f4335e00640 1 -- 172.21.15.96:0/3562492406 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f431006d5e0 msgr2=0x7f431006faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:39.755 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.754+0000 7f4335e00640 1 --2- 172.21.15.96:0/3562492406 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f431006d5e0 0x7f431006faa0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f4324034000 tx=0x7f432402e000 comp rx=0 tx=0).stop 2024-05-28T06:14:39.755 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.754+0000 7f4335e00640 1 -- 172.21.15.96:0/3562492406 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f433015fa00 msgr2=0x7f4330145110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:39.755 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.754+0000 7f4335e00640 1 --2- 172.21.15.96:0/3562492406 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f433015fa00 0x7f4330145110 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f431c00e870 tx=0x7f431c00ed30 comp rx=0 tx=0).stop 2024-05-28T06:14:39.755 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.754+0000 7f4335e00640 1 -- 172.21.15.96:0/3562492406 shutdown_connections 2024-05-28T06:14:39.755 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.754+0000 7f4335e00640 1 --2- 172.21.15.96:0/3562492406 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f431006d5e0 0x7f431006faa0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:39.755 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.754+0000 7f4335e00640 1 --2- 172.21.15.96:0/3562492406 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f43301600f0 0x7f43300b1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:39.755 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.754+0000 7f4335e00640 1 --2- 172.21.15.96:0/3562492406 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f433015fa00 0x7f4330145110 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:39.755 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.754+0000 7f4335e00640 1 --2- 172.21.15.96:0/3562492406 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f43300d0970 0x7f4330099850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:39.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.754+0000 7f4335e00640 1 -- 172.21.15.96:0/3562492406 >> 172.21.15.96:0/3562492406 conn(0x7f433006bae0 msgr2=0x7f433015eca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:39.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.754+0000 7f4335e00640 1 -- 172.21.15.96:0/3562492406 shutdown_connections 2024-05-28T06:14:39.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.754+0000 7f4335e00640 1 -- 172.21.15.96:0/3562492406 wait complete. 2024-05-28T06:14:39.780 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-05-28T06:14:39.972 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.971+0000 7f135b200640 1 Processor -- start 2024-05-28T06:14:39.972 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.972+0000 7f135b200640 1 -- start start 2024-05-28T06:14:39.974 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.973+0000 7f135b200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f135c0d3270 0x7f135c162e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:39.974 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.973+0000 7f135b200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f135c0cf620 con 0x7f135c0d03e0 2024-05-28T06:14:39.974 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.973+0000 7f135b200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f135c0cf900 con 0x7f135c15e5e0 2024-05-28T06:14:39.974 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.973+0000 7f135b200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f135c0cfbe0 con 0x7f135c0d3270 2024-05-28T06:14:39.976 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.974+0000 7f135a800640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f135c0d03e0 0x7f135c168720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:56468/0 (socket says 172.21.15.96:56468) 2024-05-28T06:14:39.976 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.974+0000 7f135a800640 1 -- 172.21.15.96:0/307192574 learned_addr learned my addr 172.21.15.96:0/307192574 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:14:39.976 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.974+0000 7f1351400640 1 --2- 172.21.15.96:0/307192574 >> v2:172.21.15.123:3300/0 conn(0x7f135c0d3270 0x7f135c162e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:39.976 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.974+0000 7f1359400640 1 -- 172.21.15.96:0/307192574 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2254564424 0 0) 0x7f135c0cf620 con 0x7f135c0d03e0 2024-05-28T06:14:39.976 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.974+0000 7f1359400640 1 -- 172.21.15.96:0/307192574 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1340003520 con 0x7f135c0d03e0 2024-05-28T06:14:39.976 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.974+0000 7f1359400640 1 -- 172.21.15.96:0/307192574 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3085923840 0 0) 0x7f1340003520 con 0x7f135c0d03e0 2024-05-28T06:14:39.976 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.975+0000 7f1359400640 1 -- 172.21.15.96:0/307192574 >> v2:172.21.15.123:3300/0 conn(0x7f135c0d3270 msgr2=0x7f135c162e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:39.976 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.975+0000 7f1359400640 1 --2- 172.21.15.96:0/307192574 >> v2:172.21.15.123:3300/0 conn(0x7f135c0d3270 0x7f135c162e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:39.976 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.975+0000 7f1359400640 1 -- 172.21.15.96:0/307192574 >> v1:172.21.15.123:6789/0 conn(0x7f135c15e5e0 legacy=0x7f135c0d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:39.976 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.975+0000 7f1359400640 1 -- 172.21.15.96:0/307192574 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f135c0cfec0 con 0x7f135c0d03e0 2024-05-28T06:14:39.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.975+0000 7f1359400640 1 -- 172.21.15.96:0/307192574 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f13540019f0 con 0x7f135c0d03e0 2024-05-28T06:14:39.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.975+0000 7f1359400640 1 -- 172.21.15.96:0/307192574 >> v1:172.21.15.96:6789/0 conn(0x7f135c0d03e0 legacy=0x7f135c168720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:39.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.975+0000 7f1359400640 1 --2- 172.21.15.96:0/307192574 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f13400039b0 0x7f1340003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:39.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.975+0000 7f1359400640 1 --2- 172.21.15.96:0/307192574 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1340004800 0x7f1340004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:39.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.975+0000 7f1359400640 1 --2- 172.21.15.96:0/307192574 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13400051c0 0x7f134000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:39.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.975+0000 7f1359400640 1 -- 172.21.15.96:0/307192574 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f135c0cfbe0 con 0x7f13400051c0 2024-05-28T06:14:39.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.975+0000 7f1359400640 1 -- 172.21.15.96:0/307192574 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f134000c060 con 0x7f1340004800 2024-05-28T06:14:39.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.975+0000 7f1359400640 1 -- 172.21.15.96:0/307192574 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f134000c260 con 0x7f13400039b0 2024-05-28T06:14:39.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.975+0000 7f135a800640 1 --2- 172.21.15.96:0/307192574 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13400051c0 0x7f134000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:39.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.975+0000 7f1359e00640 1 --2- 172.21.15.96:0/307192574 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f13400039b0 0x7f1340003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:39.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.975+0000 7f1351400640 1 --2- 172.21.15.96:0/307192574 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1340004800 0x7f1340004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:39.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.975+0000 7f135a800640 1 -- 172.21.15.96:0/307192574 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f13400039b0 msgr2=0x7f1340003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:39.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.975+0000 7f135a800640 1 --2- 172.21.15.96:0/307192574 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f13400039b0 0x7f1340003dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:39.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.975+0000 7f135a800640 1 -- 172.21.15.96:0/307192574 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1340004800 msgr2=0x7f1340004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:39.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.975+0000 7f135a800640 1 --2- 172.21.15.96:0/307192574 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1340004800 0x7f1340004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:39.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.976+0000 7f135a800640 1 -- 172.21.15.96:0/307192574 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1340003520 con 0x7f13400051c0 2024-05-28T06:14:39.979 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.976+0000 7f135a800640 1 -- 172.21.15.96:0/307192574 reap_dead start 2024-05-28T06:14:39.979 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.977+0000 7f135a800640 1 --2- 172.21.15.96:0/307192574 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13400051c0 0x7f134000b9c0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f135400a840 tx=0x7f1340005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9e2a19154344d2 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:39.979 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.977+0000 7f1359400640 1 -- 172.21.15.96:0/307192574 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f135400a460 con 0x7f13400051c0 2024-05-28T06:14:39.980 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.977+0000 7f1359400640 1 -- 172.21.15.96:0/307192574 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f135402f070 con 0x7f13400051c0 2024-05-28T06:14:39.980 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.977+0000 7f1359400640 1 -- 172.21.15.96:0/307192574 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f135402bcb0 con 0x7f13400051c0 2024-05-28T06:14:39.980 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.977+0000 7f135b200640 1 -- 172.21.15.96:0/307192574 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13400051c0 msgr2=0x7f134000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:39.980 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.977+0000 7f135b200640 1 --2- 172.21.15.96:0/307192574 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13400051c0 0x7f134000b9c0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f135400a840 tx=0x7f1340005830 comp rx=0 tx=0).stop 2024-05-28T06:14:39.980 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.977+0000 7f135b200640 1 -- 172.21.15.96:0/307192574 shutdown_connections 2024-05-28T06:14:39.980 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.977+0000 7f135b200640 1 --2- 172.21.15.96:0/307192574 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13400051c0 0x7f134000b9c0 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:39.980 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.977+0000 7f135b200640 1 -- 172.21.15.96:0/307192574 >> 172.21.15.96:0/307192574 conn(0x7f135c08bfc0 msgr2=0x7f135c0d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:39.980 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.978+0000 7f135b200640 1 -- 172.21.15.96:0/307192574 shutdown_connections 2024-05-28T06:14:39.980 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.978+0000 7f135b200640 1 -- 172.21.15.96:0/307192574 wait complete. 2024-05-28T06:14:39.982 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.979+0000 7f135b200640 1 Processor -- start 2024-05-28T06:14:39.982 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.979+0000 7f135b200640 1 -- start start 2024-05-28T06:14:39.982 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.980+0000 7f135b200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13400051c0 0x7f135c1f7790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:39.982 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.980+0000 7f135b200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f135c06d7c0 0x7f135c06dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:39.982 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.980+0000 7f135b200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f135c0b3ee0 0x7f135c0b4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:39.982 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.980+0000 7f135b200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f135c15eb40 con 0x7f13400051c0 2024-05-28T06:14:39.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.980+0000 7f135b200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f135c15e9c0 con 0x7f135c06d7c0 2024-05-28T06:14:39.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.980+0000 7f135b200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f135c15ecc0 con 0x7f135c0b3ee0 2024-05-28T06:14:39.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.980+0000 7f1359e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13400051c0 0x7f135c1f7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:39.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.980+0000 7f1359e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13400051c0 0x7f135c1f7790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35436/0 (socket says 172.21.15.96:35436) 2024-05-28T06:14:39.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.980+0000 7f1359e00640 1 -- 172.21.15.96:0/65116856 learned_addr learned my addr 172.21.15.96:0/65116856 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:39.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.981+0000 7f1359e00640 1 -- 172.21.15.96:0/65116856 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f135c0b3ee0 msgr2=0x7f135c0b4360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:39.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.981+0000 7f1359e00640 1 --2- 172.21.15.96:0/65116856 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f135c0b3ee0 0x7f135c0b4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:39.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.981+0000 7f1359e00640 1 -- 172.21.15.96:0/65116856 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f135c06d7c0 msgr2=0x7f135c06dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:39.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.981+0000 7f1359e00640 1 --2- 172.21.15.96:0/65116856 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f135c06d7c0 0x7f135c06dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:39.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.981+0000 7f1359e00640 1 -- 172.21.15.96:0/65116856 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f135c151520 con 0x7f13400051c0 2024-05-28T06:14:39.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.981+0000 7f1359e00640 1 --2- 172.21.15.96:0/65116856 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13400051c0 0x7f135c1f7790 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f134c00d520 tx=0x7f134c00d9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:39.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.981+0000 7f1352a00640 1 -- 172.21.15.96:0/65116856 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f134c005480 con 0x7f13400051c0 2024-05-28T06:14:39.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.981+0000 7f1352a00640 1 -- 172.21.15.96:0/65116856 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f134c017070 con 0x7f13400051c0 2024-05-28T06:14:39.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.981+0000 7f1352a00640 1 -- 172.21.15.96:0/65116856 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f134c003830 con 0x7f13400051c0 2024-05-28T06:14:39.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.981+0000 7f135b200640 1 -- 172.21.15.96:0/65116856 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f135c14e750 con 0x7f13400051c0 2024-05-28T06:14:39.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.982+0000 7f135b200640 1 -- 172.21.15.96:0/65116856 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f135c14ed10 con 0x7f13400051c0 2024-05-28T06:14:39.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.983+0000 7f135b200640 1 -- 172.21.15.96:0/65116856 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f135c06e180 con 0x7f13400051c0 2024-05-28T06:14:39.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.984+0000 7f1352a00640 1 -- 172.21.15.96:0/65116856 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 22) ==== 92213+0+0 (secure 0 0 0) 0x7f134c0039d0 con 0x7f13400051c0 2024-05-28T06:14:39.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.985+0000 7f1352a00640 1 --2- 172.21.15.96:0/65116856 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f133806d5e0 0x7f133806faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:39.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.985+0000 7f1352a00640 1 -- 172.21.15.96:0/65116856 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 5385+0+0 (secure 0 0 0) 0x7f134c08a580 con 0x7f13400051c0 2024-05-28T06:14:39.986 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.986+0000 7f1351400640 1 --2- 172.21.15.96:0/65116856 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f133806d5e0 0x7f133806faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:39.987 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.987+0000 7f1351400640 1 --2- 172.21.15.96:0/65116856 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f133806d5e0 0x7f133806faa0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f135c0b2680 tx=0x7f1344008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:39.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:39.989+0000 7f1352a00640 1 -- 172.21.15.96:0/65116856 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f134c05a170 con 0x7f13400051c0 2024-05-28T06:14:40.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:40.187+0000 7f135b200640 1 -- 172.21.15.96:0/65116856 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f135c0690f0 con 0x7f13400051c0 2024-05-28T06:14:40.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:40.761+0000 7f1352a00640 1 -- 172.21.15.96:0/65116856 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v65) ==== 169+0+0 (secure 0 0 0) 0x7f134c059a70 con 0x7f13400051c0 2024-05-28T06:14:40.762 INFO:teuthology.orchestra.run.smithi096.stderr:pool 'cephfs_data' removed 2024-05-28T06:14:40.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:40.766+0000 7f135b200640 1 -- 172.21.15.96:0/65116856 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f133806d5e0 msgr2=0x7f133806faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:40.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:40.766+0000 7f135b200640 1 --2- 172.21.15.96:0/65116856 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f133806d5e0 0x7f133806faa0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f135c0b2680 tx=0x7f1344008040 comp rx=0 tx=0).stop 2024-05-28T06:14:40.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:40.766+0000 7f135b200640 1 -- 172.21.15.96:0/65116856 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13400051c0 msgr2=0x7f135c1f7790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:40.767 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:40.766+0000 7f135b200640 1 --2- 172.21.15.96:0/65116856 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13400051c0 0x7f135c1f7790 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f134c00d520 tx=0x7f134c00d9e0 comp rx=0 tx=0).stop 2024-05-28T06:14:40.767 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:40.766+0000 7f135b200640 1 -- 172.21.15.96:0/65116856 shutdown_connections 2024-05-28T06:14:40.767 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:40.766+0000 7f135b200640 1 --2- 172.21.15.96:0/65116856 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f133806d5e0 0x7f133806faa0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:40.767 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:40.766+0000 7f135b200640 1 --2- 172.21.15.96:0/65116856 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f135c0b3ee0 0x7f135c0b4360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:40.767 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:40.766+0000 7f135b200640 1 --2- 172.21.15.96:0/65116856 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f135c06d7c0 0x7f135c06dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:40.767 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:40.766+0000 7f135b200640 1 --2- 172.21.15.96:0/65116856 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13400051c0 0x7f135c1f7790 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:40.767 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:40.766+0000 7f135b200640 1 -- 172.21.15.96:0/65116856 >> 172.21.15.96:0/65116856 conn(0x7f135c08bfc0 msgr2=0x7f135c0b2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:40.770 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:40.769+0000 7f135b200640 1 -- 172.21.15.96:0/65116856 shutdown_connections 2024-05-28T06:14:40.770 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:40.770+0000 7f135b200640 1 -- 172.21.15.96:0/65116856 wait complete. 2024-05-28T06:14:40.803 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/volumes/snapshot_clone_no_wait 2024-05-28T06:14:40.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:40.977+0000 7fd62ce00640 1 Processor -- start 2024-05-28T06:14:40.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:40.977+0000 7fd62ce00640 1 -- start start 2024-05-28T06:14:40.979 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:40.979+0000 7fd62ce00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd6280d0640 0x7fd62815d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:40.980 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:40.979+0000 7fd62ce00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fd6280d0a20 0x7fd6280cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:40.980 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:40.979+0000 7fd62ce00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd62815cb10 con 0x7fd6280b32c0 2024-05-28T06:14:40.980 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:40.979+0000 7fd62ce00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fd6280adbb0 con 0x7fd6280d0a20 2024-05-28T06:14:40.980 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:40.979+0000 7fd62ce00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fd62815db10 con 0x7fd6280d0640 2024-05-28T06:14:40.980 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:40.979+0000 7fd625400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fd6280d0a20 0x7fd6280cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:40.980 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:40.979+0000 7fd625400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fd6280d0a20 0x7fd6280cf950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35440/0 (socket says 172.21.15.96:35440) 2024-05-28T06:14:40.980 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:40.979+0000 7fd625400640 1 -- 172.21.15.96:0/1373338523 learned_addr learned my addr 172.21.15.96:0/1373338523 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:40.980 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:40.979+0000 7fd625400640 1 -- 172.21.15.96:0/1373338523 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd6280d0640 msgr2=0x7fd62815d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:40.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:40.979+0000 7fd625400640 1 --2- 172.21.15.96:0/1373338523 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd6280d0640 0x7fd62815d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:40.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:40.979+0000 7fd625400640 1 -- 172.21.15.96:0/1373338523 >> v1:172.21.15.123:6789/0 conn(0x7fd6280b32c0 legacy=0x7fd6280b3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:40.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:40.980+0000 7fd625400640 1 -- 172.21.15.96:0/1373338523 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6280ad5f0 con 0x7fd6280d0a20 2024-05-28T06:14:40.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:40.980+0000 7fd625400640 1 --2- 172.21.15.96:0/1373338523 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd6280d0a20 0x7fd6280cf950 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7fd610009e50 tx=0x7fd61002a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=278a2a9fa1c09c76 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:40.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:40.981+0000 7fd624a00640 1 -- 172.21.15.96:0/1373338523 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd610036070 con 0x7fd6280d0a20 2024-05-28T06:14:40.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:40.981+0000 7fd624a00640 1 -- 172.21.15.96:0/1373338523 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd610031600 con 0x7fd6280d0a20 2024-05-28T06:14:40.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:40.981+0000 7fd624a00640 1 -- 172.21.15.96:0/1373338523 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6100319d0 con 0x7fd6280d0a20 2024-05-28T06:14:40.982 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:40.981+0000 7fd62ce00640 1 -- 172.21.15.96:0/1373338523 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd6280d0a20 msgr2=0x7fd6280cf950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:40.982 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:40.981+0000 7fd62ce00640 1 --2- 172.21.15.96:0/1373338523 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd6280d0a20 0x7fd6280cf950 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7fd610009e50 tx=0x7fd61002a6c0 comp rx=0 tx=0).stop 2024-05-28T06:14:40.982 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:40.981+0000 7fd62ce00640 1 -- 172.21.15.96:0/1373338523 shutdown_connections 2024-05-28T06:14:40.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:40.981+0000 7fd62ce00640 1 --2- 172.21.15.96:0/1373338523 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd6280d0a20 0x7fd6280cf950 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:40.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:40.981+0000 7fd62ce00640 1 --2- 172.21.15.96:0/1373338523 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd6280d0640 0x7fd62815d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:40.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:40.981+0000 7fd62ce00640 1 -- 172.21.15.96:0/1373338523 >> 172.21.15.96:0/1373338523 conn(0x7fd62806bd20 msgr2=0x7fd62806c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:40.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:40.982+0000 7fd62ce00640 1 -- 172.21.15.96:0/1373338523 shutdown_connections 2024-05-28T06:14:40.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:40.982+0000 7fd62ce00640 1 -- 172.21.15.96:0/1373338523 wait complete. 2024-05-28T06:14:40.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:40.983+0000 7fd62ce00640 1 Processor -- start 2024-05-28T06:14:40.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:40.983+0000 7fd62ce00640 1 -- start start 2024-05-28T06:14:40.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:40.984+0000 7fd62ce00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd6280b32c0 0x7fd6281f2520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:40.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:40.984+0000 7fd62ce00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd6280d0640 0x7fd62809a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:40.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:40.984+0000 7fd62ce00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd6280d0a20 0x7fd628144490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:40.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:40.984+0000 7fd62ce00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd62815d0b0 con 0x7fd6280d0640 2024-05-28T06:14:40.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:40.984+0000 7fd62ce00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd62815cdd0 con 0x7fd6280d0a20 2024-05-28T06:14:40.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:40.984+0000 7fd62ce00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fd6280b3e60 con 0x7fd6280b32c0 2024-05-28T06:14:40.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:40.985+0000 7fd625400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd6280d0640 0x7fd62809a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:40.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:40.985+0000 7fd625400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd6280d0640 0x7fd62809a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35456/0 (socket says 172.21.15.96:35456) 2024-05-28T06:14:40.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:40.985+0000 7fd625400640 1 -- 172.21.15.96:0/4053268192 learned_addr learned my addr 172.21.15.96:0/4053268192 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:40.987 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:40.985+0000 7fd625e00640 1 --2- 172.21.15.96:0/4053268192 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd6280b32c0 0x7fd6281f2520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:40.987 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:40.985+0000 7fd626800640 1 --2- 172.21.15.96:0/4053268192 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd6280d0a20 0x7fd628144490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:40.987 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:40.985+0000 7fd625400640 1 -- 172.21.15.96:0/4053268192 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd6280b32c0 msgr2=0x7fd6281f2520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:40.987 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:40.985+0000 7fd625400640 1 --2- 172.21.15.96:0/4053268192 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd6280b32c0 0x7fd6281f2520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:40.987 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:40.985+0000 7fd625400640 1 -- 172.21.15.96:0/4053268192 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd6280d0a20 msgr2=0x7fd628144490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:40.987 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:40.985+0000 7fd625400640 1 --2- 172.21.15.96:0/4053268192 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd6280d0a20 0x7fd628144490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:40.987 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:40.985+0000 7fd625400640 1 -- 172.21.15.96:0/4053268192 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd62809d1e0 con 0x7fd6280d0640 2024-05-28T06:14:40.987 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:40.985+0000 7fd626800640 1 --2- 172.21.15.96:0/4053268192 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd6280d0a20 0x7fd628144490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:14:40.987 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:40.985+0000 7fd625400640 1 --2- 172.21.15.96:0/4053268192 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd6280d0640 0x7fd62809a2b0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7fd610009e50 tx=0x7fd610004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:40.987 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:40.986+0000 7fd61e000640 1 -- 172.21.15.96:0/4053268192 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd610041020 con 0x7fd6280d0640 2024-05-28T06:14:40.987 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:40.986+0000 7fd61e000640 1 -- 172.21.15.96:0/4053268192 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd610007800 con 0x7fd6280d0640 2024-05-28T06:14:40.987 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:40.986+0000 7fd62ce00640 1 -- 172.21.15.96:0/4053268192 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd62809a7f0 con 0x7fd6280d0640 2024-05-28T06:14:40.988 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:40.986+0000 7fd61e000640 1 -- 172.21.15.96:0/4053268192 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd610036040 con 0x7fd6280d0640 2024-05-28T06:14:40.988 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:40.986+0000 7fd62ce00640 1 -- 172.21.15.96:0/4053268192 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd62809ad70 con 0x7fd6280d0640 2024-05-28T06:14:40.988 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:40.987+0000 7fd62ce00640 1 -- 172.21.15.96:0/4053268192 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6280ad2a0 con 0x7fd6280d0640 2024-05-28T06:14:40.988 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:40.988+0000 7fd61e000640 1 -- 172.21.15.96:0/4053268192 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 22) ==== 92213+0+0 (secure 0 0 0) 0x7fd610044020 con 0x7fd6280d0640 2024-05-28T06:14:40.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:40.989+0000 7fd61e000640 1 --2- 172.21.15.96:0/4053268192 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd60406d5e0 0x7fd60406faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:40.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:40.989+0000 7fd61e000640 1 -- 172.21.15.96:0/4053268192 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 4971+0+0 (secure 0 0 0) 0x7fd6100ab580 con 0x7fd6280d0640 2024-05-28T06:14:40.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:40.989+0000 7fd625e00640 1 --2- 172.21.15.96:0/4053268192 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd60406d5e0 0x7fd60406faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:40.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:40.991+0000 7fd625e00640 1 --2- 172.21.15.96:0/4053268192 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd60406d5e0 0x7fd60406faa0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fd628152ed0 tx=0x7fd618008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:40.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:40.993+0000 7fd61e000640 1 -- 172.21.15.96:0/4053268192 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd6280ad2a0 con 0x7fd6280d0640 2024-05-28T06:14:41.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.207+0000 7fd62ce00640 1 -- 172.21.15.96:0/4053268192 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/snapshot_clone_no_wait"} v 0) -- 0x7fd6280689e0 con 0x7fd6280d0640 2024-05-28T06:14:41.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.214+0000 7fd61e000640 1 -- 172.21.15.96:0/4053268192 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/snapshot_clone_no_wait"}]=0 v16) ==== 117+0+0 (secure 0 0 0) 0x7fd61004b030 con 0x7fd6280d0640 2024-05-28T06:14:41.218 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.217+0000 7fd62ce00640 1 -- 172.21.15.96:0/4053268192 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd60406d5e0 msgr2=0x7fd60406faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:41.218 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.217+0000 7fd62ce00640 1 --2- 172.21.15.96:0/4053268192 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd60406d5e0 0x7fd60406faa0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fd628152ed0 tx=0x7fd618008040 comp rx=0 tx=0).stop 2024-05-28T06:14:41.218 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.217+0000 7fd62ce00640 1 -- 172.21.15.96:0/4053268192 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd6280d0640 msgr2=0x7fd62809a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:41.218 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.217+0000 7fd62ce00640 1 --2- 172.21.15.96:0/4053268192 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd6280d0640 0x7fd62809a2b0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7fd610009e50 tx=0x7fd610004a40 comp rx=0 tx=0).stop 2024-05-28T06:14:41.218 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.217+0000 7fd62ce00640 1 -- 172.21.15.96:0/4053268192 shutdown_connections 2024-05-28T06:14:41.218 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.217+0000 7fd62ce00640 1 --2- 172.21.15.96:0/4053268192 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd60406d5e0 0x7fd60406faa0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:41.218 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.217+0000 7fd62ce00640 1 --2- 172.21.15.96:0/4053268192 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd6280d0a20 0x7fd628144490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:41.218 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.217+0000 7fd62ce00640 1 --2- 172.21.15.96:0/4053268192 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd6280d0640 0x7fd62809a2b0 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:41.218 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.217+0000 7fd62ce00640 1 --2- 172.21.15.96:0/4053268192 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd6280b32c0 0x7fd6281f2520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:41.218 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.217+0000 7fd62ce00640 1 -- 172.21.15.96:0/4053268192 >> 172.21.15.96:0/4053268192 conn(0x7fd62806bd20 msgr2=0x7fd628073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:41.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.217+0000 7fd62ce00640 1 -- 172.21.15.96:0/4053268192 shutdown_connections 2024-05-28T06:14:41.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.217+0000 7fd62ce00640 1 -- 172.21.15.96:0/4053268192 wait complete. 2024-05-28T06:14:41.242 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-05-28T06:14:41.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.424+0000 7f75ddc00640 1 Processor -- start 2024-05-28T06:14:41.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.424+0000 7f75ddc00640 1 -- start start 2024-05-28T06:14:41.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.425+0000 7f75ddc00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f75d80b32c0 0x7f75d80b3700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:41.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.425+0000 7f75ddc00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75d80d0060 con 0x7f75d815d5d0 2024-05-28T06:14:41.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.425+0000 7f75ddc00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75d815cb10 con 0x7f75d80d0640 2024-05-28T06:14:41.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.425+0000 7f75ddc00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f75d80adbb0 con 0x7f75d80b32c0 2024-05-28T06:14:41.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.426+0000 7f75d7400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f75d80b32c0 0x7f75d80b3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:41.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.426+0000 7f75ce000640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f75d815d5d0 0x7f75d80cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:56484/0 (socket says 172.21.15.96:56484) 2024-05-28T06:14:41.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.426+0000 7f75ce000640 1 -- 172.21.15.96:0/1998382417 learned_addr learned my addr 172.21.15.96:0/1998382417 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:14:41.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.426+0000 7f75d7400640 1 -- 172.21.15.96:0/1998382417 >> v1:172.21.15.123:6789/0 conn(0x7f75d80d0640 legacy=0x7f75d80d0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:41.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.426+0000 7f75d7400640 1 -- 172.21.15.96:0/1998382417 >> v1:172.21.15.96:6789/0 conn(0x7f75d815d5d0 legacy=0x7f75d80cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:41.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.426+0000 7f75d7400640 1 -- 172.21.15.96:0/1998382417 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75d80ad5f0 con 0x7f75d80b32c0 2024-05-28T06:14:41.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.427+0000 7f75d7400640 1 --2- 172.21.15.96:0/1998382417 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f75d80b32c0 0x7f75d80b3700 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f75d0009d90 tx=0x7f75d002a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83f935f6acac238 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:41.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.427+0000 7f75d6000640 1 -- 172.21.15.96:0/1998382417 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75d0031600 con 0x7f75d80b32c0 2024-05-28T06:14:41.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.427+0000 7f75d6000640 1 -- 172.21.15.96:0/1998382417 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75d00317a0 con 0x7f75d80b32c0 2024-05-28T06:14:41.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.427+0000 7f75d6000640 1 -- 172.21.15.96:0/1998382417 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75d0031b70 con 0x7f75d80b32c0 2024-05-28T06:14:41.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.428+0000 7f75ddc00640 1 -- 172.21.15.96:0/1998382417 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f75d80b32c0 msgr2=0x7f75d80b3700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:41.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.428+0000 7f75ddc00640 1 --2- 172.21.15.96:0/1998382417 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f75d80b32c0 0x7f75d80b3700 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f75d0009d90 tx=0x7f75d002a610 comp rx=0 tx=0).stop 2024-05-28T06:14:41.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.428+0000 7f75ddc00640 1 -- 172.21.15.96:0/1998382417 shutdown_connections 2024-05-28T06:14:41.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.428+0000 7f75ddc00640 1 --2- 172.21.15.96:0/1998382417 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f75d80b32c0 0x7f75d80b3700 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:41.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.428+0000 7f75ddc00640 1 -- 172.21.15.96:0/1998382417 >> 172.21.15.96:0/1998382417 conn(0x7f75d806bd20 msgr2=0x7f75d806c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:41.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.428+0000 7f75ddc00640 1 -- 172.21.15.96:0/1998382417 shutdown_connections 2024-05-28T06:14:41.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.428+0000 7f75ddc00640 1 -- 172.21.15.96:0/1998382417 wait complete. 2024-05-28T06:14:41.430 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.430+0000 7f75ddc00640 1 Processor -- start 2024-05-28T06:14:41.430 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.430+0000 7f75ddc00640 1 -- start start 2024-05-28T06:14:41.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.431+0000 7f75ddc00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f75d80b32c0 0x7f75d809d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:41.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.431+0000 7f75ddc00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f75d80d0640 0x7f75d809a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:41.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.431+0000 7f75ddc00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f75d815d5d0 0x7f75d80b2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:41.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.431+0000 7f75ddc00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f75d815d0b0 con 0x7f75d80b32c0 2024-05-28T06:14:41.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.431+0000 7f75ddc00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f75d815cdd0 con 0x7f75d815d5d0 2024-05-28T06:14:41.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.431+0000 7f75ddc00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f75d815da50 con 0x7f75d80d0640 2024-05-28T06:14:41.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.431+0000 7f75d6a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f75d80b32c0 0x7f75d809d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:41.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.431+0000 7f75d6a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f75d80b32c0 0x7f75d809d330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35470/0 (socket says 172.21.15.96:35470) 2024-05-28T06:14:41.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.431+0000 7f75d6a00640 1 -- 172.21.15.96:0/1620310141 learned_addr learned my addr 172.21.15.96:0/1620310141 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:41.433 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.432+0000 7f75d7400640 1 --2- 172.21.15.96:0/1620310141 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f75d815d5d0 0x7f75d80b2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:41.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.432+0000 7f75d6a00640 1 -- 172.21.15.96:0/1620310141 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f75d80d0640 msgr2=0x7f75d809a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:41.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.432+0000 7f75d6a00640 1 --2- 172.21.15.96:0/1620310141 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f75d80d0640 0x7f75d809a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:41.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.432+0000 7f75d6a00640 1 -- 172.21.15.96:0/1620310141 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f75d815d5d0 msgr2=0x7f75d80b2140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:41.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.432+0000 7f75d6a00640 1 --2- 172.21.15.96:0/1620310141 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f75d815d5d0 0x7f75d80b2140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:41.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.432+0000 7f75d6a00640 1 -- 172.21.15.96:0/1620310141 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75d813e990 con 0x7f75d80b32c0 2024-05-28T06:14:41.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.432+0000 7f75d7400640 1 --2- 172.21.15.96:0/1620310141 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f75d815d5d0 0x7f75d80b2140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:14:41.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.432+0000 7f75d6a00640 1 --2- 172.21.15.96:0/1620310141 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f75d80b32c0 0x7f75d809d330 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f75c800c430 tx=0x7f75c800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:41.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.432+0000 7f75cf400640 1 -- 172.21.15.96:0/1620310141 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75c8011790 con 0x7f75d80b32c0 2024-05-28T06:14:41.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.432+0000 7f75cf400640 1 -- 172.21.15.96:0/1620310141 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75c8017070 con 0x7f75d80b32c0 2024-05-28T06:14:41.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.432+0000 7f75ddc00640 1 -- 172.21.15.96:0/1620310141 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75d813ec70 con 0x7f75d80b32c0 2024-05-28T06:14:41.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.432+0000 7f75cf400640 1 -- 172.21.15.96:0/1620310141 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75c8011bb0 con 0x7f75d80b32c0 2024-05-28T06:14:41.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.433+0000 7f75ddc00640 1 -- 172.21.15.96:0/1620310141 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75d813f230 con 0x7f75d80b32c0 2024-05-28T06:14:41.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.434+0000 7f75ddc00640 1 -- 172.21.15.96:0/1620310141 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75d80689e0 con 0x7f75d80b32c0 2024-05-28T06:14:41.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.435+0000 7f75cf400640 1 -- 172.21.15.96:0/1620310141 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 22) ==== 92213+0+0 (secure 0 0 0) 0x7f75c801c460 con 0x7f75d80b32c0 2024-05-28T06:14:41.436 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.436+0000 7f75cf400640 1 --2- 172.21.15.96:0/1620310141 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f75b406d610 0x7f75b406fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:41.436 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.436+0000 7f75cf400640 1 -- 172.21.15.96:0/1620310141 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 4971+0+0 (secure 0 0 0) 0x7f75c808a390 con 0x7f75d80b32c0 2024-05-28T06:14:41.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.436+0000 7f75ce000640 1 --2- 172.21.15.96:0/1620310141 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f75b406d610 0x7f75b406fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:41.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.438+0000 7f75ce000640 1 --2- 172.21.15.96:0/1620310141 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f75b406d610 0x7f75b406fad0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f75c0000f60 tx=0x7f75c0007a80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:41.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.440+0000 7f75cf400640 1 -- 172.21.15.96:0/1620310141 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f75c805a210 con 0x7f75d80b32c0 2024-05-28T06:14:41.640 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.639+0000 7f75ddc00640 1 -- 172.21.15.96:0/1620310141 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7f75d8071ef0 con 0x7f75d80b32c0 2024-05-28T06:14:41.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.646+0000 7f75cf400640 1 -- 172.21.15.96:0/1620310141 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v17) ==== 104+0+0 (secure 0 0 0) 0x7f75c8059b10 con 0x7f75d80b32c0 2024-05-28T06:14:41.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.649+0000 7f75ddc00640 1 -- 172.21.15.96:0/1620310141 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f75b406d610 msgr2=0x7f75b406fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:41.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.649+0000 7f75ddc00640 1 --2- 172.21.15.96:0/1620310141 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f75b406d610 0x7f75b406fad0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f75c0000f60 tx=0x7f75c0007a80 comp rx=0 tx=0).stop 2024-05-28T06:14:41.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.649+0000 7f75ddc00640 1 -- 172.21.15.96:0/1620310141 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f75d80b32c0 msgr2=0x7f75d809d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:41.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.649+0000 7f75ddc00640 1 --2- 172.21.15.96:0/1620310141 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f75d80b32c0 0x7f75d809d330 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f75c800c430 tx=0x7f75c800c8f0 comp rx=0 tx=0).stop 2024-05-28T06:14:41.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.649+0000 7f75ddc00640 1 -- 172.21.15.96:0/1620310141 shutdown_connections 2024-05-28T06:14:41.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.649+0000 7f75ddc00640 1 --2- 172.21.15.96:0/1620310141 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f75b406d610 0x7f75b406fad0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:41.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.649+0000 7f75ddc00640 1 --2- 172.21.15.96:0/1620310141 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f75d815d5d0 0x7f75d80b2140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:41.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.649+0000 7f75ddc00640 1 --2- 172.21.15.96:0/1620310141 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f75d80d0640 0x7f75d809a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:41.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.649+0000 7f75ddc00640 1 --2- 172.21.15.96:0/1620310141 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f75d80b32c0 0x7f75d809d330 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:41.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.649+0000 7f75ddc00640 1 -- 172.21.15.96:0/1620310141 >> 172.21.15.96:0/1620310141 conn(0x7f75d806bd20 msgr2=0x7f75d815e950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:41.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.649+0000 7f75ddc00640 1 -- 172.21.15.96:0/1620310141 shutdown_connections 2024-05-28T06:14:41.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.650+0000 7f75ddc00640 1 -- 172.21.15.96:0/1620310141 wait complete. 2024-05-28T06:14:41.669 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestCloneProgressReporter.test_progress_bars_for_7_clones' 2024-05-28T06:14:41.867 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.866+0000 7f9857200640 1 Processor -- start 2024-05-28T06:14:41.867 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.866+0000 7f9857200640 1 -- start start 2024-05-28T06:14:41.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.868+0000 7f9857200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f9858156b30 0x7f98580af6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:41.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.868+0000 7f9857200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f985806ee40 0x7f98580ad230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:41.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.868+0000 7f9857200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9858156f10 con 0x7f985806f2f0 2024-05-28T06:14:41.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.868+0000 7f9857200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f985806dda0 con 0x7f985806ee40 2024-05-28T06:14:41.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.868+0000 7f9857200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f985806e080 con 0x7f9858156b30 2024-05-28T06:14:41.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.868+0000 7f9855400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f985806ee40 0x7f98580ad230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:41.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.868+0000 7f9856800640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f985806f2f0 0x7f98580b32c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:56498/0 (socket says 172.21.15.96:56498) 2024-05-28T06:14:41.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.868+0000 7f9856800640 1 -- 172.21.15.96:0/2955975604 learned_addr learned my addr 172.21.15.96:0/2955975604 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:14:41.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.869+0000 7f9855e00640 1 --2- 172.21.15.96:0/2955975604 >> v2:172.21.15.123:3300/0 conn(0x7f9858156b30 0x7f98580af6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:41.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.869+0000 7f9855400640 1 -- 172.21.15.96:0/2955975604 >> v2:172.21.15.123:3300/0 conn(0x7f9858156b30 msgr2=0x7f98580af6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:41.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.869+0000 7f9855400640 1 --2- 172.21.15.96:0/2955975604 >> v2:172.21.15.123:3300/0 conn(0x7f9858156b30 0x7f98580af6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:41.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.869+0000 7f9855400640 1 -- 172.21.15.96:0/2955975604 >> v1:172.21.15.96:6789/0 conn(0x7f985806f2f0 legacy=0x7f98580b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:41.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.869+0000 7f9855400640 1 -- 172.21.15.96:0/2955975604 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f985806e360 con 0x7f985806ee40 2024-05-28T06:14:41.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.869+0000 7f9854a00640 1 -- 172.21.15.96:0/2955975604 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 679466308 0 0) 0x7f9858156f10 con 0x7f985806f2f0 2024-05-28T06:14:41.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.870+0000 7f9855400640 1 --2- 172.21.15.96:0/2955975604 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f985806ee40 0x7f98580ad230 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f985806c8b0 tx=0x7f984402a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6d9d19a113bd163 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:41.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.870+0000 7f9854a00640 1 -- 172.21.15.96:0/2955975604 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9844037070 con 0x7f985806ee40 2024-05-28T06:14:41.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.870+0000 7f9854a00640 1 -- 172.21.15.96:0/2955975604 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9844032440 con 0x7f985806ee40 2024-05-28T06:14:41.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.870+0000 7f9854a00640 1 -- 172.21.15.96:0/2955975604 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f984402ae60 con 0x7f985806ee40 2024-05-28T06:14:41.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.871+0000 7f9857200640 1 -- 172.21.15.96:0/2955975604 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f985806ee40 msgr2=0x7f98580ad230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:41.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.871+0000 7f9857200640 1 --2- 172.21.15.96:0/2955975604 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f985806ee40 0x7f98580ad230 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f985806c8b0 tx=0x7f984402a110 comp rx=0 tx=0).stop 2024-05-28T06:14:41.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.871+0000 7f9857200640 1 -- 172.21.15.96:0/2955975604 shutdown_connections 2024-05-28T06:14:41.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.871+0000 7f9857200640 1 --2- 172.21.15.96:0/2955975604 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f985806ee40 0x7f98580ad230 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:41.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.871+0000 7f9857200640 1 --2- 172.21.15.96:0/2955975604 >> v2:172.21.15.123:3300/0 conn(0x7f9858156b30 0x7f98580af6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:41.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.871+0000 7f9857200640 1 -- 172.21.15.96:0/2955975604 >> 172.21.15.96:0/2955975604 conn(0x7f98580b1fc0 msgr2=0x7f98580b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:41.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.871+0000 7f9857200640 1 -- 172.21.15.96:0/2955975604 shutdown_connections 2024-05-28T06:14:41.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.871+0000 7f9857200640 1 -- 172.21.15.96:0/2955975604 wait complete. 2024-05-28T06:14:41.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.873+0000 7f9857200640 1 Processor -- start 2024-05-28T06:14:41.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.873+0000 7f9857200640 1 -- start start 2024-05-28T06:14:41.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.874+0000 7f9857200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f985806ee40 0x7f9858142f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:41.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.874+0000 7f9857200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f985806f2f0 0x7f98581f5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:41.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.874+0000 7f9857200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9858156b30 0x7f9858152a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:41.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.874+0000 7f9857200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f98580afc20 con 0x7f985806f2f0 2024-05-28T06:14:41.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.874+0000 7f9857200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f98580b2c30 con 0x7f9858156b30 2024-05-28T06:14:41.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.874+0000 7f9857200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f98581f5c40 con 0x7f985806ee40 2024-05-28T06:14:41.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.874+0000 7f9855400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f985806f2f0 0x7f98581f5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:41.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.874+0000 7f9855400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f985806f2f0 0x7f98581f5700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35478/0 (socket says 172.21.15.96:35478) 2024-05-28T06:14:41.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.874+0000 7f9855400640 1 -- 172.21.15.96:0/2469321249 learned_addr learned my addr 172.21.15.96:0/2469321249 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:41.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.874+0000 7f9855400640 1 -- 172.21.15.96:0/2469321249 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f985806ee40 msgr2=0x7f9858142f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:41.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.874+0000 7f9855400640 1 --2- 172.21.15.96:0/2469321249 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f985806ee40 0x7f9858142f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:41.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.874+0000 7f9855400640 1 -- 172.21.15.96:0/2469321249 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9858156b30 msgr2=0x7f9858152a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:41.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.874+0000 7f9855400640 1 --2- 172.21.15.96:0/2469321249 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9858156b30 0x7f9858152a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:41.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.874+0000 7f9855400640 1 -- 172.21.15.96:0/2469321249 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9858152f80 con 0x7f985806f2f0 2024-05-28T06:14:41.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.875+0000 7f9855400640 1 --2- 172.21.15.96:0/2469321249 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f985806f2f0 0x7f98581f5700 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f9844000c00 tx=0x7f98440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:41.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.875+0000 7f984e000640 1 -- 172.21.15.96:0/2469321249 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9844037050 con 0x7f985806f2f0 2024-05-28T06:14:41.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.875+0000 7f984e000640 1 -- 172.21.15.96:0/2469321249 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9844038040 con 0x7f985806f2f0 2024-05-28T06:14:41.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.875+0000 7f9857200640 1 -- 172.21.15.96:0/2469321249 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9858153220 con 0x7f985806f2f0 2024-05-28T06:14:41.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.875+0000 7f984e000640 1 -- 172.21.15.96:0/2469321249 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9844004470 con 0x7f985806f2f0 2024-05-28T06:14:41.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.875+0000 7f9857200640 1 -- 172.21.15.96:0/2469321249 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98581537e0 con 0x7f985806f2f0 2024-05-28T06:14:41.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.877+0000 7f9857200640 1 -- 172.21.15.96:0/2469321249 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f985806faa0 con 0x7f985806f2f0 2024-05-28T06:14:41.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.878+0000 7f984e000640 1 -- 172.21.15.96:0/2469321249 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 22) ==== 92213+0+0 (secure 0 0 0) 0x7f9844030900 con 0x7f985806f2f0 2024-05-28T06:14:41.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.880+0000 7f984e000640 1 --2- 172.21.15.96:0/2469321249 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f983806d610 0x7f983806fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:41.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.880+0000 7f984e000640 1 -- 172.21.15.96:0/2469321249 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 4971+0+0 (secure 0 0 0) 0x7f98440ab0d0 con 0x7f985806f2f0 2024-05-28T06:14:41.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.880+0000 7f9855e00640 1 --2- 172.21.15.96:0/2469321249 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f983806d610 0x7f983806fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:41.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.882+0000 7f9855e00640 1 --2- 172.21.15.96:0/2469321249 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f983806d610 0x7f983806fad0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f9848006fd0 tx=0x7f9848008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:41.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:41.884+0000 7f984e000640 1 -- 172.21.15.96:0/2469321249 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f985806faa0 con 0x7f985806f2f0 2024-05-28T06:14:42.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:42.155+0000 7f9857200640 1 -- 172.21.15.96:0/2469321249 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgressReporter.test_progress_bars_for_7_clones"]} v 0) -- 0x7f98580689e0 con 0x7f985806f2f0 2024-05-28T06:14:42.767 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:42.767+0000 7f984e000640 1 -- 172.21.15.96:0/2469321249 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgressReporter.test_progress_bars_for_7_clones"]}]=0 v459) ==== 162+0+0 (secure 0 0 0) 0x7f984407af50 con 0x7f985806f2f0 2024-05-28T06:14:42.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:42.772+0000 7f9857200640 1 -- 172.21.15.96:0/2469321249 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f983806d610 msgr2=0x7f983806fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:42.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:42.772+0000 7f9857200640 1 --2- 172.21.15.96:0/2469321249 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f983806d610 0x7f983806fad0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f9848006fd0 tx=0x7f9848008040 comp rx=0 tx=0).stop 2024-05-28T06:14:42.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:42.772+0000 7f9857200640 1 -- 172.21.15.96:0/2469321249 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f985806f2f0 msgr2=0x7f98581f5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:42.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:42.772+0000 7f9857200640 1 --2- 172.21.15.96:0/2469321249 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f985806f2f0 0x7f98581f5700 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f9844000c00 tx=0x7f98440046c0 comp rx=0 tx=0).stop 2024-05-28T06:14:42.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:42.772+0000 7f9857200640 1 -- 172.21.15.96:0/2469321249 shutdown_connections 2024-05-28T06:14:42.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:42.772+0000 7f9857200640 1 --2- 172.21.15.96:0/2469321249 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f983806d610 0x7f983806fad0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:42.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:42.772+0000 7f9857200640 1 --2- 172.21.15.96:0/2469321249 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9858156b30 0x7f9858152a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:42.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:42.772+0000 7f9857200640 1 --2- 172.21.15.96:0/2469321249 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f985806f2f0 0x7f98581f5700 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:42.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:42.772+0000 7f9857200640 1 --2- 172.21.15.96:0/2469321249 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f985806ee40 0x7f9858142f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:42.774 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:42.772+0000 7f9857200640 1 -- 172.21.15.96:0/2469321249 >> 172.21.15.96:0/2469321249 conn(0x7f98580b1fc0 msgr2=0x7f98580b02e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:42.774 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:42.772+0000 7f9857200640 1 -- 172.21.15.96:0/2469321249 shutdown_connections 2024-05-28T06:14:42.774 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:42.772+0000 7f9857200640 1 -- 172.21.15.96:0/2469321249 wait complete. 2024-05-28T06:14:42.797 INFO:tasks.cephfs_test_runner:Test that 2 progress bars are printed when number of clone jobs is ... ok 2024-05-28T06:14:42.797 INFO:tasks.cephfs_test_runner:Starting test: test_progress_drops_when_new_jobs_are_added (tasks.cephfs.test_volumes.TestCloneProgressReporter) Test that progress indicated by progress bar for ongoing+pending clones 2024-05-28T06:14:42.797 INFO:tasks.cephfs_test_runner:test_progress_drops_when_new_jobs_are_added (tasks.cephfs.test_volumes.TestCloneProgressReporter) 2024-05-28T06:14:42.798 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:14:43.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.002+0000 7faa57400640 1 Processor -- start 2024-05-28T06:14:43.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.003+0000 7faa57400640 1 -- start start 2024-05-28T06:14:43.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.004+0000 7faa57400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faa500d0a20 0x7faa500997a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:43.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.004+0000 7faa57400640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa500b51f0 con 0x7faa500b4e10 2024-05-28T06:14:43.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.004+0000 7faa57400640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa500aef60 con 0x7faa500d0590 2024-05-28T06:14:43.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.004+0000 7faa57400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7faa500ad1d0 con 0x7faa500d0a20 2024-05-28T06:14:43.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.004+0000 7faa54a00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7faa500b4e10 0x7faa500cde30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:56500/0 (socket says 172.21.15.96:56500) 2024-05-28T06:14:43.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.004+0000 7faa54a00640 1 -- 172.21.15.96:0/3432800785 learned_addr learned my addr 172.21.15.96:0/3432800785 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:14:43.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.005+0000 7faa4f400640 1 -- 172.21.15.96:0/3432800785 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 518201693 0 0) 0x7faa500b51f0 con 0x7faa500b4e10 2024-05-28T06:14:43.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.005+0000 7faa4f400640 1 -- 172.21.15.96:0/3432800785 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa34003520 con 0x7faa500b4e10 2024-05-28T06:14:43.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.005+0000 7faa55400640 1 --2- 172.21.15.96:0/3432800785 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faa500d0a20 0x7faa500997a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:43.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.005+0000 7faa4f400640 1 -- 172.21.15.96:0/3432800785 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3679513826 0 0) 0x7faa34003520 con 0x7faa500b4e10 2024-05-28T06:14:43.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.005+0000 7faa4f400640 1 -- 172.21.15.96:0/3432800785 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faa500d0a20 msgr2=0x7faa500997a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:43.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.005+0000 7faa4f400640 1 --2- 172.21.15.96:0/3432800785 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faa500d0a20 0x7faa500997a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:43.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.005+0000 7faa4f400640 1 -- 172.21.15.96:0/3432800785 >> v1:172.21.15.123:6789/0 conn(0x7faa500d0590 legacy=0x7faa50099090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:43.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.005+0000 7faa4f400640 1 -- 172.21.15.96:0/3432800785 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa500ad470 con 0x7faa500b4e10 2024-05-28T06:14:43.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.005+0000 7faa4f400640 1 -- 172.21.15.96:0/3432800785 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7faa400019f0 con 0x7faa500b4e10 2024-05-28T06:14:43.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.005+0000 7faa4f400640 1 -- 172.21.15.96:0/3432800785 >> v1:172.21.15.96:6789/0 conn(0x7faa500b4e10 legacy=0x7faa500cde30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:43.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.005+0000 7faa4f400640 1 --2- 172.21.15.96:0/3432800785 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faa500d0a20 0x7faa340039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:43.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.005+0000 7faa4f400640 1 --2- 172.21.15.96:0/3432800785 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa34003f60 0x7faa34004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:43.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.005+0000 7faa4f400640 1 --2- 172.21.15.96:0/3432800785 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa34004d40 0x7faa340051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:43.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.005+0000 7faa4f400640 1 -- 172.21.15.96:0/3432800785 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7faa500ad1d0 con 0x7faa34004d40 2024-05-28T06:14:43.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.005+0000 7faa4f400640 1 -- 172.21.15.96:0/3432800785 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7faa3400bb50 con 0x7faa34003f60 2024-05-28T06:14:43.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.005+0000 7faa4f400640 1 -- 172.21.15.96:0/3432800785 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7faa3400bd50 con 0x7faa500d0a20 2024-05-28T06:14:43.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.006+0000 7faa55400640 1 --2- 172.21.15.96:0/3432800785 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa34004d40 0x7faa340051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:43.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.006+0000 7faa54a00640 1 --2- 172.21.15.96:0/3432800785 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faa500d0a20 0x7faa340039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:43.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.006+0000 7faa4fe00640 1 --2- 172.21.15.96:0/3432800785 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa34003f60 0x7faa34004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:43.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.006+0000 7faa55400640 1 -- 172.21.15.96:0/3432800785 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faa500d0a20 msgr2=0x7faa340039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:43.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.006+0000 7faa55400640 1 --2- 172.21.15.96:0/3432800785 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faa500d0a20 0x7faa340039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:43.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.006+0000 7faa55400640 1 -- 172.21.15.96:0/3432800785 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa34003f60 msgr2=0x7faa34004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:43.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.006+0000 7faa55400640 1 --2- 172.21.15.96:0/3432800785 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa34003f60 0x7faa34004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:43.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.006+0000 7faa55400640 1 -- 172.21.15.96:0/3432800785 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa34003520 con 0x7faa34004d40 2024-05-28T06:14:43.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.007+0000 7faa55400640 1 --2- 172.21.15.96:0/3432800785 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa34004d40 0x7faa340051c0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7faa48009ac0 tx=0x7faa34005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26acdd98235d80e8 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:43.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.008+0000 7faa4f400640 1 -- 172.21.15.96:0/3432800785 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa48036070 con 0x7faa34004d40 2024-05-28T06:14:43.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.008+0000 7faa4f400640 1 -- 172.21.15.96:0/3432800785 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa48031400 con 0x7faa34004d40 2024-05-28T06:14:43.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.008+0000 7faa4f400640 1 -- 172.21.15.96:0/3432800785 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa48009690 con 0x7faa34004d40 2024-05-28T06:14:43.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.008+0000 7faa57400640 1 -- 172.21.15.96:0/3432800785 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa34004d40 msgr2=0x7faa340051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:43.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.008+0000 7faa57400640 1 --2- 172.21.15.96:0/3432800785 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa34004d40 0x7faa340051c0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7faa48009ac0 tx=0x7faa34005830 comp rx=0 tx=0).stop 2024-05-28T06:14:43.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.008+0000 7faa55400640 1 -- 172.21.15.96:0/3432800785 reap_dead start 2024-05-28T06:14:43.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.008+0000 7faa57400640 1 -- 172.21.15.96:0/3432800785 shutdown_connections 2024-05-28T06:14:43.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.008+0000 7faa57400640 1 -- 172.21.15.96:0/3432800785 >> 172.21.15.96:0/3432800785 conn(0x7faa5006bae0 msgr2=0x7faa5006bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:43.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.009+0000 7faa57400640 1 -- 172.21.15.96:0/3432800785 shutdown_connections 2024-05-28T06:14:43.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.009+0000 7faa57400640 1 -- 172.21.15.96:0/3432800785 wait complete. 2024-05-28T06:14:43.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.010+0000 7faa57400640 1 Processor -- start 2024-05-28T06:14:43.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.011+0000 7faa57400640 1 -- start start 2024-05-28T06:14:43.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.011+0000 7faa57400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa500ca020 0x7faa5015e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:43.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.011+0000 7faa57400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa50155050 0x7faa500c5f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:43.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.011+0000 7faa57400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faa500c6480 0x7faa501e3b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:43.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.012+0000 7faa57400640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7faa500ce540 con 0x7faa500ca020 2024-05-28T06:14:43.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.012+0000 7faa57400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7faa500d0e70 con 0x7faa50155050 2024-05-28T06:14:43.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.012+0000 7faa57400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7faa500ce880 con 0x7faa500c6480 2024-05-28T06:14:43.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.012+0000 7faa54a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa500ca020 0x7faa5015e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:43.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.012+0000 7faa54a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa500ca020 0x7faa5015e310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35496/0 (socket says 172.21.15.96:35496) 2024-05-28T06:14:43.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.012+0000 7faa54a00640 1 -- 172.21.15.96:0/244864941 learned_addr learned my addr 172.21.15.96:0/244864941 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:43.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.012+0000 7faa54a00640 1 -- 172.21.15.96:0/244864941 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faa500c6480 msgr2=0x7faa501e3b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:43.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.012+0000 7faa54a00640 1 --2- 172.21.15.96:0/244864941 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faa500c6480 0x7faa501e3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:43.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.012+0000 7faa54a00640 1 -- 172.21.15.96:0/244864941 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa50155050 msgr2=0x7faa500c5f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:43.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.012+0000 7faa54a00640 1 --2- 172.21.15.96:0/244864941 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa50155050 0x7faa500c5f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:43.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.012+0000 7faa54a00640 1 -- 172.21.15.96:0/244864941 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa500c74d0 con 0x7faa500ca020 2024-05-28T06:14:43.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.012+0000 7faa54a00640 1 --2- 172.21.15.96:0/244864941 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa500ca020 0x7faa5015e310 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7faa4000e3c0 tx=0x7faa4000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:43.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.013+0000 7faa4cc00640 1 -- 172.21.15.96:0/244864941 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa4000ee10 con 0x7faa500ca020 2024-05-28T06:14:43.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.013+0000 7faa4cc00640 1 -- 172.21.15.96:0/244864941 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa40018070 con 0x7faa500ca020 2024-05-28T06:14:43.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.013+0000 7faa4cc00640 1 -- 172.21.15.96:0/244864941 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa40013660 con 0x7faa500ca020 2024-05-28T06:14:43.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.013+0000 7faa57400640 1 -- 172.21.15.96:0/244864941 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa500c77b0 con 0x7faa500ca020 2024-05-28T06:14:43.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.013+0000 7faa57400640 1 -- 172.21.15.96:0/244864941 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa500c7d70 con 0x7faa500ca020 2024-05-28T06:14:43.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.014+0000 7faa57400640 1 -- 172.21.15.96:0/244864941 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa500689e0 con 0x7faa500ca020 2024-05-28T06:14:43.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.015+0000 7faa4cc00640 1 -- 172.21.15.96:0/244864941 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 22) ==== 92213+0+0 (secure 0 0 0) 0x7faa40013800 con 0x7faa500ca020 2024-05-28T06:14:43.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.016+0000 7faa4cc00640 1 --2- 172.21.15.96:0/244864941 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7faa2c06d610 0x7faa2c06fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:43.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.016+0000 7faa4cc00640 1 -- 172.21.15.96:0/244864941 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 4971+0+0 (secure 0 0 0) 0x7faa4008af00 con 0x7faa500ca020 2024-05-28T06:14:43.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.017+0000 7faa4fe00640 1 --2- 172.21.15.96:0/244864941 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7faa2c06d610 0x7faa2c06fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:43.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.018+0000 7faa4fe00640 1 --2- 172.21.15.96:0/244864941 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7faa2c06d610 0x7faa2c06fad0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7faa38007fd0 tx=0x7faa38009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:43.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.020+0000 7faa4cc00640 1 -- 172.21.15.96:0/244864941 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7faa4005aaf0 con 0x7faa500ca020 2024-05-28T06:14:43.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.221+0000 7faa57400640 1 -- 172.21.15.96:0/244864941 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7faa500ce320 con 0x7faa500ca020 2024-05-28T06:14:43.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.223+0000 7faa4cc00640 1 -- 172.21.15.96:0/244864941 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v65) ==== 74+0+13075 (secure 0 0 0) 0x7faa4005a3f0 con 0x7faa500ca020 2024-05-28T06:14:43.224 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:14:43.224 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":65,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:14:40.751308+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.96:6835/1590650245":"2024-05-29T06:14:37.611501+0000","172.21.15.96:6833/1590650245":"2024-05-29T06:14:37.611501+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:14:43.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.226+0000 7faa57400640 1 -- 172.21.15.96:0/244864941 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7faa2c06d610 msgr2=0x7faa2c06fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:43.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.226+0000 7faa57400640 1 --2- 172.21.15.96:0/244864941 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7faa2c06d610 0x7faa2c06fad0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7faa38007fd0 tx=0x7faa38009040 comp rx=0 tx=0).stop 2024-05-28T06:14:43.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.226+0000 7faa57400640 1 -- 172.21.15.96:0/244864941 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa500ca020 msgr2=0x7faa5015e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:43.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.226+0000 7faa57400640 1 --2- 172.21.15.96:0/244864941 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa500ca020 0x7faa5015e310 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7faa4000e3c0 tx=0x7faa4000e4b0 comp rx=0 tx=0).stop 2024-05-28T06:14:43.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.226+0000 7faa57400640 1 -- 172.21.15.96:0/244864941 shutdown_connections 2024-05-28T06:14:43.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.226+0000 7faa57400640 1 --2- 172.21.15.96:0/244864941 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7faa2c06d610 0x7faa2c06fad0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:43.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.226+0000 7faa57400640 1 --2- 172.21.15.96:0/244864941 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faa500c6480 0x7faa501e3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:43.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.226+0000 7faa57400640 1 --2- 172.21.15.96:0/244864941 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa50155050 0x7faa500c5f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:43.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.226+0000 7faa57400640 1 --2- 172.21.15.96:0/244864941 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa500ca020 0x7faa5015e310 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:43.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.226+0000 7faa57400640 1 -- 172.21.15.96:0/244864941 >> 172.21.15.96:0/244864941 conn(0x7faa5006bae0 msgr2=0x7faa500ccc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:43.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.226+0000 7faa57400640 1 -- 172.21.15.96:0/244864941 shutdown_connections 2024-05-28T06:14:43.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.226+0000 7faa57400640 1 -- 172.21.15.96:0/244864941 wait complete. 2024-05-28T06:14:43.246 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-28T06:03:59.517102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-28T06:14:43.247 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-28T06:14:43.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.454+0000 7f5385e00640 1 Processor -- start 2024-05-28T06:14:43.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.454+0000 7f5385e00640 1 -- start start 2024-05-28T06:14:43.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.455+0000 7f5385e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f538015e940 0x7f53800d0590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:43.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.455+0000 7f5385e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f53800d0ad0 0x7f53800d0f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:43.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.456+0000 7f5385e00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53800d16c0 con 0x7f53800cd6d0 2024-05-28T06:14:43.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.456+0000 7f5385e00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f53800cd240 con 0x7f538015e940 2024-05-28T06:14:43.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.456+0000 7f5385e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f53800d1470 con 0x7f53800d0ad0 2024-05-28T06:14:43.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.456+0000 7f537f400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f53800cd6d0 0x7f53800cdb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:56516/0 (socket says 172.21.15.96:56516) 2024-05-28T06:14:43.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.456+0000 7f537f400640 1 -- 172.21.15.96:0/2960909437 learned_addr learned my addr 172.21.15.96:0/2960909437 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:14:43.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.456+0000 7f537e000640 1 --2- 172.21.15.96:0/2960909437 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f53800d0ad0 0x7f53800d0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:43.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.456+0000 7f537ea00640 1 --2- 172.21.15.96:0/2960909437 >> v2:172.21.15.123:3300/0 conn(0x7f538015e940 0x7f53800d0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:43.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.456+0000 7f537d600640 1 -- 172.21.15.96:0/2960909437 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3690915779 0 0) 0x7f53800d16c0 con 0x7f53800cd6d0 2024-05-28T06:14:43.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.456+0000 7f537d600640 1 -- 172.21.15.96:0/2960909437 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5364003520 con 0x7f53800cd6d0 2024-05-28T06:14:43.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.457+0000 7f537d600640 1 -- 172.21.15.96:0/2960909437 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 595308208 0 0) 0x7f5364003520 con 0x7f53800cd6d0 2024-05-28T06:14:43.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.457+0000 7f537d600640 1 -- 172.21.15.96:0/2960909437 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f53800d0ad0 msgr2=0x7f53800d0f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:43.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.457+0000 7f537d600640 1 --2- 172.21.15.96:0/2960909437 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f53800d0ad0 0x7f53800d0f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:43.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.457+0000 7f537d600640 1 -- 172.21.15.96:0/2960909437 >> v2:172.21.15.123:3300/0 conn(0x7f538015e940 msgr2=0x7f53800d0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:43.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.457+0000 7f537d600640 1 --2- 172.21.15.96:0/2960909437 >> v2:172.21.15.123:3300/0 conn(0x7f538015e940 0x7f53800d0590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:43.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.457+0000 7f537d600640 1 -- 172.21.15.96:0/2960909437 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53800ccbf0 con 0x7f53800cd6d0 2024-05-28T06:14:43.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.457+0000 7f537d600640 1 -- 172.21.15.96:0/2960909437 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f53780019f0 con 0x7f53800cd6d0 2024-05-28T06:14:43.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.457+0000 7f537d600640 1 -- 172.21.15.96:0/2960909437 >> v1:172.21.15.96:6789/0 conn(0x7f53800cd6d0 legacy=0x7f53800cdb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:43.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.457+0000 7f537d600640 1 --2- 172.21.15.96:0/2960909437 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f53800d0ad0 0x7f53640039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:43.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.457+0000 7f537d600640 1 --2- 172.21.15.96:0/2960909437 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5364003f30 0x7f5364004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:43.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.457+0000 7f537d600640 1 --2- 172.21.15.96:0/2960909437 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5364004d40 0x7f53640051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:43.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.457+0000 7f537d600640 1 -- 172.21.15.96:0/2960909437 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f53800cd240 con 0x7f5364003f30 2024-05-28T06:14:43.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.457+0000 7f537d600640 1 -- 172.21.15.96:0/2960909437 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f53800d1470 con 0x7f5364004d40 2024-05-28T06:14:43.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.457+0000 7f537d600640 1 -- 172.21.15.96:0/2960909437 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f536400b9c0 con 0x7f53800d0ad0 2024-05-28T06:14:43.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.457+0000 7f537e000640 1 --2- 172.21.15.96:0/2960909437 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5364003f30 0x7f5364004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:43.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.458+0000 7f537ea00640 1 --2- 172.21.15.96:0/2960909437 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f53800d0ad0 0x7f53640039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:43.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.458+0000 7f537e000640 1 -- 172.21.15.96:0/2960909437 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f53800d0ad0 msgr2=0x7f53640039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:43.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.458+0000 7f537e000640 1 --2- 172.21.15.96:0/2960909437 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f53800d0ad0 0x7f53640039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:43.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.458+0000 7f537e000640 1 -- 172.21.15.96:0/2960909437 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5364004d40 msgr2=0x7f53640051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:43.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.458+0000 7f537e000640 1 --2- 172.21.15.96:0/2960909437 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5364004d40 0x7f53640051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:43.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.458+0000 7f537e000640 1 -- 172.21.15.96:0/2960909437 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5364003520 con 0x7f5364003f30 2024-05-28T06:14:43.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.459+0000 7f537e000640 1 --2- 172.21.15.96:0/2960909437 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5364003f30 0x7f5364004800 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f536c009ac0 tx=0x7f5364005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c44e3d594805a093 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:43.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.459+0000 7f537d600640 1 -- 172.21.15.96:0/2960909437 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f536c036070 con 0x7f5364003f30 2024-05-28T06:14:43.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.459+0000 7f537d600640 1 -- 172.21.15.96:0/2960909437 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f536c031400 con 0x7f5364003f30 2024-05-28T06:14:43.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.459+0000 7f537d600640 1 -- 172.21.15.96:0/2960909437 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f536c009690 con 0x7f5364003f30 2024-05-28T06:14:43.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.460+0000 7f5385e00640 1 -- 172.21.15.96:0/2960909437 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5364003f30 msgr2=0x7f5364004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:43.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.460+0000 7f5385e00640 1 --2- 172.21.15.96:0/2960909437 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5364003f30 0x7f5364004800 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f536c009ac0 tx=0x7f5364005830 comp rx=0 tx=0).stop 2024-05-28T06:14:43.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.460+0000 7f537f400640 1 -- 172.21.15.96:0/2960909437 reap_dead start 2024-05-28T06:14:43.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.460+0000 7f5385e00640 1 -- 172.21.15.96:0/2960909437 shutdown_connections 2024-05-28T06:14:43.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.460+0000 7f5385e00640 1 -- 172.21.15.96:0/2960909437 >> 172.21.15.96:0/2960909437 conn(0x7f53800b3f60 msgr2=0x7f53800b4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:43.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.460+0000 7f5385e00640 1 -- 172.21.15.96:0/2960909437 shutdown_connections 2024-05-28T06:14:43.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.460+0000 7f5385e00640 1 -- 172.21.15.96:0/2960909437 wait complete. 2024-05-28T06:14:43.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.462+0000 7f5385e00640 1 Processor -- start 2024-05-28T06:14:43.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.462+0000 7f5385e00640 1 -- start start 2024-05-28T06:14:43.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.463+0000 7f5385e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5364003f30 0x7f5380158d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:43.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.463+0000 7f5385e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f53800ce170 0x7f53800ce5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:43.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.463+0000 7f5385e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f53800b6000 0x7f53800b6480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:43.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.463+0000 7f5385e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f538015ed20 con 0x7f53800b6000 2024-05-28T06:14:43.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.463+0000 7f5385e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5380070310 con 0x7f53800ce170 2024-05-28T06:14:43.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.463+0000 7f5385e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f53800ceb10 con 0x7f5364003f30 2024-05-28T06:14:43.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.463+0000 7f537f400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f53800b6000 0x7f53800b6480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:43.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.463+0000 7f537f400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f53800b6000 0x7f53800b6480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35522/0 (socket says 172.21.15.96:35522) 2024-05-28T06:14:43.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.463+0000 7f537f400640 1 -- 172.21.15.96:0/3390116933 learned_addr learned my addr 172.21.15.96:0/3390116933 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:43.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.463+0000 7f537ea00640 1 --2- 172.21.15.96:0/3390116933 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5364003f30 0x7f5380158d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:43.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.463+0000 7f537f400640 1 -- 172.21.15.96:0/3390116933 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5364003f30 msgr2=0x7f5380158d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:43.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.463+0000 7f537f400640 1 --2- 172.21.15.96:0/3390116933 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5364003f30 0x7f5380158d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:43.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.463+0000 7f537f400640 1 -- 172.21.15.96:0/3390116933 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f53800ce170 msgr2=0x7f53800ce5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:43.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.463+0000 7f537e000640 1 --2- 172.21.15.96:0/3390116933 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f53800ce170 0x7f53800ce5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:43.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.463+0000 7f537f400640 1 --2- 172.21.15.96:0/3390116933 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f53800ce170 0x7f53800ce5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:43.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.463+0000 7f537f400640 1 -- 172.21.15.96:0/3390116933 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53800cb520 con 0x7f53800b6000 2024-05-28T06:14:43.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.464+0000 7f537f400640 1 --2- 172.21.15.96:0/3390116933 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f53800b6000 0x7f53800b6480 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f53780012e0 tx=0x7f537800cf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:43.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.464+0000 7f5377400640 1 -- 172.21.15.96:0/3390116933 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f537800a610 con 0x7f53800b6000 2024-05-28T06:14:43.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.464+0000 7f5377400640 1 -- 172.21.15.96:0/3390116933 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f537800a7f0 con 0x7f53800b6000 2024-05-28T06:14:43.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.464+0000 7f5377400640 1 -- 172.21.15.96:0/3390116933 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f537800abe0 con 0x7f53800b6000 2024-05-28T06:14:43.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.464+0000 7f5385e00640 1 -- 172.21.15.96:0/3390116933 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53800cb800 con 0x7f53800b6000 2024-05-28T06:14:43.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.464+0000 7f5385e00640 1 -- 172.21.15.96:0/3390116933 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53800cbdc0 con 0x7f53800b6000 2024-05-28T06:14:43.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.465+0000 7f5385e00640 1 -- 172.21.15.96:0/3390116933 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53800689e0 con 0x7f53800b6000 2024-05-28T06:14:43.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.467+0000 7f5377400640 1 -- 172.21.15.96:0/3390116933 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 22) ==== 92213+0+0 (secure 0 0 0) 0x7f5378011070 con 0x7f53800b6000 2024-05-28T06:14:43.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.468+0000 7f5377400640 1 --2- 172.21.15.96:0/3390116933 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f536006d5e0 0x7f536006faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:43.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.468+0000 7f5377400640 1 -- 172.21.15.96:0/3390116933 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 4971+0+0 (secure 0 0 0) 0x7f5378089d30 con 0x7f53800b6000 2024-05-28T06:14:43.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.468+0000 7f537ea00640 1 --2- 172.21.15.96:0/3390116933 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f536006d5e0 0x7f536006faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:43.470 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.470+0000 7f537ea00640 1 --2- 172.21.15.96:0/3390116933 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f536006d5e0 0x7f536006faa0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f53700099d0 tx=0x7f5370008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:43.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.472+0000 7f5377400640 1 -- 172.21.15.96:0/3390116933 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5378059c10 con 0x7f53800b6000 2024-05-28T06:14:43.673 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.673+0000 7f5385e00640 1 -- 172.21.15.96:0/3390116933 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f53800cc4c0 con 0x7f53800b6000 2024-05-28T06:14:43.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.673+0000 7f5377400640 1 -- 172.21.15.96:0/3390116933 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v65) ==== 93+0+10 (secure 0 0 0) 0x7f5378059510 con 0x7f53800b6000 2024-05-28T06:14:43.674 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 1 2024-05-28T06:14:43.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.676+0000 7f5385e00640 1 -- 172.21.15.96:0/3390116933 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f536006d5e0 msgr2=0x7f536006faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:43.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.676+0000 7f5385e00640 1 --2- 172.21.15.96:0/3390116933 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f536006d5e0 0x7f536006faa0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f53700099d0 tx=0x7f5370008040 comp rx=0 tx=0).stop 2024-05-28T06:14:43.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.676+0000 7f5385e00640 1 -- 172.21.15.96:0/3390116933 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f53800b6000 msgr2=0x7f53800b6480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:43.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.676+0000 7f5385e00640 1 --2- 172.21.15.96:0/3390116933 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f53800b6000 0x7f53800b6480 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f53780012e0 tx=0x7f537800cf00 comp rx=0 tx=0).stop 2024-05-28T06:14:43.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.676+0000 7f5385e00640 1 -- 172.21.15.96:0/3390116933 shutdown_connections 2024-05-28T06:14:43.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.676+0000 7f5385e00640 1 --2- 172.21.15.96:0/3390116933 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f536006d5e0 0x7f536006faa0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:43.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.676+0000 7f5385e00640 1 --2- 172.21.15.96:0/3390116933 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f53800b6000 0x7f53800b6480 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:43.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.676+0000 7f5385e00640 1 --2- 172.21.15.96:0/3390116933 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f53800ce170 0x7f53800ce5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:43.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.676+0000 7f5385e00640 1 --2- 172.21.15.96:0/3390116933 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5364003f30 0x7f5380158d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:43.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.676+0000 7f5385e00640 1 -- 172.21.15.96:0/3390116933 >> 172.21.15.96:0/3390116933 conn(0x7f53800b3f60 msgr2=0x7f53800b1230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:43.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.676+0000 7f5385e00640 1 -- 172.21.15.96:0/3390116933 shutdown_connections 2024-05-28T06:14:43.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.676+0000 7f5385e00640 1 -- 172.21.15.96:0/3390116933 wait complete. 2024-05-28T06:14:43.696 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestCloneProgressReporter.test_progress_drops_when_new_jobs_are_added' 2024-05-28T06:14:43.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.888+0000 7f4b9cc00640 1 Processor -- start 2024-05-28T06:14:43.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.889+0000 7f4b9cc00640 1 -- start start 2024-05-28T06:14:43.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.890+0000 7f4b9cc00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f4b9815e870 0x7f4b980d0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:43.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.890+0000 7f4b9cc00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4b980d14d0 0x7f4b980cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:43.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.890+0000 7f4b9cc00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f4b980cf870 0x7f4b98162360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:43.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.890+0000 7f4b9cc00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f4b980d2150 con 0x7f4b980cf870 2024-05-28T06:14:43.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.890+0000 7f4b9cc00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f4b980cfd00 con 0x7f4b9815e870 2024-05-28T06:14:43.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.890+0000 7f4b9cc00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f4b9815ec50 con 0x7f4b980d14d0 2024-05-28T06:14:43.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.890+0000 7f4b97e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f4b980cf870 0x7f4b98162360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:43.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.890+0000 7f4b97e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f4b980cf870 0x7f4b98162360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35524/0 (socket says 172.21.15.96:35524) 2024-05-28T06:14:43.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.890+0000 7f4b97e00640 1 -- 172.21.15.96:0/707004986 learned_addr learned my addr 172.21.15.96:0/707004986 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:43.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.891+0000 7f4b97400640 1 --2- 172.21.15.96:0/707004986 >> v2:172.21.15.123:3300/0 conn(0x7f4b9815e870 0x7f4b980d0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:43.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.891+0000 7f4b97e00640 1 -- 172.21.15.96:0/707004986 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4b980d14d0 msgr2=0x7f4b980cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:43.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.891+0000 7f4b97e00640 1 --2- 172.21.15.96:0/707004986 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4b980d14d0 0x7f4b980cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:43.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.891+0000 7f4b97e00640 1 -- 172.21.15.96:0/707004986 >> v2:172.21.15.123:3300/0 conn(0x7f4b9815e870 msgr2=0x7f4b980d0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:43.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.891+0000 7f4b97e00640 1 --2- 172.21.15.96:0/707004986 >> v2:172.21.15.123:3300/0 conn(0x7f4b9815e870 0x7f4b980d0f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:43.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.891+0000 7f4b97e00640 1 -- 172.21.15.96:0/707004986 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b980cff80 con 0x7f4b980cf870 2024-05-28T06:14:43.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.892+0000 7f4b97e00640 1 --2- 172.21.15.96:0/707004986 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4b980cf870 0x7f4b98162360 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f4b9808e720 tx=0x7f4b9002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e591a60164103ce7 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:43.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.892+0000 7f4b96000640 1 -- 172.21.15.96:0/707004986 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b90037070 con 0x7f4b980cf870 2024-05-28T06:14:43.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.892+0000 7f4b96000640 1 -- 172.21.15.96:0/707004986 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b90032440 con 0x7f4b980cf870 2024-05-28T06:14:43.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.892+0000 7f4b96000640 1 -- 172.21.15.96:0/707004986 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b90032810 con 0x7f4b980cf870 2024-05-28T06:14:43.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.893+0000 7f4b9cc00640 1 -- 172.21.15.96:0/707004986 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4b980cf870 msgr2=0x7f4b98162360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:43.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.893+0000 7f4b9cc00640 1 --2- 172.21.15.96:0/707004986 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4b980cf870 0x7f4b98162360 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f4b9808e720 tx=0x7f4b9002a1e0 comp rx=0 tx=0).stop 2024-05-28T06:14:43.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.893+0000 7f4b9cc00640 1 -- 172.21.15.96:0/707004986 shutdown_connections 2024-05-28T06:14:43.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.893+0000 7f4b9cc00640 1 --2- 172.21.15.96:0/707004986 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4b980cf870 0x7f4b98162360 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:43.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.893+0000 7f4b9cc00640 1 --2- 172.21.15.96:0/707004986 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4b980d14d0 0x7f4b980cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:43.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.893+0000 7f4b9cc00640 1 --2- 172.21.15.96:0/707004986 >> v2:172.21.15.123:3300/0 conn(0x7f4b9815e870 0x7f4b980d0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:43.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.893+0000 7f4b9cc00640 1 -- 172.21.15.96:0/707004986 >> 172.21.15.96:0/707004986 conn(0x7f4b9808bfc0 msgr2=0x7f4b980d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:43.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.893+0000 7f4b9cc00640 1 -- 172.21.15.96:0/707004986 shutdown_connections 2024-05-28T06:14:43.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.893+0000 7f4b9cc00640 1 -- 172.21.15.96:0/707004986 wait complete. 2024-05-28T06:14:43.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.895+0000 7f4b9cc00640 1 Processor -- start 2024-05-28T06:14:43.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.895+0000 7f4b9cc00640 1 -- start start 2024-05-28T06:14:43.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.896+0000 7f4b9cc00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4b980cf870 0x7f4b98160f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:43.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.896+0000 7f4b9cc00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b980d14d0 0x7f4b980b8680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:43.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.896+0000 7f4b9cc00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4b9815e870 0x7f4b980cd820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:43.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.896+0000 7f4b9cc00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4b980d0820 con 0x7f4b980cf870 2024-05-28T06:14:43.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.896+0000 7f4b9cc00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f4b980d0540 con 0x7f4b980d14d0 2024-05-28T06:14:43.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.896+0000 7f4b9cc00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f4b980d4fb0 con 0x7f4b9815e870 2024-05-28T06:14:43.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.896+0000 7f4b97400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4b980cf870 0x7f4b98160f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:43.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.896+0000 7f4b97400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4b980cf870 0x7f4b98160f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35540/0 (socket says 172.21.15.96:35540) 2024-05-28T06:14:43.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.897+0000 7f4b97400640 1 -- 172.21.15.96:0/3139617546 learned_addr learned my addr 172.21.15.96:0/3139617546 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:43.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.897+0000 7f4b97e00640 1 --2- 172.21.15.96:0/3139617546 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4b9815e870 0x7f4b980cd820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:43.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.897+0000 7f4b96a00640 1 --2- 172.21.15.96:0/3139617546 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b980d14d0 0x7f4b980b8680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:43.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.897+0000 7f4b97400640 1 -- 172.21.15.96:0/3139617546 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4b9815e870 msgr2=0x7f4b980cd820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:43.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.897+0000 7f4b97400640 1 --2- 172.21.15.96:0/3139617546 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4b9815e870 0x7f4b980cd820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:43.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.897+0000 7f4b97400640 1 -- 172.21.15.96:0/3139617546 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b980d14d0 msgr2=0x7f4b980b8680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:43.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.897+0000 7f4b97400640 1 --2- 172.21.15.96:0/3139617546 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b980d14d0 0x7f4b980b8680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:43.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.897+0000 7f4b97400640 1 -- 172.21.15.96:0/3139617546 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b9815f3d0 con 0x7f4b980cf870 2024-05-28T06:14:43.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.897+0000 7f4b96a00640 1 --2- 172.21.15.96:0/3139617546 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b980d14d0 0x7f4b980b8680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:14:43.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.897+0000 7f4b97400640 1 --2- 172.21.15.96:0/3139617546 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4b980cf870 0x7f4b98160f30 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f4b8000c7f0 tx=0x7f4b8000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:43.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.898+0000 7f4b8f400640 1 -- 172.21.15.96:0/3139617546 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b80015070 con 0x7f4b980cf870 2024-05-28T06:14:43.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.898+0000 7f4b8f400640 1 -- 172.21.15.96:0/3139617546 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b80004240 con 0x7f4b980cf870 2024-05-28T06:14:43.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.898+0000 7f4b8f400640 1 -- 172.21.15.96:0/3139617546 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b80007620 con 0x7f4b980cf870 2024-05-28T06:14:43.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.898+0000 7f4b9cc00640 1 -- 172.21.15.96:0/3139617546 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b9815f6b0 con 0x7f4b980cf870 2024-05-28T06:14:43.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.898+0000 7f4b9cc00640 1 -- 172.21.15.96:0/3139617546 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b9815fc70 con 0x7f4b980cf870 2024-05-28T06:14:43.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.899+0000 7f4b9cc00640 1 -- 172.21.15.96:0/3139617546 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b980689e0 con 0x7f4b980cf870 2024-05-28T06:14:43.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.900+0000 7f4b8f400640 1 -- 172.21.15.96:0/3139617546 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 22) ==== 92213+0+0 (secure 0 0 0) 0x7f4b800079d0 con 0x7f4b980cf870 2024-05-28T06:14:43.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.901+0000 7f4b8f400640 1 --2- 172.21.15.96:0/3139617546 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4b7406d5e0 0x7f4b7406faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:43.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.901+0000 7f4b8f400640 1 -- 172.21.15.96:0/3139617546 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 4971+0+0 (secure 0 0 0) 0x7f4b8008ace0 con 0x7f4b980cf870 2024-05-28T06:14:43.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.902+0000 7f4b96a00640 1 --2- 172.21.15.96:0/3139617546 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4b7406d5e0 0x7f4b7406faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:43.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.903+0000 7f4b96a00640 1 --2- 172.21.15.96:0/3139617546 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4b7406d5e0 0x7f4b7406faa0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f4b980b8220 tx=0x7f4b880073c0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:43.906 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:43.906+0000 7f4b8f400640 1 -- 172.21.15.96:0/3139617546 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4b8005a8d0 con 0x7f4b980cf870 2024-05-28T06:14:44.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:44.184+0000 7f4b9cc00640 1 -- 172.21.15.96:0/3139617546 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgressReporter.test_progress_drops_when_new_jobs_are_added"]} v 0) -- 0x7f4b980ba500 con 0x7f4b980cf870 2024-05-28T06:14:44.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:44.779+0000 7f4b8f400640 1 -- 172.21.15.96:0/3139617546 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgressReporter.test_progress_drops_when_new_jobs_are_added"]}]=0 v461) ==== 177+0+0 (secure 0 0 0) 0x7f4b8005a1d0 con 0x7f4b980cf870 2024-05-28T06:14:44.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:44.783+0000 7f4b9cc00640 1 -- 172.21.15.96:0/3139617546 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4b7406d5e0 msgr2=0x7f4b7406faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:44.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:44.783+0000 7f4b9cc00640 1 --2- 172.21.15.96:0/3139617546 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4b7406d5e0 0x7f4b7406faa0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f4b980b8220 tx=0x7f4b880073c0 comp rx=0 tx=0).stop 2024-05-28T06:14:44.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:44.783+0000 7f4b9cc00640 1 -- 172.21.15.96:0/3139617546 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4b980cf870 msgr2=0x7f4b98160f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:44.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:44.783+0000 7f4b9cc00640 1 --2- 172.21.15.96:0/3139617546 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4b980cf870 0x7f4b98160f30 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f4b8000c7f0 tx=0x7f4b8000ccb0 comp rx=0 tx=0).stop 2024-05-28T06:14:44.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:44.783+0000 7f4b9cc00640 1 -- 172.21.15.96:0/3139617546 shutdown_connections 2024-05-28T06:14:44.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:44.783+0000 7f4b9cc00640 1 --2- 172.21.15.96:0/3139617546 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4b7406d5e0 0x7f4b7406faa0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:44.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:44.783+0000 7f4b9cc00640 1 --2- 172.21.15.96:0/3139617546 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4b9815e870 0x7f4b980cd820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:44.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:44.783+0000 7f4b9cc00640 1 --2- 172.21.15.96:0/3139617546 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b980d14d0 0x7f4b980b8680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:44.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:44.783+0000 7f4b9cc00640 1 --2- 172.21.15.96:0/3139617546 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4b980cf870 0x7f4b98160f30 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:44.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:44.783+0000 7f4b9cc00640 1 -- 172.21.15.96:0/3139617546 >> 172.21.15.96:0/3139617546 conn(0x7f4b9808bfc0 msgr2=0x7f4b9806e460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:44.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:44.783+0000 7f4b9cc00640 1 -- 172.21.15.96:0/3139617546 shutdown_connections 2024-05-28T06:14:44.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:44.783+0000 7f4b9cc00640 1 -- 172.21.15.96:0/3139617546 wait complete. 2024-05-28T06:14:44.804 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-05-28T06:14:45.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.005+0000 7fad34600640 1 Processor -- start 2024-05-28T06:14:45.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.005+0000 7fad34600640 1 -- start start 2024-05-28T06:14:45.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.007+0000 7fad34600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fad2c0b32c0 0x7fad2c0b3750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:45.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.007+0000 7fad34600640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad2c0d0060 con 0x7fad2c0d0640 2024-05-28T06:14:45.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.007+0000 7fad34600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad2c15cb10 con 0x7fad2c15d5d0 2024-05-28T06:14:45.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.007+0000 7fad34600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fad2c0adbb0 con 0x7fad2c0b32c0 2024-05-28T06:14:45.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.007+0000 7fad31c00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fad2c0d0640 0x7fad2c0d0a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:56520/0 (socket says 172.21.15.96:56520) 2024-05-28T06:14:45.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.007+0000 7fad31c00640 1 -- 172.21.15.96:0/2694027815 learned_addr learned my addr 172.21.15.96:0/2694027815 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:14:45.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.007+0000 7fad32600640 1 --2- 172.21.15.96:0/2694027815 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fad2c0b32c0 0x7fad2c0b3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:45.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.008+0000 7fad2be00640 1 -- 172.21.15.96:0/2694027815 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1651849403 0 0) 0x7fad2c0d0060 con 0x7fad2c0d0640 2024-05-28T06:14:45.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.008+0000 7fad2be00640 1 -- 172.21.15.96:0/2694027815 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fad14003520 con 0x7fad2c0d0640 2024-05-28T06:14:45.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.008+0000 7fad2be00640 1 -- 172.21.15.96:0/2694027815 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2175338933 0 0) 0x7fad14003520 con 0x7fad2c0d0640 2024-05-28T06:14:45.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.008+0000 7fad2be00640 1 -- 172.21.15.96:0/2694027815 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fad2c0b32c0 msgr2=0x7fad2c0b3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:45.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.008+0000 7fad2be00640 1 --2- 172.21.15.96:0/2694027815 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fad2c0b32c0 0x7fad2c0b3750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:45.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.008+0000 7fad2be00640 1 -- 172.21.15.96:0/2694027815 >> v1:172.21.15.123:6789/0 conn(0x7fad2c15d5d0 legacy=0x7fad2c0cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:45.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.008+0000 7fad2be00640 1 -- 172.21.15.96:0/2694027815 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad2c0ad5f0 con 0x7fad2c0d0640 2024-05-28T06:14:45.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.008+0000 7fad2be00640 1 -- 172.21.15.96:0/2694027815 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7fad20001670 con 0x7fad2c0d0640 2024-05-28T06:14:45.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.008+0000 7fad2be00640 1 -- 172.21.15.96:0/2694027815 >> v1:172.21.15.96:6789/0 conn(0x7fad2c0d0640 legacy=0x7fad2c0d0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:45.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.008+0000 7fad2be00640 1 --2- 172.21.15.96:0/2694027815 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad140039e0 0x7fad14003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:45.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.008+0000 7fad2be00640 1 --2- 172.21.15.96:0/2694027815 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fad2c0b32c0 0x7fad14004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:45.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.008+0000 7fad2be00640 1 --2- 172.21.15.96:0/2694027815 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fad14004d40 0x7fad140051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:45.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.008+0000 7fad2be00640 1 -- 172.21.15.96:0/2694027815 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fad1400ba70 con 0x7fad140039e0 2024-05-28T06:14:45.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.008+0000 7fad2be00640 1 -- 172.21.15.96:0/2694027815 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fad1400bc20 con 0x7fad14004d40 2024-05-28T06:14:45.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.008+0000 7fad2be00640 1 -- 172.21.15.96:0/2694027815 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fad1400be20 con 0x7fad2c0b32c0 2024-05-28T06:14:45.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.009+0000 7fad31c00640 1 --2- 172.21.15.96:0/2694027815 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad140039e0 0x7fad14003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:45.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.009+0000 7fad31200640 1 --2- 172.21.15.96:0/2694027815 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fad2c0b32c0 0x7fad14004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:45.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.009+0000 7fad32600640 1 --2- 172.21.15.96:0/2694027815 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fad14004d40 0x7fad140051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:45.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.009+0000 7fad31c00640 1 -- 172.21.15.96:0/2694027815 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fad2c0b32c0 msgr2=0x7fad14004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:45.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.009+0000 7fad31c00640 1 --2- 172.21.15.96:0/2694027815 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fad2c0b32c0 0x7fad14004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:45.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.009+0000 7fad31c00640 1 -- 172.21.15.96:0/2694027815 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fad14004d40 msgr2=0x7fad140051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:45.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.009+0000 7fad31c00640 1 --2- 172.21.15.96:0/2694027815 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fad14004d40 0x7fad140051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:45.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.009+0000 7fad31c00640 1 -- 172.21.15.96:0/2694027815 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad14003520 con 0x7fad140039e0 2024-05-28T06:14:45.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.011+0000 7fad31c00640 1 --2- 172.21.15.96:0/2694027815 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad140039e0 0x7fad14003e00 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7fad2000c0e0 tx=0x7fad14005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef872cd3e907391d server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:45.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.012+0000 7fad2be00640 1 -- 172.21.15.96:0/2694027815 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad20007c40 con 0x7fad140039e0 2024-05-28T06:14:45.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.012+0000 7fad2be00640 1 -- 172.21.15.96:0/2694027815 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad20005af0 con 0x7fad140039e0 2024-05-28T06:14:45.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.012+0000 7fad2be00640 1 -- 172.21.15.96:0/2694027815 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad20007070 con 0x7fad140039e0 2024-05-28T06:14:45.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.012+0000 7fad34600640 1 -- 172.21.15.96:0/2694027815 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad140039e0 msgr2=0x7fad14003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:45.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.012+0000 7fad34600640 1 --2- 172.21.15.96:0/2694027815 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad140039e0 0x7fad14003e00 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7fad2000c0e0 tx=0x7fad14005830 comp rx=0 tx=0).stop 2024-05-28T06:14:45.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.012+0000 7fad32600640 1 -- 172.21.15.96:0/2694027815 reap_dead start 2024-05-28T06:14:45.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.012+0000 7fad34600640 1 -- 172.21.15.96:0/2694027815 shutdown_connections 2024-05-28T06:14:45.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.012+0000 7fad34600640 1 -- 172.21.15.96:0/2694027815 >> 172.21.15.96:0/2694027815 conn(0x7fad2c06bd20 msgr2=0x7fad2c06c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:45.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.012+0000 7fad34600640 1 -- 172.21.15.96:0/2694027815 shutdown_connections 2024-05-28T06:14:45.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.012+0000 7fad34600640 1 -- 172.21.15.96:0/2694027815 wait complete. 2024-05-28T06:14:45.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.014+0000 7fad34600640 1 Processor -- start 2024-05-28T06:14:45.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.014+0000 7fad34600640 1 -- start start 2024-05-28T06:14:45.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.015+0000 7fad34600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad140039e0 0x7fad2c073290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:45.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.015+0000 7fad34600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fad2c15e120 0x7fad2c145f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:45.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.015+0000 7fad34600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fad2c14ab60 0x7fad2c14afe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:45.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.015+0000 7fad34600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fad2c15d0b0 con 0x7fad140039e0 2024-05-28T06:14:45.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.015+0000 7fad34600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fad2c15cdd0 con 0x7fad2c15e120 2024-05-28T06:14:45.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.015+0000 7fad34600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fad2c15da50 con 0x7fad2c14ab60 2024-05-28T06:14:45.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.016+0000 7fad31c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad140039e0 0x7fad2c073290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:45.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.016+0000 7fad31c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad140039e0 0x7fad2c073290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35554/0 (socket says 172.21.15.96:35554) 2024-05-28T06:14:45.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.016+0000 7fad31c00640 1 -- 172.21.15.96:0/1239059115 learned_addr learned my addr 172.21.15.96:0/1239059115 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:45.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.016+0000 7fad31200640 1 --2- 172.21.15.96:0/1239059115 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fad2c15e120 0x7fad2c145f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:45.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.016+0000 7fad32600640 1 --2- 172.21.15.96:0/1239059115 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fad2c14ab60 0x7fad2c14afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:45.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.016+0000 7fad31c00640 1 -- 172.21.15.96:0/1239059115 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fad2c14ab60 msgr2=0x7fad2c14afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:45.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.016+0000 7fad31c00640 1 --2- 172.21.15.96:0/1239059115 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fad2c14ab60 0x7fad2c14afe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:45.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.016+0000 7fad31c00640 1 -- 172.21.15.96:0/1239059115 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fad2c15e120 msgr2=0x7fad2c145f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:45.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.016+0000 7fad31c00640 1 --2- 172.21.15.96:0/1239059115 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fad2c15e120 0x7fad2c145f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:45.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.016+0000 7fad31c00640 1 -- 172.21.15.96:0/1239059115 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad2c0b0ae0 con 0x7fad140039e0 2024-05-28T06:14:45.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.017+0000 7fad31c00640 1 --2- 172.21.15.96:0/1239059115 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad140039e0 0x7fad2c073290 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7fad2000c200 tx=0x7fad2c0ad7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:45.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.017+0000 7fad29600640 1 -- 172.21.15.96:0/1239059115 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad20043070 con 0x7fad140039e0 2024-05-28T06:14:45.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.017+0000 7fad29600640 1 -- 172.21.15.96:0/1239059115 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad20038040 con 0x7fad140039e0 2024-05-28T06:14:45.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.017+0000 7fad29600640 1 -- 172.21.15.96:0/1239059115 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad20007070 con 0x7fad140039e0 2024-05-28T06:14:45.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.017+0000 7fad34600640 1 -- 172.21.15.96:0/1239059115 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad2c0b0d80 con 0x7fad140039e0 2024-05-28T06:14:45.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.017+0000 7fad34600640 1 -- 172.21.15.96:0/1239059115 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad2c0b1340 con 0x7fad140039e0 2024-05-28T06:14:45.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.018+0000 7fad34600640 1 -- 172.21.15.96:0/1239059115 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad2c0689e0 con 0x7fad140039e0 2024-05-28T06:14:45.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.019+0000 7fad29600640 1 -- 172.21.15.96:0/1239059115 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 22) ==== 92213+0+0 (secure 0 0 0) 0x7fad20007210 con 0x7fad140039e0 2024-05-28T06:14:45.022 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.020+0000 7fad29600640 1 --2- 172.21.15.96:0/1239059115 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fad0c06d5e0 0x7fad0c06faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:45.022 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.021+0000 7fad29600640 1 -- 172.21.15.96:0/1239059115 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 4971+0+0 (secure 0 0 0) 0x7fad200aa480 con 0x7fad140039e0 2024-05-28T06:14:45.022 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.021+0000 7fad31200640 1 --2- 172.21.15.96:0/1239059115 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fad0c06d5e0 0x7fad0c06faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:45.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.022+0000 7fad31200640 1 --2- 172.21.15.96:0/1239059115 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fad0c06d5e0 0x7fad0c06faa0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fad18000c00 tx=0x7fad18009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:45.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.025+0000 7fad29600640 1 -- 172.21.15.96:0/1239059115 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fad20079a20 con 0x7fad140039e0 2024-05-28T06:14:45.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.207+0000 7fad34600640 1 -- 172.21.15.96:0/1239059115 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7fad2c073070 con 0x7fad140039e0 2024-05-28T06:14:45.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.214+0000 7fad29600640 1 -- 172.21.15.96:0/1239059115 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v18)=0 v18) ==== 122+0+0 (secure 0 0 0) 0x7fad2007dd20 con 0x7fad140039e0 2024-05-28T06:14:45.218 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.216+0000 7fad34600640 1 -- 172.21.15.96:0/1239059115 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fad0c06d5e0 msgr2=0x7fad0c06faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:45.218 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.216+0000 7fad34600640 1 --2- 172.21.15.96:0/1239059115 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fad0c06d5e0 0x7fad0c06faa0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fad18000c00 tx=0x7fad18009040 comp rx=0 tx=0).stop 2024-05-28T06:14:45.218 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.216+0000 7fad34600640 1 -- 172.21.15.96:0/1239059115 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad140039e0 msgr2=0x7fad2c073290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:45.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.216+0000 7fad34600640 1 --2- 172.21.15.96:0/1239059115 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad140039e0 0x7fad2c073290 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7fad2000c200 tx=0x7fad2c0ad7c0 comp rx=0 tx=0).stop 2024-05-28T06:14:45.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.216+0000 7fad34600640 1 -- 172.21.15.96:0/1239059115 shutdown_connections 2024-05-28T06:14:45.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.216+0000 7fad34600640 1 --2- 172.21.15.96:0/1239059115 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fad0c06d5e0 0x7fad0c06faa0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:45.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.217+0000 7fad34600640 1 --2- 172.21.15.96:0/1239059115 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fad2c14ab60 0x7fad2c14afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:45.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.217+0000 7fad34600640 1 --2- 172.21.15.96:0/1239059115 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fad2c15e120 0x7fad2c145f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:45.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.217+0000 7fad34600640 1 --2- 172.21.15.96:0/1239059115 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad140039e0 0x7fad2c073290 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:45.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.217+0000 7fad34600640 1 -- 172.21.15.96:0/1239059115 >> 172.21.15.96:0/1239059115 conn(0x7fad2c06bd20 msgr2=0x7fad2c15e6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:45.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.217+0000 7fad34600640 1 -- 172.21.15.96:0/1239059115 shutdown_connections 2024-05-28T06:14:45.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.217+0000 7fad34600640 1 -- 172.21.15.96:0/1239059115 wait complete. 2024-05-28T06:14:45.237 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-05-28T06:14:45.238 DEBUG:teuthology.orchestra.run.smithi096:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-05-28T06:14:45.363 DEBUG:teuthology.orchestra.run.smithi123:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-05-28T06:14:45.473 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-05-28T06:14:45.473 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:14:45.474 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:14:45.532 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052731,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248996k,nr_inodes=812249,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-28T06:14:45.532 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-05-28T06:14:45.532 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:14:45.587 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052731,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248996k,nr_inodes=812249,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-05-28T06:14:45.587 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:14:45.587 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:14:45.643 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052731,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248996k,nr_inodes=812249,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-28T06:14:45.643 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-05-28T06:14:45.643 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:14:45.701 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052731,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248996k,nr_inodes=812249,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-05-28T06:14:45.702 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-05-28T06:14:45.702 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-28T06:14:45.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.909+0000 7f3e01600640 1 Processor -- start 2024-05-28T06:14:45.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.909+0000 7f3e01600640 1 -- start start 2024-05-28T06:14:45.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.911+0000 7f3e01600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f3dfc0d04a0 0x7f3dfc0d0930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:45.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.911+0000 7f3e01600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f3dfc0b3460 0x7f3dfc0b3880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:45.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.911+0000 7f3e01600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3dfc0ced40 con 0x7f3dfc0b4b60 2024-05-28T06:14:45.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.911+0000 7f3e01600640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f3dfc15fbb0 con 0x7f3dfc0b3460 2024-05-28T06:14:45.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.911+0000 7f3e01600640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f3dfc0cefc0 con 0x7f3dfc0d04a0 2024-05-28T06:14:45.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.911+0000 7f3dfb400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f3dfc0b3460 0x7f3dfc0b3880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:45.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.911+0000 7f3dfb400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f3dfc0b3460 0x7f3dfc0b3880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35558/0 (socket says 172.21.15.96:35558) 2024-05-28T06:14:45.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.911+0000 7f3dfb400640 1 -- 172.21.15.96:0/475249960 learned_addr learned my addr 172.21.15.96:0/475249960 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:45.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.911+0000 7f3dfb400640 1 -- 172.21.15.96:0/475249960 >> v2:172.21.15.123:3300/0 conn(0x7f3dfc0d04a0 msgr2=0x7f3dfc0d0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:45.914 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.911+0000 7f3dfb400640 1 --2- 172.21.15.96:0/475249960 >> v2:172.21.15.123:3300/0 conn(0x7f3dfc0d04a0 0x7f3dfc0d0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:45.914 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.911+0000 7f3dfb400640 1 -- 172.21.15.96:0/475249960 >> v1:172.21.15.123:6789/0 conn(0x7f3dfc0b4b60 legacy=0x7f3dfc0cfaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:45.914 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.911+0000 7f3dfb400640 1 -- 172.21.15.96:0/475249960 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dfc0cf2a0 con 0x7f3dfc0b3460 2024-05-28T06:14:45.914 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.912+0000 7f3dfb400640 1 --2- 172.21.15.96:0/475249960 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3dfc0b3460 0x7f3dfc0b3880 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f3dfc0d1180 tx=0x7f3df402a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60a0a4682292696c server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:45.914 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.913+0000 7f3df9600640 1 -- 172.21.15.96:0/475249960 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3df4036070 con 0x7f3dfc0b3460 2024-05-28T06:14:45.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.913+0000 7f3df9600640 1 -- 172.21.15.96:0/475249960 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3df4031600 con 0x7f3dfc0b3460 2024-05-28T06:14:45.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.913+0000 7f3df9600640 1 -- 172.21.15.96:0/475249960 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3df40319d0 con 0x7f3dfc0b3460 2024-05-28T06:14:45.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.913+0000 7f3e01600640 1 -- 172.21.15.96:0/475249960 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3dfc0b3460 msgr2=0x7f3dfc0b3880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:45.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.913+0000 7f3e01600640 1 --2- 172.21.15.96:0/475249960 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3dfc0b3460 0x7f3dfc0b3880 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f3dfc0d1180 tx=0x7f3df402a6f0 comp rx=0 tx=0).stop 2024-05-28T06:14:45.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.914+0000 7f3e01600640 1 -- 172.21.15.96:0/475249960 shutdown_connections 2024-05-28T06:14:45.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.914+0000 7f3e01600640 1 --2- 172.21.15.96:0/475249960 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3dfc0b3460 0x7f3dfc0b3880 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:45.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.914+0000 7f3e01600640 1 --2- 172.21.15.96:0/475249960 >> v2:172.21.15.123:3300/0 conn(0x7f3dfc0d04a0 0x7f3dfc0d0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:45.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.914+0000 7f3e01600640 1 -- 172.21.15.96:0/475249960 >> 172.21.15.96:0/475249960 conn(0x7f3dfc06bae0 msgr2=0x7f3dfc06bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:45.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.914+0000 7f3e01600640 1 -- 172.21.15.96:0/475249960 shutdown_connections 2024-05-28T06:14:45.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.914+0000 7f3e01600640 1 -- 172.21.15.96:0/475249960 wait complete. 2024-05-28T06:14:45.917 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.915+0000 7f3e01600640 1 Processor -- start 2024-05-28T06:14:45.917 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.916+0000 7f3e01600640 1 -- start start 2024-05-28T06:14:45.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.916+0000 7f3e01600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3dfc0b3460 0x7f3dfc099a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:45.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.916+0000 7f3e01600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3dfc0b4b60 0x7f3dfc159e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:45.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.917+0000 7f3e01600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3dfc0d04a0 0x7f3dfc161320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:45.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.917+0000 7f3e01600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f3dfc0b3f40 con 0x7f3dfc0b3460 2024-05-28T06:14:45.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.917+0000 7f3e01600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f3dfc0b3dc0 con 0x7f3dfc0b4b60 2024-05-28T06:14:45.919 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.917+0000 7f3e01600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f3dfc15ede0 con 0x7f3dfc0d04a0 2024-05-28T06:14:45.919 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.917+0000 7f3dfaa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3dfc0b3460 0x7f3dfc099a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:45.919 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.917+0000 7f3dfaa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3dfc0b3460 0x7f3dfc099a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35568/0 (socket says 172.21.15.96:35568) 2024-05-28T06:14:45.919 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.917+0000 7f3dfaa00640 1 -- 172.21.15.96:0/2944092811 learned_addr learned my addr 172.21.15.96:0/2944092811 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:45.920 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.917+0000 7f3dfaa00640 1 -- 172.21.15.96:0/2944092811 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3dfc0d04a0 msgr2=0x7f3dfc161320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:45.920 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.917+0000 7f3dfaa00640 1 --2- 172.21.15.96:0/2944092811 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3dfc0d04a0 0x7f3dfc161320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:45.920 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.917+0000 7f3dfa000640 1 --2- 172.21.15.96:0/2944092811 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3dfc0b4b60 0x7f3dfc159e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:45.920 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.917+0000 7f3dfaa00640 1 -- 172.21.15.96:0/2944092811 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3dfc0b4b60 msgr2=0x7f3dfc159e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:45.920 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.917+0000 7f3dfaa00640 1 --2- 172.21.15.96:0/2944092811 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3dfc0b4b60 0x7f3dfc159e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:45.920 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.917+0000 7f3dfaa00640 1 -- 172.21.15.96:0/2944092811 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dfc1f5690 con 0x7f3dfc0b3460 2024-05-28T06:14:45.921 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.917+0000 7f3dfa000640 1 --2- 172.21.15.96:0/2944092811 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3dfc0b4b60 0x7f3dfc159e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:14:45.921 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.918+0000 7f3dfaa00640 1 --2- 172.21.15.96:0/2944092811 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3dfc0b3460 0x7f3dfc099a10 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f3dec00c410 tx=0x7f3dec00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:45.921 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.918+0000 7f3df2a00640 1 -- 172.21.15.96:0/2944092811 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dec017070 con 0x7f3dfc0b3460 2024-05-28T06:14:45.921 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.918+0000 7f3df2a00640 1 -- 172.21.15.96:0/2944092811 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3dec0035f0 con 0x7f3dfc0b3460 2024-05-28T06:14:45.921 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.918+0000 7f3df2a00640 1 -- 172.21.15.96:0/2944092811 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dec003a20 con 0x7f3dfc0b3460 2024-05-28T06:14:45.921 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.918+0000 7f3e01600640 1 -- 172.21.15.96:0/2944092811 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3dfc1f48d0 con 0x7f3dfc0b3460 2024-05-28T06:14:45.921 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.918+0000 7f3e01600640 1 -- 172.21.15.96:0/2944092811 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3dfc1f4e90 con 0x7f3dfc0b3460 2024-05-28T06:14:45.921 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.919+0000 7f3e01600640 1 -- 172.21.15.96:0/2944092811 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3dfc0689e0 con 0x7f3dfc0b3460 2024-05-28T06:14:45.922 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.920+0000 7f3df2a00640 1 -- 172.21.15.96:0/2944092811 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 22) ==== 92213+0+0 (secure 0 0 0) 0x7f3dec003bc0 con 0x7f3dfc0b3460 2024-05-28T06:14:45.922 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.921+0000 7f3df2a00640 1 --2- 172.21.15.96:0/2944092811 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3ddc06d5e0 0x7f3ddc06faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:45.922 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.921+0000 7f3df2a00640 1 -- 172.21.15.96:0/2944092811 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 4971+0+0 (secure 0 0 0) 0x7f3dec0898b0 con 0x7f3dfc0b3460 2024-05-28T06:14:45.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.922+0000 7f3dfa000640 1 --2- 172.21.15.96:0/2944092811 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3ddc06d5e0 0x7f3ddc06faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:45.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.923+0000 7f3dfa000640 1 --2- 172.21.15.96:0/2944092811 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3ddc06d5e0 0x7f3ddc06faa0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f3dfc15aa70 tx=0x7f3de8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:45.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:45.925+0000 7f3df2a00640 1 -- 172.21.15.96:0/2944092811 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3dec059730 con 0x7f3dfc0b3460 2024-05-28T06:14:46.174 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.172+0000 7f3e01600640 1 -- 172.21.15.96:0/2944092811 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3dfc073c80 con 0x7f3dfc0b3460 2024-05-28T06:14:46.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.173+0000 7f3df2a00640 1 -- 172.21.15.96:0/2944092811 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+3872 (secure 0 0 0) 0x7f3dec0010c0 con 0x7f3dfc0b3460 2024-05-28T06:14:46.175 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:14:46.175 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":37,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5015,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/1869890774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":1869890774},{"type":"v1","addr":"172.21.15.123:6837","nonce":1869890774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":5386,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/873923319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":873923319},{"type":"v1","addr":"172.21.15.123:6835","nonce":873923319}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":6486,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6834/1041571401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":1041571401},{"type":"v1","addr":"172.21.15.96:6834","nonce":1041571401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":7044,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/2424522216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6833","nonce":2424522216},{"type":"v1","addr":"172.21.15.96:6835","nonce":2424522216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37}],"filesystems":[]} 2024-05-28T06:14:46.175 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 37 2024-05-28T06:14:46.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.176+0000 7f3e01600640 1 -- 172.21.15.96:0/2944092811 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3ddc06d5e0 msgr2=0x7f3ddc06faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:46.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.176+0000 7f3e01600640 1 --2- 172.21.15.96:0/2944092811 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3ddc06d5e0 0x7f3ddc06faa0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f3dfc15aa70 tx=0x7f3de8008040 comp rx=0 tx=0).stop 2024-05-28T06:14:46.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.176+0000 7f3e01600640 1 -- 172.21.15.96:0/2944092811 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3dfc0b3460 msgr2=0x7f3dfc099a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:46.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.176+0000 7f3e01600640 1 --2- 172.21.15.96:0/2944092811 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3dfc0b3460 0x7f3dfc099a10 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f3dec00c410 tx=0x7f3dec00c8d0 comp rx=0 tx=0).stop 2024-05-28T06:14:46.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.176+0000 7f3e01600640 1 -- 172.21.15.96:0/2944092811 shutdown_connections 2024-05-28T06:14:46.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.176+0000 7f3e01600640 1 --2- 172.21.15.96:0/2944092811 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3ddc06d5e0 0x7f3ddc06faa0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:46.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.176+0000 7f3e01600640 1 --2- 172.21.15.96:0/2944092811 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3dfc0d04a0 0x7f3dfc161320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:46.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.176+0000 7f3e01600640 1 --2- 172.21.15.96:0/2944092811 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3dfc0b4b60 0x7f3dfc159e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:46.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.176+0000 7f3e01600640 1 --2- 172.21.15.96:0/2944092811 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3dfc0b3460 0x7f3dfc099a10 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:46.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.176+0000 7f3e01600640 1 -- 172.21.15.96:0/2944092811 >> 172.21.15.96:0/2944092811 conn(0x7f3dfc06bae0 msgr2=0x7f3dfc0b0e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:46.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.176+0000 7f3e01600640 1 -- 172.21.15.96:0/2944092811 shutdown_connections 2024-05-28T06:14:46.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.176+0000 7f3e01600640 1 -- 172.21.15.96:0/2944092811 wait complete. 2024-05-28T06:14:46.197 INFO:tasks.ceph.mds.a:Restarting daemon 2024-05-28T06:14:46.197 INFO:tasks.ceph.mds.a:Stopping old one... 2024-05-28T06:14:46.197 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-05-28T06:14:46.197 INFO:teuthology.orchestra.run:waiting for 300 2024-05-28T06:14:46.197 INFO:tasks.ceph.mds.c:Restarting daemon 2024-05-28T06:14:46.197 INFO:tasks.ceph.mds.c:Stopping old one... 2024-05-28T06:14:46.198 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-05-28T06:14:46.198 INFO:teuthology.orchestra.run:waiting for 300 2024-05-28T06:14:46.198 INFO:tasks.ceph.mds.b:Restarting daemon 2024-05-28T06:14:46.198 INFO:tasks.ceph.mds.b:Stopping old one... 2024-05-28T06:14:46.198 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-05-28T06:14:46.198 INFO:teuthology.orchestra.run:waiting for 300 2024-05-28T06:14:46.198 INFO:tasks.ceph.mds.d:Restarting daemon 2024-05-28T06:14:46.199 INFO:tasks.ceph.mds.d:Stopping old one... 2024-05-28T06:14:46.199 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-05-28T06:14:46.199 INFO:teuthology.orchestra.run:waiting for 300 2024-05-28T06:14:46.215 INFO:tasks.ceph.mds.c:Stopped 2024-05-28T06:14:46.215 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-05-28T06:14:46.218 INFO:tasks.ceph.mds.a:Stopped 2024-05-28T06:14:46.218 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-05-28T06:14:46.220 INFO:tasks.ceph.mds.d:Stopped 2024-05-28T06:14:46.220 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-05-28T06:14:46.224 INFO:tasks.ceph.mds.b:Stopped 2024-05-28T06:14:46.224 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-05-28T06:14:46.264 INFO:tasks.ceph.mds.c:Started 2024-05-28T06:14:46.265 DEBUG:teuthology.parallel:result is None 2024-05-28T06:14:46.271 INFO:tasks.ceph.mds.d:Started 2024-05-28T06:14:46.271 DEBUG:teuthology.parallel:result is None 2024-05-28T06:14:46.271 INFO:tasks.ceph.mds.a:Started 2024-05-28T06:14:46.272 DEBUG:teuthology.parallel:result is None 2024-05-28T06:14:46.277 INFO:tasks.ceph.mds.b:Started 2024-05-28T06:14:46.277 DEBUG:teuthology.parallel:result is None 2024-05-28T06:14:46.277 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-05-28T06:14:46.413 INFO:tasks.ceph.mds.c.smithi096.stdout:starting mds.c at 2024-05-28T06:14:46.414 INFO:tasks.ceph.mds.a.smithi096.stdout:starting mds.a at 2024-05-28T06:14:46.423 INFO:tasks.ceph.mds.d.smithi123.stdout:starting mds.d at 2024-05-28T06:14:46.424 INFO:tasks.ceph.mds.b.smithi123.stdout:starting mds.b at 2024-05-28T06:14:46.452 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.450+0000 7f6be1600640 1 Processor -- start 2024-05-28T06:14:46.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.451+0000 7f6be1600640 1 -- start start 2024-05-28T06:14:46.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.452+0000 7f6be1600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f6bdc0b3b70 0x7f6bdc0d0930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:46.454 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.452+0000 7f6be1600640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6bdc0ced40 con 0x7f6bdc0b4b60 2024-05-28T06:14:46.454 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.452+0000 7f6be1600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6bdc0b3f90 con 0x7f6bdc0d04a0 2024-05-28T06:14:46.454 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.452+0000 7f6be1600640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f6bdc15fbb0 con 0x7f6bdc0b3b70 2024-05-28T06:14:46.454 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.452+0000 7f6bdaa00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f6bdc0b4b60 0x7f6bdc0cfaa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:56546/0 (socket says 172.21.15.96:56546) 2024-05-28T06:14:46.454 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.452+0000 7f6bdaa00640 1 -- 172.21.15.96:0/2079798472 learned_addr learned my addr 172.21.15.96:0/2079798472 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:14:46.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.452+0000 7f6bd9600640 1 -- 172.21.15.96:0/2079798472 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3659597872 0 0) 0x7f6bcc001110 con 0x7f6bdc0b4b60 2024-05-28T06:14:46.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.452+0000 7f6bd9600640 1 -- 172.21.15.96:0/2079798472 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6bc0003520 con 0x7f6bdc0b4b60 2024-05-28T06:14:46.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.453+0000 7f6bdb400640 1 --2- 172.21.15.96:0/2079798472 >> v2:172.21.15.123:3300/0 conn(0x7f6bdc0b3b70 0x7f6bdc0d0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:46.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.453+0000 7f6bd9600640 1 -- 172.21.15.96:0/2079798472 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1627659999 0 0) 0x7f6bc0003520 con 0x7f6bdc0b4b60 2024-05-28T06:14:46.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.453+0000 7f6bd9600640 1 -- 172.21.15.96:0/2079798472 >> v2:172.21.15.123:3300/0 conn(0x7f6bdc0b3b70 msgr2=0x7f6bdc0d0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:46.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.453+0000 7f6bd9600640 1 --2- 172.21.15.96:0/2079798472 >> v2:172.21.15.123:3300/0 conn(0x7f6bdc0b3b70 0x7f6bdc0d0930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:46.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.453+0000 7f6bd9600640 1 -- 172.21.15.96:0/2079798472 >> v1:172.21.15.123:6789/0 conn(0x7f6bdc0d04a0 legacy=0x7f6bdc0b3460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:46.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.453+0000 7f6bd9600640 1 -- 172.21.15.96:0/2079798472 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bdc0cefc0 con 0x7f6bdc0b4b60 2024-05-28T06:14:46.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.453+0000 7f6bd9600640 1 -- 172.21.15.96:0/2079798472 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f6bcc001840 con 0x7f6bdc0b4b60 2024-05-28T06:14:46.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.453+0000 7f6bd9600640 1 -- 172.21.15.96:0/2079798472 >> v1:172.21.15.96:6789/0 conn(0x7f6bdc0b4b60 legacy=0x7f6bdc0cfaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:46.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.453+0000 7f6bd9600640 1 --2- 172.21.15.96:0/2079798472 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6bc00039b0 0x7f6bc0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:46.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.453+0000 7f6bd9600640 1 --2- 172.21.15.96:0/2079798472 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6bc0004800 0x7f6bc0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:46.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.453+0000 7f6bd9600640 1 --2- 172.21.15.96:0/2079798472 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6bc00051c0 0x7f6bc000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:46.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.453+0000 7f6bd9600640 1 -- 172.21.15.96:0/2079798472 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f6bdc15fbb0 con 0x7f6bc00039b0 2024-05-28T06:14:46.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.453+0000 7f6bd9600640 1 -- 172.21.15.96:0/2079798472 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f6bc000c060 con 0x7f6bc0004800 2024-05-28T06:14:46.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.453+0000 7f6bd9600640 1 -- 172.21.15.96:0/2079798472 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f6bc000c260 con 0x7f6bc00051c0 2024-05-28T06:14:46.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.453+0000 7f6bdaa00640 1 --2- 172.21.15.96:0/2079798472 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6bc00039b0 0x7f6bc0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:46.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.454+0000 7f6bda000640 1 --2- 172.21.15.96:0/2079798472 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6bc0004800 0x7f6bc0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:46.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.454+0000 7f6bdb400640 1 --2- 172.21.15.96:0/2079798472 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6bc00051c0 0x7f6bc000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:46.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.454+0000 7f6bdaa00640 1 -- 172.21.15.96:0/2079798472 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6bc00051c0 msgr2=0x7f6bc000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:46.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.454+0000 7f6bdaa00640 1 --2- 172.21.15.96:0/2079798472 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6bc00051c0 0x7f6bc000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:46.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.454+0000 7f6bdaa00640 1 -- 172.21.15.96:0/2079798472 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6bc0004800 msgr2=0x7f6bc0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:46.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.454+0000 7f6bdaa00640 1 --2- 172.21.15.96:0/2079798472 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6bc0004800 0x7f6bc0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:46.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.454+0000 7f6bdaa00640 1 -- 172.21.15.96:0/2079798472 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bc0003520 con 0x7f6bc00039b0 2024-05-28T06:14:46.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.454+0000 7f6bdb400640 1 --2- 172.21.15.96:0/2079798472 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6bc00051c0 0x7f6bc000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-28T06:14:46.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.454+0000 7f6bdb400640 1 -- 172.21.15.96:0/2079798472 reap_dead start 2024-05-28T06:14:46.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.455+0000 7f6bdaa00640 1 --2- 172.21.15.96:0/2079798472 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6bc00039b0 0x7f6bc0003dd0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc00c4c0 tx=0x7f6bc0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f441495e03c79c server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:46.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.455+0000 7f6bd9600640 1 -- 172.21.15.96:0/2079798472 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bcc00c0e0 con 0x7f6bc00039b0 2024-05-28T06:14:46.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.455+0000 7f6bd9600640 1 -- 172.21.15.96:0/2079798472 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6bcc034070 con 0x7f6bc00039b0 2024-05-28T06:14:46.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.455+0000 7f6bd9600640 1 -- 172.21.15.96:0/2079798472 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bcc005d70 con 0x7f6bc00039b0 2024-05-28T06:14:46.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.455+0000 7f6be1600640 1 -- 172.21.15.96:0/2079798472 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6bc00039b0 msgr2=0x7f6bc0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:46.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.455+0000 7f6be1600640 1 --2- 172.21.15.96:0/2079798472 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6bc00039b0 0x7f6bc0003dd0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc00c4c0 tx=0x7f6bc0005830 comp rx=0 tx=0).stop 2024-05-28T06:14:46.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.456+0000 7f6be1600640 1 -- 172.21.15.96:0/2079798472 shutdown_connections 2024-05-28T06:14:46.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.456+0000 7f6be1600640 1 --2- 172.21.15.96:0/2079798472 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6bc00039b0 0x7f6bc0003dd0 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:46.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.456+0000 7f6be1600640 1 -- 172.21.15.96:0/2079798472 >> 172.21.15.96:0/2079798472 conn(0x7f6bdc06bae0 msgr2=0x7f6bdc06bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:46.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.456+0000 7f6be1600640 1 -- 172.21.15.96:0/2079798472 shutdown_connections 2024-05-28T06:14:46.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.456+0000 7f6be1600640 1 -- 172.21.15.96:0/2079798472 wait complete. 2024-05-28T06:14:46.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.457+0000 7f6be1600640 1 Processor -- start 2024-05-28T06:14:46.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.458+0000 7f6be1600640 1 -- start start 2024-05-28T06:14:46.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.458+0000 7f6be1600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6bc00039b0 0x7f6bdc09d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:46.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.458+0000 7f6be1600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6bdc160a30 0x7f6bdc160e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:46.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.458+0000 7f6be1600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6bdc15d980 0x7f6bdc15de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:46.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.458+0000 7f6be1600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f6bdc0d0930 con 0x7f6bdc15d980 2024-05-28T06:14:46.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.458+0000 7f6be1600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f6bdc0cf580 con 0x7f6bdc160a30 2024-05-28T06:14:46.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.458+0000 7f6be1600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f6bdc0d0ab0 con 0x7f6bc00039b0 2024-05-28T06:14:46.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.459+0000 7f6bdb400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6bdc15d980 0x7f6bdc15de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:46.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.459+0000 7f6bdb400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6bdc15d980 0x7f6bdc15de30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35608/0 (socket says 172.21.15.96:35608) 2024-05-28T06:14:46.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.459+0000 7f6bdb400640 1 -- 172.21.15.96:0/1676524318 learned_addr learned my addr 172.21.15.96:0/1676524318 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:46.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.459+0000 7f6bda000640 1 --2- 172.21.15.96:0/1676524318 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6bdc160a30 0x7f6bdc160e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:46.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.459+0000 7f6bdaa00640 1 --2- 172.21.15.96:0/1676524318 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6bc00039b0 0x7f6bdc09d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:46.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.459+0000 7f6bdb400640 1 -- 172.21.15.96:0/1676524318 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6bc00039b0 msgr2=0x7f6bdc09d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:46.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.459+0000 7f6bdb400640 1 --2- 172.21.15.96:0/1676524318 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6bc00039b0 0x7f6bdc09d0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:46.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.459+0000 7f6bdb400640 1 -- 172.21.15.96:0/1676524318 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6bdc160a30 msgr2=0x7f6bdc160e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:46.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.459+0000 7f6bdb400640 1 --2- 172.21.15.96:0/1676524318 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6bdc160a30 0x7f6bdc160e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:46.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.459+0000 7f6bdb400640 1 -- 172.21.15.96:0/1676524318 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bdc1613d0 con 0x7f6bdc15d980 2024-05-28T06:14:46.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.460+0000 7f6bdb400640 1 --2- 172.21.15.96:0/1676524318 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6bdc15d980 0x7f6bdc15de30 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f6bdc06eb70 tx=0x7f6bd4010590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:46.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.460+0000 7f6bd2a00640 1 -- 172.21.15.96:0/1676524318 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bd401a030 con 0x7f6bdc15d980 2024-05-28T06:14:46.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.460+0000 7f6bd2a00640 1 -- 172.21.15.96:0/1676524318 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6bd4010e30 con 0x7f6bdc15d980 2024-05-28T06:14:46.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.460+0000 7f6bd2a00640 1 -- 172.21.15.96:0/1676524318 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bd4003bb0 con 0x7f6bdc15d980 2024-05-28T06:14:46.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.460+0000 7f6be1600640 1 -- 172.21.15.96:0/1676524318 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bdc1f4200 con 0x7f6bdc15d980 2024-05-28T06:14:46.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.460+0000 7f6be1600640 1 -- 172.21.15.96:0/1676524318 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bdc1f47c0 con 0x7f6bdc15d980 2024-05-28T06:14:46.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.461+0000 7f6be1600640 1 -- 172.21.15.96:0/1676524318 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6bdc0689e0 con 0x7f6bdc15d980 2024-05-28T06:14:46.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.462+0000 7f6bd2a00640 1 -- 172.21.15.96:0/1676524318 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 22) ==== 92213+0+0 (secure 0 0 0) 0x7f6bd401d070 con 0x7f6bdc15d980 2024-05-28T06:14:46.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.463+0000 7f6bd2a00640 1 --2- 172.21.15.96:0/1676524318 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6bbc06d5e0 0x7f6bbc06faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:46.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.463+0000 7f6bd2a00640 1 -- 172.21.15.96:0/1676524318 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 4971+0+0 (secure 0 0 0) 0x7f6bd408acd0 con 0x7f6bdc15d980 2024-05-28T06:14:46.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.463+0000 7f6bdaa00640 1 --2- 172.21.15.96:0/1676524318 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6bbc06d5e0 0x7f6bbc06faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:46.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.464+0000 7f6bdaa00640 1 --2- 172.21.15.96:0/1676524318 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6bbc06d5e0 0x7f6bbc06faa0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc00c490 tx=0x7f6bcc02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:46.469 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.468+0000 7f6bd2a00640 1 -- 172.21.15.96:0/1676524318 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6bdc1613d0 con 0x7f6bdc15d980 2024-05-28T06:14:46.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:46.679+0000 7f6be1600640 1 -- 172.21.15.96:0/1676524318 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f6bdc1f4e80 con 0x7f6bdc15d980 2024-05-28T06:14:47.825 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:47.823+0000 7f6bd2a00640 1 -- 172.21.15.96:0/1676524318 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v70) ==== 97+0+0 (secure 0 0 0) 0x7f6bd405a270 con 0x7f6bdc15d980 2024-05-28T06:14:47.826 INFO:teuthology.orchestra.run.smithi096.stderr: removed all blocklist entries 2024-05-28T06:14:47.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:47.827+0000 7f6be1600640 1 -- 172.21.15.96:0/1676524318 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6bbc06d5e0 msgr2=0x7f6bbc06faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:47.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:47.827+0000 7f6be1600640 1 --2- 172.21.15.96:0/1676524318 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6bbc06d5e0 0x7f6bbc06faa0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc00c490 tx=0x7f6bcc02e000 comp rx=0 tx=0).stop 2024-05-28T06:14:47.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:47.827+0000 7f6be1600640 1 -- 172.21.15.96:0/1676524318 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6bdc15d980 msgr2=0x7f6bdc15de30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:47.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:47.827+0000 7f6be1600640 1 --2- 172.21.15.96:0/1676524318 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6bdc15d980 0x7f6bdc15de30 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f6bdc06eb70 tx=0x7f6bd4010590 comp rx=0 tx=0).stop 2024-05-28T06:14:47.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:47.827+0000 7f6be1600640 1 -- 172.21.15.96:0/1676524318 shutdown_connections 2024-05-28T06:14:47.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:47.827+0000 7f6be1600640 1 --2- 172.21.15.96:0/1676524318 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6bbc06d5e0 0x7f6bbc06faa0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:47.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:47.827+0000 7f6be1600640 1 --2- 172.21.15.96:0/1676524318 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6bdc15d980 0x7f6bdc15de30 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:47.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:47.827+0000 7f6be1600640 1 --2- 172.21.15.96:0/1676524318 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6bdc160a30 0x7f6bdc160e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:47.830 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:47.827+0000 7f6be1600640 1 --2- 172.21.15.96:0/1676524318 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6bc00039b0 0x7f6bdc09d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:47.830 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:47.827+0000 7f6be1600640 1 -- 172.21.15.96:0/1676524318 >> 172.21.15.96:0/1676524318 conn(0x7f6bdc06bae0 msgr2=0x7f6bdc0afca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:47.830 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:47.827+0000 7f6be1600640 1 -- 172.21.15.96:0/1676524318 shutdown_connections 2024-05-28T06:14:47.830 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:47.828+0000 7f6be1600640 1 -- 172.21.15.96:0/1676524318 wait complete. 2024-05-28T06:14:47.853 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-05-28T06:14:48.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.045+0000 7f589ea00640 1 Processor -- start 2024-05-28T06:14:48.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.045+0000 7f589ea00640 1 -- start start 2024-05-28T06:14:48.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.046+0000 7f589ea00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f589815b3a0 0x7f58980b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:48.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.046+0000 7f589ea00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f5898073310 0x7f589815f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:48.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.046+0000 7f589ea00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58980afd60 con 0x7f589806dc20 2024-05-28T06:14:48.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.046+0000 7f589ea00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f589806f980 con 0x7f589815b3a0 2024-05-28T06:14:48.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.046+0000 7f589ea00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f58980ae0c0 con 0x7f5898073310 2024-05-28T06:14:48.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.047+0000 7f589d600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f589815b3a0 0x7f58980b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:48.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.047+0000 7f589d600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f589815b3a0 0x7f58980b32c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35620/0 (socket says 172.21.15.96:35620) 2024-05-28T06:14:48.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.047+0000 7f589d600640 1 -- 172.21.15.96:0/2277676490 learned_addr learned my addr 172.21.15.96:0/2277676490 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:48.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.047+0000 7f589d600640 1 -- 172.21.15.96:0/2277676490 >> v2:172.21.15.123:3300/0 conn(0x7f5898073310 msgr2=0x7f589815f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:48.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.047+0000 7f589d600640 1 --2- 172.21.15.96:0/2277676490 >> v2:172.21.15.123:3300/0 conn(0x7f5898073310 0x7f589815f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:48.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.047+0000 7f589d600640 1 -- 172.21.15.96:0/2277676490 >> v1:172.21.15.123:6789/0 conn(0x7f589806dc20 legacy=0x7f5898072c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:48.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.047+0000 7f589d600640 1 -- 172.21.15.96:0/2277676490 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58980af7a0 con 0x7f589815b3a0 2024-05-28T06:14:48.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.048+0000 7f589d600640 1 --2- 172.21.15.96:0/2277676490 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f589815b3a0 0x7f58980b32c0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f5898071b50 tx=0x7f588c02c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc2cec22a1d59d8 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:48.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.048+0000 7f5897e00640 1 -- 172.21.15.96:0/2277676490 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f588c0325a0 con 0x7f589815b3a0 2024-05-28T06:14:48.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.048+0000 7f5897e00640 1 -- 172.21.15.96:0/2277676490 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f588c038070 con 0x7f589815b3a0 2024-05-28T06:14:48.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.048+0000 7f5897e00640 1 -- 172.21.15.96:0/2277676490 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f588c032960 con 0x7f589815b3a0 2024-05-28T06:14:48.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.049+0000 7f589ea00640 1 -- 172.21.15.96:0/2277676490 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f589815b3a0 msgr2=0x7f58980b32c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:48.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.049+0000 7f589ea00640 1 --2- 172.21.15.96:0/2277676490 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f589815b3a0 0x7f58980b32c0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f5898071b50 tx=0x7f588c02c8e0 comp rx=0 tx=0).stop 2024-05-28T06:14:48.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.049+0000 7f589ea00640 1 -- 172.21.15.96:0/2277676490 shutdown_connections 2024-05-28T06:14:48.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.049+0000 7f589ea00640 1 --2- 172.21.15.96:0/2277676490 >> v2:172.21.15.123:3300/0 conn(0x7f5898073310 0x7f589815f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:48.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.049+0000 7f589ea00640 1 --2- 172.21.15.96:0/2277676490 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f589815b3a0 0x7f58980b32c0 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:48.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.049+0000 7f589ea00640 1 -- 172.21.15.96:0/2277676490 >> 172.21.15.96:0/2277676490 conn(0x7f58980b1fc0 msgr2=0x7f58980b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:48.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.049+0000 7f589ea00640 1 -- 172.21.15.96:0/2277676490 shutdown_connections 2024-05-28T06:14:48.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.049+0000 7f589ea00640 1 -- 172.21.15.96:0/2277676490 wait complete. 2024-05-28T06:14:48.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.051+0000 7f589ea00640 1 Processor -- start 2024-05-28T06:14:48.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.051+0000 7f589ea00640 1 -- start start 2024-05-28T06:14:48.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.052+0000 7f589ea00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f589806dc20 0x7f58980cd920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:48.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.052+0000 7f589ea00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5898073310 0x7f58980ce220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:48.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.052+0000 7f589ea00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f589815b3a0 0x7f589814ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:48.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.052+0000 7f589ea00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f589815b780 con 0x7f589815b3a0 2024-05-28T06:14:48.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.052+0000 7f589ea00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f589815fee0 con 0x7f589806dc20 2024-05-28T06:14:48.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.052+0000 7f589ea00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f58980ae550 con 0x7f5898073310 2024-05-28T06:14:48.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.052+0000 7f589e000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f589815b3a0 0x7f589814ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:48.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.052+0000 7f589e000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f589815b3a0 0x7f589814ae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35622/0 (socket says 172.21.15.96:35622) 2024-05-28T06:14:48.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.052+0000 7f589e000640 1 -- 172.21.15.96:0/213337157 learned_addr learned my addr 172.21.15.96:0/213337157 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:48.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.053+0000 7f589e000640 1 -- 172.21.15.96:0/213337157 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5898073310 msgr2=0x7f58980ce220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:48.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.053+0000 7f589e000640 1 --2- 172.21.15.96:0/213337157 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5898073310 0x7f58980ce220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:48.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.053+0000 7f589e000640 1 -- 172.21.15.96:0/213337157 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f589806dc20 msgr2=0x7f58980cd920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:48.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.053+0000 7f589d600640 1 --2- 172.21.15.96:0/213337157 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f589806dc20 0x7f58980cd920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:48.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.053+0000 7f589e000640 1 --2- 172.21.15.96:0/213337157 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f589806dc20 0x7f58980cd920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:48.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.053+0000 7f589e000640 1 -- 172.21.15.96:0/213337157 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58980ceb20 con 0x7f589815b3a0 2024-05-28T06:14:48.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.053+0000 7f589e000640 1 --2- 172.21.15.96:0/213337157 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f589815b3a0 0x7f589814ae20 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f589000b9e0 tx=0x7f589000bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:48.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.053+0000 7f5895600640 1 -- 172.21.15.96:0/213337157 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5890019070 con 0x7f589815b3a0 2024-05-28T06:14:48.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.053+0000 7f5895600640 1 -- 172.21.15.96:0/213337157 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f589000cac0 con 0x7f589815b3a0 2024-05-28T06:14:48.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.053+0000 7f5895600640 1 -- 172.21.15.96:0/213337157 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5890014440 con 0x7f589815b3a0 2024-05-28T06:14:48.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.053+0000 7f589ea00640 1 -- 172.21.15.96:0/213337157 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58980cee00 con 0x7f589815b3a0 2024-05-28T06:14:48.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.054+0000 7f589ea00640 1 -- 172.21.15.96:0/213337157 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58980cd120 con 0x7f589815b3a0 2024-05-28T06:14:48.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.055+0000 7f589ea00640 1 -- 172.21.15.96:0/213337157 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58980689e0 con 0x7f589815b3a0 2024-05-28T06:14:48.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.056+0000 7f5895600640 1 -- 172.21.15.96:0/213337157 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 22) ==== 92213+0+0 (secure 0 0 0) 0x7f589001c070 con 0x7f589815b3a0 2024-05-28T06:14:48.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.057+0000 7f5895600640 1 --2- 172.21.15.96:0/213337157 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f587806d5e0 0x7f587806faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:48.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.057+0000 7f5895600640 1 -- 172.21.15.96:0/213337157 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 4885+0+0 (secure 0 0 0) 0x7f589008a2a0 con 0x7f589815b3a0 2024-05-28T06:14:48.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.057+0000 7f589d600640 1 --2- 172.21.15.96:0/213337157 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f587806d5e0 0x7f587806faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:48.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.058+0000 7f589d600640 1 --2- 172.21.15.96:0/213337157 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f587806d5e0 0x7f587806faa0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f588c037000 tx=0x7f588c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:48.062 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.061+0000 7f5895600640 1 -- 172.21.15.96:0/213337157 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f589005a120 con 0x7f589815b3a0 2024-05-28T06:14:48.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.340+0000 7f589ea00640 1 -- 172.21.15.96:0/213337157 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f58980ccc20 con 0x7f589815b3a0 2024-05-28T06:14:48.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.341+0000 7f5895600640 1 -- 172.21.15.96:0/213337157 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v16) ==== 80+0+5304 (secure 0 0 0) 0x7f5890059a20 con 0x7f589815b3a0 2024-05-28T06:14:48.343 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:14:48.343 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:14:48.343 INFO:teuthology.orchestra.run.smithi096.stdout: "auth_dump": [ 2024-05-28T06:14:48.344 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:14:48.344 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "mds.a", 2024-05-28T06:14:48.344 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCWc1VmyA9HHBAANJCF8I/RP5ZYviyOzo0JCQ==", 2024-05-28T06:14:48.344 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:14:48.344 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow", 2024-05-28T06:14:48.344 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow *", 2024-05-28T06:14:48.344 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow *", 2024-05-28T06:14:48.344 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:14:48.344 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:14:48.344 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:14:48.344 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:14:48.344 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "mds.b", 2024-05-28T06:14:48.344 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCWc1VmtR0EOBAA296qGHTmB/2sMztDbDJ56g==", 2024-05-28T06:14:48.345 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:14:48.345 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow", 2024-05-28T06:14:48.345 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow *", 2024-05-28T06:14:48.345 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow *", 2024-05-28T06:14:48.345 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:14:48.345 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:14:48.345 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:14:48.345 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:14:48.345 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "mds.c", 2024-05-28T06:14:48.345 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCWc1VmrLxHKhAAhkTRLDKL6RUJQGetB8JQQA==", 2024-05-28T06:14:48.345 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:14:48.345 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow", 2024-05-28T06:14:48.346 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow *", 2024-05-28T06:14:48.346 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow *", 2024-05-28T06:14:48.346 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:14:48.346 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:14:48.346 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:14:48.346 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:14:48.346 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "mds.d", 2024-05-28T06:14:48.346 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCXc1VmW2aQCxAAnkDwQzlgy39vFe2O9Wwf7Q==", 2024-05-28T06:14:48.346 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:14:48.346 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow", 2024-05-28T06:14:48.346 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow *", 2024-05-28T06:14:48.347 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow *", 2024-05-28T06:14:48.347 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:14:48.347 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:14:48.347 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:14:48.347 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:14:48.347 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "osd.0", 2024-05-28T06:14:48.347 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCbc1VmEcNAAxAAiB5AsfY2uRY1kc6h80chuw==", 2024-05-28T06:14:48.347 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:14:48.347 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow profile osd", 2024-05-28T06:14:48.347 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile osd", 2024-05-28T06:14:48.347 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:14:48.347 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:14:48.348 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:14:48.348 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:14:48.348 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "osd.1", 2024-05-28T06:14:48.348 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCec1Vm6CAaDhAA1NkYM2q4fTsgSDSwJZL3Qg==", 2024-05-28T06:14:48.348 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:14:48.348 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow profile osd", 2024-05-28T06:14:48.348 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile osd", 2024-05-28T06:14:48.348 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:14:48.348 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:14:48.348 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:14:48.348 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:14:48.348 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "osd.2", 2024-05-28T06:14:48.348 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQChc1VmKn1jFxAA+dM4gM7FR8NwBpyd833A/Q==", 2024-05-28T06:14:48.348 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:14:48.349 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow profile osd", 2024-05-28T06:14:48.349 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile osd", 2024-05-28T06:14:48.349 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:14:48.349 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:14:48.349 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:14:48.349 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:14:48.349 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "osd.3", 2024-05-28T06:14:48.349 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCkc1VmV07MIRAAHIXQGDVbL9V5QqexQ89IJA==", 2024-05-28T06:14:48.349 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:14:48.349 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow profile osd", 2024-05-28T06:14:48.349 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile osd", 2024-05-28T06:14:48.349 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:14:48.350 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:14:48.350 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:14:48.350 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:14:48.350 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "osd.4", 2024-05-28T06:14:48.350 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCqc1VmobuZMxAAlY8xhuzqlkuFrcbTouF5rA==", 2024-05-28T06:14:48.350 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:14:48.350 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow profile osd", 2024-05-28T06:14:48.350 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile osd", 2024-05-28T06:14:48.350 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:14:48.350 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:14:48.350 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:14:48.350 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:14:48.351 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "osd.5", 2024-05-28T06:14:48.351 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCuc1Vmwt3QAxAAUFj/e/wJvxljgAbcTb32Ug==", 2024-05-28T06:14:48.351 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:14:48.351 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow profile osd", 2024-05-28T06:14:48.351 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile osd", 2024-05-28T06:14:48.351 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:14:48.351 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:14:48.351 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:14:48.351 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:14:48.351 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "osd.6", 2024-05-28T06:14:48.351 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCxc1VmscerDxAAqEDAqy6/OvdfSJ+ALZ7WeQ==", 2024-05-28T06:14:48.351 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:14:48.352 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow profile osd", 2024-05-28T06:14:48.352 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile osd", 2024-05-28T06:14:48.352 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:14:48.352 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:14:48.352 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:14:48.352 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:14:48.352 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "osd.7", 2024-05-28T06:14:48.352 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQC0c1VmgxkbKhAA7dFzDTBKFmKiFFxPsyiT4Q==", 2024-05-28T06:14:48.352 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:14:48.352 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow profile osd", 2024-05-28T06:14:48.352 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile osd", 2024-05-28T06:14:48.352 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:14:48.352 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:14:48.353 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:14:48.353 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:14:48.353 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "client.0", 2024-05-28T06:14:48.353 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCXc1Vm91hbGBAASTL3HQR8XOs7wVQEHbPl/g==", 2024-05-28T06:14:48.353 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:14:48.353 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow", 2024-05-28T06:14:48.353 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow r", 2024-05-28T06:14:48.353 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-05-28T06:14:48.353 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:14:48.353 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:14:48.353 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:14:48.353 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "client.1", 2024-05-28T06:14:48.353 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCXc1VmIc77KhAAw2e0alTmVROhyCIFW4H3Sg==", 2024-05-28T06:14:48.353 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:14:48.353 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow", 2024-05-28T06:14:48.354 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow r", 2024-05-28T06:14:48.354 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow rw", 2024-05-28T06:14:48.354 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow rwx" 2024-05-28T06:14:48.354 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:14:48.354 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:14:48.354 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:14:48.354 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "client.2", 2024-05-28T06:14:48.354 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCXc1VmCS+OIRAAMV3Au6KuchLhA59QTGbv1w==", 2024-05-28T06:14:48.354 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:14:48.354 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow", 2024-05-28T06:14:48.354 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow r", 2024-05-28T06:14:48.354 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow rw", 2024-05-28T06:14:48.354 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow rwx" 2024-05-28T06:14:48.355 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:14:48.355 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:14:48.355 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:14:48.355 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "client.3", 2024-05-28T06:14:48.355 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCXc1Vm5r3pNBAAkVZ3arjywHQkEP5lmS8xqg==", 2024-05-28T06:14:48.355 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:14:48.355 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow", 2024-05-28T06:14:48.355 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow r", 2024-05-28T06:14:48.355 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow rw", 2024-05-28T06:14:48.355 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow rwx" 2024-05-28T06:14:48.355 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:14:48.355 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:14:48.355 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:14:48.356 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "client.admin", 2024-05-28T06:14:48.356 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCVc1Vm5QaXFhAAQFGKgbZ/kpGisNm2lIw0Uw==", 2024-05-28T06:14:48.356 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:14:48.356 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow *", 2024-05-28T06:14:48.356 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow *", 2024-05-28T06:14:48.356 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow *", 2024-05-28T06:14:48.356 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:14:48.356 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:14:48.356 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:14:48.356 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:14:48.356 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "mgr.x", 2024-05-28T06:14:48.356 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCWc1VmVMx/EhAAylvi37aq3hbMOhUSQWLVSg==", 2024-05-28T06:14:48.356 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:14:48.357 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow *", 2024-05-28T06:14:48.357 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile mgr", 2024-05-28T06:14:48.357 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:14:48.357 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:14:48.357 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:14:48.357 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:14:48.357 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "mgr.y", 2024-05-28T06:14:48.357 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCWc1Vmh5r7BxAAA74gn+FozehM+34eFik8kw==", 2024-05-28T06:14:48.357 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:14:48.357 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow *", 2024-05-28T06:14:48.357 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile mgr", 2024-05-28T06:14:48.357 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:14:48.357 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:14:48.357 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:14:48.358 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-05-28T06:14:48.358 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:14:48.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.344+0000 7f589ea00640 1 -- 172.21.15.96:0/213337157 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f587806d5e0 msgr2=0x7f587806faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:48.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.344+0000 7f589ea00640 1 --2- 172.21.15.96:0/213337157 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f587806d5e0 0x7f587806faa0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f588c037000 tx=0x7f588c008040 comp rx=0 tx=0).stop 2024-05-28T06:14:48.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.344+0000 7f589ea00640 1 -- 172.21.15.96:0/213337157 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f589815b3a0 msgr2=0x7f589814ae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:48.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.344+0000 7f589ea00640 1 --2- 172.21.15.96:0/213337157 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f589815b3a0 0x7f589814ae20 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f589000b9e0 tx=0x7f589000bea0 comp rx=0 tx=0).stop 2024-05-28T06:14:48.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.344+0000 7f589ea00640 1 -- 172.21.15.96:0/213337157 shutdown_connections 2024-05-28T06:14:48.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.344+0000 7f589ea00640 1 --2- 172.21.15.96:0/213337157 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f587806d5e0 0x7f587806faa0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:48.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.344+0000 7f589ea00640 1 --2- 172.21.15.96:0/213337157 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f589815b3a0 0x7f589814ae20 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:48.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.344+0000 7f589ea00640 1 --2- 172.21.15.96:0/213337157 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5898073310 0x7f58980ce220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:48.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.344+0000 7f589ea00640 1 --2- 172.21.15.96:0/213337157 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f589806dc20 0x7f58980cd920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:48.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.344+0000 7f589ea00640 1 -- 172.21.15.96:0/213337157 >> 172.21.15.96:0/213337157 conn(0x7f58980b1fc0 msgr2=0x7f58980720c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:48.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.344+0000 7f589ea00640 1 -- 172.21.15.96:0/213337157 shutdown_connections 2024-05-28T06:14:48.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.345+0000 7f589ea00640 1 -- 172.21.15.96:0/213337157 wait complete. 2024-05-28T06:14:48.366 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:14:48.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.555+0000 7f2f05400640 1 Processor -- start 2024-05-28T06:14:48.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.556+0000 7f2f05400640 1 -- start start 2024-05-28T06:14:48.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.557+0000 7f2f05400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f2f00070700 0x7f2f000af8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:48.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.557+0000 7f2f05400640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f0006e500 con 0x7f2f00162170 2024-05-28T06:14:48.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.557+0000 7f2f05400640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f0006e7e0 con 0x7f2f0015e530 2024-05-28T06:14:48.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.557+0000 7f2f05400640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f2f0006eac0 con 0x7f2f00070700 2024-05-28T06:14:48.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.557+0000 7f2f04a00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f2f00162170 0x7f2f000b5080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:56556/0 (socket says 172.21.15.96:56556) 2024-05-28T06:14:48.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.557+0000 7f2f04a00640 1 -- 172.21.15.96:0/3263666805 learned_addr learned my addr 172.21.15.96:0/3263666805 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:14:48.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.558+0000 7f2ef7400640 1 --2- 172.21.15.96:0/3263666805 >> v2:172.21.15.123:3300/0 conn(0x7f2f00070700 0x7f2f000af8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:48.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.558+0000 7f2eff400640 1 -- 172.21.15.96:0/3263666805 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2449310502 0 0) 0x7f2f0006e500 con 0x7f2f00162170 2024-05-28T06:14:48.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.558+0000 7f2eff400640 1 -- 172.21.15.96:0/3263666805 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2ee4003520 con 0x7f2f00162170 2024-05-28T06:14:48.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.558+0000 7f2eff400640 1 -- 172.21.15.96:0/3263666805 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1027709406 0 0) 0x7f2ee4003520 con 0x7f2f00162170 2024-05-28T06:14:48.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.558+0000 7f2eff400640 1 -- 172.21.15.96:0/3263666805 >> v2:172.21.15.123:3300/0 conn(0x7f2f00070700 msgr2=0x7f2f000af8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:48.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.558+0000 7f2eff400640 1 --2- 172.21.15.96:0/3263666805 >> v2:172.21.15.123:3300/0 conn(0x7f2f00070700 0x7f2f000af8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:48.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.558+0000 7f2eff400640 1 -- 172.21.15.96:0/3263666805 >> v1:172.21.15.123:6789/0 conn(0x7f2f0015e530 legacy=0x7f2f0006f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:48.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.558+0000 7f2eff400640 1 -- 172.21.15.96:0/3263666805 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f0006eda0 con 0x7f2f00162170 2024-05-28T06:14:48.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.558+0000 7f2eff400640 1 -- 172.21.15.96:0/3263666805 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f2ef8001670 con 0x7f2f00162170 2024-05-28T06:14:48.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.558+0000 7f2eff400640 1 -- 172.21.15.96:0/3263666805 >> v1:172.21.15.96:6789/0 conn(0x7f2f00162170 legacy=0x7f2f000b5080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:48.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.558+0000 7f2eff400640 1 --2- 172.21.15.96:0/3263666805 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2ee40039b0 0x7f2ee4003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:48.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.558+0000 7f2eff400640 1 --2- 172.21.15.96:0/3263666805 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ee4004800 0x7f2ee4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:48.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.558+0000 7f2eff400640 1 --2- 172.21.15.96:0/3263666805 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2ee40052e0 0x7f2ee400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:48.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.559+0000 7f2eff400640 1 -- 172.21.15.96:0/3263666805 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f2f0006eac0 con 0x7f2ee40039b0 2024-05-28T06:14:48.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.559+0000 7f2eff400640 1 -- 172.21.15.96:0/3263666805 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f2ee400c190 con 0x7f2ee4004800 2024-05-28T06:14:48.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.559+0000 7f2eff400640 1 -- 172.21.15.96:0/3263666805 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f2ee400c390 con 0x7f2ee40052e0 2024-05-28T06:14:48.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.559+0000 7f2effe00640 1 --2- 172.21.15.96:0/3263666805 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2ee40039b0 0x7f2ee4003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:48.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.559+0000 7f2effe00640 1 -- 172.21.15.96:0/3263666805 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2ee40052e0 msgr2=0x7f2ee400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:48.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.559+0000 7f2effe00640 1 --2- 172.21.15.96:0/3263666805 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2ee40052e0 0x7f2ee400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:48.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.559+0000 7f2effe00640 1 -- 172.21.15.96:0/3263666805 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ee4004800 msgr2=0x7f2ee4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:48.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.559+0000 7f2effe00640 1 --2- 172.21.15.96:0/3263666805 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ee4004800 0x7f2ee4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:48.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.559+0000 7f2effe00640 1 -- 172.21.15.96:0/3263666805 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ee4003520 con 0x7f2ee40039b0 2024-05-28T06:14:48.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.559+0000 7f2f04a00640 1 -- 172.21.15.96:0/3263666805 reap_dead start 2024-05-28T06:14:48.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.560+0000 7f2effe00640 1 --2- 172.21.15.96:0/3263666805 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2ee40039b0 0x7f2ee4003db0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f2ef0004480 tx=0x7f2ee4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54e0486ccde6142b server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:48.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.560+0000 7f2eff400640 1 -- 172.21.15.96:0/3263666805 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ef0037070 con 0x7f2ee40039b0 2024-05-28T06:14:48.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.560+0000 7f2eff400640 1 -- 172.21.15.96:0/3263666805 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ef002ad10 con 0x7f2ee40039b0 2024-05-28T06:14:48.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.560+0000 7f2eff400640 1 -- 172.21.15.96:0/3263666805 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ef0004080 con 0x7f2ee40039b0 2024-05-28T06:14:48.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.561+0000 7f2f05400640 1 -- 172.21.15.96:0/3263666805 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2ee40039b0 msgr2=0x7f2ee4003db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:48.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.561+0000 7f2f05400640 1 --2- 172.21.15.96:0/3263666805 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2ee40039b0 0x7f2ee4003db0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f2ef0004480 tx=0x7f2ee4005830 comp rx=0 tx=0).stop 2024-05-28T06:14:48.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.561+0000 7f2f05400640 1 -- 172.21.15.96:0/3263666805 shutdown_connections 2024-05-28T06:14:48.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.561+0000 7f2f05400640 1 --2- 172.21.15.96:0/3263666805 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2ee40039b0 0x7f2ee4003db0 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:48.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.561+0000 7f2f05400640 1 -- 172.21.15.96:0/3263666805 >> 172.21.15.96:0/3263666805 conn(0x7f2f0006b900 msgr2=0x7f2f0006bce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:48.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.561+0000 7f2f05400640 1 -- 172.21.15.96:0/3263666805 shutdown_connections 2024-05-28T06:14:48.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.561+0000 7f2f05400640 1 -- 172.21.15.96:0/3263666805 wait complete. 2024-05-28T06:14:48.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.563+0000 7f2f05400640 1 Processor -- start 2024-05-28T06:14:48.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.563+0000 7f2f05400640 1 -- start start 2024-05-28T06:14:48.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.564+0000 7f2f05400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ee40039b0 0x7f2f0014d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:48.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.564+0000 7f2f05400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2f0015cb70 0x7f2f0015cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:48.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.564+0000 7f2f05400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2f00156340 0x7f2f001567c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:48.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.564+0000 7f2f05400640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f2f000af570 con 0x7f2f0015cb70 2024-05-28T06:14:48.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.564+0000 7f2f05400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f2f0015e910 con 0x7f2ee40039b0 2024-05-28T06:14:48.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.564+0000 7f2f05400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f2f000af6f0 con 0x7f2f00156340 2024-05-28T06:14:48.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.564+0000 7f2ef7400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2f0015cb70 0x7f2f0015cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:48.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.564+0000 7f2ef7400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2f0015cb70 0x7f2f0015cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35638/0 (socket says 172.21.15.96:35638) 2024-05-28T06:14:48.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.564+0000 7f2ef7400640 1 -- 172.21.15.96:0/1164241580 learned_addr learned my addr 172.21.15.96:0/1164241580 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:48.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.565+0000 7f2f04a00640 1 --2- 172.21.15.96:0/1164241580 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2f00156340 0x7f2f001567c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:48.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.565+0000 7f2effe00640 1 --2- 172.21.15.96:0/1164241580 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ee40039b0 0x7f2f0014d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:48.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.565+0000 7f2ef7400640 1 -- 172.21.15.96:0/1164241580 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2f00156340 msgr2=0x7f2f001567c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:48.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.565+0000 7f2ef7400640 1 --2- 172.21.15.96:0/1164241580 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2f00156340 0x7f2f001567c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:48.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.565+0000 7f2ef7400640 1 -- 172.21.15.96:0/1164241580 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ee40039b0 msgr2=0x7f2f0014d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:48.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.565+0000 7f2ef7400640 1 --2- 172.21.15.96:0/1164241580 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ee40039b0 0x7f2f0014d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:48.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.565+0000 7f2ef7400640 1 -- 172.21.15.96:0/1164241580 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f00157070 con 0x7f2f0015cb70 2024-05-28T06:14:48.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.565+0000 7f2ef7400640 1 --2- 172.21.15.96:0/1164241580 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2f0015cb70 0x7f2f0015cff0 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f2ee800d830 tx=0x7f2ee800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:48.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.565+0000 7f2efcc00640 1 -- 172.21.15.96:0/1164241580 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ee800b660 con 0x7f2f0015cb70 2024-05-28T06:14:48.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.566+0000 7f2efcc00640 1 -- 172.21.15.96:0/1164241580 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ee8019070 con 0x7f2f0015cb70 2024-05-28T06:14:48.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.566+0000 7f2f05400640 1 -- 172.21.15.96:0/1164241580 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f00157350 con 0x7f2f0015cb70 2024-05-28T06:14:48.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.566+0000 7f2efcc00640 1 -- 172.21.15.96:0/1164241580 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ee800ba40 con 0x7f2f0015cb70 2024-05-28T06:14:48.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.566+0000 7f2f05400640 1 -- 172.21.15.96:0/1164241580 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f00157910 con 0x7f2f0015cb70 2024-05-28T06:14:48.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.567+0000 7f2f05400640 1 -- 172.21.15.96:0/1164241580 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f0006e4f0 con 0x7f2f0015cb70 2024-05-28T06:14:48.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.568+0000 7f2efcc00640 1 -- 172.21.15.96:0/1164241580 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 22) ==== 92213+0+0 (secure 0 0 0) 0x7f2ee801e460 con 0x7f2f0015cb70 2024-05-28T06:14:48.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.569+0000 7f2efcc00640 1 --2- 172.21.15.96:0/1164241580 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2edc06d610 0x7f2edc06fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:48.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.569+0000 7f2efcc00640 1 -- 172.21.15.96:0/1164241580 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 4885+0+0 (secure 0 0 0) 0x7f2ee808ada0 con 0x7f2f0015cb70 2024-05-28T06:14:48.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.569+0000 7f2effe00640 1 --2- 172.21.15.96:0/1164241580 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2edc06d610 0x7f2edc06fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:48.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.571+0000 7f2effe00640 1 --2- 172.21.15.96:0/1164241580 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2edc06d610 0x7f2edc06fad0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f2ef0034000 tx=0x7f2ef002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:48.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.574+0000 7f2efcc00640 1 -- 172.21.15.96:0/1164241580 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2f0006e4f0 con 0x7f2f0015cb70 2024-05-28T06:14:48.775 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.773+0000 7f2f05400640 1 -- 172.21.15.96:0/1164241580 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2f000689e0 con 0x7f2f0015cb70 2024-05-28T06:14:48.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.775+0000 7f2efcc00640 1 -- 172.21.15.96:0/1164241580 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v70) ==== 74+0+12946 (secure 0 0 0) 0x7f2ee805a990 con 0x7f2f0015cb70 2024-05-28T06:14:48.777 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:14:48.777 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":70,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:14:47.821111+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:14:48.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.777+0000 7f2f05400640 1 -- 172.21.15.96:0/1164241580 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2edc06d610 msgr2=0x7f2edc06fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:48.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.777+0000 7f2f05400640 1 --2- 172.21.15.96:0/1164241580 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2edc06d610 0x7f2edc06fad0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f2ef0034000 tx=0x7f2ef002e000 comp rx=0 tx=0).stop 2024-05-28T06:14:48.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.777+0000 7f2f05400640 1 -- 172.21.15.96:0/1164241580 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2f0015cb70 msgr2=0x7f2f0015cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:48.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.777+0000 7f2f05400640 1 --2- 172.21.15.96:0/1164241580 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2f0015cb70 0x7f2f0015cff0 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f2ee800d830 tx=0x7f2ee800dcf0 comp rx=0 tx=0).stop 2024-05-28T06:14:48.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.778+0000 7f2f05400640 1 -- 172.21.15.96:0/1164241580 shutdown_connections 2024-05-28T06:14:48.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.778+0000 7f2f05400640 1 --2- 172.21.15.96:0/1164241580 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2edc06d610 0x7f2edc06fad0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:48.781 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.778+0000 7f2f05400640 1 --2- 172.21.15.96:0/1164241580 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2f00156340 0x7f2f001567c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:48.781 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.778+0000 7f2f05400640 1 --2- 172.21.15.96:0/1164241580 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2f0015cb70 0x7f2f0015cff0 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:48.781 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.778+0000 7f2f05400640 1 --2- 172.21.15.96:0/1164241580 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ee40039b0 0x7f2f0014d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:48.781 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.778+0000 7f2f05400640 1 -- 172.21.15.96:0/1164241580 >> 172.21.15.96:0/1164241580 conn(0x7f2f0006b900 msgr2=0x7f2f000b5080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:48.781 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.778+0000 7f2f05400640 1 -- 172.21.15.96:0/1164241580 shutdown_connections 2024-05-28T06:14:48.781 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.778+0000 7f2f05400640 1 -- 172.21.15.96:0/1164241580 wait complete. 2024-05-28T06:14:48.800 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-28T06:03:59.517102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-28T06:14:48.800 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-28T06:14:49.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:48.999+0000 7f47f1a00640 1 Processor -- start 2024-05-28T06:14:49.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.000+0000 7f47f1a00640 1 -- start start 2024-05-28T06:14:49.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.001+0000 7f47f1a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f47ec0cf470 0x7f47ec0d1bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:49.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.001+0000 7f47f1a00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47ec0b2290 con 0x7f47ec0b49e0 2024-05-28T06:14:49.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.001+0000 7f47f1a00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47ec0d03f0 con 0x7f47ec161ed0 2024-05-28T06:14:49.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.001+0000 7f47f1a00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f47ec0d0690 con 0x7f47ec0cf470 2024-05-28T06:14:49.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.002+0000 7f47eb400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f47ec0b49e0 0x7f47ec0ced60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:52530/0 (socket says 172.21.15.96:52530) 2024-05-28T06:14:49.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.002+0000 7f47f1000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f47ec0cf470 0x7f47ec0d1bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:49.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.002+0000 7f47eb400640 1 -- 172.21.15.96:0/487348479 learned_addr learned my addr 172.21.15.96:0/487348479 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:14:49.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.002+0000 7f47f1000640 1 -- 172.21.15.96:0/487348479 >> v1:172.21.15.123:6789/0 conn(0x7f47ec161ed0 legacy=0x7f47ec06ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:49.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.002+0000 7f47f1000640 1 -- 172.21.15.96:0/487348479 >> v1:172.21.15.96:6789/0 conn(0x7f47ec0b49e0 legacy=0x7f47ec0ced60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:49.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.002+0000 7f47f1000640 1 -- 172.21.15.96:0/487348479 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47ec0b3780 con 0x7f47ec0cf470 2024-05-28T06:14:49.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.002+0000 7f47eaa00640 1 -- 172.21.15.96:0/487348479 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2388590502 0 0) 0x7f47d4001490 con 0x7f47ec0b49e0 2024-05-28T06:14:49.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.003+0000 7f47f1000640 1 --2- 172.21.15.96:0/487348479 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f47ec0cf470 0x7f47ec0d1bb0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f47e40097a0 tx=0x7f47e402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc8bc626c6d0aefd server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:49.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.004+0000 7f47eaa00640 1 -- 172.21.15.96:0/487348479 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47e4037070 con 0x7f47ec0cf470 2024-05-28T06:14:49.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.004+0000 7f47eaa00640 1 -- 172.21.15.96:0/487348479 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47e4032440 con 0x7f47ec0cf470 2024-05-28T06:14:49.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.004+0000 7f47eaa00640 1 -- 172.21.15.96:0/487348479 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47e4032810 con 0x7f47ec0cf470 2024-05-28T06:14:49.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.004+0000 7f47f1a00640 1 -- 172.21.15.96:0/487348479 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f47ec0cf470 msgr2=0x7f47ec0d1bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:49.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.004+0000 7f47f1a00640 1 --2- 172.21.15.96:0/487348479 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f47ec0cf470 0x7f47ec0d1bb0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f47e40097a0 tx=0x7f47e402a1a0 comp rx=0 tx=0).stop 2024-05-28T06:14:49.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.004+0000 7f47f1a00640 1 -- 172.21.15.96:0/487348479 shutdown_connections 2024-05-28T06:14:49.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.004+0000 7f47f1a00640 1 --2- 172.21.15.96:0/487348479 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f47ec0cf470 0x7f47ec0d1bb0 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:49.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.004+0000 7f47f1a00640 1 -- 172.21.15.96:0/487348479 >> 172.21.15.96:0/487348479 conn(0x7f47ec0b3ec0 msgr2=0x7f47ec0b42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:49.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.005+0000 7f47f1a00640 1 -- 172.21.15.96:0/487348479 shutdown_connections 2024-05-28T06:14:49.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.005+0000 7f47f1a00640 1 -- 172.21.15.96:0/487348479 wait complete. 2024-05-28T06:14:49.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.006+0000 7f47f1a00640 1 Processor -- start 2024-05-28T06:14:49.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.007+0000 7f47f1a00640 1 -- start start 2024-05-28T06:14:49.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.007+0000 7f47f1a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f47ec0b49e0 0x7f47ec15f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:49.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.007+0000 7f47f1a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f47ec0cf470 0x7f47ec09cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:49.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.007+0000 7f47f1a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f47ec161ed0 0x7f47ec0b6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:49.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.007+0000 7f47f1a00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f47ec0b4e00 con 0x7f47ec0b49e0 2024-05-28T06:14:49.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.007+0000 7f47f1a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f47ec0b1700 con 0x7f47ec0cf470 2024-05-28T06:14:49.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.007+0000 7f47f1a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f47ec0b2750 con 0x7f47ec161ed0 2024-05-28T06:14:49.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.008+0000 7f47ebe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f47ec0b49e0 0x7f47ec15f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:49.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.008+0000 7f47ebe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f47ec0b49e0 0x7f47ec15f200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54456/0 (socket says 172.21.15.96:54456) 2024-05-28T06:14:49.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.008+0000 7f47ebe00640 1 -- 172.21.15.96:0/4262419486 learned_addr learned my addr 172.21.15.96:0/4262419486 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:49.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.008+0000 7f47f1000640 1 --2- 172.21.15.96:0/4262419486 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f47ec161ed0 0x7f47ec0b6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:49.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.008+0000 7f47ebe00640 1 -- 172.21.15.96:0/4262419486 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f47ec161ed0 msgr2=0x7f47ec0b6270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:49.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.008+0000 7f47ebe00640 1 --2- 172.21.15.96:0/4262419486 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f47ec161ed0 0x7f47ec0b6270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:49.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.008+0000 7f47ebe00640 1 -- 172.21.15.96:0/4262419486 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f47ec0cf470 msgr2=0x7f47ec09cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:49.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.008+0000 7f47ebe00640 1 --2- 172.21.15.96:0/4262419486 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f47ec0cf470 0x7f47ec09cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:49.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.008+0000 7f47ebe00640 1 -- 172.21.15.96:0/4262419486 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47ec15f740 con 0x7f47ec0b49e0 2024-05-28T06:14:49.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.008+0000 7f47f1000640 1 --2- 172.21.15.96:0/4262419486 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f47ec161ed0 0x7f47ec0b6270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:14:49.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.009+0000 7f47ebe00640 1 --2- 172.21.15.96:0/4262419486 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f47ec0b49e0 0x7f47ec15f200 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f47dc00c430 tx=0x7f47dc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:49.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.009+0000 7f47e3e00640 1 -- 172.21.15.96:0/4262419486 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47dc017070 con 0x7f47ec0b49e0 2024-05-28T06:14:49.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.009+0000 7f47e3e00640 1 -- 172.21.15.96:0/4262419486 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47dc0052d0 con 0x7f47ec0b49e0 2024-05-28T06:14:49.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.009+0000 7f47e3e00640 1 -- 172.21.15.96:0/4262419486 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47dc003830 con 0x7f47ec0b49e0 2024-05-28T06:14:49.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.009+0000 7f47f1a00640 1 -- 172.21.15.96:0/4262419486 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47ec15fa10 con 0x7f47ec0b49e0 2024-05-28T06:14:49.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.009+0000 7f47f1a00640 1 -- 172.21.15.96:0/4262419486 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47ec15ffd0 con 0x7f47ec0b49e0 2024-05-28T06:14:49.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.010+0000 7f47f1a00640 1 -- 172.21.15.96:0/4262419486 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47ec0cb580 con 0x7f47ec0b49e0 2024-05-28T06:14:49.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.011+0000 7f47e3e00640 1 -- 172.21.15.96:0/4262419486 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 22) ==== 92213+0+0 (secure 0 0 0) 0x7f47dc01a070 con 0x7f47ec0b49e0 2024-05-28T06:14:49.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.012+0000 7f47e3e00640 1 --2- 172.21.15.96:0/4262419486 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f47c806d5e0 0x7f47c806faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:49.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.012+0000 7f47e3e00640 1 -- 172.21.15.96:0/4262419486 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 4885+0+0 (secure 0 0 0) 0x7f47dc0898b0 con 0x7f47ec0b49e0 2024-05-28T06:14:49.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.013+0000 7f47eb400640 1 --2- 172.21.15.96:0/4262419486 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f47c806d5e0 0x7f47c806faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:49.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.014+0000 7f47eb400640 1 --2- 172.21.15.96:0/4262419486 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f47c806d5e0 0x7f47c806faa0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f47d4007fd0 tx=0x7f47d4001680 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:49.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.016+0000 7f47e3e00640 1 -- 172.21.15.96:0/4262419486 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f47dc059790 con 0x7f47ec0b49e0 2024-05-28T06:14:49.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.215+0000 7f47f1a00640 1 -- 172.21.15.96:0/4262419486 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f47ec0689e0 con 0x7f47ec0b49e0 2024-05-28T06:14:49.218 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.216+0000 7f47e3e00640 1 -- 172.21.15.96:0/4262419486 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v70) ==== 93+0+10 (secure 0 0 0) 0x7f47dc059090 con 0x7f47ec0b49e0 2024-05-28T06:14:49.218 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 1 2024-05-28T06:14:49.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.219+0000 7f47f1a00640 1 -- 172.21.15.96:0/4262419486 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f47c806d5e0 msgr2=0x7f47c806faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:49.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.219+0000 7f47f1a00640 1 --2- 172.21.15.96:0/4262419486 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f47c806d5e0 0x7f47c806faa0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f47d4007fd0 tx=0x7f47d4001680 comp rx=0 tx=0).stop 2024-05-28T06:14:49.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.219+0000 7f47f1a00640 1 -- 172.21.15.96:0/4262419486 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f47ec0b49e0 msgr2=0x7f47ec15f200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:49.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.219+0000 7f47f1a00640 1 --2- 172.21.15.96:0/4262419486 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f47ec0b49e0 0x7f47ec15f200 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f47dc00c430 tx=0x7f47dc00c8f0 comp rx=0 tx=0).stop 2024-05-28T06:14:49.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.219+0000 7f47f1a00640 1 -- 172.21.15.96:0/4262419486 shutdown_connections 2024-05-28T06:14:49.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.219+0000 7f47f1a00640 1 --2- 172.21.15.96:0/4262419486 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f47c806d5e0 0x7f47c806faa0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:49.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.219+0000 7f47f1a00640 1 --2- 172.21.15.96:0/4262419486 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f47ec161ed0 0x7f47ec0b6270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:49.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.219+0000 7f47f1a00640 1 --2- 172.21.15.96:0/4262419486 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f47ec0cf470 0x7f47ec09cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:49.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.219+0000 7f47f1a00640 1 --2- 172.21.15.96:0/4262419486 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f47ec0b49e0 0x7f47ec15f200 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:49.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.219+0000 7f47f1a00640 1 -- 172.21.15.96:0/4262419486 >> 172.21.15.96:0/4262419486 conn(0x7f47ec0b3ec0 msgr2=0x7f47ec06d8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:49.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.219+0000 7f47f1a00640 1 -- 172.21.15.96:0/4262419486 shutdown_connections 2024-05-28T06:14:49.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.219+0000 7f47f1a00640 1 -- 172.21.15.96:0/4262419486 wait complete. 2024-05-28T06:14:49.240 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-05-28T06:14:49.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.435+0000 7fd65aa00640 1 Processor -- start 2024-05-28T06:14:49.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.436+0000 7fd65aa00640 1 -- start start 2024-05-28T06:14:49.439 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.437+0000 7fd65aa00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fd6540d2db0 0x7fd6540b3b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:49.439 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.437+0000 7fd65aa00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd654163500 con 0x7fd65415ed20 2024-05-28T06:14:49.439 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.438+0000 7fd65aa00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd654162740 con 0x7fd6540d4d60 2024-05-28T06:14:49.439 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.438+0000 7fd65aa00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fd6540d5540 con 0x7fd6540d2db0 2024-05-28T06:14:49.439 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.438+0000 7fd658a00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fd65415ed20 0x7fd65415f1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:52532/0 (socket says 172.21.15.96:52532) 2024-05-28T06:14:49.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.438+0000 7fd658a00640 1 -- 172.21.15.96:0/3901440781 learned_addr learned my addr 172.21.15.96:0/3901440781 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:14:49.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.438+0000 7fd653e00640 1 --2- 172.21.15.96:0/3901440781 >> v2:172.21.15.96:3300/0 conn(0x7fd6540d2db0 0x7fd6540b3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:49.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.438+0000 7fd653400640 1 -- 172.21.15.96:0/3901440781 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 305480122 0 0) 0x7fd654163500 con 0x7fd65415ed20 2024-05-28T06:14:49.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.438+0000 7fd653400640 1 -- 172.21.15.96:0/3901440781 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd638003520 con 0x7fd65415ed20 2024-05-28T06:14:49.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.438+0000 7fd653e00640 1 -- 172.21.15.96:0/3901440781 >> v1:172.21.15.123:6789/0 conn(0x7fd6540d4d60 legacy=0x7fd6540d3190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:49.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.438+0000 7fd653e00640 1 -- 172.21.15.96:0/3901440781 >> v1:172.21.15.96:6789/0 conn(0x7fd65415ed20 legacy=0x7fd65415f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:49.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.438+0000 7fd653e00640 1 -- 172.21.15.96:0/3901440781 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd654162a20 con 0x7fd6540d2db0 2024-05-28T06:14:49.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.439+0000 7fd653e00640 1 --2- 172.21.15.96:0/3901440781 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd6540d2db0 0x7fd6540b3b50 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7fd6440097a0 tx=0x7fd64402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70928cd5886fb1ee server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:49.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.440+0000 7fd653400640 1 -- 172.21.15.96:0/3901440781 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd644037070 con 0x7fd6540d2db0 2024-05-28T06:14:49.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.440+0000 7fd653400640 1 -- 172.21.15.96:0/3901440781 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd644032440 con 0x7fd6540d2db0 2024-05-28T06:14:49.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.440+0000 7fd653400640 1 -- 172.21.15.96:0/3901440781 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6440327f0 con 0x7fd6540d2db0 2024-05-28T06:14:49.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.440+0000 7fd65aa00640 1 -- 172.21.15.96:0/3901440781 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd6540d2db0 msgr2=0x7fd6540b3b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:49.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.440+0000 7fd65aa00640 1 --2- 172.21.15.96:0/3901440781 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd6540d2db0 0x7fd6540b3b50 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7fd6440097a0 tx=0x7fd64402a1a0 comp rx=0 tx=0).stop 2024-05-28T06:14:49.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.440+0000 7fd65aa00640 1 -- 172.21.15.96:0/3901440781 shutdown_connections 2024-05-28T06:14:49.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.440+0000 7fd65aa00640 1 --2- 172.21.15.96:0/3901440781 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd6540d2db0 0x7fd6540b3b50 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:49.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.440+0000 7fd65aa00640 1 -- 172.21.15.96:0/3901440781 >> 172.21.15.96:0/3901440781 conn(0x7fd65408bfc0 msgr2=0x7fd6540d4650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:49.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.441+0000 7fd65aa00640 1 -- 172.21.15.96:0/3901440781 shutdown_connections 2024-05-28T06:14:49.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.441+0000 7fd65aa00640 1 -- 172.21.15.96:0/3901440781 wait complete. 2024-05-28T06:14:49.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.442+0000 7fd65aa00640 1 Processor -- start 2024-05-28T06:14:49.444 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.442+0000 7fd65aa00640 1 -- start start 2024-05-28T06:14:49.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.443+0000 7fd65aa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd6540d2db0 0x7fd6540cdd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:49.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.443+0000 7fd65aa00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd6540d4d60 0x7fd6541f8de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:49.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.443+0000 7fd65aa00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd65415ed20 0x7fd6540b9790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:49.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.443+0000 7fd65aa00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd6540d51a0 con 0x7fd6540d2db0 2024-05-28T06:14:49.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.443+0000 7fd65aa00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd654162fe0 con 0x7fd65415ed20 2024-05-28T06:14:49.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.443+0000 7fd65aa00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fd6540d5320 con 0x7fd6540d4d60 2024-05-28T06:14:49.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.444+0000 7fd653e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd6540d2db0 0x7fd6540cdd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:49.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.444+0000 7fd653e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd6540d2db0 0x7fd6540cdd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54472/0 (socket says 172.21.15.96:54472) 2024-05-28T06:14:49.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.444+0000 7fd653e00640 1 -- 172.21.15.96:0/3205921498 learned_addr learned my addr 172.21.15.96:0/3205921498 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:49.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.444+0000 7fd64be00640 1 --2- 172.21.15.96:0/3205921498 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd6540d4d60 0x7fd6541f8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:49.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.444+0000 7fd653e00640 1 -- 172.21.15.96:0/3205921498 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd6540d4d60 msgr2=0x7fd6541f8de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:49.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.444+0000 7fd653e00640 1 --2- 172.21.15.96:0/3205921498 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd6540d4d60 0x7fd6541f8de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:49.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.444+0000 7fd653e00640 1 -- 172.21.15.96:0/3205921498 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd65415ed20 msgr2=0x7fd6540b9790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:49.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.444+0000 7fd653e00640 1 --2- 172.21.15.96:0/3205921498 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd65415ed20 0x7fd6540b9790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:49.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.444+0000 7fd653e00640 1 -- 172.21.15.96:0/3205921498 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6541f9320 con 0x7fd6540d2db0 2024-05-28T06:14:49.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.444+0000 7fd653e00640 1 --2- 172.21.15.96:0/3205921498 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd6540d2db0 0x7fd6540cdd10 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7fd6440097a0 tx=0x7fd6440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:49.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.444+0000 7fd650c00640 1 -- 172.21.15.96:0/3205921498 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd644037040 con 0x7fd6540d2db0 2024-05-28T06:14:49.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.445+0000 7fd650c00640 1 -- 172.21.15.96:0/3205921498 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd644004230 con 0x7fd6540d2db0 2024-05-28T06:14:49.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.445+0000 7fd650c00640 1 -- 172.21.15.96:0/3205921498 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd644003610 con 0x7fd6540d2db0 2024-05-28T06:14:49.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.445+0000 7fd65aa00640 1 -- 172.21.15.96:0/3205921498 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6540b8560 con 0x7fd6540d2db0 2024-05-28T06:14:49.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.445+0000 7fd65aa00640 1 -- 172.21.15.96:0/3205921498 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6540ccf50 con 0x7fd6540d2db0 2024-05-28T06:14:49.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.446+0000 7fd65aa00640 1 -- 172.21.15.96:0/3205921498 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6540b3920 con 0x7fd6540d2db0 2024-05-28T06:14:49.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.447+0000 7fd650c00640 1 -- 172.21.15.96:0/3205921498 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 22) ==== 92213+0+0 (secure 0 0 0) 0x7fd6440037b0 con 0x7fd6540d2db0 2024-05-28T06:14:49.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.448+0000 7fd650c00640 1 --2- 172.21.15.96:0/3205921498 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd63006d5e0 0x7fd63006faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:49.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.448+0000 7fd650c00640 1 -- 172.21.15.96:0/3205921498 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 4885+0+0 (secure 0 0 0) 0x7fd6440ab730 con 0x7fd6540d2db0 2024-05-28T06:14:49.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.448+0000 7fd64be00640 1 --2- 172.21.15.96:0/3205921498 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd63006d5e0 0x7fd63006faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:49.451 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.450+0000 7fd64be00640 1 --2- 172.21.15.96:0/3205921498 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd63006d5e0 0x7fd63006faa0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fd6540cffb0 tx=0x7fd63c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:49.454 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.452+0000 7fd650c00640 1 -- 172.21.15.96:0/3205921498 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd6540b3920 con 0x7fd6540d2db0 2024-05-28T06:14:49.640 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.638+0000 7fd65aa00640 1 -- 172.21.15.96:0/3205921498 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fd6540689e0 con 0x7fd6540d2db0 2024-05-28T06:14:49.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.639+0000 7fd650c00640 1 -- 172.21.15.96:0/3205921498 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v70) ==== 84+0+65 (secure 0 0 0) 0x7fd6541f9320 con 0x7fd6540d2db0 2024-05-28T06:14:49.641 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:14:49.641 INFO:teuthology.orchestra.run.smithi096.stdout:[ 2024-05-28T06:14:49.641 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:14:49.641 INFO:teuthology.orchestra.run.smithi096.stdout: "poolnum": 1, 2024-05-28T06:14:49.641 INFO:teuthology.orchestra.run.smithi096.stdout: "poolname": ".mgr" 2024-05-28T06:14:49.641 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:14:49.641 INFO:teuthology.orchestra.run.smithi096.stdout:] 2024-05-28T06:14:49.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.641+0000 7fd65aa00640 1 -- 172.21.15.96:0/3205921498 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd63006d5e0 msgr2=0x7fd63006faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:49.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.641+0000 7fd65aa00640 1 --2- 172.21.15.96:0/3205921498 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd63006d5e0 0x7fd63006faa0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fd6540cffb0 tx=0x7fd63c009040 comp rx=0 tx=0).stop 2024-05-28T06:14:49.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.641+0000 7fd65aa00640 1 -- 172.21.15.96:0/3205921498 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd6540d2db0 msgr2=0x7fd6540cdd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:49.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.641+0000 7fd65aa00640 1 --2- 172.21.15.96:0/3205921498 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd6540d2db0 0x7fd6540cdd10 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7fd6440097a0 tx=0x7fd6440046c0 comp rx=0 tx=0).stop 2024-05-28T06:14:49.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.641+0000 7fd65aa00640 1 -- 172.21.15.96:0/3205921498 shutdown_connections 2024-05-28T06:14:49.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.641+0000 7fd65aa00640 1 --2- 172.21.15.96:0/3205921498 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd63006d5e0 0x7fd63006faa0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:49.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.641+0000 7fd65aa00640 1 --2- 172.21.15.96:0/3205921498 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd65415ed20 0x7fd6540b9790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:49.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.641+0000 7fd65aa00640 1 --2- 172.21.15.96:0/3205921498 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd6540d4d60 0x7fd6541f8de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:49.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.641+0000 7fd65aa00640 1 --2- 172.21.15.96:0/3205921498 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd6540d2db0 0x7fd6540cdd10 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:49.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.641+0000 7fd65aa00640 1 -- 172.21.15.96:0/3205921498 >> 172.21.15.96:0/3205921498 conn(0x7fd65408bfc0 msgr2=0x7fd6540b3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:49.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.642+0000 7fd65aa00640 1 -- 172.21.15.96:0/3205921498 shutdown_connections 2024-05-28T06:14:49.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.642+0000 7fd65aa00640 1 -- 172.21.15.96:0/3205921498 wait complete. 2024-05-28T06:14:49.662 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-05-28T06:14:49.662 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-05-28T06:14:49.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.862+0000 7f5217e00640 1 Processor -- start 2024-05-28T06:14:49.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.862+0000 7f5217e00640 1 -- start start 2024-05-28T06:14:49.866 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.864+0000 7f5217e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f5210162e90 0x7f52100ce540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:49.866 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.864+0000 7f5217e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f5210163270 0x7f52100cd630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:49.866 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.864+0000 7f5217e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f521006e1b0 0x7f52100c3300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:49.866 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.864+0000 7f5217e00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f52100b3b80 con 0x7f5210162e90 2024-05-28T06:14:49.866 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.864+0000 7f5217e00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f52100b3880 con 0x7f5210163270 2024-05-28T06:14:49.866 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.864+0000 7f5217e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f52100ceb20 con 0x7f521006e1b0 2024-05-28T06:14:49.866 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.864+0000 7f5215400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f5210162e90 0x7f52100ce540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:49.866 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.864+0000 7f5215400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f5210162e90 0x7f52100ce540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54478/0 (socket says 172.21.15.96:54478) 2024-05-28T06:14:49.866 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.864+0000 7f5215400640 1 -- 172.21.15.96:0/4276912548 learned_addr learned my addr 172.21.15.96:0/4276912548 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:49.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.865+0000 7f5215e00640 1 --2- 172.21.15.96:0/4276912548 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f521006e1b0 0x7f52100c3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:49.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.865+0000 7f5215400640 1 -- 172.21.15.96:0/4276912548 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f521006e1b0 msgr2=0x7f52100c3300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:49.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.865+0000 7f5215400640 1 --2- 172.21.15.96:0/4276912548 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f521006e1b0 0x7f52100c3300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:49.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.865+0000 7f5215400640 1 -- 172.21.15.96:0/4276912548 >> v2:172.21.15.123:3300/0 conn(0x7f5210163270 msgr2=0x7f52100cd630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:49.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.865+0000 7f5214a00640 1 --2- 172.21.15.96:0/4276912548 >> v2:172.21.15.123:3300/0 conn(0x7f5210163270 0x7f52100cd630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:49.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.865+0000 7f5215400640 1 --2- 172.21.15.96:0/4276912548 >> v2:172.21.15.123:3300/0 conn(0x7f5210163270 0x7f52100cd630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:49.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.865+0000 7f5215400640 1 -- 172.21.15.96:0/4276912548 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52100cee00 con 0x7f5210162e90 2024-05-28T06:14:49.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.865+0000 7f5215e00640 1 --2- 172.21.15.96:0/4276912548 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f521006e1b0 0x7f52100c3300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:14:49.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.866+0000 7f5215400640 1 --2- 172.21.15.96:0/4276912548 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5210162e90 0x7f52100ce540 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f52100c7230 tx=0x7f520402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fda9dfdf40adeb70 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:49.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.866+0000 7f520fe00640 1 -- 172.21.15.96:0/4276912548 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5204037070 con 0x7f5210162e90 2024-05-28T06:14:49.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.866+0000 7f520fe00640 1 -- 172.21.15.96:0/4276912548 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5204032440 con 0x7f5210162e90 2024-05-28T06:14:49.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.866+0000 7f520fe00640 1 -- 172.21.15.96:0/4276912548 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5204032810 con 0x7f5210162e90 2024-05-28T06:14:49.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.867+0000 7f5217e00640 1 -- 172.21.15.96:0/4276912548 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5210162e90 msgr2=0x7f52100ce540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:49.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.867+0000 7f5217e00640 1 --2- 172.21.15.96:0/4276912548 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5210162e90 0x7f52100ce540 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f52100c7230 tx=0x7f520402a1e0 comp rx=0 tx=0).stop 2024-05-28T06:14:49.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.867+0000 7f5217e00640 1 -- 172.21.15.96:0/4276912548 shutdown_connections 2024-05-28T06:14:49.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.867+0000 7f5217e00640 1 --2- 172.21.15.96:0/4276912548 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f521006e1b0 0x7f52100c3300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:49.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.867+0000 7f5217e00640 1 --2- 172.21.15.96:0/4276912548 >> v2:172.21.15.123:3300/0 conn(0x7f5210163270 0x7f52100cd630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:49.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.867+0000 7f5217e00640 1 --2- 172.21.15.96:0/4276912548 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5210162e90 0x7f52100ce540 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:49.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.867+0000 7f5217e00640 1 -- 172.21.15.96:0/4276912548 >> 172.21.15.96:0/4276912548 conn(0x7f52100b20a0 msgr2=0x7f52100b24b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:49.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.867+0000 7f5217e00640 1 -- 172.21.15.96:0/4276912548 shutdown_connections 2024-05-28T06:14:49.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.867+0000 7f5217e00640 1 -- 172.21.15.96:0/4276912548 wait complete. 2024-05-28T06:14:49.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.868+0000 7f5217e00640 1 Processor -- start 2024-05-28T06:14:49.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.869+0000 7f5217e00640 1 -- start start 2024-05-28T06:14:49.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.869+0000 7f5217e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f521006e1b0 0x7f52101f8a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:49.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.870+0000 7f5217e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5210162e90 0x7f521015fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:49.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.870+0000 7f5217e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5210163270 0x7f52100ae790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:49.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.870+0000 7f5217e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f52100cf260 con 0x7f5210162e90 2024-05-28T06:14:49.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.870+0000 7f5217e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f52100cf0e0 con 0x7f521006e1b0 2024-05-28T06:14:49.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.870+0000 7f5217e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f521006e640 con 0x7f5210163270 2024-05-28T06:14:49.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.870+0000 7f5214a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5210162e90 0x7f521015fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:49.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.870+0000 7f5214a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5210162e90 0x7f521015fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54492/0 (socket says 172.21.15.96:54492) 2024-05-28T06:14:49.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.870+0000 7f5214a00640 1 -- 172.21.15.96:0/3994089875 learned_addr learned my addr 172.21.15.96:0/3994089875 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:49.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.870+0000 7f5215400640 1 --2- 172.21.15.96:0/3994089875 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f521006e1b0 0x7f52101f8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:49.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.870+0000 7f5215e00640 1 --2- 172.21.15.96:0/3994089875 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5210163270 0x7f52100ae790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:49.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.870+0000 7f5214a00640 1 -- 172.21.15.96:0/3994089875 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5210163270 msgr2=0x7f52100ae790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:49.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.870+0000 7f5214a00640 1 --2- 172.21.15.96:0/3994089875 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5210163270 0x7f52100ae790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:49.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.870+0000 7f5214a00640 1 -- 172.21.15.96:0/3994089875 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f521006e1b0 msgr2=0x7f52101f8a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:49.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.870+0000 7f5214a00640 1 --2- 172.21.15.96:0/3994089875 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f521006e1b0 0x7f52101f8a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:49.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.870+0000 7f5214a00640 1 -- 172.21.15.96:0/3994089875 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5210072bf0 con 0x7f5210162e90 2024-05-28T06:14:49.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.870+0000 7f5215400640 1 --2- 172.21.15.96:0/3994089875 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f521006e1b0 0x7f52101f8a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:14:49.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.871+0000 7f5214a00640 1 --2- 172.21.15.96:0/3994089875 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5210162e90 0x7f521015fc50 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f51fc00c7f0 tx=0x7f51fc00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:49.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.871+0000 7f520d600640 1 -- 172.21.15.96:0/3994089875 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51fc015070 con 0x7f5210162e90 2024-05-28T06:14:49.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.871+0000 7f520d600640 1 -- 172.21.15.96:0/3994089875 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51fc004240 con 0x7f5210162e90 2024-05-28T06:14:49.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.871+0000 7f5217e00640 1 -- 172.21.15.96:0/3994089875 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5210072ed0 con 0x7f5210162e90 2024-05-28T06:14:49.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.871+0000 7f520d600640 1 -- 172.21.15.96:0/3994089875 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51fc007620 con 0x7f5210162e90 2024-05-28T06:14:49.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.871+0000 7f5217e00640 1 -- 172.21.15.96:0/3994089875 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5210073490 con 0x7f5210162e90 2024-05-28T06:14:49.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.872+0000 7f5217e00640 1 -- 172.21.15.96:0/3994089875 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52100757d0 con 0x7f5210162e90 2024-05-28T06:14:49.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.873+0000 7f520d600640 1 -- 172.21.15.96:0/3994089875 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 22) ==== 92213+0+0 (secure 0 0 0) 0x7f51fc0079d0 con 0x7f5210162e90 2024-05-28T06:14:49.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.874+0000 7f520d600640 1 --2- 172.21.15.96:0/3994089875 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f51f006d630 0x7f51f006faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:49.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.874+0000 7f520d600640 1 -- 172.21.15.96:0/3994089875 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 4885+0+0 (secure 0 0 0) 0x7f51fc08a780 con 0x7f5210162e90 2024-05-28T06:14:49.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.875+0000 7f5215400640 1 --2- 172.21.15.96:0/3994089875 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f51f006d630 0x7f51f006faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:49.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.876+0000 7f5215400640 1 --2- 172.21.15.96:0/3994089875 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f51f006d630 0x7f51f006faf0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f5204034000 tx=0x7f520402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:49.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:49.878+0000 7f520d600640 1 -- 172.21.15.96:0/3994089875 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f51fc05a580 con 0x7f5210162e90 2024-05-28T06:14:50.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:50.080+0000 7f5217e00640 1 -- 172.21.15.96:0/3994089875 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f52100689e0 con 0x7f5210162e90 2024-05-28T06:14:50.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:50.858+0000 7f520d600640 1 -- 172.21.15.96:0/3994089875 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v71) ==== 138+0+0 (secure 0 0 0) 0x7f51fc059e80 con 0x7f5210162e90 2024-05-28T06:14:50.860 INFO:teuthology.orchestra.run.smithi096.stderr:pool 'cephfs_metadata' created 2024-05-28T06:14:50.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:50.861+0000 7f5217e00640 1 -- 172.21.15.96:0/3994089875 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f51f006d630 msgr2=0x7f51f006faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:50.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:50.861+0000 7f5217e00640 1 --2- 172.21.15.96:0/3994089875 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f51f006d630 0x7f51f006faf0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f5204034000 tx=0x7f520402e000 comp rx=0 tx=0).stop 2024-05-28T06:14:50.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:50.861+0000 7f5217e00640 1 -- 172.21.15.96:0/3994089875 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5210162e90 msgr2=0x7f521015fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:50.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:50.861+0000 7f5217e00640 1 --2- 172.21.15.96:0/3994089875 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5210162e90 0x7f521015fc50 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f51fc00c7f0 tx=0x7f51fc00ccb0 comp rx=0 tx=0).stop 2024-05-28T06:14:50.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:50.861+0000 7f5217e00640 1 -- 172.21.15.96:0/3994089875 shutdown_connections 2024-05-28T06:14:50.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:50.861+0000 7f5217e00640 1 --2- 172.21.15.96:0/3994089875 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f51f006d630 0x7f51f006faf0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:50.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:50.861+0000 7f5217e00640 1 --2- 172.21.15.96:0/3994089875 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5210163270 0x7f52100ae790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:50.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:50.861+0000 7f5217e00640 1 --2- 172.21.15.96:0/3994089875 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5210162e90 0x7f521015fc50 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:50.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:50.861+0000 7f5217e00640 1 --2- 172.21.15.96:0/3994089875 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f521006e1b0 0x7f52101f8a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:50.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:50.861+0000 7f5217e00640 1 -- 172.21.15.96:0/3994089875 >> 172.21.15.96:0/3994089875 conn(0x7f52100b20a0 msgr2=0x7f52100cb670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:50.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:50.862+0000 7f5217e00640 1 -- 172.21.15.96:0/3994089875 shutdown_connections 2024-05-28T06:14:50.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:50.862+0000 7f5217e00640 1 -- 172.21.15.96:0/3994089875 wait complete. 2024-05-28T06:14:50.890 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-05-28T06:14:51.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:51.082+0000 7f861ae00640 1 Processor -- start 2024-05-28T06:14:51.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:51.082+0000 7f861ae00640 1 -- start start 2024-05-28T06:14:51.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:51.084+0000 7f861ae00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f8614161e70 0x7f86140b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:51.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:51.084+0000 7f861ae00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86140b3d40 con 0x7f8614161a90 2024-05-28T06:14:51.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:51.084+0000 7f861ae00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86140b4020 con 0x7f86140b3800 2024-05-28T06:14:51.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:51.084+0000 7f861ae00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f86140b4300 con 0x7f8614161e70 2024-05-28T06:14:51.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:51.084+0000 7f8613e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f8614161a90 0x7f861406e470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:52538/0 (socket says 172.21.15.96:52538) 2024-05-28T06:14:51.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:51.084+0000 7f8613e00640 1 -- 172.21.15.96:0/4141050706 learned_addr learned my addr 172.21.15.96:0/4141050706 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:14:51.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:51.084+0000 7f8613400640 1 --2- 172.21.15.96:0/4141050706 >> v2:172.21.15.96:3300/0 conn(0x7f8614161e70 0x7f86140b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:51.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:51.084+0000 7f8612a00640 1 -- 172.21.15.96:0/4141050706 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3820009363 0 0) 0x7f86140b3d40 con 0x7f8614161a90 2024-05-28T06:14:51.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:51.085+0000 7f8612a00640 1 -- 172.21.15.96:0/4141050706 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f85f8003520 con 0x7f8614161a90 2024-05-28T06:14:51.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:51.085+0000 7f8613400640 1 -- 172.21.15.96:0/4141050706 >> v1:172.21.15.123:6789/0 conn(0x7f86140b3800 legacy=0x7f86140ad2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:51.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:51.085+0000 7f8612a00640 1 -- 172.21.15.96:0/4141050706 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1821210368 0 0) 0x7f85f8003520 con 0x7f8614161a90 2024-05-28T06:14:51.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:51.085+0000 7f8613400640 1 -- 172.21.15.96:0/4141050706 >> v1:172.21.15.96:6789/0 conn(0x7f8614161a90 legacy=0x7f861406e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:51.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:51.085+0000 7f8613400640 1 -- 172.21.15.96:0/4141050706 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f861406dc70 con 0x7f8614161e70 2024-05-28T06:14:51.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:51.086+0000 7f8613400640 1 --2- 172.21.15.96:0/4141050706 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8614161e70 0x7f86140b32c0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f86000097a0 tx=0x7f860002a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65ed88cd83ac64b3 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:51.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:51.086+0000 7f8612a00640 1 -- 172.21.15.96:0/4141050706 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8600037070 con 0x7f8614161e70 2024-05-28T06:14:51.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:51.086+0000 7f8612a00640 1 -- 172.21.15.96:0/4141050706 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f860002ac50 con 0x7f8614161e70 2024-05-28T06:14:51.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:51.086+0000 7f8612a00640 1 -- 172.21.15.96:0/4141050706 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8604001670 con 0x7f8614161e70 2024-05-28T06:14:51.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:51.087+0000 7f861ae00640 1 -- 172.21.15.96:0/4141050706 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8614161e70 msgr2=0x7f86140b32c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:51.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:51.087+0000 7f861ae00640 1 --2- 172.21.15.96:0/4141050706 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8614161e70 0x7f86140b32c0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f86000097a0 tx=0x7f860002a1c0 comp rx=0 tx=0).stop 2024-05-28T06:14:51.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:51.087+0000 7f861ae00640 1 -- 172.21.15.96:0/4141050706 shutdown_connections 2024-05-28T06:14:51.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:51.087+0000 7f861ae00640 1 --2- 172.21.15.96:0/4141050706 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8614161e70 0x7f86140b32c0 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:51.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:51.087+0000 7f861ae00640 1 -- 172.21.15.96:0/4141050706 >> 172.21.15.96:0/4141050706 conn(0x7f861406bd20 msgr2=0x7f861406c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:51.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:51.087+0000 7f861ae00640 1 -- 172.21.15.96:0/4141050706 shutdown_connections 2024-05-28T06:14:51.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:51.087+0000 7f861ae00640 1 -- 172.21.15.96:0/4141050706 wait complete. 2024-05-28T06:14:51.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:51.089+0000 7f861ae00640 1 Processor -- start 2024-05-28T06:14:51.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:51.089+0000 7f861ae00640 1 -- start start 2024-05-28T06:14:51.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:51.090+0000 7f861ae00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f86140b3800 0x7f86141608d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:51.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:51.090+0000 7f861ae00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8614161a90 0x7f8614160e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:51.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:51.090+0000 7f861ae00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8614161e70 0x7f861414eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:51.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:51.090+0000 7f861ae00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f861406eb80 con 0x7f86140b3800 2024-05-28T06:14:51.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:51.090+0000 7f861ae00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f86140b5020 con 0x7f8614161a90 2024-05-28T06:14:51.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:51.090+0000 7f861ae00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f86140ad9b0 con 0x7f8614161e70 2024-05-28T06:14:51.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:51.090+0000 7f8613e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f86140b3800 0x7f86141608d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:51.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:51.090+0000 7f8613e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f86140b3800 0x7f86141608d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54506/0 (socket says 172.21.15.96:54506) 2024-05-28T06:14:51.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:51.090+0000 7f8613e00640 1 -- 172.21.15.96:0/1440974511 learned_addr learned my addr 172.21.15.96:0/1440974511 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:51.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:51.091+0000 7f8613e00640 1 -- 172.21.15.96:0/1440974511 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8614161e70 msgr2=0x7f861414eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:51.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:51.091+0000 7f8613400640 1 --2- 172.21.15.96:0/1440974511 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8614161a90 0x7f8614160e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:51.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:51.091+0000 7f8618e00640 1 --2- 172.21.15.96:0/1440974511 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8614161e70 0x7f861414eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:51.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:51.091+0000 7f8613e00640 1 --2- 172.21.15.96:0/1440974511 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8614161e70 0x7f861414eb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:51.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:51.091+0000 7f8613e00640 1 -- 172.21.15.96:0/1440974511 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8614161a90 msgr2=0x7f8614160e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:51.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:51.091+0000 7f8613e00640 1 --2- 172.21.15.96:0/1440974511 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8614161a90 0x7f8614160e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:51.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:51.091+0000 7f8613e00640 1 -- 172.21.15.96:0/1440974511 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f861414f870 con 0x7f86140b3800 2024-05-28T06:14:51.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:51.091+0000 7f8613400640 1 --2- 172.21.15.96:0/1440974511 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8614161a90 0x7f8614160e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:14:51.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:51.091+0000 7f8613e00640 1 --2- 172.21.15.96:0/1440974511 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f86140b3800 0x7f86141608d0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f8604001eb0 tx=0x7f860400cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:51.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:51.091+0000 7f860be00640 1 -- 172.21.15.96:0/1440974511 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8604017070 con 0x7f86140b3800 2024-05-28T06:14:51.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:51.091+0000 7f860be00640 1 -- 172.21.15.96:0/1440974511 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8604005ad0 con 0x7f86140b3800 2024-05-28T06:14:51.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:51.091+0000 7f860be00640 1 -- 172.21.15.96:0/1440974511 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8604007070 con 0x7f86140b3800 2024-05-28T06:14:51.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:51.091+0000 7f861ae00640 1 -- 172.21.15.96:0/1440974511 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f861414fb50 con 0x7f86140b3800 2024-05-28T06:14:51.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:51.092+0000 7f861ae00640 1 -- 172.21.15.96:0/1440974511 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8614150110 con 0x7f86140b3800 2024-05-28T06:14:51.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:51.093+0000 7f861ae00640 1 -- 172.21.15.96:0/1440974511 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86140689e0 con 0x7f86140b3800 2024-05-28T06:14:51.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:51.094+0000 7f860be00640 1 -- 172.21.15.96:0/1440974511 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 22) ==== 92213+0+0 (secure 0 0 0) 0x7f860401a070 con 0x7f86140b3800 2024-05-28T06:14:51.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:51.095+0000 7f860be00640 1 --2- 172.21.15.96:0/1440974511 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f85f406d5e0 0x7f85f406faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:51.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:51.095+0000 7f860be00640 1 -- 172.21.15.96:0/1440974511 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 5255+0+0 (secure 0 0 0) 0x7f8604089f30 con 0x7f86140b3800 2024-05-28T06:14:51.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:51.096+0000 7f8613400640 1 --2- 172.21.15.96:0/1440974511 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f85f406d5e0 0x7f85f406faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:51.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:51.097+0000 7f8613400640 1 --2- 172.21.15.96:0/1440974511 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f85f406d5e0 0x7f85f406faa0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f8600034000 tx=0x7f860002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:51.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:51.101+0000 7f860be00640 1 -- 172.21.15.96:0/1440974511 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8604021020 con 0x7f86140b3800 2024-05-28T06:14:51.305 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:51.304+0000 7f861ae00640 1 -- 172.21.15.96:0/1440974511 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f86140afa60 con 0x7f86140b3800 2024-05-28T06:14:51.866 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:51.864+0000 7f860be00640 1 -- 172.21.15.96:0/1440974511 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v72) ==== 170+0+0 (secure 0 0 0) 0x7f8604059df0 con 0x7f86140b3800 2024-05-28T06:14:51.866 INFO:teuthology.orchestra.run.smithi096.stderr:pool 'cephfs_data' created 2024-05-28T06:14:51.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:51.868+0000 7f861ae00640 1 -- 172.21.15.96:0/1440974511 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f85f406d5e0 msgr2=0x7f85f406faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:51.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:51.868+0000 7f861ae00640 1 --2- 172.21.15.96:0/1440974511 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f85f406d5e0 0x7f85f406faa0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f8600034000 tx=0x7f860002e000 comp rx=0 tx=0).stop 2024-05-28T06:14:51.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:51.868+0000 7f861ae00640 1 -- 172.21.15.96:0/1440974511 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f86140b3800 msgr2=0x7f86141608d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:51.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:51.868+0000 7f861ae00640 1 --2- 172.21.15.96:0/1440974511 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f86140b3800 0x7f86141608d0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f8604001eb0 tx=0x7f860400cbe0 comp rx=0 tx=0).stop 2024-05-28T06:14:51.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:51.869+0000 7f861ae00640 1 -- 172.21.15.96:0/1440974511 shutdown_connections 2024-05-28T06:14:51.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:51.869+0000 7f861ae00640 1 --2- 172.21.15.96:0/1440974511 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f85f406d5e0 0x7f85f406faa0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:51.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:51.869+0000 7f861ae00640 1 --2- 172.21.15.96:0/1440974511 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8614161e70 0x7f861414eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:51.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:51.869+0000 7f861ae00640 1 --2- 172.21.15.96:0/1440974511 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8614161a90 0x7f8614160e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:51.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:51.869+0000 7f861ae00640 1 --2- 172.21.15.96:0/1440974511 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f86140b3800 0x7f86141608d0 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:51.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:51.869+0000 7f861ae00640 1 -- 172.21.15.96:0/1440974511 >> 172.21.15.96:0/1440974511 conn(0x7f861406bd20 msgr2=0x7f861406c100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:51.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:51.869+0000 7f861ae00640 1 -- 172.21.15.96:0/1440974511 shutdown_connections 2024-05-28T06:14:51.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:51.869+0000 7f861ae00640 1 -- 172.21.15.96:0/1440974511 wait complete. 2024-05-28T06:14:51.902 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-05-28T06:14:52.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:52.077+0000 7fc0f9a00640 1 Processor -- start 2024-05-28T06:14:52.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:52.077+0000 7fc0f9a00640 1 -- start start 2024-05-28T06:14:52.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:52.079+0000 7fc0f9a00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fc0f40d0640 0x7fc0f415d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:52.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:52.079+0000 7fc0f9a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fc0f40d0a20 0x7fc0f40cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:52.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:52.079+0000 7fc0f9a00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0f415cb10 con 0x7fc0f40b32c0 2024-05-28T06:14:52.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:52.079+0000 7fc0f9a00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fc0f40adbb0 con 0x7fc0f40d0a20 2024-05-28T06:14:52.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:52.079+0000 7fc0f9a00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fc0f415db10 con 0x7fc0f40d0640 2024-05-28T06:14:52.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:52.079+0000 7fc0f2000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fc0f40d0a20 0x7fc0f40cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:52.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:52.079+0000 7fc0f2000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fc0f40d0a20 0x7fc0f40cf950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54508/0 (socket says 172.21.15.96:54508) 2024-05-28T06:14:52.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:52.079+0000 7fc0f2000640 1 -- 172.21.15.96:0/1755825212 learned_addr learned my addr 172.21.15.96:0/1755825212 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:52.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:52.080+0000 7fc0f2000640 1 -- 172.21.15.96:0/1755825212 >> v2:172.21.15.123:3300/0 conn(0x7fc0f40d0640 msgr2=0x7fc0f415d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:52.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:52.080+0000 7fc0f2000640 1 --2- 172.21.15.96:0/1755825212 >> v2:172.21.15.123:3300/0 conn(0x7fc0f40d0640 0x7fc0f415d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:52.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:52.080+0000 7fc0f2000640 1 -- 172.21.15.96:0/1755825212 >> v1:172.21.15.123:6789/0 conn(0x7fc0f40b32c0 legacy=0x7fc0f40b3730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:52.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:52.080+0000 7fc0f2000640 1 -- 172.21.15.96:0/1755825212 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0f40ad5f0 con 0x7fc0f40d0a20 2024-05-28T06:14:52.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:52.081+0000 7fc0f2000640 1 --2- 172.21.15.96:0/1755825212 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc0f40d0a20 0x7fc0f40cf950 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7fc0f4074150 tx=0x7fc0dc02a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7eab94dbeac765c5 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:52.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:52.081+0000 7fc0f1600640 1 -- 172.21.15.96:0/1755825212 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0dc036070 con 0x7fc0f40d0a20 2024-05-28T06:14:52.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:52.081+0000 7fc0f1600640 1 -- 172.21.15.96:0/1755825212 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0dc031600 con 0x7fc0f40d0a20 2024-05-28T06:14:52.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:52.081+0000 7fc0f1600640 1 -- 172.21.15.96:0/1755825212 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0dc02ae60 con 0x7fc0f40d0a20 2024-05-28T06:14:52.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:52.082+0000 7fc0f9a00640 1 -- 172.21.15.96:0/1755825212 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc0f40d0a20 msgr2=0x7fc0f40cf950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:52.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:52.082+0000 7fc0f9a00640 1 --2- 172.21.15.96:0/1755825212 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc0f40d0a20 0x7fc0f40cf950 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7fc0f4074150 tx=0x7fc0dc02a680 comp rx=0 tx=0).stop 2024-05-28T06:14:52.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:52.082+0000 7fc0f9a00640 1 -- 172.21.15.96:0/1755825212 shutdown_connections 2024-05-28T06:14:52.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:52.082+0000 7fc0f9a00640 1 --2- 172.21.15.96:0/1755825212 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc0f40d0a20 0x7fc0f40cf950 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:52.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:52.082+0000 7fc0f9a00640 1 --2- 172.21.15.96:0/1755825212 >> v2:172.21.15.123:3300/0 conn(0x7fc0f40d0640 0x7fc0f415d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:52.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:52.082+0000 7fc0f9a00640 1 -- 172.21.15.96:0/1755825212 >> 172.21.15.96:0/1755825212 conn(0x7fc0f406bd20 msgr2=0x7fc0f406c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:52.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:52.082+0000 7fc0f9a00640 1 -- 172.21.15.96:0/1755825212 shutdown_connections 2024-05-28T06:14:52.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:52.082+0000 7fc0f9a00640 1 -- 172.21.15.96:0/1755825212 wait complete. 2024-05-28T06:14:52.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:52.084+0000 7fc0f9a00640 1 Processor -- start 2024-05-28T06:14:52.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:52.084+0000 7fc0f9a00640 1 -- start start 2024-05-28T06:14:52.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:52.085+0000 7fc0f9a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc0f40b32c0 0x7fc0f41f4940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:52.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:52.085+0000 7fc0f9a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc0f40d0640 0x7fc0f40b15d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:52.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:52.085+0000 7fc0f9a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc0f40d0a20 0x7fc0f414f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:52.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:52.085+0000 7fc0f9a00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc0f415d0b0 con 0x7fc0f40d0640 2024-05-28T06:14:52.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:52.085+0000 7fc0f9a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc0f415cdd0 con 0x7fc0f40d0a20 2024-05-28T06:14:52.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:52.085+0000 7fc0f9a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fc0f40b3e40 con 0x7fc0f40b32c0 2024-05-28T06:14:52.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:52.085+0000 7fc0f2000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc0f40d0640 0x7fc0f40b15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:52.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:52.085+0000 7fc0f2000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc0f40d0640 0x7fc0f40b15d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54524/0 (socket says 172.21.15.96:54524) 2024-05-28T06:14:52.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:52.085+0000 7fc0f2000640 1 -- 172.21.15.96:0/972698457 learned_addr learned my addr 172.21.15.96:0/972698457 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:52.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:52.086+0000 7fc0f2000640 1 -- 172.21.15.96:0/972698457 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc0f40b32c0 msgr2=0x7fc0f41f4940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:52.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:52.086+0000 7fc0f3400640 1 --2- 172.21.15.96:0/972698457 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc0f40d0a20 0x7fc0f414f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:52.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:52.086+0000 7fc0f2a00640 1 --2- 172.21.15.96:0/972698457 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc0f40b32c0 0x7fc0f41f4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:52.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:52.086+0000 7fc0f2000640 1 --2- 172.21.15.96:0/972698457 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc0f40b32c0 0x7fc0f41f4940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:52.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:52.086+0000 7fc0f2000640 1 -- 172.21.15.96:0/972698457 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc0f40d0a20 msgr2=0x7fc0f414f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:52.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:52.086+0000 7fc0f2000640 1 --2- 172.21.15.96:0/972698457 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc0f40d0a20 0x7fc0f414f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:52.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:52.086+0000 7fc0f2000640 1 -- 172.21.15.96:0/972698457 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0f4143660 con 0x7fc0f40d0640 2024-05-28T06:14:52.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:52.086+0000 7fc0f2000640 1 --2- 172.21.15.96:0/972698457 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc0f40d0640 0x7fc0f40b15d0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7fc0dc034040 tx=0x7fc0dc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:52.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:52.086+0000 7fc0eaa00640 1 -- 172.21.15.96:0/972698457 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0dc036050 con 0x7fc0f40d0640 2024-05-28T06:14:52.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:52.086+0000 7fc0eaa00640 1 -- 172.21.15.96:0/972698457 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0dc042070 con 0x7fc0f40d0640 2024-05-28T06:14:52.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:52.086+0000 7fc0eaa00640 1 -- 172.21.15.96:0/972698457 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0dc009510 con 0x7fc0f40d0640 2024-05-28T06:14:52.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:52.086+0000 7fc0f9a00640 1 -- 172.21.15.96:0/972698457 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0f4143940 con 0x7fc0f40d0640 2024-05-28T06:14:52.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:52.087+0000 7fc0f9a00640 1 -- 172.21.15.96:0/972698457 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0f4143f00 con 0x7fc0f40d0640 2024-05-28T06:14:52.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:52.088+0000 7fc0f9a00640 1 -- 172.21.15.96:0/972698457 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0f41442d0 con 0x7fc0f40d0640 2024-05-28T06:14:52.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:52.089+0000 7fc0eaa00640 1 -- 172.21.15.96:0/972698457 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 22) ==== 92213+0+0 (secure 0 0 0) 0x7fc0dc007990 con 0x7fc0f40d0640 2024-05-28T06:14:52.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:52.090+0000 7fc0eaa00640 1 --2- 172.21.15.96:0/972698457 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc0d006d610 0x7fc0d006fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:52.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:52.090+0000 7fc0eaa00640 1 -- 172.21.15.96:0/972698457 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 5637+0+0 (secure 0 0 0) 0x7fc0dc0ab430 con 0x7fc0f40d0640 2024-05-28T06:14:52.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:52.091+0000 7fc0f2a00640 1 --2- 172.21.15.96:0/972698457 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc0d006d610 0x7fc0d006fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:52.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:52.092+0000 7fc0f2a00640 1 --2- 172.21.15.96:0/972698457 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc0d006d610 0x7fc0d006fad0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fc0e400a810 tx=0x7fc0e4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:52.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:52.095+0000 7fc0eaa00640 1 -- 172.21.15.96:0/972698457 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc0f41442d0 con 0x7fc0f40d0640 2024-05-28T06:14:52.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:52.320+0000 7fc0f9a00640 1 -- 172.21.15.96:0/972698457 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fc0f415df90 con 0x7fc0f40d0640 2024-05-28T06:14:52.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:52.886+0000 7fc0eaa00640 1 -- 172.21.15.96:0/972698457 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 10 and data pool 11 v39) ==== 174+0+0 (secure 0 0 0) 0x7fc0f415df90 con 0x7fc0f40d0640 2024-05-28T06:14:52.889 INFO:teuthology.orchestra.run.smithi096.stderr:new fs with metadata pool 10 and data pool 11 2024-05-28T06:14:52.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:52.890+0000 7fc0f9a00640 1 -- 172.21.15.96:0/972698457 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc0d006d610 msgr2=0x7fc0d006fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:52.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:52.890+0000 7fc0f9a00640 1 --2- 172.21.15.96:0/972698457 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc0d006d610 0x7fc0d006fad0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fc0e400a810 tx=0x7fc0e4008040 comp rx=0 tx=0).stop 2024-05-28T06:14:52.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:52.890+0000 7fc0f9a00640 1 -- 172.21.15.96:0/972698457 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc0f40d0640 msgr2=0x7fc0f40b15d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:52.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:52.890+0000 7fc0f9a00640 1 --2- 172.21.15.96:0/972698457 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc0f40d0640 0x7fc0f40b15d0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7fc0dc034040 tx=0x7fc0dc004a40 comp rx=0 tx=0).stop 2024-05-28T06:14:52.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:52.890+0000 7fc0f9a00640 1 -- 172.21.15.96:0/972698457 shutdown_connections 2024-05-28T06:14:52.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:52.890+0000 7fc0f9a00640 1 --2- 172.21.15.96:0/972698457 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc0d006d610 0x7fc0d006fad0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:52.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:52.890+0000 7fc0f9a00640 1 --2- 172.21.15.96:0/972698457 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc0f40d0a20 0x7fc0f414f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:52.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:52.890+0000 7fc0f9a00640 1 --2- 172.21.15.96:0/972698457 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc0f40d0640 0x7fc0f40b15d0 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:52.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:52.890+0000 7fc0f9a00640 1 --2- 172.21.15.96:0/972698457 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc0f40b32c0 0x7fc0f41f4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:52.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:52.890+0000 7fc0f9a00640 1 -- 172.21.15.96:0/972698457 >> 172.21.15.96:0/972698457 conn(0x7fc0f406bd20 msgr2=0x7fc0f415eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:52.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:52.890+0000 7fc0f9a00640 1 -- 172.21.15.96:0/972698457 shutdown_connections 2024-05-28T06:14:52.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:52.890+0000 7fc0f9a00640 1 -- 172.21.15.96:0/972698457 wait complete. 2024-05-28T06:14:52.928 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:14:53.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.096+0000 7fbe7d400640 1 Processor -- start 2024-05-28T06:14:53.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.096+0000 7fbe7d400640 1 -- start start 2024-05-28T06:14:53.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.098+0000 7fbe7d400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fbe7815b3a0 0x7fbe780b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:53.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.098+0000 7fbe7d400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbe78073310 0x7fbe7815f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:53.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.098+0000 7fbe7d400640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe780afd60 con 0x7fbe7806dc20 2024-05-28T06:14:53.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.098+0000 7fbe7d400640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fbe7806f980 con 0x7fbe7815b3a0 2024-05-28T06:14:53.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.098+0000 7fbe7d400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fbe780ae0c0 con 0x7fbe78073310 2024-05-28T06:14:53.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.098+0000 7fbe77e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fbe7815b3a0 0x7fbe780b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:53.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.098+0000 7fbe77e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fbe7815b3a0 0x7fbe780b32c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54530/0 (socket says 172.21.15.96:54530) 2024-05-28T06:14:53.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.098+0000 7fbe77e00640 1 -- 172.21.15.96:0/666256784 learned_addr learned my addr 172.21.15.96:0/666256784 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:53.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.098+0000 7fbe77e00640 1 -- 172.21.15.96:0/666256784 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbe78073310 msgr2=0x7fbe7815f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:53.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.098+0000 7fbe77e00640 1 --2- 172.21.15.96:0/666256784 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbe78073310 0x7fbe7815f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:53.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.098+0000 7fbe77e00640 1 -- 172.21.15.96:0/666256784 >> v1:172.21.15.123:6789/0 conn(0x7fbe7806dc20 legacy=0x7fbe78072c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:53.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.098+0000 7fbe77e00640 1 -- 172.21.15.96:0/666256784 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe780af7a0 con 0x7fbe7815b3a0 2024-05-28T06:14:53.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.099+0000 7fbe77e00640 1 --2- 172.21.15.96:0/666256784 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe7815b3a0 0x7fbe780b32c0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7fbe6800bb10 tx=0x7fbe6802c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f9ef3697a81bb15 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:53.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.100+0000 7fbe77400640 1 -- 172.21.15.96:0/666256784 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe68032780 con 0x7fbe7815b3a0 2024-05-28T06:14:53.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.100+0000 7fbe77400640 1 -- 172.21.15.96:0/666256784 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe68038070 con 0x7fbe7815b3a0 2024-05-28T06:14:53.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.100+0000 7fbe77400640 1 -- 172.21.15.96:0/666256784 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe68032b40 con 0x7fbe7815b3a0 2024-05-28T06:14:53.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.100+0000 7fbe7d400640 1 -- 172.21.15.96:0/666256784 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe7815b3a0 msgr2=0x7fbe780b32c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:53.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.100+0000 7fbe7d400640 1 --2- 172.21.15.96:0/666256784 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe7815b3a0 0x7fbe780b32c0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7fbe6800bb10 tx=0x7fbe6802c8c0 comp rx=0 tx=0).stop 2024-05-28T06:14:53.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.101+0000 7fbe7d400640 1 -- 172.21.15.96:0/666256784 shutdown_connections 2024-05-28T06:14:53.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.101+0000 7fbe7d400640 1 --2- 172.21.15.96:0/666256784 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbe78073310 0x7fbe7815f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:53.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.101+0000 7fbe7d400640 1 --2- 172.21.15.96:0/666256784 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe7815b3a0 0x7fbe780b32c0 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:53.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.101+0000 7fbe7d400640 1 -- 172.21.15.96:0/666256784 >> 172.21.15.96:0/666256784 conn(0x7fbe780b1fc0 msgr2=0x7fbe780b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:53.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.101+0000 7fbe7d400640 1 -- 172.21.15.96:0/666256784 shutdown_connections 2024-05-28T06:14:53.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.101+0000 7fbe7d400640 1 -- 172.21.15.96:0/666256784 wait complete. 2024-05-28T06:14:53.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.102+0000 7fbe7d400640 1 Processor -- start 2024-05-28T06:14:53.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.102+0000 7fbe7d400640 1 -- start start 2024-05-28T06:14:53.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.103+0000 7fbe7d400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe7806dc20 0x7fbe781f40b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:53.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.103+0000 7fbe7d400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbe78073310 0x7fbe781591a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:53.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.103+0000 7fbe7d400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbe7815b3a0 0x7fbe781596e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:53.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.103+0000 7fbe7d400640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fbe7815b780 con 0x7fbe7806dc20 2024-05-28T06:14:53.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.103+0000 7fbe7d400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fbe7815fee0 con 0x7fbe7815b3a0 2024-05-28T06:14:53.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.103+0000 7fbe7d400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fbe780ae550 con 0x7fbe78073310 2024-05-28T06:14:53.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.104+0000 7fbe77e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe7806dc20 0x7fbe781f40b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:53.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.104+0000 7fbe77e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe7806dc20 0x7fbe781f40b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54532/0 (socket says 172.21.15.96:54532) 2024-05-28T06:14:53.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.104+0000 7fbe77e00640 1 -- 172.21.15.96:0/918556110 learned_addr learned my addr 172.21.15.96:0/918556110 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:53.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.104+0000 7fbe77e00640 1 -- 172.21.15.96:0/918556110 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbe78073310 msgr2=0x7fbe781591a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:53.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.104+0000 7fbe77e00640 1 --2- 172.21.15.96:0/918556110 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbe78073310 0x7fbe781591a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:53.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.104+0000 7fbe77e00640 1 -- 172.21.15.96:0/918556110 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbe7815b3a0 msgr2=0x7fbe781596e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:53.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.104+0000 7fbe77e00640 1 --2- 172.21.15.96:0/918556110 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbe7815b3a0 0x7fbe781596e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:53.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.104+0000 7fbe77e00640 1 -- 172.21.15.96:0/918556110 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe780ce360 con 0x7fbe7806dc20 2024-05-28T06:14:53.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.104+0000 7fbe77e00640 1 --2- 172.21.15.96:0/918556110 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe7806dc20 0x7fbe781f40b0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7fbe6800bb10 tx=0x7fbe68004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:53.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.105+0000 7fbe74c00640 1 -- 172.21.15.96:0/918556110 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe68043070 con 0x7fbe7806dc20 2024-05-28T06:14:53.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.105+0000 7fbe74c00640 1 -- 172.21.15.96:0/918556110 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe68033a40 con 0x7fbe7806dc20 2024-05-28T06:14:53.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.105+0000 7fbe7d400640 1 -- 172.21.15.96:0/918556110 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe78159d10 con 0x7fbe7806dc20 2024-05-28T06:14:53.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.105+0000 7fbe74c00640 1 -- 172.21.15.96:0/918556110 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe68033e30 con 0x7fbe7806dc20 2024-05-28T06:14:53.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.105+0000 7fbe7d400640 1 -- 172.21.15.96:0/918556110 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe7815a2d0 con 0x7fbe7806dc20 2024-05-28T06:14:53.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.106+0000 7fbe7d400640 1 -- 172.21.15.96:0/918556110 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe780689e0 con 0x7fbe7806dc20 2024-05-28T06:14:53.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.107+0000 7fbe74c00640 1 -- 172.21.15.96:0/918556110 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 22) ==== 92213+0+0 (secure 0 0 0) 0x7fbe68046070 con 0x7fbe7806dc20 2024-05-28T06:14:53.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.108+0000 7fbe74c00640 1 --2- 172.21.15.96:0/918556110 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbe5406d5e0 0x7fbe5406faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:53.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.108+0000 7fbe74c00640 1 -- 172.21.15.96:0/918556110 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 5737+0+0 (secure 0 0 0) 0x7fbe680ab090 con 0x7fbe7806dc20 2024-05-28T06:14:53.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.108+0000 7fbe6f400640 1 --2- 172.21.15.96:0/918556110 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbe5406d5e0 0x7fbe5406faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:53.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.109+0000 7fbe6f400640 1 --2- 172.21.15.96:0/918556110 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbe5406d5e0 0x7fbe5406faa0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fbe781f7dd0 tx=0x7fbe60007990 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:53.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.112+0000 7fbe74c00640 1 -- 172.21.15.96:0/918556110 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbe680b1030 con 0x7fbe7806dc20 2024-05-28T06:14:53.301 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.299+0000 7fbe7d400640 1 -- 172.21.15.96:0/918556110 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbe7806ef60 con 0x7fbe7806dc20 2024-05-28T06:14:53.304 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.302+0000 7fbe74c00640 1 -- 172.21.15.96:0/918556110 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v73) ==== 74+0+16736 (secure 0 0 0) 0x7fbe68030050 con 0x7fbe7806dc20 2024-05-28T06:14:53.304 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:14:53.305 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":73,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:14:52.322631+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-05-28T06:14:50.098829+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-05-28T06:14:51.318311+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:14:53.307 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.305+0000 7fbe7d400640 1 -- 172.21.15.96:0/918556110 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbe5406d5e0 msgr2=0x7fbe5406faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:53.307 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.305+0000 7fbe7d400640 1 --2- 172.21.15.96:0/918556110 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbe5406d5e0 0x7fbe5406faa0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fbe781f7dd0 tx=0x7fbe60007990 comp rx=0 tx=0).stop 2024-05-28T06:14:53.307 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.305+0000 7fbe7d400640 1 -- 172.21.15.96:0/918556110 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe7806dc20 msgr2=0x7fbe781f40b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:53.307 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.305+0000 7fbe7d400640 1 --2- 172.21.15.96:0/918556110 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe7806dc20 0x7fbe781f40b0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7fbe6800bb10 tx=0x7fbe68004730 comp rx=0 tx=0).stop 2024-05-28T06:14:53.307 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.305+0000 7fbe7d400640 1 -- 172.21.15.96:0/918556110 shutdown_connections 2024-05-28T06:14:53.308 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.305+0000 7fbe7d400640 1 --2- 172.21.15.96:0/918556110 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbe5406d5e0 0x7fbe5406faa0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:53.308 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.305+0000 7fbe7d400640 1 --2- 172.21.15.96:0/918556110 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbe7815b3a0 0x7fbe781596e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:53.308 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.305+0000 7fbe7d400640 1 --2- 172.21.15.96:0/918556110 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbe78073310 0x7fbe781591a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:53.308 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.305+0000 7fbe7d400640 1 --2- 172.21.15.96:0/918556110 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe7806dc20 0x7fbe781f40b0 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:53.308 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.305+0000 7fbe7d400640 1 -- 172.21.15.96:0/918556110 >> 172.21.15.96:0/918556110 conn(0x7fbe780b1fc0 msgr2=0x7fbe780b3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:53.308 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.305+0000 7fbe7d400640 1 -- 172.21.15.96:0/918556110 shutdown_connections 2024-05-28T06:14:53.308 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.305+0000 7fbe7d400640 1 -- 172.21.15.96:0/918556110 wait complete. 2024-05-28T06:14:53.324 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:14:53.517 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.516+0000 7efd0f200640 1 Processor -- start 2024-05-28T06:14:53.517 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.516+0000 7efd0f200640 1 -- start start 2024-05-28T06:14:53.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.517+0000 7efd0f200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7efd1015e870 0x7efd100d0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:53.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.517+0000 7efd0f200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efd100cfa40 0x7efd100d25e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:53.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.517+0000 7efd0f200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd1015ec50 con 0x7efd100d14d0 2024-05-28T06:14:53.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.517+0000 7efd0f200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7efd100d2150 con 0x7efd1015e870 2024-05-28T06:14:53.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.517+0000 7efd0f200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7efd100cff80 con 0x7efd100cfa40 2024-05-28T06:14:53.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.518+0000 7efd0de00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7efd1015e870 0x7efd100d0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:53.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.518+0000 7efd0d400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7efd100d14d0 0x7efd100cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:52550/0 (socket says 172.21.15.96:52550) 2024-05-28T06:14:53.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.518+0000 7efd0d400640 1 -- 172.21.15.96:0/3190946823 learned_addr learned my addr 172.21.15.96:0/3190946823 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:14:53.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.518+0000 7efd0e800640 1 --2- 172.21.15.96:0/3190946823 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efd100cfa40 0x7efd100d25e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:53.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.518+0000 7efd0de00640 1 -- 172.21.15.96:0/3190946823 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efd100cfa40 msgr2=0x7efd100d25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:53.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.518+0000 7efd0de00640 1 --2- 172.21.15.96:0/3190946823 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efd100cfa40 0x7efd100d25e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:53.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.518+0000 7efd0de00640 1 -- 172.21.15.96:0/3190946823 >> v1:172.21.15.96:6789/0 conn(0x7efd100d14d0 legacy=0x7efd100cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:53.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.518+0000 7efd0ca00640 1 -- 172.21.15.96:0/3190946823 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3792349823 0 0) 0x7efd1015ec50 con 0x7efd100d14d0 2024-05-28T06:14:53.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.518+0000 7efd0de00640 1 -- 172.21.15.96:0/3190946823 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd100d0260 con 0x7efd1015e870 2024-05-28T06:14:53.524 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.520+0000 7efd0de00640 1 --2- 172.21.15.96:0/3190946823 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efd1015e870 0x7efd100d0f90 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7efcf80097a0 tx=0x7efcf802a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e60cf54b2168983 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:53.524 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.520+0000 7efd0ca00640 1 -- 172.21.15.96:0/3190946823 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcf8037070 con 0x7efd1015e870 2024-05-28T06:14:53.524 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.520+0000 7efd0ca00640 1 -- 172.21.15.96:0/3190946823 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efcf8032440 con 0x7efd1015e870 2024-05-28T06:14:53.525 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.520+0000 7efd0ca00640 1 -- 172.21.15.96:0/3190946823 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcf8032810 con 0x7efd1015e870 2024-05-28T06:14:53.525 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.521+0000 7efd0f200640 1 -- 172.21.15.96:0/3190946823 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efd1015e870 msgr2=0x7efd100d0f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:53.525 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.521+0000 7efd0f200640 1 --2- 172.21.15.96:0/3190946823 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efd1015e870 0x7efd100d0f90 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7efcf80097a0 tx=0x7efcf802a180 comp rx=0 tx=0).stop 2024-05-28T06:14:53.525 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.521+0000 7efd0f200640 1 -- 172.21.15.96:0/3190946823 shutdown_connections 2024-05-28T06:14:53.525 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.521+0000 7efd0f200640 1 --2- 172.21.15.96:0/3190946823 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efd100cfa40 0x7efd100d25e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:53.525 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.521+0000 7efd0f200640 1 --2- 172.21.15.96:0/3190946823 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efd1015e870 0x7efd100d0f90 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:53.525 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.521+0000 7efd0f200640 1 -- 172.21.15.96:0/3190946823 >> 172.21.15.96:0/3190946823 conn(0x7efd1008bfc0 msgr2=0x7efd100d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:53.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.521+0000 7efd0f200640 1 -- 172.21.15.96:0/3190946823 shutdown_connections 2024-05-28T06:14:53.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.521+0000 7efd0f200640 1 -- 172.21.15.96:0/3190946823 wait complete. 2024-05-28T06:14:53.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.522+0000 7efd0f200640 1 Processor -- start 2024-05-28T06:14:53.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.523+0000 7efd0f200640 1 -- start start 2024-05-28T06:14:53.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.523+0000 7efd0f200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efd100cfa40 0x7efd100b6830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:53.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.523+0000 7efd0f200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efd100d14d0 0x7efd10155de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:53.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.524+0000 7efd0f200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd1015e870 0x7efd100cd820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:53.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.524+0000 7efd0f200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7efd100d4fb0 con 0x7efd100cfa40 2024-05-28T06:14:53.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.524+0000 7efd0f200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7efd100d0820 con 0x7efd1015e870 2024-05-28T06:14:53.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.524+0000 7efd0f200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7efd100d2b20 con 0x7efd100d14d0 2024-05-28T06:14:53.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.524+0000 7efd0de00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efd100cfa40 0x7efd100b6830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:53.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.524+0000 7efd0de00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efd100cfa40 0x7efd100b6830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54536/0 (socket says 172.21.15.96:54536) 2024-05-28T06:14:53.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.524+0000 7efd0de00640 1 -- 172.21.15.96:0/1924478070 learned_addr learned my addr 172.21.15.96:0/1924478070 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:53.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.524+0000 7efd0d400640 1 --2- 172.21.15.96:0/1924478070 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efd100d14d0 0x7efd10155de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:53.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.524+0000 7efd0de00640 1 -- 172.21.15.96:0/1924478070 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efd100d14d0 msgr2=0x7efd10155de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:53.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.524+0000 7efd0de00640 1 --2- 172.21.15.96:0/1924478070 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efd100d14d0 0x7efd10155de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:53.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.524+0000 7efd0de00640 1 -- 172.21.15.96:0/1924478070 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd1015e870 msgr2=0x7efd100cd820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:53.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.524+0000 7efd0e800640 1 --2- 172.21.15.96:0/1924478070 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd1015e870 0x7efd100cd820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:53.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.524+0000 7efd0de00640 1 --2- 172.21.15.96:0/1924478070 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd1015e870 0x7efd100cd820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:53.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.524+0000 7efd0de00640 1 -- 172.21.15.96:0/1924478070 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd100b73f0 con 0x7efd100cfa40 2024-05-28T06:14:53.533 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.524+0000 7efd0d400640 1 --2- 172.21.15.96:0/1924478070 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efd100d14d0 0x7efd10155de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:14:53.533 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.525+0000 7efd0de00640 1 --2- 172.21.15.96:0/1924478070 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efd100cfa40 0x7efd100b6830 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7efcf80097a0 tx=0x7efcf80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:53.533 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.525+0000 7efd06000640 1 -- 172.21.15.96:0/1924478070 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcf8037050 con 0x7efd100cfa40 2024-05-28T06:14:53.533 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.525+0000 7efd06000640 1 -- 172.21.15.96:0/1924478070 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efcf8038040 con 0x7efd100cfa40 2024-05-28T06:14:53.533 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.525+0000 7efd06000640 1 -- 172.21.15.96:0/1924478070 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcf8006e10 con 0x7efd100cfa40 2024-05-28T06:14:53.533 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.525+0000 7efd0f200640 1 -- 172.21.15.96:0/1924478070 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd100b76d0 con 0x7efd100cfa40 2024-05-28T06:14:53.533 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.525+0000 7efd0f200640 1 -- 172.21.15.96:0/1924478070 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd100b7c90 con 0x7efd100cfa40 2024-05-28T06:14:53.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.526+0000 7efd0f200640 1 -- 172.21.15.96:0/1924478070 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd100689e0 con 0x7efd100cfa40 2024-05-28T06:14:53.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.527+0000 7efd06000640 1 -- 172.21.15.96:0/1924478070 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 22) ==== 92213+0+0 (secure 0 0 0) 0x7efcf8030980 con 0x7efd100cfa40 2024-05-28T06:14:53.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.529+0000 7efd06000640 1 --2- 172.21.15.96:0/1924478070 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7efcec06d5e0 0x7efcec06faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:53.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.529+0000 7efd06000640 1 -- 172.21.15.96:0/1924478070 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 5737+0+0 (secure 0 0 0) 0x7efcf80ab0d0 con 0x7efd100cfa40 2024-05-28T06:14:53.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.529+0000 7efd0d400640 1 --2- 172.21.15.96:0/1924478070 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7efcec06d5e0 0x7efcec06faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:53.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.530+0000 7efd0d400640 1 --2- 172.21.15.96:0/1924478070 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7efcec06d5e0 0x7efcec06faa0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7efd00007fd0 tx=0x7efd00009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:53.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.533+0000 7efd06000640 1 -- 172.21.15.96:0/1924478070 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7efcf807af50 con 0x7efd100cfa40 2024-05-28T06:14:53.733 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.732+0000 7efd0f200640 1 -- 172.21.15.96:0/1924478070 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7efd100ba500 con 0x7efd100cfa40 2024-05-28T06:14:53.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.733+0000 7efd06000640 1 -- 172.21.15.96:0/1924478070 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v73) ==== 74+0+16736 (secure 0 0 0) 0x7efcf807a850 con 0x7efd100cfa40 2024-05-28T06:14:53.735 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:14:53.736 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":73,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:14:52.322631+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-05-28T06:14:50.098829+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-05-28T06:14:51.318311+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:14:53.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.736+0000 7efd0f200640 1 -- 172.21.15.96:0/1924478070 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7efcec06d5e0 msgr2=0x7efcec06faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:53.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.736+0000 7efd0f200640 1 --2- 172.21.15.96:0/1924478070 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7efcec06d5e0 0x7efcec06faa0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7efd00007fd0 tx=0x7efd00009040 comp rx=0 tx=0).stop 2024-05-28T06:14:53.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.736+0000 7efd0f200640 1 -- 172.21.15.96:0/1924478070 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efd100cfa40 msgr2=0x7efd100b6830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:53.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.736+0000 7efd0f200640 1 --2- 172.21.15.96:0/1924478070 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efd100cfa40 0x7efd100b6830 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7efcf80097a0 tx=0x7efcf80046c0 comp rx=0 tx=0).stop 2024-05-28T06:14:53.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.736+0000 7efd0f200640 1 -- 172.21.15.96:0/1924478070 shutdown_connections 2024-05-28T06:14:53.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.736+0000 7efd0f200640 1 --2- 172.21.15.96:0/1924478070 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7efcec06d5e0 0x7efcec06faa0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:53.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.736+0000 7efd0f200640 1 --2- 172.21.15.96:0/1924478070 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd1015e870 0x7efd100cd820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:53.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.736+0000 7efd0f200640 1 --2- 172.21.15.96:0/1924478070 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efd100d14d0 0x7efd10155de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:53.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.736+0000 7efd0f200640 1 --2- 172.21.15.96:0/1924478070 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efd100cfa40 0x7efd100b6830 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:53.739 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.736+0000 7efd0f200640 1 -- 172.21.15.96:0/1924478070 >> 172.21.15.96:0/1924478070 conn(0x7efd1008bfc0 msgr2=0x7efd100b10f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:53.739 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.737+0000 7efd0f200640 1 -- 172.21.15.96:0/1924478070 shutdown_connections 2024-05-28T06:14:53.739 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.737+0000 7efd0f200640 1 -- 172.21.15.96:0/1924478070 wait complete. 2024-05-28T06:14:53.757 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-05-28T06:14:53.956 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.954+0000 7fc714800640 1 Processor -- start 2024-05-28T06:14:53.956 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.954+0000 7fc714800640 1 -- start start 2024-05-28T06:14:53.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.956+0000 7fc714800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fc70c15d5d0 0x7fc70c0cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:53.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.956+0000 7fc714800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fc70c0b32c0 0x7fc70c0b3700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:53.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.956+0000 7fc714800640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc70c15cb10 con 0x7fc70c0d0640 2024-05-28T06:14:53.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.956+0000 7fc714800640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fc70c0adbb0 con 0x7fc70c15d5d0 2024-05-28T06:14:53.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.956+0000 7fc714800640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fc70c0ad5f0 con 0x7fc70c0b32c0 2024-05-28T06:14:53.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.956+0000 7fc711e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fc70c0d0640 0x7fc70c0d0a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:52552/0 (socket says 172.21.15.96:52552) 2024-05-28T06:14:53.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.956+0000 7fc711400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fc70c15d5d0 0x7fc70c0cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:53.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.956+0000 7fc711e00640 1 -- 172.21.15.96:0/3568208262 learned_addr learned my addr 172.21.15.96:0/3568208262 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:14:53.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.956+0000 7fc712800640 1 --2- 172.21.15.96:0/3568208262 >> v2:172.21.15.123:3300/0 conn(0x7fc70c0b32c0 0x7fc70c0b3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:53.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.957+0000 7fc711400640 1 -- 172.21.15.96:0/3568208262 >> v2:172.21.15.123:3300/0 conn(0x7fc70c0b32c0 msgr2=0x7fc70c0b3700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:53.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.957+0000 7fc711400640 1 --2- 172.21.15.96:0/3568208262 >> v2:172.21.15.123:3300/0 conn(0x7fc70c0b32c0 0x7fc70c0b3700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:53.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.957+0000 7fc711400640 1 -- 172.21.15.96:0/3568208262 >> v1:172.21.15.96:6789/0 conn(0x7fc70c0d0640 legacy=0x7fc70c0d0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:53.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.957+0000 7fc711400640 1 -- 172.21.15.96:0/3568208262 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc70c0ad8d0 con 0x7fc70c15d5d0 2024-05-28T06:14:53.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.958+0000 7fc711400640 1 --2- 172.21.15.96:0/3568208262 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc70c15d5d0 0x7fc70c0cf950 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7fc70c0d1200 tx=0x7fc6fc02a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5164d801a1b08f05 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:53.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.958+0000 7fc710a00640 1 -- 172.21.15.96:0/3568208262 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6fc037070 con 0x7fc70c15d5d0 2024-05-28T06:14:53.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.958+0000 7fc710a00640 1 -- 172.21.15.96:0/3568208262 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6fc032440 con 0x7fc70c15d5d0 2024-05-28T06:14:53.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.958+0000 7fc710a00640 1 -- 172.21.15.96:0/3568208262 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6fc02ae60 con 0x7fc70c15d5d0 2024-05-28T06:14:53.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.959+0000 7fc714800640 1 -- 172.21.15.96:0/3568208262 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc70c15d5d0 msgr2=0x7fc70c0cf950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:53.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.959+0000 7fc714800640 1 --2- 172.21.15.96:0/3568208262 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc70c15d5d0 0x7fc70c0cf950 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7fc70c0d1200 tx=0x7fc6fc02a110 comp rx=0 tx=0).stop 2024-05-28T06:14:53.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.959+0000 7fc714800640 1 -- 172.21.15.96:0/3568208262 shutdown_connections 2024-05-28T06:14:53.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.959+0000 7fc714800640 1 --2- 172.21.15.96:0/3568208262 >> v2:172.21.15.123:3300/0 conn(0x7fc70c0b32c0 0x7fc70c0b3700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:53.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.959+0000 7fc714800640 1 --2- 172.21.15.96:0/3568208262 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc70c15d5d0 0x7fc70c0cf950 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:53.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.959+0000 7fc714800640 1 -- 172.21.15.96:0/3568208262 >> 172.21.15.96:0/3568208262 conn(0x7fc70c06bd20 msgr2=0x7fc70c06c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:53.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.959+0000 7fc714800640 1 -- 172.21.15.96:0/3568208262 shutdown_connections 2024-05-28T06:14:53.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.959+0000 7fc714800640 1 -- 172.21.15.96:0/3568208262 wait complete. 2024-05-28T06:14:53.962 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.960+0000 7fc714800640 1 Processor -- start 2024-05-28T06:14:53.962 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.961+0000 7fc714800640 1 -- start start 2024-05-28T06:14:53.964 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.962+0000 7fc714800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc70c0b32c0 0x7fc70c1f4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:53.964 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.962+0000 7fc714800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc70c0d0640 0x7fc70c09dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:53.964 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.962+0000 7fc714800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc70c15d5d0 0x7fc70c073160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:53.964 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.962+0000 7fc714800640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc70c15da50 con 0x7fc70c0b32c0 2024-05-28T06:14:53.964 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.962+0000 7fc714800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc70c15d0b0 con 0x7fc70c0d0640 2024-05-28T06:14:53.964 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.962+0000 7fc714800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fc70c0ad2a0 con 0x7fc70c15d5d0 2024-05-28T06:14:53.964 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.962+0000 7fc711e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc70c0b32c0 0x7fc70c1f4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:53.964 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.962+0000 7fc711e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc70c0b32c0 0x7fc70c1f4440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54554/0 (socket says 172.21.15.96:54554) 2024-05-28T06:14:53.964 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.962+0000 7fc711e00640 1 -- 172.21.15.96:0/3055111090 learned_addr learned my addr 172.21.15.96:0/3055111090 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:53.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.962+0000 7fc711e00640 1 -- 172.21.15.96:0/3055111090 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc70c15d5d0 msgr2=0x7fc70c073160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:53.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.962+0000 7fc711e00640 1 --2- 172.21.15.96:0/3055111090 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc70c15d5d0 0x7fc70c073160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:53.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.962+0000 7fc711e00640 1 -- 172.21.15.96:0/3055111090 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc70c0d0640 msgr2=0x7fc70c09dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:53.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.962+0000 7fc711e00640 1 --2- 172.21.15.96:0/3055111090 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc70c0d0640 0x7fc70c09dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:53.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.962+0000 7fc711e00640 1 -- 172.21.15.96:0/3055111090 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc70c15b730 con 0x7fc70c0b32c0 2024-05-28T06:14:53.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.963+0000 7fc711e00640 1 --2- 172.21.15.96:0/3055111090 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc70c0b32c0 0x7fc70c1f4440 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7fc70000c3f0 tx=0x7fc70000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:53.966 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.963+0000 7fc706000640 1 -- 172.21.15.96:0/3055111090 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc700017070 con 0x7fc70c0b32c0 2024-05-28T06:14:53.966 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.963+0000 7fc706000640 1 -- 172.21.15.96:0/3055111090 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc700005ce0 con 0x7fc70c0b32c0 2024-05-28T06:14:53.966 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.963+0000 7fc714800640 1 -- 172.21.15.96:0/3055111090 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc70c15ba10 con 0x7fc70c0b32c0 2024-05-28T06:14:53.966 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.963+0000 7fc706000640 1 -- 172.21.15.96:0/3055111090 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7000072f0 con 0x7fc70c0b32c0 2024-05-28T06:14:53.966 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.963+0000 7fc714800640 1 -- 172.21.15.96:0/3055111090 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc70c09a5c0 con 0x7fc70c0b32c0 2024-05-28T06:14:53.966 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.964+0000 7fc714800640 1 -- 172.21.15.96:0/3055111090 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc70c156690 con 0x7fc70c0b32c0 2024-05-28T06:14:53.967 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.965+0000 7fc706000640 1 -- 172.21.15.96:0/3055111090 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 22) ==== 92213+0+0 (secure 0 0 0) 0x7fc70001a070 con 0x7fc70c0b32c0 2024-05-28T06:14:53.967 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.966+0000 7fc706000640 1 --2- 172.21.15.96:0/3055111090 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc6f006d610 0x7fc6f006fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:53.967 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.966+0000 7fc706000640 1 -- 172.21.15.96:0/3055111090 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 5737+0+0 (secure 0 0 0) 0x7fc700089930 con 0x7fc70c0b32c0 2024-05-28T06:14:53.968 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.967+0000 7fc711400640 1 --2- 172.21.15.96:0/3055111090 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc6f006d610 0x7fc6f006fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:53.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.968+0000 7fc711400640 1 --2- 172.21.15.96:0/3055111090 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc6f006d610 0x7fc6f006fad0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fc6fc034000 tx=0x7fc6fc02c000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:53.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:53.970+0000 7fc706000640 1 -- 172.21.15.96:0/3055111090 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc70c156690 con 0x7fc70c0b32c0 2024-05-28T06:14:54.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:54.205+0000 7fc714800640 1 -- 172.21.15.96:0/3055111090 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fc70c09a3b0 con 0x7fc70c0b32c0 2024-05-28T06:14:54.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:54.930+0000 7fc706000640 1 -- 172.21.15.96:0/3055111090 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v42) ==== 118+0+0 (secure 0 0 0) 0x7fc7000597b0 con 0x7fc70c0b32c0 2024-05-28T06:14:54.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:54.933+0000 7fc714800640 1 -- 172.21.15.96:0/3055111090 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc6f006d610 msgr2=0x7fc6f006fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:54.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:54.933+0000 7fc714800640 1 --2- 172.21.15.96:0/3055111090 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc6f006d610 0x7fc6f006fad0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fc6fc034000 tx=0x7fc6fc02c000 comp rx=0 tx=0).stop 2024-05-28T06:14:54.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:54.933+0000 7fc714800640 1 -- 172.21.15.96:0/3055111090 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc70c0b32c0 msgr2=0x7fc70c1f4440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:54.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:54.933+0000 7fc714800640 1 --2- 172.21.15.96:0/3055111090 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc70c0b32c0 0x7fc70c1f4440 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7fc70000c3f0 tx=0x7fc70000c8b0 comp rx=0 tx=0).stop 2024-05-28T06:14:54.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:54.934+0000 7fc714800640 1 -- 172.21.15.96:0/3055111090 shutdown_connections 2024-05-28T06:14:54.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:54.934+0000 7fc714800640 1 --2- 172.21.15.96:0/3055111090 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc6f006d610 0x7fc6f006fad0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:54.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:54.934+0000 7fc714800640 1 --2- 172.21.15.96:0/3055111090 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc70c15d5d0 0x7fc70c073160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:54.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:54.934+0000 7fc714800640 1 --2- 172.21.15.96:0/3055111090 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc70c0d0640 0x7fc70c09dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:54.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:54.934+0000 7fc714800640 1 --2- 172.21.15.96:0/3055111090 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc70c0b32c0 0x7fc70c1f4440 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:54.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:54.934+0000 7fc714800640 1 -- 172.21.15.96:0/3055111090 >> 172.21.15.96:0/3055111090 conn(0x7fc70c06bd20 msgr2=0x7fc70c15f020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:54.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:54.934+0000 7fc714800640 1 -- 172.21.15.96:0/3055111090 shutdown_connections 2024-05-28T06:14:54.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:54.934+0000 7fc714800640 1 -- 172.21.15.96:0/3055111090 wait complete. 2024-05-28T06:14:54.956 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-05-28T06:14:54.956 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-05-28T06:14:55.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:55.151+0000 7fee35400640 1 Processor -- start 2024-05-28T06:14:55.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:55.151+0000 7fee35400640 1 -- start start 2024-05-28T06:14:55.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:55.153+0000 7fee35400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fee300cef30 0x7fee300d1640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:55.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:55.153+0000 7fee35400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fee300d1b80 0x7fee300b90a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:55.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:55.153+0000 7fee35400640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee300d1f90 con 0x7fee3015e850 2024-05-28T06:14:55.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:55.153+0000 7fee35400640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fee3015ec30 con 0x7fee300cef30 2024-05-28T06:14:55.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:55.153+0000 7fee35400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fee300b8000 con 0x7fee300d1b80 2024-05-28T06:14:55.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:55.153+0000 7fee2e800640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fee3015e850 0x7fee300b77b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:52558/0 (socket says 172.21.15.96:52558) 2024-05-28T06:14:55.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:55.153+0000 7fee2e800640 1 -- 172.21.15.96:0/876953681 learned_addr learned my addr 172.21.15.96:0/876953681 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:14:55.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:55.153+0000 7fee2de00640 1 --2- 172.21.15.96:0/876953681 >> v2:172.21.15.96:3300/0 conn(0x7fee300cef30 0x7fee300d1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:55.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:55.154+0000 7fee2f200640 1 --2- 172.21.15.96:0/876953681 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fee300d1b80 0x7fee300b90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:55.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:55.154+0000 7fee2d400640 1 -- 172.21.15.96:0/876953681 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2418115557 0 0) 0x7fee20001490 con 0x7fee3015e850 2024-05-28T06:14:55.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:55.154+0000 7fee2d400640 1 -- 172.21.15.96:0/876953681 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fee14003520 con 0x7fee3015e850 2024-05-28T06:14:55.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:55.154+0000 7fee2de00640 1 -- 172.21.15.96:0/876953681 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fee300d1b80 msgr2=0x7fee300b90a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:55.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:55.154+0000 7fee2de00640 1 --2- 172.21.15.96:0/876953681 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fee300d1b80 0x7fee300b90a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:55.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:55.154+0000 7fee2de00640 1 -- 172.21.15.96:0/876953681 >> v1:172.21.15.96:6789/0 conn(0x7fee3015e850 legacy=0x7fee300b77b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:55.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:55.154+0000 7fee2de00640 1 -- 172.21.15.96:0/876953681 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee300b82e0 con 0x7fee300cef30 2024-05-28T06:14:55.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:55.155+0000 7fee2de00640 1 --2- 172.21.15.96:0/876953681 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fee300cef30 0x7fee300d1640 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7fee180097a0 tx=0x7fee1802a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe7e61933c7a8652 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:55.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:55.155+0000 7fee2d400640 1 -- 172.21.15.96:0/876953681 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee18037070 con 0x7fee300cef30 2024-05-28T06:14:55.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:55.155+0000 7fee2d400640 1 -- 172.21.15.96:0/876953681 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee1802ae50 con 0x7fee300cef30 2024-05-28T06:14:55.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:55.155+0000 7fee2d400640 1 -- 172.21.15.96:0/876953681 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee18032640 con 0x7fee300cef30 2024-05-28T06:14:55.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:55.156+0000 7fee35400640 1 -- 172.21.15.96:0/876953681 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fee300cef30 msgr2=0x7fee300d1640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:55.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:55.156+0000 7fee35400640 1 --2- 172.21.15.96:0/876953681 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fee300cef30 0x7fee300d1640 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7fee180097a0 tx=0x7fee1802a100 comp rx=0 tx=0).stop 2024-05-28T06:14:55.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:55.156+0000 7fee35400640 1 -- 172.21.15.96:0/876953681 shutdown_connections 2024-05-28T06:14:55.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:55.156+0000 7fee35400640 1 --2- 172.21.15.96:0/876953681 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fee300d1b80 0x7fee300b90a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:55.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:55.156+0000 7fee35400640 1 --2- 172.21.15.96:0/876953681 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fee300cef30 0x7fee300d1640 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:55.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:55.156+0000 7fee35400640 1 -- 172.21.15.96:0/876953681 >> 172.21.15.96:0/876953681 conn(0x7fee30067aa0 msgr2=0x7fee30067eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:55.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:55.156+0000 7fee35400640 1 -- 172.21.15.96:0/876953681 shutdown_connections 2024-05-28T06:14:55.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:55.156+0000 7fee35400640 1 -- 172.21.15.96:0/876953681 wait complete. 2024-05-28T06:14:55.159 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:55.158+0000 7fee35400640 1 Processor -- start 2024-05-28T06:14:55.159 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:55.158+0000 7fee35400640 1 -- start start 2024-05-28T06:14:55.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:55.159+0000 7fee35400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fee300cef30 0x7fee300ce840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:55.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:55.159+0000 7fee35400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fee300d1b80 0x7fee3014aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:55.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:55.159+0000 7fee35400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee3015e850 0x7fee300ab0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:55.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:55.159+0000 7fee35400640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fee300b95e0 con 0x7fee300cef30 2024-05-28T06:14:55.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:55.159+0000 7fee35400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fee300b88a0 con 0x7fee3015e850 2024-05-28T06:14:55.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:55.159+0000 7fee35400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fee300b9760 con 0x7fee300d1b80 2024-05-28T06:14:55.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:55.159+0000 7fee2e800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fee300cef30 0x7fee300ce840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:55.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:55.159+0000 7fee2e800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fee300cef30 0x7fee300ce840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54582/0 (socket says 172.21.15.96:54582) 2024-05-28T06:14:55.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:55.159+0000 7fee2e800640 1 -- 172.21.15.96:0/3039353842 learned_addr learned my addr 172.21.15.96:0/3039353842 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:55.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:55.160+0000 7fee2e800640 1 -- 172.21.15.96:0/3039353842 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fee300d1b80 msgr2=0x7fee3014aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:55.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:55.160+0000 7fee2de00640 1 --2- 172.21.15.96:0/3039353842 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fee300d1b80 0x7fee3014aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:55.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:55.160+0000 7fee2f200640 1 --2- 172.21.15.96:0/3039353842 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee3015e850 0x7fee300ab0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:55.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:55.160+0000 7fee2e800640 1 --2- 172.21.15.96:0/3039353842 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fee300d1b80 0x7fee3014aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:55.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:55.160+0000 7fee2e800640 1 -- 172.21.15.96:0/3039353842 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee3015e850 msgr2=0x7fee300ab0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:55.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:55.160+0000 7fee2e800640 1 --2- 172.21.15.96:0/3039353842 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee3015e850 0x7fee300ab0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:55.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:55.160+0000 7fee2e800640 1 -- 172.21.15.96:0/3039353842 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee300ab600 con 0x7fee300cef30 2024-05-28T06:14:55.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:55.160+0000 7fee2e800640 1 --2- 172.21.15.96:0/3039353842 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fee300cef30 0x7fee300ce840 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7fee2000cae0 tx=0x7fee2000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:55.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:55.160+0000 7fee26a00640 1 -- 172.21.15.96:0/3039353842 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee20017070 con 0x7fee300cef30 2024-05-28T06:14:55.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:55.160+0000 7fee26a00640 1 -- 172.21.15.96:0/3039353842 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee2000abe0 con 0x7fee300cef30 2024-05-28T06:14:55.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:55.160+0000 7fee26a00640 1 -- 172.21.15.96:0/3039353842 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee20012400 con 0x7fee300cef30 2024-05-28T06:14:55.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:55.160+0000 7fee35400640 1 -- 172.21.15.96:0/3039353842 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee3014dde0 con 0x7fee300cef30 2024-05-28T06:14:55.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:55.160+0000 7fee35400640 1 -- 172.21.15.96:0/3039353842 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee3014b210 con 0x7fee300cef30 2024-05-28T06:14:55.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:55.161+0000 7fee35400640 1 -- 172.21.15.96:0/3039353842 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee300689e0 con 0x7fee300cef30 2024-05-28T06:14:55.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:55.162+0000 7fee26a00640 1 -- 172.21.15.96:0/3039353842 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 22) ==== 92213+0+0 (secure 0 0 0) 0x7fee2001a070 con 0x7fee300cef30 2024-05-28T06:14:55.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:55.163+0000 7fee26a00640 1 --2- 172.21.15.96:0/3039353842 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fee0c06d5e0 0x7fee0c06faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:55.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:55.163+0000 7fee26a00640 1 -- 172.21.15.96:0/3039353842 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 5737+0+0 (secure 0 0 0) 0x7fee2008a410 con 0x7fee300cef30 2024-05-28T06:14:55.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:55.164+0000 7fee2de00640 1 --2- 172.21.15.96:0/3039353842 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fee0c06d5e0 0x7fee0c06faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:55.166 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:55.165+0000 7fee2de00640 1 --2- 172.21.15.96:0/3039353842 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fee0c06d5e0 0x7fee0c06faa0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fee18034000 tx=0x7fee1802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:55.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:55.168+0000 7fee26a00640 1 -- 172.21.15.96:0/3039353842 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fee300d1f90 con 0x7fee300cef30 2024-05-28T06:14:55.415 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:55.413+0000 7fee35400640 1 -- 172.21.15.96:0/3039353842 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fee3015d6f0 con 0x7fee300cef30 2024-05-28T06:14:55.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:55.933+0000 7fee26a00640 1 -- 172.21.15.96:0/3039353842 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v43) ==== 122+0+0 (secure 0 0 0) 0x7fee2005a200 con 0x7fee300cef30 2024-05-28T06:14:55.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:55.939+0000 7fee35400640 1 -- 172.21.15.96:0/3039353842 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fee0c06d5e0 msgr2=0x7fee0c06faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:55.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:55.939+0000 7fee35400640 1 --2- 172.21.15.96:0/3039353842 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fee0c06d5e0 0x7fee0c06faa0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fee18034000 tx=0x7fee1802e000 comp rx=0 tx=0).stop 2024-05-28T06:14:55.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:55.939+0000 7fee35400640 1 -- 172.21.15.96:0/3039353842 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fee300cef30 msgr2=0x7fee300ce840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:55.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:55.939+0000 7fee35400640 1 --2- 172.21.15.96:0/3039353842 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fee300cef30 0x7fee300ce840 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7fee2000cae0 tx=0x7fee2000a060 comp rx=0 tx=0).stop 2024-05-28T06:14:55.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:55.939+0000 7fee35400640 1 -- 172.21.15.96:0/3039353842 shutdown_connections 2024-05-28T06:14:55.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:55.939+0000 7fee35400640 1 --2- 172.21.15.96:0/3039353842 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fee0c06d5e0 0x7fee0c06faa0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:55.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:55.939+0000 7fee35400640 1 --2- 172.21.15.96:0/3039353842 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee3015e850 0x7fee300ab0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:55.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:55.939+0000 7fee35400640 1 --2- 172.21.15.96:0/3039353842 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fee300d1b80 0x7fee3014aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:55.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:55.939+0000 7fee35400640 1 --2- 172.21.15.96:0/3039353842 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fee300cef30 0x7fee300ce840 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:55.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:55.939+0000 7fee35400640 1 -- 172.21.15.96:0/3039353842 >> 172.21.15.96:0/3039353842 conn(0x7fee30067aa0 msgr2=0x7fee3015dc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:55.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:55.940+0000 7fee35400640 1 -- 172.21.15.96:0/3039353842 shutdown_connections 2024-05-28T06:14:55.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:55.940+0000 7fee35400640 1 -- 172.21.15.96:0/3039353842 wait complete. 2024-05-28T06:14:55.963 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-05-28T06:14:55.964 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-28T06:14:56.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.155+0000 7f37a5000640 1 Processor -- start 2024-05-28T06:14:56.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.155+0000 7f37a5000640 1 -- start start 2024-05-28T06:14:56.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.156+0000 7f37a5000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f37a006ee40 0x7f37a00ad230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:56.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.156+0000 7f37a5000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f37a006f2f0 0x7f37a00ad770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:56.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.157+0000 7f37a5000640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37a0156f10 con 0x7f37a0156b30 2024-05-28T06:14:56.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.157+0000 7f37a5000640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f37a006dda0 con 0x7f37a006ee40 2024-05-28T06:14:56.159 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.157+0000 7f37a5000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f37a006e080 con 0x7f37a006f2f0 2024-05-28T06:14:56.159 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.157+0000 7f379f400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f37a0156b30 0x7f37a00af6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:52566/0 (socket says 172.21.15.96:52566) 2024-05-28T06:14:56.159 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.157+0000 7f379f400640 1 -- 172.21.15.96:0/2961042526 learned_addr learned my addr 172.21.15.96:0/2961042526 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:14:56.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.157+0000 7f379fe00640 1 --2- 172.21.15.96:0/2961042526 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f37a006f2f0 0x7f37a00ad770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:56.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.157+0000 7f379e000640 1 -- 172.21.15.96:0/2961042526 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3747696421 0 0) 0x7f37a0156f10 con 0x7f37a0156b30 2024-05-28T06:14:56.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.157+0000 7f379ea00640 1 --2- 172.21.15.96:0/2961042526 >> v2:172.21.15.123:3300/0 conn(0x7f37a006ee40 0x7f37a00ad230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:56.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.157+0000 7f379e000640 1 -- 172.21.15.96:0/2961042526 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3784003520 con 0x7f37a0156b30 2024-05-28T06:14:56.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.158+0000 7f379e000640 1 -- 172.21.15.96:0/2961042526 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2225698488 0 0) 0x7f3784003520 con 0x7f37a0156b30 2024-05-28T06:14:56.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.158+0000 7f379e000640 1 -- 172.21.15.96:0/2961042526 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f37a006f2f0 msgr2=0x7f37a00ad770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:56.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.158+0000 7f379e000640 1 --2- 172.21.15.96:0/2961042526 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f37a006f2f0 0x7f37a00ad770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:56.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.158+0000 7f379e000640 1 -- 172.21.15.96:0/2961042526 >> v2:172.21.15.123:3300/0 conn(0x7f37a006ee40 msgr2=0x7f37a00ad230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:56.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.158+0000 7f379e000640 1 --2- 172.21.15.96:0/2961042526 >> v2:172.21.15.123:3300/0 conn(0x7f37a006ee40 0x7f37a00ad230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:56.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.158+0000 7f379e000640 1 -- 172.21.15.96:0/2961042526 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37a006e360 con 0x7f37a0156b30 2024-05-28T06:14:56.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.158+0000 7f379e000640 1 -- 172.21.15.96:0/2961042526 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f3790001670 con 0x7f37a0156b30 2024-05-28T06:14:56.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.158+0000 7f379e000640 1 -- 172.21.15.96:0/2961042526 >> v1:172.21.15.96:6789/0 conn(0x7f37a0156b30 legacy=0x7f37a00af6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:56.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.158+0000 7f379e000640 1 --2- 172.21.15.96:0/2961042526 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f37a006f2f0 0x7f37840039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:56.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.158+0000 7f379e000640 1 --2- 172.21.15.96:0/2961042526 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3784003fe0 0x7f3784004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:56.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.158+0000 7f379e000640 1 --2- 172.21.15.96:0/2961042526 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3784004d40 0x7f37840051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:56.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.158+0000 7f379e000640 1 -- 172.21.15.96:0/2961042526 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f37a006dda0 con 0x7f3784004d40 2024-05-28T06:14:56.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.158+0000 7f379e000640 1 -- 172.21.15.96:0/2961042526 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f37a006e080 con 0x7f3784003fe0 2024-05-28T06:14:56.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.158+0000 7f379e000640 1 -- 172.21.15.96:0/2961042526 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f378400bb80 con 0x7f37a006f2f0 2024-05-28T06:14:56.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.158+0000 7f379fe00640 1 --2- 172.21.15.96:0/2961042526 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3784004d40 0x7f37840051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:56.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.158+0000 7f379f400640 1 --2- 172.21.15.96:0/2961042526 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f37a006f2f0 0x7f37840039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:56.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.158+0000 7f379fe00640 1 -- 172.21.15.96:0/2961042526 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f37a006f2f0 msgr2=0x7f37840039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:56.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.158+0000 7f379fe00640 1 --2- 172.21.15.96:0/2961042526 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f37a006f2f0 0x7f37840039b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:56.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.158+0000 7f379fe00640 1 -- 172.21.15.96:0/2961042526 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3784003fe0 msgr2=0x7f3784004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:56.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.158+0000 7f379fe00640 1 --2- 172.21.15.96:0/2961042526 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3784003fe0 0x7f3784004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:56.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.158+0000 7f379fe00640 1 -- 172.21.15.96:0/2961042526 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3784003520 con 0x7f3784004d40 2024-05-28T06:14:56.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.159+0000 7f379fe00640 1 --2- 172.21.15.96:0/2961042526 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3784004d40 0x7f37840051c0 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f3798007360 tx=0x7f3784005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a571b14880980f5a server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:56.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.160+0000 7f379e000640 1 -- 172.21.15.96:0/2961042526 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3798009be0 con 0x7f3784004d40 2024-05-28T06:14:56.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.160+0000 7f379e000640 1 -- 172.21.15.96:0/2961042526 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3798036600 con 0x7f3784004d40 2024-05-28T06:14:56.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.160+0000 7f379e000640 1 -- 172.21.15.96:0/2961042526 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37980369d0 con 0x7f3784004d40 2024-05-28T06:14:56.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.160+0000 7f37a5000640 1 -- 172.21.15.96:0/2961042526 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3784004d40 msgr2=0x7f37840051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:56.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.160+0000 7f37a5000640 1 --2- 172.21.15.96:0/2961042526 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3784004d40 0x7f37840051c0 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f3798007360 tx=0x7f3784005830 comp rx=0 tx=0).stop 2024-05-28T06:14:56.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.160+0000 7f379fe00640 1 -- 172.21.15.96:0/2961042526 reap_dead start 2024-05-28T06:14:56.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.160+0000 7f37a5000640 1 -- 172.21.15.96:0/2961042526 shutdown_connections 2024-05-28T06:14:56.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.160+0000 7f37a5000640 1 -- 172.21.15.96:0/2961042526 >> 172.21.15.96:0/2961042526 conn(0x7f37a00b1fc0 msgr2=0x7f37a00b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:56.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.160+0000 7f37a5000640 1 -- 172.21.15.96:0/2961042526 shutdown_connections 2024-05-28T06:14:56.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.160+0000 7f37a5000640 1 -- 172.21.15.96:0/2961042526 wait complete. 2024-05-28T06:14:56.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.163+0000 7f37a5000640 1 Processor -- start 2024-05-28T06:14:56.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.163+0000 7f37a5000640 1 -- start start 2024-05-28T06:14:56.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.164+0000 7f37a5000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f37a014c3d0 0x7f37a00b4e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:56.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.164+0000 7f37a5000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f37a0150ba0 0x7f37a01f4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:56.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.164+0000 7f37a5000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f37a00b4100 0x7f37a00b4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:56.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.164+0000 7f37a5000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f37a00b2db0 con 0x7f37a0150ba0 2024-05-28T06:14:56.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.164+0000 7f37a5000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f37a00b2c30 con 0x7f37a00b4100 2024-05-28T06:14:56.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.164+0000 7f37a5000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f37a00ad770 con 0x7f37a014c3d0 2024-05-28T06:14:56.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.164+0000 7f379ea00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f37a0150ba0 0x7f37a01f4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:56.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.164+0000 7f379ea00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f37a0150ba0 0x7f37a01f4d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54604/0 (socket says 172.21.15.96:54604) 2024-05-28T06:14:56.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.164+0000 7f379ea00640 1 -- 172.21.15.96:0/672056184 learned_addr learned my addr 172.21.15.96:0/672056184 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:56.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.164+0000 7f379ea00640 1 -- 172.21.15.96:0/672056184 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f37a014c3d0 msgr2=0x7f37a00b4e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:56.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.164+0000 7f379ea00640 1 --2- 172.21.15.96:0/672056184 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f37a014c3d0 0x7f37a00b4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:56.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.164+0000 7f379ea00640 1 -- 172.21.15.96:0/672056184 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f37a00b4100 msgr2=0x7f37a00b4580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:56.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.164+0000 7f379ea00640 1 --2- 172.21.15.96:0/672056184 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f37a00b4100 0x7f37a00b4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:56.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.165+0000 7f379ea00640 1 -- 172.21.15.96:0/672056184 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37a01518d0 con 0x7f37a0150ba0 2024-05-28T06:14:56.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.165+0000 7f379ea00640 1 --2- 172.21.15.96:0/672056184 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f37a0150ba0 0x7f37a01f4d30 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f378c00d800 tx=0x7f378c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:56.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.165+0000 7f3797400640 1 -- 172.21.15.96:0/672056184 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f378c00b640 con 0x7f37a0150ba0 2024-05-28T06:14:56.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.165+0000 7f3797400640 1 -- 172.21.15.96:0/672056184 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f378c019070 con 0x7f37a0150ba0 2024-05-28T06:14:56.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.165+0000 7f37a5000640 1 -- 172.21.15.96:0/672056184 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37a0151310 con 0x7f37a0150ba0 2024-05-28T06:14:56.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.165+0000 7f3797400640 1 -- 172.21.15.96:0/672056184 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f378c00ba20 con 0x7f37a0150ba0 2024-05-28T06:14:56.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.165+0000 7f37a5000640 1 -- 172.21.15.96:0/672056184 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37a01f4530 con 0x7f37a0150ba0 2024-05-28T06:14:56.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.166+0000 7f37a5000640 1 -- 172.21.15.96:0/672056184 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37a00689e0 con 0x7f37a0150ba0 2024-05-28T06:14:56.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.167+0000 7f3797400640 1 -- 172.21.15.96:0/672056184 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 22) ==== 92213+0+0 (secure 0 0 0) 0x7f378c00bbc0 con 0x7f37a0150ba0 2024-05-28T06:14:56.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.168+0000 7f3797400640 1 --2- 172.21.15.96:0/672056184 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f378006d610 0x7f378006fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:56.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.168+0000 7f3797400640 1 -- 172.21.15.96:0/672056184 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 5737+0+0 (secure 0 0 0) 0x7f378c08ac80 con 0x7f37a0150ba0 2024-05-28T06:14:56.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.169+0000 7f379f400640 1 --2- 172.21.15.96:0/672056184 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f378006d610 0x7f378006fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:56.171 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.170+0000 7f379f400640 1 --2- 172.21.15.96:0/672056184 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f378006d610 0x7f378006fad0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f3790000f60 tx=0x7f3790009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:56.174 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.172+0000 7f3797400640 1 -- 172.21.15.96:0/672056184 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f378c05a930 con 0x7f37a0150ba0 2024-05-28T06:14:56.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.405+0000 7f37a5000640 1 -- 172.21.15.96:0/672056184 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f37a006f7f0 con 0x7f37a0150ba0 2024-05-28T06:14:56.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.406+0000 7f3797400640 1 -- 172.21.15.96:0/672056184 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) ==== 94+0+5177 (secure 0 0 0) 0x7f378c05a230 con 0x7f37a0150ba0 2024-05-28T06:14:56.408 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:14:56.408 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":43,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5291,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6834/3727695025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":3727695025},{"type":"v1","addr":"172.21.15.96:6834","nonce":3727695025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38},{"gid":5300,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/324418606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6835","nonce":324418606},{"type":"v1","addr":"172.21.15.123:6837","nonce":324418606}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38},{"gid":5725,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/417566829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6833","nonce":417566829},{"type":"v1","addr":"172.21.15.96:6835","nonce":417566829}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-28T06:14:52.322597+0000","modified":"2024-05-28T06:14:55.415769+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7146},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7146":{"gid":7146,"name":"d","rank":0,"incarnation":39,"state":"up:active","state_seq":3,"addr":"172.21.15.123:6836/76143454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":76143454},{"type":"v1","addr":"172.21.15.123:6836","nonce":76143454}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7146,"qdb_cluster":[7146]},"id":5}]} 2024-05-28T06:14:56.409 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 43 2024-05-28T06:14:56.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.409+0000 7f37a5000640 1 -- 172.21.15.96:0/672056184 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f378006d610 msgr2=0x7f378006fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:56.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.409+0000 7f37a5000640 1 --2- 172.21.15.96:0/672056184 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f378006d610 0x7f378006fad0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f3790000f60 tx=0x7f3790009040 comp rx=0 tx=0).stop 2024-05-28T06:14:56.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.409+0000 7f37a5000640 1 -- 172.21.15.96:0/672056184 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f37a0150ba0 msgr2=0x7f37a01f4d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:56.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.409+0000 7f37a5000640 1 --2- 172.21.15.96:0/672056184 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f37a0150ba0 0x7f37a01f4d30 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f378c00d800 tx=0x7f378c00dcc0 comp rx=0 tx=0).stop 2024-05-28T06:14:56.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.409+0000 7f37a5000640 1 -- 172.21.15.96:0/672056184 shutdown_connections 2024-05-28T06:14:56.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.409+0000 7f37a5000640 1 --2- 172.21.15.96:0/672056184 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f378006d610 0x7f378006fad0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:56.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.409+0000 7f37a5000640 1 --2- 172.21.15.96:0/672056184 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f37a00b4100 0x7f37a00b4580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:56.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.409+0000 7f37a5000640 1 --2- 172.21.15.96:0/672056184 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f37a0150ba0 0x7f37a01f4d30 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:56.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.409+0000 7f37a5000640 1 --2- 172.21.15.96:0/672056184 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f37a014c3d0 0x7f37a00b4e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:56.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.409+0000 7f37a5000640 1 -- 172.21.15.96:0/672056184 >> 172.21.15.96:0/672056184 conn(0x7f37a00b1fc0 msgr2=0x7f37a00722c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:56.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.409+0000 7f37a5000640 1 -- 172.21.15.96:0/672056184 shutdown_connections 2024-05-28T06:14:56.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.409+0000 7f37a5000640 1 -- 172.21.15.96:0/672056184 wait complete. 2024-05-28T06:14:56.429 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:14:56.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.623+0000 7f9858a00640 1 Processor -- start 2024-05-28T06:14:56.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.623+0000 7f9858a00640 1 -- start start 2024-05-28T06:14:56.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.625+0000 7f9858a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f985406f860 0x7f98540b3150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:56.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.625+0000 7f9858a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f98540b3690 0x7f98540d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:56.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.625+0000 7f9858a00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f98540d09e0 0x7f9854160210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:56.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.625+0000 7f9858a00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f98540d0e50 con 0x7f98540b3690 2024-05-28T06:14:56.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.625+0000 7f9858a00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f985406ed10 con 0x7f98540d09e0 2024-05-28T06:14:56.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.625+0000 7f9858a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f985415cb00 con 0x7f985406f860 2024-05-28T06:14:56.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.625+0000 7f9849400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f98540b3690 0x7f98540d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:56.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.625+0000 7f9849400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f98540b3690 0x7f98540d04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54608/0 (socket says 172.21.15.96:54608) 2024-05-28T06:14:56.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.625+0000 7f9849400640 1 -- 172.21.15.96:0/3724278471 learned_addr learned my addr 172.21.15.96:0/3724278471 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:56.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.625+0000 7f9852800640 1 --2- 172.21.15.96:0/3724278471 >> v2:172.21.15.123:3300/0 conn(0x7f98540d09e0 0x7f9854160210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:56.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.625+0000 7f9851e00640 1 --2- 172.21.15.96:0/3724278471 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f985406f860 0x7f98540b3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:56.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.626+0000 7f9849400640 1 -- 172.21.15.96:0/3724278471 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f985406f860 msgr2=0x7f98540b3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:56.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.626+0000 7f9849400640 1 --2- 172.21.15.96:0/3724278471 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f985406f860 0x7f98540b3150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:56.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.626+0000 7f9849400640 1 -- 172.21.15.96:0/3724278471 >> v2:172.21.15.123:3300/0 conn(0x7f98540d09e0 msgr2=0x7f9854160210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:56.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.626+0000 7f9849400640 1 --2- 172.21.15.96:0/3724278471 >> v2:172.21.15.123:3300/0 conn(0x7f98540d09e0 0x7f9854160210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:56.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.626+0000 7f9849400640 1 -- 172.21.15.96:0/3724278471 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98540b3cc0 con 0x7f98540b3690 2024-05-28T06:14:56.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.627+0000 7f9849400640 1 --2- 172.21.15.96:0/3724278471 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f98540b3690 0x7f98540d04a0 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f985406c900 tx=0x7f983c02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebc71d7f0baa47b2 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:56.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.627+0000 7f9851400640 1 -- 172.21.15.96:0/3724278471 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f983c037070 con 0x7f98540b3690 2024-05-28T06:14:56.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.627+0000 7f9851400640 1 -- 172.21.15.96:0/3724278471 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f983c032440 con 0x7f98540b3690 2024-05-28T06:14:56.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.627+0000 7f9851400640 1 -- 172.21.15.96:0/3724278471 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f983c032810 con 0x7f98540b3690 2024-05-28T06:14:56.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.628+0000 7f9858a00640 1 -- 172.21.15.96:0/3724278471 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f98540b3690 msgr2=0x7f98540d04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:56.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.628+0000 7f9858a00640 1 --2- 172.21.15.96:0/3724278471 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f98540b3690 0x7f98540d04a0 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f985406c900 tx=0x7f983c02a1e0 comp rx=0 tx=0).stop 2024-05-28T06:14:56.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.628+0000 7f9858a00640 1 -- 172.21.15.96:0/3724278471 shutdown_connections 2024-05-28T06:14:56.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.628+0000 7f9858a00640 1 --2- 172.21.15.96:0/3724278471 >> v2:172.21.15.123:3300/0 conn(0x7f98540d09e0 0x7f9854160210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:56.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.628+0000 7f9858a00640 1 --2- 172.21.15.96:0/3724278471 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f98540b3690 0x7f98540d04a0 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:56.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.628+0000 7f9858a00640 1 --2- 172.21.15.96:0/3724278471 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f985406f860 0x7f98540b3150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:56.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.628+0000 7f9858a00640 1 -- 172.21.15.96:0/3724278471 >> 172.21.15.96:0/3724278471 conn(0x7f985406bae0 msgr2=0x7f985406bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:56.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.628+0000 7f9858a00640 1 -- 172.21.15.96:0/3724278471 shutdown_connections 2024-05-28T06:14:56.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.628+0000 7f9858a00640 1 -- 172.21.15.96:0/3724278471 wait complete. 2024-05-28T06:14:56.631 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.630+0000 7f9858a00640 1 Processor -- start 2024-05-28T06:14:56.631 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.630+0000 7f9858a00640 1 -- start start 2024-05-28T06:14:56.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.631+0000 7f9858a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f985406f860 0x7f98541f80d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:56.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.631+0000 7f9858a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f98540b3690 0x7f98541f8610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:56.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.631+0000 7f9858a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f98540d09e0 0x7f98541f8b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:56.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.631+0000 7f9858a00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f98540b4560 con 0x7f98540d09e0 2024-05-28T06:14:56.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.631+0000 7f9858a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f98540b4280 con 0x7f98540b3690 2024-05-28T06:14:56.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.631+0000 7f9858a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f98540b4840 con 0x7f985406f860 2024-05-28T06:14:56.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.631+0000 7f9852800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f98540d09e0 0x7f98541f8b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:56.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.631+0000 7f9852800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f98540d09e0 0x7f98541f8b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54614/0 (socket says 172.21.15.96:54614) 2024-05-28T06:14:56.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.631+0000 7f9852800640 1 -- 172.21.15.96:0/2006384435 learned_addr learned my addr 172.21.15.96:0/2006384435 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:56.635 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.632+0000 7f9849400640 1 --2- 172.21.15.96:0/2006384435 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f98540b3690 0x7f98541f8610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:56.635 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.632+0000 7f9851e00640 1 --2- 172.21.15.96:0/2006384435 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f985406f860 0x7f98541f80d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:56.635 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.632+0000 7f9852800640 1 -- 172.21.15.96:0/2006384435 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f985406f860 msgr2=0x7f98541f80d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:56.635 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.632+0000 7f9852800640 1 --2- 172.21.15.96:0/2006384435 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f985406f860 0x7f98541f80d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:56.635 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.632+0000 7f9852800640 1 -- 172.21.15.96:0/2006384435 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f98540b3690 msgr2=0x7f98541f8610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:56.635 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.632+0000 7f9852800640 1 --2- 172.21.15.96:0/2006384435 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f98540b3690 0x7f98541f8610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:56.635 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.632+0000 7f9852800640 1 -- 172.21.15.96:0/2006384435 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98540ca180 con 0x7f98540d09e0 2024-05-28T06:14:56.635 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.632+0000 7f9852800640 1 --2- 172.21.15.96:0/2006384435 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f98540d09e0 0x7f98541f8b50 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f984c007e90 tx=0x7f984c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:56.635 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.632+0000 7f984aa00640 1 -- 172.21.15.96:0/2006384435 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f984c014420 con 0x7f98540d09e0 2024-05-28T06:14:56.635 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.632+0000 7f984aa00640 1 -- 172.21.15.96:0/2006384435 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f984c019070 con 0x7f98540d09e0 2024-05-28T06:14:56.635 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.632+0000 7f984aa00640 1 -- 172.21.15.96:0/2006384435 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f984c014800 con 0x7f98540d09e0 2024-05-28T06:14:56.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.632+0000 7f9858a00640 1 -- 172.21.15.96:0/2006384435 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98540c98e0 con 0x7f98540d09e0 2024-05-28T06:14:56.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.632+0000 7f9858a00640 1 -- 172.21.15.96:0/2006384435 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98540c9ea0 con 0x7f98540d09e0 2024-05-28T06:14:56.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.633+0000 7f9858a00640 1 -- 172.21.15.96:0/2006384435 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98540689e0 con 0x7f98540d09e0 2024-05-28T06:14:56.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.634+0000 7f984aa00640 1 -- 172.21.15.96:0/2006384435 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 22) ==== 92213+0+0 (secure 0 0 0) 0x7f984c0149a0 con 0x7f98540d09e0 2024-05-28T06:14:56.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.635+0000 7f984aa00640 1 --2- 172.21.15.96:0/2006384435 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f983006d5e0 0x7f983006faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:56.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.635+0000 7f984aa00640 1 -- 172.21.15.96:0/2006384435 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 5737+0+0 (secure 0 0 0) 0x7f984c08adc0 con 0x7f98540d09e0 2024-05-28T06:14:56.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.636+0000 7f9851e00640 1 --2- 172.21.15.96:0/2006384435 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f983006d5e0 0x7f983006faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:56.638 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.637+0000 7f9851e00640 1 --2- 172.21.15.96:0/2006384435 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f983006d5e0 0x7f983006faa0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f9844006fd0 tx=0x7f9844008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:56.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.640+0000 7f984aa00640 1 -- 172.21.15.96:0/2006384435 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f98540ca180 con 0x7f98540d09e0 2024-05-28T06:14:56.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.843+0000 7f9858a00640 1 -- 172.21.15.96:0/2006384435 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f98540cc040 con 0x7f98540d09e0 2024-05-28T06:14:56.847 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.846+0000 7f984aa00640 1 -- 172.21.15.96:0/2006384435 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v74) ==== 74+0+16723 (secure 0 0 0) 0x7f984c05ac40 con 0x7f98540d09e0 2024-05-28T06:14:56.848 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:14:56.848 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":74,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:14:53.906027+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-05-28T06:14:50.098829+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-05-28T06:14:51.318311+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:14:56.850 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.848+0000 7f9858a00640 1 -- 172.21.15.96:0/2006384435 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f983006d5e0 msgr2=0x7f983006faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:56.850 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.848+0000 7f9858a00640 1 --2- 172.21.15.96:0/2006384435 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f983006d5e0 0x7f983006faa0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f9844006fd0 tx=0x7f9844008040 comp rx=0 tx=0).stop 2024-05-28T06:14:56.850 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.848+0000 7f9858a00640 1 -- 172.21.15.96:0/2006384435 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f98540d09e0 msgr2=0x7f98541f8b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:56.850 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.848+0000 7f9858a00640 1 --2- 172.21.15.96:0/2006384435 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f98540d09e0 0x7f98541f8b50 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f984c007e90 tx=0x7f984c00e4b0 comp rx=0 tx=0).stop 2024-05-28T06:14:56.850 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.848+0000 7f9858a00640 1 -- 172.21.15.96:0/2006384435 shutdown_connections 2024-05-28T06:14:56.850 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.848+0000 7f9858a00640 1 --2- 172.21.15.96:0/2006384435 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f983006d5e0 0x7f983006faa0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:56.850 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.848+0000 7f9858a00640 1 --2- 172.21.15.96:0/2006384435 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f98540d09e0 0x7f98541f8b50 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:56.851 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.848+0000 7f9858a00640 1 --2- 172.21.15.96:0/2006384435 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f98540b3690 0x7f98541f8610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:56.851 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.848+0000 7f9858a00640 1 --2- 172.21.15.96:0/2006384435 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f985406f860 0x7f98541f80d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:56.851 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.848+0000 7f9858a00640 1 -- 172.21.15.96:0/2006384435 >> 172.21.15.96:0/2006384435 conn(0x7f985406bae0 msgr2=0x7f9854073050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:56.851 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.848+0000 7f9858a00640 1 -- 172.21.15.96:0/2006384435 shutdown_connections 2024-05-28T06:14:56.851 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:56.848+0000 7f9858a00640 1 -- 172.21.15.96:0/2006384435 wait complete. 2024-05-28T06:14:56.869 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-05-28T06:14:56.869 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-28T06:14:57.071 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.069+0000 7f378b200640 1 Processor -- start 2024-05-28T06:14:57.071 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.069+0000 7f378b200640 1 -- start start 2024-05-28T06:14:57.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.071+0000 7f378b200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f37840d04a0 0x7f37840d0930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:57.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.071+0000 7f378b200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f37840ad130 0x7f37840ad580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:57.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.071+0000 7f378b200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f378406ed10 con 0x7f378406f860 2024-05-28T06:14:57.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.071+0000 7f378b200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f378415cb00 con 0x7f37840d04a0 2024-05-28T06:14:57.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.071+0000 7f378b200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f37840b3cc0 con 0x7f37840ad130 2024-05-28T06:14:57.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.071+0000 7f3783e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f378406f860 0x7f37840b3150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:52574/0 (socket says 172.21.15.96:52574) 2024-05-28T06:14:57.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.071+0000 7f3783e00640 1 -- 172.21.15.96:0/4163641240 learned_addr learned my addr 172.21.15.96:0/4163641240 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:14:57.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.071+0000 7f3783400640 1 --2- 172.21.15.96:0/4163641240 >> v2:172.21.15.96:3300/0 conn(0x7f37840d04a0 0x7f37840d0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:57.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.071+0000 7f3789200640 1 --2- 172.21.15.96:0/4163641240 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f37840ad130 0x7f37840ad580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:57.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.071+0000 7f3782a00640 1 -- 172.21.15.96:0/4163641240 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 933419905 0 0) 0x7f3774001110 con 0x7f378406f860 2024-05-28T06:14:57.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.072+0000 7f3782a00640 1 -- 172.21.15.96:0/4163641240 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3768003520 con 0x7f378406f860 2024-05-28T06:14:57.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.072+0000 7f3783400640 1 -- 172.21.15.96:0/4163641240 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f37840ad130 msgr2=0x7f37840ad580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:57.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.072+0000 7f3783400640 1 --2- 172.21.15.96:0/4163641240 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f37840ad130 0x7f37840ad580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:57.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.072+0000 7f3783400640 1 -- 172.21.15.96:0/4163641240 >> v1:172.21.15.96:6789/0 conn(0x7f378406f860 legacy=0x7f37840b3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:57.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.072+0000 7f3783400640 1 -- 172.21.15.96:0/4163641240 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37840b3fa0 con 0x7f37840d04a0 2024-05-28T06:14:57.075 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.073+0000 7f3783400640 1 --2- 172.21.15.96:0/4163641240 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f37840d04a0 0x7f37840d0930 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f37700097a0 tx=0x7f377002a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=762ef0552e8efab0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:57.075 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.073+0000 7f3782a00640 1 -- 172.21.15.96:0/4163641240 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3770037070 con 0x7f37840d04a0 2024-05-28T06:14:57.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.073+0000 7f3782a00640 1 -- 172.21.15.96:0/4163641240 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3770032440 con 0x7f37840d04a0 2024-05-28T06:14:57.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.073+0000 7f3782a00640 1 -- 172.21.15.96:0/4163641240 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3770032810 con 0x7f37840d04a0 2024-05-28T06:14:57.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.074+0000 7f378b200640 1 -- 172.21.15.96:0/4163641240 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f37840d04a0 msgr2=0x7f37840d0930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:57.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.074+0000 7f378b200640 1 --2- 172.21.15.96:0/4163641240 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f37840d04a0 0x7f37840d0930 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f37700097a0 tx=0x7f377002a180 comp rx=0 tx=0).stop 2024-05-28T06:14:57.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.074+0000 7f378b200640 1 -- 172.21.15.96:0/4163641240 shutdown_connections 2024-05-28T06:14:57.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.074+0000 7f378b200640 1 --2- 172.21.15.96:0/4163641240 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f37840ad130 0x7f37840ad580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:57.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.074+0000 7f378b200640 1 --2- 172.21.15.96:0/4163641240 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f37840d04a0 0x7f37840d0930 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:57.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.074+0000 7f378b200640 1 -- 172.21.15.96:0/4163641240 >> 172.21.15.96:0/4163641240 conn(0x7f378406bae0 msgr2=0x7f378406bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:57.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.074+0000 7f378b200640 1 -- 172.21.15.96:0/4163641240 shutdown_connections 2024-05-28T06:14:57.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.074+0000 7f378b200640 1 -- 172.21.15.96:0/4163641240 wait complete. 2024-05-28T06:14:57.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.076+0000 7f378b200640 1 Processor -- start 2024-05-28T06:14:57.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.076+0000 7f378b200640 1 -- start start 2024-05-28T06:14:57.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.077+0000 7f378b200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f378406f860 0x7f37840c8270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:57.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.077+0000 7f378b200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f37840ad130 0x7f37841f54d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:57.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.077+0000 7f378b200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f37840d04a0 0x7f37840c9bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:57.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.077+0000 7f378b200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f37840b4840 con 0x7f37840ad130 2024-05-28T06:14:57.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.077+0000 7f378b200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f37840b4560 con 0x7f378406f860 2024-05-28T06:14:57.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.077+0000 7f378b200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f37840b3860 con 0x7f37840d04a0 2024-05-28T06:14:57.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.077+0000 7f3783400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f37840ad130 0x7f37841f54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:57.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.077+0000 7f3783400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f37840ad130 0x7f37841f54d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54646/0 (socket says 172.21.15.96:54646) 2024-05-28T06:14:57.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.077+0000 7f3783400640 1 -- 172.21.15.96:0/736735347 learned_addr learned my addr 172.21.15.96:0/736735347 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:57.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.077+0000 7f3783400640 1 -- 172.21.15.96:0/736735347 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f37840d04a0 msgr2=0x7f37840c9bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:57.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.077+0000 7f3783400640 1 --2- 172.21.15.96:0/736735347 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f37840d04a0 0x7f37840c9bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:57.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.077+0000 7f3783e00640 1 --2- 172.21.15.96:0/736735347 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f378406f860 0x7f37840c8270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:57.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.077+0000 7f3783400640 1 -- 172.21.15.96:0/736735347 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f378406f860 msgr2=0x7f37840c8270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:57.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.077+0000 7f3783400640 1 --2- 172.21.15.96:0/736735347 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f378406f860 0x7f37840c8270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:57.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.077+0000 7f3783400640 1 -- 172.21.15.96:0/736735347 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f378415cf60 con 0x7f37840ad130 2024-05-28T06:14:57.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.078+0000 7f3783400640 1 --2- 172.21.15.96:0/736735347 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f37840ad130 0x7f37841f54d0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f37700098c0 tx=0x7f37700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:57.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.078+0000 7f377be00640 1 -- 172.21.15.96:0/736735347 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3770037040 con 0x7f37840ad130 2024-05-28T06:14:57.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.078+0000 7f377be00640 1 -- 172.21.15.96:0/736735347 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3770042070 con 0x7f37840ad130 2024-05-28T06:14:57.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.078+0000 7f377be00640 1 -- 172.21.15.96:0/736735347 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3770030920 con 0x7f37840ad130 2024-05-28T06:14:57.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.078+0000 7f378b200640 1 -- 172.21.15.96:0/736735347 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f378415d240 con 0x7f37840ad130 2024-05-28T06:14:57.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.078+0000 7f378b200640 1 -- 172.21.15.96:0/736735347 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f378415d800 con 0x7f37840ad130 2024-05-28T06:14:57.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.079+0000 7f378b200640 1 -- 172.21.15.96:0/736735347 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37840ae590 con 0x7f37840ad130 2024-05-28T06:14:57.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.080+0000 7f377be00640 1 -- 172.21.15.96:0/736735347 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 22) ==== 92213+0+0 (secure 0 0 0) 0x7f3770030ac0 con 0x7f37840ad130 2024-05-28T06:14:57.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.081+0000 7f377be00640 1 --2- 172.21.15.96:0/736735347 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f376406d5e0 0x7f376406faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:57.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.081+0000 7f377be00640 1 -- 172.21.15.96:0/736735347 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 5737+0+0 (secure 0 0 0) 0x7f37700ab280 con 0x7f37840ad130 2024-05-28T06:14:57.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.082+0000 7f3783e00640 1 --2- 172.21.15.96:0/736735347 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f376406d5e0 0x7f376406faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:57.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.083+0000 7f3783e00640 1 --2- 172.21.15.96:0/736735347 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f376406d5e0 0x7f376406faa0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f37740012b0 tx=0x7f3774009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:57.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.085+0000 7f377be00640 1 -- 172.21.15.96:0/736735347 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f377007a820 con 0x7f37840ad130 2024-05-28T06:14:57.239 INFO:tasks.ceph.mds.d.smithi123.stderr:2024-05-28T06:14:57.237+0000 7fc869200640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-28T06:14:57.282 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.280+0000 7f378b200640 1 -- 172.21.15.96:0/736735347 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f378415cd10 con 0x7f376406d5e0 2024-05-28T06:14:57.302 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.301+0000 7f377be00640 1 -- 172.21.15.96:0/736735347 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252066 (secure 0 0 0) 0x7f378415cd10 con 0x7f376406d5e0 2024-05-28T06:14:57.306 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:14:57.307 INFO:teuthology.orchestra.run.smithi096.stderr:dumped all 2024-05-28T06:14:57.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.306+0000 7f378b200640 1 -- 172.21.15.96:0/736735347 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f376406d5e0 msgr2=0x7f376406faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:57.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.306+0000 7f378b200640 1 --2- 172.21.15.96:0/736735347 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f376406d5e0 0x7f376406faa0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f37740012b0 tx=0x7f3774009040 comp rx=0 tx=0).stop 2024-05-28T06:14:57.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.306+0000 7f378b200640 1 -- 172.21.15.96:0/736735347 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f37840ad130 msgr2=0x7f37841f54d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:57.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.306+0000 7f378b200640 1 --2- 172.21.15.96:0/736735347 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f37840ad130 0x7f37841f54d0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f37700098c0 tx=0x7f37700046c0 comp rx=0 tx=0).stop 2024-05-28T06:14:57.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.306+0000 7f378b200640 1 -- 172.21.15.96:0/736735347 shutdown_connections 2024-05-28T06:14:57.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.306+0000 7f378b200640 1 --2- 172.21.15.96:0/736735347 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f376406d5e0 0x7f376406faa0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:57.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.306+0000 7f378b200640 1 --2- 172.21.15.96:0/736735347 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f37840d04a0 0x7f37840c9bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:57.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.306+0000 7f378b200640 1 --2- 172.21.15.96:0/736735347 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f37840ad130 0x7f37841f54d0 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:57.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.306+0000 7f378b200640 1 --2- 172.21.15.96:0/736735347 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f378406f860 0x7f37840c8270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:57.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.306+0000 7f378b200640 1 -- 172.21.15.96:0/736735347 >> 172.21.15.96:0/736735347 conn(0x7f378406bae0 msgr2=0x7f3784073050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:57.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.306+0000 7f378b200640 1 -- 172.21.15.96:0/736735347 shutdown_connections 2024-05-28T06:14:57.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.306+0000 7f378b200640 1 -- 172.21.15.96:0/736735347 wait complete. 2024-05-28T06:14:57.326 INFO:teuthology.orchestra.run.smithi096.stdout:{"pg_ready":true,"pg_map":{"version":395,"stamp":"2024-05-28T06:14:55.748588+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":134,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":447816,"kb_used_data":8204,"kb_used_omap":12,"kb_used_meta":439475,"kb_avail":754526904,"statfs":{"total":773094113280,"available":772635549696,"internally_reserved":0,"allocated":8400896,"data_stored":4236852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":450022960},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.884907"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.507269+0000","last_change":"2024-05-28T06:14:52.925298+0000","last_active":"2024-05-28T06:14:54.507269+0000","last_peered":"2024-05-28T06:14:54.507269+0000","last_clean":"2024-05-28T06:14:54.507269+0000","last_became_active":"2024-05-28T06:14:52.925016+0000","last_became_peered":"2024-05-28T06:14:52.925016+0000","last_unstale":"2024-05-28T06:14:54.507269+0000","last_undegraded":"2024-05-28T06:14:54.507269+0000","last_fullsized":"2024-05-28T06:14:54.507269+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:32:21.563791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.507046+0000","last_change":"2024-05-28T06:14:52.927523+0000","last_active":"2024-05-28T06:14:54.507046+0000","last_peered":"2024-05-28T06:14:54.507046+0000","last_clean":"2024-05-28T06:14:54.507046+0000","last_became_active":"2024-05-28T06:14:52.927242+0000","last_became_peered":"2024-05-28T06:14:52.927242+0000","last_unstale":"2024-05-28T06:14:54.507046+0000","last_undegraded":"2024-05-28T06:14:54.507046+0000","last_fullsized":"2024-05-28T06:14:54.507046+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:39:34.119648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.503639+0000","last_change":"2024-05-28T06:14:52.912996+0000","last_active":"2024-05-28T06:14:54.503639+0000","last_peered":"2024-05-28T06:14:54.503639+0000","last_clean":"2024-05-28T06:14:54.503639+0000","last_became_active":"2024-05-28T06:14:52.912661+0000","last_became_peered":"2024-05-28T06:14:52.912661+0000","last_unstale":"2024-05-28T06:14:54.503639+0000","last_undegraded":"2024-05-28T06:14:54.503639+0000","last_fullsized":"2024-05-28T06:14:54.503639+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:47:56.376539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.502406+0000","last_change":"2024-05-28T06:14:52.930703+0000","last_active":"2024-05-28T06:14:54.502406+0000","last_peered":"2024-05-28T06:14:54.502406+0000","last_clean":"2024-05-28T06:14:54.502406+0000","last_became_active":"2024-05-28T06:14:52.929719+0000","last_became_peered":"2024-05-28T06:14:52.929719+0000","last_unstale":"2024-05-28T06:14:54.502406+0000","last_undegraded":"2024-05-28T06:14:54.502406+0000","last_fullsized":"2024-05-28T06:14:54.502406+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:40:46.832691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.280800+0000","last_change":"2024-05-28T06:14:52.913669+0000","last_active":"2024-05-28T06:14:54.280800+0000","last_peered":"2024-05-28T06:14:54.280800+0000","last_clean":"2024-05-28T06:14:54.280800+0000","last_became_active":"2024-05-28T06:14:52.913377+0000","last_became_peered":"2024-05-28T06:14:52.913377+0000","last_unstale":"2024-05-28T06:14:54.280800+0000","last_undegraded":"2024-05-28T06:14:54.280800+0000","last_fullsized":"2024-05-28T06:14:54.280800+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:49:39.185365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.504046+0000","last_change":"2024-05-28T06:14:52.919901+0000","last_active":"2024-05-28T06:14:54.504046+0000","last_peered":"2024-05-28T06:14:54.504046+0000","last_clean":"2024-05-28T06:14:54.504046+0000","last_became_active":"2024-05-28T06:14:52.919587+0000","last_became_peered":"2024-05-28T06:14:52.919587+0000","last_unstale":"2024-05-28T06:14:54.504046+0000","last_undegraded":"2024-05-28T06:14:54.504046+0000","last_fullsized":"2024-05-28T06:14:54.504046+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:55:22.737075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.502043+0000","last_change":"2024-05-28T06:14:52.907310+0000","last_active":"2024-05-28T06:14:54.502043+0000","last_peered":"2024-05-28T06:14:54.502043+0000","last_clean":"2024-05-28T06:14:54.502043+0000","last_became_active":"2024-05-28T06:14:52.906416+0000","last_became_peered":"2024-05-28T06:14:52.906416+0000","last_unstale":"2024-05-28T06:14:54.502043+0000","last_undegraded":"2024-05-28T06:14:54.502043+0000","last_fullsized":"2024-05-28T06:14:54.502043+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:45:53.262592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.503823+0000","last_change":"2024-05-28T06:14:52.909378+0000","last_active":"2024-05-28T06:14:54.503823+0000","last_peered":"2024-05-28T06:14:54.503823+0000","last_clean":"2024-05-28T06:14:54.503823+0000","last_became_active":"2024-05-28T06:14:52.908971+0000","last_became_peered":"2024-05-28T06:14:52.908971+0000","last_unstale":"2024-05-28T06:14:54.503823+0000","last_undegraded":"2024-05-28T06:14:54.503823+0000","last_fullsized":"2024-05-28T06:14:54.503823+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:24:46.937524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.281669+0000","last_change":"2024-05-28T06:14:52.924723+0000","last_active":"2024-05-28T06:14:54.281669+0000","last_peered":"2024-05-28T06:14:54.281669+0000","last_clean":"2024-05-28T06:14:54.281669+0000","last_became_active":"2024-05-28T06:14:52.924254+0000","last_became_peered":"2024-05-28T06:14:52.924254+0000","last_unstale":"2024-05-28T06:14:54.281669+0000","last_undegraded":"2024-05-28T06:14:54.281669+0000","last_fullsized":"2024-05-28T06:14:54.281669+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:21:31.600547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:53.923666+0000","last_change":"2024-05-28T06:14:52.920854+0000","last_active":"2024-05-28T06:14:53.923666+0000","last_peered":"2024-05-28T06:14:53.923666+0000","last_clean":"2024-05-28T06:14:53.923666+0000","last_became_active":"2024-05-28T06:14:52.920575+0000","last_became_peered":"2024-05-28T06:14:52.920575+0000","last_unstale":"2024-05-28T06:14:53.923666+0000","last_undegraded":"2024-05-28T06:14:53.923666+0000","last_fullsized":"2024-05-28T06:14:53.923666+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:56:57.854220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.506694+0000","last_change":"2024-05-28T06:14:52.927237+0000","last_active":"2024-05-28T06:14:54.506694+0000","last_peered":"2024-05-28T06:14:54.506694+0000","last_clean":"2024-05-28T06:14:54.506694+0000","last_became_active":"2024-05-28T06:14:52.926247+0000","last_became_peered":"2024-05-28T06:14:52.926247+0000","last_unstale":"2024-05-28T06:14:54.506694+0000","last_undegraded":"2024-05-28T06:14:54.506694+0000","last_fullsized":"2024-05-28T06:14:54.506694+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:34:06.413585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.502397+0000","last_change":"2024-05-28T06:14:52.921867+0000","last_active":"2024-05-28T06:14:54.502397+0000","last_peered":"2024-05-28T06:14:54.502397+0000","last_clean":"2024-05-28T06:14:54.502397+0000","last_became_active":"2024-05-28T06:14:52.921562+0000","last_became_peered":"2024-05-28T06:14:52.921562+0000","last_unstale":"2024-05-28T06:14:54.502397+0000","last_undegraded":"2024-05-28T06:14:54.502397+0000","last_fullsized":"2024-05-28T06:14:54.502397+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:42:10.393703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:53.922396+0000","last_change":"2024-05-28T06:14:52.911800+0000","last_active":"2024-05-28T06:14:53.922396+0000","last_peered":"2024-05-28T06:14:53.922396+0000","last_clean":"2024-05-28T06:14:53.922396+0000","last_became_active":"2024-05-28T06:14:52.911474+0000","last_became_peered":"2024-05-28T06:14:52.911474+0000","last_unstale":"2024-05-28T06:14:53.922396+0000","last_undegraded":"2024-05-28T06:14:53.922396+0000","last_fullsized":"2024-05-28T06:14:53.922396+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:59:01.279260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.281175+0000","last_change":"2024-05-28T06:14:52.921543+0000","last_active":"2024-05-28T06:14:54.281175+0000","last_peered":"2024-05-28T06:14:54.281175+0000","last_clean":"2024-05-28T06:14:54.281175+0000","last_became_active":"2024-05-28T06:14:52.921253+0000","last_became_peered":"2024-05-28T06:14:52.921253+0000","last_unstale":"2024-05-28T06:14:54.281175+0000","last_undegraded":"2024-05-28T06:14:54.281175+0000","last_fullsized":"2024-05-28T06:14:54.281175+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:43:59.310423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:53.922429+0000","last_change":"2024-05-28T06:14:52.917090+0000","last_active":"2024-05-28T06:14:53.922429+0000","last_peered":"2024-05-28T06:14:53.922429+0000","last_clean":"2024-05-28T06:14:53.922429+0000","last_became_active":"2024-05-28T06:14:52.916797+0000","last_became_peered":"2024-05-28T06:14:52.916797+0000","last_unstale":"2024-05-28T06:14:53.922429+0000","last_undegraded":"2024-05-28T06:14:53.922429+0000","last_fullsized":"2024-05-28T06:14:53.922429+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:45:45.895312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:53.922683+0000","last_change":"2024-05-28T06:14:52.912534+0000","last_active":"2024-05-28T06:14:53.922683+0000","last_peered":"2024-05-28T06:14:53.922683+0000","last_clean":"2024-05-28T06:14:53.922683+0000","last_became_active":"2024-05-28T06:14:52.912278+0000","last_became_peered":"2024-05-28T06:14:52.912278+0000","last_unstale":"2024-05-28T06:14:53.922683+0000","last_undegraded":"2024-05-28T06:14:53.922683+0000","last_fullsized":"2024-05-28T06:14:53.922683+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:23:03.387649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.281690+0000","last_change":"2024-05-28T06:14:52.922992+0000","last_active":"2024-05-28T06:14:54.281690+0000","last_peered":"2024-05-28T06:14:54.281690+0000","last_clean":"2024-05-28T06:14:54.281690+0000","last_became_active":"2024-05-28T06:14:52.922734+0000","last_became_peered":"2024-05-28T06:14:52.922734+0000","last_unstale":"2024-05-28T06:14:54.281690+0000","last_undegraded":"2024-05-28T06:14:54.281690+0000","last_fullsized":"2024-05-28T06:14:54.281690+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:55:49.391662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.503051+0000","last_change":"2024-05-28T06:14:52.922615+0000","last_active":"2024-05-28T06:14:54.503051+0000","last_peered":"2024-05-28T06:14:54.503051+0000","last_clean":"2024-05-28T06:14:54.503051+0000","last_became_active":"2024-05-28T06:14:52.922248+0000","last_became_peered":"2024-05-28T06:14:52.922248+0000","last_unstale":"2024-05-28T06:14:54.503051+0000","last_undegraded":"2024-05-28T06:14:54.503051+0000","last_fullsized":"2024-05-28T06:14:54.503051+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:49:24.495950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.505066+0000","last_change":"2024-05-28T06:14:52.910258+0000","last_active":"2024-05-28T06:14:54.505066+0000","last_peered":"2024-05-28T06:14:54.505066+0000","last_clean":"2024-05-28T06:14:54.505066+0000","last_became_active":"2024-05-28T06:14:52.909966+0000","last_became_peered":"2024-05-28T06:14:52.909966+0000","last_unstale":"2024-05-28T06:14:54.505066+0000","last_undegraded":"2024-05-28T06:14:54.505066+0000","last_fullsized":"2024-05-28T06:14:54.505066+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:36:20.269827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:53.922966+0000","last_change":"2024-05-28T06:14:52.926424+0000","last_active":"2024-05-28T06:14:53.922966+0000","last_peered":"2024-05-28T06:14:53.922966+0000","last_clean":"2024-05-28T06:14:53.922966+0000","last_became_active":"2024-05-28T06:14:52.926032+0000","last_became_peered":"2024-05-28T06:14:52.926032+0000","last_unstale":"2024-05-28T06:14:53.922966+0000","last_undegraded":"2024-05-28T06:14:53.922966+0000","last_fullsized":"2024-05-28T06:14:53.922966+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:20:40.686529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:53.922766+0000","last_change":"2024-05-28T06:14:52.911212+0000","last_active":"2024-05-28T06:14:53.922766+0000","last_peered":"2024-05-28T06:14:53.922766+0000","last_clean":"2024-05-28T06:14:53.922766+0000","last_became_active":"2024-05-28T06:14:52.908274+0000","last_became_peered":"2024-05-28T06:14:52.908274+0000","last_unstale":"2024-05-28T06:14:53.922766+0000","last_undegraded":"2024-05-28T06:14:53.922766+0000","last_fullsized":"2024-05-28T06:14:53.922766+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:31:26.679161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:53.922595+0000","last_change":"2024-05-28T06:14:52.921526+0000","last_active":"2024-05-28T06:14:53.922595+0000","last_peered":"2024-05-28T06:14:53.922595+0000","last_clean":"2024-05-28T06:14:53.922595+0000","last_became_active":"2024-05-28T06:14:52.921229+0000","last_became_peered":"2024-05-28T06:14:52.921229+0000","last_unstale":"2024-05-28T06:14:53.922595+0000","last_undegraded":"2024-05-28T06:14:53.922595+0000","last_fullsized":"2024-05-28T06:14:53.922595+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:55:22.737075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.507456+0000","last_change":"2024-05-28T06:14:52.924412+0000","last_active":"2024-05-28T06:14:54.507456+0000","last_peered":"2024-05-28T06:14:54.507456+0000","last_clean":"2024-05-28T06:14:54.507456+0000","last_became_active":"2024-05-28T06:14:52.924154+0000","last_became_peered":"2024-05-28T06:14:52.924154+0000","last_unstale":"2024-05-28T06:14:54.507456+0000","last_undegraded":"2024-05-28T06:14:54.507456+0000","last_fullsized":"2024-05-28T06:14:54.507456+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:25:35.889139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.503278+0000","last_change":"2024-05-28T06:14:52.918515+0000","last_active":"2024-05-28T06:14:54.503278+0000","last_peered":"2024-05-28T06:14:54.503278+0000","last_clean":"2024-05-28T06:14:54.503278+0000","last_became_active":"2024-05-28T06:14:52.918217+0000","last_became_peered":"2024-05-28T06:14:52.918217+0000","last_unstale":"2024-05-28T06:14:54.503278+0000","last_undegraded":"2024-05-28T06:14:54.503278+0000","last_fullsized":"2024-05-28T06:14:54.503278+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:59:01.279260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:53.923517+0000","last_change":"2024-05-28T06:14:52.927009+0000","last_active":"2024-05-28T06:14:53.923517+0000","last_peered":"2024-05-28T06:14:53.923517+0000","last_clean":"2024-05-28T06:14:53.923517+0000","last_became_active":"2024-05-28T06:14:52.926731+0000","last_became_peered":"2024-05-28T06:14:52.926731+0000","last_unstale":"2024-05-28T06:14:53.923517+0000","last_undegraded":"2024-05-28T06:14:53.923517+0000","last_fullsized":"2024-05-28T06:14:53.923517+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:02:32.395801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.503512+0000","last_change":"2024-05-28T06:14:52.921482+0000","last_active":"2024-05-28T06:14:54.503512+0000","last_peered":"2024-05-28T06:14:54.503512+0000","last_clean":"2024-05-28T06:14:54.503512+0000","last_became_active":"2024-05-28T06:14:52.921225+0000","last_became_peered":"2024-05-28T06:14:52.921225+0000","last_unstale":"2024-05-28T06:14:54.503512+0000","last_undegraded":"2024-05-28T06:14:54.503512+0000","last_fullsized":"2024-05-28T06:14:54.503512+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:59:01.279260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.503928+0000","last_change":"2024-05-28T06:14:52.913616+0000","last_active":"2024-05-28T06:14:54.503928+0000","last_peered":"2024-05-28T06:14:54.503928+0000","last_clean":"2024-05-28T06:14:54.503928+0000","last_became_active":"2024-05-28T06:14:52.913343+0000","last_became_peered":"2024-05-28T06:14:52.913343+0000","last_unstale":"2024-05-28T06:14:54.503928+0000","last_undegraded":"2024-05-28T06:14:54.503928+0000","last_fullsized":"2024-05-28T06:14:54.503928+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:32:41.688023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:53.923200+0000","last_change":"2024-05-28T06:14:52.922737+0000","last_active":"2024-05-28T06:14:53.923200+0000","last_peered":"2024-05-28T06:14:53.923200+0000","last_clean":"2024-05-28T06:14:53.923200+0000","last_became_active":"2024-05-28T06:14:52.922456+0000","last_became_peered":"2024-05-28T06:14:52.922456+0000","last_unstale":"2024-05-28T06:14:53.923200+0000","last_undegraded":"2024-05-28T06:14:53.923200+0000","last_fullsized":"2024-05-28T06:14:53.923200+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:26:14.502049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.503317+0000","last_change":"2024-05-28T06:14:52.927138+0000","last_active":"2024-05-28T06:14:54.503317+0000","last_peered":"2024-05-28T06:14:54.503317+0000","last_clean":"2024-05-28T06:14:54.503317+0000","last_became_active":"2024-05-28T06:14:52.926740+0000","last_became_peered":"2024-05-28T06:14:52.926740+0000","last_unstale":"2024-05-28T06:14:54.503317+0000","last_undegraded":"2024-05-28T06:14:54.503317+0000","last_fullsized":"2024-05-28T06:14:54.503317+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:30:07.820114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.502414+0000","last_change":"2024-05-28T06:14:52.917517+0000","last_active":"2024-05-28T06:14:54.502414+0000","last_peered":"2024-05-28T06:14:54.502414+0000","last_clean":"2024-05-28T06:14:54.502414+0000","last_became_active":"2024-05-28T06:14:52.916217+0000","last_became_peered":"2024-05-28T06:14:52.916217+0000","last_unstale":"2024-05-28T06:14:54.502414+0000","last_undegraded":"2024-05-28T06:14:54.502414+0000","last_fullsized":"2024-05-28T06:14:54.502414+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:24:28.642199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.502531+0000","last_change":"2024-05-28T06:14:52.925986+0000","last_active":"2024-05-28T06:14:54.502531+0000","last_peered":"2024-05-28T06:14:54.502531+0000","last_clean":"2024-05-28T06:14:54.502531+0000","last_became_active":"2024-05-28T06:14:52.923754+0000","last_became_peered":"2024-05-28T06:14:52.923754+0000","last_unstale":"2024-05-28T06:14:54.502531+0000","last_undegraded":"2024-05-28T06:14:54.502531+0000","last_fullsized":"2024-05-28T06:14:54.502531+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:13:11.797366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.504932+0000","last_change":"2024-05-28T06:14:52.908652+0000","last_active":"2024-05-28T06:14:54.504932+0000","last_peered":"2024-05-28T06:14:54.504932+0000","last_clean":"2024-05-28T06:14:54.504932+0000","last_became_active":"2024-05-28T06:14:52.908356+0000","last_became_peered":"2024-05-28T06:14:52.908356+0000","last_unstale":"2024-05-28T06:14:54.504932+0000","last_undegraded":"2024-05-28T06:14:54.504932+0000","last_fullsized":"2024-05-28T06:14:54.504932+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:31:26.679161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.506170+0000","last_change":"2024-05-28T06:14:52.931264+0000","last_active":"2024-05-28T06:14:54.506170+0000","last_peered":"2024-05-28T06:14:54.506170+0000","last_clean":"2024-05-28T06:14:54.506170+0000","last_became_active":"2024-05-28T06:14:52.930975+0000","last_became_peered":"2024-05-28T06:14:52.930975+0000","last_unstale":"2024-05-28T06:14:54.506170+0000","last_undegraded":"2024-05-28T06:14:54.506170+0000","last_fullsized":"2024-05-28T06:14:54.506170+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:20:40.686529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"73'4","reported_seq":18,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.507643+0000","last_change":"2024-05-28T06:14:51.904397+0000","last_active":"2024-05-28T06:14:54.507643+0000","last_peered":"2024-05-28T06:14:54.507643+0000","last_clean":"2024-05-28T06:14:54.507643+0000","last_became_active":"2024-05-28T06:14:51.904000+0000","last_became_peered":"2024-05-28T06:14:51.904000+0000","last_unstale":"2024-05-28T06:14:54.507643+0000","last_undegraded":"2024-05-28T06:14:54.507643+0000","last_fullsized":"2024-05-28T06:14:54.507643+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_clean_scrub_stamp":"2024-05-28T06:14:50.842466+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:56:41.907859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.281918+0000","last_change":"2024-05-28T06:14:52.924508+0000","last_active":"2024-05-28T06:14:54.281918+0000","last_peered":"2024-05-28T06:14:54.281918+0000","last_clean":"2024-05-28T06:14:54.281918+0000","last_became_active":"2024-05-28T06:14:52.923912+0000","last_became_peered":"2024-05-28T06:14:52.923912+0000","last_unstale":"2024-05-28T06:14:54.281918+0000","last_undegraded":"2024-05-28T06:14:54.281918+0000","last_fullsized":"2024-05-28T06:14:54.281918+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:18:02.973309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"73'1","reported_seq":15,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:53.923268+0000","last_change":"2024-05-28T06:14:51.894228+0000","last_active":"2024-05-28T06:14:53.923268+0000","last_peered":"2024-05-28T06:14:53.923268+0000","last_clean":"2024-05-28T06:14:53.923268+0000","last_became_active":"2024-05-28T06:14:51.893899+0000","last_became_peered":"2024-05-28T06:14:51.893899+0000","last_unstale":"2024-05-28T06:14:53.923268+0000","last_undegraded":"2024-05-28T06:14:53.923268+0000","last_fullsized":"2024-05-28T06:14:53.923268+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_clean_scrub_stamp":"2024-05-28T06:14:50.842466+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:31:11.499497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.506239+0000","last_change":"2024-05-28T06:14:52.919710+0000","last_active":"2024-05-28T06:14:54.506239+0000","last_peered":"2024-05-28T06:14:54.506239+0000","last_clean":"2024-05-28T06:14:54.506239+0000","last_became_active":"2024-05-28T06:14:52.919404+0000","last_became_peered":"2024-05-28T06:14:52.919404+0000","last_unstale":"2024-05-28T06:14:54.506239+0000","last_undegraded":"2024-05-28T06:14:54.506239+0000","last_fullsized":"2024-05-28T06:14:54.506239+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:26:14.502049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"73'2","reported_seq":16,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:53.922910+0000","last_change":"2024-05-28T06:14:51.901200+0000","last_active":"2024-05-28T06:14:53.922910+0000","last_peered":"2024-05-28T06:14:53.922910+0000","last_clean":"2024-05-28T06:14:53.922910+0000","last_became_active":"2024-05-28T06:14:51.900889+0000","last_became_peered":"2024-05-28T06:14:51.900889+0000","last_unstale":"2024-05-28T06:14:53.922910+0000","last_undegraded":"2024-05-28T06:14:53.922910+0000","last_fullsized":"2024-05-28T06:14:53.922910+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_clean_scrub_stamp":"2024-05-28T06:14:50.842466+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:40:45.816248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.506791+0000","last_change":"2024-05-28T06:14:52.912625+0000","last_active":"2024-05-28T06:14:54.506791+0000","last_peered":"2024-05-28T06:14:54.506791+0000","last_clean":"2024-05-28T06:14:54.506791+0000","last_became_active":"2024-05-28T06:14:52.912112+0000","last_became_peered":"2024-05-28T06:14:52.912112+0000","last_unstale":"2024-05-28T06:14:54.506791+0000","last_undegraded":"2024-05-28T06:14:54.506791+0000","last_fullsized":"2024-05-28T06:14:54.506791+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:15:40.810505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"73'2","reported_seq":16,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.506925+0000","last_change":"2024-05-28T06:14:51.900949+0000","last_active":"2024-05-28T06:14:54.506925+0000","last_peered":"2024-05-28T06:14:54.506925+0000","last_clean":"2024-05-28T06:14:54.506925+0000","last_became_active":"2024-05-28T06:14:51.899523+0000","last_became_peered":"2024-05-28T06:14:51.899523+0000","last_unstale":"2024-05-28T06:14:54.506925+0000","last_undegraded":"2024-05-28T06:14:54.506925+0000","last_fullsized":"2024-05-28T06:14:54.506925+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_clean_scrub_stamp":"2024-05-28T06:14:50.842466+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:06:06.859256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.502409+0000","last_change":"2024-05-28T06:14:52.916698+0000","last_active":"2024-05-28T06:14:54.502409+0000","last_peered":"2024-05-28T06:14:54.502409+0000","last_clean":"2024-05-28T06:14:54.502409+0000","last_became_active":"2024-05-28T06:14:52.915862+0000","last_became_peered":"2024-05-28T06:14:52.915862+0000","last_unstale":"2024-05-28T06:14:54.502409+0000","last_undegraded":"2024-05-28T06:14:54.502409+0000","last_fullsized":"2024-05-28T06:14:54.502409+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:23:44.334150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"73'1","reported_seq":15,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.503594+0000","last_change":"2024-05-28T06:14:51.891383+0000","last_active":"2024-05-28T06:14:54.503594+0000","last_peered":"2024-05-28T06:14:54.503594+0000","last_clean":"2024-05-28T06:14:54.503594+0000","last_became_active":"2024-05-28T06:14:51.889443+0000","last_became_peered":"2024-05-28T06:14:51.889443+0000","last_unstale":"2024-05-28T06:14:54.503594+0000","last_undegraded":"2024-05-28T06:14:54.503594+0000","last_fullsized":"2024-05-28T06:14:54.503594+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_clean_scrub_stamp":"2024-05-28T06:14:50.842466+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:12:33.974924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:53.923062+0000","last_change":"2024-05-28T06:14:52.912750+0000","last_active":"2024-05-28T06:14:53.923062+0000","last_peered":"2024-05-28T06:14:53.923062+0000","last_clean":"2024-05-28T06:14:53.923062+0000","last_became_active":"2024-05-28T06:14:52.911900+0000","last_became_peered":"2024-05-28T06:14:52.911900+0000","last_unstale":"2024-05-28T06:14:53.923062+0000","last_undegraded":"2024-05-28T06:14:53.923062+0000","last_fullsized":"2024-05-28T06:14:53.923062+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:36:20.269827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"73'1","reported_seq":15,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.504965+0000","last_change":"2024-05-28T06:14:51.901582+0000","last_active":"2024-05-28T06:14:54.504965+0000","last_peered":"2024-05-28T06:14:54.504965+0000","last_clean":"2024-05-28T06:14:54.504965+0000","last_became_active":"2024-05-28T06:14:51.900065+0000","last_became_peered":"2024-05-28T06:14:51.900065+0000","last_unstale":"2024-05-28T06:14:54.504965+0000","last_undegraded":"2024-05-28T06:14:54.504965+0000","last_fullsized":"2024-05-28T06:14:54.504965+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_clean_scrub_stamp":"2024-05-28T06:14:50.842466+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:41:01.993515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.280536+0000","last_change":"2024-05-28T06:14:52.928872+0000","last_active":"2024-05-28T06:14:54.280536+0000","last_peered":"2024-05-28T06:14:54.280536+0000","last_clean":"2024-05-28T06:14:54.280536+0000","last_became_active":"2024-05-28T06:14:52.928553+0000","last_became_peered":"2024-05-28T06:14:52.928553+0000","last_unstale":"2024-05-28T06:14:54.280536+0000","last_undegraded":"2024-05-28T06:14:54.280536+0000","last_fullsized":"2024-05-28T06:14:54.280536+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:30:38.279216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":14,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.502020+0000","last_change":"2024-05-28T06:14:51.885041+0000","last_active":"2024-05-28T06:14:54.502020+0000","last_peered":"2024-05-28T06:14:54.502020+0000","last_clean":"2024-05-28T06:14:54.502020+0000","last_became_active":"2024-05-28T06:14:51.884605+0000","last_became_peered":"2024-05-28T06:14:51.884605+0000","last_unstale":"2024-05-28T06:14:54.502020+0000","last_undegraded":"2024-05-28T06:14:54.502020+0000","last_fullsized":"2024-05-28T06:14:54.502020+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_clean_scrub_stamp":"2024-05-28T06:14:50.842466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:14:35.912548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:53.922784+0000","last_change":"2024-05-28T06:14:52.907796+0000","last_active":"2024-05-28T06:14:53.922784+0000","last_peered":"2024-05-28T06:14:53.922784+0000","last_clean":"2024-05-28T06:14:53.922784+0000","last_became_active":"2024-05-28T06:14:52.907428+0000","last_became_peered":"2024-05-28T06:14:52.907428+0000","last_unstale":"2024-05-28T06:14:53.922784+0000","last_undegraded":"2024-05-28T06:14:53.922784+0000","last_fullsized":"2024-05-28T06:14:53.922784+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:27:39.361235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":14,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.280502+0000","last_change":"2024-05-28T06:14:51.893303+0000","last_active":"2024-05-28T06:14:54.280502+0000","last_peered":"2024-05-28T06:14:54.280502+0000","last_clean":"2024-05-28T06:14:54.280502+0000","last_became_active":"2024-05-28T06:14:51.891891+0000","last_became_peered":"2024-05-28T06:14:51.891891+0000","last_unstale":"2024-05-28T06:14:54.280502+0000","last_undegraded":"2024-05-28T06:14:54.280502+0000","last_fullsized":"2024-05-28T06:14:54.280502+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_clean_scrub_stamp":"2024-05-28T06:14:50.842466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:06:21.789194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.504815+0000","last_change":"2024-05-28T06:14:52.912081+0000","last_active":"2024-05-28T06:14:54.504815+0000","last_peered":"2024-05-28T06:14:54.504815+0000","last_clean":"2024-05-28T06:14:54.504815+0000","last_became_active":"2024-05-28T06:14:52.911819+0000","last_became_peered":"2024-05-28T06:14:52.911819+0000","last_unstale":"2024-05-28T06:14:54.504815+0000","last_undegraded":"2024-05-28T06:14:54.504815+0000","last_fullsized":"2024-05-28T06:14:54.504815+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:45:45.895312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":14,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:53.923485+0000","last_change":"2024-05-28T06:14:51.889867+0000","last_active":"2024-05-28T06:14:53.923485+0000","last_peered":"2024-05-28T06:14:53.923485+0000","last_clean":"2024-05-28T06:14:53.923485+0000","last_became_active":"2024-05-28T06:14:51.889433+0000","last_became_peered":"2024-05-28T06:14:51.889433+0000","last_unstale":"2024-05-28T06:14:53.923485+0000","last_undegraded":"2024-05-28T06:14:53.923485+0000","last_fullsized":"2024-05-28T06:14:53.923485+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_clean_scrub_stamp":"2024-05-28T06:14:50.842466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:49:23.479507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.507654+0000","last_change":"2024-05-28T06:14:52.932205+0000","last_active":"2024-05-28T06:14:54.507654+0000","last_peered":"2024-05-28T06:14:54.507654+0000","last_clean":"2024-05-28T06:14:54.507654+0000","last_became_active":"2024-05-28T06:14:52.931899+0000","last_became_peered":"2024-05-28T06:14:52.931899+0000","last_unstale":"2024-05-28T06:14:54.507654+0000","last_undegraded":"2024-05-28T06:14:54.507654+0000","last_fullsized":"2024-05-28T06:14:54.507654+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:40:33.092511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":14,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.506593+0000","last_change":"2024-05-28T06:14:51.907391+0000","last_active":"2024-05-28T06:14:54.506593+0000","last_peered":"2024-05-28T06:14:54.506593+0000","last_clean":"2024-05-28T06:14:54.506593+0000","last_became_active":"2024-05-28T06:14:51.907077+0000","last_became_peered":"2024-05-28T06:14:51.907077+0000","last_unstale":"2024-05-28T06:14:54.506593+0000","last_undegraded":"2024-05-28T06:14:54.506593+0000","last_fullsized":"2024-05-28T06:14:54.506593+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_clean_scrub_stamp":"2024-05-28T06:14:50.842466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:47:24.940433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.506647+0000","last_change":"2024-05-28T06:14:52.925694+0000","last_active":"2024-05-28T06:14:54.506647+0000","last_peered":"2024-05-28T06:14:54.506647+0000","last_clean":"2024-05-28T06:14:54.506647+0000","last_became_active":"2024-05-28T06:14:52.924740+0000","last_became_peered":"2024-05-28T06:14:52.924740+0000","last_unstale":"2024-05-28T06:14:54.506647+0000","last_undegraded":"2024-05-28T06:14:54.506647+0000","last_fullsized":"2024-05-28T06:14:54.506647+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:02:32.395801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"73'3","reported_seq":17,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.503875+0000","last_change":"2024-05-28T06:14:51.903290+0000","last_active":"2024-05-28T06:14:54.503875+0000","last_peered":"2024-05-28T06:14:54.503875+0000","last_clean":"2024-05-28T06:14:54.503875+0000","last_became_active":"2024-05-28T06:14:51.902984+0000","last_became_peered":"2024-05-28T06:14:51.902984+0000","last_unstale":"2024-05-28T06:14:54.503875+0000","last_undegraded":"2024-05-28T06:14:54.503875+0000","last_fullsized":"2024-05-28T06:14:54.503875+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_clean_scrub_stamp":"2024-05-28T06:14:50.842466+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:40:32.076067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.280990+0000","last_change":"2024-05-28T06:14:52.925388+0000","last_active":"2024-05-28T06:14:54.280990+0000","last_peered":"2024-05-28T06:14:54.280990+0000","last_clean":"2024-05-28T06:14:54.280990+0000","last_became_active":"2024-05-28T06:14:52.925085+0000","last_became_peered":"2024-05-28T06:14:52.925085+0000","last_unstale":"2024-05-28T06:14:54.280990+0000","last_undegraded":"2024-05-28T06:14:54.280990+0000","last_fullsized":"2024-05-28T06:14:54.280990+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:24:46.490884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"73'6","reported_seq":20,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.502931+0000","last_change":"2024-05-28T06:14:51.894255+0000","last_active":"2024-05-28T06:14:54.502931+0000","last_peered":"2024-05-28T06:14:54.502931+0000","last_clean":"2024-05-28T06:14:54.502931+0000","last_became_active":"2024-05-28T06:14:51.893984+0000","last_became_peered":"2024-05-28T06:14:51.893984+0000","last_unstale":"2024-05-28T06:14:54.502931+0000","last_undegraded":"2024-05-28T06:14:54.502931+0000","last_fullsized":"2024-05-28T06:14:54.502931+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_clean_scrub_stamp":"2024-05-28T06:14:50.842466+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:00:05.292305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.505725+0000","last_change":"2024-05-28T06:14:52.926768+0000","last_active":"2024-05-28T06:14:54.505725+0000","last_peered":"2024-05-28T06:14:54.505725+0000","last_clean":"2024-05-28T06:14:54.505725+0000","last_became_active":"2024-05-28T06:14:52.925949+0000","last_became_peered":"2024-05-28T06:14:52.925949+0000","last_unstale":"2024-05-28T06:14:54.505725+0000","last_undegraded":"2024-05-28T06:14:54.505725+0000","last_fullsized":"2024-05-28T06:14:54.505725+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:45:45.895312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"73'2","reported_seq":16,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.502266+0000","last_change":"2024-05-28T06:14:51.900750+0000","last_active":"2024-05-28T06:14:54.502266+0000","last_peered":"2024-05-28T06:14:54.502266+0000","last_clean":"2024-05-28T06:14:54.502266+0000","last_became_active":"2024-05-28T06:14:51.900369+0000","last_became_peered":"2024-05-28T06:14:51.900369+0000","last_unstale":"2024-05-28T06:14:54.502266+0000","last_undegraded":"2024-05-28T06:14:54.502266+0000","last_fullsized":"2024-05-28T06:14:54.502266+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_clean_scrub_stamp":"2024-05-28T06:14:50.842466+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:29:53.333483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.502767+0000","last_change":"2024-05-28T06:14:52.910820+0000","last_active":"2024-05-28T06:14:54.502767+0000","last_peered":"2024-05-28T06:14:54.502767+0000","last_clean":"2024-05-28T06:14:54.502767+0000","last_became_active":"2024-05-28T06:14:52.910545+0000","last_became_peered":"2024-05-28T06:14:52.910545+0000","last_unstale":"2024-05-28T06:14:54.502767+0000","last_undegraded":"2024-05-28T06:14:54.502767+0000","last_fullsized":"2024-05-28T06:14:54.502767+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:14:25.238419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":14,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.504346+0000","last_change":"2024-05-28T06:14:51.897682+0000","last_active":"2024-05-28T06:14:54.504346+0000","last_peered":"2024-05-28T06:14:54.504346+0000","last_clean":"2024-05-28T06:14:54.504346+0000","last_became_active":"2024-05-28T06:14:51.897342+0000","last_became_peered":"2024-05-28T06:14:51.897342+0000","last_unstale":"2024-05-28T06:14:54.504346+0000","last_undegraded":"2024-05-28T06:14:54.504346+0000","last_fullsized":"2024-05-28T06:14:54.504346+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_clean_scrub_stamp":"2024-05-28T06:14:50.842466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:42:09.377259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:53.922118+0000","last_change":"2024-05-28T06:14:52.914661+0000","last_active":"2024-05-28T06:14:53.922118+0000","last_peered":"2024-05-28T06:14:53.922118+0000","last_clean":"2024-05-28T06:14:53.922118+0000","last_became_active":"2024-05-28T06:14:52.913423+0000","last_became_peered":"2024-05-28T06:14:52.913423+0000","last_unstale":"2024-05-28T06:14:53.922118+0000","last_undegraded":"2024-05-28T06:14:53.922118+0000","last_fullsized":"2024-05-28T06:14:53.922118+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:14:25.238419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":14,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.502084+0000","last_change":"2024-05-28T06:14:51.905513+0000","last_active":"2024-05-28T06:14:54.502084+0000","last_peered":"2024-05-28T06:14:54.502084+0000","last_clean":"2024-05-28T06:14:54.502084+0000","last_became_active":"2024-05-28T06:14:51.903870+0000","last_became_peered":"2024-05-28T06:14:51.903870+0000","last_unstale":"2024-05-28T06:14:54.502084+0000","last_undegraded":"2024-05-28T06:14:54.502084+0000","last_fullsized":"2024-05-28T06:14:54.502084+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_clean_scrub_stamp":"2024-05-28T06:14:50.842466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:39:33.103204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.280976+0000","last_change":"2024-05-28T06:14:52.921340+0000","last_active":"2024-05-28T06:14:54.280976+0000","last_peered":"2024-05-28T06:14:54.280976+0000","last_clean":"2024-05-28T06:14:54.280976+0000","last_became_active":"2024-05-28T06:14:52.921037+0000","last_became_peered":"2024-05-28T06:14:52.921037+0000","last_unstale":"2024-05-28T06:14:54.280976+0000","last_undegraded":"2024-05-28T06:14:54.280976+0000","last_fullsized":"2024-05-28T06:14:54.280976+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:04:35.878273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"73'8","reported_seq":22,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:53.923206+0000","last_change":"2024-05-28T06:14:51.903798+0000","last_active":"2024-05-28T06:14:53.923206+0000","last_peered":"2024-05-28T06:14:53.923206+0000","last_clean":"2024-05-28T06:14:53.923206+0000","last_became_active":"2024-05-28T06:14:51.903488+0000","last_became_peered":"2024-05-28T06:14:51.903488+0000","last_unstale":"2024-05-28T06:14:53.923206+0000","last_undegraded":"2024-05-28T06:14:53.923206+0000","last_fullsized":"2024-05-28T06:14:53.923206+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_clean_scrub_stamp":"2024-05-28T06:14:50.842466+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:42:09.377259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:53.923529+0000","last_change":"2024-05-28T06:14:52.915418+0000","last_active":"2024-05-28T06:14:53.923529+0000","last_peered":"2024-05-28T06:14:53.923529+0000","last_clean":"2024-05-28T06:14:53.923529+0000","last_became_active":"2024-05-28T06:14:52.913957+0000","last_became_peered":"2024-05-28T06:14:52.913957+0000","last_unstale":"2024-05-28T06:14:53.923529+0000","last_undegraded":"2024-05-28T06:14:53.923529+0000","last_fullsized":"2024-05-28T06:14:53.923529+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:39:34.119648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"73'1","reported_seq":15,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.506797+0000","last_change":"2024-05-28T06:14:51.903039+0000","last_active":"2024-05-28T06:14:54.506797+0000","last_peered":"2024-05-28T06:14:54.506797+0000","last_clean":"2024-05-28T06:14:54.506797+0000","last_became_active":"2024-05-28T06:14:51.902758+0000","last_became_peered":"2024-05-28T06:14:51.902758+0000","last_unstale":"2024-05-28T06:14:54.506797+0000","last_undegraded":"2024-05-28T06:14:54.506797+0000","last_fullsized":"2024-05-28T06:14:54.506797+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_clean_scrub_stamp":"2024-05-28T06:14:50.842466+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:05:54.761261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.507495+0000","last_change":"2024-05-28T06:14:52.918767+0000","last_active":"2024-05-28T06:14:54.507495+0000","last_peered":"2024-05-28T06:14:54.507495+0000","last_clean":"2024-05-28T06:14:54.507495+0000","last_became_active":"2024-05-28T06:14:52.918475+0000","last_became_peered":"2024-05-28T06:14:52.918475+0000","last_unstale":"2024-05-28T06:14:54.507495+0000","last_undegraded":"2024-05-28T06:14:54.507495+0000","last_fullsized":"2024-05-28T06:14:54.507495+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:12:34.991368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"73'2","reported_seq":16,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:53.923790+0000","last_change":"2024-05-28T06:14:51.897986+0000","last_active":"2024-05-28T06:14:53.923790+0000","last_peered":"2024-05-28T06:14:53.923790+0000","last_clean":"2024-05-28T06:14:53.923790+0000","last_became_active":"2024-05-28T06:14:51.897677+0000","last_became_peered":"2024-05-28T06:14:51.897677+0000","last_unstale":"2024-05-28T06:14:53.923790+0000","last_undegraded":"2024-05-28T06:14:53.923790+0000","last_fullsized":"2024-05-28T06:14:53.923790+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_clean_scrub_stamp":"2024-05-28T06:14:50.842466+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:05:31.746756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.502792+0000","last_change":"2024-05-28T06:14:52.918019+0000","last_active":"2024-05-28T06:14:54.502792+0000","last_peered":"2024-05-28T06:14:54.502792+0000","last_clean":"2024-05-28T06:14:54.502792+0000","last_became_active":"2024-05-28T06:14:52.915735+0000","last_became_peered":"2024-05-28T06:14:52.915735+0000","last_unstale":"2024-05-28T06:14:54.502792+0000","last_undegraded":"2024-05-28T06:14:54.502792+0000","last_fullsized":"2024-05-28T06:14:54.502792+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:05:32.763199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"73'1","reported_seq":15,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.506335+0000","last_change":"2024-05-28T06:14:51.902716+0000","last_active":"2024-05-28T06:14:54.506335+0000","last_peered":"2024-05-28T06:14:54.506335+0000","last_clean":"2024-05-28T06:14:54.506335+0000","last_became_active":"2024-05-28T06:14:51.902399+0000","last_became_peered":"2024-05-28T06:14:51.902399+0000","last_unstale":"2024-05-28T06:14:54.506335+0000","last_undegraded":"2024-05-28T06:14:54.506335+0000","last_fullsized":"2024-05-28T06:14:54.506335+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_clean_scrub_stamp":"2024-05-28T06:14:50.842466+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:25:25.569812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.281056+0000","last_change":"2024-05-28T06:14:52.925228+0000","last_active":"2024-05-28T06:14:54.281056+0000","last_peered":"2024-05-28T06:14:54.281056+0000","last_clean":"2024-05-28T06:14:54.281056+0000","last_became_active":"2024-05-28T06:14:52.924973+0000","last_became_peered":"2024-05-28T06:14:52.924973+0000","last_unstale":"2024-05-28T06:14:54.281056+0000","last_undegraded":"2024-05-28T06:14:54.281056+0000","last_fullsized":"2024-05-28T06:14:54.281056+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:36:45.488233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":14,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.506207+0000","last_change":"2024-05-28T06:14:51.905870+0000","last_active":"2024-05-28T06:14:54.506207+0000","last_peered":"2024-05-28T06:14:54.506207+0000","last_clean":"2024-05-28T06:14:54.506207+0000","last_became_active":"2024-05-28T06:14:51.905252+0000","last_became_peered":"2024-05-28T06:14:51.905252+0000","last_unstale":"2024-05-28T06:14:54.506207+0000","last_undegraded":"2024-05-28T06:14:54.506207+0000","last_fullsized":"2024-05-28T06:14:54.506207+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_clean_scrub_stamp":"2024-05-28T06:14:50.842466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:54:35.594201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.503477+0000","last_change":"2024-05-28T06:14:52.912854+0000","last_active":"2024-05-28T06:14:54.503477+0000","last_peered":"2024-05-28T06:14:54.503477+0000","last_clean":"2024-05-28T06:14:54.503477+0000","last_became_active":"2024-05-28T06:14:52.912554+0000","last_became_peered":"2024-05-28T06:14:52.912554+0000","last_unstale":"2024-05-28T06:14:54.503477+0000","last_undegraded":"2024-05-28T06:14:54.503477+0000","last_fullsized":"2024-05-28T06:14:54.503477+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:48:59.878515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":14,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.505894+0000","last_change":"2024-05-28T06:14:51.896433+0000","last_active":"2024-05-28T06:14:54.505894+0000","last_peered":"2024-05-28T06:14:54.505894+0000","last_clean":"2024-05-28T06:14:54.505894+0000","last_became_active":"2024-05-28T06:14:51.896100+0000","last_became_peered":"2024-05-28T06:14:51.896100+0000","last_unstale":"2024-05-28T06:14:54.505894+0000","last_undegraded":"2024-05-28T06:14:54.505894+0000","last_fullsized":"2024-05-28T06:14:54.505894+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_clean_scrub_stamp":"2024-05-28T06:14:50.842466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:40:45.816248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.502212+0000","last_change":"2024-05-28T06:14:52.922916+0000","last_active":"2024-05-28T06:14:54.502212+0000","last_peered":"2024-05-28T06:14:54.502212+0000","last_clean":"2024-05-28T06:14:54.502212+0000","last_became_active":"2024-05-28T06:14:52.920042+0000","last_became_peered":"2024-05-28T06:14:52.920042+0000","last_unstale":"2024-05-28T06:14:54.502212+0000","last_undegraded":"2024-05-28T06:14:54.502212+0000","last_fullsized":"2024-05-28T06:14:54.502212+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:15:52.796353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":14,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.502449+0000","last_change":"2024-05-28T06:14:51.902304+0000","last_active":"2024-05-28T06:14:54.502449+0000","last_peered":"2024-05-28T06:14:54.502449+0000","last_clean":"2024-05-28T06:14:54.502449+0000","last_became_active":"2024-05-28T06:14:51.902020+0000","last_became_peered":"2024-05-28T06:14:51.902020+0000","last_unstale":"2024-05-28T06:14:54.502449+0000","last_undegraded":"2024-05-28T06:14:54.502449+0000","last_fullsized":"2024-05-28T06:14:54.502449+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_clean_scrub_stamp":"2024-05-28T06:14:50.842466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:02:31.379357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.502445+0000","last_change":"2024-05-28T06:14:52.921867+0000","last_active":"2024-05-28T06:14:54.502445+0000","last_peered":"2024-05-28T06:14:54.502445+0000","last_clean":"2024-05-28T06:14:54.502445+0000","last_became_active":"2024-05-28T06:14:52.921434+0000","last_became_peered":"2024-05-28T06:14:52.921434+0000","last_unstale":"2024-05-28T06:14:54.502445+0000","last_undegraded":"2024-05-28T06:14:54.502445+0000","last_fullsized":"2024-05-28T06:14:54.502445+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:07:41.626704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":14,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:53.921845+0000","last_change":"2024-05-28T06:14:51.890900+0000","last_active":"2024-05-28T06:14:53.921845+0000","last_peered":"2024-05-28T06:14:53.921845+0000","last_clean":"2024-05-28T06:14:53.921845+0000","last_became_active":"2024-05-28T06:14:51.889743+0000","last_became_peered":"2024-05-28T06:14:51.889743+0000","last_unstale":"2024-05-28T06:14:53.921845+0000","last_undegraded":"2024-05-28T06:14:53.921845+0000","last_fullsized":"2024-05-28T06:14:53.921845+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_clean_scrub_stamp":"2024-05-28T06:14:50.842466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:06:28.247863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:53.923135+0000","last_change":"2024-05-28T06:14:52.916770+0000","last_active":"2024-05-28T06:14:53.923135+0000","last_peered":"2024-05-28T06:14:53.923135+0000","last_clean":"2024-05-28T06:14:53.923135+0000","last_became_active":"2024-05-28T06:14:52.916365+0000","last_became_peered":"2024-05-28T06:14:52.916365+0000","last_unstale":"2024-05-28T06:14:53.923135+0000","last_undegraded":"2024-05-28T06:14:53.923135+0000","last_fullsized":"2024-05-28T06:14:53.923135+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:34:06.413585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"73'2","reported_seq":16,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.507061+0000","last_change":"2024-05-28T06:14:51.902154+0000","last_active":"2024-05-28T06:14:54.507061+0000","last_peered":"2024-05-28T06:14:54.507061+0000","last_clean":"2024-05-28T06:14:54.507061+0000","last_became_active":"2024-05-28T06:14:51.901831+0000","last_became_peered":"2024-05-28T06:14:51.901831+0000","last_unstale":"2024-05-28T06:14:54.507061+0000","last_undegraded":"2024-05-28T06:14:54.507061+0000","last_fullsized":"2024-05-28T06:14:54.507061+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_clean_scrub_stamp":"2024-05-28T06:14:50.842466+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:49:23.479507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.504883+0000","last_change":"2024-05-28T06:14:52.919984+0000","last_active":"2024-05-28T06:14:54.504883+0000","last_peered":"2024-05-28T06:14:54.504883+0000","last_clean":"2024-05-28T06:14:54.504883+0000","last_became_active":"2024-05-28T06:14:52.919711+0000","last_became_peered":"2024-05-28T06:14:52.919711+0000","last_unstale":"2024-05-28T06:14:54.504883+0000","last_undegraded":"2024-05-28T06:14:54.504883+0000","last_fullsized":"2024-05-28T06:14:54.504883+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:27:39.361235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":14,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.504462+0000","last_change":"2024-05-28T06:14:51.901965+0000","last_active":"2024-05-28T06:14:54.504462+0000","last_peered":"2024-05-28T06:14:54.504462+0000","last_clean":"2024-05-28T06:14:54.504462+0000","last_became_active":"2024-05-28T06:14:51.901701+0000","last_became_peered":"2024-05-28T06:14:51.901701+0000","last_unstale":"2024-05-28T06:14:54.504462+0000","last_undegraded":"2024-05-28T06:14:54.504462+0000","last_fullsized":"2024-05-28T06:14:54.504462+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_clean_scrub_stamp":"2024-05-28T06:14:50.842466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:56:56.837777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.506139+0000","last_change":"2024-05-28T06:14:52.932564+0000","last_active":"2024-05-28T06:14:54.506139+0000","last_peered":"2024-05-28T06:14:54.506139+0000","last_clean":"2024-05-28T06:14:54.506139+0000","last_became_active":"2024-05-28T06:14:52.932291+0000","last_became_peered":"2024-05-28T06:14:52.932291+0000","last_unstale":"2024-05-28T06:14:54.506139+0000","last_undegraded":"2024-05-28T06:14:54.506139+0000","last_fullsized":"2024-05-28T06:14:54.506139+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:27:39.361235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":14,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.281243+0000","last_change":"2024-05-28T06:14:51.893949+0000","last_active":"2024-05-28T06:14:54.281243+0000","last_peered":"2024-05-28T06:14:54.281243+0000","last_clean":"2024-05-28T06:14:54.281243+0000","last_became_active":"2024-05-28T06:14:51.893684+0000","last_became_peered":"2024-05-28T06:14:51.893684+0000","last_unstale":"2024-05-28T06:14:54.281243+0000","last_undegraded":"2024-05-28T06:14:54.281243+0000","last_fullsized":"2024-05-28T06:14:54.281243+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_clean_scrub_stamp":"2024-05-28T06:14:50.842466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:29:09.339854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.507078+0000","last_change":"2024-05-28T06:14:52.930173+0000","last_active":"2024-05-28T06:14:54.507078+0000","last_peered":"2024-05-28T06:14:54.507078+0000","last_clean":"2024-05-28T06:14:54.507078+0000","last_became_active":"2024-05-28T06:14:52.929860+0000","last_became_peered":"2024-05-28T06:14:52.929860+0000","last_unstale":"2024-05-28T06:14:54.507078+0000","last_undegraded":"2024-05-28T06:14:54.507078+0000","last_fullsized":"2024-05-28T06:14:54.507078+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:32:04.113202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":14,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:53.922730+0000","last_change":"2024-05-28T06:14:51.887557+0000","last_active":"2024-05-28T06:14:53.922730+0000","last_peered":"2024-05-28T06:14:53.922730+0000","last_clean":"2024-05-28T06:14:53.922730+0000","last_became_active":"2024-05-28T06:14:51.885691+0000","last_became_peered":"2024-05-28T06:14:51.885691+0000","last_unstale":"2024-05-28T06:14:53.922730+0000","last_undegraded":"2024-05-28T06:14:53.922730+0000","last_fullsized":"2024-05-28T06:14:53.922730+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_clean_scrub_stamp":"2024-05-28T06:14:50.842466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:47:57.924628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.281660+0000","last_change":"2024-05-28T06:14:52.909498+0000","last_active":"2024-05-28T06:14:54.281660+0000","last_peered":"2024-05-28T06:14:54.281660+0000","last_clean":"2024-05-28T06:14:54.281660+0000","last_became_active":"2024-05-28T06:14:52.909168+0000","last_became_peered":"2024-05-28T06:14:52.909168+0000","last_unstale":"2024-05-28T06:14:54.281660+0000","last_undegraded":"2024-05-28T06:14:54.281660+0000","last_fullsized":"2024-05-28T06:14:54.281660+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:24:48.237423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":14,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.281046+0000","last_change":"2024-05-28T06:14:51.905913+0000","last_active":"2024-05-28T06:14:54.281046+0000","last_peered":"2024-05-28T06:14:54.281046+0000","last_clean":"2024-05-28T06:14:54.281046+0000","last_became_active":"2024-05-28T06:14:51.905593+0000","last_became_peered":"2024-05-28T06:14:51.905593+0000","last_unstale":"2024-05-28T06:14:54.281046+0000","last_undegraded":"2024-05-28T06:14:54.281046+0000","last_fullsized":"2024-05-28T06:14:54.281046+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_clean_scrub_stamp":"2024-05-28T06:14:50.842466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:11:41.005132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.504601+0000","last_change":"2024-05-28T06:14:52.922033+0000","last_active":"2024-05-28T06:14:54.504601+0000","last_peered":"2024-05-28T06:14:54.504601+0000","last_clean":"2024-05-28T06:14:54.504601+0000","last_became_active":"2024-05-28T06:14:52.917580+0000","last_became_peered":"2024-05-28T06:14:52.917580+0000","last_unstale":"2024-05-28T06:14:54.504601+0000","last_undegraded":"2024-05-28T06:14:54.504601+0000","last_fullsized":"2024-05-28T06:14:54.504601+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:23:03.387649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":14,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.502804+0000","last_change":"2024-05-28T06:14:51.885960+0000","last_active":"2024-05-28T06:14:54.502804+0000","last_peered":"2024-05-28T06:14:54.502804+0000","last_clean":"2024-05-28T06:14:54.502804+0000","last_became_active":"2024-05-28T06:14:51.884537+0000","last_became_peered":"2024-05-28T06:14:51.884537+0000","last_unstale":"2024-05-28T06:14:54.502804+0000","last_undegraded":"2024-05-28T06:14:54.502804+0000","last_fullsized":"2024-05-28T06:14:54.502804+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_clean_scrub_stamp":"2024-05-28T06:14:50.842466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:53:28.270897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.501951+0000","last_change":"2024-05-28T06:14:52.906062+0000","last_active":"2024-05-28T06:14:54.501951+0000","last_peered":"2024-05-28T06:14:54.501951+0000","last_clean":"2024-05-28T06:14:54.501951+0000","last_became_active":"2024-05-28T06:14:52.905684+0000","last_became_peered":"2024-05-28T06:14:52.905684+0000","last_unstale":"2024-05-28T06:14:54.501951+0000","last_undegraded":"2024-05-28T06:14:54.501951+0000","last_fullsized":"2024-05-28T06:14:54.501951+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:05:55.777704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":14,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:53.922482+0000","last_change":"2024-05-28T06:14:51.905287+0000","last_active":"2024-05-28T06:14:53.922482+0000","last_peered":"2024-05-28T06:14:53.922482+0000","last_clean":"2024-05-28T06:14:53.922482+0000","last_became_active":"2024-05-28T06:14:51.904985+0000","last_became_peered":"2024-05-28T06:14:51.904985+0000","last_unstale":"2024-05-28T06:14:53.922482+0000","last_undegraded":"2024-05-28T06:14:53.922482+0000","last_fullsized":"2024-05-28T06:14:53.922482+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_clean_scrub_stamp":"2024-05-28T06:14:50.842466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:05:54.761261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.503819+0000","last_change":"2024-05-28T06:14:52.920925+0000","last_active":"2024-05-28T06:14:54.503819+0000","last_peered":"2024-05-28T06:14:54.503819+0000","last_clean":"2024-05-28T06:14:54.503819+0000","last_became_active":"2024-05-28T06:14:52.917712+0000","last_became_peered":"2024-05-28T06:14:52.917712+0000","last_unstale":"2024-05-28T06:14:54.503819+0000","last_undegraded":"2024-05-28T06:14:54.503819+0000","last_fullsized":"2024-05-28T06:14:54.503819+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:13:07.009136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":14,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.504408+0000","last_change":"2024-05-28T06:14:51.900623+0000","last_active":"2024-05-28T06:14:54.504408+0000","last_peered":"2024-05-28T06:14:54.504408+0000","last_clean":"2024-05-28T06:14:54.504408+0000","last_became_active":"2024-05-28T06:14:51.900167+0000","last_became_peered":"2024-05-28T06:14:51.900167+0000","last_unstale":"2024-05-28T06:14:54.504408+0000","last_undegraded":"2024-05-28T06:14:54.504408+0000","last_fullsized":"2024-05-28T06:14:54.504408+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_clean_scrub_stamp":"2024-05-28T06:14:50.842466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:29:31.589118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.503640+0000","last_change":"2024-05-28T06:14:52.922154+0000","last_active":"2024-05-28T06:14:54.503640+0000","last_peered":"2024-05-28T06:14:54.503640+0000","last_clean":"2024-05-28T06:14:54.503640+0000","last_became_active":"2024-05-28T06:14:52.921876+0000","last_became_peered":"2024-05-28T06:14:52.921876+0000","last_unstale":"2024-05-28T06:14:54.503640+0000","last_undegraded":"2024-05-28T06:14:54.503640+0000","last_fullsized":"2024-05-28T06:14:54.503640+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:24:02.283756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"73'2","reported_seq":16,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.504461+0000","last_change":"2024-05-28T06:14:51.900953+0000","last_active":"2024-05-28T06:14:54.504461+0000","last_peered":"2024-05-28T06:14:54.504461+0000","last_clean":"2024-05-28T06:14:54.504461+0000","last_became_active":"2024-05-28T06:14:51.900072+0000","last_became_peered":"2024-05-28T06:14:51.900072+0000","last_unstale":"2024-05-28T06:14:54.504461+0000","last_undegraded":"2024-05-28T06:14:54.504461+0000","last_fullsized":"2024-05-28T06:14:54.504461+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_clean_scrub_stamp":"2024-05-28T06:14:50.842466+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:19:39.084377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":274,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.505218+0000","last_change":"2024-05-28T06:04:01.039129+0000","last_active":"2024-05-28T06:14:54.505218+0000","last_peered":"2024-05-28T06:14:54.505218+0000","last_clean":"2024-05-28T06:14:54.505218+0000","last_became_active":"2024-05-28T06:04:01.033906+0000","last_became_peered":"2024-05-28T06:04:01.033906+0000","last_unstale":"2024-05-28T06:14:54.505218+0000","last_undegraded":"2024-05-28T06:14:54.505218+0000","last_fullsized":"2024-05-28T06:14:54.505218+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:03:59.864237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:03:59.864237+0000","last_clean_scrub_stamp":"2024-05-28T06:03:59.864237+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:17:36.440245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574982,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54844,"kb_used_data":1456,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316996,"statfs":{"total":96636764160,"available":96580603904,"internally_reserved":0,"allocated":1490944,"data_stored":970023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":54655434},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574984,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54280,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317560,"statfs":{"total":96636764160,"available":96581181440,"internally_reserved":0,"allocated":909312,"data_stored":385459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574982,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54264,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317576,"statfs":{"total":96636764160,"available":96581197824,"internally_reserved":0,"allocated":892928,"data_stored":379129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574982,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58436,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313404,"statfs":{"total":96636764160,"available":96576925696,"internally_reserved":0,"allocated":905216,"data_stored":385437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574984,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58424,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313416,"statfs":{"total":96636764160,"available":96576937984,"internally_reserved":0,"allocated":897024,"data_stored":379151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58915271},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574983,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50128,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321712,"statfs":{"total":96636764160,"available":96585433088,"internally_reserved":0,"allocated":913408,"data_stored":386979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574983,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58444,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313396,"statfs":{"total":96636764160,"available":96576917504,"internally_reserved":0,"allocated":909312,"data_stored":381267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574982,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58996,"kb_used_data":1448,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312844,"statfs":{"total":96636764160,"available":96576352256,"internally_reserved":0,"allocated":1482752,"data_stored":969407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-28T06:14:57.331 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-28T06:14:57.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.573+0000 7f9cc5000640 1 Processor -- start 2024-05-28T06:14:57.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.573+0000 7f9cc5000640 1 -- start start 2024-05-28T06:14:57.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.575+0000 7f9cc5000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f9cc0158b80 0x7f9cc006dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:57.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.575+0000 7f9cc5000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9cc006e160 0x7f9cc0071bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:57.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.575+0000 7f9cc5000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f9cc0072100 0x7f9cc01621f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:57.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.575+0000 7f9cc5000640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f9cc006e6e0 con 0x7f9cc0072100 2024-05-28T06:14:57.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.575+0000 7f9cc5000640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f9cc006e9c0 con 0x7f9cc0158b80 2024-05-28T06:14:57.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.575+0000 7f9cc5000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f9cc006eca0 con 0x7f9cc006e160 2024-05-28T06:14:57.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.575+0000 7f9cbfe00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f9cc0072100 0x7f9cc01621f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:57.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.575+0000 7f9cbfe00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f9cc0072100 0x7f9cc01621f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54658/0 (socket says 172.21.15.96:54658) 2024-05-28T06:14:57.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.575+0000 7f9cbfe00640 1 -- 172.21.15.96:0/2041234968 learned_addr learned my addr 172.21.15.96:0/2041234968 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:57.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.575+0000 7f9cb7e00640 1 --2- 172.21.15.96:0/2041234968 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9cc006e160 0x7f9cc0071bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:57.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.576+0000 7f9cbf400640 1 --2- 172.21.15.96:0/2041234968 >> v2:172.21.15.123:3300/0 conn(0x7f9cc0158b80 0x7f9cc006dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:57.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.576+0000 7f9cbfe00640 1 -- 172.21.15.96:0/2041234968 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9cc006e160 msgr2=0x7f9cc0071bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:57.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.576+0000 7f9cbfe00640 1 --2- 172.21.15.96:0/2041234968 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9cc006e160 0x7f9cc0071bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:57.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.576+0000 7f9cbfe00640 1 -- 172.21.15.96:0/2041234968 >> v2:172.21.15.123:3300/0 conn(0x7f9cc0158b80 msgr2=0x7f9cc006dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:57.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.576+0000 7f9cbfe00640 1 --2- 172.21.15.96:0/2041234968 >> v2:172.21.15.123:3300/0 conn(0x7f9cc0158b80 0x7f9cc006dc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:57.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.576+0000 7f9cbfe00640 1 -- 172.21.15.96:0/2041234968 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cc006ef80 con 0x7f9cc0072100 2024-05-28T06:14:57.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.576+0000 7f9cbf400640 1 --2- 172.21.15.96:0/2041234968 >> v2:172.21.15.123:3300/0 conn(0x7f9cc0158b80 0x7f9cc006dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:14:57.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.577+0000 7f9cbfe00640 1 --2- 172.21.15.96:0/2041234968 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9cc0072100 0x7f9cc01621f0 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f9cc006c830 tx=0x7f9cb802a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c3ec221477ca767 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:57.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.577+0000 7f9cbea00640 1 -- 172.21.15.96:0/2041234968 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cb8036070 con 0x7f9cc0072100 2024-05-28T06:14:57.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.577+0000 7f9cbea00640 1 -- 172.21.15.96:0/2041234968 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9cb8031600 con 0x7f9cc0072100 2024-05-28T06:14:57.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.577+0000 7f9cbea00640 1 -- 172.21.15.96:0/2041234968 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cb80319d0 con 0x7f9cc0072100 2024-05-28T06:14:57.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.578+0000 7f9cc5000640 1 -- 172.21.15.96:0/2041234968 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9cc0072100 msgr2=0x7f9cc01621f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:57.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.578+0000 7f9cc5000640 1 --2- 172.21.15.96:0/2041234968 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9cc0072100 0x7f9cc01621f0 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f9cc006c830 tx=0x7f9cb802a6a0 comp rx=0 tx=0).stop 2024-05-28T06:14:57.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.578+0000 7f9cc5000640 1 -- 172.21.15.96:0/2041234968 shutdown_connections 2024-05-28T06:14:57.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.578+0000 7f9cc5000640 1 --2- 172.21.15.96:0/2041234968 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9cc0072100 0x7f9cc01621f0 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:57.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.578+0000 7f9cc5000640 1 --2- 172.21.15.96:0/2041234968 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9cc006e160 0x7f9cc0071bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:57.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.578+0000 7f9cc5000640 1 --2- 172.21.15.96:0/2041234968 >> v2:172.21.15.123:3300/0 conn(0x7f9cc0158b80 0x7f9cc006dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:57.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.578+0000 7f9cc5000640 1 -- 172.21.15.96:0/2041234968 >> 172.21.15.96:0/2041234968 conn(0x7f9cc00b1fc0 msgr2=0x7f9cc00b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:57.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.578+0000 7f9cc5000640 1 -- 172.21.15.96:0/2041234968 shutdown_connections 2024-05-28T06:14:57.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.578+0000 7f9cc5000640 1 -- 172.21.15.96:0/2041234968 wait complete. 2024-05-28T06:14:57.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.580+0000 7f9cc5000640 1 Processor -- start 2024-05-28T06:14:57.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.580+0000 7f9cc5000640 1 -- start start 2024-05-28T06:14:57.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.581+0000 7f9cc5000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9cc006e160 0x7f9cc0141ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:57.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.581+0000 7f9cc5000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9cc0072100 0x7f9cc01535a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:57.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.581+0000 7f9cc5000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9cc0158b80 0x7f9cc0153ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:57.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.581+0000 7f9cc5000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f9cc006f780 con 0x7f9cc0072100 2024-05-28T06:14:57.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.581+0000 7f9cc5000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f9cc00b2d30 con 0x7f9cc0158b80 2024-05-28T06:14:57.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.581+0000 7f9cc5000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f9cc01423c0 con 0x7f9cc006e160 2024-05-28T06:14:57.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.581+0000 7f9cb7e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9cc0072100 0x7f9cc01535a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:57.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.581+0000 7f9cb7e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9cc0072100 0x7f9cc01535a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54668/0 (socket says 172.21.15.96:54668) 2024-05-28T06:14:57.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.581+0000 7f9cb7e00640 1 -- 172.21.15.96:0/724407457 learned_addr learned my addr 172.21.15.96:0/724407457 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:57.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.581+0000 7f9cbfe00640 1 --2- 172.21.15.96:0/724407457 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9cc0158b80 0x7f9cc0153ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:57.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.581+0000 7f9cbf400640 1 --2- 172.21.15.96:0/724407457 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9cc006e160 0x7f9cc0141ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:57.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.582+0000 7f9cb7e00640 1 -- 172.21.15.96:0/724407457 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9cc006e160 msgr2=0x7f9cc0141ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:57.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.582+0000 7f9cb7e00640 1 --2- 172.21.15.96:0/724407457 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9cc006e160 0x7f9cc0141ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:57.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.582+0000 7f9cb7e00640 1 -- 172.21.15.96:0/724407457 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9cc0158b80 msgr2=0x7f9cc0153ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:57.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.582+0000 7f9cb7e00640 1 --2- 172.21.15.96:0/724407457 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9cc0158b80 0x7f9cc0153ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:57.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.582+0000 7f9cb7e00640 1 -- 172.21.15.96:0/724407457 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cc01426a0 con 0x7f9cc0072100 2024-05-28T06:14:57.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.582+0000 7f9cbfe00640 1 --2- 172.21.15.96:0/724407457 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9cc0158b80 0x7f9cc0153ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:14:57.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.582+0000 7f9cbf400640 1 --2- 172.21.15.96:0/724407457 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9cc006e160 0x7f9cc0141ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:14:57.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.582+0000 7f9cb7e00640 1 --2- 172.21.15.96:0/724407457 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9cc0072100 0x7f9cc01535a0 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f9ca800d7d0 tx=0x7f9ca800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:57.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.582+0000 7f9cb7400640 1 -- 172.21.15.96:0/724407457 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ca8004020 con 0x7f9cc0072100 2024-05-28T06:14:57.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.582+0000 7f9cb7400640 1 -- 172.21.15.96:0/724407457 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ca80041e0 con 0x7f9cc0072100 2024-05-28T06:14:57.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.582+0000 7f9cb7400640 1 -- 172.21.15.96:0/724407457 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ca80043d0 con 0x7f9cc0072100 2024-05-28T06:14:57.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.582+0000 7f9cc5000640 1 -- 172.21.15.96:0/724407457 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9cc0142980 con 0x7f9cc0072100 2024-05-28T06:14:57.586 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.582+0000 7f9cc5000640 1 -- 172.21.15.96:0/724407457 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9cc0142000 con 0x7f9cc0072100 2024-05-28T06:14:57.586 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.583+0000 7f9cc5000640 1 -- 172.21.15.96:0/724407457 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9cc00689e0 con 0x7f9cc0072100 2024-05-28T06:14:57.586 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.584+0000 7f9cb7400640 1 -- 172.21.15.96:0/724407457 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 22) ==== 92213+0+0 (secure 0 0 0) 0x7f9ca801b020 con 0x7f9cc0072100 2024-05-28T06:14:57.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.585+0000 7f9cb7400640 1 --2- 172.21.15.96:0/724407457 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9c9c06d5e0 0x7f9c9c06faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:57.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.585+0000 7f9cb7400640 1 -- 172.21.15.96:0/724407457 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 5737+0+0 (secure 0 0 0) 0x7f9ca808a720 con 0x7f9cc0072100 2024-05-28T06:14:57.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.586+0000 7f9cbf400640 1 --2- 172.21.15.96:0/724407457 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9c9c06d5e0 0x7f9c9c06faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:57.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.588+0000 7f9cbf400640 1 --2- 172.21.15.96:0/724407457 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9c9c06d5e0 0x7f9c9c06faa0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f9cb0006fd0 tx=0x7f9cb0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:57.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.590+0000 7f9cb7400640 1 -- 172.21.15.96:0/724407457 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9ca805a310 con 0x7f9cc0072100 2024-05-28T06:14:57.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.788+0000 7f9cc5000640 1 -- 172.21.15.96:0/724407457 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9cc0152c90 con 0x7f9c9c06d5e0 2024-05-28T06:14:57.810 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.808+0000 7f9cb7400640 1 -- 172.21.15.96:0/724407457 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252066 (secure 0 0 0) 0x7f9cc0152c90 con 0x7f9c9c06d5e0 2024-05-28T06:14:57.812 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:14:57.815 INFO:teuthology.orchestra.run.smithi096.stderr:dumped all 2024-05-28T06:14:57.816 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.814+0000 7f9cc5000640 1 -- 172.21.15.96:0/724407457 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9c9c06d5e0 msgr2=0x7f9c9c06faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:57.816 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.814+0000 7f9cc5000640 1 --2- 172.21.15.96:0/724407457 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9c9c06d5e0 0x7f9c9c06faa0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f9cb0006fd0 tx=0x7f9cb0008040 comp rx=0 tx=0).stop 2024-05-28T06:14:57.816 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.814+0000 7f9cc5000640 1 -- 172.21.15.96:0/724407457 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9cc0072100 msgr2=0x7f9cc01535a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:57.817 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.814+0000 7f9cc5000640 1 --2- 172.21.15.96:0/724407457 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9cc0072100 0x7f9cc01535a0 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f9ca800d7d0 tx=0x7f9ca800dc90 comp rx=0 tx=0).stop 2024-05-28T06:14:57.817 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.815+0000 7f9cc5000640 1 -- 172.21.15.96:0/724407457 shutdown_connections 2024-05-28T06:14:57.817 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.815+0000 7f9cc5000640 1 --2- 172.21.15.96:0/724407457 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9c9c06d5e0 0x7f9c9c06faa0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:57.817 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.815+0000 7f9cc5000640 1 --2- 172.21.15.96:0/724407457 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9cc0158b80 0x7f9cc0153ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:57.817 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.815+0000 7f9cc5000640 1 --2- 172.21.15.96:0/724407457 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9cc0072100 0x7f9cc01535a0 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:57.817 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.815+0000 7f9cc5000640 1 --2- 172.21.15.96:0/724407457 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9cc006e160 0x7f9cc0141ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:57.817 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.815+0000 7f9cc5000640 1 -- 172.21.15.96:0/724407457 >> 172.21.15.96:0/724407457 conn(0x7f9cc00b1fc0 msgr2=0x7f9cc0157cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:57.817 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.815+0000 7f9cc5000640 1 -- 172.21.15.96:0/724407457 shutdown_connections 2024-05-28T06:14:57.817 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:57.815+0000 7f9cc5000640 1 -- 172.21.15.96:0/724407457 wait complete. 2024-05-28T06:14:57.836 INFO:teuthology.orchestra.run.smithi096.stdout:{"pg_ready":true,"pg_map":{"version":396,"stamp":"2024-05-28T06:14:57.749570+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":134,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":447816,"kb_used_data":8204,"kb_used_omap":12,"kb_used_meta":439475,"kb_avail":754526904,"statfs":{"total":773094113280,"available":772635549696,"internally_reserved":0,"allocated":8400896,"data_stored":4236852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":450022960},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002387"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.507269+0000","last_change":"2024-05-28T06:14:52.925298+0000","last_active":"2024-05-28T06:14:54.507269+0000","last_peered":"2024-05-28T06:14:54.507269+0000","last_clean":"2024-05-28T06:14:54.507269+0000","last_became_active":"2024-05-28T06:14:52.925016+0000","last_became_peered":"2024-05-28T06:14:52.925016+0000","last_unstale":"2024-05-28T06:14:54.507269+0000","last_undegraded":"2024-05-28T06:14:54.507269+0000","last_fullsized":"2024-05-28T06:14:54.507269+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:32:21.563791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.507046+0000","last_change":"2024-05-28T06:14:52.927523+0000","last_active":"2024-05-28T06:14:54.507046+0000","last_peered":"2024-05-28T06:14:54.507046+0000","last_clean":"2024-05-28T06:14:54.507046+0000","last_became_active":"2024-05-28T06:14:52.927242+0000","last_became_peered":"2024-05-28T06:14:52.927242+0000","last_unstale":"2024-05-28T06:14:54.507046+0000","last_undegraded":"2024-05-28T06:14:54.507046+0000","last_fullsized":"2024-05-28T06:14:54.507046+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:39:34.119648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.503639+0000","last_change":"2024-05-28T06:14:52.912996+0000","last_active":"2024-05-28T06:14:54.503639+0000","last_peered":"2024-05-28T06:14:54.503639+0000","last_clean":"2024-05-28T06:14:54.503639+0000","last_became_active":"2024-05-28T06:14:52.912661+0000","last_became_peered":"2024-05-28T06:14:52.912661+0000","last_unstale":"2024-05-28T06:14:54.503639+0000","last_undegraded":"2024-05-28T06:14:54.503639+0000","last_fullsized":"2024-05-28T06:14:54.503639+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:47:56.376539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.502406+0000","last_change":"2024-05-28T06:14:52.930703+0000","last_active":"2024-05-28T06:14:54.502406+0000","last_peered":"2024-05-28T06:14:54.502406+0000","last_clean":"2024-05-28T06:14:54.502406+0000","last_became_active":"2024-05-28T06:14:52.929719+0000","last_became_peered":"2024-05-28T06:14:52.929719+0000","last_unstale":"2024-05-28T06:14:54.502406+0000","last_undegraded":"2024-05-28T06:14:54.502406+0000","last_fullsized":"2024-05-28T06:14:54.502406+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:40:46.832691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.280800+0000","last_change":"2024-05-28T06:14:52.913669+0000","last_active":"2024-05-28T06:14:54.280800+0000","last_peered":"2024-05-28T06:14:54.280800+0000","last_clean":"2024-05-28T06:14:54.280800+0000","last_became_active":"2024-05-28T06:14:52.913377+0000","last_became_peered":"2024-05-28T06:14:52.913377+0000","last_unstale":"2024-05-28T06:14:54.280800+0000","last_undegraded":"2024-05-28T06:14:54.280800+0000","last_fullsized":"2024-05-28T06:14:54.280800+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:49:39.185365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.504046+0000","last_change":"2024-05-28T06:14:52.919901+0000","last_active":"2024-05-28T06:14:54.504046+0000","last_peered":"2024-05-28T06:14:54.504046+0000","last_clean":"2024-05-28T06:14:54.504046+0000","last_became_active":"2024-05-28T06:14:52.919587+0000","last_became_peered":"2024-05-28T06:14:52.919587+0000","last_unstale":"2024-05-28T06:14:54.504046+0000","last_undegraded":"2024-05-28T06:14:54.504046+0000","last_fullsized":"2024-05-28T06:14:54.504046+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:55:22.737075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.502043+0000","last_change":"2024-05-28T06:14:52.907310+0000","last_active":"2024-05-28T06:14:54.502043+0000","last_peered":"2024-05-28T06:14:54.502043+0000","last_clean":"2024-05-28T06:14:54.502043+0000","last_became_active":"2024-05-28T06:14:52.906416+0000","last_became_peered":"2024-05-28T06:14:52.906416+0000","last_unstale":"2024-05-28T06:14:54.502043+0000","last_undegraded":"2024-05-28T06:14:54.502043+0000","last_fullsized":"2024-05-28T06:14:54.502043+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:45:53.262592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.503823+0000","last_change":"2024-05-28T06:14:52.909378+0000","last_active":"2024-05-28T06:14:54.503823+0000","last_peered":"2024-05-28T06:14:54.503823+0000","last_clean":"2024-05-28T06:14:54.503823+0000","last_became_active":"2024-05-28T06:14:52.908971+0000","last_became_peered":"2024-05-28T06:14:52.908971+0000","last_unstale":"2024-05-28T06:14:54.503823+0000","last_undegraded":"2024-05-28T06:14:54.503823+0000","last_fullsized":"2024-05-28T06:14:54.503823+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:24:46.937524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.281669+0000","last_change":"2024-05-28T06:14:52.924723+0000","last_active":"2024-05-28T06:14:54.281669+0000","last_peered":"2024-05-28T06:14:54.281669+0000","last_clean":"2024-05-28T06:14:54.281669+0000","last_became_active":"2024-05-28T06:14:52.924254+0000","last_became_peered":"2024-05-28T06:14:52.924254+0000","last_unstale":"2024-05-28T06:14:54.281669+0000","last_undegraded":"2024-05-28T06:14:54.281669+0000","last_fullsized":"2024-05-28T06:14:54.281669+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:21:31.600547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:53.923666+0000","last_change":"2024-05-28T06:14:52.920854+0000","last_active":"2024-05-28T06:14:53.923666+0000","last_peered":"2024-05-28T06:14:53.923666+0000","last_clean":"2024-05-28T06:14:53.923666+0000","last_became_active":"2024-05-28T06:14:52.920575+0000","last_became_peered":"2024-05-28T06:14:52.920575+0000","last_unstale":"2024-05-28T06:14:53.923666+0000","last_undegraded":"2024-05-28T06:14:53.923666+0000","last_fullsized":"2024-05-28T06:14:53.923666+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:56:57.854220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.506694+0000","last_change":"2024-05-28T06:14:52.927237+0000","last_active":"2024-05-28T06:14:54.506694+0000","last_peered":"2024-05-28T06:14:54.506694+0000","last_clean":"2024-05-28T06:14:54.506694+0000","last_became_active":"2024-05-28T06:14:52.926247+0000","last_became_peered":"2024-05-28T06:14:52.926247+0000","last_unstale":"2024-05-28T06:14:54.506694+0000","last_undegraded":"2024-05-28T06:14:54.506694+0000","last_fullsized":"2024-05-28T06:14:54.506694+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:34:06.413585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.502397+0000","last_change":"2024-05-28T06:14:52.921867+0000","last_active":"2024-05-28T06:14:54.502397+0000","last_peered":"2024-05-28T06:14:54.502397+0000","last_clean":"2024-05-28T06:14:54.502397+0000","last_became_active":"2024-05-28T06:14:52.921562+0000","last_became_peered":"2024-05-28T06:14:52.921562+0000","last_unstale":"2024-05-28T06:14:54.502397+0000","last_undegraded":"2024-05-28T06:14:54.502397+0000","last_fullsized":"2024-05-28T06:14:54.502397+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:42:10.393703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:53.922396+0000","last_change":"2024-05-28T06:14:52.911800+0000","last_active":"2024-05-28T06:14:53.922396+0000","last_peered":"2024-05-28T06:14:53.922396+0000","last_clean":"2024-05-28T06:14:53.922396+0000","last_became_active":"2024-05-28T06:14:52.911474+0000","last_became_peered":"2024-05-28T06:14:52.911474+0000","last_unstale":"2024-05-28T06:14:53.922396+0000","last_undegraded":"2024-05-28T06:14:53.922396+0000","last_fullsized":"2024-05-28T06:14:53.922396+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:59:01.279260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.281175+0000","last_change":"2024-05-28T06:14:52.921543+0000","last_active":"2024-05-28T06:14:54.281175+0000","last_peered":"2024-05-28T06:14:54.281175+0000","last_clean":"2024-05-28T06:14:54.281175+0000","last_became_active":"2024-05-28T06:14:52.921253+0000","last_became_peered":"2024-05-28T06:14:52.921253+0000","last_unstale":"2024-05-28T06:14:54.281175+0000","last_undegraded":"2024-05-28T06:14:54.281175+0000","last_fullsized":"2024-05-28T06:14:54.281175+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:43:59.310423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:53.922429+0000","last_change":"2024-05-28T06:14:52.917090+0000","last_active":"2024-05-28T06:14:53.922429+0000","last_peered":"2024-05-28T06:14:53.922429+0000","last_clean":"2024-05-28T06:14:53.922429+0000","last_became_active":"2024-05-28T06:14:52.916797+0000","last_became_peered":"2024-05-28T06:14:52.916797+0000","last_unstale":"2024-05-28T06:14:53.922429+0000","last_undegraded":"2024-05-28T06:14:53.922429+0000","last_fullsized":"2024-05-28T06:14:53.922429+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:45:45.895312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:53.922683+0000","last_change":"2024-05-28T06:14:52.912534+0000","last_active":"2024-05-28T06:14:53.922683+0000","last_peered":"2024-05-28T06:14:53.922683+0000","last_clean":"2024-05-28T06:14:53.922683+0000","last_became_active":"2024-05-28T06:14:52.912278+0000","last_became_peered":"2024-05-28T06:14:52.912278+0000","last_unstale":"2024-05-28T06:14:53.922683+0000","last_undegraded":"2024-05-28T06:14:53.922683+0000","last_fullsized":"2024-05-28T06:14:53.922683+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:23:03.387649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.281690+0000","last_change":"2024-05-28T06:14:52.922992+0000","last_active":"2024-05-28T06:14:54.281690+0000","last_peered":"2024-05-28T06:14:54.281690+0000","last_clean":"2024-05-28T06:14:54.281690+0000","last_became_active":"2024-05-28T06:14:52.922734+0000","last_became_peered":"2024-05-28T06:14:52.922734+0000","last_unstale":"2024-05-28T06:14:54.281690+0000","last_undegraded":"2024-05-28T06:14:54.281690+0000","last_fullsized":"2024-05-28T06:14:54.281690+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:55:49.391662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.503051+0000","last_change":"2024-05-28T06:14:52.922615+0000","last_active":"2024-05-28T06:14:54.503051+0000","last_peered":"2024-05-28T06:14:54.503051+0000","last_clean":"2024-05-28T06:14:54.503051+0000","last_became_active":"2024-05-28T06:14:52.922248+0000","last_became_peered":"2024-05-28T06:14:52.922248+0000","last_unstale":"2024-05-28T06:14:54.503051+0000","last_undegraded":"2024-05-28T06:14:54.503051+0000","last_fullsized":"2024-05-28T06:14:54.503051+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:49:24.495950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.505066+0000","last_change":"2024-05-28T06:14:52.910258+0000","last_active":"2024-05-28T06:14:54.505066+0000","last_peered":"2024-05-28T06:14:54.505066+0000","last_clean":"2024-05-28T06:14:54.505066+0000","last_became_active":"2024-05-28T06:14:52.909966+0000","last_became_peered":"2024-05-28T06:14:52.909966+0000","last_unstale":"2024-05-28T06:14:54.505066+0000","last_undegraded":"2024-05-28T06:14:54.505066+0000","last_fullsized":"2024-05-28T06:14:54.505066+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:36:20.269827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:53.922966+0000","last_change":"2024-05-28T06:14:52.926424+0000","last_active":"2024-05-28T06:14:53.922966+0000","last_peered":"2024-05-28T06:14:53.922966+0000","last_clean":"2024-05-28T06:14:53.922966+0000","last_became_active":"2024-05-28T06:14:52.926032+0000","last_became_peered":"2024-05-28T06:14:52.926032+0000","last_unstale":"2024-05-28T06:14:53.922966+0000","last_undegraded":"2024-05-28T06:14:53.922966+0000","last_fullsized":"2024-05-28T06:14:53.922966+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:20:40.686529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:53.922766+0000","last_change":"2024-05-28T06:14:52.911212+0000","last_active":"2024-05-28T06:14:53.922766+0000","last_peered":"2024-05-28T06:14:53.922766+0000","last_clean":"2024-05-28T06:14:53.922766+0000","last_became_active":"2024-05-28T06:14:52.908274+0000","last_became_peered":"2024-05-28T06:14:52.908274+0000","last_unstale":"2024-05-28T06:14:53.922766+0000","last_undegraded":"2024-05-28T06:14:53.922766+0000","last_fullsized":"2024-05-28T06:14:53.922766+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:31:26.679161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:53.922595+0000","last_change":"2024-05-28T06:14:52.921526+0000","last_active":"2024-05-28T06:14:53.922595+0000","last_peered":"2024-05-28T06:14:53.922595+0000","last_clean":"2024-05-28T06:14:53.922595+0000","last_became_active":"2024-05-28T06:14:52.921229+0000","last_became_peered":"2024-05-28T06:14:52.921229+0000","last_unstale":"2024-05-28T06:14:53.922595+0000","last_undegraded":"2024-05-28T06:14:53.922595+0000","last_fullsized":"2024-05-28T06:14:53.922595+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:55:22.737075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.507456+0000","last_change":"2024-05-28T06:14:52.924412+0000","last_active":"2024-05-28T06:14:54.507456+0000","last_peered":"2024-05-28T06:14:54.507456+0000","last_clean":"2024-05-28T06:14:54.507456+0000","last_became_active":"2024-05-28T06:14:52.924154+0000","last_became_peered":"2024-05-28T06:14:52.924154+0000","last_unstale":"2024-05-28T06:14:54.507456+0000","last_undegraded":"2024-05-28T06:14:54.507456+0000","last_fullsized":"2024-05-28T06:14:54.507456+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:25:35.889139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.503278+0000","last_change":"2024-05-28T06:14:52.918515+0000","last_active":"2024-05-28T06:14:54.503278+0000","last_peered":"2024-05-28T06:14:54.503278+0000","last_clean":"2024-05-28T06:14:54.503278+0000","last_became_active":"2024-05-28T06:14:52.918217+0000","last_became_peered":"2024-05-28T06:14:52.918217+0000","last_unstale":"2024-05-28T06:14:54.503278+0000","last_undegraded":"2024-05-28T06:14:54.503278+0000","last_fullsized":"2024-05-28T06:14:54.503278+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:59:01.279260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:53.923517+0000","last_change":"2024-05-28T06:14:52.927009+0000","last_active":"2024-05-28T06:14:53.923517+0000","last_peered":"2024-05-28T06:14:53.923517+0000","last_clean":"2024-05-28T06:14:53.923517+0000","last_became_active":"2024-05-28T06:14:52.926731+0000","last_became_peered":"2024-05-28T06:14:52.926731+0000","last_unstale":"2024-05-28T06:14:53.923517+0000","last_undegraded":"2024-05-28T06:14:53.923517+0000","last_fullsized":"2024-05-28T06:14:53.923517+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:02:32.395801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.503512+0000","last_change":"2024-05-28T06:14:52.921482+0000","last_active":"2024-05-28T06:14:54.503512+0000","last_peered":"2024-05-28T06:14:54.503512+0000","last_clean":"2024-05-28T06:14:54.503512+0000","last_became_active":"2024-05-28T06:14:52.921225+0000","last_became_peered":"2024-05-28T06:14:52.921225+0000","last_unstale":"2024-05-28T06:14:54.503512+0000","last_undegraded":"2024-05-28T06:14:54.503512+0000","last_fullsized":"2024-05-28T06:14:54.503512+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:59:01.279260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.503928+0000","last_change":"2024-05-28T06:14:52.913616+0000","last_active":"2024-05-28T06:14:54.503928+0000","last_peered":"2024-05-28T06:14:54.503928+0000","last_clean":"2024-05-28T06:14:54.503928+0000","last_became_active":"2024-05-28T06:14:52.913343+0000","last_became_peered":"2024-05-28T06:14:52.913343+0000","last_unstale":"2024-05-28T06:14:54.503928+0000","last_undegraded":"2024-05-28T06:14:54.503928+0000","last_fullsized":"2024-05-28T06:14:54.503928+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:32:41.688023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:53.923200+0000","last_change":"2024-05-28T06:14:52.922737+0000","last_active":"2024-05-28T06:14:53.923200+0000","last_peered":"2024-05-28T06:14:53.923200+0000","last_clean":"2024-05-28T06:14:53.923200+0000","last_became_active":"2024-05-28T06:14:52.922456+0000","last_became_peered":"2024-05-28T06:14:52.922456+0000","last_unstale":"2024-05-28T06:14:53.923200+0000","last_undegraded":"2024-05-28T06:14:53.923200+0000","last_fullsized":"2024-05-28T06:14:53.923200+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:26:14.502049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.503317+0000","last_change":"2024-05-28T06:14:52.927138+0000","last_active":"2024-05-28T06:14:54.503317+0000","last_peered":"2024-05-28T06:14:54.503317+0000","last_clean":"2024-05-28T06:14:54.503317+0000","last_became_active":"2024-05-28T06:14:52.926740+0000","last_became_peered":"2024-05-28T06:14:52.926740+0000","last_unstale":"2024-05-28T06:14:54.503317+0000","last_undegraded":"2024-05-28T06:14:54.503317+0000","last_fullsized":"2024-05-28T06:14:54.503317+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:30:07.820114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.502414+0000","last_change":"2024-05-28T06:14:52.917517+0000","last_active":"2024-05-28T06:14:54.502414+0000","last_peered":"2024-05-28T06:14:54.502414+0000","last_clean":"2024-05-28T06:14:54.502414+0000","last_became_active":"2024-05-28T06:14:52.916217+0000","last_became_peered":"2024-05-28T06:14:52.916217+0000","last_unstale":"2024-05-28T06:14:54.502414+0000","last_undegraded":"2024-05-28T06:14:54.502414+0000","last_fullsized":"2024-05-28T06:14:54.502414+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:24:28.642199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.502531+0000","last_change":"2024-05-28T06:14:52.925986+0000","last_active":"2024-05-28T06:14:54.502531+0000","last_peered":"2024-05-28T06:14:54.502531+0000","last_clean":"2024-05-28T06:14:54.502531+0000","last_became_active":"2024-05-28T06:14:52.923754+0000","last_became_peered":"2024-05-28T06:14:52.923754+0000","last_unstale":"2024-05-28T06:14:54.502531+0000","last_undegraded":"2024-05-28T06:14:54.502531+0000","last_fullsized":"2024-05-28T06:14:54.502531+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:13:11.797366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.504932+0000","last_change":"2024-05-28T06:14:52.908652+0000","last_active":"2024-05-28T06:14:54.504932+0000","last_peered":"2024-05-28T06:14:54.504932+0000","last_clean":"2024-05-28T06:14:54.504932+0000","last_became_active":"2024-05-28T06:14:52.908356+0000","last_became_peered":"2024-05-28T06:14:52.908356+0000","last_unstale":"2024-05-28T06:14:54.504932+0000","last_undegraded":"2024-05-28T06:14:54.504932+0000","last_fullsized":"2024-05-28T06:14:54.504932+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:31:26.679161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.506170+0000","last_change":"2024-05-28T06:14:52.931264+0000","last_active":"2024-05-28T06:14:54.506170+0000","last_peered":"2024-05-28T06:14:54.506170+0000","last_clean":"2024-05-28T06:14:54.506170+0000","last_became_active":"2024-05-28T06:14:52.930975+0000","last_became_peered":"2024-05-28T06:14:52.930975+0000","last_unstale":"2024-05-28T06:14:54.506170+0000","last_undegraded":"2024-05-28T06:14:54.506170+0000","last_fullsized":"2024-05-28T06:14:54.506170+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:20:40.686529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"73'4","reported_seq":18,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.507643+0000","last_change":"2024-05-28T06:14:51.904397+0000","last_active":"2024-05-28T06:14:54.507643+0000","last_peered":"2024-05-28T06:14:54.507643+0000","last_clean":"2024-05-28T06:14:54.507643+0000","last_became_active":"2024-05-28T06:14:51.904000+0000","last_became_peered":"2024-05-28T06:14:51.904000+0000","last_unstale":"2024-05-28T06:14:54.507643+0000","last_undegraded":"2024-05-28T06:14:54.507643+0000","last_fullsized":"2024-05-28T06:14:54.507643+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_clean_scrub_stamp":"2024-05-28T06:14:50.842466+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:56:41.907859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.281918+0000","last_change":"2024-05-28T06:14:52.924508+0000","last_active":"2024-05-28T06:14:54.281918+0000","last_peered":"2024-05-28T06:14:54.281918+0000","last_clean":"2024-05-28T06:14:54.281918+0000","last_became_active":"2024-05-28T06:14:52.923912+0000","last_became_peered":"2024-05-28T06:14:52.923912+0000","last_unstale":"2024-05-28T06:14:54.281918+0000","last_undegraded":"2024-05-28T06:14:54.281918+0000","last_fullsized":"2024-05-28T06:14:54.281918+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:18:02.973309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"73'1","reported_seq":15,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:53.923268+0000","last_change":"2024-05-28T06:14:51.894228+0000","last_active":"2024-05-28T06:14:53.923268+0000","last_peered":"2024-05-28T06:14:53.923268+0000","last_clean":"2024-05-28T06:14:53.923268+0000","last_became_active":"2024-05-28T06:14:51.893899+0000","last_became_peered":"2024-05-28T06:14:51.893899+0000","last_unstale":"2024-05-28T06:14:53.923268+0000","last_undegraded":"2024-05-28T06:14:53.923268+0000","last_fullsized":"2024-05-28T06:14:53.923268+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_clean_scrub_stamp":"2024-05-28T06:14:50.842466+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:31:11.499497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.506239+0000","last_change":"2024-05-28T06:14:52.919710+0000","last_active":"2024-05-28T06:14:54.506239+0000","last_peered":"2024-05-28T06:14:54.506239+0000","last_clean":"2024-05-28T06:14:54.506239+0000","last_became_active":"2024-05-28T06:14:52.919404+0000","last_became_peered":"2024-05-28T06:14:52.919404+0000","last_unstale":"2024-05-28T06:14:54.506239+0000","last_undegraded":"2024-05-28T06:14:54.506239+0000","last_fullsized":"2024-05-28T06:14:54.506239+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:26:14.502049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"73'2","reported_seq":16,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:53.922910+0000","last_change":"2024-05-28T06:14:51.901200+0000","last_active":"2024-05-28T06:14:53.922910+0000","last_peered":"2024-05-28T06:14:53.922910+0000","last_clean":"2024-05-28T06:14:53.922910+0000","last_became_active":"2024-05-28T06:14:51.900889+0000","last_became_peered":"2024-05-28T06:14:51.900889+0000","last_unstale":"2024-05-28T06:14:53.922910+0000","last_undegraded":"2024-05-28T06:14:53.922910+0000","last_fullsized":"2024-05-28T06:14:53.922910+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_clean_scrub_stamp":"2024-05-28T06:14:50.842466+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:40:45.816248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.506791+0000","last_change":"2024-05-28T06:14:52.912625+0000","last_active":"2024-05-28T06:14:54.506791+0000","last_peered":"2024-05-28T06:14:54.506791+0000","last_clean":"2024-05-28T06:14:54.506791+0000","last_became_active":"2024-05-28T06:14:52.912112+0000","last_became_peered":"2024-05-28T06:14:52.912112+0000","last_unstale":"2024-05-28T06:14:54.506791+0000","last_undegraded":"2024-05-28T06:14:54.506791+0000","last_fullsized":"2024-05-28T06:14:54.506791+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:15:40.810505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"73'2","reported_seq":16,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.506925+0000","last_change":"2024-05-28T06:14:51.900949+0000","last_active":"2024-05-28T06:14:54.506925+0000","last_peered":"2024-05-28T06:14:54.506925+0000","last_clean":"2024-05-28T06:14:54.506925+0000","last_became_active":"2024-05-28T06:14:51.899523+0000","last_became_peered":"2024-05-28T06:14:51.899523+0000","last_unstale":"2024-05-28T06:14:54.506925+0000","last_undegraded":"2024-05-28T06:14:54.506925+0000","last_fullsized":"2024-05-28T06:14:54.506925+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_clean_scrub_stamp":"2024-05-28T06:14:50.842466+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:06:06.859256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.502409+0000","last_change":"2024-05-28T06:14:52.916698+0000","last_active":"2024-05-28T06:14:54.502409+0000","last_peered":"2024-05-28T06:14:54.502409+0000","last_clean":"2024-05-28T06:14:54.502409+0000","last_became_active":"2024-05-28T06:14:52.915862+0000","last_became_peered":"2024-05-28T06:14:52.915862+0000","last_unstale":"2024-05-28T06:14:54.502409+0000","last_undegraded":"2024-05-28T06:14:54.502409+0000","last_fullsized":"2024-05-28T06:14:54.502409+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:23:44.334150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"73'1","reported_seq":15,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.503594+0000","last_change":"2024-05-28T06:14:51.891383+0000","last_active":"2024-05-28T06:14:54.503594+0000","last_peered":"2024-05-28T06:14:54.503594+0000","last_clean":"2024-05-28T06:14:54.503594+0000","last_became_active":"2024-05-28T06:14:51.889443+0000","last_became_peered":"2024-05-28T06:14:51.889443+0000","last_unstale":"2024-05-28T06:14:54.503594+0000","last_undegraded":"2024-05-28T06:14:54.503594+0000","last_fullsized":"2024-05-28T06:14:54.503594+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_clean_scrub_stamp":"2024-05-28T06:14:50.842466+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:12:33.974924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:53.923062+0000","last_change":"2024-05-28T06:14:52.912750+0000","last_active":"2024-05-28T06:14:53.923062+0000","last_peered":"2024-05-28T06:14:53.923062+0000","last_clean":"2024-05-28T06:14:53.923062+0000","last_became_active":"2024-05-28T06:14:52.911900+0000","last_became_peered":"2024-05-28T06:14:52.911900+0000","last_unstale":"2024-05-28T06:14:53.923062+0000","last_undegraded":"2024-05-28T06:14:53.923062+0000","last_fullsized":"2024-05-28T06:14:53.923062+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:36:20.269827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"73'1","reported_seq":15,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.504965+0000","last_change":"2024-05-28T06:14:51.901582+0000","last_active":"2024-05-28T06:14:54.504965+0000","last_peered":"2024-05-28T06:14:54.504965+0000","last_clean":"2024-05-28T06:14:54.504965+0000","last_became_active":"2024-05-28T06:14:51.900065+0000","last_became_peered":"2024-05-28T06:14:51.900065+0000","last_unstale":"2024-05-28T06:14:54.504965+0000","last_undegraded":"2024-05-28T06:14:54.504965+0000","last_fullsized":"2024-05-28T06:14:54.504965+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_clean_scrub_stamp":"2024-05-28T06:14:50.842466+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:41:01.993515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.280536+0000","last_change":"2024-05-28T06:14:52.928872+0000","last_active":"2024-05-28T06:14:54.280536+0000","last_peered":"2024-05-28T06:14:54.280536+0000","last_clean":"2024-05-28T06:14:54.280536+0000","last_became_active":"2024-05-28T06:14:52.928553+0000","last_became_peered":"2024-05-28T06:14:52.928553+0000","last_unstale":"2024-05-28T06:14:54.280536+0000","last_undegraded":"2024-05-28T06:14:54.280536+0000","last_fullsized":"2024-05-28T06:14:54.280536+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:30:38.279216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":14,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.502020+0000","last_change":"2024-05-28T06:14:51.885041+0000","last_active":"2024-05-28T06:14:54.502020+0000","last_peered":"2024-05-28T06:14:54.502020+0000","last_clean":"2024-05-28T06:14:54.502020+0000","last_became_active":"2024-05-28T06:14:51.884605+0000","last_became_peered":"2024-05-28T06:14:51.884605+0000","last_unstale":"2024-05-28T06:14:54.502020+0000","last_undegraded":"2024-05-28T06:14:54.502020+0000","last_fullsized":"2024-05-28T06:14:54.502020+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_clean_scrub_stamp":"2024-05-28T06:14:50.842466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:14:35.912548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:53.922784+0000","last_change":"2024-05-28T06:14:52.907796+0000","last_active":"2024-05-28T06:14:53.922784+0000","last_peered":"2024-05-28T06:14:53.922784+0000","last_clean":"2024-05-28T06:14:53.922784+0000","last_became_active":"2024-05-28T06:14:52.907428+0000","last_became_peered":"2024-05-28T06:14:52.907428+0000","last_unstale":"2024-05-28T06:14:53.922784+0000","last_undegraded":"2024-05-28T06:14:53.922784+0000","last_fullsized":"2024-05-28T06:14:53.922784+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:27:39.361235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":14,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.280502+0000","last_change":"2024-05-28T06:14:51.893303+0000","last_active":"2024-05-28T06:14:54.280502+0000","last_peered":"2024-05-28T06:14:54.280502+0000","last_clean":"2024-05-28T06:14:54.280502+0000","last_became_active":"2024-05-28T06:14:51.891891+0000","last_became_peered":"2024-05-28T06:14:51.891891+0000","last_unstale":"2024-05-28T06:14:54.280502+0000","last_undegraded":"2024-05-28T06:14:54.280502+0000","last_fullsized":"2024-05-28T06:14:54.280502+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_clean_scrub_stamp":"2024-05-28T06:14:50.842466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:06:21.789194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.504815+0000","last_change":"2024-05-28T06:14:52.912081+0000","last_active":"2024-05-28T06:14:54.504815+0000","last_peered":"2024-05-28T06:14:54.504815+0000","last_clean":"2024-05-28T06:14:54.504815+0000","last_became_active":"2024-05-28T06:14:52.911819+0000","last_became_peered":"2024-05-28T06:14:52.911819+0000","last_unstale":"2024-05-28T06:14:54.504815+0000","last_undegraded":"2024-05-28T06:14:54.504815+0000","last_fullsized":"2024-05-28T06:14:54.504815+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:45:45.895312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":14,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:53.923485+0000","last_change":"2024-05-28T06:14:51.889867+0000","last_active":"2024-05-28T06:14:53.923485+0000","last_peered":"2024-05-28T06:14:53.923485+0000","last_clean":"2024-05-28T06:14:53.923485+0000","last_became_active":"2024-05-28T06:14:51.889433+0000","last_became_peered":"2024-05-28T06:14:51.889433+0000","last_unstale":"2024-05-28T06:14:53.923485+0000","last_undegraded":"2024-05-28T06:14:53.923485+0000","last_fullsized":"2024-05-28T06:14:53.923485+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_clean_scrub_stamp":"2024-05-28T06:14:50.842466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:49:23.479507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.507654+0000","last_change":"2024-05-28T06:14:52.932205+0000","last_active":"2024-05-28T06:14:54.507654+0000","last_peered":"2024-05-28T06:14:54.507654+0000","last_clean":"2024-05-28T06:14:54.507654+0000","last_became_active":"2024-05-28T06:14:52.931899+0000","last_became_peered":"2024-05-28T06:14:52.931899+0000","last_unstale":"2024-05-28T06:14:54.507654+0000","last_undegraded":"2024-05-28T06:14:54.507654+0000","last_fullsized":"2024-05-28T06:14:54.507654+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:40:33.092511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":14,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.506593+0000","last_change":"2024-05-28T06:14:51.907391+0000","last_active":"2024-05-28T06:14:54.506593+0000","last_peered":"2024-05-28T06:14:54.506593+0000","last_clean":"2024-05-28T06:14:54.506593+0000","last_became_active":"2024-05-28T06:14:51.907077+0000","last_became_peered":"2024-05-28T06:14:51.907077+0000","last_unstale":"2024-05-28T06:14:54.506593+0000","last_undegraded":"2024-05-28T06:14:54.506593+0000","last_fullsized":"2024-05-28T06:14:54.506593+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_clean_scrub_stamp":"2024-05-28T06:14:50.842466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:47:24.940433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.506647+0000","last_change":"2024-05-28T06:14:52.925694+0000","last_active":"2024-05-28T06:14:54.506647+0000","last_peered":"2024-05-28T06:14:54.506647+0000","last_clean":"2024-05-28T06:14:54.506647+0000","last_became_active":"2024-05-28T06:14:52.924740+0000","last_became_peered":"2024-05-28T06:14:52.924740+0000","last_unstale":"2024-05-28T06:14:54.506647+0000","last_undegraded":"2024-05-28T06:14:54.506647+0000","last_fullsized":"2024-05-28T06:14:54.506647+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:02:32.395801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"73'3","reported_seq":17,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.503875+0000","last_change":"2024-05-28T06:14:51.903290+0000","last_active":"2024-05-28T06:14:54.503875+0000","last_peered":"2024-05-28T06:14:54.503875+0000","last_clean":"2024-05-28T06:14:54.503875+0000","last_became_active":"2024-05-28T06:14:51.902984+0000","last_became_peered":"2024-05-28T06:14:51.902984+0000","last_unstale":"2024-05-28T06:14:54.503875+0000","last_undegraded":"2024-05-28T06:14:54.503875+0000","last_fullsized":"2024-05-28T06:14:54.503875+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_clean_scrub_stamp":"2024-05-28T06:14:50.842466+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:40:32.076067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.280990+0000","last_change":"2024-05-28T06:14:52.925388+0000","last_active":"2024-05-28T06:14:54.280990+0000","last_peered":"2024-05-28T06:14:54.280990+0000","last_clean":"2024-05-28T06:14:54.280990+0000","last_became_active":"2024-05-28T06:14:52.925085+0000","last_became_peered":"2024-05-28T06:14:52.925085+0000","last_unstale":"2024-05-28T06:14:54.280990+0000","last_undegraded":"2024-05-28T06:14:54.280990+0000","last_fullsized":"2024-05-28T06:14:54.280990+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:24:46.490884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"73'6","reported_seq":20,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.502931+0000","last_change":"2024-05-28T06:14:51.894255+0000","last_active":"2024-05-28T06:14:54.502931+0000","last_peered":"2024-05-28T06:14:54.502931+0000","last_clean":"2024-05-28T06:14:54.502931+0000","last_became_active":"2024-05-28T06:14:51.893984+0000","last_became_peered":"2024-05-28T06:14:51.893984+0000","last_unstale":"2024-05-28T06:14:54.502931+0000","last_undegraded":"2024-05-28T06:14:54.502931+0000","last_fullsized":"2024-05-28T06:14:54.502931+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_clean_scrub_stamp":"2024-05-28T06:14:50.842466+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:00:05.292305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.505725+0000","last_change":"2024-05-28T06:14:52.926768+0000","last_active":"2024-05-28T06:14:54.505725+0000","last_peered":"2024-05-28T06:14:54.505725+0000","last_clean":"2024-05-28T06:14:54.505725+0000","last_became_active":"2024-05-28T06:14:52.925949+0000","last_became_peered":"2024-05-28T06:14:52.925949+0000","last_unstale":"2024-05-28T06:14:54.505725+0000","last_undegraded":"2024-05-28T06:14:54.505725+0000","last_fullsized":"2024-05-28T06:14:54.505725+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:45:45.895312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"73'2","reported_seq":16,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.502266+0000","last_change":"2024-05-28T06:14:51.900750+0000","last_active":"2024-05-28T06:14:54.502266+0000","last_peered":"2024-05-28T06:14:54.502266+0000","last_clean":"2024-05-28T06:14:54.502266+0000","last_became_active":"2024-05-28T06:14:51.900369+0000","last_became_peered":"2024-05-28T06:14:51.900369+0000","last_unstale":"2024-05-28T06:14:54.502266+0000","last_undegraded":"2024-05-28T06:14:54.502266+0000","last_fullsized":"2024-05-28T06:14:54.502266+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_clean_scrub_stamp":"2024-05-28T06:14:50.842466+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:29:53.333483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.502767+0000","last_change":"2024-05-28T06:14:52.910820+0000","last_active":"2024-05-28T06:14:54.502767+0000","last_peered":"2024-05-28T06:14:54.502767+0000","last_clean":"2024-05-28T06:14:54.502767+0000","last_became_active":"2024-05-28T06:14:52.910545+0000","last_became_peered":"2024-05-28T06:14:52.910545+0000","last_unstale":"2024-05-28T06:14:54.502767+0000","last_undegraded":"2024-05-28T06:14:54.502767+0000","last_fullsized":"2024-05-28T06:14:54.502767+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:14:25.238419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":14,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.504346+0000","last_change":"2024-05-28T06:14:51.897682+0000","last_active":"2024-05-28T06:14:54.504346+0000","last_peered":"2024-05-28T06:14:54.504346+0000","last_clean":"2024-05-28T06:14:54.504346+0000","last_became_active":"2024-05-28T06:14:51.897342+0000","last_became_peered":"2024-05-28T06:14:51.897342+0000","last_unstale":"2024-05-28T06:14:54.504346+0000","last_undegraded":"2024-05-28T06:14:54.504346+0000","last_fullsized":"2024-05-28T06:14:54.504346+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_clean_scrub_stamp":"2024-05-28T06:14:50.842466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:42:09.377259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:53.922118+0000","last_change":"2024-05-28T06:14:52.914661+0000","last_active":"2024-05-28T06:14:53.922118+0000","last_peered":"2024-05-28T06:14:53.922118+0000","last_clean":"2024-05-28T06:14:53.922118+0000","last_became_active":"2024-05-28T06:14:52.913423+0000","last_became_peered":"2024-05-28T06:14:52.913423+0000","last_unstale":"2024-05-28T06:14:53.922118+0000","last_undegraded":"2024-05-28T06:14:53.922118+0000","last_fullsized":"2024-05-28T06:14:53.922118+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:14:25.238419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":14,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.502084+0000","last_change":"2024-05-28T06:14:51.905513+0000","last_active":"2024-05-28T06:14:54.502084+0000","last_peered":"2024-05-28T06:14:54.502084+0000","last_clean":"2024-05-28T06:14:54.502084+0000","last_became_active":"2024-05-28T06:14:51.903870+0000","last_became_peered":"2024-05-28T06:14:51.903870+0000","last_unstale":"2024-05-28T06:14:54.502084+0000","last_undegraded":"2024-05-28T06:14:54.502084+0000","last_fullsized":"2024-05-28T06:14:54.502084+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_clean_scrub_stamp":"2024-05-28T06:14:50.842466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:39:33.103204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.280976+0000","last_change":"2024-05-28T06:14:52.921340+0000","last_active":"2024-05-28T06:14:54.280976+0000","last_peered":"2024-05-28T06:14:54.280976+0000","last_clean":"2024-05-28T06:14:54.280976+0000","last_became_active":"2024-05-28T06:14:52.921037+0000","last_became_peered":"2024-05-28T06:14:52.921037+0000","last_unstale":"2024-05-28T06:14:54.280976+0000","last_undegraded":"2024-05-28T06:14:54.280976+0000","last_fullsized":"2024-05-28T06:14:54.280976+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:04:35.878273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"73'8","reported_seq":22,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:53.923206+0000","last_change":"2024-05-28T06:14:51.903798+0000","last_active":"2024-05-28T06:14:53.923206+0000","last_peered":"2024-05-28T06:14:53.923206+0000","last_clean":"2024-05-28T06:14:53.923206+0000","last_became_active":"2024-05-28T06:14:51.903488+0000","last_became_peered":"2024-05-28T06:14:51.903488+0000","last_unstale":"2024-05-28T06:14:53.923206+0000","last_undegraded":"2024-05-28T06:14:53.923206+0000","last_fullsized":"2024-05-28T06:14:53.923206+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_clean_scrub_stamp":"2024-05-28T06:14:50.842466+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:42:09.377259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:53.923529+0000","last_change":"2024-05-28T06:14:52.915418+0000","last_active":"2024-05-28T06:14:53.923529+0000","last_peered":"2024-05-28T06:14:53.923529+0000","last_clean":"2024-05-28T06:14:53.923529+0000","last_became_active":"2024-05-28T06:14:52.913957+0000","last_became_peered":"2024-05-28T06:14:52.913957+0000","last_unstale":"2024-05-28T06:14:53.923529+0000","last_undegraded":"2024-05-28T06:14:53.923529+0000","last_fullsized":"2024-05-28T06:14:53.923529+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:39:34.119648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"73'1","reported_seq":15,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.506797+0000","last_change":"2024-05-28T06:14:51.903039+0000","last_active":"2024-05-28T06:14:54.506797+0000","last_peered":"2024-05-28T06:14:54.506797+0000","last_clean":"2024-05-28T06:14:54.506797+0000","last_became_active":"2024-05-28T06:14:51.902758+0000","last_became_peered":"2024-05-28T06:14:51.902758+0000","last_unstale":"2024-05-28T06:14:54.506797+0000","last_undegraded":"2024-05-28T06:14:54.506797+0000","last_fullsized":"2024-05-28T06:14:54.506797+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_clean_scrub_stamp":"2024-05-28T06:14:50.842466+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:05:54.761261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.507495+0000","last_change":"2024-05-28T06:14:52.918767+0000","last_active":"2024-05-28T06:14:54.507495+0000","last_peered":"2024-05-28T06:14:54.507495+0000","last_clean":"2024-05-28T06:14:54.507495+0000","last_became_active":"2024-05-28T06:14:52.918475+0000","last_became_peered":"2024-05-28T06:14:52.918475+0000","last_unstale":"2024-05-28T06:14:54.507495+0000","last_undegraded":"2024-05-28T06:14:54.507495+0000","last_fullsized":"2024-05-28T06:14:54.507495+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:12:34.991368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"73'2","reported_seq":16,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:53.923790+0000","last_change":"2024-05-28T06:14:51.897986+0000","last_active":"2024-05-28T06:14:53.923790+0000","last_peered":"2024-05-28T06:14:53.923790+0000","last_clean":"2024-05-28T06:14:53.923790+0000","last_became_active":"2024-05-28T06:14:51.897677+0000","last_became_peered":"2024-05-28T06:14:51.897677+0000","last_unstale":"2024-05-28T06:14:53.923790+0000","last_undegraded":"2024-05-28T06:14:53.923790+0000","last_fullsized":"2024-05-28T06:14:53.923790+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_clean_scrub_stamp":"2024-05-28T06:14:50.842466+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:05:31.746756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.502792+0000","last_change":"2024-05-28T06:14:52.918019+0000","last_active":"2024-05-28T06:14:54.502792+0000","last_peered":"2024-05-28T06:14:54.502792+0000","last_clean":"2024-05-28T06:14:54.502792+0000","last_became_active":"2024-05-28T06:14:52.915735+0000","last_became_peered":"2024-05-28T06:14:52.915735+0000","last_unstale":"2024-05-28T06:14:54.502792+0000","last_undegraded":"2024-05-28T06:14:54.502792+0000","last_fullsized":"2024-05-28T06:14:54.502792+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:05:32.763199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"73'1","reported_seq":15,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.506335+0000","last_change":"2024-05-28T06:14:51.902716+0000","last_active":"2024-05-28T06:14:54.506335+0000","last_peered":"2024-05-28T06:14:54.506335+0000","last_clean":"2024-05-28T06:14:54.506335+0000","last_became_active":"2024-05-28T06:14:51.902399+0000","last_became_peered":"2024-05-28T06:14:51.902399+0000","last_unstale":"2024-05-28T06:14:54.506335+0000","last_undegraded":"2024-05-28T06:14:54.506335+0000","last_fullsized":"2024-05-28T06:14:54.506335+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_clean_scrub_stamp":"2024-05-28T06:14:50.842466+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:25:25.569812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.281056+0000","last_change":"2024-05-28T06:14:52.925228+0000","last_active":"2024-05-28T06:14:54.281056+0000","last_peered":"2024-05-28T06:14:54.281056+0000","last_clean":"2024-05-28T06:14:54.281056+0000","last_became_active":"2024-05-28T06:14:52.924973+0000","last_became_peered":"2024-05-28T06:14:52.924973+0000","last_unstale":"2024-05-28T06:14:54.281056+0000","last_undegraded":"2024-05-28T06:14:54.281056+0000","last_fullsized":"2024-05-28T06:14:54.281056+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:36:45.488233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":14,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.506207+0000","last_change":"2024-05-28T06:14:51.905870+0000","last_active":"2024-05-28T06:14:54.506207+0000","last_peered":"2024-05-28T06:14:54.506207+0000","last_clean":"2024-05-28T06:14:54.506207+0000","last_became_active":"2024-05-28T06:14:51.905252+0000","last_became_peered":"2024-05-28T06:14:51.905252+0000","last_unstale":"2024-05-28T06:14:54.506207+0000","last_undegraded":"2024-05-28T06:14:54.506207+0000","last_fullsized":"2024-05-28T06:14:54.506207+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_clean_scrub_stamp":"2024-05-28T06:14:50.842466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:54:35.594201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.503477+0000","last_change":"2024-05-28T06:14:52.912854+0000","last_active":"2024-05-28T06:14:54.503477+0000","last_peered":"2024-05-28T06:14:54.503477+0000","last_clean":"2024-05-28T06:14:54.503477+0000","last_became_active":"2024-05-28T06:14:52.912554+0000","last_became_peered":"2024-05-28T06:14:52.912554+0000","last_unstale":"2024-05-28T06:14:54.503477+0000","last_undegraded":"2024-05-28T06:14:54.503477+0000","last_fullsized":"2024-05-28T06:14:54.503477+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:48:59.878515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":14,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.505894+0000","last_change":"2024-05-28T06:14:51.896433+0000","last_active":"2024-05-28T06:14:54.505894+0000","last_peered":"2024-05-28T06:14:54.505894+0000","last_clean":"2024-05-28T06:14:54.505894+0000","last_became_active":"2024-05-28T06:14:51.896100+0000","last_became_peered":"2024-05-28T06:14:51.896100+0000","last_unstale":"2024-05-28T06:14:54.505894+0000","last_undegraded":"2024-05-28T06:14:54.505894+0000","last_fullsized":"2024-05-28T06:14:54.505894+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_clean_scrub_stamp":"2024-05-28T06:14:50.842466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:40:45.816248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.502212+0000","last_change":"2024-05-28T06:14:52.922916+0000","last_active":"2024-05-28T06:14:54.502212+0000","last_peered":"2024-05-28T06:14:54.502212+0000","last_clean":"2024-05-28T06:14:54.502212+0000","last_became_active":"2024-05-28T06:14:52.920042+0000","last_became_peered":"2024-05-28T06:14:52.920042+0000","last_unstale":"2024-05-28T06:14:54.502212+0000","last_undegraded":"2024-05-28T06:14:54.502212+0000","last_fullsized":"2024-05-28T06:14:54.502212+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:15:52.796353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":14,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.502449+0000","last_change":"2024-05-28T06:14:51.902304+0000","last_active":"2024-05-28T06:14:54.502449+0000","last_peered":"2024-05-28T06:14:54.502449+0000","last_clean":"2024-05-28T06:14:54.502449+0000","last_became_active":"2024-05-28T06:14:51.902020+0000","last_became_peered":"2024-05-28T06:14:51.902020+0000","last_unstale":"2024-05-28T06:14:54.502449+0000","last_undegraded":"2024-05-28T06:14:54.502449+0000","last_fullsized":"2024-05-28T06:14:54.502449+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_clean_scrub_stamp":"2024-05-28T06:14:50.842466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:02:31.379357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.502445+0000","last_change":"2024-05-28T06:14:52.921867+0000","last_active":"2024-05-28T06:14:54.502445+0000","last_peered":"2024-05-28T06:14:54.502445+0000","last_clean":"2024-05-28T06:14:54.502445+0000","last_became_active":"2024-05-28T06:14:52.921434+0000","last_became_peered":"2024-05-28T06:14:52.921434+0000","last_unstale":"2024-05-28T06:14:54.502445+0000","last_undegraded":"2024-05-28T06:14:54.502445+0000","last_fullsized":"2024-05-28T06:14:54.502445+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:07:41.626704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":14,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:53.921845+0000","last_change":"2024-05-28T06:14:51.890900+0000","last_active":"2024-05-28T06:14:53.921845+0000","last_peered":"2024-05-28T06:14:53.921845+0000","last_clean":"2024-05-28T06:14:53.921845+0000","last_became_active":"2024-05-28T06:14:51.889743+0000","last_became_peered":"2024-05-28T06:14:51.889743+0000","last_unstale":"2024-05-28T06:14:53.921845+0000","last_undegraded":"2024-05-28T06:14:53.921845+0000","last_fullsized":"2024-05-28T06:14:53.921845+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_clean_scrub_stamp":"2024-05-28T06:14:50.842466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:06:28.247863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:53.923135+0000","last_change":"2024-05-28T06:14:52.916770+0000","last_active":"2024-05-28T06:14:53.923135+0000","last_peered":"2024-05-28T06:14:53.923135+0000","last_clean":"2024-05-28T06:14:53.923135+0000","last_became_active":"2024-05-28T06:14:52.916365+0000","last_became_peered":"2024-05-28T06:14:52.916365+0000","last_unstale":"2024-05-28T06:14:53.923135+0000","last_undegraded":"2024-05-28T06:14:53.923135+0000","last_fullsized":"2024-05-28T06:14:53.923135+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:34:06.413585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"73'2","reported_seq":16,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.507061+0000","last_change":"2024-05-28T06:14:51.902154+0000","last_active":"2024-05-28T06:14:54.507061+0000","last_peered":"2024-05-28T06:14:54.507061+0000","last_clean":"2024-05-28T06:14:54.507061+0000","last_became_active":"2024-05-28T06:14:51.901831+0000","last_became_peered":"2024-05-28T06:14:51.901831+0000","last_unstale":"2024-05-28T06:14:54.507061+0000","last_undegraded":"2024-05-28T06:14:54.507061+0000","last_fullsized":"2024-05-28T06:14:54.507061+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_clean_scrub_stamp":"2024-05-28T06:14:50.842466+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:49:23.479507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.504883+0000","last_change":"2024-05-28T06:14:52.919984+0000","last_active":"2024-05-28T06:14:54.504883+0000","last_peered":"2024-05-28T06:14:54.504883+0000","last_clean":"2024-05-28T06:14:54.504883+0000","last_became_active":"2024-05-28T06:14:52.919711+0000","last_became_peered":"2024-05-28T06:14:52.919711+0000","last_unstale":"2024-05-28T06:14:54.504883+0000","last_undegraded":"2024-05-28T06:14:54.504883+0000","last_fullsized":"2024-05-28T06:14:54.504883+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:27:39.361235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":14,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.504462+0000","last_change":"2024-05-28T06:14:51.901965+0000","last_active":"2024-05-28T06:14:54.504462+0000","last_peered":"2024-05-28T06:14:54.504462+0000","last_clean":"2024-05-28T06:14:54.504462+0000","last_became_active":"2024-05-28T06:14:51.901701+0000","last_became_peered":"2024-05-28T06:14:51.901701+0000","last_unstale":"2024-05-28T06:14:54.504462+0000","last_undegraded":"2024-05-28T06:14:54.504462+0000","last_fullsized":"2024-05-28T06:14:54.504462+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_clean_scrub_stamp":"2024-05-28T06:14:50.842466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:56:56.837777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.506139+0000","last_change":"2024-05-28T06:14:52.932564+0000","last_active":"2024-05-28T06:14:54.506139+0000","last_peered":"2024-05-28T06:14:54.506139+0000","last_clean":"2024-05-28T06:14:54.506139+0000","last_became_active":"2024-05-28T06:14:52.932291+0000","last_became_peered":"2024-05-28T06:14:52.932291+0000","last_unstale":"2024-05-28T06:14:54.506139+0000","last_undegraded":"2024-05-28T06:14:54.506139+0000","last_fullsized":"2024-05-28T06:14:54.506139+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:27:39.361235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":14,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.281243+0000","last_change":"2024-05-28T06:14:51.893949+0000","last_active":"2024-05-28T06:14:54.281243+0000","last_peered":"2024-05-28T06:14:54.281243+0000","last_clean":"2024-05-28T06:14:54.281243+0000","last_became_active":"2024-05-28T06:14:51.893684+0000","last_became_peered":"2024-05-28T06:14:51.893684+0000","last_unstale":"2024-05-28T06:14:54.281243+0000","last_undegraded":"2024-05-28T06:14:54.281243+0000","last_fullsized":"2024-05-28T06:14:54.281243+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_clean_scrub_stamp":"2024-05-28T06:14:50.842466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:29:09.339854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.507078+0000","last_change":"2024-05-28T06:14:52.930173+0000","last_active":"2024-05-28T06:14:54.507078+0000","last_peered":"2024-05-28T06:14:54.507078+0000","last_clean":"2024-05-28T06:14:54.507078+0000","last_became_active":"2024-05-28T06:14:52.929860+0000","last_became_peered":"2024-05-28T06:14:52.929860+0000","last_unstale":"2024-05-28T06:14:54.507078+0000","last_undegraded":"2024-05-28T06:14:54.507078+0000","last_fullsized":"2024-05-28T06:14:54.507078+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:32:04.113202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":14,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:53.922730+0000","last_change":"2024-05-28T06:14:51.887557+0000","last_active":"2024-05-28T06:14:53.922730+0000","last_peered":"2024-05-28T06:14:53.922730+0000","last_clean":"2024-05-28T06:14:53.922730+0000","last_became_active":"2024-05-28T06:14:51.885691+0000","last_became_peered":"2024-05-28T06:14:51.885691+0000","last_unstale":"2024-05-28T06:14:53.922730+0000","last_undegraded":"2024-05-28T06:14:53.922730+0000","last_fullsized":"2024-05-28T06:14:53.922730+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_clean_scrub_stamp":"2024-05-28T06:14:50.842466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:47:57.924628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.281660+0000","last_change":"2024-05-28T06:14:52.909498+0000","last_active":"2024-05-28T06:14:54.281660+0000","last_peered":"2024-05-28T06:14:54.281660+0000","last_clean":"2024-05-28T06:14:54.281660+0000","last_became_active":"2024-05-28T06:14:52.909168+0000","last_became_peered":"2024-05-28T06:14:52.909168+0000","last_unstale":"2024-05-28T06:14:54.281660+0000","last_undegraded":"2024-05-28T06:14:54.281660+0000","last_fullsized":"2024-05-28T06:14:54.281660+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:24:48.237423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":14,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.281046+0000","last_change":"2024-05-28T06:14:51.905913+0000","last_active":"2024-05-28T06:14:54.281046+0000","last_peered":"2024-05-28T06:14:54.281046+0000","last_clean":"2024-05-28T06:14:54.281046+0000","last_became_active":"2024-05-28T06:14:51.905593+0000","last_became_peered":"2024-05-28T06:14:51.905593+0000","last_unstale":"2024-05-28T06:14:54.281046+0000","last_undegraded":"2024-05-28T06:14:54.281046+0000","last_fullsized":"2024-05-28T06:14:54.281046+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_clean_scrub_stamp":"2024-05-28T06:14:50.842466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:11:41.005132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.504601+0000","last_change":"2024-05-28T06:14:52.922033+0000","last_active":"2024-05-28T06:14:54.504601+0000","last_peered":"2024-05-28T06:14:54.504601+0000","last_clean":"2024-05-28T06:14:54.504601+0000","last_became_active":"2024-05-28T06:14:52.917580+0000","last_became_peered":"2024-05-28T06:14:52.917580+0000","last_unstale":"2024-05-28T06:14:54.504601+0000","last_undegraded":"2024-05-28T06:14:54.504601+0000","last_fullsized":"2024-05-28T06:14:54.504601+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:23:03.387649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":14,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.502804+0000","last_change":"2024-05-28T06:14:51.885960+0000","last_active":"2024-05-28T06:14:54.502804+0000","last_peered":"2024-05-28T06:14:54.502804+0000","last_clean":"2024-05-28T06:14:54.502804+0000","last_became_active":"2024-05-28T06:14:51.884537+0000","last_became_peered":"2024-05-28T06:14:51.884537+0000","last_unstale":"2024-05-28T06:14:54.502804+0000","last_undegraded":"2024-05-28T06:14:54.502804+0000","last_fullsized":"2024-05-28T06:14:54.502804+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_clean_scrub_stamp":"2024-05-28T06:14:50.842466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:53:28.270897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.501951+0000","last_change":"2024-05-28T06:14:52.906062+0000","last_active":"2024-05-28T06:14:54.501951+0000","last_peered":"2024-05-28T06:14:54.501951+0000","last_clean":"2024-05-28T06:14:54.501951+0000","last_became_active":"2024-05-28T06:14:52.905684+0000","last_became_peered":"2024-05-28T06:14:52.905684+0000","last_unstale":"2024-05-28T06:14:54.501951+0000","last_undegraded":"2024-05-28T06:14:54.501951+0000","last_fullsized":"2024-05-28T06:14:54.501951+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:05:55.777704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":14,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:53.922482+0000","last_change":"2024-05-28T06:14:51.905287+0000","last_active":"2024-05-28T06:14:53.922482+0000","last_peered":"2024-05-28T06:14:53.922482+0000","last_clean":"2024-05-28T06:14:53.922482+0000","last_became_active":"2024-05-28T06:14:51.904985+0000","last_became_peered":"2024-05-28T06:14:51.904985+0000","last_unstale":"2024-05-28T06:14:53.922482+0000","last_undegraded":"2024-05-28T06:14:53.922482+0000","last_fullsized":"2024-05-28T06:14:53.922482+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_clean_scrub_stamp":"2024-05-28T06:14:50.842466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:05:54.761261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.503819+0000","last_change":"2024-05-28T06:14:52.920925+0000","last_active":"2024-05-28T06:14:54.503819+0000","last_peered":"2024-05-28T06:14:54.503819+0000","last_clean":"2024-05-28T06:14:54.503819+0000","last_became_active":"2024-05-28T06:14:52.917712+0000","last_became_peered":"2024-05-28T06:14:52.917712+0000","last_unstale":"2024-05-28T06:14:54.503819+0000","last_undegraded":"2024-05-28T06:14:54.503819+0000","last_fullsized":"2024-05-28T06:14:54.503819+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:13:07.009136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":14,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.504408+0000","last_change":"2024-05-28T06:14:51.900623+0000","last_active":"2024-05-28T06:14:54.504408+0000","last_peered":"2024-05-28T06:14:54.504408+0000","last_clean":"2024-05-28T06:14:54.504408+0000","last_became_active":"2024-05-28T06:14:51.900167+0000","last_became_peered":"2024-05-28T06:14:51.900167+0000","last_unstale":"2024-05-28T06:14:54.504408+0000","last_undegraded":"2024-05-28T06:14:54.504408+0000","last_fullsized":"2024-05-28T06:14:54.504408+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_clean_scrub_stamp":"2024-05-28T06:14:50.842466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:29:31.589118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":12,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.503640+0000","last_change":"2024-05-28T06:14:52.922154+0000","last_active":"2024-05-28T06:14:54.503640+0000","last_peered":"2024-05-28T06:14:54.503640+0000","last_clean":"2024-05-28T06:14:54.503640+0000","last_became_active":"2024-05-28T06:14:52.921876+0000","last_became_peered":"2024-05-28T06:14:52.921876+0000","last_unstale":"2024-05-28T06:14:54.503640+0000","last_undegraded":"2024-05-28T06:14:54.503640+0000","last_fullsized":"2024-05-28T06:14:54.503640+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:51.858910+0000","last_clean_scrub_stamp":"2024-05-28T06:14:51.858910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:24:02.283756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"73'2","reported_seq":16,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.504461+0000","last_change":"2024-05-28T06:14:51.900953+0000","last_active":"2024-05-28T06:14:54.504461+0000","last_peered":"2024-05-28T06:14:54.504461+0000","last_clean":"2024-05-28T06:14:54.504461+0000","last_became_active":"2024-05-28T06:14:51.900072+0000","last_became_peered":"2024-05-28T06:14:51.900072+0000","last_unstale":"2024-05-28T06:14:54.504461+0000","last_undegraded":"2024-05-28T06:14:54.504461+0000","last_fullsized":"2024-05-28T06:14:54.504461+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:14:50.842466+0000","last_clean_scrub_stamp":"2024-05-28T06:14:50.842466+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:19:39.084377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":274,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-05-28T06:14:54.505218+0000","last_change":"2024-05-28T06:04:01.039129+0000","last_active":"2024-05-28T06:14:54.505218+0000","last_peered":"2024-05-28T06:14:54.505218+0000","last_clean":"2024-05-28T06:14:54.505218+0000","last_became_active":"2024-05-28T06:04:01.033906+0000","last_became_peered":"2024-05-28T06:04:01.033906+0000","last_unstale":"2024-05-28T06:14:54.505218+0000","last_undegraded":"2024-05-28T06:14:54.505218+0000","last_fullsized":"2024-05-28T06:14:54.505218+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:03:59.864237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:03:59.864237+0000","last_clean_scrub_stamp":"2024-05-28T06:03:59.864237+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:17:36.440245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574982,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54844,"kb_used_data":1456,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316996,"statfs":{"total":96636764160,"available":96580603904,"internally_reserved":0,"allocated":1490944,"data_stored":970023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":54655434},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574984,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54280,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317560,"statfs":{"total":96636764160,"available":96581181440,"internally_reserved":0,"allocated":909312,"data_stored":385459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574982,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54264,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317576,"statfs":{"total":96636764160,"available":96581197824,"internally_reserved":0,"allocated":892928,"data_stored":379129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574982,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58436,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313404,"statfs":{"total":96636764160,"available":96576925696,"internally_reserved":0,"allocated":905216,"data_stored":385437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574984,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58424,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313416,"statfs":{"total":96636764160,"available":96576937984,"internally_reserved":0,"allocated":897024,"data_stored":379151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58915271},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574983,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50128,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321712,"statfs":{"total":96636764160,"available":96585433088,"internally_reserved":0,"allocated":913408,"data_stored":386979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574983,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58444,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313396,"statfs":{"total":96636764160,"available":96576917504,"internally_reserved":0,"allocated":909312,"data_stored":381267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574982,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58996,"kb_used_data":1448,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312844,"statfs":{"total":96636764160,"available":96576352256,"internally_reserved":0,"allocated":1482752,"data_stored":969407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-28T06:14:57.840 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-05-28T06:14:57.841 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-05-28T06:14:58.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.089+0000 7f0b7f200640 1 Processor -- start 2024-05-28T06:14:58.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.089+0000 7f0b7f200640 1 -- start start 2024-05-28T06:14:58.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.091+0000 7f0b7f200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f0b801639b0 0x7f0b800ce760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:58.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.091+0000 7f0b7f200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0b800ceee0 0x7f0b800cf350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:58.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.091+0000 7f0b7f200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b801670c0 con 0x7f0b80163d90 2024-05-28T06:14:58.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.091+0000 7f0b7f200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f0b800cf890 con 0x7f0b801639b0 2024-05-28T06:14:58.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.091+0000 7f0b7f200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f0b800ceca0 con 0x7f0b800ceee0 2024-05-28T06:14:58.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.091+0000 7f0b77e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f0b80163d90 0x7f0b80166670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:52576/0 (socket says 172.21.15.96:52576) 2024-05-28T06:14:58.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.091+0000 7f0b77e00640 1 -- 172.21.15.96:0/154943149 learned_addr learned my addr 172.21.15.96:0/154943149 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:14:58.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.092+0000 7f0b7de00640 1 --2- 172.21.15.96:0/154943149 >> v2:172.21.15.123:3300/0 conn(0x7f0b801639b0 0x7f0b800ce760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:58.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.092+0000 7f0b7e800640 1 --2- 172.21.15.96:0/154943149 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0b800ceee0 0x7f0b800cf350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:58.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.092+0000 7f0b7d400640 1 -- 172.21.15.96:0/154943149 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4140288577 0 0) 0x7f0b801670c0 con 0x7f0b80163d90 2024-05-28T06:14:58.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.092+0000 7f0b7d400640 1 -- 172.21.15.96:0/154943149 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0b64003520 con 0x7f0b80163d90 2024-05-28T06:14:58.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.092+0000 7f0b7d400640 1 -- 172.21.15.96:0/154943149 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 169182030 0 0) 0x7f0b64003520 con 0x7f0b80163d90 2024-05-28T06:14:58.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.092+0000 7f0b7d400640 1 -- 172.21.15.96:0/154943149 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0b800ceee0 msgr2=0x7f0b800cf350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:58.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.092+0000 7f0b7d400640 1 --2- 172.21.15.96:0/154943149 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0b800ceee0 0x7f0b800cf350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:58.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.092+0000 7f0b7d400640 1 -- 172.21.15.96:0/154943149 >> v2:172.21.15.123:3300/0 conn(0x7f0b801639b0 msgr2=0x7f0b800ce760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:58.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.092+0000 7f0b7d400640 1 --2- 172.21.15.96:0/154943149 >> v2:172.21.15.123:3300/0 conn(0x7f0b801639b0 0x7f0b800ce760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:58.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.092+0000 7f0b7d400640 1 -- 172.21.15.96:0/154943149 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b801673a0 con 0x7f0b80163d90 2024-05-28T06:14:58.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.092+0000 7f0b7d400640 1 -- 172.21.15.96:0/154943149 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f0b68001670 con 0x7f0b80163d90 2024-05-28T06:14:58.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.092+0000 7f0b7d400640 1 -- 172.21.15.96:0/154943149 >> v1:172.21.15.96:6789/0 conn(0x7f0b80163d90 legacy=0x7f0b80166670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:58.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.092+0000 7f0b7d400640 1 --2- 172.21.15.96:0/154943149 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0b640039e0 0x7f0b64003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:58.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.093+0000 7f0b7d400640 1 --2- 172.21.15.96:0/154943149 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0b800ceee0 0x7f0b64004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:58.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.093+0000 7f0b7d400640 1 --2- 172.21.15.96:0/154943149 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0b64004d40 0x7f0b640051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:58.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.093+0000 7f0b7d400640 1 -- 172.21.15.96:0/154943149 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f0b800cf890 con 0x7f0b640039e0 2024-05-28T06:14:58.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.093+0000 7f0b7d400640 1 -- 172.21.15.96:0/154943149 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f0b800ceca0 con 0x7f0b64004d40 2024-05-28T06:14:58.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.093+0000 7f0b7d400640 1 -- 172.21.15.96:0/154943149 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f0b6400b9f0 con 0x7f0b800ceee0 2024-05-28T06:14:58.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.093+0000 7f0b7de00640 1 --2- 172.21.15.96:0/154943149 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0b640039e0 0x7f0b64003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:58.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.093+0000 7f0b77e00640 1 --2- 172.21.15.96:0/154943149 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0b800ceee0 0x7f0b64004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:58.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.093+0000 7f0b7e800640 1 --2- 172.21.15.96:0/154943149 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0b64004d40 0x7f0b640051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:58.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.093+0000 7f0b7de00640 1 -- 172.21.15.96:0/154943149 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0b800ceee0 msgr2=0x7f0b64004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:58.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.093+0000 7f0b7de00640 1 --2- 172.21.15.96:0/154943149 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0b800ceee0 0x7f0b64004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:58.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.093+0000 7f0b7de00640 1 -- 172.21.15.96:0/154943149 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0b64004d40 msgr2=0x7f0b640051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:58.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.093+0000 7f0b7de00640 1 --2- 172.21.15.96:0/154943149 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0b64004d40 0x7f0b640051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:58.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.093+0000 7f0b7de00640 1 -- 172.21.15.96:0/154943149 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b64003520 con 0x7f0b640039e0 2024-05-28T06:14:58.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.093+0000 7f0b77e00640 1 --2- 172.21.15.96:0/154943149 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0b800ceee0 0x7f0b64004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:14:58.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.094+0000 7f0b7de00640 1 --2- 172.21.15.96:0/154943149 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0b640039e0 0x7f0b64003e00 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f0b70009ac0 tx=0x7f0b64005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b16e076c179d2c68 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:58.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.094+0000 7f0b7d400640 1 -- 172.21.15.96:0/154943149 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b70036070 con 0x7f0b640039e0 2024-05-28T06:14:58.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.095+0000 7f0b7d400640 1 -- 172.21.15.96:0/154943149 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b70031400 con 0x7f0b640039e0 2024-05-28T06:14:58.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.095+0000 7f0b7d400640 1 -- 172.21.15.96:0/154943149 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b70009690 con 0x7f0b640039e0 2024-05-28T06:14:58.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.095+0000 7f0b7f200640 1 -- 172.21.15.96:0/154943149 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0b640039e0 msgr2=0x7f0b64003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:58.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.095+0000 7f0b7f200640 1 --2- 172.21.15.96:0/154943149 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0b640039e0 0x7f0b64003e00 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f0b70009ac0 tx=0x7f0b64005830 comp rx=0 tx=0).stop 2024-05-28T06:14:58.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.095+0000 7f0b7e800640 1 -- 172.21.15.96:0/154943149 reap_dead start 2024-05-28T06:14:58.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.095+0000 7f0b7f200640 1 -- 172.21.15.96:0/154943149 shutdown_connections 2024-05-28T06:14:58.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.095+0000 7f0b7f200640 1 -- 172.21.15.96:0/154943149 >> 172.21.15.96:0/154943149 conn(0x7f0b8006b900 msgr2=0x7f0b8006bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:58.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.095+0000 7f0b7f200640 1 -- 172.21.15.96:0/154943149 shutdown_connections 2024-05-28T06:14:58.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.095+0000 7f0b7f200640 1 -- 172.21.15.96:0/154943149 wait complete. 2024-05-28T06:14:58.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.097+0000 7f0b7f200640 1 Processor -- start 2024-05-28T06:14:58.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.097+0000 7f0b7f200640 1 -- start start 2024-05-28T06:14:58.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.098+0000 7f0b7f200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0b640039e0 0x7f0b80151bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:58.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.098+0000 7f0b7f200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0b800b0ef0 0x7f0b800b1370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:58.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.098+0000 7f0b7f200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0b80073120 0x7f0b800735a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:58.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.098+0000 7f0b7f200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f0b80166d80 con 0x7f0b80073120 2024-05-28T06:14:58.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.098+0000 7f0b7f200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f0b800ce240 con 0x7f0b640039e0 2024-05-28T06:14:58.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.098+0000 7f0b7f200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f0b800afa70 con 0x7f0b800b0ef0 2024-05-28T06:14:58.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.099+0000 7f0b7e800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0b80073120 0x7f0b800735a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:58.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.099+0000 7f0b7e800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0b80073120 0x7f0b800735a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54690/0 (socket says 172.21.15.96:54690) 2024-05-28T06:14:58.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.099+0000 7f0b7e800640 1 -- 172.21.15.96:0/2575238459 learned_addr learned my addr 172.21.15.96:0/2575238459 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:58.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.099+0000 7f0b7de00640 1 --2- 172.21.15.96:0/2575238459 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0b640039e0 0x7f0b80151bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:58.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.099+0000 7f0b77e00640 1 --2- 172.21.15.96:0/2575238459 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0b800b0ef0 0x7f0b800b1370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:58.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.099+0000 7f0b7e800640 1 -- 172.21.15.96:0/2575238459 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0b800b0ef0 msgr2=0x7f0b800b1370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:58.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.099+0000 7f0b7e800640 1 --2- 172.21.15.96:0/2575238459 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0b800b0ef0 0x7f0b800b1370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:58.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.099+0000 7f0b7e800640 1 -- 172.21.15.96:0/2575238459 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0b640039e0 msgr2=0x7f0b80151bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:58.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.099+0000 7f0b7e800640 1 --2- 172.21.15.96:0/2575238459 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0b640039e0 0x7f0b80151bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:58.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.099+0000 7f0b7e800640 1 -- 172.21.15.96:0/2575238459 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b80072a80 con 0x7f0b80073120 2024-05-28T06:14:58.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.099+0000 7f0b77e00640 1 --2- 172.21.15.96:0/2575238459 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0b800b0ef0 0x7f0b800b1370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:14:58.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.099+0000 7f0b7e800640 1 --2- 172.21.15.96:0/2575238459 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0b80073120 0x7f0b800735a0 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f0b800b34d0 tx=0x7f0b78009cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:58.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.100+0000 7f0b76000640 1 -- 172.21.15.96:0/2575238459 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b78007d20 con 0x7f0b80073120 2024-05-28T06:14:58.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.100+0000 7f0b76000640 1 -- 172.21.15.96:0/2575238459 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b78019070 con 0x7f0b80073120 2024-05-28T06:14:58.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.100+0000 7f0b76000640 1 -- 172.21.15.96:0/2575238459 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b78014650 con 0x7f0b80073120 2024-05-28T06:14:58.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.100+0000 7f0b7f200640 1 -- 172.21.15.96:0/2575238459 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b800b0ca0 con 0x7f0b80073120 2024-05-28T06:14:58.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.100+0000 7f0b7f200640 1 -- 172.21.15.96:0/2575238459 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b800b01c0 con 0x7f0b80073120 2024-05-28T06:14:58.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.101+0000 7f0b7f200640 1 -- 172.21.15.96:0/2575238459 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b800689e0 con 0x7f0b80073120 2024-05-28T06:14:58.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.102+0000 7f0b76000640 1 -- 172.21.15.96:0/2575238459 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 22) ==== 92213+0+0 (secure 0 0 0) 0x7f0b780147f0 con 0x7f0b80073120 2024-05-28T06:14:58.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.103+0000 7f0b76000640 1 --2- 172.21.15.96:0/2575238459 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0b5c06d610 0x7f0b5c06fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:58.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.103+0000 7f0b76000640 1 -- 172.21.15.96:0/2575238459 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 5737+0+0 (secure 0 0 0) 0x7f0b7808ace0 con 0x7f0b80073120 2024-05-28T06:14:58.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.104+0000 7f0b7de00640 1 --2- 172.21.15.96:0/2575238459 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0b5c06d610 0x7f0b5c06fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:58.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.105+0000 7f0b7de00640 1 --2- 172.21.15.96:0/2575238459 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0b5c06d610 0x7f0b5c06fad0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f0b70009ac0 tx=0x7f0b70008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:58.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.107+0000 7f0b76000640 1 -- 172.21.15.96:0/2575238459 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0b7805a8d0 con 0x7f0b80073120 2024-05-28T06:14:58.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.391+0000 7f0b7f200640 1 -- 172.21.15.96:0/2575238459 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f0b800af480 con 0x7f0b80073120 2024-05-28T06:14:58.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.395+0000 7f0b76000640 1 -- 172.21.15.96:0/2575238459 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v17) ==== 190+0+147 (secure 0 0 0) 0x7f0b7805a1d0 con 0x7f0b80073120 2024-05-28T06:14:58.397 INFO:teuthology.orchestra.run.smithi096.stdout:[client.0] 2024-05-28T06:14:58.397 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQCXc1Vm91hbGBAASTL3HQR8XOs7wVQEHbPl/g== 2024-05-28T06:14:58.397 INFO:teuthology.orchestra.run.smithi096.stdout: caps mds = "allow" 2024-05-28T06:14:58.398 INFO:teuthology.orchestra.run.smithi096.stdout: caps mon = "allow r" 2024-05-28T06:14:58.398 INFO:teuthology.orchestra.run.smithi096.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-05-28T06:14:58.398 INFO:teuthology.orchestra.run.smithi096.stderr:updated caps for client.0 2024-05-28T06:14:58.400 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.398+0000 7f0b7f200640 1 -- 172.21.15.96:0/2575238459 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0b5c06d610 msgr2=0x7f0b5c06fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:58.400 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.398+0000 7f0b7f200640 1 --2- 172.21.15.96:0/2575238459 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0b5c06d610 0x7f0b5c06fad0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f0b70009ac0 tx=0x7f0b70008030 comp rx=0 tx=0).stop 2024-05-28T06:14:58.400 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.398+0000 7f0b7f200640 1 -- 172.21.15.96:0/2575238459 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0b80073120 msgr2=0x7f0b800735a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:58.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.398+0000 7f0b7f200640 1 --2- 172.21.15.96:0/2575238459 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0b80073120 0x7f0b800735a0 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f0b800b34d0 tx=0x7f0b78009cd0 comp rx=0 tx=0).stop 2024-05-28T06:14:58.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.399+0000 7f0b7f200640 1 -- 172.21.15.96:0/2575238459 shutdown_connections 2024-05-28T06:14:58.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.399+0000 7f0b7f200640 1 --2- 172.21.15.96:0/2575238459 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0b5c06d610 0x7f0b5c06fad0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:58.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.399+0000 7f0b7f200640 1 --2- 172.21.15.96:0/2575238459 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0b80073120 0x7f0b800735a0 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:58.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.399+0000 7f0b7f200640 1 --2- 172.21.15.96:0/2575238459 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0b800b0ef0 0x7f0b800b1370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:58.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.399+0000 7f0b7f200640 1 --2- 172.21.15.96:0/2575238459 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0b640039e0 0x7f0b80151bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:58.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.399+0000 7f0b7f200640 1 -- 172.21.15.96:0/2575238459 >> 172.21.15.96:0/2575238459 conn(0x7f0b8006b900 msgr2=0x7f0b8006e400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:58.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.399+0000 7f0b7f200640 1 -- 172.21.15.96:0/2575238459 shutdown_connections 2024-05-28T06:14:58.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.399+0000 7f0b7f200640 1 -- 172.21.15.96:0/2575238459 wait complete. 2024-05-28T06:14:58.421 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-28T06:14:58.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.622+0000 7f8b47200640 1 Processor -- start 2024-05-28T06:14:58.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.622+0000 7f8b47200640 1 -- start start 2024-05-28T06:14:58.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.623+0000 7f8b47200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8b48156b30 0x7f8b480af6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:58.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.623+0000 7f8b47200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f8b4806ee40 0x7f8b480ad230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:58.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.623+0000 7f8b47200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f8b4806f2c0 0x7f8b480ad770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:58.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.623+0000 7f8b47200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f8b48156f10 con 0x7f8b4806ee40 2024-05-28T06:14:58.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.623+0000 7f8b47200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f8b4806dda0 con 0x7f8b4806f2c0 2024-05-28T06:14:58.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.623+0000 7f8b47200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f8b4806e080 con 0x7f8b48156b30 2024-05-28T06:14:58.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.624+0000 7f8b45400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f8b4806ee40 0x7f8b480ad230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:58.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.624+0000 7f8b45400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f8b4806ee40 0x7f8b480ad230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54700/0 (socket says 172.21.15.96:54700) 2024-05-28T06:14:58.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.624+0000 7f8b45400640 1 -- 172.21.15.96:0/3212553400 learned_addr learned my addr 172.21.15.96:0/3212553400 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:58.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.624+0000 7f8b45e00640 1 --2- 172.21.15.96:0/3212553400 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8b48156b30 0x7f8b480af6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:58.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.624+0000 7f8b45400640 1 -- 172.21.15.96:0/3212553400 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8b48156b30 msgr2=0x7f8b480af6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:58.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.624+0000 7f8b45400640 1 --2- 172.21.15.96:0/3212553400 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8b48156b30 0x7f8b480af6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:58.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.624+0000 7f8b45400640 1 -- 172.21.15.96:0/3212553400 >> v2:172.21.15.123:3300/0 conn(0x7f8b4806f2c0 msgr2=0x7f8b480ad770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:58.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.624+0000 7f8b45400640 1 --2- 172.21.15.96:0/3212553400 >> v2:172.21.15.123:3300/0 conn(0x7f8b4806f2c0 0x7f8b480ad770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:58.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.624+0000 7f8b45400640 1 -- 172.21.15.96:0/3212553400 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b4806e360 con 0x7f8b4806ee40 2024-05-28T06:14:58.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.625+0000 7f8b45400640 1 --2- 172.21.15.96:0/3212553400 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8b4806ee40 0x7f8b480ad230 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f8b48071410 tx=0x7f8b3402a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf5acebb480c862c server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:58.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.626+0000 7f8b44a00640 1 -- 172.21.15.96:0/3212553400 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b34037070 con 0x7f8b4806ee40 2024-05-28T06:14:58.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.626+0000 7f8b44a00640 1 -- 172.21.15.96:0/3212553400 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b34032440 con 0x7f8b4806ee40 2024-05-28T06:14:58.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.626+0000 7f8b44a00640 1 -- 172.21.15.96:0/3212553400 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b34032810 con 0x7f8b4806ee40 2024-05-28T06:14:58.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.626+0000 7f8b47200640 1 -- 172.21.15.96:0/3212553400 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8b4806ee40 msgr2=0x7f8b480ad230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:58.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.626+0000 7f8b47200640 1 --2- 172.21.15.96:0/3212553400 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8b4806ee40 0x7f8b480ad230 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f8b48071410 tx=0x7f8b3402a140 comp rx=0 tx=0).stop 2024-05-28T06:14:58.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.627+0000 7f8b47200640 1 -- 172.21.15.96:0/3212553400 shutdown_connections 2024-05-28T06:14:58.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.627+0000 7f8b47200640 1 --2- 172.21.15.96:0/3212553400 >> v2:172.21.15.123:3300/0 conn(0x7f8b4806f2c0 0x7f8b480ad770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:58.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.627+0000 7f8b47200640 1 --2- 172.21.15.96:0/3212553400 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8b4806ee40 0x7f8b480ad230 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:58.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.627+0000 7f8b47200640 1 --2- 172.21.15.96:0/3212553400 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8b48156b30 0x7f8b480af6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:58.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.627+0000 7f8b47200640 1 -- 172.21.15.96:0/3212553400 >> 172.21.15.96:0/3212553400 conn(0x7f8b480b1fc0 msgr2=0x7f8b480b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:58.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.627+0000 7f8b47200640 1 -- 172.21.15.96:0/3212553400 shutdown_connections 2024-05-28T06:14:58.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.627+0000 7f8b47200640 1 -- 172.21.15.96:0/3212553400 wait complete. 2024-05-28T06:14:58.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.629+0000 7f8b47200640 1 Processor -- start 2024-05-28T06:14:58.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.629+0000 7f8b47200640 1 -- start start 2024-05-28T06:14:58.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.630+0000 7f8b47200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8b4806ee40 0x7f8b481f4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:58.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.630+0000 7f8b47200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b4806f2c0 0x7f8b48155860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:58.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.630+0000 7f8b47200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8b48156b30 0x7f8b48099540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:58.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.630+0000 7f8b47200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f8b480afc20 con 0x7f8b48156b30 2024-05-28T06:14:58.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.630+0000 7f8b47200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f8b480b2c30 con 0x7f8b4806f2c0 2024-05-28T06:14:58.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.630+0000 7f8b47200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f8b481f7a50 con 0x7f8b4806ee40 2024-05-28T06:14:58.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.630+0000 7f8b46800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8b48156b30 0x7f8b48099540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:58.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.631+0000 7f8b46800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8b48156b30 0x7f8b48099540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:54714/0 (socket says 172.21.15.96:54714) 2024-05-28T06:14:58.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.631+0000 7f8b46800640 1 -- 172.21.15.96:0/1211376020 learned_addr learned my addr 172.21.15.96:0/1211376020 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:58.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.631+0000 7f8b45400640 1 --2- 172.21.15.96:0/1211376020 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b4806f2c0 0x7f8b48155860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:58.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.631+0000 7f8b45e00640 1 --2- 172.21.15.96:0/1211376020 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8b4806ee40 0x7f8b481f4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:58.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.631+0000 7f8b46800640 1 -- 172.21.15.96:0/1211376020 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8b4806ee40 msgr2=0x7f8b481f4ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:58.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.631+0000 7f8b46800640 1 --2- 172.21.15.96:0/1211376020 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8b4806ee40 0x7f8b481f4ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:58.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.631+0000 7f8b46800640 1 -- 172.21.15.96:0/1211376020 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b4806f2c0 msgr2=0x7f8b48155860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:58.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.631+0000 7f8b46800640 1 --2- 172.21.15.96:0/1211376020 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b4806f2c0 0x7f8b48155860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:58.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.631+0000 7f8b46800640 1 -- 172.21.15.96:0/1211376020 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b481f7ca0 con 0x7f8b48156b30 2024-05-28T06:14:58.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.631+0000 7f8b45400640 1 --2- 172.21.15.96:0/1211376020 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b4806f2c0 0x7f8b48155860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:14:58.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.631+0000 7f8b46800640 1 --2- 172.21.15.96:0/1211376020 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8b48156b30 0x7f8b48099540 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f8b4000ba10 tx=0x7f8b4000bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:58.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.632+0000 7f8b3e000640 1 -- 172.21.15.96:0/1211376020 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b40019070 con 0x7f8b48156b30 2024-05-28T06:14:58.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.632+0000 7f8b3e000640 1 -- 172.21.15.96:0/1211376020 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b4000cae0 con 0x7f8b48156b30 2024-05-28T06:14:58.635 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.632+0000 7f8b3e000640 1 -- 172.21.15.96:0/1211376020 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b40014440 con 0x7f8b48156b30 2024-05-28T06:14:58.635 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.632+0000 7f8b47200640 1 -- 172.21.15.96:0/1211376020 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b481f7f80 con 0x7f8b48156b30 2024-05-28T06:14:58.635 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.632+0000 7f8b47200640 1 -- 172.21.15.96:0/1211376020 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b481f8540 con 0x7f8b48156b30 2024-05-28T06:14:58.635 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.633+0000 7f8b47200640 1 -- 172.21.15.96:0/1211376020 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b480689e0 con 0x7f8b48156b30 2024-05-28T06:14:58.635 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.634+0000 7f8b3e000640 1 -- 172.21.15.96:0/1211376020 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 22) ==== 92213+0+0 (secure 0 0 0) 0x7f8b4001c070 con 0x7f8b48156b30 2024-05-28T06:14:58.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.635+0000 7f8b3e000640 1 --2- 172.21.15.96:0/1211376020 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8b2806d5e0 0x7f8b2806faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:58.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.635+0000 7f8b3e000640 1 -- 172.21.15.96:0/1211376020 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 5737+0+0 (secure 0 0 0) 0x7f8b4008a250 con 0x7f8b48156b30 2024-05-28T06:14:58.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.636+0000 7f8b45e00640 1 --2- 172.21.15.96:0/1211376020 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8b2806d5e0 0x7f8b2806faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:58.638 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.637+0000 7f8b45e00640 1 --2- 172.21.15.96:0/1211376020 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8b2806d5e0 0x7f8b2806faa0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f8b3800a970 tx=0x7f8b38008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:58.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.639+0000 7f8b3e000640 1 -- 172.21.15.96:0/1211376020 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8b4005a0d0 con 0x7f8b48156b30 2024-05-28T06:14:58.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.877+0000 7f8b47200640 1 -- 172.21.15.96:0/1211376020 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8b4806f7f0 con 0x7f8b48156b30 2024-05-28T06:14:58.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.879+0000 7f8b3e000640 1 -- 172.21.15.96:0/1211376020 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) ==== 94+0+5177 (secure 0 0 0) 0x7f8b400599d0 con 0x7f8b48156b30 2024-05-28T06:14:58.881 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:14:58.881 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":43,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5291,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6834/3727695025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":3727695025},{"type":"v1","addr":"172.21.15.96:6834","nonce":3727695025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38},{"gid":5300,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/324418606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6835","nonce":324418606},{"type":"v1","addr":"172.21.15.123:6837","nonce":324418606}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38},{"gid":5725,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/417566829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6833","nonce":417566829},{"type":"v1","addr":"172.21.15.96:6835","nonce":417566829}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-28T06:14:52.322597+0000","modified":"2024-05-28T06:14:55.415769+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7146},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7146":{"gid":7146,"name":"d","rank":0,"incarnation":39,"state":"up:active","state_seq":3,"addr":"172.21.15.123:6836/76143454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":76143454},{"type":"v1","addr":"172.21.15.123:6836","nonce":76143454}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7146,"qdb_cluster":[7146]},"id":5}]} 2024-05-28T06:14:58.881 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 43 2024-05-28T06:14:58.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.881+0000 7f8b47200640 1 -- 172.21.15.96:0/1211376020 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8b2806d5e0 msgr2=0x7f8b2806faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:58.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.881+0000 7f8b47200640 1 --2- 172.21.15.96:0/1211376020 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8b2806d5e0 0x7f8b2806faa0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f8b3800a970 tx=0x7f8b38008040 comp rx=0 tx=0).stop 2024-05-28T06:14:58.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.881+0000 7f8b47200640 1 -- 172.21.15.96:0/1211376020 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8b48156b30 msgr2=0x7f8b48099540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:58.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.881+0000 7f8b47200640 1 --2- 172.21.15.96:0/1211376020 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8b48156b30 0x7f8b48099540 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f8b4000ba10 tx=0x7f8b4000bed0 comp rx=0 tx=0).stop 2024-05-28T06:14:58.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.882+0000 7f8b47200640 1 -- 172.21.15.96:0/1211376020 shutdown_connections 2024-05-28T06:14:58.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.882+0000 7f8b47200640 1 --2- 172.21.15.96:0/1211376020 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8b2806d5e0 0x7f8b2806faa0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:58.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.882+0000 7f8b47200640 1 --2- 172.21.15.96:0/1211376020 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8b48156b30 0x7f8b48099540 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:58.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.882+0000 7f8b47200640 1 --2- 172.21.15.96:0/1211376020 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b4806f2c0 0x7f8b48155860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:58.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.882+0000 7f8b47200640 1 --2- 172.21.15.96:0/1211376020 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8b4806ee40 0x7f8b481f4ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:58.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.882+0000 7f8b47200640 1 -- 172.21.15.96:0/1211376020 >> 172.21.15.96:0/1211376020 conn(0x7f8b480b1fc0 msgr2=0x7f8b480b08d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:58.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.882+0000 7f8b47200640 1 -- 172.21.15.96:0/1211376020 shutdown_connections 2024-05-28T06:14:58.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:58.882+0000 7f8b47200640 1 -- 172.21.15.96:0/1211376020 wait complete. 2024-05-28T06:14:59.305 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 43, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-05-28T06:14:52.322597+0000', 'modified': '2024-05-28T06:14:55.415769+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7146}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7146': {'gid': 7146, 'name': 'd', 'rank': 0, 'incarnation': 39, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.123:6836/76143454', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.123:6834', 'nonce': 76143454}, {'type': 'v1', 'addr': '172.21.15.123:6836', 'nonce': 76143454}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7146, 'qdb_cluster': [7146]} 2024-05-28T06:14:59.305 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-05-28T06:14:59.305 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-05-28T06:14:59.512 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.510+0000 7f92f7200640 1 Processor -- start 2024-05-28T06:14:59.512 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.510+0000 7f92f7200640 1 -- start start 2024-05-28T06:14:59.513 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.512+0000 7f92f7200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f92f8156b30 0x7f92f80af6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:59.514 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.512+0000 7f92f7200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f92f806ee40 0x7f92f80ad230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:59.514 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.512+0000 7f92f7200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92f8156f10 con 0x7f92f806f2c0 2024-05-28T06:14:59.514 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.512+0000 7f92f7200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f92f806dda0 con 0x7f92f8156b30 2024-05-28T06:14:59.514 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.512+0000 7f92f7200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f92f806e080 con 0x7f92f806ee40 2024-05-28T06:14:59.514 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.512+0000 7f92f5e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f92f8156b30 0x7f92f80af6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:59.514 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.512+0000 7f92f5e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f92f8156b30 0x7f92f80af6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:52776/0 (socket says 172.21.15.96:52776) 2024-05-28T06:14:59.514 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.512+0000 7f92f5e00640 1 -- 172.21.15.96:0/1301437417 learned_addr learned my addr 172.21.15.96:0/1301437417 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:59.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.513+0000 7f92f5400640 1 --2- 172.21.15.96:0/1301437417 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f92f806ee40 0x7f92f80ad230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:59.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.513+0000 7f92f5e00640 1 -- 172.21.15.96:0/1301437417 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f92f806ee40 msgr2=0x7f92f80ad230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:59.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.513+0000 7f92f5e00640 1 --2- 172.21.15.96:0/1301437417 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f92f806ee40 0x7f92f80ad230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:59.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.513+0000 7f92f5e00640 1 -- 172.21.15.96:0/1301437417 >> v1:172.21.15.123:6789/0 conn(0x7f92f806f2c0 legacy=0x7f92f80b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:59.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.513+0000 7f92f5e00640 1 -- 172.21.15.96:0/1301437417 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92f806e360 con 0x7f92f8156b30 2024-05-28T06:14:59.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.513+0000 7f92f5400640 1 --2- 172.21.15.96:0/1301437417 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f92f806ee40 0x7f92f80ad230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:14:59.516 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.514+0000 7f92f5e00640 1 --2- 172.21.15.96:0/1301437417 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f92f8156b30 0x7f92f80af6e0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f92e8009870 tx=0x7f92e802a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35fb351626c85507 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:59.516 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.515+0000 7f92f4a00640 1 -- 172.21.15.96:0/1301437417 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92e8037070 con 0x7f92f8156b30 2024-05-28T06:14:59.516 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.515+0000 7f92f4a00640 1 -- 172.21.15.96:0/1301437417 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92e8032440 con 0x7f92f8156b30 2024-05-28T06:14:59.516 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.515+0000 7f92f4a00640 1 -- 172.21.15.96:0/1301437417 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92e8032810 con 0x7f92f8156b30 2024-05-28T06:14:59.517 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.515+0000 7f92f7200640 1 -- 172.21.15.96:0/1301437417 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f92f8156b30 msgr2=0x7f92f80af6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:59.517 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.515+0000 7f92f7200640 1 --2- 172.21.15.96:0/1301437417 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f92f8156b30 0x7f92f80af6e0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f92e8009870 tx=0x7f92e802a180 comp rx=0 tx=0).stop 2024-05-28T06:14:59.517 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.516+0000 7f92f7200640 1 -- 172.21.15.96:0/1301437417 shutdown_connections 2024-05-28T06:14:59.517 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.516+0000 7f92f7200640 1 --2- 172.21.15.96:0/1301437417 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f92f806ee40 0x7f92f80ad230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:59.517 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.516+0000 7f92f7200640 1 --2- 172.21.15.96:0/1301437417 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f92f8156b30 0x7f92f80af6e0 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:59.517 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.516+0000 7f92f7200640 1 -- 172.21.15.96:0/1301437417 >> 172.21.15.96:0/1301437417 conn(0x7f92f80b1fc0 msgr2=0x7f92f80b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:59.518 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.516+0000 7f92f7200640 1 -- 172.21.15.96:0/1301437417 shutdown_connections 2024-05-28T06:14:59.518 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.516+0000 7f92f7200640 1 -- 172.21.15.96:0/1301437417 wait complete. 2024-05-28T06:14:59.518 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.517+0000 7f92f7200640 1 Processor -- start 2024-05-28T06:14:59.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.517+0000 7f92f7200640 1 -- start start 2024-05-28T06:14:59.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.518+0000 7f92f7200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f92f806ee40 0x7f92f814c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:59.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.518+0000 7f92f7200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f92f806f2c0 0x7f92f815b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:59.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.518+0000 7f92f7200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92f8156b30 0x7f92f815beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:59.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.518+0000 7f92f7200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f92f80afc20 con 0x7f92f806ee40 2024-05-28T06:14:59.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.518+0000 7f92f7200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f92f80b2c30 con 0x7f92f8156b30 2024-05-28T06:14:59.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.518+0000 7f92f7200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f92f814c730 con 0x7f92f806f2c0 2024-05-28T06:14:59.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.518+0000 7f92f5e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f92f806ee40 0x7f92f814c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:59.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.519+0000 7f92f5e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f92f806ee40 0x7f92f814c1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:52778/0 (socket says 172.21.15.96:52778) 2024-05-28T06:14:59.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.519+0000 7f92f5e00640 1 -- 172.21.15.96:0/71621783 learned_addr learned my addr 172.21.15.96:0/71621783 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:59.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.519+0000 7f92f5e00640 1 -- 172.21.15.96:0/71621783 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f92f806f2c0 msgr2=0x7f92f815b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:59.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.519+0000 7f92f5e00640 1 --2- 172.21.15.96:0/71621783 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f92f806f2c0 0x7f92f815b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:59.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.519+0000 7f92f5e00640 1 -- 172.21.15.96:0/71621783 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92f8156b30 msgr2=0x7f92f815beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:59.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.519+0000 7f92f5e00640 1 --2- 172.21.15.96:0/71621783 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92f8156b30 0x7f92f815beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:59.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.519+0000 7f92f5e00640 1 -- 172.21.15.96:0/71621783 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92f815e0f0 con 0x7f92f806ee40 2024-05-28T06:14:59.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.519+0000 7f92f5e00640 1 --2- 172.21.15.96:0/71621783 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f92f806ee40 0x7f92f814c1f0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f92e8000c00 tx=0x7f92e80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:59.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.520+0000 7f92ee000640 1 -- 172.21.15.96:0/71621783 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92e8037040 con 0x7f92f806ee40 2024-05-28T06:14:59.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.520+0000 7f92ee000640 1 -- 172.21.15.96:0/71621783 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92e8042070 con 0x7f92f806ee40 2024-05-28T06:14:59.523 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.520+0000 7f92f7200640 1 -- 172.21.15.96:0/71621783 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92f815e3d0 con 0x7f92f806ee40 2024-05-28T06:14:59.523 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.520+0000 7f92ee000640 1 -- 172.21.15.96:0/71621783 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92e8030bb0 con 0x7f92f806ee40 2024-05-28T06:14:59.523 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.520+0000 7f92f7200640 1 -- 172.21.15.96:0/71621783 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92f815e990 con 0x7f92f806ee40 2024-05-28T06:14:59.523 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.522+0000 7f92ee000640 1 -- 172.21.15.96:0/71621783 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 22) ==== 92213+0+0 (secure 0 0 0) 0x7f92e8047460 con 0x7f92f806ee40 2024-05-28T06:14:59.524 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.523+0000 7f92ee000640 1 --2- 172.21.15.96:0/71621783 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f92d806d5e0 0x7f92d806faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:59.524 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.523+0000 7f92ee000640 1 -- 172.21.15.96:0/71621783 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 5737+0+0 (secure 0 0 0) 0x7f92e80ab4a0 con 0x7f92f806ee40 2024-05-28T06:14:59.525 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.523+0000 7f92ff9be740 1 Processor -- start 2024-05-28T06:14:59.525 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.524+0000 7f92ff9be740 1 -- start start 2024-05-28T06:14:59.525 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.524+0000 7f92f5400640 1 --2- 172.21.15.96:0/71621783 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f92d806d5e0 0x7f92d806faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:59.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.525+0000 7f92ff9be740 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x5628dfacb490 0x5628dfacec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:59.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.525+0000 7f92ff9be740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5628dfacf150 0x5628dfaeb090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:59.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.525+0000 7f92ff9be740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5628dfaeb5d0 0x5628dfaeba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:59.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.525+0000 7f92ff9be740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5628dfa08b10 con 0x5628dfacf150 2024-05-28T06:14:59.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.525+0000 7f92ff9be740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x5628df9af2b0 con 0x5628dfaeb5d0 2024-05-28T06:14:59.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.525+0000 7f92ff9be740 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x5628dfa04ad0 con 0x5628dfacb490 2024-05-28T06:14:59.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.525+0000 7f92f5400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5628dfacf150 0x5628dfaeb090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:59.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.525+0000 7f92f5400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5628dfacf150 0x5628dfaeb090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:52792/0 (socket says 172.21.15.96:52792) 2024-05-28T06:14:59.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.525+0000 7f92f5400640 1 -- 172.21.15.96:0/894730013 learned_addr learned my addr 172.21.15.96:0/894730013 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:59.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.525+0000 7f92f5400640 1 --2- 172.21.15.96:0/71621783 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f92d806d5e0 0x7f92d806faa0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f92e4006fd0 tx=0x7f92e4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:59.528 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.525+0000 7f92f5400640 1 -- 172.21.15.96:0/894730013 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x5628dfacb490 msgr2=0x5628dfacec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:59.528 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.525+0000 7f92f5400640 1 --2- 172.21.15.96:0/894730013 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x5628dfacb490 0x5628dfacec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:59.528 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.525+0000 7f92f5400640 1 -- 172.21.15.96:0/894730013 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5628dfaeb5d0 msgr2=0x5628dfaeba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:59.528 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.525+0000 7f92f5400640 1 --2- 172.21.15.96:0/894730013 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5628dfaeb5d0 0x5628dfaeba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:59.528 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.525+0000 7f92f5400640 1 -- 172.21.15.96:0/894730013 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5628df71b2d0 con 0x5628dfacf150 2024-05-28T06:14:59.528 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.526+0000 7f92f5400640 1 --2- 172.21.15.96:0/894730013 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5628dfacf150 0x5628dfaeb090 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f92e400ca50 tx=0x7f92e400f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a25c7afa41c8e6a server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:59.529 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.526+0000 7f92e2600640 1 -- 172.21.15.96:0/894730013 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92e4023070 con 0x5628dfacf150 2024-05-28T06:14:59.529 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.526+0000 7f92e2600640 1 -- 172.21.15.96:0/894730013 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92e4013db0 con 0x5628dfacf150 2024-05-28T06:14:59.529 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.526+0000 7f92e2600640 1 -- 172.21.15.96:0/894730013 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92e401e600 con 0x5628dfacf150 2024-05-28T06:14:59.529 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.526+0000 7f92ff9be740 1 -- 172.21.15.96:0/894730013 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5628dfacf150 msgr2=0x5628dfaeb090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:59.529 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.526+0000 7f92ff9be740 1 --2- 172.21.15.96:0/894730013 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5628dfacf150 0x5628dfaeb090 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f92e400ca50 tx=0x7f92e400f000 comp rx=0 tx=0).stop 2024-05-28T06:14:59.530 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.527+0000 7f92ff9be740 1 -- 172.21.15.96:0/894730013 shutdown_connections 2024-05-28T06:14:59.530 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.527+0000 7f92ff9be740 1 --2- 172.21.15.96:0/894730013 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5628dfaeb5d0 0x5628dfaeba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:59.530 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.527+0000 7f92ff9be740 1 --2- 172.21.15.96:0/894730013 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5628dfacf150 0x5628dfaeb090 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:59.530 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.527+0000 7f92ff9be740 1 --2- 172.21.15.96:0/894730013 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x5628dfacb490 0x5628dfacec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:59.530 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.527+0000 7f92ff9be740 1 -- 172.21.15.96:0/894730013 >> 172.21.15.96:0/894730013 conn(0x5628dfaa1470 msgr2=0x5628dfaa2470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:59.530 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.527+0000 7f92ff9be740 1 -- 172.21.15.96:0/894730013 shutdown_connections 2024-05-28T06:14:59.530 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.527+0000 7f92ff9be740 1 -- 172.21.15.96:0/894730013 wait complete. 2024-05-28T06:14:59.530 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.527+0000 7f92ff9be740 1 -- start start 2024-05-28T06:14:59.530 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.528+0000 7f92ff9be740 1 Processor -- start 2024-05-28T06:14:59.533 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.528+0000 7f92ff9be740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5628dfacf150 0x5628dfac7130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:59.533 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.528+0000 7f92ff9be740 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x5628dfaeb5d0 0x5628dfad2870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:59.533 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.528+0000 7f92ff9be740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5628dfaf9a80 0x5628dfaf9f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:59.533 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.529+0000 7f92ff9be740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5628dfa9f2a0 con 0x5628dfacf150 2024-05-28T06:14:59.533 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.529+0000 7f92ff9be740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x5628df9c2570 con 0x5628dfaf9a80 2024-05-28T06:14:59.533 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.529+0000 7f92ff9be740 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x5628dfa9f580 con 0x5628dfaeb5d0 2024-05-28T06:14:59.533 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.529+0000 7f92ff9be740 10 client.0 _ll_register_callbacks cb 0x5628df990fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-28T06:14:59.533 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.529+0000 7f92f5e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5628dfacf150 0x5628dfac7130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:59.533 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.529+0000 7f92f5e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5628dfacf150 0x5628dfac7130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:52800/0 (socket says 172.21.15.96:52800) 2024-05-28T06:14:59.533 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.529+0000 7f92f5e00640 1 -- 172.21.15.96:0/1271155844 learned_addr learned my addr 172.21.15.96:0/1271155844 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:59.533 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.529+0000 7f92f5e00640 1 -- 172.21.15.96:0/1271155844 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x5628dfaeb5d0 msgr2=0x5628dfad2870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:59.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.529+0000 7f92f5e00640 1 --2- 172.21.15.96:0/1271155844 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x5628dfaeb5d0 0x5628dfad2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:59.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.529+0000 7f92f5e00640 1 -- 172.21.15.96:0/1271155844 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5628dfaf9a80 msgr2=0x5628dfaf9f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:59.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.529+0000 7f92f5e00640 1 --2- 172.21.15.96:0/1271155844 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5628dfaf9a80 0x5628dfaf9f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:59.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.529+0000 7f92f5e00640 1 -- 172.21.15.96:0/1271155844 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5628df7e56b0 con 0x5628dfacf150 2024-05-28T06:14:59.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.529+0000 7f92f5e00640 1 --2- 172.21.15.96:0/1271155844 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5628dfacf150 0x5628dfac7130 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f92e807a690 tx=0x7f92e807a3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fe7485932e43d0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:59.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.529+0000 7f92d6a00640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-05-28T06:14:59.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.529+0000 7f92d6a00640 1 -- 172.21.15.96:0/1271155844 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92e8081270 con 0x5628dfacf150 2024-05-28T06:14:59.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.530+0000 7f92d6a00640 1 -- 172.21.15.96:0/1271155844 <== mon.0 v2:172.21.15.96:3300/0 2 ==== osd_map(74..74 src has 1..74) ==== 5737+0+0 (secure 0 0 0) 0x7f92e807abf0 con 0x5628dfacf150 2024-05-28T06:14:59.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.530+0000 7f92ff9be740 1 -- 172.21.15.96:0/1271155844 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f92e4013db0 con 0x5628dfacf150 2024-05-28T06:14:59.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.530+0000 7f92d6a00640 1 -- 172.21.15.96:0/1271155844 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f92e8094050 con 0x5628dfacf150 2024-05-28T06:14:59.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.530+0000 7f92ff9be740 10 client.7290 fetch_fsmap learned FSMap version 43 2024-05-28T06:14:59.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.530+0000 7f92ff9be740 1 -- 172.21.15.96:0/1271155844 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=2+}) -- 0x5628dfad0190 con 0x5628dfacf150 2024-05-28T06:14:59.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.530+0000 7f92d6a00640 1 -- 172.21.15.96:0/1271155844 <== mon.0 v2:172.21.15.96:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92e4013db0 con 0x5628dfacf150 2024-05-28T06:14:59.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.530+0000 7f92d6a00640 1 -- 172.21.15.96:0/1271155844 <== mon.0 v2:172.21.15.96:3300/0 5 ==== fsmap(e 43) ==== 3494+0+0 (secure 0 0 0) 0x7f92e803a030 con 0x5628dfacf150 2024-05-28T06:14:59.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.530+0000 7f92ff9be740 10 client.7290 fetch_fsmap finished waiting for FSMap version 43 2024-05-28T06:14:59.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.530+0000 7f92ff9be740 10 client.7290 resolve_mds: resolved name 'd' to daemon mds.d 2024-05-28T06:14:59.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.530+0000 7f92ff9be740 20 client.7290 populate_metadata read hostname 'smithi096' 2024-05-28T06:14:59.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.530+0000 7f92ff9be740 1 --2- 172.21.15.96:0/1271155844 >> [v2:172.21.15.123:6834/76143454,v1:172.21.15.123:6836/76143454] conn(0x5628dfafd740 0x5628dfafdb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:59.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.530+0000 7f92ff9be740 4 client.7290 mds_command: new command op to 7146 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-28T06:14:59.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.530+0000 7f92ff9be740 1 -- 172.21.15.96:0/1271155844 --> [v2:172.21.15.123:6834/76143454,v1:172.21.15.123:6836/76143454] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5628dfa16530 con 0x5628dfafd740 2024-05-28T06:14:59.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.531+0000 7f92f5400640 1 --2- 172.21.15.96:0/1271155844 >> [v2:172.21.15.123:6834/76143454,v1:172.21.15.123:6836/76143454] conn(0x5628dfafd740 0x5628dfafdb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:59.536 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.532+0000 7f92f5400640 1 --2- 172.21.15.96:0/1271155844 >> [v2:172.21.15.123:6834/76143454,v1:172.21.15.123:6836/76143454] conn(0x5628dfafd740 0x5628dfafdb20 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ed2e05c99c478e2b server_cookie=c9d203fccec2bbfe in_seq=0 out_seq=0 2024-05-28T06:14:59.536 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.532+0000 7f92d6a00640 10 client.7290 ms_handle_connect on v2:172.21.15.123:6834/76143454 2024-05-28T06:14:59.536 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.534+0000 7f92d6a00640 1 -- 172.21.15.96:0/1271155844 <== mds.0 v2:172.21.15.123:6834/76143454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5628dfa16530 con 0x5628dfafd740 2024-05-28T06:14:59.536 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.534+0000 7f92d6a00640 10 client.7290 handle_command_reply: tid=0 2024-05-28T06:14:59.536 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.534+0000 7f92ff9be740 1 client.7290 shutdown 2024-05-28T06:14:59.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.535+0000 7f92ff9be740 1 -- 172.21.15.96:0/1271155844 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5628dfacf150 msgr2=0x5628dfac7130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:59.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.535+0000 7f92ff9be740 1 --2- 172.21.15.96:0/1271155844 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5628dfacf150 0x5628dfac7130 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f92e807a690 tx=0x7f92e807a3b0 comp rx=0 tx=0).stop 2024-05-28T06:14:59.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.535+0000 7f92ff9be740 1 -- 172.21.15.96:0/1271155844 shutdown_connections 2024-05-28T06:14:59.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.535+0000 7f92ff9be740 1 --2- 172.21.15.96:0/1271155844 >> [v2:172.21.15.123:6834/76143454,v1:172.21.15.123:6836/76143454] conn(0x5628dfafd740 0x5628dfafdb20 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:59.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.535+0000 7f92ff9be740 1 --2- 172.21.15.96:0/1271155844 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5628dfaf9a80 0x5628dfaf9f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:59.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.535+0000 7f92ff9be740 1 --2- 172.21.15.96:0/1271155844 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x5628dfaeb5d0 0x5628dfad2870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:59.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.535+0000 7f92ff9be740 1 --2- 172.21.15.96:0/1271155844 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5628dfacf150 0x5628dfac7130 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:59.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.535+0000 7f92d6a00640 0 client.7290 ms_handle_reset on v2:172.21.15.123:6834/76143454 2024-05-28T06:14:59.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.535+0000 7f92ff9be740 1 -- 172.21.15.96:0/1271155844 >> 172.21.15.96:0/1271155844 conn(0x5628dfacb490 msgr2=0x5628dfa9e1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:59.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.535+0000 7f92ff9be740 1 -- 172.21.15.96:0/1271155844 shutdown_connections 2024-05-28T06:14:59.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.535+0000 7f92ff9be740 1 -- 172.21.15.96:0/1271155844 wait complete. 2024-05-28T06:14:59.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.535+0000 7f92ff9be740 20 client.7290 trim_cache size 0 max 16384 2024-05-28T06:14:59.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.554+0000 7f92ff9be740 1 Processor -- start 2024-05-28T06:14:59.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.554+0000 7f92ff9be740 1 -- start start 2024-05-28T06:14:59.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.555+0000 7f92ff9be740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5628dfacf150 0x5628dfa9e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:59.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.555+0000 7f92ff9be740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5628dfaeb5d0 0x5628dfaa0100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:59.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.555+0000 7f92ff9be740 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x5628dfaf9a80 0x5628dfaa0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:59.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.555+0000 7f92ff9be740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f92f8200c40 con 0x5628dfacf150 2024-05-28T06:14:59.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.555+0000 7f92ff9be740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f92f81ffd30 con 0x5628dfaeb5d0 2024-05-28T06:14:59.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.555+0000 7f92ff9be740 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f92f815d330 con 0x5628dfaf9a80 2024-05-28T06:14:59.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.555+0000 7f92f5e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5628dfacf150 0x5628dfa9e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:59.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.555+0000 7f92f5e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5628dfacf150 0x5628dfa9e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:52816/0 (socket says 172.21.15.96:52816) 2024-05-28T06:14:59.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.555+0000 7f92f5e00640 1 -- 172.21.15.96:0/1190798486 learned_addr learned my addr 172.21.15.96:0/1190798486 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:59.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.555+0000 7f92f5e00640 1 -- 172.21.15.96:0/1190798486 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x5628dfaf9a80 msgr2=0x5628dfaa0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:59.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.555+0000 7f92f5e00640 1 --2- 172.21.15.96:0/1190798486 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x5628dfaf9a80 0x5628dfaa0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:59.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.555+0000 7f92f5e00640 1 -- 172.21.15.96:0/1190798486 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5628dfaeb5d0 msgr2=0x5628dfaa0100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:59.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.555+0000 7f92f5e00640 1 --2- 172.21.15.96:0/1190798486 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5628dfaeb5d0 0x5628dfaa0100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:59.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.555+0000 7f92f5e00640 1 -- 172.21.15.96:0/1190798486 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5628dfae51d0 con 0x5628dfacf150 2024-05-28T06:14:59.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.556+0000 7f92f5e00640 1 --2- 172.21.15.96:0/1190798486 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5628dfacf150 0x5628dfa9e590 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f92e807b0e0 tx=0x7f92e8081000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69d0f73584e98ac0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:59.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.556+0000 7f92e1c00640 1 -- 172.21.15.96:0/1190798486 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92e8076c30 con 0x5628dfacf150 2024-05-28T06:14:59.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.556+0000 7f92e1c00640 1 -- 172.21.15.96:0/1190798486 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92e8081270 con 0x5628dfacf150 2024-05-28T06:14:59.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.556+0000 7f92e1c00640 1 -- 172.21.15.96:0/1190798486 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92e80832e0 con 0x5628dfacf150 2024-05-28T06:14:59.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.556+0000 7f92ff9be740 1 -- 172.21.15.96:0/1190798486 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5628dfacf150 msgr2=0x5628dfa9e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:59.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.556+0000 7f92ff9be740 1 --2- 172.21.15.96:0/1190798486 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5628dfacf150 0x5628dfa9e590 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f92e807b0e0 tx=0x7f92e8081000 comp rx=0 tx=0).stop 2024-05-28T06:14:59.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.557+0000 7f92ff9be740 1 -- 172.21.15.96:0/1190798486 shutdown_connections 2024-05-28T06:14:59.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.557+0000 7f92ff9be740 1 --2- 172.21.15.96:0/1190798486 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x5628dfaf9a80 0x5628dfaa0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:59.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.557+0000 7f92ff9be740 1 --2- 172.21.15.96:0/1190798486 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5628dfaeb5d0 0x5628dfaa0100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:59.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.557+0000 7f92ff9be740 1 --2- 172.21.15.96:0/1190798486 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5628dfacf150 0x5628dfa9e590 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:59.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.557+0000 7f92ff9be740 1 -- 172.21.15.96:0/1190798486 >> 172.21.15.96:0/1190798486 conn(0x5628dfacb490 msgr2=0x5628dfacf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:59.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.557+0000 7f92ff9be740 1 -- 172.21.15.96:0/1190798486 shutdown_connections 2024-05-28T06:14:59.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.557+0000 7f92ff9be740 1 -- 172.21.15.96:0/1190798486 wait complete. 2024-05-28T06:14:59.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.557+0000 7f92ff9be740 1 -- start start 2024-05-28T06:14:59.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.557+0000 7f92ff9be740 1 Processor -- start 2024-05-28T06:14:59.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.558+0000 7f92ff9be740 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x5628dfacf150 0x5628dfad1100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:59.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.558+0000 7f92ff9be740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5628dfaeb5d0 0x5628dfb0d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:59.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.558+0000 7f92ff9be740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5628dfaf9a80 0x5628dfb0d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:59.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.558+0000 7f92ff9be740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5628dfad2db0 con 0x5628dfaeb5d0 2024-05-28T06:14:59.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.558+0000 7f92ff9be740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f92f815d980 con 0x5628dfaf9a80 2024-05-28T06:14:59.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.558+0000 7f92ff9be740 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x5628dfae4860 con 0x5628dfacf150 2024-05-28T06:14:59.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.558+0000 7f92ff9be740 10 client.0 _ll_register_callbacks cb 0x7f92e8006cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-28T06:14:59.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.559+0000 7f92f5400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5628dfaeb5d0 0x5628dfb0d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:59.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.559+0000 7f92f5400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5628dfaeb5d0 0x5628dfb0d4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:52818/0 (socket says 172.21.15.96:52818) 2024-05-28T06:14:59.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.559+0000 7f92f5400640 1 -- 172.21.15.96:0/3995433226 learned_addr learned my addr 172.21.15.96:0/3995433226 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:59.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.559+0000 7f92f5e00640 1 --2- 172.21.15.96:0/3995433226 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x5628dfacf150 0x5628dfad1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:59.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.559+0000 7f92f5400640 1 -- 172.21.15.96:0/3995433226 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x5628dfacf150 msgr2=0x5628dfad1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:59.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.559+0000 7f92f5400640 1 --2- 172.21.15.96:0/3995433226 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x5628dfacf150 0x5628dfad1100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:59.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.559+0000 7f92f5400640 1 -- 172.21.15.96:0/3995433226 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5628dfaf9a80 msgr2=0x5628dfb0d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:59.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.559+0000 7f92f6800640 1 --2- 172.21.15.96:0/3995433226 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5628dfaf9a80 0x5628dfb0d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:59.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.559+0000 7f92f5400640 1 --2- 172.21.15.96:0/3995433226 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5628dfaf9a80 0x5628dfb0d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:59.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.559+0000 7f92f5400640 1 -- 172.21.15.96:0/3995433226 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5628dfae9d10 con 0x5628dfaeb5d0 2024-05-28T06:14:59.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.559+0000 7f92f6800640 1 --2- 172.21.15.96:0/3995433226 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5628dfaf9a80 0x5628dfb0d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:14:59.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.559+0000 7f92f5400640 1 --2- 172.21.15.96:0/3995433226 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5628dfaeb5d0 0x5628dfb0d4a0 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f92e4021040 tx=0x7f92e40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7af948371c408b6 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:59.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.559+0000 7f92d6a00640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-05-28T06:14:59.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.560+0000 7f92d6a00640 1 -- 172.21.15.96:0/3995433226 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92e4032020 con 0x5628dfaeb5d0 2024-05-28T06:14:59.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.560+0000 7f92d6a00640 1 -- 172.21.15.96:0/3995433226 <== mon.0 v2:172.21.15.96:3300/0 2 ==== osd_map(74..74 src has 1..74) ==== 5737+0+0 (secure 0 0 0) 0x7f92e402e430 con 0x5628dfaeb5d0 2024-05-28T06:14:59.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.560+0000 7f92ff9be740 1 -- 172.21.15.96:0/3995433226 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f92e8081270 con 0x5628dfaeb5d0 2024-05-28T06:14:59.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.560+0000 7f92d6a00640 1 -- 172.21.15.96:0/3995433226 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f92e403c050 con 0x5628dfaeb5d0 2024-05-28T06:14:59.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.560+0000 7f92ff9be740 10 client.7296 fetch_fsmap learned FSMap version 43 2024-05-28T06:14:59.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.560+0000 7f92ff9be740 1 -- 172.21.15.96:0/3995433226 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=2+}) -- 0x5628dfaea610 con 0x5628dfaeb5d0 2024-05-28T06:14:59.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.560+0000 7f92d6a00640 1 -- 172.21.15.96:0/3995433226 <== mon.0 v2:172.21.15.96:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92e8081270 con 0x5628dfaeb5d0 2024-05-28T06:14:59.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.560+0000 7f92d6a00640 1 -- 172.21.15.96:0/3995433226 <== mon.0 v2:172.21.15.96:3300/0 5 ==== fsmap(e 43) ==== 3494+0+0 (secure 0 0 0) 0x7f92e403f020 con 0x5628dfaeb5d0 2024-05-28T06:14:59.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.560+0000 7f92ff9be740 10 client.7296 fetch_fsmap finished waiting for FSMap version 43 2024-05-28T06:14:59.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.560+0000 7f92ff9be740 10 client.7296 resolve_mds: resolved name 'd' to daemon mds.d 2024-05-28T06:14:59.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.560+0000 7f92ff9be740 20 client.7296 populate_metadata read hostname 'smithi096' 2024-05-28T06:14:59.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.561+0000 7f92ff9be740 1 --2- 172.21.15.96:0/3995433226 >> [v2:172.21.15.123:6834/76143454,v1:172.21.15.123:6836/76143454] conn(0x5628dfae3c80 0x5628dfae6310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:59.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.561+0000 7f92ff9be740 4 client.7296 mds_command: new command op to 7146 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-28T06:14:59.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.561+0000 7f92ff9be740 1 -- 172.21.15.96:0/3995433226 --> [v2:172.21.15.123:6834/76143454,v1:172.21.15.123:6836/76143454] -- command(tid 0: {"prefix": "status"}) -- 0x5628dfa103a0 con 0x5628dfae3c80 2024-05-28T06:14:59.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.561+0000 7f92f5e00640 1 --2- 172.21.15.96:0/3995433226 >> [v2:172.21.15.123:6834/76143454,v1:172.21.15.123:6836/76143454] conn(0x5628dfae3c80 0x5628dfae6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:59.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.562+0000 7f92f5e00640 1 --2- 172.21.15.96:0/3995433226 >> [v2:172.21.15.123:6834/76143454,v1:172.21.15.123:6836/76143454] conn(0x5628dfae3c80 0x5628dfae6310 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e137bf8121201771 server_cookie=b348050cad69a033 in_seq=0 out_seq=0 2024-05-28T06:14:59.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.562+0000 7f92d6a00640 10 client.7296 ms_handle_connect on v2:172.21.15.123:6834/76143454 2024-05-28T06:14:59.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.563+0000 7f92d6a00640 1 -- 172.21.15.96:0/3995433226 <== mds.0 v2:172.21.15.123:6834/76143454 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x5628dfa103a0 con 0x5628dfae3c80 2024-05-28T06:14:59.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.563+0000 7f92d6a00640 10 client.7296 handle_command_reply: tid=0 2024-05-28T06:14:59.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.563+0000 7f92ff9be740 1 client.7296 shutdown 2024-05-28T06:14:59.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.563+0000 7f92ff9be740 1 -- 172.21.15.96:0/3995433226 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5628dfaeb5d0 msgr2=0x5628dfb0d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:59.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.563+0000 7f92ff9be740 1 --2- 172.21.15.96:0/3995433226 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5628dfaeb5d0 0x5628dfb0d4a0 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f92e4021040 tx=0x7f92e40046c0 comp rx=0 tx=0).stop 2024-05-28T06:14:59.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.563+0000 7f92ff9be740 1 -- 172.21.15.96:0/3995433226 shutdown_connections 2024-05-28T06:14:59.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.563+0000 7f92ff9be740 1 --2- 172.21.15.96:0/3995433226 >> [v2:172.21.15.123:6834/76143454,v1:172.21.15.123:6836/76143454] conn(0x5628dfae3c80 0x5628dfae6310 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:59.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.563+0000 7f92ff9be740 1 --2- 172.21.15.96:0/3995433226 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5628dfaf9a80 0x5628dfb0d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:59.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.563+0000 7f92ff9be740 1 --2- 172.21.15.96:0/3995433226 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5628dfaeb5d0 0x5628dfb0d4a0 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:59.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.563+0000 7f92ff9be740 1 --2- 172.21.15.96:0/3995433226 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x5628dfacf150 0x5628dfad1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:59.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.563+0000 7f92d6a00640 0 client.7296 ms_handle_reset on v2:172.21.15.123:6834/76143454 2024-05-28T06:14:59.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.563+0000 7f92ff9be740 1 -- 172.21.15.96:0/3995433226 >> 172.21.15.96:0/3995433226 conn(0x5628dfacb490 msgr2=0x5628dfac6f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:59.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.563+0000 7f92ff9be740 1 -- 172.21.15.96:0/3995433226 shutdown_connections 2024-05-28T06:14:59.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.563+0000 7f92ff9be740 1 -- 172.21.15.96:0/3995433226 wait complete. 2024-05-28T06:14:59.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.563+0000 7f92ff9be740 20 client.7296 trim_cache size 0 max 16384 2024-05-28T06:14:59.568 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:14:59.568 INFO:teuthology.orchestra.run.smithi096.stdout: "cluster_fsid": "e4a831b5-e6d1-47a7-96cb-bf85154ee0d2", 2024-05-28T06:14:59.568 INFO:teuthology.orchestra.run.smithi096.stdout: "whoami": 0, 2024-05-28T06:14:59.568 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 7146, 2024-05-28T06:14:59.568 INFO:teuthology.orchestra.run.smithi096.stdout: "want_state": "up:active", 2024-05-28T06:14:59.569 INFO:teuthology.orchestra.run.smithi096.stdout: "state": "up:active", 2024-05-28T06:14:59.569 INFO:teuthology.orchestra.run.smithi096.stdout: "fs_name": "cephfs", 2024-05-28T06:14:59.569 INFO:teuthology.orchestra.run.smithi096.stdout: "rank_uptime": 6.6739334179999998, 2024-05-28T06:14:59.569 INFO:teuthology.orchestra.run.smithi096.stdout: "mdsmap_epoch": 43, 2024-05-28T06:14:59.569 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap_epoch": 74, 2024-05-28T06:14:59.569 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap_epoch_barrier": 74, 2024-05-28T06:14:59.569 INFO:teuthology.orchestra.run.smithi096.stdout: "uptime": 13.136160841000001 2024-05-28T06:14:59.569 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:14:59.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.564+0000 7f92f7200640 1 -- 172.21.15.96:0/71621783 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f92d806d5e0 msgr2=0x7f92d806faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:59.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.564+0000 7f92f7200640 1 --2- 172.21.15.96:0/71621783 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f92d806d5e0 0x7f92d806faa0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f92e4006fd0 tx=0x7f92e4008040 comp rx=0 tx=0).stop 2024-05-28T06:14:59.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.564+0000 7f92f7200640 1 -- 172.21.15.96:0/71621783 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f92f806ee40 msgr2=0x7f92f814c1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:59.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.564+0000 7f92f7200640 1 --2- 172.21.15.96:0/71621783 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f92f806ee40 0x7f92f814c1f0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f92e8000c00 tx=0x7f92e80046c0 comp rx=0 tx=0).stop 2024-05-28T06:14:59.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.564+0000 7f92f7200640 1 -- 172.21.15.96:0/71621783 shutdown_connections 2024-05-28T06:14:59.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.564+0000 7f92f7200640 1 --2- 172.21.15.96:0/71621783 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f92d806d5e0 0x7f92d806faa0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:59.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.564+0000 7f92f7200640 1 --2- 172.21.15.96:0/71621783 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92f8156b30 0x7f92f815beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:59.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.564+0000 7f92f7200640 1 --2- 172.21.15.96:0/71621783 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f92f806f2c0 0x7f92f815b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:59.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.564+0000 7f92f7200640 1 --2- 172.21.15.96:0/71621783 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f92f806ee40 0x7f92f814c1f0 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:59.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.564+0000 7f92f7200640 1 -- 172.21.15.96:0/71621783 >> 172.21.15.96:0/71621783 conn(0x7f92f80b1fc0 msgr2=0x7f92f80aff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:59.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.564+0000 7f92f7200640 1 -- 172.21.15.96:0/71621783 shutdown_connections 2024-05-28T06:14:59.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.564+0000 7f92f7200640 1 -- 172.21.15.96:0/71621783 wait complete. 2024-05-28T06:14:59.584 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-05-28T06:14:59.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.779+0000 7fa376400640 1 Processor -- start 2024-05-28T06:14:59.781 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.779+0000 7fa376400640 1 -- start start 2024-05-28T06:14:59.781 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.780+0000 7fa376400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fa3700b4b60 0x7fa3700cfaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:59.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.780+0000 7fa376400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa3700d04a0 0x7fa3700d0900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:59.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.780+0000 7fa376400640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3700ced40 con 0x7fa3700b3460 2024-05-28T06:14:59.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.780+0000 7fa376400640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fa3700d0e40 con 0x7fa3700b4b60 2024-05-28T06:14:59.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.780+0000 7fa376400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fa3700b3fe0 con 0x7fa3700d04a0 2024-05-28T06:14:59.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.781+0000 7fa366a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa3700d04a0 0x7fa3700d0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:59.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.781+0000 7fa36fe00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7fa3700b3460 0x7fa3700b38d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.96:58424/0 (socket says 172.21.15.96:58424) 2024-05-28T06:14:59.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.781+0000 7fa36fe00640 1 -- 172.21.15.96:0/4223330489 learned_addr learned my addr 172.21.15.96:0/4223330489 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:14:59.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.781+0000 7fa36f400640 1 --2- 172.21.15.96:0/4223330489 >> v2:172.21.15.123:3300/0 conn(0x7fa3700b4b60 0x7fa3700cfaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:59.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.782+0000 7fa36ea00640 1 -- 172.21.15.96:0/4223330489 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1179247979 0 0) 0x7fa368001110 con 0x7fa3700b3460 2024-05-28T06:14:59.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.782+0000 7fa36ea00640 1 -- 172.21.15.96:0/4223330489 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa354003520 con 0x7fa3700b3460 2024-05-28T06:14:59.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.782+0000 7fa36f400640 1 -- 172.21.15.96:0/4223330489 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa3700d04a0 msgr2=0x7fa3700d0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:14:59.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.782+0000 7fa36f400640 1 --2- 172.21.15.96:0/4223330489 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa3700d04a0 0x7fa3700d0900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:59.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.782+0000 7fa36f400640 1 -- 172.21.15.96:0/4223330489 >> v1:172.21.15.123:6789/0 conn(0x7fa3700b3460 legacy=0x7fa3700b38d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:59.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.782+0000 7fa36f400640 1 -- 172.21.15.96:0/4223330489 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa37015fbb0 con 0x7fa3700b4b60 2024-05-28T06:14:59.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.783+0000 7fa36f400640 1 --2- 172.21.15.96:0/4223330489 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa3700b4b60 0x7fa3700cfaa0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fa360009d90 tx=0x7fa36002a660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8936c371afd9eb99 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:59.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.784+0000 7fa36ea00640 1 -- 172.21.15.96:0/4223330489 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa360007740 con 0x7fa3700b4b60 2024-05-28T06:14:59.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.784+0000 7fa36ea00640 1 -- 172.21.15.96:0/4223330489 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3600078e0 con 0x7fa3700b4b60 2024-05-28T06:14:59.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.784+0000 7fa36ea00640 1 -- 172.21.15.96:0/4223330489 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa360007c90 con 0x7fa3700b4b60 2024-05-28T06:14:59.786 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.784+0000 7fa376400640 1 -- 172.21.15.96:0/4223330489 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa3700b4b60 msgr2=0x7fa3700cfaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:59.786 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.784+0000 7fa376400640 1 --2- 172.21.15.96:0/4223330489 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa3700b4b60 0x7fa3700cfaa0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fa360009d90 tx=0x7fa36002a660 comp rx=0 tx=0).stop 2024-05-28T06:14:59.786 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.784+0000 7fa376400640 1 -- 172.21.15.96:0/4223330489 shutdown_connections 2024-05-28T06:14:59.786 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.784+0000 7fa376400640 1 --2- 172.21.15.96:0/4223330489 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa3700d04a0 0x7fa3700d0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:59.786 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.784+0000 7fa376400640 1 --2- 172.21.15.96:0/4223330489 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa3700b4b60 0x7fa3700cfaa0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:59.786 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.784+0000 7fa376400640 1 -- 172.21.15.96:0/4223330489 >> 172.21.15.96:0/4223330489 conn(0x7fa37006bae0 msgr2=0x7fa37006bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:14:59.787 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.784+0000 7fa376400640 1 -- 172.21.15.96:0/4223330489 shutdown_connections 2024-05-28T06:14:59.787 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.784+0000 7fa376400640 1 -- 172.21.15.96:0/4223330489 wait complete. 2024-05-28T06:14:59.787 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.786+0000 7fa376400640 1 Processor -- start 2024-05-28T06:14:59.787 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.786+0000 7fa376400640 1 -- start start 2024-05-28T06:14:59.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.787+0000 7fa376400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa3700b3460 0x7fa3701f4420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:59.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.787+0000 7fa376400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa3700b4b60 0x7fa3701f4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:59.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.787+0000 7fa376400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa3700d04a0 0x7fa3701f4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:59.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.787+0000 7fa376400640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa3700cf580 con 0x7fa3700b4b60 2024-05-28T06:14:59.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.787+0000 7fa376400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa3700cf2a0 con 0x7fa3700d04a0 2024-05-28T06:14:59.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.787+0000 7fa376400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fa3700cffe0 con 0x7fa3700b3460 2024-05-28T06:14:59.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.787+0000 7fa366a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa3700b4b60 0x7fa3701f4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:59.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.787+0000 7fa366a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa3700b4b60 0x7fa3701f4960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:52824/0 (socket says 172.21.15.96:52824) 2024-05-28T06:14:59.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.787+0000 7fa366a00640 1 -- 172.21.15.96:0/1262108305 learned_addr learned my addr 172.21.15.96:0/1262108305 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:14:59.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.788+0000 7fa36f400640 1 --2- 172.21.15.96:0/1262108305 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa3700b3460 0x7fa3701f4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:59.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.788+0000 7fa36fe00640 1 --2- 172.21.15.96:0/1262108305 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa3700d04a0 0x7fa3701f4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:59.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.788+0000 7fa366a00640 1 -- 172.21.15.96:0/1262108305 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa3700b3460 msgr2=0x7fa3701f4420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:59.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.788+0000 7fa366a00640 1 --2- 172.21.15.96:0/1262108305 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa3700b3460 0x7fa3701f4420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:59.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.788+0000 7fa366a00640 1 -- 172.21.15.96:0/1262108305 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa3700d04a0 msgr2=0x7fa3701f4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:14:59.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.788+0000 7fa366a00640 1 --2- 172.21.15.96:0/1262108305 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa3700d04a0 0x7fa3701f4ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:14:59.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.788+0000 7fa366a00640 1 -- 172.21.15.96:0/1262108305 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3700b2220 con 0x7fa3700b4b60 2024-05-28T06:14:59.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.788+0000 7fa366a00640 1 --2- 172.21.15.96:0/1262108305 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa3700b4b60 0x7fa3701f4960 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7fa35800e9b0 tx=0x7fa35800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:59.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.789+0000 7fa367e00640 1 -- 172.21.15.96:0/1262108305 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa358017070 con 0x7fa3700b4b60 2024-05-28T06:14:59.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.789+0000 7fa367e00640 1 -- 172.21.15.96:0/1262108305 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa35800c9a0 con 0x7fa3700b4b60 2024-05-28T06:14:59.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.789+0000 7fa367e00640 1 -- 172.21.15.96:0/1262108305 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa35800cd90 con 0x7fa3700b4b60 2024-05-28T06:14:59.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.789+0000 7fa376400640 1 -- 172.21.15.96:0/1262108305 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3700b2500 con 0x7fa3700b4b60 2024-05-28T06:14:59.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.789+0000 7fa376400640 1 -- 172.21.15.96:0/1262108305 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3700cd880 con 0x7fa3700b4b60 2024-05-28T06:14:59.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.790+0000 7fa376400640 1 -- 172.21.15.96:0/1262108305 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3701f6760 con 0x7fa3700b4b60 2024-05-28T06:14:59.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.791+0000 7fa367e00640 1 -- 172.21.15.96:0/1262108305 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 22) ==== 92213+0+0 (secure 0 0 0) 0x7fa35801a070 con 0x7fa3700b4b60 2024-05-28T06:14:59.793 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.792+0000 7fa367e00640 1 --2- 172.21.15.96:0/1262108305 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa34c06d5e0 0x7fa34c06faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:14:59.793 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.792+0000 7fa367e00640 1 -- 172.21.15.96:0/1262108305 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 5737+0+0 (secure 0 0 0) 0x7fa35808a680 con 0x7fa3700b4b60 2024-05-28T06:14:59.794 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.793+0000 7fa36f400640 1 --2- 172.21.15.96:0/1262108305 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa34c06d5e0 0x7fa34c06faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:14:59.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.794+0000 7fa36f400640 1 --2- 172.21.15.96:0/1262108305 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa34c06d5e0 0x7fa34c06faa0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fa360034000 tx=0x7fa36002f000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:14:59.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:14:59.797+0000 7fa367e00640 1 -- 172.21.15.96:0/1262108305 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa35805a500 con 0x7fa3700b4b60 2024-05-28T06:15:00.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:00.043+0000 7fa376400640 1 -- 172.21.15.96:0/1262108305 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fa3700689e0 con 0x7fa3700b4b60 2024-05-28T06:15:00.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:00.044+0000 7fa367e00640 1 -- 172.21.15.96:0/1262108305 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v43) ==== 53+0+73 (secure 0 0 0) 0x7fa358059e00 con 0x7fa3700b4b60 2024-05-28T06:15:00.046 INFO:teuthology.orchestra.run.smithi096.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-05-28T06:15:00.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:00.047+0000 7fa376400640 1 -- 172.21.15.96:0/1262108305 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa34c06d5e0 msgr2=0x7fa34c06faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:00.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:00.047+0000 7fa376400640 1 --2- 172.21.15.96:0/1262108305 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa34c06d5e0 0x7fa34c06faa0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fa360034000 tx=0x7fa36002f000 comp rx=0 tx=0).stop 2024-05-28T06:15:00.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:00.047+0000 7fa376400640 1 -- 172.21.15.96:0/1262108305 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa3700b4b60 msgr2=0x7fa3701f4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:00.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:00.047+0000 7fa376400640 1 --2- 172.21.15.96:0/1262108305 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa3700b4b60 0x7fa3701f4960 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7fa35800e9b0 tx=0x7fa35800ee70 comp rx=0 tx=0).stop 2024-05-28T06:15:00.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:00.047+0000 7fa376400640 1 -- 172.21.15.96:0/1262108305 shutdown_connections 2024-05-28T06:15:00.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:00.047+0000 7fa376400640 1 --2- 172.21.15.96:0/1262108305 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa34c06d5e0 0x7fa34c06faa0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:00.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:00.047+0000 7fa376400640 1 --2- 172.21.15.96:0/1262108305 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa3700d04a0 0x7fa3701f4ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:00.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:00.047+0000 7fa376400640 1 --2- 172.21.15.96:0/1262108305 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa3700b4b60 0x7fa3701f4960 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:00.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:00.047+0000 7fa376400640 1 --2- 172.21.15.96:0/1262108305 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa3700b3460 0x7fa3701f4420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:00.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:00.047+0000 7fa376400640 1 -- 172.21.15.96:0/1262108305 >> 172.21.15.96:0/1262108305 conn(0x7fa37006bae0 msgr2=0x7fa3701605e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:15:00.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:00.047+0000 7fa376400640 1 -- 172.21.15.96:0/1262108305 shutdown_connections 2024-05-28T06:15:00.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:00.047+0000 7fa376400640 1 -- 172.21.15.96:0/1262108305 wait complete. 2024-05-28T06:15:00.068 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-05-28T06:15:00.068 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-28T06:15:00.068 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi096.front.sepia.ceph.com 2024-05-28T06:15:00.068 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-05-28T06:15:00.068 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-28T06:15:00.068 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-05-28T06:15:00.068 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-05-28T06:15:00.068 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-28T06:15:00.068 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-28T06:15:00.068 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:15:00.069 DEBUG:teuthology.orchestra.run.smithi096:> ip addr 2024-05-28T06:15:00.119 INFO:teuthology.orchestra.run.smithi096.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-28T06:15:00.120 INFO:teuthology.orchestra.run.smithi096.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-28T06:15:00.120 INFO:teuthology.orchestra.run.smithi096.stdout: inet 127.0.0.1/8 scope host lo 2024-05-28T06:15:00.120 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:15:00.120 INFO:teuthology.orchestra.run.smithi096.stdout: inet6 ::1/128 scope host 2024-05-28T06:15:00.120 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:15:00.120 INFO:teuthology.orchestra.run.smithi096.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-28T06:15:00.120 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether 0c:c4:7a:d8:d9:bc brd ff:ff:ff:ff:ff:ff 2024-05-28T06:15:00.120 INFO:teuthology.orchestra.run.smithi096.stdout: altname enp7s0f0 2024-05-28T06:15:00.120 INFO:teuthology.orchestra.run.smithi096.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-28T06:15:00.120 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether 0c:c4:7a:d8:d9:bd brd ff:ff:ff:ff:ff:ff 2024-05-28T06:15:00.120 INFO:teuthology.orchestra.run.smithi096.stdout: altname enp7s0f1 2024-05-28T06:15:00.121 INFO:teuthology.orchestra.run.smithi096.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-28T06:15:00.121 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether 0c:c4:7a:8f:cc:0c brd ff:ff:ff:ff:ff:ff 2024-05-28T06:15:00.121 INFO:teuthology.orchestra.run.smithi096.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-28T06:15:00.121 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether 0c:c4:7a:8f:cc:0d brd ff:ff:ff:ff:ff:ff 2024-05-28T06:15:00.121 INFO:teuthology.orchestra.run.smithi096.stdout: inet 172.21.15.96/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-05-28T06:15:00.121 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft 41937sec preferred_lft 41937sec 2024-05-28T06:15:00.121 INFO:teuthology.orchestra.run.smithi096.stdout: inet6 fe80::ec4:7aff:fe8f:cc0d/64 scope link 2024-05-28T06:15:00.121 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:15:00.121 INFO:teuthology.orchestra.run.smithi096.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-05-28T06:15:00.121 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether ae:d9:f5:b0:4f:f3 brd ff:ff:ff:ff:ff:ff 2024-05-28T06:15:00.121 INFO:teuthology.orchestra.run.smithi096.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-05-28T06:15:00.121 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:15:00.121 INFO:teuthology.orchestra.run.smithi096.stdout: inet6 fe80::604e:fcff:fe60:7892/64 scope link 2024-05-28T06:15:00.121 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:15:00.122 INFO:teuthology.orchestra.run.smithi096.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-28T06:15:00.122 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether ce:6f:19:93:44:f9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-28T06:15:00.122 INFO:teuthology.orchestra.run.smithi096.stdout: inet6 fe80::cc6f:19ff:fe93:44f9/64 scope link 2024-05-28T06:15:00.122 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:15:00.122 INFO:teuthology.orchestra.run.smithi096.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-28T06:15:00.122 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether ae:d9:f5:b0:4f:f3 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-28T06:15:00.122 INFO:teuthology.orchestra.run.smithi096.stdout: inet6 fe80::acd9:f5ff:feb0:4ff3/64 scope link 2024-05-28T06:15:00.122 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:15:00.123 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:15:00.123 DEBUG:teuthology.orchestra.run.smithi096:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-28T06:15:00.203 INFO:teuthology.orchestra.run.smithi096.stdout:1 2024-05-28T06:15:00.207 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:15:00.207 DEBUG:teuthology.orchestra.run.smithi096:> route 2024-05-28T06:15:00.257 INFO:teuthology.orchestra.run.smithi096.stdout:Kernel IP routing table 2024-05-28T06:15:00.258 INFO:teuthology.orchestra.run.smithi096.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-28T06:15:00.258 INFO:teuthology.orchestra.run.smithi096.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-05-28T06:15:00.258 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-05-28T06:15:00.258 INFO:teuthology.orchestra.run.smithi096.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-28T06:15:00.258 DEBUG:teuthology.orchestra.run.smithi096:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-28T06:15:00.258 DEBUG:teuthology.orchestra.run.smithi096:> set -e 2024-05-28T06:15:00.259 DEBUG:teuthology.orchestra.run.smithi096:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-05-28T06:15:00.259 DEBUG:teuthology.orchestra.run.smithi096:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-05-28T06:15:00.259 DEBUG:teuthology.orchestra.run.smithi096:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-05-28T06:15:00.259 DEBUG:teuthology.orchestra.run.smithi096:> ') 2024-05-28T06:15:00.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:00 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-28T06:15:00.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:00 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-28T06:15:00.456 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:15:00.457 DEBUG:teuthology.orchestra.run.smithi096:> ip netns list 2024-05-28T06:15:00.502 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-05-28T06:15:00.502 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-05-28T06:15:00.503 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-28T06:15:00.503 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:15:00.503 DEBUG:teuthology.orchestra.run.smithi096:> sudo ip link set brx.0 up 2024-05-28T06:15:00.632 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-05-28T06:15:00.632 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-28T06:15:00.632 DEBUG:teuthology.orchestra.run.smithi096:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-05-28T06:15:00.683 INFO:teuthology.orchestra.run.smithi096.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-05-28T06:15:00.684 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-28T06:15:00.684 DEBUG:teuthology.orchestra.run.smithi096:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-05-28T06:15:00.735 DEBUG:teuthology.orchestra.run.smithi096:> sudo modprobe fuse 2024-05-28T06:15:00.813 DEBUG:teuthology.orchestra.run.smithi096:> cat /proc/self/mounts | awk '{print $2}' 2024-05-28T06:15:00.862 INFO:teuthology.orchestra.run.smithi096.stdout:/proc 2024-05-28T06:15:00.862 INFO:teuthology.orchestra.run.smithi096.stdout:/sys 2024-05-28T06:15:00.862 INFO:teuthology.orchestra.run.smithi096.stdout:/dev 2024-05-28T06:15:00.862 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/security 2024-05-28T06:15:00.862 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/shm 2024-05-28T06:15:00.862 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/pts 2024-05-28T06:15:00.862 INFO:teuthology.orchestra.run.smithi096.stdout:/run 2024-05-28T06:15:00.862 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/cgroup 2024-05-28T06:15:00.862 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/pstore 2024-05-28T06:15:00.862 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/bpf 2024-05-28T06:15:00.862 INFO:teuthology.orchestra.run.smithi096.stdout:/ 2024-05-28T06:15:00.863 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/selinux 2024-05-28T06:15:00.863 INFO:teuthology.orchestra.run.smithi096.stdout:/proc/sys/fs/binfmt_misc 2024-05-28T06:15:00.863 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/mqueue 2024-05-28T06:15:00.863 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/debug 2024-05-28T06:15:00.863 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/hugepages 2024-05-28T06:15:00.863 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/tracing 2024-05-28T06:15:00.863 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/fuse/connections 2024-05-28T06:15:00.863 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/config 2024-05-28T06:15:00.863 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-sysctl.service 2024-05-28T06:15:00.863 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-28T06:15:00.863 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-28T06:15:00.863 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/nfs/rpc_pipefs 2024-05-28T06:15:00.863 INFO:teuthology.orchestra.run.smithi096.stdout:/run/user/1000 2024-05-28T06:15:00.864 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph 2024-05-28T06:15:00.864 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-0 2024-05-28T06:15:00.864 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-1 2024-05-28T06:15:00.864 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-2 2024-05-28T06:15:00.864 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-3 2024-05-28T06:15:00.864 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns 2024-05-28T06:15:00.864 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-28T06:15:00.864 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-28T06:15:00.864 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-28T06:15:00.864 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-28T06:15:00.865 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:15:00.865 DEBUG:teuthology.orchestra.run.smithi096:> ls /sys/fs/fuse/connections 2024-05-28T06:15:00.915 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-05-28T06:15:00.915 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-05-28T06:15:00.923 DEBUG:teuthology.orchestra.run.smithi096:> sudo modprobe fuse 2024-05-28T06:15:00.991 DEBUG:teuthology.orchestra.run.smithi096:> cat /proc/self/mounts | awk '{print $2}' 2024-05-28T06:15:01.032 INFO:teuthology.orchestra.run.smithi096.stdout:/proc 2024-05-28T06:15:01.032 INFO:teuthology.orchestra.run.smithi096.stdout:/sys 2024-05-28T06:15:01.032 INFO:teuthology.orchestra.run.smithi096.stdout:/dev 2024-05-28T06:15:01.032 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/security 2024-05-28T06:15:01.033 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/shm 2024-05-28T06:15:01.033 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/pts 2024-05-28T06:15:01.033 INFO:teuthology.orchestra.run.smithi096.stdout:/run 2024-05-28T06:15:01.033 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/cgroup 2024-05-28T06:15:01.033 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/pstore 2024-05-28T06:15:01.033 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/bpf 2024-05-28T06:15:01.033 INFO:teuthology.orchestra.run.smithi096.stdout:/ 2024-05-28T06:15:01.033 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/selinux 2024-05-28T06:15:01.033 INFO:teuthology.orchestra.run.smithi096.stdout:/proc/sys/fs/binfmt_misc 2024-05-28T06:15:01.033 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/mqueue 2024-05-28T06:15:01.033 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/debug 2024-05-28T06:15:01.033 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/hugepages 2024-05-28T06:15:01.033 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/tracing 2024-05-28T06:15:01.033 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/fuse/connections 2024-05-28T06:15:01.034 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/config 2024-05-28T06:15:01.034 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-sysctl.service 2024-05-28T06:15:01.034 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-28T06:15:01.034 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-28T06:15:01.034 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/nfs/rpc_pipefs 2024-05-28T06:15:01.034 INFO:teuthology.orchestra.run.smithi096.stdout:/run/user/1000 2024-05-28T06:15:01.034 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph 2024-05-28T06:15:01.034 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-0 2024-05-28T06:15:01.034 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-1 2024-05-28T06:15:01.034 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-2 2024-05-28T06:15:01.034 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-3 2024-05-28T06:15:01.034 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns 2024-05-28T06:15:01.034 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-28T06:15:01.034 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-28T06:15:01.035 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-28T06:15:01.035 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-28T06:15:01.035 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:15:01.035 DEBUG:teuthology.orchestra.run.smithi096:> ls /sys/fs/fuse/connections 2024-05-28T06:15:01.094 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi096.stderr:2024-05-28T06:15:01.093+0000 7fe254010480 -1 init, newargv = 0x5652a69cbe70 newargc=15 2024-05-28T06:15:01.095 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi096.stderr:ceph-fuse[64991]: starting ceph client 2024-05-28T06:15:01.103 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi096.stderr:ceph-fuse[64991]: starting fuse 2024-05-28T06:15:01.120 INFO:teuthology.orchestra.run.smithi096.stdout:40 2024-05-28T06:15:01.120 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40] 2024-05-28T06:15:01.121 DEBUG:teuthology.orchestra.run.smithi096:> sudo stdin-killer -- python3 -c ' 2024-05-28T06:15:01.121 DEBUG:teuthology.orchestra.run.smithi096:> import glob 2024-05-28T06:15:01.121 DEBUG:teuthology.orchestra.run.smithi096:> import re 2024-05-28T06:15:01.121 DEBUG:teuthology.orchestra.run.smithi096:> import os 2024-05-28T06:15:01.121 DEBUG:teuthology.orchestra.run.smithi096:> import subprocess 2024-05-28T06:15:01.121 DEBUG:teuthology.orchestra.run.smithi096:> 2024-05-28T06:15:01.121 DEBUG:teuthology.orchestra.run.smithi096:> def _find_admin_socket(client_name): 2024-05-28T06:15:01.121 DEBUG:teuthology.orchestra.run.smithi096:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-05-28T06:15:01.121 DEBUG:teuthology.orchestra.run.smithi096:> files = glob.glob(asok_path) 2024-05-28T06:15:01.121 DEBUG:teuthology.orchestra.run.smithi096:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-05-28T06:15:01.121 DEBUG:teuthology.orchestra.run.smithi096:> 2024-05-28T06:15:01.122 DEBUG:teuthology.orchestra.run.smithi096:> # Given a non-glob path, it better be there 2024-05-28T06:15:01.122 DEBUG:teuthology.orchestra.run.smithi096:> if "*" not in asok_path: 2024-05-28T06:15:01.122 DEBUG:teuthology.orchestra.run.smithi096:> assert(len(files) == 1) 2024-05-28T06:15:01.122 DEBUG:teuthology.orchestra.run.smithi096:> return files[0] 2024-05-28T06:15:01.122 DEBUG:teuthology.orchestra.run.smithi096:> 2024-05-28T06:15:01.122 DEBUG:teuthology.orchestra.run.smithi096:> for f in files: 2024-05-28T06:15:01.122 DEBUG:teuthology.orchestra.run.smithi096:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-05-28T06:15:01.122 DEBUG:teuthology.orchestra.run.smithi096:> if os.path.exists("/proc/{0}".format(pid)): 2024-05-28T06:15:01.122 DEBUG:teuthology.orchestra.run.smithi096:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-05-28T06:15:01.122 DEBUG:teuthology.orchestra.run.smithi096:> contents = proc_f.read() 2024-05-28T06:15:01.122 DEBUG:teuthology.orchestra.run.smithi096:> if mountpoint in contents: 2024-05-28T06:15:01.122 DEBUG:teuthology.orchestra.run.smithi096:> return f 2024-05-28T06:15:01.122 DEBUG:teuthology.orchestra.run.smithi096:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-05-28T06:15:01.122 DEBUG:teuthology.orchestra.run.smithi096:> 2024-05-28T06:15:01.123 DEBUG:teuthology.orchestra.run.smithi096:> print(_find_admin_socket("client.0")) 2024-05-28T06:15:01.123 DEBUG:teuthology.orchestra.run.smithi096:> ' 2024-05-28T06:15:01.310 INFO:teuthology.orchestra.run.smithi096.stdout:/var/run/ceph/ceph-client.0.64991.asok 2024-05-28T06:15:01.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-28T06:15:01.326 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.64991.asok 2024-05-28T06:15:01.327 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:15:01.327 DEBUG:teuthology.orchestra.run.smithi096:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.64991.asok status 2024-05-28T06:15:01.549 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:15:01.549 INFO:teuthology.orchestra.run.smithi096.stdout: "metadata": { 2024-05-28T06:15:01.549 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph_sha1": "9ed025fbf4406fb54980a0647506b15895b6d863", 2024-05-28T06:15:01.549 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph_version": "ceph version 19.0.0-3154-g9ed025fb (9ed025fbf4406fb54980a0647506b15895b6d863) squid (dev)", 2024-05-28T06:15:01.549 INFO:teuthology.orchestra.run.smithi096.stdout: "entity_id": "0", 2024-05-28T06:15:01.549 INFO:teuthology.orchestra.run.smithi096.stdout: "hostname": "smithi096", 2024-05-28T06:15:01.549 INFO:teuthology.orchestra.run.smithi096.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-05-28T06:15:01.549 INFO:teuthology.orchestra.run.smithi096.stdout: "pid": "64991", 2024-05-28T06:15:01.549 INFO:teuthology.orchestra.run.smithi096.stdout: "root": "/" 2024-05-28T06:15:01.549 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:15:01.550 INFO:teuthology.orchestra.run.smithi096.stdout: "dentry_count": 0, 2024-05-28T06:15:01.550 INFO:teuthology.orchestra.run.smithi096.stdout: "dentry_pinned_count": 0, 2024-05-28T06:15:01.550 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 7308, 2024-05-28T06:15:01.550 INFO:teuthology.orchestra.run.smithi096.stdout: "inst": { 2024-05-28T06:15:01.550 INFO:teuthology.orchestra.run.smithi096.stdout: "name": { 2024-05-28T06:15:01.550 INFO:teuthology.orchestra.run.smithi096.stdout: "type": "client", 2024-05-28T06:15:01.550 INFO:teuthology.orchestra.run.smithi096.stdout: "num": 7308 2024-05-28T06:15:01.550 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:15:01.550 INFO:teuthology.orchestra.run.smithi096.stdout: "addr": { 2024-05-28T06:15:01.550 INFO:teuthology.orchestra.run.smithi096.stdout: "type": "v1", 2024-05-28T06:15:01.550 INFO:teuthology.orchestra.run.smithi096.stdout: "addr": "192.168.0.1:0", 2024-05-28T06:15:01.550 INFO:teuthology.orchestra.run.smithi096.stdout: "nonce": 3439296019 2024-05-28T06:15:01.550 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:15:01.550 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:15:01.551 INFO:teuthology.orchestra.run.smithi096.stdout: "addr": { 2024-05-28T06:15:01.551 INFO:teuthology.orchestra.run.smithi096.stdout: "type": "v1", 2024-05-28T06:15:01.551 INFO:teuthology.orchestra.run.smithi096.stdout: "addr": "192.168.0.1:0", 2024-05-28T06:15:01.551 INFO:teuthology.orchestra.run.smithi096.stdout: "nonce": 3439296019 2024-05-28T06:15:01.551 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:15:01.551 INFO:teuthology.orchestra.run.smithi096.stdout: "inst_str": "client.7308 192.168.0.1:0/3439296019", 2024-05-28T06:15:01.551 INFO:teuthology.orchestra.run.smithi096.stdout: "addr_str": "192.168.0.1:0/3439296019", 2024-05-28T06:15:01.551 INFO:teuthology.orchestra.run.smithi096.stdout: "inode_count": 1, 2024-05-28T06:15:01.551 INFO:teuthology.orchestra.run.smithi096.stdout: "mds_epoch": 43, 2024-05-28T06:15:01.551 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_epoch": 74, 2024-05-28T06:15:01.551 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_epoch_barrier": 0, 2024-05-28T06:15:01.551 INFO:teuthology.orchestra.run.smithi096.stdout: "blocklisted": false, 2024-05-28T06:15:01.551 INFO:teuthology.orchestra.run.smithi096.stdout: "fs_name": "cephfs" 2024-05-28T06:15:01.551 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:15:01.563 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:15:01.563 DEBUG:teuthology.orchestra.run.smithi096:> stat --file-system '--printf=%T 2024-05-28T06:15:01.563 DEBUG:teuthology.orchestra.run.smithi096:> ' -- /home/ubuntu/cephtest/mnt.0 2024-05-28T06:15:01.616 INFO:teuthology.orchestra.run.smithi096.stdout:fuseblk 2024-05-28T06:15:01.617 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-05-28T06:15:01.617 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:15:01.617 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-05-28T06:15:01.702 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-05-28T06:15:01.885 DEBUG:teuthology.orchestra.run.smithi096:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:15:01.891 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:15:01.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:01.908+0000 7f9877400640 1 Processor -- start 2024-05-28T06:15:01.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:01.908+0000 7f9877400640 1 -- start start 2024-05-28T06:15:01.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:01.909+0000 7f9877400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f98700cf2f0 0x7f98700cf910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:01.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:01.909+0000 7f9877400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f9870140500 0x7f98700ad130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:01.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:01.910+0000 7f9877400640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f987006dd50 con 0x7f987006edf0 2024-05-28T06:15:01.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:01.910+0000 7f9877400640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f987006e030 con 0x7f9870140500 2024-05-28T06:15:01.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:01.910+0000 7f9877400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f987006e310 con 0x7f98700cf2f0 2024-05-28T06:15:01.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:01.910+0000 7f9874a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f98700cf2f0 0x7f98700cf910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:01.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:01.910+0000 7f9874a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f98700cf2f0 0x7f98700cf910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.96:55574/0 (socket says 172.21.15.96:55574) 2024-05-28T06:15:01.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:01.910+0000 7f9874a00640 1 -- 172.21.15.96:0/1786958357 learned_addr learned my addr 172.21.15.96:0/1786958357 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:15:01.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:01.910+0000 7f9875400640 1 --2- 172.21.15.96:0/1786958357 >> v2:172.21.15.123:3300/0 conn(0x7f9870140500 0x7f98700ad130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:01.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:01.911+0000 7f9875400640 1 -- 172.21.15.96:0/1786958357 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f98700cf2f0 msgr2=0x7f98700cf910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:15:01.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:01.911+0000 7f9875400640 1 --2- 172.21.15.96:0/1786958357 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f98700cf2f0 0x7f98700cf910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:01.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:01.911+0000 7f9875400640 1 -- 172.21.15.96:0/1786958357 >> v1:172.21.15.123:6789/0 conn(0x7f987006edf0 legacy=0x7f987013fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:01.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:01.911+0000 7f9875400640 1 -- 172.21.15.96:0/1786958357 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f987006e5f0 con 0x7f9870140500 2024-05-28T06:15:01.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:01.911+0000 7f9874a00640 1 --2- 172.21.15.96:0/1786958357 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f98700cf2f0 0x7f98700cf910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-28T06:15:01.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:01.912+0000 7f9875400640 1 --2- 172.21.15.96:0/1786958357 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9870140500 0x7f98700ad130 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f9868009d90 tx=0x7f986802a5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=97aaf420525ba74 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:15:01.914 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:01.913+0000 7f986fe00640 1 -- 172.21.15.96:0/1786958357 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9868031600 con 0x7f9870140500 2024-05-28T06:15:01.914 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:01.913+0000 7f986fe00640 1 -- 172.21.15.96:0/1786958357 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98680317a0 con 0x7f9870140500 2024-05-28T06:15:01.914 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:01.913+0000 7f986fe00640 1 -- 172.21.15.96:0/1786958357 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9868031b70 con 0x7f9870140500 2024-05-28T06:15:01.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:01.914+0000 7f9877400640 1 -- 172.21.15.96:0/1786958357 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9870140500 msgr2=0x7f98700ad130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:01.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:01.914+0000 7f9877400640 1 --2- 172.21.15.96:0/1786958357 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9870140500 0x7f98700ad130 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f9868009d90 tx=0x7f986802a5d0 comp rx=0 tx=0).stop 2024-05-28T06:15:01.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:01.914+0000 7f9877400640 1 -- 172.21.15.96:0/1786958357 shutdown_connections 2024-05-28T06:15:01.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:01.914+0000 7f9877400640 1 --2- 172.21.15.96:0/1786958357 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9870140500 0x7f98700ad130 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:01.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:01.914+0000 7f9877400640 1 --2- 172.21.15.96:0/1786958357 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f98700cf2f0 0x7f98700cf910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:01.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:01.914+0000 7f9877400640 1 -- 172.21.15.96:0/1786958357 >> 172.21.15.96:0/1786958357 conn(0x7f98700b1e50 msgr2=0x7f98700b2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:15:01.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:01.914+0000 7f9877400640 1 -- 172.21.15.96:0/1786958357 shutdown_connections 2024-05-28T06:15:01.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:01.914+0000 7f9877400640 1 -- 172.21.15.96:0/1786958357 wait complete. 2024-05-28T06:15:01.917 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:01.916+0000 7f9877400640 1 Processor -- start 2024-05-28T06:15:01.917 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:01.916+0000 7f9877400640 1 -- start start 2024-05-28T06:15:01.919 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:01.917+0000 7f9877400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f987006edf0 0x7f98700b3f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:01.919 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:01.917+0000 7f9877400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f98700cf2f0 0x7f9870143fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:01.919 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:01.917+0000 7f9877400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f98700b4490 0x7f9870144500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:01.919 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:01.917+0000 7f9877400640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f98701461a0 con 0x7f987006edf0 2024-05-28T06:15:01.919 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:01.917+0000 7f9877400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f9870146020 con 0x7f98700cf2f0 2024-05-28T06:15:01.919 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:01.917+0000 7f9877400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f98700cfe50 con 0x7f98700b4490 2024-05-28T06:15:01.919 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:01.917+0000 7f9874a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f987006edf0 0x7f98700b3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:01.919 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:01.917+0000 7f9874a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f987006edf0 0x7f98700b3f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:52838/0 (socket says 172.21.15.96:52838) 2024-05-28T06:15:01.919 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:01.917+0000 7f9874a00640 1 -- 172.21.15.96:0/1194989345 learned_addr learned my addr 172.21.15.96:0/1194989345 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:15:01.920 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:01.917+0000 7f9867e00640 1 --2- 172.21.15.96:0/1194989345 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f98700cf2f0 0x7f9870143fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:01.920 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:01.917+0000 7f9875400640 1 --2- 172.21.15.96:0/1194989345 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f98700b4490 0x7f9870144500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:01.920 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:01.918+0000 7f9874a00640 1 -- 172.21.15.96:0/1194989345 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f98700b4490 msgr2=0x7f9870144500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:01.920 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:01.918+0000 7f9874a00640 1 --2- 172.21.15.96:0/1194989345 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f98700b4490 0x7f9870144500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:01.920 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:01.918+0000 7f9874a00640 1 -- 172.21.15.96:0/1194989345 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f98700cf2f0 msgr2=0x7f9870143fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:01.920 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:01.918+0000 7f9874a00640 1 --2- 172.21.15.96:0/1194989345 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f98700cf2f0 0x7f9870143fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:01.920 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:01.918+0000 7f9874a00640 1 -- 172.21.15.96:0/1194989345 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98700b3190 con 0x7f987006edf0 2024-05-28T06:15:01.921 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:01.918+0000 7f9874a00640 1 --2- 172.21.15.96:0/1194989345 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f987006edf0 0x7f98700b3f50 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f986000ea00 tx=0x7f986000eec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:15:01.921 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:01.918+0000 7f986d600640 1 -- 172.21.15.96:0/1194989345 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9860017070 con 0x7f987006edf0 2024-05-28T06:15:01.921 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:01.918+0000 7f986d600640 1 -- 172.21.15.96:0/1194989345 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f986000ca60 con 0x7f987006edf0 2024-05-28T06:15:01.921 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:01.918+0000 7f986d600640 1 -- 172.21.15.96:0/1194989345 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f986000ce10 con 0x7f987006edf0 2024-05-28T06:15:01.921 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:01.918+0000 7f9877400640 1 -- 172.21.15.96:0/1194989345 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98700b3470 con 0x7f987006edf0 2024-05-28T06:15:01.921 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:01.918+0000 7f9877400640 1 -- 172.21.15.96:0/1194989345 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98700b3a30 con 0x7f987006edf0 2024-05-28T06:15:01.921 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:01.920+0000 7f9877400640 1 -- 172.21.15.96:0/1194989345 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98700689e0 con 0x7f987006edf0 2024-05-28T06:15:01.922 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:01.921+0000 7f986d600640 1 -- 172.21.15.96:0/1194989345 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 22) ==== 92213+0+0 (secure 0 0 0) 0x7f986001a070 con 0x7f987006edf0 2024-05-28T06:15:01.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:01.922+0000 7f986d600640 1 --2- 172.21.15.96:0/1194989345 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f984c06d5e0 0x7f984c06faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:01.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:01.922+0000 7f986d600640 1 -- 172.21.15.96:0/1194989345 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 5737+0+0 (secure 0 0 0) 0x7f986008a8e0 con 0x7f987006edf0 2024-05-28T06:15:01.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:01.922+0000 7f9867e00640 1 --2- 172.21.15.96:0/1194989345 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f984c06d5e0 0x7f984c06faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:01.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:01.923+0000 7f9867e00640 1 --2- 172.21.15.96:0/1194989345 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f984c06d5e0 0x7f984c06faa0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f9870144c10 tx=0x7f9858009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:15:01.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:01.926+0000 7f986d600640 1 -- 172.21.15.96:0/1194989345 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f986005a050 con 0x7f987006edf0 2024-05-28T06:15:02.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:02.189+0000 7f9877400640 1 -- 172.21.15.96:0/1194989345 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) -- 0x7f9870073f30 con 0x7f987006edf0 2024-05-28T06:15:02.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:02.979+0000 7f986d600640 1 -- 172.21.15.96:0/1194989345 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v44) ==== 136+0+0 (secure 0 0 0) 0x7f986005e350 con 0x7f987006edf0 2024-05-28T06:15:02.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:02.982+0000 7f9877400640 1 -- 172.21.15.96:0/1194989345 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f984c06d5e0 msgr2=0x7f984c06faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:02.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:02.982+0000 7f9877400640 1 --2- 172.21.15.96:0/1194989345 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f984c06d5e0 0x7f984c06faa0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f9870144c10 tx=0x7f9858009040 comp rx=0 tx=0).stop 2024-05-28T06:15:02.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:02.982+0000 7f9877400640 1 -- 172.21.15.96:0/1194989345 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f987006edf0 msgr2=0x7f98700b3f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:02.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:02.982+0000 7f9877400640 1 --2- 172.21.15.96:0/1194989345 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f987006edf0 0x7f98700b3f50 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f986000ea00 tx=0x7f986000eec0 comp rx=0 tx=0).stop 2024-05-28T06:15:02.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:02.982+0000 7f9877400640 1 -- 172.21.15.96:0/1194989345 shutdown_connections 2024-05-28T06:15:02.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:02.982+0000 7f9877400640 1 --2- 172.21.15.96:0/1194989345 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f984c06d5e0 0x7f984c06faa0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:02.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:02.982+0000 7f9877400640 1 --2- 172.21.15.96:0/1194989345 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f98700b4490 0x7f9870144500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:02.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:02.982+0000 7f9877400640 1 --2- 172.21.15.96:0/1194989345 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f98700cf2f0 0x7f9870143fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:02.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:02.982+0000 7f9877400640 1 --2- 172.21.15.96:0/1194989345 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f987006edf0 0x7f98700b3f50 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:02.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:02.983+0000 7f9877400640 1 -- 172.21.15.96:0/1194989345 >> 172.21.15.96:0/1194989345 conn(0x7f98700b1e50 msgr2=0x7f98700ce3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:15:02.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:02.983+0000 7f9877400640 1 -- 172.21.15.96:0/1194989345 shutdown_connections 2024-05-28T06:15:02.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:02.983+0000 7f9877400640 1 -- 172.21.15.96:0/1194989345 wait complete. 2024-05-28T06:15:03.005 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-05-28T06:15:03.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.206+0000 7f5dafe00640 1 Processor -- start 2024-05-28T06:15:03.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.207+0000 7f5dafe00640 1 -- start start 2024-05-28T06:15:03.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.208+0000 7f5dafe00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f5db015e870 0x7f5db00d0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:03.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.208+0000 7f5dafe00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5db015ec50 con 0x7f5db00cfa40 2024-05-28T06:15:03.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.208+0000 7f5dafe00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5db00cff80 con 0x7f5db00d14d0 2024-05-28T06:15:03.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.208+0000 7f5dafe00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f5db00d2150 con 0x7f5db015e870 2024-05-28T06:15:03.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.208+0000 7f5daea00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f5db015e870 0x7f5db00d0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:03.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.209+0000 7f5daea00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f5db015e870 0x7f5db00d0f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:52844/0 (socket says 172.21.15.96:52844) 2024-05-28T06:15:03.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.209+0000 7f5daea00640 1 -- 172.21.15.96:0/3233191376 learned_addr learned my addr 172.21.15.96:0/3233191376 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:15:03.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.209+0000 7f5daf400640 1 --1- 172.21.15.96:0/3233191376 >> v1:172.21.15.96:6789/0 conn(0x7f5db00cfa40 0x7f5db00d25e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:41922/0 (socket says 172.21.15.96:41922) 2024-05-28T06:15:03.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.209+0000 7f5daea00640 1 -- 172.21.15.96:0/3233191376 >> v1:172.21.15.123:6789/0 conn(0x7f5db00d14d0 legacy=0x7f5db00cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:03.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.209+0000 7f5daea00640 1 -- 172.21.15.96:0/3233191376 >> v1:172.21.15.96:6789/0 conn(0x7f5db00cfa40 legacy=0x7f5db00d25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:03.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.209+0000 7f5daea00640 1 -- 172.21.15.96:0/3233191376 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5db00d0260 con 0x7f5db015e870 2024-05-28T06:15:03.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.209+0000 7f5dad600640 1 -- 172.21.15.96:0/3233191376 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1184369503 0 0) 0x7f5db015ec50 con 0x7f5db00cfa40 2024-05-28T06:15:03.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.210+0000 7f5daea00640 1 --2- 172.21.15.96:0/3233191376 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5db015e870 0x7f5db00d0f90 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f5d98009870 tx=0x7f5d9802a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8af09dc54a727cf server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:15:03.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.210+0000 7f5dad600640 1 -- 172.21.15.96:0/3233191376 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d98037070 con 0x7f5db015e870 2024-05-28T06:15:03.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.211+0000 7f5dad600640 1 -- 172.21.15.96:0/3233191376 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d9802ae50 con 0x7f5db015e870 2024-05-28T06:15:03.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.211+0000 7f5dad600640 1 -- 172.21.15.96:0/3233191376 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d98032860 con 0x7f5db015e870 2024-05-28T06:15:03.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.211+0000 7f5dafe00640 1 -- 172.21.15.96:0/3233191376 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5db015e870 msgr2=0x7f5db00d0f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:03.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.211+0000 7f5dafe00640 1 --2- 172.21.15.96:0/3233191376 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5db015e870 0x7f5db00d0f90 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f5d98009870 tx=0x7f5d9802a300 comp rx=0 tx=0).stop 2024-05-28T06:15:03.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.211+0000 7f5dafe00640 1 -- 172.21.15.96:0/3233191376 shutdown_connections 2024-05-28T06:15:03.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.211+0000 7f5dafe00640 1 --2- 172.21.15.96:0/3233191376 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5db015e870 0x7f5db00d0f90 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:03.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.211+0000 7f5dafe00640 1 -- 172.21.15.96:0/3233191376 >> 172.21.15.96:0/3233191376 conn(0x7f5db008bfc0 msgr2=0x7f5db00d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:15:03.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.211+0000 7f5dafe00640 1 -- 172.21.15.96:0/3233191376 shutdown_connections 2024-05-28T06:15:03.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.211+0000 7f5dafe00640 1 -- 172.21.15.96:0/3233191376 wait complete. 2024-05-28T06:15:03.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.213+0000 7f5dafe00640 1 Processor -- start 2024-05-28T06:15:03.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.213+0000 7f5dafe00640 1 -- start start 2024-05-28T06:15:03.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.214+0000 7f5dafe00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5db00cfa40 0x7f5db00b5610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:03.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.214+0000 7f5dafe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5db00d14d0 0x7f5db015a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:03.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.214+0000 7f5dafe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5db015e870 0x7f5db015a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:03.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.214+0000 7f5dafe00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f5db00d4fb0 con 0x7f5db00d14d0 2024-05-28T06:15:03.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.214+0000 7f5dafe00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5db00d0820 con 0x7f5db015e870 2024-05-28T06:15:03.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.214+0000 7f5dafe00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f5db00d2cf0 con 0x7f5db00cfa40 2024-05-28T06:15:03.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.214+0000 7f5dae000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5db00d14d0 0x7f5db015a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:03.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.214+0000 7f5dae000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5db00d14d0 0x7f5db015a3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:52854/0 (socket says 172.21.15.96:52854) 2024-05-28T06:15:03.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.214+0000 7f5dae000640 1 -- 172.21.15.96:0/4014660938 learned_addr learned my addr 172.21.15.96:0/4014660938 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:15:03.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.215+0000 7f5daea00640 1 --2- 172.21.15.96:0/4014660938 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5db00cfa40 0x7f5db00b5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:03.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.215+0000 7f5dae000640 1 -- 172.21.15.96:0/4014660938 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5db00cfa40 msgr2=0x7f5db00b5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:03.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.215+0000 7f5dae000640 1 --2- 172.21.15.96:0/4014660938 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5db00cfa40 0x7f5db00b5610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:03.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.215+0000 7f5dae000640 1 -- 172.21.15.96:0/4014660938 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5db015e870 msgr2=0x7f5db015a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:03.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.215+0000 7f5dae000640 1 --2- 172.21.15.96:0/4014660938 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5db015e870 0x7f5db015a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:03.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.215+0000 7f5dae000640 1 -- 172.21.15.96:0/4014660938 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5db015c650 con 0x7f5db00d14d0 2024-05-28T06:15:03.218 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.215+0000 7f5dae000640 1 --2- 172.21.15.96:0/4014660938 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5db00d14d0 0x7f5db015a3b0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f5da000c7c0 tx=0x7f5da000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:15:03.218 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.215+0000 7f5da6a00640 1 -- 172.21.15.96:0/4014660938 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5da0003bc0 con 0x7f5db00d14d0 2024-05-28T06:15:03.218 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.215+0000 7f5da6a00640 1 -- 172.21.15.96:0/4014660938 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5da0003d80 con 0x7f5db00d14d0 2024-05-28T06:15:03.218 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.215+0000 7f5da6a00640 1 -- 172.21.15.96:0/4014660938 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5da000aa90 con 0x7f5db00d14d0 2024-05-28T06:15:03.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.216+0000 7f5dafe00640 1 -- 172.21.15.96:0/4014660938 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5db015af20 con 0x7f5db00d14d0 2024-05-28T06:15:03.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.216+0000 7f5dafe00640 1 -- 172.21.15.96:0/4014660938 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5db015b4e0 con 0x7f5db00d14d0 2024-05-28T06:15:03.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.217+0000 7f5dafe00640 1 -- 172.21.15.96:0/4014660938 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5db00689e0 con 0x7f5db00d14d0 2024-05-28T06:15:03.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.218+0000 7f5da6a00640 1 -- 172.21.15.96:0/4014660938 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 22) ==== 92213+0+0 (secure 0 0 0) 0x7f5da000ac30 con 0x7f5db00d14d0 2024-05-28T06:15:03.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.219+0000 7f5da6a00640 1 --2- 172.21.15.96:0/4014660938 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5d8c06d5e0 0x7f5d8c06faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:03.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.219+0000 7f5da6a00640 1 -- 172.21.15.96:0/4014660938 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 5737+0+0 (secure 0 0 0) 0x7f5da008a5e0 con 0x7f5db00d14d0 2024-05-28T06:15:03.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.220+0000 7f5daea00640 1 --2- 172.21.15.96:0/4014660938 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5d8c06d5e0 0x7f5d8c06faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:03.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.221+0000 7f5daea00640 1 --2- 172.21.15.96:0/4014660938 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5d8c06d5e0 0x7f5d8c06faa0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f5d98034000 tx=0x7f5d9802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:15:03.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.223+0000 7f5da6a00640 1 -- 172.21.15.96:0/4014660938 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5da005a460 con 0x7f5db00d14d0 2024-05-28T06:15:03.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.453+0000 7f5dafe00640 1 -- 172.21.15.96:0/4014660938 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7f5db00b6e60 con 0x7f5d8c06d5e0 2024-05-28T06:15:03.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.457+0000 7f5da6a00640 1 -- 172.21.15.96:0/4014660938 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7f5db00b6e60 con 0x7f5d8c06d5e0 2024-05-28T06:15:03.458 INFO:teuthology.orchestra.run.smithi096.stdout:[ 2024-05-28T06:15:03.459 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:15:03.459 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "cephfs" 2024-05-28T06:15:03.459 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:15:03.459 INFO:teuthology.orchestra.run.smithi096.stdout:] 2024-05-28T06:15:03.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.460+0000 7f5dafe00640 1 -- 172.21.15.96:0/4014660938 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5d8c06d5e0 msgr2=0x7f5d8c06faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:03.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.460+0000 7f5dafe00640 1 --2- 172.21.15.96:0/4014660938 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5d8c06d5e0 0x7f5d8c06faa0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f5d98034000 tx=0x7f5d9802e000 comp rx=0 tx=0).stop 2024-05-28T06:15:03.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.460+0000 7f5dafe00640 1 -- 172.21.15.96:0/4014660938 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5db00d14d0 msgr2=0x7f5db015a3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:03.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.460+0000 7f5dafe00640 1 --2- 172.21.15.96:0/4014660938 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5db00d14d0 0x7f5db015a3b0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f5da000c7c0 tx=0x7f5da000cc80 comp rx=0 tx=0).stop 2024-05-28T06:15:03.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.460+0000 7f5dafe00640 1 -- 172.21.15.96:0/4014660938 shutdown_connections 2024-05-28T06:15:03.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.460+0000 7f5dafe00640 1 --2- 172.21.15.96:0/4014660938 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5d8c06d5e0 0x7f5d8c06faa0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:03.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.460+0000 7f5dafe00640 1 --2- 172.21.15.96:0/4014660938 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5db015e870 0x7f5db015a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:03.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.460+0000 7f5dafe00640 1 --2- 172.21.15.96:0/4014660938 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5db00d14d0 0x7f5db015a3b0 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:03.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.460+0000 7f5dafe00640 1 --2- 172.21.15.96:0/4014660938 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5db00cfa40 0x7f5db00b5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:03.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.460+0000 7f5dafe00640 1 -- 172.21.15.96:0/4014660938 >> 172.21.15.96:0/4014660938 conn(0x7f5db008bfc0 msgr2=0x7f5db00b9790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:15:03.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.460+0000 7f5dafe00640 1 -- 172.21.15.96:0/4014660938 shutdown_connections 2024-05-28T06:15:03.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.461+0000 7f5dafe00640 1 -- 172.21.15.96:0/4014660938 wait complete. 2024-05-28T06:15:03.482 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-05-28T06:15:03.687 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.686+0000 7fbc38e00640 1 Processor -- start 2024-05-28T06:15:03.687 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.686+0000 7fbc38e00640 1 -- start start 2024-05-28T06:15:03.689 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.687+0000 7fbc38e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbc34153510 0x7fbc340b4d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:03.690 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.687+0000 7fbc38e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fbc340ccb90 0x7fbc340ccff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:03.690 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.687+0000 7fbc38e00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc3415d900 con 0x7fbc3415cb60 2024-05-28T06:15:03.690 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.687+0000 7fbc38e00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fbc341538f0 con 0x7fbc340ccb90 2024-05-28T06:15:03.690 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.687+0000 7fbc38e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fbc340cd530 con 0x7fbc34153510 2024-05-28T06:15:03.690 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.688+0000 7fbc2aa00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fbc340ccb90 0x7fbc340ccff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:03.690 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.688+0000 7fbc2aa00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fbc340ccb90 0x7fbc340ccff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:52870/0 (socket says 172.21.15.96:52870) 2024-05-28T06:15:03.690 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.688+0000 7fbc2aa00640 1 -- 172.21.15.96:0/1221552812 learned_addr learned my addr 172.21.15.96:0/1221552812 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:15:03.690 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.688+0000 7fbc33400640 1 --2- 172.21.15.96:0/1221552812 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbc34153510 0x7fbc340b4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:03.690 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.688+0000 7fbc2aa00640 1 -- 172.21.15.96:0/1221552812 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbc34153510 msgr2=0x7fbc340b4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:15:03.690 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.688+0000 7fbc2aa00640 1 --2- 172.21.15.96:0/1221552812 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbc34153510 0x7fbc340b4d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:03.690 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.688+0000 7fbc2aa00640 1 -- 172.21.15.96:0/1221552812 >> v1:172.21.15.96:6789/0 conn(0x7fbc3415cb60 legacy=0x7fbc3415cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:15:03.690 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.688+0000 7fbc2aa00640 1 -- 172.21.15.96:0/1221552812 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc340b3f90 con 0x7fbc340ccb90 2024-05-28T06:15:03.691 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.688+0000 7fbc33400640 1 --2- 172.21.15.96:0/1221552812 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbc34153510 0x7fbc340b4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:15:03.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.689+0000 7fbc2aa00640 1 --2- 172.21.15.96:0/1221552812 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc340ccb90 0x7fbc340ccff0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7fbc1c009870 tx=0x7fbc1c02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c09094a3a5f340a server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:15:03.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.689+0000 7fbc32a00640 1 -- 172.21.15.96:0/1221552812 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc1c037070 con 0x7fbc340ccb90 2024-05-28T06:15:03.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.690+0000 7fbc32a00640 1 -- 172.21.15.96:0/1221552812 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc1c032440 con 0x7fbc340ccb90 2024-05-28T06:15:03.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.690+0000 7fbc32a00640 1 -- 172.21.15.96:0/1221552812 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc1c032810 con 0x7fbc340ccb90 2024-05-28T06:15:03.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.690+0000 7fbc38e00640 1 -- 172.21.15.96:0/1221552812 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc340ccb90 msgr2=0x7fbc340ccff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:03.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.690+0000 7fbc38e00640 1 --2- 172.21.15.96:0/1221552812 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc340ccb90 0x7fbc340ccff0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7fbc1c009870 tx=0x7fbc1c02a1e0 comp rx=0 tx=0).stop 2024-05-28T06:15:03.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.690+0000 7fbc38e00640 1 -- 172.21.15.96:0/1221552812 shutdown_connections 2024-05-28T06:15:03.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.690+0000 7fbc38e00640 1 --2- 172.21.15.96:0/1221552812 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc340ccb90 0x7fbc340ccff0 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:03.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.690+0000 7fbc38e00640 1 --2- 172.21.15.96:0/1221552812 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbc34153510 0x7fbc340b4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:03.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.690+0000 7fbc38e00640 1 -- 172.21.15.96:0/1221552812 >> 172.21.15.96:0/1221552812 conn(0x7fbc3406b900 msgr2=0x7fbc3406bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:15:03.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.690+0000 7fbc38e00640 1 -- 172.21.15.96:0/1221552812 shutdown_connections 2024-05-28T06:15:03.693 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.690+0000 7fbc38e00640 1 -- 172.21.15.96:0/1221552812 wait complete. 2024-05-28T06:15:03.693 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.692+0000 7fbc38e00640 1 Processor -- start 2024-05-28T06:15:03.693 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.692+0000 7fbc38e00640 1 -- start start 2024-05-28T06:15:03.695 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.693+0000 7fbc38e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc340ccb90 0x7fbc34155570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:03.695 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.693+0000 7fbc38e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbc34153510 0x7fbc34148e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:03.695 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.693+0000 7fbc38e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc3415cb60 0x7fbc340ad130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:03.695 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.693+0000 7fbc38e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fbc340b4550 con 0x7fbc3415cb60 2024-05-28T06:15:03.695 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.693+0000 7fbc38e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fbc340b48c0 con 0x7fbc340ccb90 2024-05-28T06:15:03.695 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.693+0000 7fbc38e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fbc3406e940 con 0x7fbc34153510 2024-05-28T06:15:03.695 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.693+0000 7fbc33e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc3415cb60 0x7fbc340ad130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:03.695 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.693+0000 7fbc33e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc3415cb60 0x7fbc340ad130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:52882/0 (socket says 172.21.15.96:52882) 2024-05-28T06:15:03.695 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.693+0000 7fbc33e00640 1 -- 172.21.15.96:0/1097437903 learned_addr learned my addr 172.21.15.96:0/1097437903 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:15:03.695 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.694+0000 7fbc2aa00640 1 --2- 172.21.15.96:0/1097437903 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbc34153510 0x7fbc34148e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:03.696 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.694+0000 7fbc33400640 1 --2- 172.21.15.96:0/1097437903 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc340ccb90 0x7fbc34155570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:03.696 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.694+0000 7fbc33e00640 1 -- 172.21.15.96:0/1097437903 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbc34153510 msgr2=0x7fbc34148e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:03.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.694+0000 7fbc33e00640 1 --2- 172.21.15.96:0/1097437903 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbc34153510 0x7fbc34148e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:03.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.694+0000 7fbc33e00640 1 -- 172.21.15.96:0/1097437903 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc340ccb90 msgr2=0x7fbc34155570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:03.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.694+0000 7fbc33e00640 1 --2- 172.21.15.96:0/1097437903 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc340ccb90 0x7fbc34155570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:03.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.694+0000 7fbc33e00640 1 -- 172.21.15.96:0/1097437903 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc340adf70 con 0x7fbc3415cb60 2024-05-28T06:15:03.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.694+0000 7fbc33e00640 1 --2- 172.21.15.96:0/1097437903 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc3415cb60 0x7fbc340ad130 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7fbc2c00c730 tx=0x7fbc2c00cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:15:03.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.694+0000 7fbc2be00640 1 -- 172.21.15.96:0/1097437903 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc2c016020 con 0x7fbc3415cb60 2024-05-28T06:15:03.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.694+0000 7fbc2be00640 1 -- 172.21.15.96:0/1097437903 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc2c006d50 con 0x7fbc3415cb60 2024-05-28T06:15:03.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.694+0000 7fbc38e00640 1 -- 172.21.15.96:0/1097437903 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc34155c70 con 0x7fbc3415cb60 2024-05-28T06:15:03.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.694+0000 7fbc2be00640 1 -- 172.21.15.96:0/1097437903 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc2c003bb0 con 0x7fbc3415cb60 2024-05-28T06:15:03.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.695+0000 7fbc38e00640 1 -- 172.21.15.96:0/1097437903 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc34156230 con 0x7fbc3415cb60 2024-05-28T06:15:03.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.696+0000 7fbc38e00640 1 -- 172.21.15.96:0/1097437903 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc340689e0 con 0x7fbc3415cb60 2024-05-28T06:15:03.698 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.697+0000 7fbc2be00640 1 -- 172.21.15.96:0/1097437903 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 22) ==== 92213+0+0 (secure 0 0 0) 0x7fbc2c0050d0 con 0x7fbc3415cb60 2024-05-28T06:15:03.699 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.698+0000 7fbc2be00640 1 --2- 172.21.15.96:0/1097437903 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbc1006d610 0x7fbc1006fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:03.699 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.698+0000 7fbc2be00640 1 -- 172.21.15.96:0/1097437903 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 5737+0+0 (secure 0 0 0) 0x7fbc2c08a180 con 0x7fbc3415cb60 2024-05-28T06:15:03.700 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.699+0000 7fbc33400640 1 --2- 172.21.15.96:0/1097437903 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbc1006d610 0x7fbc1006fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:03.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.700+0000 7fbc33400640 1 --2- 172.21.15.96:0/1097437903 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbc1006d610 0x7fbc1006fad0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fbc24008850 tx=0x7fbc24008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:15:03.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.703+0000 7fbc2be00640 1 -- 172.21.15.96:0/1097437903 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbc2c020020 con 0x7fbc3415cb60 2024-05-28T06:15:03.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.902+0000 7fbc38e00640 1 -- 172.21.15.96:0/1097437903 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7fbc340b3c00 con 0x7fbc3415cb60 2024-05-28T06:15:03.904 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.902+0000 7fbc2be00640 1 -- 172.21.15.96:0/1097437903 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v18)=0 v18) ==== 122+0+0 (secure 0 0 0) 0x7fbc2c05a000 con 0x7fbc3415cb60 2024-05-28T06:15:03.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.905+0000 7fbc38e00640 1 -- 172.21.15.96:0/1097437903 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbc1006d610 msgr2=0x7fbc1006fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:03.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.905+0000 7fbc38e00640 1 --2- 172.21.15.96:0/1097437903 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbc1006d610 0x7fbc1006fad0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fbc24008850 tx=0x7fbc24008040 comp rx=0 tx=0).stop 2024-05-28T06:15:03.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.905+0000 7fbc38e00640 1 -- 172.21.15.96:0/1097437903 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc3415cb60 msgr2=0x7fbc340ad130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:03.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.905+0000 7fbc38e00640 1 --2- 172.21.15.96:0/1097437903 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc3415cb60 0x7fbc340ad130 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7fbc2c00c730 tx=0x7fbc2c00cb30 comp rx=0 tx=0).stop 2024-05-28T06:15:03.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.905+0000 7fbc38e00640 1 -- 172.21.15.96:0/1097437903 shutdown_connections 2024-05-28T06:15:03.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.905+0000 7fbc38e00640 1 --2- 172.21.15.96:0/1097437903 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbc1006d610 0x7fbc1006fad0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:03.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.905+0000 7fbc38e00640 1 --2- 172.21.15.96:0/1097437903 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc3415cb60 0x7fbc340ad130 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:03.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.905+0000 7fbc38e00640 1 --2- 172.21.15.96:0/1097437903 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbc34153510 0x7fbc34148e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:03.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.905+0000 7fbc38e00640 1 --2- 172.21.15.96:0/1097437903 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc340ccb90 0x7fbc34155570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:03.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.905+0000 7fbc38e00640 1 -- 172.21.15.96:0/1097437903 >> 172.21.15.96:0/1097437903 conn(0x7fbc3406b900 msgr2=0x7fbc340afdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:15:03.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.905+0000 7fbc38e00640 1 -- 172.21.15.96:0/1097437903 shutdown_connections 2024-05-28T06:15:03.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:03.905+0000 7fbc38e00640 1 -- 172.21.15.96:0/1097437903 wait complete. 2024-05-28T06:15:03.927 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/snapshot_clone_no_wait false 2024-05-28T06:15:04.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.128+0000 7fc932e00640 1 Processor -- start 2024-05-28T06:15:04.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.129+0000 7fc932e00640 1 -- start start 2024-05-28T06:15:04.132 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.130+0000 7fc932e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc92c161d00 0x7fc92c0b3d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:04.132 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.130+0000 7fc932e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fc92c0d1550 0x7fc92c0d27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:04.132 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.130+0000 7fc932e00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc92c0cfb40 con 0x7fc92c0d4d90 2024-05-28T06:15:04.132 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.130+0000 7fc932e00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fc92c0d2040 con 0x7fc92c0d1550 2024-05-28T06:15:04.132 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.130+0000 7fc932e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fc92c0cfdc0 con 0x7fc92c161d00 2024-05-28T06:15:04.132 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.131+0000 7fc932400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fc92c0d1550 0x7fc92c0d27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:04.133 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.131+0000 7fc932400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fc92c0d1550 0x7fc92c0d27b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:52894/0 (socket says 172.21.15.96:52894) 2024-05-28T06:15:04.133 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.131+0000 7fc932400640 1 -- 172.21.15.96:0/4048185827 learned_addr learned my addr 172.21.15.96:0/4048185827 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:15:04.133 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.131+0000 7fc931a00640 1 --2- 172.21.15.96:0/4048185827 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc92c161d00 0x7fc92c0b3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:04.133 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.131+0000 7fc932400640 1 -- 172.21.15.96:0/4048185827 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc92c161d00 msgr2=0x7fc92c0b3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:15:04.133 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.131+0000 7fc932400640 1 --2- 172.21.15.96:0/4048185827 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc92c161d00 0x7fc92c0b3d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:04.133 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.131+0000 7fc932400640 1 -- 172.21.15.96:0/4048185827 >> v1:172.21.15.123:6789/0 conn(0x7fc92c0d4d90 legacy=0x7fc92c0d0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:15:04.133 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.131+0000 7fc932400640 1 -- 172.21.15.96:0/4048185827 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc92c0d00a0 con 0x7fc92c0d1550 2024-05-28T06:15:04.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.132+0000 7fc932400640 1 --2- 172.21.15.96:0/4048185827 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc92c0d1550 0x7fc92c0d27b0 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7fc924009e50 tx=0x7fc92402a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f872e5349c98739 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:15:04.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.133+0000 7fc92be00640 1 -- 172.21.15.96:0/4048185827 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc924036070 con 0x7fc92c0d1550 2024-05-28T06:15:04.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.133+0000 7fc92be00640 1 -- 172.21.15.96:0/4048185827 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc924031600 con 0x7fc92c0d1550 2024-05-28T06:15:04.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.133+0000 7fc92be00640 1 -- 172.21.15.96:0/4048185827 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9240319d0 con 0x7fc92c0d1550 2024-05-28T06:15:04.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.133+0000 7fc932e00640 1 -- 172.21.15.96:0/4048185827 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc92c0d1550 msgr2=0x7fc92c0d27b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:04.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.133+0000 7fc932e00640 1 --2- 172.21.15.96:0/4048185827 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc92c0d1550 0x7fc92c0d27b0 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7fc924009e50 tx=0x7fc92402a6c0 comp rx=0 tx=0).stop 2024-05-28T06:15:04.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.133+0000 7fc932e00640 1 -- 172.21.15.96:0/4048185827 shutdown_connections 2024-05-28T06:15:04.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.133+0000 7fc932e00640 1 --2- 172.21.15.96:0/4048185827 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc92c0d1550 0x7fc92c0d27b0 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:04.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.133+0000 7fc932e00640 1 --2- 172.21.15.96:0/4048185827 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc92c161d00 0x7fc92c0b3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:04.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.133+0000 7fc932e00640 1 -- 172.21.15.96:0/4048185827 >> 172.21.15.96:0/4048185827 conn(0x7fc92c08bfc0 msgr2=0x7fc92c0d4470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:15:04.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.134+0000 7fc932e00640 1 -- 172.21.15.96:0/4048185827 shutdown_connections 2024-05-28T06:15:04.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.134+0000 7fc932e00640 1 -- 172.21.15.96:0/4048185827 wait complete. 2024-05-28T06:15:04.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.135+0000 7fc932e00640 1 Processor -- start 2024-05-28T06:15:04.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.135+0000 7fc932e00640 1 -- start start 2024-05-28T06:15:04.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.136+0000 7fc932e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc92c0d1550 0x7fc92c0b7090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:04.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.136+0000 7fc932e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc92c0d4d90 0x7fc92c0b9c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:04.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.136+0000 7fc932e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc92c161d00 0x7fc92c0b42a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:04.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.136+0000 7fc932e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc92c0b5560 con 0x7fc92c0d4d90 2024-05-28T06:15:04.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.136+0000 7fc932e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc92c0d2290 con 0x7fc92c0d1550 2024-05-28T06:15:04.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.136+0000 7fc932e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fc92c0b56e0 con 0x7fc92c161d00 2024-05-28T06:15:04.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.137+0000 7fc931000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc92c0d4d90 0x7fc92c0b9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:04.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.137+0000 7fc931000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc92c0d4d90 0x7fc92c0b9c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:52900/0 (socket says 172.21.15.96:52900) 2024-05-28T06:15:04.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.137+0000 7fc931000640 1 -- 172.21.15.96:0/1774819621 learned_addr learned my addr 172.21.15.96:0/1774819621 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:15:04.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.137+0000 7fc932400640 1 --2- 172.21.15.96:0/1774819621 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc92c161d00 0x7fc92c0b42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:04.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.137+0000 7fc931a00640 1 --2- 172.21.15.96:0/1774819621 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc92c0d1550 0x7fc92c0b7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:04.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.137+0000 7fc931000640 1 -- 172.21.15.96:0/1774819621 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc92c161d00 msgr2=0x7fc92c0b42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:04.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.137+0000 7fc931000640 1 --2- 172.21.15.96:0/1774819621 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc92c161d00 0x7fc92c0b42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:04.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.137+0000 7fc931000640 1 -- 172.21.15.96:0/1774819621 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc92c0d1550 msgr2=0x7fc92c0b7090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:04.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.137+0000 7fc931000640 1 --2- 172.21.15.96:0/1774819621 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc92c0d1550 0x7fc92c0b7090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:04.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.137+0000 7fc931000640 1 -- 172.21.15.96:0/1774819621 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc92c0b8ec0 con 0x7fc92c0d4d90 2024-05-28T06:15:04.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.137+0000 7fc932400640 1 --2- 172.21.15.96:0/1774819621 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc92c161d00 0x7fc92c0b42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:15:04.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.137+0000 7fc931000640 1 --2- 172.21.15.96:0/1774819621 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc92c0d4d90 0x7fc92c0b9c80 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7fc91800c430 tx=0x7fc91800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:15:04.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.138+0000 7fc929600640 1 -- 172.21.15.96:0/1774819621 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc918011790 con 0x7fc92c0d4d90 2024-05-28T06:15:04.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.138+0000 7fc929600640 1 -- 172.21.15.96:0/1774819621 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc918017070 con 0x7fc92c0d4d90 2024-05-28T06:15:04.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.138+0000 7fc929600640 1 -- 172.21.15.96:0/1774819621 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc918011bb0 con 0x7fc92c0d4d90 2024-05-28T06:15:04.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.138+0000 7fc932e00640 1 -- 172.21.15.96:0/1774819621 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc92c0b91a0 con 0x7fc92c0d4d90 2024-05-28T06:15:04.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.138+0000 7fc932e00640 1 -- 172.21.15.96:0/1774819621 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc92c0ba1c0 con 0x7fc92c0d4d90 2024-05-28T06:15:04.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.139+0000 7fc932e00640 1 -- 172.21.15.96:0/1774819621 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc92c0689e0 con 0x7fc92c0d4d90 2024-05-28T06:15:04.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.140+0000 7fc929600640 1 -- 172.21.15.96:0/1774819621 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 22) ==== 92213+0+0 (secure 0 0 0) 0x7fc91801c460 con 0x7fc92c0d4d90 2024-05-28T06:15:04.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.141+0000 7fc929600640 1 --2- 172.21.15.96:0/1774819621 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc90c06d610 0x7fc90c06fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:04.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.141+0000 7fc929600640 1 -- 172.21.15.96:0/1774819621 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 5737+0+0 (secure 0 0 0) 0x7fc91808a3c0 con 0x7fc92c0d4d90 2024-05-28T06:15:04.143 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.142+0000 7fc931a00640 1 --2- 172.21.15.96:0/1774819621 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc90c06d610 0x7fc90c06fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:04.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.143+0000 7fc931a00640 1 --2- 172.21.15.96:0/1774819621 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc90c06d610 0x7fc90c06fad0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fc920006fd0 tx=0x7fc920008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:15:04.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.145+0000 7fc929600640 1 -- 172.21.15.96:0/1774819621 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc91805a240 con 0x7fc92c0d4d90 2024-05-28T06:15:04.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.346+0000 7fc932e00640 1 -- 172.21.15.96:0/1774819621 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/snapshot_clone_no_wait}] v 0) -- 0x7fc92c0d1c40 con 0x7fc92c0d4d90 2024-05-28T06:15:04.355 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.354+0000 7fc929600640 1 -- 172.21.15.96:0/1774819621 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/snapshot_clone_no_wait}]=0 v19)=0 v19) ==== 136+0+0 (secure 0 0 0) 0x7fc918059b40 con 0x7fc92c0d4d90 2024-05-28T06:15:04.358 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.357+0000 7fc932e00640 1 -- 172.21.15.96:0/1774819621 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc90c06d610 msgr2=0x7fc90c06fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:04.358 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.357+0000 7fc932e00640 1 --2- 172.21.15.96:0/1774819621 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc90c06d610 0x7fc90c06fad0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fc920006fd0 tx=0x7fc920008040 comp rx=0 tx=0).stop 2024-05-28T06:15:04.358 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.357+0000 7fc932e00640 1 -- 172.21.15.96:0/1774819621 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc92c0d4d90 msgr2=0x7fc92c0b9c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:04.358 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.357+0000 7fc932e00640 1 --2- 172.21.15.96:0/1774819621 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc92c0d4d90 0x7fc92c0b9c80 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7fc91800c430 tx=0x7fc91800c8f0 comp rx=0 tx=0).stop 2024-05-28T06:15:04.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.357+0000 7fc932e00640 1 -- 172.21.15.96:0/1774819621 shutdown_connections 2024-05-28T06:15:04.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.357+0000 7fc932e00640 1 --2- 172.21.15.96:0/1774819621 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc90c06d610 0x7fc90c06fad0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:04.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.357+0000 7fc932e00640 1 --2- 172.21.15.96:0/1774819621 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc92c161d00 0x7fc92c0b42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:04.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.357+0000 7fc932e00640 1 --2- 172.21.15.96:0/1774819621 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc92c0d4d90 0x7fc92c0b9c80 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:04.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.357+0000 7fc932e00640 1 --2- 172.21.15.96:0/1774819621 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc92c0d1550 0x7fc92c0b7090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:04.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.357+0000 7fc932e00640 1 -- 172.21.15.96:0/1774819621 >> 172.21.15.96:0/1774819621 conn(0x7fc92c08bfc0 msgr2=0x7fc92c0b6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:15:04.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.357+0000 7fc932e00640 1 -- 172.21.15.96:0/1774819621 shutdown_connections 2024-05-28T06:15:04.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.357+0000 7fc932e00640 1 -- 172.21.15.96:0/1774819621 wait complete. 2024-05-28T06:15:04.378 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv1 --mode=777 2024-05-28T06:15:04.394 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:15:04.392+0000 7fea5da00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:15:04.394 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:15:04.392+0000 7fea5da00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:15:04.394 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:15:04.392+0000 7fea5da00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:15:04.394 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:15:04.392+0000 7fea5da00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:15:04.394 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:15:04.392+0000 7fea5da00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:15:04.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.577+0000 7faf5e800640 1 Processor -- start 2024-05-28T06:15:04.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.577+0000 7faf5e800640 1 -- start start 2024-05-28T06:15:04.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.579+0000 7faf5e800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faf5806ef00 0x7faf580d0320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:04.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.579+0000 7faf5e800640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faf5815c7d0 con 0x7faf580b4790 2024-05-28T06:15:04.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.579+0000 7faf5e800640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faf580ad9a0 con 0x7faf5815c3f0 2024-05-28T06:15:04.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.579+0000 7faf5e800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7faf580ad3e0 con 0x7faf5806ef00 2024-05-28T06:15:04.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.579+0000 7faf5de00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7faf580b4790 0x7faf580b30b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:41942/0 (socket says 172.21.15.96:41942) 2024-05-28T06:15:04.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.579+0000 7faf5de00640 1 -- 172.21.15.96:0/999288534 learned_addr learned my addr 172.21.15.96:0/999288534 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:15:04.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.579+0000 7faf5ca00640 1 --2- 172.21.15.96:0/999288534 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faf5806ef00 0x7faf580d0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:04.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.580+0000 7faf57e00640 1 -- 172.21.15.96:0/999288534 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2775356584 0 0) 0x7faf5815c7d0 con 0x7faf580b4790 2024-05-28T06:15:04.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.580+0000 7faf57e00640 1 -- 172.21.15.96:0/999288534 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faf3c003520 con 0x7faf580b4790 2024-05-28T06:15:04.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.580+0000 7faf57e00640 1 -- 172.21.15.96:0/999288534 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1582467396 0 0) 0x7faf3c003520 con 0x7faf580b4790 2024-05-28T06:15:04.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.580+0000 7faf57e00640 1 -- 172.21.15.96:0/999288534 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faf5806ef00 msgr2=0x7faf580d0320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:15:04.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.580+0000 7faf57e00640 1 --2- 172.21.15.96:0/999288534 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faf5806ef00 0x7faf580d0320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:04.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.580+0000 7faf57e00640 1 -- 172.21.15.96:0/999288534 >> v1:172.21.15.123:6789/0 conn(0x7faf5815c3f0 legacy=0x7faf581608d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:04.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.580+0000 7faf57e00640 1 -- 172.21.15.96:0/999288534 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf580ad6c0 con 0x7faf580b4790 2024-05-28T06:15:04.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.580+0000 7faf57e00640 1 -- 172.21.15.96:0/999288534 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7faf50001690 con 0x7faf580b4790 2024-05-28T06:15:04.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.580+0000 7faf57e00640 1 -- 172.21.15.96:0/999288534 >> v1:172.21.15.96:6789/0 conn(0x7faf580b4790 legacy=0x7faf580b30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:04.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.580+0000 7faf57e00640 1 --2- 172.21.15.96:0/999288534 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faf5806ef00 0x7faf3c003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:04.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.580+0000 7faf57e00640 1 --2- 172.21.15.96:0/999288534 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf3c003fd0 0x7faf3c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:04.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.580+0000 7faf57e00640 1 --2- 172.21.15.96:0/999288534 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faf3c004d40 0x7faf3c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:04.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.580+0000 7faf57e00640 1 -- 172.21.15.96:0/999288534 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7faf580ad3e0 con 0x7faf3c003fd0 2024-05-28T06:15:04.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.580+0000 7faf57e00640 1 -- 172.21.15.96:0/999288534 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7faf3c00bb20 con 0x7faf3c004d40 2024-05-28T06:15:04.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.580+0000 7faf57e00640 1 -- 172.21.15.96:0/999288534 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7faf3c00bd20 con 0x7faf5806ef00 2024-05-28T06:15:04.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.580+0000 7faf5ca00640 1 --2- 172.21.15.96:0/999288534 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf3c003fd0 0x7faf3c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:04.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.581+0000 7faf5d400640 1 --2- 172.21.15.96:0/999288534 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faf5806ef00 0x7faf3c003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:04.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.581+0000 7faf5ca00640 1 -- 172.21.15.96:0/999288534 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faf5806ef00 msgr2=0x7faf3c003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:15:04.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.581+0000 7faf5ca00640 1 --2- 172.21.15.96:0/999288534 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faf5806ef00 0x7faf3c003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:04.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.581+0000 7faf5ca00640 1 -- 172.21.15.96:0/999288534 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faf3c004d40 msgr2=0x7faf3c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:15:04.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.581+0000 7faf5ca00640 1 --2- 172.21.15.96:0/999288534 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faf3c004d40 0x7faf3c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:04.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.581+0000 7faf5ca00640 1 -- 172.21.15.96:0/999288534 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf3c003520 con 0x7faf3c003fd0 2024-05-28T06:15:04.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.582+0000 7faf5ca00640 1 --2- 172.21.15.96:0/999288534 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf3c003fd0 0x7faf3c004800 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7faf44009a90 tx=0x7faf3c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b41088420283eef8 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:15:04.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.582+0000 7faf57e00640 1 -- 172.21.15.96:0/999288534 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf44036070 con 0x7faf3c003fd0 2024-05-28T06:15:04.586 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.582+0000 7faf57e00640 1 -- 172.21.15.96:0/999288534 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf44031400 con 0x7faf3c003fd0 2024-05-28T06:15:04.586 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.582+0000 7faf57e00640 1 -- 172.21.15.96:0/999288534 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf44009690 con 0x7faf3c003fd0 2024-05-28T06:15:04.586 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.583+0000 7faf5e800640 1 -- 172.21.15.96:0/999288534 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf3c003fd0 msgr2=0x7faf3c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:04.586 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.583+0000 7faf5e800640 1 --2- 172.21.15.96:0/999288534 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf3c003fd0 0x7faf3c004800 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7faf44009a90 tx=0x7faf3c005830 comp rx=0 tx=0).stop 2024-05-28T06:15:04.586 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.583+0000 7faf5de00640 1 -- 172.21.15.96:0/999288534 reap_dead start 2024-05-28T06:15:04.586 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.583+0000 7faf5e800640 1 -- 172.21.15.96:0/999288534 shutdown_connections 2024-05-28T06:15:04.586 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.583+0000 7faf5e800640 1 -- 172.21.15.96:0/999288534 >> 172.21.15.96:0/999288534 conn(0x7faf5806bae0 msgr2=0x7faf5806bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:15:04.586 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.583+0000 7faf5e800640 1 -- 172.21.15.96:0/999288534 shutdown_connections 2024-05-28T06:15:04.586 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.583+0000 7faf5e800640 1 -- 172.21.15.96:0/999288534 wait complete. 2024-05-28T06:15:04.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.585+0000 7faf5e800640 1 Processor -- start 2024-05-28T06:15:04.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.585+0000 7faf5e800640 1 -- start start 2024-05-28T06:15:04.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.586+0000 7faf5e800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf3c003fd0 0x7faf580734e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:04.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.586+0000 7faf5e800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faf581f7680 0x7faf58152470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:04.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.586+0000 7faf5e800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faf581f5250 0x7faf581f56d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:04.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.586+0000 7faf5e800640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7faf580ad090 con 0x7faf3c003fd0 2024-05-28T06:15:04.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.586+0000 7faf5e800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7faf581603b0 con 0x7faf581f7680 2024-05-28T06:15:04.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.586+0000 7faf5e800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7faf580b37c0 con 0x7faf581f5250 2024-05-28T06:15:04.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.586+0000 7faf5d400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf3c003fd0 0x7faf580734e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:04.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.586+0000 7faf5d400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf3c003fd0 0x7faf580734e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:52918/0 (socket says 172.21.15.96:52918) 2024-05-28T06:15:04.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.586+0000 7faf5d400640 1 -- 172.21.15.96:0/306701889 learned_addr learned my addr 172.21.15.96:0/306701889 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:15:04.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.587+0000 7faf5d400640 1 -- 172.21.15.96:0/306701889 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faf581f5250 msgr2=0x7faf581f56d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:04.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.587+0000 7faf5d400640 1 --2- 172.21.15.96:0/306701889 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faf581f5250 0x7faf581f56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:04.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.587+0000 7faf5d400640 1 -- 172.21.15.96:0/306701889 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faf581f7680 msgr2=0x7faf58152470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:04.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.587+0000 7faf5d400640 1 --2- 172.21.15.96:0/306701889 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faf581f7680 0x7faf58152470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:04.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.587+0000 7faf5d400640 1 -- 172.21.15.96:0/306701889 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf581f4280 con 0x7faf3c003fd0 2024-05-28T06:15:04.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.587+0000 7faf5d400640 1 --2- 172.21.15.96:0/306701889 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf3c003fd0 0x7faf580734e0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7faf4800e8f0 tx=0x7faf4800edb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:15:04.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.587+0000 7faf55600640 1 -- 172.21.15.96:0/306701889 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf48003b80 con 0x7faf3c003fd0 2024-05-28T06:15:04.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.587+0000 7faf55600640 1 -- 172.21.15.96:0/306701889 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf48018070 con 0x7faf3c003fd0 2024-05-28T06:15:04.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.587+0000 7faf5e800640 1 -- 172.21.15.96:0/306701889 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf581f7df0 con 0x7faf3c003fd0 2024-05-28T06:15:04.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.587+0000 7faf55600640 1 -- 172.21.15.96:0/306701889 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf480050d0 con 0x7faf3c003fd0 2024-05-28T06:15:04.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.587+0000 7faf5e800640 1 -- 172.21.15.96:0/306701889 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf581f83b0 con 0x7faf3c003fd0 2024-05-28T06:15:04.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.589+0000 7faf5e800640 1 -- 172.21.15.96:0/306701889 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf581f85b0 con 0x7faf3c003fd0 2024-05-28T06:15:04.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.590+0000 7faf55600640 1 -- 172.21.15.96:0/306701889 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 22) ==== 92213+0+0 (secure 0 0 0) 0x7faf48005270 con 0x7faf3c003fd0 2024-05-28T06:15:04.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.590+0000 7faf55600640 1 --2- 172.21.15.96:0/306701889 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7faf3806d5e0 0x7faf3806faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:04.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.591+0000 7faf55600640 1 -- 172.21.15.96:0/306701889 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 5737+0+0 (secure 0 0 0) 0x7faf4808a880 con 0x7faf3c003fd0 2024-05-28T06:15:04.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.591+0000 7faf5ca00640 1 --2- 172.21.15.96:0/306701889 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7faf3806d5e0 0x7faf3806faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:04.594 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.592+0000 7faf5ca00640 1 --2- 172.21.15.96:0/306701889 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7faf3806d5e0 0x7faf3806faa0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7faf44033000 tx=0x7faf4402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:15:04.596 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.595+0000 7faf55600640 1 -- 172.21.15.96:0/306701889 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7faf4801a310 con 0x7faf3c003fd0 2024-05-28T06:15:04.826 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.824+0000 7faf5e800640 1 -- 172.21.15.96:0/306701889 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv1", "mode": "777", "target": ["mon-mgr", ""]}) -- 0x7faf580689e0 con 0x7faf3806d5e0 2024-05-28T06:15:04.841 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:15:04.840+0000 7fea59e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:15:04.841 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:15:04.840+0000 7fea59e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:15:04.841 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:15:04.840+0000 7fea59e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:15:04.841 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:15:04.840+0000 7fea59e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:15:04.841 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:15:04.840+0000 7fea59e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:15:04.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.908+0000 7faf55600640 1 -- 172.21.15.96:0/306701889 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7faf580689e0 con 0x7faf3806d5e0 2024-05-28T06:15:04.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.911+0000 7faf5e800640 1 -- 172.21.15.96:0/306701889 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7faf3806d5e0 msgr2=0x7faf3806faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:04.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.911+0000 7faf5e800640 1 --2- 172.21.15.96:0/306701889 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7faf3806d5e0 0x7faf3806faa0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7faf44033000 tx=0x7faf4402e000 comp rx=0 tx=0).stop 2024-05-28T06:15:04.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.911+0000 7faf5e800640 1 -- 172.21.15.96:0/306701889 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf3c003fd0 msgr2=0x7faf580734e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:04.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.911+0000 7faf5e800640 1 --2- 172.21.15.96:0/306701889 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf3c003fd0 0x7faf580734e0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7faf4800e8f0 tx=0x7faf4800edb0 comp rx=0 tx=0).stop 2024-05-28T06:15:04.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.911+0000 7faf5e800640 1 -- 172.21.15.96:0/306701889 shutdown_connections 2024-05-28T06:15:04.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.911+0000 7faf5e800640 1 --2- 172.21.15.96:0/306701889 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7faf3806d5e0 0x7faf3806faa0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:04.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.911+0000 7faf5e800640 1 --2- 172.21.15.96:0/306701889 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faf581f5250 0x7faf581f56d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:04.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.911+0000 7faf5e800640 1 --2- 172.21.15.96:0/306701889 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faf581f7680 0x7faf58152470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:04.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.911+0000 7faf5e800640 1 --2- 172.21.15.96:0/306701889 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf3c003fd0 0x7faf580734e0 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:04.914 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.911+0000 7faf5e800640 1 -- 172.21.15.96:0/306701889 >> 172.21.15.96:0/306701889 conn(0x7faf5806bae0 msgr2=0x7faf5815dfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:15:04.914 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.911+0000 7faf5e800640 1 -- 172.21.15.96:0/306701889 shutdown_connections 2024-05-28T06:15:04.914 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:04.912+0000 7faf5e800640 1 -- 172.21.15.96:0/306701889 wait complete. 2024-05-28T06:15:04.939 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-05-28T06:15:05.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.142+0000 7f7855600640 1 Processor -- start 2024-05-28T06:15:05.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.142+0000 7f7855600640 1 -- start start 2024-05-28T06:15:05.145 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.144+0000 7f7855600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f785015e870 0x7f78500d0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:05.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.144+0000 7f7855600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f78500d14d0 0x7f78500cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:05.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.144+0000 7f7855600640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78500cfd00 con 0x7f78500cf870 2024-05-28T06:15:05.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.144+0000 7f7855600640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f78500d2150 con 0x7f78500d14d0 2024-05-28T06:15:05.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.144+0000 7f7855600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f785015ec50 con 0x7f785015e870 2024-05-28T06:15:05.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.144+0000 7f7854c00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f78500cf870 0x7f7850162360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:41954/0 (socket says 172.21.15.96:41954) 2024-05-28T06:15:05.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.144+0000 7f7854c00640 1 -- 172.21.15.96:0/4033912954 learned_addr learned my addr 172.21.15.96:0/4033912954 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:15:05.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.145+0000 7f7847e00640 1 --2- 172.21.15.96:0/4033912954 >> v2:172.21.15.123:3300/0 conn(0x7f78500d14d0 0x7f78500cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:05.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.145+0000 7f784fe00640 1 --2- 172.21.15.96:0/4033912954 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f785015e870 0x7f78500d0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:05.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.145+0000 7f784f400640 1 -- 172.21.15.96:0/4033912954 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1153828776 0 0) 0x7f7848001490 con 0x7f78500cf870 2024-05-28T06:15:05.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.145+0000 7f784f400640 1 -- 172.21.15.96:0/4033912954 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7834003520 con 0x7f78500cf870 2024-05-28T06:15:05.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.145+0000 7f784f400640 1 -- 172.21.15.96:0/4033912954 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3449688016 0 0) 0x7f7834003520 con 0x7f78500cf870 2024-05-28T06:15:05.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.145+0000 7f784f400640 1 -- 172.21.15.96:0/4033912954 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f785015e870 msgr2=0x7f78500d0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:15:05.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.145+0000 7f784f400640 1 --2- 172.21.15.96:0/4033912954 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f785015e870 0x7f78500d0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:05.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.145+0000 7f784f400640 1 -- 172.21.15.96:0/4033912954 >> v2:172.21.15.123:3300/0 conn(0x7f78500d14d0 msgr2=0x7f78500cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:15:05.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.145+0000 7f784f400640 1 --2- 172.21.15.96:0/4033912954 >> v2:172.21.15.123:3300/0 conn(0x7f78500d14d0 0x7f78500cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:05.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.145+0000 7f784f400640 1 -- 172.21.15.96:0/4033912954 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78500cff80 con 0x7f78500cf870 2024-05-28T06:15:05.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.145+0000 7f784f400640 1 -- 172.21.15.96:0/4033912954 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f7848001bc0 con 0x7f78500cf870 2024-05-28T06:15:05.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.145+0000 7f784f400640 1 -- 172.21.15.96:0/4033912954 >> v1:172.21.15.96:6789/0 conn(0x7f78500cf870 legacy=0x7f7850162360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:05.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.145+0000 7f784f400640 1 --2- 172.21.15.96:0/4033912954 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f785015e870 0x7f78340039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:05.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.145+0000 7f784f400640 1 --2- 172.21.15.96:0/4033912954 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7834003f80 0x7f7834004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:05.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.146+0000 7f784f400640 1 --2- 172.21.15.96:0/4033912954 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7834004d40 0x7f78340051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:05.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.146+0000 7f784f400640 1 -- 172.21.15.96:0/4033912954 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f78500d2150 con 0x7f7834003f80 2024-05-28T06:15:05.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.146+0000 7f784f400640 1 -- 172.21.15.96:0/4033912954 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f785015ec50 con 0x7f7834004d40 2024-05-28T06:15:05.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.146+0000 7f784f400640 1 -- 172.21.15.96:0/4033912954 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f783400b9c0 con 0x7f785015e870 2024-05-28T06:15:05.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.146+0000 7f7847e00640 1 --2- 172.21.15.96:0/4033912954 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7834003f80 0x7f7834004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:05.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.146+0000 7f7854c00640 1 --2- 172.21.15.96:0/4033912954 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7834004d40 0x7f78340051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:05.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.146+0000 7f784fe00640 1 --2- 172.21.15.96:0/4033912954 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f785015e870 0x7f78340039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:05.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.146+0000 7f7847e00640 1 -- 172.21.15.96:0/4033912954 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f785015e870 msgr2=0x7f78340039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:15:05.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.146+0000 7f7847e00640 1 --2- 172.21.15.96:0/4033912954 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f785015e870 0x7f78340039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:05.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.146+0000 7f7847e00640 1 -- 172.21.15.96:0/4033912954 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7834004d40 msgr2=0x7f78340051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:15:05.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.146+0000 7f7847e00640 1 --2- 172.21.15.96:0/4033912954 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7834004d40 0x7f78340051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:05.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.146+0000 7f7847e00640 1 -- 172.21.15.96:0/4033912954 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7834003520 con 0x7f7834003f80 2024-05-28T06:15:05.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.147+0000 7f7847e00640 1 --2- 172.21.15.96:0/4033912954 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7834003f80 0x7f7834004800 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f7838009ae0 tx=0x7f7834005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1105c9eee201ff33 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:15:05.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.148+0000 7f784f400640 1 -- 172.21.15.96:0/4033912954 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7838036070 con 0x7f7834003f80 2024-05-28T06:15:05.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.148+0000 7f784f400640 1 -- 172.21.15.96:0/4033912954 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7838005990 con 0x7f7834003f80 2024-05-28T06:15:05.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.148+0000 7f784f400640 1 -- 172.21.15.96:0/4033912954 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7838009680 con 0x7f7834003f80 2024-05-28T06:15:05.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.148+0000 7f7855600640 1 -- 172.21.15.96:0/4033912954 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7834003f80 msgr2=0x7f7834004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:05.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.148+0000 7f7855600640 1 --2- 172.21.15.96:0/4033912954 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7834003f80 0x7f7834004800 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f7838009ae0 tx=0x7f7834005830 comp rx=0 tx=0).stop 2024-05-28T06:15:05.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.148+0000 7f7854c00640 1 -- 172.21.15.96:0/4033912954 reap_dead start 2024-05-28T06:15:05.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.148+0000 7f7855600640 1 -- 172.21.15.96:0/4033912954 shutdown_connections 2024-05-28T06:15:05.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.148+0000 7f7855600640 1 -- 172.21.15.96:0/4033912954 >> 172.21.15.96:0/4033912954 conn(0x7f785008bfc0 msgr2=0x7f78500d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:15:05.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.148+0000 7f7855600640 1 -- 172.21.15.96:0/4033912954 shutdown_connections 2024-05-28T06:15:05.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.148+0000 7f7855600640 1 -- 172.21.15.96:0/4033912954 wait complete. 2024-05-28T06:15:05.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.150+0000 7f7855600640 1 Processor -- start 2024-05-28T06:15:05.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.150+0000 7f7855600640 1 -- start start 2024-05-28T06:15:05.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.151+0000 7f7855600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7834003f80 0x7f7850160bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:05.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.151+0000 7f7855600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7850159740 0x7f78500b96f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:05.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.151+0000 7f7855600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7850159bc0 0x7f78500b9c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:05.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.151+0000 7f7855600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f78500d0820 con 0x7f7850159740 2024-05-28T06:15:05.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.151+0000 7f7855600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f78500d0540 con 0x7f7850159bc0 2024-05-28T06:15:05.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.151+0000 7f7855600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f78500d4fb0 con 0x7f7834003f80 2024-05-28T06:15:05.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.151+0000 7f7847e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7850159740 0x7f78500b96f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:05.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.152+0000 7f7847e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7850159740 0x7f78500b96f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:52938/0 (socket says 172.21.15.96:52938) 2024-05-28T06:15:05.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.152+0000 7f7847e00640 1 -- 172.21.15.96:0/2298958580 learned_addr learned my addr 172.21.15.96:0/2298958580 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:15:05.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.152+0000 7f784fe00640 1 --2- 172.21.15.96:0/2298958580 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7834003f80 0x7f7850160bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:05.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.152+0000 7f7847e00640 1 -- 172.21.15.96:0/2298958580 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7834003f80 msgr2=0x7f7850160bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:05.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.152+0000 7f7847e00640 1 --2- 172.21.15.96:0/2298958580 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7834003f80 0x7f7850160bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:05.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.152+0000 7f7854c00640 1 --2- 172.21.15.96:0/2298958580 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7850159bc0 0x7f78500b9c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:05.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.152+0000 7f7847e00640 1 -- 172.21.15.96:0/2298958580 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7850159bc0 msgr2=0x7f78500b9c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:05.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.152+0000 7f7847e00640 1 --2- 172.21.15.96:0/2298958580 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7850159bc0 0x7f78500b9c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:05.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.152+0000 7f7847e00640 1 -- 172.21.15.96:0/2298958580 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f785015f870 con 0x7f7850159740 2024-05-28T06:15:05.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.152+0000 7f7847e00640 1 --2- 172.21.15.96:0/2298958580 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7850159740 0x7f78500b96f0 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f7838034040 tx=0x7f7838004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:15:05.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.153+0000 7f784cc00640 1 -- 172.21.15.96:0/2298958580 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7838040020 con 0x7f7850159740 2024-05-28T06:15:05.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.153+0000 7f784cc00640 1 -- 172.21.15.96:0/2298958580 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7838037040 con 0x7f7850159740 2024-05-28T06:15:05.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.153+0000 7f784cc00640 1 -- 172.21.15.96:0/2298958580 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7838036030 con 0x7f7850159740 2024-05-28T06:15:05.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.153+0000 7f7855600640 1 -- 172.21.15.96:0/2298958580 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f785015fb50 con 0x7f7850159740 2024-05-28T06:15:05.159 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.153+0000 7f7855600640 1 -- 172.21.15.96:0/2298958580 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7850160110 con 0x7f7850159740 2024-05-28T06:15:05.159 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.154+0000 7f7855600640 1 -- 172.21.15.96:0/2298958580 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78500689e0 con 0x7f7850159740 2024-05-28T06:15:05.159 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.155+0000 7f784cc00640 1 -- 172.21.15.96:0/2298958580 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 22) ==== 92213+0+0 (secure 0 0 0) 0x7f7838043020 con 0x7f7850159740 2024-05-28T06:15:05.159 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.156+0000 7f784cc00640 1 --2- 172.21.15.96:0/2298958580 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f782c06d5e0 0x7f782c06faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:05.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.156+0000 7f784cc00640 1 -- 172.21.15.96:0/2298958580 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 5737+0+0 (secure 0 0 0) 0x7f78380aaa50 con 0x7f7850159740 2024-05-28T06:15:05.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.156+0000 7f784fe00640 1 --2- 172.21.15.96:0/2298958580 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f782c06d5e0 0x7f782c06faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:05.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.158+0000 7f784fe00640 1 --2- 172.21.15.96:0/2298958580 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f782c06d5e0 0x7f782c06faa0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f7840009870 tx=0x7f7840008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:15:05.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.160+0000 7f784cc00640 1 -- 172.21.15.96:0/2298958580 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f783804a030 con 0x7f7850159740 2024-05-28T06:15:05.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.397+0000 7f7855600640 1 -- 172.21.15.96:0/2298958580 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) -- 0x7f78500ba500 con 0x7f782c06d5e0 2024-05-28T06:15:05.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.408+0000 7f784cc00640 1 -- 172.21.15.96:0/2298958580 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+59 (secure 0 0 0) 0x7f78500ba500 con 0x7f782c06d5e0 2024-05-28T06:15:05.410 INFO:teuthology.orchestra.run.smithi096.stdout:/volumes/_nogroup/sv1/d9773566-6392-4a75-bc78-1cafc122ae04 2024-05-28T06:15:05.412 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.411+0000 7f7855600640 1 -- 172.21.15.96:0/2298958580 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f782c06d5e0 msgr2=0x7f782c06faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:05.412 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.411+0000 7f7855600640 1 --2- 172.21.15.96:0/2298958580 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f782c06d5e0 0x7f782c06faa0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f7840009870 tx=0x7f7840008040 comp rx=0 tx=0).stop 2024-05-28T06:15:05.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.411+0000 7f7855600640 1 -- 172.21.15.96:0/2298958580 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7850159740 msgr2=0x7f78500b96f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:05.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.411+0000 7f7855600640 1 --2- 172.21.15.96:0/2298958580 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7850159740 0x7f78500b96f0 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f7838034040 tx=0x7f7838004730 comp rx=0 tx=0).stop 2024-05-28T06:15:05.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.411+0000 7f7855600640 1 -- 172.21.15.96:0/2298958580 shutdown_connections 2024-05-28T06:15:05.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.411+0000 7f7855600640 1 --2- 172.21.15.96:0/2298958580 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f782c06d5e0 0x7f782c06faa0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:05.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.411+0000 7f7855600640 1 --2- 172.21.15.96:0/2298958580 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7850159bc0 0x7f78500b9c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:05.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.411+0000 7f7855600640 1 --2- 172.21.15.96:0/2298958580 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7850159740 0x7f78500b96f0 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:05.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.411+0000 7f7855600640 1 --2- 172.21.15.96:0/2298958580 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7834003f80 0x7f7850160bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:05.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.411+0000 7f7855600640 1 -- 172.21.15.96:0/2298958580 >> 172.21.15.96:0/2298958580 conn(0x7f785008bfc0 msgr2=0x7f78500b5e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:15:05.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.411+0000 7f7855600640 1 -- 172.21.15.96:0/2298958580 shutdown_connections 2024-05-28T06:15:05.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:05.411+0000 7f7855600640 1 -- 172.21.15.96:0/2298958580 wait complete. 2024-05-28T06:15:05.431 DEBUG:tasks.cephfs.test_volumes:filling subvolume sv1 with 3 files each 1024MB size under directory volumes/_nogroup/sv1/d9773566-6392-4a75-bc78-1cafc122ae04 2024-05-28T06:15:05.431 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:15:05.431 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:15:05.482 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:15:05.482 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/d9773566-6392-4a75-bc78-1cafc122ae04/subvolume_file.0 bs=1M conv=fdatasync count=1024 seek=0) 2024-05-28T06:15:19.678 INFO:teuthology.orchestra.run.smithi096.stderr:1024+0 records in 2024-05-28T06:15:19.678 INFO:teuthology.orchestra.run.smithi096.stderr:1024+0 records out 2024-05-28T06:15:19.678 INFO:teuthology.orchestra.run.smithi096.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 14.0676 s, 76.3 MB/s 2024-05-28T06:15:19.679 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:15:19.679 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:15:19.734 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:15:19.735 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/d9773566-6392-4a75-bc78-1cafc122ae04/subvolume_file.1 bs=1M conv=fdatasync count=1024 seek=0) 2024-05-28T06:15:32.014 DEBUG:teuthology.orchestra.run.smithi096:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:15:32.021 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:15:32.294 INFO:teuthology.orchestra.run.smithi096.stderr:1024+0 records in 2024-05-28T06:15:32.294 INFO:teuthology.orchestra.run.smithi096.stderr:1024+0 records out 2024-05-28T06:15:32.295 INFO:teuthology.orchestra.run.smithi096.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 12.4973 s, 85.9 MB/s 2024-05-28T06:15:32.296 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:15:32.296 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:15:32.347 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:15:32.348 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/d9773566-6392-4a75-bc78-1cafc122ae04/subvolume_file.2 bs=1M conv=fdatasync count=1024 seek=0) 2024-05-28T06:15:44.000 INFO:teuthology.orchestra.run.smithi096.stderr:1024+0 records in 2024-05-28T06:15:44.001 INFO:teuthology.orchestra.run.smithi096.stderr:1024+0 records out 2024-05-28T06:15:44.001 INFO:teuthology.orchestra.run.smithi096.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 11.5919 s, 92.6 MB/s 2024-05-28T06:15:44.003 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot create cephfs sv1 ss1 2024-05-28T06:15:44.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.213+0000 7f31a5000640 1 Processor -- start 2024-05-28T06:15:44.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.214+0000 7f31a5000640 1 -- start start 2024-05-28T06:15:44.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.215+0000 7f31a5000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f31a006cea0 0x7f31a015d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:44.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.215+0000 7f31a5000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f31a00b5bc0 0x7f31a00add70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:44.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.215+0000 7f31a5000640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31a00b0340 con 0x7f31a00ae2b0 2024-05-28T06:15:44.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.215+0000 7f31a5000640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f31a006d280 con 0x7f31a00b5bc0 2024-05-28T06:15:44.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.215+0000 7f31a5000640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f31a01603c0 con 0x7f31a006cea0 2024-05-28T06:15:44.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.216+0000 7f319d400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f31a00b5bc0 0x7f31a00add70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:44.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.216+0000 7f319d400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f31a00b5bc0 0x7f31a00add70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:59926/0 (socket says 172.21.15.96:59926) 2024-05-28T06:15:44.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.216+0000 7f319d400640 1 -- 172.21.15.96:0/3408627355 learned_addr learned my addr 172.21.15.96:0/3408627355 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:15:44.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.216+0000 7f319de00640 1 --2- 172.21.15.96:0/3408627355 >> v2:172.21.15.123:3300/0 conn(0x7f31a006cea0 0x7f31a015d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:44.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.216+0000 7f319d400640 1 -- 172.21.15.96:0/3408627355 >> v2:172.21.15.123:3300/0 conn(0x7f31a006cea0 msgr2=0x7f31a015d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:15:44.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.216+0000 7f319d400640 1 --2- 172.21.15.96:0/3408627355 >> v2:172.21.15.123:3300/0 conn(0x7f31a006cea0 0x7f31a015d4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:44.218 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.216+0000 7f319d400640 1 -- 172.21.15.96:0/3408627355 >> v1:172.21.15.123:6789/0 conn(0x7f31a00ae2b0 legacy=0x7f31a0152d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:15:44.218 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.216+0000 7f319d400640 1 -- 172.21.15.96:0/3408627355 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31a01606a0 con 0x7f31a00b5bc0 2024-05-28T06:15:44.218 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.217+0000 7f319d400640 1 --2- 172.21.15.96:0/3408627355 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f31a00b5bc0 0x7f31a00add70 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f31a006d4d0 tx=0x7f318802a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4a782481019db05 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:15:44.218 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.218+0000 7f319ca00640 1 -- 172.21.15.96:0/3408627355 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3188036070 con 0x7f31a00b5bc0 2024-05-28T06:15:44.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.218+0000 7f319ca00640 1 -- 172.21.15.96:0/3408627355 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3188031600 con 0x7f31a00b5bc0 2024-05-28T06:15:44.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.218+0000 7f319ca00640 1 -- 172.21.15.96:0/3408627355 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31880319d0 con 0x7f31a00b5bc0 2024-05-28T06:15:44.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.218+0000 7f31a5000640 1 -- 172.21.15.96:0/3408627355 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f31a00b5bc0 msgr2=0x7f31a00add70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:44.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.218+0000 7f31a5000640 1 --2- 172.21.15.96:0/3408627355 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f31a00b5bc0 0x7f31a00add70 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f31a006d4d0 tx=0x7f318802a6f0 comp rx=0 tx=0).stop 2024-05-28T06:15:44.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.218+0000 7f31a5000640 1 -- 172.21.15.96:0/3408627355 shutdown_connections 2024-05-28T06:15:44.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.218+0000 7f31a5000640 1 --2- 172.21.15.96:0/3408627355 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f31a00b5bc0 0x7f31a00add70 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:44.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.218+0000 7f31a5000640 1 --2- 172.21.15.96:0/3408627355 >> v2:172.21.15.123:3300/0 conn(0x7f31a006cea0 0x7f31a015d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:44.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.218+0000 7f31a5000640 1 -- 172.21.15.96:0/3408627355 >> 172.21.15.96:0/3408627355 conn(0x7f31a00d0b20 msgr2=0x7f31a00d0f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:15:44.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.218+0000 7f31a5000640 1 -- 172.21.15.96:0/3408627355 shutdown_connections 2024-05-28T06:15:44.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.218+0000 7f31a5000640 1 -- 172.21.15.96:0/3408627355 wait complete. 2024-05-28T06:15:44.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.220+0000 7f31a5000640 1 Processor -- start 2024-05-28T06:15:44.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.220+0000 7f31a5000640 1 -- start start 2024-05-28T06:15:44.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.221+0000 7f31a5000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f31a006cea0 0x7f31a014b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:44.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.221+0000 7f31a5000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f31a00ae2b0 0x7f31a01503c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:44.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.221+0000 7f31a5000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f31a00b5bc0 0x7f31a006f420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:44.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.221+0000 7f31a5000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f31a015d9e0 con 0x7f31a006cea0 2024-05-28T06:15:44.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.221+0000 7f31a5000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f31a00aff10 con 0x7f31a00ae2b0 2024-05-28T06:15:44.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.221+0000 7f31a5000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f31a0150900 con 0x7f31a00b5bc0 2024-05-28T06:15:44.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.221+0000 7f319de00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f31a006cea0 0x7f31a014b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:44.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.221+0000 7f319de00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f31a006cea0 0x7f31a014b6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:59942/0 (socket says 172.21.15.96:59942) 2024-05-28T06:15:44.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.222+0000 7f319de00640 1 -- 172.21.15.96:0/3233939361 learned_addr learned my addr 172.21.15.96:0/3233939361 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:15:44.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.222+0000 7f319de00640 1 -- 172.21.15.96:0/3233939361 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f31a00b5bc0 msgr2=0x7f31a006f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:44.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.222+0000 7f319e800640 1 --2- 172.21.15.96:0/3233939361 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f31a00b5bc0 0x7f31a006f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:44.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.222+0000 7f319de00640 1 --2- 172.21.15.96:0/3233939361 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f31a00b5bc0 0x7f31a006f420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:44.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.222+0000 7f319d400640 1 --2- 172.21.15.96:0/3233939361 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f31a00ae2b0 0x7f31a01503c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:44.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.222+0000 7f319de00640 1 -- 172.21.15.96:0/3233939361 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f31a00ae2b0 msgr2=0x7f31a01503c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:44.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.222+0000 7f319de00640 1 --2- 172.21.15.96:0/3233939361 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f31a00ae2b0 0x7f31a01503c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:44.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.222+0000 7f319de00640 1 -- 172.21.15.96:0/3233939361 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31a006ff90 con 0x7f31a006cea0 2024-05-28T06:15:44.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.222+0000 7f319de00640 1 --2- 172.21.15.96:0/3233939361 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f31a006cea0 0x7f31a014b6e0 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f319000c820 tx=0x7f319000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:15:44.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.223+0000 7f3196000640 1 -- 172.21.15.96:0/3233939361 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3190015070 con 0x7f31a006cea0 2024-05-28T06:15:44.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.223+0000 7f3196000640 1 -- 172.21.15.96:0/3233939361 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3190004240 con 0x7f31a006cea0 2024-05-28T06:15:44.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.223+0000 7f3196000640 1 -- 172.21.15.96:0/3233939361 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f319000ce10 con 0x7f31a006cea0 2024-05-28T06:15:44.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.223+0000 7f31a5000640 1 -- 172.21.15.96:0/3233939361 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31a014f8e0 con 0x7f31a006cea0 2024-05-28T06:15:44.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.223+0000 7f31a5000640 1 -- 172.21.15.96:0/3233939361 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31a014f2f0 con 0x7f31a006cea0 2024-05-28T06:15:44.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.224+0000 7f31a5000640 1 -- 172.21.15.96:0/3233939361 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31a00690f0 con 0x7f31a006cea0 2024-05-28T06:15:44.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.225+0000 7f3196000640 1 -- 172.21.15.96:0/3233939361 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 23) ==== 92315+0+0 (secure 0 0 0) 0x7f3190007830 con 0x7f31a006cea0 2024-05-28T06:15:44.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.226+0000 7f3196000640 1 --2- 172.21.15.96:0/3233939361 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f317c06d700 0x7f317c06fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:44.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.226+0000 7f3196000640 1 -- 172.21.15.96:0/3233939361 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 5737+0+0 (secure 0 0 0) 0x7f319008a7e0 con 0x7f31a006cea0 2024-05-28T06:15:44.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.227+0000 7f319d400640 1 --2- 172.21.15.96:0/3233939361 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f317c06d700 0x7f317c06fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:44.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.228+0000 7f319d400640 1 --2- 172.21.15.96:0/3233939361 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f317c06d700 0x7f317c06fbc0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f3188033000 tx=0x7f3188009000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:15:44.231 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.230+0000 7f3196000640 1 -- 172.21.15.96:0/3233939361 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f319005a3a0 con 0x7f31a006cea0 2024-05-28T06:15:44.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.464+0000 7f31a5000640 1 -- 172.21.15.96:0/3233939361 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""]}) -- 0x7f31a0070270 con 0x7f317c06d700 2024-05-28T06:15:44.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.503+0000 7f3196000640 1 -- 172.21.15.96:0/3233939361 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f31a0070270 con 0x7f317c06d700 2024-05-28T06:15:44.507 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.505+0000 7f31a5000640 1 -- 172.21.15.96:0/3233939361 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f317c06d700 msgr2=0x7f317c06fbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:44.507 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.505+0000 7f31a5000640 1 --2- 172.21.15.96:0/3233939361 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f317c06d700 0x7f317c06fbc0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f3188033000 tx=0x7f3188009000 comp rx=0 tx=0).stop 2024-05-28T06:15:44.507 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.505+0000 7f31a5000640 1 -- 172.21.15.96:0/3233939361 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f31a006cea0 msgr2=0x7f31a014b6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:44.507 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.505+0000 7f31a5000640 1 --2- 172.21.15.96:0/3233939361 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f31a006cea0 0x7f31a014b6e0 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f319000c820 tx=0x7f319000cce0 comp rx=0 tx=0).stop 2024-05-28T06:15:44.507 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.506+0000 7f31a5000640 1 -- 172.21.15.96:0/3233939361 shutdown_connections 2024-05-28T06:15:44.507 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.506+0000 7f31a5000640 1 --2- 172.21.15.96:0/3233939361 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f317c06d700 0x7f317c06fbc0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:44.507 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.506+0000 7f31a5000640 1 --2- 172.21.15.96:0/3233939361 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f31a00b5bc0 0x7f31a006f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:44.507 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.506+0000 7f31a5000640 1 --2- 172.21.15.96:0/3233939361 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f31a00ae2b0 0x7f31a01503c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:44.507 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.506+0000 7f31a5000640 1 --2- 172.21.15.96:0/3233939361 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f31a006cea0 0x7f31a014b6e0 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:44.507 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.506+0000 7f31a5000640 1 -- 172.21.15.96:0/3233939361 >> 172.21.15.96:0/3233939361 conn(0x7f31a00d0b20 msgr2=0x7f31a015ecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:15:44.508 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.506+0000 7f31a5000640 1 -- 172.21.15.96:0/3233939361 shutdown_connections 2024-05-28T06:15:44.508 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.506+0000 7f31a5000640 1 -- 172.21.15.96:0/3233939361 wait complete. 2024-05-28T06:15:44.526 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-05-28T06:15:44.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.723+0000 7fb78aa00640 1 Processor -- start 2024-05-28T06:15:44.725 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.724+0000 7fb78aa00640 1 -- start start 2024-05-28T06:15:44.727 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.725+0000 7fb78aa00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fb784161a90 0x7fb78406e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:44.727 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.725+0000 7fb78aa00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fb784161e70 0x7fb7840b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:44.727 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.725+0000 7fb78aa00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7840b3d40 con 0x7fb78406e9b0 2024-05-28T06:15:44.727 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.725+0000 7fb78aa00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fb7840b4020 con 0x7fb784161e70 2024-05-28T06:15:44.727 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.725+0000 7fb78aa00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fb7840b4300 con 0x7fb784161a90 2024-05-28T06:15:44.727 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.726+0000 7fb783400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fb784161e70 0x7fb7840b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:44.728 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.726+0000 7fb783400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fb784161e70 0x7fb7840b32c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:59948/0 (socket says 172.21.15.96:59948) 2024-05-28T06:15:44.728 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.726+0000 7fb783400640 1 -- 172.21.15.96:0/252934592 learned_addr learned my addr 172.21.15.96:0/252934592 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:15:44.728 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.726+0000 7fb783e00640 1 --2- 172.21.15.96:0/252934592 >> v2:172.21.15.123:3300/0 conn(0x7fb784161a90 0x7fb78406e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:44.728 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.726+0000 7fb783400640 1 -- 172.21.15.96:0/252934592 >> v2:172.21.15.123:3300/0 conn(0x7fb784161a90 msgr2=0x7fb78406e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:15:44.728 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.726+0000 7fb783400640 1 --2- 172.21.15.96:0/252934592 >> v2:172.21.15.123:3300/0 conn(0x7fb784161a90 0x7fb78406e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:44.728 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.726+0000 7fb783400640 1 -- 172.21.15.96:0/252934592 >> v1:172.21.15.123:6789/0 conn(0x7fb78406e9b0 legacy=0x7fb7840ad2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:15:44.728 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.726+0000 7fb783400640 1 -- 172.21.15.96:0/252934592 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb78406dc70 con 0x7fb784161e70 2024-05-28T06:15:44.729 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.727+0000 7fb783400640 1 --2- 172.21.15.96:0/252934592 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb784161e70 0x7fb7840b32c0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7fb78406fe90 tx=0x7fb76c02a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a93d7fa7e41c995a server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:15:44.729 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.728+0000 7fb782a00640 1 -- 172.21.15.96:0/252934592 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb76c037070 con 0x7fb784161e70 2024-05-28T06:15:44.730 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.728+0000 7fb782a00640 1 -- 172.21.15.96:0/252934592 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb76c032440 con 0x7fb784161e70 2024-05-28T06:15:44.730 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.728+0000 7fb782a00640 1 -- 172.21.15.96:0/252934592 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb76c032810 con 0x7fb784161e70 2024-05-28T06:15:44.730 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.729+0000 7fb78aa00640 1 -- 172.21.15.96:0/252934592 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb784161e70 msgr2=0x7fb7840b32c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:44.730 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.729+0000 7fb78aa00640 1 --2- 172.21.15.96:0/252934592 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb784161e70 0x7fb7840b32c0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7fb78406fe90 tx=0x7fb76c02a230 comp rx=0 tx=0).stop 2024-05-28T06:15:44.730 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.729+0000 7fb78aa00640 1 -- 172.21.15.96:0/252934592 shutdown_connections 2024-05-28T06:15:44.730 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.729+0000 7fb78aa00640 1 --2- 172.21.15.96:0/252934592 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb784161e70 0x7fb7840b32c0 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:44.730 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.729+0000 7fb78aa00640 1 --2- 172.21.15.96:0/252934592 >> v2:172.21.15.123:3300/0 conn(0x7fb784161a90 0x7fb78406e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:44.730 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.729+0000 7fb78aa00640 1 -- 172.21.15.96:0/252934592 >> 172.21.15.96:0/252934592 conn(0x7fb78406bd20 msgr2=0x7fb78406c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:15:44.730 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.729+0000 7fb78aa00640 1 -- 172.21.15.96:0/252934592 shutdown_connections 2024-05-28T06:15:44.730 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.729+0000 7fb78aa00640 1 -- 172.21.15.96:0/252934592 wait complete. 2024-05-28T06:15:44.731 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.731+0000 7fb78aa00640 1 Processor -- start 2024-05-28T06:15:44.732 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.731+0000 7fb78aa00640 1 -- start start 2024-05-28T06:15:44.734 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.732+0000 7fb78aa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb78406e9b0 0x7fb784155be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:44.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.732+0000 7fb78aa00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb784161a90 0x7fb784143410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:44.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.732+0000 7fb78aa00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb784161e70 0x7fb78415fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:44.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.732+0000 7fb78aa00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb7840ad9b0 con 0x7fb78406e9b0 2024-05-28T06:15:44.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.732+0000 7fb78aa00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fb7840b5020 con 0x7fb784161e70 2024-05-28T06:15:44.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.732+0000 7fb78aa00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fb7840adb30 con 0x7fb784161a90 2024-05-28T06:15:44.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.733+0000 7fb783e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb78406e9b0 0x7fb784155be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:44.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.733+0000 7fb783e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb78406e9b0 0x7fb784155be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:59960/0 (socket says 172.21.15.96:59960) 2024-05-28T06:15:44.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.733+0000 7fb783e00640 1 -- 172.21.15.96:0/176053665 learned_addr learned my addr 172.21.15.96:0/176053665 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:15:44.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.733+0000 7fb783400640 1 --2- 172.21.15.96:0/176053665 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb784161a90 0x7fb784143410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:44.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.733+0000 7fb783e00640 1 -- 172.21.15.96:0/176053665 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb784161a90 msgr2=0x7fb784143410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:44.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.733+0000 7fb783e00640 1 --2- 172.21.15.96:0/176053665 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb784161a90 0x7fb784143410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:44.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.733+0000 7fb783e00640 1 -- 172.21.15.96:0/176053665 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb784161e70 msgr2=0x7fb78415fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:44.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.733+0000 7fb783e00640 1 --2- 172.21.15.96:0/176053665 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb784161e70 0x7fb78415fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:44.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.733+0000 7fb783e00640 1 -- 172.21.15.96:0/176053665 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb784143950 con 0x7fb78406e9b0 2024-05-28T06:15:44.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.733+0000 7fb783e00640 1 --2- 172.21.15.96:0/176053665 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb78406e9b0 0x7fb784155be0 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7fb77400d800 tx=0x7fb77400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:15:44.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.734+0000 7fb77be00640 1 -- 172.21.15.96:0/176053665 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb77400b6c0 con 0x7fb78406e9b0 2024-05-28T06:15:44.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.734+0000 7fb77be00640 1 -- 172.21.15.96:0/176053665 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb774019070 con 0x7fb78406e9b0 2024-05-28T06:15:44.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.734+0000 7fb77be00640 1 -- 172.21.15.96:0/176053665 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb77400ba80 con 0x7fb78406e9b0 2024-05-28T06:15:44.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.734+0000 7fb78aa00640 1 -- 172.21.15.96:0/176053665 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb78415ec00 con 0x7fb78406e9b0 2024-05-28T06:15:44.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.734+0000 7fb78aa00640 1 -- 172.21.15.96:0/176053665 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb78415f5b0 con 0x7fb78406e9b0 2024-05-28T06:15:44.737 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.735+0000 7fb78aa00640 1 -- 172.21.15.96:0/176053665 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7840ae910 con 0x7fb78406e9b0 2024-05-28T06:15:44.737 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.736+0000 7fb77be00640 1 -- 172.21.15.96:0/176053665 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 23) ==== 92315+0+0 (secure 0 0 0) 0x7fb77400bc20 con 0x7fb78406e9b0 2024-05-28T06:15:44.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.737+0000 7fb77be00640 1 --2- 172.21.15.96:0/176053665 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb76006d700 0x7fb76006fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:44.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.737+0000 7fb77be00640 1 -- 172.21.15.96:0/176053665 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 5737+0+0 (secure 0 0 0) 0x7fb77408ae50 con 0x7fb78406e9b0 2024-05-28T06:15:44.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.738+0000 7fb783400640 1 --2- 172.21.15.96:0/176053665 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb76006d700 0x7fb76006fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:44.740 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.739+0000 7fb783400640 1 --2- 172.21.15.96:0/176053665 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb76006d700 0x7fb76006fbc0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fb76c034000 tx=0x7fb76c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:15:44.742 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.741+0000 7fb77be00640 1 -- 172.21.15.96:0/176053665 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb7840ae910 con 0x7fb78406e9b0 2024-05-28T06:15:44.974 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.973+0000 7fb78aa00640 1 -- 172.21.15.96:0/176053665 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) -- 0x7fb78415f0d0 con 0x7fb76006d700 2024-05-28T06:15:44.986 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.985+0000 7fb77be00640 1 -- 172.21.15.96:0/176053665 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+59 (secure 0 0 0) 0x7fb78415f0d0 con 0x7fb76006d700 2024-05-28T06:15:44.986 INFO:teuthology.orchestra.run.smithi096.stdout:/volumes/_nogroup/sv1/d9773566-6392-4a75-bc78-1cafc122ae04 2024-05-28T06:15:44.988 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.987+0000 7fb78aa00640 1 -- 172.21.15.96:0/176053665 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb76006d700 msgr2=0x7fb76006fbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:44.988 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.987+0000 7fb78aa00640 1 --2- 172.21.15.96:0/176053665 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb76006d700 0x7fb76006fbc0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fb76c034000 tx=0x7fb76c02e000 comp rx=0 tx=0).stop 2024-05-28T06:15:44.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.987+0000 7fb78aa00640 1 -- 172.21.15.96:0/176053665 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb78406e9b0 msgr2=0x7fb784155be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:44.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.987+0000 7fb78aa00640 1 --2- 172.21.15.96:0/176053665 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb78406e9b0 0x7fb784155be0 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7fb77400d800 tx=0x7fb77400dcc0 comp rx=0 tx=0).stop 2024-05-28T06:15:44.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.987+0000 7fb78aa00640 1 -- 172.21.15.96:0/176053665 shutdown_connections 2024-05-28T06:15:44.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.987+0000 7fb78aa00640 1 --2- 172.21.15.96:0/176053665 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb76006d700 0x7fb76006fbc0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:44.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.987+0000 7fb78aa00640 1 --2- 172.21.15.96:0/176053665 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb784161e70 0x7fb78415fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:44.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.987+0000 7fb78aa00640 1 --2- 172.21.15.96:0/176053665 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb784161a90 0x7fb784143410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:44.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.987+0000 7fb78aa00640 1 --2- 172.21.15.96:0/176053665 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb78406e9b0 0x7fb784155be0 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:44.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.987+0000 7fb78aa00640 1 -- 172.21.15.96:0/176053665 >> 172.21.15.96:0/176053665 conn(0x7fb78406bd20 msgr2=0x7fb784071c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:15:44.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.987+0000 7fb78aa00640 1 -- 172.21.15.96:0/176053665 shutdown_connections 2024-05-28T06:15:44.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:44.988+0000 7fb78aa00640 1 -- 172.21.15.96:0/176053665 wait complete. 2024-05-28T06:15:45.007 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:15:45.007 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/d9773566-6392-4a75-bc78-1cafc122ae04 2024-05-28T06:15:45.007 DEBUG:teuthology.orchestra.run.smithi096:> ) 2024-05-28T06:15:45.066 INFO:teuthology.orchestra.run.smithi096.stdout:# file: volumes/_nogroup/sv1/d9773566-6392-4a75-bc78-1cafc122ae04 2024-05-28T06:15:45.066 INFO:teuthology.orchestra.run.smithi096.stdout:ceph.dir.rbytes="3138256896" 2024-05-28T06:15:45.067 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:15:45.067 INFO:tasks.cephfs.test_volumes:r_size = 3138256896 exp_size = 3221225472 2024-05-28T06:15:47.068 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:15:47.068 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/d9773566-6392-4a75-bc78-1cafc122ae04 2024-05-28T06:15:47.068 DEBUG:teuthology.orchestra.run.smithi096:> ) 2024-05-28T06:15:47.135 INFO:teuthology.orchestra.run.smithi096.stdout:# file: volumes/_nogroup/sv1/d9773566-6392-4a75-bc78-1cafc122ae04 2024-05-28T06:15:47.136 INFO:teuthology.orchestra.run.smithi096.stdout:ceph.dir.rbytes="3221225472" 2024-05-28T06:15:47.136 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:15:47.137 INFO:tasks.cephfs.test_volumes:r_size = 3221225472 exp_size = 3221225472 2024-05-28T06:15:47.138 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone43631 2024-05-28T06:15:47.350 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:47.349+0000 7fdb7e200640 1 Processor -- start 2024-05-28T06:15:47.350 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:47.350+0000 7fdb7e200640 1 -- start start 2024-05-28T06:15:47.352 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:47.351+0000 7fdb7e200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fdb7809aa30 0x7fdb7809ae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:47.352 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:47.351+0000 7fdb7e200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdb780d1690 0x7fdb78162c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:47.352 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:47.351+0000 7fdb7e200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb7809b350 con 0x7fdb780d1230 2024-05-28T06:15:47.352 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:47.351+0000 7fdb7e200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fdb7806cd80 con 0x7fdb7809aa30 2024-05-28T06:15:47.352 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:47.351+0000 7fdb7e200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fdb7806d060 con 0x7fdb780d1690 2024-05-28T06:15:47.352 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:47.351+0000 7fdb77400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fdb7809aa30 0x7fdb7809ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:47.352 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:47.351+0000 7fdb77400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fdb7809aa30 0x7fdb7809ae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:59964/0 (socket says 172.21.15.96:59964) 2024-05-28T06:15:47.352 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:47.351+0000 7fdb77400640 1 -- 172.21.15.96:0/852230683 learned_addr learned my addr 172.21.15.96:0/852230683 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:15:47.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:47.351+0000 7fdb76a00640 1 --1- 172.21.15.96:0/852230683 >> v1:172.21.15.96:6789/0 conn(0x7fdb780d1230 0x7fdb78162540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:59122/0 (socket says 172.21.15.96:59122) 2024-05-28T06:15:47.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:47.352+0000 7fdb77400640 1 -- 172.21.15.96:0/852230683 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdb780d1690 msgr2=0x7fdb78162c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:15:47.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:47.352+0000 7fdb77400640 1 --2- 172.21.15.96:0/852230683 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdb780d1690 0x7fdb78162c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:47.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:47.352+0000 7fdb77400640 1 -- 172.21.15.96:0/852230683 >> v1:172.21.15.96:6789/0 conn(0x7fdb780d1230 legacy=0x7fdb78162540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:47.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:47.352+0000 7fdb77400640 1 -- 172.21.15.96:0/852230683 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb7806d340 con 0x7fdb7809aa30 2024-05-28T06:15:47.354 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:47.353+0000 7fdb77400640 1 --2- 172.21.15.96:0/852230683 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb7809aa30 0x7fdb7809ae10 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7fdb68009870 tx=0x7fdb6802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10e849f94100619e server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:15:47.354 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:47.353+0000 7fdb76000640 1 -- 172.21.15.96:0/852230683 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb68037070 con 0x7fdb7809aa30 2024-05-28T06:15:47.354 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:47.353+0000 7fdb76000640 1 -- 172.21.15.96:0/852230683 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb68032440 con 0x7fdb7809aa30 2024-05-28T06:15:47.354 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:47.353+0000 7fdb76000640 1 -- 172.21.15.96:0/852230683 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb68032810 con 0x7fdb7809aa30 2024-05-28T06:15:47.355 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:47.354+0000 7fdb7e200640 1 -- 172.21.15.96:0/852230683 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb7809aa30 msgr2=0x7fdb7809ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:47.355 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:47.354+0000 7fdb7e200640 1 --2- 172.21.15.96:0/852230683 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb7809aa30 0x7fdb7809ae10 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7fdb68009870 tx=0x7fdb6802a1e0 comp rx=0 tx=0).stop 2024-05-28T06:15:47.355 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:47.354+0000 7fdb7e200640 1 -- 172.21.15.96:0/852230683 shutdown_connections 2024-05-28T06:15:47.355 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:47.354+0000 7fdb7e200640 1 --2- 172.21.15.96:0/852230683 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdb780d1690 0x7fdb78162c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:47.355 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:47.354+0000 7fdb7e200640 1 --2- 172.21.15.96:0/852230683 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb7809aa30 0x7fdb7809ae10 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:47.355 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:47.354+0000 7fdb7e200640 1 -- 172.21.15.96:0/852230683 >> 172.21.15.96:0/852230683 conn(0x7fdb780b3f20 msgr2=0x7fdb780b4330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:15:47.355 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:47.354+0000 7fdb7e200640 1 -- 172.21.15.96:0/852230683 shutdown_connections 2024-05-28T06:15:47.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:47.354+0000 7fdb7e200640 1 -- 172.21.15.96:0/852230683 wait complete. 2024-05-28T06:15:47.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:47.356+0000 7fdb7e200640 1 Processor -- start 2024-05-28T06:15:47.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:47.356+0000 7fdb7e200640 1 -- start start 2024-05-28T06:15:47.358 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:47.357+0000 7fdb7e200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb7809aa30 0x7fdb7815d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:47.358 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:47.357+0000 7fdb7e200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb780d1230 0x7fdb7815ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:47.358 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:47.357+0000 7fdb7e200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdb780d1690 0x7fdb781f8470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:47.358 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:47.357+0000 7fdb7e200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fdb78162230 con 0x7fdb7809aa30 2024-05-28T06:15:47.358 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:47.357+0000 7fdb7e200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fdb7806d900 con 0x7fdb780d1230 2024-05-28T06:15:47.358 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:47.357+0000 7fdb7e200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fdb7806ca10 con 0x7fdb780d1690 2024-05-28T06:15:47.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:47.357+0000 7fdb77400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb7809aa30 0x7fdb7815d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:47.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:47.357+0000 7fdb77400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb7809aa30 0x7fdb7815d830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:59980/0 (socket says 172.21.15.96:59980) 2024-05-28T06:15:47.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:47.357+0000 7fdb77400640 1 -- 172.21.15.96:0/2794944786 learned_addr learned my addr 172.21.15.96:0/2794944786 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:15:47.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:47.358+0000 7fdb77400640 1 -- 172.21.15.96:0/2794944786 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdb780d1690 msgr2=0x7fdb781f8470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:47.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:47.358+0000 7fdb77400640 1 --2- 172.21.15.96:0/2794944786 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdb780d1690 0x7fdb781f8470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:47.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:47.358+0000 7fdb77400640 1 -- 172.21.15.96:0/2794944786 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb780d1230 msgr2=0x7fdb7815ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:47.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:47.358+0000 7fdb77400640 1 --2- 172.21.15.96:0/2794944786 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb780d1230 0x7fdb7815ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:47.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:47.358+0000 7fdb77400640 1 -- 172.21.15.96:0/2794944786 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb781f4e80 con 0x7fdb7809aa30 2024-05-28T06:15:47.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:47.358+0000 7fdb77400640 1 --2- 172.21.15.96:0/2794944786 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb7809aa30 0x7fdb7815d830 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7fdb68009870 tx=0x7fdb680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:15:47.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:47.358+0000 7fdb6f400640 1 -- 172.21.15.96:0/2794944786 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb68041020 con 0x7fdb7809aa30 2024-05-28T06:15:47.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:47.358+0000 7fdb6f400640 1 -- 172.21.15.96:0/2794944786 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb68030970 con 0x7fdb7809aa30 2024-05-28T06:15:47.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:47.358+0000 7fdb6f400640 1 -- 172.21.15.96:0/2794944786 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb68037040 con 0x7fdb7809aa30 2024-05-28T06:15:47.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:47.358+0000 7fdb7e200640 1 -- 172.21.15.96:0/2794944786 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb781f89b0 con 0x7fdb7809aa30 2024-05-28T06:15:47.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:47.359+0000 7fdb7e200640 1 -- 172.21.15.96:0/2794944786 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb781f8f60 con 0x7fdb7809aa30 2024-05-28T06:15:47.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:47.360+0000 7fdb7e200640 1 -- 172.21.15.96:0/2794944786 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb780cf180 con 0x7fdb7809aa30 2024-05-28T06:15:47.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:47.360+0000 7fdb6f400640 1 -- 172.21.15.96:0/2794944786 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 23) ==== 92315+0+0 (secure 0 0 0) 0x7fdb68044020 con 0x7fdb7809aa30 2024-05-28T06:15:47.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:47.361+0000 7fdb6f400640 1 --2- 172.21.15.96:0/2794944786 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdb5406d700 0x7fdb5406fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:47.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:47.361+0000 7fdb6f400640 1 -- 172.21.15.96:0/2794944786 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 5737+0+0 (secure 0 0 0) 0x7fdb680ab2f0 con 0x7fdb7809aa30 2024-05-28T06:15:47.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:47.362+0000 7fdb76a00640 1 --2- 172.21.15.96:0/2794944786 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdb5406d700 0x7fdb5406fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:47.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:47.363+0000 7fdb76a00640 1 --2- 172.21.15.96:0/2794944786 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdb5406d700 0x7fdb5406fbc0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fdb60007fd0 tx=0x7fdb60007d80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:15:47.367 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:47.366+0000 7fdb6f400640 1 -- 172.21.15.96:0/2794944786 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fdb780cf180 con 0x7fdb7809aa30 2024-05-28T06:15:47.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:47.601+0000 7fdb7e200640 1 -- 172.21.15.96:0/2794944786 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone43631", "target": ["mon-mgr", ""]}) -- 0x7fdb780689e0 con 0x7fdb5406d700 2024-05-28T06:15:47.767 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:47.767+0000 7fdb6f400640 1 -- 172.21.15.96:0/2794944786 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fdb780689e0 con 0x7fdb5406d700 2024-05-28T06:15:47.770 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:47.770+0000 7fdb7e200640 1 -- 172.21.15.96:0/2794944786 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdb5406d700 msgr2=0x7fdb5406fbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:47.770 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:47.770+0000 7fdb7e200640 1 --2- 172.21.15.96:0/2794944786 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdb5406d700 0x7fdb5406fbc0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fdb60007fd0 tx=0x7fdb60007d80 comp rx=0 tx=0).stop 2024-05-28T06:15:47.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:47.770+0000 7fdb7e200640 1 -- 172.21.15.96:0/2794944786 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb7809aa30 msgr2=0x7fdb7815d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:47.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:47.770+0000 7fdb7e200640 1 --2- 172.21.15.96:0/2794944786 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb7809aa30 0x7fdb7815d830 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7fdb68009870 tx=0x7fdb680046c0 comp rx=0 tx=0).stop 2024-05-28T06:15:47.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:47.770+0000 7fdb7e200640 1 -- 172.21.15.96:0/2794944786 shutdown_connections 2024-05-28T06:15:47.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:47.770+0000 7fdb7e200640 1 --2- 172.21.15.96:0/2794944786 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdb5406d700 0x7fdb5406fbc0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:47.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:47.770+0000 7fdb7e200640 1 --2- 172.21.15.96:0/2794944786 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdb780d1690 0x7fdb781f8470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:47.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:47.770+0000 7fdb7e200640 1 --2- 172.21.15.96:0/2794944786 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb780d1230 0x7fdb7815ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:47.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:47.770+0000 7fdb7e200640 1 --2- 172.21.15.96:0/2794944786 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb7809aa30 0x7fdb7815d830 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:47.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:47.770+0000 7fdb7e200640 1 -- 172.21.15.96:0/2794944786 >> 172.21.15.96:0/2794944786 conn(0x7fdb780b3f20 msgr2=0x7fdb780cea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:15:47.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:47.770+0000 7fdb7e200640 1 -- 172.21.15.96:0/2794944786 shutdown_connections 2024-05-28T06:15:47.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:47.770+0000 7fdb7e200640 1 -- 172.21.15.96:0/2794944786 wait complete. 2024-05-28T06:15:47.774 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:15:47.774+0000 7fea60c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:15:47.774 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:15:47.774+0000 7fea60c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:15:47.775 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:15:47.774+0000 7fea60c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:15:47.775 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:15:47.774+0000 7fea60c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:15:47.775 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:15:47.774+0000 7fea60c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:15:47.793 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone43632 2024-05-28T06:15:47.801 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:15:47.801+0000 7fea5f800640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:15:47.801 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:15:47.801+0000 7fea5f800640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:15:47.801 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:15:47.801+0000 7fea5f800640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:15:47.801 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:15:47.801+0000 7fea5f800640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:15:47.802 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:15:47.801+0000 7fea5f800640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:15:48.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.004+0000 7f6041a00640 1 Processor -- start 2024-05-28T06:15:48.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.004+0000 7f6041a00640 1 -- start start 2024-05-28T06:15:48.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.005+0000 7f6041a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f603c15ca40 0x7f603c06c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:48.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.005+0000 7f6041a00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f603c15b880 con 0x7f603c15e890 2024-05-28T06:15:48.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.005+0000 7f6041a00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f603c15bb40 con 0x7f603c06ce50 2024-05-28T06:15:48.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.005+0000 7f6041a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f603c15ec70 con 0x7f603c15ca40 2024-05-28T06:15:48.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.005+0000 7f603be00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f603c15e890 0x7f603c0aff80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:59130/0 (socket says 172.21.15.96:59130) 2024-05-28T06:15:48.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.005+0000 7f603be00640 1 -- 172.21.15.96:0/881057229 learned_addr learned my addr 172.21.15.96:0/881057229 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:15:48.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.006+0000 7f603b400640 1 --2- 172.21.15.96:0/881057229 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f603c15ca40 0x7f603c06c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:48.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.006+0000 7f603aa00640 1 -- 172.21.15.96:0/881057229 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3249825537 0 0) 0x7f603c15b880 con 0x7f603c15e890 2024-05-28T06:15:48.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.006+0000 7f603aa00640 1 -- 172.21.15.96:0/881057229 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6020003520 con 0x7f603c15e890 2024-05-28T06:15:48.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.006+0000 7f603aa00640 1 -- 172.21.15.96:0/881057229 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3233746254 0 0) 0x7f6020003520 con 0x7f603c15e890 2024-05-28T06:15:48.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.006+0000 7f603aa00640 1 -- 172.21.15.96:0/881057229 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f603c15ca40 msgr2=0x7f603c06c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:15:48.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.006+0000 7f603aa00640 1 --2- 172.21.15.96:0/881057229 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f603c15ca40 0x7f603c06c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:48.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.006+0000 7f603aa00640 1 -- 172.21.15.96:0/881057229 >> v1:172.21.15.123:6789/0 conn(0x7f603c06ce50 legacy=0x7f603c0b5d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:15:48.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.006+0000 7f603aa00640 1 -- 172.21.15.96:0/881057229 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f603c15be20 con 0x7f603c15e890 2024-05-28T06:15:48.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.006+0000 7f603aa00640 1 -- 172.21.15.96:0/881057229 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f602c0019f0 con 0x7f603c15e890 2024-05-28T06:15:48.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.006+0000 7f603aa00640 1 -- 172.21.15.96:0/881057229 >> v1:172.21.15.96:6789/0 conn(0x7f603c15e890 legacy=0x7f603c0aff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:48.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.006+0000 7f603aa00640 1 --2- 172.21.15.96:0/881057229 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f603c15ca40 0x7f60200039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:48.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.006+0000 7f603aa00640 1 --2- 172.21.15.96:0/881057229 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6020003ff0 0x7f6020004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:48.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.006+0000 7f603aa00640 1 --2- 172.21.15.96:0/881057229 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6020004d40 0x7f60200051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:48.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.006+0000 7f603aa00640 1 -- 172.21.15.96:0/881057229 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f602000dd60 con 0x7f6020003ff0 2024-05-28T06:15:48.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.006+0000 7f603aa00640 1 -- 172.21.15.96:0/881057229 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f602000dee0 con 0x7f6020004d40 2024-05-28T06:15:48.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.006+0000 7f603aa00640 1 -- 172.21.15.96:0/881057229 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f602000e0e0 con 0x7f603c15ca40 2024-05-28T06:15:48.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.007+0000 7f603b400640 1 --2- 172.21.15.96:0/881057229 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6020003ff0 0x7f6020004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:48.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.007+0000 7f603be00640 1 --2- 172.21.15.96:0/881057229 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f603c15ca40 0x7f60200039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:48.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.007+0000 7f6041000640 1 --2- 172.21.15.96:0/881057229 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6020004d40 0x7f60200051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:48.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.007+0000 7f603b400640 1 -- 172.21.15.96:0/881057229 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f603c15ca40 msgr2=0x7f60200039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:15:48.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.007+0000 7f603b400640 1 --2- 172.21.15.96:0/881057229 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f603c15ca40 0x7f60200039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:48.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.007+0000 7f603b400640 1 -- 172.21.15.96:0/881057229 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6020004d40 msgr2=0x7f60200051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:15:48.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.007+0000 7f603b400640 1 --2- 172.21.15.96:0/881057229 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6020004d40 0x7f60200051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:48.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.007+0000 7f603b400640 1 -- 172.21.15.96:0/881057229 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6020003520 con 0x7f6020003ff0 2024-05-28T06:15:48.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.007+0000 7f603be00640 1 --2- 172.21.15.96:0/881057229 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f603c15ca40 0x7f60200039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:15:48.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.007+0000 7f6041000640 1 --2- 172.21.15.96:0/881057229 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6020004d40 0x7f60200051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:15:48.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.008+0000 7f603b400640 1 --2- 172.21.15.96:0/881057229 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6020003ff0 0x7f6020004800 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f602400ba00 tx=0x7f6020005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b51053fbb1b501d server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:15:48.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.008+0000 7f603aa00640 1 -- 172.21.15.96:0/881057229 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6024030070 con 0x7f6020003ff0 2024-05-28T06:15:48.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.008+0000 7f603aa00640 1 -- 172.21.15.96:0/881057229 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f602402ada0 con 0x7f6020003ff0 2024-05-28T06:15:48.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.008+0000 7f603aa00640 1 -- 172.21.15.96:0/881057229 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f602400b600 con 0x7f6020003ff0 2024-05-28T06:15:48.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.009+0000 7f6041a00640 1 -- 172.21.15.96:0/881057229 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6020003ff0 msgr2=0x7f6020004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:48.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.009+0000 7f6041a00640 1 --2- 172.21.15.96:0/881057229 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6020003ff0 0x7f6020004800 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f602400ba00 tx=0x7f6020005830 comp rx=0 tx=0).stop 2024-05-28T06:15:48.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.009+0000 7f6041000640 1 -- 172.21.15.96:0/881057229 reap_dead start 2024-05-28T06:15:48.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.009+0000 7f6041a00640 1 -- 172.21.15.96:0/881057229 shutdown_connections 2024-05-28T06:15:48.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.009+0000 7f6041a00640 1 -- 172.21.15.96:0/881057229 >> 172.21.15.96:0/881057229 conn(0x7f603c072260 msgr2=0x7f603c072640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:15:48.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.009+0000 7f6041a00640 1 -- 172.21.15.96:0/881057229 shutdown_connections 2024-05-28T06:15:48.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.009+0000 7f6041a00640 1 -- 172.21.15.96:0/881057229 wait complete. 2024-05-28T06:15:48.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.011+0000 7f6041a00640 1 Processor -- start 2024-05-28T06:15:48.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.011+0000 7f6041a00640 1 -- start start 2024-05-28T06:15:48.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.012+0000 7f6041a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6020003ff0 0x7f603c14f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:48.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.012+0000 7f6041a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f603c151fc0 0x7f603c0aec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:48.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.012+0000 7f6041a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f603c0af140 0x7f603c0cef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:48.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.012+0000 7f6041a00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f603c0d04e0 con 0x7f603c0af140 2024-05-28T06:15:48.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.012+0000 7f6041a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f603c15c3e0 con 0x7f6020003ff0 2024-05-28T06:15:48.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.012+0000 7f6041a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f603c0d0660 con 0x7f603c151fc0 2024-05-28T06:15:48.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.012+0000 7f6041000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f603c0af140 0x7f603c0cef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:48.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.012+0000 7f6041000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f603c0af140 0x7f603c0cef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:59996/0 (socket says 172.21.15.96:59996) 2024-05-28T06:15:48.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.012+0000 7f6041000640 1 -- 172.21.15.96:0/3891956910 learned_addr learned my addr 172.21.15.96:0/3891956910 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:15:48.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.013+0000 7f603be00640 1 --2- 172.21.15.96:0/3891956910 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6020003ff0 0x7f603c14f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:48.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.013+0000 7f603b400640 1 --2- 172.21.15.96:0/3891956910 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f603c151fc0 0x7f603c0aec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:48.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.013+0000 7f6041000640 1 -- 172.21.15.96:0/3891956910 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f603c151fc0 msgr2=0x7f603c0aec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:48.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.013+0000 7f6041000640 1 --2- 172.21.15.96:0/3891956910 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f603c151fc0 0x7f603c0aec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:48.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.013+0000 7f6041000640 1 -- 172.21.15.96:0/3891956910 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6020003ff0 msgr2=0x7f603c14f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:48.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.013+0000 7f6041000640 1 --2- 172.21.15.96:0/3891956910 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6020003ff0 0x7f603c14f130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:48.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.013+0000 7f6041000640 1 -- 172.21.15.96:0/3891956910 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f603c1524a0 con 0x7f603c0af140 2024-05-28T06:15:48.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.013+0000 7f6041000640 1 --2- 172.21.15.96:0/3891956910 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f603c0af140 0x7f603c0cef20 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f603c159470 tx=0x7f603400d890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:15:48.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.013+0000 7f6033e00640 1 -- 172.21.15.96:0/3891956910 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6034005230 con 0x7f603c0af140 2024-05-28T06:15:48.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.013+0000 7f6033e00640 1 -- 172.21.15.96:0/3891956910 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6034005410 con 0x7f603c0af140 2024-05-28T06:15:48.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.013+0000 7f6033e00640 1 -- 172.21.15.96:0/3891956910 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6034007260 con 0x7f603c0af140 2024-05-28T06:15:48.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.013+0000 7f6041a00640 1 -- 172.21.15.96:0/3891956910 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f603c152780 con 0x7f603c0af140 2024-05-28T06:15:48.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.014+0000 7f6041a00640 1 -- 172.21.15.96:0/3891956910 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f603c152d40 con 0x7f603c0af140 2024-05-28T06:15:48.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.015+0000 7f6041a00640 1 -- 172.21.15.96:0/3891956910 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f603c0b4840 con 0x7f603c0af140 2024-05-28T06:15:48.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.016+0000 7f6033e00640 1 -- 172.21.15.96:0/3891956910 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 23) ==== 92315+0+0 (secure 0 0 0) 0x7f6034007400 con 0x7f603c0af140 2024-05-28T06:15:48.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.017+0000 7f6033e00640 1 --2- 172.21.15.96:0/3891956910 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f601806d700 0x7f601806fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:48.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.017+0000 7f6033e00640 1 -- 172.21.15.96:0/3891956910 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 5737+0+0 (secure 0 0 0) 0x7f6034089a90 con 0x7f603c0af140 2024-05-28T06:15:48.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.018+0000 7f603be00640 1 --2- 172.21.15.96:0/3891956910 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f601806d700 0x7f601806fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:48.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.019+0000 7f603be00640 1 --2- 172.21.15.96:0/3891956910 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f601806d700 0x7f601806fbc0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f602c002230 tx=0x7f602c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:15:48.022 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.021+0000 7f6033e00640 1 -- 172.21.15.96:0/3891956910 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f603c0b4840 con 0x7f603c0af140 2024-05-28T06:15:48.260 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.260+0000 7f6041a00640 1 -- 172.21.15.96:0/3891956910 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone43632", "target": ["mon-mgr", ""]}) -- 0x7f603c0690f0 con 0x7f601806d700 2024-05-28T06:15:48.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.547+0000 7f6033e00640 1 -- 172.21.15.96:0/3891956910 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mgrmap(e 24) ==== 92417+0+0 (secure 0 0 0) 0x7f60340555c0 con 0x7f603c0af140 2024-05-28T06:15:48.774 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:15:48.774+0000 7fea21000640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:15:48.774 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:15:48.774+0000 7fea21000640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:15:48.774 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:15:48.774+0000 7fea21000640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:15:48.774 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:15:48.774+0000 7fea21000640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:15:48.774 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:15:48.774+0000 7fea21000640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:15:48.972 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.972+0000 7f6033e00640 1 -- 172.21.15.96:0/3891956910 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f603c0690f0 con 0x7f601806d700 2024-05-28T06:15:48.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.976+0000 7f6041a00640 1 -- 172.21.15.96:0/3891956910 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f601806d700 msgr2=0x7f601806fbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:48.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.976+0000 7f6041a00640 1 --2- 172.21.15.96:0/3891956910 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f601806d700 0x7f601806fbc0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f602c002230 tx=0x7f602c009040 comp rx=0 tx=0).stop 2024-05-28T06:15:48.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.976+0000 7f6041a00640 1 -- 172.21.15.96:0/3891956910 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f603c0af140 msgr2=0x7f603c0cef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:48.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.976+0000 7f6041a00640 1 --2- 172.21.15.96:0/3891956910 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f603c0af140 0x7f603c0cef20 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f603c159470 tx=0x7f603400d890 comp rx=0 tx=0).stop 2024-05-28T06:15:48.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.976+0000 7f6041a00640 1 -- 172.21.15.96:0/3891956910 shutdown_connections 2024-05-28T06:15:48.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.976+0000 7f6041a00640 1 --2- 172.21.15.96:0/3891956910 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f601806d700 0x7f601806fbc0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:48.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.976+0000 7f6041a00640 1 --2- 172.21.15.96:0/3891956910 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f603c0af140 0x7f603c0cef20 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:48.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.976+0000 7f6041a00640 1 --2- 172.21.15.96:0/3891956910 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f603c151fc0 0x7f603c0aec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:48.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.976+0000 7f6041a00640 1 --2- 172.21.15.96:0/3891956910 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6020003ff0 0x7f603c14f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:48.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.976+0000 7f6041a00640 1 -- 172.21.15.96:0/3891956910 >> 172.21.15.96:0/3891956910 conn(0x7f603c072260 msgr2=0x7f603c06c050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:15:48.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.976+0000 7f6041a00640 1 -- 172.21.15.96:0/3891956910 shutdown_connections 2024-05-28T06:15:48.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:48.977+0000 7f6041a00640 1 -- 172.21.15.96:0/3891956910 wait complete. 2024-05-28T06:15:49.006 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:15:49.004+0000 7fea60200640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:15:49.007 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:15:49.004+0000 7fea60200640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:15:49.007 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:15:49.004+0000 7fea60200640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:15:49.007 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:15:49.004+0000 7fea60200640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:15:49.007 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:15:49.004+0000 7fea60200640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:15:49.017 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone43633 2024-05-28T06:15:49.269 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:49.268+0000 7f6340800640 1 Processor -- start 2024-05-28T06:15:49.269 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:49.268+0000 7f6340800640 1 -- start start 2024-05-28T06:15:49.270 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:49.269+0000 7f6340800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f63380b3a50 0x7f6338169980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:49.270 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:49.269+0000 7f6340800640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63380b5310 con 0x7f63380b3670 2024-05-28T06:15:49.270 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:49.269+0000 7f6340800640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63380d2d30 con 0x7f63380b4e80 2024-05-28T06:15:49.270 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:49.269+0000 7f6340800640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f6338162450 con 0x7f63380b3a50 2024-05-28T06:15:49.270 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:49.270+0000 7f633e800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f63380b3a50 0x7f6338169980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:49.270 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:49.270+0000 7f633de00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f63380b3670 0x7f6338162650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:43174/0 (socket says 172.21.15.96:43174) 2024-05-28T06:15:49.270 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:49.270+0000 7f633de00640 1 -- 172.21.15.96:0/801182798 learned_addr learned my addr 172.21.15.96:0/801182798 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:15:49.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:49.270+0000 7f633e800640 1 -- 172.21.15.96:0/801182798 >> v1:172.21.15.123:6789/0 conn(0x7f63380b4e80 legacy=0x7f6338162d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:15:49.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:49.270+0000 7f633e800640 1 -- 172.21.15.96:0/801182798 >> v1:172.21.15.96:6789/0 conn(0x7f63380b3670 legacy=0x7f6338162650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:49.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:49.270+0000 7f633e800640 1 -- 172.21.15.96:0/801182798 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63380d3010 con 0x7f63380b3a50 2024-05-28T06:15:49.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:49.270+0000 7f633ca00640 1 -- 172.21.15.96:0/801182798 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1280016592 0 0) 0x7f63380b5310 con 0x7f63380b3670 2024-05-28T06:15:49.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:49.271+0000 7f633e800640 1 --2- 172.21.15.96:0/801182798 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f63380b3a50 0x7f6338169980 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f6334009d90 tx=0x7f633402a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92d5f966707bed4a server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:15:49.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:49.272+0000 7f633ca00640 1 -- 172.21.15.96:0/801182798 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6334031600 con 0x7f63380b3a50 2024-05-28T06:15:49.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:49.272+0000 7f633ca00640 1 -- 172.21.15.96:0/801182798 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63340317a0 con 0x7f63380b3a50 2024-05-28T06:15:49.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:49.272+0000 7f633ca00640 1 -- 172.21.15.96:0/801182798 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6334031b70 con 0x7f63380b3a50 2024-05-28T06:15:49.273 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:49.272+0000 7f6340800640 1 -- 172.21.15.96:0/801182798 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f63380b3a50 msgr2=0x7f6338169980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:49.273 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:49.272+0000 7f6340800640 1 --2- 172.21.15.96:0/801182798 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f63380b3a50 0x7f6338169980 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f6334009d90 tx=0x7f633402a610 comp rx=0 tx=0).stop 2024-05-28T06:15:49.273 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:49.273+0000 7f6340800640 1 -- 172.21.15.96:0/801182798 shutdown_connections 2024-05-28T06:15:49.273 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:49.273+0000 7f6340800640 1 --2- 172.21.15.96:0/801182798 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f63380b3a50 0x7f6338169980 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:49.273 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:49.273+0000 7f6340800640 1 -- 172.21.15.96:0/801182798 >> 172.21.15.96:0/801182798 conn(0x7f633808bfc0 msgr2=0x7f63380d4640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:15:49.273 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:49.273+0000 7f6340800640 1 -- 172.21.15.96:0/801182798 shutdown_connections 2024-05-28T06:15:49.273 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:49.273+0000 7f6340800640 1 -- 172.21.15.96:0/801182798 wait complete. 2024-05-28T06:15:49.276 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:49.275+0000 7f6340800640 1 Processor -- start 2024-05-28T06:15:49.276 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:49.275+0000 7f6340800640 1 -- start start 2024-05-28T06:15:49.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:49.277+0000 7f6340800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f63380b3670 0x7f6338151bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:49.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:49.277+0000 7f6340800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f63380b3a50 0x7f63380c97f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:49.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:49.277+0000 7f6340800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f63380c9d30 0x7f63380cbae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:49.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:49.277+0000 7f6340800640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f6338163470 con 0x7f63380c9d30 2024-05-28T06:15:49.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:49.277+0000 7f6340800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f63380b56e0 con 0x7f63380b3a50 2024-05-28T06:15:49.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:49.277+0000 7f6340800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f6338161560 con 0x7f63380b3670 2024-05-28T06:15:49.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:49.277+0000 7f633e800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f63380c9d30 0x7f63380cbae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:49.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:49.278+0000 7f633e800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f63380c9d30 0x7f63380cbae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:50336/0 (socket says 172.21.15.96:50336) 2024-05-28T06:15:49.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:49.278+0000 7f633e800640 1 -- 172.21.15.96:0/2345422384 learned_addr learned my addr 172.21.15.96:0/2345422384 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:15:49.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:49.278+0000 7f633de00640 1 --2- 172.21.15.96:0/2345422384 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f63380b3670 0x7f6338151bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:49.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:49.278+0000 7f633d400640 1 --2- 172.21.15.96:0/2345422384 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f63380b3a50 0x7f63380c97f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:49.280 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:49.278+0000 7f633e800640 1 -- 172.21.15.96:0/2345422384 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f63380b3670 msgr2=0x7f6338151bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:49.280 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:49.278+0000 7f633e800640 1 --2- 172.21.15.96:0/2345422384 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f63380b3670 0x7f6338151bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:49.280 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:49.278+0000 7f633e800640 1 -- 172.21.15.96:0/2345422384 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f63380b3a50 msgr2=0x7f63380c97f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:49.280 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:49.278+0000 7f633e800640 1 --2- 172.21.15.96:0/2345422384 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f63380b3a50 0x7f63380c97f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:49.280 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:49.278+0000 7f633e800640 1 -- 172.21.15.96:0/2345422384 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63380ca670 con 0x7f63380c9d30 2024-05-28T06:15:49.280 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:49.278+0000 7f633e800640 1 --2- 172.21.15.96:0/2345422384 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f63380c9d30 0x7f63380cbae0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f6334009d90 tx=0x7f6334004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:15:49.280 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:49.279+0000 7f6332000640 1 -- 172.21.15.96:0/2345422384 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6334040070 con 0x7f63380c9d30 2024-05-28T06:15:49.280 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:49.279+0000 7f6332000640 1 -- 172.21.15.96:0/2345422384 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6334030b60 con 0x7f63380c9d30 2024-05-28T06:15:49.280 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:49.279+0000 7f6332000640 1 -- 172.21.15.96:0/2345422384 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f633403c420 con 0x7f63380c9d30 2024-05-28T06:15:49.280 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:49.279+0000 7f6340800640 1 -- 172.21.15.96:0/2345422384 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63380ca950 con 0x7f63380c9d30 2024-05-28T06:15:49.280 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:49.279+0000 7f6340800640 1 -- 172.21.15.96:0/2345422384 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f633815be90 con 0x7f63380c9d30 2024-05-28T06:15:49.281 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:49.280+0000 7f6340800640 1 -- 172.21.15.96:0/2345422384 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63380689e0 con 0x7f63380c9d30 2024-05-28T06:15:49.281 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:49.281+0000 7f6332000640 1 -- 172.21.15.96:0/2345422384 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 24) ==== 92417+0+0 (secure 0 0 0) 0x7f6334039040 con 0x7f63380c9d30 2024-05-28T06:15:49.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:49.282+0000 7f6332000640 1 --2- 172.21.15.96:0/2345422384 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f631c06d820 0x7f631c06fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:49.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:49.282+0000 7f6332000640 1 -- 172.21.15.96:0/2345422384 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 5737+0+0 (secure 0 0 0) 0x7f63340ab860 con 0x7f63380c9d30 2024-05-28T06:15:49.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:49.283+0000 7f633de00640 1 --2- 172.21.15.96:0/2345422384 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f631c06d820 0x7f631c06fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:49.288 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:49.287+0000 7f6332000640 1 -- 172.21.15.96:0/2345422384 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f633407ae60 con 0x7f63380c9d30 2024-05-28T06:15:49.291 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:49.291+0000 7f633de00640 1 --2- 172.21.15.96:0/2345422384 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f631c06d820 0x7f631c06fce0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f632c0010e0 tx=0x7f632c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:15:49.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:49.652+0000 7f6340800640 1 -- 172.21.15.96:0/2345422384 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone43633", "target": ["mon-mgr", ""]}) -- 0x7f6338152110 con 0x7f631c06d820 2024-05-28T06:15:50.244 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:50.243+0000 7f6332000640 1 -- 172.21.15.96:0/2345422384 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f632c00f7d0 con 0x7f631c06d820 2024-05-28T06:15:50.249 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:50.248+0000 7f6340800640 1 -- 172.21.15.96:0/2345422384 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f631c06d820 msgr2=0x7f631c06fce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:50.249 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:50.248+0000 7f6340800640 1 --2- 172.21.15.96:0/2345422384 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f631c06d820 0x7f631c06fce0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f632c0010e0 tx=0x7f632c009040 comp rx=0 tx=0).stop 2024-05-28T06:15:50.249 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:50.248+0000 7f6340800640 1 -- 172.21.15.96:0/2345422384 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f63380c9d30 msgr2=0x7f63380cbae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:50.249 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:50.248+0000 7f6340800640 1 --2- 172.21.15.96:0/2345422384 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f63380c9d30 0x7f63380cbae0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f6334009d90 tx=0x7f6334004a40 comp rx=0 tx=0).stop 2024-05-28T06:15:50.249 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:50.248+0000 7f6340800640 1 -- 172.21.15.96:0/2345422384 shutdown_connections 2024-05-28T06:15:50.250 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:50.248+0000 7f6340800640 1 --2- 172.21.15.96:0/2345422384 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f631c06d820 0x7f631c06fce0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:50.250 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:50.248+0000 7f6340800640 1 --2- 172.21.15.96:0/2345422384 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f63380c9d30 0x7f63380cbae0 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:50.250 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:50.248+0000 7f6340800640 1 --2- 172.21.15.96:0/2345422384 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f63380b3a50 0x7f63380c97f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:50.250 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:50.248+0000 7f6340800640 1 --2- 172.21.15.96:0/2345422384 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f63380b3670 0x7f6338151bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:50.250 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:50.248+0000 7f6340800640 1 -- 172.21.15.96:0/2345422384 >> 172.21.15.96:0/2345422384 conn(0x7f633808bfc0 msgr2=0x7f633806d750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:15:50.250 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:50.249+0000 7f6340800640 1 -- 172.21.15.96:0/2345422384 shutdown_connections 2024-05-28T06:15:50.250 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:50.250+0000 7f6340800640 1 -- 172.21.15.96:0/2345422384 wait complete. 2024-05-28T06:15:50.279 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone43634 2024-05-28T06:15:50.366 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:15:50.366+0000 7fea60200640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:15:50.366 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:15:50.366+0000 7fea60200640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:15:50.367 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:15:50.366+0000 7fea60200640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:15:50.367 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:15:50.366+0000 7fea60200640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:15:50.367 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:15:50.366+0000 7fea60200640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:15:50.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:50.537+0000 7fa0d8800640 1 Processor -- start 2024-05-28T06:15:50.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:50.537+0000 7fa0d8800640 1 -- start start 2024-05-28T06:15:50.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:50.539+0000 7fa0d8800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa0d015ded0 0x7fa0d00725b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:50.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:50.539+0000 7fa0d8800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fa0d006fcc0 0x7fa0d015b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:50.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:50.539+0000 7fa0d8800640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0d00b3970 con 0x7fa0d015bf30 2024-05-28T06:15:50.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:50.539+0000 7fa0d8800640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fa0d00b3c50 con 0x7fa0d006fcc0 2024-05-28T06:15:50.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:50.540+0000 7fa0d8800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fa0d00b3f30 con 0x7fa0d015ded0 2024-05-28T06:15:50.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:50.540+0000 7fa0d5e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa0d015ded0 0x7fa0d00725b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:50.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:50.540+0000 7fa0d5e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa0d015ded0 0x7fa0d00725b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.96:55100/0 (socket says 172.21.15.96:55100) 2024-05-28T06:15:50.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:50.540+0000 7fa0d5e00640 1 -- 172.21.15.96:0/306346829 learned_addr learned my addr 172.21.15.96:0/306346829 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:15:50.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:50.541+0000 7fa0d5400640 1 --2- 172.21.15.96:0/306346829 >> v2:172.21.15.123:3300/0 conn(0x7fa0d006fcc0 0x7fa0d015b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:50.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:50.541+0000 7fa0d4a00640 1 -- 172.21.15.96:0/306346829 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3106346667 0 0) 0x7fa0d00b3970 con 0x7fa0d015bf30 2024-05-28T06:15:50.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:50.542+0000 7fa0d5e00640 1 -- 172.21.15.96:0/306346829 >> v2:172.21.15.123:3300/0 conn(0x7fa0d006fcc0 msgr2=0x7fa0d015b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:15:50.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:50.542+0000 7fa0d5e00640 1 --2- 172.21.15.96:0/306346829 >> v2:172.21.15.123:3300/0 conn(0x7fa0d006fcc0 0x7fa0d015b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:50.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:50.542+0000 7fa0d5e00640 1 -- 172.21.15.96:0/306346829 >> v1:172.21.15.123:6789/0 conn(0x7fa0d015bf30 legacy=0x7fa0d00d0780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:50.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:50.542+0000 7fa0d5e00640 1 -- 172.21.15.96:0/306346829 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0d00b4210 con 0x7fa0d015ded0 2024-05-28T06:15:50.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:50.546+0000 7fa0d5e00640 1 --2- 172.21.15.96:0/306346829 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa0d015ded0 0x7fa0d00725b0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fa0c4009870 tx=0x7fa0c402a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4fdb5bb657978 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:15:50.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:50.546+0000 7fa0d4a00640 1 -- 172.21.15.96:0/306346829 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0c4037070 con 0x7fa0d015ded0 2024-05-28T06:15:50.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:50.547+0000 7fa0d4a00640 1 -- 172.21.15.96:0/306346829 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0c4032440 con 0x7fa0d015ded0 2024-05-28T06:15:50.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:50.547+0000 7fa0d4a00640 1 -- 172.21.15.96:0/306346829 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0c4032810 con 0x7fa0d015ded0 2024-05-28T06:15:50.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:50.548+0000 7fa0d8800640 1 -- 172.21.15.96:0/306346829 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa0d015ded0 msgr2=0x7fa0d00725b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:50.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:50.548+0000 7fa0d8800640 1 --2- 172.21.15.96:0/306346829 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa0d015ded0 0x7fa0d00725b0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fa0c4009870 tx=0x7fa0c402a270 comp rx=0 tx=0).stop 2024-05-28T06:15:50.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:50.549+0000 7fa0d8800640 1 -- 172.21.15.96:0/306346829 shutdown_connections 2024-05-28T06:15:50.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:50.549+0000 7fa0d8800640 1 --2- 172.21.15.96:0/306346829 >> v2:172.21.15.123:3300/0 conn(0x7fa0d006fcc0 0x7fa0d015b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:50.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:50.549+0000 7fa0d8800640 1 --2- 172.21.15.96:0/306346829 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa0d015ded0 0x7fa0d00725b0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:50.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:50.549+0000 7fa0d8800640 1 -- 172.21.15.96:0/306346829 >> 172.21.15.96:0/306346829 conn(0x7fa0d006bd20 msgr2=0x7fa0d006c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:15:50.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:50.551+0000 7fa0d8800640 1 -- 172.21.15.96:0/306346829 shutdown_connections 2024-05-28T06:15:50.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:50.551+0000 7fa0d8800640 1 -- 172.21.15.96:0/306346829 wait complete. 2024-05-28T06:15:50.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:50.555+0000 7fa0d8800640 1 Processor -- start 2024-05-28T06:15:50.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:50.555+0000 7fa0d8800640 1 -- start start 2024-05-28T06:15:50.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:50.557+0000 7fa0d8800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa0d006fcc0 0x7fa0d01438a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:50.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:50.557+0000 7fa0d8800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa0d015bf30 0x7fa0d0148aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:50.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:50.557+0000 7fa0d8800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa0d015ded0 0x7fa0d0148fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:50.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:50.557+0000 7fa0d8800640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa0d015c520 con 0x7fa0d015bf30 2024-05-28T06:15:50.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:50.557+0000 7fa0d8800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa0d015c3a0 con 0x7fa0d006fcc0 2024-05-28T06:15:50.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:50.557+0000 7fa0d8800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fa0d00d0e90 con 0x7fa0d015ded0 2024-05-28T06:15:50.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:50.557+0000 7fa0d5400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa0d015bf30 0x7fa0d0148aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:50.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:50.557+0000 7fa0d5400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa0d015bf30 0x7fa0d0148aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:50342/0 (socket says 172.21.15.96:50342) 2024-05-28T06:15:50.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:50.557+0000 7fa0d5400640 1 -- 172.21.15.96:0/4024209400 learned_addr learned my addr 172.21.15.96:0/4024209400 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:15:50.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:50.557+0000 7fa0d6800640 1 --2- 172.21.15.96:0/4024209400 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa0d015ded0 0x7fa0d0148fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:50.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:50.558+0000 7fa0d5e00640 1 --2- 172.21.15.96:0/4024209400 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa0d006fcc0 0x7fa0d01438a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:50.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:50.558+0000 7fa0d6800640 1 -- 172.21.15.96:0/4024209400 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa0d006fcc0 msgr2=0x7fa0d01438a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:50.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:50.558+0000 7fa0d6800640 1 --2- 172.21.15.96:0/4024209400 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa0d006fcc0 0x7fa0d01438a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:50.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:50.558+0000 7fa0d6800640 1 -- 172.21.15.96:0/4024209400 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa0d015bf30 msgr2=0x7fa0d0148aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:50.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:50.558+0000 7fa0d6800640 1 --2- 172.21.15.96:0/4024209400 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa0d015bf30 0x7fa0d0148aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:50.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:50.558+0000 7fa0d6800640 1 -- 172.21.15.96:0/4024209400 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0d0098250 con 0x7fa0d015ded0 2024-05-28T06:15:50.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:50.559+0000 7fa0d6800640 1 --2- 172.21.15.96:0/4024209400 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa0d015ded0 0x7fa0d0148fe0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fa0cc00c6a0 tx=0x7fa0cc00cb60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:15:50.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:50.559+0000 7fa0ca000640 1 -- 172.21.15.96:0/4024209400 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0cc003b70 con 0x7fa0d015ded0 2024-05-28T06:15:50.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:50.559+0000 7fa0ca000640 1 -- 172.21.15.96:0/4024209400 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0cc003d30 con 0x7fa0d015ded0 2024-05-28T06:15:50.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:50.559+0000 7fa0ca000640 1 -- 172.21.15.96:0/4024209400 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0cc0052b0 con 0x7fa0d015ded0 2024-05-28T06:15:50.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:50.559+0000 7fa0d8800640 1 -- 172.21.15.96:0/4024209400 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0d0143f30 con 0x7fa0d015ded0 2024-05-28T06:15:50.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:50.560+0000 7fa0d8800640 1 -- 172.21.15.96:0/4024209400 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0d01444f0 con 0x7fa0d015ded0 2024-05-28T06:15:50.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:50.562+0000 7fa0d8800640 1 -- 172.21.15.96:0/4024209400 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0d0098420 con 0x7fa0d015ded0 2024-05-28T06:15:50.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:50.563+0000 7fa0ca000640 1 -- 172.21.15.96:0/4024209400 <== mon.2 v2:172.21.15.123:3301/0 4 ==== mgrmap(e 24) ==== 92417+0+0 (secure 0 0 0) 0x7fa0cc005450 con 0x7fa0d015ded0 2024-05-28T06:15:50.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:50.565+0000 7fa0ca000640 1 --2- 172.21.15.96:0/4024209400 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa0b406d820 0x7fa0b406fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:50.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:50.565+0000 7fa0ca000640 1 -- 172.21.15.96:0/4024209400 <== mon.2 v2:172.21.15.123:3301/0 5 ==== osd_map(74..74 src has 1..74) ==== 5737+0+0 (secure 0 0 0) 0x7fa0cc089fd0 con 0x7fa0d015ded0 2024-05-28T06:15:50.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:50.565+0000 7fa0d5e00640 1 --2- 172.21.15.96:0/4024209400 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa0b406d820 0x7fa0b406fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:50.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:50.568+0000 7fa0d5e00640 1 --2- 172.21.15.96:0/4024209400 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa0b406d820 0x7fa0b406fce0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fa0c4034000 tx=0x7fa0c402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:15:50.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:50.572+0000 7fa0ca000640 1 -- 172.21.15.96:0/4024209400 <== mon.2 v2:172.21.15.123:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa0cc059550 con 0x7fa0d015ded0 2024-05-28T06:15:50.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:50.601+0000 7fa0ca000640 1 -- 172.21.15.96:0/4024209400 <== mon.2 v2:172.21.15.123:3301/0 7 ==== mgrmap(e 25) ==== 92519+0+0 (secure 0 0 0) 0x7fa0cc089b70 con 0x7fa0d015ded0 2024-05-28T06:15:50.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:50.925+0000 7fa0d8800640 1 -- 172.21.15.96:0/4024209400 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone43634", "target": ["mon-mgr", ""]}) -- 0x7fa0d00689e0 con 0x7fa0b406d820 2024-05-28T06:15:51.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:51.753+0000 7fa0ca000640 1 -- 172.21.15.96:0/4024209400 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fa0d00689e0 con 0x7fa0b406d820 2024-05-28T06:15:51.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:51.757+0000 7fa0d8800640 1 -- 172.21.15.96:0/4024209400 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa0b406d820 msgr2=0x7fa0b406fce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:51.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:51.757+0000 7fa0d8800640 1 --2- 172.21.15.96:0/4024209400 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa0b406d820 0x7fa0b406fce0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fa0c4034000 tx=0x7fa0c402e000 comp rx=0 tx=0).stop 2024-05-28T06:15:51.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:51.757+0000 7fa0d8800640 1 -- 172.21.15.96:0/4024209400 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa0d015ded0 msgr2=0x7fa0d0148fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:51.759 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:51.757+0000 7fa0d8800640 1 --2- 172.21.15.96:0/4024209400 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa0d015ded0 0x7fa0d0148fe0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fa0cc00c6a0 tx=0x7fa0cc00cb60 comp rx=0 tx=0).stop 2024-05-28T06:15:51.759 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:51.757+0000 7fa0d8800640 1 -- 172.21.15.96:0/4024209400 shutdown_connections 2024-05-28T06:15:51.759 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:51.757+0000 7fa0d8800640 1 --2- 172.21.15.96:0/4024209400 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa0b406d820 0x7fa0b406fce0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:51.759 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:51.757+0000 7fa0d8800640 1 --2- 172.21.15.96:0/4024209400 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa0d015ded0 0x7fa0d0148fe0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:51.759 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:51.757+0000 7fa0d8800640 1 --2- 172.21.15.96:0/4024209400 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa0d015bf30 0x7fa0d0148aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:51.759 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:51.757+0000 7fa0d8800640 1 --2- 172.21.15.96:0/4024209400 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa0d006fcc0 0x7fa0d01438a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:51.759 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:51.757+0000 7fa0d8800640 1 -- 172.21.15.96:0/4024209400 >> 172.21.15.96:0/4024209400 conn(0x7fa0d006bd20 msgr2=0x7fa0d00cf260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:15:51.759 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:51.758+0000 7fa0d8800640 1 -- 172.21.15.96:0/4024209400 shutdown_connections 2024-05-28T06:15:51.759 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:51.758+0000 7fa0d8800640 1 -- 172.21.15.96:0/4024209400 wait complete. 2024-05-28T06:15:51.788 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone43635 2024-05-28T06:15:52.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:52.046+0000 7f36e5800640 1 Processor -- start 2024-05-28T06:15:52.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:52.046+0000 7f36e5800640 1 -- start start 2024-05-28T06:15:52.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:52.048+0000 7f36e5800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f36e00b3670 0x7f36e0162650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:52.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:52.048+0000 7f36e5800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f36e00b4e80 0x7f36e0160550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:52.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:52.048+0000 7f36e5800640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f36e00b5310 con 0x7f36e00b3a50 2024-05-28T06:15:52.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:52.048+0000 7f36e5800640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f36e0162450 con 0x7f36e00b4e80 2024-05-28T06:15:52.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:52.048+0000 7f36e5800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f36e00d2d30 con 0x7f36e00b3670 2024-05-28T06:15:52.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:52.048+0000 7f36de000640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f36e00b3a50 0x7f36e0162b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:43180/0 (socket says 172.21.15.96:43180) 2024-05-28T06:15:52.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:52.048+0000 7f36de000640 1 -- 172.21.15.96:0/3426661836 learned_addr learned my addr 172.21.15.96:0/3426661836 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:15:52.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:52.048+0000 7f36df400640 1 --2- 172.21.15.96:0/3426661836 >> v2:172.21.15.96:3300/0 conn(0x7f36e00b4e80 0x7f36e0160550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:52.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:52.049+0000 7f36dea00640 1 --2- 172.21.15.96:0/3426661836 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f36e00b3670 0x7f36e0162650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:52.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:52.049+0000 7f36dd600640 1 -- 172.21.15.96:0/3426661836 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1285715894 0 0) 0x7f36e00b5310 con 0x7f36e00b3a50 2024-05-28T06:15:52.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:52.049+0000 7f36dd600640 1 -- 172.21.15.96:0/3426661836 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f36c4003520 con 0x7f36e00b3a50 2024-05-28T06:15:52.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:52.049+0000 7f36df400640 1 -- 172.21.15.96:0/3426661836 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f36e00b3670 msgr2=0x7f36e0162650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:15:52.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:52.049+0000 7f36df400640 1 --2- 172.21.15.96:0/3426661836 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f36e00b3670 0x7f36e0162650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:52.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:52.049+0000 7f36df400640 1 -- 172.21.15.96:0/3426661836 >> v1:172.21.15.96:6789/0 conn(0x7f36e00b3a50 legacy=0x7f36e0162b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:52.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:52.049+0000 7f36df400640 1 -- 172.21.15.96:0/3426661836 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36e00d3010 con 0x7f36e00b4e80 2024-05-28T06:15:52.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:52.051+0000 7f36df400640 1 --2- 172.21.15.96:0/3426661836 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f36e00b4e80 0x7f36e0160550 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f36d8009d90 tx=0x7f36d802a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e72429ce51f51eb server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:15:52.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:52.051+0000 7f36dd600640 1 -- 172.21.15.96:0/3426661836 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36d8031600 con 0x7f36e00b4e80 2024-05-28T06:15:52.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:52.051+0000 7f36dd600640 1 -- 172.21.15.96:0/3426661836 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36d80317a0 con 0x7f36e00b4e80 2024-05-28T06:15:52.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:52.051+0000 7f36dd600640 1 -- 172.21.15.96:0/3426661836 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36d8031b70 con 0x7f36e00b4e80 2024-05-28T06:15:52.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:52.052+0000 7f36e5800640 1 -- 172.21.15.96:0/3426661836 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f36e00b4e80 msgr2=0x7f36e0160550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:52.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:52.052+0000 7f36e5800640 1 --2- 172.21.15.96:0/3426661836 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f36e00b4e80 0x7f36e0160550 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f36d8009d90 tx=0x7f36d802a5d0 comp rx=0 tx=0).stop 2024-05-28T06:15:52.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:52.052+0000 7f36e5800640 1 -- 172.21.15.96:0/3426661836 shutdown_connections 2024-05-28T06:15:52.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:52.052+0000 7f36e5800640 1 --2- 172.21.15.96:0/3426661836 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f36e00b4e80 0x7f36e0160550 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:52.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:52.052+0000 7f36e5800640 1 --2- 172.21.15.96:0/3426661836 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f36e00b3670 0x7f36e0162650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:52.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:52.052+0000 7f36e5800640 1 -- 172.21.15.96:0/3426661836 >> 172.21.15.96:0/3426661836 conn(0x7f36e008bfc0 msgr2=0x7f36e00d4640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:15:52.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:52.053+0000 7f36e5800640 1 -- 172.21.15.96:0/3426661836 shutdown_connections 2024-05-28T06:15:52.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:52.053+0000 7f36e5800640 1 -- 172.21.15.96:0/3426661836 wait complete. 2024-05-28T06:15:52.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:52.056+0000 7f36e5800640 1 Processor -- start 2024-05-28T06:15:52.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:52.056+0000 7f36e5800640 1 -- start start 2024-05-28T06:15:52.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:52.057+0000 7f36e5800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f36e00b3670 0x7f36e0156630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:52.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:52.057+0000 7f36e5800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f36e00b3a50 0x7f36e00b1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:52.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:52.057+0000 7f36e5800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f36e015b0f0 0x7f36e015b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:52.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:52.057+0000 7f36e5800640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f36e01632a0 con 0x7f36e015b0f0 2024-05-28T06:15:52.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:52.057+0000 7f36e5800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f36e00b56e0 con 0x7f36e00b3a50 2024-05-28T06:15:52.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:52.057+0000 7f36e5800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f36e0163420 con 0x7f36e00b3670 2024-05-28T06:15:52.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:52.058+0000 7f36df400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f36e015b0f0 0x7f36e015b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:52.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:52.058+0000 7f36df400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f36e015b0f0 0x7f36e015b5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:50362/0 (socket says 172.21.15.96:50362) 2024-05-28T06:15:52.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:52.058+0000 7f36df400640 1 -- 172.21.15.96:0/2813470463 learned_addr learned my addr 172.21.15.96:0/2813470463 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:15:52.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:52.058+0000 7f36dea00640 1 --2- 172.21.15.96:0/2813470463 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f36e00b3670 0x7f36e0156630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:52.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:52.058+0000 7f36df400640 1 -- 172.21.15.96:0/2813470463 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f36e00b3670 msgr2=0x7f36e0156630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:52.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:52.058+0000 7f36df400640 1 --2- 172.21.15.96:0/2813470463 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f36e00b3670 0x7f36e0156630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:52.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:52.058+0000 7f36df400640 1 -- 172.21.15.96:0/2813470463 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f36e00b3a50 msgr2=0x7f36e00b1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:52.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:52.058+0000 7f36df400640 1 --2- 172.21.15.96:0/2813470463 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f36e00b3a50 0x7f36e00b1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:52.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:52.058+0000 7f36df400640 1 -- 172.21.15.96:0/2813470463 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36e00cb7b0 con 0x7f36e015b0f0 2024-05-28T06:15:52.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:52.059+0000 7f36df400640 1 --2- 172.21.15.96:0/2813470463 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f36e015b0f0 0x7f36e015b5a0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f36d802abc0 tx=0x7f36d8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:15:52.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:52.059+0000 7f36d6a00640 1 -- 172.21.15.96:0/2813470463 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36d8040070 con 0x7f36e015b0f0 2024-05-28T06:15:52.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:52.059+0000 7f36d6a00640 1 -- 172.21.15.96:0/2813470463 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36d803c830 con 0x7f36e015b0f0 2024-05-28T06:15:52.061 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:52.059+0000 7f36d6a00640 1 -- 172.21.15.96:0/2813470463 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36d803cbe0 con 0x7f36e015b0f0 2024-05-28T06:15:52.061 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:52.059+0000 7f36e5800640 1 -- 172.21.15.96:0/2813470463 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36e00cba90 con 0x7f36e015b0f0 2024-05-28T06:15:52.061 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:52.060+0000 7f36e5800640 1 -- 172.21.15.96:0/2813470463 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36e00cc050 con 0x7f36e015b0f0 2024-05-28T06:15:52.062 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:52.062+0000 7f36e5800640 1 -- 172.21.15.96:0/2813470463 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36e00689e0 con 0x7f36e015b0f0 2024-05-28T06:15:52.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:52.063+0000 7f36d6a00640 1 -- 172.21.15.96:0/2813470463 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 25) ==== 92519+0+0 (secure 0 0 0) 0x7f36d8039040 con 0x7f36e015b0f0 2024-05-28T06:15:52.065 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:52.065+0000 7f36d6a00640 1 --2- 172.21.15.96:0/2813470463 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f36c006d990 0x7f36c006fe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:52.065 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:52.065+0000 7f36d6a00640 1 -- 172.21.15.96:0/2813470463 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 5737+0+0 (secure 0 0 0) 0x7f36d80abc20 con 0x7f36e015b0f0 2024-05-28T06:15:52.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:52.074+0000 7f36d6a00640 1 -- 172.21.15.96:0/2813470463 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f36d8039330 con 0x7f36e015b0f0 2024-05-28T06:15:52.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:52.094+0000 7f36dea00640 1 --2- 172.21.15.96:0/2813470463 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f36c006d990 0x7f36c006fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:52.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:52.099+0000 7f36dea00640 1 --2- 172.21.15.96:0/2813470463 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f36c006d990 0x7f36c006fe50 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f36d0009a20 tx=0x7f36d0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:15:52.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:52.467+0000 7f36e5800640 1 -- 172.21.15.96:0/2813470463 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone43635", "target": ["mon-mgr", ""]}) -- 0x7f36e00ba5c0 con 0x7f36c006d990 2024-05-28T06:15:52.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:52.779+0000 7f36d6a00640 1 -- 172.21.15.96:0/2813470463 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mgrmap(e 26) ==== 92570+0+0 (secure 0 0 0) 0x7f36d8077b70 con 0x7f36e015b0f0 2024-05-28T06:15:53.269 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:53.268+0000 7f36d6a00640 1 -- 172.21.15.96:0/2813470463 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f36e00ba5c0 con 0x7f36c006d990 2024-05-28T06:15:53.274 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:53.274+0000 7f36e5800640 1 -- 172.21.15.96:0/2813470463 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f36c006d990 msgr2=0x7f36c006fe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:53.274 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:53.274+0000 7f36e5800640 1 --2- 172.21.15.96:0/2813470463 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f36c006d990 0x7f36c006fe50 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f36d0009a20 tx=0x7f36d0008040 comp rx=0 tx=0).stop 2024-05-28T06:15:53.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:53.274+0000 7f36e5800640 1 -- 172.21.15.96:0/2813470463 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f36e015b0f0 msgr2=0x7f36e015b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:53.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:53.274+0000 7f36e5800640 1 --2- 172.21.15.96:0/2813470463 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f36e015b0f0 0x7f36e015b5a0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f36d802abc0 tx=0x7f36d8004a40 comp rx=0 tx=0).stop 2024-05-28T06:15:53.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:53.274+0000 7f36e5800640 1 -- 172.21.15.96:0/2813470463 shutdown_connections 2024-05-28T06:15:53.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:53.274+0000 7f36e5800640 1 --2- 172.21.15.96:0/2813470463 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f36c006d990 0x7f36c006fe50 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:53.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:53.274+0000 7f36e5800640 1 --2- 172.21.15.96:0/2813470463 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f36e015b0f0 0x7f36e015b5a0 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:53.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:53.274+0000 7f36e5800640 1 --2- 172.21.15.96:0/2813470463 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f36e00b3a50 0x7f36e00b1fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:53.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:53.274+0000 7f36e5800640 1 --2- 172.21.15.96:0/2813470463 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f36e00b3670 0x7f36e0156630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:53.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:53.274+0000 7f36e5800640 1 -- 172.21.15.96:0/2813470463 >> 172.21.15.96:0/2813470463 conn(0x7f36e008bfc0 msgr2=0x7f36e006daa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:15:53.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:53.275+0000 7f36e5800640 1 -- 172.21.15.96:0/2813470463 shutdown_connections 2024-05-28T06:15:53.276 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:53.275+0000 7f36e5800640 1 -- 172.21.15.96:0/2813470463 wait complete. 2024-05-28T06:15:53.300 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-05-28T06:15:53.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:53.544+0000 7fef32400640 1 Processor -- start 2024-05-28T06:15:53.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:53.544+0000 7fef32400640 1 -- start start 2024-05-28T06:15:53.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:53.546+0000 7fef32400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fef2c0b3b20 0x7fef2c0cfe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:53.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:53.546+0000 7fef32400640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef2c15fed0 con 0x7fef2c15f3e0 2024-05-28T06:15:53.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:53.546+0000 7fef32400640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef2c070ce0 con 0x7fef2c0705f0 2024-05-28T06:15:53.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:53.546+0000 7fef32400640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fef2c15f1f0 con 0x7fef2c0b3b20 2024-05-28T06:15:53.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:53.547+0000 7fef2b400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fef2c15f3e0 0x7fef2c15f7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:43190/0 (socket says 172.21.15.96:43190) 2024-05-28T06:15:53.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:53.547+0000 7fef2b400640 1 -- 172.21.15.96:0/1608983575 learned_addr learned my addr 172.21.15.96:0/1608983575 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:15:53.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:53.547+0000 7fef2be00640 1 --2- 172.21.15.96:0/1608983575 >> v2:172.21.15.123:3300/0 conn(0x7fef2c0b3b20 0x7fef2c0cfe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:53.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:53.547+0000 7fef2aa00640 1 -- 172.21.15.96:0/1608983575 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2443530799 0 0) 0x7fef2c15fed0 con 0x7fef2c15f3e0 2024-05-28T06:15:53.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:53.548+0000 7fef2aa00640 1 -- 172.21.15.96:0/1608983575 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fef10003520 con 0x7fef2c15f3e0 2024-05-28T06:15:53.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:53.548+0000 7fef2be00640 1 -- 172.21.15.96:0/1608983575 >> v1:172.21.15.123:6789/0 conn(0x7fef2c0705f0 legacy=0x7fef2c0aec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:53.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:53.548+0000 7fef2be00640 1 -- 172.21.15.96:0/1608983575 >> v1:172.21.15.96:6789/0 conn(0x7fef2c15f3e0 legacy=0x7fef2c15f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:53.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:53.548+0000 7fef2aa00640 1 -- 172.21.15.96:0/1608983575 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2291350612 0 0) 0x7fef10003520 con 0x7fef2c15f3e0 2024-05-28T06:15:53.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:53.548+0000 7fef2be00640 1 -- 172.21.15.96:0/1608983575 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef2c160160 con 0x7fef2c0b3b20 2024-05-28T06:15:53.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:53.551+0000 7fef2be00640 1 --2- 172.21.15.96:0/1608983575 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef2c0b3b20 0x7fef2c0cfe40 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fef24009d90 tx=0x7fef2402a480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc80a96593e62cb server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:15:53.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:53.553+0000 7fef2aa00640 1 -- 172.21.15.96:0/1608983575 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef1c001670 con 0x7fef2c0b3b20 2024-05-28T06:15:53.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:53.553+0000 7fef2aa00640 1 -- 172.21.15.96:0/1608983575 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef24032440 con 0x7fef2c0b3b20 2024-05-28T06:15:53.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:53.553+0000 7fef2aa00640 1 -- 172.21.15.96:0/1608983575 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef24032810 con 0x7fef2c0b3b20 2024-05-28T06:15:53.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:53.554+0000 7fef32400640 1 -- 172.21.15.96:0/1608983575 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef2c0b3b20 msgr2=0x7fef2c0cfe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:53.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:53.554+0000 7fef32400640 1 --2- 172.21.15.96:0/1608983575 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef2c0b3b20 0x7fef2c0cfe40 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fef24009d90 tx=0x7fef2402a480 comp rx=0 tx=0).stop 2024-05-28T06:15:53.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:53.554+0000 7fef32400640 1 -- 172.21.15.96:0/1608983575 shutdown_connections 2024-05-28T06:15:53.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:53.554+0000 7fef32400640 1 --2- 172.21.15.96:0/1608983575 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef2c0b3b20 0x7fef2c0cfe40 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:53.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:53.554+0000 7fef32400640 1 -- 172.21.15.96:0/1608983575 >> 172.21.15.96:0/1608983575 conn(0x7fef2c09aa30 msgr2=0x7fef2c0b2f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:15:53.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:53.554+0000 7fef32400640 1 -- 172.21.15.96:0/1608983575 shutdown_connections 2024-05-28T06:15:53.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:53.554+0000 7fef32400640 1 -- 172.21.15.96:0/1608983575 wait complete. 2024-05-28T06:15:53.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:53.557+0000 7fef32400640 1 Processor -- start 2024-05-28T06:15:53.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:53.557+0000 7fef32400640 1 -- start start 2024-05-28T06:15:53.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:53.558+0000 7fef32400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef2c0705f0 0x7fef2c14f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:53.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:53.558+0000 7fef32400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fef2c0b3b20 0x7fef2c0cb0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:53.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:53.558+0000 7fef32400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef2c15f3e0 0x7fef2c1f81d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:53.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:53.558+0000 7fef32400640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fef2c0cf9b0 con 0x7fef2c15f3e0 2024-05-28T06:15:53.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:53.558+0000 7fef32400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fef2c0b08e0 con 0x7fef2c0705f0 2024-05-28T06:15:53.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:53.558+0000 7fef32400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fef2c0af340 con 0x7fef2c0b3b20 2024-05-28T06:15:53.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:53.558+0000 7fef2be00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef2c15f3e0 0x7fef2c1f81d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:53.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:53.558+0000 7fef2be00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef2c15f3e0 0x7fef2c1f81d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:50368/0 (socket says 172.21.15.96:50368) 2024-05-28T06:15:53.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:53.558+0000 7fef2be00640 1 -- 172.21.15.96:0/559014858 learned_addr learned my addr 172.21.15.96:0/559014858 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:15:53.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:53.559+0000 7fef22a00640 1 --2- 172.21.15.96:0/559014858 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fef2c0b3b20 0x7fef2c0cb0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:53.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:53.559+0000 7fef2be00640 1 -- 172.21.15.96:0/559014858 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fef2c0b3b20 msgr2=0x7fef2c0cb0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:53.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:53.559+0000 7fef2be00640 1 --2- 172.21.15.96:0/559014858 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fef2c0b3b20 0x7fef2c0cb0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:53.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:53.559+0000 7fef2be00640 1 -- 172.21.15.96:0/559014858 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef2c0705f0 msgr2=0x7fef2c14f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:53.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:53.559+0000 7fef2be00640 1 --2- 172.21.15.96:0/559014858 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef2c0705f0 0x7fef2c14f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:53.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:53.559+0000 7fef2be00640 1 -- 172.21.15.96:0/559014858 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef2c0ccc60 con 0x7fef2c15f3e0 2024-05-28T06:15:53.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:53.560+0000 7fef2be00640 1 --2- 172.21.15.96:0/559014858 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef2c15f3e0 0x7fef2c1f81d0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7fef24034040 tx=0x7fef24004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:15:53.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:53.560+0000 7fef23e00640 1 -- 172.21.15.96:0/559014858 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef24003b90 con 0x7fef2c15f3e0 2024-05-28T06:15:53.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:53.560+0000 7fef23e00640 1 -- 172.21.15.96:0/559014858 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef24003d70 con 0x7fef2c15f3e0 2024-05-28T06:15:53.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:53.560+0000 7fef32400640 1 -- 172.21.15.96:0/559014858 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef2c0ccf40 con 0x7fef2c15f3e0 2024-05-28T06:15:53.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:53.560+0000 7fef23e00640 1 -- 172.21.15.96:0/559014858 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef24030b40 con 0x7fef2c15f3e0 2024-05-28T06:15:53.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:53.560+0000 7fef32400640 1 -- 172.21.15.96:0/559014858 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef2c0cd7e0 con 0x7fef2c15f3e0 2024-05-28T06:15:53.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:53.561+0000 7fef32400640 1 -- 172.21.15.96:0/559014858 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef2c0cd330 con 0x7fef2c15f3e0 2024-05-28T06:15:53.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:53.563+0000 7fef23e00640 1 -- 172.21.15.96:0/559014858 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 26) ==== 92570+0+0 (secure 0 0 0) 0x7fef24030ce0 con 0x7fef2c15f3e0 2024-05-28T06:15:53.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:53.564+0000 7fef23e00640 1 --2- 172.21.15.96:0/559014858 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fef0806d9d0 0x7fef0806fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:53.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:53.565+0000 7fef23e00640 1 -- 172.21.15.96:0/559014858 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 5795+0+0 (secure 0 0 0) 0x7fef240abea0 con 0x7fef2c15f3e0 2024-05-28T06:15:53.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:53.572+0000 7fef23e00640 1 -- 172.21.15.96:0/559014858 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fef24042020 con 0x7fef2c15f3e0 2024-05-28T06:15:53.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:53.576+0000 7fef2b400640 1 --2- 172.21.15.96:0/559014858 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fef0806d9d0 0x7fef0806fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:53.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:53.578+0000 7fef2b400640 1 --2- 172.21.15.96:0/559014858 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fef0806d9d0 0x7fef0806fe90 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fef1c001eb0 tx=0x7fef1c0019a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:15:53.973 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:53.972+0000 7fef32400640 1 -- 172.21.15.96:0/559014858 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fef2c0cd110 con 0x7fef2c15f3e0 2024-05-28T06:15:53.976 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:53.975+0000 7fef23e00640 1 -- 172.21.15.96:0/559014858 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+2589 (secure 0 0 0) 0x7fef2407ba20 con 0x7fef2c15f3e0 2024-05-28T06:15:53.976 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:15:53.976 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:15:53.976 INFO:teuthology.orchestra.run.smithi096.stdout: "fsid": "e4a831b5-e6d1-47a7-96cb-bf85154ee0d2", 2024-05-28T06:15:53.976 INFO:teuthology.orchestra.run.smithi096.stdout: "health": { 2024-05-28T06:15:53.976 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "HEALTH_OK", 2024-05-28T06:15:53.977 INFO:teuthology.orchestra.run.smithi096.stdout: "checks": {}, 2024-05-28T06:15:53.977 INFO:teuthology.orchestra.run.smithi096.stdout: "mutes": [] 2024-05-28T06:15:53.977 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:15:53.977 INFO:teuthology.orchestra.run.smithi096.stdout: "election_epoch": 8, 2024-05-28T06:15:53.977 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum": [ 2024-05-28T06:15:53.978 INFO:teuthology.orchestra.run.smithi096.stdout: 0, 2024-05-28T06:15:53.978 INFO:teuthology.orchestra.run.smithi096.stdout: 1, 2024-05-28T06:15:53.978 INFO:teuthology.orchestra.run.smithi096.stdout: 2 2024-05-28T06:15:53.978 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:15:53.978 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_names": [ 2024-05-28T06:15:53.979 INFO:teuthology.orchestra.run.smithi096.stdout: "a", 2024-05-28T06:15:53.979 INFO:teuthology.orchestra.run.smithi096.stdout: "b", 2024-05-28T06:15:53.979 INFO:teuthology.orchestra.run.smithi096.stdout: "c" 2024-05-28T06:15:53.979 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:15:53.979 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_age": 727, 2024-05-28T06:15:53.979 INFO:teuthology.orchestra.run.smithi096.stdout: "monmap": { 2024-05-28T06:15:53.979 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 1, 2024-05-28T06:15:53.979 INFO:teuthology.orchestra.run.smithi096.stdout: "min_mon_release_name": "squid", 2024-05-28T06:15:53.979 INFO:teuthology.orchestra.run.smithi096.stdout: "num_mons": 3 2024-05-28T06:15:53.979 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:15:53.979 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap": { 2024-05-28T06:15:53.980 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 76, 2024-05-28T06:15:53.980 INFO:teuthology.orchestra.run.smithi096.stdout: "num_osds": 8, 2024-05-28T06:15:53.980 INFO:teuthology.orchestra.run.smithi096.stdout: "num_up_osds": 8, 2024-05-28T06:15:53.980 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_up_since": 1716876238, 2024-05-28T06:15:53.980 INFO:teuthology.orchestra.run.smithi096.stdout: "num_in_osds": 8, 2024-05-28T06:15:53.980 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_in_since": 1716876229, 2024-05-28T06:15:53.980 INFO:teuthology.orchestra.run.smithi096.stdout: "num_remapped_pgs": 0 2024-05-28T06:15:53.980 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:15:53.980 INFO:teuthology.orchestra.run.smithi096.stdout: "pgmap": { 2024-05-28T06:15:53.980 INFO:teuthology.orchestra.run.smithi096.stdout: "pgs_by_state": [ 2024-05-28T06:15:53.980 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:15:53.980 INFO:teuthology.orchestra.run.smithi096.stdout: "state_name": "active+clean", 2024-05-28T06:15:53.980 INFO:teuthology.orchestra.run.smithi096.stdout: "count": 97 2024-05-28T06:15:53.981 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:15:53.981 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:15:53.981 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pgs": 97, 2024-05-28T06:15:53.981 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pools": 3, 2024-05-28T06:15:53.981 INFO:teuthology.orchestra.run.smithi096.stdout: "num_objects": 914, 2024-05-28T06:15:53.982 INFO:teuthology.orchestra.run.smithi096.stdout: "data_bytes": 3696149556, 2024-05-28T06:15:53.982 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_used": 7153541120, 2024-05-28T06:15:53.982 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_avail": 765940572160, 2024-05-28T06:15:53.982 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_total": 773094113280, 2024-05-28T06:15:53.982 INFO:teuthology.orchestra.run.smithi096.stdout: "read_bytes_sec": 48211039, 2024-05-28T06:15:53.983 INFO:teuthology.orchestra.run.smithi096.stdout: "write_bytes_sec": 80367795, 2024-05-28T06:15:53.983 INFO:teuthology.orchestra.run.smithi096.stdout: "read_op_per_sec": 12, 2024-05-28T06:15:53.983 INFO:teuthology.orchestra.run.smithi096.stdout: "write_op_per_sec": 23 2024-05-28T06:15:53.983 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:15:53.983 INFO:teuthology.orchestra.run.smithi096.stdout: "fsmap": { 2024-05-28T06:15:53.983 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 44, 2024-05-28T06:15:53.983 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 5, 2024-05-28T06:15:53.983 INFO:teuthology.orchestra.run.smithi096.stdout: "up": 1, 2024-05-28T06:15:53.983 INFO:teuthology.orchestra.run.smithi096.stdout: "in": 1, 2024-05-28T06:15:53.983 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 1, 2024-05-28T06:15:53.983 INFO:teuthology.orchestra.run.smithi096.stdout: "by_rank": [ 2024-05-28T06:15:53.984 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:15:53.984 INFO:teuthology.orchestra.run.smithi096.stdout: "filesystem_id": 5, 2024-05-28T06:15:53.984 INFO:teuthology.orchestra.run.smithi096.stdout: "rank": 0, 2024-05-28T06:15:53.984 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "d", 2024-05-28T06:15:53.984 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "up:active", 2024-05-28T06:15:53.984 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 7146 2024-05-28T06:15:53.984 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:15:53.984 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:15:53.984 INFO:teuthology.orchestra.run.smithi096.stdout: "up:standby": 3 2024-05-28T06:15:53.984 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:15:53.984 INFO:teuthology.orchestra.run.smithi096.stdout: "mgrmap": { 2024-05-28T06:15:53.984 INFO:teuthology.orchestra.run.smithi096.stdout: "available": true, 2024-05-28T06:15:53.984 INFO:teuthology.orchestra.run.smithi096.stdout: "num_standbys": 1, 2024-05-28T06:15:53.984 INFO:teuthology.orchestra.run.smithi096.stdout: "modules": [ 2024-05-28T06:15:53.985 INFO:teuthology.orchestra.run.smithi096.stdout: "iostat", 2024-05-28T06:15:53.985 INFO:teuthology.orchestra.run.smithi096.stdout: "nfs", 2024-05-28T06:15:53.985 INFO:teuthology.orchestra.run.smithi096.stdout: "restful" 2024-05-28T06:15:53.985 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:15:53.985 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:15:53.986 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:15:53.986 INFO:teuthology.orchestra.run.smithi096.stdout: "servicemap": { 2024-05-28T06:15:53.986 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 12, 2024-05-28T06:15:53.986 INFO:teuthology.orchestra.run.smithi096.stdout: "modified": "2024-05-28T06:15:47.772170+0000", 2024-05-28T06:15:53.986 INFO:teuthology.orchestra.run.smithi096.stdout: "services": { 2024-05-28T06:15:53.987 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-05-28T06:15:53.987 INFO:teuthology.orchestra.run.smithi096.stdout: "daemons": { 2024-05-28T06:15:53.987 INFO:teuthology.orchestra.run.smithi096.stdout: "summary": "", 2024-05-28T06:15:53.987 INFO:teuthology.orchestra.run.smithi096.stdout: "d": { 2024-05-28T06:15:53.987 INFO:teuthology.orchestra.run.smithi096.stdout: "start_epoch": 0, 2024-05-28T06:15:53.987 INFO:teuthology.orchestra.run.smithi096.stdout: "start_stamp": "0.000000", 2024-05-28T06:15:53.987 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 0, 2024-05-28T06:15:53.987 INFO:teuthology.orchestra.run.smithi096.stdout: "addr": "(unrecognized address family 0)/0", 2024-05-28T06:15:53.988 INFO:teuthology.orchestra.run.smithi096.stdout: "metadata": {}, 2024-05-28T06:15:53.988 INFO:teuthology.orchestra.run.smithi096.stdout: "task_status": {} 2024-05-28T06:15:53.988 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:15:53.988 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:15:53.988 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:15:53.988 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:15:53.988 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:15:53.988 INFO:teuthology.orchestra.run.smithi096.stdout: "progress_events": { 2024-05-28T06:15:53.988 INFO:teuthology.orchestra.run.smithi096.stdout: "1f539510-8a8a-46db-8f0c-e233f5cc901d": { 2024-05-28T06:15:53.988 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "2 ongoing clones; Avg progress is 15.885% (2s)\n [====........................] (remaining: 10s)", 2024-05-28T06:15:53.988 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": 0.15884999930858612, 2024-05-28T06:15:53.988 INFO:teuthology.orchestra.run.smithi096.stdout: "add_to_ceph_s": true 2024-05-28T06:15:53.989 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:15:53.989 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:15:53.989 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:15:53.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:53.979+0000 7fef32400640 1 -- 172.21.15.96:0/559014858 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fef0806d9d0 msgr2=0x7fef0806fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:53.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:53.979+0000 7fef32400640 1 --2- 172.21.15.96:0/559014858 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fef0806d9d0 0x7fef0806fe90 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fef1c001eb0 tx=0x7fef1c0019a0 comp rx=0 tx=0).stop 2024-05-28T06:15:53.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:53.979+0000 7fef32400640 1 -- 172.21.15.96:0/559014858 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef2c15f3e0 msgr2=0x7fef2c1f81d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:53.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:53.979+0000 7fef32400640 1 --2- 172.21.15.96:0/559014858 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef2c15f3e0 0x7fef2c1f81d0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7fef24034040 tx=0x7fef24004a40 comp rx=0 tx=0).stop 2024-05-28T06:15:53.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:53.979+0000 7fef32400640 1 -- 172.21.15.96:0/559014858 shutdown_connections 2024-05-28T06:15:53.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:53.979+0000 7fef32400640 1 --2- 172.21.15.96:0/559014858 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fef0806d9d0 0x7fef0806fe90 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:53.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:53.979+0000 7fef32400640 1 --2- 172.21.15.96:0/559014858 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef2c15f3e0 0x7fef2c1f81d0 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:53.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:53.979+0000 7fef32400640 1 --2- 172.21.15.96:0/559014858 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fef2c0b3b20 0x7fef2c0cb0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:53.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:53.979+0000 7fef32400640 1 --2- 172.21.15.96:0/559014858 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef2c0705f0 0x7fef2c14f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:53.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:53.979+0000 7fef32400640 1 -- 172.21.15.96:0/559014858 >> 172.21.15.96:0/559014858 conn(0x7fef2c09aa30 msgr2=0x7fef2c0b2f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:15:53.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:53.980+0000 7fef32400640 1 -- 172.21.15.96:0/559014858 shutdown_connections 2024-05-28T06:15:53.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:53.980+0000 7fef32400640 1 -- 172.21.15.96:0/559014858 wait complete. 2024-05-28T06:15:55.005 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-05-28T06:15:55.267 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.266+0000 7fda99600640 1 Processor -- start 2024-05-28T06:15:55.267 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.266+0000 7fda99600640 1 -- start start 2024-05-28T06:15:55.268 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.267+0000 7fda99600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fda940d4c40 0x7fda940d2330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:55.268 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.267+0000 7fda99600640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda940d3060 con 0x7fda940d1ca0 2024-05-28T06:15:55.268 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.267+0000 7fda99600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda940d3340 con 0x7fda9415e690 2024-05-28T06:15:55.268 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.267+0000 7fda99600640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fda940d3620 con 0x7fda940d4c40 2024-05-28T06:15:55.268 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.268+0000 7fda98c00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fda940d1ca0 0x7fda940d2870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:43194/0 (socket says 172.21.15.96:43194) 2024-05-28T06:15:55.268 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.268+0000 7fda98c00640 1 -- 172.21.15.96:0/3965035821 learned_addr learned my addr 172.21.15.96:0/3965035821 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:15:55.270 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.268+0000 7fda93400640 1 --2- 172.21.15.96:0/3965035821 >> v2:172.21.15.123:3300/0 conn(0x7fda940d4c40 0x7fda940d2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:55.270 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.268+0000 7fda92a00640 1 -- 172.21.15.96:0/3965035821 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2708134716 0 0) 0x7fda940d3060 con 0x7fda940d1ca0 2024-05-28T06:15:55.270 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.268+0000 7fda92a00640 1 -- 172.21.15.96:0/3965035821 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fda78003520 con 0x7fda940d1ca0 2024-05-28T06:15:55.270 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.269+0000 7fda92a00640 1 -- 172.21.15.96:0/3965035821 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2694002138 0 0) 0x7fda78003520 con 0x7fda940d1ca0 2024-05-28T06:15:55.270 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.269+0000 7fda92a00640 1 -- 172.21.15.96:0/3965035821 >> v2:172.21.15.123:3300/0 conn(0x7fda940d4c40 msgr2=0x7fda940d2330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:15:55.270 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.269+0000 7fda92a00640 1 --2- 172.21.15.96:0/3965035821 >> v2:172.21.15.123:3300/0 conn(0x7fda940d4c40 0x7fda940d2330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:55.270 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.269+0000 7fda92a00640 1 -- 172.21.15.96:0/3965035821 >> v1:172.21.15.123:6789/0 conn(0x7fda9415e690 legacy=0x7fda940d4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:15:55.270 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.269+0000 7fda92a00640 1 -- 172.21.15.96:0/3965035821 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda940cf770 con 0x7fda940d1ca0 2024-05-28T06:15:55.270 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.269+0000 7fda92a00640 1 -- 172.21.15.96:0/3965035821 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7fda8c0019f0 con 0x7fda940d1ca0 2024-05-28T06:15:55.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.269+0000 7fda92a00640 1 -- 172.21.15.96:0/3965035821 >> v1:172.21.15.96:6789/0 conn(0x7fda940d1ca0 legacy=0x7fda940d2870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:55.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.269+0000 7fda92a00640 1 --2- 172.21.15.96:0/3965035821 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fda780039b0 0x7fda78003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:55.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.269+0000 7fda92a00640 1 --2- 172.21.15.96:0/3965035821 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fda780043d0 0x7fda780048a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:55.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.269+0000 7fda92a00640 1 --2- 172.21.15.96:0/3965035821 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fda780050a0 0x7fda7800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:55.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.269+0000 7fda92a00640 1 -- 172.21.15.96:0/3965035821 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fda940d3620 con 0x7fda780050a0 2024-05-28T06:15:55.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.269+0000 7fda92a00640 1 -- 172.21.15.96:0/3965035821 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fda7800bfe0 con 0x7fda780043d0 2024-05-28T06:15:55.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.269+0000 7fda92a00640 1 -- 172.21.15.96:0/3965035821 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fda7800c1e0 con 0x7fda780039b0 2024-05-28T06:15:55.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.269+0000 7fda98c00640 1 --2- 172.21.15.96:0/3965035821 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fda780050a0 0x7fda7800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:55.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.270+0000 7fda93400640 1 --2- 172.21.15.96:0/3965035821 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fda780043d0 0x7fda780048a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:55.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.270+0000 7fda98c00640 1 -- 172.21.15.96:0/3965035821 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fda780039b0 msgr2=0x7fda78003db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:15:55.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.270+0000 7fda98c00640 1 --2- 172.21.15.96:0/3965035821 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fda780039b0 0x7fda78003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:55.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.270+0000 7fda98c00640 1 -- 172.21.15.96:0/3965035821 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fda780043d0 msgr2=0x7fda780048a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:15:55.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.270+0000 7fda98c00640 1 --2- 172.21.15.96:0/3965035821 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fda780043d0 0x7fda780048a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:55.273 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.270+0000 7fda98c00640 1 -- 172.21.15.96:0/3965035821 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda940cf770 con 0x7fda780050a0 2024-05-28T06:15:55.273 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.270+0000 7fda93400640 1 --2- 172.21.15.96:0/3965035821 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fda780043d0 0x7fda780048a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-28T06:15:55.273 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.271+0000 7fda98c00640 1 -- 172.21.15.96:0/3965035821 reap_dead start 2024-05-28T06:15:55.273 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.271+0000 7fda98c00640 1 --2- 172.21.15.96:0/3965035821 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fda780050a0 0x7fda7800b9c0 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7fda8c00a9c0 tx=0x7fda78005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f447085af5d8adf server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:15:55.273 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.272+0000 7fda92a00640 1 -- 172.21.15.96:0/3965035821 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda8c00a590 con 0x7fda780050a0 2024-05-28T06:15:55.273 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.272+0000 7fda92a00640 1 -- 172.21.15.96:0/3965035821 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda8c02f070 con 0x7fda780050a0 2024-05-28T06:15:55.273 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.272+0000 7fda92a00640 1 -- 172.21.15.96:0/3965035821 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda8c02be60 con 0x7fda780050a0 2024-05-28T06:15:55.274 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.273+0000 7fda99600640 1 -- 172.21.15.96:0/3965035821 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fda780050a0 msgr2=0x7fda7800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:55.274 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.273+0000 7fda99600640 1 --2- 172.21.15.96:0/3965035821 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fda780050a0 0x7fda7800b9c0 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7fda8c00a9c0 tx=0x7fda78005830 comp rx=0 tx=0).stop 2024-05-28T06:15:55.274 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.273+0000 7fda99600640 1 -- 172.21.15.96:0/3965035821 shutdown_connections 2024-05-28T06:15:55.274 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.273+0000 7fda99600640 1 --2- 172.21.15.96:0/3965035821 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fda780050a0 0x7fda7800b9c0 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:55.274 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.273+0000 7fda99600640 1 -- 172.21.15.96:0/3965035821 >> 172.21.15.96:0/3965035821 conn(0x7fda9409d910 msgr2=0x7fda9408bfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:15:55.274 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.273+0000 7fda99600640 1 -- 172.21.15.96:0/3965035821 shutdown_connections 2024-05-28T06:15:55.274 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.273+0000 7fda99600640 1 -- 172.21.15.96:0/3965035821 wait complete. 2024-05-28T06:15:55.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.276+0000 7fda99600640 1 Processor -- start 2024-05-28T06:15:55.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.277+0000 7fda99600640 1 -- start start 2024-05-28T06:15:55.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.278+0000 7fda99600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fda780050a0 0x7fda940ce0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:55.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.278+0000 7fda99600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fda940ba040 0x7fda941f71d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:55.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.278+0000 7fda99600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fda940cbfa0 0x7fda941610c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:55.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.278+0000 7fda99600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fda9415ebf0 con 0x7fda780050a0 2024-05-28T06:15:55.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.278+0000 7fda99600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fda9415ea70 con 0x7fda940cbfa0 2024-05-28T06:15:55.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.278+0000 7fda99600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fda9415ed70 con 0x7fda940ba040 2024-05-28T06:15:55.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.279+0000 7fda93e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fda780050a0 0x7fda940ce0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:55.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.279+0000 7fda93e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fda780050a0 0x7fda940ce0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:50388/0 (socket says 172.21.15.96:50388) 2024-05-28T06:15:55.280 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.279+0000 7fda93e00640 1 -- 172.21.15.96:0/2291716988 learned_addr learned my addr 172.21.15.96:0/2291716988 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:15:55.280 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.279+0000 7fda93e00640 1 -- 172.21.15.96:0/2291716988 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fda940ba040 msgr2=0x7fda941f71d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:55.280 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.279+0000 7fda93e00640 1 --2- 172.21.15.96:0/2291716988 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fda940ba040 0x7fda941f71d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:55.281 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.279+0000 7fda93e00640 1 -- 172.21.15.96:0/2291716988 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fda940cbfa0 msgr2=0x7fda941610c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:55.281 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.279+0000 7fda93e00640 1 --2- 172.21.15.96:0/2291716988 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fda940cbfa0 0x7fda941610c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:55.281 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.279+0000 7fda93e00640 1 -- 172.21.15.96:0/2291716988 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda941f7ad0 con 0x7fda780050a0 2024-05-28T06:15:55.281 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.280+0000 7fda93e00640 1 --2- 172.21.15.96:0/2291716988 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fda780050a0 0x7fda940ce0d0 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7fda8400c480 tx=0x7fda8400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:15:55.281 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.280+0000 7fda8be00640 1 -- 172.21.15.96:0/2291716988 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda84016020 con 0x7fda780050a0 2024-05-28T06:15:55.281 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.280+0000 7fda99600640 1 -- 172.21.15.96:0/2291716988 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda941f7db0 con 0x7fda780050a0 2024-05-28T06:15:55.281 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.280+0000 7fda8be00640 1 -- 172.21.15.96:0/2291716988 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda84012830 con 0x7fda780050a0 2024-05-28T06:15:55.282 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.280+0000 7fda8be00640 1 -- 172.21.15.96:0/2291716988 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda84012c60 con 0x7fda780050a0 2024-05-28T06:15:55.282 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.280+0000 7fda99600640 1 -- 172.21.15.96:0/2291716988 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda940ce610 con 0x7fda780050a0 2024-05-28T06:15:55.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.282+0000 7fda99600640 1 -- 172.21.15.96:0/2291716988 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda940689e0 con 0x7fda780050a0 2024-05-28T06:15:55.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.284+0000 7fda8be00640 1 -- 172.21.15.96:0/2291716988 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 26) ==== 92570+0+0 (secure 0 0 0) 0x7fda84019020 con 0x7fda780050a0 2024-05-28T06:15:55.286 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.285+0000 7fda8be00640 1 --2- 172.21.15.96:0/2291716988 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fda7406d9d0 0x7fda7406fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:55.286 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.285+0000 7fda8be00640 1 -- 172.21.15.96:0/2291716988 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5795+0+0 (secure 0 0 0) 0x7fda8408a6d0 con 0x7fda780050a0 2024-05-28T06:15:55.291 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.290+0000 7fda8be00640 1 -- 172.21.15.96:0/2291716988 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fda84020020 con 0x7fda780050a0 2024-05-28T06:15:55.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.319+0000 7fda93400640 1 --2- 172.21.15.96:0/2291716988 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fda7406d9d0 0x7fda7406fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:55.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.331+0000 7fda93400640 1 --2- 172.21.15.96:0/2291716988 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fda7406d9d0 0x7fda7406fe90 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fda940ba480 tx=0x7fda80008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:15:55.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.719+0000 7fda99600640 1 -- 172.21.15.96:0/2291716988 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fda9406cbc0 con 0x7fda780050a0 2024-05-28T06:15:55.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.720+0000 7fda8be00640 1 -- 172.21.15.96:0/2291716988 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+2589 (secure 0 0 0) 0x7fda840182c0 con 0x7fda780050a0 2024-05-28T06:15:55.720 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:15:55.721 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:15:55.721 INFO:teuthology.orchestra.run.smithi096.stdout: "fsid": "e4a831b5-e6d1-47a7-96cb-bf85154ee0d2", 2024-05-28T06:15:55.721 INFO:teuthology.orchestra.run.smithi096.stdout: "health": { 2024-05-28T06:15:55.721 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "HEALTH_OK", 2024-05-28T06:15:55.721 INFO:teuthology.orchestra.run.smithi096.stdout: "checks": {}, 2024-05-28T06:15:55.722 INFO:teuthology.orchestra.run.smithi096.stdout: "mutes": [] 2024-05-28T06:15:55.722 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:15:55.722 INFO:teuthology.orchestra.run.smithi096.stdout: "election_epoch": 8, 2024-05-28T06:15:55.722 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum": [ 2024-05-28T06:15:55.722 INFO:teuthology.orchestra.run.smithi096.stdout: 0, 2024-05-28T06:15:55.722 INFO:teuthology.orchestra.run.smithi096.stdout: 1, 2024-05-28T06:15:55.723 INFO:teuthology.orchestra.run.smithi096.stdout: 2 2024-05-28T06:15:55.723 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:15:55.723 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_names": [ 2024-05-28T06:15:55.723 INFO:teuthology.orchestra.run.smithi096.stdout: "a", 2024-05-28T06:15:55.723 INFO:teuthology.orchestra.run.smithi096.stdout: "b", 2024-05-28T06:15:55.723 INFO:teuthology.orchestra.run.smithi096.stdout: "c" 2024-05-28T06:15:55.723 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:15:55.723 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_age": 729, 2024-05-28T06:15:55.723 INFO:teuthology.orchestra.run.smithi096.stdout: "monmap": { 2024-05-28T06:15:55.723 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 1, 2024-05-28T06:15:55.723 INFO:teuthology.orchestra.run.smithi096.stdout: "min_mon_release_name": "squid", 2024-05-28T06:15:55.723 INFO:teuthology.orchestra.run.smithi096.stdout: "num_mons": 3 2024-05-28T06:15:55.723 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:15:55.724 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap": { 2024-05-28T06:15:55.724 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 76, 2024-05-28T06:15:55.724 INFO:teuthology.orchestra.run.smithi096.stdout: "num_osds": 8, 2024-05-28T06:15:55.724 INFO:teuthology.orchestra.run.smithi096.stdout: "num_up_osds": 8, 2024-05-28T06:15:55.724 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_up_since": 1716876238, 2024-05-28T06:15:55.724 INFO:teuthology.orchestra.run.smithi096.stdout: "num_in_osds": 8, 2024-05-28T06:15:55.724 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_in_since": 1716876229, 2024-05-28T06:15:55.724 INFO:teuthology.orchestra.run.smithi096.stdout: "num_remapped_pgs": 0 2024-05-28T06:15:55.724 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:15:55.724 INFO:teuthology.orchestra.run.smithi096.stdout: "pgmap": { 2024-05-28T06:15:55.724 INFO:teuthology.orchestra.run.smithi096.stdout: "pgs_by_state": [ 2024-05-28T06:15:55.724 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:15:55.724 INFO:teuthology.orchestra.run.smithi096.stdout: "state_name": "active+clean", 2024-05-28T06:15:55.724 INFO:teuthology.orchestra.run.smithi096.stdout: "count": 97 2024-05-28T06:15:55.725 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:15:55.725 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:15:55.725 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pgs": 97, 2024-05-28T06:15:55.725 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pools": 3, 2024-05-28T06:15:55.725 INFO:teuthology.orchestra.run.smithi096.stdout: "num_objects": 914, 2024-05-28T06:15:55.725 INFO:teuthology.orchestra.run.smithi096.stdout: "data_bytes": 3696149556, 2024-05-28T06:15:55.725 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_used": 7153541120, 2024-05-28T06:15:55.725 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_avail": 765940572160, 2024-05-28T06:15:55.725 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_total": 773094113280, 2024-05-28T06:15:55.725 INFO:teuthology.orchestra.run.smithi096.stdout: "read_bytes_sec": 72315056, 2024-05-28T06:15:55.725 INFO:teuthology.orchestra.run.smithi096.stdout: "write_bytes_sec": 59237437, 2024-05-28T06:15:55.725 INFO:teuthology.orchestra.run.smithi096.stdout: "read_op_per_sec": 18, 2024-05-28T06:15:55.725 INFO:teuthology.orchestra.run.smithi096.stdout: "write_op_per_sec": 20 2024-05-28T06:15:55.726 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:15:55.726 INFO:teuthology.orchestra.run.smithi096.stdout: "fsmap": { 2024-05-28T06:15:55.726 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 44, 2024-05-28T06:15:55.726 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 5, 2024-05-28T06:15:55.726 INFO:teuthology.orchestra.run.smithi096.stdout: "up": 1, 2024-05-28T06:15:55.726 INFO:teuthology.orchestra.run.smithi096.stdout: "in": 1, 2024-05-28T06:15:55.726 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 1, 2024-05-28T06:15:55.726 INFO:teuthology.orchestra.run.smithi096.stdout: "by_rank": [ 2024-05-28T06:15:55.726 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:15:55.726 INFO:teuthology.orchestra.run.smithi096.stdout: "filesystem_id": 5, 2024-05-28T06:15:55.726 INFO:teuthology.orchestra.run.smithi096.stdout: "rank": 0, 2024-05-28T06:15:55.727 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "d", 2024-05-28T06:15:55.727 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "up:active", 2024-05-28T06:15:55.727 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 7146 2024-05-28T06:15:55.727 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:15:55.727 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:15:55.727 INFO:teuthology.orchestra.run.smithi096.stdout: "up:standby": 3 2024-05-28T06:15:55.727 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:15:55.727 INFO:teuthology.orchestra.run.smithi096.stdout: "mgrmap": { 2024-05-28T06:15:55.727 INFO:teuthology.orchestra.run.smithi096.stdout: "available": true, 2024-05-28T06:15:55.727 INFO:teuthology.orchestra.run.smithi096.stdout: "num_standbys": 1, 2024-05-28T06:15:55.727 INFO:teuthology.orchestra.run.smithi096.stdout: "modules": [ 2024-05-28T06:15:55.727 INFO:teuthology.orchestra.run.smithi096.stdout: "iostat", 2024-05-28T06:15:55.727 INFO:teuthology.orchestra.run.smithi096.stdout: "nfs", 2024-05-28T06:15:55.727 INFO:teuthology.orchestra.run.smithi096.stdout: "restful" 2024-05-28T06:15:55.728 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:15:55.728 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:15:55.728 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:15:55.728 INFO:teuthology.orchestra.run.smithi096.stdout: "servicemap": { 2024-05-28T06:15:55.728 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 12, 2024-05-28T06:15:55.728 INFO:teuthology.orchestra.run.smithi096.stdout: "modified": "2024-05-28T06:15:47.772170+0000", 2024-05-28T06:15:55.728 INFO:teuthology.orchestra.run.smithi096.stdout: "services": { 2024-05-28T06:15:55.728 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-05-28T06:15:55.728 INFO:teuthology.orchestra.run.smithi096.stdout: "daemons": { 2024-05-28T06:15:55.728 INFO:teuthology.orchestra.run.smithi096.stdout: "summary": "", 2024-05-28T06:15:55.728 INFO:teuthology.orchestra.run.smithi096.stdout: "d": { 2024-05-28T06:15:55.728 INFO:teuthology.orchestra.run.smithi096.stdout: "start_epoch": 0, 2024-05-28T06:15:55.728 INFO:teuthology.orchestra.run.smithi096.stdout: "start_stamp": "0.000000", 2024-05-28T06:15:55.729 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 0, 2024-05-28T06:15:55.729 INFO:teuthology.orchestra.run.smithi096.stdout: "addr": "(unrecognized address family 0)/0", 2024-05-28T06:15:55.729 INFO:teuthology.orchestra.run.smithi096.stdout: "metadata": {}, 2024-05-28T06:15:55.729 INFO:teuthology.orchestra.run.smithi096.stdout: "task_status": {} 2024-05-28T06:15:55.729 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:15:55.729 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:15:55.729 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:15:55.729 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:15:55.729 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:15:55.729 INFO:teuthology.orchestra.run.smithi096.stdout: "progress_events": { 2024-05-28T06:15:55.729 INFO:teuthology.orchestra.run.smithi096.stdout: "1f539510-8a8a-46db-8f0c-e233f5cc901d": { 2024-05-28T06:15:55.729 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "4 ongoing clones; Avg progress is 18.099% (4s)\n [=====.......................] (remaining: 19s)", 2024-05-28T06:15:55.729 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": 0.18098999559879303, 2024-05-28T06:15:55.730 INFO:teuthology.orchestra.run.smithi096.stdout: "add_to_ceph_s": true 2024-05-28T06:15:55.730 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:15:55.730 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:15:55.730 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:15:55.730 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.726+0000 7fda99600640 1 -- 172.21.15.96:0/2291716988 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fda7406d9d0 msgr2=0x7fda7406fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:55.731 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.726+0000 7fda99600640 1 --2- 172.21.15.96:0/2291716988 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fda7406d9d0 0x7fda7406fe90 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fda940ba480 tx=0x7fda80008040 comp rx=0 tx=0).stop 2024-05-28T06:15:55.731 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.726+0000 7fda99600640 1 -- 172.21.15.96:0/2291716988 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fda780050a0 msgr2=0x7fda940ce0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:55.731 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.726+0000 7fda99600640 1 --2- 172.21.15.96:0/2291716988 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fda780050a0 0x7fda940ce0d0 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7fda8400c480 tx=0x7fda8400c940 comp rx=0 tx=0).stop 2024-05-28T06:15:55.731 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.729+0000 7fda99600640 1 -- 172.21.15.96:0/2291716988 shutdown_connections 2024-05-28T06:15:55.731 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.729+0000 7fda99600640 1 --2- 172.21.15.96:0/2291716988 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fda7406d9d0 0x7fda7406fe90 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:55.731 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.729+0000 7fda99600640 1 --2- 172.21.15.96:0/2291716988 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fda940cbfa0 0x7fda941610c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:55.731 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.729+0000 7fda99600640 1 --2- 172.21.15.96:0/2291716988 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fda940ba040 0x7fda941f71d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:55.731 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.729+0000 7fda99600640 1 --2- 172.21.15.96:0/2291716988 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fda780050a0 0x7fda940ce0d0 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:55.731 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.729+0000 7fda99600640 1 -- 172.21.15.96:0/2291716988 >> 172.21.15.96:0/2291716988 conn(0x7fda9409d910 msgr2=0x7fda940d28a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:15:55.731 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.729+0000 7fda99600640 1 -- 172.21.15.96:0/2291716988 shutdown_connections 2024-05-28T06:15:55.732 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:55.729+0000 7fda99600640 1 -- 172.21.15.96:0/2291716988 wait complete. 2024-05-28T06:15:56.759 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-05-28T06:15:57.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.018+0000 7fccf1000640 1 Processor -- start 2024-05-28T06:15:57.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.018+0000 7fccf1000640 1 -- start start 2024-05-28T06:15:57.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.020+0000 7fccf1000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fccec15ea40 0x7fccec0d4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:57.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.020+0000 7fccf1000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fccec0d1490 0x7fccec162c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:57.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.020+0000 7fccf1000640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fccec0cf7e0 con 0x7fccec0d1040 2024-05-28T06:15:57.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.020+0000 7fccf1000640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fccec15ee20 con 0x7fccec15ea40 2024-05-28T06:15:57.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.020+0000 7fccf1000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fccec15fe50 con 0x7fccec0d1490 2024-05-28T06:15:57.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.021+0000 7fcce9e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fccec15ea40 0x7fccec0d4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:57.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.021+0000 7fcce9e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fccec15ea40 0x7fccec0d4f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:50390/0 (socket says 172.21.15.96:50390) 2024-05-28T06:15:57.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.021+0000 7fcce9e00640 1 -- 172.21.15.96:0/3854569837 learned_addr learned my addr 172.21.15.96:0/3854569837 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:15:57.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.022+0000 7fccea800640 1 --2- 172.21.15.96:0/3854569837 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fccec0d1490 0x7fccec162c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:57.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.023+0000 7fcce8a00640 1 -- 172.21.15.96:0/3854569837 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1569881351 0 0) 0x7fccec0cf7e0 con 0x7fccec0d1040 2024-05-28T06:15:57.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.023+0000 7fcce8a00640 1 -- 172.21.15.96:0/3854569837 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fccd0003520 con 0x7fccec0d1040 2024-05-28T06:15:57.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.023+0000 7fcce8a00640 1 -- 172.21.15.96:0/3854569837 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1642723168 0 0) 0x7fccd0003520 con 0x7fccec0d1040 2024-05-28T06:15:57.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.023+0000 7fcce8a00640 1 -- 172.21.15.96:0/3854569837 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fccec0d1490 msgr2=0x7fccec162c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:15:57.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.023+0000 7fcce8a00640 1 --2- 172.21.15.96:0/3854569837 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fccec0d1490 0x7fccec162c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:57.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.023+0000 7fcce8a00640 1 -- 172.21.15.96:0/3854569837 >> v2:172.21.15.96:3300/0 conn(0x7fccec15ea40 msgr2=0x7fccec0d4f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:15:57.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.023+0000 7fcce8a00640 1 --2- 172.21.15.96:0/3854569837 >> v2:172.21.15.96:3300/0 conn(0x7fccec15ea40 0x7fccec0d4f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:57.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.023+0000 7fcce8a00640 1 -- 172.21.15.96:0/3854569837 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccec0cfa80 con 0x7fccec0d1040 2024-05-28T06:15:57.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.023+0000 7fcce8a00640 1 -- 172.21.15.96:0/3854569837 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7fccd8001670 con 0x7fccec0d1040 2024-05-28T06:15:57.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.023+0000 7fcce8a00640 1 -- 172.21.15.96:0/3854569837 >> v1:172.21.15.96:6789/0 conn(0x7fccec0d1040 legacy=0x7fccec162520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:57.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.023+0000 7fcce8a00640 1 --2- 172.21.15.96:0/3854569837 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fccec0d1490 0x7fccd00039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:57.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.023+0000 7fcce8a00640 1 --2- 172.21.15.96:0/3854569837 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fccd0003f30 0x7fccd0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:57.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.023+0000 7fcce8a00640 1 --2- 172.21.15.96:0/3854569837 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fccd0004d40 0x7fccd00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:57.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.023+0000 7fcce8a00640 1 -- 172.21.15.96:0/3854569837 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fccec15ee20 con 0x7fccd0003f30 2024-05-28T06:15:57.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.023+0000 7fcce8a00640 1 -- 172.21.15.96:0/3854569837 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fccec15fe50 con 0x7fccd0004d40 2024-05-28T06:15:57.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.023+0000 7fcce8a00640 1 -- 172.21.15.96:0/3854569837 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fccd0014220 con 0x7fccec0d1490 2024-05-28T06:15:57.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.024+0000 7fcce9e00640 1 --2- 172.21.15.96:0/3854569837 >> v2:172.21.15.96:3300/0 conn(0x7fccec15ea40 0x7fccec0d4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-28T06:15:57.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.024+0000 7fcce9400640 1 --2- 172.21.15.96:0/3854569837 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fccd0003f30 0x7fccd0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:57.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.025+0000 7fcce9400640 1 -- 172.21.15.96:0/3854569837 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fccec0d1490 msgr2=0x7fccd00039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:15:57.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.025+0000 7fcce9400640 1 --2- 172.21.15.96:0/3854569837 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fccec0d1490 0x7fccd00039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:57.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.025+0000 7fcce9400640 1 -- 172.21.15.96:0/3854569837 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fccd0004d40 msgr2=0x7fccd00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:15:57.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.025+0000 7fcce9400640 1 --2- 172.21.15.96:0/3854569837 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fccd0004d40 0x7fccd00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:57.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.025+0000 7fcce9400640 1 -- 172.21.15.96:0/3854569837 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccec0cfa80 con 0x7fccd0003f30 2024-05-28T06:15:57.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.028+0000 7fcce9400640 1 --2- 172.21.15.96:0/3854569837 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fccd0003f30 0x7fccd0004800 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7fccd800c1b0 tx=0x7fccd0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=632598e7661e15b7 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:15:57.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.028+0000 7fcce8a00640 1 -- 172.21.15.96:0/3854569837 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccd8037070 con 0x7fccd0003f30 2024-05-28T06:15:57.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.028+0000 7fcce8a00640 1 -- 172.21.15.96:0/3854569837 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fccd8007270 con 0x7fccd0003f30 2024-05-28T06:15:57.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.028+0000 7fcce8a00640 1 -- 172.21.15.96:0/3854569837 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccd8007ca0 con 0x7fccd0003f30 2024-05-28T06:15:57.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.029+0000 7fccf1000640 1 -- 172.21.15.96:0/3854569837 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fccd0003f30 msgr2=0x7fccd0004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:57.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.029+0000 7fccf1000640 1 --2- 172.21.15.96:0/3854569837 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fccd0003f30 0x7fccd0004800 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7fccd800c1b0 tx=0x7fccd0005830 comp rx=0 tx=0).stop 2024-05-28T06:15:57.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.029+0000 7fccea800640 1 -- 172.21.15.96:0/3854569837 reap_dead start 2024-05-28T06:15:57.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.029+0000 7fccf1000640 1 -- 172.21.15.96:0/3854569837 shutdown_connections 2024-05-28T06:15:57.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.029+0000 7fccf1000640 1 -- 172.21.15.96:0/3854569837 >> 172.21.15.96:0/3854569837 conn(0x7fccec08bfc0 msgr2=0x7fccec0d4630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:15:57.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.030+0000 7fccf1000640 1 -- 172.21.15.96:0/3854569837 shutdown_connections 2024-05-28T06:15:57.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.030+0000 7fccf1000640 1 -- 172.21.15.96:0/3854569837 wait complete. 2024-05-28T06:15:57.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.032+0000 7fccf1000640 1 Processor -- start 2024-05-28T06:15:57.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.033+0000 7fccf1000640 1 -- start start 2024-05-28T06:15:57.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.034+0000 7fccf1000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fccd0003f30 0x7fccec158880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:57.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.034+0000 7fccf1000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fccec0b4dd0 0x7fccec1580c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:57.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.034+0000 7fccf1000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fccec0cb820 0x7fccec0cbca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:57.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.034+0000 7fccf1000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fccec0d0320 con 0x7fccd0003f30 2024-05-28T06:15:57.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.034+0000 7fccf1000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fccec0d0040 con 0x7fccec0b4dd0 2024-05-28T06:15:57.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.034+0000 7fccf1000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fccec0d18e0 con 0x7fccec0cb820 2024-05-28T06:15:57.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.034+0000 7fcce9e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fccd0003f30 0x7fccec158880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:57.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.034+0000 7fcce9e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fccd0003f30 0x7fccec158880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:50408/0 (socket says 172.21.15.96:50408) 2024-05-28T06:15:57.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.034+0000 7fcce9e00640 1 -- 172.21.15.96:0/3312231209 learned_addr learned my addr 172.21.15.96:0/3312231209 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:15:57.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.035+0000 7fcce9e00640 1 -- 172.21.15.96:0/3312231209 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fccec0cb820 msgr2=0x7fccec0cbca0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-28T06:15:57.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.035+0000 7fcce9e00640 1 --2- 172.21.15.96:0/3312231209 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fccec0cb820 0x7fccec0cbca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:57.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.035+0000 7fcce9e00640 1 -- 172.21.15.96:0/3312231209 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fccec0b4dd0 msgr2=0x7fccec1580c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:57.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.035+0000 7fcce9e00640 1 --2- 172.21.15.96:0/3312231209 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fccec0b4dd0 0x7fccec1580c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:57.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.035+0000 7fcce9e00640 1 -- 172.21.15.96:0/3312231209 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccec0b55e0 con 0x7fccd0003f30 2024-05-28T06:15:57.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.035+0000 7fcce9e00640 1 --2- 172.21.15.96:0/3312231209 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fccd0003f30 0x7fccec158880 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7fccdc00e930 tx=0x7fccdc00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:15:57.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.036+0000 7fcce2000640 1 -- 172.21.15.96:0/3312231209 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccdc004030 con 0x7fccd0003f30 2024-05-28T06:15:57.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.036+0000 7fcce2000640 1 -- 172.21.15.96:0/3312231209 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fccdc0041d0 con 0x7fccd0003f30 2024-05-28T06:15:57.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.036+0000 7fcce2000640 1 -- 172.21.15.96:0/3312231209 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccdc00c980 con 0x7fccd0003f30 2024-05-28T06:15:57.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.036+0000 7fccf1000640 1 -- 172.21.15.96:0/3312231209 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fccec0b58c0 con 0x7fccd0003f30 2024-05-28T06:15:57.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.036+0000 7fccf1000640 1 -- 172.21.15.96:0/3312231209 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fccec0b7bc0 con 0x7fccd0003f30 2024-05-28T06:15:57.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.038+0000 7fccf1000640 1 -- 172.21.15.96:0/3312231209 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fccec0b5de0 con 0x7fccd0003f30 2024-05-28T06:15:57.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.038+0000 7fcce2000640 1 -- 172.21.15.96:0/3312231209 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 26) ==== 92570+0+0 (secure 0 0 0) 0x7fccdc00cb20 con 0x7fccd0003f30 2024-05-28T06:15:57.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.040+0000 7fcce2000640 1 --2- 172.21.15.96:0/3312231209 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fcccc06d9d0 0x7fcccc06fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:57.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.040+0000 7fcce2000640 1 -- 172.21.15.96:0/3312231209 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5795+0+0 (secure 0 0 0) 0x7fccdc08aef0 con 0x7fccd0003f30 2024-05-28T06:15:57.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.046+0000 7fcce2000640 1 -- 172.21.15.96:0/3312231209 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fccdc017020 con 0x7fccd0003f30 2024-05-28T06:15:57.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.067+0000 7fcce9400640 1 --2- 172.21.15.96:0/3312231209 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fcccc06d9d0 0x7fcccc06fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:57.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.075+0000 7fcce9400640 1 --2- 172.21.15.96:0/3312231209 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fcccc06d9d0 0x7fcccc06fe90 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fccd8034000 tx=0x7fccd8009030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:15:57.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.546+0000 7fccf1000640 1 -- 172.21.15.96:0/3312231209 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fccec0689e0 con 0x7fccd0003f30 2024-05-28T06:15:57.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.546+0000 7fcce2000640 1 -- 172.21.15.96:0/3312231209 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+3600 (secure 0 0 0) 0x7fccdc05ab50 con 0x7fccd0003f30 2024-05-28T06:15:57.547 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:15:57.547 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:15:57.547 INFO:teuthology.orchestra.run.smithi096.stdout: "fsid": "e4a831b5-e6d1-47a7-96cb-bf85154ee0d2", 2024-05-28T06:15:57.547 INFO:teuthology.orchestra.run.smithi096.stdout: "health": { 2024-05-28T06:15:57.547 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "HEALTH_WARN", 2024-05-28T06:15:57.547 INFO:teuthology.orchestra.run.smithi096.stdout: "checks": { 2024-05-28T06:15:57.548 INFO:teuthology.orchestra.run.smithi096.stdout: "PG_DEGRADED": { 2024-05-28T06:15:57.548 INFO:teuthology.orchestra.run.smithi096.stdout: "severity": "HEALTH_WARN", 2024-05-28T06:15:57.548 INFO:teuthology.orchestra.run.smithi096.stdout: "summary": { 2024-05-28T06:15:57.548 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "Degraded data redundancy: 21/3542 objects degraded (0.593%), 1 pg degraded", 2024-05-28T06:15:57.548 INFO:teuthology.orchestra.run.smithi096.stdout: "count": 1 2024-05-28T06:15:57.548 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:15:57.548 INFO:teuthology.orchestra.run.smithi096.stdout: "muted": false 2024-05-28T06:15:57.548 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:15:57.548 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:15:57.548 INFO:teuthology.orchestra.run.smithi096.stdout: "mutes": [] 2024-05-28T06:15:57.548 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:15:57.548 INFO:teuthology.orchestra.run.smithi096.stdout: "election_epoch": 8, 2024-05-28T06:15:57.548 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum": [ 2024-05-28T06:15:57.549 INFO:teuthology.orchestra.run.smithi096.stdout: 0, 2024-05-28T06:15:57.549 INFO:teuthology.orchestra.run.smithi096.stdout: 1, 2024-05-28T06:15:57.549 INFO:teuthology.orchestra.run.smithi096.stdout: 2 2024-05-28T06:15:57.549 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:15:57.549 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_names": [ 2024-05-28T06:15:57.549 INFO:teuthology.orchestra.run.smithi096.stdout: "a", 2024-05-28T06:15:57.549 INFO:teuthology.orchestra.run.smithi096.stdout: "b", 2024-05-28T06:15:57.549 INFO:teuthology.orchestra.run.smithi096.stdout: "c" 2024-05-28T06:15:57.549 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:15:57.549 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_age": 731, 2024-05-28T06:15:57.549 INFO:teuthology.orchestra.run.smithi096.stdout: "monmap": { 2024-05-28T06:15:57.549 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 1, 2024-05-28T06:15:57.549 INFO:teuthology.orchestra.run.smithi096.stdout: "min_mon_release_name": "squid", 2024-05-28T06:15:57.550 INFO:teuthology.orchestra.run.smithi096.stdout: "num_mons": 3 2024-05-28T06:15:57.550 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:15:57.550 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap": { 2024-05-28T06:15:57.550 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 76, 2024-05-28T06:15:57.550 INFO:teuthology.orchestra.run.smithi096.stdout: "num_osds": 8, 2024-05-28T06:15:57.550 INFO:teuthology.orchestra.run.smithi096.stdout: "num_up_osds": 8, 2024-05-28T06:15:57.550 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_up_since": 1716876238, 2024-05-28T06:15:57.550 INFO:teuthology.orchestra.run.smithi096.stdout: "num_in_osds": 8, 2024-05-28T06:15:57.550 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_in_since": 1716876229, 2024-05-28T06:15:57.550 INFO:teuthology.orchestra.run.smithi096.stdout: "num_remapped_pgs": 0 2024-05-28T06:15:57.550 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:15:57.550 INFO:teuthology.orchestra.run.smithi096.stdout: "pgmap": { 2024-05-28T06:15:57.550 INFO:teuthology.orchestra.run.smithi096.stdout: "pgs_by_state": [ 2024-05-28T06:15:57.551 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:15:57.551 INFO:teuthology.orchestra.run.smithi096.stdout: "state_name": "active+clean", 2024-05-28T06:15:57.551 INFO:teuthology.orchestra.run.smithi096.stdout: "count": 95 2024-05-28T06:15:57.551 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:15:57.551 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:15:57.551 INFO:teuthology.orchestra.run.smithi096.stdout: "state_name": "active+recovering", 2024-05-28T06:15:57.551 INFO:teuthology.orchestra.run.smithi096.stdout: "count": 1 2024-05-28T06:15:57.551 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:15:57.551 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:15:57.551 INFO:teuthology.orchestra.run.smithi096.stdout: "state_name": "active+recovery_wait+degraded", 2024-05-28T06:15:57.551 INFO:teuthology.orchestra.run.smithi096.stdout: "count": 1 2024-05-28T06:15:57.551 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:15:57.551 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:15:57.552 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pgs": 97, 2024-05-28T06:15:57.552 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pools": 3, 2024-05-28T06:15:57.552 INFO:teuthology.orchestra.run.smithi096.stdout: "num_objects": 1771, 2024-05-28T06:15:57.552 INFO:teuthology.orchestra.run.smithi096.stdout: "data_bytes": 7286750101, 2024-05-28T06:15:57.552 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_used": 14150492160, 2024-05-28T06:15:57.552 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_avail": 758943621120, 2024-05-28T06:15:57.552 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_total": 773094113280, 2024-05-28T06:15:57.552 INFO:teuthology.orchestra.run.smithi096.stdout: "degraded_objects": 21, 2024-05-28T06:15:57.552 INFO:teuthology.orchestra.run.smithi096.stdout: "degraded_total": 3542, 2024-05-28T06:15:57.552 INFO:teuthology.orchestra.run.smithi096.stdout: "degraded_ratio": 0.0059288537549407102, 2024-05-28T06:15:57.552 INFO:teuthology.orchestra.run.smithi096.stdout: "misplaced_objects": 3, 2024-05-28T06:15:57.552 INFO:teuthology.orchestra.run.smithi096.stdout: "misplaced_total": 3542, 2024-05-28T06:15:57.552 INFO:teuthology.orchestra.run.smithi096.stdout: "misplaced_ratio": 0.00084697910784867312, 2024-05-28T06:15:57.552 INFO:teuthology.orchestra.run.smithi096.stdout: "recovering_objects_per_sec": 0, 2024-05-28T06:15:57.553 INFO:teuthology.orchestra.run.smithi096.stdout: "recovering_bytes_per_sec": 3668322, 2024-05-28T06:15:57.553 INFO:teuthology.orchestra.run.smithi096.stdout: "recovering_keys_per_sec": 0, 2024-05-28T06:15:57.553 INFO:teuthology.orchestra.run.smithi096.stdout: "num_objects_recovered": 7, 2024-05-28T06:15:57.553 INFO:teuthology.orchestra.run.smithi096.stdout: "num_bytes_recovered": 29360128, 2024-05-28T06:15:57.553 INFO:teuthology.orchestra.run.smithi096.stdout: "num_keys_recovered": 0, 2024-05-28T06:15:57.553 INFO:teuthology.orchestra.run.smithi096.stdout: "read_bytes_sec": 529813974, 2024-05-28T06:15:57.553 INFO:teuthology.orchestra.run.smithi096.stdout: "write_bytes_sec": 507864808, 2024-05-28T06:15:57.553 INFO:teuthology.orchestra.run.smithi096.stdout: "read_op_per_sec": 129, 2024-05-28T06:15:57.553 INFO:teuthology.orchestra.run.smithi096.stdout: "write_op_per_sec": 141 2024-05-28T06:15:57.553 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:15:57.553 INFO:teuthology.orchestra.run.smithi096.stdout: "fsmap": { 2024-05-28T06:15:57.553 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 44, 2024-05-28T06:15:57.553 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 5, 2024-05-28T06:15:57.554 INFO:teuthology.orchestra.run.smithi096.stdout: "up": 1, 2024-05-28T06:15:57.554 INFO:teuthology.orchestra.run.smithi096.stdout: "in": 1, 2024-05-28T06:15:57.554 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 1, 2024-05-28T06:15:57.554 INFO:teuthology.orchestra.run.smithi096.stdout: "by_rank": [ 2024-05-28T06:15:57.554 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:15:57.554 INFO:teuthology.orchestra.run.smithi096.stdout: "filesystem_id": 5, 2024-05-28T06:15:57.554 INFO:teuthology.orchestra.run.smithi096.stdout: "rank": 0, 2024-05-28T06:15:57.554 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "d", 2024-05-28T06:15:57.554 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "up:active", 2024-05-28T06:15:57.554 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 7146 2024-05-28T06:15:57.554 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:15:57.554 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:15:57.554 INFO:teuthology.orchestra.run.smithi096.stdout: "up:standby": 3 2024-05-28T06:15:57.554 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:15:57.555 INFO:teuthology.orchestra.run.smithi096.stdout: "mgrmap": { 2024-05-28T06:15:57.555 INFO:teuthology.orchestra.run.smithi096.stdout: "available": true, 2024-05-28T06:15:57.555 INFO:teuthology.orchestra.run.smithi096.stdout: "num_standbys": 1, 2024-05-28T06:15:57.555 INFO:teuthology.orchestra.run.smithi096.stdout: "modules": [ 2024-05-28T06:15:57.555 INFO:teuthology.orchestra.run.smithi096.stdout: "iostat", 2024-05-28T06:15:57.555 INFO:teuthology.orchestra.run.smithi096.stdout: "nfs", 2024-05-28T06:15:57.555 INFO:teuthology.orchestra.run.smithi096.stdout: "restful" 2024-05-28T06:15:57.555 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:15:57.555 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:15:57.555 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:15:57.555 INFO:teuthology.orchestra.run.smithi096.stdout: "servicemap": { 2024-05-28T06:15:57.555 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 12, 2024-05-28T06:15:57.555 INFO:teuthology.orchestra.run.smithi096.stdout: "modified": "2024-05-28T06:15:47.772170+0000", 2024-05-28T06:15:57.556 INFO:teuthology.orchestra.run.smithi096.stdout: "services": { 2024-05-28T06:15:57.556 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-05-28T06:15:57.556 INFO:teuthology.orchestra.run.smithi096.stdout: "daemons": { 2024-05-28T06:15:57.556 INFO:teuthology.orchestra.run.smithi096.stdout: "summary": "", 2024-05-28T06:15:57.556 INFO:teuthology.orchestra.run.smithi096.stdout: "d": { 2024-05-28T06:15:57.556 INFO:teuthology.orchestra.run.smithi096.stdout: "start_epoch": 0, 2024-05-28T06:15:57.556 INFO:teuthology.orchestra.run.smithi096.stdout: "start_stamp": "0.000000", 2024-05-28T06:15:57.556 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 0, 2024-05-28T06:15:57.556 INFO:teuthology.orchestra.run.smithi096.stdout: "addr": "(unrecognized address family 0)/0", 2024-05-28T06:15:57.556 INFO:teuthology.orchestra.run.smithi096.stdout: "metadata": {}, 2024-05-28T06:15:57.556 INFO:teuthology.orchestra.run.smithi096.stdout: "task_status": {} 2024-05-28T06:15:57.556 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:15:57.556 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:15:57.556 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:15:57.557 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:15:57.557 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:15:57.557 INFO:teuthology.orchestra.run.smithi096.stdout: "progress_events": { 2024-05-28T06:15:57.557 INFO:teuthology.orchestra.run.smithi096.stdout: "1f539510-8a8a-46db-8f0c-e233f5cc901d": { 2024-05-28T06:15:57.557 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "4 ongoing clones; Avg progress is 18.099% (4s)\n [=====.......................] (remaining: 19s)", 2024-05-28T06:15:57.557 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": 0.18098999559879303, 2024-05-28T06:15:57.557 INFO:teuthology.orchestra.run.smithi096.stdout: "add_to_ceph_s": true 2024-05-28T06:15:57.557 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:15:57.557 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:15:57.557 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:15:57.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.548+0000 7fccf1000640 1 -- 172.21.15.96:0/3312231209 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fcccc06d9d0 msgr2=0x7fcccc06fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:57.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.548+0000 7fccf1000640 1 --2- 172.21.15.96:0/3312231209 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fcccc06d9d0 0x7fcccc06fe90 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fccd8034000 tx=0x7fccd8009030 comp rx=0 tx=0).stop 2024-05-28T06:15:57.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.548+0000 7fccf1000640 1 -- 172.21.15.96:0/3312231209 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fccd0003f30 msgr2=0x7fccec158880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:57.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.548+0000 7fccf1000640 1 --2- 172.21.15.96:0/3312231209 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fccd0003f30 0x7fccec158880 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7fccdc00e930 tx=0x7fccdc00edf0 comp rx=0 tx=0).stop 2024-05-28T06:15:57.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.549+0000 7fccf1000640 1 -- 172.21.15.96:0/3312231209 shutdown_connections 2024-05-28T06:15:57.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.549+0000 7fccf1000640 1 --2- 172.21.15.96:0/3312231209 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fcccc06d9d0 0x7fcccc06fe90 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:57.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.549+0000 7fccf1000640 1 --2- 172.21.15.96:0/3312231209 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fccec0cb820 0x7fccec0cbca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:57.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.549+0000 7fccf1000640 1 --2- 172.21.15.96:0/3312231209 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fccec0b4dd0 0x7fccec1580c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:57.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.549+0000 7fccf1000640 1 --2- 172.21.15.96:0/3312231209 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fccd0003f30 0x7fccec158880 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:57.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.549+0000 7fccf1000640 1 -- 172.21.15.96:0/3312231209 >> 172.21.15.96:0/3312231209 conn(0x7fccec08bfc0 msgr2=0x7fccec15dc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:15:57.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.549+0000 7fccf1000640 1 -- 172.21.15.96:0/3312231209 shutdown_connections 2024-05-28T06:15:57.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:57.549+0000 7fccf1000640 1 -- 172.21.15.96:0/3312231209 wait complete. 2024-05-28T06:15:58.572 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-05-28T06:15:58.819 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:58.818+0000 7f2d48e00640 1 Processor -- start 2024-05-28T06:15:58.820 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:58.818+0000 7f2d48e00640 1 -- start start 2024-05-28T06:15:58.823 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:58.821+0000 7f2d48e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2d4415d170 0x7f2d440c50c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:58.823 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:58.821+0000 7f2d48e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f2d44163dc0 0x7f2d440d0460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:58.823 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:58.821+0000 7f2d48e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f2d440d09a0 0x7f2d440b4470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:58.823 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:58.821+0000 7f2d48e00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f2d440adb00 con 0x7f2d44163dc0 2024-05-28T06:15:58.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:58.821+0000 7f2d48e00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f2d440d0e30 con 0x7f2d440d09a0 2024-05-28T06:15:58.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:58.821+0000 7f2d48e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f2d4415d550 con 0x7f2d4415d170 2024-05-28T06:15:58.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:58.822+0000 7f2d43400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2d4415d170 0x7f2d440c50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:58.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:58.823+0000 7f2d43400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2d4415d170 0x7f2d440c50c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.96:48360/0 (socket says 172.21.15.96:48360) 2024-05-28T06:15:58.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:58.823+0000 7f2d43400640 1 -- 172.21.15.96:0/2912657969 learned_addr learned my addr 172.21.15.96:0/2912657969 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:15:58.825 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:58.823+0000 7f2d43e00640 1 --2- 172.21.15.96:0/2912657969 >> v2:172.21.15.123:3300/0 conn(0x7f2d440d09a0 0x7f2d440b4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:58.825 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:58.823+0000 7f2d42a00640 1 --2- 172.21.15.96:0/2912657969 >> v2:172.21.15.96:3300/0 conn(0x7f2d44163dc0 0x7f2d440d0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:58.825 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:58.823+0000 7f2d43400640 1 -- 172.21.15.96:0/2912657969 >> v2:172.21.15.123:3300/0 conn(0x7f2d440d09a0 msgr2=0x7f2d440b4470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:15:58.825 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:58.823+0000 7f2d43400640 1 --2- 172.21.15.96:0/2912657969 >> v2:172.21.15.123:3300/0 conn(0x7f2d440d09a0 0x7f2d440b4470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:58.825 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:58.824+0000 7f2d43400640 1 -- 172.21.15.96:0/2912657969 >> v2:172.21.15.96:3300/0 conn(0x7f2d44163dc0 msgr2=0x7f2d440d0460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:15:58.826 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:58.824+0000 7f2d43400640 1 --2- 172.21.15.96:0/2912657969 >> v2:172.21.15.96:3300/0 conn(0x7f2d44163dc0 0x7f2d440d0460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:58.826 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:58.824+0000 7f2d43400640 1 -- 172.21.15.96:0/2912657969 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d440ad1d0 con 0x7f2d4415d170 2024-05-28T06:15:58.826 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:58.824+0000 7f2d42a00640 1 --2- 172.21.15.96:0/2912657969 >> v2:172.21.15.96:3300/0 conn(0x7f2d44163dc0 0x7f2d440d0460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-28T06:15:58.830 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:58.830+0000 7f2d43400640 1 --2- 172.21.15.96:0/2912657969 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2d4415d170 0x7f2d440c50c0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f2d440af870 tx=0x7f2d2c02a1e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8153e1f34ee1be7 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:15:58.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:58.834+0000 7f2d42000640 1 -- 172.21.15.96:0/2912657969 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d2c037070 con 0x7f2d4415d170 2024-05-28T06:15:58.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:58.834+0000 7f2d42000640 1 -- 172.21.15.96:0/2912657969 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d2c032440 con 0x7f2d4415d170 2024-05-28T06:15:58.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:58.834+0000 7f2d42000640 1 -- 172.21.15.96:0/2912657969 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d2c032810 con 0x7f2d4415d170 2024-05-28T06:15:58.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:58.835+0000 7f2d48e00640 1 -- 172.21.15.96:0/2912657969 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2d4415d170 msgr2=0x7f2d440c50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:58.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:58.835+0000 7f2d48e00640 1 --2- 172.21.15.96:0/2912657969 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2d4415d170 0x7f2d440c50c0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f2d440af870 tx=0x7f2d2c02a1e0 comp rx=0 tx=0).stop 2024-05-28T06:15:58.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:58.835+0000 7f2d48e00640 1 -- 172.21.15.96:0/2912657969 shutdown_connections 2024-05-28T06:15:58.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:58.835+0000 7f2d48e00640 1 --2- 172.21.15.96:0/2912657969 >> v2:172.21.15.123:3300/0 conn(0x7f2d440d09a0 0x7f2d440b4470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:58.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:58.835+0000 7f2d48e00640 1 --2- 172.21.15.96:0/2912657969 >> v2:172.21.15.96:3300/0 conn(0x7f2d44163dc0 0x7f2d440d0460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:58.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:58.835+0000 7f2d48e00640 1 --2- 172.21.15.96:0/2912657969 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2d4415d170 0x7f2d440c50c0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:58.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:58.836+0000 7f2d48e00640 1 -- 172.21.15.96:0/2912657969 >> 172.21.15.96:0/2912657969 conn(0x7f2d4406bae0 msgr2=0x7f2d4406bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:15:58.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:58.836+0000 7f2d48e00640 1 -- 172.21.15.96:0/2912657969 shutdown_connections 2024-05-28T06:15:58.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:58.836+0000 7f2d48e00640 1 -- 172.21.15.96:0/2912657969 wait complete. 2024-05-28T06:15:58.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:58.838+0000 7f2d48e00640 1 Processor -- start 2024-05-28T06:15:58.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:58.838+0000 7f2d48e00640 1 -- start start 2024-05-28T06:15:58.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:58.839+0000 7f2d48e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2d440d09a0 0x7f2d440c72f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:58.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:58.839+0000 7f2d48e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2d4415d170 0x7f2d44073410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:58.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:58.839+0000 7f2d48e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2d44163dc0 0x7f2d440cf050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:58.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:58.839+0000 7f2d48e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f2d440b3150 con 0x7f2d44163dc0 2024-05-28T06:15:58.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:58.839+0000 7f2d48e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f2d440ad790 con 0x7f2d4415d170 2024-05-28T06:15:58.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:58.839+0000 7f2d48e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f2d44073950 con 0x7f2d440d09a0 2024-05-28T06:15:58.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:58.840+0000 7f2d43e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2d44163dc0 0x7f2d440cf050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:58.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:58.840+0000 7f2d43e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2d44163dc0 0x7f2d440cf050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44512/0 (socket says 172.21.15.96:44512) 2024-05-28T06:15:58.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:58.840+0000 7f2d43e00640 1 -- 172.21.15.96:0/3958585752 learned_addr learned my addr 172.21.15.96:0/3958585752 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:15:58.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:58.840+0000 7f2d43400640 1 --2- 172.21.15.96:0/3958585752 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2d440d09a0 0x7f2d440c72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:58.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:58.840+0000 7f2d43e00640 1 -- 172.21.15.96:0/3958585752 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2d440d09a0 msgr2=0x7f2d440c72f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:58.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:58.840+0000 7f2d43e00640 1 --2- 172.21.15.96:0/3958585752 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2d440d09a0 0x7f2d440c72f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:58.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:58.840+0000 7f2d43e00640 1 -- 172.21.15.96:0/3958585752 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2d4415d170 msgr2=0x7f2d44073410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:58.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:58.840+0000 7f2d43e00640 1 --2- 172.21.15.96:0/3958585752 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2d4415d170 0x7f2d44073410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:58.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:58.840+0000 7f2d43e00640 1 -- 172.21.15.96:0/3958585752 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d440c7940 con 0x7f2d44163dc0 2024-05-28T06:15:58.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:58.840+0000 7f2d43e00640 1 --2- 172.21.15.96:0/3958585752 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2d44163dc0 0x7f2d440cf050 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f2d3c00d800 tx=0x7f2d3c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:15:58.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:58.841+0000 7f2d3b400640 1 -- 172.21.15.96:0/3958585752 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d3c018020 con 0x7f2d44163dc0 2024-05-28T06:15:58.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:58.841+0000 7f2d3b400640 1 -- 172.21.15.96:0/3958585752 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d3c009660 con 0x7f2d44163dc0 2024-05-28T06:15:58.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:58.841+0000 7f2d48e00640 1 -- 172.21.15.96:0/3958585752 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d440c7c20 con 0x7f2d44163dc0 2024-05-28T06:15:58.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:58.841+0000 7f2d3b400640 1 -- 172.21.15.96:0/3958585752 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d3c009a30 con 0x7f2d44163dc0 2024-05-28T06:15:58.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:58.841+0000 7f2d48e00640 1 -- 172.21.15.96:0/3958585752 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d440c81e0 con 0x7f2d44163dc0 2024-05-28T06:15:58.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:58.843+0000 7f2d48e00640 1 -- 172.21.15.96:0/3958585752 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d440689e0 con 0x7f2d44163dc0 2024-05-28T06:15:58.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:58.844+0000 7f2d3b400640 1 -- 172.21.15.96:0/3958585752 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 26) ==== 92570+0+0 (secure 0 0 0) 0x7f2d3c01b020 con 0x7f2d44163dc0 2024-05-28T06:15:58.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:58.845+0000 7f2d3b400640 1 --2- 172.21.15.96:0/3958585752 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2d2006da00 0x7f2d2006fec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:58.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:58.846+0000 7f2d3b400640 1 -- 172.21.15.96:0/3958585752 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5795+0+0 (secure 0 0 0) 0x7f2d3c08a8b0 con 0x7f2d44163dc0 2024-05-28T06:15:58.852 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:58.852+0000 7f2d3b400640 1 -- 172.21.15.96:0/3958585752 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2d3c022030 con 0x7f2d44163dc0 2024-05-28T06:15:58.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:58.887+0000 7f2d43400640 1 --2- 172.21.15.96:0/3958585752 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2d2006da00 0x7f2d2006fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:58.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:58.899+0000 7f2d43400640 1 --2- 172.21.15.96:0/3958585752 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2d2006da00 0x7f2d2006fec0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f2d2c034000 tx=0x7f2d2c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:15:59.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:59.271+0000 7f2d48e00640 1 -- 172.21.15.96:0/3958585752 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f2d44099810 con 0x7f2d44163dc0 2024-05-28T06:15:59.273 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:59.272+0000 7f2d3b400640 1 -- 172.21.15.96:0/3958585752 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+3621 (secure 0 0 0) 0x7f2d3c05a790 con 0x7f2d44163dc0 2024-05-28T06:15:59.273 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:15:59.273 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:15:59.273 INFO:teuthology.orchestra.run.smithi096.stdout: "fsid": "e4a831b5-e6d1-47a7-96cb-bf85154ee0d2", 2024-05-28T06:15:59.273 INFO:teuthology.orchestra.run.smithi096.stdout: "health": { 2024-05-28T06:15:59.273 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "HEALTH_WARN", 2024-05-28T06:15:59.273 INFO:teuthology.orchestra.run.smithi096.stdout: "checks": { 2024-05-28T06:15:59.273 INFO:teuthology.orchestra.run.smithi096.stdout: "PG_DEGRADED": { 2024-05-28T06:15:59.273 INFO:teuthology.orchestra.run.smithi096.stdout: "severity": "HEALTH_WARN", 2024-05-28T06:15:59.273 INFO:teuthology.orchestra.run.smithi096.stdout: "summary": { 2024-05-28T06:15:59.274 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "Degraded data redundancy: 21/3542 objects degraded (0.593%), 1 pg degraded", 2024-05-28T06:15:59.274 INFO:teuthology.orchestra.run.smithi096.stdout: "count": 1 2024-05-28T06:15:59.274 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:15:59.274 INFO:teuthology.orchestra.run.smithi096.stdout: "muted": false 2024-05-28T06:15:59.274 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:15:59.274 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:15:59.274 INFO:teuthology.orchestra.run.smithi096.stdout: "mutes": [] 2024-05-28T06:15:59.274 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:15:59.274 INFO:teuthology.orchestra.run.smithi096.stdout: "election_epoch": 8, 2024-05-28T06:15:59.274 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum": [ 2024-05-28T06:15:59.274 INFO:teuthology.orchestra.run.smithi096.stdout: 0, 2024-05-28T06:15:59.274 INFO:teuthology.orchestra.run.smithi096.stdout: 1, 2024-05-28T06:15:59.274 INFO:teuthology.orchestra.run.smithi096.stdout: 2 2024-05-28T06:15:59.275 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:15:59.275 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_names": [ 2024-05-28T06:15:59.275 INFO:teuthology.orchestra.run.smithi096.stdout: "a", 2024-05-28T06:15:59.275 INFO:teuthology.orchestra.run.smithi096.stdout: "b", 2024-05-28T06:15:59.275 INFO:teuthology.orchestra.run.smithi096.stdout: "c" 2024-05-28T06:15:59.275 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:15:59.275 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_age": 732, 2024-05-28T06:15:59.275 INFO:teuthology.orchestra.run.smithi096.stdout: "monmap": { 2024-05-28T06:15:59.275 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 1, 2024-05-28T06:15:59.275 INFO:teuthology.orchestra.run.smithi096.stdout: "min_mon_release_name": "squid", 2024-05-28T06:15:59.275 INFO:teuthology.orchestra.run.smithi096.stdout: "num_mons": 3 2024-05-28T06:15:59.275 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:15:59.275 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap": { 2024-05-28T06:15:59.276 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 76, 2024-05-28T06:15:59.276 INFO:teuthology.orchestra.run.smithi096.stdout: "num_osds": 8, 2024-05-28T06:15:59.276 INFO:teuthology.orchestra.run.smithi096.stdout: "num_up_osds": 8, 2024-05-28T06:15:59.276 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_up_since": 1716876238, 2024-05-28T06:15:59.276 INFO:teuthology.orchestra.run.smithi096.stdout: "num_in_osds": 8, 2024-05-28T06:15:59.276 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_in_since": 1716876229, 2024-05-28T06:15:59.276 INFO:teuthology.orchestra.run.smithi096.stdout: "num_remapped_pgs": 0 2024-05-28T06:15:59.276 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:15:59.276 INFO:teuthology.orchestra.run.smithi096.stdout: "pgmap": { 2024-05-28T06:15:59.276 INFO:teuthology.orchestra.run.smithi096.stdout: "pgs_by_state": [ 2024-05-28T06:15:59.276 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:15:59.276 INFO:teuthology.orchestra.run.smithi096.stdout: "state_name": "active+clean", 2024-05-28T06:15:59.276 INFO:teuthology.orchestra.run.smithi096.stdout: "count": 95 2024-05-28T06:15:59.277 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:15:59.277 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:15:59.277 INFO:teuthology.orchestra.run.smithi096.stdout: "state_name": "active+recovering", 2024-05-28T06:15:59.277 INFO:teuthology.orchestra.run.smithi096.stdout: "count": 1 2024-05-28T06:15:59.277 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:15:59.277 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:15:59.277 INFO:teuthology.orchestra.run.smithi096.stdout: "state_name": "active+recovery_wait+degraded", 2024-05-28T06:15:59.277 INFO:teuthology.orchestra.run.smithi096.stdout: "count": 1 2024-05-28T06:15:59.277 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:15:59.277 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:15:59.277 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pgs": 97, 2024-05-28T06:15:59.277 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pools": 3, 2024-05-28T06:15:59.277 INFO:teuthology.orchestra.run.smithi096.stdout: "num_objects": 1771, 2024-05-28T06:15:59.278 INFO:teuthology.orchestra.run.smithi096.stdout: "data_bytes": 7286750101, 2024-05-28T06:15:59.278 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_used": 14150492160, 2024-05-28T06:15:59.278 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_avail": 758943621120, 2024-05-28T06:15:59.278 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_total": 773094113280, 2024-05-28T06:15:59.278 INFO:teuthology.orchestra.run.smithi096.stdout: "degraded_objects": 21, 2024-05-28T06:15:59.278 INFO:teuthology.orchestra.run.smithi096.stdout: "degraded_total": 3542, 2024-05-28T06:15:59.278 INFO:teuthology.orchestra.run.smithi096.stdout: "degraded_ratio": 0.0059288537549407102, 2024-05-28T06:15:59.278 INFO:teuthology.orchestra.run.smithi096.stdout: "misplaced_objects": 3, 2024-05-28T06:15:59.278 INFO:teuthology.orchestra.run.smithi096.stdout: "misplaced_total": 3542, 2024-05-28T06:15:59.278 INFO:teuthology.orchestra.run.smithi096.stdout: "misplaced_ratio": 0.00084697910784867312, 2024-05-28T06:15:59.278 INFO:teuthology.orchestra.run.smithi096.stdout: "recovering_objects_per_sec": 0, 2024-05-28T06:15:59.278 INFO:teuthology.orchestra.run.smithi096.stdout: "recovering_bytes_per_sec": 3667357, 2024-05-28T06:15:59.278 INFO:teuthology.orchestra.run.smithi096.stdout: "recovering_keys_per_sec": 0, 2024-05-28T06:15:59.279 INFO:teuthology.orchestra.run.smithi096.stdout: "num_objects_recovered": 7, 2024-05-28T06:15:59.279 INFO:teuthology.orchestra.run.smithi096.stdout: "num_bytes_recovered": 29360128, 2024-05-28T06:15:59.279 INFO:teuthology.orchestra.run.smithi096.stdout: "num_keys_recovered": 0, 2024-05-28T06:15:59.279 INFO:teuthology.orchestra.run.smithi096.stdout: "read_bytes_sec": 477806982, 2024-05-28T06:15:59.279 INFO:teuthology.orchestra.run.smithi096.stdout: "write_bytes_sec": 468414622, 2024-05-28T06:15:59.279 INFO:teuthology.orchestra.run.smithi096.stdout: "read_op_per_sec": 116, 2024-05-28T06:15:59.279 INFO:teuthology.orchestra.run.smithi096.stdout: "write_op_per_sec": 125 2024-05-28T06:15:59.279 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:15:59.279 INFO:teuthology.orchestra.run.smithi096.stdout: "fsmap": { 2024-05-28T06:15:59.279 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 44, 2024-05-28T06:15:59.279 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 5, 2024-05-28T06:15:59.279 INFO:teuthology.orchestra.run.smithi096.stdout: "up": 1, 2024-05-28T06:15:59.279 INFO:teuthology.orchestra.run.smithi096.stdout: "in": 1, 2024-05-28T06:15:59.279 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 1, 2024-05-28T06:15:59.280 INFO:teuthology.orchestra.run.smithi096.stdout: "by_rank": [ 2024-05-28T06:15:59.280 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:15:59.280 INFO:teuthology.orchestra.run.smithi096.stdout: "filesystem_id": 5, 2024-05-28T06:15:59.280 INFO:teuthology.orchestra.run.smithi096.stdout: "rank": 0, 2024-05-28T06:15:59.280 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "d", 2024-05-28T06:15:59.280 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "up:active", 2024-05-28T06:15:59.280 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 7146 2024-05-28T06:15:59.280 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:15:59.280 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:15:59.280 INFO:teuthology.orchestra.run.smithi096.stdout: "up:standby": 3 2024-05-28T06:15:59.280 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:15:59.280 INFO:teuthology.orchestra.run.smithi096.stdout: "mgrmap": { 2024-05-28T06:15:59.280 INFO:teuthology.orchestra.run.smithi096.stdout: "available": true, 2024-05-28T06:15:59.281 INFO:teuthology.orchestra.run.smithi096.stdout: "num_standbys": 1, 2024-05-28T06:15:59.281 INFO:teuthology.orchestra.run.smithi096.stdout: "modules": [ 2024-05-28T06:15:59.281 INFO:teuthology.orchestra.run.smithi096.stdout: "iostat", 2024-05-28T06:15:59.281 INFO:teuthology.orchestra.run.smithi096.stdout: "nfs", 2024-05-28T06:15:59.281 INFO:teuthology.orchestra.run.smithi096.stdout: "restful" 2024-05-28T06:15:59.281 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:15:59.281 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:15:59.281 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:15:59.281 INFO:teuthology.orchestra.run.smithi096.stdout: "servicemap": { 2024-05-28T06:15:59.281 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 13, 2024-05-28T06:15:59.281 INFO:teuthology.orchestra.run.smithi096.stdout: "modified": "2024-05-28T06:15:57.778889+0000", 2024-05-28T06:15:59.281 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:15:59.281 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:15:59.281 INFO:teuthology.orchestra.run.smithi096.stdout: "progress_events": { 2024-05-28T06:15:59.282 INFO:teuthology.orchestra.run.smithi096.stdout: "1f539510-8a8a-46db-8f0c-e233f5cc901d": { 2024-05-28T06:15:59.282 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "4 ongoing clones; Avg progress is 34.636% (7s)\n [=========...................] (remaining: 14s)", 2024-05-28T06:15:59.282 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": 0.34635999798774719, 2024-05-28T06:15:59.282 INFO:teuthology.orchestra.run.smithi096.stdout: "add_to_ceph_s": true 2024-05-28T06:15:59.282 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:15:59.282 INFO:teuthology.orchestra.run.smithi096.stdout: "897cfd0c-3f6f-4c85-8c76-d97607cbfcd3": { 2024-05-28T06:15:59.282 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "Global Recovery Event (0s)\n [............................] ", 2024-05-28T06:15:59.282 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": 0, 2024-05-28T06:15:59.282 INFO:teuthology.orchestra.run.smithi096.stdout: "add_to_ceph_s": true 2024-05-28T06:15:59.282 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:15:59.282 INFO:teuthology.orchestra.run.smithi096.stdout: "d1d4f508-f6fa-44f6-a2d0-6e7bd476fb5a": { 2024-05-28T06:15:59.282 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "5 ongoing+pending clones; Avg progress is 31.823% (0s)\n [========....................] ", 2024-05-28T06:15:59.282 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": 0.31823000311851501, 2024-05-28T06:15:59.282 INFO:teuthology.orchestra.run.smithi096.stdout: "add_to_ceph_s": true 2024-05-28T06:15:59.283 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:15:59.283 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:15:59.283 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:15:59.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:59.276+0000 7f2d48e00640 1 -- 172.21.15.96:0/3958585752 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2d2006da00 msgr2=0x7f2d2006fec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:59.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:59.276+0000 7f2d48e00640 1 --2- 172.21.15.96:0/3958585752 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2d2006da00 0x7f2d2006fec0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f2d2c034000 tx=0x7f2d2c02e000 comp rx=0 tx=0).stop 2024-05-28T06:15:59.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:59.276+0000 7f2d48e00640 1 -- 172.21.15.96:0/3958585752 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2d44163dc0 msgr2=0x7f2d440cf050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:59.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:59.276+0000 7f2d48e00640 1 --2- 172.21.15.96:0/3958585752 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2d44163dc0 0x7f2d440cf050 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f2d3c00d800 tx=0x7f2d3c00dcc0 comp rx=0 tx=0).stop 2024-05-28T06:15:59.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:59.277+0000 7f2d48e00640 1 -- 172.21.15.96:0/3958585752 shutdown_connections 2024-05-28T06:15:59.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:59.277+0000 7f2d48e00640 1 --2- 172.21.15.96:0/3958585752 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2d2006da00 0x7f2d2006fec0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:59.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:59.277+0000 7f2d48e00640 1 --2- 172.21.15.96:0/3958585752 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2d44163dc0 0x7f2d440cf050 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:59.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:59.277+0000 7f2d48e00640 1 --2- 172.21.15.96:0/3958585752 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2d4415d170 0x7f2d44073410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:59.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:59.277+0000 7f2d48e00640 1 --2- 172.21.15.96:0/3958585752 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2d440d09a0 0x7f2d440c72f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:59.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:59.277+0000 7f2d48e00640 1 -- 172.21.15.96:0/3958585752 >> 172.21.15.96:0/3958585752 conn(0x7f2d4406bae0 msgr2=0x7f2d440b4be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:15:59.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:59.277+0000 7f2d48e00640 1 -- 172.21.15.96:0/3958585752 shutdown_connections 2024-05-28T06:15:59.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:59.277+0000 7f2d48e00640 1 -- 172.21.15.96:0/3958585752 wait complete. 2024-05-28T06:15:59.310 INFO:tasks.cephfs.test_volumes:pevs - {'1f539510-8a8a-46db-8f0c-e233f5cc901d': {'message': '4 ongoing clones; Avg progress is 34.636% (7s)\n [=========...................] (remaining: 14s)', 'progress': 0.3463599979877472, 'add_to_ceph_s': True}, 'd1d4f508-f6fa-44f6-a2d0-6e7bd476fb5a': {'message': '5 ongoing+pending clones; Avg progress is 31.823% (0s)\n [========....................] ', 'progress': 0.318230003118515, 'add_to_ceph_s': True}} 2024-05-28T06:15:59.311 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone43636 2024-05-28T06:15:59.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:59.597+0000 7f59a2000640 1 Processor -- start 2024-05-28T06:15:59.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:59.597+0000 7f59a2000640 1 -- start start 2024-05-28T06:15:59.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:59.599+0000 7f59a2000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f599c0b52a0 0x7f599c162d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:59.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:59.600+0000 7f59a2000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f599c0b56d0 0x7f599c0b3cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:59.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:59.600+0000 7f59a2000640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f599c162380 con 0x7f599c161fa0 2024-05-28T06:15:59.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:59.600+0000 7f59a2000640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f599c0d33e0 con 0x7f599c0b52a0 2024-05-28T06:15:59.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:59.600+0000 7f59a2000640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f599c0b5bd0 con 0x7f599c0b56d0 2024-05-28T06:15:59.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:59.602+0000 7f599be00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f599c0b52a0 0x7f599c162d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:59.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:59.602+0000 7f599be00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f599c0b52a0 0x7f599c162d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44522/0 (socket says 172.21.15.96:44522) 2024-05-28T06:15:59.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:59.602+0000 7f599be00640 1 -- 172.21.15.96:0/4115377605 learned_addr learned my addr 172.21.15.96:0/4115377605 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:15:59.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:59.602+0000 7f599be00640 1 -- 172.21.15.96:0/4115377605 >> v2:172.21.15.123:3300/0 conn(0x7f599c0b56d0 msgr2=0x7f599c0b3cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:15:59.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:59.602+0000 7f599be00640 1 --2- 172.21.15.96:0/4115377605 >> v2:172.21.15.123:3300/0 conn(0x7f599c0b56d0 0x7f599c0b3cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:59.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:59.602+0000 7f599be00640 1 -- 172.21.15.96:0/4115377605 >> v1:172.21.15.123:6789/0 conn(0x7f599c161fa0 legacy=0x7f599c162600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:15:59.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:59.603+0000 7f599be00640 1 -- 172.21.15.96:0/4115377605 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f599c0d2e20 con 0x7f599c0b52a0 2024-05-28T06:15:59.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:59.604+0000 7f599be00640 1 --2- 172.21.15.96:0/4115377605 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f599c0b52a0 0x7f599c162d10 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f599c1636a0 tx=0x7f598802a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=524ae1591af709bc server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:15:59.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:59.605+0000 7f599b400640 1 -- 172.21.15.96:0/4115377605 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5988037070 con 0x7f599c0b52a0 2024-05-28T06:15:59.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:59.605+0000 7f599b400640 1 -- 172.21.15.96:0/4115377605 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5988032440 con 0x7f599c0b52a0 2024-05-28T06:15:59.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:59.606+0000 7f59a2000640 1 -- 172.21.15.96:0/4115377605 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f599c0b52a0 msgr2=0x7f599c162d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:59.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:59.606+0000 7f59a2000640 1 --2- 172.21.15.96:0/4115377605 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f599c0b52a0 0x7f599c162d10 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f599c1636a0 tx=0x7f598802a1c0 comp rx=0 tx=0).stop 2024-05-28T06:15:59.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:59.608+0000 7f59a2000640 1 -- 172.21.15.96:0/4115377605 shutdown_connections 2024-05-28T06:15:59.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:59.608+0000 7f59a2000640 1 --2- 172.21.15.96:0/4115377605 >> v2:172.21.15.123:3300/0 conn(0x7f599c0b56d0 0x7f599c0b3cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:59.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:59.608+0000 7f59a2000640 1 --2- 172.21.15.96:0/4115377605 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f599c0b52a0 0x7f599c162d10 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:59.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:59.608+0000 7f59a2000640 1 -- 172.21.15.96:0/4115377605 >> 172.21.15.96:0/4115377605 conn(0x7f599c08bfc0 msgr2=0x7f599c0d4470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:15:59.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:59.608+0000 7f59a2000640 1 -- 172.21.15.96:0/4115377605 shutdown_connections 2024-05-28T06:15:59.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:59.609+0000 7f59a2000640 1 -- 172.21.15.96:0/4115377605 wait complete. 2024-05-28T06:15:59.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:59.614+0000 7f59a2000640 1 Processor -- start 2024-05-28T06:15:59.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:59.615+0000 7f59a2000640 1 -- start start 2024-05-28T06:15:59.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:59.616+0000 7f59a2000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f599c0b52a0 0x7f599c0cdfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:59.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:59.616+0000 7f59a2000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f599c0b56d0 0x7f599c0b7090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:59.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:59.617+0000 7f59a2000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f599c161fa0 0x7f599c15b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:59.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:59.617+0000 7f59a2000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f599c0d51d0 con 0x7f599c0b52a0 2024-05-28T06:15:59.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:59.617+0000 7f59a2000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f599c0d4cb0 con 0x7f599c0b56d0 2024-05-28T06:15:59.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:59.617+0000 7f59a2000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f599c0d5350 con 0x7f599c161fa0 2024-05-28T06:15:59.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:59.617+0000 7f59a0c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f599c0b52a0 0x7f599c0cdfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:59.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:59.618+0000 7f59a0c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f599c0b52a0 0x7f599c0cdfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44526/0 (socket says 172.21.15.96:44526) 2024-05-28T06:15:59.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:59.618+0000 7f59a0c00640 1 -- 172.21.15.96:0/1817134853 learned_addr learned my addr 172.21.15.96:0/1817134853 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:15:59.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:59.618+0000 7f59a0c00640 1 -- 172.21.15.96:0/1817134853 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f599c161fa0 msgr2=0x7f599c15b160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-28T06:15:59.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:59.618+0000 7f59a0c00640 1 --2- 172.21.15.96:0/1817134853 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f599c161fa0 0x7f599c15b160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:59.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:59.618+0000 7f59a0c00640 1 -- 172.21.15.96:0/1817134853 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f599c0b56d0 msgr2=0x7f599c0b7090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:15:59.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:59.618+0000 7f59a0c00640 1 --2- 172.21.15.96:0/1817134853 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f599c0b56d0 0x7f599c0b7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:15:59.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:59.618+0000 7f59a0c00640 1 -- 172.21.15.96:0/1817134853 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f599c0b75d0 con 0x7f599c0b52a0 2024-05-28T06:15:59.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:59.619+0000 7f59a0c00640 1 --2- 172.21.15.96:0/1817134853 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f599c0b52a0 0x7f599c0cdfd0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f599000b3e0 tx=0x7f599000b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:15:59.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:59.621+0000 7f5998c00640 1 -- 172.21.15.96:0/1817134853 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5990015020 con 0x7f599c0b52a0 2024-05-28T06:15:59.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:59.621+0000 7f5998c00640 1 -- 172.21.15.96:0/1817134853 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5990011850 con 0x7f599c0b52a0 2024-05-28T06:15:59.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:59.621+0000 7f5998c00640 1 -- 172.21.15.96:0/1817134853 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5990011c40 con 0x7f599c0b52a0 2024-05-28T06:15:59.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:59.623+0000 7f59a2000640 1 -- 172.21.15.96:0/1817134853 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f599c0b9730 con 0x7f599c0b52a0 2024-05-28T06:15:59.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:59.623+0000 7f59a2000640 1 -- 172.21.15.96:0/1817134853 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f599c0b9cf0 con 0x7f599c0b52a0 2024-05-28T06:15:59.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:59.626+0000 7f59a2000640 1 -- 172.21.15.96:0/1817134853 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f599c0ba030 con 0x7f599c0b52a0 2024-05-28T06:15:59.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:59.630+0000 7f5998c00640 1 -- 172.21.15.96:0/1817134853 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 26) ==== 92570+0+0 (secure 0 0 0) 0x7f5990018020 con 0x7f599c0b52a0 2024-05-28T06:15:59.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:59.632+0000 7f5998c00640 1 --2- 172.21.15.96:0/1817134853 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f597c06daa0 0x7f597c06ff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:15:59.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:59.632+0000 7f5998c00640 1 -- 172.21.15.96:0/1817134853 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5795+0+0 (secure 0 0 0) 0x7f599008acc0 con 0x7f599c0b52a0 2024-05-28T06:15:59.640 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:59.640+0000 7f5998c00640 1 -- 172.21.15.96:0/1817134853 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f599c0ba030 con 0x7f599c0b52a0 2024-05-28T06:15:59.711 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:59.711+0000 7f599be00640 1 --2- 172.21.15.96:0/1817134853 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f597c06daa0 0x7f597c06ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:15:59.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:15:59.782+0000 7f599be00640 1 --2- 172.21.15.96:0/1817134853 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f597c06daa0 0x7f597c06ff60 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f5988034000 tx=0x7f598802c000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:00.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:00.008+0000 7f59a2000640 1 -- 172.21.15.96:0/1817134853 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone43636", "target": ["mon-mgr", ""]}) -- 0x7f599c08c460 con 0x7f597c06daa0 2024-05-28T06:16:01.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:01.141+0000 7f5998c00640 1 -- 172.21.15.96:0/1817134853 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f599c08c460 con 0x7f597c06daa0 2024-05-28T06:16:01.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:01.147+0000 7f59a2000640 1 -- 172.21.15.96:0/1817134853 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f597c06daa0 msgr2=0x7f597c06ff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:01.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:01.147+0000 7f59a2000640 1 --2- 172.21.15.96:0/1817134853 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f597c06daa0 0x7f597c06ff60 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f5988034000 tx=0x7f598802c000 comp rx=0 tx=0).stop 2024-05-28T06:16:01.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:01.148+0000 7f59a2000640 1 -- 172.21.15.96:0/1817134853 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f599c0b52a0 msgr2=0x7f599c0cdfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:01.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:01.148+0000 7f59a2000640 1 --2- 172.21.15.96:0/1817134853 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f599c0b52a0 0x7f599c0cdfd0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f599000b3e0 tx=0x7f599000b8a0 comp rx=0 tx=0).stop 2024-05-28T06:16:01.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:01.148+0000 7f59a2000640 1 -- 172.21.15.96:0/1817134853 shutdown_connections 2024-05-28T06:16:01.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:01.148+0000 7f59a2000640 1 --2- 172.21.15.96:0/1817134853 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f597c06daa0 0x7f597c06ff60 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:01.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:01.148+0000 7f59a2000640 1 --2- 172.21.15.96:0/1817134853 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f599c161fa0 0x7f599c15b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:01.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:01.148+0000 7f59a2000640 1 --2- 172.21.15.96:0/1817134853 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f599c0b56d0 0x7f599c0b7090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:01.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:01.148+0000 7f59a2000640 1 --2- 172.21.15.96:0/1817134853 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f599c0b52a0 0x7f599c0cdfd0 secure :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f599000b3e0 tx=0x7f599000b8a0 comp rx=0 tx=0).stop 2024-05-28T06:16:01.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:01.148+0000 7f59a2000640 1 -- 172.21.15.96:0/1817134853 >> 172.21.15.96:0/1817134853 conn(0x7f599c08bfc0 msgr2=0x7f599c0d0e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:16:01.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:01.149+0000 7f59a2000640 1 -- 172.21.15.96:0/1817134853 shutdown_connections 2024-05-28T06:16:01.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:01.149+0000 7f59a2000640 1 -- 172.21.15.96:0/1817134853 wait complete. 2024-05-28T06:16:01.185 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone43637 2024-05-28T06:16:01.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:01.482+0000 7f2fc9200640 1 Processor -- start 2024-05-28T06:16:01.672 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:01.482+0000 7f2fc9200640 1 -- start start 2024-05-28T06:16:01.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:01.483+0000 7f2fc9200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2fc4161d00 0x7f2fc40b3d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:01.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:01.483+0000 7f2fc9200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f2fc40d4d90 0x7f2fc40d0e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:01.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:01.483+0000 7f2fc9200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fc40cfb40 con 0x7f2fc40d1380 2024-05-28T06:16:01.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:01.483+0000 7f2fc9200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f2fc40d17e0 con 0x7f2fc40d4d90 2024-05-28T06:16:01.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:01.483+0000 7f2fc9200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f2fc40d2040 con 0x7f2fc4161d00 2024-05-28T06:16:01.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:01.483+0000 7f2fc2a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f2fc40d4d90 0x7f2fc40d0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:01.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:01.484+0000 7f2fc2a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f2fc40d4d90 0x7f2fc40d0e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44528/0 (socket says 172.21.15.96:44528) 2024-05-28T06:16:01.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:01.484+0000 7f2fc2a00640 1 -- 172.21.15.96:0/2935614601 learned_addr learned my addr 172.21.15.96:0/2935614601 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:16:01.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:01.484+0000 7f2fc2a00640 1 -- 172.21.15.96:0/2935614601 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2fc4161d00 msgr2=0x7f2fc40b3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:16:01.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:01.484+0000 7f2fc2a00640 1 --2- 172.21.15.96:0/2935614601 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2fc4161d00 0x7f2fc40b3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:01.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:01.484+0000 7f2fc2a00640 1 -- 172.21.15.96:0/2935614601 >> v1:172.21.15.123:6789/0 conn(0x7f2fc40d1380 legacy=0x7f2fc40d27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:16:01.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:01.484+0000 7f2fc2a00640 1 -- 172.21.15.96:0/2935614601 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fc40cfdc0 con 0x7f2fc40d4d90 2024-05-28T06:16:01.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:01.485+0000 7f2fc2a00640 1 --2- 172.21.15.96:0/2935614601 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2fc40d4d90 0x7f2fc40d0e40 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f2fb0009870 tx=0x7f2fb002a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0de20876addd258 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:01.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:01.485+0000 7f2fc2000640 1 -- 172.21.15.96:0/2935614601 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fb0037070 con 0x7f2fc40d4d90 2024-05-28T06:16:01.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:01.486+0000 7f2fc2000640 1 -- 172.21.15.96:0/2935614601 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fb0032440 con 0x7f2fc40d4d90 2024-05-28T06:16:01.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:01.486+0000 7f2fc2000640 1 -- 172.21.15.96:0/2935614601 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fb0032810 con 0x7f2fc40d4d90 2024-05-28T06:16:01.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:01.486+0000 7f2fc9200640 1 -- 172.21.15.96:0/2935614601 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2fc40d4d90 msgr2=0x7f2fc40d0e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:01.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:01.486+0000 7f2fc9200640 1 --2- 172.21.15.96:0/2935614601 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2fc40d4d90 0x7f2fc40d0e40 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f2fb0009870 tx=0x7f2fb002a210 comp rx=0 tx=0).stop 2024-05-28T06:16:01.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:01.487+0000 7f2fc9200640 1 -- 172.21.15.96:0/2935614601 shutdown_connections 2024-05-28T06:16:01.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:01.487+0000 7f2fc9200640 1 --2- 172.21.15.96:0/2935614601 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2fc40d4d90 0x7f2fc40d0e40 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:01.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:01.487+0000 7f2fc9200640 1 --2- 172.21.15.96:0/2935614601 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2fc4161d00 0x7f2fc40b3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:01.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:01.487+0000 7f2fc9200640 1 -- 172.21.15.96:0/2935614601 >> 172.21.15.96:0/2935614601 conn(0x7f2fc408bfc0 msgr2=0x7f2fc40d4470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:16:01.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:01.487+0000 7f2fc9200640 1 -- 172.21.15.96:0/2935614601 shutdown_connections 2024-05-28T06:16:01.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:01.487+0000 7f2fc9200640 1 -- 172.21.15.96:0/2935614601 wait complete. 2024-05-28T06:16:01.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:01.488+0000 7f2fc9200640 1 Processor -- start 2024-05-28T06:16:01.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:01.489+0000 7f2fc9200640 1 -- start start 2024-05-28T06:16:01.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:01.491+0000 7f2fc9200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2fc40d1380 0x7f2fc40cb620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:01.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:01.491+0000 7f2fc9200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2fc40d4d90 0x7f2fc414aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:01.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:01.491+0000 7f2fc9200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2fc4161d00 0x7f2fc414af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:01.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:01.491+0000 7f2fc9200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f2fc40d2290 con 0x7f2fc40d1380 2024-05-28T06:16:01.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:01.491+0000 7f2fc9200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f2fc40d0380 con 0x7f2fc40d4d90 2024-05-28T06:16:01.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:01.491+0000 7f2fc9200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f2fc40b5560 con 0x7f2fc4161d00 2024-05-28T06:16:01.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:01.491+0000 7f2fc3400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2fc40d1380 0x7f2fc40cb620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:01.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:01.491+0000 7f2fc3400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2fc40d1380 0x7f2fc40cb620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44542/0 (socket says 172.21.15.96:44542) 2024-05-28T06:16:01.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:01.491+0000 7f2fc3400640 1 -- 172.21.15.96:0/4070376981 learned_addr learned my addr 172.21.15.96:0/4070376981 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:16:01.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:01.491+0000 7f2fc3e00640 1 --2- 172.21.15.96:0/4070376981 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2fc4161d00 0x7f2fc414af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:01.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:01.491+0000 7f2fc2a00640 1 --2- 172.21.15.96:0/4070376981 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2fc40d4d90 0x7f2fc414aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:01.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:01.491+0000 7f2fc3400640 1 -- 172.21.15.96:0/4070376981 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2fc4161d00 msgr2=0x7f2fc414af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:01.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:01.491+0000 7f2fc3400640 1 --2- 172.21.15.96:0/4070376981 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2fc4161d00 0x7f2fc414af40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:01.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:01.491+0000 7f2fc3400640 1 -- 172.21.15.96:0/4070376981 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2fc40d4d90 msgr2=0x7f2fc414aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:01.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:01.491+0000 7f2fc3400640 1 --2- 172.21.15.96:0/4070376981 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2fc40d4d90 0x7f2fc414aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:01.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:01.491+0000 7f2fc3400640 1 -- 172.21.15.96:0/4070376981 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fc40b1f80 con 0x7f2fc40d1380 2024-05-28T06:16:01.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:01.491+0000 7f2fc3400640 1 --2- 172.21.15.96:0/4070376981 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2fc40d1380 0x7f2fc40cb620 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f2fb400b410 tx=0x7f2fb400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:01.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:01.492+0000 7f2fbb400640 1 -- 172.21.15.96:0/4070376981 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fb4015020 con 0x7f2fc40d1380 2024-05-28T06:16:01.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:01.492+0000 7f2fbb400640 1 -- 172.21.15.96:0/4070376981 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fb4011850 con 0x7f2fc40d1380 2024-05-28T06:16:01.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:01.492+0000 7f2fbb400640 1 -- 172.21.15.96:0/4070376981 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fb4011c40 con 0x7f2fc40d1380 2024-05-28T06:16:01.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:01.492+0000 7f2fc9200640 1 -- 172.21.15.96:0/4070376981 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fc40b2260 con 0x7f2fc40d1380 2024-05-28T06:16:01.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:01.492+0000 7f2fc9200640 1 -- 172.21.15.96:0/4070376981 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fc40b2820 con 0x7f2fc40d1380 2024-05-28T06:16:01.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:01.494+0000 7f2fc9200640 1 -- 172.21.15.96:0/4070376981 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2fc40689e0 con 0x7f2fc40d1380 2024-05-28T06:16:01.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:01.495+0000 7f2fbb400640 1 -- 172.21.15.96:0/4070376981 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 26) ==== 92570+0+0 (secure 0 0 0) 0x7f2fb4018020 con 0x7f2fc40d1380 2024-05-28T06:16:01.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:01.497+0000 7f2fbb400640 1 --2- 172.21.15.96:0/4070376981 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2fa406da00 0x7f2fa406fec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:01.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:01.497+0000 7f2fbb400640 1 -- 172.21.15.96:0/4070376981 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5795+0+0 (secure 0 0 0) 0x7f2fb408a570 con 0x7f2fc40d1380 2024-05-28T06:16:01.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:01.504+0000 7f2fbb400640 1 -- 172.21.15.96:0/4070376981 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2fb401f020 con 0x7f2fc40d1380 2024-05-28T06:16:01.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:01.538+0000 7f2fc2a00640 1 --2- 172.21.15.96:0/4070376981 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2fa406da00 0x7f2fa406fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:01.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:01.540+0000 7f2fc2a00640 1 --2- 172.21.15.96:0/4070376981 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2fa406da00 0x7f2fa406fec0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f2fb0034000 tx=0x7f2fb002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:01.849 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:01.849+0000 7f2fc9200640 1 -- 172.21.15.96:0/4070376981 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone43637", "target": ["mon-mgr", ""]}) -- 0x7f2fc40ce570 con 0x7f2fa406da00 2024-05-28T06:16:02.145 DEBUG:teuthology.orchestra.run.smithi096:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:16:02.156 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:16:03.174 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:03.175+0000 7f2fbb400640 1 -- 172.21.15.96:0/4070376981 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f2fc40ce570 con 0x7f2fa406da00 2024-05-28T06:16:03.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:03.179+0000 7f2fc9200640 1 -- 172.21.15.96:0/4070376981 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2fa406da00 msgr2=0x7f2fa406fec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:03.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:03.179+0000 7f2fc9200640 1 --2- 172.21.15.96:0/4070376981 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2fa406da00 0x7f2fa406fec0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f2fb0034000 tx=0x7f2fb002e000 comp rx=0 tx=0).stop 2024-05-28T06:16:03.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:03.179+0000 7f2fc9200640 1 -- 172.21.15.96:0/4070376981 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2fc40d1380 msgr2=0x7f2fc40cb620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:03.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:03.179+0000 7f2fc9200640 1 --2- 172.21.15.96:0/4070376981 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2fc40d1380 0x7f2fc40cb620 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f2fb400b410 tx=0x7f2fb400b8d0 comp rx=0 tx=0).stop 2024-05-28T06:16:03.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:03.180+0000 7f2fc9200640 1 -- 172.21.15.96:0/4070376981 shutdown_connections 2024-05-28T06:16:03.180 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:03.180+0000 7f2fc9200640 1 --2- 172.21.15.96:0/4070376981 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2fa406da00 0x7f2fa406fec0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:03.180 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:03.180+0000 7f2fc9200640 1 --2- 172.21.15.96:0/4070376981 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2fc4161d00 0x7f2fc414af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:03.180 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:03.180+0000 7f2fc9200640 1 --2- 172.21.15.96:0/4070376981 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2fc40d4d90 0x7f2fc414aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:03.180 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:03.180+0000 7f2fc9200640 1 --2- 172.21.15.96:0/4070376981 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2fc40d1380 0x7f2fc40cb620 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:03.180 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:03.180+0000 7f2fc9200640 1 -- 172.21.15.96:0/4070376981 >> 172.21.15.96:0/4070376981 conn(0x7f2fc408bfc0 msgr2=0x7f2fc40b61b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:16:03.180 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:03.180+0000 7f2fc9200640 1 -- 172.21.15.96:0/4070376981 shutdown_connections 2024-05-28T06:16:03.180 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:03.180+0000 7f2fc9200640 1 -- 172.21.15.96:0/4070376981 wait complete. 2024-05-28T06:16:03.211 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone43638 2024-05-28T06:16:03.489 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:03.490+0000 7f64dde00640 1 Processor -- start 2024-05-28T06:16:03.489 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:03.490+0000 7f64dde00640 1 -- start start 2024-05-28T06:16:03.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:03.492+0000 7f64dde00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f64d8156bd0 0x7f64d80b3af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:03.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:03.492+0000 7f64dde00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f64d80b4e30 0x7f64d815caf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:03.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:03.492+0000 7f64dde00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64d8156fb0 con 0x7f64d815d030 2024-05-28T06:16:03.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:03.492+0000 7f64dde00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f64d815dae0 con 0x7f64d8156bd0 2024-05-28T06:16:03.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:03.492+0000 7f64dde00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f64d815d520 con 0x7f64d80b4e30 2024-05-28T06:16:03.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:03.492+0000 7f64d6a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f64d8156bd0 0x7f64d80b3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:03.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:03.492+0000 7f64d6a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f64d8156bd0 0x7f64d80b3af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44546/0 (socket says 172.21.15.96:44546) 2024-05-28T06:16:03.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:03.492+0000 7f64d6a00640 1 -- 172.21.15.96:0/2355869903 learned_addr learned my addr 172.21.15.96:0/2355869903 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:16:03.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:03.493+0000 7f64d6a00640 1 -- 172.21.15.96:0/2355869903 >> v2:172.21.15.123:3300/0 conn(0x7f64d80b4e30 msgr2=0x7f64d815caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:16:03.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:03.493+0000 7f64d6a00640 1 --2- 172.21.15.96:0/2355869903 >> v2:172.21.15.123:3300/0 conn(0x7f64d80b4e30 0x7f64d815caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:03.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:03.493+0000 7f64d6a00640 1 -- 172.21.15.96:0/2355869903 >> v1:172.21.15.123:6789/0 conn(0x7f64d815d030 legacy=0x7f64d80d06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:16:03.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:03.493+0000 7f64d6a00640 1 -- 172.21.15.96:0/2355869903 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64d815d800 con 0x7f64d8156bd0 2024-05-28T06:16:03.494 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:03.495+0000 7f64d6a00640 1 --2- 172.21.15.96:0/2355869903 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f64d8156bd0 0x7f64d80b3af0 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f64d815dc60 tx=0x7f64c802a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c403ff2bb4be5ea0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:03.495 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:03.495+0000 7f64d5600640 1 -- 172.21.15.96:0/2355869903 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64c8037070 con 0x7f64d8156bd0 2024-05-28T06:16:03.495 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:03.495+0000 7f64d5600640 1 -- 172.21.15.96:0/2355869903 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64c8032440 con 0x7f64d8156bd0 2024-05-28T06:16:03.495 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:03.495+0000 7f64d5600640 1 -- 172.21.15.96:0/2355869903 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64c8032810 con 0x7f64d8156bd0 2024-05-28T06:16:03.496 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:03.497+0000 7f64dde00640 1 -- 172.21.15.96:0/2355869903 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f64d8156bd0 msgr2=0x7f64d80b3af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:03.496 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:03.497+0000 7f64dde00640 1 --2- 172.21.15.96:0/2355869903 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f64d8156bd0 0x7f64d80b3af0 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f64d815dc60 tx=0x7f64c802a180 comp rx=0 tx=0).stop 2024-05-28T06:16:03.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:03.497+0000 7f64dde00640 1 -- 172.21.15.96:0/2355869903 shutdown_connections 2024-05-28T06:16:03.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:03.497+0000 7f64dde00640 1 --2- 172.21.15.96:0/2355869903 >> v2:172.21.15.123:3300/0 conn(0x7f64d80b4e30 0x7f64d815caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:03.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:03.497+0000 7f64dde00640 1 --2- 172.21.15.96:0/2355869903 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f64d8156bd0 0x7f64d80b3af0 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:03.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:03.497+0000 7f64dde00640 1 -- 172.21.15.96:0/2355869903 >> 172.21.15.96:0/2355869903 conn(0x7f64d806bd20 msgr2=0x7f64d806c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:16:03.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:03.497+0000 7f64dde00640 1 -- 172.21.15.96:0/2355869903 shutdown_connections 2024-05-28T06:16:03.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:03.497+0000 7f64dde00640 1 -- 172.21.15.96:0/2355869903 wait complete. 2024-05-28T06:16:03.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:03.500+0000 7f64dde00640 1 Processor -- start 2024-05-28T06:16:03.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:03.501+0000 7f64dde00640 1 -- start start 2024-05-28T06:16:03.501 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:03.502+0000 7f64dde00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f64d80b4e30 0x7f64d8145ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:03.501 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:03.502+0000 7f64dde00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f64d8156bd0 0x7f64d80cccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:03.501 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:03.502+0000 7f64dde00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64d815d030 0x7f64d8154d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:03.501 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:03.502+0000 7f64dde00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f64d80b4030 con 0x7f64d80b4e30 2024-05-28T06:16:03.501 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:03.502+0000 7f64dde00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f64d80b35d0 con 0x7f64d815d030 2024-05-28T06:16:03.501 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:03.502+0000 7f64dde00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f64d80b41b0 con 0x7f64d8156bd0 2024-05-28T06:16:03.502 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:03.502+0000 7f64d6a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f64d80b4e30 0x7f64d8145ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:03.502 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:03.502+0000 7f64d6a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f64d80b4e30 0x7f64d8145ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44558/0 (socket says 172.21.15.96:44558) 2024-05-28T06:16:03.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:03.502+0000 7f64d6a00640 1 -- 172.21.15.96:0/1801912108 learned_addr learned my addr 172.21.15.96:0/1801912108 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:16:03.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:03.503+0000 7f64d6000640 1 --2- 172.21.15.96:0/1801912108 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f64d8156bd0 0x7f64d80cccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:03.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:03.503+0000 7f64d7400640 1 --2- 172.21.15.96:0/1801912108 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64d815d030 0x7f64d8154d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:03.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:03.503+0000 7f64d6a00640 1 -- 172.21.15.96:0/1801912108 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f64d8156bd0 msgr2=0x7f64d80cccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:03.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:03.503+0000 7f64d6a00640 1 --2- 172.21.15.96:0/1801912108 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f64d8156bd0 0x7f64d80cccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:03.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:03.503+0000 7f64d6a00640 1 -- 172.21.15.96:0/1801912108 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64d815d030 msgr2=0x7f64d8154d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:03.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:03.503+0000 7f64d6a00640 1 --2- 172.21.15.96:0/1801912108 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64d815d030 0x7f64d8154d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:03.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:03.503+0000 7f64d6a00640 1 -- 172.21.15.96:0/1801912108 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64d81539b0 con 0x7f64d80b4e30 2024-05-28T06:16:03.505 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:03.504+0000 7f64d6a00640 1 --2- 172.21.15.96:0/1801912108 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f64d80b4e30 0x7f64d8145ac0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f64c802a1c0 tx=0x7f64c80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:03.505 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:03.504+0000 7f64cf400640 1 -- 172.21.15.96:0/1801912108 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64c8041070 con 0x7f64d80b4e30 2024-05-28T06:16:03.505 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:03.504+0000 7f64cf400640 1 -- 172.21.15.96:0/1801912108 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64c803c870 con 0x7f64d80b4e30 2024-05-28T06:16:03.505 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:03.504+0000 7f64cf400640 1 -- 172.21.15.96:0/1801912108 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64c8037040 con 0x7f64d80b4e30 2024-05-28T06:16:03.505 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:03.504+0000 7f64dde00640 1 -- 172.21.15.96:0/1801912108 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64d8151a70 con 0x7f64d80b4e30 2024-05-28T06:16:03.506 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:03.505+0000 7f64dde00640 1 -- 172.21.15.96:0/1801912108 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64d81572d0 con 0x7f64d80b4e30 2024-05-28T06:16:03.508 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:03.508+0000 7f64cf400640 1 -- 172.21.15.96:0/1801912108 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 26) ==== 92570+0+0 (secure 0 0 0) 0x7f64c8044070 con 0x7f64d80b4e30 2024-05-28T06:16:03.508 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:03.508+0000 7f64dde00640 1 -- 172.21.15.96:0/1801912108 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64d80689e0 con 0x7f64d80b4e30 2024-05-28T06:16:03.509 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:03.510+0000 7f64cf400640 1 --2- 172.21.15.96:0/1801912108 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f64b806da50 0x7f64b806ff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:03.509 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:03.510+0000 7f64cf400640 1 -- 172.21.15.96:0/1801912108 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5795+0+0 (secure 0 0 0) 0x7f64c80ab700 con 0x7f64d80b4e30 2024-05-28T06:16:03.511 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:03.512+0000 7f64d6000640 1 --2- 172.21.15.96:0/1801912108 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f64b806da50 0x7f64b806ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:03.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:03.515+0000 7f64d6000640 1 --2- 172.21.15.96:0/1801912108 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f64b806da50 0x7f64b806ff10 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f64d806c810 tx=0x7f64c4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:03.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:03.516+0000 7f64cf400640 1 -- 172.21.15.96:0/1801912108 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f64c807b3a0 con 0x7f64d80b4e30 2024-05-28T06:16:03.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:03.902+0000 7f64dde00640 1 -- 172.21.15.96:0/1801912108 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone43638", "target": ["mon-mgr", ""]}) -- 0x7f64d815c530 con 0x7f64b806da50 2024-05-28T06:16:04.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:04.631+0000 7f64cf400640 1 -- 172.21.15.96:0/1801912108 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f64d815c530 con 0x7f64b806da50 2024-05-28T06:16:04.638 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:04.638+0000 7f64dde00640 1 -- 172.21.15.96:0/1801912108 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f64b806da50 msgr2=0x7f64b806ff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:04.638 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:04.638+0000 7f64dde00640 1 --2- 172.21.15.96:0/1801912108 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f64b806da50 0x7f64b806ff10 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f64d806c810 tx=0x7f64c4008040 comp rx=0 tx=0).stop 2024-05-28T06:16:04.638 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:04.638+0000 7f64dde00640 1 -- 172.21.15.96:0/1801912108 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f64d80b4e30 msgr2=0x7f64d8145ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:04.638 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:04.638+0000 7f64dde00640 1 --2- 172.21.15.96:0/1801912108 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f64d80b4e30 0x7f64d8145ac0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f64c802a1c0 tx=0x7f64c80046c0 comp rx=0 tx=0).stop 2024-05-28T06:16:04.638 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:04.639+0000 7f64dde00640 1 -- 172.21.15.96:0/1801912108 shutdown_connections 2024-05-28T06:16:04.639 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:04.639+0000 7f64dde00640 1 --2- 172.21.15.96:0/1801912108 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f64b806da50 0x7f64b806ff10 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:04.639 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:04.639+0000 7f64dde00640 1 --2- 172.21.15.96:0/1801912108 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64d815d030 0x7f64d8154d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:04.639 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:04.639+0000 7f64dde00640 1 --2- 172.21.15.96:0/1801912108 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f64d8156bd0 0x7f64d80cccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:04.639 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:04.639+0000 7f64dde00640 1 --2- 172.21.15.96:0/1801912108 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f64d80b4e30 0x7f64d8145ac0 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:04.639 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:04.639+0000 7f64dde00640 1 -- 172.21.15.96:0/1801912108 >> 172.21.15.96:0/1801912108 conn(0x7f64d806bd20 msgr2=0x7f64d815f8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:16:04.639 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:04.640+0000 7f64dde00640 1 -- 172.21.15.96:0/1801912108 shutdown_connections 2024-05-28T06:16:04.640 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:04.640+0000 7f64dde00640 1 -- 172.21.15.96:0/1801912108 wait complete. 2024-05-28T06:16:04.684 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone43639 2024-05-28T06:16:04.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:04.982+0000 7f5a03200640 1 Processor -- start 2024-05-28T06:16:04.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:04.982+0000 7f5a03200640 1 -- start start 2024-05-28T06:16:04.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:04.983+0000 7f5a03200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f59fc0b3d00 0x7f59fc0d0e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:04.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:04.984+0000 7f5a03200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f59fc0d1380 0x7f59fc0d27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:04.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:04.984+0000 7f5a03200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59fc0cfb40 con 0x7f59fc161d00 2024-05-28T06:16:04.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:04.984+0000 7f5a03200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f59fc0d2040 con 0x7f59fc0b3d00 2024-05-28T06:16:04.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:04.984+0000 7f5a03200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f59fc0b4140 con 0x7f59fc0d1380 2024-05-28T06:16:04.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:04.984+0000 7f5a01e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f59fc161d00 0x7f59fc0d4d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:59464/0 (socket says 172.21.15.96:59464) 2024-05-28T06:16:04.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:04.984+0000 7f5a01e00640 1 -- 172.21.15.96:0/4238191960 learned_addr learned my addr 172.21.15.96:0/4238191960 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:16:04.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:04.984+0000 7f5a01400640 1 --2- 172.21.15.96:0/4238191960 >> v2:172.21.15.96:3300/0 conn(0x7f59fc0b3d00 0x7f59fc0d0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:04.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:04.985+0000 7f5a01400640 1 -- 172.21.15.96:0/4238191960 >> v2:172.21.15.123:3300/0 conn(0x7f59fc0d1380 msgr2=0x7f59fc0d27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:16:04.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:04.985+0000 7f5a01400640 1 --2- 172.21.15.96:0/4238191960 >> v2:172.21.15.123:3300/0 conn(0x7f59fc0d1380 0x7f59fc0d27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:04.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:04.985+0000 7f5a01400640 1 -- 172.21.15.96:0/4238191960 >> v1:172.21.15.96:6789/0 conn(0x7f59fc161d00 legacy=0x7f59fc0d4d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:04.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:04.985+0000 7f5a01400640 1 -- 172.21.15.96:0/4238191960 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59fc0cfdc0 con 0x7f59fc0b3d00 2024-05-28T06:16:04.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:04.986+0000 7f5a01400640 1 --2- 172.21.15.96:0/4238191960 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f59fc0b3d00 0x7f59fc0d0e40 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f59fc0b5db0 tx=0x7f59e802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72f0a5e05e73928a server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:04.986 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:04.986+0000 7f5a00a00640 1 -- 172.21.15.96:0/4238191960 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59e8037070 con 0x7f59fc0b3d00 2024-05-28T06:16:04.986 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:04.986+0000 7f5a00a00640 1 -- 172.21.15.96:0/4238191960 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59e8032440 con 0x7f59fc0b3d00 2024-05-28T06:16:04.986 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:04.986+0000 7f5a00a00640 1 -- 172.21.15.96:0/4238191960 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59e8032810 con 0x7f59fc0b3d00 2024-05-28T06:16:04.987 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:04.987+0000 7f5a03200640 1 -- 172.21.15.96:0/4238191960 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f59fc0b3d00 msgr2=0x7f59fc0d0e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:04.987 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:04.987+0000 7f5a03200640 1 --2- 172.21.15.96:0/4238191960 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f59fc0b3d00 0x7f59fc0d0e40 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f59fc0b5db0 tx=0x7f59e802a1a0 comp rx=0 tx=0).stop 2024-05-28T06:16:04.987 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:04.987+0000 7f5a03200640 1 -- 172.21.15.96:0/4238191960 shutdown_connections 2024-05-28T06:16:04.987 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:04.987+0000 7f5a03200640 1 --2- 172.21.15.96:0/4238191960 >> v2:172.21.15.123:3300/0 conn(0x7f59fc0d1380 0x7f59fc0d27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:04.987 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:04.987+0000 7f5a03200640 1 --2- 172.21.15.96:0/4238191960 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f59fc0b3d00 0x7f59fc0d0e40 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:04.987 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:04.987+0000 7f5a03200640 1 -- 172.21.15.96:0/4238191960 >> 172.21.15.96:0/4238191960 conn(0x7f59fc08bfc0 msgr2=0x7f59fc0d4470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:16:04.987 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:04.987+0000 7f5a03200640 1 -- 172.21.15.96:0/4238191960 shutdown_connections 2024-05-28T06:16:04.987 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:04.987+0000 7f5a03200640 1 -- 172.21.15.96:0/4238191960 wait complete. 2024-05-28T06:16:04.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:04.990+0000 7f5a03200640 1 Processor -- start 2024-05-28T06:16:04.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:04.990+0000 7f5a03200640 1 -- start start 2024-05-28T06:16:04.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:04.991+0000 7f5a03200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f59fc0b3d00 0x7f59fc14e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:04.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:04.991+0000 7f5a03200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f59fc0d1380 0x7f59fc1e8280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:04.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:04.991+0000 7f5a03200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f59fc161d00 0x7f59fc0d06c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:04.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:04.991+0000 7f5a03200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f59fc0d2290 con 0x7f59fc161d00 2024-05-28T06:16:04.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:04.991+0000 7f5a03200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f59fc0d0380 con 0x7f59fc0d1380 2024-05-28T06:16:04.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:04.991+0000 7f5a03200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f59fc0b5560 con 0x7f59fc0b3d00 2024-05-28T06:16:04.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:04.992+0000 7f5a02800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f59fc161d00 0x7f59fc0d06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:04.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:04.992+0000 7f5a02800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f59fc161d00 0x7f59fc0d06c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44582/0 (socket says 172.21.15.96:44582) 2024-05-28T06:16:04.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:04.992+0000 7f5a02800640 1 -- 172.21.15.96:0/2418323309 learned_addr learned my addr 172.21.15.96:0/2418323309 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:16:04.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:04.992+0000 7f5a01e00640 1 --2- 172.21.15.96:0/2418323309 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f59fc0b3d00 0x7f59fc14e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:04.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:04.992+0000 7f5a01400640 1 --2- 172.21.15.96:0/2418323309 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f59fc0d1380 0x7f59fc1e8280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:04.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:04.992+0000 7f5a02800640 1 -- 172.21.15.96:0/2418323309 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f59fc0b3d00 msgr2=0x7f59fc14e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:04.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:04.992+0000 7f5a02800640 1 --2- 172.21.15.96:0/2418323309 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f59fc0b3d00 0x7f59fc14e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:04.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:04.992+0000 7f5a02800640 1 -- 172.21.15.96:0/2418323309 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f59fc0d1380 msgr2=0x7f59fc1e8280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:04.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:04.992+0000 7f5a02800640 1 --2- 172.21.15.96:0/2418323309 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f59fc0d1380 0x7f59fc1e8280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:04.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:04.992+0000 7f5a02800640 1 -- 172.21.15.96:0/2418323309 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59fc0cc350 con 0x7f59fc161d00 2024-05-28T06:16:04.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:04.992+0000 7f5a01400640 1 --2- 172.21.15.96:0/2418323309 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f59fc0d1380 0x7f59fc1e8280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-28T06:16:04.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:04.993+0000 7f5a02800640 1 --2- 172.21.15.96:0/2418323309 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f59fc161d00 0x7f59fc0d06c0 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f59f800b970 tx=0x7f59f800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:04.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:04.993+0000 7f59f6000640 1 -- 172.21.15.96:0/2418323309 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59f8019070 con 0x7f59fc161d00 2024-05-28T06:16:04.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:04.993+0000 7f59f6000640 1 -- 172.21.15.96:0/2418323309 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59f800cb60 con 0x7f59fc161d00 2024-05-28T06:16:04.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:04.993+0000 7f59f6000640 1 -- 172.21.15.96:0/2418323309 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59f8014440 con 0x7f59fc161d00 2024-05-28T06:16:04.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:04.993+0000 7f5a03200640 1 -- 172.21.15.96:0/2418323309 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59fc0cc630 con 0x7f59fc161d00 2024-05-28T06:16:04.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:04.993+0000 7f5a03200640 1 -- 172.21.15.96:0/2418323309 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59fc0ccbf0 con 0x7f59fc161d00 2024-05-28T06:16:04.995 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:04.994+0000 7f5a03200640 1 -- 172.21.15.96:0/2418323309 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59fc0689e0 con 0x7f59fc161d00 2024-05-28T06:16:04.995 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:04.996+0000 7f59f6000640 1 -- 172.21.15.96:0/2418323309 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 26) ==== 92570+0+0 (secure 0 0 0) 0x7f59f801c070 con 0x7f59fc161d00 2024-05-28T06:16:04.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:04.997+0000 7f59f6000640 1 --2- 172.21.15.96:0/2418323309 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f59dc06d9d0 0x7f59dc06fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:04.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:04.997+0000 7f59f6000640 1 -- 172.21.15.96:0/2418323309 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5795+0+0 (secure 0 0 0) 0x7f59f808a760 con 0x7f59fc161d00 2024-05-28T06:16:05.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:05.003+0000 7f59f6000640 1 -- 172.21.15.96:0/2418323309 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f59f805a300 con 0x7f59fc161d00 2024-05-28T06:16:05.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:05.006+0000 7f5a01e00640 1 --2- 172.21.15.96:0/2418323309 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f59dc06d9d0 0x7f59dc06fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:05.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:05.008+0000 7f5a01e00640 1 --2- 172.21.15.96:0/2418323309 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f59dc06d9d0 0x7f59dc06fe90 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f59f0005880 tx=0x7f59f0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:05.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:05.396+0000 7f5a03200640 1 -- 172.21.15.96:0/2418323309 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone43639", "target": ["mon-mgr", ""]}) -- 0x7f59fc0ce930 con 0x7f59dc06d9d0 2024-05-28T06:16:06.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.413+0000 7f59f6000640 1 -- 172.21.15.96:0/2418323309 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f59fc0ce930 con 0x7f59dc06d9d0 2024-05-28T06:16:06.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.424+0000 7f5a03200640 1 -- 172.21.15.96:0/2418323309 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f59dc06d9d0 msgr2=0x7f59dc06fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:06.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.424+0000 7f5a03200640 1 --2- 172.21.15.96:0/2418323309 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f59dc06d9d0 0x7f59dc06fe90 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f59f0005880 tx=0x7f59f0009040 comp rx=0 tx=0).stop 2024-05-28T06:16:06.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.424+0000 7f5a03200640 1 -- 172.21.15.96:0/2418323309 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f59fc161d00 msgr2=0x7f59fc0d06c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:06.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.424+0000 7f5a03200640 1 --2- 172.21.15.96:0/2418323309 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f59fc161d00 0x7f59fc0d06c0 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f59f800b970 tx=0x7f59f800be30 comp rx=0 tx=0).stop 2024-05-28T06:16:06.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.424+0000 7f5a03200640 1 -- 172.21.15.96:0/2418323309 shutdown_connections 2024-05-28T06:16:06.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.424+0000 7f5a03200640 1 --2- 172.21.15.96:0/2418323309 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f59dc06d9d0 0x7f59dc06fe90 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:06.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.424+0000 7f5a03200640 1 --2- 172.21.15.96:0/2418323309 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f59fc161d00 0x7f59fc0d06c0 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:06.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.424+0000 7f5a03200640 1 --2- 172.21.15.96:0/2418323309 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f59fc0d1380 0x7f59fc1e8280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:06.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.424+0000 7f5a03200640 1 --2- 172.21.15.96:0/2418323309 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f59fc0b3d00 0x7f59fc14e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:06.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.424+0000 7f5a03200640 1 -- 172.21.15.96:0/2418323309 >> 172.21.15.96:0/2418323309 conn(0x7f59fc08bfc0 msgr2=0x7f59fc162880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:16:06.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.424+0000 7f5a03200640 1 -- 172.21.15.96:0/2418323309 shutdown_connections 2024-05-28T06:16:06.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.424+0000 7f5a03200640 1 -- 172.21.15.96:0/2418323309 wait complete. 2024-05-28T06:16:06.470 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone436310 2024-05-28T06:16:06.760 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.759+0000 7f2080c00640 1 Processor -- start 2024-05-28T06:16:06.760 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.760+0000 7f2080c00640 1 -- start start 2024-05-28T06:16:06.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.763+0000 7f2080c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f207c0c67b0 0x7f207c0cc5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:06.763 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.763+0000 7f2080c00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f207c0cccd0 con 0x7f207c0d00a0 2024-05-28T06:16:06.763 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.763+0000 7f2080c00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f207c0ccfb0 con 0x7f207c15b750 2024-05-28T06:16:06.763 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.763+0000 7f2080c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f207c0cd290 con 0x7f207c0c67b0 2024-05-28T06:16:06.763 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.763+0000 7f2079400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f207c0d00a0 0x7f207c0cbec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:59480/0 (socket says 172.21.15.96:59480) 2024-05-28T06:16:06.763 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.763+0000 7f2079400640 1 -- 172.21.15.96:0/1743977837 learned_addr learned my addr 172.21.15.96:0/1743977837 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:16:06.764 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.764+0000 7f2078a00640 1 -- 172.21.15.96:0/1743977837 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4219077820 0 0) 0x7f207c0cccd0 con 0x7f207c0d00a0 2024-05-28T06:16:06.764 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.764+0000 7f2078a00640 1 -- 172.21.15.96:0/1743977837 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2060003520 con 0x7f207c0d00a0 2024-05-28T06:16:06.764 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.764+0000 7f207a800640 1 --2- 172.21.15.96:0/1743977837 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f207c0c67b0 0x7f207c0cc5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:06.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.765+0000 7f2078a00640 1 -- 172.21.15.96:0/1743977837 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4080962942 0 0) 0x7f2060003520 con 0x7f207c0d00a0 2024-05-28T06:16:06.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.765+0000 7f2078a00640 1 -- 172.21.15.96:0/1743977837 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f207c0c67b0 msgr2=0x7f207c0cc5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:16:06.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.765+0000 7f2078a00640 1 --2- 172.21.15.96:0/1743977837 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f207c0c67b0 0x7f207c0cc5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:06.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.765+0000 7f2078a00640 1 -- 172.21.15.96:0/1743977837 >> v1:172.21.15.123:6789/0 conn(0x7f207c15b750 legacy=0x7f207c0cf990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:06.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.765+0000 7f2078a00640 1 -- 172.21.15.96:0/1743977837 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f207c0c4fa0 con 0x7f207c0d00a0 2024-05-28T06:16:06.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.765+0000 7f2078a00640 1 -- 172.21.15.96:0/1743977837 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f2068001670 con 0x7f207c0d00a0 2024-05-28T06:16:06.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.765+0000 7f2078a00640 1 -- 172.21.15.96:0/1743977837 >> v1:172.21.15.96:6789/0 conn(0x7f207c0d00a0 legacy=0x7f207c0cbec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:06.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.765+0000 7f2078a00640 1 --2- 172.21.15.96:0/1743977837 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f20600039e0 0x7f2060003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:06.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.765+0000 7f2078a00640 1 --2- 172.21.15.96:0/1743977837 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f207c0c67b0 0x7f2060004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:06.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.765+0000 7f2078a00640 1 --2- 172.21.15.96:0/1743977837 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2060004d40 0x7f20600051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:06.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.765+0000 7f2078a00640 1 -- 172.21.15.96:0/1743977837 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f207c0cd290 con 0x7f20600039e0 2024-05-28T06:16:06.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.765+0000 7f2078a00640 1 -- 172.21.15.96:0/1743977837 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f206000ff60 con 0x7f2060004d40 2024-05-28T06:16:06.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.765+0000 7f2078a00640 1 -- 172.21.15.96:0/1743977837 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f2060010160 con 0x7f207c0c67b0 2024-05-28T06:16:06.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.766+0000 7f2079e00640 1 --2- 172.21.15.96:0/1743977837 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f20600039e0 0x7f2060003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:06.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.766+0000 7f2079400640 1 --2- 172.21.15.96:0/1743977837 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f207c0c67b0 0x7f2060004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:06.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.766+0000 7f207a800640 1 --2- 172.21.15.96:0/1743977837 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2060004d40 0x7f20600051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:06.767 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.766+0000 7f2079e00640 1 -- 172.21.15.96:0/1743977837 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f207c0c67b0 msgr2=0x7f2060004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:16:06.767 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.766+0000 7f2079e00640 1 --2- 172.21.15.96:0/1743977837 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f207c0c67b0 0x7f2060004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:06.767 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.766+0000 7f2079e00640 1 -- 172.21.15.96:0/1743977837 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2060004d40 msgr2=0x7f20600051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:16:06.767 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.767+0000 7f2079e00640 1 --2- 172.21.15.96:0/1743977837 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2060004d40 0x7f20600051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:06.767 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.767+0000 7f2079e00640 1 -- 172.21.15.96:0/1743977837 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f207c0c4fa0 con 0x7f20600039e0 2024-05-28T06:16:06.767 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.767+0000 7f207a800640 1 --2- 172.21.15.96:0/1743977837 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2060004d40 0x7f20600051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-28T06:16:06.768 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.768+0000 7f2079e00640 1 --2- 172.21.15.96:0/1743977837 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f20600039e0 0x7f2060003e00 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f206c001e50 tx=0x7f2060005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b08ae6661c18c92 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:06.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.769+0000 7f2078a00640 1 -- 172.21.15.96:0/1743977837 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f206c001a70 con 0x7f20600039e0 2024-05-28T06:16:06.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.770+0000 7f2078a00640 1 -- 172.21.15.96:0/1743977837 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f206c003970 con 0x7f20600039e0 2024-05-28T06:16:06.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.770+0000 7f2078a00640 1 -- 172.21.15.96:0/1743977837 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f206c003d60 con 0x7f20600039e0 2024-05-28T06:16:06.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.772+0000 7f2080c00640 1 -- 172.21.15.96:0/1743977837 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f20600039e0 msgr2=0x7f2060003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:06.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.772+0000 7f2080c00640 1 --2- 172.21.15.96:0/1743977837 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f20600039e0 0x7f2060003e00 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f206c001e50 tx=0x7f2060005830 comp rx=0 tx=0).stop 2024-05-28T06:16:06.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.772+0000 7f207a800640 1 -- 172.21.15.96:0/1743977837 reap_dead start 2024-05-28T06:16:06.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.772+0000 7f2080c00640 1 -- 172.21.15.96:0/1743977837 shutdown_connections 2024-05-28T06:16:06.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.772+0000 7f2080c00640 1 -- 172.21.15.96:0/1743977837 >> 172.21.15.96:0/1743977837 conn(0x7f207c0b2f40 msgr2=0x7f207c0b3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:16:06.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.773+0000 7f2080c00640 1 -- 172.21.15.96:0/1743977837 shutdown_connections 2024-05-28T06:16:06.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.773+0000 7f2080c00640 1 -- 172.21.15.96:0/1743977837 wait complete. 2024-05-28T06:16:06.775 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.776+0000 7f2080c00640 1 Processor -- start 2024-05-28T06:16:06.775 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.776+0000 7f2080c00640 1 -- start start 2024-05-28T06:16:06.777 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.777+0000 7f2080c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f207c1529f0 0x7f207c0ca900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:06.777 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.777+0000 7f2080c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f207c0c8e90 0x7f207c1537f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:06.777 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.778+0000 7f2080c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f207c153d30 0x7f207c154ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:06.777 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.778+0000 7f2080c00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f207c0cc5d0 con 0x7f207c153d30 2024-05-28T06:16:06.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.778+0000 7f2080c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f207c15bb30 con 0x7f207c1529f0 2024-05-28T06:16:06.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.778+0000 7f2080c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f207c0cc750 con 0x7f207c0c8e90 2024-05-28T06:16:06.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.778+0000 7f207a800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f207c153d30 0x7f207c154ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:06.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.778+0000 7f207a800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f207c153d30 0x7f207c154ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44598/0 (socket says 172.21.15.96:44598) 2024-05-28T06:16:06.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.778+0000 7f207a800640 1 -- 172.21.15.96:0/344813233 learned_addr learned my addr 172.21.15.96:0/344813233 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:16:06.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.779+0000 7f2079400640 1 --2- 172.21.15.96:0/344813233 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f207c0c8e90 0x7f207c1537f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:06.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.779+0000 7f2079e00640 1 --2- 172.21.15.96:0/344813233 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f207c1529f0 0x7f207c0ca900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:06.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.780+0000 7f207a800640 1 -- 172.21.15.96:0/344813233 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f207c0c8e90 msgr2=0x7f207c1537f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:06.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.780+0000 7f207a800640 1 --2- 172.21.15.96:0/344813233 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f207c0c8e90 0x7f207c1537f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:06.781 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.780+0000 7f207a800640 1 -- 172.21.15.96:0/344813233 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f207c1529f0 msgr2=0x7f207c0ca900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:06.781 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.780+0000 7f207a800640 1 --2- 172.21.15.96:0/344813233 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f207c1529f0 0x7f207c0ca900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:06.781 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.780+0000 7f207a800640 1 -- 172.21.15.96:0/344813233 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f207c0c9390 con 0x7f207c153d30 2024-05-28T06:16:06.781 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.780+0000 7f207a800640 1 --2- 172.21.15.96:0/344813233 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f207c153d30 0x7f207c154ff0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f207c09f570 tx=0x7f2074004610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:06.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.785+0000 7f2072000640 1 -- 172.21.15.96:0/344813233 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2074018020 con 0x7f207c153d30 2024-05-28T06:16:06.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.785+0000 7f2072000640 1 -- 172.21.15.96:0/344813233 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20740146a0 con 0x7f207c153d30 2024-05-28T06:16:06.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.785+0000 7f2072000640 1 -- 172.21.15.96:0/344813233 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2074014a90 con 0x7f207c153d30 2024-05-28T06:16:06.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.785+0000 7f2080c00640 1 -- 172.21.15.96:0/344813233 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f207c0c9670 con 0x7f207c153d30 2024-05-28T06:16:06.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.785+0000 7f2080c00640 1 -- 172.21.15.96:0/344813233 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f207c0c9c30 con 0x7f207c153d30 2024-05-28T06:16:06.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.787+0000 7f2080c00640 1 -- 172.21.15.96:0/344813233 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2054005290 con 0x7f207c153d30 2024-05-28T06:16:06.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.795+0000 7f2072000640 1 -- 172.21.15.96:0/344813233 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 26) ==== 92570+0+0 (secure 0 0 0) 0x7f207401b020 con 0x7f207c153d30 2024-05-28T06:16:06.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.797+0000 7f2072000640 1 --2- 172.21.15.96:0/344813233 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f205c06da50 0x7f205c06ff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:06.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.797+0000 7f2072000640 1 -- 172.21.15.96:0/344813233 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5795+0+0 (secure 0 0 0) 0x7f207408a680 con 0x7f207c153d30 2024-05-28T06:16:06.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.800+0000 7f2072000640 1 -- 172.21.15.96:0/344813233 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2074022030 con 0x7f207c153d30 2024-05-28T06:16:06.813 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.814+0000 7f2079e00640 1 --2- 172.21.15.96:0/344813233 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f205c06da50 0x7f205c06ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:06.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:06.837+0000 7f2079e00640 1 --2- 172.21.15.96:0/344813233 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f205c06da50 0x7f205c06ff10 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f206c001e50 tx=0x7f206c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:07.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:07.204+0000 7f2080c00640 1 -- 172.21.15.96:0/344813233 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone436310", "target": ["mon-mgr", ""]}) -- 0x7f2054000c80 con 0x7f205c06da50 2024-05-28T06:16:08.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:08.157+0000 7f2072000640 1 -- 172.21.15.96:0/344813233 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f2054000c80 con 0x7f205c06da50 2024-05-28T06:16:08.159 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:08.160+0000 7f2080c00640 1 -- 172.21.15.96:0/344813233 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f205c06da50 msgr2=0x7f205c06ff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:08.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:08.160+0000 7f2080c00640 1 --2- 172.21.15.96:0/344813233 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f205c06da50 0x7f205c06ff10 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f206c001e50 tx=0x7f206c02e000 comp rx=0 tx=0).stop 2024-05-28T06:16:08.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:08.160+0000 7f2080c00640 1 -- 172.21.15.96:0/344813233 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f207c153d30 msgr2=0x7f207c154ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:08.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:08.160+0000 7f2080c00640 1 --2- 172.21.15.96:0/344813233 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f207c153d30 0x7f207c154ff0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f207c09f570 tx=0x7f2074004610 comp rx=0 tx=0).stop 2024-05-28T06:16:08.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:08.160+0000 7f2080c00640 1 -- 172.21.15.96:0/344813233 shutdown_connections 2024-05-28T06:16:08.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:08.160+0000 7f2080c00640 1 --2- 172.21.15.96:0/344813233 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f205c06da50 0x7f205c06ff10 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:08.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:08.160+0000 7f2080c00640 1 --2- 172.21.15.96:0/344813233 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f207c153d30 0x7f207c154ff0 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:08.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:08.160+0000 7f2080c00640 1 --2- 172.21.15.96:0/344813233 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f207c0c8e90 0x7f207c1537f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:08.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:08.160+0000 7f2080c00640 1 --2- 172.21.15.96:0/344813233 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f207c1529f0 0x7f207c0ca900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:08.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:08.160+0000 7f2080c00640 1 -- 172.21.15.96:0/344813233 >> 172.21.15.96:0/344813233 conn(0x7f207c0b2f40 msgr2=0x7f207c0cea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:16:08.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:08.161+0000 7f2080c00640 1 -- 172.21.15.96:0/344813233 shutdown_connections 2024-05-28T06:16:08.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:08.161+0000 7f2080c00640 1 -- 172.21.15.96:0/344813233 wait complete. 2024-05-28T06:16:08.195 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone436311 2024-05-28T06:16:08.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:08.472+0000 7f1504e00640 1 Processor -- start 2024-05-28T06:16:08.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:08.473+0000 7f1504e00640 1 -- start start 2024-05-28T06:16:08.475 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:08.475+0000 7f1504e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f150015e870 0x7f15000d0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:08.475 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:08.475+0000 7f1504e00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f150015ec50 con 0x7f15000cfa40 2024-05-28T06:16:08.475 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:08.475+0000 7f1504e00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15000cff80 con 0x7f15000d14d0 2024-05-28T06:16:08.475 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:08.475+0000 7f1504e00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f15000d2150 con 0x7f150015e870 2024-05-28T06:16:08.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:08.476+0000 7f14ffe00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f15000cfa40 0x7f15000d25e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:59486/0 (socket says 172.21.15.96:59486) 2024-05-28T06:16:08.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:08.477+0000 7f14ffe00640 1 -- 172.21.15.96:0/3257261503 learned_addr learned my addr 172.21.15.96:0/3257261503 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:16:08.478 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:08.478+0000 7f14ff400640 1 --2- 172.21.15.96:0/3257261503 >> v2:172.21.15.123:3300/0 conn(0x7f150015e870 0x7f15000d0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:08.479 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:08.480+0000 7f14ff400640 1 -- 172.21.15.96:0/3257261503 >> v1:172.21.15.123:6789/0 conn(0x7f15000d14d0 legacy=0x7f15000cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:16:08.480 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:08.480+0000 7f14ff400640 1 -- 172.21.15.96:0/3257261503 >> v1:172.21.15.96:6789/0 conn(0x7f15000cfa40 legacy=0x7f15000d25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:08.480 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:08.481+0000 7f14ff400640 1 -- 172.21.15.96:0/3257261503 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15000d0260 con 0x7f150015e870 2024-05-28T06:16:08.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:08.483+0000 7f14ff400640 1 --2- 172.21.15.96:0/3257261503 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f150015e870 0x7f15000d0f90 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f14ec0097a0 tx=0x7f14ec02a1e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=114c68cf19b88c3b server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:08.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:08.486+0000 7f14fe000640 1 -- 172.21.15.96:0/3257261503 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14ec037070 con 0x7f150015e870 2024-05-28T06:16:08.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:08.486+0000 7f14fe000640 1 -- 172.21.15.96:0/3257261503 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14ec032440 con 0x7f150015e870 2024-05-28T06:16:08.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:08.486+0000 7f14fe000640 1 -- 172.21.15.96:0/3257261503 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14ec032810 con 0x7f150015e870 2024-05-28T06:16:08.486 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:08.487+0000 7f1504e00640 1 -- 172.21.15.96:0/3257261503 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f150015e870 msgr2=0x7f15000d0f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:08.486 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:08.487+0000 7f1504e00640 1 --2- 172.21.15.96:0/3257261503 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f150015e870 0x7f15000d0f90 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f14ec0097a0 tx=0x7f14ec02a1e0 comp rx=0 tx=0).stop 2024-05-28T06:16:08.486 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:08.487+0000 7f1504e00640 1 -- 172.21.15.96:0/3257261503 shutdown_connections 2024-05-28T06:16:08.486 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:08.487+0000 7f1504e00640 1 --2- 172.21.15.96:0/3257261503 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f150015e870 0x7f15000d0f90 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:08.486 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:08.487+0000 7f1504e00640 1 -- 172.21.15.96:0/3257261503 >> 172.21.15.96:0/3257261503 conn(0x7f150008bfc0 msgr2=0x7f15000d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:16:08.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:08.488+0000 7f1504e00640 1 -- 172.21.15.96:0/3257261503 shutdown_connections 2024-05-28T06:16:08.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:08.488+0000 7f1504e00640 1 -- 172.21.15.96:0/3257261503 wait complete. 2024-05-28T06:16:08.489 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:08.490+0000 7f1504e00640 1 Processor -- start 2024-05-28T06:16:08.489 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:08.490+0000 7f1504e00640 1 -- start start 2024-05-28T06:16:08.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:08.491+0000 7f1504e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f15000cfa40 0x7f1500159b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:08.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:08.491+0000 7f1504e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f15000d14d0 0x7f15000cb650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:08.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:08.491+0000 7f1504e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f150015e870 0x7f15000cd2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:08.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:08.491+0000 7f1504e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f15000d4fb0 con 0x7f150015e870 2024-05-28T06:16:08.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:08.491+0000 7f1504e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f15000d0820 con 0x7f15000d14d0 2024-05-28T06:16:08.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:08.491+0000 7f1504e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f15000d2cf0 con 0x7f15000cfa40 2024-05-28T06:16:08.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:08.492+0000 7f14ffe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f150015e870 0x7f15000cd2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:08.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:08.492+0000 7f14ffe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f150015e870 0x7f15000cd2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44600/0 (socket says 172.21.15.96:44600) 2024-05-28T06:16:08.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:08.492+0000 7f14ffe00640 1 -- 172.21.15.96:0/2439830582 learned_addr learned my addr 172.21.15.96:0/2439830582 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:16:08.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:08.492+0000 7f14ffe00640 1 -- 172.21.15.96:0/2439830582 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f15000cfa40 msgr2=0x7f1500159b50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-28T06:16:08.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:08.492+0000 7f14ffe00640 1 --2- 172.21.15.96:0/2439830582 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f15000cfa40 0x7f1500159b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:08.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:08.492+0000 7f14ffe00640 1 -- 172.21.15.96:0/2439830582 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f15000d14d0 msgr2=0x7f15000cb650 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-28T06:16:08.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:08.492+0000 7f14ffe00640 1 --2- 172.21.15.96:0/2439830582 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f15000d14d0 0x7f15000cb650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:08.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:08.492+0000 7f14ffe00640 1 -- 172.21.15.96:0/2439830582 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15001e8590 con 0x7f150015e870 2024-05-28T06:16:08.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:08.493+0000 7f14ffe00640 1 --2- 172.21.15.96:0/2439830582 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f150015e870 0x7f15000cd2c0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f14f000c3c0 tx=0x7f14f000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:08.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:08.494+0000 7f14f7400640 1 -- 172.21.15.96:0/2439830582 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14f0017070 con 0x7f150015e870 2024-05-28T06:16:08.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:08.494+0000 7f14f7400640 1 -- 172.21.15.96:0/2439830582 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14f0005ce0 con 0x7f150015e870 2024-05-28T06:16:08.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:08.494+0000 7f14f7400640 1 -- 172.21.15.96:0/2439830582 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14f00072f0 con 0x7f150015e870 2024-05-28T06:16:08.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:08.494+0000 7f1504e00640 1 -- 172.21.15.96:0/2439830582 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f150015f170 con 0x7f150015e870 2024-05-28T06:16:08.494 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:08.494+0000 7f1504e00640 1 -- 172.21.15.96:0/2439830582 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f150015f730 con 0x7f150015e870 2024-05-28T06:16:08.496 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:08.496+0000 7f14f7400640 1 -- 172.21.15.96:0/2439830582 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 26) ==== 92570+0+0 (secure 0 0 0) 0x7f14f001a070 con 0x7f150015e870 2024-05-28T06:16:08.496 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:08.497+0000 7f1504e00640 1 -- 172.21.15.96:0/2439830582 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15000689e0 con 0x7f150015e870 2024-05-28T06:16:08.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:08.498+0000 7f14f7400640 1 --2- 172.21.15.96:0/2439830582 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f14e006daa0 0x7f14e006ff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:08.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:08.498+0000 7f14f7400640 1 -- 172.21.15.96:0/2439830582 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5795+0+0 (secure 0 0 0) 0x7f14f0089de0 con 0x7f150015e870 2024-05-28T06:16:08.506 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:08.507+0000 7f14f7400640 1 -- 172.21.15.96:0/2439830582 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f15001e8590 con 0x7f150015e870 2024-05-28T06:16:08.507 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:08.508+0000 7f14ff400640 1 --2- 172.21.15.96:0/2439830582 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f14e006daa0 0x7f14e006ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:08.514 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:08.515+0000 7f14ff400640 1 --2- 172.21.15.96:0/2439830582 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f14e006daa0 0x7f14e006ff60 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f14ec034000 tx=0x7f14ec02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:08.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:08.909+0000 7f1504e00640 1 -- 172.21.15.96:0/2439830582 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone436311", "target": ["mon-mgr", ""]}) -- 0x7f15000ba500 con 0x7f14e006daa0 2024-05-28T06:16:09.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:09.789+0000 7f14f7400640 1 -- 172.21.15.96:0/2439830582 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f15000ba500 con 0x7f14e006daa0 2024-05-28T06:16:09.794 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:09.793+0000 7f1504e00640 1 -- 172.21.15.96:0/2439830582 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f14e006daa0 msgr2=0x7f14e006ff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:09.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:09.793+0000 7f1504e00640 1 --2- 172.21.15.96:0/2439830582 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f14e006daa0 0x7f14e006ff60 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f14ec034000 tx=0x7f14ec02e000 comp rx=0 tx=0).stop 2024-05-28T06:16:09.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:09.793+0000 7f1504e00640 1 -- 172.21.15.96:0/2439830582 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f150015e870 msgr2=0x7f15000cd2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:09.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:09.793+0000 7f1504e00640 1 --2- 172.21.15.96:0/2439830582 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f150015e870 0x7f15000cd2c0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f14f000c3c0 tx=0x7f14f000c880 comp rx=0 tx=0).stop 2024-05-28T06:16:09.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:09.793+0000 7f1504e00640 1 -- 172.21.15.96:0/2439830582 shutdown_connections 2024-05-28T06:16:09.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:09.793+0000 7f1504e00640 1 --2- 172.21.15.96:0/2439830582 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f14e006daa0 0x7f14e006ff60 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:09.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:09.793+0000 7f1504e00640 1 --2- 172.21.15.96:0/2439830582 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f150015e870 0x7f15000cd2c0 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:09.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:09.793+0000 7f1504e00640 1 --2- 172.21.15.96:0/2439830582 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f15000d14d0 0x7f15000cb650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:09.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:09.793+0000 7f1504e00640 1 --2- 172.21.15.96:0/2439830582 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f15000cfa40 0x7f1500159b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:09.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:09.793+0000 7f1504e00640 1 -- 172.21.15.96:0/2439830582 >> 172.21.15.96:0/2439830582 conn(0x7f150008bfc0 msgr2=0x7f15000b61c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:16:09.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:09.794+0000 7f1504e00640 1 -- 172.21.15.96:0/2439830582 shutdown_connections 2024-05-28T06:16:09.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:09.794+0000 7f1504e00640 1 -- 172.21.15.96:0/2439830582 wait complete. 2024-05-28T06:16:09.826 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone436312 2024-05-28T06:16:10.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:10.095+0000 7f17f3200640 1 Processor -- start 2024-05-28T06:16:10.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:10.095+0000 7f17f3200640 1 -- start start 2024-05-28T06:16:10.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:10.096+0000 7f17f3200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f17f40cfa40 0x7f17f4169690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:10.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:10.096+0000 7f17f3200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17f415ec50 con 0x7f17f415e870 2024-05-28T06:16:10.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:10.096+0000 7f17f3200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17f40cff80 con 0x7f17f40d16a0 2024-05-28T06:16:10.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:10.096+0000 7f17f3200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f17f40d2150 con 0x7f17f40cfa40 2024-05-28T06:16:10.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:10.097+0000 7f17f1e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f17f415e870 0x7f17f40d0f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:57366/0 (socket says 172.21.15.96:57366) 2024-05-28T06:16:10.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:10.097+0000 7f17f1e00640 1 -- 172.21.15.96:0/2368090641 learned_addr learned my addr 172.21.15.96:0/2368090641 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:16:10.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:10.097+0000 7f17f1400640 1 -- 172.21.15.96:0/2368090641 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4133237805 0 0) 0x7f17f415ec50 con 0x7f17f415e870 2024-05-28T06:16:10.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:10.097+0000 7f17f1400640 1 -- 172.21.15.96:0/2368090641 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f17d8003520 con 0x7f17f415e870 2024-05-28T06:16:10.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:10.098+0000 7f17f1400640 1 -- 172.21.15.96:0/2368090641 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3893789610 0 0) 0x7f17d8003520 con 0x7f17f415e870 2024-05-28T06:16:10.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:10.098+0000 7f17f1400640 1 -- 172.21.15.96:0/2368090641 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f17f40cfa40 msgr2=0x7f17f4169690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:16:10.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:10.098+0000 7f17f1400640 1 --2- 172.21.15.96:0/2368090641 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f17f40cfa40 0x7f17f4169690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:10.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:10.098+0000 7f17f1400640 1 -- 172.21.15.96:0/2368090641 >> v1:172.21.15.123:6789/0 conn(0x7f17f40d16a0 legacy=0x7f17f40cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:10.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:10.098+0000 7f17f1400640 1 -- 172.21.15.96:0/2368090641 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17f40d0260 con 0x7f17f415e870 2024-05-28T06:16:10.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:10.098+0000 7f17f1400640 1 -- 172.21.15.96:0/2368090641 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f17e4001670 con 0x7f17f415e870 2024-05-28T06:16:10.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:10.098+0000 7f17f1400640 1 -- 172.21.15.96:0/2368090641 >> v1:172.21.15.96:6789/0 conn(0x7f17f415e870 legacy=0x7f17f40d0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:10.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:10.098+0000 7f17f1400640 1 --2- 172.21.15.96:0/2368090641 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f17d80039e0 0x7f17d8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:10.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:10.098+0000 7f17f1400640 1 --2- 172.21.15.96:0/2368090641 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f17f40cfa40 0x7f17d8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:10.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:10.098+0000 7f17f1400640 1 --2- 172.21.15.96:0/2368090641 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f17d8004e60 0x7f17d800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:10.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:10.098+0000 7f17f1400640 1 -- 172.21.15.96:0/2368090641 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f17f40d2150 con 0x7f17d80039e0 2024-05-28T06:16:10.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:10.098+0000 7f17f1400640 1 -- 172.21.15.96:0/2368090641 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f17d80054e0 con 0x7f17d8004e60 2024-05-28T06:16:10.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:10.098+0000 7f17f1400640 1 -- 172.21.15.96:0/2368090641 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f17d800bf00 con 0x7f17f40cfa40 2024-05-28T06:16:10.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:10.099+0000 7f17f1e00640 1 --2- 172.21.15.96:0/2368090641 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f17d80039e0 0x7f17d8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:10.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:10.099+0000 7f17f1e00640 1 -- 172.21.15.96:0/2368090641 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f17f40cfa40 msgr2=0x7f17d8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:16:10.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:10.099+0000 7f17f1e00640 1 --2- 172.21.15.96:0/2368090641 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f17f40cfa40 0x7f17d8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:10.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:10.099+0000 7f17f1e00640 1 -- 172.21.15.96:0/2368090641 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f17d8004e60 msgr2=0x7f17d800b9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-28T06:16:10.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:10.099+0000 7f17f1e00640 1 --2- 172.21.15.96:0/2368090641 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f17d8004e60 0x7f17d800b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:10.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:10.099+0000 7f17f1e00640 1 -- 172.21.15.96:0/2368090641 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17d8003520 con 0x7f17d80039e0 2024-05-28T06:16:10.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:10.101+0000 7f17f1e00640 1 --2- 172.21.15.96:0/2368090641 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f17d80039e0 0x7f17d8003e00 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f17e400c290 tx=0x7f17d8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e220bf981f872c5 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:10.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:10.101+0000 7f17f1400640 1 -- 172.21.15.96:0/2368090641 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e4007d50 con 0x7f17d80039e0 2024-05-28T06:16:10.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:10.101+0000 7f17f1400640 1 -- 172.21.15.96:0/2368090641 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17e4005ad0 con 0x7f17d80039e0 2024-05-28T06:16:10.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:10.101+0000 7f17f1400640 1 -- 172.21.15.96:0/2368090641 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e4007070 con 0x7f17d80039e0 2024-05-28T06:16:10.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:10.102+0000 7f17f3200640 1 -- 172.21.15.96:0/2368090641 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f17d80039e0 msgr2=0x7f17d8003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:10.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:10.102+0000 7f17f3200640 1 --2- 172.21.15.96:0/2368090641 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f17d80039e0 0x7f17d8003e00 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f17e400c290 tx=0x7f17d8005830 comp rx=0 tx=0).stop 2024-05-28T06:16:10.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:10.102+0000 7f17f2800640 1 -- 172.21.15.96:0/2368090641 reap_dead start 2024-05-28T06:16:10.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:10.102+0000 7f17f3200640 1 -- 172.21.15.96:0/2368090641 shutdown_connections 2024-05-28T06:16:10.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:10.102+0000 7f17f3200640 1 -- 172.21.15.96:0/2368090641 >> 172.21.15.96:0/2368090641 conn(0x7f17f408bfc0 msgr2=0x7f17f40d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:16:10.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:10.102+0000 7f17f3200640 1 -- 172.21.15.96:0/2368090641 shutdown_connections 2024-05-28T06:16:10.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:10.102+0000 7f17f3200640 1 -- 172.21.15.96:0/2368090641 wait complete. 2024-05-28T06:16:10.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:10.104+0000 7f17f3200640 1 Processor -- start 2024-05-28T06:16:10.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:10.104+0000 7f17f3200640 1 -- start start 2024-05-28T06:16:10.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:10.106+0000 7f17f3200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f17f41599a0 0x7f17f40cb650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:10.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:10.106+0000 7f17f3200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f17f40cbb90 0x7f17f40cda20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:10.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:10.106+0000 7f17f3200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f17f4159d80 0x7f17f40cdf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:10.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:10.106+0000 7f17f3200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f17f40d4fb0 con 0x7f17f40cbb90 2024-05-28T06:16:10.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:10.106+0000 7f17f3200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f17f40d0820 con 0x7f17f41599a0 2024-05-28T06:16:10.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:10.106+0000 7f17f3200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f17f40d35f0 con 0x7f17f4159d80 2024-05-28T06:16:10.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:10.106+0000 7f17ebe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f17f40cbb90 0x7f17f40cda20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:10.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:10.106+0000 7f17ebe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f17f40cbb90 0x7f17f40cda20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:46866/0 (socket says 172.21.15.96:46866) 2024-05-28T06:16:10.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:10.106+0000 7f17ebe00640 1 -- 172.21.15.96:0/528744473 learned_addr learned my addr 172.21.15.96:0/528744473 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:16:10.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:10.107+0000 7f17f2800640 1 --2- 172.21.15.96:0/528744473 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f17f4159d80 0x7f17f40cdf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:10.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:10.107+0000 7f17ebe00640 1 -- 172.21.15.96:0/528744473 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f17f4159d80 msgr2=0x7f17f40cdf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:10.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:10.107+0000 7f17ebe00640 1 --2- 172.21.15.96:0/528744473 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f17f4159d80 0x7f17f40cdf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:10.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:10.107+0000 7f17ebe00640 1 -- 172.21.15.96:0/528744473 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f17f41599a0 msgr2=0x7f17f40cb650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:10.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:10.107+0000 7f17ebe00640 1 --2- 172.21.15.96:0/528744473 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f17f41599a0 0x7f17f40cb650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:10.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:10.107+0000 7f17ebe00640 1 -- 172.21.15.96:0/528744473 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17f40cd120 con 0x7f17f40cbb90 2024-05-28T06:16:10.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:10.107+0000 7f17f2800640 1 --2- 172.21.15.96:0/528744473 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f17f4159d80 0x7f17f40cdf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:16:10.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:10.107+0000 7f17ebe00640 1 --2- 172.21.15.96:0/528744473 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f17f40cbb90 0x7f17f40cda20 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f17dc001f00 tx=0x7f17dc00c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:10.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:10.108+0000 7f17ea000640 1 -- 172.21.15.96:0/528744473 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17dc0052d0 con 0x7f17f40cbb90 2024-05-28T06:16:10.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:10.108+0000 7f17f3200640 1 -- 172.21.15.96:0/528744473 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17f40cc080 con 0x7f17f40cbb90 2024-05-28T06:16:10.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:10.108+0000 7f17ea000640 1 -- 172.21.15.96:0/528744473 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17dc005470 con 0x7f17f40cbb90 2024-05-28T06:16:10.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:10.108+0000 7f17ea000640 1 -- 172.21.15.96:0/528744473 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17dc003880 con 0x7f17f40cbb90 2024-05-28T06:16:10.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:10.108+0000 7f17f3200640 1 -- 172.21.15.96:0/528744473 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17f40cc920 con 0x7f17f40cbb90 2024-05-28T06:16:10.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:10.110+0000 7f17f3200640 1 -- 172.21.15.96:0/528744473 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17f40689e0 con 0x7f17f40cbb90 2024-05-28T06:16:10.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:10.111+0000 7f17ea000640 1 -- 172.21.15.96:0/528744473 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 26) ==== 92570+0+0 (secure 0 0 0) 0x7f17dc003a20 con 0x7f17f40cbb90 2024-05-28T06:16:10.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:10.112+0000 7f17ea000640 1 --2- 172.21.15.96:0/528744473 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f17d006d9d0 0x7f17d006fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:10.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:10.112+0000 7f17ea000640 1 -- 172.21.15.96:0/528744473 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5795+0+0 (secure 0 0 0) 0x7f17dc08a660 con 0x7f17f40cbb90 2024-05-28T06:16:10.119 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:10.120+0000 7f17ea000640 1 -- 172.21.15.96:0/528744473 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f17f40cd120 con 0x7f17f40cbb90 2024-05-28T06:16:10.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:10.123+0000 7f17f1e00640 1 --2- 172.21.15.96:0/528744473 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f17d006d9d0 0x7f17d006fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:10.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:10.142+0000 7f17f1e00640 1 --2- 172.21.15.96:0/528744473 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f17d006d9d0 0x7f17d006fe90 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f17e4033000 tx=0x7f17e4009030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:10.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:10.494+0000 7f17f3200640 1 -- 172.21.15.96:0/528744473 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone436312", "target": ["mon-mgr", ""]}) -- 0x7f17f40ba500 con 0x7f17d006d9d0 2024-05-28T06:16:11.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:11.584+0000 7f17ea000640 1 -- 172.21.15.96:0/528744473 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f17f40ba500 con 0x7f17d006d9d0 2024-05-28T06:16:11.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:11.586+0000 7f17f3200640 1 -- 172.21.15.96:0/528744473 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f17d006d9d0 msgr2=0x7f17d006fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:11.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:11.586+0000 7f17f3200640 1 --2- 172.21.15.96:0/528744473 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f17d006d9d0 0x7f17d006fe90 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f17e4033000 tx=0x7f17e4009030 comp rx=0 tx=0).stop 2024-05-28T06:16:11.586 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:11.586+0000 7f17f3200640 1 -- 172.21.15.96:0/528744473 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f17f40cbb90 msgr2=0x7f17f40cda20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:11.586 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:11.586+0000 7f17f3200640 1 --2- 172.21.15.96:0/528744473 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f17f40cbb90 0x7f17f40cda20 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f17dc001f00 tx=0x7f17dc00c960 comp rx=0 tx=0).stop 2024-05-28T06:16:11.586 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:11.586+0000 7f17f3200640 1 -- 172.21.15.96:0/528744473 shutdown_connections 2024-05-28T06:16:11.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:11.586+0000 7f17f3200640 1 --2- 172.21.15.96:0/528744473 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f17d006d9d0 0x7f17d006fe90 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:11.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:11.586+0000 7f17f3200640 1 --2- 172.21.15.96:0/528744473 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f17f4159d80 0x7f17f40cdf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:11.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:11.586+0000 7f17f3200640 1 --2- 172.21.15.96:0/528744473 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f17f40cbb90 0x7f17f40cda20 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:11.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:11.586+0000 7f17f3200640 1 --2- 172.21.15.96:0/528744473 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f17f41599a0 0x7f17f40cb650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:11.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:11.587+0000 7f17f3200640 1 -- 172.21.15.96:0/528744473 >> 172.21.15.96:0/528744473 conn(0x7f17f408bfc0 msgr2=0x7f17f40b60c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:16:11.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:11.587+0000 7f17f3200640 1 -- 172.21.15.96:0/528744473 shutdown_connections 2024-05-28T06:16:11.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:11.587+0000 7f17f3200640 1 -- 172.21.15.96:0/528744473 wait complete. 2024-05-28T06:16:11.613 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone436313 2024-05-28T06:16:11.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:11.840+0000 7f5b2c600640 1 Processor -- start 2024-05-28T06:16:11.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:11.840+0000 7f5b2c600640 1 -- start start 2024-05-28T06:16:11.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:11.842+0000 7f5b2c600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f5b240ce6e0 0x7f5b240d0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:11.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:11.842+0000 7f5b2c600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f5b240d11e0 0x7f5b2409aa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:11.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:11.843+0000 7f5b2c600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b240d16c0 con 0x7f5b2415e940 2024-05-28T06:16:11.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:11.843+0000 7f5b2c600640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f5b240cd240 con 0x7f5b240d11e0 2024-05-28T06:16:11.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:11.843+0000 7f5b2c600640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f5b240ccbf0 con 0x7f5b240ce6e0 2024-05-28T06:16:11.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:11.843+0000 7f5b2a600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f5b240d11e0 0x7f5b2409aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:11.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:11.843+0000 7f5b2a600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f5b240d11e0 0x7f5b2409aa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:46872/0 (socket says 172.21.15.96:46872) 2024-05-28T06:16:11.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:11.843+0000 7f5b2a600640 1 -- 172.21.15.96:0/811575333 learned_addr learned my addr 172.21.15.96:0/811575333 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:16:11.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:11.844+0000 7f5b29200640 1 --2- 172.21.15.96:0/811575333 >> v2:172.21.15.123:3300/0 conn(0x7f5b240ce6e0 0x7f5b240d0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:11.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:11.844+0000 7f5b2a600640 1 -- 172.21.15.96:0/811575333 >> v2:172.21.15.123:3300/0 conn(0x7f5b240ce6e0 msgr2=0x7f5b240d0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:16:11.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:11.844+0000 7f5b2a600640 1 --2- 172.21.15.96:0/811575333 >> v2:172.21.15.123:3300/0 conn(0x7f5b240ce6e0 0x7f5b240d0ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:11.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:11.844+0000 7f5b2a600640 1 -- 172.21.15.96:0/811575333 >> v1:172.21.15.123:6789/0 conn(0x7f5b2415e940 legacy=0x7f5b240d0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:16:11.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:11.844+0000 7f5b2a600640 1 -- 172.21.15.96:0/811575333 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b240cced0 con 0x7f5b240d11e0 2024-05-28T06:16:11.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:11.844+0000 7f5b29200640 1 --2- 172.21.15.96:0/811575333 >> v2:172.21.15.123:3300/0 conn(0x7f5b240ce6e0 0x7f5b240d0ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:16:11.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:11.846+0000 7f5b2a600640 1 --2- 172.21.15.96:0/811575333 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5b240d11e0 0x7f5b2409aa30 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f5b240cd3c0 tx=0x7f5b1c02a6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ec569b9aa2b8174 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:11.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:11.846+0000 7f5b23e00640 1 -- 172.21.15.96:0/811575333 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b1c036070 con 0x7f5b240d11e0 2024-05-28T06:16:11.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:11.846+0000 7f5b23e00640 1 -- 172.21.15.96:0/811575333 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b1c031600 con 0x7f5b240d11e0 2024-05-28T06:16:11.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:11.846+0000 7f5b23e00640 1 -- 172.21.15.96:0/811575333 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b1c0319d0 con 0x7f5b240d11e0 2024-05-28T06:16:11.847 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:11.848+0000 7f5b2c600640 1 -- 172.21.15.96:0/811575333 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5b240d11e0 msgr2=0x7f5b2409aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:11.847 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:11.848+0000 7f5b2c600640 1 --2- 172.21.15.96:0/811575333 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5b240d11e0 0x7f5b2409aa30 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f5b240cd3c0 tx=0x7f5b1c02a6d0 comp rx=0 tx=0).stop 2024-05-28T06:16:11.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:11.848+0000 7f5b2c600640 1 -- 172.21.15.96:0/811575333 shutdown_connections 2024-05-28T06:16:11.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:11.848+0000 7f5b2c600640 1 --2- 172.21.15.96:0/811575333 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5b240d11e0 0x7f5b2409aa30 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:11.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:11.848+0000 7f5b2c600640 1 --2- 172.21.15.96:0/811575333 >> v2:172.21.15.123:3300/0 conn(0x7f5b240ce6e0 0x7f5b240d0ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:11.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:11.848+0000 7f5b2c600640 1 -- 172.21.15.96:0/811575333 >> 172.21.15.96:0/811575333 conn(0x7f5b240b3f60 msgr2=0x7f5b240b4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:16:11.849 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:11.849+0000 7f5b2c600640 1 -- 172.21.15.96:0/811575333 shutdown_connections 2024-05-28T06:16:11.849 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:11.849+0000 7f5b2c600640 1 -- 172.21.15.96:0/811575333 wait complete. 2024-05-28T06:16:11.852 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:11.853+0000 7f5b2c600640 1 Processor -- start 2024-05-28T06:16:11.852 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:11.853+0000 7f5b2c600640 1 -- start start 2024-05-28T06:16:11.854 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:11.855+0000 7f5b2c600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5b240ce6e0 0x7f5b240cbf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:11.854 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:11.855+0000 7f5b2c600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b240d11e0 0x7f5b240c6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:11.855 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:11.855+0000 7f5b2c600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5b2415e940 0x7f5b24149730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:11.855 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:11.855+0000 7f5b2c600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f5b2415ed20 con 0x7f5b2415e940 2024-05-28T06:16:11.855 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:11.855+0000 7f5b2c600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5b2409af70 con 0x7f5b240d11e0 2024-05-28T06:16:11.855 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:11.855+0000 7f5b2c600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f5b241477d0 con 0x7f5b240ce6e0 2024-05-28T06:16:11.855 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:11.855+0000 7f5b2a600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5b2415e940 0x7f5b24149730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:11.855 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:11.855+0000 7f5b2a600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5b2415e940 0x7f5b24149730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:46882/0 (socket says 172.21.15.96:46882) 2024-05-28T06:16:11.855 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:11.855+0000 7f5b2a600640 1 -- 172.21.15.96:0/3051473687 learned_addr learned my addr 172.21.15.96:0/3051473687 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:16:11.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:11.856+0000 7f5b2a600640 1 -- 172.21.15.96:0/3051473687 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5b240ce6e0 msgr2=0x7f5b240cbf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:11.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:11.856+0000 7f5b2a600640 1 --2- 172.21.15.96:0/3051473687 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5b240ce6e0 0x7f5b240cbf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:11.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:11.856+0000 7f5b2a600640 1 -- 172.21.15.96:0/3051473687 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b240d11e0 msgr2=0x7f5b240c6210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:11.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:11.856+0000 7f5b2a600640 1 --2- 172.21.15.96:0/3051473687 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b240d11e0 0x7f5b240c6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:11.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:11.856+0000 7f5b2a600640 1 -- 172.21.15.96:0/3051473687 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b240cc480 con 0x7f5b2415e940 2024-05-28T06:16:11.857 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:11.857+0000 7f5b2a600640 1 --2- 172.21.15.96:0/3051473687 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5b2415e940 0x7f5b24149730 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f5b24072160 tx=0x7f5b1c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:11.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:11.858+0000 7f5b21600640 1 -- 172.21.15.96:0/3051473687 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b1c041020 con 0x7f5b2415e940 2024-05-28T06:16:11.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:11.859+0000 7f5b21600640 1 -- 172.21.15.96:0/3051473687 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b1c008740 con 0x7f5b2415e940 2024-05-28T06:16:11.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:11.859+0000 7f5b21600640 1 -- 172.21.15.96:0/3051473687 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b1c036040 con 0x7f5b2415e940 2024-05-28T06:16:11.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:11.859+0000 7f5b2c600640 1 -- 172.21.15.96:0/3051473687 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b2415afc0 con 0x7f5b2415e940 2024-05-28T06:16:11.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:11.859+0000 7f5b2c600640 1 -- 172.21.15.96:0/3051473687 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b2415b580 con 0x7f5b2415e940 2024-05-28T06:16:11.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:11.862+0000 7f5b2c600640 1 -- 172.21.15.96:0/3051473687 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b2415b8c0 con 0x7f5b2415e940 2024-05-28T06:16:11.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:11.863+0000 7f5b21600640 1 -- 172.21.15.96:0/3051473687 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 26) ==== 92570+0+0 (secure 0 0 0) 0x7f5b1c044020 con 0x7f5b2415e940 2024-05-28T06:16:11.865 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:11.865+0000 7f5b21600640 1 --2- 172.21.15.96:0/3051473687 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5b0406da20 0x7f5b0406fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:11.865 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:11.866+0000 7f5b21600640 1 -- 172.21.15.96:0/3051473687 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5795+0+0 (secure 0 0 0) 0x7f5b1c0ab890 con 0x7f5b2415e940 2024-05-28T06:16:11.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:11.879+0000 7f5b21600640 1 -- 172.21.15.96:0/3051473687 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5b2415b8c0 con 0x7f5b2415e940 2024-05-28T06:16:11.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:11.929+0000 7f5b29c00640 1 --2- 172.21.15.96:0/3051473687 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5b0406da20 0x7f5b0406fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:11.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:11.937+0000 7f5b29c00640 1 --2- 172.21.15.96:0/3051473687 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5b0406da20 0x7f5b0406fee0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f5b10009650 tx=0x7f5b10006ba0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:12.306 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:12.307+0000 7f5b2c600640 1 -- 172.21.15.96:0/3051473687 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone436313", "target": ["mon-mgr", ""]}) -- 0x7f5b240689e0 con 0x7f5b0406da20 2024-05-28T06:16:13.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:13.948+0000 7f5b21600640 1 -- 172.21.15.96:0/3051473687 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f5b240689e0 con 0x7f5b0406da20 2024-05-28T06:16:13.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:13.952+0000 7f5b2c600640 1 -- 172.21.15.96:0/3051473687 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5b0406da20 msgr2=0x7f5b0406fee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:13.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:13.952+0000 7f5b2c600640 1 --2- 172.21.15.96:0/3051473687 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5b0406da20 0x7f5b0406fee0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f5b10009650 tx=0x7f5b10006ba0 comp rx=0 tx=0).stop 2024-05-28T06:16:13.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:13.952+0000 7f5b2c600640 1 -- 172.21.15.96:0/3051473687 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5b2415e940 msgr2=0x7f5b24149730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:13.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:13.952+0000 7f5b2c600640 1 --2- 172.21.15.96:0/3051473687 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5b2415e940 0x7f5b24149730 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f5b24072160 tx=0x7f5b1c004a40 comp rx=0 tx=0).stop 2024-05-28T06:16:13.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:13.952+0000 7f5b2c600640 1 -- 172.21.15.96:0/3051473687 shutdown_connections 2024-05-28T06:16:13.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:13.952+0000 7f5b2c600640 1 --2- 172.21.15.96:0/3051473687 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5b0406da20 0x7f5b0406fee0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:13.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:13.952+0000 7f5b2c600640 1 --2- 172.21.15.96:0/3051473687 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5b2415e940 0x7f5b24149730 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:13.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:13.952+0000 7f5b2c600640 1 --2- 172.21.15.96:0/3051473687 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b240d11e0 0x7f5b240c6210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:13.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:13.952+0000 7f5b2c600640 1 --2- 172.21.15.96:0/3051473687 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5b240ce6e0 0x7f5b240cbf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:13.953 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:13.952+0000 7f5b2c600640 1 -- 172.21.15.96:0/3051473687 >> 172.21.15.96:0/3051473687 conn(0x7f5b240b3f60 msgr2=0x7f5b240b1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:16:13.953 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:13.953+0000 7f5b2c600640 1 -- 172.21.15.96:0/3051473687 shutdown_connections 2024-05-28T06:16:13.953 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:13.953+0000 7f5b2c600640 1 -- 172.21.15.96:0/3051473687 wait complete. 2024-05-28T06:16:13.986 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone436314 2024-05-28T06:16:14.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:14.237+0000 7f6057000640 1 Processor -- start 2024-05-28T06:16:14.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:14.237+0000 7f6057000640 1 -- start start 2024-05-28T06:16:14.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:14.238+0000 7f6057000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f60500c9e00 0x7f6050141f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:14.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:14.238+0000 7f6057000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f605006bc70 0x7f605006c100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:14.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:14.238+0000 7f6057000640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60500cd360 con 0x7f605006cde0 2024-05-28T06:16:14.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:14.238+0000 7f6057000640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f6050163270 con 0x7f605006bc70 2024-05-28T06:16:14.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:14.238+0000 7f6057000640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f605006d270 con 0x7f60500c9e00 2024-05-28T06:16:14.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:14.239+0000 7f6055000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f605006bc70 0x7f605006c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:14.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:14.239+0000 7f6055000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f605006bc70 0x7f605006c100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:46890/0 (socket says 172.21.15.96:46890) 2024-05-28T06:16:14.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:14.239+0000 7f604f400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f605006cde0 0x7f60500ca1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:57376/0 (socket says 172.21.15.96:57376) 2024-05-28T06:16:14.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:14.239+0000 7f6055000640 1 -- 172.21.15.96:0/1590721372 learned_addr learned my addr 172.21.15.96:0/1590721372 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:16:14.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:14.239+0000 7f6055000640 1 -- 172.21.15.96:0/1590721372 >> v2:172.21.15.123:3300/0 conn(0x7f60500c9e00 msgr2=0x7f6050141f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:16:14.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:14.239+0000 7f6055000640 1 --2- 172.21.15.96:0/1590721372 >> v2:172.21.15.123:3300/0 conn(0x7f60500c9e00 0x7f6050141f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:14.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:14.239+0000 7f6055000640 1 -- 172.21.15.96:0/1590721372 >> v1:172.21.15.96:6789/0 conn(0x7f605006cde0 legacy=0x7f60500ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:14.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:14.239+0000 7f6055000640 1 -- 172.21.15.96:0/1590721372 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60501626f0 con 0x7f605006bc70 2024-05-28T06:16:14.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:14.239+0000 7f604ea00640 1 -- 172.21.15.96:0/1590721372 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4268903208 0 0) 0x7f60500cd360 con 0x7f605006cde0 2024-05-28T06:16:14.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:14.240+0000 7f6055000640 1 --2- 172.21.15.96:0/1590721372 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f605006bc70 0x7f605006c100 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f6048009e50 tx=0x7f604802a6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=554908ba069511d8 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:14.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:14.241+0000 7f604ea00640 1 -- 172.21.15.96:0/1590721372 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6048036070 con 0x7f605006bc70 2024-05-28T06:16:14.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:14.241+0000 7f604ea00640 1 -- 172.21.15.96:0/1590721372 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6048031600 con 0x7f605006bc70 2024-05-28T06:16:14.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:14.241+0000 7f604ea00640 1 -- 172.21.15.96:0/1590721372 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60480319d0 con 0x7f605006bc70 2024-05-28T06:16:14.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:14.241+0000 7f6057000640 1 -- 172.21.15.96:0/1590721372 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f605006bc70 msgr2=0x7f605006c100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:14.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:14.241+0000 7f6057000640 1 --2- 172.21.15.96:0/1590721372 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f605006bc70 0x7f605006c100 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f6048009e50 tx=0x7f604802a6d0 comp rx=0 tx=0).stop 2024-05-28T06:16:14.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:14.242+0000 7f6057000640 1 -- 172.21.15.96:0/1590721372 shutdown_connections 2024-05-28T06:16:14.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:14.242+0000 7f6057000640 1 --2- 172.21.15.96:0/1590721372 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f605006bc70 0x7f605006c100 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:14.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:14.242+0000 7f6057000640 1 --2- 172.21.15.96:0/1590721372 >> v2:172.21.15.123:3300/0 conn(0x7f60500c9e00 0x7f6050141f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:14.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:14.242+0000 7f6057000640 1 -- 172.21.15.96:0/1590721372 >> 172.21.15.96:0/1590721372 conn(0x7f60500cd530 msgr2=0x7f60500cd940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:16:14.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:14.242+0000 7f6057000640 1 -- 172.21.15.96:0/1590721372 shutdown_connections 2024-05-28T06:16:14.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:14.242+0000 7f6057000640 1 -- 172.21.15.96:0/1590721372 wait complete. 2024-05-28T06:16:14.243 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:14.244+0000 7f6057000640 1 Processor -- start 2024-05-28T06:16:14.244 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:14.244+0000 7f6057000640 1 -- start start 2024-05-28T06:16:14.245 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:14.245+0000 7f6057000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f605006bc70 0x7f605015df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:14.245 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:14.245+0000 7f6057000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f605006cde0 0x7f60500b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:14.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:14.245+0000 7f6057000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f60500c9e00 0x7f605015ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:14.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:14.245+0000 7f6057000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f6050162f90 con 0x7f605006cde0 2024-05-28T06:16:14.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:14.245+0000 7f6057000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f6050162cb0 con 0x7f605006bc70 2024-05-28T06:16:14.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:14.245+0000 7f6057000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f6050148110 con 0x7f60500c9e00 2024-05-28T06:16:14.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:14.245+0000 7f604f400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f605006cde0 0x7f60500b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:14.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:14.245+0000 7f604f400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f605006cde0 0x7f60500b4d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:46902/0 (socket says 172.21.15.96:46902) 2024-05-28T06:16:14.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:14.245+0000 7f604f400640 1 -- 172.21.15.96:0/360032338 learned_addr learned my addr 172.21.15.96:0/360032338 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:16:14.247 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:14.246+0000 7f604f400640 1 -- 172.21.15.96:0/360032338 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f60500c9e00 msgr2=0x7f605015ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:14.247 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:14.246+0000 7f604f400640 1 --2- 172.21.15.96:0/360032338 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f60500c9e00 0x7f605015ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:14.247 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:14.246+0000 7f604f400640 1 -- 172.21.15.96:0/360032338 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f605006bc70 msgr2=0x7f605015df60 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-28T06:16:14.247 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:14.246+0000 7f604f400640 1 --2- 172.21.15.96:0/360032338 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f605006bc70 0x7f605015df60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:14.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:14.246+0000 7f604f400640 1 -- 172.21.15.96:0/360032338 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60500b56e0 con 0x7f605006cde0 2024-05-28T06:16:14.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:14.246+0000 7f604f400640 1 --2- 172.21.15.96:0/360032338 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f605006cde0 0x7f60500b4d90 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f603c00c4d0 tx=0x7f603c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:14.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:14.247+0000 7f6047e00640 1 -- 172.21.15.96:0/360032338 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f603c018070 con 0x7f605006cde0 2024-05-28T06:16:14.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:14.247+0000 7f6047e00640 1 -- 172.21.15.96:0/360032338 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f603c007500 con 0x7f605006cde0 2024-05-28T06:16:14.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:14.247+0000 7f6047e00640 1 -- 172.21.15.96:0/360032338 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f603c004a20 con 0x7f605006cde0 2024-05-28T06:16:14.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:14.247+0000 7f6057000640 1 -- 172.21.15.96:0/360032338 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60500b59c0 con 0x7f605006cde0 2024-05-28T06:16:14.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:14.247+0000 7f6057000640 1 -- 172.21.15.96:0/360032338 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60500b5f80 con 0x7f605006cde0 2024-05-28T06:16:14.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:14.248+0000 7f6057000640 1 -- 172.21.15.96:0/360032338 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60500689e0 con 0x7f605006cde0 2024-05-28T06:16:14.249 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:14.249+0000 7f6047e00640 1 -- 172.21.15.96:0/360032338 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 26) ==== 92570+0+0 (secure 0 0 0) 0x7f603c004bc0 con 0x7f605006cde0 2024-05-28T06:16:14.250 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:14.250+0000 7f6047e00640 1 --2- 172.21.15.96:0/360032338 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f603006d9d0 0x7f603006fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:14.250 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:14.250+0000 7f6047e00640 1 -- 172.21.15.96:0/360032338 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5795+0+0 (secure 0 0 0) 0x7f603c0899e0 con 0x7f605006cde0 2024-05-28T06:16:14.259 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:14.259+0000 7f6047e00640 1 -- 172.21.15.96:0/360032338 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f603c0598c0 con 0x7f605006cde0 2024-05-28T06:16:14.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:14.310+0000 7f604fe00640 1 --2- 172.21.15.96:0/360032338 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f603006d9d0 0x7f603006fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:14.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:14.329+0000 7f604fe00640 1 --2- 172.21.15.96:0/360032338 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f603006d9d0 0x7f603006fe90 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f60400096b0 tx=0x7f6040006c20 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:14.696 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:14.695+0000 7f6057000640 1 -- 172.21.15.96:0/360032338 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone436314", "target": ["mon-mgr", ""]}) -- 0x7f6050067bb0 con 0x7f603006d9d0 2024-05-28T06:16:15.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:15.370+0000 7f6047e00640 1 -- 172.21.15.96:0/360032338 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f6050067bb0 con 0x7f603006d9d0 2024-05-28T06:16:15.378 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:15.377+0000 7f6057000640 1 -- 172.21.15.96:0/360032338 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f603006d9d0 msgr2=0x7f603006fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:15.378 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:15.377+0000 7f6057000640 1 --2- 172.21.15.96:0/360032338 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f603006d9d0 0x7f603006fe90 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f60400096b0 tx=0x7f6040006c20 comp rx=0 tx=0).stop 2024-05-28T06:16:15.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:15.377+0000 7f6057000640 1 -- 172.21.15.96:0/360032338 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f605006cde0 msgr2=0x7f60500b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:15.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:15.377+0000 7f6057000640 1 --2- 172.21.15.96:0/360032338 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f605006cde0 0x7f60500b4d90 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f603c00c4d0 tx=0x7f603c00c990 comp rx=0 tx=0).stop 2024-05-28T06:16:15.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:15.378+0000 7f6057000640 1 -- 172.21.15.96:0/360032338 shutdown_connections 2024-05-28T06:16:15.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:15.378+0000 7f6057000640 1 --2- 172.21.15.96:0/360032338 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f603006d9d0 0x7f603006fe90 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:15.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:15.378+0000 7f6057000640 1 --2- 172.21.15.96:0/360032338 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f60500c9e00 0x7f605015ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:15.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:15.378+0000 7f6057000640 1 --2- 172.21.15.96:0/360032338 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f605006cde0 0x7f60500b4d90 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:15.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:15.378+0000 7f6057000640 1 --2- 172.21.15.96:0/360032338 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f605006bc70 0x7f605015df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:15.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:15.378+0000 7f6057000640 1 -- 172.21.15.96:0/360032338 >> 172.21.15.96:0/360032338 conn(0x7f60500cd530 msgr2=0x7f60500cd910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:16:15.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:15.378+0000 7f6057000640 1 -- 172.21.15.96:0/360032338 shutdown_connections 2024-05-28T06:16:15.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:15.378+0000 7f6057000640 1 -- 172.21.15.96:0/360032338 wait complete. 2024-05-28T06:16:15.408 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone436315 2024-05-28T06:16:15.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:15.697+0000 7fd067e00640 1 Processor -- start 2024-05-28T06:16:15.699 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:15.697+0000 7fd067e00640 1 -- start start 2024-05-28T06:16:15.699 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:15.699+0000 7fd067e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd060162ac0 0x7fd0600d0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:15.700 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:15.699+0000 7fd067e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fd0600b4120 0x7fd0600d0b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:15.700 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:15.699+0000 7fd067e00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0601640e0 con 0x7fd0601633b0 2024-05-28T06:16:15.700 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:15.699+0000 7fd067e00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fd0600b3c90 con 0x7fd0600b4120 2024-05-28T06:16:15.700 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:15.699+0000 7fd067e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fd0600b3360 con 0x7fd060162ac0 2024-05-28T06:16:15.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:15.701+0000 7fd065400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd060162ac0 0x7fd0600d0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:15.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:15.701+0000 7fd064a00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fd0600b4120 0x7fd0600d0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:15.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:15.701+0000 7fd065e00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7fd0601633b0 0x7fd060163820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.96:36692/0 (socket says 172.21.15.96:36692) 2024-05-28T06:16:15.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:15.701+0000 7fd065e00640 1 -- 172.21.15.96:0/1124708334 learned_addr learned my addr 172.21.15.96:0/1124708334 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:16:15.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:15.702+0000 7fd064a00640 1 -- 172.21.15.96:0/1124708334 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd060162ac0 msgr2=0x7fd0600d0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:16:15.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:15.702+0000 7fd064a00640 1 --2- 172.21.15.96:0/1124708334 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd060162ac0 0x7fd0600d0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:15.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:15.702+0000 7fd064a00640 1 -- 172.21.15.96:0/1124708334 >> v1:172.21.15.123:6789/0 conn(0x7fd0601633b0 legacy=0x7fd060163820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:16:15.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:15.702+0000 7fd064a00640 1 -- 172.21.15.96:0/1124708334 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0600b3640 con 0x7fd0600b4120 2024-05-28T06:16:15.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:15.704+0000 7fd064a00640 1 --2- 172.21.15.96:0/1124708334 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd0600b4120 0x7fd0600d0b80 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fd04c0097a0 tx=0x7fd04c02a130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=79c8b2eb6793ffbd server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:15.705 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:15.705+0000 7fd05fe00640 1 -- 172.21.15.96:0/1124708334 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd04c037070 con 0x7fd0600b4120 2024-05-28T06:16:15.705 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:15.705+0000 7fd05fe00640 1 -- 172.21.15.96:0/1124708334 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd04c02ae40 con 0x7fd0600b4120 2024-05-28T06:16:15.705 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:15.705+0000 7fd05fe00640 1 -- 172.21.15.96:0/1124708334 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd04c032660 con 0x7fd0600b4120 2024-05-28T06:16:15.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:15.706+0000 7fd067e00640 1 -- 172.21.15.96:0/1124708334 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd0600b4120 msgr2=0x7fd0600d0b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:15.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:15.706+0000 7fd067e00640 1 --2- 172.21.15.96:0/1124708334 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd0600b4120 0x7fd0600d0b80 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fd04c0097a0 tx=0x7fd04c02a130 comp rx=0 tx=0).stop 2024-05-28T06:16:15.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:15.706+0000 7fd067e00640 1 -- 172.21.15.96:0/1124708334 shutdown_connections 2024-05-28T06:16:15.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:15.706+0000 7fd067e00640 1 --2- 172.21.15.96:0/1124708334 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd0600b4120 0x7fd0600d0b80 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:15.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:15.706+0000 7fd067e00640 1 --2- 172.21.15.96:0/1124708334 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd060162ac0 0x7fd0600d0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:15.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:15.706+0000 7fd067e00640 1 -- 172.21.15.96:0/1124708334 >> 172.21.15.96:0/1124708334 conn(0x7fd06006bd20 msgr2=0x7fd06006c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:16:15.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:15.706+0000 7fd067e00640 1 -- 172.21.15.96:0/1124708334 shutdown_connections 2024-05-28T06:16:15.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:15.706+0000 7fd067e00640 1 -- 172.21.15.96:0/1124708334 wait complete. 2024-05-28T06:16:15.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:15.709+0000 7fd067e00640 1 Processor -- start 2024-05-28T06:16:15.710 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:15.710+0000 7fd067e00640 1 -- start start 2024-05-28T06:16:15.712 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:15.712+0000 7fd067e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd0600b4120 0x7fd06014b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:15.713 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:15.713+0000 7fd067e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd060162ac0 0x7fd0600cef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:15.713 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:15.713+0000 7fd067e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd0601633b0 0x7fd06015ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:15.714 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:15.714+0000 7fd067e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd060163020 con 0x7fd0600b4120 2024-05-28T06:16:15.715 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:15.714+0000 7fd065e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd0601633b0 0x7fd06015ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:15.715 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:15.715+0000 7fd065e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd0601633b0 0x7fd06015ef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.96:50056/0 (socket says 172.21.15.96:50056) 2024-05-28T06:16:15.715 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:15.715+0000 7fd065400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd0600b4120 0x7fd06014b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:15.716 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:15.715+0000 7fd065400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd0600b4120 0x7fd06014b7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:46918/0 (socket says 172.21.15.96:46918) 2024-05-28T06:16:15.716 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:15.715+0000 7fd065400640 1 -- 172.21.15.96:0/1093143337 learned_addr learned my addr 172.21.15.96:0/1093143337 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:16:15.716 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:15.715+0000 7fd067e00640 1 -- 172.21.15.96:0/1093143337 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd060162ea0 con 0x7fd060162ac0 2024-05-28T06:16:15.716 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:15.715+0000 7fd067e00640 1 -- 172.21.15.96:0/1093143337 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fd06006dbd0 con 0x7fd0601633b0 2024-05-28T06:16:15.716 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:15.716+0000 7fd064a00640 1 --2- 172.21.15.96:0/1093143337 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd060162ac0 0x7fd0600cef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:15.718 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:15.717+0000 7fd065400640 1 -- 172.21.15.96:0/1093143337 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd0601633b0 msgr2=0x7fd06015ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:15.718 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:15.717+0000 7fd065400640 1 --2- 172.21.15.96:0/1093143337 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd0601633b0 0x7fd06015ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:15.718 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:15.717+0000 7fd065400640 1 -- 172.21.15.96:0/1093143337 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd060162ac0 msgr2=0x7fd0600cef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:15.718 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:15.717+0000 7fd065400640 1 --2- 172.21.15.96:0/1093143337 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd060162ac0 0x7fd0600cef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:15.718 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:15.717+0000 7fd065400640 1 -- 172.21.15.96:0/1093143337 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0600cd750 con 0x7fd0600b4120 2024-05-28T06:16:15.719 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:15.719+0000 7fd065400640 1 --2- 172.21.15.96:0/1093143337 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd0600b4120 0x7fd06014b7a0 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7fd05400d800 tx=0x7fd05400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:15.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:15.720+0000 7fd05d600640 1 -- 172.21.15.96:0/1093143337 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd05400b6c0 con 0x7fd0600b4120 2024-05-28T06:16:15.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:15.720+0000 7fd067e00640 1 -- 172.21.15.96:0/1093143337 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0600cda30 con 0x7fd0600b4120 2024-05-28T06:16:15.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:15.720+0000 7fd067e00640 1 -- 172.21.15.96:0/1093143337 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0600cdff0 con 0x7fd0600b4120 2024-05-28T06:16:15.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:15.721+0000 7fd05d600640 1 -- 172.21.15.96:0/1093143337 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd054019070 con 0x7fd0600b4120 2024-05-28T06:16:15.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:15.721+0000 7fd05d600640 1 -- 172.21.15.96:0/1093143337 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd05400ba80 con 0x7fd0600b4120 2024-05-28T06:16:15.727 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:15.726+0000 7fd067e00640 1 -- 172.21.15.96:0/1093143337 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd03c005290 con 0x7fd0600b4120 2024-05-28T06:16:15.728 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:15.728+0000 7fd05d600640 1 -- 172.21.15.96:0/1093143337 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 26) ==== 92570+0+0 (secure 0 0 0) 0x7fd05400bc20 con 0x7fd0600b4120 2024-05-28T06:16:15.730 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:15.729+0000 7fd05d600640 1 --2- 172.21.15.96:0/1093143337 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd04006da20 0x7fd04006fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:15.730 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:15.730+0000 7fd05d600640 1 -- 172.21.15.96:0/1093143337 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5795+0+0 (secure 0 0 0) 0x7fd05408abd0 con 0x7fd0600b4120 2024-05-28T06:16:15.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:15.738+0000 7fd05d600640 1 -- 172.21.15.96:0/1093143337 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd05401c360 con 0x7fd0600b4120 2024-05-28T06:16:15.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:15.750+0000 7fd064a00640 1 --2- 172.21.15.96:0/1093143337 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd04006da20 0x7fd04006fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:15.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:15.775+0000 7fd064a00640 1 --2- 172.21.15.96:0/1093143337 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd04006da20 0x7fd04006fee0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fd04c034000 tx=0x7fd04c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:16.145 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:16.145+0000 7fd067e00640 1 -- 172.21.15.96:0/1093143337 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone436315", "target": ["mon-mgr", ""]}) -- 0x7fd03c000c80 con 0x7fd04006da20 2024-05-28T06:16:17.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.325+0000 7fd05d600640 1 -- 172.21.15.96:0/1093143337 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fd03c000c80 con 0x7fd04006da20 2024-05-28T06:16:17.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.333+0000 7fd067e00640 1 -- 172.21.15.96:0/1093143337 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd04006da20 msgr2=0x7fd04006fee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:17.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.333+0000 7fd067e00640 1 --2- 172.21.15.96:0/1093143337 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd04006da20 0x7fd04006fee0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fd04c034000 tx=0x7fd04c02e000 comp rx=0 tx=0).stop 2024-05-28T06:16:17.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.333+0000 7fd067e00640 1 -- 172.21.15.96:0/1093143337 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd0600b4120 msgr2=0x7fd06014b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:17.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.333+0000 7fd067e00640 1 --2- 172.21.15.96:0/1093143337 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd0600b4120 0x7fd06014b7a0 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7fd05400d800 tx=0x7fd05400dcc0 comp rx=0 tx=0).stop 2024-05-28T06:16:17.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.333+0000 7fd067e00640 1 -- 172.21.15.96:0/1093143337 shutdown_connections 2024-05-28T06:16:17.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.333+0000 7fd067e00640 1 --2- 172.21.15.96:0/1093143337 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd04006da20 0x7fd04006fee0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:17.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.333+0000 7fd067e00640 1 --2- 172.21.15.96:0/1093143337 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd0601633b0 0x7fd06015ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:17.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.333+0000 7fd067e00640 1 --2- 172.21.15.96:0/1093143337 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd060162ac0 0x7fd0600cef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:17.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.333+0000 7fd067e00640 1 --2- 172.21.15.96:0/1093143337 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd0600b4120 0x7fd06014b7a0 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:17.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.333+0000 7fd067e00640 1 -- 172.21.15.96:0/1093143337 >> 172.21.15.96:0/1093143337 conn(0x7fd06006bd20 msgr2=0x7fd060163820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:16:17.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.333+0000 7fd067e00640 1 -- 172.21.15.96:0/1093143337 shutdown_connections 2024-05-28T06:16:17.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.334+0000 7fd067e00640 1 -- 172.21.15.96:0/1093143337 wait complete. 2024-05-28T06:16:17.365 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone436316 2024-05-28T06:16:17.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.621+0000 7fd845400640 1 Processor -- start 2024-05-28T06:16:17.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.621+0000 7fd845400640 1 -- start start 2024-05-28T06:16:17.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.623+0000 7fd845400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd84006d400 0x7fd84009b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:17.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.624+0000 7fd845400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fd84015fa30 0x7fd8400d0600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:17.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.624+0000 7fd845400640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd84006cfb0 con 0x7fd84006ca10 2024-05-28T06:16:17.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.624+0000 7fd845400640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fd84009abd0 con 0x7fd84015fa30 2024-05-28T06:16:17.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.624+0000 7fd845400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fd84009aeb0 con 0x7fd84006d400 2024-05-28T06:16:17.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.624+0000 7fd83f400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fd84006ca10 0x7fd84006d7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:57386/0 (socket says 172.21.15.96:57386) 2024-05-28T06:16:17.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.624+0000 7fd83f400640 1 -- 172.21.15.96:0/3327294927 learned_addr learned my addr 172.21.15.96:0/3327294927 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:16:17.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.624+0000 7fd83fe00640 1 --2- 172.21.15.96:0/3327294927 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd84006d400 0x7fd84009b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:17.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.624+0000 7fd844a00640 1 --2- 172.21.15.96:0/3327294927 >> v2:172.21.15.123:3300/0 conn(0x7fd84015fa30 0x7fd8400d0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:17.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.624+0000 7fd83ea00640 1 -- 172.21.15.96:0/3327294927 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 277920103 0 0) 0x7fd84006cfb0 con 0x7fd84006ca10 2024-05-28T06:16:17.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.624+0000 7fd83ea00640 1 -- 172.21.15.96:0/3327294927 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd824003520 con 0x7fd84006ca10 2024-05-28T06:16:17.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.625+0000 7fd83ea00640 1 -- 172.21.15.96:0/3327294927 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3740895583 0 0) 0x7fd824003520 con 0x7fd84006ca10 2024-05-28T06:16:17.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.625+0000 7fd83ea00640 1 -- 172.21.15.96:0/3327294927 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd84006d400 msgr2=0x7fd84009b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:16:17.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.625+0000 7fd83ea00640 1 --2- 172.21.15.96:0/3327294927 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd84006d400 0x7fd84009b6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:17.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.625+0000 7fd83ea00640 1 -- 172.21.15.96:0/3327294927 >> v2:172.21.15.123:3300/0 conn(0x7fd84015fa30 msgr2=0x7fd8400d0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:16:17.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.625+0000 7fd83ea00640 1 --2- 172.21.15.96:0/3327294927 >> v2:172.21.15.123:3300/0 conn(0x7fd84015fa30 0x7fd8400d0600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:17.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.625+0000 7fd83ea00640 1 -- 172.21.15.96:0/3327294927 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd84009b190 con 0x7fd84006ca10 2024-05-28T06:16:17.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.625+0000 7fd83ea00640 1 -- 172.21.15.96:0/3327294927 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7fd82c001670 con 0x7fd84006ca10 2024-05-28T06:16:17.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.625+0000 7fd83ea00640 1 -- 172.21.15.96:0/3327294927 >> v1:172.21.15.96:6789/0 conn(0x7fd84006ca10 legacy=0x7fd84006d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:17.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.625+0000 7fd83ea00640 1 --2- 172.21.15.96:0/3327294927 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd8240039b0 0x7fd824003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:17.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.625+0000 7fd83ea00640 1 --2- 172.21.15.96:0/3327294927 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd84006d400 0x7fd824004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:17.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.625+0000 7fd83ea00640 1 --2- 172.21.15.96:0/3327294927 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd824004d40 0x7fd8240051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:17.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.625+0000 7fd83ea00640 1 -- 172.21.15.96:0/3327294927 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd84009abd0 con 0x7fd824004d40 2024-05-28T06:16:17.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.625+0000 7fd83ea00640 1 -- 172.21.15.96:0/3327294927 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd84009aeb0 con 0x7fd8240039b0 2024-05-28T06:16:17.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.625+0000 7fd83ea00640 1 -- 172.21.15.96:0/3327294927 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fd82400b9c0 con 0x7fd84006d400 2024-05-28T06:16:17.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.625+0000 7fd844a00640 1 --2- 172.21.15.96:0/3327294927 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd824004d40 0x7fd8240051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:17.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.626+0000 7fd83fe00640 1 --2- 172.21.15.96:0/3327294927 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd8240039b0 0x7fd824003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:17.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.626+0000 7fd844a00640 1 -- 172.21.15.96:0/3327294927 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd84006d400 msgr2=0x7fd824004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:16:17.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.626+0000 7fd844a00640 1 --2- 172.21.15.96:0/3327294927 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd84006d400 0x7fd824004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:17.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.626+0000 7fd844a00640 1 -- 172.21.15.96:0/3327294927 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd8240039b0 msgr2=0x7fd824003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:16:17.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.626+0000 7fd844a00640 1 --2- 172.21.15.96:0/3327294927 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd8240039b0 0x7fd824003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:17.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.626+0000 7fd844a00640 1 -- 172.21.15.96:0/3327294927 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd824003520 con 0x7fd824004d40 2024-05-28T06:16:17.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.627+0000 7fd844a00640 1 --2- 172.21.15.96:0/3327294927 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd824004d40 0x7fd8240051c0 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7fd838009fc0 tx=0x7fd824005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f1c95ee9efd05a7 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:17.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.627+0000 7fd83ea00640 1 -- 172.21.15.96:0/3327294927 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd838009c10 con 0x7fd824004d40 2024-05-28T06:16:17.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.627+0000 7fd83ea00640 1 -- 172.21.15.96:0/3327294927 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd838036070 con 0x7fd824004d40 2024-05-28T06:16:17.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.627+0000 7fd83ea00640 1 -- 172.21.15.96:0/3327294927 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd83802d070 con 0x7fd824004d40 2024-05-28T06:16:17.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.628+0000 7fd845400640 1 -- 172.21.15.96:0/3327294927 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd824004d40 msgr2=0x7fd8240051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:17.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.628+0000 7fd845400640 1 --2- 172.21.15.96:0/3327294927 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd824004d40 0x7fd8240051c0 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7fd838009fc0 tx=0x7fd824005830 comp rx=0 tx=0).stop 2024-05-28T06:16:17.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.628+0000 7fd844a00640 1 -- 172.21.15.96:0/3327294927 reap_dead start 2024-05-28T06:16:17.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.628+0000 7fd845400640 1 -- 172.21.15.96:0/3327294927 shutdown_connections 2024-05-28T06:16:17.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.628+0000 7fd845400640 1 -- 172.21.15.96:0/3327294927 >> 172.21.15.96:0/3327294927 conn(0x7fd8400b3e20 msgr2=0x7fd8400b4230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:16:17.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.628+0000 7fd845400640 1 -- 172.21.15.96:0/3327294927 shutdown_connections 2024-05-28T06:16:17.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.628+0000 7fd845400640 1 -- 172.21.15.96:0/3327294927 wait complete. 2024-05-28T06:16:17.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.630+0000 7fd845400640 1 Processor -- start 2024-05-28T06:16:17.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.630+0000 7fd845400640 1 -- start start 2024-05-28T06:16:17.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.631+0000 7fd845400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd824004d40 0x7fd8400b0990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:17.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.631+0000 7fd845400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd840159a70 0x7fd8401508b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:17.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.631+0000 7fd845400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd8400af0a0 0x7fd8400b1560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:17.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.631+0000 7fd845400640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd84015ee30 con 0x7fd840159a70 2024-05-28T06:16:17.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.631+0000 7fd845400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd84015ecb0 con 0x7fd824004d40 2024-05-28T06:16:17.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.631+0000 7fd845400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fd84009b6b0 con 0x7fd8400af0a0 2024-05-28T06:16:17.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.631+0000 7fd83f400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd840159a70 0x7fd8401508b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:17.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.631+0000 7fd83f400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd840159a70 0x7fd8401508b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:46942/0 (socket says 172.21.15.96:46942) 2024-05-28T06:16:17.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.631+0000 7fd83f400640 1 -- 172.21.15.96:0/1761156986 learned_addr learned my addr 172.21.15.96:0/1761156986 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:16:17.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.631+0000 7fd83fe00640 1 --2- 172.21.15.96:0/1761156986 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd824004d40 0x7fd8400b0990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:17.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.631+0000 7fd844a00640 1 --2- 172.21.15.96:0/1761156986 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd8400af0a0 0x7fd8400b1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:17.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.631+0000 7fd83f400640 1 -- 172.21.15.96:0/1761156986 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd8400af0a0 msgr2=0x7fd8400b1560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:17.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.631+0000 7fd83f400640 1 --2- 172.21.15.96:0/1761156986 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd8400af0a0 0x7fd8400b1560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:17.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.631+0000 7fd83f400640 1 -- 172.21.15.96:0/1761156986 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd824004d40 msgr2=0x7fd8400b0990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:17.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.631+0000 7fd83f400640 1 --2- 172.21.15.96:0/1761156986 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd824004d40 0x7fd8400b0990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:17.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.632+0000 7fd83f400640 1 -- 172.21.15.96:0/1761156986 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8400af8f0 con 0x7fd840159a70 2024-05-28T06:16:17.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.632+0000 7fd83f400640 1 --2- 172.21.15.96:0/1761156986 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd840159a70 0x7fd8401508b0 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7fd82c00c700 tx=0x7fd82c00cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:17.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.632+0000 7fd837e00640 1 -- 172.21.15.96:0/1761156986 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd82c018070 con 0x7fd840159a70 2024-05-28T06:16:17.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.632+0000 7fd837e00640 1 -- 172.21.15.96:0/1761156986 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd82c0074e0 con 0x7fd840159a70 2024-05-28T06:16:17.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.632+0000 7fd837e00640 1 -- 172.21.15.96:0/1761156986 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd82c004a20 con 0x7fd840159a70 2024-05-28T06:16:17.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.632+0000 7fd845400640 1 -- 172.21.15.96:0/1761156986 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8400afbd0 con 0x7fd840159a70 2024-05-28T06:16:17.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.632+0000 7fd845400640 1 -- 172.21.15.96:0/1761156986 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8400b0190 con 0x7fd840159a70 2024-05-28T06:16:17.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.634+0000 7fd845400640 1 -- 172.21.15.96:0/1761156986 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8400690f0 con 0x7fd840159a70 2024-05-28T06:16:17.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.636+0000 7fd837e00640 1 -- 172.21.15.96:0/1761156986 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 26) ==== 92570+0+0 (secure 0 0 0) 0x7fd82c004bc0 con 0x7fd840159a70 2024-05-28T06:16:17.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.637+0000 7fd837e00640 1 --2- 172.21.15.96:0/1761156986 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd82006da00 0x7fd82006fec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:17.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.637+0000 7fd837e00640 1 -- 172.21.15.96:0/1761156986 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5795+0+0 (secure 0 0 0) 0x7fd82c089b40 con 0x7fd840159a70 2024-05-28T06:16:17.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.643+0000 7fd837e00640 1 -- 172.21.15.96:0/1761156986 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd82c059370 con 0x7fd840159a70 2024-05-28T06:16:17.657 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.657+0000 7fd83fe00640 1 --2- 172.21.15.96:0/1761156986 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd82006da00 0x7fd82006fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:17.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:17.692+0000 7fd83fe00640 1 --2- 172.21.15.96:0/1761156986 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd82006da00 0x7fd82006fec0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fd830009ac0 tx=0x7fd830008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:18.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:18.023+0000 7fd845400640 1 -- 172.21.15.96:0/1761156986 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone436316", "target": ["mon-mgr", ""]}) -- 0x7fd8400c8bb0 con 0x7fd82006da00 2024-05-28T06:16:19.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:19.003+0000 7fd837e00640 1 -- 172.21.15.96:0/1761156986 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fd8400c8bb0 con 0x7fd82006da00 2024-05-28T06:16:19.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:19.006+0000 7fd845400640 1 -- 172.21.15.96:0/1761156986 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd82006da00 msgr2=0x7fd82006fec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:19.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:19.006+0000 7fd845400640 1 --2- 172.21.15.96:0/1761156986 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd82006da00 0x7fd82006fec0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fd830009ac0 tx=0x7fd830008040 comp rx=0 tx=0).stop 2024-05-28T06:16:19.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:19.006+0000 7fd845400640 1 -- 172.21.15.96:0/1761156986 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd840159a70 msgr2=0x7fd8401508b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:19.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:19.006+0000 7fd845400640 1 --2- 172.21.15.96:0/1761156986 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd840159a70 0x7fd8401508b0 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7fd82c00c700 tx=0x7fd82c00cbc0 comp rx=0 tx=0).stop 2024-05-28T06:16:19.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:19.006+0000 7fd845400640 1 -- 172.21.15.96:0/1761156986 shutdown_connections 2024-05-28T06:16:19.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:19.006+0000 7fd845400640 1 --2- 172.21.15.96:0/1761156986 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd82006da00 0x7fd82006fec0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:19.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:19.006+0000 7fd845400640 1 --2- 172.21.15.96:0/1761156986 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd8400af0a0 0x7fd8400b1560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:19.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:19.006+0000 7fd845400640 1 --2- 172.21.15.96:0/1761156986 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd840159a70 0x7fd8401508b0 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:19.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:19.006+0000 7fd845400640 1 --2- 172.21.15.96:0/1761156986 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd824004d40 0x7fd8400b0990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:19.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:19.006+0000 7fd845400640 1 -- 172.21.15.96:0/1761156986 >> 172.21.15.96:0/1761156986 conn(0x7fd8400b3e20 msgr2=0x7fd8401602f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:16:19.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:19.006+0000 7fd845400640 1 -- 172.21.15.96:0/1761156986 shutdown_connections 2024-05-28T06:16:19.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:19.006+0000 7fd845400640 1 -- 172.21.15.96:0/1761156986 wait complete. 2024-05-28T06:16:19.034 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone436317 2024-05-28T06:16:19.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:19.311+0000 7f9ff7200640 1 Processor -- start 2024-05-28T06:16:19.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:19.311+0000 7f9ff7200640 1 -- start start 2024-05-28T06:16:19.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:19.312+0000 7f9ff7200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f9ff80d16a0 0x7f9ff80cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:19.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:19.312+0000 7f9ff7200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ff80cfce0 con 0x7f9ff80cf870 2024-05-28T06:16:19.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:19.312+0000 7f9ff7200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ff815ec50 con 0x7f9ff815e870 2024-05-28T06:16:19.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:19.312+0000 7f9ff7200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f9ff80d2150 con 0x7f9ff80d16a0 2024-05-28T06:16:19.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:19.313+0000 7f9ff5400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f9ff80d16a0 0x7f9ff80cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:19.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:19.313+0000 7f9ff6800640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f9ff80cf870 0x7f9ff80d25e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:37148/0 (socket says 172.21.15.96:37148) 2024-05-28T06:16:19.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:19.313+0000 7f9ff6800640 1 -- 172.21.15.96:0/567961628 learned_addr learned my addr 172.21.15.96:0/567961628 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:16:19.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:19.314+0000 7f9ff4a00640 1 -- 172.21.15.96:0/567961628 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4232259952 0 0) 0x7f9ff80cfce0 con 0x7f9ff80cf870 2024-05-28T06:16:19.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:19.315+0000 7f9ff4a00640 1 -- 172.21.15.96:0/567961628 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9fdc003520 con 0x7f9ff80cf870 2024-05-28T06:16:19.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:19.315+0000 7f9ff5400640 1 -- 172.21.15.96:0/567961628 >> v1:172.21.15.123:6789/0 conn(0x7f9ff815e870 legacy=0x7f9ff80d0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:19.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:19.315+0000 7f9ff5400640 1 -- 172.21.15.96:0/567961628 >> v1:172.21.15.96:6789/0 conn(0x7f9ff80cf870 legacy=0x7f9ff80d25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:19.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:19.315+0000 7f9ff5400640 1 -- 172.21.15.96:0/567961628 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ff80cff80 con 0x7f9ff80d16a0 2024-05-28T06:16:19.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:19.317+0000 7f9ff5400640 1 --2- 172.21.15.96:0/567961628 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ff80d16a0 0x7f9ff80cf330 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f9fe4009790 tx=0x7f9fe402a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec90c617f73b4fbb server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:19.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:19.317+0000 7f9ff4a00640 1 -- 172.21.15.96:0/567961628 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fe4037070 con 0x7f9ff80d16a0 2024-05-28T06:16:19.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:19.317+0000 7f9ff4a00640 1 -- 172.21.15.96:0/567961628 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9fe4032440 con 0x7f9ff80d16a0 2024-05-28T06:16:19.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:19.317+0000 7f9ff4a00640 1 -- 172.21.15.96:0/567961628 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fe4032810 con 0x7f9ff80d16a0 2024-05-28T06:16:19.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:19.318+0000 7f9ff7200640 1 -- 172.21.15.96:0/567961628 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ff80d16a0 msgr2=0x7f9ff80cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:19.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:19.318+0000 7f9ff7200640 1 --2- 172.21.15.96:0/567961628 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ff80d16a0 0x7f9ff80cf330 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f9fe4009790 tx=0x7f9fe402a210 comp rx=0 tx=0).stop 2024-05-28T06:16:19.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:19.318+0000 7f9ff7200640 1 -- 172.21.15.96:0/567961628 shutdown_connections 2024-05-28T06:16:19.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:19.318+0000 7f9ff7200640 1 --2- 172.21.15.96:0/567961628 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ff80d16a0 0x7f9ff80cf330 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:19.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:19.318+0000 7f9ff7200640 1 -- 172.21.15.96:0/567961628 >> 172.21.15.96:0/567961628 conn(0x7f9ff808bfc0 msgr2=0x7f9ff80d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:16:19.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:19.318+0000 7f9ff7200640 1 -- 172.21.15.96:0/567961628 shutdown_connections 2024-05-28T06:16:19.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:19.318+0000 7f9ff7200640 1 -- 172.21.15.96:0/567961628 wait complete. 2024-05-28T06:16:19.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:19.320+0000 7f9ff7200640 1 Processor -- start 2024-05-28T06:16:19.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:19.320+0000 7f9ff7200640 1 -- start start 2024-05-28T06:16:19.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:19.321+0000 7f9ff7200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ff80cf870 0x7f9ff80c7880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:19.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:19.321+0000 7f9ff7200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9ff80d16a0 0x7f9ff81f38d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:19.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:19.321+0000 7f9ff7200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ff815e870 0x7f9ff80cd160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:19.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:19.321+0000 7f9ff7200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f9ff80d0820 con 0x7f9ff815e870 2024-05-28T06:16:19.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:19.321+0000 7f9ff7200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f9ff80d0540 con 0x7f9ff80cf870 2024-05-28T06:16:19.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:19.321+0000 7f9ff7200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f9ff80d4fb0 con 0x7f9ff80d16a0 2024-05-28T06:16:19.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:19.322+0000 7f9ff6800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ff815e870 0x7f9ff80cd160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:19.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:19.322+0000 7f9ff6800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ff815e870 0x7f9ff80cd160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49322/0 (socket says 172.21.15.96:49322) 2024-05-28T06:16:19.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:19.322+0000 7f9ff6800640 1 -- 172.21.15.96:0/2927669581 learned_addr learned my addr 172.21.15.96:0/2927669581 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:16:19.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:19.322+0000 7f9ff5e00640 1 --2- 172.21.15.96:0/2927669581 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ff80cf870 0x7f9ff80c7880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:19.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:19.322+0000 7f9ff5400640 1 --2- 172.21.15.96:0/2927669581 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9ff80d16a0 0x7f9ff81f38d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:19.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:19.323+0000 7f9ff6800640 1 -- 172.21.15.96:0/2927669581 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9ff80d16a0 msgr2=0x7f9ff81f38d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:19.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:19.323+0000 7f9ff6800640 1 --2- 172.21.15.96:0/2927669581 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9ff80d16a0 0x7f9ff81f38d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:19.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:19.323+0000 7f9ff6800640 1 -- 172.21.15.96:0/2927669581 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ff80cf870 msgr2=0x7f9ff80c7880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:19.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:19.323+0000 7f9ff6800640 1 --2- 172.21.15.96:0/2927669581 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ff80cf870 0x7f9ff80c7880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:19.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:19.323+0000 7f9ff6800640 1 -- 172.21.15.96:0/2927669581 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ff815fb80 con 0x7f9ff815e870 2024-05-28T06:16:19.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:19.323+0000 7f9ff6800640 1 --2- 172.21.15.96:0/2927669581 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ff815e870 0x7f9ff80cd160 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f9ff000cb70 tx=0x7f9ff000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:19.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:19.323+0000 7f9fee000640 1 -- 172.21.15.96:0/2927669581 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ff0017070 con 0x7f9ff815e870 2024-05-28T06:16:19.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:19.324+0000 7f9fee000640 1 -- 172.21.15.96:0/2927669581 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ff000ad40 con 0x7f9ff815e870 2024-05-28T06:16:19.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:19.324+0000 7f9fee000640 1 -- 172.21.15.96:0/2927669581 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ff0012620 con 0x7f9ff815e870 2024-05-28T06:16:19.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:19.324+0000 7f9ff7200640 1 -- 172.21.15.96:0/2927669581 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ff815f2e0 con 0x7f9ff815e870 2024-05-28T06:16:19.325 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:19.324+0000 7f9ff7200640 1 -- 172.21.15.96:0/2927669581 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ff815f8a0 con 0x7f9ff815e870 2024-05-28T06:16:19.325 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:19.325+0000 7f9ff7200640 1 -- 172.21.15.96:0/2927669581 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ff80689e0 con 0x7f9ff815e870 2024-05-28T06:16:19.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:19.326+0000 7f9fee000640 1 -- 172.21.15.96:0/2927669581 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 26) ==== 92570+0+0 (secure 0 0 0) 0x7f9ff001a070 con 0x7f9ff815e870 2024-05-28T06:16:19.327 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:19.327+0000 7f9fee000640 1 --2- 172.21.15.96:0/2927669581 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9fd806d9d0 0x7f9fd806fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:19.327 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:19.328+0000 7f9fee000640 1 -- 172.21.15.96:0/2927669581 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5795+0+0 (secure 0 0 0) 0x7f9ff008a7b0 con 0x7f9ff815e870 2024-05-28T06:16:19.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:19.332+0000 7f9fee000640 1 -- 172.21.15.96:0/2927669581 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9ff0059ca0 con 0x7f9ff815e870 2024-05-28T06:16:19.337 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:19.337+0000 7f9ff5e00640 1 --2- 172.21.15.96:0/2927669581 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9fd806d9d0 0x7f9fd806fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:19.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:19.339+0000 7f9ff5e00640 1 --2- 172.21.15.96:0/2927669581 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9fd806d9d0 0x7f9fd806fe90 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f9fe8000f60 tx=0x7f9fe8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:19.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:19.706+0000 7f9ff7200640 1 -- 172.21.15.96:0/2927669581 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone436317", "target": ["mon-mgr", ""]}) -- 0x7f9ff80b2340 con 0x7f9fd806d9d0 2024-05-28T06:16:20.261 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.261+0000 7f9fee000640 1 -- 172.21.15.96:0/2927669581 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f9ff80b2340 con 0x7f9fd806d9d0 2024-05-28T06:16:20.268 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.268+0000 7f9ff7200640 1 -- 172.21.15.96:0/2927669581 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9fd806d9d0 msgr2=0x7f9fd806fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:20.268 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.268+0000 7f9ff7200640 1 --2- 172.21.15.96:0/2927669581 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9fd806d9d0 0x7f9fd806fe90 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f9fe8000f60 tx=0x7f9fe8009040 comp rx=0 tx=0).stop 2024-05-28T06:16:20.268 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.268+0000 7f9ff7200640 1 -- 172.21.15.96:0/2927669581 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ff815e870 msgr2=0x7f9ff80cd160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:20.268 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.268+0000 7f9ff7200640 1 --2- 172.21.15.96:0/2927669581 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ff815e870 0x7f9ff80cd160 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f9ff000cb70 tx=0x7f9ff000a430 comp rx=0 tx=0).stop 2024-05-28T06:16:20.268 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.268+0000 7f9ff7200640 1 -- 172.21.15.96:0/2927669581 shutdown_connections 2024-05-28T06:16:20.269 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.268+0000 7f9ff7200640 1 --2- 172.21.15.96:0/2927669581 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9fd806d9d0 0x7f9fd806fe90 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:20.269 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.268+0000 7f9ff7200640 1 --2- 172.21.15.96:0/2927669581 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ff815e870 0x7f9ff80cd160 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:20.269 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.268+0000 7f9ff7200640 1 --2- 172.21.15.96:0/2927669581 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9ff80d16a0 0x7f9ff81f38d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:20.269 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.268+0000 7f9ff7200640 1 --2- 172.21.15.96:0/2927669581 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ff80cf870 0x7f9ff80c7880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:20.269 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.268+0000 7f9ff7200640 1 -- 172.21.15.96:0/2927669581 >> 172.21.15.96:0/2927669581 conn(0x7f9ff808bfc0 msgr2=0x7f9ff80b10f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:16:20.269 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.268+0000 7f9ff7200640 1 -- 172.21.15.96:0/2927669581 shutdown_connections 2024-05-28T06:16:20.269 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.268+0000 7f9ff7200640 1 -- 172.21.15.96:0/2927669581 wait complete. 2024-05-28T06:16:20.296 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone436318 2024-05-28T06:16:20.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.549+0000 7f7d17200640 1 Processor -- start 2024-05-28T06:16:20.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.549+0000 7f7d17200640 1 -- start start 2024-05-28T06:16:20.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.554+0000 7f7d17200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7d18163d90 0x7f7d180ce760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:20.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.554+0000 7f7d17200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f7d180ceee0 0x7f7d180cf370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:20.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.554+0000 7f7d17200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d181670c0 con 0x7f7d181639b0 2024-05-28T06:16:20.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.554+0000 7f7d17200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f7d180ceca0 con 0x7f7d180ceee0 2024-05-28T06:16:20.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.554+0000 7f7d17200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f7d181673a0 con 0x7f7d18163d90 2024-05-28T06:16:20.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.554+0000 7f7d15e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f7d181639b0 0x7f7d18166670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:37152/0 (socket says 172.21.15.96:37152) 2024-05-28T06:16:20.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.554+0000 7f7d15e00640 1 -- 172.21.15.96:0/296370214 learned_addr learned my addr 172.21.15.96:0/296370214 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:16:20.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.555+0000 7f7d14a00640 1 -- 172.21.15.96:0/296370214 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1537289827 0 0) 0x7f7d181670c0 con 0x7f7d181639b0 2024-05-28T06:16:20.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.555+0000 7f7d14a00640 1 -- 172.21.15.96:0/296370214 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7cfc003520 con 0x7f7d181639b0 2024-05-28T06:16:20.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.555+0000 7f7d16800640 1 --2- 172.21.15.96:0/296370214 >> v2:172.21.15.123:3300/0 conn(0x7f7d180ceee0 0x7f7d180cf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:20.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.555+0000 7f7d15400640 1 --2- 172.21.15.96:0/296370214 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7d18163d90 0x7f7d180ce760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:20.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.555+0000 7f7d14a00640 1 -- 172.21.15.96:0/296370214 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2177014209 0 0) 0x7f7cfc003520 con 0x7f7d181639b0 2024-05-28T06:16:20.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.555+0000 7f7d14a00640 1 -- 172.21.15.96:0/296370214 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7d18163d90 msgr2=0x7f7d180ce760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:16:20.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.555+0000 7f7d14a00640 1 --2- 172.21.15.96:0/296370214 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7d18163d90 0x7f7d180ce760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:20.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.555+0000 7f7d14a00640 1 -- 172.21.15.96:0/296370214 >> v2:172.21.15.123:3300/0 conn(0x7f7d180ceee0 msgr2=0x7f7d180cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:16:20.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.555+0000 7f7d14a00640 1 --2- 172.21.15.96:0/296370214 >> v2:172.21.15.123:3300/0 conn(0x7f7d180ceee0 0x7f7d180cf370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:20.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.555+0000 7f7d14a00640 1 -- 172.21.15.96:0/296370214 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d180cdf60 con 0x7f7d181639b0 2024-05-28T06:16:20.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.555+0000 7f7d14a00640 1 -- 172.21.15.96:0/296370214 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f7d08001670 con 0x7f7d181639b0 2024-05-28T06:16:20.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.556+0000 7f7d14a00640 1 -- 172.21.15.96:0/296370214 >> v1:172.21.15.96:6789/0 conn(0x7f7d181639b0 legacy=0x7f7d18166670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:20.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.556+0000 7f7d14a00640 1 --2- 172.21.15.96:0/296370214 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7cfc0039b0 0x7f7cfc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:20.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.556+0000 7f7d14a00640 1 --2- 172.21.15.96:0/296370214 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7cfc004800 0x7f7cfc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:20.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.556+0000 7f7d14a00640 1 --2- 172.21.15.96:0/296370214 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7cfc0052e0 0x7f7cfc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:20.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.556+0000 7f7d14a00640 1 -- 172.21.15.96:0/296370214 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f7d180ceca0 con 0x7f7cfc0052e0 2024-05-28T06:16:20.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.556+0000 7f7d14a00640 1 -- 172.21.15.96:0/296370214 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f7d181673a0 con 0x7f7cfc004800 2024-05-28T06:16:20.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.556+0000 7f7d14a00640 1 -- 172.21.15.96:0/296370214 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f7cfc00c210 con 0x7f7cfc0039b0 2024-05-28T06:16:20.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.556+0000 7f7d16800640 1 --2- 172.21.15.96:0/296370214 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7cfc0052e0 0x7f7cfc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:20.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.557+0000 7f7d15e00640 1 --2- 172.21.15.96:0/296370214 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7cfc0039b0 0x7f7cfc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:20.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.557+0000 7f7d15400640 1 --2- 172.21.15.96:0/296370214 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7cfc004800 0x7f7cfc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:20.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.557+0000 7f7d16800640 1 -- 172.21.15.96:0/296370214 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7cfc0039b0 msgr2=0x7f7cfc003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:16:20.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.558+0000 7f7d16800640 1 --2- 172.21.15.96:0/296370214 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7cfc0039b0 0x7f7cfc003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:20.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.558+0000 7f7d16800640 1 -- 172.21.15.96:0/296370214 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7cfc004800 msgr2=0x7f7cfc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:16:20.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.558+0000 7f7d16800640 1 --2- 172.21.15.96:0/296370214 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7cfc004800 0x7f7cfc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:20.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.558+0000 7f7d16800640 1 -- 172.21.15.96:0/296370214 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cfc003520 con 0x7f7cfc0052e0 2024-05-28T06:16:20.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.558+0000 7f7d15400640 1 --2- 172.21.15.96:0/296370214 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7cfc004800 0x7f7cfc004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-28T06:16:20.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.559+0000 7f7d16800640 1 --2- 172.21.15.96:0/296370214 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7cfc0052e0 0x7f7cfc00b9c0 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f7d10007f80 tx=0x7f7cfc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99eaf831baefbde1 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:20.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.559+0000 7f7d14a00640 1 -- 172.21.15.96:0/296370214 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d10007b50 con 0x7f7cfc0052e0 2024-05-28T06:16:20.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.560+0000 7f7d14a00640 1 -- 172.21.15.96:0/296370214 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d10003b90 con 0x7f7cfc0052e0 2024-05-28T06:16:20.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.560+0000 7f7d14a00640 1 -- 172.21.15.96:0/296370214 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d10004020 con 0x7f7cfc0052e0 2024-05-28T06:16:20.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.560+0000 7f7d17200640 1 -- 172.21.15.96:0/296370214 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7cfc0052e0 msgr2=0x7f7cfc00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:20.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.560+0000 7f7d17200640 1 --2- 172.21.15.96:0/296370214 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7cfc0052e0 0x7f7cfc00b9c0 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f7d10007f80 tx=0x7f7cfc005830 comp rx=0 tx=0).stop 2024-05-28T06:16:20.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.561+0000 7f7d16800640 1 -- 172.21.15.96:0/296370214 reap_dead start 2024-05-28T06:16:20.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.561+0000 7f7d17200640 1 -- 172.21.15.96:0/296370214 shutdown_connections 2024-05-28T06:16:20.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.561+0000 7f7d17200640 1 -- 172.21.15.96:0/296370214 >> 172.21.15.96:0/296370214 conn(0x7f7d1806b900 msgr2=0x7f7d1806bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:16:20.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.561+0000 7f7d17200640 1 -- 172.21.15.96:0/296370214 shutdown_connections 2024-05-28T06:16:20.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.561+0000 7f7d17200640 1 -- 172.21.15.96:0/296370214 wait complete. 2024-05-28T06:16:20.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.563+0000 7f7d17200640 1 Processor -- start 2024-05-28T06:16:20.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.563+0000 7f7d17200640 1 -- start start 2024-05-28T06:16:20.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.564+0000 7f7d17200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7cfc0052e0 0x7f7d1814e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:20.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.564+0000 7f7d17200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7d181596b0 0x7f7d180afd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:20.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.564+0000 7f7d17200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d180b1510 0x7f7d181619e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:20.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.564+0000 7f7d17200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f7d180afa70 con 0x7f7cfc0052e0 2024-05-28T06:16:20.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.564+0000 7f7d17200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f7d18166d80 con 0x7f7d180b1510 2024-05-28T06:16:20.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.564+0000 7f7d17200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f7d180afbf0 con 0x7f7d181596b0 2024-05-28T06:16:20.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.564+0000 7f7d15e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7cfc0052e0 0x7f7d1814e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:20.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.564+0000 7f7d15e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7cfc0052e0 0x7f7d1814e5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49338/0 (socket says 172.21.15.96:49338) 2024-05-28T06:16:20.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.565+0000 7f7d15e00640 1 -- 172.21.15.96:0/511172079 learned_addr learned my addr 172.21.15.96:0/511172079 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:16:20.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.565+0000 7f7d15e00640 1 -- 172.21.15.96:0/511172079 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7d181596b0 msgr2=0x7f7d180afd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:20.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.565+0000 7f7d15e00640 1 --2- 172.21.15.96:0/511172079 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7d181596b0 0x7f7d180afd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:20.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.565+0000 7f7d15e00640 1 -- 172.21.15.96:0/511172079 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d180b1510 msgr2=0x7f7d181619e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:20.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.565+0000 7f7d15e00640 1 --2- 172.21.15.96:0/511172079 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d180b1510 0x7f7d181619e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:20.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.565+0000 7f7d15e00640 1 -- 172.21.15.96:0/511172079 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d18159eb0 con 0x7f7cfc0052e0 2024-05-28T06:16:20.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.565+0000 7f7d15e00640 1 --2- 172.21.15.96:0/511172079 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7cfc0052e0 0x7f7d1814e5f0 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f7d08004f70 tx=0x7f7d080074f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:20.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.566+0000 7f7d0e000640 1 -- 172.21.15.96:0/511172079 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d0801a070 con 0x7f7cfc0052e0 2024-05-28T06:16:20.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.566+0000 7f7d17200640 1 -- 172.21.15.96:0/511172079 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d180b0470 con 0x7f7cfc0052e0 2024-05-28T06:16:20.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.566+0000 7f7d0e000640 1 -- 172.21.15.96:0/511172079 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d0800f6d0 con 0x7f7cfc0052e0 2024-05-28T06:16:20.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.566+0000 7f7d17200640 1 -- 172.21.15.96:0/511172079 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d180b0a30 con 0x7f7cfc0052e0 2024-05-28T06:16:20.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.566+0000 7f7d0e000640 1 -- 172.21.15.96:0/511172079 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d0800fac0 con 0x7f7cfc0052e0 2024-05-28T06:16:20.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.568+0000 7f7d17200640 1 -- 172.21.15.96:0/511172079 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7cf0005290 con 0x7f7cfc0052e0 2024-05-28T06:16:20.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.571+0000 7f7d0e000640 1 -- 172.21.15.96:0/511172079 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 26) ==== 92570+0+0 (secure 0 0 0) 0x7f7d0801d070 con 0x7f7cfc0052e0 2024-05-28T06:16:20.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.573+0000 7f7d0e000640 1 --2- 172.21.15.96:0/511172079 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7cf806d9d0 0x7f7cf806fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:20.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.573+0000 7f7d0e000640 1 -- 172.21.15.96:0/511172079 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5795+0+0 (secure 0 0 0) 0x7f7d0808a310 con 0x7f7cfc0052e0 2024-05-28T06:16:20.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.576+0000 7f7d15400640 1 --2- 172.21.15.96:0/511172079 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7cf806d9d0 0x7f7cf806fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:20.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.579+0000 7f7d0e000640 1 -- 172.21.15.96:0/511172079 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7d08059eb0 con 0x7f7cfc0052e0 2024-05-28T06:16:20.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.604+0000 7f7d15400640 1 --2- 172.21.15.96:0/511172079 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7cf806d9d0 0x7f7cf806fe90 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f7d04006fd0 tx=0x7f7d04006d90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:20.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:20.936+0000 7f7d17200640 1 -- 172.21.15.96:0/511172079 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone436318", "target": ["mon-mgr", ""]}) -- 0x7f7cf0000c80 con 0x7f7cf806d9d0 2024-05-28T06:16:21.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:21.779+0000 7f7d0e000640 1 -- 172.21.15.96:0/511172079 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f7cf0000c80 con 0x7f7cf806d9d0 2024-05-28T06:16:21.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:21.783+0000 7f7d17200640 1 -- 172.21.15.96:0/511172079 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7cf806d9d0 msgr2=0x7f7cf806fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:21.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:21.783+0000 7f7d17200640 1 --2- 172.21.15.96:0/511172079 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7cf806d9d0 0x7f7cf806fe90 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f7d04006fd0 tx=0x7f7d04006d90 comp rx=0 tx=0).stop 2024-05-28T06:16:21.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:21.783+0000 7f7d17200640 1 -- 172.21.15.96:0/511172079 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7cfc0052e0 msgr2=0x7f7d1814e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:21.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:21.784+0000 7f7d17200640 1 --2- 172.21.15.96:0/511172079 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7cfc0052e0 0x7f7d1814e5f0 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f7d08004f70 tx=0x7f7d080074f0 comp rx=0 tx=0).stop 2024-05-28T06:16:21.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:21.784+0000 7f7d17200640 1 -- 172.21.15.96:0/511172079 shutdown_connections 2024-05-28T06:16:21.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:21.784+0000 7f7d17200640 1 --2- 172.21.15.96:0/511172079 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7cf806d9d0 0x7f7cf806fe90 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:21.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:21.784+0000 7f7d17200640 1 --2- 172.21.15.96:0/511172079 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d180b1510 0x7f7d181619e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:21.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:21.784+0000 7f7d17200640 1 --2- 172.21.15.96:0/511172079 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7d181596b0 0x7f7d180afd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:21.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:21.784+0000 7f7d17200640 1 --2- 172.21.15.96:0/511172079 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7cfc0052e0 0x7f7d1814e5f0 secure :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f7d08004f70 tx=0x7f7d080074f0 comp rx=0 tx=0).stop 2024-05-28T06:16:21.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:21.784+0000 7f7d17200640 1 -- 172.21.15.96:0/511172079 >> 172.21.15.96:0/511172079 conn(0x7f7d1806b900 msgr2=0x7f7d180d0070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:16:21.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:21.784+0000 7f7d17200640 1 -- 172.21.15.96:0/511172079 shutdown_connections 2024-05-28T06:16:21.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:21.784+0000 7f7d17200640 1 -- 172.21.15.96:0/511172079 wait complete. 2024-05-28T06:16:21.816 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone436319 2024-05-28T06:16:22.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:22.106+0000 7f98b6800640 1 Processor -- start 2024-05-28T06:16:22.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:22.106+0000 7f98b6800640 1 -- start start 2024-05-28T06:16:22.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:22.108+0000 7f98b6800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f98b015fc00 0x7f98b00d0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:22.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:22.108+0000 7f98b6800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f98b015ffe0 0x7f98b00d0e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:22.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:22.108+0000 7f98b6800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f98b0071240 0x7f98b006ff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:22.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:22.108+0000 7f98b6800640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f98b00cde50 con 0x7f98b0071240 2024-05-28T06:16:22.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:22.108+0000 7f98b6800640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f98b00d00e0 con 0x7f98b015fc00 2024-05-28T06:16:22.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:22.108+0000 7f98b6800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f98b00d03c0 con 0x7f98b015ffe0 2024-05-28T06:16:22.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:22.109+0000 7f98afe00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f98b0071240 0x7f98b006ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:22.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:22.109+0000 7f98afe00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f98b0071240 0x7f98b006ff30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49350/0 (socket says 172.21.15.96:49350) 2024-05-28T06:16:22.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:22.109+0000 7f98afe00640 1 -- 172.21.15.96:0/1879027885 learned_addr learned my addr 172.21.15.96:0/1879027885 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:16:22.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:22.109+0000 7f98afe00640 1 -- 172.21.15.96:0/1879027885 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f98b015ffe0 msgr2=0x7f98b00d0e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:16:22.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:22.109+0000 7f98afe00640 1 --2- 172.21.15.96:0/1879027885 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f98b015ffe0 0x7f98b00d0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:22.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:22.109+0000 7f98afe00640 1 -- 172.21.15.96:0/1879027885 >> v2:172.21.15.123:3300/0 conn(0x7f98b015fc00 msgr2=0x7f98b00d0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:16:22.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:22.109+0000 7f98afe00640 1 --2- 172.21.15.96:0/1879027885 >> v2:172.21.15.123:3300/0 conn(0x7f98b015fc00 0x7f98b00d0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:22.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:22.109+0000 7f98afe00640 1 -- 172.21.15.96:0/1879027885 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98b00cdae0 con 0x7f98b0071240 2024-05-28T06:16:22.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:22.110+0000 7f98afe00640 1 --2- 172.21.15.96:0/1879027885 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f98b0071240 0x7f98b006ff30 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f98b00cfff0 tx=0x7f98a802a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=170c354f68cf70c6 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:22.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:22.111+0000 7f98aea00640 1 -- 172.21.15.96:0/1879027885 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98a8036070 con 0x7f98b0071240 2024-05-28T06:16:22.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:22.111+0000 7f98aea00640 1 -- 172.21.15.96:0/1879027885 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98a8031600 con 0x7f98b0071240 2024-05-28T06:16:22.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:22.111+0000 7f98aea00640 1 -- 172.21.15.96:0/1879027885 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98a80319d0 con 0x7f98b0071240 2024-05-28T06:16:22.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:22.112+0000 7f98b6800640 1 -- 172.21.15.96:0/1879027885 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f98b0071240 msgr2=0x7f98b006ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:22.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:22.112+0000 7f98b6800640 1 --2- 172.21.15.96:0/1879027885 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f98b0071240 0x7f98b006ff30 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f98b00cfff0 tx=0x7f98a802a610 comp rx=0 tx=0).stop 2024-05-28T06:16:22.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:22.112+0000 7f98b6800640 1 -- 172.21.15.96:0/1879027885 shutdown_connections 2024-05-28T06:16:22.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:22.112+0000 7f98b6800640 1 --2- 172.21.15.96:0/1879027885 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f98b0071240 0x7f98b006ff30 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:22.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:22.112+0000 7f98b6800640 1 --2- 172.21.15.96:0/1879027885 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f98b015ffe0 0x7f98b00d0e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:22.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:22.112+0000 7f98b6800640 1 --2- 172.21.15.96:0/1879027885 >> v2:172.21.15.123:3300/0 conn(0x7f98b015fc00 0x7f98b00d0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:22.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:22.112+0000 7f98b6800640 1 -- 172.21.15.96:0/1879027885 >> 172.21.15.96:0/1879027885 conn(0x7f98b00b3f60 msgr2=0x7f98b00b4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:16:22.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:22.112+0000 7f98b6800640 1 -- 172.21.15.96:0/1879027885 shutdown_connections 2024-05-28T06:16:22.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:22.112+0000 7f98b6800640 1 -- 172.21.15.96:0/1879027885 wait complete. 2024-05-28T06:16:22.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:22.114+0000 7f98b6800640 1 Processor -- start 2024-05-28T06:16:22.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:22.114+0000 7f98b6800640 1 -- start start 2024-05-28T06:16:22.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:22.115+0000 7f98b6800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f98b0071240 0x7f98b01f5720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:22.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:22.115+0000 7f98b6800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f98b015fc00 0x7f98b01f1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:22.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:22.115+0000 7f98b6800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f98b015ffe0 0x7f98b014e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:22.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:22.115+0000 7f98b6800640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f98b0071d00 con 0x7f98b015fc00 2024-05-28T06:16:22.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:22.115+0000 7f98b6800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f98b00b4f10 con 0x7f98b0071240 2024-05-28T06:16:22.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:22.115+0000 7f98b6800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f98b0071e80 con 0x7f98b015ffe0 2024-05-28T06:16:22.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:22.116+0000 7f98a7e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f98b015fc00 0x7f98b01f1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:22.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:22.116+0000 7f98a7e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f98b015fc00 0x7f98b01f1930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49366/0 (socket says 172.21.15.96:49366) 2024-05-28T06:16:22.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:22.116+0000 7f98a7e00640 1 -- 172.21.15.96:0/1890649617 learned_addr learned my addr 172.21.15.96:0/1890649617 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:16:22.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:22.116+0000 7f98af400640 1 --2- 172.21.15.96:0/1890649617 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f98b0071240 0x7f98b01f5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:22.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:22.116+0000 7f98a7e00640 1 -- 172.21.15.96:0/1890649617 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f98b015ffe0 msgr2=0x7f98b014e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:22.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:22.116+0000 7f98a7e00640 1 --2- 172.21.15.96:0/1890649617 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f98b015ffe0 0x7f98b014e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:22.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:22.116+0000 7f98a7e00640 1 -- 172.21.15.96:0/1890649617 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f98b0071240 msgr2=0x7f98b01f5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:22.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:22.116+0000 7f98a7e00640 1 --2- 172.21.15.96:0/1890649617 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f98b0071240 0x7f98b01f5720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:22.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:22.116+0000 7f98a7e00640 1 -- 172.21.15.96:0/1890649617 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98b014e1f0 con 0x7f98b015fc00 2024-05-28T06:16:22.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:22.116+0000 7f98af400640 1 --2- 172.21.15.96:0/1890649617 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f98b0071240 0x7f98b01f5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-28T06:16:22.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:22.117+0000 7f98a7e00640 1 --2- 172.21.15.96:0/1890649617 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f98b015fc00 0x7f98b01f1930 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f989800b480 tx=0x7f989800b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:22.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:22.117+0000 7f98a7400640 1 -- 172.21.15.96:0/1890649617 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9898014070 con 0x7f98b015fc00 2024-05-28T06:16:22.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:22.117+0000 7f98a7400640 1 -- 172.21.15.96:0/1890649617 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9898010870 con 0x7f98b015fc00 2024-05-28T06:16:22.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:22.117+0000 7f98a7400640 1 -- 172.21.15.96:0/1890649617 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9898010c60 con 0x7f98b015fc00 2024-05-28T06:16:22.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:22.117+0000 7f98b6800640 1 -- 172.21.15.96:0/1890649617 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98b01f5dc0 con 0x7f98b015fc00 2024-05-28T06:16:22.119 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:22.118+0000 7f98b6800640 1 -- 172.21.15.96:0/1890649617 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98b01f6380 con 0x7f98b015fc00 2024-05-28T06:16:22.119 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:22.119+0000 7f98b6800640 1 -- 172.21.15.96:0/1890649617 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98b00689e0 con 0x7f98b015fc00 2024-05-28T06:16:22.120 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:22.120+0000 7f98a7400640 1 -- 172.21.15.96:0/1890649617 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 26) ==== 92570+0+0 (secure 0 0 0) 0x7f9898017070 con 0x7f98b015fc00 2024-05-28T06:16:22.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:22.121+0000 7f98a7400640 1 --2- 172.21.15.96:0/1890649617 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f988c06d9d0 0x7f988c06fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:22.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:22.121+0000 7f98a7400640 1 -- 172.21.15.96:0/1890649617 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5795+0+0 (secure 0 0 0) 0x7f989808a820 con 0x7f98b015fc00 2024-05-28T06:16:22.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:22.126+0000 7f98a7400640 1 -- 172.21.15.96:0/1890649617 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f989805a3c0 con 0x7f98b015fc00 2024-05-28T06:16:22.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:22.139+0000 7f98af400640 1 --2- 172.21.15.96:0/1890649617 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f988c06d9d0 0x7f988c06fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:22.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:22.165+0000 7f98af400640 1 --2- 172.21.15.96:0/1890649617 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f988c06d9d0 0x7f988c06fe90 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f98a0009930 tx=0x7f98a0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:22.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:22.535+0000 7f98b6800640 1 -- 172.21.15.96:0/1890649617 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone436319", "target": ["mon-mgr", ""]}) -- 0x7f98b014e3c0 con 0x7f988c06d9d0 2024-05-28T06:16:22.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:22.927+0000 7f98a7400640 1 -- 172.21.15.96:0/1890649617 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mgrmap(e 27) ==== 92519+0+0 (secure 0 0 0) 0x7f9898056330 con 0x7f98b015fc00 2024-05-28T06:16:23.867 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:23.867+0000 7f98a7400640 1 -- 172.21.15.96:0/1890649617 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f98b014e3c0 con 0x7f988c06d9d0 2024-05-28T06:16:23.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:23.871+0000 7f98b6800640 1 -- 172.21.15.96:0/1890649617 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f988c06d9d0 msgr2=0x7f988c06fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:23.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:23.871+0000 7f98b6800640 1 --2- 172.21.15.96:0/1890649617 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f988c06d9d0 0x7f988c06fe90 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f98a0009930 tx=0x7f98a0008040 comp rx=0 tx=0).stop 2024-05-28T06:16:23.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:23.871+0000 7f98b6800640 1 -- 172.21.15.96:0/1890649617 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f98b015fc00 msgr2=0x7f98b01f1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:23.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:23.871+0000 7f98b6800640 1 --2- 172.21.15.96:0/1890649617 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f98b015fc00 0x7f98b01f1930 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f989800b480 tx=0x7f989800b940 comp rx=0 tx=0).stop 2024-05-28T06:16:23.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:23.872+0000 7f98b6800640 1 -- 172.21.15.96:0/1890649617 shutdown_connections 2024-05-28T06:16:23.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:23.872+0000 7f98b6800640 1 --2- 172.21.15.96:0/1890649617 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f988c06d9d0 0x7f988c06fe90 secure :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f98a0009930 tx=0x7f98a0008040 comp rx=0 tx=0).stop 2024-05-28T06:16:23.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:23.872+0000 7f98b6800640 1 --2- 172.21.15.96:0/1890649617 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f98b015ffe0 0x7f98b014e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:23.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:23.872+0000 7f98b6800640 1 --2- 172.21.15.96:0/1890649617 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f98b015fc00 0x7f98b01f1930 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:23.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:23.872+0000 7f98b6800640 1 --2- 172.21.15.96:0/1890649617 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f98b0071240 0x7f98b01f5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:23.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:23.872+0000 7f98b6800640 1 -- 172.21.15.96:0/1890649617 >> 172.21.15.96:0/1890649617 conn(0x7f98b00b3f60 msgr2=0x7f98b00b3600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:16:23.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:23.872+0000 7f98b6800640 1 -- 172.21.15.96:0/1890649617 shutdown_connections 2024-05-28T06:16:23.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:23.873+0000 7f98b6800640 1 -- 172.21.15.96:0/1890649617 wait complete. 2024-05-28T06:16:23.907 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone436320 2024-05-28T06:16:24.166 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:24.166+0000 7ff65bc00640 1 Processor -- start 2024-05-28T06:16:24.166 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:24.166+0000 7ff65bc00640 1 -- start start 2024-05-28T06:16:24.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:24.167+0000 7ff65bc00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7ff65415e850 0x7ff6540b77b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:24.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:24.167+0000 7ff65bc00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6540b8000 con 0x7ff6540cef30 2024-05-28T06:16:24.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:24.167+0000 7ff65bc00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6540b82e0 con 0x7ff6540d1d50 2024-05-28T06:16:24.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:24.167+0000 7ff65bc00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7ff65415ec30 con 0x7ff65415e850 2024-05-28T06:16:24.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:24.167+0000 7ff653e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7ff6540cef30 0x7ff6540d1640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:37166/0 (socket says 172.21.15.96:37166) 2024-05-28T06:16:24.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:24.167+0000 7ff653e00640 1 -- 172.21.15.96:0/1537222965 learned_addr learned my addr 172.21.15.96:0/1537222965 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:16:24.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:24.168+0000 7ff659200640 1 --2- 172.21.15.96:0/1537222965 >> v2:172.21.15.123:3300/0 conn(0x7ff65415e850 0x7ff6540b77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:24.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:24.168+0000 7ff653400640 1 -- 172.21.15.96:0/1537222965 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2925268017 0 0) 0x7ff6540b8000 con 0x7ff6540cef30 2024-05-28T06:16:24.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:24.168+0000 7ff653400640 1 -- 172.21.15.96:0/1537222965 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff63c003520 con 0x7ff6540cef30 2024-05-28T06:16:24.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:24.168+0000 7ff653400640 1 -- 172.21.15.96:0/1537222965 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3551251756 0 0) 0x7ff63c003520 con 0x7ff6540cef30 2024-05-28T06:16:24.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:24.168+0000 7ff653400640 1 -- 172.21.15.96:0/1537222965 >> v2:172.21.15.123:3300/0 conn(0x7ff65415e850 msgr2=0x7ff6540b77b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:16:24.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:24.168+0000 7ff653400640 1 --2- 172.21.15.96:0/1537222965 >> v2:172.21.15.123:3300/0 conn(0x7ff65415e850 0x7ff6540b77b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:24.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:24.168+0000 7ff653400640 1 -- 172.21.15.96:0/1537222965 >> v1:172.21.15.123:6789/0 conn(0x7ff6540d1d50 legacy=0x7ff6540b90a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:16:24.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:24.168+0000 7ff653400640 1 -- 172.21.15.96:0/1537222965 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6540b85c0 con 0x7ff6540cef30 2024-05-28T06:16:24.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:24.168+0000 7ff653400640 1 -- 172.21.15.96:0/1537222965 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7ff640001670 con 0x7ff6540cef30 2024-05-28T06:16:24.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:24.168+0000 7ff653400640 1 -- 172.21.15.96:0/1537222965 >> v1:172.21.15.96:6789/0 conn(0x7ff6540cef30 legacy=0x7ff6540d1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:24.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:24.168+0000 7ff653400640 1 --2- 172.21.15.96:0/1537222965 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff63c0039b0 0x7ff63c003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:24.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:24.168+0000 7ff653400640 1 --2- 172.21.15.96:0/1537222965 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff63c004800 0x7ff63c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:24.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:24.168+0000 7ff653400640 1 --2- 172.21.15.96:0/1537222965 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff63c0052e0 0x7ff63c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:24.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:24.168+0000 7ff653400640 1 -- 172.21.15.96:0/1537222965 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff63c00c160 con 0x7ff63c0039b0 2024-05-28T06:16:24.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:24.168+0000 7ff653400640 1 -- 172.21.15.96:0/1537222965 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff63c00c310 con 0x7ff63c0052e0 2024-05-28T06:16:24.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:24.168+0000 7ff653400640 1 -- 172.21.15.96:0/1537222965 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ff63c00c510 con 0x7ff63c004800 2024-05-28T06:16:24.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:24.169+0000 7ff659200640 1 --2- 172.21.15.96:0/1537222965 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff63c0039b0 0x7ff63c003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:24.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:24.169+0000 7ff659200640 1 -- 172.21.15.96:0/1537222965 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff63c004800 msgr2=0x7ff63c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:16:24.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:24.169+0000 7ff659200640 1 --2- 172.21.15.96:0/1537222965 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff63c004800 0x7ff63c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:24.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:24.169+0000 7ff659200640 1 -- 172.21.15.96:0/1537222965 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff63c0052e0 msgr2=0x7ff63c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:16:24.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:24.169+0000 7ff659200640 1 --2- 172.21.15.96:0/1537222965 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff63c0052e0 0x7ff63c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:24.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:24.169+0000 7ff659200640 1 -- 172.21.15.96:0/1537222965 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff63c003520 con 0x7ff63c0039b0 2024-05-28T06:16:24.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:24.169+0000 7ff659c00640 1 -- 172.21.15.96:0/1537222965 reap_dead start 2024-05-28T06:16:24.171 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:24.170+0000 7ff659200640 1 --2- 172.21.15.96:0/1537222965 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff63c0039b0 0x7ff63c003db0 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7ff64800ba00 tx=0x7ff63c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=834af741d0344fa6 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:24.172 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:24.171+0000 7ff653400640 1 -- 172.21.15.96:0/1537222965 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff648030070 con 0x7ff63c0039b0 2024-05-28T06:16:24.172 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:24.171+0000 7ff653400640 1 -- 172.21.15.96:0/1537222965 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff64802ade0 con 0x7ff63c0039b0 2024-05-28T06:16:24.172 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:24.171+0000 7ff653400640 1 -- 172.21.15.96:0/1537222965 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff64800b600 con 0x7ff63c0039b0 2024-05-28T06:16:24.172 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:24.171+0000 7ff65bc00640 1 -- 172.21.15.96:0/1537222965 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff63c0039b0 msgr2=0x7ff63c003db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:24.172 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:24.171+0000 7ff65bc00640 1 --2- 172.21.15.96:0/1537222965 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff63c0039b0 0x7ff63c003db0 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7ff64800ba00 tx=0x7ff63c005830 comp rx=0 tx=0).stop 2024-05-28T06:16:24.172 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:24.172+0000 7ff65bc00640 1 -- 172.21.15.96:0/1537222965 shutdown_connections 2024-05-28T06:16:24.172 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:24.172+0000 7ff65bc00640 1 --2- 172.21.15.96:0/1537222965 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff63c0039b0 0x7ff63c003db0 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:24.172 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:24.172+0000 7ff65bc00640 1 -- 172.21.15.96:0/1537222965 >> 172.21.15.96:0/1537222965 conn(0x7ff654067aa0 msgr2=0x7ff654067eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:16:24.172 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:24.172+0000 7ff65bc00640 1 -- 172.21.15.96:0/1537222965 shutdown_connections 2024-05-28T06:16:24.172 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:24.172+0000 7ff65bc00640 1 -- 172.21.15.96:0/1537222965 wait complete. 2024-05-28T06:16:24.174 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:24.174+0000 7ff65bc00640 1 Processor -- start 2024-05-28T06:16:24.174 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:24.174+0000 7ff65bc00640 1 -- start start 2024-05-28T06:16:24.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:24.175+0000 7ff65bc00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff63c0039b0 0x7ff654151bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:24.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:24.175+0000 7ff65bc00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff6540d35a0 0x7ff6540d3a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:24.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:24.175+0000 7ff65bc00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff65415fc40 0x7ff6541600a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:24.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:24.175+0000 7ff65bc00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff6540d17c0 con 0x7ff63c0039b0 2024-05-28T06:16:24.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:24.175+0000 7ff65bc00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff6540d1640 con 0x7ff65415fc40 2024-05-28T06:16:24.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:24.175+0000 7ff65bc00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ff6540d1940 con 0x7ff6540d35a0 2024-05-28T06:16:24.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:24.175+0000 7ff659200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff63c0039b0 0x7ff654151bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:24.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:24.175+0000 7ff659200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff63c0039b0 0x7ff654151bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49384/0 (socket says 172.21.15.96:49384) 2024-05-28T06:16:24.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:24.175+0000 7ff659200640 1 -- 172.21.15.96:0/2757496191 learned_addr learned my addr 172.21.15.96:0/2757496191 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:16:24.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:24.176+0000 7ff659c00640 1 --2- 172.21.15.96:0/2757496191 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff65415fc40 0x7ff6541600a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:24.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:24.176+0000 7ff653e00640 1 --2- 172.21.15.96:0/2757496191 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff6540d35a0 0x7ff6540d3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:24.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:24.176+0000 7ff659200640 1 -- 172.21.15.96:0/2757496191 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff6540d35a0 msgr2=0x7ff6540d3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:24.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:24.176+0000 7ff659200640 1 --2- 172.21.15.96:0/2757496191 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff6540d35a0 0x7ff6540d3a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:24.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:24.176+0000 7ff659200640 1 -- 172.21.15.96:0/2757496191 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff65415fc40 msgr2=0x7ff6541600a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:24.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:24.176+0000 7ff659200640 1 --2- 172.21.15.96:0/2757496191 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff65415fc40 0x7ff6541600a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:24.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:24.176+0000 7ff659200640 1 -- 172.21.15.96:0/2757496191 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6541520f0 con 0x7ff63c0039b0 2024-05-28T06:16:24.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:24.176+0000 7ff659200640 1 --2- 172.21.15.96:0/2757496191 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff63c0039b0 0x7ff654151bb0 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7ff64802a8d0 tx=0x7ff6480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:24.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:24.177+0000 7ff650c00640 1 -- 172.21.15.96:0/2757496191 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff648043020 con 0x7ff63c0039b0 2024-05-28T06:16:24.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:24.177+0000 7ff650c00640 1 -- 172.21.15.96:0/2757496191 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff648035eb0 con 0x7ff63c0039b0 2024-05-28T06:16:24.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:24.177+0000 7ff650c00640 1 -- 172.21.15.96:0/2757496191 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff64803f880 con 0x7ff63c0039b0 2024-05-28T06:16:24.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:24.177+0000 7ff65bc00640 1 -- 172.21.15.96:0/2757496191 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6540d4040 con 0x7ff63c0039b0 2024-05-28T06:16:24.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:24.177+0000 7ff65bc00640 1 -- 172.21.15.96:0/2757496191 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6540d4600 con 0x7ff63c0039b0 2024-05-28T06:16:24.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:24.179+0000 7ff65bc00640 1 -- 172.21.15.96:0/2757496191 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6541605e0 con 0x7ff63c0039b0 2024-05-28T06:16:24.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:24.181+0000 7ff650c00640 1 -- 172.21.15.96:0/2757496191 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 27) ==== 92519+0+0 (secure 0 0 0) 0x7ff648046020 con 0x7ff63c0039b0 2024-05-28T06:16:24.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:24.184+0000 7ff650c00640 1 --2- 172.21.15.96:0/2757496191 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff63406d940 0x7ff63406fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:24.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:24.184+0000 7ff650c00640 1 -- 172.21.15.96:0/2757496191 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5795+0+0 (secure 0 0 0) 0x7ff6480ab8a0 con 0x7ff63c0039b0 2024-05-28T06:16:24.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:24.191+0000 7ff650c00640 1 -- 172.21.15.96:0/2757496191 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff64804d020 con 0x7ff63c0039b0 2024-05-28T06:16:24.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:24.206+0000 7ff653e00640 1 --2- 172.21.15.96:0/2757496191 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff63406d940 0x7ff63406fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:24.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:24.211+0000 7ff653e00640 1 --2- 172.21.15.96:0/2757496191 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff63406d940 0x7ff63406fe00 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7ff654161680 tx=0x7ff6400016a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:24.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:24.578+0000 7ff65bc00640 1 -- 172.21.15.96:0/2757496191 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone436320", "target": ["mon-mgr", ""]}) -- 0x7ff65415f720 con 0x7ff63406d940 2024-05-28T06:16:25.444 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:25.443+0000 7ff650c00640 1 -- 172.21.15.96:0/2757496191 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7ff65415f720 con 0x7ff63406d940 2024-05-28T06:16:25.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:25.450+0000 7ff65bc00640 1 -- 172.21.15.96:0/2757496191 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff63406d940 msgr2=0x7ff63406fe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:25.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:25.450+0000 7ff65bc00640 1 --2- 172.21.15.96:0/2757496191 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff63406d940 0x7ff63406fe00 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7ff654161680 tx=0x7ff6400016a0 comp rx=0 tx=0).stop 2024-05-28T06:16:25.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:25.450+0000 7ff65bc00640 1 -- 172.21.15.96:0/2757496191 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff63c0039b0 msgr2=0x7ff654151bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:25.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:25.450+0000 7ff65bc00640 1 --2- 172.21.15.96:0/2757496191 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff63c0039b0 0x7ff654151bb0 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7ff64802a8d0 tx=0x7ff6480046c0 comp rx=0 tx=0).stop 2024-05-28T06:16:25.451 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:25.451+0000 7ff65bc00640 1 -- 172.21.15.96:0/2757496191 shutdown_connections 2024-05-28T06:16:25.451 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:25.451+0000 7ff65bc00640 1 --2- 172.21.15.96:0/2757496191 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff63406d940 0x7ff63406fe00 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:25.451 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:25.451+0000 7ff65bc00640 1 --2- 172.21.15.96:0/2757496191 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff65415fc40 0x7ff6541600a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:25.451 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:25.451+0000 7ff65bc00640 1 --2- 172.21.15.96:0/2757496191 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff6540d35a0 0x7ff6540d3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:25.451 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:25.451+0000 7ff65bc00640 1 --2- 172.21.15.96:0/2757496191 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff63c0039b0 0x7ff654151bb0 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:25.451 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:25.451+0000 7ff65bc00640 1 -- 172.21.15.96:0/2757496191 >> 172.21.15.96:0/2757496191 conn(0x7ff654067aa0 msgr2=0x7ff65415f010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:16:25.452 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:25.451+0000 7ff65bc00640 1 -- 172.21.15.96:0/2757496191 shutdown_connections 2024-05-28T06:16:25.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:25.453+0000 7ff65bc00640 1 -- 172.21.15.96:0/2757496191 wait complete. 2024-05-28T06:16:27.494 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-05-28T06:16:27.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:27.773+0000 7f9945800640 1 Processor -- start 2024-05-28T06:16:27.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:27.773+0000 7f9945800640 1 -- start start 2024-05-28T06:16:27.774 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:27.774+0000 7f9945800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f9940161d60 0x7f99400d3240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:27.774 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:27.774+0000 7f9945800640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f99400d2480 con 0x7f99400d0fa0 2024-05-28T06:16:27.774 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:27.774+0000 7f9945800640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f99400d2760 con 0x7f9940162140 2024-05-28T06:16:27.774 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:27.774+0000 7f9945800640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f99400d2230 con 0x7f9940161d60 2024-05-28T06:16:27.775 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:27.775+0000 7f993f400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f99400d0fa0 0x7f99400b74b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:37180/0 (socket says 172.21.15.96:37180) 2024-05-28T06:16:27.775 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:27.775+0000 7f993f400640 1 -- 172.21.15.96:0/3675596309 learned_addr learned my addr 172.21.15.96:0/3675596309 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:16:27.775 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:27.775+0000 7f993ea00640 1 --2- 172.21.15.96:0/3675596309 >> v2:172.21.15.96:3300/0 conn(0x7f9940161d60 0x7f99400d3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:27.775 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:27.775+0000 7f993ea00640 1 -- 172.21.15.96:0/3675596309 >> v1:172.21.15.123:6789/0 conn(0x7f9940162140 legacy=0x7f99400cfd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:16:27.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:27.775+0000 7f993ea00640 1 -- 172.21.15.96:0/3675596309 >> v1:172.21.15.96:6789/0 conn(0x7f99400d0fa0 legacy=0x7f99400b74b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:27.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:27.775+0000 7f993ea00640 1 -- 172.21.15.96:0/3675596309 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99400d2a40 con 0x7f9940161d60 2024-05-28T06:16:27.777 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:27.777+0000 7f993ea00640 1 --2- 172.21.15.96:0/3675596309 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9940161d60 0x7f99400d3240 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f99400d23b0 tx=0x7f993002a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59fac49ed51b95db server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:27.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:27.778+0000 7f993d600640 1 -- 172.21.15.96:0/3675596309 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9930037070 con 0x7f9940161d60 2024-05-28T06:16:27.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:27.778+0000 7f993d600640 1 -- 172.21.15.96:0/3675596309 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9930032440 con 0x7f9940161d60 2024-05-28T06:16:27.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:27.778+0000 7f993d600640 1 -- 172.21.15.96:0/3675596309 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9930032810 con 0x7f9940161d60 2024-05-28T06:16:27.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:27.779+0000 7f9945800640 1 -- 172.21.15.96:0/3675596309 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9940161d60 msgr2=0x7f99400d3240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:27.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:27.779+0000 7f9945800640 1 --2- 172.21.15.96:0/3675596309 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9940161d60 0x7f99400d3240 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f99400d23b0 tx=0x7f993002a1c0 comp rx=0 tx=0).stop 2024-05-28T06:16:27.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:27.780+0000 7f9945800640 1 -- 172.21.15.96:0/3675596309 shutdown_connections 2024-05-28T06:16:27.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:27.780+0000 7f9945800640 1 --2- 172.21.15.96:0/3675596309 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9940161d60 0x7f99400d3240 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:27.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:27.780+0000 7f9945800640 1 -- 172.21.15.96:0/3675596309 >> 172.21.15.96:0/3675596309 conn(0x7f994008bfc0 msgr2=0x7f99400d4660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:16:27.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:27.780+0000 7f9945800640 1 -- 172.21.15.96:0/3675596309 shutdown_connections 2024-05-28T06:16:27.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:27.780+0000 7f9945800640 1 -- 172.21.15.96:0/3675596309 wait complete. 2024-05-28T06:16:27.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:27.784+0000 7f9945800640 1 Processor -- start 2024-05-28T06:16:27.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:27.784+0000 7f9945800640 1 -- start start 2024-05-28T06:16:27.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:27.785+0000 7f9945800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f99400d0fa0 0x7f99400cce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:27.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:27.785+0000 7f9945800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9940161d60 0x7f99400c8f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:27.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:27.785+0000 7f9945800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9940162140 0x7f99400c9e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:27.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:27.785+0000 7f9945800640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f99400d50e0 con 0x7f9940162140 2024-05-28T06:16:27.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:27.785+0000 7f9945800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f99400d0490 con 0x7f9940161d60 2024-05-28T06:16:27.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:27.785+0000 7f9945800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f99400d5260 con 0x7f99400d0fa0 2024-05-28T06:16:27.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:27.785+0000 7f993f400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9940162140 0x7f99400c9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:27.786 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:27.785+0000 7f993f400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9940162140 0x7f99400c9e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49404/0 (socket says 172.21.15.96:49404) 2024-05-28T06:16:27.786 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:27.785+0000 7f993f400640 1 -- 172.21.15.96:0/2167157533 learned_addr learned my addr 172.21.15.96:0/2167157533 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:16:27.786 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:27.786+0000 7f993e000640 1 --2- 172.21.15.96:0/2167157533 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9940161d60 0x7f99400c8f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:27.786 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:27.786+0000 7f993ea00640 1 --2- 172.21.15.96:0/2167157533 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f99400d0fa0 0x7f99400cce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:27.787 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:27.787+0000 7f993f400640 1 -- 172.21.15.96:0/2167157533 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f99400d0fa0 msgr2=0x7f99400cce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:27.787 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:27.787+0000 7f993f400640 1 --2- 172.21.15.96:0/2167157533 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f99400d0fa0 0x7f99400cce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:27.787 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:27.787+0000 7f993f400640 1 -- 172.21.15.96:0/2167157533 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9940161d60 msgr2=0x7f99400c8f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:27.787 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:27.787+0000 7f993f400640 1 --2- 172.21.15.96:0/2167157533 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9940161d60 0x7f99400c8f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:27.787 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:27.787+0000 7f993f400640 1 -- 172.21.15.96:0/2167157533 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99400cbd90 con 0x7f9940162140 2024-05-28T06:16:27.787 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:27.787+0000 7f993f400640 1 --2- 172.21.15.96:0/2167157533 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9940162140 0x7f99400c9e90 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f993800caa0 tx=0x7f993800cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:27.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:27.788+0000 7f9936a00640 1 -- 172.21.15.96:0/2167157533 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9938017070 con 0x7f9940162140 2024-05-28T06:16:27.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:27.788+0000 7f9936a00640 1 -- 172.21.15.96:0/2167157533 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f993800ac00 con 0x7f9940162140 2024-05-28T06:16:27.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:27.788+0000 7f9936a00640 1 -- 172.21.15.96:0/2167157533 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9938012620 con 0x7f9940162140 2024-05-28T06:16:27.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:27.789+0000 7f9945800640 1 -- 172.21.15.96:0/2167157533 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99400cc070 con 0x7f9940162140 2024-05-28T06:16:27.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:27.789+0000 7f9945800640 1 -- 172.21.15.96:0/2167157533 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99400cc630 con 0x7f9940162140 2024-05-28T06:16:27.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:27.791+0000 7f9945800640 1 -- 172.21.15.96:0/2167157533 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99400689e0 con 0x7f9940162140 2024-05-28T06:16:27.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:27.796+0000 7f9936a00640 1 -- 172.21.15.96:0/2167157533 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 27) ==== 92519+0+0 (secure 0 0 0) 0x7f993801a070 con 0x7f9940162140 2024-05-28T06:16:27.798 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:27.798+0000 7f9936a00640 1 --2- 172.21.15.96:0/2167157533 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f992006da10 0x7f992006fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:27.798 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:27.798+0000 7f9936a00640 1 -- 172.21.15.96:0/2167157533 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5795+0+0 (secure 0 0 0) 0x7f993808ab60 con 0x7f9940162140 2024-05-28T06:16:27.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:27.802+0000 7f9936a00640 1 -- 172.21.15.96:0/2167157533 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9938021020 con 0x7f9940162140 2024-05-28T06:16:27.816 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:27.814+0000 7f993ea00640 1 --2- 172.21.15.96:0/2167157533 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f992006da10 0x7f992006fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:27.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:27.883+0000 7f993ea00640 1 --2- 172.21.15.96:0/2167157533 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f992006da10 0x7f992006fed0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f9930034000 tx=0x7f993002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:28.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.225+0000 7f9945800640 1 -- 172.21.15.96:0/2167157533 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f99400d0b70 con 0x7f9940162140 2024-05-28T06:16:28.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.226+0000 7f9936a00640 1 -- 172.21.15.96:0/2167157533 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+2423 (secure 0 0 0) 0x7f993805a070 con 0x7f9940162140 2024-05-28T06:16:28.226 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:16:28.226 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:16:28.226 INFO:teuthology.orchestra.run.smithi096.stdout: "fsid": "e4a831b5-e6d1-47a7-96cb-bf85154ee0d2", 2024-05-28T06:16:28.226 INFO:teuthology.orchestra.run.smithi096.stdout: "health": { 2024-05-28T06:16:28.226 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "HEALTH_OK", 2024-05-28T06:16:28.226 INFO:teuthology.orchestra.run.smithi096.stdout: "checks": {}, 2024-05-28T06:16:28.226 INFO:teuthology.orchestra.run.smithi096.stdout: "mutes": [] 2024-05-28T06:16:28.226 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:16:28.226 INFO:teuthology.orchestra.run.smithi096.stdout: "election_epoch": 8, 2024-05-28T06:16:28.226 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum": [ 2024-05-28T06:16:28.227 INFO:teuthology.orchestra.run.smithi096.stdout: 0, 2024-05-28T06:16:28.227 INFO:teuthology.orchestra.run.smithi096.stdout: 1, 2024-05-28T06:16:28.227 INFO:teuthology.orchestra.run.smithi096.stdout: 2 2024-05-28T06:16:28.227 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:16:28.227 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_names": [ 2024-05-28T06:16:28.227 INFO:teuthology.orchestra.run.smithi096.stdout: "a", 2024-05-28T06:16:28.227 INFO:teuthology.orchestra.run.smithi096.stdout: "b", 2024-05-28T06:16:28.227 INFO:teuthology.orchestra.run.smithi096.stdout: "c" 2024-05-28T06:16:28.227 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:16:28.227 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_age": 761, 2024-05-28T06:16:28.227 INFO:teuthology.orchestra.run.smithi096.stdout: "monmap": { 2024-05-28T06:16:28.227 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 1, 2024-05-28T06:16:28.227 INFO:teuthology.orchestra.run.smithi096.stdout: "min_mon_release_name": "squid", 2024-05-28T06:16:28.228 INFO:teuthology.orchestra.run.smithi096.stdout: "num_mons": 3 2024-05-28T06:16:28.228 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:16:28.228 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap": { 2024-05-28T06:16:28.228 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 76, 2024-05-28T06:16:28.228 INFO:teuthology.orchestra.run.smithi096.stdout: "num_osds": 8, 2024-05-28T06:16:28.228 INFO:teuthology.orchestra.run.smithi096.stdout: "num_up_osds": 8, 2024-05-28T06:16:28.228 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_up_since": 1716876238, 2024-05-28T06:16:28.228 INFO:teuthology.orchestra.run.smithi096.stdout: "num_in_osds": 8, 2024-05-28T06:16:28.228 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_in_since": 1716876229, 2024-05-28T06:16:28.228 INFO:teuthology.orchestra.run.smithi096.stdout: "num_remapped_pgs": 0 2024-05-28T06:16:28.228 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:16:28.228 INFO:teuthology.orchestra.run.smithi096.stdout: "pgmap": { 2024-05-28T06:16:28.228 INFO:teuthology.orchestra.run.smithi096.stdout: "pgs_by_state": [ 2024-05-28T06:16:28.229 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:16:28.229 INFO:teuthology.orchestra.run.smithi096.stdout: "state_name": "active+clean", 2024-05-28T06:16:28.229 INFO:teuthology.orchestra.run.smithi096.stdout: "count": 97 2024-05-28T06:16:28.229 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:16:28.229 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:16:28.229 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pgs": 97, 2024-05-28T06:16:28.229 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pools": 3, 2024-05-28T06:16:28.229 INFO:teuthology.orchestra.run.smithi096.stdout: "num_objects": 7073, 2024-05-28T06:16:28.229 INFO:teuthology.orchestra.run.smithi096.stdout: "data_bytes": 29467978489, 2024-05-28T06:16:28.229 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_used": 58758504448, 2024-05-28T06:16:28.229 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_avail": 714335608832, 2024-05-28T06:16:28.229 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_total": 773094113280, 2024-05-28T06:16:28.229 INFO:teuthology.orchestra.run.smithi096.stdout: "read_bytes_sec": 600027362, 2024-05-28T06:16:28.230 INFO:teuthology.orchestra.run.smithi096.stdout: "write_bytes_sec": 599386999, 2024-05-28T06:16:28.230 INFO:teuthology.orchestra.run.smithi096.stdout: "read_op_per_sec": 145, 2024-05-28T06:16:28.230 INFO:teuthology.orchestra.run.smithi096.stdout: "write_op_per_sec": 160 2024-05-28T06:16:28.230 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:16:28.230 INFO:teuthology.orchestra.run.smithi096.stdout: "fsmap": { 2024-05-28T06:16:28.230 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 44, 2024-05-28T06:16:28.230 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 5, 2024-05-28T06:16:28.230 INFO:teuthology.orchestra.run.smithi096.stdout: "up": 1, 2024-05-28T06:16:28.230 INFO:teuthology.orchestra.run.smithi096.stdout: "in": 1, 2024-05-28T06:16:28.230 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 1, 2024-05-28T06:16:28.230 INFO:teuthology.orchestra.run.smithi096.stdout: "by_rank": [ 2024-05-28T06:16:28.230 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:16:28.230 INFO:teuthology.orchestra.run.smithi096.stdout: "filesystem_id": 5, 2024-05-28T06:16:28.231 INFO:teuthology.orchestra.run.smithi096.stdout: "rank": 0, 2024-05-28T06:16:28.231 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "d", 2024-05-28T06:16:28.231 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "up:active", 2024-05-28T06:16:28.231 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 7146 2024-05-28T06:16:28.231 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:16:28.231 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:16:28.231 INFO:teuthology.orchestra.run.smithi096.stdout: "up:standby": 3 2024-05-28T06:16:28.231 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:16:28.231 INFO:teuthology.orchestra.run.smithi096.stdout: "mgrmap": { 2024-05-28T06:16:28.231 INFO:teuthology.orchestra.run.smithi096.stdout: "available": true, 2024-05-28T06:16:28.231 INFO:teuthology.orchestra.run.smithi096.stdout: "num_standbys": 1, 2024-05-28T06:16:28.231 INFO:teuthology.orchestra.run.smithi096.stdout: "modules": [ 2024-05-28T06:16:28.231 INFO:teuthology.orchestra.run.smithi096.stdout: "iostat", 2024-05-28T06:16:28.231 INFO:teuthology.orchestra.run.smithi096.stdout: "nfs", 2024-05-28T06:16:28.232 INFO:teuthology.orchestra.run.smithi096.stdout: "restful" 2024-05-28T06:16:28.232 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:16:28.232 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:16:28.232 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:16:28.232 INFO:teuthology.orchestra.run.smithi096.stdout: "servicemap": { 2024-05-28T06:16:28.232 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 13, 2024-05-28T06:16:28.232 INFO:teuthology.orchestra.run.smithi096.stdout: "modified": "2024-05-28T06:15:57.778889+0000", 2024-05-28T06:16:28.232 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:16:28.232 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:16:28.232 INFO:teuthology.orchestra.run.smithi096.stdout: "progress_events": { 2024-05-28T06:16:28.232 INFO:teuthology.orchestra.run.smithi096.stdout: "1f539510-8a8a-46db-8f0c-e233f5cc901d": { 2024-05-28T06:16:28.232 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "4 ongoing clones; Avg progress is 27.604% (36s)\n [=======.....................] (remaining: 96s)", 2024-05-28T06:16:28.232 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": 0.27603998780250549, 2024-05-28T06:16:28.232 INFO:teuthology.orchestra.run.smithi096.stdout: "add_to_ceph_s": true 2024-05-28T06:16:28.233 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:16:28.233 INFO:teuthology.orchestra.run.smithi096.stdout: "d1d4f508-f6fa-44f6-a2d0-6e7bd476fb5a": { 2024-05-28T06:16:28.233 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "11 ongoing+pending clones; Avg progress is 20.241% (28s)\n [=====.......................] (remaining: 112s)", 2024-05-28T06:16:28.233 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": 0.20240999758243561, 2024-05-28T06:16:28.233 INFO:teuthology.orchestra.run.smithi096.stdout: "add_to_ceph_s": true 2024-05-28T06:16:28.233 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:16:28.233 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:16:28.233 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:16:28.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.228+0000 7f9945800640 1 -- 172.21.15.96:0/2167157533 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f992006da10 msgr2=0x7f992006fed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:28.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.229+0000 7f9945800640 1 --2- 172.21.15.96:0/2167157533 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f992006da10 0x7f992006fed0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f9930034000 tx=0x7f993002e000 comp rx=0 tx=0).stop 2024-05-28T06:16:28.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.229+0000 7f9945800640 1 -- 172.21.15.96:0/2167157533 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9940162140 msgr2=0x7f99400c9e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:28.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.229+0000 7f9945800640 1 --2- 172.21.15.96:0/2167157533 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9940162140 0x7f99400c9e90 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f993800caa0 tx=0x7f993800cf60 comp rx=0 tx=0).stop 2024-05-28T06:16:28.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.229+0000 7f9945800640 1 -- 172.21.15.96:0/2167157533 shutdown_connections 2024-05-28T06:16:28.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.229+0000 7f9945800640 1 --2- 172.21.15.96:0/2167157533 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f992006da10 0x7f992006fed0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:28.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.229+0000 7f9945800640 1 --2- 172.21.15.96:0/2167157533 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9940162140 0x7f99400c9e90 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:28.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.229+0000 7f9945800640 1 --2- 172.21.15.96:0/2167157533 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9940161d60 0x7f99400c8f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:28.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.229+0000 7f9945800640 1 --2- 172.21.15.96:0/2167157533 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f99400d0fa0 0x7f99400cce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:28.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.229+0000 7f9945800640 1 -- 172.21.15.96:0/2167157533 >> 172.21.15.96:0/2167157533 conn(0x7f994008bfc0 msgr2=0x7f99400b95d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:16:28.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.229+0000 7f9945800640 1 -- 172.21.15.96:0/2167157533 shutdown_connections 2024-05-28T06:16:28.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.229+0000 7f9945800640 1 -- 172.21.15.96:0/2167157533 wait complete. 2024-05-28T06:16:28.254 INFO:tasks.cephfs.test_volumes:pevs - {'1f539510-8a8a-46db-8f0c-e233f5cc901d': {'message': '4 ongoing clones; Avg progress is 27.604% (36s)\n [=======.....................] (remaining: 96s)', 'progress': 0.2760399878025055, 'add_to_ceph_s': True}, 'd1d4f508-f6fa-44f6-a2d0-6e7bd476fb5a': {'message': '11 ongoing+pending clones; Avg progress is 20.241% (28s)\n [=====.......................] (remaining: 112s)', 'progress': 0.2024099975824356, 'add_to_ceph_s': True}} 2024-05-28T06:16:28.254 INFO:tasks.cephfs.test_volumes:new_onpen_p is less than onpen_p. 2024-05-28T06:16:28.254 INFO:tasks.cephfs.test_volumes:new_onpen_p = 0.2024099975824356; onpen_p = 0.318230003118515 2024-05-28T06:16:28.254 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone43631 2024-05-28T06:16:28.478 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.477+0000 7ff9c1800640 1 Processor -- start 2024-05-28T06:16:28.478 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.478+0000 7ff9c1800640 1 -- start start 2024-05-28T06:16:28.480 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.479+0000 7ff9c1800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff9bc0cfd00 0x7ff9bc0b27f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:28.480 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.479+0000 7ff9c1800640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9bc0b25d0 con 0x7ff9bc0cde00 2024-05-28T06:16:28.480 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.479+0000 7ff9c1800640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9bc0d02e0 con 0x7ff9bc0cd370 2024-05-28T06:16:28.480 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.479+0000 7ff9c1800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ff9bc0d05c0 con 0x7ff9bc0cfd00 2024-05-28T06:16:28.480 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.480+0000 7ff9bb400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7ff9bc0cde00 0x7ff9bc0cc0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:37192/0 (socket says 172.21.15.96:37192) 2024-05-28T06:16:28.480 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.480+0000 7ff9bb400640 1 -- 172.21.15.96:0/2754400545 learned_addr learned my addr 172.21.15.96:0/2754400545 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:16:28.480 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.480+0000 7ff9b9600640 1 -- 172.21.15.96:0/2754400545 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4033750543 0 0) 0x7ff9a8001110 con 0x7ff9bc0cde00 2024-05-28T06:16:28.481 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.481+0000 7ff9b9600640 1 -- 172.21.15.96:0/2754400545 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff9a0003520 con 0x7ff9bc0cde00 2024-05-28T06:16:28.481 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.481+0000 7ff9ba000640 1 --2- 172.21.15.96:0/2754400545 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff9bc0cfd00 0x7ff9bc0b27f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:28.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.481+0000 7ff9b9600640 1 -- 172.21.15.96:0/2754400545 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 28590122 0 0) 0x7ff9bc0d02e0 con 0x7ff9bc0cd370 2024-05-28T06:16:28.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.481+0000 7ff9b9600640 1 -- 172.21.15.96:0/2754400545 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff9a8001110 con 0x7ff9bc0cd370 2024-05-28T06:16:28.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.481+0000 7ff9b9600640 1 -- 172.21.15.96:0/2754400545 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1165397675 0 0) 0x7ff9a0003520 con 0x7ff9bc0cde00 2024-05-28T06:16:28.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.481+0000 7ff9b9600640 1 -- 172.21.15.96:0/2754400545 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff9bc0cfd00 msgr2=0x7ff9bc0b27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:16:28.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.481+0000 7ff9b9600640 1 --2- 172.21.15.96:0/2754400545 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff9bc0cfd00 0x7ff9bc0b27f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:28.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.481+0000 7ff9b9600640 1 -- 172.21.15.96:0/2754400545 >> v1:172.21.15.123:6789/0 conn(0x7ff9bc0cd370 legacy=0x7ff9bc0b1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:28.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.481+0000 7ff9b9600640 1 -- 172.21.15.96:0/2754400545 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9bc0b0d80 con 0x7ff9bc0cde00 2024-05-28T06:16:28.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.481+0000 7ff9b9600640 1 -- 172.21.15.96:0/2754400545 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7ff9a8001840 con 0x7ff9bc0cde00 2024-05-28T06:16:28.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.481+0000 7ff9b9600640 1 -- 172.21.15.96:0/2754400545 >> v1:172.21.15.96:6789/0 conn(0x7ff9bc0cde00 legacy=0x7ff9bc0cc0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:28.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.481+0000 7ff9b9600640 1 --2- 172.21.15.96:0/2754400545 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff9a0003b70 0x7ff9a0003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:28.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.481+0000 7ff9b9600640 1 --2- 172.21.15.96:0/2754400545 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff9a0004800 0x7ff9a0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:28.483 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.481+0000 7ff9b9600640 1 --2- 172.21.15.96:0/2754400545 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff9a00051c0 0x7ff9a000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:28.483 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.481+0000 7ff9b9600640 1 -- 172.21.15.96:0/2754400545 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff9bc0d05c0 con 0x7ff9a00051c0 2024-05-28T06:16:28.483 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.482+0000 7ff9b9600640 1 -- 172.21.15.96:0/2754400545 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff9a000e2a0 con 0x7ff9a0004800 2024-05-28T06:16:28.483 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.482+0000 7ff9b9600640 1 -- 172.21.15.96:0/2754400545 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ff9a000e470 con 0x7ff9a0003b70 2024-05-28T06:16:28.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.482+0000 7ff9bb400640 1 --2- 172.21.15.96:0/2754400545 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff9a00051c0 0x7ff9a000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:28.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.482+0000 7ff9ba000640 1 --2- 172.21.15.96:0/2754400545 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff9a0004800 0x7ff9a0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:28.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.482+0000 7ff9baa00640 1 --2- 172.21.15.96:0/2754400545 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff9a0003b70 0x7ff9a0003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:28.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.483+0000 7ff9bb400640 1 -- 172.21.15.96:0/2754400545 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff9a0003b70 msgr2=0x7ff9a0003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:16:28.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.483+0000 7ff9bb400640 1 --2- 172.21.15.96:0/2754400545 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff9a0003b70 0x7ff9a0003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:28.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.483+0000 7ff9bb400640 1 -- 172.21.15.96:0/2754400545 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff9a0004800 msgr2=0x7ff9a0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:16:28.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.483+0000 7ff9bb400640 1 --2- 172.21.15.96:0/2754400545 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff9a0004800 0x7ff9a0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:28.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.483+0000 7ff9bb400640 1 -- 172.21.15.96:0/2754400545 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9bc0b0d80 con 0x7ff9a00051c0 2024-05-28T06:16:28.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.485+0000 7ff9bb400640 1 --2- 172.21.15.96:0/2754400545 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff9a00051c0 0x7ff9a000b9c0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7ff9a800c510 tx=0x7ff9a0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=224c539378d5499e server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:28.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.485+0000 7ff9b9600640 1 -- 172.21.15.96:0/2754400545 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9a800c0e0 con 0x7ff9a00051c0 2024-05-28T06:16:28.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.485+0000 7ff9b9600640 1 -- 172.21.15.96:0/2754400545 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9a8034070 con 0x7ff9a00051c0 2024-05-28T06:16:28.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.485+0000 7ff9b9600640 1 -- 172.21.15.96:0/2754400545 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9a8005d10 con 0x7ff9a00051c0 2024-05-28T06:16:28.486 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.486+0000 7ff9c1800640 1 -- 172.21.15.96:0/2754400545 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff9a00051c0 msgr2=0x7ff9a000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:28.486 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.486+0000 7ff9c1800640 1 --2- 172.21.15.96:0/2754400545 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff9a00051c0 0x7ff9a000b9c0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7ff9a800c510 tx=0x7ff9a0005830 comp rx=0 tx=0).stop 2024-05-28T06:16:28.486 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.486+0000 7ff9bb400640 1 -- 172.21.15.96:0/2754400545 reap_dead start 2024-05-28T06:16:28.486 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.487+0000 7ff9c1800640 1 -- 172.21.15.96:0/2754400545 shutdown_connections 2024-05-28T06:16:28.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.487+0000 7ff9c1800640 1 -- 172.21.15.96:0/2754400545 >> 172.21.15.96:0/2754400545 conn(0x7ff9bc09aa30 msgr2=0x7ff9bc0b2f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:16:28.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.488+0000 7ff9c1800640 1 -- 172.21.15.96:0/2754400545 shutdown_connections 2024-05-28T06:16:28.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.488+0000 7ff9c1800640 1 -- 172.21.15.96:0/2754400545 wait complete. 2024-05-28T06:16:28.489 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.490+0000 7ff9c1800640 1 Processor -- start 2024-05-28T06:16:28.490 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.490+0000 7ff9c1800640 1 -- start start 2024-05-28T06:16:28.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.491+0000 7ff9c1800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff9a00051c0 0x7ff9bc0c8c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:28.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.491+0000 7ff9c1800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff9bc15c930 0x7ff9bc156830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:28.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.491+0000 7ff9c1800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff9bc15d310 0x7ff9bc15d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:28.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.491+0000 7ff9c1800640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff9bc0b1f70 con 0x7ff9a00051c0 2024-05-28T06:16:28.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.491+0000 7ff9c1800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff9bc0b1340 con 0x7ff9bc15d310 2024-05-28T06:16:28.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.491+0000 7ff9c1800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ff9bc1f4860 con 0x7ff9bc15c930 2024-05-28T06:16:28.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.491+0000 7ff9baa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff9a00051c0 0x7ff9bc0c8c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:28.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.491+0000 7ff9baa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff9a00051c0 0x7ff9bc0c8c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49418/0 (socket says 172.21.15.96:49418) 2024-05-28T06:16:28.494 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.491+0000 7ff9baa00640 1 -- 172.21.15.96:0/1461202868 learned_addr learned my addr 172.21.15.96:0/1461202868 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:16:28.494 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.492+0000 7ff9bb400640 1 --2- 172.21.15.96:0/1461202868 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff9bc15d310 0x7ff9bc15d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:28.494 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.492+0000 7ff9ba000640 1 --2- 172.21.15.96:0/1461202868 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff9bc15c930 0x7ff9bc156830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:28.494 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.492+0000 7ff9baa00640 1 -- 172.21.15.96:0/1461202868 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff9bc15c930 msgr2=0x7ff9bc156830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:28.494 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.492+0000 7ff9baa00640 1 --2- 172.21.15.96:0/1461202868 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff9bc15c930 0x7ff9bc156830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:28.494 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.493+0000 7ff9baa00640 1 -- 172.21.15.96:0/1461202868 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff9bc15d310 msgr2=0x7ff9bc15d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:28.494 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.493+0000 7ff9baa00640 1 --2- 172.21.15.96:0/1461202868 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff9bc15d310 0x7ff9bc15d790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:28.494 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.493+0000 7ff9baa00640 1 -- 172.21.15.96:0/1461202868 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9bc0b5090 con 0x7ff9a00051c0 2024-05-28T06:16:28.494 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.494+0000 7ff9baa00640 1 --2- 172.21.15.96:0/1461202868 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff9a00051c0 0x7ff9bc0c8c90 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7ff9ac00e7b0 tx=0x7ff9ac00ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:28.495 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.494+0000 7ff9b2a00640 1 -- 172.21.15.96:0/1461202868 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9ac00ee60 con 0x7ff9a00051c0 2024-05-28T06:16:28.495 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.494+0000 7ff9b2a00640 1 -- 172.21.15.96:0/1461202868 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9ac018070 con 0x7ff9a00051c0 2024-05-28T06:16:28.495 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.494+0000 7ff9b2a00640 1 -- 172.21.15.96:0/1461202868 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9ac00c6c0 con 0x7ff9a00051c0 2024-05-28T06:16:28.495 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.495+0000 7ff9c1800640 1 -- 172.21.15.96:0/1461202868 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9bc0c91d0 con 0x7ff9a00051c0 2024-05-28T06:16:28.496 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.495+0000 7ff9c1800640 1 -- 172.21.15.96:0/1461202868 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9bc0b5330 con 0x7ff9a00051c0 2024-05-28T06:16:28.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.499+0000 7ff9c1800640 1 -- 172.21.15.96:0/1461202868 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9bc15dcd0 con 0x7ff9a00051c0 2024-05-28T06:16:28.501 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.501+0000 7ff9b2a00640 1 -- 172.21.15.96:0/1461202868 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 27) ==== 92519+0+0 (secure 0 0 0) 0x7ff9ac00c860 con 0x7ff9a00051c0 2024-05-28T06:16:28.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.503+0000 7ff9b2a00640 1 --2- 172.21.15.96:0/1461202868 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff99c06d9c0 0x7ff99c06fe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:28.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.503+0000 7ff9b2a00640 1 -- 172.21.15.96:0/1461202868 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5795+0+0 (secure 0 0 0) 0x7ff9ac08a7b0 con 0x7ff9a00051c0 2024-05-28T06:16:28.508 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.508+0000 7ff9b2a00640 1 -- 172.21.15.96:0/1461202868 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff9ac05a350 con 0x7ff9a00051c0 2024-05-28T06:16:28.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.520+0000 7ff9ba000640 1 --2- 172.21.15.96:0/1461202868 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff99c06d9c0 0x7ff99c06fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:28.536 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.536+0000 7ff9ba000640 1 --2- 172.21.15.96:0/1461202868 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff99c06d9c0 0x7ff99c06fe80 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7ff9b4003fd0 tx=0x7ff9b400b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:28.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:28.876+0000 7ff9c1800640 1 -- 172.21.15.96:0/1461202868 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone43631", "target": ["mon-mgr", ""]}) -- 0x7ff9bc0690f0 con 0x7ff99c06d9c0 2024-05-28T06:16:28.935 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:16:28.936+0000 7fea59e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:16:28.936 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:16:28.936+0000 7fea59e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:16:28.936 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:16:28.936+0000 7fea59e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:16:28.936 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:16:28.936+0000 7fea59e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:16:28.936 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:16:28.936+0000 7fea59e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:16:29.233 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:16:29.233+0000 7fea59e00640 -1 mgr.server reply reply (22) Invalid argument cannot cancel -- clone finished (check clone status) 2024-05-28T06:16:29.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:29.242+0000 7ff9b2a00640 1 -- 172.21.15.96:0/1461202868 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: -22 cannot cancel -- clone finished (check clone status)) ==== 60+0+0 (secure 0 0 0) 0x7ff9bc0690f0 con 0x7ff99c06d9c0 2024-05-28T06:16:29.243 INFO:teuthology.orchestra.run.smithi096.stderr:Error EINVAL: cannot cancel -- clone finished (check clone status) 2024-05-28T06:16:29.257 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-05-28T06:16:29.258 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone43632 2024-05-28T06:16:29.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:29.600+0000 7f41f3200640 1 Processor -- start 2024-05-28T06:16:29.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:29.600+0000 7f41f3200640 1 -- start start 2024-05-28T06:16:29.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:29.601+0000 7f41f3200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f41f415e870 0x7f41f40d0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:29.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:29.601+0000 7f41f3200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f41f40d14d0 0x7f41f40cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:29.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:29.601+0000 7f41f3200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f41f40cf870 0x7f41f4162360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:29.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:29.601+0000 7f41f3200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f41f40d2150 con 0x7f41f415e870 2024-05-28T06:16:29.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:29.601+0000 7f41f3200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f41f40cfd00 con 0x7f41f40cf870 2024-05-28T06:16:29.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:29.601+0000 7f41f3200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f41f415ec50 con 0x7f41f40d14d0 2024-05-28T06:16:29.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:29.601+0000 7f41f1e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f41f415e870 0x7f41f40d0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:29.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:29.601+0000 7f41f1e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f41f415e870 0x7f41f40d0f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:45770/0 (socket says 172.21.15.96:45770) 2024-05-28T06:16:29.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:29.601+0000 7f41f1e00640 1 -- 172.21.15.96:0/983744794 learned_addr learned my addr 172.21.15.96:0/983744794 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:16:29.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:29.601+0000 7f41f2800640 1 --2- 172.21.15.96:0/983744794 >> v2:172.21.15.123:3300/0 conn(0x7f41f40cf870 0x7f41f4162360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:29.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:29.602+0000 7f41f1400640 1 --2- 172.21.15.96:0/983744794 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f41f40d14d0 0x7f41f40cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:29.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:29.602+0000 7f41f1e00640 1 -- 172.21.15.96:0/983744794 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f41f40d14d0 msgr2=0x7f41f40cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:16:29.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:29.602+0000 7f41f1e00640 1 --2- 172.21.15.96:0/983744794 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f41f40d14d0 0x7f41f40cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:29.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:29.602+0000 7f41f1e00640 1 -- 172.21.15.96:0/983744794 >> v2:172.21.15.123:3300/0 conn(0x7f41f40cf870 msgr2=0x7f41f4162360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:16:29.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:29.602+0000 7f41f1e00640 1 --2- 172.21.15.96:0/983744794 >> v2:172.21.15.123:3300/0 conn(0x7f41f40cf870 0x7f41f4162360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:29.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:29.602+0000 7f41f1e00640 1 -- 172.21.15.96:0/983744794 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41f40cff80 con 0x7f41f415e870 2024-05-28T06:16:29.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:29.603+0000 7f41f1e00640 1 --2- 172.21.15.96:0/983744794 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41f415e870 0x7f41f40d0f90 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f41f40b3f60 tx=0x7f41e402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bee79237aee89d7 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:29.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:29.603+0000 7f41f0a00640 1 -- 172.21.15.96:0/983744794 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41e4037070 con 0x7f41f415e870 2024-05-28T06:16:29.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:29.603+0000 7f41f0a00640 1 -- 172.21.15.96:0/983744794 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41e4032440 con 0x7f41f415e870 2024-05-28T06:16:29.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:29.603+0000 7f41f0a00640 1 -- 172.21.15.96:0/983744794 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41e4032810 con 0x7f41f415e870 2024-05-28T06:16:29.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:29.604+0000 7f41f3200640 1 -- 172.21.15.96:0/983744794 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41f415e870 msgr2=0x7f41f40d0f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:29.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:29.604+0000 7f41f3200640 1 --2- 172.21.15.96:0/983744794 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41f415e870 0x7f41f40d0f90 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f41f40b3f60 tx=0x7f41e402a1e0 comp rx=0 tx=0).stop 2024-05-28T06:16:29.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:29.604+0000 7f41f3200640 1 -- 172.21.15.96:0/983744794 shutdown_connections 2024-05-28T06:16:29.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:29.604+0000 7f41f3200640 1 --2- 172.21.15.96:0/983744794 >> v2:172.21.15.123:3300/0 conn(0x7f41f40cf870 0x7f41f4162360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:29.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:29.604+0000 7f41f3200640 1 --2- 172.21.15.96:0/983744794 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f41f40d14d0 0x7f41f40cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:29.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:29.604+0000 7f41f3200640 1 --2- 172.21.15.96:0/983744794 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41f415e870 0x7f41f40d0f90 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:29.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:29.604+0000 7f41f3200640 1 -- 172.21.15.96:0/983744794 >> 172.21.15.96:0/983744794 conn(0x7f41f408bfc0 msgr2=0x7f41f40d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:16:29.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:29.604+0000 7f41f3200640 1 -- 172.21.15.96:0/983744794 shutdown_connections 2024-05-28T06:16:29.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:29.604+0000 7f41f3200640 1 -- 172.21.15.96:0/983744794 wait complete. 2024-05-28T06:16:29.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:29.606+0000 7f41f3200640 1 Processor -- start 2024-05-28T06:16:29.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:29.606+0000 7f41f3200640 1 -- start start 2024-05-28T06:16:29.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:29.607+0000 7f41f3200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41f40cf870 0x7f41f40b90d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:29.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:29.607+0000 7f41f3200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41f40d14d0 0x7f41f4159940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:29.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:29.607+0000 7f41f3200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f41f415e870 0x7f41f415c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:29.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:29.607+0000 7f41f3200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f41f40d0820 con 0x7f41f40cf870 2024-05-28T06:16:29.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:29.607+0000 7f41f3200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f41f40d0540 con 0x7f41f40d14d0 2024-05-28T06:16:29.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:29.607+0000 7f41f3200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f41f40d4fb0 con 0x7f41f415e870 2024-05-28T06:16:29.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:29.607+0000 7f41f1e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41f40cf870 0x7f41f40b90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:29.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:29.607+0000 7f41f1e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41f40cf870 0x7f41f40b90d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:45772/0 (socket says 172.21.15.96:45772) 2024-05-28T06:16:29.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:29.607+0000 7f41f1e00640 1 -- 172.21.15.96:0/3257401871 learned_addr learned my addr 172.21.15.96:0/3257401871 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:16:29.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:29.608+0000 7f41f1e00640 1 -- 172.21.15.96:0/3257401871 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f41f415e870 msgr2=0x7f41f415c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:29.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:29.608+0000 7f41f1e00640 1 --2- 172.21.15.96:0/3257401871 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f41f415e870 0x7f41f415c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:29.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:29.608+0000 7f41f1e00640 1 -- 172.21.15.96:0/3257401871 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41f40d14d0 msgr2=0x7f41f4159940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:29.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:29.608+0000 7f41f1e00640 1 --2- 172.21.15.96:0/3257401871 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41f40d14d0 0x7f41f4159940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:29.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:29.608+0000 7f41f1e00640 1 -- 172.21.15.96:0/3257401871 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41f41fd1e0 con 0x7f41f40cf870 2024-05-28T06:16:29.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:29.608+0000 7f41f1e00640 1 --2- 172.21.15.96:0/3257401871 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41f40cf870 0x7f41f40b90d0 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f41e402a220 tx=0x7f41e40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:29.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:29.609+0000 7f41ea000640 1 -- 172.21.15.96:0/3257401871 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41e4041070 con 0x7f41f40cf870 2024-05-28T06:16:29.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:29.609+0000 7f41ea000640 1 -- 172.21.15.96:0/3257401871 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41e403c870 con 0x7f41f40cf870 2024-05-28T06:16:29.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:29.609+0000 7f41ea000640 1 -- 172.21.15.96:0/3257401871 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41e4037040 con 0x7f41f40cf870 2024-05-28T06:16:29.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:29.609+0000 7f41f3200640 1 -- 172.21.15.96:0/3257401871 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41f40b85f0 con 0x7f41f40cf870 2024-05-28T06:16:29.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:29.609+0000 7f41f3200640 1 -- 172.21.15.96:0/3257401871 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41f40b8bb0 con 0x7f41f40cf870 2024-05-28T06:16:29.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:29.611+0000 7f41f3200640 1 -- 172.21.15.96:0/3257401871 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41f40689e0 con 0x7f41f40cf870 2024-05-28T06:16:29.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:29.612+0000 7f41ea000640 1 -- 172.21.15.96:0/3257401871 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 27) ==== 92519+0+0 (secure 0 0 0) 0x7f41e4044070 con 0x7f41f40cf870 2024-05-28T06:16:29.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:29.613+0000 7f41ea000640 1 --2- 172.21.15.96:0/3257401871 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f41d406d940 0x7f41d406fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:29.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:29.613+0000 7f41ea000640 1 -- 172.21.15.96:0/3257401871 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5795+0+0 (secure 0 0 0) 0x7f41e40ab680 con 0x7f41f40cf870 2024-05-28T06:16:29.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:29.619+0000 7f41ea000640 1 -- 172.21.15.96:0/3257401871 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f41e407b340 con 0x7f41f40cf870 2024-05-28T06:16:29.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:29.647+0000 7f41f1400640 1 --2- 172.21.15.96:0/3257401871 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f41d406d940 0x7f41d406fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:29.731 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:29.732+0000 7f41f1400640 1 --2- 172.21.15.96:0/3257401871 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f41d406d940 0x7f41d406fe00 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f41e0006fd0 tx=0x7f41e0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:29.970 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:29.970+0000 7f41f3200640 1 -- 172.21.15.96:0/3257401871 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone43632", "target": ["mon-mgr", ""]}) -- 0x7f41f4072e10 con 0x7f41d406d940 2024-05-28T06:16:30.172 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:30.172+0000 7f41ea000640 1 -- 172.21.15.96:0/3257401871 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mgrmap(e 28) ==== 92570+0+0 (secure 0 0 0) 0x7f41e403a070 con 0x7f41f40cf870 2024-05-28T06:16:30.771 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:16:30.771+0000 7fea59e00640 -1 mgr.server reply reply (22) Invalid argument cannot cancel -- clone finished (check clone status) 2024-05-28T06:16:30.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:30.773+0000 7f41ea000640 1 -- 172.21.15.96:0/3257401871 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: -22 cannot cancel -- clone finished (check clone status)) ==== 60+0+0 (secure 0 0 0) 0x7f41f4072e10 con 0x7f41d406d940 2024-05-28T06:16:30.773 INFO:teuthology.orchestra.run.smithi096.stderr:Error EINVAL: cannot cancel -- clone finished (check clone status) 2024-05-28T06:16:30.785 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-05-28T06:16:30.786 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone43633 2024-05-28T06:16:31.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:31.060+0000 7f15d5000640 1 Processor -- start 2024-05-28T06:16:31.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:31.060+0000 7f15d5000640 1 -- start start 2024-05-28T06:16:31.062 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:31.062+0000 7f15d5000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f15d00d16a0 0x7f15d00cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:31.062 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:31.062+0000 7f15d5000640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15d015ec50 con 0x7f15d015e870 2024-05-28T06:16:31.062 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:31.062+0000 7f15d5000640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15d00cff80 con 0x7f15d00cf870 2024-05-28T06:16:31.062 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:31.062+0000 7f15d5000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f15d00d2150 con 0x7f15d00d16a0 2024-05-28T06:16:31.062 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:31.062+0000 7f15cf400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f15d015e870 0x7f15d00d0f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:51456/0 (socket says 172.21.15.96:51456) 2024-05-28T06:16:31.062 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:31.062+0000 7f15cf400640 1 -- 172.21.15.96:0/3999982381 learned_addr learned my addr 172.21.15.96:0/3999982381 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:16:31.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:31.063+0000 7f15ce000640 1 -- 172.21.15.96:0/3999982381 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1369595499 0 0) 0x7f15d015ec50 con 0x7f15d015e870 2024-05-28T06:16:31.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:31.063+0000 7f15ce000640 1 -- 172.21.15.96:0/3999982381 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f15b4003520 con 0x7f15d015e870 2024-05-28T06:16:31.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:31.063+0000 7f15cea00640 1 --2- 172.21.15.96:0/3999982381 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f15d00d16a0 0x7f15d00cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:31.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:31.063+0000 7f15ce000640 1 -- 172.21.15.96:0/3999982381 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 126102843 0 0) 0x7f15b4003520 con 0x7f15d015e870 2024-05-28T06:16:31.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:31.063+0000 7f15ce000640 1 -- 172.21.15.96:0/3999982381 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f15d00d16a0 msgr2=0x7f15d00cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:16:31.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:31.063+0000 7f15ce000640 1 --2- 172.21.15.96:0/3999982381 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f15d00d16a0 0x7f15d00cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:31.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:31.063+0000 7f15ce000640 1 -- 172.21.15.96:0/3999982381 >> v1:172.21.15.123:6789/0 conn(0x7f15d00cf870 legacy=0x7f15d00d25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:16:31.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:31.063+0000 7f15ce000640 1 -- 172.21.15.96:0/3999982381 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15d00d0260 con 0x7f15d015e870 2024-05-28T06:16:31.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:31.063+0000 7f15ce000640 1 -- 172.21.15.96:0/3999982381 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f15c0001670 con 0x7f15d015e870 2024-05-28T06:16:31.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:31.063+0000 7f15cea00640 1 --2- 172.21.15.96:0/3999982381 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f15d00d16a0 0x7f15d00cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:16:31.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:31.063+0000 7f15ce000640 1 -- 172.21.15.96:0/3999982381 >> v1:172.21.15.96:6789/0 conn(0x7f15d015e870 legacy=0x7f15d00d0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:31.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:31.063+0000 7f15ce000640 1 --2- 172.21.15.96:0/3999982381 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f15b40039e0 0x7f15b4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:31.065 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:31.063+0000 7f15ce000640 1 --2- 172.21.15.96:0/3999982381 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f15d00d16a0 0x7f15b4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:31.065 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:31.063+0000 7f15ce000640 1 --2- 172.21.15.96:0/3999982381 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f15b4004db0 0x7f15b4005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:31.065 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:31.064+0000 7f15ce000640 1 -- 172.21.15.96:0/3999982381 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f15d00d2150 con 0x7f15b4004db0 2024-05-28T06:16:31.065 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:31.064+0000 7f15ce000640 1 -- 172.21.15.96:0/3999982381 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f15b400bbd0 con 0x7f15b40039e0 2024-05-28T06:16:31.065 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:31.064+0000 7f15ce000640 1 -- 172.21.15.96:0/3999982381 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f15b400bdd0 con 0x7f15d00d16a0 2024-05-28T06:16:31.065 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:31.064+0000 7f15cfe00640 1 --2- 172.21.15.96:0/3999982381 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f15b4004db0 0x7f15b4005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:31.065 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:31.064+0000 7f15cf400640 1 --2- 172.21.15.96:0/3999982381 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f15b40039e0 0x7f15b4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:31.065 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:31.064+0000 7f15cea00640 1 --2- 172.21.15.96:0/3999982381 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f15d00d16a0 0x7f15b4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:31.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:31.065+0000 7f15cea00640 1 -- 172.21.15.96:0/3999982381 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f15b40039e0 msgr2=0x7f15b4003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:16:31.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:31.065+0000 7f15cea00640 1 --2- 172.21.15.96:0/3999982381 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f15b40039e0 0x7f15b4003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:31.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:31.065+0000 7f15cea00640 1 -- 172.21.15.96:0/3999982381 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f15b4004db0 msgr2=0x7f15b4005230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:16:31.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:31.065+0000 7f15cea00640 1 --2- 172.21.15.96:0/3999982381 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f15b4004db0 0x7f15b4005230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:31.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:31.065+0000 7f15cea00640 1 -- 172.21.15.96:0/3999982381 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15b4003520 con 0x7f15d00d16a0 2024-05-28T06:16:31.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:31.065+0000 7f15cfe00640 1 --2- 172.21.15.96:0/3999982381 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f15b4004db0 0x7f15b4005230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-28T06:16:31.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:31.066+0000 7f15cea00640 1 --2- 172.21.15.96:0/3999982381 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f15d00d16a0 0x7f15b4004800 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f15bc00aa30 tx=0x7f15b4005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3d631a55f5206120 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:31.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:31.067+0000 7f15ce000640 1 -- 172.21.15.96:0/3999982381 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15bc037070 con 0x7f15d00d16a0 2024-05-28T06:16:31.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:31.067+0000 7f15ce000640 1 -- 172.21.15.96:0/3999982381 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15bc02ac40 con 0x7f15d00d16a0 2024-05-28T06:16:31.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:31.067+0000 7f15ce000640 1 -- 172.21.15.96:0/3999982381 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15bc00a630 con 0x7f15d00d16a0 2024-05-28T06:16:31.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:31.068+0000 7f15d5000640 1 -- 172.21.15.96:0/3999982381 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f15d00d16a0 msgr2=0x7f15b4004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:31.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:31.068+0000 7f15d5000640 1 --2- 172.21.15.96:0/3999982381 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f15d00d16a0 0x7f15b4004800 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f15bc00aa30 tx=0x7f15b4005830 comp rx=0 tx=0).stop 2024-05-28T06:16:31.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:31.068+0000 7f15cfe00640 1 -- 172.21.15.96:0/3999982381 reap_dead start 2024-05-28T06:16:31.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:31.068+0000 7f15d5000640 1 -- 172.21.15.96:0/3999982381 shutdown_connections 2024-05-28T06:16:31.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:31.068+0000 7f15d5000640 1 -- 172.21.15.96:0/3999982381 >> 172.21.15.96:0/3999982381 conn(0x7f15d008bfc0 msgr2=0x7f15d00d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:16:31.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:31.068+0000 7f15d5000640 1 -- 172.21.15.96:0/3999982381 shutdown_connections 2024-05-28T06:16:31.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:31.068+0000 7f15d5000640 1 -- 172.21.15.96:0/3999982381 wait complete. 2024-05-28T06:16:31.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:31.070+0000 7f15d5000640 1 Processor -- start 2024-05-28T06:16:31.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:31.070+0000 7f15d5000640 1 -- start start 2024-05-28T06:16:31.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:31.070+0000 7f15d5000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f15d00d16a0 0x7f15d00b95c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:31.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:31.071+0000 7f15d5000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f15d0152330 0x7f15d0151ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:31.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:31.071+0000 7f15d5000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f15d00ccc40 0x7f15d00cd0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:31.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:31.071+0000 7f15d5000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f15d00d0820 con 0x7f15d00d16a0 2024-05-28T06:16:31.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:31.071+0000 7f15d5000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f15d00cfd20 con 0x7f15d00ccc40 2024-05-28T06:16:31.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:31.071+0000 7f15d5000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f15d00d4fb0 con 0x7f15d0152330 2024-05-28T06:16:31.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:31.071+0000 7f15cf400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f15d00d16a0 0x7f15d00b95c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:31.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:31.071+0000 7f15cf400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f15d00d16a0 0x7f15d00b95c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:45796/0 (socket says 172.21.15.96:45796) 2024-05-28T06:16:31.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:31.071+0000 7f15cf400640 1 -- 172.21.15.96:0/586654268 learned_addr learned my addr 172.21.15.96:0/586654268 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:16:31.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:31.071+0000 7f15cfe00640 1 --2- 172.21.15.96:0/586654268 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f15d00ccc40 0x7f15d00cd0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:31.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:31.071+0000 7f15cea00640 1 --2- 172.21.15.96:0/586654268 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f15d0152330 0x7f15d0151ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:31.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:31.071+0000 7f15cf400640 1 -- 172.21.15.96:0/586654268 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f15d0152330 msgr2=0x7f15d0151ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:31.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:31.071+0000 7f15cf400640 1 --2- 172.21.15.96:0/586654268 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f15d0152330 0x7f15d0151ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:31.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:31.071+0000 7f15cf400640 1 -- 172.21.15.96:0/586654268 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f15d00ccc40 msgr2=0x7f15d00cd0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:31.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:31.071+0000 7f15cf400640 1 --2- 172.21.15.96:0/586654268 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f15d00ccc40 0x7f15d00cd0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:31.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:31.071+0000 7f15cf400640 1 -- 172.21.15.96:0/586654268 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15d00b5090 con 0x7f15d00d16a0 2024-05-28T06:16:31.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:31.072+0000 7f15cf400640 1 --2- 172.21.15.96:0/586654268 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f15d00d16a0 0x7f15d00b95c0 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f15c00072f0 tx=0x7f15c00077b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:31.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:31.072+0000 7f15c7400640 1 -- 172.21.15.96:0/586654268 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15c001a070 con 0x7f15d00d16a0 2024-05-28T06:16:31.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:31.072+0000 7f15c7400640 1 -- 172.21.15.96:0/586654268 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15c000f8e0 con 0x7f15d00d16a0 2024-05-28T06:16:31.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:31.072+0000 7f15c7400640 1 -- 172.21.15.96:0/586654268 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15c000fcd0 con 0x7f15d00d16a0 2024-05-28T06:16:31.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:31.072+0000 7f15d5000640 1 -- 172.21.15.96:0/586654268 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15d00b9bc0 con 0x7f15d00d16a0 2024-05-28T06:16:31.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:31.072+0000 7f15d5000640 1 -- 172.21.15.96:0/586654268 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15d00b5330 con 0x7f15d00d16a0 2024-05-28T06:16:31.075 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:31.073+0000 7f15d5000640 1 -- 172.21.15.96:0/586654268 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15d0069190 con 0x7f15d00d16a0 2024-05-28T06:16:31.075 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:31.074+0000 7f15c7400640 1 -- 172.21.15.96:0/586654268 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 28) ==== 92570+0+0 (secure 0 0 0) 0x7f15c001d360 con 0x7f15d00d16a0 2024-05-28T06:16:31.075 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:31.076+0000 7f15c7400640 1 --2- 172.21.15.96:0/586654268 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f15b006d9d0 0x7f15b006fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:31.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:31.076+0000 7f15c7400640 1 -- 172.21.15.96:0/586654268 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5795+0+0 (secure 0 0 0) 0x7f15c008a130 con 0x7f15d00d16a0 2024-05-28T06:16:31.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:31.078+0000 7f15cea00640 1 --2- 172.21.15.96:0/586654268 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f15b006d9d0 0x7f15b006fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:31.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:31.081+0000 7f15c7400640 1 -- 172.21.15.96:0/586654268 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f15d0069190 con 0x7f15d00d16a0 2024-05-28T06:16:31.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:31.084+0000 7f15cea00640 1 --2- 172.21.15.96:0/586654268 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f15b006d9d0 0x7f15b006fe90 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f15bc034000 tx=0x7f15bc02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:31.417 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:31.417+0000 7f15d5000640 1 -- 172.21.15.96:0/586654268 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone43633", "target": ["mon-mgr", ""]}) -- 0x7f15d00689e0 con 0x7f15b006d9d0 2024-05-28T06:16:32.360 DEBUG:teuthology.orchestra.run.smithi096:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:16:32.531 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:16:32.416+0000 7fea59e00640 -1 mgr.server reply reply (22) Invalid argument cannot cancel -- clone finished (check clone status) 2024-05-28T06:16:32.531 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-05-28T06:16:32.532 INFO:teuthology.orchestra.run.smithi096.stderr:Error EINVAL: cannot cancel -- clone finished (check clone status) 2024-05-28T06:16:32.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:32.417+0000 7f15c7400640 1 -- 172.21.15.96:0/586654268 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: -22 cannot cancel -- clone finished (check clone status)) ==== 60+0+0 (secure 0 0 0) 0x7f15d00689e0 con 0x7f15b006d9d0 2024-05-28T06:16:32.532 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone43634 2024-05-28T06:16:32.577 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:16:32.818 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:32.818+0000 7f91e0600640 1 Processor -- start 2024-05-28T06:16:32.818 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:32.818+0000 7f91e0600640 1 -- start start 2024-05-28T06:16:32.819 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:32.819+0000 7f91e0600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f91d80ce6e0 0x7f91d80d0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:32.819 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:32.819+0000 7f91e0600640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91d80d16c0 con 0x7f91d815e940 2024-05-28T06:16:32.820 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:32.819+0000 7f91e0600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91d80ccbf0 con 0x7f91d80d11e0 2024-05-28T06:16:32.820 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:32.819+0000 7f91e0600640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f91d80cd240 con 0x7f91d80ce6e0 2024-05-28T06:16:32.820 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:32.820+0000 7f91dd200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f91d80ce6e0 0x7f91d80d0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:32.820 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:32.820+0000 7f91dd200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f91d80ce6e0 0x7f91d80d0ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:45808/0 (socket says 172.21.15.96:45808) 2024-05-28T06:16:32.820 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:32.820+0000 7f91dd200640 1 -- 172.21.15.96:0/968827621 learned_addr learned my addr 172.21.15.96:0/968827621 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:16:32.820 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:32.820+0000 7f91ddc00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f91d815e940 0x7f91d80d0590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:51466/0 (socket says 172.21.15.96:51466) 2024-05-28T06:16:32.820 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:32.820+0000 7f91dd200640 1 -- 172.21.15.96:0/968827621 >> v1:172.21.15.123:6789/0 conn(0x7f91d80d11e0 legacy=0x7f91d809aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:16:32.820 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:32.820+0000 7f91dd200640 1 -- 172.21.15.96:0/968827621 >> v1:172.21.15.96:6789/0 conn(0x7f91d815e940 legacy=0x7f91d80d0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:32.821 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:32.820+0000 7f91dd200640 1 -- 172.21.15.96:0/968827621 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91d80cced0 con 0x7f91d80ce6e0 2024-05-28T06:16:32.822 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:32.822+0000 7f91dd200640 1 --2- 172.21.15.96:0/968827621 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f91d80ce6e0 0x7f91d80d0ca0 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f91c4009870 tx=0x7f91c402a2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a56687d1c7262de8 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:32.822 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:32.823+0000 7f91d7e00640 1 -- 172.21.15.96:0/968827621 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91c4037070 con 0x7f91d80ce6e0 2024-05-28T06:16:32.823 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:32.823+0000 7f91d7e00640 1 -- 172.21.15.96:0/968827621 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91c4032440 con 0x7f91d80ce6e0 2024-05-28T06:16:32.823 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:32.823+0000 7f91d7e00640 1 -- 172.21.15.96:0/968827621 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91c4032810 con 0x7f91d80ce6e0 2024-05-28T06:16:32.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:32.824+0000 7f91e0600640 1 -- 172.21.15.96:0/968827621 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f91d80ce6e0 msgr2=0x7f91d80d0ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:32.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:32.824+0000 7f91e0600640 1 --2- 172.21.15.96:0/968827621 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f91d80ce6e0 0x7f91d80d0ca0 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f91c4009870 tx=0x7f91c402a2a0 comp rx=0 tx=0).stop 2024-05-28T06:16:32.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:32.824+0000 7f91e0600640 1 -- 172.21.15.96:0/968827621 shutdown_connections 2024-05-28T06:16:32.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:32.824+0000 7f91e0600640 1 --2- 172.21.15.96:0/968827621 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f91d80ce6e0 0x7f91d80d0ca0 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:32.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:32.824+0000 7f91e0600640 1 -- 172.21.15.96:0/968827621 >> 172.21.15.96:0/968827621 conn(0x7f91d80b3f60 msgr2=0x7f91d80b4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:16:32.825 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:32.824+0000 7f91e0600640 1 -- 172.21.15.96:0/968827621 shutdown_connections 2024-05-28T06:16:32.825 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:32.824+0000 7f91e0600640 1 -- 172.21.15.96:0/968827621 wait complete. 2024-05-28T06:16:32.827 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:32.827+0000 7f91e0600640 1 Processor -- start 2024-05-28T06:16:32.827 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:32.828+0000 7f91e0600640 1 -- start start 2024-05-28T06:16:32.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:32.829+0000 7f91e0600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91d80ce6e0 0x7f91d80c6dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:32.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:32.829+0000 7f91e0600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f91d80d11e0 0x7f91d806d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:32.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:32.829+0000 7f91e0600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f91d815e940 0x7f91d806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:32.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:32.829+0000 7f91e0600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f91d815ed20 con 0x7f91d815e940 2024-05-28T06:16:32.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:32.829+0000 7f91e0600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f91d809b140 con 0x7f91d80ce6e0 2024-05-28T06:16:32.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:32.829+0000 7f91e0600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f91d8201be0 con 0x7f91d80d11e0 2024-05-28T06:16:32.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:32.829+0000 7f91de600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f91d815e940 0x7f91d806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:32.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:32.829+0000 7f91de600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f91d815e940 0x7f91d806dcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:45822/0 (socket says 172.21.15.96:45822) 2024-05-28T06:16:32.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:32.829+0000 7f91de600640 1 -- 172.21.15.96:0/297808813 learned_addr learned my addr 172.21.15.96:0/297808813 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:16:32.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:32.830+0000 7f91dd200640 1 --2- 172.21.15.96:0/297808813 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f91d80d11e0 0x7f91d806d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:32.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:32.830+0000 7f91ddc00640 1 --2- 172.21.15.96:0/297808813 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91d80ce6e0 0x7f91d80c6dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:32.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:32.830+0000 7f91de600640 1 -- 172.21.15.96:0/297808813 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f91d80d11e0 msgr2=0x7f91d806d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:32.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:32.830+0000 7f91de600640 1 --2- 172.21.15.96:0/297808813 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f91d80d11e0 0x7f91d806d790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:32.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:32.830+0000 7f91de600640 1 -- 172.21.15.96:0/297808813 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91d80ce6e0 msgr2=0x7f91d80c6dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:32.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:32.830+0000 7f91de600640 1 --2- 172.21.15.96:0/297808813 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91d80ce6e0 0x7f91d80c6dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:32.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:32.830+0000 7f91de600640 1 -- 172.21.15.96:0/297808813 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91d8201ec0 con 0x7f91d815e940 2024-05-28T06:16:32.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:32.830+0000 7f91ddc00640 1 --2- 172.21.15.96:0/297808813 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91d80ce6e0 0x7f91d80c6dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:16:32.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:32.830+0000 7f91de600640 1 --2- 172.21.15.96:0/297808813 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f91d815e940 0x7f91d806dcd0 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f91d000c780 tx=0x7f91d000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:32.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:32.831+0000 7f91d5600640 1 -- 172.21.15.96:0/297808813 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91d0016020 con 0x7f91d815e940 2024-05-28T06:16:32.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:32.831+0000 7f91d5600640 1 -- 172.21.15.96:0/297808813 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91d000aa50 con 0x7f91d815e940 2024-05-28T06:16:32.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:32.831+0000 7f91d5600640 1 -- 172.21.15.96:0/297808813 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91d000ae00 con 0x7f91d815e940 2024-05-28T06:16:32.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:32.831+0000 7f91e0600640 1 -- 172.21.15.96:0/297808813 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91d82021a0 con 0x7f91d815e940 2024-05-28T06:16:32.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:32.831+0000 7f91e0600640 1 -- 172.21.15.96:0/297808813 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91d80ca9b0 con 0x7f91d815e940 2024-05-28T06:16:32.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:32.833+0000 7f91e0600640 1 -- 172.21.15.96:0/297808813 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91d80689e0 con 0x7f91d815e940 2024-05-28T06:16:32.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:32.834+0000 7f91d5600640 1 -- 172.21.15.96:0/297808813 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 28) ==== 92570+0+0 (secure 0 0 0) 0x7f91d0019020 con 0x7f91d815e940 2024-05-28T06:16:32.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:32.836+0000 7f91d5600640 1 --2- 172.21.15.96:0/297808813 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f91b806da20 0x7f91b806fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:32.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:32.836+0000 7f91d5600640 1 -- 172.21.15.96:0/297808813 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5795+0+0 (secure 0 0 0) 0x7f91d008a690 con 0x7f91d815e940 2024-05-28T06:16:32.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:32.841+0000 7f91d5600640 1 -- 172.21.15.96:0/297808813 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f91d0020030 con 0x7f91d815e940 2024-05-28T06:16:32.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:32.854+0000 7f91ddc00640 1 --2- 172.21.15.96:0/297808813 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f91b806da20 0x7f91b806fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:32.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:32.889+0000 7f91ddc00640 1 --2- 172.21.15.96:0/297808813 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f91b806da20 0x7f91b806fee0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f91d8202400 tx=0x7f91cc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:33.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:33.181+0000 7f91e0600640 1 -- 172.21.15.96:0/297808813 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone43634", "target": ["mon-mgr", ""]}) -- 0x7f91d80d2250 con 0x7f91b806da20 2024-05-28T06:16:33.528 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:16:33.528+0000 7fea59e00640 -1 mgr.server reply reply (22) Invalid argument cannot cancel -- clone finished (check clone status) 2024-05-28T06:16:33.528 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:33.528+0000 7f91d5600640 1 -- 172.21.15.96:0/297808813 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: -22 cannot cancel -- clone finished (check clone status)) ==== 60+0+0 (secure 0 0 0) 0x7f91d80d2250 con 0x7f91b806da20 2024-05-28T06:16:33.529 INFO:teuthology.orchestra.run.smithi096.stderr:Error EINVAL: cannot cancel -- clone finished (check clone status) 2024-05-28T06:16:33.560 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-05-28T06:16:33.560 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone43635 2024-05-28T06:16:33.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:33.834+0000 7f57bb600640 1 Processor -- start 2024-05-28T06:16:33.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:33.834+0000 7f57bb600640 1 -- start start 2024-05-28T06:16:33.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:33.835+0000 7f57bb600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f57b40d0640 0x7f57b40d0a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:33.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:33.835+0000 7f57bb600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f57b40725e0 0x7f57b40ae650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:33.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:33.835+0000 7f57bb600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57b40ade90 con 0x7f57b415b6f0 2024-05-28T06:16:33.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:33.835+0000 7f57bb600640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f57b4071c40 con 0x7f57b40725e0 2024-05-28T06:16:33.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:33.835+0000 7f57bb600640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f57b40d0f60 con 0x7f57b40d0640 2024-05-28T06:16:33.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:33.836+0000 7f57b9600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f57b40725e0 0x7f57b40ae650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:33.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:33.836+0000 7f57b9600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f57b40725e0 0x7f57b40ae650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:45826/0 (socket says 172.21.15.96:45826) 2024-05-28T06:16:33.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:33.836+0000 7f57b9600640 1 -- 172.21.15.96:0/1139014527 learned_addr learned my addr 172.21.15.96:0/1139014527 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:16:33.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:33.836+0000 7f57b9600640 1 -- 172.21.15.96:0/1139014527 >> v2:172.21.15.123:3300/0 conn(0x7f57b40d0640 msgr2=0x7f57b40d0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:16:33.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:33.836+0000 7f57b9600640 1 --2- 172.21.15.96:0/1139014527 >> v2:172.21.15.123:3300/0 conn(0x7f57b40d0640 0x7f57b40d0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:33.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:33.836+0000 7f57b9600640 1 -- 172.21.15.96:0/1139014527 >> v1:172.21.15.123:6789/0 conn(0x7f57b415b6f0 legacy=0x7f57b406e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:16:33.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:33.836+0000 7f57b9600640 1 -- 172.21.15.96:0/1139014527 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57b4071e70 con 0x7f57b40725e0 2024-05-28T06:16:33.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:33.838+0000 7f57b9600640 1 --2- 172.21.15.96:0/1139014527 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f57b40725e0 0x7f57b40ae650 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f57b406fe40 tx=0x7f57ac02a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85ede30471f141b9 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:33.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:33.839+0000 7f57b3400640 1 -- 172.21.15.96:0/1139014527 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57ac036070 con 0x7f57b40725e0 2024-05-28T06:16:33.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:33.839+0000 7f57b3400640 1 -- 172.21.15.96:0/1139014527 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57ac031600 con 0x7f57b40725e0 2024-05-28T06:16:33.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:33.839+0000 7f57b3400640 1 -- 172.21.15.96:0/1139014527 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57ac0319d0 con 0x7f57b40725e0 2024-05-28T06:16:33.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:33.840+0000 7f57bb600640 1 -- 172.21.15.96:0/1139014527 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f57b40725e0 msgr2=0x7f57b40ae650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:33.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:33.840+0000 7f57bb600640 1 --2- 172.21.15.96:0/1139014527 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f57b40725e0 0x7f57b40ae650 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f57b406fe40 tx=0x7f57ac02a710 comp rx=0 tx=0).stop 2024-05-28T06:16:33.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:33.841+0000 7f57bb600640 1 -- 172.21.15.96:0/1139014527 shutdown_connections 2024-05-28T06:16:33.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:33.841+0000 7f57bb600640 1 --2- 172.21.15.96:0/1139014527 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f57b40725e0 0x7f57b40ae650 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:33.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:33.841+0000 7f57bb600640 1 --2- 172.21.15.96:0/1139014527 >> v2:172.21.15.123:3300/0 conn(0x7f57b40d0640 0x7f57b40d0a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:33.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:33.841+0000 7f57bb600640 1 -- 172.21.15.96:0/1139014527 >> 172.21.15.96:0/1139014527 conn(0x7f57b406bd20 msgr2=0x7f57b406c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:16:33.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:33.841+0000 7f57bb600640 1 -- 172.21.15.96:0/1139014527 shutdown_connections 2024-05-28T06:16:33.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:33.842+0000 7f57bb600640 1 -- 172.21.15.96:0/1139014527 wait complete. 2024-05-28T06:16:33.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:33.844+0000 7f57bb600640 1 Processor -- start 2024-05-28T06:16:33.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:33.844+0000 7f57bb600640 1 -- start start 2024-05-28T06:16:33.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:33.846+0000 7f57bb600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f57b40725e0 0x7f57b414a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:33.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:33.846+0000 7f57bb600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f57b40d0640 0x7f57b414d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:33.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:33.846+0000 7f57bb600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f57b415b6f0 0x7f57b40cd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:33.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:33.846+0000 7f57bb600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f57b40aeb90 con 0x7f57b415b6f0 2024-05-28T06:16:33.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:33.846+0000 7f57bb600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f57b40ae130 con 0x7f57b40725e0 2024-05-28T06:16:33.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:33.846+0000 7f57bb600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f57b40aed10 con 0x7f57b40d0640 2024-05-28T06:16:33.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:33.846+0000 7f57b9600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f57b415b6f0 0x7f57b40cd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:33.847 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:33.846+0000 7f57b9600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f57b415b6f0 0x7f57b40cd1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:45828/0 (socket says 172.21.15.96:45828) 2024-05-28T06:16:33.847 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:33.846+0000 7f57b9600640 1 -- 172.21.15.96:0/2940809470 learned_addr learned my addr 172.21.15.96:0/2940809470 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:16:33.847 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:33.847+0000 7f57b8c00640 1 --2- 172.21.15.96:0/2940809470 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f57b40725e0 0x7f57b414a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:33.847 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:33.847+0000 7f57b9600640 1 -- 172.21.15.96:0/2940809470 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f57b40d0640 msgr2=0x7f57b414d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:33.847 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:33.847+0000 7f57b9600640 1 --2- 172.21.15.96:0/2940809470 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f57b40d0640 0x7f57b414d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:33.847 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:33.847+0000 7f57b9600640 1 -- 172.21.15.96:0/2940809470 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f57b40725e0 msgr2=0x7f57b414a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:33.847 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:33.847+0000 7f57b9600640 1 --2- 172.21.15.96:0/2940809470 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f57b40725e0 0x7f57b414a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:33.847 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:33.847+0000 7f57b9600640 1 -- 172.21.15.96:0/2940809470 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57b40cf510 con 0x7f57b415b6f0 2024-05-28T06:16:33.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:33.848+0000 7f57b9600640 1 --2- 172.21.15.96:0/2940809470 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f57b415b6f0 0x7f57b40cd1c0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f57ac0093e0 tx=0x7f57ac004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:33.849 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:33.849+0000 7f57b0c00640 1 -- 172.21.15.96:0/2940809470 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57ac049020 con 0x7f57b415b6f0 2024-05-28T06:16:33.849 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:33.849+0000 7f57b0c00640 1 -- 172.21.15.96:0/2940809470 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57ac037040 con 0x7f57b415b6f0 2024-05-28T06:16:33.849 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:33.849+0000 7f57b0c00640 1 -- 172.21.15.96:0/2940809470 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57ac036070 con 0x7f57b415b6f0 2024-05-28T06:16:33.849 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:33.849+0000 7f57bb600640 1 -- 172.21.15.96:0/2940809470 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57b40cf7f0 con 0x7f57b415b6f0 2024-05-28T06:16:33.849 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:33.850+0000 7f57bb600640 1 -- 172.21.15.96:0/2940809470 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57b40cf030 con 0x7f57b415b6f0 2024-05-28T06:16:33.852 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:33.851+0000 7f57bb600640 1 -- 172.21.15.96:0/2940809470 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57b40cfd10 con 0x7f57b415b6f0 2024-05-28T06:16:33.854 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:33.854+0000 7f57b0c00640 1 -- 172.21.15.96:0/2940809470 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 28) ==== 92570+0+0 (secure 0 0 0) 0x7f57ac04c020 con 0x7f57b415b6f0 2024-05-28T06:16:33.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:33.856+0000 7f57b0c00640 1 --2- 172.21.15.96:0/2940809470 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f579406da20 0x7f579406fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:33.857 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:33.857+0000 7f57b0c00640 1 -- 172.21.15.96:0/2940809470 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5795+0+0 (secure 0 0 0) 0x7f57ac0b37d0 con 0x7f57b415b6f0 2024-05-28T06:16:33.866 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:33.867+0000 7f57b0c00640 1 -- 172.21.15.96:0/2940809470 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f57b40cfd10 con 0x7f57b415b6f0 2024-05-28T06:16:33.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:33.883+0000 7f57b8c00640 1 --2- 172.21.15.96:0/2940809470 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f579406da20 0x7f579406fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:33.905 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:33.905+0000 7f57b8c00640 1 --2- 172.21.15.96:0/2940809470 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f579406da20 0x7f579406fee0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f57b40b1620 tx=0x7f57a8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:34.281 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:34.281+0000 7f57bb600640 1 -- 172.21.15.96:0/2940809470 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone43635", "target": ["mon-mgr", ""]}) -- 0x7f57b40cf2c0 con 0x7f579406da20 2024-05-28T06:16:34.631 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:16:34.631+0000 7fea59e00640 -1 mgr.server reply reply (22) Invalid argument cannot cancel -- clone finished (check clone status) 2024-05-28T06:16:34.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:34.633+0000 7f57b0c00640 1 -- 172.21.15.96:0/2940809470 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: -22 cannot cancel -- clone finished (check clone status)) ==== 60+0+0 (secure 0 0 0) 0x7f57b40cf2c0 con 0x7f579406da20 2024-05-28T06:16:34.633 INFO:teuthology.orchestra.run.smithi096.stderr:Error EINVAL: cannot cancel -- clone finished (check clone status) 2024-05-28T06:16:34.653 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-05-28T06:16:34.654 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone43636 2024-05-28T06:16:34.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:34.901+0000 7fda3ec00640 1 Processor -- start 2024-05-28T06:16:34.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:34.901+0000 7fda3ec00640 1 -- start start 2024-05-28T06:16:34.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:34.902+0000 7fda3ec00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fda380725e0 0x7fda38166050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:34.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:34.902+0000 7fda3ec00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda380ade90 con 0x7fda380d0640 2024-05-28T06:16:34.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:34.902+0000 7fda3ec00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda38071e70 con 0x7fda3815b6f0 2024-05-28T06:16:34.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:34.902+0000 7fda3ec00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fda38071c40 con 0x7fda380725e0 2024-05-28T06:16:34.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:34.903+0000 7fda3cc00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fda380725e0 0x7fda38166050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:34.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:34.903+0000 7fda3cc00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fda380725e0 0x7fda38166050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:45844/0 (socket says 172.21.15.96:45844) 2024-05-28T06:16:34.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:34.903+0000 7fda3cc00640 1 -- 172.21.15.96:0/1238770234 learned_addr learned my addr 172.21.15.96:0/1238770234 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:16:34.904 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:34.903+0000 7fda3cc00640 1 -- 172.21.15.96:0/1238770234 >> v1:172.21.15.123:6789/0 conn(0x7fda3815b6f0 legacy=0x7fda3806e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:16:34.904 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:34.903+0000 7fda3cc00640 1 -- 172.21.15.96:0/1238770234 >> v1:172.21.15.96:6789/0 conn(0x7fda380d0640 legacy=0x7fda380d0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:34.904 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:34.903+0000 7fda3cc00640 1 -- 172.21.15.96:0/1238770234 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda38072130 con 0x7fda380725e0 2024-05-28T06:16:34.904 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:34.904+0000 7fda3cc00640 1 --2- 172.21.15.96:0/1238770234 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fda380725e0 0x7fda38166050 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7fda30009e50 tx=0x7fda3002a760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10ea8cc969d821c0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:34.905 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:34.905+0000 7fda36a00640 1 -- 172.21.15.96:0/1238770234 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda30036070 con 0x7fda380725e0 2024-05-28T06:16:34.905 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:34.905+0000 7fda36a00640 1 -- 172.21.15.96:0/1238770234 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda30031600 con 0x7fda380725e0 2024-05-28T06:16:34.905 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:34.905+0000 7fda36a00640 1 -- 172.21.15.96:0/1238770234 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda300319d0 con 0x7fda380725e0 2024-05-28T06:16:34.906 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:34.905+0000 7fda3ec00640 1 -- 172.21.15.96:0/1238770234 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fda380725e0 msgr2=0x7fda38166050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:34.906 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:34.905+0000 7fda3ec00640 1 --2- 172.21.15.96:0/1238770234 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fda380725e0 0x7fda38166050 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7fda30009e50 tx=0x7fda3002a760 comp rx=0 tx=0).stop 2024-05-28T06:16:34.906 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:34.906+0000 7fda3ec00640 1 -- 172.21.15.96:0/1238770234 shutdown_connections 2024-05-28T06:16:34.906 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:34.906+0000 7fda3ec00640 1 --2- 172.21.15.96:0/1238770234 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fda380725e0 0x7fda38166050 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:34.906 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:34.906+0000 7fda3ec00640 1 -- 172.21.15.96:0/1238770234 >> 172.21.15.96:0/1238770234 conn(0x7fda3806bd20 msgr2=0x7fda3806c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:16:34.906 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:34.906+0000 7fda3ec00640 1 -- 172.21.15.96:0/1238770234 shutdown_connections 2024-05-28T06:16:34.906 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:34.906+0000 7fda3ec00640 1 -- 172.21.15.96:0/1238770234 wait complete. 2024-05-28T06:16:34.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:34.908+0000 7fda3ec00640 1 Processor -- start 2024-05-28T06:16:34.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:34.908+0000 7fda3ec00640 1 -- start start 2024-05-28T06:16:34.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:34.909+0000 7fda3ec00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fda380725e0 0x7fda380cb6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:34.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:34.909+0000 7fda3ec00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fda380d0640 0x7fda3814f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:34.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:34.909+0000 7fda3ec00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fda3815b6f0 0x7fda381531f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:34.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:34.909+0000 7fda3ec00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fda380af7e0 con 0x7fda380d0640 2024-05-28T06:16:34.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:34.909+0000 7fda3ec00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fda380af660 con 0x7fda3815b6f0 2024-05-28T06:16:34.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:34.909+0000 7fda3ec00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fda380af960 con 0x7fda380725e0 2024-05-28T06:16:34.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:34.909+0000 7fda37400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fda380d0640 0x7fda3814f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:34.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:34.909+0000 7fda37400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fda380d0640 0x7fda3814f290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:45858/0 (socket says 172.21.15.96:45858) 2024-05-28T06:16:34.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:34.909+0000 7fda37400640 1 -- 172.21.15.96:0/2643839024 learned_addr learned my addr 172.21.15.96:0/2643839024 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:16:34.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:34.910+0000 7fda37e00640 1 --2- 172.21.15.96:0/2643839024 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fda380725e0 0x7fda380cb6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:34.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:34.910+0000 7fda37400640 1 -- 172.21.15.96:0/2643839024 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fda380725e0 msgr2=0x7fda380cb6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:34.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:34.910+0000 7fda37400640 1 --2- 172.21.15.96:0/2643839024 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fda380725e0 0x7fda380cb6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:34.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:34.910+0000 7fda3cc00640 1 --2- 172.21.15.96:0/2643839024 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fda3815b6f0 0x7fda381531f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:34.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:34.910+0000 7fda37400640 1 -- 172.21.15.96:0/2643839024 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fda3815b6f0 msgr2=0x7fda381531f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:34.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:34.910+0000 7fda37400640 1 --2- 172.21.15.96:0/2643839024 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fda3815b6f0 0x7fda381531f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:34.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:34.910+0000 7fda37400640 1 -- 172.21.15.96:0/2643839024 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda380cab10 con 0x7fda380d0640 2024-05-28T06:16:34.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:34.910+0000 7fda37e00640 1 --2- 172.21.15.96:0/2643839024 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fda380725e0 0x7fda380cb6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:16:34.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:34.910+0000 7fda3cc00640 1 --2- 172.21.15.96:0/2643839024 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fda3815b6f0 0x7fda381531f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:16:34.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:34.911+0000 7fda37400640 1 --2- 172.21.15.96:0/2643839024 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fda380d0640 0x7fda3814f290 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7fda2400b410 tx=0x7fda2400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:34.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:34.911+0000 7fda2fe00640 1 -- 172.21.15.96:0/2643839024 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda24015020 con 0x7fda380d0640 2024-05-28T06:16:34.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:34.911+0000 7fda2fe00640 1 -- 172.21.15.96:0/2643839024 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda24011830 con 0x7fda380d0640 2024-05-28T06:16:34.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:34.911+0000 7fda3ec00640 1 -- 172.21.15.96:0/2643839024 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda38150130 con 0x7fda380d0640 2024-05-28T06:16:34.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:34.911+0000 7fda2fe00640 1 -- 172.21.15.96:0/2643839024 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda24011c20 con 0x7fda380d0640 2024-05-28T06:16:34.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:34.911+0000 7fda3ec00640 1 -- 172.21.15.96:0/2643839024 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda381506f0 con 0x7fda380d0640 2024-05-28T06:16:34.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:34.913+0000 7fda3ec00640 1 -- 172.21.15.96:0/2643839024 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda380afae0 con 0x7fda380d0640 2024-05-28T06:16:34.917 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:34.915+0000 7fda2fe00640 1 -- 172.21.15.96:0/2643839024 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 28) ==== 92570+0+0 (secure 0 0 0) 0x7fda24018020 con 0x7fda380d0640 2024-05-28T06:16:34.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:34.916+0000 7fda2fe00640 1 --2- 172.21.15.96:0/2643839024 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fda1806d9d0 0x7fda1806fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:34.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:34.916+0000 7fda2fe00640 1 -- 172.21.15.96:0/2643839024 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5795+0+0 (secure 0 0 0) 0x7fda2408a710 con 0x7fda380d0640 2024-05-28T06:16:34.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:34.918+0000 7fda37e00640 1 --2- 172.21.15.96:0/2643839024 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fda1806d9d0 0x7fda1806fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:34.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:34.925+0000 7fda2fe00640 1 -- 172.21.15.96:0/2643839024 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fda380afae0 con 0x7fda380d0640 2024-05-28T06:16:34.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:34.934+0000 7fda37e00640 1 --2- 172.21.15.96:0/2643839024 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fda1806d9d0 0x7fda1806fe90 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fda28001680 tx=0x7fda28009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:35.262 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:35.263+0000 7fda3ec00640 1 -- 172.21.15.96:0/2643839024 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone43636", "target": ["mon-mgr", ""]}) -- 0x7fda380689e0 con 0x7fda1806d9d0 2024-05-28T06:16:35.604 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:16:35.604+0000 7fea59e00640 -1 mgr.server reply reply (22) Invalid argument cannot cancel -- clone finished (check clone status) 2024-05-28T06:16:35.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:35.609+0000 7fda2fe00640 1 -- 172.21.15.96:0/2643839024 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: -22 cannot cancel -- clone finished (check clone status)) ==== 60+0+0 (secure 0 0 0) 0x7fda380689e0 con 0x7fda1806d9d0 2024-05-28T06:16:35.609 INFO:teuthology.orchestra.run.smithi096.stderr:Error EINVAL: cannot cancel -- clone finished (check clone status) 2024-05-28T06:16:35.651 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-05-28T06:16:35.651 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone43637 2024-05-28T06:16:35.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:35.971+0000 7f1052c00640 1 Processor -- start 2024-05-28T06:16:35.973 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:35.971+0000 7f1052c00640 1 -- start start 2024-05-28T06:16:35.974 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:35.974+0000 7f1052c00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f104c0b4120 0x7f104c163ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:35.974 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:35.974+0000 7f1052c00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f104c1640e0 con 0x7f104c162ac0 2024-05-28T06:16:35.974 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:35.974+0000 7f1052c00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f104c0b3360 con 0x7f104c0d0d50 2024-05-28T06:16:35.974 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:35.974+0000 7f1052c00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f104c0b3c90 con 0x7f104c0b4120 2024-05-28T06:16:35.974 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:35.974+0000 7f1050c00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f104c0b4120 0x7f104c163ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:35.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:35.974+0000 7f1050c00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f104c0b4120 0x7f104c163ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:45872/0 (socket says 172.21.15.96:45872) 2024-05-28T06:16:35.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:35.975+0000 7f1050c00640 1 -- 172.21.15.96:0/3526055019 learned_addr learned my addr 172.21.15.96:0/3526055019 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:16:35.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:35.975+0000 7f104be00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f104c162ac0 0x7f104c0d0640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:51482/0 (socket says 172.21.15.96:51482) 2024-05-28T06:16:35.976 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:35.976+0000 7f1050c00640 1 -- 172.21.15.96:0/3526055019 >> v1:172.21.15.123:6789/0 conn(0x7f104c0d0d50 legacy=0x7f104c1633b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:16:35.976 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:35.976+0000 7f1050c00640 1 -- 172.21.15.96:0/3526055019 >> v1:172.21.15.96:6789/0 conn(0x7f104c162ac0 legacy=0x7f104c0d0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:35.976 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:35.976+0000 7f1050c00640 1 -- 172.21.15.96:0/3526055019 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f104c0b3640 con 0x7f104c0b4120 2024-05-28T06:16:35.976 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:35.976+0000 7f104aa00640 1 -- 172.21.15.96:0/3526055019 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 46802947 0 0) 0x7f104c1640e0 con 0x7f104c162ac0 2024-05-28T06:16:35.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:35.977+0000 7f1050c00640 1 --2- 172.21.15.96:0/3526055019 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f104c0b4120 0x7f104c163ac0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f1044009e50 tx=0x7f104402a760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65112ec601e88726 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:35.979 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:35.979+0000 7f104aa00640 1 -- 172.21.15.96:0/3526055019 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1044036070 con 0x7f104c0b4120 2024-05-28T06:16:35.979 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:35.979+0000 7f104aa00640 1 -- 172.21.15.96:0/3526055019 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1044031600 con 0x7f104c0b4120 2024-05-28T06:16:35.979 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:35.979+0000 7f104aa00640 1 -- 172.21.15.96:0/3526055019 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10440319d0 con 0x7f104c0b4120 2024-05-28T06:16:35.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:35.981+0000 7f1052c00640 1 -- 172.21.15.96:0/3526055019 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f104c0b4120 msgr2=0x7f104c163ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:35.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:35.981+0000 7f1052c00640 1 --2- 172.21.15.96:0/3526055019 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f104c0b4120 0x7f104c163ac0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f1044009e50 tx=0x7f104402a760 comp rx=0 tx=0).stop 2024-05-28T06:16:35.982 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:35.983+0000 7f1052c00640 1 -- 172.21.15.96:0/3526055019 shutdown_connections 2024-05-28T06:16:35.982 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:35.983+0000 7f1052c00640 1 --2- 172.21.15.96:0/3526055019 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f104c0b4120 0x7f104c163ac0 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:35.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:35.983+0000 7f1052c00640 1 -- 172.21.15.96:0/3526055019 >> 172.21.15.96:0/3526055019 conn(0x7f104c06bd20 msgr2=0x7f104c06c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:16:35.986 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:35.987+0000 7f1052c00640 1 -- 172.21.15.96:0/3526055019 shutdown_connections 2024-05-28T06:16:35.988 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:35.987+0000 7f1052c00640 1 -- 172.21.15.96:0/3526055019 wait complete. 2024-05-28T06:16:35.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:35.991+0000 7f1052c00640 1 Processor -- start 2024-05-28T06:16:35.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:35.992+0000 7f1052c00640 1 -- start start 2024-05-28T06:16:35.995 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:35.994+0000 7f1052c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f104c0b4120 0x7f104c0aea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:35.998 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:35.998+0000 7f1052c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f104c0d0d50 0x7f104c15f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:35.999 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:35.998+0000 7f1052c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f104c162ac0 0x7f104c1f8a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:35.999 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:35.998+0000 7f1052c00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f104c163020 con 0x7f104c0d0d50 2024-05-28T06:16:35.999 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:35.998+0000 7f1052c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f104c162ea0 con 0x7f104c0b4120 2024-05-28T06:16:35.999 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:35.998+0000 7f1052c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f104c16d8d0 con 0x7f104c162ac0 2024-05-28T06:16:35.999 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:35.999+0000 7f104b400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f104c0d0d50 0x7f104c15f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:35.999 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:35.999+0000 7f104b400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f104c0d0d50 0x7f104c15f850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:45884/0 (socket says 172.21.15.96:45884) 2024-05-28T06:16:35.999 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:35.999+0000 7f104b400640 1 -- 172.21.15.96:0/2661463311 learned_addr learned my addr 172.21.15.96:0/2661463311 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:16:36.000 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:35.999+0000 7f104be00640 1 --2- 172.21.15.96:0/2661463311 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f104c0b4120 0x7f104c0aea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:36.000 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:35.999+0000 7f104b400640 1 -- 172.21.15.96:0/2661463311 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f104c162ac0 msgr2=0x7f104c1f8a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:36.000 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:35.999+0000 7f104b400640 1 --2- 172.21.15.96:0/2661463311 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f104c162ac0 0x7f104c1f8a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:36.000 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:35.999+0000 7f104b400640 1 -- 172.21.15.96:0/2661463311 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f104c0b4120 msgr2=0x7f104c0aea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:36.000 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:36.000+0000 7f104b400640 1 --2- 172.21.15.96:0/2661463311 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f104c0b4120 0x7f104c0aea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:36.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:36.000+0000 7f104b400640 1 -- 172.21.15.96:0/2661463311 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f104c15fd90 con 0x7f104c0d0d50 2024-05-28T06:16:36.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:36.001+0000 7f104b400640 1 --2- 172.21.15.96:0/2661463311 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f104c0d0d50 0x7f104c15f850 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f103800c390 tx=0x7f103800c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:36.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:36.001+0000 7f1043e00640 1 -- 172.21.15.96:0/2661463311 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1038017070 con 0x7f104c0d0d50 2024-05-28T06:16:36.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:36.001+0000 7f1043e00640 1 -- 172.21.15.96:0/2661463311 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10380035f0 con 0x7f104c0d0d50 2024-05-28T06:16:36.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:36.001+0000 7f1043e00640 1 -- 172.21.15.96:0/2661463311 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1038003a20 con 0x7f104c0d0d50 2024-05-28T06:16:36.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:36.002+0000 7f1052c00640 1 -- 172.21.15.96:0/2661463311 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f104c09a790 con 0x7f104c0d0d50 2024-05-28T06:16:36.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:36.002+0000 7f1052c00640 1 -- 172.21.15.96:0/2661463311 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f104c09ad50 con 0x7f104c0d0d50 2024-05-28T06:16:36.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:36.004+0000 7f104be00640 1 --2- 172.21.15.96:0/2661463311 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f104c0b4120 0x7f104c0aea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:16:36.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:36.004+0000 7f1052c00640 1 -- 172.21.15.96:0/2661463311 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f104c0689e0 con 0x7f104c0d0d50 2024-05-28T06:16:36.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:36.006+0000 7f1043e00640 1 -- 172.21.15.96:0/2661463311 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 28) ==== 92570+0+0 (secure 0 0 0) 0x7f103801a070 con 0x7f104c0d0d50 2024-05-28T06:16:36.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:36.007+0000 7f1043e00640 1 --2- 172.21.15.96:0/2661463311 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f102c06da20 0x7f102c06fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:36.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:36.008+0000 7f1043e00640 1 -- 172.21.15.96:0/2661463311 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5795+0+0 (secure 0 0 0) 0x7f1038089ed0 con 0x7f104c0d0d50 2024-05-28T06:16:36.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:36.014+0000 7f1043e00640 1 -- 172.21.15.96:0/2661463311 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f104c15fd90 con 0x7f104c0d0d50 2024-05-28T06:16:36.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:36.052+0000 7f104be00640 1 --2- 172.21.15.96:0/2661463311 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f102c06da20 0x7f102c06fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:36.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:36.098+0000 7f104be00640 1 --2- 172.21.15.96:0/2661463311 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f102c06da20 0x7f102c06fee0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f103c000f60 tx=0x7f103c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:36.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:36.367+0000 7f1052c00640 1 -- 172.21.15.96:0/2661463311 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone43637", "target": ["mon-mgr", ""]}) -- 0x7f104c0b02a0 con 0x7f102c06da20 2024-05-28T06:16:36.883 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:16:36.882+0000 7fea59e00640 -1 mgr.server reply reply (22) Invalid argument cannot cancel -- clone finished (check clone status) 2024-05-28T06:16:36.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:36.888+0000 7f1043e00640 1 -- 172.21.15.96:0/2661463311 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: -22 cannot cancel -- clone finished (check clone status)) ==== 60+0+0 (secure 0 0 0) 0x7f104c0b02a0 con 0x7f102c06da20 2024-05-28T06:16:36.888 INFO:teuthology.orchestra.run.smithi096.stderr:Error EINVAL: cannot cancel -- clone finished (check clone status) 2024-05-28T06:16:36.909 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-05-28T06:16:36.910 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone43638 2024-05-28T06:16:37.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:37.188+0000 7fa1ab200640 1 Processor -- start 2024-05-28T06:16:37.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:37.188+0000 7fa1ab200640 1 -- start start 2024-05-28T06:16:37.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:37.190+0000 7fa1ab200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fa1a406f7e0 0x7fa1a40725c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:37.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:37.190+0000 7fa1ab200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1a406e2e0 con 0x7fa1a406f380 2024-05-28T06:16:37.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:37.190+0000 7fa1ab200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1a406e5c0 con 0x7fa1a415abe0 2024-05-28T06:16:37.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:37.190+0000 7fa1ab200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fa1a40d0d50 con 0x7fa1a406f7e0 2024-05-28T06:16:37.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:37.192+0000 7fa19be00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fa1a406f380 0x7fa1a40b45d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:51492/0 (socket says 172.21.15.96:51492) 2024-05-28T06:16:37.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:37.192+0000 7fa19be00640 1 -- 172.21.15.96:0/3541040630 learned_addr learned my addr 172.21.15.96:0/3541040630 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:16:37.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:37.192+0000 7fa1a9200640 1 --2- 172.21.15.96:0/3541040630 >> v2:172.21.15.123:3300/0 conn(0x7fa1a406f7e0 0x7fa1a40725c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:37.192 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:37.192+0000 7fa1a3400640 1 -- 172.21.15.96:0/3541040630 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2717551758 0 0) 0x7fa1a406e2e0 con 0x7fa1a406f380 2024-05-28T06:16:37.192 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:37.192+0000 7fa1a3400640 1 -- 172.21.15.96:0/3541040630 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa188003520 con 0x7fa1a406f380 2024-05-28T06:16:37.195 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:37.192+0000 7fa1a9200640 1 -- 172.21.15.96:0/3541040630 >> v1:172.21.15.123:6789/0 conn(0x7fa1a415abe0 legacy=0x7fa1a40d0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:37.196 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:37.192+0000 7fa1a9200640 1 -- 172.21.15.96:0/3541040630 >> v1:172.21.15.96:6789/0 conn(0x7fa1a406f380 legacy=0x7fa1a40b45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:37.196 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:37.193+0000 7fa1a9200640 1 -- 172.21.15.96:0/3541040630 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1a406e8a0 con 0x7fa1a406f7e0 2024-05-28T06:16:37.198 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:37.198+0000 7fa1a9200640 1 --2- 172.21.15.96:0/3541040630 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa1a406f7e0 0x7fa1a40725c0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fa19c009d90 tx=0x7fa19c02a6f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ea27b4c981012111 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:37.200 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:37.200+0000 7fa1a3400640 1 -- 172.21.15.96:0/3541040630 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa19c007740 con 0x7fa1a406f7e0 2024-05-28T06:16:37.200 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:37.200+0000 7fa1a3400640 1 -- 172.21.15.96:0/3541040630 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa19c0078e0 con 0x7fa1a406f7e0 2024-05-28T06:16:37.200 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:37.200+0000 7fa1a3400640 1 -- 172.21.15.96:0/3541040630 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa19c007cb0 con 0x7fa1a406f7e0 2024-05-28T06:16:37.201 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:37.201+0000 7fa1ab200640 1 -- 172.21.15.96:0/3541040630 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa1a406f7e0 msgr2=0x7fa1a40725c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:37.201 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:37.201+0000 7fa1ab200640 1 --2- 172.21.15.96:0/3541040630 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa1a406f7e0 0x7fa1a40725c0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fa19c009d90 tx=0x7fa19c02a6f0 comp rx=0 tx=0).stop 2024-05-28T06:16:37.201 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:37.202+0000 7fa1ab200640 1 -- 172.21.15.96:0/3541040630 shutdown_connections 2024-05-28T06:16:37.202 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:37.202+0000 7fa1ab200640 1 --2- 172.21.15.96:0/3541040630 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa1a406f7e0 0x7fa1a40725c0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:37.202 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:37.202+0000 7fa1ab200640 1 -- 172.21.15.96:0/3541040630 >> 172.21.15.96:0/3541040630 conn(0x7fa1a406bd20 msgr2=0x7fa1a406c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:16:37.202 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:37.202+0000 7fa1ab200640 1 -- 172.21.15.96:0/3541040630 shutdown_connections 2024-05-28T06:16:37.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:37.202+0000 7fa1ab200640 1 -- 172.21.15.96:0/3541040630 wait complete. 2024-05-28T06:16:37.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:37.205+0000 7fa1ab200640 1 Processor -- start 2024-05-28T06:16:37.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:37.208+0000 7fa1ab200640 1 -- start start 2024-05-28T06:16:37.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:37.209+0000 7fa1ab200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa1a406f380 0x7fa1a415d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:37.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:37.209+0000 7fa1ab200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa1a406f7e0 0x7fa1a414df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:37.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:37.209+0000 7fa1ab200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa1a415abe0 0x7fa1a409a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:37.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:37.209+0000 7fa1ab200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa1a415afc0 con 0x7fa1a415abe0 2024-05-28T06:16:37.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:37.209+0000 7fa1ab200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa1a406ee60 con 0x7fa1a406f380 2024-05-28T06:16:37.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:37.209+0000 7fa1ab200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fa1a40b4ce0 con 0x7fa1a406f7e0 2024-05-28T06:16:37.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:37.209+0000 7fa1a9200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa1a415abe0 0x7fa1a409a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:37.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:37.209+0000 7fa1a9200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa1a415abe0 0x7fa1a409a3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:45890/0 (socket says 172.21.15.96:45890) 2024-05-28T06:16:37.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:37.209+0000 7fa1a9200640 1 -- 172.21.15.96:0/3871426847 learned_addr learned my addr 172.21.15.96:0/3871426847 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:16:37.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:37.209+0000 7fa1a3e00640 1 --2- 172.21.15.96:0/3871426847 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa1a406f380 0x7fa1a415d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:37.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:37.209+0000 7fa19be00640 1 --2- 172.21.15.96:0/3871426847 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa1a406f7e0 0x7fa1a414df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:37.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:37.210+0000 7fa1a9200640 1 -- 172.21.15.96:0/3871426847 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa1a406f7e0 msgr2=0x7fa1a414df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:37.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:37.210+0000 7fa1a9200640 1 --2- 172.21.15.96:0/3871426847 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa1a406f7e0 0x7fa1a414df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:37.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:37.210+0000 7fa1a9200640 1 -- 172.21.15.96:0/3871426847 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa1a406f380 msgr2=0x7fa1a415d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:37.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:37.210+0000 7fa1a9200640 1 --2- 172.21.15.96:0/3871426847 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa1a406f380 0x7fa1a415d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:37.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:37.210+0000 7fa1a9200640 1 -- 172.21.15.96:0/3871426847 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1a414e930 con 0x7fa1a415abe0 2024-05-28T06:16:37.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:37.210+0000 7fa1a9200640 1 --2- 172.21.15.96:0/3871426847 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa1a415abe0 0x7fa1a409a3b0 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7fa19c02ace0 tx=0x7fa19c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:37.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:37.210+0000 7fa1a0c00640 1 -- 172.21.15.96:0/3871426847 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa19c040070 con 0x7fa1a415abe0 2024-05-28T06:16:37.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:37.211+0000 7fa1a0c00640 1 -- 172.21.15.96:0/3871426847 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa19c031b40 con 0x7fa1a415abe0 2024-05-28T06:16:37.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:37.211+0000 7fa1a0c00640 1 -- 172.21.15.96:0/3871426847 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa19c045460 con 0x7fa1a415abe0 2024-05-28T06:16:37.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:37.211+0000 7fa1ab200640 1 -- 172.21.15.96:0/3871426847 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1a414ec10 con 0x7fa1a415abe0 2024-05-28T06:16:37.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:37.211+0000 7fa1ab200640 1 -- 172.21.15.96:0/3871426847 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1a414fb70 con 0x7fa1a415abe0 2024-05-28T06:16:37.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:37.212+0000 7fa1ab200640 1 -- 172.21.15.96:0/3871426847 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1a40689e0 con 0x7fa1a415abe0 2024-05-28T06:16:37.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:37.213+0000 7fa1a0c00640 1 -- 172.21.15.96:0/3871426847 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 28) ==== 92570+0+0 (secure 0 0 0) 0x7fa19c039040 con 0x7fa1a415abe0 2024-05-28T06:16:37.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:37.215+0000 7fa1a0c00640 1 --2- 172.21.15.96:0/3871426847 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa18006d9d0 0x7fa18006fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:37.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:37.215+0000 7fa1a0c00640 1 -- 172.21.15.96:0/3871426847 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5795+0+0 (secure 0 0 0) 0x7fa19c0abd50 con 0x7fa1a415abe0 2024-05-28T06:16:37.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:37.221+0000 7fa1a0c00640 1 -- 172.21.15.96:0/3871426847 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa19c07b5e0 con 0x7fa1a415abe0 2024-05-28T06:16:37.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:37.243+0000 7fa1a3e00640 1 --2- 172.21.15.96:0/3871426847 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa18006d9d0 0x7fa18006fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:37.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:37.252+0000 7fa1a3e00640 1 --2- 172.21.15.96:0/3871426847 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa18006d9d0 0x7fa18006fe90 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fa1a406f2a0 tx=0x7fa194009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:37.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:37.562+0000 7fa1ab200640 1 -- 172.21.15.96:0/3871426847 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone43638", "target": ["mon-mgr", ""]}) -- 0x7fa1a414fee0 con 0x7fa18006d9d0 2024-05-28T06:16:38.515 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:16:38.516+0000 7fea59e00640 -1 mgr.server reply reply (22) Invalid argument cannot cancel -- clone finished (check clone status) 2024-05-28T06:16:38.518 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:38.518+0000 7fa1a0c00640 1 -- 172.21.15.96:0/3871426847 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: -22 cannot cancel -- clone finished (check clone status)) ==== 60+0+0 (secure 0 0 0) 0x7fa1a414fee0 con 0x7fa18006d9d0 2024-05-28T06:16:38.518 INFO:teuthology.orchestra.run.smithi096.stderr:Error EINVAL: cannot cancel -- clone finished (check clone status) 2024-05-28T06:16:38.534 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-05-28T06:16:38.535 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone43639 2024-05-28T06:16:38.826 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:38.826+0000 7ff464000640 1 Processor -- start 2024-05-28T06:16:38.826 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:38.826+0000 7ff464000640 1 -- start start 2024-05-28T06:16:38.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:38.829+0000 7ff464000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff45c15abe0 0x7ff45c06f380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:38.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:38.829+0000 7ff464000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7ff45c0725c0 0x7ff45c072a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:38.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:38.829+0000 7ff464000640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff45c06e2e0 con 0x7ff45c06f8c0 2024-05-28T06:16:38.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:38.829+0000 7ff464000640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7ff45c06e5c0 con 0x7ff45c0725c0 2024-05-28T06:16:38.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:38.829+0000 7ff464000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ff45c06e8a0 con 0x7ff45c15abe0 2024-05-28T06:16:38.830 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:38.830+0000 7ff461600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff45c15abe0 0x7ff45c06f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:38.830 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:38.830+0000 7ff461600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff45c15abe0 0x7ff45c06f380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.96:40378/0 (socket says 172.21.15.96:40378) 2024-05-28T06:16:38.830 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:38.830+0000 7ff461600640 1 -- 172.21.15.96:0/3312605775 learned_addr learned my addr 172.21.15.96:0/3312605775 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:16:38.830 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:38.831+0000 7ff461600640 1 -- 172.21.15.96:0/3312605775 >> v2:172.21.15.123:3300/0 conn(0x7ff45c0725c0 msgr2=0x7ff45c072a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:16:38.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:38.831+0000 7ff461600640 1 --2- 172.21.15.96:0/3312605775 >> v2:172.21.15.123:3300/0 conn(0x7ff45c0725c0 0x7ff45c072a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:38.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:38.831+0000 7ff461600640 1 -- 172.21.15.96:0/3312605775 >> v1:172.21.15.123:6789/0 conn(0x7ff45c06f8c0 legacy=0x7ff45c0b45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:16:38.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:38.831+0000 7ff461600640 1 -- 172.21.15.96:0/3312605775 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff45c06eb80 con 0x7ff45c15abe0 2024-05-28T06:16:38.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:38.833+0000 7ff461600640 1 --2- 172.21.15.96:0/3312605775 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff45c15abe0 0x7ff45c06f380 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff448009870 tx=0x7ff44802b420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c5595a15c503a15 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:38.833 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:38.833+0000 7ff45be00640 1 -- 172.21.15.96:0/3312605775 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff448037070 con 0x7ff45c15abe0 2024-05-28T06:16:38.833 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:38.834+0000 7ff45be00640 1 -- 172.21.15.96:0/3312605775 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff448030710 con 0x7ff45c15abe0 2024-05-28T06:16:38.833 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:38.834+0000 7ff45be00640 1 -- 172.21.15.96:0/3312605775 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff448030ae0 con 0x7ff45c15abe0 2024-05-28T06:16:38.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:38.834+0000 7ff464000640 1 -- 172.21.15.96:0/3312605775 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff45c15abe0 msgr2=0x7ff45c06f380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:38.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:38.834+0000 7ff464000640 1 --2- 172.21.15.96:0/3312605775 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff45c15abe0 0x7ff45c06f380 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff448009870 tx=0x7ff44802b420 comp rx=0 tx=0).stop 2024-05-28T06:16:38.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:38.834+0000 7ff464000640 1 -- 172.21.15.96:0/3312605775 shutdown_connections 2024-05-28T06:16:38.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:38.834+0000 7ff464000640 1 --2- 172.21.15.96:0/3312605775 >> v2:172.21.15.123:3300/0 conn(0x7ff45c0725c0 0x7ff45c072a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:38.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:38.834+0000 7ff464000640 1 --2- 172.21.15.96:0/3312605775 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff45c15abe0 0x7ff45c06f380 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:38.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:38.834+0000 7ff464000640 1 -- 172.21.15.96:0/3312605775 >> 172.21.15.96:0/3312605775 conn(0x7ff45c06bd20 msgr2=0x7ff45c06c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:16:38.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:38.835+0000 7ff464000640 1 -- 172.21.15.96:0/3312605775 shutdown_connections 2024-05-28T06:16:38.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:38.835+0000 7ff464000640 1 -- 172.21.15.96:0/3312605775 wait complete. 2024-05-28T06:16:38.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:38.837+0000 7ff464000640 1 Processor -- start 2024-05-28T06:16:38.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:38.838+0000 7ff464000640 1 -- start start 2024-05-28T06:16:38.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:38.839+0000 7ff464000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff45c06f8c0 0x7ff45c09df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:38.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:38.839+0000 7ff464000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff45c0725c0 0x7ff45c159740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:38.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:38.839+0000 7ff464000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff45c15abe0 0x7ff45c154610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:38.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:38.839+0000 7ff464000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff45c0b4ce0 con 0x7ff45c15abe0 2024-05-28T06:16:38.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:38.839+0000 7ff464000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff45c15afc0 con 0x7ff45c06f8c0 2024-05-28T06:16:38.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:38.839+0000 7ff464000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ff45c06df60 con 0x7ff45c0725c0 2024-05-28T06:16:38.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:38.840+0000 7ff461600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff45c06f8c0 0x7ff45c09df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:38.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:38.840+0000 7ff461600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff45c06f8c0 0x7ff45c09df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.96:35022/0 (socket says 172.21.15.96:35022) 2024-05-28T06:16:38.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:38.840+0000 7ff461600640 1 -- 172.21.15.96:0/742016242 learned_addr learned my addr 172.21.15.96:0/742016242 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:16:38.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:38.840+0000 7ff462000640 1 --2- 172.21.15.96:0/742016242 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff45c15abe0 0x7ff45c154610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:38.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:38.840+0000 7ff460c00640 1 --2- 172.21.15.96:0/742016242 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff45c0725c0 0x7ff45c159740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:38.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:38.840+0000 7ff462000640 1 -- 172.21.15.96:0/742016242 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff45c0725c0 msgr2=0x7ff45c159740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:38.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:38.840+0000 7ff462000640 1 --2- 172.21.15.96:0/742016242 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff45c0725c0 0x7ff45c159740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:38.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:38.840+0000 7ff462000640 1 -- 172.21.15.96:0/742016242 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff45c06f8c0 msgr2=0x7ff45c09df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:38.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:38.840+0000 7ff462000640 1 --2- 172.21.15.96:0/742016242 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff45c06f8c0 0x7ff45c09df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:38.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:38.840+0000 7ff462000640 1 -- 172.21.15.96:0/742016242 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff45c0b3120 con 0x7ff45c15abe0 2024-05-28T06:16:38.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:38.841+0000 7ff462000640 1 --2- 172.21.15.96:0/742016242 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff45c15abe0 0x7ff45c154610 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7ff45400be50 tx=0x7ff45400bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:38.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:38.841+0000 7ff459600640 1 -- 172.21.15.96:0/742016242 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff454019070 con 0x7ff45c15abe0 2024-05-28T06:16:38.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:38.841+0000 7ff459600640 1 -- 172.21.15.96:0/742016242 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff45400ce50 con 0x7ff45c15abe0 2024-05-28T06:16:38.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:38.841+0000 7ff459600640 1 -- 172.21.15.96:0/742016242 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff454014680 con 0x7ff45c15abe0 2024-05-28T06:16:38.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:38.841+0000 7ff464000640 1 -- 172.21.15.96:0/742016242 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff45c0b36a0 con 0x7ff45c15abe0 2024-05-28T06:16:38.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:38.841+0000 7ff464000640 1 -- 172.21.15.96:0/742016242 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff45c1f5a40 con 0x7ff45c15abe0 2024-05-28T06:16:38.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:38.843+0000 7ff464000640 1 -- 172.21.15.96:0/742016242 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff45c0689e0 con 0x7ff45c15abe0 2024-05-28T06:16:38.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:38.846+0000 7ff459600640 1 -- 172.21.15.96:0/742016242 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 28) ==== 92570+0+0 (secure 0 0 0) 0x7ff45401c070 con 0x7ff45c15abe0 2024-05-28T06:16:38.847 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:38.848+0000 7ff459600640 1 --2- 172.21.15.96:0/742016242 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff43c06d9d0 0x7ff43c06fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:38.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:38.848+0000 7ff459600640 1 -- 172.21.15.96:0/742016242 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5795+0+0 (secure 0 0 0) 0x7ff45408ae00 con 0x7ff45c15abe0 2024-05-28T06:16:38.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:38.854+0000 7ff459600640 1 -- 172.21.15.96:0/742016242 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff454023020 con 0x7ff45c15abe0 2024-05-28T06:16:38.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:38.875+0000 7ff461600640 1 --2- 172.21.15.96:0/742016242 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff43c06d9d0 0x7ff43c06fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:38.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:38.901+0000 7ff461600640 1 --2- 172.21.15.96:0/742016242 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff43c06d9d0 0x7ff43c06fe90 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7ff45c0b11c0 tx=0x7ff448008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:39.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:39.319+0000 7ff464000640 1 -- 172.21.15.96:0/742016242 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone43639", "target": ["mon-mgr", ""]}) -- 0x7ff45c154b50 con 0x7ff43c06d9d0 2024-05-28T06:16:39.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:39.790+0000 7ff459600640 1 -- 172.21.15.96:0/742016242 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7ff448030880 con 0x7ff43c06d9d0 2024-05-28T06:16:39.793 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:39.793+0000 7ff464000640 1 -- 172.21.15.96:0/742016242 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff43c06d9d0 msgr2=0x7ff43c06fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:39.793 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:39.793+0000 7ff464000640 1 --2- 172.21.15.96:0/742016242 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff43c06d9d0 0x7ff43c06fe90 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7ff45c0b11c0 tx=0x7ff448008030 comp rx=0 tx=0).stop 2024-05-28T06:16:39.794 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:39.793+0000 7ff464000640 1 -- 172.21.15.96:0/742016242 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff45c15abe0 msgr2=0x7ff45c154610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:39.794 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:39.793+0000 7ff464000640 1 --2- 172.21.15.96:0/742016242 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff45c15abe0 0x7ff45c154610 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7ff45400be50 tx=0x7ff45400bf40 comp rx=0 tx=0).stop 2024-05-28T06:16:39.794 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:39.794+0000 7ff464000640 1 -- 172.21.15.96:0/742016242 shutdown_connections 2024-05-28T06:16:39.794 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:39.794+0000 7ff464000640 1 --2- 172.21.15.96:0/742016242 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff43c06d9d0 0x7ff43c06fe90 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:39.794 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:39.794+0000 7ff464000640 1 --2- 172.21.15.96:0/742016242 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff45c15abe0 0x7ff45c154610 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:39.794 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:39.794+0000 7ff464000640 1 --2- 172.21.15.96:0/742016242 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff45c0725c0 0x7ff45c159740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:39.794 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:39.794+0000 7ff464000640 1 --2- 172.21.15.96:0/742016242 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff45c06f8c0 0x7ff45c09df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:39.794 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:39.794+0000 7ff464000640 1 -- 172.21.15.96:0/742016242 >> 172.21.15.96:0/742016242 conn(0x7ff45c06bd20 msgr2=0x7ff45c0afe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:16:39.794 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:39.794+0000 7ff464000640 1 -- 172.21.15.96:0/742016242 shutdown_connections 2024-05-28T06:16:39.794 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:39.794+0000 7ff464000640 1 -- 172.21.15.96:0/742016242 wait complete. 2024-05-28T06:16:39.814 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone436310 2024-05-28T06:16:40.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:40.044+0000 7f94cf200640 1 Processor -- start 2024-05-28T06:16:40.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:40.044+0000 7f94cf200640 1 -- start start 2024-05-28T06:16:40.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:40.045+0000 7f94cf200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f94d0158d40 0x7f94d015c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:40.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:40.045+0000 7f94cf200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f94d00b30b0 0x7f94d006bd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:40.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:40.045+0000 7f94cf200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f94d006c260 0x7f94d015ee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:40.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:40.045+0000 7f94cf200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f94d00ae040 con 0x7f94d00b30b0 2024-05-28T06:16:40.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:40.045+0000 7f94cf200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f94d00add50 con 0x7f94d0158d40 2024-05-28T06:16:40.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:40.045+0000 7f94cf200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f94d00b3800 con 0x7f94d006c260 2024-05-28T06:16:40.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:40.046+0000 7f94cd400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f94d00b30b0 0x7f94d006bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:40.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:40.046+0000 7f94cd400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f94d00b30b0 0x7f94d006bd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48758/0 (socket says 172.21.15.96:48758) 2024-05-28T06:16:40.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:40.046+0000 7f94cd400640 1 -- 172.21.15.96:0/1359513159 learned_addr learned my addr 172.21.15.96:0/1359513159 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:16:40.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:40.046+0000 7f94cde00640 1 --2- 172.21.15.96:0/1359513159 >> v2:172.21.15.123:3300/0 conn(0x7f94d0158d40 0x7f94d015c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:40.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:40.046+0000 7f94ce800640 1 --2- 172.21.15.96:0/1359513159 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f94d006c260 0x7f94d015ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:40.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:40.046+0000 7f94cd400640 1 -- 172.21.15.96:0/1359513159 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f94d006c260 msgr2=0x7f94d015ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:16:40.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:40.046+0000 7f94cd400640 1 --2- 172.21.15.96:0/1359513159 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f94d006c260 0x7f94d015ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:40.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:40.046+0000 7f94cd400640 1 -- 172.21.15.96:0/1359513159 >> v2:172.21.15.123:3300/0 conn(0x7f94d0158d40 msgr2=0x7f94d015c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:16:40.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:40.046+0000 7f94cd400640 1 --2- 172.21.15.96:0/1359513159 >> v2:172.21.15.123:3300/0 conn(0x7f94d0158d40 0x7f94d015c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:40.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:40.046+0000 7f94cd400640 1 -- 172.21.15.96:0/1359513159 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94d00b3ae0 con 0x7f94d00b30b0 2024-05-28T06:16:40.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:40.046+0000 7f94cde00640 1 --2- 172.21.15.96:0/1359513159 >> v2:172.21.15.123:3300/0 conn(0x7f94d0158d40 0x7f94d015c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-28T06:16:40.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:40.047+0000 7f94cd400640 1 --2- 172.21.15.96:0/1359513159 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f94d00b30b0 0x7f94d006bd20 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f94d006fc00 tx=0x7f94b802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18700465e2a1d234 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:40.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:40.048+0000 7f94cca00640 1 -- 172.21.15.96:0/1359513159 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94b8037070 con 0x7f94d00b30b0 2024-05-28T06:16:40.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:40.048+0000 7f94cca00640 1 -- 172.21.15.96:0/1359513159 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94b8032440 con 0x7f94d00b30b0 2024-05-28T06:16:40.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:40.048+0000 7f94cca00640 1 -- 172.21.15.96:0/1359513159 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94b8032810 con 0x7f94d00b30b0 2024-05-28T06:16:40.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:40.049+0000 7f94cf200640 1 -- 172.21.15.96:0/1359513159 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f94d00b30b0 msgr2=0x7f94d006bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:40.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:40.049+0000 7f94cf200640 1 --2- 172.21.15.96:0/1359513159 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f94d00b30b0 0x7f94d006bd20 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f94d006fc00 tx=0x7f94b802a1e0 comp rx=0 tx=0).stop 2024-05-28T06:16:40.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:40.050+0000 7f94cf200640 1 -- 172.21.15.96:0/1359513159 shutdown_connections 2024-05-28T06:16:40.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:40.050+0000 7f94cf200640 1 --2- 172.21.15.96:0/1359513159 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f94d006c260 0x7f94d015ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:40.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:40.050+0000 7f94cf200640 1 --2- 172.21.15.96:0/1359513159 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f94d00b30b0 0x7f94d006bd20 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:40.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:40.050+0000 7f94cf200640 1 --2- 172.21.15.96:0/1359513159 >> v2:172.21.15.123:3300/0 conn(0x7f94d0158d40 0x7f94d015c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:40.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:40.050+0000 7f94cf200640 1 -- 172.21.15.96:0/1359513159 >> 172.21.15.96:0/1359513159 conn(0x7f94d00b21d0 msgr2=0x7f94d00b25e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:16:40.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:40.050+0000 7f94cf200640 1 -- 172.21.15.96:0/1359513159 shutdown_connections 2024-05-28T06:16:40.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:40.050+0000 7f94cf200640 1 -- 172.21.15.96:0/1359513159 wait complete. 2024-05-28T06:16:40.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:40.052+0000 7f94cf200640 1 Processor -- start 2024-05-28T06:16:40.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:40.052+0000 7f94cf200640 1 -- start start 2024-05-28T06:16:40.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:40.055+0000 7f94cf200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f94d006c260 0x7f94d014c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:40.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:40.055+0000 7f94cf200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f94d00b30b0 0x7f94d01592a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:40.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:40.055+0000 7f94cf200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f94d0158d40 0x7f94d01597e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:40.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:40.055+0000 7f94cf200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f94d0159120 con 0x7f94d00b30b0 2024-05-28T06:16:40.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:40.055+0000 7f94cf200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f94d00b3510 con 0x7f94d006c260 2024-05-28T06:16:40.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:40.055+0000 7f94cf200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f94d015f340 con 0x7f94d0158d40 2024-05-28T06:16:40.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:40.055+0000 7f94cd400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f94d00b30b0 0x7f94d01592a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:40.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:40.055+0000 7f94cd400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f94d00b30b0 0x7f94d01592a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48772/0 (socket says 172.21.15.96:48772) 2024-05-28T06:16:40.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:40.055+0000 7f94cd400640 1 -- 172.21.15.96:0/1576541924 learned_addr learned my addr 172.21.15.96:0/1576541924 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:16:40.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:40.056+0000 7f94cd400640 1 -- 172.21.15.96:0/1576541924 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f94d0158d40 msgr2=0x7f94d01597e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:40.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:40.056+0000 7f94cde00640 1 --2- 172.21.15.96:0/1576541924 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f94d006c260 0x7f94d014c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:40.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:40.056+0000 7f94cd400640 1 --2- 172.21.15.96:0/1576541924 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f94d0158d40 0x7f94d01597e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:40.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:40.056+0000 7f94cd400640 1 -- 172.21.15.96:0/1576541924 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f94d006c260 msgr2=0x7f94d014c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:40.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:40.056+0000 7f94cd400640 1 --2- 172.21.15.96:0/1576541924 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f94d006c260 0x7f94d014c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:40.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:40.056+0000 7f94cd400640 1 -- 172.21.15.96:0/1576541924 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94d014d4e0 con 0x7f94d00b30b0 2024-05-28T06:16:40.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:40.056+0000 7f94cde00640 1 --2- 172.21.15.96:0/1576541924 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f94d006c260 0x7f94d014c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:16:40.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:40.057+0000 7f94cd400640 1 --2- 172.21.15.96:0/1576541924 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f94d00b30b0 0x7f94d01592a0 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f94b8009960 tx=0x7f94b80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:40.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:40.058+0000 7f94c6000640 1 -- 172.21.15.96:0/1576541924 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94b8041020 con 0x7f94d00b30b0 2024-05-28T06:16:40.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:40.058+0000 7f94c6000640 1 -- 172.21.15.96:0/1576541924 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94b8030780 con 0x7f94d00b30b0 2024-05-28T06:16:40.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:40.058+0000 7f94c6000640 1 -- 172.21.15.96:0/1576541924 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94b8037040 con 0x7f94d00b30b0 2024-05-28T06:16:40.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:40.058+0000 7f94cf200640 1 -- 172.21.15.96:0/1576541924 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94d015bd30 con 0x7f94d00b30b0 2024-05-28T06:16:40.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:40.059+0000 7f94cf200640 1 -- 172.21.15.96:0/1576541924 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94d015c2f0 con 0x7f94d00b30b0 2024-05-28T06:16:40.062 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:40.061+0000 7f94cf200640 1 -- 172.21.15.96:0/1576541924 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94d00689e0 con 0x7f94d00b30b0 2024-05-28T06:16:40.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:40.067+0000 7f94c6000640 1 -- 172.21.15.96:0/1576541924 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 28) ==== 92570+0+0 (secure 0 0 0) 0x7f94b8044020 con 0x7f94d00b30b0 2024-05-28T06:16:40.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:40.068+0000 7f94c6000640 1 --2- 172.21.15.96:0/1576541924 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f94ac06dad0 0x7f94ac06ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:40.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:40.068+0000 7f94c6000640 1 -- 172.21.15.96:0/1576541924 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5795+0+0 (secure 0 0 0) 0x7f94b80aaf20 con 0x7f94d00b30b0 2024-05-28T06:16:40.075 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:40.076+0000 7f94c6000640 1 -- 172.21.15.96:0/1576541924 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f94b804b030 con 0x7f94d00b30b0 2024-05-28T06:16:40.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:40.076+0000 7f94cde00640 1 --2- 172.21.15.96:0/1576541924 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f94ac06dad0 0x7f94ac06ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:40.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:40.084+0000 7f94cde00640 1 --2- 172.21.15.96:0/1576541924 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f94ac06dad0 0x7f94ac06ff90 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f94c000c220 tx=0x7f94c000b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:40.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:40.461+0000 7f94cf200640 1 -- 172.21.15.96:0/1576541924 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone436310", "target": ["mon-mgr", ""]}) -- 0x7f94d006f7e0 con 0x7f94ac06dad0 2024-05-28T06:16:40.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:40.872+0000 7f94c6000640 1 -- 172.21.15.96:0/1576541924 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: -22 cannot cancel -- clone finished (check clone status)) ==== 60+0+0 (secure 0 0 0) 0x7f94d006f7e0 con 0x7f94ac06dad0 2024-05-28T06:16:40.872 INFO:teuthology.orchestra.run.smithi096.stderr:Error EINVAL: cannot cancel -- clone finished (check clone status) 2024-05-28T06:16:40.873 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:16:40.872+0000 7fea59e00640 -1 mgr.server reply reply (22) Invalid argument cannot cancel -- clone finished (check clone status) 2024-05-28T06:16:40.887 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-05-28T06:16:40.887 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone436311 2024-05-28T06:16:41.145 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:41.145+0000 7f999d800640 1 Processor -- start 2024-05-28T06:16:41.145 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:41.145+0000 7f999d800640 1 -- start start 2024-05-28T06:16:41.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:41.146+0000 7f999d800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f999815e850 0x7f99980b77b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:41.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:41.146+0000 7f999d800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f99980d1d50 0x7f99980b90a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:41.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:41.146+0000 7f999d800640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f99980b8000 con 0x7f99980cef30 2024-05-28T06:16:41.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:41.146+0000 7f999d800640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f999815ec30 con 0x7f999815e850 2024-05-28T06:16:41.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:41.146+0000 7f999d800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f99980b82e0 con 0x7f99980d1d50 2024-05-28T06:16:41.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:41.147+0000 7f9996a00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f999815e850 0x7f99980b77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:41.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:41.147+0000 7f9997400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f99980d1d50 0x7f99980b90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:41.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:41.147+0000 7f9996a00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f999815e850 0x7f99980b77b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.96:35068/0 (socket says 172.21.15.96:35068) 2024-05-28T06:16:41.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:41.147+0000 7f9997400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f99980d1d50 0x7f99980b90a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.96:40424/0 (socket says 172.21.15.96:40424) 2024-05-28T06:16:41.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:41.147+0000 7f9996a00640 1 -- 172.21.15.96:0/1428737776 learned_addr learned my addr 172.21.15.96:0/1428737776 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:16:41.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:41.148+0000 7f9997400640 1 -- 172.21.15.96:0/1428737776 >> v2:172.21.15.123:3300/0 conn(0x7f999815e850 msgr2=0x7f99980b77b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:16:41.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:41.148+0000 7f9997400640 1 --2- 172.21.15.96:0/1428737776 >> v2:172.21.15.123:3300/0 conn(0x7f999815e850 0x7f99980b77b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:41.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:41.148+0000 7f9997400640 1 -- 172.21.15.96:0/1428737776 >> v1:172.21.15.123:6789/0 conn(0x7f99980cef30 legacy=0x7f99980d1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:16:41.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:41.148+0000 7f9997400640 1 -- 172.21.15.96:0/1428737776 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99980b85c0 con 0x7f99980d1d50 2024-05-28T06:16:41.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:41.149+0000 7f9997400640 1 --2- 172.21.15.96:0/1428737776 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f99980d1d50 0x7f99980b90a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f999806dba0 tx=0x7f999002a080 comp rx=0 tx=0).ready entity=mon.2 client_cookie=206e4e98d21b6625 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:41.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:41.150+0000 7f9995600640 1 -- 172.21.15.96:0/1428737776 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9990037070 con 0x7f99980d1d50 2024-05-28T06:16:41.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:41.150+0000 7f9995600640 1 -- 172.21.15.96:0/1428737776 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f999002ad90 con 0x7f99980d1d50 2024-05-28T06:16:41.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:41.150+0000 7f9995600640 1 -- 172.21.15.96:0/1428737776 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9990032660 con 0x7f99980d1d50 2024-05-28T06:16:41.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:41.151+0000 7f999d800640 1 -- 172.21.15.96:0/1428737776 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f99980d1d50 msgr2=0x7f99980b90a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:41.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:41.151+0000 7f999d800640 1 --2- 172.21.15.96:0/1428737776 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f99980d1d50 0x7f99980b90a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f999806dba0 tx=0x7f999002a080 comp rx=0 tx=0).stop 2024-05-28T06:16:41.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:41.151+0000 7f999d800640 1 -- 172.21.15.96:0/1428737776 shutdown_connections 2024-05-28T06:16:41.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:41.151+0000 7f999d800640 1 --2- 172.21.15.96:0/1428737776 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f99980d1d50 0x7f99980b90a0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:41.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:41.151+0000 7f999d800640 1 --2- 172.21.15.96:0/1428737776 >> v2:172.21.15.123:3300/0 conn(0x7f999815e850 0x7f99980b77b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:41.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:41.151+0000 7f999d800640 1 -- 172.21.15.96:0/1428737776 >> 172.21.15.96:0/1428737776 conn(0x7f9998067aa0 msgr2=0x7f9998067eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:16:41.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:41.151+0000 7f999d800640 1 -- 172.21.15.96:0/1428737776 shutdown_connections 2024-05-28T06:16:41.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:41.151+0000 7f999d800640 1 -- 172.21.15.96:0/1428737776 wait complete. 2024-05-28T06:16:41.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:41.153+0000 7f999d800640 1 Processor -- start 2024-05-28T06:16:41.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:41.153+0000 7f999d800640 1 -- start start 2024-05-28T06:16:41.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:41.154+0000 7f999d800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99980cef30 0x7f9998157520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:41.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:41.154+0000 7f999d800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f99980d1d50 0x7f99980d4630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:41.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:41.154+0000 7f999d800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f999815e850 0x7f99980d4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:41.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:41.154+0000 7f999d800640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f99980b95e0 con 0x7f99980d1d50 2024-05-28T06:16:41.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:41.154+0000 7f999d800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f99980b7cf0 con 0x7f99980cef30 2024-05-28T06:16:41.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:41.154+0000 7f999d800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f99980b9760 con 0x7f999815e850 2024-05-28T06:16:41.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:41.155+0000 7f9996000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f99980d1d50 0x7f99980d4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:41.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:41.155+0000 7f9996000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f99980d1d50 0x7f99980d4630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48778/0 (socket says 172.21.15.96:48778) 2024-05-28T06:16:41.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:41.155+0000 7f9996000640 1 -- 172.21.15.96:0/3792987225 learned_addr learned my addr 172.21.15.96:0/3792987225 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:16:41.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:41.155+0000 7f9996a00640 1 --2- 172.21.15.96:0/3792987225 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99980cef30 0x7f9998157520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:41.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:41.155+0000 7f9996000640 1 -- 172.21.15.96:0/3792987225 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f999815e850 msgr2=0x7f99980d4b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:41.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:41.155+0000 7f9996000640 1 --2- 172.21.15.96:0/3792987225 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f999815e850 0x7f99980d4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:41.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:41.155+0000 7f9996000640 1 -- 172.21.15.96:0/3792987225 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99980cef30 msgr2=0x7f9998157520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:41.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:41.155+0000 7f9996000640 1 --2- 172.21.15.96:0/3792987225 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99980cef30 0x7f9998157520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:41.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:41.155+0000 7f9996000640 1 -- 172.21.15.96:0/3792987225 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9998161550 con 0x7f99980d1d50 2024-05-28T06:16:41.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:41.156+0000 7f9996000640 1 --2- 172.21.15.96:0/3792987225 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f99980d1d50 0x7f99980d4630 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f998000c740 tx=0x7f998000cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:41.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:41.156+0000 7f998ea00640 1 -- 172.21.15.96:0/3792987225 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9980016020 con 0x7f99980d1d50 2024-05-28T06:16:41.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:41.156+0000 7f998ea00640 1 -- 172.21.15.96:0/3792987225 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f998000ce50 con 0x7f99980d1d50 2024-05-28T06:16:41.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:41.156+0000 7f998ea00640 1 -- 172.21.15.96:0/3792987225 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f998000acd0 con 0x7f99980d1d50 2024-05-28T06:16:41.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:41.156+0000 7f999d800640 1 -- 172.21.15.96:0/3792987225 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99980cb940 con 0x7f99980d1d50 2024-05-28T06:16:41.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:41.157+0000 7f999d800640 1 -- 172.21.15.96:0/3792987225 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99980cbf00 con 0x7f99980d1d50 2024-05-28T06:16:41.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:41.159+0000 7f998ea00640 1 -- 172.21.15.96:0/3792987225 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 28) ==== 92570+0+0 (secure 0 0 0) 0x7f9980019020 con 0x7f99980d1d50 2024-05-28T06:16:41.159 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:41.159+0000 7f999d800640 1 -- 172.21.15.96:0/3792987225 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99980cc100 con 0x7f99980d1d50 2024-05-28T06:16:41.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:41.160+0000 7f998ea00640 1 --2- 172.21.15.96:0/3792987225 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f997406d9d0 0x7f997406fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:41.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:41.160+0000 7f998ea00640 1 -- 172.21.15.96:0/3792987225 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5795+0+0 (secure 0 0 0) 0x7f998008a470 con 0x7f99980d1d50 2024-05-28T06:16:41.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:41.168+0000 7f998ea00640 1 -- 172.21.15.96:0/3792987225 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f99980cc100 con 0x7f99980d1d50 2024-05-28T06:16:41.196 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:41.196+0000 7f9996a00640 1 --2- 172.21.15.96:0/3792987225 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f997406d9d0 0x7f997406fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:41.325 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:41.326+0000 7f9996a00640 1 --2- 172.21.15.96:0/3792987225 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f997406d9d0 0x7f997406fe90 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f998800ba10 tx=0x7f9988008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:41.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:41.552+0000 7f999d800640 1 -- 172.21.15.96:0/3792987225 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone436311", "target": ["mon-mgr", ""]}) -- 0x7f99980cb780 con 0x7f997406d9d0 2024-05-28T06:16:42.728 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:42.728+0000 7f998ea00640 1 -- 172.21.15.96:0/3792987225 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f998800f860 con 0x7f997406d9d0 2024-05-28T06:16:42.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:42.735+0000 7f999d800640 1 -- 172.21.15.96:0/3792987225 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f997406d9d0 msgr2=0x7f997406fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:42.737 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:42.735+0000 7f999d800640 1 --2- 172.21.15.96:0/3792987225 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f997406d9d0 0x7f997406fe90 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f998800ba10 tx=0x7f9988008040 comp rx=0 tx=0).stop 2024-05-28T06:16:42.737 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:42.735+0000 7f999d800640 1 -- 172.21.15.96:0/3792987225 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f99980d1d50 msgr2=0x7f99980d4630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:42.737 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:42.735+0000 7f999d800640 1 --2- 172.21.15.96:0/3792987225 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f99980d1d50 0x7f99980d4630 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f998000c740 tx=0x7f998000cc00 comp rx=0 tx=0).stop 2024-05-28T06:16:42.737 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:42.736+0000 7f999d800640 1 -- 172.21.15.96:0/3792987225 shutdown_connections 2024-05-28T06:16:42.737 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:42.736+0000 7f999d800640 1 --2- 172.21.15.96:0/3792987225 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f997406d9d0 0x7f997406fe90 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:42.737 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:42.736+0000 7f999d800640 1 --2- 172.21.15.96:0/3792987225 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f999815e850 0x7f99980d4b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:42.737 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:42.736+0000 7f999d800640 1 --2- 172.21.15.96:0/3792987225 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f99980d1d50 0x7f99980d4630 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:42.737 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:42.736+0000 7f999d800640 1 --2- 172.21.15.96:0/3792987225 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99980cef30 0x7f9998157520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:42.737 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:42.736+0000 7f999d800640 1 -- 172.21.15.96:0/3792987225 >> 172.21.15.96:0/3792987225 conn(0x7f9998067aa0 msgr2=0x7f99980ca070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:16:42.737 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:42.736+0000 7f999d800640 1 -- 172.21.15.96:0/3792987225 shutdown_connections 2024-05-28T06:16:42.737 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:42.736+0000 7f999d800640 1 -- 172.21.15.96:0/3792987225 wait complete. 2024-05-28T06:16:42.769 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone436312 2024-05-28T06:16:43.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:43.088+0000 7faf90a00640 1 Processor -- start 2024-05-28T06:16:43.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:43.088+0000 7faf90a00640 1 -- start start 2024-05-28T06:16:43.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:43.089+0000 7faf90a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faf8c15e870 0x7faf8c0d0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:43.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:43.089+0000 7faf90a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7faf8c0d14d0 0x7faf8c0cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:43.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:43.089+0000 7faf90a00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7faf8c0cf870 0x7faf8c162360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:43.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:43.090+0000 7faf90a00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7faf8c0d2150 con 0x7faf8c0d14d0 2024-05-28T06:16:43.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:43.090+0000 7faf90a00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7faf8c0cfd00 con 0x7faf8c0cf870 2024-05-28T06:16:43.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:43.090+0000 7faf90a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7faf8c15ec50 con 0x7faf8c15e870 2024-05-28T06:16:43.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:43.090+0000 7faf89e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7faf8c0d14d0 0x7faf8c0cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:43.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:43.091+0000 7faf89e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7faf8c0d14d0 0x7faf8c0cf330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48780/0 (socket says 172.21.15.96:48780) 2024-05-28T06:16:43.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:43.091+0000 7faf89e00640 1 -- 172.21.15.96:0/324909276 learned_addr learned my addr 172.21.15.96:0/324909276 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:16:43.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:43.091+0000 7faf8b200640 1 --2- 172.21.15.96:0/324909276 >> v2:172.21.15.123:3300/0 conn(0x7faf8c0cf870 0x7faf8c162360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:43.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:43.091+0000 7faf8a800640 1 --2- 172.21.15.96:0/324909276 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faf8c15e870 0x7faf8c0d0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:43.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:43.092+0000 7faf89e00640 1 -- 172.21.15.96:0/324909276 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faf8c15e870 msgr2=0x7faf8c0d0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:16:43.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:43.092+0000 7faf89e00640 1 --2- 172.21.15.96:0/324909276 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faf8c15e870 0x7faf8c0d0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:43.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:43.092+0000 7faf89e00640 1 -- 172.21.15.96:0/324909276 >> v2:172.21.15.123:3300/0 conn(0x7faf8c0cf870 msgr2=0x7faf8c162360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:16:43.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:43.092+0000 7faf89e00640 1 --2- 172.21.15.96:0/324909276 >> v2:172.21.15.123:3300/0 conn(0x7faf8c0cf870 0x7faf8c162360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:43.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:43.092+0000 7faf89e00640 1 -- 172.21.15.96:0/324909276 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf8c0cff80 con 0x7faf8c0d14d0 2024-05-28T06:16:43.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:43.092+0000 7faf8b200640 1 --2- 172.21.15.96:0/324909276 >> v2:172.21.15.123:3300/0 conn(0x7faf8c0cf870 0x7faf8c162360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-28T06:16:43.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:43.093+0000 7faf89e00640 1 --2- 172.21.15.96:0/324909276 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf8c0d14d0 0x7faf8c0cf330 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7faf8c0b3f60 tx=0x7faf7c02a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a2325540a442956 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:43.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:43.094+0000 7faf89400640 1 -- 172.21.15.96:0/324909276 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf7c004030 con 0x7faf8c0d14d0 2024-05-28T06:16:43.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:43.094+0000 7faf89400640 1 -- 172.21.15.96:0/324909276 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf7c0041d0 con 0x7faf8c0d14d0 2024-05-28T06:16:43.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:43.094+0000 7faf89400640 1 -- 172.21.15.96:0/324909276 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf7c003610 con 0x7faf8c0d14d0 2024-05-28T06:16:43.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:43.095+0000 7faf90a00640 1 -- 172.21.15.96:0/324909276 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf8c0d14d0 msgr2=0x7faf8c0cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:43.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:43.095+0000 7faf90a00640 1 --2- 172.21.15.96:0/324909276 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf8c0d14d0 0x7faf8c0cf330 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7faf8c0b3f60 tx=0x7faf7c02a1c0 comp rx=0 tx=0).stop 2024-05-28T06:16:43.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:43.095+0000 7faf90a00640 1 -- 172.21.15.96:0/324909276 shutdown_connections 2024-05-28T06:16:43.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:43.095+0000 7faf90a00640 1 --2- 172.21.15.96:0/324909276 >> v2:172.21.15.123:3300/0 conn(0x7faf8c0cf870 0x7faf8c162360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:43.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:43.095+0000 7faf90a00640 1 --2- 172.21.15.96:0/324909276 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf8c0d14d0 0x7faf8c0cf330 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:43.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:43.095+0000 7faf90a00640 1 --2- 172.21.15.96:0/324909276 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faf8c15e870 0x7faf8c0d0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:43.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:43.095+0000 7faf90a00640 1 -- 172.21.15.96:0/324909276 >> 172.21.15.96:0/324909276 conn(0x7faf8c08bfc0 msgr2=0x7faf8c0d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:16:43.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:43.095+0000 7faf90a00640 1 -- 172.21.15.96:0/324909276 shutdown_connections 2024-05-28T06:16:43.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:43.095+0000 7faf90a00640 1 -- 172.21.15.96:0/324909276 wait complete. 2024-05-28T06:16:43.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:43.098+0000 7faf90a00640 1 Processor -- start 2024-05-28T06:16:43.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:43.099+0000 7faf90a00640 1 -- start start 2024-05-28T06:16:43.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:43.101+0000 7faf90a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf8c0cf870 0x7faf8c160dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:43.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:43.102+0000 7faf90a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faf8c0d14d0 0x7faf8c15bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:43.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:43.102+0000 7faf90a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faf8c15e870 0x7faf8c14f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:43.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:43.102+0000 7faf90a00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7faf8c0d0820 con 0x7faf8c0cf870 2024-05-28T06:16:43.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:43.102+0000 7faf8a800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf8c0cf870 0x7faf8c160dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:43.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:43.102+0000 7faf8a800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf8c0cf870 0x7faf8c160dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48794/0 (socket says 172.21.15.96:48794) 2024-05-28T06:16:43.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:43.102+0000 7faf90a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7faf8c0d0540 con 0x7faf8c0d14d0 2024-05-28T06:16:43.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:43.102+0000 7faf8a800640 1 -- 172.21.15.96:0/3430949740 learned_addr learned my addr 172.21.15.96:0/3430949740 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:16:43.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:43.102+0000 7faf90a00640 1 -- 172.21.15.96:0/3430949740 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7faf8c0d4fb0 con 0x7faf8c15e870 2024-05-28T06:16:43.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:43.103+0000 7faf8b200640 1 --2- 172.21.15.96:0/3430949740 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faf8c15e870 0x7faf8c14f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:43.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:43.103+0000 7faf89e00640 1 --2- 172.21.15.96:0/3430949740 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faf8c0d14d0 0x7faf8c15bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:43.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:43.103+0000 7faf8a800640 1 -- 172.21.15.96:0/3430949740 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faf8c15e870 msgr2=0x7faf8c14f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:43.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:43.103+0000 7faf8a800640 1 --2- 172.21.15.96:0/3430949740 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faf8c15e870 0x7faf8c14f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:43.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:43.103+0000 7faf8a800640 1 -- 172.21.15.96:0/3430949740 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faf8c0d14d0 msgr2=0x7faf8c15bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:43.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:43.103+0000 7faf8a800640 1 --2- 172.21.15.96:0/3430949740 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faf8c0d14d0 0x7faf8c15bf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:43.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:43.103+0000 7faf8a800640 1 -- 172.21.15.96:0/3430949740 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf8c15c980 con 0x7faf8c0cf870 2024-05-28T06:16:43.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:43.103+0000 7faf89e00640 1 --2- 172.21.15.96:0/3430949740 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faf8c0d14d0 0x7faf8c15bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:16:43.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:43.104+0000 7faf8a800640 1 --2- 172.21.15.96:0/3430949740 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf8c0cf870 0x7faf8c160dd0 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7faf7800d8f0 tx=0x7faf7800ddb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:43.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:43.104+0000 7faf82a00640 1 -- 172.21.15.96:0/3430949740 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf7800b660 con 0x7faf8c0cf870 2024-05-28T06:16:43.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:43.105+0000 7faf82a00640 1 -- 172.21.15.96:0/3430949740 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf78019070 con 0x7faf8c0cf870 2024-05-28T06:16:43.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:43.105+0000 7faf82a00640 1 -- 172.21.15.96:0/3430949740 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf7800ba20 con 0x7faf8c0cf870 2024-05-28T06:16:43.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:43.105+0000 7faf90a00640 1 -- 172.21.15.96:0/3430949740 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf8c15cc60 con 0x7faf8c0cf870 2024-05-28T06:16:43.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:43.105+0000 7faf90a00640 1 -- 172.21.15.96:0/3430949740 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf8c0b71a0 con 0x7faf8c0cf870 2024-05-28T06:16:43.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:43.109+0000 7faf77e00640 1 -- 172.21.15.96:0/3430949740 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf8c0689e0 con 0x7faf8c0cf870 2024-05-28T06:16:43.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:43.111+0000 7faf82a00640 1 -- 172.21.15.96:0/3430949740 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 28) ==== 92570+0+0 (secure 0 0 0) 0x7faf7800bbc0 con 0x7faf8c0cf870 2024-05-28T06:16:43.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:43.112+0000 7faf82a00640 1 --2- 172.21.15.96:0/3430949740 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7faf6c06d9d0 0x7faf6c06fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:43.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:43.112+0000 7faf82a00640 1 -- 172.21.15.96:0/3430949740 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5795+0+0 (secure 0 0 0) 0x7faf7808ab50 con 0x7faf8c0cf870 2024-05-28T06:16:43.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:43.118+0000 7faf82a00640 1 -- 172.21.15.96:0/3430949740 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7faf7805a9d0 con 0x7faf8c0cf870 2024-05-28T06:16:43.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:43.130+0000 7faf89e00640 1 --2- 172.21.15.96:0/3430949740 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7faf6c06d9d0 0x7faf6c06fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:43.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:43.149+0000 7faf89e00640 1 --2- 172.21.15.96:0/3430949740 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7faf6c06d9d0 0x7faf6c06fe90 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7faf7c033000 tx=0x7faf7c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:43.517 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:43.517+0000 7faf77e00640 1 -- 172.21.15.96:0/3430949740 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone436312", "target": ["mon-mgr", ""]}) -- 0x7faf8c0cf3c0 con 0x7faf6c06d9d0 2024-05-28T06:16:45.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:45.144+0000 7faf82a00640 1 -- 172.21.15.96:0/3430949740 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7faf8c0cf3c0 con 0x7faf6c06d9d0 2024-05-28T06:16:45.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:45.147+0000 7faf77e00640 1 -- 172.21.15.96:0/3430949740 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7faf6c06d9d0 msgr2=0x7faf6c06fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:45.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:45.147+0000 7faf77e00640 1 --2- 172.21.15.96:0/3430949740 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7faf6c06d9d0 0x7faf6c06fe90 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7faf7c033000 tx=0x7faf7c02e000 comp rx=0 tx=0).stop 2024-05-28T06:16:45.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:45.147+0000 7faf77e00640 1 -- 172.21.15.96:0/3430949740 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf8c0cf870 msgr2=0x7faf8c160dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:45.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:45.147+0000 7faf77e00640 1 --2- 172.21.15.96:0/3430949740 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf8c0cf870 0x7faf8c160dd0 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7faf7800d8f0 tx=0x7faf7800ddb0 comp rx=0 tx=0).stop 2024-05-28T06:16:45.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:45.148+0000 7faf77e00640 1 -- 172.21.15.96:0/3430949740 shutdown_connections 2024-05-28T06:16:45.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:45.148+0000 7faf77e00640 1 --2- 172.21.15.96:0/3430949740 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7faf6c06d9d0 0x7faf6c06fe90 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:45.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:45.148+0000 7faf77e00640 1 --2- 172.21.15.96:0/3430949740 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faf8c15e870 0x7faf8c14f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:45.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:45.148+0000 7faf77e00640 1 --2- 172.21.15.96:0/3430949740 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faf8c0d14d0 0x7faf8c15bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:45.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:45.148+0000 7faf77e00640 1 --2- 172.21.15.96:0/3430949740 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf8c0cf870 0x7faf8c160dd0 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:45.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:45.148+0000 7faf77e00640 1 -- 172.21.15.96:0/3430949740 >> 172.21.15.96:0/3430949740 conn(0x7faf8c08bfc0 msgr2=0x7faf8c0d2cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:16:45.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:45.148+0000 7faf77e00640 1 -- 172.21.15.96:0/3430949740 shutdown_connections 2024-05-28T06:16:45.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:45.148+0000 7faf77e00640 1 -- 172.21.15.96:0/3430949740 wait complete. 2024-05-28T06:16:45.202 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone436313 2024-05-28T06:16:45.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:45.447+0000 7fa39f400640 1 Processor -- start 2024-05-28T06:16:45.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:45.447+0000 7fa39f400640 1 -- start start 2024-05-28T06:16:45.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:45.449+0000 7fa39f400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa3980d17b0 0x7fa3980cf2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:45.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:45.449+0000 7fa39f400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fa3980b2e50 0x7fa3980cf810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:45.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:45.449+0000 7fa39f400640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3980b13d0 con 0x7fa398162130 2024-05-28T06:16:45.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:45.449+0000 7fa39f400640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fa3980d33b0 con 0x7fa3980b2e50 2024-05-28T06:16:45.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:45.449+0000 7fa39f400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fa3980b1640 con 0x7fa3980d17b0 2024-05-28T06:16:45.451 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:45.450+0000 7fa39d400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fa3980b2e50 0x7fa3980cf810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:45.451 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:45.450+0000 7fa39d400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fa3980b2e50 0x7fa3980cf810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48798/0 (socket says 172.21.15.96:48798) 2024-05-28T06:16:45.452 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:45.450+0000 7fa39d400640 1 -- 172.21.15.96:0/1571628342 learned_addr learned my addr 172.21.15.96:0/1571628342 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:16:45.452 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:45.451+0000 7fa39d400640 1 -- 172.21.15.96:0/1571628342 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa3980d17b0 msgr2=0x7fa3980cf2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:16:45.452 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:45.451+0000 7fa39d400640 1 --2- 172.21.15.96:0/1571628342 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa3980d17b0 0x7fa3980cf2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:45.452 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:45.451+0000 7fa39d400640 1 -- 172.21.15.96:0/1571628342 >> v1:172.21.15.123:6789/0 conn(0x7fa398162130 legacy=0x7fa3980d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:16:45.452 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:45.451+0000 7fa39d400640 1 -- 172.21.15.96:0/1571628342 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3980b1920 con 0x7fa3980b2e50 2024-05-28T06:16:45.452 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:45.453+0000 7fa39d400640 1 --2- 172.21.15.96:0/1571628342 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa3980b2e50 0x7fa3980cf810 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7fa390009870 tx=0x7fa39002a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e510e9c7de911f5 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:45.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:45.453+0000 7fa397400640 1 -- 172.21.15.96:0/1571628342 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa390037070 con 0x7fa3980b2e50 2024-05-28T06:16:45.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:45.453+0000 7fa397400640 1 -- 172.21.15.96:0/1571628342 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa390032440 con 0x7fa3980b2e50 2024-05-28T06:16:45.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:45.454+0000 7fa397400640 1 -- 172.21.15.96:0/1571628342 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa390032810 con 0x7fa3980b2e50 2024-05-28T06:16:45.454 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:45.455+0000 7fa39f400640 1 -- 172.21.15.96:0/1571628342 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa3980b2e50 msgr2=0x7fa3980cf810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:45.454 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:45.455+0000 7fa39f400640 1 --2- 172.21.15.96:0/1571628342 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa3980b2e50 0x7fa3980cf810 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7fa390009870 tx=0x7fa39002a210 comp rx=0 tx=0).stop 2024-05-28T06:16:45.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:45.456+0000 7fa39f400640 1 -- 172.21.15.96:0/1571628342 shutdown_connections 2024-05-28T06:16:45.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:45.456+0000 7fa39f400640 1 --2- 172.21.15.96:0/1571628342 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa3980b2e50 0x7fa3980cf810 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:45.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:45.456+0000 7fa39f400640 1 --2- 172.21.15.96:0/1571628342 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa3980d17b0 0x7fa3980cf2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:45.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:45.456+0000 7fa39f400640 1 -- 172.21.15.96:0/1571628342 >> 172.21.15.96:0/1571628342 conn(0x7fa39808bfc0 msgr2=0x7fa3980d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:16:45.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:45.456+0000 7fa39f400640 1 -- 172.21.15.96:0/1571628342 shutdown_connections 2024-05-28T06:16:45.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:45.456+0000 7fa39f400640 1 -- 172.21.15.96:0/1571628342 wait complete. 2024-05-28T06:16:45.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:45.460+0000 7fa39f400640 1 Processor -- start 2024-05-28T06:16:45.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:45.461+0000 7fa39f400640 1 -- start start 2024-05-28T06:16:45.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:45.464+0000 7fa39f400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa3980b2e50 0x7fa3980ba130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:45.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:45.464+0000 7fa39f400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa3980d17b0 0x7fa3980c8e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:45.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:45.464+0000 7fa39f400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa398162130 0x7fa3980b94d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:45.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:45.464+0000 7fa39f400640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa398162690 con 0x7fa398162130 2024-05-28T06:16:45.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:45.464+0000 7fa39f400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa398162510 con 0x7fa3980b2e50 2024-05-28T06:16:45.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:45.464+0000 7fa39f400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fa3980d3690 con 0x7fa3980d17b0 2024-05-28T06:16:45.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:45.465+0000 7fa39ca00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa3980b2e50 0x7fa3980ba130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:45.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:45.465+0000 7fa39ca00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa3980b2e50 0x7fa3980ba130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.96:35096/0 (socket says 172.21.15.96:35096) 2024-05-28T06:16:45.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:45.465+0000 7fa39ca00640 1 -- 172.21.15.96:0/989860913 learned_addr learned my addr 172.21.15.96:0/989860913 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:16:45.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:45.465+0000 7fa39d400640 1 --2- 172.21.15.96:0/989860913 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa398162130 0x7fa3980b94d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:45.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:45.465+0000 7fa397e00640 1 --2- 172.21.15.96:0/989860913 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa3980d17b0 0x7fa3980c8e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:45.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:45.466+0000 7fa39ca00640 1 -- 172.21.15.96:0/989860913 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa3980d17b0 msgr2=0x7fa3980c8e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:45.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:45.466+0000 7fa39ca00640 1 --2- 172.21.15.96:0/989860913 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa3980d17b0 0x7fa3980c8e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:45.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:45.466+0000 7fa39ca00640 1 -- 172.21.15.96:0/989860913 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa398162130 msgr2=0x7fa3980b94d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:45.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:45.466+0000 7fa39ca00640 1 --2- 172.21.15.96:0/989860913 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa398162130 0x7fa3980b94d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:45.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:45.466+0000 7fa39ca00640 1 -- 172.21.15.96:0/989860913 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3980b9a10 con 0x7fa3980b2e50 2024-05-28T06:16:45.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:45.466+0000 7fa39d400640 1 --2- 172.21.15.96:0/989860913 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa398162130 0x7fa3980b94d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-28T06:16:45.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:45.467+0000 7fa39ca00640 1 --2- 172.21.15.96:0/989860913 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa3980b2e50 0x7fa3980ba130 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fa38000b3f0 tx=0x7fa38000b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:45.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:45.467+0000 7fa394c00640 1 -- 172.21.15.96:0/989860913 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa380015020 con 0x7fa3980b2e50 2024-05-28T06:16:45.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:45.467+0000 7fa394c00640 1 -- 172.21.15.96:0/989860913 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa380011830 con 0x7fa3980b2e50 2024-05-28T06:16:45.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:45.467+0000 7fa394c00640 1 -- 172.21.15.96:0/989860913 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa380011c20 con 0x7fa3980b2e50 2024-05-28T06:16:45.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:45.467+0000 7fa39f400640 1 -- 172.21.15.96:0/989860913 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3980cdf80 con 0x7fa3980b2e50 2024-05-28T06:16:45.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:45.468+0000 7fa39f400640 1 -- 172.21.15.96:0/989860913 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3980ce520 con 0x7fa3980b2e50 2024-05-28T06:16:45.469 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:45.469+0000 7fa39f400640 1 -- 172.21.15.96:0/989860913 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3980690f0 con 0x7fa3980b2e50 2024-05-28T06:16:45.471 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:45.472+0000 7fa394c00640 1 -- 172.21.15.96:0/989860913 <== mon.1 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 28) ==== 92570+0+0 (secure 0 0 0) 0x7fa380018020 con 0x7fa3980b2e50 2024-05-28T06:16:45.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:45.473+0000 7fa394c00640 1 --2- 172.21.15.96:0/989860913 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa37406d9d0 0x7fa37406fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:45.473 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:45.473+0000 7fa394c00640 1 -- 172.21.15.96:0/989860913 <== mon.1 v2:172.21.15.123:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5795+0+0 (secure 0 0 0) 0x7fa38008a610 con 0x7fa3980b2e50 2024-05-28T06:16:45.473 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:45.474+0000 7fa397e00640 1 --2- 172.21.15.96:0/989860913 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa37406d9d0 0x7fa37406fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:45.475 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:45.475+0000 7fa397e00640 1 --2- 172.21.15.96:0/989860913 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa37406d9d0 0x7fa37406fe90 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fa39815e1a0 tx=0x7fa38800d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:45.481 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:45.481+0000 7fa394c00640 1 -- 172.21.15.96:0/989860913 <== mon.1 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa38001f030 con 0x7fa3980b2e50 2024-05-28T06:16:45.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:45.846+0000 7fa39f400640 1 -- 172.21.15.96:0/989860913 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone436313", "target": ["mon-mgr", ""]}) -- 0x7fa3980c93b0 con 0x7fa37406d9d0 2024-05-28T06:16:47.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.122+0000 7fa394c00640 1 -- 172.21.15.96:0/989860913 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fa3980c93b0 con 0x7fa37406d9d0 2024-05-28T06:16:47.125 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.126+0000 7fa39f400640 1 -- 172.21.15.96:0/989860913 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa37406d9d0 msgr2=0x7fa37406fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:47.125 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.126+0000 7fa39f400640 1 --2- 172.21.15.96:0/989860913 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa37406d9d0 0x7fa37406fe90 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fa39815e1a0 tx=0x7fa38800d040 comp rx=0 tx=0).stop 2024-05-28T06:16:47.125 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.126+0000 7fa39f400640 1 -- 172.21.15.96:0/989860913 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa3980b2e50 msgr2=0x7fa3980ba130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:47.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.126+0000 7fa39f400640 1 --2- 172.21.15.96:0/989860913 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa3980b2e50 0x7fa3980ba130 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fa38000b3f0 tx=0x7fa38000b8b0 comp rx=0 tx=0).stop 2024-05-28T06:16:47.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.127+0000 7fa39f400640 1 -- 172.21.15.96:0/989860913 shutdown_connections 2024-05-28T06:16:47.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.127+0000 7fa39f400640 1 --2- 172.21.15.96:0/989860913 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa37406d9d0 0x7fa37406fe90 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:47.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.127+0000 7fa39f400640 1 --2- 172.21.15.96:0/989860913 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa398162130 0x7fa3980b94d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:47.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.127+0000 7fa39f400640 1 --2- 172.21.15.96:0/989860913 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa3980d17b0 0x7fa3980c8e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:47.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.127+0000 7fa39f400640 1 --2- 172.21.15.96:0/989860913 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa3980b2e50 0x7fa3980ba130 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:47.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.127+0000 7fa39f400640 1 -- 172.21.15.96:0/989860913 >> 172.21.15.96:0/989860913 conn(0x7fa39808bfc0 msgr2=0x7fa398074990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:16:47.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.127+0000 7fa39f400640 1 -- 172.21.15.96:0/989860913 shutdown_connections 2024-05-28T06:16:47.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.127+0000 7fa39f400640 1 -- 172.21.15.96:0/989860913 wait complete. 2024-05-28T06:16:47.144 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone436314 2024-05-28T06:16:47.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.406+0000 7ff2ff400640 1 Processor -- start 2024-05-28T06:16:47.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.406+0000 7ff2ff400640 1 -- start start 2024-05-28T06:16:47.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.407+0000 7ff2ff400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff2f80d0640 0x7ff2f80d0a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:47.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.407+0000 7ff2ff400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7ff2f815b6f0 0x7ff2f806e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:47.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.407+0000 7ff2ff400640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2f80ade90 con 0x7ff2f80725e0 2024-05-28T06:16:47.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.407+0000 7ff2ff400640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7ff2f8071c40 con 0x7ff2f815b6f0 2024-05-28T06:16:47.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.407+0000 7ff2ff400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ff2f80d0f60 con 0x7ff2f80d0640 2024-05-28T06:16:47.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.408+0000 7ff2fd400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7ff2f80725e0 0x7ff2f80b44d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:50554/0 (socket says 172.21.15.96:50554) 2024-05-28T06:16:47.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.408+0000 7ff2fd400640 1 -- 172.21.15.96:0/2321845991 learned_addr learned my addr 172.21.15.96:0/2321845991 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:16:47.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.408+0000 7ff2f7e00640 1 --2- 172.21.15.96:0/2321845991 >> v2:172.21.15.123:3300/0 conn(0x7ff2f815b6f0 0x7ff2f806e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:47.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.409+0000 7ff2f7400640 1 -- 172.21.15.96:0/2321845991 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1981700879 0 0) 0x7ff2f80ade90 con 0x7ff2f80725e0 2024-05-28T06:16:47.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.409+0000 7ff2f7400640 1 -- 172.21.15.96:0/2321845991 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff2e0003520 con 0x7ff2f80725e0 2024-05-28T06:16:47.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.409+0000 7ff2f7400640 1 -- 172.21.15.96:0/2321845991 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3031947664 0 0) 0x7ff2e0003520 con 0x7ff2f80725e0 2024-05-28T06:16:47.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.409+0000 7ff2f7400640 1 -- 172.21.15.96:0/2321845991 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff2f80d0640 msgr2=0x7ff2f80d0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:16:47.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.409+0000 7ff2f7400640 1 --2- 172.21.15.96:0/2321845991 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff2f80d0640 0x7ff2f80d0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:47.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.409+0000 7ff2f7400640 1 -- 172.21.15.96:0/2321845991 >> v2:172.21.15.123:3300/0 conn(0x7ff2f815b6f0 msgr2=0x7ff2f806e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:16:47.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.409+0000 7ff2f7400640 1 --2- 172.21.15.96:0/2321845991 >> v2:172.21.15.123:3300/0 conn(0x7ff2f815b6f0 0x7ff2f806e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:47.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.409+0000 7ff2f7400640 1 -- 172.21.15.96:0/2321845991 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2f8071e70 con 0x7ff2f80725e0 2024-05-28T06:16:47.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.409+0000 7ff2f7400640 1 -- 172.21.15.96:0/2321845991 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7ff2f00019f0 con 0x7ff2f80725e0 2024-05-28T06:16:47.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.409+0000 7ff2f7400640 1 -- 172.21.15.96:0/2321845991 >> v1:172.21.15.96:6789/0 conn(0x7ff2f80725e0 legacy=0x7ff2f80b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:47.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.409+0000 7ff2f7400640 1 --2- 172.21.15.96:0/2321845991 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff2f80d0640 0x7ff2e00039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:47.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.409+0000 7ff2f7400640 1 --2- 172.21.15.96:0/2321845991 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff2e0003fe0 0x7ff2e0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:47.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.409+0000 7ff2f7400640 1 --2- 172.21.15.96:0/2321845991 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff2e0004d40 0x7ff2e00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:47.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.409+0000 7ff2f7400640 1 -- 172.21.15.96:0/2321845991 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff2f80d0f60 con 0x7ff2e0003fe0 2024-05-28T06:16:47.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.409+0000 7ff2f7400640 1 -- 172.21.15.96:0/2321845991 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff2e000bb50 con 0x7ff2e0004d40 2024-05-28T06:16:47.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.409+0000 7ff2f7400640 1 -- 172.21.15.96:0/2321845991 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ff2e000bd50 con 0x7ff2f80d0640 2024-05-28T06:16:47.410 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.410+0000 7ff2f7e00640 1 --2- 172.21.15.96:0/2321845991 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff2e0003fe0 0x7ff2e0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:47.410 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.410+0000 7ff2fd400640 1 --2- 172.21.15.96:0/2321845991 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff2e0004d40 0x7ff2e00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:47.410 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.410+0000 7ff2f7e00640 1 -- 172.21.15.96:0/2321845991 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff2f80d0640 msgr2=0x7ff2e00039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:16:47.410 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.410+0000 7ff2f7e00640 1 --2- 172.21.15.96:0/2321845991 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff2f80d0640 0x7ff2e00039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:47.410 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.410+0000 7ff2f7e00640 1 -- 172.21.15.96:0/2321845991 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff2e0004d40 msgr2=0x7ff2e00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:16:47.410 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.410+0000 7ff2f7e00640 1 --2- 172.21.15.96:0/2321845991 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff2e0004d40 0x7ff2e00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:47.410 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.411+0000 7ff2f7e00640 1 -- 172.21.15.96:0/2321845991 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2e0003520 con 0x7ff2e0003fe0 2024-05-28T06:16:47.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.412+0000 7ff2f7e00640 1 --2- 172.21.15.96:0/2321845991 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff2e0003fe0 0x7ff2e0004800 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7ff2e400ba30 tx=0x7ff2e0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1af9922101eb7ddc server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:47.414 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.413+0000 7ff2f7400640 1 -- 172.21.15.96:0/2321845991 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2e4030070 con 0x7ff2e0003fe0 2024-05-28T06:16:47.414 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.413+0000 7ff2f7400640 1 -- 172.21.15.96:0/2321845991 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2e402ae10 con 0x7ff2e0003fe0 2024-05-28T06:16:47.414 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.413+0000 7ff2f7400640 1 -- 172.21.15.96:0/2321845991 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2e400b600 con 0x7ff2e0003fe0 2024-05-28T06:16:47.414 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.415+0000 7ff2ff400640 1 -- 172.21.15.96:0/2321845991 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff2e0003fe0 msgr2=0x7ff2e0004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:47.415 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.415+0000 7ff2ff400640 1 --2- 172.21.15.96:0/2321845991 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff2e0003fe0 0x7ff2e0004800 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7ff2e400ba30 tx=0x7ff2e0005830 comp rx=0 tx=0).stop 2024-05-28T06:16:47.415 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.415+0000 7ff2fd400640 1 -- 172.21.15.96:0/2321845991 reap_dead start 2024-05-28T06:16:47.415 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.416+0000 7ff2ff400640 1 -- 172.21.15.96:0/2321845991 shutdown_connections 2024-05-28T06:16:47.415 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.416+0000 7ff2ff400640 1 -- 172.21.15.96:0/2321845991 >> 172.21.15.96:0/2321845991 conn(0x7ff2f806bd20 msgr2=0x7ff2f806c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:16:47.415 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.416+0000 7ff2ff400640 1 -- 172.21.15.96:0/2321845991 shutdown_connections 2024-05-28T06:16:47.415 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.416+0000 7ff2ff400640 1 -- 172.21.15.96:0/2321845991 wait complete. 2024-05-28T06:16:47.418 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.420+0000 7ff2ff400640 1 Processor -- start 2024-05-28T06:16:47.419 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.420+0000 7ff2ff400640 1 -- start start 2024-05-28T06:16:47.420 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.422+0000 7ff2ff400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff2e0003fe0 0x7ff2f814b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:47.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.422+0000 7ff2ff400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff2f815dba0 0x7ff2f815e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:47.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.422+0000 7ff2ff400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff2f815e560 0x7ff2f80cdb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:47.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.422+0000 7ff2ff400640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff2f806e890 con 0x7ff2f815dba0 2024-05-28T06:16:47.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.422+0000 7ff2ff400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff2f80ae130 con 0x7ff2e0003fe0 2024-05-28T06:16:47.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.422+0000 7ff2ff400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ff2f80af660 con 0x7ff2f815e560 2024-05-28T06:16:47.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.422+0000 7ff2f7e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff2f815dba0 0x7ff2f815e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:47.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.422+0000 7ff2f7e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff2f815dba0 0x7ff2f815e020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48834/0 (socket says 172.21.15.96:48834) 2024-05-28T06:16:47.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.422+0000 7ff2f7e00640 1 -- 172.21.15.96:0/3399830159 learned_addr learned my addr 172.21.15.96:0/3399830159 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:16:47.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.423+0000 7ff2fca00640 1 --2- 172.21.15.96:0/3399830159 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff2e0003fe0 0x7ff2f814b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:47.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.423+0000 7ff2f7e00640 1 -- 172.21.15.96:0/3399830159 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff2f815e560 msgr2=0x7ff2f80cdb60 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-28T06:16:47.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.423+0000 7ff2f7e00640 1 --2- 172.21.15.96:0/3399830159 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff2f815e560 0x7ff2f80cdb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:47.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.423+0000 7ff2f7e00640 1 -- 172.21.15.96:0/3399830159 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff2e0003fe0 msgr2=0x7ff2f814b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:47.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.423+0000 7ff2f7e00640 1 --2- 172.21.15.96:0/3399830159 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff2e0003fe0 0x7ff2f814b7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:47.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.423+0000 7ff2f7e00640 1 -- 172.21.15.96:0/3399830159 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2f815c8a0 con 0x7ff2f815dba0 2024-05-28T06:16:47.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.423+0000 7ff2fca00640 1 --2- 172.21.15.96:0/3399830159 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff2e0003fe0 0x7ff2f814b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:16:47.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.424+0000 7ff2f7e00640 1 --2- 172.21.15.96:0/3399830159 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff2f815dba0 0x7ff2f815e020 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7ff2e4036040 tx=0x7ff2e4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:47.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.425+0000 7ff2f4c00640 1 -- 172.21.15.96:0/3399830159 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2e4044070 con 0x7ff2f815dba0 2024-05-28T06:16:47.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.425+0000 7ff2f4c00640 1 -- 172.21.15.96:0/3399830159 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2e4004020 con 0x7ff2f815dba0 2024-05-28T06:16:47.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.425+0000 7ff2f4c00640 1 -- 172.21.15.96:0/3399830159 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2e40043f0 con 0x7ff2f815dba0 2024-05-28T06:16:47.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.425+0000 7ff2ff400640 1 -- 172.21.15.96:0/3399830159 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2f815cb80 con 0x7ff2f815dba0 2024-05-28T06:16:47.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.425+0000 7ff2ff400640 1 -- 172.21.15.96:0/3399830159 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2f815d680 con 0x7ff2f815dba0 2024-05-28T06:16:47.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.427+0000 7ff2ff400640 1 -- 172.21.15.96:0/3399830159 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2f80689e0 con 0x7ff2f815dba0 2024-05-28T06:16:47.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.428+0000 7ff2f4c00640 1 -- 172.21.15.96:0/3399830159 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 28) ==== 92570+0+0 (secure 0 0 0) 0x7ff2e403e6b0 con 0x7ff2f815dba0 2024-05-28T06:16:47.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.429+0000 7ff2f4c00640 1 --2- 172.21.15.96:0/3399830159 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff2d806da00 0x7ff2d806fec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:47.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.429+0000 7ff2f4c00640 1 -- 172.21.15.96:0/3399830159 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5795+0+0 (secure 0 0 0) 0x7ff2e40aee20 con 0x7ff2f815dba0 2024-05-28T06:16:47.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.439+0000 7ff2f4c00640 1 -- 172.21.15.96:0/3399830159 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff2e407ee20 con 0x7ff2f815dba0 2024-05-28T06:16:47.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.458+0000 7ff2fca00640 1 --2- 172.21.15.96:0/3399830159 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff2d806da00 0x7ff2d806fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:47.469 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.471+0000 7ff2fca00640 1 --2- 172.21.15.96:0/3399830159 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff2d806da00 0x7ff2d806fec0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7ff2ec006fd0 tx=0x7ff2ec008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:47.818 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:47.819+0000 7ff2ff400640 1 -- 172.21.15.96:0/3399830159 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone436314", "target": ["mon-mgr", ""]}) -- 0x7ff2f81494f0 con 0x7ff2d806da00 2024-05-28T06:16:49.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:49.027+0000 7ff2f4c00640 1 -- 172.21.15.96:0/3399830159 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7ff2f81494f0 con 0x7ff2d806da00 2024-05-28T06:16:49.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:49.032+0000 7ff2ff400640 1 -- 172.21.15.96:0/3399830159 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff2d806da00 msgr2=0x7ff2d806fec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:49.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:49.032+0000 7ff2ff400640 1 --2- 172.21.15.96:0/3399830159 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff2d806da00 0x7ff2d806fec0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7ff2ec006fd0 tx=0x7ff2ec008040 comp rx=0 tx=0).stop 2024-05-28T06:16:49.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:49.032+0000 7ff2ff400640 1 -- 172.21.15.96:0/3399830159 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff2f815dba0 msgr2=0x7ff2f815e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:49.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:49.032+0000 7ff2ff400640 1 --2- 172.21.15.96:0/3399830159 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff2f815dba0 0x7ff2f815e020 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7ff2e4036040 tx=0x7ff2e4004730 comp rx=0 tx=0).stop 2024-05-28T06:16:49.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:49.033+0000 7ff2ff400640 1 -- 172.21.15.96:0/3399830159 shutdown_connections 2024-05-28T06:16:49.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:49.033+0000 7ff2ff400640 1 --2- 172.21.15.96:0/3399830159 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff2d806da00 0x7ff2d806fec0 secure :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7ff2ec006fd0 tx=0x7ff2ec008040 comp rx=0 tx=0).stop 2024-05-28T06:16:49.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:49.033+0000 7ff2ff400640 1 --2- 172.21.15.96:0/3399830159 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff2f815e560 0x7ff2f80cdb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:49.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:49.033+0000 7ff2ff400640 1 --2- 172.21.15.96:0/3399830159 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff2f815dba0 0x7ff2f815e020 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:49.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:49.033+0000 7ff2ff400640 1 --2- 172.21.15.96:0/3399830159 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff2e0003fe0 0x7ff2f814b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:49.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:49.033+0000 7ff2ff400640 1 -- 172.21.15.96:0/3399830159 >> 172.21.15.96:0/3399830159 conn(0x7ff2f806bd20 msgr2=0x7ff2f815a9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:16:49.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:49.033+0000 7ff2ff400640 1 -- 172.21.15.96:0/3399830159 shutdown_connections 2024-05-28T06:16:49.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:49.033+0000 7ff2ff400640 1 -- 172.21.15.96:0/3399830159 wait complete. 2024-05-28T06:16:49.058 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone436315 2024-05-28T06:16:49.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:49.313+0000 7f21ca800640 1 Processor -- start 2024-05-28T06:16:49.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:49.314+0000 7f21ca800640 1 -- start start 2024-05-28T06:16:49.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:49.315+0000 7f21ca800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f21c40d4680 0x7f21c40d2d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:49.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:49.315+0000 7f21ca800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f21c40d3280 0x7f21c415e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:49.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:49.315+0000 7f21ca800640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21c40d0150 con 0x7f21c40d4a60 2024-05-28T06:16:49.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:49.315+0000 7f21ca800640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f21c40d28b0 con 0x7f21c40d3280 2024-05-28T06:16:49.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:49.315+0000 7f21ca800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f21c40d0430 con 0x7f21c40d4680 2024-05-28T06:16:49.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:49.316+0000 7f21c9e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f21c40d3280 0x7f21c415e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:49.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:49.316+0000 7f21c9e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f21c40d3280 0x7f21c415e2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:38946/0 (socket says 172.21.15.96:38946) 2024-05-28T06:16:49.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:49.316+0000 7f21c9e00640 1 -- 172.21.15.96:0/3495197898 learned_addr learned my addr 172.21.15.96:0/3495197898 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:16:49.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:49.317+0000 7f21c9e00640 1 -- 172.21.15.96:0/3495197898 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f21c40d4680 msgr2=0x7f21c40d2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:16:49.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:49.317+0000 7f21c9e00640 1 --2- 172.21.15.96:0/3495197898 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f21c40d4680 0x7f21c40d2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:49.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:49.317+0000 7f21c9e00640 1 -- 172.21.15.96:0/3495197898 >> v1:172.21.15.123:6789/0 conn(0x7f21c40d4a60 legacy=0x7f21c40b46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:49.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:49.317+0000 7f21c9e00640 1 -- 172.21.15.96:0/3495197898 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21c40d2260 con 0x7f21c40d3280 2024-05-28T06:16:49.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:49.318+0000 7f21c9e00640 1 --2- 172.21.15.96:0/3495197898 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f21c40d3280 0x7f21c415e2e0 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f21bc009e50 tx=0x7f21bc02a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa9607a035492912 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:49.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:49.318+0000 7f21c3e00640 1 -- 172.21.15.96:0/3495197898 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21bc036070 con 0x7f21c40d3280 2024-05-28T06:16:49.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:49.319+0000 7f21c3e00640 1 -- 172.21.15.96:0/3495197898 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21bc031600 con 0x7f21c40d3280 2024-05-28T06:16:49.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:49.319+0000 7f21c3e00640 1 -- 172.21.15.96:0/3495197898 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21bc0319d0 con 0x7f21c40d3280 2024-05-28T06:16:49.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:49.321+0000 7f21ca800640 1 -- 172.21.15.96:0/3495197898 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f21c40d3280 msgr2=0x7f21c415e2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:49.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:49.321+0000 7f21ca800640 1 --2- 172.21.15.96:0/3495197898 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f21c40d3280 0x7f21c415e2e0 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f21bc009e50 tx=0x7f21bc02a710 comp rx=0 tx=0).stop 2024-05-28T06:16:49.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:49.321+0000 7f21ca800640 1 -- 172.21.15.96:0/3495197898 shutdown_connections 2024-05-28T06:16:49.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:49.321+0000 7f21ca800640 1 --2- 172.21.15.96:0/3495197898 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f21c40d3280 0x7f21c415e2e0 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:49.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:49.321+0000 7f21ca800640 1 --2- 172.21.15.96:0/3495197898 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f21c40d4680 0x7f21c40d2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:49.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:49.321+0000 7f21ca800640 1 -- 172.21.15.96:0/3495197898 >> 172.21.15.96:0/3495197898 conn(0x7f21c409d810 msgr2=0x7f21c408bfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:16:49.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:49.321+0000 7f21ca800640 1 -- 172.21.15.96:0/3495197898 shutdown_connections 2024-05-28T06:16:49.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:49.322+0000 7f21ca800640 1 -- 172.21.15.96:0/3495197898 wait complete. 2024-05-28T06:16:49.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:49.324+0000 7f21ca800640 1 Processor -- start 2024-05-28T06:16:49.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:49.324+0000 7f21ca800640 1 -- start start 2024-05-28T06:16:49.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:49.325+0000 7f21ca800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f21c40d3280 0x7f21c40b9940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:49.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:49.325+0000 7f21ca800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f21c40d4680 0x7f21c40b7050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:49.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:49.325+0000 7f21ca800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f21c40d4a60 0x7f21c4153610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:49.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:49.325+0000 7f21ca800640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f21c40b4f30 con 0x7f21c40d3280 2024-05-28T06:16:49.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:49.325+0000 7f21ca800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f21c40b4db0 con 0x7f21c40d4a60 2024-05-28T06:16:49.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:49.325+0000 7f21ca800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f21c415e820 con 0x7f21c40d4680 2024-05-28T06:16:49.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:49.325+0000 7f21c9400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f21c40d3280 0x7f21c40b9940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:49.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:49.325+0000 7f21c9400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f21c40d3280 0x7f21c40b9940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:38956/0 (socket says 172.21.15.96:38956) 2024-05-28T06:16:49.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:49.325+0000 7f21c9400640 1 -- 172.21.15.96:0/1164009882 learned_addr learned my addr 172.21.15.96:0/1164009882 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:16:49.325 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:49.326+0000 7f21c9e00640 1 --2- 172.21.15.96:0/1164009882 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f21c40d4a60 0x7f21c4153610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:49.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:49.326+0000 7f21c9400640 1 -- 172.21.15.96:0/1164009882 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f21c40d4680 msgr2=0x7f21c40b7050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:49.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:49.326+0000 7f21c9400640 1 --2- 172.21.15.96:0/1164009882 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f21c40d4680 0x7f21c40b7050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:49.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:49.326+0000 7f21c9400640 1 -- 172.21.15.96:0/1164009882 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f21c40d4a60 msgr2=0x7f21c4153610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:49.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:49.326+0000 7f21c9400640 1 --2- 172.21.15.96:0/1164009882 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f21c40d4a60 0x7f21c4153610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:49.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:49.326+0000 7f21c9400640 1 -- 172.21.15.96:0/1164009882 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21c40b8ad0 con 0x7f21c40d3280 2024-05-28T06:16:49.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:49.326+0000 7f21c9400640 1 --2- 172.21.15.96:0/1164009882 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f21c40d3280 0x7f21c40b9940 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f21b800b4a0 tx=0x7f21b800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:49.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:49.326+0000 7f21c1600640 1 -- 172.21.15.96:0/1164009882 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21b8014070 con 0x7f21c40d3280 2024-05-28T06:16:49.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:49.326+0000 7f21c1600640 1 -- 172.21.15.96:0/1164009882 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21b8010610 con 0x7f21c40d3280 2024-05-28T06:16:49.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:49.326+0000 7f21ca800640 1 -- 172.21.15.96:0/1164009882 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21c40b7a30 con 0x7f21c40d3280 2024-05-28T06:16:49.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:49.326+0000 7f21c1600640 1 -- 172.21.15.96:0/1164009882 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21b8010a00 con 0x7f21c40d3280 2024-05-28T06:16:49.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:49.327+0000 7f21ca800640 1 -- 172.21.15.96:0/1164009882 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21c40b82d0 con 0x7f21c40d3280 2024-05-28T06:16:49.327 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:49.328+0000 7f21ca800640 1 -- 172.21.15.96:0/1164009882 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21c40689e0 con 0x7f21c40d3280 2024-05-28T06:16:49.328 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:49.329+0000 7f21c1600640 1 -- 172.21.15.96:0/1164009882 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 28) ==== 92570+0+0 (secure 0 0 0) 0x7f21b8017070 con 0x7f21c40d3280 2024-05-28T06:16:49.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:49.331+0000 7f21c1600640 1 --2- 172.21.15.96:0/1164009882 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f21a406d9d0 0x7f21a406fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:49.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:49.331+0000 7f21c1600640 1 -- 172.21.15.96:0/1164009882 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5795+0+0 (secure 0 0 0) 0x7f21b808a510 con 0x7f21c40d3280 2024-05-28T06:16:49.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:49.337+0000 7f21c8a00640 1 --2- 172.21.15.96:0/1164009882 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f21a406d9d0 0x7f21a406fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:49.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:49.342+0000 7f21c1600640 1 -- 172.21.15.96:0/1164009882 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f21b805a050 con 0x7f21c40d3280 2024-05-28T06:16:49.378 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:49.380+0000 7f21c8a00640 1 --2- 172.21.15.96:0/1164009882 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f21a406d9d0 0x7f21a406fe90 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f21b0000fb0 tx=0x7f21b00079c0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:49.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:49.710+0000 7f21ca800640 1 -- 172.21.15.96:0/1164009882 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone436315", "target": ["mon-mgr", ""]}) -- 0x7f21c40b4a90 con 0x7f21a406d9d0 2024-05-28T06:16:51.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:51.073+0000 7f21c1600640 1 -- 172.21.15.96:0/1164009882 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f21c40b4a90 con 0x7f21a406d9d0 2024-05-28T06:16:51.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:51.082+0000 7f21ca800640 1 -- 172.21.15.96:0/1164009882 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f21a406d9d0 msgr2=0x7f21a406fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:51.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:51.082+0000 7f21ca800640 1 --2- 172.21.15.96:0/1164009882 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f21a406d9d0 0x7f21a406fe90 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f21b0000fb0 tx=0x7f21b00079c0 comp rx=0 tx=0).stop 2024-05-28T06:16:51.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:51.082+0000 7f21ca800640 1 -- 172.21.15.96:0/1164009882 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f21c40d3280 msgr2=0x7f21c40b9940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:51.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:51.082+0000 7f21ca800640 1 --2- 172.21.15.96:0/1164009882 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f21c40d3280 0x7f21c40b9940 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f21b800b4a0 tx=0x7f21b800b960 comp rx=0 tx=0).stop 2024-05-28T06:16:51.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:51.082+0000 7f21ca800640 1 -- 172.21.15.96:0/1164009882 shutdown_connections 2024-05-28T06:16:51.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:51.082+0000 7f21ca800640 1 --2- 172.21.15.96:0/1164009882 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f21a406d9d0 0x7f21a406fe90 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:51.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:51.082+0000 7f21ca800640 1 --2- 172.21.15.96:0/1164009882 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f21c40d4a60 0x7f21c4153610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:51.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:51.082+0000 7f21ca800640 1 --2- 172.21.15.96:0/1164009882 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f21c40d4680 0x7f21c40b7050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:51.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:51.082+0000 7f21ca800640 1 --2- 172.21.15.96:0/1164009882 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f21c40d3280 0x7f21c40b9940 secure :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f21b800b4a0 tx=0x7f21b800b960 comp rx=0 tx=0).stop 2024-05-28T06:16:51.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:51.082+0000 7f21ca800640 1 -- 172.21.15.96:0/1164009882 >> 172.21.15.96:0/1164009882 conn(0x7f21c409d810 msgr2=0x7f21c40d0670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:16:51.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:51.084+0000 7f21ca800640 1 -- 172.21.15.96:0/1164009882 shutdown_connections 2024-05-28T06:16:51.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:51.085+0000 7f21ca800640 1 -- 172.21.15.96:0/1164009882 wait complete. 2024-05-28T06:16:51.125 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone436316 2024-05-28T06:16:51.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:51.400+0000 7f67e0c00640 1 Processor -- start 2024-05-28T06:16:51.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:51.400+0000 7f67e0c00640 1 -- start start 2024-05-28T06:16:51.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:51.401+0000 7f67e0c00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f67dc0cef30 0x7f67dc0d1640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:51.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:51.402+0000 7f67e0c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f67dc0d1b80 0x7f67dc0b90a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:51.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:51.402+0000 7f67e0c00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67dc0d1f90 con 0x7f67dc15e850 2024-05-28T06:16:51.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:51.402+0000 7f67e0c00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f67dc15ec30 con 0x7f67dc0cef30 2024-05-28T06:16:51.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:51.402+0000 7f67e0c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f67dc0b8000 con 0x7f67dc0d1b80 2024-05-28T06:16:51.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:51.402+0000 7f67d9400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f67dc0cef30 0x7f67dc0d1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:51.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:51.402+0000 7f67d9400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f67dc0cef30 0x7f67dc0d1640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:38958/0 (socket says 172.21.15.96:38958) 2024-05-28T06:16:51.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:51.402+0000 7f67d9400640 1 -- 172.21.15.96:0/2253554660 learned_addr learned my addr 172.21.15.96:0/2253554660 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:16:51.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:51.402+0000 7f67da800640 1 --2- 172.21.15.96:0/2253554660 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f67dc0d1b80 0x7f67dc0b90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:51.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:51.402+0000 7f67d9400640 1 -- 172.21.15.96:0/2253554660 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f67dc0d1b80 msgr2=0x7f67dc0b90a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:16:51.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:51.402+0000 7f67d9400640 1 --2- 172.21.15.96:0/2253554660 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f67dc0d1b80 0x7f67dc0b90a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:51.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:51.402+0000 7f67d9400640 1 -- 172.21.15.96:0/2253554660 >> v1:172.21.15.96:6789/0 conn(0x7f67dc15e850 legacy=0x7f67dc0b77b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:51.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:51.402+0000 7f67d9400640 1 -- 172.21.15.96:0/2253554660 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67dc0b82e0 con 0x7f67dc0cef30 2024-05-28T06:16:51.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:51.403+0000 7f67d9400640 1 --2- 172.21.15.96:0/2253554660 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f67dc0cef30 0x7f67dc0d1640 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f67c8009870 tx=0x7f67c802a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=233a60768e3d2d30 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:51.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:51.404+0000 7f67d8a00640 1 -- 172.21.15.96:0/2253554660 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67c8037070 con 0x7f67dc0cef30 2024-05-28T06:16:51.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:51.404+0000 7f67d8a00640 1 -- 172.21.15.96:0/2253554660 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67c8032440 con 0x7f67dc0cef30 2024-05-28T06:16:51.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:51.404+0000 7f67d8a00640 1 -- 172.21.15.96:0/2253554660 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67c8032810 con 0x7f67dc0cef30 2024-05-28T06:16:51.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:51.405+0000 7f67e0c00640 1 -- 172.21.15.96:0/2253554660 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f67dc0cef30 msgr2=0x7f67dc0d1640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:51.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:51.405+0000 7f67e0c00640 1 --2- 172.21.15.96:0/2253554660 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f67dc0cef30 0x7f67dc0d1640 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f67c8009870 tx=0x7f67c802a1c0 comp rx=0 tx=0).stop 2024-05-28T06:16:51.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:51.405+0000 7f67e0c00640 1 -- 172.21.15.96:0/2253554660 shutdown_connections 2024-05-28T06:16:51.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:51.405+0000 7f67e0c00640 1 --2- 172.21.15.96:0/2253554660 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f67dc0d1b80 0x7f67dc0b90a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:51.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:51.405+0000 7f67e0c00640 1 --2- 172.21.15.96:0/2253554660 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f67dc0cef30 0x7f67dc0d1640 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:51.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:51.405+0000 7f67e0c00640 1 -- 172.21.15.96:0/2253554660 >> 172.21.15.96:0/2253554660 conn(0x7f67dc067aa0 msgr2=0x7f67dc067eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:16:51.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:51.405+0000 7f67e0c00640 1 -- 172.21.15.96:0/2253554660 shutdown_connections 2024-05-28T06:16:51.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:51.405+0000 7f67e0c00640 1 -- 172.21.15.96:0/2253554660 wait complete. 2024-05-28T06:16:51.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:51.408+0000 7f67e0c00640 1 Processor -- start 2024-05-28T06:16:51.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:51.409+0000 7f67e0c00640 1 -- start start 2024-05-28T06:16:51.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:51.411+0000 7f67e0c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f67dc0cef30 0x7f67dc0d4780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:51.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:51.411+0000 7f67e0c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f67dc0d1b80 0x7f67dc0d2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:51.410 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:51.411+0000 7f67e0c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67dc15e850 0x7f67dc0d26b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:51.410 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:51.411+0000 7f67e0c00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f67dc0b95e0 con 0x7f67dc0d1b80 2024-05-28T06:16:51.410 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:51.411+0000 7f67e0c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f67dc0b88a0 con 0x7f67dc15e850 2024-05-28T06:16:51.410 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:51.411+0000 7f67e0c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f67dc0b9760 con 0x7f67dc0cef30 2024-05-28T06:16:51.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:51.411+0000 7f67d9400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f67dc0d1b80 0x7f67dc0d2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:51.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:51.412+0000 7f67d9400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f67dc0d1b80 0x7f67dc0d2170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:38962/0 (socket says 172.21.15.96:38962) 2024-05-28T06:16:51.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:51.412+0000 7f67d9400640 1 -- 172.21.15.96:0/2077086089 learned_addr learned my addr 172.21.15.96:0/2077086089 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:16:51.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:51.412+0000 7f67da800640 1 --2- 172.21.15.96:0/2077086089 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67dc15e850 0x7f67dc0d26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:51.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:51.412+0000 7f67d9400640 1 -- 172.21.15.96:0/2077086089 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f67dc0cef30 msgr2=0x7f67dc0d4780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:51.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:51.412+0000 7f67d9400640 1 --2- 172.21.15.96:0/2077086089 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f67dc0cef30 0x7f67dc0d4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:51.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:51.412+0000 7f67d9400640 1 -- 172.21.15.96:0/2077086089 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67dc15e850 msgr2=0x7f67dc0d26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:51.412 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:51.412+0000 7f67d9400640 1 --2- 172.21.15.96:0/2077086089 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67dc15e850 0x7f67dc0d26b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:51.412 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:51.412+0000 7f67d9400640 1 -- 172.21.15.96:0/2077086089 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67dc1f88c0 con 0x7f67dc0d1b80 2024-05-28T06:16:51.412 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:51.412+0000 7f67da800640 1 --2- 172.21.15.96:0/2077086089 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67dc15e850 0x7f67dc0d26b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-28T06:16:51.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:51.413+0000 7f67d9400640 1 --2- 172.21.15.96:0/2077086089 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f67dc0d1b80 0x7f67dc0d2170 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f67c802a200 tx=0x7f67c80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:51.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:51.413+0000 7f67d2000640 1 -- 172.21.15.96:0/2077086089 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67c8041020 con 0x7f67dc0d1b80 2024-05-28T06:16:51.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:51.413+0000 7f67d2000640 1 -- 172.21.15.96:0/2077086089 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67c80306c0 con 0x7f67dc0d1b80 2024-05-28T06:16:51.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:51.414+0000 7f67e0c00640 1 -- 172.21.15.96:0/2077086089 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67dc0d2d80 con 0x7f67dc0d1b80 2024-05-28T06:16:51.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:51.414+0000 7f67d2000640 1 -- 172.21.15.96:0/2077086089 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67c8037040 con 0x7f67dc0d1b80 2024-05-28T06:16:51.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:51.414+0000 7f67e0c00640 1 -- 172.21.15.96:0/2077086089 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67dc0d3340 con 0x7f67dc0d1b80 2024-05-28T06:16:51.414 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:51.415+0000 7f67e0c00640 1 -- 172.21.15.96:0/2077086089 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67dc0689e0 con 0x7f67dc0d1b80 2024-05-28T06:16:51.416 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:51.417+0000 7f67d2000640 1 -- 172.21.15.96:0/2077086089 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 28) ==== 92570+0+0 (secure 0 0 0) 0x7f67c8044020 con 0x7f67dc0d1b80 2024-05-28T06:16:51.418 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:51.420+0000 7f67d2000640 1 --2- 172.21.15.96:0/2077086089 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f67bc06da20 0x7f67bc06fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:51.418 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:51.420+0000 7f67d2000640 1 -- 172.21.15.96:0/2077086089 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5795+0+0 (secure 0 0 0) 0x7f67c80aabd0 con 0x7f67dc0d1b80 2024-05-28T06:16:51.419 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:51.420+0000 7f67d9e00640 1 --2- 172.21.15.96:0/2077086089 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f67bc06da20 0x7f67bc06fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:51.420 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:51.421+0000 7f67d9e00640 1 --2- 172.21.15.96:0/2077086089 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f67bc06da20 0x7f67bc06fee0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f67dc0cb2b0 tx=0x7f67cc007970 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:51.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:51.425+0000 7f67d2000640 1 -- 172.21.15.96:0/2077086089 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f67c804b070 con 0x7f67dc0d1b80 2024-05-28T06:16:51.775 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:51.776+0000 7f67e0c00640 1 -- 172.21.15.96:0/2077086089 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone436316", "target": ["mon-mgr", ""]}) -- 0x7f67dc162d60 con 0x7f67bc06da20 2024-05-28T06:16:52.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:52.883+0000 7f67d2000640 1 -- 172.21.15.96:0/2077086089 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f67dc162d60 con 0x7f67bc06da20 2024-05-28T06:16:52.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:52.888+0000 7f67e0c00640 1 -- 172.21.15.96:0/2077086089 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f67bc06da20 msgr2=0x7f67bc06fee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:52.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:52.888+0000 7f67e0c00640 1 --2- 172.21.15.96:0/2077086089 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f67bc06da20 0x7f67bc06fee0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f67dc0cb2b0 tx=0x7f67cc007970 comp rx=0 tx=0).stop 2024-05-28T06:16:52.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:52.888+0000 7f67e0c00640 1 -- 172.21.15.96:0/2077086089 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f67dc0d1b80 msgr2=0x7f67dc0d2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:52.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:52.888+0000 7f67e0c00640 1 --2- 172.21.15.96:0/2077086089 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f67dc0d1b80 0x7f67dc0d2170 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f67c802a200 tx=0x7f67c80046c0 comp rx=0 tx=0).stop 2024-05-28T06:16:52.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:52.888+0000 7f67e0c00640 1 -- 172.21.15.96:0/2077086089 shutdown_connections 2024-05-28T06:16:52.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:52.888+0000 7f67e0c00640 1 --2- 172.21.15.96:0/2077086089 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f67bc06da20 0x7f67bc06fee0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:52.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:52.888+0000 7f67e0c00640 1 --2- 172.21.15.96:0/2077086089 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67dc15e850 0x7f67dc0d26b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:52.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:52.888+0000 7f67e0c00640 1 --2- 172.21.15.96:0/2077086089 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f67dc0d1b80 0x7f67dc0d2170 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:52.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:52.888+0000 7f67e0c00640 1 --2- 172.21.15.96:0/2077086089 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f67dc0cef30 0x7f67dc0d4780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:52.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:52.888+0000 7f67e0c00640 1 -- 172.21.15.96:0/2077086089 >> 172.21.15.96:0/2077086089 conn(0x7f67dc067aa0 msgr2=0x7f67dc0b7700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:16:52.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:52.888+0000 7f67e0c00640 1 -- 172.21.15.96:0/2077086089 shutdown_connections 2024-05-28T06:16:52.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:52.888+0000 7f67e0c00640 1 -- 172.21.15.96:0/2077086089 wait complete. 2024-05-28T06:16:52.905 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone436317 2024-05-28T06:16:53.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:53.116+0000 7f6824200640 1 Processor -- start 2024-05-28T06:16:53.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:53.117+0000 7f6824200640 1 -- start start 2024-05-28T06:16:53.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:53.119+0000 7f6824200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f681c15b6f0 0x7f681c06e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:53.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:53.119+0000 7f6824200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f681c0725e0 0x7f681c0ae650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:53.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:53.119+0000 7f6824200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f681c0ade90 con 0x7f681c0d0640 2024-05-28T06:16:53.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:53.119+0000 7f6824200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f681c071c40 con 0x7f681c15b6f0 2024-05-28T06:16:53.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:53.119+0000 7f6824200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f681c071e70 con 0x7f681c0725e0 2024-05-28T06:16:53.119 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:53.120+0000 7f6820e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f681c15b6f0 0x7f681c06e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:53.119 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:53.120+0000 7f6820e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f681c15b6f0 0x7f681c06e350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:38974/0 (socket says 172.21.15.96:38974) 2024-05-28T06:16:53.119 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:53.120+0000 7f6820e00640 1 -- 172.21.15.96:0/3913156427 learned_addr learned my addr 172.21.15.96:0/3913156427 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:16:53.119 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:53.120+0000 7f6820e00640 1 -- 172.21.15.96:0/3913156427 >> v2:172.21.15.123:3300/0 conn(0x7f681c0725e0 msgr2=0x7f681c0ae650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:16:53.119 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:53.120+0000 7f6820e00640 1 --2- 172.21.15.96:0/3913156427 >> v2:172.21.15.123:3300/0 conn(0x7f681c0725e0 0x7f681c0ae650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:53.120 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:53.120+0000 7f6820e00640 1 -- 172.21.15.96:0/3913156427 >> v1:172.21.15.123:6789/0 conn(0x7f681c0d0640 legacy=0x7f681c0d0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:16:53.120 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:53.120+0000 7f6820e00640 1 -- 172.21.15.96:0/3913156427 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f681c072130 con 0x7f681c15b6f0 2024-05-28T06:16:53.120 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:53.121+0000 7f6820e00640 1 --2- 172.21.15.96:0/3913156427 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f681c15b6f0 0x7f681c06e350 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f681c0b3370 tx=0x7f680802a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b09d4dc18c4540c9 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:53.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:53.122+0000 7f681be00640 1 -- 172.21.15.96:0/3913156427 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6808037070 con 0x7f681c15b6f0 2024-05-28T06:16:53.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:53.122+0000 7f681be00640 1 -- 172.21.15.96:0/3913156427 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6808032440 con 0x7f681c15b6f0 2024-05-28T06:16:53.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:53.122+0000 7f681be00640 1 -- 172.21.15.96:0/3913156427 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6808032810 con 0x7f681c15b6f0 2024-05-28T06:16:53.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:53.123+0000 7f6824200640 1 -- 172.21.15.96:0/3913156427 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f681c15b6f0 msgr2=0x7f681c06e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:53.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:53.123+0000 7f6824200640 1 --2- 172.21.15.96:0/3913156427 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f681c15b6f0 0x7f681c06e350 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f681c0b3370 tx=0x7f680802a230 comp rx=0 tx=0).stop 2024-05-28T06:16:53.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:53.123+0000 7f6824200640 1 -- 172.21.15.96:0/3913156427 shutdown_connections 2024-05-28T06:16:53.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:53.123+0000 7f6824200640 1 --2- 172.21.15.96:0/3913156427 >> v2:172.21.15.123:3300/0 conn(0x7f681c0725e0 0x7f681c0ae650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:53.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:53.123+0000 7f6824200640 1 --2- 172.21.15.96:0/3913156427 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f681c15b6f0 0x7f681c06e350 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:53.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:53.123+0000 7f6824200640 1 -- 172.21.15.96:0/3913156427 >> 172.21.15.96:0/3913156427 conn(0x7f681c06bd20 msgr2=0x7f681c06c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:16:53.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:53.124+0000 7f6824200640 1 -- 172.21.15.96:0/3913156427 shutdown_connections 2024-05-28T06:16:53.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:53.124+0000 7f6824200640 1 -- 172.21.15.96:0/3913156427 wait complete. 2024-05-28T06:16:53.125 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:53.127+0000 7f6824200640 1 Processor -- start 2024-05-28T06:16:53.125 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:53.127+0000 7f6824200640 1 -- start start 2024-05-28T06:16:53.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:53.128+0000 7f6824200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f681c0725e0 0x7f681c146940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:53.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:53.128+0000 7f6824200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f681c0d0640 0x7f681c0cc740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:53.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:53.128+0000 7f6824200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f681c15b6f0 0x7f681c1553c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:53.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:53.128+0000 7f6824200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f681c0aeb90 con 0x7f681c0725e0 2024-05-28T06:16:53.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:53.128+0000 7f6824200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f681c06e890 con 0x7f681c0d0640 2024-05-28T06:16:53.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:53.128+0000 7f6824200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f681c0aed10 con 0x7f681c15b6f0 2024-05-28T06:16:53.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:53.129+0000 7f6821800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f681c0725e0 0x7f681c146940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:53.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:53.129+0000 7f6821800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f681c0725e0 0x7f681c146940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:38986/0 (socket says 172.21.15.96:38986) 2024-05-28T06:16:53.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:53.129+0000 7f6821800640 1 -- 172.21.15.96:0/3818349292 learned_addr learned my addr 172.21.15.96:0/3818349292 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:16:53.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:53.129+0000 7f6820e00640 1 --2- 172.21.15.96:0/3818349292 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f681c0d0640 0x7f681c0cc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:53.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:53.129+0000 7f6822200640 1 --2- 172.21.15.96:0/3818349292 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f681c15b6f0 0x7f681c1553c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:53.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:53.130+0000 7f6822200640 1 -- 172.21.15.96:0/3818349292 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f681c0d0640 msgr2=0x7f681c0cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:53.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:53.130+0000 7f6822200640 1 --2- 172.21.15.96:0/3818349292 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f681c0d0640 0x7f681c0cc740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:53.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:53.130+0000 7f6822200640 1 -- 172.21.15.96:0/3818349292 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f681c0725e0 msgr2=0x7f681c146940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:53.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:53.130+0000 7f6822200640 1 --2- 172.21.15.96:0/3818349292 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f681c0725e0 0x7f681c146940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:53.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:53.130+0000 7f6822200640 1 -- 172.21.15.96:0/3818349292 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f681c0cd040 con 0x7f681c15b6f0 2024-05-28T06:16:53.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:53.132+0000 7f6822200640 1 --2- 172.21.15.96:0/3818349292 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f681c15b6f0 0x7f681c1553c0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f681400be20 tx=0x7f681400bf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:53.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:53.134+0000 7f6819600640 1 -- 172.21.15.96:0/3818349292 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6814019070 con 0x7f681c15b6f0 2024-05-28T06:16:53.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:53.134+0000 7f6819600640 1 -- 172.21.15.96:0/3818349292 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6814014440 con 0x7f681c15b6f0 2024-05-28T06:16:53.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:53.134+0000 7f6819600640 1 -- 172.21.15.96:0/3818349292 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6814014830 con 0x7f681c15b6f0 2024-05-28T06:16:53.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:53.134+0000 7f6824200640 1 -- 172.21.15.96:0/3818349292 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f681c0cd320 con 0x7f681c15b6f0 2024-05-28T06:16:53.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:53.135+0000 7f6824200640 1 -- 172.21.15.96:0/3818349292 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f681c0cd8e0 con 0x7f681c15b6f0 2024-05-28T06:16:53.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:53.136+0000 7f6824200640 1 -- 172.21.15.96:0/3818349292 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f681c0689e0 con 0x7f681c15b6f0 2024-05-28T06:16:53.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:53.139+0000 7f6819600640 1 -- 172.21.15.96:0/3818349292 <== mon.2 v2:172.21.15.123:3301/0 4 ==== mgrmap(e 28) ==== 92570+0+0 (secure 0 0 0) 0x7f681401c070 con 0x7f681c15b6f0 2024-05-28T06:16:53.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:53.140+0000 7f6819600640 1 --2- 172.21.15.96:0/3818349292 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f67fc06da50 0x7f67fc06ff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:53.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:53.140+0000 7f6819600640 1 -- 172.21.15.96:0/3818349292 <== mon.2 v2:172.21.15.123:3301/0 5 ==== osd_map(76..76 src has 1..76) ==== 5795+0+0 (secure 0 0 0) 0x7f681408adb0 con 0x7f681c15b6f0 2024-05-28T06:16:53.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:53.143+0000 7f6821800640 1 --2- 172.21.15.96:0/3818349292 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f67fc06da50 0x7f67fc06ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:53.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:53.154+0000 7f6819600640 1 -- 172.21.15.96:0/3818349292 <== mon.2 v2:172.21.15.123:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6814023020 con 0x7f681c15b6f0 2024-05-28T06:16:53.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:53.156+0000 7f6821800640 1 --2- 172.21.15.96:0/3818349292 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f67fc06da50 0x7f67fc06ff10 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f6810001770 tx=0x7f6810009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:53.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:53.521+0000 7f6824200640 1 -- 172.21.15.96:0/3818349292 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone436317", "target": ["mon-mgr", ""]}) -- 0x7f681c0b3560 con 0x7f67fc06da50 2024-05-28T06:16:54.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:54.620+0000 7f6819600640 1 -- 172.21.15.96:0/3818349292 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f681c0b3560 con 0x7f67fc06da50 2024-05-28T06:16:54.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:54.624+0000 7f6824200640 1 -- 172.21.15.96:0/3818349292 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f67fc06da50 msgr2=0x7f67fc06ff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:54.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:54.624+0000 7f6824200640 1 --2- 172.21.15.96:0/3818349292 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f67fc06da50 0x7f67fc06ff10 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f6810001770 tx=0x7f6810009040 comp rx=0 tx=0).stop 2024-05-28T06:16:54.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:54.624+0000 7f6824200640 1 -- 172.21.15.96:0/3818349292 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f681c15b6f0 msgr2=0x7f681c1553c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:54.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:54.624+0000 7f6824200640 1 --2- 172.21.15.96:0/3818349292 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f681c15b6f0 0x7f681c1553c0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f681400be20 tx=0x7f681400bf10 comp rx=0 tx=0).stop 2024-05-28T06:16:54.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:54.625+0000 7f6824200640 1 -- 172.21.15.96:0/3818349292 shutdown_connections 2024-05-28T06:16:54.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:54.625+0000 7f6824200640 1 --2- 172.21.15.96:0/3818349292 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f67fc06da50 0x7f67fc06ff10 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:54.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:54.625+0000 7f6824200640 1 --2- 172.21.15.96:0/3818349292 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f681c15b6f0 0x7f681c1553c0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:54.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:54.625+0000 7f6824200640 1 --2- 172.21.15.96:0/3818349292 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f681c0d0640 0x7f681c0cc740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:54.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:54.625+0000 7f6824200640 1 --2- 172.21.15.96:0/3818349292 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f681c0725e0 0x7f681c146940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:54.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:54.625+0000 7f6824200640 1 -- 172.21.15.96:0/3818349292 >> 172.21.15.96:0/3818349292 conn(0x7f681c06bd20 msgr2=0x7f681c158780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:16:54.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:54.625+0000 7f6824200640 1 -- 172.21.15.96:0/3818349292 shutdown_connections 2024-05-28T06:16:54.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:54.625+0000 7f6824200640 1 -- 172.21.15.96:0/3818349292 wait complete. 2024-05-28T06:16:54.646 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone436318 2024-05-28T06:16:54.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:54.893+0000 7f877aa00640 1 Processor -- start 2024-05-28T06:16:54.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:54.893+0000 7f877aa00640 1 -- start start 2024-05-28T06:16:54.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:54.895+0000 7f877aa00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f877415e870 0x7f87740d0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:54.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:54.895+0000 7f877aa00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f87740d14d0 0x7f87740cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:54.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:54.895+0000 7f877aa00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f87740cf870 0x7f8774162360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:54.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:54.895+0000 7f877aa00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f87740d2150 con 0x7f877415e870 2024-05-28T06:16:54.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:54.895+0000 7f877aa00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f87740cfd00 con 0x7f87740cf870 2024-05-28T06:16:54.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:54.895+0000 7f877aa00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f877415ec50 con 0x7f87740d14d0 2024-05-28T06:16:54.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:54.895+0000 7f8779600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f877415e870 0x7f87740d0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:54.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:54.895+0000 7f8779600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f877415e870 0x7f87740d0f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:38996/0 (socket says 172.21.15.96:38996) 2024-05-28T06:16:54.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:54.895+0000 7f8779600640 1 -- 172.21.15.96:0/2113234532 learned_addr learned my addr 172.21.15.96:0/2113234532 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:16:54.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:54.896+0000 7f8779600640 1 -- 172.21.15.96:0/2113234532 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f87740d14d0 msgr2=0x7f87740cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:16:54.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:54.896+0000 7f8779600640 1 --2- 172.21.15.96:0/2113234532 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f87740d14d0 0x7f87740cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:54.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:54.896+0000 7f8779600640 1 -- 172.21.15.96:0/2113234532 >> v2:172.21.15.123:3300/0 conn(0x7f87740cf870 msgr2=0x7f8774162360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:16:54.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:54.896+0000 7f8779600640 1 --2- 172.21.15.96:0/2113234532 >> v2:172.21.15.123:3300/0 conn(0x7f87740cf870 0x7f8774162360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:54.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:54.896+0000 7f8779600640 1 -- 172.21.15.96:0/2113234532 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87740cff80 con 0x7f877415e870 2024-05-28T06:16:54.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:54.897+0000 7f8779600640 1 --2- 172.21.15.96:0/2113234532 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f877415e870 0x7f87740d0f90 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f87740b3f60 tx=0x7f876802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=971062198cf1ce5e server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:54.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:54.897+0000 7f8773e00640 1 -- 172.21.15.96:0/2113234532 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8768037070 con 0x7f877415e870 2024-05-28T06:16:54.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:54.897+0000 7f8773e00640 1 -- 172.21.15.96:0/2113234532 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8768032440 con 0x7f877415e870 2024-05-28T06:16:54.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:54.897+0000 7f8773e00640 1 -- 172.21.15.96:0/2113234532 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8768032810 con 0x7f877415e870 2024-05-28T06:16:54.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:54.898+0000 7f877aa00640 1 -- 172.21.15.96:0/2113234532 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f877415e870 msgr2=0x7f87740d0f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:54.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:54.898+0000 7f877aa00640 1 --2- 172.21.15.96:0/2113234532 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f877415e870 0x7f87740d0f90 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f87740b3f60 tx=0x7f876802a1e0 comp rx=0 tx=0).stop 2024-05-28T06:16:54.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:54.898+0000 7f877aa00640 1 -- 172.21.15.96:0/2113234532 shutdown_connections 2024-05-28T06:16:54.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:54.898+0000 7f877aa00640 1 --2- 172.21.15.96:0/2113234532 >> v2:172.21.15.123:3300/0 conn(0x7f87740cf870 0x7f8774162360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:54.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:54.898+0000 7f877aa00640 1 --2- 172.21.15.96:0/2113234532 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f87740d14d0 0x7f87740cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:54.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:54.898+0000 7f877aa00640 1 --2- 172.21.15.96:0/2113234532 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f877415e870 0x7f87740d0f90 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:54.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:54.898+0000 7f877aa00640 1 -- 172.21.15.96:0/2113234532 >> 172.21.15.96:0/2113234532 conn(0x7f877408bfc0 msgr2=0x7f87740d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:16:54.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:54.899+0000 7f877aa00640 1 -- 172.21.15.96:0/2113234532 shutdown_connections 2024-05-28T06:16:54.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:54.899+0000 7f877aa00640 1 -- 172.21.15.96:0/2113234532 wait complete. 2024-05-28T06:16:54.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:54.902+0000 7f877aa00640 1 Processor -- start 2024-05-28T06:16:54.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:54.902+0000 7f877aa00640 1 -- start start 2024-05-28T06:16:54.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:54.904+0000 7f877aa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f87740cf870 0x7f8774159e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:54.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:54.904+0000 7f877aa00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87740d14d0 0x7f87740b7bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:54.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:54.904+0000 7f877aa00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f877415e870 0x7f8774161030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:54.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:54.904+0000 7f877aa00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f87740d0820 con 0x7f87740cf870 2024-05-28T06:16:54.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:54.904+0000 7f877aa00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f87740d0540 con 0x7f87740d14d0 2024-05-28T06:16:54.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:54.904+0000 7f877aa00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f87740d4fb0 con 0x7f877415e870 2024-05-28T06:16:54.904 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:54.905+0000 7f8779600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f87740cf870 0x7f8774159e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:54.904 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:54.905+0000 7f8779600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f87740cf870 0x7f8774159e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:39012/0 (socket says 172.21.15.96:39012) 2024-05-28T06:16:54.904 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:54.905+0000 7f8779600640 1 -- 172.21.15.96:0/292686989 learned_addr learned my addr 172.21.15.96:0/292686989 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:16:54.906 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:54.907+0000 7f8779600640 1 -- 172.21.15.96:0/292686989 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f877415e870 msgr2=0x7f8774161030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:54.906 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:54.907+0000 7f8779600640 1 --2- 172.21.15.96:0/292686989 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f877415e870 0x7f8774161030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:54.906 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:54.907+0000 7f8779600640 1 -- 172.21.15.96:0/292686989 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87740d14d0 msgr2=0x7f87740b7bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:54.906 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:54.907+0000 7f8779600640 1 --2- 172.21.15.96:0/292686989 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87740d14d0 0x7f87740b7bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:54.906 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:54.907+0000 7f8779600640 1 -- 172.21.15.96:0/292686989 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f877415a3d0 con 0x7f87740cf870 2024-05-28T06:16:54.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:54.908+0000 7f8779600640 1 --2- 172.21.15.96:0/292686989 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f87740cf870 0x7f8774159e90 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f8768000c00 tx=0x7f87680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:54.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:54.908+0000 7f8771600640 1 -- 172.21.15.96:0/292686989 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8768041070 con 0x7f87740cf870 2024-05-28T06:16:54.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:54.909+0000 7f8771600640 1 -- 172.21.15.96:0/292686989 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f876803c870 con 0x7f87740cf870 2024-05-28T06:16:54.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:54.909+0000 7f8771600640 1 -- 172.21.15.96:0/292686989 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8768037040 con 0x7f87740cf870 2024-05-28T06:16:54.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:54.909+0000 7f877aa00640 1 -- 172.21.15.96:0/292686989 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87740b8630 con 0x7f87740cf870 2024-05-28T06:16:54.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:54.909+0000 7f877aa00640 1 -- 172.21.15.96:0/292686989 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87740b8bf0 con 0x7f87740cf870 2024-05-28T06:16:54.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:54.911+0000 7f877aa00640 1 -- 172.21.15.96:0/292686989 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8750005290 con 0x7f87740cf870 2024-05-28T06:16:54.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:54.914+0000 7f8771600640 1 -- 172.21.15.96:0/292686989 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 28) ==== 92570+0+0 (secure 0 0 0) 0x7f8768044070 con 0x7f87740cf870 2024-05-28T06:16:54.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:54.915+0000 7f8771600640 1 --2- 172.21.15.96:0/292686989 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f875806daa0 0x7f875806ff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:54.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:54.915+0000 7f8771600640 1 -- 172.21.15.96:0/292686989 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5795+0+0 (secure 0 0 0) 0x7f87680ab5d0 con 0x7f87740cf870 2024-05-28T06:16:54.922 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:54.924+0000 7f8771600640 1 -- 172.21.15.96:0/292686989 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f876807b110 con 0x7f87740cf870 2024-05-28T06:16:54.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:54.947+0000 7f8778c00640 1 --2- 172.21.15.96:0/292686989 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f875806daa0 0x7f875806ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:54.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:54.951+0000 7f8778c00640 1 --2- 172.21.15.96:0/292686989 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f875806daa0 0x7f875806ff60 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f87741f7f60 tx=0x7f8764006be0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:55.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:55.327+0000 7f877aa00640 1 -- 172.21.15.96:0/292686989 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone436318", "target": ["mon-mgr", ""]}) -- 0x7f8750000bc0 con 0x7f875806daa0 2024-05-28T06:16:56.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:56.153+0000 7f8771600640 1 -- 172.21.15.96:0/292686989 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f8750000bc0 con 0x7f875806daa0 2024-05-28T06:16:56.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:56.156+0000 7f877aa00640 1 -- 172.21.15.96:0/292686989 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f875806daa0 msgr2=0x7f875806ff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:56.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:56.156+0000 7f877aa00640 1 --2- 172.21.15.96:0/292686989 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f875806daa0 0x7f875806ff60 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f87741f7f60 tx=0x7f8764006be0 comp rx=0 tx=0).stop 2024-05-28T06:16:56.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:56.156+0000 7f877aa00640 1 -- 172.21.15.96:0/292686989 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f87740cf870 msgr2=0x7f8774159e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:56.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:56.156+0000 7f877aa00640 1 --2- 172.21.15.96:0/292686989 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f87740cf870 0x7f8774159e90 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f8768000c00 tx=0x7f87680046c0 comp rx=0 tx=0).stop 2024-05-28T06:16:56.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:56.157+0000 7f877aa00640 1 -- 172.21.15.96:0/292686989 shutdown_connections 2024-05-28T06:16:56.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:56.157+0000 7f877aa00640 1 --2- 172.21.15.96:0/292686989 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f875806daa0 0x7f875806ff60 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:56.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:56.157+0000 7f877aa00640 1 --2- 172.21.15.96:0/292686989 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f877415e870 0x7f8774161030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:56.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:56.157+0000 7f877aa00640 1 --2- 172.21.15.96:0/292686989 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87740d14d0 0x7f87740b7bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:56.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:56.157+0000 7f877aa00640 1 --2- 172.21.15.96:0/292686989 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f87740cf870 0x7f8774159e90 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:56.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:56.157+0000 7f877aa00640 1 -- 172.21.15.96:0/292686989 >> 172.21.15.96:0/292686989 conn(0x7f877408bfc0 msgr2=0x7f87740d25e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:16:56.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:56.157+0000 7f877aa00640 1 -- 172.21.15.96:0/292686989 shutdown_connections 2024-05-28T06:16:56.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:56.157+0000 7f877aa00640 1 -- 172.21.15.96:0/292686989 wait complete. 2024-05-28T06:16:56.176 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone436319 2024-05-28T06:16:56.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:56.462+0000 7fb969800640 1 Processor -- start 2024-05-28T06:16:56.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:56.463+0000 7fb969800640 1 -- start start 2024-05-28T06:16:56.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:56.466+0000 7fb969800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fb9640b3fb0 0x7fb964073a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:56.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:56.466+0000 7fb969800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fb9640b3330 0x7fb96415c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:56.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:56.466+0000 7fb969800640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb96415bf10 con 0x7fb96415ab20 2024-05-28T06:16:56.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:56.466+0000 7fb969800640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fb9640cc920 con 0x7fb9640b3330 2024-05-28T06:16:56.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:56.466+0000 7fb969800640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fb9640b17c0 con 0x7fb9640b3fb0 2024-05-28T06:16:56.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:56.467+0000 7fb962000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fb9640b3fb0 0x7fb964073a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:56.471 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:56.467+0000 7fb962000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fb9640b3fb0 0x7fb964073a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.96:40442/0 (socket says 172.21.15.96:40442) 2024-05-28T06:16:56.471 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:56.467+0000 7fb962000640 1 -- 172.21.15.96:0/4261062593 learned_addr learned my addr 172.21.15.96:0/4261062593 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:16:56.471 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:56.467+0000 7fb963400640 1 --2- 172.21.15.96:0/4261062593 >> v2:172.21.15.96:3300/0 conn(0x7fb9640b3330 0x7fb96415c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:56.471 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:56.468+0000 7fb963400640 1 -- 172.21.15.96:0/4261062593 >> v2:172.21.15.123:3300/0 conn(0x7fb9640b3fb0 msgr2=0x7fb964073a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:16:56.471 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:56.468+0000 7fb963400640 1 --2- 172.21.15.96:0/4261062593 >> v2:172.21.15.123:3300/0 conn(0x7fb9640b3fb0 0x7fb964073a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:56.471 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:56.468+0000 7fb963400640 1 -- 172.21.15.96:0/4261062593 >> v1:172.21.15.123:6789/0 conn(0x7fb96415ab20 legacy=0x7fb9640b38a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:16:56.471 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:56.468+0000 7fb963400640 1 -- 172.21.15.96:0/4261062593 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9640b1aa0 con 0x7fb9640b3330 2024-05-28T06:16:56.471 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:56.471+0000 7fb963400640 1 --2- 172.21.15.96:0/4261062593 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb9640b3330 0x7fb96415c6d0 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7fb9640b5190 tx=0x7fb954032d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55677b63a40ef376 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:56.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:56.472+0000 7fb961600640 1 -- 172.21.15.96:0/4261062593 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb954040070 con 0x7fb9640b3330 2024-05-28T06:16:56.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:56.472+0000 7fb961600640 1 -- 172.21.15.96:0/4261062593 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb954033a80 con 0x7fb9640b3330 2024-05-28T06:16:56.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:56.472+0000 7fb961600640 1 -- 172.21.15.96:0/4261062593 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb954033e50 con 0x7fb9640b3330 2024-05-28T06:16:56.473 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:56.473+0000 7fb969800640 1 -- 172.21.15.96:0/4261062593 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb9640b3330 msgr2=0x7fb96415c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:56.473 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:56.473+0000 7fb969800640 1 --2- 172.21.15.96:0/4261062593 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb9640b3330 0x7fb96415c6d0 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7fb9640b5190 tx=0x7fb954032d70 comp rx=0 tx=0).stop 2024-05-28T06:16:56.473 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:56.473+0000 7fb969800640 1 -- 172.21.15.96:0/4261062593 shutdown_connections 2024-05-28T06:16:56.473 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:56.473+0000 7fb969800640 1 --2- 172.21.15.96:0/4261062593 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb9640b3330 0x7fb96415c6d0 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:56.473 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:56.473+0000 7fb969800640 1 --2- 172.21.15.96:0/4261062593 >> v2:172.21.15.123:3300/0 conn(0x7fb9640b3fb0 0x7fb964073a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:56.473 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:56.473+0000 7fb969800640 1 -- 172.21.15.96:0/4261062593 >> 172.21.15.96:0/4261062593 conn(0x7fb9640b1fc0 msgr2=0x7fb9640b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:16:56.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:56.478+0000 7fb969800640 1 -- 172.21.15.96:0/4261062593 shutdown_connections 2024-05-28T06:16:56.478 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:56.480+0000 7fb969800640 1 -- 172.21.15.96:0/4261062593 wait complete. 2024-05-28T06:16:56.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:56.483+0000 7fb969800640 1 Processor -- start 2024-05-28T06:16:56.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:56.484+0000 7fb969800640 1 -- start start 2024-05-28T06:16:56.489 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:56.485+0000 7fb969800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb9640b3330 0x7fb964152510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:56.489 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:56.485+0000 7fb969800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb9640b3fb0 0x7fb964155490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:56.489 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:56.485+0000 7fb969800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb96415ab20 0x7fb9640994e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:56.489 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:56.485+0000 7fb969800640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb9640cc540 con 0x7fb9640b3fb0 2024-05-28T06:16:56.489 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:56.485+0000 7fb969800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fb96415c1b0 con 0x7fb9640b3330 2024-05-28T06:16:56.489 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:56.485+0000 7fb969800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fb9640cc6c0 con 0x7fb96415ab20 2024-05-28T06:16:56.489 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:56.488+0000 7fb962000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb9640b3fb0 0x7fb964155490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:56.489 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:56.488+0000 7fb962000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb9640b3fb0 0x7fb964155490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:39022/0 (socket says 172.21.15.96:39022) 2024-05-28T06:16:56.490 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:56.488+0000 7fb962000640 1 -- 172.21.15.96:0/3027318154 learned_addr learned my addr 172.21.15.96:0/3027318154 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:16:56.490 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:56.489+0000 7fb962000640 1 -- 172.21.15.96:0/3027318154 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb96415ab20 msgr2=0x7fb9640994e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-28T06:16:56.490 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:56.489+0000 7fb962000640 1 --2- 172.21.15.96:0/3027318154 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb96415ab20 0x7fb9640994e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:56.490 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:56.489+0000 7fb962000640 1 -- 172.21.15.96:0/3027318154 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb9640b3330 msgr2=0x7fb964152510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-28T06:16:56.490 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:56.489+0000 7fb962000640 1 --2- 172.21.15.96:0/3027318154 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb9640b3330 0x7fb964152510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:56.490 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:56.489+0000 7fb962000640 1 -- 172.21.15.96:0/3027318154 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9640cd060 con 0x7fb9640b3fb0 2024-05-28T06:16:56.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:56.494+0000 7fb962000640 1 --2- 172.21.15.96:0/3027318154 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb9640b3fb0 0x7fb964155490 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7fb94c009710 tx=0x7fb94c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:56.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:56.495+0000 7fb95aa00640 1 -- 172.21.15.96:0/3027318154 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb94c01c070 con 0x7fb9640b3fb0 2024-05-28T06:16:56.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:56.495+0000 7fb95aa00640 1 -- 172.21.15.96:0/3027318154 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb94c00c8d0 con 0x7fb9640b3fb0 2024-05-28T06:16:56.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:56.495+0000 7fb95aa00640 1 -- 172.21.15.96:0/3027318154 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb94c01c070 con 0x7fb9640b3fb0 2024-05-28T06:16:56.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:56.497+0000 7fb969800640 1 -- 172.21.15.96:0/3027318154 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9640cd340 con 0x7fb9640b3fb0 2024-05-28T06:16:56.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:56.499+0000 7fb969800640 1 -- 172.21.15.96:0/3027318154 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9640cd900 con 0x7fb9640b3fb0 2024-05-28T06:16:56.501 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:56.503+0000 7fb969800640 1 -- 172.21.15.96:0/3027318154 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9640689e0 con 0x7fb9640b3fb0 2024-05-28T06:16:56.505 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:56.506+0000 7fb95aa00640 1 -- 172.21.15.96:0/3027318154 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 28) ==== 92570+0+0 (secure 0 0 0) 0x7fb94c01f070 con 0x7fb9640b3fb0 2024-05-28T06:16:56.508 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:56.509+0000 7fb95aa00640 1 --2- 172.21.15.96:0/3027318154 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb94006dc00 0x7fb9400700c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:56.508 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:56.510+0000 7fb95aa00640 1 -- 172.21.15.96:0/3027318154 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5795+0+0 (secure 0 0 0) 0x7fb94c089d10 con 0x7fb9640b3fb0 2024-05-28T06:16:56.513 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:56.513+0000 7fb95aa00640 1 -- 172.21.15.96:0/3027318154 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb94c059170 con 0x7fb9640b3fb0 2024-05-28T06:16:56.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:56.532+0000 7fb962a00640 1 --2- 172.21.15.96:0/3027318154 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb94006dc00 0x7fb9400700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:56.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:56.563+0000 7fb962a00640 1 --2- 172.21.15.96:0/3027318154 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb94006dc00 0x7fb9400700c0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fb95c0095f0 tx=0x7fb95c006b50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:56.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:56.897+0000 7fb969800640 1 -- 172.21.15.96:0/3027318154 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone436319", "target": ["mon-mgr", ""]}) -- 0x7fb9640b2660 con 0x7fb94006dc00 2024-05-28T06:16:57.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:57.395+0000 7fb95aa00640 1 -- 172.21.15.96:0/3027318154 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fb9640b2660 con 0x7fb94006dc00 2024-05-28T06:16:57.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:57.399+0000 7fb969800640 1 -- 172.21.15.96:0/3027318154 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb94006dc00 msgr2=0x7fb9400700c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:57.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:57.399+0000 7fb969800640 1 --2- 172.21.15.96:0/3027318154 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb94006dc00 0x7fb9400700c0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fb95c0095f0 tx=0x7fb95c006b50 comp rx=0 tx=0).stop 2024-05-28T06:16:57.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:57.399+0000 7fb969800640 1 -- 172.21.15.96:0/3027318154 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb9640b3fb0 msgr2=0x7fb964155490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:57.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:57.399+0000 7fb969800640 1 --2- 172.21.15.96:0/3027318154 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb9640b3fb0 0x7fb964155490 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7fb94c009710 tx=0x7fb94c009bd0 comp rx=0 tx=0).stop 2024-05-28T06:16:57.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:57.399+0000 7fb969800640 1 -- 172.21.15.96:0/3027318154 shutdown_connections 2024-05-28T06:16:57.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:57.399+0000 7fb969800640 1 --2- 172.21.15.96:0/3027318154 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb94006dc00 0x7fb9400700c0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:57.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:57.399+0000 7fb969800640 1 --2- 172.21.15.96:0/3027318154 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb96415ab20 0x7fb9640994e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:57.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:57.399+0000 7fb969800640 1 --2- 172.21.15.96:0/3027318154 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb9640b3fb0 0x7fb964155490 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:57.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:57.399+0000 7fb969800640 1 --2- 172.21.15.96:0/3027318154 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb9640b3330 0x7fb964152510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:57.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:57.399+0000 7fb969800640 1 -- 172.21.15.96:0/3027318154 >> 172.21.15.96:0/3027318154 conn(0x7fb9640b1fc0 msgr2=0x7fb964159a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:16:57.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:57.399+0000 7fb969800640 1 -- 172.21.15.96:0/3027318154 shutdown_connections 2024-05-28T06:16:57.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:57.399+0000 7fb969800640 1 -- 172.21.15.96:0/3027318154 wait complete. 2024-05-28T06:16:57.419 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone436320 2024-05-28T06:16:57.666 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:57.667+0000 7fb1a8c00640 1 Processor -- start 2024-05-28T06:16:57.666 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:57.668+0000 7fb1a8c00640 1 -- start start 2024-05-28T06:16:57.669 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:57.670+0000 7fb1a8c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb1a415b6f0 0x7fb1a406e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:57.669 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:57.670+0000 7fb1a8c00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fb1a40725e0 0x7fb1a40ae650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:57.669 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:57.670+0000 7fb1a8c00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1a40ade90 con 0x7fb1a40d0640 2024-05-28T06:16:57.669 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:57.670+0000 7fb1a8c00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fb1a4071c40 con 0x7fb1a40725e0 2024-05-28T06:16:57.669 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:57.670+0000 7fb1a8c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fb1a4071e70 con 0x7fb1a415b6f0 2024-05-28T06:16:57.669 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:57.670+0000 7fb1a1400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb1a415b6f0 0x7fb1a406e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:57.669 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:57.670+0000 7fb1a1400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb1a415b6f0 0x7fb1a406e350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.96:33924/0 (socket says 172.21.15.96:33924) 2024-05-28T06:16:57.669 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:57.670+0000 7fb1a1400640 1 -- 172.21.15.96:0/796836252 learned_addr learned my addr 172.21.15.96:0/796836252 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:16:57.669 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:57.671+0000 7fb1a2800640 1 --2- 172.21.15.96:0/796836252 >> v2:172.21.15.96:3300/0 conn(0x7fb1a40725e0 0x7fb1a40ae650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:57.670 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:57.671+0000 7fb1a1400640 1 -- 172.21.15.96:0/796836252 >> v2:172.21.15.96:3300/0 conn(0x7fb1a40725e0 msgr2=0x7fb1a40ae650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:16:57.670 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:57.671+0000 7fb1a1400640 1 --2- 172.21.15.96:0/796836252 >> v2:172.21.15.96:3300/0 conn(0x7fb1a40725e0 0x7fb1a40ae650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:57.670 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:57.671+0000 7fb1a1400640 1 -- 172.21.15.96:0/796836252 >> v1:172.21.15.96:6789/0 conn(0x7fb1a40d0640 legacy=0x7fb1a40d0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:57.670 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:57.671+0000 7fb1a1400640 1 -- 172.21.15.96:0/796836252 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1a4072130 con 0x7fb1a415b6f0 2024-05-28T06:16:57.671 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:57.673+0000 7fb1a1400640 1 --2- 172.21.15.96:0/796836252 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb1a415b6f0 0x7fb1a406e350 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb190009870 tx=0x7fb19002a250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=61ee21e62a302b2e server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:57.671 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:57.673+0000 7fb1a0a00640 1 -- 172.21.15.96:0/796836252 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb190037070 con 0x7fb1a415b6f0 2024-05-28T06:16:57.671 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:57.673+0000 7fb1a0a00640 1 -- 172.21.15.96:0/796836252 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb190032440 con 0x7fb1a415b6f0 2024-05-28T06:16:57.671 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:57.673+0000 7fb1a0a00640 1 -- 172.21.15.96:0/796836252 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb190032810 con 0x7fb1a415b6f0 2024-05-28T06:16:57.673 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:57.674+0000 7fb1a8c00640 1 -- 172.21.15.96:0/796836252 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb1a415b6f0 msgr2=0x7fb1a406e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:57.673 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:57.674+0000 7fb1a8c00640 1 --2- 172.21.15.96:0/796836252 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb1a415b6f0 0x7fb1a406e350 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb190009870 tx=0x7fb19002a250 comp rx=0 tx=0).stop 2024-05-28T06:16:57.673 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:57.674+0000 7fb1a8c00640 1 -- 172.21.15.96:0/796836252 shutdown_connections 2024-05-28T06:16:57.673 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:57.674+0000 7fb1a8c00640 1 --2- 172.21.15.96:0/796836252 >> v2:172.21.15.96:3300/0 conn(0x7fb1a40725e0 0x7fb1a40ae650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:57.673 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:57.674+0000 7fb1a8c00640 1 --2- 172.21.15.96:0/796836252 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb1a415b6f0 0x7fb1a406e350 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:57.673 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:57.674+0000 7fb1a8c00640 1 -- 172.21.15.96:0/796836252 >> 172.21.15.96:0/796836252 conn(0x7fb1a406bd20 msgr2=0x7fb1a406c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:16:57.673 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:57.674+0000 7fb1a8c00640 1 -- 172.21.15.96:0/796836252 shutdown_connections 2024-05-28T06:16:57.673 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:57.674+0000 7fb1a8c00640 1 -- 172.21.15.96:0/796836252 wait complete. 2024-05-28T06:16:57.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:57.676+0000 7fb1a8c00640 1 Processor -- start 2024-05-28T06:16:57.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:57.677+0000 7fb1a8c00640 1 -- start start 2024-05-28T06:16:57.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:57.678+0000 7fb1a8c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb1a40725e0 0x7fb1a4152990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:57.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:57.678+0000 7fb1a8c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb1a40d0640 0x7fb1a414bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:57.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:57.678+0000 7fb1a8c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb1a415b6f0 0x7fb1a414c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:57.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:57.678+0000 7fb1a8c00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb1a40aeb90 con 0x7fb1a415b6f0 2024-05-28T06:16:57.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:57.678+0000 7fb1a8c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fb1a406e890 con 0x7fb1a40d0640 2024-05-28T06:16:57.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:57.678+0000 7fb1a8c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fb1a40aed10 con 0x7fb1a40725e0 2024-05-28T06:16:57.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:57.678+0000 7fb1a2800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb1a415b6f0 0x7fb1a414c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:57.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:57.678+0000 7fb1a2800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb1a415b6f0 0x7fb1a414c1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:39036/0 (socket says 172.21.15.96:39036) 2024-05-28T06:16:57.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:57.678+0000 7fb1a2800640 1 -- 172.21.15.96:0/687698849 learned_addr learned my addr 172.21.15.96:0/687698849 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:16:57.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:57.678+0000 7fb1a1e00640 1 --2- 172.21.15.96:0/687698849 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb1a40725e0 0x7fb1a4152990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:57.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:57.678+0000 7fb1a1400640 1 --2- 172.21.15.96:0/687698849 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb1a40d0640 0x7fb1a414bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:57.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:57.679+0000 7fb1a2800640 1 -- 172.21.15.96:0/687698849 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb1a40725e0 msgr2=0x7fb1a4152990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:57.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:57.679+0000 7fb1a2800640 1 --2- 172.21.15.96:0/687698849 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb1a40725e0 0x7fb1a4152990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:57.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:57.679+0000 7fb1a2800640 1 -- 172.21.15.96:0/687698849 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb1a40d0640 msgr2=0x7fb1a414bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:57.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:57.679+0000 7fb1a2800640 1 --2- 172.21.15.96:0/687698849 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb1a40d0640 0x7fb1a414bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:57.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:57.679+0000 7fb1a2800640 1 -- 172.21.15.96:0/687698849 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1a4151bf0 con 0x7fb1a415b6f0 2024-05-28T06:16:57.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:57.679+0000 7fb1a2800640 1 --2- 172.21.15.96:0/687698849 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb1a415b6f0 0x7fb1a414c1f0 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7fb19c00efc0 tx=0x7fb19c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:57.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:57.679+0000 7fb19a000640 1 -- 172.21.15.96:0/687698849 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb19c018020 con 0x7fb1a415b6f0 2024-05-28T06:16:57.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:57.680+0000 7fb19a000640 1 -- 172.21.15.96:0/687698849 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb19c014480 con 0x7fb1a415b6f0 2024-05-28T06:16:57.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:57.680+0000 7fb19a000640 1 -- 172.21.15.96:0/687698849 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb19c014870 con 0x7fb1a415b6f0 2024-05-28T06:16:57.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:57.680+0000 7fb1a8c00640 1 -- 172.21.15.96:0/687698849 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1a41504d0 con 0x7fb1a415b6f0 2024-05-28T06:16:57.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:57.680+0000 7fb1a8c00640 1 -- 172.21.15.96:0/687698849 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1a4152190 con 0x7fb1a415b6f0 2024-05-28T06:16:57.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:57.681+0000 7fb1a8c00640 1 -- 172.21.15.96:0/687698849 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1a40689e0 con 0x7fb1a415b6f0 2024-05-28T06:16:57.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:57.682+0000 7fb19a000640 1 -- 172.21.15.96:0/687698849 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 28) ==== 92570+0+0 (secure 0 0 0) 0x7fb19c005d10 con 0x7fb1a415b6f0 2024-05-28T06:16:57.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:57.683+0000 7fb19a000640 1 --2- 172.21.15.96:0/687698849 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb18406da20 0x7fb18406fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:57.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:57.683+0000 7fb19a000640 1 -- 172.21.15.96:0/687698849 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5795+0+0 (secure 0 0 0) 0x7fb19c08b470 con 0x7fb1a415b6f0 2024-05-28T06:16:57.687 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:57.688+0000 7fb19a000640 1 -- 172.21.15.96:0/687698849 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb19c022030 con 0x7fb1a415b6f0 2024-05-28T06:16:57.689 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:57.690+0000 7fb1a1e00640 1 --2- 172.21.15.96:0/687698849 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb18406da20 0x7fb18406fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:57.690 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:57.692+0000 7fb1a1e00640 1 --2- 172.21.15.96:0/687698849 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb18406da20 0x7fb18406fee0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fb194001780 tx=0x7fb194009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:58.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.026+0000 7fb1a8c00640 1 -- 172.21.15.96:0/687698849 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone436320", "target": ["mon-mgr", ""]}) -- 0x7fb1a406c440 con 0x7fb18406da20 2024-05-28T06:16:58.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.190+0000 7fb19a000640 1 -- 172.21.15.96:0/687698849 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fb1a406c440 con 0x7fb18406da20 2024-05-28T06:16:58.192 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.193+0000 7fb1a8c00640 1 -- 172.21.15.96:0/687698849 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb18406da20 msgr2=0x7fb18406fee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:58.192 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.193+0000 7fb1a8c00640 1 --2- 172.21.15.96:0/687698849 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb18406da20 0x7fb18406fee0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fb194001780 tx=0x7fb194009040 comp rx=0 tx=0).stop 2024-05-28T06:16:58.192 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.193+0000 7fb1a8c00640 1 -- 172.21.15.96:0/687698849 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb1a415b6f0 msgr2=0x7fb1a414c1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:58.192 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.193+0000 7fb1a8c00640 1 --2- 172.21.15.96:0/687698849 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb1a415b6f0 0x7fb1a414c1f0 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7fb19c00efc0 tx=0x7fb19c00c490 comp rx=0 tx=0).stop 2024-05-28T06:16:58.194 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.193+0000 7fb1a8c00640 1 -- 172.21.15.96:0/687698849 shutdown_connections 2024-05-28T06:16:58.194 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.193+0000 7fb1a8c00640 1 --2- 172.21.15.96:0/687698849 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb18406da20 0x7fb18406fee0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:58.194 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.193+0000 7fb1a8c00640 1 --2- 172.21.15.96:0/687698849 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb1a415b6f0 0x7fb1a414c1f0 unknown :-1 s=CLOSED pgs=1221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:58.194 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.194+0000 7fb1a8c00640 1 --2- 172.21.15.96:0/687698849 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb1a40d0640 0x7fb1a414bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:58.194 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.194+0000 7fb1a8c00640 1 --2- 172.21.15.96:0/687698849 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb1a40725e0 0x7fb1a4152990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:58.194 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.194+0000 7fb1a8c00640 1 -- 172.21.15.96:0/687698849 >> 172.21.15.96:0/687698849 conn(0x7fb1a406bd20 msgr2=0x7fb1a40b09f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:16:58.194 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.194+0000 7fb1a8c00640 1 -- 172.21.15.96:0/687698849 shutdown_connections 2024-05-28T06:16:58.194 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.194+0000 7fb1a8c00640 1 -- 172.21.15.96:0/687698849 wait complete. 2024-05-28T06:16:58.214 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-05-28T06:16:58.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.423+0000 7fd3d9e00640 1 Processor -- start 2024-05-28T06:16:58.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.424+0000 7fd3d9e00640 1 -- start start 2024-05-28T06:16:58.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.425+0000 7fd3d9e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fd3d40d16a0 0x7fd3d40cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:58.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.425+0000 7fd3d9e00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3d40cfce0 con 0x7fd3d40cf870 2024-05-28T06:16:58.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.425+0000 7fd3d9e00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3d415ec50 con 0x7fd3d415e870 2024-05-28T06:16:58.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.425+0000 7fd3d9e00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fd3d40d2150 con 0x7fd3d40d16a0 2024-05-28T06:16:58.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.425+0000 7fd3d9400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fd3d40cf870 0x7fd3d40d25e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:37570/0 (socket says 172.21.15.96:37570) 2024-05-28T06:16:58.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.426+0000 7fd3d9400640 1 -- 172.21.15.96:0/130300080 learned_addr learned my addr 172.21.15.96:0/130300080 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:16:58.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.426+0000 7fd3d3400640 1 -- 172.21.15.96:0/130300080 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1371299737 0 0) 0x7fd3d40cfce0 con 0x7fd3d40cf870 2024-05-28T06:16:58.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.426+0000 7fd3d3e00640 1 --2- 172.21.15.96:0/130300080 >> v2:172.21.15.123:3300/0 conn(0x7fd3d40d16a0 0x7fd3d40cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:58.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.426+0000 7fd3d3400640 1 -- 172.21.15.96:0/130300080 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd3b8003520 con 0x7fd3d40cf870 2024-05-28T06:16:58.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.426+0000 7fd3d3400640 1 -- 172.21.15.96:0/130300080 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 691744464 0 0) 0x7fd3b8003520 con 0x7fd3d40cf870 2024-05-28T06:16:58.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.426+0000 7fd3d3400640 1 -- 172.21.15.96:0/130300080 >> v2:172.21.15.123:3300/0 conn(0x7fd3d40d16a0 msgr2=0x7fd3d40cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:16:58.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.426+0000 7fd3d3400640 1 --2- 172.21.15.96:0/130300080 >> v2:172.21.15.123:3300/0 conn(0x7fd3d40d16a0 0x7fd3d40cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:58.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.426+0000 7fd3d3400640 1 -- 172.21.15.96:0/130300080 >> v1:172.21.15.123:6789/0 conn(0x7fd3d415e870 legacy=0x7fd3d40d0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:58.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.426+0000 7fd3d3400640 1 -- 172.21.15.96:0/130300080 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3d40cff80 con 0x7fd3d40cf870 2024-05-28T06:16:58.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.426+0000 7fd3d3400640 1 -- 172.21.15.96:0/130300080 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7fd3c4001670 con 0x7fd3d40cf870 2024-05-28T06:16:58.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.426+0000 7fd3d3400640 1 -- 172.21.15.96:0/130300080 >> v1:172.21.15.96:6789/0 conn(0x7fd3d40cf870 legacy=0x7fd3d40d25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:58.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.426+0000 7fd3d3400640 1 --2- 172.21.15.96:0/130300080 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd3b8003a00 0x7fd3b8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:58.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.426+0000 7fd3d3400640 1 --2- 172.21.15.96:0/130300080 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd3b8004800 0x7fd3b8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:58.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.426+0000 7fd3d3400640 1 --2- 172.21.15.96:0/130300080 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd3b8005210 0x7fd3b800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:58.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.426+0000 7fd3d3400640 1 -- 172.21.15.96:0/130300080 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd3d40d2150 con 0x7fd3b8004800 2024-05-28T06:16:58.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.426+0000 7fd3d3400640 1 -- 172.21.15.96:0/130300080 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd3b800c160 con 0x7fd3b8003a00 2024-05-28T06:16:58.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.426+0000 7fd3d3400640 1 -- 172.21.15.96:0/130300080 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fd3b800c360 con 0x7fd3b8005210 2024-05-28T06:16:58.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.427+0000 7fd3d3e00640 1 --2- 172.21.15.96:0/130300080 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd3b8004800 0x7fd3b8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:58.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.427+0000 7fd3d9400640 1 --2- 172.21.15.96:0/130300080 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd3b8005210 0x7fd3b800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:58.430 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.427+0000 7fd3d8a00640 1 --2- 172.21.15.96:0/130300080 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd3b8003a00 0x7fd3b8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:58.430 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.427+0000 7fd3d3e00640 1 -- 172.21.15.96:0/130300080 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd3b8005210 msgr2=0x7fd3b800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:16:58.430 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.427+0000 7fd3d3e00640 1 --2- 172.21.15.96:0/130300080 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd3b8005210 0x7fd3b800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:58.430 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.427+0000 7fd3d3e00640 1 -- 172.21.15.96:0/130300080 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd3b8003a00 msgr2=0x7fd3b8003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:16:58.430 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.427+0000 7fd3d3e00640 1 --2- 172.21.15.96:0/130300080 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd3b8003a00 0x7fd3b8003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:58.430 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.427+0000 7fd3d3e00640 1 -- 172.21.15.96:0/130300080 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3b8003520 con 0x7fd3b8004800 2024-05-28T06:16:58.430 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.427+0000 7fd3d9400640 1 --2- 172.21.15.96:0/130300080 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd3b8005210 0x7fd3b800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:16:58.430 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.427+0000 7fd3d8a00640 1 --2- 172.21.15.96:0/130300080 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd3b8003a00 0x7fd3b8003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:16:58.430 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.427+0000 7fd3d9400640 1 -- 172.21.15.96:0/130300080 reap_dead start 2024-05-28T06:16:58.430 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.428+0000 7fd3d3e00640 1 --2- 172.21.15.96:0/130300080 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd3b8004800 0x7fd3b8004c80 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7fd3bc00ba10 tx=0x7fd3b8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e86a0ccbf337d781 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:58.430 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.429+0000 7fd3d3400640 1 -- 172.21.15.96:0/130300080 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3bc00b5e0 con 0x7fd3b8004800 2024-05-28T06:16:58.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.429+0000 7fd3d3400640 1 -- 172.21.15.96:0/130300080 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3bc02adb0 con 0x7fd3b8004800 2024-05-28T06:16:58.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.429+0000 7fd3d3400640 1 -- 172.21.15.96:0/130300080 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3bc005bb0 con 0x7fd3b8004800 2024-05-28T06:16:58.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.429+0000 7fd3d9e00640 1 -- 172.21.15.96:0/130300080 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd3b8004800 msgr2=0x7fd3b8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:58.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.429+0000 7fd3d9e00640 1 --2- 172.21.15.96:0/130300080 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd3b8004800 0x7fd3b8004c80 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7fd3bc00ba10 tx=0x7fd3b8005830 comp rx=0 tx=0).stop 2024-05-28T06:16:58.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.429+0000 7fd3d9e00640 1 -- 172.21.15.96:0/130300080 shutdown_connections 2024-05-28T06:16:58.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.429+0000 7fd3d9e00640 1 --2- 172.21.15.96:0/130300080 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd3b8004800 0x7fd3b8004c80 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:58.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.429+0000 7fd3d9e00640 1 -- 172.21.15.96:0/130300080 >> 172.21.15.96:0/130300080 conn(0x7fd3d408bfc0 msgr2=0x7fd3d40d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:16:58.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.429+0000 7fd3d9e00640 1 -- 172.21.15.96:0/130300080 shutdown_connections 2024-05-28T06:16:58.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.429+0000 7fd3d9e00640 1 -- 172.21.15.96:0/130300080 wait complete. 2024-05-28T06:16:58.433 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.431+0000 7fd3d9e00640 1 Processor -- start 2024-05-28T06:16:58.433 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.431+0000 7fd3d9e00640 1 -- start start 2024-05-28T06:16:58.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.432+0000 7fd3d9e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd3b8004800 0x7fd3d40cb8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:58.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.432+0000 7fd3d9e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd3d4159c00 0x7fd3d40cc810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:58.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.432+0000 7fd3d9e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd3d40b9870 0x7fd3d40b9cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:58.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.432+0000 7fd3d9e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd3d40d0820 con 0x7fd3b8004800 2024-05-28T06:16:58.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.432+0000 7fd3d9e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd3d40d0540 con 0x7fd3d4159c00 2024-05-28T06:16:58.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.432+0000 7fd3d9e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fd3d40d4fb0 con 0x7fd3d40b9870 2024-05-28T06:16:58.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.433+0000 7fd3d8a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd3b8004800 0x7fd3d40cb8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:58.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.433+0000 7fd3d8a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd3b8004800 0x7fd3d40cb8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:39054/0 (socket says 172.21.15.96:39054) 2024-05-28T06:16:58.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.433+0000 7fd3d8a00640 1 -- 172.21.15.96:0/327302270 learned_addr learned my addr 172.21.15.96:0/327302270 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:16:58.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.433+0000 7fd3d9400640 1 --2- 172.21.15.96:0/327302270 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd3d40b9870 0x7fd3d40b9cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:58.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.433+0000 7fd3d8a00640 1 -- 172.21.15.96:0/327302270 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd3d40b9870 msgr2=0x7fd3d40b9cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:58.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.433+0000 7fd3d8a00640 1 --2- 172.21.15.96:0/327302270 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd3d40b9870 0x7fd3d40b9cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:58.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.433+0000 7fd3d8a00640 1 -- 172.21.15.96:0/327302270 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd3d4159c00 msgr2=0x7fd3d40cc810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:58.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.433+0000 7fd3d8a00640 1 --2- 172.21.15.96:0/327302270 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd3d4159c00 0x7fd3d40cc810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:58.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.433+0000 7fd3d8a00640 1 -- 172.21.15.96:0/327302270 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3d40ccd50 con 0x7fd3b8004800 2024-05-28T06:16:58.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.433+0000 7fd3d8a00640 1 --2- 172.21.15.96:0/327302270 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd3b8004800 0x7fd3d40cb8c0 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc002230 tx=0x7fd3cc00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:58.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.434+0000 7fd3d0c00640 1 -- 172.21.15.96:0/327302270 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3cc003da0 con 0x7fd3b8004800 2024-05-28T06:16:58.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.434+0000 7fd3d0c00640 1 -- 172.21.15.96:0/327302270 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3cc017070 con 0x7fd3b8004800 2024-05-28T06:16:58.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.434+0000 7fd3d0c00640 1 -- 172.21.15.96:0/327302270 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3cc0052b0 con 0x7fd3b8004800 2024-05-28T06:16:58.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.434+0000 7fd3d9e00640 1 -- 172.21.15.96:0/327302270 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3d40cd2d0 con 0x7fd3b8004800 2024-05-28T06:16:58.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.434+0000 7fd3d9e00640 1 -- 172.21.15.96:0/327302270 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3d40b7240 con 0x7fd3b8004800 2024-05-28T06:16:58.436 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.435+0000 7fd3d9e00640 1 -- 172.21.15.96:0/327302270 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3d4069480 con 0x7fd3b8004800 2024-05-28T06:16:58.436 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.436+0000 7fd3d0c00640 1 -- 172.21.15.96:0/327302270 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 28) ==== 92570+0+0 (secure 0 0 0) 0x7fd3cc005450 con 0x7fd3b8004800 2024-05-28T06:16:58.436 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.437+0000 7fd3d0c00640 1 --2- 172.21.15.96:0/327302270 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd3b006da00 0x7fd3b006fec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:58.436 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.437+0000 7fd3d0c00640 1 -- 172.21.15.96:0/327302270 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5795+0+0 (secure 0 0 0) 0x7fd3cc08a750 con 0x7fd3b8004800 2024-05-28T06:16:58.436 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.438+0000 7fd3d3e00640 1 --2- 172.21.15.96:0/327302270 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd3b006da00 0x7fd3b006fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:58.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.439+0000 7fd3d3e00640 1 --2- 172.21.15.96:0/327302270 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd3b006da00 0x7fd3b006fec0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fd3bc034000 tx=0x7fd3bc02c000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:58.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.442+0000 7fd3d0c00640 1 -- 172.21.15.96:0/327302270 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd3d4069480 con 0x7fd3b8004800 2024-05-28T06:16:58.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.665+0000 7fd3d9e00640 1 -- 172.21.15.96:0/327302270 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7fd3d40689e0 con 0x7fd3b006da00 2024-05-28T06:16:58.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.669+0000 7fd3d0c00640 1 -- 172.21.15.96:0/327302270 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7fd3d40689e0 con 0x7fd3b006da00 2024-05-28T06:16:58.669 INFO:teuthology.orchestra.run.smithi096.stdout:[ 2024-05-28T06:16:58.669 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:16:58.669 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "cephfs" 2024-05-28T06:16:58.669 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:16:58.669 INFO:teuthology.orchestra.run.smithi096.stdout:] 2024-05-28T06:16:58.672 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.672+0000 7fd3d9e00640 1 -- 172.21.15.96:0/327302270 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd3b006da00 msgr2=0x7fd3b006fec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:58.672 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.672+0000 7fd3d9e00640 1 --2- 172.21.15.96:0/327302270 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd3b006da00 0x7fd3b006fec0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fd3bc034000 tx=0x7fd3bc02c000 comp rx=0 tx=0).stop 2024-05-28T06:16:58.672 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.672+0000 7fd3d9e00640 1 -- 172.21.15.96:0/327302270 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd3b8004800 msgr2=0x7fd3d40cb8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:58.672 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.672+0000 7fd3d9e00640 1 --2- 172.21.15.96:0/327302270 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd3b8004800 0x7fd3d40cb8c0 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc002230 tx=0x7fd3cc00dc00 comp rx=0 tx=0).stop 2024-05-28T06:16:58.672 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.672+0000 7fd3d9e00640 1 -- 172.21.15.96:0/327302270 shutdown_connections 2024-05-28T06:16:58.672 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.672+0000 7fd3d9e00640 1 --2- 172.21.15.96:0/327302270 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd3b006da00 0x7fd3b006fec0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:58.672 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.672+0000 7fd3d9e00640 1 --2- 172.21.15.96:0/327302270 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd3d40b9870 0x7fd3d40b9cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:58.672 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.672+0000 7fd3d9e00640 1 --2- 172.21.15.96:0/327302270 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd3d4159c00 0x7fd3d40cc810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:58.672 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.672+0000 7fd3d9e00640 1 --2- 172.21.15.96:0/327302270 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd3b8004800 0x7fd3d40cb8c0 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:58.673 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.672+0000 7fd3d9e00640 1 -- 172.21.15.96:0/327302270 >> 172.21.15.96:0/327302270 conn(0x7fd3d408bfc0 msgr2=0x7fd3d40b3fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:16:58.673 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.672+0000 7fd3d9e00640 1 -- 172.21.15.96:0/327302270 shutdown_connections 2024-05-28T06:16:58.673 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.672+0000 7fd3d9e00640 1 -- 172.21.15.96:0/327302270 wait complete. 2024-05-28T06:16:58.690 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-05-28T06:16:58.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.875+0000 7f8cf7800640 1 Processor -- start 2024-05-28T06:16:58.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.876+0000 7f8cf7800640 1 -- start start 2024-05-28T06:16:58.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.877+0000 7f8cf7800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f8cf0155a00 0x7f8cf015baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:58.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.877+0000 7f8cf7800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f8cf006f390 0x7f8cf006f800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:58.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.877+0000 7f8cf7800640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8cf006e3d0 con 0x7f8cf00d0970 2024-05-28T06:16:58.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.877+0000 7f8cf7800640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f8cf006fd40 con 0x7f8cf006f390 2024-05-28T06:16:58.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.877+0000 7f8cf7800640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f8cf00ccf20 con 0x7f8cf0155a00 2024-05-28T06:16:58.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.877+0000 7f8cf5800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f8cf006f390 0x7f8cf006f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:58.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.877+0000 7f8cf5800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f8cf006f390 0x7f8cf006f800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:40308/0 (socket says 172.21.15.96:40308) 2024-05-28T06:16:58.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.877+0000 7f8cf5800640 1 -- 172.21.15.96:0/3039265708 learned_addr learned my addr 172.21.15.96:0/3039265708 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:16:58.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.878+0000 7f8cf5800640 1 -- 172.21.15.96:0/3039265708 >> v2:172.21.15.123:3300/0 conn(0x7f8cf0155a00 msgr2=0x7f8cf015baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:16:58.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.878+0000 7f8cf5800640 1 --2- 172.21.15.96:0/3039265708 >> v2:172.21.15.123:3300/0 conn(0x7f8cf0155a00 0x7f8cf015baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:58.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.878+0000 7f8cf5800640 1 -- 172.21.15.96:0/3039265708 >> v1:172.21.15.123:6789/0 conn(0x7f8cf00d0970 legacy=0x7f8cf006dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:16:58.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.878+0000 7f8cf5800640 1 -- 172.21.15.96:0/3039265708 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cf015c0b0 con 0x7f8cf006f390 2024-05-28T06:16:58.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.879+0000 7f8cf5800640 1 --2- 172.21.15.96:0/3039265708 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8cf006f390 0x7f8cf006f800 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f8cf006db90 tx=0x7f8ce802a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1422892bfe2d8106 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:58.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.879+0000 7f8cef400640 1 -- 172.21.15.96:0/3039265708 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ce8037070 con 0x7f8cf006f390 2024-05-28T06:16:58.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.879+0000 7f8cef400640 1 -- 172.21.15.96:0/3039265708 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ce8032440 con 0x7f8cf006f390 2024-05-28T06:16:58.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.879+0000 7f8cef400640 1 -- 172.21.15.96:0/3039265708 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ce8032810 con 0x7f8cf006f390 2024-05-28T06:16:58.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.880+0000 7f8cf7800640 1 -- 172.21.15.96:0/3039265708 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8cf006f390 msgr2=0x7f8cf006f800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:58.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.880+0000 7f8cf7800640 1 --2- 172.21.15.96:0/3039265708 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8cf006f390 0x7f8cf006f800 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f8cf006db90 tx=0x7f8ce802a230 comp rx=0 tx=0).stop 2024-05-28T06:16:58.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.880+0000 7f8cf7800640 1 -- 172.21.15.96:0/3039265708 shutdown_connections 2024-05-28T06:16:58.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.880+0000 7f8cf7800640 1 --2- 172.21.15.96:0/3039265708 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8cf006f390 0x7f8cf006f800 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:58.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.880+0000 7f8cf7800640 1 --2- 172.21.15.96:0/3039265708 >> v2:172.21.15.123:3300/0 conn(0x7f8cf0155a00 0x7f8cf015baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:58.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.880+0000 7f8cf7800640 1 -- 172.21.15.96:0/3039265708 >> 172.21.15.96:0/3039265708 conn(0x7f8cf0067ce0 msgr2=0x7f8cf006b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:16:58.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.880+0000 7f8cf7800640 1 -- 172.21.15.96:0/3039265708 shutdown_connections 2024-05-28T06:16:58.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.880+0000 7f8cf7800640 1 -- 172.21.15.96:0/3039265708 wait complete. 2024-05-28T06:16:58.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.882+0000 7f8cf7800640 1 Processor -- start 2024-05-28T06:16:58.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.882+0000 7f8cf7800640 1 -- start start 2024-05-28T06:16:58.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.883+0000 7f8cf7800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8cf006f390 0x7f8cf0071f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:58.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.883+0000 7f8cf7800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8cf00d0970 0x7f8cf00a9490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:58.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.883+0000 7f8cf7800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8cf0155a00 0x7f8cf0146730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:58.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.883+0000 7f8cf7800640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f8cf015c670 con 0x7f8cf0155a00 2024-05-28T06:16:58.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.883+0000 7f8cf7800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f8cf015c950 con 0x7f8cf00d0970 2024-05-28T06:16:58.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.883+0000 7f8cf7800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f8cf015f090 con 0x7f8cf006f390 2024-05-28T06:16:58.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.883+0000 7f8cf5800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8cf0155a00 0x7f8cf0146730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:58.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.883+0000 7f8cf5800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8cf0155a00 0x7f8cf0146730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:40318/0 (socket says 172.21.15.96:40318) 2024-05-28T06:16:58.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.883+0000 7f8cf5800640 1 -- 172.21.15.96:0/564444201 learned_addr learned my addr 172.21.15.96:0/564444201 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:16:58.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.883+0000 7f8cf5800640 1 -- 172.21.15.96:0/564444201 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8cf006f390 msgr2=0x7f8cf0071f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:58.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.883+0000 7f8cf5800640 1 --2- 172.21.15.96:0/564444201 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8cf006f390 0x7f8cf0071f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:58.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.883+0000 7f8cf5800640 1 -- 172.21.15.96:0/564444201 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8cf00d0970 msgr2=0x7f8cf00a9490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:58.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.883+0000 7f8cefe00640 1 --2- 172.21.15.96:0/564444201 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8cf00d0970 0x7f8cf00a9490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:58.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.883+0000 7f8cf5800640 1 --2- 172.21.15.96:0/564444201 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8cf00d0970 0x7f8cf00a9490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:58.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.883+0000 7f8cf5800640 1 -- 172.21.15.96:0/564444201 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cf00a9f10 con 0x7f8cf0155a00 2024-05-28T06:16:58.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.884+0000 7f8cf5800640 1 --2- 172.21.15.96:0/564444201 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8cf0155a00 0x7f8cf0146730 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f8ce80090b0 tx=0x7f8ce80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:58.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.884+0000 7f8cecc00640 1 -- 172.21.15.96:0/564444201 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ce8041070 con 0x7f8cf0155a00 2024-05-28T06:16:58.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.884+0000 7f8cecc00640 1 -- 172.21.15.96:0/564444201 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ce803c950 con 0x7f8cf0155a00 2024-05-28T06:16:58.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.884+0000 7f8cecc00640 1 -- 172.21.15.96:0/564444201 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ce8037040 con 0x7f8cf0155a00 2024-05-28T06:16:58.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.884+0000 7f8cf7800640 1 -- 172.21.15.96:0/564444201 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8cf00aa1f0 con 0x7f8cf0155a00 2024-05-28T06:16:58.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.884+0000 7f8cf7800640 1 -- 172.21.15.96:0/564444201 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8cf0147850 con 0x7f8cf0155a00 2024-05-28T06:16:58.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.885+0000 7f8cf7800640 1 -- 172.21.15.96:0/564444201 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8cf00689e0 con 0x7f8cf0155a00 2024-05-28T06:16:58.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.886+0000 7f8cecc00640 1 -- 172.21.15.96:0/564444201 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 28) ==== 92570+0+0 (secure 0 0 0) 0x7f8ce8044070 con 0x7f8cf0155a00 2024-05-28T06:16:58.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.887+0000 7f8cecc00640 1 --2- 172.21.15.96:0/564444201 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8cd006d9d0 0x7f8cd006fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:16:58.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.887+0000 7f8cecc00640 1 -- 172.21.15.96:0/564444201 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5795+0+0 (secure 0 0 0) 0x7f8ce80ab510 con 0x7f8cf0155a00 2024-05-28T06:16:58.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.888+0000 7f8cf4e00640 1 --2- 172.21.15.96:0/564444201 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8cd006d9d0 0x7f8cd006fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:16:58.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.889+0000 7f8cf4e00640 1 --2- 172.21.15.96:0/564444201 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8cd006d9d0 0x7f8cd006fe90 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f8ce000ab90 tx=0x7f8ce0009250 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:16:58.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:58.891+0000 7f8cecc00640 1 -- 172.21.15.96:0/564444201 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8ce807b050 con 0x7f8cf0155a00 2024-05-28T06:16:59.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:59.143+0000 7f8cf7800640 1 -- 172.21.15.96:0/564444201 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f8cf0154e80 con 0x7f8cf0155a00 2024-05-28T06:16:59.143 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:59.144+0000 7f8cecc00640 1 -- 172.21.15.96:0/564444201 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+2421 (secure 0 0 0) 0x7f8ce807a950 con 0x7f8cf0155a00 2024-05-28T06:16:59.143 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:16:59.143 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:16:59.143 INFO:teuthology.orchestra.run.smithi096.stdout: "fsid": "e4a831b5-e6d1-47a7-96cb-bf85154ee0d2", 2024-05-28T06:16:59.143 INFO:teuthology.orchestra.run.smithi096.stdout: "health": { 2024-05-28T06:16:59.144 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "HEALTH_OK", 2024-05-28T06:16:59.144 INFO:teuthology.orchestra.run.smithi096.stdout: "checks": {}, 2024-05-28T06:16:59.144 INFO:teuthology.orchestra.run.smithi096.stdout: "mutes": [] 2024-05-28T06:16:59.144 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:16:59.144 INFO:teuthology.orchestra.run.smithi096.stdout: "election_epoch": 8, 2024-05-28T06:16:59.144 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum": [ 2024-05-28T06:16:59.144 INFO:teuthology.orchestra.run.smithi096.stdout: 0, 2024-05-28T06:16:59.144 INFO:teuthology.orchestra.run.smithi096.stdout: 1, 2024-05-28T06:16:59.144 INFO:teuthology.orchestra.run.smithi096.stdout: 2 2024-05-28T06:16:59.144 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:16:59.144 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_names": [ 2024-05-28T06:16:59.144 INFO:teuthology.orchestra.run.smithi096.stdout: "a", 2024-05-28T06:16:59.144 INFO:teuthology.orchestra.run.smithi096.stdout: "b", 2024-05-28T06:16:59.144 INFO:teuthology.orchestra.run.smithi096.stdout: "c" 2024-05-28T06:16:59.144 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:16:59.145 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_age": 792, 2024-05-28T06:16:59.145 INFO:teuthology.orchestra.run.smithi096.stdout: "monmap": { 2024-05-28T06:16:59.145 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 1, 2024-05-28T06:16:59.145 INFO:teuthology.orchestra.run.smithi096.stdout: "min_mon_release_name": "squid", 2024-05-28T06:16:59.145 INFO:teuthology.orchestra.run.smithi096.stdout: "num_mons": 3 2024-05-28T06:16:59.145 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:16:59.145 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap": { 2024-05-28T06:16:59.145 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 76, 2024-05-28T06:16:59.145 INFO:teuthology.orchestra.run.smithi096.stdout: "num_osds": 8, 2024-05-28T06:16:59.145 INFO:teuthology.orchestra.run.smithi096.stdout: "num_up_osds": 8, 2024-05-28T06:16:59.145 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_up_since": 1716876238, 2024-05-28T06:16:59.145 INFO:teuthology.orchestra.run.smithi096.stdout: "num_in_osds": 8, 2024-05-28T06:16:59.145 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_in_since": 1716876229, 2024-05-28T06:16:59.145 INFO:teuthology.orchestra.run.smithi096.stdout: "num_remapped_pgs": 0 2024-05-28T06:16:59.145 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:16:59.146 INFO:teuthology.orchestra.run.smithi096.stdout: "pgmap": { 2024-05-28T06:16:59.146 INFO:teuthology.orchestra.run.smithi096.stdout: "pgs_by_state": [ 2024-05-28T06:16:59.146 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:16:59.146 INFO:teuthology.orchestra.run.smithi096.stdout: "state_name": "active+clean", 2024-05-28T06:16:59.146 INFO:teuthology.orchestra.run.smithi096.stdout: "count": 97 2024-05-28T06:16:59.146 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:16:59.146 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:16:59.146 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pgs": 97, 2024-05-28T06:16:59.146 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pools": 3, 2024-05-28T06:16:59.146 INFO:teuthology.orchestra.run.smithi096.stdout: "num_objects": 12020, 2024-05-28T06:16:59.146 INFO:teuthology.orchestra.run.smithi096.stdout: "data_bytes": 50197854485, 2024-05-28T06:16:59.146 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_used": 100391854080, 2024-05-28T06:16:59.146 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_avail": 672702259200, 2024-05-28T06:16:59.147 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_total": 773094113280, 2024-05-28T06:16:59.147 INFO:teuthology.orchestra.run.smithi096.stdout: "read_bytes_sec": 619713129, 2024-05-28T06:16:59.147 INFO:teuthology.orchestra.run.smithi096.stdout: "write_bytes_sec": 615233952, 2024-05-28T06:16:59.147 INFO:teuthology.orchestra.run.smithi096.stdout: "read_op_per_sec": 152, 2024-05-28T06:16:59.147 INFO:teuthology.orchestra.run.smithi096.stdout: "write_op_per_sec": 167 2024-05-28T06:16:59.147 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:16:59.147 INFO:teuthology.orchestra.run.smithi096.stdout: "fsmap": { 2024-05-28T06:16:59.147 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 44, 2024-05-28T06:16:59.147 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 5, 2024-05-28T06:16:59.147 INFO:teuthology.orchestra.run.smithi096.stdout: "up": 1, 2024-05-28T06:16:59.147 INFO:teuthology.orchestra.run.smithi096.stdout: "in": 1, 2024-05-28T06:16:59.147 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 1, 2024-05-28T06:16:59.147 INFO:teuthology.orchestra.run.smithi096.stdout: "by_rank": [ 2024-05-28T06:16:59.147 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:16:59.148 INFO:teuthology.orchestra.run.smithi096.stdout: "filesystem_id": 5, 2024-05-28T06:16:59.148 INFO:teuthology.orchestra.run.smithi096.stdout: "rank": 0, 2024-05-28T06:16:59.148 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "d", 2024-05-28T06:16:59.148 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "up:active", 2024-05-28T06:16:59.148 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 7146 2024-05-28T06:16:59.148 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:16:59.148 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:16:59.148 INFO:teuthology.orchestra.run.smithi096.stdout: "up:standby": 3 2024-05-28T06:16:59.148 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:16:59.148 INFO:teuthology.orchestra.run.smithi096.stdout: "mgrmap": { 2024-05-28T06:16:59.148 INFO:teuthology.orchestra.run.smithi096.stdout: "available": true, 2024-05-28T06:16:59.148 INFO:teuthology.orchestra.run.smithi096.stdout: "num_standbys": 1, 2024-05-28T06:16:59.148 INFO:teuthology.orchestra.run.smithi096.stdout: "modules": [ 2024-05-28T06:16:59.148 INFO:teuthology.orchestra.run.smithi096.stdout: "iostat", 2024-05-28T06:16:59.149 INFO:teuthology.orchestra.run.smithi096.stdout: "nfs", 2024-05-28T06:16:59.149 INFO:teuthology.orchestra.run.smithi096.stdout: "restful" 2024-05-28T06:16:59.149 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:16:59.149 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:16:59.149 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:16:59.149 INFO:teuthology.orchestra.run.smithi096.stdout: "servicemap": { 2024-05-28T06:16:59.149 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 13, 2024-05-28T06:16:59.149 INFO:teuthology.orchestra.run.smithi096.stdout: "modified": "2024-05-28T06:15:57.778889+0000", 2024-05-28T06:16:59.149 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:16:59.149 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:16:59.149 INFO:teuthology.orchestra.run.smithi096.stdout: "progress_events": { 2024-05-28T06:16:59.149 INFO:teuthology.orchestra.run.smithi096.stdout: "1f539510-8a8a-46db-8f0c-e233f5cc901d": { 2024-05-28T06:16:59.149 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "2 ongoing clones; Avg progress is 13.802% (67s)\n [===.........................] (remaining: 7m)", 2024-05-28T06:16:59.149 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": 0.13801999390125275, 2024-05-28T06:16:59.149 INFO:teuthology.orchestra.run.smithi096.stdout: "add_to_ceph_s": true 2024-05-28T06:16:59.150 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:16:59.150 INFO:teuthology.orchestra.run.smithi096.stdout: "d1d4f508-f6fa-44f6-a2d0-6e7bd476fb5a": { 2024-05-28T06:16:59.150 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "2 ongoing+pending clones; Avg progress is 13.802% (59s)\n [===.........................] (remaining: 6m)", 2024-05-28T06:16:59.150 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": 0.13801999390125275, 2024-05-28T06:16:59.150 INFO:teuthology.orchestra.run.smithi096.stdout: "add_to_ceph_s": true 2024-05-28T06:16:59.150 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:16:59.150 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:16:59.150 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:16:59.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:59.146+0000 7f8cf7800640 1 -- 172.21.15.96:0/564444201 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8cd006d9d0 msgr2=0x7f8cd006fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:59.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:59.146+0000 7f8cf7800640 1 --2- 172.21.15.96:0/564444201 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8cd006d9d0 0x7f8cd006fe90 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f8ce000ab90 tx=0x7f8ce0009250 comp rx=0 tx=0).stop 2024-05-28T06:16:59.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:59.146+0000 7f8cf7800640 1 -- 172.21.15.96:0/564444201 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8cf0155a00 msgr2=0x7f8cf0146730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:16:59.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:59.146+0000 7f8cf7800640 1 --2- 172.21.15.96:0/564444201 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8cf0155a00 0x7f8cf0146730 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f8ce80090b0 tx=0x7f8ce80046c0 comp rx=0 tx=0).stop 2024-05-28T06:16:59.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:59.146+0000 7f8cf7800640 1 -- 172.21.15.96:0/564444201 shutdown_connections 2024-05-28T06:16:59.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:59.146+0000 7f8cf7800640 1 --2- 172.21.15.96:0/564444201 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8cd006d9d0 0x7f8cd006fe90 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:59.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:59.146+0000 7f8cf7800640 1 --2- 172.21.15.96:0/564444201 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8cf0155a00 0x7f8cf0146730 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:59.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:59.146+0000 7f8cf7800640 1 --2- 172.21.15.96:0/564444201 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8cf00d0970 0x7f8cf00a9490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:59.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:59.146+0000 7f8cf7800640 1 --2- 172.21.15.96:0/564444201 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8cf006f390 0x7f8cf0071f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:16:59.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:59.146+0000 7f8cf7800640 1 -- 172.21.15.96:0/564444201 >> 172.21.15.96:0/564444201 conn(0x7f8cf0067ce0 msgr2=0x7f8cf006b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:16:59.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:59.146+0000 7f8cf7800640 1 -- 172.21.15.96:0/564444201 shutdown_connections 2024-05-28T06:16:59.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:16:59.146+0000 7f8cf7800640 1 -- 172.21.15.96:0/564444201 wait complete. 2024-05-28T06:17:02.767 DEBUG:teuthology.orchestra.run.smithi096:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:17:02.775 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:17:04.167 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-05-28T06:17:04.374 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.374+0000 7faef7200640 1 Processor -- start 2024-05-28T06:17:04.374 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.375+0000 7faef7200640 1 -- start start 2024-05-28T06:17:04.375 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.376+0000 7faef7200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7faef815b380 0x7faef815ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:04.375 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.376+0000 7faef7200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faef815fe50 con 0x7faef80d0460 2024-05-28T06:17:04.375 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.376+0000 7faef7200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faef80cec60 con 0x7faef80ce800 2024-05-28T06:17:04.375 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.376+0000 7faef7200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7faef80b3310 con 0x7faef815b380 2024-05-28T06:17:04.376 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.376+0000 7faef6800640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7faef80d0460 0x7faef80b37a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:48718/0 (socket says 172.21.15.96:48718) 2024-05-28T06:17:04.376 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.376+0000 7faef6800640 1 -- 172.21.15.96:0/1082361366 learned_addr learned my addr 172.21.15.96:0/1082361366 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:17:04.376 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.377+0000 7faef5e00640 1 --2- 172.21.15.96:0/1082361366 >> v2:172.21.15.123:3300/0 conn(0x7faef815b380 0x7faef815ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:04.376 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.377+0000 7faef5400640 1 -- 172.21.15.96:0/1082361366 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3098135458 0 0) 0x7faef815fe50 con 0x7faef80d0460 2024-05-28T06:17:04.376 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.377+0000 7faef5400640 1 -- 172.21.15.96:0/1082361366 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faedc003520 con 0x7faef80d0460 2024-05-28T06:17:04.378 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.377+0000 7faef5400640 1 -- 172.21.15.96:0/1082361366 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2506959347 0 0) 0x7faedc003520 con 0x7faef80d0460 2024-05-28T06:17:04.378 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.377+0000 7faef5400640 1 -- 172.21.15.96:0/1082361366 >> v2:172.21.15.123:3300/0 conn(0x7faef815b380 msgr2=0x7faef815ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:04.378 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.377+0000 7faef5400640 1 --2- 172.21.15.96:0/1082361366 >> v2:172.21.15.123:3300/0 conn(0x7faef815b380 0x7faef815ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:04.378 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.377+0000 7faef5400640 1 -- 172.21.15.96:0/1082361366 >> v1:172.21.15.123:6789/0 conn(0x7faef80ce800 legacy=0x7faef80ad130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:04.378 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.377+0000 7faef5400640 1 -- 172.21.15.96:0/1082361366 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faef815f5b0 con 0x7faef80d0460 2024-05-28T06:17:04.378 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.377+0000 7faef5400640 1 -- 172.21.15.96:0/1082361366 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7faef00019f0 con 0x7faef80d0460 2024-05-28T06:17:04.378 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.377+0000 7faef5400640 1 -- 172.21.15.96:0/1082361366 >> v1:172.21.15.96:6789/0 conn(0x7faef80d0460 legacy=0x7faef80b37a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:04.378 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.377+0000 7faef5400640 1 --2- 172.21.15.96:0/1082361366 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faedc0039b0 0x7faedc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:04.378 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.377+0000 7faef5400640 1 --2- 172.21.15.96:0/1082361366 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faedc004800 0x7faedc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:04.378 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.377+0000 7faef5400640 1 --2- 172.21.15.96:0/1082361366 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faedc0052e0 0x7faedc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:04.378 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.378+0000 7faef5400640 1 -- 172.21.15.96:0/1082361366 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7faedc00c180 con 0x7faedc0052e0 2024-05-28T06:17:04.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.378+0000 7faef5400640 1 -- 172.21.15.96:0/1082361366 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7faedc00c330 con 0x7faedc0039b0 2024-05-28T06:17:04.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.378+0000 7faef5400640 1 -- 172.21.15.96:0/1082361366 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7faedc00c530 con 0x7faedc004800 2024-05-28T06:17:04.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.378+0000 7faef6800640 1 --2- 172.21.15.96:0/1082361366 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faedc0052e0 0x7faedc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:04.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.378+0000 7faef6800640 1 -- 172.21.15.96:0/1082361366 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faedc004800 msgr2=0x7faedc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:04.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.378+0000 7faeed400640 1 --2- 172.21.15.96:0/1082361366 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faedc004800 0x7faedc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:04.380 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.378+0000 7faef6800640 1 --2- 172.21.15.96:0/1082361366 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faedc004800 0x7faedc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:04.380 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.378+0000 7faef6800640 1 -- 172.21.15.96:0/1082361366 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faedc0039b0 msgr2=0x7faedc003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:04.380 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.378+0000 7faef6800640 1 --2- 172.21.15.96:0/1082361366 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faedc0039b0 0x7faedc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:04.380 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.378+0000 7faef6800640 1 -- 172.21.15.96:0/1082361366 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faedc003520 con 0x7faedc0052e0 2024-05-28T06:17:04.381 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.378+0000 7faeed400640 1 --2- 172.21.15.96:0/1082361366 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faedc004800 0x7faedc004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:17:04.381 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.379+0000 7faef6800640 1 -- 172.21.15.96:0/1082361366 reap_dead start 2024-05-28T06:17:04.381 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.379+0000 7faef6800640 1 --2- 172.21.15.96:0/1082361366 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faedc0052e0 0x7faedc00b9c0 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7faef000a810 tx=0x7faedc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1739ec73dabe0f02 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:04.381 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.380+0000 7faef5400640 1 -- 172.21.15.96:0/1082361366 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faef000a460 con 0x7faedc0052e0 2024-05-28T06:17:04.381 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.380+0000 7faef5400640 1 -- 172.21.15.96:0/1082361366 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faef002f070 con 0x7faedc0052e0 2024-05-28T06:17:04.381 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.380+0000 7faef5400640 1 -- 172.21.15.96:0/1082361366 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faef002bc90 con 0x7faedc0052e0 2024-05-28T06:17:04.381 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.380+0000 7faef7200640 1 -- 172.21.15.96:0/1082361366 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faedc0052e0 msgr2=0x7faedc00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:04.381 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.380+0000 7faef7200640 1 --2- 172.21.15.96:0/1082361366 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faedc0052e0 0x7faedc00b9c0 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7faef000a810 tx=0x7faedc005830 comp rx=0 tx=0).stop 2024-05-28T06:17:04.381 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.380+0000 7faef7200640 1 -- 172.21.15.96:0/1082361366 shutdown_connections 2024-05-28T06:17:04.381 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.380+0000 7faef7200640 1 --2- 172.21.15.96:0/1082361366 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faedc0052e0 0x7faedc00b9c0 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:04.381 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.380+0000 7faef7200640 1 -- 172.21.15.96:0/1082361366 >> 172.21.15.96:0/1082361366 conn(0x7faef806bae0 msgr2=0x7faef806bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:04.381 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.380+0000 7faef7200640 1 -- 172.21.15.96:0/1082361366 shutdown_connections 2024-05-28T06:17:04.381 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.380+0000 7faef7200640 1 -- 172.21.15.96:0/1082361366 wait complete. 2024-05-28T06:17:04.382 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.382+0000 7faef7200640 1 Processor -- start 2024-05-28T06:17:04.383 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.382+0000 7faef7200640 1 -- start start 2024-05-28T06:17:04.383 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.383+0000 7faef7200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faedc0052e0 0x7faef806f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:04.383 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.383+0000 7faef7200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faef80962d0 0x7faef80cbee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:04.383 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.383+0000 7faef7200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faef8073630 0x7faef80cc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:04.383 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.383+0000 7faef7200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7faef815b760 con 0x7faef80962d0 2024-05-28T06:17:04.383 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.383+0000 7faef7200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7faef815fb70 con 0x7faef8073630 2024-05-28T06:17:04.383 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.383+0000 7faef7200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7faef815b8e0 con 0x7faedc0052e0 2024-05-28T06:17:04.383 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.383+0000 7faeed400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faef80962d0 0x7faef80cbee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:04.383 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.384+0000 7faeed400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faef80962d0 0x7faef80cbee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:40346/0 (socket says 172.21.15.96:40346) 2024-05-28T06:17:04.383 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.384+0000 7faeed400640 1 -- 172.21.15.96:0/2430593002 learned_addr learned my addr 172.21.15.96:0/2430593002 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:17:04.384 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.384+0000 7faef5e00640 1 --2- 172.21.15.96:0/2430593002 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faedc0052e0 0x7faef806f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:04.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.384+0000 7faef6800640 1 --2- 172.21.15.96:0/2430593002 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faef8073630 0x7faef80cc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:04.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.384+0000 7faeed400640 1 -- 172.21.15.96:0/2430593002 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faedc0052e0 msgr2=0x7faef806f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:04.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.384+0000 7faeed400640 1 --2- 172.21.15.96:0/2430593002 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faedc0052e0 0x7faef806f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:04.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.384+0000 7faeed400640 1 -- 172.21.15.96:0/2430593002 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faef8073630 msgr2=0x7faef80cc420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:04.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.384+0000 7faeed400640 1 --2- 172.21.15.96:0/2430593002 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faef8073630 0x7faef80cc420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:04.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.384+0000 7faeed400640 1 -- 172.21.15.96:0/2430593002 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faef806f850 con 0x7faef80962d0 2024-05-28T06:17:04.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.384+0000 7faef6800640 1 --2- 172.21.15.96:0/2430593002 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faef8073630 0x7faef80cc420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:17:04.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.384+0000 7faeed400640 1 --2- 172.21.15.96:0/2430593002 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faef80962d0 0x7faef80cbee0 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7faee000d540 tx=0x7faee000da00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:04.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.384+0000 7faeeea00640 1 -- 172.21.15.96:0/2430593002 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faee0016070 con 0x7faef80962d0 2024-05-28T06:17:04.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.385+0000 7faeeea00640 1 -- 172.21.15.96:0/2430593002 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faee0012800 con 0x7faef80962d0 2024-05-28T06:17:04.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.385+0000 7faeeea00640 1 -- 172.21.15.96:0/2430593002 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faee0012bd0 con 0x7faef80962d0 2024-05-28T06:17:04.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.385+0000 7faef7200640 1 -- 172.21.15.96:0/2430593002 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faef806fb30 con 0x7faef80962d0 2024-05-28T06:17:04.386 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.385+0000 7faef7200640 1 -- 172.21.15.96:0/2430593002 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faef80cdaa0 con 0x7faef80962d0 2024-05-28T06:17:04.386 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.386+0000 7faef7200640 1 -- 172.21.15.96:0/2430593002 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faef80cd5b0 con 0x7faef80962d0 2024-05-28T06:17:04.386 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.387+0000 7faeeea00640 1 -- 172.21.15.96:0/2430593002 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 28) ==== 92570+0+0 (secure 0 0 0) 0x7faee0019070 con 0x7faef80962d0 2024-05-28T06:17:04.387 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.388+0000 7faeeea00640 1 --2- 172.21.15.96:0/2430593002 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7faed406d9d0 0x7faed406fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:04.387 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.388+0000 7faeeea00640 1 -- 172.21.15.96:0/2430593002 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5795+0+0 (secure 0 0 0) 0x7faee008a900 con 0x7faef80962d0 2024-05-28T06:17:04.387 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.388+0000 7faef5e00640 1 --2- 172.21.15.96:0/2430593002 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7faed406d9d0 0x7faed406fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:04.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.390+0000 7faef5e00640 1 --2- 172.21.15.96:0/2430593002 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7faed406d9d0 0x7faed406fe90 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7faef809e480 tx=0x7faee8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:04.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.393+0000 7faeeea00640 1 -- 172.21.15.96:0/2430593002 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7faee005a440 con 0x7faef80962d0 2024-05-28T06:17:04.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.665+0000 7faef7200640 1 -- 172.21.15.96:0/2430593002 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7faef80689e0 con 0x7faef80962d0 2024-05-28T06:17:04.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.666+0000 7faeeea00640 1 -- 172.21.15.96:0/2430593002 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+2381 (secure 0 0 0) 0x7faee0059d40 con 0x7faef80962d0 2024-05-28T06:17:04.666 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:17:04.666 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:17:04.666 INFO:teuthology.orchestra.run.smithi096.stdout: "fsid": "e4a831b5-e6d1-47a7-96cb-bf85154ee0d2", 2024-05-28T06:17:04.666 INFO:teuthology.orchestra.run.smithi096.stdout: "health": { 2024-05-28T06:17:04.666 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "HEALTH_OK", 2024-05-28T06:17:04.666 INFO:teuthology.orchestra.run.smithi096.stdout: "checks": {}, 2024-05-28T06:17:04.666 INFO:teuthology.orchestra.run.smithi096.stdout: "mutes": [] 2024-05-28T06:17:04.666 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:17:04.666 INFO:teuthology.orchestra.run.smithi096.stdout: "election_epoch": 8, 2024-05-28T06:17:04.666 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum": [ 2024-05-28T06:17:04.667 INFO:teuthology.orchestra.run.smithi096.stdout: 0, 2024-05-28T06:17:04.667 INFO:teuthology.orchestra.run.smithi096.stdout: 1, 2024-05-28T06:17:04.667 INFO:teuthology.orchestra.run.smithi096.stdout: 2 2024-05-28T06:17:04.667 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:17:04.667 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_names": [ 2024-05-28T06:17:04.667 INFO:teuthology.orchestra.run.smithi096.stdout: "a", 2024-05-28T06:17:04.667 INFO:teuthology.orchestra.run.smithi096.stdout: "b", 2024-05-28T06:17:04.667 INFO:teuthology.orchestra.run.smithi096.stdout: "c" 2024-05-28T06:17:04.667 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:17:04.667 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_age": 798, 2024-05-28T06:17:04.667 INFO:teuthology.orchestra.run.smithi096.stdout: "monmap": { 2024-05-28T06:17:04.667 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 1, 2024-05-28T06:17:04.667 INFO:teuthology.orchestra.run.smithi096.stdout: "min_mon_release_name": "squid", 2024-05-28T06:17:04.668 INFO:teuthology.orchestra.run.smithi096.stdout: "num_mons": 3 2024-05-28T06:17:04.668 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:17:04.668 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap": { 2024-05-28T06:17:04.668 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 76, 2024-05-28T06:17:04.668 INFO:teuthology.orchestra.run.smithi096.stdout: "num_osds": 8, 2024-05-28T06:17:04.668 INFO:teuthology.orchestra.run.smithi096.stdout: "num_up_osds": 8, 2024-05-28T06:17:04.668 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_up_since": 1716876238, 2024-05-28T06:17:04.668 INFO:teuthology.orchestra.run.smithi096.stdout: "num_in_osds": 8, 2024-05-28T06:17:04.668 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_in_since": 1716876229, 2024-05-28T06:17:04.668 INFO:teuthology.orchestra.run.smithi096.stdout: "num_remapped_pgs": 0 2024-05-28T06:17:04.668 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:17:04.668 INFO:teuthology.orchestra.run.smithi096.stdout: "pgmap": { 2024-05-28T06:17:04.668 INFO:teuthology.orchestra.run.smithi096.stdout: "pgs_by_state": [ 2024-05-28T06:17:04.668 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:17:04.669 INFO:teuthology.orchestra.run.smithi096.stdout: "state_name": "active+clean", 2024-05-28T06:17:04.669 INFO:teuthology.orchestra.run.smithi096.stdout: "count": 97 2024-05-28T06:17:04.669 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:17:04.669 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:17:04.669 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pgs": 97, 2024-05-28T06:17:04.669 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pools": 3, 2024-05-28T06:17:04.669 INFO:teuthology.orchestra.run.smithi096.stdout: "num_objects": 12470, 2024-05-28T06:17:04.669 INFO:teuthology.orchestra.run.smithi096.stdout: "data_bytes": 52089716222, 2024-05-28T06:17:04.669 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_used": 104815869952, 2024-05-28T06:17:04.669 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_avail": 668278243328, 2024-05-28T06:17:04.669 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_total": 773094113280, 2024-05-28T06:17:04.669 INFO:teuthology.orchestra.run.smithi096.stdout: "read_bytes_sec": 428621488, 2024-05-28T06:17:04.669 INFO:teuthology.orchestra.run.smithi096.stdout: "write_bytes_sec": 429723180, 2024-05-28T06:17:04.669 INFO:teuthology.orchestra.run.smithi096.stdout: "read_op_per_sec": 106, 2024-05-28T06:17:04.670 INFO:teuthology.orchestra.run.smithi096.stdout: "write_op_per_sec": 121 2024-05-28T06:17:04.670 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:17:04.670 INFO:teuthology.orchestra.run.smithi096.stdout: "fsmap": { 2024-05-28T06:17:04.670 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 44, 2024-05-28T06:17:04.670 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 5, 2024-05-28T06:17:04.670 INFO:teuthology.orchestra.run.smithi096.stdout: "up": 1, 2024-05-28T06:17:04.670 INFO:teuthology.orchestra.run.smithi096.stdout: "in": 1, 2024-05-28T06:17:04.670 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 1, 2024-05-28T06:17:04.670 INFO:teuthology.orchestra.run.smithi096.stdout: "by_rank": [ 2024-05-28T06:17:04.670 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:17:04.670 INFO:teuthology.orchestra.run.smithi096.stdout: "filesystem_id": 5, 2024-05-28T06:17:04.670 INFO:teuthology.orchestra.run.smithi096.stdout: "rank": 0, 2024-05-28T06:17:04.670 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "d", 2024-05-28T06:17:04.671 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "up:active", 2024-05-28T06:17:04.671 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 7146 2024-05-28T06:17:04.671 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:17:04.671 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:17:04.671 INFO:teuthology.orchestra.run.smithi096.stdout: "up:standby": 3 2024-05-28T06:17:04.671 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:17:04.671 INFO:teuthology.orchestra.run.smithi096.stdout: "mgrmap": { 2024-05-28T06:17:04.671 INFO:teuthology.orchestra.run.smithi096.stdout: "available": true, 2024-05-28T06:17:04.671 INFO:teuthology.orchestra.run.smithi096.stdout: "num_standbys": 1, 2024-05-28T06:17:04.671 INFO:teuthology.orchestra.run.smithi096.stdout: "modules": [ 2024-05-28T06:17:04.671 INFO:teuthology.orchestra.run.smithi096.stdout: "iostat", 2024-05-28T06:17:04.671 INFO:teuthology.orchestra.run.smithi096.stdout: "nfs", 2024-05-28T06:17:04.671 INFO:teuthology.orchestra.run.smithi096.stdout: "restful" 2024-05-28T06:17:04.671 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:17:04.672 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:17:04.672 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:17:04.672 INFO:teuthology.orchestra.run.smithi096.stdout: "servicemap": { 2024-05-28T06:17:04.672 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 13, 2024-05-28T06:17:04.672 INFO:teuthology.orchestra.run.smithi096.stdout: "modified": "2024-05-28T06:15:57.778889+0000", 2024-05-28T06:17:04.672 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:17:04.672 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:17:04.672 INFO:teuthology.orchestra.run.smithi096.stdout: "progress_events": { 2024-05-28T06:17:04.672 INFO:teuthology.orchestra.run.smithi096.stdout: "1f539510-8a8a-46db-8f0c-e233f5cc901d": { 2024-05-28T06:17:04.672 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "All cloning finished, waiting 26.0 seconds before exiting (74s)\n [============================] ", 2024-05-28T06:17:04.672 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": 1, 2024-05-28T06:17:04.672 INFO:teuthology.orchestra.run.smithi096.stdout: "add_to_ceph_s": true 2024-05-28T06:17:04.672 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:17:04.673 INFO:teuthology.orchestra.run.smithi096.stdout: "d1d4f508-f6fa-44f6-a2d0-6e7bd476fb5a": { 2024-05-28T06:17:04.673 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "No pending clones left, waiting 26.0 seconds before exiting (65s)\n [============================] ", 2024-05-28T06:17:04.673 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": 1, 2024-05-28T06:17:04.673 INFO:teuthology.orchestra.run.smithi096.stdout: "add_to_ceph_s": true 2024-05-28T06:17:04.673 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:17:04.673 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:17:04.673 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:17:04.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.669+0000 7faef7200640 1 -- 172.21.15.96:0/2430593002 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7faed406d9d0 msgr2=0x7faed406fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:04.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.669+0000 7faef7200640 1 --2- 172.21.15.96:0/2430593002 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7faed406d9d0 0x7faed406fe90 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7faef809e480 tx=0x7faee8008040 comp rx=0 tx=0).stop 2024-05-28T06:17:04.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.669+0000 7faef7200640 1 -- 172.21.15.96:0/2430593002 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faef80962d0 msgr2=0x7faef80cbee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:04.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.669+0000 7faef7200640 1 --2- 172.21.15.96:0/2430593002 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faef80962d0 0x7faef80cbee0 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7faee000d540 tx=0x7faee000da00 comp rx=0 tx=0).stop 2024-05-28T06:17:04.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.669+0000 7faef7200640 1 -- 172.21.15.96:0/2430593002 shutdown_connections 2024-05-28T06:17:04.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.669+0000 7faef7200640 1 --2- 172.21.15.96:0/2430593002 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7faed406d9d0 0x7faed406fe90 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:04.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.669+0000 7faef7200640 1 --2- 172.21.15.96:0/2430593002 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faef8073630 0x7faef80cc420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:04.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.669+0000 7faef7200640 1 --2- 172.21.15.96:0/2430593002 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faef80962d0 0x7faef80cbee0 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:04.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.669+0000 7faef7200640 1 --2- 172.21.15.96:0/2430593002 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faedc0052e0 0x7faef806f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:04.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.669+0000 7faef7200640 1 -- 172.21.15.96:0/2430593002 >> 172.21.15.96:0/2430593002 conn(0x7faef806bae0 msgr2=0x7faef8152430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:04.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.669+0000 7faef7200640 1 -- 172.21.15.96:0/2430593002 shutdown_connections 2024-05-28T06:17:04.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:04.669+0000 7faef7200640 1 -- 172.21.15.96:0/2430593002 wait complete. 2024-05-28T06:17:09.688 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-05-28T06:17:09.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:09.899+0000 7f8be1c00640 1 Processor -- start 2024-05-28T06:17:09.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:09.899+0000 7f8be1c00640 1 -- start start 2024-05-28T06:17:09.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:09.901+0000 7f8be1c00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f8bdc161df0 0x7f8bdc099800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:09.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:09.901+0000 7f8be1c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8bdc0cc2b0 0x7f8bdc0d0460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:09.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:09.901+0000 7f8be1c00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8bdc1621d0 con 0x7f8bdc0d09a0 2024-05-28T06:17:09.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:09.901+0000 7f8be1c00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f8bdc0cd840 con 0x7f8bdc161df0 2024-05-28T06:17:09.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:09.901+0000 7f8be1c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f8bdc0ccf10 con 0x7f8bdc0cc2b0 2024-05-28T06:17:09.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:09.901+0000 7f8bdbe00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f8bdc161df0 0x7f8bdc099800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:09.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:09.901+0000 7f8bdbe00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f8bdc161df0 0x7f8bdc099800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:34340/0 (socket says 172.21.15.96:34340) 2024-05-28T06:17:09.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:09.901+0000 7f8bdbe00640 1 -- 172.21.15.96:0/3131055196 learned_addr learned my addr 172.21.15.96:0/3131055196 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:17:09.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:09.902+0000 7f8bdb400640 1 --2- 172.21.15.96:0/3131055196 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8bdc0cc2b0 0x7f8bdc0d0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:09.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:09.902+0000 7f8bdbe00640 1 -- 172.21.15.96:0/3131055196 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8bdc0cc2b0 msgr2=0x7f8bdc0d0460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:09.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:09.902+0000 7f8bdbe00640 1 --2- 172.21.15.96:0/3131055196 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8bdc0cc2b0 0x7f8bdc0d0460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:09.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:09.902+0000 7f8bdbe00640 1 -- 172.21.15.96:0/3131055196 >> v1:172.21.15.123:6789/0 conn(0x7f8bdc0d09a0 legacy=0x7f8bdc0c50c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:09.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:09.902+0000 7f8bdbe00640 1 -- 172.21.15.96:0/3131055196 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bdc0cd1f0 con 0x7f8bdc161df0 2024-05-28T06:17:09.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:09.902+0000 7f8bdb400640 1 --2- 172.21.15.96:0/3131055196 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8bdc0cc2b0 0x7f8bdc0d0460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:17:09.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:09.903+0000 7f8bdbe00640 1 --2- 172.21.15.96:0/3131055196 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8bdc161df0 0x7f8bdc099800 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f8bcc009870 tx=0x7f8bcc02a200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb91df87fd90969d server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:09.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:09.903+0000 7f8bdaa00640 1 -- 172.21.15.96:0/3131055196 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bcc037070 con 0x7f8bdc161df0 2024-05-28T06:17:09.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:09.903+0000 7f8bdaa00640 1 -- 172.21.15.96:0/3131055196 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8bcc032440 con 0x7f8bdc161df0 2024-05-28T06:17:09.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:09.903+0000 7f8bdaa00640 1 -- 172.21.15.96:0/3131055196 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bcc032810 con 0x7f8bdc161df0 2024-05-28T06:17:09.904 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:09.904+0000 7f8be1c00640 1 -- 172.21.15.96:0/3131055196 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8bdc161df0 msgr2=0x7f8bdc099800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:09.904 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:09.904+0000 7f8be1c00640 1 --2- 172.21.15.96:0/3131055196 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8bdc161df0 0x7f8bdc099800 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f8bcc009870 tx=0x7f8bcc02a200 comp rx=0 tx=0).stop 2024-05-28T06:17:09.904 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:09.904+0000 7f8be1c00640 1 -- 172.21.15.96:0/3131055196 shutdown_connections 2024-05-28T06:17:09.904 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:09.904+0000 7f8be1c00640 1 --2- 172.21.15.96:0/3131055196 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8bdc0cc2b0 0x7f8bdc0d0460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:09.904 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:09.904+0000 7f8be1c00640 1 --2- 172.21.15.96:0/3131055196 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8bdc161df0 0x7f8bdc099800 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:09.904 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:09.904+0000 7f8be1c00640 1 -- 172.21.15.96:0/3131055196 >> 172.21.15.96:0/3131055196 conn(0x7f8bdc06bae0 msgr2=0x7f8bdc06bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:09.904 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:09.904+0000 7f8be1c00640 1 -- 172.21.15.96:0/3131055196 shutdown_connections 2024-05-28T06:17:09.904 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:09.904+0000 7f8be1c00640 1 -- 172.21.15.96:0/3131055196 wait complete. 2024-05-28T06:17:09.905 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:09.906+0000 7f8be1c00640 1 Processor -- start 2024-05-28T06:17:09.905 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:09.906+0000 7f8be1c00640 1 -- start start 2024-05-28T06:17:09.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:09.907+0000 7f8be1c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8bdc0cc2b0 0x7f8bdc073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:09.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:09.907+0000 7f8be1c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8bdc0d09a0 0x7f8bdc160510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:09.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:09.907+0000 7f8be1c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8bdc161df0 0x7f8bdc15d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:09.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:09.907+0000 7f8be1c00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f8bdc0cffb0 con 0x7f8bdc161df0 2024-05-28T06:17:09.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:09.907+0000 7f8be1c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f8bdc099d40 con 0x7f8bdc0d09a0 2024-05-28T06:17:09.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:09.907+0000 7f8be1c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f8bdc0ccbc0 con 0x7f8bdc0cc2b0 2024-05-28T06:17:09.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:09.908+0000 7f8be1200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8bdc161df0 0x7f8bdc15d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:09.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:09.908+0000 7f8be1200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8bdc161df0 0x7f8bdc15d830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:34342/0 (socket says 172.21.15.96:34342) 2024-05-28T06:17:09.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:09.908+0000 7f8be1200640 1 -- 172.21.15.96:0/4082492659 learned_addr learned my addr 172.21.15.96:0/4082492659 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:17:09.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:09.908+0000 7f8bdbe00640 1 --2- 172.21.15.96:0/4082492659 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8bdc0cc2b0 0x7f8bdc073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:09.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:09.908+0000 7f8be1200640 1 -- 172.21.15.96:0/4082492659 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8bdc0cc2b0 msgr2=0x7f8bdc073330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:09.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:09.908+0000 7f8be1200640 1 --2- 172.21.15.96:0/4082492659 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8bdc0cc2b0 0x7f8bdc073330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:09.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:09.908+0000 7f8bdb400640 1 --2- 172.21.15.96:0/4082492659 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8bdc0d09a0 0x7f8bdc160510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:09.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:09.908+0000 7f8be1200640 1 -- 172.21.15.96:0/4082492659 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8bdc0d09a0 msgr2=0x7f8bdc160510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:09.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:09.908+0000 7f8be1200640 1 --2- 172.21.15.96:0/4082492659 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8bdc0d09a0 0x7f8bdc160510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:09.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:09.908+0000 7f8be1200640 1 -- 172.21.15.96:0/4082492659 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bdc072290 con 0x7f8bdc161df0 2024-05-28T06:17:09.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:09.908+0000 7f8be1200640 1 --2- 172.21.15.96:0/4082492659 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8bdc161df0 0x7f8bdc15d830 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f8bd400c780 tx=0x7f8bd400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:09.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:09.909+0000 7f8bd3e00640 1 -- 172.21.15.96:0/4082492659 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bd4004b60 con 0x7f8bdc161df0 2024-05-28T06:17:09.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:09.909+0000 7f8bd3e00640 1 -- 172.21.15.96:0/4082492659 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8bd4004d00 con 0x7f8bdc161df0 2024-05-28T06:17:09.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:09.909+0000 7f8bd3e00640 1 -- 172.21.15.96:0/4082492659 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bd400aa90 con 0x7f8bdc161df0 2024-05-28T06:17:09.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:09.909+0000 7f8be1c00640 1 -- 172.21.15.96:0/4082492659 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bdc072570 con 0x7f8bdc161df0 2024-05-28T06:17:09.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:09.909+0000 7f8be1c00640 1 -- 172.21.15.96:0/4082492659 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bdc072b30 con 0x7f8bdc161df0 2024-05-28T06:17:09.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:09.910+0000 7f8be1c00640 1 -- 172.21.15.96:0/4082492659 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8bdc0689e0 con 0x7f8bdc161df0 2024-05-28T06:17:09.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:09.911+0000 7f8bd3e00640 1 -- 172.21.15.96:0/4082492659 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 28) ==== 92570+0+0 (secure 0 0 0) 0x7f8bd400ac30 con 0x7f8bdc161df0 2024-05-28T06:17:09.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:09.912+0000 7f8bd3e00640 1 --2- 172.21.15.96:0/4082492659 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8bbc06d9d0 0x7f8bbc06fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:09.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:09.912+0000 7f8bd3e00640 1 -- 172.21.15.96:0/4082492659 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5795+0+0 (secure 0 0 0) 0x7f8bd408ad80 con 0x7f8bdc161df0 2024-05-28T06:17:09.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:09.912+0000 7f8bdbe00640 1 --2- 172.21.15.96:0/4082492659 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8bbc06d9d0 0x7f8bbc06fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:09.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:09.914+0000 7f8bdbe00640 1 --2- 172.21.15.96:0/4082492659 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8bbc06d9d0 0x7f8bbc06fe90 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f8bcc034000 tx=0x7f8bcc02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:09.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:09.917+0000 7f8bd3e00640 1 -- 172.21.15.96:0/4082492659 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8bd4020070 con 0x7f8bdc161df0 2024-05-28T06:17:10.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:10.191+0000 7f8be1c00640 1 -- 172.21.15.96:0/4082492659 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f8bdc06c240 con 0x7f8bdc161df0 2024-05-28T06:17:10.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:10.192+0000 7f8bd3e00640 1 -- 172.21.15.96:0/4082492659 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+2379 (secure 0 0 0) 0x7f8bd405a8c0 con 0x7f8bdc161df0 2024-05-28T06:17:10.191 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:17:10.192 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:17:10.192 INFO:teuthology.orchestra.run.smithi096.stdout: "fsid": "e4a831b5-e6d1-47a7-96cb-bf85154ee0d2", 2024-05-28T06:17:10.192 INFO:teuthology.orchestra.run.smithi096.stdout: "health": { 2024-05-28T06:17:10.192 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "HEALTH_OK", 2024-05-28T06:17:10.192 INFO:teuthology.orchestra.run.smithi096.stdout: "checks": {}, 2024-05-28T06:17:10.192 INFO:teuthology.orchestra.run.smithi096.stdout: "mutes": [] 2024-05-28T06:17:10.192 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:17:10.192 INFO:teuthology.orchestra.run.smithi096.stdout: "election_epoch": 8, 2024-05-28T06:17:10.192 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum": [ 2024-05-28T06:17:10.192 INFO:teuthology.orchestra.run.smithi096.stdout: 0, 2024-05-28T06:17:10.192 INFO:teuthology.orchestra.run.smithi096.stdout: 1, 2024-05-28T06:17:10.192 INFO:teuthology.orchestra.run.smithi096.stdout: 2 2024-05-28T06:17:10.192 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:17:10.193 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_names": [ 2024-05-28T06:17:10.193 INFO:teuthology.orchestra.run.smithi096.stdout: "a", 2024-05-28T06:17:10.193 INFO:teuthology.orchestra.run.smithi096.stdout: "b", 2024-05-28T06:17:10.193 INFO:teuthology.orchestra.run.smithi096.stdout: "c" 2024-05-28T06:17:10.193 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:17:10.193 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_age": 803, 2024-05-28T06:17:10.193 INFO:teuthology.orchestra.run.smithi096.stdout: "monmap": { 2024-05-28T06:17:10.193 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 1, 2024-05-28T06:17:10.193 INFO:teuthology.orchestra.run.smithi096.stdout: "min_mon_release_name": "squid", 2024-05-28T06:17:10.193 INFO:teuthology.orchestra.run.smithi096.stdout: "num_mons": 3 2024-05-28T06:17:10.193 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:17:10.193 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap": { 2024-05-28T06:17:10.193 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 76, 2024-05-28T06:17:10.193 INFO:teuthology.orchestra.run.smithi096.stdout: "num_osds": 8, 2024-05-28T06:17:10.194 INFO:teuthology.orchestra.run.smithi096.stdout: "num_up_osds": 8, 2024-05-28T06:17:10.194 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_up_since": 1716876238, 2024-05-28T06:17:10.194 INFO:teuthology.orchestra.run.smithi096.stdout: "num_in_osds": 8, 2024-05-28T06:17:10.194 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_in_since": 1716876229, 2024-05-28T06:17:10.194 INFO:teuthology.orchestra.run.smithi096.stdout: "num_remapped_pgs": 0 2024-05-28T06:17:10.194 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:17:10.194 INFO:teuthology.orchestra.run.smithi096.stdout: "pgmap": { 2024-05-28T06:17:10.194 INFO:teuthology.orchestra.run.smithi096.stdout: "pgs_by_state": [ 2024-05-28T06:17:10.194 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:17:10.194 INFO:teuthology.orchestra.run.smithi096.stdout: "state_name": "active+clean", 2024-05-28T06:17:10.194 INFO:teuthology.orchestra.run.smithi096.stdout: "count": 97 2024-05-28T06:17:10.194 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:17:10.194 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:17:10.195 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pgs": 97, 2024-05-28T06:17:10.195 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pools": 3, 2024-05-28T06:17:10.195 INFO:teuthology.orchestra.run.smithi096.stdout: "num_objects": 12467, 2024-05-28T06:17:10.195 INFO:teuthology.orchestra.run.smithi096.stdout: "data_bytes": 52089791936, 2024-05-28T06:17:10.195 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_used": 104815947776, 2024-05-28T06:17:10.195 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_avail": 668278165504, 2024-05-28T06:17:10.195 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_total": 773094113280, 2024-05-28T06:17:10.195 INFO:teuthology.orchestra.run.smithi096.stdout: "read_bytes_sec": 154079086, 2024-05-28T06:17:10.195 INFO:teuthology.orchestra.run.smithi096.stdout: "write_bytes_sec": 157601222, 2024-05-28T06:17:10.195 INFO:teuthology.orchestra.run.smithi096.stdout: "read_op_per_sec": 38, 2024-05-28T06:17:10.195 INFO:teuthology.orchestra.run.smithi096.stdout: "write_op_per_sec": 48 2024-05-28T06:17:10.195 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:17:10.195 INFO:teuthology.orchestra.run.smithi096.stdout: "fsmap": { 2024-05-28T06:17:10.195 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 44, 2024-05-28T06:17:10.196 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 5, 2024-05-28T06:17:10.196 INFO:teuthology.orchestra.run.smithi096.stdout: "up": 1, 2024-05-28T06:17:10.196 INFO:teuthology.orchestra.run.smithi096.stdout: "in": 1, 2024-05-28T06:17:10.196 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 1, 2024-05-28T06:17:10.196 INFO:teuthology.orchestra.run.smithi096.stdout: "by_rank": [ 2024-05-28T06:17:10.196 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:17:10.196 INFO:teuthology.orchestra.run.smithi096.stdout: "filesystem_id": 5, 2024-05-28T06:17:10.196 INFO:teuthology.orchestra.run.smithi096.stdout: "rank": 0, 2024-05-28T06:17:10.196 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "d", 2024-05-28T06:17:10.196 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "up:active", 2024-05-28T06:17:10.196 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 7146 2024-05-28T06:17:10.196 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:17:10.196 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:17:10.196 INFO:teuthology.orchestra.run.smithi096.stdout: "up:standby": 3 2024-05-28T06:17:10.197 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:17:10.197 INFO:teuthology.orchestra.run.smithi096.stdout: "mgrmap": { 2024-05-28T06:17:10.197 INFO:teuthology.orchestra.run.smithi096.stdout: "available": true, 2024-05-28T06:17:10.197 INFO:teuthology.orchestra.run.smithi096.stdout: "num_standbys": 1, 2024-05-28T06:17:10.197 INFO:teuthology.orchestra.run.smithi096.stdout: "modules": [ 2024-05-28T06:17:10.197 INFO:teuthology.orchestra.run.smithi096.stdout: "iostat", 2024-05-28T06:17:10.197 INFO:teuthology.orchestra.run.smithi096.stdout: "nfs", 2024-05-28T06:17:10.197 INFO:teuthology.orchestra.run.smithi096.stdout: "restful" 2024-05-28T06:17:10.197 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:17:10.197 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:17:10.197 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:17:10.197 INFO:teuthology.orchestra.run.smithi096.stdout: "servicemap": { 2024-05-28T06:17:10.197 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 13, 2024-05-28T06:17:10.197 INFO:teuthology.orchestra.run.smithi096.stdout: "modified": "2024-05-28T06:15:57.778889+0000", 2024-05-28T06:17:10.198 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:17:10.198 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:17:10.198 INFO:teuthology.orchestra.run.smithi096.stdout: "progress_events": { 2024-05-28T06:17:10.198 INFO:teuthology.orchestra.run.smithi096.stdout: "1f539510-8a8a-46db-8f0c-e233f5cc901d": { 2024-05-28T06:17:10.198 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "All cloning finished, waiting 20.0 seconds before exiting (80s)\n [============================] ", 2024-05-28T06:17:10.198 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": 1, 2024-05-28T06:17:10.198 INFO:teuthology.orchestra.run.smithi096.stdout: "add_to_ceph_s": true 2024-05-28T06:17:10.198 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:17:10.198 INFO:teuthology.orchestra.run.smithi096.stdout: "d1d4f508-f6fa-44f6-a2d0-6e7bd476fb5a": { 2024-05-28T06:17:10.198 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "No pending clones left, waiting 20.0 seconds before exiting (71s)\n [============================] ", 2024-05-28T06:17:10.198 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": 1, 2024-05-28T06:17:10.199 INFO:teuthology.orchestra.run.smithi096.stdout: "add_to_ceph_s": true 2024-05-28T06:17:10.199 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:17:10.199 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:17:10.199 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:17:10.199 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:10.195+0000 7f8be1c00640 1 -- 172.21.15.96:0/4082492659 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8bbc06d9d0 msgr2=0x7f8bbc06fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:10.200 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:10.195+0000 7f8be1c00640 1 --2- 172.21.15.96:0/4082492659 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8bbc06d9d0 0x7f8bbc06fe90 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f8bcc034000 tx=0x7f8bcc02e000 comp rx=0 tx=0).stop 2024-05-28T06:17:10.200 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:10.195+0000 7f8be1c00640 1 -- 172.21.15.96:0/4082492659 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8bdc161df0 msgr2=0x7f8bdc15d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:10.200 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:10.195+0000 7f8be1c00640 1 --2- 172.21.15.96:0/4082492659 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8bdc161df0 0x7f8bdc15d830 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f8bd400c780 tx=0x7f8bd400cc40 comp rx=0 tx=0).stop 2024-05-28T06:17:10.200 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:10.195+0000 7f8be1c00640 1 -- 172.21.15.96:0/4082492659 shutdown_connections 2024-05-28T06:17:10.200 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:10.195+0000 7f8be1c00640 1 --2- 172.21.15.96:0/4082492659 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8bbc06d9d0 0x7f8bbc06fe90 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:10.200 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:10.195+0000 7f8be1c00640 1 --2- 172.21.15.96:0/4082492659 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8bdc161df0 0x7f8bdc15d830 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:10.200 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:10.195+0000 7f8be1c00640 1 --2- 172.21.15.96:0/4082492659 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8bdc0d09a0 0x7f8bdc160510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:10.200 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:10.195+0000 7f8be1c00640 1 --2- 172.21.15.96:0/4082492659 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8bdc0cc2b0 0x7f8bdc073330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:10.200 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:10.195+0000 7f8be1c00640 1 -- 172.21.15.96:0/4082492659 >> 172.21.15.96:0/4082492659 conn(0x7f8bdc06bae0 msgr2=0x7f8bdc0b4cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:10.200 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:10.195+0000 7f8be1c00640 1 -- 172.21.15.96:0/4082492659 shutdown_connections 2024-05-28T06:17:10.200 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:10.196+0000 7f8be1c00640 1 -- 172.21.15.96:0/4082492659 wait complete. 2024-05-28T06:17:15.218 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-05-28T06:17:15.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:15.429+0000 7fc24c600640 1 Processor -- start 2024-05-28T06:17:15.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:15.429+0000 7fc24c600640 1 -- start start 2024-05-28T06:17:15.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:15.431+0000 7fc24c600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fc244145320 0x7fc244071cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:15.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:15.431+0000 7fc24c600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fc2440b43b0 0x7fc2440cf880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:15.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:15.431+0000 7fc24c600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2440cf0c0 con 0x7fc244072200 2024-05-28T06:17:15.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:15.431+0000 7fc24c600640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fc2440b4a20 con 0x7fc244145320 2024-05-28T06:17:15.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:15.431+0000 7fc24c600640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fc2440b4d00 con 0x7fc2440b43b0 2024-05-28T06:17:15.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:15.431+0000 7fc249c00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fc244145320 0x7fc244071cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:15.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:15.431+0000 7fc249c00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fc244145320 0x7fc244071cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:34356/0 (socket says 172.21.15.96:34356) 2024-05-28T06:17:15.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:15.431+0000 7fc249c00640 1 -- 172.21.15.96:0/3713449636 learned_addr learned my addr 172.21.15.96:0/3713449636 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:17:15.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:15.432+0000 7fc249c00640 1 -- 172.21.15.96:0/3713449636 >> v2:172.21.15.123:3300/0 conn(0x7fc2440b43b0 msgr2=0x7fc2440cf880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:15.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:15.432+0000 7fc249c00640 1 --2- 172.21.15.96:0/3713449636 >> v2:172.21.15.123:3300/0 conn(0x7fc2440b43b0 0x7fc2440cf880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:15.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:15.432+0000 7fc249c00640 1 -- 172.21.15.96:0/3713449636 >> v1:172.21.15.123:6789/0 conn(0x7fc244072200 legacy=0x7fc2440b3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:15.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:15.432+0000 7fc249c00640 1 -- 172.21.15.96:0/3713449636 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2440b4fe0 con 0x7fc244145320 2024-05-28T06:17:15.433 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:15.433+0000 7fc249c00640 1 --2- 172.21.15.96:0/3713449636 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc244145320 0x7fc244071cc0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7fc2440b5590 tx=0x7fc23802a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76d20059cee3ea24 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:15.433 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:15.433+0000 7fc243e00640 1 -- 172.21.15.96:0/3713449636 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc238037070 con 0x7fc244145320 2024-05-28T06:17:15.433 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:15.433+0000 7fc243e00640 1 -- 172.21.15.96:0/3713449636 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc238032440 con 0x7fc244145320 2024-05-28T06:17:15.433 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:15.433+0000 7fc243e00640 1 -- 172.21.15.96:0/3713449636 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc238032810 con 0x7fc244145320 2024-05-28T06:17:15.433 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:15.434+0000 7fc24c600640 1 -- 172.21.15.96:0/3713449636 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc244145320 msgr2=0x7fc244071cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:15.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:15.434+0000 7fc24c600640 1 --2- 172.21.15.96:0/3713449636 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc244145320 0x7fc244071cc0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7fc2440b5590 tx=0x7fc23802a230 comp rx=0 tx=0).stop 2024-05-28T06:17:15.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:15.434+0000 7fc24c600640 1 -- 172.21.15.96:0/3713449636 shutdown_connections 2024-05-28T06:17:15.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:15.434+0000 7fc24c600640 1 --2- 172.21.15.96:0/3713449636 >> v2:172.21.15.123:3300/0 conn(0x7fc2440b43b0 0x7fc2440cf880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:15.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:15.434+0000 7fc24c600640 1 --2- 172.21.15.96:0/3713449636 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc244145320 0x7fc244071cc0 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:15.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:15.434+0000 7fc24c600640 1 -- 172.21.15.96:0/3713449636 >> 172.21.15.96:0/3713449636 conn(0x7fc244067ce0 msgr2=0x7fc24406b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:15.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:15.434+0000 7fc24c600640 1 -- 172.21.15.96:0/3713449636 shutdown_connections 2024-05-28T06:17:15.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:15.434+0000 7fc24c600640 1 -- 172.21.15.96:0/3713449636 wait complete. 2024-05-28T06:17:15.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:15.436+0000 7fc24c600640 1 Processor -- start 2024-05-28T06:17:15.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:15.436+0000 7fc24c600640 1 -- start start 2024-05-28T06:17:15.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:15.437+0000 7fc24c600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc244072200 0x7fc244142e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:15.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:15.437+0000 7fc24c600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc2440b43b0 0x7fc2441e5e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:15.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:15.437+0000 7fc24c600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc244145320 0x7fc2441445b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:15.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:15.437+0000 7fc24c600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc2440cfdc0 con 0x7fc2440b43b0 2024-05-28T06:17:15.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:15.437+0000 7fc24c600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc244145700 con 0x7fc244072200 2024-05-28T06:17:15.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:15.437+0000 7fc24c600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fc2440cff40 con 0x7fc244145320 2024-05-28T06:17:15.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:15.438+0000 7fc249200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc2440b43b0 0x7fc2441e5e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:15.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:15.438+0000 7fc249200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc2440b43b0 0x7fc2441e5e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:34368/0 (socket says 172.21.15.96:34368) 2024-05-28T06:17:15.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:15.438+0000 7fc249200640 1 -- 172.21.15.96:0/1331367452 learned_addr learned my addr 172.21.15.96:0/1331367452 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:17:15.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:15.438+0000 7fc249c00640 1 --2- 172.21.15.96:0/1331367452 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc244072200 0x7fc244142e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:15.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:15.438+0000 7fc24a600640 1 --2- 172.21.15.96:0/1331367452 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc244145320 0x7fc2441445b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:15.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:15.438+0000 7fc249200640 1 -- 172.21.15.96:0/1331367452 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc244145320 msgr2=0x7fc2441445b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:15.439 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:15.438+0000 7fc249200640 1 --2- 172.21.15.96:0/1331367452 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc244145320 0x7fc2441445b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:15.439 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:15.438+0000 7fc249200640 1 -- 172.21.15.96:0/1331367452 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc244072200 msgr2=0x7fc244142e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:15.439 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:15.438+0000 7fc249200640 1 --2- 172.21.15.96:0/1331367452 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc244072200 0x7fc244142e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:15.439 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:15.438+0000 7fc249200640 1 -- 172.21.15.96:0/1331367452 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc244143510 con 0x7fc2440b43b0 2024-05-28T06:17:15.439 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:15.439+0000 7fc249200640 1 --2- 172.21.15.96:0/1331367452 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc2440b43b0 0x7fc2441e5e20 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7fc23000c410 tx=0x7fc23000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:15.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:15.439+0000 7fc241600640 1 -- 172.21.15.96:0/1331367452 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc230017070 con 0x7fc2440b43b0 2024-05-28T06:17:15.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:15.439+0000 7fc241600640 1 -- 172.21.15.96:0/1331367452 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2300035f0 con 0x7fc2440b43b0 2024-05-28T06:17:15.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:15.439+0000 7fc24c600640 1 -- 172.21.15.96:0/1331367452 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2441437f0 con 0x7fc2440b43b0 2024-05-28T06:17:15.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:15.439+0000 7fc241600640 1 -- 172.21.15.96:0/1331367452 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc230003a20 con 0x7fc2440b43b0 2024-05-28T06:17:15.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:15.439+0000 7fc24c600640 1 -- 172.21.15.96:0/1331367452 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc244143db0 con 0x7fc2440b43b0 2024-05-28T06:17:15.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:15.440+0000 7fc24c600640 1 -- 172.21.15.96:0/1331367452 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2440689e0 con 0x7fc2440b43b0 2024-05-28T06:17:15.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:15.441+0000 7fc241600640 1 -- 172.21.15.96:0/1331367452 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 28) ==== 92570+0+0 (secure 0 0 0) 0x7fc23001a070 con 0x7fc2440b43b0 2024-05-28T06:17:15.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:15.442+0000 7fc241600640 1 --2- 172.21.15.96:0/1331367452 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc22406d9d0 0x7fc22406fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:15.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:15.442+0000 7fc241600640 1 -- 172.21.15.96:0/1331367452 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5795+0+0 (secure 0 0 0) 0x7fc230089ed0 con 0x7fc2440b43b0 2024-05-28T06:17:15.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:15.443+0000 7fc249c00640 1 --2- 172.21.15.96:0/1331367452 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc22406d9d0 0x7fc22406fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:15.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:15.444+0000 7fc249c00640 1 --2- 172.21.15.96:0/1331367452 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc22406d9d0 0x7fc22406fe90 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fc238034000 tx=0x7fc23802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:15.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:15.448+0000 7fc241600640 1 -- 172.21.15.96:0/1331367452 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc230059a10 con 0x7fc2440b43b0 2024-05-28T06:17:15.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:15.721+0000 7fc24c600640 1 -- 172.21.15.96:0/1331367452 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fc2440cf910 con 0x7fc2440b43b0 2024-05-28T06:17:15.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:15.722+0000 7fc241600640 1 -- 172.21.15.96:0/1331367452 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+2366 (secure 0 0 0) 0x7fc230059310 con 0x7fc2440b43b0 2024-05-28T06:17:15.722 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:17:15.722 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:17:15.722 INFO:teuthology.orchestra.run.smithi096.stdout: "fsid": "e4a831b5-e6d1-47a7-96cb-bf85154ee0d2", 2024-05-28T06:17:15.722 INFO:teuthology.orchestra.run.smithi096.stdout: "health": { 2024-05-28T06:17:15.722 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "HEALTH_OK", 2024-05-28T06:17:15.722 INFO:teuthology.orchestra.run.smithi096.stdout: "checks": {}, 2024-05-28T06:17:15.722 INFO:teuthology.orchestra.run.smithi096.stdout: "mutes": [] 2024-05-28T06:17:15.722 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:17:15.722 INFO:teuthology.orchestra.run.smithi096.stdout: "election_epoch": 8, 2024-05-28T06:17:15.722 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum": [ 2024-05-28T06:17:15.722 INFO:teuthology.orchestra.run.smithi096.stdout: 0, 2024-05-28T06:17:15.722 INFO:teuthology.orchestra.run.smithi096.stdout: 1, 2024-05-28T06:17:15.723 INFO:teuthology.orchestra.run.smithi096.stdout: 2 2024-05-28T06:17:15.723 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:17:15.723 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_names": [ 2024-05-28T06:17:15.723 INFO:teuthology.orchestra.run.smithi096.stdout: "a", 2024-05-28T06:17:15.723 INFO:teuthology.orchestra.run.smithi096.stdout: "b", 2024-05-28T06:17:15.723 INFO:teuthology.orchestra.run.smithi096.stdout: "c" 2024-05-28T06:17:15.723 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:17:15.723 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_age": 809, 2024-05-28T06:17:15.723 INFO:teuthology.orchestra.run.smithi096.stdout: "monmap": { 2024-05-28T06:17:15.723 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 1, 2024-05-28T06:17:15.723 INFO:teuthology.orchestra.run.smithi096.stdout: "min_mon_release_name": "squid", 2024-05-28T06:17:15.723 INFO:teuthology.orchestra.run.smithi096.stdout: "num_mons": 3 2024-05-28T06:17:15.723 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:17:15.724 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap": { 2024-05-28T06:17:15.724 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 76, 2024-05-28T06:17:15.724 INFO:teuthology.orchestra.run.smithi096.stdout: "num_osds": 8, 2024-05-28T06:17:15.724 INFO:teuthology.orchestra.run.smithi096.stdout: "num_up_osds": 8, 2024-05-28T06:17:15.724 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_up_since": 1716876238, 2024-05-28T06:17:15.724 INFO:teuthology.orchestra.run.smithi096.stdout: "num_in_osds": 8, 2024-05-28T06:17:15.724 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_in_since": 1716876229, 2024-05-28T06:17:15.724 INFO:teuthology.orchestra.run.smithi096.stdout: "num_remapped_pgs": 0 2024-05-28T06:17:15.724 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:17:15.724 INFO:teuthology.orchestra.run.smithi096.stdout: "pgmap": { 2024-05-28T06:17:15.724 INFO:teuthology.orchestra.run.smithi096.stdout: "pgs_by_state": [ 2024-05-28T06:17:15.725 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:17:15.725 INFO:teuthology.orchestra.run.smithi096.stdout: "state_name": "active+clean", 2024-05-28T06:17:15.725 INFO:teuthology.orchestra.run.smithi096.stdout: "count": 97 2024-05-28T06:17:15.725 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:17:15.725 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:17:15.725 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pgs": 97, 2024-05-28T06:17:15.725 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pools": 3, 2024-05-28T06:17:15.725 INFO:teuthology.orchestra.run.smithi096.stdout: "num_objects": 12467, 2024-05-28T06:17:15.725 INFO:teuthology.orchestra.run.smithi096.stdout: "data_bytes": 52089791936, 2024-05-28T06:17:15.725 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_used": 104815976448, 2024-05-28T06:17:15.725 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_avail": 668278136832, 2024-05-28T06:17:15.725 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_total": 773094113280, 2024-05-28T06:17:15.725 INFO:teuthology.orchestra.run.smithi096.stdout: "read_bytes_sec": 255, 2024-05-28T06:17:15.726 INFO:teuthology.orchestra.run.smithi096.stdout: "write_bytes_sec": 7335, 2024-05-28T06:17:15.726 INFO:teuthology.orchestra.run.smithi096.stdout: "read_op_per_sec": 0, 2024-05-28T06:17:15.726 INFO:teuthology.orchestra.run.smithi096.stdout: "write_op_per_sec": 2 2024-05-28T06:17:15.726 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:17:15.726 INFO:teuthology.orchestra.run.smithi096.stdout: "fsmap": { 2024-05-28T06:17:15.726 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 44, 2024-05-28T06:17:15.726 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 5, 2024-05-28T06:17:15.726 INFO:teuthology.orchestra.run.smithi096.stdout: "up": 1, 2024-05-28T06:17:15.726 INFO:teuthology.orchestra.run.smithi096.stdout: "in": 1, 2024-05-28T06:17:15.726 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 1, 2024-05-28T06:17:15.726 INFO:teuthology.orchestra.run.smithi096.stdout: "by_rank": [ 2024-05-28T06:17:15.726 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:17:15.727 INFO:teuthology.orchestra.run.smithi096.stdout: "filesystem_id": 5, 2024-05-28T06:17:15.727 INFO:teuthology.orchestra.run.smithi096.stdout: "rank": 0, 2024-05-28T06:17:15.727 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "d", 2024-05-28T06:17:15.727 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "up:active", 2024-05-28T06:17:15.727 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 7146 2024-05-28T06:17:15.727 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:17:15.727 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:17:15.727 INFO:teuthology.orchestra.run.smithi096.stdout: "up:standby": 3 2024-05-28T06:17:15.727 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:17:15.727 INFO:teuthology.orchestra.run.smithi096.stdout: "mgrmap": { 2024-05-28T06:17:15.727 INFO:teuthology.orchestra.run.smithi096.stdout: "available": true, 2024-05-28T06:17:15.727 INFO:teuthology.orchestra.run.smithi096.stdout: "num_standbys": 1, 2024-05-28T06:17:15.728 INFO:teuthology.orchestra.run.smithi096.stdout: "modules": [ 2024-05-28T06:17:15.728 INFO:teuthology.orchestra.run.smithi096.stdout: "iostat", 2024-05-28T06:17:15.728 INFO:teuthology.orchestra.run.smithi096.stdout: "nfs", 2024-05-28T06:17:15.728 INFO:teuthology.orchestra.run.smithi096.stdout: "restful" 2024-05-28T06:17:15.728 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:17:15.728 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:17:15.728 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:17:15.728 INFO:teuthology.orchestra.run.smithi096.stdout: "servicemap": { 2024-05-28T06:17:15.728 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 13, 2024-05-28T06:17:15.728 INFO:teuthology.orchestra.run.smithi096.stdout: "modified": "2024-05-28T06:15:57.778889+0000", 2024-05-28T06:17:15.728 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:17:15.728 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:17:15.729 INFO:teuthology.orchestra.run.smithi096.stdout: "progress_events": { 2024-05-28T06:17:15.729 INFO:teuthology.orchestra.run.smithi096.stdout: "1f539510-8a8a-46db-8f0c-e233f5cc901d": { 2024-05-28T06:17:15.729 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "All cloning finished, waiting 16.0 seconds before exiting (84s)\n [============================] ", 2024-05-28T06:17:15.729 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": 1, 2024-05-28T06:17:15.729 INFO:teuthology.orchestra.run.smithi096.stdout: "add_to_ceph_s": true 2024-05-28T06:17:15.729 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:17:15.729 INFO:teuthology.orchestra.run.smithi096.stdout: "d1d4f508-f6fa-44f6-a2d0-6e7bd476fb5a": { 2024-05-28T06:17:15.729 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "No pending clones left, waiting 16.0 seconds before exiting (75s)\n [============================] ", 2024-05-28T06:17:15.729 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": 1, 2024-05-28T06:17:15.729 INFO:teuthology.orchestra.run.smithi096.stdout: "add_to_ceph_s": true 2024-05-28T06:17:15.729 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:17:15.729 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:17:15.729 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:17:15.730 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:15.724+0000 7fc24c600640 1 -- 172.21.15.96:0/1331367452 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc22406d9d0 msgr2=0x7fc22406fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:15.730 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:15.724+0000 7fc24c600640 1 --2- 172.21.15.96:0/1331367452 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc22406d9d0 0x7fc22406fe90 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fc238034000 tx=0x7fc23802e000 comp rx=0 tx=0).stop 2024-05-28T06:17:15.730 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:15.724+0000 7fc24c600640 1 -- 172.21.15.96:0/1331367452 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc2440b43b0 msgr2=0x7fc2441e5e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:15.730 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:15.724+0000 7fc24c600640 1 --2- 172.21.15.96:0/1331367452 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc2440b43b0 0x7fc2441e5e20 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7fc23000c410 tx=0x7fc23000c8d0 comp rx=0 tx=0).stop 2024-05-28T06:17:15.731 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:15.724+0000 7fc24c600640 1 -- 172.21.15.96:0/1331367452 shutdown_connections 2024-05-28T06:17:15.731 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:15.724+0000 7fc24c600640 1 --2- 172.21.15.96:0/1331367452 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc22406d9d0 0x7fc22406fe90 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:15.731 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:15.724+0000 7fc24c600640 1 --2- 172.21.15.96:0/1331367452 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc244145320 0x7fc2441445b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:15.731 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:15.724+0000 7fc24c600640 1 --2- 172.21.15.96:0/1331367452 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc2440b43b0 0x7fc2441e5e20 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:15.731 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:15.724+0000 7fc24c600640 1 --2- 172.21.15.96:0/1331367452 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc244072200 0x7fc244142e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:15.731 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:15.724+0000 7fc24c600640 1 -- 172.21.15.96:0/1331367452 >> 172.21.15.96:0/1331367452 conn(0x7fc244067ce0 msgr2=0x7fc24406b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:15.731 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:15.725+0000 7fc24c600640 1 -- 172.21.15.96:0/1331367452 shutdown_connections 2024-05-28T06:17:15.731 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:15.725+0000 7fc24c600640 1 -- 172.21.15.96:0/1331367452 wait complete. 2024-05-28T06:17:20.745 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-05-28T06:17:20.953 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:20.953+0000 7f7f3f800640 1 Processor -- start 2024-05-28T06:17:20.953 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:20.954+0000 7f7f3f800640 1 -- start start 2024-05-28T06:17:20.954 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:20.955+0000 7f7f3f800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f7f3806f180 0x7f7f38163160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:20.954 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:20.955+0000 7f7f3f800640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f3806f5b0 con 0x7f7f38162d80 2024-05-28T06:17:20.954 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:20.955+0000 7f7f3f800640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f381638c0 con 0x7f7f380b4410 2024-05-28T06:17:20.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:20.955+0000 7f7f3f800640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f7f381636a0 con 0x7f7f3806f180 2024-05-28T06:17:20.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:20.955+0000 7f7f3ce00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f7f38162d80 0x7f7f380b3d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:44240/0 (socket says 172.21.15.96:44240) 2024-05-28T06:17:20.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:20.956+0000 7f7f3ce00640 1 -- 172.21.15.96:0/3792435162 learned_addr learned my addr 172.21.15.96:0/3792435162 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:17:20.956 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:20.956+0000 7f7f37e00640 1 -- 172.21.15.96:0/3792435162 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1493471501 0 0) 0x7f7f3806f5b0 con 0x7f7f38162d80 2024-05-28T06:17:20.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:20.956+0000 7f7f37e00640 1 -- 172.21.15.96:0/3792435162 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7f1c003520 con 0x7f7f38162d80 2024-05-28T06:17:20.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:20.956+0000 7f7f3d800640 1 --2- 172.21.15.96:0/3792435162 >> v2:172.21.15.123:3300/0 conn(0x7f7f3806f180 0x7f7f38163160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:20.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:20.956+0000 7f7f37e00640 1 -- 172.21.15.96:0/3792435162 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3895492404 0 0) 0x7f7f1c003520 con 0x7f7f38162d80 2024-05-28T06:17:20.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:20.956+0000 7f7f37e00640 1 -- 172.21.15.96:0/3792435162 >> v2:172.21.15.123:3300/0 conn(0x7f7f3806f180 msgr2=0x7f7f38163160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:20.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:20.956+0000 7f7f37e00640 1 --2- 172.21.15.96:0/3792435162 >> v2:172.21.15.123:3300/0 conn(0x7f7f3806f180 0x7f7f38163160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:20.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:20.956+0000 7f7f37e00640 1 -- 172.21.15.96:0/3792435162 >> v1:172.21.15.123:6789/0 conn(0x7f7f380b4410 legacy=0x7f7f38164960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:20.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:20.956+0000 7f7f37e00640 1 -- 172.21.15.96:0/3792435162 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f38163ba0 con 0x7f7f38162d80 2024-05-28T06:17:20.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:20.956+0000 7f7f37e00640 1 -- 172.21.15.96:0/3792435162 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f7f28001670 con 0x7f7f38162d80 2024-05-28T06:17:20.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:20.956+0000 7f7f37e00640 1 -- 172.21.15.96:0/3792435162 >> v1:172.21.15.96:6789/0 conn(0x7f7f38162d80 legacy=0x7f7f380b3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:20.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:20.956+0000 7f7f37e00640 1 --2- 172.21.15.96:0/3792435162 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f1c0039b0 0x7f7f1c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:20.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:20.956+0000 7f7f37e00640 1 --2- 172.21.15.96:0/3792435162 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7f1c004800 0x7f7f1c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:20.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:20.956+0000 7f7f37e00640 1 --2- 172.21.15.96:0/3792435162 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f1c0051c0 0x7f7f1c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:20.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:20.957+0000 7f7f37e00640 1 -- 172.21.15.96:0/3792435162 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f7f381636a0 con 0x7f7f1c0039b0 2024-05-28T06:17:20.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:20.957+0000 7f7f37e00640 1 -- 172.21.15.96:0/3792435162 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f7f1c00c0f0 con 0x7f7f1c0051c0 2024-05-28T06:17:20.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:20.957+0000 7f7f37e00640 1 -- 172.21.15.96:0/3792435162 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f7f1c00c2f0 con 0x7f7f1c004800 2024-05-28T06:17:20.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:20.957+0000 7f7f3ce00640 1 --2- 172.21.15.96:0/3792435162 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f1c0039b0 0x7f7f1c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:20.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:20.957+0000 7f7f2fe00640 1 --2- 172.21.15.96:0/3792435162 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7f1c004800 0x7f7f1c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:20.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:20.957+0000 7f7f3d800640 1 --2- 172.21.15.96:0/3792435162 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f1c0051c0 0x7f7f1c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:20.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:20.957+0000 7f7f3ce00640 1 -- 172.21.15.96:0/3792435162 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7f1c004800 msgr2=0x7f7f1c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:20.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:20.957+0000 7f7f3ce00640 1 --2- 172.21.15.96:0/3792435162 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7f1c004800 0x7f7f1c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:20.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:20.957+0000 7f7f3ce00640 1 -- 172.21.15.96:0/3792435162 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f1c0051c0 msgr2=0x7f7f1c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:20.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:20.957+0000 7f7f3ce00640 1 --2- 172.21.15.96:0/3792435162 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f1c0051c0 0x7f7f1c00b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:20.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:20.957+0000 7f7f3ce00640 1 -- 172.21.15.96:0/3792435162 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f1c003520 con 0x7f7f1c0039b0 2024-05-28T06:17:20.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:20.957+0000 7f7f3d800640 1 -- 172.21.15.96:0/3792435162 reap_dead start 2024-05-28T06:17:20.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:20.957+0000 7f7f2fe00640 1 --2- 172.21.15.96:0/3792435162 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7f1c004800 0x7f7f1c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:17:20.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:20.958+0000 7f7f3ce00640 1 --2- 172.21.15.96:0/3792435162 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f1c0039b0 0x7f7f1c003dd0 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7f7f2800c240 tx=0x7f7f1c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e3f4f7d03fb47fe server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:20.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:20.961+0000 7f7f37e00640 1 -- 172.21.15.96:0/3792435162 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f28007cf0 con 0x7f7f1c0039b0 2024-05-28T06:17:20.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:20.961+0000 7f7f37e00640 1 -- 172.21.15.96:0/3792435162 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f280058d0 con 0x7f7f1c0039b0 2024-05-28T06:17:20.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:20.961+0000 7f7f37e00640 1 -- 172.21.15.96:0/3792435162 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f28005d20 con 0x7f7f1c0039b0 2024-05-28T06:17:20.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:20.962+0000 7f7f3f800640 1 -- 172.21.15.96:0/3792435162 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f1c0039b0 msgr2=0x7f7f1c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:20.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:20.962+0000 7f7f3f800640 1 --2- 172.21.15.96:0/3792435162 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f1c0039b0 0x7f7f1c003dd0 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7f7f2800c240 tx=0x7f7f1c005830 comp rx=0 tx=0).stop 2024-05-28T06:17:20.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:20.962+0000 7f7f3f800640 1 -- 172.21.15.96:0/3792435162 shutdown_connections 2024-05-28T06:17:20.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:20.962+0000 7f7f3f800640 1 --2- 172.21.15.96:0/3792435162 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f1c0039b0 0x7f7f1c003dd0 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:20.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:20.962+0000 7f7f3f800640 1 -- 172.21.15.96:0/3792435162 >> 172.21.15.96:0/3792435162 conn(0x7f7f38067ce0 msgr2=0x7f7f3806b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:20.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:20.962+0000 7f7f3f800640 1 -- 172.21.15.96:0/3792435162 shutdown_connections 2024-05-28T06:17:20.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:20.962+0000 7f7f3f800640 1 -- 172.21.15.96:0/3792435162 wait complete. 2024-05-28T06:17:20.963 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:20.964+0000 7f7f3f800640 1 Processor -- start 2024-05-28T06:17:20.963 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:20.964+0000 7f7f3f800640 1 -- start start 2024-05-28T06:17:20.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:20.965+0000 7f7f3f800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7f1c0039b0 0x7f7f380d0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:20.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:20.965+0000 7f7f3f800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f380b1040 0x7f7f380b14c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:20.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:20.965+0000 7f7f3f800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f38155680 0x7f7f38155b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:20.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:20.965+0000 7f7f3f800640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f7f38164440 con 0x7f7f38155680 2024-05-28T06:17:20.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:20.965+0000 7f7f3f800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f7f38164160 con 0x7f7f380b1040 2024-05-28T06:17:20.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:20.965+0000 7f7f3f800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f7f38163160 con 0x7f7f1c0039b0 2024-05-28T06:17:20.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:20.965+0000 7f7f3d800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f38155680 0x7f7f38155b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:20.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:20.965+0000 7f7f3d800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f38155680 0x7f7f38155b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:50502/0 (socket says 172.21.15.96:50502) 2024-05-28T06:17:20.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:20.965+0000 7f7f3d800640 1 -- 172.21.15.96:0/327927637 learned_addr learned my addr 172.21.15.96:0/327927637 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:17:20.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:20.965+0000 7f7f3ce00640 1 --2- 172.21.15.96:0/327927637 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7f1c0039b0 0x7f7f380d0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:20.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:20.966+0000 7f7f2fe00640 1 --2- 172.21.15.96:0/327927637 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f380b1040 0x7f7f380b14c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:20.966 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:20.966+0000 7f7f3d800640 1 -- 172.21.15.96:0/327927637 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7f1c0039b0 msgr2=0x7f7f380d0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:20.966 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:20.966+0000 7f7f3d800640 1 --2- 172.21.15.96:0/327927637 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7f1c0039b0 0x7f7f380d0650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:20.966 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:20.966+0000 7f7f3d800640 1 -- 172.21.15.96:0/327927637 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f380b1040 msgr2=0x7f7f380b14c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:20.966 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:20.966+0000 7f7f3d800640 1 --2- 172.21.15.96:0/327927637 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f380b1040 0x7f7f380b14c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:20.966 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:20.966+0000 7f7f3d800640 1 -- 172.21.15.96:0/327927637 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f381586d0 con 0x7f7f38155680 2024-05-28T06:17:20.966 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:20.966+0000 7f7f3d800640 1 --2- 172.21.15.96:0/327927637 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f38155680 0x7f7f38155b00 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f7f380b5870 tx=0x7f7f30009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:20.967 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:20.966+0000 7f7f35600640 1 -- 172.21.15.96:0/327927637 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f30019070 con 0x7f7f38155680 2024-05-28T06:17:20.967 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:20.966+0000 7f7f35600640 1 -- 172.21.15.96:0/327927637 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f30007d50 con 0x7f7f38155680 2024-05-28T06:17:20.967 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:20.966+0000 7f7f35600640 1 -- 172.21.15.96:0/327927637 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f30014640 con 0x7f7f38155680 2024-05-28T06:17:20.967 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:20.966+0000 7f7f3f800640 1 -- 172.21.15.96:0/327927637 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f381589b0 con 0x7f7f38155680 2024-05-28T06:17:20.967 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:20.966+0000 7f7f3f800640 1 -- 172.21.15.96:0/327927637 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f38158f70 con 0x7f7f38155680 2024-05-28T06:17:20.967 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:20.968+0000 7f7f3f800640 1 -- 172.21.15.96:0/327927637 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f38159170 con 0x7f7f38155680 2024-05-28T06:17:20.967 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:20.968+0000 7f7f35600640 1 -- 172.21.15.96:0/327927637 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 28) ==== 92570+0+0 (secure 0 0 0) 0x7f7f3001c070 con 0x7f7f38155680 2024-05-28T06:17:20.968 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:20.969+0000 7f7f35600640 1 --2- 172.21.15.96:0/327927637 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7f1406d9d0 0x7f7f1406fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:20.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:20.970+0000 7f7f35600640 1 -- 172.21.15.96:0/327927637 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5795+0+0 (secure 0 0 0) 0x7f7f3008a6e0 con 0x7f7f38155680 2024-05-28T06:17:20.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:20.970+0000 7f7f3ce00640 1 --2- 172.21.15.96:0/327927637 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7f1406d9d0 0x7f7f1406fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:20.970 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:20.971+0000 7f7f3ce00640 1 --2- 172.21.15.96:0/327927637 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7f1406d9d0 0x7f7f1406fe90 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f7f28033000 tx=0x7f7f28009030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:20.973 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:20.974+0000 7f7f35600640 1 -- 172.21.15.96:0/327927637 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7f38159170 con 0x7f7f38155680 2024-05-28T06:17:21.250 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:21.251+0000 7f7f3f800640 1 -- 172.21.15.96:0/327927637 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f7f380689e0 con 0x7f7f38155680 2024-05-28T06:17:21.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:21.252+0000 7f7f35600640 1 -- 172.21.15.96:0/327927637 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+2335 (secure 0 0 0) 0x7f7f3005a220 con 0x7f7f38155680 2024-05-28T06:17:21.252 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:17:21.252 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:17:21.252 INFO:teuthology.orchestra.run.smithi096.stdout: "fsid": "e4a831b5-e6d1-47a7-96cb-bf85154ee0d2", 2024-05-28T06:17:21.252 INFO:teuthology.orchestra.run.smithi096.stdout: "health": { 2024-05-28T06:17:21.252 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "HEALTH_OK", 2024-05-28T06:17:21.253 INFO:teuthology.orchestra.run.smithi096.stdout: "checks": {}, 2024-05-28T06:17:21.253 INFO:teuthology.orchestra.run.smithi096.stdout: "mutes": [] 2024-05-28T06:17:21.253 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:17:21.253 INFO:teuthology.orchestra.run.smithi096.stdout: "election_epoch": 8, 2024-05-28T06:17:21.253 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum": [ 2024-05-28T06:17:21.253 INFO:teuthology.orchestra.run.smithi096.stdout: 0, 2024-05-28T06:17:21.253 INFO:teuthology.orchestra.run.smithi096.stdout: 1, 2024-05-28T06:17:21.253 INFO:teuthology.orchestra.run.smithi096.stdout: 2 2024-05-28T06:17:21.253 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:17:21.253 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_names": [ 2024-05-28T06:17:21.253 INFO:teuthology.orchestra.run.smithi096.stdout: "a", 2024-05-28T06:17:21.254 INFO:teuthology.orchestra.run.smithi096.stdout: "b", 2024-05-28T06:17:21.254 INFO:teuthology.orchestra.run.smithi096.stdout: "c" 2024-05-28T06:17:21.254 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:17:21.254 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_age": 814, 2024-05-28T06:17:21.254 INFO:teuthology.orchestra.run.smithi096.stdout: "monmap": { 2024-05-28T06:17:21.254 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 1, 2024-05-28T06:17:21.254 INFO:teuthology.orchestra.run.smithi096.stdout: "min_mon_release_name": "squid", 2024-05-28T06:17:21.254 INFO:teuthology.orchestra.run.smithi096.stdout: "num_mons": 3 2024-05-28T06:17:21.254 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:17:21.254 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap": { 2024-05-28T06:17:21.254 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 76, 2024-05-28T06:17:21.254 INFO:teuthology.orchestra.run.smithi096.stdout: "num_osds": 8, 2024-05-28T06:17:21.255 INFO:teuthology.orchestra.run.smithi096.stdout: "num_up_osds": 8, 2024-05-28T06:17:21.255 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_up_since": 1716876238, 2024-05-28T06:17:21.255 INFO:teuthology.orchestra.run.smithi096.stdout: "num_in_osds": 8, 2024-05-28T06:17:21.255 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_in_since": 1716876229, 2024-05-28T06:17:21.255 INFO:teuthology.orchestra.run.smithi096.stdout: "num_remapped_pgs": 0 2024-05-28T06:17:21.255 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:17:21.255 INFO:teuthology.orchestra.run.smithi096.stdout: "pgmap": { 2024-05-28T06:17:21.255 INFO:teuthology.orchestra.run.smithi096.stdout: "pgs_by_state": [ 2024-05-28T06:17:21.255 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:17:21.255 INFO:teuthology.orchestra.run.smithi096.stdout: "state_name": "active+clean", 2024-05-28T06:17:21.255 INFO:teuthology.orchestra.run.smithi096.stdout: "count": 97 2024-05-28T06:17:21.255 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:17:21.255 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:17:21.255 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pgs": 97, 2024-05-28T06:17:21.256 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pools": 3, 2024-05-28T06:17:21.256 INFO:teuthology.orchestra.run.smithi096.stdout: "num_objects": 12467, 2024-05-28T06:17:21.256 INFO:teuthology.orchestra.run.smithi096.stdout: "data_bytes": 52089808861, 2024-05-28T06:17:21.256 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_used": 104816017408, 2024-05-28T06:17:21.256 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_avail": 668278095872, 2024-05-28T06:17:21.256 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_total": 773094113280, 2024-05-28T06:17:21.256 INFO:teuthology.orchestra.run.smithi096.stdout: "write_bytes_sec": 4350, 2024-05-28T06:17:21.256 INFO:teuthology.orchestra.run.smithi096.stdout: "read_op_per_sec": 0, 2024-05-28T06:17:21.256 INFO:teuthology.orchestra.run.smithi096.stdout: "write_op_per_sec": 0 2024-05-28T06:17:21.256 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:17:21.256 INFO:teuthology.orchestra.run.smithi096.stdout: "fsmap": { 2024-05-28T06:17:21.256 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 44, 2024-05-28T06:17:21.257 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 5, 2024-05-28T06:17:21.257 INFO:teuthology.orchestra.run.smithi096.stdout: "up": 1, 2024-05-28T06:17:21.257 INFO:teuthology.orchestra.run.smithi096.stdout: "in": 1, 2024-05-28T06:17:21.257 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 1, 2024-05-28T06:17:21.257 INFO:teuthology.orchestra.run.smithi096.stdout: "by_rank": [ 2024-05-28T06:17:21.257 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:17:21.257 INFO:teuthology.orchestra.run.smithi096.stdout: "filesystem_id": 5, 2024-05-28T06:17:21.257 INFO:teuthology.orchestra.run.smithi096.stdout: "rank": 0, 2024-05-28T06:17:21.257 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "d", 2024-05-28T06:17:21.257 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "up:active", 2024-05-28T06:17:21.257 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 7146 2024-05-28T06:17:21.257 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:17:21.257 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:17:21.257 INFO:teuthology.orchestra.run.smithi096.stdout: "up:standby": 3 2024-05-28T06:17:21.258 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:17:21.258 INFO:teuthology.orchestra.run.smithi096.stdout: "mgrmap": { 2024-05-28T06:17:21.258 INFO:teuthology.orchestra.run.smithi096.stdout: "available": true, 2024-05-28T06:17:21.258 INFO:teuthology.orchestra.run.smithi096.stdout: "num_standbys": 1, 2024-05-28T06:17:21.258 INFO:teuthology.orchestra.run.smithi096.stdout: "modules": [ 2024-05-28T06:17:21.258 INFO:teuthology.orchestra.run.smithi096.stdout: "iostat", 2024-05-28T06:17:21.258 INFO:teuthology.orchestra.run.smithi096.stdout: "nfs", 2024-05-28T06:17:21.258 INFO:teuthology.orchestra.run.smithi096.stdout: "restful" 2024-05-28T06:17:21.258 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:17:21.258 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:17:21.258 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:17:21.258 INFO:teuthology.orchestra.run.smithi096.stdout: "servicemap": { 2024-05-28T06:17:21.258 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 13, 2024-05-28T06:17:21.259 INFO:teuthology.orchestra.run.smithi096.stdout: "modified": "2024-05-28T06:15:57.778889+0000", 2024-05-28T06:17:21.259 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:17:21.259 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:17:21.259 INFO:teuthology.orchestra.run.smithi096.stdout: "progress_events": { 2024-05-28T06:17:21.259 INFO:teuthology.orchestra.run.smithi096.stdout: "1f539510-8a8a-46db-8f0c-e233f5cc901d": { 2024-05-28T06:17:21.259 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "All cloning finished, waiting 10.0 seconds before exiting (90s)\n [============================] ", 2024-05-28T06:17:21.259 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": 1, 2024-05-28T06:17:21.259 INFO:teuthology.orchestra.run.smithi096.stdout: "add_to_ceph_s": true 2024-05-28T06:17:21.259 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:17:21.259 INFO:teuthology.orchestra.run.smithi096.stdout: "d1d4f508-f6fa-44f6-a2d0-6e7bd476fb5a": { 2024-05-28T06:17:21.259 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "No pending clones left, waiting 10.0 seconds before exiting (81s)\n [============================] ", 2024-05-28T06:17:21.259 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": 1, 2024-05-28T06:17:21.259 INFO:teuthology.orchestra.run.smithi096.stdout: "add_to_ceph_s": true 2024-05-28T06:17:21.259 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:17:21.260 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:17:21.260 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:17:21.260 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:21.255+0000 7f7f3f800640 1 -- 172.21.15.96:0/327927637 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7f1406d9d0 msgr2=0x7f7f1406fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:21.260 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:21.255+0000 7f7f3f800640 1 --2- 172.21.15.96:0/327927637 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7f1406d9d0 0x7f7f1406fe90 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f7f28033000 tx=0x7f7f28009030 comp rx=0 tx=0).stop 2024-05-28T06:17:21.260 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:21.255+0000 7f7f3f800640 1 -- 172.21.15.96:0/327927637 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f38155680 msgr2=0x7f7f38155b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:21.261 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:21.255+0000 7f7f3f800640 1 --2- 172.21.15.96:0/327927637 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f38155680 0x7f7f38155b00 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f7f380b5870 tx=0x7f7f30009da0 comp rx=0 tx=0).stop 2024-05-28T06:17:21.261 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:21.255+0000 7f7f3f800640 1 -- 172.21.15.96:0/327927637 shutdown_connections 2024-05-28T06:17:21.261 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:21.255+0000 7f7f3f800640 1 --2- 172.21.15.96:0/327927637 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7f1406d9d0 0x7f7f1406fe90 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:21.261 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:21.255+0000 7f7f3f800640 1 --2- 172.21.15.96:0/327927637 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f38155680 0x7f7f38155b00 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:21.261 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:21.255+0000 7f7f3f800640 1 --2- 172.21.15.96:0/327927637 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f380b1040 0x7f7f380b14c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:21.261 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:21.255+0000 7f7f3f800640 1 --2- 172.21.15.96:0/327927637 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7f1c0039b0 0x7f7f380d0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:21.261 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:21.255+0000 7f7f3f800640 1 -- 172.21.15.96:0/327927637 >> 172.21.15.96:0/327927637 conn(0x7f7f38067ce0 msgr2=0x7f7f38160960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:21.261 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:21.255+0000 7f7f3f800640 1 -- 172.21.15.96:0/327927637 shutdown_connections 2024-05-28T06:17:21.261 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:21.255+0000 7f7f3f800640 1 -- 172.21.15.96:0/327927637 wait complete. 2024-05-28T06:17:26.273 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-05-28T06:17:26.481 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:26.481+0000 7f5e7de00640 1 Processor -- start 2024-05-28T06:17:26.481 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:26.481+0000 7f5e7de00640 1 -- start start 2024-05-28T06:17:26.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:26.483+0000 7f5e7de00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5e7806b900 0x7f5e7806bce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:26.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:26.483+0000 7f5e7de00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f5e7815ed60 0x7f5e7806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:26.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:26.483+0000 7f5e7de00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f5e780cba20 0x7f5e7806f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:26.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:26.483+0000 7f5e7de00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f5e780d05f0 con 0x7f5e780cba20 2024-05-28T06:17:26.483 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:26.483+0000 7f5e7de00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f5e7806c220 con 0x7f5e7815ed60 2024-05-28T06:17:26.483 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:26.483+0000 7f5e7de00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f5e780d0890 con 0x7f5e7806b900 2024-05-28T06:17:26.483 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:26.483+0000 7f5e77400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f5e780cba20 0x7f5e7806f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:26.483 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:26.483+0000 7f5e77400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f5e780cba20 0x7f5e7806f620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:50512/0 (socket says 172.21.15.96:50512) 2024-05-28T06:17:26.483 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:26.483+0000 7f5e77400640 1 -- 172.21.15.96:0/830041562 learned_addr learned my addr 172.21.15.96:0/830041562 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:17:26.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:26.484+0000 7f5e76a00640 1 --2- 172.21.15.96:0/830041562 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5e7806b900 0x7f5e7806bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:26.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:26.484+0000 7f5e76000640 1 --2- 172.21.15.96:0/830041562 >> v2:172.21.15.123:3300/0 conn(0x7f5e7815ed60 0x7f5e7806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:26.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:26.484+0000 7f5e77400640 1 -- 172.21.15.96:0/830041562 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5e7806b900 msgr2=0x7f5e7806bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:26.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:26.484+0000 7f5e77400640 1 --2- 172.21.15.96:0/830041562 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5e7806b900 0x7f5e7806bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:26.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:26.484+0000 7f5e77400640 1 -- 172.21.15.96:0/830041562 >> v2:172.21.15.123:3300/0 conn(0x7f5e7815ed60 msgr2=0x7f5e7806f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:26.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:26.484+0000 7f5e77400640 1 --2- 172.21.15.96:0/830041562 >> v2:172.21.15.123:3300/0 conn(0x7f5e7815ed60 0x7f5e7806f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:26.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:26.484+0000 7f5e77400640 1 -- 172.21.15.96:0/830041562 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e780d0b70 con 0x7f5e780cba20 2024-05-28T06:17:26.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:26.484+0000 7f5e76a00640 1 --2- 172.21.15.96:0/830041562 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5e7806b900 0x7f5e7806bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:17:26.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:26.484+0000 7f5e76000640 1 --2- 172.21.15.96:0/830041562 >> v2:172.21.15.123:3300/0 conn(0x7f5e7815ed60 0x7f5e7806f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:17:26.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:26.485+0000 7f5e77400640 1 --2- 172.21.15.96:0/830041562 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5e780cba20 0x7f5e7806f620 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f5e780717d0 tx=0x7f5e7002a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb82cc0e223d3539 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:26.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:26.485+0000 7f5e75600640 1 -- 172.21.15.96:0/830041562 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e70037070 con 0x7f5e780cba20 2024-05-28T06:17:26.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:26.485+0000 7f5e75600640 1 -- 172.21.15.96:0/830041562 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e70032440 con 0x7f5e780cba20 2024-05-28T06:17:26.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:26.485+0000 7f5e75600640 1 -- 172.21.15.96:0/830041562 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e70032810 con 0x7f5e780cba20 2024-05-28T06:17:26.486 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:26.486+0000 7f5e7de00640 1 -- 172.21.15.96:0/830041562 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5e780cba20 msgr2=0x7f5e7806f620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:26.486 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:26.486+0000 7f5e7de00640 1 --2- 172.21.15.96:0/830041562 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5e780cba20 0x7f5e7806f620 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f5e780717d0 tx=0x7f5e7002a140 comp rx=0 tx=0).stop 2024-05-28T06:17:26.486 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:26.486+0000 7f5e7de00640 1 -- 172.21.15.96:0/830041562 shutdown_connections 2024-05-28T06:17:26.486 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:26.486+0000 7f5e7de00640 1 --2- 172.21.15.96:0/830041562 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5e780cba20 0x7f5e7806f620 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:26.486 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:26.486+0000 7f5e7de00640 1 --2- 172.21.15.96:0/830041562 >> v2:172.21.15.123:3300/0 conn(0x7f5e7815ed60 0x7f5e7806f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:26.486 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:26.486+0000 7f5e7de00640 1 --2- 172.21.15.96:0/830041562 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5e7806b900 0x7f5e7806bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:26.486 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:26.486+0000 7f5e7de00640 1 -- 172.21.15.96:0/830041562 >> 172.21.15.96:0/830041562 conn(0x7f5e7809aa30 msgr2=0x7f5e780b2f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:26.486 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:26.486+0000 7f5e7de00640 1 -- 172.21.15.96:0/830041562 shutdown_connections 2024-05-28T06:17:26.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:26.486+0000 7f5e7de00640 1 -- 172.21.15.96:0/830041562 wait complete. 2024-05-28T06:17:26.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:26.488+0000 7f5e7de00640 1 Processor -- start 2024-05-28T06:17:26.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:26.488+0000 7f5e7de00640 1 -- start start 2024-05-28T06:17:26.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:26.489+0000 7f5e7de00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5e7806b900 0x7f5e7815d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:26.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:26.489+0000 7f5e7de00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5e780cba20 0x7f5e780b4030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:26.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:26.489+0000 7f5e7de00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e7815ed60 0x7f5e7814a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:26.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:26.489+0000 7f5e7de00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f5e780d1130 con 0x7f5e7806b900 2024-05-28T06:17:26.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:26.489+0000 7f5e7de00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5e780b3810 con 0x7f5e7815ed60 2024-05-28T06:17:26.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:26.489+0000 7f5e7de00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f5e780cd5f0 con 0x7f5e780cba20 2024-05-28T06:17:26.490 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:26.489+0000 7f5e76a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5e7806b900 0x7f5e7815d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:26.490 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:26.490+0000 7f5e76a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5e7806b900 0x7f5e7815d220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:50518/0 (socket says 172.21.15.96:50518) 2024-05-28T06:17:26.490 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:26.490+0000 7f5e76a00640 1 -- 172.21.15.96:0/3438210220 learned_addr learned my addr 172.21.15.96:0/3438210220 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:17:26.490 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:26.490+0000 7f5e76000640 1 --2- 172.21.15.96:0/3438210220 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5e780cba20 0x7f5e780b4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:26.490 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:26.490+0000 7f5e77400640 1 --2- 172.21.15.96:0/3438210220 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e7815ed60 0x7f5e7814a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:26.490 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:26.490+0000 7f5e76a00640 1 -- 172.21.15.96:0/3438210220 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5e780cba20 msgr2=0x7f5e780b4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:26.490 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:26.490+0000 7f5e76a00640 1 --2- 172.21.15.96:0/3438210220 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5e780cba20 0x7f5e780b4030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:26.490 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:26.490+0000 7f5e76a00640 1 -- 172.21.15.96:0/3438210220 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e7815ed60 msgr2=0x7f5e7814a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:26.490 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:26.490+0000 7f5e76a00640 1 --2- 172.21.15.96:0/3438210220 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e7815ed60 0x7f5e7814a010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:26.490 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:26.490+0000 7f5e76a00640 1 -- 172.21.15.96:0/3438210220 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e780c9a20 con 0x7f5e7806b900 2024-05-28T06:17:26.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:26.490+0000 7f5e76a00640 1 --2- 172.21.15.96:0/3438210220 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5e7806b900 0x7f5e7815d220 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f5e6800cf10 tx=0x7f5e6800a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:26.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:26.491+0000 7f5e6f400640 1 -- 172.21.15.96:0/3438210220 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e6801a070 con 0x7f5e7806b900 2024-05-28T06:17:26.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:26.491+0000 7f5e6f400640 1 -- 172.21.15.96:0/3438210220 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e6800d840 con 0x7f5e7806b900 2024-05-28T06:17:26.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:26.491+0000 7f5e6f400640 1 -- 172.21.15.96:0/3438210220 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e6800dc10 con 0x7f5e7806b900 2024-05-28T06:17:26.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:26.491+0000 7f5e7de00640 1 -- 172.21.15.96:0/3438210220 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e780c9d00 con 0x7f5e7806b900 2024-05-28T06:17:26.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:26.491+0000 7f5e7de00640 1 -- 172.21.15.96:0/3438210220 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e780ca2c0 con 0x7f5e7806b900 2024-05-28T06:17:26.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:26.492+0000 7f5e7de00640 1 -- 172.21.15.96:0/3438210220 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e780689e0 con 0x7f5e7806b900 2024-05-28T06:17:26.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:26.493+0000 7f5e6f400640 1 -- 172.21.15.96:0/3438210220 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 28) ==== 92570+0+0 (secure 0 0 0) 0x7f5e6801d070 con 0x7f5e7806b900 2024-05-28T06:17:26.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:26.494+0000 7f5e6f400640 1 --2- 172.21.15.96:0/3438210220 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5e5406da00 0x7f5e5406fec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:26.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:26.494+0000 7f5e6f400640 1 -- 172.21.15.96:0/3438210220 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5795+0+0 (secure 0 0 0) 0x7f5e6808b1e0 con 0x7f5e7806b900 2024-05-28T06:17:26.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:26.495+0000 7f5e76000640 1 --2- 172.21.15.96:0/3438210220 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5e5406da00 0x7f5e5406fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:26.495 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:26.496+0000 7f5e76000640 1 --2- 172.21.15.96:0/3438210220 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5e5406da00 0x7f5e5406fec0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f5e780b7150 tx=0x7f5e60008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:26.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:26.499+0000 7f5e6f400640 1 -- 172.21.15.96:0/3438210220 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5e6805a700 con 0x7f5e7806b900 2024-05-28T06:17:26.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:26.770+0000 7f5e7de00640 1 -- 172.21.15.96:0/3438210220 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f5e780b3570 con 0x7f5e7806b900 2024-05-28T06:17:26.770 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:26.771+0000 7f5e6f400640 1 -- 172.21.15.96:0/3438210220 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+2333 (secure 0 0 0) 0x7f5e6805ea00 con 0x7f5e7806b900 2024-05-28T06:17:26.770 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:17:26.770 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:17:26.770 INFO:teuthology.orchestra.run.smithi096.stdout: "fsid": "e4a831b5-e6d1-47a7-96cb-bf85154ee0d2", 2024-05-28T06:17:26.770 INFO:teuthology.orchestra.run.smithi096.stdout: "health": { 2024-05-28T06:17:26.771 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "HEALTH_OK", 2024-05-28T06:17:26.771 INFO:teuthology.orchestra.run.smithi096.stdout: "checks": {}, 2024-05-28T06:17:26.771 INFO:teuthology.orchestra.run.smithi096.stdout: "mutes": [] 2024-05-28T06:17:26.771 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:17:26.771 INFO:teuthology.orchestra.run.smithi096.stdout: "election_epoch": 8, 2024-05-28T06:17:26.771 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum": [ 2024-05-28T06:17:26.771 INFO:teuthology.orchestra.run.smithi096.stdout: 0, 2024-05-28T06:17:26.771 INFO:teuthology.orchestra.run.smithi096.stdout: 1, 2024-05-28T06:17:26.771 INFO:teuthology.orchestra.run.smithi096.stdout: 2 2024-05-28T06:17:26.771 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:17:26.771 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_names": [ 2024-05-28T06:17:26.771 INFO:teuthology.orchestra.run.smithi096.stdout: "a", 2024-05-28T06:17:26.771 INFO:teuthology.orchestra.run.smithi096.stdout: "b", 2024-05-28T06:17:26.771 INFO:teuthology.orchestra.run.smithi096.stdout: "c" 2024-05-28T06:17:26.771 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:17:26.772 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_age": 820, 2024-05-28T06:17:26.772 INFO:teuthology.orchestra.run.smithi096.stdout: "monmap": { 2024-05-28T06:17:26.772 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 1, 2024-05-28T06:17:26.772 INFO:teuthology.orchestra.run.smithi096.stdout: "min_mon_release_name": "squid", 2024-05-28T06:17:26.772 INFO:teuthology.orchestra.run.smithi096.stdout: "num_mons": 3 2024-05-28T06:17:26.772 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:17:26.772 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap": { 2024-05-28T06:17:26.772 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 76, 2024-05-28T06:17:26.772 INFO:teuthology.orchestra.run.smithi096.stdout: "num_osds": 8, 2024-05-28T06:17:26.772 INFO:teuthology.orchestra.run.smithi096.stdout: "num_up_osds": 8, 2024-05-28T06:17:26.772 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_up_since": 1716876238, 2024-05-28T06:17:26.772 INFO:teuthology.orchestra.run.smithi096.stdout: "num_in_osds": 8, 2024-05-28T06:17:26.772 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_in_since": 1716876229, 2024-05-28T06:17:26.772 INFO:teuthology.orchestra.run.smithi096.stdout: "num_remapped_pgs": 0 2024-05-28T06:17:26.773 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:17:26.773 INFO:teuthology.orchestra.run.smithi096.stdout: "pgmap": { 2024-05-28T06:17:26.773 INFO:teuthology.orchestra.run.smithi096.stdout: "pgs_by_state": [ 2024-05-28T06:17:26.773 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:17:26.773 INFO:teuthology.orchestra.run.smithi096.stdout: "state_name": "active+clean", 2024-05-28T06:17:26.773 INFO:teuthology.orchestra.run.smithi096.stdout: "count": 97 2024-05-28T06:17:26.773 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:17:26.773 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:17:26.773 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pgs": 97, 2024-05-28T06:17:26.773 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pools": 3, 2024-05-28T06:17:26.773 INFO:teuthology.orchestra.run.smithi096.stdout: "num_objects": 12467, 2024-05-28T06:17:26.773 INFO:teuthology.orchestra.run.smithi096.stdout: "data_bytes": 52089816945, 2024-05-28T06:17:26.773 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_used": 104816033792, 2024-05-28T06:17:26.773 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_avail": 668278079488, 2024-05-28T06:17:26.774 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_total": 773094113280, 2024-05-28T06:17:26.774 INFO:teuthology.orchestra.run.smithi096.stdout: "write_bytes_sec": 2558, 2024-05-28T06:17:26.774 INFO:teuthology.orchestra.run.smithi096.stdout: "read_op_per_sec": 0, 2024-05-28T06:17:26.774 INFO:teuthology.orchestra.run.smithi096.stdout: "write_op_per_sec": 0 2024-05-28T06:17:26.774 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:17:26.774 INFO:teuthology.orchestra.run.smithi096.stdout: "fsmap": { 2024-05-28T06:17:26.774 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 44, 2024-05-28T06:17:26.774 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 5, 2024-05-28T06:17:26.774 INFO:teuthology.orchestra.run.smithi096.stdout: "up": 1, 2024-05-28T06:17:26.774 INFO:teuthology.orchestra.run.smithi096.stdout: "in": 1, 2024-05-28T06:17:26.774 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 1, 2024-05-28T06:17:26.774 INFO:teuthology.orchestra.run.smithi096.stdout: "by_rank": [ 2024-05-28T06:17:26.774 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:17:26.775 INFO:teuthology.orchestra.run.smithi096.stdout: "filesystem_id": 5, 2024-05-28T06:17:26.775 INFO:teuthology.orchestra.run.smithi096.stdout: "rank": 0, 2024-05-28T06:17:26.775 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "d", 2024-05-28T06:17:26.775 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "up:active", 2024-05-28T06:17:26.775 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 7146 2024-05-28T06:17:26.775 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:17:26.775 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:17:26.775 INFO:teuthology.orchestra.run.smithi096.stdout: "up:standby": 3 2024-05-28T06:17:26.775 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:17:26.775 INFO:teuthology.orchestra.run.smithi096.stdout: "mgrmap": { 2024-05-28T06:17:26.775 INFO:teuthology.orchestra.run.smithi096.stdout: "available": true, 2024-05-28T06:17:26.775 INFO:teuthology.orchestra.run.smithi096.stdout: "num_standbys": 1, 2024-05-28T06:17:26.775 INFO:teuthology.orchestra.run.smithi096.stdout: "modules": [ 2024-05-28T06:17:26.775 INFO:teuthology.orchestra.run.smithi096.stdout: "iostat", 2024-05-28T06:17:26.776 INFO:teuthology.orchestra.run.smithi096.stdout: "nfs", 2024-05-28T06:17:26.776 INFO:teuthology.orchestra.run.smithi096.stdout: "restful" 2024-05-28T06:17:26.776 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:17:26.776 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:17:26.776 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:17:26.776 INFO:teuthology.orchestra.run.smithi096.stdout: "servicemap": { 2024-05-28T06:17:26.776 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 13, 2024-05-28T06:17:26.776 INFO:teuthology.orchestra.run.smithi096.stdout: "modified": "2024-05-28T06:15:57.778889+0000", 2024-05-28T06:17:26.776 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:17:26.776 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:17:26.776 INFO:teuthology.orchestra.run.smithi096.stdout: "progress_events": { 2024-05-28T06:17:26.776 INFO:teuthology.orchestra.run.smithi096.stdout: "1f539510-8a8a-46db-8f0c-e233f5cc901d": { 2024-05-28T06:17:26.776 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "All cloning finished, waiting 4.0 seconds before exiting (96s)\n [============================] ", 2024-05-28T06:17:26.776 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": 1, 2024-05-28T06:17:26.776 INFO:teuthology.orchestra.run.smithi096.stdout: "add_to_ceph_s": true 2024-05-28T06:17:26.777 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:17:26.777 INFO:teuthology.orchestra.run.smithi096.stdout: "d1d4f508-f6fa-44f6-a2d0-6e7bd476fb5a": { 2024-05-28T06:17:26.777 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "No pending clones left, waiting 4.0 seconds before exiting (87s)\n [============================] ", 2024-05-28T06:17:26.777 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": 1, 2024-05-28T06:17:26.777 INFO:teuthology.orchestra.run.smithi096.stdout: "add_to_ceph_s": true 2024-05-28T06:17:26.777 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:17:26.777 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:17:26.777 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:17:26.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:26.774+0000 7f5e7de00640 1 -- 172.21.15.96:0/3438210220 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5e5406da00 msgr2=0x7f5e5406fec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:26.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:26.774+0000 7f5e7de00640 1 --2- 172.21.15.96:0/3438210220 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5e5406da00 0x7f5e5406fec0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f5e780b7150 tx=0x7f5e60008040 comp rx=0 tx=0).stop 2024-05-28T06:17:26.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:26.774+0000 7f5e7de00640 1 -- 172.21.15.96:0/3438210220 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5e7806b900 msgr2=0x7f5e7815d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:26.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:26.774+0000 7f5e7de00640 1 --2- 172.21.15.96:0/3438210220 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5e7806b900 0x7f5e7815d220 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f5e6800cf10 tx=0x7f5e6800a710 comp rx=0 tx=0).stop 2024-05-28T06:17:26.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:26.774+0000 7f5e7de00640 1 -- 172.21.15.96:0/3438210220 shutdown_connections 2024-05-28T06:17:26.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:26.774+0000 7f5e7de00640 1 --2- 172.21.15.96:0/3438210220 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5e5406da00 0x7f5e5406fec0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:26.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:26.774+0000 7f5e7de00640 1 --2- 172.21.15.96:0/3438210220 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e7815ed60 0x7f5e7814a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:26.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:26.774+0000 7f5e7de00640 1 --2- 172.21.15.96:0/3438210220 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5e780cba20 0x7f5e780b4030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:26.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:26.774+0000 7f5e7de00640 1 --2- 172.21.15.96:0/3438210220 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5e7806b900 0x7f5e7815d220 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:26.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:26.774+0000 7f5e7de00640 1 -- 172.21.15.96:0/3438210220 >> 172.21.15.96:0/3438210220 conn(0x7f5e7809aa30 msgr2=0x7f5e780708b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:26.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:26.774+0000 7f5e7de00640 1 -- 172.21.15.96:0/3438210220 shutdown_connections 2024-05-28T06:17:26.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:26.774+0000 7f5e7de00640 1 -- 172.21.15.96:0/3438210220 wait complete. 2024-05-28T06:17:31.792 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-05-28T06:17:31.995 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:31.997+0000 7fa6ffa00640 1 Processor -- start 2024-05-28T06:17:31.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:31.998+0000 7fa6ffa00640 1 -- start start 2024-05-28T06:17:31.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:31.999+0000 7fa6ffa00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa6f815ea40 0x7fa6f80d4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:31.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:31.999+0000 7fa6ffa00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6f80cf7e0 con 0x7fa6f80d1040 2024-05-28T06:17:31.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:31.999+0000 7fa6ffa00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6f815fe50 con 0x7fa6f806c950 2024-05-28T06:17:31.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:31.999+0000 7fa6ffa00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fa6f815ee20 con 0x7fa6f815ea40 2024-05-28T06:17:31.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:31.999+0000 7fa6f7e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fa6f80d1040 0x7fa6f80d1460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:57724/0 (socket says 172.21.15.96:57724) 2024-05-28T06:17:31.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:31.999+0000 7fa6f7e00640 1 -- 172.21.15.96:0/1445414690 learned_addr learned my addr 172.21.15.96:0/1445414690 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:17:31.999 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.000+0000 7fa6fd000640 1 --2- 172.21.15.96:0/1445414690 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa6f815ea40 0x7fa6f80d4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:31.999 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.000+0000 7fa6f7400640 1 -- 172.21.15.96:0/1445414690 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1067254517 0 0) 0x7fa6f80cf7e0 con 0x7fa6f80d1040 2024-05-28T06:17:31.999 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.000+0000 7fa6f7400640 1 -- 172.21.15.96:0/1445414690 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa6d8003520 con 0x7fa6f80d1040 2024-05-28T06:17:31.999 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.000+0000 7fa6f7400640 1 -- 172.21.15.96:0/1445414690 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2256148689 0 0) 0x7fa6d8003520 con 0x7fa6f80d1040 2024-05-28T06:17:31.999 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.000+0000 7fa6f7400640 1 -- 172.21.15.96:0/1445414690 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa6f815ea40 msgr2=0x7fa6f80d4f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:32.000 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.000+0000 7fa6f7400640 1 --2- 172.21.15.96:0/1445414690 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa6f815ea40 0x7fa6f80d4f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:32.000 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.000+0000 7fa6f7400640 1 -- 172.21.15.96:0/1445414690 >> v1:172.21.15.123:6789/0 conn(0x7fa6f806c950 legacy=0x7fa6f806cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:32.000 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.000+0000 7fa6f7400640 1 -- 172.21.15.96:0/1445414690 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6f80cfa80 con 0x7fa6f80d1040 2024-05-28T06:17:32.000 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.000+0000 7fa6f7400640 1 -- 172.21.15.96:0/1445414690 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7fa6e00019f0 con 0x7fa6f80d1040 2024-05-28T06:17:32.000 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.000+0000 7fa6f7400640 1 -- 172.21.15.96:0/1445414690 >> v1:172.21.15.96:6789/0 conn(0x7fa6f80d1040 legacy=0x7fa6f80d1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:32.000 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.000+0000 7fa6f7400640 1 --2- 172.21.15.96:0/1445414690 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6d80039e0 0x7fa6d8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:32.000 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.000+0000 7fa6f7400640 1 --2- 172.21.15.96:0/1445414690 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6d8004800 0x7fa6d8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:32.000 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.000+0000 7fa6f7400640 1 --2- 172.21.15.96:0/1445414690 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa6f815ea40 0x7fa6d8005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:32.000 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.000+0000 7fa6f7400640 1 -- 172.21.15.96:0/1445414690 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa6f815ee20 con 0x7fa6d80039e0 2024-05-28T06:17:32.000 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.000+0000 7fa6f7400640 1 -- 172.21.15.96:0/1445414690 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa6d800baf0 con 0x7fa6d8004800 2024-05-28T06:17:32.000 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.000+0000 7fa6f7400640 1 -- 172.21.15.96:0/1445414690 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fa6d800bcf0 con 0x7fa6f815ea40 2024-05-28T06:17:32.000 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.001+0000 7fa6fd000640 1 --2- 172.21.15.96:0/1445414690 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6d80039e0 0x7fa6d8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:32.000 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.001+0000 7fa6fd000640 1 -- 172.21.15.96:0/1445414690 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa6f815ea40 msgr2=0x7fa6d8005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:32.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.001+0000 7fa6fd000640 1 --2- 172.21.15.96:0/1445414690 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa6f815ea40 0x7fa6d8005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:32.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.001+0000 7fa6fd000640 1 -- 172.21.15.96:0/1445414690 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6d8004800 msgr2=0x7fa6d8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:32.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.001+0000 7fa6fd000640 1 --2- 172.21.15.96:0/1445414690 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6d8004800 0x7fa6d8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:32.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.001+0000 7fa6fd000640 1 -- 172.21.15.96:0/1445414690 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6d8003520 con 0x7fa6d80039e0 2024-05-28T06:17:32.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.002+0000 7fa6fd000640 1 --2- 172.21.15.96:0/1445414690 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6d80039e0 0x7fa6d8003e00 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7fa6e8009ac0 tx=0x7fa6d8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b16ab248e22dc24 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:32.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.003+0000 7fa6f7400640 1 -- 172.21.15.96:0/1445414690 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6e8036070 con 0x7fa6d80039e0 2024-05-28T06:17:32.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.003+0000 7fa6f7400640 1 -- 172.21.15.96:0/1445414690 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6e8031400 con 0x7fa6d80039e0 2024-05-28T06:17:32.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.003+0000 7fa6f7400640 1 -- 172.21.15.96:0/1445414690 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6e8009690 con 0x7fa6d80039e0 2024-05-28T06:17:32.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.003+0000 7fa6ffa00640 1 -- 172.21.15.96:0/1445414690 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6d80039e0 msgr2=0x7fa6d8003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:32.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.003+0000 7fa6ffa00640 1 --2- 172.21.15.96:0/1445414690 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6d80039e0 0x7fa6d8003e00 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7fa6e8009ac0 tx=0x7fa6d8005830 comp rx=0 tx=0).stop 2024-05-28T06:17:32.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.003+0000 7fa6fda00640 1 -- 172.21.15.96:0/1445414690 reap_dead start 2024-05-28T06:17:32.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.003+0000 7fa6ffa00640 1 -- 172.21.15.96:0/1445414690 shutdown_connections 2024-05-28T06:17:32.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.003+0000 7fa6ffa00640 1 -- 172.21.15.96:0/1445414690 >> 172.21.15.96:0/1445414690 conn(0x7fa6f808bfc0 msgr2=0x7fa6f80d4630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:32.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.004+0000 7fa6ffa00640 1 -- 172.21.15.96:0/1445414690 shutdown_connections 2024-05-28T06:17:32.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.004+0000 7fa6ffa00640 1 -- 172.21.15.96:0/1445414690 wait complete. 2024-05-28T06:17:32.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.005+0000 7fa6ffa00640 1 Processor -- start 2024-05-28T06:17:32.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.005+0000 7fa6ffa00640 1 -- start start 2024-05-28T06:17:32.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.006+0000 7fa6ffa00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa6d80039e0 0x7fa6f81576b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:32.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.006+0000 7fa6ffa00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6f81e8bb0 0x7fa6f80cd100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:32.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.006+0000 7fa6ffa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6f80cd640 0x7fa6f80d39e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:32.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.006+0000 7fa6ffa00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa6f80d0320 con 0x7fa6f80cd640 2024-05-28T06:17:32.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.006+0000 7fa6ffa00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa6f80d0040 con 0x7fa6f81e8bb0 2024-05-28T06:17:32.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.006+0000 7fa6ffa00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fa6f80cf380 con 0x7fa6d80039e0 2024-05-28T06:17:32.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.007+0000 7fa6fda00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6f80cd640 0x7fa6f80d39e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:32.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.007+0000 7fa6fda00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6f80cd640 0x7fa6f80d39e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:33756/0 (socket says 172.21.15.96:33756) 2024-05-28T06:17:32.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.007+0000 7fa6fda00640 1 -- 172.21.15.96:0/3693703841 learned_addr learned my addr 172.21.15.96:0/3693703841 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:17:32.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.007+0000 7fa6fd000640 1 --2- 172.21.15.96:0/3693703841 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa6d80039e0 0x7fa6f81576b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:32.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.007+0000 7fa6f7e00640 1 --2- 172.21.15.96:0/3693703841 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6f81e8bb0 0x7fa6f80cd100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:32.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.007+0000 7fa6fda00640 1 -- 172.21.15.96:0/3693703841 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa6d80039e0 msgr2=0x7fa6f81576b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:32.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.007+0000 7fa6fda00640 1 --2- 172.21.15.96:0/3693703841 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa6d80039e0 0x7fa6f81576b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:32.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.007+0000 7fa6fda00640 1 -- 172.21.15.96:0/3693703841 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6f81e8bb0 msgr2=0x7fa6f80cd100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:32.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.007+0000 7fa6fda00640 1 --2- 172.21.15.96:0/3693703841 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6f81e8bb0 0x7fa6f80cd100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:32.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.007+0000 7fa6fda00640 1 -- 172.21.15.96:0/3693703841 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6f81e9030 con 0x7fa6f80cd640 2024-05-28T06:17:32.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.007+0000 7fa6f7e00640 1 --2- 172.21.15.96:0/3693703841 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6f81e8bb0 0x7fa6f80cd100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:17:32.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.007+0000 7fa6fda00640 1 --2- 172.21.15.96:0/3693703841 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6f80cd640 0x7fa6f80d39e0 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7fa6f806de90 tx=0x7fa6f000c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:32.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.008+0000 7fa6f4c00640 1 -- 172.21.15.96:0/3693703841 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6f00074e0 con 0x7fa6f80cd640 2024-05-28T06:17:32.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.008+0000 7fa6f4c00640 1 -- 172.21.15.96:0/3693703841 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6f0004020 con 0x7fa6f80cd640 2024-05-28T06:17:32.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.008+0000 7fa6f4c00640 1 -- 172.21.15.96:0/3693703841 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6f0004410 con 0x7fa6f80cd640 2024-05-28T06:17:32.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.008+0000 7fa6ffa00640 1 -- 172.21.15.96:0/3693703841 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6f81e9310 con 0x7fa6f80cd640 2024-05-28T06:17:32.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.008+0000 7fa6ffa00640 1 -- 172.21.15.96:0/3693703841 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6f81e98d0 con 0x7fa6f80cd640 2024-05-28T06:17:32.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.009+0000 7fa6ffa00640 1 -- 172.21.15.96:0/3693703841 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6f80689e0 con 0x7fa6f80cd640 2024-05-28T06:17:32.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.010+0000 7fa6f4c00640 1 -- 172.21.15.96:0/3693703841 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 28) ==== 92570+0+0 (secure 0 0 0) 0x7fa6f00057c0 con 0x7fa6f80cd640 2024-05-28T06:17:32.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.011+0000 7fa6f4c00640 1 --2- 172.21.15.96:0/3693703841 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa6d406d9d0 0x7fa6d406fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:32.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.012+0000 7fa6f4c00640 1 -- 172.21.15.96:0/3693703841 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5795+0+0 (secure 0 0 0) 0x7fa6f00898c0 con 0x7fa6f80cd640 2024-05-28T06:17:32.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.012+0000 7fa6fd000640 1 --2- 172.21.15.96:0/3693703841 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa6d406d9d0 0x7fa6d406fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:32.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.013+0000 7fa6fd000640 1 --2- 172.21.15.96:0/3693703841 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa6d406d9d0 0x7fa6d406fe90 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fa6e8033000 tx=0x7fa6e8008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:32.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.016+0000 7fa6f4c00640 1 -- 172.21.15.96:0/3693703841 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa6f0059740 con 0x7fa6f80cd640 2024-05-28T06:17:32.290 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.292+0000 7fa6ffa00640 1 -- 172.21.15.96:0/3693703841 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fa6f80d1c70 con 0x7fa6f80cd640 2024-05-28T06:17:32.291 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.294+0000 7fa6f4c00640 1 -- 172.21.15.96:0/3693703841 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1826 (secure 0 0 0) 0x7fa6f0007e20 con 0x7fa6f80cd640 2024-05-28T06:17:32.292 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:17:32.292 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:17:32.292 INFO:teuthology.orchestra.run.smithi096.stdout: "fsid": "e4a831b5-e6d1-47a7-96cb-bf85154ee0d2", 2024-05-28T06:17:32.292 INFO:teuthology.orchestra.run.smithi096.stdout: "health": { 2024-05-28T06:17:32.292 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "HEALTH_OK", 2024-05-28T06:17:32.292 INFO:teuthology.orchestra.run.smithi096.stdout: "checks": {}, 2024-05-28T06:17:32.292 INFO:teuthology.orchestra.run.smithi096.stdout: "mutes": [] 2024-05-28T06:17:32.292 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:17:32.292 INFO:teuthology.orchestra.run.smithi096.stdout: "election_epoch": 8, 2024-05-28T06:17:32.292 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum": [ 2024-05-28T06:17:32.292 INFO:teuthology.orchestra.run.smithi096.stdout: 0, 2024-05-28T06:17:32.292 INFO:teuthology.orchestra.run.smithi096.stdout: 1, 2024-05-28T06:17:32.292 INFO:teuthology.orchestra.run.smithi096.stdout: 2 2024-05-28T06:17:32.293 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:17:32.293 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_names": [ 2024-05-28T06:17:32.293 INFO:teuthology.orchestra.run.smithi096.stdout: "a", 2024-05-28T06:17:32.293 INFO:teuthology.orchestra.run.smithi096.stdout: "b", 2024-05-28T06:17:32.293 INFO:teuthology.orchestra.run.smithi096.stdout: "c" 2024-05-28T06:17:32.293 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:17:32.293 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_age": 825, 2024-05-28T06:17:32.293 INFO:teuthology.orchestra.run.smithi096.stdout: "monmap": { 2024-05-28T06:17:32.293 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 1, 2024-05-28T06:17:32.293 INFO:teuthology.orchestra.run.smithi096.stdout: "min_mon_release_name": "squid", 2024-05-28T06:17:32.293 INFO:teuthology.orchestra.run.smithi096.stdout: "num_mons": 3 2024-05-28T06:17:32.293 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:17:32.293 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap": { 2024-05-28T06:17:32.293 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 76, 2024-05-28T06:17:32.293 INFO:teuthology.orchestra.run.smithi096.stdout: "num_osds": 8, 2024-05-28T06:17:32.294 INFO:teuthology.orchestra.run.smithi096.stdout: "num_up_osds": 8, 2024-05-28T06:17:32.294 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_up_since": 1716876238, 2024-05-28T06:17:32.294 INFO:teuthology.orchestra.run.smithi096.stdout: "num_in_osds": 8, 2024-05-28T06:17:32.294 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_in_since": 1716876229, 2024-05-28T06:17:32.294 INFO:teuthology.orchestra.run.smithi096.stdout: "num_remapped_pgs": 0 2024-05-28T06:17:32.294 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:17:32.294 INFO:teuthology.orchestra.run.smithi096.stdout: "pgmap": { 2024-05-28T06:17:32.294 INFO:teuthology.orchestra.run.smithi096.stdout: "pgs_by_state": [ 2024-05-28T06:17:32.294 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:17:32.294 INFO:teuthology.orchestra.run.smithi096.stdout: "state_name": "active+clean", 2024-05-28T06:17:32.294 INFO:teuthology.orchestra.run.smithi096.stdout: "count": 97 2024-05-28T06:17:32.294 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:17:32.294 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:17:32.295 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pgs": 97, 2024-05-28T06:17:32.295 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pools": 3, 2024-05-28T06:17:32.295 INFO:teuthology.orchestra.run.smithi096.stdout: "num_objects": 12467, 2024-05-28T06:17:32.295 INFO:teuthology.orchestra.run.smithi096.stdout: "data_bytes": 52089816945, 2024-05-28T06:17:32.295 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_used": 104816033792, 2024-05-28T06:17:32.295 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_avail": 668278079488, 2024-05-28T06:17:32.295 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_total": 773094113280, 2024-05-28T06:17:32.295 INFO:teuthology.orchestra.run.smithi096.stdout: "write_bytes_sec": 767, 2024-05-28T06:17:32.295 INFO:teuthology.orchestra.run.smithi096.stdout: "read_op_per_sec": 0, 2024-05-28T06:17:32.295 INFO:teuthology.orchestra.run.smithi096.stdout: "write_op_per_sec": 0 2024-05-28T06:17:32.295 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:17:32.295 INFO:teuthology.orchestra.run.smithi096.stdout: "fsmap": { 2024-05-28T06:17:32.295 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 44, 2024-05-28T06:17:32.295 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 5, 2024-05-28T06:17:32.296 INFO:teuthology.orchestra.run.smithi096.stdout: "up": 1, 2024-05-28T06:17:32.296 INFO:teuthology.orchestra.run.smithi096.stdout: "in": 1, 2024-05-28T06:17:32.296 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 1, 2024-05-28T06:17:32.296 INFO:teuthology.orchestra.run.smithi096.stdout: "by_rank": [ 2024-05-28T06:17:32.296 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:17:32.296 INFO:teuthology.orchestra.run.smithi096.stdout: "filesystem_id": 5, 2024-05-28T06:17:32.296 INFO:teuthology.orchestra.run.smithi096.stdout: "rank": 0, 2024-05-28T06:17:32.296 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "d", 2024-05-28T06:17:32.296 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "up:active", 2024-05-28T06:17:32.296 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 7146 2024-05-28T06:17:32.296 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:17:32.296 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:17:32.296 INFO:teuthology.orchestra.run.smithi096.stdout: "up:standby": 3 2024-05-28T06:17:32.296 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:17:32.296 INFO:teuthology.orchestra.run.smithi096.stdout: "mgrmap": { 2024-05-28T06:17:32.297 INFO:teuthology.orchestra.run.smithi096.stdout: "available": true, 2024-05-28T06:17:32.297 INFO:teuthology.orchestra.run.smithi096.stdout: "num_standbys": 1, 2024-05-28T06:17:32.297 INFO:teuthology.orchestra.run.smithi096.stdout: "modules": [ 2024-05-28T06:17:32.297 INFO:teuthology.orchestra.run.smithi096.stdout: "iostat", 2024-05-28T06:17:32.297 INFO:teuthology.orchestra.run.smithi096.stdout: "nfs", 2024-05-28T06:17:32.297 INFO:teuthology.orchestra.run.smithi096.stdout: "restful" 2024-05-28T06:17:32.297 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:17:32.297 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:17:32.297 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:17:32.297 INFO:teuthology.orchestra.run.smithi096.stdout: "servicemap": { 2024-05-28T06:17:32.297 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 13, 2024-05-28T06:17:32.297 INFO:teuthology.orchestra.run.smithi096.stdout: "modified": "2024-05-28T06:15:57.778889+0000", 2024-05-28T06:17:32.297 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:17:32.297 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:17:32.298 INFO:teuthology.orchestra.run.smithi096.stdout: "progress_events": {} 2024-05-28T06:17:32.298 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:17:32.298 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.296+0000 7fa6ffa00640 1 -- 172.21.15.96:0/3693703841 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa6d406d9d0 msgr2=0x7fa6d406fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:32.299 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.296+0000 7fa6ffa00640 1 --2- 172.21.15.96:0/3693703841 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa6d406d9d0 0x7fa6d406fe90 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fa6e8033000 tx=0x7fa6e8008030 comp rx=0 tx=0).stop 2024-05-28T06:17:32.299 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.296+0000 7fa6ffa00640 1 -- 172.21.15.96:0/3693703841 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6f80cd640 msgr2=0x7fa6f80d39e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:32.299 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.296+0000 7fa6ffa00640 1 --2- 172.21.15.96:0/3693703841 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6f80cd640 0x7fa6f80d39e0 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7fa6f806de90 tx=0x7fa6f000c820 comp rx=0 tx=0).stop 2024-05-28T06:17:32.299 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.297+0000 7fa6ffa00640 1 -- 172.21.15.96:0/3693703841 shutdown_connections 2024-05-28T06:17:32.299 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.297+0000 7fa6ffa00640 1 --2- 172.21.15.96:0/3693703841 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa6d406d9d0 0x7fa6d406fe90 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:32.299 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.297+0000 7fa6ffa00640 1 --2- 172.21.15.96:0/3693703841 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6f80cd640 0x7fa6f80d39e0 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:32.299 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.297+0000 7fa6ffa00640 1 --2- 172.21.15.96:0/3693703841 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6f81e8bb0 0x7fa6f80cd100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:32.299 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.297+0000 7fa6ffa00640 1 --2- 172.21.15.96:0/3693703841 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa6d80039e0 0x7fa6f81576b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:32.299 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.297+0000 7fa6ffa00640 1 -- 172.21.15.96:0/3693703841 >> 172.21.15.96:0/3693703841 conn(0x7fa6f808bfc0 msgr2=0x7fa6f80b31f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:32.299 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.297+0000 7fa6ffa00640 1 -- 172.21.15.96:0/3693703841 shutdown_connections 2024-05-28T06:17:32.299 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.297+0000 7fa6ffa00640 1 -- 172.21.15.96:0/3693703841 wait complete. 2024-05-28T06:17:32.314 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume ls cephfs --format json 2024-05-28T06:17:32.513 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.515+0000 7f5317200640 1 Processor -- start 2024-05-28T06:17:32.513 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.515+0000 7f5317200640 1 -- start start 2024-05-28T06:17:32.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.517+0000 7f5317200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f53180caa40 0x7f53180ce960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:32.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.517+0000 7f5317200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f53180cae20 0x7f5318151db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:32.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.517+0000 7f5317200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53180b3e80 con 0x7f53181522f0 2024-05-28T06:17:32.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.517+0000 7f5317200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f53180d0b30 con 0x7f53180cae20 2024-05-28T06:17:32.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.517+0000 7f5317200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f53180b4160 con 0x7f53180caa40 2024-05-28T06:17:32.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.517+0000 7f5315400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f53180cae20 0x7f5318151db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:32.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.517+0000 7f5315400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f53180cae20 0x7f5318151db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:33760/0 (socket says 172.21.15.96:33760) 2024-05-28T06:17:32.516 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.517+0000 7f5315400640 1 -- 172.21.15.96:0/2422935615 learned_addr learned my addr 172.21.15.96:0/2422935615 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:17:32.516 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.517+0000 7f5316800640 1 --1- 172.21.15.96:0/2422935615 >> v1:172.21.15.96:6789/0 conn(0x7f53181522f0 0x7f53180cf710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:57734/0 (socket says 172.21.15.96:57734) 2024-05-28T06:17:32.516 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.517+0000 7f5315e00640 1 --2- 172.21.15.96:0/2422935615 >> v2:172.21.15.123:3300/0 conn(0x7f53180caa40 0x7f53180ce960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:32.517 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.518+0000 7f5315400640 1 -- 172.21.15.96:0/2422935615 >> v2:172.21.15.123:3300/0 conn(0x7f53180caa40 msgr2=0x7f53180ce960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:32.517 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.518+0000 7f5315400640 1 --2- 172.21.15.96:0/2422935615 >> v2:172.21.15.123:3300/0 conn(0x7f53180caa40 0x7f53180ce960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:32.517 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.518+0000 7f5315400640 1 -- 172.21.15.96:0/2422935615 >> v1:172.21.15.96:6789/0 conn(0x7f53181522f0 legacy=0x7f53180cf710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:32.517 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.518+0000 7f5315400640 1 -- 172.21.15.96:0/2422935615 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53180cde80 con 0x7f53180cae20 2024-05-28T06:17:32.517 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.518+0000 7f5315e00640 1 --2- 172.21.15.96:0/2422935615 >> v2:172.21.15.123:3300/0 conn(0x7f53180caa40 0x7f53180ce960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:17:32.517 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.519+0000 7f5315400640 1 --2- 172.21.15.96:0/2422935615 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f53180cae20 0x7f5318151db0 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f53180b4c10 tx=0x7f530002a1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbaa362d18f06377 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:32.518 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.519+0000 7f5314a00640 1 -- 172.21.15.96:0/2422935615 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5300037070 con 0x7f53180cae20 2024-05-28T06:17:32.518 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.519+0000 7f5314a00640 1 -- 172.21.15.96:0/2422935615 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5300032440 con 0x7f53180cae20 2024-05-28T06:17:32.518 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.519+0000 7f5314a00640 1 -- 172.21.15.96:0/2422935615 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5300032810 con 0x7f53180cae20 2024-05-28T06:17:32.518 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.520+0000 7f5317200640 1 -- 172.21.15.96:0/2422935615 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f53180cae20 msgr2=0x7f5318151db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:32.518 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.520+0000 7f5317200640 1 --2- 172.21.15.96:0/2422935615 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f53180cae20 0x7f5318151db0 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f53180b4c10 tx=0x7f530002a1f0 comp rx=0 tx=0).stop 2024-05-28T06:17:32.518 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.520+0000 7f5317200640 1 -- 172.21.15.96:0/2422935615 shutdown_connections 2024-05-28T06:17:32.518 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.520+0000 7f5317200640 1 --2- 172.21.15.96:0/2422935615 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f53180cae20 0x7f5318151db0 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:32.518 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.520+0000 7f5317200640 1 --2- 172.21.15.96:0/2422935615 >> v2:172.21.15.123:3300/0 conn(0x7f53180caa40 0x7f53180ce960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:32.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.520+0000 7f5317200640 1 -- 172.21.15.96:0/2422935615 >> 172.21.15.96:0/2422935615 conn(0x7f531806b900 msgr2=0x7f531806bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:32.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.520+0000 7f5317200640 1 -- 172.21.15.96:0/2422935615 shutdown_connections 2024-05-28T06:17:32.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.520+0000 7f5317200640 1 -- 172.21.15.96:0/2422935615 wait complete. 2024-05-28T06:17:32.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.522+0000 7f5317200640 1 Processor -- start 2024-05-28T06:17:32.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.522+0000 7f5317200640 1 -- start start 2024-05-28T06:17:32.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.522+0000 7f5317200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f53180caa40 0x7f531814d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:32.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.523+0000 7f5317200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f53180cae20 0x7f5318073380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:32.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.523+0000 7f5317200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f53181522f0 0x7f5318153c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:32.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.523+0000 7f5317200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f5318152710 con 0x7f53180cae20 2024-05-28T06:17:32.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.523+0000 7f5317200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f53180ce440 con 0x7f53180caa40 2024-05-28T06:17:32.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.523+0000 7f5317200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f53180ceea0 con 0x7f53181522f0 2024-05-28T06:17:32.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.523+0000 7f5315400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f53180cae20 0x7f5318073380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:32.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.523+0000 7f5315400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f53180cae20 0x7f5318073380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:33774/0 (socket says 172.21.15.96:33774) 2024-05-28T06:17:32.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.523+0000 7f5315400640 1 -- 172.21.15.96:0/1113868286 learned_addr learned my addr 172.21.15.96:0/1113868286 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:17:32.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.523+0000 7f5315e00640 1 --2- 172.21.15.96:0/1113868286 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f53180caa40 0x7f531814d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:32.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.523+0000 7f5315400640 1 -- 172.21.15.96:0/1113868286 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f53181522f0 msgr2=0x7f5318153c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:32.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.523+0000 7f5315400640 1 --2- 172.21.15.96:0/1113868286 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f53181522f0 0x7f5318153c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:32.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.523+0000 7f5315400640 1 -- 172.21.15.96:0/1113868286 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f53180caa40 msgr2=0x7f531814d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:32.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.523+0000 7f5315400640 1 --2- 172.21.15.96:0/1113868286 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f53180caa40 0x7f531814d200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:32.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.523+0000 7f5315400640 1 -- 172.21.15.96:0/1113868286 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53180722e0 con 0x7f53180cae20 2024-05-28T06:17:32.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.524+0000 7f5315400640 1 --2- 172.21.15.96:0/1113868286 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f53180cae20 0x7f5318073380 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f5300000c00 tx=0x7f53000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:32.523 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.524+0000 7f530e000640 1 -- 172.21.15.96:0/1113868286 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5300037040 con 0x7f53180cae20 2024-05-28T06:17:32.523 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.524+0000 7f530e000640 1 -- 172.21.15.96:0/1113868286 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5300042070 con 0x7f53180cae20 2024-05-28T06:17:32.523 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.524+0000 7f5317200640 1 -- 172.21.15.96:0/1113868286 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53180725c0 con 0x7f53180cae20 2024-05-28T06:17:32.523 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.524+0000 7f530e000640 1 -- 172.21.15.96:0/1113868286 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5300030bb0 con 0x7f53180cae20 2024-05-28T06:17:32.523 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.524+0000 7f5317200640 1 -- 172.21.15.96:0/1113868286 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5318072b80 con 0x7f53180cae20 2024-05-28T06:17:32.523 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.525+0000 7f5317200640 1 -- 172.21.15.96:0/1113868286 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53180689e0 con 0x7f53180cae20 2024-05-28T06:17:32.524 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.526+0000 7f530e000640 1 -- 172.21.15.96:0/1113868286 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 28) ==== 92570+0+0 (secure 0 0 0) 0x7f5300047460 con 0x7f53180cae20 2024-05-28T06:17:32.525 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.527+0000 7f530e000640 1 --2- 172.21.15.96:0/1113868286 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f52f406d9d0 0x7f52f406fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:32.525 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.527+0000 7f530e000640 1 -- 172.21.15.96:0/1113868286 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5795+0+0 (secure 0 0 0) 0x7f53000ab350 con 0x7f53180cae20 2024-05-28T06:17:32.525 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.528+0000 7f5315e00640 1 --2- 172.21.15.96:0/1113868286 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f52f406d9d0 0x7f52f406fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:32.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.529+0000 7f5315e00640 1 --2- 172.21.15.96:0/1113868286 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f52f406d9d0 0x7f52f406fe90 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f5308008f80 tx=0x7f53080084d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:32.529 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.532+0000 7f530e000640 1 -- 172.21.15.96:0/1113868286 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5300045360 con 0x7f53180cae20 2024-05-28T06:17:32.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.753+0000 7f5317200640 1 -- 172.21.15.96:0/1113868286 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume ls", "vol_name": "cephfs", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f53181520d0 con 0x7f52f406d9d0 2024-05-28T06:17:32.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.797+0000 7f530e000640 1 -- 172.21.15.96:0/1113868286 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+1029 (secure 0 0 0) 0x7f53181520d0 con 0x7f52f406d9d0 2024-05-28T06:17:32.796 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:17:32.796 INFO:teuthology.orchestra.run.smithi096.stdout:[ 2024-05-28T06:17:32.796 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:17:32.796 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "subvol_clone436316" 2024-05-28T06:17:32.796 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:17:32.796 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:17:32.796 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "subvol_clone436319" 2024-05-28T06:17:32.796 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:17:32.796 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:17:32.796 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "subvol_clone436313" 2024-05-28T06:17:32.796 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:17:32.796 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:17:32.796 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "subvol_clone436311" 2024-05-28T06:17:32.797 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:17:32.797 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:17:32.797 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "subvol_clone436320" 2024-05-28T06:17:32.797 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:17:32.797 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:17:32.797 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "sv1" 2024-05-28T06:17:32.797 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:17:32.797 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:17:32.797 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "subvol_clone436315" 2024-05-28T06:17:32.797 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:17:32.797 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:17:32.797 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "subvol_clone43632" 2024-05-28T06:17:32.797 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:17:32.797 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:17:32.798 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "subvol_clone43638" 2024-05-28T06:17:32.798 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:17:32.798 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:17:32.798 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "subvol_clone436312" 2024-05-28T06:17:32.798 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:17:32.798 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:17:32.798 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "subvol_clone436318" 2024-05-28T06:17:32.798 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:17:32.798 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:17:32.798 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "subvol_clone43634" 2024-05-28T06:17:32.798 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:17:32.798 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:17:32.798 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "subvol_clone436310" 2024-05-28T06:17:32.799 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:17:32.799 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:17:32.799 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "subvol_clone436314" 2024-05-28T06:17:32.799 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:17:32.799 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:17:32.799 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "subvol_clone43633" 2024-05-28T06:17:32.799 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:17:32.799 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:17:32.799 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "subvol_clone43636" 2024-05-28T06:17:32.799 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:17:32.799 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:17:32.799 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "subvol_clone43637" 2024-05-28T06:17:32.799 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:17:32.799 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:17:32.800 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "subvol_clone436317" 2024-05-28T06:17:32.800 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:17:32.800 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:17:32.800 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "subvol_clone43639" 2024-05-28T06:17:32.800 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:17:32.800 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:17:32.800 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "subvol_clone43635" 2024-05-28T06:17:32.800 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:17:32.800 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:17:32.800 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "subvol_clone43631" 2024-05-28T06:17:32.800 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:17:32.800 INFO:teuthology.orchestra.run.smithi096.stdout:] 2024-05-28T06:17:32.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.800+0000 7f5317200640 1 -- 172.21.15.96:0/1113868286 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f52f406d9d0 msgr2=0x7f52f406fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:32.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.800+0000 7f5317200640 1 --2- 172.21.15.96:0/1113868286 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f52f406d9d0 0x7f52f406fe90 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f5308008f80 tx=0x7f53080084d0 comp rx=0 tx=0).stop 2024-05-28T06:17:32.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.800+0000 7f5317200640 1 -- 172.21.15.96:0/1113868286 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f53180cae20 msgr2=0x7f5318073380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:32.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.800+0000 7f5317200640 1 --2- 172.21.15.96:0/1113868286 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f53180cae20 0x7f5318073380 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f5300000c00 tx=0x7f53000046c0 comp rx=0 tx=0).stop 2024-05-28T06:17:32.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.800+0000 7f5317200640 1 -- 172.21.15.96:0/1113868286 shutdown_connections 2024-05-28T06:17:32.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.800+0000 7f5317200640 1 --2- 172.21.15.96:0/1113868286 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f52f406d9d0 0x7f52f406fe90 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:32.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.800+0000 7f5317200640 1 --2- 172.21.15.96:0/1113868286 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f53181522f0 0x7f5318153c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:32.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.800+0000 7f5317200640 1 --2- 172.21.15.96:0/1113868286 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f53180cae20 0x7f5318073380 unknown :-1 s=CLOSED pgs=1244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:32.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.800+0000 7f5317200640 1 --2- 172.21.15.96:0/1113868286 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f53180caa40 0x7f531814d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:32.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.801+0000 7f5317200640 1 -- 172.21.15.96:0/1113868286 >> 172.21.15.96:0/1113868286 conn(0x7f531806b900 msgr2=0x7f531815ab50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:32.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.801+0000 7f5317200640 1 -- 172.21.15.96:0/1113868286 shutdown_connections 2024-05-28T06:17:32.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:32.801+0000 7f5317200640 1 -- 172.21.15.96:0/1113868286 wait complete. 2024-05-28T06:17:32.818 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone436316 2024-05-28T06:17:32.915 DEBUG:teuthology.orchestra.run.smithi096:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:17:32.921 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:17:33.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.019+0000 7f53df200640 1 Processor -- start 2024-05-28T06:17:33.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.019+0000 7f53df200640 1 -- start start 2024-05-28T06:17:33.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.021+0000 7f53df200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f53d80d34e0 0x7f53d80d4e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:33.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.021+0000 7f53df200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f53d80b5570 0x7f53d80b5a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:33.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.021+0000 7f53df200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53d80d5500 con 0x7f53d81622b0 2024-05-28T06:17:33.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.021+0000 7f53df200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f53d80b4f50 con 0x7f53d80d34e0 2024-05-28T06:17:33.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.021+0000 7f53df200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f53d80d2720 con 0x7f53d80b5570 2024-05-28T06:17:33.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.021+0000 7f53d7400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f53d80d34e0 0x7f53d80d4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:33.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.021+0000 7f53d7400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f53d80d34e0 0x7f53d80d4e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:33782/0 (socket says 172.21.15.96:33782) 2024-05-28T06:17:33.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.021+0000 7f53d7400640 1 -- 172.21.15.96:0/379368831 learned_addr learned my addr 172.21.15.96:0/379368831 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:17:33.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.022+0000 7f53dd200640 1 --2- 172.21.15.96:0/379368831 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f53d80b5570 0x7f53d80b5a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:33.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.022+0000 7f53d7400640 1 -- 172.21.15.96:0/379368831 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f53d80b5570 msgr2=0x7f53d80b5a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:33.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.022+0000 7f53d7400640 1 --2- 172.21.15.96:0/379368831 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f53d80b5570 0x7f53d80b5a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:33.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.022+0000 7f53d7400640 1 -- 172.21.15.96:0/379368831 >> v1:172.21.15.123:6789/0 conn(0x7f53d81622b0 legacy=0x7f53d80d1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:33.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.022+0000 7f53d7400640 1 -- 172.21.15.96:0/379368831 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53d80d2a00 con 0x7f53d80d34e0 2024-05-28T06:17:33.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.022+0000 7f53dd200640 1 --2- 172.21.15.96:0/379368831 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f53d80b5570 0x7f53d80b5a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:17:33.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.023+0000 7f53d7400640 1 --2- 172.21.15.96:0/379368831 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f53d80d34e0 0x7f53d80d4e00 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f53c4009870 tx=0x7f53c402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3de4b5bced1d026b server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:33.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.024+0000 7f53d6a00640 1 -- 172.21.15.96:0/379368831 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53c4037070 con 0x7f53d80d34e0 2024-05-28T06:17:33.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.024+0000 7f53d6a00640 1 -- 172.21.15.96:0/379368831 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53c4032440 con 0x7f53d80d34e0 2024-05-28T06:17:33.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.024+0000 7f53d6a00640 1 -- 172.21.15.96:0/379368831 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53c4032810 con 0x7f53d80d34e0 2024-05-28T06:17:33.022 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.024+0000 7f53df200640 1 -- 172.21.15.96:0/379368831 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f53d80d34e0 msgr2=0x7f53d80d4e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:33.022 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.024+0000 7f53df200640 1 --2- 172.21.15.96:0/379368831 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f53d80d34e0 0x7f53d80d4e00 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f53c4009870 tx=0x7f53c402a1e0 comp rx=0 tx=0).stop 2024-05-28T06:17:33.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.025+0000 7f53df200640 1 -- 172.21.15.96:0/379368831 shutdown_connections 2024-05-28T06:17:33.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.025+0000 7f53df200640 1 --2- 172.21.15.96:0/379368831 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f53d80b5570 0x7f53d80b5a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:33.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.025+0000 7f53df200640 1 --2- 172.21.15.96:0/379368831 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f53d80d34e0 0x7f53d80d4e00 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:33.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.025+0000 7f53df200640 1 -- 172.21.15.96:0/379368831 >> 172.21.15.96:0/379368831 conn(0x7f53d808bfc0 msgr2=0x7f53d80d46f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:33.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.025+0000 7f53df200640 1 -- 172.21.15.96:0/379368831 shutdown_connections 2024-05-28T06:17:33.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.025+0000 7f53df200640 1 -- 172.21.15.96:0/379368831 wait complete. 2024-05-28T06:17:33.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.026+0000 7f53df200640 1 Processor -- start 2024-05-28T06:17:33.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.027+0000 7f53df200640 1 -- start start 2024-05-28T06:17:33.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.028+0000 7f53df200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f53d80b5570 0x7f53d81f90e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:33.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.028+0000 7f53df200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f53d80d34e0 0x7f53d80b7310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:33.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.028+0000 7f53df200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f53d81622b0 0x7f53d80b7850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:33.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.028+0000 7f53df200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f53d80b40c0 con 0x7f53d80b5570 2024-05-28T06:17:33.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.028+0000 7f53df200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f53d80b3f40 con 0x7f53d81622b0 2024-05-28T06:17:33.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.028+0000 7f53df200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f53d80b4240 con 0x7f53d80d34e0 2024-05-28T06:17:33.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.028+0000 7f53d7e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f53d80b5570 0x7f53d81f90e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:33.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.028+0000 7f53d7e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f53d80b5570 0x7f53d81f90e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:33788/0 (socket says 172.21.15.96:33788) 2024-05-28T06:17:33.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.028+0000 7f53d7e00640 1 -- 172.21.15.96:0/3703296326 learned_addr learned my addr 172.21.15.96:0/3703296326 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:17:33.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.028+0000 7f53dd200640 1 --2- 172.21.15.96:0/3703296326 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f53d81622b0 0x7f53d80b7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:33.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.028+0000 7f53d7400640 1 --2- 172.21.15.96:0/3703296326 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f53d80d34e0 0x7f53d80b7310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:33.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.028+0000 7f53d7e00640 1 -- 172.21.15.96:0/3703296326 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f53d80d34e0 msgr2=0x7f53d80b7310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:33.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.028+0000 7f53d7e00640 1 --2- 172.21.15.96:0/3703296326 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f53d80d34e0 0x7f53d80b7310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:33.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.028+0000 7f53d7e00640 1 -- 172.21.15.96:0/3703296326 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f53d81622b0 msgr2=0x7f53d80b7850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:33.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.028+0000 7f53d7e00640 1 --2- 172.21.15.96:0/3703296326 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f53d81622b0 0x7f53d80b7850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:33.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.028+0000 7f53d7e00640 1 -- 172.21.15.96:0/3703296326 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53d80b96e0 con 0x7f53d80b5570 2024-05-28T06:17:33.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.029+0000 7f53d7e00640 1 --2- 172.21.15.96:0/3703296326 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f53d80b5570 0x7f53d81f90e0 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f53c800c3c0 tx=0x7f53c800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:33.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.029+0000 7f53cfe00640 1 -- 172.21.15.96:0/3703296326 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53c80052d0 con 0x7f53d80b5570 2024-05-28T06:17:33.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.029+0000 7f53cfe00640 1 -- 172.21.15.96:0/3703296326 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53c8005490 con 0x7f53d80b5570 2024-05-28T06:17:33.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.029+0000 7f53cfe00640 1 -- 172.21.15.96:0/3703296326 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53c8007260 con 0x7f53d80b5570 2024-05-28T06:17:33.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.029+0000 7f53df200640 1 -- 172.21.15.96:0/3703296326 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53d80b99c0 con 0x7f53d80b5570 2024-05-28T06:17:33.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.029+0000 7f53df200640 1 -- 172.21.15.96:0/3703296326 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53d80b9f80 con 0x7f53d80b5570 2024-05-28T06:17:33.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.030+0000 7f53df200640 1 -- 172.21.15.96:0/3703296326 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53d80ba180 con 0x7f53d80b5570 2024-05-28T06:17:33.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.031+0000 7f53cfe00640 1 -- 172.21.15.96:0/3703296326 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 28) ==== 92570+0+0 (secure 0 0 0) 0x7f53c8007400 con 0x7f53d80b5570 2024-05-28T06:17:33.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.032+0000 7f53cfe00640 1 --2- 172.21.15.96:0/3703296326 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f53b806d9d0 0x7f53b806fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:33.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.032+0000 7f53cfe00640 1 -- 172.21.15.96:0/3703296326 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5795+0+0 (secure 0 0 0) 0x7f53c8089bf0 con 0x7f53d80b5570 2024-05-28T06:17:33.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.033+0000 7f53d7400640 1 --2- 172.21.15.96:0/3703296326 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f53b806d9d0 0x7f53b806fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:33.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.034+0000 7f53d7400640 1 --2- 172.21.15.96:0/3703296326 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f53b806d9d0 0x7f53b806fe90 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f53c4034000 tx=0x7f53c402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:33.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.036+0000 7f53cfe00640 1 -- 172.21.15.96:0/3703296326 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f53c8059a70 con 0x7f53d80b5570 2024-05-28T06:17:33.265 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.267+0000 7f53df200640 1 -- 172.21.15.96:0/3703296326 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone436316", "target": ["mon-mgr", ""]}) -- 0x7f53d80b9490 con 0x7f53b806d9d0 2024-05-28T06:17:33.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.319+0000 7f53cfe00640 1 -- 172.21.15.96:0/3703296326 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f53d80b9490 con 0x7f53b806d9d0 2024-05-28T06:17:33.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.321+0000 7f53df200640 1 -- 172.21.15.96:0/3703296326 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f53b806d9d0 msgr2=0x7f53b806fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:33.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.321+0000 7f53df200640 1 --2- 172.21.15.96:0/3703296326 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f53b806d9d0 0x7f53b806fe90 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f53c4034000 tx=0x7f53c402e000 comp rx=0 tx=0).stop 2024-05-28T06:17:33.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.321+0000 7f53df200640 1 -- 172.21.15.96:0/3703296326 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f53d80b5570 msgr2=0x7f53d81f90e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:33.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.321+0000 7f53df200640 1 --2- 172.21.15.96:0/3703296326 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f53d80b5570 0x7f53d81f90e0 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f53c800c3c0 tx=0x7f53c800c880 comp rx=0 tx=0).stop 2024-05-28T06:17:33.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.321+0000 7f53df200640 1 -- 172.21.15.96:0/3703296326 shutdown_connections 2024-05-28T06:17:33.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.321+0000 7f53df200640 1 --2- 172.21.15.96:0/3703296326 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f53b806d9d0 0x7f53b806fe90 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:33.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.322+0000 7f53df200640 1 --2- 172.21.15.96:0/3703296326 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f53d81622b0 0x7f53d80b7850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:33.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.322+0000 7f53df200640 1 --2- 172.21.15.96:0/3703296326 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f53d80d34e0 0x7f53d80b7310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:33.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.322+0000 7f53df200640 1 --2- 172.21.15.96:0/3703296326 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f53d80b5570 0x7f53d81f90e0 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:33.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.322+0000 7f53df200640 1 -- 172.21.15.96:0/3703296326 >> 172.21.15.96:0/3703296326 conn(0x7f53d808bfc0 msgr2=0x7f53d80cf1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:33.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.322+0000 7f53df200640 1 -- 172.21.15.96:0/3703296326 shutdown_connections 2024-05-28T06:17:33.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.322+0000 7f53df200640 1 -- 172.21.15.96:0/3703296326 wait complete. 2024-05-28T06:17:33.326 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:17:33.327+0000 7fea5bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:17:33.326 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:17:33.327+0000 7fea5bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:17:33.326 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:17:33.327+0000 7fea5bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:17:33.326 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:17:33.327+0000 7fea5bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:17:33.326 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:17:33.327+0000 7fea5bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:17:33.339 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone436319 2024-05-28T06:17:33.355 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:17:33.357+0000 7fea5d000640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:17:33.356 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:17:33.357+0000 7fea5d000640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:17:33.356 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:17:33.357+0000 7fea5d000640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:17:33.356 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:17:33.357+0000 7fea5d000640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:17:33.356 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:17:33.357+0000 7fea5d000640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:17:33.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.593+0000 7fc35be00640 1 Processor -- start 2024-05-28T06:17:33.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.593+0000 7fc35be00640 1 -- start start 2024-05-28T06:17:33.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.595+0000 7fc35be00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fc35409b140 0x7fc3540d0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:33.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.595+0000 7fc35be00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc35409b770 0x7fc3540b7620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:33.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.595+0000 7fc35be00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3540cd240 con 0x7fc35415c7a0 2024-05-28T06:17:33.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.595+0000 7fc35be00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fc3540cde50 con 0x7fc35409b140 2024-05-28T06:17:33.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.595+0000 7fc35be00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fc3540b6360 con 0x7fc35409b770 2024-05-28T06:17:33.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.595+0000 7fc359400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fc35415c7a0 0x7fc35409aa30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:57750/0 (socket says 172.21.15.96:57750) 2024-05-28T06:17:33.594 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.595+0000 7fc359400640 1 -- 172.21.15.96:0/2731682106 learned_addr learned my addr 172.21.15.96:0/2731682106 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:17:33.595 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.596+0000 7fc353e00640 1 -- 172.21.15.96:0/2731682106 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2058540445 0 0) 0x7fc3540cd240 con 0x7fc35415c7a0 2024-05-28T06:17:33.595 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.596+0000 7fc353e00640 1 -- 172.21.15.96:0/2731682106 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc33c003520 con 0x7fc35415c7a0 2024-05-28T06:17:33.596 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.596+0000 7fc359e00640 1 --2- 172.21.15.96:0/2731682106 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc35409b770 0x7fc3540b7620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:33.596 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.596+0000 7fc358a00640 1 --2- 172.21.15.96:0/2731682106 >> v2:172.21.15.123:3300/0 conn(0x7fc35409b140 0x7fc3540d0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:33.596 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.596+0000 7fc353e00640 1 -- 172.21.15.96:0/2731682106 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3658746412 0 0) 0x7fc33c003520 con 0x7fc35415c7a0 2024-05-28T06:17:33.596 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.596+0000 7fc353e00640 1 -- 172.21.15.96:0/2731682106 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc35409b770 msgr2=0x7fc3540b7620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:33.596 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.596+0000 7fc353e00640 1 --2- 172.21.15.96:0/2731682106 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc35409b770 0x7fc3540b7620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:33.596 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.596+0000 7fc353e00640 1 -- 172.21.15.96:0/2731682106 >> v2:172.21.15.123:3300/0 conn(0x7fc35409b140 msgr2=0x7fc3540d0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:33.596 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.596+0000 7fc353e00640 1 --2- 172.21.15.96:0/2731682106 >> v2:172.21.15.123:3300/0 conn(0x7fc35409b140 0x7fc3540d0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:33.596 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.596+0000 7fc353e00640 1 -- 172.21.15.96:0/2731682106 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3540cd520 con 0x7fc35415c7a0 2024-05-28T06:17:33.596 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.596+0000 7fc353e00640 1 -- 172.21.15.96:0/2731682106 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7fc348001670 con 0x7fc35415c7a0 2024-05-28T06:17:33.596 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.596+0000 7fc353e00640 1 -- 172.21.15.96:0/2731682106 >> v1:172.21.15.96:6789/0 conn(0x7fc35415c7a0 legacy=0x7fc35409aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:33.596 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.596+0000 7fc353e00640 1 --2- 172.21.15.96:0/2731682106 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc33c0039b0 0x7fc33c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:33.596 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.596+0000 7fc353e00640 1 --2- 172.21.15.96:0/2731682106 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc33c004800 0x7fc33c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:33.596 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.596+0000 7fc353e00640 1 --2- 172.21.15.96:0/2731682106 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc35409b770 0x7fc33c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:33.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.596+0000 7fc353e00640 1 -- 172.21.15.96:0/2731682106 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc3540cde50 con 0x7fc33c0039b0 2024-05-28T06:17:33.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.596+0000 7fc353e00640 1 -- 172.21.15.96:0/2731682106 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc3540b6360 con 0x7fc33c004800 2024-05-28T06:17:33.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.596+0000 7fc353e00640 1 -- 172.21.15.96:0/2731682106 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fc33c00bb50 con 0x7fc35409b770 2024-05-28T06:17:33.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.597+0000 7fc359400640 1 --2- 172.21.15.96:0/2731682106 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc33c0039b0 0x7fc33c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:33.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.597+0000 7fc359400640 1 -- 172.21.15.96:0/2731682106 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc35409b770 msgr2=0x7fc33c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:33.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.597+0000 7fc358a00640 1 --2- 172.21.15.96:0/2731682106 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc33c004800 0x7fc33c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:33.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.597+0000 7fc359e00640 1 --2- 172.21.15.96:0/2731682106 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc35409b770 0x7fc33c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:33.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.597+0000 7fc359400640 1 --2- 172.21.15.96:0/2731682106 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc35409b770 0x7fc33c0051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:33.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.597+0000 7fc359400640 1 -- 172.21.15.96:0/2731682106 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc33c004800 msgr2=0x7fc33c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:33.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.597+0000 7fc359400640 1 --2- 172.21.15.96:0/2731682106 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc33c004800 0x7fc33c004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:33.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.597+0000 7fc359400640 1 -- 172.21.15.96:0/2731682106 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc33c003520 con 0x7fc33c0039b0 2024-05-28T06:17:33.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.598+0000 7fc359400640 1 --2- 172.21.15.96:0/2731682106 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc33c0039b0 0x7fc33c003dd0 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7fc348007fd0 tx=0x7fc33c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4cc46018ae26bb3 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:33.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.599+0000 7fc353e00640 1 -- 172.21.15.96:0/2731682106 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc348007d50 con 0x7fc33c0039b0 2024-05-28T06:17:33.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.599+0000 7fc353e00640 1 -- 172.21.15.96:0/2731682106 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc348005af0 con 0x7fc33c0039b0 2024-05-28T06:17:33.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.599+0000 7fc353e00640 1 -- 172.21.15.96:0/2731682106 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc348007070 con 0x7fc33c0039b0 2024-05-28T06:17:33.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.599+0000 7fc35be00640 1 -- 172.21.15.96:0/2731682106 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc33c0039b0 msgr2=0x7fc33c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:33.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.599+0000 7fc35be00640 1 --2- 172.21.15.96:0/2731682106 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc33c0039b0 0x7fc33c003dd0 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7fc348007fd0 tx=0x7fc33c005830 comp rx=0 tx=0).stop 2024-05-28T06:17:33.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.599+0000 7fc359e00640 1 -- 172.21.15.96:0/2731682106 reap_dead start 2024-05-28T06:17:33.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.599+0000 7fc35be00640 1 -- 172.21.15.96:0/2731682106 shutdown_connections 2024-05-28T06:17:33.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.599+0000 7fc35be00640 1 -- 172.21.15.96:0/2731682106 >> 172.21.15.96:0/2731682106 conn(0x7fc3540b3ff0 msgr2=0x7fc3540b4400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:33.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.599+0000 7fc35be00640 1 -- 172.21.15.96:0/2731682106 shutdown_connections 2024-05-28T06:17:33.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.599+0000 7fc35be00640 1 -- 172.21.15.96:0/2731682106 wait complete. 2024-05-28T06:17:33.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.601+0000 7fc35be00640 1 Processor -- start 2024-05-28T06:17:33.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.601+0000 7fc35be00640 1 -- start start 2024-05-28T06:17:33.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.602+0000 7fc35be00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc33c0039b0 0x7fc3540c5c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:33.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.602+0000 7fc35be00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3540c8ef0 0x7fc3540c6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:33.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.602+0000 7fc35be00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc3540c66c0 0x7fc3540c6b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:33.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.602+0000 7fc35be00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc35415cb80 con 0x7fc33c0039b0 2024-05-28T06:17:33.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.602+0000 7fc35be00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc3540cdae0 con 0x7fc3540c8ef0 2024-05-28T06:17:33.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.602+0000 7fc35be00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fc3540ce2e0 con 0x7fc3540c66c0 2024-05-28T06:17:33.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.602+0000 7fc359400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc33c0039b0 0x7fc3540c5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:33.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.603+0000 7fc359400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc33c0039b0 0x7fc3540c5c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:33802/0 (socket says 172.21.15.96:33802) 2024-05-28T06:17:33.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.603+0000 7fc359400640 1 -- 172.21.15.96:0/3803779904 learned_addr learned my addr 172.21.15.96:0/3803779904 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:17:33.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.603+0000 7fc359e00640 1 --2- 172.21.15.96:0/3803779904 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc3540c66c0 0x7fc3540c6b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:33.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.603+0000 7fc358a00640 1 --2- 172.21.15.96:0/3803779904 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3540c8ef0 0x7fc3540c6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:33.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.603+0000 7fc359400640 1 -- 172.21.15.96:0/3803779904 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc3540c66c0 msgr2=0x7fc3540c6b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:33.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.603+0000 7fc359400640 1 --2- 172.21.15.96:0/3803779904 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc3540c66c0 0x7fc3540c6b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:33.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.603+0000 7fc359400640 1 -- 172.21.15.96:0/3803779904 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3540c8ef0 msgr2=0x7fc3540c6180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:33.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.603+0000 7fc359400640 1 --2- 172.21.15.96:0/3803779904 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3540c8ef0 0x7fc3540c6180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:33.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.603+0000 7fc359400640 1 -- 172.21.15.96:0/3803779904 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc35415ea40 con 0x7fc33c0039b0 2024-05-28T06:17:33.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.603+0000 7fc359400640 1 --2- 172.21.15.96:0/3803779904 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc33c0039b0 0x7fc3540c5c40 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7fc34800c2d0 tx=0x7fc348001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:33.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.604+0000 7fc351600640 1 -- 172.21.15.96:0/3803779904 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc348004bf0 con 0x7fc33c0039b0 2024-05-28T06:17:33.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.604+0000 7fc351600640 1 -- 172.21.15.96:0/3803779904 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc348043070 con 0x7fc33c0039b0 2024-05-28T06:17:33.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.604+0000 7fc351600640 1 -- 172.21.15.96:0/3803779904 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc348007070 con 0x7fc33c0039b0 2024-05-28T06:17:33.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.604+0000 7fc35be00640 1 -- 172.21.15.96:0/3803779904 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc35415ed20 con 0x7fc33c0039b0 2024-05-28T06:17:33.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.604+0000 7fc35be00640 1 -- 172.21.15.96:0/3803779904 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc35415e640 con 0x7fc33c0039b0 2024-05-28T06:17:33.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.605+0000 7fc35be00640 1 -- 172.21.15.96:0/3803779904 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3540689e0 con 0x7fc33c0039b0 2024-05-28T06:17:33.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.606+0000 7fc351600640 1 -- 172.21.15.96:0/3803779904 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 28) ==== 92570+0+0 (secure 0 0 0) 0x7fc348007210 con 0x7fc33c0039b0 2024-05-28T06:17:33.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.607+0000 7fc351600640 1 --2- 172.21.15.96:0/3803779904 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc33406d9d0 0x7fc33406fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:33.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.607+0000 7fc351600640 1 -- 172.21.15.96:0/3803779904 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5795+0+0 (secure 0 0 0) 0x7fc3480aaf80 con 0x7fc33c0039b0 2024-05-28T06:17:33.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.608+0000 7fc358a00640 1 --2- 172.21.15.96:0/3803779904 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc33406d9d0 0x7fc33406fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:33.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.609+0000 7fc358a00640 1 --2- 172.21.15.96:0/3803779904 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc33406d9d0 0x7fc33406fe90 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fc3540b35a0 tx=0x7fc340008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:33.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.611+0000 7fc351600640 1 -- 172.21.15.96:0/3803779904 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc34807af20 con 0x7fc33c0039b0 2024-05-28T06:17:33.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.840+0000 7fc35be00640 1 -- 172.21.15.96:0/3803779904 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone436319", "target": ["mon-mgr", ""]}) -- 0x7fc3540cecf0 con 0x7fc33406d9d0 2024-05-28T06:17:33.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.901+0000 7fc351600640 1 -- 172.21.15.96:0/3803779904 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fc3540cecf0 con 0x7fc33406d9d0 2024-05-28T06:17:33.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.904+0000 7fc35be00640 1 -- 172.21.15.96:0/3803779904 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc33406d9d0 msgr2=0x7fc33406fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:33.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.904+0000 7fc35be00640 1 --2- 172.21.15.96:0/3803779904 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc33406d9d0 0x7fc33406fe90 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fc3540b35a0 tx=0x7fc340008040 comp rx=0 tx=0).stop 2024-05-28T06:17:33.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.904+0000 7fc35be00640 1 -- 172.21.15.96:0/3803779904 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc33c0039b0 msgr2=0x7fc3540c5c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:33.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.904+0000 7fc35be00640 1 --2- 172.21.15.96:0/3803779904 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc33c0039b0 0x7fc3540c5c40 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7fc34800c2d0 tx=0x7fc348001a40 comp rx=0 tx=0).stop 2024-05-28T06:17:33.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.904+0000 7fc35be00640 1 -- 172.21.15.96:0/3803779904 shutdown_connections 2024-05-28T06:17:33.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.904+0000 7fc35be00640 1 --2- 172.21.15.96:0/3803779904 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc33406d9d0 0x7fc33406fe90 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:33.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.904+0000 7fc35be00640 1 --2- 172.21.15.96:0/3803779904 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc3540c66c0 0x7fc3540c6b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:33.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.904+0000 7fc35be00640 1 --2- 172.21.15.96:0/3803779904 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3540c8ef0 0x7fc3540c6180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:33.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.904+0000 7fc35be00640 1 --2- 172.21.15.96:0/3803779904 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc33c0039b0 0x7fc3540c5c40 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:33.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.904+0000 7fc35be00640 1 -- 172.21.15.96:0/3803779904 >> 172.21.15.96:0/3803779904 conn(0x7fc3540b3ff0 msgr2=0x7fc35409aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:33.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.904+0000 7fc35be00640 1 -- 172.21.15.96:0/3803779904 shutdown_connections 2024-05-28T06:17:33.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:33.905+0000 7fc35be00640 1 -- 172.21.15.96:0/3803779904 wait complete. 2024-05-28T06:17:33.922 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone436313 2024-05-28T06:17:34.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.130+0000 7f08d6600640 1 Processor -- start 2024-05-28T06:17:34.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.131+0000 7f08d6600640 1 -- start start 2024-05-28T06:17:34.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.132+0000 7f08d6600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f08d00d34e0 0x7f08d00d4e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:34.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.132+0000 7f08d6600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f08d00b5570 0x7f08d00b5a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:34.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.132+0000 7f08d6600640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08d00d5500 con 0x7f08d01622b0 2024-05-28T06:17:34.131 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.132+0000 7f08d6600640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f08d00b4f50 con 0x7f08d00d34e0 2024-05-28T06:17:34.131 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.132+0000 7f08d6600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f08d00d2720 con 0x7f08d00b5570 2024-05-28T06:17:34.131 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.132+0000 7f08cf400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f08d01622b0 0x7f08d00d1b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:57754/0 (socket says 172.21.15.96:57754) 2024-05-28T06:17:34.131 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.132+0000 7f08cf400640 1 -- 172.21.15.96:0/1178240525 learned_addr learned my addr 172.21.15.96:0/1178240525 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:17:34.133 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.133+0000 7f08cea00640 1 -- 172.21.15.96:0/1178240525 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3811678597 0 0) 0x7f08d00d5500 con 0x7f08d01622b0 2024-05-28T06:17:34.133 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.133+0000 7f08c7e00640 1 --2- 172.21.15.96:0/1178240525 >> v2:172.21.15.123:3300/0 conn(0x7f08d00d34e0 0x7f08d00d4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:34.133 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.133+0000 7f08cea00640 1 -- 172.21.15.96:0/1178240525 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f08b4003520 con 0x7f08d01622b0 2024-05-28T06:17:34.133 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.133+0000 7f08cfe00640 1 --2- 172.21.15.96:0/1178240525 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f08d00b5570 0x7f08d00b5a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:34.133 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.133+0000 7f08cea00640 1 -- 172.21.15.96:0/1178240525 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 560202279 0 0) 0x7f08b4003520 con 0x7f08d01622b0 2024-05-28T06:17:34.133 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.133+0000 7f08cea00640 1 -- 172.21.15.96:0/1178240525 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f08d00b5570 msgr2=0x7f08d00b5a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:34.133 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.133+0000 7f08cea00640 1 --2- 172.21.15.96:0/1178240525 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f08d00b5570 0x7f08d00b5a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:34.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.133+0000 7f08cea00640 1 -- 172.21.15.96:0/1178240525 >> v2:172.21.15.123:3300/0 conn(0x7f08d00d34e0 msgr2=0x7f08d00d4e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:34.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.133+0000 7f08cea00640 1 --2- 172.21.15.96:0/1178240525 >> v2:172.21.15.123:3300/0 conn(0x7f08d00d34e0 0x7f08d00d4e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:34.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.133+0000 7f08cea00640 1 -- 172.21.15.96:0/1178240525 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08d00d2a00 con 0x7f08d01622b0 2024-05-28T06:17:34.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.133+0000 7f08cea00640 1 -- 172.21.15.96:0/1178240525 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f08c0001670 con 0x7f08d01622b0 2024-05-28T06:17:34.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.133+0000 7f08cea00640 1 -- 172.21.15.96:0/1178240525 >> v1:172.21.15.96:6789/0 conn(0x7f08d01622b0 legacy=0x7f08d00d1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:34.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.133+0000 7f08cea00640 1 --2- 172.21.15.96:0/1178240525 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08b40039b0 0x7f08b4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:34.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.133+0000 7f08cea00640 1 --2- 172.21.15.96:0/1178240525 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f08b4004800 0x7f08b4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:34.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.133+0000 7f08cea00640 1 --2- 172.21.15.96:0/1178240525 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f08d00b5570 0x7f08b40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:34.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.133+0000 7f08cea00640 1 -- 172.21.15.96:0/1178240525 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f08d00b4f50 con 0x7f08b4004800 2024-05-28T06:17:34.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.133+0000 7f08cea00640 1 -- 172.21.15.96:0/1178240525 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f08d00d2720 con 0x7f08b40039b0 2024-05-28T06:17:34.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.133+0000 7f08cea00640 1 -- 172.21.15.96:0/1178240525 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f08b400bb20 con 0x7f08d00b5570 2024-05-28T06:17:34.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.134+0000 7f08c7e00640 1 --2- 172.21.15.96:0/1178240525 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f08b4004800 0x7f08b4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:34.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.134+0000 7f08cf400640 1 --2- 172.21.15.96:0/1178240525 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08b40039b0 0x7f08b4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:34.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.134+0000 7f08c7e00640 1 -- 172.21.15.96:0/1178240525 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f08d00b5570 msgr2=0x7f08b40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:34.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.134+0000 7f08c7e00640 1 --2- 172.21.15.96:0/1178240525 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f08d00b5570 0x7f08b40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:34.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.134+0000 7f08c7e00640 1 -- 172.21.15.96:0/1178240525 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08b40039b0 msgr2=0x7f08b4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:34.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.134+0000 7f08c7e00640 1 --2- 172.21.15.96:0/1178240525 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08b40039b0 0x7f08b4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:34.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.134+0000 7f08c7e00640 1 -- 172.21.15.96:0/1178240525 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08b4003520 con 0x7f08b4004800 2024-05-28T06:17:34.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.135+0000 7f08c7e00640 1 --2- 172.21.15.96:0/1178240525 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f08b4004800 0x7f08b4004c80 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f08b800ba00 tx=0x7f08b4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7d00d37b8ea95fd server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:34.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.136+0000 7f08cea00640 1 -- 172.21.15.96:0/1178240525 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08b8030070 con 0x7f08b4004800 2024-05-28T06:17:34.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.136+0000 7f08cea00640 1 -- 172.21.15.96:0/1178240525 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08b802adb0 con 0x7f08b4004800 2024-05-28T06:17:34.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.136+0000 7f08cea00640 1 -- 172.21.15.96:0/1178240525 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08b800b600 con 0x7f08b4004800 2024-05-28T06:17:34.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.136+0000 7f08d6600640 1 -- 172.21.15.96:0/1178240525 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f08b4004800 msgr2=0x7f08b4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:34.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.136+0000 7f08d6600640 1 --2- 172.21.15.96:0/1178240525 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f08b4004800 0x7f08b4004c80 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f08b800ba00 tx=0x7f08b4005830 comp rx=0 tx=0).stop 2024-05-28T06:17:34.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.136+0000 7f08cfe00640 1 -- 172.21.15.96:0/1178240525 reap_dead start 2024-05-28T06:17:34.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.136+0000 7f08d6600640 1 -- 172.21.15.96:0/1178240525 shutdown_connections 2024-05-28T06:17:34.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.136+0000 7f08d6600640 1 -- 172.21.15.96:0/1178240525 >> 172.21.15.96:0/1178240525 conn(0x7f08d008bfc0 msgr2=0x7f08d00d46f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:34.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.136+0000 7f08d6600640 1 -- 172.21.15.96:0/1178240525 shutdown_connections 2024-05-28T06:17:34.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.137+0000 7f08d6600640 1 -- 172.21.15.96:0/1178240525 wait complete. 2024-05-28T06:17:34.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.138+0000 7f08d6600640 1 Processor -- start 2024-05-28T06:17:34.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.138+0000 7f08d6600640 1 -- start start 2024-05-28T06:17:34.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.139+0000 7f08d6600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f08b4004800 0x7f08d0152c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:34.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.139+0000 7f08d6600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f08d00b7730 0x7f08d00c96b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:34.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.139+0000 7f08d6600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08d00b1920 0x7f08d00b1da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:34.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.139+0000 7f08d6600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f08d00b5b80 con 0x7f08d00b7730 2024-05-28T06:17:34.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.139+0000 7f08d6600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f08d00b5a00 con 0x7f08d00b1920 2024-05-28T06:17:34.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.139+0000 7f08d6600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f08d00b5d00 con 0x7f08b4004800 2024-05-28T06:17:34.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.140+0000 7f08c7e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f08d00b7730 0x7f08d00c96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:34.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.140+0000 7f08c7e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f08d00b7730 0x7f08d00c96b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:33818/0 (socket says 172.21.15.96:33818) 2024-05-28T06:17:34.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.140+0000 7f08c7e00640 1 -- 172.21.15.96:0/1392331806 learned_addr learned my addr 172.21.15.96:0/1392331806 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:17:34.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.140+0000 7f08cf400640 1 --2- 172.21.15.96:0/1392331806 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f08b4004800 0x7f08d0152c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:34.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.140+0000 7f08c7e00640 1 -- 172.21.15.96:0/1392331806 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f08b4004800 msgr2=0x7f08d0152c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:34.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.140+0000 7f08cfe00640 1 --2- 172.21.15.96:0/1392331806 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08d00b1920 0x7f08d00b1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:34.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.140+0000 7f08c7e00640 1 --2- 172.21.15.96:0/1392331806 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f08b4004800 0x7f08d0152c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:34.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.140+0000 7f08c7e00640 1 -- 172.21.15.96:0/1392331806 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08d00b1920 msgr2=0x7f08d00b1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:34.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.140+0000 7f08c7e00640 1 --2- 172.21.15.96:0/1392331806 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08d00b1920 0x7f08d00b1da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:34.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.140+0000 7f08c7e00640 1 -- 172.21.15.96:0/1392331806 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08d00c9f90 con 0x7f08d00b7730 2024-05-28T06:17:34.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.140+0000 7f08cf400640 1 --2- 172.21.15.96:0/1392331806 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f08b4004800 0x7f08d0152c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:17:34.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.140+0000 7f08cfe00640 1 --2- 172.21.15.96:0/1392331806 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08d00b1920 0x7f08d00b1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:17:34.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.140+0000 7f08c7e00640 1 --2- 172.21.15.96:0/1392331806 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f08d00b7730 0x7f08d00c96b0 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f08b800bb20 tx=0x7f08b8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:34.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.141+0000 7f08c7400640 1 -- 172.21.15.96:0/1392331806 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08b800b600 con 0x7f08d00b7730 2024-05-28T06:17:34.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.141+0000 7f08c7400640 1 -- 172.21.15.96:0/1392331806 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08b8045070 con 0x7f08d00b7730 2024-05-28T06:17:34.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.141+0000 7f08c7400640 1 -- 172.21.15.96:0/1392331806 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08b803f8a0 con 0x7f08d00b7730 2024-05-28T06:17:34.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.141+0000 7f08d6600640 1 -- 172.21.15.96:0/1392331806 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08d00ca270 con 0x7f08d00b7730 2024-05-28T06:17:34.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.141+0000 7f08d6600640 1 -- 172.21.15.96:0/1392331806 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08d00ca830 con 0x7f08d00b7730 2024-05-28T06:17:34.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.142+0000 7f08d6600640 1 -- 172.21.15.96:0/1392331806 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08d00689e0 con 0x7f08d00b7730 2024-05-28T06:17:34.143 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.143+0000 7f08c7400640 1 -- 172.21.15.96:0/1392331806 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 28) ==== 92570+0+0 (secure 0 0 0) 0x7f08b803fa40 con 0x7f08d00b7730 2024-05-28T06:17:34.143 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.144+0000 7f08c7400640 1 --2- 172.21.15.96:0/1392331806 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f08ac06d9d0 0x7f08ac06fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:34.143 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.144+0000 7f08c7400640 1 -- 172.21.15.96:0/1392331806 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5795+0+0 (secure 0 0 0) 0x7f08b80ab560 con 0x7f08d00b7730 2024-05-28T06:17:34.143 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.145+0000 7f08cf400640 1 --2- 172.21.15.96:0/1392331806 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f08ac06d9d0 0x7f08ac06fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:34.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.146+0000 7f08cf400640 1 --2- 172.21.15.96:0/1392331806 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f08ac06d9d0 0x7f08ac06fe90 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f08c0001050 tx=0x7f08c0001580 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:34.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.149+0000 7f08c7400640 1 -- 172.21.15.96:0/1392331806 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f08b803b340 con 0x7f08d00b7730 2024-05-28T06:17:34.380 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.382+0000 7f08d6600640 1 -- 172.21.15.96:0/1392331806 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone436313", "target": ["mon-mgr", ""]}) -- 0x7f08d008c6c0 con 0x7f08ac06d9d0 2024-05-28T06:17:34.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.440+0000 7f08c7400640 1 -- 172.21.15.96:0/1392331806 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f08d008c6c0 con 0x7f08ac06d9d0 2024-05-28T06:17:34.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.443+0000 7f08d6600640 1 -- 172.21.15.96:0/1392331806 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f08ac06d9d0 msgr2=0x7f08ac06fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:34.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.443+0000 7f08d6600640 1 --2- 172.21.15.96:0/1392331806 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f08ac06d9d0 0x7f08ac06fe90 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f08c0001050 tx=0x7f08c0001580 comp rx=0 tx=0).stop 2024-05-28T06:17:34.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.443+0000 7f08d6600640 1 -- 172.21.15.96:0/1392331806 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f08d00b7730 msgr2=0x7f08d00c96b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:34.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.443+0000 7f08d6600640 1 --2- 172.21.15.96:0/1392331806 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f08d00b7730 0x7f08d00c96b0 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f08b800bb20 tx=0x7f08b8004730 comp rx=0 tx=0).stop 2024-05-28T06:17:34.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.443+0000 7f08d6600640 1 -- 172.21.15.96:0/1392331806 shutdown_connections 2024-05-28T06:17:34.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.443+0000 7f08d6600640 1 --2- 172.21.15.96:0/1392331806 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f08ac06d9d0 0x7f08ac06fe90 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:34.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.443+0000 7f08d6600640 1 --2- 172.21.15.96:0/1392331806 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08d00b1920 0x7f08d00b1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:34.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.443+0000 7f08d6600640 1 --2- 172.21.15.96:0/1392331806 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f08d00b7730 0x7f08d00c96b0 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:34.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.443+0000 7f08d6600640 1 --2- 172.21.15.96:0/1392331806 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f08b4004800 0x7f08d0152c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:34.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.443+0000 7f08d6600640 1 -- 172.21.15.96:0/1392331806 >> 172.21.15.96:0/1392331806 conn(0x7f08d008bfc0 msgr2=0x7f08d006cb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:34.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.444+0000 7f08d6600640 1 -- 172.21.15.96:0/1392331806 shutdown_connections 2024-05-28T06:17:34.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.444+0000 7f08d6600640 1 -- 172.21.15.96:0/1392331806 wait complete. 2024-05-28T06:17:34.466 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone436311 2024-05-28T06:17:34.683 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.683+0000 7f5e39000640 1 Processor -- start 2024-05-28T06:17:34.684 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.683+0000 7f5e39000640 1 -- start start 2024-05-28T06:17:34.684 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.684+0000 7f5e39000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f5e340cea30 0x7f5e3406bce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:34.684 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.684+0000 7f5e39000640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e3406c220 con 0x7f5e3415f5d0 2024-05-28T06:17:34.684 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.685+0000 7f5e39000640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e340c8110 con 0x7f5e3406b900 2024-05-28T06:17:34.684 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.685+0000 7f5e39000640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f5e340cdc70 con 0x7f5e340cea30 2024-05-28T06:17:34.684 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.685+0000 7f5e32800640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f5e3415f5d0 0x7f5e340cefc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:57756/0 (socket says 172.21.15.96:57756) 2024-05-28T06:17:34.684 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.685+0000 7f5e32800640 1 -- 172.21.15.96:0/1572819432 learned_addr learned my addr 172.21.15.96:0/1572819432 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:17:34.684 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.685+0000 7f5e2be00640 1 --2- 172.21.15.96:0/1572819432 >> v2:172.21.15.123:3300/0 conn(0x7f5e340cea30 0x7f5e3406bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:34.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.686+0000 7f5e2be00640 1 -- 172.21.15.96:0/1572819432 >> v1:172.21.15.123:6789/0 conn(0x7f5e3406b900 legacy=0x7f5e340c7950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:34.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.686+0000 7f5e2be00640 1 -- 172.21.15.96:0/1572819432 >> v1:172.21.15.96:6789/0 conn(0x7f5e3415f5d0 legacy=0x7f5e340cefc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:34.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.686+0000 7f5e2be00640 1 -- 172.21.15.96:0/1572819432 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e340cdf50 con 0x7f5e340cea30 2024-05-28T06:17:34.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.688+0000 7f5e2be00640 1 --2- 172.21.15.96:0/1572819432 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e340cea30 0x7f5e3406bce0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c0097d0 tx=0x7f5e1c02a250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3212644f62a66424 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:34.690 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.692+0000 7f5e31400640 1 -- 172.21.15.96:0/1572819432 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e1c037070 con 0x7f5e340cea30 2024-05-28T06:17:34.690 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.693+0000 7f5e31400640 1 -- 172.21.15.96:0/1572819432 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e1c032440 con 0x7f5e340cea30 2024-05-28T06:17:34.691 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.693+0000 7f5e31400640 1 -- 172.21.15.96:0/1572819432 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e1c032810 con 0x7f5e340cea30 2024-05-28T06:17:34.691 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.693+0000 7f5e39000640 1 -- 172.21.15.96:0/1572819432 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e340cea30 msgr2=0x7f5e3406bce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:34.691 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.693+0000 7f5e39000640 1 --2- 172.21.15.96:0/1572819432 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e340cea30 0x7f5e3406bce0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c0097d0 tx=0x7f5e1c02a250 comp rx=0 tx=0).stop 2024-05-28T06:17:34.691 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.693+0000 7f5e39000640 1 -- 172.21.15.96:0/1572819432 shutdown_connections 2024-05-28T06:17:34.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.693+0000 7f5e39000640 1 --2- 172.21.15.96:0/1572819432 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e340cea30 0x7f5e3406bce0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:34.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.693+0000 7f5e39000640 1 -- 172.21.15.96:0/1572819432 >> 172.21.15.96:0/1572819432 conn(0x7f5e3409aa30 msgr2=0x7f5e340b3020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:34.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.694+0000 7f5e39000640 1 -- 172.21.15.96:0/1572819432 shutdown_connections 2024-05-28T06:17:34.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.694+0000 7f5e39000640 1 -- 172.21.15.96:0/1572819432 wait complete. 2024-05-28T06:17:34.694 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.697+0000 7f5e39000640 1 Processor -- start 2024-05-28T06:17:34.695 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.697+0000 7f5e39000640 1 -- start start 2024-05-28T06:17:34.696 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.698+0000 7f5e39000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5e3406b900 0x7f5e3414d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:34.696 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.698+0000 7f5e39000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5e340cea30 0x7f5e3415b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:34.696 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.698+0000 7f5e39000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e3415f5d0 0x7f5e3414dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:34.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.698+0000 7f5e39000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f5e3415fa20 con 0x7f5e3406b900 2024-05-28T06:17:34.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.698+0000 7f5e39000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5e340ce510 con 0x7f5e3415f5d0 2024-05-28T06:17:34.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.698+0000 7f5e39000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f5e3415fba0 con 0x7f5e340cea30 2024-05-28T06:17:34.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.698+0000 7f5e31e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5e3406b900 0x7f5e3414d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:34.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.699+0000 7f5e31e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5e3406b900 0x7f5e3414d300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:33820/0 (socket says 172.21.15.96:33820) 2024-05-28T06:17:34.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.699+0000 7f5e31e00640 1 -- 172.21.15.96:0/771613659 learned_addr learned my addr 172.21.15.96:0/771613659 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:17:34.699 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.699+0000 7f5e2be00640 1 --2- 172.21.15.96:0/771613659 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5e340cea30 0x7f5e3415b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:34.699 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.699+0000 7f5e31e00640 1 -- 172.21.15.96:0/771613659 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5e340cea30 msgr2=0x7f5e3415b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:34.699 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.699+0000 7f5e31e00640 1 --2- 172.21.15.96:0/771613659 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5e340cea30 0x7f5e3415b830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:34.699 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.699+0000 7f5e32800640 1 --2- 172.21.15.96:0/771613659 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e3415f5d0 0x7f5e3414dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:34.699 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.699+0000 7f5e31e00640 1 -- 172.21.15.96:0/771613659 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e3415f5d0 msgr2=0x7f5e3414dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:34.699 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.699+0000 7f5e31e00640 1 --2- 172.21.15.96:0/771613659 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e3415f5d0 0x7f5e3414dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:34.699 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.699+0000 7f5e31e00640 1 -- 172.21.15.96:0/771613659 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e340b6ed0 con 0x7f5e3406b900 2024-05-28T06:17:34.699 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.699+0000 7f5e32800640 1 --2- 172.21.15.96:0/771613659 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e3415f5d0 0x7f5e3414dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:17:34.699 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.699+0000 7f5e31e00640 1 --2- 172.21.15.96:0/771613659 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5e3406b900 0x7f5e3414d300 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f5e2400c480 tx=0x7f5e2400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:34.700 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.700+0000 7f5e2a000640 1 -- 172.21.15.96:0/771613659 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e240052d0 con 0x7f5e3406b900 2024-05-28T06:17:34.700 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.700+0000 7f5e2a000640 1 -- 172.21.15.96:0/771613659 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e24005470 con 0x7f5e3406b900 2024-05-28T06:17:34.700 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.700+0000 7f5e39000640 1 -- 172.21.15.96:0/771613659 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e3415c360 con 0x7f5e3406b900 2024-05-28T06:17:34.700 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.700+0000 7f5e2a000640 1 -- 172.21.15.96:0/771613659 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e24012870 con 0x7f5e3406b900 2024-05-28T06:17:34.700 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.700+0000 7f5e39000640 1 -- 172.21.15.96:0/771613659 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e340b4d90 con 0x7f5e3406b900 2024-05-28T06:17:34.700 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.701+0000 7f5e39000640 1 -- 172.21.15.96:0/771613659 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e340d0970 con 0x7f5e3406b900 2024-05-28T06:17:34.700 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.702+0000 7f5e2a000640 1 -- 172.21.15.96:0/771613659 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 29) ==== 92672+0+0 (secure 0 0 0) 0x7f5e24012a10 con 0x7f5e3406b900 2024-05-28T06:17:34.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.704+0000 7f5e2a000640 1 --2- 172.21.15.96:0/771613659 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5e1006daf0 0x7f5e1006ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:34.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.704+0000 7f5e2a000640 1 -- 172.21.15.96:0/771613659 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5795+0+0 (secure 0 0 0) 0x7f5e2408a710 con 0x7f5e3406b900 2024-05-28T06:17:34.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.709+0000 7f5e2a000640 1 -- 172.21.15.96:0/771613659 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5e340d0970 con 0x7f5e3406b900 2024-05-28T06:17:34.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.710+0000 7f5e2be00640 1 --2- 172.21.15.96:0/771613659 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5e1006daf0 0x7f5e1006ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:34.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.711+0000 7f5e2be00640 1 --2- 172.21.15.96:0/771613659 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5e1006daf0 0x7f5e1006ffb0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c034000 tx=0x7f5e1c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:34.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:34.941+0000 7f5e39000640 1 -- 172.21.15.96:0/771613659 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone436311", "target": ["mon-mgr", ""]}) -- 0x7f5e3414d840 con 0x7f5e1006daf0 2024-05-28T06:17:36.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.450+0000 7f5e2a000640 1 -- 172.21.15.96:0/771613659 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f5e3414d840 con 0x7f5e1006daf0 2024-05-28T06:17:36.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.454+0000 7f5e39000640 1 -- 172.21.15.96:0/771613659 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5e1006daf0 msgr2=0x7f5e1006ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:36.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.454+0000 7f5e39000640 1 --2- 172.21.15.96:0/771613659 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5e1006daf0 0x7f5e1006ffb0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c034000 tx=0x7f5e1c02e000 comp rx=0 tx=0).stop 2024-05-28T06:17:36.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.454+0000 7f5e39000640 1 -- 172.21.15.96:0/771613659 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5e3406b900 msgr2=0x7f5e3414d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:36.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.454+0000 7f5e39000640 1 --2- 172.21.15.96:0/771613659 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5e3406b900 0x7f5e3414d300 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f5e2400c480 tx=0x7f5e2400c940 comp rx=0 tx=0).stop 2024-05-28T06:17:36.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.455+0000 7f5e39000640 1 -- 172.21.15.96:0/771613659 shutdown_connections 2024-05-28T06:17:36.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.455+0000 7f5e39000640 1 --2- 172.21.15.96:0/771613659 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5e1006daf0 0x7f5e1006ffb0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:36.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.455+0000 7f5e39000640 1 --2- 172.21.15.96:0/771613659 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e3415f5d0 0x7f5e3414dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:36.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.455+0000 7f5e39000640 1 --2- 172.21.15.96:0/771613659 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5e340cea30 0x7f5e3415b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:36.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.455+0000 7f5e39000640 1 --2- 172.21.15.96:0/771613659 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5e3406b900 0x7f5e3414d300 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:36.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.455+0000 7f5e39000640 1 -- 172.21.15.96:0/771613659 >> 172.21.15.96:0/771613659 conn(0x7f5e3409aa30 msgr2=0x7f5e3406fbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:36.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.455+0000 7f5e39000640 1 -- 172.21.15.96:0/771613659 shutdown_connections 2024-05-28T06:17:36.454 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.455+0000 7f5e39000640 1 -- 172.21.15.96:0/771613659 wait complete. 2024-05-28T06:17:36.477 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone436320 2024-05-28T06:17:36.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.694+0000 7fd73de00640 1 Processor -- start 2024-05-28T06:17:36.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.694+0000 7fd73de00640 1 -- start start 2024-05-28T06:17:36.693 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.696+0000 7fd73de00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd73815e830 0x7fd73806dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:36.694 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.696+0000 7fd73de00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fd738072310 0x7fd73806f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:36.694 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.696+0000 7fd73de00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd73806e190 con 0x7fd7380b3510 2024-05-28T06:17:36.694 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.696+0000 7fd73de00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fd73806f770 con 0x7fd738072310 2024-05-28T06:17:36.694 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.696+0000 7fd73de00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fd73806e470 con 0x7fd73815e830 2024-05-28T06:17:36.694 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.696+0000 7fd72e000640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fd7380b3510 0x7fd738071c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:57766/0 (socket says 172.21.15.96:57766) 2024-05-28T06:17:36.694 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.696+0000 7fd72e000640 1 -- 172.21.15.96:0/1017585201 learned_addr learned my addr 172.21.15.96:0/1017585201 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:17:36.695 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.696+0000 7fd736000640 1 -- 172.21.15.96:0/1017585201 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 632142108 0 0) 0x7fd73806e190 con 0x7fd7380b3510 2024-05-28T06:17:36.695 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.696+0000 7fd736a00640 1 --2- 172.21.15.96:0/1017585201 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd73815e830 0x7fd73806dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:36.695 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.696+0000 7fd736000640 1 -- 172.21.15.96:0/1017585201 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd71c003520 con 0x7fd7380b3510 2024-05-28T06:17:36.695 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.696+0000 7fd737400640 1 --2- 172.21.15.96:0/1017585201 >> v2:172.21.15.123:3300/0 conn(0x7fd738072310 0x7fd73806f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:36.695 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.697+0000 7fd736000640 1 -- 172.21.15.96:0/1017585201 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1433388760 0 0) 0x7fd71c003520 con 0x7fd7380b3510 2024-05-28T06:17:36.695 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.697+0000 7fd736a00640 1 -- 172.21.15.96:0/1017585201 >> v2:172.21.15.123:3300/0 conn(0x7fd738072310 msgr2=0x7fd73806f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:36.695 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.697+0000 7fd736a00640 1 --2- 172.21.15.96:0/1017585201 >> v2:172.21.15.123:3300/0 conn(0x7fd738072310 0x7fd73806f230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:36.696 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.697+0000 7fd736a00640 1 -- 172.21.15.96:0/1017585201 >> v1:172.21.15.96:6789/0 conn(0x7fd7380b3510 legacy=0x7fd738071c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:36.696 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.697+0000 7fd736a00640 1 -- 172.21.15.96:0/1017585201 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd73806e750 con 0x7fd73815e830 2024-05-28T06:17:36.696 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.698+0000 7fd736a00640 1 --2- 172.21.15.96:0/1017585201 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd73815e830 0x7fd73806dbb0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fd7280097a0 tx=0x7fd72802a110 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d02d2f6a4c569a50 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:36.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.703+0000 7fd736000640 1 -- 172.21.15.96:0/1017585201 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd728037070 con 0x7fd73815e830 2024-05-28T06:17:36.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.703+0000 7fd736000640 1 -- 172.21.15.96:0/1017585201 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd72802aba0 con 0x7fd73815e830 2024-05-28T06:17:36.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.703+0000 7fd736000640 1 -- 172.21.15.96:0/1017585201 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd720001670 con 0x7fd73815e830 2024-05-28T06:17:36.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.704+0000 7fd73de00640 1 -- 172.21.15.96:0/1017585201 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd73815e830 msgr2=0x7fd73806dbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:36.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.704+0000 7fd73de00640 1 --2- 172.21.15.96:0/1017585201 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd73815e830 0x7fd73806dbb0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fd7280097a0 tx=0x7fd72802a110 comp rx=0 tx=0).stop 2024-05-28T06:17:36.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.704+0000 7fd73de00640 1 -- 172.21.15.96:0/1017585201 shutdown_connections 2024-05-28T06:17:36.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.705+0000 7fd73de00640 1 --2- 172.21.15.96:0/1017585201 >> v2:172.21.15.123:3300/0 conn(0x7fd738072310 0x7fd73806f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:36.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.705+0000 7fd73de00640 1 --2- 172.21.15.96:0/1017585201 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd73815e830 0x7fd73806dbb0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:36.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.705+0000 7fd73de00640 1 -- 172.21.15.96:0/1017585201 >> 172.21.15.96:0/1017585201 conn(0x7fd7380b1e50 msgr2=0x7fd7380b2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:36.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.705+0000 7fd73de00640 1 -- 172.21.15.96:0/1017585201 shutdown_connections 2024-05-28T06:17:36.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.705+0000 7fd73de00640 1 -- 172.21.15.96:0/1017585201 wait complete. 2024-05-28T06:17:36.705 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.707+0000 7fd73de00640 1 Processor -- start 2024-05-28T06:17:36.705 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.707+0000 7fd73de00640 1 -- start start 2024-05-28T06:17:36.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.708+0000 7fd73de00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd738072310 0x7fd73815a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:36.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.708+0000 7fd73de00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd7380b3510 0x7fd73809c390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:36.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.708+0000 7fd73de00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd73815e830 0x7fd73809c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:36.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.708+0000 7fd73de00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd73815ec10 con 0x7fd7380b3510 2024-05-28T06:17:36.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.708+0000 7fd73de00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd73806ed10 con 0x7fd738072310 2024-05-28T06:17:36.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.708+0000 7fd73de00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fd7380728c0 con 0x7fd73815e830 2024-05-28T06:17:36.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.709+0000 7fd72e000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd7380b3510 0x7fd73809c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:36.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.709+0000 7fd72e000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd7380b3510 0x7fd73809c390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:33822/0 (socket says 172.21.15.96:33822) 2024-05-28T06:17:36.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.709+0000 7fd72e000640 1 -- 172.21.15.96:0/2472380756 learned_addr learned my addr 172.21.15.96:0/2472380756 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:17:36.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.709+0000 7fd737400640 1 --2- 172.21.15.96:0/2472380756 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd73815e830 0x7fd73809c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:36.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.709+0000 7fd736a00640 1 --2- 172.21.15.96:0/2472380756 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd738072310 0x7fd73815a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:36.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.709+0000 7fd72e000640 1 -- 172.21.15.96:0/2472380756 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd73815e830 msgr2=0x7fd73809c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:36.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.709+0000 7fd72e000640 1 --2- 172.21.15.96:0/2472380756 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd73815e830 0x7fd73809c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:36.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.709+0000 7fd72e000640 1 -- 172.21.15.96:0/2472380756 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd738072310 msgr2=0x7fd73815a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:36.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.709+0000 7fd72e000640 1 --2- 172.21.15.96:0/2472380756 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd738072310 0x7fd73815a910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:36.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.709+0000 7fd72e000640 1 -- 172.21.15.96:0/2472380756 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7381f43f0 con 0x7fd7380b3510 2024-05-28T06:17:36.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.710+0000 7fd72e000640 1 --2- 172.21.15.96:0/2472380756 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd7380b3510 0x7fd73809c390 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7fd7200011e0 tx=0x7fd72000cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:36.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.710+0000 7fd72fe00640 1 -- 172.21.15.96:0/2472380756 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd720017070 con 0x7fd7380b3510 2024-05-28T06:17:36.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.710+0000 7fd72fe00640 1 -- 172.21.15.96:0/2472380756 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd720005ad0 con 0x7fd7380b3510 2024-05-28T06:17:36.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.710+0000 7fd72fe00640 1 -- 172.21.15.96:0/2472380756 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd720007070 con 0x7fd7380b3510 2024-05-28T06:17:36.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.710+0000 7fd73de00640 1 -- 172.21.15.96:0/2472380756 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7380afd70 con 0x7fd7380b3510 2024-05-28T06:17:36.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.710+0000 7fd73de00640 1 -- 172.21.15.96:0/2472380756 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7380b0330 con 0x7fd7380b3510 2024-05-28T06:17:36.710 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.711+0000 7fd73de00640 1 -- 172.21.15.96:0/2472380756 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7380689e0 con 0x7fd7380b3510 2024-05-28T06:17:36.710 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.712+0000 7fd72fe00640 1 -- 172.21.15.96:0/2472380756 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 29) ==== 92672+0+0 (secure 0 0 0) 0x7fd72001a070 con 0x7fd7380b3510 2024-05-28T06:17:36.711 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.713+0000 7fd72fe00640 1 --2- 172.21.15.96:0/2472380756 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd71406daf0 0x7fd71406ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:36.711 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.713+0000 7fd72fe00640 1 -- 172.21.15.96:0/2472380756 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5795+0+0 (secure 0 0 0) 0x7fd720089fd0 con 0x7fd7380b3510 2024-05-28T06:17:36.711 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.714+0000 7fd736a00640 1 --2- 172.21.15.96:0/2472380756 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd71406daf0 0x7fd71406ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:36.712 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.715+0000 7fd736a00640 1 --2- 172.21.15.96:0/2472380756 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd71406daf0 0x7fd71406ffb0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fd728034000 tx=0x7fd72802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:36.715 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.718+0000 7fd72fe00640 1 -- 172.21.15.96:0/2472380756 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd720059ae0 con 0x7fd7380b3510 2024-05-28T06:17:36.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.935+0000 7fd73de00640 1 -- 172.21.15.96:0/2472380756 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone436320", "target": ["mon-mgr", ""]}) -- 0x7fd7381621f0 con 0x7fd71406daf0 2024-05-28T06:17:36.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.986+0000 7fd72fe00640 1 -- 172.21.15.96:0/2472380756 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fd7381621f0 con 0x7fd71406daf0 2024-05-28T06:17:36.987 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.989+0000 7fd73de00640 1 -- 172.21.15.96:0/2472380756 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd71406daf0 msgr2=0x7fd71406ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:36.987 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.989+0000 7fd73de00640 1 --2- 172.21.15.96:0/2472380756 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd71406daf0 0x7fd71406ffb0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fd728034000 tx=0x7fd72802e000 comp rx=0 tx=0).stop 2024-05-28T06:17:36.987 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.989+0000 7fd73de00640 1 -- 172.21.15.96:0/2472380756 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd7380b3510 msgr2=0x7fd73809c390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:36.987 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.989+0000 7fd73de00640 1 --2- 172.21.15.96:0/2472380756 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd7380b3510 0x7fd73809c390 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7fd7200011e0 tx=0x7fd72000cbe0 comp rx=0 tx=0).stop 2024-05-28T06:17:36.987 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.989+0000 7fd73de00640 1 -- 172.21.15.96:0/2472380756 shutdown_connections 2024-05-28T06:17:36.987 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.989+0000 7fd73de00640 1 --2- 172.21.15.96:0/2472380756 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd71406daf0 0x7fd71406ffb0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:36.987 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.989+0000 7fd73de00640 1 --2- 172.21.15.96:0/2472380756 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd73815e830 0x7fd73809c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:36.988 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.989+0000 7fd73de00640 1 --2- 172.21.15.96:0/2472380756 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd7380b3510 0x7fd73809c390 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:36.988 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.989+0000 7fd73de00640 1 --2- 172.21.15.96:0/2472380756 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd738072310 0x7fd73815a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:36.988 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.989+0000 7fd73de00640 1 -- 172.21.15.96:0/2472380756 >> 172.21.15.96:0/2472380756 conn(0x7fd7380b1e50 msgr2=0x7fd73809d3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:36.988 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.989+0000 7fd73de00640 1 -- 172.21.15.96:0/2472380756 shutdown_connections 2024-05-28T06:17:36.988 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:36.989+0000 7fd73de00640 1 -- 172.21.15.96:0/2472380756 wait complete. 2024-05-28T06:17:37.005 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot ls cephfs sv1 --format json 2024-05-28T06:17:37.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.214+0000 7f679ec00640 1 Processor -- start 2024-05-28T06:17:37.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.214+0000 7f679ec00640 1 -- start start 2024-05-28T06:17:37.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.215+0000 7f679ec00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f679815b340 0x7f679806f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:37.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.215+0000 7f679ec00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f679815c290 0x7f67980b3300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:37.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.215+0000 7f679ec00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f679806fa90 con 0x7f679815b720 2024-05-28T06:17:37.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.215+0000 7f679ec00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f67980af9b0 con 0x7f679815c290 2024-05-28T06:17:37.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.215+0000 7f679ec00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f67980aeda0 con 0x7f679815b340 2024-05-28T06:17:37.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.216+0000 7f678f400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f679815b720 0x7f679815bb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:57782/0 (socket says 172.21.15.96:57782) 2024-05-28T06:17:37.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.216+0000 7f678f400640 1 -- 172.21.15.96:0/655090999 learned_addr learned my addr 172.21.15.96:0/655090999 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:17:37.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.216+0000 7f679cc00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f679815c290 0x7f67980b3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:37.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.216+0000 7f6797e00640 1 --2- 172.21.15.96:0/655090999 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f679815b340 0x7f679806f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:37.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.216+0000 7f679cc00640 1 -- 172.21.15.96:0/655090999 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f679815b340 msgr2=0x7f679806f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:37.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.216+0000 7f679cc00640 1 --2- 172.21.15.96:0/655090999 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f679815b340 0x7f679806f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:37.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.216+0000 7f679cc00640 1 -- 172.21.15.96:0/655090999 >> v1:172.21.15.96:6789/0 conn(0x7f679815b720 legacy=0x7f679815bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:37.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.216+0000 7f679cc00640 1 -- 172.21.15.96:0/655090999 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67980af080 con 0x7f679815c290 2024-05-28T06:17:37.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.216+0000 7f6797400640 1 -- 172.21.15.96:0/655090999 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 614340944 0 0) 0x7f679806fa90 con 0x7f679815b720 2024-05-28T06:17:37.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.216+0000 7f6797e00640 1 --2- 172.21.15.96:0/655090999 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f679815b340 0x7f679806f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:17:37.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.217+0000 7f679cc00640 1 --2- 172.21.15.96:0/655090999 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f679815c290 0x7f67980b3300 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f6790009d90 tx=0x7f679002a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5096c45c7212368 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:37.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.218+0000 7f6797400640 1 -- 172.21.15.96:0/655090999 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6790007740 con 0x7f679815c290 2024-05-28T06:17:37.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.218+0000 7f6797400640 1 -- 172.21.15.96:0/655090999 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67900078e0 con 0x7f679815c290 2024-05-28T06:17:37.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.218+0000 7f6797400640 1 -- 172.21.15.96:0/655090999 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6790007cb0 con 0x7f679815c290 2024-05-28T06:17:37.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.218+0000 7f679ec00640 1 -- 172.21.15.96:0/655090999 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f679815c290 msgr2=0x7f67980b3300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:37.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.218+0000 7f679ec00640 1 --2- 172.21.15.96:0/655090999 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f679815c290 0x7f67980b3300 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f6790009d90 tx=0x7f679002a660 comp rx=0 tx=0).stop 2024-05-28T06:17:37.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.218+0000 7f679ec00640 1 -- 172.21.15.96:0/655090999 shutdown_connections 2024-05-28T06:17:37.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.218+0000 7f679ec00640 1 --2- 172.21.15.96:0/655090999 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f679815c290 0x7f67980b3300 unknown :-1 s=CLOSED pgs=1259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:37.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.218+0000 7f679ec00640 1 --2- 172.21.15.96:0/655090999 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f679815b340 0x7f679806f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:37.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.218+0000 7f679ec00640 1 -- 172.21.15.96:0/655090999 >> 172.21.15.96:0/655090999 conn(0x7f67980b1e50 msgr2=0x7f67980b2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:37.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.218+0000 7f679ec00640 1 -- 172.21.15.96:0/655090999 shutdown_connections 2024-05-28T06:17:37.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.218+0000 7f679ec00640 1 -- 172.21.15.96:0/655090999 wait complete. 2024-05-28T06:17:37.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.220+0000 7f679ec00640 1 Processor -- start 2024-05-28T06:17:37.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.220+0000 7f679ec00640 1 -- start start 2024-05-28T06:17:37.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.221+0000 7f679ec00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f679815b340 0x7f67981579c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:37.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.221+0000 7f679ec00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f679815b720 0x7f67981596a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:37.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.221+0000 7f679ec00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f679815c290 0x7f6798159be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:37.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.221+0000 7f679ec00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f67980aea30 con 0x7f679815c290 2024-05-28T06:17:37.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.221+0000 7f679ec00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f67980af640 con 0x7f679815b720 2024-05-28T06:17:37.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.221+0000 7f679ec00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f679815c8c0 con 0x7f679815b340 2024-05-28T06:17:37.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.222+0000 7f679cc00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f679815c290 0x7f6798159be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:37.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.222+0000 7f679cc00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f679815c290 0x7f6798159be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:33842/0 (socket says 172.21.15.96:33842) 2024-05-28T06:17:37.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.222+0000 7f679cc00640 1 -- 172.21.15.96:0/4286168998 learned_addr learned my addr 172.21.15.96:0/4286168998 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:17:37.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.222+0000 7f678f400640 1 --2- 172.21.15.96:0/4286168998 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f679815b720 0x7f67981596a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:37.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.222+0000 7f679cc00640 1 -- 172.21.15.96:0/4286168998 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f679815b340 msgr2=0x7f67981579c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:37.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.222+0000 7f679cc00640 1 --2- 172.21.15.96:0/4286168998 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f679815b340 0x7f67981579c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:37.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.222+0000 7f679cc00640 1 -- 172.21.15.96:0/4286168998 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f679815b720 msgr2=0x7f67981596a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:37.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.222+0000 7f679cc00640 1 --2- 172.21.15.96:0/4286168998 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f679815b720 0x7f67981596a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:37.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.222+0000 7f679cc00640 1 -- 172.21.15.96:0/4286168998 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f679814e620 con 0x7f679815c290 2024-05-28T06:17:37.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.223+0000 7f679cc00640 1 --2- 172.21.15.96:0/4286168998 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f679815c290 0x7f6798159be0 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f6790009d90 tx=0x7f6790004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:37.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.223+0000 7f6794c00640 1 -- 172.21.15.96:0/4286168998 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f679003f070 con 0x7f679815c290 2024-05-28T06:17:37.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.223+0000 7f6794c00640 1 -- 172.21.15.96:0/4286168998 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6790031bc0 con 0x7f679815c290 2024-05-28T06:17:37.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.223+0000 7f679ec00640 1 -- 172.21.15.96:0/4286168998 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f679815a400 con 0x7f679815c290 2024-05-28T06:17:37.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.223+0000 7f6794c00640 1 -- 172.21.15.96:0/4286168998 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6790003990 con 0x7f679815c290 2024-05-28T06:17:37.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.223+0000 7f679ec00640 1 -- 172.21.15.96:0/4286168998 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f679815a9c0 con 0x7f679815c290 2024-05-28T06:17:37.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.225+0000 7f679ec00640 1 -- 172.21.15.96:0/4286168998 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67980689e0 con 0x7f679815c290 2024-05-28T06:17:37.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.225+0000 7f6794c00640 1 -- 172.21.15.96:0/4286168998 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 29) ==== 92672+0+0 (secure 0 0 0) 0x7f6790003b30 con 0x7f679815c290 2024-05-28T06:17:37.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.226+0000 7f6794c00640 1 --2- 172.21.15.96:0/4286168998 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f677406daf0 0x7f677406ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:37.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.226+0000 7f6794c00640 1 -- 172.21.15.96:0/4286168998 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5795+0+0 (secure 0 0 0) 0x7f67900abb00 con 0x7f679815c290 2024-05-28T06:17:37.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.227+0000 7f6797e00640 1 --2- 172.21.15.96:0/4286168998 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f677406daf0 0x7f677406ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:37.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.231+0000 7f6797e00640 1 --2- 172.21.15.96:0/4286168998 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f677406daf0 0x7f677406ffb0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f6788006fd0 tx=0x7f6788008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:37.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.232+0000 7f6794c00640 1 -- 172.21.15.96:0/4286168998 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f679007b610 con 0x7f679815c290 2024-05-28T06:17:37.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.537+0000 7f679ec00640 1 -- 172.21.15.96:0/4286168998 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot ls", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f67980b4e60 con 0x7f677406daf0 2024-05-28T06:17:37.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.573+0000 7f6794c00640 1 -- 172.21.15.96:0/4286168998 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f67980b4e60 con 0x7f677406daf0 2024-05-28T06:17:37.571 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:17:37.571 INFO:teuthology.orchestra.run.smithi096.stdout:[ 2024-05-28T06:17:37.571 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:17:37.571 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "ss1" 2024-05-28T06:17:37.571 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:17:37.571 INFO:teuthology.orchestra.run.smithi096.stdout:] 2024-05-28T06:17:37.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.575+0000 7f679ec00640 1 -- 172.21.15.96:0/4286168998 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f677406daf0 msgr2=0x7f677406ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:37.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.575+0000 7f679ec00640 1 --2- 172.21.15.96:0/4286168998 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f677406daf0 0x7f677406ffb0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f6788006fd0 tx=0x7f6788008040 comp rx=0 tx=0).stop 2024-05-28T06:17:37.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.575+0000 7f679ec00640 1 -- 172.21.15.96:0/4286168998 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f679815c290 msgr2=0x7f6798159be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:37.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.575+0000 7f679ec00640 1 --2- 172.21.15.96:0/4286168998 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f679815c290 0x7f6798159be0 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f6790009d90 tx=0x7f6790004a40 comp rx=0 tx=0).stop 2024-05-28T06:17:37.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.575+0000 7f679ec00640 1 -- 172.21.15.96:0/4286168998 shutdown_connections 2024-05-28T06:17:37.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.575+0000 7f679ec00640 1 --2- 172.21.15.96:0/4286168998 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f677406daf0 0x7f677406ffb0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:37.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.575+0000 7f679ec00640 1 --2- 172.21.15.96:0/4286168998 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f679815c290 0x7f6798159be0 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:37.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.575+0000 7f679ec00640 1 --2- 172.21.15.96:0/4286168998 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f679815b720 0x7f67981596a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:37.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.575+0000 7f679ec00640 1 --2- 172.21.15.96:0/4286168998 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f679815b340 0x7f67981579c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:37.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.575+0000 7f679ec00640 1 -- 172.21.15.96:0/4286168998 >> 172.21.15.96:0/4286168998 conn(0x7f67980b1e50 msgr2=0x7f67981538d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:37.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.575+0000 7f679ec00640 1 -- 172.21.15.96:0/4286168998 shutdown_connections 2024-05-28T06:17:37.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.575+0000 7f679ec00640 1 -- 172.21.15.96:0/4286168998 wait complete. 2024-05-28T06:17:37.592 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-05-28T06:17:37.787 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.789+0000 7fc75ca00640 1 Processor -- start 2024-05-28T06:17:37.787 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.789+0000 7fc75ca00640 1 -- start start 2024-05-28T06:17:37.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.790+0000 7fc75ca00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fc7580d0640 0x7fc7580d0a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:37.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.790+0000 7fc75ca00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fc75815b6f0 0x7fc75806e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:37.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.790+0000 7fc75ca00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7580ade90 con 0x7fc7580725e0 2024-05-28T06:17:37.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.790+0000 7fc75ca00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fc758071c40 con 0x7fc7580d0640 2024-05-28T06:17:37.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.790+0000 7fc75ca00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fc7580d0f60 con 0x7fc75815b6f0 2024-05-28T06:17:37.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.791+0000 7fc755e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fc7580d0640 0x7fc7580d0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:37.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.791+0000 7fc756800640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fc7580725e0 0x7fc7580b44d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:57790/0 (socket says 172.21.15.96:57790) 2024-05-28T06:17:37.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.791+0000 7fc756800640 1 -- 172.21.15.96:0/2739982650 learned_addr learned my addr 172.21.15.96:0/2739982650 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:17:37.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.791+0000 7fc755400640 1 --2- 172.21.15.96:0/2739982650 >> v2:172.21.15.123:3300/0 conn(0x7fc75815b6f0 0x7fc75806e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:37.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.791+0000 7fc755e00640 1 -- 172.21.15.96:0/2739982650 >> v2:172.21.15.123:3300/0 conn(0x7fc75815b6f0 msgr2=0x7fc75806e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:37.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.791+0000 7fc755e00640 1 --2- 172.21.15.96:0/2739982650 >> v2:172.21.15.123:3300/0 conn(0x7fc75815b6f0 0x7fc75806e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:37.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.791+0000 7fc755e00640 1 -- 172.21.15.96:0/2739982650 >> v1:172.21.15.96:6789/0 conn(0x7fc7580725e0 legacy=0x7fc7580b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:37.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.791+0000 7fc755e00640 1 -- 172.21.15.96:0/2739982650 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc758071e70 con 0x7fc7580d0640 2024-05-28T06:17:37.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.791+0000 7fc755400640 1 --2- 172.21.15.96:0/2739982650 >> v2:172.21.15.123:3300/0 conn(0x7fc75815b6f0 0x7fc75806e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:17:37.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.791+0000 7fc754a00640 1 -- 172.21.15.96:0/2739982650 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3704541292 0 0) 0x7fc7580ade90 con 0x7fc7580725e0 2024-05-28T06:17:37.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.792+0000 7fc755e00640 1 --2- 172.21.15.96:0/2739982650 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc7580d0640 0x7fc7580d0a20 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7fc7580d0220 tx=0x7fc74802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=490efdbbca2d9c3c server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:37.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.792+0000 7fc754a00640 1 -- 172.21.15.96:0/2739982650 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc748037070 con 0x7fc7580d0640 2024-05-28T06:17:37.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.792+0000 7fc754a00640 1 -- 172.21.15.96:0/2739982650 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc748032440 con 0x7fc7580d0640 2024-05-28T06:17:37.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.792+0000 7fc754a00640 1 -- 172.21.15.96:0/2739982650 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc748032810 con 0x7fc7580d0640 2024-05-28T06:17:37.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.793+0000 7fc75ca00640 1 -- 172.21.15.96:0/2739982650 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc7580d0640 msgr2=0x7fc7580d0a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:37.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.793+0000 7fc75ca00640 1 --2- 172.21.15.96:0/2739982650 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc7580d0640 0x7fc7580d0a20 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7fc7580d0220 tx=0x7fc74802a1a0 comp rx=0 tx=0).stop 2024-05-28T06:17:37.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.793+0000 7fc75ca00640 1 -- 172.21.15.96:0/2739982650 shutdown_connections 2024-05-28T06:17:37.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.793+0000 7fc75ca00640 1 --2- 172.21.15.96:0/2739982650 >> v2:172.21.15.123:3300/0 conn(0x7fc75815b6f0 0x7fc75806e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:37.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.793+0000 7fc75ca00640 1 --2- 172.21.15.96:0/2739982650 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc7580d0640 0x7fc7580d0a20 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:37.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.793+0000 7fc75ca00640 1 -- 172.21.15.96:0/2739982650 >> 172.21.15.96:0/2739982650 conn(0x7fc75806bd20 msgr2=0x7fc75806c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:37.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.793+0000 7fc75ca00640 1 -- 172.21.15.96:0/2739982650 shutdown_connections 2024-05-28T06:17:37.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.793+0000 7fc75ca00640 1 -- 172.21.15.96:0/2739982650 wait complete. 2024-05-28T06:17:37.793 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.795+0000 7fc75ca00640 1 Processor -- start 2024-05-28T06:17:37.793 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.795+0000 7fc75ca00640 1 -- start start 2024-05-28T06:17:37.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.796+0000 7fc75ca00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc7580725e0 0x7fc7581593a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:37.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.796+0000 7fc75ca00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc7580d0640 0x7fc75815cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:37.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.796+0000 7fc75ca00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc75815b6f0 0x7fc75815d050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:37.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.796+0000 7fc75ca00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc75806e890 con 0x7fc75815b6f0 2024-05-28T06:17:37.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.796+0000 7fc75ca00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc7580ae130 con 0x7fc7580d0640 2024-05-28T06:17:37.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.796+0000 7fc75ca00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fc7580af660 con 0x7fc7580725e0 2024-05-28T06:17:37.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.796+0000 7fc756800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc75815b6f0 0x7fc75815d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:37.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.796+0000 7fc756800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc75815b6f0 0x7fc75815d050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:33862/0 (socket says 172.21.15.96:33862) 2024-05-28T06:17:37.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.796+0000 7fc756800640 1 -- 172.21.15.96:0/3758939090 learned_addr learned my addr 172.21.15.96:0/3758939090 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:17:37.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.797+0000 7fc756800640 1 -- 172.21.15.96:0/3758939090 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc7580725e0 msgr2=0x7fc7581593a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:37.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.797+0000 7fc755e00640 1 --2- 172.21.15.96:0/3758939090 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc7580725e0 0x7fc7581593a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:37.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.797+0000 7fc755400640 1 --2- 172.21.15.96:0/3758939090 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc7580d0640 0x7fc75815cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:37.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.797+0000 7fc756800640 1 --2- 172.21.15.96:0/3758939090 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc7580725e0 0x7fc7581593a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:37.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.797+0000 7fc756800640 1 -- 172.21.15.96:0/3758939090 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc7580d0640 msgr2=0x7fc75815cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:37.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.797+0000 7fc756800640 1 --2- 172.21.15.96:0/3758939090 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc7580d0640 0x7fc75815cb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:37.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.797+0000 7fc756800640 1 -- 172.21.15.96:0/3758939090 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc758145900 con 0x7fc75815b6f0 2024-05-28T06:17:37.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.797+0000 7fc756800640 1 --2- 172.21.15.96:0/3758939090 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc75815b6f0 0x7fc75815d050 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7fc75000c7d0 tx=0x7fc75000cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:37.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.797+0000 7fc74e000640 1 -- 172.21.15.96:0/3758939090 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc750018070 con 0x7fc75815b6f0 2024-05-28T06:17:37.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.797+0000 7fc74e000640 1 -- 172.21.15.96:0/3758939090 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc75000a640 con 0x7fc75815b6f0 2024-05-28T06:17:37.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.797+0000 7fc74e000640 1 -- 172.21.15.96:0/3758939090 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc75000aa70 con 0x7fc75815b6f0 2024-05-28T06:17:37.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.797+0000 7fc75ca00640 1 -- 172.21.15.96:0/3758939090 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc758159a80 con 0x7fc75815b6f0 2024-05-28T06:17:37.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.797+0000 7fc75ca00640 1 -- 172.21.15.96:0/3758939090 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc75815a040 con 0x7fc75815b6f0 2024-05-28T06:17:37.797 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.798+0000 7fc75ca00640 1 -- 172.21.15.96:0/3758939090 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc75815c7f0 con 0x7fc75815b6f0 2024-05-28T06:17:37.797 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.799+0000 7fc74e000640 1 -- 172.21.15.96:0/3758939090 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 29) ==== 92672+0+0 (secure 0 0 0) 0x7fc75000ac10 con 0x7fc75815b6f0 2024-05-28T06:17:37.798 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.800+0000 7fc74e000640 1 --2- 172.21.15.96:0/3758939090 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc73806daf0 0x7fc73806ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:37.798 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.800+0000 7fc74e000640 1 -- 172.21.15.96:0/3758939090 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5795+0+0 (secure 0 0 0) 0x7fc75008a130 con 0x7fc75815b6f0 2024-05-28T06:17:37.798 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.801+0000 7fc755e00640 1 --2- 172.21.15.96:0/3758939090 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc73806daf0 0x7fc73806ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:37.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.802+0000 7fc755e00640 1 --2- 172.21.15.96:0/3758939090 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc73806daf0 0x7fc73806ffb0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fc748034000 tx=0x7fc74802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:37.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:37.804+0000 7fc74e000640 1 -- 172.21.15.96:0/3758939090 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc750059c40 con 0x7fc75815b6f0 2024-05-28T06:17:38.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.021+0000 7fc75ca00640 1 -- 172.21.15.96:0/3758939090 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc7580d0c80 con 0x7fc73806daf0 2024-05-28T06:17:38.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.092+0000 7fc74e000640 1 -- 172.21.15.96:0/3758939090 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fc7580d0c80 con 0x7fc73806daf0 2024-05-28T06:17:38.090 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:17:38.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.095+0000 7fc75ca00640 1 -- 172.21.15.96:0/3758939090 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc73806daf0 msgr2=0x7fc73806ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:38.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.095+0000 7fc75ca00640 1 --2- 172.21.15.96:0/3758939090 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc73806daf0 0x7fc73806ffb0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fc748034000 tx=0x7fc74802e000 comp rx=0 tx=0).stop 2024-05-28T06:17:38.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.095+0000 7fc75ca00640 1 -- 172.21.15.96:0/3758939090 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc75815b6f0 msgr2=0x7fc75815d050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:38.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.095+0000 7fc75ca00640 1 --2- 172.21.15.96:0/3758939090 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc75815b6f0 0x7fc75815d050 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7fc75000c7d0 tx=0x7fc75000cc90 comp rx=0 tx=0).stop 2024-05-28T06:17:38.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.095+0000 7fc75ca00640 1 -- 172.21.15.96:0/3758939090 shutdown_connections 2024-05-28T06:17:38.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.095+0000 7fc75ca00640 1 --2- 172.21.15.96:0/3758939090 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc73806daf0 0x7fc73806ffb0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:38.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.095+0000 7fc75ca00640 1 --2- 172.21.15.96:0/3758939090 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc75815b6f0 0x7fc75815d050 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:38.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.095+0000 7fc75ca00640 1 --2- 172.21.15.96:0/3758939090 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc7580d0640 0x7fc75815cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:38.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.095+0000 7fc75ca00640 1 --2- 172.21.15.96:0/3758939090 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc7580725e0 0x7fc7581593a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:38.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.095+0000 7fc75ca00640 1 -- 172.21.15.96:0/3758939090 >> 172.21.15.96:0/3758939090 conn(0x7fc75806bd20 msgr2=0x7fc75814be90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:38.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.095+0000 7fc75ca00640 1 -- 172.21.15.96:0/3758939090 shutdown_connections 2024-05-28T06:17:38.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.095+0000 7fc75ca00640 1 -- 172.21.15.96:0/3758939090 wait complete. 2024-05-28T06:17:38.112 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs sv1 2024-05-28T06:17:38.307 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.309+0000 7fc738c00640 1 Processor -- start 2024-05-28T06:17:38.307 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.309+0000 7fc738c00640 1 -- start start 2024-05-28T06:17:38.308 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.310+0000 7fc738c00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fc734161a90 0x7fc73406e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:38.308 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.310+0000 7fc738c00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fc734161e70 0x7fc7340b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:38.308 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.310+0000 7fc738c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc73406e9b0 0x7fc7340ad2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:38.308 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.310+0000 7fc738c00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fc7340b3d40 con 0x7fc734161a90 2024-05-28T06:17:38.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.310+0000 7fc738c00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fc7340b4020 con 0x7fc734161e70 2024-05-28T06:17:38.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.310+0000 7fc738c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fc7340b4300 con 0x7fc73406e9b0 2024-05-28T06:17:38.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.311+0000 7fc731e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fc734161a90 0x7fc73406e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:38.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.311+0000 7fc731e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fc734161a90 0x7fc73406e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:33866/0 (socket says 172.21.15.96:33866) 2024-05-28T06:17:38.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.311+0000 7fc731e00640 1 -- 172.21.15.96:0/3065769599 learned_addr learned my addr 172.21.15.96:0/3065769599 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:17:38.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.311+0000 7fc732800640 1 --2- 172.21.15.96:0/3065769599 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc73406e9b0 0x7fc7340ad2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:38.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.311+0000 7fc731400640 1 --2- 172.21.15.96:0/3065769599 >> v2:172.21.15.123:3300/0 conn(0x7fc734161e70 0x7fc7340b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:38.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.311+0000 7fc731e00640 1 -- 172.21.15.96:0/3065769599 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc73406e9b0 msgr2=0x7fc7340ad2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:38.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.311+0000 7fc731e00640 1 --2- 172.21.15.96:0/3065769599 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc73406e9b0 0x7fc7340ad2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:38.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.311+0000 7fc731e00640 1 -- 172.21.15.96:0/3065769599 >> v2:172.21.15.123:3300/0 conn(0x7fc734161e70 msgr2=0x7fc7340b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:38.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.311+0000 7fc731e00640 1 --2- 172.21.15.96:0/3065769599 >> v2:172.21.15.123:3300/0 conn(0x7fc734161e70 0x7fc7340b32c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:38.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.311+0000 7fc731e00640 1 -- 172.21.15.96:0/3065769599 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc73406dc70 con 0x7fc734161a90 2024-05-28T06:17:38.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.311+0000 7fc732800640 1 --2- 172.21.15.96:0/3065769599 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc73406e9b0 0x7fc7340ad2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:17:38.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.311+0000 7fc731400640 1 --2- 172.21.15.96:0/3065769599 >> v2:172.21.15.123:3300/0 conn(0x7fc734161e70 0x7fc7340b32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:17:38.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.312+0000 7fc731e00640 1 --2- 172.21.15.96:0/3065769599 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc734161a90 0x7fc73406e470 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7fc7340d1350 tx=0x7fc72402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be7333a180bb61ba server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:38.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.313+0000 7fc730a00640 1 -- 172.21.15.96:0/3065769599 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc724037070 con 0x7fc734161a90 2024-05-28T06:17:38.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.313+0000 7fc730a00640 1 -- 172.21.15.96:0/3065769599 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc724032440 con 0x7fc734161a90 2024-05-28T06:17:38.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.313+0000 7fc730a00640 1 -- 172.21.15.96:0/3065769599 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc724032810 con 0x7fc734161a90 2024-05-28T06:17:38.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.313+0000 7fc738c00640 1 -- 172.21.15.96:0/3065769599 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc734161a90 msgr2=0x7fc73406e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:38.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.313+0000 7fc738c00640 1 --2- 172.21.15.96:0/3065769599 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc734161a90 0x7fc73406e470 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7fc7340d1350 tx=0x7fc72402a1e0 comp rx=0 tx=0).stop 2024-05-28T06:17:38.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.313+0000 7fc738c00640 1 -- 172.21.15.96:0/3065769599 shutdown_connections 2024-05-28T06:17:38.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.313+0000 7fc738c00640 1 --2- 172.21.15.96:0/3065769599 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc73406e9b0 0x7fc7340ad2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:38.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.313+0000 7fc738c00640 1 --2- 172.21.15.96:0/3065769599 >> v2:172.21.15.123:3300/0 conn(0x7fc734161e70 0x7fc7340b32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:38.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.313+0000 7fc738c00640 1 --2- 172.21.15.96:0/3065769599 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc734161a90 0x7fc73406e470 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:38.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.313+0000 7fc738c00640 1 -- 172.21.15.96:0/3065769599 >> 172.21.15.96:0/3065769599 conn(0x7fc73406bd20 msgr2=0x7fc73406c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:38.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.313+0000 7fc738c00640 1 -- 172.21.15.96:0/3065769599 shutdown_connections 2024-05-28T06:17:38.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.313+0000 7fc738c00640 1 -- 172.21.15.96:0/3065769599 wait complete. 2024-05-28T06:17:38.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.315+0000 7fc738c00640 1 Processor -- start 2024-05-28T06:17:38.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.315+0000 7fc738c00640 1 -- start start 2024-05-28T06:17:38.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.316+0000 7fc738c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc73406e9b0 0x7fc73409aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:38.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.316+0000 7fc738c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc734161a90 0x7fc734154110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:38.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.316+0000 7fc738c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc734161e70 0x7fc734154650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:38.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.316+0000 7fc738c00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc7340b3800 con 0x7fc734161e70 2024-05-28T06:17:38.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.316+0000 7fc738c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc7340b5020 con 0x7fc73406e9b0 2024-05-28T06:17:38.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.316+0000 7fc738c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fc7340b3980 con 0x7fc734161a90 2024-05-28T06:17:38.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.316+0000 7fc732800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc734161e70 0x7fc734154650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:38.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.316+0000 7fc732800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc734161e70 0x7fc734154650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:33870/0 (socket says 172.21.15.96:33870) 2024-05-28T06:17:38.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.316+0000 7fc732800640 1 -- 172.21.15.96:0/1204456909 learned_addr learned my addr 172.21.15.96:0/1204456909 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:17:38.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.317+0000 7fc731e00640 1 --2- 172.21.15.96:0/1204456909 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc73406e9b0 0x7fc73409aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:38.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.317+0000 7fc732800640 1 -- 172.21.15.96:0/1204456909 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc734161a90 msgr2=0x7fc734154110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:38.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.317+0000 7fc732800640 1 --2- 172.21.15.96:0/1204456909 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc734161a90 0x7fc734154110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:38.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.317+0000 7fc732800640 1 -- 172.21.15.96:0/1204456909 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc73406e9b0 msgr2=0x7fc73409aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:38.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.317+0000 7fc732800640 1 --2- 172.21.15.96:0/1204456909 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc73406e9b0 0x7fc73409aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:38.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.317+0000 7fc732800640 1 -- 172.21.15.96:0/1204456909 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc73409afb0 con 0x7fc734161e70 2024-05-28T06:17:38.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.317+0000 7fc731e00640 1 --2- 172.21.15.96:0/1204456909 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc73406e9b0 0x7fc73409aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:17:38.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.317+0000 7fc732800640 1 --2- 172.21.15.96:0/1204456909 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc734161e70 0x7fc734154650 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7fc72c00cd10 tx=0x7fc72c00a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:38.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.317+0000 7fc72a000640 1 -- 172.21.15.96:0/1204456909 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc72c00f680 con 0x7fc734161e70 2024-05-28T06:17:38.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.317+0000 7fc72a000640 1 -- 172.21.15.96:0/1204456909 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc72c00f840 con 0x7fc734161e70 2024-05-28T06:17:38.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.317+0000 7fc738c00640 1 -- 172.21.15.96:0/1204456909 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc734145470 con 0x7fc734161e70 2024-05-28T06:17:38.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.318+0000 7fc72a000640 1 -- 172.21.15.96:0/1204456909 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc72c00fc30 con 0x7fc734161e70 2024-05-28T06:17:38.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.318+0000 7fc738c00640 1 -- 172.21.15.96:0/1204456909 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc734145d10 con 0x7fc734161e70 2024-05-28T06:17:38.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.319+0000 7fc738c00640 1 -- 172.21.15.96:0/1204456909 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7340689e0 con 0x7fc734161e70 2024-05-28T06:17:38.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.320+0000 7fc72a000640 1 -- 172.21.15.96:0/1204456909 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 29) ==== 92672+0+0 (secure 0 0 0) 0x7fc72c0213f0 con 0x7fc734161e70 2024-05-28T06:17:38.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.320+0000 7fc72a000640 1 --2- 172.21.15.96:0/1204456909 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc71406db20 0x7fc71406ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:38.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.321+0000 7fc72a000640 1 -- 172.21.15.96:0/1204456909 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5795+0+0 (secure 0 0 0) 0x7fc72c08b800 con 0x7fc734161e70 2024-05-28T06:17:38.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.321+0000 7fc731e00640 1 --2- 172.21.15.96:0/1204456909 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc71406db20 0x7fc71406ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:38.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.322+0000 7fc731e00640 1 --2- 172.21.15.96:0/1204456909 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc71406db20 0x7fc71406ffe0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fc724034000 tx=0x7fc72402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:38.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.325+0000 7fc72a000640 1 -- 172.21.15.96:0/1204456909 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc72c014030 con 0x7fc734161e70 2024-05-28T06:17:38.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.548+0000 7fc738c00640 1 -- 172.21.15.96:0/1204456909 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) -- 0x7fc7340ad940 con 0x7fc71406db20 2024-05-28T06:17:38.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.593+0000 7fc72a000640 1 -- 172.21.15.96:0/1204456909 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fc7340ad940 con 0x7fc71406db20 2024-05-28T06:17:38.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.596+0000 7fc738c00640 1 -- 172.21.15.96:0/1204456909 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc71406db20 msgr2=0x7fc71406ffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:38.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.596+0000 7fc738c00640 1 --2- 172.21.15.96:0/1204456909 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc71406db20 0x7fc71406ffe0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fc724034000 tx=0x7fc72402e000 comp rx=0 tx=0).stop 2024-05-28T06:17:38.594 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.596+0000 7fc738c00640 1 -- 172.21.15.96:0/1204456909 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc734161e70 msgr2=0x7fc734154650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:38.594 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.596+0000 7fc738c00640 1 --2- 172.21.15.96:0/1204456909 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc734161e70 0x7fc734154650 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7fc72c00cd10 tx=0x7fc72c00a650 comp rx=0 tx=0).stop 2024-05-28T06:17:38.594 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.596+0000 7fc738c00640 1 -- 172.21.15.96:0/1204456909 shutdown_connections 2024-05-28T06:17:38.594 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.596+0000 7fc738c00640 1 --2- 172.21.15.96:0/1204456909 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc71406db20 0x7fc71406ffe0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:38.595 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.596+0000 7fc738c00640 1 --2- 172.21.15.96:0/1204456909 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc734161e70 0x7fc734154650 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:38.595 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.596+0000 7fc738c00640 1 --2- 172.21.15.96:0/1204456909 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc734161a90 0x7fc734154110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:38.595 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.596+0000 7fc738c00640 1 --2- 172.21.15.96:0/1204456909 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc73406e9b0 0x7fc73409aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:38.595 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.596+0000 7fc738c00640 1 -- 172.21.15.96:0/1204456909 >> 172.21.15.96:0/1204456909 conn(0x7fc73406bd20 msgr2=0x7fc7340d0610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:38.595 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.596+0000 7fc738c00640 1 -- 172.21.15.96:0/1204456909 shutdown_connections 2024-05-28T06:17:38.595 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.596+0000 7fc738c00640 1 -- 172.21.15.96:0/1204456909 wait complete. 2024-05-28T06:17:38.613 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone436315 2024-05-28T06:17:38.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.806+0000 7f9105200640 1 Processor -- start 2024-05-28T06:17:38.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.806+0000 7f9105200640 1 -- start start 2024-05-28T06:17:38.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.808+0000 7f9105200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f91000d1100 0x7f91000d1560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:38.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.808+0000 7f9105200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9100163020 con 0x7f910015e990 2024-05-28T06:17:38.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.808+0000 7f9105200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9100163300 con 0x7f91000d31d0 2024-05-28T06:17:38.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.808+0000 7f9105200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f91000d1aa0 con 0x7f91000d1100 2024-05-28T06:17:38.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.808+0000 7f90fd400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f91000d1100 0x7f91000d1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:38.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.808+0000 7f90fde00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f910015e990 0x7f9100162560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:57794/0 (socket says 172.21.15.96:57794) 2024-05-28T06:17:38.807 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.808+0000 7f90fde00640 1 -- 172.21.15.96:0/852217207 learned_addr learned my addr 172.21.15.96:0/852217207 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:17:38.807 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.809+0000 7f90fd400640 1 -- 172.21.15.96:0/852217207 >> v1:172.21.15.123:6789/0 conn(0x7f91000d31d0 legacy=0x7f91000d3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:38.807 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.809+0000 7f90fd400640 1 -- 172.21.15.96:0/852217207 >> v1:172.21.15.96:6789/0 conn(0x7f910015e990 legacy=0x7f9100162560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:38.807 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.809+0000 7f90fd400640 1 -- 172.21.15.96:0/852217207 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91000d29d0 con 0x7f91000d1100 2024-05-28T06:17:38.807 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.810+0000 7f90fd400640 1 --2- 172.21.15.96:0/852217207 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f91000d1100 0x7f91000d1560 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f90ec0097a0 tx=0x7f90ec02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bb93237683b9ab1 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:38.808 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.810+0000 7f90fca00640 1 -- 172.21.15.96:0/852217207 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90ec037070 con 0x7f91000d1100 2024-05-28T06:17:38.808 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.810+0000 7f90fca00640 1 -- 172.21.15.96:0/852217207 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90ec032440 con 0x7f91000d1100 2024-05-28T06:17:38.808 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.810+0000 7f90fca00640 1 -- 172.21.15.96:0/852217207 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90ec032810 con 0x7f91000d1100 2024-05-28T06:17:38.808 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.810+0000 7f9105200640 1 -- 172.21.15.96:0/852217207 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f91000d1100 msgr2=0x7f91000d1560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:38.809 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.810+0000 7f9105200640 1 --2- 172.21.15.96:0/852217207 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f91000d1100 0x7f91000d1560 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f90ec0097a0 tx=0x7f90ec02a1e0 comp rx=0 tx=0).stop 2024-05-28T06:17:38.809 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.810+0000 7f9105200640 1 -- 172.21.15.96:0/852217207 shutdown_connections 2024-05-28T06:17:38.809 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.810+0000 7f9105200640 1 --2- 172.21.15.96:0/852217207 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f91000d1100 0x7f91000d1560 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:38.809 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.810+0000 7f9105200640 1 -- 172.21.15.96:0/852217207 >> 172.21.15.96:0/852217207 conn(0x7f910006cb00 msgr2=0x7f910006cf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:38.809 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.811+0000 7f9105200640 1 -- 172.21.15.96:0/852217207 shutdown_connections 2024-05-28T06:17:38.809 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.811+0000 7f9105200640 1 -- 172.21.15.96:0/852217207 wait complete. 2024-05-28T06:17:38.810 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.812+0000 7f9105200640 1 Processor -- start 2024-05-28T06:17:38.810 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.812+0000 7f9105200640 1 -- start start 2024-05-28T06:17:38.812 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.813+0000 7f9105200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f91000d1100 0x7f91000b6bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:38.812 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.813+0000 7f9105200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f91000d31d0 0x7f91001e8a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:38.812 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.813+0000 7f9105200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f910015e990 0x7f91001e8f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:38.812 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.813+0000 7f9105200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f91000d2d40 con 0x7f91000d31d0 2024-05-28T06:17:38.812 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.813+0000 7f9105200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f91000d2780 con 0x7f910015e990 2024-05-28T06:17:38.812 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.813+0000 7f9105200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f910015ed70 con 0x7f91000d1100 2024-05-28T06:17:38.812 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.814+0000 7f90fd400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f91000d31d0 0x7f91001e8a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:38.813 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.814+0000 7f90fd400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f91000d31d0 0x7f91001e8a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48938/0 (socket says 172.21.15.96:48938) 2024-05-28T06:17:38.813 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.814+0000 7f90fd400640 1 -- 172.21.15.96:0/1496606733 learned_addr learned my addr 172.21.15.96:0/1496606733 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:17:38.813 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.814+0000 7f90fde00640 1 --2- 172.21.15.96:0/1496606733 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f91000d1100 0x7f91000b6bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:38.813 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.814+0000 7f90fd400640 1 -- 172.21.15.96:0/1496606733 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f91000d1100 msgr2=0x7f91000b6bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:38.813 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.814+0000 7f90fe800640 1 --2- 172.21.15.96:0/1496606733 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f910015e990 0x7f91001e8f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:38.813 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.814+0000 7f90fd400640 1 --2- 172.21.15.96:0/1496606733 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f91000d1100 0x7f91000b6bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:38.813 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.814+0000 7f90fd400640 1 -- 172.21.15.96:0/1496606733 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f910015e990 msgr2=0x7f91001e8f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:38.813 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.814+0000 7f90fd400640 1 --2- 172.21.15.96:0/1496606733 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f910015e990 0x7f91001e8f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:38.813 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.814+0000 7f90fd400640 1 -- 172.21.15.96:0/1496606733 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91000b38e0 con 0x7f91000d31d0 2024-05-28T06:17:38.813 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.814+0000 7f90fde00640 1 --2- 172.21.15.96:0/1496606733 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f91000d1100 0x7f91000b6bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:17:38.813 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.814+0000 7f90fd400640 1 --2- 172.21.15.96:0/1496606733 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f91000d31d0 0x7f91001e8a40 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f90ec000c00 tx=0x7f90ec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:38.814 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.815+0000 7f90f6000640 1 -- 172.21.15.96:0/1496606733 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90ec037040 con 0x7f91000d31d0 2024-05-28T06:17:38.814 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.815+0000 7f90f6000640 1 -- 172.21.15.96:0/1496606733 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90ec042070 con 0x7f91000d31d0 2024-05-28T06:17:38.814 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.815+0000 7f9105200640 1 -- 172.21.15.96:0/1496606733 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f910014a6d0 con 0x7f91000d31d0 2024-05-28T06:17:38.814 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.815+0000 7f90f6000640 1 -- 172.21.15.96:0/1496606733 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90ec030a70 con 0x7f91000d31d0 2024-05-28T06:17:38.814 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.815+0000 7f9105200640 1 -- 172.21.15.96:0/1496606733 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91000b4ff0 con 0x7f91000d31d0 2024-05-28T06:17:38.814 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.816+0000 7f9105200640 1 -- 172.21.15.96:0/1496606733 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91000689e0 con 0x7f91000d31d0 2024-05-28T06:17:38.816 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.818+0000 7f90f6000640 1 -- 172.21.15.96:0/1496606733 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 29) ==== 92672+0+0 (secure 0 0 0) 0x7f90ec030c10 con 0x7f91000d31d0 2024-05-28T06:17:38.817 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.819+0000 7f90f6000640 1 --2- 172.21.15.96:0/1496606733 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f90e006daf0 0x7f90e006ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:38.817 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.819+0000 7f90f6000640 1 -- 172.21.15.96:0/1496606733 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5795+0+0 (secure 0 0 0) 0x7f90ec0ab1c0 con 0x7f91000d31d0 2024-05-28T06:17:38.818 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.820+0000 7f90fde00640 1 --2- 172.21.15.96:0/1496606733 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f90e006daf0 0x7f90e006ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:38.819 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.821+0000 7f90fde00640 1 --2- 172.21.15.96:0/1496606733 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f90e006daf0 0x7f90e006ffb0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f90f0000f60 tx=0x7f90f0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:38.822 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:38.824+0000 7f90f6000640 1 -- 172.21.15.96:0/1496606733 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f90ec045360 con 0x7f91000d31d0 2024-05-28T06:17:39.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.052+0000 7f9105200640 1 -- 172.21.15.96:0/1496606733 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone436315", "target": ["mon-mgr", ""]}) -- 0x7f91000b3bc0 con 0x7f90e006daf0 2024-05-28T06:17:39.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.102+0000 7f90f6000640 1 -- 172.21.15.96:0/1496606733 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f91000b3bc0 con 0x7f90e006daf0 2024-05-28T06:17:39.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.105+0000 7f9105200640 1 -- 172.21.15.96:0/1496606733 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f90e006daf0 msgr2=0x7f90e006ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:39.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.105+0000 7f9105200640 1 --2- 172.21.15.96:0/1496606733 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f90e006daf0 0x7f90e006ffb0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f90f0000f60 tx=0x7f90f0009040 comp rx=0 tx=0).stop 2024-05-28T06:17:39.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.105+0000 7f9105200640 1 -- 172.21.15.96:0/1496606733 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f91000d31d0 msgr2=0x7f91001e8a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:39.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.105+0000 7f9105200640 1 --2- 172.21.15.96:0/1496606733 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f91000d31d0 0x7f91001e8a40 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f90ec000c00 tx=0x7f90ec0046c0 comp rx=0 tx=0).stop 2024-05-28T06:17:39.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.105+0000 7f9105200640 1 -- 172.21.15.96:0/1496606733 shutdown_connections 2024-05-28T06:17:39.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.105+0000 7f9105200640 1 --2- 172.21.15.96:0/1496606733 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f90e006daf0 0x7f90e006ffb0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:39.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.105+0000 7f9105200640 1 --2- 172.21.15.96:0/1496606733 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f910015e990 0x7f91001e8f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:39.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.105+0000 7f9105200640 1 --2- 172.21.15.96:0/1496606733 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f91000d31d0 0x7f91001e8a40 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:39.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.105+0000 7f9105200640 1 --2- 172.21.15.96:0/1496606733 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f91000d1100 0x7f91000b6bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:39.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.105+0000 7f9105200640 1 -- 172.21.15.96:0/1496606733 >> 172.21.15.96:0/1496606733 conn(0x7f910006cb00 msgr2=0x7f91000cb7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:39.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.105+0000 7f9105200640 1 -- 172.21.15.96:0/1496606733 shutdown_connections 2024-05-28T06:17:39.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.105+0000 7f9105200640 1 -- 172.21.15.96:0/1496606733 wait complete. 2024-05-28T06:17:39.122 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone43632 2024-05-28T06:17:39.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.347+0000 7fc286400640 1 Processor -- start 2024-05-28T06:17:39.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.347+0000 7fc286400640 1 -- start start 2024-05-28T06:17:39.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.348+0000 7fc286400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc2801622b0 0x7fc2800d4e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:39.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.348+0000 7fc286400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fc2800b5570 0x7fc2800b5a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:39.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.348+0000 7fc286400640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2800d5500 con 0x7fc2800d34e0 2024-05-28T06:17:39.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.348+0000 7fc286400640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fc2800b4f50 con 0x7fc2800b5570 2024-05-28T06:17:39.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.348+0000 7fc286400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fc2800d2720 con 0x7fc2801622b0 2024-05-28T06:17:39.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.349+0000 7fc27fe00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fc2800b5570 0x7fc2800b5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:39.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.349+0000 7fc27ea00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fc2800d34e0 0x7fc2800d1b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:40052/0 (socket says 172.21.15.96:40052) 2024-05-28T06:17:39.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.349+0000 7fc27ea00640 1 -- 172.21.15.96:0/3904711832 learned_addr learned my addr 172.21.15.96:0/3904711832 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:17:39.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.349+0000 7fc27fe00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fc2800b5570 0x7fc2800b5a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48954/0 (socket says 172.21.15.96:48954) 2024-05-28T06:17:39.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.349+0000 7fc27f400640 1 --2- 172.21.15.96:0/3904711832 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc2801622b0 0x7fc2800d4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:39.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.349+0000 7fc27fe00640 1 -- 172.21.15.96:0/3904711832 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc2801622b0 msgr2=0x7fc2800d4e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:39.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.349+0000 7fc27fe00640 1 --2- 172.21.15.96:0/3904711832 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc2801622b0 0x7fc2800d4e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:39.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.349+0000 7fc27fe00640 1 -- 172.21.15.96:0/3904711832 >> v1:172.21.15.96:6789/0 conn(0x7fc2800d34e0 legacy=0x7fc2800d1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:39.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.349+0000 7fc27fe00640 1 -- 172.21.15.96:0/3904711832 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2800d2a00 con 0x7fc2800b5570 2024-05-28T06:17:39.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.351+0000 7fc27fe00640 1 --2- 172.21.15.96:0/3904711832 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc2800b5570 0x7fc2800b5a20 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7fc270009d90 tx=0x7fc27002a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e780e9f3e31401b server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:39.349 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.351+0000 7fc27e000640 1 -- 172.21.15.96:0/3904711832 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc270031600 con 0x7fc2800b5570 2024-05-28T06:17:39.349 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.351+0000 7fc27e000640 1 -- 172.21.15.96:0/3904711832 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2700317a0 con 0x7fc2800b5570 2024-05-28T06:17:39.349 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.351+0000 7fc27e000640 1 -- 172.21.15.96:0/3904711832 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc270031b70 con 0x7fc2800b5570 2024-05-28T06:17:39.350 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.352+0000 7fc286400640 1 -- 172.21.15.96:0/3904711832 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc2800b5570 msgr2=0x7fc2800b5a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:39.350 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.352+0000 7fc286400640 1 --2- 172.21.15.96:0/3904711832 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc2800b5570 0x7fc2800b5a20 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7fc270009d90 tx=0x7fc27002a5d0 comp rx=0 tx=0).stop 2024-05-28T06:17:39.351 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.353+0000 7fc286400640 1 -- 172.21.15.96:0/3904711832 shutdown_connections 2024-05-28T06:17:39.351 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.353+0000 7fc286400640 1 --2- 172.21.15.96:0/3904711832 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc2800b5570 0x7fc2800b5a20 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:39.351 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.353+0000 7fc286400640 1 --2- 172.21.15.96:0/3904711832 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc2801622b0 0x7fc2800d4e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:39.351 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.353+0000 7fc286400640 1 -- 172.21.15.96:0/3904711832 >> 172.21.15.96:0/3904711832 conn(0x7fc28008bfc0 msgr2=0x7fc2800d46f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:39.351 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.353+0000 7fc286400640 1 -- 172.21.15.96:0/3904711832 shutdown_connections 2024-05-28T06:17:39.351 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.353+0000 7fc286400640 1 -- 172.21.15.96:0/3904711832 wait complete. 2024-05-28T06:17:39.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.356+0000 7fc286400640 1 Processor -- start 2024-05-28T06:17:39.354 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.356+0000 7fc286400640 1 -- start start 2024-05-28T06:17:39.355 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.357+0000 7fc286400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc2800b5570 0x7fc280159820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:39.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.357+0000 7fc286400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc2800d34e0 0x7fc2800c86e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:39.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.357+0000 7fc286400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc2801622b0 0x7fc2800c8c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:39.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.358+0000 7fc286400640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc2800b40c0 con 0x7fc2800d34e0 2024-05-28T06:17:39.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.358+0000 7fc286400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc2800b3f40 con 0x7fc2801622b0 2024-05-28T06:17:39.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.358+0000 7fc286400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fc2800b4240 con 0x7fc2800b5570 2024-05-28T06:17:39.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.358+0000 7fc27ea00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc2800d34e0 0x7fc2800c86e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:39.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.358+0000 7fc27f400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc2800b5570 0x7fc280159820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:39.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.358+0000 7fc27ea00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc2800d34e0 0x7fc2800c86e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48962/0 (socket says 172.21.15.96:48962) 2024-05-28T06:17:39.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.358+0000 7fc27ea00640 1 -- 172.21.15.96:0/3929582811 learned_addr learned my addr 172.21.15.96:0/3929582811 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:17:39.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.358+0000 7fc27fe00640 1 --2- 172.21.15.96:0/3929582811 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc2801622b0 0x7fc2800c8c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:39.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.359+0000 7fc27ea00640 1 -- 172.21.15.96:0/3929582811 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc2800b5570 msgr2=0x7fc280159820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:39.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.359+0000 7fc27ea00640 1 --2- 172.21.15.96:0/3929582811 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc2800b5570 0x7fc280159820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:39.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.359+0000 7fc27ea00640 1 -- 172.21.15.96:0/3929582811 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc2801622b0 msgr2=0x7fc2800c8c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:39.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.359+0000 7fc27ea00640 1 --2- 172.21.15.96:0/3929582811 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc2801622b0 0x7fc2800c8c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:39.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.359+0000 7fc27ea00640 1 -- 172.21.15.96:0/3929582811 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2800c9250 con 0x7fc2800d34e0 2024-05-28T06:17:39.358 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.360+0000 7fc27ea00640 1 --2- 172.21.15.96:0/3929582811 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc2800d34e0 0x7fc2800c86e0 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7fc26800c430 tx=0x7fc26800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:39.358 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.361+0000 7fc277400640 1 -- 172.21.15.96:0/3929582811 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc268017070 con 0x7fc2800d34e0 2024-05-28T06:17:39.358 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.361+0000 7fc277400640 1 -- 172.21.15.96:0/3929582811 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc268005ce0 con 0x7fc2800d34e0 2024-05-28T06:17:39.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.361+0000 7fc277400640 1 -- 172.21.15.96:0/3929582811 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2680072f0 con 0x7fc2800d34e0 2024-05-28T06:17:39.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.361+0000 7fc286400640 1 -- 172.21.15.96:0/3929582811 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2800c9530 con 0x7fc2800d34e0 2024-05-28T06:17:39.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.361+0000 7fc286400640 1 -- 172.21.15.96:0/3929582811 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2800cba80 con 0x7fc2800d34e0 2024-05-28T06:17:39.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.364+0000 7fc286400640 1 -- 172.21.15.96:0/3929582811 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2800689e0 con 0x7fc2800d34e0 2024-05-28T06:17:39.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.364+0000 7fc277400640 1 -- 172.21.15.96:0/3929582811 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 29) ==== 92672+0+0 (secure 0 0 0) 0x7fc26801a070 con 0x7fc2800d34e0 2024-05-28T06:17:39.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.366+0000 7fc277400640 1 --2- 172.21.15.96:0/3929582811 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc25c06db40 0x7fc25c070000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:39.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.366+0000 7fc277400640 1 -- 172.21.15.96:0/3929582811 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5795+0+0 (secure 0 0 0) 0x7fc268089e00 con 0x7fc2800d34e0 2024-05-28T06:17:39.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.367+0000 7fc27f400640 1 --2- 172.21.15.96:0/3929582811 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc25c06db40 0x7fc25c070000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:39.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.369+0000 7fc27f400640 1 --2- 172.21.15.96:0/3929582811 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc25c06db40 0x7fc25c070000 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fc278009aa0 tx=0x7fc278009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:39.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.374+0000 7fc277400640 1 -- 172.21.15.96:0/3929582811 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc268059300 con 0x7fc2800d34e0 2024-05-28T06:17:39.745 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.747+0000 7fc286400640 1 -- 172.21.15.96:0/3929582811 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone43632", "target": ["mon-mgr", ""]}) -- 0x7fc2800d06d0 con 0x7fc25c06db40 2024-05-28T06:17:39.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.787+0000 7fc277400640 1 -- 172.21.15.96:0/3929582811 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fc2800d06d0 con 0x7fc25c06db40 2024-05-28T06:17:39.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.791+0000 7fc286400640 1 -- 172.21.15.96:0/3929582811 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc25c06db40 msgr2=0x7fc25c070000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:39.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.791+0000 7fc286400640 1 --2- 172.21.15.96:0/3929582811 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc25c06db40 0x7fc25c070000 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fc278009aa0 tx=0x7fc278009190 comp rx=0 tx=0).stop 2024-05-28T06:17:39.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.791+0000 7fc286400640 1 -- 172.21.15.96:0/3929582811 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc2800d34e0 msgr2=0x7fc2800c86e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:39.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.791+0000 7fc286400640 1 --2- 172.21.15.96:0/3929582811 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc2800d34e0 0x7fc2800c86e0 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7fc26800c430 tx=0x7fc26800c8f0 comp rx=0 tx=0).stop 2024-05-28T06:17:39.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.791+0000 7fc286400640 1 -- 172.21.15.96:0/3929582811 shutdown_connections 2024-05-28T06:17:39.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.791+0000 7fc286400640 1 --2- 172.21.15.96:0/3929582811 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc25c06db40 0x7fc25c070000 secure :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fc278009aa0 tx=0x7fc278009190 comp rx=0 tx=0).stop 2024-05-28T06:17:39.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.791+0000 7fc286400640 1 --2- 172.21.15.96:0/3929582811 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc2801622b0 0x7fc2800c8c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:39.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.791+0000 7fc286400640 1 --2- 172.21.15.96:0/3929582811 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc2800d34e0 0x7fc2800c86e0 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:39.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.791+0000 7fc286400640 1 --2- 172.21.15.96:0/3929582811 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc2800b5570 0x7fc280159820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:39.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.791+0000 7fc286400640 1 -- 172.21.15.96:0/3929582811 >> 172.21.15.96:0/3929582811 conn(0x7fc28008bfc0 msgr2=0x7fc280161570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:39.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.791+0000 7fc286400640 1 -- 172.21.15.96:0/3929582811 shutdown_connections 2024-05-28T06:17:39.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.791+0000 7fc286400640 1 -- 172.21.15.96:0/3929582811 wait complete. 2024-05-28T06:17:39.817 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone43638 2024-05-28T06:17:39.987 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.989+0000 7f0a4a800640 1 Processor -- start 2024-05-28T06:17:39.987 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.989+0000 7f0a4a800640 1 -- start start 2024-05-28T06:17:39.988 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.990+0000 7f0a4a800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0a441600f0 0x7f0a440d0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:39.988 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.990+0000 7f0a4a800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f0a440d0970 0x7f0a44072190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:39.988 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.990+0000 7f0a4a800640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a4406fa00 con 0x7f0a4415fa00 2024-05-28T06:17:39.988 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.990+0000 7f0a4a800640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f0a440d0dc0 con 0x7f0a440d0970 2024-05-28T06:17:39.988 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.990+0000 7f0a4a800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f0a4406f160 con 0x7f0a441600f0 2024-05-28T06:17:39.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.991+0000 7f0a48a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0a441600f0 0x7f0a440d0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:39.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.991+0000 7f0a48a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0a441600f0 0x7f0a440d0430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.96:34288/0 (socket says 172.21.15.96:34288) 2024-05-28T06:17:39.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.991+0000 7f0a49e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f0a440d0970 0x7f0a44072190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:39.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.991+0000 7f0a48a00640 1 -- 172.21.15.96:0/4182922897 learned_addr learned my addr 172.21.15.96:0/4182922897 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:17:39.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.992+0000 7f0a48a00640 1 -- 172.21.15.96:0/4182922897 >> v2:172.21.15.123:3300/0 conn(0x7f0a440d0970 msgr2=0x7f0a44072190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:39.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.992+0000 7f0a48a00640 1 --2- 172.21.15.96:0/4182922897 >> v2:172.21.15.123:3300/0 conn(0x7f0a440d0970 0x7f0a44072190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:39.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.992+0000 7f0a48a00640 1 -- 172.21.15.96:0/4182922897 >> v1:172.21.15.123:6789/0 conn(0x7f0a4415fa00 legacy=0x7f0a4406e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:39.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.992+0000 7f0a48a00640 1 -- 172.21.15.96:0/4182922897 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a4406f440 con 0x7f0a441600f0 2024-05-28T06:17:39.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.993+0000 7f0a48a00640 1 --2- 172.21.15.96:0/4182922897 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0a441600f0 0x7f0a440d0430 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f0a30009870 tx=0x7f0a3002a130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e2b7d55f9b7aaa5 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:39.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.993+0000 7f0a43e00640 1 -- 172.21.15.96:0/4182922897 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a30037070 con 0x7f0a441600f0 2024-05-28T06:17:39.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.993+0000 7f0a43e00640 1 -- 172.21.15.96:0/4182922897 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a30032440 con 0x7f0a441600f0 2024-05-28T06:17:39.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.993+0000 7f0a43e00640 1 -- 172.21.15.96:0/4182922897 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a30032810 con 0x7f0a441600f0 2024-05-28T06:17:39.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.994+0000 7f0a4a800640 1 -- 172.21.15.96:0/4182922897 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0a441600f0 msgr2=0x7f0a440d0430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:39.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.994+0000 7f0a4a800640 1 --2- 172.21.15.96:0/4182922897 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0a441600f0 0x7f0a440d0430 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f0a30009870 tx=0x7f0a3002a130 comp rx=0 tx=0).stop 2024-05-28T06:17:39.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.994+0000 7f0a4a800640 1 -- 172.21.15.96:0/4182922897 shutdown_connections 2024-05-28T06:17:39.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.994+0000 7f0a4a800640 1 --2- 172.21.15.96:0/4182922897 >> v2:172.21.15.123:3300/0 conn(0x7f0a440d0970 0x7f0a44072190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:39.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.994+0000 7f0a4a800640 1 --2- 172.21.15.96:0/4182922897 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0a441600f0 0x7f0a440d0430 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:39.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.994+0000 7f0a4a800640 1 -- 172.21.15.96:0/4182922897 >> 172.21.15.96:0/4182922897 conn(0x7f0a4406bae0 msgr2=0x7f0a4406bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:39.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.994+0000 7f0a4a800640 1 -- 172.21.15.96:0/4182922897 shutdown_connections 2024-05-28T06:17:39.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.994+0000 7f0a4a800640 1 -- 172.21.15.96:0/4182922897 wait complete. 2024-05-28T06:17:39.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.996+0000 7f0a4a800640 1 Processor -- start 2024-05-28T06:17:39.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.996+0000 7f0a4a800640 1 -- start start 2024-05-28T06:17:39.995 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.997+0000 7f0a4a800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0a440d0970 0x7f0a440ccb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:39.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.997+0000 7f0a4a800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a4415fa00 0x7f0a44152070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:39.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.997+0000 7f0a4a800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0a441600f0 0x7f0a4415e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:39.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.997+0000 7f0a4a800640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f0a4415fde0 con 0x7f0a4415fa00 2024-05-28T06:17:39.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.997+0000 7f0a4a800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f0a44071c70 con 0x7f0a440d0970 2024-05-28T06:17:39.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.997+0000 7f0a4a800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f0a440726d0 con 0x7f0a441600f0 2024-05-28T06:17:39.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.997+0000 7f0a48a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a4415fa00 0x7f0a44152070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:39.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.997+0000 7f0a48a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a4415fa00 0x7f0a44152070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48972/0 (socket says 172.21.15.96:48972) 2024-05-28T06:17:39.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.997+0000 7f0a48a00640 1 -- 172.21.15.96:0/2180618318 learned_addr learned my addr 172.21.15.96:0/2180618318 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:17:39.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.998+0000 7f0a49e00640 1 --2- 172.21.15.96:0/2180618318 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0a441600f0 0x7f0a4415e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:39.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.998+0000 7f0a49400640 1 --2- 172.21.15.96:0/2180618318 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0a440d0970 0x7f0a440ccb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:39.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.998+0000 7f0a48a00640 1 -- 172.21.15.96:0/2180618318 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0a441600f0 msgr2=0x7f0a4415e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:39.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.998+0000 7f0a48a00640 1 --2- 172.21.15.96:0/2180618318 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0a441600f0 0x7f0a4415e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:39.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.998+0000 7f0a48a00640 1 -- 172.21.15.96:0/2180618318 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0a440d0970 msgr2=0x7f0a440ccb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:39.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.998+0000 7f0a48a00640 1 --2- 172.21.15.96:0/2180618318 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0a440d0970 0x7f0a440ccb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:39.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.998+0000 7f0a48a00640 1 -- 172.21.15.96:0/2180618318 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a440cd0f0 con 0x7f0a4415fa00 2024-05-28T06:17:39.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.998+0000 7f0a49e00640 1 --2- 172.21.15.96:0/2180618318 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0a441600f0 0x7f0a4415e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:17:39.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.998+0000 7f0a49400640 1 --2- 172.21.15.96:0/2180618318 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0a440d0970 0x7f0a440ccb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:17:39.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.998+0000 7f0a48a00640 1 --2- 172.21.15.96:0/2180618318 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a4415fa00 0x7f0a44152070 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7f0a3002a170 tx=0x7f0a300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:39.998 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.998+0000 7f0a41600640 1 -- 172.21.15.96:0/2180618318 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a30041030 con 0x7f0a4415fa00 2024-05-28T06:17:39.998 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.998+0000 7f0a41600640 1 -- 172.21.15.96:0/2180618318 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a30030440 con 0x7f0a4415fa00 2024-05-28T06:17:39.998 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.998+0000 7f0a41600640 1 -- 172.21.15.96:0/2180618318 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a30037040 con 0x7f0a4415fa00 2024-05-28T06:17:39.998 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.998+0000 7f0a4a800640 1 -- 172.21.15.96:0/2180618318 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a440cd3d0 con 0x7f0a4415fa00 2024-05-28T06:17:39.998 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.999+0000 7f0a4a800640 1 -- 172.21.15.96:0/2180618318 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a4415aba0 con 0x7f0a4415fa00 2024-05-28T06:17:39.998 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:39.999+0000 7f0a4a800640 1 -- 172.21.15.96:0/2180618318 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a440689e0 con 0x7f0a4415fa00 2024-05-28T06:17:39.998 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.000+0000 7f0a41600640 1 -- 172.21.15.96:0/2180618318 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 29) ==== 92672+0+0 (secure 0 0 0) 0x7f0a30044070 con 0x7f0a4415fa00 2024-05-28T06:17:39.999 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.001+0000 7f0a41600640 1 --2- 172.21.15.96:0/2180618318 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0a2406daf0 0x7f0a2406ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:39.999 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.001+0000 7f0a41600640 1 -- 172.21.15.96:0/2180618318 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5795+0+0 (secure 0 0 0) 0x7f0a300aaaf0 con 0x7f0a4415fa00 2024-05-28T06:17:39.999 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.002+0000 7f0a49400640 1 --2- 172.21.15.96:0/2180618318 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0a2406daf0 0x7f0a2406ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:40.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.003+0000 7f0a49400640 1 --2- 172.21.15.96:0/2180618318 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0a2406daf0 0x7f0a2406ffb0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f0a440ca760 tx=0x7f0a38009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:40.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.006+0000 7f0a41600640 1 -- 172.21.15.96:0/2180618318 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0a3004b070 con 0x7f0a4415fa00 2024-05-28T06:17:40.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.216+0000 7f0a4a800640 1 -- 172.21.15.96:0/2180618318 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone43638", "target": ["mon-mgr", ""]}) -- 0x7f0a4406e200 con 0x7f0a2406daf0 2024-05-28T06:17:40.267 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.269+0000 7f0a41600640 1 -- 172.21.15.96:0/2180618318 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f0a4406e200 con 0x7f0a2406daf0 2024-05-28T06:17:40.270 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.272+0000 7f0a4a800640 1 -- 172.21.15.96:0/2180618318 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0a2406daf0 msgr2=0x7f0a2406ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:40.270 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.272+0000 7f0a4a800640 1 --2- 172.21.15.96:0/2180618318 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0a2406daf0 0x7f0a2406ffb0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f0a440ca760 tx=0x7f0a38009040 comp rx=0 tx=0).stop 2024-05-28T06:17:40.270 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.272+0000 7f0a4a800640 1 -- 172.21.15.96:0/2180618318 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a4415fa00 msgr2=0x7f0a44152070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:40.270 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.272+0000 7f0a4a800640 1 --2- 172.21.15.96:0/2180618318 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a4415fa00 0x7f0a44152070 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7f0a3002a170 tx=0x7f0a300046c0 comp rx=0 tx=0).stop 2024-05-28T06:17:40.270 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.272+0000 7f0a4a800640 1 -- 172.21.15.96:0/2180618318 shutdown_connections 2024-05-28T06:17:40.270 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.272+0000 7f0a4a800640 1 --2- 172.21.15.96:0/2180618318 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0a2406daf0 0x7f0a2406ffb0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:40.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.272+0000 7f0a4a800640 1 --2- 172.21.15.96:0/2180618318 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0a441600f0 0x7f0a4415e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:40.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.272+0000 7f0a4a800640 1 --2- 172.21.15.96:0/2180618318 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a4415fa00 0x7f0a44152070 unknown :-1 s=CLOSED pgs=1272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:40.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.272+0000 7f0a4a800640 1 --2- 172.21.15.96:0/2180618318 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0a440d0970 0x7f0a440ccb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:40.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.272+0000 7f0a4a800640 1 -- 172.21.15.96:0/2180618318 >> 172.21.15.96:0/2180618318 conn(0x7f0a4406bae0 msgr2=0x7f0a4406bec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:40.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.272+0000 7f0a4a800640 1 -- 172.21.15.96:0/2180618318 shutdown_connections 2024-05-28T06:17:40.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.272+0000 7f0a4a800640 1 -- 172.21.15.96:0/2180618318 wait complete. 2024-05-28T06:17:40.288 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone436312 2024-05-28T06:17:40.483 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.485+0000 7fd80e000640 1 Processor -- start 2024-05-28T06:17:40.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.486+0000 7fd80e000640 1 -- start start 2024-05-28T06:17:40.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.487+0000 7fd80e000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fd80815e5a0 0x7fd8080d2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:40.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.487+0000 7fd80e000640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8080d3500 con 0x7fd8080cfb40 2024-05-28T06:17:40.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.487+0000 7fd80e000640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8080d2200 con 0x7fd8080d1420 2024-05-28T06:17:40.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.487+0000 7fd80e000640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fd8080d2b30 con 0x7fd80815e5a0 2024-05-28T06:17:40.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.487+0000 7fd80cc00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fd80815e5a0 0x7fd8080d2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:40.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.487+0000 7fd807e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fd8080cfb40 0x7fd8080d0d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:40064/0 (socket says 172.21.15.96:40064) 2024-05-28T06:17:40.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.487+0000 7fd807e00640 1 -- 172.21.15.96:0/1489194162 learned_addr learned my addr 172.21.15.96:0/1489194162 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:17:40.486 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.488+0000 7fd80cc00640 1 -- 172.21.15.96:0/1489194162 >> v1:172.21.15.123:6789/0 conn(0x7fd8080d1420 legacy=0x7fd80806c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:40.486 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.488+0000 7fd80cc00640 1 -- 172.21.15.96:0/1489194162 >> v1:172.21.15.96:6789/0 conn(0x7fd8080cfb40 legacy=0x7fd8080d0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:40.486 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.488+0000 7fd80cc00640 1 -- 172.21.15.96:0/1489194162 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8080d24e0 con 0x7fd80815e5a0 2024-05-28T06:17:40.486 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.489+0000 7fd80cc00640 1 --2- 172.21.15.96:0/1489194162 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd80815e5a0 0x7fd8080d2fc0 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7fd7f80097a0 tx=0x7fd7f802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbc79e696dc694cd server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:40.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.489+0000 7fd807400640 1 -- 172.21.15.96:0/1489194162 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7f8037070 con 0x7fd80815e5a0 2024-05-28T06:17:40.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.489+0000 7fd807400640 1 -- 172.21.15.96:0/1489194162 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7f8032440 con 0x7fd80815e5a0 2024-05-28T06:17:40.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.489+0000 7fd807400640 1 -- 172.21.15.96:0/1489194162 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7f8032810 con 0x7fd80815e5a0 2024-05-28T06:17:40.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.490+0000 7fd80e000640 1 -- 172.21.15.96:0/1489194162 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd80815e5a0 msgr2=0x7fd8080d2fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:40.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.490+0000 7fd80e000640 1 --2- 172.21.15.96:0/1489194162 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd80815e5a0 0x7fd8080d2fc0 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7fd7f80097a0 tx=0x7fd7f802a1e0 comp rx=0 tx=0).stop 2024-05-28T06:17:40.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.490+0000 7fd80e000640 1 -- 172.21.15.96:0/1489194162 shutdown_connections 2024-05-28T06:17:40.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.490+0000 7fd80e000640 1 --2- 172.21.15.96:0/1489194162 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd80815e5a0 0x7fd8080d2fc0 unknown :-1 s=CLOSED pgs=1274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:40.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.490+0000 7fd80e000640 1 -- 172.21.15.96:0/1489194162 >> 172.21.15.96:0/1489194162 conn(0x7fd80808bfc0 msgr2=0x7fd8080d44b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:40.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.490+0000 7fd80e000640 1 -- 172.21.15.96:0/1489194162 shutdown_connections 2024-05-28T06:17:40.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.490+0000 7fd80e000640 1 -- 172.21.15.96:0/1489194162 wait complete. 2024-05-28T06:17:40.489 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.491+0000 7fd80e000640 1 Processor -- start 2024-05-28T06:17:40.489 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.492+0000 7fd80e000640 1 -- start start 2024-05-28T06:17:40.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.492+0000 7fd80e000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd8080cfb40 0x7fd8080b3f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:40.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.492+0000 7fd80e000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd8080d1420 0x7fd80814b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:40.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.493+0000 7fd80e000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd80815e5a0 0x7fd8080b6560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:40.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.493+0000 7fd80e000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd80815eb00 con 0x7fd8080d1420 2024-05-28T06:17:40.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.493+0000 7fd80e000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd80815e980 con 0x7fd8080cfb40 2024-05-28T06:17:40.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.493+0000 7fd80e000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fd8080d5220 con 0x7fd80815e5a0 2024-05-28T06:17:40.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.493+0000 7fd807e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd8080d1420 0x7fd80814b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:40.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.493+0000 7fd807e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd8080d1420 0x7fd80814b590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48986/0 (socket says 172.21.15.96:48986) 2024-05-28T06:17:40.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.493+0000 7fd807e00640 1 -- 172.21.15.96:0/22544285 learned_addr learned my addr 172.21.15.96:0/22544285 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:17:40.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.493+0000 7fd80cc00640 1 --2- 172.21.15.96:0/22544285 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd8080cfb40 0x7fd8080b3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:40.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.493+0000 7fd80d600640 1 --2- 172.21.15.96:0/22544285 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd80815e5a0 0x7fd8080b6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:40.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.493+0000 7fd807e00640 1 -- 172.21.15.96:0/22544285 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd80815e5a0 msgr2=0x7fd8080b6560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:40.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.493+0000 7fd807e00640 1 --2- 172.21.15.96:0/22544285 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd80815e5a0 0x7fd8080b6560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:40.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.493+0000 7fd807e00640 1 -- 172.21.15.96:0/22544285 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd8080cfb40 msgr2=0x7fd8080b3f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:40.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.493+0000 7fd807e00640 1 --2- 172.21.15.96:0/22544285 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd8080cfb40 0x7fd8080b3f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:40.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.493+0000 7fd807e00640 1 -- 172.21.15.96:0/22544285 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8080b4450 con 0x7fd8080d1420 2024-05-28T06:17:40.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.493+0000 7fd80d600640 1 --2- 172.21.15.96:0/22544285 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd80815e5a0 0x7fd8080b6560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:17:40.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.494+0000 7fd807e00640 1 --2- 172.21.15.96:0/22544285 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd8080d1420 0x7fd80814b590 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7fd7f400c430 tx=0x7fd7f400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:40.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.494+0000 7fd804c00640 1 -- 172.21.15.96:0/22544285 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7f4017070 con 0x7fd8080d1420 2024-05-28T06:17:40.494 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.494+0000 7fd804c00640 1 -- 172.21.15.96:0/22544285 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7f4005ce0 con 0x7fd8080d1420 2024-05-28T06:17:40.494 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.494+0000 7fd804c00640 1 -- 172.21.15.96:0/22544285 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7f40072f0 con 0x7fd8080d1420 2024-05-28T06:17:40.494 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.494+0000 7fd80e000640 1 -- 172.21.15.96:0/22544285 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd80814ee70 con 0x7fd8080d1420 2024-05-28T06:17:40.494 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.494+0000 7fd80e000640 1 -- 172.21.15.96:0/22544285 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8080b5a80 con 0x7fd8080d1420 2024-05-28T06:17:40.494 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.495+0000 7fd80e000640 1 -- 172.21.15.96:0/22544285 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8080689e0 con 0x7fd8080d1420 2024-05-28T06:17:40.495 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.496+0000 7fd804c00640 1 -- 172.21.15.96:0/22544285 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 29) ==== 92672+0+0 (secure 0 0 0) 0x7fd7f401a070 con 0x7fd8080d1420 2024-05-28T06:17:40.495 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.497+0000 7fd804c00640 1 --2- 172.21.15.96:0/22544285 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd7e806daf0 0x7fd7e806ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:40.495 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.497+0000 7fd804c00640 1 -- 172.21.15.96:0/22544285 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5795+0+0 (secure 0 0 0) 0x7fd7f4089e50 con 0x7fd8080d1420 2024-05-28T06:17:40.495 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.497+0000 7fd80cc00640 1 --2- 172.21.15.96:0/22544285 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd7e806daf0 0x7fd7e806ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:40.496 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.499+0000 7fd80cc00640 1 --2- 172.21.15.96:0/22544285 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd7e806daf0 0x7fd7e806ffb0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fd7f8034000 tx=0x7fd7f802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:40.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.501+0000 7fd804c00640 1 -- 172.21.15.96:0/22544285 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd7f4059960 con 0x7fd8080d1420 2024-05-28T06:17:40.714 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.717+0000 7fd80e000640 1 -- 172.21.15.96:0/22544285 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone436312", "target": ["mon-mgr", ""]}) -- 0x7fd8080d1d30 con 0x7fd7e806daf0 2024-05-28T06:17:40.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.899+0000 7fd804c00640 1 -- 172.21.15.96:0/22544285 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fd8080d1d30 con 0x7fd7e806daf0 2024-05-28T06:17:40.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.901+0000 7fd80e000640 1 -- 172.21.15.96:0/22544285 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd7e806daf0 msgr2=0x7fd7e806ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:40.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.901+0000 7fd80e000640 1 --2- 172.21.15.96:0/22544285 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd7e806daf0 0x7fd7e806ffb0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fd7f8034000 tx=0x7fd7f802e000 comp rx=0 tx=0).stop 2024-05-28T06:17:40.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.901+0000 7fd80e000640 1 -- 172.21.15.96:0/22544285 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd8080d1420 msgr2=0x7fd80814b590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:40.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.901+0000 7fd80e000640 1 --2- 172.21.15.96:0/22544285 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd8080d1420 0x7fd80814b590 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7fd7f400c430 tx=0x7fd7f400c8f0 comp rx=0 tx=0).stop 2024-05-28T06:17:40.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.902+0000 7fd80e000640 1 -- 172.21.15.96:0/22544285 shutdown_connections 2024-05-28T06:17:40.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.902+0000 7fd80e000640 1 --2- 172.21.15.96:0/22544285 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd7e806daf0 0x7fd7e806ffb0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:40.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.902+0000 7fd80e000640 1 --2- 172.21.15.96:0/22544285 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd80815e5a0 0x7fd8080b6560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:40.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.902+0000 7fd80e000640 1 --2- 172.21.15.96:0/22544285 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd8080d1420 0x7fd80814b590 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:40.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.902+0000 7fd80e000640 1 --2- 172.21.15.96:0/22544285 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd8080cfb40 0x7fd8080b3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:40.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.902+0000 7fd80e000640 1 -- 172.21.15.96:0/22544285 >> 172.21.15.96:0/22544285 conn(0x7fd80808bfc0 msgr2=0x7fd8080b2bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:40.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.902+0000 7fd80e000640 1 -- 172.21.15.96:0/22544285 shutdown_connections 2024-05-28T06:17:40.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:40.902+0000 7fd80e000640 1 -- 172.21.15.96:0/22544285 wait complete. 2024-05-28T06:17:40.920 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone436318 2024-05-28T06:17:41.125 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.127+0000 7fac28a00640 1 Processor -- start 2024-05-28T06:17:41.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.127+0000 7fac28a00640 1 -- start start 2024-05-28T06:17:41.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.129+0000 7fac28a00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fac240ce560 0x7fac240b4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:41.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.129+0000 7fac28a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fac240d0860 0x7fac2415f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:41.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.129+0000 7fac28a00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac2415bb30 con 0x7fac240d0430 2024-05-28T06:17:41.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.129+0000 7fac28a00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fac240b3ba0 con 0x7fac240d0860 2024-05-28T06:17:41.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.129+0000 7fac28a00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fac2415bdd0 con 0x7fac240ce560 2024-05-28T06:17:41.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.129+0000 7fac23e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fac240d0860 0x7fac2415f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:41.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.129+0000 7fac22a00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fac240d0430 0x7fac2415f0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:40074/0 (socket says 172.21.15.96:40074) 2024-05-28T06:17:41.131 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.129+0000 7fac22a00640 1 -- 172.21.15.96:0/426957767 learned_addr learned my addr 172.21.15.96:0/426957767 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:17:41.131 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.129+0000 7fac23400640 1 --2- 172.21.15.96:0/426957767 >> v2:172.21.15.123:3300/0 conn(0x7fac240ce560 0x7fac240b4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:41.131 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.129+0000 7fac22000640 1 -- 172.21.15.96:0/426957767 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 539965838 0 0) 0x7fac2415bb30 con 0x7fac240d0430 2024-05-28T06:17:41.131 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.129+0000 7fac23e00640 1 -- 172.21.15.96:0/426957767 >> v2:172.21.15.123:3300/0 conn(0x7fac240ce560 msgr2=0x7fac240b4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:41.131 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.130+0000 7fac23e00640 1 --2- 172.21.15.96:0/426957767 >> v2:172.21.15.123:3300/0 conn(0x7fac240ce560 0x7fac240b4b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:41.131 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.130+0000 7fac23e00640 1 -- 172.21.15.96:0/426957767 >> v1:172.21.15.96:6789/0 conn(0x7fac240d0430 legacy=0x7fac2415f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:41.131 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.130+0000 7fac23e00640 1 -- 172.21.15.96:0/426957767 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac2415c0b0 con 0x7fac240d0860 2024-05-28T06:17:41.131 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.130+0000 7fac23400640 1 --2- 172.21.15.96:0/426957767 >> v2:172.21.15.123:3300/0 conn(0x7fac240ce560 0x7fac240b4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:17:41.131 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.131+0000 7fac23e00640 1 --2- 172.21.15.96:0/426957767 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fac240d0860 0x7fac2415f7d0 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7fac2406d8c0 tx=0x7fac1c02a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e4bc51467c1cc56 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:41.131 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.131+0000 7fac22000640 1 -- 172.21.15.96:0/426957767 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac1c007740 con 0x7fac240d0860 2024-05-28T06:17:41.131 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.131+0000 7fac22000640 1 -- 172.21.15.96:0/426957767 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac1c0078e0 con 0x7fac240d0860 2024-05-28T06:17:41.132 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.131+0000 7fac22000640 1 -- 172.21.15.96:0/426957767 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac1c007cb0 con 0x7fac240d0860 2024-05-28T06:17:41.132 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.131+0000 7fac28a00640 1 -- 172.21.15.96:0/426957767 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fac240d0860 msgr2=0x7fac2415f7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:41.132 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.131+0000 7fac28a00640 1 --2- 172.21.15.96:0/426957767 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fac240d0860 0x7fac2415f7d0 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7fac2406d8c0 tx=0x7fac1c02a680 comp rx=0 tx=0).stop 2024-05-28T06:17:41.132 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.131+0000 7fac28a00640 1 -- 172.21.15.96:0/426957767 shutdown_connections 2024-05-28T06:17:41.132 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.131+0000 7fac28a00640 1 --2- 172.21.15.96:0/426957767 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fac240d0860 0x7fac2415f7d0 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:41.132 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.131+0000 7fac28a00640 1 --2- 172.21.15.96:0/426957767 >> v2:172.21.15.123:3300/0 conn(0x7fac240ce560 0x7fac240b4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:41.132 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.131+0000 7fac28a00640 1 -- 172.21.15.96:0/426957767 >> 172.21.15.96:0/426957767 conn(0x7fac2406bae0 msgr2=0x7fac2406bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:41.132 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.132+0000 7fac28a00640 1 -- 172.21.15.96:0/426957767 shutdown_connections 2024-05-28T06:17:41.132 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.132+0000 7fac28a00640 1 -- 172.21.15.96:0/426957767 wait complete. 2024-05-28T06:17:41.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.133+0000 7fac28a00640 1 Processor -- start 2024-05-28T06:17:41.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.133+0000 7fac28a00640 1 -- start start 2024-05-28T06:17:41.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.134+0000 7fac28a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fac240ce560 0x7fac24149330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:41.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.134+0000 7fac28a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fac240d0430 0x7fac2409be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:41.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.134+0000 7fac28a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac240d0860 0x7fac2409c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:41.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.134+0000 7fac28a00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fac240d0cd0 con 0x7fac240ce560 2024-05-28T06:17:41.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.134+0000 7fac28a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fac2415c670 con 0x7fac240d0860 2024-05-28T06:17:41.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.134+0000 7fac28a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fac2415fd10 con 0x7fac240d0430 2024-05-28T06:17:41.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.135+0000 7fac23400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fac240ce560 0x7fac24149330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:41.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.135+0000 7fac23400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fac240ce560 0x7fac24149330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49014/0 (socket says 172.21.15.96:49014) 2024-05-28T06:17:41.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.135+0000 7fac23400640 1 -- 172.21.15.96:0/3310676132 learned_addr learned my addr 172.21.15.96:0/3310676132 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:17:41.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.135+0000 7fac22a00640 1 --2- 172.21.15.96:0/3310676132 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fac240d0430 0x7fac2409be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:41.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.135+0000 7fac23400640 1 -- 172.21.15.96:0/3310676132 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fac240d0430 msgr2=0x7fac2409be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:41.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.135+0000 7fac23400640 1 --2- 172.21.15.96:0/3310676132 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fac240d0430 0x7fac2409be70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:41.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.135+0000 7fac23e00640 1 --2- 172.21.15.96:0/3310676132 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac240d0860 0x7fac2409c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:41.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.135+0000 7fac23400640 1 -- 172.21.15.96:0/3310676132 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac240d0860 msgr2=0x7fac2409c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:41.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.135+0000 7fac23400640 1 --2- 172.21.15.96:0/3310676132 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac240d0860 0x7fac2409c3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:41.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.135+0000 7fac23400640 1 -- 172.21.15.96:0/3310676132 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac241e7070 con 0x7fac240ce560 2024-05-28T06:17:41.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.135+0000 7fac23400640 1 --2- 172.21.15.96:0/3310676132 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fac240ce560 0x7fac24149330 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7fac1400c7f0 tx=0x7fac1400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:41.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.136+0000 7fac1b400640 1 -- 172.21.15.96:0/3310676132 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac14015070 con 0x7fac240ce560 2024-05-28T06:17:41.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.136+0000 7fac1b400640 1 -- 172.21.15.96:0/3310676132 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac14007820 con 0x7fac240ce560 2024-05-28T06:17:41.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.136+0000 7fac28a00640 1 -- 172.21.15.96:0/3310676132 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac241e6d90 con 0x7fac240ce560 2024-05-28T06:17:41.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.136+0000 7fac1b400640 1 -- 172.21.15.96:0/3310676132 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac14007bd0 con 0x7fac240ce560 2024-05-28T06:17:41.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.136+0000 7fac28a00640 1 -- 172.21.15.96:0/3310676132 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac2409b040 con 0x7fac240ce560 2024-05-28T06:17:41.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.137+0000 7fac28a00640 1 -- 172.21.15.96:0/3310676132 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac240689e0 con 0x7fac240ce560 2024-05-28T06:17:41.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.138+0000 7fac1b400640 1 -- 172.21.15.96:0/3310676132 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 29) ==== 92672+0+0 (secure 0 0 0) 0x7fac14018070 con 0x7fac240ce560 2024-05-28T06:17:41.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.139+0000 7fac1b400640 1 --2- 172.21.15.96:0/3310676132 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fac0406daf0 0x7fac0406ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:41.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.139+0000 7fac1b400640 1 -- 172.21.15.96:0/3310676132 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5795+0+0 (secure 0 0 0) 0x7fac1408ab70 con 0x7fac240ce560 2024-05-28T06:17:41.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.139+0000 7fac22a00640 1 --2- 172.21.15.96:0/3310676132 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fac0406daf0 0x7fac0406ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:41.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.141+0000 7fac22a00640 1 --2- 172.21.15.96:0/3310676132 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fac0406daf0 0x7fac0406ffb0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fac241e77b0 tx=0x7fac10009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:41.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.143+0000 7fac1b400640 1 -- 172.21.15.96:0/3310676132 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fac1405a680 con 0x7fac240ce560 2024-05-28T06:17:41.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.468+0000 7fac28a00640 1 -- 172.21.15.96:0/3310676132 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone436318", "target": ["mon-mgr", ""]}) -- 0x7fac241e7c60 con 0x7fac0406daf0 2024-05-28T06:17:41.495 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.498+0000 7fac1b400640 1 -- 172.21.15.96:0/3310676132 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fac241e7c60 con 0x7fac0406daf0 2024-05-28T06:17:41.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.501+0000 7fac28a00640 1 -- 172.21.15.96:0/3310676132 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fac0406daf0 msgr2=0x7fac0406ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:41.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.501+0000 7fac28a00640 1 --2- 172.21.15.96:0/3310676132 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fac0406daf0 0x7fac0406ffb0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fac241e77b0 tx=0x7fac10009040 comp rx=0 tx=0).stop 2024-05-28T06:17:41.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.501+0000 7fac28a00640 1 -- 172.21.15.96:0/3310676132 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fac240ce560 msgr2=0x7fac24149330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:41.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.501+0000 7fac28a00640 1 --2- 172.21.15.96:0/3310676132 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fac240ce560 0x7fac24149330 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7fac1400c7f0 tx=0x7fac1400ccb0 comp rx=0 tx=0).stop 2024-05-28T06:17:41.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.501+0000 7fac28a00640 1 -- 172.21.15.96:0/3310676132 shutdown_connections 2024-05-28T06:17:41.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.501+0000 7fac28a00640 1 --2- 172.21.15.96:0/3310676132 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fac0406daf0 0x7fac0406ffb0 secure :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fac241e77b0 tx=0x7fac10009040 comp rx=0 tx=0).stop 2024-05-28T06:17:41.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.501+0000 7fac28a00640 1 --2- 172.21.15.96:0/3310676132 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac240d0860 0x7fac2409c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:41.501 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.501+0000 7fac28a00640 1 --2- 172.21.15.96:0/3310676132 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fac240d0430 0x7fac2409be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:41.501 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.501+0000 7fac28a00640 1 --2- 172.21.15.96:0/3310676132 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fac240ce560 0x7fac24149330 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:41.501 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.501+0000 7fac28a00640 1 -- 172.21.15.96:0/3310676132 >> 172.21.15.96:0/3310676132 conn(0x7fac2406bae0 msgr2=0x7fac24158460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:41.501 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.502+0000 7fac28a00640 1 -- 172.21.15.96:0/3310676132 shutdown_connections 2024-05-28T06:17:41.501 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.502+0000 7fac28a00640 1 -- 172.21.15.96:0/3310676132 wait complete. 2024-05-28T06:17:41.525 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone43634 2024-05-28T06:17:41.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.775+0000 7fa51f000640 1 Processor -- start 2024-05-28T06:17:41.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.775+0000 7fa51f000640 1 -- start start 2024-05-28T06:17:41.774 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.776+0000 7fa51f000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fa51815e870 0x7fa5180ca840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:41.774 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.776+0000 7fa51f000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fa5180cd090 0x7fa5180cfa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:41.774 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.776+0000 7fa51f000640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5180b0170 con 0x7fa5180ccc00 2024-05-28T06:17:41.774 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.776+0000 7fa51f000640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fa5180ce8d0 con 0x7fa51815e870 2024-05-28T06:17:41.774 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.776+0000 7fa51f000640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fa5180b03e0 con 0x7fa5180cd090 2024-05-28T06:17:41.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.776+0000 7fa51dc00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fa51815e870 0x7fa5180ca840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:41.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.777+0000 7fa51dc00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fa51815e870 0x7fa5180ca840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49026/0 (socket says 172.21.15.96:49026) 2024-05-28T06:17:41.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.777+0000 7fa51dc00640 1 -- 172.21.15.96:0/1665642722 learned_addr learned my addr 172.21.15.96:0/1665642722 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:17:41.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.777+0000 7fa51e600640 1 --2- 172.21.15.96:0/1665642722 >> v2:172.21.15.123:3300/0 conn(0x7fa5180cd090 0x7fa5180cfa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:41.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.777+0000 7fa51dc00640 1 -- 172.21.15.96:0/1665642722 >> v2:172.21.15.123:3300/0 conn(0x7fa5180cd090 msgr2=0x7fa5180cfa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:41.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.777+0000 7fa51dc00640 1 --2- 172.21.15.96:0/1665642722 >> v2:172.21.15.123:3300/0 conn(0x7fa5180cd090 0x7fa5180cfa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:41.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.777+0000 7fa51dc00640 1 -- 172.21.15.96:0/1665642722 >> v1:172.21.15.123:6789/0 conn(0x7fa5180ccc00 legacy=0x7fa5180cf300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:41.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.777+0000 7fa51dc00640 1 -- 172.21.15.96:0/1665642722 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5180ceb00 con 0x7fa51815e870 2024-05-28T06:17:41.777 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.779+0000 7fa51dc00640 1 --2- 172.21.15.96:0/1665642722 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa51815e870 0x7fa5180ca840 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7fa5180afb60 tx=0x7fa50c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89eb2bf0b5791e08 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:41.777 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.779+0000 7fa517e00640 1 -- 172.21.15.96:0/1665642722 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa50c037070 con 0x7fa51815e870 2024-05-28T06:17:41.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.779+0000 7fa517e00640 1 -- 172.21.15.96:0/1665642722 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa50c032440 con 0x7fa51815e870 2024-05-28T06:17:41.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.779+0000 7fa517e00640 1 -- 172.21.15.96:0/1665642722 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa50c032810 con 0x7fa51815e870 2024-05-28T06:17:41.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.781+0000 7fa51f000640 1 -- 172.21.15.96:0/1665642722 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa51815e870 msgr2=0x7fa5180ca840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:41.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.781+0000 7fa51f000640 1 --2- 172.21.15.96:0/1665642722 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa51815e870 0x7fa5180ca840 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7fa5180afb60 tx=0x7fa50c02a270 comp rx=0 tx=0).stop 2024-05-28T06:17:41.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.781+0000 7fa51f000640 1 -- 172.21.15.96:0/1665642722 shutdown_connections 2024-05-28T06:17:41.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.781+0000 7fa51f000640 1 --2- 172.21.15.96:0/1665642722 >> v2:172.21.15.123:3300/0 conn(0x7fa5180cd090 0x7fa5180cfa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:41.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.781+0000 7fa51f000640 1 --2- 172.21.15.96:0/1665642722 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa51815e870 0x7fa5180ca840 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:41.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.781+0000 7fa51f000640 1 -- 172.21.15.96:0/1665642722 >> 172.21.15.96:0/1665642722 conn(0x7fa5180b1ec0 msgr2=0x7fa5180b22d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:41.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.781+0000 7fa51f000640 1 -- 172.21.15.96:0/1665642722 shutdown_connections 2024-05-28T06:17:41.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.781+0000 7fa51f000640 1 -- 172.21.15.96:0/1665642722 wait complete. 2024-05-28T06:17:41.781 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.783+0000 7fa51f000640 1 Processor -- start 2024-05-28T06:17:41.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.784+0000 7fa51f000640 1 -- start start 2024-05-28T06:17:41.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.785+0000 7fa51f000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa5180ccc00 0x7fa518158150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:41.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.785+0000 7fa51f000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa5180cd090 0x7fa518158690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:41.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.785+0000 7fa51f000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa51815e870 0x7fa5181562e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:41.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.785+0000 7fa51f000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa5180cff50 con 0x7fa5180cd090 2024-05-28T06:17:41.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.785+0000 7fa51f000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa5180cd4a0 con 0x7fa51815e870 2024-05-28T06:17:41.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.785+0000 7fa51f000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fa5180726a0 con 0x7fa5180ccc00 2024-05-28T06:17:41.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.786+0000 7fa51d200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa5180cd090 0x7fa518158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:41.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.786+0000 7fa51e600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa51815e870 0x7fa5181562e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:41.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.786+0000 7fa51e600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa51815e870 0x7fa5181562e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.96:56776/0 (socket says 172.21.15.96:56776) 2024-05-28T06:17:41.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.786+0000 7fa51d200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa5180cd090 0x7fa518158690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49030/0 (socket says 172.21.15.96:49030) 2024-05-28T06:17:41.786 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.786+0000 7fa51e600640 1 -- 172.21.15.96:0/2329998782 learned_addr learned my addr 172.21.15.96:0/2329998782 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:17:41.786 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.786+0000 7fa51dc00640 1 --2- 172.21.15.96:0/2329998782 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa5180ccc00 0x7fa518158150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:41.786 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.787+0000 7fa51d200640 1 -- 172.21.15.96:0/2329998782 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa5180ccc00 msgr2=0x7fa518158150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:41.786 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.787+0000 7fa51d200640 1 --2- 172.21.15.96:0/2329998782 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa5180ccc00 0x7fa518158150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:41.786 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.787+0000 7fa51d200640 1 -- 172.21.15.96:0/2329998782 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa51815e870 msgr2=0x7fa5181562e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:41.786 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.787+0000 7fa51d200640 1 --2- 172.21.15.96:0/2329998782 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa51815e870 0x7fa5181562e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:41.786 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.787+0000 7fa51d200640 1 -- 172.21.15.96:0/2329998782 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa518154a00 con 0x7fa5180cd090 2024-05-28T06:17:41.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.787+0000 7fa51d200640 1 --2- 172.21.15.96:0/2329998782 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa5180cd090 0x7fa518158690 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7fa5180755b0 tx=0x7fa50400c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:41.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.788+0000 7fa515600640 1 -- 172.21.15.96:0/2329998782 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa504017070 con 0x7fa5180cd090 2024-05-28T06:17:41.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.788+0000 7fa515600640 1 -- 172.21.15.96:0/2329998782 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5040052d0 con 0x7fa5180cd090 2024-05-28T06:17:41.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.788+0000 7fa51f000640 1 -- 172.21.15.96:0/2329998782 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa518158ca0 con 0x7fa5180cd090 2024-05-28T06:17:41.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.788+0000 7fa515600640 1 -- 172.21.15.96:0/2329998782 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa504003850 con 0x7fa5180cd090 2024-05-28T06:17:41.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.788+0000 7fa51f000640 1 -- 172.21.15.96:0/2329998782 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa518159260 con 0x7fa5180cd090 2024-05-28T06:17:41.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.791+0000 7fa515600640 1 -- 172.21.15.96:0/2329998782 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 29) ==== 92672+0+0 (secure 0 0 0) 0x7fa50401a070 con 0x7fa5180cd090 2024-05-28T06:17:41.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.793+0000 7fa51f000640 1 -- 172.21.15.96:0/2329998782 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa518067bf0 con 0x7fa5180cd090 2024-05-28T06:17:41.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.793+0000 7fa515600640 1 --2- 172.21.15.96:0/2329998782 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa4f806daf0 0x7fa4f806ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:41.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.793+0000 7fa515600640 1 -- 172.21.15.96:0/2329998782 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 5851+0+0 (secure 0 0 0) 0x7fa504089d10 con 0x7fa5180cd090 2024-05-28T06:17:41.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.794+0000 7fa51dc00640 1 --2- 172.21.15.96:0/2329998782 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa4f806daf0 0x7fa4f806ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:41.793 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.796+0000 7fa51dc00640 1 --2- 172.21.15.96:0/2329998782 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa4f806daf0 0x7fa4f806ffb0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fa50c034000 tx=0x7fa50c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:41.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:41.802+0000 7fa515600640 1 -- 172.21.15.96:0/2329998782 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa518067bf0 con 0x7fa5180cd090 2024-05-28T06:17:42.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.092+0000 7fa51f000640 1 -- 172.21.15.96:0/2329998782 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone43634", "target": ["mon-mgr", ""]}) -- 0x7fa5180689e0 con 0x7fa4f806daf0 2024-05-28T06:17:42.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.136+0000 7fa515600640 1 -- 172.21.15.96:0/2329998782 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fa5180689e0 con 0x7fa4f806daf0 2024-05-28T06:17:42.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.139+0000 7fa51f000640 1 -- 172.21.15.96:0/2329998782 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa4f806daf0 msgr2=0x7fa4f806ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:42.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.139+0000 7fa51f000640 1 --2- 172.21.15.96:0/2329998782 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa4f806daf0 0x7fa4f806ffb0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fa50c034000 tx=0x7fa50c02e000 comp rx=0 tx=0).stop 2024-05-28T06:17:42.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.139+0000 7fa51f000640 1 -- 172.21.15.96:0/2329998782 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa5180cd090 msgr2=0x7fa518158690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:42.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.139+0000 7fa51f000640 1 --2- 172.21.15.96:0/2329998782 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa5180cd090 0x7fa518158690 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7fa5180755b0 tx=0x7fa50400c820 comp rx=0 tx=0).stop 2024-05-28T06:17:42.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.139+0000 7fa51f000640 1 -- 172.21.15.96:0/2329998782 shutdown_connections 2024-05-28T06:17:42.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.139+0000 7fa51f000640 1 --2- 172.21.15.96:0/2329998782 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa4f806daf0 0x7fa4f806ffb0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:42.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.139+0000 7fa51f000640 1 --2- 172.21.15.96:0/2329998782 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa51815e870 0x7fa5181562e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:42.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.139+0000 7fa51f000640 1 --2- 172.21.15.96:0/2329998782 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa5180cd090 0x7fa518158690 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:42.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.139+0000 7fa51f000640 1 --2- 172.21.15.96:0/2329998782 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa5180ccc00 0x7fa518158150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:42.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.139+0000 7fa51f000640 1 -- 172.21.15.96:0/2329998782 >> 172.21.15.96:0/2329998782 conn(0x7fa5180b1ec0 msgr2=0x7fa51806dbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:42.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.139+0000 7fa51f000640 1 -- 172.21.15.96:0/2329998782 shutdown_connections 2024-05-28T06:17:42.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.139+0000 7fa51f000640 1 -- 172.21.15.96:0/2329998782 wait complete. 2024-05-28T06:17:42.154 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone436310 2024-05-28T06:17:42.377 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.380+0000 7f41b8600640 1 Processor -- start 2024-05-28T06:17:42.378 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.380+0000 7f41b8600640 1 -- start start 2024-05-28T06:17:42.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.381+0000 7f41b8600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f41b0162c70 0x7f41b00cdb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:42.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.381+0000 7f41b8600640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41b009bc50 con 0x7f41b015fbb0 2024-05-28T06:17:42.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.381+0000 7f41b8600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41b006cf10 con 0x7f41b015e7c0 2024-05-28T06:17:42.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.381+0000 7f41b8600640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f41b006d1f0 con 0x7f41b0162c70 2024-05-28T06:17:42.380 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.382+0000 7f41b5c00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f41b015e7c0 0x7f41b009b540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.96:46368/0 (socket says 172.21.15.96:46368) 2024-05-28T06:17:42.380 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.382+0000 7f41b5c00640 1 -- 172.21.15.96:0/416270845 learned_addr learned my addr 172.21.15.96:0/416270845 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:17:42.380 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.382+0000 7f41b6600640 1 --2- 172.21.15.96:0/416270845 >> v2:172.21.15.96:3300/0 conn(0x7f41b0162c70 0x7f41b00cdb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:42.380 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.382+0000 7f41afe00640 1 -- 172.21.15.96:0/416270845 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2258428265 0 0) 0x7f41b006cf10 con 0x7f41b015e7c0 2024-05-28T06:17:42.380 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.382+0000 7f41afe00640 1 -- 172.21.15.96:0/416270845 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4198003520 con 0x7f41b015e7c0 2024-05-28T06:17:42.381 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.382+0000 7f41afe00640 1 -- 172.21.15.96:0/416270845 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2079514229 0 0) 0x7f41b009bc50 con 0x7f41b015fbb0 2024-05-28T06:17:42.381 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.382+0000 7f41b6600640 1 -- 172.21.15.96:0/416270845 >> v1:172.21.15.123:6789/0 conn(0x7f41b015e7c0 legacy=0x7f41b009b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:42.381 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.382+0000 7f41b6600640 1 -- 172.21.15.96:0/416270845 >> v1:172.21.15.96:6789/0 conn(0x7f41b015fbb0 legacy=0x7f41b0162560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:42.382 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.382+0000 7f41b6600640 1 -- 172.21.15.96:0/416270845 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41b006d4d0 con 0x7f41b0162c70 2024-05-28T06:17:42.382 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.384+0000 7f41b6600640 1 --2- 172.21.15.96:0/416270845 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41b0162c70 0x7f41b00cdb60 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f41a8009d90 tx=0x7f41a802a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a9f4fed17f5380 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:42.382 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.385+0000 7f41afe00640 1 -- 172.21.15.96:0/416270845 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41a8007740 con 0x7f41b0162c70 2024-05-28T06:17:42.382 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.385+0000 7f41afe00640 1 -- 172.21.15.96:0/416270845 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41a80078e0 con 0x7f41b0162c70 2024-05-28T06:17:42.382 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.385+0000 7f41afe00640 1 -- 172.21.15.96:0/416270845 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41a8007cb0 con 0x7f41b0162c70 2024-05-28T06:17:42.383 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.386+0000 7f41b8600640 1 -- 172.21.15.96:0/416270845 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41b0162c70 msgr2=0x7f41b00cdb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:42.384 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.386+0000 7f41b8600640 1 --2- 172.21.15.96:0/416270845 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41b0162c70 0x7f41b00cdb60 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f41a8009d90 tx=0x7f41a802a660 comp rx=0 tx=0).stop 2024-05-28T06:17:42.384 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.386+0000 7f41b8600640 1 -- 172.21.15.96:0/416270845 shutdown_connections 2024-05-28T06:17:42.384 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.386+0000 7f41b8600640 1 --2- 172.21.15.96:0/416270845 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41b0162c70 0x7f41b00cdb60 unknown :-1 s=CLOSED pgs=1282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:42.384 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.386+0000 7f41b8600640 1 -- 172.21.15.96:0/416270845 >> 172.21.15.96:0/416270845 conn(0x7f41b00b3f80 msgr2=0x7f41b00b4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:42.384 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.387+0000 7f41b8600640 1 -- 172.21.15.96:0/416270845 shutdown_connections 2024-05-28T06:17:42.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.387+0000 7f41b8600640 1 -- 172.21.15.96:0/416270845 wait complete. 2024-05-28T06:17:42.387 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.389+0000 7f41b8600640 1 Processor -- start 2024-05-28T06:17:42.387 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.390+0000 7f41b8600640 1 -- start start 2024-05-28T06:17:42.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.391+0000 7f41b8600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f41b015e7c0 0x7f41b00b0500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:42.389 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.391+0000 7f41b8600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41b015fbb0 0x7f41b00c9560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:42.389 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.391+0000 7f41b8600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41b0162c70 0x7f41b014e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:42.389 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.391+0000 7f41b8600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f41b00d1810 con 0x7f41b0162c70 2024-05-28T06:17:42.389 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.391+0000 7f41b8600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f41b006da90 con 0x7f41b015fbb0 2024-05-28T06:17:42.389 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.391+0000 7f41b8600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f41b00d1990 con 0x7f41b015e7c0 2024-05-28T06:17:42.391 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.392+0000 7f41b5200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41b015fbb0 0x7f41b00c9560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:42.391 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.392+0000 7f41b5200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41b015fbb0 0x7f41b00c9560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.96:56790/0 (socket says 172.21.15.96:56790) 2024-05-28T06:17:42.391 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.392+0000 7f41b5200640 1 -- 172.21.15.96:0/431683763 learned_addr learned my addr 172.21.15.96:0/431683763 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:17:42.391 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.392+0000 7f41b5c00640 1 --2- 172.21.15.96:0/431683763 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f41b015e7c0 0x7f41b00b0500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:42.391 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.392+0000 7f41b5200640 1 -- 172.21.15.96:0/431683763 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f41b015e7c0 msgr2=0x7f41b00b0500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:42.391 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.392+0000 7f41b5200640 1 --2- 172.21.15.96:0/431683763 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f41b015e7c0 0x7f41b00b0500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:42.391 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.392+0000 7f41b5200640 1 -- 172.21.15.96:0/431683763 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41b0162c70 msgr2=0x7f41b014e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:42.391 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.392+0000 7f41b5200640 1 --2- 172.21.15.96:0/431683763 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41b0162c70 0x7f41b014e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:42.391 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.392+0000 7f41b5200640 1 -- 172.21.15.96:0/431683763 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41b00cc370 con 0x7f41b015fbb0 2024-05-28T06:17:42.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.393+0000 7f41b5200640 1 --2- 172.21.15.96:0/431683763 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41b015fbb0 0x7f41b00c9560 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f419c00c410 tx=0x7f419c00c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:42.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.393+0000 7f41ad600640 1 -- 172.21.15.96:0/431683763 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f419c017070 con 0x7f41b015fbb0 2024-05-28T06:17:42.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.393+0000 7f41ad600640 1 -- 172.21.15.96:0/431683763 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f419c0035f0 con 0x7f41b015fbb0 2024-05-28T06:17:42.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.393+0000 7f41ad600640 1 -- 172.21.15.96:0/431683763 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f419c003a20 con 0x7f41b015fbb0 2024-05-28T06:17:42.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.393+0000 7f41b8600640 1 -- 172.21.15.96:0/431683763 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41b00cb5b0 con 0x7f41b015fbb0 2024-05-28T06:17:42.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.393+0000 7f41b8600640 1 -- 172.21.15.96:0/431683763 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41b00cbb70 con 0x7f41b015fbb0 2024-05-28T06:17:42.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.395+0000 7f41b8600640 1 -- 172.21.15.96:0/431683763 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41b00689e0 con 0x7f41b015fbb0 2024-05-28T06:17:42.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.397+0000 7f41ad600640 1 -- 172.21.15.96:0/431683763 <== mon.1 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 29) ==== 92672+0+0 (secure 0 0 0) 0x7f419c01a070 con 0x7f41b015fbb0 2024-05-28T06:17:42.395 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.398+0000 7f41ad600640 1 --2- 172.21.15.96:0/431683763 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f419006db40 0x7f4190070000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:42.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.398+0000 7f41ad600640 1 -- 172.21.15.96:0/431683763 <== mon.1 v2:172.21.15.123:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 5851+0+0 (secure 0 0 0) 0x7f419c08a320 con 0x7f41b015fbb0 2024-05-28T06:17:42.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.398+0000 7f41b5c00640 1 --2- 172.21.15.96:0/431683763 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f419006db40 0x7f4190070000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:42.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.399+0000 7f41b5c00640 1 --2- 172.21.15.96:0/431683763 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f419006db40 0x7f4190070000 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f41a40011e0 tx=0x7f41a4016040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:42.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.404+0000 7f41ad600640 1 -- 172.21.15.96:0/431683763 <== mon.1 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f419c059820 con 0x7f41b015fbb0 2024-05-28T06:17:42.743 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.744+0000 7f41b8600640 1 -- 172.21.15.96:0/431683763 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone436310", "target": ["mon-mgr", ""]}) -- 0x7f41b00d1fb0 con 0x7f419006db40 2024-05-28T06:17:42.768 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.769+0000 7f41ad600640 1 -- 172.21.15.96:0/431683763 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f41b00d1fb0 con 0x7f419006db40 2024-05-28T06:17:42.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.772+0000 7f41b8600640 1 -- 172.21.15.96:0/431683763 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f419006db40 msgr2=0x7f4190070000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:42.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.772+0000 7f41b8600640 1 --2- 172.21.15.96:0/431683763 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f419006db40 0x7f4190070000 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f41a40011e0 tx=0x7f41a4016040 comp rx=0 tx=0).stop 2024-05-28T06:17:42.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.772+0000 7f41b8600640 1 -- 172.21.15.96:0/431683763 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41b015fbb0 msgr2=0x7f41b00c9560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:42.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.772+0000 7f41b8600640 1 --2- 172.21.15.96:0/431683763 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41b015fbb0 0x7f41b00c9560 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f419c00c410 tx=0x7f419c00c8d0 comp rx=0 tx=0).stop 2024-05-28T06:17:42.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.772+0000 7f41b8600640 1 -- 172.21.15.96:0/431683763 shutdown_connections 2024-05-28T06:17:42.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.772+0000 7f41b8600640 1 --2- 172.21.15.96:0/431683763 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f419006db40 0x7f4190070000 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:42.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.772+0000 7f41b8600640 1 --2- 172.21.15.96:0/431683763 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41b0162c70 0x7f41b014e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:42.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.772+0000 7f41b8600640 1 --2- 172.21.15.96:0/431683763 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41b015fbb0 0x7f41b00c9560 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:42.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.772+0000 7f41b8600640 1 --2- 172.21.15.96:0/431683763 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f41b015e7c0 0x7f41b00b0500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:42.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.772+0000 7f41b8600640 1 -- 172.21.15.96:0/431683763 >> 172.21.15.96:0/431683763 conn(0x7f41b00b3f80 msgr2=0x7f41b009b210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:42.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.772+0000 7f41b8600640 1 -- 172.21.15.96:0/431683763 shutdown_connections 2024-05-28T06:17:42.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.772+0000 7f41b8600640 1 -- 172.21.15.96:0/431683763 wait complete. 2024-05-28T06:17:42.789 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone436314 2024-05-28T06:17:42.966 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.967+0000 7f3cf7e00640 1 Processor -- start 2024-05-28T06:17:42.967 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.968+0000 7f3cf7e00640 1 -- start start 2024-05-28T06:17:42.968 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.969+0000 7f3cf7e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f3cf815e5a0 0x7f3cf80d2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:42.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.969+0000 7f3cf7e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3cf80cfb40 0x7f3cf80d0d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:42.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.969+0000 7f3cf7e00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3cf80d3500 con 0x7f3cf80cffa0 2024-05-28T06:17:42.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.969+0000 7f3cf7e00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f3cf80d2b30 con 0x7f3cf815e5a0 2024-05-28T06:17:42.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.969+0000 7f3cf7e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f3cf80d2200 con 0x7f3cf80cfb40 2024-05-28T06:17:42.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.969+0000 7f3cf6a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f3cf815e5a0 0x7f3cf80d2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:42.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.969+0000 7f3cf7400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f3cf80cffa0 0x7f3cf806c740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:40092/0 (socket says 172.21.15.96:40092) 2024-05-28T06:17:42.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.969+0000 7f3cf7400640 1 -- 172.21.15.96:0/32457242 learned_addr learned my addr 172.21.15.96:0/32457242 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:17:42.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.970+0000 7f3cf6000640 1 --2- 172.21.15.96:0/32457242 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3cf80cfb40 0x7f3cf80d0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:42.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.970+0000 7f3cf6a00640 1 -- 172.21.15.96:0/32457242 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3cf80cfb40 msgr2=0x7f3cf80d0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:42.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.970+0000 7f3cf6a00640 1 --2- 172.21.15.96:0/32457242 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3cf80cfb40 0x7f3cf80d0d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:42.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.970+0000 7f3cf6a00640 1 -- 172.21.15.96:0/32457242 >> v1:172.21.15.96:6789/0 conn(0x7f3cf80cffa0 legacy=0x7f3cf806c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:42.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.970+0000 7f3cf6a00640 1 -- 172.21.15.96:0/32457242 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cf80d24e0 con 0x7f3cf815e5a0 2024-05-28T06:17:42.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.970+0000 7f3cf6000640 1 --2- 172.21.15.96:0/32457242 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3cf80cfb40 0x7f3cf80d0d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:17:42.970 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.971+0000 7f3cf6a00640 1 --2- 172.21.15.96:0/32457242 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3cf815e5a0 0x7f3cf80d2fc0 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f3ce80097a0 tx=0x7f3ce802a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=368382a08ddc9f1c server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:42.970 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.971+0000 7f3cf5600640 1 -- 172.21.15.96:0/32457242 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ce8037070 con 0x7f3cf815e5a0 2024-05-28T06:17:42.970 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.971+0000 7f3cf5600640 1 -- 172.21.15.96:0/32457242 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ce8032440 con 0x7f3cf815e5a0 2024-05-28T06:17:42.970 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.971+0000 7f3cf5600640 1 -- 172.21.15.96:0/32457242 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ce8032810 con 0x7f3cf815e5a0 2024-05-28T06:17:42.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.972+0000 7f3cf7e00640 1 -- 172.21.15.96:0/32457242 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3cf815e5a0 msgr2=0x7f3cf80d2fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:42.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.972+0000 7f3cf7e00640 1 --2- 172.21.15.96:0/32457242 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3cf815e5a0 0x7f3cf80d2fc0 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f3ce80097a0 tx=0x7f3ce802a180 comp rx=0 tx=0).stop 2024-05-28T06:17:42.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.972+0000 7f3cf7e00640 1 -- 172.21.15.96:0/32457242 shutdown_connections 2024-05-28T06:17:42.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.972+0000 7f3cf7e00640 1 --2- 172.21.15.96:0/32457242 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3cf80cfb40 0x7f3cf80d0d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:42.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.972+0000 7f3cf7e00640 1 --2- 172.21.15.96:0/32457242 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3cf815e5a0 0x7f3cf80d2fc0 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:42.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.972+0000 7f3cf7e00640 1 -- 172.21.15.96:0/32457242 >> 172.21.15.96:0/32457242 conn(0x7f3cf808bfc0 msgr2=0x7f3cf80d44b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:42.972 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.972+0000 7f3cf7e00640 1 -- 172.21.15.96:0/32457242 shutdown_connections 2024-05-28T06:17:42.972 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.972+0000 7f3cf7e00640 1 -- 172.21.15.96:0/32457242 wait complete. 2024-05-28T06:17:42.972 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.974+0000 7f3cf7e00640 1 Processor -- start 2024-05-28T06:17:42.972 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.974+0000 7f3cf7e00640 1 -- start start 2024-05-28T06:17:42.973 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.975+0000 7f3cf7e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3cf80cfb40 0x7f3cf80b9e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:42.973 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.975+0000 7f3cf7e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3cf80cffa0 0x7f3cf8153c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:42.974 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.975+0000 7f3cf7e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3cf815e5a0 0x7f3cf80c8d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:42.974 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.975+0000 7f3cf7e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f3cf815eb00 con 0x7f3cf80cffa0 2024-05-28T06:17:42.974 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.975+0000 7f3cf7e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f3cf815e980 con 0x7f3cf80cfb40 2024-05-28T06:17:42.974 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.975+0000 7f3cf7e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f3cf80d5220 con 0x7f3cf815e5a0 2024-05-28T06:17:42.974 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.975+0000 7f3cf6000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3cf80cffa0 0x7f3cf8153c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:42.974 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.975+0000 7f3cf6000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3cf80cffa0 0x7f3cf8153c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49058/0 (socket says 172.21.15.96:49058) 2024-05-28T06:17:42.974 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.975+0000 7f3cf6000640 1 -- 172.21.15.96:0/2669148119 learned_addr learned my addr 172.21.15.96:0/2669148119 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:17:42.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.975+0000 7f3cf6000640 1 -- 172.21.15.96:0/2669148119 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3cf815e5a0 msgr2=0x7f3cf80c8d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:42.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.975+0000 7f3cf6000640 1 --2- 172.21.15.96:0/2669148119 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3cf815e5a0 0x7f3cf80c8d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:42.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.975+0000 7f3cf6000640 1 -- 172.21.15.96:0/2669148119 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3cf80cfb40 msgr2=0x7f3cf80b9e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:42.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.975+0000 7f3cf6000640 1 --2- 172.21.15.96:0/2669148119 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3cf80cfb40 0x7f3cf80b9e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:42.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.975+0000 7f3cf6000640 1 -- 172.21.15.96:0/2669148119 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cf80c9b40 con 0x7f3cf80cffa0 2024-05-28T06:17:42.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.975+0000 7f3cf6000640 1 --2- 172.21.15.96:0/2669148119 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3cf80cffa0 0x7f3cf8153c60 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f3ce000c7f0 tx=0x7f3ce000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:42.976 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.976+0000 7f3ceea00640 1 -- 172.21.15.96:0/2669148119 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ce0015070 con 0x7f3cf80cffa0 2024-05-28T06:17:42.976 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.976+0000 7f3ceea00640 1 -- 172.21.15.96:0/2669148119 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ce0007a30 con 0x7f3cf80cffa0 2024-05-28T06:17:42.976 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.976+0000 7f3ceea00640 1 -- 172.21.15.96:0/2669148119 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ce0007e00 con 0x7f3cf80cffa0 2024-05-28T06:17:42.976 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.976+0000 7f3cf7e00640 1 -- 172.21.15.96:0/2669148119 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3cf80c9580 con 0x7f3cf80cffa0 2024-05-28T06:17:42.976 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.976+0000 7f3cf7e00640 1 -- 172.21.15.96:0/2669148119 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3cf80b5000 con 0x7f3cf80cffa0 2024-05-28T06:17:42.976 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.977+0000 7f3cf7e00640 1 -- 172.21.15.96:0/2669148119 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3cf8073460 con 0x7f3cf80cffa0 2024-05-28T06:17:42.976 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.978+0000 7f3ceea00640 1 -- 172.21.15.96:0/2669148119 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 29) ==== 92672+0+0 (secure 0 0 0) 0x7f3ce0018070 con 0x7f3cf80cffa0 2024-05-28T06:17:42.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.979+0000 7f3ceea00640 1 --2- 172.21.15.96:0/2669148119 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3cd406daf0 0x7f3cd406ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:42.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.979+0000 7f3ceea00640 1 -- 172.21.15.96:0/2669148119 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 5851+0+0 (secure 0 0 0) 0x7f3ce008a9c0 con 0x7f3cf80cffa0 2024-05-28T06:17:42.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.979+0000 7f3cf6a00640 1 --2- 172.21.15.96:0/2669148119 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3cd406daf0 0x7f3cd406ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:42.979 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.981+0000 7f3cf6a00640 1 --2- 172.21.15.96:0/2669148119 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3cd406daf0 0x7f3cd406ffb0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f3ce8034000 tx=0x7f3ce802c000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:42.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:42.983+0000 7f3ceea00640 1 -- 172.21.15.96:0/2669148119 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3cf8073460 con 0x7f3cf80cffa0 2024-05-28T06:17:43.231 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:43.232+0000 7f3cf7e00640 1 -- 172.21.15.96:0/2669148119 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone436314", "target": ["mon-mgr", ""]}) -- 0x7f3cf80c7720 con 0x7f3cd406daf0 2024-05-28T06:17:43.289 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:43.290+0000 7f3ceea00640 1 -- 172.21.15.96:0/2669148119 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f3cf80c7720 con 0x7f3cd406daf0 2024-05-28T06:17:43.291 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:43.293+0000 7f3cf7e00640 1 -- 172.21.15.96:0/2669148119 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3cd406daf0 msgr2=0x7f3cd406ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:43.292 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:43.293+0000 7f3cf7e00640 1 --2- 172.21.15.96:0/2669148119 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3cd406daf0 0x7f3cd406ffb0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f3ce8034000 tx=0x7f3ce802c000 comp rx=0 tx=0).stop 2024-05-28T06:17:43.292 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:43.293+0000 7f3cf7e00640 1 -- 172.21.15.96:0/2669148119 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3cf80cffa0 msgr2=0x7f3cf8153c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:43.292 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:43.293+0000 7f3cf7e00640 1 --2- 172.21.15.96:0/2669148119 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3cf80cffa0 0x7f3cf8153c60 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f3ce000c7f0 tx=0x7f3ce000cbf0 comp rx=0 tx=0).stop 2024-05-28T06:17:43.292 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:43.293+0000 7f3cf7e00640 1 -- 172.21.15.96:0/2669148119 shutdown_connections 2024-05-28T06:17:43.292 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:43.293+0000 7f3cf7e00640 1 --2- 172.21.15.96:0/2669148119 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3cd406daf0 0x7f3cd406ffb0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:43.293 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:43.293+0000 7f3cf7e00640 1 --2- 172.21.15.96:0/2669148119 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3cf815e5a0 0x7f3cf80c8d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:43.293 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:43.293+0000 7f3cf7e00640 1 --2- 172.21.15.96:0/2669148119 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3cf80cffa0 0x7f3cf8153c60 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:43.293 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:43.293+0000 7f3cf7e00640 1 --2- 172.21.15.96:0/2669148119 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3cf80cfb40 0x7f3cf80b9e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:43.293 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:43.293+0000 7f3cf7e00640 1 -- 172.21.15.96:0/2669148119 >> 172.21.15.96:0/2669148119 conn(0x7f3cf808bfc0 msgr2=0x7f3cf8154360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:43.293 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:43.293+0000 7f3cf7e00640 1 -- 172.21.15.96:0/2669148119 shutdown_connections 2024-05-28T06:17:43.293 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:43.294+0000 7f3cf7e00640 1 -- 172.21.15.96:0/2669148119 wait complete. 2024-05-28T06:17:43.317 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone43633 2024-05-28T06:17:43.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:43.569+0000 7f8768200640 1 Processor -- start 2024-05-28T06:17:43.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:43.570+0000 7f8768200640 1 -- start start 2024-05-28T06:17:43.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:43.571+0000 7f8768200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f876015f0d0 0x7f876015f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:43.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:43.572+0000 7f8768200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f876015f9f0 0x7f876015b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:43.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:43.572+0000 7f8768200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8760162440 con 0x7f876015b880 2024-05-28T06:17:43.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:43.572+0000 7f8768200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f87600af9b0 con 0x7f876015f0d0 2024-05-28T06:17:43.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:43.572+0000 7f8768200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f87600aeda0 con 0x7f876015f9f0 2024-05-28T06:17:43.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:43.572+0000 7f8764e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f876015f9f0 0x7f876015b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:43.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:43.573+0000 7f8764e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f876015f9f0 0x7f876015b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.96:56796/0 (socket says 172.21.15.96:56796) 2024-05-28T06:17:43.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:43.573+0000 7f8764e00640 1 -- 172.21.15.96:0/3032410249 learned_addr learned my addr 172.21.15.96:0/3032410249 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:17:43.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:43.573+0000 7f8765800640 1 --2- 172.21.15.96:0/3032410249 >> v2:172.21.15.96:3300/0 conn(0x7f876015f0d0 0x7f876015f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:43.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:43.574+0000 7f8764e00640 1 -- 172.21.15.96:0/3032410249 >> v2:172.21.15.96:3300/0 conn(0x7f876015f0d0 msgr2=0x7f876015f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:43.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:43.574+0000 7f8764e00640 1 --2- 172.21.15.96:0/3032410249 >> v2:172.21.15.96:3300/0 conn(0x7f876015f0d0 0x7f876015f4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:43.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:43.574+0000 7f8764e00640 1 -- 172.21.15.96:0/3032410249 >> v1:172.21.15.96:6789/0 conn(0x7f876015b880 legacy=0x7f876015bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:43.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:43.574+0000 7f8764e00640 1 -- 172.21.15.96:0/3032410249 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87600af080 con 0x7f876015f9f0 2024-05-28T06:17:43.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:43.574+0000 7f8765800640 1 --2- 172.21.15.96:0/3032410249 >> v2:172.21.15.96:3300/0 conn(0x7f876015f0d0 0x7f876015f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:17:43.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:43.576+0000 7f8764e00640 1 --2- 172.21.15.96:0/3032410249 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f876015f9f0 0x7f876015b340 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f876015cc20 tx=0x7f874c02a230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=296d99c68f5fb4ac server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:43.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:43.576+0000 7f875fe00640 1 -- 172.21.15.96:0/3032410249 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f874c037070 con 0x7f876015f9f0 2024-05-28T06:17:43.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:43.576+0000 7f875fe00640 1 -- 172.21.15.96:0/3032410249 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f874c032440 con 0x7f876015f9f0 2024-05-28T06:17:43.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:43.576+0000 7f875fe00640 1 -- 172.21.15.96:0/3032410249 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f874c032810 con 0x7f876015f9f0 2024-05-28T06:17:43.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:43.577+0000 7f8768200640 1 -- 172.21.15.96:0/3032410249 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f876015f9f0 msgr2=0x7f876015b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:43.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:43.577+0000 7f8768200640 1 --2- 172.21.15.96:0/3032410249 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f876015f9f0 0x7f876015b340 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f876015cc20 tx=0x7f874c02a230 comp rx=0 tx=0).stop 2024-05-28T06:17:43.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:43.577+0000 7f8768200640 1 -- 172.21.15.96:0/3032410249 shutdown_connections 2024-05-28T06:17:43.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:43.577+0000 7f8768200640 1 --2- 172.21.15.96:0/3032410249 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f876015f9f0 0x7f876015b340 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:43.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:43.577+0000 7f8768200640 1 --2- 172.21.15.96:0/3032410249 >> v2:172.21.15.96:3300/0 conn(0x7f876015f0d0 0x7f876015f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:43.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:43.577+0000 7f8768200640 1 -- 172.21.15.96:0/3032410249 >> 172.21.15.96:0/3032410249 conn(0x7f87600b1e50 msgr2=0x7f87600b2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:43.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:43.578+0000 7f8768200640 1 -- 172.21.15.96:0/3032410249 shutdown_connections 2024-05-28T06:17:43.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:43.578+0000 7f8768200640 1 -- 172.21.15.96:0/3032410249 wait complete. 2024-05-28T06:17:43.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:43.580+0000 7f8768200640 1 Processor -- start 2024-05-28T06:17:43.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:43.581+0000 7f8768200640 1 -- start start 2024-05-28T06:17:43.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:43.582+0000 7f8768200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f876015b880 0x7f8760160600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:43.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:43.582+0000 7f8768200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f876015f0d0 0x7f876009dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:43.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:43.582+0000 7f8768200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f876015f9f0 0x7f87600cddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:43.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:43.582+0000 7f8768200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f876015fe80 con 0x7f876015f9f0 2024-05-28T06:17:43.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:43.582+0000 7f8768200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f87600af640 con 0x7f876015b880 2024-05-28T06:17:43.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:43.582+0000 7f8768200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f876015c3e0 con 0x7f876015f0d0 2024-05-28T06:17:43.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:43.583+0000 7f8765800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f876015b880 0x7f8760160600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:43.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:43.583+0000 7f8764e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f876015f0d0 0x7f876009dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:43.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:43.583+0000 7f8765800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f876015b880 0x7f8760160600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.96:56808/0 (socket says 172.21.15.96:56808) 2024-05-28T06:17:43.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:43.583+0000 7f8765800640 1 -- 172.21.15.96:0/1528912350 learned_addr learned my addr 172.21.15.96:0/1528912350 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:17:43.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:43.583+0000 7f8766200640 1 --2- 172.21.15.96:0/1528912350 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f876015f9f0 0x7f87600cddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:43.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:43.583+0000 7f8766200640 1 -- 172.21.15.96:0/1528912350 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f876015f0d0 msgr2=0x7f876009dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:43.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:43.583+0000 7f8766200640 1 --2- 172.21.15.96:0/1528912350 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f876015f0d0 0x7f876009dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:43.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:43.583+0000 7f8766200640 1 -- 172.21.15.96:0/1528912350 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f876015b880 msgr2=0x7f8760160600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:43.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:43.584+0000 7f8766200640 1 --2- 172.21.15.96:0/1528912350 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f876015b880 0x7f8760160600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:43.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:43.584+0000 7f8766200640 1 -- 172.21.15.96:0/1528912350 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f876015d4d0 con 0x7f876015f9f0 2024-05-28T06:17:43.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:43.584+0000 7f8766200640 1 --2- 172.21.15.96:0/1528912350 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f876015f9f0 0x7f87600cddd0 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f875800c780 tx=0x7f875800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:43.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:43.585+0000 7f875d600640 1 -- 172.21.15.96:0/1528912350 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8758017070 con 0x7f876015f9f0 2024-05-28T06:17:43.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:43.585+0000 7f875d600640 1 -- 172.21.15.96:0/1528912350 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8758003b90 con 0x7f876015f9f0 2024-05-28T06:17:43.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:43.585+0000 7f875d600640 1 -- 172.21.15.96:0/1528912350 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8758005070 con 0x7f876015f9f0 2024-05-28T06:17:43.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:43.585+0000 7f8768200640 1 -- 172.21.15.96:0/1528912350 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8760160b40 con 0x7f876015f9f0 2024-05-28T06:17:43.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:43.586+0000 7f8768200640 1 -- 172.21.15.96:0/1528912350 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f876014f840 con 0x7f876015f9f0 2024-05-28T06:17:43.586 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:43.587+0000 7f8768200640 1 -- 172.21.15.96:0/1528912350 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87600689e0 con 0x7f876015f9f0 2024-05-28T06:17:43.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:43.589+0000 7f875d600640 1 -- 172.21.15.96:0/1528912350 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 29) ==== 92672+0+0 (secure 0 0 0) 0x7f8758005210 con 0x7f876015f9f0 2024-05-28T06:17:43.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:43.591+0000 7f875d600640 1 --2- 172.21.15.96:0/1528912350 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f874006daf0 0x7f874006ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:43.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:43.591+0000 7f875d600640 1 -- 172.21.15.96:0/1528912350 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 5851+0+0 (secure 0 0 0) 0x7f8758089f50 con 0x7f876015f9f0 2024-05-28T06:17:43.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:43.591+0000 7f8765800640 1 --2- 172.21.15.96:0/1528912350 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f874006daf0 0x7f874006ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:43.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:43.592+0000 7f8765800640 1 --2- 172.21.15.96:0/1528912350 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f874006daf0 0x7f874006ffb0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f875400aab0 tx=0x7f8754008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:43.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:43.600+0000 7f875d600640 1 -- 172.21.15.96:0/1528912350 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8758059ac0 con 0x7f876015f9f0 2024-05-28T06:17:43.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:43.978+0000 7f8768200640 1 -- 172.21.15.96:0/1528912350 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone43633", "target": ["mon-mgr", ""]}) -- 0x7f876015aeb0 con 0x7f874006daf0 2024-05-28T06:17:44.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.011+0000 7f875d600640 1 -- 172.21.15.96:0/1528912350 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f876015aeb0 con 0x7f874006daf0 2024-05-28T06:17:44.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.013+0000 7f8768200640 1 -- 172.21.15.96:0/1528912350 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f874006daf0 msgr2=0x7f874006ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:44.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.013+0000 7f8768200640 1 --2- 172.21.15.96:0/1528912350 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f874006daf0 0x7f874006ffb0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f875400aab0 tx=0x7f8754008040 comp rx=0 tx=0).stop 2024-05-28T06:17:44.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.013+0000 7f8768200640 1 -- 172.21.15.96:0/1528912350 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f876015f9f0 msgr2=0x7f87600cddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:44.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.013+0000 7f8768200640 1 --2- 172.21.15.96:0/1528912350 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f876015f9f0 0x7f87600cddd0 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f875800c780 tx=0x7f875800cc40 comp rx=0 tx=0).stop 2024-05-28T06:17:44.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.014+0000 7f8768200640 1 -- 172.21.15.96:0/1528912350 shutdown_connections 2024-05-28T06:17:44.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.014+0000 7f8768200640 1 --2- 172.21.15.96:0/1528912350 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f874006daf0 0x7f874006ffb0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:44.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.014+0000 7f8768200640 1 --2- 172.21.15.96:0/1528912350 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f876015f9f0 0x7f87600cddd0 unknown :-1 s=CLOSED pgs=1286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:44.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.014+0000 7f8768200640 1 --2- 172.21.15.96:0/1528912350 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f876015f0d0 0x7f876009dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:44.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.014+0000 7f8768200640 1 --2- 172.21.15.96:0/1528912350 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f876015b880 0x7f8760160600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:44.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.014+0000 7f8768200640 1 -- 172.21.15.96:0/1528912350 >> 172.21.15.96:0/1528912350 conn(0x7f87600b1e50 msgr2=0x7f87600b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:44.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.014+0000 7f8768200640 1 -- 172.21.15.96:0/1528912350 shutdown_connections 2024-05-28T06:17:44.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.014+0000 7f8768200640 1 -- 172.21.15.96:0/1528912350 wait complete. 2024-05-28T06:17:44.036 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone43636 2024-05-28T06:17:44.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.222+0000 7fb187200640 1 Processor -- start 2024-05-28T06:17:44.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.222+0000 7fb187200640 1 -- start start 2024-05-28T06:17:44.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.224+0000 7fb187200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb1880b4fa0 0x7fb1880cc710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:44.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.224+0000 7fb187200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fb18815ee60 0x7fb1880cb000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:44.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.224+0000 7fb187200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1880ca840 con 0x7fb18806f220 2024-05-28T06:17:44.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.224+0000 7fb187200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fb18815b040 con 0x7fb18815ee60 2024-05-28T06:17:44.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.224+0000 7fb187200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fb1880cd080 con 0x7fb1880b4fa0 2024-05-28T06:17:44.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.224+0000 7fb186800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fb18815ee60 0x7fb1880cb000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:44.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.224+0000 7fb186800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fb18815ee60 0x7fb1880cb000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49084/0 (socket says 172.21.15.96:49084) 2024-05-28T06:17:44.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.224+0000 7fb186800640 1 -- 172.21.15.96:0/3400052715 learned_addr learned my addr 172.21.15.96:0/3400052715 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:17:44.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.225+0000 7fb186800640 1 -- 172.21.15.96:0/3400052715 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb1880b4fa0 msgr2=0x7fb1880cc710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:44.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.225+0000 7fb186800640 1 --2- 172.21.15.96:0/3400052715 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb1880b4fa0 0x7fb1880cc710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:44.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.225+0000 7fb186800640 1 -- 172.21.15.96:0/3400052715 >> v1:172.21.15.123:6789/0 conn(0x7fb18806f220 legacy=0x7fb18815e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:44.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.225+0000 7fb186800640 1 -- 172.21.15.96:0/3400052715 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1880cd360 con 0x7fb18815ee60 2024-05-28T06:17:44.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.226+0000 7fb186800640 1 --2- 172.21.15.96:0/3400052715 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb18815ee60 0x7fb1880cb000 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7fb18000af60 tx=0x7fb18002b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e38e785daede5fb server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:44.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.226+0000 7fb184a00640 1 -- 172.21.15.96:0/3400052715 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb180037070 con 0x7fb18815ee60 2024-05-28T06:17:44.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.226+0000 7fb184a00640 1 -- 172.21.15.96:0/3400052715 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb180008500 con 0x7fb18815ee60 2024-05-28T06:17:44.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.226+0000 7fb184a00640 1 -- 172.21.15.96:0/3400052715 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1800088d0 con 0x7fb18815ee60 2024-05-28T06:17:44.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.226+0000 7fb187200640 1 -- 172.21.15.96:0/3400052715 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb18815ee60 msgr2=0x7fb1880cb000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:44.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.226+0000 7fb187200640 1 --2- 172.21.15.96:0/3400052715 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb18815ee60 0x7fb1880cb000 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7fb18000af60 tx=0x7fb18002b7a0 comp rx=0 tx=0).stop 2024-05-28T06:17:44.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.226+0000 7fb187200640 1 -- 172.21.15.96:0/3400052715 shutdown_connections 2024-05-28T06:17:44.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.226+0000 7fb187200640 1 --2- 172.21.15.96:0/3400052715 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb18815ee60 0x7fb1880cb000 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:44.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.226+0000 7fb187200640 1 --2- 172.21.15.96:0/3400052715 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb1880b4fa0 0x7fb1880cc710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:44.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.227+0000 7fb187200640 1 -- 172.21.15.96:0/3400052715 >> 172.21.15.96:0/3400052715 conn(0x7fb18806bae0 msgr2=0x7fb18806bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:44.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.227+0000 7fb187200640 1 -- 172.21.15.96:0/3400052715 shutdown_connections 2024-05-28T06:17:44.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.227+0000 7fb187200640 1 -- 172.21.15.96:0/3400052715 wait complete. 2024-05-28T06:17:44.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.228+0000 7fb187200640 1 Processor -- start 2024-05-28T06:17:44.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.228+0000 7fb187200640 1 -- start start 2024-05-28T06:17:44.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.229+0000 7fb187200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb18806f220 0x7fb18814eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:44.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.229+0000 7fb187200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb1880b4fa0 0x7fb188072fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:44.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.229+0000 7fb187200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb18815ee60 0x7fb1880734e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:44.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.229+0000 7fb187200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb1880ccc50 con 0x7fb18806f220 2024-05-28T06:17:44.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.229+0000 7fb187200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fb1880b31c0 con 0x7fb1880b4fa0 2024-05-28T06:17:44.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.229+0000 7fb187200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fb1880ccdd0 con 0x7fb18815ee60 2024-05-28T06:17:44.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.230+0000 7fb185e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb18806f220 0x7fb18814eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:44.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.230+0000 7fb185e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb18806f220 0x7fb18814eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49096/0 (socket says 172.21.15.96:49096) 2024-05-28T06:17:44.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.230+0000 7fb185e00640 1 -- 172.21.15.96:0/3621649232 learned_addr learned my addr 172.21.15.96:0/3621649232 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:17:44.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.230+0000 7fb186800640 1 --2- 172.21.15.96:0/3621649232 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb18815ee60 0x7fb1880734e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:44.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.230+0000 7fb185400640 1 --2- 172.21.15.96:0/3621649232 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb1880b4fa0 0x7fb188072fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:44.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.230+0000 7fb185e00640 1 -- 172.21.15.96:0/3621649232 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb18815ee60 msgr2=0x7fb1880734e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:44.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.230+0000 7fb185e00640 1 --2- 172.21.15.96:0/3621649232 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb18815ee60 0x7fb1880734e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:44.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.230+0000 7fb185e00640 1 -- 172.21.15.96:0/3621649232 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb1880b4fa0 msgr2=0x7fb188072fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:44.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.230+0000 7fb185e00640 1 --2- 172.21.15.96:0/3621649232 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb1880b4fa0 0x7fb188072fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:44.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.230+0000 7fb185e00640 1 -- 172.21.15.96:0/3621649232 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb18814f0d0 con 0x7fb18806f220 2024-05-28T06:17:44.231 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.230+0000 7fb185400640 1 --2- 172.21.15.96:0/3621649232 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb1880b4fa0 0x7fb188072fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:17:44.231 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.230+0000 7fb185e00640 1 --2- 172.21.15.96:0/3621649232 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb18806f220 0x7fb18814eb90 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7fb17800b4d0 tx=0x7fb17800b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:44.231 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.231+0000 7fb17e000640 1 -- 172.21.15.96:0/3621649232 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb178014070 con 0x7fb18806f220 2024-05-28T06:17:44.231 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.231+0000 7fb17e000640 1 -- 172.21.15.96:0/3621649232 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb178010870 con 0x7fb18806f220 2024-05-28T06:17:44.231 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.231+0000 7fb17e000640 1 -- 172.21.15.96:0/3621649232 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb178010c60 con 0x7fb18806f220 2024-05-28T06:17:44.231 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.231+0000 7fb187200640 1 -- 172.21.15.96:0/3621649232 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb188150110 con 0x7fb18806f220 2024-05-28T06:17:44.231 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.231+0000 7fb187200640 1 -- 172.21.15.96:0/3621649232 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb18814f630 con 0x7fb18806f220 2024-05-28T06:17:44.231 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.232+0000 7fb187200640 1 -- 172.21.15.96:0/3621649232 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1880689e0 con 0x7fb18806f220 2024-05-28T06:17:44.231 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.233+0000 7fb17e000640 1 -- 172.21.15.96:0/3621649232 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 29) ==== 92672+0+0 (secure 0 0 0) 0x7fb178017070 con 0x7fb18806f220 2024-05-28T06:17:44.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.234+0000 7fb17e000640 1 --2- 172.21.15.96:0/3621649232 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb16806daf0 0x7fb16806ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:44.233 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.234+0000 7fb17e000640 1 -- 172.21.15.96:0/3621649232 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 5851+0+0 (secure 0 0 0) 0x7fb17808a4c0 con 0x7fb18806f220 2024-05-28T06:17:44.233 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.235+0000 7fb185400640 1 --2- 172.21.15.96:0/3621649232 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb16806daf0 0x7fb16806ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:44.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.239+0000 7fb185400640 1 --2- 172.21.15.96:0/3621649232 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb16806daf0 0x7fb16806ffb0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fb188073af0 tx=0x7fb174009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:44.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.241+0000 7fb17e000640 1 -- 172.21.15.96:0/3621649232 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb18814f0d0 con 0x7fb18806f220 2024-05-28T06:17:44.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.623+0000 7fb187200640 1 -- 172.21.15.96:0/3621649232 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone43636", "target": ["mon-mgr", ""]}) -- 0x7fb1881f8910 con 0x7fb16806daf0 2024-05-28T06:17:44.659 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.661+0000 7fb17e000640 1 -- 172.21.15.96:0/3621649232 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fb1881f8910 con 0x7fb16806daf0 2024-05-28T06:17:44.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.664+0000 7fb187200640 1 -- 172.21.15.96:0/3621649232 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb16806daf0 msgr2=0x7fb16806ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:44.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.664+0000 7fb187200640 1 --2- 172.21.15.96:0/3621649232 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb16806daf0 0x7fb16806ffb0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fb188073af0 tx=0x7fb174009040 comp rx=0 tx=0).stop 2024-05-28T06:17:44.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.664+0000 7fb187200640 1 -- 172.21.15.96:0/3621649232 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb18806f220 msgr2=0x7fb18814eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:44.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.664+0000 7fb187200640 1 --2- 172.21.15.96:0/3621649232 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb18806f220 0x7fb18814eb90 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7fb17800b4d0 tx=0x7fb17800b990 comp rx=0 tx=0).stop 2024-05-28T06:17:44.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.664+0000 7fb187200640 1 -- 172.21.15.96:0/3621649232 shutdown_connections 2024-05-28T06:17:44.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.664+0000 7fb187200640 1 --2- 172.21.15.96:0/3621649232 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb16806daf0 0x7fb16806ffb0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:44.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.664+0000 7fb187200640 1 --2- 172.21.15.96:0/3621649232 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb18815ee60 0x7fb1880734e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:44.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.664+0000 7fb187200640 1 --2- 172.21.15.96:0/3621649232 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb1880b4fa0 0x7fb188072fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:44.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.664+0000 7fb187200640 1 --2- 172.21.15.96:0/3621649232 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb18806f220 0x7fb18814eb90 unknown :-1 s=CLOSED pgs=1288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:44.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.664+0000 7fb187200640 1 -- 172.21.15.96:0/3621649232 >> 172.21.15.96:0/3621649232 conn(0x7fb18806bae0 msgr2=0x7fb1880cb940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:44.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.665+0000 7fb187200640 1 -- 172.21.15.96:0/3621649232 shutdown_connections 2024-05-28T06:17:44.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.665+0000 7fb187200640 1 -- 172.21.15.96:0/3621649232 wait complete. 2024-05-28T06:17:44.688 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone43637 2024-05-28T06:17:44.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.896+0000 7f86fd400640 1 Processor -- start 2024-05-28T06:17:44.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.896+0000 7f86fd400640 1 -- start start 2024-05-28T06:17:44.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.897+0000 7f86fd400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f86f80cf2f0 0x7f86f80cf910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:44.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.897+0000 7f86fd400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f86f8140500 0x7f86f80ad130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:44.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.897+0000 7f86fd400640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86f806dd50 con 0x7f86f806edf0 2024-05-28T06:17:44.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.897+0000 7f86fd400640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f86f806e030 con 0x7f86f8140500 2024-05-28T06:17:44.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.897+0000 7f86fd400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f86f806e310 con 0x7f86f80cf2f0 2024-05-28T06:17:44.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.897+0000 7f86f7200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f86f8140500 0x7f86f80ad130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:44.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.897+0000 7f86f7200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f86f8140500 0x7f86f80ad130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49102/0 (socket says 172.21.15.96:49102) 2024-05-28T06:17:44.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.897+0000 7f86f7200640 1 -- 172.21.15.96:0/262725266 learned_addr learned my addr 172.21.15.96:0/262725266 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:17:44.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.897+0000 7f86efe00640 1 --1- 172.21.15.96:0/262725266 >> v1:172.21.15.96:6789/0 conn(0x7f86f806edf0 0x7f86f813fdf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:40114/0 (socket says 172.21.15.96:40114) 2024-05-28T06:17:44.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.897+0000 7f86f6800640 1 --2- 172.21.15.96:0/262725266 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f86f80cf2f0 0x7f86f80cf910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:44.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.898+0000 7f86f7200640 1 -- 172.21.15.96:0/262725266 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f86f80cf2f0 msgr2=0x7f86f80cf910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:44.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.898+0000 7f86f7200640 1 --2- 172.21.15.96:0/262725266 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f86f80cf2f0 0x7f86f80cf910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:44.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.898+0000 7f86f7200640 1 -- 172.21.15.96:0/262725266 >> v1:172.21.15.96:6789/0 conn(0x7f86f806edf0 legacy=0x7f86f813fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:44.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.898+0000 7f86f7200640 1 -- 172.21.15.96:0/262725266 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86f806e5f0 con 0x7f86f8140500 2024-05-28T06:17:44.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.898+0000 7f86f5e00640 1 -- 172.21.15.96:0/262725266 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1424584888 0 0) 0x7f86f806dd50 con 0x7f86f806edf0 2024-05-28T06:17:44.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.899+0000 7f86f7200640 1 --2- 172.21.15.96:0/262725266 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f86f8140500 0x7f86f80ad130 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f86f0009e50 tx=0x7f86f002a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=433a77a4da210e9b server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:44.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.899+0000 7f86f5e00640 1 -- 172.21.15.96:0/262725266 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86f0036070 con 0x7f86f8140500 2024-05-28T06:17:44.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.899+0000 7f86f5e00640 1 -- 172.21.15.96:0/262725266 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86f0031600 con 0x7f86f8140500 2024-05-28T06:17:44.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.899+0000 7f86f5e00640 1 -- 172.21.15.96:0/262725266 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86f00319d0 con 0x7f86f8140500 2024-05-28T06:17:44.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.899+0000 7f86fd400640 1 -- 172.21.15.96:0/262725266 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f86f8140500 msgr2=0x7f86f80ad130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:44.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.899+0000 7f86fd400640 1 --2- 172.21.15.96:0/262725266 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f86f8140500 0x7f86f80ad130 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f86f0009e50 tx=0x7f86f002a680 comp rx=0 tx=0).stop 2024-05-28T06:17:44.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.899+0000 7f86fd400640 1 -- 172.21.15.96:0/262725266 shutdown_connections 2024-05-28T06:17:44.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.899+0000 7f86fd400640 1 --2- 172.21.15.96:0/262725266 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f86f8140500 0x7f86f80ad130 unknown :-1 s=CLOSED pgs=1290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:44.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.899+0000 7f86fd400640 1 --2- 172.21.15.96:0/262725266 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f86f80cf2f0 0x7f86f80cf910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:44.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.899+0000 7f86fd400640 1 -- 172.21.15.96:0/262725266 >> 172.21.15.96:0/262725266 conn(0x7f86f80b1e50 msgr2=0x7f86f80b2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:44.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.900+0000 7f86fd400640 1 -- 172.21.15.96:0/262725266 shutdown_connections 2024-05-28T06:17:44.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.900+0000 7f86fd400640 1 -- 172.21.15.96:0/262725266 wait complete. 2024-05-28T06:17:44.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.901+0000 7f86fd400640 1 Processor -- start 2024-05-28T06:17:44.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.901+0000 7f86fd400640 1 -- start start 2024-05-28T06:17:44.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.902+0000 7f86fd400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f86f806edf0 0x7f86f81f4340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:44.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.902+0000 7f86fd400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f86f80cf2f0 0x7f86f815cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:44.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.902+0000 7f86fd400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f86f81446a0 0x7f86f80b4530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:44.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.902+0000 7f86fd400640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f86f81461a0 con 0x7f86f806edf0 2024-05-28T06:17:44.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.902+0000 7f86fd400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f86f8146020 con 0x7f86f80cf2f0 2024-05-28T06:17:44.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.902+0000 7f86fd400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f86f80cfe50 con 0x7f86f81446a0 2024-05-28T06:17:44.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.902+0000 7f86f6800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f86f806edf0 0x7f86f81f4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:44.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.902+0000 7f86f6800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f86f806edf0 0x7f86f81f4340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49106/0 (socket says 172.21.15.96:49106) 2024-05-28T06:17:44.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.902+0000 7f86f6800640 1 -- 172.21.15.96:0/1618118545 learned_addr learned my addr 172.21.15.96:0/1618118545 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:17:44.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.903+0000 7f86f7200640 1 --2- 172.21.15.96:0/1618118545 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f86f81446a0 0x7f86f80b4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:44.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.903+0000 7f86efe00640 1 --2- 172.21.15.96:0/1618118545 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f86f80cf2f0 0x7f86f815cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:44.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.903+0000 7f86f6800640 1 -- 172.21.15.96:0/1618118545 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f86f81446a0 msgr2=0x7f86f80b4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:44.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.903+0000 7f86f6800640 1 --2- 172.21.15.96:0/1618118545 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f86f81446a0 0x7f86f80b4530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:44.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.903+0000 7f86f6800640 1 -- 172.21.15.96:0/1618118545 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f86f80cf2f0 msgr2=0x7f86f815cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:44.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.903+0000 7f86f6800640 1 --2- 172.21.15.96:0/1618118545 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f86f80cf2f0 0x7f86f815cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:44.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.903+0000 7f86f6800640 1 -- 172.21.15.96:0/1618118545 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86f8144b50 con 0x7f86f806edf0 2024-05-28T06:17:44.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.903+0000 7f86efe00640 1 --2- 172.21.15.96:0/1618118545 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f86f80cf2f0 0x7f86f815cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-28T06:17:44.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.903+0000 7f86f6800640 1 --2- 172.21.15.96:0/1618118545 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f86f806edf0 0x7f86f81f4340 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7f86e800d830 tx=0x7f86e800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:44.904 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.903+0000 7f86eea00640 1 -- 172.21.15.96:0/1618118545 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86e800b6c0 con 0x7f86f806edf0 2024-05-28T06:17:44.904 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.903+0000 7f86eea00640 1 -- 172.21.15.96:0/1618118545 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86e8019070 con 0x7f86f806edf0 2024-05-28T06:17:44.904 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.903+0000 7f86eea00640 1 -- 172.21.15.96:0/1618118545 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86e800ba80 con 0x7f86f806edf0 2024-05-28T06:17:44.904 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.903+0000 7f86fd400640 1 -- 172.21.15.96:0/1618118545 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86f815d370 con 0x7f86f806edf0 2024-05-28T06:17:44.904 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.904+0000 7f86fd400640 1 -- 172.21.15.96:0/1618118545 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86f815d930 con 0x7f86f806edf0 2024-05-28T06:17:44.904 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.905+0000 7f86fd400640 1 -- 172.21.15.96:0/1618118545 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86f80689e0 con 0x7f86f806edf0 2024-05-28T06:17:44.904 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.906+0000 7f86eea00640 1 -- 172.21.15.96:0/1618118545 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 29) ==== 92672+0+0 (secure 0 0 0) 0x7f86e801e460 con 0x7f86f806edf0 2024-05-28T06:17:44.905 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.907+0000 7f86eea00640 1 --2- 172.21.15.96:0/1618118545 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f86d406db20 0x7f86d406ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:44.906 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.907+0000 7f86eea00640 1 -- 172.21.15.96:0/1618118545 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 5851+0+0 (secure 0 0 0) 0x7f86e808ada0 con 0x7f86f806edf0 2024-05-28T06:17:44.906 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.907+0000 7f86efe00640 1 --2- 172.21.15.96:0/1618118545 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f86d406db20 0x7f86d406ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:44.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.908+0000 7f86efe00640 1 --2- 172.21.15.96:0/1618118545 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f86d406db20 0x7f86d406ffe0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f86f806df30 tx=0x7f86e0007ca0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:44.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:44.911+0000 7f86eea00640 1 -- 172.21.15.96:0/1618118545 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f86e805abf0 con 0x7f86f806edf0 2024-05-28T06:17:45.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.123+0000 7f86fd400640 1 -- 172.21.15.96:0/1618118545 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone43637", "target": ["mon-mgr", ""]}) -- 0x7f86f80c9f10 con 0x7f86d406db20 2024-05-28T06:17:45.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.166+0000 7f86eea00640 1 -- 172.21.15.96:0/1618118545 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f86f80c9f10 con 0x7f86d406db20 2024-05-28T06:17:45.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.169+0000 7f86fd400640 1 -- 172.21.15.96:0/1618118545 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f86d406db20 msgr2=0x7f86d406ffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:45.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.169+0000 7f86fd400640 1 --2- 172.21.15.96:0/1618118545 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f86d406db20 0x7f86d406ffe0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f86f806df30 tx=0x7f86e0007ca0 comp rx=0 tx=0).stop 2024-05-28T06:17:45.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.169+0000 7f86fd400640 1 -- 172.21.15.96:0/1618118545 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f86f806edf0 msgr2=0x7f86f81f4340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:45.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.169+0000 7f86fd400640 1 --2- 172.21.15.96:0/1618118545 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f86f806edf0 0x7f86f81f4340 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7f86e800d830 tx=0x7f86e800dcf0 comp rx=0 tx=0).stop 2024-05-28T06:17:45.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.169+0000 7f86fd400640 1 -- 172.21.15.96:0/1618118545 shutdown_connections 2024-05-28T06:17:45.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.169+0000 7f86fd400640 1 --2- 172.21.15.96:0/1618118545 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f86d406db20 0x7f86d406ffe0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:45.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.169+0000 7f86fd400640 1 --2- 172.21.15.96:0/1618118545 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f86f81446a0 0x7f86f80b4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:45.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.169+0000 7f86fd400640 1 --2- 172.21.15.96:0/1618118545 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f86f80cf2f0 0x7f86f815cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:45.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.169+0000 7f86fd400640 1 --2- 172.21.15.96:0/1618118545 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f86f806edf0 0x7f86f81f4340 unknown :-1 s=CLOSED pgs=1291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:45.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.169+0000 7f86fd400640 1 -- 172.21.15.96:0/1618118545 >> 172.21.15.96:0/1618118545 conn(0x7f86f80b1e50 msgr2=0x7f86f80cd9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:45.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.169+0000 7f86fd400640 1 -- 172.21.15.96:0/1618118545 shutdown_connections 2024-05-28T06:17:45.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.169+0000 7f86fd400640 1 -- 172.21.15.96:0/1618118545 wait complete. 2024-05-28T06:17:45.185 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone436317 2024-05-28T06:17:45.430 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.431+0000 7f5b8ec00640 1 Processor -- start 2024-05-28T06:17:45.430 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.431+0000 7f5b8ec00640 1 -- start start 2024-05-28T06:17:45.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.433+0000 7f5b8ec00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5b880b3510 0x7f5b8806dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:45.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.433+0000 7f5b8ec00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b8806e190 con 0x7f5b8815e830 2024-05-28T06:17:45.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.433+0000 7f5b8ec00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b8806e470 con 0x7f5b88072310 2024-05-28T06:17:45.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.433+0000 7f5b8ec00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f5b8806e750 con 0x7f5b880b3510 2024-05-28T06:17:45.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.434+0000 7f5b87e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f5b8815e830 0x7f5b88071c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:40116/0 (socket says 172.21.15.96:40116) 2024-05-28T06:17:45.433 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.434+0000 7f5b87e00640 1 -- 172.21.15.96:0/3631306497 learned_addr learned my addr 172.21.15.96:0/3631306497 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:17:45.433 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.434+0000 7f5b8cc00640 1 --2- 172.21.15.96:0/3631306497 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5b880b3510 0x7f5b8806dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:45.433 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.434+0000 7f5b86a00640 1 -- 172.21.15.96:0/3631306497 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2458527797 0 0) 0x7f5b8806e470 con 0x7f5b88072310 2024-05-28T06:17:45.433 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.434+0000 7f5b86a00640 1 -- 172.21.15.96:0/3631306497 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5b6c003520 con 0x7f5b88072310 2024-05-28T06:17:45.433 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.434+0000 7f5b86a00640 1 -- 172.21.15.96:0/3631306497 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3132594743 0 0) 0x7f5b8806e190 con 0x7f5b8815e830 2024-05-28T06:17:45.433 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.435+0000 7f5b86a00640 1 -- 172.21.15.96:0/3631306497 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5b8806e470 con 0x7f5b8815e830 2024-05-28T06:17:45.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.435+0000 7f5b86a00640 1 -- 172.21.15.96:0/3631306497 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1354653666 0 0) 0x7f5b8806e470 con 0x7f5b8815e830 2024-05-28T06:17:45.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.435+0000 7f5b86a00640 1 -- 172.21.15.96:0/3631306497 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5b880b3510 msgr2=0x7f5b8806dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:45.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.435+0000 7f5b86a00640 1 --2- 172.21.15.96:0/3631306497 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5b880b3510 0x7f5b8806dbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:45.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.435+0000 7f5b86a00640 1 -- 172.21.15.96:0/3631306497 >> v1:172.21.15.123:6789/0 conn(0x7f5b88072310 legacy=0x7f5b880ad130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:45.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.435+0000 7f5b86a00640 1 -- 172.21.15.96:0/3631306497 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b8806ea30 con 0x7f5b8815e830 2024-05-28T06:17:45.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.435+0000 7f5b8cc00640 1 --2- 172.21.15.96:0/3631306497 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5b880b3510 0x7f5b8806dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-28T06:17:45.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.436+0000 7f5b86a00640 1 -- 172.21.15.96:0/3631306497 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f5b78001610 con 0x7f5b8815e830 2024-05-28T06:17:45.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.436+0000 7f5b86a00640 1 -- 172.21.15.96:0/3631306497 >> v1:172.21.15.96:6789/0 conn(0x7f5b8815e830 legacy=0x7f5b88071c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:45.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.437+0000 7f5b86a00640 1 --2- 172.21.15.96:0/3631306497 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5b880b3510 0x7f5b6c004bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:45.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.437+0000 7f5b86a00640 1 --2- 172.21.15.96:0/3631306497 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b6c005160 0x7f5b6c0037a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:45.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.437+0000 7f5b86a00640 1 --2- 172.21.15.96:0/3631306497 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5b6c003ce0 0x7f5b6c004160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:45.436 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.437+0000 7f5b86a00640 1 -- 172.21.15.96:0/3631306497 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f5b8806e750 con 0x7f5b6c003ce0 2024-05-28T06:17:45.436 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.437+0000 7f5b86a00640 1 -- 172.21.15.96:0/3631306497 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5b6c00bb50 con 0x7f5b6c005160 2024-05-28T06:17:45.436 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.437+0000 7f5b86a00640 1 -- 172.21.15.96:0/3631306497 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f5b6c00bd50 con 0x7f5b880b3510 2024-05-28T06:17:45.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.438+0000 7f5b87400640 1 --2- 172.21.15.96:0/3631306497 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b6c005160 0x7f5b6c0037a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:45.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.438+0000 7f5b87e00640 1 --2- 172.21.15.96:0/3631306497 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5b880b3510 0x7f5b6c004bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:45.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.438+0000 7f5b8cc00640 1 --2- 172.21.15.96:0/3631306497 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5b6c003ce0 0x7f5b6c004160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:45.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.439+0000 7f5b87400640 1 -- 172.21.15.96:0/3631306497 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5b880b3510 msgr2=0x7f5b6c004bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:45.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.439+0000 7f5b87400640 1 --2- 172.21.15.96:0/3631306497 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5b880b3510 0x7f5b6c004bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:45.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.439+0000 7f5b87400640 1 -- 172.21.15.96:0/3631306497 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5b6c003ce0 msgr2=0x7f5b6c004160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:45.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.439+0000 7f5b87400640 1 --2- 172.21.15.96:0/3631306497 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5b6c003ce0 0x7f5b6c004160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:45.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.439+0000 7f5b87400640 1 -- 172.21.15.96:0/3631306497 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b8806e470 con 0x7f5b6c005160 2024-05-28T06:17:45.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.439+0000 7f5b8cc00640 1 --2- 172.21.15.96:0/3631306497 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5b6c003ce0 0x7f5b6c004160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-28T06:17:45.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.441+0000 7f5b87400640 1 --2- 172.21.15.96:0/3631306497 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b6c005160 0x7f5b6c0037a0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f5b74001800 tx=0x7f5b6c005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=144f59c9f9e2a8e0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:45.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.442+0000 7f5b86a00640 1 -- 172.21.15.96:0/3631306497 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b74001450 con 0x7f5b6c005160 2024-05-28T06:17:45.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.442+0000 7f5b86a00640 1 -- 172.21.15.96:0/3631306497 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b740052f0 con 0x7f5b6c005160 2024-05-28T06:17:45.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.442+0000 7f5b86a00640 1 -- 172.21.15.96:0/3631306497 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b7402ac70 con 0x7f5b6c005160 2024-05-28T06:17:45.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.443+0000 7f5b8ec00640 1 -- 172.21.15.96:0/3631306497 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b6c005160 msgr2=0x7f5b6c0037a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:45.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.443+0000 7f5b8ec00640 1 --2- 172.21.15.96:0/3631306497 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b6c005160 0x7f5b6c0037a0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f5b74001800 tx=0x7f5b6c005830 comp rx=0 tx=0).stop 2024-05-28T06:17:45.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.443+0000 7f5b8cc00640 1 -- 172.21.15.96:0/3631306497 reap_dead start 2024-05-28T06:17:45.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.444+0000 7f5b8ec00640 1 -- 172.21.15.96:0/3631306497 shutdown_connections 2024-05-28T06:17:45.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.444+0000 7f5b8ec00640 1 -- 172.21.15.96:0/3631306497 >> 172.21.15.96:0/3631306497 conn(0x7f5b880b1e50 msgr2=0x7f5b880b2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:45.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.444+0000 7f5b8ec00640 1 -- 172.21.15.96:0/3631306497 shutdown_connections 2024-05-28T06:17:45.444 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.444+0000 7f5b8ec00640 1 -- 172.21.15.96:0/3631306497 wait complete. 2024-05-28T06:17:45.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.446+0000 7f5b8ec00640 1 Processor -- start 2024-05-28T06:17:45.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.447+0000 7f5b8ec00640 1 -- start start 2024-05-28T06:17:45.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.448+0000 7f5b8ec00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5b6c005160 0x7f5b88150750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:45.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.448+0000 7f5b8ec00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b88096d10 0x7f5b88097190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:45.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.448+0000 7f5b8ec00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5b880afd10 0x7f5b880b01c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:45.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.448+0000 7f5b8ec00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f5b880728c0 con 0x7f5b880afd10 2024-05-28T06:17:45.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.448+0000 7f5b8ec00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5b8815ec10 con 0x7f5b88096d10 2024-05-28T06:17:45.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.448+0000 7f5b8ec00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f5b88072a40 con 0x7f5b6c005160 2024-05-28T06:17:45.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.448+0000 7f5b87400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b88096d10 0x7f5b88097190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:45.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.448+0000 7f5b87e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5b6c005160 0x7f5b88150750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:45.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.448+0000 7f5b87400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b88096d10 0x7f5b88097190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.96:56846/0 (socket says 172.21.15.96:56846) 2024-05-28T06:17:45.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.448+0000 7f5b87400640 1 -- 172.21.15.96:0/1261300029 learned_addr learned my addr 172.21.15.96:0/1261300029 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:17:45.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.448+0000 7f5b8cc00640 1 --2- 172.21.15.96:0/1261300029 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5b880afd10 0x7f5b880b01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:45.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.449+0000 7f5b8cc00640 1 -- 172.21.15.96:0/1261300029 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5b6c005160 msgr2=0x7f5b88150750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:45.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.449+0000 7f5b8cc00640 1 --2- 172.21.15.96:0/1261300029 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5b6c005160 0x7f5b88150750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:45.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.449+0000 7f5b8cc00640 1 -- 172.21.15.96:0/1261300029 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b88096d10 msgr2=0x7f5b88097190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:45.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.449+0000 7f5b8cc00640 1 --2- 172.21.15.96:0/1261300029 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b88096d10 0x7f5b88097190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:45.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.449+0000 7f5b8cc00640 1 -- 172.21.15.96:0/1261300029 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b880b0d00 con 0x7f5b880afd10 2024-05-28T06:17:45.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.449+0000 7f5b8cc00640 1 --2- 172.21.15.96:0/1261300029 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5b880afd10 0x7f5b880b01c0 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f5b880d1340 tx=0x7f5b800115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:45.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.449+0000 7f5b7fe00640 1 -- 172.21.15.96:0/1261300029 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b80011db0 con 0x7f5b880afd10 2024-05-28T06:17:45.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.449+0000 7f5b7fe00640 1 -- 172.21.15.96:0/1261300029 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b8001b070 con 0x7f5b880afd10 2024-05-28T06:17:45.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.449+0000 7f5b8ec00640 1 -- 172.21.15.96:0/1261300029 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b880b0a20 con 0x7f5b880afd10 2024-05-28T06:17:45.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.450+0000 7f5b7fe00640 1 -- 172.21.15.96:0/1261300029 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b80003990 con 0x7f5b880afd10 2024-05-28T06:17:45.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.450+0000 7f5b8ec00640 1 -- 172.21.15.96:0/1261300029 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b88094b50 con 0x7f5b880afd10 2024-05-28T06:17:45.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.451+0000 7f5b8ec00640 1 -- 172.21.15.96:0/1261300029 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b880689e0 con 0x7f5b880afd10 2024-05-28T06:17:45.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.457+0000 7f5b7fe00640 1 -- 172.21.15.96:0/1261300029 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 29) ==== 92672+0+0 (secure 0 0 0) 0x7f5b80003b30 con 0x7f5b880afd10 2024-05-28T06:17:45.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.459+0000 7f5b7fe00640 1 --2- 172.21.15.96:0/1261300029 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5b6806db20 0x7f5b6806ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:45.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.459+0000 7f5b7fe00640 1 -- 172.21.15.96:0/1261300029 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 5851+0+0 (secure 0 0 0) 0x7f5b8008b020 con 0x7f5b880afd10 2024-05-28T06:17:45.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.459+0000 7f5b87e00640 1 --2- 172.21.15.96:0/1261300029 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5b6806db20 0x7f5b6806ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:45.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.461+0000 7f5b87e00640 1 --2- 172.21.15.96:0/1261300029 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5b6806db20 0x7f5b6806ffe0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f5b8809de70 tx=0x7f5b78001da0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:45.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.462+0000 7f5b7fe00640 1 -- 172.21.15.96:0/1261300029 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5b80003e20 con 0x7f5b880afd10 2024-05-28T06:17:45.763 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.764+0000 7f5b8ec00640 1 -- 172.21.15.96:0/1261300029 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone436317", "target": ["mon-mgr", ""]}) -- 0x7f5b881621f0 con 0x7f5b6806db20 2024-05-28T06:17:45.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.804+0000 7f5b7fe00640 1 -- 172.21.15.96:0/1261300029 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f5b881621f0 con 0x7f5b6806db20 2024-05-28T06:17:45.807 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.808+0000 7f5b8ec00640 1 -- 172.21.15.96:0/1261300029 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5b6806db20 msgr2=0x7f5b6806ffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:45.807 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.808+0000 7f5b8ec00640 1 --2- 172.21.15.96:0/1261300029 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5b6806db20 0x7f5b6806ffe0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f5b8809de70 tx=0x7f5b78001da0 comp rx=0 tx=0).stop 2024-05-28T06:17:45.807 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.808+0000 7f5b8ec00640 1 -- 172.21.15.96:0/1261300029 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5b880afd10 msgr2=0x7f5b880b01c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:45.807 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.808+0000 7f5b8ec00640 1 --2- 172.21.15.96:0/1261300029 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5b880afd10 0x7f5b880b01c0 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f5b880d1340 tx=0x7f5b800115a0 comp rx=0 tx=0).stop 2024-05-28T06:17:45.807 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.808+0000 7f5b8ec00640 1 -- 172.21.15.96:0/1261300029 shutdown_connections 2024-05-28T06:17:45.807 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.808+0000 7f5b8ec00640 1 --2- 172.21.15.96:0/1261300029 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5b6806db20 0x7f5b6806ffe0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:45.807 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.808+0000 7f5b8ec00640 1 --2- 172.21.15.96:0/1261300029 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5b880afd10 0x7f5b880b01c0 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:45.807 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.808+0000 7f5b8ec00640 1 --2- 172.21.15.96:0/1261300029 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b88096d10 0x7f5b88097190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:45.807 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.808+0000 7f5b8ec00640 1 --2- 172.21.15.96:0/1261300029 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5b6c005160 0x7f5b88150750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:45.807 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.808+0000 7f5b8ec00640 1 -- 172.21.15.96:0/1261300029 >> 172.21.15.96:0/1261300029 conn(0x7f5b880b1e50 msgr2=0x7f5b8809b280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:45.807 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.808+0000 7f5b8ec00640 1 -- 172.21.15.96:0/1261300029 shutdown_connections 2024-05-28T06:17:45.807 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:45.808+0000 7f5b8ec00640 1 -- 172.21.15.96:0/1261300029 wait complete. 2024-05-28T06:17:45.825 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone43639 2024-05-28T06:17:46.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.005+0000 7fd94ca00640 1 Processor -- start 2024-05-28T06:17:46.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.006+0000 7fd94ca00640 1 -- start start 2024-05-28T06:17:46.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.007+0000 7fd94ca00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fd9480d0430 0x7fd9480d0860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:46.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.007+0000 7fd94ca00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fd94815f0c0 0x7fd94815f530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:46.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.007+0000 7fd94ca00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd94815bb30 con 0x7fd9480ce560 2024-05-28T06:17:46.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.007+0000 7fd94ca00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fd9480b5290 con 0x7fd94815f0c0 2024-05-28T06:17:46.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.007+0000 7fd94ca00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fd9480d0da0 con 0x7fd9480d0430 2024-05-28T06:17:46.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.007+0000 7fd947200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fd94815f0c0 0x7fd94815f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:46.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.007+0000 7fd947200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fd94815f0c0 0x7fd94815f530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49134/0 (socket says 172.21.15.96:49134) 2024-05-28T06:17:46.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.008+0000 7fd947200640 1 -- 172.21.15.96:0/4000661565 learned_addr learned my addr 172.21.15.96:0/4000661565 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:17:46.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.008+0000 7fd947200640 1 -- 172.21.15.96:0/4000661565 >> v2:172.21.15.123:3300/0 conn(0x7fd9480d0430 msgr2=0x7fd9480d0860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:46.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.008+0000 7fd947200640 1 --2- 172.21.15.96:0/4000661565 >> v2:172.21.15.123:3300/0 conn(0x7fd9480d0430 0x7fd9480d0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:46.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.008+0000 7fd947200640 1 -- 172.21.15.96:0/4000661565 >> v1:172.21.15.123:6789/0 conn(0x7fd9480ce560 legacy=0x7fd9480b4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:46.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.008+0000 7fd947200640 1 -- 172.21.15.96:0/4000661565 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9480b3ba0 con 0x7fd94815f0c0 2024-05-28T06:17:46.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.009+0000 7fd947200640 1 --2- 172.21.15.96:0/4000661565 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd94815f0c0 0x7fd94815f530 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7fd9480d0f80 tx=0x7fd94002a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=327abff467b65141 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:46.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.009+0000 7fd945e00640 1 -- 172.21.15.96:0/4000661565 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd940036070 con 0x7fd94815f0c0 2024-05-28T06:17:46.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.009+0000 7fd945e00640 1 -- 172.21.15.96:0/4000661565 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd940031600 con 0x7fd94815f0c0 2024-05-28T06:17:46.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.009+0000 7fd945e00640 1 -- 172.21.15.96:0/4000661565 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9400319d0 con 0x7fd94815f0c0 2024-05-28T06:17:46.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.010+0000 7fd94ca00640 1 -- 172.21.15.96:0/4000661565 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd94815f0c0 msgr2=0x7fd94815f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:46.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.010+0000 7fd94ca00640 1 --2- 172.21.15.96:0/4000661565 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd94815f0c0 0x7fd94815f530 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7fd9480d0f80 tx=0x7fd94002a6f0 comp rx=0 tx=0).stop 2024-05-28T06:17:46.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.010+0000 7fd94ca00640 1 -- 172.21.15.96:0/4000661565 shutdown_connections 2024-05-28T06:17:46.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.010+0000 7fd94ca00640 1 --2- 172.21.15.96:0/4000661565 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd94815f0c0 0x7fd94815f530 unknown :-1 s=CLOSED pgs=1294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:46.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.010+0000 7fd94ca00640 1 --2- 172.21.15.96:0/4000661565 >> v2:172.21.15.123:3300/0 conn(0x7fd9480d0430 0x7fd9480d0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:46.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.010+0000 7fd94ca00640 1 -- 172.21.15.96:0/4000661565 >> 172.21.15.96:0/4000661565 conn(0x7fd94806bae0 msgr2=0x7fd94806bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:46.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.010+0000 7fd94ca00640 1 -- 172.21.15.96:0/4000661565 shutdown_connections 2024-05-28T06:17:46.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.010+0000 7fd94ca00640 1 -- 172.21.15.96:0/4000661565 wait complete. 2024-05-28T06:17:46.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.012+0000 7fd94ca00640 1 Processor -- start 2024-05-28T06:17:46.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.012+0000 7fd94ca00640 1 -- start start 2024-05-28T06:17:46.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.013+0000 7fd94ca00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9480ce560 0x7fd9480adab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:46.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.013+0000 7fd94ca00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd9480d0430 0x7fd9480b1b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:46.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.013+0000 7fd94ca00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd94815f0c0 0x7fd94809acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:46.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.013+0000 7fd94ca00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd94815c390 con 0x7fd9480ce560 2024-05-28T06:17:46.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.013+0000 7fd94ca00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd94815c0b0 con 0x7fd94815f0c0 2024-05-28T06:17:46.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.013+0000 7fd94ca00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fd94815c670 con 0x7fd9480d0430 2024-05-28T06:17:46.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.013+0000 7fd946800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9480ce560 0x7fd9480adab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:46.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.013+0000 7fd946800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9480ce560 0x7fd9480adab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49144/0 (socket says 172.21.15.96:49144) 2024-05-28T06:17:46.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.013+0000 7fd946800640 1 -- 172.21.15.96:0/1081155634 learned_addr learned my addr 172.21.15.96:0/1081155634 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:17:46.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.013+0000 7fd946800640 1 -- 172.21.15.96:0/1081155634 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd9480d0430 msgr2=0x7fd9480b1b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:46.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.013+0000 7fd946800640 1 --2- 172.21.15.96:0/1081155634 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd9480d0430 0x7fd9480b1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:46.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.013+0000 7fd946800640 1 -- 172.21.15.96:0/1081155634 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd94815f0c0 msgr2=0x7fd94809acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:46.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.013+0000 7fd946800640 1 --2- 172.21.15.96:0/1081155634 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd94815f0c0 0x7fd94809acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:46.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.013+0000 7fd946800640 1 -- 172.21.15.96:0/1081155634 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9480b20b0 con 0x7fd9480ce560 2024-05-28T06:17:46.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.014+0000 7fd946800640 1 --2- 172.21.15.96:0/1081155634 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9480ce560 0x7fd9480adab0 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7fd93800b3e0 tx=0x7fd93800b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:46.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.014+0000 7fd93f400640 1 -- 172.21.15.96:0/1081155634 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd938015020 con 0x7fd9480ce560 2024-05-28T06:17:46.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.014+0000 7fd93f400640 1 -- 172.21.15.96:0/1081155634 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd938011850 con 0x7fd9480ce560 2024-05-28T06:17:46.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.014+0000 7fd93f400640 1 -- 172.21.15.96:0/1081155634 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd938011c40 con 0x7fd9480ce560 2024-05-28T06:17:46.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.014+0000 7fd94ca00640 1 -- 172.21.15.96:0/1081155634 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd948147380 con 0x7fd9480ce560 2024-05-28T06:17:46.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.014+0000 7fd94ca00640 1 -- 172.21.15.96:0/1081155634 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd948147940 con 0x7fd9480ce560 2024-05-28T06:17:46.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.015+0000 7fd94ca00640 1 -- 172.21.15.96:0/1081155634 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9480689e0 con 0x7fd9480ce560 2024-05-28T06:17:46.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.016+0000 7fd93f400640 1 -- 172.21.15.96:0/1081155634 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 29) ==== 92672+0+0 (secure 0 0 0) 0x7fd938018020 con 0x7fd9480ce560 2024-05-28T06:17:46.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.017+0000 7fd93f400640 1 --2- 172.21.15.96:0/1081155634 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd92406daf0 0x7fd92406ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:46.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.017+0000 7fd93f400640 1 -- 172.21.15.96:0/1081155634 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 5851+0+0 (secure 0 0 0) 0x7fd93808aa30 con 0x7fd9480ce560 2024-05-28T06:17:46.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.018+0000 7fd93de00640 1 --2- 172.21.15.96:0/1081155634 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd92406daf0 0x7fd92406ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:46.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.019+0000 7fd93de00640 1 --2- 172.21.15.96:0/1081155634 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd92406daf0 0x7fd92406ffb0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fd94815b8e0 tx=0x7fd930006c90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:46.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.021+0000 7fd93f400640 1 -- 172.21.15.96:0/1081155634 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd93801f020 con 0x7fd9480ce560 2024-05-28T06:17:46.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.230+0000 7fd94ca00640 1 -- 172.21.15.96:0/1081155634 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone43639", "target": ["mon-mgr", ""]}) -- 0x7fd948147130 con 0x7fd92406daf0 2024-05-28T06:17:46.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.322+0000 7fd93f400640 1 -- 172.21.15.96:0/1081155634 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fd948147130 con 0x7fd92406daf0 2024-05-28T06:17:46.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.325+0000 7fd94ca00640 1 -- 172.21.15.96:0/1081155634 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd92406daf0 msgr2=0x7fd92406ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:46.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.325+0000 7fd94ca00640 1 --2- 172.21.15.96:0/1081155634 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd92406daf0 0x7fd92406ffb0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fd94815b8e0 tx=0x7fd930006c90 comp rx=0 tx=0).stop 2024-05-28T06:17:46.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.325+0000 7fd94ca00640 1 -- 172.21.15.96:0/1081155634 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9480ce560 msgr2=0x7fd9480adab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:46.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.325+0000 7fd94ca00640 1 --2- 172.21.15.96:0/1081155634 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9480ce560 0x7fd9480adab0 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7fd93800b3e0 tx=0x7fd93800b8a0 comp rx=0 tx=0).stop 2024-05-28T06:17:46.325 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.327+0000 7fd94ca00640 1 -- 172.21.15.96:0/1081155634 shutdown_connections 2024-05-28T06:17:46.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.327+0000 7fd94ca00640 1 --2- 172.21.15.96:0/1081155634 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd92406daf0 0x7fd92406ffb0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:46.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.327+0000 7fd94ca00640 1 --2- 172.21.15.96:0/1081155634 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd94815f0c0 0x7fd94809acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:46.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.327+0000 7fd94ca00640 1 --2- 172.21.15.96:0/1081155634 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd9480d0430 0x7fd9480b1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:46.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.327+0000 7fd94ca00640 1 --2- 172.21.15.96:0/1081155634 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9480ce560 0x7fd9480adab0 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:46.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.327+0000 7fd94ca00640 1 -- 172.21.15.96:0/1081155634 >> 172.21.15.96:0/1081155634 conn(0x7fd94806bae0 msgr2=0x7fd94806bec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:46.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.327+0000 7fd94ca00640 1 -- 172.21.15.96:0/1081155634 shutdown_connections 2024-05-28T06:17:46.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.327+0000 7fd94ca00640 1 -- 172.21.15.96:0/1081155634 wait complete. 2024-05-28T06:17:46.347 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone43635 2024-05-28T06:17:46.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.619+0000 7f7724a00640 1 Processor -- start 2024-05-28T06:17:46.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.620+0000 7f7724a00640 1 -- start start 2024-05-28T06:17:46.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.622+0000 7f7724a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f771803c0a0 0x7f77180e7780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:46.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.622+0000 7f7724a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f7718053cb0 0x7f7718054120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:46.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.622+0000 7f7724a00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7718054660 con 0x7f77180e3160 2024-05-28T06:17:46.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.622+0000 7f7724a00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f7718007b70 con 0x7f7718053cb0 2024-05-28T06:17:46.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.622+0000 7f7724a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f7718058810 con 0x7f771803c0a0 2024-05-28T06:17:46.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.622+0000 7f771e800640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f77180e3160 0x7f7718058050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:40120/0 (socket says 172.21.15.96:40120) 2024-05-28T06:17:46.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.622+0000 7f771e800640 1 -- 172.21.15.96:0/3575796490 learned_addr learned my addr 172.21.15.96:0/3575796490 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:17:46.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.622+0000 7f771f200640 1 --2- 172.21.15.96:0/3575796490 >> v2:172.21.15.96:3300/0 conn(0x7f7718053cb0 0x7f7718054120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:46.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.623+0000 7f771de00640 1 --2- 172.21.15.96:0/3575796490 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f771803c0a0 0x7f77180e7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:46.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.623+0000 7f771f200640 1 -- 172.21.15.96:0/3575796490 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f771803c0a0 msgr2=0x7f77180e7780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:46.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.623+0000 7f771f200640 1 --2- 172.21.15.96:0/3575796490 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f771803c0a0 0x7f77180e7780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:46.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.623+0000 7f771f200640 1 -- 172.21.15.96:0/3575796490 >> v1:172.21.15.96:6789/0 conn(0x7f77180e3160 legacy=0x7f7718058050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:46.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.623+0000 7f771f200640 1 -- 172.21.15.96:0/3575796490 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7718058af0 con 0x7f7718053cb0 2024-05-28T06:17:46.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.623+0000 7f771de00640 1 --2- 172.21.15.96:0/3575796490 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f771803c0a0 0x7f77180e7780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-28T06:17:46.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.627+0000 7f771f200640 1 --2- 172.21.15.96:0/3575796490 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7718053cb0 0x7f7718054120 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f772002db50 tx=0x7f77200a5a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f251450da1fb0ca server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:46.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.628+0000 7f771d400640 1 -- 172.21.15.96:0/3575796490 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77200a9910 con 0x7f7718053cb0 2024-05-28T06:17:46.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.628+0000 7f771d400640 1 -- 172.21.15.96:0/3575796490 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77200a9ab0 con 0x7f7718053cb0 2024-05-28T06:17:46.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.628+0000 7f771d400640 1 -- 172.21.15.96:0/3575796490 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77200b1400 con 0x7f7718053cb0 2024-05-28T06:17:46.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.629+0000 7f7724a00640 1 -- 172.21.15.96:0/3575796490 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7718053cb0 msgr2=0x7f7718054120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:46.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.629+0000 7f7724a00640 1 --2- 172.21.15.96:0/3575796490 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7718053cb0 0x7f7718054120 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f772002db50 tx=0x7f77200a5a90 comp rx=0 tx=0).stop 2024-05-28T06:17:46.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.629+0000 7f7724a00640 1 -- 172.21.15.96:0/3575796490 shutdown_connections 2024-05-28T06:17:46.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.629+0000 7f7724a00640 1 --2- 172.21.15.96:0/3575796490 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7718053cb0 0x7f7718054120 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:46.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.629+0000 7f7724a00640 1 --2- 172.21.15.96:0/3575796490 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f771803c0a0 0x7f77180e7780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:46.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.629+0000 7f7724a00640 1 -- 172.21.15.96:0/3575796490 >> 172.21.15.96:0/3575796490 conn(0x7f7718058ee0 msgr2=0x7f77180592c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:46.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.630+0000 7f7724a00640 1 -- 172.21.15.96:0/3575796490 shutdown_connections 2024-05-28T06:17:46.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.630+0000 7f7724a00640 1 -- 172.21.15.96:0/3575796490 wait complete. 2024-05-28T06:17:46.631 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.632+0000 7f7724a00640 1 Processor -- start 2024-05-28T06:17:46.631 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.632+0000 7f7724a00640 1 -- start start 2024-05-28T06:17:46.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.633+0000 7f7724a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f771803c0a0 0x7f771800a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:46.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.633+0000 7f7724a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7718053cb0 0x7f771804da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:46.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.633+0000 7f7724a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f77180e3160 0x7f77180e5650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:46.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.633+0000 7f7724a00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f77180e3540 con 0x7f771803c0a0 2024-05-28T06:17:46.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.633+0000 7f7724a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f7718006640 con 0x7f7718053cb0 2024-05-28T06:17:46.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.633+0000 7f7724a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f7718006b60 con 0x7f77180e3160 2024-05-28T06:17:46.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.634+0000 7f771de00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7718053cb0 0x7f771804da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:46.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.634+0000 7f771de00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7718053cb0 0x7f771804da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.96:56866/0 (socket says 172.21.15.96:56866) 2024-05-28T06:17:46.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.634+0000 7f771de00640 1 -- 172.21.15.96:0/1280413668 learned_addr learned my addr 172.21.15.96:0/1280413668 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:17:46.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.634+0000 7f771f200640 1 --2- 172.21.15.96:0/1280413668 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f77180e3160 0x7f77180e5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:46.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.635+0000 7f771e800640 1 --2- 172.21.15.96:0/1280413668 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f771803c0a0 0x7f771800a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:46.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.635+0000 7f771de00640 1 -- 172.21.15.96:0/1280413668 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f77180e3160 msgr2=0x7f77180e5650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:46.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.635+0000 7f771de00640 1 --2- 172.21.15.96:0/1280413668 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f77180e3160 0x7f77180e5650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:46.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.635+0000 7f771de00640 1 -- 172.21.15.96:0/1280413668 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f771803c0a0 msgr2=0x7f771800a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:46.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.635+0000 7f771de00640 1 --2- 172.21.15.96:0/1280413668 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f771803c0a0 0x7f771800a150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:46.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.635+0000 7f771de00640 1 -- 172.21.15.96:0/1280413668 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f771804f3a0 con 0x7f7718053cb0 2024-05-28T06:17:46.635 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.635+0000 7f771f200640 1 --2- 172.21.15.96:0/1280413668 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f77180e3160 0x7f77180e5650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-28T06:17:46.635 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.636+0000 7f771de00640 1 --2- 172.21.15.96:0/1280413668 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7718053cb0 0x7f771804da60 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f770c00e9b0 tx=0x7f770c00ee70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:46.635 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.637+0000 7f7716a00640 1 -- 172.21.15.96:0/1280413668 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f770c017070 con 0x7f7718053cb0 2024-05-28T06:17:46.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.637+0000 7f7716a00640 1 -- 172.21.15.96:0/1280413668 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f770c00ca00 con 0x7f7718053cb0 2024-05-28T06:17:46.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.637+0000 7f7716a00640 1 -- 172.21.15.96:0/1280413668 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f770c00cdf0 con 0x7f7718053cb0 2024-05-28T06:17:46.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.637+0000 7f7724a00640 1 -- 172.21.15.96:0/1280413668 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f771804e300 con 0x7f7718053cb0 2024-05-28T06:17:46.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.637+0000 7f7724a00640 1 -- 172.21.15.96:0/1280413668 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f771804e8c0 con 0x7f7718053cb0 2024-05-28T06:17:46.638 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.640+0000 7f7724a00640 1 -- 172.21.15.96:0/1280413668 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7718005390 con 0x7f7718053cb0 2024-05-28T06:17:46.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.642+0000 7f7716a00640 1 -- 172.21.15.96:0/1280413668 <== mon.1 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 29) ==== 92672+0+0 (secure 0 0 0) 0x7f770c01a070 con 0x7f7718053cb0 2024-05-28T06:17:46.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.643+0000 7f7716a00640 1 --2- 172.21.15.96:0/1280413668 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f770006daf0 0x7f770006ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:46.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.643+0000 7f7716a00640 1 -- 172.21.15.96:0/1280413668 <== mon.1 v2:172.21.15.123:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 5851+0+0 (secure 0 0 0) 0x7f770c08a670 con 0x7f7718053cb0 2024-05-28T06:17:46.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.644+0000 7f771e800640 1 --2- 172.21.15.96:0/1280413668 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f770006daf0 0x7f770006ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:46.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.646+0000 7f771e800640 1 --2- 172.21.15.96:0/1280413668 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f770006daf0 0x7f770006ffb0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f771000b830 tx=0x7f7710008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:46.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.652+0000 7f7716a00640 1 -- 172.21.15.96:0/1280413668 <== mon.1 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f770c059aa0 con 0x7f7718053cb0 2024-05-28T06:17:46.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:46.985+0000 7f7724a00640 1 -- 172.21.15.96:0/1280413668 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone43635", "target": ["mon-mgr", ""]}) -- 0x7f7718056bf0 con 0x7f770006daf0 2024-05-28T06:17:47.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.019+0000 7f7716a00640 1 -- 172.21.15.96:0/1280413668 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f7718056bf0 con 0x7f770006daf0 2024-05-28T06:17:47.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.021+0000 7f7724a00640 1 -- 172.21.15.96:0/1280413668 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f770006daf0 msgr2=0x7f770006ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:47.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.021+0000 7f7724a00640 1 --2- 172.21.15.96:0/1280413668 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f770006daf0 0x7f770006ffb0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f771000b830 tx=0x7f7710008040 comp rx=0 tx=0).stop 2024-05-28T06:17:47.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.021+0000 7f7724a00640 1 -- 172.21.15.96:0/1280413668 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7718053cb0 msgr2=0x7f771804da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:47.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.021+0000 7f7724a00640 1 --2- 172.21.15.96:0/1280413668 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7718053cb0 0x7f771804da60 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f770c00e9b0 tx=0x7f770c00ee70 comp rx=0 tx=0).stop 2024-05-28T06:17:47.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.021+0000 7f7724a00640 1 -- 172.21.15.96:0/1280413668 shutdown_connections 2024-05-28T06:17:47.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.021+0000 7f7724a00640 1 --2- 172.21.15.96:0/1280413668 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f770006daf0 0x7f770006ffb0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:47.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.021+0000 7f7724a00640 1 --2- 172.21.15.96:0/1280413668 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f77180e3160 0x7f77180e5650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:47.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.021+0000 7f7724a00640 1 --2- 172.21.15.96:0/1280413668 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7718053cb0 0x7f771804da60 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:47.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.021+0000 7f7724a00640 1 --2- 172.21.15.96:0/1280413668 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f771803c0a0 0x7f771800a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:47.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.021+0000 7f7724a00640 1 -- 172.21.15.96:0/1280413668 >> 172.21.15.96:0/1280413668 conn(0x7f7718058ee0 msgr2=0x7f7718057b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:47.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.021+0000 7f7724a00640 1 -- 172.21.15.96:0/1280413668 shutdown_connections 2024-05-28T06:17:47.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.021+0000 7f7724a00640 1 -- 172.21.15.96:0/1280413668 wait complete. 2024-05-28T06:17:47.038 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone43631 2024-05-28T06:17:47.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.228+0000 7f3cb5e00640 1 Processor -- start 2024-05-28T06:17:47.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.228+0000 7f3cb5e00640 1 -- start start 2024-05-28T06:17:47.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.229+0000 7f3cb5e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3cb00ce560 0x7f3cb00b4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:47.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.229+0000 7f3cb5e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f3cb00d0430 0x7f3cb00d0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:47.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.229+0000 7f3cb5e00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3cb00d0d70 con 0x7f3cb015f0c0 2024-05-28T06:17:47.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.229+0000 7f3cb5e00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f3cb015bb30 con 0x7f3cb00d0430 2024-05-28T06:17:47.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.229+0000 7f3cb5e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f3cb00b3ba0 con 0x7f3cb00ce560 2024-05-28T06:17:47.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.230+0000 7f3cb4a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3cb00ce560 0x7f3cb00b4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:47.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.230+0000 7f3cb4a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3cb00ce560 0x7f3cb00b4b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.96:34466/0 (socket says 172.21.15.96:34466) 2024-05-28T06:17:47.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.230+0000 7f3cb4a00640 1 -- 172.21.15.96:0/228034880 learned_addr learned my addr 172.21.15.96:0/228034880 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:17:47.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.230+0000 7f3cb5400640 1 --1- 172.21.15.96:0/228034880 >> v1:172.21.15.123:6789/0 conn(0x7f3cb015f0c0 0x7f3cb015f550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.96:46402/0 (socket says 172.21.15.96:46402) 2024-05-28T06:17:47.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.230+0000 7f3cafe00640 1 --2- 172.21.15.96:0/228034880 >> v2:172.21.15.123:3300/0 conn(0x7f3cb00d0430 0x7f3cb00d0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:47.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.231+0000 7f3caf400640 1 -- 172.21.15.96:0/228034880 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3244775362 0 0) 0x7f3cb00d0d70 con 0x7f3cb015f0c0 2024-05-28T06:17:47.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.231+0000 7f3caf400640 1 -- 172.21.15.96:0/228034880 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3c94003520 con 0x7f3cb015f0c0 2024-05-28T06:17:47.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.231+0000 7f3cb4a00640 1 -- 172.21.15.96:0/228034880 >> v2:172.21.15.123:3300/0 conn(0x7f3cb00d0430 msgr2=0x7f3cb00d0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:47.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.231+0000 7f3cb4a00640 1 --2- 172.21.15.96:0/228034880 >> v2:172.21.15.123:3300/0 conn(0x7f3cb00d0430 0x7f3cb00d0830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:47.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.231+0000 7f3cb4a00640 1 -- 172.21.15.96:0/228034880 >> v1:172.21.15.123:6789/0 conn(0x7f3cb015f0c0 legacy=0x7f3cb015f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:47.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.231+0000 7f3cb4a00640 1 -- 172.21.15.96:0/228034880 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cb015bdd0 con 0x7f3cb00ce560 2024-05-28T06:17:47.231 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.232+0000 7f3cb4a00640 1 --2- 172.21.15.96:0/228034880 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3cb00ce560 0x7f3cb00b4b80 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f3ca0009870 tx=0x7f3ca002a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e0d388ac3c193e78 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:47.231 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.233+0000 7f3caf400640 1 -- 172.21.15.96:0/228034880 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ca0037070 con 0x7f3cb00ce560 2024-05-28T06:17:47.231 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.233+0000 7f3caf400640 1 -- 172.21.15.96:0/228034880 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ca0032440 con 0x7f3cb00ce560 2024-05-28T06:17:47.231 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.233+0000 7f3caf400640 1 -- 172.21.15.96:0/228034880 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ca00327f0 con 0x7f3cb00ce560 2024-05-28T06:17:47.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.233+0000 7f3cb5e00640 1 -- 172.21.15.96:0/228034880 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3cb00ce560 msgr2=0x7f3cb00b4b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:47.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.233+0000 7f3cb5e00640 1 --2- 172.21.15.96:0/228034880 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3cb00ce560 0x7f3cb00b4b80 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f3ca0009870 tx=0x7f3ca002a270 comp rx=0 tx=0).stop 2024-05-28T06:17:47.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.233+0000 7f3cb5e00640 1 -- 172.21.15.96:0/228034880 shutdown_connections 2024-05-28T06:17:47.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.233+0000 7f3cb5e00640 1 --2- 172.21.15.96:0/228034880 >> v2:172.21.15.123:3300/0 conn(0x7f3cb00d0430 0x7f3cb00d0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:47.233 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.233+0000 7f3cb5e00640 1 --2- 172.21.15.96:0/228034880 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3cb00ce560 0x7f3cb00b4b80 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:47.233 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.233+0000 7f3cb5e00640 1 -- 172.21.15.96:0/228034880 >> 172.21.15.96:0/228034880 conn(0x7f3cb006bae0 msgr2=0x7f3cb006bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:47.233 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.233+0000 7f3cb5e00640 1 -- 172.21.15.96:0/228034880 shutdown_connections 2024-05-28T06:17:47.233 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.234+0000 7f3cb5e00640 1 -- 172.21.15.96:0/228034880 wait complete. 2024-05-28T06:17:47.233 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.235+0000 7f3cb5e00640 1 Processor -- start 2024-05-28T06:17:47.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.235+0000 7f3cb5e00640 1 -- start start 2024-05-28T06:17:47.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.236+0000 7f3cb5e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3cb00ce560 0x7f3cb00adb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:47.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.236+0000 7f3cb5e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3cb00d0430 0x7f3cb00cbcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:47.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.236+0000 7f3cb5e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3cb015f0c0 0x7f3cb00b01c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:47.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.236+0000 7f3cb5e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f3cb015c670 con 0x7f3cb00d0430 2024-05-28T06:17:47.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.236+0000 7f3cb5e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f3cb015c390 con 0x7f3cb00ce560 2024-05-28T06:17:47.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.236+0000 7f3cb5e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f3cb006f630 con 0x7f3cb015f0c0 2024-05-28T06:17:47.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.236+0000 7f3cafe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3cb00d0430 0x7f3cb00cbcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:47.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.236+0000 7f3cafe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3cb00d0430 0x7f3cb00cbcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:49182/0 (socket says 172.21.15.96:49182) 2024-05-28T06:17:47.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.236+0000 7f3cafe00640 1 -- 172.21.15.96:0/2520411022 learned_addr learned my addr 172.21.15.96:0/2520411022 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:17:47.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.237+0000 7f3cb5400640 1 --2- 172.21.15.96:0/2520411022 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3cb015f0c0 0x7f3cb00b01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:47.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.237+0000 7f3cb4a00640 1 --2- 172.21.15.96:0/2520411022 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3cb00ce560 0x7f3cb00adb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:47.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.237+0000 7f3cafe00640 1 -- 172.21.15.96:0/2520411022 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3cb015f0c0 msgr2=0x7f3cb00b01c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:47.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.237+0000 7f3cafe00640 1 --2- 172.21.15.96:0/2520411022 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3cb015f0c0 0x7f3cb00b01c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:47.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.237+0000 7f3cafe00640 1 -- 172.21.15.96:0/2520411022 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3cb00ce560 msgr2=0x7f3cb00adb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:47.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.237+0000 7f3cafe00640 1 --2- 172.21.15.96:0/2520411022 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3cb00ce560 0x7f3cb00adb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:47.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.237+0000 7f3cafe00640 1 -- 172.21.15.96:0/2520411022 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cb00ae3c0 con 0x7f3cb00d0430 2024-05-28T06:17:47.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.237+0000 7f3cafe00640 1 --2- 172.21.15.96:0/2520411022 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3cb00d0430 0x7f3cb00cbcb0 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f3c9800e9b0 tx=0x7f3c9800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:47.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.237+0000 7f3cacc00640 1 -- 172.21.15.96:0/2520411022 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c98017070 con 0x7f3cb00d0430 2024-05-28T06:17:47.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.237+0000 7f3cacc00640 1 -- 172.21.15.96:0/2520411022 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c9800c9a0 con 0x7f3cb00d0430 2024-05-28T06:17:47.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.237+0000 7f3cacc00640 1 -- 172.21.15.96:0/2520411022 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c9800cd90 con 0x7f3cb00d0430 2024-05-28T06:17:47.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.237+0000 7f3cb5e00640 1 -- 172.21.15.96:0/2520411022 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3cb00ae6a0 con 0x7f3cb00d0430 2024-05-28T06:17:47.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.238+0000 7f3cb5e00640 1 -- 172.21.15.96:0/2520411022 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3cb00aec60 con 0x7f3cb00d0430 2024-05-28T06:17:47.238 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.239+0000 7f3cb5e00640 1 -- 172.21.15.96:0/2520411022 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3cb00689e0 con 0x7f3cb00d0430 2024-05-28T06:17:47.238 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.240+0000 7f3cacc00640 1 -- 172.21.15.96:0/2520411022 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 29) ==== 92672+0+0 (secure 0 0 0) 0x7f3c9801a070 con 0x7f3cb00d0430 2024-05-28T06:17:47.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.240+0000 7f3cacc00640 1 --2- 172.21.15.96:0/2520411022 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3c8c06daf0 0x7f3c8c06ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:47.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.241+0000 7f3cacc00640 1 -- 172.21.15.96:0/2520411022 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 5851+0+0 (secure 0 0 0) 0x7f3c9808ae40 con 0x7f3cb00d0430 2024-05-28T06:17:47.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.241+0000 7f3cb4a00640 1 --2- 172.21.15.96:0/2520411022 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3c8c06daf0 0x7f3c8c06ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:47.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.242+0000 7f3cb4a00640 1 --2- 172.21.15.96:0/2520411022 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3c8c06daf0 0x7f3c8c06ffb0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f3ca0034000 tx=0x7f3ca002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:47.244 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.246+0000 7f3cacc00640 1 -- 172.21.15.96:0/2520411022 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3c9805a950 con 0x7f3cb00d0430 2024-05-28T06:17:47.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.617+0000 7f3cb5e00640 1 -- 172.21.15.96:0/2520411022 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone43631", "target": ["mon-mgr", ""]}) -- 0x7f3cb015c890 con 0x7f3c8c06daf0 2024-05-28T06:17:47.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.652+0000 7f3cacc00640 1 -- 172.21.15.96:0/2520411022 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f3cb015c890 con 0x7f3c8c06daf0 2024-05-28T06:17:47.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.656+0000 7f3cb5e00640 1 -- 172.21.15.96:0/2520411022 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3c8c06daf0 msgr2=0x7f3c8c06ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:47.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.656+0000 7f3cb5e00640 1 --2- 172.21.15.96:0/2520411022 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3c8c06daf0 0x7f3c8c06ffb0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f3ca0034000 tx=0x7f3ca002e000 comp rx=0 tx=0).stop 2024-05-28T06:17:47.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.656+0000 7f3cb5e00640 1 -- 172.21.15.96:0/2520411022 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3cb00d0430 msgr2=0x7f3cb00cbcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:47.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.656+0000 7f3cb5e00640 1 --2- 172.21.15.96:0/2520411022 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3cb00d0430 0x7f3cb00cbcb0 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f3c9800e9b0 tx=0x7f3c9800ee70 comp rx=0 tx=0).stop 2024-05-28T06:17:47.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.657+0000 7f3cb5e00640 1 -- 172.21.15.96:0/2520411022 shutdown_connections 2024-05-28T06:17:47.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.657+0000 7f3cb5e00640 1 --2- 172.21.15.96:0/2520411022 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3c8c06daf0 0x7f3c8c06ffb0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:47.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.657+0000 7f3cb5e00640 1 --2- 172.21.15.96:0/2520411022 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3cb015f0c0 0x7f3cb00b01c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:47.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.657+0000 7f3cb5e00640 1 --2- 172.21.15.96:0/2520411022 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3cb00d0430 0x7f3cb00cbcb0 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:47.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.657+0000 7f3cb5e00640 1 --2- 172.21.15.96:0/2520411022 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3cb00ce560 0x7f3cb00adb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:47.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.657+0000 7f3cb5e00640 1 -- 172.21.15.96:0/2520411022 >> 172.21.15.96:0/2520411022 conn(0x7f3cb006bae0 msgr2=0x7f3cb00d0830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:47.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.657+0000 7f3cb5e00640 1 -- 172.21.15.96:0/2520411022 shutdown_connections 2024-05-28T06:17:47.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47.657+0000 7f3cb5e00640 1 -- 172.21.15.96:0/2520411022 wait complete. 2024-05-28T06:17:47.678 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2024-05-28T06:17:47.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-28T06:17:47.818 INFO:teuthology.orchestra.run.smithi096.stdout:2 2024-05-28T06:17:47.819 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-28T06:17:47.832 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-05-28T06:17:47.832 DEBUG:teuthology.orchestra.run.smithi096:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-05-28T06:17:47.943 DEBUG:teuthology.orchestra.run.smithi123:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-05-28T06:17:48.063 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-05-28T06:17:48.063 INFO:tasks.cephfs.mount:Terminating background process 2024-05-28T06:17:48.063 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:17:48.063 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:17:48.110 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:17:48.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.349+0000 7f795e200640 1 Processor -- start 2024-05-28T06:17:48.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.350+0000 7f795e200640 1 -- start start 2024-05-28T06:17:48.350 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.351+0000 7f795e200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f7958162e70 0x7f795816d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:48.350 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.351+0000 7f795e200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79580b4350 con 0x7f79580d04a0 2024-05-28T06:17:48.350 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.352+0000 7f795e200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79580b32d0 con 0x7f795806f6e0 2024-05-28T06:17:48.350 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.352+0000 7f795e200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f79580b3ec0 con 0x7f7958162e70 2024-05-28T06:17:48.351 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.352+0000 7f7957400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f79580d04a0 0x7f79580d0880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:40122/0 (socket says 172.21.15.96:40122) 2024-05-28T06:17:48.351 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.352+0000 7f7957400640 1 -- 172.21.15.96:0/884552462 learned_addr learned my addr 172.21.15.96:0/884552462 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:17:48.351 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.352+0000 7f7957e00640 1 --2- 172.21.15.96:0/884552462 >> v2:172.21.15.123:3300/0 conn(0x7f7958162e70 0x7f795816d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:48.352 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.353+0000 7f7956000640 1 -- 172.21.15.96:0/884552462 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 368784819 0 0) 0x7f79580b32d0 con 0x7f795806f6e0 2024-05-28T06:17:48.352 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.353+0000 7f7957e00640 1 -- 172.21.15.96:0/884552462 >> v1:172.21.15.123:6789/0 conn(0x7f795806f6e0 legacy=0x7f79580cd4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:48.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.353+0000 7f7957e00640 1 -- 172.21.15.96:0/884552462 >> v1:172.21.15.96:6789/0 conn(0x7f79580d04a0 legacy=0x7f79580d0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:48.353 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.353+0000 7f7957e00640 1 -- 172.21.15.96:0/884552462 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79580b3590 con 0x7f7958162e70 2024-05-28T06:17:48.355 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.356+0000 7f7957e00640 1 --2- 172.21.15.96:0/884552462 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7958162e70 0x7f795816d7f0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f7950009d90 tx=0x7f795002a680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c95307582ec5866f server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:48.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.357+0000 7f7956000640 1 -- 172.21.15.96:0/884552462 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7950007740 con 0x7f7958162e70 2024-05-28T06:17:48.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.357+0000 7f7956000640 1 -- 172.21.15.96:0/884552462 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79500078e0 con 0x7f7958162e70 2024-05-28T06:17:48.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.357+0000 7f7956000640 1 -- 172.21.15.96:0/884552462 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7950007cb0 con 0x7f7958162e70 2024-05-28T06:17:48.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.358+0000 7f795e200640 1 -- 172.21.15.96:0/884552462 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7958162e70 msgr2=0x7f795816d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:48.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.358+0000 7f795e200640 1 --2- 172.21.15.96:0/884552462 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7958162e70 0x7f795816d7f0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f7950009d90 tx=0x7f795002a680 comp rx=0 tx=0).stop 2024-05-28T06:17:48.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.358+0000 7f795e200640 1 -- 172.21.15.96:0/884552462 shutdown_connections 2024-05-28T06:17:48.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.358+0000 7f795e200640 1 --2- 172.21.15.96:0/884552462 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7958162e70 0x7f795816d7f0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:48.358 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.358+0000 7f795e200640 1 -- 172.21.15.96:0/884552462 >> 172.21.15.96:0/884552462 conn(0x7f795806bae0 msgr2=0x7f795806bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:48.358 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.359+0000 7f795e200640 1 -- 172.21.15.96:0/884552462 shutdown_connections 2024-05-28T06:17:48.358 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.359+0000 7f795e200640 1 -- 172.21.15.96:0/884552462 wait complete. 2024-05-28T06:17:48.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.362+0000 7f795e200640 1 Processor -- start 2024-05-28T06:17:48.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.362+0000 7f795e200640 1 -- start start 2024-05-28T06:17:48.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.363+0000 7f795e200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f795806f6e0 0x7f79581f7f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:48.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.363+0000 7f795e200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f79580d04a0 0x7f79581656f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:48.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.363+0000 7f795e200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7958162e70 0x7f79580c59c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:48.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.363+0000 7f795e200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f79580cf240 con 0x7f79580d04a0 2024-05-28T06:17:48.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.363+0000 7f795e200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f79580b3b50 con 0x7f795806f6e0 2024-05-28T06:17:48.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.363+0000 7f795e200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f79580cf3c0 con 0x7f7958162e70 2024-05-28T06:17:48.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.364+0000 7f7957400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f795806f6e0 0x7f79581f7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:48.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.364+0000 7f7957400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f795806f6e0 0x7f79581f7f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.96:56926/0 (socket says 172.21.15.96:56926) 2024-05-28T06:17:48.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.364+0000 7f7957400640 1 -- 172.21.15.96:0/1147538454 learned_addr learned my addr 172.21.15.96:0/1147538454 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:17:48.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.364+0000 7f7957e00640 1 --2- 172.21.15.96:0/1147538454 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7958162e70 0x7f79580c59c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:48.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.364+0000 7f7956a00640 1 --2- 172.21.15.96:0/1147538454 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f79580d04a0 0x7f79581656f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:48.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.365+0000 7f7957400640 1 -- 172.21.15.96:0/1147538454 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7958162e70 msgr2=0x7f79580c59c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:48.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.365+0000 7f7957400640 1 --2- 172.21.15.96:0/1147538454 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7958162e70 0x7f79580c59c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:48.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.365+0000 7f7957400640 1 -- 172.21.15.96:0/1147538454 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f79580d04a0 msgr2=0x7f79581656f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:48.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.365+0000 7f7957400640 1 --2- 172.21.15.96:0/1147538454 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f79580d04a0 0x7f79581656f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:48.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.365+0000 7f7957400640 1 -- 172.21.15.96:0/1147538454 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79581641b0 con 0x7f795806f6e0 2024-05-28T06:17:48.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.365+0000 7f7956a00640 1 --2- 172.21.15.96:0/1147538454 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f79580d04a0 0x7f79581656f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-28T06:17:48.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.365+0000 7f7957400640 1 --2- 172.21.15.96:0/1147538454 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f795806f6e0 0x7f79581f7f50 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f794800c4a0 tx=0x7f794800c960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:48.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.366+0000 7f794f400640 1 -- 172.21.15.96:0/1147538454 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7948017070 con 0x7f795806f6e0 2024-05-28T06:17:48.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.366+0000 7f794f400640 1 -- 172.21.15.96:0/1147538454 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7948005ce0 con 0x7f795806f6e0 2024-05-28T06:17:48.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.366+0000 7f794f400640 1 -- 172.21.15.96:0/1147538454 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7948007270 con 0x7f795806f6e0 2024-05-28T06:17:48.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.366+0000 7f795e200640 1 -- 172.21.15.96:0/1147538454 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7958164490 con 0x7f795806f6e0 2024-05-28T06:17:48.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.366+0000 7f795e200640 1 -- 172.21.15.96:0/1147538454 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7958164a50 con 0x7f795806f6e0 2024-05-28T06:17:48.368 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.370+0000 7f794f400640 1 -- 172.21.15.96:0/1147538454 <== mon.1 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 29) ==== 92672+0+0 (secure 0 0 0) 0x7f794801a070 con 0x7f795806f6e0 2024-05-28T06:17:48.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.372+0000 7f795e200640 1 -- 172.21.15.96:0/1147538454 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79580689e0 con 0x7f795806f6e0 2024-05-28T06:17:48.373 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.374+0000 7f794f400640 1 --2- 172.21.15.96:0/1147538454 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f793806daf0 0x7f793806ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:48.373 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.374+0000 7f794f400640 1 -- 172.21.15.96:0/1147538454 <== mon.1 v2:172.21.15.123:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 5851+0+0 (secure 0 0 0) 0x7f7948089d70 con 0x7f795806f6e0 2024-05-28T06:17:48.373 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.375+0000 7f7956a00640 1 --2- 172.21.15.96:0/1147538454 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f793806daf0 0x7f793806ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:48.374 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.376+0000 7f7956a00640 1 --2- 172.21.15.96:0/1147538454 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f793806daf0 0x7f793806ffb0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f79580c4fa0 tx=0x7f7944009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:48.381 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.382+0000 7f794f400640 1 -- 172.21.15.96:0/1147538454 <== mon.1 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7948059230 con 0x7f795806f6e0 2024-05-28T06:17:48.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.590+0000 7f795e200640 1 -- 172.21.15.96:0/1147538454 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f795806cc30 con 0x7f795806f6e0 2024-05-28T06:17:48.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.594+0000 7f794f400640 1 -- 172.21.15.96:0/1147538454 <== mon.1 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) ==== 74+0+16932 (secure 0 0 0) 0x7f794805d530 con 0x7f795806f6e0 2024-05-28T06:17:48.593 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:17:48.593 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":77,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:17:41.730115+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-05-28T06:14:50.098829+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-05-28T06:14:51.318311+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":77,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"11.17","mappings":[{"from":6,"to":0}]},{"pgid":"11.28","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":11,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[{"pool":11,"snaps":[{"begin":2,"length":2}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:17:48.595 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.596+0000 7f795e200640 1 -- 172.21.15.96:0/1147538454 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f793806daf0 msgr2=0x7f793806ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:48.595 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.596+0000 7f795e200640 1 --2- 172.21.15.96:0/1147538454 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f793806daf0 0x7f793806ffb0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f79580c4fa0 tx=0x7f7944009040 comp rx=0 tx=0).stop 2024-05-28T06:17:48.595 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.596+0000 7f795e200640 1 -- 172.21.15.96:0/1147538454 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f795806f6e0 msgr2=0x7f79581f7f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:48.595 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.596+0000 7f795e200640 1 --2- 172.21.15.96:0/1147538454 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f795806f6e0 0x7f79581f7f50 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f794800c4a0 tx=0x7f794800c960 comp rx=0 tx=0).stop 2024-05-28T06:17:48.595 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.596+0000 7f795e200640 1 -- 172.21.15.96:0/1147538454 shutdown_connections 2024-05-28T06:17:48.595 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.596+0000 7f795e200640 1 --2- 172.21.15.96:0/1147538454 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f793806daf0 0x7f793806ffb0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:48.595 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.596+0000 7f795e200640 1 --2- 172.21.15.96:0/1147538454 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7958162e70 0x7f79580c59c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:48.595 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.596+0000 7f795e200640 1 --2- 172.21.15.96:0/1147538454 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f79580d04a0 0x7f79581656f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:48.595 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.596+0000 7f795e200640 1 --2- 172.21.15.96:0/1147538454 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f795806f6e0 0x7f79581f7f50 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:48.595 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.596+0000 7f795e200640 1 -- 172.21.15.96:0/1147538454 >> 172.21.15.96:0/1147538454 conn(0x7f795806bae0 msgr2=0x7f795806bec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:48.595 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.596+0000 7f795e200640 1 -- 172.21.15.96:0/1147538454 shutdown_connections 2024-05-28T06:17:48.596 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.596+0000 7f795e200640 1 -- 172.21.15.96:0/1147538454 wait complete. 2024-05-28T06:17:48.612 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-28T06:03:59.517102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-28T06:14:50.098829+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '73', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2024-05-28T06:14:51.318311+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '77', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 77, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-28T06:17:48.612 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-28T06:17:48.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.806+0000 7fd7edc00640 1 Processor -- start 2024-05-28T06:17:48.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.807+0000 7fd7edc00640 1 -- start start 2024-05-28T06:17:48.807 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.808+0000 7fd7edc00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fd7e80b4120 0x7fd7e8163ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:48.807 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.808+0000 7fd7edc00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7e81640e0 con 0x7fd7e8162ac0 2024-05-28T06:17:48.807 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.808+0000 7fd7edc00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7e80b3360 con 0x7fd7e80d0d50 2024-05-28T06:17:48.807 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.808+0000 7fd7edc00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fd7e80b3c90 con 0x7fd7e80b4120 2024-05-28T06:17:48.807 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.808+0000 7fd7e6a00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fd7e8162ac0 0x7fd7e80d0640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:59858/0 (socket says 172.21.15.96:59858) 2024-05-28T06:17:48.807 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.808+0000 7fd7e6a00640 1 -- 172.21.15.96:0/3319652543 learned_addr learned my addr 172.21.15.96:0/3319652543 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:17:48.809 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.809+0000 7fd7e6000640 1 -- 172.21.15.96:0/3319652543 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2402155298 0 0) 0x7fd7e81640e0 con 0x7fd7e8162ac0 2024-05-28T06:17:48.809 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.809+0000 7fd7e7400640 1 --2- 172.21.15.96:0/3319652543 >> v2:172.21.15.123:3300/0 conn(0x7fd7e80b4120 0x7fd7e8163ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:48.809 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.809+0000 7fd7e6000640 1 -- 172.21.15.96:0/3319652543 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd7cc003520 con 0x7fd7e8162ac0 2024-05-28T06:17:48.809 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.809+0000 7fd7e6000640 1 -- 172.21.15.96:0/3319652543 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1571958786 0 0) 0x7fd7cc003520 con 0x7fd7e8162ac0 2024-05-28T06:17:48.809 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.809+0000 7fd7e6000640 1 -- 172.21.15.96:0/3319652543 >> v2:172.21.15.123:3300/0 conn(0x7fd7e80b4120 msgr2=0x7fd7e8163ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:48.809 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.809+0000 7fd7e6000640 1 --2- 172.21.15.96:0/3319652543 >> v2:172.21.15.123:3300/0 conn(0x7fd7e80b4120 0x7fd7e8163ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:48.809 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.809+0000 7fd7e6000640 1 -- 172.21.15.96:0/3319652543 >> v1:172.21.15.123:6789/0 conn(0x7fd7e80d0d50 legacy=0x7fd7e81633b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:48.809 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.809+0000 7fd7e6000640 1 -- 172.21.15.96:0/3319652543 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7e80b3640 con 0x7fd7e8162ac0 2024-05-28T06:17:48.810 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.809+0000 7fd7e6000640 1 -- 172.21.15.96:0/3319652543 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7fd7d80019f0 con 0x7fd7e8162ac0 2024-05-28T06:17:48.810 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.809+0000 7fd7e6000640 1 -- 172.21.15.96:0/3319652543 >> v1:172.21.15.96:6789/0 conn(0x7fd7e8162ac0 legacy=0x7fd7e80d0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:48.810 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.809+0000 7fd7e6000640 1 --2- 172.21.15.96:0/3319652543 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd7cc0039b0 0x7fd7cc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:48.810 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.809+0000 7fd7e6000640 1 --2- 172.21.15.96:0/3319652543 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd7cc004800 0x7fd7cc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:48.810 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.809+0000 7fd7e6000640 1 --2- 172.21.15.96:0/3319652543 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7cc0051c0 0x7fd7cc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:48.810 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.809+0000 7fd7e6000640 1 -- 172.21.15.96:0/3319652543 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd7e80b3c90 con 0x7fd7cc004800 2024-05-28T06:17:48.810 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.809+0000 7fd7e6000640 1 -- 172.21.15.96:0/3319652543 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd7cc00c060 con 0x7fd7cc0051c0 2024-05-28T06:17:48.810 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.809+0000 7fd7e6000640 1 -- 172.21.15.96:0/3319652543 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fd7cc00c260 con 0x7fd7cc0039b0 2024-05-28T06:17:48.810 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.810+0000 7fd7de000640 1 --2- 172.21.15.96:0/3319652543 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd7cc004800 0x7fd7cc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:48.810 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.810+0000 7fd7e7400640 1 --2- 172.21.15.96:0/3319652543 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7cc0051c0 0x7fd7cc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:48.810 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.810+0000 7fd7de000640 1 -- 172.21.15.96:0/3319652543 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd7cc0039b0 msgr2=0x7fd7cc003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:48.810 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.810+0000 7fd7de000640 1 --2- 172.21.15.96:0/3319652543 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd7cc0039b0 0x7fd7cc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:48.810 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.810+0000 7fd7de000640 1 -- 172.21.15.96:0/3319652543 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7cc0051c0 msgr2=0x7fd7cc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:48.811 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.810+0000 7fd7de000640 1 --2- 172.21.15.96:0/3319652543 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7cc0051c0 0x7fd7cc00b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:48.811 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.810+0000 7fd7de000640 1 -- 172.21.15.96:0/3319652543 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7cc003520 con 0x7fd7cc004800 2024-05-28T06:17:48.811 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.810+0000 7fd7e7400640 1 -- 172.21.15.96:0/3319652543 reap_dead start 2024-05-28T06:17:48.812 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.811+0000 7fd7de000640 1 --2- 172.21.15.96:0/3319652543 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd7cc004800 0x7fd7cc004c80 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7fd7d0001750 tx=0x7fd7cc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a085dc6ae67ee821 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:48.812 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.811+0000 7fd7e6000640 1 -- 172.21.15.96:0/3319652543 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7d0001370 con 0x7fd7cc004800 2024-05-28T06:17:48.812 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.811+0000 7fd7e6000640 1 -- 172.21.15.96:0/3319652543 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7d00052d0 con 0x7fd7cc004800 2024-05-28T06:17:48.812 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.811+0000 7fd7e6000640 1 -- 172.21.15.96:0/3319652543 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7d002abc0 con 0x7fd7cc004800 2024-05-28T06:17:48.812 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.812+0000 7fd7edc00640 1 -- 172.21.15.96:0/3319652543 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd7cc004800 msgr2=0x7fd7cc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:48.812 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.812+0000 7fd7edc00640 1 --2- 172.21.15.96:0/3319652543 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd7cc004800 0x7fd7cc004c80 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7fd7d0001750 tx=0x7fd7cc005830 comp rx=0 tx=0).stop 2024-05-28T06:17:48.812 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.812+0000 7fd7edc00640 1 -- 172.21.15.96:0/3319652543 shutdown_connections 2024-05-28T06:17:48.812 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.812+0000 7fd7edc00640 1 --2- 172.21.15.96:0/3319652543 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd7cc004800 0x7fd7cc004c80 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:48.812 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.812+0000 7fd7edc00640 1 -- 172.21.15.96:0/3319652543 >> 172.21.15.96:0/3319652543 conn(0x7fd7e806bd20 msgr2=0x7fd7e806c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:48.812 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.812+0000 7fd7edc00640 1 -- 172.21.15.96:0/3319652543 shutdown_connections 2024-05-28T06:17:48.812 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.812+0000 7fd7edc00640 1 -- 172.21.15.96:0/3319652543 wait complete. 2024-05-28T06:17:48.814 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.814+0000 7fd7edc00640 1 Processor -- start 2024-05-28T06:17:48.814 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.814+0000 7fd7edc00640 1 -- start start 2024-05-28T06:17:48.814 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.814+0000 7fd7edc00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7cc004800 0x7fd7e80cf240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:48.814 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.814+0000 7fd7edc00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd7e81654a0 0x7fd7e8165920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:48.814 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.815+0000 7fd7edc00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd7e809cd30 0x7fd7e809d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:48.814 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.815+0000 7fd7edc00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd7e8163020 con 0x7fd7e81654a0 2024-05-28T06:17:48.814 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.815+0000 7fd7edc00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd7e8162ea0 con 0x7fd7cc004800 2024-05-28T06:17:48.814 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.815+0000 7fd7edc00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fd7e80d0640 con 0x7fd7e809cd30 2024-05-28T06:17:48.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.815+0000 7fd7de000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd7e81654a0 0x7fd7e8165920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:48.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.815+0000 7fd7de000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd7e81654a0 0x7fd7e8165920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:60946/0 (socket says 172.21.15.96:60946) 2024-05-28T06:17:48.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.815+0000 7fd7de000640 1 -- 172.21.15.96:0/658536839 learned_addr learned my addr 172.21.15.96:0/658536839 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:17:48.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.815+0000 7fd7de000640 1 -- 172.21.15.96:0/658536839 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd7e809cd30 msgr2=0x7fd7e809d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:48.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.815+0000 7fd7de000640 1 --2- 172.21.15.96:0/658536839 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd7e809cd30 0x7fd7e809d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:48.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.815+0000 7fd7de000640 1 -- 172.21.15.96:0/658536839 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7cc004800 msgr2=0x7fd7e80cf240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:48.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.815+0000 7fd7e6a00640 1 --2- 172.21.15.96:0/658536839 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7cc004800 0x7fd7e80cf240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:48.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.815+0000 7fd7de000640 1 --2- 172.21.15.96:0/658536839 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7cc004800 0x7fd7e80cf240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:48.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.815+0000 7fd7de000640 1 -- 172.21.15.96:0/658536839 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7e809dad0 con 0x7fd7e81654a0 2024-05-28T06:17:48.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.815+0000 7fd7e6a00640 1 --2- 172.21.15.96:0/658536839 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7cc004800 0x7fd7e80cf240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:17:48.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.816+0000 7fd7de000640 1 --2- 172.21.15.96:0/658536839 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd7e81654a0 0x7fd7e8165920 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7fd7d0001870 tx=0x7fd7d0001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:48.816 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.816+0000 7fd7df400640 1 -- 172.21.15.96:0/658536839 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7d003f030 con 0x7fd7e81654a0 2024-05-28T06:17:48.816 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.816+0000 7fd7df400640 1 -- 172.21.15.96:0/658536839 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7d0005480 con 0x7fd7e81654a0 2024-05-28T06:17:48.816 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.816+0000 7fd7edc00640 1 -- 172.21.15.96:0/658536839 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7e809ddb0 con 0x7fd7e81654a0 2024-05-28T06:17:48.816 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.816+0000 7fd7df400640 1 -- 172.21.15.96:0/658536839 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7d0037040 con 0x7fd7e81654a0 2024-05-28T06:17:48.816 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.816+0000 7fd7edc00640 1 -- 172.21.15.96:0/658536839 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7e80cce10 con 0x7fd7e81654a0 2024-05-28T06:17:48.816 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.817+0000 7fd7edc00640 1 -- 172.21.15.96:0/658536839 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7e8165e60 con 0x7fd7e81654a0 2024-05-28T06:17:48.817 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.818+0000 7fd7df400640 1 -- 172.21.15.96:0/658536839 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 29) ==== 92672+0+0 (secure 0 0 0) 0x7fd7d0042070 con 0x7fd7e81654a0 2024-05-28T06:17:48.817 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.819+0000 7fd7df400640 1 --2- 172.21.15.96:0/658536839 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd7c406daf0 0x7fd7c406ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:48.817 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.819+0000 7fd7df400640 1 -- 172.21.15.96:0/658536839 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 5851+0+0 (secure 0 0 0) 0x7fd7d0007640 con 0x7fd7e81654a0 2024-05-28T06:17:48.818 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.819+0000 7fd7e6a00640 1 --2- 172.21.15.96:0/658536839 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd7c406daf0 0x7fd7c406ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:48.819 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.821+0000 7fd7e6a00640 1 --2- 172.21.15.96:0/658536839 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd7c406daf0 0x7fd7c406ffb0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fd7e8069c40 tx=0x7fd7d8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:48.821 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:48.823+0000 7fd7df400640 1 -- 172.21.15.96:0/658536839 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd7d0079cb0 con 0x7fd7e81654a0 2024-05-28T06:17:49.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.014+0000 7fd7edc00640 1 -- 172.21.15.96:0/658536839 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd7e806c310 con 0x7fd7e81654a0 2024-05-28T06:17:49.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.015+0000 7fd7df400640 1 -- 172.21.15.96:0/658536839 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v77) ==== 93+0+10 (secure 0 0 0) 0x7fd7d00795b0 con 0x7fd7e81654a0 2024-05-28T06:17:49.014 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 1 2024-05-28T06:17:49.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.017+0000 7fd7edc00640 1 -- 172.21.15.96:0/658536839 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd7c406daf0 msgr2=0x7fd7c406ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:49.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.017+0000 7fd7edc00640 1 --2- 172.21.15.96:0/658536839 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd7c406daf0 0x7fd7c406ffb0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fd7e8069c40 tx=0x7fd7d8009040 comp rx=0 tx=0).stop 2024-05-28T06:17:49.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.017+0000 7fd7edc00640 1 -- 172.21.15.96:0/658536839 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd7e81654a0 msgr2=0x7fd7e8165920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:49.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.017+0000 7fd7edc00640 1 --2- 172.21.15.96:0/658536839 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd7e81654a0 0x7fd7e8165920 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7fd7d0001870 tx=0x7fd7d0001120 comp rx=0 tx=0).stop 2024-05-28T06:17:49.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.017+0000 7fd7edc00640 1 -- 172.21.15.96:0/658536839 shutdown_connections 2024-05-28T06:17:49.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.017+0000 7fd7edc00640 1 --2- 172.21.15.96:0/658536839 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd7c406daf0 0x7fd7c406ffb0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:49.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.017+0000 7fd7edc00640 1 --2- 172.21.15.96:0/658536839 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd7e809cd30 0x7fd7e809d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:49.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.017+0000 7fd7edc00640 1 --2- 172.21.15.96:0/658536839 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd7e81654a0 0x7fd7e8165920 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:49.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.017+0000 7fd7edc00640 1 --2- 172.21.15.96:0/658536839 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7cc004800 0x7fd7e80cf240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:49.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.018+0000 7fd7edc00640 1 -- 172.21.15.96:0/658536839 >> 172.21.15.96:0/658536839 conn(0x7fd7e806bd20 msgr2=0x7fd7e80b11b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:49.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.018+0000 7fd7edc00640 1 -- 172.21.15.96:0/658536839 shutdown_connections 2024-05-28T06:17:49.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.018+0000 7fd7edc00640 1 -- 172.21.15.96:0/658536839 wait complete. 2024-05-28T06:17:49.033 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-28T06:17:49.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.226+0000 7f5947200640 1 Processor -- start 2024-05-28T06:17:49.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.226+0000 7f5947200640 1 -- start start 2024-05-28T06:17:49.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.228+0000 7f5947200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f594015e870 0x7f59400d0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:49.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.228+0000 7f5947200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f59400d14d0 0x7f59400cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:49.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.228+0000 7f5947200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f59400cf870 0x7f5940162360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:49.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.228+0000 7f5947200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f59400d2150 con 0x7f59400d14d0 2024-05-28T06:17:49.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.228+0000 7f5947200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f59400cfd00 con 0x7f59400cf870 2024-05-28T06:17:49.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.228+0000 7f5947200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f594015ec50 con 0x7f594015e870 2024-05-28T06:17:49.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.228+0000 7f5945400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f59400d14d0 0x7f59400cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:49.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.228+0000 7f5945400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f59400d14d0 0x7f59400cf330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:60958/0 (socket says 172.21.15.96:60958) 2024-05-28T06:17:49.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.228+0000 7f5945400640 1 -- 172.21.15.96:0/1337199239 learned_addr learned my addr 172.21.15.96:0/1337199239 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:17:49.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.229+0000 7f5945400640 1 -- 172.21.15.96:0/1337199239 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f594015e870 msgr2=0x7f59400d0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:49.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.229+0000 7f5945400640 1 --2- 172.21.15.96:0/1337199239 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f594015e870 0x7f59400d0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:49.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.229+0000 7f5945400640 1 -- 172.21.15.96:0/1337199239 >> v2:172.21.15.123:3300/0 conn(0x7f59400cf870 msgr2=0x7f5940162360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:49.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.229+0000 7f5945400640 1 --2- 172.21.15.96:0/1337199239 >> v2:172.21.15.123:3300/0 conn(0x7f59400cf870 0x7f5940162360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:49.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.229+0000 7f5945400640 1 -- 172.21.15.96:0/1337199239 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59400cff80 con 0x7f59400d14d0 2024-05-28T06:17:49.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.230+0000 7f5945400640 1 --2- 172.21.15.96:0/1337199239 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f59400d14d0 0x7f59400cf330 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f59400b3f60 tx=0x7f593002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62e65b0cf72fa673 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:49.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.230+0000 7f5944a00640 1 -- 172.21.15.96:0/1337199239 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5930037070 con 0x7f59400d14d0 2024-05-28T06:17:49.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.230+0000 7f5944a00640 1 -- 172.21.15.96:0/1337199239 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5930032440 con 0x7f59400d14d0 2024-05-28T06:17:49.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.230+0000 7f5944a00640 1 -- 172.21.15.96:0/1337199239 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5930032810 con 0x7f59400d14d0 2024-05-28T06:17:49.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.230+0000 7f5947200640 1 -- 172.21.15.96:0/1337199239 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f59400d14d0 msgr2=0x7f59400cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:49.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.230+0000 7f5947200640 1 --2- 172.21.15.96:0/1337199239 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f59400d14d0 0x7f59400cf330 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f59400b3f60 tx=0x7f593002a1e0 comp rx=0 tx=0).stop 2024-05-28T06:17:49.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.230+0000 7f5947200640 1 -- 172.21.15.96:0/1337199239 shutdown_connections 2024-05-28T06:17:49.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.230+0000 7f5947200640 1 --2- 172.21.15.96:0/1337199239 >> v2:172.21.15.123:3300/0 conn(0x7f59400cf870 0x7f5940162360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:49.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.230+0000 7f5947200640 1 --2- 172.21.15.96:0/1337199239 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f59400d14d0 0x7f59400cf330 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:49.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.230+0000 7f5947200640 1 --2- 172.21.15.96:0/1337199239 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f594015e870 0x7f59400d0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:49.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.230+0000 7f5947200640 1 -- 172.21.15.96:0/1337199239 >> 172.21.15.96:0/1337199239 conn(0x7f594008bfc0 msgr2=0x7f59400d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:49.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.231+0000 7f5947200640 1 -- 172.21.15.96:0/1337199239 shutdown_connections 2024-05-28T06:17:49.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.231+0000 7f5947200640 1 -- 172.21.15.96:0/1337199239 wait complete. 2024-05-28T06:17:49.231 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.232+0000 7f5947200640 1 Processor -- start 2024-05-28T06:17:49.231 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.232+0000 7f5947200640 1 -- start start 2024-05-28T06:17:49.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.233+0000 7f5947200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f59400cf870 0x7f59401593f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:49.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.233+0000 7f5947200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f59400d14d0 0x7f5940160c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:49.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.233+0000 7f5947200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f594015e870 0x7f5940161140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:49.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.233+0000 7f5947200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f59400d0820 con 0x7f59400d14d0 2024-05-28T06:17:49.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.233+0000 7f5947200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f59400d0540 con 0x7f59400cf870 2024-05-28T06:17:49.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.233+0000 7f5947200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f59400d4fb0 con 0x7f594015e870 2024-05-28T06:17:49.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.233+0000 7f5945400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f59400d14d0 0x7f5940160c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:49.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.233+0000 7f5945400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f59400d14d0 0x7f5940160c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:60974/0 (socket says 172.21.15.96:60974) 2024-05-28T06:17:49.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.233+0000 7f5945400640 1 -- 172.21.15.96:0/2673775485 learned_addr learned my addr 172.21.15.96:0/2673775485 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:17:49.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.234+0000 7f5945400640 1 -- 172.21.15.96:0/2673775485 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f594015e870 msgr2=0x7f5940161140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:49.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.234+0000 7f5945400640 1 --2- 172.21.15.96:0/2673775485 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f594015e870 0x7f5940161140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:49.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.234+0000 7f5945400640 1 -- 172.21.15.96:0/2673775485 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f59400cf870 msgr2=0x7f59401593f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:49.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.234+0000 7f5945400640 1 --2- 172.21.15.96:0/2673775485 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f59400cf870 0x7f59401593f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:49.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.234+0000 7f5945400640 1 -- 172.21.15.96:0/2673775485 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59400ccec0 con 0x7f59400d14d0 2024-05-28T06:17:49.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.234+0000 7f5945400640 1 --2- 172.21.15.96:0/2673775485 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f59400d14d0 0x7f5940160c00 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f5930000c00 tx=0x7f59300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:49.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.234+0000 7f5936000640 1 -- 172.21.15.96:0/2673775485 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5930041070 con 0x7f59400d14d0 2024-05-28T06:17:49.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.234+0000 7f5936000640 1 -- 172.21.15.96:0/2673775485 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f593003c870 con 0x7f59400d14d0 2024-05-28T06:17:49.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.234+0000 7f5936000640 1 -- 172.21.15.96:0/2673775485 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5930037040 con 0x7f59400d14d0 2024-05-28T06:17:49.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.234+0000 7f5947200640 1 -- 172.21.15.96:0/2673775485 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59400cd1a0 con 0x7f59400d14d0 2024-05-28T06:17:49.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.234+0000 7f5947200640 1 -- 172.21.15.96:0/2673775485 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59400b7870 con 0x7f59400d14d0 2024-05-28T06:17:49.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.235+0000 7f5947200640 1 -- 172.21.15.96:0/2673775485 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59400689e0 con 0x7f59400d14d0 2024-05-28T06:17:49.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.236+0000 7f5936000640 1 -- 172.21.15.96:0/2673775485 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 29) ==== 92672+0+0 (secure 0 0 0) 0x7f5930044070 con 0x7f59400d14d0 2024-05-28T06:17:49.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.237+0000 7f5936000640 1 --2- 172.21.15.96:0/2673775485 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f592406daf0 0x7f592406ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:49.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.237+0000 7f5936000640 1 -- 172.21.15.96:0/2673775485 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 5851+0+0 (secure 0 0 0) 0x7f59300ab410 con 0x7f59400d14d0 2024-05-28T06:17:49.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.238+0000 7f5945e00640 1 --2- 172.21.15.96:0/2673775485 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f592406daf0 0x7f592406ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:49.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.239+0000 7f5945e00640 1 --2- 172.21.15.96:0/2673775485 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f592406daf0 0x7f592406ffb0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f593800ab90 tx=0x7f5938009250 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:49.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.242+0000 7f5936000640 1 -- 172.21.15.96:0/2673775485 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f593007b410 con 0x7f59400d14d0 2024-05-28T06:17:49.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.571+0000 7f5947200640 1 -- 172.21.15.96:0/2673775485 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f59400cf330 con 0x7f59400d14d0 2024-05-28T06:17:49.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.571+0000 7f5936000640 1 -- 172.21.15.96:0/2673775485 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v77) ==== 104+0+11 (secure 0 0 0) 0x7f593007ad10 con 0x7f59400d14d0 2024-05-28T06:17:49.570 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 32 2024-05-28T06:17:49.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.574+0000 7f5947200640 1 -- 172.21.15.96:0/2673775485 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f592406daf0 msgr2=0x7f592406ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:49.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.574+0000 7f5947200640 1 --2- 172.21.15.96:0/2673775485 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f592406daf0 0x7f592406ffb0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f593800ab90 tx=0x7f5938009250 comp rx=0 tx=0).stop 2024-05-28T06:17:49.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.574+0000 7f5947200640 1 -- 172.21.15.96:0/2673775485 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f59400d14d0 msgr2=0x7f5940160c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:49.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.574+0000 7f5947200640 1 --2- 172.21.15.96:0/2673775485 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f59400d14d0 0x7f5940160c00 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f5930000c00 tx=0x7f59300046c0 comp rx=0 tx=0).stop 2024-05-28T06:17:49.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.574+0000 7f5947200640 1 -- 172.21.15.96:0/2673775485 shutdown_connections 2024-05-28T06:17:49.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.574+0000 7f5947200640 1 --2- 172.21.15.96:0/2673775485 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f592406daf0 0x7f592406ffb0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:49.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.574+0000 7f5947200640 1 --2- 172.21.15.96:0/2673775485 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f594015e870 0x7f5940161140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:49.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.574+0000 7f5947200640 1 --2- 172.21.15.96:0/2673775485 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f59400d14d0 0x7f5940160c00 unknown :-1 s=CLOSED pgs=1304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:49.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.574+0000 7f5947200640 1 --2- 172.21.15.96:0/2673775485 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f59400cf870 0x7f59401593f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:49.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.574+0000 7f5947200640 1 -- 172.21.15.96:0/2673775485 >> 172.21.15.96:0/2673775485 conn(0x7f594008bfc0 msgr2=0x7f59400b1300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:49.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.574+0000 7f5947200640 1 -- 172.21.15.96:0/2673775485 shutdown_connections 2024-05-28T06:17:49.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.574+0000 7f5947200640 1 -- 172.21.15.96:0/2673775485 wait complete. 2024-05-28T06:17:49.601 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-28T06:17:49.855 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.856+0000 7f79fb800640 1 Processor -- start 2024-05-28T06:17:49.857 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.857+0000 7f79fb800640 1 -- start start 2024-05-28T06:17:49.857 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.858+0000 7f79fb800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f79f415ed80 0x7f79f40cff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:49.857 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.858+0000 7f79fb800640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79f415f160 con 0x7f79f40b7e50 2024-05-28T06:17:49.858 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.858+0000 7f79fb800640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79f4162930 con 0x7f79f40b68f0 2024-05-28T06:17:49.858 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.858+0000 7f79fb800640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f79f4163540 con 0x7f79f415ed80 2024-05-28T06:17:49.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.859+0000 7f79f3e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f79f40b7e50 0x7f79f40b61e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:59864/0 (socket says 172.21.15.96:59864) 2024-05-28T06:17:49.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.859+0000 7f79f3e00640 1 -- 172.21.15.96:0/3544923060 learned_addr learned my addr 172.21.15.96:0/3544923060 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:17:49.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.860+0000 7f79f8e00640 1 --2- 172.21.15.96:0/3544923060 >> v2:172.21.15.123:3300/0 conn(0x7f79f415ed80 0x7f79f40cff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:49.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.861+0000 7f79f3400640 1 -- 172.21.15.96:0/3544923060 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 607789881 0 0) 0x7f79f415f160 con 0x7f79f40b7e50 2024-05-28T06:17:49.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.861+0000 7f79f3400640 1 -- 172.21.15.96:0/3544923060 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f79dc003520 con 0x7f79f40b7e50 2024-05-28T06:17:49.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.861+0000 7f79f3400640 1 -- 172.21.15.96:0/3544923060 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2569233679 0 0) 0x7f79f4162930 con 0x7f79f40b68f0 2024-05-28T06:17:49.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.861+0000 7f79f3400640 1 -- 172.21.15.96:0/3544923060 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f79f415f160 con 0x7f79f40b68f0 2024-05-28T06:17:49.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.861+0000 7f79f3400640 1 -- 172.21.15.96:0/3544923060 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3628504584 0 0) 0x7f79dc003520 con 0x7f79f40b7e50 2024-05-28T06:17:49.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.861+0000 7f79f3400640 1 -- 172.21.15.96:0/3544923060 >> v2:172.21.15.123:3300/0 conn(0x7f79f415ed80 msgr2=0x7f79f40cff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:49.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.861+0000 7f79f3400640 1 --2- 172.21.15.96:0/3544923060 >> v2:172.21.15.123:3300/0 conn(0x7f79f415ed80 0x7f79f40cff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:49.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.861+0000 7f79f3400640 1 -- 172.21.15.96:0/3544923060 >> v1:172.21.15.123:6789/0 conn(0x7f79f40b68f0 legacy=0x7f79f40677b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:49.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.861+0000 7f79f3400640 1 -- 172.21.15.96:0/3544923060 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79f4162c10 con 0x7f79f40b7e50 2024-05-28T06:17:49.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.861+0000 7f79f3400640 1 -- 172.21.15.96:0/3544923060 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f79e0001670 con 0x7f79f40b7e50 2024-05-28T06:17:49.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.862+0000 7f79f3400640 1 -- 172.21.15.96:0/3544923060 >> v1:172.21.15.96:6789/0 conn(0x7f79f40b7e50 legacy=0x7f79f40b61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:49.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.862+0000 7f79f3400640 1 --2- 172.21.15.96:0/3544923060 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f79dc004ba0 0x7f79dc004fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:49.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.862+0000 7f79f3400640 1 --2- 172.21.15.96:0/3544923060 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f79dc0037a0 0x7f79dc003c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:49.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.862+0000 7f79f3400640 1 --2- 172.21.15.96:0/3544923060 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f79dc004160 0x7f79dc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:49.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.862+0000 7f79f3400640 1 -- 172.21.15.96:0/3544923060 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f79f4163540 con 0x7f79dc004ba0 2024-05-28T06:17:49.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.862+0000 7f79f3400640 1 -- 172.21.15.96:0/3544923060 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f79dc00c040 con 0x7f79dc0037a0 2024-05-28T06:17:49.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.862+0000 7f79f3400640 1 -- 172.21.15.96:0/3544923060 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f79dc00c240 con 0x7f79dc004160 2024-05-28T06:17:49.867 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.862+0000 7f79f9800640 1 --2- 172.21.15.96:0/3544923060 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f79dc004160 0x7f79dc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:49.867 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.862+0000 7f79f3e00640 1 --2- 172.21.15.96:0/3544923060 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f79dc0037a0 0x7f79dc003c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:49.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.863+0000 7f79f9800640 1 -- 172.21.15.96:0/3544923060 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f79dc0037a0 msgr2=0x7f79dc003c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:49.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.863+0000 7f79f9800640 1 --2- 172.21.15.96:0/3544923060 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f79dc0037a0 0x7f79dc003c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:49.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.863+0000 7f79f9800640 1 -- 172.21.15.96:0/3544923060 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f79dc004ba0 msgr2=0x7f79dc004fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:49.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.863+0000 7f79f9800640 1 --2- 172.21.15.96:0/3544923060 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f79dc004ba0 0x7f79dc004fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:49.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.863+0000 7f79f9800640 1 -- 172.21.15.96:0/3544923060 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79f4162c10 con 0x7f79dc004160 2024-05-28T06:17:49.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.864+0000 7f79f9800640 1 -- 172.21.15.96:0/3544923060 reap_dead start 2024-05-28T06:17:49.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.864+0000 7f79f9800640 1 --2- 172.21.15.96:0/3544923060 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f79dc004160 0x7f79dc00b9c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f79ec001fa0 tx=0x7f79dc005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c55470d5403d7b6 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:49.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.865+0000 7f79f3400640 1 -- 172.21.15.96:0/3544923060 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79ec001bf0 con 0x7f79dc004160 2024-05-28T06:17:49.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.865+0000 7f79f3400640 1 -- 172.21.15.96:0/3544923060 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79ec003b70 con 0x7f79dc004160 2024-05-28T06:17:49.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.865+0000 7f79f3400640 1 -- 172.21.15.96:0/3544923060 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79ec0050d0 con 0x7f79dc004160 2024-05-28T06:17:49.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.866+0000 7f79fb800640 1 -- 172.21.15.96:0/3544923060 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f79dc004160 msgr2=0x7f79dc00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:49.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.866+0000 7f79fb800640 1 --2- 172.21.15.96:0/3544923060 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f79dc004160 0x7f79dc00b9c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f79ec001fa0 tx=0x7f79dc005830 comp rx=0 tx=0).stop 2024-05-28T06:17:49.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.866+0000 7f79fb800640 1 -- 172.21.15.96:0/3544923060 shutdown_connections 2024-05-28T06:17:49.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.866+0000 7f79fb800640 1 --2- 172.21.15.96:0/3544923060 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f79dc004160 0x7f79dc00b9c0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:49.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.866+0000 7f79fb800640 1 -- 172.21.15.96:0/3544923060 >> 172.21.15.96:0/3544923060 conn(0x7f79f40d13c0 msgr2=0x7f79f40d17d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:49.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.867+0000 7f79fb800640 1 -- 172.21.15.96:0/3544923060 shutdown_connections 2024-05-28T06:17:49.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.867+0000 7f79fb800640 1 -- 172.21.15.96:0/3544923060 wait complete. 2024-05-28T06:17:49.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.869+0000 7f79fb800640 1 Processor -- start 2024-05-28T06:17:49.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.869+0000 7f79fb800640 1 -- start start 2024-05-28T06:17:49.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.870+0000 7f79fb800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f79dc004160 0x7f79f40d3730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:49.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.870+0000 7f79fb800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f79f4161990 0x7f79f4161e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:49.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.870+0000 7f79fb800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f79f414f7a0 0x7f79f414fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:49.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.870+0000 7f79fb800640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f79f40cff80 con 0x7f79f414f7a0 2024-05-28T06:17:49.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.870+0000 7f79fb800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f79f41631d0 con 0x7f79dc004160 2024-05-28T06:17:49.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.870+0000 7f79fb800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f79f40d0100 con 0x7f79f4161990 2024-05-28T06:17:49.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.870+0000 7f79f8e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f79dc004160 0x7f79f40d3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:49.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.871+0000 7f79f3e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f79f4161990 0x7f79f4161e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:49.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.871+0000 7f79f8e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f79dc004160 0x7f79f40d3730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.96:52908/0 (socket says 172.21.15.96:52908) 2024-05-28T06:17:49.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.871+0000 7f79f8e00640 1 -- 172.21.15.96:0/3524207799 learned_addr learned my addr 172.21.15.96:0/3524207799 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:17:49.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.871+0000 7f79f9800640 1 --2- 172.21.15.96:0/3524207799 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f79f414f7a0 0x7f79f414fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:49.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.871+0000 7f79f8e00640 1 -- 172.21.15.96:0/3524207799 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f79f4161990 msgr2=0x7f79f4161e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:49.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.871+0000 7f79f8e00640 1 --2- 172.21.15.96:0/3524207799 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f79f4161990 0x7f79f4161e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:49.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.871+0000 7f79f8e00640 1 -- 172.21.15.96:0/3524207799 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f79f414f7a0 msgr2=0x7f79f414fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:49.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.872+0000 7f79f8e00640 1 --2- 172.21.15.96:0/3524207799 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f79f414f7a0 0x7f79f414fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:49.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.872+0000 7f79f8e00640 1 -- 172.21.15.96:0/3524207799 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79f4150170 con 0x7f79dc004160 2024-05-28T06:17:49.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.872+0000 7f79f9800640 1 --2- 172.21.15.96:0/3524207799 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f79f414f7a0 0x7f79f414fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-28T06:17:49.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.875+0000 7f79f8e00640 1 --2- 172.21.15.96:0/3524207799 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f79dc004160 0x7f79f40d3730 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f79e800e7a0 tx=0x7f79e800ec60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:49.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.876+0000 7f79f0c00640 1 -- 172.21.15.96:0/3524207799 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79e8017070 con 0x7f79dc004160 2024-05-28T06:17:49.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.876+0000 7f79f0c00640 1 -- 172.21.15.96:0/3524207799 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79e800c940 con 0x7f79dc004160 2024-05-28T06:17:49.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.876+0000 7f79f0c00640 1 -- 172.21.15.96:0/3524207799 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79e800cd10 con 0x7f79dc004160 2024-05-28T06:17:49.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.876+0000 7f79fb800640 1 -- 172.21.15.96:0/3524207799 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79f40ca090 con 0x7f79dc004160 2024-05-28T06:17:49.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.876+0000 7f79fb800640 1 -- 172.21.15.96:0/3524207799 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79f40ca650 con 0x7f79dc004160 2024-05-28T06:17:49.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.878+0000 7f79fb800640 1 -- 172.21.15.96:0/3524207799 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79f40689e0 con 0x7f79dc004160 2024-05-28T06:17:49.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.880+0000 7f79f0c00640 1 -- 172.21.15.96:0/3524207799 <== mon.1 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 29) ==== 92672+0+0 (secure 0 0 0) 0x7f79e801a070 con 0x7f79dc004160 2024-05-28T06:17:49.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.881+0000 7f79f0c00640 1 --2- 172.21.15.96:0/3524207799 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f79d406db40 0x7f79d4070000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:49.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.882+0000 7f79f0c00640 1 -- 172.21.15.96:0/3524207799 <== mon.1 v2:172.21.15.123:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 5851+0+0 (secure 0 0 0) 0x7f79e8057e40 con 0x7f79dc004160 2024-05-28T06:17:49.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.882+0000 7f79f3e00640 1 --2- 172.21.15.96:0/3524207799 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f79d406db40 0x7f79d4070000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:49.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.886+0000 7f79f3e00640 1 --2- 172.21.15.96:0/3524207799 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f79d406db40 0x7f79d4070000 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f79e0000fb0 tx=0x7f79e0007040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:49.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:49.892+0000 7f79f0c00640 1 -- 172.21.15.96:0/3524207799 <== mon.1 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f79e8059da0 con 0x7f79dc004160 2024-05-28T06:17:50.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.169+0000 7f79fb800640 1 -- 172.21.15.96:0/3524207799 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f79f40d2910 con 0x7f79dc004160 2024-05-28T06:17:50.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.170+0000 7f79f0c00640 1 -- 172.21.15.96:0/3524207799 <== mon.1 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v77) ==== 100+0+11 (secure 0 0 0) 0x7f79e805e0a0 con 0x7f79dc004160 2024-05-28T06:17:50.169 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 64 2024-05-28T06:17:50.171 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.172+0000 7f79fb800640 1 -- 172.21.15.96:0/3524207799 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f79d406db40 msgr2=0x7f79d4070000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:50.171 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.172+0000 7f79fb800640 1 --2- 172.21.15.96:0/3524207799 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f79d406db40 0x7f79d4070000 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f79e0000fb0 tx=0x7f79e0007040 comp rx=0 tx=0).stop 2024-05-28T06:17:50.171 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.172+0000 7f79fb800640 1 -- 172.21.15.96:0/3524207799 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f79dc004160 msgr2=0x7f79f40d3730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:50.171 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.172+0000 7f79fb800640 1 --2- 172.21.15.96:0/3524207799 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f79dc004160 0x7f79f40d3730 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f79e800e7a0 tx=0x7f79e800ec60 comp rx=0 tx=0).stop 2024-05-28T06:17:50.171 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.172+0000 7f79fb800640 1 -- 172.21.15.96:0/3524207799 shutdown_connections 2024-05-28T06:17:50.171 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.172+0000 7f79fb800640 1 --2- 172.21.15.96:0/3524207799 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f79d406db40 0x7f79d4070000 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:50.172 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.172+0000 7f79fb800640 1 --2- 172.21.15.96:0/3524207799 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f79f414f7a0 0x7f79f414fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:50.172 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.172+0000 7f79fb800640 1 --2- 172.21.15.96:0/3524207799 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f79f4161990 0x7f79f4161e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:50.172 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.172+0000 7f79fb800640 1 --2- 172.21.15.96:0/3524207799 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f79dc004160 0x7f79f40d3730 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:50.172 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.172+0000 7f79fb800640 1 -- 172.21.15.96:0/3524207799 >> 172.21.15.96:0/3524207799 conn(0x7f79f40d13c0 msgr2=0x7f79f40ba100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:50.172 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.172+0000 7f79fb800640 1 -- 172.21.15.96:0/3524207799 shutdown_connections 2024-05-28T06:17:50.172 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.172+0000 7f79fb800640 1 -- 172.21.15.96:0/3524207799 wait complete. 2024-05-28T06:17:50.188 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-28T06:17:50.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.400+0000 7f56ee800640 1 Processor -- start 2024-05-28T06:17:50.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.401+0000 7f56ee800640 1 -- start start 2024-05-28T06:17:50.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.402+0000 7f56ee800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f56e80d2db0 0x7f56e80b3b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:50.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.402+0000 7f56ee800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f56e815ed20 0x7f56e815f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:50.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.402+0000 7f56ee800640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56e8163500 con 0x7f56e80d4d60 2024-05-28T06:17:50.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.402+0000 7f56ee800640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f56e80d5540 con 0x7f56e815ed20 2024-05-28T06:17:50.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.402+0000 7f56ee800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f56e8162740 con 0x7f56e80d2db0 2024-05-28T06:17:50.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.403+0000 7f56e6a00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f56e80d4d60 0x7f56e80d3190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:59866/0 (socket says 172.21.15.96:59866) 2024-05-28T06:17:50.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.403+0000 7f56e7400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f56e80d2db0 0x7f56e80b3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:50.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.403+0000 7f56e6a00640 1 -- 172.21.15.96:0/1653286585 learned_addr learned my addr 172.21.15.96:0/1653286585 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:17:50.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.403+0000 7f56e7e00640 1 --2- 172.21.15.96:0/1653286585 >> v2:172.21.15.96:3300/0 conn(0x7f56e815ed20 0x7f56e815f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:50.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.403+0000 7f56e7400640 1 -- 172.21.15.96:0/1653286585 >> v2:172.21.15.96:3300/0 conn(0x7f56e815ed20 msgr2=0x7f56e815f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:50.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.403+0000 7f56e6000640 1 -- 172.21.15.96:0/1653286585 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3442072265 0 0) 0x7f56e8163500 con 0x7f56e80d4d60 2024-05-28T06:17:50.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.403+0000 7f56e7400640 1 --2- 172.21.15.96:0/1653286585 >> v2:172.21.15.96:3300/0 conn(0x7f56e815ed20 0x7f56e815f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:50.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.403+0000 7f56e7400640 1 -- 172.21.15.96:0/1653286585 >> v1:172.21.15.96:6789/0 conn(0x7f56e80d4d60 legacy=0x7f56e80d3190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:50.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.404+0000 7f56e7400640 1 -- 172.21.15.96:0/1653286585 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56e8162a20 con 0x7f56e80d2db0 2024-05-28T06:17:50.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.404+0000 7f56e7e00640 1 --2- 172.21.15.96:0/1653286585 >> v2:172.21.15.96:3300/0 conn(0x7f56e815ed20 0x7f56e815f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-28T06:17:50.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.405+0000 7f56e7400640 1 --2- 172.21.15.96:0/1653286585 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f56e80d2db0 0x7f56e80b3b50 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f56d80097a0 tx=0x7f56d802a0f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d6ad5918c2c8691 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:50.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.405+0000 7f56e6000640 1 -- 172.21.15.96:0/1653286585 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56d8037070 con 0x7f56e80d2db0 2024-05-28T06:17:50.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.405+0000 7f56e6000640 1 -- 172.21.15.96:0/1653286585 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56d802ae40 con 0x7f56e80d2db0 2024-05-28T06:17:50.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.405+0000 7f56e6000640 1 -- 172.21.15.96:0/1653286585 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56d8032660 con 0x7f56e80d2db0 2024-05-28T06:17:50.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.406+0000 7f56ee800640 1 -- 172.21.15.96:0/1653286585 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f56e80d2db0 msgr2=0x7f56e80b3b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:50.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.406+0000 7f56ee800640 1 --2- 172.21.15.96:0/1653286585 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f56e80d2db0 0x7f56e80b3b50 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f56d80097a0 tx=0x7f56d802a0f0 comp rx=0 tx=0).stop 2024-05-28T06:17:50.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.406+0000 7f56ee800640 1 -- 172.21.15.96:0/1653286585 shutdown_connections 2024-05-28T06:17:50.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.406+0000 7f56ee800640 1 --2- 172.21.15.96:0/1653286585 >> v2:172.21.15.96:3300/0 conn(0x7f56e815ed20 0x7f56e815f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:50.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.406+0000 7f56ee800640 1 --2- 172.21.15.96:0/1653286585 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f56e80d2db0 0x7f56e80b3b50 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:50.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.406+0000 7f56ee800640 1 -- 172.21.15.96:0/1653286585 >> 172.21.15.96:0/1653286585 conn(0x7f56e808bfc0 msgr2=0x7f56e80d4650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:50.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.406+0000 7f56ee800640 1 -- 172.21.15.96:0/1653286585 shutdown_connections 2024-05-28T06:17:50.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.406+0000 7f56ee800640 1 -- 172.21.15.96:0/1653286585 wait complete. 2024-05-28T06:17:50.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.408+0000 7f56ee800640 1 Processor -- start 2024-05-28T06:17:50.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.408+0000 7f56ee800640 1 -- start start 2024-05-28T06:17:50.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.410+0000 7f56ee800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f56e80d2db0 0x7f56e80c8d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:50.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.410+0000 7f56ee800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f56e80d4d60 0x7f56e80b40b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:50.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.410+0000 7f56ee800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f56e815ed20 0x7f56e815a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:50.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.410+0000 7f56ee800640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f56e80d51a0 con 0x7f56e815ed20 2024-05-28T06:17:50.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.410+0000 7f56ee800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f56e8162fe0 con 0x7f56e80d2db0 2024-05-28T06:17:50.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.410+0000 7f56ee800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f56e80d5320 con 0x7f56e80d4d60 2024-05-28T06:17:50.410 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.410+0000 7f56e7400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f56e80d2db0 0x7f56e80c8d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:50.410 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.411+0000 7f56e7400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f56e80d2db0 0x7f56e80c8d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.96:52918/0 (socket says 172.21.15.96:52918) 2024-05-28T06:17:50.410 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.411+0000 7f56e7400640 1 -- 172.21.15.96:0/2065073056 learned_addr learned my addr 172.21.15.96:0/2065073056 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:17:50.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.411+0000 7f56e6a00640 1 --2- 172.21.15.96:0/2065073056 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f56e80d4d60 0x7f56e80b40b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:50.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.411+0000 7f56e7e00640 1 --2- 172.21.15.96:0/2065073056 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f56e815ed20 0x7f56e815a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:50.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.411+0000 7f56e6a00640 1 -- 172.21.15.96:0/2065073056 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f56e80d2db0 msgr2=0x7f56e80c8d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:50.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.411+0000 7f56e6a00640 1 --2- 172.21.15.96:0/2065073056 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f56e80d2db0 0x7f56e80c8d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:50.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.411+0000 7f56e6a00640 1 -- 172.21.15.96:0/2065073056 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f56e815ed20 msgr2=0x7f56e815a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:50.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.411+0000 7f56e6a00640 1 --2- 172.21.15.96:0/2065073056 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f56e815ed20 0x7f56e815a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:50.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.412+0000 7f56e6a00640 1 -- 172.21.15.96:0/2065073056 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56e80ccfc0 con 0x7f56e80d4d60 2024-05-28T06:17:50.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.412+0000 7f56e7e00640 1 --2- 172.21.15.96:0/2065073056 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f56e815ed20 0x7f56e815a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:17:50.412 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.412+0000 7f56e6a00640 1 --2- 172.21.15.96:0/2065073056 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f56e80d4d60 0x7f56e80b40b0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f56d400c480 tx=0x7f56d400c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:50.412 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.413+0000 7f56df400640 1 -- 172.21.15.96:0/2065073056 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56d4018070 con 0x7f56e80d4d60 2024-05-28T06:17:50.412 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.413+0000 7f56df400640 1 -- 172.21.15.96:0/2065073056 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56d40074e0 con 0x7f56e80d4d60 2024-05-28T06:17:50.412 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.413+0000 7f56df400640 1 -- 172.21.15.96:0/2065073056 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56d4004a20 con 0x7f56e80d4d60 2024-05-28T06:17:50.412 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.413+0000 7f56ee800640 1 -- 172.21.15.96:0/2065073056 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56e80c9320 con 0x7f56e80d4d60 2024-05-28T06:17:50.414 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.415+0000 7f56ee800640 1 -- 172.21.15.96:0/2065073056 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f56e80c98e0 con 0x7f56e80d4d60 2024-05-28T06:17:50.415 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.417+0000 7f56df400640 1 -- 172.21.15.96:0/2065073056 <== mon.2 v2:172.21.15.123:3301/0 4 ==== mgrmap(e 29) ==== 92672+0+0 (secure 0 0 0) 0x7f56d401b070 con 0x7f56e80d4d60 2024-05-28T06:17:50.417 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.418+0000 7f56ee800640 1 -- 172.21.15.96:0/2065073056 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56e80689e0 con 0x7f56e80d4d60 2024-05-28T06:17:50.418 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.419+0000 7f56df400640 1 --2- 172.21.15.96:0/2065073056 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f56c806dbc0 0x7f56c8070080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:50.418 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.419+0000 7f56df400640 1 -- 172.21.15.96:0/2065073056 <== mon.2 v2:172.21.15.123:3301/0 5 ==== osd_map(77..77 src has 1..77) ==== 5851+0+0 (secure 0 0 0) 0x7f56d408a060 con 0x7f56e80d4d60 2024-05-28T06:17:50.419 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.421+0000 7f56e7400640 1 --2- 172.21.15.96:0/2065073056 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f56c806dbc0 0x7f56c8070080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:50.420 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.422+0000 7f56e7400640 1 --2- 172.21.15.96:0/2065073056 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f56c806dbc0 0x7f56c8070080 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f56d80097a0 tx=0x7f56d802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:50.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.430+0000 7f56df400640 1 -- 172.21.15.96:0/2065073056 <== mon.2 v2:172.21.15.123:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f56d4059b70 con 0x7f56e80d4d60 2024-05-28T06:17:50.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.739+0000 7f56ee800640 1 -- 172.21.15.96:0/2065073056 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f56e80684a0 con 0x7f56e80d4d60 2024-05-28T06:17:50.739 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.740+0000 7f56df400640 1 -- 172.21.15.96:0/2065073056 <== mon.2 v2:172.21.15.123:3301/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v77) ==== 80+0+0 (secure 0 0 0) 0x7f56d4059470 con 0x7f56e80d4d60 2024-05-28T06:17:50.739 INFO:teuthology.orchestra.run.smithi096.stderr:listed 0 entries 2024-05-28T06:17:50.741 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.742+0000 7f56ee800640 1 -- 172.21.15.96:0/2065073056 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f56c806dbc0 msgr2=0x7f56c8070080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:50.741 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.742+0000 7f56ee800640 1 --2- 172.21.15.96:0/2065073056 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f56c806dbc0 0x7f56c8070080 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f56d80097a0 tx=0x7f56d802e000 comp rx=0 tx=0).stop 2024-05-28T06:17:50.742 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.742+0000 7f56ee800640 1 -- 172.21.15.96:0/2065073056 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f56e80d4d60 msgr2=0x7f56e80b40b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:50.742 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.742+0000 7f56ee800640 1 --2- 172.21.15.96:0/2065073056 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f56e80d4d60 0x7f56e80b40b0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f56d400c480 tx=0x7f56d400c940 comp rx=0 tx=0).stop 2024-05-28T06:17:50.742 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.742+0000 7f56ee800640 1 -- 172.21.15.96:0/2065073056 shutdown_connections 2024-05-28T06:17:50.742 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.742+0000 7f56ee800640 1 --2- 172.21.15.96:0/2065073056 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f56c806dbc0 0x7f56c8070080 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:50.742 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.742+0000 7f56ee800640 1 --2- 172.21.15.96:0/2065073056 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f56e815ed20 0x7f56e815a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:50.742 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.742+0000 7f56ee800640 1 --2- 172.21.15.96:0/2065073056 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f56e80d4d60 0x7f56e80b40b0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:50.742 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.742+0000 7f56ee800640 1 --2- 172.21.15.96:0/2065073056 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f56e80d2db0 0x7f56e80c8d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:50.742 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.742+0000 7f56ee800640 1 -- 172.21.15.96:0/2065073056 >> 172.21.15.96:0/2065073056 conn(0x7f56e808bfc0 msgr2=0x7f56e815ad00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:50.742 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.743+0000 7f56ee800640 1 -- 172.21.15.96:0/2065073056 shutdown_connections 2024-05-28T06:17:50.742 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:50.743+0000 7f56ee800640 1 -- 172.21.15.96:0/2065073056 wait complete. 2024-05-28T06:17:50.758 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi096.front.sepia.ceph.com... 2024-05-28T06:17:50.758 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:17:50.758 DEBUG:teuthology.orchestra.run.smithi096:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-05-28T06:17:50.819 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi096.front.sepia.ceph.com 2024-05-28T06:17:50.819 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:17:50.819 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-28T06:17:50.854 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi096.stderr:ceph-fuse[64991]: fuse finished with error 0 and tester_r 0 2024-05-28T06:17:50.969 INFO:tasks.cephfs.mount:Terminating background process 2024-05-28T06:17:50.970 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-05-28T06:17:50.970 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:17:51.007 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052731,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248996k,nr_inodes=812249,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-05-28T06:17:51.007 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi123.front.sepia.ceph.com 2024-05-28T06:17:51.007 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:17:51.008 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-05-28T06:17:51.088 INFO:teuthology.orchestra.run.smithi123.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-05-28T06:17:51.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-28T06:17:51.089 INFO:tasks.cephfs.mount:Terminating background process 2024-05-28T06:17:51.089 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:17:51.089 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:17:51.139 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052731,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248996k,nr_inodes=812249,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-28T06:17:51.140 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi096.front.sepia.ceph.com 2024-05-28T06:17:51.140 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:17:51.140 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-05-28T06:17:51.234 INFO:teuthology.orchestra.run.smithi096.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-05-28T06:17:51.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-28T06:17:51.235 INFO:tasks.cephfs.mount:Terminating background process 2024-05-28T06:17:51.236 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-05-28T06:17:51.236 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:17:51.289 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052731,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248996k,nr_inodes=812249,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-05-28T06:17:51.289 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi123.front.sepia.ceph.com 2024-05-28T06:17:51.289 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:17:51.289 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-05-28T06:17:51.337 INFO:teuthology.orchestra.run.smithi123.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-05-28T06:17:51.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-28T06:17:51.338 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-28T06:17:51.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:51.548+0000 7ff2fd600640 1 Processor -- start 2024-05-28T06:17:51.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:51.548+0000 7ff2fd600640 1 -- start start 2024-05-28T06:17:51.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:51.549+0000 7ff2fd600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7ff2f8156b30 0x7ff2f80af6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:51.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:51.550+0000 7ff2fd600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7ff2f80ad940 0x7ff2f80b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:51.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:51.550+0000 7ff2fd600640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2f8156f10 con 0x7ff2f806ee40 2024-05-28T06:17:51.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:51.550+0000 7ff2fd600640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7ff2f806dda0 con 0x7ff2f8156b30 2024-05-28T06:17:51.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:51.550+0000 7ff2fd600640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7ff2f806e080 con 0x7ff2f80ad940 2024-05-28T06:17:51.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:51.550+0000 7ff2f7400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7ff2f806ee40 0x7ff2f80ad230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:59878/0 (socket says 172.21.15.96:59878) 2024-05-28T06:17:51.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:51.550+0000 7ff2f7400640 1 -- 172.21.15.96:0/1176981099 learned_addr learned my addr 172.21.15.96:0/1176981099 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:17:51.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:51.550+0000 7ff2f7e00640 1 --2- 172.21.15.96:0/1176981099 >> v2:172.21.15.96:3300/0 conn(0x7ff2f8156b30 0x7ff2f80af6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:51.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:51.550+0000 7ff2fcc00640 1 --2- 172.21.15.96:0/1176981099 >> v2:172.21.15.123:3300/0 conn(0x7ff2f80ad940 0x7ff2f80b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:51.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:51.550+0000 7ff2f6a00640 1 -- 172.21.15.96:0/1176981099 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1367170441 0 0) 0x7ff2f8156f10 con 0x7ff2f806ee40 2024-05-28T06:17:51.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:51.550+0000 7ff2f6a00640 1 -- 172.21.15.96:0/1176981099 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff2dc003520 con 0x7ff2f806ee40 2024-05-28T06:17:51.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:51.550+0000 7ff2f7e00640 1 -- 172.21.15.96:0/1176981099 >> v2:172.21.15.123:3300/0 conn(0x7ff2f80ad940 msgr2=0x7ff2f80b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:51.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:51.550+0000 7ff2f7e00640 1 --2- 172.21.15.96:0/1176981099 >> v2:172.21.15.123:3300/0 conn(0x7ff2f80ad940 0x7ff2f80b32c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:51.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:51.550+0000 7ff2f7e00640 1 -- 172.21.15.96:0/1176981099 >> v1:172.21.15.96:6789/0 conn(0x7ff2f806ee40 legacy=0x7ff2f80ad230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:51.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:51.550+0000 7ff2f7e00640 1 -- 172.21.15.96:0/1176981099 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2f806e360 con 0x7ff2f8156b30 2024-05-28T06:17:51.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:51.551+0000 7ff2fcc00640 1 --2- 172.21.15.96:0/1176981099 >> v2:172.21.15.123:3300/0 conn(0x7ff2f80ad940 0x7ff2f80b32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:17:51.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:51.552+0000 7ff2f7e00640 1 --2- 172.21.15.96:0/1176981099 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff2f8156b30 0x7ff2f80af6e0 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7ff2f80d0180 tx=0x7ff2e802a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cef53026ca87450d server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:51.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:51.552+0000 7ff2f6a00640 1 -- 172.21.15.96:0/1176981099 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2e8037070 con 0x7ff2f8156b30 2024-05-28T06:17:51.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:51.552+0000 7ff2f6a00640 1 -- 172.21.15.96:0/1176981099 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2e8032440 con 0x7ff2f8156b30 2024-05-28T06:17:51.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:51.552+0000 7ff2f6a00640 1 -- 172.21.15.96:0/1176981099 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2e802ae60 con 0x7ff2f8156b30 2024-05-28T06:17:51.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:51.552+0000 7ff2fd600640 1 -- 172.21.15.96:0/1176981099 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff2f8156b30 msgr2=0x7ff2f80af6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:51.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:51.552+0000 7ff2fd600640 1 --2- 172.21.15.96:0/1176981099 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff2f8156b30 0x7ff2f80af6e0 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7ff2f80d0180 tx=0x7ff2e802a110 comp rx=0 tx=0).stop 2024-05-28T06:17:51.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:51.552+0000 7ff2fd600640 1 -- 172.21.15.96:0/1176981099 shutdown_connections 2024-05-28T06:17:51.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:51.552+0000 7ff2fd600640 1 --2- 172.21.15.96:0/1176981099 >> v2:172.21.15.123:3300/0 conn(0x7ff2f80ad940 0x7ff2f80b32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:51.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:51.552+0000 7ff2fd600640 1 --2- 172.21.15.96:0/1176981099 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff2f8156b30 0x7ff2f80af6e0 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:51.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:51.552+0000 7ff2fd600640 1 -- 172.21.15.96:0/1176981099 >> 172.21.15.96:0/1176981099 conn(0x7ff2f80b1fc0 msgr2=0x7ff2f80b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:51.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:51.553+0000 7ff2fd600640 1 -- 172.21.15.96:0/1176981099 shutdown_connections 2024-05-28T06:17:51.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:51.553+0000 7ff2fd600640 1 -- 172.21.15.96:0/1176981099 wait complete. 2024-05-28T06:17:51.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:51.554+0000 7ff2fd600640 1 Processor -- start 2024-05-28T06:17:51.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:51.554+0000 7ff2fd600640 1 -- start start 2024-05-28T06:17:51.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:51.555+0000 7ff2fd600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff2f806ee40 0x7ff2f8099ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:51.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:51.555+0000 7ff2fd600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff2f80ad940 0x7ff2f81f2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:51.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:51.555+0000 7ff2fd600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff2f8156b30 0x7ff2f814a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:51.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:51.555+0000 7ff2fd600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff2f80afc20 con 0x7ff2f806ee40 2024-05-28T06:17:51.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:51.555+0000 7ff2fd600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff2f80b2c30 con 0x7ff2f8156b30 2024-05-28T06:17:51.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:51.555+0000 7ff2fd600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ff2f80b5120 con 0x7ff2f80ad940 2024-05-28T06:17:51.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:51.555+0000 7ff2f7e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff2f806ee40 0x7ff2f8099ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:51.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:51.555+0000 7ff2f7e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff2f806ee40 0x7ff2f8099ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:32806/0 (socket says 172.21.15.96:32806) 2024-05-28T06:17:51.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:51.555+0000 7ff2f7e00640 1 -- 172.21.15.96:0/1609179709 learned_addr learned my addr 172.21.15.96:0/1609179709 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:17:51.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:51.556+0000 7ff2f7e00640 1 -- 172.21.15.96:0/1609179709 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff2f80ad940 msgr2=0x7ff2f81f2110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:51.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:51.556+0000 7ff2f7e00640 1 --2- 172.21.15.96:0/1609179709 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff2f80ad940 0x7ff2f81f2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:51.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:51.556+0000 7ff2f7e00640 1 -- 172.21.15.96:0/1609179709 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff2f8156b30 msgr2=0x7ff2f814a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:51.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:51.556+0000 7ff2f7e00640 1 --2- 172.21.15.96:0/1609179709 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff2f8156b30 0x7ff2f814a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:51.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:51.556+0000 7ff2f7e00640 1 -- 172.21.15.96:0/1609179709 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2f815f3f0 con 0x7ff2f806ee40 2024-05-28T06:17:51.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:51.556+0000 7ff2f7e00640 1 --2- 172.21.15.96:0/1609179709 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff2f806ee40 0x7ff2f8099ab0 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7ff2e802a150 tx=0x7ff2e80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:51.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:51.556+0000 7ff2efe00640 1 -- 172.21.15.96:0/1609179709 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2e8037040 con 0x7ff2f806ee40 2024-05-28T06:17:51.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:51.556+0000 7ff2efe00640 1 -- 172.21.15.96:0/1609179709 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2e80304c0 con 0x7ff2f806ee40 2024-05-28T06:17:51.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:51.556+0000 7ff2fd600640 1 -- 172.21.15.96:0/1609179709 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2f815f6d0 con 0x7ff2f806ee40 2024-05-28T06:17:51.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:51.556+0000 7ff2efe00640 1 -- 172.21.15.96:0/1609179709 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2e8030890 con 0x7ff2f806ee40 2024-05-28T06:17:51.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:51.556+0000 7ff2fd600640 1 -- 172.21.15.96:0/1609179709 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2f80b4db0 con 0x7ff2f806ee40 2024-05-28T06:17:51.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:51.558+0000 7ff2fd600640 1 -- 172.21.15.96:0/1609179709 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2f80689e0 con 0x7ff2f806ee40 2024-05-28T06:17:51.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:51.559+0000 7ff2efe00640 1 -- 172.21.15.96:0/1609179709 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 29) ==== 92672+0+0 (secure 0 0 0) 0x7ff2e8044020 con 0x7ff2f806ee40 2024-05-28T06:17:51.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:51.560+0000 7ff2efe00640 1 --2- 172.21.15.96:0/1609179709 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff2d806daf0 0x7ff2d806ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:51.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:51.560+0000 7ff2efe00640 1 -- 172.21.15.96:0/1609179709 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 5823+0+0 (secure 0 0 0) 0x7ff2e80aaad0 con 0x7ff2f806ee40 2024-05-28T06:17:51.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:51.560+0000 7ff2f7400640 1 --2- 172.21.15.96:0/1609179709 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff2d806daf0 0x7ff2d806ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:51.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:51.561+0000 7ff2f7400640 1 --2- 172.21.15.96:0/1609179709 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff2d806daf0 0x7ff2d806ffb0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7ff2e4000f60 tx=0x7ff2e40078f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:51.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:51.564+0000 7ff2efe00640 1 -- 172.21.15.96:0/1609179709 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff2e807a950 con 0x7ff2f806ee40 2024-05-28T06:17:51.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:51.830+0000 7ff2fd600640 1 -- 172.21.15.96:0/1609179709 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff2f80ade80 con 0x7ff2f806ee40 2024-05-28T06:17:51.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:51.831+0000 7ff2efe00640 1 -- 172.21.15.96:0/1609179709 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) ==== 94+0+5177 (secure 0 0 0) 0x7ff2e807a250 con 0x7ff2f806ee40 2024-05-28T06:17:51.829 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:17:51.830 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":44,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5291,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6834/3727695025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":3727695025},{"type":"v1","addr":"172.21.15.96:6834","nonce":3727695025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38},{"gid":5300,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/324418606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6835","nonce":324418606},{"type":"v1","addr":"172.21.15.123:6837","nonce":324418606}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38},{"gid":5725,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/417566829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6833","nonce":417566829},{"type":"v1","addr":"172.21.15.96:6835","nonce":417566829}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-28T06:14:52.322597+0000","modified":"2024-05-28T06:14:55.415769+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7146},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7146":{"gid":7146,"name":"d","rank":0,"incarnation":39,"state":"up:active","state_seq":3,"addr":"172.21.15.123:6836/76143454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":76143454},{"type":"v1","addr":"172.21.15.123:6836","nonce":76143454}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7146,"qdb_cluster":[7146]},"id":5}]} 2024-05-28T06:17:51.830 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 44 2024-05-28T06:17:51.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:51.833+0000 7ff2fd600640 1 -- 172.21.15.96:0/1609179709 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff2d806daf0 msgr2=0x7ff2d806ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:51.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:51.833+0000 7ff2fd600640 1 --2- 172.21.15.96:0/1609179709 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff2d806daf0 0x7ff2d806ffb0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7ff2e4000f60 tx=0x7ff2e40078f0 comp rx=0 tx=0).stop 2024-05-28T06:17:51.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:51.833+0000 7ff2fd600640 1 -- 172.21.15.96:0/1609179709 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff2f806ee40 msgr2=0x7ff2f8099ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:51.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:51.833+0000 7ff2fd600640 1 --2- 172.21.15.96:0/1609179709 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff2f806ee40 0x7ff2f8099ab0 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7ff2e802a150 tx=0x7ff2e80046c0 comp rx=0 tx=0).stop 2024-05-28T06:17:51.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:51.833+0000 7ff2fd600640 1 -- 172.21.15.96:0/1609179709 shutdown_connections 2024-05-28T06:17:51.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:51.833+0000 7ff2fd600640 1 --2- 172.21.15.96:0/1609179709 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff2d806daf0 0x7ff2d806ffb0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:51.833 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:51.833+0000 7ff2fd600640 1 --2- 172.21.15.96:0/1609179709 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff2f8156b30 0x7ff2f814a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:51.833 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:51.833+0000 7ff2fd600640 1 --2- 172.21.15.96:0/1609179709 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff2f80ad940 0x7ff2f81f2110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:51.833 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:51.833+0000 7ff2fd600640 1 --2- 172.21.15.96:0/1609179709 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff2f806ee40 0x7ff2f8099ab0 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:51.833 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:51.833+0000 7ff2fd600640 1 -- 172.21.15.96:0/1609179709 >> 172.21.15.96:0/1609179709 conn(0x7ff2f80b1fc0 msgr2=0x7ff2f80af060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:51.833 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:51.833+0000 7ff2fd600640 1 -- 172.21.15.96:0/1609179709 shutdown_connections 2024-05-28T06:17:51.833 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:51.833+0000 7ff2fd600640 1 -- 172.21.15.96:0/1609179709 wait complete. 2024-05-28T06:17:51.849 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:17:52.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.030+0000 7f0e09400640 1 Processor -- start 2024-05-28T06:17:52.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.030+0000 7f0e09400640 1 -- start start 2024-05-28T06:17:52.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.032+0000 7f0e09400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f0e040b32c0 0x7f0e040b3750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:52.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.032+0000 7f0e09400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f0e0406e4b0 0x7f0e0406e900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:52.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.032+0000 7f0e09400640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e0415e4f0 con 0x7f0e0415e110 2024-05-28T06:17:52.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.032+0000 7f0e09400640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f0e0406ee40 con 0x7f0e040b32c0 2024-05-28T06:17:52.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.032+0000 7f0e09400640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f0e040b3d40 con 0x7f0e0406e4b0 2024-05-28T06:17:52.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.032+0000 7f0dfb400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f0e040b32c0 0x7f0e040b3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:52.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.032+0000 7f0dfb400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f0e040b32c0 0x7f0e040b3750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:32810/0 (socket says 172.21.15.96:32810) 2024-05-28T06:17:52.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.032+0000 7f0dfb400640 1 -- 172.21.15.96:0/4230929469 learned_addr learned my addr 172.21.15.96:0/4230929469 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:17:52.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.033+0000 7f0dfb400640 1 -- 172.21.15.96:0/4230929469 >> v2:172.21.15.123:3300/0 conn(0x7f0e0406e4b0 msgr2=0x7f0e0406e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:52.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.033+0000 7f0dfb400640 1 --2- 172.21.15.96:0/4230929469 >> v2:172.21.15.123:3300/0 conn(0x7f0e0406e4b0 0x7f0e0406e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:52.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.033+0000 7f0dfb400640 1 -- 172.21.15.96:0/4230929469 >> v1:172.21.15.123:6789/0 conn(0x7f0e0415e110 legacy=0x7f0e040b4de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:52.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.033+0000 7f0dfb400640 1 -- 172.21.15.96:0/4230929469 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e040b4020 con 0x7f0e040b32c0 2024-05-28T06:17:52.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.034+0000 7f0dfb400640 1 --2- 172.21.15.96:0/4230929469 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0e040b32c0 0x7f0e040b3750 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7f0dec009870 tx=0x7f0dec02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3406be247d405cb7 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:52.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.034+0000 7f0e03400640 1 -- 172.21.15.96:0/4230929469 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dec037070 con 0x7f0e040b32c0 2024-05-28T06:17:52.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.034+0000 7f0e03400640 1 -- 172.21.15.96:0/4230929469 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0dec032440 con 0x7f0e040b32c0 2024-05-28T06:17:52.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.034+0000 7f0e03400640 1 -- 172.21.15.96:0/4230929469 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dec032810 con 0x7f0e040b32c0 2024-05-28T06:17:52.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.035+0000 7f0e09400640 1 -- 172.21.15.96:0/4230929469 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0e040b32c0 msgr2=0x7f0e040b3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:52.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.035+0000 7f0e09400640 1 --2- 172.21.15.96:0/4230929469 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0e040b32c0 0x7f0e040b3750 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7f0dec009870 tx=0x7f0dec02a270 comp rx=0 tx=0).stop 2024-05-28T06:17:52.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.035+0000 7f0e09400640 1 -- 172.21.15.96:0/4230929469 shutdown_connections 2024-05-28T06:17:52.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.035+0000 7f0e09400640 1 --2- 172.21.15.96:0/4230929469 >> v2:172.21.15.123:3300/0 conn(0x7f0e0406e4b0 0x7f0e0406e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:52.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.035+0000 7f0e09400640 1 --2- 172.21.15.96:0/4230929469 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0e040b32c0 0x7f0e040b3750 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:52.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.035+0000 7f0e09400640 1 -- 172.21.15.96:0/4230929469 >> 172.21.15.96:0/4230929469 conn(0x7f0e040b1fc0 msgr2=0x7f0e040b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:52.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.035+0000 7f0e09400640 1 -- 172.21.15.96:0/4230929469 shutdown_connections 2024-05-28T06:17:52.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.035+0000 7f0e09400640 1 -- 172.21.15.96:0/4230929469 wait complete. 2024-05-28T06:17:52.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.037+0000 7f0e09400640 1 Processor -- start 2024-05-28T06:17:52.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.037+0000 7f0e09400640 1 -- start start 2024-05-28T06:17:52.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.037+0000 7f0e09400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0e0406e4b0 0x7f0e040ae6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:52.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.038+0000 7f0e09400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0e040b32c0 0x7f0e0409a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:52.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.038+0000 7f0e09400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0e0415e110 0x7f0e0409a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:52.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.038+0000 7f0e09400640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f0e040b48c0 con 0x7f0e0406e4b0 2024-05-28T06:17:52.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.038+0000 7f0e09400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f0e040b45e0 con 0x7f0e040b32c0 2024-05-28T06:17:52.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.038+0000 7f0e09400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f0e0406f4d0 con 0x7f0e0415e110 2024-05-28T06:17:52.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.038+0000 7f0e03e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0e0406e4b0 0x7f0e040ae6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:52.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.038+0000 7f0e03e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0e0406e4b0 0x7f0e040ae6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:32826/0 (socket says 172.21.15.96:32826) 2024-05-28T06:17:52.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.038+0000 7f0e03e00640 1 -- 172.21.15.96:0/3260998359 learned_addr learned my addr 172.21.15.96:0/3260998359 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:17:52.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.038+0000 7f0dfb400640 1 --2- 172.21.15.96:0/3260998359 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0e040b32c0 0x7f0e0409a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:52.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.038+0000 7f0e03e00640 1 -- 172.21.15.96:0/3260998359 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0e0415e110 msgr2=0x7f0e0409a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:52.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.038+0000 7f0e03e00640 1 --2- 172.21.15.96:0/3260998359 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0e0415e110 0x7f0e0409a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:52.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.038+0000 7f0e03e00640 1 -- 172.21.15.96:0/3260998359 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0e040b32c0 msgr2=0x7f0e0409a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:52.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.038+0000 7f0e03e00640 1 --2- 172.21.15.96:0/3260998359 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0e040b32c0 0x7f0e0409a3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:52.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.038+0000 7f0e03e00640 1 -- 172.21.15.96:0/3260998359 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e0409d260 con 0x7f0e0406e4b0 2024-05-28T06:17:52.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.039+0000 7f0e03e00640 1 --2- 172.21.15.96:0/3260998359 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0e0406e4b0 0x7f0e040ae6b0 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f0df400b3f0 tx=0x7f0df400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:52.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.039+0000 7f0e00c00640 1 -- 172.21.15.96:0/3260998359 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0df4015020 con 0x7f0e0406e4b0 2024-05-28T06:17:52.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.039+0000 7f0e00c00640 1 -- 172.21.15.96:0/3260998359 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0df4011830 con 0x7f0e0406e4b0 2024-05-28T06:17:52.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.039+0000 7f0e00c00640 1 -- 172.21.15.96:0/3260998359 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0df4011c20 con 0x7f0e0406e4b0 2024-05-28T06:17:52.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.039+0000 7f0e09400640 1 -- 172.21.15.96:0/3260998359 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e0409d4d0 con 0x7f0e0406e4b0 2024-05-28T06:17:52.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.039+0000 7f0e09400640 1 -- 172.21.15.96:0/3260998359 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e0409da90 con 0x7f0e0406e4b0 2024-05-28T06:17:52.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.040+0000 7f0e09400640 1 -- 172.21.15.96:0/3260998359 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e040689e0 con 0x7f0e0406e4b0 2024-05-28T06:17:52.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.041+0000 7f0e00c00640 1 -- 172.21.15.96:0/3260998359 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 29) ==== 92672+0+0 (secure 0 0 0) 0x7f0df4018020 con 0x7f0e0406e4b0 2024-05-28T06:17:52.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.042+0000 7f0e00c00640 1 --2- 172.21.15.96:0/3260998359 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0de006daf0 0x7f0de006ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:52.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.042+0000 7f0e00c00640 1 -- 172.21.15.96:0/3260998359 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 5823+0+0 (secure 0 0 0) 0x7f0df408a6f0 con 0x7f0e0406e4b0 2024-05-28T06:17:52.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.043+0000 7f0dfb400640 1 --2- 172.21.15.96:0/3260998359 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0de006daf0 0x7f0de006ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:52.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.044+0000 7f0dfb400640 1 --2- 172.21.15.96:0/3260998359 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0de006daf0 0x7f0de006ffb0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f0dec034000 tx=0x7f0dec02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:52.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.046+0000 7f0e00c00640 1 -- 172.21.15.96:0/3260998359 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0df401f030 con 0x7f0e0406e4b0 2024-05-28T06:17:52.233 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.235+0000 7f0e09400640 1 -- 172.21.15.96:0/3260998359 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0e041679c0 con 0x7f0e0406e4b0 2024-05-28T06:17:52.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.237+0000 7f0e00c00640 1 -- 172.21.15.96:0/3260998359 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) ==== 74+0+16887 (secure 0 0 0) 0x7f0df40172c0 con 0x7f0e0406e4b0 2024-05-28T06:17:52.237 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:17:52.237 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":78,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:17:51.459577+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-05-28T06:14:50.098829+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-05-28T06:14:51.318311+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":77,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"11.17","mappings":[{"from":6,"to":0}]},{"pgid":"11.28","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[{"pool":11,"snaps":[{"begin":2,"length":2}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:17:52.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.240+0000 7f0e09400640 1 -- 172.21.15.96:0/3260998359 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0de006daf0 msgr2=0x7f0de006ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:52.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.240+0000 7f0e09400640 1 --2- 172.21.15.96:0/3260998359 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0de006daf0 0x7f0de006ffb0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f0dec034000 tx=0x7f0dec02e000 comp rx=0 tx=0).stop 2024-05-28T06:17:52.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.240+0000 7f0e09400640 1 -- 172.21.15.96:0/3260998359 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0e0406e4b0 msgr2=0x7f0e040ae6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:52.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.240+0000 7f0e09400640 1 --2- 172.21.15.96:0/3260998359 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0e0406e4b0 0x7f0e040ae6b0 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f0df400b3f0 tx=0x7f0df400b8b0 comp rx=0 tx=0).stop 2024-05-28T06:17:52.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.240+0000 7f0e09400640 1 -- 172.21.15.96:0/3260998359 shutdown_connections 2024-05-28T06:17:52.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.240+0000 7f0e09400640 1 --2- 172.21.15.96:0/3260998359 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0de006daf0 0x7f0de006ffb0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:52.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.240+0000 7f0e09400640 1 --2- 172.21.15.96:0/3260998359 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0e0415e110 0x7f0e0409a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:52.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.240+0000 7f0e09400640 1 --2- 172.21.15.96:0/3260998359 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0e040b32c0 0x7f0e0409a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:52.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.240+0000 7f0e09400640 1 --2- 172.21.15.96:0/3260998359 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0e0406e4b0 0x7f0e040ae6b0 unknown :-1 s=CLOSED pgs=1311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:52.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.240+0000 7f0e09400640 1 -- 172.21.15.96:0/3260998359 >> 172.21.15.96:0/3260998359 conn(0x7f0e040b1fc0 msgr2=0x7f0e040b1560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:52.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.240+0000 7f0e09400640 1 -- 172.21.15.96:0/3260998359 shutdown_connections 2024-05-28T06:17:52.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.240+0000 7f0e09400640 1 -- 172.21.15.96:0/3260998359 wait complete. 2024-05-28T06:17:52.257 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-28T06:03:59.517102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-28T06:14:50.098829+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '73', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2024-05-28T06:14:51.318311+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '77', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 77, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-28T06:17:52.257 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-28T06:17:52.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.449+0000 7f03ff200640 1 Processor -- start 2024-05-28T06:17:52.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.449+0000 7f03ff200640 1 -- start start 2024-05-28T06:17:52.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.451+0000 7f03ff200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f04000cd2a0 0x7f04000b4d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:52.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.451+0000 7f03ff200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f040015cb60 0x7f040015cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:52.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.451+0000 7f03ff200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f040015d900 con 0x7f0400153510 2024-05-28T06:17:52.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.451+0000 7f03ff200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f04001538f0 con 0x7f04000cd2a0 2024-05-28T06:17:52.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.451+0000 7f03ff200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f04000b48c0 con 0x7f040015cb60 2024-05-28T06:17:52.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.451+0000 7f03fd400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f04000cd2a0 0x7f04000b4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:52.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.451+0000 7f03fd400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f04000cd2a0 0x7f04000b4d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:32834/0 (socket says 172.21.15.96:32834) 2024-05-28T06:17:52.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.451+0000 7f03fd400640 1 -- 172.21.15.96:0/1020105937 learned_addr learned my addr 172.21.15.96:0/1020105937 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:17:52.451 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.451+0000 7f03fe800640 1 --2- 172.21.15.96:0/1020105937 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f040015cb60 0x7f040015cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:52.451 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.451+0000 7f03fd400640 1 -- 172.21.15.96:0/1020105937 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f040015cb60 msgr2=0x7f040015cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:52.451 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.451+0000 7f03fd400640 1 --2- 172.21.15.96:0/1020105937 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f040015cb60 0x7f040015cfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:52.451 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.451+0000 7f03fd400640 1 -- 172.21.15.96:0/1020105937 >> v1:172.21.15.123:6789/0 conn(0x7f0400153510 legacy=0x7f04000ccb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:52.451 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.451+0000 7f03fd400640 1 -- 172.21.15.96:0/1020105937 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04000b3f90 con 0x7f04000cd2a0 2024-05-28T06:17:52.451 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.452+0000 7f03fd400640 1 --2- 172.21.15.96:0/1020105937 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f04000cd2a0 0x7f04000b4d50 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f03ec009870 tx=0x7f03ec02a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1462a78b60412763 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:52.452 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.453+0000 7f03fca00640 1 -- 172.21.15.96:0/1020105937 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03ec037070 con 0x7f04000cd2a0 2024-05-28T06:17:52.452 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.453+0000 7f03fca00640 1 -- 172.21.15.96:0/1020105937 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03ec032440 con 0x7f04000cd2a0 2024-05-28T06:17:52.452 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.453+0000 7f03fca00640 1 -- 172.21.15.96:0/1020105937 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03ec032810 con 0x7f04000cd2a0 2024-05-28T06:17:52.452 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.453+0000 7f03ff200640 1 -- 172.21.15.96:0/1020105937 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f04000cd2a0 msgr2=0x7f04000b4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:52.452 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.453+0000 7f03ff200640 1 --2- 172.21.15.96:0/1020105937 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f04000cd2a0 0x7f04000b4d50 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f03ec009870 tx=0x7f03ec02a180 comp rx=0 tx=0).stop 2024-05-28T06:17:52.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.453+0000 7f03ff200640 1 -- 172.21.15.96:0/1020105937 shutdown_connections 2024-05-28T06:17:52.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.453+0000 7f03ff200640 1 --2- 172.21.15.96:0/1020105937 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f040015cb60 0x7f040015cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:52.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.453+0000 7f03ff200640 1 --2- 172.21.15.96:0/1020105937 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f04000cd2a0 0x7f04000b4d50 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:52.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.453+0000 7f03ff200640 1 -- 172.21.15.96:0/1020105937 >> 172.21.15.96:0/1020105937 conn(0x7f040006b900 msgr2=0x7f040006bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:52.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.453+0000 7f03ff200640 1 -- 172.21.15.96:0/1020105937 shutdown_connections 2024-05-28T06:17:52.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.454+0000 7f03ff200640 1 -- 172.21.15.96:0/1020105937 wait complete. 2024-05-28T06:17:52.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.455+0000 7f03ff200640 1 Processor -- start 2024-05-28T06:17:52.454 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.455+0000 7f03ff200640 1 -- start start 2024-05-28T06:17:52.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.456+0000 7f03ff200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f04000cd2a0 0x7f0400073680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:52.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.456+0000 7f03ff200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0400153510 0x7f04000aeb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:52.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.456+0000 7f03ff200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f040015cb60 0x7f04000cee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:52.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.456+0000 7f03ff200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f040015d4f0 con 0x7f040015cb60 2024-05-28T06:17:52.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.456+0000 7f03ff200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f04000b4550 con 0x7f0400153510 2024-05-28T06:17:52.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.456+0000 7f03ff200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f040015d670 con 0x7f04000cd2a0 2024-05-28T06:17:52.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.457+0000 7f03fe800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f040015cb60 0x7f04000cee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:52.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.457+0000 7f03fe800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f040015cb60 0x7f04000cee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:32848/0 (socket says 172.21.15.96:32848) 2024-05-28T06:17:52.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.457+0000 7f03fe800640 1 -- 172.21.15.96:0/1430909887 learned_addr learned my addr 172.21.15.96:0/1430909887 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:17:52.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.457+0000 7f03fde00640 1 --2- 172.21.15.96:0/1430909887 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f04000cd2a0 0x7f0400073680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:52.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.457+0000 7f03fd400640 1 --2- 172.21.15.96:0/1430909887 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0400153510 0x7f04000aeb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:52.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.457+0000 7f03fe800640 1 -- 172.21.15.96:0/1430909887 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f04000cd2a0 msgr2=0x7f0400073680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:52.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.457+0000 7f03fe800640 1 --2- 172.21.15.96:0/1430909887 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f04000cd2a0 0x7f0400073680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:52.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.457+0000 7f03fe800640 1 -- 172.21.15.96:0/1430909887 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0400153510 msgr2=0x7f04000aeb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:52.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.457+0000 7f03fe800640 1 --2- 172.21.15.96:0/1430909887 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0400153510 0x7f04000aeb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:52.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.457+0000 7f03fe800640 1 -- 172.21.15.96:0/1430909887 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f040014d9f0 con 0x7f040015cb60 2024-05-28T06:17:52.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.457+0000 7f03fd400640 1 --2- 172.21.15.96:0/1430909887 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0400153510 0x7f04000aeb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:17:52.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.457+0000 7f03fe800640 1 --2- 172.21.15.96:0/1430909887 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f040015cb60 0x7f04000cee80 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f03f800cc60 tx=0x7f03f800a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:52.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.458+0000 7f03f6000640 1 -- 172.21.15.96:0/1430909887 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03f8003990 con 0x7f040015cb60 2024-05-28T06:17:52.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.458+0000 7f03f6000640 1 -- 172.21.15.96:0/1430909887 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03f8003b30 con 0x7f040015cb60 2024-05-28T06:17:52.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.458+0000 7f03f6000640 1 -- 172.21.15.96:0/1430909887 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03f8004020 con 0x7f040015cb60 2024-05-28T06:17:52.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.458+0000 7f03ff200640 1 -- 172.21.15.96:0/1430909887 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f040014bde0 con 0x7f040015cb60 2024-05-28T06:17:52.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.458+0000 7f03ff200640 1 -- 172.21.15.96:0/1430909887 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04000998d0 con 0x7f040015cb60 2024-05-28T06:17:52.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.459+0000 7f03ff200640 1 -- 172.21.15.96:0/1430909887 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04000689e0 con 0x7f040015cb60 2024-05-28T06:17:52.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.460+0000 7f03f6000640 1 -- 172.21.15.96:0/1430909887 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 29) ==== 92672+0+0 (secure 0 0 0) 0x7f03f80041c0 con 0x7f040015cb60 2024-05-28T06:17:52.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.461+0000 7f03f6000640 1 --2- 172.21.15.96:0/1430909887 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f03e006daf0 0x7f03e006ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:52.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.461+0000 7f03f6000640 1 -- 172.21.15.96:0/1430909887 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 5823+0+0 (secure 0 0 0) 0x7f03f808af70 con 0x7f040015cb60 2024-05-28T06:17:52.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.462+0000 7f03fde00640 1 --2- 172.21.15.96:0/1430909887 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f03e006daf0 0x7f03e006ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:52.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.463+0000 7f03fde00640 1 --2- 172.21.15.96:0/1430909887 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f03e006daf0 0x7f03e006ffb0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f03f0007fd0 tx=0x7f03f0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:52.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.465+0000 7f03f6000640 1 -- 172.21.15.96:0/1430909887 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f03f8014030 con 0x7f040015cb60 2024-05-28T06:17:52.479 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.480+0000 7f03f6000640 1 -- 172.21.15.96:0/1430909887 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mgrmap(e 30) ==== 92621+0+0 (secure 0 0 0) 0x7f03f80569f0 con 0x7f040015cb60 2024-05-28T06:17:52.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.666+0000 7f03ff200640 1 -- 172.21.15.96:0/1430909887 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f04000ae970 con 0x7f040015cb60 2024-05-28T06:17:52.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.669+0000 7f03f6000640 1 -- 172.21.15.96:0/1430909887 <== mon.0 v2:172.21.15.96:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v78) ==== 93+0+10 (secure 0 0 0) 0x7f03f805aa80 con 0x7f040015cb60 2024-05-28T06:17:52.668 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 1 2024-05-28T06:17:52.670 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.671+0000 7f03ff200640 1 -- 172.21.15.96:0/1430909887 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f03e006daf0 msgr2=0x7f03e006ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:52.671 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.671+0000 7f03ff200640 1 --2- 172.21.15.96:0/1430909887 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f03e006daf0 0x7f03e006ffb0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f03f0007fd0 tx=0x7f03f0009040 comp rx=0 tx=0).stop 2024-05-28T06:17:52.671 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.672+0000 7f03ff200640 1 -- 172.21.15.96:0/1430909887 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f040015cb60 msgr2=0x7f04000cee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:52.671 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.672+0000 7f03ff200640 1 --2- 172.21.15.96:0/1430909887 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f040015cb60 0x7f04000cee80 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f03f800cc60 tx=0x7f03f800a5c0 comp rx=0 tx=0).stop 2024-05-28T06:17:52.671 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.672+0000 7f03ff200640 1 -- 172.21.15.96:0/1430909887 shutdown_connections 2024-05-28T06:17:52.671 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.672+0000 7f03ff200640 1 --2- 172.21.15.96:0/1430909887 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f03e006daf0 0x7f03e006ffb0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:52.671 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.672+0000 7f03ff200640 1 --2- 172.21.15.96:0/1430909887 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f040015cb60 0x7f04000cee80 unknown :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:52.671 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.672+0000 7f03ff200640 1 --2- 172.21.15.96:0/1430909887 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0400153510 0x7f04000aeb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:52.671 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.672+0000 7f03ff200640 1 --2- 172.21.15.96:0/1430909887 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f04000cd2a0 0x7f0400073680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:52.671 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.672+0000 7f03ff200640 1 -- 172.21.15.96:0/1430909887 >> 172.21.15.96:0/1430909887 conn(0x7f040006b900 msgr2=0x7f04000cb010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:52.671 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.672+0000 7f03ff200640 1 -- 172.21.15.96:0/1430909887 shutdown_connections 2024-05-28T06:17:52.671 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.672+0000 7f03ff200640 1 -- 172.21.15.96:0/1430909887 wait complete. 2024-05-28T06:17:52.690 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-28T06:17:52.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.885+0000 7fde86600640 1 Processor -- start 2024-05-28T06:17:52.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.886+0000 7fde86600640 1 -- start start 2024-05-28T06:17:52.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.887+0000 7fde86600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fde800d0640 0x7fde8015d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:52.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.887+0000 7fde86600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fde800d0a20 0x7fde800cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:52.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.887+0000 7fde86600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde8015cb10 con 0x7fde800b32c0 2024-05-28T06:17:52.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.887+0000 7fde86600640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fde800adbb0 con 0x7fde800d0a20 2024-05-28T06:17:52.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.887+0000 7fde86600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fde8015db10 con 0x7fde800d0640 2024-05-28T06:17:52.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.888+0000 7fde7f400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fde800d0640 0x7fde8015d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:52.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.888+0000 7fde7f400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fde800d0640 0x7fde8015d5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.96:50432/0 (socket says 172.21.15.96:50432) 2024-05-28T06:17:52.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.888+0000 7fde7f400640 1 -- 172.21.15.96:0/524145241 learned_addr learned my addr 172.21.15.96:0/524145241 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:17:52.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.888+0000 7fde77e00640 1 --2- 172.21.15.96:0/524145241 >> v2:172.21.15.123:3300/0 conn(0x7fde800d0a20 0x7fde800cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:52.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.889+0000 7fde7f400640 1 -- 172.21.15.96:0/524145241 >> v2:172.21.15.123:3300/0 conn(0x7fde800d0a20 msgr2=0x7fde800cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:52.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.889+0000 7fde7f400640 1 --2- 172.21.15.96:0/524145241 >> v2:172.21.15.123:3300/0 conn(0x7fde800d0a20 0x7fde800cf950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:52.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.889+0000 7fde7f400640 1 -- 172.21.15.96:0/524145241 >> v1:172.21.15.123:6789/0 conn(0x7fde800b32c0 legacy=0x7fde800b3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:52.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.889+0000 7fde7f400640 1 -- 172.21.15.96:0/524145241 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde800ad5f0 con 0x7fde800d0640 2024-05-28T06:17:52.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.890+0000 7fde7f400640 1 --2- 172.21.15.96:0/524145241 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fde800d0640 0x7fde8015d5d0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fde70009870 tx=0x7fde7002a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=82ac27645dede07c server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:52.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.891+0000 7fde7ea00640 1 -- 172.21.15.96:0/524145241 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde70037070 con 0x7fde800d0640 2024-05-28T06:17:52.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.891+0000 7fde7ea00640 1 -- 172.21.15.96:0/524145241 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde70032440 con 0x7fde800d0640 2024-05-28T06:17:52.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.891+0000 7fde7ea00640 1 -- 172.21.15.96:0/524145241 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde70032810 con 0x7fde800d0640 2024-05-28T06:17:52.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.891+0000 7fde86600640 1 -- 172.21.15.96:0/524145241 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fde800d0640 msgr2=0x7fde8015d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:52.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.891+0000 7fde86600640 1 --2- 172.21.15.96:0/524145241 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fde800d0640 0x7fde8015d5d0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fde70009870 tx=0x7fde7002a270 comp rx=0 tx=0).stop 2024-05-28T06:17:52.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.892+0000 7fde86600640 1 -- 172.21.15.96:0/524145241 shutdown_connections 2024-05-28T06:17:52.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.892+0000 7fde86600640 1 --2- 172.21.15.96:0/524145241 >> v2:172.21.15.123:3300/0 conn(0x7fde800d0a20 0x7fde800cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:52.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.892+0000 7fde86600640 1 --2- 172.21.15.96:0/524145241 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fde800d0640 0x7fde8015d5d0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:52.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.892+0000 7fde86600640 1 -- 172.21.15.96:0/524145241 >> 172.21.15.96:0/524145241 conn(0x7fde8006bd20 msgr2=0x7fde8006c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:52.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.892+0000 7fde86600640 1 -- 172.21.15.96:0/524145241 shutdown_connections 2024-05-28T06:17:52.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.892+0000 7fde86600640 1 -- 172.21.15.96:0/524145241 wait complete. 2024-05-28T06:17:52.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.894+0000 7fde86600640 1 Processor -- start 2024-05-28T06:17:52.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.894+0000 7fde86600640 1 -- start start 2024-05-28T06:17:52.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.895+0000 7fde86600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fde800b32c0 0x7fde801523e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:52.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.895+0000 7fde86600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde800d0640 0x7fde801e84c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:52.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.895+0000 7fde86600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fde800d0a20 0x7fde8015b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:52.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.895+0000 7fde86600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fde8015d0b0 con 0x7fde800d0a20 2024-05-28T06:17:52.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.895+0000 7fde86600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fde8015cdd0 con 0x7fde800d0640 2024-05-28T06:17:52.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.895+0000 7fde86600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fde800b3e60 con 0x7fde800b32c0 2024-05-28T06:17:52.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.895+0000 7fde7fe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fde800d0a20 0x7fde8015b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:52.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.895+0000 7fde7fe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fde800d0a20 0x7fde8015b240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:32860/0 (socket says 172.21.15.96:32860) 2024-05-28T06:17:52.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.895+0000 7fde7fe00640 1 -- 172.21.15.96:0/315132751 learned_addr learned my addr 172.21.15.96:0/315132751 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:17:52.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.896+0000 7fde7f400640 1 --2- 172.21.15.96:0/315132751 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fde800b32c0 0x7fde801523e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:52.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.896+0000 7fde77e00640 1 --2- 172.21.15.96:0/315132751 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde800d0640 0x7fde801e84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:52.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.896+0000 7fde7fe00640 1 -- 172.21.15.96:0/315132751 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fde800b32c0 msgr2=0x7fde801523e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:52.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.896+0000 7fde7fe00640 1 --2- 172.21.15.96:0/315132751 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fde800b32c0 0x7fde801523e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:52.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.896+0000 7fde7fe00640 1 -- 172.21.15.96:0/315132751 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde800d0640 msgr2=0x7fde801e84c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:52.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.896+0000 7fde7fe00640 1 --2- 172.21.15.96:0/315132751 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde800d0640 0x7fde801e84c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:52.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.896+0000 7fde7fe00640 1 -- 172.21.15.96:0/315132751 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde801e72f0 con 0x7fde800d0a20 2024-05-28T06:17:52.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.896+0000 7fde7fe00640 1 --2- 172.21.15.96:0/315132751 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fde800d0a20 0x7fde8015b240 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7fde7800c7a0 tx=0x7fde7800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:52.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.897+0000 7fde77400640 1 -- 172.21.15.96:0/315132751 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde78003b70 con 0x7fde800d0a20 2024-05-28T06:17:52.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.897+0000 7fde77400640 1 -- 172.21.15.96:0/315132751 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde78003d30 con 0x7fde800d0a20 2024-05-28T06:17:52.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.897+0000 7fde77400640 1 -- 172.21.15.96:0/315132751 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde7800aa90 con 0x7fde800d0a20 2024-05-28T06:17:52.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.897+0000 7fde86600640 1 -- 172.21.15.96:0/315132751 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde80152920 con 0x7fde800d0a20 2024-05-28T06:17:52.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.897+0000 7fde86600640 1 -- 172.21.15.96:0/315132751 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde801e6810 con 0x7fde800d0a20 2024-05-28T06:17:52.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.898+0000 7fde86600640 1 -- 172.21.15.96:0/315132751 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde800af7a0 con 0x7fde800d0a20 2024-05-28T06:17:52.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.899+0000 7fde77400640 1 -- 172.21.15.96:0/315132751 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 30) ==== 92621+0+0 (secure 0 0 0) 0x7fde7800ac30 con 0x7fde800d0a20 2024-05-28T06:17:52.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.900+0000 7fde77400640 1 --2- 172.21.15.96:0/315132751 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fde5c06da60 0x7fde5c06ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:52.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.900+0000 7fde77400640 1 -- 172.21.15.96:0/315132751 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 5823+0+0 (secure 0 0 0) 0x7fde7808aa30 con 0x7fde800d0a20 2024-05-28T06:17:52.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.900+0000 7fde7f400640 1 --2- 172.21.15.96:0/315132751 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fde5c06da60 0x7fde5c06ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:52.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.901+0000 7fde7f400640 1 --2- 172.21.15.96:0/315132751 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fde5c06da60 0x7fde5c06ff20 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fde70034000 tx=0x7fde7002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:52.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:52.904+0000 7fde77400640 1 -- 172.21.15.96:0/315132751 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fde800af7a0 con 0x7fde800d0a20 2024-05-28T06:17:53.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.088+0000 7fde86600640 1 -- 172.21.15.96:0/315132751 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fde800689e0 con 0x7fde800d0a20 2024-05-28T06:17:53.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.088+0000 7fde77400640 1 -- 172.21.15.96:0/315132751 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v78) ==== 104+0+11 (secure 0 0 0) 0x7fde7805a560 con 0x7fde800d0a20 2024-05-28T06:17:53.087 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 32 2024-05-28T06:17:53.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.091+0000 7fde86600640 1 -- 172.21.15.96:0/315132751 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fde5c06da60 msgr2=0x7fde5c06ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:53.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.091+0000 7fde86600640 1 --2- 172.21.15.96:0/315132751 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fde5c06da60 0x7fde5c06ff20 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fde70034000 tx=0x7fde7002e000 comp rx=0 tx=0).stop 2024-05-28T06:17:53.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.091+0000 7fde86600640 1 -- 172.21.15.96:0/315132751 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fde800d0a20 msgr2=0x7fde8015b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:53.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.091+0000 7fde86600640 1 --2- 172.21.15.96:0/315132751 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fde800d0a20 0x7fde8015b240 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7fde7800c7a0 tx=0x7fde7800cc60 comp rx=0 tx=0).stop 2024-05-28T06:17:53.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.091+0000 7fde86600640 1 -- 172.21.15.96:0/315132751 shutdown_connections 2024-05-28T06:17:53.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.091+0000 7fde86600640 1 --2- 172.21.15.96:0/315132751 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fde5c06da60 0x7fde5c06ff20 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:53.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.091+0000 7fde86600640 1 --2- 172.21.15.96:0/315132751 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fde800d0a20 0x7fde8015b240 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:53.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.091+0000 7fde86600640 1 --2- 172.21.15.96:0/315132751 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde800d0640 0x7fde801e84c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:53.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.091+0000 7fde86600640 1 --2- 172.21.15.96:0/315132751 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fde800b32c0 0x7fde801523e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:53.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.091+0000 7fde86600640 1 -- 172.21.15.96:0/315132751 >> 172.21.15.96:0/315132751 conn(0x7fde8006bd20 msgr2=0x7fde80154fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:53.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.091+0000 7fde86600640 1 -- 172.21.15.96:0/315132751 shutdown_connections 2024-05-28T06:17:53.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.091+0000 7fde86600640 1 -- 172.21.15.96:0/315132751 wait complete. 2024-05-28T06:17:53.107 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-28T06:17:53.300 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.301+0000 7fbbb1e00640 1 Processor -- start 2024-05-28T06:17:53.301 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.302+0000 7fbbb1e00640 1 -- start start 2024-05-28T06:17:53.302 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.303+0000 7fbbb1e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbbac0d0640 0x7fbbac0d0a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:53.302 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.303+0000 7fbbb1e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fbbac0725e0 0x7fbbac0ae650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:53.302 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.303+0000 7fbbb1e00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbbac0ade90 con 0x7fbbac15b6f0 2024-05-28T06:17:53.302 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.303+0000 7fbbb1e00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fbbac071c40 con 0x7fbbac0725e0 2024-05-28T06:17:53.302 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.303+0000 7fbbb1e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fbbac0d0f60 con 0x7fbbac0d0640 2024-05-28T06:17:53.304 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.304+0000 7fbba2000640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fbbac15b6f0 0x7fbbac06e350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:59886/0 (socket says 172.21.15.96:59886) 2024-05-28T06:17:53.304 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.304+0000 7fbba2000640 1 -- 172.21.15.96:0/3455132710 learned_addr learned my addr 172.21.15.96:0/3455132710 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:17:53.304 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.304+0000 7fbbaaa00640 1 --2- 172.21.15.96:0/3455132710 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbbac0d0640 0x7fbbac0d0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:53.304 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.304+0000 7fbbab400640 1 --2- 172.21.15.96:0/3455132710 >> v2:172.21.15.123:3300/0 conn(0x7fbbac0725e0 0x7fbbac0ae650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:53.304 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.304+0000 7fbbaa000640 1 -- 172.21.15.96:0/3455132710 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3435628402 0 0) 0x7fbbac0ade90 con 0x7fbbac15b6f0 2024-05-28T06:17:53.304 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.304+0000 7fbbaa000640 1 -- 172.21.15.96:0/3455132710 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbb90005730 con 0x7fbbac15b6f0 2024-05-28T06:17:53.304 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.304+0000 7fbbaa000640 1 -- 172.21.15.96:0/3455132710 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3274028974 0 0) 0x7fbb90005730 con 0x7fbbac15b6f0 2024-05-28T06:17:53.305 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.304+0000 7fbbaa000640 1 -- 172.21.15.96:0/3455132710 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbbac0d0640 msgr2=0x7fbbac0d0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:53.305 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.304+0000 7fbbaa000640 1 --2- 172.21.15.96:0/3455132710 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbbac0d0640 0x7fbbac0d0a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:53.305 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.304+0000 7fbbaa000640 1 -- 172.21.15.96:0/3455132710 >> v2:172.21.15.123:3300/0 conn(0x7fbbac0725e0 msgr2=0x7fbbac0ae650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:53.305 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.305+0000 7fbbaa000640 1 --2- 172.21.15.96:0/3455132710 >> v2:172.21.15.123:3300/0 conn(0x7fbbac0725e0 0x7fbbac0ae650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:53.305 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.305+0000 7fbbaa000640 1 -- 172.21.15.96:0/3455132710 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbac071e70 con 0x7fbbac15b6f0 2024-05-28T06:17:53.305 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.305+0000 7fbbaa000640 1 -- 172.21.15.96:0/3455132710 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7fbb94001670 con 0x7fbbac15b6f0 2024-05-28T06:17:53.305 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.305+0000 7fbbab400640 1 --2- 172.21.15.96:0/3455132710 >> v2:172.21.15.123:3300/0 conn(0x7fbbac0725e0 0x7fbbac0ae650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-28T06:17:53.305 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.305+0000 7fbbaa000640 1 -- 172.21.15.96:0/3455132710 >> v1:172.21.15.96:6789/0 conn(0x7fbbac15b6f0 legacy=0x7fbbac06e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:53.305 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.305+0000 7fbbaa000640 1 --2- 172.21.15.96:0/3455132710 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbb90005bc0 0x7fbb90005fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:53.305 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.305+0000 7fbbaa000640 1 --2- 172.21.15.96:0/3455132710 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbbac0d0640 0x7fbb90006a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:53.305 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.305+0000 7fbbaa000640 1 --2- 172.21.15.96:0/3455132710 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbb90006f50 0x7fbb900073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:53.305 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.305+0000 7fbbaa000640 1 -- 172.21.15.96:0/3455132710 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fbbac0d0f60 con 0x7fbb90006f50 2024-05-28T06:17:53.306 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.305+0000 7fbbaa000640 1 -- 172.21.15.96:0/3455132710 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fbb9000dbd0 con 0x7fbb90005bc0 2024-05-28T06:17:53.306 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.305+0000 7fbbaa000640 1 -- 172.21.15.96:0/3455132710 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fbb9000ddd0 con 0x7fbbac0d0640 2024-05-28T06:17:53.306 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.305+0000 7fbbab400640 1 --2- 172.21.15.96:0/3455132710 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbb90006f50 0x7fbb900073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:53.306 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.305+0000 7fbbaaa00640 1 --2- 172.21.15.96:0/3455132710 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbb90005bc0 0x7fbb90005fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:53.306 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.305+0000 7fbbab400640 1 -- 172.21.15.96:0/3455132710 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbbac0d0640 msgr2=0x7fbb90006a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:53.306 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.305+0000 7fbbab400640 1 --2- 172.21.15.96:0/3455132710 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbbac0d0640 0x7fbb90006a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:53.306 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.305+0000 7fbbab400640 1 -- 172.21.15.96:0/3455132710 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbb90005bc0 msgr2=0x7fbb90005fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:53.306 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.305+0000 7fbbab400640 1 --2- 172.21.15.96:0/3455132710 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbb90005bc0 0x7fbb90005fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:53.306 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.305+0000 7fbbab400640 1 -- 172.21.15.96:0/3455132710 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb90005730 con 0x7fbb90006f50 2024-05-28T06:17:53.307 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.306+0000 7fbbab400640 1 --2- 172.21.15.96:0/3455132710 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbb90006f50 0x7fbb900073d0 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7fbba40073e0 tx=0x7fbb90007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd76eaa49e7618d3 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:53.308 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.307+0000 7fbbaa000640 1 -- 172.21.15.96:0/3455132710 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbba4009c80 con 0x7fbb90006f50 2024-05-28T06:17:53.308 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.307+0000 7fbbaa000640 1 -- 172.21.15.96:0/3455132710 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbba402c680 con 0x7fbb90006f50 2024-05-28T06:17:53.308 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.307+0000 7fbbaa000640 1 -- 172.21.15.96:0/3455132710 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbba402ca30 con 0x7fbb90006f50 2024-05-28T06:17:53.308 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.307+0000 7fbbb1e00640 1 -- 172.21.15.96:0/3455132710 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbb90006f50 msgr2=0x7fbb900073d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:53.308 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.307+0000 7fbbb1e00640 1 --2- 172.21.15.96:0/3455132710 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbb90006f50 0x7fbb900073d0 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7fbba40073e0 tx=0x7fbb90007a40 comp rx=0 tx=0).stop 2024-05-28T06:17:53.308 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.307+0000 7fbbab400640 1 -- 172.21.15.96:0/3455132710 reap_dead start 2024-05-28T06:17:53.308 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.307+0000 7fbbb1e00640 1 -- 172.21.15.96:0/3455132710 shutdown_connections 2024-05-28T06:17:53.308 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.307+0000 7fbbb1e00640 1 -- 172.21.15.96:0/3455132710 >> 172.21.15.96:0/3455132710 conn(0x7fbbac06bd20 msgr2=0x7fbbac06c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:53.308 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.307+0000 7fbbb1e00640 1 -- 172.21.15.96:0/3455132710 shutdown_connections 2024-05-28T06:17:53.308 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.308+0000 7fbbb1e00640 1 -- 172.21.15.96:0/3455132710 wait complete. 2024-05-28T06:17:53.308 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.309+0000 7fbbb1e00640 1 Processor -- start 2024-05-28T06:17:53.308 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.309+0000 7fbbb1e00640 1 -- start start 2024-05-28T06:17:53.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.310+0000 7fbbb1e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbb90006f50 0x7fbbac151690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:53.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.310+0000 7fbbb1e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbbac1f4160 0x7fbbac15d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:53.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.310+0000 7fbbb1e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbbac15cb90 0x7fbbac0aeff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:53.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.310+0000 7fbbb1e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fbbac0d0a20 con 0x7fbbac1f4160 2024-05-28T06:17:53.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.310+0000 7fbbb1e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fbbac0ae130 con 0x7fbbac15cb90 2024-05-28T06:17:53.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.310+0000 7fbbb1e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fbbac0d0ba0 con 0x7fbb90006f50 2024-05-28T06:17:53.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.310+0000 7fbba2000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbbac1f4160 0x7fbbac15d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:53.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.310+0000 7fbba2000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbbac1f4160 0x7fbbac15d9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:32882/0 (socket says 172.21.15.96:32882) 2024-05-28T06:17:53.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.311+0000 7fbba2000640 1 -- 172.21.15.96:0/546512601 learned_addr learned my addr 172.21.15.96:0/546512601 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:17:53.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.311+0000 7fbbaaa00640 1 --2- 172.21.15.96:0/546512601 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbb90006f50 0x7fbbac151690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:53.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.311+0000 7fbba2000640 1 -- 172.21.15.96:0/546512601 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbb90006f50 msgr2=0x7fbbac151690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:53.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.311+0000 7fbba2000640 1 --2- 172.21.15.96:0/546512601 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbb90006f50 0x7fbbac151690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:53.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.311+0000 7fbba2000640 1 -- 172.21.15.96:0/546512601 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbbac15cb90 msgr2=0x7fbbac0aeff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:53.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.311+0000 7fbba2000640 1 --2- 172.21.15.96:0/546512601 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbbac15cb90 0x7fbbac0aeff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:53.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.311+0000 7fbba2000640 1 -- 172.21.15.96:0/546512601 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbac0ccbe0 con 0x7fbbac1f4160 2024-05-28T06:17:53.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.311+0000 7fbba2000640 1 --2- 172.21.15.96:0/546512601 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbbac1f4160 0x7fbbac15d9f0 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7fbb9400c7a0 tx=0x7fbb9400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:53.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.311+0000 7fbba3e00640 1 -- 172.21.15.96:0/546512601 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb94018070 con 0x7fbbac1f4160 2024-05-28T06:17:53.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.311+0000 7fbba3e00640 1 -- 172.21.15.96:0/546512601 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb9400ce50 con 0x7fbbac1f4160 2024-05-28T06:17:53.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.312+0000 7fbba3e00640 1 -- 172.21.15.96:0/546512601 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb940047e0 con 0x7fbbac1f4160 2024-05-28T06:17:53.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.312+0000 7fbbb1e00640 1 -- 172.21.15.96:0/546512601 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbac0ccec0 con 0x7fbbac1f4160 2024-05-28T06:17:53.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.312+0000 7fbbb1e00640 1 -- 172.21.15.96:0/546512601 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbac0cd480 con 0x7fbbac1f4160 2024-05-28T06:17:53.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.313+0000 7fbbb1e00640 1 -- 172.21.15.96:0/546512601 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbbac0689e0 con 0x7fbbac1f4160 2024-05-28T06:17:53.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.314+0000 7fbba3e00640 1 -- 172.21.15.96:0/546512601 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 30) ==== 92621+0+0 (secure 0 0 0) 0x7fbb9401b070 con 0x7fbbac1f4160 2024-05-28T06:17:53.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.315+0000 7fbba3e00640 1 --2- 172.21.15.96:0/546512601 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbb8806da60 0x7fbb8806ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:53.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.315+0000 7fbba3e00640 1 -- 172.21.15.96:0/546512601 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 5823+0+0 (secure 0 0 0) 0x7fbb94089f90 con 0x7fbbac1f4160 2024-05-28T06:17:53.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.315+0000 7fbbaaa00640 1 --2- 172.21.15.96:0/546512601 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbb8806da60 0x7fbb8806ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:53.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.316+0000 7fbbaaa00640 1 --2- 172.21.15.96:0/546512601 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbb8806da60 0x7fbb8806ff20 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fbb9c0098b0 tx=0x7fbb9c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:53.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.319+0000 7fbba3e00640 1 -- 172.21.15.96:0/546512601 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbb94059ac0 con 0x7fbbac1f4160 2024-05-28T06:17:53.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.520+0000 7fbbb1e00640 1 -- 172.21.15.96:0/546512601 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fbbac0b3ac0 con 0x7fbbac1f4160 2024-05-28T06:17:53.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.521+0000 7fbba3e00640 1 -- 172.21.15.96:0/546512601 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v78) ==== 100+0+11 (secure 0 0 0) 0x7fbb940593c0 con 0x7fbbac1f4160 2024-05-28T06:17:53.520 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 64 2024-05-28T06:17:53.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.524+0000 7fbbb1e00640 1 -- 172.21.15.96:0/546512601 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbb8806da60 msgr2=0x7fbb8806ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:53.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.524+0000 7fbbb1e00640 1 --2- 172.21.15.96:0/546512601 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbb8806da60 0x7fbb8806ff20 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fbb9c0098b0 tx=0x7fbb9c008040 comp rx=0 tx=0).stop 2024-05-28T06:17:53.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.524+0000 7fbbb1e00640 1 -- 172.21.15.96:0/546512601 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbbac1f4160 msgr2=0x7fbbac15d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:53.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.524+0000 7fbbb1e00640 1 --2- 172.21.15.96:0/546512601 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbbac1f4160 0x7fbbac15d9f0 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7fbb9400c7a0 tx=0x7fbb9400cc60 comp rx=0 tx=0).stop 2024-05-28T06:17:53.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.524+0000 7fbbb1e00640 1 -- 172.21.15.96:0/546512601 shutdown_connections 2024-05-28T06:17:53.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.524+0000 7fbbb1e00640 1 --2- 172.21.15.96:0/546512601 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbb8806da60 0x7fbb8806ff20 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:53.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.524+0000 7fbbb1e00640 1 --2- 172.21.15.96:0/546512601 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbbac15cb90 0x7fbbac0aeff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:53.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.524+0000 7fbbb1e00640 1 --2- 172.21.15.96:0/546512601 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbbac1f4160 0x7fbbac15d9f0 unknown :-1 s=CLOSED pgs=1317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:53.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.524+0000 7fbbb1e00640 1 --2- 172.21.15.96:0/546512601 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbb90006f50 0x7fbbac151690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:53.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.524+0000 7fbbb1e00640 1 -- 172.21.15.96:0/546512601 >> 172.21.15.96:0/546512601 conn(0x7fbbac06bd20 msgr2=0x7fbbac0b4d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:53.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.524+0000 7fbbb1e00640 1 -- 172.21.15.96:0/546512601 shutdown_connections 2024-05-28T06:17:53.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.525+0000 7fbbb1e00640 1 -- 172.21.15.96:0/546512601 wait complete. 2024-05-28T06:17:53.542 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-28T06:17:53.742 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.743+0000 7f07d8200640 1 Processor -- start 2024-05-28T06:17:53.742 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.743+0000 7f07d8200640 1 -- start start 2024-05-28T06:17:53.743 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.744+0000 7f07d8200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f07d006f860 0x7f07d00b3150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:53.743 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.744+0000 7f07d8200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f07d00b3690 0x7f07d00d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:53.743 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.744+0000 7f07d8200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f07d00d09e0 0x7f07d0160210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:53.743 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.744+0000 7f07d8200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f07d00d0e50 con 0x7f07d00b3690 2024-05-28T06:17:53.744 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.745+0000 7f07d8200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f07d006ed10 con 0x7f07d00d09e0 2024-05-28T06:17:53.744 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.745+0000 7f07d8200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f07d015cb00 con 0x7f07d006f860 2024-05-28T06:17:53.744 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.745+0000 7f07d4e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f07d00b3690 0x7f07d00d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:53.744 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.745+0000 7f07d4e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f07d00b3690 0x7f07d00d04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:32894/0 (socket says 172.21.15.96:32894) 2024-05-28T06:17:53.745 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.745+0000 7f07d4e00640 1 -- 172.21.15.96:0/540332354 learned_addr learned my addr 172.21.15.96:0/540332354 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:17:53.746 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.745+0000 7f07d5800640 1 --2- 172.21.15.96:0/540332354 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f07d006f860 0x7f07d00b3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:53.746 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.745+0000 7f07d4e00640 1 -- 172.21.15.96:0/540332354 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f07d006f860 msgr2=0x7f07d00b3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:53.746 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.745+0000 7f07d6200640 1 --2- 172.21.15.96:0/540332354 >> v2:172.21.15.123:3300/0 conn(0x7f07d00d09e0 0x7f07d0160210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:53.747 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.745+0000 7f07d4e00640 1 --2- 172.21.15.96:0/540332354 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f07d006f860 0x7f07d00b3150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:53.747 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.745+0000 7f07d4e00640 1 -- 172.21.15.96:0/540332354 >> v2:172.21.15.123:3300/0 conn(0x7f07d00d09e0 msgr2=0x7f07d0160210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:53.747 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.745+0000 7f07d4e00640 1 --2- 172.21.15.96:0/540332354 >> v2:172.21.15.123:3300/0 conn(0x7f07d00d09e0 0x7f07d0160210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:53.747 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.745+0000 7f07d4e00640 1 -- 172.21.15.96:0/540332354 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07d00b3cc0 con 0x7f07d00b3690 2024-05-28T06:17:53.747 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.745+0000 7f07d6200640 1 --2- 172.21.15.96:0/540332354 >> v2:172.21.15.123:3300/0 conn(0x7f07d00d09e0 0x7f07d0160210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:17:53.747 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.746+0000 7f07d4e00640 1 --2- 172.21.15.96:0/540332354 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f07d00b3690 0x7f07d00d04a0 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f07d006c900 tx=0x7f07bc02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6cfbd9f26193de9 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:53.747 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.747+0000 7f07cfe00640 1 -- 172.21.15.96:0/540332354 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07bc037070 con 0x7f07d00b3690 2024-05-28T06:17:53.747 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.747+0000 7f07cfe00640 1 -- 172.21.15.96:0/540332354 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07bc032440 con 0x7f07d00b3690 2024-05-28T06:17:53.747 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.747+0000 7f07cfe00640 1 -- 172.21.15.96:0/540332354 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07bc032810 con 0x7f07d00b3690 2024-05-28T06:17:53.747 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.747+0000 7f07d8200640 1 -- 172.21.15.96:0/540332354 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f07d00b3690 msgr2=0x7f07d00d04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:53.747 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.747+0000 7f07d8200640 1 --2- 172.21.15.96:0/540332354 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f07d00b3690 0x7f07d00d04a0 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f07d006c900 tx=0x7f07bc02a1e0 comp rx=0 tx=0).stop 2024-05-28T06:17:53.747 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.747+0000 7f07d8200640 1 -- 172.21.15.96:0/540332354 shutdown_connections 2024-05-28T06:17:53.748 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.747+0000 7f07d8200640 1 --2- 172.21.15.96:0/540332354 >> v2:172.21.15.123:3300/0 conn(0x7f07d00d09e0 0x7f07d0160210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:53.748 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.747+0000 7f07d8200640 1 --2- 172.21.15.96:0/540332354 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f07d00b3690 0x7f07d00d04a0 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:53.748 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.747+0000 7f07d8200640 1 --2- 172.21.15.96:0/540332354 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f07d006f860 0x7f07d00b3150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:53.748 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.747+0000 7f07d8200640 1 -- 172.21.15.96:0/540332354 >> 172.21.15.96:0/540332354 conn(0x7f07d006bae0 msgr2=0x7f07d006bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:53.748 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.747+0000 7f07d8200640 1 -- 172.21.15.96:0/540332354 shutdown_connections 2024-05-28T06:17:53.748 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.748+0000 7f07d8200640 1 -- 172.21.15.96:0/540332354 wait complete. 2024-05-28T06:17:53.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.749+0000 7f07d8200640 1 Processor -- start 2024-05-28T06:17:53.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.749+0000 7f07d8200640 1 -- start start 2024-05-28T06:17:53.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.750+0000 7f07d8200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f07d006f860 0x7f07d01f4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:53.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.750+0000 7f07d8200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f07d00b3690 0x7f07d01f5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:53.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.750+0000 7f07d8200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f07d00d09e0 0x7f07d01f5db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:53.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.750+0000 7f07d8200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f07d00b4560 con 0x7f07d006f860 2024-05-28T06:17:53.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.750+0000 7f07d8200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f07d00b4280 con 0x7f07d00d09e0 2024-05-28T06:17:53.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.750+0000 7f07d8200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f07d00b4840 con 0x7f07d00b3690 2024-05-28T06:17:53.750 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.750+0000 7f07d5800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f07d006f860 0x7f07d01f4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:53.750 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.750+0000 7f07d5800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f07d006f860 0x7f07d01f4fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:32900/0 (socket says 172.21.15.96:32900) 2024-05-28T06:17:53.750 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.750+0000 7f07d5800640 1 -- 172.21.15.96:0/705213958 learned_addr learned my addr 172.21.15.96:0/705213958 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:17:53.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.751+0000 7f07d5800640 1 -- 172.21.15.96:0/705213958 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f07d00b3690 msgr2=0x7f07d01f5870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:53.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.751+0000 7f07d4e00640 1 --2- 172.21.15.96:0/705213958 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f07d00b3690 0x7f07d01f5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:53.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.751+0000 7f07d6200640 1 --2- 172.21.15.96:0/705213958 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f07d00d09e0 0x7f07d01f5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:53.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.751+0000 7f07d5800640 1 --2- 172.21.15.96:0/705213958 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f07d00b3690 0x7f07d01f5870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:53.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.751+0000 7f07d5800640 1 -- 172.21.15.96:0/705213958 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f07d00d09e0 msgr2=0x7f07d01f5db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:53.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.751+0000 7f07d5800640 1 --2- 172.21.15.96:0/705213958 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f07d00d09e0 0x7f07d01f5db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:53.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.751+0000 7f07d5800640 1 -- 172.21.15.96:0/705213958 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07d00c63a0 con 0x7f07d006f860 2024-05-28T06:17:53.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.751+0000 7f07d5800640 1 --2- 172.21.15.96:0/705213958 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f07d006f860 0x7f07d01f4fd0 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f07c000ce10 tx=0x7f07c000a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:53.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.751+0000 7f07cd600640 1 -- 172.21.15.96:0/705213958 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07c0003b90 con 0x7f07d006f860 2024-05-28T06:17:53.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.752+0000 7f07cd600640 1 -- 172.21.15.96:0/705213958 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07c0003d30 con 0x7f07d006f860 2024-05-28T06:17:53.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.752+0000 7f07cd600640 1 -- 172.21.15.96:0/705213958 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07c0004240 con 0x7f07d006f860 2024-05-28T06:17:53.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.752+0000 7f07d8200640 1 -- 172.21.15.96:0/705213958 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07d00c55e0 con 0x7f07d006f860 2024-05-28T06:17:53.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.752+0000 7f07d8200640 1 -- 172.21.15.96:0/705213958 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07d00c5ba0 con 0x7f07d006f860 2024-05-28T06:17:53.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.753+0000 7f07d8200640 1 -- 172.21.15.96:0/705213958 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07d00689e0 con 0x7f07d006f860 2024-05-28T06:17:53.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.754+0000 7f07cd600640 1 -- 172.21.15.96:0/705213958 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 30) ==== 92621+0+0 (secure 0 0 0) 0x7f07c00043e0 con 0x7f07d006f860 2024-05-28T06:17:53.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.755+0000 7f07cd600640 1 --2- 172.21.15.96:0/705213958 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f07b006da60 0x7f07b006ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:53.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.755+0000 7f07cd600640 1 -- 172.21.15.96:0/705213958 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 5823+0+0 (secure 0 0 0) 0x7f07c008b580 con 0x7f07d006f860 2024-05-28T06:17:53.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.755+0000 7f07d4e00640 1 --2- 172.21.15.96:0/705213958 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f07b006da60 0x7f07b006ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:53.755 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.757+0000 7f07d4e00640 1 --2- 172.21.15.96:0/705213958 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f07b006da60 0x7f07b006ff20 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f07bc034000 tx=0x7f07bc02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:53.757 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.759+0000 7f07cd600640 1 -- 172.21.15.96:0/705213958 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f07c001d340 con 0x7f07d006f860 2024-05-28T06:17:53.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.982+0000 7f07d8200640 1 -- 172.21.15.96:0/705213958 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f07d00c6fd0 con 0x7f07d006f860 2024-05-28T06:17:53.982 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.984+0000 7f07cd600640 1 -- 172.21.15.96:0/705213958 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) ==== 94+0+5177 (secure 0 0 0) 0x7f07c005b0b0 con 0x7f07d006f860 2024-05-28T06:17:53.983 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:17:53.983 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":44,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5291,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6834/3727695025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":3727695025},{"type":"v1","addr":"172.21.15.96:6834","nonce":3727695025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38},{"gid":5300,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/324418606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6835","nonce":324418606},{"type":"v1","addr":"172.21.15.123:6837","nonce":324418606}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38},{"gid":5725,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/417566829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6833","nonce":417566829},{"type":"v1","addr":"172.21.15.96:6835","nonce":417566829}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-28T06:14:52.322597+0000","modified":"2024-05-28T06:14:55.415769+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7146},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7146":{"gid":7146,"name":"d","rank":0,"incarnation":39,"state":"up:active","state_seq":3,"addr":"172.21.15.123:6836/76143454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":76143454},{"type":"v1","addr":"172.21.15.123:6836","nonce":76143454}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7146,"qdb_cluster":[7146]},"id":5}]} 2024-05-28T06:17:53.983 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 44 2024-05-28T06:17:53.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.986+0000 7f07d8200640 1 -- 172.21.15.96:0/705213958 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f07b006da60 msgr2=0x7f07b006ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:53.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.986+0000 7f07d8200640 1 --2- 172.21.15.96:0/705213958 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f07b006da60 0x7f07b006ff20 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f07bc034000 tx=0x7f07bc02e000 comp rx=0 tx=0).stop 2024-05-28T06:17:53.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.986+0000 7f07d8200640 1 -- 172.21.15.96:0/705213958 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f07d006f860 msgr2=0x7f07d01f4fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:53.986 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.986+0000 7f07d8200640 1 --2- 172.21.15.96:0/705213958 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f07d006f860 0x7f07d01f4fd0 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f07c000ce10 tx=0x7f07c000a660 comp rx=0 tx=0).stop 2024-05-28T06:17:53.986 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.987+0000 7f07d8200640 1 -- 172.21.15.96:0/705213958 shutdown_connections 2024-05-28T06:17:53.986 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.987+0000 7f07d8200640 1 --2- 172.21.15.96:0/705213958 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f07b006da60 0x7f07b006ff20 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:53.986 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.987+0000 7f07d8200640 1 --2- 172.21.15.96:0/705213958 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f07d00d09e0 0x7f07d01f5db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:53.986 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.987+0000 7f07d8200640 1 --2- 172.21.15.96:0/705213958 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f07d00b3690 0x7f07d01f5870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:53.986 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.987+0000 7f07d8200640 1 --2- 172.21.15.96:0/705213958 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f07d006f860 0x7f07d01f4fd0 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:53.986 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.987+0000 7f07d8200640 1 -- 172.21.15.96:0/705213958 >> 172.21.15.96:0/705213958 conn(0x7f07d006bae0 msgr2=0x7f07d0073050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:53.986 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.987+0000 7f07d8200640 1 -- 172.21.15.96:0/705213958 shutdown_connections 2024-05-28T06:17:53.986 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:53.987+0000 7f07d8200640 1 -- 172.21.15.96:0/705213958 wait complete. 2024-05-28T06:17:54.004 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:17:54.201 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.202+0000 7f97d6800640 1 Processor -- start 2024-05-28T06:17:54.201 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.202+0000 7f97d6800640 1 -- start start 2024-05-28T06:17:54.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.204+0000 7f97d6800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f97d006f860 0x7f97d00b3150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:54.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.204+0000 7f97d6800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f97d00b3690 0x7f97d00d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:54.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.204+0000 7f97d6800640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97d006ed10 con 0x7f97d00d09e0 2024-05-28T06:17:54.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.204+0000 7f97d6800640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f97d00d0df0 con 0x7f97d00b3690 2024-05-28T06:17:54.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.204+0000 7f97d6800640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f97d015cb00 con 0x7f97d006f860 2024-05-28T06:17:54.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.204+0000 7f97cea00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f97d00b3690 0x7f97d00d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:54.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.204+0000 7f97cea00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f97d00b3690 0x7f97d00d04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:32902/0 (socket says 172.21.15.96:32902) 2024-05-28T06:17:54.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.204+0000 7f97cea00640 1 -- 172.21.15.96:0/3693605868 learned_addr learned my addr 172.21.15.96:0/3693605868 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:17:54.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.204+0000 7f97cea00640 1 -- 172.21.15.96:0/3693605868 >> v2:172.21.15.123:3300/0 conn(0x7f97d006f860 msgr2=0x7f97d00b3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:54.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.204+0000 7f97cea00640 1 --2- 172.21.15.96:0/3693605868 >> v2:172.21.15.123:3300/0 conn(0x7f97d006f860 0x7f97d00b3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:54.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.204+0000 7f97cea00640 1 -- 172.21.15.96:0/3693605868 >> v1:172.21.15.123:6789/0 conn(0x7f97d00d09e0 legacy=0x7f97d0160210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:54.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.205+0000 7f97cea00640 1 -- 172.21.15.96:0/3693605868 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97d00b3cc0 con 0x7f97d00b3690 2024-05-28T06:17:54.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.206+0000 7f97cea00640 1 --2- 172.21.15.96:0/3693605868 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97d00b3690 0x7f97d00d04a0 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f97d006c8d0 tx=0x7f97bc02a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bfb1c595618bf32 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:54.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.206+0000 7f97ce000640 1 -- 172.21.15.96:0/3693605868 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97bc037070 con 0x7f97d00b3690 2024-05-28T06:17:54.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.206+0000 7f97ce000640 1 -- 172.21.15.96:0/3693605868 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97bc032440 con 0x7f97d00b3690 2024-05-28T06:17:54.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.206+0000 7f97ce000640 1 -- 172.21.15.96:0/3693605868 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97bc032810 con 0x7f97d00b3690 2024-05-28T06:17:54.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.207+0000 7f97d6800640 1 -- 172.21.15.96:0/3693605868 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97d00b3690 msgr2=0x7f97d00d04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:54.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.207+0000 7f97d6800640 1 --2- 172.21.15.96:0/3693605868 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97d00b3690 0x7f97d00d04a0 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f97d006c8d0 tx=0x7f97bc02a230 comp rx=0 tx=0).stop 2024-05-28T06:17:54.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.207+0000 7f97d6800640 1 -- 172.21.15.96:0/3693605868 shutdown_connections 2024-05-28T06:17:54.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.207+0000 7f97d6800640 1 --2- 172.21.15.96:0/3693605868 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97d00b3690 0x7f97d00d04a0 unknown :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:54.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.207+0000 7f97d6800640 1 --2- 172.21.15.96:0/3693605868 >> v2:172.21.15.123:3300/0 conn(0x7f97d006f860 0x7f97d00b3150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:54.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.207+0000 7f97d6800640 1 -- 172.21.15.96:0/3693605868 >> 172.21.15.96:0/3693605868 conn(0x7f97d006bae0 msgr2=0x7f97d006bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:54.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.207+0000 7f97d6800640 1 -- 172.21.15.96:0/3693605868 shutdown_connections 2024-05-28T06:17:54.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.207+0000 7f97d6800640 1 -- 172.21.15.96:0/3693605868 wait complete. 2024-05-28T06:17:54.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.209+0000 7f97d6800640 1 Processor -- start 2024-05-28T06:17:54.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.209+0000 7f97d6800640 1 -- start start 2024-05-28T06:17:54.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.210+0000 7f97d6800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f97d006f860 0x7f97d00c9f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:54.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.210+0000 7f97d6800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f97d00b3690 0x7f97d01f45d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:54.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.210+0000 7f97d6800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97d00d09e0 0x7f97d00c73e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:54.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.210+0000 7f97d6800640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f97d00b4560 con 0x7f97d00d09e0 2024-05-28T06:17:54.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.210+0000 7f97d6800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f97d00b4280 con 0x7f97d006f860 2024-05-28T06:17:54.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.210+0000 7f97d6800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f97d00b4840 con 0x7f97d00b3690 2024-05-28T06:17:54.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.210+0000 7f97cfe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97d00d09e0 0x7f97d00c73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:54.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.210+0000 7f97cfe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97d00d09e0 0x7f97d00c73e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:32912/0 (socket says 172.21.15.96:32912) 2024-05-28T06:17:54.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.210+0000 7f97cfe00640 1 -- 172.21.15.96:0/1374275346 learned_addr learned my addr 172.21.15.96:0/1374275346 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:17:54.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.211+0000 7f97cf400640 1 --2- 172.21.15.96:0/1374275346 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f97d006f860 0x7f97d00c9f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:54.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.211+0000 7f97cea00640 1 --2- 172.21.15.96:0/1374275346 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f97d00b3690 0x7f97d01f45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:54.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.211+0000 7f97cfe00640 1 -- 172.21.15.96:0/1374275346 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f97d00b3690 msgr2=0x7f97d01f45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:54.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.211+0000 7f97cfe00640 1 --2- 172.21.15.96:0/1374275346 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f97d00b3690 0x7f97d01f45d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:54.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.211+0000 7f97cfe00640 1 -- 172.21.15.96:0/1374275346 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f97d006f860 msgr2=0x7f97d00c9f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:54.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.211+0000 7f97cfe00640 1 --2- 172.21.15.96:0/1374275346 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f97d006f860 0x7f97d00c9f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:54.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.211+0000 7f97cfe00640 1 -- 172.21.15.96:0/1374275346 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97d01f4d40 con 0x7f97d00d09e0 2024-05-28T06:17:54.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.211+0000 7f97cea00640 1 --2- 172.21.15.96:0/1374275346 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f97d00b3690 0x7f97d01f45d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:17:54.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.211+0000 7f97cfe00640 1 --2- 172.21.15.96:0/1374275346 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97d00d09e0 0x7f97d00c73e0 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f97c800c780 tx=0x7f97c800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:54.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.211+0000 7f97c7400640 1 -- 172.21.15.96:0/1374275346 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97c8004b60 con 0x7f97d00d09e0 2024-05-28T06:17:54.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.211+0000 7f97c7400640 1 -- 172.21.15.96:0/1374275346 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97c8004d00 con 0x7f97d00d09e0 2024-05-28T06:17:54.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.211+0000 7f97c7400640 1 -- 172.21.15.96:0/1374275346 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97c800aa30 con 0x7f97d00d09e0 2024-05-28T06:17:54.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.212+0000 7f97d6800640 1 -- 172.21.15.96:0/1374275346 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97d01f5020 con 0x7f97d00d09e0 2024-05-28T06:17:54.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.212+0000 7f97d6800640 1 -- 172.21.15.96:0/1374275346 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97d00cdfd0 con 0x7f97d00d09e0 2024-05-28T06:17:54.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.213+0000 7f97d6800640 1 -- 172.21.15.96:0/1374275346 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97d015b210 con 0x7f97d00d09e0 2024-05-28T06:17:54.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.214+0000 7f97c7400640 1 -- 172.21.15.96:0/1374275346 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 30) ==== 92621+0+0 (secure 0 0 0) 0x7f97c800abd0 con 0x7f97d00d09e0 2024-05-28T06:17:54.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.214+0000 7f97c7400640 1 --2- 172.21.15.96:0/1374275346 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f97b006da60 0x7f97b006ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:54.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.215+0000 7f97c7400640 1 -- 172.21.15.96:0/1374275346 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 5823+0+0 (secure 0 0 0) 0x7f97c808ad90 con 0x7f97d00d09e0 2024-05-28T06:17:54.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.215+0000 7f97cf400640 1 --2- 172.21.15.96:0/1374275346 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f97b006da60 0x7f97b006ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:54.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.216+0000 7f97cf400640 1 --2- 172.21.15.96:0/1374275346 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f97b006da60 0x7f97b006ff20 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f97d01f7b20 tx=0x7f97c0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:54.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.219+0000 7f97c7400640 1 -- 172.21.15.96:0/1374275346 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f97d015b210 con 0x7f97d00d09e0 2024-05-28T06:17:54.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.422+0000 7f97d6800640 1 -- 172.21.15.96:0/1374275346 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f97d00689e0 con 0x7f97d00d09e0 2024-05-28T06:17:54.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.424+0000 7f97c7400640 1 -- 172.21.15.96:0/1374275346 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) ==== 74+0+16887 (secure 0 0 0) 0x7f97c8020070 con 0x7f97d00d09e0 2024-05-28T06:17:54.423 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:17:54.423 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":78,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:17:51.459577+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-05-28T06:14:50.098829+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-05-28T06:14:51.318311+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":77,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"11.17","mappings":[{"from":6,"to":0}]},{"pgid":"11.28","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[{"pool":11,"snaps":[{"begin":2,"length":2}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:17:54.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.426+0000 7f97d6800640 1 -- 172.21.15.96:0/1374275346 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f97b006da60 msgr2=0x7f97b006ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:54.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.426+0000 7f97d6800640 1 --2- 172.21.15.96:0/1374275346 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f97b006da60 0x7f97b006ff20 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f97d01f7b20 tx=0x7f97c0008040 comp rx=0 tx=0).stop 2024-05-28T06:17:54.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.426+0000 7f97d6800640 1 -- 172.21.15.96:0/1374275346 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97d00d09e0 msgr2=0x7f97d00c73e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:54.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.426+0000 7f97d6800640 1 --2- 172.21.15.96:0/1374275346 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97d00d09e0 0x7f97d00c73e0 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f97c800c780 tx=0x7f97c800cc40 comp rx=0 tx=0).stop 2024-05-28T06:17:54.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.426+0000 7f97d6800640 1 -- 172.21.15.96:0/1374275346 shutdown_connections 2024-05-28T06:17:54.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.426+0000 7f97d6800640 1 --2- 172.21.15.96:0/1374275346 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f97b006da60 0x7f97b006ff20 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:54.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.426+0000 7f97d6800640 1 --2- 172.21.15.96:0/1374275346 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97d00d09e0 0x7f97d00c73e0 unknown :-1 s=CLOSED pgs=1321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:54.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.426+0000 7f97d6800640 1 --2- 172.21.15.96:0/1374275346 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f97d00b3690 0x7f97d01f45d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:54.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.426+0000 7f97d6800640 1 --2- 172.21.15.96:0/1374275346 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f97d006f860 0x7f97d00c9f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:54.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.426+0000 7f97d6800640 1 -- 172.21.15.96:0/1374275346 >> 172.21.15.96:0/1374275346 conn(0x7f97d006bae0 msgr2=0x7f97d00adfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:54.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.427+0000 7f97d6800640 1 -- 172.21.15.96:0/1374275346 shutdown_connections 2024-05-28T06:17:54.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.427+0000 7f97d6800640 1 -- 172.21.15.96:0/1374275346 wait complete. 2024-05-28T06:17:54.444 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-05-28T06:17:54.444 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-28T06:17:54.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.645+0000 7f522fe00640 1 Processor -- start 2024-05-28T06:17:54.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.645+0000 7f522fe00640 1 -- start start 2024-05-28T06:17:54.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.646+0000 7f522fe00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f523015e530 0x7f523006f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:54.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.646+0000 7f522fe00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f5230070700 0x7f52300af8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:54.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.646+0000 7f522fe00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f5230162170 0x7f52300afe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:54.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.646+0000 7f522fe00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f523006e500 con 0x7f5230070700 2024-05-28T06:17:54.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.646+0000 7f522fe00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f523006e7e0 con 0x7f5230162170 2024-05-28T06:17:54.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.646+0000 7f522fe00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f523006eac0 con 0x7f523015e530 2024-05-28T06:17:54.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.647+0000 7f5226000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f5230070700 0x7f52300af8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:54.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.647+0000 7f522f400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f5230162170 0x7f52300afe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:54.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.647+0000 7f5226000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f5230070700 0x7f52300af8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:32922/0 (socket says 172.21.15.96:32922) 2024-05-28T06:17:54.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.647+0000 7f5226000640 1 -- 172.21.15.96:0/2914725122 learned_addr learned my addr 172.21.15.96:0/2914725122 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:17:54.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.647+0000 7f522f400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f5230162170 0x7f52300afe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.96:53052/0 (socket says 172.21.15.96:53052) 2024-05-28T06:17:54.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.647+0000 7f522ea00640 1 --2- 172.21.15.96:0/2914725122 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f523015e530 0x7f523006f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:54.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.647+0000 7f5226000640 1 -- 172.21.15.96:0/2914725122 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f523015e530 msgr2=0x7f523006f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:54.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.647+0000 7f5226000640 1 --2- 172.21.15.96:0/2914725122 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f523015e530 0x7f523006f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:54.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.647+0000 7f5226000640 1 -- 172.21.15.96:0/2914725122 >> v2:172.21.15.123:3300/0 conn(0x7f5230162170 msgr2=0x7f52300afe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:54.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.647+0000 7f5226000640 1 --2- 172.21.15.96:0/2914725122 >> v2:172.21.15.123:3300/0 conn(0x7f5230162170 0x7f52300afe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:54.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.647+0000 7f5226000640 1 -- 172.21.15.96:0/2914725122 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f523006eda0 con 0x7f5230070700 2024-05-28T06:17:54.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.649+0000 7f5226000640 1 --2- 172.21.15.96:0/2914725122 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5230070700 0x7f52300af8e0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f5230070e40 tx=0x7f521802a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b91e607d8e2ae7da server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:54.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.649+0000 7f522e000640 1 -- 172.21.15.96:0/2914725122 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5218036070 con 0x7f5230070700 2024-05-28T06:17:54.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.649+0000 7f522e000640 1 -- 172.21.15.96:0/2914725122 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5218031600 con 0x7f5230070700 2024-05-28T06:17:54.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.649+0000 7f522e000640 1 -- 172.21.15.96:0/2914725122 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52180319d0 con 0x7f5230070700 2024-05-28T06:17:54.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.649+0000 7f522fe00640 1 -- 172.21.15.96:0/2914725122 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5230070700 msgr2=0x7f52300af8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:54.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.649+0000 7f522fe00640 1 --2- 172.21.15.96:0/2914725122 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5230070700 0x7f52300af8e0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f5230070e40 tx=0x7f521802a610 comp rx=0 tx=0).stop 2024-05-28T06:17:54.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.650+0000 7f522fe00640 1 -- 172.21.15.96:0/2914725122 shutdown_connections 2024-05-28T06:17:54.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.650+0000 7f522fe00640 1 --2- 172.21.15.96:0/2914725122 >> v2:172.21.15.123:3300/0 conn(0x7f5230162170 0x7f52300afe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:54.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.650+0000 7f522fe00640 1 --2- 172.21.15.96:0/2914725122 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5230070700 0x7f52300af8e0 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:54.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.650+0000 7f522fe00640 1 --2- 172.21.15.96:0/2914725122 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f523015e530 0x7f523006f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:54.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.650+0000 7f522fe00640 1 -- 172.21.15.96:0/2914725122 >> 172.21.15.96:0/2914725122 conn(0x7f523006b900 msgr2=0x7f523006bce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:54.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.650+0000 7f522fe00640 1 -- 172.21.15.96:0/2914725122 shutdown_connections 2024-05-28T06:17:54.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.650+0000 7f522fe00640 1 -- 172.21.15.96:0/2914725122 wait complete. 2024-05-28T06:17:54.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.651+0000 7f522fe00640 1 Processor -- start 2024-05-28T06:17:54.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.651+0000 7f522fe00640 1 -- start start 2024-05-28T06:17:54.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.652+0000 7f522fe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5230070700 0x7f52301f44f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:54.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.652+0000 7f522fe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f523015e530 0x7f523015fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:54.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.652+0000 7f522fe00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5230162170 0x7f5230153320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:54.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.652+0000 7f522fe00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f52300af570 con 0x7f5230070700 2024-05-28T06:17:54.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.652+0000 7f522fe00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f523015e910 con 0x7f523015e530 2024-05-28T06:17:54.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.652+0000 7f522fe00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f52300af6f0 con 0x7f5230162170 2024-05-28T06:17:54.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.653+0000 7f522ea00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5230070700 0x7f52301f44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:54.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.653+0000 7f522ea00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5230070700 0x7f52301f44f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:32934/0 (socket says 172.21.15.96:32934) 2024-05-28T06:17:54.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.653+0000 7f522ea00640 1 -- 172.21.15.96:0/778923056 learned_addr learned my addr 172.21.15.96:0/778923056 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:17:54.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.653+0000 7f522f400640 1 --2- 172.21.15.96:0/778923056 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5230162170 0x7f5230153320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:54.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.653+0000 7f5226000640 1 --2- 172.21.15.96:0/778923056 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f523015e530 0x7f523015fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:54.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.653+0000 7f522ea00640 1 -- 172.21.15.96:0/778923056 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5230162170 msgr2=0x7f5230153320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:54.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.653+0000 7f522ea00640 1 --2- 172.21.15.96:0/778923056 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5230162170 0x7f5230153320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:54.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.653+0000 7f522ea00640 1 -- 172.21.15.96:0/778923056 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f523015e530 msgr2=0x7f523015fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:54.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.653+0000 7f522ea00640 1 --2- 172.21.15.96:0/778923056 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f523015e530 0x7f523015fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:54.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.653+0000 7f522ea00640 1 -- 172.21.15.96:0/778923056 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52301f4b00 con 0x7f5230070700 2024-05-28T06:17:54.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.653+0000 7f5226000640 1 --2- 172.21.15.96:0/778923056 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f523015e530 0x7f523015fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:17:54.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.653+0000 7f522ea00640 1 --2- 172.21.15.96:0/778923056 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5230070700 0x7f52301f44f0 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f522000d830 tx=0x7f522000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:54.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.654+0000 7f5227e00640 1 -- 172.21.15.96:0/778923056 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5220018030 con 0x7f5230070700 2024-05-28T06:17:54.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.654+0000 7f5227e00640 1 -- 172.21.15.96:0/778923056 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f522000be10 con 0x7f5230070700 2024-05-28T06:17:54.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.654+0000 7f5227e00640 1 -- 172.21.15.96:0/778923056 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52200098b0 con 0x7f5230070700 2024-05-28T06:17:54.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.654+0000 7f522fe00640 1 -- 172.21.15.96:0/778923056 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52301f4de0 con 0x7f5230070700 2024-05-28T06:17:54.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.654+0000 7f522fe00640 1 -- 172.21.15.96:0/778923056 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52301f53a0 con 0x7f5230070700 2024-05-28T06:17:54.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.655+0000 7f522fe00640 1 -- 172.21.15.96:0/778923056 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52300d0470 con 0x7f5230070700 2024-05-28T06:17:54.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.656+0000 7f5227e00640 1 -- 172.21.15.96:0/778923056 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 30) ==== 92621+0+0 (secure 0 0 0) 0x7f522001b070 con 0x7f5230070700 2024-05-28T06:17:54.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.657+0000 7f5227e00640 1 --2- 172.21.15.96:0/778923056 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f520c06da60 0x7f520c06ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:54.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.657+0000 7f5227e00640 1 -- 172.21.15.96:0/778923056 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 5823+0+0 (secure 0 0 0) 0x7f522008abc0 con 0x7f5230070700 2024-05-28T06:17:54.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.657+0000 7f5226000640 1 --2- 172.21.15.96:0/778923056 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f520c06da60 0x7f520c06ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:54.657 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.659+0000 7f5226000640 1 --2- 172.21.15.96:0/778923056 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f520c06da60 0x7f520c06ff20 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f5218033000 tx=0x7f5218009000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:54.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.661+0000 7f5227e00640 1 -- 172.21.15.96:0/778923056 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f52300d0470 con 0x7f5230070700 2024-05-28T06:17:54.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.902+0000 7f522fe00640 1 -- 172.21.15.96:0/778923056 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f52300689e0 con 0x7f5230070700 2024-05-28T06:17:54.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.903+0000 7f5227e00640 1 -- 172.21.15.96:0/778923056 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) ==== 94+0+5177 (secure 0 0 0) 0x7f522005a750 con 0x7f5230070700 2024-05-28T06:17:54.902 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:17:54.902 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":44,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5291,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6834/3727695025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":3727695025},{"type":"v1","addr":"172.21.15.96:6834","nonce":3727695025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38},{"gid":5300,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/324418606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6835","nonce":324418606},{"type":"v1","addr":"172.21.15.123:6837","nonce":324418606}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38},{"gid":5725,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/417566829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6833","nonce":417566829},{"type":"v1","addr":"172.21.15.96:6835","nonce":417566829}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-28T06:14:52.322597+0000","modified":"2024-05-28T06:14:55.415769+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7146},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7146":{"gid":7146,"name":"d","rank":0,"incarnation":39,"state":"up:active","state_seq":3,"addr":"172.21.15.123:6836/76143454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":76143454},{"type":"v1","addr":"172.21.15.123:6836","nonce":76143454}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7146,"qdb_cluster":[7146]},"id":5}]} 2024-05-28T06:17:54.903 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 44 2024-05-28T06:17:54.905 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.906+0000 7f522fe00640 1 -- 172.21.15.96:0/778923056 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f520c06da60 msgr2=0x7f520c06ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:54.905 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.906+0000 7f522fe00640 1 --2- 172.21.15.96:0/778923056 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f520c06da60 0x7f520c06ff20 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f5218033000 tx=0x7f5218009000 comp rx=0 tx=0).stop 2024-05-28T06:17:54.905 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.906+0000 7f522fe00640 1 -- 172.21.15.96:0/778923056 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5230070700 msgr2=0x7f52301f44f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:54.905 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.906+0000 7f522fe00640 1 --2- 172.21.15.96:0/778923056 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5230070700 0x7f52301f44f0 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f522000d830 tx=0x7f522000dcf0 comp rx=0 tx=0).stop 2024-05-28T06:17:54.905 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.906+0000 7f522fe00640 1 -- 172.21.15.96:0/778923056 shutdown_connections 2024-05-28T06:17:54.905 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.906+0000 7f522fe00640 1 --2- 172.21.15.96:0/778923056 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f520c06da60 0x7f520c06ff20 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:54.905 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.906+0000 7f522fe00640 1 --2- 172.21.15.96:0/778923056 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5230162170 0x7f5230153320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:54.905 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.906+0000 7f522fe00640 1 --2- 172.21.15.96:0/778923056 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f523015e530 0x7f523015fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:54.906 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.906+0000 7f522fe00640 1 --2- 172.21.15.96:0/778923056 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5230070700 0x7f52301f44f0 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:54.906 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.906+0000 7f522fe00640 1 -- 172.21.15.96:0/778923056 >> 172.21.15.96:0/778923056 conn(0x7f523006b900 msgr2=0x7f52300af870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:54.906 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.906+0000 7f522fe00640 1 -- 172.21.15.96:0/778923056 shutdown_connections 2024-05-28T06:17:54.906 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:54.906+0000 7f522fe00640 1 -- 172.21.15.96:0/778923056 wait complete. 2024-05-28T06:17:54.925 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-28T06:17:55.125 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.126+0000 7fbf7d000640 1 Processor -- start 2024-05-28T06:17:55.125 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.126+0000 7fbf7d000640 1 -- start start 2024-05-28T06:17:55.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.128+0000 7fbf7d000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fbf780b4b60 0x7fbf780cfaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:55.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.128+0000 7fbf7d000640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf780ced40 con 0x7fbf780d04a0 2024-05-28T06:17:55.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.128+0000 7fbf7d000640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf780b3f90 con 0x7fbf780b3b70 2024-05-28T06:17:55.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.128+0000 7fbf7d000640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fbf7815fbb0 con 0x7fbf780b4b60 2024-05-28T06:17:55.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.128+0000 7fbf6d400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fbf780d04a0 0x7fbf780b3460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:59896/0 (socket says 172.21.15.96:59896) 2024-05-28T06:17:55.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.128+0000 7fbf6d400640 1 -- 172.21.15.96:0/3297334581 learned_addr learned my addr 172.21.15.96:0/3297334581 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:17:55.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.129+0000 7fbf75400640 1 -- 172.21.15.96:0/3297334581 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1822526931 0 0) 0x7fbf60001110 con 0x7fbf780d04a0 2024-05-28T06:17:55.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.129+0000 7fbf75400640 1 -- 172.21.15.96:0/3297334581 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf5c003520 con 0x7fbf780d04a0 2024-05-28T06:17:55.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.129+0000 7fbf75e00640 1 --2- 172.21.15.96:0/3297334581 >> v2:172.21.15.123:3300/0 conn(0x7fbf780b4b60 0x7fbf780cfaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:55.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.129+0000 7fbf75400640 1 -- 172.21.15.96:0/3297334581 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4063970195 0 0) 0x7fbf5c003520 con 0x7fbf780d04a0 2024-05-28T06:17:55.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.129+0000 7fbf75400640 1 -- 172.21.15.96:0/3297334581 >> v2:172.21.15.123:3300/0 conn(0x7fbf780b4b60 msgr2=0x7fbf780cfaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:55.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.129+0000 7fbf75400640 1 --2- 172.21.15.96:0/3297334581 >> v2:172.21.15.123:3300/0 conn(0x7fbf780b4b60 0x7fbf780cfaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:55.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.129+0000 7fbf75400640 1 -- 172.21.15.96:0/3297334581 >> v1:172.21.15.123:6789/0 conn(0x7fbf780b3b70 legacy=0x7fbf7815ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:55.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.129+0000 7fbf75400640 1 -- 172.21.15.96:0/3297334581 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf780cefc0 con 0x7fbf780d04a0 2024-05-28T06:17:55.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.129+0000 7fbf75400640 1 -- 172.21.15.96:0/3297334581 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7fbf60001840 con 0x7fbf780d04a0 2024-05-28T06:17:55.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.129+0000 7fbf75400640 1 -- 172.21.15.96:0/3297334581 >> v1:172.21.15.96:6789/0 conn(0x7fbf780d04a0 legacy=0x7fbf780b3460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:55.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.129+0000 7fbf75400640 1 --2- 172.21.15.96:0/3297334581 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbf5c0039b0 0x7fbf5c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:55.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.129+0000 7fbf75400640 1 --2- 172.21.15.96:0/3297334581 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbf5c004800 0x7fbf5c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:55.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.129+0000 7fbf75400640 1 --2- 172.21.15.96:0/3297334581 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf5c0051c0 0x7fbf5c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:55.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.129+0000 7fbf75400640 1 -- 172.21.15.96:0/3297334581 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fbf7815fbb0 con 0x7fbf5c0051c0 2024-05-28T06:17:55.131 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.129+0000 7fbf75400640 1 -- 172.21.15.96:0/3297334581 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fbf5c00c110 con 0x7fbf5c004800 2024-05-28T06:17:55.131 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.129+0000 7fbf75400640 1 -- 172.21.15.96:0/3297334581 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fbf5c00c310 con 0x7fbf5c0039b0 2024-05-28T06:17:55.131 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.130+0000 7fbf76800640 1 --2- 172.21.15.96:0/3297334581 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf5c0051c0 0x7fbf5c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:55.131 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.130+0000 7fbf6d400640 1 --2- 172.21.15.96:0/3297334581 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbf5c004800 0x7fbf5c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:55.131 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.130+0000 7fbf75e00640 1 --2- 172.21.15.96:0/3297334581 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbf5c0039b0 0x7fbf5c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:55.131 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.130+0000 7fbf76800640 1 -- 172.21.15.96:0/3297334581 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbf5c0039b0 msgr2=0x7fbf5c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:55.131 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.130+0000 7fbf76800640 1 --2- 172.21.15.96:0/3297334581 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbf5c0039b0 0x7fbf5c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:55.131 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.130+0000 7fbf76800640 1 -- 172.21.15.96:0/3297334581 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbf5c004800 msgr2=0x7fbf5c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:55.131 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.130+0000 7fbf76800640 1 --2- 172.21.15.96:0/3297334581 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbf5c004800 0x7fbf5c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:55.131 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.130+0000 7fbf76800640 1 -- 172.21.15.96:0/3297334581 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf5c003520 con 0x7fbf5c0051c0 2024-05-28T06:17:55.131 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.130+0000 7fbf75e00640 1 --2- 172.21.15.96:0/3297334581 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbf5c0039b0 0x7fbf5c003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:17:55.133 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.131+0000 7fbf76800640 1 -- 172.21.15.96:0/3297334581 reap_dead start 2024-05-28T06:17:55.133 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.131+0000 7fbf76800640 1 --2- 172.21.15.96:0/3297334581 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf5c0051c0 0x7fbf5c00b9c0 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7fbf70001e50 tx=0x7fbf5c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d57611351efbbf2 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:55.133 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.131+0000 7fbf75400640 1 -- 172.21.15.96:0/3297334581 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf70001a70 con 0x7fbf5c0051c0 2024-05-28T06:17:55.133 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.131+0000 7fbf75400640 1 -- 172.21.15.96:0/3297334581 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf70003970 con 0x7fbf5c0051c0 2024-05-28T06:17:55.133 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.131+0000 7fbf75400640 1 -- 172.21.15.96:0/3297334581 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf70003d60 con 0x7fbf5c0051c0 2024-05-28T06:17:55.133 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.132+0000 7fbf7d000640 1 -- 172.21.15.96:0/3297334581 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf5c0051c0 msgr2=0x7fbf5c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:55.133 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.132+0000 7fbf7d000640 1 --2- 172.21.15.96:0/3297334581 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf5c0051c0 0x7fbf5c00b9c0 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7fbf70001e50 tx=0x7fbf5c005830 comp rx=0 tx=0).stop 2024-05-28T06:17:55.133 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.132+0000 7fbf7d000640 1 -- 172.21.15.96:0/3297334581 shutdown_connections 2024-05-28T06:17:55.133 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.132+0000 7fbf7d000640 1 --2- 172.21.15.96:0/3297334581 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf5c0051c0 0x7fbf5c00b9c0 unknown :-1 s=CLOSED pgs=1325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:55.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.132+0000 7fbf7d000640 1 -- 172.21.15.96:0/3297334581 >> 172.21.15.96:0/3297334581 conn(0x7fbf7806bae0 msgr2=0x7fbf7806bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:55.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.132+0000 7fbf7d000640 1 -- 172.21.15.96:0/3297334581 shutdown_connections 2024-05-28T06:17:55.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.132+0000 7fbf7d000640 1 -- 172.21.15.96:0/3297334581 wait complete. 2024-05-28T06:17:55.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.134+0000 7fbf7d000640 1 Processor -- start 2024-05-28T06:17:55.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.134+0000 7fbf7d000640 1 -- start start 2024-05-28T06:17:55.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.135+0000 7fbf7d000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbf5c0051c0 0x7fbf78099a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:55.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.135+0000 7fbf7d000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf780c3d70 0x7fbf780c41f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:55.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.135+0000 7fbf7d000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbf780adb40 0x7fbf780adfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:55.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.135+0000 7fbf7d000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fbf780d0930 con 0x7fbf780c3d70 2024-05-28T06:17:55.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.135+0000 7fbf7d000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fbf780cf580 con 0x7fbf780adb40 2024-05-28T06:17:55.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.135+0000 7fbf7d000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fbf780d0ab0 con 0x7fbf5c0051c0 2024-05-28T06:17:55.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.135+0000 7fbf6d400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf780c3d70 0x7fbf780c41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:55.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.135+0000 7fbf6d400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf780c3d70 0x7fbf780c41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:32954/0 (socket says 172.21.15.96:32954) 2024-05-28T06:17:55.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.135+0000 7fbf6d400640 1 -- 172.21.15.96:0/2945584638 learned_addr learned my addr 172.21.15.96:0/2945584638 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:17:55.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.136+0000 7fbf75e00640 1 --2- 172.21.15.96:0/2945584638 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbf5c0051c0 0x7fbf78099a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:55.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.136+0000 7fbf6d400640 1 -- 172.21.15.96:0/2945584638 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbf5c0051c0 msgr2=0x7fbf78099a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:55.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.136+0000 7fbf76800640 1 --2- 172.21.15.96:0/2945584638 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbf780adb40 0x7fbf780adfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:55.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.136+0000 7fbf6d400640 1 --2- 172.21.15.96:0/2945584638 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbf5c0051c0 0x7fbf78099a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:55.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.136+0000 7fbf6d400640 1 -- 172.21.15.96:0/2945584638 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbf780adb40 msgr2=0x7fbf780adfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:55.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.136+0000 7fbf6d400640 1 --2- 172.21.15.96:0/2945584638 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbf780adb40 0x7fbf780adfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:55.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.136+0000 7fbf6d400640 1 -- 172.21.15.96:0/2945584638 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf78156830 con 0x7fbf780c3d70 2024-05-28T06:17:55.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.136+0000 7fbf6d400640 1 --2- 172.21.15.96:0/2945584638 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf780c3d70 0x7fbf780c41f0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7fbf6000e7e0 tx=0x7fbf6000eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:55.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.136+0000 7fbf6ea00640 1 -- 172.21.15.96:0/2945584638 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf6000ee90 con 0x7fbf780c3d70 2024-05-28T06:17:55.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.136+0000 7fbf6ea00640 1 -- 172.21.15.96:0/2945584638 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf60018070 con 0x7fbf780c3d70 2024-05-28T06:17:55.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.136+0000 7fbf6ea00640 1 -- 172.21.15.96:0/2945584638 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf6000cab0 con 0x7fbf780c3d70 2024-05-28T06:17:55.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.136+0000 7fbf7d000640 1 -- 172.21.15.96:0/2945584638 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf780ae720 con 0x7fbf780c3d70 2024-05-28T06:17:55.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.136+0000 7fbf7d000640 1 -- 172.21.15.96:0/2945584638 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf7815bc40 con 0x7fbf780c3d70 2024-05-28T06:17:55.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.138+0000 7fbf7d000640 1 -- 172.21.15.96:0/2945584638 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf780689e0 con 0x7fbf780c3d70 2024-05-28T06:17:55.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.138+0000 7fbf6ea00640 1 -- 172.21.15.96:0/2945584638 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 30) ==== 92621+0+0 (secure 0 0 0) 0x7fbf6000cc50 con 0x7fbf780c3d70 2024-05-28T06:17:55.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.139+0000 7fbf6ea00640 1 --2- 172.21.15.96:0/2945584638 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbf5406dc30 0x7fbf540700f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:55.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.139+0000 7fbf6ea00640 1 -- 172.21.15.96:0/2945584638 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 5823+0+0 (secure 0 0 0) 0x7fbf6008a5d0 con 0x7fbf780c3d70 2024-05-28T06:17:55.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.140+0000 7fbf75e00640 1 --2- 172.21.15.96:0/2945584638 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbf5406dc30 0x7fbf540700f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:55.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.141+0000 7fbf75e00640 1 --2- 172.21.15.96:0/2945584638 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbf5406dc30 0x7fbf540700f0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fbf780af440 tx=0x7fbf68008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:55.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.144+0000 7fbf6ea00640 1 -- 172.21.15.96:0/2945584638 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbf780ced40 con 0x7fbf780c3d70 2024-05-28T06:17:55.383 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.385+0000 7fbf7d000640 1 -- 172.21.15.96:0/2945584638 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbf7806bf30 con 0x7fbf780c3d70 2024-05-28T06:17:55.384 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.386+0000 7fbf6ea00640 1 -- 172.21.15.96:0/2945584638 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) ==== 94+0+5177 (secure 0 0 0) 0x7fbf6005a100 con 0x7fbf780c3d70 2024-05-28T06:17:55.385 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:17:55.385 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":44,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5291,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6834/3727695025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":3727695025},{"type":"v1","addr":"172.21.15.96:6834","nonce":3727695025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38},{"gid":5300,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/324418606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6835","nonce":324418606},{"type":"v1","addr":"172.21.15.123:6837","nonce":324418606}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38},{"gid":5725,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/417566829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6833","nonce":417566829},{"type":"v1","addr":"172.21.15.96:6835","nonce":417566829}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-28T06:14:52.322597+0000","modified":"2024-05-28T06:14:55.415769+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7146},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7146":{"gid":7146,"name":"d","rank":0,"incarnation":39,"state":"up:active","state_seq":3,"addr":"172.21.15.123:6836/76143454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":76143454},{"type":"v1","addr":"172.21.15.123:6836","nonce":76143454}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7146,"qdb_cluster":[7146]},"id":5}]} 2024-05-28T06:17:55.385 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 44 2024-05-28T06:17:55.387 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.389+0000 7fbf7d000640 1 -- 172.21.15.96:0/2945584638 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbf5406dc30 msgr2=0x7fbf540700f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:55.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.389+0000 7fbf7d000640 1 --2- 172.21.15.96:0/2945584638 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbf5406dc30 0x7fbf540700f0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fbf780af440 tx=0x7fbf68008040 comp rx=0 tx=0).stop 2024-05-28T06:17:55.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.389+0000 7fbf7d000640 1 -- 172.21.15.96:0/2945584638 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf780c3d70 msgr2=0x7fbf780c41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:55.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.389+0000 7fbf7d000640 1 --2- 172.21.15.96:0/2945584638 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf780c3d70 0x7fbf780c41f0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7fbf6000e7e0 tx=0x7fbf6000eca0 comp rx=0 tx=0).stop 2024-05-28T06:17:55.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.389+0000 7fbf7d000640 1 -- 172.21.15.96:0/2945584638 shutdown_connections 2024-05-28T06:17:55.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.389+0000 7fbf7d000640 1 --2- 172.21.15.96:0/2945584638 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbf5406dc30 0x7fbf540700f0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:55.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.389+0000 7fbf7d000640 1 --2- 172.21.15.96:0/2945584638 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbf780adb40 0x7fbf780adfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:55.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.389+0000 7fbf7d000640 1 --2- 172.21.15.96:0/2945584638 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf780c3d70 0x7fbf780c41f0 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:55.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.389+0000 7fbf7d000640 1 --2- 172.21.15.96:0/2945584638 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbf5c0051c0 0x7fbf78099a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:55.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.389+0000 7fbf7d000640 1 -- 172.21.15.96:0/2945584638 >> 172.21.15.96:0/2945584638 conn(0x7fbf7806bae0 msgr2=0x7fbf7815fe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:55.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.389+0000 7fbf7d000640 1 -- 172.21.15.96:0/2945584638 shutdown_connections 2024-05-28T06:17:55.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.389+0000 7fbf7d000640 1 -- 172.21.15.96:0/2945584638 wait complete. 2024-05-28T06:17:55.407 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:17:55.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.609+0000 7f22fc600640 1 Processor -- start 2024-05-28T06:17:55.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.609+0000 7f22fc600640 1 -- start start 2024-05-28T06:17:55.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.611+0000 7f22fc600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f22f40d04a0 0x7f22f40d0880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:55.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.611+0000 7f22fc600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f22f406f6e0 0x7f22f40cd4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:55.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.611+0000 7f22fc600640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22f40b4350 con 0x7f22f4162e70 2024-05-28T06:17:55.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.611+0000 7f22fc600640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f22f40b3ec0 con 0x7f22f40d04a0 2024-05-28T06:17:55.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.611+0000 7f22fc600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f22f40b32d0 con 0x7f22f406f6e0 2024-05-28T06:17:55.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.611+0000 7f22f9c00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f22f40d04a0 0x7f22f40d0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:55.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.611+0000 7f22fa600640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f22f4162e70 0x7f22f406e0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:59898/0 (socket says 172.21.15.96:59898) 2024-05-28T06:17:55.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.611+0000 7f22fa600640 1 -- 172.21.15.96:0/3443846553 learned_addr learned my addr 172.21.15.96:0/3443846553 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:17:55.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.612+0000 7f22f9c00640 1 -- 172.21.15.96:0/3443846553 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f22f406f6e0 msgr2=0x7f22f40cd4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:55.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.612+0000 7f22f9200640 1 --2- 172.21.15.96:0/3443846553 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f22f406f6e0 0x7f22f40cd4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:55.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.612+0000 7f22f3e00640 1 -- 172.21.15.96:0/3443846553 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1385239739 0 0) 0x7f22f40b4350 con 0x7f22f4162e70 2024-05-28T06:17:55.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.612+0000 7f22f9c00640 1 --2- 172.21.15.96:0/3443846553 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f22f406f6e0 0x7f22f40cd4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:55.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.612+0000 7f22f9c00640 1 -- 172.21.15.96:0/3443846553 >> v1:172.21.15.96:6789/0 conn(0x7f22f4162e70 legacy=0x7f22f406e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:55.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.612+0000 7f22f9c00640 1 -- 172.21.15.96:0/3443846553 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22f40b3590 con 0x7f22f40d04a0 2024-05-28T06:17:55.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.612+0000 7f22f9200640 1 --2- 172.21.15.96:0/3443846553 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f22f406f6e0 0x7f22f40cd4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:17:55.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.613+0000 7f22f9c00640 1 --2- 172.21.15.96:0/3443846553 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f22f40d04a0 0x7f22f40d0880 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f22e80097a0 tx=0x7f22e802a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a779ce3f3dc29430 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:55.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.613+0000 7f22f3e00640 1 -- 172.21.15.96:0/3443846553 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22e8037070 con 0x7f22f40d04a0 2024-05-28T06:17:55.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.613+0000 7f22f3e00640 1 -- 172.21.15.96:0/3443846553 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22e8032440 con 0x7f22f40d04a0 2024-05-28T06:17:55.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.613+0000 7f22f3e00640 1 -- 172.21.15.96:0/3443846553 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22e8032810 con 0x7f22f40d04a0 2024-05-28T06:17:55.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.614+0000 7f22fc600640 1 -- 172.21.15.96:0/3443846553 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f22f40d04a0 msgr2=0x7f22f40d0880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:55.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.614+0000 7f22fc600640 1 --2- 172.21.15.96:0/3443846553 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f22f40d04a0 0x7f22f40d0880 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f22e80097a0 tx=0x7f22e802a180 comp rx=0 tx=0).stop 2024-05-28T06:17:55.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.614+0000 7f22fc600640 1 -- 172.21.15.96:0/3443846553 shutdown_connections 2024-05-28T06:17:55.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.614+0000 7f22fc600640 1 --2- 172.21.15.96:0/3443846553 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f22f406f6e0 0x7f22f40cd4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:55.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.614+0000 7f22fc600640 1 --2- 172.21.15.96:0/3443846553 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f22f40d04a0 0x7f22f40d0880 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:55.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.614+0000 7f22fc600640 1 -- 172.21.15.96:0/3443846553 >> 172.21.15.96:0/3443846553 conn(0x7f22f406bae0 msgr2=0x7f22f406bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:55.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.614+0000 7f22fc600640 1 -- 172.21.15.96:0/3443846553 shutdown_connections 2024-05-28T06:17:55.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.614+0000 7f22fc600640 1 -- 172.21.15.96:0/3443846553 wait complete. 2024-05-28T06:17:55.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.616+0000 7f22fc600640 1 Processor -- start 2024-05-28T06:17:55.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.616+0000 7f22fc600640 1 -- start start 2024-05-28T06:17:55.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.617+0000 7f22fc600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f22f406f6e0 0x7f22f40c5f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:55.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.617+0000 7f22fc600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f22f40d04a0 0x7f22f4165510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:55.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.617+0000 7f22fc600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f22f4162e70 0x7f22f415a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:55.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.617+0000 7f22fc600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f22f40d0dc0 con 0x7f22f40d04a0 2024-05-28T06:17:55.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.617+0000 7f22fc600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f22f40b3b50 con 0x7f22f4162e70 2024-05-28T06:17:55.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.617+0000 7f22fc600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f22f40cda30 con 0x7f22f406f6e0 2024-05-28T06:17:55.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.617+0000 7f22f9200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f22f40d04a0 0x7f22f4165510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:55.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.617+0000 7f22f9200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f22f40d04a0 0x7f22f4165510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:32958/0 (socket says 172.21.15.96:32958) 2024-05-28T06:17:55.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.617+0000 7f22f9200640 1 -- 172.21.15.96:0/434985081 learned_addr learned my addr 172.21.15.96:0/434985081 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:17:55.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.617+0000 7f22f9c00640 1 --2- 172.21.15.96:0/434985081 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f22f406f6e0 0x7f22f40c5f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:55.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.618+0000 7f22fa600640 1 --2- 172.21.15.96:0/434985081 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f22f4162e70 0x7f22f415a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:55.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.618+0000 7f22f9200640 1 -- 172.21.15.96:0/434985081 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f22f406f6e0 msgr2=0x7f22f40c5f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:55.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.618+0000 7f22f9200640 1 --2- 172.21.15.96:0/434985081 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f22f406f6e0 0x7f22f40c5f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:55.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.618+0000 7f22f9200640 1 -- 172.21.15.96:0/434985081 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f22f4162e70 msgr2=0x7f22f415a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:55.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.618+0000 7f22f9200640 1 --2- 172.21.15.96:0/434985081 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f22f4162e70 0x7f22f415a330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:55.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.618+0000 7f22f9200640 1 -- 172.21.15.96:0/434985081 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22f415e120 con 0x7f22f40d04a0 2024-05-28T06:17:55.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.618+0000 7f22f9200640 1 --2- 172.21.15.96:0/434985081 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f22f40d04a0 0x7f22f4165510 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f22e000c7f0 tx=0x7f22e000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:55.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.618+0000 7f22f1600640 1 -- 172.21.15.96:0/434985081 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22e0004230 con 0x7f22f40d04a0 2024-05-28T06:17:55.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.618+0000 7f22f1600640 1 -- 172.21.15.96:0/434985081 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22e00043d0 con 0x7f22f40d04a0 2024-05-28T06:17:55.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.618+0000 7f22f1600640 1 -- 172.21.15.96:0/434985081 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22e0007a70 con 0x7f22f40d04a0 2024-05-28T06:17:55.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.618+0000 7f22fc600640 1 -- 172.21.15.96:0/434985081 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22f40c6600 con 0x7f22f40d04a0 2024-05-28T06:17:55.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.618+0000 7f22fc600640 1 -- 172.21.15.96:0/434985081 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22f40c6bc0 con 0x7f22f40d04a0 2024-05-28T06:17:55.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.620+0000 7f22fc600640 1 -- 172.21.15.96:0/434985081 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22f40689e0 con 0x7f22f40d04a0 2024-05-28T06:17:55.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.621+0000 7f22f1600640 1 -- 172.21.15.96:0/434985081 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 30) ==== 92621+0+0 (secure 0 0 0) 0x7f22e0007c10 con 0x7f22f40d04a0 2024-05-28T06:17:55.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.622+0000 7f22f1600640 1 --2- 172.21.15.96:0/434985081 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f22d406da60 0x7f22d406ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:55.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.622+0000 7f22f1600640 1 -- 172.21.15.96:0/434985081 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 5823+0+0 (secure 0 0 0) 0x7f22e008aaf0 con 0x7f22f40d04a0 2024-05-28T06:17:55.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.623+0000 7f22f9c00640 1 --2- 172.21.15.96:0/434985081 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f22d406da60 0x7f22d406ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:55.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.624+0000 7f22f9c00640 1 --2- 172.21.15.96:0/434985081 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f22d406da60 0x7f22d406ff20 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f22e8034000 tx=0x7f22e802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:55.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.626+0000 7f22f1600640 1 -- 172.21.15.96:0/434985081 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f22f415e120 con 0x7f22f40d04a0 2024-05-28T06:17:55.826 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.828+0000 7f22fc600640 1 -- 172.21.15.96:0/434985081 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f22f4160ac0 con 0x7f22f40d04a0 2024-05-28T06:17:55.828 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.830+0000 7f22f1600640 1 -- 172.21.15.96:0/434985081 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) ==== 74+0+16887 (secure 0 0 0) 0x7f22e0015020 con 0x7f22f40d04a0 2024-05-28T06:17:55.829 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:17:55.829 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":78,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:17:51.459577+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-05-28T06:14:50.098829+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-05-28T06:14:51.318311+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":77,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"11.17","mappings":[{"from":6,"to":0}]},{"pgid":"11.28","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[{"pool":11,"snaps":[{"begin":2,"length":2}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:17:55.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.832+0000 7f22fc600640 1 -- 172.21.15.96:0/434985081 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f22d406da60 msgr2=0x7f22d406ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:55.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.832+0000 7f22fc600640 1 --2- 172.21.15.96:0/434985081 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f22d406da60 0x7f22d406ff20 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f22e8034000 tx=0x7f22e802e000 comp rx=0 tx=0).stop 2024-05-28T06:17:55.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.832+0000 7f22fc600640 1 -- 172.21.15.96:0/434985081 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f22f40d04a0 msgr2=0x7f22f4165510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:55.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.832+0000 7f22fc600640 1 --2- 172.21.15.96:0/434985081 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f22f40d04a0 0x7f22f4165510 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f22e000c7f0 tx=0x7f22e000ccb0 comp rx=0 tx=0).stop 2024-05-28T06:17:55.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.833+0000 7f22fc600640 1 -- 172.21.15.96:0/434985081 shutdown_connections 2024-05-28T06:17:55.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.833+0000 7f22fc600640 1 --2- 172.21.15.96:0/434985081 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f22d406da60 0x7f22d406ff20 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:55.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.833+0000 7f22fc600640 1 --2- 172.21.15.96:0/434985081 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f22f4162e70 0x7f22f415a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:55.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.833+0000 7f22fc600640 1 --2- 172.21.15.96:0/434985081 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f22f40d04a0 0x7f22f4165510 unknown :-1 s=CLOSED pgs=1329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:55.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.833+0000 7f22fc600640 1 --2- 172.21.15.96:0/434985081 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f22f406f6e0 0x7f22f40c5f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:55.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.833+0000 7f22fc600640 1 -- 172.21.15.96:0/434985081 >> 172.21.15.96:0/434985081 conn(0x7f22f406bae0 msgr2=0x7f22f415fca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:55.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.833+0000 7f22fc600640 1 -- 172.21.15.96:0/434985081 shutdown_connections 2024-05-28T06:17:55.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:55.833+0000 7f22fc600640 1 -- 172.21.15.96:0/434985081 wait complete. 2024-05-28T06:17:55.851 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-05-28T06:17:56.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.061+0000 7f3dcc000640 1 Processor -- start 2024-05-28T06:17:56.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.061+0000 7f3dcc000640 1 -- start start 2024-05-28T06:17:56.061 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.062+0000 7f3dcc000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f3dc40b4e10 0x7f3dc40cde30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:56.061 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.062+0000 7f3dcc000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f3dc40ce370 0x7f3dc40d0590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:56.061 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.062+0000 7f3dcc000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3dc40d0ad0 0x7f3dc4099090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:56.062 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.062+0000 7f3dcc000640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f3dc40d0f40 con 0x7f3dc40b4e10 2024-05-28T06:17:56.062 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.063+0000 7f3dcc000640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f3dc40b51f0 con 0x7f3dc40ce370 2024-05-28T06:17:56.062 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.063+0000 7f3dcc000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f3dc40ad1d0 con 0x7f3dc40d0ad0 2024-05-28T06:17:56.062 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.063+0000 7f3dc9600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f3dc40b4e10 0x7f3dc40cde30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:56.062 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.063+0000 7f3dc9600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f3dc40b4e10 0x7f3dc40cde30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:32960/0 (socket says 172.21.15.96:32960) 2024-05-28T06:17:56.062 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.063+0000 7f3dc9600640 1 -- 172.21.15.96:0/1984567359 learned_addr learned my addr 172.21.15.96:0/1984567359 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:17:56.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.063+0000 7f3dc8c00640 1 --2- 172.21.15.96:0/1984567359 >> v2:172.21.15.123:3300/0 conn(0x7f3dc40ce370 0x7f3dc40d0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:56.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.063+0000 7f3dca000640 1 --2- 172.21.15.96:0/1984567359 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3dc40d0ad0 0x7f3dc4099090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:56.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.063+0000 7f3dc9600640 1 -- 172.21.15.96:0/1984567359 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3dc40d0ad0 msgr2=0x7f3dc4099090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:56.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.063+0000 7f3dc9600640 1 --2- 172.21.15.96:0/1984567359 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3dc40d0ad0 0x7f3dc4099090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:56.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.063+0000 7f3dc9600640 1 -- 172.21.15.96:0/1984567359 >> v2:172.21.15.123:3300/0 conn(0x7f3dc40ce370 msgr2=0x7f3dc40d0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:56.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.063+0000 7f3dc9600640 1 --2- 172.21.15.96:0/1984567359 >> v2:172.21.15.123:3300/0 conn(0x7f3dc40ce370 0x7f3dc40d0590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:56.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.063+0000 7f3dc9600640 1 -- 172.21.15.96:0/1984567359 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dc40ad470 con 0x7f3dc40b4e10 2024-05-28T06:17:56.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.064+0000 7f3dc9600640 1 --2- 172.21.15.96:0/1984567359 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3dc40b4e10 0x7f3dc40cde30 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f3dc40ca860 tx=0x7f3db802c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e532dba8471e33cc server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:56.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.065+0000 7f3dc3e00640 1 -- 172.21.15.96:0/1984567359 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3db8032710 con 0x7f3dc40b4e10 2024-05-28T06:17:56.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.065+0000 7f3dc3e00640 1 -- 172.21.15.96:0/1984567359 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3db80328b0 con 0x7f3dc40b4e10 2024-05-28T06:17:56.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.065+0000 7f3dc3e00640 1 -- 172.21.15.96:0/1984567359 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3db8032ca0 con 0x7f3dc40b4e10 2024-05-28T06:17:56.065 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.065+0000 7f3dcc000640 1 -- 172.21.15.96:0/1984567359 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3dc40b4e10 msgr2=0x7f3dc40cde30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:56.065 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.065+0000 7f3dcc000640 1 --2- 172.21.15.96:0/1984567359 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3dc40b4e10 0x7f3dc40cde30 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f3dc40ca860 tx=0x7f3db802c810 comp rx=0 tx=0).stop 2024-05-28T06:17:56.065 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.065+0000 7f3dcc000640 1 -- 172.21.15.96:0/1984567359 shutdown_connections 2024-05-28T06:17:56.065 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.065+0000 7f3dcc000640 1 --2- 172.21.15.96:0/1984567359 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3dc40d0ad0 0x7f3dc4099090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:56.065 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.065+0000 7f3dcc000640 1 --2- 172.21.15.96:0/1984567359 >> v2:172.21.15.123:3300/0 conn(0x7f3dc40ce370 0x7f3dc40d0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:56.065 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.065+0000 7f3dcc000640 1 --2- 172.21.15.96:0/1984567359 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3dc40b4e10 0x7f3dc40cde30 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:56.065 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.065+0000 7f3dcc000640 1 -- 172.21.15.96:0/1984567359 >> 172.21.15.96:0/1984567359 conn(0x7f3dc406bae0 msgr2=0x7f3dc406bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:56.065 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.066+0000 7f3dcc000640 1 -- 172.21.15.96:0/1984567359 shutdown_connections 2024-05-28T06:17:56.065 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.066+0000 7f3dcc000640 1 -- 172.21.15.96:0/1984567359 wait complete. 2024-05-28T06:17:56.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.067+0000 7f3dcc000640 1 Processor -- start 2024-05-28T06:17:56.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.067+0000 7f3dcc000640 1 -- start start 2024-05-28T06:17:56.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.068+0000 7f3dcc000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3dc40b4e10 0x7f3dc4099a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:56.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.068+0000 7f3dcc000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3dc40ce370 0x7f3dc414f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:56.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.068+0000 7f3dcc000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3dc40d0ad0 0x7f3dc40cb740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:56.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.068+0000 7f3dcc000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f3dc40cea00 con 0x7f3dc40b4e10 2024-05-28T06:17:56.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.068+0000 7f3dcc000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f3dc40ce880 con 0x7f3dc40ce370 2024-05-28T06:17:56.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.068+0000 7f3dcc000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f3dc40b3150 con 0x7f3dc40d0ad0 2024-05-28T06:17:56.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.068+0000 7f3dc9600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3dc40b4e10 0x7f3dc4099a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:56.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.069+0000 7f3dc9600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3dc40b4e10 0x7f3dc4099a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:32970/0 (socket says 172.21.15.96:32970) 2024-05-28T06:17:56.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.069+0000 7f3dc9600640 1 -- 172.21.15.96:0/437507995 learned_addr learned my addr 172.21.15.96:0/437507995 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:17:56.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.069+0000 7f3dc8c00640 1 --2- 172.21.15.96:0/437507995 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3dc40ce370 0x7f3dc414f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:56.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.069+0000 7f3dc9600640 1 -- 172.21.15.96:0/437507995 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3dc40d0ad0 msgr2=0x7f3dc40cb740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:56.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.069+0000 7f3dc9600640 1 --2- 172.21.15.96:0/437507995 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3dc40d0ad0 0x7f3dc40cb740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:56.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.069+0000 7f3dc9600640 1 -- 172.21.15.96:0/437507995 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3dc40ce370 msgr2=0x7f3dc414f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:56.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.069+0000 7f3dc9600640 1 --2- 172.21.15.96:0/437507995 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3dc40ce370 0x7f3dc414f9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:56.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.069+0000 7f3dc9600640 1 -- 172.21.15.96:0/437507995 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dc40c4eb0 con 0x7f3dc40b4e10 2024-05-28T06:17:56.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.069+0000 7f3dc9600640 1 --2- 172.21.15.96:0/437507995 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3dc40b4e10 0x7f3dc4099a10 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7f3db802c850 tx=0x7f3db80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:56.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.070+0000 7f3dc1600640 1 -- 172.21.15.96:0/437507995 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3db8003810 con 0x7f3dc40b4e10 2024-05-28T06:17:56.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.070+0000 7f3dc1600640 1 -- 172.21.15.96:0/437507995 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3db8043070 con 0x7f3dc40b4e10 2024-05-28T06:17:56.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.070+0000 7f3dc1600640 1 -- 172.21.15.96:0/437507995 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3db8003c10 con 0x7f3dc40b4e10 2024-05-28T06:17:56.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.070+0000 7f3dcc000640 1 -- 172.21.15.96:0/437507995 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3dc40c5190 con 0x7f3dc40b4e10 2024-05-28T06:17:56.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.070+0000 7f3dcc000640 1 -- 172.21.15.96:0/437507995 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3dc40c5750 con 0x7f3dc40b4e10 2024-05-28T06:17:56.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.071+0000 7f3dcc000640 1 -- 172.21.15.96:0/437507995 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3dc40689e0 con 0x7f3dc40b4e10 2024-05-28T06:17:56.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.072+0000 7f3dc1600640 1 -- 172.21.15.96:0/437507995 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 30) ==== 92621+0+0 (secure 0 0 0) 0x7f3db803d070 con 0x7f3dc40b4e10 2024-05-28T06:17:56.071 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.073+0000 7f3dc1600640 1 --2- 172.21.15.96:0/437507995 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3da406da60 0x7f3da406ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:56.071 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.073+0000 7f3dc1600640 1 -- 172.21.15.96:0/437507995 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 5823+0+0 (secure 0 0 0) 0x7f3db80ab740 con 0x7f3dc40b4e10 2024-05-28T06:17:56.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.073+0000 7f3dc8c00640 1 --2- 172.21.15.96:0/437507995 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3da406da60 0x7f3da406ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:56.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.075+0000 7f3dc8c00640 1 --2- 172.21.15.96:0/437507995 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3da406da60 0x7f3da406ff20 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f3dc40ce0c0 tx=0x7f3db0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:56.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.078+0000 7f3dc1600640 1 -- 172.21.15.96:0/437507995 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3db8046360 con 0x7f3dc40b4e10 2024-05-28T06:17:56.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.319+0000 7f3dcc000640 1 -- 172.21.15.96:0/437507995 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f3dc415b840 con 0x7f3dc40b4e10 2024-05-28T06:17:56.319 INFO:tasks.ceph.mon.a.smithi096.stderr:2024-05-28T06:17:56.321+0000 7fb2fca00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-28T06:17:56.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.325+0000 7f3dc1600640 1 -- 172.21.15.96:0/437507995 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v45) ==== 161+0+0 (secure 0 0 0) 0x7f3db807b270 con 0x7f3dc40b4e10 2024-05-28T06:17:56.324 INFO:teuthology.orchestra.run.smithi096.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-28T06:17:56.327 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.328+0000 7f3dcc000640 1 -- 172.21.15.96:0/437507995 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3da406da60 msgr2=0x7f3da406ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:56.327 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.328+0000 7f3dcc000640 1 --2- 172.21.15.96:0/437507995 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3da406da60 0x7f3da406ff20 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f3dc40ce0c0 tx=0x7f3db0008040 comp rx=0 tx=0).stop 2024-05-28T06:17:56.327 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.328+0000 7f3dcc000640 1 -- 172.21.15.96:0/437507995 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3dc40b4e10 msgr2=0x7f3dc4099a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:56.327 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.328+0000 7f3dcc000640 1 --2- 172.21.15.96:0/437507995 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3dc40b4e10 0x7f3dc4099a10 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7f3db802c850 tx=0x7f3db80046c0 comp rx=0 tx=0).stop 2024-05-28T06:17:56.327 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.329+0000 7f3dcc000640 1 -- 172.21.15.96:0/437507995 shutdown_connections 2024-05-28T06:17:56.328 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.329+0000 7f3dcc000640 1 --2- 172.21.15.96:0/437507995 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3da406da60 0x7f3da406ff20 secure :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f3dc40ce0c0 tx=0x7f3db0008040 comp rx=0 tx=0).stop 2024-05-28T06:17:56.328 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.329+0000 7f3dcc000640 1 --2- 172.21.15.96:0/437507995 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3dc40d0ad0 0x7f3dc40cb740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:56.328 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.329+0000 7f3dcc000640 1 --2- 172.21.15.96:0/437507995 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3dc40ce370 0x7f3dc414f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:56.328 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.329+0000 7f3dcc000640 1 --2- 172.21.15.96:0/437507995 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3dc40b4e10 0x7f3dc4099a10 unknown :-1 s=CLOSED pgs=1331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:56.328 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.329+0000 7f3dcc000640 1 -- 172.21.15.96:0/437507995 >> 172.21.15.96:0/437507995 conn(0x7f3dc406bae0 msgr2=0x7f3dc40aee20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:56.328 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.329+0000 7f3dcc000640 1 -- 172.21.15.96:0/437507995 shutdown_connections 2024-05-28T06:17:56.328 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.330+0000 7f3dcc000640 1 -- 172.21.15.96:0/437507995 wait complete. 2024-05-28T06:17:56.347 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-05-28T06:17:56.447 INFO:tasks.ceph.mds.d.smithi123.stdout:starting mds.d at 2024-05-28T06:17:56.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.545+0000 7feb03c00640 1 Processor -- start 2024-05-28T06:17:56.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.545+0000 7feb03c00640 1 -- start start 2024-05-28T06:17:56.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.546+0000 7feb03c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7feafc0d0640 0x7feafc0d0a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:56.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.546+0000 7feb03c00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feafc0ade90 con 0x7feafc0725e0 2024-05-28T06:17:56.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.546+0000 7feb03c00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feafc0d0f60 con 0x7feafc15b6f0 2024-05-28T06:17:56.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.546+0000 7feb03c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7feafc071c40 con 0x7feafc0d0640 2024-05-28T06:17:56.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.547+0000 7feb01c00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7feafc0725e0 0x7feafc0ae650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:59904/0 (socket says 172.21.15.96:59904) 2024-05-28T06:17:56.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.547+0000 7feb01c00640 1 -- 172.21.15.96:0/1245311994 learned_addr learned my addr 172.21.15.96:0/1245311994 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:17:56.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.547+0000 7feb01200640 1 --2- 172.21.15.96:0/1245311994 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7feafc0d0640 0x7feafc0d0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:56.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.547+0000 7feafb400640 1 -- 172.21.15.96:0/1245311994 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1603317480 0 0) 0x7feafc0ade90 con 0x7feafc0725e0 2024-05-28T06:17:56.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.547+0000 7feafb400640 1 -- 172.21.15.96:0/1245311994 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feae4005730 con 0x7feafc0725e0 2024-05-28T06:17:56.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.547+0000 7feafb400640 1 -- 172.21.15.96:0/1245311994 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 102128116 0 0) 0x7feae4005730 con 0x7feafc0725e0 2024-05-28T06:17:56.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.548+0000 7feafb400640 1 -- 172.21.15.96:0/1245311994 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7feafc0d0640 msgr2=0x7feafc0d0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:56.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.548+0000 7feafb400640 1 --2- 172.21.15.96:0/1245311994 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7feafc0d0640 0x7feafc0d0a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:56.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.548+0000 7feafb400640 1 -- 172.21.15.96:0/1245311994 >> v1:172.21.15.123:6789/0 conn(0x7feafc15b6f0 legacy=0x7feafc06e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:56.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.548+0000 7feafb400640 1 -- 172.21.15.96:0/1245311994 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feafc071e70 con 0x7feafc0725e0 2024-05-28T06:17:56.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.548+0000 7feafb400640 1 -- 172.21.15.96:0/1245311994 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7feaf40019f0 con 0x7feafc0725e0 2024-05-28T06:17:56.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.548+0000 7feafb400640 1 -- 172.21.15.96:0/1245311994 >> v1:172.21.15.96:6789/0 conn(0x7feafc0725e0 legacy=0x7feafc0ae650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:56.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.548+0000 7feafb400640 1 --2- 172.21.15.96:0/1245311994 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feae4005bf0 0x7feae4006010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:56.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.548+0000 7feafb400640 1 --2- 172.21.15.96:0/1245311994 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feae4006a10 0x7feae4006e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:56.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.548+0000 7feafb400640 1 --2- 172.21.15.96:0/1245311994 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7feafc0d0640 0x7feae40073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:56.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.548+0000 7feafb400640 1 -- 172.21.15.96:0/1245311994 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7feae400dc80 con 0x7feae4005bf0 2024-05-28T06:17:56.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.548+0000 7feafb400640 1 -- 172.21.15.96:0/1245311994 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7feae400de00 con 0x7feae4006a10 2024-05-28T06:17:56.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.548+0000 7feafb400640 1 -- 172.21.15.96:0/1245311994 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7feae400e000 con 0x7feafc0d0640 2024-05-28T06:17:56.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.548+0000 7feb01200640 1 --2- 172.21.15.96:0/1245311994 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feae4005bf0 0x7feae4006010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:56.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.548+0000 7feb01c00640 1 --2- 172.21.15.96:0/1245311994 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7feafc0d0640 0x7feae40073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:56.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.548+0000 7feb01200640 1 -- 172.21.15.96:0/1245311994 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7feafc0d0640 msgr2=0x7feae40073d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:56.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.548+0000 7feb01200640 1 --2- 172.21.15.96:0/1245311994 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7feafc0d0640 0x7feae40073d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:56.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.548+0000 7feb01200640 1 -- 172.21.15.96:0/1245311994 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feae4006a10 msgr2=0x7feae4006e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:56.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.548+0000 7feb01200640 1 --2- 172.21.15.96:0/1245311994 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feae4006a10 0x7feae4006e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:56.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.548+0000 7feb01200640 1 -- 172.21.15.96:0/1245311994 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feae4005730 con 0x7feae4005bf0 2024-05-28T06:17:56.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.549+0000 7feb01200640 1 --2- 172.21.15.96:0/1245311994 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feae4005bf0 0x7feae4006010 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7feaf0009a90 tx=0x7feae4007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b76aab7d1a995276 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:56.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.550+0000 7feafb400640 1 -- 172.21.15.96:0/1245311994 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaf0036070 con 0x7feae4005bf0 2024-05-28T06:17:56.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.550+0000 7feafb400640 1 -- 172.21.15.96:0/1245311994 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feaf0031400 con 0x7feae4005bf0 2024-05-28T06:17:56.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.550+0000 7feafb400640 1 -- 172.21.15.96:0/1245311994 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaf0009690 con 0x7feae4005bf0 2024-05-28T06:17:56.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.550+0000 7feb03c00640 1 -- 172.21.15.96:0/1245311994 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feae4005bf0 msgr2=0x7feae4006010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:56.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.550+0000 7feb03c00640 1 --2- 172.21.15.96:0/1245311994 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feae4005bf0 0x7feae4006010 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7feaf0009a90 tx=0x7feae4007a40 comp rx=0 tx=0).stop 2024-05-28T06:17:56.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.550+0000 7feb01c00640 1 -- 172.21.15.96:0/1245311994 reap_dead start 2024-05-28T06:17:56.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.550+0000 7feb03c00640 1 -- 172.21.15.96:0/1245311994 shutdown_connections 2024-05-28T06:17:56.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.550+0000 7feb03c00640 1 -- 172.21.15.96:0/1245311994 >> 172.21.15.96:0/1245311994 conn(0x7feafc06bd20 msgr2=0x7feafc06c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:56.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.550+0000 7feb03c00640 1 -- 172.21.15.96:0/1245311994 shutdown_connections 2024-05-28T06:17:56.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.550+0000 7feb03c00640 1 -- 172.21.15.96:0/1245311994 wait complete. 2024-05-28T06:17:56.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.552+0000 7feb03c00640 1 Processor -- start 2024-05-28T06:17:56.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.552+0000 7feb03c00640 1 -- start start 2024-05-28T06:17:56.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.553+0000 7feb03c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7feae4005bf0 0x7feafc146e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:56.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.553+0000 7feb03c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feafc1466e0 0x7feafc15c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:56.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.553+0000 7feb03c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feafc15c990 0x7feafc159850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:56.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.553+0000 7feb03c00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7feafc0d0a20 con 0x7feafc1466e0 2024-05-28T06:17:56.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.553+0000 7feb03c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7feafc0ae130 con 0x7feafc15c990 2024-05-28T06:17:56.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.553+0000 7feb03c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7feafc0d0ba0 con 0x7feae4005bf0 2024-05-28T06:17:56.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.553+0000 7feafbe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feafc1466e0 0x7feafc15c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:56.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.553+0000 7feafbe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feafc1466e0 0x7feafc15c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:32982/0 (socket says 172.21.15.96:32982) 2024-05-28T06:17:56.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.553+0000 7feafbe00640 1 -- 172.21.15.96:0/2513569553 learned_addr learned my addr 172.21.15.96:0/2513569553 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:17:56.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.554+0000 7feb01200640 1 --2- 172.21.15.96:0/2513569553 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7feae4005bf0 0x7feafc146e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:56.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.554+0000 7feb01c00640 1 --2- 172.21.15.96:0/2513569553 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feafc15c990 0x7feafc159850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:56.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.554+0000 7feafbe00640 1 -- 172.21.15.96:0/2513569553 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7feae4005bf0 msgr2=0x7feafc146e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:56.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.554+0000 7feafbe00640 1 --2- 172.21.15.96:0/2513569553 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7feae4005bf0 0x7feafc146e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:56.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.554+0000 7feafbe00640 1 -- 172.21.15.96:0/2513569553 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feafc15c990 msgr2=0x7feafc159850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:56.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.554+0000 7feafbe00640 1 --2- 172.21.15.96:0/2513569553 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feafc15c990 0x7feafc159850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:56.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.554+0000 7feafbe00640 1 -- 172.21.15.96:0/2513569553 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feafc146b60 con 0x7feafc1466e0 2024-05-28T06:17:56.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.554+0000 7feb01c00640 1 --2- 172.21.15.96:0/2513569553 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feafc15c990 0x7feafc159850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:17:56.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.554+0000 7feafbe00640 1 --2- 172.21.15.96:0/2513569553 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feafc1466e0 0x7feafc15c450 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7feae800c500 tx=0x7feae800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:56.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.554+0000 7feaf8c00640 1 -- 172.21.15.96:0/2513569553 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feae80052d0 con 0x7feafc1466e0 2024-05-28T06:17:56.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.555+0000 7feaf8c00640 1 -- 172.21.15.96:0/2513569553 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feae8005470 con 0x7feafc1466e0 2024-05-28T06:17:56.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.555+0000 7feb03c00640 1 -- 172.21.15.96:0/2513569553 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feafc0cda60 con 0x7feafc1466e0 2024-05-28T06:17:56.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.555+0000 7feaf8c00640 1 -- 172.21.15.96:0/2513569553 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feae8012870 con 0x7feafc1466e0 2024-05-28T06:17:56.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.555+0000 7feb03c00640 1 -- 172.21.15.96:0/2513569553 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feafc0ce020 con 0x7feafc1466e0 2024-05-28T06:17:56.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.556+0000 7feb03c00640 1 -- 172.21.15.96:0/2513569553 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feafc0689e0 con 0x7feafc1466e0 2024-05-28T06:17:56.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.557+0000 7feaf8c00640 1 -- 172.21.15.96:0/2513569553 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 30) ==== 92621+0+0 (secure 0 0 0) 0x7feae8012a10 con 0x7feafc1466e0 2024-05-28T06:17:56.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.558+0000 7feaf8c00640 1 --2- 172.21.15.96:0/2513569553 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7feadc06da90 0x7feadc06ff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:56.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.558+0000 7feaf8c00640 1 -- 172.21.15.96:0/2513569553 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 5881+0+0 (secure 0 0 0) 0x7feae808a730 con 0x7feafc1466e0 2024-05-28T06:17:56.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.558+0000 7feb01200640 1 --2- 172.21.15.96:0/2513569553 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7feadc06da90 0x7feadc06ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:56.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.560+0000 7feb01200640 1 --2- 172.21.15.96:0/2513569553 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7feadc06da90 0x7feadc06ff50 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7feaf0033000 tx=0x7feaf0008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:56.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.563+0000 7feaf8c00640 1 -- 172.21.15.96:0/2513569553 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7feae805a2c0 con 0x7feafc1466e0 2024-05-28T06:17:56.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:56.801+0000 7feb03c00640 1 -- 172.21.15.96:0/2513569553 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7feafc0b0ba0 con 0x7feafc1466e0 2024-05-28T06:17:57.327 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:57.329+0000 7feaf8c00640 1 -- 172.21.15.96:0/2513569553 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v46) ==== 104+0+0 (secure 0 0 0) 0x7feae8059bc0 con 0x7feafc1466e0 2024-05-28T06:17:57.331 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:57.332+0000 7feb03c00640 1 -- 172.21.15.96:0/2513569553 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7feadc06da90 msgr2=0x7feadc06ff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:57.331 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:57.332+0000 7feb03c00640 1 --2- 172.21.15.96:0/2513569553 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7feadc06da90 0x7feadc06ff50 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7feaf0033000 tx=0x7feaf0008030 comp rx=0 tx=0).stop 2024-05-28T06:17:57.331 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:57.333+0000 7feb03c00640 1 -- 172.21.15.96:0/2513569553 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feafc1466e0 msgr2=0x7feafc15c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:57.331 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:57.333+0000 7feb03c00640 1 --2- 172.21.15.96:0/2513569553 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feafc1466e0 0x7feafc15c450 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7feae800c500 tx=0x7feae800c9c0 comp rx=0 tx=0).stop 2024-05-28T06:17:57.331 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:57.333+0000 7feb03c00640 1 -- 172.21.15.96:0/2513569553 shutdown_connections 2024-05-28T06:17:57.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:57.333+0000 7feb03c00640 1 --2- 172.21.15.96:0/2513569553 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7feadc06da90 0x7feadc06ff50 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:57.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:57.333+0000 7feb03c00640 1 --2- 172.21.15.96:0/2513569553 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feafc15c990 0x7feafc159850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:57.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:57.333+0000 7feb03c00640 1 --2- 172.21.15.96:0/2513569553 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feafc1466e0 0x7feafc15c450 unknown :-1 s=CLOSED pgs=1334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:57.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:57.333+0000 7feb03c00640 1 --2- 172.21.15.96:0/2513569553 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7feae4005bf0 0x7feafc146e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:57.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:57.333+0000 7feb03c00640 1 -- 172.21.15.96:0/2513569553 >> 172.21.15.96:0/2513569553 conn(0x7feafc06bd20 msgr2=0x7feafc0afe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:57.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:57.333+0000 7feb03c00640 1 -- 172.21.15.96:0/2513569553 shutdown_connections 2024-05-28T06:17:57.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:57.333+0000 7feb03c00640 1 -- 172.21.15.96:0/2513569553 wait complete. 2024-05-28T06:17:57.355 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-05-28T06:17:57.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:57.560+0000 7f87c1e00640 1 Processor -- start 2024-05-28T06:17:57.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:57.560+0000 7f87c1e00640 1 -- start start 2024-05-28T06:17:57.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:57.562+0000 7f87c1e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f87bc0cf2f0 0x7f87bc0cf910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:57.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:57.562+0000 7f87c1e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f87bc06edf0 0x7f87bc13fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:57.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:57.562+0000 7f87c1e00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87bc06dd50 con 0x7f87bc06f250 2024-05-28T06:17:57.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:57.562+0000 7f87c1e00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f87bc06e030 con 0x7f87bc0cf2f0 2024-05-28T06:17:57.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:57.562+0000 7f87c1e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f87bc06e310 con 0x7f87bc06edf0 2024-05-28T06:17:57.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:57.562+0000 7f87baa00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f87bc0cf2f0 0x7f87bc0cf910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:57.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:57.562+0000 7f87bb400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f87bc06f250 0x7f87bc0ad130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:59920/0 (socket says 172.21.15.96:59920) 2024-05-28T06:17:57.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:57.562+0000 7f87bb400640 1 -- 172.21.15.96:0/1472058592 learned_addr learned my addr 172.21.15.96:0/1472058592 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:17:57.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:57.563+0000 7f87b2000640 1 --2- 172.21.15.96:0/1472058592 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f87bc06edf0 0x7f87bc13fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:57.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:57.563+0000 7f87ba000640 1 -- 172.21.15.96:0/1472058592 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3456561503 0 0) 0x7f87bc06dd50 con 0x7f87bc06f250 2024-05-28T06:17:57.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:57.563+0000 7f87ba000640 1 -- 172.21.15.96:0/1472058592 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f87a0003520 con 0x7f87bc06f250 2024-05-28T06:17:57.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:57.563+0000 7f87baa00640 1 -- 172.21.15.96:0/1472058592 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f87bc06edf0 msgr2=0x7f87bc13fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:57.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:57.563+0000 7f87baa00640 1 --2- 172.21.15.96:0/1472058592 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f87bc06edf0 0x7f87bc13fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:57.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:57.563+0000 7f87baa00640 1 -- 172.21.15.96:0/1472058592 >> v1:172.21.15.96:6789/0 conn(0x7f87bc06f250 legacy=0x7f87bc0ad130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:57.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:57.563+0000 7f87baa00640 1 -- 172.21.15.96:0/1472058592 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87bc06e5f0 con 0x7f87bc0cf2f0 2024-05-28T06:17:57.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:57.564+0000 7f87baa00640 1 --2- 172.21.15.96:0/1472058592 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f87bc0cf2f0 0x7f87bc0cf910 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7f87ac009d90 tx=0x7f87ac02a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4df8a554c6ad600 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:57.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:57.564+0000 7f87ba000640 1 -- 172.21.15.96:0/1472058592 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87ac031600 con 0x7f87bc0cf2f0 2024-05-28T06:17:57.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:57.564+0000 7f87ba000640 1 -- 172.21.15.96:0/1472058592 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87ac0317a0 con 0x7f87bc0cf2f0 2024-05-28T06:17:57.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:57.564+0000 7f87ba000640 1 -- 172.21.15.96:0/1472058592 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87ac031b70 con 0x7f87bc0cf2f0 2024-05-28T06:17:57.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:57.565+0000 7f87c1e00640 1 -- 172.21.15.96:0/1472058592 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f87bc0cf2f0 msgr2=0x7f87bc0cf910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:57.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:57.565+0000 7f87c1e00640 1 --2- 172.21.15.96:0/1472058592 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f87bc0cf2f0 0x7f87bc0cf910 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7f87ac009d90 tx=0x7f87ac02a5d0 comp rx=0 tx=0).stop 2024-05-28T06:17:57.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:57.565+0000 7f87c1e00640 1 -- 172.21.15.96:0/1472058592 shutdown_connections 2024-05-28T06:17:57.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:57.565+0000 7f87c1e00640 1 --2- 172.21.15.96:0/1472058592 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f87bc06edf0 0x7f87bc13fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:57.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:57.565+0000 7f87c1e00640 1 --2- 172.21.15.96:0/1472058592 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f87bc0cf2f0 0x7f87bc0cf910 unknown :-1 s=CLOSED pgs=1336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:57.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:57.565+0000 7f87c1e00640 1 -- 172.21.15.96:0/1472058592 >> 172.21.15.96:0/1472058592 conn(0x7f87bc0b1e50 msgr2=0x7f87bc0b2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:57.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:57.565+0000 7f87c1e00640 1 -- 172.21.15.96:0/1472058592 shutdown_connections 2024-05-28T06:17:57.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:57.565+0000 7f87c1e00640 1 -- 172.21.15.96:0/1472058592 wait complete. 2024-05-28T06:17:57.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:57.567+0000 7f87c1e00640 1 Processor -- start 2024-05-28T06:17:57.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:57.567+0000 7f87c1e00640 1 -- start start 2024-05-28T06:17:57.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:57.568+0000 7f87c1e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f87bc06edf0 0x7f87bc159760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:57.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:57.568+0000 7f87c1e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f87bc06f250 0x7f87bc15d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:57.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:57.568+0000 7f87c1e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87bc0cf2f0 0x7f87bc15dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:57.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:57.568+0000 7f87c1e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f87bc1461a0 con 0x7f87bc06edf0 2024-05-28T06:17:57.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:57.568+0000 7f87c1e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f87bc146020 con 0x7f87bc0cf2f0 2024-05-28T06:17:57.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:57.568+0000 7f87c1e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f87bc0cfe50 con 0x7f87bc06f250 2024-05-28T06:17:57.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:57.568+0000 7f87baa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f87bc06edf0 0x7f87bc159760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:57.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:57.568+0000 7f87baa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f87bc06edf0 0x7f87bc159760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:33000/0 (socket says 172.21.15.96:33000) 2024-05-28T06:17:57.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:57.568+0000 7f87baa00640 1 -- 172.21.15.96:0/4101578283 learned_addr learned my addr 172.21.15.96:0/4101578283 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:17:57.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:57.569+0000 7f87bb400640 1 --2- 172.21.15.96:0/4101578283 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87bc0cf2f0 0x7f87bc15dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:57.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:57.569+0000 7f87b2000640 1 --2- 172.21.15.96:0/4101578283 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f87bc06f250 0x7f87bc15d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:57.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:57.569+0000 7f87baa00640 1 -- 172.21.15.96:0/4101578283 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f87bc06f250 msgr2=0x7f87bc15d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:57.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:57.569+0000 7f87baa00640 1 --2- 172.21.15.96:0/4101578283 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f87bc06f250 0x7f87bc15d720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:57.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:57.569+0000 7f87baa00640 1 -- 172.21.15.96:0/4101578283 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87bc0cf2f0 msgr2=0x7f87bc15dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:57.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:57.569+0000 7f87baa00640 1 --2- 172.21.15.96:0/4101578283 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87bc0cf2f0 0x7f87bc15dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:57.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:57.569+0000 7f87baa00640 1 -- 172.21.15.96:0/4101578283 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87bc159f90 con 0x7f87bc06edf0 2024-05-28T06:17:57.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:57.569+0000 7f87bb400640 1 --2- 172.21.15.96:0/4101578283 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87bc0cf2f0 0x7f87bc15dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:17:57.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:57.569+0000 7f87baa00640 1 --2- 172.21.15.96:0/4101578283 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f87bc06edf0 0x7f87bc159760 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7f87ac009d90 tx=0x7f87ac004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:57.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:57.569+0000 7f87b3e00640 1 -- 172.21.15.96:0/4101578283 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87ac040070 con 0x7f87bc06edf0 2024-05-28T06:17:57.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:57.569+0000 7f87b3e00640 1 -- 172.21.15.96:0/4101578283 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87ac030be0 con 0x7f87bc06edf0 2024-05-28T06:17:57.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:57.569+0000 7f87b3e00640 1 -- 172.21.15.96:0/4101578283 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87ac009450 con 0x7f87bc06edf0 2024-05-28T06:17:57.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:57.570+0000 7f87c1e00640 1 -- 172.21.15.96:0/4101578283 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87bc0b40c0 con 0x7f87bc06edf0 2024-05-28T06:17:57.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:57.570+0000 7f87c1e00640 1 -- 172.21.15.96:0/4101578283 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87bc0b4680 con 0x7f87bc06edf0 2024-05-28T06:17:57.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:57.571+0000 7f87c1e00640 1 -- 172.21.15.96:0/4101578283 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87bc0689e0 con 0x7f87bc06edf0 2024-05-28T06:17:57.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:57.572+0000 7f87b3e00640 1 -- 172.21.15.96:0/4101578283 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 30) ==== 92621+0+0 (secure 0 0 0) 0x7f87ac039040 con 0x7f87bc06edf0 2024-05-28T06:17:57.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:57.572+0000 7f87b3e00640 1 --2- 172.21.15.96:0/4101578283 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f879806da60 0x7f879806ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:57.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:57.573+0000 7f87b3e00640 1 -- 172.21.15.96:0/4101578283 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 5881+0+0 (secure 0 0 0) 0x7f87ac0ac170 con 0x7f87bc06edf0 2024-05-28T06:17:57.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:57.573+0000 7f87b2000640 1 --2- 172.21.15.96:0/4101578283 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f879806da60 0x7f879806ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:57.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:57.574+0000 7f87b2000640 1 --2- 172.21.15.96:0/4101578283 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f879806da60 0x7f879806ff20 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f87a4006fd0 tx=0x7f87a4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:57.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:57.577+0000 7f87b3e00640 1 -- 172.21.15.96:0/4101578283 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f87ac07bdc0 con 0x7f87bc06edf0 2024-05-28T06:17:57.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:57.779+0000 7f87c1e00640 1 -- 172.21.15.96:0/4101578283 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f87bc0b4d70 con 0x7f87bc06edf0 2024-05-28T06:17:58.513 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:58.514+0000 7f87b3e00640 1 -- 172.21.15.96:0/4101578283 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v80) ==== 181+0+0 (secure 0 0 0) 0x7f87ac07b6c0 con 0x7f87bc06edf0 2024-05-28T06:17:58.514 INFO:teuthology.orchestra.run.smithi096.stderr:pool 'cephfs_metadata' removed 2024-05-28T06:17:58.518 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:58.519+0000 7f87c1e00640 1 -- 172.21.15.96:0/4101578283 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f879806da60 msgr2=0x7f879806ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:58.518 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:58.519+0000 7f87c1e00640 1 --2- 172.21.15.96:0/4101578283 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f879806da60 0x7f879806ff20 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f87a4006fd0 tx=0x7f87a4008040 comp rx=0 tx=0).stop 2024-05-28T06:17:58.518 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:58.519+0000 7f87c1e00640 1 -- 172.21.15.96:0/4101578283 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f87bc06edf0 msgr2=0x7f87bc159760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:58.518 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:58.519+0000 7f87c1e00640 1 --2- 172.21.15.96:0/4101578283 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f87bc06edf0 0x7f87bc159760 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7f87ac009d90 tx=0x7f87ac004a40 comp rx=0 tx=0).stop 2024-05-28T06:17:58.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:58.520+0000 7f87c1e00640 1 -- 172.21.15.96:0/4101578283 shutdown_connections 2024-05-28T06:17:58.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:58.520+0000 7f87c1e00640 1 --2- 172.21.15.96:0/4101578283 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f879806da60 0x7f879806ff20 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:58.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:58.520+0000 7f87c1e00640 1 --2- 172.21.15.96:0/4101578283 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87bc0cf2f0 0x7f87bc15dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:58.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:58.520+0000 7f87c1e00640 1 --2- 172.21.15.96:0/4101578283 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f87bc06f250 0x7f87bc15d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:58.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:58.520+0000 7f87c1e00640 1 --2- 172.21.15.96:0/4101578283 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f87bc06edf0 0x7f87bc159760 unknown :-1 s=CLOSED pgs=1337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:58.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:58.520+0000 7f87c1e00640 1 -- 172.21.15.96:0/4101578283 >> 172.21.15.96:0/4101578283 conn(0x7f87bc0b1e50 msgr2=0x7f87bc15b810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:58.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:58.521+0000 7f87c1e00640 1 -- 172.21.15.96:0/4101578283 shutdown_connections 2024-05-28T06:17:58.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:58.522+0000 7f87c1e00640 1 -- 172.21.15.96:0/4101578283 wait complete. 2024-05-28T06:17:58.547 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-05-28T06:17:58.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:58.740+0000 7fc866200640 1 Processor -- start 2024-05-28T06:17:58.739 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:58.740+0000 7fc866200640 1 -- start start 2024-05-28T06:17:58.740 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:58.741+0000 7fc866200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fc86015ae60 0x7fc860071c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:58.740 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:58.741+0000 7fc866200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc8600af380 0x7fc8600b3150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:58.740 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:58.741+0000 7fc866200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8600721e0 con 0x7fc86006e440 2024-05-28T06:17:58.741 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:58.741+0000 7fc866200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fc8600adad0 con 0x7fc86015ae60 2024-05-28T06:17:58.741 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:58.741+0000 7fc866200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fc8600addb0 con 0x7fc8600af380 2024-05-28T06:17:58.741 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:58.742+0000 7fc856a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fc86015ae60 0x7fc860071c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:58.741 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:58.742+0000 7fc856a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fc86015ae60 0x7fc860071c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:33004/0 (socket says 172.21.15.96:33004) 2024-05-28T06:17:58.741 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:58.742+0000 7fc856a00640 1 -- 172.21.15.96:0/2827135404 learned_addr learned my addr 172.21.15.96:0/2827135404 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:17:58.742 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:58.742+0000 7fc856a00640 1 -- 172.21.15.96:0/2827135404 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc8600af380 msgr2=0x7fc8600b3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:58.742 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:58.742+0000 7fc856a00640 1 --2- 172.21.15.96:0/2827135404 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc8600af380 0x7fc8600b3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:58.742 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:58.742+0000 7fc856a00640 1 -- 172.21.15.96:0/2827135404 >> v1:172.21.15.123:6789/0 conn(0x7fc86006e440 legacy=0x7fc86006e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:58.742 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:58.742+0000 7fc856a00640 1 -- 172.21.15.96:0/2827135404 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8600ae090 con 0x7fc86015ae60 2024-05-28T06:17:58.742 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:58.743+0000 7fc856a00640 1 --2- 172.21.15.96:0/2827135404 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc86015ae60 0x7fc860071c00 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7fc84800bb00 tx=0x7fc84802c690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a02f5e3c0aeca2d8 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:58.742 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:58.743+0000 7fc85ea00640 1 -- 172.21.15.96:0/2827135404 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc84803a070 con 0x7fc86015ae60 2024-05-28T06:17:58.742 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:58.743+0000 7fc85ea00640 1 -- 172.21.15.96:0/2827135404 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc848032480 con 0x7fc86015ae60 2024-05-28T06:17:58.742 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:58.743+0000 7fc85ea00640 1 -- 172.21.15.96:0/2827135404 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc848032850 con 0x7fc86015ae60 2024-05-28T06:17:58.743 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:58.744+0000 7fc866200640 1 -- 172.21.15.96:0/2827135404 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc86015ae60 msgr2=0x7fc860071c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:58.744 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:58.744+0000 7fc866200640 1 --2- 172.21.15.96:0/2827135404 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc86015ae60 0x7fc860071c00 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7fc84800bb00 tx=0x7fc84802c690 comp rx=0 tx=0).stop 2024-05-28T06:17:58.744 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:58.744+0000 7fc866200640 1 -- 172.21.15.96:0/2827135404 shutdown_connections 2024-05-28T06:17:58.744 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:58.744+0000 7fc866200640 1 --2- 172.21.15.96:0/2827135404 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc8600af380 0x7fc8600b3150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:58.744 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:58.744+0000 7fc866200640 1 --2- 172.21.15.96:0/2827135404 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc86015ae60 0x7fc860071c00 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:58.744 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:58.744+0000 7fc866200640 1 -- 172.21.15.96:0/2827135404 >> 172.21.15.96:0/2827135404 conn(0x7fc8600b1e50 msgr2=0x7fc8600b2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:58.744 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:58.744+0000 7fc866200640 1 -- 172.21.15.96:0/2827135404 shutdown_connections 2024-05-28T06:17:58.744 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:58.744+0000 7fc866200640 1 -- 172.21.15.96:0/2827135404 wait complete. 2024-05-28T06:17:58.746 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:58.746+0000 7fc866200640 1 Processor -- start 2024-05-28T06:17:58.746 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:58.746+0000 7fc866200640 1 -- start start 2024-05-28T06:17:58.746 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:58.747+0000 7fc866200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc86006e440 0x7fc8601496e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:58.746 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:58.747+0000 7fc866200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc8600af380 0x7fc860159ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:58.746 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:58.747+0000 7fc866200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc86015ae60 0x7fc8600b0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:58.746 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:58.747+0000 7fc866200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc8600ad6f0 con 0x7fc86015ae60 2024-05-28T06:17:58.747 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:58.747+0000 7fc866200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc86006fa50 con 0x7fc86006e440 2024-05-28T06:17:58.747 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:58.747+0000 7fc866200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fc8600ad870 con 0x7fc8600af380 2024-05-28T06:17:58.747 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:58.747+0000 7fc85fe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc86015ae60 0x7fc8600b0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:58.747 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:58.747+0000 7fc85fe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc86015ae60 0x7fc8600b0dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:33008/0 (socket says 172.21.15.96:33008) 2024-05-28T06:17:58.747 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:58.747+0000 7fc85fe00640 1 -- 172.21.15.96:0/1682688422 learned_addr learned my addr 172.21.15.96:0/1682688422 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:17:58.747 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:58.748+0000 7fc85fe00640 1 -- 172.21.15.96:0/1682688422 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc8600af380 msgr2=0x7fc860159ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:58.747 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:58.748+0000 7fc856a00640 1 --2- 172.21.15.96:0/1682688422 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc8600af380 0x7fc860159ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:58.747 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:58.748+0000 7fc85fe00640 1 --2- 172.21.15.96:0/1682688422 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc8600af380 0x7fc860159ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:58.747 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:58.748+0000 7fc85fe00640 1 -- 172.21.15.96:0/1682688422 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc86006e440 msgr2=0x7fc8601496e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:58.747 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:58.748+0000 7fc85fe00640 1 --2- 172.21.15.96:0/1682688422 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc86006e440 0x7fc8601496e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:58.747 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:58.748+0000 7fc85fe00640 1 -- 172.21.15.96:0/1682688422 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8601551c0 con 0x7fc86015ae60 2024-05-28T06:17:58.748 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:58.748+0000 7fc85fe00640 1 --2- 172.21.15.96:0/1682688422 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc86015ae60 0x7fc8600b0dd0 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7fc85800b970 tx=0x7fc85800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:58.748 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:58.748+0000 7fc857e00640 1 -- 172.21.15.96:0/1682688422 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc858019070 con 0x7fc86015ae60 2024-05-28T06:17:58.748 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:58.748+0000 7fc857e00640 1 -- 172.21.15.96:0/1682688422 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc85800cb60 con 0x7fc86015ae60 2024-05-28T06:17:58.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:58.748+0000 7fc857e00640 1 -- 172.21.15.96:0/1682688422 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc858014440 con 0x7fc86015ae60 2024-05-28T06:17:58.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:58.748+0000 7fc866200640 1 -- 172.21.15.96:0/1682688422 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8601554a0 con 0x7fc86015ae60 2024-05-28T06:17:58.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:58.748+0000 7fc866200640 1 -- 172.21.15.96:0/1682688422 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc860155a60 con 0x7fc86015ae60 2024-05-28T06:17:58.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:58.750+0000 7fc866200640 1 -- 172.21.15.96:0/1682688422 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8600689e0 con 0x7fc86015ae60 2024-05-28T06:17:58.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:58.751+0000 7fc857e00640 1 -- 172.21.15.96:0/1682688422 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 30) ==== 92621+0+0 (secure 0 0 0) 0x7fc85801c070 con 0x7fc86015ae60 2024-05-28T06:17:58.750 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:58.751+0000 7fc857e00640 1 --2- 172.21.15.96:0/1682688422 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc83c06da60 0x7fc83c06ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:58.750 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:58.752+0000 7fc857e00640 1 -- 172.21.15.96:0/1682688422 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 5443+0+0 (secure 0 0 0) 0x7fc85808a7f0 con 0x7fc86015ae60 2024-05-28T06:17:58.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:58.752+0000 7fc85f400640 1 --2- 172.21.15.96:0/1682688422 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc83c06da60 0x7fc83c06ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:58.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:58.753+0000 7fc85f400640 1 --2- 172.21.15.96:0/1682688422 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc83c06da60 0x7fc83c06ff20 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fc850009650 tx=0x7fc850006ba0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:58.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:58.756+0000 7fc857e00640 1 -- 172.21.15.96:0/1682688422 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc85805a320 con 0x7fc86015ae60 2024-05-28T06:17:58.956 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:58.957+0000 7fc866200640 1 -- 172.21.15.96:0/1682688422 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fc860073180 con 0x7fc86015ae60 2024-05-28T06:17:59.525 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:59.526+0000 7fc857e00640 1 -- 172.21.15.96:0/1682688422 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v81) ==== 169+0+0 (secure 0 0 0) 0x7fc858059c20 con 0x7fc86015ae60 2024-05-28T06:17:59.525 INFO:teuthology.orchestra.run.smithi096.stderr:pool 'cephfs_data' removed 2024-05-28T06:17:59.528 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:59.530+0000 7fc866200640 1 -- 172.21.15.96:0/1682688422 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc83c06da60 msgr2=0x7fc83c06ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:59.528 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:59.530+0000 7fc866200640 1 --2- 172.21.15.96:0/1682688422 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc83c06da60 0x7fc83c06ff20 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fc850009650 tx=0x7fc850006ba0 comp rx=0 tx=0).stop 2024-05-28T06:17:59.528 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:59.530+0000 7fc866200640 1 -- 172.21.15.96:0/1682688422 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc86015ae60 msgr2=0x7fc8600b0dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:59.528 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:59.530+0000 7fc866200640 1 --2- 172.21.15.96:0/1682688422 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc86015ae60 0x7fc8600b0dd0 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7fc85800b970 tx=0x7fc85800be30 comp rx=0 tx=0).stop 2024-05-28T06:17:59.529 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:59.530+0000 7fc866200640 1 -- 172.21.15.96:0/1682688422 shutdown_connections 2024-05-28T06:17:59.529 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:59.530+0000 7fc866200640 1 --2- 172.21.15.96:0/1682688422 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc83c06da60 0x7fc83c06ff20 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:59.529 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:59.530+0000 7fc866200640 1 --2- 172.21.15.96:0/1682688422 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc86015ae60 0x7fc8600b0dd0 unknown :-1 s=CLOSED pgs=1339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:59.529 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:59.530+0000 7fc866200640 1 --2- 172.21.15.96:0/1682688422 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc8600af380 0x7fc860159ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:59.529 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:59.530+0000 7fc866200640 1 --2- 172.21.15.96:0/1682688422 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc86006e440 0x7fc8601496e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:59.529 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:59.530+0000 7fc866200640 1 -- 172.21.15.96:0/1682688422 >> 172.21.15.96:0/1682688422 conn(0x7fc8600b1e50 msgr2=0x7fc86015f1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:59.530 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:59.531+0000 7fc866200640 1 -- 172.21.15.96:0/1682688422 shutdown_connections 2024-05-28T06:17:59.530 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:59.531+0000 7fc866200640 1 -- 172.21.15.96:0/1682688422 wait complete. 2024-05-28T06:17:59.554 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/volumes/snapshot_clone_no_wait 2024-05-28T06:17:59.750 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:59.751+0000 7fd043600640 1 Processor -- start 2024-05-28T06:17:59.750 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:59.751+0000 7fd043600640 1 -- start start 2024-05-28T06:17:59.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:59.752+0000 7fd043600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd03c0d0640 0x7fd03c15d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:59.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:59.752+0000 7fd043600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fd03c0d0a20 0x7fd03c0cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:59.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:59.752+0000 7fd043600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd03c15cb10 con 0x7fd03c0b32c0 2024-05-28T06:17:59.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:59.752+0000 7fd043600640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fd03c0adbb0 con 0x7fd03c0d0a20 2024-05-28T06:17:59.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:59.752+0000 7fd043600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fd03c15db10 con 0x7fd03c0d0640 2024-05-28T06:17:59.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:59.753+0000 7fd03be00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fd03c0d0a20 0x7fd03c0cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:59.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:59.753+0000 7fd03be00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fd03c0d0a20 0x7fd03c0cf950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:60124/0 (socket says 172.21.15.96:60124) 2024-05-28T06:17:59.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:59.753+0000 7fd03be00640 1 -- 172.21.15.96:0/2252423671 learned_addr learned my addr 172.21.15.96:0/2252423671 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:17:59.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:59.753+0000 7fd040c00640 1 --2- 172.21.15.96:0/2252423671 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd03c0d0640 0x7fd03c15d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:59.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:59.753+0000 7fd03be00640 1 -- 172.21.15.96:0/2252423671 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd03c0d0640 msgr2=0x7fd03c15d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:59.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:59.753+0000 7fd03be00640 1 --2- 172.21.15.96:0/2252423671 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd03c0d0640 0x7fd03c15d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:59.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:59.753+0000 7fd03be00640 1 -- 172.21.15.96:0/2252423671 >> v1:172.21.15.123:6789/0 conn(0x7fd03c0b32c0 legacy=0x7fd03c0b3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:17:59.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:59.753+0000 7fd03be00640 1 -- 172.21.15.96:0/2252423671 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd03c0ad5f0 con 0x7fd03c0d0a20 2024-05-28T06:17:59.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:59.753+0000 7fd040c00640 1 --2- 172.21.15.96:0/2252423671 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd03c0d0640 0x7fd03c15d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:17:59.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:59.754+0000 7fd03be00640 1 --2- 172.21.15.96:0/2252423671 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd03c0d0a20 0x7fd03c0cf950 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7fd028009870 tx=0x7fd02802a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dac56f9a4531c30e server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:59.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:59.755+0000 7fd03b400640 1 -- 172.21.15.96:0/2252423671 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd028037070 con 0x7fd03c0d0a20 2024-05-28T06:17:59.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:59.755+0000 7fd03b400640 1 -- 172.21.15.96:0/2252423671 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd028032440 con 0x7fd03c0d0a20 2024-05-28T06:17:59.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:59.755+0000 7fd03b400640 1 -- 172.21.15.96:0/2252423671 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd028032810 con 0x7fd03c0d0a20 2024-05-28T06:17:59.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:59.755+0000 7fd043600640 1 -- 172.21.15.96:0/2252423671 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd03c0d0a20 msgr2=0x7fd03c0cf950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:59.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:59.755+0000 7fd043600640 1 --2- 172.21.15.96:0/2252423671 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd03c0d0a20 0x7fd03c0cf950 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7fd028009870 tx=0x7fd02802a210 comp rx=0 tx=0).stop 2024-05-28T06:17:59.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:59.755+0000 7fd043600640 1 -- 172.21.15.96:0/2252423671 shutdown_connections 2024-05-28T06:17:59.755 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:59.755+0000 7fd043600640 1 --2- 172.21.15.96:0/2252423671 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd03c0d0a20 0x7fd03c0cf950 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:59.755 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:59.755+0000 7fd043600640 1 --2- 172.21.15.96:0/2252423671 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd03c0d0640 0x7fd03c15d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:59.755 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:59.755+0000 7fd043600640 1 -- 172.21.15.96:0/2252423671 >> 172.21.15.96:0/2252423671 conn(0x7fd03c06bd20 msgr2=0x7fd03c06c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:17:59.755 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:59.756+0000 7fd043600640 1 -- 172.21.15.96:0/2252423671 shutdown_connections 2024-05-28T06:17:59.755 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:59.756+0000 7fd043600640 1 -- 172.21.15.96:0/2252423671 wait complete. 2024-05-28T06:17:59.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:59.757+0000 7fd043600640 1 Processor -- start 2024-05-28T06:17:59.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:59.758+0000 7fd043600640 1 -- start start 2024-05-28T06:17:59.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:59.759+0000 7fd043600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd03c0b32c0 0x7fd03c06f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:59.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:59.759+0000 7fd043600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd03c0d0640 0x7fd03c14cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:59.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:59.759+0000 7fd043600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd03c0d0a20 0x7fd03c1f4b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:59.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:59.759+0000 7fd043600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd03c15d0b0 con 0x7fd03c0d0a20 2024-05-28T06:17:59.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:59.759+0000 7fd043600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd03c15cdd0 con 0x7fd03c0d0640 2024-05-28T06:17:59.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:59.759+0000 7fd043600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fd03c0b3e60 con 0x7fd03c0b32c0 2024-05-28T06:17:59.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:59.759+0000 7fd041600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd03c0d0a20 0x7fd03c1f4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:59.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:59.759+0000 7fd041600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd03c0d0a20 0x7fd03c1f4b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:60134/0 (socket says 172.21.15.96:60134) 2024-05-28T06:17:59.759 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:59.759+0000 7fd041600640 1 -- 172.21.15.96:0/3186529256 learned_addr learned my addr 172.21.15.96:0/3186529256 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:17:59.759 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:59.759+0000 7fd040c00640 1 --2- 172.21.15.96:0/3186529256 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd03c0b32c0 0x7fd03c06f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:59.759 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:59.759+0000 7fd041600640 1 -- 172.21.15.96:0/3186529256 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd03c0b32c0 msgr2=0x7fd03c06f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:59.759 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:59.759+0000 7fd03be00640 1 --2- 172.21.15.96:0/3186529256 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd03c0d0640 0x7fd03c14cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:59.759 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:59.759+0000 7fd041600640 1 --2- 172.21.15.96:0/3186529256 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd03c0b32c0 0x7fd03c06f1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:59.759 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:59.759+0000 7fd041600640 1 -- 172.21.15.96:0/3186529256 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd03c0d0640 msgr2=0x7fd03c14cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:17:59.759 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:59.759+0000 7fd041600640 1 --2- 172.21.15.96:0/3186529256 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd03c0d0640 0x7fd03c14cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:17:59.759 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:59.759+0000 7fd041600640 1 -- 172.21.15.96:0/3186529256 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd03c06e470 con 0x7fd03c0d0a20 2024-05-28T06:17:59.759 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:59.760+0000 7fd041600640 1 --2- 172.21.15.96:0/3186529256 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd03c0d0a20 0x7fd03c1f4b80 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7fd03400c780 tx=0x7fd03400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:59.760 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:59.760+0000 7fd038c00640 1 -- 172.21.15.96:0/3186529256 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd034004b60 con 0x7fd03c0d0a20 2024-05-28T06:17:59.760 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:59.760+0000 7fd038c00640 1 -- 172.21.15.96:0/3186529256 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd034004d00 con 0x7fd03c0d0a20 2024-05-28T06:17:59.760 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:59.760+0000 7fd038c00640 1 -- 172.21.15.96:0/3186529256 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd03400aa90 con 0x7fd03c0d0a20 2024-05-28T06:17:59.760 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:59.760+0000 7fd043600640 1 -- 172.21.15.96:0/3186529256 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd03c06e710 con 0x7fd03c0d0a20 2024-05-28T06:17:59.760 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:59.760+0000 7fd043600640 1 -- 172.21.15.96:0/3186529256 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd03c15d820 con 0x7fd03c0d0a20 2024-05-28T06:17:59.760 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:59.761+0000 7fd043600640 1 -- 172.21.15.96:0/3186529256 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd03c0689e0 con 0x7fd03c0d0a20 2024-05-28T06:17:59.761 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:59.762+0000 7fd038c00640 1 -- 172.21.15.96:0/3186529256 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 30) ==== 92621+0+0 (secure 0 0 0) 0x7fd03400ac30 con 0x7fd03c0d0a20 2024-05-28T06:17:59.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:59.763+0000 7fd038c00640 1 --2- 172.21.15.96:0/3186529256 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd01c06da60 0x7fd01c06ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:17:59.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:59.763+0000 7fd038c00640 1 -- 172.21.15.96:0/3186529256 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 4971+0+0 (secure 0 0 0) 0x7fd03408afc0 con 0x7fd03c0d0a20 2024-05-28T06:17:59.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:59.764+0000 7fd040c00640 1 --2- 172.21.15.96:0/3186529256 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd01c06da60 0x7fd01c06ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:17:59.763 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:59.765+0000 7fd040c00640 1 --2- 172.21.15.96:0/3186529256 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd01c06da60 0x7fd01c06ff20 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fd03c155f80 tx=0x7fd030008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:17:59.767 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:17:59.768+0000 7fd038c00640 1 -- 172.21.15.96:0/3186529256 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd03405aaf0 con 0x7fd03c0d0a20 2024-05-28T06:18:00.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.011+0000 7fd043600640 1 -- 172.21.15.96:0/3186529256 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/snapshot_clone_no_wait"} v 0) -- 0x7fd03c0726b0 con 0x7fd03c0d0a20 2024-05-28T06:18:00.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.017+0000 7fd038c00640 1 -- 172.21.15.96:0/3186529256 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/snapshot_clone_no_wait"}]=0 v20) ==== 117+0+0 (secure 0 0 0) 0x7fd03405a3f0 con 0x7fd03c0d0a20 2024-05-28T06:18:00.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.021+0000 7fd043600640 1 -- 172.21.15.96:0/3186529256 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd01c06da60 msgr2=0x7fd01c06ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:00.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.021+0000 7fd043600640 1 --2- 172.21.15.96:0/3186529256 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd01c06da60 0x7fd01c06ff20 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fd03c155f80 tx=0x7fd030008040 comp rx=0 tx=0).stop 2024-05-28T06:18:00.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.021+0000 7fd043600640 1 -- 172.21.15.96:0/3186529256 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd03c0d0a20 msgr2=0x7fd03c1f4b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:00.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.021+0000 7fd043600640 1 --2- 172.21.15.96:0/3186529256 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd03c0d0a20 0x7fd03c1f4b80 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7fd03400c780 tx=0x7fd03400cc40 comp rx=0 tx=0).stop 2024-05-28T06:18:00.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.021+0000 7fd043600640 1 -- 172.21.15.96:0/3186529256 shutdown_connections 2024-05-28T06:18:00.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.021+0000 7fd043600640 1 --2- 172.21.15.96:0/3186529256 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd01c06da60 0x7fd01c06ff20 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:00.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.021+0000 7fd043600640 1 --2- 172.21.15.96:0/3186529256 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd03c0d0a20 0x7fd03c1f4b80 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:00.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.021+0000 7fd043600640 1 --2- 172.21.15.96:0/3186529256 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd03c0d0640 0x7fd03c14cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:00.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.021+0000 7fd043600640 1 --2- 172.21.15.96:0/3186529256 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd03c0b32c0 0x7fd03c06f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:00.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.021+0000 7fd043600640 1 -- 172.21.15.96:0/3186529256 >> 172.21.15.96:0/3186529256 conn(0x7fd03c06bd20 msgr2=0x7fd03c154cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:18:00.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.021+0000 7fd043600640 1 -- 172.21.15.96:0/3186529256 shutdown_connections 2024-05-28T06:18:00.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.021+0000 7fd043600640 1 -- 172.21.15.96:0/3186529256 wait complete. 2024-05-28T06:18:00.040 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-05-28T06:18:00.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.219+0000 7f0dc3200640 1 Processor -- start 2024-05-28T06:18:00.218 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.219+0000 7f0dc3200640 1 -- start start 2024-05-28T06:18:00.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.220+0000 7f0dc3200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0dc4153510 0x7f0dc40b4d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:00.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.220+0000 7f0dc3200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0dc415d900 con 0x7f0dc40ccb90 2024-05-28T06:18:00.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.220+0000 7f0dc3200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0dc40b3f90 con 0x7f0dc415d270 2024-05-28T06:18:00.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.220+0000 7f0dc3200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f0dc41538f0 con 0x7f0dc4153510 2024-05-28T06:18:00.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.221+0000 7f0dc1400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f0dc40ccb90 0x7f0dc415cb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:48854/0 (socket says 172.21.15.96:48854) 2024-05-28T06:18:00.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.221+0000 7f0dc1400640 1 -- 172.21.15.96:0/3586305006 learned_addr learned my addr 172.21.15.96:0/3586305006 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:18:00.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.221+0000 7f0dc0a00640 1 -- 172.21.15.96:0/3586305006 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2422094163 0 0) 0x7f0dc415d900 con 0x7f0dc40ccb90 2024-05-28T06:18:00.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.221+0000 7f0dc0a00640 1 -- 172.21.15.96:0/3586305006 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0da8003520 con 0x7f0dc40ccb90 2024-05-28T06:18:00.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.221+0000 7f0dc1e00640 1 --2- 172.21.15.96:0/3586305006 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0dc4153510 0x7f0dc40b4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:00.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.221+0000 7f0dc0a00640 1 -- 172.21.15.96:0/3586305006 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 214804318 0 0) 0x7f0da8003520 con 0x7f0dc40ccb90 2024-05-28T06:18:00.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.221+0000 7f0dc0a00640 1 -- 172.21.15.96:0/3586305006 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0dc4153510 msgr2=0x7f0dc40b4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:18:00.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.221+0000 7f0dc0a00640 1 --2- 172.21.15.96:0/3586305006 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0dc4153510 0x7f0dc40b4d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:00.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.221+0000 7f0dc0a00640 1 -- 172.21.15.96:0/3586305006 >> v1:172.21.15.123:6789/0 conn(0x7f0dc415d270 legacy=0x7f0dc406e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:18:00.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.221+0000 7f0dc0a00640 1 -- 172.21.15.96:0/3586305006 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dc40b4270 con 0x7f0dc40ccb90 2024-05-28T06:18:00.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.221+0000 7f0dc0a00640 1 -- 172.21.15.96:0/3586305006 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f0db0001670 con 0x7f0dc40ccb90 2024-05-28T06:18:00.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.222+0000 7f0dc0a00640 1 -- 172.21.15.96:0/3586305006 >> v1:172.21.15.96:6789/0 conn(0x7f0dc40ccb90 legacy=0x7f0dc415cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:00.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.222+0000 7f0dc0a00640 1 --2- 172.21.15.96:0/3586305006 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0da80039e0 0x7f0da8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:00.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.222+0000 7f0dc0a00640 1 --2- 172.21.15.96:0/3586305006 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0da8004800 0x7f0da8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:00.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.222+0000 7f0dc0a00640 1 --2- 172.21.15.96:0/3586305006 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0dc4153510 0x7f0da800fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:00.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.222+0000 7f0dc0a00640 1 -- 172.21.15.96:0/3586305006 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f0da80055d0 con 0x7f0da80039e0 2024-05-28T06:18:00.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.222+0000 7f0dc0a00640 1 -- 172.21.15.96:0/3586305006 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f0da8010310 con 0x7f0da8004800 2024-05-28T06:18:00.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.222+0000 7f0dc0a00640 1 -- 172.21.15.96:0/3586305006 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f0da8010490 con 0x7f0dc4153510 2024-05-28T06:18:00.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.222+0000 7f0dc1e00640 1 --2- 172.21.15.96:0/3586305006 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0da80039e0 0x7f0da8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:00.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.222+0000 7f0dc2800640 1 --2- 172.21.15.96:0/3586305006 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0dc4153510 0x7f0da800fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:00.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.222+0000 7f0dc1e00640 1 -- 172.21.15.96:0/3586305006 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0dc4153510 msgr2=0x7f0da800fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:18:00.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.222+0000 7f0dc1e00640 1 --2- 172.21.15.96:0/3586305006 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0dc4153510 0x7f0da800fdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:00.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.222+0000 7f0dc1e00640 1 -- 172.21.15.96:0/3586305006 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0da8004800 msgr2=0x7f0da8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:18:00.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.222+0000 7f0dc1e00640 1 --2- 172.21.15.96:0/3586305006 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0da8004800 0x7f0da8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:00.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.222+0000 7f0dc1e00640 1 -- 172.21.15.96:0/3586305006 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0da8003520 con 0x7f0da80039e0 2024-05-28T06:18:00.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.223+0000 7f0dc1e00640 1 --2- 172.21.15.96:0/3586305006 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0da80039e0 0x7f0da8003de0 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7f0db400a9d0 tx=0x7f0da8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45179cff0e2ec35 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:00.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.223+0000 7f0dc0a00640 1 -- 172.21.15.96:0/3586305006 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0db4037070 con 0x7f0da80039e0 2024-05-28T06:18:00.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.223+0000 7f0dc0a00640 1 -- 172.21.15.96:0/3586305006 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0db402ab80 con 0x7f0da80039e0 2024-05-28T06:18:00.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.223+0000 7f0dc0a00640 1 -- 172.21.15.96:0/3586305006 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0db400a570 con 0x7f0da80039e0 2024-05-28T06:18:00.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.224+0000 7f0dc3200640 1 -- 172.21.15.96:0/3586305006 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0da80039e0 msgr2=0x7f0da8003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:00.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.224+0000 7f0dc3200640 1 --2- 172.21.15.96:0/3586305006 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0da80039e0 0x7f0da8003de0 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7f0db400a9d0 tx=0x7f0da8005830 comp rx=0 tx=0).stop 2024-05-28T06:18:00.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.224+0000 7f0dc2800640 1 -- 172.21.15.96:0/3586305006 reap_dead start 2024-05-28T06:18:00.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.224+0000 7f0dc3200640 1 -- 172.21.15.96:0/3586305006 shutdown_connections 2024-05-28T06:18:00.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.224+0000 7f0dc3200640 1 -- 172.21.15.96:0/3586305006 >> 172.21.15.96:0/3586305006 conn(0x7f0dc406b900 msgr2=0x7f0dc406bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:18:00.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.224+0000 7f0dc3200640 1 -- 172.21.15.96:0/3586305006 shutdown_connections 2024-05-28T06:18:00.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.224+0000 7f0dc3200640 1 -- 172.21.15.96:0/3586305006 wait complete. 2024-05-28T06:18:00.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.226+0000 7f0dc3200640 1 Processor -- start 2024-05-28T06:18:00.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.226+0000 7f0dc3200640 1 -- start start 2024-05-28T06:18:00.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.227+0000 7f0dc3200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0da80039e0 0x7f0dc4071a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:00.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.227+0000 7f0dc3200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0dc414b4f0 0x7f0dc4099090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:00.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.227+0000 7f0dc3200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0dc40b0e10 0x7f0dc40b1290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:00.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.227+0000 7f0dc3200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f0dc40b4d50 con 0x7f0dc40b0e10 2024-05-28T06:18:00.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.227+0000 7f0dc3200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f0dc40b48c0 con 0x7f0dc414b4f0 2024-05-28T06:18:00.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.227+0000 7f0dc3200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f0dc40b4ed0 con 0x7f0da80039e0 2024-05-28T06:18:00.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.227+0000 7f0dc2800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0dc40b0e10 0x7f0dc40b1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:00.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.228+0000 7f0dc2800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0dc40b0e10 0x7f0dc40b1290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:60154/0 (socket says 172.21.15.96:60154) 2024-05-28T06:18:00.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.228+0000 7f0dc2800640 1 -- 172.21.15.96:0/3208827015 learned_addr learned my addr 172.21.15.96:0/3208827015 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:18:00.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.228+0000 7f0dc1e00640 1 --2- 172.21.15.96:0/3208827015 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0da80039e0 0x7f0dc4071a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:00.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.228+0000 7f0dc2800640 1 -- 172.21.15.96:0/3208827015 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0da80039e0 msgr2=0x7f0dc4071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:00.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.228+0000 7f0dc1400640 1 --2- 172.21.15.96:0/3208827015 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0dc414b4f0 0x7f0dc4099090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:00.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.228+0000 7f0dc2800640 1 --2- 172.21.15.96:0/3208827015 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0da80039e0 0x7f0dc4071a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:00.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.228+0000 7f0dc2800640 1 -- 172.21.15.96:0/3208827015 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0dc414b4f0 msgr2=0x7f0dc4099090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:00.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.228+0000 7f0dc2800640 1 --2- 172.21.15.96:0/3208827015 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0dc414b4f0 0x7f0dc4099090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:00.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.228+0000 7f0dc2800640 1 -- 172.21.15.96:0/3208827015 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dc40995d0 con 0x7f0dc40b0e10 2024-05-28T06:18:00.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.228+0000 7f0dc2800640 1 --2- 172.21.15.96:0/3208827015 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0dc40b0e10 0x7f0dc40b1290 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f0dc40cdce0 tx=0x7f0dbc00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:00.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.228+0000 7f0dba000640 1 -- 172.21.15.96:0/3208827015 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dbc01a070 con 0x7f0dc40b0e10 2024-05-28T06:18:00.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.229+0000 7f0dba000640 1 -- 172.21.15.96:0/3208827015 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0dbc004aa0 con 0x7f0dc40b0e10 2024-05-28T06:18:00.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.229+0000 7f0dba000640 1 -- 172.21.15.96:0/3208827015 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dbc005030 con 0x7f0dc40b0e10 2024-05-28T06:18:00.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.229+0000 7f0dc3200640 1 -- 172.21.15.96:0/3208827015 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0dc40721f0 con 0x7f0dc40b0e10 2024-05-28T06:18:00.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.229+0000 7f0dc3200640 1 -- 172.21.15.96:0/3208827015 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0dc41502a0 con 0x7f0dc40b0e10 2024-05-28T06:18:00.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.230+0000 7f0dc3200640 1 -- 172.21.15.96:0/3208827015 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0dc40681e0 con 0x7f0dc40b0e10 2024-05-28T06:18:00.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.231+0000 7f0dba000640 1 -- 172.21.15.96:0/3208827015 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 30) ==== 92621+0+0 (secure 0 0 0) 0x7f0dbc01d070 con 0x7f0dc40b0e10 2024-05-28T06:18:00.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.232+0000 7f0dba000640 1 --2- 172.21.15.96:0/3208827015 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0da406da60 0x7f0da406ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:00.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.232+0000 7f0dba000640 1 -- 172.21.15.96:0/3208827015 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 4971+0+0 (secure 0 0 0) 0x7f0dbc089d30 con 0x7f0dc40b0e10 2024-05-28T06:18:00.231 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.233+0000 7f0dc1e00640 1 --2- 172.21.15.96:0/3208827015 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0da406da60 0x7f0da406ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:00.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.234+0000 7f0dc1e00640 1 --2- 172.21.15.96:0/3208827015 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0da406da60 0x7f0da406ff20 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f0db4034000 tx=0x7f0db402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:00.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.236+0000 7f0dba000640 1 -- 172.21.15.96:0/3208827015 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0dc40681e0 con 0x7f0dc40b0e10 2024-05-28T06:18:00.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.436+0000 7f0dc3200640 1 -- 172.21.15.96:0/3208827015 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7f0dc40689e0 con 0x7f0dc40b0e10 2024-05-28T06:18:00.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.442+0000 7f0dba000640 1 -- 172.21.15.96:0/3208827015 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v21) ==== 104+0+0 (secure 0 0 0) 0x7f0dbc059860 con 0x7f0dc40b0e10 2024-05-28T06:18:00.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.444+0000 7f0dc3200640 1 -- 172.21.15.96:0/3208827015 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0da406da60 msgr2=0x7f0da406ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:00.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.444+0000 7f0dc3200640 1 --2- 172.21.15.96:0/3208827015 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0da406da60 0x7f0da406ff20 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f0db4034000 tx=0x7f0db402e000 comp rx=0 tx=0).stop 2024-05-28T06:18:00.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.444+0000 7f0dc3200640 1 -- 172.21.15.96:0/3208827015 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0dc40b0e10 msgr2=0x7f0dc40b1290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:00.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.444+0000 7f0dc3200640 1 --2- 172.21.15.96:0/3208827015 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0dc40b0e10 0x7f0dc40b1290 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f0dc40cdce0 tx=0x7f0dbc00dc00 comp rx=0 tx=0).stop 2024-05-28T06:18:00.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.444+0000 7f0dc3200640 1 -- 172.21.15.96:0/3208827015 shutdown_connections 2024-05-28T06:18:00.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.444+0000 7f0dc3200640 1 --2- 172.21.15.96:0/3208827015 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0da406da60 0x7f0da406ff20 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:00.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.444+0000 7f0dc3200640 1 --2- 172.21.15.96:0/3208827015 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0dc40b0e10 0x7f0dc40b1290 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:00.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.444+0000 7f0dc3200640 1 --2- 172.21.15.96:0/3208827015 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0dc414b4f0 0x7f0dc4099090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:00.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.444+0000 7f0dc3200640 1 --2- 172.21.15.96:0/3208827015 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0da80039e0 0x7f0dc4071a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:00.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.444+0000 7f0dc3200640 1 -- 172.21.15.96:0/3208827015 >> 172.21.15.96:0/3208827015 conn(0x7f0dc406b900 msgr2=0x7f0dc40d00a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:18:00.444 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.445+0000 7f0dc3200640 1 -- 172.21.15.96:0/3208827015 shutdown_connections 2024-05-28T06:18:00.444 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.445+0000 7f0dc3200640 1 -- 172.21.15.96:0/3208827015 wait complete. 2024-05-28T06:18:00.461 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestCloneProgressReporter.test_progress_drops_when_new_jobs_are_added' 2024-05-28T06:18:00.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.651+0000 7f9976a00640 1 Processor -- start 2024-05-28T06:18:00.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.651+0000 7f9976a00640 1 -- start start 2024-05-28T06:18:00.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.653+0000 7f9976a00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f99700d16a0 0x7f99700cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:00.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.653+0000 7f9976a00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f99700cfce0 con 0x7f997015e870 2024-05-28T06:18:00.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.653+0000 7f9976a00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f997015ec50 con 0x7f99700cf870 2024-05-28T06:18:00.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.653+0000 7f9976a00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f99700d2150 con 0x7f99700d16a0 2024-05-28T06:18:00.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.653+0000 7f9975600640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f997015e870 0x7f99700d0f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:48860/0 (socket says 172.21.15.96:48860) 2024-05-28T06:18:00.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.653+0000 7f9975600640 1 -- 172.21.15.96:0/154112953 learned_addr learned my addr 172.21.15.96:0/154112953 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:18:00.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.653+0000 7f9974c00640 1 --2- 172.21.15.96:0/154112953 >> v2:172.21.15.123:3300/0 conn(0x7f99700d16a0 0x7f99700cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:00.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.653+0000 7f996fe00640 1 -- 172.21.15.96:0/154112953 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3502261794 0 0) 0x7f99700cfce0 con 0x7f997015e870 2024-05-28T06:18:00.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.653+0000 7f996fe00640 1 -- 172.21.15.96:0/154112953 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9954003520 con 0x7f997015e870 2024-05-28T06:18:00.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.654+0000 7f996fe00640 1 -- 172.21.15.96:0/154112953 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1574920583 0 0) 0x7f9954003520 con 0x7f997015e870 2024-05-28T06:18:00.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.654+0000 7f996fe00640 1 -- 172.21.15.96:0/154112953 >> v2:172.21.15.123:3300/0 conn(0x7f99700d16a0 msgr2=0x7f99700cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:18:00.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.654+0000 7f996fe00640 1 --2- 172.21.15.96:0/154112953 >> v2:172.21.15.123:3300/0 conn(0x7f99700d16a0 0x7f99700cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:00.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.654+0000 7f996fe00640 1 -- 172.21.15.96:0/154112953 >> v1:172.21.15.123:6789/0 conn(0x7f99700cf870 legacy=0x7f99700d25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:18:00.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.654+0000 7f996fe00640 1 -- 172.21.15.96:0/154112953 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99700cff80 con 0x7f997015e870 2024-05-28T06:18:00.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.654+0000 7f996fe00640 1 -- 172.21.15.96:0/154112953 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f99600019f0 con 0x7f997015e870 2024-05-28T06:18:00.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.654+0000 7f996fe00640 1 -- 172.21.15.96:0/154112953 >> v1:172.21.15.96:6789/0 conn(0x7f997015e870 legacy=0x7f99700d0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:00.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.654+0000 7f996fe00640 1 --2- 172.21.15.96:0/154112953 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f99540039b0 0x7f9954003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:00.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.654+0000 7f996fe00640 1 --2- 172.21.15.96:0/154112953 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9954004800 0x7f9954004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:00.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.654+0000 7f996fe00640 1 --2- 172.21.15.96:0/154112953 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f99540051c0 0x7f995400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:00.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.654+0000 7f996fe00640 1 -- 172.21.15.96:0/154112953 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f99700d2150 con 0x7f99540051c0 2024-05-28T06:18:00.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.654+0000 7f996fe00640 1 -- 172.21.15.96:0/154112953 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f995400c0f0 con 0x7f9954004800 2024-05-28T06:18:00.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.654+0000 7f996fe00640 1 -- 172.21.15.96:0/154112953 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f995400c2f0 con 0x7f99540039b0 2024-05-28T06:18:00.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.654+0000 7f9976000640 1 --2- 172.21.15.96:0/154112953 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f99540051c0 0x7f995400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:00.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.654+0000 7f9974c00640 1 --2- 172.21.15.96:0/154112953 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9954004800 0x7f9954004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:00.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.655+0000 7f9975600640 1 --2- 172.21.15.96:0/154112953 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f99540039b0 0x7f9954003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:00.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.655+0000 7f9976000640 1 -- 172.21.15.96:0/154112953 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f99540039b0 msgr2=0x7f9954003db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:18:00.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.655+0000 7f9976000640 1 --2- 172.21.15.96:0/154112953 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f99540039b0 0x7f9954003db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:00.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.655+0000 7f9976000640 1 -- 172.21.15.96:0/154112953 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9954004800 msgr2=0x7f9954004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:18:00.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.655+0000 7f9976000640 1 --2- 172.21.15.96:0/154112953 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9954004800 0x7f9954004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:00.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.655+0000 7f9976000640 1 -- 172.21.15.96:0/154112953 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9954003520 con 0x7f99540051c0 2024-05-28T06:18:00.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.655+0000 7f9976000640 1 -- 172.21.15.96:0/154112953 reap_dead start 2024-05-28T06:18:00.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.656+0000 7f9976000640 1 --2- 172.21.15.96:0/154112953 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f99540051c0 0x7f995400b9c0 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7f9968003d80 tx=0x7f9954005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a9b0fce09eb65ad server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:00.659 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.656+0000 7f996fe00640 1 -- 172.21.15.96:0/154112953 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9968003980 con 0x7f99540051c0 2024-05-28T06:18:00.659 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.656+0000 7f996fe00640 1 -- 172.21.15.96:0/154112953 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9968007240 con 0x7f99540051c0 2024-05-28T06:18:00.659 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.656+0000 7f996fe00640 1 -- 172.21.15.96:0/154112953 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9968007630 con 0x7f99540051c0 2024-05-28T06:18:00.659 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.656+0000 7f9976a00640 1 -- 172.21.15.96:0/154112953 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f99540051c0 msgr2=0x7f995400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:00.659 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.656+0000 7f9976a00640 1 --2- 172.21.15.96:0/154112953 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f99540051c0 0x7f995400b9c0 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7f9968003d80 tx=0x7f9954005830 comp rx=0 tx=0).stop 2024-05-28T06:18:00.659 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.656+0000 7f9976a00640 1 -- 172.21.15.96:0/154112953 shutdown_connections 2024-05-28T06:18:00.659 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.656+0000 7f9976a00640 1 --2- 172.21.15.96:0/154112953 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f99540051c0 0x7f995400b9c0 unknown :-1 s=CLOSED pgs=1346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:00.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.656+0000 7f9976a00640 1 -- 172.21.15.96:0/154112953 >> 172.21.15.96:0/154112953 conn(0x7f997008bfc0 msgr2=0x7f99700d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:18:00.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.657+0000 7f9976a00640 1 -- 172.21.15.96:0/154112953 shutdown_connections 2024-05-28T06:18:00.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.657+0000 7f9976a00640 1 -- 172.21.15.96:0/154112953 wait complete. 2024-05-28T06:18:00.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.658+0000 7f9976a00640 1 Processor -- start 2024-05-28T06:18:00.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.658+0000 7f9976a00640 1 -- start start 2024-05-28T06:18:00.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.659+0000 7f9976a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99540051c0 0x7f99700cda70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:00.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.659+0000 7f9976a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f99701f7790 0x7f99700b9270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:00.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.659+0000 7f9976a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f99701f4890 0x7f99701f4d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:00.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.659+0000 7f9976a00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f99700d0820 con 0x7f99701f4890 2024-05-28T06:18:00.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.659+0000 7f9976a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f99700d0540 con 0x7f99540051c0 2024-05-28T06:18:00.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.659+0000 7f9976a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f99700d4fb0 con 0x7f99701f7790 2024-05-28T06:18:00.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.659+0000 7f9976000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f99701f4890 0x7f99701f4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:00.661 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.659+0000 7f9976000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f99701f4890 0x7f99701f4d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:60176/0 (socket says 172.21.15.96:60176) 2024-05-28T06:18:00.661 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.659+0000 7f9976000640 1 -- 172.21.15.96:0/1466997255 learned_addr learned my addr 172.21.15.96:0/1466997255 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:18:00.661 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.660+0000 7f9975600640 1 --2- 172.21.15.96:0/1466997255 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99540051c0 0x7f99700cda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:00.661 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.660+0000 7f9974c00640 1 --2- 172.21.15.96:0/1466997255 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f99701f7790 0x7f99700b9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:00.661 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.660+0000 7f9976000640 1 -- 172.21.15.96:0/1466997255 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f99701f7790 msgr2=0x7f99700b9270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:00.661 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.660+0000 7f9976000640 1 --2- 172.21.15.96:0/1466997255 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f99701f7790 0x7f99700b9270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:00.661 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.660+0000 7f9976000640 1 -- 172.21.15.96:0/1466997255 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99540051c0 msgr2=0x7f99700cda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:00.661 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.660+0000 7f9976000640 1 --2- 172.21.15.96:0/1466997255 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99540051c0 0x7f99700cda70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:00.661 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.660+0000 7f9976000640 1 -- 172.21.15.96:0/1466997255 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99701f7bf0 con 0x7f99701f4890 2024-05-28T06:18:00.661 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.660+0000 7f9975600640 1 --2- 172.21.15.96:0/1466997255 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99540051c0 0x7f99700cda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:18:00.661 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.660+0000 7f9976000640 1 --2- 172.21.15.96:0/1466997255 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f99701f4890 0x7f99701f4d10 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f9968003d80 tx=0x7f99680036f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:00.661 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.660+0000 7f996d600640 1 -- 172.21.15.96:0/1466997255 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99680067d0 con 0x7f99701f4890 2024-05-28T06:18:00.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.661+0000 7f996d600640 1 -- 172.21.15.96:0/1466997255 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9968042070 con 0x7f99701f4890 2024-05-28T06:18:00.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.661+0000 7f9976a00640 1 -- 172.21.15.96:0/1466997255 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99701f7e70 con 0x7f99701f4890 2024-05-28T06:18:00.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.661+0000 7f996d600640 1 -- 172.21.15.96:0/1466997255 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9968006b90 con 0x7f99701f4890 2024-05-28T06:18:00.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.661+0000 7f9976a00640 1 -- 172.21.15.96:0/1466997255 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99701f8430 con 0x7f99701f4890 2024-05-28T06:18:00.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.662+0000 7f9976a00640 1 -- 172.21.15.96:0/1466997255 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99700689e0 con 0x7f99701f4890 2024-05-28T06:18:00.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.663+0000 7f996d600640 1 -- 172.21.15.96:0/1466997255 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 30) ==== 92621+0+0 (secure 0 0 0) 0x7f9968047460 con 0x7f99701f4890 2024-05-28T06:18:00.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.664+0000 7f996d600640 1 --2- 172.21.15.96:0/1466997255 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f995006da60 0x7f995006ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:00.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.664+0000 7f996d600640 1 -- 172.21.15.96:0/1466997255 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 4971+0+0 (secure 0 0 0) 0x7f99680aa850 con 0x7f99701f4890 2024-05-28T06:18:00.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.664+0000 7f9975600640 1 --2- 172.21.15.96:0/1466997255 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f995006da60 0x7f995006ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:00.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.665+0000 7f9975600640 1 --2- 172.21.15.96:0/1466997255 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f995006da60 0x7f995006ff20 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f9960001ca0 tx=0x7f9960009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:00.666 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.668+0000 7f996d600640 1 -- 172.21.15.96:0/1466997255 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f99701f7bf0 con 0x7f99701f4890 2024-05-28T06:18:00.919 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:00.920+0000 7f9976a00640 1 -- 172.21.15.96:0/1466997255 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgressReporter.test_progress_drops_when_new_jobs_are_added"]} v 0) -- 0x7f99700ba500 con 0x7f99701f4890 2024-05-28T06:18:01.536 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.538+0000 7f996d600640 1 -- 172.21.15.96:0/1466997255 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgressReporter.test_progress_drops_when_new_jobs_are_added"]}]=0 v612) ==== 174+0+0 (secure 0 0 0) 0x7f996807a710 con 0x7f99701f4890 2024-05-28T06:18:01.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.541+0000 7f9976a00640 1 -- 172.21.15.96:0/1466997255 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f995006da60 msgr2=0x7f995006ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:01.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.541+0000 7f9976a00640 1 --2- 172.21.15.96:0/1466997255 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f995006da60 0x7f995006ff20 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f9960001ca0 tx=0x7f9960009040 comp rx=0 tx=0).stop 2024-05-28T06:18:01.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.541+0000 7f9976a00640 1 -- 172.21.15.96:0/1466997255 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f99701f4890 msgr2=0x7f99701f4d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:01.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.541+0000 7f9976a00640 1 --2- 172.21.15.96:0/1466997255 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f99701f4890 0x7f99701f4d10 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f9968003d80 tx=0x7f99680036f0 comp rx=0 tx=0).stop 2024-05-28T06:18:01.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.541+0000 7f9976a00640 1 -- 172.21.15.96:0/1466997255 shutdown_connections 2024-05-28T06:18:01.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.541+0000 7f9976a00640 1 --2- 172.21.15.96:0/1466997255 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f995006da60 0x7f995006ff20 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:01.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.541+0000 7f9976a00640 1 --2- 172.21.15.96:0/1466997255 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f99701f4890 0x7f99701f4d10 unknown :-1 s=CLOSED pgs=1347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:01.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.541+0000 7f9976a00640 1 --2- 172.21.15.96:0/1466997255 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f99701f7790 0x7f99700b9270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:01.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.541+0000 7f9976a00640 1 --2- 172.21.15.96:0/1466997255 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99540051c0 0x7f99700cda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:01.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.541+0000 7f9976a00640 1 -- 172.21.15.96:0/1466997255 >> 172.21.15.96:0/1466997255 conn(0x7f997008bfc0 msgr2=0x7f99700b54c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:18:01.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.542+0000 7f9976a00640 1 -- 172.21.15.96:0/1466997255 shutdown_connections 2024-05-28T06:18:01.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.542+0000 7f9976a00640 1 -- 172.21.15.96:0/1466997255 wait complete. 2024-05-28T06:18:01.562 INFO:tasks.cephfs_test_runner:Test that progress indicated by progress bar for ongoing+pending clones ... ok 2024-05-28T06:18:01.563 INFO:tasks.cephfs_test_runner:Starting test: test_stats_are_printed (tasks.cephfs.test_volumes.TestCloneProgressReporter) Test that the command "ceph fs clone status" prints progress stats 2024-05-28T06:18:01.563 INFO:tasks.cephfs_test_runner:test_stats_are_printed (tasks.cephfs.test_volumes.TestCloneProgressReporter) 2024-05-28T06:18:01.563 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:18:01.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.763+0000 7f269a600640 1 Processor -- start 2024-05-28T06:18:01.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.763+0000 7f269a600640 1 -- start start 2024-05-28T06:18:01.763 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.765+0000 7f269a600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f26940d0590 0x7f26940d09f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:01.763 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.765+0000 7f269a600640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26940b51f0 con 0x7f2694099090 2024-05-28T06:18:01.763 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.765+0000 7f269a600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26940aef60 con 0x7f26940b4e10 2024-05-28T06:18:01.763 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.765+0000 7f269a600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f26940d0f30 con 0x7f26940d0590 2024-05-28T06:18:01.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.765+0000 7f2693e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f2694099090 0x7f26940994e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:48866/0 (socket says 172.21.15.96:48866) 2024-05-28T06:18:01.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.765+0000 7f2693e00640 1 -- 172.21.15.96:0/1024437967 learned_addr learned my addr 172.21.15.96:0/1024437967 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:18:01.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.766+0000 7f2692a00640 1 --2- 172.21.15.96:0/1024437967 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f26940d0590 0x7f26940d09f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:01.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.766+0000 7f2692000640 1 -- 172.21.15.96:0/1024437967 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1370704301 0 0) 0x7f26940b51f0 con 0x7f2694099090 2024-05-28T06:18:01.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.766+0000 7f2692000640 1 -- 172.21.15.96:0/1024437967 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2678003520 con 0x7f2694099090 2024-05-28T06:18:01.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.766+0000 7f2692000640 1 -- 172.21.15.96:0/1024437967 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 312318301 0 0) 0x7f2678003520 con 0x7f2694099090 2024-05-28T06:18:01.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.766+0000 7f2692000640 1 -- 172.21.15.96:0/1024437967 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f26940d0590 msgr2=0x7f26940d09f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:18:01.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.766+0000 7f2692000640 1 --2- 172.21.15.96:0/1024437967 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f26940d0590 0x7f26940d09f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:01.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.766+0000 7f2692000640 1 -- 172.21.15.96:0/1024437967 >> v1:172.21.15.123:6789/0 conn(0x7f26940b4e10 legacy=0x7f26940cde30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:01.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.766+0000 7f2692000640 1 -- 172.21.15.96:0/1024437967 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26940ad1d0 con 0x7f2694099090 2024-05-28T06:18:01.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.766+0000 7f2692000640 1 -- 172.21.15.96:0/1024437967 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f268c0019f0 con 0x7f2694099090 2024-05-28T06:18:01.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.766+0000 7f2692000640 1 -- 172.21.15.96:0/1024437967 >> v1:172.21.15.96:6789/0 conn(0x7f2694099090 legacy=0x7f26940994e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:01.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.766+0000 7f2692000640 1 --2- 172.21.15.96:0/1024437967 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26780039e0 0x7f2678003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:01.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.766+0000 7f2692000640 1 --2- 172.21.15.96:0/1024437967 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2678004800 0x7f2678004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:01.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.766+0000 7f2692000640 1 --2- 172.21.15.96:0/1024437967 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f26940d0590 0x7f2678005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:01.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.766+0000 7f2692000640 1 -- 172.21.15.96:0/1024437967 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f267800bac0 con 0x7f2678004800 2024-05-28T06:18:01.767 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.766+0000 7f2692000640 1 -- 172.21.15.96:0/1024437967 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f267800bc70 con 0x7f26780039e0 2024-05-28T06:18:01.767 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.766+0000 7f2692000640 1 -- 172.21.15.96:0/1024437967 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f267800e080 con 0x7f26940d0590 2024-05-28T06:18:01.767 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.766+0000 7f2692a00640 1 --2- 172.21.15.96:0/1024437967 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2678004800 0x7f2678004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:01.767 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.767+0000 7f2692a00640 1 -- 172.21.15.96:0/1024437967 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f26940d0590 msgr2=0x7f2678005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:18:01.767 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.767+0000 7f2692a00640 1 --2- 172.21.15.96:0/1024437967 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f26940d0590 0x7f2678005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:01.767 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.767+0000 7f2693400640 1 --2- 172.21.15.96:0/1024437967 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26780039e0 0x7f2678003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:01.767 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.767+0000 7f2692a00640 1 -- 172.21.15.96:0/1024437967 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26780039e0 msgr2=0x7f2678003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:18:01.767 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.767+0000 7f2692a00640 1 --2- 172.21.15.96:0/1024437967 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26780039e0 0x7f2678003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:01.767 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.767+0000 7f2692a00640 1 -- 172.21.15.96:0/1024437967 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2678003520 con 0x7f2678004800 2024-05-28T06:18:01.767 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.767+0000 7f2693400640 1 --2- 172.21.15.96:0/1024437967 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26780039e0 0x7f2678003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:18:01.768 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.768+0000 7f2692a00640 1 --2- 172.21.15.96:0/1024437967 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2678004800 0x7f2678004c80 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f2680009ac0 tx=0x7f2678005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42c571755622f5e8 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:01.768 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.768+0000 7f2692000640 1 -- 172.21.15.96:0/1024437967 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2680036070 con 0x7f2678004800 2024-05-28T06:18:01.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.768+0000 7f2692000640 1 -- 172.21.15.96:0/1024437967 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2680031400 con 0x7f2678004800 2024-05-28T06:18:01.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.768+0000 7f2692000640 1 -- 172.21.15.96:0/1024437967 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2680009690 con 0x7f2678004800 2024-05-28T06:18:01.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.769+0000 7f269a600640 1 -- 172.21.15.96:0/1024437967 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2678004800 msgr2=0x7f2678004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:01.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.769+0000 7f269a600640 1 --2- 172.21.15.96:0/1024437967 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2678004800 0x7f2678004c80 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f2680009ac0 tx=0x7f2678005830 comp rx=0 tx=0).stop 2024-05-28T06:18:01.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.769+0000 7f2693e00640 1 -- 172.21.15.96:0/1024437967 reap_dead start 2024-05-28T06:18:01.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.769+0000 7f269a600640 1 -- 172.21.15.96:0/1024437967 shutdown_connections 2024-05-28T06:18:01.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.769+0000 7f269a600640 1 -- 172.21.15.96:0/1024437967 >> 172.21.15.96:0/1024437967 conn(0x7f269406bae0 msgr2=0x7f269406bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:18:01.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.769+0000 7f269a600640 1 -- 172.21.15.96:0/1024437967 shutdown_connections 2024-05-28T06:18:01.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.769+0000 7f269a600640 1 -- 172.21.15.96:0/1024437967 wait complete. 2024-05-28T06:18:01.770 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.771+0000 7f269a600640 1 Processor -- start 2024-05-28T06:18:01.770 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.771+0000 7f269a600640 1 -- start start 2024-05-28T06:18:01.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.772+0000 7f269a600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2678004800 0x7f26940c7c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:01.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.772+0000 7f269a600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2694158190 0x7f2694072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:01.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.772+0000 7f269a600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f26941585d0 0x7f26941575e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:01.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.772+0000 7f269a600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f26940ce540 con 0x7f26941585d0 2024-05-28T06:18:01.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.772+0000 7f269a600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f26940ad750 con 0x7f2694158190 2024-05-28T06:18:01.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.772+0000 7f269a600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f26940ce880 con 0x7f2678004800 2024-05-28T06:18:01.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.772+0000 7f2693e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f26941585d0 0x7f26941575e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:01.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.772+0000 7f2693e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f26941585d0 0x7f26941575e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:60206/0 (socket says 172.21.15.96:60206) 2024-05-28T06:18:01.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.772+0000 7f2693e00640 1 -- 172.21.15.96:0/1171206029 learned_addr learned my addr 172.21.15.96:0/1171206029 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:18:01.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.772+0000 7f2692a00640 1 --2- 172.21.15.96:0/1171206029 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2694158190 0x7f2694072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:01.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.772+0000 7f2693400640 1 --2- 172.21.15.96:0/1171206029 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2678004800 0x7f26940c7c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:01.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.772+0000 7f2693e00640 1 -- 172.21.15.96:0/1171206029 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2678004800 msgr2=0x7f26940c7c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:01.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.772+0000 7f2693e00640 1 --2- 172.21.15.96:0/1171206029 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2678004800 0x7f26940c7c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:01.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.773+0000 7f2693e00640 1 -- 172.21.15.96:0/1171206029 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2694158190 msgr2=0x7f2694072df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:01.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.773+0000 7f2693e00640 1 --2- 172.21.15.96:0/1171206029 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2694158190 0x7f2694072df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:01.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.773+0000 7f2693e00640 1 -- 172.21.15.96:0/1171206029 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26940b2c30 con 0x7f26941585d0 2024-05-28T06:18:01.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.773+0000 7f2692a00640 1 --2- 172.21.15.96:0/1171206029 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2694158190 0x7f2694072df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:18:01.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.773+0000 7f2693e00640 1 --2- 172.21.15.96:0/1171206029 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f26941585d0 0x7f26941575e0 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f26940ca7f0 tx=0x7f268c00cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:01.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.773+0000 7f268b400640 1 -- 172.21.15.96:0/1171206029 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f268c004d60 con 0x7f26941585d0 2024-05-28T06:18:01.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.773+0000 7f268b400640 1 -- 172.21.15.96:0/1171206029 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f268c00a610 con 0x7f26941585d0 2024-05-28T06:18:01.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.773+0000 7f268b400640 1 -- 172.21.15.96:0/1171206029 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f268c00aa00 con 0x7f26941585d0 2024-05-28T06:18:01.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.773+0000 7f269a600640 1 -- 172.21.15.96:0/1171206029 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26940b2690 con 0x7f26941585d0 2024-05-28T06:18:01.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.773+0000 7f269a600640 1 -- 172.21.15.96:0/1171206029 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26940c7460 con 0x7f26941585d0 2024-05-28T06:18:01.774 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.774+0000 7f269a600640 1 -- 172.21.15.96:0/1171206029 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26940689e0 con 0x7f26941585d0 2024-05-28T06:18:01.774 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.775+0000 7f268b400640 1 -- 172.21.15.96:0/1171206029 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 30) ==== 92621+0+0 (secure 0 0 0) 0x7f268c00aba0 con 0x7f26941585d0 2024-05-28T06:18:01.774 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.776+0000 7f268b400640 1 --2- 172.21.15.96:0/1171206029 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f267406da60 0x7f267406ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:01.775 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.776+0000 7f268b400640 1 -- 172.21.15.96:0/1171206029 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 4971+0+0 (secure 0 0 0) 0x7f268c089d20 con 0x7f26941585d0 2024-05-28T06:18:01.775 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.777+0000 7f2693400640 1 --2- 172.21.15.96:0/1171206029 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f267406da60 0x7f267406ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:01.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.778+0000 7f2693400640 1 --2- 172.21.15.96:0/1171206029 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f267406da60 0x7f267406ff20 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f2684001610 tx=0x7f2684009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:01.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.781+0000 7f268b400640 1 -- 172.21.15.96:0/1171206029 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f268c01a070 con 0x7f26941585d0 2024-05-28T06:18:01.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.979+0000 7f269a600640 1 -- 172.21.15.96:0/1171206029 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f269406bf90 con 0x7f26941585d0 2024-05-28T06:18:01.979 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.980+0000 7f268b400640 1 -- 172.21.15.96:0/1171206029 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+13074 (secure 0 0 0) 0x7f268c01a250 con 0x7f26941585d0 2024-05-28T06:18:01.979 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:18:01.979 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":81,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:17:59.516443+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.123:6836/76143454":"2024-05-29T06:17:56.322084+0000","172.21.15.123:6834/76143454":"2024-05-29T06:17:56.322084+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:18:01.982 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.983+0000 7f269a600640 1 -- 172.21.15.96:0/1171206029 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f267406da60 msgr2=0x7f267406ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:01.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.983+0000 7f269a600640 1 --2- 172.21.15.96:0/1171206029 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f267406da60 0x7f267406ff20 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f2684001610 tx=0x7f2684009040 comp rx=0 tx=0).stop 2024-05-28T06:18:01.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.983+0000 7f269a600640 1 -- 172.21.15.96:0/1171206029 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f26941585d0 msgr2=0x7f26941575e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:01.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.983+0000 7f269a600640 1 --2- 172.21.15.96:0/1171206029 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f26941585d0 0x7f26941575e0 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f26940ca7f0 tx=0x7f268c00cbe0 comp rx=0 tx=0).stop 2024-05-28T06:18:01.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.983+0000 7f269a600640 1 -- 172.21.15.96:0/1171206029 shutdown_connections 2024-05-28T06:18:01.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.983+0000 7f269a600640 1 --2- 172.21.15.96:0/1171206029 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f267406da60 0x7f267406ff20 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:01.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.983+0000 7f269a600640 1 --2- 172.21.15.96:0/1171206029 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f26941585d0 0x7f26941575e0 unknown :-1 s=CLOSED pgs=1350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:01.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.983+0000 7f269a600640 1 --2- 172.21.15.96:0/1171206029 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2694158190 0x7f2694072df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:01.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.983+0000 7f269a600640 1 --2- 172.21.15.96:0/1171206029 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2678004800 0x7f26940c7c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:01.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.983+0000 7f269a600640 1 -- 172.21.15.96:0/1171206029 >> 172.21.15.96:0/1171206029 conn(0x7f269406bae0 msgr2=0x7f269406e780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:18:01.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.983+0000 7f269a600640 1 -- 172.21.15.96:0/1171206029 shutdown_connections 2024-05-28T06:18:01.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:01.984+0000 7f269a600640 1 -- 172.21.15.96:0/1171206029 wait complete. 2024-05-28T06:18:02.002 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-28T06:03:59.517102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-28T06:18:02.002 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-28T06:18:02.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.204+0000 7fbaab600640 1 Processor -- start 2024-05-28T06:18:02.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.205+0000 7fbaab600640 1 -- start start 2024-05-28T06:18:02.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.206+0000 7fbaab600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fbaa4161e70 0x7fbaa40b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:02.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.206+0000 7fbaab600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbaa40b3800 0x7fbaa40ad2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:02.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.206+0000 7fbaab600640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbaa40b3d40 con 0x7fbaa4161a90 2024-05-28T06:18:02.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.206+0000 7fbaab600640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fbaa40b4020 con 0x7fbaa4161e70 2024-05-28T06:18:02.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.206+0000 7fbaab600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fbaa40b4300 con 0x7fbaa40b3800 2024-05-28T06:18:02.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.207+0000 7fba9be00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fbaa4161e70 0x7fbaa40b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:02.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.207+0000 7fbaa8c00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fbaa4161a90 0x7fbaa406e470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:48872/0 (socket says 172.21.15.96:48872) 2024-05-28T06:18:02.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.207+0000 7fbaa8c00640 1 -- 172.21.15.96:0/868612597 learned_addr learned my addr 172.21.15.96:0/868612597 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:18:02.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.207+0000 7fbaa9600640 1 --2- 172.21.15.96:0/868612597 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbaa40b3800 0x7fbaa40ad2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:02.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.207+0000 7fba9be00640 1 -- 172.21.15.96:0/868612597 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbaa40b3800 msgr2=0x7fbaa40ad2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:18:02.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.207+0000 7fba9be00640 1 --2- 172.21.15.96:0/868612597 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbaa40b3800 0x7fbaa40ad2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:02.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.207+0000 7fba9be00640 1 -- 172.21.15.96:0/868612597 >> v1:172.21.15.96:6789/0 conn(0x7fbaa4161a90 legacy=0x7fbaa406e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:02.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.207+0000 7fbaa3e00640 1 -- 172.21.15.96:0/868612597 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3483907466 0 0) 0x7fbaa40b3d40 con 0x7fbaa4161a90 2024-05-28T06:18:02.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.207+0000 7fba9be00640 1 -- 172.21.15.96:0/868612597 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbaa406dc70 con 0x7fbaa4161e70 2024-05-28T06:18:02.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.208+0000 7fba9be00640 1 --2- 172.21.15.96:0/868612597 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbaa4161e70 0x7fbaa40b32c0 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7fba8c0097a0 tx=0x7fba8c02a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=470609149e656ce0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:02.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.209+0000 7fbaa3e00640 1 -- 172.21.15.96:0/868612597 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba8c037070 con 0x7fbaa4161e70 2024-05-28T06:18:02.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.209+0000 7fbaa3e00640 1 -- 172.21.15.96:0/868612597 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba8c032440 con 0x7fbaa4161e70 2024-05-28T06:18:02.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.209+0000 7fbaa3e00640 1 -- 172.21.15.96:0/868612597 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba8c032810 con 0x7fbaa4161e70 2024-05-28T06:18:02.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.209+0000 7fbaab600640 1 -- 172.21.15.96:0/868612597 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbaa4161e70 msgr2=0x7fbaa40b32c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:02.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.209+0000 7fbaab600640 1 --2- 172.21.15.96:0/868612597 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbaa4161e70 0x7fbaa40b32c0 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7fba8c0097a0 tx=0x7fba8c02a180 comp rx=0 tx=0).stop 2024-05-28T06:18:02.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.209+0000 7fbaab600640 1 -- 172.21.15.96:0/868612597 shutdown_connections 2024-05-28T06:18:02.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.209+0000 7fbaab600640 1 --2- 172.21.15.96:0/868612597 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbaa40b3800 0x7fbaa40ad2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:02.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.209+0000 7fbaab600640 1 --2- 172.21.15.96:0/868612597 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbaa4161e70 0x7fbaa40b32c0 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:02.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.209+0000 7fbaab600640 1 -- 172.21.15.96:0/868612597 >> 172.21.15.96:0/868612597 conn(0x7fbaa406bd20 msgr2=0x7fbaa406c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:18:02.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.209+0000 7fbaab600640 1 -- 172.21.15.96:0/868612597 shutdown_connections 2024-05-28T06:18:02.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.209+0000 7fbaab600640 1 -- 172.21.15.96:0/868612597 wait complete. 2024-05-28T06:18:02.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.211+0000 7fbaab600640 1 Processor -- start 2024-05-28T06:18:02.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.211+0000 7fbaab600640 1 -- start start 2024-05-28T06:18:02.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.212+0000 7fbaab600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbaa40b3800 0x7fbaa4155210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:02.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.212+0000 7fbaab600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbaa4161a90 0x7fbaa4159fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:02.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.212+0000 7fbaab600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbaa4161e70 0x7fbaa41472c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:02.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.212+0000 7fbaab600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fbaa406eb80 con 0x7fbaa40b3800 2024-05-28T06:18:02.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.212+0000 7fbaab600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fbaa40b5020 con 0x7fbaa4161e70 2024-05-28T06:18:02.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.212+0000 7fbaab600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fbaa40ad7e0 con 0x7fbaa4161a90 2024-05-28T06:18:02.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.212+0000 7fbaa8c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbaa40b3800 0x7fbaa4155210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:02.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.213+0000 7fbaa8c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbaa40b3800 0x7fbaa4155210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:60210/0 (socket says 172.21.15.96:60210) 2024-05-28T06:18:02.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.213+0000 7fbaa8c00640 1 -- 172.21.15.96:0/2474615488 learned_addr learned my addr 172.21.15.96:0/2474615488 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:18:02.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.213+0000 7fba9be00640 1 --2- 172.21.15.96:0/2474615488 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbaa4161a90 0x7fbaa4159fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:02.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.213+0000 7fbaa8c00640 1 -- 172.21.15.96:0/2474615488 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbaa4161a90 msgr2=0x7fbaa4159fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:02.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.213+0000 7fbaa8c00640 1 --2- 172.21.15.96:0/2474615488 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbaa4161a90 0x7fbaa4159fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:02.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.213+0000 7fbaa8c00640 1 -- 172.21.15.96:0/2474615488 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbaa4161e70 msgr2=0x7fbaa41472c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:02.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.213+0000 7fbaa8c00640 1 --2- 172.21.15.96:0/2474615488 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbaa4161e70 0x7fbaa41472c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:02.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.213+0000 7fbaa8c00640 1 -- 172.21.15.96:0/2474615488 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbaa415a4e0 con 0x7fbaa40b3800 2024-05-28T06:18:02.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.213+0000 7fba9be00640 1 --2- 172.21.15.96:0/2474615488 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbaa4161a90 0x7fbaa4159fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:18:02.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.213+0000 7fbaa8c00640 1 --2- 172.21.15.96:0/2474615488 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbaa40b3800 0x7fbaa4155210 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7fba9400c4d0 tx=0x7fba9400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:02.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.214+0000 7fbaa1600640 1 -- 172.21.15.96:0/2474615488 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba94018070 con 0x7fbaa40b3800 2024-05-28T06:18:02.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.214+0000 7fbaa1600640 1 -- 172.21.15.96:0/2474615488 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba94007500 con 0x7fbaa40b3800 2024-05-28T06:18:02.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.214+0000 7fbaa1600640 1 -- 172.21.15.96:0/2474615488 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba94004a20 con 0x7fbaa40b3800 2024-05-28T06:18:02.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.214+0000 7fbaab600640 1 -- 172.21.15.96:0/2474615488 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbaa41558a0 con 0x7fbaa40b3800 2024-05-28T06:18:02.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.214+0000 7fbaab600640 1 -- 172.21.15.96:0/2474615488 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbaa4155e60 con 0x7fbaa40b3800 2024-05-28T06:18:02.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.215+0000 7fbaab600640 1 -- 172.21.15.96:0/2474615488 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbaa40689e0 con 0x7fbaa40b3800 2024-05-28T06:18:02.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.216+0000 7fbaa1600640 1 -- 172.21.15.96:0/2474615488 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 30) ==== 92621+0+0 (secure 0 0 0) 0x7fba94004bc0 con 0x7fbaa40b3800 2024-05-28T06:18:02.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.217+0000 7fbaa1600640 1 --2- 172.21.15.96:0/2474615488 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fba8006da60 0x7fba8006ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:02.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.217+0000 7fbaa1600640 1 -- 172.21.15.96:0/2474615488 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 4971+0+0 (secure 0 0 0) 0x7fba94089d40 con 0x7fbaa40b3800 2024-05-28T06:18:02.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.218+0000 7fba9be00640 1 --2- 172.21.15.96:0/2474615488 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fba8006da60 0x7fba8006ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:02.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.219+0000 7fba9be00640 1 --2- 172.21.15.96:0/2474615488 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fba8006da60 0x7fba8006ff20 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fba8c034000 tx=0x7fba8c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:02.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.221+0000 7fbaa1600640 1 -- 172.21.15.96:0/2474615488 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fba94059870 con 0x7fbaa40b3800 2024-05-28T06:18:02.420 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.422+0000 7fbaab600640 1 -- 172.21.15.96:0/2474615488 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fbaa406f1e0 con 0x7fbaa40b3800 2024-05-28T06:18:02.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.423+0000 7fbaa1600640 1 -- 172.21.15.96:0/2474615488 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7fba94059170 con 0x7fbaa40b3800 2024-05-28T06:18:02.421 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 1 2024-05-28T06:18:02.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.426+0000 7fbaab600640 1 -- 172.21.15.96:0/2474615488 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fba8006da60 msgr2=0x7fba8006ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:02.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.426+0000 7fbaab600640 1 --2- 172.21.15.96:0/2474615488 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fba8006da60 0x7fba8006ff20 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fba8c034000 tx=0x7fba8c02e000 comp rx=0 tx=0).stop 2024-05-28T06:18:02.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.426+0000 7fbaab600640 1 -- 172.21.15.96:0/2474615488 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbaa40b3800 msgr2=0x7fbaa4155210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:02.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.426+0000 7fbaab600640 1 --2- 172.21.15.96:0/2474615488 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbaa40b3800 0x7fbaa4155210 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7fba9400c4d0 tx=0x7fba9400c990 comp rx=0 tx=0).stop 2024-05-28T06:18:02.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.426+0000 7fbaab600640 1 -- 172.21.15.96:0/2474615488 shutdown_connections 2024-05-28T06:18:02.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.426+0000 7fbaab600640 1 --2- 172.21.15.96:0/2474615488 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fba8006da60 0x7fba8006ff20 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:02.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.426+0000 7fbaab600640 1 --2- 172.21.15.96:0/2474615488 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbaa4161e70 0x7fbaa41472c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:02.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.426+0000 7fbaab600640 1 --2- 172.21.15.96:0/2474615488 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbaa4161a90 0x7fbaa4159fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:02.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.426+0000 7fbaab600640 1 --2- 172.21.15.96:0/2474615488 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbaa40b3800 0x7fbaa4155210 unknown :-1 s=CLOSED pgs=1353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:02.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.426+0000 7fbaab600640 1 -- 172.21.15.96:0/2474615488 >> 172.21.15.96:0/2474615488 conn(0x7fbaa406bd20 msgr2=0x7fbaa40d0610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:18:02.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.426+0000 7fbaab600640 1 -- 172.21.15.96:0/2474615488 shutdown_connections 2024-05-28T06:18:02.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.426+0000 7fbaab600640 1 -- 172.21.15.96:0/2474615488 wait complete. 2024-05-28T06:18:02.445 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestCloneProgressReporter.test_stats_are_printed' 2024-05-28T06:18:02.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.649+0000 7fd991400640 1 Processor -- start 2024-05-28T06:18:02.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.649+0000 7fd991400640 1 -- start start 2024-05-28T06:18:02.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.650+0000 7fd991400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd98c0b4e10 0x7fd98c0cde30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:02.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.651+0000 7fd991400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fd98c0ce370 0x7fd98c0d0590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:02.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.651+0000 7fd991400640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd98c0b51f0 con 0x7fd98c0d0ad0 2024-05-28T06:18:02.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.651+0000 7fd991400640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fd98c0d0f20 con 0x7fd98c0ce370 2024-05-28T06:18:02.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.651+0000 7fd991400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fd98c0aef60 con 0x7fd98c0b4e10 2024-05-28T06:18:02.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.651+0000 7fd98b200640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fd98c0d0ad0 0x7fd98c099090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:48876/0 (socket says 172.21.15.96:48876) 2024-05-28T06:18:02.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.651+0000 7fd98b200640 1 -- 172.21.15.96:0/4071755679 learned_addr learned my addr 172.21.15.96:0/4071755679 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:18:02.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.651+0000 7fd98a800640 1 --2- 172.21.15.96:0/4071755679 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd98c0b4e10 0x7fd98c0cde30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:02.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.651+0000 7fd989e00640 1 --2- 172.21.15.96:0/4071755679 >> v2:172.21.15.123:3300/0 conn(0x7fd98c0ce370 0x7fd98c0d0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:02.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.651+0000 7fd989400640 1 -- 172.21.15.96:0/4071755679 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1832407207 0 0) 0x7fd98c0b51f0 con 0x7fd98c0d0ad0 2024-05-28T06:18:02.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.652+0000 7fd989400640 1 -- 172.21.15.96:0/4071755679 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd970003520 con 0x7fd98c0d0ad0 2024-05-28T06:18:02.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.652+0000 7fd989400640 1 -- 172.21.15.96:0/4071755679 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3837742840 0 0) 0x7fd970003520 con 0x7fd98c0d0ad0 2024-05-28T06:18:02.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.652+0000 7fd989400640 1 -- 172.21.15.96:0/4071755679 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd98c0b4e10 msgr2=0x7fd98c0cde30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:18:02.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.652+0000 7fd989400640 1 --2- 172.21.15.96:0/4071755679 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd98c0b4e10 0x7fd98c0cde30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:02.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.652+0000 7fd989400640 1 -- 172.21.15.96:0/4071755679 >> v2:172.21.15.123:3300/0 conn(0x7fd98c0ce370 msgr2=0x7fd98c0d0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:18:02.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.652+0000 7fd989400640 1 --2- 172.21.15.96:0/4071755679 >> v2:172.21.15.123:3300/0 conn(0x7fd98c0ce370 0x7fd98c0d0590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:02.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.652+0000 7fd989400640 1 -- 172.21.15.96:0/4071755679 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd98c0ad1d0 con 0x7fd98c0d0ad0 2024-05-28T06:18:02.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.652+0000 7fd989400640 1 -- 172.21.15.96:0/4071755679 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7fd984001670 con 0x7fd98c0d0ad0 2024-05-28T06:18:02.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.652+0000 7fd989400640 1 -- 172.21.15.96:0/4071755679 >> v1:172.21.15.96:6789/0 conn(0x7fd98c0d0ad0 legacy=0x7fd98c099090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:02.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.652+0000 7fd989400640 1 --2- 172.21.15.96:0/4071755679 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd98c0b4e10 0x7fd9700039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:02.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.652+0000 7fd989400640 1 --2- 172.21.15.96:0/4071755679 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd970003f80 0x7fd970004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:02.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.652+0000 7fd989400640 1 --2- 172.21.15.96:0/4071755679 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd970004d40 0x7fd9700051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:02.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.652+0000 7fd989400640 1 -- 172.21.15.96:0/4071755679 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd98c0d0f20 con 0x7fd970003f80 2024-05-28T06:18:02.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.652+0000 7fd989400640 1 -- 172.21.15.96:0/4071755679 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd98c0aef60 con 0x7fd970004d40 2024-05-28T06:18:02.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.652+0000 7fd989400640 1 -- 172.21.15.96:0/4071755679 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fd97000dc80 con 0x7fd98c0b4e10 2024-05-28T06:18:02.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.652+0000 7fd989e00640 1 --2- 172.21.15.96:0/4071755679 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd970003f80 0x7fd970004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:02.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.653+0000 7fd98a800640 1 --2- 172.21.15.96:0/4071755679 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd98c0b4e10 0x7fd9700039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:02.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.653+0000 7fd98b200640 1 --2- 172.21.15.96:0/4071755679 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd970004d40 0x7fd9700051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:02.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.653+0000 7fd989e00640 1 -- 172.21.15.96:0/4071755679 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd98c0b4e10 msgr2=0x7fd9700039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:18:02.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.653+0000 7fd989e00640 1 --2- 172.21.15.96:0/4071755679 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd98c0b4e10 0x7fd9700039b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:02.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.653+0000 7fd989e00640 1 -- 172.21.15.96:0/4071755679 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd970004d40 msgr2=0x7fd9700051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:18:02.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.653+0000 7fd989e00640 1 --2- 172.21.15.96:0/4071755679 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd970004d40 0x7fd9700051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:02.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.653+0000 7fd989e00640 1 -- 172.21.15.96:0/4071755679 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd970003520 con 0x7fd970003f80 2024-05-28T06:18:02.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.653+0000 7fd98b200640 1 --2- 172.21.15.96:0/4071755679 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd970004d40 0x7fd9700051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:18:02.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.654+0000 7fd989e00640 1 --2- 172.21.15.96:0/4071755679 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd970003f80 0x7fd970004800 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7fd974009a90 tx=0x7fd970005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d9b1a448d475761 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:02.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.654+0000 7fd989400640 1 -- 172.21.15.96:0/4071755679 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd974036070 con 0x7fd970003f80 2024-05-28T06:18:02.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.654+0000 7fd989400640 1 -- 172.21.15.96:0/4071755679 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd974031400 con 0x7fd970003f80 2024-05-28T06:18:02.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.654+0000 7fd989400640 1 -- 172.21.15.96:0/4071755679 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd974009690 con 0x7fd970003f80 2024-05-28T06:18:02.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.655+0000 7fd991400640 1 -- 172.21.15.96:0/4071755679 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd970003f80 msgr2=0x7fd970004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:02.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.655+0000 7fd991400640 1 --2- 172.21.15.96:0/4071755679 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd970003f80 0x7fd970004800 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7fd974009a90 tx=0x7fd970005830 comp rx=0 tx=0).stop 2024-05-28T06:18:02.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.655+0000 7fd98b200640 1 -- 172.21.15.96:0/4071755679 reap_dead start 2024-05-28T06:18:02.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.655+0000 7fd991400640 1 -- 172.21.15.96:0/4071755679 shutdown_connections 2024-05-28T06:18:02.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.655+0000 7fd991400640 1 -- 172.21.15.96:0/4071755679 >> 172.21.15.96:0/4071755679 conn(0x7fd98c06bae0 msgr2=0x7fd98c06bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:18:02.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.655+0000 7fd991400640 1 -- 172.21.15.96:0/4071755679 shutdown_connections 2024-05-28T06:18:02.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.655+0000 7fd991400640 1 -- 172.21.15.96:0/4071755679 wait complete. 2024-05-28T06:18:02.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.657+0000 7fd991400640 1 Processor -- start 2024-05-28T06:18:02.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.657+0000 7fd991400640 1 -- start start 2024-05-28T06:18:02.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.658+0000 7fd991400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd970003f80 0x7fd98c0cb520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:02.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.658+0000 7fd991400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd98c155850 0x7fd98c1f4330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:02.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.658+0000 7fd991400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd98c071b30 0x7fd98c071fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:02.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.658+0000 7fd991400640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd98c0ce880 con 0x7fd970003f80 2024-05-28T06:18:02.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.658+0000 7fd991400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd98c0ad750 con 0x7fd98c155850 2024-05-28T06:18:02.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.658+0000 7fd991400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fd98c0cea00 con 0x7fd98c071b30 2024-05-28T06:18:02.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.658+0000 7fd98a800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd970003f80 0x7fd98c0cb520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:02.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.658+0000 7fd98a800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd970003f80 0x7fd98c0cb520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:60236/0 (socket says 172.21.15.96:60236) 2024-05-28T06:18:02.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.658+0000 7fd98a800640 1 -- 172.21.15.96:0/3391058663 learned_addr learned my addr 172.21.15.96:0/3391058663 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:18:02.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.658+0000 7fd98a800640 1 -- 172.21.15.96:0/3391058663 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd98c071b30 msgr2=0x7fd98c071fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:02.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.658+0000 7fd98a800640 1 --2- 172.21.15.96:0/3391058663 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd98c071b30 0x7fd98c071fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:02.659 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.658+0000 7fd98a800640 1 -- 172.21.15.96:0/3391058663 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd98c155850 msgr2=0x7fd98c1f4330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:02.659 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.658+0000 7fd98a800640 1 --2- 172.21.15.96:0/3391058663 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd98c155850 0x7fd98c1f4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:02.659 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.658+0000 7fd98a800640 1 -- 172.21.15.96:0/3391058663 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd98c0c74b0 con 0x7fd970003f80 2024-05-28T06:18:02.659 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.659+0000 7fd98a800640 1 --2- 172.21.15.96:0/3391058663 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd970003f80 0x7fd98c0cb520 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7fd97c00f350 tx=0x7fd97c00f810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:02.659 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.659+0000 7fd982a00640 1 -- 172.21.15.96:0/3391058663 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd97c01c070 con 0x7fd970003f80 2024-05-28T06:18:02.659 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.659+0000 7fd982a00640 1 -- 172.21.15.96:0/3391058663 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd97c017620 con 0x7fd970003f80 2024-05-28T06:18:02.659 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.659+0000 7fd991400640 1 -- 172.21.15.96:0/3391058663 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd98c155cf0 con 0x7fd970003f80 2024-05-28T06:18:02.659 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.659+0000 7fd982a00640 1 -- 172.21.15.96:0/3391058663 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd97c017a10 con 0x7fd970003f80 2024-05-28T06:18:02.659 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.659+0000 7fd991400640 1 -- 172.21.15.96:0/3391058663 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd98c1f4870 con 0x7fd970003f80 2024-05-28T06:18:02.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.660+0000 7fd991400640 1 -- 172.21.15.96:0/3391058663 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd98c0689e0 con 0x7fd970003f80 2024-05-28T06:18:02.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.661+0000 7fd982a00640 1 -- 172.21.15.96:0/3391058663 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 30) ==== 92621+0+0 (secure 0 0 0) 0x7fd97c01f070 con 0x7fd970003f80 2024-05-28T06:18:02.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.662+0000 7fd982a00640 1 --2- 172.21.15.96:0/3391058663 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd96806da90 0x7fd96806ff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:02.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.662+0000 7fd982a00640 1 -- 172.21.15.96:0/3391058663 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 4971+0+0 (secure 0 0 0) 0x7fd97c08ac00 con 0x7fd970003f80 2024-05-28T06:18:02.661 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.663+0000 7fd989e00640 1 --2- 172.21.15.96:0/3391058663 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd96806da90 0x7fd96806ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:02.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.664+0000 7fd989e00640 1 --2- 172.21.15.96:0/3391058663 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd96806da90 0x7fd96806ff50 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fd974033000 tx=0x7fd97402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:02.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.667+0000 7fd982a00640 1 -- 172.21.15.96:0/3391058663 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd97c012070 con 0x7fd970003f80 2024-05-28T06:18:02.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:02.946+0000 7fd991400640 1 -- 172.21.15.96:0/3391058663 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgressReporter.test_stats_are_printed"]} v 0) -- 0x7fd98c06f540 con 0x7fd970003f80 2024-05-28T06:18:03.025 DEBUG:teuthology.orchestra.run.smithi096:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:18:03.033 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:18:03.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:03.550+0000 7fd982a00640 1 -- 172.21.15.96:0/3391058663 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgressReporter.test_stats_are_printed"]}]=0 v614) ==== 156+0+0 (secure 0 0 0) 0x7fd97c05a1a0 con 0x7fd970003f80 2024-05-28T06:18:03.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:03.554+0000 7fd991400640 1 -- 172.21.15.96:0/3391058663 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd96806da90 msgr2=0x7fd96806ff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:03.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:03.554+0000 7fd991400640 1 --2- 172.21.15.96:0/3391058663 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd96806da90 0x7fd96806ff50 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fd974033000 tx=0x7fd97402e000 comp rx=0 tx=0).stop 2024-05-28T06:18:03.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:03.554+0000 7fd991400640 1 -- 172.21.15.96:0/3391058663 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd970003f80 msgr2=0x7fd98c0cb520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:03.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:03.554+0000 7fd991400640 1 --2- 172.21.15.96:0/3391058663 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd970003f80 0x7fd98c0cb520 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7fd97c00f350 tx=0x7fd97c00f810 comp rx=0 tx=0).stop 2024-05-28T06:18:03.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:03.554+0000 7fd991400640 1 -- 172.21.15.96:0/3391058663 shutdown_connections 2024-05-28T06:18:03.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:03.554+0000 7fd991400640 1 --2- 172.21.15.96:0/3391058663 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd96806da90 0x7fd96806ff50 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:03.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:03.554+0000 7fd991400640 1 --2- 172.21.15.96:0/3391058663 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd98c071b30 0x7fd98c071fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:03.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:03.554+0000 7fd991400640 1 --2- 172.21.15.96:0/3391058663 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd98c155850 0x7fd98c1f4330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:03.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:03.554+0000 7fd991400640 1 --2- 172.21.15.96:0/3391058663 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd970003f80 0x7fd98c0cb520 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:03.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:03.554+0000 7fd991400640 1 -- 172.21.15.96:0/3391058663 >> 172.21.15.96:0/3391058663 conn(0x7fd98c06bae0 msgr2=0x7fd98c0b3a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:18:03.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:03.555+0000 7fd991400640 1 -- 172.21.15.96:0/3391058663 shutdown_connections 2024-05-28T06:18:03.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:03.555+0000 7fd991400640 1 -- 172.21.15.96:0/3391058663 wait complete. 2024-05-28T06:18:03.579 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-05-28T06:18:03.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:03.783+0000 7f7718400640 1 Processor -- start 2024-05-28T06:18:03.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:03.783+0000 7f7718400640 1 -- start start 2024-05-28T06:18:03.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:03.785+0000 7f7718400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f77100d0640 0x7f77100d0a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:03.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:03.785+0000 7f7718400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f771015b6f0 0x7f771006e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:03.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:03.785+0000 7f7718400640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77100ade90 con 0x7f77100725e0 2024-05-28T06:18:03.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:03.785+0000 7f7718400640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f7710071c40 con 0x7f77100d0640 2024-05-28T06:18:03.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:03.785+0000 7f7718400640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f77100d0f60 con 0x7f771015b6f0 2024-05-28T06:18:03.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:03.785+0000 7f7715a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f77100d0640 0x7f77100d0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:03.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:03.785+0000 7f7715a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f77100d0640 0x7f77100d0a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:60240/0 (socket says 172.21.15.96:60240) 2024-05-28T06:18:03.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:03.785+0000 7f7715a00640 1 -- 172.21.15.96:0/3702718202 learned_addr learned my addr 172.21.15.96:0/3702718202 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:18:03.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:03.786+0000 7f7715a00640 1 -- 172.21.15.96:0/3702718202 >> v2:172.21.15.123:3300/0 conn(0x7f771015b6f0 msgr2=0x7f771006e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:18:03.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:03.786+0000 7f7715a00640 1 --2- 172.21.15.96:0/3702718202 >> v2:172.21.15.123:3300/0 conn(0x7f771015b6f0 0x7f771006e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:03.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:03.786+0000 7f7715a00640 1 -- 172.21.15.96:0/3702718202 >> v1:172.21.15.123:6789/0 conn(0x7f77100725e0 legacy=0x7f77100b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:18:03.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:03.786+0000 7f7715a00640 1 -- 172.21.15.96:0/3702718202 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7710071e70 con 0x7f77100d0640 2024-05-28T06:18:03.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:03.787+0000 7f7715a00640 1 --2- 172.21.15.96:0/3702718202 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f77100d0640 0x7f77100d0a20 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7f77100d0220 tx=0x7f770402c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ba410fc721e7c7e server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:03.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:03.787+0000 7f770fe00640 1 -- 172.21.15.96:0/3702718202 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77040325a0 con 0x7f77100d0640 2024-05-28T06:18:03.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:03.787+0000 7f770fe00640 1 -- 172.21.15.96:0/3702718202 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7704038070 con 0x7f77100d0640 2024-05-28T06:18:03.786 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:03.787+0000 7f770fe00640 1 -- 172.21.15.96:0/3702718202 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7704032960 con 0x7f77100d0640 2024-05-28T06:18:03.786 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:03.788+0000 7f7718400640 1 -- 172.21.15.96:0/3702718202 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f77100d0640 msgr2=0x7f77100d0a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:03.787 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:03.788+0000 7f7718400640 1 --2- 172.21.15.96:0/3702718202 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f77100d0640 0x7f77100d0a20 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7f77100d0220 tx=0x7f770402c8e0 comp rx=0 tx=0).stop 2024-05-28T06:18:03.787 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:03.788+0000 7f7718400640 1 -- 172.21.15.96:0/3702718202 shutdown_connections 2024-05-28T06:18:03.787 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:03.788+0000 7f7718400640 1 --2- 172.21.15.96:0/3702718202 >> v2:172.21.15.123:3300/0 conn(0x7f771015b6f0 0x7f771006e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:03.787 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:03.788+0000 7f7718400640 1 --2- 172.21.15.96:0/3702718202 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f77100d0640 0x7f77100d0a20 unknown :-1 s=CLOSED pgs=1357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:03.787 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:03.788+0000 7f7718400640 1 -- 172.21.15.96:0/3702718202 >> 172.21.15.96:0/3702718202 conn(0x7f771006bd20 msgr2=0x7f771006c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:18:03.787 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:03.788+0000 7f7718400640 1 -- 172.21.15.96:0/3702718202 shutdown_connections 2024-05-28T06:18:03.787 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:03.788+0000 7f7718400640 1 -- 172.21.15.96:0/3702718202 wait complete. 2024-05-28T06:18:03.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:03.790+0000 7f7718400640 1 Processor -- start 2024-05-28T06:18:03.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:03.790+0000 7f7718400640 1 -- start start 2024-05-28T06:18:03.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:03.792+0000 7f7718400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f77100725e0 0x7f77101485d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:03.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:03.792+0000 7f7718400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f77100d0640 0x7f771014b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:03.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:03.792+0000 7f7718400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f771015b6f0 0x7f77101f0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:03.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:03.792+0000 7f7718400640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f771006e890 con 0x7f77100d0640 2024-05-28T06:18:03.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:03.792+0000 7f7718400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f77100ae130 con 0x7f771015b6f0 2024-05-28T06:18:03.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:03.792+0000 7f7718400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f77100af660 con 0x7f77100725e0 2024-05-28T06:18:03.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:03.792+0000 7f7715000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f77100d0640 0x7f771014b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:03.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:03.792+0000 7f7715000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f77100d0640 0x7f771014b330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:60246/0 (socket says 172.21.15.96:60246) 2024-05-28T06:18:03.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:03.792+0000 7f7715000640 1 -- 172.21.15.96:0/1616643222 learned_addr learned my addr 172.21.15.96:0/1616643222 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:18:03.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:03.792+0000 7f7715000640 1 -- 172.21.15.96:0/1616643222 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f77100725e0 msgr2=0x7f77101485d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:03.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:03.792+0000 7f7715000640 1 --2- 172.21.15.96:0/1616643222 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f77100725e0 0x7f77101485d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:03.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:03.792+0000 7f7715000640 1 -- 172.21.15.96:0/1616643222 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f771015b6f0 msgr2=0x7f77101f0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:03.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:03.792+0000 7f7715000640 1 --2- 172.21.15.96:0/1616643222 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f771015b6f0 0x7f77101f0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:03.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:03.792+0000 7f7715000640 1 -- 172.21.15.96:0/1616643222 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7710148e60 con 0x7f77100d0640 2024-05-28T06:18:03.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:03.793+0000 7f7715000640 1 --2- 172.21.15.96:0/1616643222 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f77100d0640 0x7f771014b330 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f76fc00b440 tx=0x7f76fc00b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:03.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:03.793+0000 7f770d600640 1 -- 172.21.15.96:0/1616643222 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76fc015020 con 0x7f77100d0640 2024-05-28T06:18:03.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:03.793+0000 7f770d600640 1 -- 172.21.15.96:0/1616643222 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76fc011890 con 0x7f77100d0640 2024-05-28T06:18:03.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:03.793+0000 7f770d600640 1 -- 172.21.15.96:0/1616643222 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76fc011c80 con 0x7f77100d0640 2024-05-28T06:18:03.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:03.793+0000 7f7718400640 1 -- 172.21.15.96:0/1616643222 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f771014a570 con 0x7f77100d0640 2024-05-28T06:18:03.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:03.793+0000 7f7718400640 1 -- 172.21.15.96:0/1616643222 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f771014ab30 con 0x7f77100d0640 2024-05-28T06:18:03.793 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:03.794+0000 7f7718400640 1 -- 172.21.15.96:0/1616643222 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77100689e0 con 0x7f77100d0640 2024-05-28T06:18:03.793 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:03.795+0000 7f770d600640 1 -- 172.21.15.96:0/1616643222 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 30) ==== 92621+0+0 (secure 0 0 0) 0x7f76fc018020 con 0x7f77100d0640 2024-05-28T06:18:03.794 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:03.796+0000 7f770d600640 1 --2- 172.21.15.96:0/1616643222 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f76f006da60 0x7f76f006ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:03.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:03.796+0000 7f770d600640 1 -- 172.21.15.96:0/1616643222 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 4971+0+0 (secure 0 0 0) 0x7f76fc08ab90 con 0x7f77100d0640 2024-05-28T06:18:03.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:03.797+0000 7f7715a00640 1 --2- 172.21.15.96:0/1616643222 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f76f006da60 0x7f76f006ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:03.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:03.798+0000 7f7715a00640 1 --2- 172.21.15.96:0/1616643222 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f76f006da60 0x7f76f006ff20 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f7704037000 tx=0x7f7704031000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:03.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:03.801+0000 7f770d600640 1 -- 172.21.15.96:0/1616643222 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f76fc01f020 con 0x7f77100d0640 2024-05-28T06:18:04.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.002+0000 7f7718400640 1 -- 172.21.15.96:0/1616643222 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f77100cd750 con 0x7f77100d0640 2024-05-28T06:18:04.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.009+0000 7f770d600640 1 -- 172.21.15.96:0/1616643222 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v22)=0 v22) ==== 122+0+0 (secure 0 0 0) 0x7f76fc00d040 con 0x7f77100d0640 2024-05-28T06:18:04.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.012+0000 7f7718400640 1 -- 172.21.15.96:0/1616643222 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f76f006da60 msgr2=0x7f76f006ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:04.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.012+0000 7f7718400640 1 --2- 172.21.15.96:0/1616643222 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f76f006da60 0x7f76f006ff20 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f7704037000 tx=0x7f7704031000 comp rx=0 tx=0).stop 2024-05-28T06:18:04.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.012+0000 7f7718400640 1 -- 172.21.15.96:0/1616643222 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f77100d0640 msgr2=0x7f771014b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:04.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.012+0000 7f7718400640 1 --2- 172.21.15.96:0/1616643222 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f77100d0640 0x7f771014b330 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f76fc00b440 tx=0x7f76fc00b900 comp rx=0 tx=0).stop 2024-05-28T06:18:04.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.012+0000 7f7718400640 1 -- 172.21.15.96:0/1616643222 shutdown_connections 2024-05-28T06:18:04.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.012+0000 7f7718400640 1 --2- 172.21.15.96:0/1616643222 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f76f006da60 0x7f76f006ff20 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:04.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.012+0000 7f7718400640 1 --2- 172.21.15.96:0/1616643222 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f771015b6f0 0x7f77101f0bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:04.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.012+0000 7f7718400640 1 --2- 172.21.15.96:0/1616643222 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f77100d0640 0x7f771014b330 unknown :-1 s=CLOSED pgs=1358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:04.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.012+0000 7f7718400640 1 --2- 172.21.15.96:0/1616643222 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f77100725e0 0x7f77101485d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:04.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.012+0000 7f7718400640 1 -- 172.21.15.96:0/1616643222 >> 172.21.15.96:0/1616643222 conn(0x7f771006bd20 msgr2=0x7f77100b1150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:18:04.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.012+0000 7f7718400640 1 -- 172.21.15.96:0/1616643222 shutdown_connections 2024-05-28T06:18:04.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.012+0000 7f7718400640 1 -- 172.21.15.96:0/1616643222 wait complete. 2024-05-28T06:18:04.030 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-05-28T06:18:04.030 DEBUG:teuthology.orchestra.run.smithi096:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-05-28T06:18:04.136 DEBUG:teuthology.orchestra.run.smithi123:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-05-28T06:18:04.248 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-05-28T06:18:04.249 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:18:04.249 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:18:04.306 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052731,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248996k,nr_inodes=812249,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-28T06:18:04.306 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-05-28T06:18:04.306 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:18:04.362 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052731,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248996k,nr_inodes=812249,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-05-28T06:18:04.362 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:18:04.362 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:18:04.419 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052731,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248996k,nr_inodes=812249,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-28T06:18:04.419 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-05-28T06:18:04.419 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:18:04.475 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052731,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248996k,nr_inodes=812249,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-05-28T06:18:04.476 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-05-28T06:18:04.476 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-28T06:18:04.683 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.685+0000 7fc4a9600640 1 Processor -- start 2024-05-28T06:18:04.684 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.685+0000 7fc4a9600640 1 -- start start 2024-05-28T06:18:04.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.686+0000 7fc4a9600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fc4a406ee40 0x7fc4a40ad230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:04.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.686+0000 7fc4a9600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc4a406f2f0 0x7fc4a40ad770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:04.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.686+0000 7fc4a9600640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4a4156f10 con 0x7fc4a4156b30 2024-05-28T06:18:04.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.686+0000 7fc4a9600640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fc4a406dda0 con 0x7fc4a406ee40 2024-05-28T06:18:04.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.686+0000 7fc4a9600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fc4a406e080 con 0x7fc4a406f2f0 2024-05-28T06:18:04.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.687+0000 7fc4a3e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fc4a4156b30 0x7fc4a40af6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:48886/0 (socket says 172.21.15.96:48886) 2024-05-28T06:18:04.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.687+0000 7fc4a3400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fc4a406ee40 0x7fc4a40ad230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:04.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.687+0000 7fc4a3e00640 1 -- 172.21.15.96:0/3691041297 learned_addr learned my addr 172.21.15.96:0/3691041297 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:18:04.686 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.687+0000 7fc4a8c00640 1 --2- 172.21.15.96:0/3691041297 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc4a406f2f0 0x7fc4a40ad770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:04.686 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.687+0000 7fc4a3400640 1 -- 172.21.15.96:0/3691041297 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc4a406f2f0 msgr2=0x7fc4a40ad770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:18:04.686 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.687+0000 7fc4a3400640 1 --2- 172.21.15.96:0/3691041297 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc4a406f2f0 0x7fc4a40ad770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:04.686 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.687+0000 7fc4a3400640 1 -- 172.21.15.96:0/3691041297 >> v1:172.21.15.96:6789/0 conn(0x7fc4a4156b30 legacy=0x7fc4a40af6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:04.687 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.687+0000 7fc4a2a00640 1 -- 172.21.15.96:0/3691041297 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3260116489 0 0) 0x7fc4a4156f10 con 0x7fc4a4156b30 2024-05-28T06:18:04.687 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.687+0000 7fc4a3400640 1 -- 172.21.15.96:0/3691041297 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4a406e360 con 0x7fc4a406ee40 2024-05-28T06:18:04.688 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.688+0000 7fc4a3400640 1 --2- 172.21.15.96:0/3691041297 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc4a406ee40 0x7fc4a40ad230 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7fc4900097a0 tx=0x7fc49002a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f346bc6604de7329 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:04.688 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.689+0000 7fc4a2a00640 1 -- 172.21.15.96:0/3691041297 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc490037070 con 0x7fc4a406ee40 2024-05-28T06:18:04.688 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.689+0000 7fc4a2a00640 1 -- 172.21.15.96:0/3691041297 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc49002ae50 con 0x7fc4a406ee40 2024-05-28T06:18:04.688 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.689+0000 7fc4a2a00640 1 -- 172.21.15.96:0/3691041297 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc490032660 con 0x7fc4a406ee40 2024-05-28T06:18:04.688 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.689+0000 7fc4a9600640 1 -- 172.21.15.96:0/3691041297 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc4a406ee40 msgr2=0x7fc4a40ad230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:04.688 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.689+0000 7fc4a9600640 1 --2- 172.21.15.96:0/3691041297 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc4a406ee40 0x7fc4a40ad230 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7fc4900097a0 tx=0x7fc49002a100 comp rx=0 tx=0).stop 2024-05-28T06:18:04.688 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.689+0000 7fc4a9600640 1 -- 172.21.15.96:0/3691041297 shutdown_connections 2024-05-28T06:18:04.688 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.689+0000 7fc4a9600640 1 --2- 172.21.15.96:0/3691041297 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc4a406f2f0 0x7fc4a40ad770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:04.688 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.689+0000 7fc4a9600640 1 --2- 172.21.15.96:0/3691041297 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc4a406ee40 0x7fc4a40ad230 unknown :-1 s=CLOSED pgs=1360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:04.688 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.689+0000 7fc4a9600640 1 -- 172.21.15.96:0/3691041297 >> 172.21.15.96:0/3691041297 conn(0x7fc4a40b1fc0 msgr2=0x7fc4a40b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:18:04.688 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.689+0000 7fc4a9600640 1 -- 172.21.15.96:0/3691041297 shutdown_connections 2024-05-28T06:18:04.689 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.689+0000 7fc4a9600640 1 -- 172.21.15.96:0/3691041297 wait complete. 2024-05-28T06:18:04.689 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.691+0000 7fc4a9600640 1 Processor -- start 2024-05-28T06:18:04.690 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.691+0000 7fc4a9600640 1 -- start start 2024-05-28T06:18:04.691 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.692+0000 7fc4a9600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4a406ee40 0x7fc4a4157180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:04.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.692+0000 7fc4a9600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc4a406f2f0 0x7fc4a4072c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:04.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.692+0000 7fc4a9600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc4a4156b30 0x7fc4a4158130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:04.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.692+0000 7fc4a9600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc4a40b2db0 con 0x7fc4a406f2f0 2024-05-28T06:18:04.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.692+0000 7fc4a9600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc4a40b2c30 con 0x7fc4a406ee40 2024-05-28T06:18:04.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.693+0000 7fc4a9600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fc4a40b32c0 con 0x7fc4a4156b30 2024-05-28T06:18:04.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.693+0000 7fc4a3400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc4a406f2f0 0x7fc4a4072c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:04.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.693+0000 7fc4a3400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc4a406f2f0 0x7fc4a4072c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:60264/0 (socket says 172.21.15.96:60264) 2024-05-28T06:18:04.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.693+0000 7fc4a3400640 1 -- 172.21.15.96:0/4077652930 learned_addr learned my addr 172.21.15.96:0/4077652930 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:18:04.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.693+0000 7fc4a3e00640 1 --2- 172.21.15.96:0/4077652930 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4a406ee40 0x7fc4a4157180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:04.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.693+0000 7fc4a8c00640 1 --2- 172.21.15.96:0/4077652930 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc4a4156b30 0x7fc4a4158130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:04.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.693+0000 7fc4a3400640 1 -- 172.21.15.96:0/4077652930 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc4a4156b30 msgr2=0x7fc4a4158130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:04.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.693+0000 7fc4a3400640 1 --2- 172.21.15.96:0/4077652930 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc4a4156b30 0x7fc4a4158130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:04.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.693+0000 7fc4a3400640 1 -- 172.21.15.96:0/4077652930 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4a406ee40 msgr2=0x7fc4a4157180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:04.693 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.693+0000 7fc4a3400640 1 --2- 172.21.15.96:0/4077652930 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4a406ee40 0x7fc4a4157180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:04.693 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.693+0000 7fc4a3400640 1 -- 172.21.15.96:0/4077652930 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4a4157d20 con 0x7fc4a406f2f0 2024-05-28T06:18:04.693 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.694+0000 7fc4a3400640 1 --2- 172.21.15.96:0/4077652930 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc4a406f2f0 0x7fc4a4072c20 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7fc4900097a0 tx=0x7fc4900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:04.693 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.694+0000 7fc49be00640 1 -- 172.21.15.96:0/4077652930 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc490037040 con 0x7fc4a406f2f0 2024-05-28T06:18:04.694 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.694+0000 7fc49be00640 1 -- 172.21.15.96:0/4077652930 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc490004030 con 0x7fc4a406f2f0 2024-05-28T06:18:04.694 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.694+0000 7fc49be00640 1 -- 172.21.15.96:0/4077652930 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc490004420 con 0x7fc4a406f2f0 2024-05-28T06:18:04.694 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.694+0000 7fc4a9600640 1 -- 172.21.15.96:0/4077652930 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4a414f6d0 con 0x7fc4a406f2f0 2024-05-28T06:18:04.694 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.694+0000 7fc4a9600640 1 -- 172.21.15.96:0/4077652930 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4a414fc50 con 0x7fc4a406f2f0 2024-05-28T06:18:04.694 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.695+0000 7fc4a9600640 1 -- 172.21.15.96:0/4077652930 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4a40689e0 con 0x7fc4a406f2f0 2024-05-28T06:18:04.694 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.696+0000 7fc49be00640 1 -- 172.21.15.96:0/4077652930 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 30) ==== 92621+0+0 (secure 0 0 0) 0x7fc490003610 con 0x7fc4a406f2f0 2024-05-28T06:18:04.695 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.697+0000 7fc49be00640 1 --2- 172.21.15.96:0/4077652930 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc48406da60 0x7fc48406ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:04.696 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.697+0000 7fc49be00640 1 -- 172.21.15.96:0/4077652930 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 4971+0+0 (secure 0 0 0) 0x7fc4900ab720 con 0x7fc4a406f2f0 2024-05-28T06:18:04.696 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.698+0000 7fc4a3e00640 1 --2- 172.21.15.96:0/4077652930 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc48406da60 0x7fc48406ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:04.698 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.699+0000 7fc4a3e00640 1 --2- 172.21.15.96:0/4077652930 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc48406da60 0x7fc48406ff20 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fc494000f60 tx=0x7fc494007c00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:04.700 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.702+0000 7fc49be00640 1 -- 172.21.15.96:0/4077652930 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc49004c070 con 0x7fc4a406f2f0 2024-05-28T06:18:04.943 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.945+0000 7fc4a9600640 1 -- 172.21.15.96:0/4077652930 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc4a415bc30 con 0x7fc4a406f2f0 2024-05-28T06:18:04.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.946+0000 7fc49be00640 1 -- 172.21.15.96:0/4077652930 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) ==== 94+0+3869 (secure 0 0 0) 0x7fc49007b2b0 con 0x7fc4a406f2f0 2024-05-28T06:18:04.944 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:18:04.945 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":46,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5291,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6834/3727695025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":3727695025},{"type":"v1","addr":"172.21.15.96:6834","nonce":3727695025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38},{"gid":5300,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/324418606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6835","nonce":324418606},{"type":"v1","addr":"172.21.15.123:6837","nonce":324418606}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38},{"gid":5725,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/417566829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6833","nonce":417566829},{"type":"v1","addr":"172.21.15.96:6835","nonce":417566829}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38},{"gid":6127,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6836/1357326476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":1357326476},{"type":"v1","addr":"172.21.15.123:6836","nonce":1357326476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46}],"filesystems":[]} 2024-05-28T06:18:04.945 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 46 2024-05-28T06:18:04.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.948+0000 7fc4a9600640 1 -- 172.21.15.96:0/4077652930 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc48406da60 msgr2=0x7fc48406ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:04.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.948+0000 7fc4a9600640 1 --2- 172.21.15.96:0/4077652930 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc48406da60 0x7fc48406ff20 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fc494000f60 tx=0x7fc494007c00 comp rx=0 tx=0).stop 2024-05-28T06:18:04.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.948+0000 7fc4a9600640 1 -- 172.21.15.96:0/4077652930 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc4a406f2f0 msgr2=0x7fc4a4072c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:04.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.948+0000 7fc4a9600640 1 --2- 172.21.15.96:0/4077652930 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc4a406f2f0 0x7fc4a4072c20 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7fc4900097a0 tx=0x7fc4900046c0 comp rx=0 tx=0).stop 2024-05-28T06:18:04.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.948+0000 7fc4a9600640 1 -- 172.21.15.96:0/4077652930 shutdown_connections 2024-05-28T06:18:04.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.948+0000 7fc4a9600640 1 --2- 172.21.15.96:0/4077652930 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc48406da60 0x7fc48406ff20 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:04.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.948+0000 7fc4a9600640 1 --2- 172.21.15.96:0/4077652930 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc4a4156b30 0x7fc4a4158130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:04.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.948+0000 7fc4a9600640 1 --2- 172.21.15.96:0/4077652930 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc4a406f2f0 0x7fc4a4072c20 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:04.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.948+0000 7fc4a9600640 1 --2- 172.21.15.96:0/4077652930 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4a406ee40 0x7fc4a4157180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:04.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.948+0000 7fc4a9600640 1 -- 172.21.15.96:0/4077652930 >> 172.21.15.96:0/4077652930 conn(0x7fc4a40b1fc0 msgr2=0x7fc4a415a6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:18:04.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.949+0000 7fc4a9600640 1 -- 172.21.15.96:0/4077652930 shutdown_connections 2024-05-28T06:18:04.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:04.949+0000 7fc4a9600640 1 -- 172.21.15.96:0/4077652930 wait complete. 2024-05-28T06:18:04.966 INFO:tasks.ceph.mds.a:Restarting daemon 2024-05-28T06:18:04.967 INFO:tasks.ceph.mds.a:Stopping old one... 2024-05-28T06:18:04.967 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-05-28T06:18:04.967 INFO:teuthology.orchestra.run:waiting for 300 2024-05-28T06:18:04.967 INFO:tasks.ceph.mds.c:Restarting daemon 2024-05-28T06:18:04.967 INFO:tasks.ceph.mds.c:Stopping old one... 2024-05-28T06:18:04.967 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-05-28T06:18:04.968 INFO:teuthology.orchestra.run:waiting for 300 2024-05-28T06:18:04.968 INFO:tasks.ceph.mds.b:Restarting daemon 2024-05-28T06:18:04.968 INFO:tasks.ceph.mds.b:Stopping old one... 2024-05-28T06:18:04.968 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-05-28T06:18:04.968 INFO:teuthology.orchestra.run:waiting for 300 2024-05-28T06:18:04.968 INFO:tasks.ceph.mds.d:Restarting daemon 2024-05-28T06:18:04.968 INFO:tasks.ceph.mds.d:Stopping old one... 2024-05-28T06:18:04.968 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-05-28T06:18:04.969 INFO:teuthology.orchestra.run:waiting for 300 2024-05-28T06:18:04.988 INFO:tasks.ceph.mds.b:Stopped 2024-05-28T06:18:04.988 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-05-28T06:18:04.998 INFO:tasks.ceph.mds.d:Stopped 2024-05-28T06:18:04.999 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-05-28T06:18:05.000 INFO:tasks.ceph.mds.a:Stopped 2024-05-28T06:18:05.000 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-05-28T06:18:05.001 INFO:tasks.ceph.mds.c:Stopped 2024-05-28T06:18:05.001 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-05-28T06:18:05.046 INFO:tasks.ceph.mds.b:Started 2024-05-28T06:18:05.046 DEBUG:teuthology.parallel:result is None 2024-05-28T06:18:05.049 INFO:tasks.ceph.mds.a:Started 2024-05-28T06:18:05.049 DEBUG:teuthology.parallel:result is None 2024-05-28T06:18:05.052 INFO:tasks.ceph.mds.d:Started 2024-05-28T06:18:05.052 DEBUG:teuthology.parallel:result is None 2024-05-28T06:18:05.055 INFO:tasks.ceph.mds.c:Started 2024-05-28T06:18:05.055 DEBUG:teuthology.parallel:result is None 2024-05-28T06:18:05.056 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-05-28T06:18:05.198 INFO:tasks.ceph.mds.a.smithi096.stdout:starting mds.a at 2024-05-28T06:18:05.199 INFO:tasks.ceph.mds.c.smithi096.stdout:starting mds.c at 2024-05-28T06:18:05.202 INFO:tasks.ceph.mds.b.smithi123.stdout:starting mds.b at 2024-05-28T06:18:05.205 INFO:tasks.ceph.mds.d.smithi123.stdout:starting mds.d at 2024-05-28T06:18:05.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.237+0000 7fcdca800640 1 Processor -- start 2024-05-28T06:18:05.253 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.237+0000 7fcdca800640 1 -- start start 2024-05-28T06:18:05.257 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.238+0000 7fcdca800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcdc409b140 0x7fcdc40b1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:05.257 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.238+0000 7fcdca800640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcdc40d0ef0 con 0x7fcdc40b37a0 2024-05-28T06:18:05.257 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.238+0000 7fcdca800640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcdc40d11d0 con 0x7fcdc40d0780 2024-05-28T06:18:05.257 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.238+0000 7fcdca800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fcdc40d14b0 con 0x7fcdc409b140 2024-05-28T06:18:05.258 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.238+0000 7fcdc3400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fcdc40b37a0 0x7fcdc406d950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:48926/0 (socket says 172.21.15.96:48926) 2024-05-28T06:18:05.258 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.238+0000 7fcdc3400640 1 -- 172.21.15.96:0/2048837095 learned_addr learned my addr 172.21.15.96:0/2048837095 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:18:05.258 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.238+0000 7fcdc3e00640 1 --2- 172.21.15.96:0/2048837095 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcdc409b140 0x7fcdc40b1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:05.258 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.239+0000 7fcdc2a00640 1 -- 172.21.15.96:0/2048837095 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2492501158 0 0) 0x7fcdc40d0ef0 con 0x7fcdc40b37a0 2024-05-28T06:18:05.258 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.239+0000 7fcdc2a00640 1 -- 172.21.15.96:0/2048837095 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcda8003520 con 0x7fcdc40b37a0 2024-05-28T06:18:05.258 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.239+0000 7fcdc3e00640 1 -- 172.21.15.96:0/2048837095 >> v1:172.21.15.123:6789/0 conn(0x7fcdc40d0780 legacy=0x7fcdc409aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:05.258 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.239+0000 7fcdc3e00640 1 -- 172.21.15.96:0/2048837095 >> v1:172.21.15.96:6789/0 conn(0x7fcdc40b37a0 legacy=0x7fcdc406d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:05.258 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.239+0000 7fcdc3e00640 1 -- 172.21.15.96:0/2048837095 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdc40d1790 con 0x7fcdc409b140 2024-05-28T06:18:05.258 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.240+0000 7fcdc3e00640 1 --2- 172.21.15.96:0/2048837095 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcdc409b140 0x7fcdc40b1da0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fcdbc009dc0 tx=0x7fcdbc02a760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db0b1aacffdfae2e server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:05.258 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.240+0000 7fcdc2a00640 1 -- 172.21.15.96:0/2048837095 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdbc007740 con 0x7fcdc409b140 2024-05-28T06:18:05.258 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.240+0000 7fcdc2a00640 1 -- 172.21.15.96:0/2048837095 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcdbc0078e0 con 0x7fcdc409b140 2024-05-28T06:18:05.258 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.240+0000 7fcdc2a00640 1 -- 172.21.15.96:0/2048837095 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdbc007cb0 con 0x7fcdc409b140 2024-05-28T06:18:05.259 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.241+0000 7fcdca800640 1 -- 172.21.15.96:0/2048837095 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcdc409b140 msgr2=0x7fcdc40b1da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:05.259 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.241+0000 7fcdca800640 1 --2- 172.21.15.96:0/2048837095 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcdc409b140 0x7fcdc40b1da0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fcdbc009dc0 tx=0x7fcdbc02a760 comp rx=0 tx=0).stop 2024-05-28T06:18:05.259 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.241+0000 7fcdca800640 1 -- 172.21.15.96:0/2048837095 shutdown_connections 2024-05-28T06:18:05.259 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.241+0000 7fcdca800640 1 --2- 172.21.15.96:0/2048837095 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcdc409b140 0x7fcdc40b1da0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:05.259 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.241+0000 7fcdca800640 1 -- 172.21.15.96:0/2048837095 >> 172.21.15.96:0/2048837095 conn(0x7fcdc40b3f40 msgr2=0x7fcdc40b4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:18:05.259 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.241+0000 7fcdca800640 1 -- 172.21.15.96:0/2048837095 shutdown_connections 2024-05-28T06:18:05.259 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.241+0000 7fcdca800640 1 -- 172.21.15.96:0/2048837095 wait complete. 2024-05-28T06:18:05.259 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.243+0000 7fcdca800640 1 Processor -- start 2024-05-28T06:18:05.259 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.243+0000 7fcdca800640 1 -- start start 2024-05-28T06:18:05.259 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.244+0000 7fcdca800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcdc409b140 0x7fcdc415b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:05.259 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.244+0000 7fcdca800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcdc40b37a0 0x7fcdc415d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:05.259 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.244+0000 7fcdca800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcdc40d0780 0x7fcdc415d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:05.259 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.244+0000 7fcdca800640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fcdc40b3b80 con 0x7fcdc40b37a0 2024-05-28T06:18:05.260 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.244+0000 7fcdca800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fcdc40d0bf0 con 0x7fcdc409b140 2024-05-28T06:18:05.260 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.244+0000 7fcdca800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fcdc40b3d00 con 0x7fcdc40d0780 2024-05-28T06:18:05.260 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.244+0000 7fcdbbe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcdc40b37a0 0x7fcdc415d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:05.260 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.244+0000 7fcdbbe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcdc40b37a0 0x7fcdc415d280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:60306/0 (socket says 172.21.15.96:60306) 2024-05-28T06:18:05.260 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.244+0000 7fcdbbe00640 1 -- 172.21.15.96:0/3729229223 learned_addr learned my addr 172.21.15.96:0/3729229223 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:18:05.260 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.244+0000 7fcdc3400640 1 --2- 172.21.15.96:0/3729229223 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcdc409b140 0x7fcdc415b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:05.260 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.244+0000 7fcdc3e00640 1 --2- 172.21.15.96:0/3729229223 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcdc40d0780 0x7fcdc415d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:05.260 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.244+0000 7fcdbbe00640 1 -- 172.21.15.96:0/3729229223 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcdc40d0780 msgr2=0x7fcdc415d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:05.260 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.244+0000 7fcdbbe00640 1 --2- 172.21.15.96:0/3729229223 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcdc40d0780 0x7fcdc415d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:05.260 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.244+0000 7fcdbbe00640 1 -- 172.21.15.96:0/3729229223 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcdc409b140 msgr2=0x7fcdc415b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:05.260 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.244+0000 7fcdbbe00640 1 --2- 172.21.15.96:0/3729229223 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcdc409b140 0x7fcdc415b910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:05.260 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.244+0000 7fcdbbe00640 1 -- 172.21.15.96:0/3729229223 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdc40cb400 con 0x7fcdc40b37a0 2024-05-28T06:18:05.260 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.245+0000 7fcdbbe00640 1 --2- 172.21.15.96:0/3729229223 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcdc40b37a0 0x7fcdc415d280 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7fcdac00c450 tx=0x7fcdac00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:05.260 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.245+0000 7fcdbb400640 1 -- 172.21.15.96:0/3729229223 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdac0052d0 con 0x7fcdc40b37a0 2024-05-28T06:18:05.261 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.245+0000 7fcdbb400640 1 -- 172.21.15.96:0/3729229223 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcdac005470 con 0x7fcdc40b37a0 2024-05-28T06:18:05.261 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.245+0000 7fcdbb400640 1 -- 172.21.15.96:0/3729229223 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdac012870 con 0x7fcdc40b37a0 2024-05-28T06:18:05.261 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.245+0000 7fcdca800640 1 -- 172.21.15.96:0/3729229223 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdc40cb6e0 con 0x7fcdc40b37a0 2024-05-28T06:18:05.261 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.245+0000 7fcdca800640 1 -- 172.21.15.96:0/3729229223 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdc40cbca0 con 0x7fcdc40b37a0 2024-05-28T06:18:05.261 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.247+0000 7fcdca800640 1 -- 172.21.15.96:0/3729229223 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcdc40689e0 con 0x7fcdc40b37a0 2024-05-28T06:18:05.261 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.248+0000 7fcdbb400640 1 -- 172.21.15.96:0/3729229223 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 30) ==== 92621+0+0 (secure 0 0 0) 0x7fcdac012a10 con 0x7fcdc40b37a0 2024-05-28T06:18:05.261 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.249+0000 7fcdbb400640 1 --2- 172.21.15.96:0/3729229223 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fcda006da60 0x7fcda006ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:05.261 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.249+0000 7fcdbb400640 1 -- 172.21.15.96:0/3729229223 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 4971+0+0 (secure 0 0 0) 0x7fcdac08a4e0 con 0x7fcdc40b37a0 2024-05-28T06:18:05.261 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.249+0000 7fcdc3400640 1 --2- 172.21.15.96:0/3729229223 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fcda006da60 0x7fcda006ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:05.261 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.250+0000 7fcdc3400640 1 --2- 172.21.15.96:0/3729229223 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fcda006da60 0x7fcda006ff20 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fcdb40011e0 tx=0x7fcdb4007d30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:05.261 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.253+0000 7fcdbb400640 1 -- 172.21.15.96:0/3729229223 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fcdac059a20 con 0x7fcdc40b37a0 2024-05-28T06:18:05.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.449+0000 7fcdca800640 1 -- 172.21.15.96:0/3729229223 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7fcdc40b4b30 con 0x7fcdc40b37a0 2024-05-28T06:18:05.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.583+0000 7fcdbb400640 1 -- 172.21.15.96:0/3729229223 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v83) ==== 97+0+0 (secure 0 0 0) 0x7fcdac05dd20 con 0x7fcdc40b37a0 2024-05-28T06:18:05.582 INFO:teuthology.orchestra.run.smithi096.stderr: removed all blocklist entries 2024-05-28T06:18:05.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.586+0000 7fcdca800640 1 -- 172.21.15.96:0/3729229223 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fcda006da60 msgr2=0x7fcda006ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:05.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.586+0000 7fcdca800640 1 --2- 172.21.15.96:0/3729229223 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fcda006da60 0x7fcda006ff20 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fcdb40011e0 tx=0x7fcdb4007d30 comp rx=0 tx=0).stop 2024-05-28T06:18:05.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.586+0000 7fcdca800640 1 -- 172.21.15.96:0/3729229223 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcdc40b37a0 msgr2=0x7fcdc415d280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:05.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.586+0000 7fcdca800640 1 --2- 172.21.15.96:0/3729229223 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcdc40b37a0 0x7fcdc415d280 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7fcdac00c450 tx=0x7fcdac00c910 comp rx=0 tx=0).stop 2024-05-28T06:18:05.586 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.587+0000 7fcdca800640 1 -- 172.21.15.96:0/3729229223 shutdown_connections 2024-05-28T06:18:05.586 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.587+0000 7fcdca800640 1 --2- 172.21.15.96:0/3729229223 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fcda006da60 0x7fcda006ff20 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:05.586 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.587+0000 7fcdca800640 1 --2- 172.21.15.96:0/3729229223 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcdc40d0780 0x7fcdc415d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:05.586 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.587+0000 7fcdca800640 1 --2- 172.21.15.96:0/3729229223 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcdc40b37a0 0x7fcdc415d280 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:05.586 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.587+0000 7fcdca800640 1 --2- 172.21.15.96:0/3729229223 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcdc409b140 0x7fcdc415b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:05.586 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.587+0000 7fcdca800640 1 -- 172.21.15.96:0/3729229223 >> 172.21.15.96:0/3729229223 conn(0x7fcdc40b3f40 msgr2=0x7fcdc409aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:18:05.586 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.587+0000 7fcdca800640 1 -- 172.21.15.96:0/3729229223 shutdown_connections 2024-05-28T06:18:05.586 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.587+0000 7fcdca800640 1 -- 172.21.15.96:0/3729229223 wait complete. 2024-05-28T06:18:05.606 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-05-28T06:18:05.797 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.798+0000 7f121ee00640 1 Processor -- start 2024-05-28T06:18:05.797 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.798+0000 7f121ee00640 1 -- start start 2024-05-28T06:18:05.798 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.799+0000 7f121ee00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f12180d04a0 0x7f12180d0930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:05.798 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.799+0000 7f121ee00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f12180ad130 0x7f12180ad580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:05.798 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.799+0000 7f121ee00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f121806ed10 con 0x7f121806f860 2024-05-28T06:18:05.798 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.799+0000 7f121ee00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f121815cb00 con 0x7f12180d04a0 2024-05-28T06:18:05.798 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.799+0000 7f121ee00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f12180b3cc0 con 0x7f12180ad130 2024-05-28T06:18:05.798 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.800+0000 7f1217e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f121806f860 0x7f12180b3150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:48936/0 (socket says 172.21.15.96:48936) 2024-05-28T06:18:05.798 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.800+0000 7f1217e00640 1 -- 172.21.15.96:0/291481258 learned_addr learned my addr 172.21.15.96:0/291481258 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:18:05.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.800+0000 7f121ce00640 1 --2- 172.21.15.96:0/291481258 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f12180ad130 0x7f12180ad580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:05.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.800+0000 7f1217400640 1 --2- 172.21.15.96:0/291481258 >> v2:172.21.15.123:3300/0 conn(0x7f12180d04a0 0x7f12180d0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:05.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.800+0000 7f1216a00640 1 -- 172.21.15.96:0/291481258 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3118439515 0 0) 0x7f1208001110 con 0x7f121806f860 2024-05-28T06:18:05.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.800+0000 7f1216a00640 1 -- 172.21.15.96:0/291481258 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f11fc003520 con 0x7f121806f860 2024-05-28T06:18:05.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.801+0000 7f1216a00640 1 -- 172.21.15.96:0/291481258 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3186855351 0 0) 0x7f11fc003520 con 0x7f121806f860 2024-05-28T06:18:05.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.801+0000 7f1216a00640 1 -- 172.21.15.96:0/291481258 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f12180ad130 msgr2=0x7f12180ad580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:18:05.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.801+0000 7f1216a00640 1 --2- 172.21.15.96:0/291481258 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f12180ad130 0x7f12180ad580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:05.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.801+0000 7f1216a00640 1 -- 172.21.15.96:0/291481258 >> v2:172.21.15.123:3300/0 conn(0x7f12180d04a0 msgr2=0x7f12180d0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:18:05.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.801+0000 7f1216a00640 1 --2- 172.21.15.96:0/291481258 >> v2:172.21.15.123:3300/0 conn(0x7f12180d04a0 0x7f12180d0930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:05.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.801+0000 7f1216a00640 1 -- 172.21.15.96:0/291481258 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12180b3fa0 con 0x7f121806f860 2024-05-28T06:18:05.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.801+0000 7f1216a00640 1 -- 172.21.15.96:0/291481258 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f1208001840 con 0x7f121806f860 2024-05-28T06:18:05.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.801+0000 7f1216a00640 1 -- 172.21.15.96:0/291481258 >> v1:172.21.15.96:6789/0 conn(0x7f121806f860 legacy=0x7f12180b3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:05.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.801+0000 7f1216a00640 1 --2- 172.21.15.96:0/291481258 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11fc0039b0 0x7f11fc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:05.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.801+0000 7f1216a00640 1 --2- 172.21.15.96:0/291481258 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f11fc004800 0x7f11fc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:05.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.801+0000 7f1216a00640 1 --2- 172.21.15.96:0/291481258 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f12180ad130 0x7f11fc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:05.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.801+0000 7f1216a00640 1 -- 172.21.15.96:0/291481258 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f121815cb00 con 0x7f11fc0039b0 2024-05-28T06:18:05.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.801+0000 7f1216a00640 1 -- 172.21.15.96:0/291481258 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f12180b3cc0 con 0x7f11fc004800 2024-05-28T06:18:05.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.801+0000 7f1216a00640 1 -- 172.21.15.96:0/291481258 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f11fc00b9c0 con 0x7f12180ad130 2024-05-28T06:18:05.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.801+0000 7f1217e00640 1 --2- 172.21.15.96:0/291481258 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11fc0039b0 0x7f11fc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:05.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.801+0000 7f1217400640 1 --2- 172.21.15.96:0/291481258 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f11fc004800 0x7f11fc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:05.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.802+0000 7f121ce00640 1 --2- 172.21.15.96:0/291481258 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f12180ad130 0x7f11fc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:05.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.802+0000 7f1217e00640 1 -- 172.21.15.96:0/291481258 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f12180ad130 msgr2=0x7f11fc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:18:05.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.802+0000 7f1217e00640 1 --2- 172.21.15.96:0/291481258 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f12180ad130 0x7f11fc0051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:05.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.802+0000 7f1217e00640 1 -- 172.21.15.96:0/291481258 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f11fc004800 msgr2=0x7f11fc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:18:05.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.802+0000 7f1217e00640 1 --2- 172.21.15.96:0/291481258 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f11fc004800 0x7f11fc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:05.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.802+0000 7f1217e00640 1 -- 172.21.15.96:0/291481258 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11fc003520 con 0x7f11fc0039b0 2024-05-28T06:18:05.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.803+0000 7f1217e00640 1 --2- 172.21.15.96:0/291481258 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11fc0039b0 0x7f11fc003dd0 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7f120800c3e0 tx=0x7f11fc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36e34699aad3f21e server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:05.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.803+0000 7f1216a00640 1 -- 172.21.15.96:0/291481258 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1208032020 con 0x7f11fc0039b0 2024-05-28T06:18:05.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.803+0000 7f1216a00640 1 -- 172.21.15.96:0/291481258 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1208005af0 con 0x7f11fc0039b0 2024-05-28T06:18:05.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.803+0000 7f1216a00640 1 -- 172.21.15.96:0/291481258 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f120800c0e0 con 0x7f11fc0039b0 2024-05-28T06:18:05.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.804+0000 7f121ee00640 1 -- 172.21.15.96:0/291481258 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11fc0039b0 msgr2=0x7f11fc003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:05.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.804+0000 7f121ee00640 1 --2- 172.21.15.96:0/291481258 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11fc0039b0 0x7f11fc003dd0 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7f120800c3e0 tx=0x7f11fc005830 comp rx=0 tx=0).stop 2024-05-28T06:18:05.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.804+0000 7f121ce00640 1 -- 172.21.15.96:0/291481258 reap_dead start 2024-05-28T06:18:05.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.804+0000 7f121ee00640 1 -- 172.21.15.96:0/291481258 shutdown_connections 2024-05-28T06:18:05.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.804+0000 7f121ee00640 1 -- 172.21.15.96:0/291481258 >> 172.21.15.96:0/291481258 conn(0x7f121806bae0 msgr2=0x7f121806bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:18:05.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.804+0000 7f121ee00640 1 -- 172.21.15.96:0/291481258 shutdown_connections 2024-05-28T06:18:05.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.804+0000 7f121ee00640 1 -- 172.21.15.96:0/291481258 wait complete. 2024-05-28T06:18:05.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.806+0000 7f121ee00640 1 Processor -- start 2024-05-28T06:18:05.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.806+0000 7f121ee00640 1 -- start start 2024-05-28T06:18:05.808 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.807+0000 7f121ee00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f121815b460 0x7f12180c4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:05.808 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.807+0000 7f121ee00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f121815bbf0 0x7f12180c9110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:05.808 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.807+0000 7f121ee00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f121815c120 0x7f12180c9650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:05.808 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.807+0000 7f121ee00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f12180b4840 con 0x7f121815c120 2024-05-28T06:18:05.808 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.807+0000 7f121ee00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f12180b4560 con 0x7f121815b460 2024-05-28T06:18:05.809 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.807+0000 7f121ee00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f12180b3860 con 0x7f121815bbf0 2024-05-28T06:18:05.809 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.807+0000 7f121ce00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f121815c120 0x7f12180c9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:05.809 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.807+0000 7f121ce00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f121815c120 0x7f12180c9650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:60326/0 (socket says 172.21.15.96:60326) 2024-05-28T06:18:05.809 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.807+0000 7f121ce00640 1 -- 172.21.15.96:0/138417586 learned_addr learned my addr 172.21.15.96:0/138417586 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:18:05.809 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.807+0000 7f1217e00640 1 --2- 172.21.15.96:0/138417586 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f121815b460 0x7f12180c4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:05.809 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.807+0000 7f1217400640 1 --2- 172.21.15.96:0/138417586 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f121815bbf0 0x7f12180c9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:05.809 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.807+0000 7f121ce00640 1 -- 172.21.15.96:0/138417586 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f121815bbf0 msgr2=0x7f12180c9110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:05.809 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.807+0000 7f121ce00640 1 --2- 172.21.15.96:0/138417586 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f121815bbf0 0x7f12180c9110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:05.809 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.807+0000 7f121ce00640 1 -- 172.21.15.96:0/138417586 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f121815b460 msgr2=0x7f12180c4fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:05.810 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.807+0000 7f121ce00640 1 --2- 172.21.15.96:0/138417586 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f121815b460 0x7f12180c4fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:05.810 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.807+0000 7f121ce00640 1 -- 172.21.15.96:0/138417586 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12180c6720 con 0x7f121815c120 2024-05-28T06:18:05.810 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.807+0000 7f1217e00640 1 --2- 172.21.15.96:0/138417586 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f121815b460 0x7f12180c4fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:18:05.810 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.808+0000 7f121ce00640 1 --2- 172.21.15.96:0/138417586 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f121815c120 0x7f12180c9650 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f121806c900 tx=0x7f121000f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:05.810 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.808+0000 7f120fe00640 1 -- 172.21.15.96:0/138417586 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f121001c070 con 0x7f121815c120 2024-05-28T06:18:05.810 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.808+0000 7f120fe00640 1 -- 172.21.15.96:0/138417586 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1210017400 con 0x7f121815c120 2024-05-28T06:18:05.810 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.808+0000 7f120fe00640 1 -- 172.21.15.96:0/138417586 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12100177f0 con 0x7f121815c120 2024-05-28T06:18:05.810 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.808+0000 7f121ee00640 1 -- 172.21.15.96:0/138417586 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12180c5520 con 0x7f121815c120 2024-05-28T06:18:05.810 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.808+0000 7f121ee00640 1 -- 172.21.15.96:0/138417586 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12180c4500 con 0x7f121815c120 2024-05-28T06:18:05.810 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.809+0000 7f121ee00640 1 -- 172.21.15.96:0/138417586 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12180689e0 con 0x7f121815c120 2024-05-28T06:18:05.810 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.810+0000 7f120fe00640 1 -- 172.21.15.96:0/138417586 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 30) ==== 92621+0+0 (secure 0 0 0) 0x7f1210005d10 con 0x7f121815c120 2024-05-28T06:18:05.811 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.811+0000 7f120fe00640 1 --2- 172.21.15.96:0/138417586 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f11f806da60 0x7f11f806ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:05.811 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.811+0000 7f120fe00640 1 -- 172.21.15.96:0/138417586 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 4885+0+0 (secure 0 0 0) 0x7f121008af50 con 0x7f121815c120 2024-05-28T06:18:05.811 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.811+0000 7f1217e00640 1 --2- 172.21.15.96:0/138417586 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f11f806da60 0x7f11f806ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:05.812 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.813+0000 7f1217e00640 1 --2- 172.21.15.96:0/138417586 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f11f806da60 0x7f11f806ff20 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f1208033000 tx=0x7f120803b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:05.813 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:05.815+0000 7f120fe00640 1 -- 172.21.15.96:0/138417586 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1210012070 con 0x7f121815c120 2024-05-28T06:18:06.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.086+0000 7f121ee00640 1 -- 172.21.15.96:0/138417586 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f121806cc30 con 0x7f121815c120 2024-05-28T06:18:06.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.087+0000 7f120fe00640 1 -- 172.21.15.96:0/138417586 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v18) ==== 80+0+5304 (secure 0 0 0) 0x7f121005aa80 con 0x7f121815c120 2024-05-28T06:18:06.086 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:18:06.086 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:18:06.086 INFO:teuthology.orchestra.run.smithi096.stdout: "auth_dump": [ 2024-05-28T06:18:06.086 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:18:06.086 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "mds.a", 2024-05-28T06:18:06.086 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCWc1VmyA9HHBAANJCF8I/RP5ZYviyOzo0JCQ==", 2024-05-28T06:18:06.086 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:18:06.086 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow", 2024-05-28T06:18:06.087 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow *", 2024-05-28T06:18:06.087 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow *", 2024-05-28T06:18:06.087 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:18:06.087 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:18:06.087 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:18:06.087 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:18:06.087 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "mds.b", 2024-05-28T06:18:06.087 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCWc1VmtR0EOBAA296qGHTmB/2sMztDbDJ56g==", 2024-05-28T06:18:06.087 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:18:06.087 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow", 2024-05-28T06:18:06.087 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow *", 2024-05-28T06:18:06.087 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow *", 2024-05-28T06:18:06.087 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:18:06.087 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:18:06.088 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:18:06.088 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:18:06.088 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "mds.c", 2024-05-28T06:18:06.088 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCWc1VmrLxHKhAAhkTRLDKL6RUJQGetB8JQQA==", 2024-05-28T06:18:06.088 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:18:06.088 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow", 2024-05-28T06:18:06.088 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow *", 2024-05-28T06:18:06.088 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow *", 2024-05-28T06:18:06.088 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:18:06.088 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:18:06.088 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:18:06.088 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:18:06.089 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "mds.d", 2024-05-28T06:18:06.089 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCXc1VmW2aQCxAAnkDwQzlgy39vFe2O9Wwf7Q==", 2024-05-28T06:18:06.089 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:18:06.089 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow", 2024-05-28T06:18:06.089 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow *", 2024-05-28T06:18:06.089 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow *", 2024-05-28T06:18:06.089 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:18:06.089 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:18:06.089 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:18:06.089 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:18:06.090 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "osd.0", 2024-05-28T06:18:06.090 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCbc1VmEcNAAxAAiB5AsfY2uRY1kc6h80chuw==", 2024-05-28T06:18:06.090 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:18:06.090 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow profile osd", 2024-05-28T06:18:06.090 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile osd", 2024-05-28T06:18:06.090 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:18:06.090 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:18:06.090 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:18:06.090 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:18:06.090 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "osd.1", 2024-05-28T06:18:06.090 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCec1Vm6CAaDhAA1NkYM2q4fTsgSDSwJZL3Qg==", 2024-05-28T06:18:06.090 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:18:06.090 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow profile osd", 2024-05-28T06:18:06.091 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile osd", 2024-05-28T06:18:06.091 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:18:06.091 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:18:06.091 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:18:06.091 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:18:06.091 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "osd.2", 2024-05-28T06:18:06.091 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQChc1VmKn1jFxAA+dM4gM7FR8NwBpyd833A/Q==", 2024-05-28T06:18:06.091 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:18:06.091 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow profile osd", 2024-05-28T06:18:06.091 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile osd", 2024-05-28T06:18:06.091 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:18:06.091 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:18:06.091 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:18:06.091 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:18:06.091 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "osd.3", 2024-05-28T06:18:06.092 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCkc1VmV07MIRAAHIXQGDVbL9V5QqexQ89IJA==", 2024-05-28T06:18:06.092 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:18:06.092 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow profile osd", 2024-05-28T06:18:06.092 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile osd", 2024-05-28T06:18:06.092 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:18:06.092 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:18:06.092 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:18:06.092 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:18:06.092 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "osd.4", 2024-05-28T06:18:06.092 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCqc1VmobuZMxAAlY8xhuzqlkuFrcbTouF5rA==", 2024-05-28T06:18:06.092 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:18:06.092 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow profile osd", 2024-05-28T06:18:06.092 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile osd", 2024-05-28T06:18:06.093 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:18:06.093 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:18:06.093 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:18:06.093 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:18:06.093 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "osd.5", 2024-05-28T06:18:06.093 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCuc1Vmwt3QAxAAUFj/e/wJvxljgAbcTb32Ug==", 2024-05-28T06:18:06.093 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:18:06.093 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow profile osd", 2024-05-28T06:18:06.093 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile osd", 2024-05-28T06:18:06.093 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:18:06.093 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:18:06.093 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:18:06.093 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:18:06.093 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "osd.6", 2024-05-28T06:18:06.093 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCxc1VmscerDxAAqEDAqy6/OvdfSJ+ALZ7WeQ==", 2024-05-28T06:18:06.094 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:18:06.094 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow profile osd", 2024-05-28T06:18:06.094 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile osd", 2024-05-28T06:18:06.094 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:18:06.094 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:18:06.094 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:18:06.094 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:18:06.094 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "osd.7", 2024-05-28T06:18:06.094 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQC0c1VmgxkbKhAA7dFzDTBKFmKiFFxPsyiT4Q==", 2024-05-28T06:18:06.094 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:18:06.094 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow profile osd", 2024-05-28T06:18:06.094 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile osd", 2024-05-28T06:18:06.094 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:18:06.095 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:18:06.095 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:18:06.095 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:18:06.095 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "client.0", 2024-05-28T06:18:06.095 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCXc1Vm91hbGBAASTL3HQR8XOs7wVQEHbPl/g==", 2024-05-28T06:18:06.095 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:18:06.095 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow", 2024-05-28T06:18:06.095 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow r", 2024-05-28T06:18:06.095 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-05-28T06:18:06.095 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:18:06.095 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:18:06.095 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:18:06.095 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "client.1", 2024-05-28T06:18:06.095 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCXc1VmIc77KhAAw2e0alTmVROhyCIFW4H3Sg==", 2024-05-28T06:18:06.095 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:18:06.096 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow", 2024-05-28T06:18:06.096 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow r", 2024-05-28T06:18:06.096 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow rw", 2024-05-28T06:18:06.096 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow rwx" 2024-05-28T06:18:06.096 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:18:06.096 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:18:06.096 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:18:06.096 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "client.2", 2024-05-28T06:18:06.096 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCXc1VmCS+OIRAAMV3Au6KuchLhA59QTGbv1w==", 2024-05-28T06:18:06.096 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:18:06.096 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow", 2024-05-28T06:18:06.096 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow r", 2024-05-28T06:18:06.096 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow rw", 2024-05-28T06:18:06.097 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow rwx" 2024-05-28T06:18:06.097 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:18:06.097 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:18:06.097 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:18:06.097 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "client.3", 2024-05-28T06:18:06.097 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCXc1Vm5r3pNBAAkVZ3arjywHQkEP5lmS8xqg==", 2024-05-28T06:18:06.097 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:18:06.097 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow", 2024-05-28T06:18:06.097 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow r", 2024-05-28T06:18:06.097 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow rw", 2024-05-28T06:18:06.097 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow rwx" 2024-05-28T06:18:06.097 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:18:06.097 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:18:06.097 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:18:06.098 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "client.admin", 2024-05-28T06:18:06.098 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCVc1Vm5QaXFhAAQFGKgbZ/kpGisNm2lIw0Uw==", 2024-05-28T06:18:06.098 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:18:06.098 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow *", 2024-05-28T06:18:06.098 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow *", 2024-05-28T06:18:06.098 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow *", 2024-05-28T06:18:06.098 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:18:06.098 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:18:06.098 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:18:06.099 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:18:06.099 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "mgr.x", 2024-05-28T06:18:06.099 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCWc1VmVMx/EhAAylvi37aq3hbMOhUSQWLVSg==", 2024-05-28T06:18:06.099 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:18:06.099 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow *", 2024-05-28T06:18:06.099 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile mgr", 2024-05-28T06:18:06.099 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:18:06.100 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:18:06.100 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:18:06.100 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:18:06.100 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "mgr.y", 2024-05-28T06:18:06.100 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCWc1Vmh5r7BxAAA74gn+FozehM+34eFik8kw==", 2024-05-28T06:18:06.100 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:18:06.101 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow *", 2024-05-28T06:18:06.101 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile mgr", 2024-05-28T06:18:06.101 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:18:06.101 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:18:06.101 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:18:06.101 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-05-28T06:18:06.102 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:18:06.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.090+0000 7f121ee00640 1 -- 172.21.15.96:0/138417586 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f11f806da60 msgr2=0x7f11f806ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:06.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.090+0000 7f121ee00640 1 --2- 172.21.15.96:0/138417586 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f11f806da60 0x7f11f806ff20 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f1208033000 tx=0x7f120803b040 comp rx=0 tx=0).stop 2024-05-28T06:18:06.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.090+0000 7f121ee00640 1 -- 172.21.15.96:0/138417586 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f121815c120 msgr2=0x7f12180c9650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:06.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.090+0000 7f121ee00640 1 --2- 172.21.15.96:0/138417586 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f121815c120 0x7f12180c9650 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f121806c900 tx=0x7f121000f4a0 comp rx=0 tx=0).stop 2024-05-28T06:18:06.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.090+0000 7f121ee00640 1 -- 172.21.15.96:0/138417586 shutdown_connections 2024-05-28T06:18:06.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.090+0000 7f121ee00640 1 --2- 172.21.15.96:0/138417586 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f11f806da60 0x7f11f806ff20 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:06.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.090+0000 7f121ee00640 1 --2- 172.21.15.96:0/138417586 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f121815c120 0x7f12180c9650 unknown :-1 s=CLOSED pgs=1373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:06.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.090+0000 7f121ee00640 1 --2- 172.21.15.96:0/138417586 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f121815bbf0 0x7f12180c9110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:06.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.090+0000 7f121ee00640 1 --2- 172.21.15.96:0/138417586 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f121815b460 0x7f12180c4fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:06.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.090+0000 7f121ee00640 1 -- 172.21.15.96:0/138417586 >> 172.21.15.96:0/138417586 conn(0x7f121806bae0 msgr2=0x7f12180b0f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:18:06.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.090+0000 7f121ee00640 1 -- 172.21.15.96:0/138417586 shutdown_connections 2024-05-28T06:18:06.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.090+0000 7f121ee00640 1 -- 172.21.15.96:0/138417586 wait complete. 2024-05-28T06:18:06.108 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:18:06.296 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.297+0000 7fdbb8e00640 1 Processor -- start 2024-05-28T06:18:06.296 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.298+0000 7fdbb8e00640 1 -- start start 2024-05-28T06:18:06.297 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.299+0000 7fdbb8e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fdbb406ee40 0x7fdbb40ad230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:06.297 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.299+0000 7fdbb8e00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdbb4156f10 con 0x7fdbb406f2f0 2024-05-28T06:18:06.297 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.299+0000 7fdbb8e00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdbb406dda0 con 0x7fdbb4156b30 2024-05-28T06:18:06.297 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.299+0000 7fdbb8e00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fdbb406e080 con 0x7fdbb406ee40 2024-05-28T06:18:06.297 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.299+0000 7fdbb3e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fdbb406f2f0 0x7fdbb40b32c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:48950/0 (socket says 172.21.15.96:48950) 2024-05-28T06:18:06.298 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.299+0000 7fdbb3e00640 1 -- 172.21.15.96:0/4054345955 learned_addr learned my addr 172.21.15.96:0/4054345955 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:18:06.299 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.300+0000 7fdbb2a00640 1 -- 172.21.15.96:0/4054345955 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1649519565 0 0) 0x7fdbb4156f10 con 0x7fdbb406f2f0 2024-05-28T06:18:06.300 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.300+0000 7fdbb2a00640 1 -- 172.21.15.96:0/4054345955 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdb98003520 con 0x7fdbb406f2f0 2024-05-28T06:18:06.300 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.300+0000 7fdbaaa00640 1 --2- 172.21.15.96:0/4054345955 >> v2:172.21.15.123:3300/0 conn(0x7fdbb406ee40 0x7fdbb40ad230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:06.300 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.300+0000 7fdbb2a00640 1 -- 172.21.15.96:0/4054345955 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2052710447 0 0) 0x7fdb98003520 con 0x7fdbb406f2f0 2024-05-28T06:18:06.300 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.300+0000 7fdbb2a00640 1 -- 172.21.15.96:0/4054345955 >> v2:172.21.15.123:3300/0 conn(0x7fdbb406ee40 msgr2=0x7fdbb40ad230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:18:06.300 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.300+0000 7fdbb2a00640 1 --2- 172.21.15.96:0/4054345955 >> v2:172.21.15.123:3300/0 conn(0x7fdbb406ee40 0x7fdbb40ad230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:06.300 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.300+0000 7fdbb2a00640 1 -- 172.21.15.96:0/4054345955 >> v1:172.21.15.123:6789/0 conn(0x7fdbb4156b30 legacy=0x7fdbb40af6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:18:06.300 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.300+0000 7fdbb2a00640 1 -- 172.21.15.96:0/4054345955 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbb406e360 con 0x7fdbb406f2f0 2024-05-28T06:18:06.300 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.300+0000 7fdbb2a00640 1 -- 172.21.15.96:0/4054345955 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7fdbac0019f0 con 0x7fdbb406f2f0 2024-05-28T06:18:06.300 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.300+0000 7fdbb2a00640 1 -- 172.21.15.96:0/4054345955 >> v1:172.21.15.96:6789/0 conn(0x7fdbb406f2f0 legacy=0x7fdbb40b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:06.300 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.300+0000 7fdbb2a00640 1 --2- 172.21.15.96:0/4054345955 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb98003a00 0x7fdb98003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:06.301 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.300+0000 7fdbb2a00640 1 --2- 172.21.15.96:0/4054345955 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb98004800 0x7fdb98004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:06.301 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.300+0000 7fdbb2a00640 1 --2- 172.21.15.96:0/4054345955 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdb980052e0 0x7fdb9800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:06.301 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.300+0000 7fdbb2a00640 1 -- 172.21.15.96:0/4054345955 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fdbb406e080 con 0x7fdb98004800 2024-05-28T06:18:06.301 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.300+0000 7fdbb2a00640 1 -- 172.21.15.96:0/4054345955 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fdb9800c200 con 0x7fdb98003a00 2024-05-28T06:18:06.301 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.300+0000 7fdbb2a00640 1 -- 172.21.15.96:0/4054345955 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fdb9800c400 con 0x7fdb980052e0 2024-05-28T06:18:06.301 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.300+0000 7fdbaaa00640 1 --2- 172.21.15.96:0/4054345955 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb98004800 0x7fdb98004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:06.302 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.301+0000 7fdbb3400640 1 --2- 172.21.15.96:0/4054345955 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb98003a00 0x7fdb98003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:06.302 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.301+0000 7fdbb3e00640 1 --2- 172.21.15.96:0/4054345955 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdb980052e0 0x7fdb9800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:06.302 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.301+0000 7fdbaaa00640 1 -- 172.21.15.96:0/4054345955 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdb980052e0 msgr2=0x7fdb9800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:18:06.302 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.301+0000 7fdbaaa00640 1 --2- 172.21.15.96:0/4054345955 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdb980052e0 0x7fdb9800b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:06.302 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.301+0000 7fdbaaa00640 1 -- 172.21.15.96:0/4054345955 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb98003a00 msgr2=0x7fdb98003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:18:06.302 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.301+0000 7fdbaaa00640 1 --2- 172.21.15.96:0/4054345955 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb98003a00 0x7fdb98003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:06.302 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.301+0000 7fdbaaa00640 1 -- 172.21.15.96:0/4054345955 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb98003520 con 0x7fdb98004800 2024-05-28T06:18:06.303 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.301+0000 7fdbb3e00640 1 -- 172.21.15.96:0/4054345955 reap_dead start 2024-05-28T06:18:06.308 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.302+0000 7fdbaaa00640 1 --2- 172.21.15.96:0/4054345955 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb98004800 0x7fdb98004c80 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c00ba30 tx=0x7fdb98005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4d437cf5deb1dc0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:06.308 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.302+0000 7fdbb2a00640 1 -- 172.21.15.96:0/4054345955 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb9c030070 con 0x7fdb98004800 2024-05-28T06:18:06.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.302+0000 7fdbb2a00640 1 -- 172.21.15.96:0/4054345955 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb9c02ade0 con 0x7fdb98004800 2024-05-28T06:18:06.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.302+0000 7fdbb2a00640 1 -- 172.21.15.96:0/4054345955 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb9c00b600 con 0x7fdb98004800 2024-05-28T06:18:06.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.303+0000 7fdbb8e00640 1 -- 172.21.15.96:0/4054345955 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb98004800 msgr2=0x7fdb98004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:06.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.303+0000 7fdbb8e00640 1 --2- 172.21.15.96:0/4054345955 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb98004800 0x7fdb98004c80 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c00ba30 tx=0x7fdb98005830 comp rx=0 tx=0).stop 2024-05-28T06:18:06.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.303+0000 7fdbb8e00640 1 -- 172.21.15.96:0/4054345955 shutdown_connections 2024-05-28T06:18:06.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.303+0000 7fdbb8e00640 1 --2- 172.21.15.96:0/4054345955 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb98004800 0x7fdb98004c80 unknown :-1 s=CLOSED pgs=1375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:06.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.303+0000 7fdbb8e00640 1 -- 172.21.15.96:0/4054345955 >> 172.21.15.96:0/4054345955 conn(0x7fdbb40b1fc0 msgr2=0x7fdbb40b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:18:06.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.303+0000 7fdbb8e00640 1 -- 172.21.15.96:0/4054345955 shutdown_connections 2024-05-28T06:18:06.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.303+0000 7fdbb8e00640 1 -- 172.21.15.96:0/4054345955 wait complete. 2024-05-28T06:18:06.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.305+0000 7fdbb8e00640 1 Processor -- start 2024-05-28T06:18:06.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.305+0000 7fdbb8e00640 1 -- start start 2024-05-28T06:18:06.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.306+0000 7fdbb8e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb98004800 0x7fdbb414b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:06.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.306+0000 7fdbb8e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdbb415a8f0 0x7fdbb4071bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:06.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.306+0000 7fdbb8e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdbb414ce10 0x7fdbb4072100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:06.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.306+0000 7fdbb8e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fdbb40b2db0 con 0x7fdb98004800 2024-05-28T06:18:06.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.306+0000 7fdbb8e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fdbb40b2c30 con 0x7fdbb415a8f0 2024-05-28T06:18:06.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.306+0000 7fdbb8e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fdbb40af6e0 con 0x7fdbb414ce10 2024-05-28T06:18:06.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.306+0000 7fdbb3400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb98004800 0x7fdbb414b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:06.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.306+0000 7fdbb3400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb98004800 0x7fdbb414b910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:60346/0 (socket says 172.21.15.96:60346) 2024-05-28T06:18:06.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.306+0000 7fdbb3400640 1 -- 172.21.15.96:0/705807947 learned_addr learned my addr 172.21.15.96:0/705807947 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:18:06.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.307+0000 7fdbb3400640 1 -- 172.21.15.96:0/705807947 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdbb414ce10 msgr2=0x7fdbb4072100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:06.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.307+0000 7fdbb3e00640 1 --2- 172.21.15.96:0/705807947 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdbb414ce10 0x7fdbb4072100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:06.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.307+0000 7fdbb3400640 1 --2- 172.21.15.96:0/705807947 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdbb414ce10 0x7fdbb4072100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:06.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.307+0000 7fdbb3400640 1 -- 172.21.15.96:0/705807947 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdbb415a8f0 msgr2=0x7fdbb4071bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:06.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.307+0000 7fdbaaa00640 1 --2- 172.21.15.96:0/705807947 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdbb415a8f0 0x7fdbb4071bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:06.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.307+0000 7fdbb3400640 1 --2- 172.21.15.96:0/705807947 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdbb415a8f0 0x7fdbb4071bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:06.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.307+0000 7fdbb3400640 1 -- 172.21.15.96:0/705807947 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbb4072c70 con 0x7fdb98004800 2024-05-28T06:18:06.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.307+0000 7fdbb3400640 1 --2- 172.21.15.96:0/705807947 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb98004800 0x7fdbb414b910 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7fdba400e440 tx=0x7fdba400e840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:06.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.307+0000 7fdbabe00640 1 -- 172.21.15.96:0/705807947 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdba4019070 con 0x7fdb98004800 2024-05-28T06:18:06.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.307+0000 7fdbabe00640 1 -- 172.21.15.96:0/705807947 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdba40052e0 con 0x7fdb98004800 2024-05-28T06:18:06.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.308+0000 7fdbb8e00640 1 -- 172.21.15.96:0/705807947 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdbb40734d0 con 0x7fdb98004800 2024-05-28T06:18:06.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.308+0000 7fdbabe00640 1 -- 172.21.15.96:0/705807947 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdba4003830 con 0x7fdb98004800 2024-05-28T06:18:06.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.308+0000 7fdbb8e00640 1 -- 172.21.15.96:0/705807947 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbb40731f0 con 0x7fdb98004800 2024-05-28T06:18:06.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.309+0000 7fdbb8e00640 1 -- 172.21.15.96:0/705807947 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdbb40689e0 con 0x7fdb98004800 2024-05-28T06:18:06.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.310+0000 7fdbabe00640 1 -- 172.21.15.96:0/705807947 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 30) ==== 92621+0+0 (secure 0 0 0) 0x7fdba401c070 con 0x7fdb98004800 2024-05-28T06:18:06.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.310+0000 7fdbabe00640 1 --2- 172.21.15.96:0/705807947 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdb9006da90 0x7fdb9006ff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:06.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.311+0000 7fdbabe00640 1 -- 172.21.15.96:0/705807947 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 4885+0+0 (secure 0 0 0) 0x7fdba4089f30 con 0x7fdb98004800 2024-05-28T06:18:06.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.311+0000 7fdbaaa00640 1 --2- 172.21.15.96:0/705807947 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdb9006da90 0x7fdb9006ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:06.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.312+0000 7fdbaaa00640 1 --2- 172.21.15.96:0/705807947 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdb9006da90 0x7fdb9006ff50 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c00ba00 tx=0x7fdb9c030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:06.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.315+0000 7fdbabe00640 1 -- 172.21.15.96:0/705807947 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fdba4059410 con 0x7fdb98004800 2024-05-28T06:18:06.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.502+0000 7fdbb8e00640 1 -- 172.21.15.96:0/705807947 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdbb415b110 con 0x7fdb98004800 2024-05-28T06:18:06.502 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.503+0000 7fdbabe00640 1 -- 172.21.15.96:0/705807947 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+12947 (secure 0 0 0) 0x7fdba405d710 con 0x7fdb98004800 2024-05-28T06:18:06.502 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:18:06.503 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":85,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:18:05.595494+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:18:06.505 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.506+0000 7fdbb8e00640 1 -- 172.21.15.96:0/705807947 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdb9006da90 msgr2=0x7fdb9006ff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:06.505 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.506+0000 7fdbb8e00640 1 --2- 172.21.15.96:0/705807947 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdb9006da90 0x7fdb9006ff50 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c00ba00 tx=0x7fdb9c030000 comp rx=0 tx=0).stop 2024-05-28T06:18:06.506 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.506+0000 7fdbb8e00640 1 -- 172.21.15.96:0/705807947 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb98004800 msgr2=0x7fdbb414b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:06.506 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.506+0000 7fdbb8e00640 1 --2- 172.21.15.96:0/705807947 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb98004800 0x7fdbb414b910 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7fdba400e440 tx=0x7fdba400e840 comp rx=0 tx=0).stop 2024-05-28T06:18:06.506 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.506+0000 7fdbb8e00640 1 -- 172.21.15.96:0/705807947 shutdown_connections 2024-05-28T06:18:06.506 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.506+0000 7fdbb8e00640 1 --2- 172.21.15.96:0/705807947 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdb9006da90 0x7fdb9006ff50 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:06.506 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.506+0000 7fdbb8e00640 1 --2- 172.21.15.96:0/705807947 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdbb414ce10 0x7fdbb4072100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:06.506 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.506+0000 7fdbb8e00640 1 --2- 172.21.15.96:0/705807947 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdbb415a8f0 0x7fdbb4071bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:06.507 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.506+0000 7fdbb8e00640 1 --2- 172.21.15.96:0/705807947 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb98004800 0x7fdbb414b910 unknown :-1 s=CLOSED pgs=1376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:06.507 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.506+0000 7fdbb8e00640 1 -- 172.21.15.96:0/705807947 >> 172.21.15.96:0/705807947 conn(0x7fdbb40b1fc0 msgr2=0x7fdbb40ad230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:18:06.507 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.506+0000 7fdbb8e00640 1 -- 172.21.15.96:0/705807947 shutdown_connections 2024-05-28T06:18:06.507 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.506+0000 7fdbb8e00640 1 -- 172.21.15.96:0/705807947 wait complete. 2024-05-28T06:18:06.523 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-28T06:03:59.517102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-28T06:18:06.524 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-28T06:18:06.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.721+0000 7f3e14800640 1 Processor -- start 2024-05-28T06:18:06.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.721+0000 7f3e14800640 1 -- start start 2024-05-28T06:18:06.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.723+0000 7f3e14800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f3e0c0ba070 0x7f3e0c0b6160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:06.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.723+0000 7f3e14800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f3e0c0b66a0 0x7f3e0c0b7880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:06.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.723+0000 7f3e14800640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e0c0d32e0 con 0x7f3e0c15e940 2024-05-28T06:18:06.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.723+0000 7f3e14800640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f3e0c0b6b10 con 0x7f3e0c0ba070 2024-05-28T06:18:06.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.723+0000 7f3e14800640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f3e0c15ed20 con 0x7f3e0c0b66a0 2024-05-28T06:18:06.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.723+0000 7f3e11400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f3e0c0ba070 0x7f3e0c0b6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:06.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.723+0000 7f3e11400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f3e0c0ba070 0x7f3e0c0b6160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:60362/0 (socket says 172.21.15.96:60362) 2024-05-28T06:18:06.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.723+0000 7f3e11400640 1 -- 172.21.15.96:0/88463033 learned_addr learned my addr 172.21.15.96:0/88463033 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:18:06.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.724+0000 7f3e12800640 1 --2- 172.21.15.96:0/88463033 >> v2:172.21.15.123:3300/0 conn(0x7f3e0c0b66a0 0x7f3e0c0b7880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:06.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.724+0000 7f3e11400640 1 -- 172.21.15.96:0/88463033 >> v2:172.21.15.123:3300/0 conn(0x7f3e0c0b66a0 msgr2=0x7f3e0c0b7880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:18:06.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.724+0000 7f3e11400640 1 --2- 172.21.15.96:0/88463033 >> v2:172.21.15.123:3300/0 conn(0x7f3e0c0b66a0 0x7f3e0c0b7880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:06.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.724+0000 7f3e11400640 1 -- 172.21.15.96:0/88463033 >> v1:172.21.15.96:6789/0 conn(0x7f3e0c15e940 legacy=0x7f3e0c0d2b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:06.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.724+0000 7f3e11400640 1 -- 172.21.15.96:0/88463033 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e0c0d35c0 con 0x7f3e0c0ba070 2024-05-28T06:18:06.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.724+0000 7f3e12800640 1 --2- 172.21.15.96:0/88463033 >> v2:172.21.15.123:3300/0 conn(0x7f3e0c0b66a0 0x7f3e0c0b7880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:18:06.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.725+0000 7f3e11400640 1 --2- 172.21.15.96:0/88463033 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3e0c0ba070 0x7f3e0c0b6160 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f3e0c0b7650 tx=0x7f3dfc02a2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aef51a9dff7b4c53 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:06.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.725+0000 7f3e10a00640 1 -- 172.21.15.96:0/88463033 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dfc004030 con 0x7f3e0c0ba070 2024-05-28T06:18:06.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.725+0000 7f3e10a00640 1 -- 172.21.15.96:0/88463033 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3dfc0041d0 con 0x7f3e0c0ba070 2024-05-28T06:18:06.725 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.725+0000 7f3e10a00640 1 -- 172.21.15.96:0/88463033 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dfc003610 con 0x7f3e0c0ba070 2024-05-28T06:18:06.725 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.726+0000 7f3e14800640 1 -- 172.21.15.96:0/88463033 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3e0c0ba070 msgr2=0x7f3e0c0b6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:06.725 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.726+0000 7f3e14800640 1 --2- 172.21.15.96:0/88463033 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3e0c0ba070 0x7f3e0c0b6160 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f3e0c0b7650 tx=0x7f3dfc02a2a0 comp rx=0 tx=0).stop 2024-05-28T06:18:06.725 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.726+0000 7f3e14800640 1 -- 172.21.15.96:0/88463033 shutdown_connections 2024-05-28T06:18:06.725 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.726+0000 7f3e14800640 1 --2- 172.21.15.96:0/88463033 >> v2:172.21.15.123:3300/0 conn(0x7f3e0c0b66a0 0x7f3e0c0b7880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:06.725 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.726+0000 7f3e14800640 1 --2- 172.21.15.96:0/88463033 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3e0c0ba070 0x7f3e0c0b6160 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:06.725 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.726+0000 7f3e14800640 1 -- 172.21.15.96:0/88463033 >> 172.21.15.96:0/88463033 conn(0x7f3e0c06d150 msgr2=0x7f3e0c0d1160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:18:06.725 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.726+0000 7f3e14800640 1 -- 172.21.15.96:0/88463033 shutdown_connections 2024-05-28T06:18:06.725 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.726+0000 7f3e14800640 1 -- 172.21.15.96:0/88463033 wait complete. 2024-05-28T06:18:06.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.728+0000 7f3e14800640 1 Processor -- start 2024-05-28T06:18:06.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.728+0000 7f3e14800640 1 -- start start 2024-05-28T06:18:06.727 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.729+0000 7f3e14800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3e0c0b66a0 0x7f3e0c1f3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:06.727 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.729+0000 7f3e14800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3e0c0ba070 0x7f3e0c158210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:06.727 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.729+0000 7f3e14800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3e0c15e940 0x7f3e0c0b4270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:06.727 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.729+0000 7f3e14800640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f3e0c0d3e60 con 0x7f3e0c15e940 2024-05-28T06:18:06.728 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.729+0000 7f3e14800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f3e0c0d3b80 con 0x7f3e0c0ba070 2024-05-28T06:18:06.728 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.729+0000 7f3e14800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f3e0c0d4380 con 0x7f3e0c0b66a0 2024-05-28T06:18:06.728 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.729+0000 7f3e12800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3e0c15e940 0x7f3e0c0b4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:06.728 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.729+0000 7f3e12800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3e0c15e940 0x7f3e0c0b4270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:60378/0 (socket says 172.21.15.96:60378) 2024-05-28T06:18:06.728 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.729+0000 7f3e12800640 1 -- 172.21.15.96:0/1566472650 learned_addr learned my addr 172.21.15.96:0/1566472650 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:18:06.729 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.729+0000 7f3e12800640 1 -- 172.21.15.96:0/1566472650 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3e0c0b66a0 msgr2=0x7f3e0c1f3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:06.729 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.729+0000 7f3e11e00640 1 --2- 172.21.15.96:0/1566472650 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3e0c0b66a0 0x7f3e0c1f3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:06.729 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.729+0000 7f3e12800640 1 --2- 172.21.15.96:0/1566472650 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3e0c0b66a0 0x7f3e0c1f3eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:06.729 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.730+0000 7f3e12800640 1 -- 172.21.15.96:0/1566472650 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3e0c0ba070 msgr2=0x7f3e0c158210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:06.729 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.730+0000 7f3e12800640 1 --2- 172.21.15.96:0/1566472650 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3e0c0ba070 0x7f3e0c158210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:06.729 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.730+0000 7f3e12800640 1 -- 172.21.15.96:0/1566472650 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e0c1f7d70 con 0x7f3e0c15e940 2024-05-28T06:18:06.729 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.730+0000 7f3e12800640 1 --2- 172.21.15.96:0/1566472650 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3e0c15e940 0x7f3e0c0b4270 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f3e0800c820 tx=0x7f3e0800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:06.730 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.730+0000 7f3e02000640 1 -- 172.21.15.96:0/1566472650 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e08015070 con 0x7f3e0c15e940 2024-05-28T06:18:06.730 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.730+0000 7f3e02000640 1 -- 172.21.15.96:0/1566472650 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e08007a30 con 0x7f3e0c15e940 2024-05-28T06:18:06.730 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.730+0000 7f3e02000640 1 -- 172.21.15.96:0/1566472650 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e08007e20 con 0x7f3e0c15e940 2024-05-28T06:18:06.730 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.730+0000 7f3e14800640 1 -- 172.21.15.96:0/1566472650 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e0c1f8050 con 0x7f3e0c15e940 2024-05-28T06:18:06.730 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.730+0000 7f3e14800640 1 -- 172.21.15.96:0/1566472650 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e0c1f8610 con 0x7f3e0c15e940 2024-05-28T06:18:06.731 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.731+0000 7f3e14800640 1 -- 172.21.15.96:0/1566472650 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e0c0689e0 con 0x7f3e0c15e940 2024-05-28T06:18:06.731 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.732+0000 7f3e02000640 1 -- 172.21.15.96:0/1566472650 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 30) ==== 92621+0+0 (secure 0 0 0) 0x7f3e08018070 con 0x7f3e0c15e940 2024-05-28T06:18:06.731 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.733+0000 7f3e02000640 1 --2- 172.21.15.96:0/1566472650 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3df006da60 0x7f3df006ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:06.731 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.733+0000 7f3e02000640 1 -- 172.21.15.96:0/1566472650 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 4885+0+0 (secure 0 0 0) 0x7f3e0808aa50 con 0x7f3e0c15e940 2024-05-28T06:18:06.732 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.734+0000 7f3e11e00640 1 --2- 172.21.15.96:0/1566472650 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3df006da60 0x7f3df006ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:06.733 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.735+0000 7f3e11e00640 1 --2- 172.21.15.96:0/1566472650 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3df006da60 0x7f3df006ff20 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f3e04007fd0 tx=0x7f3e04009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:06.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.738+0000 7f3e02000640 1 -- 172.21.15.96:0/1566472650 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3e0805a580 con 0x7f3e0c15e940 2024-05-28T06:18:06.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.937+0000 7f3e14800640 1 -- 172.21.15.96:0/1566472650 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f3e0c0cad10 con 0x7f3e0c15e940 2024-05-28T06:18:06.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.938+0000 7f3e02000640 1 -- 172.21.15.96:0/1566472650 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v85) ==== 93+0+10 (secure 0 0 0) 0x7f3e08059e80 con 0x7f3e0c15e940 2024-05-28T06:18:06.936 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 1 2024-05-28T06:18:06.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.940+0000 7f3e14800640 1 -- 172.21.15.96:0/1566472650 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3df006da60 msgr2=0x7f3df006ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:06.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.940+0000 7f3e14800640 1 --2- 172.21.15.96:0/1566472650 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3df006da60 0x7f3df006ff20 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f3e04007fd0 tx=0x7f3e04009040 comp rx=0 tx=0).stop 2024-05-28T06:18:06.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.940+0000 7f3e14800640 1 -- 172.21.15.96:0/1566472650 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3e0c15e940 msgr2=0x7f3e0c0b4270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:06.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.940+0000 7f3e14800640 1 --2- 172.21.15.96:0/1566472650 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3e0c15e940 0x7f3e0c0b4270 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f3e0800c820 tx=0x7f3e0800cce0 comp rx=0 tx=0).stop 2024-05-28T06:18:06.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.941+0000 7f3e14800640 1 -- 172.21.15.96:0/1566472650 shutdown_connections 2024-05-28T06:18:06.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.941+0000 7f3e14800640 1 --2- 172.21.15.96:0/1566472650 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3df006da60 0x7f3df006ff20 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:06.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.941+0000 7f3e14800640 1 --2- 172.21.15.96:0/1566472650 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3e0c15e940 0x7f3e0c0b4270 unknown :-1 s=CLOSED pgs=1379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:06.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.941+0000 7f3e14800640 1 --2- 172.21.15.96:0/1566472650 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3e0c0ba070 0x7f3e0c158210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:06.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.941+0000 7f3e14800640 1 --2- 172.21.15.96:0/1566472650 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3e0c0b66a0 0x7f3e0c1f3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:06.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.941+0000 7f3e14800640 1 -- 172.21.15.96:0/1566472650 >> 172.21.15.96:0/1566472650 conn(0x7f3e0c06d150 msgr2=0x7f3e0c15d950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:18:06.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.941+0000 7f3e14800640 1 -- 172.21.15.96:0/1566472650 shutdown_connections 2024-05-28T06:18:06.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:06.941+0000 7f3e14800640 1 -- 172.21.15.96:0/1566472650 wait complete. 2024-05-28T06:18:06.957 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-05-28T06:18:07.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.152+0000 7f883dc00640 1 Processor -- start 2024-05-28T06:18:07.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.152+0000 7f883dc00640 1 -- start start 2024-05-28T06:18:07.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.154+0000 7f883dc00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f8838158b80 0x7f883806dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:07.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.154+0000 7f883dc00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f883806e6e0 con 0x7f883806e160 2024-05-28T06:18:07.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.154+0000 7f883dc00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f883806e9c0 con 0x7f88380722d0 2024-05-28T06:18:07.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.154+0000 7f883dc00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f883806eca0 con 0x7f8838158b80 2024-05-28T06:18:07.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.154+0000 7f8837e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f8838158b80 0x7f883806dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:07.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.154+0000 7f882fe00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f883806e160 0x7f8838071bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:48970/0 (socket says 172.21.15.96:48970) 2024-05-28T06:18:07.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.154+0000 7f882fe00640 1 -- 172.21.15.96:0/319500094 learned_addr learned my addr 172.21.15.96:0/319500094 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:18:07.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.155+0000 7f8837400640 1 -- 172.21.15.96:0/319500094 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 659387929 0 0) 0x7f883806e6e0 con 0x7f883806e160 2024-05-28T06:18:07.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.155+0000 7f8837e00640 1 -- 172.21.15.96:0/319500094 >> v1:172.21.15.123:6789/0 conn(0x7f88380722d0 legacy=0x7f88380b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:07.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.155+0000 7f8837e00640 1 -- 172.21.15.96:0/319500094 >> v1:172.21.15.96:6789/0 conn(0x7f883806e160 legacy=0x7f8838071bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:07.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.155+0000 7f8837e00640 1 -- 172.21.15.96:0/319500094 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f883806ef80 con 0x7f8838158b80 2024-05-28T06:18:07.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.157+0000 7f8837e00640 1 --2- 172.21.15.96:0/319500094 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8838158b80 0x7f883806dc20 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f88280097a0 tx=0x7f882802a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e221e524b611a424 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:07.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.157+0000 7f8837400640 1 -- 172.21.15.96:0/319500094 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8828037070 con 0x7f8838158b80 2024-05-28T06:18:07.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.157+0000 7f8837400640 1 -- 172.21.15.96:0/319500094 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8828032440 con 0x7f8838158b80 2024-05-28T06:18:07.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.157+0000 7f8837400640 1 -- 172.21.15.96:0/319500094 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8828032810 con 0x7f8838158b80 2024-05-28T06:18:07.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.157+0000 7f883dc00640 1 -- 172.21.15.96:0/319500094 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8838158b80 msgr2=0x7f883806dc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:07.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.157+0000 7f883dc00640 1 --2- 172.21.15.96:0/319500094 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8838158b80 0x7f883806dc20 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f88280097a0 tx=0x7f882802a230 comp rx=0 tx=0).stop 2024-05-28T06:18:07.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.158+0000 7f883dc00640 1 -- 172.21.15.96:0/319500094 shutdown_connections 2024-05-28T06:18:07.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.158+0000 7f883dc00640 1 --2- 172.21.15.96:0/319500094 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8838158b80 0x7f883806dc20 unknown :-1 s=CLOSED pgs=1381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:07.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.158+0000 7f883dc00640 1 -- 172.21.15.96:0/319500094 >> 172.21.15.96:0/319500094 conn(0x7f88380b1fc0 msgr2=0x7f88380b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:18:07.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.158+0000 7f883dc00640 1 -- 172.21.15.96:0/319500094 shutdown_connections 2024-05-28T06:18:07.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.158+0000 7f883dc00640 1 -- 172.21.15.96:0/319500094 wait complete. 2024-05-28T06:18:07.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.159+0000 7f883dc00640 1 Processor -- start 2024-05-28T06:18:07.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.160+0000 7f883dc00640 1 -- start start 2024-05-28T06:18:07.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.160+0000 7f883dc00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f883806e160 0x7f8838144910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:07.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.160+0000 7f883dc00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f88380722d0 0x7f88380ad230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:07.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.160+0000 7f883dc00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8838158b80 0x7f883809b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:07.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.160+0000 7f883dc00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f883806f780 con 0x7f88380722d0 2024-05-28T06:18:07.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.160+0000 7f883dc00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f88380b2d30 con 0x7f8838158b80 2024-05-28T06:18:07.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.160+0000 7f883dc00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f883809a380 con 0x7f883806e160 2024-05-28T06:18:07.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.161+0000 7f882fe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f88380722d0 0x7f88380ad230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:07.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.161+0000 7f882fe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f88380722d0 0x7f88380ad230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:60394/0 (socket says 172.21.15.96:60394) 2024-05-28T06:18:07.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.161+0000 7f882fe00640 1 -- 172.21.15.96:0/1958849086 learned_addr learned my addr 172.21.15.96:0/1958849086 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:18:07.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.161+0000 7f8837e00640 1 --2- 172.21.15.96:0/1958849086 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f883806e160 0x7f8838144910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:07.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.161+0000 7f882fe00640 1 -- 172.21.15.96:0/1958849086 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f883806e160 msgr2=0x7f8838144910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:07.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.161+0000 7f882fe00640 1 --2- 172.21.15.96:0/1958849086 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f883806e160 0x7f8838144910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:07.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.161+0000 7f882fe00640 1 -- 172.21.15.96:0/1958849086 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8838158b80 msgr2=0x7f883809b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:07.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.161+0000 7f882fe00640 1 --2- 172.21.15.96:0/1958849086 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8838158b80 0x7f883809b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:07.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.161+0000 7f882fe00640 1 -- 172.21.15.96:0/1958849086 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f883809a630 con 0x7f88380722d0 2024-05-28T06:18:07.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.161+0000 7f882fe00640 1 --2- 172.21.15.96:0/1958849086 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f88380722d0 0x7f88380ad230 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7f882000c490 tx=0x7f882000c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:07.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.162+0000 7f8834c00640 1 -- 172.21.15.96:0/1958849086 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8820018070 con 0x7f88380722d0 2024-05-28T06:18:07.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.162+0000 7f8834c00640 1 -- 172.21.15.96:0/1958849086 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88200074e0 con 0x7f88380722d0 2024-05-28T06:18:07.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.162+0000 7f8834c00640 1 -- 172.21.15.96:0/1958849086 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8820004a20 con 0x7f88380722d0 2024-05-28T06:18:07.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.162+0000 7f883dc00640 1 -- 172.21.15.96:0/1958849086 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f883809a970 con 0x7f88380722d0 2024-05-28T06:18:07.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.162+0000 7f883dc00640 1 -- 172.21.15.96:0/1958849086 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8838144e50 con 0x7f88380722d0 2024-05-28T06:18:07.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.163+0000 7f883dc00640 1 -- 172.21.15.96:0/1958849086 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88380689e0 con 0x7f88380722d0 2024-05-28T06:18:07.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.164+0000 7f8834c00640 1 -- 172.21.15.96:0/1958849086 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 30) ==== 92621+0+0 (secure 0 0 0) 0x7f882001b070 con 0x7f88380722d0 2024-05-28T06:18:07.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.165+0000 7f8834c00640 1 --2- 172.21.15.96:0/1958849086 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f881406da60 0x7f881406ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:07.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.165+0000 7f8834c00640 1 -- 172.21.15.96:0/1958849086 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 4885+0+0 (secure 0 0 0) 0x7f8820089db0 con 0x7f88380722d0 2024-05-28T06:18:07.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.166+0000 7f8837e00640 1 --2- 172.21.15.96:0/1958849086 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f881406da60 0x7f881406ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:07.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.167+0000 7f8837e00640 1 --2- 172.21.15.96:0/1958849086 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f881406da60 0x7f881406ff20 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f8828034000 tx=0x7f882802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:07.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.170+0000 7f8834c00640 1 -- 172.21.15.96:0/1958849086 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f883809a970 con 0x7f88380722d0 2024-05-28T06:18:07.354 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.356+0000 7f883dc00640 1 -- 172.21.15.96:0/1958849086 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f883809ab50 con 0x7f88380722d0 2024-05-28T06:18:07.355 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.357+0000 7f8834c00640 1 -- 172.21.15.96:0/1958849086 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v85) ==== 84+0+65 (secure 0 0 0) 0x7f88200598e0 con 0x7f88380722d0 2024-05-28T06:18:07.355 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:18:07.355 INFO:teuthology.orchestra.run.smithi096.stdout:[ 2024-05-28T06:18:07.355 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:18:07.355 INFO:teuthology.orchestra.run.smithi096.stdout: "poolnum": 1, 2024-05-28T06:18:07.355 INFO:teuthology.orchestra.run.smithi096.stdout: "poolname": ".mgr" 2024-05-28T06:18:07.356 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:18:07.356 INFO:teuthology.orchestra.run.smithi096.stdout:] 2024-05-28T06:18:07.358 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.359+0000 7f883dc00640 1 -- 172.21.15.96:0/1958849086 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f881406da60 msgr2=0x7f881406ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:07.358 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.359+0000 7f883dc00640 1 --2- 172.21.15.96:0/1958849086 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f881406da60 0x7f881406ff20 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f8828034000 tx=0x7f882802e000 comp rx=0 tx=0).stop 2024-05-28T06:18:07.358 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.359+0000 7f883dc00640 1 -- 172.21.15.96:0/1958849086 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f88380722d0 msgr2=0x7f88380ad230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:07.358 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.359+0000 7f883dc00640 1 --2- 172.21.15.96:0/1958849086 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f88380722d0 0x7f88380ad230 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7f882000c490 tx=0x7f882000c950 comp rx=0 tx=0).stop 2024-05-28T06:18:07.358 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.359+0000 7f883dc00640 1 -- 172.21.15.96:0/1958849086 shutdown_connections 2024-05-28T06:18:07.358 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.359+0000 7f883dc00640 1 --2- 172.21.15.96:0/1958849086 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f881406da60 0x7f881406ff20 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:07.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.359+0000 7f883dc00640 1 --2- 172.21.15.96:0/1958849086 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8838158b80 0x7f883809b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:07.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.359+0000 7f883dc00640 1 --2- 172.21.15.96:0/1958849086 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f88380722d0 0x7f88380ad230 unknown :-1 s=CLOSED pgs=1382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:07.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.359+0000 7f883dc00640 1 --2- 172.21.15.96:0/1958849086 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f883806e160 0x7f8838144910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:07.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.359+0000 7f883dc00640 1 -- 172.21.15.96:0/1958849086 >> 172.21.15.96:0/1958849086 conn(0x7f88380b1fc0 msgr2=0x7f88380b2f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:18:07.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.359+0000 7f883dc00640 1 -- 172.21.15.96:0/1958849086 shutdown_connections 2024-05-28T06:18:07.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.359+0000 7f883dc00640 1 -- 172.21.15.96:0/1958849086 wait complete. 2024-05-28T06:18:07.376 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-05-28T06:18:07.376 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-05-28T06:18:07.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.578+0000 7f9300800640 1 Processor -- start 2024-05-28T06:18:07.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.579+0000 7f9300800640 1 -- start start 2024-05-28T06:18:07.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.580+0000 7f9300800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f92f815e9b0 0x7f92f80b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:07.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.580+0000 7f9300800640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92f80d0500 con 0x7f92f80cd2e0 2024-05-28T06:18:07.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.580+0000 7f9300800640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92f80d07e0 con 0x7f92f80cfff0 2024-05-28T06:18:07.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.580+0000 7f9300800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f92f80d0ac0 con 0x7f92f815e9b0 2024-05-28T06:18:07.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.580+0000 7f92fe800640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f92f80cd2e0 0x7f92f806e8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:48972/0 (socket says 172.21.15.96:48972) 2024-05-28T06:18:07.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.580+0000 7f92fe800640 1 -- 172.21.15.96:0/188154831 learned_addr learned my addr 172.21.15.96:0/188154831 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:18:07.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.581+0000 7f92fde00640 1 --2- 172.21.15.96:0/188154831 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f92f815e9b0 0x7f92f80b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:07.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.581+0000 7f92fca00640 1 -- 172.21.15.96:0/188154831 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3522185754 0 0) 0x7f92f80d0500 con 0x7f92f80cd2e0 2024-05-28T06:18:07.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.581+0000 7f92fca00640 1 -- 172.21.15.96:0/188154831 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f92e0003520 con 0x7f92f80cd2e0 2024-05-28T06:18:07.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.581+0000 7f92fca00640 1 -- 172.21.15.96:0/188154831 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1200560085 0 0) 0x7f92e0003520 con 0x7f92f80cd2e0 2024-05-28T06:18:07.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.581+0000 7f92fca00640 1 -- 172.21.15.96:0/188154831 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f92f815e9b0 msgr2=0x7f92f80b4dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:18:07.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.581+0000 7f92fca00640 1 --2- 172.21.15.96:0/188154831 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f92f815e9b0 0x7f92f80b4dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:07.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.581+0000 7f92fca00640 1 -- 172.21.15.96:0/188154831 >> v1:172.21.15.123:6789/0 conn(0x7f92f80cfff0 legacy=0x7f92f80ccbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:07.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.581+0000 7f92fca00640 1 -- 172.21.15.96:0/188154831 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92f80d0da0 con 0x7f92f80cd2e0 2024-05-28T06:18:07.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.581+0000 7f92fca00640 1 -- 172.21.15.96:0/188154831 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f92f40019f0 con 0x7f92f80cd2e0 2024-05-28T06:18:07.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.581+0000 7f92fca00640 1 -- 172.21.15.96:0/188154831 >> v1:172.21.15.96:6789/0 conn(0x7f92f80cd2e0 legacy=0x7f92f806e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:07.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.582+0000 7f92fca00640 1 --2- 172.21.15.96:0/188154831 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92e00039e0 0x7f92e0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:07.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.582+0000 7f92fca00640 1 --2- 172.21.15.96:0/188154831 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f92f815e9b0 0x7f92e0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:07.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.582+0000 7f92fca00640 1 --2- 172.21.15.96:0/188154831 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f92e0004d40 0x7f92e00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:07.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.582+0000 7f92fca00640 1 -- 172.21.15.96:0/188154831 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f92f80d0ac0 con 0x7f92e0004d40 2024-05-28T06:18:07.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.582+0000 7f92fca00640 1 -- 172.21.15.96:0/188154831 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f92e000ba70 con 0x7f92e00039e0 2024-05-28T06:18:07.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.582+0000 7f92fca00640 1 -- 172.21.15.96:0/188154831 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f92e000bc70 con 0x7f92f815e9b0 2024-05-28T06:18:07.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.582+0000 7f92fe800640 1 --2- 172.21.15.96:0/188154831 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f92e0004d40 0x7f92e00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:07.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.582+0000 7f92fe800640 1 -- 172.21.15.96:0/188154831 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f92f815e9b0 msgr2=0x7f92e0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:18:07.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.582+0000 7f92fd400640 1 --2- 172.21.15.96:0/188154831 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f92f815e9b0 0x7f92e0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:07.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.582+0000 7f92fe800640 1 --2- 172.21.15.96:0/188154831 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f92f815e9b0 0x7f92e0004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:07.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.582+0000 7f92fe800640 1 -- 172.21.15.96:0/188154831 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92e00039e0 msgr2=0x7f92e0003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:18:07.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.582+0000 7f92fe800640 1 --2- 172.21.15.96:0/188154831 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92e00039e0 0x7f92e0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:07.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.582+0000 7f92fe800640 1 -- 172.21.15.96:0/188154831 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92e0003520 con 0x7f92e0004d40 2024-05-28T06:18:07.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.583+0000 7f92fe800640 1 --2- 172.21.15.96:0/188154831 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f92e0004d40 0x7f92e00051c0 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f92f4002230 tx=0x7f92e0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=623fda88934ab1fb server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:07.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.584+0000 7f92fca00640 1 -- 172.21.15.96:0/188154831 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92f400a4a0 con 0x7f92e0004d40 2024-05-28T06:18:07.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.584+0000 7f92fca00640 1 -- 172.21.15.96:0/188154831 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92f402f070 con 0x7f92e0004d40 2024-05-28T06:18:07.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.584+0000 7f92fca00640 1 -- 172.21.15.96:0/188154831 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92f402bc50 con 0x7f92e0004d40 2024-05-28T06:18:07.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.584+0000 7f9300800640 1 -- 172.21.15.96:0/188154831 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f92e0004d40 msgr2=0x7f92e00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:07.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.584+0000 7f9300800640 1 --2- 172.21.15.96:0/188154831 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f92e0004d40 0x7f92e00051c0 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f92f4002230 tx=0x7f92e0005830 comp rx=0 tx=0).stop 2024-05-28T06:18:07.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.584+0000 7f92fe800640 1 -- 172.21.15.96:0/188154831 reap_dead start 2024-05-28T06:18:07.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.584+0000 7f9300800640 1 -- 172.21.15.96:0/188154831 shutdown_connections 2024-05-28T06:18:07.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.584+0000 7f9300800640 1 -- 172.21.15.96:0/188154831 >> 172.21.15.96:0/188154831 conn(0x7f92f80b20a0 msgr2=0x7f92f80b24b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:18:07.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.584+0000 7f9300800640 1 -- 172.21.15.96:0/188154831 shutdown_connections 2024-05-28T06:18:07.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.584+0000 7f9300800640 1 -- 172.21.15.96:0/188154831 wait complete. 2024-05-28T06:18:07.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.586+0000 7f9300800640 1 Processor -- start 2024-05-28T06:18:07.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.586+0000 7f9300800640 1 -- start start 2024-05-28T06:18:07.586 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.587+0000 7f9300800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92f81f75e0 0x7f92f80c8da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:07.586 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.587+0000 7f9300800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f92f80aee80 0x7f92f80c7400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:07.586 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.587+0000 7f9300800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f92f815abc0 0x7f92f815b040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:07.586 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.587+0000 7f9300800640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f92f815ed90 con 0x7f92f815abc0 2024-05-28T06:18:07.586 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.587+0000 7f9300800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f92f80cc8f0 con 0x7f92f81f75e0 2024-05-28T06:18:07.586 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.587+0000 7f9300800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f92f815ef10 con 0x7f92f80aee80 2024-05-28T06:18:07.586 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.587+0000 7f92fe800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f92f815abc0 0x7f92f815b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:07.586 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.588+0000 7f92fe800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f92f815abc0 0x7f92f815b040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:60410/0 (socket says 172.21.15.96:60410) 2024-05-28T06:18:07.586 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.588+0000 7f92fe800640 1 -- 172.21.15.96:0/126448690 learned_addr learned my addr 172.21.15.96:0/126448690 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:18:07.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.588+0000 7f92fd400640 1 --2- 172.21.15.96:0/126448690 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f92f80aee80 0x7f92f80c7400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:07.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.588+0000 7f92fde00640 1 --2- 172.21.15.96:0/126448690 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92f81f75e0 0x7f92f80c8da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:07.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.588+0000 7f92fe800640 1 -- 172.21.15.96:0/126448690 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f92f80aee80 msgr2=0x7f92f80c7400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:07.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.588+0000 7f92fe800640 1 --2- 172.21.15.96:0/126448690 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f92f80aee80 0x7f92f80c7400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:07.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.588+0000 7f92fe800640 1 -- 172.21.15.96:0/126448690 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92f81f75e0 msgr2=0x7f92f80c8da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:07.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.588+0000 7f92fe800640 1 --2- 172.21.15.96:0/126448690 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92f81f75e0 0x7f92f80c8da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:07.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.588+0000 7f92fe800640 1 -- 172.21.15.96:0/126448690 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92f80c7d00 con 0x7f92f815abc0 2024-05-28T06:18:07.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.588+0000 7f92fe800640 1 --2- 172.21.15.96:0/126448690 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f92f815abc0 0x7f92f815b040 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7f92f402b4a0 tx=0x7f92f402d070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:07.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.591+0000 7f92f2000640 1 -- 172.21.15.96:0/126448690 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92f402be30 con 0x7f92f815abc0 2024-05-28T06:18:07.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.591+0000 7f92f2000640 1 -- 172.21.15.96:0/126448690 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92f400a480 con 0x7f92f815abc0 2024-05-28T06:18:07.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.591+0000 7f9300800640 1 -- 172.21.15.96:0/126448690 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92f80c7fe0 con 0x7f92f815abc0 2024-05-28T06:18:07.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.591+0000 7f92f2000640 1 -- 172.21.15.96:0/126448690 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92f4034670 con 0x7f92f815abc0 2024-05-28T06:18:07.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.592+0000 7f9300800640 1 -- 172.21.15.96:0/126448690 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92f80c85a0 con 0x7f92f815abc0 2024-05-28T06:18:07.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.593+0000 7f9300800640 1 -- 172.21.15.96:0/126448690 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92f80757d0 con 0x7f92f815abc0 2024-05-28T06:18:07.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.594+0000 7f92f2000640 1 -- 172.21.15.96:0/126448690 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 30) ==== 92621+0+0 (secure 0 0 0) 0x7f92f4034810 con 0x7f92f815abc0 2024-05-28T06:18:07.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.595+0000 7f92f2000640 1 --2- 172.21.15.96:0/126448690 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f92dc06dae0 0x7f92dc06ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:07.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.595+0000 7f92f2000640 1 -- 172.21.15.96:0/126448690 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 4885+0+0 (secure 0 0 0) 0x7f92f40aa630 con 0x7f92f815abc0 2024-05-28T06:18:07.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.595+0000 7f92fde00640 1 --2- 172.21.15.96:0/126448690 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f92dc06dae0 0x7f92dc06ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:07.595 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.597+0000 7f92fde00640 1 --2- 172.21.15.96:0/126448690 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f92dc06dae0 0x7f92dc06ffa0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f92ec004770 tx=0x7f92ec008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:07.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.599+0000 7f92f2000640 1 -- 172.21.15.96:0/126448690 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f92f407a4b0 con 0x7f92f815abc0 2024-05-28T06:18:07.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:07.800+0000 7f9300800640 1 -- 172.21.15.96:0/126448690 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f92f806e440 con 0x7f92f815abc0 2024-05-28T06:18:08.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.604+0000 7f92f2000640 1 -- 172.21.15.96:0/126448690 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v86) ==== 138+0+0 (secure 0 0 0) 0x7f92f4079db0 con 0x7f92f815abc0 2024-05-28T06:18:08.603 INFO:teuthology.orchestra.run.smithi096.stderr:pool 'cephfs_metadata' created 2024-05-28T06:18:08.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.607+0000 7f9300800640 1 -- 172.21.15.96:0/126448690 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f92dc06dae0 msgr2=0x7f92dc06ffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:08.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.607+0000 7f9300800640 1 --2- 172.21.15.96:0/126448690 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f92dc06dae0 0x7f92dc06ffa0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f92ec004770 tx=0x7f92ec008040 comp rx=0 tx=0).stop 2024-05-28T06:18:08.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.607+0000 7f9300800640 1 -- 172.21.15.96:0/126448690 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f92f815abc0 msgr2=0x7f92f815b040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:08.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.607+0000 7f9300800640 1 --2- 172.21.15.96:0/126448690 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f92f815abc0 0x7f92f815b040 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7f92f402b4a0 tx=0x7f92f402d070 comp rx=0 tx=0).stop 2024-05-28T06:18:08.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.608+0000 7f9300800640 1 -- 172.21.15.96:0/126448690 shutdown_connections 2024-05-28T06:18:08.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.608+0000 7f9300800640 1 --2- 172.21.15.96:0/126448690 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f92dc06dae0 0x7f92dc06ffa0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:08.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.608+0000 7f9300800640 1 --2- 172.21.15.96:0/126448690 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f92f815abc0 0x7f92f815b040 unknown :-1 s=CLOSED pgs=1385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:08.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.608+0000 7f9300800640 1 --2- 172.21.15.96:0/126448690 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f92f80aee80 0x7f92f80c7400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:08.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.608+0000 7f9300800640 1 --2- 172.21.15.96:0/126448690 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92f81f75e0 0x7f92f80c8da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:08.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.608+0000 7f9300800640 1 -- 172.21.15.96:0/126448690 >> 172.21.15.96:0/126448690 conn(0x7f92f80b20a0 msgr2=0x7f92f80b0030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:18:08.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.608+0000 7f9300800640 1 -- 172.21.15.96:0/126448690 shutdown_connections 2024-05-28T06:18:08.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.608+0000 7f9300800640 1 -- 172.21.15.96:0/126448690 wait complete. 2024-05-28T06:18:08.633 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-05-28T06:18:08.828 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.829+0000 7ff45cc00640 1 Processor -- start 2024-05-28T06:18:08.828 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.829+0000 7ff45cc00640 1 -- start start 2024-05-28T06:18:08.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.831+0000 7ff45cc00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff45815b6f0 0x7ff45806e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:08.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.831+0000 7ff45cc00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4580ade90 con 0x7ff4580d0640 2024-05-28T06:18:08.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.831+0000 7ff45cc00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff458071e70 con 0x7ff4580725e0 2024-05-28T06:18:08.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.831+0000 7ff45cc00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ff458071c40 con 0x7ff45815b6f0 2024-05-28T06:18:08.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.831+0000 7ff455e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7ff4580d0640 0x7ff4580d0a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:49864/0 (socket says 172.21.15.96:49864) 2024-05-28T06:18:08.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.831+0000 7ff455e00640 1 -- 172.21.15.96:0/1943530758 learned_addr learned my addr 172.21.15.96:0/1943530758 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:18:08.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.831+0000 7ff455400640 1 --2- 172.21.15.96:0/1943530758 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff45815b6f0 0x7ff45806e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:08.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.831+0000 7ff454a00640 1 -- 172.21.15.96:0/1943530758 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1122805371 0 0) 0x7ff4580ade90 con 0x7ff4580d0640 2024-05-28T06:18:08.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.832+0000 7ff454a00640 1 -- 172.21.15.96:0/1943530758 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff438005730 con 0x7ff4580d0640 2024-05-28T06:18:08.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.832+0000 7ff454a00640 1 -- 172.21.15.96:0/1943530758 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 198310725 0 0) 0x7ff438005730 con 0x7ff4580d0640 2024-05-28T06:18:08.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.832+0000 7ff454a00640 1 -- 172.21.15.96:0/1943530758 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff45815b6f0 msgr2=0x7ff45806e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:18:08.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.832+0000 7ff454a00640 1 --2- 172.21.15.96:0/1943530758 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff45815b6f0 0x7ff45806e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:08.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.832+0000 7ff454a00640 1 -- 172.21.15.96:0/1943530758 >> v1:172.21.15.123:6789/0 conn(0x7ff4580725e0 legacy=0x7ff4580ae650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:08.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.832+0000 7ff454a00640 1 -- 172.21.15.96:0/1943530758 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff458072130 con 0x7ff4580d0640 2024-05-28T06:18:08.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.832+0000 7ff454a00640 1 -- 172.21.15.96:0/1943530758 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7ff448001670 con 0x7ff4580d0640 2024-05-28T06:18:08.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.832+0000 7ff454a00640 1 -- 172.21.15.96:0/1943530758 >> v1:172.21.15.96:6789/0 conn(0x7ff4580d0640 legacy=0x7ff4580d0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:08.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.832+0000 7ff454a00640 1 --2- 172.21.15.96:0/1943530758 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff438005bf0 0x7ff438006010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:08.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.832+0000 7ff454a00640 1 --2- 172.21.15.96:0/1943530758 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff438006a10 0x7ff438006e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:08.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.832+0000 7ff454a00640 1 --2- 172.21.15.96:0/1943530758 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff45815b6f0 0x7ff438007410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:08.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.832+0000 7ff454a00640 1 -- 172.21.15.96:0/1943530758 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff43800dcd0 con 0x7ff438005bf0 2024-05-28T06:18:08.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.832+0000 7ff454a00640 1 -- 172.21.15.96:0/1943530758 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff43800de80 con 0x7ff438006a10 2024-05-28T06:18:08.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.832+0000 7ff454a00640 1 -- 172.21.15.96:0/1943530758 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ff43800e080 con 0x7ff45815b6f0 2024-05-28T06:18:08.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.832+0000 7ff455e00640 1 --2- 172.21.15.96:0/1943530758 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff438005bf0 0x7ff438006010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:08.833 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.833+0000 7ff456800640 1 --2- 172.21.15.96:0/1943530758 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff45815b6f0 0x7ff438007410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:08.833 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.833+0000 7ff455e00640 1 -- 172.21.15.96:0/1943530758 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff45815b6f0 msgr2=0x7ff438007410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:18:08.833 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.833+0000 7ff455400640 1 --2- 172.21.15.96:0/1943530758 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff438006a10 0x7ff438006e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:08.833 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.833+0000 7ff455e00640 1 --2- 172.21.15.96:0/1943530758 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff45815b6f0 0x7ff438007410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:08.833 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.833+0000 7ff455e00640 1 -- 172.21.15.96:0/1943530758 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff438006a10 msgr2=0x7ff438006e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:18:08.833 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.833+0000 7ff455e00640 1 --2- 172.21.15.96:0/1943530758 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff438006a10 0x7ff438006e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:08.833 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.833+0000 7ff455e00640 1 -- 172.21.15.96:0/1943530758 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff438005730 con 0x7ff438005bf0 2024-05-28T06:18:08.833 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.833+0000 7ff456800640 1 --2- 172.21.15.96:0/1943530758 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff45815b6f0 0x7ff438007410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:18:08.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.834+0000 7ff455e00640 1 --2- 172.21.15.96:0/1943530758 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff438005bf0 0x7ff438006010 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7ff44800c210 tx=0x7ff438007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f8a08ae0058d42 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:08.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.834+0000 7ff454a00640 1 -- 172.21.15.96:0/1943530758 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff448007d30 con 0x7ff438005bf0 2024-05-28T06:18:08.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.834+0000 7ff454a00640 1 -- 172.21.15.96:0/1943530758 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff448005ad0 con 0x7ff438005bf0 2024-05-28T06:18:08.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.834+0000 7ff454a00640 1 -- 172.21.15.96:0/1943530758 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff448007070 con 0x7ff438005bf0 2024-05-28T06:18:08.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.835+0000 7ff45cc00640 1 -- 172.21.15.96:0/1943530758 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff438005bf0 msgr2=0x7ff438006010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:08.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.835+0000 7ff45cc00640 1 --2- 172.21.15.96:0/1943530758 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff438005bf0 0x7ff438006010 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7ff44800c210 tx=0x7ff438007a40 comp rx=0 tx=0).stop 2024-05-28T06:18:08.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.835+0000 7ff456800640 1 -- 172.21.15.96:0/1943530758 reap_dead start 2024-05-28T06:18:08.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.835+0000 7ff45cc00640 1 -- 172.21.15.96:0/1943530758 shutdown_connections 2024-05-28T06:18:08.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.835+0000 7ff45cc00640 1 -- 172.21.15.96:0/1943530758 >> 172.21.15.96:0/1943530758 conn(0x7ff45806bd20 msgr2=0x7ff45806c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:18:08.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.835+0000 7ff45cc00640 1 -- 172.21.15.96:0/1943530758 shutdown_connections 2024-05-28T06:18:08.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.835+0000 7ff45cc00640 1 -- 172.21.15.96:0/1943530758 wait complete. 2024-05-28T06:18:08.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.837+0000 7ff45cc00640 1 Processor -- start 2024-05-28T06:18:08.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.837+0000 7ff45cc00640 1 -- start start 2024-05-28T06:18:08.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.838+0000 7ff45cc00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff438005bf0 0x7ff4581ef420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:08.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.838+0000 7ff45cc00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff4581598f0 0x7ff458159d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:08.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.838+0000 7ff45cc00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff458142dc0 0x7ff458143240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:08.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.838+0000 7ff45cc00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff45806e4d0 con 0x7ff438005bf0 2024-05-28T06:18:08.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.838+0000 7ff45cc00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff45806e350 con 0x7ff458142dc0 2024-05-28T06:18:08.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.838+0000 7ff45cc00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ff45806e650 con 0x7ff4581598f0 2024-05-28T06:18:08.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.838+0000 7ff455e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff438005bf0 0x7ff4581ef420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:08.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.838+0000 7ff455e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff438005bf0 0x7ff4581ef420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48108/0 (socket says 172.21.15.96:48108) 2024-05-28T06:18:08.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.838+0000 7ff455e00640 1 -- 172.21.15.96:0/607498660 learned_addr learned my addr 172.21.15.96:0/607498660 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:18:08.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.838+0000 7ff455400640 1 --2- 172.21.15.96:0/607498660 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff4581598f0 0x7ff458159d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:08.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.838+0000 7ff455e00640 1 -- 172.21.15.96:0/607498660 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff4581598f0 msgr2=0x7ff458159d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:08.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.838+0000 7ff455e00640 1 --2- 172.21.15.96:0/607498660 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff4581598f0 0x7ff458159d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:08.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.838+0000 7ff455e00640 1 -- 172.21.15.96:0/607498660 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff458142dc0 msgr2=0x7ff458143240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:08.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.838+0000 7ff456800640 1 --2- 172.21.15.96:0/607498660 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff458142dc0 0x7ff458143240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:08.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.839+0000 7ff455e00640 1 --2- 172.21.15.96:0/607498660 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff458142dc0 0x7ff458143240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:08.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.839+0000 7ff455e00640 1 -- 172.21.15.96:0/607498660 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff458143b80 con 0x7ff438005bf0 2024-05-28T06:18:08.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.839+0000 7ff456800640 1 --2- 172.21.15.96:0/607498660 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff458142dc0 0x7ff458143240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:18:08.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.839+0000 7ff455e00640 1 --2- 172.21.15.96:0/607498660 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff438005bf0 0x7ff4581ef420 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7ff44800c330 tx=0x7ff448001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:08.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.839+0000 7ff44e000640 1 -- 172.21.15.96:0/607498660 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff448043070 con 0x7ff438005bf0 2024-05-28T06:18:08.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.839+0000 7ff44e000640 1 -- 172.21.15.96:0/607498660 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff448004bf0 con 0x7ff438005bf0 2024-05-28T06:18:08.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.839+0000 7ff44e000640 1 -- 172.21.15.96:0/607498660 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff448007d10 con 0x7ff438005bf0 2024-05-28T06:18:08.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.839+0000 7ff45cc00640 1 -- 172.21.15.96:0/607498660 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4581f3180 con 0x7ff438005bf0 2024-05-28T06:18:08.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.839+0000 7ff45cc00640 1 -- 172.21.15.96:0/607498660 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4581f3740 con 0x7ff438005bf0 2024-05-28T06:18:08.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.841+0000 7ff45cc00640 1 -- 172.21.15.96:0/607498660 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4580689e0 con 0x7ff438005bf0 2024-05-28T06:18:08.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.841+0000 7ff44e000640 1 -- 172.21.15.96:0/607498660 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 30) ==== 92621+0+0 (secure 0 0 0) 0x7ff448007070 con 0x7ff438005bf0 2024-05-28T06:18:08.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.842+0000 7ff44e000640 1 --2- 172.21.15.96:0/607498660 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff43406da90 0x7ff43406ff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:08.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.843+0000 7ff44e000640 1 -- 172.21.15.96:0/607498660 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5255+0+0 (secure 0 0 0) 0x7ff4480aa700 con 0x7ff438005bf0 2024-05-28T06:18:08.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.843+0000 7ff455400640 1 --2- 172.21.15.96:0/607498660 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff43406da90 0x7ff43406ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:08.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.844+0000 7ff455400640 1 --2- 172.21.15.96:0/607498660 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff43406da90 0x7ff43406ff50 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7ff45815a4c0 tx=0x7ff44000b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:08.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:08.848+0000 7ff44e000640 1 -- 172.21.15.96:0/607498660 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff448079be0 con 0x7ff438005bf0 2024-05-28T06:18:09.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:09.043+0000 7ff45cc00640 1 -- 172.21.15.96:0/607498660 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7ff4580b0ba0 con 0x7ff438005bf0 2024-05-28T06:18:09.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:09.611+0000 7ff44e000640 1 -- 172.21.15.96:0/607498660 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v87) ==== 170+0+0 (secure 0 0 0) 0x7ff44807dee0 con 0x7ff438005bf0 2024-05-28T06:18:09.610 INFO:teuthology.orchestra.run.smithi096.stderr:pool 'cephfs_data' created 2024-05-28T06:18:09.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:09.614+0000 7ff45cc00640 1 -- 172.21.15.96:0/607498660 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff43406da90 msgr2=0x7ff43406ff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:09.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:09.614+0000 7ff45cc00640 1 --2- 172.21.15.96:0/607498660 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff43406da90 0x7ff43406ff50 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7ff45815a4c0 tx=0x7ff44000b040 comp rx=0 tx=0).stop 2024-05-28T06:18:09.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:09.614+0000 7ff45cc00640 1 -- 172.21.15.96:0/607498660 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff438005bf0 msgr2=0x7ff4581ef420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:09.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:09.614+0000 7ff45cc00640 1 --2- 172.21.15.96:0/607498660 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff438005bf0 0x7ff4581ef420 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7ff44800c330 tx=0x7ff448001a40 comp rx=0 tx=0).stop 2024-05-28T06:18:09.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:09.615+0000 7ff45cc00640 1 -- 172.21.15.96:0/607498660 shutdown_connections 2024-05-28T06:18:09.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:09.615+0000 7ff45cc00640 1 --2- 172.21.15.96:0/607498660 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff43406da90 0x7ff43406ff50 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:09.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:09.615+0000 7ff45cc00640 1 --2- 172.21.15.96:0/607498660 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff458142dc0 0x7ff458143240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:09.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:09.615+0000 7ff45cc00640 1 --2- 172.21.15.96:0/607498660 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff4581598f0 0x7ff458159d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:09.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:09.615+0000 7ff45cc00640 1 --2- 172.21.15.96:0/607498660 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff438005bf0 0x7ff4581ef420 secure :-1 s=CLOSED pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7ff44800c330 tx=0x7ff448001a40 comp rx=0 tx=0).stop 2024-05-28T06:18:09.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:09.615+0000 7ff45cc00640 1 -- 172.21.15.96:0/607498660 >> 172.21.15.96:0/607498660 conn(0x7ff45806bd20 msgr2=0x7ff45806c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:18:09.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:09.615+0000 7ff45cc00640 1 -- 172.21.15.96:0/607498660 shutdown_connections 2024-05-28T06:18:09.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:09.615+0000 7ff45cc00640 1 -- 172.21.15.96:0/607498660 wait complete. 2024-05-28T06:18:09.650 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-05-28T06:18:09.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:09.826+0000 7f1ece800640 1 Processor -- start 2024-05-28T06:18:09.825 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:09.826+0000 7f1ece800640 1 -- start start 2024-05-28T06:18:09.826 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:09.827+0000 7f1ece800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f1ec80caa40 0x7f1ec80ce960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:09.826 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:09.827+0000 7f1ece800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f1ec80cae20 0x7f1ec8151db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:09.826 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:09.827+0000 7f1ece800640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ec80b3e80 con 0x7f1ec81522f0 2024-05-28T06:18:09.827 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:09.827+0000 7f1ece800640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f1ec80d0b30 con 0x7f1ec80cae20 2024-05-28T06:18:09.827 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:09.827+0000 7f1ece800640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f1ec80b4160 con 0x7f1ec80caa40 2024-05-28T06:18:09.827 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:09.828+0000 7f1ecca00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f1ec80cae20 0x7f1ec8151db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:09.827 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:09.828+0000 7f1ecca00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f1ec80cae20 0x7f1ec8151db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48110/0 (socket says 172.21.15.96:48110) 2024-05-28T06:18:09.827 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:09.828+0000 7f1ecca00640 1 -- 172.21.15.96:0/1885169562 learned_addr learned my addr 172.21.15.96:0/1885169562 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:18:09.827 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:09.828+0000 7f1ecca00640 1 -- 172.21.15.96:0/1885169562 >> v2:172.21.15.123:3300/0 conn(0x7f1ec80caa40 msgr2=0x7f1ec80ce960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:18:09.827 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:09.828+0000 7f1ecca00640 1 --2- 172.21.15.96:0/1885169562 >> v2:172.21.15.123:3300/0 conn(0x7f1ec80caa40 0x7f1ec80ce960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:09.827 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:09.828+0000 7f1ecca00640 1 -- 172.21.15.96:0/1885169562 >> v1:172.21.15.96:6789/0 conn(0x7f1ec81522f0 legacy=0x7f1ec80cf710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:09.827 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:09.828+0000 7f1ecca00640 1 -- 172.21.15.96:0/1885169562 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ec80cde80 con 0x7f1ec80cae20 2024-05-28T06:18:09.828 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:09.829+0000 7f1ecca00640 1 --2- 172.21.15.96:0/1885169562 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1ec80cae20 0x7f1ec8151db0 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f1ec80b4c10 tx=0x7f1eb402a1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d284cba9ce9e4592 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:09.828 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:09.829+0000 7f1ec7e00640 1 -- 172.21.15.96:0/1885169562 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1eb4037070 con 0x7f1ec80cae20 2024-05-28T06:18:09.828 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:09.830+0000 7f1ec7e00640 1 -- 172.21.15.96:0/1885169562 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1eb4032440 con 0x7f1ec80cae20 2024-05-28T06:18:09.828 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:09.830+0000 7f1ec7e00640 1 -- 172.21.15.96:0/1885169562 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1eb4032810 con 0x7f1ec80cae20 2024-05-28T06:18:09.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:09.830+0000 7f1ece800640 1 -- 172.21.15.96:0/1885169562 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1ec80cae20 msgr2=0x7f1ec8151db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:09.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:09.830+0000 7f1ece800640 1 --2- 172.21.15.96:0/1885169562 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1ec80cae20 0x7f1ec8151db0 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f1ec80b4c10 tx=0x7f1eb402a1f0 comp rx=0 tx=0).stop 2024-05-28T06:18:09.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:09.830+0000 7f1ece800640 1 -- 172.21.15.96:0/1885169562 shutdown_connections 2024-05-28T06:18:09.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:09.830+0000 7f1ece800640 1 --2- 172.21.15.96:0/1885169562 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1ec80cae20 0x7f1ec8151db0 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:09.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:09.830+0000 7f1ece800640 1 --2- 172.21.15.96:0/1885169562 >> v2:172.21.15.123:3300/0 conn(0x7f1ec80caa40 0x7f1ec80ce960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:09.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:09.830+0000 7f1ece800640 1 -- 172.21.15.96:0/1885169562 >> 172.21.15.96:0/1885169562 conn(0x7f1ec806b900 msgr2=0x7f1ec806bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:18:09.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:09.830+0000 7f1ece800640 1 -- 172.21.15.96:0/1885169562 shutdown_connections 2024-05-28T06:18:09.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:09.830+0000 7f1ece800640 1 -- 172.21.15.96:0/1885169562 wait complete. 2024-05-28T06:18:09.830 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:09.832+0000 7f1ece800640 1 Processor -- start 2024-05-28T06:18:09.830 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:09.832+0000 7f1ece800640 1 -- start start 2024-05-28T06:18:09.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:09.833+0000 7f1ece800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1ec80caa40 0x7f1ec81465e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:09.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:09.833+0000 7f1ece800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ec80cae20 0x7f1ec8154b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:09.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:09.833+0000 7f1ece800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1ec81522f0 0x7f1ec8148840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:09.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:09.833+0000 7f1ece800640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f1ec8152710 con 0x7f1ec81522f0 2024-05-28T06:18:09.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:09.833+0000 7f1ece800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f1ec80ce440 con 0x7f1ec80cae20 2024-05-28T06:18:09.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:09.833+0000 7f1ece800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f1ec80ceea0 con 0x7f1ec80caa40 2024-05-28T06:18:09.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:09.833+0000 7f1ecde00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1ec81522f0 0x7f1ec8148840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:09.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:09.833+0000 7f1ecde00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1ec81522f0 0x7f1ec8148840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48120/0 (socket says 172.21.15.96:48120) 2024-05-28T06:18:09.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:09.833+0000 7f1ecde00640 1 -- 172.21.15.96:0/3717780484 learned_addr learned my addr 172.21.15.96:0/3717780484 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:18:09.833 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:09.834+0000 7f1ecca00640 1 --2- 172.21.15.96:0/3717780484 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ec80cae20 0x7f1ec8154b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:09.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:09.834+0000 7f1ecd400640 1 --2- 172.21.15.96:0/3717780484 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1ec80caa40 0x7f1ec81465e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:09.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:09.834+0000 7f1ecde00640 1 -- 172.21.15.96:0/3717780484 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1ec80caa40 msgr2=0x7f1ec81465e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:09.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:09.834+0000 7f1ecde00640 1 --2- 172.21.15.96:0/3717780484 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1ec80caa40 0x7f1ec81465e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:09.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:09.834+0000 7f1ecde00640 1 -- 172.21.15.96:0/3717780484 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ec80cae20 msgr2=0x7f1ec8154b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:09.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:09.834+0000 7f1ecde00640 1 --2- 172.21.15.96:0/3717780484 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ec80cae20 0x7f1ec8154b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:09.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:09.834+0000 7f1ecde00640 1 -- 172.21.15.96:0/3717780484 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ec80b1410 con 0x7f1ec81522f0 2024-05-28T06:18:09.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:09.834+0000 7f1ecd400640 1 --2- 172.21.15.96:0/3717780484 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1ec80caa40 0x7f1ec81465e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:18:09.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:09.834+0000 7f1ecde00640 1 --2- 172.21.15.96:0/3717780484 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1ec81522f0 0x7f1ec8148840 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f1ec000c6b0 tx=0x7f1ec000cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:09.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:09.834+0000 7f1ec5600640 1 -- 172.21.15.96:0/3717780484 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ec0003b70 con 0x7f1ec81522f0 2024-05-28T06:18:09.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:09.834+0000 7f1ec5600640 1 -- 172.21.15.96:0/3717780484 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ec0003d30 con 0x7f1ec81522f0 2024-05-28T06:18:09.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:09.834+0000 7f1ec5600640 1 -- 172.21.15.96:0/3717780484 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ec0005310 con 0x7f1ec81522f0 2024-05-28T06:18:09.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:09.834+0000 7f1ece800640 1 -- 172.21.15.96:0/3717780484 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ec80b16f0 con 0x7f1ec81522f0 2024-05-28T06:18:09.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:09.835+0000 7f1ece800640 1 -- 172.21.15.96:0/3717780484 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ec80ad340 con 0x7f1ec81522f0 2024-05-28T06:18:09.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:09.836+0000 7f1ece800640 1 -- 172.21.15.96:0/3717780484 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ec80ad130 con 0x7f1ec81522f0 2024-05-28T06:18:09.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:09.837+0000 7f1ec5600640 1 -- 172.21.15.96:0/3717780484 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 30) ==== 92621+0+0 (secure 0 0 0) 0x7f1ec00054b0 con 0x7f1ec81522f0 2024-05-28T06:18:09.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:09.838+0000 7f1ec5600640 1 --2- 172.21.15.96:0/3717780484 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1ea806da60 0x7f1ea806ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:09.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:09.838+0000 7f1ec5600640 1 -- 172.21.15.96:0/3717780484 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 5637+0+0 (secure 0 0 0) 0x7f1ec008a8b0 con 0x7f1ec81522f0 2024-05-28T06:18:09.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:09.839+0000 7f1ecd400640 1 --2- 172.21.15.96:0/3717780484 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1ea806da60 0x7f1ea806ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:09.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:09.840+0000 7f1ecd400640 1 --2- 172.21.15.96:0/3717780484 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1ea806da60 0x7f1ea806ff20 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f1ebc006fd0 tx=0x7f1ebc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:09.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:09.843+0000 7f1ec5600640 1 -- 172.21.15.96:0/3717780484 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1ec005a440 con 0x7f1ec81522f0 2024-05-28T06:18:10.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:10.067+0000 7f1ece800640 1 -- 172.21.15.96:0/3717780484 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f1ec80689e0 con 0x7f1ec81522f0 2024-05-28T06:18:10.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:10.663+0000 7f1ec5600640 1 -- 172.21.15.96:0/3717780484 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 12 and data pool 13 v49) ==== 174+0+0 (secure 0 0 0) 0x7f1ec0059d40 con 0x7f1ec81522f0 2024-05-28T06:18:10.663 INFO:teuthology.orchestra.run.smithi096.stderr:new fs with metadata pool 12 and data pool 13 2024-05-28T06:18:10.669 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:10.671+0000 7f1ece800640 1 -- 172.21.15.96:0/3717780484 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1ea806da60 msgr2=0x7f1ea806ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:10.670 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:10.671+0000 7f1ece800640 1 --2- 172.21.15.96:0/3717780484 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1ea806da60 0x7f1ea806ff20 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f1ebc006fd0 tx=0x7f1ebc008040 comp rx=0 tx=0).stop 2024-05-28T06:18:10.670 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:10.671+0000 7f1ece800640 1 -- 172.21.15.96:0/3717780484 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1ec81522f0 msgr2=0x7f1ec8148840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:10.670 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:10.671+0000 7f1ece800640 1 --2- 172.21.15.96:0/3717780484 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1ec81522f0 0x7f1ec8148840 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f1ec000c6b0 tx=0x7f1ec000cb70 comp rx=0 tx=0).stop 2024-05-28T06:18:10.670 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:10.671+0000 7f1ece800640 1 -- 172.21.15.96:0/3717780484 shutdown_connections 2024-05-28T06:18:10.670 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:10.671+0000 7f1ece800640 1 --2- 172.21.15.96:0/3717780484 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1ea806da60 0x7f1ea806ff20 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:10.670 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:10.671+0000 7f1ece800640 1 --2- 172.21.15.96:0/3717780484 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1ec81522f0 0x7f1ec8148840 unknown :-1 s=CLOSED pgs=1391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:10.670 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:10.671+0000 7f1ece800640 1 --2- 172.21.15.96:0/3717780484 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ec80cae20 0x7f1ec8154b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:10.670 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:10.671+0000 7f1ece800640 1 --2- 172.21.15.96:0/3717780484 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1ec80caa40 0x7f1ec81465e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:10.670 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:10.671+0000 7f1ece800640 1 -- 172.21.15.96:0/3717780484 >> 172.21.15.96:0/3717780484 conn(0x7f1ec806b900 msgr2=0x7f1ec806f590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:18:10.671 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:10.672+0000 7f1ece800640 1 -- 172.21.15.96:0/3717780484 shutdown_connections 2024-05-28T06:18:10.671 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:10.672+0000 7f1ece800640 1 -- 172.21.15.96:0/3717780484 wait complete. 2024-05-28T06:18:10.700 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:18:10.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:10.870+0000 7f130fe00640 1 Processor -- start 2024-05-28T06:18:10.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:10.870+0000 7f130fe00640 1 -- start start 2024-05-28T06:18:10.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:10.871+0000 7f130fe00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f131015fae0 0x7f13100d4e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:10.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:10.871+0000 7f130fe00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f13100b6570 0x7f13100cf1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:10.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:10.871+0000 7f130fe00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13100d3300 con 0x7f1310160380 2024-05-28T06:18:10.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:10.871+0000 7f130fe00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f13100ced60 con 0x7f13100b6570 2024-05-28T06:18:10.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:10.871+0000 7f130fe00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f13100d35e0 con 0x7f131015fae0 2024-05-28T06:18:10.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:10.872+0000 7f130e000640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f1310160380 0x7f13100b5e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:49884/0 (socket says 172.21.15.96:49884) 2024-05-28T06:18:10.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:10.872+0000 7f130f400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f13100b6570 0x7f13100cf1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:10.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:10.872+0000 7f130e000640 1 -- 172.21.15.96:0/1828122335 learned_addr learned my addr 172.21.15.96:0/1828122335 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:18:10.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:10.872+0000 7f130ea00640 1 --2- 172.21.15.96:0/1828122335 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f131015fae0 0x7f13100d4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:10.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:10.872+0000 7f130f400640 1 -- 172.21.15.96:0/1828122335 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f131015fae0 msgr2=0x7f13100d4e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:18:10.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:10.872+0000 7f130f400640 1 --2- 172.21.15.96:0/1828122335 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f131015fae0 0x7f13100d4e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:10.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:10.872+0000 7f130f400640 1 -- 172.21.15.96:0/1828122335 >> v1:172.21.15.96:6789/0 conn(0x7f1310160380 legacy=0x7f13100b5e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:10.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:10.872+0000 7f130f400640 1 -- 172.21.15.96:0/1828122335 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13100d38c0 con 0x7f13100b6570 2024-05-28T06:18:10.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:10.872+0000 7f130d600640 1 -- 172.21.15.96:0/1828122335 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 804004913 0 0) 0x7f13100d3300 con 0x7f1310160380 2024-05-28T06:18:10.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:10.873+0000 7f130f400640 1 --2- 172.21.15.96:0/1828122335 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13100b6570 0x7f13100cf1f0 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f13080097a0 tx=0x7f130802a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca41c1aa1d1f5918 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:10.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:10.874+0000 7f130d600640 1 -- 172.21.15.96:0/1828122335 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1308037070 con 0x7f13100b6570 2024-05-28T06:18:10.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:10.874+0000 7f130d600640 1 -- 172.21.15.96:0/1828122335 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1308032440 con 0x7f13100b6570 2024-05-28T06:18:10.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:10.874+0000 7f130d600640 1 -- 172.21.15.96:0/1828122335 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1308032810 con 0x7f13100b6570 2024-05-28T06:18:10.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:10.874+0000 7f130fe00640 1 -- 172.21.15.96:0/1828122335 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13100b6570 msgr2=0x7f13100cf1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:10.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:10.874+0000 7f130fe00640 1 --2- 172.21.15.96:0/1828122335 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13100b6570 0x7f13100cf1f0 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f13080097a0 tx=0x7f130802a180 comp rx=0 tx=0).stop 2024-05-28T06:18:10.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:10.874+0000 7f130fe00640 1 -- 172.21.15.96:0/1828122335 shutdown_connections 2024-05-28T06:18:10.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:10.874+0000 7f130fe00640 1 --2- 172.21.15.96:0/1828122335 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13100b6570 0x7f13100cf1f0 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:10.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:10.874+0000 7f130fe00640 1 --2- 172.21.15.96:0/1828122335 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f131015fae0 0x7f13100d4e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:10.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:10.874+0000 7f130fe00640 1 -- 172.21.15.96:0/1828122335 >> 172.21.15.96:0/1828122335 conn(0x7f131008bfc0 msgr2=0x7f13100d4500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:18:10.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:10.874+0000 7f130fe00640 1 -- 172.21.15.96:0/1828122335 shutdown_connections 2024-05-28T06:18:10.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:10.874+0000 7f130fe00640 1 -- 172.21.15.96:0/1828122335 wait complete. 2024-05-28T06:18:10.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:10.876+0000 7f130fe00640 1 Processor -- start 2024-05-28T06:18:10.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:10.876+0000 7f130fe00640 1 -- start start 2024-05-28T06:18:10.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:10.877+0000 7f130fe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f13100b6570 0x7f13100cd010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:10.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:10.877+0000 7f130fe00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f131015fae0 0x7f131015af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:10.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:10.877+0000 7f130fe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1310160380 0x7f131015b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:10.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:10.877+0000 7f130fe00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f13100b5430 con 0x7f1310160380 2024-05-28T06:18:10.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:10.877+0000 7f130fe00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f131015fec0 con 0x7f13100b6570 2024-05-28T06:18:10.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:10.877+0000 7f130fe00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f13100b55b0 con 0x7f131015fae0 2024-05-28T06:18:10.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:10.877+0000 7f130f400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1310160380 0x7f131015b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:10.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:10.877+0000 7f130f400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1310160380 0x7f131015b490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48132/0 (socket says 172.21.15.96:48132) 2024-05-28T06:18:10.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:10.877+0000 7f130f400640 1 -- 172.21.15.96:0/3088942251 learned_addr learned my addr 172.21.15.96:0/3088942251 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:18:10.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:10.878+0000 7f130f400640 1 -- 172.21.15.96:0/3088942251 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f131015fae0 msgr2=0x7f131015af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:10.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:10.878+0000 7f130f400640 1 --2- 172.21.15.96:0/3088942251 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f131015fae0 0x7f131015af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:10.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:10.878+0000 7f130f400640 1 -- 172.21.15.96:0/3088942251 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f13100b6570 msgr2=0x7f13100cd010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:10.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:10.878+0000 7f130f400640 1 --2- 172.21.15.96:0/3088942251 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f13100b6570 0x7f13100cd010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:10.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:10.878+0000 7f130f400640 1 -- 172.21.15.96:0/3088942251 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13100cd730 con 0x7f1310160380 2024-05-28T06:18:10.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:10.878+0000 7f130f400640 1 --2- 172.21.15.96:0/3088942251 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1310160380 0x7f131015b490 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f13080098c0 tx=0x7f13080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:10.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:10.878+0000 7f1306a00640 1 -- 172.21.15.96:0/3088942251 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1308037040 con 0x7f1310160380 2024-05-28T06:18:10.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:10.878+0000 7f1306a00640 1 -- 172.21.15.96:0/3088942251 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1308042070 con 0x7f1310160380 2024-05-28T06:18:10.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:10.878+0000 7f1306a00640 1 -- 172.21.15.96:0/3088942251 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1308030b30 con 0x7f1310160380 2024-05-28T06:18:10.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:10.878+0000 7f130fe00640 1 -- 172.21.15.96:0/3088942251 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13100cda10 con 0x7f1310160380 2024-05-28T06:18:10.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:10.878+0000 7f130fe00640 1 -- 172.21.15.96:0/3088942251 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13100cdfd0 con 0x7f1310160380 2024-05-28T06:18:10.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:10.879+0000 7f130fe00640 1 -- 172.21.15.96:0/3088942251 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13100689e0 con 0x7f1310160380 2024-05-28T06:18:10.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:10.880+0000 7f1306a00640 1 -- 172.21.15.96:0/3088942251 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 30) ==== 92621+0+0 (secure 0 0 0) 0x7f1308030cd0 con 0x7f1310160380 2024-05-28T06:18:10.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:10.882+0000 7f1306a00640 1 --2- 172.21.15.96:0/3088942251 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f12ec06da60 0x7f12ec06ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:10.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:10.882+0000 7f1306a00640 1 -- 172.21.15.96:0/3088942251 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 5737+0+0 (secure 0 0 0) 0x7f13080ab590 con 0x7f1310160380 2024-05-28T06:18:10.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:10.882+0000 7f130ea00640 1 --2- 172.21.15.96:0/3088942251 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f12ec06da60 0x7f12ec06ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:10.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:10.884+0000 7f130ea00640 1 --2- 172.21.15.96:0/3088942251 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f12ec06da60 0x7f12ec06ff20 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f13000070b0 tx=0x7f1300007040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:10.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:10.887+0000 7f1306a00640 1 -- 172.21.15.96:0/3088942251 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1308032e20 con 0x7f1310160380 2024-05-28T06:18:11.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.089+0000 7f130fe00640 1 -- 172.21.15.96:0/3088942251 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f13100b6c90 con 0x7f1310160380 2024-05-28T06:18:11.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.091+0000 7f1306a00640 1 -- 172.21.15.96:0/3088942251 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+16728 (secure 0 0 0) 0x7f1308045360 con 0x7f1310160380 2024-05-28T06:18:11.090 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:18:11.090 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":88,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:18:10.069661+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-05-28T06:18:07.815707+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-05-28T06:18:09.061362+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:18:11.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.094+0000 7f130fe00640 1 -- 172.21.15.96:0/3088942251 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f12ec06da60 msgr2=0x7f12ec06ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:11.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.094+0000 7f130fe00640 1 --2- 172.21.15.96:0/3088942251 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f12ec06da60 0x7f12ec06ff20 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f13000070b0 tx=0x7f1300007040 comp rx=0 tx=0).stop 2024-05-28T06:18:11.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.094+0000 7f130fe00640 1 -- 172.21.15.96:0/3088942251 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1310160380 msgr2=0x7f131015b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:11.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.094+0000 7f130fe00640 1 --2- 172.21.15.96:0/3088942251 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1310160380 0x7f131015b490 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f13080098c0 tx=0x7f13080046c0 comp rx=0 tx=0).stop 2024-05-28T06:18:11.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.094+0000 7f130fe00640 1 -- 172.21.15.96:0/3088942251 shutdown_connections 2024-05-28T06:18:11.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.094+0000 7f130fe00640 1 --2- 172.21.15.96:0/3088942251 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f12ec06da60 0x7f12ec06ff20 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:11.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.094+0000 7f130fe00640 1 --2- 172.21.15.96:0/3088942251 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1310160380 0x7f131015b490 unknown :-1 s=CLOSED pgs=1394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:11.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.094+0000 7f130fe00640 1 --2- 172.21.15.96:0/3088942251 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f131015fae0 0x7f131015af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:11.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.094+0000 7f130fe00640 1 --2- 172.21.15.96:0/3088942251 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f13100b6570 0x7f13100cd010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:11.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.094+0000 7f130fe00640 1 -- 172.21.15.96:0/3088942251 >> 172.21.15.96:0/3088942251 conn(0x7f131008bfc0 msgr2=0x7f13101607d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:18:11.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.094+0000 7f130fe00640 1 -- 172.21.15.96:0/3088942251 shutdown_connections 2024-05-28T06:18:11.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.095+0000 7f130fe00640 1 -- 172.21.15.96:0/3088942251 wait complete. 2024-05-28T06:18:11.111 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:18:11.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.314+0000 7f73dea00640 1 Processor -- start 2024-05-28T06:18:11.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.314+0000 7f73dea00640 1 -- start start 2024-05-28T06:18:11.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.316+0000 7f73dea00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f73d80b4e10 0x7f73d80cde30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:11.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.316+0000 7f73dea00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73d80b51f0 con 0x7f73d80d0ca0 2024-05-28T06:18:11.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.316+0000 7f73dea00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73d80aef60 con 0x7f73d80ce370 2024-05-28T06:18:11.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.316+0000 7f73dea00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f73d80ad1d0 con 0x7f73d80b4e10 2024-05-28T06:18:11.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.316+0000 7f73d7e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f73d80b4e10 0x7f73d80cde30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:11.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.316+0000 7f73dca00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f73d80d0ca0 0x7f73d8099090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:49900/0 (socket says 172.21.15.96:49900) 2024-05-28T06:18:11.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.316+0000 7f73dca00640 1 -- 172.21.15.96:0/2368563684 learned_addr learned my addr 172.21.15.96:0/2368563684 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:18:11.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.317+0000 7f73d7e00640 1 -- 172.21.15.96:0/2368563684 >> v1:172.21.15.123:6789/0 conn(0x7f73d80ce370 legacy=0x7f73d80d0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:18:11.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.317+0000 7f73d7e00640 1 -- 172.21.15.96:0/2368563684 >> v1:172.21.15.96:6789/0 conn(0x7f73d80d0ca0 legacy=0x7f73d8099090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:11.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.317+0000 7f73d7e00640 1 -- 172.21.15.96:0/2368563684 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73d80ad470 con 0x7f73d80b4e10 2024-05-28T06:18:11.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.318+0000 7f73d7e00640 1 --2- 172.21.15.96:0/2368563684 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f73d80b4e10 0x7f73d80cde30 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7f73c80097a0 tx=0x7f73c802a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa35453215d2693d server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:11.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.318+0000 7f73d6a00640 1 -- 172.21.15.96:0/2368563684 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73c8037070 con 0x7f73d80b4e10 2024-05-28T06:18:11.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.318+0000 7f73d6a00640 1 -- 172.21.15.96:0/2368563684 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73c802ae50 con 0x7f73d80b4e10 2024-05-28T06:18:11.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.318+0000 7f73d6a00640 1 -- 172.21.15.96:0/2368563684 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73c8032660 con 0x7f73d80b4e10 2024-05-28T06:18:11.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.318+0000 7f73dea00640 1 -- 172.21.15.96:0/2368563684 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f73d80b4e10 msgr2=0x7f73d80cde30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:11.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.318+0000 7f73dea00640 1 --2- 172.21.15.96:0/2368563684 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f73d80b4e10 0x7f73d80cde30 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7f73c80097a0 tx=0x7f73c802a140 comp rx=0 tx=0).stop 2024-05-28T06:18:11.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.319+0000 7f73dea00640 1 -- 172.21.15.96:0/2368563684 shutdown_connections 2024-05-28T06:18:11.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.319+0000 7f73dea00640 1 --2- 172.21.15.96:0/2368563684 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f73d80b4e10 0x7f73d80cde30 unknown :-1 s=CLOSED pgs=1396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:11.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.319+0000 7f73dea00640 1 -- 172.21.15.96:0/2368563684 >> 172.21.15.96:0/2368563684 conn(0x7f73d806bae0 msgr2=0x7f73d806bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:18:11.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.319+0000 7f73dea00640 1 -- 172.21.15.96:0/2368563684 shutdown_connections 2024-05-28T06:18:11.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.319+0000 7f73dea00640 1 -- 172.21.15.96:0/2368563684 wait complete. 2024-05-28T06:18:11.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.320+0000 7f73dea00640 1 Processor -- start 2024-05-28T06:18:11.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.321+0000 7f73dea00640 1 -- start start 2024-05-28T06:18:11.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.321+0000 7f73dea00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f73d80b4e10 0x7f73d81f7590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:11.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.321+0000 7f73dea00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f73d80ce370 0x7f73d80c95a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:11.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.322+0000 7f73dea00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f73d80d0ca0 0x7f73d815ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:11.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.322+0000 7f73dea00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f73d80cea00 con 0x7f73d80b4e10 2024-05-28T06:18:11.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.322+0000 7f73dea00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f73d80ce880 con 0x7f73d80d0ca0 2024-05-28T06:18:11.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.322+0000 7f73dea00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f73d80b3150 con 0x7f73d80ce370 2024-05-28T06:18:11.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.322+0000 7f73d7e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f73d80b4e10 0x7f73d81f7590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:11.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.322+0000 7f73d7e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f73d80b4e10 0x7f73d81f7590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48154/0 (socket says 172.21.15.96:48154) 2024-05-28T06:18:11.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.322+0000 7f73d7e00640 1 -- 172.21.15.96:0/2216679114 learned_addr learned my addr 172.21.15.96:0/2216679114 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:18:11.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.322+0000 7f73d7400640 1 --2- 172.21.15.96:0/2216679114 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f73d80ce370 0x7f73d80c95a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:11.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.322+0000 7f73d7e00640 1 -- 172.21.15.96:0/2216679114 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f73d80ce370 msgr2=0x7f73d80c95a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:11.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.322+0000 7f73d7e00640 1 --2- 172.21.15.96:0/2216679114 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f73d80ce370 0x7f73d80c95a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:11.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.322+0000 7f73d7e00640 1 -- 172.21.15.96:0/2216679114 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f73d80d0ca0 msgr2=0x7f73d815ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:11.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.322+0000 7f73d7e00640 1 --2- 172.21.15.96:0/2216679114 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f73d80d0ca0 0x7f73d815ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:11.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.322+0000 7f73d7e00640 1 -- 172.21.15.96:0/2216679114 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73d80c6190 con 0x7f73d80b4e10 2024-05-28T06:18:11.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.323+0000 7f73d7e00640 1 --2- 172.21.15.96:0/2216679114 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f73d80b4e10 0x7f73d81f7590 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7f73c80098c0 tx=0x7f73c80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:11.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.323+0000 7f73cfe00640 1 -- 172.21.15.96:0/2216679114 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73c8042070 con 0x7f73d80b4e10 2024-05-28T06:18:11.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.323+0000 7f73cfe00640 1 -- 172.21.15.96:0/2216679114 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73c803e670 con 0x7f73d80b4e10 2024-05-28T06:18:11.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.323+0000 7f73dea00640 1 -- 172.21.15.96:0/2216679114 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73d80c6430 con 0x7f73d80b4e10 2024-05-28T06:18:11.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.323+0000 7f73cfe00640 1 -- 172.21.15.96:0/2216679114 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73c8037040 con 0x7f73d80b4e10 2024-05-28T06:18:11.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.323+0000 7f73dea00640 1 -- 172.21.15.96:0/2216679114 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73d80c69f0 con 0x7f73d80b4e10 2024-05-28T06:18:11.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.324+0000 7f73dea00640 1 -- 172.21.15.96:0/2216679114 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73d80689e0 con 0x7f73d80b4e10 2024-05-28T06:18:11.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.326+0000 7f73cfe00640 1 -- 172.21.15.96:0/2216679114 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 30) ==== 92621+0+0 (secure 0 0 0) 0x7f73c8045070 con 0x7f73d80b4e10 2024-05-28T06:18:11.325 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.327+0000 7f73cfe00640 1 --2- 172.21.15.96:0/2216679114 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f73b806da60 0x7f73b806ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:11.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.327+0000 7f73cfe00640 1 -- 172.21.15.96:0/2216679114 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 5737+0+0 (secure 0 0 0) 0x7f73c80ab540 con 0x7f73d80b4e10 2024-05-28T06:18:11.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.327+0000 7f73d7400640 1 --2- 172.21.15.96:0/2216679114 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f73b806da60 0x7f73b806ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:11.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.328+0000 7f73d7400640 1 --2- 172.21.15.96:0/2216679114 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f73b806da60 0x7f73b806ff20 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f73d80aeda0 tx=0x7f73c4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:11.328 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.330+0000 7f73cfe00640 1 -- 172.21.15.96:0/2216679114 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f73c807b070 con 0x7f73d80b4e10 2024-05-28T06:18:11.533 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.534+0000 7f73dea00640 1 -- 172.21.15.96:0/2216679114 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f73d806f540 con 0x7f73d80b4e10 2024-05-28T06:18:11.536 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.537+0000 7f73cfe00640 1 -- 172.21.15.96:0/2216679114 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+16728 (secure 0 0 0) 0x7f73c807a970 con 0x7f73d80b4e10 2024-05-28T06:18:11.536 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:18:11.536 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":88,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:18:10.069661+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-05-28T06:18:07.815707+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-05-28T06:18:09.061362+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:18:11.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.540+0000 7f73dea00640 1 -- 172.21.15.96:0/2216679114 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f73b806da60 msgr2=0x7f73b806ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:11.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.540+0000 7f73dea00640 1 --2- 172.21.15.96:0/2216679114 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f73b806da60 0x7f73b806ff20 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f73d80aeda0 tx=0x7f73c4009040 comp rx=0 tx=0).stop 2024-05-28T06:18:11.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.540+0000 7f73dea00640 1 -- 172.21.15.96:0/2216679114 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f73d80b4e10 msgr2=0x7f73d81f7590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:11.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.540+0000 7f73dea00640 1 --2- 172.21.15.96:0/2216679114 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f73d80b4e10 0x7f73d81f7590 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7f73c80098c0 tx=0x7f73c80046c0 comp rx=0 tx=0).stop 2024-05-28T06:18:11.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.540+0000 7f73dea00640 1 -- 172.21.15.96:0/2216679114 shutdown_connections 2024-05-28T06:18:11.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.540+0000 7f73dea00640 1 --2- 172.21.15.96:0/2216679114 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f73b806da60 0x7f73b806ff20 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:11.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.540+0000 7f73dea00640 1 --2- 172.21.15.96:0/2216679114 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f73d80d0ca0 0x7f73d815ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:11.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.540+0000 7f73dea00640 1 --2- 172.21.15.96:0/2216679114 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f73d80ce370 0x7f73d80c95a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:11.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.541+0000 7f73dea00640 1 --2- 172.21.15.96:0/2216679114 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f73d80b4e10 0x7f73d81f7590 unknown :-1 s=CLOSED pgs=1397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:11.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.541+0000 7f73dea00640 1 -- 172.21.15.96:0/2216679114 >> 172.21.15.96:0/2216679114 conn(0x7f73d806bae0 msgr2=0x7f73d80ca840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:18:11.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.541+0000 7f73dea00640 1 -- 172.21.15.96:0/2216679114 shutdown_connections 2024-05-28T06:18:11.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.541+0000 7f73dea00640 1 -- 172.21.15.96:0/2216679114 wait complete. 2024-05-28T06:18:11.561 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-05-28T06:18:11.592 INFO:tasks.ceph.mds.c.smithi096.stderr:2024-05-28T06:18:11.593+0000 7f2420800640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-28T06:18:11.770 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.772+0000 7f5f30800640 1 Processor -- start 2024-05-28T06:18:11.770 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.772+0000 7f5f30800640 1 -- start start 2024-05-28T06:18:11.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.773+0000 7f5f30800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f5f280d0640 0x7f5f2815d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:11.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.773+0000 7f5f30800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5f280b32c0 0x7f5f280b3730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:11.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.774+0000 7f5f30800640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f280d0060 con 0x7f5f280d0a20 2024-05-28T06:18:11.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.774+0000 7f5f30800640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f5f280adbb0 con 0x7f5f280d0640 2024-05-28T06:18:11.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.774+0000 7f5f30800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f5f2815db10 con 0x7f5f280b32c0 2024-05-28T06:18:11.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.774+0000 7f5f2de00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f5f280d0640 0x7f5f2815d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:11.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.774+0000 7f5f2de00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f5f280d0640 0x7f5f2815d5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48164/0 (socket says 172.21.15.96:48164) 2024-05-28T06:18:11.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.774+0000 7f5f2de00640 1 -- 172.21.15.96:0/1461281316 learned_addr learned my addr 172.21.15.96:0/1461281316 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:18:11.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.774+0000 7f5f2e800640 1 --2- 172.21.15.96:0/1461281316 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5f280b32c0 0x7f5f280b3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:11.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.774+0000 7f5f2de00640 1 -- 172.21.15.96:0/1461281316 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5f280b32c0 msgr2=0x7f5f280b3730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:18:11.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.774+0000 7f5f2de00640 1 --2- 172.21.15.96:0/1461281316 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5f280b32c0 0x7f5f280b3730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:11.774 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.774+0000 7f5f2de00640 1 -- 172.21.15.96:0/1461281316 >> v1:172.21.15.123:6789/0 conn(0x7f5f280d0a20 legacy=0x7f5f280cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:18:11.774 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.774+0000 7f5f2de00640 1 -- 172.21.15.96:0/1461281316 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f2815cb10 con 0x7f5f280d0640 2024-05-28T06:18:11.774 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.776+0000 7f5f2de00640 1 --2- 172.21.15.96:0/1461281316 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5f280d0640 0x7f5f2815d5d0 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c009870 tx=0x7f5f1c02a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e955c7ba5824bff server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:11.774 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.776+0000 7f5f2ca00640 1 -- 172.21.15.96:0/1461281316 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f1c037070 con 0x7f5f280d0640 2024-05-28T06:18:11.775 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.776+0000 7f5f2ca00640 1 -- 172.21.15.96:0/1461281316 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f1c032440 con 0x7f5f280d0640 2024-05-28T06:18:11.775 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.776+0000 7f5f2ca00640 1 -- 172.21.15.96:0/1461281316 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f1c032810 con 0x7f5f280d0640 2024-05-28T06:18:11.775 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.776+0000 7f5f30800640 1 -- 172.21.15.96:0/1461281316 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5f280d0640 msgr2=0x7f5f2815d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:11.775 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.776+0000 7f5f30800640 1 --2- 172.21.15.96:0/1461281316 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5f280d0640 0x7f5f2815d5d0 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c009870 tx=0x7f5f1c02a210 comp rx=0 tx=0).stop 2024-05-28T06:18:11.775 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.777+0000 7f5f30800640 1 -- 172.21.15.96:0/1461281316 shutdown_connections 2024-05-28T06:18:11.775 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.777+0000 7f5f30800640 1 --2- 172.21.15.96:0/1461281316 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5f280b32c0 0x7f5f280b3730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:11.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.777+0000 7f5f30800640 1 --2- 172.21.15.96:0/1461281316 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5f280d0640 0x7f5f2815d5d0 unknown :-1 s=CLOSED pgs=1398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:11.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.777+0000 7f5f30800640 1 -- 172.21.15.96:0/1461281316 >> 172.21.15.96:0/1461281316 conn(0x7f5f2806bd20 msgr2=0x7f5f2806c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:18:11.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.777+0000 7f5f30800640 1 -- 172.21.15.96:0/1461281316 shutdown_connections 2024-05-28T06:18:11.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.777+0000 7f5f30800640 1 -- 172.21.15.96:0/1461281316 wait complete. 2024-05-28T06:18:11.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.778+0000 7f5f30800640 1 Processor -- start 2024-05-28T06:18:11.777 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.779+0000 7f5f30800640 1 -- start start 2024-05-28T06:18:11.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.779+0000 7f5f30800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5f280b32c0 0x7f5f28143370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:11.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.779+0000 7f5f30800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5f280d0640 0x7f5f28145c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:11.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.780+0000 7f5f30800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f280d0a20 0x7f5f2815b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:11.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.780+0000 7f5f30800640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f5f2815cdd0 con 0x7f5f280b32c0 2024-05-28T06:18:11.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.780+0000 7f5f30800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5f280ad8d0 con 0x7f5f280d0a20 2024-05-28T06:18:11.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.780+0000 7f5f30800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f5f2815d0b0 con 0x7f5f280d0640 2024-05-28T06:18:11.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.780+0000 7f5f2de00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5f280b32c0 0x7f5f28143370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:11.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.780+0000 7f5f2de00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5f280b32c0 0x7f5f28143370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48178/0 (socket says 172.21.15.96:48178) 2024-05-28T06:18:11.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.780+0000 7f5f2de00640 1 -- 172.21.15.96:0/1751982686 learned_addr learned my addr 172.21.15.96:0/1751982686 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:18:11.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.780+0000 7f5f2de00640 1 -- 172.21.15.96:0/1751982686 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5f280d0640 msgr2=0x7f5f28145c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:11.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.780+0000 7f5f2e800640 1 --2- 172.21.15.96:0/1751982686 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f280d0a20 0x7f5f2815b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:11.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.780+0000 7f5f2d400640 1 --2- 172.21.15.96:0/1751982686 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5f280d0640 0x7f5f28145c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:11.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.780+0000 7f5f2de00640 1 --2- 172.21.15.96:0/1751982686 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5f280d0640 0x7f5f28145c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:11.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.780+0000 7f5f2de00640 1 -- 172.21.15.96:0/1751982686 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f280d0a20 msgr2=0x7f5f2815b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:11.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.780+0000 7f5f2de00640 1 --2- 172.21.15.96:0/1751982686 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f280d0a20 0x7f5f2815b240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:11.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.780+0000 7f5f2de00640 1 -- 172.21.15.96:0/1751982686 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f281466e0 con 0x7f5f280b32c0 2024-05-28T06:18:11.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.781+0000 7f5f2de00640 1 --2- 172.21.15.96:0/1751982686 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5f280b32c0 0x7f5f28143370 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c009870 tx=0x7f5f1c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:11.781 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.781+0000 7f5f22000640 1 -- 172.21.15.96:0/1751982686 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f1c041020 con 0x7f5f280b32c0 2024-05-28T06:18:11.781 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.781+0000 7f5f22000640 1 -- 172.21.15.96:0/1751982686 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f1c030970 con 0x7f5f280b32c0 2024-05-28T06:18:11.781 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.781+0000 7f5f22000640 1 -- 172.21.15.96:0/1751982686 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f1c037040 con 0x7f5f280b32c0 2024-05-28T06:18:11.781 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.781+0000 7f5f30800640 1 -- 172.21.15.96:0/1751982686 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f281469c0 con 0x7f5f280b32c0 2024-05-28T06:18:11.781 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.781+0000 7f5f30800640 1 -- 172.21.15.96:0/1751982686 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f281440a0 con 0x7f5f280b32c0 2024-05-28T06:18:11.781 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.782+0000 7f5f30800640 1 -- 172.21.15.96:0/1751982686 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f280689e0 con 0x7f5f280b32c0 2024-05-28T06:18:11.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.784+0000 7f5f22000640 1 -- 172.21.15.96:0/1751982686 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 30) ==== 92621+0+0 (secure 0 0 0) 0x7f5f1c044020 con 0x7f5f280b32c0 2024-05-28T06:18:11.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.784+0000 7f5f22000640 1 --2- 172.21.15.96:0/1751982686 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5f0806da60 0x7f5f0806ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:11.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.785+0000 7f5f22000640 1 -- 172.21.15.96:0/1751982686 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 5737+0+0 (secure 0 0 0) 0x7f5f1c0abab0 con 0x7f5f280b32c0 2024-05-28T06:18:11.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.785+0000 7f5f2d400640 1 --2- 172.21.15.96:0/1751982686 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5f0806da60 0x7f5f0806ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:11.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.786+0000 7f5f2d400640 1 --2- 172.21.15.96:0/1751982686 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5f0806da60 0x7f5f0806ff20 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f5f280cf7b0 tx=0x7f5f14008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:11.786 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:11.788+0000 7f5f22000640 1 -- 172.21.15.96:0/1751982686 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5f1c04b030 con 0x7f5f280b32c0 2024-05-28T06:18:12.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.026+0000 7f5f30800640 1 -- 172.21.15.96:0/1751982686 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f5f28071ef0 con 0x7f5f280b32c0 2024-05-28T06:18:12.672 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.673+0000 7f5f22000640 1 -- 172.21.15.96:0/1751982686 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v51) ==== 118+0+0 (secure 0 0 0) 0x7f5f1c07b670 con 0x7f5f280b32c0 2024-05-28T06:18:12.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.677+0000 7f5f30800640 1 -- 172.21.15.96:0/1751982686 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5f0806da60 msgr2=0x7f5f0806ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:12.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.678+0000 7f5f30800640 1 --2- 172.21.15.96:0/1751982686 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5f0806da60 0x7f5f0806ff20 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f5f280cf7b0 tx=0x7f5f14008040 comp rx=0 tx=0).stop 2024-05-28T06:18:12.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.678+0000 7f5f30800640 1 -- 172.21.15.96:0/1751982686 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5f280b32c0 msgr2=0x7f5f28143370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:12.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.678+0000 7f5f30800640 1 --2- 172.21.15.96:0/1751982686 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5f280b32c0 0x7f5f28143370 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c009870 tx=0x7f5f1c0046c0 comp rx=0 tx=0).stop 2024-05-28T06:18:12.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.678+0000 7f5f30800640 1 -- 172.21.15.96:0/1751982686 shutdown_connections 2024-05-28T06:18:12.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.678+0000 7f5f30800640 1 --2- 172.21.15.96:0/1751982686 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5f0806da60 0x7f5f0806ff20 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:12.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.678+0000 7f5f30800640 1 --2- 172.21.15.96:0/1751982686 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f280d0a20 0x7f5f2815b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:12.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.678+0000 7f5f30800640 1 --2- 172.21.15.96:0/1751982686 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5f280d0640 0x7f5f28145c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:12.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.678+0000 7f5f30800640 1 --2- 172.21.15.96:0/1751982686 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5f280b32c0 0x7f5f28143370 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:12.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.678+0000 7f5f30800640 1 -- 172.21.15.96:0/1751982686 >> 172.21.15.96:0/1751982686 conn(0x7f5f2806bd20 msgr2=0x7f5f2815ef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:18:12.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.678+0000 7f5f30800640 1 -- 172.21.15.96:0/1751982686 shutdown_connections 2024-05-28T06:18:12.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.678+0000 7f5f30800640 1 -- 172.21.15.96:0/1751982686 wait complete. 2024-05-28T06:18:12.698 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-05-28T06:18:12.698 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-05-28T06:18:12.953 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.955+0000 7fbc3ee00640 1 Processor -- start 2024-05-28T06:18:12.954 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.955+0000 7fbc3ee00640 1 -- start start 2024-05-28T06:18:12.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.956+0000 7fbc3ee00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fbc380d0640 0x7fbc3815d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:12.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.957+0000 7fbc3ee00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc380d0060 con 0x7fbc380d0a20 2024-05-28T06:18:12.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.957+0000 7fbc3ee00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc3815cb10 con 0x7fbc380b32c0 2024-05-28T06:18:12.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.957+0000 7fbc3ee00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fbc380adbb0 con 0x7fbc380d0640 2024-05-28T06:18:12.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.957+0000 7fbc37400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fbc380d0a20 0x7fbc380cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:49906/0 (socket says 172.21.15.96:49906) 2024-05-28T06:18:12.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.957+0000 7fbc37400640 1 -- 172.21.15.96:0/1889407263 learned_addr learned my addr 172.21.15.96:0/1889407263 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:18:12.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.957+0000 7fbc37e00640 1 --2- 172.21.15.96:0/1889407263 >> v2:172.21.15.123:3300/0 conn(0x7fbc380d0640 0x7fbc3815d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:12.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.957+0000 7fbc36a00640 1 -- 172.21.15.96:0/1889407263 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 886112229 0 0) 0x7fbc380d0060 con 0x7fbc380d0a20 2024-05-28T06:18:12.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.958+0000 7fbc36a00640 1 -- 172.21.15.96:0/1889407263 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc1c003520 con 0x7fbc380d0a20 2024-05-28T06:18:12.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.958+0000 7fbc36a00640 1 -- 172.21.15.96:0/1889407263 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1523814859 0 0) 0x7fbc1c003520 con 0x7fbc380d0a20 2024-05-28T06:18:12.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.958+0000 7fbc36a00640 1 -- 172.21.15.96:0/1889407263 >> v2:172.21.15.123:3300/0 conn(0x7fbc380d0640 msgr2=0x7fbc3815d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:18:12.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.958+0000 7fbc36a00640 1 --2- 172.21.15.96:0/1889407263 >> v2:172.21.15.123:3300/0 conn(0x7fbc380d0640 0x7fbc3815d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:12.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.958+0000 7fbc36a00640 1 -- 172.21.15.96:0/1889407263 >> v1:172.21.15.123:6789/0 conn(0x7fbc380b32c0 legacy=0x7fbc380b3730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:12.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.958+0000 7fbc36a00640 1 -- 172.21.15.96:0/1889407263 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc380ad5f0 con 0x7fbc380d0a20 2024-05-28T06:18:12.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.958+0000 7fbc36a00640 1 -- 172.21.15.96:0/1889407263 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7fbc24001670 con 0x7fbc380d0a20 2024-05-28T06:18:12.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.958+0000 7fbc36a00640 1 -- 172.21.15.96:0/1889407263 >> v1:172.21.15.96:6789/0 conn(0x7fbc380d0a20 legacy=0x7fbc380cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:12.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.958+0000 7fbc36a00640 1 --2- 172.21.15.96:0/1889407263 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbc1c0039b0 0x7fbc1c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:12.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.958+0000 7fbc36a00640 1 --2- 172.21.15.96:0/1889407263 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc1c004800 0x7fbc1c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:12.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.958+0000 7fbc36a00640 1 --2- 172.21.15.96:0/1889407263 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc1c0051c0 0x7fbc1c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:12.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.958+0000 7fbc36a00640 1 -- 172.21.15.96:0/1889407263 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fbc1c00c130 con 0x7fbc1c0051c0 2024-05-28T06:18:12.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.958+0000 7fbc36a00640 1 -- 172.21.15.96:0/1889407263 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fbc1c00c2e0 con 0x7fbc1c004800 2024-05-28T06:18:12.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.958+0000 7fbc36a00640 1 -- 172.21.15.96:0/1889407263 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fbc1c00c4e0 con 0x7fbc1c0039b0 2024-05-28T06:18:12.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.958+0000 7fbc3ce00640 1 --2- 172.21.15.96:0/1889407263 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc1c0051c0 0x7fbc1c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:12.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.959+0000 7fbc37400640 1 --2- 172.21.15.96:0/1889407263 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc1c004800 0x7fbc1c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:12.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.959+0000 7fbc37e00640 1 --2- 172.21.15.96:0/1889407263 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbc1c0039b0 0x7fbc1c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:12.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.959+0000 7fbc3ce00640 1 -- 172.21.15.96:0/1889407263 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbc1c0039b0 msgr2=0x7fbc1c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:18:12.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.959+0000 7fbc3ce00640 1 --2- 172.21.15.96:0/1889407263 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbc1c0039b0 0x7fbc1c003dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:12.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.959+0000 7fbc3ce00640 1 -- 172.21.15.96:0/1889407263 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc1c004800 msgr2=0x7fbc1c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:18:12.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.959+0000 7fbc3ce00640 1 --2- 172.21.15.96:0/1889407263 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc1c004800 0x7fbc1c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:12.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.959+0000 7fbc3ce00640 1 -- 172.21.15.96:0/1889407263 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc1c003520 con 0x7fbc1c0051c0 2024-05-28T06:18:12.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.960+0000 7fbc3ce00640 1 -- 172.21.15.96:0/1889407263 reap_dead start 2024-05-28T06:18:12.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.960+0000 7fbc3ce00640 1 --2- 172.21.15.96:0/1889407263 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc1c0051c0 0x7fbc1c00b9c0 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7fbc30001e50 tx=0x7fbc1c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b57ba20c39ac0476 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:12.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.960+0000 7fbc36a00640 1 -- 172.21.15.96:0/1889407263 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc30001a70 con 0x7fbc1c0051c0 2024-05-28T06:18:12.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.960+0000 7fbc36a00640 1 -- 172.21.15.96:0/1889407263 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc30003970 con 0x7fbc1c0051c0 2024-05-28T06:18:12.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.960+0000 7fbc36a00640 1 -- 172.21.15.96:0/1889407263 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc30003d60 con 0x7fbc1c0051c0 2024-05-28T06:18:12.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.961+0000 7fbc3ee00640 1 -- 172.21.15.96:0/1889407263 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc1c0051c0 msgr2=0x7fbc1c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:12.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.961+0000 7fbc3ee00640 1 --2- 172.21.15.96:0/1889407263 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc1c0051c0 0x7fbc1c00b9c0 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7fbc30001e50 tx=0x7fbc1c005830 comp rx=0 tx=0).stop 2024-05-28T06:18:12.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.961+0000 7fbc3ee00640 1 -- 172.21.15.96:0/1889407263 shutdown_connections 2024-05-28T06:18:12.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.961+0000 7fbc3ee00640 1 --2- 172.21.15.96:0/1889407263 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc1c0051c0 0x7fbc1c00b9c0 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:12.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.961+0000 7fbc3ee00640 1 -- 172.21.15.96:0/1889407263 >> 172.21.15.96:0/1889407263 conn(0x7fbc3806bd20 msgr2=0x7fbc3806c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:18:12.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.961+0000 7fbc3ee00640 1 -- 172.21.15.96:0/1889407263 shutdown_connections 2024-05-28T06:18:12.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.961+0000 7fbc3ee00640 1 -- 172.21.15.96:0/1889407263 wait complete. 2024-05-28T06:18:12.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.963+0000 7fbc3ee00640 1 Processor -- start 2024-05-28T06:18:12.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.963+0000 7fbc3ee00640 1 -- start start 2024-05-28T06:18:12.966 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.964+0000 7fbc3ee00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbc1c0051c0 0x7fbc38147740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:12.966 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.964+0000 7fbc3ee00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc3814c160 0x7fbc3814c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:12.966 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.964+0000 7fbc3ee00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc380b1640 0x7fbc380b1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:12.966 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.964+0000 7fbc3ee00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fbc3815d0b0 con 0x7fbc3814c160 2024-05-28T06:18:12.967 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.964+0000 7fbc3ee00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fbc3815cdd0 con 0x7fbc380b1640 2024-05-28T06:18:12.967 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.964+0000 7fbc3ee00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fbc380ad2a0 con 0x7fbc1c0051c0 2024-05-28T06:18:12.967 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.964+0000 7fbc37400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc3814c160 0x7fbc3814c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:12.967 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.964+0000 7fbc37400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc3814c160 0x7fbc3814c5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48194/0 (socket says 172.21.15.96:48194) 2024-05-28T06:18:12.967 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.964+0000 7fbc37400640 1 -- 172.21.15.96:0/3113711535 learned_addr learned my addr 172.21.15.96:0/3113711535 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:18:12.968 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.965+0000 7fbc37400640 1 -- 172.21.15.96:0/3113711535 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbc1c0051c0 msgr2=0x7fbc38147740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:12.968 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.965+0000 7fbc37e00640 1 --2- 172.21.15.96:0/3113711535 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbc1c0051c0 0x7fbc38147740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:12.968 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.965+0000 7fbc3ce00640 1 --2- 172.21.15.96:0/3113711535 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc380b1640 0x7fbc380b1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:12.968 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.965+0000 7fbc37400640 1 --2- 172.21.15.96:0/3113711535 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbc1c0051c0 0x7fbc38147740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:12.968 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.965+0000 7fbc37400640 1 -- 172.21.15.96:0/3113711535 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc380b1640 msgr2=0x7fbc380b1ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:12.968 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.965+0000 7fbc37400640 1 --2- 172.21.15.96:0/3113711535 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc380b1640 0x7fbc380b1ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:12.968 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.965+0000 7fbc37400640 1 -- 172.21.15.96:0/3113711535 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc38148460 con 0x7fbc3814c160 2024-05-28T06:18:12.968 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.965+0000 7fbc37400640 1 --2- 172.21.15.96:0/3113711535 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc3814c160 0x7fbc3814c5e0 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7fbc2400e830 tx=0x7fbc2400ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:12.968 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.965+0000 7fbc2fe00640 1 -- 172.21.15.96:0/3113711535 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc2400ee20 con 0x7fbc3814c160 2024-05-28T06:18:12.968 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.965+0000 7fbc2fe00640 1 -- 172.21.15.96:0/3113711535 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc24018070 con 0x7fbc3814c160 2024-05-28T06:18:12.968 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.965+0000 7fbc2fe00640 1 -- 172.21.15.96:0/3113711535 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc2400cb10 con 0x7fbc3814c160 2024-05-28T06:18:12.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.965+0000 7fbc3ee00640 1 -- 172.21.15.96:0/3113711535 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc38147c80 con 0x7fbc3814c160 2024-05-28T06:18:12.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.965+0000 7fbc3ee00640 1 -- 172.21.15.96:0/3113711535 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc381489e0 con 0x7fbc3814c160 2024-05-28T06:18:12.970 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.966+0000 7fbc3ee00640 1 -- 172.21.15.96:0/3113711535 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc380689e0 con 0x7fbc3814c160 2024-05-28T06:18:12.970 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.967+0000 7fbc2fe00640 1 -- 172.21.15.96:0/3113711535 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 30) ==== 92621+0+0 (secure 0 0 0) 0x7fbc2400ccb0 con 0x7fbc3814c160 2024-05-28T06:18:12.970 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.969+0000 7fbc2fe00640 1 --2- 172.21.15.96:0/3113711535 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbc1806da60 0x7fbc1806ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:12.970 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.969+0000 7fbc2fe00640 1 -- 172.21.15.96:0/3113711535 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 5737+0+0 (secure 0 0 0) 0x7fbc2408a900 con 0x7fbc3814c160 2024-05-28T06:18:12.970 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.969+0000 7fbc37e00640 1 --2- 172.21.15.96:0/3113711535 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbc1806da60 0x7fbc1806ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:12.970 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.971+0000 7fbc37e00640 1 --2- 172.21.15.96:0/3113711535 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbc1806da60 0x7fbc1806ff20 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fbc28009910 tx=0x7fbc28008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:12.972 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:12.974+0000 7fbc2fe00640 1 -- 172.21.15.96:0/3113711535 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbc2405a430 con 0x7fbc3814c160 2024-05-28T06:18:13.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:13.210+0000 7fbc3ee00640 1 -- 172.21.15.96:0/3113711535 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fbc380b2000 con 0x7fbc3814c160 2024-05-28T06:18:13.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:13.685+0000 7fbc2fe00640 1 -- 172.21.15.96:0/3113711535 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v52) ==== 122+0+0 (secure 0 0 0) 0x7fbc24059d30 con 0x7fbc3814c160 2024-05-28T06:18:13.687 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:13.689+0000 7fbc3ee00640 1 -- 172.21.15.96:0/3113711535 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbc1806da60 msgr2=0x7fbc1806ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:13.687 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:13.689+0000 7fbc3ee00640 1 --2- 172.21.15.96:0/3113711535 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbc1806da60 0x7fbc1806ff20 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fbc28009910 tx=0x7fbc28008040 comp rx=0 tx=0).stop 2024-05-28T06:18:13.688 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:13.689+0000 7fbc3ee00640 1 -- 172.21.15.96:0/3113711535 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc3814c160 msgr2=0x7fbc3814c5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:13.688 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:13.689+0000 7fbc3ee00640 1 --2- 172.21.15.96:0/3113711535 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc3814c160 0x7fbc3814c5e0 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7fbc2400e830 tx=0x7fbc2400ecf0 comp rx=0 tx=0).stop 2024-05-28T06:18:13.688 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:13.689+0000 7fbc3ee00640 1 -- 172.21.15.96:0/3113711535 shutdown_connections 2024-05-28T06:18:13.688 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:13.689+0000 7fbc3ee00640 1 --2- 172.21.15.96:0/3113711535 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbc1806da60 0x7fbc1806ff20 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:13.688 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:13.689+0000 7fbc3ee00640 1 --2- 172.21.15.96:0/3113711535 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc380b1640 0x7fbc380b1ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:13.688 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:13.689+0000 7fbc3ee00640 1 --2- 172.21.15.96:0/3113711535 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc3814c160 0x7fbc3814c5e0 unknown :-1 s=CLOSED pgs=1402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:13.688 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:13.689+0000 7fbc3ee00640 1 --2- 172.21.15.96:0/3113711535 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbc1c0051c0 0x7fbc38147740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:13.688 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:13.689+0000 7fbc3ee00640 1 -- 172.21.15.96:0/3113711535 >> 172.21.15.96:0/3113711535 conn(0x7fbc3806bd20 msgr2=0x7fbc3815f3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:18:13.688 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:13.689+0000 7fbc3ee00640 1 -- 172.21.15.96:0/3113711535 shutdown_connections 2024-05-28T06:18:13.688 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:13.689+0000 7fbc3ee00640 1 -- 172.21.15.96:0/3113711535 wait complete. 2024-05-28T06:18:13.709 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-05-28T06:18:13.709 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-28T06:18:13.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:13.902+0000 7f04e7800640 1 Processor -- start 2024-05-28T06:18:13.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:13.902+0000 7f04e7800640 1 -- start start 2024-05-28T06:18:13.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:13.904+0000 7f04e7800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f04e00b4b60 0x7f04e00cfaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:13.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:13.904+0000 7f04e7800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f04e00b3b70 0x7f04e00d0900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:13.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:13.904+0000 7f04e7800640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04e00ced40 con 0x7f04e00d04a0 2024-05-28T06:18:13.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:13.904+0000 7f04e7800640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f04e00b3fe0 con 0x7f04e00b3b70 2024-05-28T06:18:13.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:13.904+0000 7f04e7800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f04e00d0e40 con 0x7f04e00b4b60 2024-05-28T06:18:13.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:13.904+0000 7f04d7e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f04e00d04a0 0x7f04e00b3460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:49920/0 (socket says 172.21.15.96:49920) 2024-05-28T06:18:13.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:13.904+0000 7f04d7e00640 1 -- 172.21.15.96:0/2698745686 learned_addr learned my addr 172.21.15.96:0/2698745686 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:18:13.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:13.904+0000 7f04e5800640 1 --2- 172.21.15.96:0/2698745686 >> v2:172.21.15.96:3300/0 conn(0x7f04e00b3b70 0x7f04e00d0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:13.904 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:13.904+0000 7f04e4e00640 1 --2- 172.21.15.96:0/2698745686 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f04e00b4b60 0x7f04e00cfaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:13.904 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:13.904+0000 7f04dfe00640 1 -- 172.21.15.96:0/2698745686 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4007347671 0 0) 0x7f04c8001110 con 0x7f04e00d04a0 2024-05-28T06:18:13.904 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:13.905+0000 7f04dfe00640 1 -- 172.21.15.96:0/2698745686 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f04c4003520 con 0x7f04e00d04a0 2024-05-28T06:18:13.904 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:13.905+0000 7f04e5800640 1 -- 172.21.15.96:0/2698745686 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f04e00b4b60 msgr2=0x7f04e00cfaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:18:13.904 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:13.905+0000 7f04e5800640 1 --2- 172.21.15.96:0/2698745686 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f04e00b4b60 0x7f04e00cfaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:13.904 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:13.905+0000 7f04e5800640 1 -- 172.21.15.96:0/2698745686 >> v1:172.21.15.96:6789/0 conn(0x7f04e00d04a0 legacy=0x7f04e00b3460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:13.904 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:13.905+0000 7f04e5800640 1 -- 172.21.15.96:0/2698745686 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04e015fbb0 con 0x7f04e00b3b70 2024-05-28T06:18:13.905 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:13.906+0000 7f04e5800640 1 --2- 172.21.15.96:0/2698745686 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f04e00b3b70 0x7f04e00d0900 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f04d8009d90 tx=0x7f04d802a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44fd298d7d72b8c4 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:13.905 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:13.906+0000 7f04dfe00640 1 -- 172.21.15.96:0/2698745686 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04d8007740 con 0x7f04e00b3b70 2024-05-28T06:18:13.905 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:13.906+0000 7f04dfe00640 1 -- 172.21.15.96:0/2698745686 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04d80078e0 con 0x7f04e00b3b70 2024-05-28T06:18:13.905 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:13.906+0000 7f04dfe00640 1 -- 172.21.15.96:0/2698745686 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04d8007cb0 con 0x7f04e00b3b70 2024-05-28T06:18:13.906 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:13.906+0000 7f04e7800640 1 -- 172.21.15.96:0/2698745686 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f04e00b3b70 msgr2=0x7f04e00d0900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:13.906 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:13.906+0000 7f04e7800640 1 --2- 172.21.15.96:0/2698745686 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f04e00b3b70 0x7f04e00d0900 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f04d8009d90 tx=0x7f04d802a660 comp rx=0 tx=0).stop 2024-05-28T06:18:13.906 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:13.907+0000 7f04e7800640 1 -- 172.21.15.96:0/2698745686 shutdown_connections 2024-05-28T06:18:13.906 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:13.907+0000 7f04e7800640 1 --2- 172.21.15.96:0/2698745686 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f04e00b3b70 0x7f04e00d0900 unknown :-1 s=CLOSED pgs=1404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:13.906 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:13.907+0000 7f04e7800640 1 --2- 172.21.15.96:0/2698745686 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f04e00b4b60 0x7f04e00cfaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:13.906 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:13.907+0000 7f04e7800640 1 -- 172.21.15.96:0/2698745686 >> 172.21.15.96:0/2698745686 conn(0x7f04e006bae0 msgr2=0x7f04e006bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:18:13.906 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:13.907+0000 7f04e7800640 1 -- 172.21.15.96:0/2698745686 shutdown_connections 2024-05-28T06:18:13.906 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:13.907+0000 7f04e7800640 1 -- 172.21.15.96:0/2698745686 wait complete. 2024-05-28T06:18:13.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:13.908+0000 7f04e7800640 1 Processor -- start 2024-05-28T06:18:13.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:13.908+0000 7f04e7800640 1 -- start start 2024-05-28T06:18:13.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:13.909+0000 7f04e7800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f04e00b3b70 0x7f04e014d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:13.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:13.909+0000 7f04e7800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f04e00b4b60 0x7f04e00c5b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:13.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:13.909+0000 7f04e7800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f04e00d04a0 0x7f04e00c60b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:13.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:13.909+0000 7f04e7800640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f04e00cf580 con 0x7f04e00d04a0 2024-05-28T06:18:13.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:13.909+0000 7f04e7800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f04e00cf2a0 con 0x7f04e00b4b60 2024-05-28T06:18:13.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:13.909+0000 7f04e7800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f04e00cffe0 con 0x7f04e00b3b70 2024-05-28T06:18:13.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:13.910+0000 7f04e5800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f04e00d04a0 0x7f04e00c60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:13.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:13.910+0000 7f04e5800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f04e00d04a0 0x7f04e00c60b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48214/0 (socket says 172.21.15.96:48214) 2024-05-28T06:18:13.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:13.910+0000 7f04e5800640 1 -- 172.21.15.96:0/603072316 learned_addr learned my addr 172.21.15.96:0/603072316 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:18:13.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:13.910+0000 7f04e5800640 1 -- 172.21.15.96:0/603072316 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f04e00b3b70 msgr2=0x7f04e014d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:13.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:13.910+0000 7f04e4e00640 1 --2- 172.21.15.96:0/603072316 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f04e00b3b70 0x7f04e014d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:13.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:13.910+0000 7f04d7e00640 1 --2- 172.21.15.96:0/603072316 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f04e00b4b60 0x7f04e00c5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:13.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:13.910+0000 7f04e5800640 1 --2- 172.21.15.96:0/603072316 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f04e00b3b70 0x7f04e014d590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:13.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:13.910+0000 7f04e5800640 1 -- 172.21.15.96:0/603072316 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f04e00b4b60 msgr2=0x7f04e00c5b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:13.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:13.910+0000 7f04e5800640 1 --2- 172.21.15.96:0/603072316 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f04e00b4b60 0x7f04e00c5b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:13.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:13.910+0000 7f04e5800640 1 -- 172.21.15.96:0/603072316 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04e015bc20 con 0x7f04e00d04a0 2024-05-28T06:18:13.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:13.910+0000 7f04d7e00640 1 --2- 172.21.15.96:0/603072316 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f04e00b4b60 0x7f04e00c5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:18:13.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:13.910+0000 7f04e5800640 1 --2- 172.21.15.96:0/603072316 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f04e00d04a0 0x7f04e00c60b0 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f04d8009eb0 tx=0x7f04d8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:13.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:13.911+0000 7f04dd600640 1 -- 172.21.15.96:0/603072316 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04d803f070 con 0x7f04e00d04a0 2024-05-28T06:18:13.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:13.911+0000 7f04dd600640 1 -- 172.21.15.96:0/603072316 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04d8031bc0 con 0x7f04e00d04a0 2024-05-28T06:18:13.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:13.911+0000 7f04dd600640 1 -- 172.21.15.96:0/603072316 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04d8009450 con 0x7f04e00d04a0 2024-05-28T06:18:13.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:13.911+0000 7f04e7800640 1 -- 172.21.15.96:0/603072316 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04e0073440 con 0x7f04e00d04a0 2024-05-28T06:18:13.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:13.911+0000 7f04e7800640 1 -- 172.21.15.96:0/603072316 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04e015c1a0 con 0x7f04e00d04a0 2024-05-28T06:18:13.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:13.912+0000 7f04e7800640 1 -- 172.21.15.96:0/603072316 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04e00705d0 con 0x7f04e00d04a0 2024-05-28T06:18:13.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:13.913+0000 7f04dd600640 1 -- 172.21.15.96:0/603072316 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 30) ==== 92621+0+0 (secure 0 0 0) 0x7f04d803b410 con 0x7f04e00d04a0 2024-05-28T06:18:13.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:13.914+0000 7f04dd600640 1 --2- 172.21.15.96:0/603072316 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f04bc06da60 0x7f04bc06ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:13.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:13.914+0000 7f04dd600640 1 -- 172.21.15.96:0/603072316 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 5737+0+0 (secure 0 0 0) 0x7f04d80abf50 con 0x7f04e00d04a0 2024-05-28T06:18:13.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:13.914+0000 7f04e4e00640 1 --2- 172.21.15.96:0/603072316 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f04bc06da60 0x7f04bc06ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:13.914 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:13.916+0000 7f04e4e00640 1 --2- 172.21.15.96:0/603072316 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f04bc06da60 0x7f04bc06ff20 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f04d0006fd0 tx=0x7f04d0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:13.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:13.918+0000 7f04dd600640 1 -- 172.21.15.96:0/603072316 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f04e00705d0 con 0x7f04e00d04a0 2024-05-28T06:18:14.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.163+0000 7f04e7800640 1 -- 172.21.15.96:0/603072316 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f04e00689e0 con 0x7f04e00d04a0 2024-05-28T06:18:14.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.164+0000 7f04dd600640 1 -- 172.21.15.96:0/603072316 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) ==== 94+0+5189 (secure 0 0 0) 0x7f04d807bdd0 con 0x7f04e00d04a0 2024-05-28T06:18:14.163 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:18:14.163 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":52,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5645,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6834/1355344409","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":1355344409},{"type":"v1","addr":"172.21.15.96:6834","nonce":1355344409}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":47},{"gid":5651,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/1967688533","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":1967688533},{"type":"v1","addr":"172.21.15.123:6835","nonce":1967688533}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48},{"gid":6169,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/3362018278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":3362018278},{"type":"v1","addr":"172.21.15.123:6837","nonce":3362018278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-28T06:18:10.069593+0000","modified":"2024-05-28T06:18:13.212203+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8103},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8103":{"gid":8103,"name":"c","rank":0,"incarnation":49,"state":"up:active","state_seq":3,"addr":"172.21.15.96:6835/1614743177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6833","nonce":1614743177},{"type":"v1","addr":"172.21.15.96:6835","nonce":1614743177}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8103,"qdb_cluster":[8103]},"id":6}]} 2024-05-28T06:18:14.163 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 52 2024-05-28T06:18:14.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.168+0000 7f04e7800640 1 -- 172.21.15.96:0/603072316 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f04bc06da60 msgr2=0x7f04bc06ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:14.166 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.168+0000 7f04e7800640 1 --2- 172.21.15.96:0/603072316 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f04bc06da60 0x7f04bc06ff20 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f04d0006fd0 tx=0x7f04d0008040 comp rx=0 tx=0).stop 2024-05-28T06:18:14.166 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.168+0000 7f04e7800640 1 -- 172.21.15.96:0/603072316 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f04e00d04a0 msgr2=0x7f04e00c60b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:14.166 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.168+0000 7f04e7800640 1 --2- 172.21.15.96:0/603072316 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f04e00d04a0 0x7f04e00c60b0 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f04d8009eb0 tx=0x7f04d8004a40 comp rx=0 tx=0).stop 2024-05-28T06:18:14.166 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.168+0000 7f04e7800640 1 -- 172.21.15.96:0/603072316 shutdown_connections 2024-05-28T06:18:14.166 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.168+0000 7f04e7800640 1 --2- 172.21.15.96:0/603072316 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f04bc06da60 0x7f04bc06ff20 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:14.166 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.168+0000 7f04e7800640 1 --2- 172.21.15.96:0/603072316 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f04e00d04a0 0x7f04e00c60b0 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:14.166 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.168+0000 7f04e7800640 1 --2- 172.21.15.96:0/603072316 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f04e00b4b60 0x7f04e00c5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:14.166 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.168+0000 7f04e7800640 1 --2- 172.21.15.96:0/603072316 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f04e00b3b70 0x7f04e014d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:14.166 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.168+0000 7f04e7800640 1 -- 172.21.15.96:0/603072316 >> 172.21.15.96:0/603072316 conn(0x7f04e006bae0 msgr2=0x7f04e00ae470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:18:14.166 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.168+0000 7f04e7800640 1 -- 172.21.15.96:0/603072316 shutdown_connections 2024-05-28T06:18:14.166 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.168+0000 7f04e7800640 1 -- 172.21.15.96:0/603072316 wait complete. 2024-05-28T06:18:14.185 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:18:14.383 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.384+0000 7f3d3b200640 1 Processor -- start 2024-05-28T06:18:14.383 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.385+0000 7f3d3b200640 1 -- start start 2024-05-28T06:18:14.384 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.387+0000 7f3d3b200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3d3c06ee40 0x7f3d3c0ad230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:14.384 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.387+0000 7f3d3b200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f3d3c06f2c0 0x7f3d3c0ad770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:14.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.387+0000 7f3d3b200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d3c156f10 con 0x7f3d3c156b30 2024-05-28T06:18:14.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.387+0000 7f3d3b200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f3d3c06dda0 con 0x7f3d3c06f2c0 2024-05-28T06:18:14.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.387+0000 7f3d3b200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f3d3c06e080 con 0x7f3d3c06ee40 2024-05-28T06:18:14.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.387+0000 7f3d39e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f3d3c156b30 0x7f3d3c0af6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:49932/0 (socket says 172.21.15.96:49932) 2024-05-28T06:18:14.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.387+0000 7f3d39e00640 1 -- 172.21.15.96:0/85747645 learned_addr learned my addr 172.21.15.96:0/85747645 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:18:14.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.387+0000 7f3d39400640 1 -- 172.21.15.96:0/85747645 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4228920638 0 0) 0x7f3d3c156f10 con 0x7f3d3c156b30 2024-05-28T06:18:14.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.387+0000 7f3d39400640 1 -- 172.21.15.96:0/85747645 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d20003520 con 0x7f3d3c156b30 2024-05-28T06:18:14.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.387+0000 7f3d33e00640 1 --2- 172.21.15.96:0/85747645 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3d3c06ee40 0x7f3d3c0ad230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:14.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.387+0000 7f3d3a800640 1 --2- 172.21.15.96:0/85747645 >> v2:172.21.15.123:3300/0 conn(0x7f3d3c06f2c0 0x7f3d3c0ad770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:14.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.387+0000 7f3d39400640 1 -- 172.21.15.96:0/85747645 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1579814984 0 0) 0x7f3d20003520 con 0x7f3d3c156b30 2024-05-28T06:18:14.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.387+0000 7f3d39400640 1 -- 172.21.15.96:0/85747645 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3d3c06ee40 msgr2=0x7f3d3c0ad230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:18:14.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.387+0000 7f3d39400640 1 --2- 172.21.15.96:0/85747645 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3d3c06ee40 0x7f3d3c0ad230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:14.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.387+0000 7f3d39400640 1 -- 172.21.15.96:0/85747645 >> v2:172.21.15.123:3300/0 conn(0x7f3d3c06f2c0 msgr2=0x7f3d3c0ad770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:18:14.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.387+0000 7f3d39400640 1 --2- 172.21.15.96:0/85747645 >> v2:172.21.15.123:3300/0 conn(0x7f3d3c06f2c0 0x7f3d3c0ad770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:14.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.387+0000 7f3d39400640 1 -- 172.21.15.96:0/85747645 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d3c06e360 con 0x7f3d3c156b30 2024-05-28T06:18:14.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.387+0000 7f3d39400640 1 -- 172.21.15.96:0/85747645 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f3d2c001670 con 0x7f3d3c156b30 2024-05-28T06:18:14.389 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.387+0000 7f3d3a800640 1 --2- 172.21.15.96:0/85747645 >> v2:172.21.15.123:3300/0 conn(0x7f3d3c06f2c0 0x7f3d3c0ad770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-28T06:18:14.389 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.387+0000 7f3d39400640 1 -- 172.21.15.96:0/85747645 >> v1:172.21.15.96:6789/0 conn(0x7f3d3c156b30 legacy=0x7f3d3c0af6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:14.389 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.387+0000 7f3d39400640 1 --2- 172.21.15.96:0/85747645 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d200039b0 0x7f3d20003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:14.389 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.387+0000 7f3d39400640 1 --2- 172.21.15.96:0/85747645 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3d3c06ee40 0x7f3d20004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:14.389 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.387+0000 7f3d39400640 1 --2- 172.21.15.96:0/85747645 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3d20004d40 0x7f3d200051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:14.389 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.387+0000 7f3d39400640 1 -- 172.21.15.96:0/85747645 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f3d3c06dda0 con 0x7f3d20004d40 2024-05-28T06:18:14.389 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.387+0000 7f3d39400640 1 -- 172.21.15.96:0/85747645 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f3d3c06e080 con 0x7f3d200039b0 2024-05-28T06:18:14.389 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.387+0000 7f3d39400640 1 -- 172.21.15.96:0/85747645 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f3d2000bb80 con 0x7f3d3c06ee40 2024-05-28T06:18:14.389 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.388+0000 7f3d3a800640 1 --2- 172.21.15.96:0/85747645 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3d20004d40 0x7f3d200051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:14.389 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.388+0000 7f3d39e00640 1 --2- 172.21.15.96:0/85747645 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d200039b0 0x7f3d20003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:14.389 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.388+0000 7f3d3a800640 1 -- 172.21.15.96:0/85747645 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3d3c06ee40 msgr2=0x7f3d20004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:18:14.389 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.388+0000 7f3d3a800640 1 --2- 172.21.15.96:0/85747645 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3d3c06ee40 0x7f3d20004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:14.390 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.388+0000 7f3d3a800640 1 -- 172.21.15.96:0/85747645 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d200039b0 msgr2=0x7f3d20003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:18:14.390 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.388+0000 7f3d3a800640 1 --2- 172.21.15.96:0/85747645 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d200039b0 0x7f3d20003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:14.390 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.388+0000 7f3d3a800640 1 -- 172.21.15.96:0/85747645 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d20003520 con 0x7f3d20004d40 2024-05-28T06:18:14.390 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.388+0000 7f3d39e00640 1 --2- 172.21.15.96:0/85747645 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d200039b0 0x7f3d20003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:18:14.390 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.389+0000 7f3d3a800640 1 --2- 172.21.15.96:0/85747645 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3d20004d40 0x7f3d200051c0 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f3d340073e0 tx=0x7f3d20005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19b00c7383430fe3 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:14.390 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.389+0000 7f3d39400640 1 -- 172.21.15.96:0/85747645 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d34009c80 con 0x7f3d20004d40 2024-05-28T06:18:14.390 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.389+0000 7f3d39400640 1 -- 172.21.15.96:0/85747645 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d3402c680 con 0x7f3d20004d40 2024-05-28T06:18:14.390 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.389+0000 7f3d39400640 1 -- 172.21.15.96:0/85747645 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d3402ca30 con 0x7f3d20004d40 2024-05-28T06:18:14.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.390+0000 7f3d3b200640 1 -- 172.21.15.96:0/85747645 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3d20004d40 msgr2=0x7f3d200051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:14.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.390+0000 7f3d3b200640 1 --2- 172.21.15.96:0/85747645 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3d20004d40 0x7f3d200051c0 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f3d340073e0 tx=0x7f3d20005830 comp rx=0 tx=0).stop 2024-05-28T06:18:14.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.390+0000 7f3d3a800640 1 -- 172.21.15.96:0/85747645 reap_dead start 2024-05-28T06:18:14.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.390+0000 7f3d3b200640 1 -- 172.21.15.96:0/85747645 shutdown_connections 2024-05-28T06:18:14.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.390+0000 7f3d3b200640 1 -- 172.21.15.96:0/85747645 >> 172.21.15.96:0/85747645 conn(0x7f3d3c0b1fc0 msgr2=0x7f3d3c0b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:18:14.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.390+0000 7f3d3b200640 1 -- 172.21.15.96:0/85747645 shutdown_connections 2024-05-28T06:18:14.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.390+0000 7f3d3b200640 1 -- 172.21.15.96:0/85747645 wait complete. 2024-05-28T06:18:14.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.392+0000 7f3d3b200640 1 Processor -- start 2024-05-28T06:18:14.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.392+0000 7f3d3b200640 1 -- start start 2024-05-28T06:18:14.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.393+0000 7f3d3b200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3d20004d40 0x7f3d3c0b0c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:14.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.393+0000 7f3d3b200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3d3c0af110 0x7f3d3c0af590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:14.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.393+0000 7f3d3b200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d3c0afad0 0x7f3d3c072560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:14.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.393+0000 7f3d3b200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f3d3c0b2db0 con 0x7f3d3c0af110 2024-05-28T06:18:14.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.393+0000 7f3d3b200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f3d3c0b2c30 con 0x7f3d3c0afad0 2024-05-28T06:18:14.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.393+0000 7f3d3b200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f3d3c0ad230 con 0x7f3d20004d40 2024-05-28T06:18:14.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.393+0000 7f3d33e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3d3c0af110 0x7f3d3c0af590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:14.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.393+0000 7f3d33e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3d3c0af110 0x7f3d3c0af590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48240/0 (socket says 172.21.15.96:48240) 2024-05-28T06:18:14.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.393+0000 7f3d33e00640 1 -- 172.21.15.96:0/4200975743 learned_addr learned my addr 172.21.15.96:0/4200975743 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:18:14.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.394+0000 7f3d33e00640 1 -- 172.21.15.96:0/4200975743 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3d20004d40 msgr2=0x7f3d3c0b0c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:14.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.394+0000 7f3d33e00640 1 --2- 172.21.15.96:0/4200975743 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3d20004d40 0x7f3d3c0b0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:14.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.394+0000 7f3d33e00640 1 -- 172.21.15.96:0/4200975743 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d3c0afad0 msgr2=0x7f3d3c072560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:14.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.394+0000 7f3d33e00640 1 --2- 172.21.15.96:0/4200975743 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d3c0afad0 0x7f3d3c072560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:14.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.394+0000 7f3d33e00640 1 -- 172.21.15.96:0/4200975743 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d3c0aff50 con 0x7f3d3c0af110 2024-05-28T06:18:14.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.394+0000 7f3d33e00640 1 --2- 172.21.15.96:0/4200975743 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3d3c0af110 0x7f3d3c0af590 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f3d2400d830 tx=0x7f3d2400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:14.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.394+0000 7f3d32000640 1 -- 172.21.15.96:0/4200975743 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d24018020 con 0x7f3d3c0af110 2024-05-28T06:18:14.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.394+0000 7f3d32000640 1 -- 172.21.15.96:0/4200975743 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d2400b5e0 con 0x7f3d3c0af110 2024-05-28T06:18:14.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.394+0000 7f3d32000640 1 -- 172.21.15.96:0/4200975743 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d2400b9b0 con 0x7f3d3c0af110 2024-05-28T06:18:14.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.394+0000 7f3d3b200640 1 -- 172.21.15.96:0/4200975743 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d3c14e160 con 0x7f3d3c0af110 2024-05-28T06:18:14.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.394+0000 7f3d3b200640 1 -- 172.21.15.96:0/4200975743 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d3c071d60 con 0x7f3d3c0af110 2024-05-28T06:18:14.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.395+0000 7f3d3b200640 1 -- 172.21.15.96:0/4200975743 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d3c0689e0 con 0x7f3d3c0af110 2024-05-28T06:18:14.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.397+0000 7f3d32000640 1 -- 172.21.15.96:0/4200975743 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 30) ==== 92621+0+0 (secure 0 0 0) 0x7f3d2401b020 con 0x7f3d3c0af110 2024-05-28T06:18:14.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.398+0000 7f3d32000640 1 --2- 172.21.15.96:0/4200975743 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3d1806da60 0x7f3d1806ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:14.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.398+0000 7f3d32000640 1 -- 172.21.15.96:0/4200975743 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 5737+0+0 (secure 0 0 0) 0x7f3d2408aec0 con 0x7f3d3c0af110 2024-05-28T06:18:14.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.398+0000 7f3d39e00640 1 --2- 172.21.15.96:0/4200975743 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3d1806da60 0x7f3d1806ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:14.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.400+0000 7f3d39e00640 1 --2- 172.21.15.96:0/4200975743 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3d1806da60 0x7f3d1806ff20 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f3d2c001eb0 tx=0x7f3d2c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:14.400 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.402+0000 7f3d32000640 1 -- 172.21.15.96:0/4200975743 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3d24022030 con 0x7f3d3c0af110 2024-05-28T06:18:14.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.605+0000 7f3d3b200640 1 -- 172.21.15.96:0/4200975743 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3d3c06f7f0 con 0x7f3d3c0af110 2024-05-28T06:18:14.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.607+0000 7f3d32000640 1 -- 172.21.15.96:0/4200975743 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+16715 (secure 0 0 0) 0x7f3d2405a9f0 con 0x7f3d3c0af110 2024-05-28T06:18:14.605 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:18:14.606 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":89,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:18:11.659770+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-05-28T06:18:07.815707+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-05-28T06:18:09.061362+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:18:14.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.610+0000 7f3d3b200640 1 -- 172.21.15.96:0/4200975743 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3d1806da60 msgr2=0x7f3d1806ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:14.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.610+0000 7f3d3b200640 1 --2- 172.21.15.96:0/4200975743 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3d1806da60 0x7f3d1806ff20 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f3d2c001eb0 tx=0x7f3d2c009040 comp rx=0 tx=0).stop 2024-05-28T06:18:14.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.610+0000 7f3d3b200640 1 -- 172.21.15.96:0/4200975743 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3d3c0af110 msgr2=0x7f3d3c0af590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:14.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.610+0000 7f3d3b200640 1 --2- 172.21.15.96:0/4200975743 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3d3c0af110 0x7f3d3c0af590 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f3d2400d830 tx=0x7f3d2400dcf0 comp rx=0 tx=0).stop 2024-05-28T06:18:14.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.610+0000 7f3d3b200640 1 -- 172.21.15.96:0/4200975743 shutdown_connections 2024-05-28T06:18:14.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.610+0000 7f3d3b200640 1 --2- 172.21.15.96:0/4200975743 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3d1806da60 0x7f3d1806ff20 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:14.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.610+0000 7f3d3b200640 1 --2- 172.21.15.96:0/4200975743 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d3c0afad0 0x7f3d3c072560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:14.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.610+0000 7f3d3b200640 1 --2- 172.21.15.96:0/4200975743 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3d3c0af110 0x7f3d3c0af590 unknown :-1 s=CLOSED pgs=1408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:14.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.610+0000 7f3d3b200640 1 --2- 172.21.15.96:0/4200975743 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3d20004d40 0x7f3d3c0b0c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:14.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.610+0000 7f3d3b200640 1 -- 172.21.15.96:0/4200975743 >> 172.21.15.96:0/4200975743 conn(0x7f3d3c0b1fc0 msgr2=0x7f3d3c155410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:18:14.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.610+0000 7f3d3b200640 1 -- 172.21.15.96:0/4200975743 shutdown_connections 2024-05-28T06:18:14.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.610+0000 7f3d3b200640 1 -- 172.21.15.96:0/4200975743 wait complete. 2024-05-28T06:18:14.627 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-05-28T06:18:14.627 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-28T06:18:14.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.841+0000 7f0d9b400640 1 Processor -- start 2024-05-28T06:18:14.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.841+0000 7f0d9b400640 1 -- start start 2024-05-28T06:18:14.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.842+0000 7f0d9b400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f0d940b4b60 0x7f0d940cfaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:14.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.842+0000 7f0d9b400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f0d940d04a0 0x7f0d940d0930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:14.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.842+0000 7f0d9b400640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d940ced40 con 0x7f0d940b3460 2024-05-28T06:18:14.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.842+0000 7f0d9b400640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f0d940b3f90 con 0x7f0d940b4b60 2024-05-28T06:18:14.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.842+0000 7f0d9b400640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f0d9415fbb0 con 0x7f0d940d04a0 2024-05-28T06:18:14.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.843+0000 7f0d98a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f0d940b4b60 0x7f0d940cfaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:14.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.843+0000 7f0d99400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f0d940b3460 0x7f0d940b3880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:49938/0 (socket says 172.21.15.96:49938) 2024-05-28T06:18:14.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.843+0000 7f0d99400640 1 -- 172.21.15.96:0/2794022202 learned_addr learned my addr 172.21.15.96:0/2794022202 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:18:14.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.843+0000 7f0d8be00640 1 --2- 172.21.15.96:0/2794022202 >> v2:172.21.15.123:3300/0 conn(0x7f0d940d04a0 0x7f0d940d0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:14.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.843+0000 7f0d98a00640 1 -- 172.21.15.96:0/2794022202 >> v2:172.21.15.123:3300/0 conn(0x7f0d940d04a0 msgr2=0x7f0d940d0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:18:14.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.843+0000 7f0d98a00640 1 --2- 172.21.15.96:0/2794022202 >> v2:172.21.15.123:3300/0 conn(0x7f0d940d04a0 0x7f0d940d0930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:14.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.843+0000 7f0d98a00640 1 -- 172.21.15.96:0/2794022202 >> v1:172.21.15.96:6789/0 conn(0x7f0d940b3460 legacy=0x7f0d940b3880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:14.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.843+0000 7f0d98a00640 1 -- 172.21.15.96:0/2794022202 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d940cefc0 con 0x7f0d940b4b60 2024-05-28T06:18:14.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.843+0000 7f0d93e00640 1 -- 172.21.15.96:0/2794022202 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3589328174 0 0) 0x7f0d8c001490 con 0x7f0d940b3460 2024-05-28T06:18:14.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.844+0000 7f0d98a00640 1 --2- 172.21.15.96:0/2794022202 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0d940b4b60 0x7f0d940cfaa0 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7f0d940d1180 tx=0x7f0d8402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f639ca76bf533a1a server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:14.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.844+0000 7f0d93e00640 1 -- 172.21.15.96:0/2794022202 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d84037070 con 0x7f0d940b4b60 2024-05-28T06:18:14.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.844+0000 7f0d93e00640 1 -- 172.21.15.96:0/2794022202 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d84032440 con 0x7f0d940b4b60 2024-05-28T06:18:14.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.844+0000 7f0d93e00640 1 -- 172.21.15.96:0/2794022202 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d84032810 con 0x7f0d940b4b60 2024-05-28T06:18:14.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.844+0000 7f0d9b400640 1 -- 172.21.15.96:0/2794022202 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0d940b4b60 msgr2=0x7f0d940cfaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:14.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.844+0000 7f0d9b400640 1 --2- 172.21.15.96:0/2794022202 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0d940b4b60 0x7f0d940cfaa0 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7f0d940d1180 tx=0x7f0d8402a1a0 comp rx=0 tx=0).stop 2024-05-28T06:18:14.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.846+0000 7f0d9b400640 1 -- 172.21.15.96:0/2794022202 shutdown_connections 2024-05-28T06:18:14.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.846+0000 7f0d9b400640 1 --2- 172.21.15.96:0/2794022202 >> v2:172.21.15.123:3300/0 conn(0x7f0d940d04a0 0x7f0d940d0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:14.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.846+0000 7f0d9b400640 1 --2- 172.21.15.96:0/2794022202 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0d940b4b60 0x7f0d940cfaa0 unknown :-1 s=CLOSED pgs=1410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:14.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.846+0000 7f0d9b400640 1 -- 172.21.15.96:0/2794022202 >> 172.21.15.96:0/2794022202 conn(0x7f0d9406bae0 msgr2=0x7f0d9406bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:18:14.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.846+0000 7f0d9b400640 1 -- 172.21.15.96:0/2794022202 shutdown_connections 2024-05-28T06:18:14.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.846+0000 7f0d9b400640 1 -- 172.21.15.96:0/2794022202 wait complete. 2024-05-28T06:18:14.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.847+0000 7f0d9b400640 1 Processor -- start 2024-05-28T06:18:14.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.848+0000 7f0d9b400640 1 -- start start 2024-05-28T06:18:14.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.848+0000 7f0d9b400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0d940b3460 0x7f0d940c40b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:14.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.848+0000 7f0d9b400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0d940b4b60 0x7f0d9415d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:14.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.848+0000 7f0d9b400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d940d04a0 0x7f0d940ad340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:14.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.848+0000 7f0d9b400640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f0d940cffe0 con 0x7f0d940b4b60 2024-05-28T06:18:14.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.848+0000 7f0d9b400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f0d940cf580 con 0x7f0d940d04a0 2024-05-28T06:18:14.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.848+0000 7f0d9b400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f0d9415ede0 con 0x7f0d940b3460 2024-05-28T06:18:14.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.849+0000 7f0d8be00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0d940b4b60 0x7f0d9415d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:14.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.849+0000 7f0d8be00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0d940b4b60 0x7f0d9415d3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48266/0 (socket says 172.21.15.96:48266) 2024-05-28T06:18:14.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.849+0000 7f0d8be00640 1 -- 172.21.15.96:0/300357643 learned_addr learned my addr 172.21.15.96:0/300357643 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:18:14.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.849+0000 7f0d99400640 1 --2- 172.21.15.96:0/300357643 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d940d04a0 0x7f0d940ad340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:14.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.849+0000 7f0d98a00640 1 --2- 172.21.15.96:0/300357643 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0d940b3460 0x7f0d940c40b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:14.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.849+0000 7f0d8be00640 1 -- 172.21.15.96:0/300357643 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0d940b3460 msgr2=0x7f0d940c40b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:14.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.849+0000 7f0d8be00640 1 --2- 172.21.15.96:0/300357643 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0d940b3460 0x7f0d940c40b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:14.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.849+0000 7f0d8be00640 1 -- 172.21.15.96:0/300357643 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d940d04a0 msgr2=0x7f0d940ad340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:14.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.849+0000 7f0d8be00640 1 --2- 172.21.15.96:0/300357643 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d940d04a0 0x7f0d940ad340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:14.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.849+0000 7f0d8be00640 1 -- 172.21.15.96:0/300357643 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d940adb90 con 0x7f0d940b4b60 2024-05-28T06:18:14.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.849+0000 7f0d99400640 1 --2- 172.21.15.96:0/300357643 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d940d04a0 0x7f0d940ad340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:18:14.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.850+0000 7f0d8be00640 1 --2- 172.21.15.96:0/300357643 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0d940b4b60 0x7f0d9415d3f0 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7f0d7c00d620 tx=0x7f0d7c00dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:14.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.850+0000 7f0d91600640 1 -- 172.21.15.96:0/300357643 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d7c00b620 con 0x7f0d940b4b60 2024-05-28T06:18:14.849 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.850+0000 7f0d91600640 1 -- 172.21.15.96:0/300357643 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d7c019070 con 0x7f0d940b4b60 2024-05-28T06:18:14.849 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.850+0000 7f0d91600640 1 -- 172.21.15.96:0/300357643 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d7c00ba00 con 0x7f0d940b4b60 2024-05-28T06:18:14.849 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.850+0000 7f0d9b400640 1 -- 172.21.15.96:0/300357643 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d940ade70 con 0x7f0d940b4b60 2024-05-28T06:18:14.849 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.850+0000 7f0d9b400640 1 -- 172.21.15.96:0/300357643 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d940ae430 con 0x7f0d940b4b60 2024-05-28T06:18:14.849 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.851+0000 7f0d9b400640 1 -- 172.21.15.96:0/300357643 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d940689e0 con 0x7f0d940b4b60 2024-05-28T06:18:14.849 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.852+0000 7f0d91600640 1 -- 172.21.15.96:0/300357643 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 30) ==== 92621+0+0 (secure 0 0 0) 0x7f0d7c00bba0 con 0x7f0d940b4b60 2024-05-28T06:18:14.850 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.853+0000 7f0d91600640 1 --2- 172.21.15.96:0/300357643 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0d7006da60 0x7f0d7006ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:14.850 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.854+0000 7f0d91600640 1 -- 172.21.15.96:0/300357643 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 5737+0+0 (secure 0 0 0) 0x7f0d7c08ae70 con 0x7f0d940b4b60 2024-05-28T06:18:14.851 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.854+0000 7f0d98a00640 1 --2- 172.21.15.96:0/300357643 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0d7006da60 0x7f0d7006ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:14.852 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.855+0000 7f0d98a00640 1 --2- 172.21.15.96:0/300357643 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0d7006da60 0x7f0d7006ff20 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f0d84034000 tx=0x7f0d8402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:14.855 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:14.858+0000 7f0d91600640 1 -- 172.21.15.96:0/300357643 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0d7c05a9a0 con 0x7f0d940b4b60 2024-05-28T06:18:15.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.049+0000 7f0d9b400640 1 -- 172.21.15.96:0/300357643 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0d9415f340 con 0x7f0d7006da60 2024-05-28T06:18:15.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.069+0000 7f0d91600640 1 -- 172.21.15.96:0/300357643 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+247514 (secure 0 0 0) 0x7f0d9415f340 con 0x7f0d7006da60 2024-05-28T06:18:15.069 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:18:15.072 INFO:teuthology.orchestra.run.smithi096.stderr:dumped all 2024-05-28T06:18:15.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.075+0000 7f0d9b400640 1 -- 172.21.15.96:0/300357643 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0d7006da60 msgr2=0x7f0d7006ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:15.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.075+0000 7f0d9b400640 1 --2- 172.21.15.96:0/300357643 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0d7006da60 0x7f0d7006ff20 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f0d84034000 tx=0x7f0d8402e000 comp rx=0 tx=0).stop 2024-05-28T06:18:15.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.075+0000 7f0d9b400640 1 -- 172.21.15.96:0/300357643 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0d940b4b60 msgr2=0x7f0d9415d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:15.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.075+0000 7f0d9b400640 1 --2- 172.21.15.96:0/300357643 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0d940b4b60 0x7f0d9415d3f0 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7f0d7c00d620 tx=0x7f0d7c00dae0 comp rx=0 tx=0).stop 2024-05-28T06:18:15.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.075+0000 7f0d9b400640 1 -- 172.21.15.96:0/300357643 shutdown_connections 2024-05-28T06:18:15.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.076+0000 7f0d9b400640 1 --2- 172.21.15.96:0/300357643 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0d7006da60 0x7f0d7006ff20 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:15.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.076+0000 7f0d9b400640 1 --2- 172.21.15.96:0/300357643 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d940d04a0 0x7f0d940ad340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:15.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.076+0000 7f0d9b400640 1 --2- 172.21.15.96:0/300357643 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0d940b4b60 0x7f0d9415d3f0 unknown :-1 s=CLOSED pgs=1411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:15.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.076+0000 7f0d9b400640 1 --2- 172.21.15.96:0/300357643 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0d940b3460 0x7f0d940c40b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:15.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.076+0000 7f0d9b400640 1 -- 172.21.15.96:0/300357643 >> 172.21.15.96:0/300357643 conn(0x7f0d9406bae0 msgr2=0x7f0d9406f4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:18:15.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.076+0000 7f0d9b400640 1 -- 172.21.15.96:0/300357643 shutdown_connections 2024-05-28T06:18:15.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.076+0000 7f0d9b400640 1 -- 172.21.15.96:0/300357643 wait complete. 2024-05-28T06:18:15.092 INFO:teuthology.orchestra.run.smithi096.stdout:{"pg_ready":true,"pg_map":{"version":509,"stamp":"2024-05-28T06:18:13.862529+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590438,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":160,"num_write_kb":2417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118,"ondisk_log_size":118,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":35,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":812332,"kb_used_data":9456,"kb_used_omap":12,"kb_used_meta":802739,"kb_avail":754162388,"statfs":{"total":773094113280,"available":772262285312,"internally_reserved":0,"allocated":9682944,"data_stored":4833559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":822005296},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.253304"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.635935+0000","last_change":"2024-05-28T06:18:09.635935+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.635935+0000","last_undegraded":"2024-05-28T06:18:09.635935+0000","last_fullsized":"2024-05-28T06:18:09.635935+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.639835+0000","last_change":"2024-05-28T06:18:09.639835+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.639835+0000","last_undegraded":"2024-05-28T06:18:09.639835+0000","last_fullsized":"2024-05-28T06:18:09.639835+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-28T06:18:10.688685+0000","last_change":"2024-05-28T06:18:10.684086+0000","last_active":"2024-05-28T06:18:10.688685+0000","last_peered":"2024-05-28T06:18:10.688685+0000","last_clean":"2024-05-28T06:18:10.688685+0000","last_became_active":"2024-05-28T06:18:10.683345+0000","last_became_peered":"2024-05-28T06:18:10.683345+0000","last_unstale":"2024-05-28T06:18:10.688685+0000","last_undegraded":"2024-05-28T06:18:10.688685+0000","last_fullsized":"2024-05-28T06:18:10.688685+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:36:04.760890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.629536+0000","last_change":"2024-05-28T06:18:09.629536+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.629536+0000","last_undegraded":"2024-05-28T06:18:09.629536+0000","last_fullsized":"2024-05-28T06:18:09.629536+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.637122+0000","last_change":"2024-05-28T06:18:09.637122+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.637122+0000","last_undegraded":"2024-05-28T06:18:09.637122+0000","last_fullsized":"2024-05-28T06:18:09.637122+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.631959+0000","last_change":"2024-05-28T06:18:09.631959+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.631959+0000","last_undegraded":"2024-05-28T06:18:09.631959+0000","last_fullsized":"2024-05-28T06:18:09.631959+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.628109+0000","last_change":"2024-05-28T06:18:09.628109+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.628109+0000","last_undegraded":"2024-05-28T06:18:09.628109+0000","last_fullsized":"2024-05-28T06:18:09.628109+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.628197+0000","last_change":"2024-05-28T06:18:09.628197+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.628197+0000","last_undegraded":"2024-05-28T06:18:09.628197+0000","last_fullsized":"2024-05-28T06:18:09.628197+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.633228+0000","last_change":"2024-05-28T06:18:09.633228+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.633228+0000","last_undegraded":"2024-05-28T06:18:09.633228+0000","last_fullsized":"2024-05-28T06:18:09.633228+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.626053+0000","last_change":"2024-05-28T06:18:09.626053+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.626053+0000","last_undegraded":"2024-05-28T06:18:09.626053+0000","last_fullsized":"2024-05-28T06:18:09.626053+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.640699+0000","last_change":"2024-05-28T06:18:09.640699+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.640699+0000","last_undegraded":"2024-05-28T06:18:09.640699+0000","last_fullsized":"2024-05-28T06:18:09.640699+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.627727+0000","last_change":"2024-05-28T06:18:09.627727+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.627727+0000","last_undegraded":"2024-05-28T06:18:09.627727+0000","last_fullsized":"2024-05-28T06:18:09.627727+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-28T06:18:10.676482+0000","last_change":"2024-05-28T06:18:10.673510+0000","last_active":"2024-05-28T06:18:10.676482+0000","last_peered":"2024-05-28T06:18:10.676482+0000","last_clean":"2024-05-28T06:18:10.676482+0000","last_became_active":"2024-05-28T06:18:10.673194+0000","last_became_peered":"2024-05-28T06:18:10.673194+0000","last_unstale":"2024-05-28T06:18:10.676482+0000","last_undegraded":"2024-05-28T06:18:10.676482+0000","last_fullsized":"2024-05-28T06:18:10.676482+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:10:10.001167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.626455+0000","last_change":"2024-05-28T06:18:09.626455+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.626455+0000","last_undegraded":"2024-05-28T06:18:09.626455+0000","last_fullsized":"2024-05-28T06:18:09.626455+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.628788+0000","last_change":"2024-05-28T06:18:09.628788+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.628788+0000","last_undegraded":"2024-05-28T06:18:09.628788+0000","last_fullsized":"2024-05-28T06:18:09.628788+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.626723+0000","last_change":"2024-05-28T06:18:09.626723+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.626723+0000","last_undegraded":"2024-05-28T06:18:09.626723+0000","last_fullsized":"2024-05-28T06:18:09.626723+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.635456+0000","last_change":"2024-05-28T06:18:09.635456+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.635456+0000","last_undegraded":"2024-05-28T06:18:09.635456+0000","last_fullsized":"2024-05-28T06:18:09.635456+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.626844+0000","last_change":"2024-05-28T06:18:09.626844+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.626844+0000","last_undegraded":"2024-05-28T06:18:09.626844+0000","last_fullsized":"2024-05-28T06:18:09.626844+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.638675+0000","last_change":"2024-05-28T06:18:09.638675+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.638675+0000","last_undegraded":"2024-05-28T06:18:09.638675+0000","last_fullsized":"2024-05-28T06:18:09.638675+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.628326+0000","last_change":"2024-05-28T06:18:09.628326+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.628326+0000","last_undegraded":"2024-05-28T06:18:09.628326+0000","last_fullsized":"2024-05-28T06:18:09.628326+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.626431+0000","last_change":"2024-05-28T06:18:09.626431+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.626431+0000","last_undegraded":"2024-05-28T06:18:09.626431+0000","last_fullsized":"2024-05-28T06:18:09.626431+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.632230+0000","last_change":"2024-05-28T06:18:09.632230+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.632230+0000","last_undegraded":"2024-05-28T06:18:09.632230+0000","last_fullsized":"2024-05-28T06:18:09.632230+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.634278+0000","last_change":"2024-05-28T06:18:09.634278+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.634278+0000","last_undegraded":"2024-05-28T06:18:09.634278+0000","last_fullsized":"2024-05-28T06:18:09.634278+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.625935+0000","last_change":"2024-05-28T06:18:09.625935+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.625935+0000","last_undegraded":"2024-05-28T06:18:09.625935+0000","last_fullsized":"2024-05-28T06:18:09.625935+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.637179+0000","last_change":"2024-05-28T06:18:09.637179+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.637179+0000","last_undegraded":"2024-05-28T06:18:09.637179+0000","last_fullsized":"2024-05-28T06:18:09.637179+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-28T06:18:10.688666+0000","last_change":"2024-05-28T06:18:10.682743+0000","last_active":"2024-05-28T06:18:10.688666+0000","last_peered":"2024-05-28T06:18:10.688666+0000","last_clean":"2024-05-28T06:18:10.688666+0000","last_became_active":"2024-05-28T06:18:10.682451+0000","last_became_peered":"2024-05-28T06:18:10.682451+0000","last_unstale":"2024-05-28T06:18:10.688666+0000","last_undegraded":"2024-05-28T06:18:10.688666+0000","last_fullsized":"2024-05-28T06:18:10.688666+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:38:26.397686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.636728+0000","last_change":"2024-05-28T06:18:09.636728+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.636728+0000","last_undegraded":"2024-05-28T06:18:09.636728+0000","last_fullsized":"2024-05-28T06:18:09.636728+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.627185+0000","last_change":"2024-05-28T06:18:09.627185+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.627185+0000","last_undegraded":"2024-05-28T06:18:09.627185+0000","last_fullsized":"2024-05-28T06:18:09.627185+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.632024+0000","last_change":"2024-05-28T06:18:09.632024+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.632024+0000","last_undegraded":"2024-05-28T06:18:09.632024+0000","last_fullsized":"2024-05-28T06:18:09.632024+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.631106+0000","last_change":"2024-05-28T06:18:09.631106+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.631106+0000","last_undegraded":"2024-05-28T06:18:09.631106+0000","last_fullsized":"2024-05-28T06:18:09.631106+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.626336+0000","last_change":"2024-05-28T06:18:09.626336+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.626336+0000","last_undegraded":"2024-05-28T06:18:09.626336+0000","last_fullsized":"2024-05-28T06:18:09.626336+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.626394+0000","last_change":"2024-05-28T06:18:09.626394+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.626394+0000","last_undegraded":"2024-05-28T06:18:09.626394+0000","last_fullsized":"2024-05-28T06:18:09.626394+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.636228+0000","last_change":"2024-05-28T06:18:09.636228+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.636228+0000","last_undegraded":"2024-05-28T06:18:09.636228+0000","last_fullsized":"2024-05-28T06:18:09.636228+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-28T06:18:09.645386+0000","last_change":"2024-05-28T06:18:09.643542+0000","last_active":"2024-05-28T06:18:09.645386+0000","last_peered":"2024-05-28T06:18:09.645386+0000","last_clean":"2024-05-28T06:18:09.645386+0000","last_became_active":"2024-05-28T06:18:09.641808+0000","last_became_peered":"2024-05-28T06:18:09.641808+0000","last_unstale":"2024-05-28T06:18:09.645386+0000","last_undegraded":"2024-05-28T06:18:09.645386+0000","last_fullsized":"2024-05-28T06:18:09.645386+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:24:16.141801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.635380+0000","last_change":"2024-05-28T06:18:09.635380+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.635380+0000","last_undegraded":"2024-05-28T06:18:09.635380+0000","last_fullsized":"2024-05-28T06:18:09.635380+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-28T06:18:09.648769+0000","last_change":"2024-05-28T06:18:09.647214+0000","last_active":"2024-05-28T06:18:09.648769+0000","last_peered":"2024-05-28T06:18:09.648769+0000","last_clean":"2024-05-28T06:18:09.648769+0000","last_became_active":"2024-05-28T06:18:09.646937+0000","last_became_peered":"2024-05-28T06:18:09.646937+0000","last_unstale":"2024-05-28T06:18:09.648769+0000","last_undegraded":"2024-05-28T06:18:09.648769+0000","last_fullsized":"2024-05-28T06:18:09.648769+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:32:35.049306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.626694+0000","last_change":"2024-05-28T06:18:09.626694+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.626694+0000","last_undegraded":"2024-05-28T06:18:09.626694+0000","last_fullsized":"2024-05-28T06:18:09.626694+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-28T06:18:09.650076+0000","last_change":"2024-05-28T06:18:09.646080+0000","last_active":"2024-05-28T06:18:09.650076+0000","last_peered":"2024-05-28T06:18:09.650076+0000","last_clean":"2024-05-28T06:18:09.650076+0000","last_became_active":"2024-05-28T06:18:09.645807+0000","last_became_peered":"2024-05-28T06:18:09.645807+0000","last_unstale":"2024-05-28T06:18:09.650076+0000","last_undegraded":"2024-05-28T06:18:09.650076+0000","last_fullsized":"2024-05-28T06:18:09.650076+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:50:32.547130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-28T06:18:10.680587+0000","last_change":"2024-05-28T06:18:10.677395+0000","last_active":"2024-05-28T06:18:10.680587+0000","last_peered":"2024-05-28T06:18:10.680587+0000","last_clean":"2024-05-28T06:18:10.680587+0000","last_became_active":"2024-05-28T06:18:10.677100+0000","last_became_peered":"2024-05-28T06:18:10.677100+0000","last_unstale":"2024-05-28T06:18:10.680587+0000","last_undegraded":"2024-05-28T06:18:10.680587+0000","last_fullsized":"2024-05-28T06:18:10.680587+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:35:48.065461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-28T06:18:09.648814+0000","last_change":"2024-05-28T06:18:09.643971+0000","last_active":"2024-05-28T06:18:09.648814+0000","last_peered":"2024-05-28T06:18:09.648814+0000","last_clean":"2024-05-28T06:18:09.648814+0000","last_became_active":"2024-05-28T06:18:09.643691+0000","last_became_peered":"2024-05-28T06:18:09.643691+0000","last_unstale":"2024-05-28T06:18:09.648814+0000","last_undegraded":"2024-05-28T06:18:09.648814+0000","last_fullsized":"2024-05-28T06:18:09.648814+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:48:38.824030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.630680+0000","last_change":"2024-05-28T06:18:09.630680+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.630680+0000","last_undegraded":"2024-05-28T06:18:09.630680+0000","last_fullsized":"2024-05-28T06:18:09.630680+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-28T06:18:09.647612+0000","last_change":"2024-05-28T06:18:09.645873+0000","last_active":"2024-05-28T06:18:09.647612+0000","last_peered":"2024-05-28T06:18:09.647612+0000","last_clean":"2024-05-28T06:18:09.647612+0000","last_became_active":"2024-05-28T06:18:09.645561+0000","last_became_peered":"2024-05-28T06:18:09.645561+0000","last_unstale":"2024-05-28T06:18:09.647612+0000","last_undegraded":"2024-05-28T06:18:09.647612+0000","last_fullsized":"2024-05-28T06:18:09.647612+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:50:17.041537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.627460+0000","last_change":"2024-05-28T06:18:09.627460+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.627460+0000","last_undegraded":"2024-05-28T06:18:09.627460+0000","last_fullsized":"2024-05-28T06:18:09.627460+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-28T06:18:09.644497+0000","last_change":"2024-05-28T06:18:09.641147+0000","last_active":"2024-05-28T06:18:09.644497+0000","last_peered":"2024-05-28T06:18:09.644497+0000","last_clean":"2024-05-28T06:18:09.644497+0000","last_became_active":"2024-05-28T06:18:09.640830+0000","last_became_peered":"2024-05-28T06:18:09.640830+0000","last_unstale":"2024-05-28T06:18:09.644497+0000","last_undegraded":"2024-05-28T06:18:09.644497+0000","last_fullsized":"2024-05-28T06:18:09.644497+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:46:24.986403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-28T06:18:10.688719+0000","last_change":"2024-05-28T06:18:10.684510+0000","last_active":"2024-05-28T06:18:10.688719+0000","last_peered":"2024-05-28T06:18:10.688719+0000","last_clean":"2024-05-28T06:18:10.688719+0000","last_became_active":"2024-05-28T06:18:10.684209+0000","last_became_peered":"2024-05-28T06:18:10.684209+0000","last_unstale":"2024-05-28T06:18:10.688719+0000","last_undegraded":"2024-05-28T06:18:10.688719+0000","last_fullsized":"2024-05-28T06:18:10.688719+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T18:13:08.105068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-28T06:18:09.651440+0000","last_change":"2024-05-28T06:18:09.647844+0000","last_active":"2024-05-28T06:18:09.651440+0000","last_peered":"2024-05-28T06:18:09.651440+0000","last_clean":"2024-05-28T06:18:09.651440+0000","last_became_active":"2024-05-28T06:18:09.647527+0000","last_became_peered":"2024-05-28T06:18:09.647527+0000","last_unstale":"2024-05-28T06:18:09.651440+0000","last_undegraded":"2024-05-28T06:18:09.651440+0000","last_fullsized":"2024-05-28T06:18:09.651440+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T18:02:58.935614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.626057+0000","last_change":"2024-05-28T06:18:09.626057+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.626057+0000","last_undegraded":"2024-05-28T06:18:09.626057+0000","last_fullsized":"2024-05-28T06:18:09.626057+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-28T06:18:09.649566+0000","last_change":"2024-05-28T06:18:09.648064+0000","last_active":"2024-05-28T06:18:09.649566+0000","last_peered":"2024-05-28T06:18:09.649566+0000","last_clean":"2024-05-28T06:18:09.649566+0000","last_became_active":"2024-05-28T06:18:09.647787+0000","last_became_peered":"2024-05-28T06:18:09.647787+0000","last_unstale":"2024-05-28T06:18:09.649566+0000","last_undegraded":"2024-05-28T06:18:09.649566+0000","last_fullsized":"2024-05-28T06:18:09.649566+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:53:34.184416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.626358+0000","last_change":"2024-05-28T06:18:09.626358+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.626358+0000","last_undegraded":"2024-05-28T06:18:09.626358+0000","last_fullsized":"2024-05-28T06:18:09.626358+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-28T06:18:09.645619+0000","last_change":"2024-05-28T06:18:09.643958+0000","last_active":"2024-05-28T06:18:09.645619+0000","last_peered":"2024-05-28T06:18:09.645619+0000","last_clean":"2024-05-28T06:18:09.645619+0000","last_became_active":"2024-05-28T06:18:09.643658+0000","last_became_peered":"2024-05-28T06:18:09.643658+0000","last_unstale":"2024-05-28T06:18:09.645619+0000","last_undegraded":"2024-05-28T06:18:09.645619+0000","last_fullsized":"2024-05-28T06:18:09.645619+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:22:37.288639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.632712+0000","last_change":"2024-05-28T06:18:09.632712+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.632712+0000","last_undegraded":"2024-05-28T06:18:09.632712+0000","last_fullsized":"2024-05-28T06:18:09.632712+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-28T06:18:09.646827+0000","last_change":"2024-05-28T06:18:09.645952+0000","last_active":"2024-05-28T06:18:09.646827+0000","last_peered":"2024-05-28T06:18:09.646827+0000","last_clean":"2024-05-28T06:18:09.646827+0000","last_became_active":"2024-05-28T06:18:09.644892+0000","last_became_peered":"2024-05-28T06:18:09.644892+0000","last_unstale":"2024-05-28T06:18:09.646827+0000","last_undegraded":"2024-05-28T06:18:09.646827+0000","last_fullsized":"2024-05-28T06:18:09.646827+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:44:26.345831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.636113+0000","last_change":"2024-05-28T06:18:09.636113+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.636113+0000","last_undegraded":"2024-05-28T06:18:09.636113+0000","last_fullsized":"2024-05-28T06:18:09.636113+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-28T06:18:09.649034+0000","last_change":"2024-05-28T06:18:09.648424+0000","last_active":"2024-05-28T06:18:09.649034+0000","last_peered":"2024-05-28T06:18:09.649034+0000","last_clean":"2024-05-28T06:18:09.649034+0000","last_became_active":"2024-05-28T06:18:09.648138+0000","last_became_peered":"2024-05-28T06:18:09.648138+0000","last_unstale":"2024-05-28T06:18:09.649034+0000","last_undegraded":"2024-05-28T06:18:09.649034+0000","last_fullsized":"2024-05-28T06:18:09.649034+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:57:25.848771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.630542+0000","last_change":"2024-05-28T06:18:09.630542+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.630542+0000","last_undegraded":"2024-05-28T06:18:09.630542+0000","last_fullsized":"2024-05-28T06:18:09.630542+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-28T06:18:09.648677+0000","last_change":"2024-05-28T06:18:09.645431+0000","last_active":"2024-05-28T06:18:09.648677+0000","last_peered":"2024-05-28T06:18:09.648677+0000","last_clean":"2024-05-28T06:18:09.648677+0000","last_became_active":"2024-05-28T06:18:09.645141+0000","last_became_peered":"2024-05-28T06:18:09.645141+0000","last_unstale":"2024-05-28T06:18:09.648677+0000","last_undegraded":"2024-05-28T06:18:09.648677+0000","last_fullsized":"2024-05-28T06:18:09.648677+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:37:17.604820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.633752+0000","last_change":"2024-05-28T06:18:09.633752+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.633752+0000","last_undegraded":"2024-05-28T06:18:09.633752+0000","last_fullsized":"2024-05-28T06:18:09.633752+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-28T06:18:09.649440+0000","last_change":"2024-05-28T06:18:09.647148+0000","last_active":"2024-05-28T06:18:09.649440+0000","last_peered":"2024-05-28T06:18:09.649440+0000","last_clean":"2024-05-28T06:18:09.649440+0000","last_became_active":"2024-05-28T06:18:09.646882+0000","last_became_peered":"2024-05-28T06:18:09.646882+0000","last_unstale":"2024-05-28T06:18:09.649440+0000","last_undegraded":"2024-05-28T06:18:09.649440+0000","last_fullsized":"2024-05-28T06:18:09.649440+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:06:17.684186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-28T06:18:10.688761+0000","last_change":"2024-05-28T06:18:10.687447+0000","last_active":"2024-05-28T06:18:10.688761+0000","last_peered":"2024-05-28T06:18:10.688761+0000","last_clean":"2024-05-28T06:18:10.688761+0000","last_became_active":"2024-05-28T06:18:10.687161+0000","last_became_peered":"2024-05-28T06:18:10.687161+0000","last_unstale":"2024-05-28T06:18:10.688761+0000","last_undegraded":"2024-05-28T06:18:10.688761+0000","last_fullsized":"2024-05-28T06:18:10.688761+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:52:52.857149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-28T06:18:09.649258+0000","last_change":"2024-05-28T06:18:09.647907+0000","last_active":"2024-05-28T06:18:09.649258+0000","last_peered":"2024-05-28T06:18:09.649258+0000","last_clean":"2024-05-28T06:18:09.649258+0000","last_became_active":"2024-05-28T06:18:09.647240+0000","last_became_peered":"2024-05-28T06:18:09.647240+0000","last_unstale":"2024-05-28T06:18:09.649258+0000","last_undegraded":"2024-05-28T06:18:09.649258+0000","last_fullsized":"2024-05-28T06:18:09.649258+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:42:11.297790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.630019+0000","last_change":"2024-05-28T06:18:09.630019+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.630019+0000","last_undegraded":"2024-05-28T06:18:09.630019+0000","last_fullsized":"2024-05-28T06:18:09.630019+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-28T06:18:09.648641+0000","last_change":"2024-05-28T06:18:09.645407+0000","last_active":"2024-05-28T06:18:09.648641+0000","last_peered":"2024-05-28T06:18:09.648641+0000","last_clean":"2024-05-28T06:18:09.648641+0000","last_became_active":"2024-05-28T06:18:09.644308+0000","last_became_peered":"2024-05-28T06:18:09.644308+0000","last_unstale":"2024-05-28T06:18:09.648641+0000","last_undegraded":"2024-05-28T06:18:09.648641+0000","last_fullsized":"2024-05-28T06:18:09.648641+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:50:32.547130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.639445+0000","last_change":"2024-05-28T06:18:09.639445+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.639445+0000","last_undegraded":"2024-05-28T06:18:09.639445+0000","last_fullsized":"2024-05-28T06:18:09.639445+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-28T06:18:09.646760+0000","last_change":"2024-05-28T06:18:09.643338+0000","last_active":"2024-05-28T06:18:09.646760+0000","last_peered":"2024-05-28T06:18:09.646760+0000","last_clean":"2024-05-28T06:18:09.646760+0000","last_became_active":"2024-05-28T06:18:09.643035+0000","last_became_peered":"2024-05-28T06:18:09.643035+0000","last_unstale":"2024-05-28T06:18:09.646760+0000","last_undegraded":"2024-05-28T06:18:09.646760+0000","last_fullsized":"2024-05-28T06:18:09.646760+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:15:01.390069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-28T06:18:10.676395+0000","last_change":"2024-05-28T06:18:10.673463+0000","last_active":"2024-05-28T06:18:10.676395+0000","last_peered":"2024-05-28T06:18:10.676395+0000","last_clean":"2024-05-28T06:18:10.676395+0000","last_became_active":"2024-05-28T06:18:10.673150+0000","last_became_peered":"2024-05-28T06:18:10.673150+0000","last_unstale":"2024-05-28T06:18:10.676395+0000","last_undegraded":"2024-05-28T06:18:10.676395+0000","last_fullsized":"2024-05-28T06:18:10.676395+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:43:28.316407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-28T06:18:09.642014+0000","last_change":"2024-05-28T06:18:09.638887+0000","last_active":"2024-05-28T06:18:09.642014+0000","last_peered":"2024-05-28T06:18:09.642014+0000","last_clean":"2024-05-28T06:18:09.642014+0000","last_became_active":"2024-05-28T06:18:09.638560+0000","last_became_peered":"2024-05-28T06:18:09.638560+0000","last_unstale":"2024-05-28T06:18:09.642014+0000","last_undegraded":"2024-05-28T06:18:09.642014+0000","last_fullsized":"2024-05-28T06:18:09.642014+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:50:17.041537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.634893+0000","last_change":"2024-05-28T06:18:09.634893+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.634893+0000","last_undegraded":"2024-05-28T06:18:09.634893+0000","last_fullsized":"2024-05-28T06:18:09.634893+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-28T06:18:09.651585+0000","last_change":"2024-05-28T06:18:09.647691+0000","last_active":"2024-05-28T06:18:09.651585+0000","last_peered":"2024-05-28T06:18:09.651585+0000","last_clean":"2024-05-28T06:18:09.651585+0000","last_became_active":"2024-05-28T06:18:09.647419+0000","last_became_peered":"2024-05-28T06:18:09.647419+0000","last_unstale":"2024-05-28T06:18:09.651585+0000","last_undegraded":"2024-05-28T06:18:09.651585+0000","last_fullsized":"2024-05-28T06:18:09.651585+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:37:17.604820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.629576+0000","last_change":"2024-05-28T06:18:09.629576+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.629576+0000","last_undegraded":"2024-05-28T06:18:09.629576+0000","last_fullsized":"2024-05-28T06:18:09.629576+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-28T06:18:09.651564+0000","last_change":"2024-05-28T06:18:09.649938+0000","last_active":"2024-05-28T06:18:09.651564+0000","last_peered":"2024-05-28T06:18:09.651564+0000","last_clean":"2024-05-28T06:18:09.651564+0000","last_became_active":"2024-05-28T06:18:09.649011+0000","last_became_peered":"2024-05-28T06:18:09.649011+0000","last_unstale":"2024-05-28T06:18:09.651564+0000","last_undegraded":"2024-05-28T06:18:09.651564+0000","last_fullsized":"2024-05-28T06:18:09.651564+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:33:46.549475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.624892+0000","last_change":"2024-05-28T06:18:09.624892+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.624892+0000","last_undegraded":"2024-05-28T06:18:09.624892+0000","last_fullsized":"2024-05-28T06:18:09.624892+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-28T06:18:09.640111+0000","last_change":"2024-05-28T06:18:09.631399+0000","last_active":"2024-05-28T06:18:09.640111+0000","last_peered":"2024-05-28T06:18:09.640111+0000","last_clean":"2024-05-28T06:18:09.640111+0000","last_became_active":"2024-05-28T06:18:09.630708+0000","last_became_peered":"2024-05-28T06:18:09.630708+0000","last_unstale":"2024-05-28T06:18:09.640111+0000","last_undegraded":"2024-05-28T06:18:09.640111+0000","last_fullsized":"2024-05-28T06:18:09.640111+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T18:13:07.090030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.629949+0000","last_change":"2024-05-28T06:18:09.629949+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.629949+0000","last_undegraded":"2024-05-28T06:18:09.629949+0000","last_fullsized":"2024-05-28T06:18:09.629949+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-28T06:18:09.647176+0000","last_change":"2024-05-28T06:18:09.645491+0000","last_active":"2024-05-28T06:18:09.647176+0000","last_peered":"2024-05-28T06:18:09.647176+0000","last_clean":"2024-05-28T06:18:09.647176+0000","last_became_active":"2024-05-28T06:18:09.645198+0000","last_became_peered":"2024-05-28T06:18:09.645198+0000","last_unstale":"2024-05-28T06:18:09.647176+0000","last_undegraded":"2024-05-28T06:18:09.647176+0000","last_fullsized":"2024-05-28T06:18:09.647176+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:27:19.014710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.624502+0000","last_change":"2024-05-28T06:18:09.624502+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.624502+0000","last_undegraded":"2024-05-28T06:18:09.624502+0000","last_fullsized":"2024-05-28T06:18:09.624502+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-28T06:18:09.640139+0000","last_change":"2024-05-28T06:18:09.638348+0000","last_active":"2024-05-28T06:18:09.640139+0000","last_peered":"2024-05-28T06:18:09.640139+0000","last_clean":"2024-05-28T06:18:09.640139+0000","last_became_active":"2024-05-28T06:18:09.638008+0000","last_became_peered":"2024-05-28T06:18:09.638008+0000","last_unstale":"2024-05-28T06:18:09.640139+0000","last_undegraded":"2024-05-28T06:18:09.640139+0000","last_fullsized":"2024-05-28T06:18:09.640139+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:52:16.609469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.630530+0000","last_change":"2024-05-28T06:18:09.630530+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.630530+0000","last_undegraded":"2024-05-28T06:18:09.630530+0000","last_fullsized":"2024-05-28T06:18:09.630530+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-28T06:18:09.637741+0000","last_change":"2024-05-28T06:18:09.632017+0000","last_active":"2024-05-28T06:18:09.637741+0000","last_peered":"2024-05-28T06:18:09.637741+0000","last_clean":"2024-05-28T06:18:09.637741+0000","last_became_active":"2024-05-28T06:18:09.631229+0000","last_became_peered":"2024-05-28T06:18:09.631229+0000","last_unstale":"2024-05-28T06:18:09.637741+0000","last_undegraded":"2024-05-28T06:18:09.637741+0000","last_fullsized":"2024-05-28T06:18:09.637741+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:52:51.842112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.634408+0000","last_change":"2024-05-28T06:18:09.634408+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.634408+0000","last_undegraded":"2024-05-28T06:18:09.634408+0000","last_fullsized":"2024-05-28T06:18:09.634408+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-28T06:18:09.649128+0000","last_change":"2024-05-28T06:18:09.646324+0000","last_active":"2024-05-28T06:18:09.649128+0000","last_peered":"2024-05-28T06:18:09.649128+0000","last_clean":"2024-05-28T06:18:09.649128+0000","last_became_active":"2024-05-28T06:18:09.646028+0000","last_became_peered":"2024-05-28T06:18:09.646028+0000","last_unstale":"2024-05-28T06:18:09.649128+0000","last_undegraded":"2024-05-28T06:18:09.649128+0000","last_fullsized":"2024-05-28T06:18:09.649128+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:09:40.380342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-28T06:18:10.675571+0000","last_change":"2024-05-28T06:18:10.673086+0000","last_active":"2024-05-28T06:18:10.675571+0000","last_peered":"2024-05-28T06:18:10.675571+0000","last_clean":"2024-05-28T06:18:10.675571+0000","last_became_active":"2024-05-28T06:18:10.672555+0000","last_became_peered":"2024-05-28T06:18:10.672555+0000","last_unstale":"2024-05-28T06:18:10.675571+0000","last_undegraded":"2024-05-28T06:18:10.675571+0000","last_fullsized":"2024-05-28T06:18:10.675571+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:30:38.196353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-28T06:18:09.649680+0000","last_change":"2024-05-28T06:18:09.646621+0000","last_active":"2024-05-28T06:18:09.649680+0000","last_peered":"2024-05-28T06:18:09.649680+0000","last_clean":"2024-05-28T06:18:09.649680+0000","last_became_active":"2024-05-28T06:18:09.646289+0000","last_became_peered":"2024-05-28T06:18:09.646289+0000","last_unstale":"2024-05-28T06:18:09.649680+0000","last_undegraded":"2024-05-28T06:18:09.649680+0000","last_fullsized":"2024-05-28T06:18:09.649680+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:07:03.763128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.626049+0000","last_change":"2024-05-28T06:18:09.626049+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.626049+0000","last_undegraded":"2024-05-28T06:18:09.626049+0000","last_fullsized":"2024-05-28T06:18:09.626049+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-28T06:18:09.644557+0000","last_change":"2024-05-28T06:18:09.642139+0000","last_active":"2024-05-28T06:18:09.644557+0000","last_peered":"2024-05-28T06:18:09.644557+0000","last_clean":"2024-05-28T06:18:09.644557+0000","last_became_active":"2024-05-28T06:18:09.641858+0000","last_became_peered":"2024-05-28T06:18:09.641858+0000","last_unstale":"2024-05-28T06:18:09.644557+0000","last_undegraded":"2024-05-28T06:18:09.644557+0000","last_fullsized":"2024-05-28T06:18:09.644557+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:32:35.049306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.628587+0000","last_change":"2024-05-28T06:18:09.628587+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.628587+0000","last_undegraded":"2024-05-28T06:18:09.628587+0000","last_fullsized":"2024-05-28T06:18:09.628587+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-28T06:18:09.640418+0000","last_change":"2024-05-28T06:18:09.638402+0000","last_active":"2024-05-28T06:18:09.640418+0000","last_peered":"2024-05-28T06:18:09.640418+0000","last_clean":"2024-05-28T06:18:09.640418+0000","last_became_active":"2024-05-28T06:18:09.637801+0000","last_became_peered":"2024-05-28T06:18:09.637801+0000","last_unstale":"2024-05-28T06:18:09.640418+0000","last_undegraded":"2024-05-28T06:18:09.640418+0000","last_fullsized":"2024-05-28T06:18:09.640418+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:18:25.285292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.629922+0000","last_change":"2024-05-28T06:18:09.629922+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.629922+0000","last_undegraded":"2024-05-28T06:18:09.629922+0000","last_fullsized":"2024-05-28T06:18:09.629922+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-28T06:18:09.645575+0000","last_change":"2024-05-28T06:18:09.642946+0000","last_active":"2024-05-28T06:18:09.645575+0000","last_peered":"2024-05-28T06:18:09.645575+0000","last_clean":"2024-05-28T06:18:09.645575+0000","last_became_active":"2024-05-28T06:18:09.642546+0000","last_became_peered":"2024-05-28T06:18:09.642546+0000","last_unstale":"2024-05-28T06:18:09.645575+0000","last_undegraded":"2024-05-28T06:18:09.645575+0000","last_fullsized":"2024-05-28T06:18:09.645575+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:41:26.564391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.630093+0000","last_change":"2024-05-28T06:18:09.630093+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.630093+0000","last_undegraded":"2024-05-28T06:18:09.630093+0000","last_fullsized":"2024-05-28T06:18:09.630093+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-28T06:18:09.644520+0000","last_change":"2024-05-28T06:18:09.643305+0000","last_active":"2024-05-28T06:18:09.644520+0000","last_peered":"2024-05-28T06:18:09.644520+0000","last_clean":"2024-05-28T06:18:09.644520+0000","last_became_active":"2024-05-28T06:18:09.642959+0000","last_became_peered":"2024-05-28T06:18:09.642959+0000","last_unstale":"2024-05-28T06:18:09.644520+0000","last_undegraded":"2024-05-28T06:18:09.644520+0000","last_fullsized":"2024-05-28T06:18:09.644520+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:18:25.285292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-28T06:18:10.688312+0000","last_change":"2024-05-28T06:18:10.682613+0000","last_active":"2024-05-28T06:18:10.688312+0000","last_peered":"2024-05-28T06:18:10.688312+0000","last_clean":"2024-05-28T06:18:10.688312+0000","last_became_active":"2024-05-28T06:18:10.682019+0000","last_became_peered":"2024-05-28T06:18:10.682019+0000","last_unstale":"2024-05-28T06:18:10.688312+0000","last_undegraded":"2024-05-28T06:18:10.688312+0000","last_fullsized":"2024-05-28T06:18:10.688312+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:40:59.653395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-28T06:18:09.640358+0000","last_change":"2024-05-28T06:18:09.637819+0000","last_active":"2024-05-28T06:18:09.640358+0000","last_peered":"2024-05-28T06:18:09.640358+0000","last_clean":"2024-05-28T06:18:09.640358+0000","last_became_active":"2024-05-28T06:18:09.637503+0000","last_became_peered":"2024-05-28T06:18:09.637503+0000","last_unstale":"2024-05-28T06:18:09.640358+0000","last_undegraded":"2024-05-28T06:18:09.640358+0000","last_fullsized":"2024-05-28T06:18:09.640358+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:46:24.986403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.633740+0000","last_change":"2024-05-28T06:18:09.633740+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.633740+0000","last_undegraded":"2024-05-28T06:18:09.633740+0000","last_fullsized":"2024-05-28T06:18:09.633740+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-28T06:18:09.646406+0000","last_change":"2024-05-28T06:18:09.641057+0000","last_active":"2024-05-28T06:18:09.646406+0000","last_peered":"2024-05-28T06:18:09.646406+0000","last_clean":"2024-05-28T06:18:09.646406+0000","last_became_active":"2024-05-28T06:18:09.640751+0000","last_became_peered":"2024-05-28T06:18:09.640751+0000","last_unstale":"2024-05-28T06:18:09.646406+0000","last_undegraded":"2024-05-28T06:18:09.646406+0000","last_fullsized":"2024-05-28T06:18:09.646406+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:15:39.926872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.634397+0000","last_change":"2024-05-28T06:18:09.634397+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.634397+0000","last_undegraded":"2024-05-28T06:18:09.634397+0000","last_fullsized":"2024-05-28T06:18:09.634397+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"88'1","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-28T06:18:10.687989+0000","last_change":"2024-05-28T06:18:09.649924+0000","last_active":"2024-05-28T06:18:10.687989+0000","last_peered":"2024-05-28T06:18:10.687989+0000","last_clean":"2024-05-28T06:18:10.687989+0000","last_became_active":"2024-05-28T06:18:09.649696+0000","last_became_peered":"2024-05-28T06:18:09.649696+0000","last_unstale":"2024-05-28T06:18:10.687989+0000","last_undegraded":"2024-05-28T06:18:10.687989+0000","last_fullsized":"2024-05-28T06:18:10.687989+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:10:32.947652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":303,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-28T06:18:10.663404+0000","last_change":"2024-05-28T06:04:01.039129+0000","last_active":"2024-05-28T06:18:10.663404+0000","last_peered":"2024-05-28T06:18:10.663404+0000","last_clean":"2024-05-28T06:18:10.663404+0000","last_became_active":"2024-05-28T06:04:01.033906+0000","last_became_peered":"2024-05-28T06:04:01.033906+0000","last_unstale":"2024-05-28T06:18:10.663404+0000","last_undegraded":"2024-05-28T06:18:10.663404+0000","last_fullsized":"2024-05-28T06:18:10.663404+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:03:59.864237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:03:59.864237+0000","last_clean_scrub_stamp":"2024-05-28T06:03:59.864237+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:17:36.440245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575021,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":122992,"kb_used_data":1640,"kb_used_omap":1,"kb_used_meta":121342,"kb_avail":94248848,"statfs":{"total":96636764160,"available":96510820352,"internally_reserved":0,"allocated":1679360,"data_stored":1057712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":124254666},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575023,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":122404,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":121342,"kb_avail":94249436,"statfs":{"total":96636764160,"available":96511422464,"internally_reserved":0,"allocated":1069056,"data_stored":460471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":124254662},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575021,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76576,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94295264,"statfs":{"total":96636764160,"available":96558350336,"internally_reserved":0,"allocated":1056768,"data_stored":454168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575021,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64084,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94307756,"statfs":{"total":96636764160,"available":96571142144,"internally_reserved":0,"allocated":1056768,"data_stored":454168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575023,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":122388,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":121342,"kb_avail":94249452,"statfs":{"total":96636764160,"available":96511438848,"internally_reserved":0,"allocated":1056768,"data_stored":454168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":124254663},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575022,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":122392,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":121342,"kb_avail":94249448,"statfs":{"total":96636764160,"available":96511434752,"internally_reserved":0,"allocated":1056768,"data_stored":454168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124254661},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575022,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":104340,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":103294,"kb_avail":94267500,"statfs":{"total":96636764160,"available":96529920000,"internally_reserved":0,"allocated":1056768,"data_stored":454168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105773509},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575021,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77156,"kb_used_data":1612,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94294684,"statfs":{"total":96636764160,"available":96557756416,"internally_reserved":0,"allocated":1650688,"data_stored":1044536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-28T06:18:15.097 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-28T06:18:15.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.335+0000 7f1c1ec00640 1 Processor -- start 2024-05-28T06:18:15.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.335+0000 7f1c1ec00640 1 -- start start 2024-05-28T06:18:15.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.336+0000 7f1c1ec00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f1c1815b3a0 0x7f1c180b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:15.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.336+0000 7f1c1ec00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f1c18073310 0x7f1c1815f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:15.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.336+0000 7f1c1ec00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c180afd60 con 0x7f1c1806dc20 2024-05-28T06:18:15.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.336+0000 7f1c1ec00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f1c1806f980 con 0x7f1c18073310 2024-05-28T06:18:15.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.336+0000 7f1c1ec00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f1c180ae0c0 con 0x7f1c1815b3a0 2024-05-28T06:18:15.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.337+0000 7f1c1ce00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f1c1806dc20 0x7f1c18072c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:49942/0 (socket says 172.21.15.96:49942) 2024-05-28T06:18:15.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.337+0000 7f1c1ce00640 1 -- 172.21.15.96:0/792106847 learned_addr learned my addr 172.21.15.96:0/792106847 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:18:15.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.337+0000 7f1c1e200640 1 --2- 172.21.15.96:0/792106847 >> v2:172.21.15.96:3300/0 conn(0x7f1c18073310 0x7f1c1815f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:15.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.337+0000 7f1c1d800640 1 --2- 172.21.15.96:0/792106847 >> v2:172.21.15.123:3300/0 conn(0x7f1c1815b3a0 0x7f1c180b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:15.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.337+0000 7f1c1e200640 1 -- 172.21.15.96:0/792106847 >> v2:172.21.15.123:3300/0 conn(0x7f1c1815b3a0 msgr2=0x7f1c180b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:18:15.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.337+0000 7f1c1e200640 1 --2- 172.21.15.96:0/792106847 >> v2:172.21.15.123:3300/0 conn(0x7f1c1815b3a0 0x7f1c180b32c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:15.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.337+0000 7f1c1e200640 1 -- 172.21.15.96:0/792106847 >> v1:172.21.15.96:6789/0 conn(0x7f1c1806dc20 legacy=0x7f1c18072c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:15.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.337+0000 7f1c1e200640 1 -- 172.21.15.96:0/792106847 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c180af7a0 con 0x7f1c18073310 2024-05-28T06:18:15.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.337+0000 7f1c17e00640 1 -- 172.21.15.96:0/792106847 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2570434052 0 0) 0x7f1c180afd60 con 0x7f1c1806dc20 2024-05-28T06:18:15.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.338+0000 7f1c1e200640 1 --2- 172.21.15.96:0/792106847 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1c18073310 0x7f1c1815f9a0 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7f1c18072bd0 tx=0x7f1c1002a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26b963e50fb5ba37 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:15.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.339+0000 7f1c17e00640 1 -- 172.21.15.96:0/792106847 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c10007740 con 0x7f1c18073310 2024-05-28T06:18:15.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.339+0000 7f1c17e00640 1 -- 172.21.15.96:0/792106847 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c100078e0 con 0x7f1c18073310 2024-05-28T06:18:15.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.339+0000 7f1c17e00640 1 -- 172.21.15.96:0/792106847 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c10007cb0 con 0x7f1c18073310 2024-05-28T06:18:15.337 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.339+0000 7f1c1ec00640 1 -- 172.21.15.96:0/792106847 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1c18073310 msgr2=0x7f1c1815f9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:15.337 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.339+0000 7f1c1ec00640 1 --2- 172.21.15.96:0/792106847 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1c18073310 0x7f1c1815f9a0 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7f1c18072bd0 tx=0x7f1c1002a680 comp rx=0 tx=0).stop 2024-05-28T06:18:15.337 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.339+0000 7f1c1ec00640 1 -- 172.21.15.96:0/792106847 shutdown_connections 2024-05-28T06:18:15.337 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.339+0000 7f1c1ec00640 1 --2- 172.21.15.96:0/792106847 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1c18073310 0x7f1c1815f9a0 unknown :-1 s=CLOSED pgs=1413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:15.337 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.339+0000 7f1c1ec00640 1 --2- 172.21.15.96:0/792106847 >> v2:172.21.15.123:3300/0 conn(0x7f1c1815b3a0 0x7f1c180b32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:15.337 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.339+0000 7f1c1ec00640 1 -- 172.21.15.96:0/792106847 >> 172.21.15.96:0/792106847 conn(0x7f1c180b1fc0 msgr2=0x7f1c180b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:18:15.337 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.339+0000 7f1c1ec00640 1 -- 172.21.15.96:0/792106847 shutdown_connections 2024-05-28T06:18:15.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.339+0000 7f1c1ec00640 1 -- 172.21.15.96:0/792106847 wait complete. 2024-05-28T06:18:15.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.341+0000 7f1c1ec00640 1 Processor -- start 2024-05-28T06:18:15.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.341+0000 7f1c1ec00640 1 -- start start 2024-05-28T06:18:15.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.342+0000 7f1c1ec00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c1806dc20 0x7f1c1815e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:15.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.342+0000 7f1c1ec00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1c18073310 0x7f1c180cb140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:15.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.342+0000 7f1c1ec00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1c1815b3a0 0x7f1c1814a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:15.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.342+0000 7f1c1ec00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f1c1815b780 con 0x7f1c18073310 2024-05-28T06:18:15.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.342+0000 7f1c1ec00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f1c1815fee0 con 0x7f1c1806dc20 2024-05-28T06:18:15.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.342+0000 7f1c1ec00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f1c180ae550 con 0x7f1c1815b3a0 2024-05-28T06:18:15.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.342+0000 7f1c1ce00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1c18073310 0x7f1c180cb140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:15.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.342+0000 7f1c1ce00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1c18073310 0x7f1c180cb140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48272/0 (socket says 172.21.15.96:48272) 2024-05-28T06:18:15.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.342+0000 7f1c1ce00640 1 -- 172.21.15.96:0/3783549172 learned_addr learned my addr 172.21.15.96:0/3783549172 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:18:15.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.343+0000 7f1c1d800640 1 --2- 172.21.15.96:0/3783549172 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c1806dc20 0x7f1c1815e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:15.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.343+0000 7f1c1e200640 1 --2- 172.21.15.96:0/3783549172 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1c1815b3a0 0x7f1c1814a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:15.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.343+0000 7f1c1ce00640 1 -- 172.21.15.96:0/3783549172 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1c1815b3a0 msgr2=0x7f1c1814a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:15.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.343+0000 7f1c1ce00640 1 --2- 172.21.15.96:0/3783549172 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1c1815b3a0 0x7f1c1814a500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:15.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.343+0000 7f1c1ce00640 1 -- 172.21.15.96:0/3783549172 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c1806dc20 msgr2=0x7f1c1815e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:15.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.343+0000 7f1c1ce00640 1 --2- 172.21.15.96:0/3783549172 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c1806dc20 0x7f1c1815e0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:15.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.343+0000 7f1c1ce00640 1 -- 172.21.15.96:0/3783549172 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c1814e800 con 0x7f1c18073310 2024-05-28T06:18:15.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.343+0000 7f1c1ce00640 1 --2- 172.21.15.96:0/3783549172 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1c18073310 0x7f1c180cb140 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f1c0400c4d0 tx=0x7f1c0400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:15.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.343+0000 7f1c15600640 1 -- 172.21.15.96:0/3783549172 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c04018070 con 0x7f1c18073310 2024-05-28T06:18:15.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.343+0000 7f1c15600640 1 -- 172.21.15.96:0/3783549172 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c04007500 con 0x7f1c18073310 2024-05-28T06:18:15.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.343+0000 7f1c1ec00640 1 -- 172.21.15.96:0/3783549172 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c1815d3c0 con 0x7f1c18073310 2024-05-28T06:18:15.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.343+0000 7f1c15600640 1 -- 172.21.15.96:0/3783549172 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c04004a20 con 0x7f1c18073310 2024-05-28T06:18:15.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.344+0000 7f1c1ec00640 1 -- 172.21.15.96:0/3783549172 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c1815e630 con 0x7f1c18073310 2024-05-28T06:18:15.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.345+0000 7f1c1ec00640 1 -- 172.21.15.96:0/3783549172 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c180689e0 con 0x7f1c18073310 2024-05-28T06:18:15.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.346+0000 7f1c15600640 1 -- 172.21.15.96:0/3783549172 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 30) ==== 92621+0+0 (secure 0 0 0) 0x7f1c0401b070 con 0x7f1c18073310 2024-05-28T06:18:15.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.347+0000 7f1c15600640 1 --2- 172.21.15.96:0/3783549172 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1bf806da60 0x7f1bf806ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:15.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.347+0000 7f1c15600640 1 -- 172.21.15.96:0/3783549172 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 5737+0+0 (secure 0 0 0) 0x7f1c04089da0 con 0x7f1c18073310 2024-05-28T06:18:15.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.348+0000 7f1c1d800640 1 --2- 172.21.15.96:0/3783549172 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1bf806da60 0x7f1bf806ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:15.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.349+0000 7f1c1d800640 1 --2- 172.21.15.96:0/3783549172 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1bf806da60 0x7f1bf806ff20 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c006fd0 tx=0x7f1c0c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:15.349 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.351+0000 7f1c15600640 1 -- 172.21.15.96:0/3783549172 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1c040598d0 con 0x7f1c18073310 2024-05-28T06:18:15.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.547+0000 7f1c1ec00640 1 -- 172.21.15.96:0/3783549172 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1c180b4f00 con 0x7f1bf806da60 2024-05-28T06:18:15.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.566+0000 7f1c15600640 1 -- 172.21.15.96:0/3783549172 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+247514 (secure 0 0 0) 0x7f1c180b4f00 con 0x7f1bf806da60 2024-05-28T06:18:15.566 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:18:15.568 INFO:teuthology.orchestra.run.smithi096.stderr:dumped all 2024-05-28T06:18:15.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.571+0000 7f1c1ec00640 1 -- 172.21.15.96:0/3783549172 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1bf806da60 msgr2=0x7f1bf806ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:15.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.571+0000 7f1c1ec00640 1 --2- 172.21.15.96:0/3783549172 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1bf806da60 0x7f1bf806ff20 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c006fd0 tx=0x7f1c0c008040 comp rx=0 tx=0).stop 2024-05-28T06:18:15.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.571+0000 7f1c1ec00640 1 -- 172.21.15.96:0/3783549172 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1c18073310 msgr2=0x7f1c180cb140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:15.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.571+0000 7f1c1ec00640 1 --2- 172.21.15.96:0/3783549172 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1c18073310 0x7f1c180cb140 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f1c0400c4d0 tx=0x7f1c0400c990 comp rx=0 tx=0).stop 2024-05-28T06:18:15.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.571+0000 7f1c1ec00640 1 -- 172.21.15.96:0/3783549172 shutdown_connections 2024-05-28T06:18:15.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.571+0000 7f1c1ec00640 1 --2- 172.21.15.96:0/3783549172 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1bf806da60 0x7f1bf806ff20 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:15.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.571+0000 7f1c1ec00640 1 --2- 172.21.15.96:0/3783549172 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1c1815b3a0 0x7f1c1814a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:15.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.571+0000 7f1c1ec00640 1 --2- 172.21.15.96:0/3783549172 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1c18073310 0x7f1c180cb140 unknown :-1 s=CLOSED pgs=1414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:15.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.571+0000 7f1c1ec00640 1 --2- 172.21.15.96:0/3783549172 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c1806dc20 0x7f1c1815e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:15.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.571+0000 7f1c1ec00640 1 -- 172.21.15.96:0/3783549172 >> 172.21.15.96:0/3783549172 conn(0x7f1c180b1fc0 msgr2=0x7f1c180b3440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:18:15.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.571+0000 7f1c1ec00640 1 -- 172.21.15.96:0/3783549172 shutdown_connections 2024-05-28T06:18:15.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.571+0000 7f1c1ec00640 1 -- 172.21.15.96:0/3783549172 wait complete. 2024-05-28T06:18:15.588 INFO:teuthology.orchestra.run.smithi096.stdout:{"pg_ready":true,"pg_map":{"version":509,"stamp":"2024-05-28T06:18:13.862529+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590438,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":160,"num_write_kb":2417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118,"ondisk_log_size":118,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":35,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":812332,"kb_used_data":9456,"kb_used_omap":12,"kb_used_meta":802739,"kb_avail":754162388,"statfs":{"total":773094113280,"available":772262285312,"internally_reserved":0,"allocated":9682944,"data_stored":4833559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":822005296},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.253304"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.635935+0000","last_change":"2024-05-28T06:18:09.635935+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.635935+0000","last_undegraded":"2024-05-28T06:18:09.635935+0000","last_fullsized":"2024-05-28T06:18:09.635935+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.639835+0000","last_change":"2024-05-28T06:18:09.639835+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.639835+0000","last_undegraded":"2024-05-28T06:18:09.639835+0000","last_fullsized":"2024-05-28T06:18:09.639835+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-28T06:18:10.688685+0000","last_change":"2024-05-28T06:18:10.684086+0000","last_active":"2024-05-28T06:18:10.688685+0000","last_peered":"2024-05-28T06:18:10.688685+0000","last_clean":"2024-05-28T06:18:10.688685+0000","last_became_active":"2024-05-28T06:18:10.683345+0000","last_became_peered":"2024-05-28T06:18:10.683345+0000","last_unstale":"2024-05-28T06:18:10.688685+0000","last_undegraded":"2024-05-28T06:18:10.688685+0000","last_fullsized":"2024-05-28T06:18:10.688685+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:36:04.760890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.629536+0000","last_change":"2024-05-28T06:18:09.629536+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.629536+0000","last_undegraded":"2024-05-28T06:18:09.629536+0000","last_fullsized":"2024-05-28T06:18:09.629536+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.637122+0000","last_change":"2024-05-28T06:18:09.637122+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.637122+0000","last_undegraded":"2024-05-28T06:18:09.637122+0000","last_fullsized":"2024-05-28T06:18:09.637122+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.631959+0000","last_change":"2024-05-28T06:18:09.631959+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.631959+0000","last_undegraded":"2024-05-28T06:18:09.631959+0000","last_fullsized":"2024-05-28T06:18:09.631959+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.628109+0000","last_change":"2024-05-28T06:18:09.628109+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.628109+0000","last_undegraded":"2024-05-28T06:18:09.628109+0000","last_fullsized":"2024-05-28T06:18:09.628109+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.628197+0000","last_change":"2024-05-28T06:18:09.628197+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.628197+0000","last_undegraded":"2024-05-28T06:18:09.628197+0000","last_fullsized":"2024-05-28T06:18:09.628197+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.633228+0000","last_change":"2024-05-28T06:18:09.633228+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.633228+0000","last_undegraded":"2024-05-28T06:18:09.633228+0000","last_fullsized":"2024-05-28T06:18:09.633228+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.626053+0000","last_change":"2024-05-28T06:18:09.626053+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.626053+0000","last_undegraded":"2024-05-28T06:18:09.626053+0000","last_fullsized":"2024-05-28T06:18:09.626053+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.640699+0000","last_change":"2024-05-28T06:18:09.640699+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.640699+0000","last_undegraded":"2024-05-28T06:18:09.640699+0000","last_fullsized":"2024-05-28T06:18:09.640699+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.627727+0000","last_change":"2024-05-28T06:18:09.627727+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.627727+0000","last_undegraded":"2024-05-28T06:18:09.627727+0000","last_fullsized":"2024-05-28T06:18:09.627727+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-28T06:18:10.676482+0000","last_change":"2024-05-28T06:18:10.673510+0000","last_active":"2024-05-28T06:18:10.676482+0000","last_peered":"2024-05-28T06:18:10.676482+0000","last_clean":"2024-05-28T06:18:10.676482+0000","last_became_active":"2024-05-28T06:18:10.673194+0000","last_became_peered":"2024-05-28T06:18:10.673194+0000","last_unstale":"2024-05-28T06:18:10.676482+0000","last_undegraded":"2024-05-28T06:18:10.676482+0000","last_fullsized":"2024-05-28T06:18:10.676482+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:10:10.001167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.626455+0000","last_change":"2024-05-28T06:18:09.626455+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.626455+0000","last_undegraded":"2024-05-28T06:18:09.626455+0000","last_fullsized":"2024-05-28T06:18:09.626455+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.628788+0000","last_change":"2024-05-28T06:18:09.628788+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.628788+0000","last_undegraded":"2024-05-28T06:18:09.628788+0000","last_fullsized":"2024-05-28T06:18:09.628788+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.626723+0000","last_change":"2024-05-28T06:18:09.626723+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.626723+0000","last_undegraded":"2024-05-28T06:18:09.626723+0000","last_fullsized":"2024-05-28T06:18:09.626723+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.635456+0000","last_change":"2024-05-28T06:18:09.635456+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.635456+0000","last_undegraded":"2024-05-28T06:18:09.635456+0000","last_fullsized":"2024-05-28T06:18:09.635456+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.626844+0000","last_change":"2024-05-28T06:18:09.626844+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.626844+0000","last_undegraded":"2024-05-28T06:18:09.626844+0000","last_fullsized":"2024-05-28T06:18:09.626844+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.638675+0000","last_change":"2024-05-28T06:18:09.638675+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.638675+0000","last_undegraded":"2024-05-28T06:18:09.638675+0000","last_fullsized":"2024-05-28T06:18:09.638675+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.628326+0000","last_change":"2024-05-28T06:18:09.628326+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.628326+0000","last_undegraded":"2024-05-28T06:18:09.628326+0000","last_fullsized":"2024-05-28T06:18:09.628326+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.626431+0000","last_change":"2024-05-28T06:18:09.626431+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.626431+0000","last_undegraded":"2024-05-28T06:18:09.626431+0000","last_fullsized":"2024-05-28T06:18:09.626431+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.632230+0000","last_change":"2024-05-28T06:18:09.632230+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.632230+0000","last_undegraded":"2024-05-28T06:18:09.632230+0000","last_fullsized":"2024-05-28T06:18:09.632230+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.634278+0000","last_change":"2024-05-28T06:18:09.634278+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.634278+0000","last_undegraded":"2024-05-28T06:18:09.634278+0000","last_fullsized":"2024-05-28T06:18:09.634278+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.625935+0000","last_change":"2024-05-28T06:18:09.625935+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.625935+0000","last_undegraded":"2024-05-28T06:18:09.625935+0000","last_fullsized":"2024-05-28T06:18:09.625935+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.637179+0000","last_change":"2024-05-28T06:18:09.637179+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.637179+0000","last_undegraded":"2024-05-28T06:18:09.637179+0000","last_fullsized":"2024-05-28T06:18:09.637179+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-28T06:18:10.688666+0000","last_change":"2024-05-28T06:18:10.682743+0000","last_active":"2024-05-28T06:18:10.688666+0000","last_peered":"2024-05-28T06:18:10.688666+0000","last_clean":"2024-05-28T06:18:10.688666+0000","last_became_active":"2024-05-28T06:18:10.682451+0000","last_became_peered":"2024-05-28T06:18:10.682451+0000","last_unstale":"2024-05-28T06:18:10.688666+0000","last_undegraded":"2024-05-28T06:18:10.688666+0000","last_fullsized":"2024-05-28T06:18:10.688666+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:38:26.397686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.636728+0000","last_change":"2024-05-28T06:18:09.636728+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.636728+0000","last_undegraded":"2024-05-28T06:18:09.636728+0000","last_fullsized":"2024-05-28T06:18:09.636728+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.627185+0000","last_change":"2024-05-28T06:18:09.627185+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.627185+0000","last_undegraded":"2024-05-28T06:18:09.627185+0000","last_fullsized":"2024-05-28T06:18:09.627185+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.632024+0000","last_change":"2024-05-28T06:18:09.632024+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.632024+0000","last_undegraded":"2024-05-28T06:18:09.632024+0000","last_fullsized":"2024-05-28T06:18:09.632024+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.631106+0000","last_change":"2024-05-28T06:18:09.631106+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.631106+0000","last_undegraded":"2024-05-28T06:18:09.631106+0000","last_fullsized":"2024-05-28T06:18:09.631106+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.626336+0000","last_change":"2024-05-28T06:18:09.626336+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.626336+0000","last_undegraded":"2024-05-28T06:18:09.626336+0000","last_fullsized":"2024-05-28T06:18:09.626336+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.626394+0000","last_change":"2024-05-28T06:18:09.626394+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.626394+0000","last_undegraded":"2024-05-28T06:18:09.626394+0000","last_fullsized":"2024-05-28T06:18:09.626394+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.636228+0000","last_change":"2024-05-28T06:18:09.636228+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.636228+0000","last_undegraded":"2024-05-28T06:18:09.636228+0000","last_fullsized":"2024-05-28T06:18:09.636228+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-28T06:18:09.645386+0000","last_change":"2024-05-28T06:18:09.643542+0000","last_active":"2024-05-28T06:18:09.645386+0000","last_peered":"2024-05-28T06:18:09.645386+0000","last_clean":"2024-05-28T06:18:09.645386+0000","last_became_active":"2024-05-28T06:18:09.641808+0000","last_became_peered":"2024-05-28T06:18:09.641808+0000","last_unstale":"2024-05-28T06:18:09.645386+0000","last_undegraded":"2024-05-28T06:18:09.645386+0000","last_fullsized":"2024-05-28T06:18:09.645386+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:24:16.141801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.635380+0000","last_change":"2024-05-28T06:18:09.635380+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.635380+0000","last_undegraded":"2024-05-28T06:18:09.635380+0000","last_fullsized":"2024-05-28T06:18:09.635380+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-28T06:18:09.648769+0000","last_change":"2024-05-28T06:18:09.647214+0000","last_active":"2024-05-28T06:18:09.648769+0000","last_peered":"2024-05-28T06:18:09.648769+0000","last_clean":"2024-05-28T06:18:09.648769+0000","last_became_active":"2024-05-28T06:18:09.646937+0000","last_became_peered":"2024-05-28T06:18:09.646937+0000","last_unstale":"2024-05-28T06:18:09.648769+0000","last_undegraded":"2024-05-28T06:18:09.648769+0000","last_fullsized":"2024-05-28T06:18:09.648769+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:32:35.049306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.626694+0000","last_change":"2024-05-28T06:18:09.626694+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.626694+0000","last_undegraded":"2024-05-28T06:18:09.626694+0000","last_fullsized":"2024-05-28T06:18:09.626694+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-28T06:18:09.650076+0000","last_change":"2024-05-28T06:18:09.646080+0000","last_active":"2024-05-28T06:18:09.650076+0000","last_peered":"2024-05-28T06:18:09.650076+0000","last_clean":"2024-05-28T06:18:09.650076+0000","last_became_active":"2024-05-28T06:18:09.645807+0000","last_became_peered":"2024-05-28T06:18:09.645807+0000","last_unstale":"2024-05-28T06:18:09.650076+0000","last_undegraded":"2024-05-28T06:18:09.650076+0000","last_fullsized":"2024-05-28T06:18:09.650076+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:50:32.547130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-28T06:18:10.680587+0000","last_change":"2024-05-28T06:18:10.677395+0000","last_active":"2024-05-28T06:18:10.680587+0000","last_peered":"2024-05-28T06:18:10.680587+0000","last_clean":"2024-05-28T06:18:10.680587+0000","last_became_active":"2024-05-28T06:18:10.677100+0000","last_became_peered":"2024-05-28T06:18:10.677100+0000","last_unstale":"2024-05-28T06:18:10.680587+0000","last_undegraded":"2024-05-28T06:18:10.680587+0000","last_fullsized":"2024-05-28T06:18:10.680587+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:35:48.065461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-28T06:18:09.648814+0000","last_change":"2024-05-28T06:18:09.643971+0000","last_active":"2024-05-28T06:18:09.648814+0000","last_peered":"2024-05-28T06:18:09.648814+0000","last_clean":"2024-05-28T06:18:09.648814+0000","last_became_active":"2024-05-28T06:18:09.643691+0000","last_became_peered":"2024-05-28T06:18:09.643691+0000","last_unstale":"2024-05-28T06:18:09.648814+0000","last_undegraded":"2024-05-28T06:18:09.648814+0000","last_fullsized":"2024-05-28T06:18:09.648814+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:48:38.824030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.630680+0000","last_change":"2024-05-28T06:18:09.630680+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.630680+0000","last_undegraded":"2024-05-28T06:18:09.630680+0000","last_fullsized":"2024-05-28T06:18:09.630680+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-28T06:18:09.647612+0000","last_change":"2024-05-28T06:18:09.645873+0000","last_active":"2024-05-28T06:18:09.647612+0000","last_peered":"2024-05-28T06:18:09.647612+0000","last_clean":"2024-05-28T06:18:09.647612+0000","last_became_active":"2024-05-28T06:18:09.645561+0000","last_became_peered":"2024-05-28T06:18:09.645561+0000","last_unstale":"2024-05-28T06:18:09.647612+0000","last_undegraded":"2024-05-28T06:18:09.647612+0000","last_fullsized":"2024-05-28T06:18:09.647612+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:50:17.041537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.627460+0000","last_change":"2024-05-28T06:18:09.627460+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.627460+0000","last_undegraded":"2024-05-28T06:18:09.627460+0000","last_fullsized":"2024-05-28T06:18:09.627460+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-28T06:18:09.644497+0000","last_change":"2024-05-28T06:18:09.641147+0000","last_active":"2024-05-28T06:18:09.644497+0000","last_peered":"2024-05-28T06:18:09.644497+0000","last_clean":"2024-05-28T06:18:09.644497+0000","last_became_active":"2024-05-28T06:18:09.640830+0000","last_became_peered":"2024-05-28T06:18:09.640830+0000","last_unstale":"2024-05-28T06:18:09.644497+0000","last_undegraded":"2024-05-28T06:18:09.644497+0000","last_fullsized":"2024-05-28T06:18:09.644497+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:46:24.986403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-28T06:18:10.688719+0000","last_change":"2024-05-28T06:18:10.684510+0000","last_active":"2024-05-28T06:18:10.688719+0000","last_peered":"2024-05-28T06:18:10.688719+0000","last_clean":"2024-05-28T06:18:10.688719+0000","last_became_active":"2024-05-28T06:18:10.684209+0000","last_became_peered":"2024-05-28T06:18:10.684209+0000","last_unstale":"2024-05-28T06:18:10.688719+0000","last_undegraded":"2024-05-28T06:18:10.688719+0000","last_fullsized":"2024-05-28T06:18:10.688719+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T18:13:08.105068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-28T06:18:09.651440+0000","last_change":"2024-05-28T06:18:09.647844+0000","last_active":"2024-05-28T06:18:09.651440+0000","last_peered":"2024-05-28T06:18:09.651440+0000","last_clean":"2024-05-28T06:18:09.651440+0000","last_became_active":"2024-05-28T06:18:09.647527+0000","last_became_peered":"2024-05-28T06:18:09.647527+0000","last_unstale":"2024-05-28T06:18:09.651440+0000","last_undegraded":"2024-05-28T06:18:09.651440+0000","last_fullsized":"2024-05-28T06:18:09.651440+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T18:02:58.935614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.626057+0000","last_change":"2024-05-28T06:18:09.626057+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.626057+0000","last_undegraded":"2024-05-28T06:18:09.626057+0000","last_fullsized":"2024-05-28T06:18:09.626057+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-28T06:18:09.649566+0000","last_change":"2024-05-28T06:18:09.648064+0000","last_active":"2024-05-28T06:18:09.649566+0000","last_peered":"2024-05-28T06:18:09.649566+0000","last_clean":"2024-05-28T06:18:09.649566+0000","last_became_active":"2024-05-28T06:18:09.647787+0000","last_became_peered":"2024-05-28T06:18:09.647787+0000","last_unstale":"2024-05-28T06:18:09.649566+0000","last_undegraded":"2024-05-28T06:18:09.649566+0000","last_fullsized":"2024-05-28T06:18:09.649566+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:53:34.184416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.626358+0000","last_change":"2024-05-28T06:18:09.626358+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.626358+0000","last_undegraded":"2024-05-28T06:18:09.626358+0000","last_fullsized":"2024-05-28T06:18:09.626358+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-28T06:18:09.645619+0000","last_change":"2024-05-28T06:18:09.643958+0000","last_active":"2024-05-28T06:18:09.645619+0000","last_peered":"2024-05-28T06:18:09.645619+0000","last_clean":"2024-05-28T06:18:09.645619+0000","last_became_active":"2024-05-28T06:18:09.643658+0000","last_became_peered":"2024-05-28T06:18:09.643658+0000","last_unstale":"2024-05-28T06:18:09.645619+0000","last_undegraded":"2024-05-28T06:18:09.645619+0000","last_fullsized":"2024-05-28T06:18:09.645619+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:22:37.288639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.632712+0000","last_change":"2024-05-28T06:18:09.632712+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.632712+0000","last_undegraded":"2024-05-28T06:18:09.632712+0000","last_fullsized":"2024-05-28T06:18:09.632712+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-28T06:18:09.646827+0000","last_change":"2024-05-28T06:18:09.645952+0000","last_active":"2024-05-28T06:18:09.646827+0000","last_peered":"2024-05-28T06:18:09.646827+0000","last_clean":"2024-05-28T06:18:09.646827+0000","last_became_active":"2024-05-28T06:18:09.644892+0000","last_became_peered":"2024-05-28T06:18:09.644892+0000","last_unstale":"2024-05-28T06:18:09.646827+0000","last_undegraded":"2024-05-28T06:18:09.646827+0000","last_fullsized":"2024-05-28T06:18:09.646827+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:44:26.345831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.636113+0000","last_change":"2024-05-28T06:18:09.636113+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.636113+0000","last_undegraded":"2024-05-28T06:18:09.636113+0000","last_fullsized":"2024-05-28T06:18:09.636113+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-28T06:18:09.649034+0000","last_change":"2024-05-28T06:18:09.648424+0000","last_active":"2024-05-28T06:18:09.649034+0000","last_peered":"2024-05-28T06:18:09.649034+0000","last_clean":"2024-05-28T06:18:09.649034+0000","last_became_active":"2024-05-28T06:18:09.648138+0000","last_became_peered":"2024-05-28T06:18:09.648138+0000","last_unstale":"2024-05-28T06:18:09.649034+0000","last_undegraded":"2024-05-28T06:18:09.649034+0000","last_fullsized":"2024-05-28T06:18:09.649034+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:57:25.848771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.630542+0000","last_change":"2024-05-28T06:18:09.630542+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.630542+0000","last_undegraded":"2024-05-28T06:18:09.630542+0000","last_fullsized":"2024-05-28T06:18:09.630542+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-28T06:18:09.648677+0000","last_change":"2024-05-28T06:18:09.645431+0000","last_active":"2024-05-28T06:18:09.648677+0000","last_peered":"2024-05-28T06:18:09.648677+0000","last_clean":"2024-05-28T06:18:09.648677+0000","last_became_active":"2024-05-28T06:18:09.645141+0000","last_became_peered":"2024-05-28T06:18:09.645141+0000","last_unstale":"2024-05-28T06:18:09.648677+0000","last_undegraded":"2024-05-28T06:18:09.648677+0000","last_fullsized":"2024-05-28T06:18:09.648677+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:37:17.604820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.633752+0000","last_change":"2024-05-28T06:18:09.633752+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.633752+0000","last_undegraded":"2024-05-28T06:18:09.633752+0000","last_fullsized":"2024-05-28T06:18:09.633752+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-28T06:18:09.649440+0000","last_change":"2024-05-28T06:18:09.647148+0000","last_active":"2024-05-28T06:18:09.649440+0000","last_peered":"2024-05-28T06:18:09.649440+0000","last_clean":"2024-05-28T06:18:09.649440+0000","last_became_active":"2024-05-28T06:18:09.646882+0000","last_became_peered":"2024-05-28T06:18:09.646882+0000","last_unstale":"2024-05-28T06:18:09.649440+0000","last_undegraded":"2024-05-28T06:18:09.649440+0000","last_fullsized":"2024-05-28T06:18:09.649440+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:06:17.684186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-28T06:18:10.688761+0000","last_change":"2024-05-28T06:18:10.687447+0000","last_active":"2024-05-28T06:18:10.688761+0000","last_peered":"2024-05-28T06:18:10.688761+0000","last_clean":"2024-05-28T06:18:10.688761+0000","last_became_active":"2024-05-28T06:18:10.687161+0000","last_became_peered":"2024-05-28T06:18:10.687161+0000","last_unstale":"2024-05-28T06:18:10.688761+0000","last_undegraded":"2024-05-28T06:18:10.688761+0000","last_fullsized":"2024-05-28T06:18:10.688761+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:52:52.857149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-28T06:18:09.649258+0000","last_change":"2024-05-28T06:18:09.647907+0000","last_active":"2024-05-28T06:18:09.649258+0000","last_peered":"2024-05-28T06:18:09.649258+0000","last_clean":"2024-05-28T06:18:09.649258+0000","last_became_active":"2024-05-28T06:18:09.647240+0000","last_became_peered":"2024-05-28T06:18:09.647240+0000","last_unstale":"2024-05-28T06:18:09.649258+0000","last_undegraded":"2024-05-28T06:18:09.649258+0000","last_fullsized":"2024-05-28T06:18:09.649258+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:42:11.297790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.630019+0000","last_change":"2024-05-28T06:18:09.630019+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.630019+0000","last_undegraded":"2024-05-28T06:18:09.630019+0000","last_fullsized":"2024-05-28T06:18:09.630019+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-28T06:18:09.648641+0000","last_change":"2024-05-28T06:18:09.645407+0000","last_active":"2024-05-28T06:18:09.648641+0000","last_peered":"2024-05-28T06:18:09.648641+0000","last_clean":"2024-05-28T06:18:09.648641+0000","last_became_active":"2024-05-28T06:18:09.644308+0000","last_became_peered":"2024-05-28T06:18:09.644308+0000","last_unstale":"2024-05-28T06:18:09.648641+0000","last_undegraded":"2024-05-28T06:18:09.648641+0000","last_fullsized":"2024-05-28T06:18:09.648641+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:50:32.547130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.639445+0000","last_change":"2024-05-28T06:18:09.639445+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.639445+0000","last_undegraded":"2024-05-28T06:18:09.639445+0000","last_fullsized":"2024-05-28T06:18:09.639445+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-28T06:18:09.646760+0000","last_change":"2024-05-28T06:18:09.643338+0000","last_active":"2024-05-28T06:18:09.646760+0000","last_peered":"2024-05-28T06:18:09.646760+0000","last_clean":"2024-05-28T06:18:09.646760+0000","last_became_active":"2024-05-28T06:18:09.643035+0000","last_became_peered":"2024-05-28T06:18:09.643035+0000","last_unstale":"2024-05-28T06:18:09.646760+0000","last_undegraded":"2024-05-28T06:18:09.646760+0000","last_fullsized":"2024-05-28T06:18:09.646760+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:15:01.390069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-28T06:18:10.676395+0000","last_change":"2024-05-28T06:18:10.673463+0000","last_active":"2024-05-28T06:18:10.676395+0000","last_peered":"2024-05-28T06:18:10.676395+0000","last_clean":"2024-05-28T06:18:10.676395+0000","last_became_active":"2024-05-28T06:18:10.673150+0000","last_became_peered":"2024-05-28T06:18:10.673150+0000","last_unstale":"2024-05-28T06:18:10.676395+0000","last_undegraded":"2024-05-28T06:18:10.676395+0000","last_fullsized":"2024-05-28T06:18:10.676395+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:43:28.316407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-28T06:18:09.642014+0000","last_change":"2024-05-28T06:18:09.638887+0000","last_active":"2024-05-28T06:18:09.642014+0000","last_peered":"2024-05-28T06:18:09.642014+0000","last_clean":"2024-05-28T06:18:09.642014+0000","last_became_active":"2024-05-28T06:18:09.638560+0000","last_became_peered":"2024-05-28T06:18:09.638560+0000","last_unstale":"2024-05-28T06:18:09.642014+0000","last_undegraded":"2024-05-28T06:18:09.642014+0000","last_fullsized":"2024-05-28T06:18:09.642014+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:50:17.041537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.634893+0000","last_change":"2024-05-28T06:18:09.634893+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.634893+0000","last_undegraded":"2024-05-28T06:18:09.634893+0000","last_fullsized":"2024-05-28T06:18:09.634893+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-28T06:18:09.651585+0000","last_change":"2024-05-28T06:18:09.647691+0000","last_active":"2024-05-28T06:18:09.651585+0000","last_peered":"2024-05-28T06:18:09.651585+0000","last_clean":"2024-05-28T06:18:09.651585+0000","last_became_active":"2024-05-28T06:18:09.647419+0000","last_became_peered":"2024-05-28T06:18:09.647419+0000","last_unstale":"2024-05-28T06:18:09.651585+0000","last_undegraded":"2024-05-28T06:18:09.651585+0000","last_fullsized":"2024-05-28T06:18:09.651585+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:37:17.604820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.629576+0000","last_change":"2024-05-28T06:18:09.629576+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.629576+0000","last_undegraded":"2024-05-28T06:18:09.629576+0000","last_fullsized":"2024-05-28T06:18:09.629576+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-28T06:18:09.651564+0000","last_change":"2024-05-28T06:18:09.649938+0000","last_active":"2024-05-28T06:18:09.651564+0000","last_peered":"2024-05-28T06:18:09.651564+0000","last_clean":"2024-05-28T06:18:09.651564+0000","last_became_active":"2024-05-28T06:18:09.649011+0000","last_became_peered":"2024-05-28T06:18:09.649011+0000","last_unstale":"2024-05-28T06:18:09.651564+0000","last_undegraded":"2024-05-28T06:18:09.651564+0000","last_fullsized":"2024-05-28T06:18:09.651564+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:33:46.549475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.624892+0000","last_change":"2024-05-28T06:18:09.624892+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.624892+0000","last_undegraded":"2024-05-28T06:18:09.624892+0000","last_fullsized":"2024-05-28T06:18:09.624892+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-28T06:18:09.640111+0000","last_change":"2024-05-28T06:18:09.631399+0000","last_active":"2024-05-28T06:18:09.640111+0000","last_peered":"2024-05-28T06:18:09.640111+0000","last_clean":"2024-05-28T06:18:09.640111+0000","last_became_active":"2024-05-28T06:18:09.630708+0000","last_became_peered":"2024-05-28T06:18:09.630708+0000","last_unstale":"2024-05-28T06:18:09.640111+0000","last_undegraded":"2024-05-28T06:18:09.640111+0000","last_fullsized":"2024-05-28T06:18:09.640111+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T18:13:07.090030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.629949+0000","last_change":"2024-05-28T06:18:09.629949+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.629949+0000","last_undegraded":"2024-05-28T06:18:09.629949+0000","last_fullsized":"2024-05-28T06:18:09.629949+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-28T06:18:09.647176+0000","last_change":"2024-05-28T06:18:09.645491+0000","last_active":"2024-05-28T06:18:09.647176+0000","last_peered":"2024-05-28T06:18:09.647176+0000","last_clean":"2024-05-28T06:18:09.647176+0000","last_became_active":"2024-05-28T06:18:09.645198+0000","last_became_peered":"2024-05-28T06:18:09.645198+0000","last_unstale":"2024-05-28T06:18:09.647176+0000","last_undegraded":"2024-05-28T06:18:09.647176+0000","last_fullsized":"2024-05-28T06:18:09.647176+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:27:19.014710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.624502+0000","last_change":"2024-05-28T06:18:09.624502+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.624502+0000","last_undegraded":"2024-05-28T06:18:09.624502+0000","last_fullsized":"2024-05-28T06:18:09.624502+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-28T06:18:09.640139+0000","last_change":"2024-05-28T06:18:09.638348+0000","last_active":"2024-05-28T06:18:09.640139+0000","last_peered":"2024-05-28T06:18:09.640139+0000","last_clean":"2024-05-28T06:18:09.640139+0000","last_became_active":"2024-05-28T06:18:09.638008+0000","last_became_peered":"2024-05-28T06:18:09.638008+0000","last_unstale":"2024-05-28T06:18:09.640139+0000","last_undegraded":"2024-05-28T06:18:09.640139+0000","last_fullsized":"2024-05-28T06:18:09.640139+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:52:16.609469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.630530+0000","last_change":"2024-05-28T06:18:09.630530+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.630530+0000","last_undegraded":"2024-05-28T06:18:09.630530+0000","last_fullsized":"2024-05-28T06:18:09.630530+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-28T06:18:09.637741+0000","last_change":"2024-05-28T06:18:09.632017+0000","last_active":"2024-05-28T06:18:09.637741+0000","last_peered":"2024-05-28T06:18:09.637741+0000","last_clean":"2024-05-28T06:18:09.637741+0000","last_became_active":"2024-05-28T06:18:09.631229+0000","last_became_peered":"2024-05-28T06:18:09.631229+0000","last_unstale":"2024-05-28T06:18:09.637741+0000","last_undegraded":"2024-05-28T06:18:09.637741+0000","last_fullsized":"2024-05-28T06:18:09.637741+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:52:51.842112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.634408+0000","last_change":"2024-05-28T06:18:09.634408+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.634408+0000","last_undegraded":"2024-05-28T06:18:09.634408+0000","last_fullsized":"2024-05-28T06:18:09.634408+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-28T06:18:09.649128+0000","last_change":"2024-05-28T06:18:09.646324+0000","last_active":"2024-05-28T06:18:09.649128+0000","last_peered":"2024-05-28T06:18:09.649128+0000","last_clean":"2024-05-28T06:18:09.649128+0000","last_became_active":"2024-05-28T06:18:09.646028+0000","last_became_peered":"2024-05-28T06:18:09.646028+0000","last_unstale":"2024-05-28T06:18:09.649128+0000","last_undegraded":"2024-05-28T06:18:09.649128+0000","last_fullsized":"2024-05-28T06:18:09.649128+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:09:40.380342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-28T06:18:10.675571+0000","last_change":"2024-05-28T06:18:10.673086+0000","last_active":"2024-05-28T06:18:10.675571+0000","last_peered":"2024-05-28T06:18:10.675571+0000","last_clean":"2024-05-28T06:18:10.675571+0000","last_became_active":"2024-05-28T06:18:10.672555+0000","last_became_peered":"2024-05-28T06:18:10.672555+0000","last_unstale":"2024-05-28T06:18:10.675571+0000","last_undegraded":"2024-05-28T06:18:10.675571+0000","last_fullsized":"2024-05-28T06:18:10.675571+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:30:38.196353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-28T06:18:09.649680+0000","last_change":"2024-05-28T06:18:09.646621+0000","last_active":"2024-05-28T06:18:09.649680+0000","last_peered":"2024-05-28T06:18:09.649680+0000","last_clean":"2024-05-28T06:18:09.649680+0000","last_became_active":"2024-05-28T06:18:09.646289+0000","last_became_peered":"2024-05-28T06:18:09.646289+0000","last_unstale":"2024-05-28T06:18:09.649680+0000","last_undegraded":"2024-05-28T06:18:09.649680+0000","last_fullsized":"2024-05-28T06:18:09.649680+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:07:03.763128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.626049+0000","last_change":"2024-05-28T06:18:09.626049+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.626049+0000","last_undegraded":"2024-05-28T06:18:09.626049+0000","last_fullsized":"2024-05-28T06:18:09.626049+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-28T06:18:09.644557+0000","last_change":"2024-05-28T06:18:09.642139+0000","last_active":"2024-05-28T06:18:09.644557+0000","last_peered":"2024-05-28T06:18:09.644557+0000","last_clean":"2024-05-28T06:18:09.644557+0000","last_became_active":"2024-05-28T06:18:09.641858+0000","last_became_peered":"2024-05-28T06:18:09.641858+0000","last_unstale":"2024-05-28T06:18:09.644557+0000","last_undegraded":"2024-05-28T06:18:09.644557+0000","last_fullsized":"2024-05-28T06:18:09.644557+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:32:35.049306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.628587+0000","last_change":"2024-05-28T06:18:09.628587+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.628587+0000","last_undegraded":"2024-05-28T06:18:09.628587+0000","last_fullsized":"2024-05-28T06:18:09.628587+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-28T06:18:09.640418+0000","last_change":"2024-05-28T06:18:09.638402+0000","last_active":"2024-05-28T06:18:09.640418+0000","last_peered":"2024-05-28T06:18:09.640418+0000","last_clean":"2024-05-28T06:18:09.640418+0000","last_became_active":"2024-05-28T06:18:09.637801+0000","last_became_peered":"2024-05-28T06:18:09.637801+0000","last_unstale":"2024-05-28T06:18:09.640418+0000","last_undegraded":"2024-05-28T06:18:09.640418+0000","last_fullsized":"2024-05-28T06:18:09.640418+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:18:25.285292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.629922+0000","last_change":"2024-05-28T06:18:09.629922+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.629922+0000","last_undegraded":"2024-05-28T06:18:09.629922+0000","last_fullsized":"2024-05-28T06:18:09.629922+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-28T06:18:09.645575+0000","last_change":"2024-05-28T06:18:09.642946+0000","last_active":"2024-05-28T06:18:09.645575+0000","last_peered":"2024-05-28T06:18:09.645575+0000","last_clean":"2024-05-28T06:18:09.645575+0000","last_became_active":"2024-05-28T06:18:09.642546+0000","last_became_peered":"2024-05-28T06:18:09.642546+0000","last_unstale":"2024-05-28T06:18:09.645575+0000","last_undegraded":"2024-05-28T06:18:09.645575+0000","last_fullsized":"2024-05-28T06:18:09.645575+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:41:26.564391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.630093+0000","last_change":"2024-05-28T06:18:09.630093+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.630093+0000","last_undegraded":"2024-05-28T06:18:09.630093+0000","last_fullsized":"2024-05-28T06:18:09.630093+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-28T06:18:09.644520+0000","last_change":"2024-05-28T06:18:09.643305+0000","last_active":"2024-05-28T06:18:09.644520+0000","last_peered":"2024-05-28T06:18:09.644520+0000","last_clean":"2024-05-28T06:18:09.644520+0000","last_became_active":"2024-05-28T06:18:09.642959+0000","last_became_peered":"2024-05-28T06:18:09.642959+0000","last_unstale":"2024-05-28T06:18:09.644520+0000","last_undegraded":"2024-05-28T06:18:09.644520+0000","last_fullsized":"2024-05-28T06:18:09.644520+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:18:25.285292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-28T06:18:10.688312+0000","last_change":"2024-05-28T06:18:10.682613+0000","last_active":"2024-05-28T06:18:10.688312+0000","last_peered":"2024-05-28T06:18:10.688312+0000","last_clean":"2024-05-28T06:18:10.688312+0000","last_became_active":"2024-05-28T06:18:10.682019+0000","last_became_peered":"2024-05-28T06:18:10.682019+0000","last_unstale":"2024-05-28T06:18:10.688312+0000","last_undegraded":"2024-05-28T06:18:10.688312+0000","last_fullsized":"2024-05-28T06:18:10.688312+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:40:59.653395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-28T06:18:09.640358+0000","last_change":"2024-05-28T06:18:09.637819+0000","last_active":"2024-05-28T06:18:09.640358+0000","last_peered":"2024-05-28T06:18:09.640358+0000","last_clean":"2024-05-28T06:18:09.640358+0000","last_became_active":"2024-05-28T06:18:09.637503+0000","last_became_peered":"2024-05-28T06:18:09.637503+0000","last_unstale":"2024-05-28T06:18:09.640358+0000","last_undegraded":"2024-05-28T06:18:09.640358+0000","last_fullsized":"2024-05-28T06:18:09.640358+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:46:24.986403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.633740+0000","last_change":"2024-05-28T06:18:09.633740+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.633740+0000","last_undegraded":"2024-05-28T06:18:09.633740+0000","last_fullsized":"2024-05-28T06:18:09.633740+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-05-28T06:18:09.646406+0000","last_change":"2024-05-28T06:18:09.641057+0000","last_active":"2024-05-28T06:18:09.646406+0000","last_peered":"2024-05-28T06:18:09.646406+0000","last_clean":"2024-05-28T06:18:09.646406+0000","last_became_active":"2024-05-28T06:18:09.640751+0000","last_became_peered":"2024-05-28T06:18:09.640751+0000","last_unstale":"2024-05-28T06:18:09.646406+0000","last_undegraded":"2024-05-28T06:18:09.646406+0000","last_fullsized":"2024-05-28T06:18:09.646406+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:15:39.926872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-05-28T06:18:09.634397+0000","last_change":"2024-05-28T06:18:09.634397+0000","last_active":"2024-05-28T06:18:09.604901+0000","last_peered":"2024-05-28T06:18:09.604901+0000","last_clean":"2024-05-28T06:18:09.604901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:18:09.634397+0000","last_undegraded":"2024-05-28T06:18:09.634397+0000","last_fullsized":"2024-05-28T06:18:09.634397+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"88'1","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-28T06:18:10.687989+0000","last_change":"2024-05-28T06:18:09.649924+0000","last_active":"2024-05-28T06:18:10.687989+0000","last_peered":"2024-05-28T06:18:10.687989+0000","last_clean":"2024-05-28T06:18:10.687989+0000","last_became_active":"2024-05-28T06:18:09.649696+0000","last_became_peered":"2024-05-28T06:18:09.649696+0000","last_unstale":"2024-05-28T06:18:10.687989+0000","last_undegraded":"2024-05-28T06:18:10.687989+0000","last_fullsized":"2024-05-28T06:18:10.687989+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:10:32.947652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":303,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-28T06:18:10.663404+0000","last_change":"2024-05-28T06:04:01.039129+0000","last_active":"2024-05-28T06:18:10.663404+0000","last_peered":"2024-05-28T06:18:10.663404+0000","last_clean":"2024-05-28T06:18:10.663404+0000","last_became_active":"2024-05-28T06:04:01.033906+0000","last_became_peered":"2024-05-28T06:04:01.033906+0000","last_unstale":"2024-05-28T06:18:10.663404+0000","last_undegraded":"2024-05-28T06:18:10.663404+0000","last_fullsized":"2024-05-28T06:18:10.663404+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:03:59.864237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:03:59.864237+0000","last_clean_scrub_stamp":"2024-05-28T06:03:59.864237+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:17:36.440245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575021,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":122992,"kb_used_data":1640,"kb_used_omap":1,"kb_used_meta":121342,"kb_avail":94248848,"statfs":{"total":96636764160,"available":96510820352,"internally_reserved":0,"allocated":1679360,"data_stored":1057712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":124254666},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575023,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":122404,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":121342,"kb_avail":94249436,"statfs":{"total":96636764160,"available":96511422464,"internally_reserved":0,"allocated":1069056,"data_stored":460471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":124254662},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575021,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76576,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94295264,"statfs":{"total":96636764160,"available":96558350336,"internally_reserved":0,"allocated":1056768,"data_stored":454168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575021,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64084,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94307756,"statfs":{"total":96636764160,"available":96571142144,"internally_reserved":0,"allocated":1056768,"data_stored":454168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575023,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":122388,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":121342,"kb_avail":94249452,"statfs":{"total":96636764160,"available":96511438848,"internally_reserved":0,"allocated":1056768,"data_stored":454168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":124254663},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575022,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":122392,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":121342,"kb_avail":94249448,"statfs":{"total":96636764160,"available":96511434752,"internally_reserved":0,"allocated":1056768,"data_stored":454168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124254661},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575022,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":104340,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":103294,"kb_avail":94267500,"statfs":{"total":96636764160,"available":96529920000,"internally_reserved":0,"allocated":1056768,"data_stored":454168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105773509},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575021,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77156,"kb_used_data":1612,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94294684,"statfs":{"total":96636764160,"available":96557756416,"internally_reserved":0,"allocated":1650688,"data_stored":1044536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-28T06:18:15.593 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-28T06:18:15.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.837+0000 7fcd6ce00640 1 Processor -- start 2024-05-28T06:18:15.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.837+0000 7fcd6ce00640 1 -- start start 2024-05-28T06:18:15.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.838+0000 7fcd6ce00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcd680722d0 0x7fcd6816d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:15.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.838+0000 7fcd6ce00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd6806f590 con 0x7fcd68162d20 2024-05-28T06:18:15.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.838+0000 7fcd6ce00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd6806f870 con 0x7fcd68162940 2024-05-28T06:18:15.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.839+0000 7fcd6ce00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fcd680ad3d0 con 0x7fcd680722d0 2024-05-28T06:18:15.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.839+0000 7fcd66a00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fcd68162d20 0x7fcd68071bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:49952/0 (socket says 172.21.15.96:49952) 2024-05-28T06:18:15.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.839+0000 7fcd66a00640 1 -- 172.21.15.96:0/1919073313 learned_addr learned my addr 172.21.15.96:0/1919073313 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:18:15.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.839+0000 7fcd67e00640 1 --2- 172.21.15.96:0/1919073313 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcd680722d0 0x7fcd6816d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:15.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.839+0000 7fcd66000640 1 -- 172.21.15.96:0/1919073313 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1208023598 0 0) 0x7fcd6806f590 con 0x7fcd68162d20 2024-05-28T06:18:15.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.839+0000 7fcd66000640 1 -- 172.21.15.96:0/1919073313 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcd4c003520 con 0x7fcd68162d20 2024-05-28T06:18:15.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.840+0000 7fcd66000640 1 -- 172.21.15.96:0/1919073313 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1967445069 0 0) 0x7fcd4c003520 con 0x7fcd68162d20 2024-05-28T06:18:15.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.840+0000 7fcd66000640 1 -- 172.21.15.96:0/1919073313 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcd680722d0 msgr2=0x7fcd6816d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:18:15.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.840+0000 7fcd66000640 1 --2- 172.21.15.96:0/1919073313 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcd680722d0 0x7fcd6816d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:15.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.840+0000 7fcd66000640 1 -- 172.21.15.96:0/1919073313 >> v1:172.21.15.123:6789/0 conn(0x7fcd68162940 legacy=0x7fcd6806eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:15.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.840+0000 7fcd66000640 1 -- 172.21.15.96:0/1919073313 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd6806f210 con 0x7fcd68162d20 2024-05-28T06:18:15.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.840+0000 7fcd66000640 1 -- 172.21.15.96:0/1919073313 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7fcd58001670 con 0x7fcd68162d20 2024-05-28T06:18:15.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.840+0000 7fcd66000640 1 -- 172.21.15.96:0/1919073313 >> v1:172.21.15.96:6789/0 conn(0x7fcd68162d20 legacy=0x7fcd68071bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:15.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.840+0000 7fcd66000640 1 --2- 172.21.15.96:0/1919073313 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcd4c0039e0 0x7fcd4c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:15.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.840+0000 7fcd66000640 1 --2- 172.21.15.96:0/1919073313 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcd680722d0 0x7fcd4c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:15.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.840+0000 7fcd66000640 1 --2- 172.21.15.96:0/1919073313 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcd4c004d40 0x7fcd4c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:15.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.840+0000 7fcd66000640 1 -- 172.21.15.96:0/1919073313 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fcd680ad3d0 con 0x7fcd4c004d40 2024-05-28T06:18:15.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.840+0000 7fcd66000640 1 -- 172.21.15.96:0/1919073313 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fcd4c00bbb0 con 0x7fcd4c0039e0 2024-05-28T06:18:15.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.840+0000 7fcd66000640 1 -- 172.21.15.96:0/1919073313 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fcd4c00bdb0 con 0x7fcd680722d0 2024-05-28T06:18:15.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.840+0000 7fcd67e00640 1 --2- 172.21.15.96:0/1919073313 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcd4c004d40 0x7fcd4c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:15.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.841+0000 7fcd67e00640 1 -- 172.21.15.96:0/1919073313 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcd680722d0 msgr2=0x7fcd4c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:18:15.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.841+0000 7fcd67400640 1 --2- 172.21.15.96:0/1919073313 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcd4c0039e0 0x7fcd4c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:15.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.841+0000 7fcd67e00640 1 --2- 172.21.15.96:0/1919073313 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcd680722d0 0x7fcd4c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:15.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.841+0000 7fcd67e00640 1 -- 172.21.15.96:0/1919073313 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcd4c0039e0 msgr2=0x7fcd4c003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:18:15.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.841+0000 7fcd67e00640 1 --2- 172.21.15.96:0/1919073313 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcd4c0039e0 0x7fcd4c003e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:15.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.841+0000 7fcd67e00640 1 -- 172.21.15.96:0/1919073313 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd4c003520 con 0x7fcd4c004d40 2024-05-28T06:18:15.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.842+0000 7fcd67e00640 1 --2- 172.21.15.96:0/1919073313 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcd4c004d40 0x7fcd4c0051c0 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7fcd60009fc0 tx=0x7fcd4c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35ff40f83a08ac05 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:15.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.842+0000 7fcd66000640 1 -- 172.21.15.96:0/1919073313 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd60009c10 con 0x7fcd4c004d40 2024-05-28T06:18:15.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.842+0000 7fcd66000640 1 -- 172.21.15.96:0/1919073313 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd60036070 con 0x7fcd4c004d40 2024-05-28T06:18:15.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.842+0000 7fcd66000640 1 -- 172.21.15.96:0/1919073313 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd6002d070 con 0x7fcd4c004d40 2024-05-28T06:18:15.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.842+0000 7fcd6ce00640 1 -- 172.21.15.96:0/1919073313 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcd4c004d40 msgr2=0x7fcd4c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:15.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.842+0000 7fcd6ce00640 1 --2- 172.21.15.96:0/1919073313 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcd4c004d40 0x7fcd4c0051c0 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7fcd60009fc0 tx=0x7fcd4c005830 comp rx=0 tx=0).stop 2024-05-28T06:18:15.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.842+0000 7fcd67e00640 1 -- 172.21.15.96:0/1919073313 reap_dead start 2024-05-28T06:18:15.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.843+0000 7fcd6ce00640 1 -- 172.21.15.96:0/1919073313 shutdown_connections 2024-05-28T06:18:15.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.843+0000 7fcd6ce00640 1 -- 172.21.15.96:0/1919073313 >> 172.21.15.96:0/1919073313 conn(0x7fcd680b1fc0 msgr2=0x7fcd680b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:18:15.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.843+0000 7fcd6ce00640 1 -- 172.21.15.96:0/1919073313 shutdown_connections 2024-05-28T06:18:15.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.843+0000 7fcd6ce00640 1 -- 172.21.15.96:0/1919073313 wait complete. 2024-05-28T06:18:15.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.844+0000 7fcd6ce00640 1 Processor -- start 2024-05-28T06:18:15.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.844+0000 7fcd6ce00640 1 -- start start 2024-05-28T06:18:15.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.845+0000 7fcd6ce00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcd6809c150 0x7fcd681f8fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:15.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.845+0000 7fcd6ce00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcd680ccca0 0x7fcd6809c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:15.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.845+0000 7fcd6ce00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcd680ce2e0 0x7fcd6809ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:15.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.845+0000 7fcd6ce00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fcd680aed40 con 0x7fcd680ccca0 2024-05-28T06:18:15.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.845+0000 7fcd6ce00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fcd680aebc0 con 0x7fcd6809c150 2024-05-28T06:18:15.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.845+0000 7fcd6ce00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fcd680aeec0 con 0x7fcd680ce2e0 2024-05-28T06:18:15.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.846+0000 7fcd66a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcd680ccca0 0x7fcd6809c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:15.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.846+0000 7fcd66a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcd680ccca0 0x7fcd6809c530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48290/0 (socket says 172.21.15.96:48290) 2024-05-28T06:18:15.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.846+0000 7fcd66a00640 1 -- 172.21.15.96:0/3069187130 learned_addr learned my addr 172.21.15.96:0/3069187130 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:18:15.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.846+0000 7fcd67400640 1 --2- 172.21.15.96:0/3069187130 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcd6809c150 0x7fcd681f8fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:15.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.846+0000 7fcd67e00640 1 --2- 172.21.15.96:0/3069187130 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcd680ce2e0 0x7fcd6809ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:15.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.846+0000 7fcd66a00640 1 -- 172.21.15.96:0/3069187130 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcd680ce2e0 msgr2=0x7fcd6809ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:15.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.846+0000 7fcd66a00640 1 --2- 172.21.15.96:0/3069187130 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcd680ce2e0 0x7fcd6809ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:15.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.846+0000 7fcd66a00640 1 -- 172.21.15.96:0/3069187130 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcd6809c150 msgr2=0x7fcd681f8fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:15.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.846+0000 7fcd66a00640 1 --2- 172.21.15.96:0/3069187130 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcd6809c150 0x7fcd681f8fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:15.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.846+0000 7fcd66a00640 1 -- 172.21.15.96:0/3069187130 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd681f9510 con 0x7fcd680ccca0 2024-05-28T06:18:15.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.846+0000 7fcd67400640 1 --2- 172.21.15.96:0/3069187130 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcd6809c150 0x7fcd681f8fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:18:15.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.846+0000 7fcd66a00640 1 --2- 172.21.15.96:0/3069187130 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcd680ccca0 0x7fcd6809c530 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7fcd5800c660 tx=0x7fcd5800cb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:15.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.847+0000 7fcd5f400640 1 -- 172.21.15.96:0/3069187130 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd58018070 con 0x7fcd680ccca0 2024-05-28T06:18:15.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.847+0000 7fcd5f400640 1 -- 172.21.15.96:0/3069187130 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd58004ac0 con 0x7fcd680ccca0 2024-05-28T06:18:15.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.847+0000 7fcd5f400640 1 -- 172.21.15.96:0/3069187130 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd580070d0 con 0x7fcd680ccca0 2024-05-28T06:18:15.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.847+0000 7fcd6ce00640 1 -- 172.21.15.96:0/3069187130 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd680cd240 con 0x7fcd680ccca0 2024-05-28T06:18:15.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.847+0000 7fcd6ce00640 1 -- 172.21.15.96:0/3069187130 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd680cd800 con 0x7fcd680ccca0 2024-05-28T06:18:15.847 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.848+0000 7fcd6ce00640 1 -- 172.21.15.96:0/3069187130 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd680689e0 con 0x7fcd680ccca0 2024-05-28T06:18:15.847 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.849+0000 7fcd5f400640 1 -- 172.21.15.96:0/3069187130 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 30) ==== 92621+0+0 (secure 0 0 0) 0x7fcd58007270 con 0x7fcd680ccca0 2024-05-28T06:18:15.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.850+0000 7fcd5f400640 1 --2- 172.21.15.96:0/3069187130 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fcd4806da60 0x7fcd4806ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:15.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.851+0000 7fcd5f400640 1 -- 172.21.15.96:0/3069187130 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 5737+0+0 (secure 0 0 0) 0x7fcd5808a040 con 0x7fcd680ccca0 2024-05-28T06:18:15.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.851+0000 7fcd67400640 1 --2- 172.21.15.96:0/3069187130 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fcd4806da60 0x7fcd4806ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:15.849 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.852+0000 7fcd67400640 1 --2- 172.21.15.96:0/3069187130 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fcd4806da60 0x7fcd4806ff20 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fcd540016f0 tx=0x7fcd54009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:15.852 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:15.855+0000 7fcd5f400640 1 -- 172.21.15.96:0/3069187130 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fcd58059b70 con 0x7fcd680ccca0 2024-05-28T06:18:16.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.039+0000 7fcd6ce00640 1 -- 172.21.15.96:0/3069187130 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fcd68162130 con 0x7fcd4806da60 2024-05-28T06:18:16.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.058+0000 7fcd5f400640 1 -- 172.21.15.96:0/3069187130 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252091 (secure 0 0 0) 0x7fcd68162130 con 0x7fcd4806da60 2024-05-28T06:18:16.060 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:18:16.062 INFO:teuthology.orchestra.run.smithi096.stderr:dumped all 2024-05-28T06:18:16.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.062+0000 7fcd6ce00640 1 -- 172.21.15.96:0/3069187130 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fcd4806da60 msgr2=0x7fcd4806ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:16.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.062+0000 7fcd6ce00640 1 --2- 172.21.15.96:0/3069187130 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fcd4806da60 0x7fcd4806ff20 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fcd540016f0 tx=0x7fcd54009040 comp rx=0 tx=0).stop 2024-05-28T06:18:16.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.063+0000 7fcd6ce00640 1 -- 172.21.15.96:0/3069187130 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcd680ccca0 msgr2=0x7fcd6809c530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:16.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.063+0000 7fcd6ce00640 1 --2- 172.21.15.96:0/3069187130 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcd680ccca0 0x7fcd6809c530 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7fcd5800c660 tx=0x7fcd5800cb20 comp rx=0 tx=0).stop 2024-05-28T06:18:16.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.063+0000 7fcd6ce00640 1 -- 172.21.15.96:0/3069187130 shutdown_connections 2024-05-28T06:18:16.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.063+0000 7fcd6ce00640 1 --2- 172.21.15.96:0/3069187130 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fcd4806da60 0x7fcd4806ff20 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:16.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.063+0000 7fcd6ce00640 1 --2- 172.21.15.96:0/3069187130 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcd680ce2e0 0x7fcd6809ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:16.065 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.063+0000 7fcd6ce00640 1 --2- 172.21.15.96:0/3069187130 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcd680ccca0 0x7fcd6809c530 unknown :-1 s=CLOSED pgs=1417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:16.065 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.063+0000 7fcd6ce00640 1 --2- 172.21.15.96:0/3069187130 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcd6809c150 0x7fcd681f8fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:16.065 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.063+0000 7fcd6ce00640 1 -- 172.21.15.96:0/3069187130 >> 172.21.15.96:0/3069187130 conn(0x7fcd680b1fc0 msgr2=0x7fcd680afec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:18:16.065 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.063+0000 7fcd6ce00640 1 -- 172.21.15.96:0/3069187130 shutdown_connections 2024-05-28T06:18:16.065 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.063+0000 7fcd6ce00640 1 -- 172.21.15.96:0/3069187130 wait complete. 2024-05-28T06:18:16.079 INFO:teuthology.orchestra.run.smithi096.stdout:{"pg_ready":true,"pg_map":{"version":510,"stamp":"2024-05-28T06:18:15.863167+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":154,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":812628,"kb_used_data":9756,"kb_used_omap":12,"kb_used_meta":802739,"kb_avail":754162092,"statfs":{"total":773094113280,"available":772261982208,"internally_reserved":0,"allocated":9990144,"data_stored":4968044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":822005296},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.248409"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.416525+0000","last_change":"2024-05-28T06:18:10.671822+0000","last_active":"2024-05-28T06:18:12.416525+0000","last_peered":"2024-05-28T06:18:12.416525+0000","last_clean":"2024-05-28T06:18:12.416525+0000","last_became_active":"2024-05-28T06:18:10.671188+0000","last_became_peered":"2024-05-28T06:18:10.671188+0000","last_unstale":"2024-05-28T06:18:12.416525+0000","last_undegraded":"2024-05-28T06:18:12.416525+0000","last_fullsized":"2024-05-28T06:18:12.416525+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:40:59.653395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.676054+0000","last_change":"2024-05-28T06:18:10.673470+0000","last_active":"2024-05-28T06:18:11.676054+0000","last_peered":"2024-05-28T06:18:11.676054+0000","last_clean":"2024-05-28T06:18:11.676054+0000","last_became_active":"2024-05-28T06:18:10.673112+0000","last_became_peered":"2024-05-28T06:18:10.673112+0000","last_unstale":"2024-05-28T06:18:11.676054+0000","last_undegraded":"2024-05-28T06:18:11.676054+0000","last_fullsized":"2024-05-28T06:18:11.676054+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:50:18.056574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-28T06:18:10.688685+0000","last_change":"2024-05-28T06:18:10.684086+0000","last_active":"2024-05-28T06:18:10.688685+0000","last_peered":"2024-05-28T06:18:10.688685+0000","last_clean":"2024-05-28T06:18:10.688685+0000","last_became_active":"2024-05-28T06:18:10.683345+0000","last_became_peered":"2024-05-28T06:18:10.683345+0000","last_unstale":"2024-05-28T06:18:10.688685+0000","last_undegraded":"2024-05-28T06:18:10.688685+0000","last_fullsized":"2024-05-28T06:18:10.688685+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:36:04.760890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.417075+0000","last_change":"2024-05-28T06:18:10.667705+0000","last_active":"2024-05-28T06:18:12.417075+0000","last_peered":"2024-05-28T06:18:12.417075+0000","last_clean":"2024-05-28T06:18:12.417075+0000","last_became_active":"2024-05-28T06:18:10.667442+0000","last_became_peered":"2024-05-28T06:18:10.667442+0000","last_unstale":"2024-05-28T06:18:12.417075+0000","last_undegraded":"2024-05-28T06:18:12.417075+0000","last_fullsized":"2024-05-28T06:18:12.417075+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:51:19.311871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.669932+0000","last_change":"2024-05-28T06:18:10.660951+0000","last_active":"2024-05-28T06:18:11.669932+0000","last_peered":"2024-05-28T06:18:11.669932+0000","last_clean":"2024-05-28T06:18:11.669932+0000","last_became_active":"2024-05-28T06:18:10.660437+0000","last_became_peered":"2024-05-28T06:18:10.660437+0000","last_unstale":"2024-05-28T06:18:11.669932+0000","last_undegraded":"2024-05-28T06:18:11.669932+0000","last_fullsized":"2024-05-28T06:18:11.669932+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:03:21.839812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.419909+0000","last_change":"2024-05-28T06:18:10.680300+0000","last_active":"2024-05-28T06:18:12.419909+0000","last_peered":"2024-05-28T06:18:12.419909+0000","last_clean":"2024-05-28T06:18:12.419909+0000","last_became_active":"2024-05-28T06:18:10.679570+0000","last_became_peered":"2024-05-28T06:18:10.679570+0000","last_unstale":"2024-05-28T06:18:12.419909+0000","last_undegraded":"2024-05-28T06:18:12.419909+0000","last_fullsized":"2024-05-28T06:18:12.419909+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:10:10.001167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.415870+0000","last_change":"2024-05-28T06:18:10.670026+0000","last_active":"2024-05-28T06:18:12.415870+0000","last_peered":"2024-05-28T06:18:12.415870+0000","last_clean":"2024-05-28T06:18:12.415870+0000","last_became_active":"2024-05-28T06:18:10.669644+0000","last_became_peered":"2024-05-28T06:18:10.669644+0000","last_unstale":"2024-05-28T06:18:12.415870+0000","last_undegraded":"2024-05-28T06:18:12.415870+0000","last_fullsized":"2024-05-28T06:18:12.415870+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:04:19.667824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.418225+0000","last_change":"2024-05-28T06:18:10.680190+0000","last_active":"2024-05-28T06:18:12.418225+0000","last_peered":"2024-05-28T06:18:12.418225+0000","last_clean":"2024-05-28T06:18:12.418225+0000","last_became_active":"2024-05-28T06:18:10.679764+0000","last_became_peered":"2024-05-28T06:18:10.679764+0000","last_unstale":"2024-05-28T06:18:12.418225+0000","last_undegraded":"2024-05-28T06:18:12.418225+0000","last_fullsized":"2024-05-28T06:18:12.418225+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:43:28.316407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.420071+0000","last_change":"2024-05-28T06:18:10.685588+0000","last_active":"2024-05-28T06:18:12.420071+0000","last_peered":"2024-05-28T06:18:12.420071+0000","last_clean":"2024-05-28T06:18:12.420071+0000","last_became_active":"2024-05-28T06:18:10.685361+0000","last_became_peered":"2024-05-28T06:18:10.685361+0000","last_unstale":"2024-05-28T06:18:12.420071+0000","last_undegraded":"2024-05-28T06:18:12.420071+0000","last_fullsized":"2024-05-28T06:18:12.420071+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:40:59.653395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.417249+0000","last_change":"2024-05-28T06:18:10.679458+0000","last_active":"2024-05-28T06:18:12.417249+0000","last_peered":"2024-05-28T06:18:12.417249+0000","last_clean":"2024-05-28T06:18:12.417249+0000","last_became_active":"2024-05-28T06:18:10.679143+0000","last_became_peered":"2024-05-28T06:18:10.679143+0000","last_unstale":"2024-05-28T06:18:12.417249+0000","last_undegraded":"2024-05-28T06:18:12.417249+0000","last_fullsized":"2024-05-28T06:18:12.417249+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:42:12.312828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.675913+0000","last_change":"2024-05-28T06:18:10.675557+0000","last_active":"2024-05-28T06:18:11.675913+0000","last_peered":"2024-05-28T06:18:11.675913+0000","last_clean":"2024-05-28T06:18:11.675913+0000","last_became_active":"2024-05-28T06:18:10.675292+0000","last_became_peered":"2024-05-28T06:18:10.675292+0000","last_unstale":"2024-05-28T06:18:11.675913+0000","last_undegraded":"2024-05-28T06:18:11.675913+0000","last_fullsized":"2024-05-28T06:18:11.675913+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:46:26.001440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.416176+0000","last_change":"2024-05-28T06:18:10.677581+0000","last_active":"2024-05-28T06:18:12.416176+0000","last_peered":"2024-05-28T06:18:12.416176+0000","last_clean":"2024-05-28T06:18:12.416176+0000","last_became_active":"2024-05-28T06:18:10.675476+0000","last_became_peered":"2024-05-28T06:18:10.675476+0000","last_unstale":"2024-05-28T06:18:12.416176+0000","last_undegraded":"2024-05-28T06:18:12.416176+0000","last_fullsized":"2024-05-28T06:18:12.416176+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T18:13:08.105068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-28T06:18:10.676482+0000","last_change":"2024-05-28T06:18:10.673510+0000","last_active":"2024-05-28T06:18:10.676482+0000","last_peered":"2024-05-28T06:18:10.676482+0000","last_clean":"2024-05-28T06:18:10.676482+0000","last_became_active":"2024-05-28T06:18:10.673194+0000","last_became_peered":"2024-05-28T06:18:10.673194+0000","last_unstale":"2024-05-28T06:18:10.676482+0000","last_undegraded":"2024-05-28T06:18:10.676482+0000","last_fullsized":"2024-05-28T06:18:10.676482+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:10:10.001167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.416894+0000","last_change":"2024-05-28T06:18:10.676724+0000","last_active":"2024-05-28T06:18:12.416894+0000","last_peered":"2024-05-28T06:18:12.416894+0000","last_clean":"2024-05-28T06:18:12.416894+0000","last_became_active":"2024-05-28T06:18:10.676457+0000","last_became_peered":"2024-05-28T06:18:10.676457+0000","last_unstale":"2024-05-28T06:18:12.416894+0000","last_undegraded":"2024-05-28T06:18:12.416894+0000","last_fullsized":"2024-05-28T06:18:12.416894+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:50:29.852024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.415556+0000","last_change":"2024-05-28T06:18:10.667679+0000","last_active":"2024-05-28T06:18:12.415556+0000","last_peered":"2024-05-28T06:18:12.415556+0000","last_clean":"2024-05-28T06:18:12.415556+0000","last_became_active":"2024-05-28T06:18:10.667355+0000","last_became_peered":"2024-05-28T06:18:10.667355+0000","last_unstale":"2024-05-28T06:18:12.415556+0000","last_undegraded":"2024-05-28T06:18:12.415556+0000","last_fullsized":"2024-05-28T06:18:12.415556+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:57:41.516876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.669444+0000","last_change":"2024-05-28T06:18:10.676070+0000","last_active":"2024-05-28T06:18:11.669444+0000","last_peered":"2024-05-28T06:18:11.669444+0000","last_clean":"2024-05-28T06:18:11.669444+0000","last_became_active":"2024-05-28T06:18:10.675759+0000","last_became_peered":"2024-05-28T06:18:10.675759+0000","last_unstale":"2024-05-28T06:18:11.669444+0000","last_undegraded":"2024-05-28T06:18:11.669444+0000","last_fullsized":"2024-05-28T06:18:11.669444+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:19:46.201909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.670757+0000","last_change":"2024-05-28T06:18:10.655314+0000","last_active":"2024-05-28T06:18:11.670757+0000","last_peered":"2024-05-28T06:18:11.670757+0000","last_clean":"2024-05-28T06:18:11.670757+0000","last_became_active":"2024-05-28T06:18:10.654580+0000","last_became_peered":"2024-05-28T06:18:10.654580+0000","last_unstale":"2024-05-28T06:18:11.670757+0000","last_undegraded":"2024-05-28T06:18:11.670757+0000","last_fullsized":"2024-05-28T06:18:11.670757+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:10:10.001167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.417262+0000","last_change":"2024-05-28T06:18:10.670459+0000","last_active":"2024-05-28T06:18:12.417262+0000","last_peered":"2024-05-28T06:18:12.417262+0000","last_clean":"2024-05-28T06:18:12.417262+0000","last_became_active":"2024-05-28T06:18:10.670119+0000","last_became_peered":"2024-05-28T06:18:10.670119+0000","last_unstale":"2024-05-28T06:18:12.417262+0000","last_undegraded":"2024-05-28T06:18:12.417262+0000","last_fullsized":"2024-05-28T06:18:12.417262+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:09:41.395379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.668772+0000","last_change":"2024-05-28T06:18:10.662056+0000","last_active":"2024-05-28T06:18:11.668772+0000","last_peered":"2024-05-28T06:18:11.668772+0000","last_clean":"2024-05-28T06:18:11.668772+0000","last_became_active":"2024-05-28T06:18:10.661589+0000","last_became_peered":"2024-05-28T06:18:10.661589+0000","last_unstale":"2024-05-28T06:18:11.668772+0000","last_undegraded":"2024-05-28T06:18:11.668772+0000","last_fullsized":"2024-05-28T06:18:11.668772+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:52:52.857149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.417036+0000","last_change":"2024-05-28T06:18:10.678748+0000","last_active":"2024-05-28T06:18:12.417036+0000","last_peered":"2024-05-28T06:18:12.417036+0000","last_clean":"2024-05-28T06:18:12.417036+0000","last_became_active":"2024-05-28T06:18:10.678444+0000","last_became_peered":"2024-05-28T06:18:10.678444+0000","last_unstale":"2024-05-28T06:18:12.417036+0000","last_undegraded":"2024-05-28T06:18:12.417036+0000","last_fullsized":"2024-05-28T06:18:12.417036+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:30:00.901425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.416302+0000","last_change":"2024-05-28T06:18:10.678968+0000","last_active":"2024-05-28T06:18:12.416302+0000","last_peered":"2024-05-28T06:18:12.416302+0000","last_clean":"2024-05-28T06:18:12.416302+0000","last_became_active":"2024-05-28T06:18:10.678693+0000","last_became_peered":"2024-05-28T06:18:10.678693+0000","last_unstale":"2024-05-28T06:18:12.416302+0000","last_undegraded":"2024-05-28T06:18:12.416302+0000","last_fullsized":"2024-05-28T06:18:12.416302+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:24:00.461410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.675477+0000","last_change":"2024-05-28T06:18:10.688872+0000","last_active":"2024-05-28T06:18:11.675477+0000","last_peered":"2024-05-28T06:18:11.675477+0000","last_clean":"2024-05-28T06:18:11.675477+0000","last_became_active":"2024-05-28T06:18:10.688588+0000","last_became_peered":"2024-05-28T06:18:10.688588+0000","last_unstale":"2024-05-28T06:18:11.675477+0000","last_undegraded":"2024-05-28T06:18:11.675477+0000","last_fullsized":"2024-05-28T06:18:11.675477+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:58:16.081516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.415595+0000","last_change":"2024-05-28T06:18:10.683026+0000","last_active":"2024-05-28T06:18:12.415595+0000","last_peered":"2024-05-28T06:18:12.415595+0000","last_clean":"2024-05-28T06:18:12.415595+0000","last_became_active":"2024-05-28T06:18:10.682741+0000","last_became_peered":"2024-05-28T06:18:10.682741+0000","last_unstale":"2024-05-28T06:18:12.415595+0000","last_undegraded":"2024-05-28T06:18:12.415595+0000","last_fullsized":"2024-05-28T06:18:12.415595+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:43:28.316407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.415588+0000","last_change":"2024-05-28T06:18:10.665505+0000","last_active":"2024-05-28T06:18:12.415588+0000","last_peered":"2024-05-28T06:18:12.415588+0000","last_clean":"2024-05-28T06:18:12.415588+0000","last_became_active":"2024-05-28T06:18:10.665204+0000","last_became_peered":"2024-05-28T06:18:10.665204+0000","last_unstale":"2024-05-28T06:18:12.415588+0000","last_undegraded":"2024-05-28T06:18:12.415588+0000","last_fullsized":"2024-05-28T06:18:12.415588+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:13:16.431201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.676140+0000","last_change":"2024-05-28T06:18:10.684581+0000","last_active":"2024-05-28T06:18:11.676140+0000","last_peered":"2024-05-28T06:18:11.676140+0000","last_clean":"2024-05-28T06:18:11.676140+0000","last_became_active":"2024-05-28T06:18:10.684250+0000","last_became_peered":"2024-05-28T06:18:10.684250+0000","last_unstale":"2024-05-28T06:18:11.676140+0000","last_undegraded":"2024-05-28T06:18:11.676140+0000","last_fullsized":"2024-05-28T06:18:11.676140+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:32:36.064344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-28T06:18:10.688666+0000","last_change":"2024-05-28T06:18:10.682743+0000","last_active":"2024-05-28T06:18:10.688666+0000","last_peered":"2024-05-28T06:18:10.688666+0000","last_clean":"2024-05-28T06:18:10.688666+0000","last_became_active":"2024-05-28T06:18:10.682451+0000","last_became_peered":"2024-05-28T06:18:10.682451+0000","last_unstale":"2024-05-28T06:18:10.688666+0000","last_undegraded":"2024-05-28T06:18:10.688666+0000","last_fullsized":"2024-05-28T06:18:10.688666+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:38:26.397686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.670303+0000","last_change":"2024-05-28T06:18:10.661976+0000","last_active":"2024-05-28T06:18:11.670303+0000","last_peered":"2024-05-28T06:18:11.670303+0000","last_clean":"2024-05-28T06:18:11.670303+0000","last_became_active":"2024-05-28T06:18:10.661460+0000","last_became_peered":"2024-05-28T06:18:10.661460+0000","last_unstale":"2024-05-28T06:18:11.670303+0000","last_undegraded":"2024-05-28T06:18:11.670303+0000","last_fullsized":"2024-05-28T06:18:11.670303+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:30:38.196353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.417151+0000","last_change":"2024-05-28T06:18:10.666860+0000","last_active":"2024-05-28T06:18:12.417151+0000","last_peered":"2024-05-28T06:18:12.417151+0000","last_clean":"2024-05-28T06:18:12.417151+0000","last_became_active":"2024-05-28T06:18:10.666305+0000","last_became_peered":"2024-05-28T06:18:10.666305+0000","last_unstale":"2024-05-28T06:18:12.417151+0000","last_undegraded":"2024-05-28T06:18:12.417151+0000","last_fullsized":"2024-05-28T06:18:12.417151+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:52:52.857149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.676511+0000","last_change":"2024-05-28T06:18:10.668290+0000","last_active":"2024-05-28T06:18:11.676511+0000","last_peered":"2024-05-28T06:18:11.676511+0000","last_clean":"2024-05-28T06:18:11.676511+0000","last_became_active":"2024-05-28T06:18:10.666561+0000","last_became_peered":"2024-05-28T06:18:10.666561+0000","last_unstale":"2024-05-28T06:18:11.676511+0000","last_undegraded":"2024-05-28T06:18:11.676511+0000","last_fullsized":"2024-05-28T06:18:11.676511+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:24:17.156838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.675494+0000","last_change":"2024-05-28T06:18:10.675895+0000","last_active":"2024-05-28T06:18:11.675494+0000","last_peered":"2024-05-28T06:18:11.675494+0000","last_clean":"2024-05-28T06:18:11.675494+0000","last_became_active":"2024-05-28T06:18:10.675623+0000","last_became_peered":"2024-05-28T06:18:10.675623+0000","last_unstale":"2024-05-28T06:18:11.675494+0000","last_undegraded":"2024-05-28T06:18:11.675494+0000","last_fullsized":"2024-05-28T06:18:11.675494+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:17:45.643378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.669373+0000","last_change":"2024-05-28T06:18:10.667388+0000","last_active":"2024-05-28T06:18:11.669373+0000","last_peered":"2024-05-28T06:18:11.669373+0000","last_clean":"2024-05-28T06:18:11.669373+0000","last_became_active":"2024-05-28T06:18:10.667062+0000","last_became_peered":"2024-05-28T06:18:10.667062+0000","last_unstale":"2024-05-28T06:18:11.669373+0000","last_undegraded":"2024-05-28T06:18:11.669373+0000","last_fullsized":"2024-05-28T06:18:11.669373+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:20:23.727599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.415807+0000","last_change":"2024-05-28T06:18:10.676303+0000","last_active":"2024-05-28T06:18:12.415807+0000","last_peered":"2024-05-28T06:18:12.415807+0000","last_clean":"2024-05-28T06:18:12.415807+0000","last_became_active":"2024-05-28T06:18:10.675998+0000","last_became_peered":"2024-05-28T06:18:10.675998+0000","last_unstale":"2024-05-28T06:18:12.415807+0000","last_undegraded":"2024-05-28T06:18:12.415807+0000","last_fullsized":"2024-05-28T06:18:12.415807+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:10:10.001167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.670796+0000","last_change":"2024-05-28T06:18:10.671446+0000","last_active":"2024-05-28T06:18:11.670796+0000","last_peered":"2024-05-28T06:18:11.670796+0000","last_clean":"2024-05-28T06:18:11.670796+0000","last_became_active":"2024-05-28T06:18:10.671173+0000","last_became_peered":"2024-05-28T06:18:10.671173+0000","last_unstale":"2024-05-28T06:18:11.670796+0000","last_undegraded":"2024-05-28T06:18:11.670796+0000","last_fullsized":"2024-05-28T06:18:11.670796+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:35:48.065461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"88'4","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.417211+0000","last_change":"2024-05-28T06:18:09.643542+0000","last_active":"2024-05-28T06:18:12.417211+0000","last_peered":"2024-05-28T06:18:12.417211+0000","last_clean":"2024-05-28T06:18:12.417211+0000","last_became_active":"2024-05-28T06:18:09.641808+0000","last_became_peered":"2024-05-28T06:18:09.641808+0000","last_unstale":"2024-05-28T06:18:12.417211+0000","last_undegraded":"2024-05-28T06:18:12.417211+0000","last_fullsized":"2024-05-28T06:18:12.417211+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:50:32.547130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.416707+0000","last_change":"2024-05-28T06:18:10.682953+0000","last_active":"2024-05-28T06:18:12.416707+0000","last_peered":"2024-05-28T06:18:12.416707+0000","last_clean":"2024-05-28T06:18:12.416707+0000","last_became_active":"2024-05-28T06:18:10.682621+0000","last_became_peered":"2024-05-28T06:18:10.682621+0000","last_unstale":"2024-05-28T06:18:12.416707+0000","last_undegraded":"2024-05-28T06:18:12.416707+0000","last_fullsized":"2024-05-28T06:18:12.416707+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:38:26.397686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"88'1","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.671160+0000","last_change":"2024-05-28T06:18:09.647214+0000","last_active":"2024-05-28T06:18:11.671160+0000","last_peered":"2024-05-28T06:18:11.671160+0000","last_clean":"2024-05-28T06:18:11.671160+0000","last_became_active":"2024-05-28T06:18:09.646937+0000","last_became_peered":"2024-05-28T06:18:09.646937+0000","last_unstale":"2024-05-28T06:18:11.671160+0000","last_undegraded":"2024-05-28T06:18:11.671160+0000","last_fullsized":"2024-05-28T06:18:11.671160+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:45:34.943093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.417520+0000","last_change":"2024-05-28T06:18:10.681586+0000","last_active":"2024-05-28T06:18:12.417520+0000","last_peered":"2024-05-28T06:18:12.417520+0000","last_clean":"2024-05-28T06:18:12.417520+0000","last_became_active":"2024-05-28T06:18:10.681223+0000","last_became_peered":"2024-05-28T06:18:10.681223+0000","last_unstale":"2024-05-28T06:18:12.417520+0000","last_undegraded":"2024-05-28T06:18:12.417520+0000","last_fullsized":"2024-05-28T06:18:12.417520+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:17:36.803843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"88'2","reported_seq":16,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.668839+0000","last_change":"2024-05-28T06:18:09.646080+0000","last_active":"2024-05-28T06:18:11.668839+0000","last_peered":"2024-05-28T06:18:11.668839+0000","last_clean":"2024-05-28T06:18:11.668839+0000","last_became_active":"2024-05-28T06:18:09.645807+0000","last_became_peered":"2024-05-28T06:18:09.645807+0000","last_unstale":"2024-05-28T06:18:11.668839+0000","last_undegraded":"2024-05-28T06:18:11.668839+0000","last_fullsized":"2024-05-28T06:18:11.668839+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:30:37.181316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-28T06:18:10.680587+0000","last_change":"2024-05-28T06:18:10.677395+0000","last_active":"2024-05-28T06:18:10.680587+0000","last_peered":"2024-05-28T06:18:10.680587+0000","last_clean":"2024-05-28T06:18:10.680587+0000","last_became_active":"2024-05-28T06:18:10.677100+0000","last_became_peered":"2024-05-28T06:18:10.677100+0000","last_unstale":"2024-05-28T06:18:10.680587+0000","last_undegraded":"2024-05-28T06:18:10.680587+0000","last_fullsized":"2024-05-28T06:18:10.680587+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:35:48.065461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"88'2","reported_seq":16,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.668500+0000","last_change":"2024-05-28T06:18:09.643971+0000","last_active":"2024-05-28T06:18:11.668500+0000","last_peered":"2024-05-28T06:18:11.668500+0000","last_clean":"2024-05-28T06:18:11.668500+0000","last_became_active":"2024-05-28T06:18:09.643691+0000","last_became_peered":"2024-05-28T06:18:09.643691+0000","last_unstale":"2024-05-28T06:18:11.668500+0000","last_undegraded":"2024-05-28T06:18:11.668500+0000","last_fullsized":"2024-05-28T06:18:11.668500+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:03:20.824775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.675683+0000","last_change":"2024-05-28T06:18:10.667892+0000","last_active":"2024-05-28T06:18:11.675683+0000","last_peered":"2024-05-28T06:18:11.675683+0000","last_clean":"2024-05-28T06:18:11.675683+0000","last_became_active":"2024-05-28T06:18:10.667622+0000","last_became_peered":"2024-05-28T06:18:10.667622+0000","last_unstale":"2024-05-28T06:18:11.675683+0000","last_undegraded":"2024-05-28T06:18:11.675683+0000","last_fullsized":"2024-05-28T06:18:11.675683+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:15:40.941909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"88'1","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.670434+0000","last_change":"2024-05-28T06:18:09.645873+0000","last_active":"2024-05-28T06:18:11.670434+0000","last_peered":"2024-05-28T06:18:11.670434+0000","last_clean":"2024-05-28T06:18:11.670434+0000","last_became_active":"2024-05-28T06:18:09.645561+0000","last_became_peered":"2024-05-28T06:18:09.645561+0000","last_unstale":"2024-05-28T06:18:11.670434+0000","last_undegraded":"2024-05-28T06:18:11.670434+0000","last_fullsized":"2024-05-28T06:18:11.670434+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:33:21.519096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.670041+0000","last_change":"2024-05-28T06:18:10.653334+0000","last_active":"2024-05-28T06:18:11.670041+0000","last_peered":"2024-05-28T06:18:11.670041+0000","last_clean":"2024-05-28T06:18:11.670041+0000","last_became_active":"2024-05-28T06:18:10.652195+0000","last_became_peered":"2024-05-28T06:18:10.652195+0000","last_unstale":"2024-05-28T06:18:11.670041+0000","last_undegraded":"2024-05-28T06:18:11.670041+0000","last_fullsized":"2024-05-28T06:18:11.670041+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:10:33.962690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"88'1","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.669219+0000","last_change":"2024-05-28T06:18:09.641147+0000","last_active":"2024-05-28T06:18:11.669219+0000","last_peered":"2024-05-28T06:18:11.669219+0000","last_clean":"2024-05-28T06:18:11.669219+0000","last_became_active":"2024-05-28T06:18:09.640830+0000","last_became_peered":"2024-05-28T06:18:09.640830+0000","last_unstale":"2024-05-28T06:18:11.669219+0000","last_undegraded":"2024-05-28T06:18:11.669219+0000","last_fullsized":"2024-05-28T06:18:11.669219+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:44:26.345831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-28T06:18:10.688719+0000","last_change":"2024-05-28T06:18:10.684510+0000","last_active":"2024-05-28T06:18:10.688719+0000","last_peered":"2024-05-28T06:18:10.688719+0000","last_clean":"2024-05-28T06:18:10.688719+0000","last_became_active":"2024-05-28T06:18:10.684209+0000","last_became_peered":"2024-05-28T06:18:10.684209+0000","last_unstale":"2024-05-28T06:18:10.688719+0000","last_undegraded":"2024-05-28T06:18:10.688719+0000","last_fullsized":"2024-05-28T06:18:10.688719+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T18:13:08.105068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.667919+0000","last_change":"2024-05-28T06:18:09.647844+0000","last_active":"2024-05-28T06:18:11.667919+0000","last_peered":"2024-05-28T06:18:11.667919+0000","last_clean":"2024-05-28T06:18:11.667919+0000","last_became_active":"2024-05-28T06:18:09.647527+0000","last_became_peered":"2024-05-28T06:18:09.647527+0000","last_unstale":"2024-05-28T06:18:11.667919+0000","last_undegraded":"2024-05-28T06:18:11.667919+0000","last_fullsized":"2024-05-28T06:18:11.667919+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:20:22.712562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.416481+0000","last_change":"2024-05-28T06:18:10.667540+0000","last_active":"2024-05-28T06:18:12.416481+0000","last_peered":"2024-05-28T06:18:12.416481+0000","last_clean":"2024-05-28T06:18:12.416481+0000","last_became_active":"2024-05-28T06:18:10.667232+0000","last_became_peered":"2024-05-28T06:18:10.667232+0000","last_unstale":"2024-05-28T06:18:12.416481+0000","last_undegraded":"2024-05-28T06:18:12.416481+0000","last_fullsized":"2024-05-28T06:18:12.416481+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:45:48.725150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.675347+0000","last_change":"2024-05-28T06:18:09.648064+0000","last_active":"2024-05-28T06:18:11.675347+0000","last_peered":"2024-05-28T06:18:11.675347+0000","last_clean":"2024-05-28T06:18:11.675347+0000","last_became_active":"2024-05-28T06:18:09.647787+0000","last_became_peered":"2024-05-28T06:18:09.647787+0000","last_unstale":"2024-05-28T06:18:11.675347+0000","last_undegraded":"2024-05-28T06:18:11.675347+0000","last_fullsized":"2024-05-28T06:18:11.675347+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:26:13.635159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.418311+0000","last_change":"2024-05-28T06:18:10.674109+0000","last_active":"2024-05-28T06:18:12.418311+0000","last_peered":"2024-05-28T06:18:12.418311+0000","last_clean":"2024-05-28T06:18:12.418311+0000","last_became_active":"2024-05-28T06:18:10.673531+0000","last_became_peered":"2024-05-28T06:18:10.673531+0000","last_unstale":"2024-05-28T06:18:12.418311+0000","last_undegraded":"2024-05-28T06:18:12.418311+0000","last_fullsized":"2024-05-28T06:18:12.418311+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:15:33.611029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.417249+0000","last_change":"2024-05-28T06:18:09.643958+0000","last_active":"2024-05-28T06:18:12.417249+0000","last_peered":"2024-05-28T06:18:12.417249+0000","last_clean":"2024-05-28T06:18:12.417249+0000","last_became_active":"2024-05-28T06:18:09.643658+0000","last_became_peered":"2024-05-28T06:18:09.643658+0000","last_unstale":"2024-05-28T06:18:12.417249+0000","last_undegraded":"2024-05-28T06:18:12.417249+0000","last_fullsized":"2024-05-28T06:18:12.417249+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:37:17.604820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.420418+0000","last_change":"2024-05-28T06:18:10.680643+0000","last_active":"2024-05-28T06:18:12.420418+0000","last_peered":"2024-05-28T06:18:12.420418+0000","last_clean":"2024-05-28T06:18:12.420418+0000","last_became_active":"2024-05-28T06:18:10.677505+0000","last_became_peered":"2024-05-28T06:18:10.677505+0000","last_unstale":"2024-05-28T06:18:12.420418+0000","last_undegraded":"2024-05-28T06:18:12.420418+0000","last_fullsized":"2024-05-28T06:18:12.420418+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:36:04.760890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.417061+0000","last_change":"2024-05-28T06:18:09.645952+0000","last_active":"2024-05-28T06:18:12.417061+0000","last_peered":"2024-05-28T06:18:12.417061+0000","last_clean":"2024-05-28T06:18:12.417061+0000","last_became_active":"2024-05-28T06:18:09.644892+0000","last_became_peered":"2024-05-28T06:18:09.644892+0000","last_unstale":"2024-05-28T06:18:12.417061+0000","last_undegraded":"2024-05-28T06:18:12.417061+0000","last_fullsized":"2024-05-28T06:18:12.417061+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:48:38.824030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.668434+0000","last_change":"2024-05-28T06:18:10.678535+0000","last_active":"2024-05-28T06:18:11.668434+0000","last_peered":"2024-05-28T06:18:11.668434+0000","last_clean":"2024-05-28T06:18:11.668434+0000","last_became_active":"2024-05-28T06:18:10.678261+0000","last_became_peered":"2024-05-28T06:18:10.678261+0000","last_unstale":"2024-05-28T06:18:11.668434+0000","last_undegraded":"2024-05-28T06:18:11.668434+0000","last_fullsized":"2024-05-28T06:18:11.668434+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T18:13:08.105068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"88'3","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.416171+0000","last_change":"2024-05-28T06:18:09.648424+0000","last_active":"2024-05-28T06:18:12.416171+0000","last_peered":"2024-05-28T06:18:12.416171+0000","last_clean":"2024-05-28T06:18:12.416171+0000","last_became_active":"2024-05-28T06:18:09.648138+0000","last_became_peered":"2024-05-28T06:18:09.648138+0000","last_unstale":"2024-05-28T06:18:12.416171+0000","last_undegraded":"2024-05-28T06:18:12.416171+0000","last_fullsized":"2024-05-28T06:18:12.416171+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:33:21.519096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.670379+0000","last_change":"2024-05-28T06:18:10.666292+0000","last_active":"2024-05-28T06:18:11.670379+0000","last_peered":"2024-05-28T06:18:11.670379+0000","last_clean":"2024-05-28T06:18:11.670379+0000","last_became_active":"2024-05-28T06:18:10.665709+0000","last_became_peered":"2024-05-28T06:18:10.665709+0000","last_unstale":"2024-05-28T06:18:11.670379+0000","last_undegraded":"2024-05-28T06:18:11.670379+0000","last_fullsized":"2024-05-28T06:18:11.670379+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:43:28.316407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"88'6","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.419335+0000","last_change":"2024-05-28T06:18:09.645431+0000","last_active":"2024-05-28T06:18:12.419335+0000","last_peered":"2024-05-28T06:18:12.419335+0000","last_clean":"2024-05-28T06:18:12.419335+0000","last_became_active":"2024-05-28T06:18:09.645141+0000","last_became_peered":"2024-05-28T06:18:09.645141+0000","last_unstale":"2024-05-28T06:18:12.419335+0000","last_undegraded":"2024-05-28T06:18:12.419335+0000","last_fullsized":"2024-05-28T06:18:12.419335+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:33:46.549475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.668301+0000","last_change":"2024-05-28T06:18:10.677586+0000","last_active":"2024-05-28T06:18:11.668301+0000","last_peered":"2024-05-28T06:18:11.668301+0000","last_clean":"2024-05-28T06:18:11.668301+0000","last_became_active":"2024-05-28T06:18:10.677186+0000","last_became_peered":"2024-05-28T06:18:10.677186+0000","last_unstale":"2024-05-28T06:18:11.668301+0000","last_undegraded":"2024-05-28T06:18:11.668301+0000","last_fullsized":"2024-05-28T06:18:11.668301+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:36:04.760890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"88'2","reported_seq":16,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.675765+0000","last_change":"2024-05-28T06:18:09.647148+0000","last_active":"2024-05-28T06:18:11.675765+0000","last_peered":"2024-05-28T06:18:11.675765+0000","last_clean":"2024-05-28T06:18:11.675765+0000","last_became_active":"2024-05-28T06:18:09.646882+0000","last_became_peered":"2024-05-28T06:18:09.646882+0000","last_unstale":"2024-05-28T06:18:11.675765+0000","last_undegraded":"2024-05-28T06:18:11.675765+0000","last_fullsized":"2024-05-28T06:18:11.675765+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:17:12.740244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-28T06:18:10.688761+0000","last_change":"2024-05-28T06:18:10.687447+0000","last_active":"2024-05-28T06:18:10.688761+0000","last_peered":"2024-05-28T06:18:10.688761+0000","last_clean":"2024-05-28T06:18:10.688761+0000","last_became_active":"2024-05-28T06:18:10.687161+0000","last_became_peered":"2024-05-28T06:18:10.687161+0000","last_unstale":"2024-05-28T06:18:10.688761+0000","last_undegraded":"2024-05-28T06:18:10.688761+0000","last_fullsized":"2024-05-28T06:18:10.688761+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:52:52.857149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.417055+0000","last_change":"2024-05-28T06:18:09.647907+0000","last_active":"2024-05-28T06:18:12.417055+0000","last_peered":"2024-05-28T06:18:12.417055+0000","last_clean":"2024-05-28T06:18:12.417055+0000","last_became_active":"2024-05-28T06:18:09.647240+0000","last_became_peered":"2024-05-28T06:18:09.647240+0000","last_unstale":"2024-05-28T06:18:12.417055+0000","last_undegraded":"2024-05-28T06:18:12.417055+0000","last_fullsized":"2024-05-28T06:18:12.417055+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:21:28.556698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.675523+0000","last_change":"2024-05-28T06:18:10.670763+0000","last_active":"2024-05-28T06:18:11.675523+0000","last_peered":"2024-05-28T06:18:11.675523+0000","last_clean":"2024-05-28T06:18:11.675523+0000","last_became_active":"2024-05-28T06:18:10.670493+0000","last_became_peered":"2024-05-28T06:18:10.670493+0000","last_unstale":"2024-05-28T06:18:11.675523+0000","last_undegraded":"2024-05-28T06:18:11.675523+0000","last_fullsized":"2024-05-28T06:18:11.675523+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:23:19.474029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.419186+0000","last_change":"2024-05-28T06:18:09.645407+0000","last_active":"2024-05-28T06:18:12.419186+0000","last_peered":"2024-05-28T06:18:12.419186+0000","last_clean":"2024-05-28T06:18:12.419186+0000","last_became_active":"2024-05-28T06:18:09.644308+0000","last_became_peered":"2024-05-28T06:18:09.644308+0000","last_unstale":"2024-05-28T06:18:12.419186+0000","last_undegraded":"2024-05-28T06:18:12.419186+0000","last_fullsized":"2024-05-28T06:18:12.419186+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T18:02:58.935614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.675215+0000","last_change":"2024-05-28T06:18:10.674096+0000","last_active":"2024-05-28T06:18:11.675215+0000","last_peered":"2024-05-28T06:18:11.675215+0000","last_clean":"2024-05-28T06:18:11.675215+0000","last_became_active":"2024-05-28T06:18:10.673401+0000","last_became_peered":"2024-05-28T06:18:10.673401+0000","last_unstale":"2024-05-28T06:18:11.675215+0000","last_undegraded":"2024-05-28T06:18:11.675215+0000","last_fullsized":"2024-05-28T06:18:11.675215+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:29:01.599345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"88'8","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.667571+0000","last_change":"2024-05-28T06:18:09.643338+0000","last_active":"2024-05-28T06:18:11.667571+0000","last_peered":"2024-05-28T06:18:11.667571+0000","last_clean":"2024-05-28T06:18:11.667571+0000","last_became_active":"2024-05-28T06:18:09.643035+0000","last_became_peered":"2024-05-28T06:18:09.643035+0000","last_unstale":"2024-05-28T06:18:11.667571+0000","last_undegraded":"2024-05-28T06:18:11.667571+0000","last_fullsized":"2024-05-28T06:18:11.667571+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:50:23.564808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-28T06:18:10.676395+0000","last_change":"2024-05-28T06:18:10.673463+0000","last_active":"2024-05-28T06:18:10.676395+0000","last_peered":"2024-05-28T06:18:10.676395+0000","last_clean":"2024-05-28T06:18:10.676395+0000","last_became_active":"2024-05-28T06:18:10.673150+0000","last_became_peered":"2024-05-28T06:18:10.673150+0000","last_unstale":"2024-05-28T06:18:10.676395+0000","last_undegraded":"2024-05-28T06:18:10.676395+0000","last_fullsized":"2024-05-28T06:18:10.676395+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:43:28.316407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"88'1","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.417570+0000","last_change":"2024-05-28T06:18:09.638887+0000","last_active":"2024-05-28T06:18:12.417570+0000","last_peered":"2024-05-28T06:18:12.417570+0000","last_clean":"2024-05-28T06:18:12.417570+0000","last_became_active":"2024-05-28T06:18:09.638560+0000","last_became_peered":"2024-05-28T06:18:09.638560+0000","last_unstale":"2024-05-28T06:18:12.417570+0000","last_undegraded":"2024-05-28T06:18:12.417570+0000","last_fullsized":"2024-05-28T06:18:12.417570+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T18:02:58.935614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.416958+0000","last_change":"2024-05-28T06:18:10.672644+0000","last_active":"2024-05-28T06:18:12.416958+0000","last_peered":"2024-05-28T06:18:12.416958+0000","last_clean":"2024-05-28T06:18:12.416958+0000","last_became_active":"2024-05-28T06:18:10.672376+0000","last_became_peered":"2024-05-28T06:18:10.672376+0000","last_unstale":"2024-05-28T06:18:12.416958+0000","last_undegraded":"2024-05-28T06:18:12.416958+0000","last_fullsized":"2024-05-28T06:18:12.416958+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:36:04.760890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"88'2","reported_seq":16,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.668844+0000","last_change":"2024-05-28T06:18:09.647691+0000","last_active":"2024-05-28T06:18:11.668844+0000","last_peered":"2024-05-28T06:18:11.668844+0000","last_clean":"2024-05-28T06:18:11.668844+0000","last_became_active":"2024-05-28T06:18:09.647419+0000","last_became_peered":"2024-05-28T06:18:09.647419+0000","last_unstale":"2024-05-28T06:18:11.668844+0000","last_undegraded":"2024-05-28T06:18:11.668844+0000","last_fullsized":"2024-05-28T06:18:11.668844+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:43:27.301370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.670827+0000","last_change":"2024-05-28T06:18:10.660698+0000","last_active":"2024-05-28T06:18:11.670827+0000","last_peered":"2024-05-28T06:18:11.670827+0000","last_clean":"2024-05-28T06:18:11.670827+0000","last_became_active":"2024-05-28T06:18:10.660019+0000","last_became_peered":"2024-05-28T06:18:10.660019+0000","last_unstale":"2024-05-28T06:18:11.670827+0000","last_undegraded":"2024-05-28T06:18:11.670827+0000","last_fullsized":"2024-05-28T06:18:11.670827+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:40:59.653395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.668528+0000","last_change":"2024-05-28T06:18:09.649938+0000","last_active":"2024-05-28T06:18:11.668528+0000","last_peered":"2024-05-28T06:18:11.668528+0000","last_clean":"2024-05-28T06:18:11.668528+0000","last_became_active":"2024-05-28T06:18:09.649011+0000","last_became_peered":"2024-05-28T06:18:09.649011+0000","last_unstale":"2024-05-28T06:18:11.668528+0000","last_undegraded":"2024-05-28T06:18:11.668528+0000","last_fullsized":"2024-05-28T06:18:11.668528+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:10:32.947652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.417302+0000","last_change":"2024-05-28T06:18:10.668617+0000","last_active":"2024-05-28T06:18:12.417302+0000","last_peered":"2024-05-28T06:18:12.417302+0000","last_clean":"2024-05-28T06:18:12.417302+0000","last_became_active":"2024-05-28T06:18:10.668324+0000","last_became_peered":"2024-05-28T06:18:10.668324+0000","last_unstale":"2024-05-28T06:18:12.417302+0000","last_undegraded":"2024-05-28T06:18:12.417302+0000","last_fullsized":"2024-05-28T06:18:12.417302+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:07:04.778166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"88'2","reported_seq":16,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.417711+0000","last_change":"2024-05-28T06:18:09.631399+0000","last_active":"2024-05-28T06:18:12.417711+0000","last_peered":"2024-05-28T06:18:12.417711+0000","last_clean":"2024-05-28T06:18:12.417711+0000","last_became_active":"2024-05-28T06:18:09.630708+0000","last_became_peered":"2024-05-28T06:18:09.630708+0000","last_unstale":"2024-05-28T06:18:12.417711+0000","last_undegraded":"2024-05-28T06:18:12.417711+0000","last_fullsized":"2024-05-28T06:18:12.417711+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:19:09.598059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.420330+0000","last_change":"2024-05-28T06:18:10.677756+0000","last_active":"2024-05-28T06:18:12.420330+0000","last_peered":"2024-05-28T06:18:12.420330+0000","last_clean":"2024-05-28T06:18:12.420330+0000","last_became_active":"2024-05-28T06:18:10.677433+0000","last_became_peered":"2024-05-28T06:18:10.677433+0000","last_unstale":"2024-05-28T06:18:12.420330+0000","last_undegraded":"2024-05-28T06:18:12.420330+0000","last_fullsized":"2024-05-28T06:18:12.420330+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:38:26.397686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.417551+0000","last_change":"2024-05-28T06:18:09.645491+0000","last_active":"2024-05-28T06:18:12.417551+0000","last_peered":"2024-05-28T06:18:12.417551+0000","last_clean":"2024-05-28T06:18:12.417551+0000","last_became_active":"2024-05-28T06:18:09.645198+0000","last_became_peered":"2024-05-28T06:18:09.645198+0000","last_unstale":"2024-05-28T06:18:12.417551+0000","last_undegraded":"2024-05-28T06:18:12.417551+0000","last_fullsized":"2024-05-28T06:18:12.417551+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:58:20.418372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.415731+0000","last_change":"2024-05-28T06:18:10.670797+0000","last_active":"2024-05-28T06:18:12.415731+0000","last_peered":"2024-05-28T06:18:12.415731+0000","last_clean":"2024-05-28T06:18:12.415731+0000","last_became_active":"2024-05-28T06:18:10.670524+0000","last_became_peered":"2024-05-28T06:18:10.670524+0000","last_unstale":"2024-05-28T06:18:12.415731+0000","last_undegraded":"2024-05-28T06:18:12.415731+0000","last_fullsized":"2024-05-28T06:18:12.415731+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:52:46.719566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.416008+0000","last_change":"2024-05-28T06:18:09.638348+0000","last_active":"2024-05-28T06:18:12.416008+0000","last_peered":"2024-05-28T06:18:12.416008+0000","last_clean":"2024-05-28T06:18:12.416008+0000","last_became_active":"2024-05-28T06:18:09.638008+0000","last_became_peered":"2024-05-28T06:18:09.638008+0000","last_unstale":"2024-05-28T06:18:12.416008+0000","last_undegraded":"2024-05-28T06:18:12.416008+0000","last_fullsized":"2024-05-28T06:18:12.416008+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:35:58.418977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.420424+0000","last_change":"2024-05-28T06:18:10.684680+0000","last_active":"2024-05-28T06:18:12.420424+0000","last_peered":"2024-05-28T06:18:12.420424+0000","last_clean":"2024-05-28T06:18:12.420424+0000","last_became_active":"2024-05-28T06:18:10.684197+0000","last_became_peered":"2024-05-28T06:18:10.684197+0000","last_unstale":"2024-05-28T06:18:12.420424+0000","last_undegraded":"2024-05-28T06:18:12.420424+0000","last_fullsized":"2024-05-28T06:18:12.420424+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T18:13:08.105068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.415239+0000","last_change":"2024-05-28T06:18:09.632017+0000","last_active":"2024-05-28T06:18:12.415239+0000","last_peered":"2024-05-28T06:18:12.415239+0000","last_clean":"2024-05-28T06:18:12.415239+0000","last_became_active":"2024-05-28T06:18:09.631229+0000","last_became_peered":"2024-05-28T06:18:09.631229+0000","last_unstale":"2024-05-28T06:18:12.415239+0000","last_undegraded":"2024-05-28T06:18:12.415239+0000","last_fullsized":"2024-05-28T06:18:12.415239+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:17:35.788806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.670828+0000","last_change":"2024-05-28T06:18:10.673882+0000","last_active":"2024-05-28T06:18:11.670828+0000","last_peered":"2024-05-28T06:18:11.670828+0000","last_clean":"2024-05-28T06:18:11.670828+0000","last_became_active":"2024-05-28T06:18:10.672435+0000","last_became_peered":"2024-05-28T06:18:10.672435+0000","last_unstale":"2024-05-28T06:18:11.670828+0000","last_undegraded":"2024-05-28T06:18:11.670828+0000","last_fullsized":"2024-05-28T06:18:11.670828+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:38:26.397686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"88'2","reported_seq":16,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.676296+0000","last_change":"2024-05-28T06:18:09.646324+0000","last_active":"2024-05-28T06:18:11.676296+0000","last_peered":"2024-05-28T06:18:11.676296+0000","last_clean":"2024-05-28T06:18:11.676296+0000","last_became_active":"2024-05-28T06:18:09.646028+0000","last_became_peered":"2024-05-28T06:18:09.646028+0000","last_unstale":"2024-05-28T06:18:11.676296+0000","last_undegraded":"2024-05-28T06:18:11.676296+0000","last_fullsized":"2024-05-28T06:18:11.676296+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:27:14.572603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-28T06:18:10.675571+0000","last_change":"2024-05-28T06:18:10.673086+0000","last_active":"2024-05-28T06:18:10.675571+0000","last_peered":"2024-05-28T06:18:10.675571+0000","last_clean":"2024-05-28T06:18:10.675571+0000","last_became_active":"2024-05-28T06:18:10.672555+0000","last_became_peered":"2024-05-28T06:18:10.672555+0000","last_unstale":"2024-05-28T06:18:10.675571+0000","last_undegraded":"2024-05-28T06:18:10.675571+0000","last_fullsized":"2024-05-28T06:18:10.675571+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:30:38.196353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"89'1","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.676181+0000","last_change":"2024-05-28T06:18:09.646621+0000","last_active":"2024-05-28T06:18:11.676181+0000","last_peered":"2024-05-28T06:18:11.676181+0000","last_clean":"2024-05-28T06:18:11.676181+0000","last_became_active":"2024-05-28T06:18:09.646289+0000","last_became_peered":"2024-05-28T06:18:09.646289+0000","last_unstale":"2024-05-28T06:18:11.676181+0000","last_undegraded":"2024-05-28T06:18:11.676181+0000","last_fullsized":"2024-05-28T06:18:11.676181+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:47:35.036840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.417590+0000","last_change":"2024-05-28T06:18:10.670457+0000","last_active":"2024-05-28T06:18:12.417590+0000","last_peered":"2024-05-28T06:18:12.417590+0000","last_clean":"2024-05-28T06:18:12.417590+0000","last_became_active":"2024-05-28T06:18:10.670145+0000","last_became_peered":"2024-05-28T06:18:10.670145+0000","last_unstale":"2024-05-28T06:18:12.417590+0000","last_undegraded":"2024-05-28T06:18:12.417590+0000","last_fullsized":"2024-05-28T06:18:12.417590+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:19:10.613096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.416460+0000","last_change":"2024-05-28T06:18:09.642139+0000","last_active":"2024-05-28T06:18:12.416460+0000","last_peered":"2024-05-28T06:18:12.416460+0000","last_clean":"2024-05-28T06:18:12.416460+0000","last_became_active":"2024-05-28T06:18:09.641858+0000","last_became_peered":"2024-05-28T06:18:09.641858+0000","last_unstale":"2024-05-28T06:18:12.416460+0000","last_undegraded":"2024-05-28T06:18:12.416460+0000","last_fullsized":"2024-05-28T06:18:12.416460+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:14:09.079098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.670999+0000","last_change":"2024-05-28T06:18:10.666029+0000","last_active":"2024-05-28T06:18:11.670999+0000","last_peered":"2024-05-28T06:18:11.670999+0000","last_clean":"2024-05-28T06:18:11.670999+0000","last_became_active":"2024-05-28T06:18:10.665253+0000","last_became_peered":"2024-05-28T06:18:10.665253+0000","last_unstale":"2024-05-28T06:18:11.670999+0000","last_undegraded":"2024-05-28T06:18:11.670999+0000","last_fullsized":"2024-05-28T06:18:11.670999+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:36:04.760890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.417769+0000","last_change":"2024-05-28T06:18:09.638402+0000","last_active":"2024-05-28T06:18:12.417769+0000","last_peered":"2024-05-28T06:18:12.417769+0000","last_clean":"2024-05-28T06:18:12.417769+0000","last_became_active":"2024-05-28T06:18:09.637801+0000","last_became_peered":"2024-05-28T06:18:09.637801+0000","last_unstale":"2024-05-28T06:18:12.417769+0000","last_undegraded":"2024-05-28T06:18:12.417769+0000","last_fullsized":"2024-05-28T06:18:12.417769+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:45:34.943093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.420016+0000","last_change":"2024-05-28T06:18:10.681723+0000","last_active":"2024-05-28T06:18:12.420016+0000","last_peered":"2024-05-28T06:18:12.420016+0000","last_clean":"2024-05-28T06:18:12.420016+0000","last_became_active":"2024-05-28T06:18:10.681397+0000","last_became_peered":"2024-05-28T06:18:10.681397+0000","last_unstale":"2024-05-28T06:18:12.420016+0000","last_undegraded":"2024-05-28T06:18:12.420016+0000","last_fullsized":"2024-05-28T06:18:12.420016+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:35:48.065461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.668379+0000","last_change":"2024-05-28T06:18:09.642946+0000","last_active":"2024-05-28T06:18:11.668379+0000","last_peered":"2024-05-28T06:18:11.668379+0000","last_clean":"2024-05-28T06:18:11.668379+0000","last_became_active":"2024-05-28T06:18:09.642546+0000","last_became_peered":"2024-05-28T06:18:09.642546+0000","last_unstale":"2024-05-28T06:18:11.668379+0000","last_undegraded":"2024-05-28T06:18:11.668379+0000","last_fullsized":"2024-05-28T06:18:11.668379+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:19:45.186872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.668346+0000","last_change":"2024-05-28T06:18:10.685239+0000","last_active":"2024-05-28T06:18:11.668346+0000","last_peered":"2024-05-28T06:18:11.668346+0000","last_clean":"2024-05-28T06:18:11.668346+0000","last_became_active":"2024-05-28T06:18:10.681969+0000","last_became_peered":"2024-05-28T06:18:10.681969+0000","last_unstale":"2024-05-28T06:18:11.668346+0000","last_undegraded":"2024-05-28T06:18:11.668346+0000","last_fullsized":"2024-05-28T06:18:11.668346+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:38:26.397686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.669511+0000","last_change":"2024-05-28T06:18:09.643305+0000","last_active":"2024-05-28T06:18:11.669511+0000","last_peered":"2024-05-28T06:18:11.669511+0000","last_clean":"2024-05-28T06:18:11.669511+0000","last_became_active":"2024-05-28T06:18:09.642959+0000","last_became_peered":"2024-05-28T06:18:09.642959+0000","last_unstale":"2024-05-28T06:18:11.669511+0000","last_undegraded":"2024-05-28T06:18:11.669511+0000","last_fullsized":"2024-05-28T06:18:11.669511+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:57:25.848771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-28T06:18:10.688312+0000","last_change":"2024-05-28T06:18:10.682613+0000","last_active":"2024-05-28T06:18:10.688312+0000","last_peered":"2024-05-28T06:18:10.688312+0000","last_clean":"2024-05-28T06:18:10.688312+0000","last_became_active":"2024-05-28T06:18:10.682019+0000","last_became_peered":"2024-05-28T06:18:10.682019+0000","last_unstale":"2024-05-28T06:18:10.688312+0000","last_undegraded":"2024-05-28T06:18:10.688312+0000","last_fullsized":"2024-05-28T06:18:10.688312+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:40:59.653395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.416028+0000","last_change":"2024-05-28T06:18:09.637819+0000","last_active":"2024-05-28T06:18:12.416028+0000","last_peered":"2024-05-28T06:18:12.416028+0000","last_clean":"2024-05-28T06:18:12.416028+0000","last_became_active":"2024-05-28T06:18:09.637503+0000","last_became_peered":"2024-05-28T06:18:09.637503+0000","last_unstale":"2024-05-28T06:18:12.416028+0000","last_undegraded":"2024-05-28T06:18:12.416028+0000","last_fullsized":"2024-05-28T06:18:12.416028+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:41:26.564391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.415971+0000","last_change":"2024-05-28T06:18:10.668846+0000","last_active":"2024-05-28T06:18:12.415971+0000","last_peered":"2024-05-28T06:18:12.415971+0000","last_clean":"2024-05-28T06:18:12.415971+0000","last_became_active":"2024-05-28T06:18:10.665081+0000","last_became_peered":"2024-05-28T06:18:10.665081+0000","last_unstale":"2024-05-28T06:18:12.415971+0000","last_undegraded":"2024-05-28T06:18:12.415971+0000","last_fullsized":"2024-05-28T06:18:12.415971+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:35:48.065461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.669169+0000","last_change":"2024-05-28T06:18:09.641057+0000","last_active":"2024-05-28T06:18:11.669169+0000","last_peered":"2024-05-28T06:18:11.669169+0000","last_clean":"2024-05-28T06:18:11.669169+0000","last_became_active":"2024-05-28T06:18:09.640751+0000","last_became_peered":"2024-05-28T06:18:09.640751+0000","last_unstale":"2024-05-28T06:18:11.669169+0000","last_undegraded":"2024-05-28T06:18:11.669169+0000","last_fullsized":"2024-05-28T06:18:11.669169+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:14:09.079098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.675977+0000","last_change":"2024-05-28T06:18:10.673680+0000","last_active":"2024-05-28T06:18:11.675977+0000","last_peered":"2024-05-28T06:18:11.675977+0000","last_clean":"2024-05-28T06:18:11.675977+0000","last_became_active":"2024-05-28T06:18:10.673261+0000","last_became_peered":"2024-05-28T06:18:10.673261+0000","last_unstale":"2024-05-28T06:18:11.675977+0000","last_undegraded":"2024-05-28T06:18:11.675977+0000","last_fullsized":"2024-05-28T06:18:11.675977+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:18:26.300329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"88'1","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-28T06:18:10.687989+0000","last_change":"2024-05-28T06:18:09.649924+0000","last_active":"2024-05-28T06:18:10.687989+0000","last_peered":"2024-05-28T06:18:10.687989+0000","last_clean":"2024-05-28T06:18:10.687989+0000","last_became_active":"2024-05-28T06:18:09.649696+0000","last_became_peered":"2024-05-28T06:18:09.649696+0000","last_unstale":"2024-05-28T06:18:10.687989+0000","last_undegraded":"2024-05-28T06:18:10.687989+0000","last_fullsized":"2024-05-28T06:18:10.687989+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:10:32.947652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":303,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-28T06:18:10.663404+0000","last_change":"2024-05-28T06:04:01.039129+0000","last_active":"2024-05-28T06:18:10.663404+0000","last_peered":"2024-05-28T06:18:10.663404+0000","last_clean":"2024-05-28T06:18:10.663404+0000","last_became_active":"2024-05-28T06:04:01.033906+0000","last_became_peered":"2024-05-28T06:04:01.033906+0000","last_unstale":"2024-05-28T06:18:10.663404+0000","last_undegraded":"2024-05-28T06:18:10.663404+0000","last_fullsized":"2024-05-28T06:18:10.663404+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:03:59.864237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:03:59.864237+0000","last_clean_scrub_stamp":"2024-05-28T06:03:59.864237+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:17:36.440245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575021,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":122992,"kb_used_data":1640,"kb_used_omap":1,"kb_used_meta":121342,"kb_avail":94248848,"statfs":{"total":96636764160,"available":96510820352,"internally_reserved":0,"allocated":1679360,"data_stored":1057712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":124254666},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575024,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":122440,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":121342,"kb_avail":94249400,"statfs":{"total":96636764160,"available":96511385600,"internally_reserved":0,"allocated":1105920,"data_stored":474296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":124254662},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575022,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76616,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94295224,"statfs":{"total":96636764160,"available":96558309376,"internally_reserved":0,"allocated":1110016,"data_stored":475236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575022,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64124,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94307716,"statfs":{"total":96636764160,"available":96571101184,"internally_reserved":0,"allocated":1093632,"data_stored":473602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575024,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":122440,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":121342,"kb_avail":94249400,"statfs":{"total":96636764160,"available":96511385600,"internally_reserved":0,"allocated":1105920,"data_stored":475214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":124254663},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575023,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":122432,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":121342,"kb_avail":94249408,"statfs":{"total":96636764160,"available":96511393792,"internally_reserved":0,"allocated":1097728,"data_stored":474184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124254661},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575023,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":104392,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":103294,"kb_avail":94267448,"statfs":{"total":96636764160,"available":96529866752,"internally_reserved":0,"allocated":1105920,"data_stored":473796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105773509},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575022,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77192,"kb_used_data":1652,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94294648,"statfs":{"total":96636764160,"available":96557719552,"internally_reserved":0,"allocated":1691648,"data_stored":1064004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-28T06:18:16.083 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-05-28T06:18:16.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.319+0000 7f1377200640 1 Processor -- start 2024-05-28T06:18:16.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.319+0000 7f1377200640 1 -- start start 2024-05-28T06:18:16.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.321+0000 7f1377200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f13780d0460 0x7f13780d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:16.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.321+0000 7f1377200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13780cd1a0 con 0x7f13780ccc30 2024-05-28T06:18:16.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.321+0000 7f1377200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13780cd480 con 0x7f137815c6b0 2024-05-28T06:18:16.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.321+0000 7f1377200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f137806e3a0 con 0x7f13780d0460 2024-05-28T06:18:16.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.321+0000 7f136fe00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f13780ccc30 0x7f137806dc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:49956/0 (socket says 172.21.15.96:49956) 2024-05-28T06:18:16.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.321+0000 7f136fe00640 1 -- 172.21.15.96:0/893321227 learned_addr learned my addr 172.21.15.96:0/893321227 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:18:16.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.321+0000 7f1376800640 1 --2- 172.21.15.96:0/893321227 >> v2:172.21.15.96:3300/0 conn(0x7f13780d0460 0x7f13780d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:16.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.322+0000 7f1376800640 1 -- 172.21.15.96:0/893321227 >> v1:172.21.15.123:6789/0 conn(0x7f137815c6b0 legacy=0x7f13780b3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:18:16.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.322+0000 7f1376800640 1 -- 172.21.15.96:0/893321227 >> v1:172.21.15.96:6789/0 conn(0x7f13780ccc30 legacy=0x7f137806dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:16.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.322+0000 7f1376800640 1 -- 172.21.15.96:0/893321227 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13780cd760 con 0x7f13780d0460 2024-05-28T06:18:16.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.323+0000 7f1376800640 1 --2- 172.21.15.96:0/893321227 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13780d0460 0x7f13780d08b0 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f1370009dc0 tx=0x7f137002a6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa48c6e116d272e7 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:16.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.323+0000 7f1375400640 1 -- 172.21.15.96:0/893321227 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1370007740 con 0x7f13780d0460 2024-05-28T06:18:16.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.323+0000 7f1375400640 1 -- 172.21.15.96:0/893321227 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13700078e0 con 0x7f13780d0460 2024-05-28T06:18:16.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.323+0000 7f1375400640 1 -- 172.21.15.96:0/893321227 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1370007cb0 con 0x7f13780d0460 2024-05-28T06:18:16.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.323+0000 7f1377200640 1 -- 172.21.15.96:0/893321227 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13780d0460 msgr2=0x7f13780d08b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:16.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.323+0000 7f1377200640 1 --2- 172.21.15.96:0/893321227 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13780d0460 0x7f13780d08b0 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f1370009dc0 tx=0x7f137002a6d0 comp rx=0 tx=0).stop 2024-05-28T06:18:16.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.324+0000 7f1377200640 1 -- 172.21.15.96:0/893321227 shutdown_connections 2024-05-28T06:18:16.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.324+0000 7f1377200640 1 --2- 172.21.15.96:0/893321227 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13780d0460 0x7f13780d08b0 unknown :-1 s=CLOSED pgs=1419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:16.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.324+0000 7f1377200640 1 -- 172.21.15.96:0/893321227 >> 172.21.15.96:0/893321227 conn(0x7f137806bae0 msgr2=0x7f137806bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:18:16.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.324+0000 7f1377200640 1 -- 172.21.15.96:0/893321227 shutdown_connections 2024-05-28T06:18:16.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.324+0000 7f1377200640 1 -- 172.21.15.96:0/893321227 wait complete. 2024-05-28T06:18:16.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.325+0000 7f1377200640 1 Processor -- start 2024-05-28T06:18:16.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.325+0000 7f1377200640 1 -- start start 2024-05-28T06:18:16.325 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.326+0000 7f1377200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f13780ccc30 0x7f13780ae690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:16.325 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.326+0000 7f1377200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13780d0460 0x7f13780ad130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:16.325 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.326+0000 7f1377200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f137815c6b0 0x7f137815aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:16.325 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.326+0000 7f1377200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f13780cba10 con 0x7f13780d0460 2024-05-28T06:18:16.325 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.326+0000 7f1377200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f13780d0df0 con 0x7f137815c6b0 2024-05-28T06:18:16.325 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.326+0000 7f1377200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f13780cbf30 con 0x7f13780ccc30 2024-05-28T06:18:16.325 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.327+0000 7f136fe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13780d0460 0x7f13780ad130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:16.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.327+0000 7f136fe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13780d0460 0x7f13780ad130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48310/0 (socket says 172.21.15.96:48310) 2024-05-28T06:18:16.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.327+0000 7f136fe00640 1 -- 172.21.15.96:0/2184775752 learned_addr learned my addr 172.21.15.96:0/2184775752 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:18:16.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.327+0000 7f136fe00640 1 -- 172.21.15.96:0/2184775752 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f13780ccc30 msgr2=0x7f13780ae690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:16.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.327+0000 7f1375e00640 1 --2- 172.21.15.96:0/2184775752 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f13780ccc30 0x7f13780ae690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:16.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.327+0000 7f1376800640 1 --2- 172.21.15.96:0/2184775752 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f137815c6b0 0x7f137815aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:16.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.327+0000 7f136fe00640 1 --2- 172.21.15.96:0/2184775752 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f13780ccc30 0x7f13780ae690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:16.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.327+0000 7f136fe00640 1 -- 172.21.15.96:0/2184775752 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f137815c6b0 msgr2=0x7f137815aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:16.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.327+0000 7f136fe00640 1 --2- 172.21.15.96:0/2184775752 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f137815c6b0 0x7f137815aa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:16.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.327+0000 7f136fe00640 1 -- 172.21.15.96:0/2184775752 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13781f4130 con 0x7f13780d0460 2024-05-28T06:18:16.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.327+0000 7f1375e00640 1 --2- 172.21.15.96:0/2184775752 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f13780ccc30 0x7f13780ae690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:18:16.327 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.327+0000 7f136fe00640 1 --2- 172.21.15.96:0/2184775752 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13780d0460 0x7f13780ad130 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f136000c400 tx=0x7f136000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:16.327 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.328+0000 7f136e000640 1 -- 172.21.15.96:0/2184775752 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1360017070 con 0x7f13780d0460 2024-05-28T06:18:16.327 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.328+0000 7f136e000640 1 -- 172.21.15.96:0/2184775752 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1360005ce0 con 0x7f13780d0460 2024-05-28T06:18:16.327 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.328+0000 7f136e000640 1 -- 172.21.15.96:0/2184775752 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13600072f0 con 0x7f13780d0460 2024-05-28T06:18:16.327 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.328+0000 7f1377200640 1 -- 172.21.15.96:0/2184775752 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13781f7bb0 con 0x7f13780d0460 2024-05-28T06:18:16.327 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.328+0000 7f1377200640 1 -- 172.21.15.96:0/2184775752 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13781f8170 con 0x7f13780d0460 2024-05-28T06:18:16.328 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.329+0000 7f1377200640 1 -- 172.21.15.96:0/2184775752 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13780689e0 con 0x7f13780d0460 2024-05-28T06:18:16.328 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.330+0000 7f136e000640 1 -- 172.21.15.96:0/2184775752 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 30) ==== 92621+0+0 (secure 0 0 0) 0x7f136001a070 con 0x7f13780d0460 2024-05-28T06:18:16.328 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.331+0000 7f136e000640 1 --2- 172.21.15.96:0/2184775752 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f135406da60 0x7f135406ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:16.328 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.331+0000 7f136e000640 1 -- 172.21.15.96:0/2184775752 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 5737+0+0 (secure 0 0 0) 0x7f1360089e40 con 0x7f13780d0460 2024-05-28T06:18:16.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.331+0000 7f1375e00640 1 --2- 172.21.15.96:0/2184775752 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f135406da60 0x7f135406ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:16.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.333+0000 7f1375e00640 1 --2- 172.21.15.96:0/2184775752 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f135406da60 0x7f135406ff20 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f13780ca380 tx=0x7f1368008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:16.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.335+0000 7f136e000640 1 -- 172.21.15.96:0/2184775752 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f13600599d0 con 0x7f13780d0460 2024-05-28T06:18:16.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.607+0000 7f1377200640 1 -- 172.21.15.96:0/2184775752 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7f137806cc30 con 0x7f13780d0460 2024-05-28T06:18:16.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.609+0000 7f136e000640 1 -- 172.21.15.96:0/2184775752 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1098 (secure 0 0 0) 0x7f13600592d0 con 0x7f13780d0460 2024-05-28T06:18:16.606 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:18:16.606 INFO:teuthology.orchestra.run.smithi096.stdout:{"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":870,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":89,"num_osds":8,"num_up_osds":8,"osd_up_since":1716876238,"num_in_osds":8,"osd_in_since":1716876229,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"creating+peering","count":55},{"state_name":"active+clean","count":42}],"num_pgs":97,"num_pools":3,"num_objects":3,"data_bytes":590438,"bytes_used":831827968,"bytes_avail":772262285312,"bytes_total":773094113280,"inactive_pgs_ratio":0.56701028347015381,"write_bytes_sec":194,"read_op_per_sec":0,"write_op_per_sec":0},"fsmap":{"epoch":52,"id":6,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":6,"rank":0,"name":"c","status":"up:active","gid":8103}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":13,"modified":"2024-05-28T06:15:57.778889+0000","services":{}},"progress_events":{}} 2024-05-28T06:18:16.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.611+0000 7f1377200640 1 -- 172.21.15.96:0/2184775752 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f135406da60 msgr2=0x7f135406ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:16.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.611+0000 7f1377200640 1 --2- 172.21.15.96:0/2184775752 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f135406da60 0x7f135406ff20 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f13780ca380 tx=0x7f1368008040 comp rx=0 tx=0).stop 2024-05-28T06:18:16.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.611+0000 7f1377200640 1 -- 172.21.15.96:0/2184775752 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13780d0460 msgr2=0x7f13780ad130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:16.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.611+0000 7f1377200640 1 --2- 172.21.15.96:0/2184775752 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13780d0460 0x7f13780ad130 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f136000c400 tx=0x7f136000c8c0 comp rx=0 tx=0).stop 2024-05-28T06:18:16.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.611+0000 7f1377200640 1 -- 172.21.15.96:0/2184775752 shutdown_connections 2024-05-28T06:18:16.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.611+0000 7f1377200640 1 --2- 172.21.15.96:0/2184775752 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f135406da60 0x7f135406ff20 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:16.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.611+0000 7f1377200640 1 --2- 172.21.15.96:0/2184775752 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f137815c6b0 0x7f137815aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:16.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.611+0000 7f1377200640 1 --2- 172.21.15.96:0/2184775752 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13780d0460 0x7f13780ad130 unknown :-1 s=CLOSED pgs=1420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:16.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.611+0000 7f1377200640 1 --2- 172.21.15.96:0/2184775752 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f13780ccc30 0x7f13780ae690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:16.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.611+0000 7f1377200640 1 -- 172.21.15.96:0/2184775752 >> 172.21.15.96:0/2184775752 conn(0x7f137806bae0 msgr2=0x7f137806bec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:18:16.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.612+0000 7f1377200640 1 -- 172.21.15.96:0/2184775752 shutdown_connections 2024-05-28T06:18:16.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.612+0000 7f1377200640 1 -- 172.21.15.96:0/2184775752 wait complete. 2024-05-28T06:18:16.628 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-05-28T06:18:16.628 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-28T06:18:16.826 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.827+0000 7f4e13e00640 1 Processor -- start 2024-05-28T06:18:16.826 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.827+0000 7f4e13e00640 1 -- start start 2024-05-28T06:18:16.826 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.828+0000 7f4e13e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4e14156b30 0x7f4e140af6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:16.826 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.828+0000 7f4e13e00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e14156f10 con 0x7f4e140ad940 2024-05-28T06:18:16.826 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.829+0000 7f4e13e00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e1406dda0 con 0x7f4e1406ee40 2024-05-28T06:18:16.827 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.829+0000 7f4e13e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f4e1406e080 con 0x7f4e14156b30 2024-05-28T06:18:16.827 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.829+0000 7f4e13400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f4e140ad940 0x7f4e140b32c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:49962/0 (socket says 172.21.15.96:49962) 2024-05-28T06:18:16.827 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.829+0000 7f4e13400640 1 -- 172.21.15.96:0/2233558739 learned_addr learned my addr 172.21.15.96:0/2233558739 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:18:16.828 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.829+0000 7f4e12a00640 1 --2- 172.21.15.96:0/2233558739 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4e14156b30 0x7f4e140af6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:16.828 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.829+0000 7f4e12000640 1 -- 172.21.15.96:0/2233558739 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3332154361 0 0) 0x7f4e14156f10 con 0x7f4e140ad940 2024-05-28T06:18:16.828 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.829+0000 7f4e12000640 1 -- 172.21.15.96:0/2233558739 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4df8003520 con 0x7f4e140ad940 2024-05-28T06:18:16.828 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.830+0000 7f4e12000640 1 -- 172.21.15.96:0/2233558739 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3697932018 0 0) 0x7f4df8003520 con 0x7f4e140ad940 2024-05-28T06:18:16.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.830+0000 7f4e12000640 1 -- 172.21.15.96:0/2233558739 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4e14156b30 msgr2=0x7f4e140af6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:18:16.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.830+0000 7f4e12000640 1 --2- 172.21.15.96:0/2233558739 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4e14156b30 0x7f4e140af6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:16.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.830+0000 7f4e12000640 1 -- 172.21.15.96:0/2233558739 >> v1:172.21.15.123:6789/0 conn(0x7f4e1406ee40 legacy=0x7f4e140ad230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:16.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.830+0000 7f4e12000640 1 -- 172.21.15.96:0/2233558739 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e1406e360 con 0x7f4e140ad940 2024-05-28T06:18:16.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.830+0000 7f4e12000640 1 -- 172.21.15.96:0/2233558739 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f4e0c0019f0 con 0x7f4e140ad940 2024-05-28T06:18:16.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.830+0000 7f4e12000640 1 -- 172.21.15.96:0/2233558739 >> v1:172.21.15.96:6789/0 conn(0x7f4e140ad940 legacy=0x7f4e140b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:16.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.830+0000 7f4e12000640 1 --2- 172.21.15.96:0/2233558739 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4df80039e0 0x7f4df8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:16.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.830+0000 7f4e12000640 1 --2- 172.21.15.96:0/2233558739 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4df8004800 0x7f4df8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:16.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.830+0000 7f4e12000640 1 --2- 172.21.15.96:0/2233558739 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4e14156b30 0x7f4df800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:16.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.830+0000 7f4e12000640 1 -- 172.21.15.96:0/2233558739 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4e1406e080 con 0x7f4df80039e0 2024-05-28T06:18:16.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.830+0000 7f4e12000640 1 -- 172.21.15.96:0/2233558739 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f4df80054f0 con 0x7f4df8004800 2024-05-28T06:18:16.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.830+0000 7f4e12000640 1 -- 172.21.15.96:0/2233558739 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f4df800bf00 con 0x7f4e14156b30 2024-05-28T06:18:16.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.830+0000 7f4e12a00640 1 --2- 172.21.15.96:0/2233558739 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4df80039e0 0x7f4df8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:16.830 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.830+0000 7f4e12a00640 1 -- 172.21.15.96:0/2233558739 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4e14156b30 msgr2=0x7f4df800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:18:16.830 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.830+0000 7f4e12a00640 1 --2- 172.21.15.96:0/2233558739 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4e14156b30 0x7f4df800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:16.830 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.830+0000 7f4e12a00640 1 -- 172.21.15.96:0/2233558739 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4df8004800 msgr2=0x7f4df8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:18:16.830 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.830+0000 7f4e12a00640 1 --2- 172.21.15.96:0/2233558739 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4df8004800 0x7f4df8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:16.830 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.831+0000 7f4e12a00640 1 -- 172.21.15.96:0/2233558739 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4df8003520 con 0x7f4df80039e0 2024-05-28T06:18:16.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.831+0000 7f4e12a00640 1 --2- 172.21.15.96:0/2233558739 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4df80039e0 0x7f4df8003e00 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7f4e0400ba30 tx=0x7f4df8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c85aeb3d70abd648 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:16.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.832+0000 7f4e12000640 1 -- 172.21.15.96:0/2233558739 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e04030070 con 0x7f4df80039e0 2024-05-28T06:18:16.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.832+0000 7f4e12000640 1 -- 172.21.15.96:0/2233558739 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e0402ae10 con 0x7f4df80039e0 2024-05-28T06:18:16.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.832+0000 7f4e12000640 1 -- 172.21.15.96:0/2233558739 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e0400b600 con 0x7f4df80039e0 2024-05-28T06:18:16.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.832+0000 7f4e13e00640 1 -- 172.21.15.96:0/2233558739 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4df80039e0 msgr2=0x7f4df8003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:16.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.832+0000 7f4e13e00640 1 --2- 172.21.15.96:0/2233558739 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4df80039e0 0x7f4df8003e00 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7f4e0400ba30 tx=0x7f4df8005830 comp rx=0 tx=0).stop 2024-05-28T06:18:16.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.832+0000 7f4e13400640 1 -- 172.21.15.96:0/2233558739 reap_dead start 2024-05-28T06:18:16.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.832+0000 7f4e13e00640 1 -- 172.21.15.96:0/2233558739 shutdown_connections 2024-05-28T06:18:16.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.832+0000 7f4e13e00640 1 -- 172.21.15.96:0/2233558739 >> 172.21.15.96:0/2233558739 conn(0x7f4e140b1fc0 msgr2=0x7f4e140b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:18:16.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.832+0000 7f4e13e00640 1 -- 172.21.15.96:0/2233558739 shutdown_connections 2024-05-28T06:18:16.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.832+0000 7f4e13e00640 1 -- 172.21.15.96:0/2233558739 wait complete. 2024-05-28T06:18:16.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.834+0000 7f4e13e00640 1 Processor -- start 2024-05-28T06:18:16.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.834+0000 7f4e13e00640 1 -- start start 2024-05-28T06:18:16.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.835+0000 7f4e13e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4df80039e0 0x7f4e14154c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:16.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.835+0000 7f4e13e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4e14156f10 0x7f4e140999f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:16.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.835+0000 7f4e13e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e1413e470 0x7f4e1413e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:16.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.835+0000 7f4e13e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4e140b2db0 con 0x7f4e1413e470 2024-05-28T06:18:16.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.835+0000 7f4e13e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f4e140b2c30 con 0x7f4df80039e0 2024-05-28T06:18:16.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.835+0000 7f4e13e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f4e140b39d0 con 0x7f4e14156f10 2024-05-28T06:18:16.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.835+0000 7f4e13400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e1413e470 0x7f4e1413e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:16.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.835+0000 7f4e13400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e1413e470 0x7f4e1413e8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48326/0 (socket says 172.21.15.96:48326) 2024-05-28T06:18:16.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.835+0000 7f4e13400640 1 -- 172.21.15.96:0/2865580435 learned_addr learned my addr 172.21.15.96:0/2865580435 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:18:16.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.836+0000 7f4e12a00640 1 --2- 172.21.15.96:0/2865580435 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4df80039e0 0x7f4e14154c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:16.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.836+0000 7f4e0a000640 1 --2- 172.21.15.96:0/2865580435 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4e14156f10 0x7f4e140999f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:16.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.836+0000 7f4e13400640 1 -- 172.21.15.96:0/2865580435 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4e14156f10 msgr2=0x7f4e140999f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:16.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.836+0000 7f4e13400640 1 --2- 172.21.15.96:0/2865580435 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4e14156f10 0x7f4e140999f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:16.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.836+0000 7f4e13400640 1 -- 172.21.15.96:0/2865580435 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4df80039e0 msgr2=0x7f4e14154c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:16.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.836+0000 7f4e13400640 1 --2- 172.21.15.96:0/2865580435 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4df80039e0 0x7f4e14154c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:16.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.836+0000 7f4e13400640 1 -- 172.21.15.96:0/2865580435 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e1413ee30 con 0x7f4e1413e470 2024-05-28T06:18:16.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.836+0000 7f4e13400640 1 --2- 172.21.15.96:0/2865580435 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e1413e470 0x7f4e1413e8f0 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7f4e0c00c770 tx=0x7f4e0c00cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:16.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.836+0000 7f4e0b400640 1 -- 172.21.15.96:0/2865580435 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e0c00ce80 con 0x7f4e1413e470 2024-05-28T06:18:16.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.836+0000 7f4e0b400640 1 -- 172.21.15.96:0/2865580435 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e0c00a640 con 0x7f4e1413e470 2024-05-28T06:18:16.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.836+0000 7f4e0b400640 1 -- 172.21.15.96:0/2865580435 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e0c00aa30 con 0x7f4e1413e470 2024-05-28T06:18:16.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.837+0000 7f4e13e00640 1 -- 172.21.15.96:0/2865580435 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e14157390 con 0x7f4e1413e470 2024-05-28T06:18:16.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.837+0000 7f4e13e00640 1 -- 172.21.15.96:0/2865580435 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e141541a0 con 0x7f4e1413e470 2024-05-28T06:18:16.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.838+0000 7f4e13e00640 1 -- 172.21.15.96:0/2865580435 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e140689e0 con 0x7f4e1413e470 2024-05-28T06:18:16.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.839+0000 7f4e0b400640 1 -- 172.21.15.96:0/2865580435 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 30) ==== 92621+0+0 (secure 0 0 0) 0x7f4e0c00abd0 con 0x7f4e1413e470 2024-05-28T06:18:16.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.840+0000 7f4e0b400640 1 --2- 172.21.15.96:0/2865580435 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4df006da60 0x7f4df006ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:16.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.840+0000 7f4e0b400640 1 -- 172.21.15.96:0/2865580435 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 5737+0+0 (secure 0 0 0) 0x7f4e0c08a350 con 0x7f4e1413e470 2024-05-28T06:18:16.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.840+0000 7f4e12a00640 1 --2- 172.21.15.96:0/2865580435 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4df006da60 0x7f4df006ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:16.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.842+0000 7f4e12a00640 1 --2- 172.21.15.96:0/2865580435 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4df006da60 0x7f4df006ff20 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f4e0400ba00 tx=0x7f4e04030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:16.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:16.844+0000 7f4e0b400640 1 -- 172.21.15.96:0/2865580435 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4e0c059df0 con 0x7f4e1413e470 2024-05-28T06:18:17.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:17.042+0000 7f4e13e00640 1 -- 172.21.15.96:0/2865580435 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4e1406f7f0 con 0x7f4df006da60 2024-05-28T06:18:17.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:17.062+0000 7f4e0b400640 1 -- 172.21.15.96:0/2865580435 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252091 (secure 0 0 0) 0x7f4e1406f7f0 con 0x7f4df006da60 2024-05-28T06:18:17.063 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:18:17.064 INFO:teuthology.orchestra.run.smithi096.stderr:dumped all 2024-05-28T06:18:17.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:17.067+0000 7f4e13e00640 1 -- 172.21.15.96:0/2865580435 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4df006da60 msgr2=0x7f4df006ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:17.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:17.067+0000 7f4e13e00640 1 --2- 172.21.15.96:0/2865580435 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4df006da60 0x7f4df006ff20 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f4e0400ba00 tx=0x7f4e04030000 comp rx=0 tx=0).stop 2024-05-28T06:18:17.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:17.067+0000 7f4e13e00640 1 -- 172.21.15.96:0/2865580435 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e1413e470 msgr2=0x7f4e1413e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:17.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:17.067+0000 7f4e13e00640 1 --2- 172.21.15.96:0/2865580435 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e1413e470 0x7f4e1413e8f0 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7f4e0c00c770 tx=0x7f4e0c00cc30 comp rx=0 tx=0).stop 2024-05-28T06:18:17.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:17.067+0000 7f4e13e00640 1 -- 172.21.15.96:0/2865580435 shutdown_connections 2024-05-28T06:18:17.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:17.067+0000 7f4e13e00640 1 --2- 172.21.15.96:0/2865580435 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4df006da60 0x7f4df006ff20 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:17.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:17.067+0000 7f4e13e00640 1 --2- 172.21.15.96:0/2865580435 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e1413e470 0x7f4e1413e8f0 unknown :-1 s=CLOSED pgs=1423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:17.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:17.067+0000 7f4e13e00640 1 --2- 172.21.15.96:0/2865580435 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4e14156f10 0x7f4e140999f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:17.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:17.067+0000 7f4e13e00640 1 --2- 172.21.15.96:0/2865580435 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4df80039e0 0x7f4e14154c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:17.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:17.067+0000 7f4e13e00640 1 -- 172.21.15.96:0/2865580435 >> 172.21.15.96:0/2865580435 conn(0x7f4e140b1fc0 msgr2=0x7f4e1415f280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:18:17.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:17.068+0000 7f4e13e00640 1 -- 172.21.15.96:0/2865580435 shutdown_connections 2024-05-28T06:18:17.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:17.068+0000 7f4e13e00640 1 -- 172.21.15.96:0/2865580435 wait complete. 2024-05-28T06:18:17.084 INFO:teuthology.orchestra.run.smithi096.stdout:{"pg_ready":true,"pg_map":{"version":510,"stamp":"2024-05-28T06:18:15.863167+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":154,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":812628,"kb_used_data":9756,"kb_used_omap":12,"kb_used_meta":802739,"kb_avail":754162092,"statfs":{"total":773094113280,"available":772261982208,"internally_reserved":0,"allocated":9990144,"data_stored":4968044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":822005296},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.248409"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.416525+0000","last_change":"2024-05-28T06:18:10.671822+0000","last_active":"2024-05-28T06:18:12.416525+0000","last_peered":"2024-05-28T06:18:12.416525+0000","last_clean":"2024-05-28T06:18:12.416525+0000","last_became_active":"2024-05-28T06:18:10.671188+0000","last_became_peered":"2024-05-28T06:18:10.671188+0000","last_unstale":"2024-05-28T06:18:12.416525+0000","last_undegraded":"2024-05-28T06:18:12.416525+0000","last_fullsized":"2024-05-28T06:18:12.416525+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:40:59.653395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.676054+0000","last_change":"2024-05-28T06:18:10.673470+0000","last_active":"2024-05-28T06:18:11.676054+0000","last_peered":"2024-05-28T06:18:11.676054+0000","last_clean":"2024-05-28T06:18:11.676054+0000","last_became_active":"2024-05-28T06:18:10.673112+0000","last_became_peered":"2024-05-28T06:18:10.673112+0000","last_unstale":"2024-05-28T06:18:11.676054+0000","last_undegraded":"2024-05-28T06:18:11.676054+0000","last_fullsized":"2024-05-28T06:18:11.676054+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:50:18.056574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-28T06:18:10.688685+0000","last_change":"2024-05-28T06:18:10.684086+0000","last_active":"2024-05-28T06:18:10.688685+0000","last_peered":"2024-05-28T06:18:10.688685+0000","last_clean":"2024-05-28T06:18:10.688685+0000","last_became_active":"2024-05-28T06:18:10.683345+0000","last_became_peered":"2024-05-28T06:18:10.683345+0000","last_unstale":"2024-05-28T06:18:10.688685+0000","last_undegraded":"2024-05-28T06:18:10.688685+0000","last_fullsized":"2024-05-28T06:18:10.688685+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:36:04.760890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.417075+0000","last_change":"2024-05-28T06:18:10.667705+0000","last_active":"2024-05-28T06:18:12.417075+0000","last_peered":"2024-05-28T06:18:12.417075+0000","last_clean":"2024-05-28T06:18:12.417075+0000","last_became_active":"2024-05-28T06:18:10.667442+0000","last_became_peered":"2024-05-28T06:18:10.667442+0000","last_unstale":"2024-05-28T06:18:12.417075+0000","last_undegraded":"2024-05-28T06:18:12.417075+0000","last_fullsized":"2024-05-28T06:18:12.417075+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:51:19.311871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.669932+0000","last_change":"2024-05-28T06:18:10.660951+0000","last_active":"2024-05-28T06:18:11.669932+0000","last_peered":"2024-05-28T06:18:11.669932+0000","last_clean":"2024-05-28T06:18:11.669932+0000","last_became_active":"2024-05-28T06:18:10.660437+0000","last_became_peered":"2024-05-28T06:18:10.660437+0000","last_unstale":"2024-05-28T06:18:11.669932+0000","last_undegraded":"2024-05-28T06:18:11.669932+0000","last_fullsized":"2024-05-28T06:18:11.669932+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:03:21.839812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.419909+0000","last_change":"2024-05-28T06:18:10.680300+0000","last_active":"2024-05-28T06:18:12.419909+0000","last_peered":"2024-05-28T06:18:12.419909+0000","last_clean":"2024-05-28T06:18:12.419909+0000","last_became_active":"2024-05-28T06:18:10.679570+0000","last_became_peered":"2024-05-28T06:18:10.679570+0000","last_unstale":"2024-05-28T06:18:12.419909+0000","last_undegraded":"2024-05-28T06:18:12.419909+0000","last_fullsized":"2024-05-28T06:18:12.419909+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:10:10.001167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.415870+0000","last_change":"2024-05-28T06:18:10.670026+0000","last_active":"2024-05-28T06:18:12.415870+0000","last_peered":"2024-05-28T06:18:12.415870+0000","last_clean":"2024-05-28T06:18:12.415870+0000","last_became_active":"2024-05-28T06:18:10.669644+0000","last_became_peered":"2024-05-28T06:18:10.669644+0000","last_unstale":"2024-05-28T06:18:12.415870+0000","last_undegraded":"2024-05-28T06:18:12.415870+0000","last_fullsized":"2024-05-28T06:18:12.415870+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:04:19.667824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.418225+0000","last_change":"2024-05-28T06:18:10.680190+0000","last_active":"2024-05-28T06:18:12.418225+0000","last_peered":"2024-05-28T06:18:12.418225+0000","last_clean":"2024-05-28T06:18:12.418225+0000","last_became_active":"2024-05-28T06:18:10.679764+0000","last_became_peered":"2024-05-28T06:18:10.679764+0000","last_unstale":"2024-05-28T06:18:12.418225+0000","last_undegraded":"2024-05-28T06:18:12.418225+0000","last_fullsized":"2024-05-28T06:18:12.418225+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:43:28.316407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.420071+0000","last_change":"2024-05-28T06:18:10.685588+0000","last_active":"2024-05-28T06:18:12.420071+0000","last_peered":"2024-05-28T06:18:12.420071+0000","last_clean":"2024-05-28T06:18:12.420071+0000","last_became_active":"2024-05-28T06:18:10.685361+0000","last_became_peered":"2024-05-28T06:18:10.685361+0000","last_unstale":"2024-05-28T06:18:12.420071+0000","last_undegraded":"2024-05-28T06:18:12.420071+0000","last_fullsized":"2024-05-28T06:18:12.420071+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:40:59.653395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.417249+0000","last_change":"2024-05-28T06:18:10.679458+0000","last_active":"2024-05-28T06:18:12.417249+0000","last_peered":"2024-05-28T06:18:12.417249+0000","last_clean":"2024-05-28T06:18:12.417249+0000","last_became_active":"2024-05-28T06:18:10.679143+0000","last_became_peered":"2024-05-28T06:18:10.679143+0000","last_unstale":"2024-05-28T06:18:12.417249+0000","last_undegraded":"2024-05-28T06:18:12.417249+0000","last_fullsized":"2024-05-28T06:18:12.417249+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:42:12.312828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.675913+0000","last_change":"2024-05-28T06:18:10.675557+0000","last_active":"2024-05-28T06:18:11.675913+0000","last_peered":"2024-05-28T06:18:11.675913+0000","last_clean":"2024-05-28T06:18:11.675913+0000","last_became_active":"2024-05-28T06:18:10.675292+0000","last_became_peered":"2024-05-28T06:18:10.675292+0000","last_unstale":"2024-05-28T06:18:11.675913+0000","last_undegraded":"2024-05-28T06:18:11.675913+0000","last_fullsized":"2024-05-28T06:18:11.675913+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:46:26.001440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.416176+0000","last_change":"2024-05-28T06:18:10.677581+0000","last_active":"2024-05-28T06:18:12.416176+0000","last_peered":"2024-05-28T06:18:12.416176+0000","last_clean":"2024-05-28T06:18:12.416176+0000","last_became_active":"2024-05-28T06:18:10.675476+0000","last_became_peered":"2024-05-28T06:18:10.675476+0000","last_unstale":"2024-05-28T06:18:12.416176+0000","last_undegraded":"2024-05-28T06:18:12.416176+0000","last_fullsized":"2024-05-28T06:18:12.416176+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T18:13:08.105068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-28T06:18:10.676482+0000","last_change":"2024-05-28T06:18:10.673510+0000","last_active":"2024-05-28T06:18:10.676482+0000","last_peered":"2024-05-28T06:18:10.676482+0000","last_clean":"2024-05-28T06:18:10.676482+0000","last_became_active":"2024-05-28T06:18:10.673194+0000","last_became_peered":"2024-05-28T06:18:10.673194+0000","last_unstale":"2024-05-28T06:18:10.676482+0000","last_undegraded":"2024-05-28T06:18:10.676482+0000","last_fullsized":"2024-05-28T06:18:10.676482+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:10:10.001167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.416894+0000","last_change":"2024-05-28T06:18:10.676724+0000","last_active":"2024-05-28T06:18:12.416894+0000","last_peered":"2024-05-28T06:18:12.416894+0000","last_clean":"2024-05-28T06:18:12.416894+0000","last_became_active":"2024-05-28T06:18:10.676457+0000","last_became_peered":"2024-05-28T06:18:10.676457+0000","last_unstale":"2024-05-28T06:18:12.416894+0000","last_undegraded":"2024-05-28T06:18:12.416894+0000","last_fullsized":"2024-05-28T06:18:12.416894+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:50:29.852024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.415556+0000","last_change":"2024-05-28T06:18:10.667679+0000","last_active":"2024-05-28T06:18:12.415556+0000","last_peered":"2024-05-28T06:18:12.415556+0000","last_clean":"2024-05-28T06:18:12.415556+0000","last_became_active":"2024-05-28T06:18:10.667355+0000","last_became_peered":"2024-05-28T06:18:10.667355+0000","last_unstale":"2024-05-28T06:18:12.415556+0000","last_undegraded":"2024-05-28T06:18:12.415556+0000","last_fullsized":"2024-05-28T06:18:12.415556+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:57:41.516876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.669444+0000","last_change":"2024-05-28T06:18:10.676070+0000","last_active":"2024-05-28T06:18:11.669444+0000","last_peered":"2024-05-28T06:18:11.669444+0000","last_clean":"2024-05-28T06:18:11.669444+0000","last_became_active":"2024-05-28T06:18:10.675759+0000","last_became_peered":"2024-05-28T06:18:10.675759+0000","last_unstale":"2024-05-28T06:18:11.669444+0000","last_undegraded":"2024-05-28T06:18:11.669444+0000","last_fullsized":"2024-05-28T06:18:11.669444+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:19:46.201909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.670757+0000","last_change":"2024-05-28T06:18:10.655314+0000","last_active":"2024-05-28T06:18:11.670757+0000","last_peered":"2024-05-28T06:18:11.670757+0000","last_clean":"2024-05-28T06:18:11.670757+0000","last_became_active":"2024-05-28T06:18:10.654580+0000","last_became_peered":"2024-05-28T06:18:10.654580+0000","last_unstale":"2024-05-28T06:18:11.670757+0000","last_undegraded":"2024-05-28T06:18:11.670757+0000","last_fullsized":"2024-05-28T06:18:11.670757+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:10:10.001167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.417262+0000","last_change":"2024-05-28T06:18:10.670459+0000","last_active":"2024-05-28T06:18:12.417262+0000","last_peered":"2024-05-28T06:18:12.417262+0000","last_clean":"2024-05-28T06:18:12.417262+0000","last_became_active":"2024-05-28T06:18:10.670119+0000","last_became_peered":"2024-05-28T06:18:10.670119+0000","last_unstale":"2024-05-28T06:18:12.417262+0000","last_undegraded":"2024-05-28T06:18:12.417262+0000","last_fullsized":"2024-05-28T06:18:12.417262+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:09:41.395379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.668772+0000","last_change":"2024-05-28T06:18:10.662056+0000","last_active":"2024-05-28T06:18:11.668772+0000","last_peered":"2024-05-28T06:18:11.668772+0000","last_clean":"2024-05-28T06:18:11.668772+0000","last_became_active":"2024-05-28T06:18:10.661589+0000","last_became_peered":"2024-05-28T06:18:10.661589+0000","last_unstale":"2024-05-28T06:18:11.668772+0000","last_undegraded":"2024-05-28T06:18:11.668772+0000","last_fullsized":"2024-05-28T06:18:11.668772+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:52:52.857149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.417036+0000","last_change":"2024-05-28T06:18:10.678748+0000","last_active":"2024-05-28T06:18:12.417036+0000","last_peered":"2024-05-28T06:18:12.417036+0000","last_clean":"2024-05-28T06:18:12.417036+0000","last_became_active":"2024-05-28T06:18:10.678444+0000","last_became_peered":"2024-05-28T06:18:10.678444+0000","last_unstale":"2024-05-28T06:18:12.417036+0000","last_undegraded":"2024-05-28T06:18:12.417036+0000","last_fullsized":"2024-05-28T06:18:12.417036+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:30:00.901425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.416302+0000","last_change":"2024-05-28T06:18:10.678968+0000","last_active":"2024-05-28T06:18:12.416302+0000","last_peered":"2024-05-28T06:18:12.416302+0000","last_clean":"2024-05-28T06:18:12.416302+0000","last_became_active":"2024-05-28T06:18:10.678693+0000","last_became_peered":"2024-05-28T06:18:10.678693+0000","last_unstale":"2024-05-28T06:18:12.416302+0000","last_undegraded":"2024-05-28T06:18:12.416302+0000","last_fullsized":"2024-05-28T06:18:12.416302+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:24:00.461410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.675477+0000","last_change":"2024-05-28T06:18:10.688872+0000","last_active":"2024-05-28T06:18:11.675477+0000","last_peered":"2024-05-28T06:18:11.675477+0000","last_clean":"2024-05-28T06:18:11.675477+0000","last_became_active":"2024-05-28T06:18:10.688588+0000","last_became_peered":"2024-05-28T06:18:10.688588+0000","last_unstale":"2024-05-28T06:18:11.675477+0000","last_undegraded":"2024-05-28T06:18:11.675477+0000","last_fullsized":"2024-05-28T06:18:11.675477+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:58:16.081516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.415595+0000","last_change":"2024-05-28T06:18:10.683026+0000","last_active":"2024-05-28T06:18:12.415595+0000","last_peered":"2024-05-28T06:18:12.415595+0000","last_clean":"2024-05-28T06:18:12.415595+0000","last_became_active":"2024-05-28T06:18:10.682741+0000","last_became_peered":"2024-05-28T06:18:10.682741+0000","last_unstale":"2024-05-28T06:18:12.415595+0000","last_undegraded":"2024-05-28T06:18:12.415595+0000","last_fullsized":"2024-05-28T06:18:12.415595+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:43:28.316407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.415588+0000","last_change":"2024-05-28T06:18:10.665505+0000","last_active":"2024-05-28T06:18:12.415588+0000","last_peered":"2024-05-28T06:18:12.415588+0000","last_clean":"2024-05-28T06:18:12.415588+0000","last_became_active":"2024-05-28T06:18:10.665204+0000","last_became_peered":"2024-05-28T06:18:10.665204+0000","last_unstale":"2024-05-28T06:18:12.415588+0000","last_undegraded":"2024-05-28T06:18:12.415588+0000","last_fullsized":"2024-05-28T06:18:12.415588+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:13:16.431201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.676140+0000","last_change":"2024-05-28T06:18:10.684581+0000","last_active":"2024-05-28T06:18:11.676140+0000","last_peered":"2024-05-28T06:18:11.676140+0000","last_clean":"2024-05-28T06:18:11.676140+0000","last_became_active":"2024-05-28T06:18:10.684250+0000","last_became_peered":"2024-05-28T06:18:10.684250+0000","last_unstale":"2024-05-28T06:18:11.676140+0000","last_undegraded":"2024-05-28T06:18:11.676140+0000","last_fullsized":"2024-05-28T06:18:11.676140+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:32:36.064344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-28T06:18:10.688666+0000","last_change":"2024-05-28T06:18:10.682743+0000","last_active":"2024-05-28T06:18:10.688666+0000","last_peered":"2024-05-28T06:18:10.688666+0000","last_clean":"2024-05-28T06:18:10.688666+0000","last_became_active":"2024-05-28T06:18:10.682451+0000","last_became_peered":"2024-05-28T06:18:10.682451+0000","last_unstale":"2024-05-28T06:18:10.688666+0000","last_undegraded":"2024-05-28T06:18:10.688666+0000","last_fullsized":"2024-05-28T06:18:10.688666+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:38:26.397686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.670303+0000","last_change":"2024-05-28T06:18:10.661976+0000","last_active":"2024-05-28T06:18:11.670303+0000","last_peered":"2024-05-28T06:18:11.670303+0000","last_clean":"2024-05-28T06:18:11.670303+0000","last_became_active":"2024-05-28T06:18:10.661460+0000","last_became_peered":"2024-05-28T06:18:10.661460+0000","last_unstale":"2024-05-28T06:18:11.670303+0000","last_undegraded":"2024-05-28T06:18:11.670303+0000","last_fullsized":"2024-05-28T06:18:11.670303+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:30:38.196353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.417151+0000","last_change":"2024-05-28T06:18:10.666860+0000","last_active":"2024-05-28T06:18:12.417151+0000","last_peered":"2024-05-28T06:18:12.417151+0000","last_clean":"2024-05-28T06:18:12.417151+0000","last_became_active":"2024-05-28T06:18:10.666305+0000","last_became_peered":"2024-05-28T06:18:10.666305+0000","last_unstale":"2024-05-28T06:18:12.417151+0000","last_undegraded":"2024-05-28T06:18:12.417151+0000","last_fullsized":"2024-05-28T06:18:12.417151+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:52:52.857149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.676511+0000","last_change":"2024-05-28T06:18:10.668290+0000","last_active":"2024-05-28T06:18:11.676511+0000","last_peered":"2024-05-28T06:18:11.676511+0000","last_clean":"2024-05-28T06:18:11.676511+0000","last_became_active":"2024-05-28T06:18:10.666561+0000","last_became_peered":"2024-05-28T06:18:10.666561+0000","last_unstale":"2024-05-28T06:18:11.676511+0000","last_undegraded":"2024-05-28T06:18:11.676511+0000","last_fullsized":"2024-05-28T06:18:11.676511+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:24:17.156838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.675494+0000","last_change":"2024-05-28T06:18:10.675895+0000","last_active":"2024-05-28T06:18:11.675494+0000","last_peered":"2024-05-28T06:18:11.675494+0000","last_clean":"2024-05-28T06:18:11.675494+0000","last_became_active":"2024-05-28T06:18:10.675623+0000","last_became_peered":"2024-05-28T06:18:10.675623+0000","last_unstale":"2024-05-28T06:18:11.675494+0000","last_undegraded":"2024-05-28T06:18:11.675494+0000","last_fullsized":"2024-05-28T06:18:11.675494+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:17:45.643378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.669373+0000","last_change":"2024-05-28T06:18:10.667388+0000","last_active":"2024-05-28T06:18:11.669373+0000","last_peered":"2024-05-28T06:18:11.669373+0000","last_clean":"2024-05-28T06:18:11.669373+0000","last_became_active":"2024-05-28T06:18:10.667062+0000","last_became_peered":"2024-05-28T06:18:10.667062+0000","last_unstale":"2024-05-28T06:18:11.669373+0000","last_undegraded":"2024-05-28T06:18:11.669373+0000","last_fullsized":"2024-05-28T06:18:11.669373+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:20:23.727599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.415807+0000","last_change":"2024-05-28T06:18:10.676303+0000","last_active":"2024-05-28T06:18:12.415807+0000","last_peered":"2024-05-28T06:18:12.415807+0000","last_clean":"2024-05-28T06:18:12.415807+0000","last_became_active":"2024-05-28T06:18:10.675998+0000","last_became_peered":"2024-05-28T06:18:10.675998+0000","last_unstale":"2024-05-28T06:18:12.415807+0000","last_undegraded":"2024-05-28T06:18:12.415807+0000","last_fullsized":"2024-05-28T06:18:12.415807+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:10:10.001167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.670796+0000","last_change":"2024-05-28T06:18:10.671446+0000","last_active":"2024-05-28T06:18:11.670796+0000","last_peered":"2024-05-28T06:18:11.670796+0000","last_clean":"2024-05-28T06:18:11.670796+0000","last_became_active":"2024-05-28T06:18:10.671173+0000","last_became_peered":"2024-05-28T06:18:10.671173+0000","last_unstale":"2024-05-28T06:18:11.670796+0000","last_undegraded":"2024-05-28T06:18:11.670796+0000","last_fullsized":"2024-05-28T06:18:11.670796+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:35:48.065461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"88'4","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.417211+0000","last_change":"2024-05-28T06:18:09.643542+0000","last_active":"2024-05-28T06:18:12.417211+0000","last_peered":"2024-05-28T06:18:12.417211+0000","last_clean":"2024-05-28T06:18:12.417211+0000","last_became_active":"2024-05-28T06:18:09.641808+0000","last_became_peered":"2024-05-28T06:18:09.641808+0000","last_unstale":"2024-05-28T06:18:12.417211+0000","last_undegraded":"2024-05-28T06:18:12.417211+0000","last_fullsized":"2024-05-28T06:18:12.417211+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:50:32.547130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.416707+0000","last_change":"2024-05-28T06:18:10.682953+0000","last_active":"2024-05-28T06:18:12.416707+0000","last_peered":"2024-05-28T06:18:12.416707+0000","last_clean":"2024-05-28T06:18:12.416707+0000","last_became_active":"2024-05-28T06:18:10.682621+0000","last_became_peered":"2024-05-28T06:18:10.682621+0000","last_unstale":"2024-05-28T06:18:12.416707+0000","last_undegraded":"2024-05-28T06:18:12.416707+0000","last_fullsized":"2024-05-28T06:18:12.416707+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:38:26.397686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"88'1","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.671160+0000","last_change":"2024-05-28T06:18:09.647214+0000","last_active":"2024-05-28T06:18:11.671160+0000","last_peered":"2024-05-28T06:18:11.671160+0000","last_clean":"2024-05-28T06:18:11.671160+0000","last_became_active":"2024-05-28T06:18:09.646937+0000","last_became_peered":"2024-05-28T06:18:09.646937+0000","last_unstale":"2024-05-28T06:18:11.671160+0000","last_undegraded":"2024-05-28T06:18:11.671160+0000","last_fullsized":"2024-05-28T06:18:11.671160+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:45:34.943093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.417520+0000","last_change":"2024-05-28T06:18:10.681586+0000","last_active":"2024-05-28T06:18:12.417520+0000","last_peered":"2024-05-28T06:18:12.417520+0000","last_clean":"2024-05-28T06:18:12.417520+0000","last_became_active":"2024-05-28T06:18:10.681223+0000","last_became_peered":"2024-05-28T06:18:10.681223+0000","last_unstale":"2024-05-28T06:18:12.417520+0000","last_undegraded":"2024-05-28T06:18:12.417520+0000","last_fullsized":"2024-05-28T06:18:12.417520+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:17:36.803843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"88'2","reported_seq":16,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.668839+0000","last_change":"2024-05-28T06:18:09.646080+0000","last_active":"2024-05-28T06:18:11.668839+0000","last_peered":"2024-05-28T06:18:11.668839+0000","last_clean":"2024-05-28T06:18:11.668839+0000","last_became_active":"2024-05-28T06:18:09.645807+0000","last_became_peered":"2024-05-28T06:18:09.645807+0000","last_unstale":"2024-05-28T06:18:11.668839+0000","last_undegraded":"2024-05-28T06:18:11.668839+0000","last_fullsized":"2024-05-28T06:18:11.668839+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:30:37.181316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-28T06:18:10.680587+0000","last_change":"2024-05-28T06:18:10.677395+0000","last_active":"2024-05-28T06:18:10.680587+0000","last_peered":"2024-05-28T06:18:10.680587+0000","last_clean":"2024-05-28T06:18:10.680587+0000","last_became_active":"2024-05-28T06:18:10.677100+0000","last_became_peered":"2024-05-28T06:18:10.677100+0000","last_unstale":"2024-05-28T06:18:10.680587+0000","last_undegraded":"2024-05-28T06:18:10.680587+0000","last_fullsized":"2024-05-28T06:18:10.680587+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:35:48.065461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"88'2","reported_seq":16,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.668500+0000","last_change":"2024-05-28T06:18:09.643971+0000","last_active":"2024-05-28T06:18:11.668500+0000","last_peered":"2024-05-28T06:18:11.668500+0000","last_clean":"2024-05-28T06:18:11.668500+0000","last_became_active":"2024-05-28T06:18:09.643691+0000","last_became_peered":"2024-05-28T06:18:09.643691+0000","last_unstale":"2024-05-28T06:18:11.668500+0000","last_undegraded":"2024-05-28T06:18:11.668500+0000","last_fullsized":"2024-05-28T06:18:11.668500+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:03:20.824775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.675683+0000","last_change":"2024-05-28T06:18:10.667892+0000","last_active":"2024-05-28T06:18:11.675683+0000","last_peered":"2024-05-28T06:18:11.675683+0000","last_clean":"2024-05-28T06:18:11.675683+0000","last_became_active":"2024-05-28T06:18:10.667622+0000","last_became_peered":"2024-05-28T06:18:10.667622+0000","last_unstale":"2024-05-28T06:18:11.675683+0000","last_undegraded":"2024-05-28T06:18:11.675683+0000","last_fullsized":"2024-05-28T06:18:11.675683+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:15:40.941909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"88'1","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.670434+0000","last_change":"2024-05-28T06:18:09.645873+0000","last_active":"2024-05-28T06:18:11.670434+0000","last_peered":"2024-05-28T06:18:11.670434+0000","last_clean":"2024-05-28T06:18:11.670434+0000","last_became_active":"2024-05-28T06:18:09.645561+0000","last_became_peered":"2024-05-28T06:18:09.645561+0000","last_unstale":"2024-05-28T06:18:11.670434+0000","last_undegraded":"2024-05-28T06:18:11.670434+0000","last_fullsized":"2024-05-28T06:18:11.670434+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:33:21.519096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.670041+0000","last_change":"2024-05-28T06:18:10.653334+0000","last_active":"2024-05-28T06:18:11.670041+0000","last_peered":"2024-05-28T06:18:11.670041+0000","last_clean":"2024-05-28T06:18:11.670041+0000","last_became_active":"2024-05-28T06:18:10.652195+0000","last_became_peered":"2024-05-28T06:18:10.652195+0000","last_unstale":"2024-05-28T06:18:11.670041+0000","last_undegraded":"2024-05-28T06:18:11.670041+0000","last_fullsized":"2024-05-28T06:18:11.670041+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:10:33.962690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"88'1","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.669219+0000","last_change":"2024-05-28T06:18:09.641147+0000","last_active":"2024-05-28T06:18:11.669219+0000","last_peered":"2024-05-28T06:18:11.669219+0000","last_clean":"2024-05-28T06:18:11.669219+0000","last_became_active":"2024-05-28T06:18:09.640830+0000","last_became_peered":"2024-05-28T06:18:09.640830+0000","last_unstale":"2024-05-28T06:18:11.669219+0000","last_undegraded":"2024-05-28T06:18:11.669219+0000","last_fullsized":"2024-05-28T06:18:11.669219+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:44:26.345831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-28T06:18:10.688719+0000","last_change":"2024-05-28T06:18:10.684510+0000","last_active":"2024-05-28T06:18:10.688719+0000","last_peered":"2024-05-28T06:18:10.688719+0000","last_clean":"2024-05-28T06:18:10.688719+0000","last_became_active":"2024-05-28T06:18:10.684209+0000","last_became_peered":"2024-05-28T06:18:10.684209+0000","last_unstale":"2024-05-28T06:18:10.688719+0000","last_undegraded":"2024-05-28T06:18:10.688719+0000","last_fullsized":"2024-05-28T06:18:10.688719+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T18:13:08.105068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.667919+0000","last_change":"2024-05-28T06:18:09.647844+0000","last_active":"2024-05-28T06:18:11.667919+0000","last_peered":"2024-05-28T06:18:11.667919+0000","last_clean":"2024-05-28T06:18:11.667919+0000","last_became_active":"2024-05-28T06:18:09.647527+0000","last_became_peered":"2024-05-28T06:18:09.647527+0000","last_unstale":"2024-05-28T06:18:11.667919+0000","last_undegraded":"2024-05-28T06:18:11.667919+0000","last_fullsized":"2024-05-28T06:18:11.667919+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:20:22.712562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.416481+0000","last_change":"2024-05-28T06:18:10.667540+0000","last_active":"2024-05-28T06:18:12.416481+0000","last_peered":"2024-05-28T06:18:12.416481+0000","last_clean":"2024-05-28T06:18:12.416481+0000","last_became_active":"2024-05-28T06:18:10.667232+0000","last_became_peered":"2024-05-28T06:18:10.667232+0000","last_unstale":"2024-05-28T06:18:12.416481+0000","last_undegraded":"2024-05-28T06:18:12.416481+0000","last_fullsized":"2024-05-28T06:18:12.416481+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:45:48.725150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.675347+0000","last_change":"2024-05-28T06:18:09.648064+0000","last_active":"2024-05-28T06:18:11.675347+0000","last_peered":"2024-05-28T06:18:11.675347+0000","last_clean":"2024-05-28T06:18:11.675347+0000","last_became_active":"2024-05-28T06:18:09.647787+0000","last_became_peered":"2024-05-28T06:18:09.647787+0000","last_unstale":"2024-05-28T06:18:11.675347+0000","last_undegraded":"2024-05-28T06:18:11.675347+0000","last_fullsized":"2024-05-28T06:18:11.675347+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:26:13.635159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.418311+0000","last_change":"2024-05-28T06:18:10.674109+0000","last_active":"2024-05-28T06:18:12.418311+0000","last_peered":"2024-05-28T06:18:12.418311+0000","last_clean":"2024-05-28T06:18:12.418311+0000","last_became_active":"2024-05-28T06:18:10.673531+0000","last_became_peered":"2024-05-28T06:18:10.673531+0000","last_unstale":"2024-05-28T06:18:12.418311+0000","last_undegraded":"2024-05-28T06:18:12.418311+0000","last_fullsized":"2024-05-28T06:18:12.418311+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:15:33.611029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.417249+0000","last_change":"2024-05-28T06:18:09.643958+0000","last_active":"2024-05-28T06:18:12.417249+0000","last_peered":"2024-05-28T06:18:12.417249+0000","last_clean":"2024-05-28T06:18:12.417249+0000","last_became_active":"2024-05-28T06:18:09.643658+0000","last_became_peered":"2024-05-28T06:18:09.643658+0000","last_unstale":"2024-05-28T06:18:12.417249+0000","last_undegraded":"2024-05-28T06:18:12.417249+0000","last_fullsized":"2024-05-28T06:18:12.417249+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:37:17.604820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.420418+0000","last_change":"2024-05-28T06:18:10.680643+0000","last_active":"2024-05-28T06:18:12.420418+0000","last_peered":"2024-05-28T06:18:12.420418+0000","last_clean":"2024-05-28T06:18:12.420418+0000","last_became_active":"2024-05-28T06:18:10.677505+0000","last_became_peered":"2024-05-28T06:18:10.677505+0000","last_unstale":"2024-05-28T06:18:12.420418+0000","last_undegraded":"2024-05-28T06:18:12.420418+0000","last_fullsized":"2024-05-28T06:18:12.420418+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:36:04.760890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.417061+0000","last_change":"2024-05-28T06:18:09.645952+0000","last_active":"2024-05-28T06:18:12.417061+0000","last_peered":"2024-05-28T06:18:12.417061+0000","last_clean":"2024-05-28T06:18:12.417061+0000","last_became_active":"2024-05-28T06:18:09.644892+0000","last_became_peered":"2024-05-28T06:18:09.644892+0000","last_unstale":"2024-05-28T06:18:12.417061+0000","last_undegraded":"2024-05-28T06:18:12.417061+0000","last_fullsized":"2024-05-28T06:18:12.417061+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:48:38.824030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.668434+0000","last_change":"2024-05-28T06:18:10.678535+0000","last_active":"2024-05-28T06:18:11.668434+0000","last_peered":"2024-05-28T06:18:11.668434+0000","last_clean":"2024-05-28T06:18:11.668434+0000","last_became_active":"2024-05-28T06:18:10.678261+0000","last_became_peered":"2024-05-28T06:18:10.678261+0000","last_unstale":"2024-05-28T06:18:11.668434+0000","last_undegraded":"2024-05-28T06:18:11.668434+0000","last_fullsized":"2024-05-28T06:18:11.668434+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T18:13:08.105068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"88'3","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.416171+0000","last_change":"2024-05-28T06:18:09.648424+0000","last_active":"2024-05-28T06:18:12.416171+0000","last_peered":"2024-05-28T06:18:12.416171+0000","last_clean":"2024-05-28T06:18:12.416171+0000","last_became_active":"2024-05-28T06:18:09.648138+0000","last_became_peered":"2024-05-28T06:18:09.648138+0000","last_unstale":"2024-05-28T06:18:12.416171+0000","last_undegraded":"2024-05-28T06:18:12.416171+0000","last_fullsized":"2024-05-28T06:18:12.416171+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:33:21.519096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.670379+0000","last_change":"2024-05-28T06:18:10.666292+0000","last_active":"2024-05-28T06:18:11.670379+0000","last_peered":"2024-05-28T06:18:11.670379+0000","last_clean":"2024-05-28T06:18:11.670379+0000","last_became_active":"2024-05-28T06:18:10.665709+0000","last_became_peered":"2024-05-28T06:18:10.665709+0000","last_unstale":"2024-05-28T06:18:11.670379+0000","last_undegraded":"2024-05-28T06:18:11.670379+0000","last_fullsized":"2024-05-28T06:18:11.670379+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:43:28.316407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"88'6","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.419335+0000","last_change":"2024-05-28T06:18:09.645431+0000","last_active":"2024-05-28T06:18:12.419335+0000","last_peered":"2024-05-28T06:18:12.419335+0000","last_clean":"2024-05-28T06:18:12.419335+0000","last_became_active":"2024-05-28T06:18:09.645141+0000","last_became_peered":"2024-05-28T06:18:09.645141+0000","last_unstale":"2024-05-28T06:18:12.419335+0000","last_undegraded":"2024-05-28T06:18:12.419335+0000","last_fullsized":"2024-05-28T06:18:12.419335+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:33:46.549475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.668301+0000","last_change":"2024-05-28T06:18:10.677586+0000","last_active":"2024-05-28T06:18:11.668301+0000","last_peered":"2024-05-28T06:18:11.668301+0000","last_clean":"2024-05-28T06:18:11.668301+0000","last_became_active":"2024-05-28T06:18:10.677186+0000","last_became_peered":"2024-05-28T06:18:10.677186+0000","last_unstale":"2024-05-28T06:18:11.668301+0000","last_undegraded":"2024-05-28T06:18:11.668301+0000","last_fullsized":"2024-05-28T06:18:11.668301+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:36:04.760890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"88'2","reported_seq":16,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.675765+0000","last_change":"2024-05-28T06:18:09.647148+0000","last_active":"2024-05-28T06:18:11.675765+0000","last_peered":"2024-05-28T06:18:11.675765+0000","last_clean":"2024-05-28T06:18:11.675765+0000","last_became_active":"2024-05-28T06:18:09.646882+0000","last_became_peered":"2024-05-28T06:18:09.646882+0000","last_unstale":"2024-05-28T06:18:11.675765+0000","last_undegraded":"2024-05-28T06:18:11.675765+0000","last_fullsized":"2024-05-28T06:18:11.675765+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:17:12.740244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-28T06:18:10.688761+0000","last_change":"2024-05-28T06:18:10.687447+0000","last_active":"2024-05-28T06:18:10.688761+0000","last_peered":"2024-05-28T06:18:10.688761+0000","last_clean":"2024-05-28T06:18:10.688761+0000","last_became_active":"2024-05-28T06:18:10.687161+0000","last_became_peered":"2024-05-28T06:18:10.687161+0000","last_unstale":"2024-05-28T06:18:10.688761+0000","last_undegraded":"2024-05-28T06:18:10.688761+0000","last_fullsized":"2024-05-28T06:18:10.688761+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:52:52.857149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.417055+0000","last_change":"2024-05-28T06:18:09.647907+0000","last_active":"2024-05-28T06:18:12.417055+0000","last_peered":"2024-05-28T06:18:12.417055+0000","last_clean":"2024-05-28T06:18:12.417055+0000","last_became_active":"2024-05-28T06:18:09.647240+0000","last_became_peered":"2024-05-28T06:18:09.647240+0000","last_unstale":"2024-05-28T06:18:12.417055+0000","last_undegraded":"2024-05-28T06:18:12.417055+0000","last_fullsized":"2024-05-28T06:18:12.417055+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:21:28.556698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.675523+0000","last_change":"2024-05-28T06:18:10.670763+0000","last_active":"2024-05-28T06:18:11.675523+0000","last_peered":"2024-05-28T06:18:11.675523+0000","last_clean":"2024-05-28T06:18:11.675523+0000","last_became_active":"2024-05-28T06:18:10.670493+0000","last_became_peered":"2024-05-28T06:18:10.670493+0000","last_unstale":"2024-05-28T06:18:11.675523+0000","last_undegraded":"2024-05-28T06:18:11.675523+0000","last_fullsized":"2024-05-28T06:18:11.675523+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:23:19.474029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.419186+0000","last_change":"2024-05-28T06:18:09.645407+0000","last_active":"2024-05-28T06:18:12.419186+0000","last_peered":"2024-05-28T06:18:12.419186+0000","last_clean":"2024-05-28T06:18:12.419186+0000","last_became_active":"2024-05-28T06:18:09.644308+0000","last_became_peered":"2024-05-28T06:18:09.644308+0000","last_unstale":"2024-05-28T06:18:12.419186+0000","last_undegraded":"2024-05-28T06:18:12.419186+0000","last_fullsized":"2024-05-28T06:18:12.419186+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T18:02:58.935614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.675215+0000","last_change":"2024-05-28T06:18:10.674096+0000","last_active":"2024-05-28T06:18:11.675215+0000","last_peered":"2024-05-28T06:18:11.675215+0000","last_clean":"2024-05-28T06:18:11.675215+0000","last_became_active":"2024-05-28T06:18:10.673401+0000","last_became_peered":"2024-05-28T06:18:10.673401+0000","last_unstale":"2024-05-28T06:18:11.675215+0000","last_undegraded":"2024-05-28T06:18:11.675215+0000","last_fullsized":"2024-05-28T06:18:11.675215+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:29:01.599345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"88'8","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.667571+0000","last_change":"2024-05-28T06:18:09.643338+0000","last_active":"2024-05-28T06:18:11.667571+0000","last_peered":"2024-05-28T06:18:11.667571+0000","last_clean":"2024-05-28T06:18:11.667571+0000","last_became_active":"2024-05-28T06:18:09.643035+0000","last_became_peered":"2024-05-28T06:18:09.643035+0000","last_unstale":"2024-05-28T06:18:11.667571+0000","last_undegraded":"2024-05-28T06:18:11.667571+0000","last_fullsized":"2024-05-28T06:18:11.667571+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:50:23.564808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-28T06:18:10.676395+0000","last_change":"2024-05-28T06:18:10.673463+0000","last_active":"2024-05-28T06:18:10.676395+0000","last_peered":"2024-05-28T06:18:10.676395+0000","last_clean":"2024-05-28T06:18:10.676395+0000","last_became_active":"2024-05-28T06:18:10.673150+0000","last_became_peered":"2024-05-28T06:18:10.673150+0000","last_unstale":"2024-05-28T06:18:10.676395+0000","last_undegraded":"2024-05-28T06:18:10.676395+0000","last_fullsized":"2024-05-28T06:18:10.676395+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:43:28.316407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"88'1","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.417570+0000","last_change":"2024-05-28T06:18:09.638887+0000","last_active":"2024-05-28T06:18:12.417570+0000","last_peered":"2024-05-28T06:18:12.417570+0000","last_clean":"2024-05-28T06:18:12.417570+0000","last_became_active":"2024-05-28T06:18:09.638560+0000","last_became_peered":"2024-05-28T06:18:09.638560+0000","last_unstale":"2024-05-28T06:18:12.417570+0000","last_undegraded":"2024-05-28T06:18:12.417570+0000","last_fullsized":"2024-05-28T06:18:12.417570+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T18:02:58.935614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.416958+0000","last_change":"2024-05-28T06:18:10.672644+0000","last_active":"2024-05-28T06:18:12.416958+0000","last_peered":"2024-05-28T06:18:12.416958+0000","last_clean":"2024-05-28T06:18:12.416958+0000","last_became_active":"2024-05-28T06:18:10.672376+0000","last_became_peered":"2024-05-28T06:18:10.672376+0000","last_unstale":"2024-05-28T06:18:12.416958+0000","last_undegraded":"2024-05-28T06:18:12.416958+0000","last_fullsized":"2024-05-28T06:18:12.416958+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:36:04.760890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"88'2","reported_seq":16,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.668844+0000","last_change":"2024-05-28T06:18:09.647691+0000","last_active":"2024-05-28T06:18:11.668844+0000","last_peered":"2024-05-28T06:18:11.668844+0000","last_clean":"2024-05-28T06:18:11.668844+0000","last_became_active":"2024-05-28T06:18:09.647419+0000","last_became_peered":"2024-05-28T06:18:09.647419+0000","last_unstale":"2024-05-28T06:18:11.668844+0000","last_undegraded":"2024-05-28T06:18:11.668844+0000","last_fullsized":"2024-05-28T06:18:11.668844+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:43:27.301370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.670827+0000","last_change":"2024-05-28T06:18:10.660698+0000","last_active":"2024-05-28T06:18:11.670827+0000","last_peered":"2024-05-28T06:18:11.670827+0000","last_clean":"2024-05-28T06:18:11.670827+0000","last_became_active":"2024-05-28T06:18:10.660019+0000","last_became_peered":"2024-05-28T06:18:10.660019+0000","last_unstale":"2024-05-28T06:18:11.670827+0000","last_undegraded":"2024-05-28T06:18:11.670827+0000","last_fullsized":"2024-05-28T06:18:11.670827+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:40:59.653395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.668528+0000","last_change":"2024-05-28T06:18:09.649938+0000","last_active":"2024-05-28T06:18:11.668528+0000","last_peered":"2024-05-28T06:18:11.668528+0000","last_clean":"2024-05-28T06:18:11.668528+0000","last_became_active":"2024-05-28T06:18:09.649011+0000","last_became_peered":"2024-05-28T06:18:09.649011+0000","last_unstale":"2024-05-28T06:18:11.668528+0000","last_undegraded":"2024-05-28T06:18:11.668528+0000","last_fullsized":"2024-05-28T06:18:11.668528+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:10:32.947652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.417302+0000","last_change":"2024-05-28T06:18:10.668617+0000","last_active":"2024-05-28T06:18:12.417302+0000","last_peered":"2024-05-28T06:18:12.417302+0000","last_clean":"2024-05-28T06:18:12.417302+0000","last_became_active":"2024-05-28T06:18:10.668324+0000","last_became_peered":"2024-05-28T06:18:10.668324+0000","last_unstale":"2024-05-28T06:18:12.417302+0000","last_undegraded":"2024-05-28T06:18:12.417302+0000","last_fullsized":"2024-05-28T06:18:12.417302+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:07:04.778166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"88'2","reported_seq":16,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.417711+0000","last_change":"2024-05-28T06:18:09.631399+0000","last_active":"2024-05-28T06:18:12.417711+0000","last_peered":"2024-05-28T06:18:12.417711+0000","last_clean":"2024-05-28T06:18:12.417711+0000","last_became_active":"2024-05-28T06:18:09.630708+0000","last_became_peered":"2024-05-28T06:18:09.630708+0000","last_unstale":"2024-05-28T06:18:12.417711+0000","last_undegraded":"2024-05-28T06:18:12.417711+0000","last_fullsized":"2024-05-28T06:18:12.417711+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:19:09.598059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.420330+0000","last_change":"2024-05-28T06:18:10.677756+0000","last_active":"2024-05-28T06:18:12.420330+0000","last_peered":"2024-05-28T06:18:12.420330+0000","last_clean":"2024-05-28T06:18:12.420330+0000","last_became_active":"2024-05-28T06:18:10.677433+0000","last_became_peered":"2024-05-28T06:18:10.677433+0000","last_unstale":"2024-05-28T06:18:12.420330+0000","last_undegraded":"2024-05-28T06:18:12.420330+0000","last_fullsized":"2024-05-28T06:18:12.420330+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:38:26.397686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.417551+0000","last_change":"2024-05-28T06:18:09.645491+0000","last_active":"2024-05-28T06:18:12.417551+0000","last_peered":"2024-05-28T06:18:12.417551+0000","last_clean":"2024-05-28T06:18:12.417551+0000","last_became_active":"2024-05-28T06:18:09.645198+0000","last_became_peered":"2024-05-28T06:18:09.645198+0000","last_unstale":"2024-05-28T06:18:12.417551+0000","last_undegraded":"2024-05-28T06:18:12.417551+0000","last_fullsized":"2024-05-28T06:18:12.417551+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:58:20.418372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.415731+0000","last_change":"2024-05-28T06:18:10.670797+0000","last_active":"2024-05-28T06:18:12.415731+0000","last_peered":"2024-05-28T06:18:12.415731+0000","last_clean":"2024-05-28T06:18:12.415731+0000","last_became_active":"2024-05-28T06:18:10.670524+0000","last_became_peered":"2024-05-28T06:18:10.670524+0000","last_unstale":"2024-05-28T06:18:12.415731+0000","last_undegraded":"2024-05-28T06:18:12.415731+0000","last_fullsized":"2024-05-28T06:18:12.415731+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:52:46.719566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.416008+0000","last_change":"2024-05-28T06:18:09.638348+0000","last_active":"2024-05-28T06:18:12.416008+0000","last_peered":"2024-05-28T06:18:12.416008+0000","last_clean":"2024-05-28T06:18:12.416008+0000","last_became_active":"2024-05-28T06:18:09.638008+0000","last_became_peered":"2024-05-28T06:18:09.638008+0000","last_unstale":"2024-05-28T06:18:12.416008+0000","last_undegraded":"2024-05-28T06:18:12.416008+0000","last_fullsized":"2024-05-28T06:18:12.416008+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:35:58.418977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.420424+0000","last_change":"2024-05-28T06:18:10.684680+0000","last_active":"2024-05-28T06:18:12.420424+0000","last_peered":"2024-05-28T06:18:12.420424+0000","last_clean":"2024-05-28T06:18:12.420424+0000","last_became_active":"2024-05-28T06:18:10.684197+0000","last_became_peered":"2024-05-28T06:18:10.684197+0000","last_unstale":"2024-05-28T06:18:12.420424+0000","last_undegraded":"2024-05-28T06:18:12.420424+0000","last_fullsized":"2024-05-28T06:18:12.420424+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T18:13:08.105068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.415239+0000","last_change":"2024-05-28T06:18:09.632017+0000","last_active":"2024-05-28T06:18:12.415239+0000","last_peered":"2024-05-28T06:18:12.415239+0000","last_clean":"2024-05-28T06:18:12.415239+0000","last_became_active":"2024-05-28T06:18:09.631229+0000","last_became_peered":"2024-05-28T06:18:09.631229+0000","last_unstale":"2024-05-28T06:18:12.415239+0000","last_undegraded":"2024-05-28T06:18:12.415239+0000","last_fullsized":"2024-05-28T06:18:12.415239+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:17:35.788806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.670828+0000","last_change":"2024-05-28T06:18:10.673882+0000","last_active":"2024-05-28T06:18:11.670828+0000","last_peered":"2024-05-28T06:18:11.670828+0000","last_clean":"2024-05-28T06:18:11.670828+0000","last_became_active":"2024-05-28T06:18:10.672435+0000","last_became_peered":"2024-05-28T06:18:10.672435+0000","last_unstale":"2024-05-28T06:18:11.670828+0000","last_undegraded":"2024-05-28T06:18:11.670828+0000","last_fullsized":"2024-05-28T06:18:11.670828+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:38:26.397686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"88'2","reported_seq":16,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.676296+0000","last_change":"2024-05-28T06:18:09.646324+0000","last_active":"2024-05-28T06:18:11.676296+0000","last_peered":"2024-05-28T06:18:11.676296+0000","last_clean":"2024-05-28T06:18:11.676296+0000","last_became_active":"2024-05-28T06:18:09.646028+0000","last_became_peered":"2024-05-28T06:18:09.646028+0000","last_unstale":"2024-05-28T06:18:11.676296+0000","last_undegraded":"2024-05-28T06:18:11.676296+0000","last_fullsized":"2024-05-28T06:18:11.676296+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:27:14.572603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-28T06:18:10.675571+0000","last_change":"2024-05-28T06:18:10.673086+0000","last_active":"2024-05-28T06:18:10.675571+0000","last_peered":"2024-05-28T06:18:10.675571+0000","last_clean":"2024-05-28T06:18:10.675571+0000","last_became_active":"2024-05-28T06:18:10.672555+0000","last_became_peered":"2024-05-28T06:18:10.672555+0000","last_unstale":"2024-05-28T06:18:10.675571+0000","last_undegraded":"2024-05-28T06:18:10.675571+0000","last_fullsized":"2024-05-28T06:18:10.675571+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:30:38.196353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"89'1","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.676181+0000","last_change":"2024-05-28T06:18:09.646621+0000","last_active":"2024-05-28T06:18:11.676181+0000","last_peered":"2024-05-28T06:18:11.676181+0000","last_clean":"2024-05-28T06:18:11.676181+0000","last_became_active":"2024-05-28T06:18:09.646289+0000","last_became_peered":"2024-05-28T06:18:09.646289+0000","last_unstale":"2024-05-28T06:18:11.676181+0000","last_undegraded":"2024-05-28T06:18:11.676181+0000","last_fullsized":"2024-05-28T06:18:11.676181+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:47:35.036840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.417590+0000","last_change":"2024-05-28T06:18:10.670457+0000","last_active":"2024-05-28T06:18:12.417590+0000","last_peered":"2024-05-28T06:18:12.417590+0000","last_clean":"2024-05-28T06:18:12.417590+0000","last_became_active":"2024-05-28T06:18:10.670145+0000","last_became_peered":"2024-05-28T06:18:10.670145+0000","last_unstale":"2024-05-28T06:18:12.417590+0000","last_undegraded":"2024-05-28T06:18:12.417590+0000","last_fullsized":"2024-05-28T06:18:12.417590+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:19:10.613096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.416460+0000","last_change":"2024-05-28T06:18:09.642139+0000","last_active":"2024-05-28T06:18:12.416460+0000","last_peered":"2024-05-28T06:18:12.416460+0000","last_clean":"2024-05-28T06:18:12.416460+0000","last_became_active":"2024-05-28T06:18:09.641858+0000","last_became_peered":"2024-05-28T06:18:09.641858+0000","last_unstale":"2024-05-28T06:18:12.416460+0000","last_undegraded":"2024-05-28T06:18:12.416460+0000","last_fullsized":"2024-05-28T06:18:12.416460+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:14:09.079098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.670999+0000","last_change":"2024-05-28T06:18:10.666029+0000","last_active":"2024-05-28T06:18:11.670999+0000","last_peered":"2024-05-28T06:18:11.670999+0000","last_clean":"2024-05-28T06:18:11.670999+0000","last_became_active":"2024-05-28T06:18:10.665253+0000","last_became_peered":"2024-05-28T06:18:10.665253+0000","last_unstale":"2024-05-28T06:18:11.670999+0000","last_undegraded":"2024-05-28T06:18:11.670999+0000","last_fullsized":"2024-05-28T06:18:11.670999+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:36:04.760890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.417769+0000","last_change":"2024-05-28T06:18:09.638402+0000","last_active":"2024-05-28T06:18:12.417769+0000","last_peered":"2024-05-28T06:18:12.417769+0000","last_clean":"2024-05-28T06:18:12.417769+0000","last_became_active":"2024-05-28T06:18:09.637801+0000","last_became_peered":"2024-05-28T06:18:09.637801+0000","last_unstale":"2024-05-28T06:18:12.417769+0000","last_undegraded":"2024-05-28T06:18:12.417769+0000","last_fullsized":"2024-05-28T06:18:12.417769+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:45:34.943093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.420016+0000","last_change":"2024-05-28T06:18:10.681723+0000","last_active":"2024-05-28T06:18:12.420016+0000","last_peered":"2024-05-28T06:18:12.420016+0000","last_clean":"2024-05-28T06:18:12.420016+0000","last_became_active":"2024-05-28T06:18:10.681397+0000","last_became_peered":"2024-05-28T06:18:10.681397+0000","last_unstale":"2024-05-28T06:18:12.420016+0000","last_undegraded":"2024-05-28T06:18:12.420016+0000","last_fullsized":"2024-05-28T06:18:12.420016+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:35:48.065461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.668379+0000","last_change":"2024-05-28T06:18:09.642946+0000","last_active":"2024-05-28T06:18:11.668379+0000","last_peered":"2024-05-28T06:18:11.668379+0000","last_clean":"2024-05-28T06:18:11.668379+0000","last_became_active":"2024-05-28T06:18:09.642546+0000","last_became_peered":"2024-05-28T06:18:09.642546+0000","last_unstale":"2024-05-28T06:18:11.668379+0000","last_undegraded":"2024-05-28T06:18:11.668379+0000","last_fullsized":"2024-05-28T06:18:11.668379+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:19:45.186872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.668346+0000","last_change":"2024-05-28T06:18:10.685239+0000","last_active":"2024-05-28T06:18:11.668346+0000","last_peered":"2024-05-28T06:18:11.668346+0000","last_clean":"2024-05-28T06:18:11.668346+0000","last_became_active":"2024-05-28T06:18:10.681969+0000","last_became_peered":"2024-05-28T06:18:10.681969+0000","last_unstale":"2024-05-28T06:18:11.668346+0000","last_undegraded":"2024-05-28T06:18:11.668346+0000","last_fullsized":"2024-05-28T06:18:11.668346+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:38:26.397686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.669511+0000","last_change":"2024-05-28T06:18:09.643305+0000","last_active":"2024-05-28T06:18:11.669511+0000","last_peered":"2024-05-28T06:18:11.669511+0000","last_clean":"2024-05-28T06:18:11.669511+0000","last_became_active":"2024-05-28T06:18:09.642959+0000","last_became_peered":"2024-05-28T06:18:09.642959+0000","last_unstale":"2024-05-28T06:18:11.669511+0000","last_undegraded":"2024-05-28T06:18:11.669511+0000","last_fullsized":"2024-05-28T06:18:11.669511+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:57:25.848771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-28T06:18:10.688312+0000","last_change":"2024-05-28T06:18:10.682613+0000","last_active":"2024-05-28T06:18:10.688312+0000","last_peered":"2024-05-28T06:18:10.688312+0000","last_clean":"2024-05-28T06:18:10.688312+0000","last_became_active":"2024-05-28T06:18:10.682019+0000","last_became_peered":"2024-05-28T06:18:10.682019+0000","last_unstale":"2024-05-28T06:18:10.688312+0000","last_undegraded":"2024-05-28T06:18:10.688312+0000","last_fullsized":"2024-05-28T06:18:10.688312+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:40:59.653395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.416028+0000","last_change":"2024-05-28T06:18:09.637819+0000","last_active":"2024-05-28T06:18:12.416028+0000","last_peered":"2024-05-28T06:18:12.416028+0000","last_clean":"2024-05-28T06:18:12.416028+0000","last_became_active":"2024-05-28T06:18:09.637503+0000","last_became_peered":"2024-05-28T06:18:09.637503+0000","last_unstale":"2024-05-28T06:18:12.416028+0000","last_undegraded":"2024-05-28T06:18:12.416028+0000","last_fullsized":"2024-05-28T06:18:12.416028+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:41:26.564391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.415971+0000","last_change":"2024-05-28T06:18:10.668846+0000","last_active":"2024-05-28T06:18:12.415971+0000","last_peered":"2024-05-28T06:18:12.415971+0000","last_clean":"2024-05-28T06:18:12.415971+0000","last_became_active":"2024-05-28T06:18:10.665081+0000","last_became_peered":"2024-05-28T06:18:10.665081+0000","last_unstale":"2024-05-28T06:18:12.415971+0000","last_undegraded":"2024-05-28T06:18:12.415971+0000","last_fullsized":"2024-05-28T06:18:12.415971+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:35:48.065461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.669169+0000","last_change":"2024-05-28T06:18:09.641057+0000","last_active":"2024-05-28T06:18:11.669169+0000","last_peered":"2024-05-28T06:18:11.669169+0000","last_clean":"2024-05-28T06:18:11.669169+0000","last_became_active":"2024-05-28T06:18:09.640751+0000","last_became_peered":"2024-05-28T06:18:09.640751+0000","last_unstale":"2024-05-28T06:18:11.669169+0000","last_undegraded":"2024-05-28T06:18:11.669169+0000","last_fullsized":"2024-05-28T06:18:11.669169+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:14:09.079098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.675977+0000","last_change":"2024-05-28T06:18:10.673680+0000","last_active":"2024-05-28T06:18:11.675977+0000","last_peered":"2024-05-28T06:18:11.675977+0000","last_clean":"2024-05-28T06:18:11.675977+0000","last_became_active":"2024-05-28T06:18:10.673261+0000","last_became_peered":"2024-05-28T06:18:10.673261+0000","last_unstale":"2024-05-28T06:18:11.675977+0000","last_undegraded":"2024-05-28T06:18:11.675977+0000","last_fullsized":"2024-05-28T06:18:11.675977+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:18:26.300329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"88'1","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-28T06:18:10.687989+0000","last_change":"2024-05-28T06:18:09.649924+0000","last_active":"2024-05-28T06:18:10.687989+0000","last_peered":"2024-05-28T06:18:10.687989+0000","last_clean":"2024-05-28T06:18:10.687989+0000","last_became_active":"2024-05-28T06:18:09.649696+0000","last_became_peered":"2024-05-28T06:18:09.649696+0000","last_unstale":"2024-05-28T06:18:10.687989+0000","last_undegraded":"2024-05-28T06:18:10.687989+0000","last_fullsized":"2024-05-28T06:18:10.687989+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:10:32.947652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":303,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-28T06:18:10.663404+0000","last_change":"2024-05-28T06:04:01.039129+0000","last_active":"2024-05-28T06:18:10.663404+0000","last_peered":"2024-05-28T06:18:10.663404+0000","last_clean":"2024-05-28T06:18:10.663404+0000","last_became_active":"2024-05-28T06:04:01.033906+0000","last_became_peered":"2024-05-28T06:04:01.033906+0000","last_unstale":"2024-05-28T06:18:10.663404+0000","last_undegraded":"2024-05-28T06:18:10.663404+0000","last_fullsized":"2024-05-28T06:18:10.663404+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:03:59.864237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:03:59.864237+0000","last_clean_scrub_stamp":"2024-05-28T06:03:59.864237+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:17:36.440245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575021,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":122992,"kb_used_data":1640,"kb_used_omap":1,"kb_used_meta":121342,"kb_avail":94248848,"statfs":{"total":96636764160,"available":96510820352,"internally_reserved":0,"allocated":1679360,"data_stored":1057712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":124254666},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575024,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":122440,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":121342,"kb_avail":94249400,"statfs":{"total":96636764160,"available":96511385600,"internally_reserved":0,"allocated":1105920,"data_stored":474296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":124254662},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575022,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76616,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94295224,"statfs":{"total":96636764160,"available":96558309376,"internally_reserved":0,"allocated":1110016,"data_stored":475236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575022,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64124,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94307716,"statfs":{"total":96636764160,"available":96571101184,"internally_reserved":0,"allocated":1093632,"data_stored":473602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575024,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":122440,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":121342,"kb_avail":94249400,"statfs":{"total":96636764160,"available":96511385600,"internally_reserved":0,"allocated":1105920,"data_stored":475214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":124254663},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575023,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":122432,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":121342,"kb_avail":94249408,"statfs":{"total":96636764160,"available":96511393792,"internally_reserved":0,"allocated":1097728,"data_stored":474184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124254661},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575023,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":104392,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":103294,"kb_avail":94267448,"statfs":{"total":96636764160,"available":96529866752,"internally_reserved":0,"allocated":1105920,"data_stored":473796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105773509},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575022,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77192,"kb_used_data":1652,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94294648,"statfs":{"total":96636764160,"available":96557719552,"internally_reserved":0,"allocated":1691648,"data_stored":1064004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-28T06:18:20.090 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-28T06:18:20.300 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.302+0000 7f71dd600640 1 Processor -- start 2024-05-28T06:18:20.300 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.302+0000 7f71dd600640 1 -- start start 2024-05-28T06:18:20.301 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.304+0000 7f71dd600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f71d8162940 0x7f71d806eb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:20.301 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.304+0000 7f71dd600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f71d80722d0 0x7f71d80adbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:20.301 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.304+0000 7f71dd600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71d806f590 con 0x7f71d8162d20 2024-05-28T06:18:20.302 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.304+0000 7f71dd600640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f71d806f870 con 0x7f71d80722d0 2024-05-28T06:18:20.302 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.304+0000 7f71dd600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f71d80ad3d0 con 0x7f71d8162940 2024-05-28T06:18:20.302 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.304+0000 7f71dcc00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f71d80722d0 0x7f71d80adbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:20.302 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.304+0000 7f71dcc00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f71d80722d0 0x7f71d80adbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:40088/0 (socket says 172.21.15.96:40088) 2024-05-28T06:18:20.302 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.304+0000 7f71dcc00640 1 -- 172.21.15.96:0/455176397 learned_addr learned my addr 172.21.15.96:0/455176397 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:18:20.302 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.305+0000 7f71dcc00640 1 -- 172.21.15.96:0/455176397 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f71d8162940 msgr2=0x7f71d806eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:18:20.303 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.305+0000 7f71d7e00640 1 --2- 172.21.15.96:0/455176397 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f71d8162940 0x7f71d806eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:20.303 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.305+0000 7f71dcc00640 1 --2- 172.21.15.96:0/455176397 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f71d8162940 0x7f71d806eb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:20.303 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.305+0000 7f71dcc00640 1 -- 172.21.15.96:0/455176397 >> v1:172.21.15.123:6789/0 conn(0x7f71d8162d20 legacy=0x7f71d8071bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:18:20.303 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.305+0000 7f71dcc00640 1 -- 172.21.15.96:0/455176397 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71d806dde0 con 0x7f71d80722d0 2024-05-28T06:18:20.303 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.306+0000 7f71dcc00640 1 --2- 172.21.15.96:0/455176397 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f71d80722d0 0x7f71d80adbb0 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7f71d0009870 tx=0x7f71d002a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e87bd4361166ba37 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:20.304 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.306+0000 7f71d6a00640 1 -- 172.21.15.96:0/455176397 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71d0037070 con 0x7f71d80722d0 2024-05-28T06:18:20.304 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.306+0000 7f71d6a00640 1 -- 172.21.15.96:0/455176397 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71d002ae10 con 0x7f71d80722d0 2024-05-28T06:18:20.304 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.306+0000 7f71d6a00640 1 -- 172.21.15.96:0/455176397 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71d0032660 con 0x7f71d80722d0 2024-05-28T06:18:20.304 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.307+0000 7f71dd600640 1 -- 172.21.15.96:0/455176397 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f71d80722d0 msgr2=0x7f71d80adbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:20.304 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.307+0000 7f71dd600640 1 --2- 172.21.15.96:0/455176397 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f71d80722d0 0x7f71d80adbb0 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7f71d0009870 tx=0x7f71d002a100 comp rx=0 tx=0).stop 2024-05-28T06:18:20.305 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.307+0000 7f71dd600640 1 -- 172.21.15.96:0/455176397 shutdown_connections 2024-05-28T06:18:20.305 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.307+0000 7f71dd600640 1 --2- 172.21.15.96:0/455176397 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f71d80722d0 0x7f71d80adbb0 unknown :-1 s=CLOSED pgs=1424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:20.305 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.307+0000 7f71dd600640 1 --2- 172.21.15.96:0/455176397 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f71d8162940 0x7f71d806eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:20.305 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.307+0000 7f71dd600640 1 -- 172.21.15.96:0/455176397 >> 172.21.15.96:0/455176397 conn(0x7f71d80b1fc0 msgr2=0x7f71d80b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:18:20.305 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.307+0000 7f71dd600640 1 -- 172.21.15.96:0/455176397 shutdown_connections 2024-05-28T06:18:20.305 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.307+0000 7f71dd600640 1 -- 172.21.15.96:0/455176397 wait complete. 2024-05-28T06:18:20.306 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.309+0000 7f71dd600640 1 Processor -- start 2024-05-28T06:18:20.306 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.309+0000 7f71dd600640 1 -- start start 2024-05-28T06:18:20.307 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.310+0000 7f71dd600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f71d80722d0 0x7f71d814c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:20.307 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.310+0000 7f71dd600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f71d8162940 0x7f71d80cf480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:20.307 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.310+0000 7f71dd600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f71d8162d20 0x7f71d814d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:20.308 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.310+0000 7f71dd600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f71d80ae0f0 con 0x7f71d8162d20 2024-05-28T06:18:20.308 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.310+0000 7f71dd600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f71d806f2d0 con 0x7f71d80722d0 2024-05-28T06:18:20.308 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.310+0000 7f71dd600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f71d80ae270 con 0x7f71d8162940 2024-05-28T06:18:20.308 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.310+0000 7f71dcc00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f71d8162d20 0x7f71d814d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:20.308 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.310+0000 7f71dcc00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f71d8162d20 0x7f71d814d0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:40102/0 (socket says 172.21.15.96:40102) 2024-05-28T06:18:20.308 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.310+0000 7f71dcc00640 1 -- 172.21.15.96:0/1287521067 learned_addr learned my addr 172.21.15.96:0/1287521067 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:18:20.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.311+0000 7f71d7e00640 1 --2- 172.21.15.96:0/1287521067 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f71d80722d0 0x7f71d814c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:20.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.311+0000 7f71d7400640 1 --2- 172.21.15.96:0/1287521067 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f71d8162940 0x7f71d80cf480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:20.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.311+0000 7f71dcc00640 1 -- 172.21.15.96:0/1287521067 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f71d8162940 msgr2=0x7f71d80cf480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:20.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.311+0000 7f71dcc00640 1 --2- 172.21.15.96:0/1287521067 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f71d8162940 0x7f71d80cf480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:20.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.311+0000 7f71dcc00640 1 -- 172.21.15.96:0/1287521067 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f71d80722d0 msgr2=0x7f71d814c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:20.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.311+0000 7f71dcc00640 1 --2- 172.21.15.96:0/1287521067 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f71d80722d0 0x7f71d814c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:20.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.311+0000 7f71dcc00640 1 -- 172.21.15.96:0/1287521067 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71d815cc60 con 0x7f71d8162d20 2024-05-28T06:18:20.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.311+0000 7f71d7e00640 1 --2- 172.21.15.96:0/1287521067 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f71d80722d0 0x7f71d814c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:18:20.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.311+0000 7f71d7400640 1 --2- 172.21.15.96:0/1287521067 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f71d8162940 0x7f71d80cf480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:18:20.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.311+0000 7f71dcc00640 1 --2- 172.21.15.96:0/1287521067 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f71d8162d20 0x7f71d814d0e0 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7f71d002a140 tx=0x7f71d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:20.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.311+0000 7f71cfe00640 1 -- 172.21.15.96:0/1287521067 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71d0041070 con 0x7f71d8162d20 2024-05-28T06:18:20.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.312+0000 7f71cfe00640 1 -- 172.21.15.96:0/1287521067 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71d003c680 con 0x7f71d8162d20 2024-05-28T06:18:20.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.312+0000 7f71cfe00640 1 -- 172.21.15.96:0/1287521067 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71d0037040 con 0x7f71d8162d20 2024-05-28T06:18:20.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.312+0000 7f71dd600640 1 -- 172.21.15.96:0/1287521067 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71d815cf40 con 0x7f71d8162d20 2024-05-28T06:18:20.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.312+0000 7f71dd600640 1 -- 172.21.15.96:0/1287521067 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71d815d500 con 0x7f71d8162d20 2024-05-28T06:18:20.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.313+0000 7f71dd600640 1 -- 172.21.15.96:0/1287521067 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71d80689e0 con 0x7f71d8162d20 2024-05-28T06:18:20.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.314+0000 7f71cfe00640 1 -- 172.21.15.96:0/1287521067 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 30) ==== 92621+0+0 (secure 0 0 0) 0x7f71d0044070 con 0x7f71d8162d20 2024-05-28T06:18:20.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.315+0000 7f71cfe00640 1 --2- 172.21.15.96:0/1287521067 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f71b806da90 0x7f71b806ff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:20.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.315+0000 7f71cfe00640 1 -- 172.21.15.96:0/1287521067 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 5737+0+0 (secure 0 0 0) 0x7f71d00ab310 con 0x7f71d8162d20 2024-05-28T06:18:20.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.315+0000 7f71d7e00640 1 --2- 172.21.15.96:0/1287521067 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f71b806da90 0x7f71b806ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:20.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.317+0000 7f71d7e00640 1 --2- 172.21.15.96:0/1287521067 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f71b806da90 0x7f71b806ff50 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f71c8007f50 tx=0x7f71c80074a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:20.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.319+0000 7f71cfe00640 1 -- 172.21.15.96:0/1287521067 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f71d007af60 con 0x7f71d8162d20 2024-05-28T06:18:20.514 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.517+0000 7f71dd600640 1 -- 172.21.15.96:0/1287521067 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f71d80b1b20 con 0x7f71b806da90 2024-05-28T06:18:20.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.535+0000 7f71cfe00640 1 -- 172.21.15.96:0/1287521067 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252066 (secure 0 0 0) 0x7f71d80b1b20 con 0x7f71b806da90 2024-05-28T06:18:20.537 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:18:20.539 INFO:teuthology.orchestra.run.smithi096.stderr:dumped all 2024-05-28T06:18:20.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.540+0000 7f71dd600640 1 -- 172.21.15.96:0/1287521067 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f71b806da90 msgr2=0x7f71b806ff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:20.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.540+0000 7f71dd600640 1 --2- 172.21.15.96:0/1287521067 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f71b806da90 0x7f71b806ff50 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f71c8007f50 tx=0x7f71c80074a0 comp rx=0 tx=0).stop 2024-05-28T06:18:20.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.540+0000 7f71dd600640 1 -- 172.21.15.96:0/1287521067 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f71d8162d20 msgr2=0x7f71d814d0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:20.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.540+0000 7f71dd600640 1 --2- 172.21.15.96:0/1287521067 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f71d8162d20 0x7f71d814d0e0 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7f71d002a140 tx=0x7f71d00046c0 comp rx=0 tx=0).stop 2024-05-28T06:18:20.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.540+0000 7f71dd600640 1 -- 172.21.15.96:0/1287521067 shutdown_connections 2024-05-28T06:18:20.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.540+0000 7f71dd600640 1 --2- 172.21.15.96:0/1287521067 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f71b806da90 0x7f71b806ff50 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:20.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.540+0000 7f71dd600640 1 --2- 172.21.15.96:0/1287521067 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f71d8162d20 0x7f71d814d0e0 unknown :-1 s=CLOSED pgs=1425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:20.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.540+0000 7f71dd600640 1 --2- 172.21.15.96:0/1287521067 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f71d8162940 0x7f71d80cf480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:20.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.540+0000 7f71dd600640 1 --2- 172.21.15.96:0/1287521067 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f71d80722d0 0x7f71d814c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:20.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.540+0000 7f71dd600640 1 -- 172.21.15.96:0/1287521067 >> 172.21.15.96:0/1287521067 conn(0x7f71d80b1fc0 msgr2=0x7f71d80b2b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:18:20.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.540+0000 7f71dd600640 1 -- 172.21.15.96:0/1287521067 shutdown_connections 2024-05-28T06:18:20.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.540+0000 7f71dd600640 1 -- 172.21.15.96:0/1287521067 wait complete. 2024-05-28T06:18:20.556 INFO:teuthology.orchestra.run.smithi096.stdout:{"pg_ready":true,"pg_map":{"version":512,"stamp":"2024-05-28T06:18:19.864887+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":812644,"kb_used_data":9768,"kb_used_omap":12,"kb_used_meta":802739,"kb_avail":754162076,"statfs":{"total":773094113280,"available":772261965824,"internally_reserved":0,"allocated":10002432,"data_stored":4974372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":822005296},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.225503"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.416525+0000","last_change":"2024-05-28T06:18:10.671822+0000","last_active":"2024-05-28T06:18:12.416525+0000","last_peered":"2024-05-28T06:18:12.416525+0000","last_clean":"2024-05-28T06:18:12.416525+0000","last_became_active":"2024-05-28T06:18:10.671188+0000","last_became_peered":"2024-05-28T06:18:10.671188+0000","last_unstale":"2024-05-28T06:18:12.416525+0000","last_undegraded":"2024-05-28T06:18:12.416525+0000","last_fullsized":"2024-05-28T06:18:12.416525+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:40:59.653395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.676054+0000","last_change":"2024-05-28T06:18:10.673470+0000","last_active":"2024-05-28T06:18:11.676054+0000","last_peered":"2024-05-28T06:18:11.676054+0000","last_clean":"2024-05-28T06:18:11.676054+0000","last_became_active":"2024-05-28T06:18:10.673112+0000","last_became_peered":"2024-05-28T06:18:10.673112+0000","last_unstale":"2024-05-28T06:18:11.676054+0000","last_undegraded":"2024-05-28T06:18:11.676054+0000","last_fullsized":"2024-05-28T06:18:11.676054+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:50:18.056574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.420670+0000","last_change":"2024-05-28T06:18:10.684086+0000","last_active":"2024-05-28T06:18:12.420670+0000","last_peered":"2024-05-28T06:18:12.420670+0000","last_clean":"2024-05-28T06:18:12.420670+0000","last_became_active":"2024-05-28T06:18:10.683345+0000","last_became_peered":"2024-05-28T06:18:10.683345+0000","last_unstale":"2024-05-28T06:18:12.420670+0000","last_undegraded":"2024-05-28T06:18:12.420670+0000","last_fullsized":"2024-05-28T06:18:12.420670+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:57:41.516876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.417075+0000","last_change":"2024-05-28T06:18:10.667705+0000","last_active":"2024-05-28T06:18:12.417075+0000","last_peered":"2024-05-28T06:18:12.417075+0000","last_clean":"2024-05-28T06:18:12.417075+0000","last_became_active":"2024-05-28T06:18:10.667442+0000","last_became_peered":"2024-05-28T06:18:10.667442+0000","last_unstale":"2024-05-28T06:18:12.417075+0000","last_undegraded":"2024-05-28T06:18:12.417075+0000","last_fullsized":"2024-05-28T06:18:12.417075+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:51:19.311871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.669932+0000","last_change":"2024-05-28T06:18:10.660951+0000","last_active":"2024-05-28T06:18:11.669932+0000","last_peered":"2024-05-28T06:18:11.669932+0000","last_clean":"2024-05-28T06:18:11.669932+0000","last_became_active":"2024-05-28T06:18:10.660437+0000","last_became_peered":"2024-05-28T06:18:10.660437+0000","last_unstale":"2024-05-28T06:18:11.669932+0000","last_undegraded":"2024-05-28T06:18:11.669932+0000","last_fullsized":"2024-05-28T06:18:11.669932+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:03:21.839812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.419909+0000","last_change":"2024-05-28T06:18:10.680300+0000","last_active":"2024-05-28T06:18:12.419909+0000","last_peered":"2024-05-28T06:18:12.419909+0000","last_clean":"2024-05-28T06:18:12.419909+0000","last_became_active":"2024-05-28T06:18:10.679570+0000","last_became_peered":"2024-05-28T06:18:10.679570+0000","last_unstale":"2024-05-28T06:18:12.419909+0000","last_undegraded":"2024-05-28T06:18:12.419909+0000","last_fullsized":"2024-05-28T06:18:12.419909+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:10:10.001167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.415870+0000","last_change":"2024-05-28T06:18:10.670026+0000","last_active":"2024-05-28T06:18:12.415870+0000","last_peered":"2024-05-28T06:18:12.415870+0000","last_clean":"2024-05-28T06:18:12.415870+0000","last_became_active":"2024-05-28T06:18:10.669644+0000","last_became_peered":"2024-05-28T06:18:10.669644+0000","last_unstale":"2024-05-28T06:18:12.415870+0000","last_undegraded":"2024-05-28T06:18:12.415870+0000","last_fullsized":"2024-05-28T06:18:12.415870+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:04:19.667824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.418225+0000","last_change":"2024-05-28T06:18:10.680190+0000","last_active":"2024-05-28T06:18:12.418225+0000","last_peered":"2024-05-28T06:18:12.418225+0000","last_clean":"2024-05-28T06:18:12.418225+0000","last_became_active":"2024-05-28T06:18:10.679764+0000","last_became_peered":"2024-05-28T06:18:10.679764+0000","last_unstale":"2024-05-28T06:18:12.418225+0000","last_undegraded":"2024-05-28T06:18:12.418225+0000","last_fullsized":"2024-05-28T06:18:12.418225+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:43:28.316407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.420071+0000","last_change":"2024-05-28T06:18:10.685588+0000","last_active":"2024-05-28T06:18:12.420071+0000","last_peered":"2024-05-28T06:18:12.420071+0000","last_clean":"2024-05-28T06:18:12.420071+0000","last_became_active":"2024-05-28T06:18:10.685361+0000","last_became_peered":"2024-05-28T06:18:10.685361+0000","last_unstale":"2024-05-28T06:18:12.420071+0000","last_undegraded":"2024-05-28T06:18:12.420071+0000","last_fullsized":"2024-05-28T06:18:12.420071+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:40:59.653395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.417249+0000","last_change":"2024-05-28T06:18:10.679458+0000","last_active":"2024-05-28T06:18:12.417249+0000","last_peered":"2024-05-28T06:18:12.417249+0000","last_clean":"2024-05-28T06:18:12.417249+0000","last_became_active":"2024-05-28T06:18:10.679143+0000","last_became_peered":"2024-05-28T06:18:10.679143+0000","last_unstale":"2024-05-28T06:18:12.417249+0000","last_undegraded":"2024-05-28T06:18:12.417249+0000","last_fullsized":"2024-05-28T06:18:12.417249+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:42:12.312828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.675913+0000","last_change":"2024-05-28T06:18:10.675557+0000","last_active":"2024-05-28T06:18:11.675913+0000","last_peered":"2024-05-28T06:18:11.675913+0000","last_clean":"2024-05-28T06:18:11.675913+0000","last_became_active":"2024-05-28T06:18:10.675292+0000","last_became_peered":"2024-05-28T06:18:10.675292+0000","last_unstale":"2024-05-28T06:18:11.675913+0000","last_undegraded":"2024-05-28T06:18:11.675913+0000","last_fullsized":"2024-05-28T06:18:11.675913+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:46:26.001440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.416176+0000","last_change":"2024-05-28T06:18:10.677581+0000","last_active":"2024-05-28T06:18:12.416176+0000","last_peered":"2024-05-28T06:18:12.416176+0000","last_clean":"2024-05-28T06:18:12.416176+0000","last_became_active":"2024-05-28T06:18:10.675476+0000","last_became_peered":"2024-05-28T06:18:10.675476+0000","last_unstale":"2024-05-28T06:18:12.416176+0000","last_undegraded":"2024-05-28T06:18:12.416176+0000","last_fullsized":"2024-05-28T06:18:12.416176+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T18:13:08.105068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.418925+0000","last_change":"2024-05-28T06:18:10.673510+0000","last_active":"2024-05-28T06:18:12.418925+0000","last_peered":"2024-05-28T06:18:12.418925+0000","last_clean":"2024-05-28T06:18:12.418925+0000","last_became_active":"2024-05-28T06:18:10.673194+0000","last_became_peered":"2024-05-28T06:18:10.673194+0000","last_unstale":"2024-05-28T06:18:12.418925+0000","last_undegraded":"2024-05-28T06:18:12.418925+0000","last_fullsized":"2024-05-28T06:18:12.418925+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:19:10.613096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.416894+0000","last_change":"2024-05-28T06:18:10.676724+0000","last_active":"2024-05-28T06:18:12.416894+0000","last_peered":"2024-05-28T06:18:12.416894+0000","last_clean":"2024-05-28T06:18:12.416894+0000","last_became_active":"2024-05-28T06:18:10.676457+0000","last_became_peered":"2024-05-28T06:18:10.676457+0000","last_unstale":"2024-05-28T06:18:12.416894+0000","last_undegraded":"2024-05-28T06:18:12.416894+0000","last_fullsized":"2024-05-28T06:18:12.416894+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:50:29.852024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.415556+0000","last_change":"2024-05-28T06:18:10.667679+0000","last_active":"2024-05-28T06:18:12.415556+0000","last_peered":"2024-05-28T06:18:12.415556+0000","last_clean":"2024-05-28T06:18:12.415556+0000","last_became_active":"2024-05-28T06:18:10.667355+0000","last_became_peered":"2024-05-28T06:18:10.667355+0000","last_unstale":"2024-05-28T06:18:12.415556+0000","last_undegraded":"2024-05-28T06:18:12.415556+0000","last_fullsized":"2024-05-28T06:18:12.415556+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:57:41.516876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.669444+0000","last_change":"2024-05-28T06:18:10.676070+0000","last_active":"2024-05-28T06:18:11.669444+0000","last_peered":"2024-05-28T06:18:11.669444+0000","last_clean":"2024-05-28T06:18:11.669444+0000","last_became_active":"2024-05-28T06:18:10.675759+0000","last_became_peered":"2024-05-28T06:18:10.675759+0000","last_unstale":"2024-05-28T06:18:11.669444+0000","last_undegraded":"2024-05-28T06:18:11.669444+0000","last_fullsized":"2024-05-28T06:18:11.669444+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:19:46.201909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.670757+0000","last_change":"2024-05-28T06:18:10.655314+0000","last_active":"2024-05-28T06:18:11.670757+0000","last_peered":"2024-05-28T06:18:11.670757+0000","last_clean":"2024-05-28T06:18:11.670757+0000","last_became_active":"2024-05-28T06:18:10.654580+0000","last_became_peered":"2024-05-28T06:18:10.654580+0000","last_unstale":"2024-05-28T06:18:11.670757+0000","last_undegraded":"2024-05-28T06:18:11.670757+0000","last_fullsized":"2024-05-28T06:18:11.670757+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:10:10.001167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.417262+0000","last_change":"2024-05-28T06:18:10.670459+0000","last_active":"2024-05-28T06:18:12.417262+0000","last_peered":"2024-05-28T06:18:12.417262+0000","last_clean":"2024-05-28T06:18:12.417262+0000","last_became_active":"2024-05-28T06:18:10.670119+0000","last_became_peered":"2024-05-28T06:18:10.670119+0000","last_unstale":"2024-05-28T06:18:12.417262+0000","last_undegraded":"2024-05-28T06:18:12.417262+0000","last_fullsized":"2024-05-28T06:18:12.417262+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:09:41.395379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.668772+0000","last_change":"2024-05-28T06:18:10.662056+0000","last_active":"2024-05-28T06:18:11.668772+0000","last_peered":"2024-05-28T06:18:11.668772+0000","last_clean":"2024-05-28T06:18:11.668772+0000","last_became_active":"2024-05-28T06:18:10.661589+0000","last_became_peered":"2024-05-28T06:18:10.661589+0000","last_unstale":"2024-05-28T06:18:11.668772+0000","last_undegraded":"2024-05-28T06:18:11.668772+0000","last_fullsized":"2024-05-28T06:18:11.668772+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:52:52.857149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.417036+0000","last_change":"2024-05-28T06:18:10.678748+0000","last_active":"2024-05-28T06:18:12.417036+0000","last_peered":"2024-05-28T06:18:12.417036+0000","last_clean":"2024-05-28T06:18:12.417036+0000","last_became_active":"2024-05-28T06:18:10.678444+0000","last_became_peered":"2024-05-28T06:18:10.678444+0000","last_unstale":"2024-05-28T06:18:12.417036+0000","last_undegraded":"2024-05-28T06:18:12.417036+0000","last_fullsized":"2024-05-28T06:18:12.417036+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:30:00.901425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.416302+0000","last_change":"2024-05-28T06:18:10.678968+0000","last_active":"2024-05-28T06:18:12.416302+0000","last_peered":"2024-05-28T06:18:12.416302+0000","last_clean":"2024-05-28T06:18:12.416302+0000","last_became_active":"2024-05-28T06:18:10.678693+0000","last_became_peered":"2024-05-28T06:18:10.678693+0000","last_unstale":"2024-05-28T06:18:12.416302+0000","last_undegraded":"2024-05-28T06:18:12.416302+0000","last_fullsized":"2024-05-28T06:18:12.416302+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:24:00.461410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.675477+0000","last_change":"2024-05-28T06:18:10.688872+0000","last_active":"2024-05-28T06:18:11.675477+0000","last_peered":"2024-05-28T06:18:11.675477+0000","last_clean":"2024-05-28T06:18:11.675477+0000","last_became_active":"2024-05-28T06:18:10.688588+0000","last_became_peered":"2024-05-28T06:18:10.688588+0000","last_unstale":"2024-05-28T06:18:11.675477+0000","last_undegraded":"2024-05-28T06:18:11.675477+0000","last_fullsized":"2024-05-28T06:18:11.675477+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:58:16.081516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.415595+0000","last_change":"2024-05-28T06:18:10.683026+0000","last_active":"2024-05-28T06:18:12.415595+0000","last_peered":"2024-05-28T06:18:12.415595+0000","last_clean":"2024-05-28T06:18:12.415595+0000","last_became_active":"2024-05-28T06:18:10.682741+0000","last_became_peered":"2024-05-28T06:18:10.682741+0000","last_unstale":"2024-05-28T06:18:12.415595+0000","last_undegraded":"2024-05-28T06:18:12.415595+0000","last_fullsized":"2024-05-28T06:18:12.415595+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:43:28.316407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.415588+0000","last_change":"2024-05-28T06:18:10.665505+0000","last_active":"2024-05-28T06:18:12.415588+0000","last_peered":"2024-05-28T06:18:12.415588+0000","last_clean":"2024-05-28T06:18:12.415588+0000","last_became_active":"2024-05-28T06:18:10.665204+0000","last_became_peered":"2024-05-28T06:18:10.665204+0000","last_unstale":"2024-05-28T06:18:12.415588+0000","last_undegraded":"2024-05-28T06:18:12.415588+0000","last_fullsized":"2024-05-28T06:18:12.415588+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:13:16.431201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.676140+0000","last_change":"2024-05-28T06:18:10.684581+0000","last_active":"2024-05-28T06:18:11.676140+0000","last_peered":"2024-05-28T06:18:11.676140+0000","last_clean":"2024-05-28T06:18:11.676140+0000","last_became_active":"2024-05-28T06:18:10.684250+0000","last_became_peered":"2024-05-28T06:18:10.684250+0000","last_unstale":"2024-05-28T06:18:11.676140+0000","last_undegraded":"2024-05-28T06:18:11.676140+0000","last_fullsized":"2024-05-28T06:18:11.676140+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:32:36.064344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.418982+0000","last_change":"2024-05-28T06:18:10.682743+0000","last_active":"2024-05-28T06:18:12.418982+0000","last_peered":"2024-05-28T06:18:12.418982+0000","last_clean":"2024-05-28T06:18:12.418982+0000","last_became_active":"2024-05-28T06:18:10.682451+0000","last_became_peered":"2024-05-28T06:18:10.682451+0000","last_unstale":"2024-05-28T06:18:12.418982+0000","last_undegraded":"2024-05-28T06:18:12.418982+0000","last_fullsized":"2024-05-28T06:18:12.418982+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:13:37.293057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.670303+0000","last_change":"2024-05-28T06:18:10.661976+0000","last_active":"2024-05-28T06:18:11.670303+0000","last_peered":"2024-05-28T06:18:11.670303+0000","last_clean":"2024-05-28T06:18:11.670303+0000","last_became_active":"2024-05-28T06:18:10.661460+0000","last_became_peered":"2024-05-28T06:18:10.661460+0000","last_unstale":"2024-05-28T06:18:11.670303+0000","last_undegraded":"2024-05-28T06:18:11.670303+0000","last_fullsized":"2024-05-28T06:18:11.670303+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:30:38.196353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.417151+0000","last_change":"2024-05-28T06:18:10.666860+0000","last_active":"2024-05-28T06:18:12.417151+0000","last_peered":"2024-05-28T06:18:12.417151+0000","last_clean":"2024-05-28T06:18:12.417151+0000","last_became_active":"2024-05-28T06:18:10.666305+0000","last_became_peered":"2024-05-28T06:18:10.666305+0000","last_unstale":"2024-05-28T06:18:12.417151+0000","last_undegraded":"2024-05-28T06:18:12.417151+0000","last_fullsized":"2024-05-28T06:18:12.417151+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:52:52.857149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.676511+0000","last_change":"2024-05-28T06:18:10.668290+0000","last_active":"2024-05-28T06:18:11.676511+0000","last_peered":"2024-05-28T06:18:11.676511+0000","last_clean":"2024-05-28T06:18:11.676511+0000","last_became_active":"2024-05-28T06:18:10.666561+0000","last_became_peered":"2024-05-28T06:18:10.666561+0000","last_unstale":"2024-05-28T06:18:11.676511+0000","last_undegraded":"2024-05-28T06:18:11.676511+0000","last_fullsized":"2024-05-28T06:18:11.676511+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:24:17.156838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.675494+0000","last_change":"2024-05-28T06:18:10.675895+0000","last_active":"2024-05-28T06:18:11.675494+0000","last_peered":"2024-05-28T06:18:11.675494+0000","last_clean":"2024-05-28T06:18:11.675494+0000","last_became_active":"2024-05-28T06:18:10.675623+0000","last_became_peered":"2024-05-28T06:18:10.675623+0000","last_unstale":"2024-05-28T06:18:11.675494+0000","last_undegraded":"2024-05-28T06:18:11.675494+0000","last_fullsized":"2024-05-28T06:18:11.675494+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:17:45.643378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.669373+0000","last_change":"2024-05-28T06:18:10.667388+0000","last_active":"2024-05-28T06:18:11.669373+0000","last_peered":"2024-05-28T06:18:11.669373+0000","last_clean":"2024-05-28T06:18:11.669373+0000","last_became_active":"2024-05-28T06:18:10.667062+0000","last_became_peered":"2024-05-28T06:18:10.667062+0000","last_unstale":"2024-05-28T06:18:11.669373+0000","last_undegraded":"2024-05-28T06:18:11.669373+0000","last_fullsized":"2024-05-28T06:18:11.669373+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:20:23.727599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.415807+0000","last_change":"2024-05-28T06:18:10.676303+0000","last_active":"2024-05-28T06:18:12.415807+0000","last_peered":"2024-05-28T06:18:12.415807+0000","last_clean":"2024-05-28T06:18:12.415807+0000","last_became_active":"2024-05-28T06:18:10.675998+0000","last_became_peered":"2024-05-28T06:18:10.675998+0000","last_unstale":"2024-05-28T06:18:12.415807+0000","last_undegraded":"2024-05-28T06:18:12.415807+0000","last_fullsized":"2024-05-28T06:18:12.415807+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:10:10.001167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.670796+0000","last_change":"2024-05-28T06:18:10.671446+0000","last_active":"2024-05-28T06:18:11.670796+0000","last_peered":"2024-05-28T06:18:11.670796+0000","last_clean":"2024-05-28T06:18:11.670796+0000","last_became_active":"2024-05-28T06:18:10.671173+0000","last_became_peered":"2024-05-28T06:18:10.671173+0000","last_unstale":"2024-05-28T06:18:11.670796+0000","last_undegraded":"2024-05-28T06:18:11.670796+0000","last_fullsized":"2024-05-28T06:18:11.670796+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:35:48.065461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"88'4","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.417211+0000","last_change":"2024-05-28T06:18:09.643542+0000","last_active":"2024-05-28T06:18:12.417211+0000","last_peered":"2024-05-28T06:18:12.417211+0000","last_clean":"2024-05-28T06:18:12.417211+0000","last_became_active":"2024-05-28T06:18:09.641808+0000","last_became_peered":"2024-05-28T06:18:09.641808+0000","last_unstale":"2024-05-28T06:18:12.417211+0000","last_undegraded":"2024-05-28T06:18:12.417211+0000","last_fullsized":"2024-05-28T06:18:12.417211+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:50:32.547130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.416707+0000","last_change":"2024-05-28T06:18:10.682953+0000","last_active":"2024-05-28T06:18:12.416707+0000","last_peered":"2024-05-28T06:18:12.416707+0000","last_clean":"2024-05-28T06:18:12.416707+0000","last_became_active":"2024-05-28T06:18:10.682621+0000","last_became_peered":"2024-05-28T06:18:10.682621+0000","last_unstale":"2024-05-28T06:18:12.416707+0000","last_undegraded":"2024-05-28T06:18:12.416707+0000","last_fullsized":"2024-05-28T06:18:12.416707+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:38:26.397686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"88'1","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.671160+0000","last_change":"2024-05-28T06:18:09.647214+0000","last_active":"2024-05-28T06:18:11.671160+0000","last_peered":"2024-05-28T06:18:11.671160+0000","last_clean":"2024-05-28T06:18:11.671160+0000","last_became_active":"2024-05-28T06:18:09.646937+0000","last_became_peered":"2024-05-28T06:18:09.646937+0000","last_unstale":"2024-05-28T06:18:11.671160+0000","last_undegraded":"2024-05-28T06:18:11.671160+0000","last_fullsized":"2024-05-28T06:18:11.671160+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:45:34.943093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.417520+0000","last_change":"2024-05-28T06:18:10.681586+0000","last_active":"2024-05-28T06:18:12.417520+0000","last_peered":"2024-05-28T06:18:12.417520+0000","last_clean":"2024-05-28T06:18:12.417520+0000","last_became_active":"2024-05-28T06:18:10.681223+0000","last_became_peered":"2024-05-28T06:18:10.681223+0000","last_unstale":"2024-05-28T06:18:12.417520+0000","last_undegraded":"2024-05-28T06:18:12.417520+0000","last_fullsized":"2024-05-28T06:18:12.417520+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:17:36.803843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"88'2","reported_seq":16,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.668839+0000","last_change":"2024-05-28T06:18:09.646080+0000","last_active":"2024-05-28T06:18:11.668839+0000","last_peered":"2024-05-28T06:18:11.668839+0000","last_clean":"2024-05-28T06:18:11.668839+0000","last_became_active":"2024-05-28T06:18:09.645807+0000","last_became_peered":"2024-05-28T06:18:09.645807+0000","last_unstale":"2024-05-28T06:18:11.668839+0000","last_undegraded":"2024-05-28T06:18:11.668839+0000","last_fullsized":"2024-05-28T06:18:11.668839+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:30:37.181316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.418946+0000","last_change":"2024-05-28T06:18:10.677395+0000","last_active":"2024-05-28T06:18:12.418946+0000","last_peered":"2024-05-28T06:18:12.418946+0000","last_clean":"2024-05-28T06:18:12.418946+0000","last_became_active":"2024-05-28T06:18:10.677100+0000","last_became_peered":"2024-05-28T06:18:10.677100+0000","last_unstale":"2024-05-28T06:18:12.418946+0000","last_undegraded":"2024-05-28T06:18:12.418946+0000","last_fullsized":"2024-05-28T06:18:12.418946+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:21:29.571735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"88'2","reported_seq":16,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.668500+0000","last_change":"2024-05-28T06:18:09.643971+0000","last_active":"2024-05-28T06:18:11.668500+0000","last_peered":"2024-05-28T06:18:11.668500+0000","last_clean":"2024-05-28T06:18:11.668500+0000","last_became_active":"2024-05-28T06:18:09.643691+0000","last_became_peered":"2024-05-28T06:18:09.643691+0000","last_unstale":"2024-05-28T06:18:11.668500+0000","last_undegraded":"2024-05-28T06:18:11.668500+0000","last_fullsized":"2024-05-28T06:18:11.668500+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:03:20.824775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.675683+0000","last_change":"2024-05-28T06:18:10.667892+0000","last_active":"2024-05-28T06:18:11.675683+0000","last_peered":"2024-05-28T06:18:11.675683+0000","last_clean":"2024-05-28T06:18:11.675683+0000","last_became_active":"2024-05-28T06:18:10.667622+0000","last_became_peered":"2024-05-28T06:18:10.667622+0000","last_unstale":"2024-05-28T06:18:11.675683+0000","last_undegraded":"2024-05-28T06:18:11.675683+0000","last_fullsized":"2024-05-28T06:18:11.675683+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:15:40.941909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"88'1","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.670434+0000","last_change":"2024-05-28T06:18:09.645873+0000","last_active":"2024-05-28T06:18:11.670434+0000","last_peered":"2024-05-28T06:18:11.670434+0000","last_clean":"2024-05-28T06:18:11.670434+0000","last_became_active":"2024-05-28T06:18:09.645561+0000","last_became_peered":"2024-05-28T06:18:09.645561+0000","last_unstale":"2024-05-28T06:18:11.670434+0000","last_undegraded":"2024-05-28T06:18:11.670434+0000","last_fullsized":"2024-05-28T06:18:11.670434+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:33:21.519096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.670041+0000","last_change":"2024-05-28T06:18:10.653334+0000","last_active":"2024-05-28T06:18:11.670041+0000","last_peered":"2024-05-28T06:18:11.670041+0000","last_clean":"2024-05-28T06:18:11.670041+0000","last_became_active":"2024-05-28T06:18:10.652195+0000","last_became_peered":"2024-05-28T06:18:10.652195+0000","last_unstale":"2024-05-28T06:18:11.670041+0000","last_undegraded":"2024-05-28T06:18:11.670041+0000","last_fullsized":"2024-05-28T06:18:11.670041+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:10:33.962690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"88'1","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.669219+0000","last_change":"2024-05-28T06:18:09.641147+0000","last_active":"2024-05-28T06:18:11.669219+0000","last_peered":"2024-05-28T06:18:11.669219+0000","last_clean":"2024-05-28T06:18:11.669219+0000","last_became_active":"2024-05-28T06:18:09.640830+0000","last_became_peered":"2024-05-28T06:18:09.640830+0000","last_unstale":"2024-05-28T06:18:11.669219+0000","last_undegraded":"2024-05-28T06:18:11.669219+0000","last_fullsized":"2024-05-28T06:18:11.669219+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:44:26.345831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.416654+0000","last_change":"2024-05-28T06:18:10.684510+0000","last_active":"2024-05-28T06:18:12.416654+0000","last_peered":"2024-05-28T06:18:12.416654+0000","last_clean":"2024-05-28T06:18:12.416654+0000","last_became_active":"2024-05-28T06:18:10.684209+0000","last_became_peered":"2024-05-28T06:18:10.684209+0000","last_unstale":"2024-05-28T06:18:12.416654+0000","last_undegraded":"2024-05-28T06:18:12.416654+0000","last_fullsized":"2024-05-28T06:18:12.416654+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:42:12.312828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.667919+0000","last_change":"2024-05-28T06:18:09.647844+0000","last_active":"2024-05-28T06:18:11.667919+0000","last_peered":"2024-05-28T06:18:11.667919+0000","last_clean":"2024-05-28T06:18:11.667919+0000","last_became_active":"2024-05-28T06:18:09.647527+0000","last_became_peered":"2024-05-28T06:18:09.647527+0000","last_unstale":"2024-05-28T06:18:11.667919+0000","last_undegraded":"2024-05-28T06:18:11.667919+0000","last_fullsized":"2024-05-28T06:18:11.667919+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:20:22.712562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.416481+0000","last_change":"2024-05-28T06:18:10.667540+0000","last_active":"2024-05-28T06:18:12.416481+0000","last_peered":"2024-05-28T06:18:12.416481+0000","last_clean":"2024-05-28T06:18:12.416481+0000","last_became_active":"2024-05-28T06:18:10.667232+0000","last_became_peered":"2024-05-28T06:18:10.667232+0000","last_unstale":"2024-05-28T06:18:12.416481+0000","last_undegraded":"2024-05-28T06:18:12.416481+0000","last_fullsized":"2024-05-28T06:18:12.416481+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:45:48.725150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.675347+0000","last_change":"2024-05-28T06:18:09.648064+0000","last_active":"2024-05-28T06:18:11.675347+0000","last_peered":"2024-05-28T06:18:11.675347+0000","last_clean":"2024-05-28T06:18:11.675347+0000","last_became_active":"2024-05-28T06:18:09.647787+0000","last_became_peered":"2024-05-28T06:18:09.647787+0000","last_unstale":"2024-05-28T06:18:11.675347+0000","last_undegraded":"2024-05-28T06:18:11.675347+0000","last_fullsized":"2024-05-28T06:18:11.675347+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:26:13.635159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.418311+0000","last_change":"2024-05-28T06:18:10.674109+0000","last_active":"2024-05-28T06:18:12.418311+0000","last_peered":"2024-05-28T06:18:12.418311+0000","last_clean":"2024-05-28T06:18:12.418311+0000","last_became_active":"2024-05-28T06:18:10.673531+0000","last_became_peered":"2024-05-28T06:18:10.673531+0000","last_unstale":"2024-05-28T06:18:12.418311+0000","last_undegraded":"2024-05-28T06:18:12.418311+0000","last_fullsized":"2024-05-28T06:18:12.418311+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:15:33.611029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.417249+0000","last_change":"2024-05-28T06:18:09.643958+0000","last_active":"2024-05-28T06:18:12.417249+0000","last_peered":"2024-05-28T06:18:12.417249+0000","last_clean":"2024-05-28T06:18:12.417249+0000","last_became_active":"2024-05-28T06:18:09.643658+0000","last_became_peered":"2024-05-28T06:18:09.643658+0000","last_unstale":"2024-05-28T06:18:12.417249+0000","last_undegraded":"2024-05-28T06:18:12.417249+0000","last_fullsized":"2024-05-28T06:18:12.417249+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:37:17.604820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.420418+0000","last_change":"2024-05-28T06:18:10.680643+0000","last_active":"2024-05-28T06:18:12.420418+0000","last_peered":"2024-05-28T06:18:12.420418+0000","last_clean":"2024-05-28T06:18:12.420418+0000","last_became_active":"2024-05-28T06:18:10.677505+0000","last_became_peered":"2024-05-28T06:18:10.677505+0000","last_unstale":"2024-05-28T06:18:12.420418+0000","last_undegraded":"2024-05-28T06:18:12.420418+0000","last_fullsized":"2024-05-28T06:18:12.420418+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:36:04.760890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.417061+0000","last_change":"2024-05-28T06:18:09.645952+0000","last_active":"2024-05-28T06:18:12.417061+0000","last_peered":"2024-05-28T06:18:12.417061+0000","last_clean":"2024-05-28T06:18:12.417061+0000","last_became_active":"2024-05-28T06:18:09.644892+0000","last_became_peered":"2024-05-28T06:18:09.644892+0000","last_unstale":"2024-05-28T06:18:12.417061+0000","last_undegraded":"2024-05-28T06:18:12.417061+0000","last_fullsized":"2024-05-28T06:18:12.417061+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:48:38.824030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.668434+0000","last_change":"2024-05-28T06:18:10.678535+0000","last_active":"2024-05-28T06:18:11.668434+0000","last_peered":"2024-05-28T06:18:11.668434+0000","last_clean":"2024-05-28T06:18:11.668434+0000","last_became_active":"2024-05-28T06:18:10.678261+0000","last_became_peered":"2024-05-28T06:18:10.678261+0000","last_unstale":"2024-05-28T06:18:11.668434+0000","last_undegraded":"2024-05-28T06:18:11.668434+0000","last_fullsized":"2024-05-28T06:18:11.668434+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T18:13:08.105068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"88'3","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.416171+0000","last_change":"2024-05-28T06:18:09.648424+0000","last_active":"2024-05-28T06:18:12.416171+0000","last_peered":"2024-05-28T06:18:12.416171+0000","last_clean":"2024-05-28T06:18:12.416171+0000","last_became_active":"2024-05-28T06:18:09.648138+0000","last_became_peered":"2024-05-28T06:18:09.648138+0000","last_unstale":"2024-05-28T06:18:12.416171+0000","last_undegraded":"2024-05-28T06:18:12.416171+0000","last_fullsized":"2024-05-28T06:18:12.416171+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:33:21.519096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.670379+0000","last_change":"2024-05-28T06:18:10.666292+0000","last_active":"2024-05-28T06:18:11.670379+0000","last_peered":"2024-05-28T06:18:11.670379+0000","last_clean":"2024-05-28T06:18:11.670379+0000","last_became_active":"2024-05-28T06:18:10.665709+0000","last_became_peered":"2024-05-28T06:18:10.665709+0000","last_unstale":"2024-05-28T06:18:11.670379+0000","last_undegraded":"2024-05-28T06:18:11.670379+0000","last_fullsized":"2024-05-28T06:18:11.670379+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:43:28.316407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"88'6","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.419335+0000","last_change":"2024-05-28T06:18:09.645431+0000","last_active":"2024-05-28T06:18:12.419335+0000","last_peered":"2024-05-28T06:18:12.419335+0000","last_clean":"2024-05-28T06:18:12.419335+0000","last_became_active":"2024-05-28T06:18:09.645141+0000","last_became_peered":"2024-05-28T06:18:09.645141+0000","last_unstale":"2024-05-28T06:18:12.419335+0000","last_undegraded":"2024-05-28T06:18:12.419335+0000","last_fullsized":"2024-05-28T06:18:12.419335+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:33:46.549475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.668301+0000","last_change":"2024-05-28T06:18:10.677586+0000","last_active":"2024-05-28T06:18:11.668301+0000","last_peered":"2024-05-28T06:18:11.668301+0000","last_clean":"2024-05-28T06:18:11.668301+0000","last_became_active":"2024-05-28T06:18:10.677186+0000","last_became_peered":"2024-05-28T06:18:10.677186+0000","last_unstale":"2024-05-28T06:18:11.668301+0000","last_undegraded":"2024-05-28T06:18:11.668301+0000","last_fullsized":"2024-05-28T06:18:11.668301+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:36:04.760890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"88'2","reported_seq":16,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.675765+0000","last_change":"2024-05-28T06:18:09.647148+0000","last_active":"2024-05-28T06:18:11.675765+0000","last_peered":"2024-05-28T06:18:11.675765+0000","last_clean":"2024-05-28T06:18:11.675765+0000","last_became_active":"2024-05-28T06:18:09.646882+0000","last_became_peered":"2024-05-28T06:18:09.646882+0000","last_unstale":"2024-05-28T06:18:11.675765+0000","last_undegraded":"2024-05-28T06:18:11.675765+0000","last_fullsized":"2024-05-28T06:18:11.675765+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:17:12.740244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.419382+0000","last_change":"2024-05-28T06:18:10.687447+0000","last_active":"2024-05-28T06:18:12.419382+0000","last_peered":"2024-05-28T06:18:12.419382+0000","last_clean":"2024-05-28T06:18:12.419382+0000","last_became_active":"2024-05-28T06:18:10.687161+0000","last_became_peered":"2024-05-28T06:18:10.687161+0000","last_unstale":"2024-05-28T06:18:12.419382+0000","last_undegraded":"2024-05-28T06:18:12.419382+0000","last_fullsized":"2024-05-28T06:18:12.419382+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:34:23.542328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.417055+0000","last_change":"2024-05-28T06:18:09.647907+0000","last_active":"2024-05-28T06:18:12.417055+0000","last_peered":"2024-05-28T06:18:12.417055+0000","last_clean":"2024-05-28T06:18:12.417055+0000","last_became_active":"2024-05-28T06:18:09.647240+0000","last_became_peered":"2024-05-28T06:18:09.647240+0000","last_unstale":"2024-05-28T06:18:12.417055+0000","last_undegraded":"2024-05-28T06:18:12.417055+0000","last_fullsized":"2024-05-28T06:18:12.417055+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:21:28.556698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.675523+0000","last_change":"2024-05-28T06:18:10.670763+0000","last_active":"2024-05-28T06:18:11.675523+0000","last_peered":"2024-05-28T06:18:11.675523+0000","last_clean":"2024-05-28T06:18:11.675523+0000","last_became_active":"2024-05-28T06:18:10.670493+0000","last_became_peered":"2024-05-28T06:18:10.670493+0000","last_unstale":"2024-05-28T06:18:11.675523+0000","last_undegraded":"2024-05-28T06:18:11.675523+0000","last_fullsized":"2024-05-28T06:18:11.675523+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:23:19.474029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.419186+0000","last_change":"2024-05-28T06:18:09.645407+0000","last_active":"2024-05-28T06:18:12.419186+0000","last_peered":"2024-05-28T06:18:12.419186+0000","last_clean":"2024-05-28T06:18:12.419186+0000","last_became_active":"2024-05-28T06:18:09.644308+0000","last_became_peered":"2024-05-28T06:18:09.644308+0000","last_unstale":"2024-05-28T06:18:12.419186+0000","last_undegraded":"2024-05-28T06:18:12.419186+0000","last_fullsized":"2024-05-28T06:18:12.419186+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T18:02:58.935614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.675215+0000","last_change":"2024-05-28T06:18:10.674096+0000","last_active":"2024-05-28T06:18:11.675215+0000","last_peered":"2024-05-28T06:18:11.675215+0000","last_clean":"2024-05-28T06:18:11.675215+0000","last_became_active":"2024-05-28T06:18:10.673401+0000","last_became_peered":"2024-05-28T06:18:10.673401+0000","last_unstale":"2024-05-28T06:18:11.675215+0000","last_undegraded":"2024-05-28T06:18:11.675215+0000","last_fullsized":"2024-05-28T06:18:11.675215+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:29:01.599345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"88'8","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.667571+0000","last_change":"2024-05-28T06:18:09.643338+0000","last_active":"2024-05-28T06:18:11.667571+0000","last_peered":"2024-05-28T06:18:11.667571+0000","last_clean":"2024-05-28T06:18:11.667571+0000","last_became_active":"2024-05-28T06:18:09.643035+0000","last_became_peered":"2024-05-28T06:18:09.643035+0000","last_unstale":"2024-05-28T06:18:11.667571+0000","last_undegraded":"2024-05-28T06:18:11.667571+0000","last_fullsized":"2024-05-28T06:18:11.667571+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:50:23.564808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.419466+0000","last_change":"2024-05-28T06:18:10.673463+0000","last_active":"2024-05-28T06:18:12.419466+0000","last_peered":"2024-05-28T06:18:12.419466+0000","last_clean":"2024-05-28T06:18:12.419466+0000","last_became_active":"2024-05-28T06:18:10.673150+0000","last_became_peered":"2024-05-28T06:18:10.673150+0000","last_unstale":"2024-05-28T06:18:12.419466+0000","last_undegraded":"2024-05-28T06:18:12.419466+0000","last_fullsized":"2024-05-28T06:18:12.419466+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:45:03.376863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"88'1","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.417570+0000","last_change":"2024-05-28T06:18:09.638887+0000","last_active":"2024-05-28T06:18:12.417570+0000","last_peered":"2024-05-28T06:18:12.417570+0000","last_clean":"2024-05-28T06:18:12.417570+0000","last_became_active":"2024-05-28T06:18:09.638560+0000","last_became_peered":"2024-05-28T06:18:09.638560+0000","last_unstale":"2024-05-28T06:18:12.417570+0000","last_undegraded":"2024-05-28T06:18:12.417570+0000","last_fullsized":"2024-05-28T06:18:12.417570+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T18:02:58.935614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.416958+0000","last_change":"2024-05-28T06:18:10.672644+0000","last_active":"2024-05-28T06:18:12.416958+0000","last_peered":"2024-05-28T06:18:12.416958+0000","last_clean":"2024-05-28T06:18:12.416958+0000","last_became_active":"2024-05-28T06:18:10.672376+0000","last_became_peered":"2024-05-28T06:18:10.672376+0000","last_unstale":"2024-05-28T06:18:12.416958+0000","last_undegraded":"2024-05-28T06:18:12.416958+0000","last_fullsized":"2024-05-28T06:18:12.416958+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:36:04.760890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"88'2","reported_seq":16,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.668844+0000","last_change":"2024-05-28T06:18:09.647691+0000","last_active":"2024-05-28T06:18:11.668844+0000","last_peered":"2024-05-28T06:18:11.668844+0000","last_clean":"2024-05-28T06:18:11.668844+0000","last_became_active":"2024-05-28T06:18:09.647419+0000","last_became_peered":"2024-05-28T06:18:09.647419+0000","last_unstale":"2024-05-28T06:18:11.668844+0000","last_undegraded":"2024-05-28T06:18:11.668844+0000","last_fullsized":"2024-05-28T06:18:11.668844+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:43:27.301370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.670827+0000","last_change":"2024-05-28T06:18:10.660698+0000","last_active":"2024-05-28T06:18:11.670827+0000","last_peered":"2024-05-28T06:18:11.670827+0000","last_clean":"2024-05-28T06:18:11.670827+0000","last_became_active":"2024-05-28T06:18:10.660019+0000","last_became_peered":"2024-05-28T06:18:10.660019+0000","last_unstale":"2024-05-28T06:18:11.670827+0000","last_undegraded":"2024-05-28T06:18:11.670827+0000","last_fullsized":"2024-05-28T06:18:11.670827+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:40:59.653395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.668528+0000","last_change":"2024-05-28T06:18:09.649938+0000","last_active":"2024-05-28T06:18:11.668528+0000","last_peered":"2024-05-28T06:18:11.668528+0000","last_clean":"2024-05-28T06:18:11.668528+0000","last_became_active":"2024-05-28T06:18:09.649011+0000","last_became_peered":"2024-05-28T06:18:09.649011+0000","last_unstale":"2024-05-28T06:18:11.668528+0000","last_undegraded":"2024-05-28T06:18:11.668528+0000","last_fullsized":"2024-05-28T06:18:11.668528+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:10:32.947652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.417302+0000","last_change":"2024-05-28T06:18:10.668617+0000","last_active":"2024-05-28T06:18:12.417302+0000","last_peered":"2024-05-28T06:18:12.417302+0000","last_clean":"2024-05-28T06:18:12.417302+0000","last_became_active":"2024-05-28T06:18:10.668324+0000","last_became_peered":"2024-05-28T06:18:10.668324+0000","last_unstale":"2024-05-28T06:18:12.417302+0000","last_undegraded":"2024-05-28T06:18:12.417302+0000","last_fullsized":"2024-05-28T06:18:12.417302+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:07:04.778166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"88'2","reported_seq":16,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.417711+0000","last_change":"2024-05-28T06:18:09.631399+0000","last_active":"2024-05-28T06:18:12.417711+0000","last_peered":"2024-05-28T06:18:12.417711+0000","last_clean":"2024-05-28T06:18:12.417711+0000","last_became_active":"2024-05-28T06:18:09.630708+0000","last_became_peered":"2024-05-28T06:18:09.630708+0000","last_unstale":"2024-05-28T06:18:12.417711+0000","last_undegraded":"2024-05-28T06:18:12.417711+0000","last_fullsized":"2024-05-28T06:18:12.417711+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:19:09.598059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.420330+0000","last_change":"2024-05-28T06:18:10.677756+0000","last_active":"2024-05-28T06:18:12.420330+0000","last_peered":"2024-05-28T06:18:12.420330+0000","last_clean":"2024-05-28T06:18:12.420330+0000","last_became_active":"2024-05-28T06:18:10.677433+0000","last_became_peered":"2024-05-28T06:18:10.677433+0000","last_unstale":"2024-05-28T06:18:12.420330+0000","last_undegraded":"2024-05-28T06:18:12.420330+0000","last_fullsized":"2024-05-28T06:18:12.420330+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:38:26.397686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.417551+0000","last_change":"2024-05-28T06:18:09.645491+0000","last_active":"2024-05-28T06:18:12.417551+0000","last_peered":"2024-05-28T06:18:12.417551+0000","last_clean":"2024-05-28T06:18:12.417551+0000","last_became_active":"2024-05-28T06:18:09.645198+0000","last_became_peered":"2024-05-28T06:18:09.645198+0000","last_unstale":"2024-05-28T06:18:12.417551+0000","last_undegraded":"2024-05-28T06:18:12.417551+0000","last_fullsized":"2024-05-28T06:18:12.417551+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:58:20.418372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.415731+0000","last_change":"2024-05-28T06:18:10.670797+0000","last_active":"2024-05-28T06:18:12.415731+0000","last_peered":"2024-05-28T06:18:12.415731+0000","last_clean":"2024-05-28T06:18:12.415731+0000","last_became_active":"2024-05-28T06:18:10.670524+0000","last_became_peered":"2024-05-28T06:18:10.670524+0000","last_unstale":"2024-05-28T06:18:12.415731+0000","last_undegraded":"2024-05-28T06:18:12.415731+0000","last_fullsized":"2024-05-28T06:18:12.415731+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:52:46.719566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.416008+0000","last_change":"2024-05-28T06:18:09.638348+0000","last_active":"2024-05-28T06:18:12.416008+0000","last_peered":"2024-05-28T06:18:12.416008+0000","last_clean":"2024-05-28T06:18:12.416008+0000","last_became_active":"2024-05-28T06:18:09.638008+0000","last_became_peered":"2024-05-28T06:18:09.638008+0000","last_unstale":"2024-05-28T06:18:12.416008+0000","last_undegraded":"2024-05-28T06:18:12.416008+0000","last_fullsized":"2024-05-28T06:18:12.416008+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:35:58.418977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.420424+0000","last_change":"2024-05-28T06:18:10.684680+0000","last_active":"2024-05-28T06:18:12.420424+0000","last_peered":"2024-05-28T06:18:12.420424+0000","last_clean":"2024-05-28T06:18:12.420424+0000","last_became_active":"2024-05-28T06:18:10.684197+0000","last_became_peered":"2024-05-28T06:18:10.684197+0000","last_unstale":"2024-05-28T06:18:12.420424+0000","last_undegraded":"2024-05-28T06:18:12.420424+0000","last_fullsized":"2024-05-28T06:18:12.420424+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T18:13:08.105068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.415239+0000","last_change":"2024-05-28T06:18:09.632017+0000","last_active":"2024-05-28T06:18:12.415239+0000","last_peered":"2024-05-28T06:18:12.415239+0000","last_clean":"2024-05-28T06:18:12.415239+0000","last_became_active":"2024-05-28T06:18:09.631229+0000","last_became_peered":"2024-05-28T06:18:09.631229+0000","last_unstale":"2024-05-28T06:18:12.415239+0000","last_undegraded":"2024-05-28T06:18:12.415239+0000","last_fullsized":"2024-05-28T06:18:12.415239+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:17:35.788806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.670828+0000","last_change":"2024-05-28T06:18:10.673882+0000","last_active":"2024-05-28T06:18:11.670828+0000","last_peered":"2024-05-28T06:18:11.670828+0000","last_clean":"2024-05-28T06:18:11.670828+0000","last_became_active":"2024-05-28T06:18:10.672435+0000","last_became_peered":"2024-05-28T06:18:10.672435+0000","last_unstale":"2024-05-28T06:18:11.670828+0000","last_undegraded":"2024-05-28T06:18:11.670828+0000","last_fullsized":"2024-05-28T06:18:11.670828+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:38:26.397686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"88'2","reported_seq":16,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.676296+0000","last_change":"2024-05-28T06:18:09.646324+0000","last_active":"2024-05-28T06:18:11.676296+0000","last_peered":"2024-05-28T06:18:11.676296+0000","last_clean":"2024-05-28T06:18:11.676296+0000","last_became_active":"2024-05-28T06:18:09.646028+0000","last_became_peered":"2024-05-28T06:18:09.646028+0000","last_unstale":"2024-05-28T06:18:11.676296+0000","last_undegraded":"2024-05-28T06:18:11.676296+0000","last_fullsized":"2024-05-28T06:18:11.676296+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:27:14.572603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.419448+0000","last_change":"2024-05-28T06:18:10.673086+0000","last_active":"2024-05-28T06:18:12.419448+0000","last_peered":"2024-05-28T06:18:12.419448+0000","last_clean":"2024-05-28T06:18:12.419448+0000","last_became_active":"2024-05-28T06:18:10.672555+0000","last_became_peered":"2024-05-28T06:18:10.672555+0000","last_unstale":"2024-05-28T06:18:12.419448+0000","last_undegraded":"2024-05-28T06:18:12.419448+0000","last_fullsized":"2024-05-28T06:18:12.419448+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:48:55.924965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"89'1","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.676181+0000","last_change":"2024-05-28T06:18:09.646621+0000","last_active":"2024-05-28T06:18:11.676181+0000","last_peered":"2024-05-28T06:18:11.676181+0000","last_clean":"2024-05-28T06:18:11.676181+0000","last_became_active":"2024-05-28T06:18:09.646289+0000","last_became_peered":"2024-05-28T06:18:09.646289+0000","last_unstale":"2024-05-28T06:18:11.676181+0000","last_undegraded":"2024-05-28T06:18:11.676181+0000","last_fullsized":"2024-05-28T06:18:11.676181+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:47:35.036840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.417590+0000","last_change":"2024-05-28T06:18:10.670457+0000","last_active":"2024-05-28T06:18:12.417590+0000","last_peered":"2024-05-28T06:18:12.417590+0000","last_clean":"2024-05-28T06:18:12.417590+0000","last_became_active":"2024-05-28T06:18:10.670145+0000","last_became_peered":"2024-05-28T06:18:10.670145+0000","last_unstale":"2024-05-28T06:18:12.417590+0000","last_undegraded":"2024-05-28T06:18:12.417590+0000","last_fullsized":"2024-05-28T06:18:12.417590+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:19:10.613096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.416460+0000","last_change":"2024-05-28T06:18:09.642139+0000","last_active":"2024-05-28T06:18:12.416460+0000","last_peered":"2024-05-28T06:18:12.416460+0000","last_clean":"2024-05-28T06:18:12.416460+0000","last_became_active":"2024-05-28T06:18:09.641858+0000","last_became_peered":"2024-05-28T06:18:09.641858+0000","last_unstale":"2024-05-28T06:18:12.416460+0000","last_undegraded":"2024-05-28T06:18:12.416460+0000","last_fullsized":"2024-05-28T06:18:12.416460+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:14:09.079098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.670999+0000","last_change":"2024-05-28T06:18:10.666029+0000","last_active":"2024-05-28T06:18:11.670999+0000","last_peered":"2024-05-28T06:18:11.670999+0000","last_clean":"2024-05-28T06:18:11.670999+0000","last_became_active":"2024-05-28T06:18:10.665253+0000","last_became_peered":"2024-05-28T06:18:10.665253+0000","last_unstale":"2024-05-28T06:18:11.670999+0000","last_undegraded":"2024-05-28T06:18:11.670999+0000","last_fullsized":"2024-05-28T06:18:11.670999+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:36:04.760890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.417769+0000","last_change":"2024-05-28T06:18:09.638402+0000","last_active":"2024-05-28T06:18:12.417769+0000","last_peered":"2024-05-28T06:18:12.417769+0000","last_clean":"2024-05-28T06:18:12.417769+0000","last_became_active":"2024-05-28T06:18:09.637801+0000","last_became_peered":"2024-05-28T06:18:09.637801+0000","last_unstale":"2024-05-28T06:18:12.417769+0000","last_undegraded":"2024-05-28T06:18:12.417769+0000","last_fullsized":"2024-05-28T06:18:12.417769+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:45:34.943093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.420016+0000","last_change":"2024-05-28T06:18:10.681723+0000","last_active":"2024-05-28T06:18:12.420016+0000","last_peered":"2024-05-28T06:18:12.420016+0000","last_clean":"2024-05-28T06:18:12.420016+0000","last_became_active":"2024-05-28T06:18:10.681397+0000","last_became_peered":"2024-05-28T06:18:10.681397+0000","last_unstale":"2024-05-28T06:18:12.420016+0000","last_undegraded":"2024-05-28T06:18:12.420016+0000","last_fullsized":"2024-05-28T06:18:12.420016+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:35:48.065461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.668379+0000","last_change":"2024-05-28T06:18:09.642946+0000","last_active":"2024-05-28T06:18:11.668379+0000","last_peered":"2024-05-28T06:18:11.668379+0000","last_clean":"2024-05-28T06:18:11.668379+0000","last_became_active":"2024-05-28T06:18:09.642546+0000","last_became_peered":"2024-05-28T06:18:09.642546+0000","last_unstale":"2024-05-28T06:18:11.668379+0000","last_undegraded":"2024-05-28T06:18:11.668379+0000","last_fullsized":"2024-05-28T06:18:11.668379+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:19:45.186872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.668346+0000","last_change":"2024-05-28T06:18:10.685239+0000","last_active":"2024-05-28T06:18:11.668346+0000","last_peered":"2024-05-28T06:18:11.668346+0000","last_clean":"2024-05-28T06:18:11.668346+0000","last_became_active":"2024-05-28T06:18:10.681969+0000","last_became_peered":"2024-05-28T06:18:10.681969+0000","last_unstale":"2024-05-28T06:18:11.668346+0000","last_undegraded":"2024-05-28T06:18:11.668346+0000","last_fullsized":"2024-05-28T06:18:11.668346+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:38:26.397686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.669511+0000","last_change":"2024-05-28T06:18:09.643305+0000","last_active":"2024-05-28T06:18:11.669511+0000","last_peered":"2024-05-28T06:18:11.669511+0000","last_clean":"2024-05-28T06:18:11.669511+0000","last_became_active":"2024-05-28T06:18:09.642959+0000","last_became_peered":"2024-05-28T06:18:09.642959+0000","last_unstale":"2024-05-28T06:18:11.669511+0000","last_undegraded":"2024-05-28T06:18:11.669511+0000","last_fullsized":"2024-05-28T06:18:11.669511+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:57:25.848771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.417952+0000","last_change":"2024-05-28T06:18:10.682613+0000","last_active":"2024-05-28T06:18:12.417952+0000","last_peered":"2024-05-28T06:18:12.417952+0000","last_clean":"2024-05-28T06:18:12.417952+0000","last_became_active":"2024-05-28T06:18:10.682019+0000","last_became_peered":"2024-05-28T06:18:10.682019+0000","last_unstale":"2024-05-28T06:18:12.417952+0000","last_undegraded":"2024-05-28T06:18:12.417952+0000","last_fullsized":"2024-05-28T06:18:12.417952+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:17:36.803843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.416028+0000","last_change":"2024-05-28T06:18:09.637819+0000","last_active":"2024-05-28T06:18:12.416028+0000","last_peered":"2024-05-28T06:18:12.416028+0000","last_clean":"2024-05-28T06:18:12.416028+0000","last_became_active":"2024-05-28T06:18:09.637503+0000","last_became_peered":"2024-05-28T06:18:09.637503+0000","last_unstale":"2024-05-28T06:18:12.416028+0000","last_undegraded":"2024-05-28T06:18:12.416028+0000","last_fullsized":"2024-05-28T06:18:12.416028+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:41:26.564391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.415971+0000","last_change":"2024-05-28T06:18:10.668846+0000","last_active":"2024-05-28T06:18:12.415971+0000","last_peered":"2024-05-28T06:18:12.415971+0000","last_clean":"2024-05-28T06:18:12.415971+0000","last_became_active":"2024-05-28T06:18:10.665081+0000","last_became_peered":"2024-05-28T06:18:10.665081+0000","last_unstale":"2024-05-28T06:18:12.415971+0000","last_undegraded":"2024-05-28T06:18:12.415971+0000","last_fullsized":"2024-05-28T06:18:12.415971+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:35:48.065461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.669169+0000","last_change":"2024-05-28T06:18:09.641057+0000","last_active":"2024-05-28T06:18:11.669169+0000","last_peered":"2024-05-28T06:18:11.669169+0000","last_clean":"2024-05-28T06:18:11.669169+0000","last_became_active":"2024-05-28T06:18:09.640751+0000","last_became_peered":"2024-05-28T06:18:09.640751+0000","last_unstale":"2024-05-28T06:18:11.669169+0000","last_undegraded":"2024-05-28T06:18:11.669169+0000","last_fullsized":"2024-05-28T06:18:11.669169+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:14:09.079098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:11.675977+0000","last_change":"2024-05-28T06:18:10.673680+0000","last_active":"2024-05-28T06:18:11.675977+0000","last_peered":"2024-05-28T06:18:11.675977+0000","last_clean":"2024-05-28T06:18:11.675977+0000","last_became_active":"2024-05-28T06:18:10.673261+0000","last_became_peered":"2024-05-28T06:18:10.673261+0000","last_unstale":"2024-05-28T06:18:11.675977+0000","last_undegraded":"2024-05-28T06:18:11.675977+0000","last_fullsized":"2024-05-28T06:18:11.675977+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:09.604901+0000","last_clean_scrub_stamp":"2024-05-28T06:18:09.604901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:18:26.300329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"88'1","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.419222+0000","last_change":"2024-05-28T06:18:09.649924+0000","last_active":"2024-05-28T06:18:12.419222+0000","last_peered":"2024-05-28T06:18:12.419222+0000","last_clean":"2024-05-28T06:18:12.419222+0000","last_became_active":"2024-05-28T06:18:09.649696+0000","last_became_peered":"2024-05-28T06:18:09.649696+0000","last_unstale":"2024-05-28T06:18:12.419222+0000","last_undegraded":"2024-05-28T06:18:12.419222+0000","last_fullsized":"2024-05-28T06:18:12.419222+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:18:08.589864+0000","last_clean_scrub_stamp":"2024-05-28T06:18:08.589864+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:10:32.947652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":305,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-28T06:18:12.418348+0000","last_change":"2024-05-28T06:04:01.039129+0000","last_active":"2024-05-28T06:18:12.418348+0000","last_peered":"2024-05-28T06:18:12.418348+0000","last_clean":"2024-05-28T06:18:12.418348+0000","last_became_active":"2024-05-28T06:04:01.033906+0000","last_became_peered":"2024-05-28T06:04:01.033906+0000","last_unstale":"2024-05-28T06:18:12.418348+0000","last_undegraded":"2024-05-28T06:18:12.418348+0000","last_fullsized":"2024-05-28T06:18:12.418348+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:03:59.864237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:03:59.864237+0000","last_clean_scrub_stamp":"2024-05-28T06:03:59.864237+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:17:36.440245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575022,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":123008,"kb_used_data":1652,"kb_used_omap":1,"kb_used_meta":121342,"kb_avail":94248832,"statfs":{"total":96636764160,"available":96510803968,"internally_reserved":0,"allocated":1691648,"data_stored":1064040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":124254666},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575024,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":122440,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":121342,"kb_avail":94249400,"statfs":{"total":96636764160,"available":96511385600,"internally_reserved":0,"allocated":1105920,"data_stored":474296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":124254662},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575022,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76616,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94295224,"statfs":{"total":96636764160,"available":96558309376,"internally_reserved":0,"allocated":1110016,"data_stored":475236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575022,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64124,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94307716,"statfs":{"total":96636764160,"available":96571101184,"internally_reserved":0,"allocated":1093632,"data_stored":473602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575024,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":122440,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":121342,"kb_avail":94249400,"statfs":{"total":96636764160,"available":96511385600,"internally_reserved":0,"allocated":1105920,"data_stored":475214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":124254663},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575024,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":122432,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":121342,"kb_avail":94249408,"statfs":{"total":96636764160,"available":96511393792,"internally_reserved":0,"allocated":1097728,"data_stored":474184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124254661},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575024,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":104392,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":103294,"kb_avail":94267448,"statfs":{"total":96636764160,"available":96529866752,"internally_reserved":0,"allocated":1105920,"data_stored":473796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105773509},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575023,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77192,"kb_used_data":1652,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94294648,"statfs":{"total":96636764160,"available":96557719552,"internally_reserved":0,"allocated":1691648,"data_stored":1064004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-28T06:18:20.618 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-05-28T06:18:20.619 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-05-28T06:18:20.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.838+0000 7fbdccc00640 1 Processor -- start 2024-05-28T06:18:20.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.838+0000 7fbdccc00640 1 -- start start 2024-05-28T06:18:20.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.839+0000 7fbdccc00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fbdc80d0640 0x7fbdc815d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:20.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.839+0000 7fbdccc00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbdc80d0060 con 0x7fbdc80b32c0 2024-05-28T06:18:20.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.839+0000 7fbdccc00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbdc815cb10 con 0x7fbdc80d0a20 2024-05-28T06:18:20.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.839+0000 7fbdccc00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fbdc80adbb0 con 0x7fbdc80d0640 2024-05-28T06:18:20.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.840+0000 7fbdc6800640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fbdc80b32c0 0x7fbdc80b3730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:43816/0 (socket says 172.21.15.96:43816) 2024-05-28T06:18:20.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.840+0000 7fbdc6800640 1 -- 172.21.15.96:0/4027930409 learned_addr learned my addr 172.21.15.96:0/4027930409 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:18:20.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.840+0000 7fbdc5e00640 1 --2- 172.21.15.96:0/4027930409 >> v2:172.21.15.123:3300/0 conn(0x7fbdc80d0640 0x7fbdc815d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:20.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.840+0000 7fbdc4a00640 1 -- 172.21.15.96:0/4027930409 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4146966655 0 0) 0x7fbdc80d0060 con 0x7fbdc80b32c0 2024-05-28T06:18:20.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.841+0000 7fbdc4a00640 1 -- 172.21.15.96:0/4027930409 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbdac003520 con 0x7fbdc80b32c0 2024-05-28T06:18:20.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.841+0000 7fbdc4a00640 1 -- 172.21.15.96:0/4027930409 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3624699010 0 0) 0x7fbdac003520 con 0x7fbdc80b32c0 2024-05-28T06:18:20.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.841+0000 7fbdc4a00640 1 -- 172.21.15.96:0/4027930409 >> v2:172.21.15.123:3300/0 conn(0x7fbdc80d0640 msgr2=0x7fbdc815d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:18:20.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.841+0000 7fbdc4a00640 1 --2- 172.21.15.96:0/4027930409 >> v2:172.21.15.123:3300/0 conn(0x7fbdc80d0640 0x7fbdc815d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:20.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.841+0000 7fbdc4a00640 1 -- 172.21.15.96:0/4027930409 >> v1:172.21.15.123:6789/0 conn(0x7fbdc80d0a20 legacy=0x7fbdc80cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:20.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.841+0000 7fbdc4a00640 1 -- 172.21.15.96:0/4027930409 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdc80ad5f0 con 0x7fbdc80b32c0 2024-05-28T06:18:20.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.841+0000 7fbdc4a00640 1 -- 172.21.15.96:0/4027930409 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7fbdc00019f0 con 0x7fbdc80b32c0 2024-05-28T06:18:20.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.841+0000 7fbdc4a00640 1 -- 172.21.15.96:0/4027930409 >> v1:172.21.15.96:6789/0 conn(0x7fbdc80b32c0 legacy=0x7fbdc80b3730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:20.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.841+0000 7fbdc4a00640 1 --2- 172.21.15.96:0/4027930409 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbdac003a00 0x7fbdac003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:20.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.841+0000 7fbdc4a00640 1 --2- 172.21.15.96:0/4027930409 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbdac004800 0x7fbdac004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:20.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.841+0000 7fbdc4a00640 1 --2- 172.21.15.96:0/4027930409 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbdac005210 0x7fbdac00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:20.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.841+0000 7fbdc4a00640 1 -- 172.21.15.96:0/4027930409 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fbdac00c130 con 0x7fbdac005210 2024-05-28T06:18:20.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.841+0000 7fbdc4a00640 1 -- 172.21.15.96:0/4027930409 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fbdac00c2e0 con 0x7fbdac004800 2024-05-28T06:18:20.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.841+0000 7fbdc4a00640 1 -- 172.21.15.96:0/4027930409 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fbdac00c4e0 con 0x7fbdac003a00 2024-05-28T06:18:20.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.842+0000 7fbdc6800640 1 --2- 172.21.15.96:0/4027930409 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbdac005210 0x7fbdac00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:20.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.842+0000 7fbdc5e00640 1 --2- 172.21.15.96:0/4027930409 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbdac003a00 0x7fbdac003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:20.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.842+0000 7fbdc5400640 1 --2- 172.21.15.96:0/4027930409 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbdac004800 0x7fbdac004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:20.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.842+0000 7fbdc6800640 1 -- 172.21.15.96:0/4027930409 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbdac003a00 msgr2=0x7fbdac003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:18:20.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.842+0000 7fbdc6800640 1 --2- 172.21.15.96:0/4027930409 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbdac003a00 0x7fbdac003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:20.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.842+0000 7fbdc6800640 1 -- 172.21.15.96:0/4027930409 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbdac004800 msgr2=0x7fbdac004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:18:20.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.842+0000 7fbdc6800640 1 --2- 172.21.15.96:0/4027930409 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbdac004800 0x7fbdac004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:20.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.842+0000 7fbdc6800640 1 -- 172.21.15.96:0/4027930409 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdac003520 con 0x7fbdac005210 2024-05-28T06:18:20.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.843+0000 7fbdc6800640 1 -- 172.21.15.96:0/4027930409 reap_dead start 2024-05-28T06:18:20.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.843+0000 7fbdc6800640 1 --2- 172.21.15.96:0/4027930409 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbdac005210 0x7fbdac00b9c0 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7fbdc000a760 tx=0x7fbdac005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c672f25a2f5473 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:20.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.844+0000 7fbdc4a00640 1 -- 172.21.15.96:0/4027930409 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdc000a3b0 con 0x7fbdac005210 2024-05-28T06:18:20.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.844+0000 7fbdc4a00640 1 -- 172.21.15.96:0/4027930409 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbdc002f070 con 0x7fbdac005210 2024-05-28T06:18:20.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.844+0000 7fbdc4a00640 1 -- 172.21.15.96:0/4027930409 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdc002bbc0 con 0x7fbdac005210 2024-05-28T06:18:20.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.844+0000 7fbdccc00640 1 -- 172.21.15.96:0/4027930409 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbdac005210 msgr2=0x7fbdac00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:20.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.844+0000 7fbdccc00640 1 --2- 172.21.15.96:0/4027930409 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbdac005210 0x7fbdac00b9c0 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7fbdc000a760 tx=0x7fbdac005830 comp rx=0 tx=0).stop 2024-05-28T06:18:20.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.845+0000 7fbdccc00640 1 -- 172.21.15.96:0/4027930409 shutdown_connections 2024-05-28T06:18:20.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.845+0000 7fbdccc00640 1 --2- 172.21.15.96:0/4027930409 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbdac005210 0x7fbdac00b9c0 unknown :-1 s=CLOSED pgs=1427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:20.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.845+0000 7fbdccc00640 1 -- 172.21.15.96:0/4027930409 >> 172.21.15.96:0/4027930409 conn(0x7fbdc806bd20 msgr2=0x7fbdc806c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:18:20.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.845+0000 7fbdccc00640 1 -- 172.21.15.96:0/4027930409 shutdown_connections 2024-05-28T06:18:20.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.845+0000 7fbdccc00640 1 -- 172.21.15.96:0/4027930409 wait complete. 2024-05-28T06:18:20.847 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.847+0000 7fbdccc00640 1 Processor -- start 2024-05-28T06:18:20.847 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.847+0000 7fbdccc00640 1 -- start start 2024-05-28T06:18:20.847 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.848+0000 7fbdccc00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbdac005210 0x7fbdc81f16b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:20.847 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.848+0000 7fbdccc00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbdc814a590 0x7fbdc81484f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:20.847 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.848+0000 7fbdccc00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbdc8099af0 0x7fbdc8148a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:20.847 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.848+0000 7fbdccc00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fbdc815d0b0 con 0x7fbdac005210 2024-05-28T06:18:20.847 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.848+0000 7fbdccc00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fbdc815cdd0 con 0x7fbdc814a590 2024-05-28T06:18:20.847 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.848+0000 7fbdccc00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fbdc80ad2a0 con 0x7fbdc8099af0 2024-05-28T06:18:20.847 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.848+0000 7fbdc5e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbdac005210 0x7fbdc81f16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:20.847 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.848+0000 7fbdc5e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbdac005210 0x7fbdc81f16b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:40120/0 (socket says 172.21.15.96:40120) 2024-05-28T06:18:20.847 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.848+0000 7fbdc5e00640 1 -- 172.21.15.96:0/2587290456 learned_addr learned my addr 172.21.15.96:0/2587290456 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:18:20.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.848+0000 7fbdc5e00640 1 -- 172.21.15.96:0/2587290456 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbdc8099af0 msgr2=0x7fbdc8148a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:20.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.848+0000 7fbdc5e00640 1 --2- 172.21.15.96:0/2587290456 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbdc8099af0 0x7fbdc8148a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:20.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.848+0000 7fbdc5e00640 1 -- 172.21.15.96:0/2587290456 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbdc814a590 msgr2=0x7fbdc81484f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:20.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.848+0000 7fbdc5e00640 1 --2- 172.21.15.96:0/2587290456 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbdc814a590 0x7fbdc81484f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:20.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.848+0000 7fbdc5e00640 1 -- 172.21.15.96:0/2587290456 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdc81f1bf0 con 0x7fbdac005210 2024-05-28T06:18:20.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.849+0000 7fbdc5e00640 1 --2- 172.21.15.96:0/2587290456 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbdac005210 0x7fbdc81f16b0 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7fbdb8011540 tx=0x7fbdb8011a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:20.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.849+0000 7fbdbe000640 1 -- 172.21.15.96:0/2587290456 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdb8015930 con 0x7fbdac005210 2024-05-28T06:18:20.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.849+0000 7fbdbe000640 1 -- 172.21.15.96:0/2587290456 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbdb8015ad0 con 0x7fbdac005210 2024-05-28T06:18:20.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.849+0000 7fbdbe000640 1 -- 172.21.15.96:0/2587290456 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdb800f460 con 0x7fbdac005210 2024-05-28T06:18:20.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.849+0000 7fbdccc00640 1 -- 172.21.15.96:0/2587290456 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbdc814aa50 con 0x7fbdac005210 2024-05-28T06:18:20.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.849+0000 7fbdccc00640 1 -- 172.21.15.96:0/2587290456 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbdc814b010 con 0x7fbdac005210 2024-05-28T06:18:20.849 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.850+0000 7fbdccc00640 1 -- 172.21.15.96:0/2587290456 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbdc80689e0 con 0x7fbdac005210 2024-05-28T06:18:20.849 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.851+0000 7fbdbe000640 1 -- 172.21.15.96:0/2587290456 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 30) ==== 92621+0+0 (secure 0 0 0) 0x7fbdb800f600 con 0x7fbdac005210 2024-05-28T06:18:20.849 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.852+0000 7fbdbe000640 1 --2- 172.21.15.96:0/2587290456 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbda806da60 0x7fbda806ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:20.850 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.852+0000 7fbdbe000640 1 -- 172.21.15.96:0/2587290456 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 5737+0+0 (secure 0 0 0) 0x7fbdb808b370 con 0x7fbdac005210 2024-05-28T06:18:20.850 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.853+0000 7fbdc5400640 1 --2- 172.21.15.96:0/2587290456 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbda806da60 0x7fbda806ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:20.851 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.854+0000 7fbdc5400640 1 --2- 172.21.15.96:0/2587290456 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbda806da60 0x7fbda806ff20 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fbdb4007fd0 tx=0x7fbdb4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:20.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:20.856+0000 7fbdbe000640 1 -- 172.21.15.96:0/2587290456 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbdb8019030 con 0x7fbdac005210 2024-05-28T06:18:21.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:21.123+0000 7fbdccc00640 1 -- 172.21.15.96:0/2587290456 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7fbdc8156970 con 0x7fbdac005210 2024-05-28T06:18:21.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:21.130+0000 7fbdbe000640 1 -- 172.21.15.96:0/2587290456 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v19) ==== 190+0+147 (secure 0 0 0) 0x7fbdb805aea0 con 0x7fbdac005210 2024-05-28T06:18:21.127 INFO:teuthology.orchestra.run.smithi096.stdout:[client.0] 2024-05-28T06:18:21.128 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQCXc1Vm91hbGBAASTL3HQR8XOs7wVQEHbPl/g== 2024-05-28T06:18:21.128 INFO:teuthology.orchestra.run.smithi096.stdout: caps mds = "allow" 2024-05-28T06:18:21.128 INFO:teuthology.orchestra.run.smithi096.stdout: caps mon = "allow r" 2024-05-28T06:18:21.128 INFO:teuthology.orchestra.run.smithi096.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-05-28T06:18:21.128 INFO:teuthology.orchestra.run.smithi096.stderr:updated caps for client.0 2024-05-28T06:18:21.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:21.132+0000 7fbdccc00640 1 -- 172.21.15.96:0/2587290456 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbda806da60 msgr2=0x7fbda806ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:21.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:21.132+0000 7fbdccc00640 1 --2- 172.21.15.96:0/2587290456 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbda806da60 0x7fbda806ff20 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fbdb4007fd0 tx=0x7fbdb4009040 comp rx=0 tx=0).stop 2024-05-28T06:18:21.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:21.132+0000 7fbdccc00640 1 -- 172.21.15.96:0/2587290456 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbdac005210 msgr2=0x7fbdc81f16b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:21.131 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:21.132+0000 7fbdccc00640 1 --2- 172.21.15.96:0/2587290456 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbdac005210 0x7fbdc81f16b0 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7fbdb8011540 tx=0x7fbdb8011a00 comp rx=0 tx=0).stop 2024-05-28T06:18:21.131 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:21.133+0000 7fbdccc00640 1 -- 172.21.15.96:0/2587290456 shutdown_connections 2024-05-28T06:18:21.131 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:21.133+0000 7fbdccc00640 1 --2- 172.21.15.96:0/2587290456 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbda806da60 0x7fbda806ff20 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:21.131 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:21.133+0000 7fbdccc00640 1 --2- 172.21.15.96:0/2587290456 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbdc8099af0 0x7fbdc8148a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:21.131 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:21.133+0000 7fbdccc00640 1 --2- 172.21.15.96:0/2587290456 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbdc814a590 0x7fbdc81484f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:21.131 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:21.133+0000 7fbdccc00640 1 --2- 172.21.15.96:0/2587290456 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbdac005210 0x7fbdc81f16b0 unknown :-1 s=CLOSED pgs=1428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:21.131 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:21.133+0000 7fbdccc00640 1 -- 172.21.15.96:0/2587290456 >> 172.21.15.96:0/2587290456 conn(0x7fbdc806bd20 msgr2=0x7fbdc815dcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:18:21.131 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:21.133+0000 7fbdccc00640 1 -- 172.21.15.96:0/2587290456 shutdown_connections 2024-05-28T06:18:21.131 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:21.133+0000 7fbdccc00640 1 -- 172.21.15.96:0/2587290456 wait complete. 2024-05-28T06:18:21.148 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-28T06:18:21.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:21.332+0000 7f0d6d000640 1 Processor -- start 2024-05-28T06:18:21.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:21.332+0000 7f0d6d000640 1 -- start start 2024-05-28T06:18:21.331 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:21.334+0000 7f0d6d000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0d68161fb0 0x7f0d6809b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:21.331 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:21.334+0000 7f0d6d000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f0d6809b980 0x7f0d680b1a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:21.331 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:21.334+0000 7f0d6d000640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d680d0630 con 0x7f0d680b15e0 2024-05-28T06:18:21.331 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:21.334+0000 7f0d6d000640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f0d680d0400 con 0x7f0d6809b980 2024-05-28T06:18:21.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:21.334+0000 7f0d6d000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f0d68162390 con 0x7f0d68161fb0 2024-05-28T06:18:21.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:21.334+0000 7f0d5ea00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f0d6809b980 0x7f0d680b1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:21.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:21.334+0000 7f0d5ea00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f0d6809b980 0x7f0d680b1a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:40126/0 (socket says 172.21.15.96:40126) 2024-05-28T06:18:21.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:21.334+0000 7f0d5ea00640 1 -- 172.21.15.96:0/1144951756 learned_addr learned my addr 172.21.15.96:0/1144951756 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:18:21.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:21.335+0000 7f0d67400640 1 --2- 172.21.15.96:0/1144951756 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0d68161fb0 0x7f0d6809b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:21.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:21.335+0000 7f0d5ea00640 1 -- 172.21.15.96:0/1144951756 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0d68161fb0 msgr2=0x7f0d6809b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:18:21.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:21.335+0000 7f0d5ea00640 1 --2- 172.21.15.96:0/1144951756 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0d68161fb0 0x7f0d6809b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:21.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:21.335+0000 7f0d5ea00640 1 -- 172.21.15.96:0/1144951756 >> v1:172.21.15.96:6789/0 conn(0x7f0d680b15e0 legacy=0x7f0d680d13f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:21.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:21.335+0000 7f0d5ea00640 1 -- 172.21.15.96:0/1144951756 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d680d0910 con 0x7f0d6809b980 2024-05-28T06:18:21.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:21.335+0000 7f0d66a00640 1 -- 172.21.15.96:0/1144951756 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1602510969 0 0) 0x7f0d680d0630 con 0x7f0d680b15e0 2024-05-28T06:18:21.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:21.336+0000 7f0d5ea00640 1 --2- 172.21.15.96:0/1144951756 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0d6809b980 0x7f0d680b1a10 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f0d50009870 tx=0x7f0d5002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2520f3f5985dced9 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:21.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:21.336+0000 7f0d66a00640 1 -- 172.21.15.96:0/1144951756 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d50037070 con 0x7f0d6809b980 2024-05-28T06:18:21.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:21.336+0000 7f0d66a00640 1 -- 172.21.15.96:0/1144951756 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d50032440 con 0x7f0d6809b980 2024-05-28T06:18:21.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:21.336+0000 7f0d66a00640 1 -- 172.21.15.96:0/1144951756 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d50032810 con 0x7f0d6809b980 2024-05-28T06:18:21.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:21.337+0000 7f0d6d000640 1 -- 172.21.15.96:0/1144951756 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0d6809b980 msgr2=0x7f0d680b1a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:21.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:21.337+0000 7f0d6d000640 1 --2- 172.21.15.96:0/1144951756 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0d6809b980 0x7f0d680b1a10 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f0d50009870 tx=0x7f0d5002a270 comp rx=0 tx=0).stop 2024-05-28T06:18:21.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:21.337+0000 7f0d6d000640 1 -- 172.21.15.96:0/1144951756 shutdown_connections 2024-05-28T06:18:21.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:21.337+0000 7f0d6d000640 1 --2- 172.21.15.96:0/1144951756 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0d6809b980 0x7f0d680b1a10 unknown :-1 s=CLOSED pgs=1430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:21.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:21.337+0000 7f0d6d000640 1 --2- 172.21.15.96:0/1144951756 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0d68161fb0 0x7f0d6809b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:21.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:21.337+0000 7f0d6d000640 1 -- 172.21.15.96:0/1144951756 >> 172.21.15.96:0/1144951756 conn(0x7f0d680d23b0 msgr2=0x7f0d680d27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:18:21.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:21.337+0000 7f0d6d000640 1 -- 172.21.15.96:0/1144951756 shutdown_connections 2024-05-28T06:18:21.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:21.337+0000 7f0d6d000640 1 -- 172.21.15.96:0/1144951756 wait complete. 2024-05-28T06:18:21.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:21.338+0000 7f0d6d000640 1 Processor -- start 2024-05-28T06:18:21.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:21.339+0000 7f0d6d000640 1 -- start start 2024-05-28T06:18:21.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:21.339+0000 7f0d6d000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0d6809b980 0x7f0d681574c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:21.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:21.339+0000 7f0d6d000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d680b15e0 0x7f0d680cbc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:21.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:21.340+0000 7f0d6d000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0d68161fb0 0x7f0d6809aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:21.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:21.340+0000 7f0d6d000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f0d680d0f60 con 0x7f0d6809b980 2024-05-28T06:18:21.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:21.340+0000 7f0d6d000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f0d6809be20 con 0x7f0d680b15e0 2024-05-28T06:18:21.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:21.340+0000 7f0d6d000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f0d680d1b00 con 0x7f0d68161fb0 2024-05-28T06:18:21.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:21.340+0000 7f0d67400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0d6809b980 0x7f0d681574c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:21.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:21.340+0000 7f0d67400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0d6809b980 0x7f0d681574c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:40138/0 (socket says 172.21.15.96:40138) 2024-05-28T06:18:21.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:21.340+0000 7f0d67400640 1 -- 172.21.15.96:0/2102254263 learned_addr learned my addr 172.21.15.96:0/2102254263 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:18:21.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:21.340+0000 7f0d67400640 1 -- 172.21.15.96:0/2102254263 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0d68161fb0 msgr2=0x7f0d6809aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:21.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:21.340+0000 7f0d67400640 1 --2- 172.21.15.96:0/2102254263 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0d68161fb0 0x7f0d6809aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:21.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:21.340+0000 7f0d67400640 1 -- 172.21.15.96:0/2102254263 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d680b15e0 msgr2=0x7f0d680cbc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:21.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:21.340+0000 7f0d67400640 1 --2- 172.21.15.96:0/2102254263 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d680b15e0 0x7f0d680cbc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:21.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:21.340+0000 7f0d67400640 1 -- 172.21.15.96:0/2102254263 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d68160700 con 0x7f0d6809b980 2024-05-28T06:18:21.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:21.340+0000 7f0d67400640 1 --2- 172.21.15.96:0/2102254263 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0d6809b980 0x7f0d681574c0 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7f0d5800d620 tx=0x7f0d5800dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:21.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:21.341+0000 7f0d5fe00640 1 -- 172.21.15.96:0/2102254263 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d5800b620 con 0x7f0d6809b980 2024-05-28T06:18:21.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:21.341+0000 7f0d5fe00640 1 -- 172.21.15.96:0/2102254263 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d58019070 con 0x7f0d6809b980 2024-05-28T06:18:21.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:21.341+0000 7f0d5fe00640 1 -- 172.21.15.96:0/2102254263 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d5800ba00 con 0x7f0d6809b980 2024-05-28T06:18:21.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:21.341+0000 7f0d6d000640 1 -- 172.21.15.96:0/2102254263 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d681609a0 con 0x7f0d6809b980 2024-05-28T06:18:21.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:21.341+0000 7f0d6d000640 1 -- 172.21.15.96:0/2102254263 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d68160f60 con 0x7f0d6809b980 2024-05-28T06:18:21.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:21.342+0000 7f0d6d000640 1 -- 172.21.15.96:0/2102254263 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d680689e0 con 0x7f0d6809b980 2024-05-28T06:18:21.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:21.343+0000 7f0d5fe00640 1 -- 172.21.15.96:0/2102254263 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 30) ==== 92621+0+0 (secure 0 0 0) 0x7f0d5800bba0 con 0x7f0d6809b980 2024-05-28T06:18:21.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:21.344+0000 7f0d5fe00640 1 --2- 172.21.15.96:0/2102254263 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0d4406da60 0x7f0d4406ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:21.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:21.344+0000 7f0d5fe00640 1 -- 172.21.15.96:0/2102254263 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 5737+0+0 (secure 0 0 0) 0x7f0d5808ae10 con 0x7f0d6809b980 2024-05-28T06:18:21.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:21.345+0000 7f0d5ea00640 1 --2- 172.21.15.96:0/2102254263 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0d4406da60 0x7f0d4406ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:21.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:21.346+0000 7f0d5ea00640 1 --2- 172.21.15.96:0/2102254263 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0d4406da60 0x7f0d4406ff20 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f0d50034000 tx=0x7f0d5002c000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:21.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:21.348+0000 7f0d5fe00640 1 -- 172.21.15.96:0/2102254263 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0d5805a9a0 con 0x7f0d6809b980 2024-05-28T06:18:21.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:21.585+0000 7f0d6d000640 1 -- 172.21.15.96:0/2102254263 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0d680c5c40 con 0x7f0d6809b980 2024-05-28T06:18:21.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:21.587+0000 7f0d5fe00640 1 -- 172.21.15.96:0/2102254263 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) ==== 94+0+5189 (secure 0 0 0) 0x7f0d5805a2a0 con 0x7f0d6809b980 2024-05-28T06:18:21.584 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:18:21.585 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":52,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5645,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6834/1355344409","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":1355344409},{"type":"v1","addr":"172.21.15.96:6834","nonce":1355344409}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":47},{"gid":5651,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/1967688533","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":1967688533},{"type":"v1","addr":"172.21.15.123:6835","nonce":1967688533}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48},{"gid":6169,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/3362018278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":3362018278},{"type":"v1","addr":"172.21.15.123:6837","nonce":3362018278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-28T06:18:10.069593+0000","modified":"2024-05-28T06:18:13.212203+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8103},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8103":{"gid":8103,"name":"c","rank":0,"incarnation":49,"state":"up:active","state_seq":3,"addr":"172.21.15.96:6835/1614743177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6833","nonce":1614743177},{"type":"v1","addr":"172.21.15.96:6835","nonce":1614743177}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8103,"qdb_cluster":[8103]},"id":6}]} 2024-05-28T06:18:21.585 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 52 2024-05-28T06:18:21.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:21.589+0000 7f0d6d000640 1 -- 172.21.15.96:0/2102254263 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0d4406da60 msgr2=0x7f0d4406ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:21.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:21.589+0000 7f0d6d000640 1 --2- 172.21.15.96:0/2102254263 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0d4406da60 0x7f0d4406ff20 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f0d50034000 tx=0x7f0d5002c000 comp rx=0 tx=0).stop 2024-05-28T06:18:21.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:21.589+0000 7f0d6d000640 1 -- 172.21.15.96:0/2102254263 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0d6809b980 msgr2=0x7f0d681574c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:21.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:21.589+0000 7f0d6d000640 1 --2- 172.21.15.96:0/2102254263 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0d6809b980 0x7f0d681574c0 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7f0d5800d620 tx=0x7f0d5800dae0 comp rx=0 tx=0).stop 2024-05-28T06:18:21.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:21.589+0000 7f0d6d000640 1 -- 172.21.15.96:0/2102254263 shutdown_connections 2024-05-28T06:18:21.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:21.589+0000 7f0d6d000640 1 --2- 172.21.15.96:0/2102254263 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0d4406da60 0x7f0d4406ff20 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:21.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:21.589+0000 7f0d6d000640 1 --2- 172.21.15.96:0/2102254263 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0d68161fb0 0x7f0d6809aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:21.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:21.589+0000 7f0d6d000640 1 --2- 172.21.15.96:0/2102254263 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d680b15e0 0x7f0d680cbc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:21.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:21.589+0000 7f0d6d000640 1 --2- 172.21.15.96:0/2102254263 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0d6809b980 0x7f0d681574c0 unknown :-1 s=CLOSED pgs=1431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:21.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:21.589+0000 7f0d6d000640 1 -- 172.21.15.96:0/2102254263 >> 172.21.15.96:0/2102254263 conn(0x7f0d680d23b0 msgr2=0x7f0d68070040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:18:21.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:21.589+0000 7f0d6d000640 1 -- 172.21.15.96:0/2102254263 shutdown_connections 2024-05-28T06:18:21.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:21.590+0000 7f0d6d000640 1 -- 172.21.15.96:0/2102254263 wait complete. 2024-05-28T06:18:22.008 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 52, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-05-28T06:18:10.069593+0000', 'modified': '2024-05-28T06:18:13.212203+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8103}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8103': {'gid': 8103, 'name': 'c', 'rank': 0, 'incarnation': 49, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.96:6835/1614743177', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.96:6833', 'nonce': 1614743177}, {'type': 'v1', 'addr': '172.21.15.96:6835', 'nonce': 1614743177}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8103, 'qdb_cluster': [8103]} 2024-05-28T06:18:22.008 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-05-28T06:18:22.008 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-05-28T06:18:22.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.217+0000 7fa4b9200640 1 Processor -- start 2024-05-28T06:18:22.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.218+0000 7fa4b9200640 1 -- start start 2024-05-28T06:18:22.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.219+0000 7fa4b9200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa4b4156b30 0x7fa4b40af6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:22.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.219+0000 7fa4b9200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fa4b40ad940 0x7fa4b40b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:22.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.219+0000 7fa4b9200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4b4156f10 con 0x7fa4b406ee40 2024-05-28T06:18:22.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.219+0000 7fa4b9200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fa4b406dda0 con 0x7fa4b40ad940 2024-05-28T06:18:22.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.219+0000 7fa4b9200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fa4b406e080 con 0x7fa4b4156b30 2024-05-28T06:18:22.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.220+0000 7fa4b3400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa4b4156b30 0x7fa4b40af6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:22.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.220+0000 7fa4aaa00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7fa4b406ee40 0x7fa4b40ad230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.96:36276/0 (socket says 172.21.15.96:36276) 2024-05-28T06:18:22.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.220+0000 7fa4b3400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa4b4156b30 0x7fa4b40af6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.96:59736/0 (socket says 172.21.15.96:59736) 2024-05-28T06:18:22.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.220+0000 7fa4aaa00640 1 -- 172.21.15.96:0/4018848497 learned_addr learned my addr 172.21.15.96:0/4018848497 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:18:22.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.220+0000 7fa4b3e00640 1 --2- 172.21.15.96:0/4018848497 >> v2:172.21.15.123:3300/0 conn(0x7fa4b40ad940 0x7fa4b40b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:22.218 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.221+0000 7fa4b3400640 1 -- 172.21.15.96:0/4018848497 >> v2:172.21.15.123:3300/0 conn(0x7fa4b40ad940 msgr2=0x7fa4b40b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:18:22.218 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.221+0000 7fa4b3400640 1 --2- 172.21.15.96:0/4018848497 >> v2:172.21.15.123:3300/0 conn(0x7fa4b40ad940 0x7fa4b40b32c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:22.218 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.221+0000 7fa4b3400640 1 -- 172.21.15.96:0/4018848497 >> v1:172.21.15.123:6789/0 conn(0x7fa4b406ee40 legacy=0x7fa4b40ad230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:22.218 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.221+0000 7fa4b3400640 1 -- 172.21.15.96:0/4018848497 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4b406e360 con 0x7fa4b4156b30 2024-05-28T06:18:22.218 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.221+0000 7fa4b3e00640 1 --2- 172.21.15.96:0/4018848497 >> v2:172.21.15.123:3300/0 conn(0x7fa4b40ad940 0x7fa4b40b32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-28T06:18:22.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.222+0000 7fa4b3400640 1 --2- 172.21.15.96:0/4018848497 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa4b4156b30 0x7fa4b40af6e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fa4a4009d90 tx=0x7fa4a402a5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af86a07aa10bd425 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:22.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.222+0000 7fa4b2a00640 1 -- 172.21.15.96:0/4018848497 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4a4031600 con 0x7fa4b4156b30 2024-05-28T06:18:22.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.222+0000 7fa4b2a00640 1 -- 172.21.15.96:0/4018848497 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4a40317a0 con 0x7fa4b4156b30 2024-05-28T06:18:22.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.222+0000 7fa4b2a00640 1 -- 172.21.15.96:0/4018848497 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4a4031b70 con 0x7fa4b4156b30 2024-05-28T06:18:22.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.223+0000 7fa4b9200640 1 -- 172.21.15.96:0/4018848497 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa4b4156b30 msgr2=0x7fa4b40af6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:22.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.223+0000 7fa4b9200640 1 --2- 172.21.15.96:0/4018848497 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa4b4156b30 0x7fa4b40af6e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fa4a4009d90 tx=0x7fa4a402a5d0 comp rx=0 tx=0).stop 2024-05-28T06:18:22.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.223+0000 7fa4b9200640 1 -- 172.21.15.96:0/4018848497 shutdown_connections 2024-05-28T06:18:22.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.223+0000 7fa4b9200640 1 --2- 172.21.15.96:0/4018848497 >> v2:172.21.15.123:3300/0 conn(0x7fa4b40ad940 0x7fa4b40b32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:22.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.223+0000 7fa4b9200640 1 --2- 172.21.15.96:0/4018848497 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa4b4156b30 0x7fa4b40af6e0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:22.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.223+0000 7fa4b9200640 1 -- 172.21.15.96:0/4018848497 >> 172.21.15.96:0/4018848497 conn(0x7fa4b40b1fc0 msgr2=0x7fa4b40b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:18:22.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.223+0000 7fa4b9200640 1 -- 172.21.15.96:0/4018848497 shutdown_connections 2024-05-28T06:18:22.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.223+0000 7fa4b9200640 1 -- 172.21.15.96:0/4018848497 wait complete. 2024-05-28T06:18:22.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.225+0000 7fa4b9200640 1 Processor -- start 2024-05-28T06:18:22.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.225+0000 7fa4b9200640 1 -- start start 2024-05-28T06:18:22.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.226+0000 7fa4b9200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa4b406ee40 0x7fa4b41543b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:22.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.226+0000 7fa4b9200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa4b40ad940 0x7fa4b41595f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:22.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.226+0000 7fa4b9200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa4b4156b30 0x7fa4b41f40f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:22.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.226+0000 7fa4b9200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa4b40afc20 con 0x7fa4b406ee40 2024-05-28T06:18:22.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.226+0000 7fa4b9200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa4b40b2c30 con 0x7fa4b40ad940 2024-05-28T06:18:22.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.226+0000 7fa4b9200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fa4b4154ab0 con 0x7fa4b4156b30 2024-05-28T06:18:22.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.226+0000 7fa4b3400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa4b406ee40 0x7fa4b41543b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:22.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.226+0000 7fa4b3400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa4b406ee40 0x7fa4b41543b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:40152/0 (socket says 172.21.15.96:40152) 2024-05-28T06:18:22.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.226+0000 7fa4b3400640 1 -- 172.21.15.96:0/2350700441 learned_addr learned my addr 172.21.15.96:0/2350700441 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:18:22.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.226+0000 7fa4b3400640 1 -- 172.21.15.96:0/2350700441 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa4b4156b30 msgr2=0x7fa4b41f40f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:22.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.226+0000 7fa4b3400640 1 --2- 172.21.15.96:0/2350700441 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa4b4156b30 0x7fa4b41f40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:22.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.226+0000 7fa4b3400640 1 -- 172.21.15.96:0/2350700441 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa4b40ad940 msgr2=0x7fa4b41595f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:22.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.226+0000 7fa4b3400640 1 --2- 172.21.15.96:0/2350700441 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa4b40ad940 0x7fa4b41595f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:22.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.226+0000 7fa4b3400640 1 -- 172.21.15.96:0/2350700441 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4b4154d90 con 0x7fa4b406ee40 2024-05-28T06:18:22.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.227+0000 7fa4b3400640 1 --2- 172.21.15.96:0/2350700441 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa4b406ee40 0x7fa4b41543b0 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7fa4a4009d90 tx=0x7fa4a4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:22.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.227+0000 7fa4abe00640 1 -- 172.21.15.96:0/2350700441 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4a4040070 con 0x7fa4b406ee40 2024-05-28T06:18:22.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.227+0000 7fa4abe00640 1 -- 172.21.15.96:0/2350700441 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4a4030be0 con 0x7fa4b406ee40 2024-05-28T06:18:22.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.227+0000 7fa4b9200640 1 -- 172.21.15.96:0/2350700441 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4b4155070 con 0x7fa4b406ee40 2024-05-28T06:18:22.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.227+0000 7fa4abe00640 1 -- 172.21.15.96:0/2350700441 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4a4009450 con 0x7fa4b406ee40 2024-05-28T06:18:22.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.227+0000 7fa4b9200640 1 -- 172.21.15.96:0/2350700441 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4b415d230 con 0x7fa4b406ee40 2024-05-28T06:18:22.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.229+0000 7fa4abe00640 1 -- 172.21.15.96:0/2350700441 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 30) ==== 92621+0+0 (secure 0 0 0) 0x7fa4a4039040 con 0x7fa4b406ee40 2024-05-28T06:18:22.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.230+0000 7fa4abe00640 1 --2- 172.21.15.96:0/2350700441 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa49006da60 0x7fa49006ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:22.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.230+0000 7fa4abe00640 1 -- 172.21.15.96:0/2350700441 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 5737+0+0 (secure 0 0 0) 0x7fa4a40ac2b0 con 0x7fa4b406ee40 2024-05-28T06:18:22.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.231+0000 7fa4bd5f9740 1 Processor -- start 2024-05-28T06:18:22.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.231+0000 7fa4aaa00640 1 --2- 172.21.15.96:0/2350700441 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa49006da60 0x7fa49006ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:22.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.231+0000 7fa4bd5f9740 1 -- start start 2024-05-28T06:18:22.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.232+0000 7fa4bd5f9740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c6f1860490 0x55c6f1863c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:22.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.232+0000 7fa4bd5f9740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c6f1864150 0x55c6f1880090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:22.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.232+0000 7fa4bd5f9740 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x55c6f18805d0 0x55c6f1880a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:22.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.232+0000 7fa4bd5f9740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55c6f179db10 con 0x55c6f1864150 2024-05-28T06:18:22.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.232+0000 7fa4bd5f9740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55c6f17442b0 con 0x55c6f1860490 2024-05-28T06:18:22.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.232+0000 7fa4bd5f9740 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x55c6f1799ad0 con 0x55c6f18805d0 2024-05-28T06:18:22.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.232+0000 7fa4aaa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c6f1864150 0x55c6f1880090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:22.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.232+0000 7fa4aaa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c6f1864150 0x55c6f1880090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:40162/0 (socket says 172.21.15.96:40162) 2024-05-28T06:18:22.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.232+0000 7fa4aaa00640 1 -- 172.21.15.96:0/1719851034 learned_addr learned my addr 172.21.15.96:0/1719851034 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:18:22.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.232+0000 7fa4aaa00640 1 --2- 172.21.15.96:0/2350700441 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa49006da60 0x7fa49006ff20 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fa4b4144fb0 tx=0x7fa49c007a40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:22.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.232+0000 7fa4b3400640 1 --2- 172.21.15.96:0/1719851034 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c6f1860490 0x55c6f1863c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:22.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.232+0000 7fa4aaa00640 1 -- 172.21.15.96:0/1719851034 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x55c6f18805d0 msgr2=0x55c6f1880a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:18:22.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.232+0000 7fa4aaa00640 1 --2- 172.21.15.96:0/1719851034 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x55c6f18805d0 0x55c6f1880a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:22.231 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.232+0000 7fa4aaa00640 1 -- 172.21.15.96:0/1719851034 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c6f1860490 msgr2=0x55c6f1863c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:18:22.231 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.232+0000 7fa4aaa00640 1 --2- 172.21.15.96:0/1719851034 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c6f1860490 0x55c6f1863c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:22.231 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.232+0000 7fa4aaa00640 1 -- 172.21.15.96:0/1719851034 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c6f14b02d0 con 0x55c6f1864150 2024-05-28T06:18:22.231 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.232+0000 7fa4b3400640 1 --2- 172.21.15.96:0/1719851034 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c6f1860490 0x55c6f1863c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:18:22.231 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.232+0000 7fa4aaa00640 1 --2- 172.21.15.96:0/1719851034 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c6f1864150 0x55c6f1880090 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7fa4b40ad730 tx=0x7fa49c00f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b286e30021296250 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:22.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.233+0000 7fa4a3e00640 1 -- 172.21.15.96:0/1719851034 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa49c023070 con 0x55c6f1864150 2024-05-28T06:18:22.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.233+0000 7fa4a3e00640 1 -- 172.21.15.96:0/1719851034 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa49c016030 con 0x55c6f1864150 2024-05-28T06:18:22.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.233+0000 7fa4a3e00640 1 -- 172.21.15.96:0/1719851034 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa49c018050 con 0x55c6f1864150 2024-05-28T06:18:22.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.233+0000 7fa4bd5f9740 1 -- 172.21.15.96:0/1719851034 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c6f1864150 msgr2=0x55c6f1880090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:22.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.233+0000 7fa4bd5f9740 1 --2- 172.21.15.96:0/1719851034 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c6f1864150 0x55c6f1880090 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7fa4b40ad730 tx=0x7fa49c00f000 comp rx=0 tx=0).stop 2024-05-28T06:18:22.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.234+0000 7fa4bd5f9740 1 -- 172.21.15.96:0/1719851034 shutdown_connections 2024-05-28T06:18:22.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.234+0000 7fa4bd5f9740 1 --2- 172.21.15.96:0/1719851034 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x55c6f18805d0 0x55c6f1880a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:22.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.234+0000 7fa4bd5f9740 1 --2- 172.21.15.96:0/1719851034 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c6f1864150 0x55c6f1880090 unknown :-1 s=CLOSED pgs=1433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:22.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.234+0000 7fa4bd5f9740 1 --2- 172.21.15.96:0/1719851034 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c6f1860490 0x55c6f1863c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:22.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.234+0000 7fa4bd5f9740 1 -- 172.21.15.96:0/1719851034 >> 172.21.15.96:0/1719851034 conn(0x55c6f1836470 msgr2=0x55c6f1837470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:18:22.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.234+0000 7fa4bd5f9740 1 -- 172.21.15.96:0/1719851034 shutdown_connections 2024-05-28T06:18:22.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.234+0000 7fa4bd5f9740 1 -- 172.21.15.96:0/1719851034 wait complete. 2024-05-28T06:18:22.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.234+0000 7fa4bd5f9740 1 -- start start 2024-05-28T06:18:22.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.234+0000 7fa4bd5f9740 1 Processor -- start 2024-05-28T06:18:22.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.235+0000 7fa4bd5f9740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c6f1864150 0x55c6f1879240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:22.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.235+0000 7fa4bd5f9740 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x55c6f18805d0 0x55c6f1866810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:22.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.235+0000 7fa4bd5f9740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c6f185c500 0x55c6f1866d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:22.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.235+0000 7fa4bd5f9740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55c6f18342a0 con 0x55c6f1864150 2024-05-28T06:18:22.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.235+0000 7fa4bd5f9740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55c6f1757570 con 0x55c6f185c500 2024-05-28T06:18:22.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.235+0000 7fa4bd5f9740 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x55c6f1834580 con 0x55c6f18805d0 2024-05-28T06:18:22.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.235+0000 7fa4bd5f9740 10 client.0 _ll_register_callbacks cb 0x55c6f1725fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-28T06:18:22.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.236+0000 7fa4b3400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c6f1864150 0x55c6f1879240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:22.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.236+0000 7fa4b3400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c6f1864150 0x55c6f1879240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:40178/0 (socket says 172.21.15.96:40178) 2024-05-28T06:18:22.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.236+0000 7fa4b3400640 1 -- 172.21.15.96:0/872340253 learned_addr learned my addr 172.21.15.96:0/872340253 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:18:22.238 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.236+0000 7fa4aaa00640 1 --2- 172.21.15.96:0/872340253 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x55c6f18805d0 0x55c6f1866810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:22.238 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.236+0000 7fa4b3400640 1 -- 172.21.15.96:0/872340253 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x55c6f18805d0 msgr2=0x55c6f1866810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:18:22.238 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.236+0000 7fa4b3e00640 1 --2- 172.21.15.96:0/872340253 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c6f185c500 0x55c6f1866d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:22.238 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.236+0000 7fa4b3400640 1 --2- 172.21.15.96:0/872340253 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x55c6f18805d0 0x55c6f1866810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:22.238 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.236+0000 7fa4b3400640 1 -- 172.21.15.96:0/872340253 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c6f185c500 msgr2=0x55c6f1866d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:18:22.238 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.236+0000 7fa4b3400640 1 --2- 172.21.15.96:0/872340253 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c6f185c500 0x55c6f1866d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:22.238 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.236+0000 7fa4b3400640 1 -- 172.21.15.96:0/872340253 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c6f157a6b0 con 0x55c6f1864150 2024-05-28T06:18:22.238 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.236+0000 7fa4b3400640 1 --2- 172.21.15.96:0/872340253 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c6f1864150 0x55c6f1879240 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7fa4a4083eb0 tx=0x7fa4a407bfc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f602a8cdf92a0407 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:22.238 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.236+0000 7fa4a0c00640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-05-28T06:18:22.238 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.237+0000 7fa4a0c00640 1 -- 172.21.15.96:0/872340253 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4a40843e0 con 0x55c6f1864150 2024-05-28T06:18:22.238 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.237+0000 7fa4a0c00640 1 -- 172.21.15.96:0/872340253 <== mon.0 v2:172.21.15.96:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 5737+0+0 (secure 0 0 0) 0x7fa4a40820b0 con 0x55c6f1864150 2024-05-28T06:18:22.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.237+0000 7fa4bd5f9740 1 -- 172.21.15.96:0/872340253 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa49c016030 con 0x55c6f1864150 2024-05-28T06:18:22.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.237+0000 7fa4a0c00640 1 -- 172.21.15.96:0/872340253 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=52) ==== 24+0+0 (secure 0 0 0) 0x7fa4a4079b00 con 0x55c6f1864150 2024-05-28T06:18:22.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.237+0000 7fa4bd5f9740 10 client.8280 fetch_fsmap learned FSMap version 52 2024-05-28T06:18:22.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.237+0000 7fa4bd5f9740 1 -- 172.21.15.96:0/872340253 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=52,monmap=2+}) -- 0x55c6f1864db0 con 0x55c6f1864150 2024-05-28T06:18:22.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.237+0000 7fa4a0c00640 1 -- 172.21.15.96:0/872340253 <== mon.0 v2:172.21.15.96:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa49c016030 con 0x55c6f1864150 2024-05-28T06:18:22.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.237+0000 7fa4a0c00640 1 -- 172.21.15.96:0/872340253 <== mon.0 v2:172.21.15.96:3300/0 5 ==== fsmap(e 52) ==== 3494+0+0 (secure 0 0 0) 0x7fa4a404d030 con 0x55c6f1864150 2024-05-28T06:18:22.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.237+0000 7fa4bd5f9740 10 client.8280 fetch_fsmap finished waiting for FSMap version 52 2024-05-28T06:18:22.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.237+0000 7fa4bd5f9740 10 client.8280 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-28T06:18:22.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.237+0000 7fa4bd5f9740 20 client.8280 populate_metadata read hostname 'smithi096' 2024-05-28T06:18:22.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.237+0000 7fa4bd5f9740 1 --2- 172.21.15.96:0/872340253 >> [v2:172.21.15.96:6833/1614743177,v1:172.21.15.96:6835/1614743177] conn(0x55c6f188f040 0x55c6f189d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:22.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.237+0000 7fa4bd5f9740 4 client.8280 mds_command: new command op to 8103 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-28T06:18:22.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.237+0000 7fa4bd5f9740 1 -- 172.21.15.96:0/872340253 --> [v2:172.21.15.96:6833/1614743177,v1:172.21.15.96:6835/1614743177] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c6f17ab530 con 0x55c6f188f040 2024-05-28T06:18:22.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.238+0000 7fa4aaa00640 1 --2- 172.21.15.96:0/872340253 >> [v2:172.21.15.96:6833/1614743177,v1:172.21.15.96:6835/1614743177] conn(0x55c6f188f040 0x55c6f189d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:22.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.238+0000 7fa4a0c00640 10 client.8280 ms_handle_connect on v2:172.21.15.96:6833/1614743177 2024-05-28T06:18:22.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.238+0000 7fa4aaa00640 1 --2- 172.21.15.96:0/872340253 >> [v2:172.21.15.96:6833/1614743177,v1:172.21.15.96:6835/1614743177] conn(0x55c6f188f040 0x55c6f189d2a0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=77a82434a12317a8 server_cookie=6e78178162bfc97e in_seq=0 out_seq=0 2024-05-28T06:18:22.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.239+0000 7fa4a0c00640 1 -- 172.21.15.96:0/872340253 <== mds.0 v2:172.21.15.96:6833/1614743177 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c6f17ab530 con 0x55c6f188f040 2024-05-28T06:18:22.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.239+0000 7fa4a0c00640 10 client.8280 handle_command_reply: tid=0 2024-05-28T06:18:22.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.239+0000 7fa4bd5f9740 1 client.8280 shutdown 2024-05-28T06:18:22.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.239+0000 7fa4bd5f9740 1 -- 172.21.15.96:0/872340253 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c6f1864150 msgr2=0x55c6f1879240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:22.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.239+0000 7fa4bd5f9740 1 --2- 172.21.15.96:0/872340253 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c6f1864150 0x55c6f1879240 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7fa4a4083eb0 tx=0x7fa4a407bfc0 comp rx=0 tx=0).stop 2024-05-28T06:18:22.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.239+0000 7fa4bd5f9740 1 -- 172.21.15.96:0/872340253 shutdown_connections 2024-05-28T06:18:22.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.239+0000 7fa4bd5f9740 1 --2- 172.21.15.96:0/872340253 >> [v2:172.21.15.96:6833/1614743177,v1:172.21.15.96:6835/1614743177] conn(0x55c6f188f040 0x55c6f189d2a0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:22.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.239+0000 7fa4bd5f9740 1 --2- 172.21.15.96:0/872340253 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c6f185c500 0x55c6f1866d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:22.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.239+0000 7fa4bd5f9740 1 --2- 172.21.15.96:0/872340253 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x55c6f18805d0 0x55c6f1866810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:22.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.239+0000 7fa4bd5f9740 1 --2- 172.21.15.96:0/872340253 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c6f1864150 0x55c6f1879240 unknown :-1 s=CLOSED pgs=1434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:22.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.239+0000 7fa4bd5f9740 1 -- 172.21.15.96:0/872340253 >> 172.21.15.96:0/872340253 conn(0x55c6f1860490 msgr2=0x55c6f1839e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:18:22.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.239+0000 7fa4a0c00640 0 client.8280 ms_handle_reset on v2:172.21.15.96:6833/1614743177 2024-05-28T06:18:22.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.240+0000 7fa4bd5f9740 1 -- 172.21.15.96:0/872340253 shutdown_connections 2024-05-28T06:18:22.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.240+0000 7fa4bd5f9740 1 -- 172.21.15.96:0/872340253 wait complete. 2024-05-28T06:18:22.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.240+0000 7fa4bd5f9740 20 client.8280 trim_cache size 0 max 16384 2024-05-28T06:18:22.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.259+0000 7fa4bd5f9740 1 Processor -- start 2024-05-28T06:18:22.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.259+0000 7fa4bd5f9740 1 -- start start 2024-05-28T06:18:22.257 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.260+0000 7fa4bd5f9740 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x55c6f185c500 0x55c6f1866bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:22.257 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.260+0000 7fa4bd5f9740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c6f1864150 0x55c6f1879210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:22.257 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.260+0000 7fa4bd5f9740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c6f18805d0 0x55c6f1866640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:22.257 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.260+0000 7fa4bd5f9740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa4b4200d40 con 0x55c6f1864150 2024-05-28T06:18:22.257 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.260+0000 7fa4bd5f9740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa4b415ea60 con 0x55c6f18805d0 2024-05-28T06:18:22.257 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.260+0000 7fa4bd5f9740 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fa4b4158e60 con 0x55c6f185c500 2024-05-28T06:18:22.259 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.260+0000 7fa4aaa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c6f1864150 0x55c6f1879210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:22.259 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.260+0000 7fa4aaa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c6f1864150 0x55c6f1879210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:40190/0 (socket says 172.21.15.96:40190) 2024-05-28T06:18:22.260 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.260+0000 7fa4aaa00640 1 -- 172.21.15.96:0/758678683 learned_addr learned my addr 172.21.15.96:0/758678683 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:18:22.260 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.261+0000 7fa4aaa00640 1 -- 172.21.15.96:0/758678683 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x55c6f185c500 msgr2=0x55c6f1866bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:18:22.260 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.261+0000 7fa4aaa00640 1 --2- 172.21.15.96:0/758678683 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x55c6f185c500 0x55c6f1866bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:22.260 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.261+0000 7fa4aaa00640 1 -- 172.21.15.96:0/758678683 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c6f18805d0 msgr2=0x55c6f1866640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:18:22.260 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.261+0000 7fa4aaa00640 1 --2- 172.21.15.96:0/758678683 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c6f18805d0 0x55c6f1866640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:22.260 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.261+0000 7fa4aaa00640 1 -- 172.21.15.96:0/758678683 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c6f182ecf0 con 0x55c6f1864150 2024-05-28T06:18:22.260 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.261+0000 7fa4aaa00640 1 --2- 172.21.15.96:0/758678683 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c6f1864150 0x55c6f1879210 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7fa49c021040 tx=0x7fa49c013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=923765a61d09b582 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:22.260 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.261+0000 7fa4a3400640 1 -- 172.21.15.96:0/758678683 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa49c032070 con 0x55c6f1864150 2024-05-28T06:18:22.260 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.261+0000 7fa4a3400640 1 -- 172.21.15.96:0/758678683 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa49c016040 con 0x55c6f1864150 2024-05-28T06:18:22.261 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.261+0000 7fa4a3400640 1 -- 172.21.15.96:0/758678683 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa49c018040 con 0x55c6f1864150 2024-05-28T06:18:22.261 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.262+0000 7fa4bd5f9740 1 -- 172.21.15.96:0/758678683 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c6f1864150 msgr2=0x55c6f1879210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:22.261 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.262+0000 7fa4bd5f9740 1 --2- 172.21.15.96:0/758678683 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c6f1864150 0x55c6f1879210 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7fa49c021040 tx=0x7fa49c013000 comp rx=0 tx=0).stop 2024-05-28T06:18:22.261 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.262+0000 7fa4bd5f9740 1 -- 172.21.15.96:0/758678683 shutdown_connections 2024-05-28T06:18:22.261 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.262+0000 7fa4bd5f9740 1 --2- 172.21.15.96:0/758678683 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c6f18805d0 0x55c6f1866640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:22.261 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.262+0000 7fa4bd5f9740 1 --2- 172.21.15.96:0/758678683 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c6f1864150 0x55c6f1879210 unknown :-1 s=CLOSED pgs=1435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:22.261 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.262+0000 7fa4bd5f9740 1 --2- 172.21.15.96:0/758678683 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x55c6f185c500 0x55c6f1866bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:22.261 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.262+0000 7fa4bd5f9740 1 -- 172.21.15.96:0/758678683 >> 172.21.15.96:0/758678683 conn(0x55c6f1860490 msgr2=0x55c6f182f8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:18:22.262 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.262+0000 7fa4bd5f9740 1 -- 172.21.15.96:0/758678683 shutdown_connections 2024-05-28T06:18:22.262 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.262+0000 7fa4bd5f9740 1 -- 172.21.15.96:0/758678683 wait complete. 2024-05-28T06:18:22.267 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.263+0000 7fa4bd5f9740 1 -- start start 2024-05-28T06:18:22.267 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.263+0000 7fa4bd5f9740 1 Processor -- start 2024-05-28T06:18:22.267 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.264+0000 7fa4bd5f9740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c6f185c500 0x55c6f18824a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:22.267 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.264+0000 7fa4bd5f9740 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x55c6f1864150 0x55c6f18f7c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:22.268 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.264+0000 7fa4bd5f9740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c6f18805d0 0x55c6f18751c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:22.268 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.264+0000 7fa4bd5f9740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55c6f18342a0 con 0x55c6f18805d0 2024-05-28T06:18:22.268 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.264+0000 7fa4bd5f9740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa4b415e410 con 0x55c6f185c500 2024-05-28T06:18:22.268 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.264+0000 7fa4bd5f9740 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x55c6f1867590 con 0x55c6f1864150 2024-05-28T06:18:22.268 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.264+0000 7fa4bd5f9740 10 client.0 _ll_register_callbacks cb 0x7fa4a4009b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-28T06:18:22.268 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.264+0000 7fa4b3e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c6f18805d0 0x55c6f18751c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:22.268 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.264+0000 7fa4b3e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c6f18805d0 0x55c6f18751c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:40204/0 (socket says 172.21.15.96:40204) 2024-05-28T06:18:22.268 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.264+0000 7fa4b3e00640 1 -- 172.21.15.96:0/2002940542 learned_addr learned my addr 172.21.15.96:0/2002940542 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:18:22.269 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.265+0000 7fa4aaa00640 1 --2- 172.21.15.96:0/2002940542 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x55c6f1864150 0x55c6f18f7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:22.269 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.265+0000 7fa4b3e00640 1 -- 172.21.15.96:0/2002940542 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x55c6f1864150 msgr2=0x55c6f18f7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:18:22.269 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.265+0000 7fa4b3e00640 1 --2- 172.21.15.96:0/2002940542 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x55c6f1864150 0x55c6f18f7c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:22.269 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.265+0000 7fa4b3400640 1 --2- 172.21.15.96:0/2002940542 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c6f185c500 0x55c6f18824a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:22.269 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.265+0000 7fa4b3e00640 1 -- 172.21.15.96:0/2002940542 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c6f185c500 msgr2=0x55c6f18824a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:18:22.269 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.265+0000 7fa4b3e00640 1 --2- 172.21.15.96:0/2002940542 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c6f185c500 0x55c6f18824a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:22.269 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.265+0000 7fa4b3e00640 1 -- 172.21.15.96:0/2002940542 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c6f18803f0 con 0x55c6f18805d0 2024-05-28T06:18:22.269 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.265+0000 7fa4b3e00640 1 --2- 172.21.15.96:0/2002940542 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c6f18805d0 0x55c6f18751c0 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7fa4ac010420 tx=0x7fa4ac0108e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=682579c0c8246637 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:22.270 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.265+0000 7fa4a0c00640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-05-28T06:18:22.270 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.265+0000 7fa4a0c00640 1 -- 172.21.15.96:0/2002940542 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4ac018020 con 0x55c6f18805d0 2024-05-28T06:18:22.270 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.265+0000 7fa4a0c00640 1 -- 172.21.15.96:0/2002940542 <== mon.0 v2:172.21.15.96:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 5737+0+0 (secure 0 0 0) 0x7fa4ac004440 con 0x55c6f18805d0 2024-05-28T06:18:22.270 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.265+0000 7fa4bd5f9740 1 -- 172.21.15.96:0/2002940542 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa49c016040 con 0x55c6f18805d0 2024-05-28T06:18:22.270 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.265+0000 7fa4a0c00640 1 -- 172.21.15.96:0/2002940542 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=52) ==== 24+0+0 (secure 0 0 0) 0x7fa4ac022050 con 0x55c6f18805d0 2024-05-28T06:18:22.270 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.265+0000 7fa4bd5f9740 10 client.8286 fetch_fsmap learned FSMap version 52 2024-05-28T06:18:22.270 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.265+0000 7fa4bd5f9740 1 -- 172.21.15.96:0/2002940542 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=52,monmap=2+}) -- 0x55c6f1882cd0 con 0x55c6f18805d0 2024-05-28T06:18:22.270 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.266+0000 7fa4a0c00640 1 -- 172.21.15.96:0/2002940542 <== mon.0 v2:172.21.15.96:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa49c016040 con 0x55c6f18805d0 2024-05-28T06:18:22.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.266+0000 7fa4a0c00640 1 -- 172.21.15.96:0/2002940542 <== mon.0 v2:172.21.15.96:3300/0 5 ==== fsmap(e 52) ==== 3494+0+0 (secure 0 0 0) 0x7fa4ac025020 con 0x55c6f18805d0 2024-05-28T06:18:22.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.266+0000 7fa4bd5f9740 10 client.8286 fetch_fsmap finished waiting for FSMap version 52 2024-05-28T06:18:22.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.266+0000 7fa4bd5f9740 10 client.8286 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-28T06:18:22.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.266+0000 7fa4bd5f9740 20 client.8286 populate_metadata read hostname 'smithi096' 2024-05-28T06:18:22.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.266+0000 7fa4bd5f9740 1 --2- 172.21.15.96:0/2002940542 >> [v2:172.21.15.96:6833/1614743177,v1:172.21.15.96:6835/1614743177] conn(0x55c6f1862880 0x55c6f1862c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:22.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.266+0000 7fa4bd5f9740 4 client.8286 mds_command: new command op to 8103 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-28T06:18:22.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.266+0000 7fa4bd5f9740 1 -- 172.21.15.96:0/2002940542 --> [v2:172.21.15.96:6833/1614743177,v1:172.21.15.96:6835/1614743177] -- command(tid 0: {"prefix": "status"}) -- 0x55c6f17a53a0 con 0x55c6f1862880 2024-05-28T06:18:22.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.266+0000 7fa4b3400640 1 --2- 172.21.15.96:0/2002940542 >> [v2:172.21.15.96:6833/1614743177,v1:172.21.15.96:6835/1614743177] conn(0x55c6f1862880 0x55c6f1862c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:22.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.267+0000 7fa4a0c00640 10 client.8286 ms_handle_connect on v2:172.21.15.96:6833/1614743177 2024-05-28T06:18:22.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.267+0000 7fa4b3400640 1 --2- 172.21.15.96:0/2002940542 >> [v2:172.21.15.96:6833/1614743177,v1:172.21.15.96:6835/1614743177] conn(0x55c6f1862880 0x55c6f1862c60 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cac2b7e34065b116 server_cookie=f66d355e9de497b9 in_seq=0 out_seq=0 2024-05-28T06:18:22.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.267+0000 7fa4a0c00640 1 -- 172.21.15.96:0/2002940542 <== mds.0 v2:172.21.15.96:6833/1614743177 1 ==== command_reply(tid 0: 0 ) ==== 8+0+324 (crc 0 0 0) 0x55c6f17a53a0 con 0x55c6f1862880 2024-05-28T06:18:22.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.267+0000 7fa4a0c00640 10 client.8286 handle_command_reply: tid=0 2024-05-28T06:18:22.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.267+0000 7fa4bd5f9740 1 client.8286 shutdown 2024-05-28T06:18:22.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.267+0000 7fa4bd5f9740 1 -- 172.21.15.96:0/2002940542 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c6f18805d0 msgr2=0x55c6f18751c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:22.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.267+0000 7fa4bd5f9740 1 --2- 172.21.15.96:0/2002940542 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c6f18805d0 0x55c6f18751c0 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7fa4ac010420 tx=0x7fa4ac0108e0 comp rx=0 tx=0).stop 2024-05-28T06:18:22.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.267+0000 7fa4bd5f9740 1 -- 172.21.15.96:0/2002940542 shutdown_connections 2024-05-28T06:18:22.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.267+0000 7fa4bd5f9740 1 --2- 172.21.15.96:0/2002940542 >> [v2:172.21.15.96:6833/1614743177,v1:172.21.15.96:6835/1614743177] conn(0x55c6f1862880 0x55c6f1862c60 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:22.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.267+0000 7fa4bd5f9740 1 --2- 172.21.15.96:0/2002940542 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c6f18805d0 0x55c6f18751c0 unknown :-1 s=CLOSED pgs=1436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:22.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.267+0000 7fa4bd5f9740 1 --2- 172.21.15.96:0/2002940542 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x55c6f1864150 0x55c6f18f7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:22.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.267+0000 7fa4bd5f9740 1 --2- 172.21.15.96:0/2002940542 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55c6f185c500 0x55c6f18824a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:22.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.267+0000 7fa4bd5f9740 1 -- 172.21.15.96:0/2002940542 >> 172.21.15.96:0/2002940542 conn(0x55c6f1860490 msgr2=0x55c6f1834fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:18:22.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.267+0000 7fa4a0c00640 0 client.8286 ms_handle_reset on v2:172.21.15.96:6833/1614743177 2024-05-28T06:18:22.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.267+0000 7fa4bd5f9740 1 -- 172.21.15.96:0/2002940542 shutdown_connections 2024-05-28T06:18:22.273 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.267+0000 7fa4bd5f9740 1 -- 172.21.15.96:0/2002940542 wait complete. 2024-05-28T06:18:22.273 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.267+0000 7fa4bd5f9740 20 client.8286 trim_cache size 0 max 16384 2024-05-28T06:18:22.273 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:18:22.273 INFO:teuthology.orchestra.run.smithi096.stdout: "cluster_fsid": "e4a831b5-e6d1-47a7-96cb-bf85154ee0d2", 2024-05-28T06:18:22.273 INFO:teuthology.orchestra.run.smithi096.stdout: "whoami": 0, 2024-05-28T06:18:22.273 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 8103, 2024-05-28T06:18:22.273 INFO:teuthology.orchestra.run.smithi096.stdout: "want_state": "up:active", 2024-05-28T06:18:22.273 INFO:teuthology.orchestra.run.smithi096.stdout: "state": "up:active", 2024-05-28T06:18:22.273 INFO:teuthology.orchestra.run.smithi096.stdout: "fs_name": "cephfs", 2024-05-28T06:18:22.273 INFO:teuthology.orchestra.run.smithi096.stdout: "rank_uptime": 11.60241315, 2024-05-28T06:18:22.273 INFO:teuthology.orchestra.run.smithi096.stdout: "mdsmap_epoch": 52, 2024-05-28T06:18:22.274 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap_epoch": 89, 2024-05-28T06:18:22.274 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap_epoch_barrier": 89, 2024-05-28T06:18:22.274 INFO:teuthology.orchestra.run.smithi096.stdout: "uptime": 17.060131997999999 2024-05-28T06:18:22.274 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:18:22.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.268+0000 7fa4b9200640 1 -- 172.21.15.96:0/2350700441 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa49006da60 msgr2=0x7fa49006ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:22.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.268+0000 7fa4b9200640 1 --2- 172.21.15.96:0/2350700441 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa49006da60 0x7fa49006ff20 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fa4b4144fb0 tx=0x7fa49c007a40 comp rx=0 tx=0).stop 2024-05-28T06:18:22.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.268+0000 7fa4b9200640 1 -- 172.21.15.96:0/2350700441 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa4b406ee40 msgr2=0x7fa4b41543b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:22.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.268+0000 7fa4b9200640 1 --2- 172.21.15.96:0/2350700441 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa4b406ee40 0x7fa4b41543b0 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7fa4a4009d90 tx=0x7fa4a4004a40 comp rx=0 tx=0).stop 2024-05-28T06:18:22.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.268+0000 7fa4b9200640 1 -- 172.21.15.96:0/2350700441 shutdown_connections 2024-05-28T06:18:22.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.268+0000 7fa4b9200640 1 --2- 172.21.15.96:0/2350700441 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa49006da60 0x7fa49006ff20 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:22.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.268+0000 7fa4b9200640 1 --2- 172.21.15.96:0/2350700441 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa4b4156b30 0x7fa4b41f40f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:22.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.268+0000 7fa4b9200640 1 --2- 172.21.15.96:0/2350700441 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa4b40ad940 0x7fa4b41595f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:22.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.268+0000 7fa4b9200640 1 --2- 172.21.15.96:0/2350700441 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa4b406ee40 0x7fa4b41543b0 unknown :-1 s=CLOSED pgs=1432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:22.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.268+0000 7fa4b9200640 1 -- 172.21.15.96:0/2350700441 >> 172.21.15.96:0/2350700441 conn(0x7fa4b40b1fc0 msgr2=0x7fa4b40af060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:18:22.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.268+0000 7fa4b9200640 1 -- 172.21.15.96:0/2350700441 shutdown_connections 2024-05-28T06:18:22.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.268+0000 7fa4b9200640 1 -- 172.21.15.96:0/2350700441 wait complete. 2024-05-28T06:18:22.281 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-05-28T06:18:22.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.456+0000 7fc899a00640 1 Processor -- start 2024-05-28T06:18:22.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.456+0000 7fc899a00640 1 -- start start 2024-05-28T06:18:22.454 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.457+0000 7fc899a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fc89406ee40 0x7fc8940ad230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:22.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.457+0000 7fc899a00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc894156f10 con 0x7fc89406f2f0 2024-05-28T06:18:22.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.457+0000 7fc899a00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc89406dda0 con 0x7fc894156b30 2024-05-28T06:18:22.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.457+0000 7fc899a00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fc89406e080 con 0x7fc89406ee40 2024-05-28T06:18:22.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.457+0000 7fc88be00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fc89406ee40 0x7fc8940ad230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:22.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.458+0000 7fc88be00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fc89406ee40 0x7fc8940ad230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:40214/0 (socket says 172.21.15.96:40214) 2024-05-28T06:18:22.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.458+0000 7fc88be00640 1 -- 172.21.15.96:0/3036988261 learned_addr learned my addr 172.21.15.96:0/3036988261 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:18:22.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.458+0000 7fc88be00640 1 -- 172.21.15.96:0/3036988261 >> v1:172.21.15.123:6789/0 conn(0x7fc894156b30 legacy=0x7fc8940af6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:18:22.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.458+0000 7fc88be00640 1 -- 172.21.15.96:0/3036988261 >> v1:172.21.15.96:6789/0 conn(0x7fc89406f2f0 legacy=0x7fc8940b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:22.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.458+0000 7fc88be00640 1 -- 172.21.15.96:0/3036988261 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc89406e360 con 0x7fc89406ee40 2024-05-28T06:18:22.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.459+0000 7fc88be00640 1 --2- 172.21.15.96:0/3036988261 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc89406ee40 0x7fc8940ad230 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7fc87c009870 tx=0x7fc87c02a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b61c62522a876b57 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:22.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.459+0000 7fc893400640 1 -- 172.21.15.96:0/3036988261 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc87c037070 con 0x7fc89406ee40 2024-05-28T06:18:22.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.459+0000 7fc893400640 1 -- 172.21.15.96:0/3036988261 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc87c032440 con 0x7fc89406ee40 2024-05-28T06:18:22.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.459+0000 7fc893400640 1 -- 172.21.15.96:0/3036988261 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc87c032810 con 0x7fc89406ee40 2024-05-28T06:18:22.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.460+0000 7fc899a00640 1 -- 172.21.15.96:0/3036988261 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc89406ee40 msgr2=0x7fc8940ad230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:22.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.460+0000 7fc899a00640 1 --2- 172.21.15.96:0/3036988261 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc89406ee40 0x7fc8940ad230 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7fc87c009870 tx=0x7fc87c02a210 comp rx=0 tx=0).stop 2024-05-28T06:18:22.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.460+0000 7fc899a00640 1 -- 172.21.15.96:0/3036988261 shutdown_connections 2024-05-28T06:18:22.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.460+0000 7fc899a00640 1 --2- 172.21.15.96:0/3036988261 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc89406ee40 0x7fc8940ad230 unknown :-1 s=CLOSED pgs=1438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:22.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.460+0000 7fc899a00640 1 -- 172.21.15.96:0/3036988261 >> 172.21.15.96:0/3036988261 conn(0x7fc8940b1fc0 msgr2=0x7fc8940b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:18:22.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.460+0000 7fc899a00640 1 -- 172.21.15.96:0/3036988261 shutdown_connections 2024-05-28T06:18:22.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.460+0000 7fc899a00640 1 -- 172.21.15.96:0/3036988261 wait complete. 2024-05-28T06:18:22.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.462+0000 7fc899a00640 1 Processor -- start 2024-05-28T06:18:22.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.462+0000 7fc899a00640 1 -- start start 2024-05-28T06:18:22.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.463+0000 7fc899a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc89406ee40 0x7fc89414c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:22.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.463+0000 7fc899a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc89406f2f0 0x7fc894157110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:22.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.463+0000 7fc899a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc894156b30 0x7fc8941535d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:22.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.463+0000 7fc899a00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc8940b2db0 con 0x7fc89406f2f0 2024-05-28T06:18:22.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.463+0000 7fc899a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc8940b2c30 con 0x7fc894156b30 2024-05-28T06:18:22.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.463+0000 7fc899a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fc8940b39d0 con 0x7fc89406ee40 2024-05-28T06:18:22.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.463+0000 7fc88be00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc89406f2f0 0x7fc894157110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:22.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.463+0000 7fc88be00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc89406f2f0 0x7fc894157110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:40218/0 (socket says 172.21.15.96:40218) 2024-05-28T06:18:22.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.463+0000 7fc88be00640 1 -- 172.21.15.96:0/3866809464 learned_addr learned my addr 172.21.15.96:0/3866809464 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:18:22.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.463+0000 7fc88be00640 1 -- 172.21.15.96:0/3866809464 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc89406ee40 msgr2=0x7fc89414c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:22.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.463+0000 7fc88be00640 1 --2- 172.21.15.96:0/3866809464 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc89406ee40 0x7fc89414c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:22.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.463+0000 7fc88be00640 1 -- 172.21.15.96:0/3866809464 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc894156b30 msgr2=0x7fc8941535d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:22.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.463+0000 7fc88be00640 1 --2- 172.21.15.96:0/3866809464 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc894156b30 0x7fc8941535d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:22.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.463+0000 7fc88be00640 1 -- 172.21.15.96:0/3866809464 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8941e6cf0 con 0x7fc89406f2f0 2024-05-28T06:18:22.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.464+0000 7fc88be00640 1 --2- 172.21.15.96:0/3866809464 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc89406f2f0 0x7fc894157110 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7fc87c000c00 tx=0x7fc87c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:22.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.464+0000 7fc890c00640 1 -- 172.21.15.96:0/3866809464 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc87c037040 con 0x7fc89406f2f0 2024-05-28T06:18:22.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.464+0000 7fc890c00640 1 -- 172.21.15.96:0/3866809464 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc87c042070 con 0x7fc89406f2f0 2024-05-28T06:18:22.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.464+0000 7fc890c00640 1 -- 172.21.15.96:0/3866809464 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc87c030bb0 con 0x7fc89406f2f0 2024-05-28T06:18:22.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.464+0000 7fc899a00640 1 -- 172.21.15.96:0/3866809464 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc89414c480 con 0x7fc89406f2f0 2024-05-28T06:18:22.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.464+0000 7fc899a00640 1 -- 172.21.15.96:0/3866809464 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8941e6a10 con 0x7fc89406f2f0 2024-05-28T06:18:22.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.465+0000 7fc899a00640 1 -- 172.21.15.96:0/3866809464 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8940689e0 con 0x7fc89406f2f0 2024-05-28T06:18:22.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.466+0000 7fc890c00640 1 -- 172.21.15.96:0/3866809464 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 30) ==== 92621+0+0 (secure 0 0 0) 0x7fc87c047460 con 0x7fc89406f2f0 2024-05-28T06:18:22.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.467+0000 7fc890c00640 1 --2- 172.21.15.96:0/3866809464 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc87006dae0 0x7fc87006ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:22.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.467+0000 7fc890c00640 1 -- 172.21.15.96:0/3866809464 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 5737+0+0 (secure 0 0 0) 0x7fc87c0ab6d0 con 0x7fc89406f2f0 2024-05-28T06:18:22.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.468+0000 7fc893e00640 1 --2- 172.21.15.96:0/3866809464 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc87006dae0 0x7fc87006ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:22.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.469+0000 7fc893e00640 1 --2- 172.21.15.96:0/3866809464 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc87006dae0 0x7fc87006ffa0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fc884006fd0 tx=0x7fc884006bc0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:22.469 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.472+0000 7fc890c00640 1 -- 172.21.15.96:0/3866809464 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc8941e6cf0 con 0x7fc89406f2f0 2024-05-28T06:18:22.711 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.714+0000 7fc899a00640 1 -- 172.21.15.96:0/3866809464 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fc89406f7f0 con 0x7fc89406f2f0 2024-05-28T06:18:22.712 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.715+0000 7fc890c00640 1 -- 172.21.15.96:0/3866809464 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v52) ==== 53+0+73 (secure 0 0 0) 0x7fc87c045360 con 0x7fc89406f2f0 2024-05-28T06:18:22.712 INFO:teuthology.orchestra.run.smithi096.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-05-28T06:18:22.714 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.717+0000 7fc899a00640 1 -- 172.21.15.96:0/3866809464 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc87006dae0 msgr2=0x7fc87006ffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:22.715 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.717+0000 7fc899a00640 1 --2- 172.21.15.96:0/3866809464 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc87006dae0 0x7fc87006ffa0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fc884006fd0 tx=0x7fc884006bc0 comp rx=0 tx=0).stop 2024-05-28T06:18:22.715 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.718+0000 7fc899a00640 1 -- 172.21.15.96:0/3866809464 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc89406f2f0 msgr2=0x7fc894157110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:22.715 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.718+0000 7fc899a00640 1 --2- 172.21.15.96:0/3866809464 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc89406f2f0 0x7fc894157110 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7fc87c000c00 tx=0x7fc87c0046c0 comp rx=0 tx=0).stop 2024-05-28T06:18:22.715 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.718+0000 7fc899a00640 1 -- 172.21.15.96:0/3866809464 shutdown_connections 2024-05-28T06:18:22.715 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.718+0000 7fc899a00640 1 --2- 172.21.15.96:0/3866809464 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc87006dae0 0x7fc87006ffa0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:22.715 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.718+0000 7fc899a00640 1 --2- 172.21.15.96:0/3866809464 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc894156b30 0x7fc8941535d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:22.715 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.718+0000 7fc899a00640 1 --2- 172.21.15.96:0/3866809464 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc89406f2f0 0x7fc894157110 unknown :-1 s=CLOSED pgs=1439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:22.716 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.718+0000 7fc899a00640 1 --2- 172.21.15.96:0/3866809464 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc89406ee40 0x7fc89414c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:22.716 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.718+0000 7fc899a00640 1 -- 172.21.15.96:0/3866809464 >> 172.21.15.96:0/3866809464 conn(0x7fc8940b1fc0 msgr2=0x7fc8940b3ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:18:22.716 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.718+0000 7fc899a00640 1 -- 172.21.15.96:0/3866809464 shutdown_connections 2024-05-28T06:18:22.716 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:22.718+0000 7fc899a00640 1 -- 172.21.15.96:0/3866809464 wait complete. 2024-05-28T06:18:22.733 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-05-28T06:18:22.733 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-28T06:18:22.733 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi096.front.sepia.ceph.com 2024-05-28T06:18:22.733 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-05-28T06:18:22.733 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-28T06:18:22.734 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-05-28T06:18:22.734 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-05-28T06:18:22.734 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-28T06:18:22.734 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-28T06:18:22.734 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:18:22.734 DEBUG:teuthology.orchestra.run.smithi096:> ip addr 2024-05-28T06:18:22.784 INFO:teuthology.orchestra.run.smithi096.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-28T06:18:22.785 INFO:teuthology.orchestra.run.smithi096.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-28T06:18:22.785 INFO:teuthology.orchestra.run.smithi096.stdout: inet 127.0.0.1/8 scope host lo 2024-05-28T06:18:22.785 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:18:22.785 INFO:teuthology.orchestra.run.smithi096.stdout: inet6 ::1/128 scope host 2024-05-28T06:18:22.785 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:18:22.785 INFO:teuthology.orchestra.run.smithi096.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-28T06:18:22.785 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether 0c:c4:7a:d8:d9:bc brd ff:ff:ff:ff:ff:ff 2024-05-28T06:18:22.785 INFO:teuthology.orchestra.run.smithi096.stdout: altname enp7s0f0 2024-05-28T06:18:22.785 INFO:teuthology.orchestra.run.smithi096.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-28T06:18:22.785 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether 0c:c4:7a:d8:d9:bd brd ff:ff:ff:ff:ff:ff 2024-05-28T06:18:22.785 INFO:teuthology.orchestra.run.smithi096.stdout: altname enp7s0f1 2024-05-28T06:18:22.785 INFO:teuthology.orchestra.run.smithi096.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-28T06:18:22.785 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether 0c:c4:7a:8f:cc:0c brd ff:ff:ff:ff:ff:ff 2024-05-28T06:18:22.785 INFO:teuthology.orchestra.run.smithi096.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-28T06:18:22.786 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether 0c:c4:7a:8f:cc:0d brd ff:ff:ff:ff:ff:ff 2024-05-28T06:18:22.786 INFO:teuthology.orchestra.run.smithi096.stdout: inet 172.21.15.96/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-05-28T06:18:22.786 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft 41734sec preferred_lft 41734sec 2024-05-28T06:18:22.786 INFO:teuthology.orchestra.run.smithi096.stdout: inet6 fe80::ec4:7aff:fe8f:cc0d/64 scope link 2024-05-28T06:18:22.786 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:18:22.786 INFO:teuthology.orchestra.run.smithi096.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-05-28T06:18:22.786 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether ae:d9:f5:b0:4f:f3 brd ff:ff:ff:ff:ff:ff 2024-05-28T06:18:22.786 INFO:teuthology.orchestra.run.smithi096.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-05-28T06:18:22.786 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:18:22.786 INFO:teuthology.orchestra.run.smithi096.stdout: inet6 fe80::604e:fcff:fe60:7892/64 scope link 2024-05-28T06:18:22.786 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:18:22.786 INFO:teuthology.orchestra.run.smithi096.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-28T06:18:22.787 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether ce:6f:19:93:44:f9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-28T06:18:22.787 INFO:teuthology.orchestra.run.smithi096.stdout: inet6 fe80::cc6f:19ff:fe93:44f9/64 scope link 2024-05-28T06:18:22.787 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:18:22.787 INFO:teuthology.orchestra.run.smithi096.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-28T06:18:22.787 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether ae:d9:f5:b0:4f:f3 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-28T06:18:22.787 INFO:teuthology.orchestra.run.smithi096.stdout: inet6 fe80::acd9:f5ff:feb0:4ff3/64 scope link 2024-05-28T06:18:22.787 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:18:22.787 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:18:22.787 DEBUG:teuthology.orchestra.run.smithi096:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-28T06:18:22.856 INFO:teuthology.orchestra.run.smithi096.stdout:1 2024-05-28T06:18:22.860 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:18:22.860 DEBUG:teuthology.orchestra.run.smithi096:> route 2024-05-28T06:18:22.912 INFO:teuthology.orchestra.run.smithi096.stdout:Kernel IP routing table 2024-05-28T06:18:22.912 INFO:teuthology.orchestra.run.smithi096.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-28T06:18:22.912 INFO:teuthology.orchestra.run.smithi096.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-05-28T06:18:22.912 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-05-28T06:18:22.912 INFO:teuthology.orchestra.run.smithi096.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-28T06:18:22.913 DEBUG:teuthology.orchestra.run.smithi096:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-28T06:18:22.913 DEBUG:teuthology.orchestra.run.smithi096:> set -e 2024-05-28T06:18:22.913 DEBUG:teuthology.orchestra.run.smithi096:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-05-28T06:18:22.913 DEBUG:teuthology.orchestra.run.smithi096:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-05-28T06:18:22.913 DEBUG:teuthology.orchestra.run.smithi096:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-05-28T06:18:22.913 DEBUG:teuthology.orchestra.run.smithi096:> ') 2024-05-28T06:18:23.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:23 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-28T06:18:23.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:23 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-28T06:18:23.118 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:18:23.118 DEBUG:teuthology.orchestra.run.smithi096:> ip netns list 2024-05-28T06:18:23.165 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-05-28T06:18:23.165 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-05-28T06:18:23.166 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-28T06:18:23.166 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:18:23.166 DEBUG:teuthology.orchestra.run.smithi096:> sudo ip link set brx.0 up 2024-05-28T06:18:23.249 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-05-28T06:18:23.250 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-28T06:18:23.250 DEBUG:teuthology.orchestra.run.smithi096:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-05-28T06:18:23.304 INFO:teuthology.orchestra.run.smithi096.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-05-28T06:18:23.304 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-28T06:18:23.304 DEBUG:teuthology.orchestra.run.smithi096:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-05-28T06:18:23.354 DEBUG:teuthology.orchestra.run.smithi096:> sudo modprobe fuse 2024-05-28T06:18:23.432 DEBUG:teuthology.orchestra.run.smithi096:> cat /proc/self/mounts | awk '{print $2}' 2024-05-28T06:18:23.484 INFO:teuthology.orchestra.run.smithi096.stdout:/proc 2024-05-28T06:18:23.484 INFO:teuthology.orchestra.run.smithi096.stdout:/sys 2024-05-28T06:18:23.484 INFO:teuthology.orchestra.run.smithi096.stdout:/dev 2024-05-28T06:18:23.484 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/security 2024-05-28T06:18:23.485 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/shm 2024-05-28T06:18:23.485 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/pts 2024-05-28T06:18:23.485 INFO:teuthology.orchestra.run.smithi096.stdout:/run 2024-05-28T06:18:23.485 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/cgroup 2024-05-28T06:18:23.485 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/pstore 2024-05-28T06:18:23.485 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/bpf 2024-05-28T06:18:23.485 INFO:teuthology.orchestra.run.smithi096.stdout:/ 2024-05-28T06:18:23.485 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/selinux 2024-05-28T06:18:23.485 INFO:teuthology.orchestra.run.smithi096.stdout:/proc/sys/fs/binfmt_misc 2024-05-28T06:18:23.485 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/mqueue 2024-05-28T06:18:23.485 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/debug 2024-05-28T06:18:23.485 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/hugepages 2024-05-28T06:18:23.485 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/tracing 2024-05-28T06:18:23.486 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/fuse/connections 2024-05-28T06:18:23.486 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/config 2024-05-28T06:18:23.486 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-sysctl.service 2024-05-28T06:18:23.486 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-28T06:18:23.486 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-28T06:18:23.486 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/nfs/rpc_pipefs 2024-05-28T06:18:23.486 INFO:teuthology.orchestra.run.smithi096.stdout:/run/user/1000 2024-05-28T06:18:23.486 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph 2024-05-28T06:18:23.486 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-0 2024-05-28T06:18:23.486 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-1 2024-05-28T06:18:23.486 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-2 2024-05-28T06:18:23.486 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-3 2024-05-28T06:18:23.487 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns 2024-05-28T06:18:23.487 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-28T06:18:23.487 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-28T06:18:23.487 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-28T06:18:23.487 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-28T06:18:23.488 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:18:23.488 DEBUG:teuthology.orchestra.run.smithi096:> ls /sys/fs/fuse/connections 2024-05-28T06:18:23.538 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-05-28T06:18:23.538 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-05-28T06:18:23.543 DEBUG:teuthology.orchestra.run.smithi096:> sudo modprobe fuse 2024-05-28T06:18:23.608 DEBUG:teuthology.orchestra.run.smithi096:> cat /proc/self/mounts | awk '{print $2}' 2024-05-28T06:18:23.648 INFO:teuthology.orchestra.run.smithi096.stdout:/proc 2024-05-28T06:18:23.648 INFO:teuthology.orchestra.run.smithi096.stdout:/sys 2024-05-28T06:18:23.648 INFO:teuthology.orchestra.run.smithi096.stdout:/dev 2024-05-28T06:18:23.648 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/security 2024-05-28T06:18:23.648 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/shm 2024-05-28T06:18:23.649 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/pts 2024-05-28T06:18:23.649 INFO:teuthology.orchestra.run.smithi096.stdout:/run 2024-05-28T06:18:23.649 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/cgroup 2024-05-28T06:18:23.649 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/pstore 2024-05-28T06:18:23.649 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/bpf 2024-05-28T06:18:23.649 INFO:teuthology.orchestra.run.smithi096.stdout:/ 2024-05-28T06:18:23.649 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/selinux 2024-05-28T06:18:23.649 INFO:teuthology.orchestra.run.smithi096.stdout:/proc/sys/fs/binfmt_misc 2024-05-28T06:18:23.649 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/mqueue 2024-05-28T06:18:23.649 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/debug 2024-05-28T06:18:23.649 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/hugepages 2024-05-28T06:18:23.650 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/tracing 2024-05-28T06:18:23.650 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/fuse/connections 2024-05-28T06:18:23.650 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/config 2024-05-28T06:18:23.650 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-sysctl.service 2024-05-28T06:18:23.650 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-28T06:18:23.650 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-28T06:18:23.650 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/nfs/rpc_pipefs 2024-05-28T06:18:23.650 INFO:teuthology.orchestra.run.smithi096.stdout:/run/user/1000 2024-05-28T06:18:23.650 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph 2024-05-28T06:18:23.651 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-0 2024-05-28T06:18:23.651 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-1 2024-05-28T06:18:23.651 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-2 2024-05-28T06:18:23.651 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-3 2024-05-28T06:18:23.651 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns 2024-05-28T06:18:23.651 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-28T06:18:23.651 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-28T06:18:23.651 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-28T06:18:23.651 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-28T06:18:23.652 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:18:23.652 DEBUG:teuthology.orchestra.run.smithi096:> ls /sys/fs/fuse/connections 2024-05-28T06:18:23.706 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi096.stderr:2024-05-28T06:18:23.709+0000 7fa21900c480 -1 init, newargv = 0x55560c027940 newargc=15 2024-05-28T06:18:23.707 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi096.stderr:ceph-fuse[71726]: starting ceph client 2024-05-28T06:18:23.715 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi096.stderr:ceph-fuse[71726]: starting fuse 2024-05-28T06:18:23.734 INFO:teuthology.orchestra.run.smithi096.stdout:40 2024-05-28T06:18:23.734 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40] 2024-05-28T06:18:23.734 DEBUG:teuthology.orchestra.run.smithi096:> sudo stdin-killer -- python3 -c ' 2024-05-28T06:18:23.734 DEBUG:teuthology.orchestra.run.smithi096:> import glob 2024-05-28T06:18:23.734 DEBUG:teuthology.orchestra.run.smithi096:> import re 2024-05-28T06:18:23.734 DEBUG:teuthology.orchestra.run.smithi096:> import os 2024-05-28T06:18:23.735 DEBUG:teuthology.orchestra.run.smithi096:> import subprocess 2024-05-28T06:18:23.735 DEBUG:teuthology.orchestra.run.smithi096:> 2024-05-28T06:18:23.735 DEBUG:teuthology.orchestra.run.smithi096:> def _find_admin_socket(client_name): 2024-05-28T06:18:23.735 DEBUG:teuthology.orchestra.run.smithi096:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-05-28T06:18:23.735 DEBUG:teuthology.orchestra.run.smithi096:> files = glob.glob(asok_path) 2024-05-28T06:18:23.735 DEBUG:teuthology.orchestra.run.smithi096:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-05-28T06:18:23.735 DEBUG:teuthology.orchestra.run.smithi096:> 2024-05-28T06:18:23.735 DEBUG:teuthology.orchestra.run.smithi096:> # Given a non-glob path, it better be there 2024-05-28T06:18:23.735 DEBUG:teuthology.orchestra.run.smithi096:> if "*" not in asok_path: 2024-05-28T06:18:23.735 DEBUG:teuthology.orchestra.run.smithi096:> assert(len(files) == 1) 2024-05-28T06:18:23.735 DEBUG:teuthology.orchestra.run.smithi096:> return files[0] 2024-05-28T06:18:23.735 DEBUG:teuthology.orchestra.run.smithi096:> 2024-05-28T06:18:23.735 DEBUG:teuthology.orchestra.run.smithi096:> for f in files: 2024-05-28T06:18:23.735 DEBUG:teuthology.orchestra.run.smithi096:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-05-28T06:18:23.735 DEBUG:teuthology.orchestra.run.smithi096:> if os.path.exists("/proc/{0}".format(pid)): 2024-05-28T06:18:23.736 DEBUG:teuthology.orchestra.run.smithi096:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-05-28T06:18:23.736 DEBUG:teuthology.orchestra.run.smithi096:> contents = proc_f.read() 2024-05-28T06:18:23.736 DEBUG:teuthology.orchestra.run.smithi096:> if mountpoint in contents: 2024-05-28T06:18:23.736 DEBUG:teuthology.orchestra.run.smithi096:> return f 2024-05-28T06:18:23.736 DEBUG:teuthology.orchestra.run.smithi096:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-05-28T06:18:23.736 DEBUG:teuthology.orchestra.run.smithi096:> 2024-05-28T06:18:23.736 DEBUG:teuthology.orchestra.run.smithi096:> print(_find_admin_socket("client.0")) 2024-05-28T06:18:23.736 DEBUG:teuthology.orchestra.run.smithi096:> ' 2024-05-28T06:18:23.927 INFO:teuthology.orchestra.run.smithi096.stdout:/var/run/ceph/ceph-client.0.71726.asok 2024-05-28T06:18:23.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:23 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-28T06:18:23.944 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.71726.asok 2024-05-28T06:18:23.944 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:18:23.944 DEBUG:teuthology.orchestra.run.smithi096:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.71726.asok status 2024-05-28T06:18:24.169 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:18:24.169 INFO:teuthology.orchestra.run.smithi096.stdout: "metadata": { 2024-05-28T06:18:24.169 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph_sha1": "9ed025fbf4406fb54980a0647506b15895b6d863", 2024-05-28T06:18:24.170 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph_version": "ceph version 19.0.0-3154-g9ed025fb (9ed025fbf4406fb54980a0647506b15895b6d863) squid (dev)", 2024-05-28T06:18:24.170 INFO:teuthology.orchestra.run.smithi096.stdout: "entity_id": "0", 2024-05-28T06:18:24.170 INFO:teuthology.orchestra.run.smithi096.stdout: "hostname": "smithi096", 2024-05-28T06:18:24.170 INFO:teuthology.orchestra.run.smithi096.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-05-28T06:18:24.170 INFO:teuthology.orchestra.run.smithi096.stdout: "pid": "71726", 2024-05-28T06:18:24.170 INFO:teuthology.orchestra.run.smithi096.stdout: "root": "/" 2024-05-28T06:18:24.170 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:18:24.170 INFO:teuthology.orchestra.run.smithi096.stdout: "dentry_count": 0, 2024-05-28T06:18:24.170 INFO:teuthology.orchestra.run.smithi096.stdout: "dentry_pinned_count": 0, 2024-05-28T06:18:24.170 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 8301, 2024-05-28T06:18:24.170 INFO:teuthology.orchestra.run.smithi096.stdout: "inst": { 2024-05-28T06:18:24.170 INFO:teuthology.orchestra.run.smithi096.stdout: "name": { 2024-05-28T06:18:24.170 INFO:teuthology.orchestra.run.smithi096.stdout: "type": "client", 2024-05-28T06:18:24.171 INFO:teuthology.orchestra.run.smithi096.stdout: "num": 8301 2024-05-28T06:18:24.171 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:18:24.171 INFO:teuthology.orchestra.run.smithi096.stdout: "addr": { 2024-05-28T06:18:24.171 INFO:teuthology.orchestra.run.smithi096.stdout: "type": "v1", 2024-05-28T06:18:24.171 INFO:teuthology.orchestra.run.smithi096.stdout: "addr": "192.168.0.1:0", 2024-05-28T06:18:24.171 INFO:teuthology.orchestra.run.smithi096.stdout: "nonce": 1005249312 2024-05-28T06:18:24.171 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:18:24.171 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:18:24.171 INFO:teuthology.orchestra.run.smithi096.stdout: "addr": { 2024-05-28T06:18:24.171 INFO:teuthology.orchestra.run.smithi096.stdout: "type": "v1", 2024-05-28T06:18:24.171 INFO:teuthology.orchestra.run.smithi096.stdout: "addr": "192.168.0.1:0", 2024-05-28T06:18:24.171 INFO:teuthology.orchestra.run.smithi096.stdout: "nonce": 1005249312 2024-05-28T06:18:24.171 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:18:24.172 INFO:teuthology.orchestra.run.smithi096.stdout: "inst_str": "client.8301 192.168.0.1:0/1005249312", 2024-05-28T06:18:24.172 INFO:teuthology.orchestra.run.smithi096.stdout: "addr_str": "192.168.0.1:0/1005249312", 2024-05-28T06:18:24.172 INFO:teuthology.orchestra.run.smithi096.stdout: "inode_count": 1, 2024-05-28T06:18:24.172 INFO:teuthology.orchestra.run.smithi096.stdout: "mds_epoch": 52, 2024-05-28T06:18:24.172 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_epoch": 89, 2024-05-28T06:18:24.172 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_epoch_barrier": 0, 2024-05-28T06:18:24.172 INFO:teuthology.orchestra.run.smithi096.stdout: "blocklisted": false, 2024-05-28T06:18:24.172 INFO:teuthology.orchestra.run.smithi096.stdout: "fs_name": "cephfs" 2024-05-28T06:18:24.172 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:18:24.183 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:18:24.184 DEBUG:teuthology.orchestra.run.smithi096:> stat --file-system '--printf=%T 2024-05-28T06:18:24.184 DEBUG:teuthology.orchestra.run.smithi096:> ' -- /home/ubuntu/cephtest/mnt.0 2024-05-28T06:18:24.233 INFO:teuthology.orchestra.run.smithi096.stdout:fuseblk 2024-05-28T06:18:24.233 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-05-28T06:18:24.233 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:18:24.234 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-05-28T06:18:24.318 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-05-28T06:18:24.525 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:24.528+0000 7f51fc600640 1 Processor -- start 2024-05-28T06:18:24.525 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:24.528+0000 7f51fc600640 1 -- start start 2024-05-28T06:18:24.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:24.529+0000 7f51fc600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f51f415e830 0x7f51f406dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:24.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:24.529+0000 7f51fc600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f51f40b3510 0x7f51f406f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:24.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:24.529+0000 7f51fc600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51f406e190 con 0x7f51f4071c00 2024-05-28T06:18:24.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:24.529+0000 7f51fc600640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f51f406f770 con 0x7f51f415e830 2024-05-28T06:18:24.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:24.529+0000 7f51fc600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f51f406e470 con 0x7f51f40b3510 2024-05-28T06:18:24.528 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:24.530+0000 7f51f9200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f51f40b3510 0x7f51f406f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:24.528 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:24.530+0000 7f51fa600640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f51f4071c00 0x7f51f40ad130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.96:36304/0 (socket says 172.21.15.96:36304) 2024-05-28T06:18:24.528 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:24.530+0000 7f51fa600640 1 -- 172.21.15.96:0/1133149841 learned_addr learned my addr 172.21.15.96:0/1133149841 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:18:24.528 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:24.530+0000 7f51f9c00640 1 --2- 172.21.15.96:0/1133149841 >> v2:172.21.15.123:3300/0 conn(0x7f51f415e830 0x7f51f406dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:24.528 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:24.531+0000 7f51f9200640 1 -- 172.21.15.96:0/1133149841 >> v2:172.21.15.123:3300/0 conn(0x7f51f415e830 msgr2=0x7f51f406dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:18:24.528 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:24.531+0000 7f51f9200640 1 --2- 172.21.15.96:0/1133149841 >> v2:172.21.15.123:3300/0 conn(0x7f51f415e830 0x7f51f406dbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:24.529 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:24.531+0000 7f51f9200640 1 -- 172.21.15.96:0/1133149841 >> v1:172.21.15.123:6789/0 conn(0x7f51f4071c00 legacy=0x7f51f40ad130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:24.529 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:24.531+0000 7f51f9200640 1 -- 172.21.15.96:0/1133149841 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51f406e750 con 0x7f51f40b3510 2024-05-28T06:18:24.529 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:24.532+0000 7f51f9200640 1 --2- 172.21.15.96:0/1133149841 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f51f40b3510 0x7f51f406f230 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f51e00097a0 tx=0x7f51e002a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d4a18bae2143d58 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:24.530 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:24.533+0000 7f51f3e00640 1 -- 172.21.15.96:0/1133149841 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51e0037070 con 0x7f51f40b3510 2024-05-28T06:18:24.530 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:24.533+0000 7f51f3e00640 1 -- 172.21.15.96:0/1133149841 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51e0032440 con 0x7f51f40b3510 2024-05-28T06:18:24.530 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:24.533+0000 7f51f3e00640 1 -- 172.21.15.96:0/1133149841 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51e0032810 con 0x7f51f40b3510 2024-05-28T06:18:24.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:24.534+0000 7f51fc600640 1 -- 172.21.15.96:0/1133149841 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f51f40b3510 msgr2=0x7f51f406f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:24.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:24.534+0000 7f51fc600640 1 --2- 172.21.15.96:0/1133149841 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f51f40b3510 0x7f51f406f230 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f51e00097a0 tx=0x7f51e002a180 comp rx=0 tx=0).stop 2024-05-28T06:18:24.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:24.534+0000 7f51fc600640 1 -- 172.21.15.96:0/1133149841 shutdown_connections 2024-05-28T06:18:24.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:24.534+0000 7f51fc600640 1 --2- 172.21.15.96:0/1133149841 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f51f40b3510 0x7f51f406f230 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:24.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:24.534+0000 7f51fc600640 1 --2- 172.21.15.96:0/1133149841 >> v2:172.21.15.123:3300/0 conn(0x7f51f415e830 0x7f51f406dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:24.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:24.534+0000 7f51fc600640 1 -- 172.21.15.96:0/1133149841 >> 172.21.15.96:0/1133149841 conn(0x7f51f40b1e50 msgr2=0x7f51f40b2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:18:24.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:24.534+0000 7f51fc600640 1 -- 172.21.15.96:0/1133149841 shutdown_connections 2024-05-28T06:18:24.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:24.534+0000 7f51fc600640 1 -- 172.21.15.96:0/1133149841 wait complete. 2024-05-28T06:18:24.533 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:24.536+0000 7f51fc600640 1 Processor -- start 2024-05-28T06:18:24.533 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:24.536+0000 7f51fc600640 1 -- start start 2024-05-28T06:18:24.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:24.537+0000 7f51fc600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f51f4071c00 0x7f51f4156780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:24.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:24.537+0000 7f51fc600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f51f40b3510 0x7f51f41f1a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:24.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:24.537+0000 7f51fc600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f51f415e830 0x7f51f41f4e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:24.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:24.537+0000 7f51fc600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f51f415ec10 con 0x7f51f415e830 2024-05-28T06:18:24.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:24.537+0000 7f51fc600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f51f406ed10 con 0x7f51f40b3510 2024-05-28T06:18:24.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:24.537+0000 7f51fc600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f51f40728c0 con 0x7f51f4071c00 2024-05-28T06:18:24.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:24.537+0000 7f51fa600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f51f415e830 0x7f51f41f4e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:24.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:24.537+0000 7f51fa600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f51f415e830 0x7f51f41f4e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:40232/0 (socket says 172.21.15.96:40232) 2024-05-28T06:18:24.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:24.537+0000 7f51fa600640 1 -- 172.21.15.96:0/365982199 learned_addr learned my addr 172.21.15.96:0/365982199 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:18:24.536 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:24.538+0000 7f51f9200640 1 --2- 172.21.15.96:0/365982199 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f51f40b3510 0x7f51f41f1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:24.536 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:24.538+0000 7f51f9c00640 1 --2- 172.21.15.96:0/365982199 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f51f4071c00 0x7f51f4156780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:24.536 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:24.538+0000 7f51fa600640 1 -- 172.21.15.96:0/365982199 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f51f4071c00 msgr2=0x7f51f4156780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:24.536 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:24.538+0000 7f51fa600640 1 --2- 172.21.15.96:0/365982199 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f51f4071c00 0x7f51f4156780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:24.536 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:24.538+0000 7f51fa600640 1 -- 172.21.15.96:0/365982199 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f51f40b3510 msgr2=0x7f51f41f1a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:24.536 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:24.538+0000 7f51fa600640 1 --2- 172.21.15.96:0/365982199 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f51f40b3510 0x7f51f41f1a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:24.536 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:24.538+0000 7f51fa600640 1 -- 172.21.15.96:0/365982199 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51f41f53b0 con 0x7f51f415e830 2024-05-28T06:18:24.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:24.538+0000 7f51fa600640 1 --2- 172.21.15.96:0/365982199 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f51f415e830 0x7f51f41f4e70 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7f51ec00cbf0 tx=0x7f51ec00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:24.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:24.538+0000 7f51f1600640 1 -- 172.21.15.96:0/365982199 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51ec017070 con 0x7f51f415e830 2024-05-28T06:18:24.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:24.538+0000 7f51f1600640 1 -- 172.21.15.96:0/365982199 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51ec00ad40 con 0x7f51f415e830 2024-05-28T06:18:24.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:24.538+0000 7f51f1600640 1 -- 172.21.15.96:0/365982199 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51ec012600 con 0x7f51f415e830 2024-05-28T06:18:24.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:24.539+0000 7f51fc600640 1 -- 172.21.15.96:0/365982199 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51f4156d80 con 0x7f51f415e830 2024-05-28T06:18:24.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:24.539+0000 7f51fc600640 1 -- 172.21.15.96:0/365982199 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51f4157340 con 0x7f51f415e830 2024-05-28T06:18:24.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:24.540+0000 7f51fc600640 1 -- 172.21.15.96:0/365982199 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51f40689e0 con 0x7f51f415e830 2024-05-28T06:18:24.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:24.541+0000 7f51f1600640 1 -- 172.21.15.96:0/365982199 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 30) ==== 92621+0+0 (secure 0 0 0) 0x7f51ec01a070 con 0x7f51f415e830 2024-05-28T06:18:24.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:24.542+0000 7f51f1600640 1 --2- 172.21.15.96:0/365982199 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f51d406da60 0x7f51d406ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:24.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:24.542+0000 7f51f1600640 1 -- 172.21.15.96:0/365982199 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 5737+0+0 (secure 0 0 0) 0x7f51ec08a730 con 0x7f51f415e830 2024-05-28T06:18:24.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:24.543+0000 7f51f9c00640 1 --2- 172.21.15.96:0/365982199 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f51d406da60 0x7f51d406ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:24.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:24.544+0000 7f51f9c00640 1 --2- 172.21.15.96:0/365982199 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f51d406da60 0x7f51d406ff20 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f51e8009a30 tx=0x7f51e8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:24.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:24.546+0000 7f51f1600640 1 -- 172.21.15.96:0/365982199 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f51ec059c10 con 0x7f51f415e830 2024-05-28T06:18:24.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:24.791+0000 7f51fc600640 1 -- 172.21.15.96:0/365982199 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) -- 0x7f51f406cc20 con 0x7f51f415e830 2024-05-28T06:18:25.759 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:25.762+0000 7f51f1600640 1 -- 172.21.15.96:0/365982199 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v53) ==== 136+0+0 (secure 0 0 0) 0x7f51ec05df10 con 0x7f51f415e830 2024-05-28T06:18:25.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:25.766+0000 7f51fc600640 1 -- 172.21.15.96:0/365982199 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f51d406da60 msgr2=0x7f51d406ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:25.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:25.766+0000 7f51fc600640 1 --2- 172.21.15.96:0/365982199 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f51d406da60 0x7f51d406ff20 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f51e8009a30 tx=0x7f51e8008040 comp rx=0 tx=0).stop 2024-05-28T06:18:25.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:25.766+0000 7f51fc600640 1 -- 172.21.15.96:0/365982199 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f51f415e830 msgr2=0x7f51f41f4e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:25.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:25.766+0000 7f51fc600640 1 --2- 172.21.15.96:0/365982199 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f51f415e830 0x7f51f41f4e70 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7f51ec00cbf0 tx=0x7f51ec00a430 comp rx=0 tx=0).stop 2024-05-28T06:18:25.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:25.766+0000 7f51fc600640 1 -- 172.21.15.96:0/365982199 shutdown_connections 2024-05-28T06:18:25.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:25.766+0000 7f51fc600640 1 --2- 172.21.15.96:0/365982199 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f51d406da60 0x7f51d406ff20 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:25.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:25.766+0000 7f51fc600640 1 --2- 172.21.15.96:0/365982199 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f51f415e830 0x7f51f41f4e70 unknown :-1 s=CLOSED pgs=1443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:25.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:25.766+0000 7f51fc600640 1 --2- 172.21.15.96:0/365982199 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f51f40b3510 0x7f51f41f1a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:25.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:25.766+0000 7f51fc600640 1 --2- 172.21.15.96:0/365982199 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f51f4071c00 0x7f51f4156780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:25.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:25.766+0000 7f51fc600640 1 -- 172.21.15.96:0/365982199 >> 172.21.15.96:0/365982199 conn(0x7f51f40b1e50 msgr2=0x7f51f40b2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:18:25.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:25.766+0000 7f51fc600640 1 -- 172.21.15.96:0/365982199 shutdown_connections 2024-05-28T06:18:25.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:25.766+0000 7f51fc600640 1 -- 172.21.15.96:0/365982199 wait complete. 2024-05-28T06:18:25.872 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-05-28T06:18:26.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.091+0000 7f23a5600640 1 Processor -- start 2024-05-28T06:18:26.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.091+0000 7f23a5600640 1 -- start start 2024-05-28T06:18:26.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.093+0000 7f23a5600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f23a015e870 0x7f23a00d0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:26.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.093+0000 7f23a5600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f23a00d14d0 0x7f23a00cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:26.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.093+0000 7f23a5600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23a00cfd00 con 0x7f23a00cf870 2024-05-28T06:18:26.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.093+0000 7f23a5600640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f23a00d2150 con 0x7f23a015e870 2024-05-28T06:18:26.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.093+0000 7f23a5600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f23a015ec50 con 0x7f23a00d14d0 2024-05-28T06:18:26.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.094+0000 7f239f400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f23a00d14d0 0x7f23a00cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:26.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.094+0000 7f239fe00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f23a015e870 0x7f23a00d0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:26.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.094+0000 7f239fe00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f23a015e870 0x7f23a00d0f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.96:48432/0 (socket says 172.21.15.96:48432) 2024-05-28T06:18:26.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.094+0000 7f239fe00640 1 -- 172.21.15.96:0/3989093889 learned_addr learned my addr 172.21.15.96:0/3989093889 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:18:26.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.094+0000 7f239f400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f23a00d14d0 0x7f23a00cf330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.96:59836/0 (socket says 172.21.15.96:59836) 2024-05-28T06:18:26.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.094+0000 7f23a4c00640 1 --1- 172.21.15.96:0/3989093889 >> v1:172.21.15.123:6789/0 conn(0x7f23a00cf870 0x7f23a0162360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.96:36310/0 (socket says 172.21.15.96:36310) 2024-05-28T06:18:26.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.095+0000 7f239f400640 1 -- 172.21.15.96:0/3989093889 >> v2:172.21.15.123:3300/0 conn(0x7f23a015e870 msgr2=0x7f23a00d0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:18:26.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.095+0000 7f239f400640 1 --2- 172.21.15.96:0/3989093889 >> v2:172.21.15.123:3300/0 conn(0x7f23a015e870 0x7f23a00d0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:26.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.095+0000 7f239f400640 1 -- 172.21.15.96:0/3989093889 >> v1:172.21.15.123:6789/0 conn(0x7f23a00cf870 legacy=0x7f23a0162360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:26.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.095+0000 7f239f400640 1 -- 172.21.15.96:0/3989093889 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23a00cff80 con 0x7f23a00d14d0 2024-05-28T06:18:26.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.096+0000 7f239f400640 1 --2- 172.21.15.96:0/3989093889 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f23a00d14d0 0x7f23a00cf330 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f2388009de0 tx=0x7f238802a6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=851e0e17dbe86d61 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:26.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.097+0000 7f239ea00640 1 -- 172.21.15.96:0/3989093889 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2388007740 con 0x7f23a00d14d0 2024-05-28T06:18:26.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.097+0000 7f239ea00640 1 -- 172.21.15.96:0/3989093889 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23880078e0 con 0x7f23a00d14d0 2024-05-28T06:18:26.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.097+0000 7f239ea00640 1 -- 172.21.15.96:0/3989093889 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2388007cb0 con 0x7f23a00d14d0 2024-05-28T06:18:26.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.098+0000 7f23a5600640 1 -- 172.21.15.96:0/3989093889 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f23a00d14d0 msgr2=0x7f23a00cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:26.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.098+0000 7f23a5600640 1 --2- 172.21.15.96:0/3989093889 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f23a00d14d0 0x7f23a00cf330 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f2388009de0 tx=0x7f238802a6c0 comp rx=0 tx=0).stop 2024-05-28T06:18:26.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.098+0000 7f23a5600640 1 -- 172.21.15.96:0/3989093889 shutdown_connections 2024-05-28T06:18:26.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.098+0000 7f23a5600640 1 --2- 172.21.15.96:0/3989093889 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f23a00d14d0 0x7f23a00cf330 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:26.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.098+0000 7f23a5600640 1 --2- 172.21.15.96:0/3989093889 >> v2:172.21.15.123:3300/0 conn(0x7f23a015e870 0x7f23a00d0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:26.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.098+0000 7f23a5600640 1 -- 172.21.15.96:0/3989093889 >> 172.21.15.96:0/3989093889 conn(0x7f23a008bfc0 msgr2=0x7f23a00d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:18:26.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.098+0000 7f23a5600640 1 -- 172.21.15.96:0/3989093889 shutdown_connections 2024-05-28T06:18:26.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.098+0000 7f23a5600640 1 -- 172.21.15.96:0/3989093889 wait complete. 2024-05-28T06:18:26.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.100+0000 7f23a5600640 1 Processor -- start 2024-05-28T06:18:26.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.100+0000 7f23a5600640 1 -- start start 2024-05-28T06:18:26.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.101+0000 7f23a5600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f23a00cf870 0x7f23a014f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:26.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.101+0000 7f23a5600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f23a00d14d0 0x7f23a00b8790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:26.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.101+0000 7f23a5600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f23a015e870 0x7f23a00b8cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:26.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.101+0000 7f23a5600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f23a00d0820 con 0x7f23a015e870 2024-05-28T06:18:26.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.101+0000 7f23a5600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f23a00d0540 con 0x7f23a00cf870 2024-05-28T06:18:26.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.101+0000 7f23a5600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f23a00d4fb0 con 0x7f23a00d14d0 2024-05-28T06:18:26.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.101+0000 7f23a4c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f23a015e870 0x7f23a00b8cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:26.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.102+0000 7f23a4c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f23a015e870 0x7f23a00b8cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:40234/0 (socket says 172.21.15.96:40234) 2024-05-28T06:18:26.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.102+0000 7f23a4c00640 1 -- 172.21.15.96:0/2570149768 learned_addr learned my addr 172.21.15.96:0/2570149768 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:18:26.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.102+0000 7f239fe00640 1 --2- 172.21.15.96:0/2570149768 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f23a00cf870 0x7f23a014f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:26.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.102+0000 7f239f400640 1 --2- 172.21.15.96:0/2570149768 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f23a00d14d0 0x7f23a00b8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:26.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.102+0000 7f23a4c00640 1 -- 172.21.15.96:0/2570149768 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f23a00d14d0 msgr2=0x7f23a00b8790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:26.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.102+0000 7f23a4c00640 1 --2- 172.21.15.96:0/2570149768 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f23a00d14d0 0x7f23a00b8790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:26.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.102+0000 7f23a4c00640 1 -- 172.21.15.96:0/2570149768 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f23a00cf870 msgr2=0x7f23a014f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:26.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.102+0000 7f23a4c00640 1 --2- 172.21.15.96:0/2570149768 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f23a00cf870 0x7f23a014f9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:26.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.102+0000 7f23a4c00640 1 -- 172.21.15.96:0/2570149768 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23a014ff10 con 0x7f23a015e870 2024-05-28T06:18:26.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.102+0000 7f23a4c00640 1 --2- 172.21.15.96:0/2570149768 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f23a015e870 0x7f23a00b8cd0 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f239800c3f0 tx=0x7f239800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:26.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.103+0000 7f2397e00640 1 -- 172.21.15.96:0/2570149768 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23980074e0 con 0x7f23a015e870 2024-05-28T06:18:26.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.103+0000 7f2397e00640 1 -- 172.21.15.96:0/2570149768 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2398007680 con 0x7f23a015e870 2024-05-28T06:18:26.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.103+0000 7f2397e00640 1 -- 172.21.15.96:0/2570149768 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2398004a00 con 0x7f23a015e870 2024-05-28T06:18:26.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.103+0000 7f23a5600640 1 -- 172.21.15.96:0/2570149768 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23a015ffe0 con 0x7f23a015e870 2024-05-28T06:18:26.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.103+0000 7f23a5600640 1 -- 172.21.15.96:0/2570149768 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23a0160350 con 0x7f23a015e870 2024-05-28T06:18:26.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.104+0000 7f23a5600640 1 -- 172.21.15.96:0/2570149768 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23a00689e0 con 0x7f23a015e870 2024-05-28T06:18:26.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.105+0000 7f2397e00640 1 -- 172.21.15.96:0/2570149768 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 30) ==== 92621+0+0 (secure 0 0 0) 0x7f239801a020 con 0x7f23a015e870 2024-05-28T06:18:26.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.106+0000 7f2397e00640 1 --2- 172.21.15.96:0/2570149768 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f237c06da90 0x7f237c06ff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:26.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.106+0000 7f2397e00640 1 -- 172.21.15.96:0/2570149768 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 5737+0+0 (secure 0 0 0) 0x7f239808a0e0 con 0x7f23a015e870 2024-05-28T06:18:26.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.107+0000 7f239fe00640 1 --2- 172.21.15.96:0/2570149768 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f237c06da90 0x7f237c06ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:26.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.108+0000 7f239fe00640 1 --2- 172.21.15.96:0/2570149768 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f237c06da90 0x7f237c06ff50 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f2390009bd0 tx=0x7f2390009240 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:26.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.111+0000 7f2397e00640 1 -- 172.21.15.96:0/2570149768 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f23a014ff10 con 0x7f23a015e870 2024-05-28T06:18:26.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.338+0000 7f23a5600640 1 -- 172.21.15.96:0/2570149768 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7f23a00b4130 con 0x7f237c06da90 2024-05-28T06:18:26.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.341+0000 7f2397e00640 1 -- 172.21.15.96:0/2570149768 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7f23a00b4130 con 0x7f237c06da90 2024-05-28T06:18:26.339 INFO:teuthology.orchestra.run.smithi096.stdout:[ 2024-05-28T06:18:26.339 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:18:26.339 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "cephfs" 2024-05-28T06:18:26.339 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:18:26.339 INFO:teuthology.orchestra.run.smithi096.stdout:] 2024-05-28T06:18:26.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.344+0000 7f23a5600640 1 -- 172.21.15.96:0/2570149768 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f237c06da90 msgr2=0x7f237c06ff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:26.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.344+0000 7f23a5600640 1 --2- 172.21.15.96:0/2570149768 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f237c06da90 0x7f237c06ff50 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f2390009bd0 tx=0x7f2390009240 comp rx=0 tx=0).stop 2024-05-28T06:18:26.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.344+0000 7f23a5600640 1 -- 172.21.15.96:0/2570149768 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f23a015e870 msgr2=0x7f23a00b8cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:26.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.344+0000 7f23a5600640 1 --2- 172.21.15.96:0/2570149768 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f23a015e870 0x7f23a00b8cd0 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f239800c3f0 tx=0x7f239800c8b0 comp rx=0 tx=0).stop 2024-05-28T06:18:26.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.345+0000 7f23a5600640 1 -- 172.21.15.96:0/2570149768 shutdown_connections 2024-05-28T06:18:26.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.345+0000 7f23a5600640 1 --2- 172.21.15.96:0/2570149768 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f237c06da90 0x7f237c06ff50 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:26.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.345+0000 7f23a5600640 1 --2- 172.21.15.96:0/2570149768 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f23a015e870 0x7f23a00b8cd0 unknown :-1 s=CLOSED pgs=1444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:26.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.345+0000 7f23a5600640 1 --2- 172.21.15.96:0/2570149768 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f23a00d14d0 0x7f23a00b8790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:26.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.345+0000 7f23a5600640 1 --2- 172.21.15.96:0/2570149768 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f23a00cf870 0x7f23a014f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:26.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.345+0000 7f23a5600640 1 -- 172.21.15.96:0/2570149768 >> 172.21.15.96:0/2570149768 conn(0x7f23a008bfc0 msgr2=0x7f23a00b10f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:18:26.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.345+0000 7f23a5600640 1 -- 172.21.15.96:0/2570149768 shutdown_connections 2024-05-28T06:18:26.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.345+0000 7f23a5600640 1 -- 172.21.15.96:0/2570149768 wait complete. 2024-05-28T06:18:26.362 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-05-28T06:18:26.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.569+0000 7f6f8d400640 1 Processor -- start 2024-05-28T06:18:26.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.569+0000 7f6f8d400640 1 -- start start 2024-05-28T06:18:26.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.571+0000 7f6f8d400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f6f8815d5d0 0x7f6f880cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:26.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.571+0000 7f6f8d400640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f8815cb10 con 0x7f6f880b32c0 2024-05-28T06:18:26.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.571+0000 7f6f8d400640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f880ad5f0 con 0x7f6f880d0640 2024-05-28T06:18:26.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.571+0000 7f6f8d400640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f6f880adbb0 con 0x7f6f8815d5d0 2024-05-28T06:18:26.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.571+0000 7f6f87200640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f6f880b32c0 0x7f6f880b3700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:43832/0 (socket says 172.21.15.96:43832) 2024-05-28T06:18:26.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.571+0000 7f6f87200640 1 -- 172.21.15.96:0/2513770852 learned_addr learned my addr 172.21.15.96:0/2513770852 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:18:26.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.572+0000 7f6f7de00640 1 --2- 172.21.15.96:0/2513770852 >> v2:172.21.15.123:3300/0 conn(0x7f6f8815d5d0 0x7f6f880cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:26.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.572+0000 7f6f85e00640 1 -- 172.21.15.96:0/2513770852 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1411720368 0 0) 0x7f6f8815cb10 con 0x7f6f880b32c0 2024-05-28T06:18:26.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.572+0000 7f6f85e00640 1 -- 172.21.15.96:0/2513770852 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6f6c003520 con 0x7f6f880b32c0 2024-05-28T06:18:26.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.572+0000 7f6f85e00640 1 -- 172.21.15.96:0/2513770852 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1172159441 0 0) 0x7f6f6c003520 con 0x7f6f880b32c0 2024-05-28T06:18:26.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.572+0000 7f6f85e00640 1 -- 172.21.15.96:0/2513770852 >> v2:172.21.15.123:3300/0 conn(0x7f6f8815d5d0 msgr2=0x7f6f880cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:18:26.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.572+0000 7f6f85e00640 1 --2- 172.21.15.96:0/2513770852 >> v2:172.21.15.123:3300/0 conn(0x7f6f8815d5d0 0x7f6f880cf950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:26.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.572+0000 7f6f85e00640 1 -- 172.21.15.96:0/2513770852 >> v1:172.21.15.123:6789/0 conn(0x7f6f880d0640 legacy=0x7f6f880d0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:26.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.572+0000 7f6f85e00640 1 -- 172.21.15.96:0/2513770852 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f880ad8d0 con 0x7f6f880b32c0 2024-05-28T06:18:26.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.572+0000 7f6f85e00640 1 -- 172.21.15.96:0/2513770852 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f6f800019f0 con 0x7f6f880b32c0 2024-05-28T06:18:26.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.572+0000 7f6f85e00640 1 -- 172.21.15.96:0/2513770852 >> v1:172.21.15.96:6789/0 conn(0x7f6f880b32c0 legacy=0x7f6f880b3700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:26.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.572+0000 7f6f85e00640 1 --2- 172.21.15.96:0/2513770852 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f6c003a00 0x7f6f6c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:26.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.572+0000 7f6f85e00640 1 --2- 172.21.15.96:0/2513770852 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6f6c004800 0x7f6f6c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:26.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.572+0000 7f6f85e00640 1 --2- 172.21.15.96:0/2513770852 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6f6c005210 0x7f6f6c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:26.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.572+0000 7f6f85e00640 1 -- 172.21.15.96:0/2513770852 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f6f6c00c130 con 0x7f6f6c004800 2024-05-28T06:18:26.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.572+0000 7f6f85e00640 1 -- 172.21.15.96:0/2513770852 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f6f6c00c2e0 con 0x7f6f6c003a00 2024-05-28T06:18:26.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.572+0000 7f6f85e00640 1 -- 172.21.15.96:0/2513770852 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f6f6c00c4e0 con 0x7f6f6c005210 2024-05-28T06:18:26.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.573+0000 7f6f7de00640 1 --2- 172.21.15.96:0/2513770852 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6f6c004800 0x7f6f6c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:26.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.573+0000 7f6f7de00640 1 -- 172.21.15.96:0/2513770852 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6f6c005210 msgr2=0x7f6f6c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:18:26.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.573+0000 7f6f7de00640 1 --2- 172.21.15.96:0/2513770852 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6f6c005210 0x7f6f6c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:26.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.573+0000 7f6f7de00640 1 -- 172.21.15.96:0/2513770852 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f6c003a00 msgr2=0x7f6f6c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:18:26.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.573+0000 7f6f7de00640 1 --2- 172.21.15.96:0/2513770852 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f6c003a00 0x7f6f6c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:26.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.573+0000 7f6f7de00640 1 -- 172.21.15.96:0/2513770852 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f6c003520 con 0x7f6f6c004800 2024-05-28T06:18:26.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.573+0000 7f6f87200640 1 -- 172.21.15.96:0/2513770852 reap_dead start 2024-05-28T06:18:26.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.574+0000 7f6f7de00640 1 --2- 172.21.15.96:0/2513770852 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6f6c004800 0x7f6f6c004c80 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f6f70009a90 tx=0x7f6f6c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2371329aab9e57d0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:26.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.574+0000 7f6f85e00640 1 -- 172.21.15.96:0/2513770852 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f70036070 con 0x7f6f6c004800 2024-05-28T06:18:26.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.574+0000 7f6f85e00640 1 -- 172.21.15.96:0/2513770852 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f70031400 con 0x7f6f6c004800 2024-05-28T06:18:26.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.574+0000 7f6f85e00640 1 -- 172.21.15.96:0/2513770852 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f70009690 con 0x7f6f6c004800 2024-05-28T06:18:26.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.575+0000 7f6f8d400640 1 -- 172.21.15.96:0/2513770852 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6f6c004800 msgr2=0x7f6f6c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:26.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.575+0000 7f6f8d400640 1 --2- 172.21.15.96:0/2513770852 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6f6c004800 0x7f6f6c004c80 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f6f70009a90 tx=0x7f6f6c005830 comp rx=0 tx=0).stop 2024-05-28T06:18:26.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.575+0000 7f6f8d400640 1 -- 172.21.15.96:0/2513770852 shutdown_connections 2024-05-28T06:18:26.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.575+0000 7f6f8d400640 1 --2- 172.21.15.96:0/2513770852 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6f6c004800 0x7f6f6c004c80 unknown :-1 s=CLOSED pgs=1446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:26.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.575+0000 7f6f8d400640 1 -- 172.21.15.96:0/2513770852 >> 172.21.15.96:0/2513770852 conn(0x7f6f8806bd20 msgr2=0x7f6f8806c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:18:26.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.575+0000 7f6f8d400640 1 -- 172.21.15.96:0/2513770852 shutdown_connections 2024-05-28T06:18:26.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.575+0000 7f6f8d400640 1 -- 172.21.15.96:0/2513770852 wait complete. 2024-05-28T06:18:26.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.577+0000 7f6f8d400640 1 Processor -- start 2024-05-28T06:18:26.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.577+0000 7f6f8d400640 1 -- start start 2024-05-28T06:18:26.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.578+0000 7f6f8d400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6f6c004800 0x7f6f881f54d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:26.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.578+0000 7f6f8d400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6f88137c80 0x7f6f88138100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:26.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.578+0000 7f6f8d400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f88152e70 0x7f6f881532f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:26.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.578+0000 7f6f8d400640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f6f8815da50 con 0x7f6f88137c80 2024-05-28T06:18:26.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.578+0000 7f6f8d400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f6f8815d0b0 con 0x7f6f88152e70 2024-05-28T06:18:26.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.578+0000 7f6f8d400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f6f880ad2a0 con 0x7f6f6c004800 2024-05-28T06:18:26.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.578+0000 7f6f7de00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6f88137c80 0x7f6f88138100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:26.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.578+0000 7f6f7de00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6f88137c80 0x7f6f88138100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:40260/0 (socket says 172.21.15.96:40260) 2024-05-28T06:18:26.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.578+0000 7f6f7de00640 1 -- 172.21.15.96:0/1707943867 learned_addr learned my addr 172.21.15.96:0/1707943867 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:18:26.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.579+0000 7f6f87200640 1 --2- 172.21.15.96:0/1707943867 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f88152e70 0x7f6f881532f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:26.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.579+0000 7f6f86800640 1 --2- 172.21.15.96:0/1707943867 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6f6c004800 0x7f6f881f54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:26.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.579+0000 7f6f7de00640 1 -- 172.21.15.96:0/1707943867 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6f6c004800 msgr2=0x7f6f881f54d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:26.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.579+0000 7f6f7de00640 1 --2- 172.21.15.96:0/1707943867 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6f6c004800 0x7f6f881f54d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:26.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.579+0000 7f6f7de00640 1 -- 172.21.15.96:0/1707943867 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f88152e70 msgr2=0x7f6f881532f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:26.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.579+0000 7f6f7de00640 1 --2- 172.21.15.96:0/1707943867 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f88152e70 0x7f6f881532f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:26.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.579+0000 7f6f7de00640 1 -- 172.21.15.96:0/1707943867 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f881387f0 con 0x7f6f88137c80 2024-05-28T06:18:26.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.579+0000 7f6f86800640 1 --2- 172.21.15.96:0/1707943867 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6f6c004800 0x7f6f881f54d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:18:26.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.579+0000 7f6f7de00640 1 --2- 172.21.15.96:0/1707943867 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6f88137c80 0x7f6f88138100 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f6f70034040 tx=0x7f6f70004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:26.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.579+0000 7f6f7f400640 1 -- 172.21.15.96:0/1707943867 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f7003f030 con 0x7f6f88137c80 2024-05-28T06:18:26.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.580+0000 7f6f7f400640 1 -- 172.21.15.96:0/1707943867 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f70037040 con 0x7f6f88137c80 2024-05-28T06:18:26.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.580+0000 7f6f7f400640 1 -- 172.21.15.96:0/1707943867 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f70036070 con 0x7f6f88137c80 2024-05-28T06:18:26.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.580+0000 7f6f8d400640 1 -- 172.21.15.96:0/1707943867 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f88138ad0 con 0x7f6f88137c80 2024-05-28T06:18:26.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.580+0000 7f6f8d400640 1 -- 172.21.15.96:0/1707943867 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f88139090 con 0x7f6f88137c80 2024-05-28T06:18:26.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.581+0000 7f6f8d400640 1 -- 172.21.15.96:0/1707943867 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f880689e0 con 0x7f6f88137c80 2024-05-28T06:18:26.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.582+0000 7f6f7f400640 1 -- 172.21.15.96:0/1707943867 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 30) ==== 92621+0+0 (secure 0 0 0) 0x7f6f70042070 con 0x7f6f88137c80 2024-05-28T06:18:26.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.583+0000 7f6f7f400640 1 --2- 172.21.15.96:0/1707943867 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6f6406da60 0x7f6f6406ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:26.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.583+0000 7f6f7f400640 1 -- 172.21.15.96:0/1707943867 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 5737+0+0 (secure 0 0 0) 0x7f6f700aa0c0 con 0x7f6f88137c80 2024-05-28T06:18:26.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.584+0000 7f6f86800640 1 --2- 172.21.15.96:0/1707943867 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6f6406da60 0x7f6f6406ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:26.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.585+0000 7f6f86800640 1 --2- 172.21.15.96:0/1707943867 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6f6406da60 0x7f6f6406ff20 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f6f881567a0 tx=0x7f6f78009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:26.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.588+0000 7f6f7f400640 1 -- 172.21.15.96:0/1707943867 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6f700044e0 con 0x7f6f88137c80 2024-05-28T06:18:26.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.794+0000 7f6f8d400640 1 -- 172.21.15.96:0/1707943867 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f6f8815dce0 con 0x7f6f88137c80 2024-05-28T06:18:26.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.795+0000 7f6f7f400640 1 -- 172.21.15.96:0/1707943867 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v22)=0 v22) ==== 122+0+0 (secure 0 0 0) 0x7f6f700795a0 con 0x7f6f88137c80 2024-05-28T06:18:26.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.798+0000 7f6f8d400640 1 -- 172.21.15.96:0/1707943867 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6f6406da60 msgr2=0x7f6f6406ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:26.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.798+0000 7f6f8d400640 1 --2- 172.21.15.96:0/1707943867 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6f6406da60 0x7f6f6406ff20 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f6f881567a0 tx=0x7f6f78009040 comp rx=0 tx=0).stop 2024-05-28T06:18:26.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.798+0000 7f6f8d400640 1 -- 172.21.15.96:0/1707943867 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6f88137c80 msgr2=0x7f6f88138100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:26.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.798+0000 7f6f8d400640 1 --2- 172.21.15.96:0/1707943867 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6f88137c80 0x7f6f88138100 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f6f70034040 tx=0x7f6f70004730 comp rx=0 tx=0).stop 2024-05-28T06:18:26.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.798+0000 7f6f8d400640 1 -- 172.21.15.96:0/1707943867 shutdown_connections 2024-05-28T06:18:26.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.798+0000 7f6f8d400640 1 --2- 172.21.15.96:0/1707943867 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6f6406da60 0x7f6f6406ff20 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:26.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.798+0000 7f6f8d400640 1 --2- 172.21.15.96:0/1707943867 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f88152e70 0x7f6f881532f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:26.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.798+0000 7f6f8d400640 1 --2- 172.21.15.96:0/1707943867 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6f88137c80 0x7f6f88138100 unknown :-1 s=CLOSED pgs=1447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:26.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.798+0000 7f6f8d400640 1 --2- 172.21.15.96:0/1707943867 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6f6c004800 0x7f6f881f54d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:26.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.798+0000 7f6f8d400640 1 -- 172.21.15.96:0/1707943867 >> 172.21.15.96:0/1707943867 conn(0x7f6f8806bd20 msgr2=0x7f6f8815e7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:18:26.797 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.798+0000 7f6f8d400640 1 -- 172.21.15.96:0/1707943867 shutdown_connections 2024-05-28T06:18:26.797 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:26.798+0000 7f6f8d400640 1 -- 172.21.15.96:0/1707943867 wait complete. 2024-05-28T06:18:26.816 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv1 --mode=777 2024-05-28T06:18:27.022 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.024+0000 7f4fb9c00640 1 Processor -- start 2024-05-28T06:18:27.022 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.024+0000 7f4fb9c00640 1 -- start start 2024-05-28T06:18:27.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.025+0000 7f4fb9c00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f4fb40b4540 0x7f4fb40ccd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:27.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.025+0000 7f4fb9c00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4fb415cc10 con 0x7f4fb40b40b0 2024-05-28T06:18:27.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.025+0000 7f4fb9c00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4fb40b4b90 con 0x7f4fb415c830 2024-05-28T06:18:27.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.025+0000 7f4fb9c00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f4fb40b4970 con 0x7f4fb40b4540 2024-05-28T06:18:27.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.025+0000 7f4fb2000640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f4fb40b40b0 0x7f4fb40d08e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:43846/0 (socket says 172.21.15.96:43846) 2024-05-28T06:18:27.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.025+0000 7f4fb2000640 1 -- 172.21.15.96:0/3509442506 learned_addr learned my addr 172.21.15.96:0/3509442506 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:18:27.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.026+0000 7f4fb1600640 1 -- 172.21.15.96:0/3509442506 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1860042339 0 0) 0x7f4fb415cc10 con 0x7f4fb40b40b0 2024-05-28T06:18:27.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.026+0000 7f4fb1600640 1 -- 172.21.15.96:0/3509442506 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f98003520 con 0x7f4fb40b40b0 2024-05-28T06:18:27.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.026+0000 7f4fb3400640 1 --2- 172.21.15.96:0/3509442506 >> v2:172.21.15.123:3300/0 conn(0x7f4fb40b4540 0x7f4fb40ccd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:27.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.026+0000 7f4fb1600640 1 -- 172.21.15.96:0/3509442506 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4146603575 0 0) 0x7f4f98003520 con 0x7f4fb40b40b0 2024-05-28T06:18:27.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.026+0000 7f4fb1600640 1 -- 172.21.15.96:0/3509442506 >> v2:172.21.15.123:3300/0 conn(0x7f4fb40b4540 msgr2=0x7f4fb40ccd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:18:27.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.026+0000 7f4fb1600640 1 --2- 172.21.15.96:0/3509442506 >> v2:172.21.15.123:3300/0 conn(0x7f4fb40b4540 0x7f4fb40ccd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:27.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.026+0000 7f4fb1600640 1 -- 172.21.15.96:0/3509442506 >> v1:172.21.15.123:6789/0 conn(0x7f4fb415c830 legacy=0x7f4fb40ce230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:18:27.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.026+0000 7f4fb1600640 1 -- 172.21.15.96:0/3509442506 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fb40b4e70 con 0x7f4fb40b40b0 2024-05-28T06:18:27.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.026+0000 7f4fb1600640 1 -- 172.21.15.96:0/3509442506 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f4f9c001670 con 0x7f4fb40b40b0 2024-05-28T06:18:27.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.026+0000 7f4fb1600640 1 -- 172.21.15.96:0/3509442506 >> v1:172.21.15.96:6789/0 conn(0x7f4fb40b40b0 legacy=0x7f4fb40d08e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:27.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.026+0000 7f4fb1600640 1 --2- 172.21.15.96:0/3509442506 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f98003a00 0x7f4f98003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:27.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.026+0000 7f4fb1600640 1 --2- 172.21.15.96:0/3509442506 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4f98004800 0x7f4f98004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:27.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.027+0000 7f4fb1600640 1 --2- 172.21.15.96:0/3509442506 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4f98005210 0x7f4f9800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:27.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.027+0000 7f4fb1600640 1 -- 172.21.15.96:0/3509442506 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4fb40b4970 con 0x7f4f98003a00 2024-05-28T06:18:27.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.027+0000 7f4fb1600640 1 -- 172.21.15.96:0/3509442506 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f4f9800c140 con 0x7f4f98004800 2024-05-28T06:18:27.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.027+0000 7f4fb1600640 1 -- 172.21.15.96:0/3509442506 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f4f9800c340 con 0x7f4f98005210 2024-05-28T06:18:27.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.027+0000 7f4fb2a00640 1 --2- 172.21.15.96:0/3509442506 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f98003a00 0x7f4f98003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:27.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.027+0000 7f4fb2000640 1 --2- 172.21.15.96:0/3509442506 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4f98004800 0x7f4f98004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:27.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.027+0000 7f4fb2a00640 1 -- 172.21.15.96:0/3509442506 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4f98005210 msgr2=0x7f4f9800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:18:27.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.027+0000 7f4fb2a00640 1 --2- 172.21.15.96:0/3509442506 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4f98005210 0x7f4f9800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:27.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.027+0000 7f4fb2a00640 1 -- 172.21.15.96:0/3509442506 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4f98004800 msgr2=0x7f4f98004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:18:27.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.027+0000 7f4fb2a00640 1 --2- 172.21.15.96:0/3509442506 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4f98004800 0x7f4f98004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:27.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.027+0000 7f4fb2a00640 1 -- 172.21.15.96:0/3509442506 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f98003520 con 0x7f4f98003a00 2024-05-28T06:18:27.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.027+0000 7f4fb3400640 1 -- 172.21.15.96:0/3509442506 reap_dead start 2024-05-28T06:18:27.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.028+0000 7f4fb2a00640 1 --2- 172.21.15.96:0/3509442506 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f98003a00 0x7f4f98003e00 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f4fa4004480 tx=0x7f4f98005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15409561b11cabc8 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:27.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.028+0000 7f4fb1600640 1 -- 172.21.15.96:0/3509442506 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fa4037070 con 0x7f4f98003a00 2024-05-28T06:18:27.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.029+0000 7f4fb1600640 1 -- 172.21.15.96:0/3509442506 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fa402ad30 con 0x7f4f98003a00 2024-05-28T06:18:27.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.029+0000 7f4fb1600640 1 -- 172.21.15.96:0/3509442506 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fa4004080 con 0x7f4f98003a00 2024-05-28T06:18:27.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.029+0000 7f4fb9c00640 1 -- 172.21.15.96:0/3509442506 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f98003a00 msgr2=0x7f4f98003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:27.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.029+0000 7f4fb9c00640 1 --2- 172.21.15.96:0/3509442506 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f98003a00 0x7f4f98003e00 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f4fa4004480 tx=0x7f4f98005830 comp rx=0 tx=0).stop 2024-05-28T06:18:27.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.029+0000 7f4fb9c00640 1 -- 172.21.15.96:0/3509442506 shutdown_connections 2024-05-28T06:18:27.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.029+0000 7f4fb9c00640 1 --2- 172.21.15.96:0/3509442506 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f98003a00 0x7f4f98003e00 unknown :-1 s=CLOSED pgs=1449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:27.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.029+0000 7f4fb9c00640 1 -- 172.21.15.96:0/3509442506 >> 172.21.15.96:0/3509442506 conn(0x7f4fb40b20a0 msgr2=0x7f4fb40b24b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:18:27.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.029+0000 7f4fb9c00640 1 -- 172.21.15.96:0/3509442506 shutdown_connections 2024-05-28T06:18:27.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.029+0000 7f4fb9c00640 1 -- 172.21.15.96:0/3509442506 wait complete. 2024-05-28T06:18:27.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.031+0000 7f4fb9c00640 1 Processor -- start 2024-05-28T06:18:27.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.031+0000 7f4fb9c00640 1 -- start start 2024-05-28T06:18:27.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.032+0000 7f4fb9c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4f98003a00 0x7f4fb4071de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:27.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.032+0000 7f4fb9c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4fb4072320 0x7f4fb415ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:27.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.032+0000 7f4fb9c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4fb415f100 0x7f4fb415f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:27.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.032+0000 7f4fb9c00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4fb40cffe0 con 0x7f4fb415f100 2024-05-28T06:18:27.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.032+0000 7f4fb9c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f4fb40d03c0 con 0x7f4f98003a00 2024-05-28T06:18:27.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.032+0000 7f4fb9c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f4fb40d0160 con 0x7f4fb4072320 2024-05-28T06:18:27.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.032+0000 7f4fb3400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4fb415f100 0x7f4fb415f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:27.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.033+0000 7f4fb3400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4fb415f100 0x7f4fb415f580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:40270/0 (socket says 172.21.15.96:40270) 2024-05-28T06:18:27.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.033+0000 7f4fb3400640 1 -- 172.21.15.96:0/819028100 learned_addr learned my addr 172.21.15.96:0/819028100 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:18:27.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.033+0000 7f4fb2a00640 1 --2- 172.21.15.96:0/819028100 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4f98003a00 0x7f4fb4071de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:27.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.033+0000 7f4fb3400640 1 -- 172.21.15.96:0/819028100 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4fb4072320 msgr2=0x7f4fb415ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:27.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.033+0000 7f4fb2000640 1 --2- 172.21.15.96:0/819028100 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4fb4072320 0x7f4fb415ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:27.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.033+0000 7f4fb3400640 1 --2- 172.21.15.96:0/819028100 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4fb4072320 0x7f4fb415ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:27.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.033+0000 7f4fb3400640 1 -- 172.21.15.96:0/819028100 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4f98003a00 msgr2=0x7f4fb4071de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:27.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.033+0000 7f4fb3400640 1 --2- 172.21.15.96:0/819028100 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4f98003a00 0x7f4fb4071de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:27.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.033+0000 7f4fb3400640 1 -- 172.21.15.96:0/819028100 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fb415fac0 con 0x7f4fb415f100 2024-05-28T06:18:27.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.033+0000 7f4fb2a00640 1 --2- 172.21.15.96:0/819028100 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4f98003a00 0x7f4fb4071de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:18:27.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.033+0000 7f4fb2000640 1 --2- 172.21.15.96:0/819028100 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4fb4072320 0x7f4fb415ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:18:27.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.033+0000 7f4fb3400640 1 --2- 172.21.15.96:0/819028100 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4fb415f100 0x7f4fb415f580 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f4fb406d750 tx=0x7f4fac00d8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:27.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.033+0000 7f4faaa00640 1 -- 172.21.15.96:0/819028100 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fac004020 con 0x7f4fb415f100 2024-05-28T06:18:27.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.034+0000 7f4faaa00640 1 -- 172.21.15.96:0/819028100 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fac0041c0 con 0x7f4fb415f100 2024-05-28T06:18:27.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.034+0000 7f4faaa00640 1 -- 172.21.15.96:0/819028100 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fac0043f0 con 0x7f4fb415f100 2024-05-28T06:18:27.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.034+0000 7f4fb9c00640 1 -- 172.21.15.96:0/819028100 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fb4072b50 con 0x7f4fb415f100 2024-05-28T06:18:27.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.034+0000 7f4fb9c00640 1 -- 172.21.15.96:0/819028100 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fb4073110 con 0x7f4fb415f100 2024-05-28T06:18:27.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.035+0000 7f4fb9c00640 1 -- 172.21.15.96:0/819028100 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4fb40757d0 con 0x7f4fb415f100 2024-05-28T06:18:27.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.036+0000 7f4faaa00640 1 -- 172.21.15.96:0/819028100 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 30) ==== 92621+0+0 (secure 0 0 0) 0x7f4fac01a020 con 0x7f4fb415f100 2024-05-28T06:18:27.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.037+0000 7f4faaa00640 1 --2- 172.21.15.96:0/819028100 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4f9006da90 0x7f4f9006ff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:27.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.037+0000 7f4faaa00640 1 -- 172.21.15.96:0/819028100 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 5737+0+0 (secure 0 0 0) 0x7f4fac08a1d0 con 0x7f4fb415f100 2024-05-28T06:18:27.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.038+0000 7f4fb2a00640 1 --2- 172.21.15.96:0/819028100 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4f9006da90 0x7f4f9006ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:27.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.039+0000 7f4fb2a00640 1 --2- 172.21.15.96:0/819028100 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4f9006da90 0x7f4f9006ff50 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f4fa4034000 tx=0x7f4fa402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:27.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.042+0000 7f4faaa00640 1 -- 172.21.15.96:0/819028100 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4fac059d00 con 0x7f4fb415f100 2024-05-28T06:18:27.266 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.268+0000 7f4fb9c00640 1 -- 172.21.15.96:0/819028100 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv1", "mode": "777", "target": ["mon-mgr", ""]}) -- 0x7f4fb406e260 con 0x7f4f9006da90 2024-05-28T06:18:27.284 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:18:27.286+0000 7fea59e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:18:27.284 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:18:27.286+0000 7fea59e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:18:27.284 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:18:27.286+0000 7fea59e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:18:27.284 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:18:27.286+0000 7fea59e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:18:27.284 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:18:27.286+0000 7fea59e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:18:27.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.367+0000 7f4faaa00640 1 -- 172.21.15.96:0/819028100 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f4fb406e260 con 0x7f4f9006da90 2024-05-28T06:18:27.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.371+0000 7f4fb9c00640 1 -- 172.21.15.96:0/819028100 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4f9006da90 msgr2=0x7f4f9006ff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:27.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.371+0000 7f4fb9c00640 1 --2- 172.21.15.96:0/819028100 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4f9006da90 0x7f4f9006ff50 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f4fa4034000 tx=0x7f4fa402e000 comp rx=0 tx=0).stop 2024-05-28T06:18:27.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.371+0000 7f4fb9c00640 1 -- 172.21.15.96:0/819028100 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4fb415f100 msgr2=0x7f4fb415f580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:27.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.371+0000 7f4fb9c00640 1 --2- 172.21.15.96:0/819028100 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4fb415f100 0x7f4fb415f580 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f4fb406d750 tx=0x7f4fac00d8a0 comp rx=0 tx=0).stop 2024-05-28T06:18:27.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.371+0000 7f4fb9c00640 1 -- 172.21.15.96:0/819028100 shutdown_connections 2024-05-28T06:18:27.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.371+0000 7f4fb9c00640 1 --2- 172.21.15.96:0/819028100 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4f9006da90 0x7f4f9006ff50 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:27.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.371+0000 7f4fb9c00640 1 --2- 172.21.15.96:0/819028100 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4fb415f100 0x7f4fb415f580 unknown :-1 s=CLOSED pgs=1450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:27.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.371+0000 7f4fb9c00640 1 --2- 172.21.15.96:0/819028100 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4fb4072320 0x7f4fb415ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:27.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.371+0000 7f4fb9c00640 1 --2- 172.21.15.96:0/819028100 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4f98003a00 0x7f4fb4071de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:27.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.371+0000 7f4fb9c00640 1 -- 172.21.15.96:0/819028100 >> 172.21.15.96:0/819028100 conn(0x7f4fb40b20a0 msgr2=0x7f4fb40cec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:18:27.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.371+0000 7f4fb9c00640 1 -- 172.21.15.96:0/819028100 shutdown_connections 2024-05-28T06:18:27.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.371+0000 7f4fb9c00640 1 -- 172.21.15.96:0/819028100 wait complete. 2024-05-28T06:18:27.391 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-05-28T06:18:27.596 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.598+0000 7f0137200640 1 Processor -- start 2024-05-28T06:18:27.596 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.598+0000 7f0137200640 1 -- start start 2024-05-28T06:18:27.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.600+0000 7f0137200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f01380cd2a0 0x7f01380b4d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:27.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.600+0000 7f0137200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f013815cb60 0x7f013815cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:27.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.600+0000 7f0137200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f013815d900 con 0x7f0138153510 2024-05-28T06:18:27.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.600+0000 7f0137200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f01381538f0 con 0x7f01380cd2a0 2024-05-28T06:18:27.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.600+0000 7f0137200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f01380b48c0 con 0x7f013815cb60 2024-05-28T06:18:27.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.600+0000 7f0135e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f0138153510 0x7f01380ccb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:43856/0 (socket says 172.21.15.96:43856) 2024-05-28T06:18:27.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.600+0000 7f0135e00640 1 -- 172.21.15.96:0/3512071961 learned_addr learned my addr 172.21.15.96:0/3512071961 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:18:27.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.600+0000 7f012fe00640 1 --2- 172.21.15.96:0/3512071961 >> v2:172.21.15.123:3300/0 conn(0x7f01380cd2a0 0x7f01380b4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:27.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.600+0000 7f0136800640 1 --2- 172.21.15.96:0/3512071961 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f013815cb60 0x7f013815cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:27.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.601+0000 7f0135400640 1 -- 172.21.15.96:0/3512071961 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 974911192 0 0) 0x7f013815d900 con 0x7f0138153510 2024-05-28T06:18:27.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.601+0000 7f0135400640 1 -- 172.21.15.96:0/3512071961 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f011c003520 con 0x7f0138153510 2024-05-28T06:18:27.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.601+0000 7f012fe00640 1 -- 172.21.15.96:0/3512071961 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f013815cb60 msgr2=0x7f013815cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:18:27.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.601+0000 7f012fe00640 1 --2- 172.21.15.96:0/3512071961 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f013815cb60 0x7f013815cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:27.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.601+0000 7f012fe00640 1 -- 172.21.15.96:0/3512071961 >> v1:172.21.15.96:6789/0 conn(0x7f0138153510 legacy=0x7f01380ccb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:27.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.601+0000 7f0135400640 1 -- 172.21.15.96:0/3512071961 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 656963536 0 0) 0x7f011c003520 con 0x7f0138153510 2024-05-28T06:18:27.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.601+0000 7f012fe00640 1 -- 172.21.15.96:0/3512071961 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01380b3f90 con 0x7f01380cd2a0 2024-05-28T06:18:27.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.601+0000 7f0136800640 1 --2- 172.21.15.96:0/3512071961 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f013815cb60 0x7f013815cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-28T06:18:27.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.603+0000 7f012fe00640 1 --2- 172.21.15.96:0/3512071961 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01380cd2a0 0x7f01380b4d50 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f01200097a0 tx=0x7f012002a080 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9deb0238e8143d01 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:27.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.603+0000 7f0135400640 1 -- 172.21.15.96:0/3512071961 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0120037070 con 0x7f01380cd2a0 2024-05-28T06:18:27.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.603+0000 7f0135400640 1 -- 172.21.15.96:0/3512071961 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f012002ab10 con 0x7f01380cd2a0 2024-05-28T06:18:27.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.603+0000 7f0135400640 1 -- 172.21.15.96:0/3512071961 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0128001670 con 0x7f01380cd2a0 2024-05-28T06:18:27.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.604+0000 7f0137200640 1 -- 172.21.15.96:0/3512071961 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01380cd2a0 msgr2=0x7f01380b4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:27.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.604+0000 7f0137200640 1 --2- 172.21.15.96:0/3512071961 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01380cd2a0 0x7f01380b4d50 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f01200097a0 tx=0x7f012002a080 comp rx=0 tx=0).stop 2024-05-28T06:18:27.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.604+0000 7f0137200640 1 -- 172.21.15.96:0/3512071961 shutdown_connections 2024-05-28T06:18:27.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.604+0000 7f0137200640 1 --2- 172.21.15.96:0/3512071961 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f013815cb60 0x7f013815cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:27.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.604+0000 7f0137200640 1 --2- 172.21.15.96:0/3512071961 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01380cd2a0 0x7f01380b4d50 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:27.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.604+0000 7f0137200640 1 -- 172.21.15.96:0/3512071961 >> 172.21.15.96:0/3512071961 conn(0x7f013806b900 msgr2=0x7f013806bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:18:27.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.604+0000 7f0137200640 1 -- 172.21.15.96:0/3512071961 shutdown_connections 2024-05-28T06:18:27.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.604+0000 7f0137200640 1 -- 172.21.15.96:0/3512071961 wait complete. 2024-05-28T06:18:27.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.606+0000 7f0137200640 1 Processor -- start 2024-05-28T06:18:27.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.607+0000 7f0137200640 1 -- start start 2024-05-28T06:18:27.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.608+0000 7f0137200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f01380cd2a0 0x7f013815edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:27.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.608+0000 7f0137200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0138153510 0x7f0138154d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:27.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.608+0000 7f0137200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f013815cb60 0x7f01380aea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:27.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.608+0000 7f0137200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f013815d4f0 con 0x7f01380cd2a0 2024-05-28T06:18:27.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.608+0000 7f0137200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f01380b4550 con 0x7f013815cb60 2024-05-28T06:18:27.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.608+0000 7f0137200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f013815d670 con 0x7f0138153510 2024-05-28T06:18:27.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.608+0000 7f0135e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f01380cd2a0 0x7f013815edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:27.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.608+0000 7f0135e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f01380cd2a0 0x7f013815edb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:40278/0 (socket says 172.21.15.96:40278) 2024-05-28T06:18:27.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.608+0000 7f0135e00640 1 -- 172.21.15.96:0/1328658006 learned_addr learned my addr 172.21.15.96:0/1328658006 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:18:27.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.608+0000 7f0135e00640 1 -- 172.21.15.96:0/1328658006 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0138153510 msgr2=0x7f0138154d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:27.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.608+0000 7f0135e00640 1 --2- 172.21.15.96:0/1328658006 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0138153510 0x7f0138154d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:27.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.608+0000 7f0135e00640 1 -- 172.21.15.96:0/1328658006 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f013815cb60 msgr2=0x7f01380aea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:27.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.608+0000 7f0135e00640 1 --2- 172.21.15.96:0/1328658006 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f013815cb60 0x7f01380aea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:27.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.608+0000 7f0135e00640 1 -- 172.21.15.96:0/1328658006 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0138153cd0 con 0x7f01380cd2a0 2024-05-28T06:18:27.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.609+0000 7f0135e00640 1 --2- 172.21.15.96:0/1328658006 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f01380cd2a0 0x7f013815edb0 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7f012800c7b0 tx=0x7f012800cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:27.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.609+0000 7f012e000640 1 -- 172.21.15.96:0/1328658006 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0128017070 con 0x7f01380cd2a0 2024-05-28T06:18:27.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.609+0000 7f012e000640 1 -- 172.21.15.96:0/1328658006 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0128005ad0 con 0x7f01380cd2a0 2024-05-28T06:18:27.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.609+0000 7f012e000640 1 -- 172.21.15.96:0/1328658006 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0128007070 con 0x7f01380cd2a0 2024-05-28T06:18:27.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.609+0000 7f0137200640 1 -- 172.21.15.96:0/1328658006 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0138153fb0 con 0x7f01380cd2a0 2024-05-28T06:18:27.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.609+0000 7f0137200640 1 -- 172.21.15.96:0/1328658006 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0138154570 con 0x7f01380cd2a0 2024-05-28T06:18:27.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.611+0000 7f0137200640 1 -- 172.21.15.96:0/1328658006 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01380cd9b0 con 0x7f01380cd2a0 2024-05-28T06:18:27.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.612+0000 7f012e000640 1 -- 172.21.15.96:0/1328658006 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 30) ==== 92621+0+0 (secure 0 0 0) 0x7f012801a070 con 0x7f01380cd2a0 2024-05-28T06:18:27.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.612+0000 7f012e000640 1 --2- 172.21.15.96:0/1328658006 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f011406da60 0x7f011406ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:18:27.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.613+0000 7f012e000640 1 -- 172.21.15.96:0/1328658006 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 5737+0+0 (secure 0 0 0) 0x7f0128089f10 con 0x7f01380cd2a0 2024-05-28T06:18:27.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.613+0000 7f012fe00640 1 --2- 172.21.15.96:0/1328658006 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f011406da60 0x7f011406ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:18:27.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.614+0000 7f012fe00640 1 --2- 172.21.15.96:0/1328658006 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f011406da60 0x7f011406ff20 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f0120034000 tx=0x7f012002c000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:18:27.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.617+0000 7f012e000640 1 -- 172.21.15.96:0/1328658006 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f01380cd9b0 con 0x7f01380cd2a0 2024-05-28T06:18:27.849 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.851+0000 7f0137200640 1 -- 172.21.15.96:0/1328658006 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) -- 0x7f01380b3c40 con 0x7f011406da60 2024-05-28T06:18:27.860 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.862+0000 7f012e000640 1 -- 172.21.15.96:0/1328658006 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+59 (secure 0 0 0) 0x7f01380b3c40 con 0x7f011406da60 2024-05-28T06:18:27.861 INFO:teuthology.orchestra.run.smithi096.stdout:/volumes/_nogroup/sv1/babc868a-5738-42d7-be15-95a0a95d1980 2024-05-28T06:18:27.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.865+0000 7f0137200640 1 -- 172.21.15.96:0/1328658006 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f011406da60 msgr2=0x7f011406ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:27.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.865+0000 7f0137200640 1 --2- 172.21.15.96:0/1328658006 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f011406da60 0x7f011406ff20 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f0120034000 tx=0x7f012002c000 comp rx=0 tx=0).stop 2024-05-28T06:18:27.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.865+0000 7f0137200640 1 -- 172.21.15.96:0/1328658006 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f01380cd2a0 msgr2=0x7f013815edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:18:27.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.865+0000 7f0137200640 1 --2- 172.21.15.96:0/1328658006 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f01380cd2a0 0x7f013815edb0 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7f012800c7b0 tx=0x7f012800cc70 comp rx=0 tx=0).stop 2024-05-28T06:18:27.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.865+0000 7f0137200640 1 -- 172.21.15.96:0/1328658006 shutdown_connections 2024-05-28T06:18:27.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.865+0000 7f0137200640 1 --2- 172.21.15.96:0/1328658006 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f011406da60 0x7f011406ff20 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:27.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.865+0000 7f0137200640 1 --2- 172.21.15.96:0/1328658006 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f013815cb60 0x7f01380aea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:27.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.865+0000 7f0137200640 1 --2- 172.21.15.96:0/1328658006 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0138153510 0x7f0138154d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:27.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.865+0000 7f0137200640 1 --2- 172.21.15.96:0/1328658006 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f01380cd2a0 0x7f013815edb0 unknown :-1 s=CLOSED pgs=1452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:18:27.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.865+0000 7f0137200640 1 -- 172.21.15.96:0/1328658006 >> 172.21.15.96:0/1328658006 conn(0x7f013806b900 msgr2=0x7f013806bce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:18:27.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.865+0000 7f0137200640 1 -- 172.21.15.96:0/1328658006 shutdown_connections 2024-05-28T06:18:27.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:18:27.865+0000 7f0137200640 1 -- 172.21.15.96:0/1328658006 wait complete. 2024-05-28T06:18:27.884 DEBUG:tasks.cephfs.test_volumes:filling subvolume sv1 with 3 files each 1024MB size under directory volumes/_nogroup/sv1/babc868a-5738-42d7-be15-95a0a95d1980 2024-05-28T06:18:27.884 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:18:27.884 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:18:27.931 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:18:27.931 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/babc868a-5738-42d7-be15-95a0a95d1980/subvolume_file.0 bs=1M conv=fdatasync count=1024 seek=0) 2024-05-28T06:18:33.189 DEBUG:teuthology.orchestra.run.smithi096:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:18:33.197 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:18:42.147 INFO:teuthology.orchestra.run.smithi096.stderr:1024+0 records in 2024-05-28T06:18:42.148 INFO:teuthology.orchestra.run.smithi096.stderr:1024+0 records out 2024-05-28T06:18:42.148 INFO:teuthology.orchestra.run.smithi096.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 14.093 s, 76.2 MB/s 2024-05-28T06:18:42.149 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:18:42.150 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:18:42.201 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:18:42.201 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/babc868a-5738-42d7-be15-95a0a95d1980/subvolume_file.1 bs=1M conv=fdatasync count=1024 seek=0) 2024-05-28T06:18:54.822 INFO:teuthology.orchestra.run.smithi096.stderr:1024+0 records in 2024-05-28T06:18:54.822 INFO:teuthology.orchestra.run.smithi096.stderr:1024+0 records out 2024-05-28T06:18:54.822 INFO:teuthology.orchestra.run.smithi096.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 12.5625 s, 85.5 MB/s 2024-05-28T06:18:54.824 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:18:54.824 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:18:54.872 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:18:54.873 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/babc868a-5738-42d7-be15-95a0a95d1980/subvolume_file.2 bs=1M conv=fdatasync count=1024 seek=0) 2024-05-28T06:19:03.352 DEBUG:teuthology.orchestra.run.smithi096:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:19:03.360 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:19:05.994 INFO:teuthology.orchestra.run.smithi096.stderr:1024+0 records in 2024-05-28T06:19:05.994 INFO:teuthology.orchestra.run.smithi096.stderr:1024+0 records out 2024-05-28T06:19:05.994 INFO:teuthology.orchestra.run.smithi096.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 11.0203 s, 97.4 MB/s 2024-05-28T06:19:05.996 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot create cephfs sv1 ss1 2024-05-28T06:19:06.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.209+0000 7f92e7200640 1 Processor -- start 2024-05-28T06:19:06.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.210+0000 7f92e7200640 1 -- start start 2024-05-28T06:19:06.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.211+0000 7f92e7200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f92e815fa00 0x7f92e806e970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:06.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.211+0000 7f92e7200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f92e81600f0 0x7f92e80d0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:06.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.211+0000 7f92e7200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92e80d0d90 con 0x7f92e80d0970 2024-05-28T06:19:06.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.211+0000 7f92e7200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f92e806fa00 con 0x7f92e815fa00 2024-05-28T06:19:06.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.211+0000 7f92e7200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f92e806eeb0 con 0x7f92e81600f0 2024-05-28T06:19:06.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.212+0000 7f92e5e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f92e815fa00 0x7f92e806e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:06.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.212+0000 7f92e5e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f92e815fa00 0x7f92e806e970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:46566/0 (socket says 172.21.15.96:46566) 2024-05-28T06:19:06.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.212+0000 7f92e5e00640 1 -- 172.21.15.96:0/747718936 learned_addr learned my addr 172.21.15.96:0/747718936 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:19:06.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.212+0000 7f92dd400640 1 --2- 172.21.15.96:0/747718936 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f92e81600f0 0x7f92e80d0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:06.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.212+0000 7f92e5e00640 1 -- 172.21.15.96:0/747718936 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f92e81600f0 msgr2=0x7f92e80d0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:19:06.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.212+0000 7f92e5e00640 1 --2- 172.21.15.96:0/747718936 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f92e81600f0 0x7f92e80d0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:06.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.212+0000 7f92e5e00640 1 -- 172.21.15.96:0/747718936 >> v1:172.21.15.123:6789/0 conn(0x7f92e80d0970 legacy=0x7f92e80b3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:19:06.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.212+0000 7f92e5e00640 1 -- 172.21.15.96:0/747718936 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92e806f160 con 0x7f92e815fa00 2024-05-28T06:19:06.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.213+0000 7f92e5e00640 1 --2- 172.21.15.96:0/747718936 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f92e815fa00 0x7f92e806e970 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7f92d8009870 tx=0x7f92d802a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af3db00438e9ae19 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:06.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.214+0000 7f92e5400640 1 -- 172.21.15.96:0/747718936 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92d8037070 con 0x7f92e815fa00 2024-05-28T06:19:06.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.214+0000 7f92e5400640 1 -- 172.21.15.96:0/747718936 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92d8032440 con 0x7f92e815fa00 2024-05-28T06:19:06.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.214+0000 7f92e5400640 1 -- 172.21.15.96:0/747718936 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92d8032810 con 0x7f92e815fa00 2024-05-28T06:19:06.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.214+0000 7f92e7200640 1 -- 172.21.15.96:0/747718936 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f92e815fa00 msgr2=0x7f92e806e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:06.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.214+0000 7f92e7200640 1 --2- 172.21.15.96:0/747718936 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f92e815fa00 0x7f92e806e970 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7f92d8009870 tx=0x7f92d802a180 comp rx=0 tx=0).stop 2024-05-28T06:19:06.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.214+0000 7f92e7200640 1 -- 172.21.15.96:0/747718936 shutdown_connections 2024-05-28T06:19:06.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.214+0000 7f92e7200640 1 --2- 172.21.15.96:0/747718936 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f92e81600f0 0x7f92e80d0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:06.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.214+0000 7f92e7200640 1 --2- 172.21.15.96:0/747718936 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f92e815fa00 0x7f92e806e970 unknown :-1 s=CLOSED pgs=1453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:06.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.214+0000 7f92e7200640 1 -- 172.21.15.96:0/747718936 >> 172.21.15.96:0/747718936 conn(0x7f92e806bae0 msgr2=0x7f92e806bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:19:06.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.214+0000 7f92e7200640 1 -- 172.21.15.96:0/747718936 shutdown_connections 2024-05-28T06:19:06.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.214+0000 7f92e7200640 1 -- 172.21.15.96:0/747718936 wait complete. 2024-05-28T06:19:06.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.216+0000 7f92e7200640 1 Processor -- start 2024-05-28T06:19:06.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.216+0000 7f92e7200640 1 -- start start 2024-05-28T06:19:06.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.217+0000 7f92e7200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f92e80d0970 0x7f92e814f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:06.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.217+0000 7f92e7200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f92e815fa00 0x7f92e809dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:06.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.217+0000 7f92e7200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92e81600f0 0x7f92e809b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:06.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.217+0000 7f92e7200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f92e8071c70 con 0x7f92e80d0970 2024-05-28T06:19:06.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.217+0000 7f92e7200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f92e806f720 con 0x7f92e81600f0 2024-05-28T06:19:06.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.217+0000 7f92e7200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f92e815fde0 con 0x7f92e815fa00 2024-05-28T06:19:06.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.217+0000 7f92e5e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f92e80d0970 0x7f92e814f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:06.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.218+0000 7f92e5e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f92e80d0970 0x7f92e814f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:46580/0 (socket says 172.21.15.96:46580) 2024-05-28T06:19:06.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.218+0000 7f92e5e00640 1 -- 172.21.15.96:0/1556363535 learned_addr learned my addr 172.21.15.96:0/1556363535 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:19:06.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.218+0000 7f92dd400640 1 --2- 172.21.15.96:0/1556363535 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f92e815fa00 0x7f92e809dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:06.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.218+0000 7f92e6800640 1 --2- 172.21.15.96:0/1556363535 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92e81600f0 0x7f92e809b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:06.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.218+0000 7f92e5e00640 1 -- 172.21.15.96:0/1556363535 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f92e815fa00 msgr2=0x7f92e809dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:06.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.218+0000 7f92e5e00640 1 --2- 172.21.15.96:0/1556363535 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f92e815fa00 0x7f92e809dc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:06.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.218+0000 7f92e5e00640 1 -- 172.21.15.96:0/1556363535 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92e81600f0 msgr2=0x7f92e809b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:06.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.218+0000 7f92e5e00640 1 --2- 172.21.15.96:0/1556363535 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92e81600f0 0x7f92e809b9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:06.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.218+0000 7f92e5e00640 1 -- 172.21.15.96:0/1556363535 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92e809b320 con 0x7f92e80d0970 2024-05-28T06:19:06.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.218+0000 7f92e5e00640 1 --2- 172.21.15.96:0/1556363535 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f92e80d0970 0x7f92e814f750 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7f92d8009870 tx=0x7f92d80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:06.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.219+0000 7f92dea00640 1 -- 172.21.15.96:0/1556363535 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92d8041020 con 0x7f92e80d0970 2024-05-28T06:19:06.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.219+0000 7f92dea00640 1 -- 172.21.15.96:0/1556363535 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92d80306c0 con 0x7f92e80d0970 2024-05-28T06:19:06.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.219+0000 7f92dea00640 1 -- 172.21.15.96:0/1556363535 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92d8037040 con 0x7f92e80d0970 2024-05-28T06:19:06.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.219+0000 7f92e7200640 1 -- 172.21.15.96:0/1556363535 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92e809ad60 con 0x7f92e80d0970 2024-05-28T06:19:06.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.219+0000 7f92e7200640 1 -- 172.21.15.96:0/1556363535 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92e815ba60 con 0x7f92e80d0970 2024-05-28T06:19:06.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.220+0000 7f92e7200640 1 -- 172.21.15.96:0/1556363535 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92e80689e0 con 0x7f92e80d0970 2024-05-28T06:19:06.218 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.221+0000 7f92dea00640 1 -- 172.21.15.96:0/1556363535 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 31) ==== 92672+0+0 (secure 0 0 0) 0x7f92d8044020 con 0x7f92e80d0970 2024-05-28T06:19:06.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.223+0000 7f92dea00640 1 --2- 172.21.15.96:0/1556363535 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f92c406daf0 0x7f92c406ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:06.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.223+0000 7f92dea00640 1 -- 172.21.15.96:0/1556363535 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 5766+0+0 (secure 0 0 0) 0x7f92d80aaf60 con 0x7f92e80d0970 2024-05-28T06:19:06.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.223+0000 7f92dd400640 1 --2- 172.21.15.96:0/1556363535 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f92c406daf0 0x7f92c406ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:06.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.225+0000 7f92dd400640 1 --2- 172.21.15.96:0/1556363535 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f92c406daf0 0x7f92c406ffb0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f92d0006fd0 tx=0x7f92d0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:06.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.228+0000 7f92dea00640 1 -- 172.21.15.96:0/1556363535 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f92d804b030 con 0x7f92e80d0970 2024-05-28T06:19:06.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.459+0000 7f92e7200640 1 -- 172.21.15.96:0/1556363535 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""]}) -- 0x7f92e815e660 con 0x7f92c406daf0 2024-05-28T06:19:06.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.497+0000 7f92dea00640 1 -- 172.21.15.96:0/1556363535 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f92e815e660 con 0x7f92c406daf0 2024-05-28T06:19:06.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.499+0000 7f92e7200640 1 -- 172.21.15.96:0/1556363535 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f92c406daf0 msgr2=0x7f92c406ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:06.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.499+0000 7f92e7200640 1 --2- 172.21.15.96:0/1556363535 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f92c406daf0 0x7f92c406ffb0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f92d0006fd0 tx=0x7f92d0008040 comp rx=0 tx=0).stop 2024-05-28T06:19:06.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.499+0000 7f92e7200640 1 -- 172.21.15.96:0/1556363535 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f92e80d0970 msgr2=0x7f92e814f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:06.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.499+0000 7f92e7200640 1 --2- 172.21.15.96:0/1556363535 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f92e80d0970 0x7f92e814f750 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7f92d8009870 tx=0x7f92d80046c0 comp rx=0 tx=0).stop 2024-05-28T06:19:06.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.500+0000 7f92e7200640 1 -- 172.21.15.96:0/1556363535 shutdown_connections 2024-05-28T06:19:06.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.500+0000 7f92e7200640 1 --2- 172.21.15.96:0/1556363535 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f92c406daf0 0x7f92c406ffb0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:06.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.500+0000 7f92e7200640 1 --2- 172.21.15.96:0/1556363535 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92e81600f0 0x7f92e809b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:06.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.500+0000 7f92e7200640 1 --2- 172.21.15.96:0/1556363535 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f92e815fa00 0x7f92e809dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:06.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.500+0000 7f92e7200640 1 --2- 172.21.15.96:0/1556363535 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f92e80d0970 0x7f92e814f750 unknown :-1 s=CLOSED pgs=1454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:06.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.500+0000 7f92e7200640 1 -- 172.21.15.96:0/1556363535 >> 172.21.15.96:0/1556363535 conn(0x7f92e806bae0 msgr2=0x7f92e80b2f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:19:06.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.500+0000 7f92e7200640 1 -- 172.21.15.96:0/1556363535 shutdown_connections 2024-05-28T06:19:06.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.500+0000 7f92e7200640 1 -- 172.21.15.96:0/1556363535 wait complete. 2024-05-28T06:19:06.516 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-05-28T06:19:06.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.723+0000 7f4ffde00640 1 Processor -- start 2024-05-28T06:19:06.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.724+0000 7f4ffde00640 1 -- start start 2024-05-28T06:19:06.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.725+0000 7f4ffde00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f4ff815ed80 0x7f4ff80cff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:06.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.725+0000 7f4ffde00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4ff80b7e50 0x7f4ff80cedb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:06.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.725+0000 7f4ffde00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ff815f160 con 0x7f4ff80b61e0 2024-05-28T06:19:06.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.725+0000 7f4ffde00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f4ff8163540 con 0x7f4ff815ed80 2024-05-28T06:19:06.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.725+0000 7f4ffde00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f4ff8162930 con 0x7f4ff80b7e50 2024-05-28T06:19:06.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.726+0000 7f4ff6a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f4ff815ed80 0x7f4ff80cff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:06.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.726+0000 7f4ff6a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f4ff815ed80 0x7f4ff80cff80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:46586/0 (socket says 172.21.15.96:46586) 2024-05-28T06:19:06.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.726+0000 7f4ff6a00640 1 -- 172.21.15.96:0/2824268532 learned_addr learned my addr 172.21.15.96:0/2824268532 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:19:06.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.726+0000 7f4ff6000640 1 --2- 172.21.15.96:0/2824268532 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4ff80b7e50 0x7f4ff80cedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:06.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.726+0000 7f4ff6a00640 1 -- 172.21.15.96:0/2824268532 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4ff80b7e50 msgr2=0x7f4ff80cedb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:19:06.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.726+0000 7f4ff6a00640 1 --2- 172.21.15.96:0/2824268532 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4ff80b7e50 0x7f4ff80cedb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:06.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.726+0000 7f4ff6a00640 1 -- 172.21.15.96:0/2824268532 >> v1:172.21.15.123:6789/0 conn(0x7f4ff80b61e0 legacy=0x7f4ff80d2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:19:06.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.726+0000 7f4ff6a00640 1 -- 172.21.15.96:0/2824268532 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ff8162c10 con 0x7f4ff815ed80 2024-05-28T06:19:06.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.727+0000 7f4ff6a00640 1 --2- 172.21.15.96:0/2824268532 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4ff815ed80 0x7f4ff80cff80 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f4fe8009870 tx=0x7f4fe802a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc964048665783af server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:06.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.728+0000 7f4ff5600640 1 -- 172.21.15.96:0/2824268532 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fe8037070 con 0x7f4ff815ed80 2024-05-28T06:19:06.725 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.728+0000 7f4ff5600640 1 -- 172.21.15.96:0/2824268532 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fe8032440 con 0x7f4ff815ed80 2024-05-28T06:19:06.725 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.728+0000 7f4ff5600640 1 -- 172.21.15.96:0/2824268532 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fe8032810 con 0x7f4ff815ed80 2024-05-28T06:19:06.725 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.729+0000 7f4ffde00640 1 -- 172.21.15.96:0/2824268532 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4ff815ed80 msgr2=0x7f4ff80cff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:06.725 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.729+0000 7f4ffde00640 1 --2- 172.21.15.96:0/2824268532 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4ff815ed80 0x7f4ff80cff80 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f4fe8009870 tx=0x7f4fe802a210 comp rx=0 tx=0).stop 2024-05-28T06:19:06.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.729+0000 7f4ffde00640 1 -- 172.21.15.96:0/2824268532 shutdown_connections 2024-05-28T06:19:06.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.729+0000 7f4ffde00640 1 --2- 172.21.15.96:0/2824268532 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4ff80b7e50 0x7f4ff80cedb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:06.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.729+0000 7f4ffde00640 1 --2- 172.21.15.96:0/2824268532 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4ff815ed80 0x7f4ff80cff80 unknown :-1 s=CLOSED pgs=1455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:06.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.729+0000 7f4ffde00640 1 -- 172.21.15.96:0/2824268532 >> 172.21.15.96:0/2824268532 conn(0x7f4ff80d13c0 msgr2=0x7f4ff80d17d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:19:06.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.729+0000 7f4ffde00640 1 -- 172.21.15.96:0/2824268532 shutdown_connections 2024-05-28T06:19:06.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.729+0000 7f4ffde00640 1 -- 172.21.15.96:0/2824268532 wait complete. 2024-05-28T06:19:06.727 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.731+0000 7f4ffde00640 1 Processor -- start 2024-05-28T06:19:06.727 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.731+0000 7f4ffde00640 1 -- start start 2024-05-28T06:19:06.729 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.732+0000 7f4ffde00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4ff80b61e0 0x7f4ff81f90a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:06.729 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.732+0000 7f4ffde00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4ff80b7e50 0x7f4ff814fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:06.729 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.732+0000 7f4ffde00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4ff815ed80 0x7f4ff81507b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:06.730 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.732+0000 7f4ffde00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4ff80b6670 con 0x7f4ff80b61e0 2024-05-28T06:19:06.730 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.732+0000 7f4ffde00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f4ff81631d0 con 0x7f4ff80b7e50 2024-05-28T06:19:06.730 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.732+0000 7f4ffde00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f4ff80b67f0 con 0x7f4ff815ed80 2024-05-28T06:19:06.730 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.732+0000 7f4ff6a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4ff80b61e0 0x7f4ff81f90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:06.730 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.733+0000 7f4ff6a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4ff80b61e0 0x7f4ff81f90a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:46594/0 (socket says 172.21.15.96:46594) 2024-05-28T06:19:06.730 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.733+0000 7f4ff6a00640 1 -- 172.21.15.96:0/1188678549 learned_addr learned my addr 172.21.15.96:0/1188678549 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:19:06.730 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.733+0000 7f4ff7400640 1 --2- 172.21.15.96:0/1188678549 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4ff815ed80 0x7f4ff81507b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:06.730 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.733+0000 7f4ff6000640 1 --2- 172.21.15.96:0/1188678549 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4ff80b7e50 0x7f4ff814fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:06.730 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.733+0000 7f4ff6a00640 1 -- 172.21.15.96:0/1188678549 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4ff815ed80 msgr2=0x7f4ff81507b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:06.730 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.733+0000 7f4ff6a00640 1 --2- 172.21.15.96:0/1188678549 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4ff815ed80 0x7f4ff81507b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:06.730 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.733+0000 7f4ff6a00640 1 -- 172.21.15.96:0/1188678549 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4ff80b7e50 msgr2=0x7f4ff814fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:06.730 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.733+0000 7f4ff6a00640 1 --2- 172.21.15.96:0/1188678549 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4ff80b7e50 0x7f4ff814fc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:06.731 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.733+0000 7f4ff6a00640 1 -- 172.21.15.96:0/1188678549 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ff8151590 con 0x7f4ff80b61e0 2024-05-28T06:19:06.731 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.733+0000 7f4ff6a00640 1 --2- 172.21.15.96:0/1188678549 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4ff80b61e0 0x7f4ff81f90a0 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f4fe8000c00 tx=0x7f4fe80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:06.732 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.734+0000 7f4fef400640 1 -- 172.21.15.96:0/1188678549 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fe8041020 con 0x7f4ff80b61e0 2024-05-28T06:19:06.732 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.734+0000 7f4fef400640 1 -- 172.21.15.96:0/1188678549 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fe80306c0 con 0x7f4ff80b61e0 2024-05-28T06:19:06.732 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.734+0000 7f4ffde00640 1 -- 172.21.15.96:0/1188678549 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ff81521e0 con 0x7f4ff80b61e0 2024-05-28T06:19:06.732 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.734+0000 7f4fef400640 1 -- 172.21.15.96:0/1188678549 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fe8037040 con 0x7f4ff80b61e0 2024-05-28T06:19:06.732 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.734+0000 7f4ffde00640 1 -- 172.21.15.96:0/1188678549 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ff81527a0 con 0x7f4ff80b61e0 2024-05-28T06:19:06.732 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.735+0000 7f4ffde00640 1 -- 172.21.15.96:0/1188678549 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ff80689e0 con 0x7f4ff80b61e0 2024-05-28T06:19:06.732 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.736+0000 7f4fef400640 1 -- 172.21.15.96:0/1188678549 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 31) ==== 92672+0+0 (secure 0 0 0) 0x7f4fe8044020 con 0x7f4ff80b61e0 2024-05-28T06:19:06.733 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.737+0000 7f4fef400640 1 --2- 172.21.15.96:0/1188678549 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4fd406daf0 0x7f4fd406ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:06.733 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.737+0000 7f4fef400640 1 -- 172.21.15.96:0/1188678549 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 5766+0+0 (secure 0 0 0) 0x7f4fe80aaf60 con 0x7f4ff80b61e0 2024-05-28T06:19:06.734 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.738+0000 7f4ff6000640 1 --2- 172.21.15.96:0/1188678549 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4fd406daf0 0x7f4fd406ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:06.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.739+0000 7f4ff6000640 1 --2- 172.21.15.96:0/1188678549 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4fd406daf0 0x7f4fd406ffb0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f4ff80b5c80 tx=0x7f4fe000b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:06.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.741+0000 7f4fef400640 1 -- 172.21.15.96:0/1188678549 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4fe804b030 con 0x7f4ff80b61e0 2024-05-28T06:19:06.970 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.973+0000 7f4ffde00640 1 -- 172.21.15.96:0/1188678549 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) -- 0x7f4ff80d2910 con 0x7f4fd406daf0 2024-05-28T06:19:06.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.985+0000 7f4fef400640 1 -- 172.21.15.96:0/1188678549 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+59 (secure 0 0 0) 0x7f4ff80d2910 con 0x7f4fd406daf0 2024-05-28T06:19:06.981 INFO:teuthology.orchestra.run.smithi096.stdout:/volumes/_nogroup/sv1/babc868a-5738-42d7-be15-95a0a95d1980 2024-05-28T06:19:06.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.987+0000 7f4ffde00640 1 -- 172.21.15.96:0/1188678549 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4fd406daf0 msgr2=0x7f4fd406ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:06.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.987+0000 7f4ffde00640 1 --2- 172.21.15.96:0/1188678549 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4fd406daf0 0x7f4fd406ffb0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f4ff80b5c80 tx=0x7f4fe000b040 comp rx=0 tx=0).stop 2024-05-28T06:19:06.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.987+0000 7f4ffde00640 1 -- 172.21.15.96:0/1188678549 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4ff80b61e0 msgr2=0x7f4ff81f90a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:06.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.987+0000 7f4ffde00640 1 --2- 172.21.15.96:0/1188678549 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4ff80b61e0 0x7f4ff81f90a0 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f4fe8000c00 tx=0x7f4fe80046c0 comp rx=0 tx=0).stop 2024-05-28T06:19:06.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.987+0000 7f4ffde00640 1 -- 172.21.15.96:0/1188678549 shutdown_connections 2024-05-28T06:19:06.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.987+0000 7f4ffde00640 1 --2- 172.21.15.96:0/1188678549 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4fd406daf0 0x7f4fd406ffb0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:06.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.987+0000 7f4ffde00640 1 --2- 172.21.15.96:0/1188678549 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4ff815ed80 0x7f4ff81507b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:06.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.987+0000 7f4ffde00640 1 --2- 172.21.15.96:0/1188678549 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4ff80b7e50 0x7f4ff814fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:06.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.987+0000 7f4ffde00640 1 --2- 172.21.15.96:0/1188678549 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4ff80b61e0 0x7f4ff81f90a0 unknown :-1 s=CLOSED pgs=1456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:06.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.987+0000 7f4ffde00640 1 -- 172.21.15.96:0/1188678549 >> 172.21.15.96:0/1188678549 conn(0x7f4ff80d13c0 msgr2=0x7f4ff80d2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:19:06.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.987+0000 7f4ffde00640 1 -- 172.21.15.96:0/1188678549 shutdown_connections 2024-05-28T06:19:06.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:06.987+0000 7f4ffde00640 1 -- 172.21.15.96:0/1188678549 wait complete. 2024-05-28T06:19:07.002 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:19:07.002 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/babc868a-5738-42d7-be15-95a0a95d1980 2024-05-28T06:19:07.002 DEBUG:teuthology.orchestra.run.smithi096:> ) 2024-05-28T06:19:07.058 INFO:teuthology.orchestra.run.smithi096.stdout:# file: volumes/_nogroup/sv1/babc868a-5738-42d7-be15-95a0a95d1980 2024-05-28T06:19:07.059 INFO:teuthology.orchestra.run.smithi096.stdout:ceph.dir.rbytes="3138256896" 2024-05-28T06:19:07.059 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:19:07.060 INFO:tasks.cephfs.test_volumes:r_size = 3138256896 exp_size = 3221225472 2024-05-28T06:19:09.060 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:19:09.061 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/babc868a-5738-42d7-be15-95a0a95d1980 2024-05-28T06:19:09.061 DEBUG:teuthology.orchestra.run.smithi096:> ) 2024-05-28T06:19:09.124 INFO:teuthology.orchestra.run.smithi096.stdout:# file: volumes/_nogroup/sv1/babc868a-5738-42d7-be15-95a0a95d1980 2024-05-28T06:19:09.124 INFO:teuthology.orchestra.run.smithi096.stdout:ceph.dir.rbytes="3138256896" 2024-05-28T06:19:09.124 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:19:09.125 INFO:tasks.cephfs.test_volumes:r_size = 3138256896 exp_size = 3221225472 2024-05-28T06:19:11.126 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:19:11.126 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/babc868a-5738-42d7-be15-95a0a95d1980 2024-05-28T06:19:11.126 DEBUG:teuthology.orchestra.run.smithi096:> ) 2024-05-28T06:19:11.192 INFO:teuthology.orchestra.run.smithi096.stdout:# file: volumes/_nogroup/sv1/babc868a-5738-42d7-be15-95a0a95d1980 2024-05-28T06:19:11.192 INFO:teuthology.orchestra.run.smithi096.stdout:ceph.dir.rbytes="3221225472" 2024-05-28T06:19:11.192 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:19:11.193 INFO:tasks.cephfs.test_volumes:r_size = 3221225472 exp_size = 3221225472 2024-05-28T06:19:11.194 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 ss1clone1 2024-05-28T06:19:11.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.406+0000 7efde2600640 1 Processor -- start 2024-05-28T06:19:11.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.406+0000 7efde2600640 1 -- start start 2024-05-28T06:19:11.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.407+0000 7efde2600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7efddc0ce1f0 0x7efddc0b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:11.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.408+0000 7efde2600640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efddc0b3580 con 0x7efddc0b3d60 2024-05-28T06:19:11.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.408+0000 7efde2600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efddc15f490 con 0x7efddc0ce5d0 2024-05-28T06:19:11.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.408+0000 7efde2600640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7efddc15f770 con 0x7efddc0ce1f0 2024-05-28T06:19:11.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.408+0000 7efde1c00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7efddc0b3d60 0x7efddc0ccb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:44914/0 (socket says 172.21.15.96:44914) 2024-05-28T06:19:11.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.408+0000 7efde1c00640 1 -- 172.21.15.96:0/463330226 learned_addr learned my addr 172.21.15.96:0/463330226 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:19:11.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.408+0000 7efde1200640 1 --2- 172.21.15.96:0/463330226 >> v2:172.21.15.123:3300/0 conn(0x7efddc0ce1f0 0x7efddc0b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:11.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.409+0000 7efddb400640 1 -- 172.21.15.96:0/463330226 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2741062208 0 0) 0x7efddc0b3580 con 0x7efddc0b3d60 2024-05-28T06:19:11.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.409+0000 7efddb400640 1 -- 172.21.15.96:0/463330226 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efdc4003520 con 0x7efddc0b3d60 2024-05-28T06:19:11.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.409+0000 7efddb400640 1 -- 172.21.15.96:0/463330226 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 361075430 0 0) 0x7efdc4003520 con 0x7efddc0b3d60 2024-05-28T06:19:11.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.409+0000 7efddb400640 1 -- 172.21.15.96:0/463330226 >> v2:172.21.15.123:3300/0 conn(0x7efddc0ce1f0 msgr2=0x7efddc0b4dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:19:11.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.409+0000 7efddb400640 1 --2- 172.21.15.96:0/463330226 >> v2:172.21.15.123:3300/0 conn(0x7efddc0ce1f0 0x7efddc0b4dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:11.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.409+0000 7efddb400640 1 -- 172.21.15.96:0/463330226 >> v1:172.21.15.123:6789/0 conn(0x7efddc0ce5d0 legacy=0x7efddc15ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:11.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.409+0000 7efddb400640 1 -- 172.21.15.96:0/463330226 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efddc15fa50 con 0x7efddc0b3d60 2024-05-28T06:19:11.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.409+0000 7efddb400640 1 -- 172.21.15.96:0/463330226 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7efdd40019f0 con 0x7efddc0b3d60 2024-05-28T06:19:11.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.409+0000 7efddb400640 1 -- 172.21.15.96:0/463330226 >> v1:172.21.15.96:6789/0 conn(0x7efddc0b3d60 legacy=0x7efddc0ccb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:11.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.409+0000 7efddb400640 1 --2- 172.21.15.96:0/463330226 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efdc4003a00 0x7efdc4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:11.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.409+0000 7efddb400640 1 --2- 172.21.15.96:0/463330226 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efdc4004800 0x7efdc4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:11.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.409+0000 7efddb400640 1 --2- 172.21.15.96:0/463330226 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efdc40052e0 0x7efdc400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:11.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.409+0000 7efddb400640 1 -- 172.21.15.96:0/463330226 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7efddc15f770 con 0x7efdc40052e0 2024-05-28T06:19:11.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.409+0000 7efddb400640 1 -- 172.21.15.96:0/463330226 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7efdc400c1e0 con 0x7efdc4003a00 2024-05-28T06:19:11.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.409+0000 7efddb400640 1 -- 172.21.15.96:0/463330226 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7efdc400c3e0 con 0x7efdc4004800 2024-05-28T06:19:11.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.409+0000 7efde1c00640 1 --2- 172.21.15.96:0/463330226 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efdc40052e0 0x7efdc400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:11.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.410+0000 7efde1200640 1 --2- 172.21.15.96:0/463330226 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efdc4003a00 0x7efdc4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:11.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.410+0000 7efddbe00640 1 --2- 172.21.15.96:0/463330226 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efdc4004800 0x7efdc4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:11.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.410+0000 7efde1c00640 1 -- 172.21.15.96:0/463330226 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efdc4004800 msgr2=0x7efdc4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:19:11.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.410+0000 7efde1c00640 1 --2- 172.21.15.96:0/463330226 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efdc4004800 0x7efdc4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:11.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.410+0000 7efde1c00640 1 -- 172.21.15.96:0/463330226 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efdc4003a00 msgr2=0x7efdc4003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:19:11.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.410+0000 7efde1c00640 1 --2- 172.21.15.96:0/463330226 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efdc4003a00 0x7efdc4003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:11.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.410+0000 7efde1c00640 1 -- 172.21.15.96:0/463330226 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdc4003520 con 0x7efdc40052e0 2024-05-28T06:19:11.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.410+0000 7efddbe00640 1 --2- 172.21.15.96:0/463330226 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efdc4004800 0x7efdc4004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:19:11.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.410+0000 7efde1200640 1 --2- 172.21.15.96:0/463330226 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efdc4003a00 0x7efdc4003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:19:11.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.411+0000 7efde1c00640 1 -- 172.21.15.96:0/463330226 reap_dead start 2024-05-28T06:19:11.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.411+0000 7efde1c00640 1 --2- 172.21.15.96:0/463330226 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efdc40052e0 0x7efdc400b9c0 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7efdd400a770 tx=0x7efdc4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd1effe447eb58db server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:11.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.411+0000 7efddb400640 1 -- 172.21.15.96:0/463330226 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdd400a390 con 0x7efdc40052e0 2024-05-28T06:19:11.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.411+0000 7efddb400640 1 -- 172.21.15.96:0/463330226 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efdd402f070 con 0x7efdc40052e0 2024-05-28T06:19:11.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.411+0000 7efddb400640 1 -- 172.21.15.96:0/463330226 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdd402bbe0 con 0x7efdc40052e0 2024-05-28T06:19:11.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.412+0000 7efde2600640 1 -- 172.21.15.96:0/463330226 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efdc40052e0 msgr2=0x7efdc400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:11.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.412+0000 7efde2600640 1 --2- 172.21.15.96:0/463330226 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efdc40052e0 0x7efdc400b9c0 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7efdd400a770 tx=0x7efdc4005830 comp rx=0 tx=0).stop 2024-05-28T06:19:11.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.412+0000 7efde2600640 1 -- 172.21.15.96:0/463330226 shutdown_connections 2024-05-28T06:19:11.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.412+0000 7efde2600640 1 --2- 172.21.15.96:0/463330226 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efdc40052e0 0x7efdc400b9c0 unknown :-1 s=CLOSED pgs=1458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:11.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.412+0000 7efde2600640 1 -- 172.21.15.96:0/463330226 >> 172.21.15.96:0/463330226 conn(0x7efddc06b900 msgr2=0x7efddc06bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:19:11.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.412+0000 7efde2600640 1 -- 172.21.15.96:0/463330226 shutdown_connections 2024-05-28T06:19:11.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.412+0000 7efde2600640 1 -- 172.21.15.96:0/463330226 wait complete. 2024-05-28T06:19:11.412 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.414+0000 7efde2600640 1 Processor -- start 2024-05-28T06:19:11.412 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.414+0000 7efde2600640 1 -- start start 2024-05-28T06:19:11.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.415+0000 7efde2600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efdc40052e0 0x7efddc0998f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:11.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.415+0000 7efde2600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efddc156ed0 0x7efddc157350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:11.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.415+0000 7efde2600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efddc1e90a0 0x7efddc1e9530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:11.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.415+0000 7efde2600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7efddc0b4f50 con 0x7efdc40052e0 2024-05-28T06:19:11.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.415+0000 7efde2600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7efddc0b4dd0 con 0x7efddc1e90a0 2024-05-28T06:19:11.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.415+0000 7efde2600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7efddc0b50d0 con 0x7efddc156ed0 2024-05-28T06:19:11.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.415+0000 7efde1200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efdc40052e0 0x7efddc0998f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:11.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.415+0000 7efde1200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efdc40052e0 0x7efddc0998f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:36718/0 (socket says 172.21.15.96:36718) 2024-05-28T06:19:11.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.415+0000 7efde1200640 1 -- 172.21.15.96:0/3413644161 learned_addr learned my addr 172.21.15.96:0/3413644161 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:19:11.414 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.416+0000 7efde1200640 1 -- 172.21.15.96:0/3413644161 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efddc156ed0 msgr2=0x7efddc157350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:11.414 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.416+0000 7efde1200640 1 --2- 172.21.15.96:0/3413644161 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efddc156ed0 0x7efddc157350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:11.415 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.416+0000 7efde1200640 1 -- 172.21.15.96:0/3413644161 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efddc1e90a0 msgr2=0x7efddc1e9530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:11.415 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.416+0000 7efde1200640 1 --2- 172.21.15.96:0/3413644161 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efddc1e90a0 0x7efddc1e9530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:11.415 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.416+0000 7efde1200640 1 -- 172.21.15.96:0/3413644161 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efddc1e9dd0 con 0x7efdc40052e0 2024-05-28T06:19:11.415 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.416+0000 7efde1200640 1 --2- 172.21.15.96:0/3413644161 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efdc40052e0 0x7efddc0998f0 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7efdd00108b0 tx=0x7efdd0010d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:11.415 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.416+0000 7efdd8c00640 1 -- 172.21.15.96:0/3413644161 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdd000f040 con 0x7efdc40052e0 2024-05-28T06:19:11.415 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.416+0000 7efdd8c00640 1 -- 172.21.15.96:0/3413644161 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efdd000eb60 con 0x7efdc40052e0 2024-05-28T06:19:11.415 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.416+0000 7efde2600640 1 -- 172.21.15.96:0/3413644161 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efddc1ea0b0 con 0x7efdc40052e0 2024-05-28T06:19:11.415 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.416+0000 7efdd8c00640 1 -- 172.21.15.96:0/3413644161 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdd001c400 con 0x7efdc40052e0 2024-05-28T06:19:11.415 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.417+0000 7efde2600640 1 -- 172.21.15.96:0/3413644161 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efddc1ea670 con 0x7efdc40052e0 2024-05-28T06:19:11.415 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.418+0000 7efde2600640 1 -- 172.21.15.96:0/3413644161 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efddc0689e0 con 0x7efdc40052e0 2024-05-28T06:19:11.416 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.419+0000 7efdd8c00640 1 -- 172.21.15.96:0/3413644161 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 31) ==== 92672+0+0 (secure 0 0 0) 0x7efdd001c5a0 con 0x7efdc40052e0 2024-05-28T06:19:11.417 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.419+0000 7efdd8c00640 1 --2- 172.21.15.96:0/3413644161 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7efdbc06db20 0x7efdbc06ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:11.417 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.420+0000 7efdd8c00640 1 -- 172.21.15.96:0/3413644161 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 5766+0+0 (secure 0 0 0) 0x7efdd008b160 con 0x7efdc40052e0 2024-05-28T06:19:11.417 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.420+0000 7efddbe00640 1 --2- 172.21.15.96:0/3413644161 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7efdbc06db20 0x7efdbc06ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:11.418 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.421+0000 7efddbe00640 1 --2- 172.21.15.96:0/3413644161 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7efdbc06db20 0x7efdbc06ffe0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7efdc8000f60 tx=0x7efdc8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:11.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.424+0000 7efdd8c00640 1 -- 172.21.15.96:0/3413644161 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7efdd0018030 con 0x7efdc40052e0 2024-05-28T06:19:11.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.665+0000 7efde2600640 1 -- 172.21.15.96:0/3413644161 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "ss1clone1", "target": ["mon-mgr", ""]}) -- 0x7efddc0b3580 con 0x7efdbc06db20 2024-05-28T06:19:11.812 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.814+0000 7efdd8c00640 1 -- 172.21.15.96:0/3413644161 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7efddc0b3580 con 0x7efdbc06db20 2024-05-28T06:19:11.816 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.818+0000 7efde2600640 1 -- 172.21.15.96:0/3413644161 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7efdbc06db20 msgr2=0x7efdbc06ffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:11.816 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.818+0000 7efde2600640 1 --2- 172.21.15.96:0/3413644161 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7efdbc06db20 0x7efdbc06ffe0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7efdc8000f60 tx=0x7efdc8009040 comp rx=0 tx=0).stop 2024-05-28T06:19:11.816 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.818+0000 7efde2600640 1 -- 172.21.15.96:0/3413644161 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efdc40052e0 msgr2=0x7efddc0998f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:11.816 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.818+0000 7efde2600640 1 --2- 172.21.15.96:0/3413644161 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efdc40052e0 0x7efddc0998f0 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7efdd00108b0 tx=0x7efdd0010d70 comp rx=0 tx=0).stop 2024-05-28T06:19:11.816 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.818+0000 7efde2600640 1 -- 172.21.15.96:0/3413644161 shutdown_connections 2024-05-28T06:19:11.816 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.818+0000 7efde2600640 1 --2- 172.21.15.96:0/3413644161 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7efdbc06db20 0x7efdbc06ffe0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:11.816 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.818+0000 7efde2600640 1 --2- 172.21.15.96:0/3413644161 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efddc1e90a0 0x7efddc1e9530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:11.816 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.818+0000 7efde2600640 1 --2- 172.21.15.96:0/3413644161 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efddc156ed0 0x7efddc157350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:11.817 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.818+0000 7efde2600640 1 --2- 172.21.15.96:0/3413644161 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efdc40052e0 0x7efddc0998f0 unknown :-1 s=CLOSED pgs=1459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:11.817 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.818+0000 7efde2600640 1 -- 172.21.15.96:0/3413644161 >> 172.21.15.96:0/3413644161 conn(0x7efddc06b900 msgr2=0x7efddc072eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:19:11.817 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.818+0000 7efde2600640 1 -- 172.21.15.96:0/3413644161 shutdown_connections 2024-05-28T06:19:11.817 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:11.818+0000 7efde2600640 1 -- 172.21.15.96:0/3413644161 wait complete. 2024-05-28T06:19:11.819 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:19:11.822+0000 7fea60c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:19:11.819 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:19:11.822+0000 7fea60c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:19:11.819 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:19:11.822+0000 7fea60c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:19:11.819 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:19:11.822+0000 7fea60c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:19:11.820 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:19:11.822+0000 7fea60c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:19:11.837 INFO:tasks.cephfs.test_volumes:expected clone state = ('in-progress',) 2024-05-28T06:19:11.837 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-05-28T06:19:11.849 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:19:11.852+0000 7fea5f800640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:19:11.849 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:19:11.852+0000 7fea5f800640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:19:11.849 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:19:11.852+0000 7fea5f800640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:19:11.850 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:19:11.852+0000 7fea5f800640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:19:11.850 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:19:11.852+0000 7fea5f800640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:19:12.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:12.034+0000 7f643a800640 1 Processor -- start 2024-05-28T06:19:12.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:12.034+0000 7f643a800640 1 -- start start 2024-05-28T06:19:12.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:12.036+0000 7f643a800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f643415b6f0 0x7f643406e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:12.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:12.036+0000 7f643a800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f64340725e0 0x7f64340ae650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:12.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:12.036+0000 7f643a800640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64340ade90 con 0x7f64340d0640 2024-05-28T06:19:12.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:12.036+0000 7f643a800640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f6434071c40 con 0x7f643415b6f0 2024-05-28T06:19:12.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:12.036+0000 7f643a800640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f6434071e70 con 0x7f64340725e0 2024-05-28T06:19:12.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:12.036+0000 7f6433400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f64340d0640 0x7f64340d0a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:44928/0 (socket says 172.21.15.96:44928) 2024-05-28T06:19:12.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:12.036+0000 7f6433400640 1 -- 172.21.15.96:0/590991570 learned_addr learned my addr 172.21.15.96:0/590991570 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:19:12.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:12.036+0000 7f6432a00640 1 --2- 172.21.15.96:0/590991570 >> v2:172.21.15.96:3300/0 conn(0x7f643415b6f0 0x7f643406e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:12.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:12.037+0000 7f6433e00640 1 --2- 172.21.15.96:0/590991570 >> v2:172.21.15.123:3300/0 conn(0x7f64340725e0 0x7f64340ae650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:12.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:12.037+0000 7f6432a00640 1 -- 172.21.15.96:0/590991570 >> v2:172.21.15.123:3300/0 conn(0x7f64340725e0 msgr2=0x7f64340ae650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:19:12.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:12.037+0000 7f6432a00640 1 --2- 172.21.15.96:0/590991570 >> v2:172.21.15.123:3300/0 conn(0x7f64340725e0 0x7f64340ae650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:12.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:12.037+0000 7f6432a00640 1 -- 172.21.15.96:0/590991570 >> v1:172.21.15.96:6789/0 conn(0x7f64340d0640 legacy=0x7f64340d0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:12.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:12.037+0000 7f6432000640 1 -- 172.21.15.96:0/590991570 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4112307813 0 0) 0x7f64340ade90 con 0x7f64340d0640 2024-05-28T06:19:12.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:12.037+0000 7f6432a00640 1 -- 172.21.15.96:0/590991570 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6434072130 con 0x7f643415b6f0 2024-05-28T06:19:12.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:12.038+0000 7f6432a00640 1 --2- 172.21.15.96:0/590991570 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f643415b6f0 0x7f643406e350 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7f64340b3370 tx=0x7f642402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b783c07b77a329d server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:12.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:12.038+0000 7f6432000640 1 -- 172.21.15.96:0/590991570 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6424037070 con 0x7f643415b6f0 2024-05-28T06:19:12.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:12.038+0000 7f6432000640 1 -- 172.21.15.96:0/590991570 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6424032440 con 0x7f643415b6f0 2024-05-28T06:19:12.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:12.038+0000 7f6432000640 1 -- 172.21.15.96:0/590991570 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6424032810 con 0x7f643415b6f0 2024-05-28T06:19:12.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:12.039+0000 7f643a800640 1 -- 172.21.15.96:0/590991570 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f643415b6f0 msgr2=0x7f643406e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:12.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:12.039+0000 7f643a800640 1 --2- 172.21.15.96:0/590991570 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f643415b6f0 0x7f643406e350 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7f64340b3370 tx=0x7f642402a1a0 comp rx=0 tx=0).stop 2024-05-28T06:19:12.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:12.039+0000 7f643a800640 1 -- 172.21.15.96:0/590991570 shutdown_connections 2024-05-28T06:19:12.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:12.039+0000 7f643a800640 1 --2- 172.21.15.96:0/590991570 >> v2:172.21.15.123:3300/0 conn(0x7f64340725e0 0x7f64340ae650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:12.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:12.039+0000 7f643a800640 1 --2- 172.21.15.96:0/590991570 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f643415b6f0 0x7f643406e350 unknown :-1 s=CLOSED pgs=1461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:12.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:12.039+0000 7f643a800640 1 -- 172.21.15.96:0/590991570 >> 172.21.15.96:0/590991570 conn(0x7f643406bd20 msgr2=0x7f643406c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:19:12.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:12.039+0000 7f643a800640 1 -- 172.21.15.96:0/590991570 shutdown_connections 2024-05-28T06:19:12.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:12.039+0000 7f643a800640 1 -- 172.21.15.96:0/590991570 wait complete. 2024-05-28T06:19:12.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:12.041+0000 7f643a800640 1 Processor -- start 2024-05-28T06:19:12.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:12.041+0000 7f643a800640 1 -- start start 2024-05-28T06:19:12.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:12.042+0000 7f643a800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f64340725e0 0x7f64340b48f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:12.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:12.042+0000 7f643a800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64340d0640 0x7f643414c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:12.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:12.042+0000 7f643a800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f643415b6f0 0x7f64341553c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:12.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:12.042+0000 7f643a800640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f64340aeb90 con 0x7f643415b6f0 2024-05-28T06:19:12.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:12.042+0000 7f643a800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f643406e890 con 0x7f64340d0640 2024-05-28T06:19:12.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:12.042+0000 7f643a800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f64340aed10 con 0x7f64340725e0 2024-05-28T06:19:12.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:12.042+0000 7f6433e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f643415b6f0 0x7f64341553c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:12.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:12.042+0000 7f6433e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f643415b6f0 0x7f64341553c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:36738/0 (socket says 172.21.15.96:36738) 2024-05-28T06:19:12.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:12.042+0000 7f6433e00640 1 -- 172.21.15.96:0/2418497060 learned_addr learned my addr 172.21.15.96:0/2418497060 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:19:12.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:12.043+0000 7f6432a00640 1 --2- 172.21.15.96:0/2418497060 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64340d0640 0x7f643414c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:12.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:12.043+0000 7f6433e00640 1 -- 172.21.15.96:0/2418497060 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f64340725e0 msgr2=0x7f64340b48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:12.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:12.043+0000 7f6433e00640 1 --2- 172.21.15.96:0/2418497060 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f64340725e0 0x7f64340b48f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:12.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:12.043+0000 7f6433e00640 1 -- 172.21.15.96:0/2418497060 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64340d0640 msgr2=0x7f643414c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:12.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:12.043+0000 7f6433e00640 1 --2- 172.21.15.96:0/2418497060 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64340d0640 0x7f643414c7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:12.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:12.043+0000 7f6433e00640 1 -- 172.21.15.96:0/2418497060 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6434158a40 con 0x7f643415b6f0 2024-05-28T06:19:12.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:12.043+0000 7f6433e00640 1 --2- 172.21.15.96:0/2418497060 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f643415b6f0 0x7f64341553c0 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7f642c007e70 tx=0x7f642c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:12.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:12.043+0000 7f642b400640 1 -- 172.21.15.96:0/2418497060 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f642c014620 con 0x7f643415b6f0 2024-05-28T06:19:12.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:12.043+0000 7f642b400640 1 -- 172.21.15.96:0/2418497060 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f642c019070 con 0x7f643415b6f0 2024-05-28T06:19:12.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:12.043+0000 7f643a800640 1 -- 172.21.15.96:0/2418497060 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6434158d20 con 0x7f643415b6f0 2024-05-28T06:19:12.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:12.043+0000 7f642b400640 1 -- 172.21.15.96:0/2418497060 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f642c014a20 con 0x7f643415b6f0 2024-05-28T06:19:12.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:12.044+0000 7f643a800640 1 -- 172.21.15.96:0/2418497060 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64341592e0 con 0x7f643415b6f0 2024-05-28T06:19:12.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:12.045+0000 7f643a800640 1 -- 172.21.15.96:0/2418497060 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64340689e0 con 0x7f643415b6f0 2024-05-28T06:19:12.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:12.046+0000 7f642b400640 1 -- 172.21.15.96:0/2418497060 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 31) ==== 92672+0+0 (secure 0 0 0) 0x7f642c014bc0 con 0x7f643415b6f0 2024-05-28T06:19:12.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:12.047+0000 7f642b400640 1 --2- 172.21.15.96:0/2418497060 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f641006db20 0x7f641006ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:12.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:12.047+0000 7f642b400640 1 -- 172.21.15.96:0/2418497060 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 5766+0+0 (secure 0 0 0) 0x7f642c08b3e0 con 0x7f643415b6f0 2024-05-28T06:19:12.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:12.047+0000 7f6433400640 1 --2- 172.21.15.96:0/2418497060 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f641006db20 0x7f641006ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:12.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:12.049+0000 7f6433400640 1 --2- 172.21.15.96:0/2418497060 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f641006db20 0x7f641006ffe0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f641c007fd0 tx=0x7f641c007920 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:12.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:12.051+0000 7f642b400640 1 -- 172.21.15.96:0/2418497060 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f642c011260 con 0x7f643415b6f0 2024-05-28T06:19:12.290 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:12.293+0000 7f643a800640 1 -- 172.21.15.96:0/2418497060 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) -- 0x7f64340afe10 con 0x7f641006db20 2024-05-28T06:19:12.818 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:19:12.821+0000 7fea03a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:19:12.818 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:19:12.821+0000 7fea03a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:19:12.818 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:19:12.821+0000 7fea03a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:19:12.818 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:19:12.821+0000 7fea03a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:19:12.818 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:19:12.821+0000 7fea03a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:19:12.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:12.879+0000 7f642b400640 1 -- 172.21.15.96:0/2418497060 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+275 (secure 0 0 0) 0x7f64340afe10 con 0x7f641006db20 2024-05-28T06:19:12.877 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:19:12.877 INFO:teuthology.orchestra.run.smithi096.stdout: "status": { 2024-05-28T06:19:12.877 INFO:teuthology.orchestra.run.smithi096.stdout: "state": "in-progress", 2024-05-28T06:19:12.877 INFO:teuthology.orchestra.run.smithi096.stdout: "source": { 2024-05-28T06:19:12.877 INFO:teuthology.orchestra.run.smithi096.stdout: "volume": "cephfs", 2024-05-28T06:19:12.877 INFO:teuthology.orchestra.run.smithi096.stdout: "subvolume": "sv1", 2024-05-28T06:19:12.877 INFO:teuthology.orchestra.run.smithi096.stdout: "snapshot": "ss1" 2024-05-28T06:19:12.877 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:19:12.877 INFO:teuthology.orchestra.run.smithi096.stdout: "progress_report": { 2024-05-28T06:19:12.877 INFO:teuthology.orchestra.run.smithi096.stdout: "percentage cloned": 0.0, 2024-05-28T06:19:12.878 INFO:teuthology.orchestra.run.smithi096.stdout: "amount cloned": "184/3.0G", 2024-05-28T06:19:12.878 INFO:teuthology.orchestra.run.smithi096.stdout: "files cloned": "3/6" 2024-05-28T06:19:12.878 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:12.878 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:12.878 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:19:12.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:12.882+0000 7f643a800640 1 -- 172.21.15.96:0/2418497060 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f641006db20 msgr2=0x7f641006ffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:12.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:12.882+0000 7f643a800640 1 --2- 172.21.15.96:0/2418497060 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f641006db20 0x7f641006ffe0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f641c007fd0 tx=0x7f641c007920 comp rx=0 tx=0).stop 2024-05-28T06:19:12.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:12.882+0000 7f643a800640 1 -- 172.21.15.96:0/2418497060 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f643415b6f0 msgr2=0x7f64341553c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:12.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:12.882+0000 7f643a800640 1 --2- 172.21.15.96:0/2418497060 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f643415b6f0 0x7f64341553c0 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7f642c007e70 tx=0x7f642c00e4b0 comp rx=0 tx=0).stop 2024-05-28T06:19:12.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:12.883+0000 7f643a800640 1 -- 172.21.15.96:0/2418497060 shutdown_connections 2024-05-28T06:19:12.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:12.883+0000 7f643a800640 1 --2- 172.21.15.96:0/2418497060 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f641006db20 0x7f641006ffe0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:12.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:12.883+0000 7f643a800640 1 --2- 172.21.15.96:0/2418497060 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f643415b6f0 0x7f64341553c0 unknown :-1 s=CLOSED pgs=1462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:12.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:12.883+0000 7f643a800640 1 --2- 172.21.15.96:0/2418497060 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64340d0640 0x7f643414c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:12.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:12.883+0000 7f643a800640 1 --2- 172.21.15.96:0/2418497060 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f64340725e0 0x7f64340b48f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:12.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:12.883+0000 7f643a800640 1 -- 172.21.15.96:0/2418497060 >> 172.21.15.96:0/2418497060 conn(0x7f643406bd20 msgr2=0x7f6434072c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:19:12.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:12.883+0000 7f643a800640 1 -- 172.21.15.96:0/2418497060 shutdown_connections 2024-05-28T06:19:12.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:12.883+0000 7f643a800640 1 -- 172.21.15.96:0/2418497060 wait complete. 2024-05-28T06:19:12.903 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-05-28T06:19:13.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.114+0000 7f979a200640 1 Processor -- start 2024-05-28T06:19:13.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.114+0000 7f979a200640 1 -- start start 2024-05-28T06:19:13.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.115+0000 7f979a200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f97940d0640 0x7f97940d0a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:13.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.115+0000 7f979a200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97940ade90 con 0x7f97940725e0 2024-05-28T06:19:13.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.116+0000 7f979a200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97940d0f60 con 0x7f979415b6f0 2024-05-28T06:19:13.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.116+0000 7f979a200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f9794071c40 con 0x7f97940d0640 2024-05-28T06:19:13.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.116+0000 7f9793400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f97940d0640 0x7f97940d0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:13.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.116+0000 7f9793400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f97940d0640 0x7f97940d0a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:36754/0 (socket says 172.21.15.96:36754) 2024-05-28T06:19:13.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.116+0000 7f9793400640 1 -- 172.21.15.96:0/1106375874 learned_addr learned my addr 172.21.15.96:0/1106375874 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:19:13.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.116+0000 7f9793400640 1 -- 172.21.15.96:0/1106375874 >> v1:172.21.15.123:6789/0 conn(0x7f979415b6f0 legacy=0x7f979406e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:19:13.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.116+0000 7f9793400640 1 -- 172.21.15.96:0/1106375874 >> v1:172.21.15.96:6789/0 conn(0x7f97940725e0 legacy=0x7f97940ae650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:13.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.116+0000 7f9793400640 1 -- 172.21.15.96:0/1106375874 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9794071e70 con 0x7f97940d0640 2024-05-28T06:19:13.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.118+0000 7f9793400640 1 --2- 172.21.15.96:0/1106375874 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97940d0640 0x7f97940d0a20 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7f979406fe40 tx=0x7f977c02a2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e02e78e895737e7 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:13.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.118+0000 7f9792000640 1 -- 172.21.15.96:0/1106375874 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f977c037070 con 0x7f97940d0640 2024-05-28T06:19:13.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.118+0000 7f9792000640 1 -- 172.21.15.96:0/1106375874 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f977c032440 con 0x7f97940d0640 2024-05-28T06:19:13.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.118+0000 7f9792000640 1 -- 172.21.15.96:0/1106375874 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f977c032810 con 0x7f97940d0640 2024-05-28T06:19:13.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.119+0000 7f979a200640 1 -- 172.21.15.96:0/1106375874 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97940d0640 msgr2=0x7f97940d0a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:13.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.119+0000 7f979a200640 1 --2- 172.21.15.96:0/1106375874 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97940d0640 0x7f97940d0a20 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7f979406fe40 tx=0x7f977c02a2a0 comp rx=0 tx=0).stop 2024-05-28T06:19:13.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.120+0000 7f979a200640 1 -- 172.21.15.96:0/1106375874 shutdown_connections 2024-05-28T06:19:13.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.120+0000 7f979a200640 1 --2- 172.21.15.96:0/1106375874 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97940d0640 0x7f97940d0a20 unknown :-1 s=CLOSED pgs=1464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:13.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.120+0000 7f979a200640 1 -- 172.21.15.96:0/1106375874 >> 172.21.15.96:0/1106375874 conn(0x7f979406bd20 msgr2=0x7f979406c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:19:13.120 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.122+0000 7f979a200640 1 -- 172.21.15.96:0/1106375874 shutdown_connections 2024-05-28T06:19:13.120 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.122+0000 7f979a200640 1 -- 172.21.15.96:0/1106375874 wait complete. 2024-05-28T06:19:13.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.126+0000 7f979a200640 1 Processor -- start 2024-05-28T06:19:13.125 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.126+0000 7f979a200640 1 -- start start 2024-05-28T06:19:13.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.128+0000 7f979a200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f97940725e0 0x7f979414f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:13.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.128+0000 7f979a200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97940d0640 0x7f979415c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:13.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.128+0000 7f979a200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f979415b6f0 0x7f97941f8b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:13.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.128+0000 7f979a200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f97940aed60 con 0x7f97940d0640 2024-05-28T06:19:13.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.128+0000 7f979a200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f97940ae130 con 0x7f979415b6f0 2024-05-28T06:19:13.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.128+0000 7f979a200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f97940aeee0 con 0x7f97940725e0 2024-05-28T06:19:13.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.129+0000 7f9792a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97940d0640 0x7f979415c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:13.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.129+0000 7f9792a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97940d0640 0x7f979415c4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:36756/0 (socket says 172.21.15.96:36756) 2024-05-28T06:19:13.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.129+0000 7f9792a00640 1 -- 172.21.15.96:0/2499900697 learned_addr learned my addr 172.21.15.96:0/2499900697 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:19:13.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.130+0000 7f9793400640 1 --2- 172.21.15.96:0/2499900697 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f97940725e0 0x7f979414f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:13.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.130+0000 7f9792a00640 1 -- 172.21.15.96:0/2499900697 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f97940725e0 msgr2=0x7f979414f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:13.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.130+0000 7f9792a00640 1 --2- 172.21.15.96:0/2499900697 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f97940725e0 0x7f979414f320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:13.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.130+0000 7f9792a00640 1 -- 172.21.15.96:0/2499900697 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f979415b6f0 msgr2=0x7f97941f8b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:13.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.130+0000 7f9792a00640 1 --2- 172.21.15.96:0/2499900697 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f979415b6f0 0x7f97941f8b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:13.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.130+0000 7f9792a00640 1 -- 172.21.15.96:0/2499900697 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f979415cf30 con 0x7f97940d0640 2024-05-28T06:19:13.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.131+0000 7f9792a00640 1 --2- 172.21.15.96:0/2499900697 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97940d0640 0x7f979415c4e0 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7f978400b3e0 tx=0x7f978400b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:13.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.131+0000 7f978b400640 1 -- 172.21.15.96:0/2499900697 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9784015020 con 0x7f97940d0640 2024-05-28T06:19:13.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.131+0000 7f978b400640 1 -- 172.21.15.96:0/2499900697 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9784011850 con 0x7f97940d0640 2024-05-28T06:19:13.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.131+0000 7f978b400640 1 -- 172.21.15.96:0/2499900697 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9784011c40 con 0x7f97940d0640 2024-05-28T06:19:13.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.131+0000 7f979a200640 1 -- 172.21.15.96:0/2499900697 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f979415d210 con 0x7f97940d0640 2024-05-28T06:19:13.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.132+0000 7f979a200640 1 -- 172.21.15.96:0/2499900697 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f979415d7d0 con 0x7f97940d0640 2024-05-28T06:19:13.131 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.134+0000 7f979a200640 1 -- 172.21.15.96:0/2499900697 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97940689e0 con 0x7f97940d0640 2024-05-28T06:19:13.133 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.135+0000 7f978b400640 1 -- 172.21.15.96:0/2499900697 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 32) ==== 92774+0+0 (secure 0 0 0) 0x7f9784018020 con 0x7f97940d0640 2024-05-28T06:19:13.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.137+0000 7f978b400640 1 --2- 172.21.15.96:0/2499900697 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f977006dc10 0x7f97700700d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:13.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.137+0000 7f978b400640 1 -- 172.21.15.96:0/2499900697 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 5766+0+0 (secure 0 0 0) 0x7f978408aae0 con 0x7f97940d0640 2024-05-28T06:19:13.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.138+0000 7f9793400640 1 --2- 172.21.15.96:0/2499900697 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f977006dc10 0x7f97700700d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:13.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.139+0000 7f9793400640 1 --2- 172.21.15.96:0/2499900697 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f977006dc10 0x7f97700700d0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f977c034000 tx=0x7f977c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:13.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.142+0000 7f978b400640 1 -- 172.21.15.96:0/2499900697 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f978401f020 con 0x7f97940d0640 2024-05-28T06:19:13.475 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.477+0000 7f979a200640 1 -- 172.21.15.96:0/2499900697 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) -- 0x7f979415ed30 con 0x7f977006dc10 2024-05-28T06:19:13.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.541+0000 7f978b400640 1 -- 172.21.15.96:0/2499900697 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+275 (secure 0 0 0) 0x7f979415ed30 con 0x7f977006dc10 2024-05-28T06:19:13.541 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:19:13.541 INFO:teuthology.orchestra.run.smithi096.stdout: "status": { 2024-05-28T06:19:13.541 INFO:teuthology.orchestra.run.smithi096.stdout: "state": "in-progress", 2024-05-28T06:19:13.541 INFO:teuthology.orchestra.run.smithi096.stdout: "source": { 2024-05-28T06:19:13.541 INFO:teuthology.orchestra.run.smithi096.stdout: "volume": "cephfs", 2024-05-28T06:19:13.542 INFO:teuthology.orchestra.run.smithi096.stdout: "subvolume": "sv1", 2024-05-28T06:19:13.542 INFO:teuthology.orchestra.run.smithi096.stdout: "snapshot": "ss1" 2024-05-28T06:19:13.542 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:19:13.542 INFO:teuthology.orchestra.run.smithi096.stdout: "progress_report": { 2024-05-28T06:19:13.542 INFO:teuthology.orchestra.run.smithi096.stdout: "percentage cloned": 0.0, 2024-05-28T06:19:13.542 INFO:teuthology.orchestra.run.smithi096.stdout: "amount cloned": "184/3.0G", 2024-05-28T06:19:13.542 INFO:teuthology.orchestra.run.smithi096.stdout: "files cloned": "4/6" 2024-05-28T06:19:13.542 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:13.542 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:13.542 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:19:13.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.547+0000 7f979a200640 1 -- 172.21.15.96:0/2499900697 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f977006dc10 msgr2=0x7f97700700d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:13.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.547+0000 7f979a200640 1 --2- 172.21.15.96:0/2499900697 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f977006dc10 0x7f97700700d0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f977c034000 tx=0x7f977c02e000 comp rx=0 tx=0).stop 2024-05-28T06:19:13.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.547+0000 7f979a200640 1 -- 172.21.15.96:0/2499900697 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97940d0640 msgr2=0x7f979415c4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:13.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.547+0000 7f979a200640 1 --2- 172.21.15.96:0/2499900697 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97940d0640 0x7f979415c4e0 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7f978400b3e0 tx=0x7f978400b8a0 comp rx=0 tx=0).stop 2024-05-28T06:19:13.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.547+0000 7f979a200640 1 -- 172.21.15.96:0/2499900697 shutdown_connections 2024-05-28T06:19:13.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.547+0000 7f979a200640 1 --2- 172.21.15.96:0/2499900697 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f977006dc10 0x7f97700700d0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:13.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.547+0000 7f979a200640 1 --2- 172.21.15.96:0/2499900697 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f979415b6f0 0x7f97941f8b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:13.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.547+0000 7f979a200640 1 --2- 172.21.15.96:0/2499900697 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97940d0640 0x7f979415c4e0 secure :-1 s=CLOSED pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7f978400b3e0 tx=0x7f978400b8a0 comp rx=0 tx=0).stop 2024-05-28T06:19:13.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.547+0000 7f979a200640 1 --2- 172.21.15.96:0/2499900697 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f97940725e0 0x7f979414f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:13.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.547+0000 7f979a200640 1 -- 172.21.15.96:0/2499900697 >> 172.21.15.96:0/2499900697 conn(0x7f979406bd20 msgr2=0x7f979406f270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:19:13.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.550+0000 7f979a200640 1 -- 172.21.15.96:0/2499900697 shutdown_connections 2024-05-28T06:19:13.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.550+0000 7f979a200640 1 -- 172.21.15.96:0/2499900697 wait complete. 2024-05-28T06:19:13.571 INFO:tasks.cephfs.test_volumes:percentage cloned = 0.0 2024-05-28T06:19:13.571 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-05-28T06:19:13.851 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.853+0000 7fb2a2600640 1 Processor -- start 2024-05-28T06:19:13.852 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.853+0000 7fb2a2600640 1 -- start start 2024-05-28T06:19:13.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.855+0000 7fb2a2600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fb29c15e9c0 0x7fb29c0ba330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:13.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.855+0000 7fb2a2600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb29c0cf870 0x7fb29c0b88a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:13.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.855+0000 7fb2a2600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb29c0b9180 con 0x7fb29c0cf430 2024-05-28T06:19:13.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.855+0000 7fb2a2600640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fb29c0b9460 con 0x7fb29c15e9c0 2024-05-28T06:19:13.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.855+0000 7fb2a2600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fb29c0b9740 con 0x7fb29c0cf870 2024-05-28T06:19:13.857 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.856+0000 7fb29b400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fb29c15e9c0 0x7fb29c0ba330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:13.857 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.856+0000 7fb29b400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fb29c15e9c0 0x7fb29c0ba330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:36764/0 (socket says 172.21.15.96:36764) 2024-05-28T06:19:13.857 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.856+0000 7fb29b400640 1 -- 172.21.15.96:0/3536490754 learned_addr learned my addr 172.21.15.96:0/3536490754 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:19:13.857 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.856+0000 7fb29b400640 1 -- 172.21.15.96:0/3536490754 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb29c0cf870 msgr2=0x7fb29c0b88a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:19:13.857 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.856+0000 7fb29b400640 1 --2- 172.21.15.96:0/3536490754 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb29c0cf870 0x7fb29c0b88a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:13.857 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.856+0000 7fb29b400640 1 -- 172.21.15.96:0/3536490754 >> v1:172.21.15.123:6789/0 conn(0x7fb29c0cf430 legacy=0x7fb29c0d2ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:13.858 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.857+0000 7fb29b400640 1 -- 172.21.15.96:0/3536490754 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb29c0b9a20 con 0x7fb29c15e9c0 2024-05-28T06:19:13.858 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.858+0000 7fb29b400640 1 --2- 172.21.15.96:0/3536490754 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb29c15e9c0 0x7fb29c0ba330 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7fb28c009870 tx=0x7fb28c032af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=644a70c215398d32 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:13.858 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.859+0000 7fb29a000640 1 -- 172.21.15.96:0/3536490754 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb28c034850 con 0x7fb29c15e9c0 2024-05-28T06:19:13.858 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.859+0000 7fb29a000640 1 -- 172.21.15.96:0/3536490754 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb28c0349f0 con 0x7fb29c15e9c0 2024-05-28T06:19:13.858 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.859+0000 7fb29a000640 1 -- 172.21.15.96:0/3536490754 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb28c034dc0 con 0x7fb29c15e9c0 2024-05-28T06:19:13.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.860+0000 7fb2a2600640 1 -- 172.21.15.96:0/3536490754 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb29c15e9c0 msgr2=0x7fb29c0ba330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:13.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.860+0000 7fb2a2600640 1 --2- 172.21.15.96:0/3536490754 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb29c15e9c0 0x7fb29c0ba330 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7fb28c009870 tx=0x7fb28c032af0 comp rx=0 tx=0).stop 2024-05-28T06:19:13.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.860+0000 7fb2a2600640 1 -- 172.21.15.96:0/3536490754 shutdown_connections 2024-05-28T06:19:13.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.860+0000 7fb2a2600640 1 --2- 172.21.15.96:0/3536490754 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb29c0cf870 0x7fb29c0b88a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:13.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.860+0000 7fb2a2600640 1 --2- 172.21.15.96:0/3536490754 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb29c15e9c0 0x7fb29c0ba330 unknown :-1 s=CLOSED pgs=1466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:13.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.860+0000 7fb2a2600640 1 -- 172.21.15.96:0/3536490754 >> 172.21.15.96:0/3536490754 conn(0x7fb29c067aa0 msgr2=0x7fb29c067eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:19:13.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.861+0000 7fb2a2600640 1 -- 172.21.15.96:0/3536490754 shutdown_connections 2024-05-28T06:19:13.860 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.861+0000 7fb2a2600640 1 -- 172.21.15.96:0/3536490754 wait complete. 2024-05-28T06:19:13.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.863+0000 7fb2a2600640 1 Processor -- start 2024-05-28T06:19:13.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.864+0000 7fb2a2600640 1 -- start start 2024-05-28T06:19:13.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.865+0000 7fb2a2600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb29c0cf430 0x7fb29c155d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:13.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.865+0000 7fb2a2600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb29c0cf870 0x7fb29c0cb830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:13.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.865+0000 7fb2a2600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb29c15e9c0 0x7fb29c161490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:13.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.865+0000 7fb2a2600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb29c0b8f60 con 0x7fb29c0cf870 2024-05-28T06:19:13.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.865+0000 7fb2a2600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fb29c0b8de0 con 0x7fb29c0cf430 2024-05-28T06:19:13.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.865+0000 7fb2a2600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fb29c0d35b0 con 0x7fb29c15e9c0 2024-05-28T06:19:13.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.866+0000 7fb29b400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb29c0cf430 0x7fb29c155d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:13.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.866+0000 7fb29be00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb29c15e9c0 0x7fb29c161490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:13.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.866+0000 7fb29b400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb29c0cf430 0x7fb29c155d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.96:53038/0 (socket says 172.21.15.96:53038) 2024-05-28T06:19:13.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.866+0000 7fb29b400640 1 -- 172.21.15.96:0/886671681 learned_addr learned my addr 172.21.15.96:0/886671681 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:19:13.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.867+0000 7fb29aa00640 1 --2- 172.21.15.96:0/886671681 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb29c0cf870 0x7fb29c0cb830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:13.865 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.867+0000 7fb29b400640 1 -- 172.21.15.96:0/886671681 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb29c15e9c0 msgr2=0x7fb29c161490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:13.865 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.867+0000 7fb29b400640 1 --2- 172.21.15.96:0/886671681 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb29c15e9c0 0x7fb29c161490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:13.865 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.867+0000 7fb29b400640 1 -- 172.21.15.96:0/886671681 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb29c0cf870 msgr2=0x7fb29c0cb830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:13.865 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.867+0000 7fb29b400640 1 --2- 172.21.15.96:0/886671681 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb29c0cf870 0x7fb29c0cb830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:13.865 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.867+0000 7fb29b400640 1 -- 172.21.15.96:0/886671681 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb29c1603f0 con 0x7fb29c0cf430 2024-05-28T06:19:13.865 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.867+0000 7fb29be00640 1 --2- 172.21.15.96:0/886671681 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb29c15e9c0 0x7fb29c161490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-28T06:19:13.866 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.868+0000 7fb29b400640 1 --2- 172.21.15.96:0/886671681 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb29c0cf430 0x7fb29c155d10 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fb28c009990 tx=0x7fb28c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:13.866 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.868+0000 7fb293400640 1 -- 172.21.15.96:0/886671681 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb28c03ea40 con 0x7fb29c0cf430 2024-05-28T06:19:13.866 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.868+0000 7fb293400640 1 -- 172.21.15.96:0/886671681 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb28c04e070 con 0x7fb29c0cf430 2024-05-28T06:19:13.866 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.868+0000 7fb293400640 1 -- 172.21.15.96:0/886671681 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb28c03ee20 con 0x7fb29c0cf430 2024-05-28T06:19:13.866 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.869+0000 7fb2a2600640 1 -- 172.21.15.96:0/886671681 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb29c1606d0 con 0x7fb29c0cf430 2024-05-28T06:19:13.866 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.869+0000 7fb2a2600640 1 -- 172.21.15.96:0/886671681 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb29c160c90 con 0x7fb29c0cf430 2024-05-28T06:19:13.867 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.870+0000 7fb2a2600640 1 -- 172.21.15.96:0/886671681 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb29c0689e0 con 0x7fb29c0cf430 2024-05-28T06:19:13.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.873+0000 7fb293400640 1 -- 172.21.15.96:0/886671681 <== mon.1 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 32) ==== 92774+0+0 (secure 0 0 0) 0x7fb28c06ed10 con 0x7fb29c0cf430 2024-05-28T06:19:13.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.876+0000 7fb293400640 1 --2- 172.21.15.96:0/886671681 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb27c06dc10 0x7fb27c0700d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:13.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.876+0000 7fb293400640 1 -- 172.21.15.96:0/886671681 <== mon.1 v2:172.21.15.123:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 5766+0+0 (secure 0 0 0) 0x7fb28c0b3f10 con 0x7fb29c0cf430 2024-05-28T06:19:13.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.877+0000 7fb29aa00640 1 --2- 172.21.15.96:0/886671681 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb27c06dc10 0x7fb27c0700d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:13.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.881+0000 7fb293400640 1 -- 172.21.15.96:0/886671681 <== mon.1 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb28c057020 con 0x7fb29c0cf430 2024-05-28T06:19:13.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:13.886+0000 7fb29aa00640 1 --2- 172.21.15.96:0/886671681 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb27c06dc10 0x7fb27c0700d0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fb29c0d0340 tx=0x7fb288009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:14.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:14.150+0000 7fb293400640 1 -- 172.21.15.96:0/886671681 <== mon.1 v2:172.21.15.123:3300/0 7 ==== mgrmap(e 33) ==== 92825+0+0 (secure 0 0 0) 0x7fb28c047050 con 0x7fb29c0cf430 2024-05-28T06:19:14.195 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:14.197+0000 7fb2a2600640 1 -- 172.21.15.96:0/886671681 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) -- 0x7fb29c0cbd70 con 0x7fb27c06dc10 2024-05-28T06:19:14.279 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:19:14.279 INFO:teuthology.orchestra.run.smithi096.stdout: "status": { 2024-05-28T06:19:14.279 INFO:teuthology.orchestra.run.smithi096.stdout: "state": "in-progress", 2024-05-28T06:19:14.279 INFO:teuthology.orchestra.run.smithi096.stdout: "source": { 2024-05-28T06:19:14.279 INFO:teuthology.orchestra.run.smithi096.stdout: "volume": "cephfs", 2024-05-28T06:19:14.279 INFO:teuthology.orchestra.run.smithi096.stdout: "subvolume": "sv1", 2024-05-28T06:19:14.279 INFO:teuthology.orchestra.run.smithi096.stdout: "snapshot": "ss1" 2024-05-28T06:19:14.279 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:19:14.279 INFO:teuthology.orchestra.run.smithi096.stdout: "progress_report": { 2024-05-28T06:19:14.279 INFO:teuthology.orchestra.run.smithi096.stdout: "percentage cloned": 0.0, 2024-05-28T06:19:14.280 INFO:teuthology.orchestra.run.smithi096.stdout: "amount cloned": "184/3.0G", 2024-05-28T06:19:14.280 INFO:teuthology.orchestra.run.smithi096.stdout: "files cloned": "4/6" 2024-05-28T06:19:14.280 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:14.280 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:14.280 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:19:14.280 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:14.281+0000 7fb293400640 1 -- 172.21.15.96:0/886671681 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+275 (secure 0 0 0) 0x7fb29c0cbd70 con 0x7fb27c06dc10 2024-05-28T06:19:14.282 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:14.284+0000 7fb2a2600640 1 -- 172.21.15.96:0/886671681 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb27c06dc10 msgr2=0x7fb27c0700d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:14.282 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:14.285+0000 7fb2a2600640 1 --2- 172.21.15.96:0/886671681 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb27c06dc10 0x7fb27c0700d0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fb29c0d0340 tx=0x7fb288009040 comp rx=0 tx=0).stop 2024-05-28T06:19:14.282 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:14.285+0000 7fb2a2600640 1 -- 172.21.15.96:0/886671681 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb29c0cf430 msgr2=0x7fb29c155d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:14.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:14.285+0000 7fb2a2600640 1 --2- 172.21.15.96:0/886671681 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb29c0cf430 0x7fb29c155d10 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fb28c009990 tx=0x7fb28c0046c0 comp rx=0 tx=0).stop 2024-05-28T06:19:14.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:14.285+0000 7fb2a2600640 1 -- 172.21.15.96:0/886671681 shutdown_connections 2024-05-28T06:19:14.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:14.285+0000 7fb2a2600640 1 --2- 172.21.15.96:0/886671681 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb27c06dc10 0x7fb27c0700d0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:14.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:14.285+0000 7fb2a2600640 1 --2- 172.21.15.96:0/886671681 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb29c15e9c0 0x7fb29c161490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:14.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:14.285+0000 7fb2a2600640 1 --2- 172.21.15.96:0/886671681 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb29c0cf870 0x7fb29c0cb830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:14.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:14.285+0000 7fb2a2600640 1 --2- 172.21.15.96:0/886671681 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb29c0cf430 0x7fb29c155d10 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:14.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:14.285+0000 7fb2a2600640 1 -- 172.21.15.96:0/886671681 >> 172.21.15.96:0/886671681 conn(0x7fb29c067aa0 msgr2=0x7fb29c0b41a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:19:14.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:14.285+0000 7fb2a2600640 1 -- 172.21.15.96:0/886671681 shutdown_connections 2024-05-28T06:19:14.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:14.285+0000 7fb2a2600640 1 -- 172.21.15.96:0/886671681 wait complete. 2024-05-28T06:19:14.313 INFO:tasks.cephfs.test_volumes:percentage cloned = 0.0 2024-05-28T06:19:14.313 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-05-28T06:19:14.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:14.573+0000 7feaf2600640 1 Processor -- start 2024-05-28T06:19:14.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:14.574+0000 7feaf2600640 1 -- start start 2024-05-28T06:19:14.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:14.575+0000 7feaf2600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7feaec15d5d0 0x7feaec0cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:14.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:14.575+0000 7feaf2600640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feaec15cb10 con 0x7feaec0d0640 2024-05-28T06:19:14.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:14.575+0000 7feaf2600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feaec0ad5f0 con 0x7feaec0b32c0 2024-05-28T06:19:14.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:14.575+0000 7feaf2600640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7feaec0adbb0 con 0x7feaec15d5d0 2024-05-28T06:19:14.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:14.575+0000 7feae3e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7feaec15d5d0 0x7feaec0cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:14.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:14.575+0000 7feaeb400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7feaec0d0640 0x7feaec0d0a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:44952/0 (socket says 172.21.15.96:44952) 2024-05-28T06:19:14.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:14.575+0000 7feaeb400640 1 -- 172.21.15.96:0/1613072187 learned_addr learned my addr 172.21.15.96:0/1613072187 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:19:14.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:14.575+0000 7feae3e00640 1 -- 172.21.15.96:0/1613072187 >> v1:172.21.15.123:6789/0 conn(0x7feaec0b32c0 legacy=0x7feaec0b3700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:19:14.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:14.575+0000 7feae3e00640 1 -- 172.21.15.96:0/1613072187 >> v1:172.21.15.96:6789/0 conn(0x7feaec0d0640 legacy=0x7feaec0d0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:14.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:14.576+0000 7feae3e00640 1 -- 172.21.15.96:0/1613072187 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feaec0ad8d0 con 0x7feaec15d5d0 2024-05-28T06:19:14.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:14.577+0000 7feae3e00640 1 --2- 172.21.15.96:0/1613072187 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feaec15d5d0 0x7feaec0cf950 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7fead40097a0 tx=0x7fead402a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afb9b747d47b9441 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:14.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:14.577+0000 7feaeaa00640 1 -- 172.21.15.96:0/1613072187 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fead4037070 con 0x7feaec15d5d0 2024-05-28T06:19:14.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:14.577+0000 7feaeaa00640 1 -- 172.21.15.96:0/1613072187 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fead402ae50 con 0x7feaec15d5d0 2024-05-28T06:19:14.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:14.577+0000 7feaeaa00640 1 -- 172.21.15.96:0/1613072187 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fead4032660 con 0x7feaec15d5d0 2024-05-28T06:19:14.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:14.578+0000 7feaf2600640 1 -- 172.21.15.96:0/1613072187 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feaec15d5d0 msgr2=0x7feaec0cf950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:14.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:14.578+0000 7feaf2600640 1 --2- 172.21.15.96:0/1613072187 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feaec15d5d0 0x7feaec0cf950 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7fead40097a0 tx=0x7fead402a140 comp rx=0 tx=0).stop 2024-05-28T06:19:14.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:14.578+0000 7feaf2600640 1 -- 172.21.15.96:0/1613072187 shutdown_connections 2024-05-28T06:19:14.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:14.578+0000 7feaf2600640 1 --2- 172.21.15.96:0/1613072187 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feaec15d5d0 0x7feaec0cf950 unknown :-1 s=CLOSED pgs=1468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:14.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:14.578+0000 7feaf2600640 1 -- 172.21.15.96:0/1613072187 >> 172.21.15.96:0/1613072187 conn(0x7feaec06bd20 msgr2=0x7feaec06c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:19:14.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:14.578+0000 7feaf2600640 1 -- 172.21.15.96:0/1613072187 shutdown_connections 2024-05-28T06:19:14.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:14.578+0000 7feaf2600640 1 -- 172.21.15.96:0/1613072187 wait complete. 2024-05-28T06:19:14.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:14.579+0000 7feaf2600640 1 Processor -- start 2024-05-28T06:19:14.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:14.580+0000 7feaf2600640 1 -- start start 2024-05-28T06:19:14.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:14.581+0000 7feaf2600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feaec0b32c0 0x7feaec15ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:14.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:14.581+0000 7feaf2600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7feaec0d0640 0x7feaec14bd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:14.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:14.581+0000 7feaf2600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feaec15d5d0 0x7feaec14f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:14.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:14.581+0000 7feaf2600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7feaec15da50 con 0x7feaec15d5d0 2024-05-28T06:19:14.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:14.581+0000 7feaf2600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7feaec15d0b0 con 0x7feaec0b32c0 2024-05-28T06:19:14.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:14.581+0000 7feaf2600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7feaec0b3e10 con 0x7feaec0d0640 2024-05-28T06:19:14.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:14.581+0000 7feaebe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feaec15d5d0 0x7feaec14f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:14.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:14.581+0000 7feaebe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feaec15d5d0 0x7feaec14f1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:36792/0 (socket says 172.21.15.96:36792) 2024-05-28T06:19:14.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:14.581+0000 7feaebe00640 1 -- 172.21.15.96:0/3914621023 learned_addr learned my addr 172.21.15.96:0/3914621023 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:19:14.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:14.581+0000 7feaeb400640 1 --2- 172.21.15.96:0/3914621023 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feaec0b32c0 0x7feaec15ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:14.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:14.581+0000 7feae3e00640 1 --2- 172.21.15.96:0/3914621023 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7feaec0d0640 0x7feaec14bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:14.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:14.581+0000 7feaebe00640 1 -- 172.21.15.96:0/3914621023 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7feaec0d0640 msgr2=0x7feaec14bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:14.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:14.581+0000 7feaebe00640 1 --2- 172.21.15.96:0/3914621023 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7feaec0d0640 0x7feaec14bd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:14.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:14.581+0000 7feaebe00640 1 -- 172.21.15.96:0/3914621023 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feaec0b32c0 msgr2=0x7feaec15ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:14.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:14.581+0000 7feaebe00640 1 --2- 172.21.15.96:0/3914621023 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feaec0b32c0 0x7feaec15ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:14.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:14.581+0000 7feaebe00640 1 -- 172.21.15.96:0/3914621023 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feaec14cb30 con 0x7feaec15d5d0 2024-05-28T06:19:14.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:14.581+0000 7feae3e00640 1 --2- 172.21.15.96:0/3914621023 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7feaec0d0640 0x7feaec14bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:19:14.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:14.582+0000 7feaebe00640 1 --2- 172.21.15.96:0/3914621023 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feaec15d5d0 0x7feaec14f1f0 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7feae400c780 tx=0x7feae400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:14.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:14.582+0000 7feae3400640 1 -- 172.21.15.96:0/3914621023 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feae4004b60 con 0x7feaec15d5d0 2024-05-28T06:19:14.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:14.582+0000 7feae3400640 1 -- 172.21.15.96:0/3914621023 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feae4004d00 con 0x7feaec15d5d0 2024-05-28T06:19:14.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:14.582+0000 7feaf2600640 1 -- 172.21.15.96:0/3914621023 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feaec14c890 con 0x7feaec15d5d0 2024-05-28T06:19:14.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:14.582+0000 7feae3400640 1 -- 172.21.15.96:0/3914621023 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feae400aa90 con 0x7feaec15d5d0 2024-05-28T06:19:14.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:14.582+0000 7feaf2600640 1 -- 172.21.15.96:0/3914621023 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feaec0b2ac0 con 0x7feaec15d5d0 2024-05-28T06:19:14.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:14.584+0000 7feaf2600640 1 -- 172.21.15.96:0/3914621023 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feaec0b4490 con 0x7feaec15d5d0 2024-05-28T06:19:14.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:14.584+0000 7feae3400640 1 -- 172.21.15.96:0/3914621023 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 33) ==== 92825+0+0 (secure 0 0 0) 0x7feae400ac30 con 0x7feaec15d5d0 2024-05-28T06:19:14.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:14.588+0000 7feae3400640 1 --2- 172.21.15.96:0/3914621023 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7feac806dca0 0x7feac8070160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:14.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:14.588+0000 7feae3400640 1 -- 172.21.15.96:0/3914621023 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 5766+0+0 (secure 0 0 0) 0x7feae408aa40 con 0x7feaec15d5d0 2024-05-28T06:19:14.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:14.590+0000 7feaeb400640 1 --2- 172.21.15.96:0/3914621023 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7feac806dca0 0x7feac8070160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:14.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:14.592+0000 7feae3400640 1 -- 172.21.15.96:0/3914621023 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7feaec0b4490 con 0x7feaec15d5d0 2024-05-28T06:19:14.594 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:14.596+0000 7feaeb400640 1 --2- 172.21.15.96:0/3914621023 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7feac806dca0 0x7feac8070160 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7feadc000f60 tx=0x7feadc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:14.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:14.900+0000 7feaf2600640 1 -- 172.21.15.96:0/3914621023 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) -- 0x7feaec0b2ea0 con 0x7feac806dca0 2024-05-28T06:19:14.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:14.940+0000 7feae3400640 1 -- 172.21.15.96:0/3914621023 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+279 (secure 0 0 0) 0x7feaec0b2ea0 con 0x7feac806dca0 2024-05-28T06:19:14.938 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:19:14.938 INFO:teuthology.orchestra.run.smithi096.stdout: "status": { 2024-05-28T06:19:14.938 INFO:teuthology.orchestra.run.smithi096.stdout: "state": "in-progress", 2024-05-28T06:19:14.938 INFO:teuthology.orchestra.run.smithi096.stdout: "source": { 2024-05-28T06:19:14.938 INFO:teuthology.orchestra.run.smithi096.stdout: "volume": "cephfs", 2024-05-28T06:19:14.938 INFO:teuthology.orchestra.run.smithi096.stdout: "subvolume": "sv1", 2024-05-28T06:19:14.938 INFO:teuthology.orchestra.run.smithi096.stdout: "snapshot": "ss1" 2024-05-28T06:19:14.938 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:19:14.938 INFO:teuthology.orchestra.run.smithi096.stdout: "progress_report": { 2024-05-28T06:19:14.938 INFO:teuthology.orchestra.run.smithi096.stdout: "percentage cloned": 23.958, 2024-05-28T06:19:14.938 INFO:teuthology.orchestra.run.smithi096.stdout: "amount cloned": "736M/3.0G", 2024-05-28T06:19:14.939 INFO:teuthology.orchestra.run.smithi096.stdout: "files cloned": "4/6" 2024-05-28T06:19:14.939 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:14.939 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:14.939 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:19:14.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:14.943+0000 7feaf2600640 1 -- 172.21.15.96:0/3914621023 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7feac806dca0 msgr2=0x7feac8070160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:14.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:14.943+0000 7feaf2600640 1 --2- 172.21.15.96:0/3914621023 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7feac806dca0 0x7feac8070160 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7feadc000f60 tx=0x7feadc009040 comp rx=0 tx=0).stop 2024-05-28T06:19:14.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:14.943+0000 7feaf2600640 1 -- 172.21.15.96:0/3914621023 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feaec15d5d0 msgr2=0x7feaec14f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:14.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:14.943+0000 7feaf2600640 1 --2- 172.21.15.96:0/3914621023 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feaec15d5d0 0x7feaec14f1f0 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7feae400c780 tx=0x7feae400cc40 comp rx=0 tx=0).stop 2024-05-28T06:19:14.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:14.943+0000 7feaf2600640 1 -- 172.21.15.96:0/3914621023 shutdown_connections 2024-05-28T06:19:14.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:14.943+0000 7feaf2600640 1 --2- 172.21.15.96:0/3914621023 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7feac806dca0 0x7feac8070160 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:14.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:14.943+0000 7feaf2600640 1 --2- 172.21.15.96:0/3914621023 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feaec15d5d0 0x7feaec14f1f0 unknown :-1 s=CLOSED pgs=1469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:14.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:14.943+0000 7feaf2600640 1 --2- 172.21.15.96:0/3914621023 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7feaec0d0640 0x7feaec14bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:14.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:14.943+0000 7feaf2600640 1 --2- 172.21.15.96:0/3914621023 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feaec0b32c0 0x7feaec15ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:14.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:14.943+0000 7feaf2600640 1 -- 172.21.15.96:0/3914621023 >> 172.21.15.96:0/3914621023 conn(0x7feaec06bd20 msgr2=0x7feaec154420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:19:14.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:14.944+0000 7feaf2600640 1 -- 172.21.15.96:0/3914621023 shutdown_connections 2024-05-28T06:19:14.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:14.944+0000 7feaf2600640 1 -- 172.21.15.96:0/3914621023 wait complete. 2024-05-28T06:19:14.968 INFO:tasks.cephfs.test_volumes:percentage cloned = 23.958 2024-05-28T06:19:14.968 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-05-28T06:19:15.197 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.199+0000 7f04ca800640 1 Processor -- start 2024-05-28T06:19:15.197 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.199+0000 7f04ca800640 1 -- start start 2024-05-28T06:19:15.198 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.201+0000 7f04ca800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f04c40d0ac0 0x7f04c415d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:15.199 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.201+0000 7f04ca800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f04c40cd010 0x7f04c409aa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:15.199 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.201+0000 7f04ca800640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04c40cd4d0 con 0x7f04c415c9d0 2024-05-28T06:19:15.199 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.201+0000 7f04ca800640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f04c415dd30 con 0x7f04c40cd010 2024-05-28T06:19:15.199 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.201+0000 7f04ca800640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f04c40ce0e0 con 0x7f04c40d0ac0 2024-05-28T06:19:15.199 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.201+0000 7f04c3e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f04c40cd010 0x7f04c409aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:15.199 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.202+0000 7f04c3e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f04c40cd010 0x7f04c409aa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:36800/0 (socket says 172.21.15.96:36800) 2024-05-28T06:19:15.200 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.202+0000 7f04c3e00640 1 -- 172.21.15.96:0/3918906661 learned_addr learned my addr 172.21.15.96:0/3918906661 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:19:15.200 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.202+0000 7f04c3e00640 1 -- 172.21.15.96:0/3918906661 >> v2:172.21.15.123:3300/0 conn(0x7f04c40d0ac0 msgr2=0x7f04c415d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:19:15.200 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.202+0000 7f04c3e00640 1 --2- 172.21.15.96:0/3918906661 >> v2:172.21.15.123:3300/0 conn(0x7f04c40d0ac0 0x7f04c415d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:15.200 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.202+0000 7f04c3e00640 1 -- 172.21.15.96:0/3918906661 >> v1:172.21.15.123:6789/0 conn(0x7f04c415c9d0 legacy=0x7f04c40d0ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:19:15.200 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.202+0000 7f04c3e00640 1 -- 172.21.15.96:0/3918906661 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04c40cd7b0 con 0x7f04c40cd010 2024-05-28T06:19:15.201 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.203+0000 7f04c3e00640 1 --2- 172.21.15.96:0/3918906661 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f04c40cd010 0x7f04c409aa30 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f04bc009e50 tx=0x7f04bc02a7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ed7e4cde47ffc06 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:15.201 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.204+0000 7f04c2a00640 1 -- 172.21.15.96:0/3918906661 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04bc036070 con 0x7f04c40cd010 2024-05-28T06:19:15.201 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.204+0000 7f04c2a00640 1 -- 172.21.15.96:0/3918906661 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04bc031600 con 0x7f04c40cd010 2024-05-28T06:19:15.201 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.204+0000 7f04c2a00640 1 -- 172.21.15.96:0/3918906661 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04bc0319d0 con 0x7f04c40cd010 2024-05-28T06:19:15.202 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.204+0000 7f04ca800640 1 -- 172.21.15.96:0/3918906661 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f04c40cd010 msgr2=0x7f04c409aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:15.202 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.204+0000 7f04ca800640 1 --2- 172.21.15.96:0/3918906661 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f04c40cd010 0x7f04c409aa30 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f04bc009e50 tx=0x7f04bc02a7a0 comp rx=0 tx=0).stop 2024-05-28T06:19:15.202 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.204+0000 7f04ca800640 1 -- 172.21.15.96:0/3918906661 shutdown_connections 2024-05-28T06:19:15.202 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.204+0000 7f04ca800640 1 --2- 172.21.15.96:0/3918906661 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f04c40cd010 0x7f04c409aa30 unknown :-1 s=CLOSED pgs=1470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:15.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.204+0000 7f04ca800640 1 --2- 172.21.15.96:0/3918906661 >> v2:172.21.15.123:3300/0 conn(0x7f04c40d0ac0 0x7f04c415d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:15.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.204+0000 7f04ca800640 1 -- 172.21.15.96:0/3918906661 >> 172.21.15.96:0/3918906661 conn(0x7f04c40b3ff0 msgr2=0x7f04c40b4400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:19:15.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.205+0000 7f04ca800640 1 -- 172.21.15.96:0/3918906661 shutdown_connections 2024-05-28T06:19:15.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.205+0000 7f04ca800640 1 -- 172.21.15.96:0/3918906661 wait complete. 2024-05-28T06:19:15.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.206+0000 7f04ca800640 1 Processor -- start 2024-05-28T06:19:15.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.207+0000 7f04ca800640 1 -- start start 2024-05-28T06:19:15.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.208+0000 7f04ca800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f04c40cd010 0x7f04c41f8d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:15.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.208+0000 7f04ca800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f04c40d0ac0 0x7f04c41f7ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:15.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.208+0000 7f04ca800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f04c415c9d0 0x7f04c40b2490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:15.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.208+0000 7f04ca800640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f04c40d0280 con 0x7f04c40cd010 2024-05-28T06:19:15.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.208+0000 7f04ca800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f04c40cdd70 con 0x7f04c415c9d0 2024-05-28T06:19:15.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.208+0000 7f04ca800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f04c40d0400 con 0x7f04c40d0ac0 2024-05-28T06:19:15.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.209+0000 7f04c3400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f04c40cd010 0x7f04c41f8d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:15.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.209+0000 7f04c3400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f04c40cd010 0x7f04c41f8d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:36810/0 (socket says 172.21.15.96:36810) 2024-05-28T06:19:15.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.209+0000 7f04c3400640 1 -- 172.21.15.96:0/3428841519 learned_addr learned my addr 172.21.15.96:0/3428841519 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:19:15.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.209+0000 7f04c3e00640 1 --2- 172.21.15.96:0/3428841519 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f04c415c9d0 0x7f04c40b2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:15.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.209+0000 7f04bbe00640 1 --2- 172.21.15.96:0/3428841519 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f04c40d0ac0 0x7f04c41f7ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:15.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.209+0000 7f04c3400640 1 -- 172.21.15.96:0/3428841519 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f04c40d0ac0 msgr2=0x7f04c41f7ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:15.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.209+0000 7f04c3400640 1 --2- 172.21.15.96:0/3428841519 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f04c40d0ac0 0x7f04c41f7ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:15.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.209+0000 7f04c3400640 1 -- 172.21.15.96:0/3428841519 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f04c415c9d0 msgr2=0x7f04c40b2490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:15.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.209+0000 7f04c3400640 1 --2- 172.21.15.96:0/3428841519 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f04c415c9d0 0x7f04c40b2490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:15.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.209+0000 7f04c3400640 1 -- 172.21.15.96:0/3428841519 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04c40c6700 con 0x7f04c40cd010 2024-05-28T06:19:15.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.209+0000 7f04bbe00640 1 --2- 172.21.15.96:0/3428841519 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f04c40d0ac0 0x7f04c41f7ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:19:15.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.210+0000 7f04c3400640 1 --2- 172.21.15.96:0/3428841519 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f04c40cd010 0x7f04c41f8d50 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7f04b400b4a0 tx=0x7f04b400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:15.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.210+0000 7f04bb400640 1 -- 172.21.15.96:0/3428841519 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04b4014070 con 0x7f04c40cd010 2024-05-28T06:19:15.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.210+0000 7f04bb400640 1 -- 172.21.15.96:0/3428841519 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04b4010610 con 0x7f04c40cd010 2024-05-28T06:19:15.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.210+0000 7f04bb400640 1 -- 172.21.15.96:0/3428841519 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04b4010a00 con 0x7f04c40cd010 2024-05-28T06:19:15.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.210+0000 7f04ca800640 1 -- 172.21.15.96:0/3428841519 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04c40c69e0 con 0x7f04c40cd010 2024-05-28T06:19:15.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.210+0000 7f04ca800640 1 -- 172.21.15.96:0/3428841519 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04c40c6fa0 con 0x7f04c40cd010 2024-05-28T06:19:15.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.212+0000 7f04ca800640 1 -- 172.21.15.96:0/3428841519 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04c40690f0 con 0x7f04c40cd010 2024-05-28T06:19:15.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.215+0000 7f04bb400640 1 -- 172.21.15.96:0/3428841519 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 33) ==== 92825+0+0 (secure 0 0 0) 0x7f04b4017070 con 0x7f04c40cd010 2024-05-28T06:19:15.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.217+0000 7f04bb400640 1 --2- 172.21.15.96:0/3428841519 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f04a006dcf0 0x7f04a00701b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:15.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.217+0000 7f04bb400640 1 -- 172.21.15.96:0/3428841519 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 5766+0+0 (secure 0 0 0) 0x7f04b408a580 con 0x7f04c40cd010 2024-05-28T06:19:15.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.220+0000 7f04bbe00640 1 --2- 172.21.15.96:0/3428841519 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f04a006dcf0 0x7f04a00701b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:15.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.224+0000 7f04bb400640 1 -- 172.21.15.96:0/3428841519 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f04b4059a00 con 0x7f04c40cd010 2024-05-28T06:19:15.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.244+0000 7f04bbe00640 1 --2- 172.21.15.96:0/3428841519 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f04a006dcf0 0x7f04a00701b0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f04c40b7d80 tx=0x7f04ac008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:15.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.578+0000 7f04ca800640 1 -- 172.21.15.96:0/3428841519 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) -- 0x7f04c40ce910 con 0x7f04a006dcf0 2024-05-28T06:19:15.617 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:19:15.617 INFO:teuthology.orchestra.run.smithi096.stdout: "status": { 2024-05-28T06:19:15.618 INFO:teuthology.orchestra.run.smithi096.stdout: "state": "in-progress", 2024-05-28T06:19:15.618 INFO:teuthology.orchestra.run.smithi096.stdout: "source": { 2024-05-28T06:19:15.618 INFO:teuthology.orchestra.run.smithi096.stdout: "volume": "cephfs", 2024-05-28T06:19:15.618 INFO:teuthology.orchestra.run.smithi096.stdout: "subvolume": "sv1", 2024-05-28T06:19:15.618 INFO:teuthology.orchestra.run.smithi096.stdout: "snapshot": "ss1" 2024-05-28T06:19:15.618 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:19:15.618 INFO:teuthology.orchestra.run.smithi096.stdout: "progress_report": { 2024-05-28T06:19:15.618 INFO:teuthology.orchestra.run.smithi096.stdout: "percentage cloned": 42.188, 2024-05-28T06:19:15.618 INFO:teuthology.orchestra.run.smithi096.stdout: "amount cloned": "1.2G/3.0G", 2024-05-28T06:19:15.618 INFO:teuthology.orchestra.run.smithi096.stdout: "files cloned": "5/6" 2024-05-28T06:19:15.618 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:15.618 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:15.619 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:19:15.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.619+0000 7f04bb400640 1 -- 172.21.15.96:0/3428841519 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+279 (secure 0 0 0) 0x7f04c40ce910 con 0x7f04a006dcf0 2024-05-28T06:19:15.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.622+0000 7f04ca800640 1 -- 172.21.15.96:0/3428841519 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f04a006dcf0 msgr2=0x7f04a00701b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:15.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.622+0000 7f04ca800640 1 --2- 172.21.15.96:0/3428841519 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f04a006dcf0 0x7f04a00701b0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f04c40b7d80 tx=0x7f04ac008040 comp rx=0 tx=0).stop 2024-05-28T06:19:15.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.622+0000 7f04ca800640 1 -- 172.21.15.96:0/3428841519 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f04c40cd010 msgr2=0x7f04c41f8d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:15.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.622+0000 7f04ca800640 1 --2- 172.21.15.96:0/3428841519 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f04c40cd010 0x7f04c41f8d50 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7f04b400b4a0 tx=0x7f04b400b960 comp rx=0 tx=0).stop 2024-05-28T06:19:15.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.622+0000 7f04ca800640 1 -- 172.21.15.96:0/3428841519 shutdown_connections 2024-05-28T06:19:15.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.622+0000 7f04ca800640 1 --2- 172.21.15.96:0/3428841519 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f04a006dcf0 0x7f04a00701b0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:15.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.622+0000 7f04ca800640 1 --2- 172.21.15.96:0/3428841519 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f04c415c9d0 0x7f04c40b2490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:15.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.622+0000 7f04ca800640 1 --2- 172.21.15.96:0/3428841519 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f04c40d0ac0 0x7f04c41f7ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:15.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.622+0000 7f04ca800640 1 --2- 172.21.15.96:0/3428841519 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f04c40cd010 0x7f04c41f8d50 unknown :-1 s=CLOSED pgs=1471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:15.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.622+0000 7f04ca800640 1 -- 172.21.15.96:0/3428841519 >> 172.21.15.96:0/3428841519 conn(0x7f04c40b3ff0 msgr2=0x7f04c40c4f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:19:15.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.623+0000 7f04ca800640 1 -- 172.21.15.96:0/3428841519 shutdown_connections 2024-05-28T06:19:15.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.623+0000 7f04ca800640 1 -- 172.21.15.96:0/3428841519 wait complete. 2024-05-28T06:19:15.650 INFO:tasks.cephfs.test_volumes:percentage cloned = 42.188 2024-05-28T06:19:15.650 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-05-28T06:19:15.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.895+0000 7f7fed000640 1 Processor -- start 2024-05-28T06:19:15.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.895+0000 7f7fed000640 1 -- start start 2024-05-28T06:19:15.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.896+0000 7f7fed000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f7fe815ed80 0x7f7fe80cff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:15.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.896+0000 7f7fed000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f7fe80b7e50 0x7f7fe80cedb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:15.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.897+0000 7f7fed000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7fe80b61e0 0x7f7fe80b6650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:15.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.897+0000 7f7fed000640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f7fe80b6b90 con 0x7f7fe815ed80 2024-05-28T06:19:15.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.897+0000 7f7fed000640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f7fe8163540 con 0x7f7fe80b7e50 2024-05-28T06:19:15.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.897+0000 7f7fed000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f7fe815f160 con 0x7f7fe80b61e0 2024-05-28T06:19:15.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.897+0000 7f7fe5e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f7fe815ed80 0x7f7fe80cff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:15.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.897+0000 7f7fe5e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f7fe815ed80 0x7f7fe80cff80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:36812/0 (socket says 172.21.15.96:36812) 2024-05-28T06:19:15.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.897+0000 7f7fe5e00640 1 -- 172.21.15.96:0/2242963364 learned_addr learned my addr 172.21.15.96:0/2242963364 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:19:15.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.897+0000 7f7fe6800640 1 --2- 172.21.15.96:0/2242963364 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7fe80b61e0 0x7f7fe80b6650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:15.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.897+0000 7f7fe5e00640 1 -- 172.21.15.96:0/2242963364 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7fe80b61e0 msgr2=0x7f7fe80b6650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:19:15.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.897+0000 7f7fe5e00640 1 --2- 172.21.15.96:0/2242963364 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7fe80b61e0 0x7f7fe80b6650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:15.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.897+0000 7f7fe5e00640 1 -- 172.21.15.96:0/2242963364 >> v2:172.21.15.123:3300/0 conn(0x7f7fe80b7e50 msgr2=0x7f7fe80cedb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:19:15.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.898+0000 7f7fe5e00640 1 --2- 172.21.15.96:0/2242963364 >> v2:172.21.15.123:3300/0 conn(0x7f7fe80b7e50 0x7f7fe80cedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:15.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.898+0000 7f7fe5e00640 1 -- 172.21.15.96:0/2242963364 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fe8162930 con 0x7f7fe815ed80 2024-05-28T06:19:15.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.898+0000 7f7fe6800640 1 --2- 172.21.15.96:0/2242963364 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7fe80b61e0 0x7f7fe80b6650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:19:15.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.899+0000 7f7fe5e00640 1 --2- 172.21.15.96:0/2242963364 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7fe815ed80 0x7f7fe80cff80 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7f7fe808ce10 tx=0x7f7fd802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4c1b44cdf455274 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:15.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.899+0000 7f7fe4a00640 1 -- 172.21.15.96:0/2242963364 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fd8037070 con 0x7f7fe815ed80 2024-05-28T06:19:15.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.899+0000 7f7fe4a00640 1 -- 172.21.15.96:0/2242963364 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7fd8032440 con 0x7f7fe815ed80 2024-05-28T06:19:15.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.899+0000 7f7fe4a00640 1 -- 172.21.15.96:0/2242963364 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fd8032810 con 0x7f7fe815ed80 2024-05-28T06:19:15.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.900+0000 7f7fed000640 1 -- 172.21.15.96:0/2242963364 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7fe815ed80 msgr2=0x7f7fe80cff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:15.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.900+0000 7f7fed000640 1 --2- 172.21.15.96:0/2242963364 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7fe815ed80 0x7f7fe80cff80 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7f7fe808ce10 tx=0x7f7fd802a1e0 comp rx=0 tx=0).stop 2024-05-28T06:19:15.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.900+0000 7f7fed000640 1 -- 172.21.15.96:0/2242963364 shutdown_connections 2024-05-28T06:19:15.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.900+0000 7f7fed000640 1 --2- 172.21.15.96:0/2242963364 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7fe80b61e0 0x7f7fe80b6650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:15.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.900+0000 7f7fed000640 1 --2- 172.21.15.96:0/2242963364 >> v2:172.21.15.123:3300/0 conn(0x7f7fe80b7e50 0x7f7fe80cedb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:15.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.900+0000 7f7fed000640 1 --2- 172.21.15.96:0/2242963364 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7fe815ed80 0x7f7fe80cff80 unknown :-1 s=CLOSED pgs=1472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:15.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.900+0000 7f7fed000640 1 -- 172.21.15.96:0/2242963364 >> 172.21.15.96:0/2242963364 conn(0x7f7fe80d13c0 msgr2=0x7f7fe80d17d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:19:15.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.900+0000 7f7fed000640 1 -- 172.21.15.96:0/2242963364 shutdown_connections 2024-05-28T06:19:15.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.900+0000 7f7fed000640 1 -- 172.21.15.96:0/2242963364 wait complete. 2024-05-28T06:19:15.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.902+0000 7f7fed000640 1 Processor -- start 2024-05-28T06:19:15.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.902+0000 7f7fed000640 1 -- start start 2024-05-28T06:19:15.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.903+0000 7f7fed000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7fe80b61e0 0x7f7fe8161720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:15.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.903+0000 7f7fed000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7fe80b7e50 0x7f7fe81603e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:15.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.903+0000 7f7fed000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7fe815ed80 0x7f7fe8160920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:15.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.903+0000 7f7fed000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f7fe81631d0 con 0x7f7fe815ed80 2024-05-28T06:19:15.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.903+0000 7f7fed000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f7fe8162ef0 con 0x7f7fe80b7e50 2024-05-28T06:19:15.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.903+0000 7f7fed000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f7fe80d2b30 con 0x7f7fe80b61e0 2024-05-28T06:19:15.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.904+0000 7f7fe6800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7fe815ed80 0x7f7fe8160920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:15.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.904+0000 7f7fe6800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7fe815ed80 0x7f7fe8160920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:36818/0 (socket says 172.21.15.96:36818) 2024-05-28T06:19:15.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.904+0000 7f7fe6800640 1 -- 172.21.15.96:0/3754662737 learned_addr learned my addr 172.21.15.96:0/3754662737 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:19:15.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.904+0000 7f7fe5400640 1 --2- 172.21.15.96:0/3754662737 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7fe80b7e50 0x7f7fe81603e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:15.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.905+0000 7f7fe6800640 1 -- 172.21.15.96:0/3754662737 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7fe80b61e0 msgr2=0x7f7fe8161720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-28T06:19:15.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.905+0000 7f7fe6800640 1 --2- 172.21.15.96:0/3754662737 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7fe80b61e0 0x7f7fe8161720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:15.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.905+0000 7f7fe6800640 1 -- 172.21.15.96:0/3754662737 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7fe80b7e50 msgr2=0x7f7fe81603e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:15.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.905+0000 7f7fe6800640 1 --2- 172.21.15.96:0/3754662737 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7fe80b7e50 0x7f7fe81603e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:15.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.905+0000 7f7fe6800640 1 -- 172.21.15.96:0/3754662737 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fe80cbf80 con 0x7f7fe815ed80 2024-05-28T06:19:15.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.905+0000 7f7fe5400640 1 --2- 172.21.15.96:0/3754662737 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7fe80b7e50 0x7f7fe81603e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-28T06:19:15.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.905+0000 7f7fe6800640 1 --2- 172.21.15.96:0/3754662737 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7fe815ed80 0x7f7fe8160920 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7f7fe000cef0 tx=0x7f7fe000a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:15.904 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.906+0000 7f7fde000640 1 -- 172.21.15.96:0/3754662737 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fe001a070 con 0x7f7fe815ed80 2024-05-28T06:19:15.904 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.906+0000 7f7fed000640 1 -- 172.21.15.96:0/3754662737 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fe80cbca0 con 0x7f7fe815ed80 2024-05-28T06:19:15.904 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.906+0000 7f7fed000640 1 -- 172.21.15.96:0/3754662737 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fe80c8900 con 0x7f7fe815ed80 2024-05-28T06:19:15.904 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.906+0000 7f7fde000640 1 -- 172.21.15.96:0/3754662737 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7fe000d860 con 0x7f7fe815ed80 2024-05-28T06:19:15.904 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.906+0000 7f7fde000640 1 -- 172.21.15.96:0/3754662737 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fe000dc50 con 0x7f7fe815ed80 2024-05-28T06:19:15.906 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.909+0000 7f7fed000640 1 -- 172.21.15.96:0/3754662737 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7fc0005290 con 0x7f7fe815ed80 2024-05-28T06:19:15.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.909+0000 7f7fde000640 1 -- 172.21.15.96:0/3754662737 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 33) ==== 92825+0+0 (secure 0 0 0) 0x7f7fe001d070 con 0x7f7fe815ed80 2024-05-28T06:19:15.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.911+0000 7f7fde000640 1 --2- 172.21.15.96:0/3754662737 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7fc806dca0 0x7f7fc8070160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:15.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.911+0000 7f7fde000640 1 -- 172.21.15.96:0/3754662737 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 5766+0+0 (secure 0 0 0) 0x7f7fe008b3e0 con 0x7f7fe815ed80 2024-05-28T06:19:15.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.912+0000 7f7fe5e00640 1 --2- 172.21.15.96:0/3754662737 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7fc806dca0 0x7f7fc8070160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:15.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.913+0000 7f7fe5e00640 1 --2- 172.21.15.96:0/3754662737 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7fc806dca0 0x7f7fc8070160 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f7fd8034000 tx=0x7f7fd802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:15.917 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:15.920+0000 7f7fde000640 1 -- 172.21.15.96:0/3754662737 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7fe005af70 con 0x7f7fe815ed80 2024-05-28T06:19:16.266 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.268+0000 7f7fed000640 1 -- 172.21.15.96:0/3754662737 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) -- 0x7f7fc0000bc0 con 0x7f7fc806dca0 2024-05-28T06:19:16.303 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.305+0000 7f7fde000640 1 -- 172.21.15.96:0/3754662737 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+279 (secure 0 0 0) 0x7f7fc0000bc0 con 0x7f7fc806dca0 2024-05-28T06:19:16.304 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:19:16.304 INFO:teuthology.orchestra.run.smithi096.stdout: "status": { 2024-05-28T06:19:16.304 INFO:teuthology.orchestra.run.smithi096.stdout: "state": "in-progress", 2024-05-28T06:19:16.304 INFO:teuthology.orchestra.run.smithi096.stdout: "source": { 2024-05-28T06:19:16.304 INFO:teuthology.orchestra.run.smithi096.stdout: "volume": "cephfs", 2024-05-28T06:19:16.304 INFO:teuthology.orchestra.run.smithi096.stdout: "subvolume": "sv1", 2024-05-28T06:19:16.304 INFO:teuthology.orchestra.run.smithi096.stdout: "snapshot": "ss1" 2024-05-28T06:19:16.304 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:19:16.304 INFO:teuthology.orchestra.run.smithi096.stdout: "progress_report": { 2024-05-28T06:19:16.304 INFO:teuthology.orchestra.run.smithi096.stdout: "percentage cloned": 42.188, 2024-05-28T06:19:16.304 INFO:teuthology.orchestra.run.smithi096.stdout: "amount cloned": "1.2G/3.0G", 2024-05-28T06:19:16.305 INFO:teuthology.orchestra.run.smithi096.stdout: "files cloned": "5/6" 2024-05-28T06:19:16.305 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:16.305 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:16.305 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:19:16.307 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.309+0000 7f7fed000640 1 -- 172.21.15.96:0/3754662737 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7fc806dca0 msgr2=0x7f7fc8070160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:16.307 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.309+0000 7f7fed000640 1 --2- 172.21.15.96:0/3754662737 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7fc806dca0 0x7f7fc8070160 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f7fd8034000 tx=0x7f7fd802e000 comp rx=0 tx=0).stop 2024-05-28T06:19:16.307 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.309+0000 7f7fed000640 1 -- 172.21.15.96:0/3754662737 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7fe815ed80 msgr2=0x7f7fe8160920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:16.307 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.309+0000 7f7fed000640 1 --2- 172.21.15.96:0/3754662737 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7fe815ed80 0x7f7fe8160920 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7f7fe000cef0 tx=0x7f7fe000a710 comp rx=0 tx=0).stop 2024-05-28T06:19:16.308 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.311+0000 7f7fed000640 1 -- 172.21.15.96:0/3754662737 shutdown_connections 2024-05-28T06:19:16.308 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.311+0000 7f7fed000640 1 --2- 172.21.15.96:0/3754662737 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7fc806dca0 0x7f7fc8070160 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:16.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.311+0000 7f7fed000640 1 --2- 172.21.15.96:0/3754662737 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7fe815ed80 0x7f7fe8160920 unknown :-1 s=CLOSED pgs=1473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:16.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.311+0000 7f7fed000640 1 --2- 172.21.15.96:0/3754662737 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7fe80b7e50 0x7f7fe81603e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:16.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.311+0000 7f7fed000640 1 --2- 172.21.15.96:0/3754662737 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7fe80b61e0 0x7f7fe8161720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:16.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.311+0000 7f7fed000640 1 -- 172.21.15.96:0/3754662737 >> 172.21.15.96:0/3754662737 conn(0x7f7fe80d13c0 msgr2=0x7f7fe81596e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:19:16.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.311+0000 7f7fed000640 1 -- 172.21.15.96:0/3754662737 shutdown_connections 2024-05-28T06:19:16.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.311+0000 7f7fed000640 1 -- 172.21.15.96:0/3754662737 wait complete. 2024-05-28T06:19:16.336 INFO:tasks.cephfs.test_volumes:percentage cloned = 42.188 2024-05-28T06:19:16.337 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-05-28T06:19:16.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.609+0000 7f55e0400640 1 Processor -- start 2024-05-28T06:19:16.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.609+0000 7f55e0400640 1 -- start start 2024-05-28T06:19:16.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.610+0000 7f55e0400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f55d8162130 0x7f55d80d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:16.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.610+0000 7f55e0400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f55d80d15e0 0x7f55d80b1e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:16.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.610+0000 7f55e0400640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55d80b13d0 con 0x7f55d80b2380 2024-05-28T06:19:16.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.610+0000 7f55e0400640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f55d80d1a40 con 0x7f55d8162130 2024-05-28T06:19:16.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.610+0000 7f55e0400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f55d80d33b0 con 0x7f55d80d15e0 2024-05-28T06:19:16.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.611+0000 7f55de400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f55d80b2380 0x7f55d80b2810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:44954/0 (socket says 172.21.15.96:44954) 2024-05-28T06:19:16.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.611+0000 7f55de400640 1 -- 172.21.15.96:0/3268835126 learned_addr learned my addr 172.21.15.96:0/3268835126 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:19:16.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.611+0000 7f55dda00640 1 --2- 172.21.15.96:0/3268835126 >> v2:172.21.15.123:3300/0 conn(0x7f55d8162130 0x7f55d80d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:16.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.611+0000 7f55dd000640 1 --2- 172.21.15.96:0/3268835126 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f55d80d15e0 0x7f55d80b1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:16.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.611+0000 7f55d7e00640 1 -- 172.21.15.96:0/3268835126 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1510846201 0 0) 0x7f55d80b13d0 con 0x7f55d80b2380 2024-05-28T06:19:16.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.611+0000 7f55d7e00640 1 -- 172.21.15.96:0/3268835126 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f55c0003520 con 0x7f55d80b2380 2024-05-28T06:19:16.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.611+0000 7f55d7e00640 1 -- 172.21.15.96:0/3268835126 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2159704112 0 0) 0x7f55c0003520 con 0x7f55d80b2380 2024-05-28T06:19:16.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.611+0000 7f55d7e00640 1 -- 172.21.15.96:0/3268835126 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f55d80d15e0 msgr2=0x7f55d80b1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:19:16.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.611+0000 7f55d7e00640 1 --2- 172.21.15.96:0/3268835126 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f55d80d15e0 0x7f55d80b1e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:16.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.612+0000 7f55d7e00640 1 -- 172.21.15.96:0/3268835126 >> v2:172.21.15.123:3300/0 conn(0x7f55d8162130 msgr2=0x7f55d80d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:19:16.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.612+0000 7f55d7e00640 1 --2- 172.21.15.96:0/3268835126 >> v2:172.21.15.123:3300/0 conn(0x7f55d8162130 0x7f55d80d10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:16.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.612+0000 7f55d7e00640 1 -- 172.21.15.96:0/3268835126 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55d80b1640 con 0x7f55d80b2380 2024-05-28T06:19:16.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.612+0000 7f55d7e00640 1 -- 172.21.15.96:0/3268835126 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f55d00019f0 con 0x7f55d80b2380 2024-05-28T06:19:16.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.612+0000 7f55d7e00640 1 -- 172.21.15.96:0/3268835126 >> v1:172.21.15.96:6789/0 conn(0x7f55d80b2380 legacy=0x7f55d80b2810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:16.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.612+0000 7f55d7e00640 1 --2- 172.21.15.96:0/3268835126 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f55c00039b0 0x7f55c0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:16.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.612+0000 7f55d7e00640 1 --2- 172.21.15.96:0/3268835126 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f55c0004800 0x7f55c0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:16.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.612+0000 7f55d7e00640 1 --2- 172.21.15.96:0/3268835126 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f55c00051c0 0x7f55c000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:16.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.612+0000 7f55d7e00640 1 -- 172.21.15.96:0/3268835126 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f55d80d33b0 con 0x7f55c0004800 2024-05-28T06:19:16.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.612+0000 7f55d7e00640 1 -- 172.21.15.96:0/3268835126 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f55c000bf00 con 0x7f55c00051c0 2024-05-28T06:19:16.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.612+0000 7f55d7e00640 1 -- 172.21.15.96:0/3268835126 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f55c000c080 con 0x7f55c00039b0 2024-05-28T06:19:16.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.612+0000 7f55dda00640 1 --2- 172.21.15.96:0/3268835126 >> v2:172.21.15.123:3300/0 conn(0x7f55d8162130 0x7f55d80d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-28T06:19:16.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.612+0000 7f55dd000640 1 --2- 172.21.15.96:0/3268835126 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f55d80d15e0 0x7f55d80b1e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-28T06:19:16.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.612+0000 7f55de400640 1 --2- 172.21.15.96:0/3268835126 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f55c00051c0 0x7f55c000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:16.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.612+0000 7f55dd000640 1 --2- 172.21.15.96:0/3268835126 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f55c0004800 0x7f55c0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:16.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.612+0000 7f55dda00640 1 --2- 172.21.15.96:0/3268835126 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f55c00039b0 0x7f55c0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:16.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.613+0000 7f55dd000640 1 -- 172.21.15.96:0/3268835126 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f55c00039b0 msgr2=0x7f55c0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:19:16.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.613+0000 7f55dd000640 1 --2- 172.21.15.96:0/3268835126 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f55c00039b0 0x7f55c0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:16.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.613+0000 7f55dd000640 1 -- 172.21.15.96:0/3268835126 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f55c00051c0 msgr2=0x7f55c000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:19:16.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.613+0000 7f55dd000640 1 --2- 172.21.15.96:0/3268835126 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f55c00051c0 0x7f55c000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:16.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.613+0000 7f55dd000640 1 -- 172.21.15.96:0/3268835126 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55c0003520 con 0x7f55c0004800 2024-05-28T06:19:16.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.614+0000 7f55dd000640 1 --2- 172.21.15.96:0/3268835126 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f55c0004800 0x7f55c0004c80 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7f55c4009b00 tx=0x7f55c0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5f4ca833c664a49 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:16.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.614+0000 7f55d7e00640 1 -- 172.21.15.96:0/3268835126 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55c4036070 con 0x7f55c0004800 2024-05-28T06:19:16.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.614+0000 7f55d7e00640 1 -- 172.21.15.96:0/3268835126 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55c4004030 con 0x7f55c0004800 2024-05-28T06:19:16.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.614+0000 7f55d7e00640 1 -- 172.21.15.96:0/3268835126 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55c40097d0 con 0x7f55c0004800 2024-05-28T06:19:16.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.615+0000 7f55e0400640 1 -- 172.21.15.96:0/3268835126 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f55c0004800 msgr2=0x7f55c0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:16.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.615+0000 7f55e0400640 1 --2- 172.21.15.96:0/3268835126 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f55c0004800 0x7f55c0004c80 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7f55c4009b00 tx=0x7f55c0005830 comp rx=0 tx=0).stop 2024-05-28T06:19:16.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.615+0000 7f55de400640 1 -- 172.21.15.96:0/3268835126 reap_dead start 2024-05-28T06:19:16.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.615+0000 7f55e0400640 1 -- 172.21.15.96:0/3268835126 shutdown_connections 2024-05-28T06:19:16.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.615+0000 7f55e0400640 1 -- 172.21.15.96:0/3268835126 >> 172.21.15.96:0/3268835126 conn(0x7f55d808bfc0 msgr2=0x7f55d80d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:19:16.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.615+0000 7f55e0400640 1 -- 172.21.15.96:0/3268835126 shutdown_connections 2024-05-28T06:19:16.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.615+0000 7f55e0400640 1 -- 172.21.15.96:0/3268835126 wait complete. 2024-05-28T06:19:16.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.617+0000 7f55e0400640 1 Processor -- start 2024-05-28T06:19:16.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.617+0000 7f55e0400640 1 -- start start 2024-05-28T06:19:16.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.618+0000 7f55e0400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f55c0004800 0x7f55d80b7570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:16.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.618+0000 7f55e0400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f55d80cb7b0 0x7f55d815e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:16.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.618+0000 7f55e0400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f55d80cce10 0x7f55d815eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:16.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.618+0000 7f55e0400640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f55d80d3170 con 0x7f55d80cce10 2024-05-28T06:19:16.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.618+0000 7f55e0400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f55d80d2ff0 con 0x7f55c0004800 2024-05-28T06:19:16.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.618+0000 7f55e0400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f55d8162510 con 0x7f55d80cb7b0 2024-05-28T06:19:16.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.618+0000 7f55de400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f55d80cce10 0x7f55d815eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:16.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.618+0000 7f55de400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f55d80cce10 0x7f55d815eef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:36834/0 (socket says 172.21.15.96:36834) 2024-05-28T06:19:16.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.618+0000 7f55de400640 1 -- 172.21.15.96:0/1060828750 learned_addr learned my addr 172.21.15.96:0/1060828750 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:19:16.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.619+0000 7f55dda00640 1 --2- 172.21.15.96:0/1060828750 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f55c0004800 0x7f55d80b7570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:16.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.619+0000 7f55dd000640 1 --2- 172.21.15.96:0/1060828750 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f55d80cb7b0 0x7f55d815e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:16.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.619+0000 7f55de400640 1 -- 172.21.15.96:0/1060828750 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f55d80cb7b0 msgr2=0x7f55d815e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:16.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.619+0000 7f55de400640 1 --2- 172.21.15.96:0/1060828750 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f55d80cb7b0 0x7f55d815e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:16.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.619+0000 7f55de400640 1 -- 172.21.15.96:0/1060828750 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f55c0004800 msgr2=0x7f55d80b7570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:16.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.619+0000 7f55de400640 1 --2- 172.21.15.96:0/1060828750 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f55c0004800 0x7f55d80b7570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:16.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.619+0000 7f55de400640 1 -- 172.21.15.96:0/1060828750 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55d815f8c0 con 0x7f55d80cce10 2024-05-28T06:19:16.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.619+0000 7f55de400640 1 --2- 172.21.15.96:0/1060828750 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f55d80cce10 0x7f55d815eef0 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7f55d808db20 tx=0x7f55d0007bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:16.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.620+0000 7f55d5600640 1 -- 172.21.15.96:0/1060828750 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55d0017020 con 0x7f55d80cce10 2024-05-28T06:19:16.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.620+0000 7f55d5600640 1 -- 172.21.15.96:0/1060828750 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55d000ab90 con 0x7f55d80cce10 2024-05-28T06:19:16.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.620+0000 7f55e0400640 1 -- 172.21.15.96:0/1060828750 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55d80cbd70 con 0x7f55d80cce10 2024-05-28T06:19:16.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.620+0000 7f55d5600640 1 -- 172.21.15.96:0/1060828750 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55d00133f0 con 0x7f55d80cce10 2024-05-28T06:19:16.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.620+0000 7f55e0400640 1 -- 172.21.15.96:0/1060828750 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55d80cc330 con 0x7f55d80cce10 2024-05-28T06:19:16.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.621+0000 7f55e0400640 1 -- 172.21.15.96:0/1060828750 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55d80690f0 con 0x7f55d80cce10 2024-05-28T06:19:16.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.622+0000 7f55d5600640 1 -- 172.21.15.96:0/1060828750 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 33) ==== 92825+0+0 (secure 0 0 0) 0x7f55d001a020 con 0x7f55d80cce10 2024-05-28T06:19:16.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.623+0000 7f55d5600640 1 --2- 172.21.15.96:0/1060828750 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f55b806dcd0 0x7f55b8070190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:16.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.623+0000 7f55d5600640 1 -- 172.21.15.96:0/1060828750 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 5766+0+0 (secure 0 0 0) 0x7f55d008b380 con 0x7f55d80cce10 2024-05-28T06:19:16.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.624+0000 7f55dda00640 1 --2- 172.21.15.96:0/1060828750 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f55b806dcd0 0x7f55b8070190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:16.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.624+0000 7f55dda00640 1 --2- 172.21.15.96:0/1060828750 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f55b806dcd0 0x7f55b8070190 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f55cc009c90 tx=0x7f55cc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:16.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:16.628+0000 7f55d5600640 1 -- 172.21.15.96:0/1060828750 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f55d0021020 con 0x7f55d80cce10 2024-05-28T06:19:17.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.015+0000 7f55e0400640 1 -- 172.21.15.96:0/1060828750 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) -- 0x7f55d80d13a0 con 0x7f55b806dcd0 2024-05-28T06:19:17.075 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.077+0000 7f55d5600640 1 -- 172.21.15.96:0/1060828750 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+279 (secure 0 0 0) 0x7f55d80d13a0 con 0x7f55b806dcd0 2024-05-28T06:19:17.075 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:19:17.075 INFO:teuthology.orchestra.run.smithi096.stdout: "status": { 2024-05-28T06:19:17.075 INFO:teuthology.orchestra.run.smithi096.stdout: "state": "in-progress", 2024-05-28T06:19:17.075 INFO:teuthology.orchestra.run.smithi096.stdout: "source": { 2024-05-28T06:19:17.075 INFO:teuthology.orchestra.run.smithi096.stdout: "volume": "cephfs", 2024-05-28T06:19:17.075 INFO:teuthology.orchestra.run.smithi096.stdout: "subvolume": "sv1", 2024-05-28T06:19:17.075 INFO:teuthology.orchestra.run.smithi096.stdout: "snapshot": "ss1" 2024-05-28T06:19:17.076 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:19:17.076 INFO:teuthology.orchestra.run.smithi096.stdout: "progress_report": { 2024-05-28T06:19:17.076 INFO:teuthology.orchestra.run.smithi096.stdout: "percentage cloned": 64.323, 2024-05-28T06:19:17.076 INFO:teuthology.orchestra.run.smithi096.stdout: "amount cloned": "1.9G/3.0G", 2024-05-28T06:19:17.076 INFO:teuthology.orchestra.run.smithi096.stdout: "files cloned": "5/6" 2024-05-28T06:19:17.076 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:17.076 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:17.076 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:19:17.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.080+0000 7f55e0400640 1 -- 172.21.15.96:0/1060828750 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f55b806dcd0 msgr2=0x7f55b8070190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:17.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.080+0000 7f55e0400640 1 --2- 172.21.15.96:0/1060828750 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f55b806dcd0 0x7f55b8070190 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f55cc009c90 tx=0x7f55cc008040 comp rx=0 tx=0).stop 2024-05-28T06:19:17.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.080+0000 7f55e0400640 1 -- 172.21.15.96:0/1060828750 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f55d80cce10 msgr2=0x7f55d815eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:17.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.080+0000 7f55e0400640 1 --2- 172.21.15.96:0/1060828750 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f55d80cce10 0x7f55d815eef0 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7f55d808db20 tx=0x7f55d0007bd0 comp rx=0 tx=0).stop 2024-05-28T06:19:17.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.080+0000 7f55e0400640 1 -- 172.21.15.96:0/1060828750 shutdown_connections 2024-05-28T06:19:17.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.080+0000 7f55e0400640 1 --2- 172.21.15.96:0/1060828750 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f55b806dcd0 0x7f55b8070190 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:17.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.080+0000 7f55e0400640 1 --2- 172.21.15.96:0/1060828750 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f55d80cce10 0x7f55d815eef0 unknown :-1 s=CLOSED pgs=1476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:17.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.080+0000 7f55e0400640 1 --2- 172.21.15.96:0/1060828750 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f55d80cb7b0 0x7f55d815e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:17.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.080+0000 7f55e0400640 1 --2- 172.21.15.96:0/1060828750 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f55c0004800 0x7f55d80b7570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:17.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.080+0000 7f55e0400640 1 -- 172.21.15.96:0/1060828750 >> 172.21.15.96:0/1060828750 conn(0x7f55d808bfc0 msgr2=0x7f55d806cb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:19:17.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.081+0000 7f55e0400640 1 -- 172.21.15.96:0/1060828750 shutdown_connections 2024-05-28T06:19:17.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.081+0000 7f55e0400640 1 -- 172.21.15.96:0/1060828750 wait complete. 2024-05-28T06:19:17.108 INFO:tasks.cephfs.test_volumes:percentage cloned = 64.323 2024-05-28T06:19:17.108 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-05-28T06:19:17.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.364+0000 7f1522000640 1 Processor -- start 2024-05-28T06:19:17.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.364+0000 7f1522000640 1 -- start start 2024-05-28T06:19:17.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.366+0000 7f1522000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f151c162130 0x7f151c0d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:17.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.366+0000 7f1522000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f151c0d15e0 0x7f151c0b1e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:17.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.366+0000 7f1522000640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f151c0b13d0 con 0x7f151c0b2380 2024-05-28T06:19:17.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.366+0000 7f1522000640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f151c0d33b0 con 0x7f151c162130 2024-05-28T06:19:17.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.366+0000 7f1522000640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f151c0b1640 con 0x7f151c0d15e0 2024-05-28T06:19:17.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.367+0000 7f151b400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f151c162130 0x7f151c0d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:17.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.367+0000 7f151b400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f151c162130 0x7f151c0d10a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:36848/0 (socket says 172.21.15.96:36848) 2024-05-28T06:19:17.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.367+0000 7f151b400640 1 -- 172.21.15.96:0/3972814919 learned_addr learned my addr 172.21.15.96:0/3972814919 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:19:17.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.367+0000 7f151aa00640 1 --2- 172.21.15.96:0/3972814919 >> v2:172.21.15.123:3300/0 conn(0x7f151c0d15e0 0x7f151c0b1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:17.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.367+0000 7f151b400640 1 -- 172.21.15.96:0/3972814919 >> v2:172.21.15.123:3300/0 conn(0x7f151c0d15e0 msgr2=0x7f151c0b1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:19:17.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.367+0000 7f151b400640 1 --2- 172.21.15.96:0/3972814919 >> v2:172.21.15.123:3300/0 conn(0x7f151c0d15e0 0x7f151c0b1e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:17.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.367+0000 7f151b400640 1 -- 172.21.15.96:0/3972814919 >> v1:172.21.15.123:6789/0 conn(0x7f151c0b2380 legacy=0x7f151c0b27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:19:17.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.367+0000 7f151b400640 1 -- 172.21.15.96:0/3972814919 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f151c0b1920 con 0x7f151c162130 2024-05-28T06:19:17.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.368+0000 7f151b400640 1 --2- 172.21.15.96:0/3972814919 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f151c162130 0x7f151c0d10a0 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f151c0b7250 tx=0x7f150c02a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e15e25de52af047 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:17.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.369+0000 7f151a000640 1 -- 172.21.15.96:0/3972814919 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f150c037070 con 0x7f151c162130 2024-05-28T06:19:17.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.369+0000 7f151a000640 1 -- 172.21.15.96:0/3972814919 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f150c032440 con 0x7f151c162130 2024-05-28T06:19:17.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.369+0000 7f151a000640 1 -- 172.21.15.96:0/3972814919 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f150c032810 con 0x7f151c162130 2024-05-28T06:19:17.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.369+0000 7f1522000640 1 -- 172.21.15.96:0/3972814919 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f151c162130 msgr2=0x7f151c0d10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:17.367 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.369+0000 7f1522000640 1 --2- 172.21.15.96:0/3972814919 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f151c162130 0x7f151c0d10a0 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f151c0b7250 tx=0x7f150c02a230 comp rx=0 tx=0).stop 2024-05-28T06:19:17.368 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.370+0000 7f1522000640 1 -- 172.21.15.96:0/3972814919 shutdown_connections 2024-05-28T06:19:17.368 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.370+0000 7f1522000640 1 --2- 172.21.15.96:0/3972814919 >> v2:172.21.15.123:3300/0 conn(0x7f151c0d15e0 0x7f151c0b1e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:17.368 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.370+0000 7f1522000640 1 --2- 172.21.15.96:0/3972814919 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f151c162130 0x7f151c0d10a0 unknown :-1 s=CLOSED pgs=1477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:17.368 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.370+0000 7f1522000640 1 -- 172.21.15.96:0/3972814919 >> 172.21.15.96:0/3972814919 conn(0x7f151c08bfc0 msgr2=0x7f151c0d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:19:17.368 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.370+0000 7f1522000640 1 -- 172.21.15.96:0/3972814919 shutdown_connections 2024-05-28T06:19:17.368 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.370+0000 7f1522000640 1 -- 172.21.15.96:0/3972814919 wait complete. 2024-05-28T06:19:17.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.371+0000 7f1522000640 1 Processor -- start 2024-05-28T06:19:17.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.372+0000 7f1522000640 1 -- start start 2024-05-28T06:19:17.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.373+0000 7f1522000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f151c0b2380 0x7f151c0b8010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:17.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.373+0000 7f1522000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f151c0d15e0 0x7f151c0b8770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:17.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.373+0000 7f1522000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f151c162130 0x7f151c1f8b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:17.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.373+0000 7f1522000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f151c162690 con 0x7f151c0d15e0 2024-05-28T06:19:17.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.373+0000 7f1522000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f151c162510 con 0x7f151c0b2380 2024-05-28T06:19:17.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.373+0000 7f1522000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f151c0d3690 con 0x7f151c162130 2024-05-28T06:19:17.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.373+0000 7f151aa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f151c0d15e0 0x7f151c0b8770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:17.373 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.373+0000 7f151aa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f151c0d15e0 0x7f151c0b8770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:36858/0 (socket says 172.21.15.96:36858) 2024-05-28T06:19:17.373 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.373+0000 7f151aa00640 1 -- 172.21.15.96:0/3919084580 learned_addr learned my addr 172.21.15.96:0/3919084580 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:19:17.373 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.373+0000 7f151b400640 1 --2- 172.21.15.96:0/3919084580 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f151c0b2380 0x7f151c0b8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:17.373 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.373+0000 7f151be00640 1 --2- 172.21.15.96:0/3919084580 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f151c162130 0x7f151c1f8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:17.373 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.374+0000 7f151aa00640 1 -- 172.21.15.96:0/3919084580 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f151c162130 msgr2=0x7f151c1f8b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:17.373 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.374+0000 7f151aa00640 1 --2- 172.21.15.96:0/3919084580 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f151c162130 0x7f151c1f8b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:17.373 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.374+0000 7f151aa00640 1 -- 172.21.15.96:0/3919084580 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f151c0b2380 msgr2=0x7f151c0b8010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:17.373 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.374+0000 7f151aa00640 1 --2- 172.21.15.96:0/3919084580 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f151c0b2380 0x7f151c0b8010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:17.373 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.374+0000 7f151aa00640 1 -- 172.21.15.96:0/3919084580 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f151c1f9d70 con 0x7f151c0d15e0 2024-05-28T06:19:17.373 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.374+0000 7f151aa00640 1 --2- 172.21.15.96:0/3919084580 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f151c0d15e0 0x7f151c0b8770 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f150800d800 tx=0x7f150800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:17.373 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.374+0000 7f1513400640 1 -- 172.21.15.96:0/3919084580 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f150800b6c0 con 0x7f151c0d15e0 2024-05-28T06:19:17.373 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.374+0000 7f1513400640 1 -- 172.21.15.96:0/3919084580 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1508019070 con 0x7f151c0d15e0 2024-05-28T06:19:17.373 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.374+0000 7f1522000640 1 -- 172.21.15.96:0/3919084580 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f151c1fa050 con 0x7f151c0d15e0 2024-05-28T06:19:17.373 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.374+0000 7f1513400640 1 -- 172.21.15.96:0/3919084580 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f150800ba80 con 0x7f151c0d15e0 2024-05-28T06:19:17.374 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.374+0000 7f1522000640 1 -- 172.21.15.96:0/3919084580 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f151c1fa610 con 0x7f151c0d15e0 2024-05-28T06:19:17.374 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.375+0000 7f1522000640 1 -- 172.21.15.96:0/3919084580 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f151c0690f0 con 0x7f151c0d15e0 2024-05-28T06:19:17.374 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.377+0000 7f1513400640 1 -- 172.21.15.96:0/3919084580 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 33) ==== 92825+0+0 (secure 0 0 0) 0x7f150800bc20 con 0x7f151c0d15e0 2024-05-28T06:19:17.375 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.377+0000 7f1513400640 1 --2- 172.21.15.96:0/3919084580 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f14fc06dca0 0x7f14fc070160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:17.375 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.378+0000 7f1513400640 1 -- 172.21.15.96:0/3919084580 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 5766+0+0 (secure 0 0 0) 0x7f150808b0b0 con 0x7f151c0d15e0 2024-05-28T06:19:17.375 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.378+0000 7f151b400640 1 --2- 172.21.15.96:0/3919084580 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f14fc06dca0 0x7f14fc070160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:17.380 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.382+0000 7f151b400640 1 --2- 172.21.15.96:0/3919084580 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f14fc06dca0 0x7f14fc070160 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f150c034000 tx=0x7f150c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:17.380 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.383+0000 7f1513400640 1 -- 172.21.15.96:0/3919084580 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f150805ab80 con 0x7f151c0d15e0 2024-05-28T06:19:17.718 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.720+0000 7f1522000640 1 -- 172.21.15.96:0/3919084580 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) -- 0x7f151c06d900 con 0x7f14fc06dca0 2024-05-28T06:19:17.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.756+0000 7f1513400640 1 -- 172.21.15.96:0/3919084580 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+278 (secure 0 0 0) 0x7f151c06d900 con 0x7f14fc06dca0 2024-05-28T06:19:17.754 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:19:17.754 INFO:teuthology.orchestra.run.smithi096.stdout: "status": { 2024-05-28T06:19:17.754 INFO:teuthology.orchestra.run.smithi096.stdout: "state": "in-progress", 2024-05-28T06:19:17.754 INFO:teuthology.orchestra.run.smithi096.stdout: "source": { 2024-05-28T06:19:17.754 INFO:teuthology.orchestra.run.smithi096.stdout: "volume": "cephfs", 2024-05-28T06:19:17.754 INFO:teuthology.orchestra.run.smithi096.stdout: "subvolume": "sv1", 2024-05-28T06:19:17.754 INFO:teuthology.orchestra.run.smithi096.stdout: "snapshot": "ss1" 2024-05-28T06:19:17.754 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:19:17.754 INFO:teuthology.orchestra.run.smithi096.stdout: "progress_report": { 2024-05-28T06:19:17.754 INFO:teuthology.orchestra.run.smithi096.stdout: "percentage cloned": 87.24, 2024-05-28T06:19:17.755 INFO:teuthology.orchestra.run.smithi096.stdout: "amount cloned": "2.6G/3.0G", 2024-05-28T06:19:17.755 INFO:teuthology.orchestra.run.smithi096.stdout: "files cloned": "6/6" 2024-05-28T06:19:17.755 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:17.755 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:17.755 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:19:17.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.760+0000 7f1522000640 1 -- 172.21.15.96:0/3919084580 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f14fc06dca0 msgr2=0x7f14fc070160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:17.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.760+0000 7f1522000640 1 --2- 172.21.15.96:0/3919084580 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f14fc06dca0 0x7f14fc070160 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f150c034000 tx=0x7f150c02e000 comp rx=0 tx=0).stop 2024-05-28T06:19:17.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.760+0000 7f1522000640 1 -- 172.21.15.96:0/3919084580 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f151c0d15e0 msgr2=0x7f151c0b8770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:17.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.760+0000 7f1522000640 1 --2- 172.21.15.96:0/3919084580 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f151c0d15e0 0x7f151c0b8770 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f150800d800 tx=0x7f150800dcc0 comp rx=0 tx=0).stop 2024-05-28T06:19:17.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.760+0000 7f1522000640 1 -- 172.21.15.96:0/3919084580 shutdown_connections 2024-05-28T06:19:17.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.760+0000 7f1522000640 1 --2- 172.21.15.96:0/3919084580 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f14fc06dca0 0x7f14fc070160 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:17.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.760+0000 7f1522000640 1 --2- 172.21.15.96:0/3919084580 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f151c162130 0x7f151c1f8b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:17.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.760+0000 7f1522000640 1 --2- 172.21.15.96:0/3919084580 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f151c0d15e0 0x7f151c0b8770 unknown :-1 s=CLOSED pgs=1478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:17.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.760+0000 7f1522000640 1 --2- 172.21.15.96:0/3919084580 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f151c0b2380 0x7f151c0b8010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:17.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.760+0000 7f1522000640 1 -- 172.21.15.96:0/3919084580 >> 172.21.15.96:0/3919084580 conn(0x7f151c08bfc0 msgr2=0x7f151c06de20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:19:17.760 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.763+0000 7f1522000640 1 -- 172.21.15.96:0/3919084580 shutdown_connections 2024-05-28T06:19:17.761 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:17.763+0000 7f1522000640 1 -- 172.21.15.96:0/3919084580 wait complete. 2024-05-28T06:19:17.783 INFO:tasks.cephfs.test_volumes:percentage cloned = 87.24 2024-05-28T06:19:17.783 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-05-28T06:19:18.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.015+0000 7fdb4a600640 1 Processor -- start 2024-05-28T06:19:18.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.015+0000 7fdb4a600640 1 -- start start 2024-05-28T06:19:18.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.016+0000 7fdb4a600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fdb440d0640 0x7fdb4415d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:18.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.016+0000 7fdb4a600640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb440d0060 con 0x7fdb440b32c0 2024-05-28T06:19:18.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.016+0000 7fdb4a600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb4415cb10 con 0x7fdb440d0a20 2024-05-28T06:19:18.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.016+0000 7fdb4a600640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fdb440adbb0 con 0x7fdb440d0640 2024-05-28T06:19:18.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.017+0000 7fdb43e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fdb440b32c0 0x7fdb440b3730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:44966/0 (socket says 172.21.15.96:44966) 2024-05-28T06:19:18.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.017+0000 7fdb43e00640 1 -- 172.21.15.96:0/2732421630 learned_addr learned my addr 172.21.15.96:0/2732421630 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:19:18.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.017+0000 7fdb43400640 1 --2- 172.21.15.96:0/2732421630 >> v2:172.21.15.123:3300/0 conn(0x7fdb440d0640 0x7fdb4415d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:18.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.017+0000 7fdb42000640 1 -- 172.21.15.96:0/2732421630 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3832624 0 0) 0x7fdb440d0060 con 0x7fdb440b32c0 2024-05-28T06:19:18.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.017+0000 7fdb42000640 1 -- 172.21.15.96:0/2732421630 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdb28003520 con 0x7fdb440b32c0 2024-05-28T06:19:18.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.017+0000 7fdb42000640 1 -- 172.21.15.96:0/2732421630 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1575980659 0 0) 0x7fdb4415cb10 con 0x7fdb440d0a20 2024-05-28T06:19:18.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.017+0000 7fdb42000640 1 -- 172.21.15.96:0/2732421630 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdb440d0060 con 0x7fdb440d0a20 2024-05-28T06:19:18.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.018+0000 7fdb42000640 1 -- 172.21.15.96:0/2732421630 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3687896157 0 0) 0x7fdb28003520 con 0x7fdb440b32c0 2024-05-28T06:19:18.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.018+0000 7fdb42000640 1 -- 172.21.15.96:0/2732421630 >> v2:172.21.15.123:3300/0 conn(0x7fdb440d0640 msgr2=0x7fdb4415d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:19:18.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.018+0000 7fdb42000640 1 --2- 172.21.15.96:0/2732421630 >> v2:172.21.15.123:3300/0 conn(0x7fdb440d0640 0x7fdb4415d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:18.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.018+0000 7fdb42000640 1 -- 172.21.15.96:0/2732421630 >> v1:172.21.15.123:6789/0 conn(0x7fdb440d0a20 legacy=0x7fdb440cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:18.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.018+0000 7fdb42000640 1 -- 172.21.15.96:0/2732421630 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb440ad5f0 con 0x7fdb440b32c0 2024-05-28T06:19:18.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.018+0000 7fdb42000640 1 -- 172.21.15.96:0/2732421630 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7fdb3c0019f0 con 0x7fdb440b32c0 2024-05-28T06:19:18.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.018+0000 7fdb42000640 1 -- 172.21.15.96:0/2732421630 >> v1:172.21.15.96:6789/0 conn(0x7fdb440b32c0 legacy=0x7fdb440b3730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:18.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.018+0000 7fdb42000640 1 --2- 172.21.15.96:0/2732421630 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb28003b40 0x7fdb28003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:18.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.018+0000 7fdb42000640 1 --2- 172.21.15.96:0/2732421630 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdb28004800 0x7fdb28004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:18.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.018+0000 7fdb42000640 1 --2- 172.21.15.96:0/2732421630 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb280051c0 0x7fdb2800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:18.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.018+0000 7fdb42000640 1 -- 172.21.15.96:0/2732421630 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fdb2800c0c0 con 0x7fdb280051c0 2024-05-28T06:19:18.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.018+0000 7fdb42000640 1 -- 172.21.15.96:0/2732421630 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fdb2800c270 con 0x7fdb28003b40 2024-05-28T06:19:18.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.018+0000 7fdb42000640 1 -- 172.21.15.96:0/2732421630 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fdb2800c470 con 0x7fdb28004800 2024-05-28T06:19:18.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.018+0000 7fdb43400640 1 --2- 172.21.15.96:0/2732421630 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb28003b40 0x7fdb28003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:18.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.018+0000 7fdb43e00640 1 --2- 172.21.15.96:0/2732421630 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb280051c0 0x7fdb2800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:18.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.018+0000 7fdb42a00640 1 --2- 172.21.15.96:0/2732421630 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdb28004800 0x7fdb28004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:18.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.019+0000 7fdb43e00640 1 -- 172.21.15.96:0/2732421630 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdb28004800 msgr2=0x7fdb28004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:19:18.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.019+0000 7fdb43e00640 1 --2- 172.21.15.96:0/2732421630 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdb28004800 0x7fdb28004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:18.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.019+0000 7fdb43e00640 1 -- 172.21.15.96:0/2732421630 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb28003b40 msgr2=0x7fdb28003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:19:18.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.019+0000 7fdb43e00640 1 --2- 172.21.15.96:0/2732421630 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb28003b40 0x7fdb28003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:18.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.019+0000 7fdb43e00640 1 -- 172.21.15.96:0/2732421630 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb28003520 con 0x7fdb280051c0 2024-05-28T06:19:18.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.019+0000 7fdb43400640 1 --2- 172.21.15.96:0/2732421630 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb28003b40 0x7fdb28003f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-28T06:19:18.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.020+0000 7fdb43e00640 1 -- 172.21.15.96:0/2732421630 reap_dead start 2024-05-28T06:19:18.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.020+0000 7fdb43e00640 1 --2- 172.21.15.96:0/2732421630 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb280051c0 0x7fdb2800b9c0 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7fdb3c00a7f0 tx=0x7fdb28005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bcf69b6cd0ea915 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:18.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.020+0000 7fdb42000640 1 -- 172.21.15.96:0/2732421630 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb34009690 con 0x7fdb280051c0 2024-05-28T06:19:18.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.020+0000 7fdb42000640 1 -- 172.21.15.96:0/2732421630 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb3c02f070 con 0x7fdb280051c0 2024-05-28T06:19:18.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.020+0000 7fdb42000640 1 -- 172.21.15.96:0/2732421630 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb3c00a410 con 0x7fdb280051c0 2024-05-28T06:19:18.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.021+0000 7fdb4a600640 1 -- 172.21.15.96:0/2732421630 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb280051c0 msgr2=0x7fdb2800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:18.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.021+0000 7fdb4a600640 1 --2- 172.21.15.96:0/2732421630 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb280051c0 0x7fdb2800b9c0 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7fdb3c00a7f0 tx=0x7fdb28005830 comp rx=0 tx=0).stop 2024-05-28T06:19:18.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.022+0000 7fdb4a600640 1 -- 172.21.15.96:0/2732421630 shutdown_connections 2024-05-28T06:19:18.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.022+0000 7fdb4a600640 1 --2- 172.21.15.96:0/2732421630 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb280051c0 0x7fdb2800b9c0 unknown :-1 s=CLOSED pgs=1480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:18.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.022+0000 7fdb4a600640 1 -- 172.21.15.96:0/2732421630 >> 172.21.15.96:0/2732421630 conn(0x7fdb4406bd20 msgr2=0x7fdb4406c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:19:18.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.022+0000 7fdb4a600640 1 -- 172.21.15.96:0/2732421630 shutdown_connections 2024-05-28T06:19:18.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.022+0000 7fdb4a600640 1 -- 172.21.15.96:0/2732421630 wait complete. 2024-05-28T06:19:18.022 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.024+0000 7fdb4a600640 1 Processor -- start 2024-05-28T06:19:18.022 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.025+0000 7fdb4a600640 1 -- start start 2024-05-28T06:19:18.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.026+0000 7fdb4a600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb280051c0 0x7fdb44155c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:18.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.026+0000 7fdb4a600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb44152e60 0x7fdb44147540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:18.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.026+0000 7fdb4a600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdb440aef80 0x7fdb440af430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:18.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.026+0000 7fdb4a600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fdb4415d0b0 con 0x7fdb44152e60 2024-05-28T06:19:18.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.026+0000 7fdb4a600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fdb4415cdd0 con 0x7fdb280051c0 2024-05-28T06:19:18.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.026+0000 7fdb4a600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fdb440ad2a0 con 0x7fdb440aef80 2024-05-28T06:19:18.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.026+0000 7fdb42a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb44152e60 0x7fdb44147540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:18.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.026+0000 7fdb42a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb44152e60 0x7fdb44147540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:36890/0 (socket says 172.21.15.96:36890) 2024-05-28T06:19:18.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.026+0000 7fdb42a00640 1 -- 172.21.15.96:0/1138925166 learned_addr learned my addr 172.21.15.96:0/1138925166 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:19:18.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.026+0000 7fdb42a00640 1 -- 172.21.15.96:0/1138925166 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdb440aef80 msgr2=0x7fdb440af430 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-28T06:19:18.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.026+0000 7fdb42a00640 1 --2- 172.21.15.96:0/1138925166 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdb440aef80 0x7fdb440af430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:18.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.026+0000 7fdb42a00640 1 -- 172.21.15.96:0/1138925166 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb280051c0 msgr2=0x7fdb44155c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:18.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.026+0000 7fdb42a00640 1 --2- 172.21.15.96:0/1138925166 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb280051c0 0x7fdb44155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:18.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.026+0000 7fdb42a00640 1 -- 172.21.15.96:0/1138925166 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb4406d9c0 con 0x7fdb44152e60 2024-05-28T06:19:18.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.027+0000 7fdb42a00640 1 --2- 172.21.15.96:0/1138925166 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb44152e60 0x7fdb44147540 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7fdb3000f7a0 tx=0x7fdb3000fc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:18.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.027+0000 7fdb3b400640 1 -- 172.21.15.96:0/1138925166 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb3001a070 con 0x7fdb44152e60 2024-05-28T06:19:18.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.027+0000 7fdb3b400640 1 -- 172.21.15.96:0/1138925166 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb3000fe50 con 0x7fdb44152e60 2024-05-28T06:19:18.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.027+0000 7fdb4a600640 1 -- 172.21.15.96:0/1138925166 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb44153670 con 0x7fdb44152e60 2024-05-28T06:19:18.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.027+0000 7fdb3b400640 1 -- 172.21.15.96:0/1138925166 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb30007050 con 0x7fdb44152e60 2024-05-28T06:19:18.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.027+0000 7fdb4a600640 1 -- 172.21.15.96:0/1138925166 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb4406dc80 con 0x7fdb44152e60 2024-05-28T06:19:18.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.029+0000 7fdb4a600640 1 -- 172.21.15.96:0/1138925166 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb440689e0 con 0x7fdb44152e60 2024-05-28T06:19:18.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.030+0000 7fdb3b400640 1 -- 172.21.15.96:0/1138925166 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 33) ==== 92825+0+0 (secure 0 0 0) 0x7fdb3001d070 con 0x7fdb44152e60 2024-05-28T06:19:18.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.032+0000 7fdb3b400640 1 --2- 172.21.15.96:0/1138925166 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdb2406dcd0 0x7fdb24070190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:18.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.032+0000 7fdb3b400640 1 -- 172.21.15.96:0/1138925166 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 5766+0+0 (secure 0 0 0) 0x7fdb3008a300 con 0x7fdb44152e60 2024-05-28T06:19:18.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.033+0000 7fdb43400640 1 --2- 172.21.15.96:0/1138925166 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdb2406dcd0 0x7fdb24070190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:18.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.038+0000 7fdb43400640 1 --2- 172.21.15.96:0/1138925166 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdb2406dcd0 0x7fdb24070190 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fdb34006b10 tx=0x7fdb34006aa0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:18.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.039+0000 7fdb3b400640 1 -- 172.21.15.96:0/1138925166 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fdb30059e30 con 0x7fdb44152e60 2024-05-28T06:19:18.384 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.387+0000 7fdb4a600640 1 -- 172.21.15.96:0/1138925166 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) -- 0x7fdb4415cb10 con 0x7fdb2406dcd0 2024-05-28T06:19:18.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.408+0000 7fdb3b400640 1 -- 172.21.15.96:0/1138925166 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+278 (secure 0 0 0) 0x7fdb4415cb10 con 0x7fdb2406dcd0 2024-05-28T06:19:18.406 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:19:18.406 INFO:teuthology.orchestra.run.smithi096.stdout: "status": { 2024-05-28T06:19:18.406 INFO:teuthology.orchestra.run.smithi096.stdout: "state": "in-progress", 2024-05-28T06:19:18.406 INFO:teuthology.orchestra.run.smithi096.stdout: "source": { 2024-05-28T06:19:18.406 INFO:teuthology.orchestra.run.smithi096.stdout: "volume": "cephfs", 2024-05-28T06:19:18.406 INFO:teuthology.orchestra.run.smithi096.stdout: "subvolume": "sv1", 2024-05-28T06:19:18.407 INFO:teuthology.orchestra.run.smithi096.stdout: "snapshot": "ss1" 2024-05-28T06:19:18.407 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:19:18.407 INFO:teuthology.orchestra.run.smithi096.stdout: "progress_report": { 2024-05-28T06:19:18.407 INFO:teuthology.orchestra.run.smithi096.stdout: "percentage cloned": 100.0, 2024-05-28T06:19:18.407 INFO:teuthology.orchestra.run.smithi096.stdout: "amount cloned": "3.0G/3.0G", 2024-05-28T06:19:18.407 INFO:teuthology.orchestra.run.smithi096.stdout: "files cloned": "6/6" 2024-05-28T06:19:18.407 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:18.407 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:18.407 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:19:18.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.411+0000 7fdb4a600640 1 -- 172.21.15.96:0/1138925166 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdb2406dcd0 msgr2=0x7fdb24070190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:18.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.411+0000 7fdb4a600640 1 --2- 172.21.15.96:0/1138925166 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdb2406dcd0 0x7fdb24070190 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fdb34006b10 tx=0x7fdb34006aa0 comp rx=0 tx=0).stop 2024-05-28T06:19:18.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.411+0000 7fdb4a600640 1 -- 172.21.15.96:0/1138925166 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb44152e60 msgr2=0x7fdb44147540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:18.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.411+0000 7fdb4a600640 1 --2- 172.21.15.96:0/1138925166 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb44152e60 0x7fdb44147540 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7fdb3000f7a0 tx=0x7fdb3000fc60 comp rx=0 tx=0).stop 2024-05-28T06:19:18.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.411+0000 7fdb4a600640 1 -- 172.21.15.96:0/1138925166 shutdown_connections 2024-05-28T06:19:18.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.411+0000 7fdb4a600640 1 --2- 172.21.15.96:0/1138925166 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdb2406dcd0 0x7fdb24070190 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:18.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.411+0000 7fdb4a600640 1 --2- 172.21.15.96:0/1138925166 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdb440aef80 0x7fdb440af430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:18.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.411+0000 7fdb4a600640 1 --2- 172.21.15.96:0/1138925166 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb44152e60 0x7fdb44147540 unknown :-1 s=CLOSED pgs=1481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:18.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.411+0000 7fdb4a600640 1 --2- 172.21.15.96:0/1138925166 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb280051c0 0x7fdb44155c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:18.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.411+0000 7fdb4a600640 1 -- 172.21.15.96:0/1138925166 >> 172.21.15.96:0/1138925166 conn(0x7fdb4406bd20 msgr2=0x7fdb4415f710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:19:18.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.411+0000 7fdb4a600640 1 -- 172.21.15.96:0/1138925166 shutdown_connections 2024-05-28T06:19:18.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.411+0000 7fdb4a600640 1 -- 172.21.15.96:0/1138925166 wait complete. 2024-05-28T06:19:18.427 INFO:tasks.cephfs.test_volumes:percentage cloned = 100.0 2024-05-28T06:19:18.428 INFO:tasks.cephfs.test_volumes:expected clone state = ('complete',) 2024-05-28T06:19:18.428 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-05-28T06:19:18.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.614+0000 7f5fd7600640 1 Processor -- start 2024-05-28T06:19:18.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.614+0000 7f5fd7600640 1 -- start start 2024-05-28T06:19:18.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.615+0000 7f5fd7600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f5fd00d0640 0x7f5fd015d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:18.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.615+0000 7f5fd7600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5fd00d0a20 0x7f5fd00cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:18.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.616+0000 7f5fd7600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f5fd00b32c0 0x7f5fd00b3750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:18.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.616+0000 7f5fd7600640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f5fd00adbb0 con 0x7f5fd00d0640 2024-05-28T06:19:18.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.616+0000 7f5fd7600640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f5fd015db10 con 0x7f5fd00b32c0 2024-05-28T06:19:18.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.616+0000 7f5fd7600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f5fd015cb10 con 0x7f5fd00d0a20 2024-05-28T06:19:18.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.616+0000 7f5fd4c00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f5fd00d0640 0x7f5fd015d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:18.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.616+0000 7f5fd4c00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f5fd00d0640 0x7f5fd015d5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:36892/0 (socket says 172.21.15.96:36892) 2024-05-28T06:19:18.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.616+0000 7f5fd4c00640 1 -- 172.21.15.96:0/830873131 learned_addr learned my addr 172.21.15.96:0/830873131 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:19:18.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.616+0000 7f5fd4c00640 1 -- 172.21.15.96:0/830873131 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5fd00d0a20 msgr2=0x7f5fd00cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:19:18.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.616+0000 7f5fd4c00640 1 --2- 172.21.15.96:0/830873131 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5fd00d0a20 0x7f5fd00cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:18.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.616+0000 7f5fd4c00640 1 -- 172.21.15.96:0/830873131 >> v2:172.21.15.123:3300/0 conn(0x7f5fd00b32c0 msgr2=0x7f5fd00b3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:19:18.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.616+0000 7f5fd4c00640 1 --2- 172.21.15.96:0/830873131 >> v2:172.21.15.123:3300/0 conn(0x7f5fd00b32c0 0x7f5fd00b3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:18.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.616+0000 7f5fd4c00640 1 -- 172.21.15.96:0/830873131 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fd00ad5f0 con 0x7f5fd00d0640 2024-05-28T06:19:18.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.617+0000 7f5fd4c00640 1 --2- 172.21.15.96:0/830873131 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5fd00d0640 0x7f5fd015d5d0 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7f5fd0160030 tx=0x7f5fc402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=682023b80b932e3d server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:18.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.618+0000 7f5fcf400640 1 -- 172.21.15.96:0/830873131 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fc4037070 con 0x7f5fd00d0640 2024-05-28T06:19:18.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.618+0000 7f5fcf400640 1 -- 172.21.15.96:0/830873131 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5fc4032440 con 0x7f5fd00d0640 2024-05-28T06:19:18.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.618+0000 7f5fcf400640 1 -- 172.21.15.96:0/830873131 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fc4032810 con 0x7f5fd00d0640 2024-05-28T06:19:18.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.618+0000 7f5fd7600640 1 -- 172.21.15.96:0/830873131 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5fd00d0640 msgr2=0x7f5fd015d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:18.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.618+0000 7f5fd7600640 1 --2- 172.21.15.96:0/830873131 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5fd00d0640 0x7f5fd015d5d0 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7f5fd0160030 tx=0x7f5fc402a1e0 comp rx=0 tx=0).stop 2024-05-28T06:19:18.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.618+0000 7f5fd7600640 1 -- 172.21.15.96:0/830873131 shutdown_connections 2024-05-28T06:19:18.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.618+0000 7f5fd7600640 1 --2- 172.21.15.96:0/830873131 >> v2:172.21.15.123:3300/0 conn(0x7f5fd00b32c0 0x7f5fd00b3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:18.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.618+0000 7f5fd7600640 1 --2- 172.21.15.96:0/830873131 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5fd00d0a20 0x7f5fd00cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:18.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.618+0000 7f5fd7600640 1 --2- 172.21.15.96:0/830873131 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5fd00d0640 0x7f5fd015d5d0 unknown :-1 s=CLOSED pgs=1482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:18.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.618+0000 7f5fd7600640 1 -- 172.21.15.96:0/830873131 >> 172.21.15.96:0/830873131 conn(0x7f5fd006bd20 msgr2=0x7f5fd006c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:19:18.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.618+0000 7f5fd7600640 1 -- 172.21.15.96:0/830873131 shutdown_connections 2024-05-28T06:19:18.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.618+0000 7f5fd7600640 1 -- 172.21.15.96:0/830873131 wait complete. 2024-05-28T06:19:18.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.620+0000 7f5fd7600640 1 Processor -- start 2024-05-28T06:19:18.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.620+0000 7f5fd7600640 1 -- start start 2024-05-28T06:19:18.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.621+0000 7f5fd7600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5fd00b32c0 0x7f5fd01f3c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:18.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.621+0000 7f5fd7600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5fd00d0640 0x7f5fd01f79b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:18.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.621+0000 7f5fd7600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5fd00d0a20 0x7f5fd0152ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:18.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.621+0000 7f5fd7600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f5fd015d0b0 con 0x7f5fd00b32c0 2024-05-28T06:19:18.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.621+0000 7f5fd7600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5fd015cdd0 con 0x7f5fd00d0a20 2024-05-28T06:19:18.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.621+0000 7f5fd7600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f5fd00ad2a0 con 0x7f5fd00d0640 2024-05-28T06:19:18.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.621+0000 7f5fd4c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5fd00b32c0 0x7f5fd01f3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:18.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.621+0000 7f5fd4c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5fd00b32c0 0x7f5fd01f3c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:36906/0 (socket says 172.21.15.96:36906) 2024-05-28T06:19:18.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.621+0000 7f5fd4c00640 1 -- 172.21.15.96:0/684658846 learned_addr learned my addr 172.21.15.96:0/684658846 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:19:18.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.622+0000 7f5fd4c00640 1 -- 172.21.15.96:0/684658846 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5fd00d0640 msgr2=0x7f5fd01f79b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:18.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.622+0000 7f5fd4c00640 1 --2- 172.21.15.96:0/684658846 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5fd00d0640 0x7f5fd01f79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:18.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.622+0000 7f5fd4c00640 1 -- 172.21.15.96:0/684658846 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5fd00d0a20 msgr2=0x7f5fd0152ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:18.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.622+0000 7f5fd4c00640 1 --2- 172.21.15.96:0/684658846 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5fd00d0a20 0x7f5fd0152ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:18.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.622+0000 7f5fd4c00640 1 -- 172.21.15.96:0/684658846 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fd01f8870 con 0x7f5fd00b32c0 2024-05-28T06:19:18.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.622+0000 7f5fd4c00640 1 --2- 172.21.15.96:0/684658846 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5fd00b32c0 0x7f5fd01f3c60 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7f5fc4000c00 tx=0x7f5fc40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:18.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.622+0000 7f5fccc00640 1 -- 172.21.15.96:0/684658846 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fc4041070 con 0x7f5fd00b32c0 2024-05-28T06:19:18.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.622+0000 7f5fccc00640 1 -- 172.21.15.96:0/684658846 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5fc403c890 con 0x7f5fd00b32c0 2024-05-28T06:19:18.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.622+0000 7f5fd7600640 1 -- 172.21.15.96:0/684658846 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fd01f82b0 con 0x7f5fd00b32c0 2024-05-28T06:19:18.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.622+0000 7f5fccc00640 1 -- 172.21.15.96:0/684658846 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fc4037040 con 0x7f5fd00b32c0 2024-05-28T06:19:18.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.623+0000 7f5fd7600640 1 -- 172.21.15.96:0/684658846 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fd014d5d0 con 0x7f5fd00b32c0 2024-05-28T06:19:18.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.624+0000 7f5fd7600640 1 -- 172.21.15.96:0/684658846 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5fd00689e0 con 0x7f5fd00b32c0 2024-05-28T06:19:18.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.625+0000 7f5fccc00640 1 -- 172.21.15.96:0/684658846 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 33) ==== 92825+0+0 (secure 0 0 0) 0x7f5fc4044070 con 0x7f5fd00b32c0 2024-05-28T06:19:18.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.625+0000 7f5fccc00640 1 --2- 172.21.15.96:0/684658846 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5fb006dcd0 0x7f5fb0070190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:18.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.625+0000 7f5fccc00640 1 -- 172.21.15.96:0/684658846 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 5766+0+0 (secure 0 0 0) 0x7f5fc40ab710 con 0x7f5fd00b32c0 2024-05-28T06:19:18.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.626+0000 7f5fcfe00640 1 --2- 172.21.15.96:0/684658846 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5fb006dcd0 0x7f5fb0070190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:18.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.627+0000 7f5fcfe00640 1 --2- 172.21.15.96:0/684658846 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5fb006dcd0 0x7f5fb0070190 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f5fd009e5f0 tx=0x7f5fbc006bf0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:18.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.630+0000 7f5fccc00640 1 -- 172.21.15.96:0/684658846 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5fc407b300 con 0x7f5fd00b32c0 2024-05-28T06:19:18.867 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:18.870+0000 7f5fd7600640 1 -- 172.21.15.96:0/684658846 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) -- 0x7f5fd00737d0 con 0x7f5fb006dcd0 2024-05-28T06:19:20.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.237+0000 7f5fccc00640 1 -- 172.21.15.96:0/684658846 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+46 (secure 0 0 0) 0x7f5fd00737d0 con 0x7f5fb006dcd0 2024-05-28T06:19:20.235 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:19:20.235 INFO:teuthology.orchestra.run.smithi096.stdout: "status": { 2024-05-28T06:19:20.236 INFO:teuthology.orchestra.run.smithi096.stdout: "state": "complete" 2024-05-28T06:19:20.236 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:20.236 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:19:20.238 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.240+0000 7f5fd7600640 1 -- 172.21.15.96:0/684658846 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5fb006dcd0 msgr2=0x7f5fb0070190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:20.238 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.240+0000 7f5fd7600640 1 --2- 172.21.15.96:0/684658846 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5fb006dcd0 0x7f5fb0070190 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f5fd009e5f0 tx=0x7f5fbc006bf0 comp rx=0 tx=0).stop 2024-05-28T06:19:20.238 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.240+0000 7f5fd7600640 1 -- 172.21.15.96:0/684658846 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5fd00b32c0 msgr2=0x7f5fd01f3c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:20.238 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.240+0000 7f5fd7600640 1 --2- 172.21.15.96:0/684658846 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5fd00b32c0 0x7f5fd01f3c60 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7f5fc4000c00 tx=0x7f5fc40046c0 comp rx=0 tx=0).stop 2024-05-28T06:19:20.238 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.241+0000 7f5fd7600640 1 -- 172.21.15.96:0/684658846 shutdown_connections 2024-05-28T06:19:20.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.241+0000 7f5fd7600640 1 --2- 172.21.15.96:0/684658846 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5fb006dcd0 0x7f5fb0070190 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:20.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.241+0000 7f5fd7600640 1 --2- 172.21.15.96:0/684658846 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5fd00d0a20 0x7f5fd0152ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:20.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.241+0000 7f5fd7600640 1 --2- 172.21.15.96:0/684658846 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5fd00d0640 0x7f5fd01f79b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:20.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.241+0000 7f5fd7600640 1 --2- 172.21.15.96:0/684658846 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5fd00b32c0 0x7f5fd01f3c60 unknown :-1 s=CLOSED pgs=1483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:20.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.241+0000 7f5fd7600640 1 -- 172.21.15.96:0/684658846 >> 172.21.15.96:0/684658846 conn(0x7f5fd006bd20 msgr2=0x7f5fd015f600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:19:20.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.241+0000 7f5fd7600640 1 -- 172.21.15.96:0/684658846 shutdown_connections 2024-05-28T06:19:20.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.241+0000 7f5fd7600640 1 -- 172.21.15.96:0/684658846 wait complete. 2024-05-28T06:19:20.262 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-05-28T06:19:20.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.465+0000 7f80f2600640 1 Processor -- start 2024-05-28T06:19:20.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.465+0000 7f80f2600640 1 -- start start 2024-05-28T06:19:20.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.466+0000 7f80f2600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f80ec06f860 0x7f80ec0b3150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:20.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.466+0000 7f80f2600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f80ec0d0bb0 0x7f80ec0ad130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:20.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.466+0000 7f80f2600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80ec06ed10 con 0x7f80ec0b3690 2024-05-28T06:19:20.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.466+0000 7f80f2600640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f80ec15cb00 con 0x7f80ec0d0bb0 2024-05-28T06:19:20.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.466+0000 7f80f2600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f80ec0b3cc0 con 0x7f80ec06f860 2024-05-28T06:19:20.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.467+0000 7f80ebe00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f80ec0d0bb0 0x7f80ec0ad130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:20.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.467+0000 7f80ebe00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f80ec0d0bb0 0x7f80ec0ad130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:57446/0 (socket says 172.21.15.96:57446) 2024-05-28T06:19:20.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.467+0000 7f80ebe00640 1 -- 172.21.15.96:0/3958692822 learned_addr learned my addr 172.21.15.96:0/3958692822 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:19:20.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.467+0000 7f80ebe00640 1 -- 172.21.15.96:0/3958692822 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f80ec06f860 msgr2=0x7f80ec0b3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:19:20.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.467+0000 7f80eb400640 1 --2- 172.21.15.96:0/3958692822 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f80ec06f860 0x7f80ec0b3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:20.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.467+0000 7f80ebe00640 1 --2- 172.21.15.96:0/3958692822 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f80ec06f860 0x7f80ec0b3150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:20.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.467+0000 7f80ebe00640 1 -- 172.21.15.96:0/3958692822 >> v1:172.21.15.123:6789/0 conn(0x7f80ec0b3690 legacy=0x7f80ec0d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:19:20.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.467+0000 7f80ebe00640 1 -- 172.21.15.96:0/3958692822 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80ec0b3fa0 con 0x7f80ec0d0bb0 2024-05-28T06:19:20.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.468+0000 7f80ebe00640 1 --2- 172.21.15.96:0/3958692822 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f80ec0d0bb0 0x7f80ec0ad130 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7f80e4009e50 tx=0x7f80e402a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c48cafcd4ea4125b server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:20.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.468+0000 7f80ea000640 1 -- 172.21.15.96:0/3958692822 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80e4036070 con 0x7f80ec0d0bb0 2024-05-28T06:19:20.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.468+0000 7f80ea000640 1 -- 172.21.15.96:0/3958692822 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80e4031600 con 0x7f80ec0d0bb0 2024-05-28T06:19:20.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.468+0000 7f80ea000640 1 -- 172.21.15.96:0/3958692822 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80e40319d0 con 0x7f80ec0d0bb0 2024-05-28T06:19:20.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.469+0000 7f80f2600640 1 -- 172.21.15.96:0/3958692822 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f80ec0d0bb0 msgr2=0x7f80ec0ad130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:20.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.469+0000 7f80f2600640 1 --2- 172.21.15.96:0/3958692822 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f80ec0d0bb0 0x7f80ec0ad130 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7f80e4009e50 tx=0x7f80e402a6c0 comp rx=0 tx=0).stop 2024-05-28T06:19:20.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.469+0000 7f80f2600640 1 -- 172.21.15.96:0/3958692822 shutdown_connections 2024-05-28T06:19:20.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.469+0000 7f80f2600640 1 --2- 172.21.15.96:0/3958692822 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f80ec0d0bb0 0x7f80ec0ad130 unknown :-1 s=CLOSED pgs=1484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:20.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.469+0000 7f80f2600640 1 --2- 172.21.15.96:0/3958692822 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f80ec06f860 0x7f80ec0b3150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:20.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.469+0000 7f80f2600640 1 -- 172.21.15.96:0/3958692822 >> 172.21.15.96:0/3958692822 conn(0x7f80ec06bae0 msgr2=0x7f80ec06bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:19:20.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.469+0000 7f80f2600640 1 -- 172.21.15.96:0/3958692822 shutdown_connections 2024-05-28T06:19:20.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.469+0000 7f80f2600640 1 -- 172.21.15.96:0/3958692822 wait complete. 2024-05-28T06:19:20.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.471+0000 7f80f2600640 1 Processor -- start 2024-05-28T06:19:20.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.471+0000 7f80f2600640 1 -- start start 2024-05-28T06:19:20.470 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.472+0000 7f80f2600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f80ec06f860 0x7f80ec1f7440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:20.470 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.472+0000 7f80f2600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f80ec0b3690 0x7f80ec0c3a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:20.470 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.472+0000 7f80f2600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f80ec0d0bb0 0x7f80ec0b08b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:20.470 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.472+0000 7f80f2600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f80ec0b4840 con 0x7f80ec0d0bb0 2024-05-28T06:19:20.470 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.472+0000 7f80f2600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f80ec0b4560 con 0x7f80ec06f860 2024-05-28T06:19:20.470 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.472+0000 7f80f2600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f80ec0ad670 con 0x7f80ec0b3690 2024-05-28T06:19:20.470 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.473+0000 7f80ebe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f80ec0d0bb0 0x7f80ec0b08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:20.470 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.473+0000 7f80ebe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f80ec0d0bb0 0x7f80ec0b08b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:57450/0 (socket says 172.21.15.96:57450) 2024-05-28T06:19:20.470 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.473+0000 7f80ebe00640 1 -- 172.21.15.96:0/2038473224 learned_addr learned my addr 172.21.15.96:0/2038473224 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:19:20.471 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.473+0000 7f80eb400640 1 --2- 172.21.15.96:0/2038473224 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f80ec06f860 0x7f80ec1f7440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:20.471 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.473+0000 7f80eaa00640 1 --2- 172.21.15.96:0/2038473224 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f80ec0b3690 0x7f80ec0c3a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:20.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.473+0000 7f80ebe00640 1 -- 172.21.15.96:0/2038473224 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f80ec0b3690 msgr2=0x7f80ec0c3a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:20.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.473+0000 7f80ebe00640 1 --2- 172.21.15.96:0/2038473224 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f80ec0b3690 0x7f80ec0c3a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:20.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.473+0000 7f80ebe00640 1 -- 172.21.15.96:0/2038473224 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f80ec06f860 msgr2=0x7f80ec1f7440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:20.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.473+0000 7f80ebe00640 1 --2- 172.21.15.96:0/2038473224 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f80ec06f860 0x7f80ec1f7440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:20.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.473+0000 7f80ebe00640 1 -- 172.21.15.96:0/2038473224 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80ec0c4350 con 0x7f80ec0d0bb0 2024-05-28T06:19:20.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.473+0000 7f80ebe00640 1 --2- 172.21.15.96:0/2038473224 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f80ec0d0bb0 0x7f80ec0b08b0 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7f80e4009f70 tx=0x7f80e4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:20.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.474+0000 7f80e3400640 1 -- 172.21.15.96:0/2038473224 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80e4041020 con 0x7f80ec0d0bb0 2024-05-28T06:19:20.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.474+0000 7f80e3400640 1 -- 172.21.15.96:0/2038473224 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80e4007780 con 0x7f80ec0d0bb0 2024-05-28T06:19:20.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.474+0000 7f80e3400640 1 -- 172.21.15.96:0/2038473224 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80e4036040 con 0x7f80ec0d0bb0 2024-05-28T06:19:20.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.474+0000 7f80f2600640 1 -- 172.21.15.96:0/2038473224 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80ec0c4630 con 0x7f80ec0d0bb0 2024-05-28T06:19:20.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.474+0000 7f80f2600640 1 -- 172.21.15.96:0/2038473224 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80ec0c4bf0 con 0x7f80ec0d0bb0 2024-05-28T06:19:20.473 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.475+0000 7f80f2600640 1 -- 172.21.15.96:0/2038473224 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80ec0689e0 con 0x7f80ec0d0bb0 2024-05-28T06:19:20.473 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.476+0000 7f80e3400640 1 -- 172.21.15.96:0/2038473224 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 33) ==== 92825+0+0 (secure 0 0 0) 0x7f80e4044020 con 0x7f80ec0d0bb0 2024-05-28T06:19:20.474 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.477+0000 7f80e3400640 1 --2- 172.21.15.96:0/2038473224 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f80c806dca0 0x7f80c8070160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:20.474 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.477+0000 7f80e3400640 1 -- 172.21.15.96:0/2038473224 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 5766+0+0 (secure 0 0 0) 0x7f80e40ab5b0 con 0x7f80ec0d0bb0 2024-05-28T06:19:20.475 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.478+0000 7f80eb400640 1 --2- 172.21.15.96:0/2038473224 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f80c806dca0 0x7f80c8070160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:20.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.479+0000 7f80eb400640 1 --2- 172.21.15.96:0/2038473224 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f80c806dca0 0x7f80c8070160 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f80dc00a970 tx=0x7f80dc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:20.479 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.482+0000 7f80e3400640 1 -- 172.21.15.96:0/2038473224 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f80e404b030 con 0x7f80ec0d0bb0 2024-05-28T06:19:20.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.706+0000 7f80f2600640 1 -- 172.21.15.96:0/2038473224 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7f80ec06e2f0 con 0x7f80c806dca0 2024-05-28T06:19:20.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.710+0000 7f80e3400640 1 -- 172.21.15.96:0/2038473224 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7f80ec06e2f0 con 0x7f80c806dca0 2024-05-28T06:19:20.707 INFO:teuthology.orchestra.run.smithi096.stdout:[ 2024-05-28T06:19:20.707 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:19:20.707 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "cephfs" 2024-05-28T06:19:20.707 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:20.708 INFO:teuthology.orchestra.run.smithi096.stdout:] 2024-05-28T06:19:20.710 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.712+0000 7f80f2600640 1 -- 172.21.15.96:0/2038473224 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f80c806dca0 msgr2=0x7f80c8070160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:20.710 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.712+0000 7f80f2600640 1 --2- 172.21.15.96:0/2038473224 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f80c806dca0 0x7f80c8070160 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f80dc00a970 tx=0x7f80dc008040 comp rx=0 tx=0).stop 2024-05-28T06:19:20.710 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.712+0000 7f80f2600640 1 -- 172.21.15.96:0/2038473224 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f80ec0d0bb0 msgr2=0x7f80ec0b08b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:20.710 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.712+0000 7f80f2600640 1 --2- 172.21.15.96:0/2038473224 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f80ec0d0bb0 0x7f80ec0b08b0 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7f80e4009f70 tx=0x7f80e4004a40 comp rx=0 tx=0).stop 2024-05-28T06:19:20.710 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.712+0000 7f80f2600640 1 -- 172.21.15.96:0/2038473224 shutdown_connections 2024-05-28T06:19:20.710 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.712+0000 7f80f2600640 1 --2- 172.21.15.96:0/2038473224 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f80c806dca0 0x7f80c8070160 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:20.710 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.712+0000 7f80f2600640 1 --2- 172.21.15.96:0/2038473224 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f80ec0d0bb0 0x7f80ec0b08b0 unknown :-1 s=CLOSED pgs=1485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:20.710 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.712+0000 7f80f2600640 1 --2- 172.21.15.96:0/2038473224 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f80ec0b3690 0x7f80ec0c3a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:20.710 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.712+0000 7f80f2600640 1 --2- 172.21.15.96:0/2038473224 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f80ec06f860 0x7f80ec1f7440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:20.711 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.712+0000 7f80f2600640 1 -- 172.21.15.96:0/2038473224 >> 172.21.15.96:0/2038473224 conn(0x7f80ec06bae0 msgr2=0x7f80ec0c2da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:19:20.711 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.712+0000 7f80f2600640 1 -- 172.21.15.96:0/2038473224 shutdown_connections 2024-05-28T06:19:20.711 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.712+0000 7f80f2600640 1 -- 172.21.15.96:0/2038473224 wait complete. 2024-05-28T06:19:20.730 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-05-28T06:19:20.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.934+0000 7fda02200640 1 Processor -- start 2024-05-28T06:19:20.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.935+0000 7fda02200640 1 -- start start 2024-05-28T06:19:20.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.936+0000 7fda02200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd9fc09aa30 0x7fd9fc09ae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:20.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.936+0000 7fda02200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fd9fc15ee10 0x7fd9fc1624e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:20.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.936+0000 7fda02200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9fc09b350 con 0x7fd9fc06fee0 2024-05-28T06:19:20.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.936+0000 7fda02200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fd9fc15fbf0 con 0x7fd9fc15ee10 2024-05-28T06:19:20.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.936+0000 7fda02200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fd9fc15f2c0 con 0x7fd9fc09aa30 2024-05-28T06:19:20.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.936+0000 7fd9fbe00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fd9fc06fee0 0x7fd9fc162a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:35986/0 (socket says 172.21.15.96:35986) 2024-05-28T06:19:20.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.936+0000 7fd9fbe00640 1 -- 172.21.15.96:0/937966142 learned_addr learned my addr 172.21.15.96:0/937966142 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:19:20.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.937+0000 7fd9f2a00640 1 --2- 172.21.15.96:0/937966142 >> v2:172.21.15.123:3300/0 conn(0x7fd9fc15ee10 0x7fd9fc1624e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:20.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.937+0000 7fd9fb400640 1 --2- 172.21.15.96:0/937966142 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd9fc09aa30 0x7fd9fc09ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:20.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.937+0000 7fd9faa00640 1 -- 172.21.15.96:0/937966142 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1133968493 0 0) 0x7fd9fc09b350 con 0x7fd9fc06fee0 2024-05-28T06:19:20.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.937+0000 7fd9faa00640 1 -- 172.21.15.96:0/937966142 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd9e0003520 con 0x7fd9fc06fee0 2024-05-28T06:19:20.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.937+0000 7fd9faa00640 1 -- 172.21.15.96:0/937966142 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3662248053 0 0) 0x7fd9e0003520 con 0x7fd9fc06fee0 2024-05-28T06:19:20.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.937+0000 7fd9faa00640 1 -- 172.21.15.96:0/937966142 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd9fc09aa30 msgr2=0x7fd9fc09ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:19:20.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.937+0000 7fd9faa00640 1 --2- 172.21.15.96:0/937966142 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd9fc09aa30 0x7fd9fc09ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:20.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.937+0000 7fd9faa00640 1 -- 172.21.15.96:0/937966142 >> v2:172.21.15.123:3300/0 conn(0x7fd9fc15ee10 msgr2=0x7fd9fc1624e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:19:20.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.937+0000 7fd9faa00640 1 --2- 172.21.15.96:0/937966142 >> v2:172.21.15.123:3300/0 conn(0x7fd9fc15ee10 0x7fd9fc1624e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:20.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.937+0000 7fd9faa00640 1 -- 172.21.15.96:0/937966142 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9fc15f5a0 con 0x7fd9fc06fee0 2024-05-28T06:19:20.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.937+0000 7fd9faa00640 1 -- 172.21.15.96:0/937966142 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7fd9f4001670 con 0x7fd9fc06fee0 2024-05-28T06:19:20.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.937+0000 7fd9faa00640 1 -- 172.21.15.96:0/937966142 >> v1:172.21.15.96:6789/0 conn(0x7fd9fc06fee0 legacy=0x7fd9fc162a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:20.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.937+0000 7fd9faa00640 1 --2- 172.21.15.96:0/937966142 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9e00039b0 0x7fd9e0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:20.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.937+0000 7fd9faa00640 1 --2- 172.21.15.96:0/937966142 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9e0004800 0x7fd9e0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:20.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.937+0000 7fd9faa00640 1 --2- 172.21.15.96:0/937966142 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd9fc09aa30 0x7fd9e00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:20.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.938+0000 7fd9faa00640 1 -- 172.21.15.96:0/937966142 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd9fc15fbf0 con 0x7fd9e00039b0 2024-05-28T06:19:20.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.938+0000 7fd9faa00640 1 -- 172.21.15.96:0/937966142 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd9fc15f2c0 con 0x7fd9e0004800 2024-05-28T06:19:20.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.938+0000 7fd9faa00640 1 -- 172.21.15.96:0/937966142 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fd9e000ba10 con 0x7fd9fc09aa30 2024-05-28T06:19:20.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.938+0000 7fd9fb400640 1 --2- 172.21.15.96:0/937966142 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9e00039b0 0x7fd9e0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:20.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.938+0000 7fd9fb400640 1 -- 172.21.15.96:0/937966142 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd9fc09aa30 msgr2=0x7fd9e00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:19:20.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.938+0000 7fd9fb400640 1 --2- 172.21.15.96:0/937966142 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd9fc09aa30 0x7fd9e00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:20.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.938+0000 7fd9fb400640 1 -- 172.21.15.96:0/937966142 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9e0004800 msgr2=0x7fd9e0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:19:20.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.938+0000 7fd9fb400640 1 --2- 172.21.15.96:0/937966142 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9e0004800 0x7fd9e0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:20.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.938+0000 7fd9fb400640 1 -- 172.21.15.96:0/937966142 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9e0003520 con 0x7fd9e00039b0 2024-05-28T06:19:20.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.939+0000 7fd9fb400640 1 --2- 172.21.15.96:0/937966142 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9e00039b0 0x7fd9e0003dd0 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7fd9ec00e170 tx=0x7fd9e0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5425d1cbad83d09 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:20.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.939+0000 7fd9faa00640 1 -- 172.21.15.96:0/937966142 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9ec034020 con 0x7fd9e00039b0 2024-05-28T06:19:20.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.939+0000 7fd9faa00640 1 -- 172.21.15.96:0/937966142 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9ec030430 con 0x7fd9e00039b0 2024-05-28T06:19:20.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.939+0000 7fd9faa00640 1 -- 172.21.15.96:0/937966142 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9ec009c10 con 0x7fd9e00039b0 2024-05-28T06:19:20.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.940+0000 7fda02200640 1 -- 172.21.15.96:0/937966142 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9e00039b0 msgr2=0x7fd9e0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:20.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.940+0000 7fda02200640 1 --2- 172.21.15.96:0/937966142 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9e00039b0 0x7fd9e0003dd0 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7fd9ec00e170 tx=0x7fd9e0005830 comp rx=0 tx=0).stop 2024-05-28T06:19:20.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.940+0000 7fd9fbe00640 1 -- 172.21.15.96:0/937966142 reap_dead start 2024-05-28T06:19:20.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.940+0000 7fda02200640 1 -- 172.21.15.96:0/937966142 shutdown_connections 2024-05-28T06:19:20.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.940+0000 7fda02200640 1 -- 172.21.15.96:0/937966142 >> 172.21.15.96:0/937966142 conn(0x7fd9fc0b3f80 msgr2=0x7fd9fc0b4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:19:20.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.940+0000 7fda02200640 1 -- 172.21.15.96:0/937966142 shutdown_connections 2024-05-28T06:19:20.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.941+0000 7fda02200640 1 -- 172.21.15.96:0/937966142 wait complete. 2024-05-28T06:19:20.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.942+0000 7fda02200640 1 Processor -- start 2024-05-28T06:19:20.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.942+0000 7fda02200640 1 -- start start 2024-05-28T06:19:20.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.943+0000 7fda02200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9e00039b0 0x7fd9fc1f8ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:20.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.943+0000 7fda02200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9fc0c8400 0x7fd9fc1f4880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:20.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.943+0000 7fda02200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd9fc15ff80 0x7fd9fc160430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:20.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.943+0000 7fda02200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd9fc0b1b30 con 0x7fd9fc0c8400 2024-05-28T06:19:20.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.943+0000 7fda02200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd9fc0b19b0 con 0x7fd9e00039b0 2024-05-28T06:19:20.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.943+0000 7fda02200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fd9fc163130 con 0x7fd9fc15ff80 2024-05-28T06:19:20.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.943+0000 7fd9f2a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9fc0c8400 0x7fd9fc1f4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:20.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.944+0000 7fd9f2a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9fc0c8400 0x7fd9fc1f4880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:57464/0 (socket says 172.21.15.96:57464) 2024-05-28T06:19:20.943 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.944+0000 7fd9f2a00640 1 -- 172.21.15.96:0/3055292944 learned_addr learned my addr 172.21.15.96:0/3055292944 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:19:20.943 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.944+0000 7fd9f2a00640 1 -- 172.21.15.96:0/3055292944 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd9fc15ff80 msgr2=0x7fd9fc160430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:20.943 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.944+0000 7fd9f2a00640 1 --2- 172.21.15.96:0/3055292944 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd9fc15ff80 0x7fd9fc160430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:20.943 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.944+0000 7fd9f2a00640 1 -- 172.21.15.96:0/3055292944 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9e00039b0 msgr2=0x7fd9fc1f8ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:20.943 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.944+0000 7fd9fb400640 1 --2- 172.21.15.96:0/3055292944 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9e00039b0 0x7fd9fc1f8ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:20.943 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.944+0000 7fd9f2a00640 1 --2- 172.21.15.96:0/3055292944 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9e00039b0 0x7fd9fc1f8ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:20.943 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.944+0000 7fd9f2a00640 1 -- 172.21.15.96:0/3055292944 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9fc1f4dc0 con 0x7fd9fc0c8400 2024-05-28T06:19:20.943 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.944+0000 7fd9f2a00640 1 --2- 172.21.15.96:0/3055292944 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9fc0c8400 0x7fd9fc1f4880 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7fd9e400e7c0 tx=0x7fd9e400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:20.943 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.944+0000 7fd9f3e00640 1 -- 172.21.15.96:0/3055292944 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9e4017070 con 0x7fd9fc0c8400 2024-05-28T06:19:20.943 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.944+0000 7fd9f3e00640 1 -- 172.21.15.96:0/3055292944 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9e400c940 con 0x7fd9fc0c8400 2024-05-28T06:19:20.943 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.944+0000 7fd9f3e00640 1 -- 172.21.15.96:0/3055292944 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9e400cd30 con 0x7fd9fc0c8400 2024-05-28T06:19:20.943 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.945+0000 7fda02200640 1 -- 172.21.15.96:0/3055292944 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9fc0c7360 con 0x7fd9fc0c8400 2024-05-28T06:19:20.943 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.945+0000 7fda02200640 1 -- 172.21.15.96:0/3055292944 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9fc0c7920 con 0x7fd9fc0c8400 2024-05-28T06:19:20.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.946+0000 7fda02200640 1 -- 172.21.15.96:0/3055292944 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9fc0689e0 con 0x7fd9fc0c8400 2024-05-28T06:19:20.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.947+0000 7fd9f3e00640 1 -- 172.21.15.96:0/3055292944 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 33) ==== 92825+0+0 (secure 0 0 0) 0x7fd9e401a070 con 0x7fd9fc0c8400 2024-05-28T06:19:20.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.948+0000 7fd9f3e00640 1 --2- 172.21.15.96:0/3055292944 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd9d806dca0 0x7fd9d8070160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:20.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.948+0000 7fd9f3e00640 1 -- 172.21.15.96:0/3055292944 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 5766+0+0 (secure 0 0 0) 0x7fd9e408aa70 con 0x7fd9fc0c8400 2024-05-28T06:19:20.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.949+0000 7fd9fb400640 1 --2- 172.21.15.96:0/3055292944 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd9d806dca0 0x7fd9d8070160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:20.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.950+0000 7fd9fb400640 1 --2- 172.21.15.96:0/3055292944 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd9d806dca0 0x7fd9d8070160 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fd9ec035000 tx=0x7fd9ec02f000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:20.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:20.953+0000 7fd9f3e00640 1 -- 172.21.15.96:0/3055292944 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd9e405a540 con 0x7fd9fc0c8400 2024-05-28T06:19:21.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:21.232+0000 7fda02200640 1 -- 172.21.15.96:0/3055292944 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fd9fc0b4e00 con 0x7fd9fc0c8400 2024-05-28T06:19:21.231 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:21.233+0000 7fd9f3e00640 1 -- 172.21.15.96:0/3055292944 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+3020 (secure 0 0 0) 0x7fd9e4059e40 con 0x7fd9fc0c8400 2024-05-28T06:19:21.231 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:19:21.231 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:19:21.231 INFO:teuthology.orchestra.run.smithi096.stdout: "fsid": "e4a831b5-e6d1-47a7-96cb-bf85154ee0d2", 2024-05-28T06:19:21.231 INFO:teuthology.orchestra.run.smithi096.stdout: "health": { 2024-05-28T06:19:21.232 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "HEALTH_OK", 2024-05-28T06:19:21.232 INFO:teuthology.orchestra.run.smithi096.stdout: "checks": {}, 2024-05-28T06:19:21.232 INFO:teuthology.orchestra.run.smithi096.stdout: "mutes": [] 2024-05-28T06:19:21.232 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:19:21.232 INFO:teuthology.orchestra.run.smithi096.stdout: "election_epoch": 8, 2024-05-28T06:19:21.232 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum": [ 2024-05-28T06:19:21.232 INFO:teuthology.orchestra.run.smithi096.stdout: 0, 2024-05-28T06:19:21.232 INFO:teuthology.orchestra.run.smithi096.stdout: 1, 2024-05-28T06:19:21.232 INFO:teuthology.orchestra.run.smithi096.stdout: 2 2024-05-28T06:19:21.232 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:19:21.232 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_names": [ 2024-05-28T06:19:21.233 INFO:teuthology.orchestra.run.smithi096.stdout: "a", 2024-05-28T06:19:21.233 INFO:teuthology.orchestra.run.smithi096.stdout: "b", 2024-05-28T06:19:21.233 INFO:teuthology.orchestra.run.smithi096.stdout: "c" 2024-05-28T06:19:21.233 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:19:21.233 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_age": 934, 2024-05-28T06:19:21.233 INFO:teuthology.orchestra.run.smithi096.stdout: "monmap": { 2024-05-28T06:19:21.233 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 1, 2024-05-28T06:19:21.233 INFO:teuthology.orchestra.run.smithi096.stdout: "min_mon_release_name": "squid", 2024-05-28T06:19:21.233 INFO:teuthology.orchestra.run.smithi096.stdout: "num_mons": 3 2024-05-28T06:19:21.233 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:19:21.233 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap": { 2024-05-28T06:19:21.234 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 91, 2024-05-28T06:19:21.234 INFO:teuthology.orchestra.run.smithi096.stdout: "num_osds": 8, 2024-05-28T06:19:21.234 INFO:teuthology.orchestra.run.smithi096.stdout: "num_up_osds": 8, 2024-05-28T06:19:21.234 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_up_since": 1716876238, 2024-05-28T06:19:21.234 INFO:teuthology.orchestra.run.smithi096.stdout: "num_in_osds": 8, 2024-05-28T06:19:21.234 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_in_since": 1716876229, 2024-05-28T06:19:21.234 INFO:teuthology.orchestra.run.smithi096.stdout: "num_remapped_pgs": 0 2024-05-28T06:19:21.234 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:19:21.234 INFO:teuthology.orchestra.run.smithi096.stdout: "pgmap": { 2024-05-28T06:19:21.234 INFO:teuthology.orchestra.run.smithi096.stdout: "pgs_by_state": [ 2024-05-28T06:19:21.234 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:19:21.234 INFO:teuthology.orchestra.run.smithi096.stdout: "state_name": "active+clean", 2024-05-28T06:19:21.234 INFO:teuthology.orchestra.run.smithi096.stdout: "count": 97 2024-05-28T06:19:21.235 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:21.235 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:19:21.235 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pgs": 97, 2024-05-28T06:19:21.235 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pools": 3, 2024-05-28T06:19:21.235 INFO:teuthology.orchestra.run.smithi096.stdout: "num_objects": 1284, 2024-05-28T06:19:21.235 INFO:teuthology.orchestra.run.smithi096.stdout: "data_bytes": 5260582528, 2024-05-28T06:19:21.235 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_used": 11361648640, 2024-05-28T06:19:21.235 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_avail": 761732464640, 2024-05-28T06:19:21.235 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_total": 773094113280, 2024-05-28T06:19:21.235 INFO:teuthology.orchestra.run.smithi096.stdout: "read_bytes_sec": 176097828, 2024-05-28T06:19:21.235 INFO:teuthology.orchestra.run.smithi096.stdout: "write_bytes_sec": 178206849, 2024-05-28T06:19:21.236 INFO:teuthology.orchestra.run.smithi096.stdout: "read_op_per_sec": 42, 2024-05-28T06:19:21.236 INFO:teuthology.orchestra.run.smithi096.stdout: "write_op_per_sec": 47 2024-05-28T06:19:21.236 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:19:21.236 INFO:teuthology.orchestra.run.smithi096.stdout: "fsmap": { 2024-05-28T06:19:21.236 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 53, 2024-05-28T06:19:21.236 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 6, 2024-05-28T06:19:21.236 INFO:teuthology.orchestra.run.smithi096.stdout: "up": 1, 2024-05-28T06:19:21.236 INFO:teuthology.orchestra.run.smithi096.stdout: "in": 1, 2024-05-28T06:19:21.236 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 1, 2024-05-28T06:19:21.236 INFO:teuthology.orchestra.run.smithi096.stdout: "by_rank": [ 2024-05-28T06:19:21.237 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:19:21.237 INFO:teuthology.orchestra.run.smithi096.stdout: "filesystem_id": 6, 2024-05-28T06:19:21.237 INFO:teuthology.orchestra.run.smithi096.stdout: "rank": 0, 2024-05-28T06:19:21.237 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "c", 2024-05-28T06:19:21.237 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "up:active", 2024-05-28T06:19:21.237 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 8103 2024-05-28T06:19:21.237 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:21.237 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:19:21.237 INFO:teuthology.orchestra.run.smithi096.stdout: "up:standby": 3 2024-05-28T06:19:21.237 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:19:21.237 INFO:teuthology.orchestra.run.smithi096.stdout: "mgrmap": { 2024-05-28T06:19:21.237 INFO:teuthology.orchestra.run.smithi096.stdout: "available": true, 2024-05-28T06:19:21.237 INFO:teuthology.orchestra.run.smithi096.stdout: "num_standbys": 1, 2024-05-28T06:19:21.237 INFO:teuthology.orchestra.run.smithi096.stdout: "modules": [ 2024-05-28T06:19:21.238 INFO:teuthology.orchestra.run.smithi096.stdout: "iostat", 2024-05-28T06:19:21.238 INFO:teuthology.orchestra.run.smithi096.stdout: "nfs", 2024-05-28T06:19:21.238 INFO:teuthology.orchestra.run.smithi096.stdout: "restful" 2024-05-28T06:19:21.238 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:19:21.238 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:19:21.238 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:19:21.238 INFO:teuthology.orchestra.run.smithi096.stdout: "servicemap": { 2024-05-28T06:19:21.238 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 16, 2024-05-28T06:19:21.238 INFO:teuthology.orchestra.run.smithi096.stdout: "modified": "2024-05-28T06:19:17.891181+0000", 2024-05-28T06:19:21.238 INFO:teuthology.orchestra.run.smithi096.stdout: "services": { 2024-05-28T06:19:21.238 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-05-28T06:19:21.238 INFO:teuthology.orchestra.run.smithi096.stdout: "daemons": { 2024-05-28T06:19:21.238 INFO:teuthology.orchestra.run.smithi096.stdout: "summary": "", 2024-05-28T06:19:21.239 INFO:teuthology.orchestra.run.smithi096.stdout: "y": { 2024-05-28T06:19:21.239 INFO:teuthology.orchestra.run.smithi096.stdout: "start_epoch": 0, 2024-05-28T06:19:21.239 INFO:teuthology.orchestra.run.smithi096.stdout: "start_stamp": "0.000000", 2024-05-28T06:19:21.239 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 0, 2024-05-28T06:19:21.239 INFO:teuthology.orchestra.run.smithi096.stdout: "addr": "(unrecognized address family 0)/0", 2024-05-28T06:19:21.239 INFO:teuthology.orchestra.run.smithi096.stdout: "metadata": {}, 2024-05-28T06:19:21.239 INFO:teuthology.orchestra.run.smithi096.stdout: "task_status": {} 2024-05-28T06:19:21.239 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:21.239 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:21.239 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:19:21.239 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-05-28T06:19:21.239 INFO:teuthology.orchestra.run.smithi096.stdout: "daemons": { 2024-05-28T06:19:21.239 INFO:teuthology.orchestra.run.smithi096.stdout: "summary": "", 2024-05-28T06:19:21.239 INFO:teuthology.orchestra.run.smithi096.stdout: "b": { 2024-05-28T06:19:21.240 INFO:teuthology.orchestra.run.smithi096.stdout: "start_epoch": 0, 2024-05-28T06:19:21.240 INFO:teuthology.orchestra.run.smithi096.stdout: "start_stamp": "0.000000", 2024-05-28T06:19:21.240 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 0, 2024-05-28T06:19:21.240 INFO:teuthology.orchestra.run.smithi096.stdout: "addr": "(unrecognized address family 0)/0", 2024-05-28T06:19:21.240 INFO:teuthology.orchestra.run.smithi096.stdout: "metadata": {}, 2024-05-28T06:19:21.240 INFO:teuthology.orchestra.run.smithi096.stdout: "task_status": {} 2024-05-28T06:19:21.240 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:21.240 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:21.240 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:21.240 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:21.240 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:19:21.240 INFO:teuthology.orchestra.run.smithi096.stdout: "progress_events": { 2024-05-28T06:19:21.240 INFO:teuthology.orchestra.run.smithi096.stdout: "7877b5be-3dc0-42ef-b4f3-0e5ec502edf5": { 2024-05-28T06:19:21.241 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "All cloning finished, waiting 30.0 seconds before exiting (6s)\n [============================] ", 2024-05-28T06:19:21.241 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": 1, 2024-05-28T06:19:21.241 INFO:teuthology.orchestra.run.smithi096.stdout: "add_to_ceph_s": true 2024-05-28T06:19:21.241 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:21.241 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:21.241 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:19:21.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:21.236+0000 7fda02200640 1 -- 172.21.15.96:0/3055292944 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd9d806dca0 msgr2=0x7fd9d8070160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:21.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:21.236+0000 7fda02200640 1 --2- 172.21.15.96:0/3055292944 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd9d806dca0 0x7fd9d8070160 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fd9ec035000 tx=0x7fd9ec02f000 comp rx=0 tx=0).stop 2024-05-28T06:19:21.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:21.236+0000 7fda02200640 1 -- 172.21.15.96:0/3055292944 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9fc0c8400 msgr2=0x7fd9fc1f4880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:21.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:21.236+0000 7fda02200640 1 --2- 172.21.15.96:0/3055292944 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9fc0c8400 0x7fd9fc1f4880 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7fd9e400e7c0 tx=0x7fd9e400ec80 comp rx=0 tx=0).stop 2024-05-28T06:19:21.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:21.236+0000 7fda02200640 1 -- 172.21.15.96:0/3055292944 shutdown_connections 2024-05-28T06:19:21.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:21.236+0000 7fda02200640 1 --2- 172.21.15.96:0/3055292944 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd9d806dca0 0x7fd9d8070160 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:21.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:21.236+0000 7fda02200640 1 --2- 172.21.15.96:0/3055292944 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd9fc15ff80 0x7fd9fc160430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:21.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:21.236+0000 7fda02200640 1 --2- 172.21.15.96:0/3055292944 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9fc0c8400 0x7fd9fc1f4880 unknown :-1 s=CLOSED pgs=1488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:21.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:21.236+0000 7fda02200640 1 --2- 172.21.15.96:0/3055292944 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9e00039b0 0x7fd9fc1f8ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:21.243 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:21.236+0000 7fda02200640 1 -- 172.21.15.96:0/3055292944 >> 172.21.15.96:0/3055292944 conn(0x7fd9fc0b3f80 msgr2=0x7fd9fc070510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:19:21.243 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:21.237+0000 7fda02200640 1 -- 172.21.15.96:0/3055292944 shutdown_connections 2024-05-28T06:19:21.243 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:21.237+0000 7fda02200640 1 -- 172.21.15.96:0/3055292944 wait complete. 2024-05-28T06:19:26.252 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-05-28T06:19:26.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:26.463+0000 7f80fce00640 1 Processor -- start 2024-05-28T06:19:26.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:26.463+0000 7f80fce00640 1 -- start start 2024-05-28T06:19:26.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:26.464+0000 7f80fce00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f80f815c6b0 0x7f80f80b3db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:26.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:26.464+0000 7f80fce00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f80f80ccc30 0x7f80f806dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:26.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:26.464+0000 7f80fce00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80f80cd1a0 con 0x7f80f80d0460 2024-05-28T06:19:26.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:26.464+0000 7f80fce00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f80f80cd480 con 0x7f80f80ccc30 2024-05-28T06:19:26.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:26.465+0000 7f80fce00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f80f80cd760 con 0x7f80f815c6b0 2024-05-28T06:19:26.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:26.465+0000 7f80f7400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f80f815c6b0 0x7f80f80b3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:26.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:26.465+0000 7f80f7400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f80f815c6b0 0x7f80f80b3db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.96:53906/0 (socket says 172.21.15.96:53906) 2024-05-28T06:19:26.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:26.465+0000 7f80f7400640 1 -- 172.21.15.96:0/795506185 learned_addr learned my addr 172.21.15.96:0/795506185 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:19:26.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:26.465+0000 7f80f6a00640 1 --2- 172.21.15.96:0/795506185 >> v2:172.21.15.123:3300/0 conn(0x7f80f80ccc30 0x7f80f806dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:26.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:26.466+0000 7f80f7400640 1 -- 172.21.15.96:0/795506185 >> v2:172.21.15.123:3300/0 conn(0x7f80f80ccc30 msgr2=0x7f80f806dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:19:26.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:26.466+0000 7f80f7400640 1 --2- 172.21.15.96:0/795506185 >> v2:172.21.15.123:3300/0 conn(0x7f80f80ccc30 0x7f80f806dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:26.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:26.466+0000 7f80f7400640 1 -- 172.21.15.96:0/795506185 >> v1:172.21.15.123:6789/0 conn(0x7f80f80d0460 legacy=0x7f80f80cbf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:26.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:26.466+0000 7f80f7400640 1 -- 172.21.15.96:0/795506185 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80f80cda40 con 0x7f80f815c6b0 2024-05-28T06:19:26.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:26.466+0000 7f80f6a00640 1 --2- 172.21.15.96:0/795506185 >> v2:172.21.15.123:3300/0 conn(0x7f80f80ccc30 0x7f80f806dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-28T06:19:26.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:26.468+0000 7f80f7400640 1 --2- 172.21.15.96:0/795506185 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f80f815c6b0 0x7f80f80b3db0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f80e8009870 tx=0x7f80e802a250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=771d780402f07f95 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:26.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:26.469+0000 7f80f6000640 1 -- 172.21.15.96:0/795506185 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80e8037070 con 0x7f80f815c6b0 2024-05-28T06:19:26.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:26.469+0000 7f80f6000640 1 -- 172.21.15.96:0/795506185 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80e8032440 con 0x7f80f815c6b0 2024-05-28T06:19:26.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:26.469+0000 7f80f6000640 1 -- 172.21.15.96:0/795506185 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80e8032810 con 0x7f80f815c6b0 2024-05-28T06:19:26.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:26.469+0000 7f80fce00640 1 -- 172.21.15.96:0/795506185 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f80f815c6b0 msgr2=0x7f80f80b3db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:26.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:26.469+0000 7f80fce00640 1 --2- 172.21.15.96:0/795506185 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f80f815c6b0 0x7f80f80b3db0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f80e8009870 tx=0x7f80e802a250 comp rx=0 tx=0).stop 2024-05-28T06:19:26.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:26.469+0000 7f80fce00640 1 -- 172.21.15.96:0/795506185 shutdown_connections 2024-05-28T06:19:26.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:26.469+0000 7f80fce00640 1 --2- 172.21.15.96:0/795506185 >> v2:172.21.15.123:3300/0 conn(0x7f80f80ccc30 0x7f80f806dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:26.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:26.469+0000 7f80fce00640 1 --2- 172.21.15.96:0/795506185 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f80f815c6b0 0x7f80f80b3db0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:26.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:26.469+0000 7f80fce00640 1 -- 172.21.15.96:0/795506185 >> 172.21.15.96:0/795506185 conn(0x7f80f806bae0 msgr2=0x7f80f806bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:19:26.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:26.470+0000 7f80fce00640 1 -- 172.21.15.96:0/795506185 shutdown_connections 2024-05-28T06:19:26.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:26.470+0000 7f80fce00640 1 -- 172.21.15.96:0/795506185 wait complete. 2024-05-28T06:19:26.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:26.471+0000 7f80fce00640 1 Processor -- start 2024-05-28T06:19:26.469 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:26.471+0000 7f80fce00640 1 -- start start 2024-05-28T06:19:26.470 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:26.472+0000 7f80fce00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f80f80ccc30 0x7f80f80c9d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:26.470 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:26.472+0000 7f80fce00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f80f80d0460 0x7f80f81505f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:26.471 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:26.472+0000 7f80fce00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f80f815c6b0 0x7f80f80cf1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:26.471 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:26.472+0000 7f80fce00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f80f80b4470 con 0x7f80f815c6b0 2024-05-28T06:19:26.471 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:26.472+0000 7f80fce00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f80f80b42f0 con 0x7f80f80d0460 2024-05-28T06:19:26.471 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:26.472+0000 7f80fce00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f80f806e1d0 con 0x7f80f80ccc30 2024-05-28T06:19:26.471 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:26.473+0000 7f80f7e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f80f815c6b0 0x7f80f80cf1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:26.471 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:26.473+0000 7f80f7e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f80f815c6b0 0x7f80f80cf1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:57472/0 (socket says 172.21.15.96:57472) 2024-05-28T06:19:26.471 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:26.473+0000 7f80f7e00640 1 -- 172.21.15.96:0/1388546185 learned_addr learned my addr 172.21.15.96:0/1388546185 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:19:26.471 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:26.473+0000 7f80f6a00640 1 --2- 172.21.15.96:0/1388546185 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f80f80d0460 0x7f80f81505f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:26.471 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:26.473+0000 7f80f7400640 1 --2- 172.21.15.96:0/1388546185 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f80f80ccc30 0x7f80f80c9d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:26.471 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:26.473+0000 7f80f7e00640 1 -- 172.21.15.96:0/1388546185 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f80f80ccc30 msgr2=0x7f80f80c9d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:26.471 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:26.473+0000 7f80f7e00640 1 --2- 172.21.15.96:0/1388546185 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f80f80ccc30 0x7f80f80c9d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:26.471 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:26.473+0000 7f80f7e00640 1 -- 172.21.15.96:0/1388546185 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f80f80d0460 msgr2=0x7f80f81505f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:26.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:26.473+0000 7f80f7e00640 1 --2- 172.21.15.96:0/1388546185 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f80f80d0460 0x7f80f81505f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:26.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:26.473+0000 7f80f7e00640 1 -- 172.21.15.96:0/1388546185 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80f80c87d0 con 0x7f80f815c6b0 2024-05-28T06:19:26.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:26.473+0000 7f80f6a00640 1 --2- 172.21.15.96:0/1388546185 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f80f80d0460 0x7f80f81505f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-28T06:19:26.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:26.474+0000 7f80f7e00640 1 --2- 172.21.15.96:0/1388546185 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f80f815c6b0 0x7f80f80cf1a0 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7f80f000c7a0 tx=0x7f80f000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:26.473 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:26.474+0000 7f80ef400640 1 -- 172.21.15.96:0/1388546185 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80f0003b70 con 0x7f80f815c6b0 2024-05-28T06:19:26.473 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:26.474+0000 7f80ef400640 1 -- 172.21.15.96:0/1388546185 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80f0003d30 con 0x7f80f815c6b0 2024-05-28T06:19:26.473 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:26.474+0000 7f80ef400640 1 -- 172.21.15.96:0/1388546185 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80f000aa90 con 0x7f80f815c6b0 2024-05-28T06:19:26.473 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:26.474+0000 7f80fce00640 1 -- 172.21.15.96:0/1388546185 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80f80c8ab0 con 0x7f80f815c6b0 2024-05-28T06:19:26.473 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:26.474+0000 7f80fce00640 1 -- 172.21.15.96:0/1388546185 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80f80c9070 con 0x7f80f815c6b0 2024-05-28T06:19:26.473 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:26.475+0000 7f80fce00640 1 -- 172.21.15.96:0/1388546185 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80f80689e0 con 0x7f80f815c6b0 2024-05-28T06:19:26.473 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:26.476+0000 7f80ef400640 1 -- 172.21.15.96:0/1388546185 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 33) ==== 92825+0+0 (secure 0 0 0) 0x7f80f000ac30 con 0x7f80f815c6b0 2024-05-28T06:19:26.475 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:26.478+0000 7f80ef400640 1 --2- 172.21.15.96:0/1388546185 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f80d806dca0 0x7f80d8070160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:26.475 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:26.478+0000 7f80ef400640 1 -- 172.21.15.96:0/1388546185 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 5766+0+0 (secure 0 0 0) 0x7f80f008a7b0 con 0x7f80f815c6b0 2024-05-28T06:19:26.475 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:26.478+0000 7f80f7400640 1 --2- 172.21.15.96:0/1388546185 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f80d806dca0 0x7f80d8070160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:26.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:26.480+0000 7f80f7400640 1 --2- 172.21.15.96:0/1388546185 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f80d806dca0 0x7f80d8070160 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f80e8034000 tx=0x7f80e802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:26.480 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:26.483+0000 7f80ef400640 1 -- 172.21.15.96:0/1388546185 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f80f005a630 con 0x7f80f815c6b0 2024-05-28T06:19:26.745 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:26.748+0000 7f80fce00640 1 -- 172.21.15.96:0/1388546185 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f80f8073d60 con 0x7f80f815c6b0 2024-05-28T06:19:26.746 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:26.749+0000 7f80ef400640 1 -- 172.21.15.96:0/1388546185 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+3021 (secure 0 0 0) 0x7f80f0059f30 con 0x7f80f815c6b0 2024-05-28T06:19:26.746 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:19:26.746 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:19:26.747 INFO:teuthology.orchestra.run.smithi096.stdout: "fsid": "e4a831b5-e6d1-47a7-96cb-bf85154ee0d2", 2024-05-28T06:19:26.747 INFO:teuthology.orchestra.run.smithi096.stdout: "health": { 2024-05-28T06:19:26.747 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "HEALTH_OK", 2024-05-28T06:19:26.747 INFO:teuthology.orchestra.run.smithi096.stdout: "checks": {}, 2024-05-28T06:19:26.747 INFO:teuthology.orchestra.run.smithi096.stdout: "mutes": [] 2024-05-28T06:19:26.747 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:19:26.747 INFO:teuthology.orchestra.run.smithi096.stdout: "election_epoch": 8, 2024-05-28T06:19:26.747 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum": [ 2024-05-28T06:19:26.747 INFO:teuthology.orchestra.run.smithi096.stdout: 0, 2024-05-28T06:19:26.747 INFO:teuthology.orchestra.run.smithi096.stdout: 1, 2024-05-28T06:19:26.747 INFO:teuthology.orchestra.run.smithi096.stdout: 2 2024-05-28T06:19:26.748 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:19:26.748 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_names": [ 2024-05-28T06:19:26.748 INFO:teuthology.orchestra.run.smithi096.stdout: "a", 2024-05-28T06:19:26.748 INFO:teuthology.orchestra.run.smithi096.stdout: "b", 2024-05-28T06:19:26.748 INFO:teuthology.orchestra.run.smithi096.stdout: "c" 2024-05-28T06:19:26.748 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:19:26.748 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_age": 940, 2024-05-28T06:19:26.748 INFO:teuthology.orchestra.run.smithi096.stdout: "monmap": { 2024-05-28T06:19:26.748 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 1, 2024-05-28T06:19:26.748 INFO:teuthology.orchestra.run.smithi096.stdout: "min_mon_release_name": "squid", 2024-05-28T06:19:26.749 INFO:teuthology.orchestra.run.smithi096.stdout: "num_mons": 3 2024-05-28T06:19:26.749 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:19:26.749 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap": { 2024-05-28T06:19:26.749 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 91, 2024-05-28T06:19:26.749 INFO:teuthology.orchestra.run.smithi096.stdout: "num_osds": 8, 2024-05-28T06:19:26.749 INFO:teuthology.orchestra.run.smithi096.stdout: "num_up_osds": 8, 2024-05-28T06:19:26.749 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_up_since": 1716876238, 2024-05-28T06:19:26.749 INFO:teuthology.orchestra.run.smithi096.stdout: "num_in_osds": 8, 2024-05-28T06:19:26.749 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_in_since": 1716876229, 2024-05-28T06:19:26.749 INFO:teuthology.orchestra.run.smithi096.stdout: "num_remapped_pgs": 0 2024-05-28T06:19:26.749 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:19:26.749 INFO:teuthology.orchestra.run.smithi096.stdout: "pgmap": { 2024-05-28T06:19:26.750 INFO:teuthology.orchestra.run.smithi096.stdout: "pgs_by_state": [ 2024-05-28T06:19:26.750 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:19:26.750 INFO:teuthology.orchestra.run.smithi096.stdout: "state_name": "active+clean", 2024-05-28T06:19:26.750 INFO:teuthology.orchestra.run.smithi096.stdout: "count": 97 2024-05-28T06:19:26.750 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:26.750 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:19:26.750 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pgs": 97, 2024-05-28T06:19:26.750 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pools": 3, 2024-05-28T06:19:26.750 INFO:teuthology.orchestra.run.smithi096.stdout: "num_objects": 1565, 2024-05-28T06:19:26.750 INFO:teuthology.orchestra.run.smithi096.stdout: "data_bytes": 6443504700, 2024-05-28T06:19:26.750 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_used": 13777838080, 2024-05-28T06:19:26.751 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_avail": 759316275200, 2024-05-28T06:19:26.751 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_total": 773094113280, 2024-05-28T06:19:26.751 INFO:teuthology.orchestra.run.smithi096.stdout: "read_bytes_sec": 268337316, 2024-05-28T06:19:26.751 INFO:teuthology.orchestra.run.smithi096.stdout: "write_bytes_sec": 268360603, 2024-05-28T06:19:26.751 INFO:teuthology.orchestra.run.smithi096.stdout: "read_op_per_sec": 65, 2024-05-28T06:19:26.751 INFO:teuthology.orchestra.run.smithi096.stdout: "write_op_per_sec": 73 2024-05-28T06:19:26.751 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:19:26.751 INFO:teuthology.orchestra.run.smithi096.stdout: "fsmap": { 2024-05-28T06:19:26.751 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 53, 2024-05-28T06:19:26.751 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 6, 2024-05-28T06:19:26.751 INFO:teuthology.orchestra.run.smithi096.stdout: "up": 1, 2024-05-28T06:19:26.751 INFO:teuthology.orchestra.run.smithi096.stdout: "in": 1, 2024-05-28T06:19:26.752 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 1, 2024-05-28T06:19:26.752 INFO:teuthology.orchestra.run.smithi096.stdout: "by_rank": [ 2024-05-28T06:19:26.752 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:19:26.752 INFO:teuthology.orchestra.run.smithi096.stdout: "filesystem_id": 6, 2024-05-28T06:19:26.752 INFO:teuthology.orchestra.run.smithi096.stdout: "rank": 0, 2024-05-28T06:19:26.752 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "c", 2024-05-28T06:19:26.752 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "up:active", 2024-05-28T06:19:26.752 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 8103 2024-05-28T06:19:26.752 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:26.752 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:19:26.752 INFO:teuthology.orchestra.run.smithi096.stdout: "up:standby": 3 2024-05-28T06:19:26.753 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:19:26.753 INFO:teuthology.orchestra.run.smithi096.stdout: "mgrmap": { 2024-05-28T06:19:26.753 INFO:teuthology.orchestra.run.smithi096.stdout: "available": true, 2024-05-28T06:19:26.753 INFO:teuthology.orchestra.run.smithi096.stdout: "num_standbys": 1, 2024-05-28T06:19:26.753 INFO:teuthology.orchestra.run.smithi096.stdout: "modules": [ 2024-05-28T06:19:26.753 INFO:teuthology.orchestra.run.smithi096.stdout: "iostat", 2024-05-28T06:19:26.753 INFO:teuthology.orchestra.run.smithi096.stdout: "nfs", 2024-05-28T06:19:26.753 INFO:teuthology.orchestra.run.smithi096.stdout: "restful" 2024-05-28T06:19:26.753 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:19:26.753 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:19:26.753 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:19:26.753 INFO:teuthology.orchestra.run.smithi096.stdout: "servicemap": { 2024-05-28T06:19:26.754 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 16, 2024-05-28T06:19:26.754 INFO:teuthology.orchestra.run.smithi096.stdout: "modified": "2024-05-28T06:19:17.891181+0000", 2024-05-28T06:19:26.754 INFO:teuthology.orchestra.run.smithi096.stdout: "services": { 2024-05-28T06:19:26.754 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-05-28T06:19:26.754 INFO:teuthology.orchestra.run.smithi096.stdout: "daemons": { 2024-05-28T06:19:26.754 INFO:teuthology.orchestra.run.smithi096.stdout: "summary": "", 2024-05-28T06:19:26.754 INFO:teuthology.orchestra.run.smithi096.stdout: "y": { 2024-05-28T06:19:26.755 INFO:teuthology.orchestra.run.smithi096.stdout: "start_epoch": 0, 2024-05-28T06:19:26.755 INFO:teuthology.orchestra.run.smithi096.stdout: "start_stamp": "0.000000", 2024-05-28T06:19:26.755 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 0, 2024-05-28T06:19:26.755 INFO:teuthology.orchestra.run.smithi096.stdout: "addr": "(unrecognized address family 0)/0", 2024-05-28T06:19:26.755 INFO:teuthology.orchestra.run.smithi096.stdout: "metadata": {}, 2024-05-28T06:19:26.755 INFO:teuthology.orchestra.run.smithi096.stdout: "task_status": {} 2024-05-28T06:19:26.755 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:26.755 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:26.755 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:19:26.755 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-05-28T06:19:26.755 INFO:teuthology.orchestra.run.smithi096.stdout: "daemons": { 2024-05-28T06:19:26.756 INFO:teuthology.orchestra.run.smithi096.stdout: "summary": "", 2024-05-28T06:19:26.756 INFO:teuthology.orchestra.run.smithi096.stdout: "b": { 2024-05-28T06:19:26.756 INFO:teuthology.orchestra.run.smithi096.stdout: "start_epoch": 0, 2024-05-28T06:19:26.756 INFO:teuthology.orchestra.run.smithi096.stdout: "start_stamp": "0.000000", 2024-05-28T06:19:26.756 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 0, 2024-05-28T06:19:26.756 INFO:teuthology.orchestra.run.smithi096.stdout: "addr": "(unrecognized address family 0)/0", 2024-05-28T06:19:26.756 INFO:teuthology.orchestra.run.smithi096.stdout: "metadata": {}, 2024-05-28T06:19:26.756 INFO:teuthology.orchestra.run.smithi096.stdout: "task_status": {} 2024-05-28T06:19:26.756 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:26.756 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:26.756 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:26.757 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:26.757 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:19:26.757 INFO:teuthology.orchestra.run.smithi096.stdout: "progress_events": { 2024-05-28T06:19:26.757 INFO:teuthology.orchestra.run.smithi096.stdout: "7877b5be-3dc0-42ef-b4f3-0e5ec502edf5": { 2024-05-28T06:19:26.757 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "All cloning finished, waiting 24.0 seconds before exiting (12s)\n [============================] ", 2024-05-28T06:19:26.757 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": 1, 2024-05-28T06:19:26.757 INFO:teuthology.orchestra.run.smithi096.stdout: "add_to_ceph_s": true 2024-05-28T06:19:26.757 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:26.757 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:26.757 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:19:26.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:26.751+0000 7f80fce00640 1 -- 172.21.15.96:0/1388546185 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f80d806dca0 msgr2=0x7f80d8070160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:26.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:26.751+0000 7f80fce00640 1 --2- 172.21.15.96:0/1388546185 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f80d806dca0 0x7f80d8070160 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f80e8034000 tx=0x7f80e802e000 comp rx=0 tx=0).stop 2024-05-28T06:19:26.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:26.751+0000 7f80fce00640 1 -- 172.21.15.96:0/1388546185 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f80f815c6b0 msgr2=0x7f80f80cf1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:26.759 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:26.751+0000 7f80fce00640 1 --2- 172.21.15.96:0/1388546185 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f80f815c6b0 0x7f80f80cf1a0 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7f80f000c7a0 tx=0x7f80f000cc60 comp rx=0 tx=0).stop 2024-05-28T06:19:26.759 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:26.751+0000 7f80fce00640 1 -- 172.21.15.96:0/1388546185 shutdown_connections 2024-05-28T06:19:26.759 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:26.751+0000 7f80fce00640 1 --2- 172.21.15.96:0/1388546185 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f80d806dca0 0x7f80d8070160 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:26.759 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:26.751+0000 7f80fce00640 1 --2- 172.21.15.96:0/1388546185 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f80f815c6b0 0x7f80f80cf1a0 unknown :-1 s=CLOSED pgs=1489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:26.759 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:26.751+0000 7f80fce00640 1 --2- 172.21.15.96:0/1388546185 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f80f80d0460 0x7f80f81505f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:26.759 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:26.751+0000 7f80fce00640 1 --2- 172.21.15.96:0/1388546185 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f80f80ccc30 0x7f80f80c9d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:26.759 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:26.751+0000 7f80fce00640 1 -- 172.21.15.96:0/1388546185 >> 172.21.15.96:0/1388546185 conn(0x7f80f806bae0 msgr2=0x7f80f80cbd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:19:26.759 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:26.751+0000 7f80fce00640 1 -- 172.21.15.96:0/1388546185 shutdown_connections 2024-05-28T06:19:26.759 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:26.751+0000 7f80fce00640 1 -- 172.21.15.96:0/1388546185 wait complete. 2024-05-28T06:19:31.768 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-05-28T06:19:31.979 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:31.982+0000 7fc135000640 1 Processor -- start 2024-05-28T06:19:31.979 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:31.982+0000 7fc135000640 1 -- start start 2024-05-28T06:19:31.980 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:31.983+0000 7fc135000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fc1300ccc30 0x7fc13006dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:31.980 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:31.983+0000 7fc135000640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1300cd1a0 con 0x7fc1300d0460 2024-05-28T06:19:31.980 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:31.983+0000 7fc135000640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1300cd480 con 0x7fc13015c6b0 2024-05-28T06:19:31.980 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:31.983+0000 7fc135000640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fc1300cd760 con 0x7fc1300ccc30 2024-05-28T06:19:31.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:31.983+0000 7fc126a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fc1300ccc30 0x7fc13006dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:31.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:31.983+0000 7fc12fe00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fc1300d0460 0x7fc1300cbf30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:39272/0 (socket says 172.21.15.96:39272) 2024-05-28T06:19:31.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:31.983+0000 7fc126a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fc1300ccc30 0x7fc13006dc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35884/0 (socket says 172.21.15.96:35884) 2024-05-28T06:19:31.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:31.984+0000 7fc12fe00640 1 -- 172.21.15.96:0/335997794 learned_addr learned my addr 172.21.15.96:0/335997794 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:19:31.982 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:31.984+0000 7fc126a00640 1 -- 172.21.15.96:0/335997794 >> v1:172.21.15.123:6789/0 conn(0x7fc13015c6b0 legacy=0x7fc1300b3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:19:31.982 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:31.984+0000 7fc126a00640 1 -- 172.21.15.96:0/335997794 >> v1:172.21.15.96:6789/0 conn(0x7fc1300d0460 legacy=0x7fc1300cbf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:31.982 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:31.984+0000 7fc126a00640 1 -- 172.21.15.96:0/335997794 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1300cda40 con 0x7fc1300ccc30 2024-05-28T06:19:31.982 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:31.985+0000 7fc126a00640 1 --2- 172.21.15.96:0/335997794 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc1300ccc30 0x7fc13006dc90 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7fc1180097d0 tx=0x7fc11802a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36feab13372929d9 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:31.982 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:31.985+0000 7fc12ea00640 1 -- 172.21.15.96:0/335997794 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc118037070 con 0x7fc1300ccc30 2024-05-28T06:19:31.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:31.985+0000 7fc12ea00640 1 -- 172.21.15.96:0/335997794 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc118032440 con 0x7fc1300ccc30 2024-05-28T06:19:31.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:31.985+0000 7fc12ea00640 1 -- 172.21.15.96:0/335997794 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc118032810 con 0x7fc1300ccc30 2024-05-28T06:19:31.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:31.986+0000 7fc135000640 1 -- 172.21.15.96:0/335997794 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc1300ccc30 msgr2=0x7fc13006dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:31.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:31.986+0000 7fc135000640 1 --2- 172.21.15.96:0/335997794 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc1300ccc30 0x7fc13006dc90 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7fc1180097d0 tx=0x7fc11802a210 comp rx=0 tx=0).stop 2024-05-28T06:19:31.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:31.986+0000 7fc135000640 1 -- 172.21.15.96:0/335997794 shutdown_connections 2024-05-28T06:19:31.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:31.986+0000 7fc135000640 1 --2- 172.21.15.96:0/335997794 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc1300ccc30 0x7fc13006dc90 unknown :-1 s=CLOSED pgs=1491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:31.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:31.986+0000 7fc135000640 1 -- 172.21.15.96:0/335997794 >> 172.21.15.96:0/335997794 conn(0x7fc13006bae0 msgr2=0x7fc13006bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:19:31.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:31.986+0000 7fc135000640 1 -- 172.21.15.96:0/335997794 shutdown_connections 2024-05-28T06:19:31.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:31.986+0000 7fc135000640 1 -- 172.21.15.96:0/335997794 wait complete. 2024-05-28T06:19:31.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:31.988+0000 7fc135000640 1 Processor -- start 2024-05-28T06:19:31.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:31.988+0000 7fc135000640 1 -- start start 2024-05-28T06:19:31.987 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:31.989+0000 7fc135000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc1300ccc30 0x7fc130154de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:31.987 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:31.989+0000 7fc135000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc1300d0460 0x7fc1300c7d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:31.987 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:31.989+0000 7fc135000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc13015c6b0 0x7fc1301508b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:31.987 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:31.989+0000 7fc135000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc13006e350 con 0x7fc1300d0460 2024-05-28T06:19:31.987 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:31.989+0000 7fc135000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc13006e1d0 con 0x7fc13015c6b0 2024-05-28T06:19:31.987 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:31.989+0000 7fc135000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fc1300cc640 con 0x7fc1300ccc30 2024-05-28T06:19:31.987 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:31.989+0000 7fc126a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc1300d0460 0x7fc1300c7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:31.987 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:31.989+0000 7fc126a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc1300d0460 0x7fc1300c7d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35896/0 (socket says 172.21.15.96:35896) 2024-05-28T06:19:31.987 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:31.989+0000 7fc126a00640 1 -- 172.21.15.96:0/339926809 learned_addr learned my addr 172.21.15.96:0/339926809 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:19:31.988 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:31.989+0000 7fc126a00640 1 -- 172.21.15.96:0/339926809 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc1300ccc30 msgr2=0x7fc130154de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:31.988 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:31.989+0000 7fc126a00640 1 --2- 172.21.15.96:0/339926809 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc1300ccc30 0x7fc130154de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:31.988 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:31.990+0000 7fc126a00640 1 -- 172.21.15.96:0/339926809 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc13015c6b0 msgr2=0x7fc1301508b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:31.988 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:31.990+0000 7fc126a00640 1 --2- 172.21.15.96:0/339926809 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc13015c6b0 0x7fc1301508b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:31.988 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:31.990+0000 7fc126a00640 1 -- 172.21.15.96:0/339926809 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc130155320 con 0x7fc1300d0460 2024-05-28T06:19:31.988 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:31.990+0000 7fc126a00640 1 --2- 172.21.15.96:0/339926809 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc1300d0460 0x7fc1300c7d00 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7fc1180097d0 tx=0x7fc1180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:31.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:31.990+0000 7fc127e00640 1 -- 172.21.15.96:0/339926809 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc118041070 con 0x7fc1300d0460 2024-05-28T06:19:31.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:31.990+0000 7fc127e00640 1 -- 172.21.15.96:0/339926809 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc118004240 con 0x7fc1300d0460 2024-05-28T06:19:31.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:31.990+0000 7fc135000640 1 -- 172.21.15.96:0/339926809 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1300c8530 con 0x7fc1300d0460 2024-05-28T06:19:31.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:31.990+0000 7fc127e00640 1 -- 172.21.15.96:0/339926809 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc118037040 con 0x7fc1300d0460 2024-05-28T06:19:31.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:31.990+0000 7fc135000640 1 -- 172.21.15.96:0/339926809 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1300b00a0 con 0x7fc1300d0460 2024-05-28T06:19:31.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:31.992+0000 7fc135000640 1 -- 172.21.15.96:0/339926809 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1300cefa0 con 0x7fc1300d0460 2024-05-28T06:19:31.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:31.992+0000 7fc127e00640 1 -- 172.21.15.96:0/339926809 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 33) ==== 92825+0+0 (secure 0 0 0) 0x7fc118030770 con 0x7fc1300d0460 2024-05-28T06:19:31.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:31.993+0000 7fc127e00640 1 --2- 172.21.15.96:0/339926809 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc10c06dca0 0x7fc10c070160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:31.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:31.993+0000 7fc127e00640 1 -- 172.21.15.96:0/339926809 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 5766+0+0 (secure 0 0 0) 0x7fc1180ab0e0 con 0x7fc1300d0460 2024-05-28T06:19:31.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:31.994+0000 7fc12f400640 1 --2- 172.21.15.96:0/339926809 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc10c06dca0 0x7fc10c070160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:31.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:31.995+0000 7fc12f400640 1 --2- 172.21.15.96:0/339926809 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc10c06dca0 0x7fc10c070160 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fc120006fd0 tx=0x7fc120006bc0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:31.995 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:31.998+0000 7fc127e00640 1 -- 172.21.15.96:0/339926809 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc1300cefa0 con 0x7fc1300d0460 2024-05-28T06:19:32.264 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:32.267+0000 7fc135000640 1 -- 172.21.15.96:0/339926809 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fc1300b0440 con 0x7fc1300d0460 2024-05-28T06:19:32.265 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:32.268+0000 7fc127e00640 1 -- 172.21.15.96:0/339926809 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+3021 (secure 0 0 0) 0x7fc118032e20 con 0x7fc1300d0460 2024-05-28T06:19:32.266 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:19:32.266 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:19:32.266 INFO:teuthology.orchestra.run.smithi096.stdout: "fsid": "e4a831b5-e6d1-47a7-96cb-bf85154ee0d2", 2024-05-28T06:19:32.266 INFO:teuthology.orchestra.run.smithi096.stdout: "health": { 2024-05-28T06:19:32.266 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "HEALTH_OK", 2024-05-28T06:19:32.266 INFO:teuthology.orchestra.run.smithi096.stdout: "checks": {}, 2024-05-28T06:19:32.266 INFO:teuthology.orchestra.run.smithi096.stdout: "mutes": [] 2024-05-28T06:19:32.266 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:19:32.266 INFO:teuthology.orchestra.run.smithi096.stdout: "election_epoch": 8, 2024-05-28T06:19:32.267 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum": [ 2024-05-28T06:19:32.267 INFO:teuthology.orchestra.run.smithi096.stdout: 0, 2024-05-28T06:19:32.267 INFO:teuthology.orchestra.run.smithi096.stdout: 1, 2024-05-28T06:19:32.267 INFO:teuthology.orchestra.run.smithi096.stdout: 2 2024-05-28T06:19:32.267 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:19:32.267 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_names": [ 2024-05-28T06:19:32.267 INFO:teuthology.orchestra.run.smithi096.stdout: "a", 2024-05-28T06:19:32.267 INFO:teuthology.orchestra.run.smithi096.stdout: "b", 2024-05-28T06:19:32.267 INFO:teuthology.orchestra.run.smithi096.stdout: "c" 2024-05-28T06:19:32.267 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:19:32.267 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_age": 945, 2024-05-28T06:19:32.267 INFO:teuthology.orchestra.run.smithi096.stdout: "monmap": { 2024-05-28T06:19:32.267 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 1, 2024-05-28T06:19:32.268 INFO:teuthology.orchestra.run.smithi096.stdout: "min_mon_release_name": "squid", 2024-05-28T06:19:32.268 INFO:teuthology.orchestra.run.smithi096.stdout: "num_mons": 3 2024-05-28T06:19:32.268 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:19:32.268 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap": { 2024-05-28T06:19:32.268 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 91, 2024-05-28T06:19:32.268 INFO:teuthology.orchestra.run.smithi096.stdout: "num_osds": 8, 2024-05-28T06:19:32.268 INFO:teuthology.orchestra.run.smithi096.stdout: "num_up_osds": 8, 2024-05-28T06:19:32.268 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_up_since": 1716876238, 2024-05-28T06:19:32.268 INFO:teuthology.orchestra.run.smithi096.stdout: "num_in_osds": 8, 2024-05-28T06:19:32.268 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_in_since": 1716876229, 2024-05-28T06:19:32.268 INFO:teuthology.orchestra.run.smithi096.stdout: "num_remapped_pgs": 0 2024-05-28T06:19:32.268 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:19:32.269 INFO:teuthology.orchestra.run.smithi096.stdout: "pgmap": { 2024-05-28T06:19:32.269 INFO:teuthology.orchestra.run.smithi096.stdout: "pgs_by_state": [ 2024-05-28T06:19:32.269 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:19:32.269 INFO:teuthology.orchestra.run.smithi096.stdout: "state_name": "active+clean", 2024-05-28T06:19:32.269 INFO:teuthology.orchestra.run.smithi096.stdout: "count": 97 2024-05-28T06:19:32.269 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:32.269 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:19:32.269 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pgs": 97, 2024-05-28T06:19:32.269 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pools": 3, 2024-05-28T06:19:32.269 INFO:teuthology.orchestra.run.smithi096.stdout: "num_objects": 1565, 2024-05-28T06:19:32.269 INFO:teuthology.orchestra.run.smithi096.stdout: "data_bytes": 6443504700, 2024-05-28T06:19:32.269 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_used": 13777838080, 2024-05-28T06:19:32.269 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_avail": 759316275200, 2024-05-28T06:19:32.269 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_total": 773094113280, 2024-05-28T06:19:32.270 INFO:teuthology.orchestra.run.smithi096.stdout: "read_bytes_sec": 161070873, 2024-05-28T06:19:32.270 INFO:teuthology.orchestra.run.smithi096.stdout: "write_bytes_sec": 169118864, 2024-05-28T06:19:32.270 INFO:teuthology.orchestra.run.smithi096.stdout: "read_op_per_sec": 38, 2024-05-28T06:19:32.270 INFO:teuthology.orchestra.run.smithi096.stdout: "write_op_per_sec": 45 2024-05-28T06:19:32.270 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:19:32.270 INFO:teuthology.orchestra.run.smithi096.stdout: "fsmap": { 2024-05-28T06:19:32.270 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 53, 2024-05-28T06:19:32.270 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 6, 2024-05-28T06:19:32.270 INFO:teuthology.orchestra.run.smithi096.stdout: "up": 1, 2024-05-28T06:19:32.270 INFO:teuthology.orchestra.run.smithi096.stdout: "in": 1, 2024-05-28T06:19:32.270 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 1, 2024-05-28T06:19:32.270 INFO:teuthology.orchestra.run.smithi096.stdout: "by_rank": [ 2024-05-28T06:19:32.270 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:19:32.271 INFO:teuthology.orchestra.run.smithi096.stdout: "filesystem_id": 6, 2024-05-28T06:19:32.271 INFO:teuthology.orchestra.run.smithi096.stdout: "rank": 0, 2024-05-28T06:19:32.271 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "c", 2024-05-28T06:19:32.271 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "up:active", 2024-05-28T06:19:32.271 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 8103 2024-05-28T06:19:32.271 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:32.271 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:19:32.271 INFO:teuthology.orchestra.run.smithi096.stdout: "up:standby": 3 2024-05-28T06:19:32.271 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:19:32.271 INFO:teuthology.orchestra.run.smithi096.stdout: "mgrmap": { 2024-05-28T06:19:32.271 INFO:teuthology.orchestra.run.smithi096.stdout: "available": true, 2024-05-28T06:19:32.271 INFO:teuthology.orchestra.run.smithi096.stdout: "num_standbys": 1, 2024-05-28T06:19:32.271 INFO:teuthology.orchestra.run.smithi096.stdout: "modules": [ 2024-05-28T06:19:32.271 INFO:teuthology.orchestra.run.smithi096.stdout: "iostat", 2024-05-28T06:19:32.272 INFO:teuthology.orchestra.run.smithi096.stdout: "nfs", 2024-05-28T06:19:32.272 INFO:teuthology.orchestra.run.smithi096.stdout: "restful" 2024-05-28T06:19:32.272 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:19:32.272 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:19:32.272 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:19:32.272 INFO:teuthology.orchestra.run.smithi096.stdout: "servicemap": { 2024-05-28T06:19:32.272 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 16, 2024-05-28T06:19:32.272 INFO:teuthology.orchestra.run.smithi096.stdout: "modified": "2024-05-28T06:19:17.891181+0000", 2024-05-28T06:19:32.272 INFO:teuthology.orchestra.run.smithi096.stdout: "services": { 2024-05-28T06:19:32.272 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-05-28T06:19:32.272 INFO:teuthology.orchestra.run.smithi096.stdout: "daemons": { 2024-05-28T06:19:32.272 INFO:teuthology.orchestra.run.smithi096.stdout: "summary": "", 2024-05-28T06:19:32.272 INFO:teuthology.orchestra.run.smithi096.stdout: "y": { 2024-05-28T06:19:32.272 INFO:teuthology.orchestra.run.smithi096.stdout: "start_epoch": 0, 2024-05-28T06:19:32.273 INFO:teuthology.orchestra.run.smithi096.stdout: "start_stamp": "0.000000", 2024-05-28T06:19:32.273 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 0, 2024-05-28T06:19:32.273 INFO:teuthology.orchestra.run.smithi096.stdout: "addr": "(unrecognized address family 0)/0", 2024-05-28T06:19:32.273 INFO:teuthology.orchestra.run.smithi096.stdout: "metadata": {}, 2024-05-28T06:19:32.273 INFO:teuthology.orchestra.run.smithi096.stdout: "task_status": {} 2024-05-28T06:19:32.273 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:32.273 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:32.273 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:19:32.273 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-05-28T06:19:32.273 INFO:teuthology.orchestra.run.smithi096.stdout: "daemons": { 2024-05-28T06:19:32.273 INFO:teuthology.orchestra.run.smithi096.stdout: "summary": "", 2024-05-28T06:19:32.273 INFO:teuthology.orchestra.run.smithi096.stdout: "b": { 2024-05-28T06:19:32.273 INFO:teuthology.orchestra.run.smithi096.stdout: "start_epoch": 0, 2024-05-28T06:19:32.274 INFO:teuthology.orchestra.run.smithi096.stdout: "start_stamp": "0.000000", 2024-05-28T06:19:32.274 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 0, 2024-05-28T06:19:32.274 INFO:teuthology.orchestra.run.smithi096.stdout: "addr": "(unrecognized address family 0)/0", 2024-05-28T06:19:32.274 INFO:teuthology.orchestra.run.smithi096.stdout: "metadata": {}, 2024-05-28T06:19:32.274 INFO:teuthology.orchestra.run.smithi096.stdout: "task_status": {} 2024-05-28T06:19:32.274 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:32.274 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:32.274 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:32.274 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:32.274 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:19:32.274 INFO:teuthology.orchestra.run.smithi096.stdout: "progress_events": { 2024-05-28T06:19:32.274 INFO:teuthology.orchestra.run.smithi096.stdout: "7877b5be-3dc0-42ef-b4f3-0e5ec502edf5": { 2024-05-28T06:19:32.274 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "All cloning finished, waiting 20.0 seconds before exiting (16s)\n [============================] ", 2024-05-28T06:19:32.275 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": 1, 2024-05-28T06:19:32.275 INFO:teuthology.orchestra.run.smithi096.stdout: "add_to_ceph_s": true 2024-05-28T06:19:32.275 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:32.275 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:32.275 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:19:32.276 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:32.270+0000 7fc135000640 1 -- 172.21.15.96:0/339926809 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc10c06dca0 msgr2=0x7fc10c070160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:32.276 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:32.270+0000 7fc135000640 1 --2- 172.21.15.96:0/339926809 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc10c06dca0 0x7fc10c070160 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fc120006fd0 tx=0x7fc120006bc0 comp rx=0 tx=0).stop 2024-05-28T06:19:32.276 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:32.271+0000 7fc135000640 1 -- 172.21.15.96:0/339926809 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc1300d0460 msgr2=0x7fc1300c7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:32.276 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:32.271+0000 7fc135000640 1 --2- 172.21.15.96:0/339926809 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc1300d0460 0x7fc1300c7d00 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7fc1180097d0 tx=0x7fc1180046c0 comp rx=0 tx=0).stop 2024-05-28T06:19:32.276 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:32.271+0000 7fc135000640 1 -- 172.21.15.96:0/339926809 shutdown_connections 2024-05-28T06:19:32.276 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:32.271+0000 7fc135000640 1 --2- 172.21.15.96:0/339926809 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc10c06dca0 0x7fc10c070160 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:32.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:32.271+0000 7fc135000640 1 --2- 172.21.15.96:0/339926809 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc13015c6b0 0x7fc1301508b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:32.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:32.271+0000 7fc135000640 1 --2- 172.21.15.96:0/339926809 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc1300d0460 0x7fc1300c7d00 unknown :-1 s=CLOSED pgs=1492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:32.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:32.271+0000 7fc135000640 1 --2- 172.21.15.96:0/339926809 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc1300ccc30 0x7fc130154de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:32.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:32.271+0000 7fc135000640 1 -- 172.21.15.96:0/339926809 >> 172.21.15.96:0/339926809 conn(0x7fc13006bae0 msgr2=0x7fc1300c6ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:19:32.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:32.271+0000 7fc135000640 1 -- 172.21.15.96:0/339926809 shutdown_connections 2024-05-28T06:19:32.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:32.271+0000 7fc135000640 1 -- 172.21.15.96:0/339926809 wait complete. 2024-05-28T06:19:33.490 DEBUG:teuthology.orchestra.run.smithi096:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:19:33.498 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:19:37.287 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-05-28T06:19:37.496 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:37.498+0000 7f2c2fe00640 1 Processor -- start 2024-05-28T06:19:37.496 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:37.498+0000 7f2c2fe00640 1 -- start start 2024-05-28T06:19:37.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:37.499+0000 7f2c2fe00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2c2815e9e0 0x7f2c280ad950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:37.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:37.499+0000 7f2c2fe00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f2c280afa30 0x7f2c280d0260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:37.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:37.499+0000 7f2c2fe00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f2c280d07a0 0x7f2c28071cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:37.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:37.499+0000 7f2c2fe00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f2c280aee00 con 0x7f2c280d07a0 2024-05-28T06:19:37.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:37.499+0000 7f2c2fe00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f2c280ae1f0 con 0x7f2c280afa30 2024-05-28T06:19:37.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:37.499+0000 7f2c2fe00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f2c280ae4d0 con 0x7f2c2815e9e0 2024-05-28T06:19:37.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:37.500+0000 7f2c2de00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f2c280d07a0 0x7f2c28071cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:37.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:37.500+0000 7f2c2de00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f2c280d07a0 0x7f2c28071cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35898/0 (socket says 172.21.15.96:35898) 2024-05-28T06:19:37.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:37.500+0000 7f2c2de00640 1 -- 172.21.15.96:0/2852759973 learned_addr learned my addr 172.21.15.96:0/2852759973 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:19:37.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:37.500+0000 7f2c2d400640 1 --2- 172.21.15.96:0/2852759973 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2c2815e9e0 0x7f2c280ad950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:37.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:37.500+0000 7f2c2de00640 1 -- 172.21.15.96:0/2852759973 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2c2815e9e0 msgr2=0x7f2c280ad950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:19:37.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:37.500+0000 7f2c2de00640 1 --2- 172.21.15.96:0/2852759973 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2c2815e9e0 0x7f2c280ad950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:37.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:37.500+0000 7f2c2ca00640 1 --2- 172.21.15.96:0/2852759973 >> v2:172.21.15.123:3300/0 conn(0x7f2c280afa30 0x7f2c280d0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:37.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:37.500+0000 7f2c2de00640 1 -- 172.21.15.96:0/2852759973 >> v2:172.21.15.123:3300/0 conn(0x7f2c280afa30 msgr2=0x7f2c280d0260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:19:37.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:37.500+0000 7f2c2de00640 1 --2- 172.21.15.96:0/2852759973 >> v2:172.21.15.123:3300/0 conn(0x7f2c280afa30 0x7f2c280d0260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:37.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:37.500+0000 7f2c2de00640 1 -- 172.21.15.96:0/2852759973 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c280ae7b0 con 0x7f2c280d07a0 2024-05-28T06:19:37.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:37.501+0000 7f2c2de00640 1 --2- 172.21.15.96:0/2852759973 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2c280d07a0 0x7f2c28071cc0 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7f2c28072d90 tx=0x7f2c2002a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55ed76a95a5e683c server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:37.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:37.502+0000 7f2c27e00640 1 -- 172.21.15.96:0/2852759973 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c20036070 con 0x7f2c280d07a0 2024-05-28T06:19:37.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:37.502+0000 7f2c27e00640 1 -- 172.21.15.96:0/2852759973 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c20031600 con 0x7f2c280d07a0 2024-05-28T06:19:37.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:37.502+0000 7f2c27e00640 1 -- 172.21.15.96:0/2852759973 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c200319d0 con 0x7f2c280d07a0 2024-05-28T06:19:37.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:37.502+0000 7f2c2fe00640 1 -- 172.21.15.96:0/2852759973 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2c280d07a0 msgr2=0x7f2c28071cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:37.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:37.502+0000 7f2c2fe00640 1 --2- 172.21.15.96:0/2852759973 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2c280d07a0 0x7f2c28071cc0 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7f2c28072d90 tx=0x7f2c2002a730 comp rx=0 tx=0).stop 2024-05-28T06:19:37.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:37.503+0000 7f2c2fe00640 1 -- 172.21.15.96:0/2852759973 shutdown_connections 2024-05-28T06:19:37.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:37.503+0000 7f2c2fe00640 1 --2- 172.21.15.96:0/2852759973 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2c280d07a0 0x7f2c28071cc0 unknown :-1 s=CLOSED pgs=1493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:37.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:37.503+0000 7f2c2fe00640 1 --2- 172.21.15.96:0/2852759973 >> v2:172.21.15.123:3300/0 conn(0x7f2c280afa30 0x7f2c280d0260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:37.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:37.503+0000 7f2c2fe00640 1 --2- 172.21.15.96:0/2852759973 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2c2815e9e0 0x7f2c280ad950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:37.501 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:37.503+0000 7f2c2fe00640 1 -- 172.21.15.96:0/2852759973 >> 172.21.15.96:0/2852759973 conn(0x7f2c28067ce0 msgr2=0x7f2c2806b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:19:37.501 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:37.503+0000 7f2c2fe00640 1 -- 172.21.15.96:0/2852759973 shutdown_connections 2024-05-28T06:19:37.501 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:37.503+0000 7f2c2fe00640 1 -- 172.21.15.96:0/2852759973 wait complete. 2024-05-28T06:19:37.502 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:37.504+0000 7f2c2fe00640 1 Processor -- start 2024-05-28T06:19:37.502 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:37.505+0000 7f2c2fe00640 1 -- start start 2024-05-28T06:19:37.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:37.506+0000 7f2c2fe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2c280afa30 0x7f2c2814bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:37.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:37.506+0000 7f2c2fe00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2c280d07a0 0x7f2c2815edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:37.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:37.506+0000 7f2c2fe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c2815e9e0 0x7f2c280b0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:37.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:37.506+0000 7f2c2fe00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f2c28072200 con 0x7f2c280afa30 2024-05-28T06:19:37.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:37.506+0000 7f2c2fe00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f2c280ade90 con 0x7f2c2815e9e0 2024-05-28T06:19:37.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:37.506+0000 7f2c2fe00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f2c28072380 con 0x7f2c280d07a0 2024-05-28T06:19:37.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:37.506+0000 7f2c2d400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2c280afa30 0x7f2c2814bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:37.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:37.506+0000 7f2c2d400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2c280afa30 0x7f2c2814bc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35904/0 (socket says 172.21.15.96:35904) 2024-05-28T06:19:37.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:37.506+0000 7f2c2d400640 1 -- 172.21.15.96:0/450283206 learned_addr learned my addr 172.21.15.96:0/450283206 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:19:37.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:37.506+0000 7f2c2d400640 1 -- 172.21.15.96:0/450283206 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2c280d07a0 msgr2=0x7f2c2815edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:37.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:37.506+0000 7f2c2d400640 1 --2- 172.21.15.96:0/450283206 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2c280d07a0 0x7f2c2815edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:37.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:37.506+0000 7f2c2de00640 1 --2- 172.21.15.96:0/450283206 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c2815e9e0 0x7f2c280b0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:37.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:37.506+0000 7f2c2d400640 1 -- 172.21.15.96:0/450283206 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c2815e9e0 msgr2=0x7f2c280b0ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:37.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:37.506+0000 7f2c2d400640 1 --2- 172.21.15.96:0/450283206 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c2815e9e0 0x7f2c280b0ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:37.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:37.506+0000 7f2c2d400640 1 -- 172.21.15.96:0/450283206 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c281617d0 con 0x7f2c280afa30 2024-05-28T06:19:37.505 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:37.507+0000 7f2c2d400640 1 --2- 172.21.15.96:0/450283206 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2c280afa30 0x7f2c2814bc80 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7f2c1c00c7f0 tx=0x7f2c1c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:37.505 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:37.507+0000 7f2c25600640 1 -- 172.21.15.96:0/450283206 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c1c015070 con 0x7f2c280afa30 2024-05-28T06:19:37.506 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:37.507+0000 7f2c25600640 1 -- 172.21.15.96:0/450283206 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c1c004220 con 0x7f2c280afa30 2024-05-28T06:19:37.506 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:37.507+0000 7f2c25600640 1 -- 172.21.15.96:0/450283206 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c1c007620 con 0x7f2c280afa30 2024-05-28T06:19:37.506 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:37.507+0000 7f2c2fe00640 1 -- 172.21.15.96:0/450283206 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c28161ab0 con 0x7f2c280afa30 2024-05-28T06:19:37.506 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:37.507+0000 7f2c2fe00640 1 -- 172.21.15.96:0/450283206 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c2814c1c0 con 0x7f2c280afa30 2024-05-28T06:19:37.506 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:37.508+0000 7f2c2fe00640 1 -- 172.21.15.96:0/450283206 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c2815e1a0 con 0x7f2c280afa30 2024-05-28T06:19:37.506 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:37.509+0000 7f2c25600640 1 -- 172.21.15.96:0/450283206 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 33) ==== 92825+0+0 (secure 0 0 0) 0x7f2c1c0079d0 con 0x7f2c280afa30 2024-05-28T06:19:37.507 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:37.510+0000 7f2c25600640 1 --2- 172.21.15.96:0/450283206 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2c0806dca0 0x7f2c08070160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:37.507 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:37.510+0000 7f2c25600640 1 -- 172.21.15.96:0/450283206 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 5766+0+0 (secure 0 0 0) 0x7f2c1c08ae80 con 0x7f2c280afa30 2024-05-28T06:19:37.508 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:37.511+0000 7f2c2ca00640 1 --2- 172.21.15.96:0/450283206 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2c0806dca0 0x7f2c08070160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:37.509 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:37.512+0000 7f2c2ca00640 1 --2- 172.21.15.96:0/450283206 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2c0806dca0 0x7f2c08070160 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f2c14006fd0 tx=0x7f2c14008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:37.512 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:37.515+0000 7f2c25600640 1 -- 172.21.15.96:0/450283206 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2c2815e1a0 con 0x7f2c280afa30 2024-05-28T06:19:37.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:37.791+0000 7f2c2fe00640 1 -- 172.21.15.96:0/450283206 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f2c280689e0 con 0x7f2c280afa30 2024-05-28T06:19:37.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:37.792+0000 7f2c25600640 1 -- 172.21.15.96:0/450283206 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+3008 (secure 0 0 0) 0x7f2c1c05a950 con 0x7f2c280afa30 2024-05-28T06:19:37.789 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:19:37.790 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:19:37.790 INFO:teuthology.orchestra.run.smithi096.stdout: "fsid": "e4a831b5-e6d1-47a7-96cb-bf85154ee0d2", 2024-05-28T06:19:37.790 INFO:teuthology.orchestra.run.smithi096.stdout: "health": { 2024-05-28T06:19:37.790 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "HEALTH_OK", 2024-05-28T06:19:37.790 INFO:teuthology.orchestra.run.smithi096.stdout: "checks": {}, 2024-05-28T06:19:37.790 INFO:teuthology.orchestra.run.smithi096.stdout: "mutes": [] 2024-05-28T06:19:37.790 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:19:37.790 INFO:teuthology.orchestra.run.smithi096.stdout: "election_epoch": 8, 2024-05-28T06:19:37.790 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum": [ 2024-05-28T06:19:37.790 INFO:teuthology.orchestra.run.smithi096.stdout: 0, 2024-05-28T06:19:37.790 INFO:teuthology.orchestra.run.smithi096.stdout: 1, 2024-05-28T06:19:37.790 INFO:teuthology.orchestra.run.smithi096.stdout: 2 2024-05-28T06:19:37.790 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:19:37.791 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_names": [ 2024-05-28T06:19:37.791 INFO:teuthology.orchestra.run.smithi096.stdout: "a", 2024-05-28T06:19:37.791 INFO:teuthology.orchestra.run.smithi096.stdout: "b", 2024-05-28T06:19:37.791 INFO:teuthology.orchestra.run.smithi096.stdout: "c" 2024-05-28T06:19:37.791 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:19:37.791 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_age": 951, 2024-05-28T06:19:37.791 INFO:teuthology.orchestra.run.smithi096.stdout: "monmap": { 2024-05-28T06:19:37.791 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 1, 2024-05-28T06:19:37.791 INFO:teuthology.orchestra.run.smithi096.stdout: "min_mon_release_name": "squid", 2024-05-28T06:19:37.791 INFO:teuthology.orchestra.run.smithi096.stdout: "num_mons": 3 2024-05-28T06:19:37.791 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:19:37.791 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap": { 2024-05-28T06:19:37.791 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 91, 2024-05-28T06:19:37.791 INFO:teuthology.orchestra.run.smithi096.stdout: "num_osds": 8, 2024-05-28T06:19:37.792 INFO:teuthology.orchestra.run.smithi096.stdout: "num_up_osds": 8, 2024-05-28T06:19:37.792 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_up_since": 1716876238, 2024-05-28T06:19:37.792 INFO:teuthology.orchestra.run.smithi096.stdout: "num_in_osds": 8, 2024-05-28T06:19:37.792 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_in_since": 1716876229, 2024-05-28T06:19:37.792 INFO:teuthology.orchestra.run.smithi096.stdout: "num_remapped_pgs": 0 2024-05-28T06:19:37.792 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:19:37.792 INFO:teuthology.orchestra.run.smithi096.stdout: "pgmap": { 2024-05-28T06:19:37.792 INFO:teuthology.orchestra.run.smithi096.stdout: "pgs_by_state": [ 2024-05-28T06:19:37.792 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:19:37.792 INFO:teuthology.orchestra.run.smithi096.stdout: "state_name": "active+clean", 2024-05-28T06:19:37.792 INFO:teuthology.orchestra.run.smithi096.stdout: "count": 97 2024-05-28T06:19:37.792 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:37.792 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:19:37.792 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pgs": 97, 2024-05-28T06:19:37.793 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pools": 3, 2024-05-28T06:19:37.793 INFO:teuthology.orchestra.run.smithi096.stdout: "num_objects": 1565, 2024-05-28T06:19:37.793 INFO:teuthology.orchestra.run.smithi096.stdout: "data_bytes": 6443529106, 2024-05-28T06:19:37.793 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_used": 13777887232, 2024-05-28T06:19:37.793 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_avail": 759316226048, 2024-05-28T06:19:37.793 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_total": 773094113280, 2024-05-28T06:19:37.793 INFO:teuthology.orchestra.run.smithi096.stdout: "read_bytes_sec": 341, 2024-05-28T06:19:37.793 INFO:teuthology.orchestra.run.smithi096.stdout: "write_bytes_sec": 4862, 2024-05-28T06:19:37.793 INFO:teuthology.orchestra.run.smithi096.stdout: "read_op_per_sec": 0, 2024-05-28T06:19:37.793 INFO:teuthology.orchestra.run.smithi096.stdout: "write_op_per_sec": 2 2024-05-28T06:19:37.793 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:19:37.793 INFO:teuthology.orchestra.run.smithi096.stdout: "fsmap": { 2024-05-28T06:19:37.793 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 53, 2024-05-28T06:19:37.793 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 6, 2024-05-28T06:19:37.794 INFO:teuthology.orchestra.run.smithi096.stdout: "up": 1, 2024-05-28T06:19:37.794 INFO:teuthology.orchestra.run.smithi096.stdout: "in": 1, 2024-05-28T06:19:37.794 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 1, 2024-05-28T06:19:37.794 INFO:teuthology.orchestra.run.smithi096.stdout: "by_rank": [ 2024-05-28T06:19:37.794 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:19:37.794 INFO:teuthology.orchestra.run.smithi096.stdout: "filesystem_id": 6, 2024-05-28T06:19:37.794 INFO:teuthology.orchestra.run.smithi096.stdout: "rank": 0, 2024-05-28T06:19:37.794 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "c", 2024-05-28T06:19:37.794 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "up:active", 2024-05-28T06:19:37.794 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 8103 2024-05-28T06:19:37.794 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:37.794 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:19:37.794 INFO:teuthology.orchestra.run.smithi096.stdout: "up:standby": 3 2024-05-28T06:19:37.794 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:19:37.795 INFO:teuthology.orchestra.run.smithi096.stdout: "mgrmap": { 2024-05-28T06:19:37.795 INFO:teuthology.orchestra.run.smithi096.stdout: "available": true, 2024-05-28T06:19:37.795 INFO:teuthology.orchestra.run.smithi096.stdout: "num_standbys": 1, 2024-05-28T06:19:37.795 INFO:teuthology.orchestra.run.smithi096.stdout: "modules": [ 2024-05-28T06:19:37.795 INFO:teuthology.orchestra.run.smithi096.stdout: "iostat", 2024-05-28T06:19:37.795 INFO:teuthology.orchestra.run.smithi096.stdout: "nfs", 2024-05-28T06:19:37.795 INFO:teuthology.orchestra.run.smithi096.stdout: "restful" 2024-05-28T06:19:37.795 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:19:37.795 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:19:37.795 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:19:37.795 INFO:teuthology.orchestra.run.smithi096.stdout: "servicemap": { 2024-05-28T06:19:37.795 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 16, 2024-05-28T06:19:37.795 INFO:teuthology.orchestra.run.smithi096.stdout: "modified": "2024-05-28T06:19:17.891181+0000", 2024-05-28T06:19:37.796 INFO:teuthology.orchestra.run.smithi096.stdout: "services": { 2024-05-28T06:19:37.796 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-05-28T06:19:37.796 INFO:teuthology.orchestra.run.smithi096.stdout: "daemons": { 2024-05-28T06:19:37.796 INFO:teuthology.orchestra.run.smithi096.stdout: "summary": "", 2024-05-28T06:19:37.796 INFO:teuthology.orchestra.run.smithi096.stdout: "y": { 2024-05-28T06:19:37.796 INFO:teuthology.orchestra.run.smithi096.stdout: "start_epoch": 0, 2024-05-28T06:19:37.796 INFO:teuthology.orchestra.run.smithi096.stdout: "start_stamp": "0.000000", 2024-05-28T06:19:37.796 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 0, 2024-05-28T06:19:37.796 INFO:teuthology.orchestra.run.smithi096.stdout: "addr": "(unrecognized address family 0)/0", 2024-05-28T06:19:37.796 INFO:teuthology.orchestra.run.smithi096.stdout: "metadata": {}, 2024-05-28T06:19:37.796 INFO:teuthology.orchestra.run.smithi096.stdout: "task_status": {} 2024-05-28T06:19:37.796 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:37.796 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:37.796 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:19:37.797 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-05-28T06:19:37.797 INFO:teuthology.orchestra.run.smithi096.stdout: "daemons": { 2024-05-28T06:19:37.797 INFO:teuthology.orchestra.run.smithi096.stdout: "summary": "", 2024-05-28T06:19:37.797 INFO:teuthology.orchestra.run.smithi096.stdout: "b": { 2024-05-28T06:19:37.797 INFO:teuthology.orchestra.run.smithi096.stdout: "start_epoch": 0, 2024-05-28T06:19:37.797 INFO:teuthology.orchestra.run.smithi096.stdout: "start_stamp": "0.000000", 2024-05-28T06:19:37.797 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 0, 2024-05-28T06:19:37.797 INFO:teuthology.orchestra.run.smithi096.stdout: "addr": "(unrecognized address family 0)/0", 2024-05-28T06:19:37.797 INFO:teuthology.orchestra.run.smithi096.stdout: "metadata": {}, 2024-05-28T06:19:37.797 INFO:teuthology.orchestra.run.smithi096.stdout: "task_status": {} 2024-05-28T06:19:37.797 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:37.797 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:37.797 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:37.797 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:37.798 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:19:37.798 INFO:teuthology.orchestra.run.smithi096.stdout: "progress_events": { 2024-05-28T06:19:37.798 INFO:teuthology.orchestra.run.smithi096.stdout: "7877b5be-3dc0-42ef-b4f3-0e5ec502edf5": { 2024-05-28T06:19:37.798 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "All cloning finished, waiting 14.0 seconds before exiting (22s)\n [============================] ", 2024-05-28T06:19:37.798 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": 1, 2024-05-28T06:19:37.798 INFO:teuthology.orchestra.run.smithi096.stdout: "add_to_ceph_s": true 2024-05-28T06:19:37.798 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:37.798 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:37.798 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:19:37.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:37.794+0000 7f2c2fe00640 1 -- 172.21.15.96:0/450283206 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2c0806dca0 msgr2=0x7f2c08070160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:37.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:37.794+0000 7f2c2fe00640 1 --2- 172.21.15.96:0/450283206 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2c0806dca0 0x7f2c08070160 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f2c14006fd0 tx=0x7f2c14008040 comp rx=0 tx=0).stop 2024-05-28T06:19:37.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:37.794+0000 7f2c2fe00640 1 -- 172.21.15.96:0/450283206 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2c280afa30 msgr2=0x7f2c2814bc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:37.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:37.794+0000 7f2c2fe00640 1 --2- 172.21.15.96:0/450283206 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2c280afa30 0x7f2c2814bc80 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7f2c1c00c7f0 tx=0x7f2c1c00ccb0 comp rx=0 tx=0).stop 2024-05-28T06:19:37.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:37.795+0000 7f2c2fe00640 1 -- 172.21.15.96:0/450283206 shutdown_connections 2024-05-28T06:19:37.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:37.795+0000 7f2c2fe00640 1 --2- 172.21.15.96:0/450283206 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2c0806dca0 0x7f2c08070160 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:37.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:37.795+0000 7f2c2fe00640 1 --2- 172.21.15.96:0/450283206 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c2815e9e0 0x7f2c280b0ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:37.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:37.795+0000 7f2c2fe00640 1 --2- 172.21.15.96:0/450283206 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2c280d07a0 0x7f2c2815edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:37.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:37.795+0000 7f2c2fe00640 1 --2- 172.21.15.96:0/450283206 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2c280afa30 0x7f2c2814bc80 unknown :-1 s=CLOSED pgs=1494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:37.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:37.795+0000 7f2c2fe00640 1 -- 172.21.15.96:0/450283206 >> 172.21.15.96:0/450283206 conn(0x7f2c28067ce0 msgr2=0x7f2c2815da90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:19:37.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:37.795+0000 7f2c2fe00640 1 -- 172.21.15.96:0/450283206 shutdown_connections 2024-05-28T06:19:37.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:37.795+0000 7f2c2fe00640 1 -- 172.21.15.96:0/450283206 wait complete. 2024-05-28T06:19:42.813 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-05-28T06:19:43.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.022+0000 7fafc8e00640 1 Processor -- start 2024-05-28T06:19:43.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.022+0000 7fafc8e00640 1 -- start start 2024-05-28T06:19:43.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.023+0000 7fafc8e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fafc415e690 0x7fafc40d4530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:43.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.023+0000 7fafc8e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fafc40d1ca0 0x7fafc40d4a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:43.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.023+0000 7fafc8e00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fafc40d3060 con 0x7fafc415ea70 2024-05-28T06:19:43.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.023+0000 7fafc8e00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fafc40d3340 con 0x7fafc40d1ca0 2024-05-28T06:19:43.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.023+0000 7fafc8e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fafc40d3620 con 0x7fafc415e690 2024-05-28T06:19:43.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.023+0000 7fafc3e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fafc415ea70 0x7fafc40d2330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:35458/0 (socket says 172.21.15.96:35458) 2024-05-28T06:19:43.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.023+0000 7fafc3e00640 1 -- 172.21.15.96:0/2663139743 learned_addr learned my addr 172.21.15.96:0/2663139743 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:19:43.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.023+0000 7fafc2000640 1 -- 172.21.15.96:0/2663139743 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 808212694 0 0) 0x7fafc40d3060 con 0x7fafc415ea70 2024-05-28T06:19:43.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.024+0000 7fafc2000640 1 -- 172.21.15.96:0/2663139743 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fafa8003520 con 0x7fafc415ea70 2024-05-28T06:19:43.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.024+0000 7fafc2a00640 1 --2- 172.21.15.96:0/2663139743 >> v2:172.21.15.123:3300/0 conn(0x7fafc40d1ca0 0x7fafc40d4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:43.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.024+0000 7fafc3400640 1 --2- 172.21.15.96:0/2663139743 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fafc415e690 0x7fafc40d4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:43.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.024+0000 7fafc2000640 1 -- 172.21.15.96:0/2663139743 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3184982711 0 0) 0x7fafa8003520 con 0x7fafc415ea70 2024-05-28T06:19:43.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.024+0000 7fafc2000640 1 -- 172.21.15.96:0/2663139743 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fafc415e690 msgr2=0x7fafc40d4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:19:43.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.024+0000 7fafc2000640 1 --2- 172.21.15.96:0/2663139743 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fafc415e690 0x7fafc40d4530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:43.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.024+0000 7fafc2000640 1 -- 172.21.15.96:0/2663139743 >> v2:172.21.15.123:3300/0 conn(0x7fafc40d1ca0 msgr2=0x7fafc40d4a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:19:43.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.024+0000 7fafc2000640 1 --2- 172.21.15.96:0/2663139743 >> v2:172.21.15.123:3300/0 conn(0x7fafc40d1ca0 0x7fafc40d4a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:43.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.024+0000 7fafc2000640 1 -- 172.21.15.96:0/2663139743 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafc40cf770 con 0x7fafc415ea70 2024-05-28T06:19:43.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.024+0000 7fafc2000640 1 -- 172.21.15.96:0/2663139743 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7fafbc0019f0 con 0x7fafc415ea70 2024-05-28T06:19:43.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.024+0000 7fafc2000640 1 -- 172.21.15.96:0/2663139743 >> v1:172.21.15.96:6789/0 conn(0x7fafc415ea70 legacy=0x7fafc40d2330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:43.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.024+0000 7fafc2000640 1 --2- 172.21.15.96:0/2663139743 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fafa80039b0 0x7fafa8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:43.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.024+0000 7fafc2000640 1 --2- 172.21.15.96:0/2663139743 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fafc415e690 0x7fafa8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:43.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.024+0000 7fafc2000640 1 --2- 172.21.15.96:0/2663139743 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fafa8004d40 0x7fafa80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:43.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.024+0000 7fafc2000640 1 -- 172.21.15.96:0/2663139743 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fafc40d3340 con 0x7fafa8004d40 2024-05-28T06:19:43.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.024+0000 7fafc2000640 1 -- 172.21.15.96:0/2663139743 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fafc40d3620 con 0x7fafa80039b0 2024-05-28T06:19:43.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.024+0000 7fafc2000640 1 -- 172.21.15.96:0/2663139743 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fafa800b9c0 con 0x7fafc415e690 2024-05-28T06:19:43.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.024+0000 7fafc3e00640 1 --2- 172.21.15.96:0/2663139743 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fafa8004d40 0x7fafa80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:43.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.025+0000 7fafc3400640 1 --2- 172.21.15.96:0/2663139743 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fafa80039b0 0x7fafa8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:43.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.025+0000 7fafc3e00640 1 -- 172.21.15.96:0/2663139743 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fafc415e690 msgr2=0x7fafa8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:19:43.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.025+0000 7fafc2a00640 1 --2- 172.21.15.96:0/2663139743 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fafc415e690 0x7fafa8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:43.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.025+0000 7fafc3e00640 1 --2- 172.21.15.96:0/2663139743 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fafc415e690 0x7fafa8004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:43.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.025+0000 7fafc3e00640 1 -- 172.21.15.96:0/2663139743 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fafa80039b0 msgr2=0x7fafa8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:19:43.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.025+0000 7fafc3e00640 1 --2- 172.21.15.96:0/2663139743 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fafa80039b0 0x7fafa8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:43.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.025+0000 7fafc3e00640 1 -- 172.21.15.96:0/2663139743 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafa8003520 con 0x7fafa8004d40 2024-05-28T06:19:43.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.025+0000 7fafc3400640 1 --2- 172.21.15.96:0/2663139743 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fafa80039b0 0x7fafa8003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:19:43.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.026+0000 7fafc3e00640 1 --2- 172.21.15.96:0/2663139743 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fafa8004d40 0x7fafa80051c0 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7fafbc00a880 tx=0x7fafa8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff93a90c70341b68 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:43.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.026+0000 7fafc2000640 1 -- 172.21.15.96:0/2663139743 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafbc00a4a0 con 0x7fafa8004d40 2024-05-28T06:19:43.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.026+0000 7fafc2000640 1 -- 172.21.15.96:0/2663139743 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fafbc02f070 con 0x7fafa8004d40 2024-05-28T06:19:43.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.026+0000 7fafc2000640 1 -- 172.21.15.96:0/2663139743 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafbc02bcc0 con 0x7fafa8004d40 2024-05-28T06:19:43.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.027+0000 7fafc8e00640 1 -- 172.21.15.96:0/2663139743 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fafa8004d40 msgr2=0x7fafa80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:43.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.027+0000 7fafc8e00640 1 --2- 172.21.15.96:0/2663139743 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fafa8004d40 0x7fafa80051c0 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7fafbc00a880 tx=0x7fafa8005830 comp rx=0 tx=0).stop 2024-05-28T06:19:43.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.027+0000 7fafc3e00640 1 -- 172.21.15.96:0/2663139743 reap_dead start 2024-05-28T06:19:43.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.027+0000 7fafc8e00640 1 -- 172.21.15.96:0/2663139743 shutdown_connections 2024-05-28T06:19:43.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.027+0000 7fafc8e00640 1 -- 172.21.15.96:0/2663139743 >> 172.21.15.96:0/2663139743 conn(0x7fafc409d910 msgr2=0x7fafc408bfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:19:43.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.027+0000 7fafc8e00640 1 -- 172.21.15.96:0/2663139743 shutdown_connections 2024-05-28T06:19:43.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.027+0000 7fafc8e00640 1 -- 172.21.15.96:0/2663139743 wait complete. 2024-05-28T06:19:43.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.029+0000 7fafc8e00640 1 Processor -- start 2024-05-28T06:19:43.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.029+0000 7fafc8e00640 1 -- start start 2024-05-28T06:19:43.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.030+0000 7fafc8e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fafa8004d40 0x7fafc40b9920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:43.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.030+0000 7fafc8e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fafc41f8310 0x7fafc40b7150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:43.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.030+0000 7fafc8e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fafc41f8790 0x7fafc40b7690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:43.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.030+0000 7fafc8e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fafc40d46b0 con 0x7fafc41f8310 2024-05-28T06:19:43.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.030+0000 7fafc8e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fafc40d4530 con 0x7fafa8004d40 2024-05-28T06:19:43.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.030+0000 7fafc8e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fafc40d4830 con 0x7fafc41f8790 2024-05-28T06:19:43.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.030+0000 7fafc2a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fafc41f8310 0x7fafc40b7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:43.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.030+0000 7fafc2a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fafc41f8310 0x7fafc40b7150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:38526/0 (socket says 172.21.15.96:38526) 2024-05-28T06:19:43.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.030+0000 7fafc2a00640 1 -- 172.21.15.96:0/3205439586 learned_addr learned my addr 172.21.15.96:0/3205439586 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:19:43.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.030+0000 7fafc3e00640 1 --2- 172.21.15.96:0/3205439586 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fafc41f8790 0x7fafc40b7690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:43.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.030+0000 7fafc2a00640 1 -- 172.21.15.96:0/3205439586 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fafc41f8790 msgr2=0x7fafc40b7690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:43.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.030+0000 7fafc2a00640 1 --2- 172.21.15.96:0/3205439586 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fafc41f8790 0x7fafc40b7690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:43.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.030+0000 7fafc2a00640 1 -- 172.21.15.96:0/3205439586 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fafa8004d40 msgr2=0x7fafc40b9920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:43.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.030+0000 7fafc2a00640 1 --2- 172.21.15.96:0/3205439586 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fafa8004d40 0x7fafc40b9920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:43.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.030+0000 7fafc2a00640 1 -- 172.21.15.96:0/3205439586 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafc41f8d60 con 0x7fafc41f8310 2024-05-28T06:19:43.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.031+0000 7fafc2a00640 1 --2- 172.21.15.96:0/3205439586 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fafc41f8310 0x7fafc40b7150 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7fafac00e840 tx=0x7fafac00ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:43.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.031+0000 7fafbb400640 1 -- 172.21.15.96:0/3205439586 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafac017070 con 0x7fafc41f8310 2024-05-28T06:19:43.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.031+0000 7fafbb400640 1 -- 172.21.15.96:0/3205439586 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fafac005990 con 0x7fafc41f8310 2024-05-28T06:19:43.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.031+0000 7fafbb400640 1 -- 172.21.15.96:0/3205439586 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafac00f040 con 0x7fafc41f8310 2024-05-28T06:19:43.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.031+0000 7fafc8e00640 1 -- 172.21.15.96:0/3205439586 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafc41f9040 con 0x7fafc41f8310 2024-05-28T06:19:43.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.031+0000 7fafc8e00640 1 -- 172.21.15.96:0/3205439586 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafc40b4e50 con 0x7fafc41f8310 2024-05-28T06:19:43.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.032+0000 7fafc8e00640 1 -- 172.21.15.96:0/3205439586 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fafc40b4960 con 0x7fafc41f8310 2024-05-28T06:19:43.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.033+0000 7fafbb400640 1 -- 172.21.15.96:0/3205439586 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 33) ==== 92825+0+0 (secure 0 0 0) 0x7fafac00c6d0 con 0x7fafc41f8310 2024-05-28T06:19:43.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.034+0000 7fafbb400640 1 --2- 172.21.15.96:0/3205439586 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fafa006dcd0 0x7fafa0070190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:43.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.034+0000 7fafbb400640 1 -- 172.21.15.96:0/3205439586 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 5766+0+0 (secure 0 0 0) 0x7fafac08a6f0 con 0x7fafc41f8310 2024-05-28T06:19:43.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.035+0000 7fafc3400640 1 --2- 172.21.15.96:0/3205439586 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fafa006dcd0 0x7fafa0070190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:43.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.036+0000 7fafc3400640 1 --2- 172.21.15.96:0/3205439586 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fafa006dcd0 0x7fafa0070190 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fafb4009a00 tx=0x7fafb4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:43.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.040+0000 7fafbb400640 1 -- 172.21.15.96:0/3205439586 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fafac05a1c0 con 0x7fafc41f8310 2024-05-28T06:19:43.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.314+0000 7fafc8e00640 1 -- 172.21.15.96:0/3205439586 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fafc40689e0 con 0x7fafc41f8310 2024-05-28T06:19:43.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.315+0000 7fafbb400640 1 -- 172.21.15.96:0/3205439586 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+2976 (secure 0 0 0) 0x7fafac059ac0 con 0x7fafc41f8310 2024-05-28T06:19:43.311 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:19:43.311 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:19:43.312 INFO:teuthology.orchestra.run.smithi096.stdout: "fsid": "e4a831b5-e6d1-47a7-96cb-bf85154ee0d2", 2024-05-28T06:19:43.312 INFO:teuthology.orchestra.run.smithi096.stdout: "health": { 2024-05-28T06:19:43.312 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "HEALTH_OK", 2024-05-28T06:19:43.312 INFO:teuthology.orchestra.run.smithi096.stdout: "checks": {}, 2024-05-28T06:19:43.312 INFO:teuthology.orchestra.run.smithi096.stdout: "mutes": [] 2024-05-28T06:19:43.312 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:19:43.312 INFO:teuthology.orchestra.run.smithi096.stdout: "election_epoch": 8, 2024-05-28T06:19:43.312 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum": [ 2024-05-28T06:19:43.312 INFO:teuthology.orchestra.run.smithi096.stdout: 0, 2024-05-28T06:19:43.312 INFO:teuthology.orchestra.run.smithi096.stdout: 1, 2024-05-28T06:19:43.312 INFO:teuthology.orchestra.run.smithi096.stdout: 2 2024-05-28T06:19:43.312 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:19:43.312 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_names": [ 2024-05-28T06:19:43.312 INFO:teuthology.orchestra.run.smithi096.stdout: "a", 2024-05-28T06:19:43.313 INFO:teuthology.orchestra.run.smithi096.stdout: "b", 2024-05-28T06:19:43.313 INFO:teuthology.orchestra.run.smithi096.stdout: "c" 2024-05-28T06:19:43.313 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:19:43.313 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_age": 956, 2024-05-28T06:19:43.313 INFO:teuthology.orchestra.run.smithi096.stdout: "monmap": { 2024-05-28T06:19:43.313 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 1, 2024-05-28T06:19:43.313 INFO:teuthology.orchestra.run.smithi096.stdout: "min_mon_release_name": "squid", 2024-05-28T06:19:43.313 INFO:teuthology.orchestra.run.smithi096.stdout: "num_mons": 3 2024-05-28T06:19:43.313 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:19:43.313 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap": { 2024-05-28T06:19:43.313 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 91, 2024-05-28T06:19:43.313 INFO:teuthology.orchestra.run.smithi096.stdout: "num_osds": 8, 2024-05-28T06:19:43.313 INFO:teuthology.orchestra.run.smithi096.stdout: "num_up_osds": 8, 2024-05-28T06:19:43.313 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_up_since": 1716876238, 2024-05-28T06:19:43.314 INFO:teuthology.orchestra.run.smithi096.stdout: "num_in_osds": 8, 2024-05-28T06:19:43.314 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_in_since": 1716876229, 2024-05-28T06:19:43.314 INFO:teuthology.orchestra.run.smithi096.stdout: "num_remapped_pgs": 0 2024-05-28T06:19:43.314 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:19:43.314 INFO:teuthology.orchestra.run.smithi096.stdout: "pgmap": { 2024-05-28T06:19:43.314 INFO:teuthology.orchestra.run.smithi096.stdout: "pgs_by_state": [ 2024-05-28T06:19:43.314 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:19:43.314 INFO:teuthology.orchestra.run.smithi096.stdout: "state_name": "active+clean", 2024-05-28T06:19:43.314 INFO:teuthology.orchestra.run.smithi096.stdout: "count": 97 2024-05-28T06:19:43.314 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:43.314 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:19:43.314 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pgs": 97, 2024-05-28T06:19:43.314 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pools": 3, 2024-05-28T06:19:43.315 INFO:teuthology.orchestra.run.smithi096.stdout: "num_objects": 1565, 2024-05-28T06:19:43.315 INFO:teuthology.orchestra.run.smithi096.stdout: "data_bytes": 6443529106, 2024-05-28T06:19:43.315 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_used": 13777887232, 2024-05-28T06:19:43.315 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_avail": 759316226048, 2024-05-28T06:19:43.315 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_total": 773094113280, 2024-05-28T06:19:43.315 INFO:teuthology.orchestra.run.smithi096.stdout: "write_bytes_sec": 2303, 2024-05-28T06:19:43.315 INFO:teuthology.orchestra.run.smithi096.stdout: "read_op_per_sec": 0, 2024-05-28T06:19:43.315 INFO:teuthology.orchestra.run.smithi096.stdout: "write_op_per_sec": 0 2024-05-28T06:19:43.315 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:19:43.315 INFO:teuthology.orchestra.run.smithi096.stdout: "fsmap": { 2024-05-28T06:19:43.315 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 53, 2024-05-28T06:19:43.315 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 6, 2024-05-28T06:19:43.315 INFO:teuthology.orchestra.run.smithi096.stdout: "up": 1, 2024-05-28T06:19:43.315 INFO:teuthology.orchestra.run.smithi096.stdout: "in": 1, 2024-05-28T06:19:43.316 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 1, 2024-05-28T06:19:43.316 INFO:teuthology.orchestra.run.smithi096.stdout: "by_rank": [ 2024-05-28T06:19:43.316 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:19:43.316 INFO:teuthology.orchestra.run.smithi096.stdout: "filesystem_id": 6, 2024-05-28T06:19:43.316 INFO:teuthology.orchestra.run.smithi096.stdout: "rank": 0, 2024-05-28T06:19:43.316 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "c", 2024-05-28T06:19:43.316 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "up:active", 2024-05-28T06:19:43.316 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 8103 2024-05-28T06:19:43.316 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:43.316 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:19:43.316 INFO:teuthology.orchestra.run.smithi096.stdout: "up:standby": 3 2024-05-28T06:19:43.316 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:19:43.316 INFO:teuthology.orchestra.run.smithi096.stdout: "mgrmap": { 2024-05-28T06:19:43.316 INFO:teuthology.orchestra.run.smithi096.stdout: "available": true, 2024-05-28T06:19:43.317 INFO:teuthology.orchestra.run.smithi096.stdout: "num_standbys": 1, 2024-05-28T06:19:43.317 INFO:teuthology.orchestra.run.smithi096.stdout: "modules": [ 2024-05-28T06:19:43.317 INFO:teuthology.orchestra.run.smithi096.stdout: "iostat", 2024-05-28T06:19:43.317 INFO:teuthology.orchestra.run.smithi096.stdout: "nfs", 2024-05-28T06:19:43.317 INFO:teuthology.orchestra.run.smithi096.stdout: "restful" 2024-05-28T06:19:43.317 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:19:43.317 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:19:43.317 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:19:43.317 INFO:teuthology.orchestra.run.smithi096.stdout: "servicemap": { 2024-05-28T06:19:43.317 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 16, 2024-05-28T06:19:43.317 INFO:teuthology.orchestra.run.smithi096.stdout: "modified": "2024-05-28T06:19:17.891181+0000", 2024-05-28T06:19:43.317 INFO:teuthology.orchestra.run.smithi096.stdout: "services": { 2024-05-28T06:19:43.317 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-05-28T06:19:43.317 INFO:teuthology.orchestra.run.smithi096.stdout: "daemons": { 2024-05-28T06:19:43.318 INFO:teuthology.orchestra.run.smithi096.stdout: "summary": "", 2024-05-28T06:19:43.318 INFO:teuthology.orchestra.run.smithi096.stdout: "y": { 2024-05-28T06:19:43.318 INFO:teuthology.orchestra.run.smithi096.stdout: "start_epoch": 0, 2024-05-28T06:19:43.318 INFO:teuthology.orchestra.run.smithi096.stdout: "start_stamp": "0.000000", 2024-05-28T06:19:43.318 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 0, 2024-05-28T06:19:43.318 INFO:teuthology.orchestra.run.smithi096.stdout: "addr": "(unrecognized address family 0)/0", 2024-05-28T06:19:43.318 INFO:teuthology.orchestra.run.smithi096.stdout: "metadata": {}, 2024-05-28T06:19:43.318 INFO:teuthology.orchestra.run.smithi096.stdout: "task_status": {} 2024-05-28T06:19:43.318 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:43.318 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:43.318 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:19:43.318 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-05-28T06:19:43.318 INFO:teuthology.orchestra.run.smithi096.stdout: "daemons": { 2024-05-28T06:19:43.318 INFO:teuthology.orchestra.run.smithi096.stdout: "summary": "", 2024-05-28T06:19:43.319 INFO:teuthology.orchestra.run.smithi096.stdout: "b": { 2024-05-28T06:19:43.319 INFO:teuthology.orchestra.run.smithi096.stdout: "start_epoch": 0, 2024-05-28T06:19:43.319 INFO:teuthology.orchestra.run.smithi096.stdout: "start_stamp": "0.000000", 2024-05-28T06:19:43.319 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 0, 2024-05-28T06:19:43.319 INFO:teuthology.orchestra.run.smithi096.stdout: "addr": "(unrecognized address family 0)/0", 2024-05-28T06:19:43.319 INFO:teuthology.orchestra.run.smithi096.stdout: "metadata": {}, 2024-05-28T06:19:43.319 INFO:teuthology.orchestra.run.smithi096.stdout: "task_status": {} 2024-05-28T06:19:43.319 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:43.319 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:43.319 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:43.319 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:43.319 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:19:43.319 INFO:teuthology.orchestra.run.smithi096.stdout: "progress_events": { 2024-05-28T06:19:43.319 INFO:teuthology.orchestra.run.smithi096.stdout: "7877b5be-3dc0-42ef-b4f3-0e5ec502edf5": { 2024-05-28T06:19:43.320 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "All cloning finished, waiting 8.0 seconds before exiting (28s)\n [============================] ", 2024-05-28T06:19:43.320 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": 1, 2024-05-28T06:19:43.320 INFO:teuthology.orchestra.run.smithi096.stdout: "add_to_ceph_s": true 2024-05-28T06:19:43.320 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:43.320 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:43.320 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:19:43.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.317+0000 7fafc8e00640 1 -- 172.21.15.96:0/3205439586 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fafa006dcd0 msgr2=0x7fafa0070190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:43.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.317+0000 7fafc8e00640 1 --2- 172.21.15.96:0/3205439586 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fafa006dcd0 0x7fafa0070190 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fafb4009a00 tx=0x7fafb4008040 comp rx=0 tx=0).stop 2024-05-28T06:19:43.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.317+0000 7fafc8e00640 1 -- 172.21.15.96:0/3205439586 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fafc41f8310 msgr2=0x7fafc40b7150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:43.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.317+0000 7fafc8e00640 1 --2- 172.21.15.96:0/3205439586 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fafc41f8310 0x7fafc40b7150 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7fafac00e840 tx=0x7fafac00ed00 comp rx=0 tx=0).stop 2024-05-28T06:19:43.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.318+0000 7fafc8e00640 1 -- 172.21.15.96:0/3205439586 shutdown_connections 2024-05-28T06:19:43.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.318+0000 7fafc8e00640 1 --2- 172.21.15.96:0/3205439586 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fafa006dcd0 0x7fafa0070190 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:43.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.318+0000 7fafc8e00640 1 --2- 172.21.15.96:0/3205439586 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fafc41f8790 0x7fafc40b7690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:43.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.318+0000 7fafc8e00640 1 --2- 172.21.15.96:0/3205439586 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fafc41f8310 0x7fafc40b7150 unknown :-1 s=CLOSED pgs=1497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:43.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.318+0000 7fafc8e00640 1 --2- 172.21.15.96:0/3205439586 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fafa8004d40 0x7fafc40b9920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:43.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.318+0000 7fafc8e00640 1 -- 172.21.15.96:0/3205439586 >> 172.21.15.96:0/3205439586 conn(0x7fafc409d910 msgr2=0x7fafc40d11b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:19:43.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.318+0000 7fafc8e00640 1 -- 172.21.15.96:0/3205439586 shutdown_connections 2024-05-28T06:19:43.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:43.318+0000 7fafc8e00640 1 -- 172.21.15.96:0/3205439586 wait complete. 2024-05-28T06:19:48.333 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-05-28T06:19:48.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:48.546+0000 7f4015800640 1 Processor -- start 2024-05-28T06:19:48.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:48.547+0000 7f4015800640 1 -- start start 2024-05-28T06:19:48.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:48.548+0000 7f4015800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f4010161df0 0x7f4010099800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:48.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:48.548+0000 7f4015800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f40100d0b70 0x7f40100cefa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:48.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:48.548+0000 7f4015800640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40101621d0 con 0x7f40100cc2b0 2024-05-28T06:19:48.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:48.548+0000 7f4015800640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f40100cd840 con 0x7f4010161df0 2024-05-28T06:19:48.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:48.548+0000 7f4015800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f40100ccf10 con 0x7f40100d0b70 2024-05-28T06:19:48.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:48.549+0000 7f400fe00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f4010161df0 0x7f4010099800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:48.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:48.549+0000 7f4007400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f40100cc2b0 0x7f40100d0460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:35464/0 (socket says 172.21.15.96:35464) 2024-05-28T06:19:48.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:48.549+0000 7f4007400640 1 -- 172.21.15.96:0/57951109 learned_addr learned my addr 172.21.15.96:0/57951109 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:19:48.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:48.549+0000 7f4014e00640 1 --2- 172.21.15.96:0/57951109 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f40100d0b70 0x7f40100cefa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:48.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:48.549+0000 7f400fe00640 1 -- 172.21.15.96:0/57951109 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f40100d0b70 msgr2=0x7f40100cefa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:19:48.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:48.549+0000 7f400fe00640 1 --2- 172.21.15.96:0/57951109 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f40100d0b70 0x7f40100cefa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:48.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:48.549+0000 7f400fe00640 1 -- 172.21.15.96:0/57951109 >> v1:172.21.15.96:6789/0 conn(0x7f40100cc2b0 legacy=0x7f40100d0460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:48.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:48.549+0000 7f400fe00640 1 -- 172.21.15.96:0/57951109 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40100cd1f0 con 0x7f4010161df0 2024-05-28T06:19:48.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:48.549+0000 7f400f400640 1 -- 172.21.15.96:0/57951109 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3482601781 0 0) 0x7f3ff8001110 con 0x7f40100cc2b0 2024-05-28T06:19:48.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:48.551+0000 7f400fe00640 1 --2- 172.21.15.96:0/57951109 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4010161df0 0x7f4010099800 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7f40000097a0 tx=0x7f400002a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bd67c29ed8b1900 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:48.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:48.551+0000 7f400f400640 1 -- 172.21.15.96:0/57951109 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4000037070 con 0x7f4010161df0 2024-05-28T06:19:48.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:48.551+0000 7f400f400640 1 -- 172.21.15.96:0/57951109 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4000032440 con 0x7f4010161df0 2024-05-28T06:19:48.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:48.551+0000 7f400f400640 1 -- 172.21.15.96:0/57951109 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4000032810 con 0x7f4010161df0 2024-05-28T06:19:48.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:48.552+0000 7f4015800640 1 -- 172.21.15.96:0/57951109 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4010161df0 msgr2=0x7f4010099800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:48.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:48.552+0000 7f4015800640 1 --2- 172.21.15.96:0/57951109 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4010161df0 0x7f4010099800 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7f40000097a0 tx=0x7f400002a180 comp rx=0 tx=0).stop 2024-05-28T06:19:48.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:48.552+0000 7f4015800640 1 -- 172.21.15.96:0/57951109 shutdown_connections 2024-05-28T06:19:48.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:48.552+0000 7f4015800640 1 --2- 172.21.15.96:0/57951109 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f40100d0b70 0x7f40100cefa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:48.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:48.552+0000 7f4015800640 1 --2- 172.21.15.96:0/57951109 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4010161df0 0x7f4010099800 unknown :-1 s=CLOSED pgs=1499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:48.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:48.552+0000 7f4015800640 1 -- 172.21.15.96:0/57951109 >> 172.21.15.96:0/57951109 conn(0x7f401006bae0 msgr2=0x7f401006bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:19:48.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:48.552+0000 7f4015800640 1 -- 172.21.15.96:0/57951109 shutdown_connections 2024-05-28T06:19:48.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:48.552+0000 7f4015800640 1 -- 172.21.15.96:0/57951109 wait complete. 2024-05-28T06:19:48.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:48.554+0000 7f4015800640 1 Processor -- start 2024-05-28T06:19:48.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:48.554+0000 7f4015800640 1 -- start start 2024-05-28T06:19:48.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:48.555+0000 7f4015800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f40100cc2b0 0x7f40101f8090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:48.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:48.555+0000 7f4015800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40100d0b70 0x7f401014b050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:48.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:48.555+0000 7f4015800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4010161df0 0x7f401014d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:48.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:48.555+0000 7f4015800640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f40100ccbc0 con 0x7f40100d0b70 2024-05-28T06:19:48.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:48.555+0000 7f4015800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f4010099d40 con 0x7f4010161df0 2024-05-28T06:19:48.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:48.555+0000 7f4015800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f40100ca870 con 0x7f40100cc2b0 2024-05-28T06:19:48.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:48.555+0000 7f4007400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40100d0b70 0x7f401014b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:48.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:48.555+0000 7f4007400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40100d0b70 0x7f401014b050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:38550/0 (socket says 172.21.15.96:38550) 2024-05-28T06:19:48.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:48.555+0000 7f4007400640 1 -- 172.21.15.96:0/3456327440 learned_addr learned my addr 172.21.15.96:0/3456327440 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:19:48.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:48.555+0000 7f400fe00640 1 --2- 172.21.15.96:0/3456327440 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f40100cc2b0 0x7f40101f8090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:48.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:48.556+0000 7f4014e00640 1 --2- 172.21.15.96:0/3456327440 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4010161df0 0x7f401014d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:48.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:48.556+0000 7f4007400640 1 -- 172.21.15.96:0/3456327440 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f40100cc2b0 msgr2=0x7f40101f8090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:48.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:48.556+0000 7f4007400640 1 --2- 172.21.15.96:0/3456327440 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f40100cc2b0 0x7f40101f8090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:48.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:48.556+0000 7f4007400640 1 -- 172.21.15.96:0/3456327440 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4010161df0 msgr2=0x7f401014d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:48.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:48.556+0000 7f4007400640 1 --2- 172.21.15.96:0/3456327440 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4010161df0 0x7f401014d950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:48.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:48.556+0000 7f4007400640 1 -- 172.21.15.96:0/3456327440 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40100c5ce0 con 0x7f40100d0b70 2024-05-28T06:19:48.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:48.556+0000 7f4007400640 1 --2- 172.21.15.96:0/3456327440 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40100d0b70 0x7f401014b050 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f3ff800c7b0 tx=0x7f3ff800cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:48.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:48.556+0000 7f400cc00640 1 -- 172.21.15.96:0/3456327440 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ff8018070 con 0x7f40100d0b70 2024-05-28T06:19:48.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:48.556+0000 7f400cc00640 1 -- 172.21.15.96:0/3456327440 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ff80072d0 con 0x7f40100d0b70 2024-05-28T06:19:48.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:48.556+0000 7f400cc00640 1 -- 172.21.15.96:0/3456327440 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ff80047e0 con 0x7f40100d0b70 2024-05-28T06:19:48.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:48.556+0000 7f4015800640 1 -- 172.21.15.96:0/3456327440 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f401014b590 con 0x7f40100d0b70 2024-05-28T06:19:48.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:48.556+0000 7f4015800640 1 -- 172.21.15.96:0/3456327440 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f401014e480 con 0x7f40100d0b70 2024-05-28T06:19:48.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:48.558+0000 7f4015800640 1 -- 172.21.15.96:0/3456327440 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40100689e0 con 0x7f40100d0b70 2024-05-28T06:19:48.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:48.559+0000 7f400cc00640 1 -- 172.21.15.96:0/3456327440 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 33) ==== 92825+0+0 (secure 0 0 0) 0x7f3ff8004980 con 0x7f40100d0b70 2024-05-28T06:19:48.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:48.560+0000 7f400cc00640 1 --2- 172.21.15.96:0/3456327440 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3fec06dca0 0x7f3fec070160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:48.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:48.560+0000 7f400cc00640 1 -- 172.21.15.96:0/3456327440 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 5766+0+0 (secure 0 0 0) 0x7f3ff8089f60 con 0x7f40100d0b70 2024-05-28T06:19:48.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:48.561+0000 7f400fe00640 1 --2- 172.21.15.96:0/3456327440 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3fec06dca0 0x7f3fec070160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:48.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:48.562+0000 7f400fe00640 1 --2- 172.21.15.96:0/3456327440 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3fec06dca0 0x7f3fec070160 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f4000034000 tx=0x7f400002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:48.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:48.565+0000 7f400cc00640 1 -- 172.21.15.96:0/3456327440 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f40101621d0 con 0x7f40100d0b70 2024-05-28T06:19:48.827 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:48.831+0000 7f4015800640 1 -- 172.21.15.96:0/3456327440 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f40100af410 con 0x7f40100d0b70 2024-05-28T06:19:48.828 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:48.832+0000 7f400cc00640 1 -- 172.21.15.96:0/3456327440 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+2882 (secure 0 0 0) 0x7f3ff8004c70 con 0x7f40100d0b70 2024-05-28T06:19:48.828 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:19:48.828 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:19:48.828 INFO:teuthology.orchestra.run.smithi096.stdout: "fsid": "e4a831b5-e6d1-47a7-96cb-bf85154ee0d2", 2024-05-28T06:19:48.828 INFO:teuthology.orchestra.run.smithi096.stdout: "health": { 2024-05-28T06:19:48.828 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "HEALTH_OK", 2024-05-28T06:19:48.829 INFO:teuthology.orchestra.run.smithi096.stdout: "checks": {}, 2024-05-28T06:19:48.829 INFO:teuthology.orchestra.run.smithi096.stdout: "mutes": [] 2024-05-28T06:19:48.829 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:19:48.829 INFO:teuthology.orchestra.run.smithi096.stdout: "election_epoch": 8, 2024-05-28T06:19:48.829 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum": [ 2024-05-28T06:19:48.829 INFO:teuthology.orchestra.run.smithi096.stdout: 0, 2024-05-28T06:19:48.829 INFO:teuthology.orchestra.run.smithi096.stdout: 1, 2024-05-28T06:19:48.829 INFO:teuthology.orchestra.run.smithi096.stdout: 2 2024-05-28T06:19:48.829 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:19:48.829 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_names": [ 2024-05-28T06:19:48.829 INFO:teuthology.orchestra.run.smithi096.stdout: "a", 2024-05-28T06:19:48.829 INFO:teuthology.orchestra.run.smithi096.stdout: "b", 2024-05-28T06:19:48.829 INFO:teuthology.orchestra.run.smithi096.stdout: "c" 2024-05-28T06:19:48.829 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:19:48.830 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_age": 962, 2024-05-28T06:19:48.830 INFO:teuthology.orchestra.run.smithi096.stdout: "monmap": { 2024-05-28T06:19:48.830 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 1, 2024-05-28T06:19:48.830 INFO:teuthology.orchestra.run.smithi096.stdout: "min_mon_release_name": "squid", 2024-05-28T06:19:48.830 INFO:teuthology.orchestra.run.smithi096.stdout: "num_mons": 3 2024-05-28T06:19:48.830 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:19:48.830 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap": { 2024-05-28T06:19:48.830 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 91, 2024-05-28T06:19:48.830 INFO:teuthology.orchestra.run.smithi096.stdout: "num_osds": 8, 2024-05-28T06:19:48.830 INFO:teuthology.orchestra.run.smithi096.stdout: "num_up_osds": 8, 2024-05-28T06:19:48.830 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_up_since": 1716876238, 2024-05-28T06:19:48.830 INFO:teuthology.orchestra.run.smithi096.stdout: "num_in_osds": 8, 2024-05-28T06:19:48.831 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_in_since": 1716876229, 2024-05-28T06:19:48.831 INFO:teuthology.orchestra.run.smithi096.stdout: "num_remapped_pgs": 0 2024-05-28T06:19:48.831 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:19:48.831 INFO:teuthology.orchestra.run.smithi096.stdout: "pgmap": { 2024-05-28T06:19:48.831 INFO:teuthology.orchestra.run.smithi096.stdout: "pgs_by_state": [ 2024-05-28T06:19:48.831 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:19:48.831 INFO:teuthology.orchestra.run.smithi096.stdout: "state_name": "active+clean", 2024-05-28T06:19:48.831 INFO:teuthology.orchestra.run.smithi096.stdout: "count": 97 2024-05-28T06:19:48.831 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:48.831 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:19:48.831 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pgs": 97, 2024-05-28T06:19:48.831 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pools": 3, 2024-05-28T06:19:48.831 INFO:teuthology.orchestra.run.smithi096.stdout: "num_objects": 1565, 2024-05-28T06:19:48.831 INFO:teuthology.orchestra.run.smithi096.stdout: "data_bytes": 6443529106, 2024-05-28T06:19:48.831 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_used": 13777887232, 2024-05-28T06:19:48.832 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_avail": 759316226048, 2024-05-28T06:19:48.832 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_total": 773094113280 2024-05-28T06:19:48.832 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:19:48.832 INFO:teuthology.orchestra.run.smithi096.stdout: "fsmap": { 2024-05-28T06:19:48.832 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 53, 2024-05-28T06:19:48.832 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 6, 2024-05-28T06:19:48.832 INFO:teuthology.orchestra.run.smithi096.stdout: "up": 1, 2024-05-28T06:19:48.832 INFO:teuthology.orchestra.run.smithi096.stdout: "in": 1, 2024-05-28T06:19:48.832 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 1, 2024-05-28T06:19:48.832 INFO:teuthology.orchestra.run.smithi096.stdout: "by_rank": [ 2024-05-28T06:19:48.832 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:19:48.832 INFO:teuthology.orchestra.run.smithi096.stdout: "filesystem_id": 6, 2024-05-28T06:19:48.833 INFO:teuthology.orchestra.run.smithi096.stdout: "rank": 0, 2024-05-28T06:19:48.833 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "c", 2024-05-28T06:19:48.833 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "up:active", 2024-05-28T06:19:48.833 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 8103 2024-05-28T06:19:48.833 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:48.833 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:19:48.833 INFO:teuthology.orchestra.run.smithi096.stdout: "up:standby": 3 2024-05-28T06:19:48.833 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:19:48.833 INFO:teuthology.orchestra.run.smithi096.stdout: "mgrmap": { 2024-05-28T06:19:48.833 INFO:teuthology.orchestra.run.smithi096.stdout: "available": true, 2024-05-28T06:19:48.833 INFO:teuthology.orchestra.run.smithi096.stdout: "num_standbys": 1, 2024-05-28T06:19:48.833 INFO:teuthology.orchestra.run.smithi096.stdout: "modules": [ 2024-05-28T06:19:48.833 INFO:teuthology.orchestra.run.smithi096.stdout: "iostat", 2024-05-28T06:19:48.833 INFO:teuthology.orchestra.run.smithi096.stdout: "nfs", 2024-05-28T06:19:48.834 INFO:teuthology.orchestra.run.smithi096.stdout: "restful" 2024-05-28T06:19:48.834 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:19:48.834 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:19:48.834 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:19:48.834 INFO:teuthology.orchestra.run.smithi096.stdout: "servicemap": { 2024-05-28T06:19:48.834 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 16, 2024-05-28T06:19:48.834 INFO:teuthology.orchestra.run.smithi096.stdout: "modified": "2024-05-28T06:19:17.891181+0000", 2024-05-28T06:19:48.834 INFO:teuthology.orchestra.run.smithi096.stdout: "services": { 2024-05-28T06:19:48.834 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-05-28T06:19:48.834 INFO:teuthology.orchestra.run.smithi096.stdout: "daemons": { 2024-05-28T06:19:48.834 INFO:teuthology.orchestra.run.smithi096.stdout: "summary": "", 2024-05-28T06:19:48.834 INFO:teuthology.orchestra.run.smithi096.stdout: "y": { 2024-05-28T06:19:48.834 INFO:teuthology.orchestra.run.smithi096.stdout: "start_epoch": 0, 2024-05-28T06:19:48.834 INFO:teuthology.orchestra.run.smithi096.stdout: "start_stamp": "0.000000", 2024-05-28T06:19:48.835 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 0, 2024-05-28T06:19:48.835 INFO:teuthology.orchestra.run.smithi096.stdout: "addr": "(unrecognized address family 0)/0", 2024-05-28T06:19:48.835 INFO:teuthology.orchestra.run.smithi096.stdout: "metadata": {}, 2024-05-28T06:19:48.835 INFO:teuthology.orchestra.run.smithi096.stdout: "task_status": {} 2024-05-28T06:19:48.835 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:48.835 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:48.835 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:19:48.835 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-05-28T06:19:48.835 INFO:teuthology.orchestra.run.smithi096.stdout: "daemons": { 2024-05-28T06:19:48.835 INFO:teuthology.orchestra.run.smithi096.stdout: "summary": "", 2024-05-28T06:19:48.835 INFO:teuthology.orchestra.run.smithi096.stdout: "b": { 2024-05-28T06:19:48.835 INFO:teuthology.orchestra.run.smithi096.stdout: "start_epoch": 0, 2024-05-28T06:19:48.835 INFO:teuthology.orchestra.run.smithi096.stdout: "start_stamp": "0.000000", 2024-05-28T06:19:48.835 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 0, 2024-05-28T06:19:48.836 INFO:teuthology.orchestra.run.smithi096.stdout: "addr": "(unrecognized address family 0)/0", 2024-05-28T06:19:48.836 INFO:teuthology.orchestra.run.smithi096.stdout: "metadata": {}, 2024-05-28T06:19:48.836 INFO:teuthology.orchestra.run.smithi096.stdout: "task_status": {} 2024-05-28T06:19:48.836 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:48.836 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:48.836 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:48.836 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:48.836 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:19:48.836 INFO:teuthology.orchestra.run.smithi096.stdout: "progress_events": { 2024-05-28T06:19:48.836 INFO:teuthology.orchestra.run.smithi096.stdout: "7877b5be-3dc0-42ef-b4f3-0e5ec502edf5": { 2024-05-28T06:19:48.836 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "All cloning finished, waiting 2.0 seconds before exiting (34s)\n [============================] ", 2024-05-28T06:19:48.836 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": 1, 2024-05-28T06:19:48.836 INFO:teuthology.orchestra.run.smithi096.stdout: "add_to_ceph_s": true 2024-05-28T06:19:48.836 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:48.837 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:48.837 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:19:48.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:48.834+0000 7f4015800640 1 -- 172.21.15.96:0/3456327440 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3fec06dca0 msgr2=0x7f3fec070160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:48.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:48.834+0000 7f4015800640 1 --2- 172.21.15.96:0/3456327440 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3fec06dca0 0x7f3fec070160 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f4000034000 tx=0x7f400002e000 comp rx=0 tx=0).stop 2024-05-28T06:19:48.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:48.834+0000 7f4015800640 1 -- 172.21.15.96:0/3456327440 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40100d0b70 msgr2=0x7f401014b050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:48.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:48.834+0000 7f4015800640 1 --2- 172.21.15.96:0/3456327440 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40100d0b70 0x7f401014b050 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f3ff800c7b0 tx=0x7f3ff800cc70 comp rx=0 tx=0).stop 2024-05-28T06:19:48.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:48.834+0000 7f4015800640 1 -- 172.21.15.96:0/3456327440 shutdown_connections 2024-05-28T06:19:48.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:48.834+0000 7f4015800640 1 --2- 172.21.15.96:0/3456327440 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3fec06dca0 0x7f3fec070160 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:48.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:48.834+0000 7f4015800640 1 --2- 172.21.15.96:0/3456327440 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4010161df0 0x7f401014d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:48.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:48.834+0000 7f4015800640 1 --2- 172.21.15.96:0/3456327440 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40100d0b70 0x7f401014b050 unknown :-1 s=CLOSED pgs=1500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:48.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:48.834+0000 7f4015800640 1 --2- 172.21.15.96:0/3456327440 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f40100cc2b0 0x7f40101f8090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:48.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:48.834+0000 7f4015800640 1 -- 172.21.15.96:0/3456327440 >> 172.21.15.96:0/3456327440 conn(0x7f401006bae0 msgr2=0x7f401006bec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:19:48.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:48.835+0000 7f4015800640 1 -- 172.21.15.96:0/3456327440 shutdown_connections 2024-05-28T06:19:48.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:48.835+0000 7f4015800640 1 -- 172.21.15.96:0/3456327440 wait complete. 2024-05-28T06:19:53.850 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-05-28T06:19:54.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.062+0000 7fa499e00640 1 Processor -- start 2024-05-28T06:19:54.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.062+0000 7fa499e00640 1 -- start start 2024-05-28T06:19:54.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.063+0000 7fa499e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fa49406b900 0x7fa49406bce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:54.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.063+0000 7fa499e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fa4940cba20 0x7fa49406f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:54.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.063+0000 7fa499e00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4940d05f0 con 0x7fa49415ed60 2024-05-28T06:19:54.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.063+0000 7fa499e00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fa49406c220 con 0x7fa49406b900 2024-05-28T06:19:54.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.063+0000 7fa499e00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fa4940d0890 con 0x7fa4940cba20 2024-05-28T06:19:54.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.064+0000 7fa492a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fa49406b900 0x7fa49406bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:54.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.064+0000 7fa492a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fa49406b900 0x7fa49406bce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51548/0 (socket says 172.21.15.96:51548) 2024-05-28T06:19:54.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.064+0000 7fa492a00640 1 -- 172.21.15.96:0/3145586443 learned_addr learned my addr 172.21.15.96:0/3145586443 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:19:54.061 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.064+0000 7fa493400640 1 --2- 172.21.15.96:0/3145586443 >> v2:172.21.15.123:3300/0 conn(0x7fa4940cba20 0x7fa49406f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:54.061 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.064+0000 7fa492a00640 1 -- 172.21.15.96:0/3145586443 >> v2:172.21.15.123:3300/0 conn(0x7fa4940cba20 msgr2=0x7fa49406f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:19:54.061 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.064+0000 7fa492a00640 1 --2- 172.21.15.96:0/3145586443 >> v2:172.21.15.123:3300/0 conn(0x7fa4940cba20 0x7fa49406f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:54.061 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.064+0000 7fa492a00640 1 -- 172.21.15.96:0/3145586443 >> v1:172.21.15.123:6789/0 conn(0x7fa49415ed60 legacy=0x7fa49406f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:19:54.061 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.064+0000 7fa492a00640 1 -- 172.21.15.96:0/3145586443 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4940d0b70 con 0x7fa49406b900 2024-05-28T06:19:54.061 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.064+0000 7fa493400640 1 --2- 172.21.15.96:0/3145586443 >> v2:172.21.15.123:3300/0 conn(0x7fa4940cba20 0x7fa49406f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:19:54.062 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.066+0000 7fa492a00640 1 --2- 172.21.15.96:0/3145586443 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa49406b900 0x7fa49406bce0 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7fa49406c4c0 tx=0x7fa48402a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29f5aa50572ad976 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:54.062 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.066+0000 7fa492000640 1 -- 172.21.15.96:0/3145586443 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa484037070 con 0x7fa49406b900 2024-05-28T06:19:54.062 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.066+0000 7fa492000640 1 -- 172.21.15.96:0/3145586443 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa48402ae20 con 0x7fa49406b900 2024-05-28T06:19:54.062 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.066+0000 7fa492000640 1 -- 172.21.15.96:0/3145586443 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa484032660 con 0x7fa49406b900 2024-05-28T06:19:54.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.067+0000 7fa499e00640 1 -- 172.21.15.96:0/3145586443 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa49406b900 msgr2=0x7fa49406bce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:54.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.067+0000 7fa499e00640 1 --2- 172.21.15.96:0/3145586443 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa49406b900 0x7fa49406bce0 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7fa49406c4c0 tx=0x7fa48402a110 comp rx=0 tx=0).stop 2024-05-28T06:19:54.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.067+0000 7fa499e00640 1 -- 172.21.15.96:0/3145586443 shutdown_connections 2024-05-28T06:19:54.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.067+0000 7fa499e00640 1 --2- 172.21.15.96:0/3145586443 >> v2:172.21.15.123:3300/0 conn(0x7fa4940cba20 0x7fa49406f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:54.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.067+0000 7fa499e00640 1 --2- 172.21.15.96:0/3145586443 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa49406b900 0x7fa49406bce0 unknown :-1 s=CLOSED pgs=1501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:54.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.067+0000 7fa499e00640 1 -- 172.21.15.96:0/3145586443 >> 172.21.15.96:0/3145586443 conn(0x7fa49409aa30 msgr2=0x7fa4940b2f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:19:54.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.067+0000 7fa499e00640 1 -- 172.21.15.96:0/3145586443 shutdown_connections 2024-05-28T06:19:54.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.067+0000 7fa499e00640 1 -- 172.21.15.96:0/3145586443 wait complete. 2024-05-28T06:19:54.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.068+0000 7fa499e00640 1 Processor -- start 2024-05-28T06:19:54.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.071+0000 7fa499e00640 1 -- start start 2024-05-28T06:19:54.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.071+0000 7fa499e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa49406b900 0x7fa4940c4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:54.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.072+0000 7fa499e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa4940cba20 0x7fa49414bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:54.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.072+0000 7fa499e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa49415ed60 0x7fa49406ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:54.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.072+0000 7fa499e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa4940d1130 con 0x7fa49415ed60 2024-05-28T06:19:54.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.072+0000 7fa499e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa4940b3810 con 0x7fa4940cba20 2024-05-28T06:19:54.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.072+0000 7fa499e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fa49406fd30 con 0x7fa49406b900 2024-05-28T06:19:54.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.072+0000 7fa493400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa49415ed60 0x7fa49406ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:54.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.072+0000 7fa493400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa49415ed60 0x7fa49406ef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51560/0 (socket says 172.21.15.96:51560) 2024-05-28T06:19:54.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.072+0000 7fa493400640 1 -- 172.21.15.96:0/2575456320 learned_addr learned my addr 172.21.15.96:0/2575456320 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:19:54.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.072+0000 7fa48a000640 1 --2- 172.21.15.96:0/2575456320 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa4940cba20 0x7fa49414bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:54.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.072+0000 7fa493400640 1 -- 172.21.15.96:0/2575456320 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa49406b900 msgr2=0x7fa4940c4f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:54.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.072+0000 7fa493400640 1 --2- 172.21.15.96:0/2575456320 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa49406b900 0x7fa4940c4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:54.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.072+0000 7fa493400640 1 -- 172.21.15.96:0/2575456320 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa4940cba20 msgr2=0x7fa49414bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:54.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.072+0000 7fa493400640 1 --2- 172.21.15.96:0/2575456320 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa4940cba20 0x7fa49414bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:54.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.072+0000 7fa493400640 1 -- 172.21.15.96:0/2575456320 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa49414f040 con 0x7fa49415ed60 2024-05-28T06:19:54.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.073+0000 7fa493400640 1 --2- 172.21.15.96:0/2575456320 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa49415ed60 0x7fa49406ef30 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7fa48c00cd80 tx=0x7fa48c00a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:54.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.073+0000 7fa48be00640 1 -- 172.21.15.96:0/2575456320 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa48c00f680 con 0x7fa49415ed60 2024-05-28T06:19:54.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.073+0000 7fa48be00640 1 -- 172.21.15.96:0/2575456320 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa48c00f840 con 0x7fa49415ed60 2024-05-28T06:19:54.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.073+0000 7fa48be00640 1 -- 172.21.15.96:0/2575456320 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa48c00fc30 con 0x7fa49415ed60 2024-05-28T06:19:54.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.073+0000 7fa499e00640 1 -- 172.21.15.96:0/2575456320 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4940b56f0 con 0x7fa49415ed60 2024-05-28T06:19:54.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.073+0000 7fa499e00640 1 -- 172.21.15.96:0/2575456320 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4940c77d0 con 0x7fa49415ed60 2024-05-28T06:19:54.071 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.075+0000 7fa499e00640 1 -- 172.21.15.96:0/2575456320 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4940b58c0 con 0x7fa49415ed60 2024-05-28T06:19:54.071 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.075+0000 7fa48be00640 1 -- 172.21.15.96:0/2575456320 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 33) ==== 92825+0+0 (secure 0 0 0) 0x7fa48c0213f0 con 0x7fa49415ed60 2024-05-28T06:19:54.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.076+0000 7fa48be00640 1 --2- 172.21.15.96:0/2575456320 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa47006dca0 0x7fa470070160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:54.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.076+0000 7fa48be00640 1 -- 172.21.15.96:0/2575456320 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 5766+0+0 (secure 0 0 0) 0x7fa48c08b920 con 0x7fa49415ed60 2024-05-28T06:19:54.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.077+0000 7fa492a00640 1 --2- 172.21.15.96:0/2575456320 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa47006dca0 0x7fa470070160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:54.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.078+0000 7fa492a00640 1 --2- 172.21.15.96:0/2575456320 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa47006dca0 0x7fa470070160 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fa484034000 tx=0x7fa48402c000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:54.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.081+0000 7fa48be00640 1 -- 172.21.15.96:0/2575456320 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa48c014030 con 0x7fa49415ed60 2024-05-28T06:19:54.368 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.371+0000 7fa499e00640 1 -- 172.21.15.96:0/2575456320 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fa4940689e0 con 0x7fa49415ed60 2024-05-28T06:19:54.368 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.372+0000 7fa48be00640 1 -- 172.21.15.96:0/2575456320 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+2628 (secure 0 0 0) 0x7fa48c05b3f0 con 0x7fa49415ed60 2024-05-28T06:19:54.368 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:19:54.368 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:19:54.369 INFO:teuthology.orchestra.run.smithi096.stdout: "fsid": "e4a831b5-e6d1-47a7-96cb-bf85154ee0d2", 2024-05-28T06:19:54.369 INFO:teuthology.orchestra.run.smithi096.stdout: "health": { 2024-05-28T06:19:54.369 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "HEALTH_OK", 2024-05-28T06:19:54.369 INFO:teuthology.orchestra.run.smithi096.stdout: "checks": {}, 2024-05-28T06:19:54.369 INFO:teuthology.orchestra.run.smithi096.stdout: "mutes": [] 2024-05-28T06:19:54.369 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:19:54.369 INFO:teuthology.orchestra.run.smithi096.stdout: "election_epoch": 8, 2024-05-28T06:19:54.369 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum": [ 2024-05-28T06:19:54.369 INFO:teuthology.orchestra.run.smithi096.stdout: 0, 2024-05-28T06:19:54.369 INFO:teuthology.orchestra.run.smithi096.stdout: 1, 2024-05-28T06:19:54.370 INFO:teuthology.orchestra.run.smithi096.stdout: 2 2024-05-28T06:19:54.370 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:19:54.370 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_names": [ 2024-05-28T06:19:54.370 INFO:teuthology.orchestra.run.smithi096.stdout: "a", 2024-05-28T06:19:54.370 INFO:teuthology.orchestra.run.smithi096.stdout: "b", 2024-05-28T06:19:54.370 INFO:teuthology.orchestra.run.smithi096.stdout: "c" 2024-05-28T06:19:54.370 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:19:54.370 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_age": 967, 2024-05-28T06:19:54.370 INFO:teuthology.orchestra.run.smithi096.stdout: "monmap": { 2024-05-28T06:19:54.371 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 1, 2024-05-28T06:19:54.371 INFO:teuthology.orchestra.run.smithi096.stdout: "min_mon_release_name": "squid", 2024-05-28T06:19:54.371 INFO:teuthology.orchestra.run.smithi096.stdout: "num_mons": 3 2024-05-28T06:19:54.371 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:19:54.371 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap": { 2024-05-28T06:19:54.371 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 91, 2024-05-28T06:19:54.371 INFO:teuthology.orchestra.run.smithi096.stdout: "num_osds": 8, 2024-05-28T06:19:54.371 INFO:teuthology.orchestra.run.smithi096.stdout: "num_up_osds": 8, 2024-05-28T06:19:54.371 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_up_since": 1716876238, 2024-05-28T06:19:54.371 INFO:teuthology.orchestra.run.smithi096.stdout: "num_in_osds": 8, 2024-05-28T06:19:54.371 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_in_since": 1716876229, 2024-05-28T06:19:54.371 INFO:teuthology.orchestra.run.smithi096.stdout: "num_remapped_pgs": 0 2024-05-28T06:19:54.371 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:19:54.372 INFO:teuthology.orchestra.run.smithi096.stdout: "pgmap": { 2024-05-28T06:19:54.372 INFO:teuthology.orchestra.run.smithi096.stdout: "pgs_by_state": [ 2024-05-28T06:19:54.372 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:19:54.372 INFO:teuthology.orchestra.run.smithi096.stdout: "state_name": "active+clean", 2024-05-28T06:19:54.372 INFO:teuthology.orchestra.run.smithi096.stdout: "count": 97 2024-05-28T06:19:54.372 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:54.372 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:19:54.372 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pgs": 97, 2024-05-28T06:19:54.372 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pools": 3, 2024-05-28T06:19:54.372 INFO:teuthology.orchestra.run.smithi096.stdout: "num_objects": 1565, 2024-05-28T06:19:54.373 INFO:teuthology.orchestra.run.smithi096.stdout: "data_bytes": 6443529106, 2024-05-28T06:19:54.373 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_used": 13777887232, 2024-05-28T06:19:54.373 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_avail": 759316226048, 2024-05-28T06:19:54.373 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_total": 773094113280 2024-05-28T06:19:54.373 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:19:54.373 INFO:teuthology.orchestra.run.smithi096.stdout: "fsmap": { 2024-05-28T06:19:54.373 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 53, 2024-05-28T06:19:54.373 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 6, 2024-05-28T06:19:54.373 INFO:teuthology.orchestra.run.smithi096.stdout: "up": 1, 2024-05-28T06:19:54.373 INFO:teuthology.orchestra.run.smithi096.stdout: "in": 1, 2024-05-28T06:19:54.373 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 1, 2024-05-28T06:19:54.374 INFO:teuthology.orchestra.run.smithi096.stdout: "by_rank": [ 2024-05-28T06:19:54.374 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:19:54.374 INFO:teuthology.orchestra.run.smithi096.stdout: "filesystem_id": 6, 2024-05-28T06:19:54.374 INFO:teuthology.orchestra.run.smithi096.stdout: "rank": 0, 2024-05-28T06:19:54.374 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "c", 2024-05-28T06:19:54.374 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "up:active", 2024-05-28T06:19:54.374 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 8103 2024-05-28T06:19:54.374 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:54.374 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:19:54.374 INFO:teuthology.orchestra.run.smithi096.stdout: "up:standby": 3 2024-05-28T06:19:54.374 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:19:54.375 INFO:teuthology.orchestra.run.smithi096.stdout: "mgrmap": { 2024-05-28T06:19:54.375 INFO:teuthology.orchestra.run.smithi096.stdout: "available": true, 2024-05-28T06:19:54.375 INFO:teuthology.orchestra.run.smithi096.stdout: "num_standbys": 1, 2024-05-28T06:19:54.375 INFO:teuthology.orchestra.run.smithi096.stdout: "modules": [ 2024-05-28T06:19:54.375 INFO:teuthology.orchestra.run.smithi096.stdout: "iostat", 2024-05-28T06:19:54.375 INFO:teuthology.orchestra.run.smithi096.stdout: "nfs", 2024-05-28T06:19:54.375 INFO:teuthology.orchestra.run.smithi096.stdout: "restful" 2024-05-28T06:19:54.375 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:19:54.375 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:19:54.375 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:19:54.375 INFO:teuthology.orchestra.run.smithi096.stdout: "servicemap": { 2024-05-28T06:19:54.375 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 16, 2024-05-28T06:19:54.376 INFO:teuthology.orchestra.run.smithi096.stdout: "modified": "2024-05-28T06:19:17.891181+0000", 2024-05-28T06:19:54.376 INFO:teuthology.orchestra.run.smithi096.stdout: "services": { 2024-05-28T06:19:54.376 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": { 2024-05-28T06:19:54.376 INFO:teuthology.orchestra.run.smithi096.stdout: "daemons": { 2024-05-28T06:19:54.376 INFO:teuthology.orchestra.run.smithi096.stdout: "summary": "", 2024-05-28T06:19:54.376 INFO:teuthology.orchestra.run.smithi096.stdout: "y": { 2024-05-28T06:19:54.376 INFO:teuthology.orchestra.run.smithi096.stdout: "start_epoch": 0, 2024-05-28T06:19:54.376 INFO:teuthology.orchestra.run.smithi096.stdout: "start_stamp": "0.000000", 2024-05-28T06:19:54.376 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 0, 2024-05-28T06:19:54.376 INFO:teuthology.orchestra.run.smithi096.stdout: "addr": "(unrecognized address family 0)/0", 2024-05-28T06:19:54.376 INFO:teuthology.orchestra.run.smithi096.stdout: "metadata": {}, 2024-05-28T06:19:54.376 INFO:teuthology.orchestra.run.smithi096.stdout: "task_status": {} 2024-05-28T06:19:54.377 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:54.377 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:54.377 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:19:54.377 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": { 2024-05-28T06:19:54.377 INFO:teuthology.orchestra.run.smithi096.stdout: "daemons": { 2024-05-28T06:19:54.377 INFO:teuthology.orchestra.run.smithi096.stdout: "summary": "", 2024-05-28T06:19:54.377 INFO:teuthology.orchestra.run.smithi096.stdout: "b": { 2024-05-28T06:19:54.377 INFO:teuthology.orchestra.run.smithi096.stdout: "start_epoch": 0, 2024-05-28T06:19:54.377 INFO:teuthology.orchestra.run.smithi096.stdout: "start_stamp": "0.000000", 2024-05-28T06:19:54.377 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 0, 2024-05-28T06:19:54.377 INFO:teuthology.orchestra.run.smithi096.stdout: "addr": "(unrecognized address family 0)/0", 2024-05-28T06:19:54.377 INFO:teuthology.orchestra.run.smithi096.stdout: "metadata": {}, 2024-05-28T06:19:54.377 INFO:teuthology.orchestra.run.smithi096.stdout: "task_status": {} 2024-05-28T06:19:54.377 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:54.378 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:54.378 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:54.378 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:54.378 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:19:54.378 INFO:teuthology.orchestra.run.smithi096.stdout: "progress_events": {} 2024-05-28T06:19:54.378 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:19:54.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.375+0000 7fa499e00640 1 -- 172.21.15.96:0/2575456320 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa47006dca0 msgr2=0x7fa470070160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:54.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.375+0000 7fa499e00640 1 --2- 172.21.15.96:0/2575456320 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa47006dca0 0x7fa470070160 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fa484034000 tx=0x7fa48402c000 comp rx=0 tx=0).stop 2024-05-28T06:19:54.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.375+0000 7fa499e00640 1 -- 172.21.15.96:0/2575456320 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa49415ed60 msgr2=0x7fa49406ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:54.380 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.375+0000 7fa499e00640 1 --2- 172.21.15.96:0/2575456320 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa49415ed60 0x7fa49406ef30 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7fa48c00cd80 tx=0x7fa48c00a650 comp rx=0 tx=0).stop 2024-05-28T06:19:54.380 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.375+0000 7fa499e00640 1 -- 172.21.15.96:0/2575456320 shutdown_connections 2024-05-28T06:19:54.380 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.375+0000 7fa499e00640 1 --2- 172.21.15.96:0/2575456320 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa47006dca0 0x7fa470070160 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:54.380 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.375+0000 7fa499e00640 1 --2- 172.21.15.96:0/2575456320 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa49415ed60 0x7fa49406ef30 unknown :-1 s=CLOSED pgs=1502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:54.380 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.375+0000 7fa499e00640 1 --2- 172.21.15.96:0/2575456320 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa4940cba20 0x7fa49414bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:54.380 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.375+0000 7fa499e00640 1 --2- 172.21.15.96:0/2575456320 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa49406b900 0x7fa4940c4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:54.380 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.375+0000 7fa499e00640 1 -- 172.21.15.96:0/2575456320 >> 172.21.15.96:0/2575456320 conn(0x7fa49409aa30 msgr2=0x7fa4940aed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:19:54.380 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.375+0000 7fa499e00640 1 -- 172.21.15.96:0/2575456320 shutdown_connections 2024-05-28T06:19:54.380 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.375+0000 7fa499e00640 1 -- 172.21.15.96:0/2575456320 wait complete. 2024-05-28T06:19:54.392 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume ls cephfs --format json 2024-05-28T06:19:54.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.593+0000 7f3345800640 1 Processor -- start 2024-05-28T06:19:54.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.593+0000 7f3345800640 1 -- start start 2024-05-28T06:19:54.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.594+0000 7f3345800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f33400cae20 0x7f3340151db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:54.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.594+0000 7f3345800640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33400b3e80 con 0x7f33400caa40 2024-05-28T06:19:54.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.595+0000 7f3345800640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33400cde80 con 0x7f33401522f0 2024-05-28T06:19:54.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.595+0000 7f3345800640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f33400d0b30 con 0x7f33400cae20 2024-05-28T06:19:54.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.595+0000 7f333fe00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f33400caa40 0x7f33400ce960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:36962/0 (socket says 172.21.15.96:36962) 2024-05-28T06:19:54.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.595+0000 7f333fe00640 1 -- 172.21.15.96:0/3822518574 learned_addr learned my addr 172.21.15.96:0/3822518574 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:19:54.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.595+0000 7f333f400640 1 --2- 172.21.15.96:0/3822518574 >> v2:172.21.15.96:3300/0 conn(0x7f33400cae20 0x7f3340151db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:54.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.595+0000 7f333ea00640 1 -- 172.21.15.96:0/3822518574 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3419740392 0 0) 0x7f33400b3e80 con 0x7f33400caa40 2024-05-28T06:19:54.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.595+0000 7f333f400640 1 -- 172.21.15.96:0/3822518574 >> v1:172.21.15.123:6789/0 conn(0x7f33401522f0 legacy=0x7f33400cf710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:19:54.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.595+0000 7f333f400640 1 -- 172.21.15.96:0/3822518574 >> v1:172.21.15.96:6789/0 conn(0x7f33400caa40 legacy=0x7f33400ce960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:54.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.595+0000 7f333f400640 1 -- 172.21.15.96:0/3822518574 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33400ce160 con 0x7f33400cae20 2024-05-28T06:19:54.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.596+0000 7f333f400640 1 --2- 172.21.15.96:0/3822518574 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f33400cae20 0x7f3340151db0 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7f332c0097a0 tx=0x7f332c02c350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=503ceb4cde05c429 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:54.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.597+0000 7f333ea00640 1 -- 172.21.15.96:0/3822518574 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f332c039070 con 0x7f33400cae20 2024-05-28T06:19:54.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.597+0000 7f333ea00640 1 -- 172.21.15.96:0/3822518574 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f332c034640 con 0x7f33400cae20 2024-05-28T06:19:54.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.597+0000 7f333ea00640 1 -- 172.21.15.96:0/3822518574 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f332c02ce60 con 0x7f33400cae20 2024-05-28T06:19:54.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.597+0000 7f3345800640 1 -- 172.21.15.96:0/3822518574 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f33400cae20 msgr2=0x7f3340151db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:54.594 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.597+0000 7f3345800640 1 --2- 172.21.15.96:0/3822518574 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f33400cae20 0x7f3340151db0 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7f332c0097a0 tx=0x7f332c02c350 comp rx=0 tx=0).stop 2024-05-28T06:19:54.594 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.597+0000 7f3345800640 1 -- 172.21.15.96:0/3822518574 shutdown_connections 2024-05-28T06:19:54.594 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.597+0000 7f3345800640 1 --2- 172.21.15.96:0/3822518574 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f33400cae20 0x7f3340151db0 unknown :-1 s=CLOSED pgs=1504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:54.594 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.597+0000 7f3345800640 1 -- 172.21.15.96:0/3822518574 >> 172.21.15.96:0/3822518574 conn(0x7f334006b900 msgr2=0x7f334006bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:19:54.594 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.597+0000 7f3345800640 1 -- 172.21.15.96:0/3822518574 shutdown_connections 2024-05-28T06:19:54.594 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.597+0000 7f3345800640 1 -- 172.21.15.96:0/3822518574 wait complete. 2024-05-28T06:19:54.595 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.599+0000 7f3345800640 1 Processor -- start 2024-05-28T06:19:54.595 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.599+0000 7f3345800640 1 -- start start 2024-05-28T06:19:54.596 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.600+0000 7f3345800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f33400caa40 0x7f3340156840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:54.596 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.600+0000 7f3345800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f33400cae20 0x7f3340152920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:54.596 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.600+0000 7f3345800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f33401522f0 0x7f334014b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:54.596 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.600+0000 7f3345800640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f33400cf070 con 0x7f33400cae20 2024-05-28T06:19:54.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.600+0000 7f3345800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f33400b4160 con 0x7f33401522f0 2024-05-28T06:19:54.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.600+0000 7f3345800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f3340152710 con 0x7f33400caa40 2024-05-28T06:19:54.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.600+0000 7f333f400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f33400cae20 0x7f3340152920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:54.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.600+0000 7f333f400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f33400cae20 0x7f3340152920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51588/0 (socket says 172.21.15.96:51588) 2024-05-28T06:19:54.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.600+0000 7f333f400640 1 -- 172.21.15.96:0/938349422 learned_addr learned my addr 172.21.15.96:0/938349422 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:19:54.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.601+0000 7f333f400640 1 -- 172.21.15.96:0/938349422 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f33400caa40 msgr2=0x7f3340156840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:54.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.601+0000 7f333f400640 1 --2- 172.21.15.96:0/938349422 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f33400caa40 0x7f3340156840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:54.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.601+0000 7f333f400640 1 -- 172.21.15.96:0/938349422 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f33401522f0 msgr2=0x7f334014b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:54.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.601+0000 7f333f400640 1 --2- 172.21.15.96:0/938349422 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f33401522f0 0x7f334014b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:54.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.601+0000 7f333f400640 1 -- 172.21.15.96:0/938349422 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3340072e40 con 0x7f33400cae20 2024-05-28T06:19:54.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.601+0000 7f333f400640 1 --2- 172.21.15.96:0/938349422 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f33400cae20 0x7f3340152920 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f332c0097a0 tx=0x7f332c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:54.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.601+0000 7f3337e00640 1 -- 172.21.15.96:0/938349422 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f332c039040 con 0x7f33400cae20 2024-05-28T06:19:54.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.601+0000 7f3337e00640 1 -- 172.21.15.96:0/938349422 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f332c004030 con 0x7f33400cae20 2024-05-28T06:19:54.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.601+0000 7f3337e00640 1 -- 172.21.15.96:0/938349422 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f332c02ce60 con 0x7f33400cae20 2024-05-28T06:19:54.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.601+0000 7f3345800640 1 -- 172.21.15.96:0/938349422 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f334014f7c0 con 0x7f33400cae20 2024-05-28T06:19:54.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.602+0000 7f3345800640 1 -- 172.21.15.96:0/938349422 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f334014fd80 con 0x7f33400cae20 2024-05-28T06:19:54.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.603+0000 7f3345800640 1 -- 172.21.15.96:0/938349422 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f334015b2b0 con 0x7f33400cae20 2024-05-28T06:19:54.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.604+0000 7f3337e00640 1 -- 172.21.15.96:0/938349422 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 33) ==== 92825+0+0 (secure 0 0 0) 0x7f332c032770 con 0x7f33400cae20 2024-05-28T06:19:54.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.604+0000 7f3337e00640 1 --2- 172.21.15.96:0/938349422 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f332006dcd0 0x7f3320070190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:54.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.605+0000 7f3337e00640 1 -- 172.21.15.96:0/938349422 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 5766+0+0 (secure 0 0 0) 0x7f332c0ab530 con 0x7f33400cae20 2024-05-28T06:19:54.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.605+0000 7f333fe00640 1 --2- 172.21.15.96:0/938349422 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f332006dcd0 0x7f3320070190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:54.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.606+0000 7f333fe00640 1 --2- 172.21.15.96:0/938349422 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f332006dcd0 0x7f3320070190 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f3330007fd0 tx=0x7f3330007920 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:54.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.609+0000 7f3337e00640 1 -- 172.21.15.96:0/938349422 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f334015b2b0 con 0x7f33400cae20 2024-05-28T06:19:54.826 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.830+0000 7f3345800640 1 -- 172.21.15.96:0/938349422 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume ls", "vol_name": "cephfs", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f33400689e0 con 0x7f332006dcd0 2024-05-28T06:19:54.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.842+0000 7f3337e00640 1 -- 172.21.15.96:0/938349422 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+79 (secure 0 0 0) 0x7f33400689e0 con 0x7f332006dcd0 2024-05-28T06:19:54.838 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:19:54.838 INFO:teuthology.orchestra.run.smithi096.stdout:[ 2024-05-28T06:19:54.838 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:19:54.838 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "sv1" 2024-05-28T06:19:54.838 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:19:54.838 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:19:54.838 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "ss1clone1" 2024-05-28T06:19:54.838 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:54.838 INFO:teuthology.orchestra.run.smithi096.stdout:] 2024-05-28T06:19:54.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.844+0000 7f3345800640 1 -- 172.21.15.96:0/938349422 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f332006dcd0 msgr2=0x7f3320070190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:54.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.844+0000 7f3345800640 1 --2- 172.21.15.96:0/938349422 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f332006dcd0 0x7f3320070190 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f3330007fd0 tx=0x7f3330007920 comp rx=0 tx=0).stop 2024-05-28T06:19:54.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.844+0000 7f3345800640 1 -- 172.21.15.96:0/938349422 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f33400cae20 msgr2=0x7f3340152920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:54.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.844+0000 7f3345800640 1 --2- 172.21.15.96:0/938349422 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f33400cae20 0x7f3340152920 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f332c0097a0 tx=0x7f332c0046c0 comp rx=0 tx=0).stop 2024-05-28T06:19:54.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.844+0000 7f3345800640 1 -- 172.21.15.96:0/938349422 shutdown_connections 2024-05-28T06:19:54.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.844+0000 7f3345800640 1 --2- 172.21.15.96:0/938349422 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f332006dcd0 0x7f3320070190 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:54.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.844+0000 7f3345800640 1 --2- 172.21.15.96:0/938349422 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f33401522f0 0x7f334014b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:54.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.844+0000 7f3345800640 1 --2- 172.21.15.96:0/938349422 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f33400cae20 0x7f3340152920 unknown :-1 s=CLOSED pgs=1505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:54.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.844+0000 7f3345800640 1 --2- 172.21.15.96:0/938349422 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f33400caa40 0x7f3340156840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:54.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.844+0000 7f3345800640 1 -- 172.21.15.96:0/938349422 >> 172.21.15.96:0/938349422 conn(0x7f334006b900 msgr2=0x7f334015aba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:19:54.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.845+0000 7f3345800640 1 -- 172.21.15.96:0/938349422 shutdown_connections 2024-05-28T06:19:54.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:54.845+0000 7f3345800640 1 -- 172.21.15.96:0/938349422 wait complete. 2024-05-28T06:19:54.859 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot ls cephfs sv1 --format json 2024-05-28T06:19:55.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.063+0000 7ff075200640 1 Processor -- start 2024-05-28T06:19:55.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.063+0000 7ff075200640 1 -- start start 2024-05-28T06:19:55.061 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.064+0000 7ff075200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7ff070140500 0x7ff0700ad130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:55.061 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.064+0000 7ff075200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0700d0020 con 0x7ff0700cf2f0 2024-05-28T06:19:55.061 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.064+0000 7ff075200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff07006dd50 con 0x7ff07006edf0 2024-05-28T06:19:55.061 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.064+0000 7ff075200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7ff07006e030 con 0x7ff070140500 2024-05-28T06:19:55.061 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.065+0000 7ff06de00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7ff0700cf2f0 0x7ff0700cf910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:36964/0 (socket says 172.21.15.96:36964) 2024-05-28T06:19:55.061 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.065+0000 7ff06de00640 1 -- 172.21.15.96:0/1364519451 learned_addr learned my addr 172.21.15.96:0/1364519451 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:19:55.061 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.065+0000 7ff06e800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7ff070140500 0x7ff0700ad130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:55.062 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.065+0000 7ff06e800640 1 -- 172.21.15.96:0/1364519451 >> v1:172.21.15.123:6789/0 conn(0x7ff07006edf0 legacy=0x7ff07013fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:19:55.062 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.065+0000 7ff06e800640 1 -- 172.21.15.96:0/1364519451 >> v1:172.21.15.96:6789/0 conn(0x7ff0700cf2f0 legacy=0x7ff0700cf910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:55.062 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.065+0000 7ff06e800640 1 -- 172.21.15.96:0/1364519451 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff07006e310 con 0x7ff070140500 2024-05-28T06:19:55.062 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.065+0000 7ff06d400640 1 -- 172.21.15.96:0/1364519451 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 468738381 0 0) 0x7ff0700d0020 con 0x7ff0700cf2f0 2024-05-28T06:19:55.062 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.066+0000 7ff06e800640 1 --2- 172.21.15.96:0/1364519451 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff070140500 0x7ff0700ad130 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7ff068009d90 tx=0x7ff06802a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10f313de496eecdb server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:55.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.066+0000 7ff06d400640 1 -- 172.21.15.96:0/1364519451 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff068031600 con 0x7ff070140500 2024-05-28T06:19:55.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.067+0000 7ff06d400640 1 -- 172.21.15.96:0/1364519451 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0680317a0 con 0x7ff070140500 2024-05-28T06:19:55.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.067+0000 7ff06d400640 1 -- 172.21.15.96:0/1364519451 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff068031b70 con 0x7ff070140500 2024-05-28T06:19:55.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.067+0000 7ff075200640 1 -- 172.21.15.96:0/1364519451 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff070140500 msgr2=0x7ff0700ad130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:55.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.067+0000 7ff075200640 1 --2- 172.21.15.96:0/1364519451 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff070140500 0x7ff0700ad130 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7ff068009d90 tx=0x7ff06802a610 comp rx=0 tx=0).stop 2024-05-28T06:19:55.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.067+0000 7ff075200640 1 -- 172.21.15.96:0/1364519451 shutdown_connections 2024-05-28T06:19:55.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.067+0000 7ff075200640 1 --2- 172.21.15.96:0/1364519451 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff070140500 0x7ff0700ad130 unknown :-1 s=CLOSED pgs=1507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:55.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.067+0000 7ff075200640 1 -- 172.21.15.96:0/1364519451 >> 172.21.15.96:0/1364519451 conn(0x7ff0700b1e50 msgr2=0x7ff0700b2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:19:55.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.067+0000 7ff075200640 1 -- 172.21.15.96:0/1364519451 shutdown_connections 2024-05-28T06:19:55.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.067+0000 7ff075200640 1 -- 172.21.15.96:0/1364519451 wait complete. 2024-05-28T06:19:55.065 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.069+0000 7ff075200640 1 Processor -- start 2024-05-28T06:19:55.065 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.069+0000 7ff075200640 1 -- start start 2024-05-28T06:19:55.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.070+0000 7ff075200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff07006edf0 0x7ff070073380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:55.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.070+0000 7ff075200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0700cf2f0 0x7ff07015cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:55.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.070+0000 7ff075200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff0701541b0 0x7ff07015d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:55.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.070+0000 7ff075200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff070146020 con 0x7ff0701541b0 2024-05-28T06:19:55.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.070+0000 7ff075200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff07006e8d0 con 0x7ff0700cf2f0 2024-05-28T06:19:55.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.070+0000 7ff075200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ff0701461a0 con 0x7ff07006edf0 2024-05-28T06:19:55.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.070+0000 7ff06e800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff0701541b0 0x7ff07015d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:55.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.070+0000 7ff06e800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff0701541b0 0x7ff07015d1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51616/0 (socket says 172.21.15.96:51616) 2024-05-28T06:19:55.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.070+0000 7ff06e800640 1 -- 172.21.15.96:0/1891101861 learned_addr learned my addr 172.21.15.96:0/1891101861 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:19:55.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.071+0000 7ff06de00640 1 --2- 172.21.15.96:0/1891101861 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff07006edf0 0x7ff070073380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:55.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.071+0000 7ff06e800640 1 -- 172.21.15.96:0/1891101861 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff07006edf0 msgr2=0x7ff070073380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:55.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.071+0000 7ff06e800640 1 --2- 172.21.15.96:0/1891101861 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff07006edf0 0x7ff070073380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:55.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.071+0000 7ff06e800640 1 -- 172.21.15.96:0/1891101861 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0700cf2f0 msgr2=0x7ff07015cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:55.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.071+0000 7ff06e800640 1 --2- 172.21.15.96:0/1891101861 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0700cf2f0 0x7ff07015cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:55.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.071+0000 7ff06e800640 1 -- 172.21.15.96:0/1891101861 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0700cb7b0 con 0x7ff0701541b0 2024-05-28T06:19:55.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.071+0000 7ff06e800640 1 --2- 172.21.15.96:0/1891101861 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff0701541b0 0x7ff07015d1b0 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7ff068009d90 tx=0x7ff068004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:55.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.071+0000 7ff066000640 1 -- 172.21.15.96:0/1891101861 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff068040070 con 0x7ff0701541b0 2024-05-28T06:19:55.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.071+0000 7ff066000640 1 -- 172.21.15.96:0/1891101861 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff068030be0 con 0x7ff0701541b0 2024-05-28T06:19:55.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.071+0000 7ff066000640 1 -- 172.21.15.96:0/1891101861 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff068009450 con 0x7ff0701541b0 2024-05-28T06:19:55.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.071+0000 7ff075200640 1 -- 172.21.15.96:0/1891101861 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0700cba90 con 0x7ff0701541b0 2024-05-28T06:19:55.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.071+0000 7ff075200640 1 -- 172.21.15.96:0/1891101861 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff070072e60 con 0x7ff0701541b0 2024-05-28T06:19:55.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.073+0000 7ff075200640 1 -- 172.21.15.96:0/1891101861 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0700cbf50 con 0x7ff0701541b0 2024-05-28T06:19:55.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.074+0000 7ff066000640 1 -- 172.21.15.96:0/1891101861 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 33) ==== 92825+0+0 (secure 0 0 0) 0x7ff068039040 con 0x7ff0701541b0 2024-05-28T06:19:55.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.074+0000 7ff066000640 1 --2- 172.21.15.96:0/1891101861 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff04c06dcd0 0x7ff04c070190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:55.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.074+0000 7ff066000640 1 -- 172.21.15.96:0/1891101861 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 5766+0+0 (secure 0 0 0) 0x7ff0680abc20 con 0x7ff0701541b0 2024-05-28T06:19:55.071 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.075+0000 7ff06de00640 1 --2- 172.21.15.96:0/1891101861 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff04c06dcd0 0x7ff04c070190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:55.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.076+0000 7ff06de00640 1 --2- 172.21.15.96:0/1891101861 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff04c06dcd0 0x7ff04c070190 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7ff060007fd0 tx=0x7ff060009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:55.075 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.079+0000 7ff066000640 1 -- 172.21.15.96:0/1891101861 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff0700cbf50 con 0x7ff0701541b0 2024-05-28T06:19:55.308 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.311+0000 7ff075200640 1 -- 172.21.15.96:0/1891101861 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot ls", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff0700b2b00 con 0x7ff04c06dcd0 2024-05-28T06:19:55.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.339+0000 7ff066000640 1 -- 172.21.15.96:0/1891101861 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7ff0700b2b00 con 0x7ff04c06dcd0 2024-05-28T06:19:55.336 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:19:55.336 INFO:teuthology.orchestra.run.smithi096.stdout:[ 2024-05-28T06:19:55.336 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:19:55.336 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "ss1" 2024-05-28T06:19:55.336 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:19:55.337 INFO:teuthology.orchestra.run.smithi096.stdout:] 2024-05-28T06:19:55.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.342+0000 7ff075200640 1 -- 172.21.15.96:0/1891101861 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff04c06dcd0 msgr2=0x7ff04c070190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:55.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.342+0000 7ff075200640 1 --2- 172.21.15.96:0/1891101861 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff04c06dcd0 0x7ff04c070190 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7ff060007fd0 tx=0x7ff060009040 comp rx=0 tx=0).stop 2024-05-28T06:19:55.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.342+0000 7ff075200640 1 -- 172.21.15.96:0/1891101861 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff0701541b0 msgr2=0x7ff07015d1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:55.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.342+0000 7ff075200640 1 --2- 172.21.15.96:0/1891101861 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff0701541b0 0x7ff07015d1b0 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7ff068009d90 tx=0x7ff068004a40 comp rx=0 tx=0).stop 2024-05-28T06:19:55.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.342+0000 7ff075200640 1 -- 172.21.15.96:0/1891101861 shutdown_connections 2024-05-28T06:19:55.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.342+0000 7ff075200640 1 --2- 172.21.15.96:0/1891101861 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff04c06dcd0 0x7ff04c070190 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:55.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.342+0000 7ff075200640 1 --2- 172.21.15.96:0/1891101861 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff0701541b0 0x7ff07015d1b0 unknown :-1 s=CLOSED pgs=1508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:55.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.342+0000 7ff075200640 1 --2- 172.21.15.96:0/1891101861 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0700cf2f0 0x7ff07015cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:55.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.342+0000 7ff075200640 1 --2- 172.21.15.96:0/1891101861 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff07006edf0 0x7ff070073380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:55.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.342+0000 7ff075200640 1 -- 172.21.15.96:0/1891101861 >> 172.21.15.96:0/1891101861 conn(0x7ff0700b1e50 msgr2=0x7ff07015bc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:19:55.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.342+0000 7ff075200640 1 -- 172.21.15.96:0/1891101861 shutdown_connections 2024-05-28T06:19:55.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.342+0000 7ff075200640 1 -- 172.21.15.96:0/1891101861 wait complete. 2024-05-28T06:19:55.357 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-05-28T06:19:55.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.568+0000 7fd73fa00640 1 Processor -- start 2024-05-28T06:19:55.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.568+0000 7fd73fa00640 1 -- start start 2024-05-28T06:19:55.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.570+0000 7fd73fa00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd7380b4120 0x7fd738163ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:55.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.570+0000 7fd73fa00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7381640e0 con 0x7fd738162ac0 2024-05-28T06:19:55.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.570+0000 7fd73fa00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7380b3360 con 0x7fd7380d0d50 2024-05-28T06:19:55.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.570+0000 7fd73fa00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fd7380b3c90 con 0x7fd7380b4120 2024-05-28T06:19:55.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.570+0000 7fd73d000640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fd738162ac0 0x7fd7380d0640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:36966/0 (socket says 172.21.15.96:36966) 2024-05-28T06:19:55.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.570+0000 7fd73d000640 1 -- 172.21.15.96:0/1972379347 learned_addr learned my addr 172.21.15.96:0/1972379347 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:19:55.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.571+0000 7fd73da00640 1 --2- 172.21.15.96:0/1972379347 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd7380b4120 0x7fd738163ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:55.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.571+0000 7fd737400640 1 -- 172.21.15.96:0/1972379347 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4224775027 0 0) 0x7fd7381640e0 con 0x7fd738162ac0 2024-05-28T06:19:55.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.571+0000 7fd737400640 1 -- 172.21.15.96:0/1972379347 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd720003520 con 0x7fd738162ac0 2024-05-28T06:19:55.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.571+0000 7fd737400640 1 -- 172.21.15.96:0/1972379347 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2677308088 0 0) 0x7fd720003520 con 0x7fd738162ac0 2024-05-28T06:19:55.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.571+0000 7fd737400640 1 -- 172.21.15.96:0/1972379347 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd7380b4120 msgr2=0x7fd738163ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:19:55.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.571+0000 7fd737400640 1 --2- 172.21.15.96:0/1972379347 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd7380b4120 0x7fd738163ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:55.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.571+0000 7fd737400640 1 -- 172.21.15.96:0/1972379347 >> v1:172.21.15.123:6789/0 conn(0x7fd7380d0d50 legacy=0x7fd7381633b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:55.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.571+0000 7fd737400640 1 -- 172.21.15.96:0/1972379347 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7380b3640 con 0x7fd738162ac0 2024-05-28T06:19:55.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.571+0000 7fd737400640 1 -- 172.21.15.96:0/1972379347 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7fd72c001670 con 0x7fd738162ac0 2024-05-28T06:19:55.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.571+0000 7fd737400640 1 -- 172.21.15.96:0/1972379347 >> v1:172.21.15.96:6789/0 conn(0x7fd738162ac0 legacy=0x7fd7380d0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:55.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.571+0000 7fd737400640 1 --2- 172.21.15.96:0/1972379347 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd7380b4120 0x7fd7200039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:55.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.571+0000 7fd737400640 1 --2- 172.21.15.96:0/1972379347 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd720003f60 0x7fd720004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:55.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.571+0000 7fd737400640 1 --2- 172.21.15.96:0/1972379347 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd720004d40 0x7fd7200051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:55.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.571+0000 7fd737400640 1 -- 172.21.15.96:0/1972379347 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd7380b3c90 con 0x7fd720004d40 2024-05-28T06:19:55.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.571+0000 7fd737400640 1 -- 172.21.15.96:0/1972379347 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd72000baa0 con 0x7fd720003f60 2024-05-28T06:19:55.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.571+0000 7fd737400640 1 -- 172.21.15.96:0/1972379347 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fd72000bca0 con 0x7fd7380b4120 2024-05-28T06:19:55.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.572+0000 7fd73da00640 1 --2- 172.21.15.96:0/1972379347 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd720004d40 0x7fd7200051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:55.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.572+0000 7fd73d000640 1 --2- 172.21.15.96:0/1972379347 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd7380b4120 0x7fd7200039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:55.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.572+0000 7fd737e00640 1 --2- 172.21.15.96:0/1972379347 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd720003f60 0x7fd720004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:55.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.572+0000 7fd73da00640 1 -- 172.21.15.96:0/1972379347 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd7380b4120 msgr2=0x7fd7200039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:19:55.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.572+0000 7fd73da00640 1 --2- 172.21.15.96:0/1972379347 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd7380b4120 0x7fd7200039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:55.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.572+0000 7fd73da00640 1 -- 172.21.15.96:0/1972379347 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd720003f60 msgr2=0x7fd720004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:19:55.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.572+0000 7fd73da00640 1 --2- 172.21.15.96:0/1972379347 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd720003f60 0x7fd720004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:55.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.572+0000 7fd73da00640 1 -- 172.21.15.96:0/1972379347 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd720003520 con 0x7fd720004d40 2024-05-28T06:19:55.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.573+0000 7fd73da00640 1 --2- 172.21.15.96:0/1972379347 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd720004d40 0x7fd7200051c0 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7fd730009f90 tx=0x7fd720005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34425c0290449920 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:55.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.573+0000 7fd737400640 1 -- 172.21.15.96:0/1972379347 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd730009c10 con 0x7fd720004d40 2024-05-28T06:19:55.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.573+0000 7fd737400640 1 -- 172.21.15.96:0/1972379347 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd730036070 con 0x7fd720004d40 2024-05-28T06:19:55.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.573+0000 7fd737400640 1 -- 172.21.15.96:0/1972379347 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd730031850 con 0x7fd720004d40 2024-05-28T06:19:55.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.574+0000 7fd73fa00640 1 -- 172.21.15.96:0/1972379347 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd720004d40 msgr2=0x7fd7200051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:55.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.574+0000 7fd73fa00640 1 --2- 172.21.15.96:0/1972379347 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd720004d40 0x7fd7200051c0 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7fd730009f90 tx=0x7fd720005830 comp rx=0 tx=0).stop 2024-05-28T06:19:55.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.574+0000 7fd73da00640 1 -- 172.21.15.96:0/1972379347 reap_dead start 2024-05-28T06:19:55.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.574+0000 7fd73fa00640 1 -- 172.21.15.96:0/1972379347 shutdown_connections 2024-05-28T06:19:55.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.574+0000 7fd73fa00640 1 -- 172.21.15.96:0/1972379347 >> 172.21.15.96:0/1972379347 conn(0x7fd73806bd20 msgr2=0x7fd73806c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:19:55.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.574+0000 7fd73fa00640 1 -- 172.21.15.96:0/1972379347 shutdown_connections 2024-05-28T06:19:55.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.574+0000 7fd73fa00640 1 -- 172.21.15.96:0/1972379347 wait complete. 2024-05-28T06:19:55.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.576+0000 7fd73fa00640 1 Processor -- start 2024-05-28T06:19:55.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.576+0000 7fd73fa00640 1 -- start start 2024-05-28T06:19:55.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.577+0000 7fd73fa00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd720004d40 0x7fd7381528a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:55.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.577+0000 7fd73fa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd73815c950 0x7fd738160e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:55.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.577+0000 7fd73fa00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd7380ce940 0x7fd7380cedf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:55.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.577+0000 7fd73fa00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd738163020 con 0x7fd73815c950 2024-05-28T06:19:55.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.577+0000 7fd73fa00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd738162ea0 con 0x7fd720004d40 2024-05-28T06:19:55.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.577+0000 7fd73fa00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fd738163ac0 con 0x7fd7380ce940 2024-05-28T06:19:55.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.577+0000 7fd737e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd73815c950 0x7fd738160e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:55.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.577+0000 7fd737e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd73815c950 0x7fd738160e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51632/0 (socket says 172.21.15.96:51632) 2024-05-28T06:19:55.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.577+0000 7fd737e00640 1 -- 172.21.15.96:0/3059543635 learned_addr learned my addr 172.21.15.96:0/3059543635 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:19:55.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.578+0000 7fd73da00640 1 --2- 172.21.15.96:0/3059543635 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd7380ce940 0x7fd7380cedf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:55.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.578+0000 7fd73d000640 1 --2- 172.21.15.96:0/3059543635 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd720004d40 0x7fd7381528a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:55.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.578+0000 7fd737e00640 1 -- 172.21.15.96:0/3059543635 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd7380ce940 msgr2=0x7fd7380cedf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:55.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.578+0000 7fd737e00640 1 --2- 172.21.15.96:0/3059543635 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd7380ce940 0x7fd7380cedf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:55.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.578+0000 7fd737e00640 1 -- 172.21.15.96:0/3059543635 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd720004d40 msgr2=0x7fd7381528a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:55.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.578+0000 7fd737e00640 1 --2- 172.21.15.96:0/3059543635 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd720004d40 0x7fd7381528a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:55.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.578+0000 7fd737e00640 1 -- 172.21.15.96:0/3059543635 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd738152de0 con 0x7fd73815c950 2024-05-28T06:19:55.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.578+0000 7fd737e00640 1 --2- 172.21.15.96:0/3059543635 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd73815c950 0x7fd738160e90 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7fd72400e580 tx=0x7fd72400ea40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:55.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.578+0000 7fd734c00640 1 -- 172.21.15.96:0/3059543635 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7240052e0 con 0x7fd73815c950 2024-05-28T06:19:55.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.578+0000 7fd734c00640 1 -- 172.21.15.96:0/3059543635 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd724018070 con 0x7fd73815c950 2024-05-28T06:19:55.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.578+0000 7fd73fa00640 1 -- 172.21.15.96:0/3059543635 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd73815f3c0 con 0x7fd73815c950 2024-05-28T06:19:55.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.578+0000 7fd734c00640 1 -- 172.21.15.96:0/3059543635 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd724003810 con 0x7fd73815c950 2024-05-28T06:19:55.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.579+0000 7fd73fa00640 1 -- 172.21.15.96:0/3059543635 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd73815f980 con 0x7fd73815c950 2024-05-28T06:19:55.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.579+0000 7fd73fa00640 1 -- 172.21.15.96:0/3059543635 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7380689e0 con 0x7fd73815c950 2024-05-28T06:19:55.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.580+0000 7fd734c00640 1 -- 172.21.15.96:0/3059543635 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 33) ==== 92825+0+0 (secure 0 0 0) 0x7fd7240039b0 con 0x7fd73815c950 2024-05-28T06:19:55.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.581+0000 7fd734c00640 1 --2- 172.21.15.96:0/3059543635 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd71806dcd0 0x7fd718070190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:55.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.582+0000 7fd734c00640 1 -- 172.21.15.96:0/3059543635 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 5766+0+0 (secure 0 0 0) 0x7fd72408a520 con 0x7fd73815c950 2024-05-28T06:19:55.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.582+0000 7fd73d000640 1 --2- 172.21.15.96:0/3059543635 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd71806dcd0 0x7fd718070190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:55.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.583+0000 7fd73d000640 1 --2- 172.21.15.96:0/3059543635 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd71806dcd0 0x7fd718070190 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fd72c001050 tx=0x7fd72c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:55.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.586+0000 7fd734c00640 1 -- 172.21.15.96:0/3059543635 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd72405a320 con 0x7fd73815c950 2024-05-28T06:19:55.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.818+0000 7fd73fa00640 1 -- 172.21.15.96:0/3059543635 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd7380ce560 con 0x7fd71806dcd0 2024-05-28T06:19:55.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.887+0000 7fd734c00640 1 -- 172.21.15.96:0/3059543635 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fd7380ce560 con 0x7fd71806dcd0 2024-05-28T06:19:55.884 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:19:55.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.890+0000 7fd73fa00640 1 -- 172.21.15.96:0/3059543635 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd71806dcd0 msgr2=0x7fd718070190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:55.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.890+0000 7fd73fa00640 1 --2- 172.21.15.96:0/3059543635 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd71806dcd0 0x7fd718070190 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fd72c001050 tx=0x7fd72c009040 comp rx=0 tx=0).stop 2024-05-28T06:19:55.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.890+0000 7fd73fa00640 1 -- 172.21.15.96:0/3059543635 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd73815c950 msgr2=0x7fd738160e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:55.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.890+0000 7fd73fa00640 1 --2- 172.21.15.96:0/3059543635 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd73815c950 0x7fd738160e90 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7fd72400e580 tx=0x7fd72400ea40 comp rx=0 tx=0).stop 2024-05-28T06:19:55.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.890+0000 7fd73fa00640 1 -- 172.21.15.96:0/3059543635 shutdown_connections 2024-05-28T06:19:55.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.890+0000 7fd73fa00640 1 --2- 172.21.15.96:0/3059543635 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd71806dcd0 0x7fd718070190 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:55.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.890+0000 7fd73fa00640 1 --2- 172.21.15.96:0/3059543635 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd7380ce940 0x7fd7380cedf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:55.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.890+0000 7fd73fa00640 1 --2- 172.21.15.96:0/3059543635 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd73815c950 0x7fd738160e90 unknown :-1 s=CLOSED pgs=1511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:55.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.890+0000 7fd73fa00640 1 --2- 172.21.15.96:0/3059543635 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd720004d40 0x7fd7381528a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:55.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.891+0000 7fd73fa00640 1 -- 172.21.15.96:0/3059543635 >> 172.21.15.96:0/3059543635 conn(0x7fd73806bd20 msgr2=0x7fd738071c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:19:55.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.891+0000 7fd73fa00640 1 -- 172.21.15.96:0/3059543635 shutdown_connections 2024-05-28T06:19:55.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:55.891+0000 7fd73fa00640 1 -- 172.21.15.96:0/3059543635 wait complete. 2024-05-28T06:19:55.908 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs sv1 2024-05-28T06:19:56.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.112+0000 7f2784400640 1 Processor -- start 2024-05-28T06:19:56.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.113+0000 7f2784400640 1 -- start start 2024-05-28T06:19:56.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.114+0000 7f2784400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f277c15d5d0 0x7f277c0cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:56.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.114+0000 7f2784400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f277c0b32c0 0x7f277c0b3750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:56.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.114+0000 7f2784400640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f277c15cb10 con 0x7f277c0d0640 2024-05-28T06:19:56.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.114+0000 7f2784400640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f277c0adbb0 con 0x7f277c0b32c0 2024-05-28T06:19:56.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.114+0000 7f2784400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f277c0ad5f0 con 0x7f277c15d5d0 2024-05-28T06:19:56.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.114+0000 7f2781a00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f277c0d0640 0x7f277c0d0a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:36978/0 (socket says 172.21.15.96:36978) 2024-05-28T06:19:56.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.114+0000 7f2781a00640 1 -- 172.21.15.96:0/868776872 learned_addr learned my addr 172.21.15.96:0/868776872 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:19:56.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.115+0000 7f277be00640 1 -- 172.21.15.96:0/868776872 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1927806385 0 0) 0x7f277c15cb10 con 0x7f277c0d0640 2024-05-28T06:19:56.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.115+0000 7f2781000640 1 --2- 172.21.15.96:0/868776872 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f277c15d5d0 0x7f277c0cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:56.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.115+0000 7f277be00640 1 -- 172.21.15.96:0/868776872 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2760003520 con 0x7f277c0d0640 2024-05-28T06:19:56.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.115+0000 7f2782400640 1 --2- 172.21.15.96:0/868776872 >> v2:172.21.15.123:3300/0 conn(0x7f277c0b32c0 0x7f277c0b3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:56.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.115+0000 7f277be00640 1 -- 172.21.15.96:0/868776872 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2230528839 0 0) 0x7f2760003520 con 0x7f277c0d0640 2024-05-28T06:19:56.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.115+0000 7f277be00640 1 -- 172.21.15.96:0/868776872 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f277c15d5d0 msgr2=0x7f277c0cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:19:56.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.115+0000 7f277be00640 1 --2- 172.21.15.96:0/868776872 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f277c15d5d0 0x7f277c0cf950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:56.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.115+0000 7f277be00640 1 -- 172.21.15.96:0/868776872 >> v2:172.21.15.123:3300/0 conn(0x7f277c0b32c0 msgr2=0x7f277c0b3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:19:56.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.115+0000 7f277be00640 1 --2- 172.21.15.96:0/868776872 >> v2:172.21.15.123:3300/0 conn(0x7f277c0b32c0 0x7f277c0b3750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:56.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.115+0000 7f277be00640 1 -- 172.21.15.96:0/868776872 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f277c0ad8d0 con 0x7f277c0d0640 2024-05-28T06:19:56.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.115+0000 7f277be00640 1 -- 172.21.15.96:0/868776872 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f276c001670 con 0x7f277c0d0640 2024-05-28T06:19:56.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.115+0000 7f277be00640 1 -- 172.21.15.96:0/868776872 >> v1:172.21.15.96:6789/0 conn(0x7f277c0d0640 legacy=0x7f277c0d0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:56.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.115+0000 7f277be00640 1 --2- 172.21.15.96:0/868776872 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f27600039b0 0x7f2760003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:56.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.115+0000 7f277be00640 1 --2- 172.21.15.96:0/868776872 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2760004800 0x7f2760004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:56.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.115+0000 7f277be00640 1 --2- 172.21.15.96:0/868776872 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f277c15d5d0 0x7f27600051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:56.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.115+0000 7f277be00640 1 -- 172.21.15.96:0/868776872 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f277c0ad5f0 con 0x7f2760004800 2024-05-28T06:19:56.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.115+0000 7f277be00640 1 -- 172.21.15.96:0/868776872 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f276000b9c0 con 0x7f27600039b0 2024-05-28T06:19:56.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.115+0000 7f277be00640 1 -- 172.21.15.96:0/868776872 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f276000bb40 con 0x7f277c15d5d0 2024-05-28T06:19:56.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.115+0000 7f2782400640 1 --2- 172.21.15.96:0/868776872 >> v2:172.21.15.123:3300/0 conn(0x7f277c0b32c0 0x7f277c0b3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-28T06:19:56.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.116+0000 7f2781000640 1 --2- 172.21.15.96:0/868776872 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2760004800 0x7f2760004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:56.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.116+0000 7f2781000640 1 -- 172.21.15.96:0/868776872 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f277c15d5d0 msgr2=0x7f27600051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:19:56.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.116+0000 7f2781000640 1 --2- 172.21.15.96:0/868776872 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f277c15d5d0 0x7f27600051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:56.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.116+0000 7f2781000640 1 -- 172.21.15.96:0/868776872 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f27600039b0 msgr2=0x7f2760003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:19:56.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.116+0000 7f2781000640 1 --2- 172.21.15.96:0/868776872 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f27600039b0 0x7f2760003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:56.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.116+0000 7f2781000640 1 -- 172.21.15.96:0/868776872 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2760003520 con 0x7f2760004800 2024-05-28T06:19:56.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.117+0000 7f2781000640 1 --2- 172.21.15.96:0/868776872 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2760004800 0x7f2760004c80 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7f2768009fa0 tx=0x7f2760005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ee02dc7043c6e9f server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:56.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.117+0000 7f277be00640 1 -- 172.21.15.96:0/868776872 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2768009bf0 con 0x7f2760004800 2024-05-28T06:19:56.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.117+0000 7f277be00640 1 -- 172.21.15.96:0/868776872 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2768036070 con 0x7f2760004800 2024-05-28T06:19:56.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.117+0000 7f277be00640 1 -- 172.21.15.96:0/868776872 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2768030880 con 0x7f2760004800 2024-05-28T06:19:56.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.118+0000 7f2784400640 1 -- 172.21.15.96:0/868776872 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2760004800 msgr2=0x7f2760004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:56.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.118+0000 7f2784400640 1 --2- 172.21.15.96:0/868776872 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2760004800 0x7f2760004c80 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7f2768009fa0 tx=0x7f2760005830 comp rx=0 tx=0).stop 2024-05-28T06:19:56.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.118+0000 7f2782400640 1 -- 172.21.15.96:0/868776872 reap_dead start 2024-05-28T06:19:56.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.118+0000 7f2784400640 1 -- 172.21.15.96:0/868776872 shutdown_connections 2024-05-28T06:19:56.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.118+0000 7f2784400640 1 -- 172.21.15.96:0/868776872 >> 172.21.15.96:0/868776872 conn(0x7f277c06bd20 msgr2=0x7f277c06c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:19:56.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.118+0000 7f2784400640 1 -- 172.21.15.96:0/868776872 shutdown_connections 2024-05-28T06:19:56.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.118+0000 7f2784400640 1 -- 172.21.15.96:0/868776872 wait complete. 2024-05-28T06:19:56.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.120+0000 7f2784400640 1 Processor -- start 2024-05-28T06:19:56.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.120+0000 7f2784400640 1 -- start start 2024-05-28T06:19:56.120 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.121+0000 7f2784400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2760004800 0x7f277c14ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:56.120 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.121+0000 7f2784400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f277c147ff0 0x7f277c14c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:56.120 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.121+0000 7f2784400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f277c0af1e0 0x7f277c0af660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:56.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.121+0000 7f2784400640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f277c15da50 con 0x7f2760004800 2024-05-28T06:19:56.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.121+0000 7f2784400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f277c15d0b0 con 0x7f277c147ff0 2024-05-28T06:19:56.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.121+0000 7f2784400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f277c0ad2a0 con 0x7f277c0af1e0 2024-05-28T06:19:56.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.121+0000 7f2781a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2760004800 0x7f277c14ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:56.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.121+0000 7f2781a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2760004800 0x7f277c14ae60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51648/0 (socket says 172.21.15.96:51648) 2024-05-28T06:19:56.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.121+0000 7f2781a00640 1 -- 172.21.15.96:0/1561435197 learned_addr learned my addr 172.21.15.96:0/1561435197 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:19:56.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.122+0000 7f2781000640 1 --2- 172.21.15.96:0/1561435197 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f277c147ff0 0x7f277c14c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:56.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.122+0000 7f2782400640 1 --2- 172.21.15.96:0/1561435197 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f277c0af1e0 0x7f277c0af660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:56.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.122+0000 7f2781a00640 1 -- 172.21.15.96:0/1561435197 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f277c0af1e0 msgr2=0x7f277c0af660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:56.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.122+0000 7f2781a00640 1 --2- 172.21.15.96:0/1561435197 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f277c0af1e0 0x7f277c0af660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:56.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.122+0000 7f2781a00640 1 -- 172.21.15.96:0/1561435197 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f277c147ff0 msgr2=0x7f277c14c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:56.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.122+0000 7f2781a00640 1 --2- 172.21.15.96:0/1561435197 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f277c147ff0 0x7f277c14c4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:56.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.122+0000 7f2781a00640 1 -- 172.21.15.96:0/1561435197 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f277c149dc0 con 0x7f2760004800 2024-05-28T06:19:56.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.122+0000 7f2781a00640 1 --2- 172.21.15.96:0/1561435197 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2760004800 0x7f277c14ae60 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f276c00c780 tx=0x7f276c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:56.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.122+0000 7f2779600640 1 -- 172.21.15.96:0/1561435197 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f276c018070 con 0x7f2760004800 2024-05-28T06:19:56.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.122+0000 7f2779600640 1 -- 172.21.15.96:0/1561435197 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f276c007580 con 0x7f2760004800 2024-05-28T06:19:56.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.122+0000 7f2779600640 1 -- 172.21.15.96:0/1561435197 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f276c004a20 con 0x7f2760004800 2024-05-28T06:19:56.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.122+0000 7f2784400640 1 -- 172.21.15.96:0/1561435197 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f277c14a0a0 con 0x7f2760004800 2024-05-28T06:19:56.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.123+0000 7f2784400640 1 -- 172.21.15.96:0/1561435197 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f277c14a660 con 0x7f2760004800 2024-05-28T06:19:56.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.124+0000 7f2784400640 1 -- 172.21.15.96:0/1561435197 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f277c0689e0 con 0x7f2760004800 2024-05-28T06:19:56.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.125+0000 7f2779600640 1 -- 172.21.15.96:0/1561435197 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 33) ==== 92825+0+0 (secure 0 0 0) 0x7f276c004bc0 con 0x7f2760004800 2024-05-28T06:19:56.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.126+0000 7f2779600640 1 --2- 172.21.15.96:0/1561435197 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f275c06dca0 0x7f275c070160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:56.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.126+0000 7f2779600640 1 -- 172.21.15.96:0/1561435197 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 5766+0+0 (secure 0 0 0) 0x7f276c08a090 con 0x7f2760004800 2024-05-28T06:19:56.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.126+0000 7f2781000640 1 --2- 172.21.15.96:0/1561435197 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f275c06dca0 0x7f275c070160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:56.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.128+0000 7f2781000640 1 --2- 172.21.15.96:0/1561435197 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f275c06dca0 0x7f275c070160 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f2768035000 tx=0x7f276802f000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:56.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.130+0000 7f2779600640 1 -- 172.21.15.96:0/1561435197 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f276c059b60 con 0x7f2760004800 2024-05-28T06:19:56.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.362+0000 7f2784400640 1 -- 172.21.15.96:0/1561435197 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) -- 0x7f277c15cb10 con 0x7f275c06dca0 2024-05-28T06:19:56.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.405+0000 7f2779600640 1 -- 172.21.15.96:0/1561435197 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f277c15cb10 con 0x7f275c06dca0 2024-05-28T06:19:56.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.409+0000 7f2784400640 1 -- 172.21.15.96:0/1561435197 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f275c06dca0 msgr2=0x7f275c070160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:56.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.409+0000 7f2784400640 1 --2- 172.21.15.96:0/1561435197 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f275c06dca0 0x7f275c070160 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f2768035000 tx=0x7f276802f000 comp rx=0 tx=0).stop 2024-05-28T06:19:56.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.409+0000 7f2784400640 1 -- 172.21.15.96:0/1561435197 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2760004800 msgr2=0x7f277c14ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:56.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.409+0000 7f2784400640 1 --2- 172.21.15.96:0/1561435197 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2760004800 0x7f277c14ae60 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f276c00c780 tx=0x7f276c00cc40 comp rx=0 tx=0).stop 2024-05-28T06:19:56.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.409+0000 7f2784400640 1 -- 172.21.15.96:0/1561435197 shutdown_connections 2024-05-28T06:19:56.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.409+0000 7f2784400640 1 --2- 172.21.15.96:0/1561435197 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f275c06dca0 0x7f275c070160 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:56.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.409+0000 7f2784400640 1 --2- 172.21.15.96:0/1561435197 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f277c0af1e0 0x7f277c0af660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:56.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.409+0000 7f2784400640 1 --2- 172.21.15.96:0/1561435197 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f277c147ff0 0x7f277c14c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:56.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.409+0000 7f2784400640 1 --2- 172.21.15.96:0/1561435197 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2760004800 0x7f277c14ae60 unknown :-1 s=CLOSED pgs=1514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:56.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.409+0000 7f2784400640 1 -- 172.21.15.96:0/1561435197 >> 172.21.15.96:0/1561435197 conn(0x7f277c06bd20 msgr2=0x7f277c153a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:19:56.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.409+0000 7f2784400640 1 -- 172.21.15.96:0/1561435197 shutdown_connections 2024-05-28T06:19:56.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.409+0000 7f2784400640 1 -- 172.21.15.96:0/1561435197 wait complete. 2024-05-28T06:19:56.412 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:19:56.416+0000 7fea5bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:19:56.412 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:19:56.416+0000 7fea5bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:19:56.413 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:19:56.416+0000 7fea5bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:19:56.413 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:19:56.416+0000 7fea5bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:19:56.413 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:19:56.416+0000 7fea5bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:19:56.427 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs ss1clone1 2024-05-28T06:19:56.442 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:19:56.445+0000 7fea5da00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:19:56.442 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:19:56.445+0000 7fea5da00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:19:56.442 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:19:56.445+0000 7fea5da00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:19:56.442 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:19:56.445+0000 7fea5da00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:19:56.442 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:19:56.445+0000 7fea5da00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:19:56.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.631+0000 7f093b800640 1 Processor -- start 2024-05-28T06:19:56.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.631+0000 7f093b800640 1 -- start start 2024-05-28T06:19:56.631 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.633+0000 7f093b800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f093406b900 0x7f09340c7950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:56.631 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.633+0000 7f093b800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f093415f5d0 0x7f093415fa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:56.631 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.633+0000 7f093b800640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09340c7e90 con 0x7f09340cea30 2024-05-28T06:19:56.631 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.633+0000 7f093b800640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f09340c8110 con 0x7f093415f5d0 2024-05-28T06:19:56.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.633+0000 7f093b800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f09340cdc70 con 0x7f093406b900 2024-05-28T06:19:56.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.633+0000 7f0939800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f093415f5d0 0x7f093415fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:56.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.634+0000 7f0939800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f093415f5d0 0x7f093415fa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51656/0 (socket says 172.21.15.96:51656) 2024-05-28T06:19:56.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.633+0000 7f0933e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f09340cea30 0x7f093406bce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:36990/0 (socket says 172.21.15.96:36990) 2024-05-28T06:19:56.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.634+0000 7f0939800640 1 -- 172.21.15.96:0/2155290583 learned_addr learned my addr 172.21.15.96:0/2155290583 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:19:56.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.634+0000 7f0938e00640 1 --2- 172.21.15.96:0/2155290583 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f093406b900 0x7f09340c7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:56.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.634+0000 7f0939800640 1 -- 172.21.15.96:0/2155290583 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f093406b900 msgr2=0x7f09340c7950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:19:56.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.634+0000 7f0939800640 1 --2- 172.21.15.96:0/2155290583 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f093406b900 0x7f09340c7950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:56.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.634+0000 7f0939800640 1 -- 172.21.15.96:0/2155290583 >> v1:172.21.15.96:6789/0 conn(0x7f09340cea30 legacy=0x7f093406bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:56.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.634+0000 7f0933400640 1 -- 172.21.15.96:0/2155290583 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3277013571 0 0) 0x7f0920001110 con 0x7f09340cea30 2024-05-28T06:19:56.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.634+0000 7f0939800640 1 -- 172.21.15.96:0/2155290583 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09340cdf50 con 0x7f093415f5d0 2024-05-28T06:19:56.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.635+0000 7f0939800640 1 --2- 172.21.15.96:0/2155290583 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f093415f5d0 0x7f093415fa60 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f092c009e50 tx=0x7f092c02a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc2b53ae905c9263 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:56.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.635+0000 7f0933400640 1 -- 172.21.15.96:0/2155290583 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f092c036070 con 0x7f093415f5d0 2024-05-28T06:19:56.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.635+0000 7f0933400640 1 -- 172.21.15.96:0/2155290583 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f092c031600 con 0x7f093415f5d0 2024-05-28T06:19:56.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.636+0000 7f0933400640 1 -- 172.21.15.96:0/2155290583 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f092c0319d0 con 0x7f093415f5d0 2024-05-28T06:19:56.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.636+0000 7f093b800640 1 -- 172.21.15.96:0/2155290583 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f093415f5d0 msgr2=0x7f093415fa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:56.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.636+0000 7f093b800640 1 --2- 172.21.15.96:0/2155290583 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f093415f5d0 0x7f093415fa60 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f092c009e50 tx=0x7f092c02a710 comp rx=0 tx=0).stop 2024-05-28T06:19:56.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.636+0000 7f093b800640 1 -- 172.21.15.96:0/2155290583 shutdown_connections 2024-05-28T06:19:56.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.636+0000 7f093b800640 1 --2- 172.21.15.96:0/2155290583 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f093415f5d0 0x7f093415fa60 unknown :-1 s=CLOSED pgs=1517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:56.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.636+0000 7f093b800640 1 --2- 172.21.15.96:0/2155290583 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f093406b900 0x7f09340c7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:56.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.636+0000 7f093b800640 1 -- 172.21.15.96:0/2155290583 >> 172.21.15.96:0/2155290583 conn(0x7f093409aa30 msgr2=0x7f09340b3020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:19:56.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.636+0000 7f093b800640 1 -- 172.21.15.96:0/2155290583 shutdown_connections 2024-05-28T06:19:56.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.636+0000 7f093b800640 1 -- 172.21.15.96:0/2155290583 wait complete. 2024-05-28T06:19:56.635 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.638+0000 7f093b800640 1 Processor -- start 2024-05-28T06:19:56.635 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.638+0000 7f093b800640 1 -- start start 2024-05-28T06:19:56.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.639+0000 7f093b800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f093406b900 0x7f09341f8950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:56.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.639+0000 7f093b800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f09340cea30 0x7f0934161240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:56.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.639+0000 7f093b800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f093415f5d0 0x7f093414fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:56.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.639+0000 7f093b800640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f09340cefc0 con 0x7f093415f5d0 2024-05-28T06:19:56.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.639+0000 7f093b800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f09340ce510 con 0x7f093406b900 2024-05-28T06:19:56.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.639+0000 7f093b800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f09340cf140 con 0x7f09340cea30 2024-05-28T06:19:56.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.639+0000 7f0939800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f093415f5d0 0x7f093414fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:56.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.639+0000 7f0939800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f093415f5d0 0x7f093414fe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51664/0 (socket says 172.21.15.96:51664) 2024-05-28T06:19:56.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.639+0000 7f0939800640 1 -- 172.21.15.96:0/3001540565 learned_addr learned my addr 172.21.15.96:0/3001540565 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:19:56.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.640+0000 7f0933e00640 1 --2- 172.21.15.96:0/3001540565 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f09340cea30 0x7f0934161240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:56.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.640+0000 7f0938e00640 1 --2- 172.21.15.96:0/3001540565 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f093406b900 0x7f09341f8950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:56.638 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.640+0000 7f0939800640 1 -- 172.21.15.96:0/3001540565 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f09340cea30 msgr2=0x7f0934161240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:56.638 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.640+0000 7f0939800640 1 --2- 172.21.15.96:0/3001540565 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f09340cea30 0x7f0934161240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:56.638 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.640+0000 7f0939800640 1 -- 172.21.15.96:0/3001540565 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f093406b900 msgr2=0x7f09341f8950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:56.638 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.640+0000 7f0939800640 1 --2- 172.21.15.96:0/3001540565 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f093406b900 0x7f09341f8950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:56.638 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.640+0000 7f0939800640 1 -- 172.21.15.96:0/3001540565 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09340b3bc0 con 0x7f093415f5d0 2024-05-28T06:19:56.639 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.640+0000 7f0939800640 1 --2- 172.21.15.96:0/3001540565 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f093415f5d0 0x7f093414fe90 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f092c034040 tx=0x7f092c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:56.639 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.640+0000 7f0930c00640 1 -- 172.21.15.96:0/3001540565 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f092c041070 con 0x7f093415f5d0 2024-05-28T06:19:56.639 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.640+0000 7f0930c00640 1 -- 172.21.15.96:0/3001540565 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f092c003990 con 0x7f093415f5d0 2024-05-28T06:19:56.639 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.641+0000 7f0930c00640 1 -- 172.21.15.96:0/3001540565 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f092c036040 con 0x7f093415f5d0 2024-05-28T06:19:56.639 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.641+0000 7f093b800640 1 -- 172.21.15.96:0/3001540565 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09340ca480 con 0x7f093415f5d0 2024-05-28T06:19:56.639 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.641+0000 7f093b800640 1 -- 172.21.15.96:0/3001540565 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09340caa40 con 0x7f093415f5d0 2024-05-28T06:19:56.639 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.642+0000 7f093b800640 1 -- 172.21.15.96:0/3001540565 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09340690f0 con 0x7f093415f5d0 2024-05-28T06:19:56.640 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.643+0000 7f0930c00640 1 -- 172.21.15.96:0/3001540565 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 33) ==== 92825+0+0 (secure 0 0 0) 0x7f092c0077a0 con 0x7f093415f5d0 2024-05-28T06:19:56.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.643+0000 7f0930c00640 1 --2- 172.21.15.96:0/3001540565 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f091406dca0 0x7f0914070160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:56.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.644+0000 7f0930c00640 1 -- 172.21.15.96:0/3001540565 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 5766+0+0 (secure 0 0 0) 0x7f092c0ab2a0 con 0x7f093415f5d0 2024-05-28T06:19:56.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.644+0000 7f0938e00640 1 --2- 172.21.15.96:0/3001540565 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f091406dca0 0x7f0914070160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:56.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.646+0000 7f0938e00640 1 --2- 172.21.15.96:0/3001540565 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f091406dca0 0x7f0914070160 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f0928006fd0 tx=0x7f0928008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:56.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.653+0000 7f0930c00640 1 -- 172.21.15.96:0/3001540565 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f092c04b020 con 0x7f093415f5d0 2024-05-28T06:19:56.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.900+0000 7f093b800640 1 -- 172.21.15.96:0/3001540565 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "ss1clone1", "target": ["mon-mgr", ""]}) -- 0x7f093409af10 con 0x7f091406dca0 2024-05-28T06:19:56.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.938+0000 7f0930c00640 1 -- 172.21.15.96:0/3001540565 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f093409af10 con 0x7f091406dca0 2024-05-28T06:19:56.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.941+0000 7f093b800640 1 -- 172.21.15.96:0/3001540565 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f091406dca0 msgr2=0x7f0914070160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:56.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.941+0000 7f093b800640 1 --2- 172.21.15.96:0/3001540565 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f091406dca0 0x7f0914070160 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f0928006fd0 tx=0x7f0928008040 comp rx=0 tx=0).stop 2024-05-28T06:19:56.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.941+0000 7f093b800640 1 -- 172.21.15.96:0/3001540565 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f093415f5d0 msgr2=0x7f093414fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:56.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.941+0000 7f093b800640 1 --2- 172.21.15.96:0/3001540565 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f093415f5d0 0x7f093414fe90 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f092c034040 tx=0x7f092c004a40 comp rx=0 tx=0).stop 2024-05-28T06:19:56.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.941+0000 7f093b800640 1 -- 172.21.15.96:0/3001540565 shutdown_connections 2024-05-28T06:19:56.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.941+0000 7f093b800640 1 --2- 172.21.15.96:0/3001540565 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f091406dca0 0x7f0914070160 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:56.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.941+0000 7f093b800640 1 --2- 172.21.15.96:0/3001540565 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f093415f5d0 0x7f093414fe90 unknown :-1 s=CLOSED pgs=1518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:56.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.941+0000 7f093b800640 1 --2- 172.21.15.96:0/3001540565 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f09340cea30 0x7f0934161240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:56.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.941+0000 7f093b800640 1 --2- 172.21.15.96:0/3001540565 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f093406b900 0x7f09341f8950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:56.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.941+0000 7f093b800640 1 -- 172.21.15.96:0/3001540565 >> 172.21.15.96:0/3001540565 conn(0x7f093409aa30 msgr2=0x7f093406eec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:19:56.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.941+0000 7f093b800640 1 -- 172.21.15.96:0/3001540565 shutdown_connections 2024-05-28T06:19:56.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:56.941+0000 7f093b800640 1 -- 172.21.15.96:0/3001540565 wait complete. 2024-05-28T06:19:56.958 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2024-05-28T06:19:57.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:57 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-28T06:19:57.067 INFO:teuthology.orchestra.run.smithi096.stdout:2 2024-05-28T06:19:57.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-28T06:19:57.077 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-05-28T06:19:57.077 DEBUG:teuthology.orchestra.run.smithi096:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-05-28T06:19:57.181 DEBUG:teuthology.orchestra.run.smithi123:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-05-28T06:19:57.299 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-05-28T06:19:57.299 INFO:tasks.cephfs.mount:Terminating background process 2024-05-28T06:19:57.299 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:19:57.299 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:19:57.359 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:19:57.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:57.567+0000 7f4eeea00640 1 Processor -- start 2024-05-28T06:19:57.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:57.567+0000 7f4eeea00640 1 -- start start 2024-05-28T06:19:57.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:57.568+0000 7f4eeea00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f4ee80b4e10 0x7f4ee80cde30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:57.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:57.569+0000 7f4eeea00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f4ee80ce370 0x7f4ee80d0590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:57.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:57.569+0000 7f4eeea00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4ee80d0ad0 0x7f4ee8099090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:57.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:57.569+0000 7f4eeea00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f4ee80d0f40 con 0x7f4ee80ce370 2024-05-28T06:19:57.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:57.569+0000 7f4eeea00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f4ee80b51f0 con 0x7f4ee80b4e10 2024-05-28T06:19:57.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:57.569+0000 7f4eeea00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f4ee80ad1d0 con 0x7f4ee80d0ad0 2024-05-28T06:19:57.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:57.569+0000 7f4ee7400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f4ee80ce370 0x7f4ee80d0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:57.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:57.569+0000 7f4ee7400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f4ee80ce370 0x7f4ee80d0590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51680/0 (socket says 172.21.15.96:51680) 2024-05-28T06:19:57.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:57.569+0000 7f4ee7400640 1 -- 172.21.15.96:0/1137584542 learned_addr learned my addr 172.21.15.96:0/1137584542 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:19:57.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:57.569+0000 7f4ee7e00640 1 --2- 172.21.15.96:0/1137584542 >> v2:172.21.15.123:3300/0 conn(0x7f4ee80b4e10 0x7f4ee80cde30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:57.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:57.569+0000 7f4ee7400640 1 -- 172.21.15.96:0/1137584542 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4ee80d0ad0 msgr2=0x7f4ee8099090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:19:57.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:57.569+0000 7f4ee7400640 1 --2- 172.21.15.96:0/1137584542 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4ee80d0ad0 0x7f4ee8099090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:57.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:57.569+0000 7f4ee7400640 1 -- 172.21.15.96:0/1137584542 >> v2:172.21.15.123:3300/0 conn(0x7f4ee80b4e10 msgr2=0x7f4ee80cde30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:19:57.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:57.569+0000 7f4ee7400640 1 --2- 172.21.15.96:0/1137584542 >> v2:172.21.15.123:3300/0 conn(0x7f4ee80b4e10 0x7f4ee80cde30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:57.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:57.569+0000 7f4ee7400640 1 -- 172.21.15.96:0/1137584542 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ee80ad470 con 0x7f4ee80ce370 2024-05-28T06:19:57.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:57.570+0000 7f4ee7400640 1 --2- 172.21.15.96:0/1137584542 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4ee80ce370 0x7f4ee80d0590 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f4ee806f080 tx=0x7f4ed402a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b48b53ad7c35ac2 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:57.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:57.571+0000 7f4ee6a00640 1 -- 172.21.15.96:0/1137584542 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ed4037070 con 0x7f4ee80ce370 2024-05-28T06:19:57.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:57.571+0000 7f4ee6a00640 1 -- 172.21.15.96:0/1137584542 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ed4032440 con 0x7f4ee80ce370 2024-05-28T06:19:57.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:57.571+0000 7f4ee6a00640 1 -- 172.21.15.96:0/1137584542 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ed4032810 con 0x7f4ee80ce370 2024-05-28T06:19:57.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:57.571+0000 7f4eeea00640 1 -- 172.21.15.96:0/1137584542 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4ee80ce370 msgr2=0x7f4ee80d0590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:57.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:57.571+0000 7f4eeea00640 1 --2- 172.21.15.96:0/1137584542 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4ee80ce370 0x7f4ee80d0590 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f4ee806f080 tx=0x7f4ed402a140 comp rx=0 tx=0).stop 2024-05-28T06:19:57.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:57.571+0000 7f4eeea00640 1 -- 172.21.15.96:0/1137584542 shutdown_connections 2024-05-28T06:19:57.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:57.571+0000 7f4eeea00640 1 --2- 172.21.15.96:0/1137584542 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4ee80d0ad0 0x7f4ee8099090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:57.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:57.571+0000 7f4eeea00640 1 --2- 172.21.15.96:0/1137584542 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4ee80ce370 0x7f4ee80d0590 unknown :-1 s=CLOSED pgs=1519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:57.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:57.571+0000 7f4eeea00640 1 --2- 172.21.15.96:0/1137584542 >> v2:172.21.15.123:3300/0 conn(0x7f4ee80b4e10 0x7f4ee80cde30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:57.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:57.571+0000 7f4eeea00640 1 -- 172.21.15.96:0/1137584542 >> 172.21.15.96:0/1137584542 conn(0x7f4ee806bae0 msgr2=0x7f4ee806bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:19:57.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:57.571+0000 7f4eeea00640 1 -- 172.21.15.96:0/1137584542 shutdown_connections 2024-05-28T06:19:57.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:57.572+0000 7f4eeea00640 1 -- 172.21.15.96:0/1137584542 wait complete. 2024-05-28T06:19:57.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:57.573+0000 7f4eeea00640 1 Processor -- start 2024-05-28T06:19:57.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:57.573+0000 7f4eeea00640 1 -- start start 2024-05-28T06:19:57.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:57.574+0000 7f4eeea00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4ee80b4e10 0x7f4ee80afdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:57.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:57.574+0000 7f4eeea00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4ee80ce370 0x7f4ee80c7330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:57.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:57.574+0000 7f4eeea00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4ee80d0ad0 0x7f4ee815ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:57.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:57.574+0000 7f4eeea00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4ee80cea00 con 0x7f4ee80d0ad0 2024-05-28T06:19:57.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:57.574+0000 7f4eeea00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f4ee80ce880 con 0x7f4ee80ce370 2024-05-28T06:19:57.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:57.574+0000 7f4eeea00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f4ee80b3150 con 0x7f4ee80b4e10 2024-05-28T06:19:57.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:57.575+0000 7f4eeca00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4ee80d0ad0 0x7f4ee815ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:57.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:57.575+0000 7f4eeca00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4ee80d0ad0 0x7f4ee815ae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51690/0 (socket says 172.21.15.96:51690) 2024-05-28T06:19:57.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:57.575+0000 7f4eeca00640 1 -- 172.21.15.96:0/247354344 learned_addr learned my addr 172.21.15.96:0/247354344 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:19:57.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:57.575+0000 7f4ee7e00640 1 --2- 172.21.15.96:0/247354344 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4ee80b4e10 0x7f4ee80afdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:57.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:57.575+0000 7f4ee7400640 1 --2- 172.21.15.96:0/247354344 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4ee80ce370 0x7f4ee80c7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:57.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:57.575+0000 7f4eeca00640 1 -- 172.21.15.96:0/247354344 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4ee80b4e10 msgr2=0x7f4ee80afdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:57.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:57.575+0000 7f4eeca00640 1 --2- 172.21.15.96:0/247354344 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4ee80b4e10 0x7f4ee80afdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:57.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:57.575+0000 7f4eeca00640 1 -- 172.21.15.96:0/247354344 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4ee80ce370 msgr2=0x7f4ee80c7330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:57.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:57.575+0000 7f4eeca00640 1 --2- 172.21.15.96:0/247354344 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4ee80ce370 0x7f4ee80c7330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:57.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:57.575+0000 7f4eeca00640 1 -- 172.21.15.96:0/247354344 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ee80b2640 con 0x7f4ee80d0ad0 2024-05-28T06:19:57.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:57.576+0000 7f4eeca00640 1 --2- 172.21.15.96:0/247354344 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4ee80d0ad0 0x7f4ee815ae80 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f4ee000b970 tx=0x7f4ee000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:57.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:57.576+0000 7f4edfe00640 1 -- 172.21.15.96:0/247354344 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ee0019070 con 0x7f4ee80d0ad0 2024-05-28T06:19:57.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:57.576+0000 7f4edfe00640 1 -- 172.21.15.96:0/247354344 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ee000cb60 con 0x7f4ee80d0ad0 2024-05-28T06:19:57.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:57.576+0000 7f4edfe00640 1 -- 172.21.15.96:0/247354344 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ee0014440 con 0x7f4ee80d0ad0 2024-05-28T06:19:57.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:57.576+0000 7f4eeea00640 1 -- 172.21.15.96:0/247354344 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ee80b2920 con 0x7f4ee80d0ad0 2024-05-28T06:19:57.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:57.576+0000 7f4eeea00640 1 -- 172.21.15.96:0/247354344 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ee80725e0 con 0x7f4ee80d0ad0 2024-05-28T06:19:57.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:57.577+0000 7f4eeea00640 1 -- 172.21.15.96:0/247354344 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ee80689e0 con 0x7f4ee80d0ad0 2024-05-28T06:19:57.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:57.578+0000 7f4edfe00640 1 -- 172.21.15.96:0/247354344 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 33) ==== 92825+0+0 (secure 0 0 0) 0x7f4ee001c070 con 0x7f4ee80d0ad0 2024-05-28T06:19:57.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:57.579+0000 7f4edfe00640 1 --2- 172.21.15.96:0/247354344 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4ec806dca0 0x7f4ec8070160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:57.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:57.579+0000 7f4edfe00640 1 -- 172.21.15.96:0/247354344 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 5766+0+0 (secure 0 0 0) 0x7f4ee008a810 con 0x7f4ee80d0ad0 2024-05-28T06:19:57.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:57.580+0000 7f4ee7e00640 1 --2- 172.21.15.96:0/247354344 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4ec806dca0 0x7f4ec8070160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:57.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:57.581+0000 7f4ee7e00640 1 --2- 172.21.15.96:0/247354344 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4ec806dca0 0x7f4ec8070160 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f4ee806c130 tx=0x7f4ed8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:57.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:57.583+0000 7f4edfe00640 1 -- 172.21.15.96:0/247354344 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4ee0059cf0 con 0x7f4ee80d0ad0 2024-05-28T06:19:57.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:57.785+0000 7f4eeea00640 1 -- 172.21.15.96:0/247354344 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4ee806f540 con 0x7f4ee80d0ad0 2024-05-28T06:19:57.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:57.788+0000 7f4edfe00640 1 -- 172.21.15.96:0/247354344 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+16765 (secure 0 0 0) 0x7f4ee005dff0 con 0x7f4ee80d0ad0 2024-05-28T06:19:57.786 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:19:57.786 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":91,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:18:53.991525+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-05-28T06:18:07.815707+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-05-28T06:18:09.061362+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"13.3b","mappings":[{"from":1,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:19:57.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:57.791+0000 7f4eeea00640 1 -- 172.21.15.96:0/247354344 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4ec806dca0 msgr2=0x7f4ec8070160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:57.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:57.791+0000 7f4eeea00640 1 --2- 172.21.15.96:0/247354344 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4ec806dca0 0x7f4ec8070160 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f4ee806c130 tx=0x7f4ed8008040 comp rx=0 tx=0).stop 2024-05-28T06:19:57.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:57.791+0000 7f4eeea00640 1 -- 172.21.15.96:0/247354344 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4ee80d0ad0 msgr2=0x7f4ee815ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:57.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:57.791+0000 7f4eeea00640 1 --2- 172.21.15.96:0/247354344 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4ee80d0ad0 0x7f4ee815ae80 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f4ee000b970 tx=0x7f4ee000be30 comp rx=0 tx=0).stop 2024-05-28T06:19:57.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:57.791+0000 7f4eeea00640 1 -- 172.21.15.96:0/247354344 shutdown_connections 2024-05-28T06:19:57.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:57.791+0000 7f4eeea00640 1 --2- 172.21.15.96:0/247354344 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4ec806dca0 0x7f4ec8070160 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:57.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:57.791+0000 7f4eeea00640 1 --2- 172.21.15.96:0/247354344 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4ee80d0ad0 0x7f4ee815ae80 unknown :-1 s=CLOSED pgs=1520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:57.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:57.791+0000 7f4eeea00640 1 --2- 172.21.15.96:0/247354344 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4ee80ce370 0x7f4ee80c7330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:57.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:57.791+0000 7f4eeea00640 1 --2- 172.21.15.96:0/247354344 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4ee80b4e10 0x7f4ee80afdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:57.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:57.791+0000 7f4eeea00640 1 -- 172.21.15.96:0/247354344 >> 172.21.15.96:0/247354344 conn(0x7f4ee806bae0 msgr2=0x7f4ee80cb480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:19:57.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:57.791+0000 7f4eeea00640 1 -- 172.21.15.96:0/247354344 shutdown_connections 2024-05-28T06:19:57.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:57.791+0000 7f4eeea00640 1 -- 172.21.15.96:0/247354344 wait complete. 2024-05-28T06:19:57.808 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-28T06:03:59.517102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-28T06:18:07.815707+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2024-05-28T06:18:09.061362+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-28T06:19:57.809 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-28T06:19:58.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.007+0000 7f7f57000640 1 Processor -- start 2024-05-28T06:19:58.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.008+0000 7f7f57000640 1 -- start start 2024-05-28T06:19:58.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.009+0000 7f7f57000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f7f500ccd20 0x7f7f5006b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:58.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.009+0000 7f7f57000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7f500cd550 0x7f7f5006f740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:58.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.009+0000 7f7f57000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f7f5006fc80 0x7f7f500cb9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:58.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.009+0000 7f7f57000640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f7f5006c660 con 0x7f7f5006fc80 2024-05-28T06:19:58.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.009+0000 7f7f57000640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f7f5006f2b0 con 0x7f7f500ccd20 2024-05-28T06:19:58.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.009+0000 7f7f57000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f7f500cf140 con 0x7f7f500cd550 2024-05-28T06:19:58.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.009+0000 7f7f55000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f7f5006fc80 0x7f7f500cb9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:58.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.010+0000 7f7f55000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f7f5006fc80 0x7f7f500cb9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51702/0 (socket says 172.21.15.96:51702) 2024-05-28T06:19:58.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.010+0000 7f7f55000640 1 -- 172.21.15.96:0/2176564526 learned_addr learned my addr 172.21.15.96:0/2176564526 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:19:58.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.010+0000 7f7f55000640 1 -- 172.21.15.96:0/2176564526 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7f500cd550 msgr2=0x7f7f5006f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:19:58.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.010+0000 7f7f4f400640 1 --2- 172.21.15.96:0/2176564526 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7f500cd550 0x7f7f5006f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:58.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.010+0000 7f7f4fe00640 1 --2- 172.21.15.96:0/2176564526 >> v2:172.21.15.123:3300/0 conn(0x7f7f500ccd20 0x7f7f5006b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:58.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.010+0000 7f7f55000640 1 --2- 172.21.15.96:0/2176564526 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7f500cd550 0x7f7f5006f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:58.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.010+0000 7f7f55000640 1 -- 172.21.15.96:0/2176564526 >> v2:172.21.15.123:3300/0 conn(0x7f7f500ccd20 msgr2=0x7f7f5006b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:19:58.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.010+0000 7f7f55000640 1 --2- 172.21.15.96:0/2176564526 >> v2:172.21.15.123:3300/0 conn(0x7f7f500ccd20 0x7f7f5006b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:58.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.010+0000 7f7f55000640 1 -- 172.21.15.96:0/2176564526 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f5006ef80 con 0x7f7f5006fc80 2024-05-28T06:19:58.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.011+0000 7f7f55000640 1 --2- 172.21.15.96:0/2176564526 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f5006fc80 0x7f7f500cb9a0 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f7f50073090 tx=0x7f7f4802a640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84395e6693e6104d server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:58.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.011+0000 7f7f4ea00640 1 -- 172.21.15.96:0/2176564526 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f48036070 con 0x7f7f5006fc80 2024-05-28T06:19:58.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.011+0000 7f7f4ea00640 1 -- 172.21.15.96:0/2176564526 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f48031600 con 0x7f7f5006fc80 2024-05-28T06:19:58.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.011+0000 7f7f4ea00640 1 -- 172.21.15.96:0/2176564526 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f480319d0 con 0x7f7f5006fc80 2024-05-28T06:19:58.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.012+0000 7f7f57000640 1 -- 172.21.15.96:0/2176564526 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f5006fc80 msgr2=0x7f7f500cb9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:58.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.012+0000 7f7f57000640 1 --2- 172.21.15.96:0/2176564526 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f5006fc80 0x7f7f500cb9a0 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f7f50073090 tx=0x7f7f4802a640 comp rx=0 tx=0).stop 2024-05-28T06:19:58.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.012+0000 7f7f57000640 1 -- 172.21.15.96:0/2176564526 shutdown_connections 2024-05-28T06:19:58.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.012+0000 7f7f57000640 1 --2- 172.21.15.96:0/2176564526 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f5006fc80 0x7f7f500cb9a0 unknown :-1 s=CLOSED pgs=1521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:58.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.012+0000 7f7f57000640 1 --2- 172.21.15.96:0/2176564526 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7f500cd550 0x7f7f5006f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:58.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.012+0000 7f7f57000640 1 --2- 172.21.15.96:0/2176564526 >> v2:172.21.15.123:3300/0 conn(0x7f7f500ccd20 0x7f7f5006b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:58.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.012+0000 7f7f57000640 1 -- 172.21.15.96:0/2176564526 >> 172.21.15.96:0/2176564526 conn(0x7f7f500b2f40 msgr2=0x7f7f500b3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:19:58.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.012+0000 7f7f57000640 1 -- 172.21.15.96:0/2176564526 shutdown_connections 2024-05-28T06:19:58.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.012+0000 7f7f57000640 1 -- 172.21.15.96:0/2176564526 wait complete. 2024-05-28T06:19:58.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.014+0000 7f7f57000640 1 Processor -- start 2024-05-28T06:19:58.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.014+0000 7f7f57000640 1 -- start start 2024-05-28T06:19:58.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.015+0000 7f7f57000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7f5006fc80 0x7f7f500b6c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:58.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.015+0000 7f7f57000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f500ccd20 0x7f7f5014d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:58.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.015+0000 7f7f57000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f500cd550 0x7f7f500b4a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:58.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.015+0000 7f7f57000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f7f5006c0a0 con 0x7f7f500cd550 2024-05-28T06:19:58.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.015+0000 7f7f57000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f7f5006be40 con 0x7f7f500ccd20 2024-05-28T06:19:58.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.015+0000 7f7f57000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f7f5006c380 con 0x7f7f5006fc80 2024-05-28T06:19:58.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.015+0000 7f7f55000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f500cd550 0x7f7f500b4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:58.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.015+0000 7f7f55000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f500cd550 0x7f7f500b4a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51718/0 (socket says 172.21.15.96:51718) 2024-05-28T06:19:58.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.015+0000 7f7f55000640 1 -- 172.21.15.96:0/1315183302 learned_addr learned my addr 172.21.15.96:0/1315183302 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:19:58.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.016+0000 7f7f4fe00640 1 --2- 172.21.15.96:0/1315183302 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7f5006fc80 0x7f7f500b6c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:58.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.016+0000 7f7f55000640 1 -- 172.21.15.96:0/1315183302 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7f5006fc80 msgr2=0x7f7f500b6c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:58.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.016+0000 7f7f55000640 1 --2- 172.21.15.96:0/1315183302 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7f5006fc80 0x7f7f500b6c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:58.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.016+0000 7f7f55000640 1 -- 172.21.15.96:0/1315183302 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f500ccd20 msgr2=0x7f7f5014d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:58.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.016+0000 7f7f55000640 1 --2- 172.21.15.96:0/1315183302 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f500ccd20 0x7f7f5014d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:58.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.016+0000 7f7f55000640 1 -- 172.21.15.96:0/1315183302 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f500b4f50 con 0x7f7f500cd550 2024-05-28T06:19:58.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.016+0000 7f7f55000640 1 --2- 172.21.15.96:0/1315183302 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f500cd550 0x7f7f500b4a10 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f7f48034040 tx=0x7f7f48004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:58.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.016+0000 7f7f47e00640 1 -- 172.21.15.96:0/1315183302 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f48036050 con 0x7f7f500cd550 2024-05-28T06:19:58.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.016+0000 7f7f47e00640 1 -- 172.21.15.96:0/1315183302 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f48042070 con 0x7f7f500cd550 2024-05-28T06:19:58.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.016+0000 7f7f57000640 1 -- 172.21.15.96:0/1315183302 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f500b51f0 con 0x7f7f500cd550 2024-05-28T06:19:58.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.016+0000 7f7f47e00640 1 -- 172.21.15.96:0/1315183302 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f48009510 con 0x7f7f500cd550 2024-05-28T06:19:58.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.017+0000 7f7f57000640 1 -- 172.21.15.96:0/1315183302 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f500b57b0 con 0x7f7f500cd550 2024-05-28T06:19:58.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.018+0000 7f7f57000640 1 -- 172.21.15.96:0/1315183302 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f500cfe40 con 0x7f7f500cd550 2024-05-28T06:19:58.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.019+0000 7f7f47e00640 1 -- 172.21.15.96:0/1315183302 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 33) ==== 92825+0+0 (secure 0 0 0) 0x7f7f48007bd0 con 0x7f7f500cd550 2024-05-28T06:19:58.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.019+0000 7f7f47e00640 1 --2- 172.21.15.96:0/1315183302 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7f2c06dcd0 0x7f7f2c070190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:58.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.020+0000 7f7f47e00640 1 -- 172.21.15.96:0/1315183302 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 5766+0+0 (secure 0 0 0) 0x7f7f480abb10 con 0x7f7f500cd550 2024-05-28T06:19:58.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.020+0000 7f7f4fe00640 1 --2- 172.21.15.96:0/1315183302 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7f2c06dcd0 0x7f7f2c070190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:58.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.021+0000 7f7f4fe00640 1 --2- 172.21.15.96:0/1315183302 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7f2c06dcd0 0x7f7f2c070190 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f7f4000a990 tx=0x7f7f40008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:58.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.024+0000 7f7f47e00640 1 -- 172.21.15.96:0/1315183302 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7f500cfe40 con 0x7f7f500cd550 2024-05-28T06:19:58.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.206+0000 7f7f57000640 1 -- 172.21.15.96:0/1315183302 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f7f500b0650 con 0x7f7f500cd550 2024-05-28T06:19:58.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.207+0000 7f7f47e00640 1 -- 172.21.15.96:0/1315183302 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) ==== 93+0+10 (secure 0 0 0) 0x7f7f4802ae20 con 0x7f7f500cd550 2024-05-28T06:19:58.204 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 1 2024-05-28T06:19:58.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.209+0000 7f7f57000640 1 -- 172.21.15.96:0/1315183302 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7f2c06dcd0 msgr2=0x7f7f2c070190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:58.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.209+0000 7f7f57000640 1 --2- 172.21.15.96:0/1315183302 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7f2c06dcd0 0x7f7f2c070190 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f7f4000a990 tx=0x7f7f40008040 comp rx=0 tx=0).stop 2024-05-28T06:19:58.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.209+0000 7f7f57000640 1 -- 172.21.15.96:0/1315183302 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f500cd550 msgr2=0x7f7f500b4a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:58.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.209+0000 7f7f57000640 1 --2- 172.21.15.96:0/1315183302 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f500cd550 0x7f7f500b4a10 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f7f48034040 tx=0x7f7f48004a40 comp rx=0 tx=0).stop 2024-05-28T06:19:58.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.209+0000 7f7f57000640 1 -- 172.21.15.96:0/1315183302 shutdown_connections 2024-05-28T06:19:58.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.209+0000 7f7f57000640 1 --2- 172.21.15.96:0/1315183302 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7f2c06dcd0 0x7f7f2c070190 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:58.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.209+0000 7f7f57000640 1 --2- 172.21.15.96:0/1315183302 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f500cd550 0x7f7f500b4a10 unknown :-1 s=CLOSED pgs=1522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:58.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.209+0000 7f7f57000640 1 --2- 172.21.15.96:0/1315183302 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f500ccd20 0x7f7f5014d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:58.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.209+0000 7f7f57000640 1 --2- 172.21.15.96:0/1315183302 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7f5006fc80 0x7f7f500b6c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:58.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.209+0000 7f7f57000640 1 -- 172.21.15.96:0/1315183302 >> 172.21.15.96:0/1315183302 conn(0x7f7f500b2f40 msgr2=0x7f7f500ae220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:19:58.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.210+0000 7f7f57000640 1 -- 172.21.15.96:0/1315183302 shutdown_connections 2024-05-28T06:19:58.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.210+0000 7f7f57000640 1 -- 172.21.15.96:0/1315183302 wait complete. 2024-05-28T06:19:58.225 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-28T06:19:58.416 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.418+0000 7fa3fe200640 1 Processor -- start 2024-05-28T06:19:58.416 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.419+0000 7fa3fe200640 1 -- start start 2024-05-28T06:19:58.418 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.420+0000 7fa3fe200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa3f80d3940 0x7fa3f80cf560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:58.418 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.420+0000 7fa3fe200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fa3f81620e0 0x7fa3f80cfaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:58.418 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.420+0000 7fa3fe200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3f80d28a0 con 0x7fa3f8161d00 2024-05-28T06:19:58.418 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.420+0000 7fa3fe200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fa3f80d2670 con 0x7fa3f81620e0 2024-05-28T06:19:58.418 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.420+0000 7fa3fe200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fa3f80d34b0 con 0x7fa3f80d3940 2024-05-28T06:19:58.418 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.421+0000 7fa3fce00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fa3f8161d00 0x7fa3f80d4e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:37004/0 (socket says 172.21.15.96:37004) 2024-05-28T06:19:58.418 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.421+0000 7fa3fce00640 1 -- 172.21.15.96:0/1057904912 learned_addr learned my addr 172.21.15.96:0/1057904912 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:19:58.418 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.421+0000 7fa3fd800640 1 --2- 172.21.15.96:0/1057904912 >> v2:172.21.15.96:3300/0 conn(0x7fa3f81620e0 0x7fa3f80cfaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:58.418 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.421+0000 7fa3f7e00640 1 --2- 172.21.15.96:0/1057904912 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa3f80d3940 0x7fa3f80cf560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:58.419 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.421+0000 7fa3fd800640 1 -- 172.21.15.96:0/1057904912 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa3f80d3940 msgr2=0x7fa3f80cf560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:19:58.419 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.421+0000 7fa3fd800640 1 --2- 172.21.15.96:0/1057904912 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa3f80d3940 0x7fa3f80cf560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:58.419 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.421+0000 7fa3fd800640 1 -- 172.21.15.96:0/1057904912 >> v1:172.21.15.96:6789/0 conn(0x7fa3f8161d00 legacy=0x7fa3f80d4e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:58.419 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.421+0000 7fa3fd800640 1 -- 172.21.15.96:0/1057904912 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3f80d2b80 con 0x7fa3f81620e0 2024-05-28T06:19:58.419 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.422+0000 7fa3fd800640 1 --2- 172.21.15.96:0/1057904912 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa3f81620e0 0x7fa3f80cfaa0 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7fa3f0009d90 tx=0x7fa3f002a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7aa629477b294b38 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:58.420 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.423+0000 7fa3f7400640 1 -- 172.21.15.96:0/1057904912 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3f0031600 con 0x7fa3f81620e0 2024-05-28T06:19:58.420 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.423+0000 7fa3f7400640 1 -- 172.21.15.96:0/1057904912 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3f00317a0 con 0x7fa3f81620e0 2024-05-28T06:19:58.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.423+0000 7fa3f7400640 1 -- 172.21.15.96:0/1057904912 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3f0031b70 con 0x7fa3f81620e0 2024-05-28T06:19:58.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.423+0000 7fa3fe200640 1 -- 172.21.15.96:0/1057904912 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa3f81620e0 msgr2=0x7fa3f80cfaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:58.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.423+0000 7fa3fe200640 1 --2- 172.21.15.96:0/1057904912 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa3f81620e0 0x7fa3f80cfaa0 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7fa3f0009d90 tx=0x7fa3f002a5d0 comp rx=0 tx=0).stop 2024-05-28T06:19:58.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.423+0000 7fa3fe200640 1 -- 172.21.15.96:0/1057904912 shutdown_connections 2024-05-28T06:19:58.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.423+0000 7fa3fe200640 1 --2- 172.21.15.96:0/1057904912 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa3f81620e0 0x7fa3f80cfaa0 unknown :-1 s=CLOSED pgs=1524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:58.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.423+0000 7fa3fe200640 1 --2- 172.21.15.96:0/1057904912 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa3f80d3940 0x7fa3f80cf560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:58.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.423+0000 7fa3fe200640 1 -- 172.21.15.96:0/1057904912 >> 172.21.15.96:0/1057904912 conn(0x7fa3f808bfc0 msgr2=0x7fa3f80d4500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:19:58.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.423+0000 7fa3fe200640 1 -- 172.21.15.96:0/1057904912 shutdown_connections 2024-05-28T06:19:58.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.423+0000 7fa3fe200640 1 -- 172.21.15.96:0/1057904912 wait complete. 2024-05-28T06:19:58.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.425+0000 7fa3fe200640 1 Processor -- start 2024-05-28T06:19:58.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.425+0000 7fa3fe200640 1 -- start start 2024-05-28T06:19:58.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.426+0000 7fa3fe200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa3f80d3940 0x7fa3f81504a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:58.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.426+0000 7fa3fe200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa3f8161d00 0x7fa3f814d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:58.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.426+0000 7fa3fe200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa3f81620e0 0x7fa3f814d040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:58.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.426+0000 7fa3fe200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa3f80ced60 con 0x7fa3f80d3940 2024-05-28T06:19:58.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.426+0000 7fa3fe200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa3f80d3140 con 0x7fa3f8161d00 2024-05-28T06:19:58.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.426+0000 7fa3fe200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fa3f80ceee0 con 0x7fa3f81620e0 2024-05-28T06:19:58.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.426+0000 7fa3fce00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa3f80d3940 0x7fa3f81504a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:58.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.426+0000 7fa3fce00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa3f80d3940 0x7fa3f81504a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51742/0 (socket says 172.21.15.96:51742) 2024-05-28T06:19:58.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.426+0000 7fa3fce00640 1 -- 172.21.15.96:0/1276101395 learned_addr learned my addr 172.21.15.96:0/1276101395 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:19:58.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.427+0000 7fa3f7e00640 1 --2- 172.21.15.96:0/1276101395 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa3f8161d00 0x7fa3f814d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:58.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.427+0000 7fa3fd800640 1 --2- 172.21.15.96:0/1276101395 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa3f81620e0 0x7fa3f814d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:58.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.427+0000 7fa3fce00640 1 -- 172.21.15.96:0/1276101395 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa3f81620e0 msgr2=0x7fa3f814d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:58.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.427+0000 7fa3fce00640 1 --2- 172.21.15.96:0/1276101395 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa3f81620e0 0x7fa3f814d040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:58.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.427+0000 7fa3fce00640 1 -- 172.21.15.96:0/1276101395 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa3f8161d00 msgr2=0x7fa3f814d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:58.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.427+0000 7fa3fce00640 1 --2- 172.21.15.96:0/1276101395 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa3f8161d00 0x7fa3f814d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:58.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.427+0000 7fa3fce00640 1 -- 172.21.15.96:0/1276101395 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3f814e050 con 0x7fa3f80d3940 2024-05-28T06:19:58.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.427+0000 7fa3f7e00640 1 --2- 172.21.15.96:0/1276101395 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa3f8161d00 0x7fa3f814d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:19:58.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.427+0000 7fa3fce00640 1 --2- 172.21.15.96:0/1276101395 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa3f80d3940 0x7fa3f81504a0 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7fa3e800c400 tx=0x7fa3e800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:58.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.428+0000 7fa3f4c00640 1 -- 172.21.15.96:0/1276101395 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3e8017070 con 0x7fa3f80d3940 2024-05-28T06:19:58.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.428+0000 7fa3f4c00640 1 -- 172.21.15.96:0/1276101395 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3e8005ce0 con 0x7fa3f80d3940 2024-05-28T06:19:58.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.428+0000 7fa3fe200640 1 -- 172.21.15.96:0/1276101395 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3f814e330 con 0x7fa3f80d3940 2024-05-28T06:19:58.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.428+0000 7fa3f4c00640 1 -- 172.21.15.96:0/1276101395 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3e80072f0 con 0x7fa3f80d3940 2024-05-28T06:19:58.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.428+0000 7fa3fe200640 1 -- 172.21.15.96:0/1276101395 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3f814e8f0 con 0x7fa3f80d3940 2024-05-28T06:19:58.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.429+0000 7fa3fe200640 1 -- 172.21.15.96:0/1276101395 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3f80689e0 con 0x7fa3f80d3940 2024-05-28T06:19:58.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.430+0000 7fa3f4c00640 1 -- 172.21.15.96:0/1276101395 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 33) ==== 92825+0+0 (secure 0 0 0) 0x7fa3e801a070 con 0x7fa3f80d3940 2024-05-28T06:19:58.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.431+0000 7fa3f4c00640 1 --2- 172.21.15.96:0/1276101395 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa3d806dcd0 0x7fa3d8070190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:58.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.431+0000 7fa3f4c00640 1 -- 172.21.15.96:0/1276101395 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 5766+0+0 (secure 0 0 0) 0x7fa3e808a040 con 0x7fa3f80d3940 2024-05-28T06:19:58.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.431+0000 7fa3f7e00640 1 --2- 172.21.15.96:0/1276101395 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa3d806dcd0 0x7fa3d8070190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:58.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.432+0000 7fa3f7e00640 1 --2- 172.21.15.96:0/1276101395 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa3d806dcd0 0x7fa3d8070190 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fa3f815eeb0 tx=0x7fa3e4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:58.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.435+0000 7fa3f4c00640 1 -- 172.21.15.96:0/1276101395 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa3e8059ac0 con 0x7fa3f80d3940 2024-05-28T06:19:58.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.635+0000 7fa3fe200640 1 -- 172.21.15.96:0/1276101395 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fa3f80b6c90 con 0x7fa3f80d3940 2024-05-28T06:19:58.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.636+0000 7fa3f4c00640 1 -- 172.21.15.96:0/1276101395 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) ==== 104+0+11 (secure 0 0 0) 0x7fa3e80593c0 con 0x7fa3f80d3940 2024-05-28T06:19:58.634 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 32 2024-05-28T06:19:58.635 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.638+0000 7fa3fe200640 1 -- 172.21.15.96:0/1276101395 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa3d806dcd0 msgr2=0x7fa3d8070190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:58.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.638+0000 7fa3fe200640 1 --2- 172.21.15.96:0/1276101395 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa3d806dcd0 0x7fa3d8070190 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fa3f815eeb0 tx=0x7fa3e4008040 comp rx=0 tx=0).stop 2024-05-28T06:19:58.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.638+0000 7fa3fe200640 1 -- 172.21.15.96:0/1276101395 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa3f80d3940 msgr2=0x7fa3f81504a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:58.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.638+0000 7fa3fe200640 1 --2- 172.21.15.96:0/1276101395 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa3f80d3940 0x7fa3f81504a0 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7fa3e800c400 tx=0x7fa3e800c8c0 comp rx=0 tx=0).stop 2024-05-28T06:19:58.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.639+0000 7fa3fe200640 1 -- 172.21.15.96:0/1276101395 shutdown_connections 2024-05-28T06:19:58.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.639+0000 7fa3fe200640 1 --2- 172.21.15.96:0/1276101395 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa3d806dcd0 0x7fa3d8070190 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:58.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.639+0000 7fa3fe200640 1 --2- 172.21.15.96:0/1276101395 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa3f81620e0 0x7fa3f814d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:58.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.639+0000 7fa3fe200640 1 --2- 172.21.15.96:0/1276101395 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa3f8161d00 0x7fa3f814d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:58.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.639+0000 7fa3fe200640 1 --2- 172.21.15.96:0/1276101395 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa3f80d3940 0x7fa3f81504a0 unknown :-1 s=CLOSED pgs=1525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:58.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.639+0000 7fa3fe200640 1 -- 172.21.15.96:0/1276101395 >> 172.21.15.96:0/1276101395 conn(0x7fa3f808bfc0 msgr2=0x7fa3f81600b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:19:58.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.639+0000 7fa3fe200640 1 -- 172.21.15.96:0/1276101395 shutdown_connections 2024-05-28T06:19:58.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.639+0000 7fa3fe200640 1 -- 172.21.15.96:0/1276101395 wait complete. 2024-05-28T06:19:58.655 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-28T06:19:58.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.871+0000 7ff444e00640 1 Processor -- start 2024-05-28T06:19:58.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.872+0000 7ff444e00640 1 -- start start 2024-05-28T06:19:58.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.873+0000 7ff444e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7ff4400d1910 0x7ff4400d5120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:58.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.873+0000 7ff444e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7ff44015ed50 0x7ff4400cf3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:58.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.873+0000 7ff444e00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4400b3d20 con 0x7ff4400d2bd0 2024-05-28T06:19:58.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.873+0000 7ff444e00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7ff44015f2b0 con 0x7ff44015ed50 2024-05-28T06:19:58.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.873+0000 7ff444e00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7ff44015f590 con 0x7ff4400d1910 2024-05-28T06:19:58.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.874+0000 7ff43de00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7ff4400d2bd0 0x7ff4400d2fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:55690/0 (socket says 172.21.15.96:55690) 2024-05-28T06:19:58.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.874+0000 7ff43de00640 1 -- 172.21.15.96:0/1563637070 learned_addr learned my addr 172.21.15.96:0/1563637070 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:19:58.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.874+0000 7ff43e800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7ff44015ed50 0x7ff4400cf3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:58.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.875+0000 7ff43d400640 1 --2- 172.21.15.96:0/1563637070 >> v2:172.21.15.123:3300/0 conn(0x7ff4400d1910 0x7ff4400d5120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:58.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.875+0000 7ff43ca00640 1 -- 172.21.15.96:0/1563637070 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2566984584 0 0) 0x7ff4400b3d20 con 0x7ff4400d2bd0 2024-05-28T06:19:58.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.875+0000 7ff43e800640 1 -- 172.21.15.96:0/1563637070 >> v2:172.21.15.123:3300/0 conn(0x7ff4400d1910 msgr2=0x7ff4400d5120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:19:58.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.875+0000 7ff43e800640 1 --2- 172.21.15.96:0/1563637070 >> v2:172.21.15.123:3300/0 conn(0x7ff4400d1910 0x7ff4400d5120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:58.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.875+0000 7ff43e800640 1 -- 172.21.15.96:0/1563637070 >> v1:172.21.15.96:6789/0 conn(0x7ff4400d2bd0 legacy=0x7ff4400d2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:58.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.875+0000 7ff43e800640 1 -- 172.21.15.96:0/1563637070 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4400b39f0 con 0x7ff44015ed50 2024-05-28T06:19:58.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.875+0000 7ff43d400640 1 --2- 172.21.15.96:0/1563637070 >> v2:172.21.15.123:3300/0 conn(0x7ff4400d1910 0x7ff4400d5120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:19:58.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.877+0000 7ff43e800640 1 --2- 172.21.15.96:0/1563637070 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff44015ed50 0x7ff4400cf3e0 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7ff4400d3e90 tx=0x7ff42802a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6585f793666b79df server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:58.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.877+0000 7ff43ca00640 1 -- 172.21.15.96:0/1563637070 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff428037070 con 0x7ff44015ed50 2024-05-28T06:19:58.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.877+0000 7ff43ca00640 1 -- 172.21.15.96:0/1563637070 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff428032440 con 0x7ff44015ed50 2024-05-28T06:19:58.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.877+0000 7ff43ca00640 1 -- 172.21.15.96:0/1563637070 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff42802ae60 con 0x7ff44015ed50 2024-05-28T06:19:58.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.878+0000 7ff444e00640 1 -- 172.21.15.96:0/1563637070 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff44015ed50 msgr2=0x7ff4400cf3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:58.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.878+0000 7ff444e00640 1 --2- 172.21.15.96:0/1563637070 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff44015ed50 0x7ff4400cf3e0 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7ff4400d3e90 tx=0x7ff42802a110 comp rx=0 tx=0).stop 2024-05-28T06:19:58.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.879+0000 7ff444e00640 1 -- 172.21.15.96:0/1563637070 shutdown_connections 2024-05-28T06:19:58.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.879+0000 7ff444e00640 1 --2- 172.21.15.96:0/1563637070 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff44015ed50 0x7ff4400cf3e0 unknown :-1 s=CLOSED pgs=1527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:58.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.879+0000 7ff444e00640 1 --2- 172.21.15.96:0/1563637070 >> v2:172.21.15.123:3300/0 conn(0x7ff4400d1910 0x7ff4400d5120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:58.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.879+0000 7ff444e00640 1 -- 172.21.15.96:0/1563637070 >> 172.21.15.96:0/1563637070 conn(0x7ff44008bfc0 msgr2=0x7ff4400d45f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:19:58.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.879+0000 7ff444e00640 1 -- 172.21.15.96:0/1563637070 shutdown_connections 2024-05-28T06:19:58.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.879+0000 7ff444e00640 1 -- 172.21.15.96:0/1563637070 wait complete. 2024-05-28T06:19:58.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.881+0000 7ff444e00640 1 Processor -- start 2024-05-28T06:19:58.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.881+0000 7ff444e00640 1 -- start start 2024-05-28T06:19:58.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.882+0000 7ff444e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4400d1910 0x7ff4401f51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:58.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.883+0000 7ff444e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff4400d2bd0 0x7ff440159560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:58.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.883+0000 7ff444e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff44015ed50 0x7ff4401f7e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:58.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.883+0000 7ff444e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff4400cfaa0 con 0x7ff4400d2bd0 2024-05-28T06:19:58.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.883+0000 7ff444e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff4400cf920 con 0x7ff4400d1910 2024-05-28T06:19:58.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.883+0000 7ff444e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ff4400cfc20 con 0x7ff44015ed50 2024-05-28T06:19:58.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.883+0000 7ff43d400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff4400d2bd0 0x7ff440159560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:58.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.883+0000 7ff43d400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff4400d2bd0 0x7ff440159560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:55724/0 (socket says 172.21.15.96:55724) 2024-05-28T06:19:58.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.883+0000 7ff43d400640 1 -- 172.21.15.96:0/1207394944 learned_addr learned my addr 172.21.15.96:0/1207394944 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:19:58.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.883+0000 7ff43de00640 1 --2- 172.21.15.96:0/1207394944 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4400d1910 0x7ff4401f51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:58.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.884+0000 7ff43e800640 1 --2- 172.21.15.96:0/1207394944 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff44015ed50 0x7ff4401f7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:58.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.884+0000 7ff43d400640 1 -- 172.21.15.96:0/1207394944 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff44015ed50 msgr2=0x7ff4401f7e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:58.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.884+0000 7ff43d400640 1 --2- 172.21.15.96:0/1207394944 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff44015ed50 0x7ff4401f7e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:58.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.884+0000 7ff43d400640 1 -- 172.21.15.96:0/1207394944 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4400d1910 msgr2=0x7ff4401f51b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:58.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.884+0000 7ff43d400640 1 --2- 172.21.15.96:0/1207394944 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4400d1910 0x7ff4401f51b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:58.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.884+0000 7ff43d400640 1 -- 172.21.15.96:0/1207394944 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4401f8770 con 0x7ff4400d2bd0 2024-05-28T06:19:58.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.884+0000 7ff43e800640 1 --2- 172.21.15.96:0/1207394944 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff44015ed50 0x7ff4401f7e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:19:58.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.884+0000 7ff43d400640 1 --2- 172.21.15.96:0/1207394944 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff4400d2bd0 0x7ff440159560 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7ff43000c820 tx=0x7ff43000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:58.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.884+0000 7ff436000640 1 -- 172.21.15.96:0/1207394944 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff430005b90 con 0x7ff4400d2bd0 2024-05-28T06:19:58.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.884+0000 7ff436000640 1 -- 172.21.15.96:0/1207394944 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff430005d30 con 0x7ff4400d2bd0 2024-05-28T06:19:58.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.884+0000 7ff436000640 1 -- 172.21.15.96:0/1207394944 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff430007a30 con 0x7ff4400d2bd0 2024-05-28T06:19:58.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.885+0000 7ff444e00640 1 -- 172.21.15.96:0/1207394944 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4401f8a50 con 0x7ff4400d2bd0 2024-05-28T06:19:58.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.885+0000 7ff444e00640 1 -- 172.21.15.96:0/1207394944 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4401f56f0 con 0x7ff4400d2bd0 2024-05-28T06:19:58.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.886+0000 7ff444e00640 1 -- 172.21.15.96:0/1207394944 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4400690f0 con 0x7ff4400d2bd0 2024-05-28T06:19:58.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.888+0000 7ff436000640 1 -- 172.21.15.96:0/1207394944 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 34) ==== 92927+0+0 (secure 0 0 0) 0x7ff430007bd0 con 0x7ff4400d2bd0 2024-05-28T06:19:58.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.890+0000 7ff436000640 1 --2- 172.21.15.96:0/1207394944 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff41c06ddc0 0x7ff41c070280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:58.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.890+0000 7ff436000640 1 -- 172.21.15.96:0/1207394944 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 5766+0+0 (secure 0 0 0) 0x7ff43008ad30 con 0x7ff4400d2bd0 2024-05-28T06:19:58.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.891+0000 7ff43de00640 1 --2- 172.21.15.96:0/1207394944 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff41c06ddc0 0x7ff41c070280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:58.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.892+0000 7ff43de00640 1 --2- 172.21.15.96:0/1207394944 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff41c06ddc0 0x7ff41c070280 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7ff4380012e0 tx=0x7ff438007ef0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:58.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:58.898+0000 7ff436000640 1 -- 172.21.15.96:0/1207394944 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff430015020 con 0x7ff4400d2bd0 2024-05-28T06:19:59.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:59.281+0000 7ff444e00640 1 -- 172.21.15.96:0/1207394944 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7ff4400d29b0 con 0x7ff4400d2bd0 2024-05-28T06:19:59.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:59.283+0000 7ff436000640 1 -- 172.21.15.96:0/1207394944 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) ==== 100+0+11 (secure 0 0 0) 0x7ff43005a7d0 con 0x7ff4400d2bd0 2024-05-28T06:19:59.280 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 64 2024-05-28T06:19:59.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:59.287+0000 7ff444e00640 1 -- 172.21.15.96:0/1207394944 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff41c06ddc0 msgr2=0x7ff41c070280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:59.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:59.287+0000 7ff444e00640 1 --2- 172.21.15.96:0/1207394944 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff41c06ddc0 0x7ff41c070280 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7ff4380012e0 tx=0x7ff438007ef0 comp rx=0 tx=0).stop 2024-05-28T06:19:59.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:59.287+0000 7ff444e00640 1 -- 172.21.15.96:0/1207394944 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff4400d2bd0 msgr2=0x7ff440159560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:59.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:59.287+0000 7ff444e00640 1 --2- 172.21.15.96:0/1207394944 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff4400d2bd0 0x7ff440159560 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7ff43000c820 tx=0x7ff43000cc20 comp rx=0 tx=0).stop 2024-05-28T06:19:59.285 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:59.288+0000 7ff444e00640 1 -- 172.21.15.96:0/1207394944 shutdown_connections 2024-05-28T06:19:59.285 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:59.288+0000 7ff444e00640 1 --2- 172.21.15.96:0/1207394944 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff41c06ddc0 0x7ff41c070280 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:59.285 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:59.288+0000 7ff444e00640 1 --2- 172.21.15.96:0/1207394944 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff44015ed50 0x7ff4401f7e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:59.285 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:59.288+0000 7ff444e00640 1 --2- 172.21.15.96:0/1207394944 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff4400d2bd0 0x7ff440159560 unknown :-1 s=CLOSED pgs=1528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:59.285 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:59.288+0000 7ff444e00640 1 --2- 172.21.15.96:0/1207394944 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4400d1910 0x7ff4401f51b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:59.285 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:59.288+0000 7ff444e00640 1 -- 172.21.15.96:0/1207394944 >> 172.21.15.96:0/1207394944 conn(0x7ff44008bfc0 msgr2=0x7ff44006d660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:19:59.285 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:59.288+0000 7ff444e00640 1 -- 172.21.15.96:0/1207394944 shutdown_connections 2024-05-28T06:19:59.286 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:59.288+0000 7ff444e00640 1 -- 172.21.15.96:0/1207394944 wait complete. 2024-05-28T06:19:59.303 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-28T06:19:59.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:59.466+0000 7f3985200640 1 Processor -- start 2024-05-28T06:19:59.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:59.466+0000 7f3985200640 1 -- start start 2024-05-28T06:19:59.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:59.467+0000 7f3985200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f39800d04a0 0x7f39800d0900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:59.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:59.467+0000 7f3985200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f39800ad130 0x7f39800ad5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:59.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:59.467+0000 7f3985200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f398006ed10 con 0x7f398006f860 2024-05-28T06:19:59.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:59.467+0000 7f3985200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f39800d0e40 con 0x7f39800ad130 2024-05-28T06:19:59.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:59.467+0000 7f3985200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f398015cb00 con 0x7f39800d04a0 2024-05-28T06:19:59.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:59.467+0000 7f397e800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f39800ad130 0x7f39800ad5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:59.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:59.467+0000 7f397e800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f39800ad130 0x7f39800ad5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:55736/0 (socket says 172.21.15.96:55736) 2024-05-28T06:19:59.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:59.467+0000 7f397e800640 1 -- 172.21.15.96:0/3501636792 learned_addr learned my addr 172.21.15.96:0/3501636792 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:19:59.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:59.468+0000 7f3975400640 1 --2- 172.21.15.96:0/3501636792 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f39800d04a0 0x7f39800d0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:59.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:59.468+0000 7f397e800640 1 -- 172.21.15.96:0/3501636792 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f39800d04a0 msgr2=0x7f39800d0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:19:59.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:59.468+0000 7f397e800640 1 --2- 172.21.15.96:0/3501636792 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f39800d04a0 0x7f39800d0900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:59.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:59.468+0000 7f397e800640 1 -- 172.21.15.96:0/3501636792 >> v1:172.21.15.123:6789/0 conn(0x7f398006f860 legacy=0x7f39800b3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:19:59.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:59.468+0000 7f397e800640 1 -- 172.21.15.96:0/3501636792 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39800b3cc0 con 0x7f39800ad130 2024-05-28T06:19:59.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:59.468+0000 7f3975400640 1 --2- 172.21.15.96:0/3501636792 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f39800d04a0 0x7f39800d0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:19:59.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:59.469+0000 7f397e800640 1 --2- 172.21.15.96:0/3501636792 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f39800ad130 0x7f39800ad5a0 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7f3978009e50 tx=0x7f397802a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a22c9957b377ad4 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:59.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:59.469+0000 7f397d400640 1 -- 172.21.15.96:0/3501636792 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3978036070 con 0x7f39800ad130 2024-05-28T06:19:59.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:59.469+0000 7f397d400640 1 -- 172.21.15.96:0/3501636792 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3978031600 con 0x7f39800ad130 2024-05-28T06:19:59.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:59.469+0000 7f397d400640 1 -- 172.21.15.96:0/3501636792 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39780319d0 con 0x7f39800ad130 2024-05-28T06:19:59.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:59.470+0000 7f3985200640 1 -- 172.21.15.96:0/3501636792 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f39800ad130 msgr2=0x7f39800ad5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:59.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:59.470+0000 7f3985200640 1 --2- 172.21.15.96:0/3501636792 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f39800ad130 0x7f39800ad5a0 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7f3978009e50 tx=0x7f397802a6c0 comp rx=0 tx=0).stop 2024-05-28T06:19:59.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:59.470+0000 7f3985200640 1 -- 172.21.15.96:0/3501636792 shutdown_connections 2024-05-28T06:19:59.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:59.470+0000 7f3985200640 1 --2- 172.21.15.96:0/3501636792 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f39800ad130 0x7f39800ad5a0 unknown :-1 s=CLOSED pgs=1529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:59.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:59.470+0000 7f3985200640 1 --2- 172.21.15.96:0/3501636792 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f39800d04a0 0x7f39800d0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:59.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:59.470+0000 7f3985200640 1 -- 172.21.15.96:0/3501636792 >> 172.21.15.96:0/3501636792 conn(0x7f398006bae0 msgr2=0x7f398006bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:19:59.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:59.470+0000 7f3985200640 1 -- 172.21.15.96:0/3501636792 shutdown_connections 2024-05-28T06:19:59.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:59.470+0000 7f3985200640 1 -- 172.21.15.96:0/3501636792 wait complete. 2024-05-28T06:19:59.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:59.471+0000 7f3985200640 1 Processor -- start 2024-05-28T06:19:59.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:59.471+0000 7f3985200640 1 -- start start 2024-05-28T06:19:59.469 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:59.472+0000 7f3985200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f398006f860 0x7f3980157180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:59.469 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:59.472+0000 7f3985200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f39800ad130 0x7f3980099650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:59.469 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:59.472+0000 7f3985200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f39800d04a0 0x7f3980158b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:59.470 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:59.472+0000 7f3985200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f39800b4560 con 0x7f39800ad130 2024-05-28T06:19:59.470 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:59.472+0000 7f3985200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f39800b4280 con 0x7f398006f860 2024-05-28T06:19:59.470 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:59.472+0000 7f3985200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f39800b4840 con 0x7f39800d04a0 2024-05-28T06:19:59.470 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:59.472+0000 7f3975400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f39800ad130 0x7f3980099650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:59.470 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:59.473+0000 7f3975400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f39800ad130 0x7f3980099650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:55744/0 (socket says 172.21.15.96:55744) 2024-05-28T06:19:59.470 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:59.473+0000 7f3975400640 1 -- 172.21.15.96:0/2447927986 learned_addr learned my addr 172.21.15.96:0/2447927986 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:19:59.474 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:59.473+0000 7f3975400640 1 -- 172.21.15.96:0/2447927986 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f39800d04a0 msgr2=0x7f3980158b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:59.475 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:59.473+0000 7f3975400640 1 --2- 172.21.15.96:0/2447927986 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f39800d04a0 0x7f3980158b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:59.475 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:59.473+0000 7f3975400640 1 -- 172.21.15.96:0/2447927986 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f398006f860 msgr2=0x7f3980157180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:59.475 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:59.473+0000 7f3975400640 1 --2- 172.21.15.96:0/2447927986 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f398006f860 0x7f3980157180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:59.475 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:59.473+0000 7f3975400640 1 -- 172.21.15.96:0/2447927986 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3980159f50 con 0x7f39800ad130 2024-05-28T06:19:59.475 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:59.473+0000 7f3975400640 1 --2- 172.21.15.96:0/2447927986 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f39800ad130 0x7f3980099650 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f396800c7f0 tx=0x7f396800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:59.475 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:59.473+0000 7f3976a00640 1 -- 172.21.15.96:0/2447927986 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3968015070 con 0x7f39800ad130 2024-05-28T06:19:59.475 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:59.473+0000 7f3976a00640 1 -- 172.21.15.96:0/2447927986 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3968007a30 con 0x7f39800ad130 2024-05-28T06:19:59.475 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:59.473+0000 7f3976a00640 1 -- 172.21.15.96:0/2447927986 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3968007e00 con 0x7f39800ad130 2024-05-28T06:19:59.475 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:59.473+0000 7f3985200640 1 -- 172.21.15.96:0/2447927986 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3980159990 con 0x7f39800ad130 2024-05-28T06:19:59.475 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:59.474+0000 7f3985200640 1 -- 172.21.15.96:0/2447927986 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3980099b90 con 0x7f39800ad130 2024-05-28T06:19:59.475 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:59.475+0000 7f3985200640 1 -- 172.21.15.96:0/2447927986 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39800689e0 con 0x7f39800ad130 2024-05-28T06:19:59.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:59.475+0000 7f3976a00640 1 -- 172.21.15.96:0/2447927986 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 34) ==== 92927+0+0 (secure 0 0 0) 0x7f3968018070 con 0x7f39800ad130 2024-05-28T06:19:59.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:59.476+0000 7f3976a00640 1 --2- 172.21.15.96:0/2447927986 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f395c06ddc0 0x7f395c070280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:19:59.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:59.476+0000 7f3976a00640 1 -- 172.21.15.96:0/2447927986 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 5766+0+0 (secure 0 0 0) 0x7f396808a840 con 0x7f39800ad130 2024-05-28T06:19:59.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:59.477+0000 7f397de00640 1 --2- 172.21.15.96:0/2447927986 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f395c06ddc0 0x7f395c070280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:19:59.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:59.478+0000 7f397de00640 1 --2- 172.21.15.96:0/2447927986 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f395c06ddc0 0x7f395c070280 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f3970006fd0 tx=0x7f3970006bc0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:19:59.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:59.481+0000 7f3976a00640 1 -- 172.21.15.96:0/2447927986 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f396805a6c0 con 0x7f39800ad130 2024-05-28T06:19:59.673 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:59.676+0000 7f3985200640 1 -- 172.21.15.96:0/2447927986 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f398006e2f0 con 0x7f39800ad130 2024-05-28T06:19:59.673 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:59.676+0000 7f3976a00640 1 -- 172.21.15.96:0/2447927986 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v91) ==== 80+0+0 (secure 0 0 0) 0x7f3968059fc0 con 0x7f39800ad130 2024-05-28T06:19:59.674 INFO:teuthology.orchestra.run.smithi096.stderr:listed 0 entries 2024-05-28T06:19:59.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:59.679+0000 7f3985200640 1 -- 172.21.15.96:0/2447927986 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f395c06ddc0 msgr2=0x7f395c070280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:59.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:59.679+0000 7f3985200640 1 --2- 172.21.15.96:0/2447927986 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f395c06ddc0 0x7f395c070280 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f3970006fd0 tx=0x7f3970006bc0 comp rx=0 tx=0).stop 2024-05-28T06:19:59.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:59.679+0000 7f3985200640 1 -- 172.21.15.96:0/2447927986 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f39800ad130 msgr2=0x7f3980099650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:19:59.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:59.679+0000 7f3985200640 1 --2- 172.21.15.96:0/2447927986 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f39800ad130 0x7f3980099650 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f396800c7f0 tx=0x7f396800cbf0 comp rx=0 tx=0).stop 2024-05-28T06:19:59.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:59.679+0000 7f3985200640 1 -- 172.21.15.96:0/2447927986 shutdown_connections 2024-05-28T06:19:59.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:59.679+0000 7f3985200640 1 --2- 172.21.15.96:0/2447927986 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f395c06ddc0 0x7f395c070280 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:59.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:59.679+0000 7f3985200640 1 --2- 172.21.15.96:0/2447927986 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f39800d04a0 0x7f3980158b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:59.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:59.679+0000 7f3985200640 1 --2- 172.21.15.96:0/2447927986 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f39800ad130 0x7f3980099650 unknown :-1 s=CLOSED pgs=1530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:59.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:59.679+0000 7f3985200640 1 --2- 172.21.15.96:0/2447927986 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f398006f860 0x7f3980157180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:19:59.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:59.679+0000 7f3985200640 1 -- 172.21.15.96:0/2447927986 >> 172.21.15.96:0/2447927986 conn(0x7f398006bae0 msgr2=0x7f3980071b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:19:59.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:59.679+0000 7f3985200640 1 -- 172.21.15.96:0/2447927986 shutdown_connections 2024-05-28T06:19:59.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:19:59.679+0000 7f3985200640 1 -- 172.21.15.96:0/2447927986 wait complete. 2024-05-28T06:19:59.694 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi096.front.sepia.ceph.com... 2024-05-28T06:19:59.694 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:19:59.694 DEBUG:teuthology.orchestra.run.smithi096:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-05-28T06:19:59.768 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi096.stderr:ceph-fuse[71726]: fuse finished with error 0 and tester_r 0 2024-05-28T06:19:59.772 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi096.front.sepia.ceph.com 2024-05-28T06:19:59.772 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:19:59.772 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-28T06:19:59.984 INFO:tasks.cephfs.mount:Terminating background process 2024-05-28T06:19:59.984 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-05-28T06:19:59.984 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:20:00.047 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052731,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248996k,nr_inodes=812249,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-05-28T06:20:00.047 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi123.front.sepia.ceph.com 2024-05-28T06:20:00.047 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:20:00.047 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-05-28T06:20:00.105 INFO:teuthology.orchestra.run.smithi123.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-05-28T06:20:00.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-28T06:20:00.106 INFO:tasks.cephfs.mount:Terminating background process 2024-05-28T06:20:00.106 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:20:00.106 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:20:00.170 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052731,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248996k,nr_inodes=812249,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-28T06:20:00.170 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi096.front.sepia.ceph.com 2024-05-28T06:20:00.170 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:20:00.170 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-05-28T06:20:00.229 INFO:teuthology.orchestra.run.smithi096.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-05-28T06:20:00.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-28T06:20:00.230 INFO:tasks.cephfs.mount:Terminating background process 2024-05-28T06:20:00.230 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-05-28T06:20:00.230 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:20:00.266 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052731,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248996k,nr_inodes=812249,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-05-28T06:20:00.266 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi123.front.sepia.ceph.com 2024-05-28T06:20:00.266 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:20:00.266 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-05-28T06:20:00.349 INFO:teuthology.orchestra.run.smithi123.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-05-28T06:20:00.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-28T06:20:00.350 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-28T06:20:00.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.545+0000 7efc37200640 1 Processor -- start 2024-05-28T06:20:00.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.545+0000 7efc37200640 1 -- start start 2024-05-28T06:20:00.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.546+0000 7efc37200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7efc3815b3a0 0x7efc380b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:00.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.546+0000 7efc37200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7efc3806dc20 0x7efc3815f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:00.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.547+0000 7efc37200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efc38072c00 0x7efc38073050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:00.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.547+0000 7efc37200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7efc3806f980 con 0x7efc3806dc20 2024-05-28T06:20:00.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.547+0000 7efc37200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7efc38073590 con 0x7efc3815b3a0 2024-05-28T06:20:00.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.547+0000 7efc37200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7efc380ae0c0 con 0x7efc38072c00 2024-05-28T06:20:00.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.547+0000 7efc2fe00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7efc3806dc20 0x7efc3815f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:00.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.547+0000 7efc2fe00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7efc3806dc20 0x7efc3815f9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:55748/0 (socket says 172.21.15.96:55748) 2024-05-28T06:20:00.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.547+0000 7efc2fe00640 1 -- 172.21.15.96:0/2350440265 learned_addr learned my addr 172.21.15.96:0/2350440265 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:20:00.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.547+0000 7efc2fe00640 1 -- 172.21.15.96:0/2350440265 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efc38072c00 msgr2=0x7efc38073050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:00.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.547+0000 7efc2fe00640 1 --2- 172.21.15.96:0/2350440265 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efc38072c00 0x7efc38073050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:00.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.547+0000 7efc2fe00640 1 -- 172.21.15.96:0/2350440265 >> v2:172.21.15.123:3300/0 conn(0x7efc3815b3a0 msgr2=0x7efc380b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:00.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.547+0000 7efc2fe00640 1 --2- 172.21.15.96:0/2350440265 >> v2:172.21.15.123:3300/0 conn(0x7efc3815b3a0 0x7efc380b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:00.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.547+0000 7efc2fe00640 1 -- 172.21.15.96:0/2350440265 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc380afd60 con 0x7efc3806dc20 2024-05-28T06:20:00.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.548+0000 7efc2fe00640 1 --2- 172.21.15.96:0/2350440265 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efc3806dc20 0x7efc3815f9a0 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7efc38072bd0 tx=0x7efc2002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac973749c50355f8 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:00.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.549+0000 7efc35400640 1 -- 172.21.15.96:0/2350440265 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc20037070 con 0x7efc3806dc20 2024-05-28T06:20:00.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.549+0000 7efc35400640 1 -- 172.21.15.96:0/2350440265 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc20032440 con 0x7efc3806dc20 2024-05-28T06:20:00.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.549+0000 7efc35400640 1 -- 172.21.15.96:0/2350440265 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc20032810 con 0x7efc3806dc20 2024-05-28T06:20:00.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.549+0000 7efc37200640 1 -- 172.21.15.96:0/2350440265 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efc3806dc20 msgr2=0x7efc3815f9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:00.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.549+0000 7efc37200640 1 --2- 172.21.15.96:0/2350440265 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efc3806dc20 0x7efc3815f9a0 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7efc38072bd0 tx=0x7efc2002a1e0 comp rx=0 tx=0).stop 2024-05-28T06:20:00.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.549+0000 7efc37200640 1 -- 172.21.15.96:0/2350440265 shutdown_connections 2024-05-28T06:20:00.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.549+0000 7efc37200640 1 --2- 172.21.15.96:0/2350440265 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efc38072c00 0x7efc38073050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:00.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.549+0000 7efc37200640 1 --2- 172.21.15.96:0/2350440265 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efc3806dc20 0x7efc3815f9a0 unknown :-1 s=CLOSED pgs=1531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:00.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.549+0000 7efc37200640 1 --2- 172.21.15.96:0/2350440265 >> v2:172.21.15.123:3300/0 conn(0x7efc3815b3a0 0x7efc380b32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:00.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.549+0000 7efc37200640 1 -- 172.21.15.96:0/2350440265 >> 172.21.15.96:0/2350440265 conn(0x7efc380b1fc0 msgr2=0x7efc380b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:00.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.549+0000 7efc37200640 1 -- 172.21.15.96:0/2350440265 shutdown_connections 2024-05-28T06:20:00.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.549+0000 7efc37200640 1 -- 172.21.15.96:0/2350440265 wait complete. 2024-05-28T06:20:00.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.551+0000 7efc37200640 1 Processor -- start 2024-05-28T06:20:00.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.551+0000 7efc37200640 1 -- start start 2024-05-28T06:20:00.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.552+0000 7efc37200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc3806dc20 0x7efc38152110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:00.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.552+0000 7efc37200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efc38072c00 0x7efc381482b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:00.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.552+0000 7efc37200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efc3815b3a0 0x7efc38147510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:00.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.552+0000 7efc37200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7efc380afa80 con 0x7efc3815b3a0 2024-05-28T06:20:00.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.552+0000 7efc37200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7efc3815fee0 con 0x7efc3806dc20 2024-05-28T06:20:00.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.552+0000 7efc37200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7efc3815b780 con 0x7efc38072c00 2024-05-28T06:20:00.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.552+0000 7efc36800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efc3815b3a0 0x7efc38147510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:00.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.552+0000 7efc36800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efc3815b3a0 0x7efc38147510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:55758/0 (socket says 172.21.15.96:55758) 2024-05-28T06:20:00.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.552+0000 7efc36800640 1 -- 172.21.15.96:0/337218867 learned_addr learned my addr 172.21.15.96:0/337218867 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:20:00.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.553+0000 7efc2fe00640 1 --2- 172.21.15.96:0/337218867 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efc38072c00 0x7efc381482b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:00.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.553+0000 7efc36800640 1 -- 172.21.15.96:0/337218867 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efc38072c00 msgr2=0x7efc381482b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:00.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.553+0000 7efc35e00640 1 --2- 172.21.15.96:0/337218867 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc3806dc20 0x7efc38152110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:00.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.553+0000 7efc36800640 1 --2- 172.21.15.96:0/337218867 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efc38072c00 0x7efc381482b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:00.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.553+0000 7efc36800640 1 -- 172.21.15.96:0/337218867 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc3806dc20 msgr2=0x7efc38152110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:00.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.553+0000 7efc36800640 1 --2- 172.21.15.96:0/337218867 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc3806dc20 0x7efc38152110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:00.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.553+0000 7efc36800640 1 -- 172.21.15.96:0/337218867 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc380cd990 con 0x7efc3815b3a0 2024-05-28T06:20:00.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.553+0000 7efc36800640 1 --2- 172.21.15.96:0/337218867 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efc3815b3a0 0x7efc38147510 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7efc3000b9e0 tx=0x7efc3000bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:00.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.553+0000 7efc2e000640 1 -- 172.21.15.96:0/337218867 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc30019070 con 0x7efc3815b3a0 2024-05-28T06:20:00.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.553+0000 7efc2e000640 1 -- 172.21.15.96:0/337218867 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc3000cac0 con 0x7efc3815b3a0 2024-05-28T06:20:00.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.553+0000 7efc37200640 1 -- 172.21.15.96:0/337218867 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc380cdc70 con 0x7efc3815b3a0 2024-05-28T06:20:00.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.553+0000 7efc2e000640 1 -- 172.21.15.96:0/337218867 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc30014440 con 0x7efc3815b3a0 2024-05-28T06:20:00.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.554+0000 7efc37200640 1 -- 172.21.15.96:0/337218867 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc38146ff0 con 0x7efc3815b3a0 2024-05-28T06:20:00.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.555+0000 7efc37200640 1 -- 172.21.15.96:0/337218867 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc38146b00 con 0x7efc3815b3a0 2024-05-28T06:20:00.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.555+0000 7efc2e000640 1 -- 172.21.15.96:0/337218867 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 34) ==== 92927+0+0 (secure 0 0 0) 0x7efc3001c070 con 0x7efc3815b3a0 2024-05-28T06:20:00.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.557+0000 7efc2e000640 1 --2- 172.21.15.96:0/337218867 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7efc1406ddc0 0x7efc14070280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:00.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.557+0000 7efc2e000640 1 -- 172.21.15.96:0/337218867 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 5766+0+0 (secure 0 0 0) 0x7efc3008a560 con 0x7efc3815b3a0 2024-05-28T06:20:00.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.557+0000 7efc35e00640 1 --2- 172.21.15.96:0/337218867 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7efc1406ddc0 0x7efc14070280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:00.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.559+0000 7efc35e00640 1 --2- 172.21.15.96:0/337218867 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7efc1406ddc0 0x7efc14070280 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7efc3815e9e0 tx=0x7efc28008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:00.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.561+0000 7efc2e000640 1 -- 172.21.15.96:0/337218867 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7efc3005a3e0 con 0x7efc3815b3a0 2024-05-28T06:20:00.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.786+0000 7efc37200640 1 -- 172.21.15.96:0/337218867 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efc380689e0 con 0x7efc3815b3a0 2024-05-28T06:20:00.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.788+0000 7efc2e000640 1 -- 172.21.15.96:0/337218867 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) ==== 94+0+5189 (secure 0 0 0) 0x7efc30059ce0 con 0x7efc3815b3a0 2024-05-28T06:20:00.785 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:20:00.785 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":53,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5645,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6834/1355344409","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":1355344409},{"type":"v1","addr":"172.21.15.96:6834","nonce":1355344409}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":47},{"gid":5651,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/1967688533","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":1967688533},{"type":"v1","addr":"172.21.15.123:6835","nonce":1967688533}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48},{"gid":6169,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/3362018278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":3362018278},{"type":"v1","addr":"172.21.15.123:6837","nonce":3362018278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-28T06:18:10.069593+0000","modified":"2024-05-28T06:18:13.212203+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8103},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8103":{"gid":8103,"name":"c","rank":0,"incarnation":49,"state":"up:active","state_seq":3,"addr":"172.21.15.96:6835/1614743177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6833","nonce":1614743177},{"type":"v1","addr":"172.21.15.96:6835","nonce":1614743177}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8103,"qdb_cluster":[8103]},"id":6}]} 2024-05-28T06:20:00.785 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 53 2024-05-28T06:20:00.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.791+0000 7efc37200640 1 -- 172.21.15.96:0/337218867 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7efc1406ddc0 msgr2=0x7efc14070280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:00.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.791+0000 7efc37200640 1 --2- 172.21.15.96:0/337218867 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7efc1406ddc0 0x7efc14070280 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7efc3815e9e0 tx=0x7efc28008040 comp rx=0 tx=0).stop 2024-05-28T06:20:00.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.791+0000 7efc37200640 1 -- 172.21.15.96:0/337218867 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efc3815b3a0 msgr2=0x7efc38147510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:00.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.791+0000 7efc37200640 1 --2- 172.21.15.96:0/337218867 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efc3815b3a0 0x7efc38147510 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7efc3000b9e0 tx=0x7efc3000bea0 comp rx=0 tx=0).stop 2024-05-28T06:20:00.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.791+0000 7efc37200640 1 -- 172.21.15.96:0/337218867 shutdown_connections 2024-05-28T06:20:00.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.791+0000 7efc37200640 1 --2- 172.21.15.96:0/337218867 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7efc1406ddc0 0x7efc14070280 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:00.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.791+0000 7efc37200640 1 --2- 172.21.15.96:0/337218867 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efc3815b3a0 0x7efc38147510 unknown :-1 s=CLOSED pgs=1532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:00.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.791+0000 7efc37200640 1 --2- 172.21.15.96:0/337218867 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efc38072c00 0x7efc381482b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:00.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.791+0000 7efc37200640 1 --2- 172.21.15.96:0/337218867 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc3806dc20 0x7efc38152110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:00.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.791+0000 7efc37200640 1 -- 172.21.15.96:0/337218867 >> 172.21.15.96:0/337218867 conn(0x7efc380b1fc0 msgr2=0x7efc380b38c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:00.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.791+0000 7efc37200640 1 -- 172.21.15.96:0/337218867 shutdown_connections 2024-05-28T06:20:00.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.791+0000 7efc37200640 1 -- 172.21.15.96:0/337218867 wait complete. 2024-05-28T06:20:00.806 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:20:00.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.988+0000 7fe86f000640 1 Processor -- start 2024-05-28T06:20:00.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.988+0000 7fe86f000640 1 -- start start 2024-05-28T06:20:00.987 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.989+0000 7fe86f000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fe86815b3a0 0x7fe8680b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:00.987 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.989+0000 7fe86f000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe868073310 0x7fe86815f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:00.987 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.989+0000 7fe86f000640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8680afd60 con 0x7fe86806dc20 2024-05-28T06:20:00.987 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.989+0000 7fe86f000640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fe86806f980 con 0x7fe86815b3a0 2024-05-28T06:20:00.987 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.989+0000 7fe86f000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fe8680ae0c0 con 0x7fe868073310 2024-05-28T06:20:00.987 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.989+0000 7fe86dc00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fe86815b3a0 0x7fe8680b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:00.987 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.989+0000 7fe86dc00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fe86815b3a0 0x7fe8680b32c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:55772/0 (socket says 172.21.15.96:55772) 2024-05-28T06:20:00.987 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.989+0000 7fe86dc00640 1 -- 172.21.15.96:0/3966581043 learned_addr learned my addr 172.21.15.96:0/3966581043 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:20:00.987 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.990+0000 7fe86e600640 1 --2- 172.21.15.96:0/3966581043 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe868073310 0x7fe86815f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:00.987 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.990+0000 7fe86dc00640 1 -- 172.21.15.96:0/3966581043 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe868073310 msgr2=0x7fe86815f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:00.988 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.990+0000 7fe86dc00640 1 --2- 172.21.15.96:0/3966581043 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe868073310 0x7fe86815f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:00.988 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.990+0000 7fe86dc00640 1 -- 172.21.15.96:0/3966581043 >> v1:172.21.15.123:6789/0 conn(0x7fe86806dc20 legacy=0x7fe868072c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:00.988 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.990+0000 7fe86dc00640 1 -- 172.21.15.96:0/3966581043 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8680af7a0 con 0x7fe86815b3a0 2024-05-28T06:20:00.988 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.990+0000 7fe86e600640 1 --2- 172.21.15.96:0/3966581043 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe868073310 0x7fe86815f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:20:00.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.991+0000 7fe86dc00640 1 --2- 172.21.15.96:0/3966581043 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe86815b3a0 0x7fe8680b32c0 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7fe85c009870 tx=0x7fe85c02a200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55c1e86083f7ebf5 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:00.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.991+0000 7fe867e00640 1 -- 172.21.15.96:0/3966581043 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe85c037070 con 0x7fe86815b3a0 2024-05-28T06:20:00.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.991+0000 7fe867e00640 1 -- 172.21.15.96:0/3966581043 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe85c032440 con 0x7fe86815b3a0 2024-05-28T06:20:00.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.991+0000 7fe867e00640 1 -- 172.21.15.96:0/3966581043 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe85c032810 con 0x7fe86815b3a0 2024-05-28T06:20:00.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.992+0000 7fe86f000640 1 -- 172.21.15.96:0/3966581043 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe86815b3a0 msgr2=0x7fe8680b32c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:00.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.992+0000 7fe86f000640 1 --2- 172.21.15.96:0/3966581043 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe86815b3a0 0x7fe8680b32c0 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7fe85c009870 tx=0x7fe85c02a200 comp rx=0 tx=0).stop 2024-05-28T06:20:00.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.992+0000 7fe86f000640 1 -- 172.21.15.96:0/3966581043 shutdown_connections 2024-05-28T06:20:00.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.992+0000 7fe86f000640 1 --2- 172.21.15.96:0/3966581043 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe868073310 0x7fe86815f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:00.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.992+0000 7fe86f000640 1 --2- 172.21.15.96:0/3966581043 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe86815b3a0 0x7fe8680b32c0 unknown :-1 s=CLOSED pgs=1533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:00.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.992+0000 7fe86f000640 1 -- 172.21.15.96:0/3966581043 >> 172.21.15.96:0/3966581043 conn(0x7fe8680b1fc0 msgr2=0x7fe8680b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:00.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.992+0000 7fe86f000640 1 -- 172.21.15.96:0/3966581043 shutdown_connections 2024-05-28T06:20:00.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.992+0000 7fe86f000640 1 -- 172.21.15.96:0/3966581043 wait complete. 2024-05-28T06:20:00.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.993+0000 7fe86f000640 1 Processor -- start 2024-05-28T06:20:00.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.994+0000 7fe86f000640 1 -- start start 2024-05-28T06:20:00.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.994+0000 7fe86f000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe86806dc20 0x7fe86814d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:00.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.994+0000 7fe86f000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe868073310 0x7fe868157d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:00.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.994+0000 7fe86f000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe86815b3a0 0x7fe868159a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:00.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.994+0000 7fe86f000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe86815b780 con 0x7fe86806dc20 2024-05-28T06:20:00.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.994+0000 7fe86f000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe86815fee0 con 0x7fe86815b3a0 2024-05-28T06:20:00.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.994+0000 7fe86f000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fe8680ae550 con 0x7fe868073310 2024-05-28T06:20:00.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.995+0000 7fe86dc00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe86806dc20 0x7fe86814d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:00.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.995+0000 7fe86dc00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe86806dc20 0x7fe86814d250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:55782/0 (socket says 172.21.15.96:55782) 2024-05-28T06:20:00.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.995+0000 7fe86dc00640 1 -- 172.21.15.96:0/2021774439 learned_addr learned my addr 172.21.15.96:0/2021774439 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:20:00.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.995+0000 7fe86d200640 1 --2- 172.21.15.96:0/2021774439 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe868073310 0x7fe868157d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:00.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.995+0000 7fe86e600640 1 --2- 172.21.15.96:0/2021774439 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe86815b3a0 0x7fe868159a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:00.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.995+0000 7fe86dc00640 1 -- 172.21.15.96:0/2021774439 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe868073310 msgr2=0x7fe868157d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:00.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.995+0000 7fe86dc00640 1 --2- 172.21.15.96:0/2021774439 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe868073310 0x7fe868157d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:00.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.995+0000 7fe86dc00640 1 -- 172.21.15.96:0/2021774439 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe86815b3a0 msgr2=0x7fe868159a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:00.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.995+0000 7fe86dc00640 1 --2- 172.21.15.96:0/2021774439 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe86815b3a0 0x7fe868159a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:00.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.995+0000 7fe86dc00640 1 -- 172.21.15.96:0/2021774439 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8681f2b80 con 0x7fe86806dc20 2024-05-28T06:20:00.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.995+0000 7fe86d200640 1 --2- 172.21.15.96:0/2021774439 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe868073310 0x7fe868157d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:20:00.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.995+0000 7fe86dc00640 1 --2- 172.21.15.96:0/2021774439 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe86806dc20 0x7fe86814d250 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7fe85c009870 tx=0x7fe85c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:00.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.996+0000 7fe865600640 1 -- 172.21.15.96:0/2021774439 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe85c041070 con 0x7fe86806dc20 2024-05-28T06:20:00.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.996+0000 7fe865600640 1 -- 172.21.15.96:0/2021774439 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe85c03c870 con 0x7fe86806dc20 2024-05-28T06:20:00.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.996+0000 7fe865600640 1 -- 172.21.15.96:0/2021774439 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe85c037040 con 0x7fe86806dc20 2024-05-28T06:20:00.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.996+0000 7fe86f000640 1 -- 172.21.15.96:0/2021774439 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8681f2e60 con 0x7fe86806dc20 2024-05-28T06:20:00.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.996+0000 7fe86f000640 1 -- 172.21.15.96:0/2021774439 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8681f3420 con 0x7fe86806dc20 2024-05-28T06:20:00.995 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.997+0000 7fe86f000640 1 -- 172.21.15.96:0/2021774439 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8680689e0 con 0x7fe86806dc20 2024-05-28T06:20:00.995 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.998+0000 7fe865600640 1 -- 172.21.15.96:0/2021774439 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 34) ==== 92927+0+0 (secure 0 0 0) 0x7fe85c044070 con 0x7fe86806dc20 2024-05-28T06:20:00.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.999+0000 7fe865600640 1 --2- 172.21.15.96:0/2021774439 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe84806ddc0 0x7fe848070280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:00.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.999+0000 7fe865600640 1 -- 172.21.15.96:0/2021774439 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 5822+0+0 (secure 0 0 0) 0x7fe85c0ab4a0 con 0x7fe86806dc20 2024-05-28T06:20:00.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:00.999+0000 7fe86d200640 1 --2- 172.21.15.96:0/2021774439 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe84806ddc0 0x7fe848070280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:00.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.000+0000 7fe86d200640 1 --2- 172.21.15.96:0/2021774439 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe84806ddc0 0x7fe848070280 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fe854007fd0 tx=0x7fe854009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:00.999 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.003+0000 7fe865600640 1 -- 172.21.15.96:0/2021774439 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe85c07b320 con 0x7fe86806dc20 2024-05-28T06:20:01.196 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.199+0000 7fe86f000640 1 -- 172.21.15.96:0/2021774439 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe86814f730 con 0x7fe86806dc20 2024-05-28T06:20:01.199 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.202+0000 7fe865600640 1 -- 172.21.15.96:0/2021774439 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) ==== 74+0+16876 (secure 0 0 0) 0x7fe85c07ac20 con 0x7fe86806dc20 2024-05-28T06:20:01.199 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:20:01.199 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":92,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:20:00.747831+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-05-28T06:18:07.815707+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-05-28T06:18:09.061362+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":92,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"13.3b","mappings":[{"from":1,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":13,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[{"pool":13,"snaps":[{"begin":2,"length":2}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:20:01.202 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.205+0000 7fe86f000640 1 -- 172.21.15.96:0/2021774439 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe84806ddc0 msgr2=0x7fe848070280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:01.202 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.205+0000 7fe86f000640 1 --2- 172.21.15.96:0/2021774439 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe84806ddc0 0x7fe848070280 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fe854007fd0 tx=0x7fe854009040 comp rx=0 tx=0).stop 2024-05-28T06:20:01.202 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.205+0000 7fe86f000640 1 -- 172.21.15.96:0/2021774439 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe86806dc20 msgr2=0x7fe86814d250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:01.202 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.205+0000 7fe86f000640 1 --2- 172.21.15.96:0/2021774439 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe86806dc20 0x7fe86814d250 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7fe85c009870 tx=0x7fe85c0046c0 comp rx=0 tx=0).stop 2024-05-28T06:20:01.202 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.205+0000 7fe86f000640 1 -- 172.21.15.96:0/2021774439 shutdown_connections 2024-05-28T06:20:01.202 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.205+0000 7fe86f000640 1 --2- 172.21.15.96:0/2021774439 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe84806ddc0 0x7fe848070280 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:01.202 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.205+0000 7fe86f000640 1 --2- 172.21.15.96:0/2021774439 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe86815b3a0 0x7fe868159a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:01.202 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.205+0000 7fe86f000640 1 --2- 172.21.15.96:0/2021774439 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe868073310 0x7fe868157d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:01.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.205+0000 7fe86f000640 1 --2- 172.21.15.96:0/2021774439 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe86806dc20 0x7fe86814d250 unknown :-1 s=CLOSED pgs=1534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:01.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.205+0000 7fe86f000640 1 -- 172.21.15.96:0/2021774439 >> 172.21.15.96:0/2021774439 conn(0x7fe8680b1fc0 msgr2=0x7fe8680b3d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:01.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.205+0000 7fe86f000640 1 -- 172.21.15.96:0/2021774439 shutdown_connections 2024-05-28T06:20:01.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.205+0000 7fe86f000640 1 -- 172.21.15.96:0/2021774439 wait complete. 2024-05-28T06:20:01.221 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-28T06:03:59.517102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-28T06:18:07.815707+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2024-05-28T06:18:09.061362+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '92', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 92, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-28T06:20:01.221 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-28T06:20:01.419 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.422+0000 7fe910600640 1 Processor -- start 2024-05-28T06:20:01.419 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.422+0000 7fe910600640 1 -- start start 2024-05-28T06:20:01.420 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.423+0000 7fe910600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fe9080d0640 0x7fe90815d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:01.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.423+0000 7fe910600640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9080d0060 con 0x7fe9080b32c0 2024-05-28T06:20:01.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.423+0000 7fe910600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe90815cb10 con 0x7fe9080d0a20 2024-05-28T06:20:01.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.423+0000 7fe910600640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fe9080adbb0 con 0x7fe9080d0640 2024-05-28T06:20:01.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.424+0000 7fe90e600640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fe9080b32c0 0x7fe9080b3730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:55702/0 (socket says 172.21.15.96:55702) 2024-05-28T06:20:01.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.424+0000 7fe90e600640 1 -- 172.21.15.96:0/488457064 learned_addr learned my addr 172.21.15.96:0/488457064 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:20:01.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.424+0000 7fe90dc00640 1 --2- 172.21.15.96:0/488457064 >> v2:172.21.15.123:3300/0 conn(0x7fe9080d0640 0x7fe90815d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:01.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.424+0000 7fe907e00640 1 -- 172.21.15.96:0/488457064 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3486568885 0 0) 0x7fe9080d0060 con 0x7fe9080b32c0 2024-05-28T06:20:01.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.425+0000 7fe907e00640 1 -- 172.21.15.96:0/488457064 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe8f0003520 con 0x7fe9080b32c0 2024-05-28T06:20:01.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.425+0000 7fe907e00640 1 -- 172.21.15.96:0/488457064 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1737895624 0 0) 0x7fe8f0003520 con 0x7fe9080b32c0 2024-05-28T06:20:01.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.425+0000 7fe907e00640 1 -- 172.21.15.96:0/488457064 >> v2:172.21.15.123:3300/0 conn(0x7fe9080d0640 msgr2=0x7fe90815d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:01.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.425+0000 7fe907e00640 1 --2- 172.21.15.96:0/488457064 >> v2:172.21.15.123:3300/0 conn(0x7fe9080d0640 0x7fe90815d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:01.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.425+0000 7fe907e00640 1 -- 172.21.15.96:0/488457064 >> v1:172.21.15.123:6789/0 conn(0x7fe9080d0a20 legacy=0x7fe9080cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:01.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.425+0000 7fe907e00640 1 -- 172.21.15.96:0/488457064 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9080ad5f0 con 0x7fe9080b32c0 2024-05-28T06:20:01.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.425+0000 7fe907e00640 1 -- 172.21.15.96:0/488457064 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7fe9000019f0 con 0x7fe9080b32c0 2024-05-28T06:20:01.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.425+0000 7fe907e00640 1 -- 172.21.15.96:0/488457064 >> v1:172.21.15.96:6789/0 conn(0x7fe9080b32c0 legacy=0x7fe9080b3730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:01.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.425+0000 7fe907e00640 1 --2- 172.21.15.96:0/488457064 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe8f00039b0 0x7fe8f0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:01.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.425+0000 7fe907e00640 1 --2- 172.21.15.96:0/488457064 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe8f0004800 0x7fe8f0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:01.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.425+0000 7fe907e00640 1 --2- 172.21.15.96:0/488457064 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe8f00051c0 0x7fe8f000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:01.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.425+0000 7fe907e00640 1 -- 172.21.15.96:0/488457064 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe8f000c130 con 0x7fe8f00051c0 2024-05-28T06:20:01.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.425+0000 7fe907e00640 1 -- 172.21.15.96:0/488457064 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe8f000c2e0 con 0x7fe8f0004800 2024-05-28T06:20:01.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.425+0000 7fe907e00640 1 -- 172.21.15.96:0/488457064 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fe8f000c4e0 con 0x7fe8f00039b0 2024-05-28T06:20:01.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.425+0000 7fe90e600640 1 --2- 172.21.15.96:0/488457064 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe8f00051c0 0x7fe8f000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:01.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.426+0000 7fe90dc00640 1 --2- 172.21.15.96:0/488457064 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe8f00039b0 0x7fe8f0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:01.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.426+0000 7fe90d200640 1 --2- 172.21.15.96:0/488457064 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe8f0004800 0x7fe8f0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:01.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.426+0000 7fe90e600640 1 -- 172.21.15.96:0/488457064 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe8f00039b0 msgr2=0x7fe8f0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:01.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.426+0000 7fe90e600640 1 --2- 172.21.15.96:0/488457064 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe8f00039b0 0x7fe8f0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:01.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.426+0000 7fe90e600640 1 -- 172.21.15.96:0/488457064 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe8f0004800 msgr2=0x7fe8f0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:01.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.426+0000 7fe90e600640 1 --2- 172.21.15.96:0/488457064 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe8f0004800 0x7fe8f0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:01.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.426+0000 7fe90e600640 1 -- 172.21.15.96:0/488457064 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8f0003520 con 0x7fe8f00051c0 2024-05-28T06:20:01.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.426+0000 7fe90d200640 1 --2- 172.21.15.96:0/488457064 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe8f0004800 0x7fe8f0004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:20:01.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.427+0000 7fe90e600640 1 -- 172.21.15.96:0/488457064 reap_dead start 2024-05-28T06:20:01.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.427+0000 7fe90e600640 1 --2- 172.21.15.96:0/488457064 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe8f00051c0 0x7fe8f000b9c0 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7fe90000a860 tx=0x7fe8f0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=756351aec183ea3e server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:01.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.428+0000 7fe907e00640 1 -- 172.21.15.96:0/488457064 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe90000a480 con 0x7fe8f00051c0 2024-05-28T06:20:01.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.428+0000 7fe907e00640 1 -- 172.21.15.96:0/488457064 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe90002f070 con 0x7fe8f00051c0 2024-05-28T06:20:01.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.428+0000 7fe907e00640 1 -- 172.21.15.96:0/488457064 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe90002bca0 con 0x7fe8f00051c0 2024-05-28T06:20:01.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.428+0000 7fe910600640 1 -- 172.21.15.96:0/488457064 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe8f00051c0 msgr2=0x7fe8f000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:01.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.428+0000 7fe910600640 1 --2- 172.21.15.96:0/488457064 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe8f00051c0 0x7fe8f000b9c0 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7fe90000a860 tx=0x7fe8f0005830 comp rx=0 tx=0).stop 2024-05-28T06:20:01.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.429+0000 7fe910600640 1 -- 172.21.15.96:0/488457064 shutdown_connections 2024-05-28T06:20:01.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.429+0000 7fe910600640 1 --2- 172.21.15.96:0/488457064 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe8f00051c0 0x7fe8f000b9c0 unknown :-1 s=CLOSED pgs=1536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:01.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.429+0000 7fe910600640 1 -- 172.21.15.96:0/488457064 >> 172.21.15.96:0/488457064 conn(0x7fe90806bd20 msgr2=0x7fe90806c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:01.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.429+0000 7fe910600640 1 -- 172.21.15.96:0/488457064 shutdown_connections 2024-05-28T06:20:01.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.429+0000 7fe910600640 1 -- 172.21.15.96:0/488457064 wait complete. 2024-05-28T06:20:01.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.431+0000 7fe910600640 1 Processor -- start 2024-05-28T06:20:01.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.431+0000 7fe910600640 1 -- start start 2024-05-28T06:20:01.430 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.432+0000 7fe910600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe8f00051c0 0x7fe90814aed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:01.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.432+0000 7fe910600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe908148e70 0x7fe9081471f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:01.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.432+0000 7fe910600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe9081f8470 0x7fe90815b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:01.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.432+0000 7fe910600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe90815d0b0 con 0x7fe9081f8470 2024-05-28T06:20:01.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.432+0000 7fe910600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe90815cdd0 con 0x7fe908148e70 2024-05-28T06:20:01.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.432+0000 7fe910600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fe9080ad2a0 con 0x7fe8f00051c0 2024-05-28T06:20:01.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.433+0000 7fe90e600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe9081f8470 0x7fe90815b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:01.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.433+0000 7fe90e600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe9081f8470 0x7fe90815b240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:55798/0 (socket says 172.21.15.96:55798) 2024-05-28T06:20:01.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.433+0000 7fe90e600640 1 -- 172.21.15.96:0/3626086344 learned_addr learned my addr 172.21.15.96:0/3626086344 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:20:01.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.433+0000 7fe90dc00640 1 --2- 172.21.15.96:0/3626086344 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe8f00051c0 0x7fe90814aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:01.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.433+0000 7fe90d200640 1 --2- 172.21.15.96:0/3626086344 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe908148e70 0x7fe9081471f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:01.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.433+0000 7fe90e600640 1 -- 172.21.15.96:0/3626086344 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe8f00051c0 msgr2=0x7fe90814aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:01.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.433+0000 7fe90e600640 1 --2- 172.21.15.96:0/3626086344 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe8f00051c0 0x7fe90814aed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:01.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.433+0000 7fe90e600640 1 -- 172.21.15.96:0/3626086344 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe908148e70 msgr2=0x7fe9081471f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:01.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.433+0000 7fe90e600640 1 --2- 172.21.15.96:0/3626086344 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe908148e70 0x7fe9081471f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:01.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.433+0000 7fe90e600640 1 -- 172.21.15.96:0/3626086344 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe908149300 con 0x7fe9081f8470 2024-05-28T06:20:01.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.434+0000 7fe90e600640 1 --2- 172.21.15.96:0/3626086344 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe9081f8470 0x7fe90815b240 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7fe90002b530 tx=0x7fe90002d070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:01.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.434+0000 7fe905600640 1 -- 172.21.15.96:0/3626086344 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe90002bc90 con 0x7fe9081f8470 2024-05-28T06:20:01.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.434+0000 7fe905600640 1 -- 172.21.15.96:0/3626086344 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe90002be30 con 0x7fe9081f8470 2024-05-28T06:20:01.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.434+0000 7fe905600640 1 -- 172.21.15.96:0/3626086344 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe900034670 con 0x7fe9081f8470 2024-05-28T06:20:01.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.434+0000 7fe910600640 1 -- 172.21.15.96:0/3626086344 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9081495e0 con 0x7fe9081f8470 2024-05-28T06:20:01.433 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.434+0000 7fe910600640 1 -- 172.21.15.96:0/3626086344 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9081f7c70 con 0x7fe9081f8470 2024-05-28T06:20:01.433 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.435+0000 7fe910600640 1 -- 172.21.15.96:0/3626086344 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9080ae5a0 con 0x7fe9081f8470 2024-05-28T06:20:01.433 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.436+0000 7fe905600640 1 -- 172.21.15.96:0/3626086344 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 34) ==== 92927+0+0 (secure 0 0 0) 0x7fe900034810 con 0x7fe9081f8470 2024-05-28T06:20:01.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.438+0000 7fe905600640 1 --2- 172.21.15.96:0/3626086344 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe8e806ddc0 0x7fe8e8070280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:01.436 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.439+0000 7fe905600640 1 -- 172.21.15.96:0/3626086344 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 5822+0+0 (secure 0 0 0) 0x7fe9000aa670 con 0x7fe9081f8470 2024-05-28T06:20:01.436 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.439+0000 7fe90dc00640 1 --2- 172.21.15.96:0/3626086344 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe8e806ddc0 0x7fe8e8070280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:01.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.440+0000 7fe90dc00640 1 --2- 172.21.15.96:0/3626086344 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe8e806ddc0 0x7fe8e8070280 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc009a80 tx=0x7fe8fc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:01.439 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.442+0000 7fe905600640 1 -- 172.21.15.96:0/3626086344 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe9080ae5a0 con 0x7fe9081f8470 2024-05-28T06:20:01.638 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.641+0000 7fe910600640 1 -- 172.21.15.96:0/3626086344 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe9080689e0 con 0x7fe9081f8470 2024-05-28T06:20:01.638 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.641+0000 7fe905600640 1 -- 172.21.15.96:0/3626086344 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v92) ==== 93+0+10 (secure 0 0 0) 0x7fe900046060 con 0x7fe9081f8470 2024-05-28T06:20:01.638 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 1 2024-05-28T06:20:01.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.644+0000 7fe910600640 1 -- 172.21.15.96:0/3626086344 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe8e806ddc0 msgr2=0x7fe8e8070280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:01.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.644+0000 7fe910600640 1 --2- 172.21.15.96:0/3626086344 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe8e806ddc0 0x7fe8e8070280 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc009a80 tx=0x7fe8fc008040 comp rx=0 tx=0).stop 2024-05-28T06:20:01.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.644+0000 7fe910600640 1 -- 172.21.15.96:0/3626086344 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe9081f8470 msgr2=0x7fe90815b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:01.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.644+0000 7fe910600640 1 --2- 172.21.15.96:0/3626086344 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe9081f8470 0x7fe90815b240 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7fe90002b530 tx=0x7fe90002d070 comp rx=0 tx=0).stop 2024-05-28T06:20:01.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.644+0000 7fe910600640 1 -- 172.21.15.96:0/3626086344 shutdown_connections 2024-05-28T06:20:01.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.644+0000 7fe910600640 1 --2- 172.21.15.96:0/3626086344 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe8e806ddc0 0x7fe8e8070280 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:01.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.644+0000 7fe910600640 1 --2- 172.21.15.96:0/3626086344 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe9081f8470 0x7fe90815b240 unknown :-1 s=CLOSED pgs=1537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:01.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.644+0000 7fe910600640 1 --2- 172.21.15.96:0/3626086344 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe908148e70 0x7fe9081471f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:01.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.644+0000 7fe910600640 1 --2- 172.21.15.96:0/3626086344 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe8f00051c0 0x7fe90814aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:01.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.644+0000 7fe910600640 1 -- 172.21.15.96:0/3626086344 >> 172.21.15.96:0/3626086344 conn(0x7fe90806bd20 msgr2=0x7fe90815e1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:01.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.645+0000 7fe910600640 1 -- 172.21.15.96:0/3626086344 shutdown_connections 2024-05-28T06:20:01.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.645+0000 7fe910600640 1 -- 172.21.15.96:0/3626086344 wait complete. 2024-05-28T06:20:01.662 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-28T06:20:01.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.867+0000 7f69b0c00640 1 Processor -- start 2024-05-28T06:20:01.865 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.867+0000 7f69b0c00640 1 -- start start 2024-05-28T06:20:01.866 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.869+0000 7f69b0c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f69ac0ce1f0 0x7f69ac0b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:01.866 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.869+0000 7f69b0c00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f69ac0ce5d0 0x7f69ac15ec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:01.866 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.869+0000 7f69b0c00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69ac0b3580 con 0x7f69ac0b3d60 2024-05-28T06:20:01.866 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.869+0000 7f69b0c00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f69ac15f490 con 0x7f69ac0ce5d0 2024-05-28T06:20:01.867 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.869+0000 7f69b0c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f69ac15f770 con 0x7f69ac0ce1f0 2024-05-28T06:20:01.867 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.869+0000 7f69abe00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f69ac0b3d60 0x7f69ac0ccb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:55704/0 (socket says 172.21.15.96:55704) 2024-05-28T06:20:01.867 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.869+0000 7f69abe00640 1 -- 172.21.15.96:0/2242099532 learned_addr learned my addr 172.21.15.96:0/2242099532 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:20:01.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.870+0000 7f69ab400640 1 --2- 172.21.15.96:0/2242099532 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f69ac0ce1f0 0x7f69ac0b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:01.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.870+0000 7f69aaa00640 1 --2- 172.21.15.96:0/2242099532 >> v2:172.21.15.123:3300/0 conn(0x7f69ac0ce5d0 0x7f69ac15ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:01.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.870+0000 7f69aa000640 1 -- 172.21.15.96:0/2242099532 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3971406476 0 0) 0x7f69ac0b3580 con 0x7f69ac0b3d60 2024-05-28T06:20:01.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.870+0000 7f69aa000640 1 -- 172.21.15.96:0/2242099532 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6990003520 con 0x7f69ac0b3d60 2024-05-28T06:20:01.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.870+0000 7f69aa000640 1 -- 172.21.15.96:0/2242099532 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2738876768 0 0) 0x7f6990003520 con 0x7f69ac0b3d60 2024-05-28T06:20:01.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.870+0000 7f69aa000640 1 -- 172.21.15.96:0/2242099532 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f69ac0ce1f0 msgr2=0x7f69ac0b4dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:01.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.870+0000 7f69aa000640 1 --2- 172.21.15.96:0/2242099532 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f69ac0ce1f0 0x7f69ac0b4dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:01.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.870+0000 7f69aa000640 1 -- 172.21.15.96:0/2242099532 >> v2:172.21.15.123:3300/0 conn(0x7f69ac0ce5d0 msgr2=0x7f69ac15ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:01.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.870+0000 7f69aa000640 1 --2- 172.21.15.96:0/2242099532 >> v2:172.21.15.123:3300/0 conn(0x7f69ac0ce5d0 0x7f69ac15ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:01.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.870+0000 7f69aa000640 1 -- 172.21.15.96:0/2242099532 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69ac15fa50 con 0x7f69ac0b3d60 2024-05-28T06:20:01.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.870+0000 7f69aa000640 1 -- 172.21.15.96:0/2242099532 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f69a4001670 con 0x7f69ac0b3d60 2024-05-28T06:20:01.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.870+0000 7f69aa000640 1 -- 172.21.15.96:0/2242099532 >> v1:172.21.15.96:6789/0 conn(0x7f69ac0b3d60 legacy=0x7f69ac0ccb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:01.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.870+0000 7f69aa000640 1 --2- 172.21.15.96:0/2242099532 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f69ac0ce1f0 0x7f69900039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:01.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.870+0000 7f69aa000640 1 --2- 172.21.15.96:0/2242099532 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6990003f30 0x7f6990004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:01.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.870+0000 7f69aa000640 1 --2- 172.21.15.96:0/2242099532 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6990004d40 0x7f69900051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:01.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.870+0000 7f69aa000640 1 -- 172.21.15.96:0/2242099532 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f69ac15f490 con 0x7f6990003f30 2024-05-28T06:20:01.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.870+0000 7f69aa000640 1 -- 172.21.15.96:0/2242099532 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f69ac15f770 con 0x7f6990004d40 2024-05-28T06:20:01.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.870+0000 7f69aa000640 1 -- 172.21.15.96:0/2242099532 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f699000baa0 con 0x7f69ac0ce1f0 2024-05-28T06:20:01.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.871+0000 7f69aaa00640 1 --2- 172.21.15.96:0/2242099532 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6990003f30 0x7f6990004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:01.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.871+0000 7f69ab400640 1 --2- 172.21.15.96:0/2242099532 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f69ac0ce1f0 0x7f69900039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:01.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.871+0000 7f69aaa00640 1 -- 172.21.15.96:0/2242099532 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f69ac0ce1f0 msgr2=0x7f69900039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:01.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.871+0000 7f69aaa00640 1 --2- 172.21.15.96:0/2242099532 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f69ac0ce1f0 0x7f69900039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:01.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.871+0000 7f69abe00640 1 --2- 172.21.15.96:0/2242099532 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6990004d40 0x7f69900051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:01.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.871+0000 7f69aaa00640 1 -- 172.21.15.96:0/2242099532 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6990004d40 msgr2=0x7f69900051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:01.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.871+0000 7f69aaa00640 1 --2- 172.21.15.96:0/2242099532 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6990004d40 0x7f69900051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:01.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.871+0000 7f69aaa00640 1 -- 172.21.15.96:0/2242099532 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6990003520 con 0x7f6990003f30 2024-05-28T06:20:01.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.871+0000 7f69abe00640 1 --2- 172.21.15.96:0/2242099532 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6990004d40 0x7f69900051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:20:01.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.871+0000 7f69ab400640 1 --2- 172.21.15.96:0/2242099532 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f69ac0ce1f0 0x7f69900039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:20:01.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.872+0000 7f69aaa00640 1 --2- 172.21.15.96:0/2242099532 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6990003f30 0x7f6990004800 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7f6998009fc0 tx=0x7f6990005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4450348aca135c34 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:01.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.873+0000 7f69aa000640 1 -- 172.21.15.96:0/2242099532 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6998009c10 con 0x7f6990003f30 2024-05-28T06:20:01.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.873+0000 7f69aa000640 1 -- 172.21.15.96:0/2242099532 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6998036070 con 0x7f6990003f30 2024-05-28T06:20:01.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.873+0000 7f69aa000640 1 -- 172.21.15.96:0/2242099532 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f699802d070 con 0x7f6990003f30 2024-05-28T06:20:01.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.873+0000 7f69b0c00640 1 -- 172.21.15.96:0/2242099532 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6990003f30 msgr2=0x7f6990004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:01.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.873+0000 7f69b0c00640 1 --2- 172.21.15.96:0/2242099532 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6990003f30 0x7f6990004800 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7f6998009fc0 tx=0x7f6990005830 comp rx=0 tx=0).stop 2024-05-28T06:20:01.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.873+0000 7f69abe00640 1 -- 172.21.15.96:0/2242099532 reap_dead start 2024-05-28T06:20:01.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.873+0000 7f69b0c00640 1 -- 172.21.15.96:0/2242099532 shutdown_connections 2024-05-28T06:20:01.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.873+0000 7f69b0c00640 1 -- 172.21.15.96:0/2242099532 >> 172.21.15.96:0/2242099532 conn(0x7f69ac06b900 msgr2=0x7f69ac06bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:01.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.873+0000 7f69b0c00640 1 -- 172.21.15.96:0/2242099532 shutdown_connections 2024-05-28T06:20:01.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.873+0000 7f69b0c00640 1 -- 172.21.15.96:0/2242099532 wait complete. 2024-05-28T06:20:01.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.875+0000 7f69b0c00640 1 Processor -- start 2024-05-28T06:20:01.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.875+0000 7f69b0c00640 1 -- start start 2024-05-28T06:20:01.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.876+0000 7f69b0c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6990003f30 0x7f69ac157e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:01.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.876+0000 7f69b0c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69ac0b2890 0x7f69ac160050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:01.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.876+0000 7f69b0c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f69ac157330 0x7f69ac1577b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:01.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.876+0000 7f69b0c00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f69ac0cd2a0 con 0x7f6990003f30 2024-05-28T06:20:01.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.876+0000 7f69b0c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f69ac15f180 con 0x7f69ac0b2890 2024-05-28T06:20:01.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.876+0000 7f69b0c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f69ac0cd420 con 0x7f69ac157330 2024-05-28T06:20:01.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.876+0000 7f69ab400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6990003f30 0x7f69ac157e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:01.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.876+0000 7f69ab400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6990003f30 0x7f69ac157e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:55822/0 (socket says 172.21.15.96:55822) 2024-05-28T06:20:01.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.876+0000 7f69ab400640 1 -- 172.21.15.96:0/1291116623 learned_addr learned my addr 172.21.15.96:0/1291116623 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:20:01.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.877+0000 7f69abe00640 1 --2- 172.21.15.96:0/1291116623 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f69ac157330 0x7f69ac1577b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:01.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.877+0000 7f69aaa00640 1 --2- 172.21.15.96:0/1291116623 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69ac0b2890 0x7f69ac160050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:01.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.877+0000 7f69ab400640 1 -- 172.21.15.96:0/1291116623 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f69ac157330 msgr2=0x7f69ac1577b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:01.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.877+0000 7f69ab400640 1 --2- 172.21.15.96:0/1291116623 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f69ac157330 0x7f69ac1577b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:01.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.877+0000 7f69ab400640 1 -- 172.21.15.96:0/1291116623 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69ac0b2890 msgr2=0x7f69ac160050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:01.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.877+0000 7f69ab400640 1 --2- 172.21.15.96:0/1291116623 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69ac0b2890 0x7f69ac160050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:01.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.877+0000 7f69ab400640 1 -- 172.21.15.96:0/1291116623 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69ac0b2d10 con 0x7f6990003f30 2024-05-28T06:20:01.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.877+0000 7f69aaa00640 1 --2- 172.21.15.96:0/1291116623 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69ac0b2890 0x7f69ac160050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:20:01.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.877+0000 7f69abe00640 1 --2- 172.21.15.96:0/1291116623 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f69ac157330 0x7f69ac1577b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:20:01.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.877+0000 7f69ab400640 1 --2- 172.21.15.96:0/1291116623 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6990003f30 0x7f69ac157e80 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7f699c0108b0 tx=0x7f699c010d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:01.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.877+0000 7f69a3400640 1 -- 172.21.15.96:0/1291116623 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f699c00f040 con 0x7f6990003f30 2024-05-28T06:20:01.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.878+0000 7f69a3400640 1 -- 172.21.15.96:0/1291116623 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f699c00eb60 con 0x7f6990003f30 2024-05-28T06:20:01.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.878+0000 7f69a3400640 1 -- 172.21.15.96:0/1291116623 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f699c01c400 con 0x7f6990003f30 2024-05-28T06:20:01.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.878+0000 7f69b0c00640 1 -- 172.21.15.96:0/1291116623 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69ac0b1170 con 0x7f6990003f30 2024-05-28T06:20:01.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.878+0000 7f69b0c00640 1 -- 172.21.15.96:0/1291116623 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69ac0b16f0 con 0x7f6990003f30 2024-05-28T06:20:01.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.879+0000 7f69b0c00640 1 -- 172.21.15.96:0/1291116623 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69ac0b1db0 con 0x7f6990003f30 2024-05-28T06:20:01.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.880+0000 7f69a3400640 1 -- 172.21.15.96:0/1291116623 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 34) ==== 92927+0+0 (secure 0 0 0) 0x7f699c01c5a0 con 0x7f6990003f30 2024-05-28T06:20:01.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.881+0000 7f69a3400640 1 --2- 172.21.15.96:0/1291116623 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f698c06ddc0 0x7f698c070280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:01.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.881+0000 7f69a3400640 1 -- 172.21.15.96:0/1291116623 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 5822+0+0 (secure 0 0 0) 0x7f699c08b250 con 0x7f6990003f30 2024-05-28T06:20:01.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.882+0000 7f69aaa00640 1 --2- 172.21.15.96:0/1291116623 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f698c06ddc0 0x7f698c070280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:01.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.883+0000 7f69aaa00640 1 --2- 172.21.15.96:0/1291116623 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f698c06ddc0 0x7f698c070280 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f6998035000 tx=0x7f699802f000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:01.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:01.886+0000 7f69a3400640 1 -- 172.21.15.96:0/1291116623 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f69ac0b2d10 con 0x7f6990003f30 2024-05-28T06:20:02.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.086+0000 7f69b0c00640 1 -- 172.21.15.96:0/1291116623 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f69ac0689e0 con 0x7f6990003f30 2024-05-28T06:20:02.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.086+0000 7f69a3400640 1 -- 172.21.15.96:0/1291116623 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v92) ==== 104+0+11 (secure 0 0 0) 0x7f699c018030 con 0x7f6990003f30 2024-05-28T06:20:02.084 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 32 2024-05-28T06:20:02.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.089+0000 7f69b0c00640 1 -- 172.21.15.96:0/1291116623 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f698c06ddc0 msgr2=0x7f698c070280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:02.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.089+0000 7f69b0c00640 1 --2- 172.21.15.96:0/1291116623 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f698c06ddc0 0x7f698c070280 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f6998035000 tx=0x7f699802f000 comp rx=0 tx=0).stop 2024-05-28T06:20:02.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.089+0000 7f69b0c00640 1 -- 172.21.15.96:0/1291116623 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6990003f30 msgr2=0x7f69ac157e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:02.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.089+0000 7f69b0c00640 1 --2- 172.21.15.96:0/1291116623 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6990003f30 0x7f69ac157e80 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7f699c0108b0 tx=0x7f699c010d70 comp rx=0 tx=0).stop 2024-05-28T06:20:02.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.089+0000 7f69b0c00640 1 -- 172.21.15.96:0/1291116623 shutdown_connections 2024-05-28T06:20:02.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.089+0000 7f69b0c00640 1 --2- 172.21.15.96:0/1291116623 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f698c06ddc0 0x7f698c070280 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:02.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.089+0000 7f69b0c00640 1 --2- 172.21.15.96:0/1291116623 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f69ac157330 0x7f69ac1577b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:02.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.089+0000 7f69b0c00640 1 --2- 172.21.15.96:0/1291116623 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69ac0b2890 0x7f69ac160050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:02.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.089+0000 7f69b0c00640 1 --2- 172.21.15.96:0/1291116623 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6990003f30 0x7f69ac157e80 unknown :-1 s=CLOSED pgs=1540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:02.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.089+0000 7f69b0c00640 1 -- 172.21.15.96:0/1291116623 >> 172.21.15.96:0/1291116623 conn(0x7f69ac06b900 msgr2=0x7f69ac06f3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:02.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.089+0000 7f69b0c00640 1 -- 172.21.15.96:0/1291116623 shutdown_connections 2024-05-28T06:20:02.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.089+0000 7f69b0c00640 1 -- 172.21.15.96:0/1291116623 wait complete. 2024-05-28T06:20:02.106 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-28T06:20:02.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.313+0000 7f8402600640 1 Processor -- start 2024-05-28T06:20:02.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.313+0000 7f8402600640 1 -- start start 2024-05-28T06:20:02.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.314+0000 7f8402600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f83fc0cae20 0x7f83fc151db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:02.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.314+0000 7f8402600640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83fc152780 con 0x7f83fc0caa40 2024-05-28T06:20:02.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.314+0000 7f8402600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83fc0b3e80 con 0x7f83fc1522f0 2024-05-28T06:20:02.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.314+0000 7f8402600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f83fc0d0b30 con 0x7f83fc0cae20 2024-05-28T06:20:02.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.315+0000 7f8401200640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f83fc0caa40 0x7f83fc0ce960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:55714/0 (socket says 172.21.15.96:55714) 2024-05-28T06:20:02.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.315+0000 7f8401200640 1 -- 172.21.15.96:0/1161134724 learned_addr learned my addr 172.21.15.96:0/1161134724 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:20:02.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.315+0000 7f83fbe00640 1 --2- 172.21.15.96:0/1161134724 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f83fc0cae20 0x7f83fc151db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:02.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.315+0000 7f83fb400640 1 -- 172.21.15.96:0/1161134724 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1617059504 0 0) 0x7f83fc152780 con 0x7f83fc0caa40 2024-05-28T06:20:02.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.315+0000 7f83fb400640 1 -- 172.21.15.96:0/1161134724 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f83e0003520 con 0x7f83fc0caa40 2024-05-28T06:20:02.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.316+0000 7f83fb400640 1 -- 172.21.15.96:0/1161134724 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2956832927 0 0) 0x7f83e0003520 con 0x7f83fc0caa40 2024-05-28T06:20:02.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.316+0000 7f83fb400640 1 -- 172.21.15.96:0/1161134724 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f83fc0cae20 msgr2=0x7f83fc151db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:02.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.316+0000 7f83fb400640 1 --2- 172.21.15.96:0/1161134724 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f83fc0cae20 0x7f83fc151db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:02.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.316+0000 7f83fb400640 1 -- 172.21.15.96:0/1161134724 >> v1:172.21.15.123:6789/0 conn(0x7f83fc1522f0 legacy=0x7f83fc0cf710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:02.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.316+0000 7f83fb400640 1 -- 172.21.15.96:0/1161134724 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83fc0cde80 con 0x7f83fc0caa40 2024-05-28T06:20:02.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.316+0000 7f83fb400640 1 -- 172.21.15.96:0/1161134724 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f83ec0019f0 con 0x7f83fc0caa40 2024-05-28T06:20:02.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.316+0000 7f83fb400640 1 -- 172.21.15.96:0/1161134724 >> v1:172.21.15.96:6789/0 conn(0x7f83fc0caa40 legacy=0x7f83fc0ce960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:02.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.316+0000 7f83fb400640 1 --2- 172.21.15.96:0/1161134724 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f83e00039e0 0x7f83e0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:02.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.316+0000 7f83fb400640 1 --2- 172.21.15.96:0/1161134724 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83e0004800 0x7f83e0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:02.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.316+0000 7f83fb400640 1 --2- 172.21.15.96:0/1161134724 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f83fc0cae20 0x7f83e0005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:02.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.316+0000 7f83fb400640 1 -- 172.21.15.96:0/1161134724 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f83fc0d0b30 con 0x7f83e0004800 2024-05-28T06:20:02.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.316+0000 7f83fb400640 1 -- 172.21.15.96:0/1161134724 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f83e000baf0 con 0x7f83e00039e0 2024-05-28T06:20:02.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.316+0000 7f83fb400640 1 -- 172.21.15.96:0/1161134724 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f83e000bcf0 con 0x7f83fc0cae20 2024-05-28T06:20:02.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.316+0000 7f83fbe00640 1 --2- 172.21.15.96:0/1161134724 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83e0004800 0x7f83e0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:02.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.317+0000 7f8401200640 1 --2- 172.21.15.96:0/1161134724 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f83e00039e0 0x7f83e0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:02.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.317+0000 7f8401c00640 1 --2- 172.21.15.96:0/1161134724 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f83fc0cae20 0x7f83e0005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:02.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.317+0000 7f83fbe00640 1 -- 172.21.15.96:0/1161134724 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f83fc0cae20 msgr2=0x7f83e0005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:02.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.317+0000 7f83fbe00640 1 --2- 172.21.15.96:0/1161134724 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f83fc0cae20 0x7f83e0005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:02.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.317+0000 7f83fbe00640 1 -- 172.21.15.96:0/1161134724 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f83e00039e0 msgr2=0x7f83e0003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:02.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.317+0000 7f83fbe00640 1 --2- 172.21.15.96:0/1161134724 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f83e00039e0 0x7f83e0003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:02.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.317+0000 7f83fbe00640 1 -- 172.21.15.96:0/1161134724 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83e0003520 con 0x7f83e0004800 2024-05-28T06:20:02.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.317+0000 7f8401c00640 1 --2- 172.21.15.96:0/1161134724 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f83fc0cae20 0x7f83e0005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:20:02.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.318+0000 7f83fbe00640 1 --2- 172.21.15.96:0/1161134724 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83e0004800 0x7f83e0004c80 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7f83e4009ac0 tx=0x7f83e0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=815b6608a9a42830 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:02.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.318+0000 7f83fb400640 1 -- 172.21.15.96:0/1161134724 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83e4036070 con 0x7f83e0004800 2024-05-28T06:20:02.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.318+0000 7f83fb400640 1 -- 172.21.15.96:0/1161134724 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83e4031400 con 0x7f83e0004800 2024-05-28T06:20:02.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.318+0000 7f83fb400640 1 -- 172.21.15.96:0/1161134724 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83e4009690 con 0x7f83e0004800 2024-05-28T06:20:02.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.319+0000 7f8402600640 1 -- 172.21.15.96:0/1161134724 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83e0004800 msgr2=0x7f83e0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:02.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.319+0000 7f8402600640 1 --2- 172.21.15.96:0/1161134724 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83e0004800 0x7f83e0004c80 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7f83e4009ac0 tx=0x7f83e0005830 comp rx=0 tx=0).stop 2024-05-28T06:20:02.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.319+0000 7f8401c00640 1 -- 172.21.15.96:0/1161134724 reap_dead start 2024-05-28T06:20:02.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.319+0000 7f8402600640 1 -- 172.21.15.96:0/1161134724 shutdown_connections 2024-05-28T06:20:02.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.319+0000 7f8402600640 1 -- 172.21.15.96:0/1161134724 >> 172.21.15.96:0/1161134724 conn(0x7f83fc06b900 msgr2=0x7f83fc06bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:02.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.319+0000 7f8402600640 1 -- 172.21.15.96:0/1161134724 shutdown_connections 2024-05-28T06:20:02.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.319+0000 7f8402600640 1 -- 172.21.15.96:0/1161134724 wait complete. 2024-05-28T06:20:02.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.321+0000 7f8402600640 1 Processor -- start 2024-05-28T06:20:02.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.321+0000 7f8402600640 1 -- start start 2024-05-28T06:20:02.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.322+0000 7f8402600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f83e0004800 0x7f83fc1f2210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:02.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.322+0000 7f8402600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f83fc151240 0x7f83fc0b0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:02.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.322+0000 7f8402600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83fc0b0fe0 0x7f83fc154450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:02.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.322+0000 7f8402600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f83fc0cf070 con 0x7f83fc0b0fe0 2024-05-28T06:20:02.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.322+0000 7f8402600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f83fc0b4160 con 0x7f83fc151240 2024-05-28T06:20:02.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.322+0000 7f8402600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f83fc0ce440 con 0x7f83e0004800 2024-05-28T06:20:02.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.322+0000 7f8401c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83fc0b0fe0 0x7f83fc154450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:02.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.322+0000 7f8401c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83fc0b0fe0 0x7f83fc154450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:55838/0 (socket says 172.21.15.96:55838) 2024-05-28T06:20:02.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.322+0000 7f8401c00640 1 -- 172.21.15.96:0/1993299563 learned_addr learned my addr 172.21.15.96:0/1993299563 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:20:02.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.322+0000 7f83fbe00640 1 --2- 172.21.15.96:0/1993299563 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f83fc151240 0x7f83fc0b0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:02.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.322+0000 7f8401200640 1 --2- 172.21.15.96:0/1993299563 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f83e0004800 0x7f83fc1f2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:02.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.322+0000 7f8401c00640 1 -- 172.21.15.96:0/1993299563 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f83e0004800 msgr2=0x7f83fc1f2210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:02.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.323+0000 7f8401c00640 1 --2- 172.21.15.96:0/1993299563 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f83e0004800 0x7f83fc1f2210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:02.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.323+0000 7f8401c00640 1 -- 172.21.15.96:0/1993299563 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f83fc151240 msgr2=0x7f83fc0b0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:02.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.323+0000 7f8401c00640 1 --2- 172.21.15.96:0/1993299563 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f83fc151240 0x7f83fc0b0aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:02.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.323+0000 7f8401c00640 1 -- 172.21.15.96:0/1993299563 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83fc1560d0 con 0x7f83fc0b0fe0 2024-05-28T06:20:02.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.323+0000 7f83fbe00640 1 --2- 172.21.15.96:0/1993299563 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f83fc151240 0x7f83fc0b0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:20:02.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.323+0000 7f8401c00640 1 --2- 172.21.15.96:0/1993299563 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83fc0b0fe0 0x7f83fc154450 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7f83fc06c8b0 tx=0x7f83f400d890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:02.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.323+0000 7f83f8c00640 1 -- 172.21.15.96:0/1993299563 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83f4007030 con 0x7f83fc0b0fe0 2024-05-28T06:20:02.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.323+0000 7f83f8c00640 1 -- 172.21.15.96:0/1993299563 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83f40071f0 con 0x7f83fc0b0fe0 2024-05-28T06:20:02.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.323+0000 7f8402600640 1 -- 172.21.15.96:0/1993299563 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83fc1563b0 con 0x7f83fc0b0fe0 2024-05-28T06:20:02.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.323+0000 7f83f8c00640 1 -- 172.21.15.96:0/1993299563 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83f40075e0 con 0x7f83fc0b0fe0 2024-05-28T06:20:02.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.323+0000 7f8402600640 1 -- 172.21.15.96:0/1993299563 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83fc156970 con 0x7f83fc0b0fe0 2024-05-28T06:20:02.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.324+0000 7f8402600640 1 -- 172.21.15.96:0/1993299563 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83fc156d50 con 0x7f83fc0b0fe0 2024-05-28T06:20:02.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.325+0000 7f83f8c00640 1 -- 172.21.15.96:0/1993299563 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 34) ==== 92927+0+0 (secure 0 0 0) 0x7f83f401b020 con 0x7f83fc0b0fe0 2024-05-28T06:20:02.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.326+0000 7f83f8c00640 1 --2- 172.21.15.96:0/1993299563 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f83d806ddf0 0x7f83d80702b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:02.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.326+0000 7f83f8c00640 1 -- 172.21.15.96:0/1993299563 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 5822+0+0 (secure 0 0 0) 0x7f83f408a2c0 con 0x7f83fc0b0fe0 2024-05-28T06:20:02.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.327+0000 7f8401200640 1 --2- 172.21.15.96:0/1993299563 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f83d806ddf0 0x7f83d80702b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:02.325 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.328+0000 7f8401200640 1 --2- 172.21.15.96:0/1993299563 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f83d806ddf0 0x7f83d80702b0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f83ec002230 tx=0x7f83ec001360 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:02.327 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.331+0000 7f83f8c00640 1 -- 172.21.15.96:0/1993299563 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f83f405a140 con 0x7f83fc0b0fe0 2024-05-28T06:20:02.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.530+0000 7f8402600640 1 -- 172.21.15.96:0/1993299563 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f83fc0689e0 con 0x7f83fc0b0fe0 2024-05-28T06:20:02.528 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.531+0000 7f83f8c00640 1 -- 172.21.15.96:0/1993299563 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v92) ==== 100+0+11 (secure 0 0 0) 0x7f83f4059a40 con 0x7f83fc0b0fe0 2024-05-28T06:20:02.528 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 64 2024-05-28T06:20:02.530 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.533+0000 7f8402600640 1 -- 172.21.15.96:0/1993299563 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f83d806ddf0 msgr2=0x7f83d80702b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:02.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.533+0000 7f8402600640 1 --2- 172.21.15.96:0/1993299563 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f83d806ddf0 0x7f83d80702b0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f83ec002230 tx=0x7f83ec001360 comp rx=0 tx=0).stop 2024-05-28T06:20:02.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.533+0000 7f8402600640 1 -- 172.21.15.96:0/1993299563 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83fc0b0fe0 msgr2=0x7f83fc154450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:02.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.533+0000 7f8402600640 1 --2- 172.21.15.96:0/1993299563 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83fc0b0fe0 0x7f83fc154450 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7f83fc06c8b0 tx=0x7f83f400d890 comp rx=0 tx=0).stop 2024-05-28T06:20:02.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.534+0000 7f8402600640 1 -- 172.21.15.96:0/1993299563 shutdown_connections 2024-05-28T06:20:02.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.534+0000 7f8402600640 1 --2- 172.21.15.96:0/1993299563 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f83d806ddf0 0x7f83d80702b0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:02.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.534+0000 7f8402600640 1 --2- 172.21.15.96:0/1993299563 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83fc0b0fe0 0x7f83fc154450 unknown :-1 s=CLOSED pgs=1543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:02.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.534+0000 7f8402600640 1 --2- 172.21.15.96:0/1993299563 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f83fc151240 0x7f83fc0b0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:02.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.534+0000 7f8402600640 1 --2- 172.21.15.96:0/1993299563 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f83e0004800 0x7f83fc1f2210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:02.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.534+0000 7f8402600640 1 -- 172.21.15.96:0/1993299563 >> 172.21.15.96:0/1993299563 conn(0x7f83fc06b900 msgr2=0x7f83fc0ae750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:02.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.534+0000 7f8402600640 1 -- 172.21.15.96:0/1993299563 shutdown_connections 2024-05-28T06:20:02.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.534+0000 7f8402600640 1 -- 172.21.15.96:0/1993299563 wait complete. 2024-05-28T06:20:02.549 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-28T06:20:02.742 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.745+0000 7f3cfbe00640 1 Processor -- start 2024-05-28T06:20:02.743 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.745+0000 7f3cfbe00640 1 -- start start 2024-05-28T06:20:02.748 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.746+0000 7f3cfbe00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f3cfc156b30 0x7f3cfc0af6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:02.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.746+0000 7f3cfbe00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3cfc0ad940 0x7f3cfc0b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:02.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.746+0000 7f3cfbe00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3cfc156f10 con 0x7f3cfc06ee40 2024-05-28T06:20:02.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.746+0000 7f3cfbe00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f3cfc06dda0 con 0x7f3cfc156b30 2024-05-28T06:20:02.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.746+0000 7f3cfbe00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f3cfc06e080 con 0x7f3cfc0ad940 2024-05-28T06:20:02.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.747+0000 7f3cfa000640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f3cfc06ee40 0x7f3cfc0ad230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:55716/0 (socket says 172.21.15.96:55716) 2024-05-28T06:20:02.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.747+0000 7f3cfa000640 1 -- 172.21.15.96:0/4294493297 learned_addr learned my addr 172.21.15.96:0/4294493297 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:20:02.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.747+0000 7f3cfaa00640 1 --2- 172.21.15.96:0/4294493297 >> v2:172.21.15.123:3300/0 conn(0x7f3cfc156b30 0x7f3cfc0af6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:02.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.747+0000 7f3cfb400640 1 --2- 172.21.15.96:0/4294493297 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3cfc0ad940 0x7f3cfc0b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:02.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.747+0000 7f3cf9600640 1 -- 172.21.15.96:0/4294493297 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 185866553 0 0) 0x7f3cfc156f10 con 0x7f3cfc06ee40 2024-05-28T06:20:02.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.747+0000 7f3cf9600640 1 -- 172.21.15.96:0/4294493297 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3ce0003520 con 0x7f3cfc06ee40 2024-05-28T06:20:02.750 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.748+0000 7f3cf9600640 1 -- 172.21.15.96:0/4294493297 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 824038008 0 0) 0x7f3ce0003520 con 0x7f3cfc06ee40 2024-05-28T06:20:02.750 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.748+0000 7f3cf9600640 1 -- 172.21.15.96:0/4294493297 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3cfc0ad940 msgr2=0x7f3cfc0b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:02.750 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.748+0000 7f3cf9600640 1 --2- 172.21.15.96:0/4294493297 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3cfc0ad940 0x7f3cfc0b32c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:02.750 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.748+0000 7f3cf9600640 1 -- 172.21.15.96:0/4294493297 >> v2:172.21.15.123:3300/0 conn(0x7f3cfc156b30 msgr2=0x7f3cfc0af6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:02.750 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.748+0000 7f3cf9600640 1 --2- 172.21.15.96:0/4294493297 >> v2:172.21.15.123:3300/0 conn(0x7f3cfc156b30 0x7f3cfc0af6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:02.750 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.748+0000 7f3cf9600640 1 -- 172.21.15.96:0/4294493297 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cfc06e360 con 0x7f3cfc06ee40 2024-05-28T06:20:02.750 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.748+0000 7f3cf9600640 1 -- 172.21.15.96:0/4294493297 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f3ce8001670 con 0x7f3cfc06ee40 2024-05-28T06:20:02.750 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.748+0000 7f3cf9600640 1 -- 172.21.15.96:0/4294493297 >> v1:172.21.15.96:6789/0 conn(0x7f3cfc06ee40 legacy=0x7f3cfc0ad230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:02.750 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.748+0000 7f3cf9600640 1 --2- 172.21.15.96:0/4294493297 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3ce00039b0 0x7f3ce0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:02.750 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.748+0000 7f3cf9600640 1 --2- 172.21.15.96:0/4294493297 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3cfc0ad940 0x7f3ce0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:02.750 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.748+0000 7f3cf9600640 1 --2- 172.21.15.96:0/4294493297 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ce0004d40 0x7f3ce00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:02.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.748+0000 7f3cf9600640 1 -- 172.21.15.96:0/4294493297 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f3cfc06dda0 con 0x7f3ce00039b0 2024-05-28T06:20:02.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.748+0000 7f3cf9600640 1 -- 172.21.15.96:0/4294493297 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f3cfc06e080 con 0x7f3ce0004d40 2024-05-28T06:20:02.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.748+0000 7f3cf9600640 1 -- 172.21.15.96:0/4294493297 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f3ce000bb80 con 0x7f3cfc0ad940 2024-05-28T06:20:02.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.748+0000 7f3cfaa00640 1 --2- 172.21.15.96:0/4294493297 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3ce00039b0 0x7f3ce0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:02.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.748+0000 7f3cfa000640 1 --2- 172.21.15.96:0/4294493297 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3cfc0ad940 0x7f3ce0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:02.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.748+0000 7f3cfaa00640 1 -- 172.21.15.96:0/4294493297 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3cfc0ad940 msgr2=0x7f3ce0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:02.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.748+0000 7f3cfaa00640 1 --2- 172.21.15.96:0/4294493297 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3cfc0ad940 0x7f3ce0004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:02.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.748+0000 7f3cfaa00640 1 -- 172.21.15.96:0/4294493297 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ce0004d40 msgr2=0x7f3ce00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:02.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.748+0000 7f3cfaa00640 1 --2- 172.21.15.96:0/4294493297 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ce0004d40 0x7f3ce00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:02.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.749+0000 7f3cfaa00640 1 -- 172.21.15.96:0/4294493297 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ce0003520 con 0x7f3ce00039b0 2024-05-28T06:20:02.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.750+0000 7f3cfaa00640 1 --2- 172.21.15.96:0/4294493297 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3ce00039b0 0x7f3ce0003dd0 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7f3cec00ba30 tx=0x7f3ce0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42bf35a723bb4256 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:02.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.750+0000 7f3cf9600640 1 -- 172.21.15.96:0/4294493297 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cec030070 con 0x7f3ce00039b0 2024-05-28T06:20:02.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.750+0000 7f3cf9600640 1 -- 172.21.15.96:0/4294493297 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3cec02ae00 con 0x7f3ce00039b0 2024-05-28T06:20:02.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.750+0000 7f3cf9600640 1 -- 172.21.15.96:0/4294493297 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cec00b600 con 0x7f3ce00039b0 2024-05-28T06:20:02.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.750+0000 7f3cfbe00640 1 -- 172.21.15.96:0/4294493297 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3ce00039b0 msgr2=0x7f3ce0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:02.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.750+0000 7f3cfbe00640 1 --2- 172.21.15.96:0/4294493297 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3ce00039b0 0x7f3ce0003dd0 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7f3cec00ba30 tx=0x7f3ce0005830 comp rx=0 tx=0).stop 2024-05-28T06:20:02.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.751+0000 7f3cfb400640 1 -- 172.21.15.96:0/4294493297 reap_dead start 2024-05-28T06:20:02.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.751+0000 7f3cfbe00640 1 -- 172.21.15.96:0/4294493297 shutdown_connections 2024-05-28T06:20:02.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.751+0000 7f3cfbe00640 1 -- 172.21.15.96:0/4294493297 >> 172.21.15.96:0/4294493297 conn(0x7f3cfc0b1fc0 msgr2=0x7f3cfc0b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:02.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.751+0000 7f3cfbe00640 1 -- 172.21.15.96:0/4294493297 shutdown_connections 2024-05-28T06:20:02.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.751+0000 7f3cfbe00640 1 -- 172.21.15.96:0/4294493297 wait complete. 2024-05-28T06:20:02.755 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.752+0000 7f3cfbe00640 1 Processor -- start 2024-05-28T06:20:02.755 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.752+0000 7f3cfbe00640 1 -- start start 2024-05-28T06:20:02.755 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.753+0000 7f3cfbe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3ce00039b0 0x7f3cfc15f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:02.755 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.753+0000 7f3cfbe00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3cfc155120 0x7f3cfc1555a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:02.755 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.753+0000 7f3cfbe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3cfc157f60 0x7f3cfc1583e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:02.755 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.753+0000 7f3cfbe00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f3cfc0afc20 con 0x7f3ce00039b0 2024-05-28T06:20:02.755 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.753+0000 7f3cfbe00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f3cfc0b2c30 con 0x7f3cfc157f60 2024-05-28T06:20:02.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.753+0000 7f3cfbe00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f3cfc158920 con 0x7f3cfc155120 2024-05-28T06:20:02.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.754+0000 7f3cfaa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3ce00039b0 0x7f3cfc15f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:02.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.754+0000 7f3cfaa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3ce00039b0 0x7f3cfc15f3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:55856/0 (socket says 172.21.15.96:55856) 2024-05-28T06:20:02.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.754+0000 7f3cfaa00640 1 -- 172.21.15.96:0/2313440236 learned_addr learned my addr 172.21.15.96:0/2313440236 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:20:02.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.754+0000 7f3cfaa00640 1 -- 172.21.15.96:0/2313440236 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3cfc155120 msgr2=0x7f3cfc1555a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:02.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.754+0000 7f3cfa000640 1 --2- 172.21.15.96:0/2313440236 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3cfc155120 0x7f3cfc1555a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:02.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.754+0000 7f3cfaa00640 1 --2- 172.21.15.96:0/2313440236 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3cfc155120 0x7f3cfc1555a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:02.757 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.754+0000 7f3cfaa00640 1 -- 172.21.15.96:0/2313440236 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3cfc157f60 msgr2=0x7f3cfc1583e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:02.757 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.754+0000 7f3cfaa00640 1 --2- 172.21.15.96:0/2313440236 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3cfc157f60 0x7f3cfc1583e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:02.757 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.754+0000 7f3cfaa00640 1 -- 172.21.15.96:0/2313440236 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cfc1544d0 con 0x7f3ce00039b0 2024-05-28T06:20:02.757 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.754+0000 7f3cfaa00640 1 --2- 172.21.15.96:0/2313440236 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3ce00039b0 0x7f3cfc15f3a0 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7f3cec02a8f0 tx=0x7f3cec004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:02.757 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.755+0000 7f3cf3400640 1 -- 172.21.15.96:0/2313440236 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cec042030 con 0x7f3ce00039b0 2024-05-28T06:20:02.757 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.755+0000 7f3cf3400640 1 -- 172.21.15.96:0/2313440236 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3cec02ac00 con 0x7f3ce00039b0 2024-05-28T06:20:02.757 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.755+0000 7f3cfbe00640 1 -- 172.21.15.96:0/2313440236 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3cfc158b70 con 0x7f3ce00039b0 2024-05-28T06:20:02.757 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.755+0000 7f3cf3400640 1 -- 172.21.15.96:0/2313440236 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cec004240 con 0x7f3ce00039b0 2024-05-28T06:20:02.757 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.755+0000 7f3cfbe00640 1 -- 172.21.15.96:0/2313440236 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3cfc15cd80 con 0x7f3ce00039b0 2024-05-28T06:20:02.757 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.756+0000 7f3cfbe00640 1 -- 172.21.15.96:0/2313440236 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3cfc155ae0 con 0x7f3ce00039b0 2024-05-28T06:20:02.757 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.757+0000 7f3cf3400640 1 -- 172.21.15.96:0/2313440236 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 34) ==== 92927+0+0 (secure 0 0 0) 0x7f3cec045070 con 0x7f3ce00039b0 2024-05-28T06:20:02.757 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.758+0000 7f3cf3400640 1 --2- 172.21.15.96:0/2313440236 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3cdc06ddf0 0x7f3cdc0702b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:02.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.758+0000 7f3cf3400640 1 -- 172.21.15.96:0/2313440236 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 5822+0+0 (secure 0 0 0) 0x7f3cec0aac80 con 0x7f3ce00039b0 2024-05-28T06:20:02.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.759+0000 7f3cfa000640 1 --2- 172.21.15.96:0/2313440236 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3cdc06ddf0 0x7f3cdc0702b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:02.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.760+0000 7f3cfa000640 1 --2- 172.21.15.96:0/2313440236 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3cdc06ddf0 0x7f3cdc0702b0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f3ce8001eb0 tx=0x7f3ce8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:02.760 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:02.763+0000 7f3cf3400640 1 -- 172.21.15.96:0/2313440236 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3cec04c070 con 0x7f3ce00039b0 2024-05-28T06:20:03.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.007+0000 7f3cfbe00640 1 -- 172.21.15.96:0/2313440236 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3cfc0689e0 con 0x7f3ce00039b0 2024-05-28T06:20:03.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.008+0000 7f3cf3400640 1 -- 172.21.15.96:0/2313440236 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) ==== 94+0+5189 (secure 0 0 0) 0x7f3cfc1544d0 con 0x7f3ce00039b0 2024-05-28T06:20:03.005 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:20:03.006 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":53,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5645,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6834/1355344409","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":1355344409},{"type":"v1","addr":"172.21.15.96:6834","nonce":1355344409}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":47},{"gid":5651,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/1967688533","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":1967688533},{"type":"v1","addr":"172.21.15.123:6835","nonce":1967688533}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48},{"gid":6169,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/3362018278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":3362018278},{"type":"v1","addr":"172.21.15.123:6837","nonce":3362018278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-28T06:18:10.069593+0000","modified":"2024-05-28T06:18:13.212203+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8103},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8103":{"gid":8103,"name":"c","rank":0,"incarnation":49,"state":"up:active","state_seq":3,"addr":"172.21.15.96:6835/1614743177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6833","nonce":1614743177},{"type":"v1","addr":"172.21.15.96:6835","nonce":1614743177}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8103,"qdb_cluster":[8103]},"id":6}]} 2024-05-28T06:20:03.006 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 53 2024-05-28T06:20:03.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.011+0000 7f3cfbe00640 1 -- 172.21.15.96:0/2313440236 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3cdc06ddf0 msgr2=0x7f3cdc0702b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:03.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.011+0000 7f3cfbe00640 1 --2- 172.21.15.96:0/2313440236 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3cdc06ddf0 0x7f3cdc0702b0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f3ce8001eb0 tx=0x7f3ce8009040 comp rx=0 tx=0).stop 2024-05-28T06:20:03.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.011+0000 7f3cfbe00640 1 -- 172.21.15.96:0/2313440236 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3ce00039b0 msgr2=0x7f3cfc15f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:03.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.011+0000 7f3cfbe00640 1 --2- 172.21.15.96:0/2313440236 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3ce00039b0 0x7f3cfc15f3a0 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7f3cec02a8f0 tx=0x7f3cec004730 comp rx=0 tx=0).stop 2024-05-28T06:20:03.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.011+0000 7f3cfbe00640 1 -- 172.21.15.96:0/2313440236 shutdown_connections 2024-05-28T06:20:03.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.011+0000 7f3cfbe00640 1 --2- 172.21.15.96:0/2313440236 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3cdc06ddf0 0x7f3cdc0702b0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:03.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.011+0000 7f3cfbe00640 1 --2- 172.21.15.96:0/2313440236 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3cfc157f60 0x7f3cfc1583e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:03.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.011+0000 7f3cfbe00640 1 --2- 172.21.15.96:0/2313440236 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3cfc155120 0x7f3cfc1555a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:03.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.011+0000 7f3cfbe00640 1 --2- 172.21.15.96:0/2313440236 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3ce00039b0 0x7f3cfc15f3a0 unknown :-1 s=CLOSED pgs=1546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:03.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.011+0000 7f3cfbe00640 1 -- 172.21.15.96:0/2313440236 >> 172.21.15.96:0/2313440236 conn(0x7f3cfc0b1fc0 msgr2=0x7f3cfc0b04b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:03.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.011+0000 7f3cfbe00640 1 -- 172.21.15.96:0/2313440236 shutdown_connections 2024-05-28T06:20:03.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.011+0000 7f3cfbe00640 1 -- 172.21.15.96:0/2313440236 wait complete. 2024-05-28T06:20:03.027 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:20:03.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.228+0000 7faf5be00640 1 Processor -- start 2024-05-28T06:20:03.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.228+0000 7faf5be00640 1 -- start start 2024-05-28T06:20:03.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.230+0000 7faf5be00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faf5c15e870 0x7faf5c0d0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:03.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.230+0000 7faf5be00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faf5c15ec50 con 0x7faf5c0d14d0 2024-05-28T06:20:03.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.230+0000 7faf5be00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faf5c0cff80 con 0x7faf5c0cfa40 2024-05-28T06:20:03.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.230+0000 7faf5be00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7faf5c0d2150 con 0x7faf5c15e870 2024-05-28T06:20:03.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.230+0000 7faf53e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7faf5c0d14d0 0x7faf5c0cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:55718/0 (socket says 172.21.15.96:55718) 2024-05-28T06:20:03.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.230+0000 7faf53e00640 1 -- 172.21.15.96:0/130160200 learned_addr learned my addr 172.21.15.96:0/130160200 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:20:03.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.231+0000 7faf5aa00640 1 --2- 172.21.15.96:0/130160200 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faf5c15e870 0x7faf5c0d0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:03.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.231+0000 7faf5a000640 1 -- 172.21.15.96:0/130160200 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 289971421 0 0) 0x7faf5c15ec50 con 0x7faf5c0d14d0 2024-05-28T06:20:03.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.231+0000 7faf5a000640 1 -- 172.21.15.96:0/130160200 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faf40003520 con 0x7faf5c0d14d0 2024-05-28T06:20:03.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.231+0000 7faf5a000640 1 -- 172.21.15.96:0/130160200 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3887996910 0 0) 0x7faf40003520 con 0x7faf5c0d14d0 2024-05-28T06:20:03.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.231+0000 7faf5a000640 1 -- 172.21.15.96:0/130160200 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faf5c15e870 msgr2=0x7faf5c0d0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:03.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.231+0000 7faf5a000640 1 --2- 172.21.15.96:0/130160200 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faf5c15e870 0x7faf5c0d0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:03.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.231+0000 7faf5a000640 1 -- 172.21.15.96:0/130160200 >> v1:172.21.15.123:6789/0 conn(0x7faf5c0cfa40 legacy=0x7faf5c0d25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:03.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.231+0000 7faf5a000640 1 -- 172.21.15.96:0/130160200 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf5c0d0260 con 0x7faf5c0d14d0 2024-05-28T06:20:03.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.231+0000 7faf5a000640 1 -- 172.21.15.96:0/130160200 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7faf440016c0 con 0x7faf5c0d14d0 2024-05-28T06:20:03.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.231+0000 7faf5a000640 1 -- 172.21.15.96:0/130160200 >> v1:172.21.15.96:6789/0 conn(0x7faf5c0d14d0 legacy=0x7faf5c0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:03.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.231+0000 7faf5a000640 1 --2- 172.21.15.96:0/130160200 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf400039e0 0x7faf40003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:03.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.231+0000 7faf5a000640 1 --2- 172.21.15.96:0/130160200 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faf40004800 0x7faf40004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:03.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.231+0000 7faf5a000640 1 --2- 172.21.15.96:0/130160200 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faf5c15e870 0x7faf400051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:03.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.231+0000 7faf5a000640 1 -- 172.21.15.96:0/130160200 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7faf5c0d2150 con 0x7faf400039e0 2024-05-28T06:20:03.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.231+0000 7faf5a000640 1 -- 172.21.15.96:0/130160200 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7faf4000baa0 con 0x7faf40004800 2024-05-28T06:20:03.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.231+0000 7faf5a000640 1 -- 172.21.15.96:0/130160200 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7faf4000bca0 con 0x7faf5c15e870 2024-05-28T06:20:03.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.232+0000 7faf5aa00640 1 --2- 172.21.15.96:0/130160200 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf400039e0 0x7faf40003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:03.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.232+0000 7faf5aa00640 1 -- 172.21.15.96:0/130160200 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faf5c15e870 msgr2=0x7faf400051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:03.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.232+0000 7faf5aa00640 1 --2- 172.21.15.96:0/130160200 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faf5c15e870 0x7faf400051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:03.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.232+0000 7faf5aa00640 1 -- 172.21.15.96:0/130160200 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faf40004800 msgr2=0x7faf40004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:03.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.232+0000 7faf5aa00640 1 --2- 172.21.15.96:0/130160200 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faf40004800 0x7faf40004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:03.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.232+0000 7faf5aa00640 1 -- 172.21.15.96:0/130160200 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf40003520 con 0x7faf400039e0 2024-05-28T06:20:03.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.233+0000 7faf5aa00640 1 --2- 172.21.15.96:0/130160200 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf400039e0 0x7faf40003e00 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7faf4c009a90 tx=0x7faf40005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68ca5fcd59874bfd server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:03.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.233+0000 7faf5a000640 1 -- 172.21.15.96:0/130160200 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf4c036070 con 0x7faf400039e0 2024-05-28T06:20:03.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.233+0000 7faf5a000640 1 -- 172.21.15.96:0/130160200 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf4c031400 con 0x7faf400039e0 2024-05-28T06:20:03.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.233+0000 7faf5a000640 1 -- 172.21.15.96:0/130160200 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf4c009690 con 0x7faf400039e0 2024-05-28T06:20:03.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.234+0000 7faf5be00640 1 -- 172.21.15.96:0/130160200 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf400039e0 msgr2=0x7faf40003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:03.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.234+0000 7faf5be00640 1 --2- 172.21.15.96:0/130160200 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf400039e0 0x7faf40003e00 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7faf4c009a90 tx=0x7faf40005830 comp rx=0 tx=0).stop 2024-05-28T06:20:03.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.234+0000 7faf5b400640 1 -- 172.21.15.96:0/130160200 reap_dead start 2024-05-28T06:20:03.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.234+0000 7faf5be00640 1 -- 172.21.15.96:0/130160200 shutdown_connections 2024-05-28T06:20:03.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.234+0000 7faf5be00640 1 -- 172.21.15.96:0/130160200 >> 172.21.15.96:0/130160200 conn(0x7faf5c08bfc0 msgr2=0x7faf5c0d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:03.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.234+0000 7faf5be00640 1 -- 172.21.15.96:0/130160200 shutdown_connections 2024-05-28T06:20:03.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.234+0000 7faf5be00640 1 -- 172.21.15.96:0/130160200 wait complete. 2024-05-28T06:20:03.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.235+0000 7faf5be00640 1 Processor -- start 2024-05-28T06:20:03.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.236+0000 7faf5be00640 1 -- start start 2024-05-28T06:20:03.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.236+0000 7faf5be00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf400039e0 0x7faf5c0b5270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:03.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.236+0000 7faf5be00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faf5c0c9920 0x7faf5c0c9d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:03.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.236+0000 7faf5be00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faf5c158d40 0x7faf5c1591c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:03.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.236+0000 7faf5be00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7faf5c0d4fb0 con 0x7faf400039e0 2024-05-28T06:20:03.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.236+0000 7faf5be00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7faf5c0d0820 con 0x7faf5c0c9920 2024-05-28T06:20:03.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.236+0000 7faf5be00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7faf5c0d0f90 con 0x7faf5c158d40 2024-05-28T06:20:03.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.237+0000 7faf5aa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf400039e0 0x7faf5c0b5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:03.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.237+0000 7faf5aa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf400039e0 0x7faf5c0b5270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:55880/0 (socket says 172.21.15.96:55880) 2024-05-28T06:20:03.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.237+0000 7faf5aa00640 1 -- 172.21.15.96:0/2843994276 learned_addr learned my addr 172.21.15.96:0/2843994276 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:20:03.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.237+0000 7faf53e00640 1 --2- 172.21.15.96:0/2843994276 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faf5c0c9920 0x7faf5c0c9d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:03.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.237+0000 7faf5aa00640 1 -- 172.21.15.96:0/2843994276 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faf5c158d40 msgr2=0x7faf5c1591c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:03.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.237+0000 7faf5aa00640 1 --2- 172.21.15.96:0/2843994276 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faf5c158d40 0x7faf5c1591c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:03.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.237+0000 7faf5aa00640 1 -- 172.21.15.96:0/2843994276 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faf5c0c9920 msgr2=0x7faf5c0c9d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:03.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.237+0000 7faf5aa00640 1 --2- 172.21.15.96:0/2843994276 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faf5c0c9920 0x7faf5c0c9d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:03.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.237+0000 7faf5aa00640 1 -- 172.21.15.96:0/2843994276 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf5c0ca3c0 con 0x7faf400039e0 2024-05-28T06:20:03.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.237+0000 7faf53e00640 1 --2- 172.21.15.96:0/2843994276 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faf5c0c9920 0x7faf5c0c9d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:20:03.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.237+0000 7faf5aa00640 1 --2- 172.21.15.96:0/2843994276 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf400039e0 0x7faf5c0b5270 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7faf4c009bb0 tx=0x7faf4c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:03.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.238+0000 7faf52a00640 1 -- 172.21.15.96:0/2843994276 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf4c040020 con 0x7faf400039e0 2024-05-28T06:20:03.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.238+0000 7faf52a00640 1 -- 172.21.15.96:0/2843994276 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf4c003bc0 con 0x7faf400039e0 2024-05-28T06:20:03.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.238+0000 7faf52a00640 1 -- 172.21.15.96:0/2843994276 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf4c036040 con 0x7faf400039e0 2024-05-28T06:20:03.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.238+0000 7faf5be00640 1 -- 172.21.15.96:0/2843994276 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf5c0cb780 con 0x7faf400039e0 2024-05-28T06:20:03.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.238+0000 7faf5be00640 1 -- 172.21.15.96:0/2843994276 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf5c159700 con 0x7faf400039e0 2024-05-28T06:20:03.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.239+0000 7faf5be00640 1 -- 172.21.15.96:0/2843994276 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf5c0689e0 con 0x7faf400039e0 2024-05-28T06:20:03.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.240+0000 7faf52a00640 1 -- 172.21.15.96:0/2843994276 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 34) ==== 92927+0+0 (secure 0 0 0) 0x7faf4c043020 con 0x7faf400039e0 2024-05-28T06:20:03.469 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.241+0000 7faf52a00640 1 --2- 172.21.15.96:0/2843994276 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7faf3806ddc0 0x7faf38070280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:03.469 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.241+0000 7faf52a00640 1 -- 172.21.15.96:0/2843994276 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 5822+0+0 (secure 0 0 0) 0x7faf4c0aac00 con 0x7faf400039e0 2024-05-28T06:20:03.469 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.242+0000 7faf53e00640 1 --2- 172.21.15.96:0/2843994276 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7faf3806ddc0 0x7faf38070280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:03.469 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.243+0000 7faf53e00640 1 --2- 172.21.15.96:0/2843994276 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7faf3806ddc0 0x7faf38070280 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7faf44001eb0 tx=0x7faf44009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:03.469 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.246+0000 7faf52a00640 1 -- 172.21.15.96:0/2843994276 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7faf4c04a030 con 0x7faf400039e0 2024-05-28T06:20:03.469 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.434+0000 7faf5be00640 1 -- 172.21.15.96:0/2843994276 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7faf5c0ba500 con 0x7faf400039e0 2024-05-28T06:20:03.469 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.437+0000 7faf52a00640 1 -- 172.21.15.96:0/2843994276 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) ==== 74+0+16876 (secure 0 0 0) 0x7faf4c0044e0 con 0x7faf400039e0 2024-05-28T06:20:03.469 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:20:03.470 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":92,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:20:00.747831+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-05-28T06:18:07.815707+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-05-28T06:18:09.061362+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":92,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"13.3b","mappings":[{"from":1,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":13,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[{"pool":13,"snaps":[{"begin":2,"length":2}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:20:03.471 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.439+0000 7faf5be00640 1 -- 172.21.15.96:0/2843994276 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7faf3806ddc0 msgr2=0x7faf38070280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:03.471 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.439+0000 7faf5be00640 1 --2- 172.21.15.96:0/2843994276 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7faf3806ddc0 0x7faf38070280 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7faf44001eb0 tx=0x7faf44009040 comp rx=0 tx=0).stop 2024-05-28T06:20:03.471 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.439+0000 7faf5be00640 1 -- 172.21.15.96:0/2843994276 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf400039e0 msgr2=0x7faf5c0b5270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:03.471 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.439+0000 7faf5be00640 1 --2- 172.21.15.96:0/2843994276 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf400039e0 0x7faf5c0b5270 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7faf4c009bb0 tx=0x7faf4c004730 comp rx=0 tx=0).stop 2024-05-28T06:20:03.471 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.440+0000 7faf5be00640 1 -- 172.21.15.96:0/2843994276 shutdown_connections 2024-05-28T06:20:03.471 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.440+0000 7faf5be00640 1 --2- 172.21.15.96:0/2843994276 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7faf3806ddc0 0x7faf38070280 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:03.471 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.440+0000 7faf5be00640 1 --2- 172.21.15.96:0/2843994276 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faf5c158d40 0x7faf5c1591c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:03.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.440+0000 7faf5be00640 1 --2- 172.21.15.96:0/2843994276 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faf5c0c9920 0x7faf5c0c9d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:03.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.440+0000 7faf5be00640 1 --2- 172.21.15.96:0/2843994276 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf400039e0 0x7faf5c0b5270 unknown :-1 s=CLOSED pgs=1549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:03.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.440+0000 7faf5be00640 1 -- 172.21.15.96:0/2843994276 >> 172.21.15.96:0/2843994276 conn(0x7faf5c08bfc0 msgr2=0x7faf5c0b21f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:03.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.440+0000 7faf5be00640 1 -- 172.21.15.96:0/2843994276 shutdown_connections 2024-05-28T06:20:03.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.440+0000 7faf5be00640 1 -- 172.21.15.96:0/2843994276 wait complete. 2024-05-28T06:20:03.473 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-05-28T06:20:03.473 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-28T06:20:03.622 DEBUG:teuthology.orchestra.run.smithi096:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:20:03.630 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:20:03.672 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.675+0000 7f2db3800640 1 Processor -- start 2024-05-28T06:20:03.672 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.675+0000 7f2db3800640 1 -- start start 2024-05-28T06:20:03.673 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.676+0000 7f2db3800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2dac0d3350 0x7f2dac0cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:03.673 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.676+0000 7f2db3800640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2dac162580 con 0x7f2dac0cf850 2024-05-28T06:20:03.673 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.676+0000 7f2db3800640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2dac0cfff0 con 0x7f2dac1621a0 2024-05-28T06:20:03.673 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.676+0000 7f2db3800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f2dac0b39c0 con 0x7f2dac0d3350 2024-05-28T06:20:03.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.677+0000 7f2db1800640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f2dac0cf850 0x7f2dac0b3f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:55730/0 (socket says 172.21.15.96:55730) 2024-05-28T06:20:03.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.677+0000 7f2db1800640 1 -- 172.21.15.96:0/539934011 learned_addr learned my addr 172.21.15.96:0/539934011 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:20:03.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.677+0000 7f2dabe00640 1 --2- 172.21.15.96:0/539934011 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2dac0d3350 0x7f2dac0cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:03.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.677+0000 7f2dab400640 1 -- 172.21.15.96:0/539934011 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3273057089 0 0) 0x7f2dac162580 con 0x7f2dac0cf850 2024-05-28T06:20:03.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.677+0000 7f2dab400640 1 -- 172.21.15.96:0/539934011 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2d94003520 con 0x7f2dac0cf850 2024-05-28T06:20:03.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.677+0000 7f2dab400640 1 -- 172.21.15.96:0/539934011 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3832437250 0 0) 0x7f2dac0cfff0 con 0x7f2dac1621a0 2024-05-28T06:20:03.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.677+0000 7f2dabe00640 1 -- 172.21.15.96:0/539934011 >> v1:172.21.15.123:6789/0 conn(0x7f2dac1621a0 legacy=0x7f2dac0d2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:03.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.677+0000 7f2dabe00640 1 -- 172.21.15.96:0/539934011 >> v1:172.21.15.96:6789/0 conn(0x7f2dac0cf850 legacy=0x7f2dac0b3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:03.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.677+0000 7f2dabe00640 1 -- 172.21.15.96:0/539934011 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2dac0d02d0 con 0x7f2dac0d3350 2024-05-28T06:20:03.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.679+0000 7f2dabe00640 1 --2- 172.21.15.96:0/539934011 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2dac0d3350 0x7f2dac0cf310 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f2d980097d0 tx=0x7f2d9802a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=26dc490bf50eeca1 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:03.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.679+0000 7f2dab400640 1 -- 172.21.15.96:0/539934011 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d98037070 con 0x7f2dac0d3350 2024-05-28T06:20:03.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.679+0000 7f2dab400640 1 -- 172.21.15.96:0/539934011 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d98032440 con 0x7f2dac0d3350 2024-05-28T06:20:03.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.679+0000 7f2dab400640 1 -- 172.21.15.96:0/539934011 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d980327f0 con 0x7f2dac0d3350 2024-05-28T06:20:03.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.680+0000 7f2db3800640 1 -- 172.21.15.96:0/539934011 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2dac0d3350 msgr2=0x7f2dac0cf310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:03.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.680+0000 7f2db3800640 1 --2- 172.21.15.96:0/539934011 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2dac0d3350 0x7f2dac0cf310 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f2d980097d0 tx=0x7f2d9802a180 comp rx=0 tx=0).stop 2024-05-28T06:20:03.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.680+0000 7f2db3800640 1 -- 172.21.15.96:0/539934011 shutdown_connections 2024-05-28T06:20:03.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.680+0000 7f2db3800640 1 --2- 172.21.15.96:0/539934011 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2dac0d3350 0x7f2dac0cf310 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:03.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.680+0000 7f2db3800640 1 -- 172.21.15.96:0/539934011 >> 172.21.15.96:0/539934011 conn(0x7f2dac08bfc0 msgr2=0x7f2dac0d4650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:03.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.680+0000 7f2db3800640 1 -- 172.21.15.96:0/539934011 shutdown_connections 2024-05-28T06:20:03.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.680+0000 7f2db3800640 1 -- 172.21.15.96:0/539934011 wait complete. 2024-05-28T06:20:03.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.681+0000 7f2db3800640 1 Processor -- start 2024-05-28T06:20:03.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.682+0000 7f2db3800640 1 -- start start 2024-05-28T06:20:03.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.682+0000 7f2db3800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2dac0cf850 0x7f2dac1f4670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:03.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.683+0000 7f2db3800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2dac0d3350 0x7f2dac0cccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:03.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.683+0000 7f2db3800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2dac1621a0 0x7f2dac0b7e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:03.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.683+0000 7f2db3800640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f2dac0d0890 con 0x7f2dac0d3350 2024-05-28T06:20:03.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.683+0000 7f2db3800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f2dac0d0b70 con 0x7f2dac1621a0 2024-05-28T06:20:03.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.683+0000 7f2db3800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f2dac0cfcc0 con 0x7f2dac0cf850 2024-05-28T06:20:03.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.683+0000 7f2dabe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2dac0d3350 0x7f2dac0cccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:03.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.683+0000 7f2dabe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2dac0d3350 0x7f2dac0cccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:55882/0 (socket says 172.21.15.96:55882) 2024-05-28T06:20:03.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.683+0000 7f2dabe00640 1 -- 172.21.15.96:0/3366577923 learned_addr learned my addr 172.21.15.96:0/3366577923 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:20:03.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.683+0000 7f2db0e00640 1 --2- 172.21.15.96:0/3366577923 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2dac0cf850 0x7f2dac1f4670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:03.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.683+0000 7f2db1800640 1 --2- 172.21.15.96:0/3366577923 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2dac1621a0 0x7f2dac0b7e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:03.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.684+0000 7f2db0e00640 1 -- 172.21.15.96:0/3366577923 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2dac1621a0 msgr2=0x7f2dac0b7e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:03.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.684+0000 7f2db0e00640 1 --2- 172.21.15.96:0/3366577923 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2dac1621a0 0x7f2dac0b7e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:03.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.684+0000 7f2db0e00640 1 -- 172.21.15.96:0/3366577923 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2dac0d3350 msgr2=0x7f2dac0cccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:03.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.684+0000 7f2db0e00640 1 --2- 172.21.15.96:0/3366577923 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2dac0d3350 0x7f2dac0cccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:03.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.684+0000 7f2db0e00640 1 -- 172.21.15.96:0/3366577923 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2dac1f4cc0 con 0x7f2dac0cf850 2024-05-28T06:20:03.683 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.684+0000 7f2dabe00640 1 --2- 172.21.15.96:0/3366577923 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2dac0d3350 0x7f2dac0cccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-28T06:20:03.683 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.684+0000 7f2db0e00640 1 --2- 172.21.15.96:0/3366577923 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2dac0cf850 0x7f2dac1f4670 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f2da000c440 tx=0x7f2da000c840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:03.683 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.684+0000 7f2da8c00640 1 -- 172.21.15.96:0/3366577923 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2da0017070 con 0x7f2dac0cf850 2024-05-28T06:20:03.683 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.684+0000 7f2da8c00640 1 -- 172.21.15.96:0/3366577923 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2da00035f0 con 0x7f2dac0cf850 2024-05-28T06:20:03.683 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.684+0000 7f2da8c00640 1 -- 172.21.15.96:0/3366577923 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2da00039e0 con 0x7f2dac0cf850 2024-05-28T06:20:03.683 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.684+0000 7f2db3800640 1 -- 172.21.15.96:0/3366577923 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2dac1f4fa0 con 0x7f2dac0cf850 2024-05-28T06:20:03.683 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.684+0000 7f2db3800640 1 -- 172.21.15.96:0/3366577923 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2dac0b83c0 con 0x7f2dac0cf850 2024-05-28T06:20:03.684 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.686+0000 7f2db3800640 1 -- 172.21.15.96:0/3366577923 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2dac0689e0 con 0x7f2dac0cf850 2024-05-28T06:20:03.684 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.687+0000 7f2da8c00640 1 -- 172.21.15.96:0/3366577923 <== mon.2 v2:172.21.15.123:3301/0 4 ==== mgrmap(e 34) ==== 92927+0+0 (secure 0 0 0) 0x7f2da001a070 con 0x7f2dac0cf850 2024-05-28T06:20:03.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.688+0000 7f2da8c00640 1 --2- 172.21.15.96:0/3366577923 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2d8c06ddc0 0x7f2d8c070280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:03.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.688+0000 7f2da8c00640 1 -- 172.21.15.96:0/3366577923 <== mon.2 v2:172.21.15.123:3301/0 5 ==== osd_map(92..92 src has 1..92) ==== 5822+0+0 (secure 0 0 0) 0x7f2da0089ee0 con 0x7f2dac0cf850 2024-05-28T06:20:03.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.689+0000 7f2dabe00640 1 --2- 172.21.15.96:0/3366577923 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2d8c06ddc0 0x7f2d8c070280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:03.686 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.690+0000 7f2dabe00640 1 --2- 172.21.15.96:0/3366577923 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2d8c06ddc0 0x7f2d8c070280 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f2d98009720 tx=0x7f2d9802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:03.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.700+0000 7f2da8c00640 1 -- 172.21.15.96:0/3366577923 <== mon.2 v2:172.21.15.123:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2da00597d0 con 0x7f2dac0cf850 2024-05-28T06:20:03.943 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.946+0000 7f2db3800640 1 -- 172.21.15.96:0/3366577923 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2dac0b59e0 con 0x7f2dac0cf850 2024-05-28T06:20:03.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.948+0000 7f2da8c00640 1 -- 172.21.15.96:0/3366577923 <== mon.2 v2:172.21.15.123:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) ==== 94+0+5189 (secure 0 0 0) 0x7f2da005dad0 con 0x7f2dac0cf850 2024-05-28T06:20:03.945 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:20:03.945 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":53,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5645,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6834/1355344409","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":1355344409},{"type":"v1","addr":"172.21.15.96:6834","nonce":1355344409}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":47},{"gid":5651,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/1967688533","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":1967688533},{"type":"v1","addr":"172.21.15.123:6835","nonce":1967688533}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48},{"gid":6169,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/3362018278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":3362018278},{"type":"v1","addr":"172.21.15.123:6837","nonce":3362018278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-28T06:18:10.069593+0000","modified":"2024-05-28T06:18:13.212203+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8103},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8103":{"gid":8103,"name":"c","rank":0,"incarnation":49,"state":"up:active","state_seq":3,"addr":"172.21.15.96:6835/1614743177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6833","nonce":1614743177},{"type":"v1","addr":"172.21.15.96:6835","nonce":1614743177}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8103,"qdb_cluster":[8103]},"id":6}]} 2024-05-28T06:20:03.945 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 53 2024-05-28T06:20:03.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.951+0000 7f2db3800640 1 -- 172.21.15.96:0/3366577923 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2d8c06ddc0 msgr2=0x7f2d8c070280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:03.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.951+0000 7f2db3800640 1 --2- 172.21.15.96:0/3366577923 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2d8c06ddc0 0x7f2d8c070280 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f2d98009720 tx=0x7f2d9802e000 comp rx=0 tx=0).stop 2024-05-28T06:20:03.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.951+0000 7f2db3800640 1 -- 172.21.15.96:0/3366577923 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2dac0cf850 msgr2=0x7f2dac1f4670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:03.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.951+0000 7f2db3800640 1 --2- 172.21.15.96:0/3366577923 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2dac0cf850 0x7f2dac1f4670 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f2da000c440 tx=0x7f2da000c840 comp rx=0 tx=0).stop 2024-05-28T06:20:03.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.951+0000 7f2db3800640 1 -- 172.21.15.96:0/3366577923 shutdown_connections 2024-05-28T06:20:03.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.951+0000 7f2db3800640 1 --2- 172.21.15.96:0/3366577923 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2d8c06ddc0 0x7f2d8c070280 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:03.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.951+0000 7f2db3800640 1 --2- 172.21.15.96:0/3366577923 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2dac1621a0 0x7f2dac0b7e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:03.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.951+0000 7f2db3800640 1 --2- 172.21.15.96:0/3366577923 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2dac0d3350 0x7f2dac0cccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:03.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.951+0000 7f2db3800640 1 --2- 172.21.15.96:0/3366577923 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2dac0cf850 0x7f2dac1f4670 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:03.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.951+0000 7f2db3800640 1 -- 172.21.15.96:0/3366577923 >> 172.21.15.96:0/3366577923 conn(0x7f2dac08bfc0 msgr2=0x7f2dac0d2550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:03.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.951+0000 7f2db3800640 1 -- 172.21.15.96:0/3366577923 shutdown_connections 2024-05-28T06:20:03.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:03.951+0000 7f2db3800640 1 -- 172.21.15.96:0/3366577923 wait complete. 2024-05-28T06:20:03.968 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-28T06:20:04.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.173+0000 7f3fe9400640 1 Processor -- start 2024-05-28T06:20:04.171 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.173+0000 7f3fe9400640 1 -- start start 2024-05-28T06:20:04.172 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.175+0000 7f3fe9400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f3fe4156b30 0x7f3fe40af6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:04.172 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.175+0000 7f3fe9400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3fe406ee40 0x7f3fe40ad230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:04.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.175+0000 7f3fe9400640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3fe4156f10 con 0x7f3fe406f2c0 2024-05-28T06:20:04.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.175+0000 7f3fe9400640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f3fe406dda0 con 0x7f3fe4156b30 2024-05-28T06:20:04.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.175+0000 7f3fe9400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f3fe406e080 con 0x7f3fe406ee40 2024-05-28T06:20:04.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.175+0000 7f3fe3e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f3fe4156b30 0x7f3fe40af6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:04.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.176+0000 7f3fe3e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f3fe4156b30 0x7f3fe40af6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:55898/0 (socket says 172.21.15.96:55898) 2024-05-28T06:20:04.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.176+0000 7f3fe3e00640 1 -- 172.21.15.96:0/2500852287 learned_addr learned my addr 172.21.15.96:0/2500852287 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:20:04.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.176+0000 7f3fe3400640 1 --2- 172.21.15.96:0/2500852287 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3fe406ee40 0x7f3fe40ad230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:04.174 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.176+0000 7f3fe3e00640 1 -- 172.21.15.96:0/2500852287 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3fe406ee40 msgr2=0x7f3fe40ad230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:04.174 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.176+0000 7f3fe3e00640 1 --2- 172.21.15.96:0/2500852287 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3fe406ee40 0x7f3fe40ad230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:04.174 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.176+0000 7f3fe3e00640 1 -- 172.21.15.96:0/2500852287 >> v1:172.21.15.123:6789/0 conn(0x7f3fe406f2c0 legacy=0x7f3fe40b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:04.174 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.176+0000 7f3fe3e00640 1 -- 172.21.15.96:0/2500852287 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fe406e360 con 0x7f3fe4156b30 2024-05-28T06:20:04.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.178+0000 7f3fe3e00640 1 --2- 172.21.15.96:0/2500852287 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3fe4156b30 0x7f3fe40af6e0 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7f3fd4009870 tx=0x7f3fd402a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94f42dc082011f7a server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:04.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.178+0000 7f3fe2a00640 1 -- 172.21.15.96:0/2500852287 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fd4037070 con 0x7f3fe4156b30 2024-05-28T06:20:04.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.178+0000 7f3fe2a00640 1 -- 172.21.15.96:0/2500852287 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3fd4032440 con 0x7f3fe4156b30 2024-05-28T06:20:04.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.178+0000 7f3fe2a00640 1 -- 172.21.15.96:0/2500852287 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fd4032810 con 0x7f3fe4156b30 2024-05-28T06:20:04.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.179+0000 7f3fe9400640 1 -- 172.21.15.96:0/2500852287 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3fe4156b30 msgr2=0x7f3fe40af6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:04.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.179+0000 7f3fe9400640 1 --2- 172.21.15.96:0/2500852287 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3fe4156b30 0x7f3fe40af6e0 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7f3fd4009870 tx=0x7f3fd402a1c0 comp rx=0 tx=0).stop 2024-05-28T06:20:04.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.179+0000 7f3fe9400640 1 -- 172.21.15.96:0/2500852287 shutdown_connections 2024-05-28T06:20:04.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.179+0000 7f3fe9400640 1 --2- 172.21.15.96:0/2500852287 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3fe406ee40 0x7f3fe40ad230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:04.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.179+0000 7f3fe9400640 1 --2- 172.21.15.96:0/2500852287 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3fe4156b30 0x7f3fe40af6e0 unknown :-1 s=CLOSED pgs=1551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:04.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.179+0000 7f3fe9400640 1 -- 172.21.15.96:0/2500852287 >> 172.21.15.96:0/2500852287 conn(0x7f3fe40b1fc0 msgr2=0x7f3fe40b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:04.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.179+0000 7f3fe9400640 1 -- 172.21.15.96:0/2500852287 shutdown_connections 2024-05-28T06:20:04.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.179+0000 7f3fe9400640 1 -- 172.21.15.96:0/2500852287 wait complete. 2024-05-28T06:20:04.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.181+0000 7f3fe9400640 1 Processor -- start 2024-05-28T06:20:04.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.181+0000 7f3fe9400640 1 -- start start 2024-05-28T06:20:04.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.182+0000 7f3fe9400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3fe406ee40 0x7f3fe414c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:04.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.182+0000 7f3fe9400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3fe406f2c0 0x7f3fe4151d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:04.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.182+0000 7f3fe9400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3fe4156b30 0x7f3fe41522a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:04.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.182+0000 7f3fe9400640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f3fe40afc20 con 0x7f3fe406f2c0 2024-05-28T06:20:04.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.182+0000 7f3fe9400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f3fe40b2c30 con 0x7f3fe4156b30 2024-05-28T06:20:04.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.182+0000 7f3fe9400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f3fe415e5d0 con 0x7f3fe406ee40 2024-05-28T06:20:04.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.182+0000 7f3fe3400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3fe406f2c0 0x7f3fe4151d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:04.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.182+0000 7f3fe3400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3fe406f2c0 0x7f3fe4151d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:55914/0 (socket says 172.21.15.96:55914) 2024-05-28T06:20:04.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.182+0000 7f3fe3400640 1 -- 172.21.15.96:0/1748061876 learned_addr learned my addr 172.21.15.96:0/1748061876 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:20:04.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.182+0000 7f3fe3e00640 1 --2- 172.21.15.96:0/1748061876 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3fe406ee40 0x7f3fe414c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:04.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.182+0000 7f3fe3400640 1 -- 172.21.15.96:0/1748061876 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3fe406ee40 msgr2=0x7f3fe414c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:04.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.182+0000 7f3fe3400640 1 --2- 172.21.15.96:0/1748061876 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3fe406ee40 0x7f3fe414c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:04.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.182+0000 7f3fe3400640 1 -- 172.21.15.96:0/1748061876 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3fe4156b30 msgr2=0x7f3fe41522a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:04.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.182+0000 7f3fe8a00640 1 --2- 172.21.15.96:0/1748061876 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3fe4156b30 0x7f3fe41522a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:04.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.182+0000 7f3fe3400640 1 --2- 172.21.15.96:0/1748061876 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3fe4156b30 0x7f3fe41522a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:04.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.182+0000 7f3fe3400640 1 -- 172.21.15.96:0/1748061876 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fe415e8b0 con 0x7f3fe406f2c0 2024-05-28T06:20:04.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.183+0000 7f3fe3400640 1 --2- 172.21.15.96:0/1748061876 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3fe406f2c0 0x7f3fe4151d60 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f3fd0007e40 tx=0x7f3fd000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:04.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.183+0000 7f3fdbe00640 1 -- 172.21.15.96:0/1748061876 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fd0014620 con 0x7f3fe406f2c0 2024-05-28T06:20:04.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.183+0000 7f3fdbe00640 1 -- 172.21.15.96:0/1748061876 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3fd0019070 con 0x7f3fe406f2c0 2024-05-28T06:20:04.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.183+0000 7f3fdbe00640 1 -- 172.21.15.96:0/1748061876 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fd0014a00 con 0x7f3fe406f2c0 2024-05-28T06:20:04.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.183+0000 7f3fe9400640 1 -- 172.21.15.96:0/1748061876 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3fe415eb90 con 0x7f3fe406f2c0 2024-05-28T06:20:04.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.183+0000 7f3fe9400640 1 -- 172.21.15.96:0/1748061876 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3fe415f150 con 0x7f3fe406f2c0 2024-05-28T06:20:04.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.184+0000 7f3fe9400640 1 -- 172.21.15.96:0/1748061876 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3fe40689e0 con 0x7f3fe406f2c0 2024-05-28T06:20:04.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.185+0000 7f3fdbe00640 1 -- 172.21.15.96:0/1748061876 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 34) ==== 92927+0+0 (secure 0 0 0) 0x7f3fd0014ba0 con 0x7f3fe406f2c0 2024-05-28T06:20:04.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.186+0000 7f3fdbe00640 1 --2- 172.21.15.96:0/1748061876 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3fc406ddf0 0x7f3fc40702b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:04.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.186+0000 7f3fdbe00640 1 -- 172.21.15.96:0/1748061876 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 5822+0+0 (secure 0 0 0) 0x7f3fd008b410 con 0x7f3fe406f2c0 2024-05-28T06:20:04.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.187+0000 7f3fe3e00640 1 --2- 172.21.15.96:0/1748061876 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3fc406ddf0 0x7f3fc40702b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:04.185 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.188+0000 7f3fe3e00640 1 --2- 172.21.15.96:0/1748061876 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3fc406ddf0 0x7f3fc40702b0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f3fd4034000 tx=0x7f3fd402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:04.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.191+0000 7f3fdbe00640 1 -- 172.21.15.96:0/1748061876 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3fd005af10 con 0x7f3fe406f2c0 2024-05-28T06:20:04.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.428+0000 7f3fe9400640 1 -- 172.21.15.96:0/1748061876 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3fe4073bb0 con 0x7f3fe406f2c0 2024-05-28T06:20:04.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.429+0000 7f3fdbe00640 1 -- 172.21.15.96:0/1748061876 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) ==== 94+0+5189 (secure 0 0 0) 0x7f3fd005a810 con 0x7f3fe406f2c0 2024-05-28T06:20:04.427 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:20:04.427 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":53,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5645,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6834/1355344409","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":1355344409},{"type":"v1","addr":"172.21.15.96:6834","nonce":1355344409}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":47},{"gid":5651,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/1967688533","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":1967688533},{"type":"v1","addr":"172.21.15.123:6835","nonce":1967688533}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48},{"gid":6169,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/3362018278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":3362018278},{"type":"v1","addr":"172.21.15.123:6837","nonce":3362018278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-28T06:18:10.069593+0000","modified":"2024-05-28T06:18:13.212203+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8103},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8103":{"gid":8103,"name":"c","rank":0,"incarnation":49,"state":"up:active","state_seq":3,"addr":"172.21.15.96:6835/1614743177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6833","nonce":1614743177},{"type":"v1","addr":"172.21.15.96:6835","nonce":1614743177}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8103,"qdb_cluster":[8103]},"id":6}]} 2024-05-28T06:20:04.427 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 53 2024-05-28T06:20:04.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.432+0000 7f3fe9400640 1 -- 172.21.15.96:0/1748061876 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3fc406ddf0 msgr2=0x7f3fc40702b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:04.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.432+0000 7f3fe9400640 1 --2- 172.21.15.96:0/1748061876 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3fc406ddf0 0x7f3fc40702b0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f3fd4034000 tx=0x7f3fd402e000 comp rx=0 tx=0).stop 2024-05-28T06:20:04.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.432+0000 7f3fe9400640 1 -- 172.21.15.96:0/1748061876 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3fe406f2c0 msgr2=0x7f3fe4151d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:04.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.432+0000 7f3fe9400640 1 --2- 172.21.15.96:0/1748061876 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3fe406f2c0 0x7f3fe4151d60 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f3fd0007e40 tx=0x7f3fd000e4b0 comp rx=0 tx=0).stop 2024-05-28T06:20:04.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.432+0000 7f3fe9400640 1 -- 172.21.15.96:0/1748061876 shutdown_connections 2024-05-28T06:20:04.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.432+0000 7f3fe9400640 1 --2- 172.21.15.96:0/1748061876 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3fc406ddf0 0x7f3fc40702b0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:04.430 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.432+0000 7f3fe9400640 1 --2- 172.21.15.96:0/1748061876 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3fe4156b30 0x7f3fe41522a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:04.430 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.432+0000 7f3fe9400640 1 --2- 172.21.15.96:0/1748061876 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3fe406f2c0 0x7f3fe4151d60 unknown :-1 s=CLOSED pgs=1552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:04.430 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.432+0000 7f3fe9400640 1 --2- 172.21.15.96:0/1748061876 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3fe406ee40 0x7f3fe414c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:04.430 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.432+0000 7f3fe9400640 1 -- 172.21.15.96:0/1748061876 >> 172.21.15.96:0/1748061876 conn(0x7f3fe40b1fc0 msgr2=0x7f3fe40b0d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:04.430 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.432+0000 7f3fe9400640 1 -- 172.21.15.96:0/1748061876 shutdown_connections 2024-05-28T06:20:04.430 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.432+0000 7f3fe9400640 1 -- 172.21.15.96:0/1748061876 wait complete. 2024-05-28T06:20:04.447 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:20:04.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.640+0000 7f9eecc00640 1 Processor -- start 2024-05-28T06:20:04.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.640+0000 7f9eecc00640 1 -- start start 2024-05-28T06:20:04.638 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.641+0000 7f9eecc00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f9ee80ad940 0x7f9ee80b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:04.639 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.641+0000 7f9eecc00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ee8156f10 con 0x7f9ee806ee40 2024-05-28T06:20:04.639 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.641+0000 7f9eecc00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ee806dda0 con 0x7f9ee8156b30 2024-05-28T06:20:04.639 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.641+0000 7f9eecc00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f9ee806e080 con 0x7f9ee80ad940 2024-05-28T06:20:04.639 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.642+0000 7f9ee6a00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f9ee806ee40 0x7f9ee80ad230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:55742/0 (socket says 172.21.15.96:55742) 2024-05-28T06:20:04.639 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.642+0000 7f9ee6a00640 1 -- 172.21.15.96:0/2529350940 learned_addr learned my addr 172.21.15.96:0/2529350940 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:20:04.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.642+0000 7f9ee6000640 1 -- 172.21.15.96:0/2529350940 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1714175973 0 0) 0x7f9ee8156f10 con 0x7f9ee806ee40 2024-05-28T06:20:04.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.642+0000 7f9ee7e00640 1 --2- 172.21.15.96:0/2529350940 >> v2:172.21.15.123:3300/0 conn(0x7f9ee80ad940 0x7f9ee80b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:04.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.642+0000 7f9ee6000640 1 -- 172.21.15.96:0/2529350940 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9ecc003520 con 0x7f9ee806ee40 2024-05-28T06:20:04.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.643+0000 7f9ee6000640 1 -- 172.21.15.96:0/2529350940 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1535496073 0 0) 0x7f9ecc003520 con 0x7f9ee806ee40 2024-05-28T06:20:04.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.643+0000 7f9ee6000640 1 -- 172.21.15.96:0/2529350940 >> v2:172.21.15.123:3300/0 conn(0x7f9ee80ad940 msgr2=0x7f9ee80b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:04.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.643+0000 7f9ee6000640 1 --2- 172.21.15.96:0/2529350940 >> v2:172.21.15.123:3300/0 conn(0x7f9ee80ad940 0x7f9ee80b32c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:04.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.643+0000 7f9ee6000640 1 -- 172.21.15.96:0/2529350940 >> v1:172.21.15.123:6789/0 conn(0x7f9ee8156b30 legacy=0x7f9ee80af6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:04.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.643+0000 7f9ee6000640 1 -- 172.21.15.96:0/2529350940 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ee806e360 con 0x7f9ee806ee40 2024-05-28T06:20:04.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.643+0000 7f9ee6000640 1 -- 172.21.15.96:0/2529350940 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f9ed4001670 con 0x7f9ee806ee40 2024-05-28T06:20:04.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.643+0000 7f9ee6000640 1 -- 172.21.15.96:0/2529350940 >> v1:172.21.15.96:6789/0 conn(0x7f9ee806ee40 legacy=0x7f9ee80ad230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:04.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.643+0000 7f9ee6000640 1 --2- 172.21.15.96:0/2529350940 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9ecc003a00 0x7f9ecc003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:04.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.643+0000 7f9ee6000640 1 --2- 172.21.15.96:0/2529350940 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ecc004800 0x7f9ecc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:04.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.643+0000 7f9ee6000640 1 --2- 172.21.15.96:0/2529350940 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ecc0052e0 0x7f9ecc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:04.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.643+0000 7f9ee6000640 1 -- 172.21.15.96:0/2529350940 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f9ee806e080 con 0x7f9ecc004800 2024-05-28T06:20:04.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.643+0000 7f9ee6000640 1 -- 172.21.15.96:0/2529350940 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f9ecc00c200 con 0x7f9ecc0052e0 2024-05-28T06:20:04.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.643+0000 7f9ee6000640 1 -- 172.21.15.96:0/2529350940 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f9ecc00c400 con 0x7f9ecc003a00 2024-05-28T06:20:04.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.643+0000 7f9ee6a00640 1 --2- 172.21.15.96:0/2529350940 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ecc004800 0x7f9ecc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:04.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.643+0000 7f9ee7400640 1 --2- 172.21.15.96:0/2529350940 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9ecc003a00 0x7f9ecc003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:04.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.644+0000 7f9ee7e00640 1 --2- 172.21.15.96:0/2529350940 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ecc0052e0 0x7f9ecc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:04.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.644+0000 7f9ee6a00640 1 -- 172.21.15.96:0/2529350940 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9ecc003a00 msgr2=0x7f9ecc003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:04.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.644+0000 7f9ee6a00640 1 --2- 172.21.15.96:0/2529350940 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9ecc003a00 0x7f9ecc003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:04.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.644+0000 7f9ee6a00640 1 -- 172.21.15.96:0/2529350940 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ecc0052e0 msgr2=0x7f9ecc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:04.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.644+0000 7f9ee6a00640 1 --2- 172.21.15.96:0/2529350940 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ecc0052e0 0x7f9ecc00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:04.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.644+0000 7f9ee6a00640 1 -- 172.21.15.96:0/2529350940 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ecc003520 con 0x7f9ecc004800 2024-05-28T06:20:04.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.644+0000 7f9ee7400640 1 --2- 172.21.15.96:0/2529350940 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9ecc003a00 0x7f9ecc003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:20:04.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.644+0000 7f9ee7e00640 1 --2- 172.21.15.96:0/2529350940 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ecc0052e0 0x7f9ecc00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:20:04.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.644+0000 7f9ee7e00640 1 -- 172.21.15.96:0/2529350940 reap_dead start 2024-05-28T06:20:04.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.645+0000 7f9ee6a00640 1 --2- 172.21.15.96:0/2529350940 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ecc004800 0x7f9ecc004c80 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7f9ed400c0e0 tx=0x7f9ecc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba824be996857c25 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:04.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.645+0000 7f9ee6000640 1 -- 172.21.15.96:0/2529350940 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ed4007c20 con 0x7f9ecc004800 2024-05-28T06:20:04.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.645+0000 7f9ee6000640 1 -- 172.21.15.96:0/2529350940 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ed4005af0 con 0x7f9ecc004800 2024-05-28T06:20:04.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.645+0000 7f9ee6000640 1 -- 172.21.15.96:0/2529350940 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ed4007070 con 0x7f9ecc004800 2024-05-28T06:20:04.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.646+0000 7f9eecc00640 1 -- 172.21.15.96:0/2529350940 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ecc004800 msgr2=0x7f9ecc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:04.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.646+0000 7f9eecc00640 1 --2- 172.21.15.96:0/2529350940 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ecc004800 0x7f9ecc004c80 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7f9ed400c0e0 tx=0x7f9ecc005830 comp rx=0 tx=0).stop 2024-05-28T06:20:04.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.646+0000 7f9eecc00640 1 -- 172.21.15.96:0/2529350940 shutdown_connections 2024-05-28T06:20:04.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.646+0000 7f9eecc00640 1 --2- 172.21.15.96:0/2529350940 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ecc004800 0x7f9ecc004c80 unknown :-1 s=CLOSED pgs=1554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:04.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.646+0000 7f9eecc00640 1 -- 172.21.15.96:0/2529350940 >> 172.21.15.96:0/2529350940 conn(0x7f9ee80b1fc0 msgr2=0x7f9ee80b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:04.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.646+0000 7f9eecc00640 1 -- 172.21.15.96:0/2529350940 shutdown_connections 2024-05-28T06:20:04.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.646+0000 7f9eecc00640 1 -- 172.21.15.96:0/2529350940 wait complete. 2024-05-28T06:20:04.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.648+0000 7f9eecc00640 1 Processor -- start 2024-05-28T06:20:04.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.648+0000 7f9eecc00640 1 -- start start 2024-05-28T06:20:04.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.649+0000 7f9eecc00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9ecc004800 0x7f9ee81f23b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:04.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.649+0000 7f9eecc00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ee8099850 0x7f9ee8160010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:04.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.649+0000 7f9eecc00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ee8099c90 0x7f9ee8160550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:04.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.649+0000 7f9eecc00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f9ee80b2db0 con 0x7f9ee8099c90 2024-05-28T06:20:04.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.649+0000 7f9eecc00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f9ee80b2c30 con 0x7f9ee8099850 2024-05-28T06:20:04.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.649+0000 7f9eecc00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f9ee806f2f0 con 0x7f9ecc004800 2024-05-28T06:20:04.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.649+0000 7f9ee7e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ee8099c90 0x7f9ee8160550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:04.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.649+0000 7f9ee7e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ee8099c90 0x7f9ee8160550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:55932/0 (socket says 172.21.15.96:55932) 2024-05-28T06:20:04.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.649+0000 7f9ee7e00640 1 -- 172.21.15.96:0/1446916289 learned_addr learned my addr 172.21.15.96:0/1446916289 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:20:04.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.649+0000 7f9ee7400640 1 --2- 172.21.15.96:0/1446916289 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9ecc004800 0x7f9ee81f23b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:04.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.649+0000 7f9ee6a00640 1 --2- 172.21.15.96:0/1446916289 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ee8099850 0x7f9ee8160010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:04.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.649+0000 7f9ee7e00640 1 -- 172.21.15.96:0/1446916289 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9ecc004800 msgr2=0x7f9ee81f23b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:04.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.649+0000 7f9ee7e00640 1 --2- 172.21.15.96:0/1446916289 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9ecc004800 0x7f9ee81f23b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:04.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.649+0000 7f9ee7e00640 1 -- 172.21.15.96:0/1446916289 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ee8099850 msgr2=0x7f9ee8160010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:04.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.649+0000 7f9ee7e00640 1 --2- 172.21.15.96:0/1446916289 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ee8099850 0x7f9ee8160010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:04.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.649+0000 7f9ee7e00640 1 -- 172.21.15.96:0/1446916289 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ee8144f80 con 0x7f9ee8099c90 2024-05-28T06:20:04.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.649+0000 7f9ee7400640 1 --2- 172.21.15.96:0/1446916289 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9ecc004800 0x7f9ee81f23b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:20:04.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.650+0000 7f9ee7e00640 1 --2- 172.21.15.96:0/1446916289 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ee8099c90 0x7f9ee8160550 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7f9ee806db70 tx=0x7f9ee0009d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:04.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.650+0000 7f9edf400640 1 -- 172.21.15.96:0/1446916289 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ee0019070 con 0x7f9ee8099c90 2024-05-28T06:20:04.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.650+0000 7f9edf400640 1 -- 172.21.15.96:0/1446916289 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ee0007cf0 con 0x7f9ee8099c90 2024-05-28T06:20:04.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.650+0000 7f9edf400640 1 -- 172.21.15.96:0/1446916289 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ee0014640 con 0x7f9ee8099c90 2024-05-28T06:20:04.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.650+0000 7f9eecc00640 1 -- 172.21.15.96:0/1446916289 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ee8145260 con 0x7f9ee8099c90 2024-05-28T06:20:04.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.650+0000 7f9eecc00640 1 -- 172.21.15.96:0/1446916289 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ee8145820 con 0x7f9ee8099c90 2024-05-28T06:20:04.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.651+0000 7f9eecc00640 1 -- 172.21.15.96:0/1446916289 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ee80689e0 con 0x7f9ee8099c90 2024-05-28T06:20:04.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.652+0000 7f9edf400640 1 -- 172.21.15.96:0/1446916289 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 34) ==== 92927+0+0 (secure 0 0 0) 0x7f9ee001c070 con 0x7f9ee8099c90 2024-05-28T06:20:04.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.654+0000 7f9edf400640 1 --2- 172.21.15.96:0/1446916289 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9ec806ddc0 0x7f9ec8070280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:04.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.654+0000 7f9edf400640 1 -- 172.21.15.96:0/1446916289 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 5822+0+0 (secure 0 0 0) 0x7f9ee008a4b0 con 0x7f9ee8099c90 2024-05-28T06:20:04.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.654+0000 7f9ee7400640 1 --2- 172.21.15.96:0/1446916289 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9ec806ddc0 0x7f9ec8070280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:04.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.655+0000 7f9ee7400640 1 --2- 172.21.15.96:0/1446916289 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9ec806ddc0 0x7f9ec8070280 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f9ee8051690 tx=0x7f9ed8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:04.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.658+0000 7f9edf400640 1 -- 172.21.15.96:0/1446916289 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9ee005a330 con 0x7f9ee8099c90 2024-05-28T06:20:04.858 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.861+0000 7f9eecc00640 1 -- 172.21.15.96:0/1446916289 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9ee806f7f0 con 0x7f9ee8099c90 2024-05-28T06:20:04.860 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.863+0000 7f9edf400640 1 -- 172.21.15.96:0/1446916289 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) ==== 74+0+16876 (secure 0 0 0) 0x7f9ee0059c30 con 0x7f9ee8099c90 2024-05-28T06:20:04.861 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:20:04.861 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":92,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:20:00.747831+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-05-28T06:18:07.815707+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-05-28T06:18:09.061362+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":92,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"13.3b","mappings":[{"from":1,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":13,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[{"pool":13,"snaps":[{"begin":2,"length":2}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:20:04.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.866+0000 7f9eecc00640 1 -- 172.21.15.96:0/1446916289 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9ec806ddc0 msgr2=0x7f9ec8070280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:04.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.866+0000 7f9eecc00640 1 --2- 172.21.15.96:0/1446916289 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9ec806ddc0 0x7f9ec8070280 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f9ee8051690 tx=0x7f9ed8009040 comp rx=0 tx=0).stop 2024-05-28T06:20:04.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.866+0000 7f9eecc00640 1 -- 172.21.15.96:0/1446916289 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ee8099c90 msgr2=0x7f9ee8160550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:04.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.866+0000 7f9eecc00640 1 --2- 172.21.15.96:0/1446916289 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ee8099c90 0x7f9ee8160550 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7f9ee806db70 tx=0x7f9ee0009d50 comp rx=0 tx=0).stop 2024-05-28T06:20:04.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.867+0000 7f9eecc00640 1 -- 172.21.15.96:0/1446916289 shutdown_connections 2024-05-28T06:20:04.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.867+0000 7f9eecc00640 1 --2- 172.21.15.96:0/1446916289 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9ec806ddc0 0x7f9ec8070280 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:04.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.867+0000 7f9eecc00640 1 --2- 172.21.15.96:0/1446916289 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ee8099c90 0x7f9ee8160550 unknown :-1 s=CLOSED pgs=1555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:04.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.867+0000 7f9eecc00640 1 --2- 172.21.15.96:0/1446916289 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ee8099850 0x7f9ee8160010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:04.865 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.867+0000 7f9eecc00640 1 --2- 172.21.15.96:0/1446916289 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9ecc004800 0x7f9ee81f23b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:04.865 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.867+0000 7f9eecc00640 1 -- 172.21.15.96:0/1446916289 >> 172.21.15.96:0/1446916289 conn(0x7f9ee80b1fc0 msgr2=0x7f9ee815acf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:04.865 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.867+0000 7f9eecc00640 1 -- 172.21.15.96:0/1446916289 shutdown_connections 2024-05-28T06:20:04.865 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:04.867+0000 7f9eecc00640 1 -- 172.21.15.96:0/1446916289 wait complete. 2024-05-28T06:20:04.884 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-05-28T06:20:05.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.072+0000 7fe58b600640 1 Processor -- start 2024-05-28T06:20:05.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.072+0000 7fe58b600640 1 -- start start 2024-05-28T06:20:05.071 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.074+0000 7fe58b600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe5840707d0 0x7fe58406ee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:05.071 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.074+0000 7fe58b600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fe58406c310 0x7fe5840d1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:05.071 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.074+0000 7fe58b600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe58415fbc0 con 0x7fe5840b01e0 2024-05-28T06:20:05.071 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.074+0000 7fe58b600640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fe584070d50 con 0x7fe58406c310 2024-05-28T06:20:05.071 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.074+0000 7fe58b600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fe5840b6140 con 0x7fe5840707d0 2024-05-28T06:20:05.071 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.074+0000 7fe589600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fe58406c310 0x7fe5840d1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:05.071 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.074+0000 7fe589600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fe58406c310 0x7fe5840d1470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:55944/0 (socket says 172.21.15.96:55944) 2024-05-28T06:20:05.071 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.074+0000 7fe589600640 1 -- 172.21.15.96:0/1559797188 learned_addr learned my addr 172.21.15.96:0/1559797188 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:20:05.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.074+0000 7fe589600640 1 -- 172.21.15.96:0/1559797188 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe5840707d0 msgr2=0x7fe58406ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:05.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.074+0000 7fe589600640 1 --2- 172.21.15.96:0/1559797188 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe5840707d0 0x7fe58406ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:05.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.074+0000 7fe589600640 1 -- 172.21.15.96:0/1559797188 >> v1:172.21.15.123:6789/0 conn(0x7fe5840b01e0 legacy=0x7fe5840b05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:05.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.074+0000 7fe589600640 1 -- 172.21.15.96:0/1559797188 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5840b6420 con 0x7fe58406c310 2024-05-28T06:20:05.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.076+0000 7fe589600640 1 --2- 172.21.15.96:0/1559797188 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe58406c310 0x7fe5840d1470 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7fe57c009870 tx=0x7fe57c02a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca5474753404e8d9 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:05.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.076+0000 7fe583400640 1 -- 172.21.15.96:0/1559797188 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe57c037070 con 0x7fe58406c310 2024-05-28T06:20:05.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.076+0000 7fe583400640 1 -- 172.21.15.96:0/1559797188 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe57c032440 con 0x7fe58406c310 2024-05-28T06:20:05.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.076+0000 7fe583400640 1 -- 172.21.15.96:0/1559797188 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe57c032810 con 0x7fe58406c310 2024-05-28T06:20:05.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.076+0000 7fe58b600640 1 -- 172.21.15.96:0/1559797188 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe58406c310 msgr2=0x7fe5840d1470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:05.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.076+0000 7fe58b600640 1 --2- 172.21.15.96:0/1559797188 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe58406c310 0x7fe5840d1470 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7fe57c009870 tx=0x7fe57c02a210 comp rx=0 tx=0).stop 2024-05-28T06:20:05.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.077+0000 7fe58b600640 1 -- 172.21.15.96:0/1559797188 shutdown_connections 2024-05-28T06:20:05.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.077+0000 7fe58b600640 1 --2- 172.21.15.96:0/1559797188 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe58406c310 0x7fe5840d1470 unknown :-1 s=CLOSED pgs=1556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:05.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.077+0000 7fe58b600640 1 --2- 172.21.15.96:0/1559797188 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe5840707d0 0x7fe58406ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:05.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.077+0000 7fe58b600640 1 -- 172.21.15.96:0/1559797188 >> 172.21.15.96:0/1559797188 conn(0x7fe58409aa30 msgr2=0x7fe5840b6840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:05.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.077+0000 7fe58b600640 1 -- 172.21.15.96:0/1559797188 shutdown_connections 2024-05-28T06:20:05.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.077+0000 7fe58b600640 1 -- 172.21.15.96:0/1559797188 wait complete. 2024-05-28T06:20:05.075 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.078+0000 7fe58b600640 1 Processor -- start 2024-05-28T06:20:05.075 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.078+0000 7fe58b600640 1 -- start start 2024-05-28T06:20:05.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.079+0000 7fe58b600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe58406c310 0x7fe584152a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:05.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.079+0000 7fe58b600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe5840707d0 0x7fe58415cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:05.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.079+0000 7fe58b600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe5840b01e0 0x7fe58415d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:05.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.079+0000 7fe58b600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe5840d19b0 con 0x7fe5840707d0 2024-05-28T06:20:05.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.079+0000 7fe58b600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe5840b5df0 con 0x7fe5840b01e0 2024-05-28T06:20:05.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.079+0000 7fe58b600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fe5840d1b30 con 0x7fe58406c310 2024-05-28T06:20:05.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.080+0000 7fe583e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe5840707d0 0x7fe58415cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:05.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.080+0000 7fe583e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe5840707d0 0x7fe58415cc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:55954/0 (socket says 172.21.15.96:55954) 2024-05-28T06:20:05.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.080+0000 7fe583e00640 1 -- 172.21.15.96:0/518818296 learned_addr learned my addr 172.21.15.96:0/518818296 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:20:05.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.080+0000 7fe588c00640 1 --2- 172.21.15.96:0/518818296 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe58406c310 0x7fe584152a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:05.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.080+0000 7fe589600640 1 --2- 172.21.15.96:0/518818296 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe5840b01e0 0x7fe58415d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:05.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.080+0000 7fe583e00640 1 -- 172.21.15.96:0/518818296 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe58406c310 msgr2=0x7fe584152a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:05.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.080+0000 7fe583e00640 1 --2- 172.21.15.96:0/518818296 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe58406c310 0x7fe584152a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:05.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.080+0000 7fe583e00640 1 -- 172.21.15.96:0/518818296 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe5840b01e0 msgr2=0x7fe58415d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:05.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.080+0000 7fe583e00640 1 --2- 172.21.15.96:0/518818296 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe5840b01e0 0x7fe58415d1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:05.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.080+0000 7fe583e00640 1 -- 172.21.15.96:0/518818296 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe584152f60 con 0x7fe5840707d0 2024-05-28T06:20:05.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.080+0000 7fe583e00640 1 --2- 172.21.15.96:0/518818296 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe5840707d0 0x7fe58415cc80 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7fe57400ba50 tx=0x7fe57400bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:05.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.081+0000 7fe580c00640 1 -- 172.21.15.96:0/518818296 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe574019070 con 0x7fe5840707d0 2024-05-28T06:20:05.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.081+0000 7fe580c00640 1 -- 172.21.15.96:0/518818296 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe57400cba0 con 0x7fe5840707d0 2024-05-28T06:20:05.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.081+0000 7fe580c00640 1 -- 172.21.15.96:0/518818296 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe574014440 con 0x7fe5840707d0 2024-05-28T06:20:05.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.081+0000 7fe58b600640 1 -- 172.21.15.96:0/518818296 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5841560f0 con 0x7fe5840707d0 2024-05-28T06:20:05.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.081+0000 7fe58b600640 1 -- 172.21.15.96:0/518818296 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5841566b0 con 0x7fe5840707d0 2024-05-28T06:20:05.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.082+0000 7fe58b600640 1 -- 172.21.15.96:0/518818296 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5840689e0 con 0x7fe5840707d0 2024-05-28T06:20:05.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.083+0000 7fe580c00640 1 -- 172.21.15.96:0/518818296 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 34) ==== 92927+0+0 (secure 0 0 0) 0x7fe57401c070 con 0x7fe5840707d0 2024-05-28T06:20:05.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.084+0000 7fe580c00640 1 --2- 172.21.15.96:0/518818296 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe56006ddf0 0x7fe5600702b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:05.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.084+0000 7fe580c00640 1 -- 172.21.15.96:0/518818296 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 5822+0+0 (secure 0 0 0) 0x7fe57408a550 con 0x7fe5840707d0 2024-05-28T06:20:05.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.085+0000 7fe588c00640 1 --2- 172.21.15.96:0/518818296 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe56006ddf0 0x7fe5600702b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:05.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.086+0000 7fe588c00640 1 --2- 172.21.15.96:0/518818296 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe56006ddf0 0x7fe5600702b0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fe56c006fd0 tx=0x7fe56c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:05.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.089+0000 7fe580c00640 1 -- 172.21.15.96:0/518818296 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe57405a3d0 con 0x7fe5840707d0 2024-05-28T06:20:05.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.338+0000 7fe58b600640 1 -- 172.21.15.96:0/518818296 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fe58406ddb0 con 0x7fe5840707d0 2024-05-28T06:20:05.336 INFO:tasks.ceph.mon.a.smithi096.stderr:2024-05-28T06:20:05.339+0000 7fb2fca00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-28T06:20:05.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.342+0000 7fe580c00640 1 -- 172.21.15.96:0/518818296 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v54) ==== 161+0+0 (secure 0 0 0) 0x7fe574059cd0 con 0x7fe5840707d0 2024-05-28T06:20:05.339 INFO:teuthology.orchestra.run.smithi096.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-28T06:20:05.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.346+0000 7fe58b600640 1 -- 172.21.15.96:0/518818296 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe56006ddf0 msgr2=0x7fe5600702b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:05.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.346+0000 7fe58b600640 1 --2- 172.21.15.96:0/518818296 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe56006ddf0 0x7fe5600702b0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fe56c006fd0 tx=0x7fe56c008040 comp rx=0 tx=0).stop 2024-05-28T06:20:05.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.346+0000 7fe58b600640 1 -- 172.21.15.96:0/518818296 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe5840707d0 msgr2=0x7fe58415cc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:05.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.346+0000 7fe58b600640 1 --2- 172.21.15.96:0/518818296 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe5840707d0 0x7fe58415cc80 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7fe57400ba50 tx=0x7fe57400bf10 comp rx=0 tx=0).stop 2024-05-28T06:20:05.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.346+0000 7fe58b600640 1 -- 172.21.15.96:0/518818296 shutdown_connections 2024-05-28T06:20:05.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.346+0000 7fe58b600640 1 --2- 172.21.15.96:0/518818296 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe56006ddf0 0x7fe5600702b0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:05.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.346+0000 7fe58b600640 1 --2- 172.21.15.96:0/518818296 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe5840b01e0 0x7fe58415d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:05.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.346+0000 7fe58b600640 1 --2- 172.21.15.96:0/518818296 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe5840707d0 0x7fe58415cc80 unknown :-1 s=CLOSED pgs=1557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:05.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.346+0000 7fe58b600640 1 --2- 172.21.15.96:0/518818296 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe58406c310 0x7fe584152a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:05.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.346+0000 7fe58b600640 1 -- 172.21.15.96:0/518818296 >> 172.21.15.96:0/518818296 conn(0x7fe58409aa30 msgr2=0x7fe58415a1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:05.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.346+0000 7fe58b600640 1 -- 172.21.15.96:0/518818296 shutdown_connections 2024-05-28T06:20:05.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.346+0000 7fe58b600640 1 -- 172.21.15.96:0/518818296 wait complete. 2024-05-28T06:20:05.362 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-05-28T06:20:05.442 INFO:tasks.ceph.mds.c.smithi096.stdout:starting mds.c at 2024-05-28T06:20:05.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.546+0000 7f2c46400640 1 Processor -- start 2024-05-28T06:20:05.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.546+0000 7f2c46400640 1 -- start start 2024-05-28T06:20:05.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.547+0000 7f2c46400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2c400d0640 0x7f2c4015d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:05.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.547+0000 7f2c46400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f2c400d0a20 0x7f2c400cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:05.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.547+0000 7f2c46400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f2c400b32c0 0x7f2c400b3750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:05.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.547+0000 7f2c46400640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f2c400adbb0 con 0x7f2c400b32c0 2024-05-28T06:20:05.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.547+0000 7f2c46400640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f2c4015db10 con 0x7f2c400d0a20 2024-05-28T06:20:05.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.547+0000 7f2c46400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f2c4015cb10 con 0x7f2c400d0640 2024-05-28T06:20:05.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.547+0000 7f2c3fe00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f2c400b32c0 0x7f2c400b3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:05.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.547+0000 7f2c3fe00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f2c400b32c0 0x7f2c400b3750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:55980/0 (socket says 172.21.15.96:55980) 2024-05-28T06:20:05.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.547+0000 7f2c3fe00640 1 -- 172.21.15.96:0/3661465331 learned_addr learned my addr 172.21.15.96:0/3661465331 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:20:05.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.548+0000 7f2c3fe00640 1 -- 172.21.15.96:0/3661465331 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2c400d0640 msgr2=0x7f2c4015d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:05.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.548+0000 7f2c3fe00640 1 --2- 172.21.15.96:0/3661465331 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2c400d0640 0x7f2c4015d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:05.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.548+0000 7f2c3fe00640 1 -- 172.21.15.96:0/3661465331 >> v2:172.21.15.123:3300/0 conn(0x7f2c400d0a20 msgr2=0x7f2c400cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:05.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.548+0000 7f2c3fe00640 1 --2- 172.21.15.96:0/3661465331 >> v2:172.21.15.123:3300/0 conn(0x7f2c400d0a20 0x7f2c400cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:05.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.548+0000 7f2c3fe00640 1 -- 172.21.15.96:0/3661465331 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c400ad5f0 con 0x7f2c400b32c0 2024-05-28T06:20:05.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.549+0000 7f2c3fe00640 1 --2- 172.21.15.96:0/3661465331 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2c400b32c0 0x7f2c400b3750 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7f2c400b6580 tx=0x7f2c3802a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91f9a63c723d6444 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:05.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.549+0000 7f2c3e000640 1 -- 172.21.15.96:0/3661465331 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c38036070 con 0x7f2c400b32c0 2024-05-28T06:20:05.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.549+0000 7f2c3e000640 1 -- 172.21.15.96:0/3661465331 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c38031600 con 0x7f2c400b32c0 2024-05-28T06:20:05.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.549+0000 7f2c3e000640 1 -- 172.21.15.96:0/3661465331 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c380319d0 con 0x7f2c400b32c0 2024-05-28T06:20:05.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.550+0000 7f2c46400640 1 -- 172.21.15.96:0/3661465331 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2c400b32c0 msgr2=0x7f2c400b3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:05.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.550+0000 7f2c46400640 1 --2- 172.21.15.96:0/3661465331 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2c400b32c0 0x7f2c400b3750 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7f2c400b6580 tx=0x7f2c3802a6a0 comp rx=0 tx=0).stop 2024-05-28T06:20:05.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.550+0000 7f2c46400640 1 -- 172.21.15.96:0/3661465331 shutdown_connections 2024-05-28T06:20:05.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.550+0000 7f2c46400640 1 --2- 172.21.15.96:0/3661465331 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2c400b32c0 0x7f2c400b3750 unknown :-1 s=CLOSED pgs=1561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:05.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.550+0000 7f2c46400640 1 --2- 172.21.15.96:0/3661465331 >> v2:172.21.15.123:3300/0 conn(0x7f2c400d0a20 0x7f2c400cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:05.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.550+0000 7f2c46400640 1 --2- 172.21.15.96:0/3661465331 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2c400d0640 0x7f2c4015d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:05.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.550+0000 7f2c46400640 1 -- 172.21.15.96:0/3661465331 >> 172.21.15.96:0/3661465331 conn(0x7f2c4006bd20 msgr2=0x7f2c4006c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:05.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.550+0000 7f2c46400640 1 -- 172.21.15.96:0/3661465331 shutdown_connections 2024-05-28T06:20:05.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.550+0000 7f2c46400640 1 -- 172.21.15.96:0/3661465331 wait complete. 2024-05-28T06:20:05.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.552+0000 7f2c46400640 1 Processor -- start 2024-05-28T06:20:05.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.552+0000 7f2c46400640 1 -- start start 2024-05-28T06:20:05.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.553+0000 7f2c46400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c400b32c0 0x7f2c401e8830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:05.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.553+0000 7f2c46400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2c400d0640 0x7f2c401e68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:05.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.553+0000 7f2c46400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2c400d0a20 0x7f2c401e6de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:05.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.553+0000 7f2c46400640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f2c4015d0b0 con 0x7f2c400d0a20 2024-05-28T06:20:05.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.553+0000 7f2c46400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f2c4015cdd0 con 0x7f2c400b32c0 2024-05-28T06:20:05.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.553+0000 7f2c46400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f2c400ad2a0 con 0x7f2c400d0640 2024-05-28T06:20:05.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.553+0000 7f2c3fe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2c400d0a20 0x7f2c401e6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:05.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.553+0000 7f2c3fe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2c400d0a20 0x7f2c401e6de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:55984/0 (socket says 172.21.15.96:55984) 2024-05-28T06:20:05.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.553+0000 7f2c3fe00640 1 -- 172.21.15.96:0/2295520694 learned_addr learned my addr 172.21.15.96:0/2295520694 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:20:05.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.553+0000 7f2c3fe00640 1 -- 172.21.15.96:0/2295520694 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2c400d0640 msgr2=0x7f2c401e68a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:05.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.553+0000 7f2c3f400640 1 --2- 172.21.15.96:0/2295520694 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c400b32c0 0x7f2c401e8830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:05.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.553+0000 7f2c3fe00640 1 --2- 172.21.15.96:0/2295520694 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2c400d0640 0x7f2c401e68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:05.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.553+0000 7f2c3fe00640 1 -- 172.21.15.96:0/2295520694 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c400b32c0 msgr2=0x7f2c401e8830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:05.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.553+0000 7f2c3fe00640 1 --2- 172.21.15.96:0/2295520694 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c400b32c0 0x7f2c401e8830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:05.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.553+0000 7f2c3fe00640 1 -- 172.21.15.96:0/2295520694 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c40153fd0 con 0x7f2c400d0a20 2024-05-28T06:20:05.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.554+0000 7f2c3fe00640 1 --2- 172.21.15.96:0/2295520694 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2c400d0a20 0x7f2c401e6de0 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7f2c38009f40 tx=0x7f2c38004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:05.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.554+0000 7f2c37400640 1 -- 172.21.15.96:0/2295520694 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c38041020 con 0x7f2c400d0a20 2024-05-28T06:20:05.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.554+0000 7f2c37400640 1 -- 172.21.15.96:0/2295520694 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c38007780 con 0x7f2c400d0a20 2024-05-28T06:20:05.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.554+0000 7f2c46400640 1 -- 172.21.15.96:0/2295520694 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c40148f30 con 0x7f2c400d0a20 2024-05-28T06:20:05.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.554+0000 7f2c37400640 1 -- 172.21.15.96:0/2295520694 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c38036040 con 0x7f2c400d0a20 2024-05-28T06:20:05.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.554+0000 7f2c46400640 1 -- 172.21.15.96:0/2295520694 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c40151040 con 0x7f2c400d0a20 2024-05-28T06:20:05.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.555+0000 7f2c46400640 1 -- 172.21.15.96:0/2295520694 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c40150380 con 0x7f2c400d0a20 2024-05-28T06:20:05.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.556+0000 7f2c37400640 1 -- 172.21.15.96:0/2295520694 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 34) ==== 92927+0+0 (secure 0 0 0) 0x7f2c38044020 con 0x7f2c400d0a20 2024-05-28T06:20:05.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.557+0000 7f2c37400640 1 --2- 172.21.15.96:0/2295520694 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2c2006ddc0 0x7f2c20070280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:05.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.557+0000 7f2c37400640 1 -- 172.21.15.96:0/2295520694 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 5880+0+0 (secure 0 0 0) 0x7f2c380ab6f0 con 0x7f2c400d0a20 2024-05-28T06:20:05.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.558+0000 7f2c3f400640 1 --2- 172.21.15.96:0/2295520694 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2c2006ddc0 0x7f2c20070280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:05.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.559+0000 7f2c3f400640 1 --2- 172.21.15.96:0/2295520694 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2c2006ddc0 0x7f2c20070280 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f2c3000a810 tx=0x7f2c30008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:05.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.562+0000 7f2c37400640 1 -- 172.21.15.96:0/2295520694 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2c40150380 con 0x7f2c400d0a20 2024-05-28T06:20:05.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:05.806+0000 7f2c46400640 1 -- 172.21.15.96:0/2295520694 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f2c400689e0 con 0x7f2c400d0a20 2024-05-28T06:20:06.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.345+0000 7f2c37400640 1 -- 172.21.15.96:0/2295520694 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v55) ==== 104+0+0 (secure 0 0 0) 0x7f2c3804b030 con 0x7f2c400d0a20 2024-05-28T06:20:06.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.349+0000 7f2c46400640 1 -- 172.21.15.96:0/2295520694 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2c2006ddc0 msgr2=0x7f2c20070280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:06.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.349+0000 7f2c46400640 1 --2- 172.21.15.96:0/2295520694 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2c2006ddc0 0x7f2c20070280 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f2c3000a810 tx=0x7f2c30008040 comp rx=0 tx=0).stop 2024-05-28T06:20:06.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.349+0000 7f2c46400640 1 -- 172.21.15.96:0/2295520694 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2c400d0a20 msgr2=0x7f2c401e6de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:06.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.349+0000 7f2c46400640 1 --2- 172.21.15.96:0/2295520694 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2c400d0a20 0x7f2c401e6de0 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7f2c38009f40 tx=0x7f2c38004a40 comp rx=0 tx=0).stop 2024-05-28T06:20:06.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.349+0000 7f2c46400640 1 -- 172.21.15.96:0/2295520694 shutdown_connections 2024-05-28T06:20:06.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.349+0000 7f2c46400640 1 --2- 172.21.15.96:0/2295520694 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2c2006ddc0 0x7f2c20070280 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:06.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.349+0000 7f2c46400640 1 --2- 172.21.15.96:0/2295520694 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2c400d0a20 0x7f2c401e6de0 unknown :-1 s=CLOSED pgs=1562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:06.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.349+0000 7f2c46400640 1 --2- 172.21.15.96:0/2295520694 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2c400d0640 0x7f2c401e68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:06.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.349+0000 7f2c46400640 1 --2- 172.21.15.96:0/2295520694 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2c400b32c0 0x7f2c401e8830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:06.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.349+0000 7f2c46400640 1 -- 172.21.15.96:0/2295520694 >> 172.21.15.96:0/2295520694 conn(0x7f2c4006bd20 msgr2=0x7f2c4015e1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:06.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.349+0000 7f2c46400640 1 -- 172.21.15.96:0/2295520694 shutdown_connections 2024-05-28T06:20:06.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.349+0000 7f2c46400640 1 -- 172.21.15.96:0/2295520694 wait complete. 2024-05-28T06:20:06.369 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-05-28T06:20:06.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.574+0000 7f20fa800640 1 Processor -- start 2024-05-28T06:20:06.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.574+0000 7f20fa800640 1 -- start start 2024-05-28T06:20:06.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.575+0000 7f20fa800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f20f415e830 0x7f20f406dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:06.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.575+0000 7f20fa800640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20f406e190 con 0x7f20f40b3510 2024-05-28T06:20:06.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.576+0000 7f20fa800640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20f406e470 con 0x7f20f4072310 2024-05-28T06:20:06.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.576+0000 7f20fa800640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f20f406e750 con 0x7f20f415e830 2024-05-28T06:20:06.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.576+0000 7f20f2a00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f20f40b3510 0x7f20f4071c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:55768/0 (socket says 172.21.15.96:55768) 2024-05-28T06:20:06.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.576+0000 7f20f2a00640 1 -- 172.21.15.96:0/3922510931 learned_addr learned my addr 172.21.15.96:0/3922510931 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:20:06.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.576+0000 7f20f3400640 1 --2- 172.21.15.96:0/3922510931 >> v2:172.21.15.123:3300/0 conn(0x7f20f415e830 0x7f20f406dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:06.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.576+0000 7f20f2000640 1 -- 172.21.15.96:0/3922510931 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1761754090 0 0) 0x7f20f406e190 con 0x7f20f40b3510 2024-05-28T06:20:06.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.576+0000 7f20f2000640 1 -- 172.21.15.96:0/3922510931 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f20d8003520 con 0x7f20f40b3510 2024-05-28T06:20:06.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.577+0000 7f20f2000640 1 -- 172.21.15.96:0/3922510931 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2035919750 0 0) 0x7f20d8003520 con 0x7f20f40b3510 2024-05-28T06:20:06.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.577+0000 7f20f2000640 1 -- 172.21.15.96:0/3922510931 >> v2:172.21.15.123:3300/0 conn(0x7f20f415e830 msgr2=0x7f20f406dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:06.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.577+0000 7f20f2000640 1 --2- 172.21.15.96:0/3922510931 >> v2:172.21.15.123:3300/0 conn(0x7f20f415e830 0x7f20f406dbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:06.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.577+0000 7f20f2000640 1 -- 172.21.15.96:0/3922510931 >> v1:172.21.15.123:6789/0 conn(0x7f20f4072310 legacy=0x7f20f40ad130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:06.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.577+0000 7f20f2000640 1 -- 172.21.15.96:0/3922510931 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20f406ea30 con 0x7f20f40b3510 2024-05-28T06:20:06.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.577+0000 7f20f2000640 1 -- 172.21.15.96:0/3922510931 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f20e0001670 con 0x7f20f40b3510 2024-05-28T06:20:06.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.577+0000 7f20f2000640 1 -- 172.21.15.96:0/3922510931 >> v1:172.21.15.96:6789/0 conn(0x7f20f40b3510 legacy=0x7f20f4071c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:06.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.577+0000 7f20f2000640 1 --2- 172.21.15.96:0/3922510931 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f20d80039b0 0x7f20d8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:06.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.577+0000 7f20f2000640 1 --2- 172.21.15.96:0/3922510931 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f20d8004800 0x7f20d8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:06.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.577+0000 7f20f2000640 1 --2- 172.21.15.96:0/3922510931 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f20d80051c0 0x7f20d800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:06.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.577+0000 7f20f2000640 1 -- 172.21.15.96:0/3922510931 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f20f406e750 con 0x7f20d80039b0 2024-05-28T06:20:06.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.577+0000 7f20f2000640 1 -- 172.21.15.96:0/3922510931 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f20d800c060 con 0x7f20d8004800 2024-05-28T06:20:06.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.577+0000 7f20f2000640 1 -- 172.21.15.96:0/3922510931 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f20d800c260 con 0x7f20d80051c0 2024-05-28T06:20:06.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.577+0000 7f20f3400640 1 --2- 172.21.15.96:0/3922510931 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f20d80039b0 0x7f20d8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:06.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.577+0000 7f20f2a00640 1 --2- 172.21.15.96:0/3922510931 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f20d8004800 0x7f20d8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:06.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.577+0000 7f20f3400640 1 -- 172.21.15.96:0/3922510931 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f20d80051c0 msgr2=0x7f20d800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:06.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.577+0000 7f20f3400640 1 --2- 172.21.15.96:0/3922510931 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f20d80051c0 0x7f20d800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:06.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.578+0000 7f20f3400640 1 -- 172.21.15.96:0/3922510931 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f20d8004800 msgr2=0x7f20d8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:06.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.578+0000 7f20f3400640 1 --2- 172.21.15.96:0/3922510931 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f20d8004800 0x7f20d8004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:06.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.578+0000 7f20f3400640 1 -- 172.21.15.96:0/3922510931 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20d8003520 con 0x7f20d80039b0 2024-05-28T06:20:06.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.578+0000 7f20f3e00640 1 -- 172.21.15.96:0/3922510931 reap_dead start 2024-05-28T06:20:06.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.579+0000 7f20f3400640 1 --2- 172.21.15.96:0/3922510931 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f20d80039b0 0x7f20d8003dd0 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7f20e4009ac0 tx=0x7f20d8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab1b946612ebd430 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:06.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.579+0000 7f20f2000640 1 -- 172.21.15.96:0/3922510931 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20e4036070 con 0x7f20d80039b0 2024-05-28T06:20:06.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.579+0000 7f20f2000640 1 -- 172.21.15.96:0/3922510931 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20e4031400 con 0x7f20d80039b0 2024-05-28T06:20:06.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.579+0000 7f20f2000640 1 -- 172.21.15.96:0/3922510931 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20e4009690 con 0x7f20d80039b0 2024-05-28T06:20:06.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.579+0000 7f20fa800640 1 -- 172.21.15.96:0/3922510931 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f20d80039b0 msgr2=0x7f20d8003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:06.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.579+0000 7f20fa800640 1 --2- 172.21.15.96:0/3922510931 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f20d80039b0 0x7f20d8003dd0 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7f20e4009ac0 tx=0x7f20d8005830 comp rx=0 tx=0).stop 2024-05-28T06:20:06.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.579+0000 7f20fa800640 1 -- 172.21.15.96:0/3922510931 shutdown_connections 2024-05-28T06:20:06.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.579+0000 7f20fa800640 1 --2- 172.21.15.96:0/3922510931 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f20d80039b0 0x7f20d8003dd0 unknown :-1 s=CLOSED pgs=1564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:06.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.579+0000 7f20fa800640 1 -- 172.21.15.96:0/3922510931 >> 172.21.15.96:0/3922510931 conn(0x7f20f40b1e50 msgr2=0x7f20f40b2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:06.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.580+0000 7f20fa800640 1 -- 172.21.15.96:0/3922510931 shutdown_connections 2024-05-28T06:20:06.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.580+0000 7f20fa800640 1 -- 172.21.15.96:0/3922510931 wait complete. 2024-05-28T06:20:06.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.581+0000 7f20fa800640 1 Processor -- start 2024-05-28T06:20:06.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.581+0000 7f20fa800640 1 -- start start 2024-05-28T06:20:06.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.582+0000 7f20fa800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f20d80039b0 0x7f20f40cef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:06.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.582+0000 7f20fa800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f20f4161b90 0x7f20f41f81b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:06.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.582+0000 7f20fa800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f20f41527b0 0x7f20f41f86f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:06.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.582+0000 7f20fa800640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f20f40728c0 con 0x7f20d80039b0 2024-05-28T06:20:06.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.582+0000 7f20fa800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f20f415ec10 con 0x7f20f4161b90 2024-05-28T06:20:06.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.582+0000 7f20fa800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f20f4072a40 con 0x7f20f41527b0 2024-05-28T06:20:06.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.583+0000 7f20f3400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f20d80039b0 0x7f20f40cef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:06.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.583+0000 7f20f3400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f20d80039b0 0x7f20f40cef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:56006/0 (socket says 172.21.15.96:56006) 2024-05-28T06:20:06.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.583+0000 7f20f3400640 1 -- 172.21.15.96:0/2587476902 learned_addr learned my addr 172.21.15.96:0/2587476902 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:20:06.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.583+0000 7f20f2a00640 1 --2- 172.21.15.96:0/2587476902 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f20f4161b90 0x7f20f41f81b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:06.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.583+0000 7f20f3400640 1 -- 172.21.15.96:0/2587476902 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f20f41527b0 msgr2=0x7f20f41f86f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:06.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.583+0000 7f20f3400640 1 --2- 172.21.15.96:0/2587476902 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f20f41527b0 0x7f20f41f86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:06.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.583+0000 7f20f3400640 1 -- 172.21.15.96:0/2587476902 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f20f4161b90 msgr2=0x7f20f41f81b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:06.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.583+0000 7f20f3400640 1 --2- 172.21.15.96:0/2587476902 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f20f4161b90 0x7f20f41f81b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:06.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.583+0000 7f20f3400640 1 -- 172.21.15.96:0/2587476902 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20f409ae60 con 0x7f20d80039b0 2024-05-28T06:20:06.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.583+0000 7f20f2a00640 1 --2- 172.21.15.96:0/2587476902 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f20f4161b90 0x7f20f41f81b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:20:06.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.584+0000 7f20f3400640 1 --2- 172.21.15.96:0/2587476902 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f20d80039b0 0x7f20f40cef50 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7f20e4009be0 tx=0x7f20e4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:06.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.584+0000 7f20eb400640 1 -- 172.21.15.96:0/2587476902 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20e403f020 con 0x7f20d80039b0 2024-05-28T06:20:06.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.584+0000 7f20eb400640 1 -- 172.21.15.96:0/2587476902 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20e4037040 con 0x7f20d80039b0 2024-05-28T06:20:06.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.584+0000 7f20eb400640 1 -- 172.21.15.96:0/2587476902 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20e4036070 con 0x7f20d80039b0 2024-05-28T06:20:06.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.584+0000 7f20fa800640 1 -- 172.21.15.96:0/2587476902 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20f409b140 con 0x7f20d80039b0 2024-05-28T06:20:06.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.584+0000 7f20fa800640 1 -- 172.21.15.96:0/2587476902 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20f40cf490 con 0x7f20d80039b0 2024-05-28T06:20:06.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.585+0000 7f20fa800640 1 -- 172.21.15.96:0/2587476902 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20f409b660 con 0x7f20d80039b0 2024-05-28T06:20:06.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.586+0000 7f20eb400640 1 -- 172.21.15.96:0/2587476902 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 34) ==== 92927+0+0 (secure 0 0 0) 0x7f20e4042020 con 0x7f20d80039b0 2024-05-28T06:20:06.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.587+0000 7f20eb400640 1 --2- 172.21.15.96:0/2587476902 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f20d406ddc0 0x7f20d4070280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:06.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.588+0000 7f20eb400640 1 -- 172.21.15.96:0/2587476902 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 5880+0+0 (secure 0 0 0) 0x7f20e40aa820 con 0x7f20d80039b0 2024-05-28T06:20:06.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.588+0000 7f20f2a00640 1 --2- 172.21.15.96:0/2587476902 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f20d406ddc0 0x7f20d4070280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:06.586 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.589+0000 7f20f2a00640 1 --2- 172.21.15.96:0/2587476902 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f20d406ddc0 0x7f20d4070280 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f20e0001eb0 tx=0x7f20e0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:06.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.592+0000 7f20eb400640 1 -- 172.21.15.96:0/2587476902 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f20e4049030 con 0x7f20d80039b0 2024-05-28T06:20:06.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:06.793+0000 7f20fa800640 1 -- 172.21.15.96:0/2587476902 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f20f40689e0 con 0x7f20d80039b0 2024-05-28T06:20:07.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:07.805+0000 7f20eb400640 1 -- 172.21.15.96:0/2587476902 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v94) ==== 181+0+0 (secure 0 0 0) 0x7f20e407a6a0 con 0x7f20d80039b0 2024-05-28T06:20:07.803 INFO:teuthology.orchestra.run.smithi096.stderr:pool 'cephfs_metadata' removed 2024-05-28T06:20:07.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:07.808+0000 7f20fa800640 1 -- 172.21.15.96:0/2587476902 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f20d406ddc0 msgr2=0x7f20d4070280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:07.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:07.808+0000 7f20fa800640 1 --2- 172.21.15.96:0/2587476902 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f20d406ddc0 0x7f20d4070280 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f20e0001eb0 tx=0x7f20e0009040 comp rx=0 tx=0).stop 2024-05-28T06:20:07.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:07.808+0000 7f20fa800640 1 -- 172.21.15.96:0/2587476902 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f20d80039b0 msgr2=0x7f20f40cef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:07.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:07.808+0000 7f20fa800640 1 --2- 172.21.15.96:0/2587476902 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f20d80039b0 0x7f20f40cef50 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7f20e4009be0 tx=0x7f20e4004730 comp rx=0 tx=0).stop 2024-05-28T06:20:07.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:07.809+0000 7f20fa800640 1 -- 172.21.15.96:0/2587476902 shutdown_connections 2024-05-28T06:20:07.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:07.809+0000 7f20fa800640 1 --2- 172.21.15.96:0/2587476902 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f20d406ddc0 0x7f20d4070280 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:07.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:07.809+0000 7f20fa800640 1 --2- 172.21.15.96:0/2587476902 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f20f41527b0 0x7f20f41f86f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:07.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:07.809+0000 7f20fa800640 1 --2- 172.21.15.96:0/2587476902 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f20f4161b90 0x7f20f41f81b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:07.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:07.809+0000 7f20fa800640 1 --2- 172.21.15.96:0/2587476902 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f20d80039b0 0x7f20f40cef50 unknown :-1 s=CLOSED pgs=1565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:07.807 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:07.809+0000 7f20fa800640 1 -- 172.21.15.96:0/2587476902 >> 172.21.15.96:0/2587476902 conn(0x7f20f40b1e50 msgr2=0x7f20f40cf730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:07.807 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:07.809+0000 7f20fa800640 1 -- 172.21.15.96:0/2587476902 shutdown_connections 2024-05-28T06:20:07.807 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:07.809+0000 7f20fa800640 1 -- 172.21.15.96:0/2587476902 wait complete. 2024-05-28T06:20:07.831 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-05-28T06:20:08.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:08.032+0000 7fc5c7600640 1 Processor -- start 2024-05-28T06:20:08.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:08.033+0000 7fc5c7600640 1 -- start start 2024-05-28T06:20:08.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:08.034+0000 7fc5c7600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fc5c015b340 0x7fc5c006f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:08.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:08.034+0000 7fc5c7600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fc5c015b720 0x7fc5c015bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:08.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:08.034+0000 7fc5c7600640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5c006fa90 con 0x7fc5c015c0f0 2024-05-28T06:20:08.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:08.034+0000 7fc5c7600640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fc5c00af9b0 con 0x7fc5c015b340 2024-05-28T06:20:08.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:08.034+0000 7fc5c7600640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fc5c00aeda0 con 0x7fc5c015b720 2024-05-28T06:20:08.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:08.034+0000 7fc5c4c00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fc5c015b340 0x7fc5c006f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:08.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:08.034+0000 7fc5c5600640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fc5c015c0f0 0x7fc5c015f0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:55784/0 (socket says 172.21.15.96:55784) 2024-05-28T06:20:08.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:08.034+0000 7fc5c5600640 1 -- 172.21.15.96:0/89748227 learned_addr learned my addr 172.21.15.96:0/89748227 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:20:08.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:08.035+0000 7fc5b7e00640 1 --2- 172.21.15.96:0/89748227 >> v2:172.21.15.123:3300/0 conn(0x7fc5c015b720 0x7fc5c015bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:08.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:08.035+0000 7fc5c4c00640 1 -- 172.21.15.96:0/89748227 >> v2:172.21.15.123:3300/0 conn(0x7fc5c015b720 msgr2=0x7fc5c015bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:08.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:08.035+0000 7fc5c4c00640 1 --2- 172.21.15.96:0/89748227 >> v2:172.21.15.123:3300/0 conn(0x7fc5c015b720 0x7fc5c015bbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:08.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:08.035+0000 7fc5c4c00640 1 -- 172.21.15.96:0/89748227 >> v1:172.21.15.96:6789/0 conn(0x7fc5c015c0f0 legacy=0x7fc5c015f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:08.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:08.035+0000 7fc5c4c00640 1 -- 172.21.15.96:0/89748227 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5c00af080 con 0x7fc5c015b340 2024-05-28T06:20:08.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:08.035+0000 7fc5bfe00640 1 -- 172.21.15.96:0/89748227 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1369862910 0 0) 0x7fc5c006fa90 con 0x7fc5c015c0f0 2024-05-28T06:20:08.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:08.036+0000 7fc5c4c00640 1 --2- 172.21.15.96:0/89748227 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc5c015b340 0x7fc5c006f550 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7fc5c015cc20 tx=0x7fc5b002a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ae0052a0766ab6b server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:08.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:08.036+0000 7fc5bfe00640 1 -- 172.21.15.96:0/89748227 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b0037070 con 0x7fc5c015b340 2024-05-28T06:20:08.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:08.037+0000 7fc5bfe00640 1 -- 172.21.15.96:0/89748227 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5b0032440 con 0x7fc5c015b340 2024-05-28T06:20:08.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:08.037+0000 7fc5bfe00640 1 -- 172.21.15.96:0/89748227 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b0032810 con 0x7fc5c015b340 2024-05-28T06:20:08.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:08.037+0000 7fc5c7600640 1 -- 172.21.15.96:0/89748227 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc5c015b340 msgr2=0x7fc5c006f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:08.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:08.037+0000 7fc5c7600640 1 --2- 172.21.15.96:0/89748227 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc5c015b340 0x7fc5c006f550 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7fc5c015cc20 tx=0x7fc5b002a1a0 comp rx=0 tx=0).stop 2024-05-28T06:20:08.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:08.037+0000 7fc5c7600640 1 -- 172.21.15.96:0/89748227 shutdown_connections 2024-05-28T06:20:08.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:08.037+0000 7fc5c7600640 1 --2- 172.21.15.96:0/89748227 >> v2:172.21.15.123:3300/0 conn(0x7fc5c015b720 0x7fc5c015bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:08.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:08.037+0000 7fc5c7600640 1 --2- 172.21.15.96:0/89748227 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc5c015b340 0x7fc5c006f550 unknown :-1 s=CLOSED pgs=1567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:08.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:08.037+0000 7fc5c7600640 1 -- 172.21.15.96:0/89748227 >> 172.21.15.96:0/89748227 conn(0x7fc5c00b1e50 msgr2=0x7fc5c00b2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:08.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:08.037+0000 7fc5c7600640 1 -- 172.21.15.96:0/89748227 shutdown_connections 2024-05-28T06:20:08.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:08.037+0000 7fc5c7600640 1 -- 172.21.15.96:0/89748227 wait complete. 2024-05-28T06:20:08.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:08.039+0000 7fc5c7600640 1 Processor -- start 2024-05-28T06:20:08.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:08.039+0000 7fc5c7600640 1 -- start start 2024-05-28T06:20:08.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:08.040+0000 7fc5c7600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc5c015b340 0x7fc5c0155a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:08.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:08.040+0000 7fc5c7600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc5c015b720 0x7fc5c00cedd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:08.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:08.040+0000 7fc5c7600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5c015c0f0 0x7fc5c00cf310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:08.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:08.040+0000 7fc5c7600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc5c00aea30 con 0x7fc5c015b340 2024-05-28T06:20:08.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:08.040+0000 7fc5c7600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc5c00af640 con 0x7fc5c015c0f0 2024-05-28T06:20:08.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:08.040+0000 7fc5c7600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fc5c015c8c0 con 0x7fc5c015b720 2024-05-28T06:20:08.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:08.040+0000 7fc5c4c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc5c015b340 0x7fc5c0155a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:08.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:08.040+0000 7fc5c4c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc5c015b340 0x7fc5c0155a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:56020/0 (socket says 172.21.15.96:56020) 2024-05-28T06:20:08.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:08.040+0000 7fc5c4c00640 1 -- 172.21.15.96:0/955922389 learned_addr learned my addr 172.21.15.96:0/955922389 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:20:08.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:08.041+0000 7fc5c4c00640 1 -- 172.21.15.96:0/955922389 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc5c015b720 msgr2=0x7fc5c00cedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:08.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:08.041+0000 7fc5c4c00640 1 --2- 172.21.15.96:0/955922389 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc5c015b720 0x7fc5c00cedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:08.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:08.041+0000 7fc5c4c00640 1 -- 172.21.15.96:0/955922389 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5c015c0f0 msgr2=0x7fc5c00cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:08.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:08.041+0000 7fc5c4c00640 1 --2- 172.21.15.96:0/955922389 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5c015c0f0 0x7fc5c00cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:08.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:08.041+0000 7fc5c4c00640 1 -- 172.21.15.96:0/955922389 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5c01574d0 con 0x7fc5c015b340 2024-05-28T06:20:08.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:08.041+0000 7fc5c4c00640 1 --2- 172.21.15.96:0/955922389 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc5c015b340 0x7fc5c0155a50 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7fc5b0000c00 tx=0x7fc5b00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:08.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:08.041+0000 7fc5bd600640 1 -- 172.21.15.96:0/955922389 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b0037040 con 0x7fc5c015b340 2024-05-28T06:20:08.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:08.041+0000 7fc5bd600640 1 -- 172.21.15.96:0/955922389 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5b0042070 con 0x7fc5c015b340 2024-05-28T06:20:08.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:08.041+0000 7fc5c7600640 1 -- 172.21.15.96:0/955922389 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5c01577b0 con 0x7fc5c015b340 2024-05-28T06:20:08.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:08.041+0000 7fc5bd600640 1 -- 172.21.15.96:0/955922389 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b0030bb0 con 0x7fc5c015b340 2024-05-28T06:20:08.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:08.042+0000 7fc5c7600640 1 -- 172.21.15.96:0/955922389 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5c0157d70 con 0x7fc5c015b340 2024-05-28T06:20:08.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:08.043+0000 7fc5c7600640 1 -- 172.21.15.96:0/955922389 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5c006fa80 con 0x7fc5c015b340 2024-05-28T06:20:08.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:08.044+0000 7fc5bd600640 1 -- 172.21.15.96:0/955922389 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 34) ==== 92927+0+0 (secure 0 0 0) 0x7fc5b0047460 con 0x7fc5c015b340 2024-05-28T06:20:08.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:08.044+0000 7fc5bd600640 1 --2- 172.21.15.96:0/955922389 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc59c06ddc0 0x7fc59c070280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:08.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:08.045+0000 7fc5bd600640 1 -- 172.21.15.96:0/955922389 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 5442+0+0 (secure 0 0 0) 0x7fc5b00ab860 con 0x7fc5c015b340 2024-05-28T06:20:08.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:08.045+0000 7fc5b7e00640 1 --2- 172.21.15.96:0/955922389 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc59c06ddc0 0x7fc59c070280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:08.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:08.046+0000 7fc5b7e00640 1 --2- 172.21.15.96:0/955922389 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc59c06ddc0 0x7fc59c070280 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fc5a8006fd0 tx=0x7fc5a8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:08.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:08.049+0000 7fc5bd600640 1 -- 172.21.15.96:0/955922389 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc5c006fa80 con 0x7fc5c015b340 2024-05-28T06:20:08.249 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:08.252+0000 7fc5c7600640 1 -- 172.21.15.96:0/955922389 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fc5c00689e0 con 0x7fc5c015b340 2024-05-28T06:20:08.816 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:08.819+0000 7fc5bd600640 1 -- 172.21.15.96:0/955922389 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v95) ==== 169+0+0 (secure 0 0 0) 0x7fc5b0032e20 con 0x7fc5c015b340 2024-05-28T06:20:08.817 INFO:teuthology.orchestra.run.smithi096.stderr:pool 'cephfs_data' removed 2024-05-28T06:20:08.820 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:08.822+0000 7fc5c7600640 1 -- 172.21.15.96:0/955922389 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc59c06ddc0 msgr2=0x7fc59c070280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:08.820 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:08.822+0000 7fc5c7600640 1 --2- 172.21.15.96:0/955922389 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc59c06ddc0 0x7fc59c070280 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fc5a8006fd0 tx=0x7fc5a8008040 comp rx=0 tx=0).stop 2024-05-28T06:20:08.820 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:08.822+0000 7fc5c7600640 1 -- 172.21.15.96:0/955922389 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc5c015b340 msgr2=0x7fc5c0155a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:08.820 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:08.822+0000 7fc5c7600640 1 --2- 172.21.15.96:0/955922389 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc5c015b340 0x7fc5c0155a50 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7fc5b0000c00 tx=0x7fc5b00046c0 comp rx=0 tx=0).stop 2024-05-28T06:20:08.820 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:08.823+0000 7fc5c7600640 1 -- 172.21.15.96:0/955922389 shutdown_connections 2024-05-28T06:20:08.820 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:08.823+0000 7fc5c7600640 1 --2- 172.21.15.96:0/955922389 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc59c06ddc0 0x7fc59c070280 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:08.820 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:08.823+0000 7fc5c7600640 1 --2- 172.21.15.96:0/955922389 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5c015c0f0 0x7fc5c00cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:08.820 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:08.823+0000 7fc5c7600640 1 --2- 172.21.15.96:0/955922389 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc5c015b720 0x7fc5c00cedd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:08.820 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:08.823+0000 7fc5c7600640 1 --2- 172.21.15.96:0/955922389 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc5c015b340 0x7fc5c0155a50 unknown :-1 s=CLOSED pgs=1568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:08.820 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:08.823+0000 7fc5c7600640 1 -- 172.21.15.96:0/955922389 >> 172.21.15.96:0/955922389 conn(0x7fc5c00b1e50 msgr2=0x7fc5c015f620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:08.821 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:08.823+0000 7fc5c7600640 1 -- 172.21.15.96:0/955922389 shutdown_connections 2024-05-28T06:20:08.821 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:08.823+0000 7fc5c7600640 1 -- 172.21.15.96:0/955922389 wait complete. 2024-05-28T06:20:08.847 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-05-28T06:20:09.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.029+0000 7f3df1600640 1 Processor -- start 2024-05-28T06:20:09.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.029+0000 7f3df1600640 1 -- start start 2024-05-28T06:20:09.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.031+0000 7f3df1600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f3dec161a90 0x7f3dec06e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:09.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.031+0000 7f3df1600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3dec161e70 0x7f3dec0b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:09.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.031+0000 7f3df1600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3dec0b3d40 con 0x7f3dec06e9b0 2024-05-28T06:20:09.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.031+0000 7f3df1600640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f3dec0b4020 con 0x7f3dec161a90 2024-05-28T06:20:09.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.031+0000 7f3df1600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f3dec0b4300 con 0x7f3dec161e70 2024-05-28T06:20:09.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.031+0000 7f3deaa00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f3dec161a90 0x7f3dec06e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:09.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.031+0000 7f3deaa00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f3dec161a90 0x7f3dec06e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:50872/0 (socket says 172.21.15.96:50872) 2024-05-28T06:20:09.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.031+0000 7f3deaa00640 1 -- 172.21.15.96:0/1508421553 learned_addr learned my addr 172.21.15.96:0/1508421553 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:20:09.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.031+0000 7f3deaa00640 1 -- 172.21.15.96:0/1508421553 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3dec161e70 msgr2=0x7f3dec0b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:09.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.031+0000 7f3deaa00640 1 --2- 172.21.15.96:0/1508421553 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3dec161e70 0x7f3dec0b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:09.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.031+0000 7f3deaa00640 1 -- 172.21.15.96:0/1508421553 >> v1:172.21.15.123:6789/0 conn(0x7f3dec06e9b0 legacy=0x7f3dec0ad2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:09.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.031+0000 7f3deaa00640 1 -- 172.21.15.96:0/1508421553 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dec06dc70 con 0x7f3dec161a90 2024-05-28T06:20:09.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.032+0000 7f3deaa00640 1 --2- 172.21.15.96:0/1508421553 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3dec161a90 0x7f3dec06e470 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f3ddc009870 tx=0x7f3ddc02a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ffc7b5d921d360c server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:09.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.033+0000 7f3de9600640 1 -- 172.21.15.96:0/1508421553 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ddc037070 con 0x7f3dec161a90 2024-05-28T06:20:09.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.033+0000 7f3de9600640 1 -- 172.21.15.96:0/1508421553 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ddc032440 con 0x7f3dec161a90 2024-05-28T06:20:09.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.033+0000 7f3de9600640 1 -- 172.21.15.96:0/1508421553 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ddc032810 con 0x7f3dec161a90 2024-05-28T06:20:09.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.033+0000 7f3df1600640 1 -- 172.21.15.96:0/1508421553 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3dec161a90 msgr2=0x7f3dec06e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:09.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.033+0000 7f3df1600640 1 --2- 172.21.15.96:0/1508421553 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3dec161a90 0x7f3dec06e470 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f3ddc009870 tx=0x7f3ddc02a210 comp rx=0 tx=0).stop 2024-05-28T06:20:09.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.034+0000 7f3df1600640 1 -- 172.21.15.96:0/1508421553 shutdown_connections 2024-05-28T06:20:09.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.034+0000 7f3df1600640 1 --2- 172.21.15.96:0/1508421553 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3dec161e70 0x7f3dec0b32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:09.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.034+0000 7f3df1600640 1 --2- 172.21.15.96:0/1508421553 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3dec161a90 0x7f3dec06e470 unknown :-1 s=CLOSED pgs=1569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:09.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.034+0000 7f3df1600640 1 -- 172.21.15.96:0/1508421553 >> 172.21.15.96:0/1508421553 conn(0x7f3dec06bd20 msgr2=0x7f3dec06c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:09.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.034+0000 7f3df1600640 1 -- 172.21.15.96:0/1508421553 shutdown_connections 2024-05-28T06:20:09.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.034+0000 7f3df1600640 1 -- 172.21.15.96:0/1508421553 wait complete. 2024-05-28T06:20:09.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.035+0000 7f3df1600640 1 Processor -- start 2024-05-28T06:20:09.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.036+0000 7f3df1600640 1 -- start start 2024-05-28T06:20:09.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.036+0000 7f3df1600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3dec06e9b0 0x7f3dec1ea240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:09.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.036+0000 7f3df1600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3dec161a90 0x7f3dec154280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:09.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.036+0000 7f3df1600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3dec161e70 0x7f3dec1547c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:09.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.036+0000 7f3df1600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f3dec0ad9b0 con 0x7f3dec161e70 2024-05-28T06:20:09.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.036+0000 7f3df1600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f3dec0b5020 con 0x7f3dec06e9b0 2024-05-28T06:20:09.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.036+0000 7f3df1600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f3dec0adb30 con 0x7f3dec161a90 2024-05-28T06:20:09.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.037+0000 7f3deb400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3dec161e70 0x7f3dec1547c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:09.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.037+0000 7f3deb400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3dec161e70 0x7f3dec1547c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:50882/0 (socket says 172.21.15.96:50882) 2024-05-28T06:20:09.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.037+0000 7f3deb400640 1 -- 172.21.15.96:0/486228005 learned_addr learned my addr 172.21.15.96:0/486228005 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:20:09.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.037+0000 7f3dea000640 1 --2- 172.21.15.96:0/486228005 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3dec161a90 0x7f3dec154280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:09.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.037+0000 7f3deaa00640 1 --2- 172.21.15.96:0/486228005 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3dec06e9b0 0x7f3dec1ea240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:09.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.037+0000 7f3deb400640 1 -- 172.21.15.96:0/486228005 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3dec161a90 msgr2=0x7f3dec154280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:09.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.037+0000 7f3deb400640 1 --2- 172.21.15.96:0/486228005 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3dec161a90 0x7f3dec154280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:09.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.037+0000 7f3deb400640 1 -- 172.21.15.96:0/486228005 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3dec06e9b0 msgr2=0x7f3dec1ea240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:09.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.037+0000 7f3deb400640 1 --2- 172.21.15.96:0/486228005 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3dec06e9b0 0x7f3dec1ea240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:09.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.037+0000 7f3deb400640 1 -- 172.21.15.96:0/486228005 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dec09a6e0 con 0x7f3dec161e70 2024-05-28T06:20:09.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.037+0000 7f3dea000640 1 --2- 172.21.15.96:0/486228005 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3dec161a90 0x7f3dec154280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:20:09.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.037+0000 7f3deb400640 1 --2- 172.21.15.96:0/486228005 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3dec161e70 0x7f3dec1547c0 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7f3de400b990 tx=0x7f3de400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:09.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.038+0000 7f3de2a00640 1 -- 172.21.15.96:0/486228005 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3de4019070 con 0x7f3dec161e70 2024-05-28T06:20:09.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.038+0000 7f3de2a00640 1 -- 172.21.15.96:0/486228005 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3de400cba0 con 0x7f3dec161e70 2024-05-28T06:20:09.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.038+0000 7f3de2a00640 1 -- 172.21.15.96:0/486228005 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3de4014440 con 0x7f3dec161e70 2024-05-28T06:20:09.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.038+0000 7f3df1600640 1 -- 172.21.15.96:0/486228005 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3dec09d6a0 con 0x7f3dec161e70 2024-05-28T06:20:09.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.038+0000 7f3df1600640 1 -- 172.21.15.96:0/486228005 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3dec09dc60 con 0x7f3dec161e70 2024-05-28T06:20:09.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.039+0000 7f3df1600640 1 -- 172.21.15.96:0/486228005 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3dec0689e0 con 0x7f3dec161e70 2024-05-28T06:20:09.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.040+0000 7f3de2a00640 1 -- 172.21.15.96:0/486228005 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 34) ==== 92927+0+0 (secure 0 0 0) 0x7f3de401c070 con 0x7f3dec161e70 2024-05-28T06:20:09.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.041+0000 7f3de2a00640 1 --2- 172.21.15.96:0/486228005 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3dcc06ddc0 0x7f3dcc070280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:09.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.041+0000 7f3de2a00640 1 -- 172.21.15.96:0/486228005 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 4999+0+0 (secure 0 0 0) 0x7f3de408a5e0 con 0x7f3dec161e70 2024-05-28T06:20:09.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.042+0000 7f3deaa00640 1 --2- 172.21.15.96:0/486228005 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3dcc06ddc0 0x7f3dcc070280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:09.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.043+0000 7f3deaa00640 1 --2- 172.21.15.96:0/486228005 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3dcc06ddc0 0x7f3dcc070280 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f3ddc034000 tx=0x7f3ddc02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:09.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.045+0000 7f3de2a00640 1 -- 172.21.15.96:0/486228005 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3dec09a6e0 con 0x7f3dec161e70 2024-05-28T06:20:09.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.245+0000 7f3df1600640 1 -- 172.21.15.96:0/486228005 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7f3dec1ea780 con 0x7f3dec161e70 2024-05-28T06:20:09.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.251+0000 7f3de2a00640 1 -- 172.21.15.96:0/486228005 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v23) ==== 104+0+0 (secure 0 0 0) 0x7f3de405a460 con 0x7f3dec161e70 2024-05-28T06:20:09.257 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.260+0000 7f3df1600640 1 -- 172.21.15.96:0/486228005 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3dcc06ddc0 msgr2=0x7f3dcc070280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:09.257 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.260+0000 7f3df1600640 1 --2- 172.21.15.96:0/486228005 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3dcc06ddc0 0x7f3dcc070280 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f3ddc034000 tx=0x7f3ddc02e000 comp rx=0 tx=0).stop 2024-05-28T06:20:09.257 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.260+0000 7f3df1600640 1 -- 172.21.15.96:0/486228005 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3dec161e70 msgr2=0x7f3dec1547c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:09.257 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.260+0000 7f3df1600640 1 --2- 172.21.15.96:0/486228005 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3dec161e70 0x7f3dec1547c0 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7f3de400b990 tx=0x7f3de400be50 comp rx=0 tx=0).stop 2024-05-28T06:20:09.257 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.260+0000 7f3df1600640 1 -- 172.21.15.96:0/486228005 shutdown_connections 2024-05-28T06:20:09.257 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.260+0000 7f3df1600640 1 --2- 172.21.15.96:0/486228005 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3dcc06ddc0 0x7f3dcc070280 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:09.257 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.260+0000 7f3df1600640 1 --2- 172.21.15.96:0/486228005 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3dec161e70 0x7f3dec1547c0 unknown :-1 s=CLOSED pgs=1570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:09.258 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.260+0000 7f3df1600640 1 --2- 172.21.15.96:0/486228005 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3dec161a90 0x7f3dec154280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:09.258 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.260+0000 7f3df1600640 1 --2- 172.21.15.96:0/486228005 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3dec06e9b0 0x7f3dec1ea240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:09.258 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.260+0000 7f3df1600640 1 -- 172.21.15.96:0/486228005 >> 172.21.15.96:0/486228005 conn(0x7f3dec06bd20 msgr2=0x7f3dec0b1510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:09.258 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.261+0000 7f3df1600640 1 -- 172.21.15.96:0/486228005 shutdown_connections 2024-05-28T06:20:09.258 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.261+0000 7f3df1600640 1 -- 172.21.15.96:0/486228005 wait complete. 2024-05-28T06:20:09.286 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestCloneProgressReporter.test_stats_are_printed' 2024-05-28T06:20:09.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.456+0000 7fadc8400640 1 Processor -- start 2024-05-28T06:20:09.454 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.457+0000 7fadc8400640 1 -- start start 2024-05-28T06:20:09.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.458+0000 7fadc8400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fadc00b4e10 0x7fadc00cde30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:09.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.458+0000 7fadc8400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fadc00ce370 0x7fadc00d0590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:09.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.458+0000 7fadc8400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fadc00d0ad0 0x7fadc0099090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:09.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.458+0000 7fadc8400640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fadc00d0f40 con 0x7fadc00ce370 2024-05-28T06:20:09.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.458+0000 7fadc8400640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fadc00b51f0 con 0x7fadc00b4e10 2024-05-28T06:20:09.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.458+0000 7fadc8400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fadc00ad1d0 con 0x7fadc00d0ad0 2024-05-28T06:20:09.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.458+0000 7fadc5000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fadc00ce370 0x7fadc00d0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:09.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.458+0000 7fadc5000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fadc00ce370 0x7fadc00d0590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:50884/0 (socket says 172.21.15.96:50884) 2024-05-28T06:20:09.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.458+0000 7fadc5000640 1 -- 172.21.15.96:0/543278091 learned_addr learned my addr 172.21.15.96:0/543278091 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:20:09.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.459+0000 7fadc6400640 1 --2- 172.21.15.96:0/543278091 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fadc00d0ad0 0x7fadc0099090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:09.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.459+0000 7fadc5a00640 1 --2- 172.21.15.96:0/543278091 >> v2:172.21.15.123:3300/0 conn(0x7fadc00b4e10 0x7fadc00cde30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:09.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.459+0000 7fadc5000640 1 -- 172.21.15.96:0/543278091 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fadc00d0ad0 msgr2=0x7fadc0099090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:09.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.459+0000 7fadc5000640 1 --2- 172.21.15.96:0/543278091 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fadc00d0ad0 0x7fadc0099090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:09.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.459+0000 7fadc5000640 1 -- 172.21.15.96:0/543278091 >> v2:172.21.15.123:3300/0 conn(0x7fadc00b4e10 msgr2=0x7fadc00cde30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:09.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.459+0000 7fadc5000640 1 --2- 172.21.15.96:0/543278091 >> v2:172.21.15.123:3300/0 conn(0x7fadc00b4e10 0x7fadc00cde30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:09.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.459+0000 7fadc5000640 1 -- 172.21.15.96:0/543278091 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadc00ad470 con 0x7fadc00ce370 2024-05-28T06:20:09.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.460+0000 7fadc5000640 1 --2- 172.21.15.96:0/543278091 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fadc00ce370 0x7fadc00d0590 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7fadc006f080 tx=0x7fadb002a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f14e3343c4da6750 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:09.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.460+0000 7fadbfe00640 1 -- 172.21.15.96:0/543278091 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadb0037070 con 0x7fadc00ce370 2024-05-28T06:20:09.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.460+0000 7fadbfe00640 1 -- 172.21.15.96:0/543278091 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fadb0032440 con 0x7fadc00ce370 2024-05-28T06:20:09.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.460+0000 7fadbfe00640 1 -- 172.21.15.96:0/543278091 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadb0032810 con 0x7fadc00ce370 2024-05-28T06:20:09.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.461+0000 7fadc8400640 1 -- 172.21.15.96:0/543278091 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fadc00ce370 msgr2=0x7fadc00d0590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:09.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.461+0000 7fadc8400640 1 --2- 172.21.15.96:0/543278091 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fadc00ce370 0x7fadc00d0590 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7fadc006f080 tx=0x7fadb002a140 comp rx=0 tx=0).stop 2024-05-28T06:20:09.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.461+0000 7fadc8400640 1 -- 172.21.15.96:0/543278091 shutdown_connections 2024-05-28T06:20:09.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.461+0000 7fadc8400640 1 --2- 172.21.15.96:0/543278091 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fadc00d0ad0 0x7fadc0099090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:09.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.461+0000 7fadc8400640 1 --2- 172.21.15.96:0/543278091 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fadc00ce370 0x7fadc00d0590 unknown :-1 s=CLOSED pgs=1571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:09.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.461+0000 7fadc8400640 1 --2- 172.21.15.96:0/543278091 >> v2:172.21.15.123:3300/0 conn(0x7fadc00b4e10 0x7fadc00cde30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:09.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.461+0000 7fadc8400640 1 -- 172.21.15.96:0/543278091 >> 172.21.15.96:0/543278091 conn(0x7fadc006bae0 msgr2=0x7fadc006bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:09.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.461+0000 7fadc8400640 1 -- 172.21.15.96:0/543278091 shutdown_connections 2024-05-28T06:20:09.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.461+0000 7fadc8400640 1 -- 172.21.15.96:0/543278091 wait complete. 2024-05-28T06:20:09.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.463+0000 7fadc8400640 1 Processor -- start 2024-05-28T06:20:09.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.463+0000 7fadc8400640 1 -- start start 2024-05-28T06:20:09.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.464+0000 7fadc8400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fadc00b4e10 0x7fadc00c5990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:09.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.464+0000 7fadc8400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fadc00ce370 0x7fadc00c6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:09.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.464+0000 7fadc8400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fadc00d0ad0 0x7fadc01567f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:09.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.464+0000 7fadc8400640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fadc00cea00 con 0x7fadc00d0ad0 2024-05-28T06:20:09.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.464+0000 7fadc8400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fadc00ce880 con 0x7fadc00b4e10 2024-05-28T06:20:09.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.464+0000 7fadc8400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fadc00b3150 con 0x7fadc00ce370 2024-05-28T06:20:09.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.465+0000 7fadc6400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fadc00d0ad0 0x7fadc01567f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:09.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.465+0000 7fadc6400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fadc00d0ad0 0x7fadc01567f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:50888/0 (socket says 172.21.15.96:50888) 2024-05-28T06:20:09.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.465+0000 7fadc6400640 1 -- 172.21.15.96:0/410404930 learned_addr learned my addr 172.21.15.96:0/410404930 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:20:09.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.465+0000 7fadc5000640 1 --2- 172.21.15.96:0/410404930 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fadc00ce370 0x7fadc00c6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:09.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.465+0000 7fadc5a00640 1 --2- 172.21.15.96:0/410404930 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fadc00b4e10 0x7fadc00c5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:09.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.465+0000 7fadc6400640 1 -- 172.21.15.96:0/410404930 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fadc00ce370 msgr2=0x7fadc00c6170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:09.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.465+0000 7fadc6400640 1 --2- 172.21.15.96:0/410404930 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fadc00ce370 0x7fadc00c6170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:09.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.465+0000 7fadc6400640 1 -- 172.21.15.96:0/410404930 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fadc00b4e10 msgr2=0x7fadc00c5990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:09.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.465+0000 7fadc6400640 1 --2- 172.21.15.96:0/410404930 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fadc00b4e10 0x7fadc00c5990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:09.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.465+0000 7fadc6400640 1 -- 172.21.15.96:0/410404930 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadc00c6bd0 con 0x7fadc00d0ad0 2024-05-28T06:20:09.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.465+0000 7fadc5000640 1 --2- 172.21.15.96:0/410404930 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fadc00ce370 0x7fadc00c6170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:20:09.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.465+0000 7fadc6400640 1 --2- 172.21.15.96:0/410404930 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fadc00d0ad0 0x7fadc01567f0 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7fadb8007e70 tx=0x7fadb800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:09.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.465+0000 7fadbd600640 1 -- 172.21.15.96:0/410404930 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadb8014620 con 0x7fadc00d0ad0 2024-05-28T06:20:09.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.466+0000 7fadbd600640 1 -- 172.21.15.96:0/410404930 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fadb8019070 con 0x7fadc00d0ad0 2024-05-28T06:20:09.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.466+0000 7fadbd600640 1 -- 172.21.15.96:0/410404930 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadb8014a20 con 0x7fadc00d0ad0 2024-05-28T06:20:09.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.466+0000 7fadc8400640 1 -- 172.21.15.96:0/410404930 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fadc01f5660 con 0x7fadc00d0ad0 2024-05-28T06:20:09.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.466+0000 7fadc8400640 1 -- 172.21.15.96:0/410404930 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fadc015bc90 con 0x7fadc00d0ad0 2024-05-28T06:20:09.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.467+0000 7fadc8400640 1 -- 172.21.15.96:0/410404930 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fadc00689e0 con 0x7fadc00d0ad0 2024-05-28T06:20:09.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.468+0000 7fadbd600640 1 -- 172.21.15.96:0/410404930 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 34) ==== 92927+0+0 (secure 0 0 0) 0x7fadb8014bc0 con 0x7fadc00d0ad0 2024-05-28T06:20:09.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.469+0000 7fadbd600640 1 --2- 172.21.15.96:0/410404930 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fada406ddc0 0x7fada4070280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:09.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.469+0000 7fadbd600640 1 -- 172.21.15.96:0/410404930 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 4999+0+0 (secure 0 0 0) 0x7fadb808b5b0 con 0x7fadc00d0ad0 2024-05-28T06:20:09.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.469+0000 7fadc5a00640 1 --2- 172.21.15.96:0/410404930 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fada406ddc0 0x7fada4070280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:09.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.471+0000 7fadc5a00640 1 --2- 172.21.15.96:0/410404930 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fada406ddc0 0x7fada4070280 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fadb4006fd0 tx=0x7fadb4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:09.469 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.473+0000 7fadbd600640 1 -- 172.21.15.96:0/410404930 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fadb805b050 con 0x7fadc00d0ad0 2024-05-28T06:20:09.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.738+0000 7fadc8400640 1 -- 172.21.15.96:0/410404930 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgressReporter.test_stats_are_printed"]} v 0) -- 0x7fadc006f540 con 0x7fadc00d0ad0 2024-05-28T06:20:09.819 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.822+0000 7fadbd600640 1 -- 172.21.15.96:0/410404930 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgressReporter.test_stats_are_printed"]}]=0 v706) ==== 153+0+0 (secure 0 0 0) 0x7fadb805a950 con 0x7fadc00d0ad0 2024-05-28T06:20:09.822 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.825+0000 7fadc8400640 1 -- 172.21.15.96:0/410404930 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fada406ddc0 msgr2=0x7fada4070280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:09.822 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.825+0000 7fadc8400640 1 --2- 172.21.15.96:0/410404930 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fada406ddc0 0x7fada4070280 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fadb4006fd0 tx=0x7fadb4008040 comp rx=0 tx=0).stop 2024-05-28T06:20:09.822 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.825+0000 7fadc8400640 1 -- 172.21.15.96:0/410404930 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fadc00d0ad0 msgr2=0x7fadc01567f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:09.823 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.825+0000 7fadc8400640 1 --2- 172.21.15.96:0/410404930 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fadc00d0ad0 0x7fadc01567f0 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7fadb8007e70 tx=0x7fadb800e4b0 comp rx=0 tx=0).stop 2024-05-28T06:20:09.823 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.825+0000 7fadc8400640 1 -- 172.21.15.96:0/410404930 shutdown_connections 2024-05-28T06:20:09.823 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.825+0000 7fadc8400640 1 --2- 172.21.15.96:0/410404930 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fada406ddc0 0x7fada4070280 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:09.823 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.825+0000 7fadc8400640 1 --2- 172.21.15.96:0/410404930 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fadc00d0ad0 0x7fadc01567f0 unknown :-1 s=CLOSED pgs=1572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:09.823 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.825+0000 7fadc8400640 1 --2- 172.21.15.96:0/410404930 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fadc00ce370 0x7fadc00c6170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:09.823 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.825+0000 7fadc8400640 1 --2- 172.21.15.96:0/410404930 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fadc00b4e10 0x7fadc00c5990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:09.823 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.825+0000 7fadc8400640 1 -- 172.21.15.96:0/410404930 >> 172.21.15.96:0/410404930 conn(0x7fadc006bae0 msgr2=0x7fadc006dbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:09.823 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.825+0000 7fadc8400640 1 -- 172.21.15.96:0/410404930 shutdown_connections 2024-05-28T06:20:09.823 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:09.825+0000 7fadc8400640 1 -- 172.21.15.96:0/410404930 wait complete. 2024-05-28T06:20:09.842 INFO:tasks.cephfs_test_runner:Test that the command "ceph fs clone status" prints progress stats ... ok 2024-05-28T06:20:09.842 INFO:tasks.cephfs_test_runner:Starting test: test_when_1_clone_is_cancelled (tasks.cephfs.test_volumes.TestCloneProgressReporter) Test that the progress bar, that is printed for 1 ongoing clone job, is 2024-05-28T06:20:09.842 INFO:tasks.cephfs_test_runner:test_when_1_clone_is_cancelled (tasks.cephfs.test_volumes.TestCloneProgressReporter) 2024-05-28T06:20:09.843 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:20:10.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.032+0000 7fc7a4a00640 1 Processor -- start 2024-05-28T06:20:10.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.032+0000 7fc7a4a00640 1 -- start start 2024-05-28T06:20:10.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.034+0000 7fc7a4a00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fc7a0071bc0 0x7fc7a0072070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:10.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.034+0000 7fc7a4a00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7a006e6e0 con 0x7fc7a00b32c0 2024-05-28T06:20:10.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.034+0000 7fc7a4a00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7a006e9c0 con 0x7fc7a0158b80 2024-05-28T06:20:10.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.034+0000 7fc7a4a00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fc7a006eca0 con 0x7fc7a0071bc0 2024-05-28T06:20:10.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.034+0000 7fc79fe00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fc7a00b32c0 0x7fc7a00b36a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:33688/0 (socket says 172.21.15.96:33688) 2024-05-28T06:20:10.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.034+0000 7fc79fe00640 1 -- 172.21.15.96:0/1268526599 learned_addr learned my addr 172.21.15.96:0/1268526599 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:20:10.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.035+0000 7fc79ea00640 1 --2- 172.21.15.96:0/1268526599 >> v2:172.21.15.123:3300/0 conn(0x7fc7a0071bc0 0x7fc7a0072070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:10.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.035+0000 7fc79e000640 1 -- 172.21.15.96:0/1268526599 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2525940597 0 0) 0x7fc7a006e6e0 con 0x7fc7a00b32c0 2024-05-28T06:20:10.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.035+0000 7fc79e000640 1 -- 172.21.15.96:0/1268526599 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc784003520 con 0x7fc7a00b32c0 2024-05-28T06:20:10.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.035+0000 7fc79e000640 1 -- 172.21.15.96:0/1268526599 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 135540450 0 0) 0x7fc784003520 con 0x7fc7a00b32c0 2024-05-28T06:20:10.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.035+0000 7fc79e000640 1 -- 172.21.15.96:0/1268526599 >> v2:172.21.15.123:3300/0 conn(0x7fc7a0071bc0 msgr2=0x7fc7a0072070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:10.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.035+0000 7fc79e000640 1 --2- 172.21.15.96:0/1268526599 >> v2:172.21.15.123:3300/0 conn(0x7fc7a0071bc0 0x7fc7a0072070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:10.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.035+0000 7fc79e000640 1 -- 172.21.15.96:0/1268526599 >> v1:172.21.15.123:6789/0 conn(0x7fc7a0158b80 legacy=0x7fc7a006dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:10.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.035+0000 7fc79e000640 1 -- 172.21.15.96:0/1268526599 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7a006ef80 con 0x7fc7a00b32c0 2024-05-28T06:20:10.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.035+0000 7fc79e000640 1 -- 172.21.15.96:0/1268526599 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7fc7980019f0 con 0x7fc7a00b32c0 2024-05-28T06:20:10.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.035+0000 7fc79e000640 1 -- 172.21.15.96:0/1268526599 >> v1:172.21.15.96:6789/0 conn(0x7fc7a00b32c0 legacy=0x7fc7a00b36a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:10.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.035+0000 7fc79e000640 1 --2- 172.21.15.96:0/1268526599 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc784003a00 0x7fc784003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:10.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.035+0000 7fc79e000640 1 --2- 172.21.15.96:0/1268526599 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc784004800 0x7fc784004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:10.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.035+0000 7fc79e000640 1 --2- 172.21.15.96:0/1268526599 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc784005210 0x7fc78400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:10.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.035+0000 7fc79e000640 1 -- 172.21.15.96:0/1268526599 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc7a006eca0 con 0x7fc784003a00 2024-05-28T06:20:10.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.035+0000 7fc79e000640 1 -- 172.21.15.96:0/1268526599 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc78400c160 con 0x7fc784005210 2024-05-28T06:20:10.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.035+0000 7fc79e000640 1 -- 172.21.15.96:0/1268526599 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fc78400c360 con 0x7fc784004800 2024-05-28T06:20:10.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.035+0000 7fc79f400640 1 --2- 172.21.15.96:0/1268526599 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc784003a00 0x7fc784003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:10.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.036+0000 7fc79f400640 1 -- 172.21.15.96:0/1268526599 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc784004800 msgr2=0x7fc784004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:10.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.036+0000 7fc79f400640 1 --2- 172.21.15.96:0/1268526599 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc784004800 0x7fc784004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:10.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.036+0000 7fc79f400640 1 -- 172.21.15.96:0/1268526599 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc784005210 msgr2=0x7fc78400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:10.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.036+0000 7fc79f400640 1 --2- 172.21.15.96:0/1268526599 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc784005210 0x7fc78400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:10.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.036+0000 7fc79f400640 1 -- 172.21.15.96:0/1268526599 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc784003520 con 0x7fc784003a00 2024-05-28T06:20:10.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.036+0000 7fc79fe00640 1 -- 172.21.15.96:0/1268526599 reap_dead start 2024-05-28T06:20:10.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.037+0000 7fc79f400640 1 --2- 172.21.15.96:0/1268526599 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc784003a00 0x7fc784003e20 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7fc790001750 tx=0x7fc784005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf02a1822f6b9a68 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:10.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.037+0000 7fc79e000640 1 -- 172.21.15.96:0/1268526599 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc790001370 con 0x7fc784003a00 2024-05-28T06:20:10.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.037+0000 7fc79e000640 1 -- 172.21.15.96:0/1268526599 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7900052d0 con 0x7fc784003a00 2024-05-28T06:20:10.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.037+0000 7fc79e000640 1 -- 172.21.15.96:0/1268526599 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc79002abe0 con 0x7fc784003a00 2024-05-28T06:20:10.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.038+0000 7fc7a4a00640 1 -- 172.21.15.96:0/1268526599 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc784003a00 msgr2=0x7fc784003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:10.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.038+0000 7fc7a4a00640 1 --2- 172.21.15.96:0/1268526599 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc784003a00 0x7fc784003e20 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7fc790001750 tx=0x7fc784005830 comp rx=0 tx=0).stop 2024-05-28T06:20:10.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.038+0000 7fc7a4a00640 1 -- 172.21.15.96:0/1268526599 shutdown_connections 2024-05-28T06:20:10.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.038+0000 7fc7a4a00640 1 --2- 172.21.15.96:0/1268526599 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc784003a00 0x7fc784003e20 unknown :-1 s=CLOSED pgs=1574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:10.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.038+0000 7fc7a4a00640 1 -- 172.21.15.96:0/1268526599 >> 172.21.15.96:0/1268526599 conn(0x7fc7a00b1fc0 msgr2=0x7fc7a00b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:10.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.038+0000 7fc7a4a00640 1 -- 172.21.15.96:0/1268526599 shutdown_connections 2024-05-28T06:20:10.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.038+0000 7fc7a4a00640 1 -- 172.21.15.96:0/1268526599 wait complete. 2024-05-28T06:20:10.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.040+0000 7fc7a4a00640 1 Processor -- start 2024-05-28T06:20:10.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.040+0000 7fc7a4a00640 1 -- start start 2024-05-28T06:20:10.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.041+0000 7fc7a4a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc784003a00 0x7fc7a01576d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:10.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.041+0000 7fc7a4a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc7a015f460 0x7fc7a015f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:10.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.041+0000 7fc7a4a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc7a01f4920 0x7fc7a01f4da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:10.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.041+0000 7fc7a4a00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc7a006f780 con 0x7fc7a015f460 2024-05-28T06:20:10.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.041+0000 7fc7a4a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc7a00b2d30 con 0x7fc7a01f4920 2024-05-28T06:20:10.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.041+0000 7fc7a4a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fc7a01619c0 con 0x7fc784003a00 2024-05-28T06:20:10.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.041+0000 7fc79ea00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc7a015f460 0x7fc7a015f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:10.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.041+0000 7fc79ea00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc7a015f460 0x7fc7a015f8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:50914/0 (socket says 172.21.15.96:50914) 2024-05-28T06:20:10.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.041+0000 7fc79ea00640 1 -- 172.21.15.96:0/1433552947 learned_addr learned my addr 172.21.15.96:0/1433552947 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:20:10.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.041+0000 7fc79ea00640 1 -- 172.21.15.96:0/1433552947 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc784003a00 msgr2=0x7fc7a01576d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:10.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.042+0000 7fc79ea00640 1 --2- 172.21.15.96:0/1433552947 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc784003a00 0x7fc7a01576d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:10.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.042+0000 7fc79fe00640 1 --2- 172.21.15.96:0/1433552947 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc7a01f4920 0x7fc7a01f4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:10.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.042+0000 7fc79ea00640 1 -- 172.21.15.96:0/1433552947 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc7a01f4920 msgr2=0x7fc7a01f4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:10.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.042+0000 7fc79ea00640 1 --2- 172.21.15.96:0/1433552947 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc7a01f4920 0x7fc7a01f4da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:10.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.042+0000 7fc79ea00640 1 -- 172.21.15.96:0/1433552947 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7a0156630 con 0x7fc7a015f460 2024-05-28T06:20:10.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.042+0000 7fc79ea00640 1 --2- 172.21.15.96:0/1433552947 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc7a015f460 0x7fc7a015f8e0 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7fc78c00d530 tx=0x7fc78c00d9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:10.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.042+0000 7fc797400640 1 -- 172.21.15.96:0/1433552947 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc78c0125f0 con 0x7fc7a015f460 2024-05-28T06:20:10.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.042+0000 7fc797400640 1 -- 172.21.15.96:0/1433552947 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc78c019070 con 0x7fc7a015f460 2024-05-28T06:20:10.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.042+0000 7fc797400640 1 -- 172.21.15.96:0/1433552947 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc78c0129d0 con 0x7fc7a015f460 2024-05-28T06:20:10.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.042+0000 7fc7a4a00640 1 -- 172.21.15.96:0/1433552947 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7a0156910 con 0x7fc7a015f460 2024-05-28T06:20:10.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.042+0000 7fc7a4a00640 1 -- 172.21.15.96:0/1433552947 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7a0156ed0 con 0x7fc7a015f460 2024-05-28T06:20:10.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.043+0000 7fc7a4a00640 1 -- 172.21.15.96:0/1433552947 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7a00689e0 con 0x7fc7a015f460 2024-05-28T06:20:10.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.044+0000 7fc797400640 1 -- 172.21.15.96:0/1433552947 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 34) ==== 92927+0+0 (secure 0 0 0) 0x7fc78c012b70 con 0x7fc7a015f460 2024-05-28T06:20:10.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.045+0000 7fc797400640 1 --2- 172.21.15.96:0/1433552947 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc78006ddf0 0x7fc7800702b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:10.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.045+0000 7fc797400640 1 -- 172.21.15.96:0/1433552947 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 4999+0+0 (secure 0 0 0) 0x7fc78c08ab00 con 0x7fc7a015f460 2024-05-28T06:20:10.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.046+0000 7fc79f400640 1 --2- 172.21.15.96:0/1433552947 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc78006ddf0 0x7fc7800702b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:10.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.047+0000 7fc79f400640 1 --2- 172.21.15.96:0/1433552947 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc78006ddf0 0x7fc7800702b0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fc790032000 tx=0x7fc79002c000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:10.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.050+0000 7fc797400640 1 -- 172.21.15.96:0/1433552947 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc78c05a980 con 0x7fc7a015f460 2024-05-28T06:20:10.249 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.253+0000 7fc7a4a00640 1 -- 172.21.15.96:0/1433552947 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc7a006e170 con 0x7fc7a015f460 2024-05-28T06:20:10.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.254+0000 7fc797400640 1 -- 172.21.15.96:0/1433552947 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) ==== 74+0+13121 (secure 0 0 0) 0x7fc78c05a280 con 0x7fc7a015f460 2024-05-28T06:20:10.251 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:20:10.252 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":95,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:20:08.807423+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.96:6835/1614743177":"2024-05-29T06:20:05.340366+0000","172.21.15.96:6833/1614743177":"2024-05-29T06:20:05.340366+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":13,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:20:10.254 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.257+0000 7fc7a4a00640 1 -- 172.21.15.96:0/1433552947 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc78006ddf0 msgr2=0x7fc7800702b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:10.254 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.257+0000 7fc7a4a00640 1 --2- 172.21.15.96:0/1433552947 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc78006ddf0 0x7fc7800702b0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fc790032000 tx=0x7fc79002c000 comp rx=0 tx=0).stop 2024-05-28T06:20:10.254 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.257+0000 7fc7a4a00640 1 -- 172.21.15.96:0/1433552947 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc7a015f460 msgr2=0x7fc7a015f8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:10.254 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.257+0000 7fc7a4a00640 1 --2- 172.21.15.96:0/1433552947 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc7a015f460 0x7fc7a015f8e0 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7fc78c00d530 tx=0x7fc78c00d9f0 comp rx=0 tx=0).stop 2024-05-28T06:20:10.254 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.257+0000 7fc7a4a00640 1 -- 172.21.15.96:0/1433552947 shutdown_connections 2024-05-28T06:20:10.254 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.257+0000 7fc7a4a00640 1 --2- 172.21.15.96:0/1433552947 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc78006ddf0 0x7fc7800702b0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:10.254 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.257+0000 7fc7a4a00640 1 --2- 172.21.15.96:0/1433552947 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc7a01f4920 0x7fc7a01f4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:10.254 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.257+0000 7fc7a4a00640 1 --2- 172.21.15.96:0/1433552947 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc7a015f460 0x7fc7a015f8e0 unknown :-1 s=CLOSED pgs=1575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:10.254 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.257+0000 7fc7a4a00640 1 --2- 172.21.15.96:0/1433552947 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc784003a00 0x7fc7a01576d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:10.255 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.257+0000 7fc7a4a00640 1 -- 172.21.15.96:0/1433552947 >> 172.21.15.96:0/1433552947 conn(0x7fc7a00b1fc0 msgr2=0x7fc7a00b14c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:10.255 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.257+0000 7fc7a4a00640 1 -- 172.21.15.96:0/1433552947 shutdown_connections 2024-05-28T06:20:10.255 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.257+0000 7fc7a4a00640 1 -- 172.21.15.96:0/1433552947 wait complete. 2024-05-28T06:20:10.274 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-28T06:03:59.517102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-28T06:20:10.275 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-28T06:20:10.479 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.482+0000 7fd5a9600640 1 Processor -- start 2024-05-28T06:20:10.479 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.482+0000 7fd5a9600640 1 -- start start 2024-05-28T06:20:10.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.484+0000 7fd5a9600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fd5a406f5e0 0x7fd5a40cdd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:10.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.484+0000 7fd5a9600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fd5a415afd0 0x7fd5a40cc230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:10.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.484+0000 7fd5a9600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd5a40cc770 0x7fd5a40ccb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:10.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.484+0000 7fd5a9600640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fd5a406f9c0 con 0x7fd5a415afd0 2024-05-28T06:20:10.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.484+0000 7fd5a9600640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fd5a415b520 con 0x7fd5a406f5e0 2024-05-28T06:20:10.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.484+0000 7fd5a9600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fd5a415b800 con 0x7fd5a40cc770 2024-05-28T06:20:10.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.484+0000 7fd5a3400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fd5a415afd0 0x7fd5a40cc230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:10.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.484+0000 7fd5a3400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fd5a415afd0 0x7fd5a40cc230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:50920/0 (socket says 172.21.15.96:50920) 2024-05-28T06:20:10.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.484+0000 7fd5a3400640 1 -- 172.21.15.96:0/1319178574 learned_addr learned my addr 172.21.15.96:0/1319178574 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:20:10.483 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.485+0000 7fd5a3e00640 1 --2- 172.21.15.96:0/1319178574 >> v2:172.21.15.123:3300/0 conn(0x7fd5a406f5e0 0x7fd5a40cdd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:10.483 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.485+0000 7fd5a8c00640 1 --2- 172.21.15.96:0/1319178574 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd5a40cc770 0x7fd5a40ccb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:10.483 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.485+0000 7fd5a3400640 1 -- 172.21.15.96:0/1319178574 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd5a40cc770 msgr2=0x7fd5a40ccb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:10.483 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.485+0000 7fd5a3400640 1 --2- 172.21.15.96:0/1319178574 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd5a40cc770 0x7fd5a40ccb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:10.483 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.485+0000 7fd5a3400640 1 -- 172.21.15.96:0/1319178574 >> v2:172.21.15.123:3300/0 conn(0x7fd5a406f5e0 msgr2=0x7fd5a40cdd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:10.483 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.485+0000 7fd5a3400640 1 --2- 172.21.15.96:0/1319178574 >> v2:172.21.15.123:3300/0 conn(0x7fd5a406f5e0 0x7fd5a40cdd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:10.483 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.485+0000 7fd5a3400640 1 -- 172.21.15.96:0/1319178574 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5a415bae0 con 0x7fd5a415afd0 2024-05-28T06:20:10.483 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.485+0000 7fd5a3e00640 1 --2- 172.21.15.96:0/1319178574 >> v2:172.21.15.123:3300/0 conn(0x7fd5a406f5e0 0x7fd5a40cdd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:20:10.483 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.485+0000 7fd5a8c00640 1 --2- 172.21.15.96:0/1319178574 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd5a40cc770 0x7fd5a40ccb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:20:10.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.486+0000 7fd5a3400640 1 --2- 172.21.15.96:0/1319178574 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd5a415afd0 0x7fd5a40cc230 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7fd5a40ccb60 tx=0x7fd59002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=592a4e41ec47c723 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:10.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.486+0000 7fd5a2a00640 1 -- 172.21.15.96:0/1319178574 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd590037070 con 0x7fd5a415afd0 2024-05-28T06:20:10.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.486+0000 7fd5a2a00640 1 -- 172.21.15.96:0/1319178574 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd590032440 con 0x7fd5a415afd0 2024-05-28T06:20:10.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.486+0000 7fd5a2a00640 1 -- 172.21.15.96:0/1319178574 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd590032810 con 0x7fd5a415afd0 2024-05-28T06:20:10.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.487+0000 7fd5a9600640 1 -- 172.21.15.96:0/1319178574 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd5a415afd0 msgr2=0x7fd5a40cc230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:10.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.487+0000 7fd5a9600640 1 --2- 172.21.15.96:0/1319178574 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd5a415afd0 0x7fd5a40cc230 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7fd5a40ccb60 tx=0x7fd59002a1e0 comp rx=0 tx=0).stop 2024-05-28T06:20:10.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.487+0000 7fd5a9600640 1 -- 172.21.15.96:0/1319178574 shutdown_connections 2024-05-28T06:20:10.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.487+0000 7fd5a9600640 1 --2- 172.21.15.96:0/1319178574 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd5a40cc770 0x7fd5a40ccb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:10.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.487+0000 7fd5a9600640 1 --2- 172.21.15.96:0/1319178574 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd5a415afd0 0x7fd5a40cc230 unknown :-1 s=CLOSED pgs=1576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:10.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.487+0000 7fd5a9600640 1 --2- 172.21.15.96:0/1319178574 >> v2:172.21.15.123:3300/0 conn(0x7fd5a406f5e0 0x7fd5a40cdd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:10.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.487+0000 7fd5a9600640 1 -- 172.21.15.96:0/1319178574 >> 172.21.15.96:0/1319178574 conn(0x7fd5a406b900 msgr2=0x7fd5a406bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:10.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.487+0000 7fd5a9600640 1 -- 172.21.15.96:0/1319178574 shutdown_connections 2024-05-28T06:20:10.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.487+0000 7fd5a9600640 1 -- 172.21.15.96:0/1319178574 wait complete. 2024-05-28T06:20:10.486 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.489+0000 7fd5a9600640 1 Processor -- start 2024-05-28T06:20:10.486 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.489+0000 7fd5a9600640 1 -- start start 2024-05-28T06:20:10.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.490+0000 7fd5a9600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd5a406f5e0 0x7fd5a415cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:10.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.490+0000 7fd5a9600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd5a415afd0 0x7fd5a41465e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:10.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.490+0000 7fd5a9600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd5a4158760 0x7fd5a4159350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:10.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.490+0000 7fd5a9600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd5a40cd250 con 0x7fd5a415afd0 2024-05-28T06:20:10.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.490+0000 7fd5a9600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd5a40cd0d0 con 0x7fd5a4158760 2024-05-28T06:20:10.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.490+0000 7fd5a9600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fd5a40cd3d0 con 0x7fd5a406f5e0 2024-05-28T06:20:10.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.490+0000 7fd5a3400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd5a415afd0 0x7fd5a41465e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:10.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.490+0000 7fd5a3400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd5a415afd0 0x7fd5a41465e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:50924/0 (socket says 172.21.15.96:50924) 2024-05-28T06:20:10.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.490+0000 7fd5a3400640 1 -- 172.21.15.96:0/3570975906 learned_addr learned my addr 172.21.15.96:0/3570975906 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:20:10.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.490+0000 7fd5a3400640 1 -- 172.21.15.96:0/3570975906 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd5a406f5e0 msgr2=0x7fd5a415cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:10.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.490+0000 7fd5a3400640 1 --2- 172.21.15.96:0/3570975906 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd5a406f5e0 0x7fd5a415cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:10.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.490+0000 7fd5a3400640 1 -- 172.21.15.96:0/3570975906 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd5a4158760 msgr2=0x7fd5a4159350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:10.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.490+0000 7fd5a3400640 1 --2- 172.21.15.96:0/3570975906 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd5a4158760 0x7fd5a4159350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:10.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.490+0000 7fd5a3400640 1 -- 172.21.15.96:0/3570975906 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5a4146b20 con 0x7fd5a415afd0 2024-05-28T06:20:10.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.491+0000 7fd5a3400640 1 --2- 172.21.15.96:0/3570975906 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd5a415afd0 0x7fd5a41465e0 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7fd59002a220 tx=0x7fd5900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:10.489 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.491+0000 7fd59be00640 1 -- 172.21.15.96:0/3570975906 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd590037040 con 0x7fd5a415afd0 2024-05-28T06:20:10.489 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.491+0000 7fd59be00640 1 -- 172.21.15.96:0/3570975906 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd590042070 con 0x7fd5a415afd0 2024-05-28T06:20:10.489 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.491+0000 7fd59be00640 1 -- 172.21.15.96:0/3570975906 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd590030bb0 con 0x7fd5a415afd0 2024-05-28T06:20:10.489 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.491+0000 7fd5a9600640 1 -- 172.21.15.96:0/3570975906 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5a415d140 con 0x7fd5a415afd0 2024-05-28T06:20:10.490 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.491+0000 7fd5a9600640 1 -- 172.21.15.96:0/3570975906 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5a415d700 con 0x7fd5a415afd0 2024-05-28T06:20:10.490 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.492+0000 7fd5a9600640 1 -- 172.21.15.96:0/3570975906 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5a40689e0 con 0x7fd5a415afd0 2024-05-28T06:20:10.490 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.493+0000 7fd59be00640 1 -- 172.21.15.96:0/3570975906 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 34) ==== 92927+0+0 (secure 0 0 0) 0x7fd590047460 con 0x7fd5a415afd0 2024-05-28T06:20:10.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.494+0000 7fd59be00640 1 --2- 172.21.15.96:0/3570975906 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd58406ddc0 0x7fd584070280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:10.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.494+0000 7fd59be00640 1 -- 172.21.15.96:0/3570975906 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 4999+0+0 (secure 0 0 0) 0x7fd5900ab8b0 con 0x7fd5a415afd0 2024-05-28T06:20:10.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.495+0000 7fd5a3e00640 1 --2- 172.21.15.96:0/3570975906 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd58406ddc0 0x7fd584070280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:10.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.496+0000 7fd5a3e00640 1 --2- 172.21.15.96:0/3570975906 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd58406ddc0 0x7fd584070280 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fd59400a8b0 tx=0x7fd594008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:10.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.500+0000 7fd59be00640 1 -- 172.21.15.96:0/3570975906 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd5a4146b20 con 0x7fd5a415afd0 2024-05-28T06:20:10.698 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.701+0000 7fd5a9600640 1 -- 172.21.15.96:0/3570975906 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd5a40b4690 con 0x7fd5a415afd0 2024-05-28T06:20:10.699 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.702+0000 7fd59be00640 1 -- 172.21.15.96:0/3570975906 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v95) ==== 93+0+10 (secure 0 0 0) 0x7fd590045360 con 0x7fd5a415afd0 2024-05-28T06:20:10.699 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 1 2024-05-28T06:20:10.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.704+0000 7fd5a9600640 1 -- 172.21.15.96:0/3570975906 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd58406ddc0 msgr2=0x7fd584070280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:10.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.704+0000 7fd5a9600640 1 --2- 172.21.15.96:0/3570975906 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd58406ddc0 0x7fd584070280 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fd59400a8b0 tx=0x7fd594008040 comp rx=0 tx=0).stop 2024-05-28T06:20:10.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.705+0000 7fd5a9600640 1 -- 172.21.15.96:0/3570975906 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd5a415afd0 msgr2=0x7fd5a41465e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:10.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.705+0000 7fd5a9600640 1 --2- 172.21.15.96:0/3570975906 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd5a415afd0 0x7fd5a41465e0 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7fd59002a220 tx=0x7fd5900046c0 comp rx=0 tx=0).stop 2024-05-28T06:20:10.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.705+0000 7fd5a9600640 1 -- 172.21.15.96:0/3570975906 shutdown_connections 2024-05-28T06:20:10.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.705+0000 7fd5a9600640 1 --2- 172.21.15.96:0/3570975906 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd58406ddc0 0x7fd584070280 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:10.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.705+0000 7fd5a9600640 1 --2- 172.21.15.96:0/3570975906 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd5a4158760 0x7fd5a4159350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:10.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.705+0000 7fd5a9600640 1 --2- 172.21.15.96:0/3570975906 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd5a415afd0 0x7fd5a41465e0 unknown :-1 s=CLOSED pgs=1577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:10.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.705+0000 7fd5a9600640 1 --2- 172.21.15.96:0/3570975906 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd5a406f5e0 0x7fd5a415cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:10.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.705+0000 7fd5a9600640 1 -- 172.21.15.96:0/3570975906 >> 172.21.15.96:0/3570975906 conn(0x7fd5a406b900 msgr2=0x7fd5a415e650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:10.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.705+0000 7fd5a9600640 1 -- 172.21.15.96:0/3570975906 shutdown_connections 2024-05-28T06:20:10.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.705+0000 7fd5a9600640 1 -- 172.21.15.96:0/3570975906 wait complete. 2024-05-28T06:20:10.722 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestCloneProgressReporter.test_when_1_clone_is_cancelled' 2024-05-28T06:20:10.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.918+0000 7f6859600640 1 Processor -- start 2024-05-28T06:20:10.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.918+0000 7f6859600640 1 -- start start 2024-05-28T06:20:10.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.919+0000 7f6859600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f685406f860 0x7f68540b3150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:10.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.920+0000 7f6859600640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f685406ed10 con 0x7f68540b3690 2024-05-28T06:20:10.917 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.920+0000 7f6859600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f685415cb00 con 0x7f68540d0bb0 2024-05-28T06:20:10.917 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.920+0000 7f6859600640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f68540b3cc0 con 0x7f685406f860 2024-05-28T06:20:10.917 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.920+0000 7f6852000640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f68540b3690 0x7f68540d04a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:33702/0 (socket says 172.21.15.96:33702) 2024-05-28T06:20:10.917 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.920+0000 7f6852000640 1 -- 172.21.15.96:0/1039689740 learned_addr learned my addr 172.21.15.96:0/1039689740 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:20:10.919 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.920+0000 7f6851600640 1 -- 172.21.15.96:0/1039689740 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1317123833 0 0) 0x7f6840001110 con 0x7f68540b3690 2024-05-28T06:20:10.919 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.920+0000 7f6851600640 1 -- 172.21.15.96:0/1039689740 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6838003520 con 0x7f68540b3690 2024-05-28T06:20:10.919 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.921+0000 7f6852a00640 1 --2- 172.21.15.96:0/1039689740 >> v2:172.21.15.123:3300/0 conn(0x7f685406f860 0x7f68540b3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:10.919 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.921+0000 7f6851600640 1 -- 172.21.15.96:0/1039689740 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 260574309 0 0) 0x7f6838003520 con 0x7f68540b3690 2024-05-28T06:20:10.919 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.921+0000 7f6851600640 1 -- 172.21.15.96:0/1039689740 >> v2:172.21.15.123:3300/0 conn(0x7f685406f860 msgr2=0x7f68540b3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:10.919 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.921+0000 7f6851600640 1 --2- 172.21.15.96:0/1039689740 >> v2:172.21.15.123:3300/0 conn(0x7f685406f860 0x7f68540b3150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:10.919 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.921+0000 7f6851600640 1 -- 172.21.15.96:0/1039689740 >> v1:172.21.15.123:6789/0 conn(0x7f68540d0bb0 legacy=0x7f68540ad130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:10.919 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.921+0000 7f6851600640 1 -- 172.21.15.96:0/1039689740 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68540b3fa0 con 0x7f68540b3690 2024-05-28T06:20:10.919 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.921+0000 7f6851600640 1 -- 172.21.15.96:0/1039689740 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f6840001840 con 0x7f68540b3690 2024-05-28T06:20:10.919 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.921+0000 7f6851600640 1 -- 172.21.15.96:0/1039689740 >> v1:172.21.15.96:6789/0 conn(0x7f68540b3690 legacy=0x7f68540d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:10.919 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.921+0000 7f6851600640 1 --2- 172.21.15.96:0/1039689740 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6838003a00 0x7f6838003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:10.920 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.921+0000 7f6851600640 1 --2- 172.21.15.96:0/1039689740 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6838004800 0x7f6838004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:10.920 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.921+0000 7f6851600640 1 --2- 172.21.15.96:0/1039689740 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6838005210 0x7f683800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:10.920 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.921+0000 7f6851600640 1 -- 172.21.15.96:0/1039689740 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f68540b3cc0 con 0x7f6838004800 2024-05-28T06:20:10.920 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.921+0000 7f6851600640 1 -- 172.21.15.96:0/1039689740 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f683800c140 con 0x7f6838005210 2024-05-28T06:20:10.920 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.921+0000 7f6851600640 1 -- 172.21.15.96:0/1039689740 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f683800c340 con 0x7f6838003a00 2024-05-28T06:20:10.920 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.921+0000 7f6852000640 1 --2- 172.21.15.96:0/1039689740 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6838004800 0x7f6838004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:10.920 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.922+0000 7f6852000640 1 -- 172.21.15.96:0/1039689740 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6838003a00 msgr2=0x7f6838003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:10.920 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.922+0000 7f6852000640 1 --2- 172.21.15.96:0/1039689740 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6838003a00 0x7f6838003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:10.920 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.922+0000 7f6852000640 1 -- 172.21.15.96:0/1039689740 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6838005210 msgr2=0x7f683800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:10.920 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.922+0000 7f6852000640 1 --2- 172.21.15.96:0/1039689740 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6838005210 0x7f683800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:10.920 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.922+0000 7f6852000640 1 -- 172.21.15.96:0/1039689740 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6838003520 con 0x7f6838004800 2024-05-28T06:20:10.920 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.922+0000 7f6853400640 1 -- 172.21.15.96:0/1039689740 reap_dead start 2024-05-28T06:20:10.921 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.923+0000 7f6852000640 1 --2- 172.21.15.96:0/1039689740 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6838004800 0x7f6838004c80 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7f6840007fd0 tx=0x7f6838005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d0adf6055754f9d server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:10.921 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.923+0000 7f6851600640 1 -- 172.21.15.96:0/1039689740 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6840036070 con 0x7f6838004800 2024-05-28T06:20:10.922 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.923+0000 7f6851600640 1 -- 172.21.15.96:0/1039689740 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6840005ad0 con 0x7f6838004800 2024-05-28T06:20:10.922 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.923+0000 7f6851600640 1 -- 172.21.15.96:0/1039689740 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6840007e30 con 0x7f6838004800 2024-05-28T06:20:10.922 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.923+0000 7f6859600640 1 -- 172.21.15.96:0/1039689740 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6838004800 msgr2=0x7f6838004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:10.922 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.923+0000 7f6859600640 1 --2- 172.21.15.96:0/1039689740 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6838004800 0x7f6838004c80 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7f6840007fd0 tx=0x7f6838005830 comp rx=0 tx=0).stop 2024-05-28T06:20:10.922 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.923+0000 7f6859600640 1 -- 172.21.15.96:0/1039689740 shutdown_connections 2024-05-28T06:20:10.922 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.923+0000 7f6859600640 1 --2- 172.21.15.96:0/1039689740 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6838004800 0x7f6838004c80 unknown :-1 s=CLOSED pgs=1579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:10.922 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.923+0000 7f6859600640 1 -- 172.21.15.96:0/1039689740 >> 172.21.15.96:0/1039689740 conn(0x7f685406bae0 msgr2=0x7f685406bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:10.922 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.924+0000 7f6859600640 1 -- 172.21.15.96:0/1039689740 shutdown_connections 2024-05-28T06:20:10.922 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.924+0000 7f6859600640 1 -- 172.21.15.96:0/1039689740 wait complete. 2024-05-28T06:20:10.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.925+0000 7f6859600640 1 Processor -- start 2024-05-28T06:20:10.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.926+0000 7f6859600640 1 -- start start 2024-05-28T06:20:10.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.926+0000 7f6859600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6838004800 0x7f68540cced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:10.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.926+0000 7f6859600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f685415e730 0x7f68540c5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:10.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.926+0000 7f6859600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f68540c58b0 0x7f6854099570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:10.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.926+0000 7f6859600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f68540b4840 con 0x7f685415e730 2024-05-28T06:20:10.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.926+0000 7f6859600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f68540b4560 con 0x7f68540c58b0 2024-05-28T06:20:10.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.926+0000 7f6859600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f68540b3150 con 0x7f6838004800 2024-05-28T06:20:10.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.927+0000 7f6852000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f685415e730 0x7f68540c5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:10.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.927+0000 7f6852000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f685415e730 0x7f68540c5370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:50932/0 (socket says 172.21.15.96:50932) 2024-05-28T06:20:10.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.927+0000 7f6852000640 1 -- 172.21.15.96:0/3573223351 learned_addr learned my addr 172.21.15.96:0/3573223351 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:20:10.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.927+0000 7f6852a00640 1 --2- 172.21.15.96:0/3573223351 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6838004800 0x7f68540cced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:10.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.927+0000 7f6853400640 1 --2- 172.21.15.96:0/3573223351 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f68540c58b0 0x7f6854099570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:10.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.927+0000 7f6852000640 1 -- 172.21.15.96:0/3573223351 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6838004800 msgr2=0x7f68540cced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:10.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.927+0000 7f6852000640 1 --2- 172.21.15.96:0/3573223351 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6838004800 0x7f68540cced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:10.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.927+0000 7f6852000640 1 -- 172.21.15.96:0/3573223351 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f68540c58b0 msgr2=0x7f6854099570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:10.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.927+0000 7f6852000640 1 --2- 172.21.15.96:0/3573223351 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f68540c58b0 0x7f6854099570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:10.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.927+0000 7f6852000640 1 -- 172.21.15.96:0/3573223351 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f685415ec00 con 0x7f685415e730 2024-05-28T06:20:10.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.928+0000 7f6852000640 1 --2- 172.21.15.96:0/3573223351 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f685415e730 0x7f68540c5370 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7f684002af00 tx=0x7f68540b4170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:10.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.928+0000 7f684aa00640 1 -- 172.21.15.96:0/3573223351 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6840036030 con 0x7f685415e730 2024-05-28T06:20:10.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.928+0000 7f684aa00640 1 -- 172.21.15.96:0/3573223351 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6840042070 con 0x7f685415e730 2024-05-28T06:20:10.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.928+0000 7f684aa00640 1 -- 172.21.15.96:0/3573223351 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6840004bf0 con 0x7f685415e730 2024-05-28T06:20:10.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.928+0000 7f6859600640 1 -- 172.21.15.96:0/3573223351 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f685415eee0 con 0x7f685415e730 2024-05-28T06:20:10.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.928+0000 7f6859600640 1 -- 172.21.15.96:0/3573223351 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f685415f4a0 con 0x7f685415e730 2024-05-28T06:20:10.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.929+0000 7f6859600640 1 -- 172.21.15.96:0/3573223351 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68540689e0 con 0x7f685415e730 2024-05-28T06:20:10.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.930+0000 7f684aa00640 1 -- 172.21.15.96:0/3573223351 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 34) ==== 92927+0+0 (secure 0 0 0) 0x7f6840007070 con 0x7f685415e730 2024-05-28T06:20:10.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.931+0000 7f684aa00640 1 --2- 172.21.15.96:0/3573223351 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f683406ddc0 0x7f6834070280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:10.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.931+0000 7f684aa00640 1 -- 172.21.15.96:0/3573223351 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 4999+0+0 (secure 0 0 0) 0x7f68400aaa90 con 0x7f685415e730 2024-05-28T06:20:10.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.932+0000 7f6852a00640 1 --2- 172.21.15.96:0/3573223351 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f683406ddc0 0x7f6834070280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:10.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.933+0000 7f6852a00640 1 --2- 172.21.15.96:0/3573223351 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f683406ddc0 0x7f6834070280 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f6844006a90 tx=0x7f6844008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:10.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:10.935+0000 7f684aa00640 1 -- 172.21.15.96:0/3573223351 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f684004b020 con 0x7f685415e730 2024-05-28T06:20:11.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:11.213+0000 7f6859600640 1 -- 172.21.15.96:0/3573223351 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgressReporter.test_when_1_clone_is_cancelled"]} v 0) -- 0x7f68540689e0 con 0x7f685415e730 2024-05-28T06:20:11.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:11.835+0000 7f684aa00640 1 -- 172.21.15.96:0/3573223351 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgressReporter.test_when_1_clone_is_cancelled"]}]=0 v708) ==== 164+0+0 (secure 0 0 0) 0x7f685406ed10 con 0x7f685415e730 2024-05-28T06:20:11.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:11.839+0000 7f6859600640 1 -- 172.21.15.96:0/3573223351 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f683406ddc0 msgr2=0x7f6834070280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:11.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:11.839+0000 7f6859600640 1 --2- 172.21.15.96:0/3573223351 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f683406ddc0 0x7f6834070280 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f6844006a90 tx=0x7f6844008040 comp rx=0 tx=0).stop 2024-05-28T06:20:11.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:11.839+0000 7f6859600640 1 -- 172.21.15.96:0/3573223351 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f685415e730 msgr2=0x7f68540c5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:11.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:11.839+0000 7f6859600640 1 --2- 172.21.15.96:0/3573223351 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f685415e730 0x7f68540c5370 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7f684002af00 tx=0x7f68540b4170 comp rx=0 tx=0).stop 2024-05-28T06:20:11.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:11.839+0000 7f6859600640 1 -- 172.21.15.96:0/3573223351 shutdown_connections 2024-05-28T06:20:11.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:11.839+0000 7f6859600640 1 --2- 172.21.15.96:0/3573223351 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f683406ddc0 0x7f6834070280 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:11.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:11.839+0000 7f6859600640 1 --2- 172.21.15.96:0/3573223351 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f68540c58b0 0x7f6854099570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:11.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:11.839+0000 7f6859600640 1 --2- 172.21.15.96:0/3573223351 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f685415e730 0x7f68540c5370 unknown :-1 s=CLOSED pgs=1580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:11.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:11.839+0000 7f6859600640 1 --2- 172.21.15.96:0/3573223351 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6838004800 0x7f68540cced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:11.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:11.839+0000 7f6859600640 1 -- 172.21.15.96:0/3573223351 >> 172.21.15.96:0/3573223351 conn(0x7f685406bae0 msgr2=0x7f68540737b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:11.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:11.839+0000 7f6859600640 1 -- 172.21.15.96:0/3573223351 shutdown_connections 2024-05-28T06:20:11.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:11.839+0000 7f6859600640 1 -- 172.21.15.96:0/3573223351 wait complete. 2024-05-28T06:20:11.857 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-05-28T06:20:12.061 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.064+0000 7f4149a00640 1 Processor -- start 2024-05-28T06:20:12.061 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.064+0000 7f4149a00640 1 -- start start 2024-05-28T06:20:12.062 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.065+0000 7f4149a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f41440cd2a0 0x7f41440b4d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:12.062 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.066+0000 7f4149a00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f414415cb60 0x7f414415cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:12.062 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.066+0000 7f4149a00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f414415d900 con 0x7f4144153510 2024-05-28T06:20:12.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.066+0000 7f4149a00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f41441538f0 con 0x7f414415cb60 2024-05-28T06:20:12.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.066+0000 7f4149a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f41440b48c0 con 0x7f41440cd2a0 2024-05-28T06:20:12.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.066+0000 7f4143400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f41440cd2a0 0x7f41440b4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:12.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.067+0000 7f4143e00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f4144153510 0x7f41440ccb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.96:58390/0 (socket says 172.21.15.96:58390) 2024-05-28T06:20:12.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.067+0000 7f4143e00640 1 -- 172.21.15.96:0/2518629639 learned_addr learned my addr 172.21.15.96:0/2518629639 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:20:12.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.066+0000 7f4149000640 1 --2- 172.21.15.96:0/2518629639 >> v2:172.21.15.123:3300/0 conn(0x7f414415cb60 0x7f414415cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:12.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.067+0000 7f4142a00640 1 -- 172.21.15.96:0/2518629639 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3410516706 0 0) 0x7f414415d900 con 0x7f4144153510 2024-05-28T06:20:12.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.067+0000 7f4142a00640 1 -- 172.21.15.96:0/2518629639 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4128003520 con 0x7f4144153510 2024-05-28T06:20:12.065 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.068+0000 7f4149000640 1 -- 172.21.15.96:0/2518629639 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f41440cd2a0 msgr2=0x7f41440b4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:12.065 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.068+0000 7f4149000640 1 --2- 172.21.15.96:0/2518629639 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f41440cd2a0 0x7f41440b4d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:12.065 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.068+0000 7f4149000640 1 -- 172.21.15.96:0/2518629639 >> v1:172.21.15.123:6789/0 conn(0x7f4144153510 legacy=0x7f41440ccb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:12.065 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.068+0000 7f4149000640 1 -- 172.21.15.96:0/2518629639 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41440b3f90 con 0x7f414415cb60 2024-05-28T06:20:12.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.069+0000 7f4149000640 1 --2- 172.21.15.96:0/2518629639 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f414415cb60 0x7f414415cfd0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f413c009d90 tx=0x7f413c02a5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5c60815a96185aa9 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:12.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.070+0000 7f4142a00640 1 -- 172.21.15.96:0/2518629639 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f413c031600 con 0x7f414415cb60 2024-05-28T06:20:12.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.070+0000 7f4142a00640 1 -- 172.21.15.96:0/2518629639 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f413c0317a0 con 0x7f414415cb60 2024-05-28T06:20:12.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.070+0000 7f4142a00640 1 -- 172.21.15.96:0/2518629639 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f413c031b50 con 0x7f414415cb60 2024-05-28T06:20:12.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.070+0000 7f4149a00640 1 -- 172.21.15.96:0/2518629639 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f414415cb60 msgr2=0x7f414415cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:12.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.070+0000 7f4149a00640 1 --2- 172.21.15.96:0/2518629639 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f414415cb60 0x7f414415cfd0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f413c009d90 tx=0x7f413c02a5d0 comp rx=0 tx=0).stop 2024-05-28T06:20:12.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.070+0000 7f4149a00640 1 -- 172.21.15.96:0/2518629639 shutdown_connections 2024-05-28T06:20:12.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.070+0000 7f4149a00640 1 --2- 172.21.15.96:0/2518629639 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f414415cb60 0x7f414415cfd0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:12.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.070+0000 7f4149a00640 1 --2- 172.21.15.96:0/2518629639 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f41440cd2a0 0x7f41440b4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:12.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.070+0000 7f4149a00640 1 -- 172.21.15.96:0/2518629639 >> 172.21.15.96:0/2518629639 conn(0x7f414406b900 msgr2=0x7f414406bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:12.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.070+0000 7f4149a00640 1 -- 172.21.15.96:0/2518629639 shutdown_connections 2024-05-28T06:20:12.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.070+0000 7f4149a00640 1 -- 172.21.15.96:0/2518629639 wait complete. 2024-05-28T06:20:12.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.072+0000 7f4149a00640 1 Processor -- start 2024-05-28T06:20:12.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.072+0000 7f4149a00640 1 -- start start 2024-05-28T06:20:12.071 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.073+0000 7f4149a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41440cd2a0 0x7f41441f4d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:12.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.073+0000 7f4149a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4144153510 0x7f414414ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:12.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.073+0000 7f4149a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f414415cb60 0x7f41441f15f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:12.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.073+0000 7f4149a00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f414415d510 con 0x7f41440cd2a0 2024-05-28T06:20:12.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.073+0000 7f4149a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f41440b4550 con 0x7f414415cb60 2024-05-28T06:20:12.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.073+0000 7f4149a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f414415d690 con 0x7f4144153510 2024-05-28T06:20:12.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.074+0000 7f4143e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41440cd2a0 0x7f41441f4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:12.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.074+0000 7f4143e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41440cd2a0 0x7f41441f4d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:50934/0 (socket says 172.21.15.96:50934) 2024-05-28T06:20:12.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.074+0000 7f4143e00640 1 -- 172.21.15.96:0/3339115209 learned_addr learned my addr 172.21.15.96:0/3339115209 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:20:12.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.074+0000 7f4149000640 1 --2- 172.21.15.96:0/3339115209 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f414415cb60 0x7f41441f15f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:12.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.074+0000 7f4143400640 1 --2- 172.21.15.96:0/3339115209 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4144153510 0x7f414414ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:12.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.074+0000 7f4143e00640 1 -- 172.21.15.96:0/3339115209 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4144153510 msgr2=0x7f414414ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:12.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.074+0000 7f4143e00640 1 --2- 172.21.15.96:0/3339115209 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4144153510 0x7f414414ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:12.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.074+0000 7f4143e00640 1 -- 172.21.15.96:0/3339115209 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f414415cb60 msgr2=0x7f41441f15f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:12.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.074+0000 7f4143e00640 1 --2- 172.21.15.96:0/3339115209 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f414415cb60 0x7f41441f15f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:12.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.074+0000 7f4143e00640 1 -- 172.21.15.96:0/3339115209 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41440cde80 con 0x7f41440cd2a0 2024-05-28T06:20:12.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.074+0000 7f4143400640 1 --2- 172.21.15.96:0/3339115209 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4144153510 0x7f414414ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:20:12.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.074+0000 7f4143e00640 1 --2- 172.21.15.96:0/3339115209 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41440cd2a0 0x7f41441f4d10 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7f413400c430 tx=0x7f413400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:12.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.075+0000 7f413be00640 1 -- 172.21.15.96:0/3339115209 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4134017070 con 0x7f41440cd2a0 2024-05-28T06:20:12.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.075+0000 7f413be00640 1 -- 172.21.15.96:0/3339115209 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4134005ce0 con 0x7f41440cd2a0 2024-05-28T06:20:12.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.075+0000 7f413be00640 1 -- 172.21.15.96:0/3339115209 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41340072f0 con 0x7f41440cd2a0 2024-05-28T06:20:12.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.075+0000 7f4149a00640 1 -- 172.21.15.96:0/3339115209 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41440ce160 con 0x7f41440cd2a0 2024-05-28T06:20:12.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.075+0000 7f4149a00640 1 -- 172.21.15.96:0/3339115209 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f414414d470 con 0x7f41440cd2a0 2024-05-28T06:20:12.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.076+0000 7f4149a00640 1 -- 172.21.15.96:0/3339115209 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41440ccb10 con 0x7f41440cd2a0 2024-05-28T06:20:12.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.077+0000 7f413be00640 1 -- 172.21.15.96:0/3339115209 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 34) ==== 92927+0+0 (secure 0 0 0) 0x7f413401a070 con 0x7f41440cd2a0 2024-05-28T06:20:12.075 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.078+0000 7f413be00640 1 --2- 172.21.15.96:0/3339115209 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f412406ddf0 0x7f41240702b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:12.075 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.078+0000 7f413be00640 1 -- 172.21.15.96:0/3339115209 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 4999+0+0 (secure 0 0 0) 0x7f4134089d20 con 0x7f41440cd2a0 2024-05-28T06:20:12.075 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.079+0000 7f4143400640 1 --2- 172.21.15.96:0/3339115209 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f412406ddf0 0x7f41240702b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:12.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.080+0000 7f4143400640 1 --2- 172.21.15.96:0/3339115209 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f412406ddf0 0x7f41240702b0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f414415fc10 tx=0x7f4130008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:12.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.083+0000 7f413be00640 1 -- 172.21.15.96:0/3339115209 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4134059ba0 con 0x7f41440cd2a0 2024-05-28T06:20:12.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.274+0000 7f4149a00640 1 -- 172.21.15.96:0/3339115209 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f41440689e0 con 0x7f41440cd2a0 2024-05-28T06:20:12.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.281+0000 7f413be00640 1 -- 172.21.15.96:0/3339115209 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v24)=0 v24) ==== 122+0+0 (secure 0 0 0) 0x7f41340594a0 con 0x7f41440cd2a0 2024-05-28T06:20:12.281 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.284+0000 7f4149a00640 1 -- 172.21.15.96:0/3339115209 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f412406ddf0 msgr2=0x7f41240702b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:12.281 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.284+0000 7f4149a00640 1 --2- 172.21.15.96:0/3339115209 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f412406ddf0 0x7f41240702b0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f414415fc10 tx=0x7f4130008040 comp rx=0 tx=0).stop 2024-05-28T06:20:12.282 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.284+0000 7f4149a00640 1 -- 172.21.15.96:0/3339115209 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41440cd2a0 msgr2=0x7f41441f4d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:12.282 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.284+0000 7f4149a00640 1 --2- 172.21.15.96:0/3339115209 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41440cd2a0 0x7f41441f4d10 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7f413400c430 tx=0x7f413400c8f0 comp rx=0 tx=0).stop 2024-05-28T06:20:12.282 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.284+0000 7f4149a00640 1 -- 172.21.15.96:0/3339115209 shutdown_connections 2024-05-28T06:20:12.282 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.284+0000 7f4149a00640 1 --2- 172.21.15.96:0/3339115209 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f412406ddf0 0x7f41240702b0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:12.282 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.284+0000 7f4149a00640 1 --2- 172.21.15.96:0/3339115209 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f414415cb60 0x7f41441f15f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:12.282 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.284+0000 7f4149a00640 1 --2- 172.21.15.96:0/3339115209 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4144153510 0x7f414414ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:12.282 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.284+0000 7f4149a00640 1 --2- 172.21.15.96:0/3339115209 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41440cd2a0 0x7f41441f4d10 unknown :-1 s=CLOSED pgs=1581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:12.282 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.284+0000 7f4149a00640 1 -- 172.21.15.96:0/3339115209 >> 172.21.15.96:0/3339115209 conn(0x7f414406b900 msgr2=0x7f414406bce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:12.282 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.285+0000 7f4149a00640 1 -- 172.21.15.96:0/3339115209 shutdown_connections 2024-05-28T06:20:12.282 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.285+0000 7f4149a00640 1 -- 172.21.15.96:0/3339115209 wait complete. 2024-05-28T06:20:12.300 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-05-28T06:20:12.301 DEBUG:teuthology.orchestra.run.smithi096:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-05-28T06:20:12.397 DEBUG:teuthology.orchestra.run.smithi123:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-05-28T06:20:12.512 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-05-28T06:20:12.512 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:20:12.512 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:20:12.566 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052731,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248996k,nr_inodes=812249,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-28T06:20:12.567 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-05-28T06:20:12.567 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:20:12.623 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052731,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248996k,nr_inodes=812249,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-05-28T06:20:12.623 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:20:12.623 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:20:12.680 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052731,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248996k,nr_inodes=812249,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-28T06:20:12.680 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-05-28T06:20:12.680 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:20:12.738 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052731,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248996k,nr_inodes=812249,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-05-28T06:20:12.739 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-05-28T06:20:12.739 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-28T06:20:12.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.950+0000 7fa707e00640 1 Processor -- start 2024-05-28T06:20:12.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.950+0000 7fa707e00640 1 -- start start 2024-05-28T06:20:12.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.952+0000 7fa707e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fa708156b30 0x7fa7080af6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:12.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.952+0000 7fa707e00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa708156f10 con 0x7fa70806ee40 2024-05-28T06:20:12.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.952+0000 7fa707e00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa70806dda0 con 0x7fa7080ad940 2024-05-28T06:20:12.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.952+0000 7fa707e00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fa70806e080 con 0x7fa708156b30 2024-05-28T06:20:12.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.952+0000 7fa706a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fa708156b30 0x7fa7080af6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:12.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.952+0000 7fa6fe000640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fa70806ee40 0x7fa7080ad230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:33714/0 (socket says 172.21.15.96:33714) 2024-05-28T06:20:12.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.952+0000 7fa6fe000640 1 -- 172.21.15.96:0/1052551371 learned_addr learned my addr 172.21.15.96:0/1052551371 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:20:12.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.953+0000 7fa706a00640 1 -- 172.21.15.96:0/1052551371 >> v1:172.21.15.123:6789/0 conn(0x7fa7080ad940 legacy=0x7fa7080b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:12.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.953+0000 7fa706a00640 1 -- 172.21.15.96:0/1052551371 >> v1:172.21.15.96:6789/0 conn(0x7fa70806ee40 legacy=0x7fa7080ad230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:12.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.953+0000 7fa706a00640 1 -- 172.21.15.96:0/1052551371 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa70806e360 con 0x7fa708156b30 2024-05-28T06:20:12.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.954+0000 7fa706a00640 1 --2- 172.21.15.96:0/1052551371 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa708156b30 0x7fa7080af6e0 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7fa6f80097a0 tx=0x7fa6f802a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc9fd1476e4d778d server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:12.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.954+0000 7fa706000640 1 -- 172.21.15.96:0/1052551371 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6f8037070 con 0x7fa708156b30 2024-05-28T06:20:12.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.954+0000 7fa706000640 1 -- 172.21.15.96:0/1052551371 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6f802ae50 con 0x7fa708156b30 2024-05-28T06:20:12.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.954+0000 7fa706000640 1 -- 172.21.15.96:0/1052551371 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6f8032660 con 0x7fa708156b30 2024-05-28T06:20:12.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.955+0000 7fa707e00640 1 -- 172.21.15.96:0/1052551371 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa708156b30 msgr2=0x7fa7080af6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:12.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.955+0000 7fa707e00640 1 --2- 172.21.15.96:0/1052551371 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa708156b30 0x7fa7080af6e0 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7fa6f80097a0 tx=0x7fa6f802a140 comp rx=0 tx=0).stop 2024-05-28T06:20:12.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.955+0000 7fa707e00640 1 -- 172.21.15.96:0/1052551371 shutdown_connections 2024-05-28T06:20:12.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.955+0000 7fa707e00640 1 --2- 172.21.15.96:0/1052551371 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa708156b30 0x7fa7080af6e0 unknown :-1 s=CLOSED pgs=1583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:12.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.955+0000 7fa707e00640 1 -- 172.21.15.96:0/1052551371 >> 172.21.15.96:0/1052551371 conn(0x7fa7080b1fc0 msgr2=0x7fa7080b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:12.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.955+0000 7fa707e00640 1 -- 172.21.15.96:0/1052551371 shutdown_connections 2024-05-28T06:20:12.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.955+0000 7fa707e00640 1 -- 172.21.15.96:0/1052551371 wait complete. 2024-05-28T06:20:12.953 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.957+0000 7fa707e00640 1 Processor -- start 2024-05-28T06:20:12.953 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.957+0000 7fa707e00640 1 -- start start 2024-05-28T06:20:12.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.958+0000 7fa707e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa70806ee40 0x7fa70815f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:12.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.958+0000 7fa707e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa7080ad940 0x7fa708099910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:12.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.958+0000 7fa707e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa708156b30 0x7fa7080b30b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:12.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.958+0000 7fa707e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa7080afc20 con 0x7fa70806ee40 2024-05-28T06:20:12.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.958+0000 7fa707e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa7080b2c30 con 0x7fa708156b30 2024-05-28T06:20:12.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.958+0000 7fa707e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fa708159f50 con 0x7fa7080ad940 2024-05-28T06:20:12.956 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.958+0000 7fa706a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa70806ee40 0x7fa70815f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:12.956 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.958+0000 7fa706a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa70806ee40 0x7fa70815f400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:50950/0 (socket says 172.21.15.96:50950) 2024-05-28T06:20:12.956 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.958+0000 7fa706a00640 1 -- 172.21.15.96:0/2218892518 learned_addr learned my addr 172.21.15.96:0/2218892518 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:20:12.956 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.958+0000 7fa706a00640 1 -- 172.21.15.96:0/2218892518 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa7080ad940 msgr2=0x7fa708099910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:12.956 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.958+0000 7fa706a00640 1 --2- 172.21.15.96:0/2218892518 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa7080ad940 0x7fa708099910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:12.956 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.959+0000 7fa706a00640 1 -- 172.21.15.96:0/2218892518 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa708156b30 msgr2=0x7fa7080b30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:12.956 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.959+0000 7fa706a00640 1 --2- 172.21.15.96:0/2218892518 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa708156b30 0x7fa7080b30b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:12.956 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.959+0000 7fa706a00640 1 -- 172.21.15.96:0/2218892518 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa70815b8b0 con 0x7fa70806ee40 2024-05-28T06:20:12.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.959+0000 7fa706a00640 1 --2- 172.21.15.96:0/2218892518 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa70806ee40 0x7fa70815f400 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7fa6f80098c0 tx=0x7fa6f80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:12.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.959+0000 7fa6ff400640 1 -- 172.21.15.96:0/2218892518 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6f8041020 con 0x7fa70806ee40 2024-05-28T06:20:12.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.959+0000 7fa6ff400640 1 -- 172.21.15.96:0/2218892518 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6f803c870 con 0x7fa70806ee40 2024-05-28T06:20:12.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.959+0000 7fa6ff400640 1 -- 172.21.15.96:0/2218892518 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6f8037040 con 0x7fa70806ee40 2024-05-28T06:20:12.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.959+0000 7fa707e00640 1 -- 172.21.15.96:0/2218892518 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa70815bb90 con 0x7fa70806ee40 2024-05-28T06:20:12.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.959+0000 7fa707e00640 1 -- 172.21.15.96:0/2218892518 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa70815c150 con 0x7fa70806ee40 2024-05-28T06:20:12.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.960+0000 7fa707e00640 1 -- 172.21.15.96:0/2218892518 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7080689e0 con 0x7fa70806ee40 2024-05-28T06:20:12.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.961+0000 7fa6ff400640 1 -- 172.21.15.96:0/2218892518 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 34) ==== 92927+0+0 (secure 0 0 0) 0x7fa6f8044020 con 0x7fa70806ee40 2024-05-28T06:20:12.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.962+0000 7fa6ff400640 1 --2- 172.21.15.96:0/2218892518 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa6e406ddc0 0x7fa6e4070280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:12.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.962+0000 7fa6ff400640 1 -- 172.21.15.96:0/2218892518 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 4999+0+0 (secure 0 0 0) 0x7fa6f80ab0d0 con 0x7fa70806ee40 2024-05-28T06:20:12.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.963+0000 7fa6fe000640 1 --2- 172.21.15.96:0/2218892518 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa6e406ddc0 0x7fa6e4070280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:12.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.964+0000 7fa6fe000640 1 --2- 172.21.15.96:0/2218892518 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa6e406ddc0 0x7fa6e4070280 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fa6f0005880 tx=0x7fa6f0007aa0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:12.963 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:12.967+0000 7fa6ff400640 1 -- 172.21.15.96:0/2218892518 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa6f804b030 con 0x7fa70806ee40 2024-05-28T06:20:13.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.210+0000 7fa707e00640 1 -- 172.21.15.96:0/2218892518 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa7080b0bc0 con 0x7fa70806ee40 2024-05-28T06:20:13.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.211+0000 7fa6ff400640 1 -- 172.21.15.96:0/2218892518 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) ==== 94+0+3875 (secure 0 0 0) 0x7fa6f807af90 con 0x7fa70806ee40 2024-05-28T06:20:13.208 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:20:13.208 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":55,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5645,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6834/1355344409","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":1355344409},{"type":"v1","addr":"172.21.15.96:6834","nonce":1355344409}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":47},{"gid":5651,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/1967688533","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":1967688533},{"type":"v1","addr":"172.21.15.123:6835","nonce":1967688533}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48},{"gid":6169,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/3362018278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":3362018278},{"type":"v1","addr":"172.21.15.123:6837","nonce":3362018278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48},{"gid":8646,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/1871560877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6833","nonce":1871560877},{"type":"v1","addr":"172.21.15.96:6835","nonce":1871560877}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":55}],"filesystems":[]} 2024-05-28T06:20:13.208 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 55 2024-05-28T06:20:13.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.214+0000 7fa707e00640 1 -- 172.21.15.96:0/2218892518 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa6e406ddc0 msgr2=0x7fa6e4070280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:13.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.214+0000 7fa707e00640 1 --2- 172.21.15.96:0/2218892518 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa6e406ddc0 0x7fa6e4070280 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fa6f0005880 tx=0x7fa6f0007aa0 comp rx=0 tx=0).stop 2024-05-28T06:20:13.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.214+0000 7fa707e00640 1 -- 172.21.15.96:0/2218892518 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa70806ee40 msgr2=0x7fa70815f400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:13.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.214+0000 7fa707e00640 1 --2- 172.21.15.96:0/2218892518 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa70806ee40 0x7fa70815f400 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7fa6f80098c0 tx=0x7fa6f80046c0 comp rx=0 tx=0).stop 2024-05-28T06:20:13.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.214+0000 7fa707e00640 1 -- 172.21.15.96:0/2218892518 shutdown_connections 2024-05-28T06:20:13.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.214+0000 7fa707e00640 1 --2- 172.21.15.96:0/2218892518 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa6e406ddc0 0x7fa6e4070280 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:13.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.214+0000 7fa707e00640 1 --2- 172.21.15.96:0/2218892518 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa708156b30 0x7fa7080b30b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:13.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.214+0000 7fa707e00640 1 --2- 172.21.15.96:0/2218892518 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa7080ad940 0x7fa708099910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:13.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.214+0000 7fa707e00640 1 --2- 172.21.15.96:0/2218892518 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa70806ee40 0x7fa70815f400 unknown :-1 s=CLOSED pgs=1584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:13.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.214+0000 7fa707e00640 1 -- 172.21.15.96:0/2218892518 >> 172.21.15.96:0/2218892518 conn(0x7fa7080b1fc0 msgr2=0x7fa7080af060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:13.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.214+0000 7fa707e00640 1 -- 172.21.15.96:0/2218892518 shutdown_connections 2024-05-28T06:20:13.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.214+0000 7fa707e00640 1 -- 172.21.15.96:0/2218892518 wait complete. 2024-05-28T06:20:13.231 INFO:tasks.ceph.mds.a:Restarting daemon 2024-05-28T06:20:13.231 INFO:tasks.ceph.mds.a:Stopping old one... 2024-05-28T06:20:13.231 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-05-28T06:20:13.231 INFO:teuthology.orchestra.run:waiting for 300 2024-05-28T06:20:13.231 INFO:tasks.ceph.mds.c:Restarting daemon 2024-05-28T06:20:13.232 INFO:tasks.ceph.mds.c:Stopping old one... 2024-05-28T06:20:13.232 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-05-28T06:20:13.232 INFO:teuthology.orchestra.run:waiting for 300 2024-05-28T06:20:13.232 INFO:tasks.ceph.mds.b:Restarting daemon 2024-05-28T06:20:13.232 INFO:tasks.ceph.mds.b:Stopping old one... 2024-05-28T06:20:13.232 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-05-28T06:20:13.232 INFO:teuthology.orchestra.run:waiting for 300 2024-05-28T06:20:13.233 INFO:tasks.ceph.mds.d:Restarting daemon 2024-05-28T06:20:13.233 INFO:tasks.ceph.mds.d:Stopping old one... 2024-05-28T06:20:13.233 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-05-28T06:20:13.233 INFO:teuthology.orchestra.run:waiting for 300 2024-05-28T06:20:13.251 INFO:tasks.ceph.mds.a:Stopped 2024-05-28T06:20:13.251 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-05-28T06:20:13.253 INFO:tasks.ceph.mds.c:Stopped 2024-05-28T06:20:13.254 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-05-28T06:20:13.257 INFO:tasks.ceph.mds.d:Stopped 2024-05-28T06:20:13.257 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-05-28T06:20:13.259 INFO:tasks.ceph.mds.b:Stopped 2024-05-28T06:20:13.259 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-05-28T06:20:13.299 INFO:tasks.ceph.mds.a:Started 2024-05-28T06:20:13.300 DEBUG:teuthology.parallel:result is None 2024-05-28T06:20:13.304 INFO:tasks.ceph.mds.c:Started 2024-05-28T06:20:13.304 DEBUG:teuthology.parallel:result is None 2024-05-28T06:20:13.305 INFO:tasks.ceph.mds.d:Started 2024-05-28T06:20:13.305 DEBUG:teuthology.parallel:result is None 2024-05-28T06:20:13.311 INFO:tasks.ceph.mds.b:Started 2024-05-28T06:20:13.311 DEBUG:teuthology.parallel:result is None 2024-05-28T06:20:13.312 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-05-28T06:20:13.448 INFO:tasks.ceph.mds.c.smithi096.stdout:starting mds.c at 2024-05-28T06:20:13.449 INFO:tasks.ceph.mds.a.smithi096.stdout:starting mds.a at 2024-05-28T06:20:13.456 INFO:tasks.ceph.mds.d.smithi123.stdout:starting mds.d at 2024-05-28T06:20:13.456 INFO:tasks.ceph.mds.b.smithi123.stdout:starting mds.b at 2024-05-28T06:20:13.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.491+0000 7f67d3e00640 1 Processor -- start 2024-05-28T06:20:13.489 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.491+0000 7f67d3e00640 1 -- start start 2024-05-28T06:20:13.490 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.493+0000 7f67d3e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f67d415e870 0x7f67d40d0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:13.490 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.493+0000 7f67d3e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f67d40cfa40 0x7f67d40d25e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:13.490 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.493+0000 7f67d3e00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67d415ec50 con 0x7f67d40d14d0 2024-05-28T06:20:13.490 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.493+0000 7f67d3e00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f67d40d2150 con 0x7f67d415e870 2024-05-28T06:20:13.490 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.493+0000 7f67d3e00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f67d40cff80 con 0x7f67d40cfa40 2024-05-28T06:20:13.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.493+0000 7f67d2a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f67d415e870 0x7f67d40d0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:13.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.493+0000 7f67d2a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f67d415e870 0x7f67d40d0f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51004/0 (socket says 172.21.15.96:51004) 2024-05-28T06:20:13.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.493+0000 7f67d2a00640 1 -- 172.21.15.96:0/3297732628 learned_addr learned my addr 172.21.15.96:0/3297732628 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:20:13.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.493+0000 7f67d3400640 1 --2- 172.21.15.96:0/3297732628 >> v2:172.21.15.123:3300/0 conn(0x7f67d40cfa40 0x7f67d40d25e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:13.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.493+0000 7f67d2a00640 1 -- 172.21.15.96:0/3297732628 >> v2:172.21.15.123:3300/0 conn(0x7f67d40cfa40 msgr2=0x7f67d40d25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:13.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.493+0000 7f67d2a00640 1 --2- 172.21.15.96:0/3297732628 >> v2:172.21.15.123:3300/0 conn(0x7f67d40cfa40 0x7f67d40d25e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:13.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.493+0000 7f67d2a00640 1 -- 172.21.15.96:0/3297732628 >> v1:172.21.15.123:6789/0 conn(0x7f67d40d14d0 legacy=0x7f67d40cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:13.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.493+0000 7f67d2a00640 1 -- 172.21.15.96:0/3297732628 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67d40d0260 con 0x7f67d415e870 2024-05-28T06:20:13.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.493+0000 7f67d2000640 1 -- 172.21.15.96:0/3297732628 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1879698481 0 0) 0x7f67d415ec50 con 0x7f67d40d14d0 2024-05-28T06:20:13.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.494+0000 7f67d2a00640 1 --2- 172.21.15.96:0/3297732628 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f67d415e870 0x7f67d40d0f90 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7f67d408cad0 tx=0x7f67c402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16957bb78585025c server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:13.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.495+0000 7f67d2000640 1 -- 172.21.15.96:0/3297732628 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67c4037070 con 0x7f67d415e870 2024-05-28T06:20:13.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.495+0000 7f67d2000640 1 -- 172.21.15.96:0/3297732628 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67c4032440 con 0x7f67d415e870 2024-05-28T06:20:13.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.495+0000 7f67d2000640 1 -- 172.21.15.96:0/3297732628 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67c4032810 con 0x7f67d415e870 2024-05-28T06:20:13.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.495+0000 7f67d3e00640 1 -- 172.21.15.96:0/3297732628 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f67d415e870 msgr2=0x7f67d40d0f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:13.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.495+0000 7f67d3e00640 1 --2- 172.21.15.96:0/3297732628 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f67d415e870 0x7f67d40d0f90 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7f67d408cad0 tx=0x7f67c402a270 comp rx=0 tx=0).stop 2024-05-28T06:20:13.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.496+0000 7f67d3e00640 1 -- 172.21.15.96:0/3297732628 shutdown_connections 2024-05-28T06:20:13.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.496+0000 7f67d3e00640 1 --2- 172.21.15.96:0/3297732628 >> v2:172.21.15.123:3300/0 conn(0x7f67d40cfa40 0x7f67d40d25e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:13.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.496+0000 7f67d3e00640 1 --2- 172.21.15.96:0/3297732628 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f67d415e870 0x7f67d40d0f90 unknown :-1 s=CLOSED pgs=1592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:13.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.496+0000 7f67d3e00640 1 -- 172.21.15.96:0/3297732628 >> 172.21.15.96:0/3297732628 conn(0x7f67d408bfc0 msgr2=0x7f67d40d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:13.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.496+0000 7f67d3e00640 1 -- 172.21.15.96:0/3297732628 shutdown_connections 2024-05-28T06:20:13.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.496+0000 7f67d3e00640 1 -- 172.21.15.96:0/3297732628 wait complete. 2024-05-28T06:20:13.494 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.498+0000 7f67d3e00640 1 Processor -- start 2024-05-28T06:20:13.494 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.498+0000 7f67d3e00640 1 -- start start 2024-05-28T06:20:13.496 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.499+0000 7f67d3e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f67d40cfa40 0x7f67d4159bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:13.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.499+0000 7f67d3e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67d40d14d0 0x7f67d40cd660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:13.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.499+0000 7f67d3e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f67d415e870 0x7f67d40cdba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:13.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.499+0000 7f67d3e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f67d40d4fb0 con 0x7f67d40cfa40 2024-05-28T06:20:13.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.499+0000 7f67d3e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f67d40d0820 con 0x7f67d40d14d0 2024-05-28T06:20:13.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.499+0000 7f67d3e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f67d40d2b20 con 0x7f67d415e870 2024-05-28T06:20:13.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.499+0000 7f67d2a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f67d40cfa40 0x7f67d4159bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:13.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.499+0000 7f67d2a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f67d40cfa40 0x7f67d4159bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51010/0 (socket says 172.21.15.96:51010) 2024-05-28T06:20:13.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.499+0000 7f67d2a00640 1 -- 172.21.15.96:0/1041081604 learned_addr learned my addr 172.21.15.96:0/1041081604 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:20:13.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.499+0000 7f67ca000640 1 --2- 172.21.15.96:0/1041081604 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67d40d14d0 0x7f67d40cd660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:13.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.499+0000 7f67d2a00640 1 -- 172.21.15.96:0/1041081604 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f67d415e870 msgr2=0x7f67d40cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:13.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.499+0000 7f67d3400640 1 --2- 172.21.15.96:0/1041081604 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f67d415e870 0x7f67d40cdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:13.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.499+0000 7f67d2a00640 1 --2- 172.21.15.96:0/1041081604 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f67d415e870 0x7f67d40cdba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:13.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.499+0000 7f67d2a00640 1 -- 172.21.15.96:0/1041081604 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67d40d14d0 msgr2=0x7f67d40cd660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:13.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.499+0000 7f67d2a00640 1 --2- 172.21.15.96:0/1041081604 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67d40d14d0 0x7f67d40cd660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:13.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.499+0000 7f67d2a00640 1 -- 172.21.15.96:0/1041081604 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67d415a4e0 con 0x7f67d40cfa40 2024-05-28T06:20:13.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.499+0000 7f67ca000640 1 --2- 172.21.15.96:0/1041081604 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67d40d14d0 0x7f67d40cd660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:20:13.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.500+0000 7f67d2a00640 1 --2- 172.21.15.96:0/1041081604 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f67d40cfa40 0x7f67d4159bf0 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7f67c4000c00 tx=0x7f67c40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:13.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.500+0000 7f67cb400640 1 -- 172.21.15.96:0/1041081604 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67c4037040 con 0x7f67d40cfa40 2024-05-28T06:20:13.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.500+0000 7f67cb400640 1 -- 172.21.15.96:0/1041081604 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67c4042070 con 0x7f67d40cfa40 2024-05-28T06:20:13.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.500+0000 7f67cb400640 1 -- 172.21.15.96:0/1041081604 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67c4030880 con 0x7f67d40cfa40 2024-05-28T06:20:13.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.500+0000 7f67d3e00640 1 -- 172.21.15.96:0/1041081604 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67d415a7c0 con 0x7f67d40cfa40 2024-05-28T06:20:13.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.500+0000 7f67d3e00640 1 -- 172.21.15.96:0/1041081604 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67d415ad80 con 0x7f67d40cfa40 2024-05-28T06:20:13.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.501+0000 7f67d3e00640 1 -- 172.21.15.96:0/1041081604 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67d40689e0 con 0x7f67d40cfa40 2024-05-28T06:20:13.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.503+0000 7f67cb400640 1 -- 172.21.15.96:0/1041081604 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 34) ==== 92927+0+0 (secure 0 0 0) 0x7f67c4030a20 con 0x7f67d40cfa40 2024-05-28T06:20:13.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.504+0000 7f67cb400640 1 --2- 172.21.15.96:0/1041081604 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f67b006ddc0 0x7f67b0070280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:13.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.504+0000 7f67cb400640 1 -- 172.21.15.96:0/1041081604 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 4999+0+0 (secure 0 0 0) 0x7f67c40ab4e0 con 0x7f67d40cfa40 2024-05-28T06:20:13.501 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.504+0000 7f67ca000640 1 --2- 172.21.15.96:0/1041081604 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f67b006ddc0 0x7f67b0070280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:13.501 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.505+0000 7f67ca000640 1 --2- 172.21.15.96:0/1041081604 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f67b006ddc0 0x7f67b0070280 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f67d40ce250 tx=0x7f67bc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:13.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.508+0000 7f67cb400640 1 -- 172.21.15.96:0/1041081604 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f67c407af80 con 0x7f67d40cfa40 2024-05-28T06:20:13.712 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.715+0000 7f67d3e00640 1 -- 172.21.15.96:0/1041081604 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f67d40ba500 con 0x7f67d40cfa40 2024-05-28T06:20:13.866 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.869+0000 7f67cb400640 1 -- 172.21.15.96:0/1041081604 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v97) ==== 97+0+0 (secure 0 0 0) 0x7f67c407a880 con 0x7f67d40cfa40 2024-05-28T06:20:13.866 INFO:teuthology.orchestra.run.smithi096.stderr: removed all blocklist entries 2024-05-28T06:20:13.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.873+0000 7f67d3e00640 1 -- 172.21.15.96:0/1041081604 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f67b006ddc0 msgr2=0x7f67b0070280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:13.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.873+0000 7f67d3e00640 1 --2- 172.21.15.96:0/1041081604 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f67b006ddc0 0x7f67b0070280 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f67d40ce250 tx=0x7f67bc009040 comp rx=0 tx=0).stop 2024-05-28T06:20:13.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.873+0000 7f67d3e00640 1 -- 172.21.15.96:0/1041081604 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f67d40cfa40 msgr2=0x7f67d4159bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:13.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.873+0000 7f67d3e00640 1 --2- 172.21.15.96:0/1041081604 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f67d40cfa40 0x7f67d4159bf0 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7f67c4000c00 tx=0x7f67c40046c0 comp rx=0 tx=0).stop 2024-05-28T06:20:13.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.874+0000 7f67d3e00640 1 -- 172.21.15.96:0/1041081604 shutdown_connections 2024-05-28T06:20:13.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.874+0000 7f67d3e00640 1 --2- 172.21.15.96:0/1041081604 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f67b006ddc0 0x7f67b0070280 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:13.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.874+0000 7f67d3e00640 1 --2- 172.21.15.96:0/1041081604 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f67d415e870 0x7f67d40cdba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:13.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.874+0000 7f67d3e00640 1 --2- 172.21.15.96:0/1041081604 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67d40d14d0 0x7f67d40cd660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:13.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.874+0000 7f67d3e00640 1 --2- 172.21.15.96:0/1041081604 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f67d40cfa40 0x7f67d4159bf0 unknown :-1 s=CLOSED pgs=1593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:13.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.874+0000 7f67d3e00640 1 -- 172.21.15.96:0/1041081604 >> 172.21.15.96:0/1041081604 conn(0x7f67d408bfc0 msgr2=0x7f67d415d4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:13.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.874+0000 7f67d3e00640 1 -- 172.21.15.96:0/1041081604 shutdown_connections 2024-05-28T06:20:13.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:13.874+0000 7f67d3e00640 1 -- 172.21.15.96:0/1041081604 wait complete. 2024-05-28T06:20:13.891 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-05-28T06:20:14.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.079+0000 7fc3f6000640 1 Processor -- start 2024-05-28T06:20:14.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.079+0000 7fc3f6000640 1 -- start start 2024-05-28T06:20:14.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.080+0000 7fc3f6000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fc3f0156b30 0x7fc3f00af6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:14.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.080+0000 7fc3f6000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fc3f006ee40 0x7fc3f00ad230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:14.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.080+0000 7fc3f6000640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3f0156f10 con 0x7fc3f006f2f0 2024-05-28T06:20:14.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.080+0000 7fc3f6000640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fc3f006dda0 con 0x7fc3f0156b30 2024-05-28T06:20:14.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.080+0000 7fc3f6000640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fc3f006e080 con 0x7fc3f006ee40 2024-05-28T06:20:14.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.081+0000 7fc3f4c00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fc3f0156b30 0x7fc3f00af6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:14.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.081+0000 7fc3f4c00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fc3f0156b30 0x7fc3f00af6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51026/0 (socket says 172.21.15.96:51026) 2024-05-28T06:20:14.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.081+0000 7fc3f4c00640 1 -- 172.21.15.96:0/3189781289 learned_addr learned my addr 172.21.15.96:0/3189781289 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:20:14.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.081+0000 7fc3f4c00640 1 -- 172.21.15.96:0/3189781289 >> v2:172.21.15.123:3300/0 conn(0x7fc3f006ee40 msgr2=0x7fc3f00ad230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:14.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.081+0000 7fc3f4c00640 1 --2- 172.21.15.96:0/3189781289 >> v2:172.21.15.123:3300/0 conn(0x7fc3f006ee40 0x7fc3f00ad230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:14.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.081+0000 7fc3f4c00640 1 -- 172.21.15.96:0/3189781289 >> v1:172.21.15.123:6789/0 conn(0x7fc3f006f2f0 legacy=0x7fc3f00b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:14.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.081+0000 7fc3f4c00640 1 -- 172.21.15.96:0/3189781289 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3f006e360 con 0x7fc3f0156b30 2024-05-28T06:20:14.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.082+0000 7fc3f4c00640 1 --2- 172.21.15.96:0/3189781289 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc3f0156b30 0x7fc3f00af6e0 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7fc3f006fdf0 tx=0x7fc3e002a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c627a2f58fdc1736 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:14.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.082+0000 7fc3efe00640 1 -- 172.21.15.96:0/3189781289 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3e0037070 con 0x7fc3f0156b30 2024-05-28T06:20:14.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.083+0000 7fc3efe00640 1 -- 172.21.15.96:0/3189781289 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3e0032440 con 0x7fc3f0156b30 2024-05-28T06:20:14.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.083+0000 7fc3efe00640 1 -- 172.21.15.96:0/3189781289 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3e0032810 con 0x7fc3f0156b30 2024-05-28T06:20:14.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.083+0000 7fc3f6000640 1 -- 172.21.15.96:0/3189781289 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc3f0156b30 msgr2=0x7fc3f00af6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:14.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.083+0000 7fc3f6000640 1 --2- 172.21.15.96:0/3189781289 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc3f0156b30 0x7fc3f00af6e0 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7fc3f006fdf0 tx=0x7fc3e002a1a0 comp rx=0 tx=0).stop 2024-05-28T06:20:14.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.083+0000 7fc3f6000640 1 -- 172.21.15.96:0/3189781289 shutdown_connections 2024-05-28T06:20:14.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.083+0000 7fc3f6000640 1 --2- 172.21.15.96:0/3189781289 >> v2:172.21.15.123:3300/0 conn(0x7fc3f006ee40 0x7fc3f00ad230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:14.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.083+0000 7fc3f6000640 1 --2- 172.21.15.96:0/3189781289 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc3f0156b30 0x7fc3f00af6e0 unknown :-1 s=CLOSED pgs=1594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:14.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.083+0000 7fc3f6000640 1 -- 172.21.15.96:0/3189781289 >> 172.21.15.96:0/3189781289 conn(0x7fc3f00b1fc0 msgr2=0x7fc3f00b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:14.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.083+0000 7fc3f6000640 1 -- 172.21.15.96:0/3189781289 shutdown_connections 2024-05-28T06:20:14.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.083+0000 7fc3f6000640 1 -- 172.21.15.96:0/3189781289 wait complete. 2024-05-28T06:20:14.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.085+0000 7fc3f6000640 1 Processor -- start 2024-05-28T06:20:14.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.085+0000 7fc3f6000640 1 -- start start 2024-05-28T06:20:14.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.086+0000 7fc3f6000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc3f006ee40 0x7fc3f01f3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:14.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.086+0000 7fc3f6000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3f006f2f0 0x7fc3f014ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:14.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.086+0000 7fc3f6000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc3f0156b30 0x7fc3f014b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:14.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.086+0000 7fc3f6000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc3f00afc20 con 0x7fc3f0156b30 2024-05-28T06:20:14.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.086+0000 7fc3f6000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc3f00b2c30 con 0x7fc3f006f2f0 2024-05-28T06:20:14.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.086+0000 7fc3f6000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fc3f0158be0 con 0x7fc3f006ee40 2024-05-28T06:20:14.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.086+0000 7fc3f5600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc3f0156b30 0x7fc3f014b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:14.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.086+0000 7fc3f5600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc3f0156b30 0x7fc3f014b140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51038/0 (socket says 172.21.15.96:51038) 2024-05-28T06:20:14.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.086+0000 7fc3f5600640 1 -- 172.21.15.96:0/1254371206 learned_addr learned my addr 172.21.15.96:0/1254371206 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:20:14.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.087+0000 7fc3f5600640 1 -- 172.21.15.96:0/1254371206 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc3f006ee40 msgr2=0x7fc3f01f3ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:14.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.087+0000 7fc3f5600640 1 --2- 172.21.15.96:0/1254371206 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc3f006ee40 0x7fc3f01f3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:14.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.087+0000 7fc3f5600640 1 -- 172.21.15.96:0/1254371206 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3f006f2f0 msgr2=0x7fc3f014ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:14.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.087+0000 7fc3f5600640 1 --2- 172.21.15.96:0/1254371206 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3f006f2f0 0x7fc3f014ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:14.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.087+0000 7fc3f5600640 1 -- 172.21.15.96:0/1254371206 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3f0073010 con 0x7fc3f0156b30 2024-05-28T06:20:14.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.087+0000 7fc3f5600640 1 --2- 172.21.15.96:0/1254371206 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc3f0156b30 0x7fc3f014b140 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7fc3e800ba10 tx=0x7fc3e800bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:14.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.087+0000 7fc3ed600640 1 -- 172.21.15.96:0/1254371206 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3e8019070 con 0x7fc3f0156b30 2024-05-28T06:20:14.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.087+0000 7fc3ed600640 1 -- 172.21.15.96:0/1254371206 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3e800cb20 con 0x7fc3f0156b30 2024-05-28T06:20:14.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.087+0000 7fc3ed600640 1 -- 172.21.15.96:0/1254371206 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3e8014440 con 0x7fc3f0156b30 2024-05-28T06:20:14.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.087+0000 7fc3f6000640 1 -- 172.21.15.96:0/1254371206 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3f00b1040 con 0x7fc3f0156b30 2024-05-28T06:20:14.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.088+0000 7fc3f6000640 1 -- 172.21.15.96:0/1254371206 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3f0073590 con 0x7fc3f0156b30 2024-05-28T06:20:14.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.088+0000 7fc3f6000640 1 -- 172.21.15.96:0/1254371206 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3f00689e0 con 0x7fc3f0156b30 2024-05-28T06:20:14.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.089+0000 7fc3ed600640 1 -- 172.21.15.96:0/1254371206 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 34) ==== 92927+0+0 (secure 0 0 0) 0x7fc3e801c070 con 0x7fc3f0156b30 2024-05-28T06:20:14.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.090+0000 7fc3ed600640 1 --2- 172.21.15.96:0/1254371206 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc3cc06ddc0 0x7fc3cc070280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:14.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.090+0000 7fc3ed600640 1 -- 172.21.15.96:0/1254371206 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 4913+0+0 (secure 0 0 0) 0x7fc3e808a510 con 0x7fc3f0156b30 2024-05-28T06:20:14.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.091+0000 7fc3f4c00640 1 --2- 172.21.15.96:0/1254371206 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc3cc06ddc0 0x7fc3cc070280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:14.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.092+0000 7fc3f4c00640 1 --2- 172.21.15.96:0/1254371206 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc3cc06ddc0 0x7fc3cc070280 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fc3e0034000 tx=0x7fc3e002c000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:14.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.094+0000 7fc3ed600640 1 -- 172.21.15.96:0/1254371206 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc3e805a390 con 0x7fc3f0156b30 2024-05-28T06:20:14.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.347+0000 7fc3f6000640 1 -- 172.21.15.96:0/1254371206 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7fc3f006f7f0 con 0x7fc3f0156b30 2024-05-28T06:20:14.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.347+0000 7fc3ed600640 1 -- 172.21.15.96:0/1254371206 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v20) ==== 80+0+5304 (secure 0 0 0) 0x7fc3e8059c90 con 0x7fc3f0156b30 2024-05-28T06:20:14.345 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:20:14.345 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:20:14.345 INFO:teuthology.orchestra.run.smithi096.stdout: "auth_dump": [ 2024-05-28T06:20:14.345 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:20:14.345 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "mds.a", 2024-05-28T06:20:14.345 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCWc1VmyA9HHBAANJCF8I/RP5ZYviyOzo0JCQ==", 2024-05-28T06:20:14.345 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:20:14.345 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow", 2024-05-28T06:20:14.346 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow *", 2024-05-28T06:20:14.346 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow *", 2024-05-28T06:20:14.346 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:20:14.346 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:20:14.346 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:20:14.346 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:20:14.346 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "mds.b", 2024-05-28T06:20:14.346 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCWc1VmtR0EOBAA296qGHTmB/2sMztDbDJ56g==", 2024-05-28T06:20:14.346 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:20:14.346 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow", 2024-05-28T06:20:14.346 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow *", 2024-05-28T06:20:14.347 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow *", 2024-05-28T06:20:14.347 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:20:14.347 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:20:14.347 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:20:14.347 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:20:14.347 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "mds.c", 2024-05-28T06:20:14.347 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCWc1VmrLxHKhAAhkTRLDKL6RUJQGetB8JQQA==", 2024-05-28T06:20:14.347 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:20:14.347 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow", 2024-05-28T06:20:14.347 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow *", 2024-05-28T06:20:14.347 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow *", 2024-05-28T06:20:14.347 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:20:14.347 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:20:14.348 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:20:14.348 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:20:14.348 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "mds.d", 2024-05-28T06:20:14.348 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCXc1VmW2aQCxAAnkDwQzlgy39vFe2O9Wwf7Q==", 2024-05-28T06:20:14.348 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:20:14.348 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow", 2024-05-28T06:20:14.348 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow *", 2024-05-28T06:20:14.348 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow *", 2024-05-28T06:20:14.348 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:20:14.348 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:20:14.348 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:20:14.348 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:20:14.348 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "osd.0", 2024-05-28T06:20:14.348 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCbc1VmEcNAAxAAiB5AsfY2uRY1kc6h80chuw==", 2024-05-28T06:20:14.349 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:20:14.349 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow profile osd", 2024-05-28T06:20:14.349 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile osd", 2024-05-28T06:20:14.349 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:20:14.349 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:20:14.349 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:20:14.349 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:20:14.349 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "osd.1", 2024-05-28T06:20:14.349 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCec1Vm6CAaDhAA1NkYM2q4fTsgSDSwJZL3Qg==", 2024-05-28T06:20:14.349 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:20:14.349 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow profile osd", 2024-05-28T06:20:14.349 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile osd", 2024-05-28T06:20:14.349 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:20:14.350 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:20:14.350 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:20:14.350 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:20:14.350 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "osd.2", 2024-05-28T06:20:14.350 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQChc1VmKn1jFxAA+dM4gM7FR8NwBpyd833A/Q==", 2024-05-28T06:20:14.350 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:20:14.350 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow profile osd", 2024-05-28T06:20:14.350 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile osd", 2024-05-28T06:20:14.350 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:20:14.350 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:20:14.350 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:20:14.350 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:20:14.350 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "osd.3", 2024-05-28T06:20:14.350 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCkc1VmV07MIRAAHIXQGDVbL9V5QqexQ89IJA==", 2024-05-28T06:20:14.351 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:20:14.351 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow profile osd", 2024-05-28T06:20:14.351 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile osd", 2024-05-28T06:20:14.351 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:20:14.351 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:20:14.351 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:20:14.351 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:20:14.351 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "osd.4", 2024-05-28T06:20:14.351 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCqc1VmobuZMxAAlY8xhuzqlkuFrcbTouF5rA==", 2024-05-28T06:20:14.351 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:20:14.351 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow profile osd", 2024-05-28T06:20:14.351 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile osd", 2024-05-28T06:20:14.351 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:20:14.352 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:20:14.352 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:20:14.352 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:20:14.352 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "osd.5", 2024-05-28T06:20:14.352 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCuc1Vmwt3QAxAAUFj/e/wJvxljgAbcTb32Ug==", 2024-05-28T06:20:14.352 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:20:14.352 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow profile osd", 2024-05-28T06:20:14.352 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile osd", 2024-05-28T06:20:14.352 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:20:14.352 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:20:14.352 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:20:14.352 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:20:14.352 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "osd.6", 2024-05-28T06:20:14.353 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCxc1VmscerDxAAqEDAqy6/OvdfSJ+ALZ7WeQ==", 2024-05-28T06:20:14.353 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:20:14.353 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow profile osd", 2024-05-28T06:20:14.353 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile osd", 2024-05-28T06:20:14.353 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:20:14.353 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:20:14.353 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:20:14.353 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:20:14.353 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "osd.7", 2024-05-28T06:20:14.353 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQC0c1VmgxkbKhAA7dFzDTBKFmKiFFxPsyiT4Q==", 2024-05-28T06:20:14.353 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:20:14.353 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow profile osd", 2024-05-28T06:20:14.353 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile osd", 2024-05-28T06:20:14.354 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:20:14.354 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:20:14.354 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:20:14.354 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:20:14.354 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "client.0", 2024-05-28T06:20:14.354 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCXc1Vm91hbGBAASTL3HQR8XOs7wVQEHbPl/g==", 2024-05-28T06:20:14.354 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:20:14.354 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow", 2024-05-28T06:20:14.354 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow r", 2024-05-28T06:20:14.354 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-05-28T06:20:14.354 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:20:14.354 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:20:14.354 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:20:14.355 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "client.1", 2024-05-28T06:20:14.355 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCXc1VmIc77KhAAw2e0alTmVROhyCIFW4H3Sg==", 2024-05-28T06:20:14.355 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:20:14.355 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow", 2024-05-28T06:20:14.355 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow r", 2024-05-28T06:20:14.355 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow rw", 2024-05-28T06:20:14.355 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow rwx" 2024-05-28T06:20:14.355 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:20:14.355 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:20:14.355 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:20:14.355 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "client.2", 2024-05-28T06:20:14.355 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCXc1VmCS+OIRAAMV3Au6KuchLhA59QTGbv1w==", 2024-05-28T06:20:14.355 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:20:14.356 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow", 2024-05-28T06:20:14.356 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow r", 2024-05-28T06:20:14.356 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow rw", 2024-05-28T06:20:14.356 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow rwx" 2024-05-28T06:20:14.356 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:20:14.356 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:20:14.356 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:20:14.356 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "client.3", 2024-05-28T06:20:14.356 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCXc1Vm5r3pNBAAkVZ3arjywHQkEP5lmS8xqg==", 2024-05-28T06:20:14.356 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:20:14.356 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow", 2024-05-28T06:20:14.356 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow r", 2024-05-28T06:20:14.356 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow rw", 2024-05-28T06:20:14.357 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow rwx" 2024-05-28T06:20:14.357 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:20:14.357 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:20:14.357 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:20:14.357 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "client.admin", 2024-05-28T06:20:14.357 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCVc1Vm5QaXFhAAQFGKgbZ/kpGisNm2lIw0Uw==", 2024-05-28T06:20:14.357 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:20:14.357 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow *", 2024-05-28T06:20:14.357 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow *", 2024-05-28T06:20:14.357 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow *", 2024-05-28T06:20:14.357 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:20:14.357 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:20:14.357 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:20:14.358 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:20:14.358 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "mgr.x", 2024-05-28T06:20:14.358 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCWc1VmVMx/EhAAylvi37aq3hbMOhUSQWLVSg==", 2024-05-28T06:20:14.358 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:20:14.358 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow *", 2024-05-28T06:20:14.358 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile mgr", 2024-05-28T06:20:14.358 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:20:14.358 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:20:14.358 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:20:14.358 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:20:14.358 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "mgr.y", 2024-05-28T06:20:14.358 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCWc1Vmh5r7BxAAA74gn+FozehM+34eFik8kw==", 2024-05-28T06:20:14.359 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:20:14.359 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow *", 2024-05-28T06:20:14.359 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile mgr", 2024-05-28T06:20:14.359 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:20:14.359 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:20:14.359 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:20:14.359 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-05-28T06:20:14.359 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:20:14.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.350+0000 7fc3f6000640 1 -- 172.21.15.96:0/1254371206 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc3cc06ddc0 msgr2=0x7fc3cc070280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:14.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.350+0000 7fc3f6000640 1 --2- 172.21.15.96:0/1254371206 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc3cc06ddc0 0x7fc3cc070280 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fc3e0034000 tx=0x7fc3e002c000 comp rx=0 tx=0).stop 2024-05-28T06:20:14.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.350+0000 7fc3f6000640 1 -- 172.21.15.96:0/1254371206 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc3f0156b30 msgr2=0x7fc3f014b140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:14.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.350+0000 7fc3f6000640 1 --2- 172.21.15.96:0/1254371206 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc3f0156b30 0x7fc3f014b140 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7fc3e800ba10 tx=0x7fc3e800bed0 comp rx=0 tx=0).stop 2024-05-28T06:20:14.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.350+0000 7fc3f6000640 1 -- 172.21.15.96:0/1254371206 shutdown_connections 2024-05-28T06:20:14.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.350+0000 7fc3f6000640 1 --2- 172.21.15.96:0/1254371206 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc3cc06ddc0 0x7fc3cc070280 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:14.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.350+0000 7fc3f6000640 1 --2- 172.21.15.96:0/1254371206 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc3f0156b30 0x7fc3f014b140 unknown :-1 s=CLOSED pgs=1595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:14.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.350+0000 7fc3f6000640 1 --2- 172.21.15.96:0/1254371206 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3f006f2f0 0x7fc3f014ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:14.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.350+0000 7fc3f6000640 1 --2- 172.21.15.96:0/1254371206 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc3f006ee40 0x7fc3f01f3ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:14.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.350+0000 7fc3f6000640 1 -- 172.21.15.96:0/1254371206 >> 172.21.15.96:0/1254371206 conn(0x7fc3f00b1fc0 msgr2=0x7fc3f015b530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:14.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.350+0000 7fc3f6000640 1 -- 172.21.15.96:0/1254371206 shutdown_connections 2024-05-28T06:20:14.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.350+0000 7fc3f6000640 1 -- 172.21.15.96:0/1254371206 wait complete. 2024-05-28T06:20:14.365 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:20:14.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.561+0000 7f9871a00640 1 Processor -- start 2024-05-28T06:20:14.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.561+0000 7f9871a00640 1 -- start start 2024-05-28T06:20:14.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.562+0000 7f9871a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f986c073310 0x7f986c0b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:14.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.562+0000 7f9871a00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f986c06dc20 0x7f986c15f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:14.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.563+0000 7f9871a00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f986c0afd60 con 0x7f986c15b3a0 2024-05-28T06:20:14.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.563+0000 7f9871a00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f986c06f980 con 0x7f986c06dc20 2024-05-28T06:20:14.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.563+0000 7f9871a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f986c0ae0c0 con 0x7f986c073310 2024-05-28T06:20:14.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.563+0000 7f986b400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f986c073310 0x7f986c0b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:14.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.563+0000 7f9871000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f986c06dc20 0x7f986c15f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:14.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.563+0000 7f986be00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f986c15b3a0 0x7f986c072c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.96:58436/0 (socket says 172.21.15.96:58436) 2024-05-28T06:20:14.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.563+0000 7f986b400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f986c073310 0x7f986c0b32c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.96:59666/0 (socket says 172.21.15.96:59666) 2024-05-28T06:20:14.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.563+0000 7f986be00640 1 -- 172.21.15.96:0/4077914351 learned_addr learned my addr 172.21.15.96:0/4077914351 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:20:14.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.564+0000 7f986b400640 1 -- 172.21.15.96:0/4077914351 >> v2:172.21.15.123:3300/0 conn(0x7f986c06dc20 msgr2=0x7f986c15f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:14.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.564+0000 7f986b400640 1 --2- 172.21.15.96:0/4077914351 >> v2:172.21.15.123:3300/0 conn(0x7f986c06dc20 0x7f986c15f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:14.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.564+0000 7f986b400640 1 -- 172.21.15.96:0/4077914351 >> v1:172.21.15.123:6789/0 conn(0x7f986c15b3a0 legacy=0x7f986c072c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:14.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.564+0000 7f986b400640 1 -- 172.21.15.96:0/4077914351 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f986c0af7a0 con 0x7f986c073310 2024-05-28T06:20:14.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.565+0000 7f986b400640 1 --2- 172.21.15.96:0/4077914351 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f986c073310 0x7f986c0b32c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f985800ba40 tx=0x7f985802c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d97441c5d946c4c7 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:14.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.566+0000 7f986aa00640 1 -- 172.21.15.96:0/4077914351 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9858032750 con 0x7f986c073310 2024-05-28T06:20:14.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.566+0000 7f986aa00640 1 -- 172.21.15.96:0/4077914351 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9858030050 con 0x7f986c073310 2024-05-28T06:20:14.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.566+0000 7f986aa00640 1 -- 172.21.15.96:0/4077914351 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9858032b10 con 0x7f986c073310 2024-05-28T06:20:14.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.567+0000 7f9871a00640 1 -- 172.21.15.96:0/4077914351 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f986c073310 msgr2=0x7f986c0b32c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:14.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.567+0000 7f9871a00640 1 --2- 172.21.15.96:0/4077914351 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f986c073310 0x7f986c0b32c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f985800ba40 tx=0x7f985802c850 comp rx=0 tx=0).stop 2024-05-28T06:20:14.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.567+0000 7f9871a00640 1 -- 172.21.15.96:0/4077914351 shutdown_connections 2024-05-28T06:20:14.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.567+0000 7f9871a00640 1 --2- 172.21.15.96:0/4077914351 >> v2:172.21.15.123:3300/0 conn(0x7f986c06dc20 0x7f986c15f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:14.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.567+0000 7f9871a00640 1 --2- 172.21.15.96:0/4077914351 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f986c073310 0x7f986c0b32c0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:14.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.567+0000 7f9871a00640 1 -- 172.21.15.96:0/4077914351 >> 172.21.15.96:0/4077914351 conn(0x7f986c0b1fc0 msgr2=0x7f986c0b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:14.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.567+0000 7f9871a00640 1 -- 172.21.15.96:0/4077914351 shutdown_connections 2024-05-28T06:20:14.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.567+0000 7f9871a00640 1 -- 172.21.15.96:0/4077914351 wait complete. 2024-05-28T06:20:14.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.569+0000 7f9871a00640 1 Processor -- start 2024-05-28T06:20:14.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.569+0000 7f9871a00640 1 -- start start 2024-05-28T06:20:14.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.570+0000 7f9871a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f986c06dc20 0x7f986c14e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:14.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.570+0000 7f9871a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f986c073310 0x7f986c0ce1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:14.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.570+0000 7f9871a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f986c15b3a0 0x7f986c157220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:14.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.570+0000 7f9871a00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f986c15b780 con 0x7f986c15b3a0 2024-05-28T06:20:14.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.570+0000 7f9871a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f986c15fee0 con 0x7f986c073310 2024-05-28T06:20:14.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.570+0000 7f9871a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f986c0ae550 con 0x7f986c06dc20 2024-05-28T06:20:14.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.571+0000 7f9871000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f986c15b3a0 0x7f986c157220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:14.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.571+0000 7f9871000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f986c15b3a0 0x7f986c157220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51048/0 (socket says 172.21.15.96:51048) 2024-05-28T06:20:14.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.571+0000 7f9871000640 1 -- 172.21.15.96:0/3408931132 learned_addr learned my addr 172.21.15.96:0/3408931132 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:20:14.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.571+0000 7f986b400640 1 --2- 172.21.15.96:0/3408931132 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f986c073310 0x7f986c0ce1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:14.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.571+0000 7f986be00640 1 --2- 172.21.15.96:0/3408931132 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f986c06dc20 0x7f986c14e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:14.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.571+0000 7f9871000640 1 -- 172.21.15.96:0/3408931132 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f986c06dc20 msgr2=0x7f986c14e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:14.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.571+0000 7f9871000640 1 --2- 172.21.15.96:0/3408931132 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f986c06dc20 0x7f986c14e380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:14.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.571+0000 7f9871000640 1 -- 172.21.15.96:0/3408931132 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f986c073310 msgr2=0x7f986c0ce1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:14.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.571+0000 7f9871000640 1 --2- 172.21.15.96:0/3408931132 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f986c073310 0x7f986c0ce1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:14.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.571+0000 7f9871000640 1 -- 172.21.15.96:0/3408931132 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f986c0cd150 con 0x7f986c15b3a0 2024-05-28T06:20:14.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.572+0000 7f9871000640 1 --2- 172.21.15.96:0/3408931132 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f986c15b3a0 0x7f986c157220 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f986400efc0 tx=0x7f986400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:14.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.572+0000 7f9863e00640 1 -- 172.21.15.96:0/3408931132 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9864018020 con 0x7f986c15b3a0 2024-05-28T06:20:14.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.572+0000 7f9863e00640 1 -- 172.21.15.96:0/3408931132 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9864009b10 con 0x7f986c15b3a0 2024-05-28T06:20:14.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.572+0000 7f9863e00640 1 -- 172.21.15.96:0/3408931132 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98640146c0 con 0x7f986c15b3a0 2024-05-28T06:20:14.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.572+0000 7f9871a00640 1 -- 172.21.15.96:0/3408931132 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f986c0cd430 con 0x7f986c15b3a0 2024-05-28T06:20:14.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.572+0000 7f9871a00640 1 -- 172.21.15.96:0/3408931132 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f986c0cd9f0 con 0x7f986c15b3a0 2024-05-28T06:20:14.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.573+0000 7f9871a00640 1 -- 172.21.15.96:0/3408931132 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f986c0689e0 con 0x7f986c15b3a0 2024-05-28T06:20:14.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.574+0000 7f9863e00640 1 -- 172.21.15.96:0/3408931132 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 34) ==== 92927+0+0 (secure 0 0 0) 0x7f9864005d10 con 0x7f986c15b3a0 2024-05-28T06:20:14.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.575+0000 7f9863e00640 1 --2- 172.21.15.96:0/3408931132 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f984c06ddc0 0x7f984c070280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:14.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.576+0000 7f9863e00640 1 -- 172.21.15.96:0/3408931132 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 4913+0+0 (secure 0 0 0) 0x7f986408b660 con 0x7f986c15b3a0 2024-05-28T06:20:14.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.576+0000 7f986be00640 1 --2- 172.21.15.96:0/3408931132 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f984c06ddc0 0x7f984c070280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:14.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.577+0000 7f986be00640 1 --2- 172.21.15.96:0/3408931132 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f984c06ddc0 0x7f984c070280 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f985c005880 tx=0x7f985c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:14.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.580+0000 7f9863e00640 1 -- 172.21.15.96:0/3408931132 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9864022030 con 0x7f986c15b3a0 2024-05-28T06:20:14.761 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.764+0000 7f9871a00640 1 -- 172.21.15.96:0/3408931132 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f986c0b38a0 con 0x7f986c15b3a0 2024-05-28T06:20:14.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.765+0000 7f9863e00640 1 -- 172.21.15.96:0/3408931132 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v99) ==== 74+0+12992 (secure 0 0 0) 0x7f9864010040 con 0x7f986c15b3a0 2024-05-28T06:20:14.762 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:20:14.762 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":99,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:20:13.885380+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":13,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:20:14.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.767+0000 7f9871a00640 1 -- 172.21.15.96:0/3408931132 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f984c06ddc0 msgr2=0x7f984c070280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:14.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.767+0000 7f9871a00640 1 --2- 172.21.15.96:0/3408931132 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f984c06ddc0 0x7f984c070280 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f985c005880 tx=0x7f985c009040 comp rx=0 tx=0).stop 2024-05-28T06:20:14.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.767+0000 7f9871a00640 1 -- 172.21.15.96:0/3408931132 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f986c15b3a0 msgr2=0x7f986c157220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:14.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.767+0000 7f9871a00640 1 --2- 172.21.15.96:0/3408931132 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f986c15b3a0 0x7f986c157220 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f986400efc0 tx=0x7f986400c490 comp rx=0 tx=0).stop 2024-05-28T06:20:14.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.768+0000 7f9871a00640 1 -- 172.21.15.96:0/3408931132 shutdown_connections 2024-05-28T06:20:14.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.768+0000 7f9871a00640 1 --2- 172.21.15.96:0/3408931132 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f984c06ddc0 0x7f984c070280 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:14.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.768+0000 7f9871a00640 1 --2- 172.21.15.96:0/3408931132 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f986c15b3a0 0x7f986c157220 unknown :-1 s=CLOSED pgs=1596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:14.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.768+0000 7f9871a00640 1 --2- 172.21.15.96:0/3408931132 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f986c073310 0x7f986c0ce1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:14.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.768+0000 7f9871a00640 1 --2- 172.21.15.96:0/3408931132 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f986c06dc20 0x7f986c14e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:14.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.768+0000 7f9871a00640 1 -- 172.21.15.96:0/3408931132 >> 172.21.15.96:0/3408931132 conn(0x7f986c0b1fc0 msgr2=0x7f986c0b4120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:14.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.768+0000 7f9871a00640 1 -- 172.21.15.96:0/3408931132 shutdown_connections 2024-05-28T06:20:14.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.768+0000 7f9871a00640 1 -- 172.21.15.96:0/3408931132 wait complete. 2024-05-28T06:20:14.783 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-28T06:03:59.517102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-28T06:20:14.784 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-28T06:20:14.970 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.971+0000 7f8557200640 1 Processor -- start 2024-05-28T06:20:14.970 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.972+0000 7f8557200640 1 -- start start 2024-05-28T06:20:14.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.973+0000 7f8557200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f8550160280 0x7f8550162860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:14.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.973+0000 7f8557200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8550162da0 con 0x7f8550162480 2024-05-28T06:20:14.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.973+0000 7f8557200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f855006d1f0 con 0x7f855009b5d0 2024-05-28T06:20:14.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.973+0000 7f8557200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f855006d4d0 con 0x7f8550160280 2024-05-28T06:20:14.976 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.973+0000 7f854fe00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f8550162480 0x7f855006ca10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:33736/0 (socket says 172.21.15.96:33736) 2024-05-28T06:20:14.976 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.973+0000 7f854fe00640 1 -- 172.21.15.96:0/4032444299 learned_addr learned my addr 172.21.15.96:0/4032444299 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:20:14.976 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.974+0000 7f854ea00640 1 -- 172.21.15.96:0/4032444299 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1183131367 0 0) 0x7f8550162da0 con 0x7f8550162480 2024-05-28T06:20:14.976 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.974+0000 7f854ea00640 1 -- 172.21.15.96:0/4032444299 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8534003520 con 0x7f8550162480 2024-05-28T06:20:14.976 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.974+0000 7f854f400640 1 --2- 172.21.15.96:0/4032444299 >> v2:172.21.15.123:3300/0 conn(0x7f8550160280 0x7f8550162860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:14.976 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.974+0000 7f854ea00640 1 -- 172.21.15.96:0/4032444299 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2510986692 0 0) 0x7f8534003520 con 0x7f8550162480 2024-05-28T06:20:14.976 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.974+0000 7f854ea00640 1 -- 172.21.15.96:0/4032444299 >> v2:172.21.15.123:3300/0 conn(0x7f8550160280 msgr2=0x7f8550162860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:14.976 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.974+0000 7f854ea00640 1 --2- 172.21.15.96:0/4032444299 >> v2:172.21.15.123:3300/0 conn(0x7f8550160280 0x7f8550162860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:14.976 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.974+0000 7f854ea00640 1 -- 172.21.15.96:0/4032444299 >> v1:172.21.15.123:6789/0 conn(0x7f855009b5d0 legacy=0x7f855015f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:14.976 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.974+0000 7f854ea00640 1 -- 172.21.15.96:0/4032444299 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f855006d7b0 con 0x7f8550162480 2024-05-28T06:20:14.976 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.974+0000 7f854ea00640 1 -- 172.21.15.96:0/4032444299 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f85400019f0 con 0x7f8550162480 2024-05-28T06:20:14.976 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.974+0000 7f854ea00640 1 -- 172.21.15.96:0/4032444299 >> v1:172.21.15.96:6789/0 conn(0x7f8550162480 legacy=0x7f855006ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:14.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.974+0000 7f854ea00640 1 --2- 172.21.15.96:0/4032444299 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8534003a00 0x7f8534003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:14.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.974+0000 7f854ea00640 1 --2- 172.21.15.96:0/4032444299 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8534004800 0x7f8534004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:14.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.974+0000 7f854ea00640 1 --2- 172.21.15.96:0/4032444299 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8534005210 0x7f853400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:14.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.974+0000 7f854ea00640 1 -- 172.21.15.96:0/4032444299 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f855006d4d0 con 0x7f8534004800 2024-05-28T06:20:14.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.974+0000 7f854ea00640 1 -- 172.21.15.96:0/4032444299 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f853400c160 con 0x7f8534005210 2024-05-28T06:20:14.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.974+0000 7f854ea00640 1 -- 172.21.15.96:0/4032444299 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f853400c360 con 0x7f8534003a00 2024-05-28T06:20:14.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.975+0000 7f854f400640 1 --2- 172.21.15.96:0/4032444299 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8534004800 0x7f8534004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:14.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.975+0000 7f854fe00640 1 --2- 172.21.15.96:0/4032444299 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8534003a00 0x7f8534003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:14.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.975+0000 7f854f400640 1 -- 172.21.15.96:0/4032444299 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8534003a00 msgr2=0x7f8534003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:14.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.975+0000 7f854f400640 1 --2- 172.21.15.96:0/4032444299 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8534003a00 0x7f8534003e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:14.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.975+0000 7f854f400640 1 -- 172.21.15.96:0/4032444299 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8534005210 msgr2=0x7f853400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:14.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.975+0000 7f8555200640 1 --2- 172.21.15.96:0/4032444299 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8534005210 0x7f853400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:14.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.975+0000 7f854f400640 1 --2- 172.21.15.96:0/4032444299 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8534005210 0x7f853400b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:14.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.975+0000 7f854f400640 1 -- 172.21.15.96:0/4032444299 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8534003520 con 0x7f8534004800 2024-05-28T06:20:14.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.975+0000 7f8555200640 1 -- 172.21.15.96:0/4032444299 reap_dead start 2024-05-28T06:20:14.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.976+0000 7f854f400640 1 --2- 172.21.15.96:0/4032444299 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8534004800 0x7f8534004c80 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7f853800ba30 tx=0x7f8534005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eeba4314522ea870 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:14.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.976+0000 7f854ea00640 1 -- 172.21.15.96:0/4032444299 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8538030070 con 0x7f8534004800 2024-05-28T06:20:14.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.976+0000 7f854ea00640 1 -- 172.21.15.96:0/4032444299 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f853802ade0 con 0x7f8534004800 2024-05-28T06:20:14.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.976+0000 7f854ea00640 1 -- 172.21.15.96:0/4032444299 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f853800b600 con 0x7f8534004800 2024-05-28T06:20:14.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.977+0000 7f8557200640 1 -- 172.21.15.96:0/4032444299 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8534004800 msgr2=0x7f8534004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:14.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.977+0000 7f8557200640 1 --2- 172.21.15.96:0/4032444299 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8534004800 0x7f8534004c80 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7f853800ba30 tx=0x7f8534005830 comp rx=0 tx=0).stop 2024-05-28T06:20:14.979 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.977+0000 7f8557200640 1 -- 172.21.15.96:0/4032444299 shutdown_connections 2024-05-28T06:20:14.979 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.977+0000 7f8557200640 1 --2- 172.21.15.96:0/4032444299 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8534004800 0x7f8534004c80 unknown :-1 s=CLOSED pgs=1598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:14.979 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.977+0000 7f8557200640 1 -- 172.21.15.96:0/4032444299 >> 172.21.15.96:0/4032444299 conn(0x7f85500d2430 msgr2=0x7f85500d2840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:14.979 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.977+0000 7f8557200640 1 -- 172.21.15.96:0/4032444299 shutdown_connections 2024-05-28T06:20:14.979 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.977+0000 7f8557200640 1 -- 172.21.15.96:0/4032444299 wait complete. 2024-05-28T06:20:14.979 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.979+0000 7f8557200640 1 Processor -- start 2024-05-28T06:20:14.979 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.979+0000 7f8557200640 1 -- start start 2024-05-28T06:20:14.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.980+0000 7f8557200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8534004800 0x7f8550150780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:14.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.980+0000 7f8557200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f855015ad20 0x7f855015ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:14.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.980+0000 7f8557200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f855015b180 0x7f855015cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:14.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.980+0000 7f8557200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f855009ba60 con 0x7f855015ad20 2024-05-28T06:20:14.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.980+0000 7f8557200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f85500d08a0 con 0x7f8534004800 2024-05-28T06:20:14.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.980+0000 7f8557200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f8550162860 con 0x7f855015b180 2024-05-28T06:20:14.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.980+0000 7f854f400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f855015ad20 0x7f855015ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:14.982 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.980+0000 7f854f400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f855015ad20 0x7f855015ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51058/0 (socket says 172.21.15.96:51058) 2024-05-28T06:20:14.982 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.980+0000 7f854f400640 1 -- 172.21.15.96:0/140901427 learned_addr learned my addr 172.21.15.96:0/140901427 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:20:14.982 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.980+0000 7f854f400640 1 -- 172.21.15.96:0/140901427 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f855015b180 msgr2=0x7f855015cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:14.982 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.980+0000 7f854f400640 1 --2- 172.21.15.96:0/140901427 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f855015b180 0x7f855015cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:14.982 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.980+0000 7f854f400640 1 -- 172.21.15.96:0/140901427 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8534004800 msgr2=0x7f8550150780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:14.982 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.980+0000 7f854f400640 1 --2- 172.21.15.96:0/140901427 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8534004800 0x7f8550150780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:14.982 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.980+0000 7f854f400640 1 -- 172.21.15.96:0/140901427 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8550150cc0 con 0x7f855015ad20 2024-05-28T06:20:14.982 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.981+0000 7f854f400640 1 --2- 172.21.15.96:0/140901427 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f855015ad20 0x7f855015ca50 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7f8538035d90 tx=0x7f85380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:14.982 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.981+0000 7f8547e00640 1 -- 172.21.15.96:0/140901427 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8538043030 con 0x7f855015ad20 2024-05-28T06:20:14.982 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.981+0000 7f8547e00640 1 -- 172.21.15.96:0/140901427 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f853802abe0 con 0x7f855015ad20 2024-05-28T06:20:14.982 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.981+0000 7f8547e00640 1 -- 172.21.15.96:0/140901427 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f853800b600 con 0x7f855015ad20 2024-05-28T06:20:14.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.981+0000 7f8557200640 1 -- 172.21.15.96:0/140901427 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f855015b690 con 0x7f855015ad20 2024-05-28T06:20:14.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.981+0000 7f8557200640 1 -- 172.21.15.96:0/140901427 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f855015bc50 con 0x7f855015ad20 2024-05-28T06:20:14.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.982+0000 7f8557200640 1 -- 172.21.15.96:0/140901427 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85500689e0 con 0x7f855015ad20 2024-05-28T06:20:14.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.983+0000 7f8547e00640 1 -- 172.21.15.96:0/140901427 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 34) ==== 92927+0+0 (secure 0 0 0) 0x7f8538046070 con 0x7f855015ad20 2024-05-28T06:20:14.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.984+0000 7f8547e00640 1 --2- 172.21.15.96:0/140901427 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f852c06ddf0 0x7f852c0702b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:14.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.984+0000 7f8547e00640 1 -- 172.21.15.96:0/140901427 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 4913+0+0 (secure 0 0 0) 0x7f85380ab400 con 0x7f855015ad20 2024-05-28T06:20:14.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.985+0000 7f854fe00640 1 --2- 172.21.15.96:0/140901427 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f852c06ddf0 0x7f852c0702b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:14.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.986+0000 7f854fe00640 1 --2- 172.21.15.96:0/140901427 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f852c06ddf0 0x7f852c0702b0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f8540007c40 tx=0x7f8540009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:14.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:14.988+0000 7f8547e00640 1 -- 172.21.15.96:0/140901427 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f853803c3a0 con 0x7f855015ad20 2024-05-28T06:20:15.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.186+0000 7f8557200640 1 -- 172.21.15.96:0/140901427 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f855009d9a0 con 0x7f855015ad20 2024-05-28T06:20:15.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.187+0000 7f8547e00640 1 -- 172.21.15.96:0/140901427 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v99) ==== 93+0+10 (secure 0 0 0) 0x7f853807be20 con 0x7f855015ad20 2024-05-28T06:20:15.184 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 1 2024-05-28T06:20:15.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.190+0000 7f8557200640 1 -- 172.21.15.96:0/140901427 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f852c06ddf0 msgr2=0x7f852c0702b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:15.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.190+0000 7f8557200640 1 --2- 172.21.15.96:0/140901427 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f852c06ddf0 0x7f852c0702b0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f8540007c40 tx=0x7f8540009040 comp rx=0 tx=0).stop 2024-05-28T06:20:15.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.190+0000 7f8557200640 1 -- 172.21.15.96:0/140901427 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f855015ad20 msgr2=0x7f855015ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:15.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.190+0000 7f8557200640 1 --2- 172.21.15.96:0/140901427 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f855015ad20 0x7f855015ca50 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7f8538035d90 tx=0x7f85380046c0 comp rx=0 tx=0).stop 2024-05-28T06:20:15.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.190+0000 7f8557200640 1 -- 172.21.15.96:0/140901427 shutdown_connections 2024-05-28T06:20:15.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.190+0000 7f8557200640 1 --2- 172.21.15.96:0/140901427 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f852c06ddf0 0x7f852c0702b0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:15.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.190+0000 7f8557200640 1 --2- 172.21.15.96:0/140901427 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f855015b180 0x7f855015cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:15.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.190+0000 7f8557200640 1 --2- 172.21.15.96:0/140901427 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f855015ad20 0x7f855015ca50 unknown :-1 s=CLOSED pgs=1599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:15.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.190+0000 7f8557200640 1 --2- 172.21.15.96:0/140901427 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8534004800 0x7f8550150780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:15.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.190+0000 7f8557200640 1 -- 172.21.15.96:0/140901427 >> 172.21.15.96:0/140901427 conn(0x7f85500d2430 msgr2=0x7f85500d0c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:15.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.190+0000 7f8557200640 1 -- 172.21.15.96:0/140901427 shutdown_connections 2024-05-28T06:20:15.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.190+0000 7f8557200640 1 -- 172.21.15.96:0/140901427 wait complete. 2024-05-28T06:20:15.206 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-05-28T06:20:15.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.407+0000 7fdceda00640 1 Processor -- start 2024-05-28T06:20:15.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.408+0000 7fdceda00640 1 -- start start 2024-05-28T06:20:15.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.409+0000 7fdceda00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fdce815e870 0x7fdce80d0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:15.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.409+0000 7fdceda00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdce815ec50 con 0x7fdce80d14d0 2024-05-28T06:20:15.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.409+0000 7fdceda00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdce80cff80 con 0x7fdce80cfa40 2024-05-28T06:20:15.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.409+0000 7fdceda00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fdce80d2150 con 0x7fdce815e870 2024-05-28T06:20:15.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.409+0000 7fdce7e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fdce815e870 0x7fdce80d0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:15.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.409+0000 7fdce7400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fdce80d14d0 0x7fdce80cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:33746/0 (socket says 172.21.15.96:33746) 2024-05-28T06:20:15.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.410+0000 7fdce7400640 1 -- 172.21.15.96:0/4061681435 learned_addr learned my addr 172.21.15.96:0/4061681435 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:20:15.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.410+0000 7fdce7e00640 1 -- 172.21.15.96:0/4061681435 >> v1:172.21.15.123:6789/0 conn(0x7fdce80cfa40 legacy=0x7fdce80d25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:15.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.410+0000 7fdce7e00640 1 -- 172.21.15.96:0/4061681435 >> v1:172.21.15.96:6789/0 conn(0x7fdce80d14d0 legacy=0x7fdce80cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:15.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.410+0000 7fdce7e00640 1 -- 172.21.15.96:0/4061681435 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdce80d0260 con 0x7fdce815e870 2024-05-28T06:20:15.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.410+0000 7fdce6a00640 1 -- 172.21.15.96:0/4061681435 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 887351669 0 0) 0x7fdce815ec50 con 0x7fdce80d14d0 2024-05-28T06:20:15.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.411+0000 7fdce7e00640 1 --2- 172.21.15.96:0/4061681435 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdce815e870 0x7fdce80d0f90 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7fdcd80097a0 tx=0x7fdcd802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be13ab12823cbb02 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:15.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.411+0000 7fdce6a00640 1 -- 172.21.15.96:0/4061681435 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcd8037070 con 0x7fdce815e870 2024-05-28T06:20:15.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.411+0000 7fdce6a00640 1 -- 172.21.15.96:0/4061681435 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdcd8032440 con 0x7fdce815e870 2024-05-28T06:20:15.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.412+0000 7fdce6a00640 1 -- 172.21.15.96:0/4061681435 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcd8032810 con 0x7fdce815e870 2024-05-28T06:20:15.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.412+0000 7fdceda00640 1 -- 172.21.15.96:0/4061681435 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdce815e870 msgr2=0x7fdce80d0f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:15.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.412+0000 7fdceda00640 1 --2- 172.21.15.96:0/4061681435 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdce815e870 0x7fdce80d0f90 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7fdcd80097a0 tx=0x7fdcd802a1e0 comp rx=0 tx=0).stop 2024-05-28T06:20:15.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.412+0000 7fdceda00640 1 -- 172.21.15.96:0/4061681435 shutdown_connections 2024-05-28T06:20:15.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.412+0000 7fdceda00640 1 --2- 172.21.15.96:0/4061681435 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdce815e870 0x7fdce80d0f90 unknown :-1 s=CLOSED pgs=1601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:15.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.412+0000 7fdceda00640 1 -- 172.21.15.96:0/4061681435 >> 172.21.15.96:0/4061681435 conn(0x7fdce808bfc0 msgr2=0x7fdce80d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:15.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.412+0000 7fdceda00640 1 -- 172.21.15.96:0/4061681435 shutdown_connections 2024-05-28T06:20:15.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.412+0000 7fdceda00640 1 -- 172.21.15.96:0/4061681435 wait complete. 2024-05-28T06:20:15.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.414+0000 7fdceda00640 1 Processor -- start 2024-05-28T06:20:15.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.414+0000 7fdceda00640 1 -- start start 2024-05-28T06:20:15.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.415+0000 7fdceda00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdce80cfa40 0x7fdce8150ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:15.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.415+0000 7fdceda00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdce80d14d0 0x7fdce8161040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:15.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.415+0000 7fdceda00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdce80c9e40 0x7fdce8161580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:15.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.415+0000 7fdceda00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fdce80d4fb0 con 0x7fdce80c9e40 2024-05-28T06:20:15.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.415+0000 7fdceda00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fdce80d0820 con 0x7fdce80d14d0 2024-05-28T06:20:15.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.415+0000 7fdceda00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fdce80d2cf0 con 0x7fdce80cfa40 2024-05-28T06:20:15.414 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.415+0000 7fdced000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdce80c9e40 0x7fdce8161580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:15.414 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.415+0000 7fdced000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdce80c9e40 0x7fdce8161580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51074/0 (socket says 172.21.15.96:51074) 2024-05-28T06:20:15.414 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.415+0000 7fdced000640 1 -- 172.21.15.96:0/3248725197 learned_addr learned my addr 172.21.15.96:0/3248725197 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:20:15.414 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.416+0000 7fdce7e00640 1 --2- 172.21.15.96:0/3248725197 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdce80cfa40 0x7fdce8150ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:15.414 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.416+0000 7fdce7400640 1 --2- 172.21.15.96:0/3248725197 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdce80d14d0 0x7fdce8161040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:15.414 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.416+0000 7fdced000640 1 -- 172.21.15.96:0/3248725197 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdce80cfa40 msgr2=0x7fdce8150ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:15.414 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.416+0000 7fdced000640 1 --2- 172.21.15.96:0/3248725197 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdce80cfa40 0x7fdce8150ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:15.414 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.416+0000 7fdced000640 1 -- 172.21.15.96:0/3248725197 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdce80d14d0 msgr2=0x7fdce8161040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:15.414 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.416+0000 7fdced000640 1 --2- 172.21.15.96:0/3248725197 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdce80d14d0 0x7fdce8161040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:15.414 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.416+0000 7fdced000640 1 -- 172.21.15.96:0/3248725197 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdce80c8da0 con 0x7fdce80c9e40 2024-05-28T06:20:15.414 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.416+0000 7fdced000640 1 --2- 172.21.15.96:0/3248725197 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdce80c9e40 0x7fdce8161580 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7fdce000c410 tx=0x7fdce000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:15.414 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.416+0000 7fdcdfe00640 1 -- 172.21.15.96:0/3248725197 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdce0017070 con 0x7fdce80c9e40 2024-05-28T06:20:15.415 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.416+0000 7fdcdfe00640 1 -- 172.21.15.96:0/3248725197 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdce0007230 con 0x7fdce80c9e40 2024-05-28T06:20:15.415 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.416+0000 7fdceda00640 1 -- 172.21.15.96:0/3248725197 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdce80c9080 con 0x7fdce80c9e40 2024-05-28T06:20:15.415 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.416+0000 7fdcdfe00640 1 -- 172.21.15.96:0/3248725197 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdce0007660 con 0x7fdce80c9e40 2024-05-28T06:20:15.415 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.416+0000 7fdceda00640 1 -- 172.21.15.96:0/3248725197 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdce80c9640 con 0x7fdce80c9e40 2024-05-28T06:20:15.415 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.418+0000 7fdceda00640 1 -- 172.21.15.96:0/3248725197 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdce80689e0 con 0x7fdce80c9e40 2024-05-28T06:20:15.415 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.419+0000 7fdcdfe00640 1 -- 172.21.15.96:0/3248725197 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 34) ==== 92927+0+0 (secure 0 0 0) 0x7fdce001a070 con 0x7fdce80c9e40 2024-05-28T06:20:15.416 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.420+0000 7fdcdfe00640 1 --2- 172.21.15.96:0/3248725197 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdcc406ddc0 0x7fdcc4070280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:15.416 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.420+0000 7fdcdfe00640 1 -- 172.21.15.96:0/3248725197 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 4913+0+0 (secure 0 0 0) 0x7fdce0089c10 con 0x7fdce80c9e40 2024-05-28T06:20:15.417 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.420+0000 7fdce7e00640 1 --2- 172.21.15.96:0/3248725197 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdcc406ddc0 0x7fdcc4070280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:15.418 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.422+0000 7fdce7e00640 1 --2- 172.21.15.96:0/3248725197 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdcc406ddc0 0x7fdcc4070280 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fdcd8034000 tx=0x7fdcd802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:15.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.424+0000 7fdcdfe00640 1 -- 172.21.15.96:0/3248725197 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fdce0059a90 con 0x7fdce80c9e40 2024-05-28T06:20:15.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.625+0000 7fdceda00640 1 -- 172.21.15.96:0/3248725197 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fdce80ba500 con 0x7fdce80c9e40 2024-05-28T06:20:15.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.627+0000 7fdcdfe00640 1 -- 172.21.15.96:0/3248725197 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v99) ==== 84+0+65 (secure 0 0 0) 0x7fdce0059390 con 0x7fdce80c9e40 2024-05-28T06:20:15.624 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:20:15.624 INFO:teuthology.orchestra.run.smithi096.stdout:[ 2024-05-28T06:20:15.624 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:20:15.624 INFO:teuthology.orchestra.run.smithi096.stdout: "poolnum": 1, 2024-05-28T06:20:15.624 INFO:teuthology.orchestra.run.smithi096.stdout: "poolname": ".mgr" 2024-05-28T06:20:15.624 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:20:15.624 INFO:teuthology.orchestra.run.smithi096.stdout:] 2024-05-28T06:20:15.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.629+0000 7fdceda00640 1 -- 172.21.15.96:0/3248725197 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdcc406ddc0 msgr2=0x7fdcc4070280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:15.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.629+0000 7fdceda00640 1 --2- 172.21.15.96:0/3248725197 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdcc406ddc0 0x7fdcc4070280 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fdcd8034000 tx=0x7fdcd802e000 comp rx=0 tx=0).stop 2024-05-28T06:20:15.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.629+0000 7fdceda00640 1 -- 172.21.15.96:0/3248725197 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdce80c9e40 msgr2=0x7fdce8161580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:15.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.629+0000 7fdceda00640 1 --2- 172.21.15.96:0/3248725197 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdce80c9e40 0x7fdce8161580 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7fdce000c410 tx=0x7fdce000c8d0 comp rx=0 tx=0).stop 2024-05-28T06:20:15.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.629+0000 7fdceda00640 1 -- 172.21.15.96:0/3248725197 shutdown_connections 2024-05-28T06:20:15.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.629+0000 7fdceda00640 1 --2- 172.21.15.96:0/3248725197 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdcc406ddc0 0x7fdcc4070280 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:15.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.629+0000 7fdceda00640 1 --2- 172.21.15.96:0/3248725197 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdce80c9e40 0x7fdce8161580 unknown :-1 s=CLOSED pgs=1602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:15.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.629+0000 7fdceda00640 1 --2- 172.21.15.96:0/3248725197 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdce80d14d0 0x7fdce8161040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:15.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.629+0000 7fdceda00640 1 --2- 172.21.15.96:0/3248725197 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdce80cfa40 0x7fdce8150ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:15.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.629+0000 7fdceda00640 1 -- 172.21.15.96:0/3248725197 >> 172.21.15.96:0/3248725197 conn(0x7fdce808bfc0 msgr2=0x7fdce80b3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:15.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.629+0000 7fdceda00640 1 -- 172.21.15.96:0/3248725197 shutdown_connections 2024-05-28T06:20:15.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.630+0000 7fdceda00640 1 -- 172.21.15.96:0/3248725197 wait complete. 2024-05-28T06:20:15.646 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-05-28T06:20:15.646 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-05-28T06:20:15.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.858+0000 7f06e4000640 1 Processor -- start 2024-05-28T06:20:15.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.858+0000 7f06e4000640 1 -- start start 2024-05-28T06:20:15.858 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.860+0000 7f06e4000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f06dc162e90 0x7f06dc0ce540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:15.858 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.860+0000 7f06e4000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f06dc06e1b0 0x7f06dc0c3300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:15.858 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.860+0000 7f06e4000640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06dc0b3880 con 0x7f06dc163270 2024-05-28T06:20:15.858 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.860+0000 7f06e4000640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f06dc0b3b80 con 0x7f06dc162e90 2024-05-28T06:20:15.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.860+0000 7f06e4000640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f06dc0ceb20 con 0x7f06dc06e1b0 2024-05-28T06:20:15.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.860+0000 7f06e1600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f06dc162e90 0x7f06dc0ce540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:15.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.860+0000 7f06e1600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f06dc162e90 0x7f06dc0ce540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51088/0 (socket says 172.21.15.96:51088) 2024-05-28T06:20:15.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.860+0000 7f06e1600640 1 -- 172.21.15.96:0/2277447081 learned_addr learned my addr 172.21.15.96:0/2277447081 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:20:15.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.861+0000 7f06e1600640 1 -- 172.21.15.96:0/2277447081 >> v2:172.21.15.123:3300/0 conn(0x7f06dc06e1b0 msgr2=0x7f06dc0c3300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:15.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.861+0000 7f06e2000640 1 --2- 172.21.15.96:0/2277447081 >> v2:172.21.15.123:3300/0 conn(0x7f06dc06e1b0 0x7f06dc0c3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:15.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.861+0000 7f06e1600640 1 --2- 172.21.15.96:0/2277447081 >> v2:172.21.15.123:3300/0 conn(0x7f06dc06e1b0 0x7f06dc0c3300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:15.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.861+0000 7f06e1600640 1 -- 172.21.15.96:0/2277447081 >> v1:172.21.15.123:6789/0 conn(0x7f06dc163270 legacy=0x7f06dc0cd630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:15.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.861+0000 7f06e1600640 1 -- 172.21.15.96:0/2277447081 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06dc0cee00 con 0x7f06dc162e90 2024-05-28T06:20:15.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.861+0000 7f06e2000640 1 --2- 172.21.15.96:0/2277447081 >> v2:172.21.15.123:3300/0 conn(0x7f06dc06e1b0 0x7f06dc0c3300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:20:15.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.862+0000 7f06e1600640 1 --2- 172.21.15.96:0/2277447081 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f06dc162e90 0x7f06dc0ce540 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7f06dc0ce4c0 tx=0x7f06d002a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c506e87cfcfa85b9 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:15.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.862+0000 7f06dbe00640 1 -- 172.21.15.96:0/2277447081 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06d0037070 con 0x7f06dc162e90 2024-05-28T06:20:15.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.863+0000 7f06dbe00640 1 -- 172.21.15.96:0/2277447081 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06d0032440 con 0x7f06dc162e90 2024-05-28T06:20:15.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.863+0000 7f06dbe00640 1 -- 172.21.15.96:0/2277447081 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06d0032810 con 0x7f06dc162e90 2024-05-28T06:20:15.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.864+0000 7f06e4000640 1 -- 172.21.15.96:0/2277447081 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f06dc162e90 msgr2=0x7f06dc0ce540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:15.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.864+0000 7f06e4000640 1 --2- 172.21.15.96:0/2277447081 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f06dc162e90 0x7f06dc0ce540 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7f06dc0ce4c0 tx=0x7f06d002a230 comp rx=0 tx=0).stop 2024-05-28T06:20:15.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.864+0000 7f06e4000640 1 -- 172.21.15.96:0/2277447081 shutdown_connections 2024-05-28T06:20:15.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.864+0000 7f06e4000640 1 --2- 172.21.15.96:0/2277447081 >> v2:172.21.15.123:3300/0 conn(0x7f06dc06e1b0 0x7f06dc0c3300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:15.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.864+0000 7f06e4000640 1 --2- 172.21.15.96:0/2277447081 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f06dc162e90 0x7f06dc0ce540 unknown :-1 s=CLOSED pgs=1603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:15.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.864+0000 7f06e4000640 1 -- 172.21.15.96:0/2277447081 >> 172.21.15.96:0/2277447081 conn(0x7f06dc0b20a0 msgr2=0x7f06dc0b24b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:15.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.864+0000 7f06e4000640 1 -- 172.21.15.96:0/2277447081 shutdown_connections 2024-05-28T06:20:15.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.864+0000 7f06e4000640 1 -- 172.21.15.96:0/2277447081 wait complete. 2024-05-28T06:20:15.867 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.866+0000 7f06e4000640 1 Processor -- start 2024-05-28T06:20:15.867 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.866+0000 7f06e4000640 1 -- start start 2024-05-28T06:20:15.867 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.867+0000 7f06e4000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f06dc06e1b0 0x7f06dc15f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:15.867 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.867+0000 7f06e4000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f06dc162e90 0x7f06dc0c7400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:15.867 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.867+0000 7f06e4000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f06dc163270 0x7f06dc0c9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:15.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.867+0000 7f06e4000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f06dc06e7c0 con 0x7f06dc162e90 2024-05-28T06:20:15.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.867+0000 7f06e4000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f06dc06e640 con 0x7f06dc06e1b0 2024-05-28T06:20:15.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.867+0000 7f06e4000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f06dc0c3840 con 0x7f06dc163270 2024-05-28T06:20:15.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.867+0000 7f06e0c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f06dc162e90 0x7f06dc0c7400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:15.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.867+0000 7f06e0c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f06dc162e90 0x7f06dc0c7400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51094/0 (socket says 172.21.15.96:51094) 2024-05-28T06:20:15.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.867+0000 7f06e0c00640 1 -- 172.21.15.96:0/331414644 learned_addr learned my addr 172.21.15.96:0/331414644 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:20:15.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.868+0000 7f06e2000640 1 --2- 172.21.15.96:0/331414644 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f06dc163270 0x7f06dc0c9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:15.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.868+0000 7f06e1600640 1 --2- 172.21.15.96:0/331414644 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f06dc06e1b0 0x7f06dc15f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:15.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.868+0000 7f06e0c00640 1 -- 172.21.15.96:0/331414644 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f06dc163270 msgr2=0x7f06dc0c9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:15.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.868+0000 7f06e0c00640 1 --2- 172.21.15.96:0/331414644 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f06dc163270 0x7f06dc0c9cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:15.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.868+0000 7f06e0c00640 1 -- 172.21.15.96:0/331414644 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f06dc06e1b0 msgr2=0x7f06dc15f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:15.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.868+0000 7f06e0c00640 1 --2- 172.21.15.96:0/331414644 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f06dc06e1b0 0x7f06dc15f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:15.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.868+0000 7f06e0c00640 1 -- 172.21.15.96:0/331414644 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06dc0c7b00 con 0x7f06dc162e90 2024-05-28T06:20:15.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.868+0000 7f06e2000640 1 --2- 172.21.15.96:0/331414644 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f06dc163270 0x7f06dc0c9cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:20:15.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.868+0000 7f06e1600640 1 --2- 172.21.15.96:0/331414644 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f06dc06e1b0 0x7f06dc15f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:20:15.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.868+0000 7f06e0c00640 1 --2- 172.21.15.96:0/331414644 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f06dc162e90 0x7f06dc0c7400 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7f06c800b420 tx=0x7f06c800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:15.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.868+0000 7f06d9600640 1 -- 172.21.15.96:0/331414644 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06c8011830 con 0x7f06dc162e90 2024-05-28T06:20:15.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.868+0000 7f06d9600640 1 -- 172.21.15.96:0/331414644 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06c8016070 con 0x7f06dc162e90 2024-05-28T06:20:15.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.868+0000 7f06e4000640 1 -- 172.21.15.96:0/331414644 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06dc0c7de0 con 0x7f06dc162e90 2024-05-28T06:20:15.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.869+0000 7f06d9600640 1 -- 172.21.15.96:0/331414644 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06c8011c10 con 0x7f06dc162e90 2024-05-28T06:20:15.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.869+0000 7f06e4000640 1 -- 172.21.15.96:0/331414644 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06dc0c83a0 con 0x7f06dc162e90 2024-05-28T06:20:15.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.870+0000 7f06e4000640 1 -- 172.21.15.96:0/331414644 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06dc0757d0 con 0x7f06dc162e90 2024-05-28T06:20:15.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.871+0000 7f06d9600640 1 -- 172.21.15.96:0/331414644 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 34) ==== 92927+0+0 (secure 0 0 0) 0x7f06c801b460 con 0x7f06dc162e90 2024-05-28T06:20:15.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.872+0000 7f06d9600640 1 --2- 172.21.15.96:0/331414644 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f06bc06ddc0 0x7f06bc070280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:15.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.872+0000 7f06d9600640 1 -- 172.21.15.96:0/331414644 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 4913+0+0 (secure 0 0 0) 0x7f06c808a730 con 0x7f06dc162e90 2024-05-28T06:20:15.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.873+0000 7f06e1600640 1 --2- 172.21.15.96:0/331414644 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f06bc06ddc0 0x7f06bc070280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:15.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.874+0000 7f06e1600640 1 --2- 172.21.15.96:0/331414644 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f06bc06ddc0 0x7f06bc070280 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f06d0034000 tx=0x7f06d002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:15.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:15.876+0000 7f06d9600640 1 -- 172.21.15.96:0/331414644 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f06c8020020 con 0x7f06dc162e90 2024-05-28T06:20:16.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:16.077+0000 7f06e4000640 1 -- 172.21.15.96:0/331414644 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f06dc0689e0 con 0x7f06dc162e90 2024-05-28T06:20:16.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:16.879+0000 7f06d9600640 1 -- 172.21.15.96:0/331414644 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v100) ==== 138+0+0 (secure 0 0 0) 0x7f06c805a210 con 0x7f06dc162e90 2024-05-28T06:20:16.876 INFO:teuthology.orchestra.run.smithi096.stderr:pool 'cephfs_metadata' created 2024-05-28T06:20:16.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:16.883+0000 7f06e4000640 1 -- 172.21.15.96:0/331414644 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f06bc06ddc0 msgr2=0x7f06bc070280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:16.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:16.883+0000 7f06e4000640 1 --2- 172.21.15.96:0/331414644 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f06bc06ddc0 0x7f06bc070280 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f06d0034000 tx=0x7f06d002e000 comp rx=0 tx=0).stop 2024-05-28T06:20:16.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:16.883+0000 7f06e4000640 1 -- 172.21.15.96:0/331414644 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f06dc162e90 msgr2=0x7f06dc0c7400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:16.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:16.883+0000 7f06e4000640 1 --2- 172.21.15.96:0/331414644 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f06dc162e90 0x7f06dc0c7400 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7f06c800b420 tx=0x7f06c800b8e0 comp rx=0 tx=0).stop 2024-05-28T06:20:16.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:16.883+0000 7f06e4000640 1 -- 172.21.15.96:0/331414644 shutdown_connections 2024-05-28T06:20:16.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:16.883+0000 7f06e4000640 1 --2- 172.21.15.96:0/331414644 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f06bc06ddc0 0x7f06bc070280 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:16.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:16.883+0000 7f06e4000640 1 --2- 172.21.15.96:0/331414644 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f06dc163270 0x7f06dc0c9cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:16.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:16.883+0000 7f06e4000640 1 --2- 172.21.15.96:0/331414644 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f06dc162e90 0x7f06dc0c7400 unknown :-1 s=CLOSED pgs=1604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:16.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:16.883+0000 7f06e4000640 1 --2- 172.21.15.96:0/331414644 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f06dc06e1b0 0x7f06dc15f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:16.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:16.883+0000 7f06e4000640 1 -- 172.21.15.96:0/331414644 >> 172.21.15.96:0/331414644 conn(0x7f06dc0b20a0 msgr2=0x7f06dc0b03f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:16.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:16.883+0000 7f06e4000640 1 -- 172.21.15.96:0/331414644 shutdown_connections 2024-05-28T06:20:16.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:16.884+0000 7f06e4000640 1 -- 172.21.15.96:0/331414644 wait complete. 2024-05-28T06:20:16.911 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-05-28T06:20:17.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.104+0000 7fa69ba00640 1 Processor -- start 2024-05-28T06:20:17.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.104+0000 7fa69ba00640 1 -- start start 2024-05-28T06:20:17.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.105+0000 7fa69ba00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fa6940b32c0 0x7fa6940b3700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:17.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.105+0000 7fa69ba00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6940d0060 con 0x7fa6940d0640 2024-05-28T06:20:17.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.105+0000 7fa69ba00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa69415cb10 con 0x7fa69415d5d0 2024-05-28T06:20:17.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.105+0000 7fa69ba00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fa6940adbb0 con 0x7fa6940b32c0 2024-05-28T06:20:17.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.105+0000 7fa699000640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fa6940d0640 0x7fa6940d0a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:33760/0 (socket says 172.21.15.96:33760) 2024-05-28T06:20:17.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.105+0000 7fa699000640 1 -- 172.21.15.96:0/80310768 learned_addr learned my addr 172.21.15.96:0/80310768 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:20:17.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.106+0000 7fa699a00640 1 --2- 172.21.15.96:0/80310768 >> v2:172.21.15.123:3300/0 conn(0x7fa6940b32c0 0x7fa6940b3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:17.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.106+0000 7fa693e00640 1 -- 172.21.15.96:0/80310768 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 42757791 0 0) 0x7fa6940d0060 con 0x7fa6940d0640 2024-05-28T06:20:17.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.106+0000 7fa693e00640 1 -- 172.21.15.96:0/80310768 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa678003520 con 0x7fa6940d0640 2024-05-28T06:20:17.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.106+0000 7fa693e00640 1 -- 172.21.15.96:0/80310768 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1657256085 0 0) 0x7fa678003520 con 0x7fa6940d0640 2024-05-28T06:20:17.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.106+0000 7fa693e00640 1 -- 172.21.15.96:0/80310768 >> v2:172.21.15.123:3300/0 conn(0x7fa6940b32c0 msgr2=0x7fa6940b3700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:17.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.106+0000 7fa693e00640 1 --2- 172.21.15.96:0/80310768 >> v2:172.21.15.123:3300/0 conn(0x7fa6940b32c0 0x7fa6940b3700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:17.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.106+0000 7fa693e00640 1 -- 172.21.15.96:0/80310768 >> v1:172.21.15.123:6789/0 conn(0x7fa69415d5d0 legacy=0x7fa6940cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:17.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.106+0000 7fa693e00640 1 -- 172.21.15.96:0/80310768 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6940ad5f0 con 0x7fa6940d0640 2024-05-28T06:20:17.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.106+0000 7fa693e00640 1 -- 172.21.15.96:0/80310768 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7fa684001670 con 0x7fa6940d0640 2024-05-28T06:20:17.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.106+0000 7fa693e00640 1 -- 172.21.15.96:0/80310768 >> v1:172.21.15.96:6789/0 conn(0x7fa6940d0640 legacy=0x7fa6940d0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:17.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.107+0000 7fa693e00640 1 --2- 172.21.15.96:0/80310768 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6780039b0 0x7fa678003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:17.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.107+0000 7fa693e00640 1 --2- 172.21.15.96:0/80310768 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa678004800 0x7fa678004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:17.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.107+0000 7fa693e00640 1 --2- 172.21.15.96:0/80310768 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6780051c0 0x7fa67800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:17.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.107+0000 7fa693e00640 1 -- 172.21.15.96:0/80310768 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa67800c130 con 0x7fa6780051c0 2024-05-28T06:20:17.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.107+0000 7fa693e00640 1 -- 172.21.15.96:0/80310768 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa67800c2e0 con 0x7fa6780039b0 2024-05-28T06:20:17.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.107+0000 7fa693e00640 1 -- 172.21.15.96:0/80310768 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fa67800c4e0 con 0x7fa678004800 2024-05-28T06:20:17.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.107+0000 7fa699a00640 1 --2- 172.21.15.96:0/80310768 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6780051c0 0x7fa67800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:17.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.107+0000 7fa699000640 1 --2- 172.21.15.96:0/80310768 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6780039b0 0x7fa678003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:17.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.107+0000 7fa699a00640 1 -- 172.21.15.96:0/80310768 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa678004800 msgr2=0x7fa678004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:17.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.107+0000 7fa699a00640 1 --2- 172.21.15.96:0/80310768 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa678004800 0x7fa678004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:17.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.107+0000 7fa699a00640 1 -- 172.21.15.96:0/80310768 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6780039b0 msgr2=0x7fa678003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:17.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.107+0000 7fa699a00640 1 --2- 172.21.15.96:0/80310768 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6780039b0 0x7fa678003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:17.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.107+0000 7fa699a00640 1 -- 172.21.15.96:0/80310768 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa678003520 con 0x7fa6780051c0 2024-05-28T06:20:17.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.107+0000 7fa699000640 1 --2- 172.21.15.96:0/80310768 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6780039b0 0x7fa678003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:20:17.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.108+0000 7fa699a00640 1 -- 172.21.15.96:0/80310768 reap_dead start 2024-05-28T06:20:17.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.108+0000 7fa699a00640 1 --2- 172.21.15.96:0/80310768 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6780051c0 0x7fa67800b9c0 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7fa68c009fc0 tx=0x7fa678005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4927e5248fcc00b server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:17.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.109+0000 7fa693e00640 1 -- 172.21.15.96:0/80310768 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa68c035020 con 0x7fa6780051c0 2024-05-28T06:20:17.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.109+0000 7fa693e00640 1 -- 172.21.15.96:0/80310768 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa68c003bd0 con 0x7fa6780051c0 2024-05-28T06:20:17.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.109+0000 7fa693e00640 1 -- 172.21.15.96:0/80310768 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa68c009c10 con 0x7fa6780051c0 2024-05-28T06:20:17.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.109+0000 7fa69ba00640 1 -- 172.21.15.96:0/80310768 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6780051c0 msgr2=0x7fa67800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:17.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.109+0000 7fa69ba00640 1 --2- 172.21.15.96:0/80310768 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6780051c0 0x7fa67800b9c0 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7fa68c009fc0 tx=0x7fa678005830 comp rx=0 tx=0).stop 2024-05-28T06:20:17.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.109+0000 7fa69ba00640 1 -- 172.21.15.96:0/80310768 shutdown_connections 2024-05-28T06:20:17.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.109+0000 7fa69ba00640 1 --2- 172.21.15.96:0/80310768 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6780051c0 0x7fa67800b9c0 unknown :-1 s=CLOSED pgs=1606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:17.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.109+0000 7fa69ba00640 1 -- 172.21.15.96:0/80310768 >> 172.21.15.96:0/80310768 conn(0x7fa69406bd20 msgr2=0x7fa69406c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:17.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.109+0000 7fa69ba00640 1 -- 172.21.15.96:0/80310768 shutdown_connections 2024-05-28T06:20:17.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.109+0000 7fa69ba00640 1 -- 172.21.15.96:0/80310768 wait complete. 2024-05-28T06:20:17.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.111+0000 7fa69ba00640 1 Processor -- start 2024-05-28T06:20:17.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.111+0000 7fa69ba00640 1 -- start start 2024-05-28T06:20:17.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.112+0000 7fa69ba00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6780051c0 0x7fa6941f3a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:17.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.112+0000 7fa69ba00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa694154e30 0x7fa69406ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:17.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.112+0000 7fa69ba00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa69406ef50 0x7fa6940b17c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:17.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.112+0000 7fa69ba00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa69415d0b0 con 0x7fa69406ef50 2024-05-28T06:20:17.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.112+0000 7fa69ba00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa69415cdd0 con 0x7fa6780051c0 2024-05-28T06:20:17.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.112+0000 7fa69ba00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fa69415da50 con 0x7fa694154e30 2024-05-28T06:20:17.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.113+0000 7fa699a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa69406ef50 0x7fa6940b17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:17.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.113+0000 7fa699a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa69406ef50 0x7fa6940b17c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51114/0 (socket says 172.21.15.96:51114) 2024-05-28T06:20:17.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.113+0000 7fa699a00640 1 -- 172.21.15.96:0/4240776080 learned_addr learned my addr 172.21.15.96:0/4240776080 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:20:17.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.113+0000 7fa699000640 1 --2- 172.21.15.96:0/4240776080 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6780051c0 0x7fa6941f3a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:17.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.113+0000 7fa68be00640 1 --2- 172.21.15.96:0/4240776080 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa694154e30 0x7fa69406ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:17.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.113+0000 7fa699a00640 1 -- 172.21.15.96:0/4240776080 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa694154e30 msgr2=0x7fa69406ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:17.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.113+0000 7fa699a00640 1 --2- 172.21.15.96:0/4240776080 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa694154e30 0x7fa69406ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:17.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.113+0000 7fa699a00640 1 -- 172.21.15.96:0/4240776080 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6780051c0 msgr2=0x7fa6941f3a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:17.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.113+0000 7fa699a00640 1 --2- 172.21.15.96:0/4240776080 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6780051c0 0x7fa6941f3a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:17.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.113+0000 7fa699a00640 1 -- 172.21.15.96:0/4240776080 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa694155fc0 con 0x7fa69406ef50 2024-05-28T06:20:17.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.113+0000 7fa699a00640 1 --2- 172.21.15.96:0/4240776080 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa69406ef50 0x7fa6940b17c0 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7fa68c00e200 tx=0x7fa68c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:17.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.114+0000 7fa691600640 1 -- 172.21.15.96:0/4240776080 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa68c035030 con 0x7fa69406ef50 2024-05-28T06:20:17.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.114+0000 7fa691600640 1 -- 172.21.15.96:0/4240776080 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa68c043070 con 0x7fa69406ef50 2024-05-28T06:20:17.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.114+0000 7fa69ba00640 1 -- 172.21.15.96:0/4240776080 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6940b0310 con 0x7fa69406ef50 2024-05-28T06:20:17.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.114+0000 7fa691600640 1 -- 172.21.15.96:0/4240776080 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa68c02d070 con 0x7fa69406ef50 2024-05-28T06:20:17.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.114+0000 7fa69ba00640 1 -- 172.21.15.96:0/4240776080 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6940b08d0 con 0x7fa69406ef50 2024-05-28T06:20:17.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.115+0000 7fa69ba00640 1 -- 172.21.15.96:0/4240776080 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa69406f3d0 con 0x7fa69406ef50 2024-05-28T06:20:17.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.116+0000 7fa691600640 1 -- 172.21.15.96:0/4240776080 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 34) ==== 92927+0+0 (secure 0 0 0) 0x7fa68c005d10 con 0x7fa69406ef50 2024-05-28T06:20:17.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.117+0000 7fa691600640 1 --2- 172.21.15.96:0/4240776080 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa67006ddf0 0x7fa6700702b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:17.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.117+0000 7fa691600640 1 -- 172.21.15.96:0/4240776080 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 5283+0+0 (secure 0 0 0) 0x7fa68c0ab570 con 0x7fa69406ef50 2024-05-28T06:20:17.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.118+0000 7fa699000640 1 --2- 172.21.15.96:0/4240776080 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa67006ddf0 0x7fa6700702b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:17.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.119+0000 7fa699000640 1 --2- 172.21.15.96:0/4240776080 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa67006ddf0 0x7fa6700702b0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fa684001920 tx=0x7fa684009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:17.119 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.123+0000 7fa691600640 1 -- 172.21.15.96:0/4240776080 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa694155fc0 con 0x7fa69406ef50 2024-05-28T06:20:17.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.324+0000 7fa69ba00640 1 -- 172.21.15.96:0/4240776080 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fa694071ef0 con 0x7fa69406ef50 2024-05-28T06:20:17.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.885+0000 7fa691600640 1 -- 172.21.15.96:0/4240776080 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v101) ==== 170+0+0 (secure 0 0 0) 0x7fa68c07b3f0 con 0x7fa69406ef50 2024-05-28T06:20:17.883 INFO:teuthology.orchestra.run.smithi096.stderr:pool 'cephfs_data' created 2024-05-28T06:20:17.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.890+0000 7fa69ba00640 1 -- 172.21.15.96:0/4240776080 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa67006ddf0 msgr2=0x7fa6700702b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:17.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.890+0000 7fa69ba00640 1 --2- 172.21.15.96:0/4240776080 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa67006ddf0 0x7fa6700702b0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fa684001920 tx=0x7fa684009040 comp rx=0 tx=0).stop 2024-05-28T06:20:17.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.890+0000 7fa69ba00640 1 -- 172.21.15.96:0/4240776080 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa69406ef50 msgr2=0x7fa6940b17c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:17.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.890+0000 7fa69ba00640 1 --2- 172.21.15.96:0/4240776080 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa69406ef50 0x7fa6940b17c0 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7fa68c00e200 tx=0x7fa68c009750 comp rx=0 tx=0).stop 2024-05-28T06:20:17.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.891+0000 7fa69ba00640 1 -- 172.21.15.96:0/4240776080 shutdown_connections 2024-05-28T06:20:17.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.891+0000 7fa69ba00640 1 --2- 172.21.15.96:0/4240776080 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa67006ddf0 0x7fa6700702b0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:17.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.891+0000 7fa69ba00640 1 --2- 172.21.15.96:0/4240776080 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa69406ef50 0x7fa6940b17c0 unknown :-1 s=CLOSED pgs=1607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:17.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.891+0000 7fa69ba00640 1 --2- 172.21.15.96:0/4240776080 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa694154e30 0x7fa69406ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:17.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.891+0000 7fa69ba00640 1 --2- 172.21.15.96:0/4240776080 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6780051c0 0x7fa6941f3a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:17.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.891+0000 7fa69ba00640 1 -- 172.21.15.96:0/4240776080 >> 172.21.15.96:0/4240776080 conn(0x7fa69406bd20 msgr2=0x7fa694151b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:17.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.891+0000 7fa69ba00640 1 -- 172.21.15.96:0/4240776080 shutdown_connections 2024-05-28T06:20:17.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:17.891+0000 7fa69ba00640 1 -- 172.21.15.96:0/4240776080 wait complete. 2024-05-28T06:20:17.927 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-05-28T06:20:18.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.107+0000 7fd0fc800640 1 Processor -- start 2024-05-28T06:20:18.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.107+0000 7fd0fc800640 1 -- start start 2024-05-28T06:20:18.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.108+0000 7fd0fc800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd0f40d0640 0x7fd0f40d0a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:18.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.108+0000 7fd0fc800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fd0f415b6f0 0x7fd0f406e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:18.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.108+0000 7fd0fc800640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0f40ade90 con 0x7fd0f40725e0 2024-05-28T06:20:18.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.108+0000 7fd0fc800640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fd0f4071c40 con 0x7fd0f415b6f0 2024-05-28T06:20:18.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.108+0000 7fd0fc800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fd0f40d0f60 con 0x7fd0f40d0640 2024-05-28T06:20:18.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.108+0000 7fd0fa800640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fd0f40725e0 0x7fd0f40b44d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:33774/0 (socket says 172.21.15.96:33774) 2024-05-28T06:20:18.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.108+0000 7fd0fa800640 1 -- 172.21.15.96:0/274043689 learned_addr learned my addr 172.21.15.96:0/274043689 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:20:18.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.109+0000 7fd0f9e00640 1 --2- 172.21.15.96:0/274043689 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd0f40d0640 0x7fd0f40d0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:18.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.109+0000 7fd0f8a00640 1 -- 172.21.15.96:0/274043689 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1401468104 0 0) 0x7fd0f40ade90 con 0x7fd0f40725e0 2024-05-28T06:20:18.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.109+0000 7fd0f8a00640 1 -- 172.21.15.96:0/274043689 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd0dc003520 con 0x7fd0f40725e0 2024-05-28T06:20:18.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.109+0000 7fd0f9400640 1 --2- 172.21.15.96:0/274043689 >> v2:172.21.15.123:3300/0 conn(0x7fd0f415b6f0 0x7fd0f406e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:18.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.109+0000 7fd0f8a00640 1 -- 172.21.15.96:0/274043689 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1503582198 0 0) 0x7fd0dc003520 con 0x7fd0f40725e0 2024-05-28T06:20:18.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.109+0000 7fd0f8a00640 1 -- 172.21.15.96:0/274043689 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd0f40d0640 msgr2=0x7fd0f40d0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:18.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.109+0000 7fd0f8a00640 1 --2- 172.21.15.96:0/274043689 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd0f40d0640 0x7fd0f40d0a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:18.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.109+0000 7fd0f8a00640 1 -- 172.21.15.96:0/274043689 >> v2:172.21.15.123:3300/0 conn(0x7fd0f415b6f0 msgr2=0x7fd0f406e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:18.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.109+0000 7fd0f8a00640 1 --2- 172.21.15.96:0/274043689 >> v2:172.21.15.123:3300/0 conn(0x7fd0f415b6f0 0x7fd0f406e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:18.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.109+0000 7fd0f8a00640 1 -- 172.21.15.96:0/274043689 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0f4071e70 con 0x7fd0f40725e0 2024-05-28T06:20:18.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.109+0000 7fd0f8a00640 1 -- 172.21.15.96:0/274043689 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7fd0f00019f0 con 0x7fd0f40725e0 2024-05-28T06:20:18.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.109+0000 7fd0f8a00640 1 -- 172.21.15.96:0/274043689 >> v1:172.21.15.96:6789/0 conn(0x7fd0f40725e0 legacy=0x7fd0f40b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:18.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.109+0000 7fd0f8a00640 1 --2- 172.21.15.96:0/274043689 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd0dc0039b0 0x7fd0dc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:18.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.109+0000 7fd0f8a00640 1 --2- 172.21.15.96:0/274043689 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd0f40d0640 0x7fd0dc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:18.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.109+0000 7fd0f8a00640 1 --2- 172.21.15.96:0/274043689 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd0dc004d40 0x7fd0dc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:18.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.109+0000 7fd0f8a00640 1 -- 172.21.15.96:0/274043689 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd0f40d0f60 con 0x7fd0dc0039b0 2024-05-28T06:20:18.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.109+0000 7fd0f8a00640 1 -- 172.21.15.96:0/274043689 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd0dc00b9c0 con 0x7fd0dc004d40 2024-05-28T06:20:18.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.109+0000 7fd0f8a00640 1 -- 172.21.15.96:0/274043689 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fd0dc00bb40 con 0x7fd0f40d0640 2024-05-28T06:20:18.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.110+0000 7fd0f9e00640 1 --2- 172.21.15.96:0/274043689 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd0dc0039b0 0x7fd0dc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:18.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.110+0000 7fd0f9400640 1 --2- 172.21.15.96:0/274043689 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd0f40d0640 0x7fd0dc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:18.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.110+0000 7fd0fa800640 1 --2- 172.21.15.96:0/274043689 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd0dc004d40 0x7fd0dc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:18.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.110+0000 7fd0f9e00640 1 -- 172.21.15.96:0/274043689 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd0f40d0640 msgr2=0x7fd0dc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:18.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.110+0000 7fd0f9e00640 1 --2- 172.21.15.96:0/274043689 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd0f40d0640 0x7fd0dc004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:18.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.110+0000 7fd0f9e00640 1 -- 172.21.15.96:0/274043689 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd0dc004d40 msgr2=0x7fd0dc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:18.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.110+0000 7fd0f9e00640 1 --2- 172.21.15.96:0/274043689 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd0dc004d40 0x7fd0dc0051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:18.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.110+0000 7fd0f9e00640 1 -- 172.21.15.96:0/274043689 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0dc003520 con 0x7fd0dc0039b0 2024-05-28T06:20:18.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.111+0000 7fd0f9e00640 1 --2- 172.21.15.96:0/274043689 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd0dc0039b0 0x7fd0dc003dd0 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7fd0e8009af0 tx=0x7fd0dc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e819079c662508f3 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:18.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.111+0000 7fd0f8a00640 1 -- 172.21.15.96:0/274043689 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0e8036070 con 0x7fd0dc0039b0 2024-05-28T06:20:18.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.111+0000 7fd0f8a00640 1 -- 172.21.15.96:0/274043689 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0e8031400 con 0x7fd0dc0039b0 2024-05-28T06:20:18.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.111+0000 7fd0f8a00640 1 -- 172.21.15.96:0/274043689 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0e8009690 con 0x7fd0dc0039b0 2024-05-28T06:20:18.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.112+0000 7fd0fc800640 1 -- 172.21.15.96:0/274043689 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd0dc0039b0 msgr2=0x7fd0dc003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:18.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.112+0000 7fd0fc800640 1 --2- 172.21.15.96:0/274043689 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd0dc0039b0 0x7fd0dc003dd0 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7fd0e8009af0 tx=0x7fd0dc005830 comp rx=0 tx=0).stop 2024-05-28T06:20:18.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.112+0000 7fd0fa800640 1 -- 172.21.15.96:0/274043689 reap_dead start 2024-05-28T06:20:18.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.112+0000 7fd0fc800640 1 -- 172.21.15.96:0/274043689 shutdown_connections 2024-05-28T06:20:18.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.112+0000 7fd0fc800640 1 -- 172.21.15.96:0/274043689 >> 172.21.15.96:0/274043689 conn(0x7fd0f406bd20 msgr2=0x7fd0f406c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:18.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.112+0000 7fd0fc800640 1 -- 172.21.15.96:0/274043689 shutdown_connections 2024-05-28T06:20:18.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.112+0000 7fd0fc800640 1 -- 172.21.15.96:0/274043689 wait complete. 2024-05-28T06:20:18.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.114+0000 7fd0fc800640 1 Processor -- start 2024-05-28T06:20:18.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.114+0000 7fd0fc800640 1 -- start start 2024-05-28T06:20:18.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.115+0000 7fd0fc800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd0dc0039b0 0x7fd0f415bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:18.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.115+0000 7fd0fc800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd0f415e3f0 0x7fd0f409bb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:18.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.115+0000 7fd0fc800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd0f409c0d0 0x7fd0f415d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:18.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.115+0000 7fd0fc800640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd0f406e890 con 0x7fd0f415e3f0 2024-05-28T06:20:18.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.115+0000 7fd0fc800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd0f40ae130 con 0x7fd0dc0039b0 2024-05-28T06:20:18.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.115+0000 7fd0fc800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fd0f40af660 con 0x7fd0f409c0d0 2024-05-28T06:20:18.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.115+0000 7fd0f9400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd0f415e3f0 0x7fd0f409bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:18.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.115+0000 7fd0f9400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd0f415e3f0 0x7fd0f409bb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51128/0 (socket says 172.21.15.96:51128) 2024-05-28T06:20:18.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.115+0000 7fd0f9400640 1 -- 172.21.15.96:0/1117696432 learned_addr learned my addr 172.21.15.96:0/1117696432 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:20:18.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.116+0000 7fd0fa800640 1 --2- 172.21.15.96:0/1117696432 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd0f409c0d0 0x7fd0f415d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:18.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.116+0000 7fd0f9e00640 1 --2- 172.21.15.96:0/1117696432 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd0dc0039b0 0x7fd0f415bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:18.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.116+0000 7fd0f9400640 1 -- 172.21.15.96:0/1117696432 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd0f409c0d0 msgr2=0x7fd0f415d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:18.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.116+0000 7fd0f9400640 1 --2- 172.21.15.96:0/1117696432 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd0f409c0d0 0x7fd0f415d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:18.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.116+0000 7fd0f9400640 1 -- 172.21.15.96:0/1117696432 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd0dc0039b0 msgr2=0x7fd0f415bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:18.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.116+0000 7fd0f9400640 1 --2- 172.21.15.96:0/1117696432 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd0dc0039b0 0x7fd0f415bad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:18.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.116+0000 7fd0f9400640 1 -- 172.21.15.96:0/1117696432 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0f415c240 con 0x7fd0f415e3f0 2024-05-28T06:20:18.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.116+0000 7fd0f9400640 1 --2- 172.21.15.96:0/1117696432 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd0f415e3f0 0x7fd0f409bb90 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7fd0e400f600 tx=0x7fd0e400fac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:18.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.116+0000 7fd0ee000640 1 -- 172.21.15.96:0/1117696432 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0e4019020 con 0x7fd0f415e3f0 2024-05-28T06:20:18.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.116+0000 7fd0ee000640 1 -- 172.21.15.96:0/1117696432 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0e4014870 con 0x7fd0f415e3f0 2024-05-28T06:20:18.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.117+0000 7fd0ee000640 1 -- 172.21.15.96:0/1117696432 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0e4014c20 con 0x7fd0f415e3f0 2024-05-28T06:20:18.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.117+0000 7fd0fc800640 1 -- 172.21.15.96:0/1117696432 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0f415c520 con 0x7fd0f415e3f0 2024-05-28T06:20:18.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.117+0000 7fd0fc800640 1 -- 172.21.15.96:0/1117696432 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0f415cae0 con 0x7fd0f415e3f0 2024-05-28T06:20:18.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.118+0000 7fd0fc800640 1 -- 172.21.15.96:0/1117696432 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0f40689e0 con 0x7fd0f415e3f0 2024-05-28T06:20:18.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.119+0000 7fd0ee000640 1 -- 172.21.15.96:0/1117696432 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 34) ==== 92927+0+0 (secure 0 0 0) 0x7fd0e401c020 con 0x7fd0f415e3f0 2024-05-28T06:20:18.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.120+0000 7fd0ee000640 1 --2- 172.21.15.96:0/1117696432 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd0d806ddc0 0x7fd0d8070280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:18.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.120+0000 7fd0ee000640 1 -- 172.21.15.96:0/1117696432 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 5665+0+0 (secure 0 0 0) 0x7fd0e408b720 con 0x7fd0f415e3f0 2024-05-28T06:20:18.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.121+0000 7fd0f9e00640 1 --2- 172.21.15.96:0/1117696432 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd0d806ddc0 0x7fd0d8070280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:18.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.122+0000 7fd0f9e00640 1 --2- 172.21.15.96:0/1117696432 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd0d806ddc0 0x7fd0d8070280 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fd0e8033000 tx=0x7fd0e8008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:18.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.124+0000 7fd0ee000640 1 -- 172.21.15.96:0/1117696432 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd0e4023020 con 0x7fd0f415e3f0 2024-05-28T06:20:18.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.361+0000 7fd0fc800640 1 -- 172.21.15.96:0/1117696432 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fd0f40b4e90 con 0x7fd0f415e3f0 2024-05-28T06:20:18.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.927+0000 7fd0ee000640 1 -- 172.21.15.96:0/1117696432 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 14 and data pool 15 v58) ==== 174+0+0 (secure 0 0 0) 0x7fd0e4012360 con 0x7fd0f415e3f0 2024-05-28T06:20:18.926 INFO:teuthology.orchestra.run.smithi096.stderr:new fs with metadata pool 14 and data pool 15 2024-05-28T06:20:18.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.935+0000 7fd0fc800640 1 -- 172.21.15.96:0/1117696432 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd0d806ddc0 msgr2=0x7fd0d8070280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:18.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.935+0000 7fd0fc800640 1 --2- 172.21.15.96:0/1117696432 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd0d806ddc0 0x7fd0d8070280 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fd0e8033000 tx=0x7fd0e8008030 comp rx=0 tx=0).stop 2024-05-28T06:20:18.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.935+0000 7fd0fc800640 1 -- 172.21.15.96:0/1117696432 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd0f415e3f0 msgr2=0x7fd0f409bb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:18.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.935+0000 7fd0fc800640 1 --2- 172.21.15.96:0/1117696432 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd0f415e3f0 0x7fd0f409bb90 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7fd0e400f600 tx=0x7fd0e400fac0 comp rx=0 tx=0).stop 2024-05-28T06:20:18.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.936+0000 7fd0fc800640 1 -- 172.21.15.96:0/1117696432 shutdown_connections 2024-05-28T06:20:18.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.936+0000 7fd0fc800640 1 --2- 172.21.15.96:0/1117696432 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd0d806ddc0 0x7fd0d8070280 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:18.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.936+0000 7fd0fc800640 1 --2- 172.21.15.96:0/1117696432 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd0f409c0d0 0x7fd0f415d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:18.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.936+0000 7fd0fc800640 1 --2- 172.21.15.96:0/1117696432 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd0f415e3f0 0x7fd0f409bb90 unknown :-1 s=CLOSED pgs=1610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:18.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.936+0000 7fd0fc800640 1 --2- 172.21.15.96:0/1117696432 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd0dc0039b0 0x7fd0f415bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:18.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.936+0000 7fd0fc800640 1 -- 172.21.15.96:0/1117696432 >> 172.21.15.96:0/1117696432 conn(0x7fd0f406bd20 msgr2=0x7fd0f40aff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:18.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.937+0000 7fd0fc800640 1 -- 172.21.15.96:0/1117696432 shutdown_connections 2024-05-28T06:20:18.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:18.937+0000 7fd0fc800640 1 -- 172.21.15.96:0/1117696432 wait complete. 2024-05-28T06:20:18.966 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:20:19.133 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.136+0000 7f1f9ae00640 1 Processor -- start 2024-05-28T06:20:19.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.137+0000 7f1f9ae00640 1 -- start start 2024-05-28T06:20:19.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.138+0000 7f1f9ae00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f1f940d04a0 0x7f1f940d0930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:19.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.138+0000 7f1f9ae00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f940ced40 con 0x7f1f940b4b60 2024-05-28T06:20:19.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.138+0000 7f1f9ae00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f940b3f90 con 0x7f1f940b3460 2024-05-28T06:20:19.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.138+0000 7f1f9ae00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f1f9415fbb0 con 0x7f1f940d04a0 2024-05-28T06:20:19.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.138+0000 7f1f8b400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f1f940d04a0 0x7f1f940d0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:19.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.138+0000 7f1f8b400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f1f940d04a0 0x7f1f940d0930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58662/0 (socket says 172.21.15.96:58662) 2024-05-28T06:20:19.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.138+0000 7f1f8b400640 1 -- 172.21.15.96:0/2423915910 learned_addr learned my addr 172.21.15.96:0/2423915910 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:20:19.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.139+0000 7f1f8b400640 1 -- 172.21.15.96:0/2423915910 >> v1:172.21.15.123:6789/0 conn(0x7f1f940b3460 legacy=0x7f1f940b3880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:19.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.139+0000 7f1f8b400640 1 -- 172.21.15.96:0/2423915910 >> v1:172.21.15.96:6789/0 conn(0x7f1f940b4b60 legacy=0x7f1f940cfaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:19.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.139+0000 7f1f8b400640 1 -- 172.21.15.96:0/2423915910 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f940cefc0 con 0x7f1f940d04a0 2024-05-28T06:20:19.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.139+0000 7f1f93400640 1 -- 172.21.15.96:0/2423915910 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 368415908 0 0) 0x7f1f84001110 con 0x7f1f940b4b60 2024-05-28T06:20:19.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.140+0000 7f1f8b400640 1 --2- 172.21.15.96:0/2423915910 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1f940d04a0 0x7f1f940d0930 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7f1f7c009870 tx=0x7f1f7c02a2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ed385c5a6995e82 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:19.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.140+0000 7f1f93400640 1 -- 172.21.15.96:0/2423915910 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f7c037070 con 0x7f1f940d04a0 2024-05-28T06:20:19.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.140+0000 7f1f93400640 1 -- 172.21.15.96:0/2423915910 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f7c032440 con 0x7f1f940d04a0 2024-05-28T06:20:19.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.140+0000 7f1f93400640 1 -- 172.21.15.96:0/2423915910 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f7c032810 con 0x7f1f940d04a0 2024-05-28T06:20:19.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.141+0000 7f1f9ae00640 1 -- 172.21.15.96:0/2423915910 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1f940d04a0 msgr2=0x7f1f940d0930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:19.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.141+0000 7f1f9ae00640 1 --2- 172.21.15.96:0/2423915910 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1f940d04a0 0x7f1f940d0930 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7f1f7c009870 tx=0x7f1f7c02a2a0 comp rx=0 tx=0).stop 2024-05-28T06:20:19.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.141+0000 7f1f9ae00640 1 -- 172.21.15.96:0/2423915910 shutdown_connections 2024-05-28T06:20:19.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.141+0000 7f1f9ae00640 1 --2- 172.21.15.96:0/2423915910 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1f940d04a0 0x7f1f940d0930 unknown :-1 s=CLOSED pgs=1612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:19.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.141+0000 7f1f9ae00640 1 -- 172.21.15.96:0/2423915910 >> 172.21.15.96:0/2423915910 conn(0x7f1f9406bae0 msgr2=0x7f1f9406bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:19.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.141+0000 7f1f9ae00640 1 -- 172.21.15.96:0/2423915910 shutdown_connections 2024-05-28T06:20:19.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.141+0000 7f1f9ae00640 1 -- 172.21.15.96:0/2423915910 wait complete. 2024-05-28T06:20:19.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.142+0000 7f1f9ae00640 1 Processor -- start 2024-05-28T06:20:19.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.143+0000 7f1f9ae00640 1 -- start start 2024-05-28T06:20:19.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.143+0000 7f1f9ae00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1f940b3460 0x7f1f940ad350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:19.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.143+0000 7f1f9ae00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1f940b4b60 0x7f1f940ad890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:19.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.143+0000 7f1f9ae00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1f940d04a0 0x7f1f9415b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:19.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.143+0000 7f1f9ae00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f1f9415ede0 con 0x7f1f940b4b60 2024-05-28T06:20:19.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.143+0000 7f1f9ae00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f1f940cf580 con 0x7f1f940d04a0 2024-05-28T06:20:19.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.143+0000 7f1f9ae00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f1f9415ef60 con 0x7f1f940b3460 2024-05-28T06:20:19.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.144+0000 7f1f8b400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1f940b4b60 0x7f1f940ad890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:19.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.144+0000 7f1f8b400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1f940b4b60 0x7f1f940ad890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58674/0 (socket says 172.21.15.96:58674) 2024-05-28T06:20:19.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.144+0000 7f1f8b400640 1 -- 172.21.15.96:0/1798643450 learned_addr learned my addr 172.21.15.96:0/1798643450 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:20:19.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.144+0000 7f1f8b400640 1 -- 172.21.15.96:0/1798643450 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1f940b3460 msgr2=0x7f1f940ad350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:19.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.144+0000 7f1f8b400640 1 --2- 172.21.15.96:0/1798643450 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1f940b3460 0x7f1f940ad350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:19.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.144+0000 7f1f8b400640 1 -- 172.21.15.96:0/1798643450 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1f940d04a0 msgr2=0x7f1f9415b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:19.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.144+0000 7f1f8b400640 1 --2- 172.21.15.96:0/1798643450 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1f940d04a0 0x7f1f9415b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:19.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.144+0000 7f1f8b400640 1 -- 172.21.15.96:0/1798643450 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f940ae140 con 0x7f1f940b4b60 2024-05-28T06:20:19.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.144+0000 7f1f8b400640 1 --2- 172.21.15.96:0/1798643450 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1f940b4b60 0x7f1f940ad890 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7f1f7c000c00 tx=0x7f1f7c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:19.143 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.145+0000 7f1f90c00640 1 -- 172.21.15.96:0/1798643450 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f7c041070 con 0x7f1f940b4b60 2024-05-28T06:20:19.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.145+0000 7f1f90c00640 1 -- 172.21.15.96:0/1798643450 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f7c03c890 con 0x7f1f940b4b60 2024-05-28T06:20:19.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.145+0000 7f1f90c00640 1 -- 172.21.15.96:0/1798643450 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f7c037040 con 0x7f1f940b4b60 2024-05-28T06:20:19.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.145+0000 7f1f9ae00640 1 -- 172.21.15.96:0/1798643450 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f940ae420 con 0x7f1f940b4b60 2024-05-28T06:20:19.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.145+0000 7f1f9ae00640 1 -- 172.21.15.96:0/1798643450 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f940ae9e0 con 0x7f1f940b4b60 2024-05-28T06:20:19.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.146+0000 7f1f9ae00640 1 -- 172.21.15.96:0/1798643450 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f940689e0 con 0x7f1f940b4b60 2024-05-28T06:20:19.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.147+0000 7f1f90c00640 1 -- 172.21.15.96:0/1798643450 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 34) ==== 92927+0+0 (secure 0 0 0) 0x7f1f7c044070 con 0x7f1f940b4b60 2024-05-28T06:20:19.145 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.148+0000 7f1f90c00640 1 --2- 172.21.15.96:0/1798643450 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1f7006ddc0 0x7f1f70070280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:19.145 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.148+0000 7f1f90c00640 1 -- 172.21.15.96:0/1798643450 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 5765+0+0 (secure 0 0 0) 0x7f1f7c0ab4c0 con 0x7f1f940b4b60 2024-05-28T06:20:19.145 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.148+0000 7f1f93e00640 1 --2- 172.21.15.96:0/1798643450 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1f7006ddc0 0x7f1f70070280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:19.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.150+0000 7f1f93e00640 1 --2- 172.21.15.96:0/1798643450 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1f7006ddc0 0x7f1f70070280 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f1f84001670 tx=0x7f1f84007b60 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:19.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.152+0000 7f1f90c00640 1 -- 172.21.15.96:0/1798643450 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1f940b3f90 con 0x7f1f940b4b60 2024-05-28T06:20:19.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.351+0000 7f1f9ae00640 1 -- 172.21.15.96:0/1798643450 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1f9406c380 con 0x7f1f940b4b60 2024-05-28T06:20:19.351 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.355+0000 7f1f90c00640 1 -- 172.21.15.96:0/1798643450 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v102) ==== 74+0+16732 (secure 0 0 0) 0x7f1f7c07b340 con 0x7f1f940b4b60 2024-05-28T06:20:19.352 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:20:19.352 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":102,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:20:18.363077+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-05-28T06:20:16.097747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-05-28T06:20:17.338574+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":13,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:20:19.355 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.357+0000 7f1f9ae00640 1 -- 172.21.15.96:0/1798643450 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1f7006ddc0 msgr2=0x7f1f70070280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:19.355 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.357+0000 7f1f9ae00640 1 --2- 172.21.15.96:0/1798643450 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1f7006ddc0 0x7f1f70070280 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f1f84001670 tx=0x7f1f84007b60 comp rx=0 tx=0).stop 2024-05-28T06:20:19.355 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.357+0000 7f1f9ae00640 1 -- 172.21.15.96:0/1798643450 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1f940b4b60 msgr2=0x7f1f940ad890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:19.355 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.357+0000 7f1f9ae00640 1 --2- 172.21.15.96:0/1798643450 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1f940b4b60 0x7f1f940ad890 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7f1f7c000c00 tx=0x7f1f7c0046c0 comp rx=0 tx=0).stop 2024-05-28T06:20:19.355 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.358+0000 7f1f9ae00640 1 -- 172.21.15.96:0/1798643450 shutdown_connections 2024-05-28T06:20:19.355 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.358+0000 7f1f9ae00640 1 --2- 172.21.15.96:0/1798643450 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1f7006ddc0 0x7f1f70070280 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:19.355 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.358+0000 7f1f9ae00640 1 --2- 172.21.15.96:0/1798643450 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1f940d04a0 0x7f1f9415b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:19.355 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.358+0000 7f1f9ae00640 1 --2- 172.21.15.96:0/1798643450 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1f940b4b60 0x7f1f940ad890 unknown :-1 s=CLOSED pgs=1613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:19.355 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.358+0000 7f1f9ae00640 1 --2- 172.21.15.96:0/1798643450 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1f940b3460 0x7f1f940ad350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:19.355 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.358+0000 7f1f9ae00640 1 -- 172.21.15.96:0/1798643450 >> 172.21.15.96:0/1798643450 conn(0x7f1f9406bae0 msgr2=0x7f1f9415f2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:19.355 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.358+0000 7f1f9ae00640 1 -- 172.21.15.96:0/1798643450 shutdown_connections 2024-05-28T06:20:19.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.358+0000 7f1f9ae00640 1 -- 172.21.15.96:0/1798643450 wait complete. 2024-05-28T06:20:19.374 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:20:19.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.576+0000 7f8273e00640 1 Processor -- start 2024-05-28T06:20:19.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.576+0000 7f8273e00640 1 -- start start 2024-05-28T06:20:19.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.578+0000 7f8273e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8274156b30 0x7f82740af6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:19.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.578+0000 7f8273e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f827406ee40 0x7f82740ad230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:19.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.578+0000 7f8273e00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8274156f10 con 0x7f827406f2c0 2024-05-28T06:20:19.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.578+0000 7f8273e00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f827406dda0 con 0x7f827406ee40 2024-05-28T06:20:19.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.578+0000 7f8273e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f827406e080 con 0x7f8274156b30 2024-05-28T06:20:19.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.578+0000 7f8273400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f827406f2c0 0x7f82740b32c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:51056/0 (socket says 172.21.15.96:51056) 2024-05-28T06:20:19.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.578+0000 7f8273400640 1 -- 172.21.15.96:0/1772520999 learned_addr learned my addr 172.21.15.96:0/1772520999 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:20:19.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.578+0000 7f826be00640 1 --2- 172.21.15.96:0/1772520999 >> v2:172.21.15.123:3300/0 conn(0x7f827406ee40 0x7f82740ad230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:19.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.578+0000 7f8272a00640 1 --2- 172.21.15.96:0/1772520999 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8274156b30 0x7f82740af6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:19.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.578+0000 7f8272000640 1 -- 172.21.15.96:0/1772520999 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1754257320 0 0) 0x7f8274156f10 con 0x7f827406f2c0 2024-05-28T06:20:19.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.579+0000 7f8272000640 1 -- 172.21.15.96:0/1772520999 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8258003520 con 0x7f827406f2c0 2024-05-28T06:20:19.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.579+0000 7f8272000640 1 -- 172.21.15.96:0/1772520999 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3481006648 0 0) 0x7f8258003520 con 0x7f827406f2c0 2024-05-28T06:20:19.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.579+0000 7f8272000640 1 -- 172.21.15.96:0/1772520999 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8274156b30 msgr2=0x7f82740af6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:19.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.579+0000 7f8272000640 1 --2- 172.21.15.96:0/1772520999 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8274156b30 0x7f82740af6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:19.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.579+0000 7f8272000640 1 -- 172.21.15.96:0/1772520999 >> v2:172.21.15.123:3300/0 conn(0x7f827406ee40 msgr2=0x7f82740ad230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:19.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.579+0000 7f8272000640 1 --2- 172.21.15.96:0/1772520999 >> v2:172.21.15.123:3300/0 conn(0x7f827406ee40 0x7f82740ad230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:19.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.579+0000 7f8272000640 1 -- 172.21.15.96:0/1772520999 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f827406e360 con 0x7f827406f2c0 2024-05-28T06:20:19.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.579+0000 7f8272000640 1 -- 172.21.15.96:0/1772520999 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f826c0019f0 con 0x7f827406f2c0 2024-05-28T06:20:19.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.579+0000 7f826be00640 1 --2- 172.21.15.96:0/1772520999 >> v2:172.21.15.123:3300/0 conn(0x7f827406ee40 0x7f82740ad230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-28T06:20:19.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.579+0000 7f8272000640 1 -- 172.21.15.96:0/1772520999 >> v1:172.21.15.96:6789/0 conn(0x7f827406f2c0 legacy=0x7f82740b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:19.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.579+0000 7f8272000640 1 --2- 172.21.15.96:0/1772520999 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f82580039b0 0x7f8258003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:19.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.579+0000 7f8272000640 1 --2- 172.21.15.96:0/1772520999 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8258004800 0x7f8258004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:19.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.579+0000 7f8272000640 1 --2- 172.21.15.96:0/1772520999 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f82580051c0 0x7f825800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:19.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.579+0000 7f8272000640 1 -- 172.21.15.96:0/1772520999 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f827406dda0 con 0x7f82580051c0 2024-05-28T06:20:19.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.579+0000 7f8272000640 1 -- 172.21.15.96:0/1772520999 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f827406e080 con 0x7f8258004800 2024-05-28T06:20:19.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.579+0000 7f8272000640 1 -- 172.21.15.96:0/1772520999 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f825800c0e0 con 0x7f82580039b0 2024-05-28T06:20:19.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.579+0000 7f8272a00640 1 --2- 172.21.15.96:0/1772520999 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8274156b30 0x7f82740af6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-28T06:20:19.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.579+0000 7f8273400640 1 --2- 172.21.15.96:0/1772520999 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f82580051c0 0x7f825800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:19.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.580+0000 7f826be00640 1 --2- 172.21.15.96:0/1772520999 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8258004800 0x7f8258004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:19.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.580+0000 7f8273400640 1 -- 172.21.15.96:0/1772520999 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f82580039b0 msgr2=0x7f8258003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:19.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.580+0000 7f8273400640 1 --2- 172.21.15.96:0/1772520999 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f82580039b0 0x7f8258003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:19.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.580+0000 7f8273400640 1 -- 172.21.15.96:0/1772520999 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8258004800 msgr2=0x7f8258004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:19.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.580+0000 7f8273400640 1 --2- 172.21.15.96:0/1772520999 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8258004800 0x7f8258004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:19.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.580+0000 7f8273400640 1 -- 172.21.15.96:0/1772520999 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8258003520 con 0x7f82580051c0 2024-05-28T06:20:19.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.580+0000 7f826be00640 1 --2- 172.21.15.96:0/1772520999 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8258004800 0x7f8258004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:20:19.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.581+0000 7f8273400640 1 --2- 172.21.15.96:0/1772520999 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f82580051c0 0x7f825800b9c0 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7f826c00a7d0 tx=0x7f8258005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f10ca104240f608c server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:19.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.581+0000 7f8272000640 1 -- 172.21.15.96:0/1772520999 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f826c00a3f0 con 0x7f82580051c0 2024-05-28T06:20:19.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.581+0000 7f8272000640 1 -- 172.21.15.96:0/1772520999 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f826c02f070 con 0x7f82580051c0 2024-05-28T06:20:19.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.581+0000 7f8272000640 1 -- 172.21.15.96:0/1772520999 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f826c02bc90 con 0x7f82580051c0 2024-05-28T06:20:19.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.581+0000 7f8273e00640 1 -- 172.21.15.96:0/1772520999 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f82580051c0 msgr2=0x7f825800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:19.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.581+0000 7f8273e00640 1 --2- 172.21.15.96:0/1772520999 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f82580051c0 0x7f825800b9c0 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7f826c00a7d0 tx=0x7f8258005830 comp rx=0 tx=0).stop 2024-05-28T06:20:19.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.581+0000 7f8273400640 1 -- 172.21.15.96:0/1772520999 reap_dead start 2024-05-28T06:20:19.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.582+0000 7f8273e00640 1 -- 172.21.15.96:0/1772520999 shutdown_connections 2024-05-28T06:20:19.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.582+0000 7f8273e00640 1 -- 172.21.15.96:0/1772520999 >> 172.21.15.96:0/1772520999 conn(0x7f82740b1fc0 msgr2=0x7f82740b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:19.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.582+0000 7f8273e00640 1 -- 172.21.15.96:0/1772520999 shutdown_connections 2024-05-28T06:20:19.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.582+0000 7f8273e00640 1 -- 172.21.15.96:0/1772520999 wait complete. 2024-05-28T06:20:19.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.583+0000 7f8273e00640 1 Processor -- start 2024-05-28T06:20:19.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.583+0000 7f8273e00640 1 -- start start 2024-05-28T06:20:19.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.584+0000 7f8273e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f82580051c0 0x7f827415f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:19.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.584+0000 7f8273e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f827415d950 0x7f8274150160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:19.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.584+0000 7f8273e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f82740726b0 0x7f8274072b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:19.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.584+0000 7f8273e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f82740b2db0 con 0x7f82740726b0 2024-05-28T06:20:19.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.584+0000 7f8273e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f82740b2c30 con 0x7f82580051c0 2024-05-28T06:20:19.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.584+0000 7f8273e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f82740b39d0 con 0x7f827415d950 2024-05-28T06:20:19.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.585+0000 7f8273400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f82740726b0 0x7f8274072b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:19.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.585+0000 7f8273400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f82740726b0 0x7f8274072b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58698/0 (socket says 172.21.15.96:58698) 2024-05-28T06:20:19.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.585+0000 7f8273400640 1 -- 172.21.15.96:0/1775440459 learned_addr learned my addr 172.21.15.96:0/1775440459 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:20:19.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.585+0000 7f8272a00640 1 --2- 172.21.15.96:0/1775440459 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f82580051c0 0x7f827415f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:19.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.585+0000 7f826be00640 1 --2- 172.21.15.96:0/1775440459 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f827415d950 0x7f8274150160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:19.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.585+0000 7f8273400640 1 -- 172.21.15.96:0/1775440459 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f827415d950 msgr2=0x7f8274150160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:19.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.585+0000 7f8273400640 1 --2- 172.21.15.96:0/1775440459 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f827415d950 0x7f8274150160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:19.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.585+0000 7f8273400640 1 -- 172.21.15.96:0/1775440459 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f82580051c0 msgr2=0x7f827415f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:19.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.585+0000 7f8273400640 1 --2- 172.21.15.96:0/1775440459 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f82580051c0 0x7f827415f260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:19.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.585+0000 7f8273400640 1 -- 172.21.15.96:0/1775440459 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f827415e1c0 con 0x7f82740726b0 2024-05-28T06:20:19.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.585+0000 7f826be00640 1 --2- 172.21.15.96:0/1775440459 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f827415d950 0x7f8274150160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:20:19.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.585+0000 7f8273400640 1 --2- 172.21.15.96:0/1775440459 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f82740726b0 0x7f8274072b30 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f826c00a8f0 tx=0x7f826c001cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:19.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.586+0000 7f826aa00640 1 -- 172.21.15.96:0/1775440459 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f826c00a3f0 con 0x7f82740726b0 2024-05-28T06:20:19.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.586+0000 7f826aa00640 1 -- 172.21.15.96:0/1775440459 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f826c044070 con 0x7f82740726b0 2024-05-28T06:20:19.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.586+0000 7f8273e00640 1 -- 172.21.15.96:0/1775440459 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f827415e4a0 con 0x7f82740726b0 2024-05-28T06:20:19.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.586+0000 7f826aa00640 1 -- 172.21.15.96:0/1775440459 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f826c001f40 con 0x7f82740726b0 2024-05-28T06:20:19.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.586+0000 7f8273e00640 1 -- 172.21.15.96:0/1775440459 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f827415ea60 con 0x7f82740726b0 2024-05-28T06:20:19.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.587+0000 7f8273e00640 1 -- 172.21.15.96:0/1775440459 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82740689e0 con 0x7f82740726b0 2024-05-28T06:20:19.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.588+0000 7f826aa00640 1 -- 172.21.15.96:0/1775440459 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 34) ==== 92927+0+0 (secure 0 0 0) 0x7f826c034690 con 0x7f82740726b0 2024-05-28T06:20:19.586 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.589+0000 7f826aa00640 1 --2- 172.21.15.96:0/1775440459 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f825006ddf0 0x7f82500702b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:19.586 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.590+0000 7f826aa00640 1 -- 172.21.15.96:0/1775440459 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 5765+0+0 (secure 0 0 0) 0x7f826c0aae40 con 0x7f82740726b0 2024-05-28T06:20:19.586 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.590+0000 7f8272a00640 1 --2- 172.21.15.96:0/1775440459 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f825006ddf0 0x7f82500702b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:19.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.591+0000 7f8272a00640 1 --2- 172.21.15.96:0/1775440459 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f825006ddf0 0x7f82500702b0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f8264006fd0 tx=0x7f8264008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:19.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.594+0000 7f826aa00640 1 -- 172.21.15.96:0/1775440459 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f826c07ac90 con 0x7f82740726b0 2024-05-28T06:20:19.793 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.796+0000 7f8273e00640 1 -- 172.21.15.96:0/1775440459 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8274073bb0 con 0x7f82740726b0 2024-05-28T06:20:19.794 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.797+0000 7f826aa00640 1 -- 172.21.15.96:0/1775440459 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v102) ==== 74+0+16732 (secure 0 0 0) 0x7f826c07a590 con 0x7f82740726b0 2024-05-28T06:20:19.795 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:20:19.795 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":102,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:20:18.363077+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-05-28T06:20:16.097747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-05-28T06:20:17.338574+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":13,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:20:19.797 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.800+0000 7f8273e00640 1 -- 172.21.15.96:0/1775440459 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f825006ddf0 msgr2=0x7f82500702b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:19.797 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.800+0000 7f8273e00640 1 --2- 172.21.15.96:0/1775440459 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f825006ddf0 0x7f82500702b0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f8264006fd0 tx=0x7f8264008040 comp rx=0 tx=0).stop 2024-05-28T06:20:19.797 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.800+0000 7f8273e00640 1 -- 172.21.15.96:0/1775440459 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f82740726b0 msgr2=0x7f8274072b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:19.797 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.800+0000 7f8273e00640 1 --2- 172.21.15.96:0/1775440459 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f82740726b0 0x7f8274072b30 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f826c00a8f0 tx=0x7f826c001cf0 comp rx=0 tx=0).stop 2024-05-28T06:20:19.797 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.800+0000 7f8273e00640 1 -- 172.21.15.96:0/1775440459 shutdown_connections 2024-05-28T06:20:19.797 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.800+0000 7f8273e00640 1 --2- 172.21.15.96:0/1775440459 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f825006ddf0 0x7f82500702b0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:19.798 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.800+0000 7f8273e00640 1 --2- 172.21.15.96:0/1775440459 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f82740726b0 0x7f8274072b30 unknown :-1 s=CLOSED pgs=1616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:19.798 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.800+0000 7f8273e00640 1 --2- 172.21.15.96:0/1775440459 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f827415d950 0x7f8274150160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:19.798 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.800+0000 7f8273e00640 1 --2- 172.21.15.96:0/1775440459 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f82580051c0 0x7f827415f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:19.798 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.800+0000 7f8273e00640 1 -- 172.21.15.96:0/1775440459 >> 172.21.15.96:0/1775440459 conn(0x7f82740b1fc0 msgr2=0x7f82740b4980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:19.798 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.800+0000 7f8273e00640 1 -- 172.21.15.96:0/1775440459 shutdown_connections 2024-05-28T06:20:19.798 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:19.800+0000 7f8273e00640 1 -- 172.21.15.96:0/1775440459 wait complete. 2024-05-28T06:20:19.816 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-05-28T06:20:20.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:20.022+0000 7f835c200640 1 Processor -- start 2024-05-28T06:20:20.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:20.023+0000 7f835c200640 1 -- start start 2024-05-28T06:20:20.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:20.024+0000 7f835c200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f835415d5d0 0x7f83540cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:20.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:20.024+0000 7f835c200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f83540b32c0 0x7f83540b3700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:20.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:20.024+0000 7f835c200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f835415cb10 con 0x7f83540d0640 2024-05-28T06:20:20.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:20.024+0000 7f835c200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f83540adbb0 con 0x7f83540b32c0 2024-05-28T06:20:20.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:20.024+0000 7f835c200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f83540ad5f0 con 0x7f835415d5d0 2024-05-28T06:20:20.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:20.025+0000 7f835a200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f83540b32c0 0x7f83540b3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:20.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:20.025+0000 7f835a200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f83540b32c0 0x7f83540b3700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58708/0 (socket says 172.21.15.96:58708) 2024-05-28T06:20:20.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:20.025+0000 7f835a200640 1 -- 172.21.15.96:0/2931477288 learned_addr learned my addr 172.21.15.96:0/2931477288 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:20:20.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:20.025+0000 7f8358e00640 1 --2- 172.21.15.96:0/2931477288 >> v2:172.21.15.123:3300/0 conn(0x7f835415d5d0 0x7f83540cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:20.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:20.025+0000 7f835a200640 1 -- 172.21.15.96:0/2931477288 >> v2:172.21.15.123:3300/0 conn(0x7f835415d5d0 msgr2=0x7f83540cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:20.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:20.025+0000 7f835a200640 1 --2- 172.21.15.96:0/2931477288 >> v2:172.21.15.123:3300/0 conn(0x7f835415d5d0 0x7f83540cf950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:20.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:20.025+0000 7f835a200640 1 -- 172.21.15.96:0/2931477288 >> v1:172.21.15.96:6789/0 conn(0x7f83540d0640 legacy=0x7f83540d0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:20.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:20.025+0000 7f835a200640 1 -- 172.21.15.96:0/2931477288 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83540ad8d0 con 0x7f83540b32c0 2024-05-28T06:20:20.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:20.025+0000 7f8353e00640 1 -- 172.21.15.96:0/2931477288 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 454582352 0 0) 0x7f835415cb10 con 0x7f83540d0640 2024-05-28T06:20:20.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:20.026+0000 7f835a200640 1 --2- 172.21.15.96:0/2931477288 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83540b32c0 0x7f83540b3700 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7f8354074120 tx=0x7f834c02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49454b7823a78a71 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:20.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:20.027+0000 7f8353e00640 1 -- 172.21.15.96:0/2931477288 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f834c037070 con 0x7f83540b32c0 2024-05-28T06:20:20.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:20.027+0000 7f8353e00640 1 -- 172.21.15.96:0/2931477288 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f834c032440 con 0x7f83540b32c0 2024-05-28T06:20:20.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:20.027+0000 7f8353e00640 1 -- 172.21.15.96:0/2931477288 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f834c032810 con 0x7f83540b32c0 2024-05-28T06:20:20.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:20.027+0000 7f835c200640 1 -- 172.21.15.96:0/2931477288 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83540b32c0 msgr2=0x7f83540b3700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:20.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:20.027+0000 7f835c200640 1 --2- 172.21.15.96:0/2931477288 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83540b32c0 0x7f83540b3700 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7f8354074120 tx=0x7f834c02a1e0 comp rx=0 tx=0).stop 2024-05-28T06:20:20.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:20.027+0000 7f835c200640 1 -- 172.21.15.96:0/2931477288 shutdown_connections 2024-05-28T06:20:20.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:20.027+0000 7f835c200640 1 --2- 172.21.15.96:0/2931477288 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83540b32c0 0x7f83540b3700 unknown :-1 s=CLOSED pgs=1618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:20.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:20.027+0000 7f835c200640 1 --2- 172.21.15.96:0/2931477288 >> v2:172.21.15.123:3300/0 conn(0x7f835415d5d0 0x7f83540cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:20.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:20.027+0000 7f835c200640 1 -- 172.21.15.96:0/2931477288 >> 172.21.15.96:0/2931477288 conn(0x7f835406bd20 msgr2=0x7f835406c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:20.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:20.028+0000 7f835c200640 1 -- 172.21.15.96:0/2931477288 shutdown_connections 2024-05-28T06:20:20.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:20.028+0000 7f835c200640 1 -- 172.21.15.96:0/2931477288 wait complete. 2024-05-28T06:20:20.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:20.029+0000 7f835c200640 1 Processor -- start 2024-05-28T06:20:20.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:20.029+0000 7f835c200640 1 -- start start 2024-05-28T06:20:20.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:20.030+0000 7f835c200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f83540b32c0 0x7f835409c320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:20.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:20.030+0000 7f835c200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83540d0640 0x7f835409c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:20.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:20.030+0000 7f835c200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f835415d5d0 0x7f835409a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:20.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:20.030+0000 7f835c200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f835415da50 con 0x7f83540d0640 2024-05-28T06:20:20.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:20.030+0000 7f835c200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f835415d0b0 con 0x7f83540b32c0 2024-05-28T06:20:20.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:20.030+0000 7f835c200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f83540ad2a0 con 0x7f835415d5d0 2024-05-28T06:20:20.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:20.031+0000 7f8358e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83540d0640 0x7f835409c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:20.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:20.031+0000 7f8358e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83540d0640 0x7f835409c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58720/0 (socket says 172.21.15.96:58720) 2024-05-28T06:20:20.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:20.031+0000 7f8358e00640 1 -- 172.21.15.96:0/2995747576 learned_addr learned my addr 172.21.15.96:0/2995747576 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:20:20.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:20.031+0000 7f8358e00640 1 -- 172.21.15.96:0/2995747576 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f835415d5d0 msgr2=0x7f835409a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:20.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:20.031+0000 7f8358e00640 1 --2- 172.21.15.96:0/2995747576 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f835415d5d0 0x7f835409a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:20.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:20.031+0000 7f8359800640 1 --2- 172.21.15.96:0/2995747576 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f83540b32c0 0x7f835409c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:20.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:20.031+0000 7f8358e00640 1 -- 172.21.15.96:0/2995747576 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f83540b32c0 msgr2=0x7f835409c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:20.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:20.031+0000 7f8358e00640 1 --2- 172.21.15.96:0/2995747576 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f83540b32c0 0x7f835409c320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:20.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:20.031+0000 7f8358e00640 1 -- 172.21.15.96:0/2995747576 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f835409bc60 con 0x7f83540d0640 2024-05-28T06:20:20.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:20.031+0000 7f8358e00640 1 --2- 172.21.15.96:0/2995747576 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83540d0640 0x7f835409c860 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7f834000d800 tx=0x7f834000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:20.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:20.032+0000 7f8351600640 1 -- 172.21.15.96:0/2995747576 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f834000b6c0 con 0x7f83540d0640 2024-05-28T06:20:20.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:20.032+0000 7f8351600640 1 -- 172.21.15.96:0/2995747576 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8340019070 con 0x7f83540d0640 2024-05-28T06:20:20.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:20.032+0000 7f8351600640 1 -- 172.21.15.96:0/2995747576 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f834000ba80 con 0x7f83540d0640 2024-05-28T06:20:20.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:20.032+0000 7f835c200640 1 -- 172.21.15.96:0/2995747576 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f835409b0e0 con 0x7f83540d0640 2024-05-28T06:20:20.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:20.032+0000 7f835c200640 1 -- 172.21.15.96:0/2995747576 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f835409b6a0 con 0x7f83540d0640 2024-05-28T06:20:20.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:20.033+0000 7f835c200640 1 -- 172.21.15.96:0/2995747576 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83540689e0 con 0x7f83540d0640 2024-05-28T06:20:20.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:20.034+0000 7f8351600640 1 -- 172.21.15.96:0/2995747576 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 34) ==== 92927+0+0 (secure 0 0 0) 0x7f834000bc20 con 0x7f83540d0640 2024-05-28T06:20:20.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:20.035+0000 7f8351600640 1 --2- 172.21.15.96:0/2995747576 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f833406ddc0 0x7f8334070280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:20.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:20.035+0000 7f8351600640 1 -- 172.21.15.96:0/2995747576 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 5765+0+0 (secure 0 0 0) 0x7f834008b140 con 0x7f83540d0640 2024-05-28T06:20:20.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:20.036+0000 7f8359800640 1 --2- 172.21.15.96:0/2995747576 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f833406ddc0 0x7f8334070280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:20.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:20.037+0000 7f8359800640 1 --2- 172.21.15.96:0/2995747576 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f833406ddc0 0x7f8334070280 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f83480012e0 tx=0x7f8348009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:20.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:20.039+0000 7f8351600640 1 -- 172.21.15.96:0/2995747576 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f834005abe0 con 0x7f83540d0640 2024-05-28T06:20:20.276 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:20.279+0000 7f835c200640 1 -- 172.21.15.96:0/2995747576 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f8354071ef0 con 0x7f83540d0640 2024-05-28T06:20:20.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:20.944+0000 7f8351600640 1 -- 172.21.15.96:0/2995747576 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v61) ==== 118+0+0 (secure 0 0 0) 0x7f834005a4e0 con 0x7f83540d0640 2024-05-28T06:20:20.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:20.947+0000 7f835c200640 1 -- 172.21.15.96:0/2995747576 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f833406ddc0 msgr2=0x7f8334070280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:20.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:20.947+0000 7f835c200640 1 --2- 172.21.15.96:0/2995747576 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f833406ddc0 0x7f8334070280 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f83480012e0 tx=0x7f8348009040 comp rx=0 tx=0).stop 2024-05-28T06:20:20.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:20.947+0000 7f835c200640 1 -- 172.21.15.96:0/2995747576 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83540d0640 msgr2=0x7f835409c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:20.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:20.947+0000 7f835c200640 1 --2- 172.21.15.96:0/2995747576 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83540d0640 0x7f835409c860 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7f834000d800 tx=0x7f834000dcc0 comp rx=0 tx=0).stop 2024-05-28T06:20:20.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:20.947+0000 7f835c200640 1 -- 172.21.15.96:0/2995747576 shutdown_connections 2024-05-28T06:20:20.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:20.947+0000 7f835c200640 1 --2- 172.21.15.96:0/2995747576 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f833406ddc0 0x7f8334070280 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:20.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:20.947+0000 7f835c200640 1 --2- 172.21.15.96:0/2995747576 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f835415d5d0 0x7f835409a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:20.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:20.947+0000 7f835c200640 1 --2- 172.21.15.96:0/2995747576 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83540d0640 0x7f835409c860 unknown :-1 s=CLOSED pgs=1619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:20.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:20.947+0000 7f835c200640 1 --2- 172.21.15.96:0/2995747576 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f83540b32c0 0x7f835409c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:20.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:20.948+0000 7f835c200640 1 -- 172.21.15.96:0/2995747576 >> 172.21.15.96:0/2995747576 conn(0x7f835406bd20 msgr2=0x7f83540b3df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:20.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:20.948+0000 7f835c200640 1 -- 172.21.15.96:0/2995747576 shutdown_connections 2024-05-28T06:20:20.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:20.948+0000 7f835c200640 1 -- 172.21.15.96:0/2995747576 wait complete. 2024-05-28T06:20:20.966 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-05-28T06:20:20.966 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-05-28T06:20:21.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:21.172+0000 7fe3eea00640 1 Processor -- start 2024-05-28T06:20:21.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:21.172+0000 7fe3eea00640 1 -- start start 2024-05-28T06:20:21.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:21.173+0000 7fe3eea00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fe3e80ce1f0 0x7fe3e80b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:21.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:21.173+0000 7fe3eea00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe3e80ce5d0 0x7fe3e815ec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:21.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:21.174+0000 7fe3eea00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3e80b3580 con 0x7fe3e80b3d60 2024-05-28T06:20:21.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:21.174+0000 7fe3eea00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fe3e815f490 con 0x7fe3e80ce1f0 2024-05-28T06:20:21.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:21.174+0000 7fe3eea00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fe3e815f770 con 0x7fe3e80ce5d0 2024-05-28T06:20:21.171 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:21.174+0000 7fe3ecc00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe3e80ce5d0 0x7fe3e815ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:21.171 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:21.174+0000 7fe3ed600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fe3e80ce1f0 0x7fe3e80b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:21.171 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:21.174+0000 7fe3ee000640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7fe3e80b3d60 0x7fe3e80ccb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.96:43834/0 (socket says 172.21.15.96:43834) 2024-05-28T06:20:21.171 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:21.175+0000 7fe3ee000640 1 -- 172.21.15.96:0/1562395247 learned_addr learned my addr 172.21.15.96:0/1562395247 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:20:21.172 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:21.175+0000 7fe3ed600640 1 -- 172.21.15.96:0/1562395247 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe3e80ce5d0 msgr2=0x7fe3e815ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:21.172 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:21.175+0000 7fe3ed600640 1 --2- 172.21.15.96:0/1562395247 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe3e80ce5d0 0x7fe3e815ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:21.172 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:21.175+0000 7fe3ed600640 1 -- 172.21.15.96:0/1562395247 >> v1:172.21.15.123:6789/0 conn(0x7fe3e80b3d60 legacy=0x7fe3e80ccb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:21.172 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:21.175+0000 7fe3ed600640 1 -- 172.21.15.96:0/1562395247 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3e815fa50 con 0x7fe3e80ce1f0 2024-05-28T06:20:21.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:21.177+0000 7fe3ed600640 1 --2- 172.21.15.96:0/1562395247 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3e80ce1f0 0x7fe3e80b4dd0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fe3dc0097a0 tx=0x7fe3dc02a060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ddccaede5aa9dbd1 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:21.174 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:21.177+0000 7fe3e7e00640 1 -- 172.21.15.96:0/1562395247 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3dc037070 con 0x7fe3e80ce1f0 2024-05-28T06:20:21.174 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:21.177+0000 7fe3e7e00640 1 -- 172.21.15.96:0/1562395247 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3dc02adb0 con 0x7fe3e80ce1f0 2024-05-28T06:20:21.174 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:21.177+0000 7fe3e7e00640 1 -- 172.21.15.96:0/1562395247 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3dc032660 con 0x7fe3e80ce1f0 2024-05-28T06:20:21.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:21.178+0000 7fe3eea00640 1 -- 172.21.15.96:0/1562395247 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3e80ce1f0 msgr2=0x7fe3e80b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:21.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:21.178+0000 7fe3eea00640 1 --2- 172.21.15.96:0/1562395247 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3e80ce1f0 0x7fe3e80b4dd0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fe3dc0097a0 tx=0x7fe3dc02a060 comp rx=0 tx=0).stop 2024-05-28T06:20:21.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:21.178+0000 7fe3eea00640 1 -- 172.21.15.96:0/1562395247 shutdown_connections 2024-05-28T06:20:21.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:21.178+0000 7fe3eea00640 1 --2- 172.21.15.96:0/1562395247 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe3e80ce5d0 0x7fe3e815ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:21.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:21.178+0000 7fe3eea00640 1 --2- 172.21.15.96:0/1562395247 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3e80ce1f0 0x7fe3e80b4dd0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:21.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:21.178+0000 7fe3eea00640 1 -- 172.21.15.96:0/1562395247 >> 172.21.15.96:0/1562395247 conn(0x7fe3e806b900 msgr2=0x7fe3e806bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:21.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:21.178+0000 7fe3eea00640 1 -- 172.21.15.96:0/1562395247 shutdown_connections 2024-05-28T06:20:21.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:21.178+0000 7fe3eea00640 1 -- 172.21.15.96:0/1562395247 wait complete. 2024-05-28T06:20:21.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:21.180+0000 7fe3eea00640 1 Processor -- start 2024-05-28T06:20:21.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:21.180+0000 7fe3eea00640 1 -- start start 2024-05-28T06:20:21.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:21.181+0000 7fe3eea00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe3e80b3d60 0x7fe3e80b1b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:21.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:21.181+0000 7fe3eea00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe3e80ce1f0 0x7fe3e80b24d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:21.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:21.181+0000 7fe3eea00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3e80ce5d0 0x7fe3e806f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:21.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:21.181+0000 7fe3eea00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe3e80cd2a0 con 0x7fe3e80b3d60 2024-05-28T06:20:21.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:21.181+0000 7fe3eea00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe3e815f180 con 0x7fe3e80ce5d0 2024-05-28T06:20:21.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:21.181+0000 7fe3eea00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fe3e80cd420 con 0x7fe3e80ce1f0 2024-05-28T06:20:21.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:21.181+0000 7fe3ed600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe3e80b3d60 0x7fe3e80b1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:21.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:21.181+0000 7fe3ed600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe3e80b3d60 0x7fe3e80b1b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58736/0 (socket says 172.21.15.96:58736) 2024-05-28T06:20:21.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:21.181+0000 7fe3ed600640 1 -- 172.21.15.96:0/3208797733 learned_addr learned my addr 172.21.15.96:0/3208797733 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:20:21.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:21.182+0000 7fe3ecc00640 1 --2- 172.21.15.96:0/3208797733 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe3e80ce1f0 0x7fe3e80b24d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:21.180 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:21.182+0000 7fe3ee000640 1 --2- 172.21.15.96:0/3208797733 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3e80ce5d0 0x7fe3e806f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:21.180 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:21.182+0000 7fe3ed600640 1 -- 172.21.15.96:0/3208797733 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe3e80ce1f0 msgr2=0x7fe3e80b24d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:21.180 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:21.182+0000 7fe3ed600640 1 --2- 172.21.15.96:0/3208797733 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe3e80ce1f0 0x7fe3e80b24d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:21.180 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:21.182+0000 7fe3ed600640 1 -- 172.21.15.96:0/3208797733 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3e80ce5d0 msgr2=0x7fe3e806f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:21.180 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:21.182+0000 7fe3ed600640 1 --2- 172.21.15.96:0/3208797733 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3e80ce5d0 0x7fe3e806f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:21.180 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:21.182+0000 7fe3ed600640 1 -- 172.21.15.96:0/3208797733 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3e81531d0 con 0x7fe3e80b3d60 2024-05-28T06:20:21.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:21.182+0000 7fe3ecc00640 1 --2- 172.21.15.96:0/3208797733 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe3e80ce1f0 0x7fe3e80b24d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-28T06:20:21.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:21.182+0000 7fe3ee000640 1 --2- 172.21.15.96:0/3208797733 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3e80ce5d0 0x7fe3e806f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:20:21.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:21.182+0000 7fe3ed600640 1 --2- 172.21.15.96:0/3208797733 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe3e80b3d60 0x7fe3e80b1b70 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7fe3dc035040 tx=0x7fe3dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:21.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:21.183+0000 7fe3e5600640 1 -- 172.21.15.96:0/3208797733 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3dc041070 con 0x7fe3e80b3d60 2024-05-28T06:20:21.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:21.183+0000 7fe3e5600640 1 -- 172.21.15.96:0/3208797733 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3dc03c680 con 0x7fe3e80b3d60 2024-05-28T06:20:21.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:21.183+0000 7fe3eea00640 1 -- 172.21.15.96:0/3208797733 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3e81534b0 con 0x7fe3e80b3d60 2024-05-28T06:20:21.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:21.183+0000 7fe3e5600640 1 -- 172.21.15.96:0/3208797733 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3dc037040 con 0x7fe3e80b3d60 2024-05-28T06:20:21.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:21.183+0000 7fe3eea00640 1 -- 172.21.15.96:0/3208797733 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3e80afce0 con 0x7fe3e80b3d60 2024-05-28T06:20:21.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:21.184+0000 7fe3eea00640 1 -- 172.21.15.96:0/3208797733 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3e80689e0 con 0x7fe3e80b3d60 2024-05-28T06:20:21.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:21.185+0000 7fe3e5600640 1 -- 172.21.15.96:0/3208797733 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 34) ==== 92927+0+0 (secure 0 0 0) 0x7fe3dc03a070 con 0x7fe3e80b3d60 2024-05-28T06:20:21.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:21.186+0000 7fe3e5600640 1 --2- 172.21.15.96:0/3208797733 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe3c806ddc0 0x7fe3c8070280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:21.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:21.186+0000 7fe3e5600640 1 -- 172.21.15.96:0/3208797733 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 5765+0+0 (secure 0 0 0) 0x7fe3dc0aaf50 con 0x7fe3e80b3d60 2024-05-28T06:20:21.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:21.187+0000 7fe3ecc00640 1 --2- 172.21.15.96:0/3208797733 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe3c806ddc0 0x7fe3c8070280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:21.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:21.188+0000 7fe3ecc00640 1 --2- 172.21.15.96:0/3208797733 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe3c806ddc0 0x7fe3c8070280 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fe3e81544d0 tx=0x7fe3d4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:21.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:21.190+0000 7fe3e5600640 1 -- 172.21.15.96:0/3208797733 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe3dc07a9f0 con 0x7fe3e80b3d60 2024-05-28T06:20:21.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:21.424+0000 7fe3eea00640 1 -- 172.21.15.96:0/3208797733 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fe3e80ca070 con 0x7fe3e80b3d60 2024-05-28T06:20:21.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:21.947+0000 7fe3e5600640 1 -- 172.21.15.96:0/3208797733 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v62) ==== 122+0+0 (secure 0 0 0) 0x7fe3dc07a2f0 con 0x7fe3e80b3d60 2024-05-28T06:20:21.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:21.951+0000 7fe3eea00640 1 -- 172.21.15.96:0/3208797733 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe3c806ddc0 msgr2=0x7fe3c8070280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:21.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:21.951+0000 7fe3eea00640 1 --2- 172.21.15.96:0/3208797733 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe3c806ddc0 0x7fe3c8070280 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fe3e81544d0 tx=0x7fe3d4008040 comp rx=0 tx=0).stop 2024-05-28T06:20:21.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:21.951+0000 7fe3eea00640 1 -- 172.21.15.96:0/3208797733 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe3e80b3d60 msgr2=0x7fe3e80b1b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:21.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:21.951+0000 7fe3eea00640 1 --2- 172.21.15.96:0/3208797733 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe3e80b3d60 0x7fe3e80b1b70 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7fe3dc035040 tx=0x7fe3dc0046c0 comp rx=0 tx=0).stop 2024-05-28T06:20:21.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:21.951+0000 7fe3eea00640 1 -- 172.21.15.96:0/3208797733 shutdown_connections 2024-05-28T06:20:21.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:21.951+0000 7fe3eea00640 1 --2- 172.21.15.96:0/3208797733 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe3c806ddc0 0x7fe3c8070280 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:21.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:21.951+0000 7fe3eea00640 1 --2- 172.21.15.96:0/3208797733 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe3e80ce5d0 0x7fe3e806f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:21.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:21.951+0000 7fe3eea00640 1 --2- 172.21.15.96:0/3208797733 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe3e80ce1f0 0x7fe3e80b24d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:21.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:21.951+0000 7fe3eea00640 1 --2- 172.21.15.96:0/3208797733 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe3e80b3d60 0x7fe3e80b1b70 unknown :-1 s=CLOSED pgs=1620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:21.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:21.951+0000 7fe3eea00640 1 -- 172.21.15.96:0/3208797733 >> 172.21.15.96:0/3208797733 conn(0x7fe3e806b900 msgr2=0x7fe3e80ad130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:21.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:21.951+0000 7fe3eea00640 1 -- 172.21.15.96:0/3208797733 shutdown_connections 2024-05-28T06:20:21.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:21.951+0000 7fe3eea00640 1 -- 172.21.15.96:0/3208797733 wait complete. 2024-05-28T06:20:21.968 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-05-28T06:20:21.968 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-28T06:20:22.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.156+0000 7f16de400640 1 Processor -- start 2024-05-28T06:20:22.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.156+0000 7f16de400640 1 -- start start 2024-05-28T06:20:22.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.158+0000 7f16de400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f16d80d24d0 0x7f16d80d52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:22.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.158+0000 7f16de400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f16d80d1650 0x7f16d80d1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:22.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.158+0000 7f16de400640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16d80d4320 con 0x7f16d815e9d0 2024-05-28T06:20:22.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.158+0000 7f16de400640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f16d815edb0 con 0x7f16d80d1650 2024-05-28T06:20:22.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.158+0000 7f16de400640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f16d80d2000 con 0x7f16d80d24d0 2024-05-28T06:20:22.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.158+0000 7f16d7e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f16d80d1650 0x7f16d80d1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:22.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.158+0000 7f16d7e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f16d80d1650 0x7f16d80d1ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58746/0 (socket says 172.21.15.96:58746) 2024-05-28T06:20:22.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.158+0000 7f16d7e00640 1 -- 172.21.15.96:0/988605952 learned_addr learned my addr 172.21.15.96:0/988605952 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:20:22.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.158+0000 7f16d7e00640 1 -- 172.21.15.96:0/988605952 >> v2:172.21.15.123:3300/0 conn(0x7f16d80d24d0 msgr2=0x7f16d80d52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:22.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.158+0000 7f16d7e00640 1 --2- 172.21.15.96:0/988605952 >> v2:172.21.15.123:3300/0 conn(0x7f16d80d24d0 0x7f16d80d52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:22.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.158+0000 7f16d7e00640 1 -- 172.21.15.96:0/988605952 >> v1:172.21.15.123:6789/0 conn(0x7f16d815e9d0 legacy=0x7f16d80d3c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:22.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.158+0000 7f16d7e00640 1 -- 172.21.15.96:0/988605952 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16d80d2b70 con 0x7f16d80d1650 2024-05-28T06:20:22.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.159+0000 7f16d7e00640 1 --2- 172.21.15.96:0/988605952 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f16d80d1650 0x7f16d80d1ac0 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7f16d80b3dc0 tx=0x7f16d002a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f88954109dba8f57 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:22.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.160+0000 7f16d6a00640 1 -- 172.21.15.96:0/988605952 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16d0037070 con 0x7f16d80d1650 2024-05-28T06:20:22.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.160+0000 7f16d6a00640 1 -- 172.21.15.96:0/988605952 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16d0032440 con 0x7f16d80d1650 2024-05-28T06:20:22.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.160+0000 7f16d6a00640 1 -- 172.21.15.96:0/988605952 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16d0032810 con 0x7f16d80d1650 2024-05-28T06:20:22.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.160+0000 7f16de400640 1 -- 172.21.15.96:0/988605952 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f16d80d1650 msgr2=0x7f16d80d1ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:22.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.160+0000 7f16de400640 1 --2- 172.21.15.96:0/988605952 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f16d80d1650 0x7f16d80d1ac0 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7f16d80b3dc0 tx=0x7f16d002a1a0 comp rx=0 tx=0).stop 2024-05-28T06:20:22.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.160+0000 7f16de400640 1 -- 172.21.15.96:0/988605952 shutdown_connections 2024-05-28T06:20:22.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.160+0000 7f16de400640 1 --2- 172.21.15.96:0/988605952 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f16d80d1650 0x7f16d80d1ac0 unknown :-1 s=CLOSED pgs=1621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:22.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.160+0000 7f16de400640 1 --2- 172.21.15.96:0/988605952 >> v2:172.21.15.123:3300/0 conn(0x7f16d80d24d0 0x7f16d80d52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:22.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.160+0000 7f16de400640 1 -- 172.21.15.96:0/988605952 >> 172.21.15.96:0/988605952 conn(0x7f16d8067890 msgr2=0x7f16d8067ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:22.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.160+0000 7f16de400640 1 -- 172.21.15.96:0/988605952 shutdown_connections 2024-05-28T06:20:22.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.160+0000 7f16de400640 1 -- 172.21.15.96:0/988605952 wait complete. 2024-05-28T06:20:22.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.162+0000 7f16de400640 1 Processor -- start 2024-05-28T06:20:22.159 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.162+0000 7f16de400640 1 -- start start 2024-05-28T06:20:22.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.163+0000 7f16de400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f16d80d1650 0x7f16d81f74b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:22.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.163+0000 7f16de400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f16d80d24d0 0x7f16d80cf190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:22.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.163+0000 7f16de400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f16d815e9d0 0x7f16d80ad1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:22.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.163+0000 7f16de400640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f16d80d3410 con 0x7f16d815e9d0 2024-05-28T06:20:22.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.163+0000 7f16de400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f16d80d3130 con 0x7f16d80d24d0 2024-05-28T06:20:22.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.163+0000 7f16de400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f16d80d36f0 con 0x7f16d80d1650 2024-05-28T06:20:22.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.163+0000 7f16d7e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f16d815e9d0 0x7f16d80ad1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:22.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.163+0000 7f16d7e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f16d815e9d0 0x7f16d80ad1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58754/0 (socket says 172.21.15.96:58754) 2024-05-28T06:20:22.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.163+0000 7f16d7e00640 1 -- 172.21.15.96:0/181332781 learned_addr learned my addr 172.21.15.96:0/181332781 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:20:22.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.164+0000 7f16cea00640 1 --2- 172.21.15.96:0/181332781 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f16d80d24d0 0x7f16d80cf190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:22.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.164+0000 7f16d7400640 1 --2- 172.21.15.96:0/181332781 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f16d80d1650 0x7f16d81f74b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:22.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.164+0000 7f16d7e00640 1 -- 172.21.15.96:0/181332781 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f16d80d1650 msgr2=0x7f16d81f74b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:22.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.164+0000 7f16d7e00640 1 --2- 172.21.15.96:0/181332781 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f16d80d1650 0x7f16d81f74b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:22.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.164+0000 7f16d7e00640 1 -- 172.21.15.96:0/181332781 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f16d80d24d0 msgr2=0x7f16d80cf190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:22.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.164+0000 7f16d7e00640 1 --2- 172.21.15.96:0/181332781 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f16d80d24d0 0x7f16d80cf190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:22.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.164+0000 7f16d7e00640 1 -- 172.21.15.96:0/181332781 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16d80cd5d0 con 0x7f16d815e9d0 2024-05-28T06:20:22.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.164+0000 7f16d7e00640 1 --2- 172.21.15.96:0/181332781 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f16d815e9d0 0x7f16d80ad1b0 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f16d00090b0 tx=0x7f16d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:22.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.164+0000 7f16cfe00640 1 -- 172.21.15.96:0/181332781 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16d0041070 con 0x7f16d815e9d0 2024-05-28T06:20:22.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.164+0000 7f16cfe00640 1 -- 172.21.15.96:0/181332781 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16d003c950 con 0x7f16d815e9d0 2024-05-28T06:20:22.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.164+0000 7f16cfe00640 1 -- 172.21.15.96:0/181332781 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16d0037040 con 0x7f16d815e9d0 2024-05-28T06:20:22.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.164+0000 7f16de400640 1 -- 172.21.15.96:0/181332781 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16d80cd8b0 con 0x7f16d815e9d0 2024-05-28T06:20:22.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.165+0000 7f16de400640 1 -- 172.21.15.96:0/181332781 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16d80cde70 con 0x7f16d815e9d0 2024-05-28T06:20:22.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.166+0000 7f16de400640 1 -- 172.21.15.96:0/181332781 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16d80689e0 con 0x7f16d815e9d0 2024-05-28T06:20:22.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.167+0000 7f16cfe00640 1 -- 172.21.15.96:0/181332781 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 34) ==== 92927+0+0 (secure 0 0 0) 0x7f16d0044070 con 0x7f16d815e9d0 2024-05-28T06:20:22.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.167+0000 7f16cfe00640 1 --2- 172.21.15.96:0/181332781 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f16b406ddc0 0x7f16b4070280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:22.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.168+0000 7f16cfe00640 1 -- 172.21.15.96:0/181332781 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 5765+0+0 (secure 0 0 0) 0x7f16d00ab360 con 0x7f16d815e9d0 2024-05-28T06:20:22.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.168+0000 7f16d7400640 1 --2- 172.21.15.96:0/181332781 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f16b406ddc0 0x7f16b4070280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:22.166 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.169+0000 7f16d7400640 1 --2- 172.21.15.96:0/181332781 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f16b406ddc0 0x7f16b4070280 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f16c8007dc0 tx=0x7f16c80072f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:22.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.172+0000 7f16cfe00640 1 -- 172.21.15.96:0/181332781 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f16d007b1b0 con 0x7f16d815e9d0 2024-05-28T06:20:22.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.405+0000 7f16de400640 1 -- 172.21.15.96:0/181332781 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f16d815b180 con 0x7f16d815e9d0 2024-05-28T06:20:22.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.406+0000 7f16cfe00640 1 -- 172.21.15.96:0/181332781 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) ==== 94+0+5189 (secure 0 0 0) 0x7f16d007aab0 con 0x7f16d815e9d0 2024-05-28T06:20:22.403 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:20:22.403 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":62,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5861,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/1001966057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6833","nonce":1001966057},{"type":"v1","addr":"172.21.15.96:6835","nonce":1001966057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":57},{"gid":5864,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6836/4285246962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":4285246962},{"type":"v1","addr":"172.21.15.123:6836","nonce":4285246962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":57},{"gid":6445,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6834/3302807355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":3302807355},{"type":"v1","addr":"172.21.15.96:6834","nonce":3302807355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-28T06:20:18.363024+0000","modified":"2024-05-28T06:20:21.426136+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8739},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8739":{"gid":8739,"name":"b","rank":0,"incarnation":58,"state":"up:active","state_seq":3,"addr":"172.21.15.123:6837/3372660768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6835","nonce":3372660768},{"type":"v1","addr":"172.21.15.123:6837","nonce":3372660768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8739,"qdb_cluster":[8739]},"id":7}]} 2024-05-28T06:20:22.404 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 62 2024-05-28T06:20:22.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.409+0000 7f16de400640 1 -- 172.21.15.96:0/181332781 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f16b406ddc0 msgr2=0x7f16b4070280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:22.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.409+0000 7f16de400640 1 --2- 172.21.15.96:0/181332781 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f16b406ddc0 0x7f16b4070280 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f16c8007dc0 tx=0x7f16c80072f0 comp rx=0 tx=0).stop 2024-05-28T06:20:22.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.409+0000 7f16de400640 1 -- 172.21.15.96:0/181332781 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f16d815e9d0 msgr2=0x7f16d80ad1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:22.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.409+0000 7f16de400640 1 --2- 172.21.15.96:0/181332781 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f16d815e9d0 0x7f16d80ad1b0 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f16d00090b0 tx=0x7f16d00046c0 comp rx=0 tx=0).stop 2024-05-28T06:20:22.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.409+0000 7f16de400640 1 -- 172.21.15.96:0/181332781 shutdown_connections 2024-05-28T06:20:22.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.409+0000 7f16de400640 1 --2- 172.21.15.96:0/181332781 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f16b406ddc0 0x7f16b4070280 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:22.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.409+0000 7f16de400640 1 --2- 172.21.15.96:0/181332781 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f16d815e9d0 0x7f16d80ad1b0 unknown :-1 s=CLOSED pgs=1622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:22.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.409+0000 7f16de400640 1 --2- 172.21.15.96:0/181332781 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f16d80d24d0 0x7f16d80cf190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:22.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.409+0000 7f16de400640 1 --2- 172.21.15.96:0/181332781 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f16d80d1650 0x7f16d81f74b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:22.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.409+0000 7f16de400640 1 -- 172.21.15.96:0/181332781 >> 172.21.15.96:0/181332781 conn(0x7f16d8067890 msgr2=0x7f16d8159a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:22.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.409+0000 7f16de400640 1 -- 172.21.15.96:0/181332781 shutdown_connections 2024-05-28T06:20:22.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.409+0000 7f16de400640 1 -- 172.21.15.96:0/181332781 wait complete. 2024-05-28T06:20:22.424 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:20:22.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.602+0000 7f0707000640 1 Processor -- start 2024-05-28T06:20:22.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.603+0000 7f0707000640 1 -- start start 2024-05-28T06:20:22.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.604+0000 7f0707000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f070006dc20 0x7f07000b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:22.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.604+0000 7f0707000640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07000afd60 con 0x7f070015b3a0 2024-05-28T06:20:22.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.604+0000 7f0707000640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07000af7a0 con 0x7f0700073310 2024-05-28T06:20:22.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.604+0000 7f0707000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f070006f980 con 0x7f070006dc20 2024-05-28T06:20:22.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.604+0000 7f0705c00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f070015b3a0 0x7f0700072c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:51078/0 (socket says 172.21.15.96:51078) 2024-05-28T06:20:22.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.604+0000 7f0705c00640 1 -- 172.21.15.96:0/738141747 learned_addr learned my addr 172.21.15.96:0/738141747 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:20:22.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.604+0000 7f0706600640 1 --2- 172.21.15.96:0/738141747 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f070006dc20 0x7f07000b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:22.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.604+0000 7f06ffe00640 1 -- 172.21.15.96:0/738141747 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1974463652 0 0) 0x7f07000afd60 con 0x7f070015b3a0 2024-05-28T06:20:22.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.604+0000 7f06ffe00640 1 -- 172.21.15.96:0/738141747 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f06e8003520 con 0x7f070015b3a0 2024-05-28T06:20:22.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.605+0000 7f06ffe00640 1 -- 172.21.15.96:0/738141747 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1874527821 0 0) 0x7f06e8003520 con 0x7f070015b3a0 2024-05-28T06:20:22.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.605+0000 7f06ffe00640 1 -- 172.21.15.96:0/738141747 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f070006dc20 msgr2=0x7f07000b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:22.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.605+0000 7f06ffe00640 1 --2- 172.21.15.96:0/738141747 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f070006dc20 0x7f07000b32c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:22.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.605+0000 7f06ffe00640 1 -- 172.21.15.96:0/738141747 >> v1:172.21.15.123:6789/0 conn(0x7f0700073310 legacy=0x7f07000ae550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:22.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.605+0000 7f06ffe00640 1 -- 172.21.15.96:0/738141747 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07000afa80 con 0x7f070015b3a0 2024-05-28T06:20:22.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.605+0000 7f06ffe00640 1 -- 172.21.15.96:0/738141747 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f06f4001670 con 0x7f070015b3a0 2024-05-28T06:20:22.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.605+0000 7f06ffe00640 1 -- 172.21.15.96:0/738141747 >> v1:172.21.15.96:6789/0 conn(0x7f070015b3a0 legacy=0x7f0700072c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:22.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.605+0000 7f06ffe00640 1 --2- 172.21.15.96:0/738141747 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f070006dc20 0x7f06e80039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:22.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.605+0000 7f06ffe00640 1 --2- 172.21.15.96:0/738141747 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f06e8003ff0 0x7f06e8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:22.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.605+0000 7f06ffe00640 1 --2- 172.21.15.96:0/738141747 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f06e8004d40 0x7f06e80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:22.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.605+0000 7f06ffe00640 1 -- 172.21.15.96:0/738141747 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f070006f980 con 0x7f06e8004d40 2024-05-28T06:20:22.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.605+0000 7f06ffe00640 1 -- 172.21.15.96:0/738141747 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f06e800dde0 con 0x7f06e8003ff0 2024-05-28T06:20:22.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.605+0000 7f06ffe00640 1 -- 172.21.15.96:0/738141747 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f06e800dfe0 con 0x7f070006dc20 2024-05-28T06:20:22.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.605+0000 7f0706600640 1 --2- 172.21.15.96:0/738141747 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f06e8004d40 0x7f06e80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:22.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.606+0000 7f0705c00640 1 --2- 172.21.15.96:0/738141747 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f070006dc20 0x7f06e80039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:22.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.606+0000 7f0706600640 1 -- 172.21.15.96:0/738141747 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f070006dc20 msgr2=0x7f06e80039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:22.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.606+0000 7f0706600640 1 --2- 172.21.15.96:0/738141747 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f070006dc20 0x7f06e80039e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:22.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.606+0000 7f0706600640 1 -- 172.21.15.96:0/738141747 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f06e8003ff0 msgr2=0x7f06e8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:22.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.606+0000 7f0705200640 1 --2- 172.21.15.96:0/738141747 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f06e8003ff0 0x7f06e8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:22.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.606+0000 7f0706600640 1 --2- 172.21.15.96:0/738141747 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f06e8003ff0 0x7f06e8004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:22.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.606+0000 7f0706600640 1 -- 172.21.15.96:0/738141747 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06e8003520 con 0x7f06e8004d40 2024-05-28T06:20:22.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.607+0000 7f0706600640 1 --2- 172.21.15.96:0/738141747 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f06e8004d40 0x7f06e80051c0 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7f06f8007360 tx=0x7f06e8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75903a64c9945108 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:22.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.607+0000 7f06ffe00640 1 -- 172.21.15.96:0/738141747 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06f8009be0 con 0x7f06e8004d40 2024-05-28T06:20:22.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.607+0000 7f06ffe00640 1 -- 172.21.15.96:0/738141747 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06f8036600 con 0x7f06e8004d40 2024-05-28T06:20:22.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.607+0000 7f06ffe00640 1 -- 172.21.15.96:0/738141747 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06f80369d0 con 0x7f06e8004d40 2024-05-28T06:20:22.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.607+0000 7f0707000640 1 -- 172.21.15.96:0/738141747 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f06e8004d40 msgr2=0x7f06e80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:22.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.607+0000 7f0707000640 1 --2- 172.21.15.96:0/738141747 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f06e8004d40 0x7f06e80051c0 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7f06f8007360 tx=0x7f06e8005830 comp rx=0 tx=0).stop 2024-05-28T06:20:22.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.607+0000 7f0706600640 1 -- 172.21.15.96:0/738141747 reap_dead start 2024-05-28T06:20:22.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.607+0000 7f0707000640 1 -- 172.21.15.96:0/738141747 shutdown_connections 2024-05-28T06:20:22.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.607+0000 7f0707000640 1 -- 172.21.15.96:0/738141747 >> 172.21.15.96:0/738141747 conn(0x7f07000b1fc0 msgr2=0x7f07000b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:22.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.608+0000 7f0707000640 1 -- 172.21.15.96:0/738141747 shutdown_connections 2024-05-28T06:20:22.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.608+0000 7f0707000640 1 -- 172.21.15.96:0/738141747 wait complete. 2024-05-28T06:20:22.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.609+0000 7f0707000640 1 Processor -- start 2024-05-28T06:20:22.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.609+0000 7f0707000640 1 -- start start 2024-05-28T06:20:22.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.610+0000 7f0707000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f06e8004d40 0x7f0700153090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:22.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.610+0000 7f0707000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0700151a60 0x7f07001564f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:22.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.610+0000 7f0707000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f07001f3c20 0x7f0700144e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:22.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.610+0000 7f0707000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f07000b32c0 con 0x7f07001f3c20 2024-05-28T06:20:22.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.610+0000 7f0707000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f07000ae0c0 con 0x7f0700151a60 2024-05-28T06:20:22.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.610+0000 7f0707000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f07000b3440 con 0x7f06e8004d40 2024-05-28T06:20:22.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.610+0000 7f0706600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f07001f3c20 0x7f0700144e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:22.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.610+0000 7f0706600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f07001f3c20 0x7f0700144e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58784/0 (socket says 172.21.15.96:58784) 2024-05-28T06:20:22.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.610+0000 7f0706600640 1 -- 172.21.15.96:0/2172150684 learned_addr learned my addr 172.21.15.96:0/2172150684 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:20:22.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.611+0000 7f0706600640 1 -- 172.21.15.96:0/2172150684 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f06e8004d40 msgr2=0x7f0700153090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:22.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.611+0000 7f0705200640 1 --2- 172.21.15.96:0/2172150684 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0700151a60 0x7f07001564f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:22.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.611+0000 7f0705c00640 1 --2- 172.21.15.96:0/2172150684 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f06e8004d40 0x7f0700153090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:22.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.611+0000 7f0706600640 1 --2- 172.21.15.96:0/2172150684 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f06e8004d40 0x7f0700153090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:22.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.611+0000 7f0706600640 1 -- 172.21.15.96:0/2172150684 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0700151a60 msgr2=0x7f07001564f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:22.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.611+0000 7f0706600640 1 --2- 172.21.15.96:0/2172150684 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0700151a60 0x7f07001564f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:22.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.611+0000 7f0706600640 1 -- 172.21.15.96:0/2172150684 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07001535d0 con 0x7f07001f3c20 2024-05-28T06:20:22.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.611+0000 7f0705200640 1 --2- 172.21.15.96:0/2172150684 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0700151a60 0x7f07001564f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:20:22.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.611+0000 7f0706600640 1 --2- 172.21.15.96:0/2172150684 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f07001f3c20 0x7f0700144e40 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7f06f8007480 tx=0x7f06f8004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:22.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.611+0000 7f06fd600640 1 -- 172.21.15.96:0/2172150684 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06f8009be0 con 0x7f07001f3c20 2024-05-28T06:20:22.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.611+0000 7f06fd600640 1 -- 172.21.15.96:0/2172150684 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06f8036400 con 0x7f07001f3c20 2024-05-28T06:20:22.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.611+0000 7f06fd600640 1 -- 172.21.15.96:0/2172150684 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06f8038500 con 0x7f07001f3c20 2024-05-28T06:20:22.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.611+0000 7f0707000640 1 -- 172.21.15.96:0/2172150684 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0700151ff0 con 0x7f07001f3c20 2024-05-28T06:20:22.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.612+0000 7f0707000640 1 -- 172.21.15.96:0/2172150684 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07001525b0 con 0x7f07001f3c20 2024-05-28T06:20:22.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.613+0000 7f0707000640 1 -- 172.21.15.96:0/2172150684 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07000689e0 con 0x7f07001f3c20 2024-05-28T06:20:22.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.613+0000 7f06fd600640 1 -- 172.21.15.96:0/2172150684 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 34) ==== 92927+0+0 (secure 0 0 0) 0x7f06f802e620 con 0x7f07001f3c20 2024-05-28T06:20:22.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.614+0000 7f06fd600640 1 --2- 172.21.15.96:0/2172150684 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f06e006ddf0 0x7f06e00702b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:22.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.614+0000 7f06fd600640 1 -- 172.21.15.96:0/2172150684 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 5765+0+0 (secure 0 0 0) 0x7f06f80aad40 con 0x7f07001f3c20 2024-05-28T06:20:22.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.615+0000 7f0705c00640 1 --2- 172.21.15.96:0/2172150684 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f06e006ddf0 0x7f06e00702b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:22.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.616+0000 7f0705c00640 1 --2- 172.21.15.96:0/2172150684 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f06e006ddf0 0x7f06e00702b0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f06f4000f60 tx=0x7f06f4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:22.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.619+0000 7f06fd600640 1 -- 172.21.15.96:0/2172150684 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f06f8048060 con 0x7f07001f3c20 2024-05-28T06:20:22.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.803+0000 7f0707000640 1 -- 172.21.15.96:0/2172150684 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f07000b2e90 con 0x7f07001f3c20 2024-05-28T06:20:22.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.806+0000 7f06fd600640 1 -- 172.21.15.96:0/2172150684 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v103) ==== 74+0+16719 (secure 0 0 0) 0x7f06f8032030 con 0x7f07001f3c20 2024-05-28T06:20:22.803 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:20:22.803 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":103,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:20:19.907412+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-05-28T06:20:16.097747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-05-28T06:20:17.338574+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"103","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":13,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:20:22.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.808+0000 7f0707000640 1 -- 172.21.15.96:0/2172150684 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f06e006ddf0 msgr2=0x7f06e00702b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:22.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.808+0000 7f0707000640 1 --2- 172.21.15.96:0/2172150684 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f06e006ddf0 0x7f06e00702b0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f06f4000f60 tx=0x7f06f4009040 comp rx=0 tx=0).stop 2024-05-28T06:20:22.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.808+0000 7f0707000640 1 -- 172.21.15.96:0/2172150684 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f07001f3c20 msgr2=0x7f0700144e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:22.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.808+0000 7f0707000640 1 --2- 172.21.15.96:0/2172150684 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f07001f3c20 0x7f0700144e40 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7f06f8007480 tx=0x7f06f8004ab0 comp rx=0 tx=0).stop 2024-05-28T06:20:22.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.809+0000 7f0707000640 1 -- 172.21.15.96:0/2172150684 shutdown_connections 2024-05-28T06:20:22.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.809+0000 7f0707000640 1 --2- 172.21.15.96:0/2172150684 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f06e006ddf0 0x7f06e00702b0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:22.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.809+0000 7f0707000640 1 --2- 172.21.15.96:0/2172150684 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f07001f3c20 0x7f0700144e40 unknown :-1 s=CLOSED pgs=1625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:22.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.809+0000 7f0707000640 1 --2- 172.21.15.96:0/2172150684 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0700151a60 0x7f07001564f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:22.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.809+0000 7f0707000640 1 --2- 172.21.15.96:0/2172150684 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f06e8004d40 0x7f0700153090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:22.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.809+0000 7f0707000640 1 -- 172.21.15.96:0/2172150684 >> 172.21.15.96:0/2172150684 conn(0x7f07000b1fc0 msgr2=0x7f07000b4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:22.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.809+0000 7f0707000640 1 -- 172.21.15.96:0/2172150684 shutdown_connections 2024-05-28T06:20:22.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:22.809+0000 7f0707000640 1 -- 172.21.15.96:0/2172150684 wait complete. 2024-05-28T06:20:22.824 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-05-28T06:20:22.824 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-28T06:20:23.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.024+0000 7fd4f2200640 1 Processor -- start 2024-05-28T06:20:23.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.024+0000 7fd4f2200640 1 -- start start 2024-05-28T06:20:23.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.026+0000 7fd4f2200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fd4ec0b4e10 0x7fd4ec0cde30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:23.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.026+0000 7fd4f2200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd4ec0d0ca0 0x7fd4ec099090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:23.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.026+0000 7fd4f2200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd4ec0b51f0 con 0x7fd4ec0ce370 2024-05-28T06:20:23.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.026+0000 7fd4f2200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fd4ec0aef60 con 0x7fd4ec0b4e10 2024-05-28T06:20:23.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.026+0000 7fd4f2200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fd4ec0ad1d0 con 0x7fd4ec0d0ca0 2024-05-28T06:20:23.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.026+0000 7fd4eb400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fd4ec0b4e10 0x7fd4ec0cde30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:23.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.026+0000 7fd4eb400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fd4ec0b4e10 0x7fd4ec0cde30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58790/0 (socket says 172.21.15.96:58790) 2024-05-28T06:20:23.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.026+0000 7fd4eb400640 1 -- 172.21.15.96:0/502643072 learned_addr learned my addr 172.21.15.96:0/502643072 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:20:23.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.027+0000 7fd4ebe00640 1 --2- 172.21.15.96:0/502643072 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd4ec0d0ca0 0x7fd4ec099090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:23.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.027+0000 7fd4eb400640 1 -- 172.21.15.96:0/502643072 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd4ec0d0ca0 msgr2=0x7fd4ec099090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:23.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.027+0000 7fd4eb400640 1 --2- 172.21.15.96:0/502643072 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd4ec0d0ca0 0x7fd4ec099090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:23.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.027+0000 7fd4eb400640 1 -- 172.21.15.96:0/502643072 >> v1:172.21.15.123:6789/0 conn(0x7fd4ec0ce370 legacy=0x7fd4ec0d0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:23.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.027+0000 7fd4eb400640 1 -- 172.21.15.96:0/502643072 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4ec0ad470 con 0x7fd4ec0b4e10 2024-05-28T06:20:23.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.028+0000 7fd4eb400640 1 --2- 172.21.15.96:0/502643072 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd4ec0b4e10 0x7fd4ec0cde30 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc00bb10 tx=0x7fd4dc02c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e00dd0ea55e9d999 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:23.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.029+0000 7fd4ea000640 1 -- 172.21.15.96:0/502643072 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4dc032790 con 0x7fd4ec0b4e10 2024-05-28T06:20:23.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.029+0000 7fd4ea000640 1 -- 172.21.15.96:0/502643072 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4dc038070 con 0x7fd4ec0b4e10 2024-05-28T06:20:23.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.029+0000 7fd4ea000640 1 -- 172.21.15.96:0/502643072 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4dc032b50 con 0x7fd4ec0b4e10 2024-05-28T06:20:23.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.029+0000 7fd4f2200640 1 -- 172.21.15.96:0/502643072 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd4ec0b4e10 msgr2=0x7fd4ec0cde30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:23.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.029+0000 7fd4f2200640 1 --2- 172.21.15.96:0/502643072 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd4ec0b4e10 0x7fd4ec0cde30 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc00bb10 tx=0x7fd4dc02c890 comp rx=0 tx=0).stop 2024-05-28T06:20:23.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.029+0000 7fd4f2200640 1 -- 172.21.15.96:0/502643072 shutdown_connections 2024-05-28T06:20:23.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.029+0000 7fd4f2200640 1 --2- 172.21.15.96:0/502643072 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd4ec0d0ca0 0x7fd4ec099090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:23.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.029+0000 7fd4f2200640 1 --2- 172.21.15.96:0/502643072 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd4ec0b4e10 0x7fd4ec0cde30 unknown :-1 s=CLOSED pgs=1626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:23.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.030+0000 7fd4f2200640 1 -- 172.21.15.96:0/502643072 >> 172.21.15.96:0/502643072 conn(0x7fd4ec06bae0 msgr2=0x7fd4ec06bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:23.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.030+0000 7fd4f2200640 1 -- 172.21.15.96:0/502643072 shutdown_connections 2024-05-28T06:20:23.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.030+0000 7fd4f2200640 1 -- 172.21.15.96:0/502643072 wait complete. 2024-05-28T06:20:23.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.031+0000 7fd4f2200640 1 Processor -- start 2024-05-28T06:20:23.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.031+0000 7fd4f2200640 1 -- start start 2024-05-28T06:20:23.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.032+0000 7fd4f2200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd4ec0b4e10 0x7fd4ec0996e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:23.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.032+0000 7fd4f2200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd4ec0ce370 0x7fd4ec072b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:23.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.032+0000 7fd4f2200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd4ec0d0ca0 0x7fd4ec0730c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:23.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.032+0000 7fd4f2200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd4ec0cea00 con 0x7fd4ec0ce370 2024-05-28T06:20:23.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.032+0000 7fd4f2200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd4ec0ce880 con 0x7fd4ec0d0ca0 2024-05-28T06:20:23.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.032+0000 7fd4f2200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fd4ec0b3150 con 0x7fd4ec0b4e10 2024-05-28T06:20:23.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.033+0000 7fd4eaa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd4ec0ce370 0x7fd4ec072b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:23.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.033+0000 7fd4eaa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd4ec0ce370 0x7fd4ec072b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58796/0 (socket says 172.21.15.96:58796) 2024-05-28T06:20:23.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.033+0000 7fd4eaa00640 1 -- 172.21.15.96:0/216563673 learned_addr learned my addr 172.21.15.96:0/216563673 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:20:23.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.033+0000 7fd4eb400640 1 --2- 172.21.15.96:0/216563673 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd4ec0b4e10 0x7fd4ec0996e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:23.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.033+0000 7fd4ebe00640 1 --2- 172.21.15.96:0/216563673 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd4ec0d0ca0 0x7fd4ec0730c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:23.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.033+0000 7fd4eaa00640 1 -- 172.21.15.96:0/216563673 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd4ec0b4e10 msgr2=0x7fd4ec0996e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:23.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.033+0000 7fd4eaa00640 1 --2- 172.21.15.96:0/216563673 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd4ec0b4e10 0x7fd4ec0996e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:23.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.033+0000 7fd4eaa00640 1 -- 172.21.15.96:0/216563673 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd4ec0d0ca0 msgr2=0x7fd4ec0730c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:23.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.033+0000 7fd4eaa00640 1 --2- 172.21.15.96:0/216563673 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd4ec0d0ca0 0x7fd4ec0730c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:23.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.033+0000 7fd4eaa00640 1 -- 172.21.15.96:0/216563673 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4ec15bc20 con 0x7fd4ec0ce370 2024-05-28T06:20:23.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.033+0000 7fd4eaa00640 1 --2- 172.21.15.96:0/216563673 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd4ec0ce370 0x7fd4ec072b80 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7fd4d400c730 tx=0x7fd4d400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:23.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.034+0000 7fd4e3400640 1 -- 172.21.15.96:0/216563673 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4d4016020 con 0x7fd4ec0ce370 2024-05-28T06:20:23.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.034+0000 7fd4e3400640 1 -- 172.21.15.96:0/216563673 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4d4006d50 con 0x7fd4ec0ce370 2024-05-28T06:20:23.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.034+0000 7fd4f2200640 1 -- 172.21.15.96:0/216563673 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4ec15bf00 con 0x7fd4ec0ce370 2024-05-28T06:20:23.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.034+0000 7fd4e3400640 1 -- 172.21.15.96:0/216563673 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4d4003bb0 con 0x7fd4ec0ce370 2024-05-28T06:20:23.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.034+0000 7fd4f2200640 1 -- 172.21.15.96:0/216563673 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4ec15c4c0 con 0x7fd4ec0ce370 2024-05-28T06:20:23.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.035+0000 7fd4f2200640 1 -- 172.21.15.96:0/216563673 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4ec0689e0 con 0x7fd4ec0ce370 2024-05-28T06:20:23.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.036+0000 7fd4e3400640 1 -- 172.21.15.96:0/216563673 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 34) ==== 92927+0+0 (secure 0 0 0) 0x7fd4d400a850 con 0x7fd4ec0ce370 2024-05-28T06:20:23.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.037+0000 7fd4e3400640 1 --2- 172.21.15.96:0/216563673 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd4c806ddc0 0x7fd4c8070280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:23.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.037+0000 7fd4e3400640 1 -- 172.21.15.96:0/216563673 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 5765+0+0 (secure 0 0 0) 0x7fd4d408a530 con 0x7fd4ec0ce370 2024-05-28T06:20:23.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.038+0000 7fd4eb400640 1 --2- 172.21.15.96:0/216563673 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd4c806ddc0 0x7fd4c8070280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:23.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.039+0000 7fd4eb400640 1 --2- 172.21.15.96:0/216563673 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd4c806ddc0 0x7fd4c8070280 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc037000 tx=0x7fd4dc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:23.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.042+0000 7fd4e3400640 1 -- 172.21.15.96:0/216563673 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd4d4020020 con 0x7fd4ec0ce370 2024-05-28T06:20:23.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.239+0000 7fd4f2200640 1 -- 172.21.15.96:0/216563673 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd4ec06f540 con 0x7fd4c806ddc0 2024-05-28T06:20:23.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.260+0000 7fd4e3400640 1 -- 172.21.15.96:0/216563673 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252451 (secure 0 0 0) 0x7fd4ec06f540 con 0x7fd4c806ddc0 2024-05-28T06:20:23.261 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:20:23.262 INFO:teuthology.orchestra.run.smithi096.stderr:dumped all 2024-05-28T06:20:23.263 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.266+0000 7fd4f2200640 1 -- 172.21.15.96:0/216563673 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd4c806ddc0 msgr2=0x7fd4c8070280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:23.264 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.266+0000 7fd4f2200640 1 --2- 172.21.15.96:0/216563673 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd4c806ddc0 0x7fd4c8070280 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc037000 tx=0x7fd4dc008040 comp rx=0 tx=0).stop 2024-05-28T06:20:23.264 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.266+0000 7fd4f2200640 1 -- 172.21.15.96:0/216563673 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd4ec0ce370 msgr2=0x7fd4ec072b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:23.264 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.266+0000 7fd4f2200640 1 --2- 172.21.15.96:0/216563673 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd4ec0ce370 0x7fd4ec072b80 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7fd4d400c730 tx=0x7fd4d400cbf0 comp rx=0 tx=0).stop 2024-05-28T06:20:23.264 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.266+0000 7fd4f2200640 1 -- 172.21.15.96:0/216563673 shutdown_connections 2024-05-28T06:20:23.264 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.266+0000 7fd4f2200640 1 --2- 172.21.15.96:0/216563673 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd4c806ddc0 0x7fd4c8070280 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:23.264 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.266+0000 7fd4f2200640 1 --2- 172.21.15.96:0/216563673 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd4ec0d0ca0 0x7fd4ec0730c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:23.264 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.266+0000 7fd4f2200640 1 --2- 172.21.15.96:0/216563673 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd4ec0ce370 0x7fd4ec072b80 unknown :-1 s=CLOSED pgs=1627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:23.264 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.266+0000 7fd4f2200640 1 --2- 172.21.15.96:0/216563673 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd4ec0b4e10 0x7fd4ec0996e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:23.264 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.266+0000 7fd4f2200640 1 -- 172.21.15.96:0/216563673 >> 172.21.15.96:0/216563673 conn(0x7fd4ec06bae0 msgr2=0x7fd4ec0ca890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:23.264 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.266+0000 7fd4f2200640 1 -- 172.21.15.96:0/216563673 shutdown_connections 2024-05-28T06:20:23.265 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.266+0000 7fd4f2200640 1 -- 172.21.15.96:0/216563673 wait complete. 2024-05-28T06:20:23.281 INFO:teuthology.orchestra.run.smithi096.stdout:{"pg_ready":true,"pg_map":{"version":587,"stamp":"2024-05-28T06:20:21.915176+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":136,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":880592,"kb_used_data":11092,"kb_used_omap":12,"kb_used_meta":869363,"kb_avail":754094128,"statfs":{"total":773094113280,"available":772192387072,"internally_reserved":0,"allocated":11358208,"data_stored":5588288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":890228272},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.032546"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.954908+0000","last_change":"2024-05-28T06:20:18.944123+0000","last_active":"2024-05-28T06:20:18.954908+0000","last_peered":"2024-05-28T06:20:18.954908+0000","last_clean":"2024-05-28T06:20:18.954908+0000","last_became_active":"2024-05-28T06:20:18.942920+0000","last_became_peered":"2024-05-28T06:20:18.942920+0000","last_unstale":"2024-05-28T06:20:18.954908+0000","last_undegraded":"2024-05-28T06:20:18.954908+0000","last_fullsized":"2024-05-28T06:20:18.954908+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:37:54.494431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.947560+0000","last_change":"2024-05-28T06:20:18.940202+0000","last_active":"2024-05-28T06:20:18.947560+0000","last_peered":"2024-05-28T06:20:18.947560+0000","last_clean":"2024-05-28T06:20:18.947560+0000","last_became_active":"2024-05-28T06:20:18.939917+0000","last_became_peered":"2024-05-28T06:20:18.939917+0000","last_unstale":"2024-05-28T06:20:18.947560+0000","last_undegraded":"2024-05-28T06:20:18.947560+0000","last_fullsized":"2024-05-28T06:20:18.947560+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:49:52.448395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.955120+0000","last_change":"2024-05-28T06:20:18.944516+0000","last_active":"2024-05-28T06:20:18.955120+0000","last_peered":"2024-05-28T06:20:18.955120+0000","last_clean":"2024-05-28T06:20:18.955120+0000","last_became_active":"2024-05-28T06:20:18.943234+0000","last_became_peered":"2024-05-28T06:20:18.943234+0000","last_unstale":"2024-05-28T06:20:18.955120+0000","last_undegraded":"2024-05-28T06:20:18.955120+0000","last_fullsized":"2024-05-28T06:20:18.955120+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:04:37.338955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.948997+0000","last_change":"2024-05-28T06:20:18.945783+0000","last_active":"2024-05-28T06:20:18.948997+0000","last_peered":"2024-05-28T06:20:18.948997+0000","last_clean":"2024-05-28T06:20:18.948997+0000","last_became_active":"2024-05-28T06:20:18.945479+0000","last_became_peered":"2024-05-28T06:20:18.945479+0000","last_unstale":"2024-05-28T06:20:18.948997+0000","last_undegraded":"2024-05-28T06:20:18.948997+0000","last_fullsized":"2024-05-28T06:20:18.948997+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:53:31.071486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.948862+0000","last_change":"2024-05-28T06:20:18.941714+0000","last_active":"2024-05-28T06:20:18.948862+0000","last_peered":"2024-05-28T06:20:18.948862+0000","last_clean":"2024-05-28T06:20:18.948862+0000","last_became_active":"2024-05-28T06:20:18.941310+0000","last_became_peered":"2024-05-28T06:20:18.941310+0000","last_unstale":"2024-05-28T06:20:18.948862+0000","last_undegraded":"2024-05-28T06:20:18.948862+0000","last_fullsized":"2024-05-28T06:20:18.948862+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:19:42.753227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-05-28T06:20:20.415278+0000","last_change":"2024-05-28T06:20:18.944910+0000","last_active":"2024-05-28T06:20:20.415278+0000","last_peered":"2024-05-28T06:20:20.415278+0000","last_clean":"2024-05-28T06:20:20.415278+0000","last_became_active":"2024-05-28T06:20:18.944631+0000","last_became_peered":"2024-05-28T06:20:18.944631+0000","last_unstale":"2024-05-28T06:20:20.415278+0000","last_undegraded":"2024-05-28T06:20:20.415278+0000","last_fullsized":"2024-05-28T06:20:20.415278+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:55:45.904700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-05-28T06:20:20.415280+0000","last_change":"2024-05-28T06:20:18.945364+0000","last_active":"2024-05-28T06:20:20.415280+0000","last_peered":"2024-05-28T06:20:20.415280+0000","last_clean":"2024-05-28T06:20:20.415280+0000","last_became_active":"2024-05-28T06:20:18.944879+0000","last_became_peered":"2024-05-28T06:20:18.944879+0000","last_unstale":"2024-05-28T06:20:20.415280+0000","last_undegraded":"2024-05-28T06:20:20.415280+0000","last_fullsized":"2024-05-28T06:20:20.415280+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:20:02.787394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.934711+0000","last_change":"2024-05-28T06:20:18.932617+0000","last_active":"2024-05-28T06:20:18.934711+0000","last_peered":"2024-05-28T06:20:18.934711+0000","last_clean":"2024-05-28T06:20:18.934711+0000","last_became_active":"2024-05-28T06:20:18.932263+0000","last_became_peered":"2024-05-28T06:20:18.932263+0000","last_unstale":"2024-05-28T06:20:18.934711+0000","last_undegraded":"2024-05-28T06:20:18.934711+0000","last_fullsized":"2024-05-28T06:20:18.934711+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:52:01.724740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-05-28T06:20:19.926369+0000","last_change":"2024-05-28T06:20:18.940604+0000","last_active":"2024-05-28T06:20:19.926369+0000","last_peered":"2024-05-28T06:20:19.926369+0000","last_clean":"2024-05-28T06:20:19.926369+0000","last_became_active":"2024-05-28T06:20:18.940288+0000","last_became_peered":"2024-05-28T06:20:18.940288+0000","last_unstale":"2024-05-28T06:20:19.926369+0000","last_undegraded":"2024-05-28T06:20:19.926369+0000","last_fullsized":"2024-05-28T06:20:19.926369+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:13:51.233304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.951547+0000","last_change":"2024-05-28T06:20:18.950570+0000","last_active":"2024-05-28T06:20:18.951547+0000","last_peered":"2024-05-28T06:20:18.951547+0000","last_clean":"2024-05-28T06:20:18.951547+0000","last_became_active":"2024-05-28T06:20:18.950151+0000","last_became_peered":"2024-05-28T06:20:18.950151+0000","last_unstale":"2024-05-28T06:20:18.951547+0000","last_undegraded":"2024-05-28T06:20:18.951547+0000","last_fullsized":"2024-05-28T06:20:18.951547+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:13:34.791207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-05-28T06:20:20.416022+0000","last_change":"2024-05-28T06:20:18.947205+0000","last_active":"2024-05-28T06:20:20.416022+0000","last_peered":"2024-05-28T06:20:20.416022+0000","last_clean":"2024-05-28T06:20:20.416022+0000","last_became_active":"2024-05-28T06:20:18.946278+0000","last_became_peered":"2024-05-28T06:20:18.946278+0000","last_unstale":"2024-05-28T06:20:20.416022+0000","last_undegraded":"2024-05-28T06:20:20.416022+0000","last_fullsized":"2024-05-28T06:20:20.416022+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T18:02:36.378093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-05-28T06:20:19.926077+0000","last_change":"2024-05-28T06:20:18.939540+0000","last_active":"2024-05-28T06:20:19.926077+0000","last_peered":"2024-05-28T06:20:19.926077+0000","last_clean":"2024-05-28T06:20:19.926077+0000","last_became_active":"2024-05-28T06:20:18.938306+0000","last_became_peered":"2024-05-28T06:20:18.938306+0000","last_unstale":"2024-05-28T06:20:19.926077+0000","last_undegraded":"2024-05-28T06:20:19.926077+0000","last_fullsized":"2024-05-28T06:20:19.926077+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:23:32.531274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.949054+0000","last_change":"2024-05-28T06:20:18.947647+0000","last_active":"2024-05-28T06:20:18.949054+0000","last_peered":"2024-05-28T06:20:18.949054+0000","last_clean":"2024-05-28T06:20:18.949054+0000","last_became_active":"2024-05-28T06:20:18.944391+0000","last_became_peered":"2024-05-28T06:20:18.944391+0000","last_unstale":"2024-05-28T06:20:18.949054+0000","last_undegraded":"2024-05-28T06:20:18.949054+0000","last_fullsized":"2024-05-28T06:20:18.949054+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:06:21.231285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.947233+0000","last_change":"2024-05-28T06:20:18.940843+0000","last_active":"2024-05-28T06:20:18.947233+0000","last_peered":"2024-05-28T06:20:18.947233+0000","last_clean":"2024-05-28T06:20:18.947233+0000","last_became_active":"2024-05-28T06:20:18.939324+0000","last_became_peered":"2024-05-28T06:20:18.939324+0000","last_unstale":"2024-05-28T06:20:18.947233+0000","last_undegraded":"2024-05-28T06:20:18.947233+0000","last_fullsized":"2024-05-28T06:20:18.947233+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:55:01.131906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.947400+0000","last_change":"2024-05-28T06:20:18.942145+0000","last_active":"2024-05-28T06:20:18.947400+0000","last_peered":"2024-05-28T06:20:18.947400+0000","last_clean":"2024-05-28T06:20:18.947400+0000","last_became_active":"2024-05-28T06:20:18.941877+0000","last_became_peered":"2024-05-28T06:20:18.941877+0000","last_unstale":"2024-05-28T06:20:18.947400+0000","last_undegraded":"2024-05-28T06:20:18.947400+0000","last_fullsized":"2024-05-28T06:20:18.947400+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:54:18.917443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-05-28T06:20:20.415912+0000","last_change":"2024-05-28T06:20:18.942276+0000","last_active":"2024-05-28T06:20:20.415912+0000","last_peered":"2024-05-28T06:20:20.415912+0000","last_clean":"2024-05-28T06:20:20.415912+0000","last_became_active":"2024-05-28T06:20:18.941966+0000","last_became_peered":"2024-05-28T06:20:18.941966+0000","last_unstale":"2024-05-28T06:20:20.415912+0000","last_undegraded":"2024-05-28T06:20:20.415912+0000","last_fullsized":"2024-05-28T06:20:20.415912+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:06:38.458525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.940444+0000","last_change":"2024-05-28T06:20:18.937719+0000","last_active":"2024-05-28T06:20:18.940444+0000","last_peered":"2024-05-28T06:20:18.940444+0000","last_clean":"2024-05-28T06:20:18.940444+0000","last_became_active":"2024-05-28T06:20:18.935237+0000","last_became_peered":"2024-05-28T06:20:18.935237+0000","last_unstale":"2024-05-28T06:20:18.940444+0000","last_undegraded":"2024-05-28T06:20:18.940444+0000","last_fullsized":"2024-05-28T06:20:18.940444+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:13:34.791207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.951546+0000","last_change":"2024-05-28T06:20:18.949468+0000","last_active":"2024-05-28T06:20:18.951546+0000","last_peered":"2024-05-28T06:20:18.951546+0000","last_clean":"2024-05-28T06:20:18.951546+0000","last_became_active":"2024-05-28T06:20:18.948150+0000","last_became_peered":"2024-05-28T06:20:18.948150+0000","last_unstale":"2024-05-28T06:20:18.951546+0000","last_undegraded":"2024-05-28T06:20:18.951546+0000","last_fullsized":"2024-05-28T06:20:18.951546+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:23:32.531274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.952847+0000","last_change":"2024-05-28T06:20:18.951827+0000","last_active":"2024-05-28T06:20:18.952847+0000","last_peered":"2024-05-28T06:20:18.952847+0000","last_clean":"2024-05-28T06:20:18.952847+0000","last_became_active":"2024-05-28T06:20:18.951264+0000","last_became_peered":"2024-05-28T06:20:18.951264+0000","last_unstale":"2024-05-28T06:20:18.952847+0000","last_undegraded":"2024-05-28T06:20:18.952847+0000","last_fullsized":"2024-05-28T06:20:18.952847+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:59:49.791633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-05-28T06:20:19.926398+0000","last_change":"2024-05-28T06:20:18.942158+0000","last_active":"2024-05-28T06:20:19.926398+0000","last_peered":"2024-05-28T06:20:19.926398+0000","last_clean":"2024-05-28T06:20:19.926398+0000","last_became_active":"2024-05-28T06:20:18.941531+0000","last_became_peered":"2024-05-28T06:20:18.941531+0000","last_unstale":"2024-05-28T06:20:19.926398+0000","last_undegraded":"2024-05-28T06:20:19.926398+0000","last_fullsized":"2024-05-28T06:20:19.926398+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:49:52.448395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.947549+0000","last_change":"2024-05-28T06:20:18.944885+0000","last_active":"2024-05-28T06:20:18.947549+0000","last_peered":"2024-05-28T06:20:18.947549+0000","last_clean":"2024-05-28T06:20:18.947549+0000","last_became_active":"2024-05-28T06:20:18.944394+0000","last_became_peered":"2024-05-28T06:20:18.944394+0000","last_unstale":"2024-05-28T06:20:18.947549+0000","last_undegraded":"2024-05-28T06:20:18.947549+0000","last_fullsized":"2024-05-28T06:20:18.947549+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:15:45.567814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.947545+0000","last_change":"2024-05-28T06:20:18.940292+0000","last_active":"2024-05-28T06:20:18.947545+0000","last_peered":"2024-05-28T06:20:18.947545+0000","last_clean":"2024-05-28T06:20:18.947545+0000","last_became_active":"2024-05-28T06:20:18.939893+0000","last_became_peered":"2024-05-28T06:20:18.939893+0000","last_unstale":"2024-05-28T06:20:18.947545+0000","last_undegraded":"2024-05-28T06:20:18.947545+0000","last_fullsized":"2024-05-28T06:20:18.947545+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:25:49.657873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.947510+0000","last_change":"2024-05-28T06:20:18.941107+0000","last_active":"2024-05-28T06:20:18.947510+0000","last_peered":"2024-05-28T06:20:18.947510+0000","last_clean":"2024-05-28T06:20:18.947510+0000","last_became_active":"2024-05-28T06:20:18.939567+0000","last_became_peered":"2024-05-28T06:20:18.939567+0000","last_unstale":"2024-05-28T06:20:18.947510+0000","last_undegraded":"2024-05-28T06:20:18.947510+0000","last_fullsized":"2024-05-28T06:20:18.947510+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:48:15.007285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-05-28T06:20:20.415423+0000","last_change":"2024-05-28T06:20:18.952910+0000","last_active":"2024-05-28T06:20:20.415423+0000","last_peered":"2024-05-28T06:20:20.415423+0000","last_clean":"2024-05-28T06:20:20.415423+0000","last_became_active":"2024-05-28T06:20:18.952586+0000","last_became_peered":"2024-05-28T06:20:18.952586+0000","last_unstale":"2024-05-28T06:20:20.415423+0000","last_undegraded":"2024-05-28T06:20:20.415423+0000","last_fullsized":"2024-05-28T06:20:20.415423+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:12:30.286713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.950664+0000","last_change":"2024-05-28T06:20:18.946721+0000","last_active":"2024-05-28T06:20:18.950664+0000","last_peered":"2024-05-28T06:20:18.950664+0000","last_clean":"2024-05-28T06:20:18.950664+0000","last_became_active":"2024-05-28T06:20:18.944468+0000","last_became_peered":"2024-05-28T06:20:18.944468+0000","last_unstale":"2024-05-28T06:20:18.950664+0000","last_undegraded":"2024-05-28T06:20:18.950664+0000","last_fullsized":"2024-05-28T06:20:18.950664+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:23:50.885000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.939414+0000","last_change":"2024-05-28T06:20:18.935988+0000","last_active":"2024-05-28T06:20:18.939414+0000","last_peered":"2024-05-28T06:20:18.939414+0000","last_clean":"2024-05-28T06:20:18.939414+0000","last_became_active":"2024-05-28T06:20:18.935456+0000","last_became_peered":"2024-05-28T06:20:18.935456+0000","last_unstale":"2024-05-28T06:20:18.939414+0000","last_undegraded":"2024-05-28T06:20:18.939414+0000","last_fullsized":"2024-05-28T06:20:18.939414+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:06:21.231285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-05-28T06:20:20.415538+0000","last_change":"2024-05-28T06:20:18.953983+0000","last_active":"2024-05-28T06:20:20.415538+0000","last_peered":"2024-05-28T06:20:20.415538+0000","last_clean":"2024-05-28T06:20:20.415538+0000","last_became_active":"2024-05-28T06:20:18.953585+0000","last_became_peered":"2024-05-28T06:20:18.953585+0000","last_unstale":"2024-05-28T06:20:20.415538+0000","last_undegraded":"2024-05-28T06:20:20.415538+0000","last_fullsized":"2024-05-28T06:20:20.415538+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:34:04.919342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.950379+0000","last_change":"2024-05-28T06:20:18.947685+0000","last_active":"2024-05-28T06:20:18.950379+0000","last_peered":"2024-05-28T06:20:18.950379+0000","last_clean":"2024-05-28T06:20:18.950379+0000","last_became_active":"2024-05-28T06:20:18.947377+0000","last_became_peered":"2024-05-28T06:20:18.947377+0000","last_unstale":"2024-05-28T06:20:18.950379+0000","last_undegraded":"2024-05-28T06:20:18.950379+0000","last_fullsized":"2024-05-28T06:20:18.950379+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:54:18.917443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.945209+0000","last_change":"2024-05-28T06:20:18.939097+0000","last_active":"2024-05-28T06:20:18.945209+0000","last_peered":"2024-05-28T06:20:18.945209+0000","last_clean":"2024-05-28T06:20:18.945209+0000","last_became_active":"2024-05-28T06:20:18.937724+0000","last_became_peered":"2024-05-28T06:20:18.937724+0000","last_unstale":"2024-05-28T06:20:18.945209+0000","last_undegraded":"2024-05-28T06:20:18.945209+0000","last_fullsized":"2024-05-28T06:20:18.945209+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:23:32.531274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.956338+0000","last_change":"2024-05-28T06:20:18.953508+0000","last_active":"2024-05-28T06:20:18.956338+0000","last_peered":"2024-05-28T06:20:18.956338+0000","last_clean":"2024-05-28T06:20:18.956338+0000","last_became_active":"2024-05-28T06:20:18.953203+0000","last_became_peered":"2024-05-28T06:20:18.953203+0000","last_unstale":"2024-05-28T06:20:18.956338+0000","last_undegraded":"2024-05-28T06:20:18.956338+0000","last_fullsized":"2024-05-28T06:20:18.956338+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:47:56.999907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.947376+0000","last_change":"2024-05-28T06:20:18.942039+0000","last_active":"2024-05-28T06:20:18.947376+0000","last_peered":"2024-05-28T06:20:18.947376+0000","last_clean":"2024-05-28T06:20:18.947376+0000","last_became_active":"2024-05-28T06:20:18.940022+0000","last_became_peered":"2024-05-28T06:20:18.940022+0000","last_unstale":"2024-05-28T06:20:18.947376+0000","last_undegraded":"2024-05-28T06:20:18.947376+0000","last_fullsized":"2024-05-28T06:20:18.947376+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:23:50.885000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.950488+0000","last_change":"2024-05-28T06:20:18.942741+0000","last_active":"2024-05-28T06:20:18.950488+0000","last_peered":"2024-05-28T06:20:18.950488+0000","last_clean":"2024-05-28T06:20:18.950488+0000","last_became_active":"2024-05-28T06:20:18.942451+0000","last_became_peered":"2024-05-28T06:20:18.942451+0000","last_unstale":"2024-05-28T06:20:18.950488+0000","last_undegraded":"2024-05-28T06:20:18.950488+0000","last_fullsized":"2024-05-28T06:20:18.950488+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:44:20.546170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.951195+0000","last_change":"2024-05-28T06:20:18.948939+0000","last_active":"2024-05-28T06:20:18.951195+0000","last_peered":"2024-05-28T06:20:18.951195+0000","last_clean":"2024-05-28T06:20:18.951195+0000","last_became_active":"2024-05-28T06:20:18.948623+0000","last_became_peered":"2024-05-28T06:20:18.948623+0000","last_unstale":"2024-05-28T06:20:18.951195+0000","last_undegraded":"2024-05-28T06:20:18.951195+0000","last_fullsized":"2024-05-28T06:20:18.951195+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:47:11.651620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"102'4","reported_seq":16,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.962097+0000","last_change":"2024-05-28T06:20:17.921647+0000","last_active":"2024-05-28T06:20:18.962097+0000","last_peered":"2024-05-28T06:20:18.962097+0000","last_clean":"2024-05-28T06:20:18.962097+0000","last_became_active":"2024-05-28T06:20:17.920107+0000","last_became_peered":"2024-05-28T06:20:17.920107+0000","last_unstale":"2024-05-28T06:20:18.962097+0000","last_undegraded":"2024-05-28T06:20:18.962097+0000","last_fullsized":"2024-05-28T06:20:18.962097+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_clean_scrub_stamp":"2024-05-28T06:20:16.867179+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:23:49.872522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.949362+0000","last_change":"2024-05-28T06:20:18.948352+0000","last_active":"2024-05-28T06:20:18.949362+0000","last_peered":"2024-05-28T06:20:18.949362+0000","last_clean":"2024-05-28T06:20:18.949362+0000","last_became_active":"2024-05-28T06:20:18.948042+0000","last_became_peered":"2024-05-28T06:20:18.948042+0000","last_unstale":"2024-05-28T06:20:18.949362+0000","last_undegraded":"2024-05-28T06:20:18.949362+0000","last_fullsized":"2024-05-28T06:20:18.949362+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:13:34.791207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"102'1","reported_seq":15,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-05-28T06:20:20.415833+0000","last_change":"2024-05-28T06:20:17.912807+0000","last_active":"2024-05-28T06:20:20.415833+0000","last_peered":"2024-05-28T06:20:20.415833+0000","last_clean":"2024-05-28T06:20:20.415833+0000","last_became_active":"2024-05-28T06:20:17.912373+0000","last_became_peered":"2024-05-28T06:20:17.912373+0000","last_unstale":"2024-05-28T06:20:20.415833+0000","last_undegraded":"2024-05-28T06:20:20.415833+0000","last_fullsized":"2024-05-28T06:20:20.415833+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_clean_scrub_stamp":"2024-05-28T06:20:16.867179+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:23:49.872522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.954341+0000","last_change":"2024-05-28T06:20:18.943347+0000","last_active":"2024-05-28T06:20:18.954341+0000","last_peered":"2024-05-28T06:20:18.954341+0000","last_clean":"2024-05-28T06:20:18.954341+0000","last_became_active":"2024-05-28T06:20:18.942963+0000","last_became_peered":"2024-05-28T06:20:18.942963+0000","last_unstale":"2024-05-28T06:20:18.954341+0000","last_undegraded":"2024-05-28T06:20:18.954341+0000","last_fullsized":"2024-05-28T06:20:18.954341+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:06:30.855462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"102'2","reported_seq":14,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.961311+0000","last_change":"2024-05-28T06:20:17.917213+0000","last_active":"2024-05-28T06:20:18.961311+0000","last_peered":"2024-05-28T06:20:18.961311+0000","last_clean":"2024-05-28T06:20:18.961311+0000","last_became_active":"2024-05-28T06:20:17.915714+0000","last_became_peered":"2024-05-28T06:20:17.915714+0000","last_unstale":"2024-05-28T06:20:18.961311+0000","last_undegraded":"2024-05-28T06:20:18.961311+0000","last_fullsized":"2024-05-28T06:20:18.961311+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_clean_scrub_stamp":"2024-05-28T06:20:16.867179+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:33:28.764945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-05-28T06:20:19.925983+0000","last_change":"2024-05-28T06:20:18.944335+0000","last_active":"2024-05-28T06:20:19.925983+0000","last_peered":"2024-05-28T06:20:19.925983+0000","last_clean":"2024-05-28T06:20:19.925983+0000","last_became_active":"2024-05-28T06:20:18.944050+0000","last_became_peered":"2024-05-28T06:20:18.944050+0000","last_unstale":"2024-05-28T06:20:19.925983+0000","last_undegraded":"2024-05-28T06:20:19.925983+0000","last_fullsized":"2024-05-28T06:20:19.925983+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:13:34.791207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"102'2","reported_seq":14,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.962230+0000","last_change":"2024-05-28T06:20:17.922556+0000","last_active":"2024-05-28T06:20:18.962230+0000","last_peered":"2024-05-28T06:20:18.962230+0000","last_clean":"2024-05-28T06:20:18.962230+0000","last_became_active":"2024-05-28T06:20:17.920828+0000","last_became_peered":"2024-05-28T06:20:17.920828+0000","last_unstale":"2024-05-28T06:20:18.962230+0000","last_undegraded":"2024-05-28T06:20:18.962230+0000","last_fullsized":"2024-05-28T06:20:18.962230+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_clean_scrub_stamp":"2024-05-28T06:20:16.867179+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:26:05.221654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.956245+0000","last_change":"2024-05-28T06:20:18.952189+0000","last_active":"2024-05-28T06:20:18.956245+0000","last_peered":"2024-05-28T06:20:18.956245+0000","last_clean":"2024-05-28T06:20:18.956245+0000","last_became_active":"2024-05-28T06:20:18.951183+0000","last_became_peered":"2024-05-28T06:20:18.951183+0000","last_unstale":"2024-05-28T06:20:18.956245+0000","last_undegraded":"2024-05-28T06:20:18.956245+0000","last_fullsized":"2024-05-28T06:20:18.956245+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:29:20.621430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"102'1","reported_seq":13,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.956839+0000","last_change":"2024-05-28T06:20:17.924313+0000","last_active":"2024-05-28T06:20:18.956839+0000","last_peered":"2024-05-28T06:20:18.956839+0000","last_clean":"2024-05-28T06:20:18.956839+0000","last_became_active":"2024-05-28T06:20:17.923726+0000","last_became_peered":"2024-05-28T06:20:17.923726+0000","last_unstale":"2024-05-28T06:20:18.956839+0000","last_undegraded":"2024-05-28T06:20:18.956839+0000","last_fullsized":"2024-05-28T06:20:18.956839+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_clean_scrub_stamp":"2024-05-28T06:20:16.867179+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:52:19.137425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-05-28T06:20:20.415359+0000","last_change":"2024-05-28T06:20:18.945771+0000","last_active":"2024-05-28T06:20:20.415359+0000","last_peered":"2024-05-28T06:20:20.415359+0000","last_clean":"2024-05-28T06:20:20.415359+0000","last_became_active":"2024-05-28T06:20:18.944047+0000","last_became_peered":"2024-05-28T06:20:18.944047+0000","last_unstale":"2024-05-28T06:20:20.415359+0000","last_undegraded":"2024-05-28T06:20:20.415359+0000","last_fullsized":"2024-05-28T06:20:20.415359+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:33:27.455861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"102'1","reported_seq":13,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.960209+0000","last_change":"2024-05-28T06:20:17.913811+0000","last_active":"2024-05-28T06:20:18.960209+0000","last_peered":"2024-05-28T06:20:18.960209+0000","last_clean":"2024-05-28T06:20:18.960209+0000","last_became_active":"2024-05-28T06:20:17.912572+0000","last_became_peered":"2024-05-28T06:20:17.912572+0000","last_unstale":"2024-05-28T06:20:18.960209+0000","last_undegraded":"2024-05-28T06:20:18.960209+0000","last_fullsized":"2024-05-28T06:20:18.960209+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_clean_scrub_stamp":"2024-05-28T06:20:16.867179+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:29:19.608952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.949038+0000","last_change":"2024-05-28T06:20:18.947323+0000","last_active":"2024-05-28T06:20:18.949038+0000","last_peered":"2024-05-28T06:20:18.949038+0000","last_clean":"2024-05-28T06:20:18.949038+0000","last_became_active":"2024-05-28T06:20:18.946955+0000","last_became_peered":"2024-05-28T06:20:18.946955+0000","last_unstale":"2024-05-28T06:20:18.949038+0000","last_undegraded":"2024-05-28T06:20:18.949038+0000","last_fullsized":"2024-05-28T06:20:18.949038+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:52:01.724740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.936794+0000","last_change":"2024-05-28T06:20:17.916892+0000","last_active":"2024-05-28T06:20:18.936794+0000","last_peered":"2024-05-28T06:20:18.936794+0000","last_clean":"2024-05-28T06:20:18.936794+0000","last_became_active":"2024-05-28T06:20:17.916596+0000","last_became_peered":"2024-05-28T06:20:17.916596+0000","last_unstale":"2024-05-28T06:20:18.936794+0000","last_undegraded":"2024-05-28T06:20:18.936794+0000","last_fullsized":"2024-05-28T06:20:18.936794+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_clean_scrub_stamp":"2024-05-28T06:20:16.867179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:50:07.888165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-05-28T06:20:19.925308+0000","last_change":"2024-05-28T06:20:18.948033+0000","last_active":"2024-05-28T06:20:19.925308+0000","last_peered":"2024-05-28T06:20:19.925308+0000","last_clean":"2024-05-28T06:20:19.925308+0000","last_became_active":"2024-05-28T06:20:18.946352+0000","last_became_peered":"2024-05-28T06:20:18.946352+0000","last_unstale":"2024-05-28T06:20:19.925308+0000","last_undegraded":"2024-05-28T06:20:19.925308+0000","last_fullsized":"2024-05-28T06:20:19.925308+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:52:01.724740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.932899+0000","last_change":"2024-05-28T06:20:17.904261+0000","last_active":"2024-05-28T06:20:18.932899+0000","last_peered":"2024-05-28T06:20:18.932899+0000","last_clean":"2024-05-28T06:20:18.932899+0000","last_became_active":"2024-05-28T06:20:17.903943+0000","last_became_peered":"2024-05-28T06:20:17.903943+0000","last_unstale":"2024-05-28T06:20:18.932899+0000","last_undegraded":"2024-05-28T06:20:18.932899+0000","last_fullsized":"2024-05-28T06:20:18.932899+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_clean_scrub_stamp":"2024-05-28T06:20:16.867179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:48:51.382403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.925775+0000","last_change":"2024-05-28T06:20:18.923372+0000","last_active":"2024-05-28T06:20:18.925775+0000","last_peered":"2024-05-28T06:20:18.925775+0000","last_clean":"2024-05-28T06:20:18.925775+0000","last_became_active":"2024-05-28T06:20:18.922207+0000","last_became_peered":"2024-05-28T06:20:18.922207+0000","last_unstale":"2024-05-28T06:20:18.925775+0000","last_undegraded":"2024-05-28T06:20:18.925775+0000","last_fullsized":"2024-05-28T06:20:18.925775+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:53:31.071486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.931612+0000","last_change":"2024-05-28T06:20:17.924331+0000","last_active":"2024-05-28T06:20:18.931612+0000","last_peered":"2024-05-28T06:20:18.931612+0000","last_clean":"2024-05-28T06:20:18.931612+0000","last_became_active":"2024-05-28T06:20:17.924029+0000","last_became_peered":"2024-05-28T06:20:17.924029+0000","last_unstale":"2024-05-28T06:20:18.931612+0000","last_undegraded":"2024-05-28T06:20:18.931612+0000","last_fullsized":"2024-05-28T06:20:18.931612+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_clean_scrub_stamp":"2024-05-28T06:20:16.867179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:49:44.721933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-05-28T06:20:19.926494+0000","last_change":"2024-05-28T06:20:18.945763+0000","last_active":"2024-05-28T06:20:19.926494+0000","last_peered":"2024-05-28T06:20:19.926494+0000","last_clean":"2024-05-28T06:20:19.926494+0000","last_became_active":"2024-05-28T06:20:18.945474+0000","last_became_peered":"2024-05-28T06:20:18.945474+0000","last_unstale":"2024-05-28T06:20:19.926494+0000","last_undegraded":"2024-05-28T06:20:19.926494+0000","last_fullsized":"2024-05-28T06:20:19.926494+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:25:49.657873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.923744+0000","last_change":"2024-05-28T06:20:17.912420+0000","last_active":"2024-05-28T06:20:18.923744+0000","last_peered":"2024-05-28T06:20:18.923744+0000","last_clean":"2024-05-28T06:20:18.923744+0000","last_became_active":"2024-05-28T06:20:17.912084+0000","last_became_peered":"2024-05-28T06:20:17.912084+0000","last_unstale":"2024-05-28T06:20:18.923744+0000","last_undegraded":"2024-05-28T06:20:18.923744+0000","last_fullsized":"2024-05-28T06:20:18.923744+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_clean_scrub_stamp":"2024-05-28T06:20:16.867179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:19:41.740749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.950358+0000","last_change":"2024-05-28T06:20:18.947754+0000","last_active":"2024-05-28T06:20:18.950358+0000","last_peered":"2024-05-28T06:20:18.950358+0000","last_clean":"2024-05-28T06:20:18.950358+0000","last_became_active":"2024-05-28T06:20:18.947477+0000","last_became_peered":"2024-05-28T06:20:18.947477+0000","last_unstale":"2024-05-28T06:20:18.950358+0000","last_undegraded":"2024-05-28T06:20:18.950358+0000","last_fullsized":"2024-05-28T06:20:18.950358+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:53:31.071486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"102'3","reported_seq":17,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-05-28T06:20:20.416206+0000","last_change":"2024-05-28T06:20:17.926192+0000","last_active":"2024-05-28T06:20:20.416206+0000","last_peered":"2024-05-28T06:20:20.416206+0000","last_clean":"2024-05-28T06:20:20.416206+0000","last_became_active":"2024-05-28T06:20:17.925809+0000","last_became_peered":"2024-05-28T06:20:17.925809+0000","last_unstale":"2024-05-28T06:20:20.416206+0000","last_undegraded":"2024-05-28T06:20:20.416206+0000","last_fullsized":"2024-05-28T06:20:20.416206+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_clean_scrub_stamp":"2024-05-28T06:20:16.867179+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:54:17.904965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-05-28T06:20:20.415846+0000","last_change":"2024-05-28T06:20:18.954383+0000","last_active":"2024-05-28T06:20:20.415846+0000","last_peered":"2024-05-28T06:20:20.415846+0000","last_clean":"2024-05-28T06:20:20.415846+0000","last_became_active":"2024-05-28T06:20:18.951938+0000","last_became_peered":"2024-05-28T06:20:18.951938+0000","last_unstale":"2024-05-28T06:20:20.415846+0000","last_undegraded":"2024-05-28T06:20:20.415846+0000","last_fullsized":"2024-05-28T06:20:20.415846+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:28:46.723017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"102'6","reported_seq":20,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-05-28T06:20:19.926561+0000","last_change":"2024-05-28T06:20:17.919818+0000","last_active":"2024-05-28T06:20:19.926561+0000","last_peered":"2024-05-28T06:20:19.926561+0000","last_clean":"2024-05-28T06:20:19.926561+0000","last_became_active":"2024-05-28T06:20:17.919236+0000","last_became_peered":"2024-05-28T06:20:17.919236+0000","last_unstale":"2024-05-28T06:20:19.926561+0000","last_undegraded":"2024-05-28T06:20:19.926561+0000","last_fullsized":"2024-05-28T06:20:19.926561+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_clean_scrub_stamp":"2024-05-28T06:20:16.867179+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:26:05.221654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.937569+0000","last_change":"2024-05-28T06:20:18.932847+0000","last_active":"2024-05-28T06:20:18.937569+0000","last_peered":"2024-05-28T06:20:18.937569+0000","last_clean":"2024-05-28T06:20:18.937569+0000","last_became_active":"2024-05-28T06:20:18.932532+0000","last_became_peered":"2024-05-28T06:20:18.932532+0000","last_unstale":"2024-05-28T06:20:18.937569+0000","last_undegraded":"2024-05-28T06:20:18.937569+0000","last_fullsized":"2024-05-28T06:20:18.937569+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:34:28.359961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"102'2","reported_seq":14,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.962591+0000","last_change":"2024-05-28T06:20:17.921616+0000","last_active":"2024-05-28T06:20:18.962591+0000","last_peered":"2024-05-28T06:20:18.962591+0000","last_clean":"2024-05-28T06:20:18.962591+0000","last_became_active":"2024-05-28T06:20:17.921332+0000","last_became_peered":"2024-05-28T06:20:17.921332+0000","last_unstale":"2024-05-28T06:20:18.962591+0000","last_undegraded":"2024-05-28T06:20:18.962591+0000","last_fullsized":"2024-05-28T06:20:18.962591+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_clean_scrub_stamp":"2024-05-28T06:20:16.867179+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:21:17.875374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.955217+0000","last_change":"2024-05-28T06:20:18.949515+0000","last_active":"2024-05-28T06:20:18.955217+0000","last_peered":"2024-05-28T06:20:18.955217+0000","last_clean":"2024-05-28T06:20:18.955217+0000","last_became_active":"2024-05-28T06:20:18.949214+0000","last_became_peered":"2024-05-28T06:20:18.949214+0000","last_unstale":"2024-05-28T06:20:18.955217+0000","last_undegraded":"2024-05-28T06:20:18.955217+0000","last_fullsized":"2024-05-28T06:20:18.955217+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:24:26.011451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.940635+0000","last_change":"2024-05-28T06:20:17.923138+0000","last_active":"2024-05-28T06:20:18.940635+0000","last_peered":"2024-05-28T06:20:18.940635+0000","last_clean":"2024-05-28T06:20:18.940635+0000","last_became_active":"2024-05-28T06:20:17.922501+0000","last_became_peered":"2024-05-28T06:20:17.922501+0000","last_unstale":"2024-05-28T06:20:18.940635+0000","last_undegraded":"2024-05-28T06:20:18.940635+0000","last_fullsized":"2024-05-28T06:20:18.940635+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_clean_scrub_stamp":"2024-05-28T06:20:16.867179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T18:15:15.367346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-05-28T06:20:19.924857+0000","last_change":"2024-05-28T06:20:18.942969+0000","last_active":"2024-05-28T06:20:19.924857+0000","last_peered":"2024-05-28T06:20:19.924857+0000","last_clean":"2024-05-28T06:20:19.924857+0000","last_became_active":"2024-05-28T06:20:18.939044+0000","last_became_peered":"2024-05-28T06:20:18.939044+0000","last_unstale":"2024-05-28T06:20:19.924857+0000","last_undegraded":"2024-05-28T06:20:19.924857+0000","last_fullsized":"2024-05-28T06:20:19.924857+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:53:31.071486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":14,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-05-28T06:20:19.925345+0000","last_change":"2024-05-28T06:20:17.926049+0000","last_active":"2024-05-28T06:20:19.925345+0000","last_peered":"2024-05-28T06:20:19.925345+0000","last_clean":"2024-05-28T06:20:19.925345+0000","last_became_active":"2024-05-28T06:20:17.923234+0000","last_became_peered":"2024-05-28T06:20:17.923234+0000","last_unstale":"2024-05-28T06:20:19.925345+0000","last_undegraded":"2024-05-28T06:20:19.925345+0000","last_fullsized":"2024-05-28T06:20:19.925345+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_clean_scrub_stamp":"2024-05-28T06:20:16.867179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:04:54.249701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.947481+0000","last_change":"2024-05-28T06:20:18.941003+0000","last_active":"2024-05-28T06:20:18.947481+0000","last_peered":"2024-05-28T06:20:18.947481+0000","last_clean":"2024-05-28T06:20:18.947481+0000","last_became_active":"2024-05-28T06:20:18.940699+0000","last_became_peered":"2024-05-28T06:20:18.940699+0000","last_unstale":"2024-05-28T06:20:18.947481+0000","last_undegraded":"2024-05-28T06:20:18.947481+0000","last_fullsized":"2024-05-28T06:20:18.947481+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:44:20.587585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"102'8","reported_seq":20,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.962777+0000","last_change":"2024-05-28T06:20:17.917887+0000","last_active":"2024-05-28T06:20:18.962777+0000","last_peered":"2024-05-28T06:20:18.962777+0000","last_clean":"2024-05-28T06:20:18.962777+0000","last_became_active":"2024-05-28T06:20:17.917597+0000","last_became_peered":"2024-05-28T06:20:17.917597+0000","last_unstale":"2024-05-28T06:20:18.962777+0000","last_undegraded":"2024-05-28T06:20:18.962777+0000","last_fullsized":"2024-05-28T06:20:18.962777+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_clean_scrub_stamp":"2024-05-28T06:20:16.867179+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:15:03.433459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.950690+0000","last_change":"2024-05-28T06:20:18.944640+0000","last_active":"2024-05-28T06:20:18.950690+0000","last_peered":"2024-05-28T06:20:18.950690+0000","last_clean":"2024-05-28T06:20:18.950690+0000","last_became_active":"2024-05-28T06:20:18.944125+0000","last_became_peered":"2024-05-28T06:20:18.944125+0000","last_unstale":"2024-05-28T06:20:18.950690+0000","last_undegraded":"2024-05-28T06:20:18.950690+0000","last_fullsized":"2024-05-28T06:20:18.950690+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:34:28.359961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"102'2","reported_seq":16,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-05-28T06:20:19.925894+0000","last_change":"2024-05-28T06:20:17.922062+0000","last_active":"2024-05-28T06:20:19.925894+0000","last_peered":"2024-05-28T06:20:19.925894+0000","last_clean":"2024-05-28T06:20:19.925894+0000","last_became_active":"2024-05-28T06:20:17.921772+0000","last_became_peered":"2024-05-28T06:20:17.921772+0000","last_unstale":"2024-05-28T06:20:19.925894+0000","last_undegraded":"2024-05-28T06:20:19.925894+0000","last_fullsized":"2024-05-28T06:20:19.925894+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_clean_scrub_stamp":"2024-05-28T06:20:16.867179+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:29:19.608952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.956278+0000","last_change":"2024-05-28T06:20:18.952158+0000","last_active":"2024-05-28T06:20:18.956278+0000","last_peered":"2024-05-28T06:20:18.956278+0000","last_clean":"2024-05-28T06:20:18.956278+0000","last_became_active":"2024-05-28T06:20:18.949205+0000","last_became_peered":"2024-05-28T06:20:18.949205+0000","last_unstale":"2024-05-28T06:20:18.956278+0000","last_undegraded":"2024-05-28T06:20:18.956278+0000","last_fullsized":"2024-05-28T06:20:18.956278+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:04:55.262180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"102'1","reported_seq":13,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.957102+0000","last_change":"2024-05-28T06:20:17.923109+0000","last_active":"2024-05-28T06:20:18.957102+0000","last_peered":"2024-05-28T06:20:18.957102+0000","last_clean":"2024-05-28T06:20:18.957102+0000","last_became_active":"2024-05-28T06:20:17.921903+0000","last_became_peered":"2024-05-28T06:20:17.921903+0000","last_unstale":"2024-05-28T06:20:18.957102+0000","last_undegraded":"2024-05-28T06:20:18.957102+0000","last_fullsized":"2024-05-28T06:20:18.957102+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_clean_scrub_stamp":"2024-05-28T06:20:16.867179+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:33:28.764945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-05-28T06:20:20.416066+0000","last_change":"2024-05-28T06:20:18.954256+0000","last_active":"2024-05-28T06:20:20.416066+0000","last_peered":"2024-05-28T06:20:20.416066+0000","last_clean":"2024-05-28T06:20:20.416066+0000","last_became_active":"2024-05-28T06:20:18.951535+0000","last_became_peered":"2024-05-28T06:20:18.951535+0000","last_unstale":"2024-05-28T06:20:20.416066+0000","last_undegraded":"2024-05-28T06:20:20.416066+0000","last_fullsized":"2024-05-28T06:20:20.416066+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T18:08:21.610036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.912722+0000","last_change":"2024-05-28T06:20:17.927124+0000","last_active":"2024-05-28T06:20:18.912722+0000","last_peered":"2024-05-28T06:20:18.912722+0000","last_clean":"2024-05-28T06:20:18.912722+0000","last_became_active":"2024-05-28T06:20:17.926859+0000","last_became_peered":"2024-05-28T06:20:17.926859+0000","last_unstale":"2024-05-28T06:20:18.912722+0000","last_undegraded":"2024-05-28T06:20:18.912722+0000","last_fullsized":"2024-05-28T06:20:18.912722+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_clean_scrub_stamp":"2024-05-28T06:20:16.867179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:50:07.888165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.947401+0000","last_change":"2024-05-28T06:20:18.945503+0000","last_active":"2024-05-28T06:20:18.947401+0000","last_peered":"2024-05-28T06:20:18.947401+0000","last_clean":"2024-05-28T06:20:18.947401+0000","last_became_active":"2024-05-28T06:20:18.945195+0000","last_became_peered":"2024-05-28T06:20:18.945195+0000","last_unstale":"2024-05-28T06:20:18.947401+0000","last_undegraded":"2024-05-28T06:20:18.947401+0000","last_fullsized":"2024-05-28T06:20:18.947401+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:36:31.817085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"102'2","reported_seq":14,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.961406+0000","last_change":"2024-05-28T06:20:17.917817+0000","last_active":"2024-05-28T06:20:18.961406+0000","last_peered":"2024-05-28T06:20:18.961406+0000","last_clean":"2024-05-28T06:20:18.961406+0000","last_became_active":"2024-05-28T06:20:17.916979+0000","last_became_peered":"2024-05-28T06:20:17.916979+0000","last_unstale":"2024-05-28T06:20:18.961406+0000","last_undegraded":"2024-05-28T06:20:18.961406+0000","last_fullsized":"2024-05-28T06:20:18.961406+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_clean_scrub_stamp":"2024-05-28T06:20:16.867179+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:41:42.790025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.947526+0000","last_change":"2024-05-28T06:20:18.944664+0000","last_active":"2024-05-28T06:20:18.947526+0000","last_peered":"2024-05-28T06:20:18.947526+0000","last_clean":"2024-05-28T06:20:18.947526+0000","last_became_active":"2024-05-28T06:20:18.941309+0000","last_became_peered":"2024-05-28T06:20:18.941309+0000","last_unstale":"2024-05-28T06:20:18.947526+0000","last_undegraded":"2024-05-28T06:20:18.947526+0000","last_fullsized":"2024-05-28T06:20:18.947526+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:23:37.846492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.942965+0000","last_change":"2024-05-28T06:20:17.925585+0000","last_active":"2024-05-28T06:20:18.942965+0000","last_peered":"2024-05-28T06:20:18.942965+0000","last_clean":"2024-05-28T06:20:18.942965+0000","last_became_active":"2024-05-28T06:20:17.925283+0000","last_became_peered":"2024-05-28T06:20:17.925283+0000","last_unstale":"2024-05-28T06:20:18.942965+0000","last_undegraded":"2024-05-28T06:20:18.942965+0000","last_fullsized":"2024-05-28T06:20:18.942965+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_clean_scrub_stamp":"2024-05-28T06:20:16.867179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:59:51.754198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.947484+0000","last_change":"2024-05-28T06:20:18.943551+0000","last_active":"2024-05-28T06:20:18.947484+0000","last_peered":"2024-05-28T06:20:18.947484+0000","last_clean":"2024-05-28T06:20:18.947484+0000","last_became_active":"2024-05-28T06:20:18.943259+0000","last_became_peered":"2024-05-28T06:20:18.943259+0000","last_unstale":"2024-05-28T06:20:18.947484+0000","last_undegraded":"2024-05-28T06:20:18.947484+0000","last_fullsized":"2024-05-28T06:20:18.947484+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:19:45.078600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.942761+0000","last_change":"2024-05-28T06:20:17.918736+0000","last_active":"2024-05-28T06:20:18.942761+0000","last_peered":"2024-05-28T06:20:18.942761+0000","last_clean":"2024-05-28T06:20:18.942761+0000","last_became_active":"2024-05-28T06:20:17.918447+0000","last_became_peered":"2024-05-28T06:20:17.918447+0000","last_unstale":"2024-05-28T06:20:18.942761+0000","last_undegraded":"2024-05-28T06:20:18.942761+0000","last_fullsized":"2024-05-28T06:20:18.942761+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_clean_scrub_stamp":"2024-05-28T06:20:16.867179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:48:48.325930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.951463+0000","last_change":"2024-05-28T06:20:18.950512+0000","last_active":"2024-05-28T06:20:18.951463+0000","last_peered":"2024-05-28T06:20:18.951463+0000","last_clean":"2024-05-28T06:20:18.951463+0000","last_became_active":"2024-05-28T06:20:18.950183+0000","last_became_peered":"2024-05-28T06:20:18.950183+0000","last_unstale":"2024-05-28T06:20:18.951463+0000","last_undegraded":"2024-05-28T06:20:18.951463+0000","last_fullsized":"2024-05-28T06:20:18.951463+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:06:21.231285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.912441+0000","last_change":"2024-05-28T06:20:17.910325+0000","last_active":"2024-05-28T06:20:18.912441+0000","last_peered":"2024-05-28T06:20:18.912441+0000","last_clean":"2024-05-28T06:20:18.912441+0000","last_became_active":"2024-05-28T06:20:17.910055+0000","last_became_peered":"2024-05-28T06:20:17.910055+0000","last_unstale":"2024-05-28T06:20:18.912441+0000","last_undegraded":"2024-05-28T06:20:18.912441+0000","last_fullsized":"2024-05-28T06:20:18.912441+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_clean_scrub_stamp":"2024-05-28T06:20:16.867179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:24:06.175231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.946687+0000","last_change":"2024-05-28T06:20:18.939534+0000","last_active":"2024-05-28T06:20:18.946687+0000","last_peered":"2024-05-28T06:20:18.946687+0000","last_clean":"2024-05-28T06:20:18.946687+0000","last_became_active":"2024-05-28T06:20:18.939173+0000","last_became_peered":"2024-05-28T06:20:18.939173+0000","last_unstale":"2024-05-28T06:20:18.946687+0000","last_undegraded":"2024-05-28T06:20:18.946687+0000","last_fullsized":"2024-05-28T06:20:18.946687+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:13:51.233304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"102'2","reported_seq":14,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.958192+0000","last_change":"2024-05-28T06:20:17.927698+0000","last_active":"2024-05-28T06:20:18.958192+0000","last_peered":"2024-05-28T06:20:18.958192+0000","last_clean":"2024-05-28T06:20:18.958192+0000","last_became_active":"2024-05-28T06:20:17.927383+0000","last_became_peered":"2024-05-28T06:20:17.927383+0000","last_unstale":"2024-05-28T06:20:18.958192+0000","last_undegraded":"2024-05-28T06:20:18.958192+0000","last_fullsized":"2024-05-28T06:20:18.958192+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_clean_scrub_stamp":"2024-05-28T06:20:16.867179+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:54:02.951428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.955355+0000","last_change":"2024-05-28T06:20:18.950303+0000","last_active":"2024-05-28T06:20:18.955355+0000","last_peered":"2024-05-28T06:20:18.955355+0000","last_clean":"2024-05-28T06:20:18.955355+0000","last_became_active":"2024-05-28T06:20:18.949736+0000","last_became_peered":"2024-05-28T06:20:18.949736+0000","last_unstale":"2024-05-28T06:20:18.955355+0000","last_undegraded":"2024-05-28T06:20:18.955355+0000","last_fullsized":"2024-05-28T06:20:18.955355+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:32:52.994598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.915478+0000","last_change":"2024-05-28T06:20:17.917237+0000","last_active":"2024-05-28T06:20:18.915478+0000","last_peered":"2024-05-28T06:20:18.915478+0000","last_clean":"2024-05-28T06:20:18.915478+0000","last_became_active":"2024-05-28T06:20:17.915703+0000","last_became_peered":"2024-05-28T06:20:17.915703+0000","last_unstale":"2024-05-28T06:20:18.915478+0000","last_undegraded":"2024-05-28T06:20:18.915478+0000","last_fullsized":"2024-05-28T06:20:18.915478+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_clean_scrub_stamp":"2024-05-28T06:20:16.867179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:54:17.904965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.956040+0000","last_change":"2024-05-28T06:20:18.954100+0000","last_active":"2024-05-28T06:20:18.956040+0000","last_peered":"2024-05-28T06:20:18.956040+0000","last_clean":"2024-05-28T06:20:18.956040+0000","last_became_active":"2024-05-28T06:20:18.953338+0000","last_became_peered":"2024-05-28T06:20:18.953338+0000","last_unstale":"2024-05-28T06:20:18.956040+0000","last_undegraded":"2024-05-28T06:20:18.956040+0000","last_fullsized":"2024-05-28T06:20:18.956040+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:39:20.591183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.912813+0000","last_change":"2024-05-28T06:20:17.928473+0000","last_active":"2024-05-28T06:20:18.912813+0000","last_peered":"2024-05-28T06:20:18.912813+0000","last_clean":"2024-05-28T06:20:18.912813+0000","last_became_active":"2024-05-28T06:20:17.928180+0000","last_became_peered":"2024-05-28T06:20:17.928180+0000","last_unstale":"2024-05-28T06:20:18.912813+0000","last_undegraded":"2024-05-28T06:20:18.912813+0000","last_fullsized":"2024-05-28T06:20:18.912813+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_clean_scrub_stamp":"2024-05-28T06:20:16.867179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:19:41.740749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.948756+0000","last_change":"2024-05-28T06:20:18.943224+0000","last_active":"2024-05-28T06:20:18.948756+0000","last_peered":"2024-05-28T06:20:18.948756+0000","last_clean":"2024-05-28T06:20:18.948756+0000","last_became_active":"2024-05-28T06:20:18.941234+0000","last_became_peered":"2024-05-28T06:20:18.941234+0000","last_unstale":"2024-05-28T06:20:18.948756+0000","last_undegraded":"2024-05-28T06:20:18.948756+0000","last_fullsized":"2024-05-28T06:20:18.948756+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:41:43.802503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.935946+0000","last_change":"2024-05-28T06:20:17.923709+0000","last_active":"2024-05-28T06:20:18.935946+0000","last_peered":"2024-05-28T06:20:18.935946+0000","last_clean":"2024-05-28T06:20:18.935946+0000","last_became_active":"2024-05-28T06:20:17.922798+0000","last_became_peered":"2024-05-28T06:20:17.922798+0000","last_unstale":"2024-05-28T06:20:18.935946+0000","last_undegraded":"2024-05-28T06:20:18.935946+0000","last_fullsized":"2024-05-28T06:20:18.935946+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_clean_scrub_stamp":"2024-05-28T06:20:16.867179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:24:06.175231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.947210+0000","last_change":"2024-05-28T06:20:18.939976+0000","last_active":"2024-05-28T06:20:18.947210+0000","last_peered":"2024-05-28T06:20:18.947210+0000","last_clean":"2024-05-28T06:20:18.947210+0000","last_became_active":"2024-05-28T06:20:18.939633+0000","last_became_peered":"2024-05-28T06:20:18.939633+0000","last_unstale":"2024-05-28T06:20:18.947210+0000","last_undegraded":"2024-05-28T06:20:18.947210+0000","last_fullsized":"2024-05-28T06:20:18.947210+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:40:34.672443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.913651+0000","last_change":"2024-05-28T06:20:17.917929+0000","last_active":"2024-05-28T06:20:18.913651+0000","last_peered":"2024-05-28T06:20:18.913651+0000","last_clean":"2024-05-28T06:20:18.913651+0000","last_became_active":"2024-05-28T06:20:17.916446+0000","last_became_peered":"2024-05-28T06:20:17.916446+0000","last_unstale":"2024-05-28T06:20:18.913651+0000","last_undegraded":"2024-05-28T06:20:18.913651+0000","last_fullsized":"2024-05-28T06:20:18.913651+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_clean_scrub_stamp":"2024-05-28T06:20:16.867179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:34:27.347482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-05-28T06:20:19.925542+0000","last_change":"2024-05-28T06:20:18.941825+0000","last_active":"2024-05-28T06:20:19.925542+0000","last_peered":"2024-05-28T06:20:19.925542+0000","last_clean":"2024-05-28T06:20:19.925542+0000","last_became_active":"2024-05-28T06:20:18.941470+0000","last_became_peered":"2024-05-28T06:20:18.941470+0000","last_unstale":"2024-05-28T06:20:19.925542+0000","last_undegraded":"2024-05-28T06:20:19.925542+0000","last_fullsized":"2024-05-28T06:20:19.925542+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:06:21.231285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.929972+0000","last_change":"2024-05-28T06:20:17.918870+0000","last_active":"2024-05-28T06:20:18.929972+0000","last_peered":"2024-05-28T06:20:18.929972+0000","last_clean":"2024-05-28T06:20:18.929972+0000","last_became_active":"2024-05-28T06:20:17.918328+0000","last_became_peered":"2024-05-28T06:20:17.918328+0000","last_unstale":"2024-05-28T06:20:18.929972+0000","last_undegraded":"2024-05-28T06:20:18.929972+0000","last_fullsized":"2024-05-28T06:20:18.929972+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_clean_scrub_stamp":"2024-05-28T06:20:16.867179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:26:05.221654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.946287+0000","last_change":"2024-05-28T06:20:18.939717+0000","last_active":"2024-05-28T06:20:18.946287+0000","last_peered":"2024-05-28T06:20:18.946287+0000","last_clean":"2024-05-28T06:20:18.946287+0000","last_became_active":"2024-05-28T06:20:18.939426+0000","last_became_peered":"2024-05-28T06:20:18.939426+0000","last_unstale":"2024-05-28T06:20:18.946287+0000","last_undegraded":"2024-05-28T06:20:18.946287+0000","last_fullsized":"2024-05-28T06:20:18.946287+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:43:07.928152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.924112+0000","last_change":"2024-05-28T06:20:17.906604+0000","last_active":"2024-05-28T06:20:18.924112+0000","last_peered":"2024-05-28T06:20:18.924112+0000","last_clean":"2024-05-28T06:20:18.924112+0000","last_became_active":"2024-05-28T06:20:17.906310+0000","last_became_peered":"2024-05-28T06:20:17.906310+0000","last_unstale":"2024-05-28T06:20:18.924112+0000","last_undegraded":"2024-05-28T06:20:18.924112+0000","last_fullsized":"2024-05-28T06:20:18.924112+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_clean_scrub_stamp":"2024-05-28T06:20:16.867179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:41:42.790025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.949799+0000","last_change":"2024-05-28T06:20:18.948057+0000","last_active":"2024-05-28T06:20:18.949799+0000","last_peered":"2024-05-28T06:20:18.949799+0000","last_clean":"2024-05-28T06:20:18.949799+0000","last_became_active":"2024-05-28T06:20:18.946998+0000","last_became_peered":"2024-05-28T06:20:18.946998+0000","last_unstale":"2024-05-28T06:20:18.949799+0000","last_undegraded":"2024-05-28T06:20:18.949799+0000","last_fullsized":"2024-05-28T06:20:18.949799+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:52:01.724740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.940414+0000","last_change":"2024-05-28T06:20:17.916573+0000","last_active":"2024-05-28T06:20:18.940414+0000","last_peered":"2024-05-28T06:20:18.940414+0000","last_clean":"2024-05-28T06:20:18.940414+0000","last_became_active":"2024-05-28T06:20:17.916172+0000","last_became_peered":"2024-05-28T06:20:17.916172+0000","last_unstale":"2024-05-28T06:20:18.940414+0000","last_undegraded":"2024-05-28T06:20:18.940414+0000","last_fullsized":"2024-05-28T06:20:18.940414+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_clean_scrub_stamp":"2024-05-28T06:20:16.867179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:38:12.023168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.947418+0000","last_change":"2024-05-28T06:20:18.945549+0000","last_active":"2024-05-28T06:20:18.947418+0000","last_peered":"2024-05-28T06:20:18.947418+0000","last_clean":"2024-05-28T06:20:18.947418+0000","last_became_active":"2024-05-28T06:20:18.945251+0000","last_became_peered":"2024-05-28T06:20:18.945251+0000","last_unstale":"2024-05-28T06:20:18.947418+0000","last_undegraded":"2024-05-28T06:20:18.947418+0000","last_fullsized":"2024-05-28T06:20:18.947418+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:51:04.199722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"102'1","reported_seq":15,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-05-28T06:20:20.415660+0000","last_change":"2024-05-28T06:20:17.913519+0000","last_active":"2024-05-28T06:20:20.415660+0000","last_peered":"2024-05-28T06:20:20.415660+0000","last_clean":"2024-05-28T06:20:20.415660+0000","last_became_active":"2024-05-28T06:20:17.910345+0000","last_became_peered":"2024-05-28T06:20:17.910345+0000","last_unstale":"2024-05-28T06:20:20.415660+0000","last_undegraded":"2024-05-28T06:20:20.415660+0000","last_fullsized":"2024-05-28T06:20:20.415660+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_clean_scrub_stamp":"2024-05-28T06:20:16.867179+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:34:27.347482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":332,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-05-28T06:20:20.415582+0000","last_change":"2024-05-28T06:04:01.039129+0000","last_active":"2024-05-28T06:20:20.415582+0000","last_peered":"2024-05-28T06:20:20.415582+0000","last_clean":"2024-05-28T06:20:20.415582+0000","last_became_active":"2024-05-28T06:04:01.033906+0000","last_became_peered":"2024-05-28T06:04:01.033906+0000","last_unstale":"2024-05-28T06:20:20.415582+0000","last_undegraded":"2024-05-28T06:20:20.415582+0000","last_fullsized":"2024-05-28T06:20:20.415582+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:03:59.864237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:03:59.864237+0000","last_clean_scrub_stamp":"2024-05-28T06:03:59.864237+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:17:36.440245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575047,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":127356,"kb_used_data":1832,"kb_used_omap":1,"kb_used_meta":125502,"kb_avail":94244484,"statfs":{"total":96636764160,"available":96506351616,"internally_reserved":0,"allocated":1875968,"data_stored":1146383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":128514506},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575049,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":126760,"kb_used_data":1240,"kb_used_omap":1,"kb_used_meta":125502,"kb_avail":94245080,"statfs":{"total":96636764160,"available":96506961920,"internally_reserved":0,"allocated":1269760,"data_stored":550199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":128514502},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575047,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76768,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94295072,"statfs":{"total":96636764160,"available":96558153728,"internally_reserved":0,"allocated":1265664,"data_stored":549617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575047,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":86520,"kb_used_data":1252,"kb_used_omap":1,"kb_used_meta":85246,"kb_avail":94285320,"statfs":{"total":96636764160,"available":96548167680,"internally_reserved":0,"allocated":1282048,"data_stored":551283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87292357},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575049,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":126764,"kb_used_data":1248,"kb_used_omap":1,"kb_used_meta":125502,"kb_avail":94245076,"statfs":{"total":96636764160,"available":96506957824,"internally_reserved":0,"allocated":1277952,"data_stored":551181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":128514503},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575048,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":144804,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":143550,"kb_avail":94227036,"statfs":{"total":96636764160,"available":96488484864,"internally_reserved":0,"allocated":1265664,"data_stored":549581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":146995653},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575048,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":126752,"kb_used_data":1232,"kb_used_omap":1,"kb_used_meta":125502,"kb_avail":94245088,"statfs":{"total":96636764160,"available":96506970112,"internally_reserved":0,"allocated":1261568,"data_stored":549547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":128514501},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575047,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64868,"kb_used_data":1816,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94306972,"statfs":{"total":96636764160,"available":96570339328,"internally_reserved":0,"allocated":1859584,"data_stored":1140497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-28T06:20:23.285 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-28T06:20:23.524 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.527+0000 7f6144000640 1 Processor -- start 2024-05-28T06:20:23.524 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.528+0000 7f6144000640 1 -- start start 2024-05-28T06:20:23.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.529+0000 7f6144000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f613c0b4b60 0x7f613c0cfaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:23.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.529+0000 7f6144000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f613c0d04a0 0x7f613c0d0930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:23.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.529+0000 7f6144000640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f613c0ced40 con 0x7f613c0b3460 2024-05-28T06:20:23.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.529+0000 7f6144000640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f613c0b3f90 con 0x7f613c0d04a0 2024-05-28T06:20:23.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.529+0000 7f6144000640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f613c15fbb0 con 0x7f613c0b4b60 2024-05-28T06:20:23.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.529+0000 7f6140c00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f613c0d04a0 0x7f613c0d0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:23.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.529+0000 7f6140c00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f613c0d04a0 0x7f613c0d0930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58798/0 (socket says 172.21.15.96:58798) 2024-05-28T06:20:23.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.529+0000 7f6140c00640 1 -- 172.21.15.96:0/3165886205 learned_addr learned my addr 172.21.15.96:0/3165886205 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:20:23.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.530+0000 7f6141600640 1 --2- 172.21.15.96:0/3165886205 >> v2:172.21.15.123:3300/0 conn(0x7f613c0b4b60 0x7f613c0cfaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:23.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.530+0000 7f6140c00640 1 -- 172.21.15.96:0/3165886205 >> v2:172.21.15.123:3300/0 conn(0x7f613c0b4b60 msgr2=0x7f613c0cfaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:23.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.530+0000 7f6140c00640 1 --2- 172.21.15.96:0/3165886205 >> v2:172.21.15.123:3300/0 conn(0x7f613c0b4b60 0x7f613c0cfaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:23.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.530+0000 7f6140c00640 1 -- 172.21.15.96:0/3165886205 >> v1:172.21.15.123:6789/0 conn(0x7f613c0b3460 legacy=0x7f613c0b3880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:23.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.530+0000 7f6140c00640 1 -- 172.21.15.96:0/3165886205 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f613c0cefc0 con 0x7f613c0d04a0 2024-05-28T06:20:23.528 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.531+0000 7f6140c00640 1 --2- 172.21.15.96:0/3165886205 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f613c0d04a0 0x7f613c0d0930 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f613c0cdd80 tx=0x7f612802a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0ab51847c3fc455 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:23.528 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.531+0000 7f613be00640 1 -- 172.21.15.96:0/3165886205 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6128037070 con 0x7f613c0d04a0 2024-05-28T06:20:23.528 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.531+0000 7f613be00640 1 -- 172.21.15.96:0/3165886205 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6128032440 con 0x7f613c0d04a0 2024-05-28T06:20:23.528 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.531+0000 7f613be00640 1 -- 172.21.15.96:0/3165886205 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6128032810 con 0x7f613c0d04a0 2024-05-28T06:20:23.530 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.532+0000 7f6144000640 1 -- 172.21.15.96:0/3165886205 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f613c0d04a0 msgr2=0x7f613c0d0930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:23.530 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.532+0000 7f6144000640 1 --2- 172.21.15.96:0/3165886205 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f613c0d04a0 0x7f613c0d0930 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f613c0cdd80 tx=0x7f612802a230 comp rx=0 tx=0).stop 2024-05-28T06:20:23.530 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.532+0000 7f6144000640 1 -- 172.21.15.96:0/3165886205 shutdown_connections 2024-05-28T06:20:23.530 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.532+0000 7f6144000640 1 --2- 172.21.15.96:0/3165886205 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f613c0d04a0 0x7f613c0d0930 unknown :-1 s=CLOSED pgs=1628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:23.530 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.532+0000 7f6144000640 1 --2- 172.21.15.96:0/3165886205 >> v2:172.21.15.123:3300/0 conn(0x7f613c0b4b60 0x7f613c0cfaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:23.530 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.532+0000 7f6144000640 1 -- 172.21.15.96:0/3165886205 >> 172.21.15.96:0/3165886205 conn(0x7f613c06bae0 msgr2=0x7f613c06bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:23.530 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.532+0000 7f6144000640 1 -- 172.21.15.96:0/3165886205 shutdown_connections 2024-05-28T06:20:23.530 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.532+0000 7f6144000640 1 -- 172.21.15.96:0/3165886205 wait complete. 2024-05-28T06:20:23.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.534+0000 7f6144000640 1 Processor -- start 2024-05-28T06:20:23.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.534+0000 7f6144000640 1 -- start start 2024-05-28T06:20:23.533 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.535+0000 7f6144000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f613c0b3460 0x7f613c09c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:23.533 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.535+0000 7f6144000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f613c0b4b60 0x7f613c09a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:23.533 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.535+0000 7f6144000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f613c0d04a0 0x7f613c09a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:23.533 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.535+0000 7f6144000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f613c0cffe0 con 0x7f613c0b4b60 2024-05-28T06:20:23.533 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.535+0000 7f6144000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f613c0cf580 con 0x7f613c0b3460 2024-05-28T06:20:23.533 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.535+0000 7f6144000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f613c15ede0 con 0x7f613c0d04a0 2024-05-28T06:20:23.533 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.536+0000 7f6140c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f613c0b4b60 0x7f613c09a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:23.533 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.536+0000 7f6140c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f613c0b4b60 0x7f613c09a400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58800/0 (socket says 172.21.15.96:58800) 2024-05-28T06:20:23.533 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.536+0000 7f6140c00640 1 -- 172.21.15.96:0/1969228422 learned_addr learned my addr 172.21.15.96:0/1969228422 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:20:23.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.536+0000 7f6141600640 1 --2- 172.21.15.96:0/1969228422 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f613c0b3460 0x7f613c09c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:23.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.536+0000 7f6142000640 1 --2- 172.21.15.96:0/1969228422 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f613c0d04a0 0x7f613c09a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:23.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.536+0000 7f6140c00640 1 -- 172.21.15.96:0/1969228422 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f613c0d04a0 msgr2=0x7f613c09a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:23.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.536+0000 7f6140c00640 1 --2- 172.21.15.96:0/1969228422 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f613c0d04a0 0x7f613c09a940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:23.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.536+0000 7f6140c00640 1 -- 172.21.15.96:0/1969228422 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f613c0b3460 msgr2=0x7f613c09c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:23.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.536+0000 7f6140c00640 1 --2- 172.21.15.96:0/1969228422 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f613c0b3460 0x7f613c09c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:23.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.536+0000 7f6140c00640 1 -- 172.21.15.96:0/1969228422 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f613c09ae80 con 0x7f613c0b4b60 2024-05-28T06:20:23.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.536+0000 7f6141600640 1 --2- 172.21.15.96:0/1969228422 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f613c0b3460 0x7f613c09c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:20:23.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.536+0000 7f6140c00640 1 --2- 172.21.15.96:0/1969228422 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f613c0b4b60 0x7f613c09a400 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7f612802a270 tx=0x7f61280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:23.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.537+0000 7f6139600640 1 -- 172.21.15.96:0/1969228422 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6128041020 con 0x7f613c0b4b60 2024-05-28T06:20:23.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.537+0000 7f6139600640 1 -- 172.21.15.96:0/1969228422 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6128030990 con 0x7f613c0b4b60 2024-05-28T06:20:23.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.537+0000 7f6144000640 1 -- 172.21.15.96:0/1969228422 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f613c09b150 con 0x7f613c0b4b60 2024-05-28T06:20:23.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.537+0000 7f6139600640 1 -- 172.21.15.96:0/1969228422 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6128037040 con 0x7f613c0b4b60 2024-05-28T06:20:23.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.537+0000 7f6144000640 1 -- 172.21.15.96:0/1969228422 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f613c09b710 con 0x7f613c0b4b60 2024-05-28T06:20:23.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.538+0000 7f6144000640 1 -- 172.21.15.96:0/1969228422 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f613c0689e0 con 0x7f613c0b4b60 2024-05-28T06:20:23.536 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.539+0000 7f6139600640 1 -- 172.21.15.96:0/1969228422 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 34) ==== 92927+0+0 (secure 0 0 0) 0x7f6128044020 con 0x7f613c0b4b60 2024-05-28T06:20:23.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.540+0000 7f6139600640 1 --2- 172.21.15.96:0/1969228422 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f611c06ddc0 0x7f611c070280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:23.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.540+0000 7f6139600640 1 -- 172.21.15.96:0/1969228422 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 5765+0+0 (secure 0 0 0) 0x7f61280ab290 con 0x7f613c0b4b60 2024-05-28T06:20:23.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.541+0000 7f6141600640 1 --2- 172.21.15.96:0/1969228422 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f611c06ddc0 0x7f611c070280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:23.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.542+0000 7f6141600640 1 --2- 172.21.15.96:0/1969228422 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f611c06ddc0 0x7f611c070280 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f6130008890 tx=0x7f6130008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:23.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.544+0000 7f6139600640 1 -- 172.21.15.96:0/1969228422 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f612804b030 con 0x7f613c0b4b60 2024-05-28T06:20:23.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.739+0000 7f6144000640 1 -- 172.21.15.96:0/1969228422 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f613c0b12a0 con 0x7f611c06ddc0 2024-05-28T06:20:23.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.759+0000 7f6139600640 1 -- 172.21.15.96:0/1969228422 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252451 (secure 0 0 0) 0x7f613c0b12a0 con 0x7f611c06ddc0 2024-05-28T06:20:23.759 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:20:23.760 INFO:teuthology.orchestra.run.smithi096.stderr:dumped all 2024-05-28T06:20:23.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.764+0000 7f6144000640 1 -- 172.21.15.96:0/1969228422 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f611c06ddc0 msgr2=0x7f611c070280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:23.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.764+0000 7f6144000640 1 --2- 172.21.15.96:0/1969228422 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f611c06ddc0 0x7f611c070280 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f6130008890 tx=0x7f6130008040 comp rx=0 tx=0).stop 2024-05-28T06:20:23.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.764+0000 7f6144000640 1 -- 172.21.15.96:0/1969228422 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f613c0b4b60 msgr2=0x7f613c09a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:23.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.764+0000 7f6144000640 1 --2- 172.21.15.96:0/1969228422 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f613c0b4b60 0x7f613c09a400 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7f612802a270 tx=0x7f61280046c0 comp rx=0 tx=0).stop 2024-05-28T06:20:23.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.764+0000 7f6144000640 1 -- 172.21.15.96:0/1969228422 shutdown_connections 2024-05-28T06:20:23.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.764+0000 7f6144000640 1 --2- 172.21.15.96:0/1969228422 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f611c06ddc0 0x7f611c070280 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:23.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.764+0000 7f6144000640 1 --2- 172.21.15.96:0/1969228422 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f613c0d04a0 0x7f613c09a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:23.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.764+0000 7f6144000640 1 --2- 172.21.15.96:0/1969228422 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f613c0b4b60 0x7f613c09a400 unknown :-1 s=CLOSED pgs=1629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:23.763 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.764+0000 7f6144000640 1 --2- 172.21.15.96:0/1969228422 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f613c0b3460 0x7f613c09c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:23.763 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.764+0000 7f6144000640 1 -- 172.21.15.96:0/1969228422 >> 172.21.15.96:0/1969228422 conn(0x7f613c06bae0 msgr2=0x7f613c074a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:23.763 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.764+0000 7f6144000640 1 -- 172.21.15.96:0/1969228422 shutdown_connections 2024-05-28T06:20:23.763 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:23.764+0000 7f6144000640 1 -- 172.21.15.96:0/1969228422 wait complete. 2024-05-28T06:20:23.782 INFO:teuthology.orchestra.run.smithi096.stdout:{"pg_ready":true,"pg_map":{"version":587,"stamp":"2024-05-28T06:20:21.915176+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":136,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":880592,"kb_used_data":11092,"kb_used_omap":12,"kb_used_meta":869363,"kb_avail":754094128,"statfs":{"total":773094113280,"available":772192387072,"internally_reserved":0,"allocated":11358208,"data_stored":5588288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":890228272},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.032546"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.954908+0000","last_change":"2024-05-28T06:20:18.944123+0000","last_active":"2024-05-28T06:20:18.954908+0000","last_peered":"2024-05-28T06:20:18.954908+0000","last_clean":"2024-05-28T06:20:18.954908+0000","last_became_active":"2024-05-28T06:20:18.942920+0000","last_became_peered":"2024-05-28T06:20:18.942920+0000","last_unstale":"2024-05-28T06:20:18.954908+0000","last_undegraded":"2024-05-28T06:20:18.954908+0000","last_fullsized":"2024-05-28T06:20:18.954908+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:37:54.494431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.947560+0000","last_change":"2024-05-28T06:20:18.940202+0000","last_active":"2024-05-28T06:20:18.947560+0000","last_peered":"2024-05-28T06:20:18.947560+0000","last_clean":"2024-05-28T06:20:18.947560+0000","last_became_active":"2024-05-28T06:20:18.939917+0000","last_became_peered":"2024-05-28T06:20:18.939917+0000","last_unstale":"2024-05-28T06:20:18.947560+0000","last_undegraded":"2024-05-28T06:20:18.947560+0000","last_fullsized":"2024-05-28T06:20:18.947560+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:49:52.448395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.955120+0000","last_change":"2024-05-28T06:20:18.944516+0000","last_active":"2024-05-28T06:20:18.955120+0000","last_peered":"2024-05-28T06:20:18.955120+0000","last_clean":"2024-05-28T06:20:18.955120+0000","last_became_active":"2024-05-28T06:20:18.943234+0000","last_became_peered":"2024-05-28T06:20:18.943234+0000","last_unstale":"2024-05-28T06:20:18.955120+0000","last_undegraded":"2024-05-28T06:20:18.955120+0000","last_fullsized":"2024-05-28T06:20:18.955120+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:04:37.338955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.948997+0000","last_change":"2024-05-28T06:20:18.945783+0000","last_active":"2024-05-28T06:20:18.948997+0000","last_peered":"2024-05-28T06:20:18.948997+0000","last_clean":"2024-05-28T06:20:18.948997+0000","last_became_active":"2024-05-28T06:20:18.945479+0000","last_became_peered":"2024-05-28T06:20:18.945479+0000","last_unstale":"2024-05-28T06:20:18.948997+0000","last_undegraded":"2024-05-28T06:20:18.948997+0000","last_fullsized":"2024-05-28T06:20:18.948997+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:53:31.071486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.948862+0000","last_change":"2024-05-28T06:20:18.941714+0000","last_active":"2024-05-28T06:20:18.948862+0000","last_peered":"2024-05-28T06:20:18.948862+0000","last_clean":"2024-05-28T06:20:18.948862+0000","last_became_active":"2024-05-28T06:20:18.941310+0000","last_became_peered":"2024-05-28T06:20:18.941310+0000","last_unstale":"2024-05-28T06:20:18.948862+0000","last_undegraded":"2024-05-28T06:20:18.948862+0000","last_fullsized":"2024-05-28T06:20:18.948862+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:19:42.753227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-05-28T06:20:20.415278+0000","last_change":"2024-05-28T06:20:18.944910+0000","last_active":"2024-05-28T06:20:20.415278+0000","last_peered":"2024-05-28T06:20:20.415278+0000","last_clean":"2024-05-28T06:20:20.415278+0000","last_became_active":"2024-05-28T06:20:18.944631+0000","last_became_peered":"2024-05-28T06:20:18.944631+0000","last_unstale":"2024-05-28T06:20:20.415278+0000","last_undegraded":"2024-05-28T06:20:20.415278+0000","last_fullsized":"2024-05-28T06:20:20.415278+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:55:45.904700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-05-28T06:20:20.415280+0000","last_change":"2024-05-28T06:20:18.945364+0000","last_active":"2024-05-28T06:20:20.415280+0000","last_peered":"2024-05-28T06:20:20.415280+0000","last_clean":"2024-05-28T06:20:20.415280+0000","last_became_active":"2024-05-28T06:20:18.944879+0000","last_became_peered":"2024-05-28T06:20:18.944879+0000","last_unstale":"2024-05-28T06:20:20.415280+0000","last_undegraded":"2024-05-28T06:20:20.415280+0000","last_fullsized":"2024-05-28T06:20:20.415280+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:20:02.787394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.934711+0000","last_change":"2024-05-28T06:20:18.932617+0000","last_active":"2024-05-28T06:20:18.934711+0000","last_peered":"2024-05-28T06:20:18.934711+0000","last_clean":"2024-05-28T06:20:18.934711+0000","last_became_active":"2024-05-28T06:20:18.932263+0000","last_became_peered":"2024-05-28T06:20:18.932263+0000","last_unstale":"2024-05-28T06:20:18.934711+0000","last_undegraded":"2024-05-28T06:20:18.934711+0000","last_fullsized":"2024-05-28T06:20:18.934711+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:52:01.724740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-05-28T06:20:19.926369+0000","last_change":"2024-05-28T06:20:18.940604+0000","last_active":"2024-05-28T06:20:19.926369+0000","last_peered":"2024-05-28T06:20:19.926369+0000","last_clean":"2024-05-28T06:20:19.926369+0000","last_became_active":"2024-05-28T06:20:18.940288+0000","last_became_peered":"2024-05-28T06:20:18.940288+0000","last_unstale":"2024-05-28T06:20:19.926369+0000","last_undegraded":"2024-05-28T06:20:19.926369+0000","last_fullsized":"2024-05-28T06:20:19.926369+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:13:51.233304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.951547+0000","last_change":"2024-05-28T06:20:18.950570+0000","last_active":"2024-05-28T06:20:18.951547+0000","last_peered":"2024-05-28T06:20:18.951547+0000","last_clean":"2024-05-28T06:20:18.951547+0000","last_became_active":"2024-05-28T06:20:18.950151+0000","last_became_peered":"2024-05-28T06:20:18.950151+0000","last_unstale":"2024-05-28T06:20:18.951547+0000","last_undegraded":"2024-05-28T06:20:18.951547+0000","last_fullsized":"2024-05-28T06:20:18.951547+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:13:34.791207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-05-28T06:20:20.416022+0000","last_change":"2024-05-28T06:20:18.947205+0000","last_active":"2024-05-28T06:20:20.416022+0000","last_peered":"2024-05-28T06:20:20.416022+0000","last_clean":"2024-05-28T06:20:20.416022+0000","last_became_active":"2024-05-28T06:20:18.946278+0000","last_became_peered":"2024-05-28T06:20:18.946278+0000","last_unstale":"2024-05-28T06:20:20.416022+0000","last_undegraded":"2024-05-28T06:20:20.416022+0000","last_fullsized":"2024-05-28T06:20:20.416022+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T18:02:36.378093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-05-28T06:20:19.926077+0000","last_change":"2024-05-28T06:20:18.939540+0000","last_active":"2024-05-28T06:20:19.926077+0000","last_peered":"2024-05-28T06:20:19.926077+0000","last_clean":"2024-05-28T06:20:19.926077+0000","last_became_active":"2024-05-28T06:20:18.938306+0000","last_became_peered":"2024-05-28T06:20:18.938306+0000","last_unstale":"2024-05-28T06:20:19.926077+0000","last_undegraded":"2024-05-28T06:20:19.926077+0000","last_fullsized":"2024-05-28T06:20:19.926077+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:23:32.531274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.949054+0000","last_change":"2024-05-28T06:20:18.947647+0000","last_active":"2024-05-28T06:20:18.949054+0000","last_peered":"2024-05-28T06:20:18.949054+0000","last_clean":"2024-05-28T06:20:18.949054+0000","last_became_active":"2024-05-28T06:20:18.944391+0000","last_became_peered":"2024-05-28T06:20:18.944391+0000","last_unstale":"2024-05-28T06:20:18.949054+0000","last_undegraded":"2024-05-28T06:20:18.949054+0000","last_fullsized":"2024-05-28T06:20:18.949054+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:06:21.231285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.947233+0000","last_change":"2024-05-28T06:20:18.940843+0000","last_active":"2024-05-28T06:20:18.947233+0000","last_peered":"2024-05-28T06:20:18.947233+0000","last_clean":"2024-05-28T06:20:18.947233+0000","last_became_active":"2024-05-28T06:20:18.939324+0000","last_became_peered":"2024-05-28T06:20:18.939324+0000","last_unstale":"2024-05-28T06:20:18.947233+0000","last_undegraded":"2024-05-28T06:20:18.947233+0000","last_fullsized":"2024-05-28T06:20:18.947233+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:55:01.131906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.947400+0000","last_change":"2024-05-28T06:20:18.942145+0000","last_active":"2024-05-28T06:20:18.947400+0000","last_peered":"2024-05-28T06:20:18.947400+0000","last_clean":"2024-05-28T06:20:18.947400+0000","last_became_active":"2024-05-28T06:20:18.941877+0000","last_became_peered":"2024-05-28T06:20:18.941877+0000","last_unstale":"2024-05-28T06:20:18.947400+0000","last_undegraded":"2024-05-28T06:20:18.947400+0000","last_fullsized":"2024-05-28T06:20:18.947400+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:54:18.917443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-05-28T06:20:20.415912+0000","last_change":"2024-05-28T06:20:18.942276+0000","last_active":"2024-05-28T06:20:20.415912+0000","last_peered":"2024-05-28T06:20:20.415912+0000","last_clean":"2024-05-28T06:20:20.415912+0000","last_became_active":"2024-05-28T06:20:18.941966+0000","last_became_peered":"2024-05-28T06:20:18.941966+0000","last_unstale":"2024-05-28T06:20:20.415912+0000","last_undegraded":"2024-05-28T06:20:20.415912+0000","last_fullsized":"2024-05-28T06:20:20.415912+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:06:38.458525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.940444+0000","last_change":"2024-05-28T06:20:18.937719+0000","last_active":"2024-05-28T06:20:18.940444+0000","last_peered":"2024-05-28T06:20:18.940444+0000","last_clean":"2024-05-28T06:20:18.940444+0000","last_became_active":"2024-05-28T06:20:18.935237+0000","last_became_peered":"2024-05-28T06:20:18.935237+0000","last_unstale":"2024-05-28T06:20:18.940444+0000","last_undegraded":"2024-05-28T06:20:18.940444+0000","last_fullsized":"2024-05-28T06:20:18.940444+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:13:34.791207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.951546+0000","last_change":"2024-05-28T06:20:18.949468+0000","last_active":"2024-05-28T06:20:18.951546+0000","last_peered":"2024-05-28T06:20:18.951546+0000","last_clean":"2024-05-28T06:20:18.951546+0000","last_became_active":"2024-05-28T06:20:18.948150+0000","last_became_peered":"2024-05-28T06:20:18.948150+0000","last_unstale":"2024-05-28T06:20:18.951546+0000","last_undegraded":"2024-05-28T06:20:18.951546+0000","last_fullsized":"2024-05-28T06:20:18.951546+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:23:32.531274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.952847+0000","last_change":"2024-05-28T06:20:18.951827+0000","last_active":"2024-05-28T06:20:18.952847+0000","last_peered":"2024-05-28T06:20:18.952847+0000","last_clean":"2024-05-28T06:20:18.952847+0000","last_became_active":"2024-05-28T06:20:18.951264+0000","last_became_peered":"2024-05-28T06:20:18.951264+0000","last_unstale":"2024-05-28T06:20:18.952847+0000","last_undegraded":"2024-05-28T06:20:18.952847+0000","last_fullsized":"2024-05-28T06:20:18.952847+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:59:49.791633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-05-28T06:20:19.926398+0000","last_change":"2024-05-28T06:20:18.942158+0000","last_active":"2024-05-28T06:20:19.926398+0000","last_peered":"2024-05-28T06:20:19.926398+0000","last_clean":"2024-05-28T06:20:19.926398+0000","last_became_active":"2024-05-28T06:20:18.941531+0000","last_became_peered":"2024-05-28T06:20:18.941531+0000","last_unstale":"2024-05-28T06:20:19.926398+0000","last_undegraded":"2024-05-28T06:20:19.926398+0000","last_fullsized":"2024-05-28T06:20:19.926398+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:49:52.448395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.947549+0000","last_change":"2024-05-28T06:20:18.944885+0000","last_active":"2024-05-28T06:20:18.947549+0000","last_peered":"2024-05-28T06:20:18.947549+0000","last_clean":"2024-05-28T06:20:18.947549+0000","last_became_active":"2024-05-28T06:20:18.944394+0000","last_became_peered":"2024-05-28T06:20:18.944394+0000","last_unstale":"2024-05-28T06:20:18.947549+0000","last_undegraded":"2024-05-28T06:20:18.947549+0000","last_fullsized":"2024-05-28T06:20:18.947549+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:15:45.567814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.947545+0000","last_change":"2024-05-28T06:20:18.940292+0000","last_active":"2024-05-28T06:20:18.947545+0000","last_peered":"2024-05-28T06:20:18.947545+0000","last_clean":"2024-05-28T06:20:18.947545+0000","last_became_active":"2024-05-28T06:20:18.939893+0000","last_became_peered":"2024-05-28T06:20:18.939893+0000","last_unstale":"2024-05-28T06:20:18.947545+0000","last_undegraded":"2024-05-28T06:20:18.947545+0000","last_fullsized":"2024-05-28T06:20:18.947545+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:25:49.657873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.947510+0000","last_change":"2024-05-28T06:20:18.941107+0000","last_active":"2024-05-28T06:20:18.947510+0000","last_peered":"2024-05-28T06:20:18.947510+0000","last_clean":"2024-05-28T06:20:18.947510+0000","last_became_active":"2024-05-28T06:20:18.939567+0000","last_became_peered":"2024-05-28T06:20:18.939567+0000","last_unstale":"2024-05-28T06:20:18.947510+0000","last_undegraded":"2024-05-28T06:20:18.947510+0000","last_fullsized":"2024-05-28T06:20:18.947510+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:48:15.007285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-05-28T06:20:20.415423+0000","last_change":"2024-05-28T06:20:18.952910+0000","last_active":"2024-05-28T06:20:20.415423+0000","last_peered":"2024-05-28T06:20:20.415423+0000","last_clean":"2024-05-28T06:20:20.415423+0000","last_became_active":"2024-05-28T06:20:18.952586+0000","last_became_peered":"2024-05-28T06:20:18.952586+0000","last_unstale":"2024-05-28T06:20:20.415423+0000","last_undegraded":"2024-05-28T06:20:20.415423+0000","last_fullsized":"2024-05-28T06:20:20.415423+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:12:30.286713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.950664+0000","last_change":"2024-05-28T06:20:18.946721+0000","last_active":"2024-05-28T06:20:18.950664+0000","last_peered":"2024-05-28T06:20:18.950664+0000","last_clean":"2024-05-28T06:20:18.950664+0000","last_became_active":"2024-05-28T06:20:18.944468+0000","last_became_peered":"2024-05-28T06:20:18.944468+0000","last_unstale":"2024-05-28T06:20:18.950664+0000","last_undegraded":"2024-05-28T06:20:18.950664+0000","last_fullsized":"2024-05-28T06:20:18.950664+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:23:50.885000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.939414+0000","last_change":"2024-05-28T06:20:18.935988+0000","last_active":"2024-05-28T06:20:18.939414+0000","last_peered":"2024-05-28T06:20:18.939414+0000","last_clean":"2024-05-28T06:20:18.939414+0000","last_became_active":"2024-05-28T06:20:18.935456+0000","last_became_peered":"2024-05-28T06:20:18.935456+0000","last_unstale":"2024-05-28T06:20:18.939414+0000","last_undegraded":"2024-05-28T06:20:18.939414+0000","last_fullsized":"2024-05-28T06:20:18.939414+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:06:21.231285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-05-28T06:20:20.415538+0000","last_change":"2024-05-28T06:20:18.953983+0000","last_active":"2024-05-28T06:20:20.415538+0000","last_peered":"2024-05-28T06:20:20.415538+0000","last_clean":"2024-05-28T06:20:20.415538+0000","last_became_active":"2024-05-28T06:20:18.953585+0000","last_became_peered":"2024-05-28T06:20:18.953585+0000","last_unstale":"2024-05-28T06:20:20.415538+0000","last_undegraded":"2024-05-28T06:20:20.415538+0000","last_fullsized":"2024-05-28T06:20:20.415538+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:34:04.919342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.950379+0000","last_change":"2024-05-28T06:20:18.947685+0000","last_active":"2024-05-28T06:20:18.950379+0000","last_peered":"2024-05-28T06:20:18.950379+0000","last_clean":"2024-05-28T06:20:18.950379+0000","last_became_active":"2024-05-28T06:20:18.947377+0000","last_became_peered":"2024-05-28T06:20:18.947377+0000","last_unstale":"2024-05-28T06:20:18.950379+0000","last_undegraded":"2024-05-28T06:20:18.950379+0000","last_fullsized":"2024-05-28T06:20:18.950379+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:54:18.917443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.945209+0000","last_change":"2024-05-28T06:20:18.939097+0000","last_active":"2024-05-28T06:20:18.945209+0000","last_peered":"2024-05-28T06:20:18.945209+0000","last_clean":"2024-05-28T06:20:18.945209+0000","last_became_active":"2024-05-28T06:20:18.937724+0000","last_became_peered":"2024-05-28T06:20:18.937724+0000","last_unstale":"2024-05-28T06:20:18.945209+0000","last_undegraded":"2024-05-28T06:20:18.945209+0000","last_fullsized":"2024-05-28T06:20:18.945209+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:23:32.531274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.956338+0000","last_change":"2024-05-28T06:20:18.953508+0000","last_active":"2024-05-28T06:20:18.956338+0000","last_peered":"2024-05-28T06:20:18.956338+0000","last_clean":"2024-05-28T06:20:18.956338+0000","last_became_active":"2024-05-28T06:20:18.953203+0000","last_became_peered":"2024-05-28T06:20:18.953203+0000","last_unstale":"2024-05-28T06:20:18.956338+0000","last_undegraded":"2024-05-28T06:20:18.956338+0000","last_fullsized":"2024-05-28T06:20:18.956338+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:47:56.999907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.947376+0000","last_change":"2024-05-28T06:20:18.942039+0000","last_active":"2024-05-28T06:20:18.947376+0000","last_peered":"2024-05-28T06:20:18.947376+0000","last_clean":"2024-05-28T06:20:18.947376+0000","last_became_active":"2024-05-28T06:20:18.940022+0000","last_became_peered":"2024-05-28T06:20:18.940022+0000","last_unstale":"2024-05-28T06:20:18.947376+0000","last_undegraded":"2024-05-28T06:20:18.947376+0000","last_fullsized":"2024-05-28T06:20:18.947376+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:23:50.885000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.950488+0000","last_change":"2024-05-28T06:20:18.942741+0000","last_active":"2024-05-28T06:20:18.950488+0000","last_peered":"2024-05-28T06:20:18.950488+0000","last_clean":"2024-05-28T06:20:18.950488+0000","last_became_active":"2024-05-28T06:20:18.942451+0000","last_became_peered":"2024-05-28T06:20:18.942451+0000","last_unstale":"2024-05-28T06:20:18.950488+0000","last_undegraded":"2024-05-28T06:20:18.950488+0000","last_fullsized":"2024-05-28T06:20:18.950488+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:44:20.546170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.951195+0000","last_change":"2024-05-28T06:20:18.948939+0000","last_active":"2024-05-28T06:20:18.951195+0000","last_peered":"2024-05-28T06:20:18.951195+0000","last_clean":"2024-05-28T06:20:18.951195+0000","last_became_active":"2024-05-28T06:20:18.948623+0000","last_became_peered":"2024-05-28T06:20:18.948623+0000","last_unstale":"2024-05-28T06:20:18.951195+0000","last_undegraded":"2024-05-28T06:20:18.951195+0000","last_fullsized":"2024-05-28T06:20:18.951195+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:47:11.651620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"102'4","reported_seq":16,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.962097+0000","last_change":"2024-05-28T06:20:17.921647+0000","last_active":"2024-05-28T06:20:18.962097+0000","last_peered":"2024-05-28T06:20:18.962097+0000","last_clean":"2024-05-28T06:20:18.962097+0000","last_became_active":"2024-05-28T06:20:17.920107+0000","last_became_peered":"2024-05-28T06:20:17.920107+0000","last_unstale":"2024-05-28T06:20:18.962097+0000","last_undegraded":"2024-05-28T06:20:18.962097+0000","last_fullsized":"2024-05-28T06:20:18.962097+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_clean_scrub_stamp":"2024-05-28T06:20:16.867179+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:23:49.872522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.949362+0000","last_change":"2024-05-28T06:20:18.948352+0000","last_active":"2024-05-28T06:20:18.949362+0000","last_peered":"2024-05-28T06:20:18.949362+0000","last_clean":"2024-05-28T06:20:18.949362+0000","last_became_active":"2024-05-28T06:20:18.948042+0000","last_became_peered":"2024-05-28T06:20:18.948042+0000","last_unstale":"2024-05-28T06:20:18.949362+0000","last_undegraded":"2024-05-28T06:20:18.949362+0000","last_fullsized":"2024-05-28T06:20:18.949362+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:13:34.791207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"102'1","reported_seq":15,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-05-28T06:20:20.415833+0000","last_change":"2024-05-28T06:20:17.912807+0000","last_active":"2024-05-28T06:20:20.415833+0000","last_peered":"2024-05-28T06:20:20.415833+0000","last_clean":"2024-05-28T06:20:20.415833+0000","last_became_active":"2024-05-28T06:20:17.912373+0000","last_became_peered":"2024-05-28T06:20:17.912373+0000","last_unstale":"2024-05-28T06:20:20.415833+0000","last_undegraded":"2024-05-28T06:20:20.415833+0000","last_fullsized":"2024-05-28T06:20:20.415833+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_clean_scrub_stamp":"2024-05-28T06:20:16.867179+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:23:49.872522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.954341+0000","last_change":"2024-05-28T06:20:18.943347+0000","last_active":"2024-05-28T06:20:18.954341+0000","last_peered":"2024-05-28T06:20:18.954341+0000","last_clean":"2024-05-28T06:20:18.954341+0000","last_became_active":"2024-05-28T06:20:18.942963+0000","last_became_peered":"2024-05-28T06:20:18.942963+0000","last_unstale":"2024-05-28T06:20:18.954341+0000","last_undegraded":"2024-05-28T06:20:18.954341+0000","last_fullsized":"2024-05-28T06:20:18.954341+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:06:30.855462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"102'2","reported_seq":14,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.961311+0000","last_change":"2024-05-28T06:20:17.917213+0000","last_active":"2024-05-28T06:20:18.961311+0000","last_peered":"2024-05-28T06:20:18.961311+0000","last_clean":"2024-05-28T06:20:18.961311+0000","last_became_active":"2024-05-28T06:20:17.915714+0000","last_became_peered":"2024-05-28T06:20:17.915714+0000","last_unstale":"2024-05-28T06:20:18.961311+0000","last_undegraded":"2024-05-28T06:20:18.961311+0000","last_fullsized":"2024-05-28T06:20:18.961311+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_clean_scrub_stamp":"2024-05-28T06:20:16.867179+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:33:28.764945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-05-28T06:20:19.925983+0000","last_change":"2024-05-28T06:20:18.944335+0000","last_active":"2024-05-28T06:20:19.925983+0000","last_peered":"2024-05-28T06:20:19.925983+0000","last_clean":"2024-05-28T06:20:19.925983+0000","last_became_active":"2024-05-28T06:20:18.944050+0000","last_became_peered":"2024-05-28T06:20:18.944050+0000","last_unstale":"2024-05-28T06:20:19.925983+0000","last_undegraded":"2024-05-28T06:20:19.925983+0000","last_fullsized":"2024-05-28T06:20:19.925983+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:13:34.791207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"102'2","reported_seq":14,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.962230+0000","last_change":"2024-05-28T06:20:17.922556+0000","last_active":"2024-05-28T06:20:18.962230+0000","last_peered":"2024-05-28T06:20:18.962230+0000","last_clean":"2024-05-28T06:20:18.962230+0000","last_became_active":"2024-05-28T06:20:17.920828+0000","last_became_peered":"2024-05-28T06:20:17.920828+0000","last_unstale":"2024-05-28T06:20:18.962230+0000","last_undegraded":"2024-05-28T06:20:18.962230+0000","last_fullsized":"2024-05-28T06:20:18.962230+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_clean_scrub_stamp":"2024-05-28T06:20:16.867179+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:26:05.221654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.956245+0000","last_change":"2024-05-28T06:20:18.952189+0000","last_active":"2024-05-28T06:20:18.956245+0000","last_peered":"2024-05-28T06:20:18.956245+0000","last_clean":"2024-05-28T06:20:18.956245+0000","last_became_active":"2024-05-28T06:20:18.951183+0000","last_became_peered":"2024-05-28T06:20:18.951183+0000","last_unstale":"2024-05-28T06:20:18.956245+0000","last_undegraded":"2024-05-28T06:20:18.956245+0000","last_fullsized":"2024-05-28T06:20:18.956245+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:29:20.621430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"102'1","reported_seq":13,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.956839+0000","last_change":"2024-05-28T06:20:17.924313+0000","last_active":"2024-05-28T06:20:18.956839+0000","last_peered":"2024-05-28T06:20:18.956839+0000","last_clean":"2024-05-28T06:20:18.956839+0000","last_became_active":"2024-05-28T06:20:17.923726+0000","last_became_peered":"2024-05-28T06:20:17.923726+0000","last_unstale":"2024-05-28T06:20:18.956839+0000","last_undegraded":"2024-05-28T06:20:18.956839+0000","last_fullsized":"2024-05-28T06:20:18.956839+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_clean_scrub_stamp":"2024-05-28T06:20:16.867179+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:52:19.137425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-05-28T06:20:20.415359+0000","last_change":"2024-05-28T06:20:18.945771+0000","last_active":"2024-05-28T06:20:20.415359+0000","last_peered":"2024-05-28T06:20:20.415359+0000","last_clean":"2024-05-28T06:20:20.415359+0000","last_became_active":"2024-05-28T06:20:18.944047+0000","last_became_peered":"2024-05-28T06:20:18.944047+0000","last_unstale":"2024-05-28T06:20:20.415359+0000","last_undegraded":"2024-05-28T06:20:20.415359+0000","last_fullsized":"2024-05-28T06:20:20.415359+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:33:27.455861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"102'1","reported_seq":13,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.960209+0000","last_change":"2024-05-28T06:20:17.913811+0000","last_active":"2024-05-28T06:20:18.960209+0000","last_peered":"2024-05-28T06:20:18.960209+0000","last_clean":"2024-05-28T06:20:18.960209+0000","last_became_active":"2024-05-28T06:20:17.912572+0000","last_became_peered":"2024-05-28T06:20:17.912572+0000","last_unstale":"2024-05-28T06:20:18.960209+0000","last_undegraded":"2024-05-28T06:20:18.960209+0000","last_fullsized":"2024-05-28T06:20:18.960209+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_clean_scrub_stamp":"2024-05-28T06:20:16.867179+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:29:19.608952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.949038+0000","last_change":"2024-05-28T06:20:18.947323+0000","last_active":"2024-05-28T06:20:18.949038+0000","last_peered":"2024-05-28T06:20:18.949038+0000","last_clean":"2024-05-28T06:20:18.949038+0000","last_became_active":"2024-05-28T06:20:18.946955+0000","last_became_peered":"2024-05-28T06:20:18.946955+0000","last_unstale":"2024-05-28T06:20:18.949038+0000","last_undegraded":"2024-05-28T06:20:18.949038+0000","last_fullsized":"2024-05-28T06:20:18.949038+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:52:01.724740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.936794+0000","last_change":"2024-05-28T06:20:17.916892+0000","last_active":"2024-05-28T06:20:18.936794+0000","last_peered":"2024-05-28T06:20:18.936794+0000","last_clean":"2024-05-28T06:20:18.936794+0000","last_became_active":"2024-05-28T06:20:17.916596+0000","last_became_peered":"2024-05-28T06:20:17.916596+0000","last_unstale":"2024-05-28T06:20:18.936794+0000","last_undegraded":"2024-05-28T06:20:18.936794+0000","last_fullsized":"2024-05-28T06:20:18.936794+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_clean_scrub_stamp":"2024-05-28T06:20:16.867179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:50:07.888165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-05-28T06:20:19.925308+0000","last_change":"2024-05-28T06:20:18.948033+0000","last_active":"2024-05-28T06:20:19.925308+0000","last_peered":"2024-05-28T06:20:19.925308+0000","last_clean":"2024-05-28T06:20:19.925308+0000","last_became_active":"2024-05-28T06:20:18.946352+0000","last_became_peered":"2024-05-28T06:20:18.946352+0000","last_unstale":"2024-05-28T06:20:19.925308+0000","last_undegraded":"2024-05-28T06:20:19.925308+0000","last_fullsized":"2024-05-28T06:20:19.925308+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:52:01.724740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.932899+0000","last_change":"2024-05-28T06:20:17.904261+0000","last_active":"2024-05-28T06:20:18.932899+0000","last_peered":"2024-05-28T06:20:18.932899+0000","last_clean":"2024-05-28T06:20:18.932899+0000","last_became_active":"2024-05-28T06:20:17.903943+0000","last_became_peered":"2024-05-28T06:20:17.903943+0000","last_unstale":"2024-05-28T06:20:18.932899+0000","last_undegraded":"2024-05-28T06:20:18.932899+0000","last_fullsized":"2024-05-28T06:20:18.932899+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_clean_scrub_stamp":"2024-05-28T06:20:16.867179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:48:51.382403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.925775+0000","last_change":"2024-05-28T06:20:18.923372+0000","last_active":"2024-05-28T06:20:18.925775+0000","last_peered":"2024-05-28T06:20:18.925775+0000","last_clean":"2024-05-28T06:20:18.925775+0000","last_became_active":"2024-05-28T06:20:18.922207+0000","last_became_peered":"2024-05-28T06:20:18.922207+0000","last_unstale":"2024-05-28T06:20:18.925775+0000","last_undegraded":"2024-05-28T06:20:18.925775+0000","last_fullsized":"2024-05-28T06:20:18.925775+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:53:31.071486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.931612+0000","last_change":"2024-05-28T06:20:17.924331+0000","last_active":"2024-05-28T06:20:18.931612+0000","last_peered":"2024-05-28T06:20:18.931612+0000","last_clean":"2024-05-28T06:20:18.931612+0000","last_became_active":"2024-05-28T06:20:17.924029+0000","last_became_peered":"2024-05-28T06:20:17.924029+0000","last_unstale":"2024-05-28T06:20:18.931612+0000","last_undegraded":"2024-05-28T06:20:18.931612+0000","last_fullsized":"2024-05-28T06:20:18.931612+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_clean_scrub_stamp":"2024-05-28T06:20:16.867179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:49:44.721933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-05-28T06:20:19.926494+0000","last_change":"2024-05-28T06:20:18.945763+0000","last_active":"2024-05-28T06:20:19.926494+0000","last_peered":"2024-05-28T06:20:19.926494+0000","last_clean":"2024-05-28T06:20:19.926494+0000","last_became_active":"2024-05-28T06:20:18.945474+0000","last_became_peered":"2024-05-28T06:20:18.945474+0000","last_unstale":"2024-05-28T06:20:19.926494+0000","last_undegraded":"2024-05-28T06:20:19.926494+0000","last_fullsized":"2024-05-28T06:20:19.926494+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:25:49.657873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.923744+0000","last_change":"2024-05-28T06:20:17.912420+0000","last_active":"2024-05-28T06:20:18.923744+0000","last_peered":"2024-05-28T06:20:18.923744+0000","last_clean":"2024-05-28T06:20:18.923744+0000","last_became_active":"2024-05-28T06:20:17.912084+0000","last_became_peered":"2024-05-28T06:20:17.912084+0000","last_unstale":"2024-05-28T06:20:18.923744+0000","last_undegraded":"2024-05-28T06:20:18.923744+0000","last_fullsized":"2024-05-28T06:20:18.923744+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_clean_scrub_stamp":"2024-05-28T06:20:16.867179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:19:41.740749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.950358+0000","last_change":"2024-05-28T06:20:18.947754+0000","last_active":"2024-05-28T06:20:18.950358+0000","last_peered":"2024-05-28T06:20:18.950358+0000","last_clean":"2024-05-28T06:20:18.950358+0000","last_became_active":"2024-05-28T06:20:18.947477+0000","last_became_peered":"2024-05-28T06:20:18.947477+0000","last_unstale":"2024-05-28T06:20:18.950358+0000","last_undegraded":"2024-05-28T06:20:18.950358+0000","last_fullsized":"2024-05-28T06:20:18.950358+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:53:31.071486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"102'3","reported_seq":17,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-05-28T06:20:20.416206+0000","last_change":"2024-05-28T06:20:17.926192+0000","last_active":"2024-05-28T06:20:20.416206+0000","last_peered":"2024-05-28T06:20:20.416206+0000","last_clean":"2024-05-28T06:20:20.416206+0000","last_became_active":"2024-05-28T06:20:17.925809+0000","last_became_peered":"2024-05-28T06:20:17.925809+0000","last_unstale":"2024-05-28T06:20:20.416206+0000","last_undegraded":"2024-05-28T06:20:20.416206+0000","last_fullsized":"2024-05-28T06:20:20.416206+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_clean_scrub_stamp":"2024-05-28T06:20:16.867179+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:54:17.904965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-05-28T06:20:20.415846+0000","last_change":"2024-05-28T06:20:18.954383+0000","last_active":"2024-05-28T06:20:20.415846+0000","last_peered":"2024-05-28T06:20:20.415846+0000","last_clean":"2024-05-28T06:20:20.415846+0000","last_became_active":"2024-05-28T06:20:18.951938+0000","last_became_peered":"2024-05-28T06:20:18.951938+0000","last_unstale":"2024-05-28T06:20:20.415846+0000","last_undegraded":"2024-05-28T06:20:20.415846+0000","last_fullsized":"2024-05-28T06:20:20.415846+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:28:46.723017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"102'6","reported_seq":20,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-05-28T06:20:19.926561+0000","last_change":"2024-05-28T06:20:17.919818+0000","last_active":"2024-05-28T06:20:19.926561+0000","last_peered":"2024-05-28T06:20:19.926561+0000","last_clean":"2024-05-28T06:20:19.926561+0000","last_became_active":"2024-05-28T06:20:17.919236+0000","last_became_peered":"2024-05-28T06:20:17.919236+0000","last_unstale":"2024-05-28T06:20:19.926561+0000","last_undegraded":"2024-05-28T06:20:19.926561+0000","last_fullsized":"2024-05-28T06:20:19.926561+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_clean_scrub_stamp":"2024-05-28T06:20:16.867179+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:26:05.221654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.937569+0000","last_change":"2024-05-28T06:20:18.932847+0000","last_active":"2024-05-28T06:20:18.937569+0000","last_peered":"2024-05-28T06:20:18.937569+0000","last_clean":"2024-05-28T06:20:18.937569+0000","last_became_active":"2024-05-28T06:20:18.932532+0000","last_became_peered":"2024-05-28T06:20:18.932532+0000","last_unstale":"2024-05-28T06:20:18.937569+0000","last_undegraded":"2024-05-28T06:20:18.937569+0000","last_fullsized":"2024-05-28T06:20:18.937569+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:34:28.359961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"102'2","reported_seq":14,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.962591+0000","last_change":"2024-05-28T06:20:17.921616+0000","last_active":"2024-05-28T06:20:18.962591+0000","last_peered":"2024-05-28T06:20:18.962591+0000","last_clean":"2024-05-28T06:20:18.962591+0000","last_became_active":"2024-05-28T06:20:17.921332+0000","last_became_peered":"2024-05-28T06:20:17.921332+0000","last_unstale":"2024-05-28T06:20:18.962591+0000","last_undegraded":"2024-05-28T06:20:18.962591+0000","last_fullsized":"2024-05-28T06:20:18.962591+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_clean_scrub_stamp":"2024-05-28T06:20:16.867179+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:21:17.875374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.955217+0000","last_change":"2024-05-28T06:20:18.949515+0000","last_active":"2024-05-28T06:20:18.955217+0000","last_peered":"2024-05-28T06:20:18.955217+0000","last_clean":"2024-05-28T06:20:18.955217+0000","last_became_active":"2024-05-28T06:20:18.949214+0000","last_became_peered":"2024-05-28T06:20:18.949214+0000","last_unstale":"2024-05-28T06:20:18.955217+0000","last_undegraded":"2024-05-28T06:20:18.955217+0000","last_fullsized":"2024-05-28T06:20:18.955217+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:24:26.011451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.940635+0000","last_change":"2024-05-28T06:20:17.923138+0000","last_active":"2024-05-28T06:20:18.940635+0000","last_peered":"2024-05-28T06:20:18.940635+0000","last_clean":"2024-05-28T06:20:18.940635+0000","last_became_active":"2024-05-28T06:20:17.922501+0000","last_became_peered":"2024-05-28T06:20:17.922501+0000","last_unstale":"2024-05-28T06:20:18.940635+0000","last_undegraded":"2024-05-28T06:20:18.940635+0000","last_fullsized":"2024-05-28T06:20:18.940635+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_clean_scrub_stamp":"2024-05-28T06:20:16.867179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T18:15:15.367346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-05-28T06:20:19.924857+0000","last_change":"2024-05-28T06:20:18.942969+0000","last_active":"2024-05-28T06:20:19.924857+0000","last_peered":"2024-05-28T06:20:19.924857+0000","last_clean":"2024-05-28T06:20:19.924857+0000","last_became_active":"2024-05-28T06:20:18.939044+0000","last_became_peered":"2024-05-28T06:20:18.939044+0000","last_unstale":"2024-05-28T06:20:19.924857+0000","last_undegraded":"2024-05-28T06:20:19.924857+0000","last_fullsized":"2024-05-28T06:20:19.924857+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:53:31.071486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":14,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-05-28T06:20:19.925345+0000","last_change":"2024-05-28T06:20:17.926049+0000","last_active":"2024-05-28T06:20:19.925345+0000","last_peered":"2024-05-28T06:20:19.925345+0000","last_clean":"2024-05-28T06:20:19.925345+0000","last_became_active":"2024-05-28T06:20:17.923234+0000","last_became_peered":"2024-05-28T06:20:17.923234+0000","last_unstale":"2024-05-28T06:20:19.925345+0000","last_undegraded":"2024-05-28T06:20:19.925345+0000","last_fullsized":"2024-05-28T06:20:19.925345+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_clean_scrub_stamp":"2024-05-28T06:20:16.867179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:04:54.249701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.947481+0000","last_change":"2024-05-28T06:20:18.941003+0000","last_active":"2024-05-28T06:20:18.947481+0000","last_peered":"2024-05-28T06:20:18.947481+0000","last_clean":"2024-05-28T06:20:18.947481+0000","last_became_active":"2024-05-28T06:20:18.940699+0000","last_became_peered":"2024-05-28T06:20:18.940699+0000","last_unstale":"2024-05-28T06:20:18.947481+0000","last_undegraded":"2024-05-28T06:20:18.947481+0000","last_fullsized":"2024-05-28T06:20:18.947481+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:44:20.587585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"102'8","reported_seq":20,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.962777+0000","last_change":"2024-05-28T06:20:17.917887+0000","last_active":"2024-05-28T06:20:18.962777+0000","last_peered":"2024-05-28T06:20:18.962777+0000","last_clean":"2024-05-28T06:20:18.962777+0000","last_became_active":"2024-05-28T06:20:17.917597+0000","last_became_peered":"2024-05-28T06:20:17.917597+0000","last_unstale":"2024-05-28T06:20:18.962777+0000","last_undegraded":"2024-05-28T06:20:18.962777+0000","last_fullsized":"2024-05-28T06:20:18.962777+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_clean_scrub_stamp":"2024-05-28T06:20:16.867179+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:15:03.433459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.950690+0000","last_change":"2024-05-28T06:20:18.944640+0000","last_active":"2024-05-28T06:20:18.950690+0000","last_peered":"2024-05-28T06:20:18.950690+0000","last_clean":"2024-05-28T06:20:18.950690+0000","last_became_active":"2024-05-28T06:20:18.944125+0000","last_became_peered":"2024-05-28T06:20:18.944125+0000","last_unstale":"2024-05-28T06:20:18.950690+0000","last_undegraded":"2024-05-28T06:20:18.950690+0000","last_fullsized":"2024-05-28T06:20:18.950690+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:34:28.359961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"102'2","reported_seq":16,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-05-28T06:20:19.925894+0000","last_change":"2024-05-28T06:20:17.922062+0000","last_active":"2024-05-28T06:20:19.925894+0000","last_peered":"2024-05-28T06:20:19.925894+0000","last_clean":"2024-05-28T06:20:19.925894+0000","last_became_active":"2024-05-28T06:20:17.921772+0000","last_became_peered":"2024-05-28T06:20:17.921772+0000","last_unstale":"2024-05-28T06:20:19.925894+0000","last_undegraded":"2024-05-28T06:20:19.925894+0000","last_fullsized":"2024-05-28T06:20:19.925894+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_clean_scrub_stamp":"2024-05-28T06:20:16.867179+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:29:19.608952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.956278+0000","last_change":"2024-05-28T06:20:18.952158+0000","last_active":"2024-05-28T06:20:18.956278+0000","last_peered":"2024-05-28T06:20:18.956278+0000","last_clean":"2024-05-28T06:20:18.956278+0000","last_became_active":"2024-05-28T06:20:18.949205+0000","last_became_peered":"2024-05-28T06:20:18.949205+0000","last_unstale":"2024-05-28T06:20:18.956278+0000","last_undegraded":"2024-05-28T06:20:18.956278+0000","last_fullsized":"2024-05-28T06:20:18.956278+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:04:55.262180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"102'1","reported_seq":13,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.957102+0000","last_change":"2024-05-28T06:20:17.923109+0000","last_active":"2024-05-28T06:20:18.957102+0000","last_peered":"2024-05-28T06:20:18.957102+0000","last_clean":"2024-05-28T06:20:18.957102+0000","last_became_active":"2024-05-28T06:20:17.921903+0000","last_became_peered":"2024-05-28T06:20:17.921903+0000","last_unstale":"2024-05-28T06:20:18.957102+0000","last_undegraded":"2024-05-28T06:20:18.957102+0000","last_fullsized":"2024-05-28T06:20:18.957102+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_clean_scrub_stamp":"2024-05-28T06:20:16.867179+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:33:28.764945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-05-28T06:20:20.416066+0000","last_change":"2024-05-28T06:20:18.954256+0000","last_active":"2024-05-28T06:20:20.416066+0000","last_peered":"2024-05-28T06:20:20.416066+0000","last_clean":"2024-05-28T06:20:20.416066+0000","last_became_active":"2024-05-28T06:20:18.951535+0000","last_became_peered":"2024-05-28T06:20:18.951535+0000","last_unstale":"2024-05-28T06:20:20.416066+0000","last_undegraded":"2024-05-28T06:20:20.416066+0000","last_fullsized":"2024-05-28T06:20:20.416066+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T18:08:21.610036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.912722+0000","last_change":"2024-05-28T06:20:17.927124+0000","last_active":"2024-05-28T06:20:18.912722+0000","last_peered":"2024-05-28T06:20:18.912722+0000","last_clean":"2024-05-28T06:20:18.912722+0000","last_became_active":"2024-05-28T06:20:17.926859+0000","last_became_peered":"2024-05-28T06:20:17.926859+0000","last_unstale":"2024-05-28T06:20:18.912722+0000","last_undegraded":"2024-05-28T06:20:18.912722+0000","last_fullsized":"2024-05-28T06:20:18.912722+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_clean_scrub_stamp":"2024-05-28T06:20:16.867179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:50:07.888165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.947401+0000","last_change":"2024-05-28T06:20:18.945503+0000","last_active":"2024-05-28T06:20:18.947401+0000","last_peered":"2024-05-28T06:20:18.947401+0000","last_clean":"2024-05-28T06:20:18.947401+0000","last_became_active":"2024-05-28T06:20:18.945195+0000","last_became_peered":"2024-05-28T06:20:18.945195+0000","last_unstale":"2024-05-28T06:20:18.947401+0000","last_undegraded":"2024-05-28T06:20:18.947401+0000","last_fullsized":"2024-05-28T06:20:18.947401+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:36:31.817085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"102'2","reported_seq":14,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.961406+0000","last_change":"2024-05-28T06:20:17.917817+0000","last_active":"2024-05-28T06:20:18.961406+0000","last_peered":"2024-05-28T06:20:18.961406+0000","last_clean":"2024-05-28T06:20:18.961406+0000","last_became_active":"2024-05-28T06:20:17.916979+0000","last_became_peered":"2024-05-28T06:20:17.916979+0000","last_unstale":"2024-05-28T06:20:18.961406+0000","last_undegraded":"2024-05-28T06:20:18.961406+0000","last_fullsized":"2024-05-28T06:20:18.961406+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_clean_scrub_stamp":"2024-05-28T06:20:16.867179+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:41:42.790025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.947526+0000","last_change":"2024-05-28T06:20:18.944664+0000","last_active":"2024-05-28T06:20:18.947526+0000","last_peered":"2024-05-28T06:20:18.947526+0000","last_clean":"2024-05-28T06:20:18.947526+0000","last_became_active":"2024-05-28T06:20:18.941309+0000","last_became_peered":"2024-05-28T06:20:18.941309+0000","last_unstale":"2024-05-28T06:20:18.947526+0000","last_undegraded":"2024-05-28T06:20:18.947526+0000","last_fullsized":"2024-05-28T06:20:18.947526+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:23:37.846492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.942965+0000","last_change":"2024-05-28T06:20:17.925585+0000","last_active":"2024-05-28T06:20:18.942965+0000","last_peered":"2024-05-28T06:20:18.942965+0000","last_clean":"2024-05-28T06:20:18.942965+0000","last_became_active":"2024-05-28T06:20:17.925283+0000","last_became_peered":"2024-05-28T06:20:17.925283+0000","last_unstale":"2024-05-28T06:20:18.942965+0000","last_undegraded":"2024-05-28T06:20:18.942965+0000","last_fullsized":"2024-05-28T06:20:18.942965+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_clean_scrub_stamp":"2024-05-28T06:20:16.867179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:59:51.754198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.947484+0000","last_change":"2024-05-28T06:20:18.943551+0000","last_active":"2024-05-28T06:20:18.947484+0000","last_peered":"2024-05-28T06:20:18.947484+0000","last_clean":"2024-05-28T06:20:18.947484+0000","last_became_active":"2024-05-28T06:20:18.943259+0000","last_became_peered":"2024-05-28T06:20:18.943259+0000","last_unstale":"2024-05-28T06:20:18.947484+0000","last_undegraded":"2024-05-28T06:20:18.947484+0000","last_fullsized":"2024-05-28T06:20:18.947484+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:19:45.078600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.942761+0000","last_change":"2024-05-28T06:20:17.918736+0000","last_active":"2024-05-28T06:20:18.942761+0000","last_peered":"2024-05-28T06:20:18.942761+0000","last_clean":"2024-05-28T06:20:18.942761+0000","last_became_active":"2024-05-28T06:20:17.918447+0000","last_became_peered":"2024-05-28T06:20:17.918447+0000","last_unstale":"2024-05-28T06:20:18.942761+0000","last_undegraded":"2024-05-28T06:20:18.942761+0000","last_fullsized":"2024-05-28T06:20:18.942761+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_clean_scrub_stamp":"2024-05-28T06:20:16.867179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:48:48.325930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.951463+0000","last_change":"2024-05-28T06:20:18.950512+0000","last_active":"2024-05-28T06:20:18.951463+0000","last_peered":"2024-05-28T06:20:18.951463+0000","last_clean":"2024-05-28T06:20:18.951463+0000","last_became_active":"2024-05-28T06:20:18.950183+0000","last_became_peered":"2024-05-28T06:20:18.950183+0000","last_unstale":"2024-05-28T06:20:18.951463+0000","last_undegraded":"2024-05-28T06:20:18.951463+0000","last_fullsized":"2024-05-28T06:20:18.951463+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:06:21.231285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.912441+0000","last_change":"2024-05-28T06:20:17.910325+0000","last_active":"2024-05-28T06:20:18.912441+0000","last_peered":"2024-05-28T06:20:18.912441+0000","last_clean":"2024-05-28T06:20:18.912441+0000","last_became_active":"2024-05-28T06:20:17.910055+0000","last_became_peered":"2024-05-28T06:20:17.910055+0000","last_unstale":"2024-05-28T06:20:18.912441+0000","last_undegraded":"2024-05-28T06:20:18.912441+0000","last_fullsized":"2024-05-28T06:20:18.912441+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_clean_scrub_stamp":"2024-05-28T06:20:16.867179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:24:06.175231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.946687+0000","last_change":"2024-05-28T06:20:18.939534+0000","last_active":"2024-05-28T06:20:18.946687+0000","last_peered":"2024-05-28T06:20:18.946687+0000","last_clean":"2024-05-28T06:20:18.946687+0000","last_became_active":"2024-05-28T06:20:18.939173+0000","last_became_peered":"2024-05-28T06:20:18.939173+0000","last_unstale":"2024-05-28T06:20:18.946687+0000","last_undegraded":"2024-05-28T06:20:18.946687+0000","last_fullsized":"2024-05-28T06:20:18.946687+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:13:51.233304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"102'2","reported_seq":14,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.958192+0000","last_change":"2024-05-28T06:20:17.927698+0000","last_active":"2024-05-28T06:20:18.958192+0000","last_peered":"2024-05-28T06:20:18.958192+0000","last_clean":"2024-05-28T06:20:18.958192+0000","last_became_active":"2024-05-28T06:20:17.927383+0000","last_became_peered":"2024-05-28T06:20:17.927383+0000","last_unstale":"2024-05-28T06:20:18.958192+0000","last_undegraded":"2024-05-28T06:20:18.958192+0000","last_fullsized":"2024-05-28T06:20:18.958192+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_clean_scrub_stamp":"2024-05-28T06:20:16.867179+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:54:02.951428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.955355+0000","last_change":"2024-05-28T06:20:18.950303+0000","last_active":"2024-05-28T06:20:18.955355+0000","last_peered":"2024-05-28T06:20:18.955355+0000","last_clean":"2024-05-28T06:20:18.955355+0000","last_became_active":"2024-05-28T06:20:18.949736+0000","last_became_peered":"2024-05-28T06:20:18.949736+0000","last_unstale":"2024-05-28T06:20:18.955355+0000","last_undegraded":"2024-05-28T06:20:18.955355+0000","last_fullsized":"2024-05-28T06:20:18.955355+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:32:52.994598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.915478+0000","last_change":"2024-05-28T06:20:17.917237+0000","last_active":"2024-05-28T06:20:18.915478+0000","last_peered":"2024-05-28T06:20:18.915478+0000","last_clean":"2024-05-28T06:20:18.915478+0000","last_became_active":"2024-05-28T06:20:17.915703+0000","last_became_peered":"2024-05-28T06:20:17.915703+0000","last_unstale":"2024-05-28T06:20:18.915478+0000","last_undegraded":"2024-05-28T06:20:18.915478+0000","last_fullsized":"2024-05-28T06:20:18.915478+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_clean_scrub_stamp":"2024-05-28T06:20:16.867179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:54:17.904965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.956040+0000","last_change":"2024-05-28T06:20:18.954100+0000","last_active":"2024-05-28T06:20:18.956040+0000","last_peered":"2024-05-28T06:20:18.956040+0000","last_clean":"2024-05-28T06:20:18.956040+0000","last_became_active":"2024-05-28T06:20:18.953338+0000","last_became_peered":"2024-05-28T06:20:18.953338+0000","last_unstale":"2024-05-28T06:20:18.956040+0000","last_undegraded":"2024-05-28T06:20:18.956040+0000","last_fullsized":"2024-05-28T06:20:18.956040+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:39:20.591183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.912813+0000","last_change":"2024-05-28T06:20:17.928473+0000","last_active":"2024-05-28T06:20:18.912813+0000","last_peered":"2024-05-28T06:20:18.912813+0000","last_clean":"2024-05-28T06:20:18.912813+0000","last_became_active":"2024-05-28T06:20:17.928180+0000","last_became_peered":"2024-05-28T06:20:17.928180+0000","last_unstale":"2024-05-28T06:20:18.912813+0000","last_undegraded":"2024-05-28T06:20:18.912813+0000","last_fullsized":"2024-05-28T06:20:18.912813+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_clean_scrub_stamp":"2024-05-28T06:20:16.867179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:19:41.740749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.948756+0000","last_change":"2024-05-28T06:20:18.943224+0000","last_active":"2024-05-28T06:20:18.948756+0000","last_peered":"2024-05-28T06:20:18.948756+0000","last_clean":"2024-05-28T06:20:18.948756+0000","last_became_active":"2024-05-28T06:20:18.941234+0000","last_became_peered":"2024-05-28T06:20:18.941234+0000","last_unstale":"2024-05-28T06:20:18.948756+0000","last_undegraded":"2024-05-28T06:20:18.948756+0000","last_fullsized":"2024-05-28T06:20:18.948756+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:41:43.802503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.935946+0000","last_change":"2024-05-28T06:20:17.923709+0000","last_active":"2024-05-28T06:20:18.935946+0000","last_peered":"2024-05-28T06:20:18.935946+0000","last_clean":"2024-05-28T06:20:18.935946+0000","last_became_active":"2024-05-28T06:20:17.922798+0000","last_became_peered":"2024-05-28T06:20:17.922798+0000","last_unstale":"2024-05-28T06:20:18.935946+0000","last_undegraded":"2024-05-28T06:20:18.935946+0000","last_fullsized":"2024-05-28T06:20:18.935946+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_clean_scrub_stamp":"2024-05-28T06:20:16.867179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:24:06.175231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.947210+0000","last_change":"2024-05-28T06:20:18.939976+0000","last_active":"2024-05-28T06:20:18.947210+0000","last_peered":"2024-05-28T06:20:18.947210+0000","last_clean":"2024-05-28T06:20:18.947210+0000","last_became_active":"2024-05-28T06:20:18.939633+0000","last_became_peered":"2024-05-28T06:20:18.939633+0000","last_unstale":"2024-05-28T06:20:18.947210+0000","last_undegraded":"2024-05-28T06:20:18.947210+0000","last_fullsized":"2024-05-28T06:20:18.947210+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:40:34.672443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.913651+0000","last_change":"2024-05-28T06:20:17.917929+0000","last_active":"2024-05-28T06:20:18.913651+0000","last_peered":"2024-05-28T06:20:18.913651+0000","last_clean":"2024-05-28T06:20:18.913651+0000","last_became_active":"2024-05-28T06:20:17.916446+0000","last_became_peered":"2024-05-28T06:20:17.916446+0000","last_unstale":"2024-05-28T06:20:18.913651+0000","last_undegraded":"2024-05-28T06:20:18.913651+0000","last_fullsized":"2024-05-28T06:20:18.913651+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_clean_scrub_stamp":"2024-05-28T06:20:16.867179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:34:27.347482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-05-28T06:20:19.925542+0000","last_change":"2024-05-28T06:20:18.941825+0000","last_active":"2024-05-28T06:20:19.925542+0000","last_peered":"2024-05-28T06:20:19.925542+0000","last_clean":"2024-05-28T06:20:19.925542+0000","last_became_active":"2024-05-28T06:20:18.941470+0000","last_became_peered":"2024-05-28T06:20:18.941470+0000","last_unstale":"2024-05-28T06:20:19.925542+0000","last_undegraded":"2024-05-28T06:20:19.925542+0000","last_fullsized":"2024-05-28T06:20:19.925542+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:06:21.231285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.929972+0000","last_change":"2024-05-28T06:20:17.918870+0000","last_active":"2024-05-28T06:20:18.929972+0000","last_peered":"2024-05-28T06:20:18.929972+0000","last_clean":"2024-05-28T06:20:18.929972+0000","last_became_active":"2024-05-28T06:20:17.918328+0000","last_became_peered":"2024-05-28T06:20:17.918328+0000","last_unstale":"2024-05-28T06:20:18.929972+0000","last_undegraded":"2024-05-28T06:20:18.929972+0000","last_fullsized":"2024-05-28T06:20:18.929972+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_clean_scrub_stamp":"2024-05-28T06:20:16.867179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:26:05.221654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.946287+0000","last_change":"2024-05-28T06:20:18.939717+0000","last_active":"2024-05-28T06:20:18.946287+0000","last_peered":"2024-05-28T06:20:18.946287+0000","last_clean":"2024-05-28T06:20:18.946287+0000","last_became_active":"2024-05-28T06:20:18.939426+0000","last_became_peered":"2024-05-28T06:20:18.939426+0000","last_unstale":"2024-05-28T06:20:18.946287+0000","last_undegraded":"2024-05-28T06:20:18.946287+0000","last_fullsized":"2024-05-28T06:20:18.946287+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:43:07.928152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.924112+0000","last_change":"2024-05-28T06:20:17.906604+0000","last_active":"2024-05-28T06:20:18.924112+0000","last_peered":"2024-05-28T06:20:18.924112+0000","last_clean":"2024-05-28T06:20:18.924112+0000","last_became_active":"2024-05-28T06:20:17.906310+0000","last_became_peered":"2024-05-28T06:20:17.906310+0000","last_unstale":"2024-05-28T06:20:18.924112+0000","last_undegraded":"2024-05-28T06:20:18.924112+0000","last_fullsized":"2024-05-28T06:20:18.924112+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_clean_scrub_stamp":"2024-05-28T06:20:16.867179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:41:42.790025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.949799+0000","last_change":"2024-05-28T06:20:18.948057+0000","last_active":"2024-05-28T06:20:18.949799+0000","last_peered":"2024-05-28T06:20:18.949799+0000","last_clean":"2024-05-28T06:20:18.949799+0000","last_became_active":"2024-05-28T06:20:18.946998+0000","last_became_peered":"2024-05-28T06:20:18.946998+0000","last_unstale":"2024-05-28T06:20:18.949799+0000","last_undegraded":"2024-05-28T06:20:18.949799+0000","last_fullsized":"2024-05-28T06:20:18.949799+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:52:01.724740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.940414+0000","last_change":"2024-05-28T06:20:17.916573+0000","last_active":"2024-05-28T06:20:18.940414+0000","last_peered":"2024-05-28T06:20:18.940414+0000","last_clean":"2024-05-28T06:20:18.940414+0000","last_became_active":"2024-05-28T06:20:17.916172+0000","last_became_peered":"2024-05-28T06:20:17.916172+0000","last_unstale":"2024-05-28T06:20:18.940414+0000","last_undegraded":"2024-05-28T06:20:18.940414+0000","last_fullsized":"2024-05-28T06:20:18.940414+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_clean_scrub_stamp":"2024-05-28T06:20:16.867179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:38:12.023168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-05-28T06:20:18.947418+0000","last_change":"2024-05-28T06:20:18.945549+0000","last_active":"2024-05-28T06:20:18.947418+0000","last_peered":"2024-05-28T06:20:18.947418+0000","last_clean":"2024-05-28T06:20:18.947418+0000","last_became_active":"2024-05-28T06:20:18.945251+0000","last_became_peered":"2024-05-28T06:20:18.945251+0000","last_unstale":"2024-05-28T06:20:18.947418+0000","last_undegraded":"2024-05-28T06:20:18.947418+0000","last_fullsized":"2024-05-28T06:20:18.947418+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:17.879658+0000","last_clean_scrub_stamp":"2024-05-28T06:20:17.879658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:51:04.199722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"102'1","reported_seq":15,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-05-28T06:20:20.415660+0000","last_change":"2024-05-28T06:20:17.913519+0000","last_active":"2024-05-28T06:20:20.415660+0000","last_peered":"2024-05-28T06:20:20.415660+0000","last_clean":"2024-05-28T06:20:20.415660+0000","last_became_active":"2024-05-28T06:20:17.910345+0000","last_became_peered":"2024-05-28T06:20:17.910345+0000","last_unstale":"2024-05-28T06:20:20.415660+0000","last_undegraded":"2024-05-28T06:20:20.415660+0000","last_fullsized":"2024-05-28T06:20:20.415660+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:20:16.867179+0000","last_clean_scrub_stamp":"2024-05-28T06:20:16.867179+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:34:27.347482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":332,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-05-28T06:20:20.415582+0000","last_change":"2024-05-28T06:04:01.039129+0000","last_active":"2024-05-28T06:20:20.415582+0000","last_peered":"2024-05-28T06:20:20.415582+0000","last_clean":"2024-05-28T06:20:20.415582+0000","last_became_active":"2024-05-28T06:04:01.033906+0000","last_became_peered":"2024-05-28T06:04:01.033906+0000","last_unstale":"2024-05-28T06:20:20.415582+0000","last_undegraded":"2024-05-28T06:20:20.415582+0000","last_fullsized":"2024-05-28T06:20:20.415582+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:03:59.864237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:03:59.864237+0000","last_clean_scrub_stamp":"2024-05-28T06:03:59.864237+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:17:36.440245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575047,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":127356,"kb_used_data":1832,"kb_used_omap":1,"kb_used_meta":125502,"kb_avail":94244484,"statfs":{"total":96636764160,"available":96506351616,"internally_reserved":0,"allocated":1875968,"data_stored":1146383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":128514506},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575049,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":126760,"kb_used_data":1240,"kb_used_omap":1,"kb_used_meta":125502,"kb_avail":94245080,"statfs":{"total":96636764160,"available":96506961920,"internally_reserved":0,"allocated":1269760,"data_stored":550199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":128514502},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575047,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76768,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94295072,"statfs":{"total":96636764160,"available":96558153728,"internally_reserved":0,"allocated":1265664,"data_stored":549617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575047,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":86520,"kb_used_data":1252,"kb_used_omap":1,"kb_used_meta":85246,"kb_avail":94285320,"statfs":{"total":96636764160,"available":96548167680,"internally_reserved":0,"allocated":1282048,"data_stored":551283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87292357},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575049,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":126764,"kb_used_data":1248,"kb_used_omap":1,"kb_used_meta":125502,"kb_avail":94245076,"statfs":{"total":96636764160,"available":96506957824,"internally_reserved":0,"allocated":1277952,"data_stored":551181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":128514503},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575048,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":144804,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":143550,"kb_avail":94227036,"statfs":{"total":96636764160,"available":96488484864,"internally_reserved":0,"allocated":1265664,"data_stored":549581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":146995653},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575048,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":126752,"kb_used_data":1232,"kb_used_omap":1,"kb_used_meta":125502,"kb_avail":94245088,"statfs":{"total":96636764160,"available":96506970112,"internally_reserved":0,"allocated":1261568,"data_stored":549547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":128514501},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575047,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64868,"kb_used_data":1816,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94306972,"statfs":{"total":96636764160,"available":96570339328,"internally_reserved":0,"allocated":1859584,"data_stored":1140497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-28T06:20:23.786 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-05-28T06:20:23.786 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-05-28T06:20:24.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.043+0000 7f2ac5600640 1 Processor -- start 2024-05-28T06:20:24.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.043+0000 7f2ac5600640 1 -- start start 2024-05-28T06:20:24.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.044+0000 7f2ac5600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2ac00d0640 0x7f2ac015d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:24.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.045+0000 7f2ac5600640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ac00d0060 con 0x7f2ac00b32c0 2024-05-28T06:20:24.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.045+0000 7f2ac5600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ac015cb10 con 0x7f2ac00d0a20 2024-05-28T06:20:24.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.045+0000 7f2ac5600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f2ac00adbb0 con 0x7f2ac00d0640 2024-05-28T06:20:24.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.045+0000 7f2abf400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f2ac00b32c0 0x7f2ac00b3750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:51084/0 (socket says 172.21.15.96:51084) 2024-05-28T06:20:24.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.045+0000 7f2abf400640 1 -- 172.21.15.96:0/531872549 learned_addr learned my addr 172.21.15.96:0/531872549 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:20:24.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.045+0000 7f2abea00640 1 --2- 172.21.15.96:0/531872549 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2ac00d0640 0x7f2ac015d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:24.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.046+0000 7f2abe000640 1 -- 172.21.15.96:0/531872549 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 724616357 0 0) 0x7f2ac00d0060 con 0x7f2ac00b32c0 2024-05-28T06:20:24.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.046+0000 7f2abe000640 1 -- 172.21.15.96:0/531872549 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2aa4003520 con 0x7f2ac00b32c0 2024-05-28T06:20:24.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.046+0000 7f2abe000640 1 -- 172.21.15.96:0/531872549 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 516003664 0 0) 0x7f2aa4003520 con 0x7f2ac00b32c0 2024-05-28T06:20:24.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.046+0000 7f2abe000640 1 -- 172.21.15.96:0/531872549 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2ac00d0640 msgr2=0x7f2ac015d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:24.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.046+0000 7f2abe000640 1 --2- 172.21.15.96:0/531872549 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2ac00d0640 0x7f2ac015d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:24.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.046+0000 7f2abe000640 1 -- 172.21.15.96:0/531872549 >> v1:172.21.15.123:6789/0 conn(0x7f2ac00d0a20 legacy=0x7f2ac00cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:24.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.046+0000 7f2abe000640 1 -- 172.21.15.96:0/531872549 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ac00ad5f0 con 0x7f2ac00b32c0 2024-05-28T06:20:24.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.046+0000 7f2abe000640 1 -- 172.21.15.96:0/531872549 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f2ab80019f0 con 0x7f2ac00b32c0 2024-05-28T06:20:24.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.046+0000 7f2abe000640 1 -- 172.21.15.96:0/531872549 >> v1:172.21.15.96:6789/0 conn(0x7f2ac00b32c0 legacy=0x7f2ac00b3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:24.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.046+0000 7f2abe000640 1 --2- 172.21.15.96:0/531872549 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2aa40039e0 0x7f2aa4003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:24.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.046+0000 7f2abe000640 1 --2- 172.21.15.96:0/531872549 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2ac00d0640 0x7f2aa4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:24.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.046+0000 7f2abe000640 1 --2- 172.21.15.96:0/531872549 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2aa4004d40 0x7f2aa40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:24.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.046+0000 7f2abe000640 1 -- 172.21.15.96:0/531872549 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f2aa400ba70 con 0x7f2aa40039e0 2024-05-28T06:20:24.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.046+0000 7f2abe000640 1 -- 172.21.15.96:0/531872549 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f2aa400bbf0 con 0x7f2aa4004d40 2024-05-28T06:20:24.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.046+0000 7f2abe000640 1 -- 172.21.15.96:0/531872549 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f2aa400bdf0 con 0x7f2ac00d0640 2024-05-28T06:20:24.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.047+0000 7f2abea00640 1 --2- 172.21.15.96:0/531872549 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2aa40039e0 0x7f2aa4003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:24.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.047+0000 7f2ab6000640 1 --2- 172.21.15.96:0/531872549 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2ac00d0640 0x7f2aa4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:24.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.047+0000 7f2abf400640 1 --2- 172.21.15.96:0/531872549 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2aa4004d40 0x7f2aa40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:24.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.047+0000 7f2abea00640 1 -- 172.21.15.96:0/531872549 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2ac00d0640 msgr2=0x7f2aa4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:24.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.047+0000 7f2abea00640 1 --2- 172.21.15.96:0/531872549 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2ac00d0640 0x7f2aa4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:24.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.047+0000 7f2abea00640 1 -- 172.21.15.96:0/531872549 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2aa4004d40 msgr2=0x7f2aa40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:24.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.047+0000 7f2abea00640 1 --2- 172.21.15.96:0/531872549 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2aa4004d40 0x7f2aa40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:24.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.047+0000 7f2abea00640 1 -- 172.21.15.96:0/531872549 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2aa4003520 con 0x7f2aa40039e0 2024-05-28T06:20:24.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.047+0000 7f2abf400640 1 --2- 172.21.15.96:0/531872549 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2aa4004d40 0x7f2aa40051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:20:24.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.048+0000 7f2abea00640 1 --2- 172.21.15.96:0/531872549 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2aa40039e0 0x7f2aa4003e00 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7f2ab0009a90 tx=0x7f2aa4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9430836529450dbe server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:24.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.048+0000 7f2abe000640 1 -- 172.21.15.96:0/531872549 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ab0036070 con 0x7f2aa40039e0 2024-05-28T06:20:24.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.048+0000 7f2abe000640 1 -- 172.21.15.96:0/531872549 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ab0031400 con 0x7f2aa40039e0 2024-05-28T06:20:24.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.048+0000 7f2abe000640 1 -- 172.21.15.96:0/531872549 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ab0009690 con 0x7f2aa40039e0 2024-05-28T06:20:24.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.049+0000 7f2ac5600640 1 -- 172.21.15.96:0/531872549 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2aa40039e0 msgr2=0x7f2aa4003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:24.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.049+0000 7f2ac5600640 1 --2- 172.21.15.96:0/531872549 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2aa40039e0 0x7f2aa4003e00 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7f2ab0009a90 tx=0x7f2aa4005830 comp rx=0 tx=0).stop 2024-05-28T06:20:24.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.049+0000 7f2abf400640 1 -- 172.21.15.96:0/531872549 reap_dead start 2024-05-28T06:20:24.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.049+0000 7f2ac5600640 1 -- 172.21.15.96:0/531872549 shutdown_connections 2024-05-28T06:20:24.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.049+0000 7f2ac5600640 1 -- 172.21.15.96:0/531872549 >> 172.21.15.96:0/531872549 conn(0x7f2ac006bd20 msgr2=0x7f2ac006c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:24.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.049+0000 7f2ac5600640 1 -- 172.21.15.96:0/531872549 shutdown_connections 2024-05-28T06:20:24.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.049+0000 7f2ac5600640 1 -- 172.21.15.96:0/531872549 wait complete. 2024-05-28T06:20:24.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.051+0000 7f2ac5600640 1 Processor -- start 2024-05-28T06:20:24.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.051+0000 7f2ac5600640 1 -- start start 2024-05-28T06:20:24.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.052+0000 7f2ac5600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2aa40039e0 0x7f2ac01f3c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:24.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.052+0000 7f2ac5600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ac006e3d0 0x7f2ac015b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:24.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.052+0000 7f2ac5600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2ac015b780 0x7f2ac01544c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:24.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.052+0000 7f2ac5600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f2ac015d0b0 con 0x7f2ac015b780 2024-05-28T06:20:24.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.052+0000 7f2ac5600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f2ac015cdd0 con 0x7f2ac006e3d0 2024-05-28T06:20:24.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.052+0000 7f2ac5600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f2ac00b3e60 con 0x7f2aa40039e0 2024-05-28T06:20:24.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.052+0000 7f2abf400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2ac015b780 0x7f2ac01544c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:24.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.052+0000 7f2abf400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2ac015b780 0x7f2ac01544c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58818/0 (socket says 172.21.15.96:58818) 2024-05-28T06:20:24.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.052+0000 7f2abf400640 1 -- 172.21.15.96:0/2668523219 learned_addr learned my addr 172.21.15.96:0/2668523219 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:20:24.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.053+0000 7f2ab6000640 1 --2- 172.21.15.96:0/2668523219 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ac006e3d0 0x7f2ac015b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:24.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.053+0000 7f2abea00640 1 --2- 172.21.15.96:0/2668523219 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2aa40039e0 0x7f2ac01f3c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:24.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.053+0000 7f2abf400640 1 -- 172.21.15.96:0/2668523219 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2aa40039e0 msgr2=0x7f2ac01f3c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:24.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.053+0000 7f2abf400640 1 --2- 172.21.15.96:0/2668523219 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2aa40039e0 0x7f2ac01f3c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:24.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.053+0000 7f2abf400640 1 -- 172.21.15.96:0/2668523219 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ac006e3d0 msgr2=0x7f2ac015b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:24.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.053+0000 7f2abf400640 1 --2- 172.21.15.96:0/2668523219 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ac006e3d0 0x7f2ac015b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:24.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.053+0000 7f2abf400640 1 -- 172.21.15.96:0/2668523219 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ac0153110 con 0x7f2ac015b780 2024-05-28T06:20:24.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.053+0000 7f2ab6000640 1 --2- 172.21.15.96:0/2668523219 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ac006e3d0 0x7f2ac015b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:20:24.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.053+0000 7f2abf400640 1 --2- 172.21.15.96:0/2668523219 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2ac015b780 0x7f2ac01544c0 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7f2ac00aff00 tx=0x7f2ab800b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:24.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.053+0000 7f2ab7400640 1 -- 172.21.15.96:0/2668523219 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ab800bb20 con 0x7f2ac015b780 2024-05-28T06:20:24.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.053+0000 7f2ab7400640 1 -- 172.21.15.96:0/2668523219 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ab800bcc0 con 0x7f2ac015b780 2024-05-28T06:20:24.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.053+0000 7f2ab7400640 1 -- 172.21.15.96:0/2668523219 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ab80072b0 con 0x7f2ac015b780 2024-05-28T06:20:24.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.053+0000 7f2ac5600640 1 -- 172.21.15.96:0/2668523219 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ac01533f0 con 0x7f2ac015b780 2024-05-28T06:20:24.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.053+0000 7f2ac5600640 1 -- 172.21.15.96:0/2668523219 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ac01539b0 con 0x7f2ac015b780 2024-05-28T06:20:24.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.055+0000 7f2ac5600640 1 -- 172.21.15.96:0/2668523219 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ac0149fb0 con 0x7f2ac015b780 2024-05-28T06:20:24.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.056+0000 7f2ab7400640 1 -- 172.21.15.96:0/2668523219 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 34) ==== 92927+0+0 (secure 0 0 0) 0x7f2ab801b020 con 0x7f2ac015b780 2024-05-28T06:20:24.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.057+0000 7f2ab7400640 1 --2- 172.21.15.96:0/2668523219 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2a9c06ddc0 0x7f2a9c070280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:24.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.057+0000 7f2ab7400640 1 -- 172.21.15.96:0/2668523219 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 5765+0+0 (secure 0 0 0) 0x7f2ab808aae0 con 0x7f2ac015b780 2024-05-28T06:20:24.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.057+0000 7f2abea00640 1 --2- 172.21.15.96:0/2668523219 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2a9c06ddc0 0x7f2a9c070280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:24.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.059+0000 7f2abea00640 1 --2- 172.21.15.96:0/2668523219 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2a9c06ddc0 0x7f2a9c070280 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f2ab0033000 tx=0x7f2ab0008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:24.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.061+0000 7f2ab7400640 1 -- 172.21.15.96:0/2668523219 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2ab805a9c0 con 0x7f2ac015b780 2024-05-28T06:20:24.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.332+0000 7f2ac5600640 1 -- 172.21.15.96:0/2668523219 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f2ac00689e0 con 0x7f2ac015b780 2024-05-28T06:20:24.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.337+0000 7f2ab7400640 1 -- 172.21.15.96:0/2668523219 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v21) ==== 190+0+147 (secure 0 0 0) 0x7f2ab805a2c0 con 0x7f2ac015b780 2024-05-28T06:20:24.334 INFO:teuthology.orchestra.run.smithi096.stdout:[client.0] 2024-05-28T06:20:24.334 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQCXc1Vm91hbGBAASTL3HQR8XOs7wVQEHbPl/g== 2024-05-28T06:20:24.334 INFO:teuthology.orchestra.run.smithi096.stdout: caps mds = "allow" 2024-05-28T06:20:24.334 INFO:teuthology.orchestra.run.smithi096.stdout: caps mon = "allow r" 2024-05-28T06:20:24.334 INFO:teuthology.orchestra.run.smithi096.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-05-28T06:20:24.334 INFO:teuthology.orchestra.run.smithi096.stderr:updated caps for client.0 2024-05-28T06:20:24.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.341+0000 7f2ac5600640 1 -- 172.21.15.96:0/2668523219 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2a9c06ddc0 msgr2=0x7f2a9c070280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:24.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.341+0000 7f2ac5600640 1 --2- 172.21.15.96:0/2668523219 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2a9c06ddc0 0x7f2a9c070280 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f2ab0033000 tx=0x7f2ab0008030 comp rx=0 tx=0).stop 2024-05-28T06:20:24.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.341+0000 7f2ac5600640 1 -- 172.21.15.96:0/2668523219 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2ac015b780 msgr2=0x7f2ac01544c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:24.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.341+0000 7f2ac5600640 1 --2- 172.21.15.96:0/2668523219 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2ac015b780 0x7f2ac01544c0 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7f2ac00aff00 tx=0x7f2ab800b4a0 comp rx=0 tx=0).stop 2024-05-28T06:20:24.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.341+0000 7f2ac5600640 1 -- 172.21.15.96:0/2668523219 shutdown_connections 2024-05-28T06:20:24.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.341+0000 7f2ac5600640 1 --2- 172.21.15.96:0/2668523219 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2a9c06ddc0 0x7f2a9c070280 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:24.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.341+0000 7f2ac5600640 1 --2- 172.21.15.96:0/2668523219 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2ac015b780 0x7f2ac01544c0 unknown :-1 s=CLOSED pgs=1632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:24.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.341+0000 7f2ac5600640 1 --2- 172.21.15.96:0/2668523219 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ac006e3d0 0x7f2ac015b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:24.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.341+0000 7f2ac5600640 1 --2- 172.21.15.96:0/2668523219 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2aa40039e0 0x7f2ac01f3c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:24.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.341+0000 7f2ac5600640 1 -- 172.21.15.96:0/2668523219 >> 172.21.15.96:0/2668523219 conn(0x7f2ac006bd20 msgr2=0x7f2ac00aff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:24.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.341+0000 7f2ac5600640 1 -- 172.21.15.96:0/2668523219 shutdown_connections 2024-05-28T06:20:24.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.341+0000 7f2ac5600640 1 -- 172.21.15.96:0/2668523219 wait complete. 2024-05-28T06:20:24.356 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-28T06:20:24.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.548+0000 7fd707200640 1 Processor -- start 2024-05-28T06:20:24.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.548+0000 7fd707200640 1 -- start start 2024-05-28T06:20:24.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.550+0000 7fd707200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd7080d16a0 0x7fd7080cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:24.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.550+0000 7fd707200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fd7080cf870 0x7fd7080d25e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:24.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.550+0000 7fd707200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd70815ec50 con 0x7fd70815e870 2024-05-28T06:20:24.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.550+0000 7fd707200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fd7080d2150 con 0x7fd7080cf870 2024-05-28T06:20:24.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.550+0000 7fd707200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fd7080cfd20 con 0x7fd7080d16a0 2024-05-28T06:20:24.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.550+0000 7fd706800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fd7080cf870 0x7fd7080d25e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:24.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.550+0000 7fd705e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fd70815e870 0x7fd7080d0f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:51098/0 (socket says 172.21.15.96:51098) 2024-05-28T06:20:24.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.550+0000 7fd705e00640 1 -- 172.21.15.96:0/3645617174 learned_addr learned my addr 172.21.15.96:0/3645617174 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:20:24.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.551+0000 7fd6fd400640 1 --2- 172.21.15.96:0/3645617174 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd7080d16a0 0x7fd7080cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:24.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.551+0000 7fd706800640 1 -- 172.21.15.96:0/3645617174 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd7080d16a0 msgr2=0x7fd7080cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:24.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.551+0000 7fd706800640 1 --2- 172.21.15.96:0/3645617174 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd7080d16a0 0x7fd7080cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:24.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.551+0000 7fd706800640 1 -- 172.21.15.96:0/3645617174 >> v1:172.21.15.96:6789/0 conn(0x7fd70815e870 legacy=0x7fd7080d0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:24.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.551+0000 7fd706800640 1 -- 172.21.15.96:0/3645617174 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7080cff80 con 0x7fd7080cf870 2024-05-28T06:20:24.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.551+0000 7fd6fd400640 1 --2- 172.21.15.96:0/3645617174 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd7080d16a0 0x7fd7080cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:20:24.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.552+0000 7fd706800640 1 --2- 172.21.15.96:0/3645617174 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd7080cf870 0x7fd7080d25e0 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7fd700009d90 tx=0x7fd70002a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e8ee31a53858c57 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:24.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.552+0000 7fd705400640 1 -- 172.21.15.96:0/3645617174 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd700007740 con 0x7fd7080cf870 2024-05-28T06:20:24.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.552+0000 7fd705400640 1 -- 172.21.15.96:0/3645617174 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7000078e0 con 0x7fd7080cf870 2024-05-28T06:20:24.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.552+0000 7fd705400640 1 -- 172.21.15.96:0/3645617174 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd700007cb0 con 0x7fd7080cf870 2024-05-28T06:20:24.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.552+0000 7fd707200640 1 -- 172.21.15.96:0/3645617174 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd7080cf870 msgr2=0x7fd7080d25e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:24.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.552+0000 7fd707200640 1 --2- 172.21.15.96:0/3645617174 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd7080cf870 0x7fd7080d25e0 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7fd700009d90 tx=0x7fd70002a660 comp rx=0 tx=0).stop 2024-05-28T06:20:24.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.553+0000 7fd707200640 1 -- 172.21.15.96:0/3645617174 shutdown_connections 2024-05-28T06:20:24.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.553+0000 7fd707200640 1 --2- 172.21.15.96:0/3645617174 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd7080cf870 0x7fd7080d25e0 unknown :-1 s=CLOSED pgs=1634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:24.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.553+0000 7fd707200640 1 --2- 172.21.15.96:0/3645617174 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd7080d16a0 0x7fd7080cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:24.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.553+0000 7fd707200640 1 -- 172.21.15.96:0/3645617174 >> 172.21.15.96:0/3645617174 conn(0x7fd70808bfc0 msgr2=0x7fd7080d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:24.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.553+0000 7fd707200640 1 -- 172.21.15.96:0/3645617174 shutdown_connections 2024-05-28T06:20:24.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.553+0000 7fd707200640 1 -- 172.21.15.96:0/3645617174 wait complete. 2024-05-28T06:20:24.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.554+0000 7fd707200640 1 Processor -- start 2024-05-28T06:20:24.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.554+0000 7fd707200640 1 -- start start 2024-05-28T06:20:24.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.555+0000 7fd707200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd7080cf870 0x7fd708159740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:24.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.555+0000 7fd707200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7080d16a0 0x7fd7080cc870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:24.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.555+0000 7fd707200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd70815e870 0x7fd7080b95e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:24.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.555+0000 7fd707200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd7080d0820 con 0x7fd70815e870 2024-05-28T06:20:24.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.555+0000 7fd707200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd7080d0540 con 0x7fd7080d16a0 2024-05-28T06:20:24.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.555+0000 7fd707200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fd7080d4fb0 con 0x7fd7080cf870 2024-05-28T06:20:24.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.556+0000 7fd706800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd70815e870 0x7fd7080b95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:24.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.556+0000 7fd706800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd70815e870 0x7fd7080b95e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58838/0 (socket says 172.21.15.96:58838) 2024-05-28T06:20:24.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.556+0000 7fd706800640 1 -- 172.21.15.96:0/993921730 learned_addr learned my addr 172.21.15.96:0/993921730 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:20:24.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.556+0000 7fd705e00640 1 --2- 172.21.15.96:0/993921730 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd7080cf870 0x7fd708159740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:24.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.556+0000 7fd706800640 1 -- 172.21.15.96:0/993921730 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd7080cf870 msgr2=0x7fd708159740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:24.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.556+0000 7fd706800640 1 --2- 172.21.15.96:0/993921730 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd7080cf870 0x7fd708159740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:24.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.556+0000 7fd6fd400640 1 --2- 172.21.15.96:0/993921730 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7080d16a0 0x7fd7080cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:24.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.556+0000 7fd706800640 1 -- 172.21.15.96:0/993921730 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7080d16a0 msgr2=0x7fd7080cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:24.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.556+0000 7fd706800640 1 --2- 172.21.15.96:0/993921730 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7080d16a0 0x7fd7080cc870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:24.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.556+0000 7fd706800640 1 -- 172.21.15.96:0/993921730 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7081f82e0 con 0x7fd70815e870 2024-05-28T06:20:24.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.556+0000 7fd706800640 1 --2- 172.21.15.96:0/993921730 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd70815e870 0x7fd7080b95e0 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7fd700009d90 tx=0x7fd700004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:24.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.557+0000 7fd6fea00640 1 -- 172.21.15.96:0/993921730 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd700040070 con 0x7fd70815e870 2024-05-28T06:20:24.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.557+0000 7fd6fea00640 1 -- 172.21.15.96:0/993921730 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd700031c90 con 0x7fd70815e870 2024-05-28T06:20:24.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.557+0000 7fd6fea00640 1 -- 172.21.15.96:0/993921730 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd70003c640 con 0x7fd70815e870 2024-05-28T06:20:24.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.557+0000 7fd707200640 1 -- 172.21.15.96:0/993921730 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7081f85c0 con 0x7fd70815e870 2024-05-28T06:20:24.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.557+0000 7fd707200640 1 -- 172.21.15.96:0/993921730 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7081f8b80 con 0x7fd70815e870 2024-05-28T06:20:24.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.558+0000 7fd707200640 1 -- 172.21.15.96:0/993921730 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7080689e0 con 0x7fd70815e870 2024-05-28T06:20:24.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.559+0000 7fd6fea00640 1 -- 172.21.15.96:0/993921730 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 34) ==== 92927+0+0 (secure 0 0 0) 0x7fd700039040 con 0x7fd70815e870 2024-05-28T06:20:24.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.560+0000 7fd6fea00640 1 --2- 172.21.15.96:0/993921730 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd6e406ddc0 0x7fd6e4070280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:24.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.560+0000 7fd6fea00640 1 -- 172.21.15.96:0/993921730 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 5765+0+0 (secure 0 0 0) 0x7fd7000aba80 con 0x7fd70815e870 2024-05-28T06:20:24.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.560+0000 7fd705e00640 1 --2- 172.21.15.96:0/993921730 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd6e406ddc0 0x7fd6e4070280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:24.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.562+0000 7fd705e00640 1 --2- 172.21.15.96:0/993921730 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd6e406ddc0 0x7fd6e4070280 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fd6f8000f60 tx=0x7fd6f8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:24.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.564+0000 7fd6fea00640 1 -- 172.21.15.96:0/993921730 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd700039330 con 0x7fd70815e870 2024-05-28T06:20:24.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.799+0000 7fd707200640 1 -- 172.21.15.96:0/993921730 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd7080ba500 con 0x7fd70815e870 2024-05-28T06:20:24.797 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.800+0000 7fd6fea00640 1 -- 172.21.15.96:0/993921730 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) ==== 94+0+5189 (secure 0 0 0) 0x7fd70007ba20 con 0x7fd70815e870 2024-05-28T06:20:24.798 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:20:24.798 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":62,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5861,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/1001966057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6833","nonce":1001966057},{"type":"v1","addr":"172.21.15.96:6835","nonce":1001966057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":57},{"gid":5864,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6836/4285246962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":4285246962},{"type":"v1","addr":"172.21.15.123:6836","nonce":4285246962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":57},{"gid":6445,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6834/3302807355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":3302807355},{"type":"v1","addr":"172.21.15.96:6834","nonce":3302807355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-28T06:20:18.363024+0000","modified":"2024-05-28T06:20:21.426136+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8739},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8739":{"gid":8739,"name":"b","rank":0,"incarnation":58,"state":"up:active","state_seq":3,"addr":"172.21.15.123:6837/3372660768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6835","nonce":3372660768},{"type":"v1","addr":"172.21.15.123:6837","nonce":3372660768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8739,"qdb_cluster":[8739]},"id":7}]} 2024-05-28T06:20:24.798 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 62 2024-05-28T06:20:24.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.803+0000 7fd707200640 1 -- 172.21.15.96:0/993921730 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd6e406ddc0 msgr2=0x7fd6e4070280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:24.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.803+0000 7fd707200640 1 --2- 172.21.15.96:0/993921730 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd6e406ddc0 0x7fd6e4070280 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fd6f8000f60 tx=0x7fd6f8009040 comp rx=0 tx=0).stop 2024-05-28T06:20:24.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.803+0000 7fd707200640 1 -- 172.21.15.96:0/993921730 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd70815e870 msgr2=0x7fd7080b95e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:24.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.803+0000 7fd707200640 1 --2- 172.21.15.96:0/993921730 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd70815e870 0x7fd7080b95e0 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7fd700009d90 tx=0x7fd700004a40 comp rx=0 tx=0).stop 2024-05-28T06:20:24.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.804+0000 7fd707200640 1 -- 172.21.15.96:0/993921730 shutdown_connections 2024-05-28T06:20:24.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.804+0000 7fd707200640 1 --2- 172.21.15.96:0/993921730 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd6e406ddc0 0x7fd6e4070280 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:24.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.804+0000 7fd707200640 1 --2- 172.21.15.96:0/993921730 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd70815e870 0x7fd7080b95e0 unknown :-1 s=CLOSED pgs=1635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:24.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.804+0000 7fd707200640 1 --2- 172.21.15.96:0/993921730 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7080d16a0 0x7fd7080cc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:24.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.804+0000 7fd707200640 1 --2- 172.21.15.96:0/993921730 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd7080cf870 0x7fd708159740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:24.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.804+0000 7fd707200640 1 -- 172.21.15.96:0/993921730 >> 172.21.15.96:0/993921730 conn(0x7fd70808bfc0 msgr2=0x7fd7080d2cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:24.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.804+0000 7fd707200640 1 -- 172.21.15.96:0/993921730 shutdown_connections 2024-05-28T06:20:24.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:24.804+0000 7fd707200640 1 -- 172.21.15.96:0/993921730 wait complete. 2024-05-28T06:20:24.873 INFO:tasks.ceph.mds.b.smithi123.stderr:2024-05-28T06:20:24.877+0000 7f1b51a00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-28T06:20:25.221 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 62, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-05-28T06:20:18.363024+0000', 'modified': '2024-05-28T06:20:21.426136+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8739}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8739': {'gid': 8739, 'name': 'b', 'rank': 0, 'incarnation': 58, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.123:6837/3372660768', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.123:6835', 'nonce': 3372660768}, {'type': 'v1', 'addr': '172.21.15.123:6837', 'nonce': 3372660768}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8739, 'qdb_cluster': [8739]} 2024-05-28T06:20:25.221 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-05-28T06:20:25.222 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-05-28T06:20:25.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.446+0000 7ff791600640 1 Processor -- start 2024-05-28T06:20:25.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.446+0000 7ff791600640 1 -- start start 2024-05-28T06:20:25.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.448+0000 7ff791600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff78c0d04a0 0x7ff78c0d0880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:25.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.448+0000 7ff791600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7ff78c06f6e0 0x7ff78c0cd4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:25.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.448+0000 7ff791600640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff78c0b4350 con 0x7ff78c162e70 2024-05-28T06:20:25.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.448+0000 7ff791600640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7ff78c0b3ec0 con 0x7ff78c06f6e0 2024-05-28T06:20:25.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.448+0000 7ff791600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ff78c0b32d0 con 0x7ff78c0d04a0 2024-05-28T06:20:25.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.448+0000 7ff78b400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7ff78c162e70 0x7ff78c06e0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:51108/0 (socket says 172.21.15.96:51108) 2024-05-28T06:20:25.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.448+0000 7ff78b400640 1 -- 172.21.15.96:0/1462416280 learned_addr learned my addr 172.21.15.96:0/1462416280 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:20:25.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.448+0000 7ff78a000640 1 --2- 172.21.15.96:0/1462416280 >> v2:172.21.15.123:3300/0 conn(0x7ff78c06f6e0 0x7ff78c0cd4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:25.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.448+0000 7ff78aa00640 1 --2- 172.21.15.96:0/1462416280 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff78c0d04a0 0x7ff78c0d0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:25.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.449+0000 7ff789600640 1 -- 172.21.15.96:0/1462416280 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2116670578 0 0) 0x7ff78c0b4350 con 0x7ff78c162e70 2024-05-28T06:20:25.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.449+0000 7ff789600640 1 -- 172.21.15.96:0/1462416280 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff770003520 con 0x7ff78c162e70 2024-05-28T06:20:25.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.449+0000 7ff789600640 1 -- 172.21.15.96:0/1462416280 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 65877135 0 0) 0x7ff770003520 con 0x7ff78c162e70 2024-05-28T06:20:25.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.449+0000 7ff789600640 1 -- 172.21.15.96:0/1462416280 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff78c0d04a0 msgr2=0x7ff78c0d0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:25.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.449+0000 7ff789600640 1 --2- 172.21.15.96:0/1462416280 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff78c0d04a0 0x7ff78c0d0880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:25.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.449+0000 7ff789600640 1 -- 172.21.15.96:0/1462416280 >> v2:172.21.15.123:3300/0 conn(0x7ff78c06f6e0 msgr2=0x7ff78c0cd4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:25.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.449+0000 7ff789600640 1 --2- 172.21.15.96:0/1462416280 >> v2:172.21.15.123:3300/0 conn(0x7ff78c06f6e0 0x7ff78c0cd4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:25.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.449+0000 7ff789600640 1 -- 172.21.15.96:0/1462416280 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff78c0b3590 con 0x7ff78c162e70 2024-05-28T06:20:25.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.449+0000 7ff789600640 1 -- 172.21.15.96:0/1462416280 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7ff7840019f0 con 0x7ff78c162e70 2024-05-28T06:20:25.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.449+0000 7ff789600640 1 -- 172.21.15.96:0/1462416280 >> v1:172.21.15.96:6789/0 conn(0x7ff78c162e70 legacy=0x7ff78c06e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:25.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.449+0000 7ff789600640 1 --2- 172.21.15.96:0/1462416280 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff7700039b0 0x7ff770003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:25.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.449+0000 7ff789600640 1 --2- 172.21.15.96:0/1462416280 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff770004800 0x7ff770004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:25.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.449+0000 7ff789600640 1 --2- 172.21.15.96:0/1462416280 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff7700051c0 0x7ff77000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:25.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.449+0000 7ff789600640 1 -- 172.21.15.96:0/1462416280 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff78c0b3ec0 con 0x7ff770004800 2024-05-28T06:20:25.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.449+0000 7ff789600640 1 -- 172.21.15.96:0/1462416280 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff78c0b32d0 con 0x7ff7700051c0 2024-05-28T06:20:25.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.449+0000 7ff789600640 1 -- 172.21.15.96:0/1462416280 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ff77000c000 con 0x7ff7700039b0 2024-05-28T06:20:25.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.449+0000 7ff78a000640 1 --2- 172.21.15.96:0/1462416280 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff770004800 0x7ff770004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:25.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.450+0000 7ff78b400640 1 --2- 172.21.15.96:0/1462416280 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff7700051c0 0x7ff77000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:25.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.450+0000 7ff78aa00640 1 --2- 172.21.15.96:0/1462416280 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff7700039b0 0x7ff770003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:25.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.450+0000 7ff78a000640 1 -- 172.21.15.96:0/1462416280 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff7700039b0 msgr2=0x7ff770003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:25.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.450+0000 7ff78a000640 1 --2- 172.21.15.96:0/1462416280 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff7700039b0 0x7ff770003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:25.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.450+0000 7ff78a000640 1 -- 172.21.15.96:0/1462416280 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff7700051c0 msgr2=0x7ff77000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:25.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.450+0000 7ff78a000640 1 --2- 172.21.15.96:0/1462416280 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff7700051c0 0x7ff77000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:25.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.450+0000 7ff78a000640 1 -- 172.21.15.96:0/1462416280 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff770003520 con 0x7ff770004800 2024-05-28T06:20:25.451 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.450+0000 7ff78aa00640 1 --2- 172.21.15.96:0/1462416280 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff7700039b0 0x7ff770003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:20:25.451 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.451+0000 7ff78a000640 1 --2- 172.21.15.96:0/1462416280 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff770004800 0x7ff770004c80 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7ff77800ba40 tx=0x7ff770005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5efbf4ac633747e server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:25.451 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.452+0000 7ff789600640 1 -- 172.21.15.96:0/1462416280 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff778030070 con 0x7ff770004800 2024-05-28T06:20:25.451 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.452+0000 7ff789600640 1 -- 172.21.15.96:0/1462416280 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff77802ae10 con 0x7ff770004800 2024-05-28T06:20:25.451 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.452+0000 7ff789600640 1 -- 172.21.15.96:0/1462416280 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff77800b610 con 0x7ff770004800 2024-05-28T06:20:25.454 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.452+0000 7ff791600640 1 -- 172.21.15.96:0/1462416280 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff770004800 msgr2=0x7ff770004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:25.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.452+0000 7ff791600640 1 --2- 172.21.15.96:0/1462416280 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff770004800 0x7ff770004c80 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7ff77800ba40 tx=0x7ff770005830 comp rx=0 tx=0).stop 2024-05-28T06:20:25.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.452+0000 7ff78b400640 1 -- 172.21.15.96:0/1462416280 reap_dead start 2024-05-28T06:20:25.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.452+0000 7ff791600640 1 -- 172.21.15.96:0/1462416280 shutdown_connections 2024-05-28T06:20:25.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.452+0000 7ff791600640 1 -- 172.21.15.96:0/1462416280 >> 172.21.15.96:0/1462416280 conn(0x7ff78c06bae0 msgr2=0x7ff78c06bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:25.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.452+0000 7ff791600640 1 -- 172.21.15.96:0/1462416280 shutdown_connections 2024-05-28T06:20:25.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.452+0000 7ff791600640 1 -- 172.21.15.96:0/1462416280 wait complete. 2024-05-28T06:20:25.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.454+0000 7ff791600640 1 Processor -- start 2024-05-28T06:20:25.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.454+0000 7ff791600640 1 -- start start 2024-05-28T06:20:25.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.455+0000 7ff791600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff770004800 0x7ff78c163bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:25.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.455+0000 7ff791600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff78c15a1c0 0x7ff78c0c5b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:25.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.455+0000 7ff791600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff78c15a640 0x7ff78c0c6090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:25.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.455+0000 7ff791600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff78c0cda30 con 0x7ff770004800 2024-05-28T06:20:25.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.455+0000 7ff791600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff78c0b3b50 con 0x7ff78c15a640 2024-05-28T06:20:25.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.455+0000 7ff791600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ff78c0cf240 con 0x7ff78c15a1c0 2024-05-28T06:20:25.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.455+0000 7ff78aa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff770004800 0x7ff78c163bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:25.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.456+0000 7ff78aa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff770004800 0x7ff78c163bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58866/0 (socket says 172.21.15.96:58866) 2024-05-28T06:20:25.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.456+0000 7ff78aa00640 1 -- 172.21.15.96:0/3457003582 learned_addr learned my addr 172.21.15.96:0/3457003582 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:20:25.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.456+0000 7ff78b400640 1 --2- 172.21.15.96:0/3457003582 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff78c15a640 0x7ff78c0c6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:25.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.456+0000 7ff78a000640 1 --2- 172.21.15.96:0/3457003582 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff78c15a1c0 0x7ff78c0c5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:25.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.456+0000 7ff78aa00640 1 -- 172.21.15.96:0/3457003582 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff78c15a1c0 msgr2=0x7ff78c0c5b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:25.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.456+0000 7ff78aa00640 1 --2- 172.21.15.96:0/3457003582 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff78c15a1c0 0x7ff78c0c5b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:25.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.456+0000 7ff78aa00640 1 -- 172.21.15.96:0/3457003582 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff78c15a640 msgr2=0x7ff78c0c6090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:25.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.456+0000 7ff78aa00640 1 --2- 172.21.15.96:0/3457003582 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff78c15a640 0x7ff78c0c6090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:25.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.456+0000 7ff78aa00640 1 -- 172.21.15.96:0/3457003582 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff78c0c4ab0 con 0x7ff770004800 2024-05-28T06:20:25.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.456+0000 7ff78a000640 1 --2- 172.21.15.96:0/3457003582 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff78c15a1c0 0x7ff78c0c5b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:20:25.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.456+0000 7ff78b400640 1 --2- 172.21.15.96:0/3457003582 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff78c15a640 0x7ff78c0c6090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:20:25.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.456+0000 7ff78aa00640 1 --2- 172.21.15.96:0/3457003582 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff770004800 0x7ff78c163bc0 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7ff77c00e7c0 tx=0x7ff77c00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:25.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.457+0000 7ff782a00640 1 -- 172.21.15.96:0/3457003582 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff77c017070 con 0x7ff770004800 2024-05-28T06:20:25.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.457+0000 7ff782a00640 1 -- 172.21.15.96:0/3457003582 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff77c00c6d0 con 0x7ff770004800 2024-05-28T06:20:25.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.457+0000 7ff782a00640 1 -- 172.21.15.96:0/3457003582 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff77c00f040 con 0x7ff770004800 2024-05-28T06:20:25.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.457+0000 7ff791600640 1 -- 172.21.15.96:0/3457003582 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff78c0c4d90 con 0x7ff770004800 2024-05-28T06:20:25.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.457+0000 7ff791600640 1 -- 172.21.15.96:0/3457003582 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff78c0c5350 con 0x7ff770004800 2024-05-28T06:20:25.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.459+0000 7ff782a00640 1 -- 172.21.15.96:0/3457003582 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 34) ==== 92927+0+0 (secure 0 0 0) 0x7ff77c01a070 con 0x7ff770004800 2024-05-28T06:20:25.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.460+0000 7ff782a00640 1 --2- 172.21.15.96:0/3457003582 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff76c06ddc0 0x7ff76c070280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:25.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.460+0000 7ff782a00640 1 -- 172.21.15.96:0/3457003582 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 5765+0+0 (secure 0 0 0) 0x7ff77c08a390 con 0x7ff770004800 2024-05-28T06:20:25.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.461+0000 7ff79441b740 1 Processor -- start 2024-05-28T06:20:25.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.461+0000 7ff78a000640 1 --2- 172.21.15.96:0/3457003582 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff76c06ddc0 0x7ff76c070280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:25.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.461+0000 7ff79441b740 1 -- start start 2024-05-28T06:20:25.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.462+0000 7ff79441b740 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x564bf1d072a0 0x564bf1d0aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:25.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.462+0000 7ff79441b740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564bf1d0af60 0x564bf1d26ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:25.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.462+0000 7ff79441b740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564bf1d273e0 0x564bf1d27840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:25.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.462+0000 7ff79441b740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x564bf1c44b10 con 0x564bf1d273e0 2024-05-28T06:20:25.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.462+0000 7ff79441b740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x564bf1beb2b0 con 0x564bf1d0af60 2024-05-28T06:20:25.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.462+0000 7ff79441b740 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x564bf1c40ad0 con 0x564bf1d072a0 2024-05-28T06:20:25.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.462+0000 7ff78b400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564bf1d273e0 0x564bf1d27840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:25.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.462+0000 7ff78b400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564bf1d273e0 0x564bf1d27840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58872/0 (socket says 172.21.15.96:58872) 2024-05-28T06:20:25.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.462+0000 7ff78b400640 1 -- 172.21.15.96:0/2793859744 learned_addr learned my addr 172.21.15.96:0/2793859744 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:20:25.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.462+0000 7ff78a000640 1 --2- 172.21.15.96:0/3457003582 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff76c06ddc0 0x7ff76c070280 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7ff77800ba10 tx=0x7ff778030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:25.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.462+0000 7ff78aa00640 1 --2- 172.21.15.96:0/2793859744 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x564bf1d072a0 0x564bf1d0aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:25.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.462+0000 7ff78b400640 1 -- 172.21.15.96:0/2793859744 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x564bf1d072a0 msgr2=0x564bf1d0aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:25.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.462+0000 7ff78a000640 1 --2- 172.21.15.96:0/2793859744 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564bf1d0af60 0x564bf1d26ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:25.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.462+0000 7ff78b400640 1 --2- 172.21.15.96:0/2793859744 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x564bf1d072a0 0x564bf1d0aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:25.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.463+0000 7ff78b400640 1 -- 172.21.15.96:0/2793859744 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564bf1d0af60 msgr2=0x564bf1d26ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:25.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.463+0000 7ff78b400640 1 --2- 172.21.15.96:0/2793859744 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564bf1d0af60 0x564bf1d26ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:25.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.463+0000 7ff78b400640 1 -- 172.21.15.96:0/2793859744 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564bf19572d0 con 0x564bf1d273e0 2024-05-28T06:20:25.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.463+0000 7ff78b400640 1 --2- 172.21.15.96:0/2793859744 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564bf1d273e0 0x564bf1d27840 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7ff784000f80 tx=0x7ff784007420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d4665b7bc675157 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:25.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.463+0000 7ff777000640 1 -- 172.21.15.96:0/2793859744 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff784019070 con 0x564bf1d273e0 2024-05-28T06:20:25.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.463+0000 7ff777000640 1 -- 172.21.15.96:0/2793859744 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff784014630 con 0x564bf1d273e0 2024-05-28T06:20:25.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.463+0000 7ff777000640 1 -- 172.21.15.96:0/2793859744 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff784010040 con 0x564bf1d273e0 2024-05-28T06:20:25.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.464+0000 7ff79441b740 1 -- 172.21.15.96:0/2793859744 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564bf1d273e0 msgr2=0x564bf1d27840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:25.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.464+0000 7ff79441b740 1 --2- 172.21.15.96:0/2793859744 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564bf1d273e0 0x564bf1d27840 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7ff784000f80 tx=0x7ff784007420 comp rx=0 tx=0).stop 2024-05-28T06:20:25.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.464+0000 7ff79441b740 1 -- 172.21.15.96:0/2793859744 shutdown_connections 2024-05-28T06:20:25.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.464+0000 7ff79441b740 1 --2- 172.21.15.96:0/2793859744 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564bf1d273e0 0x564bf1d27840 unknown :-1 s=CLOSED pgs=1639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:25.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.464+0000 7ff79441b740 1 --2- 172.21.15.96:0/2793859744 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564bf1d0af60 0x564bf1d26ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:25.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.464+0000 7ff79441b740 1 --2- 172.21.15.96:0/2793859744 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x564bf1d072a0 0x564bf1d0aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:25.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.464+0000 7ff79441b740 1 -- 172.21.15.96:0/2793859744 >> 172.21.15.96:0/2793859744 conn(0x564bf1cdd280 msgr2=0x564bf1cde280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:25.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.464+0000 7ff79441b740 1 -- 172.21.15.96:0/2793859744 shutdown_connections 2024-05-28T06:20:25.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.464+0000 7ff79441b740 1 -- 172.21.15.96:0/2793859744 wait complete. 2024-05-28T06:20:25.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.465+0000 7ff79441b740 1 -- start start 2024-05-28T06:20:25.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.465+0000 7ff79441b740 1 Processor -- start 2024-05-28T06:20:25.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.466+0000 7ff79441b740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564bf1d0af60 0x564bf1d090e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:25.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.466+0000 7ff79441b740 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x564bf1d273e0 0x564bf1d21260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:25.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.466+0000 7ff79441b740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564bf1d26330 0x564bf1d217a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:25.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.466+0000 7ff79441b740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x564bf1cdb0b0 con 0x564bf1d0af60 2024-05-28T06:20:25.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.466+0000 7ff79441b740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x564bf1bfe570 con 0x564bf1d26330 2024-05-28T06:20:25.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.466+0000 7ff79441b740 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x564bf1cdb390 con 0x564bf1d273e0 2024-05-28T06:20:25.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.466+0000 7ff79441b740 10 client.0 _ll_register_callbacks cb 0x564bf1bccfc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-28T06:20:25.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.466+0000 7ff78aa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564bf1d0af60 0x564bf1d090e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:25.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.466+0000 7ff78aa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564bf1d0af60 0x564bf1d090e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58884/0 (socket says 172.21.15.96:58884) 2024-05-28T06:20:25.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.466+0000 7ff78aa00640 1 -- 172.21.15.96:0/3123254136 learned_addr learned my addr 172.21.15.96:0/3123254136 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:20:25.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.466+0000 7ff78aa00640 1 -- 172.21.15.96:0/3123254136 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x564bf1d273e0 msgr2=0x564bf1d21260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:25.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.466+0000 7ff78aa00640 1 --2- 172.21.15.96:0/3123254136 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x564bf1d273e0 0x564bf1d21260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:25.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.466+0000 7ff78aa00640 1 -- 172.21.15.96:0/3123254136 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564bf1d26330 msgr2=0x564bf1d217a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:25.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.466+0000 7ff78aa00640 1 --2- 172.21.15.96:0/3123254136 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564bf1d26330 0x564bf1d217a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:25.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.466+0000 7ff78aa00640 1 -- 172.21.15.96:0/3123254136 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564bf1a216b0 con 0x564bf1d0af60 2024-05-28T06:20:25.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.467+0000 7ff78aa00640 1 --2- 172.21.15.96:0/3123254136 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564bf1d0af60 0x564bf1d090e0 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7ff77c0621c0 tx=0x7ff77c059b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8008b368f654a494 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:25.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.467+0000 7ff76b400640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-05-28T06:20:25.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.467+0000 7ff76b400640 1 -- 172.21.15.96:0/3123254136 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff77c088060 con 0x564bf1d0af60 2024-05-28T06:20:25.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.467+0000 7ff76b400640 1 -- 172.21.15.96:0/3123254136 <== mon.0 v2:172.21.15.96:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 5765+0+0 (secure 0 0 0) 0x7ff77c059160 con 0x564bf1d0af60 2024-05-28T06:20:25.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.467+0000 7ff79441b740 1 -- 172.21.15.96:0/3123254136 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff784014630 con 0x564bf1d0af60 2024-05-28T06:20:25.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.467+0000 7ff76b400640 1 -- 172.21.15.96:0/3123254136 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7ff77c072070 con 0x564bf1d0af60 2024-05-28T06:20:25.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.467+0000 7ff79441b740 10 client.8865 fetch_fsmap learned FSMap version 62 2024-05-28T06:20:25.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.467+0000 7ff79441b740 1 -- 172.21.15.96:0/3123254136 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=2+}) -- 0x564bf1d03960 con 0x564bf1d0af60 2024-05-28T06:20:25.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.467+0000 7ff76b400640 1 -- 172.21.15.96:0/3123254136 <== mon.0 v2:172.21.15.96:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff784014630 con 0x564bf1d0af60 2024-05-28T06:20:25.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.468+0000 7ff76b400640 1 -- 172.21.15.96:0/3123254136 <== mon.0 v2:172.21.15.96:3300/0 5 ==== fsmap(e 62) ==== 3494+0+0 (secure 0 0 0) 0x7ff77c073020 con 0x564bf1d0af60 2024-05-28T06:20:25.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.468+0000 7ff79441b740 10 client.8865 fetch_fsmap finished waiting for FSMap version 62 2024-05-28T06:20:25.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.468+0000 7ff79441b740 10 client.8865 resolve_mds: resolved name 'b' to daemon mds.b 2024-05-28T06:20:25.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.468+0000 7ff79441b740 20 client.8865 populate_metadata read hostname 'smithi096' 2024-05-28T06:20:25.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.468+0000 7ff79441b740 1 --2- 172.21.15.96:0/3123254136 >> [v2:172.21.15.123:6835/3372660768,v1:172.21.15.123:6837/3372660768] conn(0x564bf1d0cfb0 0x564bf1d43a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:25.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.468+0000 7ff79441b740 4 client.8865 mds_command: new command op to 8739 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-28T06:20:25.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.468+0000 7ff79441b740 1 -- 172.21.15.96:0/3123254136 --> [v2:172.21.15.123:6835/3372660768,v1:172.21.15.123:6837/3372660768] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564bf1c52530 con 0x564bf1d0cfb0 2024-05-28T06:20:25.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.468+0000 7ff78a000640 1 --2- 172.21.15.96:0/3123254136 >> [v2:172.21.15.123:6835/3372660768,v1:172.21.15.123:6837/3372660768] conn(0x564bf1d0cfb0 0x564bf1d43a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:25.469 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.469+0000 7ff78a000640 1 --2- 172.21.15.96:0/3123254136 >> [v2:172.21.15.123:6835/3372660768,v1:172.21.15.123:6837/3372660768] conn(0x564bf1d0cfb0 0x564bf1d43a40 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=37b41fe3c663a058 server_cookie=55e07c85b29def6c in_seq=0 out_seq=0 2024-05-28T06:20:25.469 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.469+0000 7ff76b400640 10 client.8865 ms_handle_connect on v2:172.21.15.123:6835/3372660768 2024-05-28T06:20:25.469 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.471+0000 7ff76b400640 1 -- 172.21.15.96:0/3123254136 <== mds.0 v2:172.21.15.123:6835/3372660768 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564bf1c52530 con 0x564bf1d0cfb0 2024-05-28T06:20:25.469 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.471+0000 7ff76b400640 10 client.8865 handle_command_reply: tid=0 2024-05-28T06:20:25.470 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.471+0000 7ff79441b740 1 client.8865 shutdown 2024-05-28T06:20:25.470 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.472+0000 7ff79441b740 1 -- 172.21.15.96:0/3123254136 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564bf1d0af60 msgr2=0x564bf1d090e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:25.470 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.472+0000 7ff79441b740 1 --2- 172.21.15.96:0/3123254136 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564bf1d0af60 0x564bf1d090e0 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7ff77c0621c0 tx=0x7ff77c059b10 comp rx=0 tx=0).stop 2024-05-28T06:20:25.470 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.472+0000 7ff79441b740 1 -- 172.21.15.96:0/3123254136 shutdown_connections 2024-05-28T06:20:25.470 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.472+0000 7ff79441b740 1 --2- 172.21.15.96:0/3123254136 >> [v2:172.21.15.123:6835/3372660768,v1:172.21.15.123:6837/3372660768] conn(0x564bf1d0cfb0 0x564bf1d43a40 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:25.470 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.472+0000 7ff79441b740 1 --2- 172.21.15.96:0/3123254136 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564bf1d26330 0x564bf1d217a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:25.470 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.472+0000 7ff79441b740 1 --2- 172.21.15.96:0/3123254136 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x564bf1d273e0 0x564bf1d21260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:25.470 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.472+0000 7ff79441b740 1 --2- 172.21.15.96:0/3123254136 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564bf1d0af60 0x564bf1d090e0 unknown :-1 s=CLOSED pgs=1640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:25.470 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.472+0000 7ff79441b740 1 -- 172.21.15.96:0/3123254136 >> 172.21.15.96:0/3123254136 conn(0x564bf1d072a0 msgr2=0x564bf1cd9ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:25.470 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.472+0000 7ff76b400640 0 client.8865 ms_handle_reset on v2:172.21.15.123:6835/3372660768 2024-05-28T06:20:25.470 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.472+0000 7ff79441b740 1 -- 172.21.15.96:0/3123254136 shutdown_connections 2024-05-28T06:20:25.470 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.472+0000 7ff79441b740 1 -- 172.21.15.96:0/3123254136 wait complete. 2024-05-28T06:20:25.470 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.472+0000 7ff79441b740 20 client.8865 trim_cache size 0 max 16384 2024-05-28T06:20:25.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.491+0000 7ff79441b740 1 Processor -- start 2024-05-28T06:20:25.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.491+0000 7ff79441b740 1 -- start start 2024-05-28T06:20:25.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.492+0000 7ff79441b740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564bf1d0af60 0x564bf1cde7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:25.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.492+0000 7ff79441b740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564bf1d26330 0x564bf1d21120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:25.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.492+0000 7ff79441b740 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x564bf1d273e0 0x564bf1d21660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:25.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.492+0000 7ff79441b740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff78c204a20 con 0x564bf1d0af60 2024-05-28T06:20:25.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.492+0000 7ff79441b740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff78c161270 con 0x564bf1d26330 2024-05-28T06:20:25.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.492+0000 7ff79441b740 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ff78c1618c0 con 0x564bf1d273e0 2024-05-28T06:20:25.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.492+0000 7ff78aa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564bf1d0af60 0x564bf1cde7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:25.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.492+0000 7ff78aa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564bf1d0af60 0x564bf1cde7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58898/0 (socket says 172.21.15.96:58898) 2024-05-28T06:20:25.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.492+0000 7ff78aa00640 1 -- 172.21.15.96:0/2197638313 learned_addr learned my addr 172.21.15.96:0/2197638313 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:20:25.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.492+0000 7ff78aa00640 1 -- 172.21.15.96:0/2197638313 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x564bf1d273e0 msgr2=0x564bf1d21660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:25.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.492+0000 7ff78aa00640 1 --2- 172.21.15.96:0/2197638313 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x564bf1d273e0 0x564bf1d21660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:25.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.492+0000 7ff78aa00640 1 -- 172.21.15.96:0/2197638313 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564bf1d26330 msgr2=0x564bf1d21120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:25.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.492+0000 7ff78aa00640 1 --2- 172.21.15.96:0/2197638313 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564bf1d26330 0x564bf1d21120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:25.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.492+0000 7ff78aa00640 1 -- 172.21.15.96:0/2197638313 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564bf1ce0820 con 0x564bf1d0af60 2024-05-28T06:20:25.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.493+0000 7ff78aa00640 1 --2- 172.21.15.96:0/2197638313 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564bf1d0af60 0x564bf1cde7a0 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7ff77c05e080 tx=0x7ff77c014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1026e6de9362ea09 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:25.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.493+0000 7ff776600640 1 -- 172.21.15.96:0/2197638313 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff77c07d020 con 0x564bf1d0af60 2024-05-28T06:20:25.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.493+0000 7ff776600640 1 -- 172.21.15.96:0/2197638313 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff77c055a60 con 0x564bf1d0af60 2024-05-28T06:20:25.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.493+0000 7ff776600640 1 -- 172.21.15.96:0/2197638313 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff77c021040 con 0x564bf1d0af60 2024-05-28T06:20:25.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.493+0000 7ff79441b740 1 -- 172.21.15.96:0/2197638313 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564bf1d0af60 msgr2=0x564bf1cde7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:25.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.493+0000 7ff79441b740 1 --2- 172.21.15.96:0/2197638313 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564bf1d0af60 0x564bf1cde7a0 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7ff77c05e080 tx=0x7ff77c014000 comp rx=0 tx=0).stop 2024-05-28T06:20:25.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.493+0000 7ff79441b740 1 -- 172.21.15.96:0/2197638313 shutdown_connections 2024-05-28T06:20:25.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.493+0000 7ff79441b740 1 --2- 172.21.15.96:0/2197638313 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x564bf1d273e0 0x564bf1d21660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:25.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.493+0000 7ff79441b740 1 --2- 172.21.15.96:0/2197638313 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564bf1d26330 0x564bf1d21120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:25.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.493+0000 7ff79441b740 1 --2- 172.21.15.96:0/2197638313 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564bf1d0af60 0x564bf1cde7a0 unknown :-1 s=CLOSED pgs=1641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:25.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.493+0000 7ff79441b740 1 -- 172.21.15.96:0/2197638313 >> 172.21.15.96:0/2197638313 conn(0x564bf1d072a0 msgr2=0x564bf1cde050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:25.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.494+0000 7ff79441b740 1 -- 172.21.15.96:0/2197638313 shutdown_connections 2024-05-28T06:20:25.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.494+0000 7ff79441b740 1 -- 172.21.15.96:0/2197638313 wait complete. 2024-05-28T06:20:25.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.494+0000 7ff79441b740 1 -- start start 2024-05-28T06:20:25.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.494+0000 7ff79441b740 1 Processor -- start 2024-05-28T06:20:25.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.495+0000 7ff79441b740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564bf1d0af60 0x564bf1d06270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:25.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.495+0000 7ff79441b740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564bf1d26330 0x564bf1d067b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:25.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.495+0000 7ff79441b740 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x564bf1d273e0 0x564bf1d47c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:25.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.495+0000 7ff79441b740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x564bf1d0ce30 con 0x564bf1d0af60 2024-05-28T06:20:25.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.495+0000 7ff79441b740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x564bf1d0c990 con 0x564bf1d26330 2024-05-28T06:20:25.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.495+0000 7ff79441b740 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x564bf1d0d440 con 0x564bf1d273e0 2024-05-28T06:20:25.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.495+0000 7ff79441b740 10 client.0 _ll_register_callbacks cb 0x7ff77c0594e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-28T06:20:25.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.495+0000 7ff78aa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564bf1d0af60 0x564bf1d06270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:25.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.496+0000 7ff78aa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564bf1d0af60 0x564bf1d06270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58908/0 (socket says 172.21.15.96:58908) 2024-05-28T06:20:25.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.496+0000 7ff78aa00640 1 -- 172.21.15.96:0/2121092028 learned_addr learned my addr 172.21.15.96:0/2121092028 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:20:25.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.496+0000 7ff78aa00640 1 -- 172.21.15.96:0/2121092028 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x564bf1d273e0 msgr2=0x564bf1d47c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:25.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.496+0000 7ff78a000640 1 --2- 172.21.15.96:0/2121092028 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564bf1d26330 0x564bf1d067b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:25.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.496+0000 7ff78b400640 1 --2- 172.21.15.96:0/2121092028 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x564bf1d273e0 0x564bf1d47c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:25.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.496+0000 7ff78aa00640 1 --2- 172.21.15.96:0/2121092028 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x564bf1d273e0 0x564bf1d47c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:25.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.496+0000 7ff78aa00640 1 -- 172.21.15.96:0/2121092028 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564bf1d26330 msgr2=0x564bf1d067b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:25.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.496+0000 7ff78aa00640 1 --2- 172.21.15.96:0/2121092028 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564bf1d26330 0x564bf1d067b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:25.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.496+0000 7ff78aa00640 1 -- 172.21.15.96:0/2121092028 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564bf1d49630 con 0x564bf1d0af60 2024-05-28T06:20:25.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.496+0000 7ff78aa00640 1 --2- 172.21.15.96:0/2121092028 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564bf1d0af60 0x564bf1d06270 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7ff77c056490 tx=0x7ff77c05c070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df28f234f5ca97b7 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:25.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.496+0000 7ff76b400640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-05-28T06:20:25.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.496+0000 7ff76b400640 1 -- 172.21.15.96:0/2121092028 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff77c07d050 con 0x564bf1d0af60 2024-05-28T06:20:25.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.496+0000 7ff76b400640 1 -- 172.21.15.96:0/2121092028 <== mon.0 v2:172.21.15.96:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 5765+0+0 (secure 0 0 0) 0x7ff77c059230 con 0x564bf1d0af60 2024-05-28T06:20:25.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.496+0000 7ff79441b740 1 -- 172.21.15.96:0/2121092028 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff77c055a60 con 0x564bf1d0af60 2024-05-28T06:20:25.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.497+0000 7ff76b400640 1 -- 172.21.15.96:0/2121092028 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7ff77c021070 con 0x564bf1d0af60 2024-05-28T06:20:25.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.497+0000 7ff79441b740 10 client.8871 fetch_fsmap learned FSMap version 62 2024-05-28T06:20:25.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.497+0000 7ff79441b740 1 -- 172.21.15.96:0/2121092028 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=2+}) -- 0x564bf1d403e0 con 0x564bf1d0af60 2024-05-28T06:20:25.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.497+0000 7ff76b400640 1 -- 172.21.15.96:0/2121092028 <== mon.0 v2:172.21.15.96:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff77c055a60 con 0x564bf1d0af60 2024-05-28T06:20:25.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.497+0000 7ff76b400640 1 -- 172.21.15.96:0/2121092028 <== mon.0 v2:172.21.15.96:3300/0 5 ==== fsmap(e 62) ==== 3494+0+0 (secure 0 0 0) 0x7ff77c073040 con 0x564bf1d0af60 2024-05-28T06:20:25.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.497+0000 7ff79441b740 10 client.8871 fetch_fsmap finished waiting for FSMap version 62 2024-05-28T06:20:25.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.497+0000 7ff79441b740 10 client.8871 resolve_mds: resolved name 'b' to daemon mds.b 2024-05-28T06:20:25.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.497+0000 7ff79441b740 20 client.8871 populate_metadata read hostname 'smithi096' 2024-05-28T06:20:25.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.497+0000 7ff79441b740 1 --2- 172.21.15.96:0/2121092028 >> [v2:172.21.15.123:6835/3372660768,v1:172.21.15.123:6837/3372660768] conn(0x564bf1d48210 0x564bf1d1b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:25.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.497+0000 7ff79441b740 4 client.8871 mds_command: new command op to 8739 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-28T06:20:25.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.497+0000 7ff79441b740 1 -- 172.21.15.96:0/2121092028 --> [v2:172.21.15.123:6835/3372660768,v1:172.21.15.123:6837/3372660768] -- command(tid 0: {"prefix": "status"}) -- 0x564bf1c4c3a0 con 0x564bf1d48210 2024-05-28T06:20:25.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.498+0000 7ff78a000640 1 --2- 172.21.15.96:0/2121092028 >> [v2:172.21.15.123:6835/3372660768,v1:172.21.15.123:6837/3372660768] conn(0x564bf1d48210 0x564bf1d1b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:25.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.499+0000 7ff78a000640 1 --2- 172.21.15.96:0/2121092028 >> [v2:172.21.15.123:6835/3372660768,v1:172.21.15.123:6837/3372660768] conn(0x564bf1d48210 0x564bf1d1b3b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=13e6ebc66b6710f2 server_cookie=65108794ac29de56 in_seq=0 out_seq=0 2024-05-28T06:20:25.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.499+0000 7ff76b400640 10 client.8871 ms_handle_connect on v2:172.21.15.123:6835/3372660768 2024-05-28T06:20:25.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.499+0000 7ff76b400640 1 -- 172.21.15.96:0/2121092028 <== mds.0 v2:172.21.15.123:6835/3372660768 1 ==== command_reply(tid 0: 0 ) ==== 8+0+333 (crc 0 0 0) 0x564bf1c4c3a0 con 0x564bf1d48210 2024-05-28T06:20:25.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.499+0000 7ff76b400640 10 client.8871 handle_command_reply: tid=0 2024-05-28T06:20:25.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.499+0000 7ff79441b740 1 client.8871 shutdown 2024-05-28T06:20:25.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.499+0000 7ff79441b740 1 -- 172.21.15.96:0/2121092028 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564bf1d0af60 msgr2=0x564bf1d06270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:25.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.499+0000 7ff79441b740 1 --2- 172.21.15.96:0/2121092028 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564bf1d0af60 0x564bf1d06270 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7ff77c056490 tx=0x7ff77c05c070 comp rx=0 tx=0).stop 2024-05-28T06:20:25.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.500+0000 7ff79441b740 1 -- 172.21.15.96:0/2121092028 shutdown_connections 2024-05-28T06:20:25.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.500+0000 7ff79441b740 1 --2- 172.21.15.96:0/2121092028 >> [v2:172.21.15.123:6835/3372660768,v1:172.21.15.123:6837/3372660768] conn(0x564bf1d48210 0x564bf1d1b3b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:25.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.500+0000 7ff79441b740 1 --2- 172.21.15.96:0/2121092028 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x564bf1d273e0 0x564bf1d47c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:25.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.500+0000 7ff79441b740 1 --2- 172.21.15.96:0/2121092028 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564bf1d26330 0x564bf1d067b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:25.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.500+0000 7ff76b400640 0 client.8871 ms_handle_reset on v2:172.21.15.123:6835/3372660768 2024-05-28T06:20:25.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.500+0000 7ff79441b740 1 --2- 172.21.15.96:0/2121092028 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564bf1d0af60 0x564bf1d06270 unknown :-1 s=CLOSED pgs=1642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:25.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.500+0000 7ff79441b740 1 -- 172.21.15.96:0/2121092028 >> 172.21.15.96:0/2121092028 conn(0x564bf1d072a0 msgr2=0x564bf1ce0c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:25.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.500+0000 7ff79441b740 1 -- 172.21.15.96:0/2121092028 shutdown_connections 2024-05-28T06:20:25.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.500+0000 7ff79441b740 1 -- 172.21.15.96:0/2121092028 wait complete. 2024-05-28T06:20:25.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.500+0000 7ff79441b740 20 client.8871 trim_cache size 0 max 16384 2024-05-28T06:20:25.501 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:20:25.501 INFO:teuthology.orchestra.run.smithi096.stdout: "cluster_fsid": "e4a831b5-e6d1-47a7-96cb-bf85154ee0d2", 2024-05-28T06:20:25.501 INFO:teuthology.orchestra.run.smithi096.stdout: "whoami": 0, 2024-05-28T06:20:25.501 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 8739, 2024-05-28T06:20:25.501 INFO:teuthology.orchestra.run.smithi096.stdout: "want_state": "up:active", 2024-05-28T06:20:25.501 INFO:teuthology.orchestra.run.smithi096.stdout: "state": "up:active", 2024-05-28T06:20:25.501 INFO:teuthology.orchestra.run.smithi096.stdout: "fs_name": "cephfs", 2024-05-28T06:20:25.501 INFO:teuthology.orchestra.run.smithi096.stdout: "rank_uptime": 6.5716837760000004, 2024-05-28T06:20:25.501 INFO:teuthology.orchestra.run.smithi096.stdout: "mdsmap_epoch": 62, 2024-05-28T06:20:25.501 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap_epoch": 103, 2024-05-28T06:20:25.501 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap_epoch_barrier": 103, 2024-05-28T06:20:25.502 INFO:teuthology.orchestra.run.smithi096.stdout: "uptime": 12.035128115999999 2024-05-28T06:20:25.502 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:20:25.502 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.500+0000 7ff791600640 1 -- 172.21.15.96:0/3457003582 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff76c06ddc0 msgr2=0x7ff76c070280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:25.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.500+0000 7ff791600640 1 --2- 172.21.15.96:0/3457003582 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff76c06ddc0 0x7ff76c070280 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7ff77800ba10 tx=0x7ff778030000 comp rx=0 tx=0).stop 2024-05-28T06:20:25.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.500+0000 7ff791600640 1 -- 172.21.15.96:0/3457003582 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff770004800 msgr2=0x7ff78c163bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:25.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.500+0000 7ff791600640 1 --2- 172.21.15.96:0/3457003582 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff770004800 0x7ff78c163bc0 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7ff77c00e7c0 tx=0x7ff77c00ec80 comp rx=0 tx=0).stop 2024-05-28T06:20:25.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.500+0000 7ff791600640 1 -- 172.21.15.96:0/3457003582 shutdown_connections 2024-05-28T06:20:25.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.500+0000 7ff791600640 1 --2- 172.21.15.96:0/3457003582 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff76c06ddc0 0x7ff76c070280 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:25.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.500+0000 7ff791600640 1 --2- 172.21.15.96:0/3457003582 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff78c15a640 0x7ff78c0c6090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:25.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.500+0000 7ff791600640 1 --2- 172.21.15.96:0/3457003582 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff78c15a1c0 0x7ff78c0c5b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:25.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.500+0000 7ff791600640 1 --2- 172.21.15.96:0/3457003582 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff770004800 0x7ff78c163bc0 unknown :-1 s=CLOSED pgs=1638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:25.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.500+0000 7ff791600640 1 -- 172.21.15.96:0/3457003582 >> 172.21.15.96:0/3457003582 conn(0x7ff78c06bae0 msgr2=0x7ff78c073560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:25.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.501+0000 7ff791600640 1 -- 172.21.15.96:0/3457003582 shutdown_connections 2024-05-28T06:20:25.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.501+0000 7ff791600640 1 -- 172.21.15.96:0/3457003582 wait complete. 2024-05-28T06:20:25.516 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-05-28T06:20:25.712 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.715+0000 7fd71b200640 1 Processor -- start 2024-05-28T06:20:25.712 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.715+0000 7fd71b200640 1 -- start start 2024-05-28T06:20:25.713 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.716+0000 7fd71b200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fd71c0cfa40 0x7fd71c169690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:25.713 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.716+0000 7fd71b200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd71c15ec50 con 0x7fd71c0d16a0 2024-05-28T06:20:25.713 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.716+0000 7fd71b200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd71c0cff80 con 0x7fd71c15e870 2024-05-28T06:20:25.713 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.716+0000 7fd71b200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fd71c0d2150 con 0x7fd71c0cfa40 2024-05-28T06:20:25.714 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.717+0000 7fd719400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fd71c0d16a0 0x7fd71c0cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:51122/0 (socket says 172.21.15.96:51122) 2024-05-28T06:20:25.714 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.717+0000 7fd719400640 1 -- 172.21.15.96:0/3861787711 learned_addr learned my addr 172.21.15.96:0/3861787711 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:20:25.714 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.717+0000 7fd71a800640 1 --2- 172.21.15.96:0/3861787711 >> v2:172.21.15.96:3300/0 conn(0x7fd71c0cfa40 0x7fd71c169690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:25.714 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.717+0000 7fd718a00640 1 -- 172.21.15.96:0/3861787711 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 287736114 0 0) 0x7fd71c15ec50 con 0x7fd71c0d16a0 2024-05-28T06:20:25.714 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.717+0000 7fd718a00640 1 -- 172.21.15.96:0/3861787711 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd700003520 con 0x7fd71c0d16a0 2024-05-28T06:20:25.715 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.717+0000 7fd71a800640 1 -- 172.21.15.96:0/3861787711 >> v1:172.21.15.123:6789/0 conn(0x7fd71c15e870 legacy=0x7fd71c0d0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:25.715 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.717+0000 7fd71a800640 1 -- 172.21.15.96:0/3861787711 >> v1:172.21.15.96:6789/0 conn(0x7fd71c0d16a0 legacy=0x7fd71c0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:25.715 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.717+0000 7fd71a800640 1 -- 172.21.15.96:0/3861787711 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd71c0d0260 con 0x7fd71c0cfa40 2024-05-28T06:20:25.717 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.718+0000 7fd71a800640 1 --2- 172.21.15.96:0/3861787711 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd71c0cfa40 0x7fd71c169690 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7fd714009d90 tx=0x7fd71402a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2300fa959e8396eb server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:25.717 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.719+0000 7fd718a00640 1 -- 172.21.15.96:0/3861787711 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd714007740 con 0x7fd71c0cfa40 2024-05-28T06:20:25.717 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.719+0000 7fd718a00640 1 -- 172.21.15.96:0/3861787711 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7140078e0 con 0x7fd71c0cfa40 2024-05-28T06:20:25.717 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.719+0000 7fd718a00640 1 -- 172.21.15.96:0/3861787711 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd714007c90 con 0x7fd71c0cfa40 2024-05-28T06:20:25.717 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.719+0000 7fd71b200640 1 -- 172.21.15.96:0/3861787711 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd71c0cfa40 msgr2=0x7fd71c169690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:25.717 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.719+0000 7fd71b200640 1 --2- 172.21.15.96:0/3861787711 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd71c0cfa40 0x7fd71c169690 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7fd714009d90 tx=0x7fd71402a6a0 comp rx=0 tx=0).stop 2024-05-28T06:20:25.717 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.719+0000 7fd71b200640 1 -- 172.21.15.96:0/3861787711 shutdown_connections 2024-05-28T06:20:25.718 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.719+0000 7fd71b200640 1 --2- 172.21.15.96:0/3861787711 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd71c0cfa40 0x7fd71c169690 unknown :-1 s=CLOSED pgs=1644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:25.718 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.719+0000 7fd71b200640 1 -- 172.21.15.96:0/3861787711 >> 172.21.15.96:0/3861787711 conn(0x7fd71c08bfc0 msgr2=0x7fd71c0d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:25.718 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.719+0000 7fd71b200640 1 -- 172.21.15.96:0/3861787711 shutdown_connections 2024-05-28T06:20:25.718 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.719+0000 7fd71b200640 1 -- 172.21.15.96:0/3861787711 wait complete. 2024-05-28T06:20:25.718 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.721+0000 7fd71b200640 1 Processor -- start 2024-05-28T06:20:25.718 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.721+0000 7fd71b200640 1 -- start start 2024-05-28T06:20:25.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.722+0000 7fd71b200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd71c0cfa40 0x7fd71c0cae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:25.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.722+0000 7fd71b200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd71c0d16a0 0x7fd71c159b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:25.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.722+0000 7fd71b200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd71c15e870 0x7fd71c0c80b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:25.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.722+0000 7fd71b200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd71c0d4fb0 con 0x7fd71c15e870 2024-05-28T06:20:25.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.722+0000 7fd71b200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd71c0d0820 con 0x7fd71c0cfa40 2024-05-28T06:20:25.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.722+0000 7fd71b200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fd71c0d35f0 con 0x7fd71c0d16a0 2024-05-28T06:20:25.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.723+0000 7fd71a800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd71c15e870 0x7fd71c0c80b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:25.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.723+0000 7fd71a800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd71c15e870 0x7fd71c0c80b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58924/0 (socket says 172.21.15.96:58924) 2024-05-28T06:20:25.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.723+0000 7fd71a800640 1 -- 172.21.15.96:0/1281089045 learned_addr learned my addr 172.21.15.96:0/1281089045 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:20:25.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.723+0000 7fd719e00640 1 --2- 172.21.15.96:0/1281089045 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd71c0cfa40 0x7fd71c0cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:25.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.723+0000 7fd719400640 1 --2- 172.21.15.96:0/1281089045 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd71c0d16a0 0x7fd71c159b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:25.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.723+0000 7fd71a800640 1 -- 172.21.15.96:0/1281089045 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd71c0d16a0 msgr2=0x7fd71c159b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:25.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.723+0000 7fd71a800640 1 --2- 172.21.15.96:0/1281089045 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd71c0d16a0 0x7fd71c159b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:25.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.723+0000 7fd71a800640 1 -- 172.21.15.96:0/1281089045 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd71c0cfa40 msgr2=0x7fd71c0cae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:25.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.723+0000 7fd71a800640 1 --2- 172.21.15.96:0/1281089045 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd71c0cfa40 0x7fd71c0cae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:25.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.723+0000 7fd71a800640 1 -- 172.21.15.96:0/1281089045 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd71c15fda0 con 0x7fd71c15e870 2024-05-28T06:20:25.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.723+0000 7fd719400640 1 --2- 172.21.15.96:0/1281089045 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd71c0d16a0 0x7fd71c159b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:20:25.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.723+0000 7fd71a800640 1 --2- 172.21.15.96:0/1281089045 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd71c15e870 0x7fd71c0c80b0 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7fd714009eb0 tx=0x7fd714004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:25.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.724+0000 7fd712000640 1 -- 172.21.15.96:0/1281089045 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd71403f070 con 0x7fd71c15e870 2024-05-28T06:20:25.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.724+0000 7fd712000640 1 -- 172.21.15.96:0/1281089045 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd714031bc0 con 0x7fd71c15e870 2024-05-28T06:20:25.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.724+0000 7fd712000640 1 -- 172.21.15.96:0/1281089045 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd714009450 con 0x7fd71c15e870 2024-05-28T06:20:25.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.724+0000 7fd71b200640 1 -- 172.21.15.96:0/1281089045 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd71c160080 con 0x7fd71c15e870 2024-05-28T06:20:25.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.724+0000 7fd71b200640 1 -- 172.21.15.96:0/1281089045 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd71c160640 con 0x7fd71c15e870 2024-05-28T06:20:25.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.725+0000 7fd71b200640 1 -- 172.21.15.96:0/1281089045 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd71c0689e0 con 0x7fd71c15e870 2024-05-28T06:20:25.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.726+0000 7fd712000640 1 -- 172.21.15.96:0/1281089045 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 34) ==== 92927+0+0 (secure 0 0 0) 0x7fd714043410 con 0x7fd71c15e870 2024-05-28T06:20:25.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.727+0000 7fd712000640 1 --2- 172.21.15.96:0/1281089045 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd6fc06ddc0 0x7fd6fc070280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:25.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.727+0000 7fd712000640 1 -- 172.21.15.96:0/1281089045 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 5765+0+0 (secure 0 0 0) 0x7fd7140ac020 con 0x7fd71c15e870 2024-05-28T06:20:25.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.727+0000 7fd719e00640 1 --2- 172.21.15.96:0/1281089045 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd6fc06ddc0 0x7fd6fc070280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:25.725 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.728+0000 7fd719e00640 1 --2- 172.21.15.96:0/1281089045 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd6fc06ddc0 0x7fd6fc070280 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fd71c162240 tx=0x7fd70c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:25.727 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.731+0000 7fd712000640 1 -- 172.21.15.96:0/1281089045 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd71407bb20 con 0x7fd71c15e870 2024-05-28T06:20:25.953 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.956+0000 7fd71b200640 1 -- 172.21.15.96:0/1281089045 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fd71c0ba500 con 0x7fd71c15e870 2024-05-28T06:20:25.954 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.957+0000 7fd712000640 1 -- 172.21.15.96:0/1281089045 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v62) ==== 53+0+73 (secure 0 0 0) 0x7fd71407b420 con 0x7fd71c15e870 2024-05-28T06:20:25.954 INFO:teuthology.orchestra.run.smithi096.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-05-28T06:20:25.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.960+0000 7fd71b200640 1 -- 172.21.15.96:0/1281089045 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd6fc06ddc0 msgr2=0x7fd6fc070280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:25.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.960+0000 7fd71b200640 1 --2- 172.21.15.96:0/1281089045 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd6fc06ddc0 0x7fd6fc070280 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fd71c162240 tx=0x7fd70c009040 comp rx=0 tx=0).stop 2024-05-28T06:20:25.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.960+0000 7fd71b200640 1 -- 172.21.15.96:0/1281089045 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd71c15e870 msgr2=0x7fd71c0c80b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:25.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.960+0000 7fd71b200640 1 --2- 172.21.15.96:0/1281089045 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd71c15e870 0x7fd71c0c80b0 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7fd714009eb0 tx=0x7fd714004a40 comp rx=0 tx=0).stop 2024-05-28T06:20:25.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.960+0000 7fd71b200640 1 -- 172.21.15.96:0/1281089045 shutdown_connections 2024-05-28T06:20:25.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.960+0000 7fd71b200640 1 --2- 172.21.15.96:0/1281089045 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd6fc06ddc0 0x7fd6fc070280 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:25.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.960+0000 7fd71b200640 1 --2- 172.21.15.96:0/1281089045 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd71c15e870 0x7fd71c0c80b0 unknown :-1 s=CLOSED pgs=1645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:25.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.960+0000 7fd71b200640 1 --2- 172.21.15.96:0/1281089045 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd71c0d16a0 0x7fd71c159b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:25.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.960+0000 7fd71b200640 1 --2- 172.21.15.96:0/1281089045 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd71c0cfa40 0x7fd71c0cae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:25.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.960+0000 7fd71b200640 1 -- 172.21.15.96:0/1281089045 >> 172.21.15.96:0/1281089045 conn(0x7fd71c08bfc0 msgr2=0x7fd71c0b4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:25.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.960+0000 7fd71b200640 1 -- 172.21.15.96:0/1281089045 shutdown_connections 2024-05-28T06:20:25.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:25.960+0000 7fd71b200640 1 -- 172.21.15.96:0/1281089045 wait complete. 2024-05-28T06:20:25.978 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-05-28T06:20:25.978 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-28T06:20:25.978 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi096.front.sepia.ceph.com 2024-05-28T06:20:25.978 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-05-28T06:20:25.978 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-28T06:20:25.979 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-05-28T06:20:25.979 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-05-28T06:20:25.979 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-28T06:20:25.979 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-28T06:20:25.979 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:20:25.979 DEBUG:teuthology.orchestra.run.smithi096:> ip addr 2024-05-28T06:20:26.028 INFO:teuthology.orchestra.run.smithi096.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-28T06:20:26.028 INFO:teuthology.orchestra.run.smithi096.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-28T06:20:26.028 INFO:teuthology.orchestra.run.smithi096.stdout: inet 127.0.0.1/8 scope host lo 2024-05-28T06:20:26.028 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:20:26.028 INFO:teuthology.orchestra.run.smithi096.stdout: inet6 ::1/128 scope host 2024-05-28T06:20:26.028 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:20:26.029 INFO:teuthology.orchestra.run.smithi096.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-28T06:20:26.029 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether 0c:c4:7a:d8:d9:bc brd ff:ff:ff:ff:ff:ff 2024-05-28T06:20:26.029 INFO:teuthology.orchestra.run.smithi096.stdout: altname enp7s0f0 2024-05-28T06:20:26.029 INFO:teuthology.orchestra.run.smithi096.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-28T06:20:26.029 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether 0c:c4:7a:d8:d9:bd brd ff:ff:ff:ff:ff:ff 2024-05-28T06:20:26.029 INFO:teuthology.orchestra.run.smithi096.stdout: altname enp7s0f1 2024-05-28T06:20:26.029 INFO:teuthology.orchestra.run.smithi096.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-28T06:20:26.029 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether 0c:c4:7a:8f:cc:0c brd ff:ff:ff:ff:ff:ff 2024-05-28T06:20:26.029 INFO:teuthology.orchestra.run.smithi096.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-28T06:20:26.029 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether 0c:c4:7a:8f:cc:0d brd ff:ff:ff:ff:ff:ff 2024-05-28T06:20:26.029 INFO:teuthology.orchestra.run.smithi096.stdout: inet 172.21.15.96/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-05-28T06:20:26.029 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft 41611sec preferred_lft 41611sec 2024-05-28T06:20:26.029 INFO:teuthology.orchestra.run.smithi096.stdout: inet6 fe80::ec4:7aff:fe8f:cc0d/64 scope link 2024-05-28T06:20:26.030 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:20:26.030 INFO:teuthology.orchestra.run.smithi096.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-05-28T06:20:26.030 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether ae:d9:f5:b0:4f:f3 brd ff:ff:ff:ff:ff:ff 2024-05-28T06:20:26.030 INFO:teuthology.orchestra.run.smithi096.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-05-28T06:20:26.030 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:20:26.030 INFO:teuthology.orchestra.run.smithi096.stdout: inet6 fe80::604e:fcff:fe60:7892/64 scope link 2024-05-28T06:20:26.030 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:20:26.030 INFO:teuthology.orchestra.run.smithi096.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-28T06:20:26.030 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether ce:6f:19:93:44:f9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-28T06:20:26.030 INFO:teuthology.orchestra.run.smithi096.stdout: inet6 fe80::cc6f:19ff:fe93:44f9/64 scope link 2024-05-28T06:20:26.030 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:20:26.030 INFO:teuthology.orchestra.run.smithi096.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-28T06:20:26.030 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether ae:d9:f5:b0:4f:f3 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-28T06:20:26.031 INFO:teuthology.orchestra.run.smithi096.stdout: inet6 fe80::acd9:f5ff:feb0:4ff3/64 scope link 2024-05-28T06:20:26.031 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:20:26.031 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:20:26.031 DEBUG:teuthology.orchestra.run.smithi096:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-28T06:20:26.148 INFO:teuthology.orchestra.run.smithi096.stdout:1 2024-05-28T06:20:26.153 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:20:26.153 DEBUG:teuthology.orchestra.run.smithi096:> route 2024-05-28T06:20:26.204 INFO:teuthology.orchestra.run.smithi096.stdout:Kernel IP routing table 2024-05-28T06:20:26.204 INFO:teuthology.orchestra.run.smithi096.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-28T06:20:26.204 INFO:teuthology.orchestra.run.smithi096.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-05-28T06:20:26.204 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-05-28T06:20:26.204 INFO:teuthology.orchestra.run.smithi096.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-28T06:20:26.205 DEBUG:teuthology.orchestra.run.smithi096:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-28T06:20:26.205 DEBUG:teuthology.orchestra.run.smithi096:> set -e 2024-05-28T06:20:26.205 DEBUG:teuthology.orchestra.run.smithi096:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-05-28T06:20:26.205 DEBUG:teuthology.orchestra.run.smithi096:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-05-28T06:20:26.205 DEBUG:teuthology.orchestra.run.smithi096:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-05-28T06:20:26.206 DEBUG:teuthology.orchestra.run.smithi096:> ') 2024-05-28T06:20:26.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:26 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-28T06:20:26.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:26 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-28T06:20:26.413 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:20:26.413 DEBUG:teuthology.orchestra.run.smithi096:> ip netns list 2024-05-28T06:20:26.463 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-05-28T06:20:26.463 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-05-28T06:20:26.463 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-28T06:20:26.464 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:20:26.464 DEBUG:teuthology.orchestra.run.smithi096:> sudo ip link set brx.0 up 2024-05-28T06:20:26.544 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-05-28T06:20:26.544 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-28T06:20:26.544 DEBUG:teuthology.orchestra.run.smithi096:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-05-28T06:20:26.598 INFO:teuthology.orchestra.run.smithi096.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-05-28T06:20:26.598 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-28T06:20:26.598 DEBUG:teuthology.orchestra.run.smithi096:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-05-28T06:20:26.652 DEBUG:teuthology.orchestra.run.smithi096:> sudo modprobe fuse 2024-05-28T06:20:26.733 DEBUG:teuthology.orchestra.run.smithi096:> cat /proc/self/mounts | awk '{print $2}' 2024-05-28T06:20:26.783 INFO:teuthology.orchestra.run.smithi096.stdout:/proc 2024-05-28T06:20:26.783 INFO:teuthology.orchestra.run.smithi096.stdout:/sys 2024-05-28T06:20:26.784 INFO:teuthology.orchestra.run.smithi096.stdout:/dev 2024-05-28T06:20:26.784 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/security 2024-05-28T06:20:26.784 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/shm 2024-05-28T06:20:26.784 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/pts 2024-05-28T06:20:26.784 INFO:teuthology.orchestra.run.smithi096.stdout:/run 2024-05-28T06:20:26.784 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/cgroup 2024-05-28T06:20:26.784 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/pstore 2024-05-28T06:20:26.784 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/bpf 2024-05-28T06:20:26.784 INFO:teuthology.orchestra.run.smithi096.stdout:/ 2024-05-28T06:20:26.784 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/selinux 2024-05-28T06:20:26.784 INFO:teuthology.orchestra.run.smithi096.stdout:/proc/sys/fs/binfmt_misc 2024-05-28T06:20:26.785 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/mqueue 2024-05-28T06:20:26.785 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/debug 2024-05-28T06:20:26.785 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/hugepages 2024-05-28T06:20:26.785 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/tracing 2024-05-28T06:20:26.785 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/fuse/connections 2024-05-28T06:20:26.785 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/config 2024-05-28T06:20:26.786 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-sysctl.service 2024-05-28T06:20:26.786 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-28T06:20:26.786 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-28T06:20:26.786 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/nfs/rpc_pipefs 2024-05-28T06:20:26.786 INFO:teuthology.orchestra.run.smithi096.stdout:/run/user/1000 2024-05-28T06:20:26.786 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph 2024-05-28T06:20:26.786 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-0 2024-05-28T06:20:26.786 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-1 2024-05-28T06:20:26.786 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-2 2024-05-28T06:20:26.787 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-3 2024-05-28T06:20:26.787 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns 2024-05-28T06:20:26.787 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-28T06:20:26.787 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-28T06:20:26.787 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-28T06:20:26.787 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-28T06:20:26.788 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:20:26.789 DEBUG:teuthology.orchestra.run.smithi096:> ls /sys/fs/fuse/connections 2024-05-28T06:20:26.838 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-05-28T06:20:26.839 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-05-28T06:20:26.844 DEBUG:teuthology.orchestra.run.smithi096:> sudo modprobe fuse 2024-05-28T06:20:26.912 DEBUG:teuthology.orchestra.run.smithi096:> cat /proc/self/mounts | awk '{print $2}' 2024-05-28T06:20:26.955 INFO:teuthology.orchestra.run.smithi096.stdout:/proc 2024-05-28T06:20:26.955 INFO:teuthology.orchestra.run.smithi096.stdout:/sys 2024-05-28T06:20:26.955 INFO:teuthology.orchestra.run.smithi096.stdout:/dev 2024-05-28T06:20:26.955 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/security 2024-05-28T06:20:26.955 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/shm 2024-05-28T06:20:26.955 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/pts 2024-05-28T06:20:26.955 INFO:teuthology.orchestra.run.smithi096.stdout:/run 2024-05-28T06:20:26.955 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/cgroup 2024-05-28T06:20:26.955 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/pstore 2024-05-28T06:20:26.955 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/bpf 2024-05-28T06:20:26.955 INFO:teuthology.orchestra.run.smithi096.stdout:/ 2024-05-28T06:20:26.956 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/selinux 2024-05-28T06:20:26.956 INFO:teuthology.orchestra.run.smithi096.stdout:/proc/sys/fs/binfmt_misc 2024-05-28T06:20:26.956 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/mqueue 2024-05-28T06:20:26.956 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/debug 2024-05-28T06:20:26.956 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/hugepages 2024-05-28T06:20:26.956 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/tracing 2024-05-28T06:20:26.956 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/fuse/connections 2024-05-28T06:20:26.956 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/config 2024-05-28T06:20:26.956 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-sysctl.service 2024-05-28T06:20:26.956 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-28T06:20:26.956 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-28T06:20:26.956 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/nfs/rpc_pipefs 2024-05-28T06:20:26.956 INFO:teuthology.orchestra.run.smithi096.stdout:/run/user/1000 2024-05-28T06:20:26.956 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph 2024-05-28T06:20:26.957 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-0 2024-05-28T06:20:26.957 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-1 2024-05-28T06:20:26.957 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-2 2024-05-28T06:20:26.957 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-3 2024-05-28T06:20:26.957 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns 2024-05-28T06:20:26.957 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-28T06:20:26.957 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-28T06:20:26.957 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-28T06:20:26.958 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-28T06:20:26.959 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:20:26.959 DEBUG:teuthology.orchestra.run.smithi096:> ls /sys/fs/fuse/connections 2024-05-28T06:20:27.017 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi096.stderr:2024-05-28T06:20:27.020+0000 7f91cff4e480 -1 init, newargv = 0x55c341e34800 newargc=15 2024-05-28T06:20:27.017 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi096.stderr:ceph-fuse[75861]: starting ceph client 2024-05-28T06:20:27.027 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi096.stderr:ceph-fuse[75861]: starting fuse 2024-05-28T06:20:27.039 INFO:teuthology.orchestra.run.smithi096.stdout:40 2024-05-28T06:20:27.039 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40] 2024-05-28T06:20:27.039 DEBUG:teuthology.orchestra.run.smithi096:> sudo stdin-killer -- python3 -c ' 2024-05-28T06:20:27.039 DEBUG:teuthology.orchestra.run.smithi096:> import glob 2024-05-28T06:20:27.039 DEBUG:teuthology.orchestra.run.smithi096:> import re 2024-05-28T06:20:27.039 DEBUG:teuthology.orchestra.run.smithi096:> import os 2024-05-28T06:20:27.039 DEBUG:teuthology.orchestra.run.smithi096:> import subprocess 2024-05-28T06:20:27.039 DEBUG:teuthology.orchestra.run.smithi096:> 2024-05-28T06:20:27.039 DEBUG:teuthology.orchestra.run.smithi096:> def _find_admin_socket(client_name): 2024-05-28T06:20:27.040 DEBUG:teuthology.orchestra.run.smithi096:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-05-28T06:20:27.040 DEBUG:teuthology.orchestra.run.smithi096:> files = glob.glob(asok_path) 2024-05-28T06:20:27.040 DEBUG:teuthology.orchestra.run.smithi096:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-05-28T06:20:27.040 DEBUG:teuthology.orchestra.run.smithi096:> 2024-05-28T06:20:27.040 DEBUG:teuthology.orchestra.run.smithi096:> # Given a non-glob path, it better be there 2024-05-28T06:20:27.040 DEBUG:teuthology.orchestra.run.smithi096:> if "*" not in asok_path: 2024-05-28T06:20:27.040 DEBUG:teuthology.orchestra.run.smithi096:> assert(len(files) == 1) 2024-05-28T06:20:27.040 DEBUG:teuthology.orchestra.run.smithi096:> return files[0] 2024-05-28T06:20:27.040 DEBUG:teuthology.orchestra.run.smithi096:> 2024-05-28T06:20:27.040 DEBUG:teuthology.orchestra.run.smithi096:> for f in files: 2024-05-28T06:20:27.040 DEBUG:teuthology.orchestra.run.smithi096:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-05-28T06:20:27.040 DEBUG:teuthology.orchestra.run.smithi096:> if os.path.exists("/proc/{0}".format(pid)): 2024-05-28T06:20:27.040 DEBUG:teuthology.orchestra.run.smithi096:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-05-28T06:20:27.040 DEBUG:teuthology.orchestra.run.smithi096:> contents = proc_f.read() 2024-05-28T06:20:27.040 DEBUG:teuthology.orchestra.run.smithi096:> if mountpoint in contents: 2024-05-28T06:20:27.041 DEBUG:teuthology.orchestra.run.smithi096:> return f 2024-05-28T06:20:27.041 DEBUG:teuthology.orchestra.run.smithi096:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-05-28T06:20:27.041 DEBUG:teuthology.orchestra.run.smithi096:> 2024-05-28T06:20:27.041 DEBUG:teuthology.orchestra.run.smithi096:> print(_find_admin_socket("client.0")) 2024-05-28T06:20:27.041 DEBUG:teuthology.orchestra.run.smithi096:> ' 2024-05-28T06:20:27.226 INFO:teuthology.orchestra.run.smithi096.stdout:/var/run/ceph/ceph-client.0.75861.asok 2024-05-28T06:20:27.231 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-28T06:20:27.244 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.75861.asok 2024-05-28T06:20:27.244 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:20:27.244 DEBUG:teuthology.orchestra.run.smithi096:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.75861.asok status 2024-05-28T06:20:27.471 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:20:27.471 INFO:teuthology.orchestra.run.smithi096.stdout: "metadata": { 2024-05-28T06:20:27.471 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph_sha1": "9ed025fbf4406fb54980a0647506b15895b6d863", 2024-05-28T06:20:27.471 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph_version": "ceph version 19.0.0-3154-g9ed025fb (9ed025fbf4406fb54980a0647506b15895b6d863) squid (dev)", 2024-05-28T06:20:27.471 INFO:teuthology.orchestra.run.smithi096.stdout: "entity_id": "0", 2024-05-28T06:20:27.471 INFO:teuthology.orchestra.run.smithi096.stdout: "hostname": "smithi096", 2024-05-28T06:20:27.471 INFO:teuthology.orchestra.run.smithi096.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-05-28T06:20:27.471 INFO:teuthology.orchestra.run.smithi096.stdout: "pid": "75861", 2024-05-28T06:20:27.471 INFO:teuthology.orchestra.run.smithi096.stdout: "root": "/" 2024-05-28T06:20:27.472 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:20:27.472 INFO:teuthology.orchestra.run.smithi096.stdout: "dentry_count": 0, 2024-05-28T06:20:27.472 INFO:teuthology.orchestra.run.smithi096.stdout: "dentry_pinned_count": 0, 2024-05-28T06:20:27.472 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 8889, 2024-05-28T06:20:27.472 INFO:teuthology.orchestra.run.smithi096.stdout: "inst": { 2024-05-28T06:20:27.472 INFO:teuthology.orchestra.run.smithi096.stdout: "name": { 2024-05-28T06:20:27.472 INFO:teuthology.orchestra.run.smithi096.stdout: "type": "client", 2024-05-28T06:20:27.472 INFO:teuthology.orchestra.run.smithi096.stdout: "num": 8889 2024-05-28T06:20:27.472 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:20:27.472 INFO:teuthology.orchestra.run.smithi096.stdout: "addr": { 2024-05-28T06:20:27.472 INFO:teuthology.orchestra.run.smithi096.stdout: "type": "v1", 2024-05-28T06:20:27.472 INFO:teuthology.orchestra.run.smithi096.stdout: "addr": "192.168.0.1:0", 2024-05-28T06:20:27.472 INFO:teuthology.orchestra.run.smithi096.stdout: "nonce": 472262842 2024-05-28T06:20:27.472 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:20:27.473 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:20:27.473 INFO:teuthology.orchestra.run.smithi096.stdout: "addr": { 2024-05-28T06:20:27.473 INFO:teuthology.orchestra.run.smithi096.stdout: "type": "v1", 2024-05-28T06:20:27.473 INFO:teuthology.orchestra.run.smithi096.stdout: "addr": "192.168.0.1:0", 2024-05-28T06:20:27.473 INFO:teuthology.orchestra.run.smithi096.stdout: "nonce": 472262842 2024-05-28T06:20:27.473 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:20:27.473 INFO:teuthology.orchestra.run.smithi096.stdout: "inst_str": "client.8889 192.168.0.1:0/472262842", 2024-05-28T06:20:27.473 INFO:teuthology.orchestra.run.smithi096.stdout: "addr_str": "192.168.0.1:0/472262842", 2024-05-28T06:20:27.473 INFO:teuthology.orchestra.run.smithi096.stdout: "inode_count": 1, 2024-05-28T06:20:27.473 INFO:teuthology.orchestra.run.smithi096.stdout: "mds_epoch": 62, 2024-05-28T06:20:27.473 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_epoch": 103, 2024-05-28T06:20:27.473 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_epoch_barrier": 0, 2024-05-28T06:20:27.473 INFO:teuthology.orchestra.run.smithi096.stdout: "blocklisted": false, 2024-05-28T06:20:27.474 INFO:teuthology.orchestra.run.smithi096.stdout: "fs_name": "cephfs" 2024-05-28T06:20:27.474 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:20:27.486 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:20:27.486 DEBUG:teuthology.orchestra.run.smithi096:> stat --file-system '--printf=%T 2024-05-28T06:20:27.487 DEBUG:teuthology.orchestra.run.smithi096:> ' -- /home/ubuntu/cephtest/mnt.0 2024-05-28T06:20:27.535 INFO:teuthology.orchestra.run.smithi096.stdout:fuseblk 2024-05-28T06:20:27.535 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-05-28T06:20:27.536 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:20:27.536 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-05-28T06:20:27.665 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-05-28T06:20:27.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:27.880+0000 7f66cd200640 1 Processor -- start 2024-05-28T06:20:27.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:27.880+0000 7f66cd200640 1 -- start start 2024-05-28T06:20:27.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:27.882+0000 7f66cd200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f66c815e5e0 0x7f66c8162e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:27.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:27.882+0000 7f66cd200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66c80cf620 con 0x7f66c80d3270 2024-05-28T06:20:27.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:27.882+0000 7f66cd200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66c80cf900 con 0x7f66c80d03e0 2024-05-28T06:20:27.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:27.882+0000 7f66cd200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f66c80cfbe0 con 0x7f66c815e5e0 2024-05-28T06:20:27.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:27.882+0000 7f66c7e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f66c80d3270 0x7f66c8168720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:51138/0 (socket says 172.21.15.96:51138) 2024-05-28T06:20:27.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:27.882+0000 7f66c7e00640 1 -- 172.21.15.96:0/45811864 learned_addr learned my addr 172.21.15.96:0/45811864 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:20:27.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:27.883+0000 7f66c7400640 1 --2- 172.21.15.96:0/45811864 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f66c815e5e0 0x7f66c8162e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:27.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:27.883+0000 7f66c6a00640 1 -- 172.21.15.96:0/45811864 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3620990454 0 0) 0x7f66c80cf620 con 0x7f66c80d3270 2024-05-28T06:20:27.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:27.883+0000 7f66c6a00640 1 -- 172.21.15.96:0/45811864 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f66ac003520 con 0x7f66c80d3270 2024-05-28T06:20:27.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:27.883+0000 7f66c6a00640 1 -- 172.21.15.96:0/45811864 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 695608583 0 0) 0x7f66ac003520 con 0x7f66c80d3270 2024-05-28T06:20:27.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:27.883+0000 7f66c6a00640 1 -- 172.21.15.96:0/45811864 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f66c815e5e0 msgr2=0x7f66c8162e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:27.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:27.883+0000 7f66c6a00640 1 --2- 172.21.15.96:0/45811864 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f66c815e5e0 0x7f66c8162e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:27.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:27.883+0000 7f66c6a00640 1 -- 172.21.15.96:0/45811864 >> v1:172.21.15.123:6789/0 conn(0x7f66c80d03e0 legacy=0x7f66c80d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:27.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:27.883+0000 7f66c6a00640 1 -- 172.21.15.96:0/45811864 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66c80cfec0 con 0x7f66c80d3270 2024-05-28T06:20:27.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:27.883+0000 7f66c6a00640 1 -- 172.21.15.96:0/45811864 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f66c00019f0 con 0x7f66c80d3270 2024-05-28T06:20:27.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:27.883+0000 7f66c6a00640 1 -- 172.21.15.96:0/45811864 >> v1:172.21.15.96:6789/0 conn(0x7f66c80d3270 legacy=0x7f66c8168720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:27.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:27.883+0000 7f66c6a00640 1 --2- 172.21.15.96:0/45811864 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f66ac0039e0 0x7f66ac003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:27.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:27.883+0000 7f66c6a00640 1 --2- 172.21.15.96:0/45811864 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f66c815e5e0 0x7f66ac004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:27.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:27.883+0000 7f66c6a00640 1 --2- 172.21.15.96:0/45811864 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66ac004d40 0x7f66ac0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:27.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:27.883+0000 7f66c6a00640 1 -- 172.21.15.96:0/45811864 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f66c80cfbe0 con 0x7f66ac0039e0 2024-05-28T06:20:27.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:27.883+0000 7f66c6a00640 1 -- 172.21.15.96:0/45811864 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f66ac00bb50 con 0x7f66ac004d40 2024-05-28T06:20:27.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:27.883+0000 7f66c6a00640 1 -- 172.21.15.96:0/45811864 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f66ac00bd20 con 0x7f66c815e5e0 2024-05-28T06:20:27.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:27.884+0000 7f66c7400640 1 --2- 172.21.15.96:0/45811864 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f66ac0039e0 0x7f66ac003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:27.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:27.884+0000 7f66c7e00640 1 --2- 172.21.15.96:0/45811864 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66ac004d40 0x7f66ac0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:27.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:27.884+0000 7f66c7400640 1 -- 172.21.15.96:0/45811864 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f66c815e5e0 msgr2=0x7f66ac004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:27.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:27.884+0000 7f66c7400640 1 --2- 172.21.15.96:0/45811864 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f66c815e5e0 0x7f66ac004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:27.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:27.884+0000 7f66c7400640 1 -- 172.21.15.96:0/45811864 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66ac004d40 msgr2=0x7f66ac0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:27.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:27.884+0000 7f66c7400640 1 --2- 172.21.15.96:0/45811864 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66ac004d40 0x7f66ac0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:27.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:27.884+0000 7f66c7400640 1 -- 172.21.15.96:0/45811864 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66ac003520 con 0x7f66ac0039e0 2024-05-28T06:20:27.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:27.884+0000 7f66c7e00640 1 --2- 172.21.15.96:0/45811864 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66ac004d40 0x7f66ac0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:20:27.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:27.885+0000 7f66c7400640 1 --2- 172.21.15.96:0/45811864 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f66ac0039e0 0x7f66ac003e00 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7f66b800ba60 tx=0x7f66ac005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c389aec6fa3fb396 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:27.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:27.885+0000 7f66c6a00640 1 -- 172.21.15.96:0/45811864 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66b8030070 con 0x7f66ac0039e0 2024-05-28T06:20:27.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:27.885+0000 7f66c6a00640 1 -- 172.21.15.96:0/45811864 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66b802ae40 con 0x7f66ac0039e0 2024-05-28T06:20:27.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:27.885+0000 7f66c6a00640 1 -- 172.21.15.96:0/45811864 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66b800b600 con 0x7f66ac0039e0 2024-05-28T06:20:27.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:27.886+0000 7f66cd200640 1 -- 172.21.15.96:0/45811864 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f66ac0039e0 msgr2=0x7f66ac003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:27.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:27.886+0000 7f66cd200640 1 --2- 172.21.15.96:0/45811864 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f66ac0039e0 0x7f66ac003e00 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7f66b800ba60 tx=0x7f66ac005830 comp rx=0 tx=0).stop 2024-05-28T06:20:27.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:27.886+0000 7f66c7e00640 1 -- 172.21.15.96:0/45811864 reap_dead start 2024-05-28T06:20:27.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:27.886+0000 7f66cd200640 1 -- 172.21.15.96:0/45811864 shutdown_connections 2024-05-28T06:20:27.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:27.886+0000 7f66cd200640 1 -- 172.21.15.96:0/45811864 >> 172.21.15.96:0/45811864 conn(0x7f66c808bfc0 msgr2=0x7f66c80d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:27.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:27.886+0000 7f66cd200640 1 -- 172.21.15.96:0/45811864 shutdown_connections 2024-05-28T06:20:27.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:27.886+0000 7f66cd200640 1 -- 172.21.15.96:0/45811864 wait complete. 2024-05-28T06:20:27.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:27.888+0000 7f66cd200640 1 Processor -- start 2024-05-28T06:20:27.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:27.888+0000 7f66cd200640 1 -- start start 2024-05-28T06:20:27.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:27.889+0000 7f66cd200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f66ac0039e0 0x7f66c80ca6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:27.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:27.889+0000 7f66cd200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66c815c850 0x7f66c80b53d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:27.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:27.889+0000 7f66cd200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f66c80b5910 0x7f66c81ea130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:27.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:27.889+0000 7f66cd200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f66c815eb40 con 0x7f66c80b5910 2024-05-28T06:20:27.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:27.889+0000 7f66cd200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f66c815e9c0 con 0x7f66c815c850 2024-05-28T06:20:27.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:27.889+0000 7f66cd200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f66c815ecc0 con 0x7f66ac0039e0 2024-05-28T06:20:27.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:27.889+0000 7f66c7e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f66c80b5910 0x7f66c81ea130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:27.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:27.889+0000 7f66c7e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f66c80b5910 0x7f66c81ea130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58948/0 (socket says 172.21.15.96:58948) 2024-05-28T06:20:27.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:27.890+0000 7f66c7e00640 1 -- 172.21.15.96:0/934597459 learned_addr learned my addr 172.21.15.96:0/934597459 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:20:27.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:27.890+0000 7f66bfe00640 1 --2- 172.21.15.96:0/934597459 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66c815c850 0x7f66c80b53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:27.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:27.890+0000 7f66c7400640 1 --2- 172.21.15.96:0/934597459 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f66ac0039e0 0x7f66c80ca6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:27.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:27.890+0000 7f66c7e00640 1 -- 172.21.15.96:0/934597459 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f66ac0039e0 msgr2=0x7f66c80ca6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:27.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:27.890+0000 7f66c7e00640 1 --2- 172.21.15.96:0/934597459 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f66ac0039e0 0x7f66c80ca6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:27.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:27.890+0000 7f66c7e00640 1 -- 172.21.15.96:0/934597459 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66c815c850 msgr2=0x7f66c80b53d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:27.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:27.890+0000 7f66c7e00640 1 --2- 172.21.15.96:0/934597459 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66c815c850 0x7f66c80b53d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:27.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:27.890+0000 7f66c7e00640 1 -- 172.21.15.96:0/934597459 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66c815d890 con 0x7f66c80b5910 2024-05-28T06:20:27.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:27.890+0000 7f66bfe00640 1 --2- 172.21.15.96:0/934597459 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66c815c850 0x7f66c80b53d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:20:27.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:27.890+0000 7f66c7e00640 1 --2- 172.21.15.96:0/934597459 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f66c80b5910 0x7f66c81ea130 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7f66c808d9e0 tx=0x7f66c000b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:27.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:27.890+0000 7f66bf400640 1 -- 172.21.15.96:0/934597459 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66c0004d60 con 0x7f66c80b5910 2024-05-28T06:20:27.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:27.891+0000 7f66bf400640 1 -- 172.21.15.96:0/934597459 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66c000bb80 con 0x7f66c80b5910 2024-05-28T06:20:27.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:27.891+0000 7f66cd200640 1 -- 172.21.15.96:0/934597459 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66c80c9aa0 con 0x7f66c80b5910 2024-05-28T06:20:27.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:27.891+0000 7f66bf400640 1 -- 172.21.15.96:0/934597459 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66c0007070 con 0x7f66c80b5910 2024-05-28T06:20:27.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:27.891+0000 7f66cd200640 1 -- 172.21.15.96:0/934597459 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66c80b48f0 con 0x7f66c80b5910 2024-05-28T06:20:27.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:27.892+0000 7f66cd200640 1 -- 172.21.15.96:0/934597459 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66c80690f0 con 0x7f66c80b5910 2024-05-28T06:20:27.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:27.893+0000 7f66bf400640 1 -- 172.21.15.96:0/934597459 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 34) ==== 92927+0+0 (secure 0 0 0) 0x7f66c001b020 con 0x7f66c80b5910 2024-05-28T06:20:27.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:27.894+0000 7f66bf400640 1 --2- 172.21.15.96:0/934597459 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f66a406ddf0 0x7f66a40702b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:27.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:27.894+0000 7f66bf400640 1 -- 172.21.15.96:0/934597459 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 5765+0+0 (secure 0 0 0) 0x7f66c008a9d0 con 0x7f66c80b5910 2024-05-28T06:20:27.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:27.894+0000 7f66c7400640 1 --2- 172.21.15.96:0/934597459 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f66a406ddf0 0x7f66a40702b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:27.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:27.896+0000 7f66c7400640 1 --2- 172.21.15.96:0/934597459 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f66a406ddf0 0x7f66a40702b0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f66b800ba30 tx=0x7f66b8030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:27.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:27.899+0000 7f66bf400640 1 -- 172.21.15.96:0/934597459 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f66c005a850 con 0x7f66c80b5910 2024-05-28T06:20:28.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:28.143+0000 7f66cd200640 1 -- 172.21.15.96:0/934597459 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) -- 0x7f66c80cac30 con 0x7f66c80b5910 2024-05-28T06:20:28.987 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:28.991+0000 7f66bf400640 1 -- 172.21.15.96:0/934597459 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v63) ==== 136+0+0 (secure 0 0 0) 0x7f66c005a150 con 0x7f66c80b5910 2024-05-28T06:20:28.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:28.994+0000 7f66cd200640 1 -- 172.21.15.96:0/934597459 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f66a406ddf0 msgr2=0x7f66a40702b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:28.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:28.994+0000 7f66cd200640 1 --2- 172.21.15.96:0/934597459 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f66a406ddf0 0x7f66a40702b0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f66b800ba30 tx=0x7f66b8030000 comp rx=0 tx=0).stop 2024-05-28T06:20:28.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:28.994+0000 7f66cd200640 1 -- 172.21.15.96:0/934597459 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f66c80b5910 msgr2=0x7f66c81ea130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:28.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:28.994+0000 7f66cd200640 1 --2- 172.21.15.96:0/934597459 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f66c80b5910 0x7f66c81ea130 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7f66c808d9e0 tx=0x7f66c000b4a0 comp rx=0 tx=0).stop 2024-05-28T06:20:28.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:28.994+0000 7f66cd200640 1 -- 172.21.15.96:0/934597459 shutdown_connections 2024-05-28T06:20:28.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:28.994+0000 7f66cd200640 1 --2- 172.21.15.96:0/934597459 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f66a406ddf0 0x7f66a40702b0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:28.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:28.994+0000 7f66cd200640 1 --2- 172.21.15.96:0/934597459 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f66c80b5910 0x7f66c81ea130 unknown :-1 s=CLOSED pgs=1651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:28.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:28.994+0000 7f66cd200640 1 --2- 172.21.15.96:0/934597459 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66c815c850 0x7f66c80b53d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:28.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:28.995+0000 7f66cd200640 1 --2- 172.21.15.96:0/934597459 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f66ac0039e0 0x7f66c80ca6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:28.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:28.995+0000 7f66cd200640 1 -- 172.21.15.96:0/934597459 >> 172.21.15.96:0/934597459 conn(0x7f66c808bfc0 msgr2=0x7f66c80d1700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:28.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:28.995+0000 7f66cd200640 1 -- 172.21.15.96:0/934597459 shutdown_connections 2024-05-28T06:20:28.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:28.995+0000 7f66cd200640 1 -- 172.21.15.96:0/934597459 wait complete. 2024-05-28T06:20:29.012 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-05-28T06:20:29.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.216+0000 7fde01a00640 1 Processor -- start 2024-05-28T06:20:29.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.216+0000 7fde01a00640 1 -- start start 2024-05-28T06:20:29.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.218+0000 7fde01a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fddfc0d04a0 0x7fddfc0d0900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:29.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.218+0000 7fde01a00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fddfc0ad130 0x7fddfc0ad5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:29.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.218+0000 7fde01a00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fddfc06ed10 con 0x7fddfc06f860 2024-05-28T06:20:29.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.218+0000 7fde01a00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fddfc0d0e40 con 0x7fddfc0ad130 2024-05-28T06:20:29.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.218+0000 7fde01a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fddfc15cb00 con 0x7fddfc0d04a0 2024-05-28T06:20:29.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.218+0000 7fddfaa00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fddfc06f860 0x7fddfc0b3150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:53962/0 (socket says 172.21.15.96:53962) 2024-05-28T06:20:29.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.218+0000 7fddfaa00640 1 -- 172.21.15.96:0/469355339 learned_addr learned my addr 172.21.15.96:0/469355339 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:20:29.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.218+0000 7fddf3e00640 1 --2- 172.21.15.96:0/469355339 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fddfc0d04a0 0x7fddfc0d0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:29.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.218+0000 7fddfa000640 1 -- 172.21.15.96:0/469355339 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1027891241 0 0) 0x7fddec001110 con 0x7fddfc06f860 2024-05-28T06:20:29.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.218+0000 7fddfa000640 1 -- 172.21.15.96:0/469355339 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdde0003520 con 0x7fddfc06f860 2024-05-28T06:20:29.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.218+0000 7fddfb400640 1 --2- 172.21.15.96:0/469355339 >> v2:172.21.15.123:3300/0 conn(0x7fddfc0ad130 0x7fddfc0ad5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:29.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.219+0000 7fddfa000640 1 -- 172.21.15.96:0/469355339 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2489378623 0 0) 0x7fdde0003520 con 0x7fddfc06f860 2024-05-28T06:20:29.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.219+0000 7fddfa000640 1 -- 172.21.15.96:0/469355339 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fddfc0d04a0 msgr2=0x7fddfc0d0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:29.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.219+0000 7fddfa000640 1 --2- 172.21.15.96:0/469355339 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fddfc0d04a0 0x7fddfc0d0900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:29.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.219+0000 7fddfa000640 1 -- 172.21.15.96:0/469355339 >> v2:172.21.15.123:3300/0 conn(0x7fddfc0ad130 msgr2=0x7fddfc0ad5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:29.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.219+0000 7fddfa000640 1 --2- 172.21.15.96:0/469355339 >> v2:172.21.15.123:3300/0 conn(0x7fddfc0ad130 0x7fddfc0ad5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:29.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.219+0000 7fddfa000640 1 -- 172.21.15.96:0/469355339 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddfc0b3cc0 con 0x7fddfc06f860 2024-05-28T06:20:29.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.219+0000 7fddfa000640 1 -- 172.21.15.96:0/469355339 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7fddec001840 con 0x7fddfc06f860 2024-05-28T06:20:29.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.219+0000 7fddfa000640 1 -- 172.21.15.96:0/469355339 >> v1:172.21.15.96:6789/0 conn(0x7fddfc06f860 legacy=0x7fddfc0b3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:29.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.219+0000 7fddfa000640 1 --2- 172.21.15.96:0/469355339 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdde00039b0 0x7fdde0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:29.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.219+0000 7fddfa000640 1 --2- 172.21.15.96:0/469355339 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdde0004800 0x7fdde0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:29.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.219+0000 7fddfa000640 1 --2- 172.21.15.96:0/469355339 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdde00051c0 0x7fdde000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:29.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.219+0000 7fddfa000640 1 -- 172.21.15.96:0/469355339 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fddfc15cb00 con 0x7fdde00039b0 2024-05-28T06:20:29.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.219+0000 7fddfa000640 1 -- 172.21.15.96:0/469355339 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fdde000bfe0 con 0x7fdde00051c0 2024-05-28T06:20:29.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.219+0000 7fddfa000640 1 -- 172.21.15.96:0/469355339 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fdde000c1e0 con 0x7fdde0004800 2024-05-28T06:20:29.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.219+0000 7fddfb400640 1 --2- 172.21.15.96:0/469355339 >> v2:172.21.15.123:3300/0 conn(0x7fddfc0ad130 0x7fddfc0ad5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-28T06:20:29.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.219+0000 7fddf3e00640 1 --2- 172.21.15.96:0/469355339 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fddfc0d04a0 0x7fddfc0d0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-28T06:20:29.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.219+0000 7fddfaa00640 1 --2- 172.21.15.96:0/469355339 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdde00039b0 0x7fdde0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:29.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.219+0000 7fddfaa00640 1 -- 172.21.15.96:0/469355339 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdde0004800 msgr2=0x7fdde0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:29.218 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.219+0000 7fddfaa00640 1 --2- 172.21.15.96:0/469355339 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdde0004800 0x7fdde0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:29.218 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.219+0000 7fddfaa00640 1 -- 172.21.15.96:0/469355339 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdde00051c0 msgr2=0x7fdde000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:29.218 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.219+0000 7fddfaa00640 1 --2- 172.21.15.96:0/469355339 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdde00051c0 0x7fdde000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:29.218 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.219+0000 7fddfaa00640 1 -- 172.21.15.96:0/469355339 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdde0003520 con 0x7fdde00039b0 2024-05-28T06:20:29.218 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.220+0000 7fddfaa00640 1 --2- 172.21.15.96:0/469355339 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdde00039b0 0x7fdde0003dd0 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7fddec00c320 tx=0x7fdde0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ecd03462ff22241 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:29.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.221+0000 7fddfa000640 1 -- 172.21.15.96:0/469355339 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddec005ad0 con 0x7fdde00039b0 2024-05-28T06:20:29.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.221+0000 7fddfa000640 1 -- 172.21.15.96:0/469355339 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fddec034070 con 0x7fdde00039b0 2024-05-28T06:20:29.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.221+0000 7fddfa000640 1 -- 172.21.15.96:0/469355339 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddec007070 con 0x7fdde00039b0 2024-05-28T06:20:29.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.221+0000 7fde01a00640 1 -- 172.21.15.96:0/469355339 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdde00039b0 msgr2=0x7fdde0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:29.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.221+0000 7fde01a00640 1 --2- 172.21.15.96:0/469355339 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdde00039b0 0x7fdde0003dd0 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7fddec00c320 tx=0x7fdde0005830 comp rx=0 tx=0).stop 2024-05-28T06:20:29.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.221+0000 7fddfb400640 1 -- 172.21.15.96:0/469355339 reap_dead start 2024-05-28T06:20:29.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.221+0000 7fde01a00640 1 -- 172.21.15.96:0/469355339 shutdown_connections 2024-05-28T06:20:29.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.221+0000 7fde01a00640 1 -- 172.21.15.96:0/469355339 >> 172.21.15.96:0/469355339 conn(0x7fddfc06bae0 msgr2=0x7fddfc06bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:29.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.221+0000 7fde01a00640 1 -- 172.21.15.96:0/469355339 shutdown_connections 2024-05-28T06:20:29.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.221+0000 7fde01a00640 1 -- 172.21.15.96:0/469355339 wait complete. 2024-05-28T06:20:29.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.223+0000 7fde01a00640 1 Processor -- start 2024-05-28T06:20:29.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.223+0000 7fde01a00640 1 -- start start 2024-05-28T06:20:29.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.224+0000 7fde01a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdde00039b0 0x7fddfc0997c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:29.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.224+0000 7fde01a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fddfc15f880 0x7fddfc15cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:29.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.224+0000 7fde01a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fddfc152fb0 0x7fddfc153430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:29.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.224+0000 7fde01a00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fddfc0b4560 con 0x7fdde00039b0 2024-05-28T06:20:29.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.224+0000 7fde01a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fddfc0b4280 con 0x7fddfc15f880 2024-05-28T06:20:29.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.224+0000 7fde01a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fddfc0b4840 con 0x7fddfc152fb0 2024-05-28T06:20:29.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.224+0000 7fddfaa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdde00039b0 0x7fddfc0997c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:29.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.225+0000 7fddfaa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdde00039b0 0x7fddfc0997c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51546/0 (socket says 172.21.15.96:51546) 2024-05-28T06:20:29.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.225+0000 7fddfaa00640 1 -- 172.21.15.96:0/1846442499 learned_addr learned my addr 172.21.15.96:0/1846442499 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:20:29.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.225+0000 7fddf3e00640 1 --2- 172.21.15.96:0/1846442499 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fddfc15f880 0x7fddfc15cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:29.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.225+0000 7fddfb400640 1 --2- 172.21.15.96:0/1846442499 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fddfc152fb0 0x7fddfc153430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:29.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.225+0000 7fddfaa00640 1 -- 172.21.15.96:0/1846442499 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fddfc152fb0 msgr2=0x7fddfc153430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:29.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.225+0000 7fddfaa00640 1 --2- 172.21.15.96:0/1846442499 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fddfc152fb0 0x7fddfc153430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:29.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.225+0000 7fddfaa00640 1 -- 172.21.15.96:0/1846442499 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fddfc15f880 msgr2=0x7fddfc15cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:29.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.225+0000 7fddfaa00640 1 --2- 172.21.15.96:0/1846442499 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fddfc15f880 0x7fddfc15cd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:29.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.225+0000 7fddfaa00640 1 -- 172.21.15.96:0/1846442499 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddfc15e6b0 con 0x7fdde00039b0 2024-05-28T06:20:29.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.225+0000 7fddfaa00640 1 --2- 172.21.15.96:0/1846442499 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdde00039b0 0x7fddfc0997c0 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7fddec02af70 tx=0x7fddec001b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:29.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.226+0000 7fddf2a00640 1 -- 172.21.15.96:0/1846442499 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddec007480 con 0x7fdde00039b0 2024-05-28T06:20:29.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.226+0000 7fddf2a00640 1 -- 172.21.15.96:0/1846442499 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fddec007e50 con 0x7fdde00039b0 2024-05-28T06:20:29.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.226+0000 7fde01a00640 1 -- 172.21.15.96:0/1846442499 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fddfc15d610 con 0x7fdde00039b0 2024-05-28T06:20:29.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.226+0000 7fddf2a00640 1 -- 172.21.15.96:0/1846442499 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddec007840 con 0x7fdde00039b0 2024-05-28T06:20:29.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.226+0000 7fde01a00640 1 -- 172.21.15.96:0/1846442499 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fddfc15dbd0 con 0x7fdde00039b0 2024-05-28T06:20:29.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.227+0000 7fde01a00640 1 -- 172.21.15.96:0/1846442499 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fddfc0689e0 con 0x7fdde00039b0 2024-05-28T06:20:29.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.228+0000 7fddf2a00640 1 -- 172.21.15.96:0/1846442499 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 34) ==== 92927+0+0 (secure 0 0 0) 0x7fddec03a410 con 0x7fdde00039b0 2024-05-28T06:20:29.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.229+0000 7fddf2a00640 1 --2- 172.21.15.96:0/1846442499 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fddd806df60 0x7fddd8070420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:29.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.229+0000 7fddf2a00640 1 -- 172.21.15.96:0/1846442499 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 5765+0+0 (secure 0 0 0) 0x7fddec0ab990 con 0x7fdde00039b0 2024-05-28T06:20:29.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.229+0000 7fddf3e00640 1 --2- 172.21.15.96:0/1846442499 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fddd806df60 0x7fddd8070420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:29.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.230+0000 7fddf3e00640 1 --2- 172.21.15.96:0/1846442499 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fddd806df60 0x7fddd8070420 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fddfc072850 tx=0x7fdde4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:29.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.233+0000 7fddf2a00640 1 -- 172.21.15.96:0/1846442499 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fddec049060 con 0x7fdde00039b0 2024-05-28T06:20:29.436 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.440+0000 7fde01a00640 1 -- 172.21.15.96:0/1846442499 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7fddfc06ddb0 con 0x7fddd806df60 2024-05-28T06:20:29.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.444+0000 7fddf2a00640 1 -- 172.21.15.96:0/1846442499 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7fddfc06ddb0 con 0x7fddd806df60 2024-05-28T06:20:29.440 INFO:teuthology.orchestra.run.smithi096.stdout:[ 2024-05-28T06:20:29.440 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:20:29.440 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "cephfs" 2024-05-28T06:20:29.440 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:20:29.440 INFO:teuthology.orchestra.run.smithi096.stdout:] 2024-05-28T06:20:29.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.446+0000 7fde01a00640 1 -- 172.21.15.96:0/1846442499 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fddd806df60 msgr2=0x7fddd8070420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:29.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.446+0000 7fde01a00640 1 --2- 172.21.15.96:0/1846442499 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fddd806df60 0x7fddd8070420 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fddfc072850 tx=0x7fdde4008040 comp rx=0 tx=0).stop 2024-05-28T06:20:29.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.446+0000 7fde01a00640 1 -- 172.21.15.96:0/1846442499 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdde00039b0 msgr2=0x7fddfc0997c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:29.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.446+0000 7fde01a00640 1 --2- 172.21.15.96:0/1846442499 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdde00039b0 0x7fddfc0997c0 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7fddec02af70 tx=0x7fddec001b40 comp rx=0 tx=0).stop 2024-05-28T06:20:29.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.447+0000 7fde01a00640 1 -- 172.21.15.96:0/1846442499 shutdown_connections 2024-05-28T06:20:29.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.447+0000 7fde01a00640 1 --2- 172.21.15.96:0/1846442499 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fddd806df60 0x7fddd8070420 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:29.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.447+0000 7fde01a00640 1 --2- 172.21.15.96:0/1846442499 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fddfc152fb0 0x7fddfc153430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:29.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.447+0000 7fde01a00640 1 --2- 172.21.15.96:0/1846442499 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fddfc15f880 0x7fddfc15cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:29.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.447+0000 7fde01a00640 1 --2- 172.21.15.96:0/1846442499 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdde00039b0 0x7fddfc0997c0 unknown :-1 s=CLOSED pgs=1654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:29.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.447+0000 7fde01a00640 1 -- 172.21.15.96:0/1846442499 >> 172.21.15.96:0/1846442499 conn(0x7fddfc06bae0 msgr2=0x7fddfc0af9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:29.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.447+0000 7fde01a00640 1 -- 172.21.15.96:0/1846442499 shutdown_connections 2024-05-28T06:20:29.444 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.447+0000 7fde01a00640 1 -- 172.21.15.96:0/1846442499 wait complete. 2024-05-28T06:20:29.461 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-05-28T06:20:29.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.668+0000 7f3693e00640 1 Processor -- start 2024-05-28T06:20:29.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.669+0000 7f3693e00640 1 -- start start 2024-05-28T06:20:29.666 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.670+0000 7f3693e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f3694153510 0x7f36940b4d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:29.666 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.670+0000 7f3693e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f36940ccb90 0x7f36940ccff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:29.666 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.670+0000 7f3693e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f369415cb60 0x7f369415cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:29.666 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.670+0000 7f3693e00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f36941538f0 con 0x7f369415cb60 2024-05-28T06:20:29.666 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.670+0000 7f3693e00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f36940cd530 con 0x7f3694153510 2024-05-28T06:20:29.666 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.670+0000 7f3693e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f36940b48c0 con 0x7f36940ccb90 2024-05-28T06:20:29.666 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.670+0000 7f3693400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f369415cb60 0x7f369415cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:29.666 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.670+0000 7f3693400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f369415cb60 0x7f369415cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51552/0 (socket says 172.21.15.96:51552) 2024-05-28T06:20:29.667 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.670+0000 7f3693400640 1 -- 172.21.15.96:0/1864026996 learned_addr learned my addr 172.21.15.96:0/1864026996 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:20:29.667 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.671+0000 7f3692000640 1 --2- 172.21.15.96:0/1864026996 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f36940ccb90 0x7f36940ccff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:29.667 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.671+0000 7f3692a00640 1 --2- 172.21.15.96:0/1864026996 >> v2:172.21.15.123:3300/0 conn(0x7f3694153510 0x7f36940b4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:29.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.671+0000 7f3693400640 1 -- 172.21.15.96:0/1864026996 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f36940ccb90 msgr2=0x7f36940ccff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:29.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.671+0000 7f3693400640 1 --2- 172.21.15.96:0/1864026996 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f36940ccb90 0x7f36940ccff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:29.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.671+0000 7f3693400640 1 -- 172.21.15.96:0/1864026996 >> v2:172.21.15.123:3300/0 conn(0x7f3694153510 msgr2=0x7f36940b4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:29.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.671+0000 7f3693400640 1 --2- 172.21.15.96:0/1864026996 >> v2:172.21.15.123:3300/0 conn(0x7f3694153510 0x7f36940b4d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:29.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.671+0000 7f3693400640 1 -- 172.21.15.96:0/1864026996 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f369415d900 con 0x7f369415cb60 2024-05-28T06:20:29.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.672+0000 7f3693400640 1 --2- 172.21.15.96:0/1864026996 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f369415cb60 0x7f369415cff0 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7f36940b3970 tx=0x7f368c02a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d34c33ed129fc69 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:29.669 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.672+0000 7f3691600640 1 -- 172.21.15.96:0/1864026996 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f368c037070 con 0x7f369415cb60 2024-05-28T06:20:29.669 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.672+0000 7f3691600640 1 -- 172.21.15.96:0/1864026996 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f368c032440 con 0x7f369415cb60 2024-05-28T06:20:29.669 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.672+0000 7f3691600640 1 -- 172.21.15.96:0/1864026996 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f368c032810 con 0x7f369415cb60 2024-05-28T06:20:29.669 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.673+0000 7f3693e00640 1 -- 172.21.15.96:0/1864026996 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f369415cb60 msgr2=0x7f369415cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:29.669 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.673+0000 7f3693e00640 1 --2- 172.21.15.96:0/1864026996 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f369415cb60 0x7f369415cff0 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7f36940b3970 tx=0x7f368c02a140 comp rx=0 tx=0).stop 2024-05-28T06:20:29.669 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.673+0000 7f3693e00640 1 -- 172.21.15.96:0/1864026996 shutdown_connections 2024-05-28T06:20:29.669 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.673+0000 7f3693e00640 1 --2- 172.21.15.96:0/1864026996 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f369415cb60 0x7f369415cff0 unknown :-1 s=CLOSED pgs=1655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:29.669 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.673+0000 7f3693e00640 1 --2- 172.21.15.96:0/1864026996 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f36940ccb90 0x7f36940ccff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:29.670 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.673+0000 7f3693e00640 1 --2- 172.21.15.96:0/1864026996 >> v2:172.21.15.123:3300/0 conn(0x7f3694153510 0x7f36940b4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:29.670 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.673+0000 7f3693e00640 1 -- 172.21.15.96:0/1864026996 >> 172.21.15.96:0/1864026996 conn(0x7f369406b900 msgr2=0x7f369406bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:29.670 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.673+0000 7f3693e00640 1 -- 172.21.15.96:0/1864026996 shutdown_connections 2024-05-28T06:20:29.670 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.673+0000 7f3693e00640 1 -- 172.21.15.96:0/1864026996 wait complete. 2024-05-28T06:20:29.670 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.675+0000 7f3693e00640 1 Processor -- start 2024-05-28T06:20:29.670 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.675+0000 7f3693e00640 1 -- start start 2024-05-28T06:20:29.672 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.676+0000 7f3693e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f36940ccb90 0x7f3694149770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:29.672 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.676+0000 7f3693e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3694153510 0x7f3694071a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:29.672 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.676+0000 7f3693e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f369415cb60 0x7f3694148930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:29.672 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.676+0000 7f3693e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f36940b4550 con 0x7f3694153510 2024-05-28T06:20:29.672 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.676+0000 7f3693e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f36940b4270 con 0x7f369415cb60 2024-05-28T06:20:29.672 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.676+0000 7f3693e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f369415d530 con 0x7f36940ccb90 2024-05-28T06:20:29.672 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.676+0000 7f3692000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3694153510 0x7f3694071a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:29.672 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.676+0000 7f3692000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3694153510 0x7f3694071a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51562/0 (socket says 172.21.15.96:51562) 2024-05-28T06:20:29.673 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.676+0000 7f3692000640 1 -- 172.21.15.96:0/2442149743 learned_addr learned my addr 172.21.15.96:0/2442149743 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:20:29.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.677+0000 7f3692a00640 1 --2- 172.21.15.96:0/2442149743 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f36940ccb90 0x7f3694149770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:29.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.677+0000 7f3693400640 1 --2- 172.21.15.96:0/2442149743 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f369415cb60 0x7f3694148930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:29.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.677+0000 7f3692000640 1 -- 172.21.15.96:0/2442149743 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f36940ccb90 msgr2=0x7f3694149770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:29.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.677+0000 7f3692000640 1 --2- 172.21.15.96:0/2442149743 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f36940ccb90 0x7f3694149770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:29.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.677+0000 7f3692000640 1 -- 172.21.15.96:0/2442149743 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f369415cb60 msgr2=0x7f3694148930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:29.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.677+0000 7f3692000640 1 --2- 172.21.15.96:0/2442149743 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f369415cb60 0x7f3694148930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:29.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.677+0000 7f3692000640 1 -- 172.21.15.96:0/2442149743 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3694071fc0 con 0x7f3694153510 2024-05-28T06:20:29.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.677+0000 7f3692a00640 1 --2- 172.21.15.96:0/2442149743 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f36940ccb90 0x7f3694149770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:20:29.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.677+0000 7f3692000640 1 --2- 172.21.15.96:0/2442149743 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3694153510 0x7f3694071a80 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7f367c007ef0 tx=0x7f367c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:29.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.677+0000 7f368aa00640 1 -- 172.21.15.96:0/2442149743 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f367c014620 con 0x7f3694153510 2024-05-28T06:20:29.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.677+0000 7f368aa00640 1 -- 172.21.15.96:0/2442149743 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f367c019070 con 0x7f3694153510 2024-05-28T06:20:29.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.677+0000 7f368aa00640 1 -- 172.21.15.96:0/2442149743 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f367c014a00 con 0x7f3694153510 2024-05-28T06:20:29.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.677+0000 7f3693e00640 1 -- 172.21.15.96:0/2442149743 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3694072260 con 0x7f3694153510 2024-05-28T06:20:29.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.678+0000 7f3693e00640 1 -- 172.21.15.96:0/2442149743 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3694072820 con 0x7f3694153510 2024-05-28T06:20:29.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.679+0000 7f3693e00640 1 -- 172.21.15.96:0/2442149743 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36940681e0 con 0x7f3694153510 2024-05-28T06:20:29.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.680+0000 7f368aa00640 1 -- 172.21.15.96:0/2442149743 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 34) ==== 92927+0+0 (secure 0 0 0) 0x7f367c014ba0 con 0x7f3694153510 2024-05-28T06:20:29.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.680+0000 7f368aa00640 1 --2- 172.21.15.96:0/2442149743 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f367006ddf0 0x7f36700702b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:29.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.681+0000 7f368aa00640 1 -- 172.21.15.96:0/2442149743 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 5765+0+0 (secure 0 0 0) 0x7f367c08b530 con 0x7f3694153510 2024-05-28T06:20:29.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.681+0000 7f3692a00640 1 --2- 172.21.15.96:0/2442149743 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f367006ddf0 0x7f36700702b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:29.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.682+0000 7f3692a00640 1 --2- 172.21.15.96:0/2442149743 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f367006ddf0 0x7f36700702b0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f3684006fd0 tx=0x7f3684008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:29.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.685+0000 7f368aa00640 1 -- 172.21.15.96:0/2442149743 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f36940681e0 con 0x7f3694153510 2024-05-28T06:20:29.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.883+0000 7f3693e00640 1 -- 172.21.15.96:0/2442149743 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f36940689e0 con 0x7f3694153510 2024-05-28T06:20:29.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.884+0000 7f368aa00640 1 -- 172.21.15.96:0/2442149743 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v24)=0 v24) ==== 122+0+0 (secure 0 0 0) 0x7f367c05afd0 con 0x7f3694153510 2024-05-28T06:20:29.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.887+0000 7f3693e00640 1 -- 172.21.15.96:0/2442149743 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f367006ddf0 msgr2=0x7f36700702b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:29.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.887+0000 7f3693e00640 1 --2- 172.21.15.96:0/2442149743 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f367006ddf0 0x7f36700702b0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f3684006fd0 tx=0x7f3684008040 comp rx=0 tx=0).stop 2024-05-28T06:20:29.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.887+0000 7f3693e00640 1 -- 172.21.15.96:0/2442149743 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3694153510 msgr2=0x7f3694071a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:29.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.887+0000 7f3693e00640 1 --2- 172.21.15.96:0/2442149743 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3694153510 0x7f3694071a80 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7f367c007ef0 tx=0x7f367c00e4b0 comp rx=0 tx=0).stop 2024-05-28T06:20:29.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.887+0000 7f3693e00640 1 -- 172.21.15.96:0/2442149743 shutdown_connections 2024-05-28T06:20:29.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.887+0000 7f3693e00640 1 --2- 172.21.15.96:0/2442149743 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f367006ddf0 0x7f36700702b0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:29.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.887+0000 7f3693e00640 1 --2- 172.21.15.96:0/2442149743 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f369415cb60 0x7f3694148930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:29.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.887+0000 7f3693e00640 1 --2- 172.21.15.96:0/2442149743 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3694153510 0x7f3694071a80 unknown :-1 s=CLOSED pgs=1656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:29.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.887+0000 7f3693e00640 1 --2- 172.21.15.96:0/2442149743 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f36940ccb90 0x7f3694149770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:29.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.887+0000 7f3693e00640 1 -- 172.21.15.96:0/2442149743 >> 172.21.15.96:0/2442149743 conn(0x7f369406b900 msgr2=0x7f36940d00f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:29.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.887+0000 7f3693e00640 1 -- 172.21.15.96:0/2442149743 shutdown_connections 2024-05-28T06:20:29.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:29.887+0000 7f3693e00640 1 -- 172.21.15.96:0/2442149743 wait complete. 2024-05-28T06:20:29.903 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv1 --mode=777 2024-05-28T06:20:30.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.110+0000 7fb006800640 1 Processor -- start 2024-05-28T06:20:30.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.110+0000 7fb006800640 1 -- start start 2024-05-28T06:20:30.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.111+0000 7fb006800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fb00006ea70 0x7fb0000aeb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:30.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.111+0000 7fb006800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fb00006dd10 0x7fb0000ad1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:30.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.111+0000 7fb006800640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb00015bb10 con 0x7fb0000b31c0 2024-05-28T06:20:30.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.111+0000 7fb006800640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fb0000ae6b0 con 0x7fb00006dd10 2024-05-28T06:20:30.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.111+0000 7fb006800640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fb0000adaa0 con 0x7fb00006ea70 2024-05-28T06:20:30.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.112+0000 7faffea00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fb00006dd10 0x7fb0000ad1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:30.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.112+0000 7faffea00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fb00006dd10 0x7fb0000ad1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51572/0 (socket says 172.21.15.96:51572) 2024-05-28T06:20:30.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.112+0000 7faffea00640 1 -- 172.21.15.96:0/3282092076 learned_addr learned my addr 172.21.15.96:0/3282092076 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:20:30.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.112+0000 7fafff400640 1 --2- 172.21.15.96:0/3282092076 >> v2:172.21.15.123:3300/0 conn(0x7fb00006ea70 0x7fb0000aeb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:30.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.112+0000 7faffea00640 1 -- 172.21.15.96:0/3282092076 >> v2:172.21.15.123:3300/0 conn(0x7fb00006ea70 msgr2=0x7fb0000aeb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:30.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.112+0000 7faffea00640 1 --2- 172.21.15.96:0/3282092076 >> v2:172.21.15.123:3300/0 conn(0x7fb00006ea70 0x7fb0000aeb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:30.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.112+0000 7faffea00640 1 -- 172.21.15.96:0/3282092076 >> v1:172.21.15.123:6789/0 conn(0x7fb0000b31c0 legacy=0x7fb0000ca980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:30.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.112+0000 7faffea00640 1 -- 172.21.15.96:0/3282092076 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0000add80 con 0x7fb00006dd10 2024-05-28T06:20:30.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.113+0000 7faffea00640 1 --2- 172.21.15.96:0/3282092076 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb00006dd10 0x7fb0000ad1a0 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7fb00006f280 tx=0x7fafec02a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22d316d595b4813c server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:30.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.113+0000 7faffe000640 1 -- 172.21.15.96:0/3282092076 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafec037070 con 0x7fb00006dd10 2024-05-28T06:20:30.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.114+0000 7faffe000640 1 -- 172.21.15.96:0/3282092076 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fafec032440 con 0x7fb00006dd10 2024-05-28T06:20:30.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.114+0000 7faffe000640 1 -- 172.21.15.96:0/3282092076 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafec032810 con 0x7fb00006dd10 2024-05-28T06:20:30.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.114+0000 7fb006800640 1 -- 172.21.15.96:0/3282092076 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb00006dd10 msgr2=0x7fb0000ad1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:30.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.114+0000 7fb006800640 1 --2- 172.21.15.96:0/3282092076 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb00006dd10 0x7fb0000ad1a0 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7fb00006f280 tx=0x7fafec02a230 comp rx=0 tx=0).stop 2024-05-28T06:20:30.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.114+0000 7fb006800640 1 -- 172.21.15.96:0/3282092076 shutdown_connections 2024-05-28T06:20:30.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.114+0000 7fb006800640 1 --2- 172.21.15.96:0/3282092076 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb00006dd10 0x7fb0000ad1a0 unknown :-1 s=CLOSED pgs=1657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:30.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.114+0000 7fb006800640 1 --2- 172.21.15.96:0/3282092076 >> v2:172.21.15.123:3300/0 conn(0x7fb00006ea70 0x7fb0000aeb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:30.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.114+0000 7fb006800640 1 -- 172.21.15.96:0/3282092076 >> 172.21.15.96:0/3282092076 conn(0x7fb0000b20a0 msgr2=0x7fb0000b24b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:30.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.114+0000 7fb006800640 1 -- 172.21.15.96:0/3282092076 shutdown_connections 2024-05-28T06:20:30.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.114+0000 7fb006800640 1 -- 172.21.15.96:0/3282092076 wait complete. 2024-05-28T06:20:30.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.116+0000 7fb006800640 1 Processor -- start 2024-05-28T06:20:30.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.116+0000 7fb006800640 1 -- start start 2024-05-28T06:20:30.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.117+0000 7fb006800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb00006dd10 0x7fb000099a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:30.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.117+0000 7fb006800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb00006ea70 0x7fb00015a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:30.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.117+0000 7fb006800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb0000b31c0 0x7fb00015ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:30.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.117+0000 7fb006800640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb00006ee50 con 0x7fb00006dd10 2024-05-28T06:20:30.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.117+0000 7fb006800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fb0000ae340 con 0x7fb00006ea70 2024-05-28T06:20:30.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.117+0000 7fb006800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fb0000ad6e0 con 0x7fb0000b31c0 2024-05-28T06:20:30.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.117+0000 7fafff400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb00006dd10 0x7fb000099a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:30.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.118+0000 7fafff400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb00006dd10 0x7fb000099a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51576/0 (socket says 172.21.15.96:51576) 2024-05-28T06:20:30.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.118+0000 7fafff400640 1 -- 172.21.15.96:0/864258521 learned_addr learned my addr 172.21.15.96:0/864258521 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:20:30.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.118+0000 7fafff400640 1 -- 172.21.15.96:0/864258521 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb0000b31c0 msgr2=0x7fb00015ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:30.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.118+0000 7fafffe00640 1 --2- 172.21.15.96:0/864258521 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb0000b31c0 0x7fb00015ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:30.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.118+0000 7faffea00640 1 --2- 172.21.15.96:0/864258521 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb00006ea70 0x7fb00015a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:30.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.118+0000 7fafff400640 1 --2- 172.21.15.96:0/864258521 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb0000b31c0 0x7fb00015ae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:30.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.118+0000 7fafff400640 1 -- 172.21.15.96:0/864258521 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb00006ea70 msgr2=0x7fb00015a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:30.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.118+0000 7fafff400640 1 --2- 172.21.15.96:0/864258521 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb00006ea70 0x7fb00015a940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:30.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.118+0000 7fafff400640 1 -- 172.21.15.96:0/864258521 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb000153020 con 0x7fb00006dd10 2024-05-28T06:20:30.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.118+0000 7fafffe00640 1 --2- 172.21.15.96:0/864258521 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb0000b31c0 0x7fb00015ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:20:30.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.118+0000 7fafff400640 1 --2- 172.21.15.96:0/864258521 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb00006dd10 0x7fb000099a10 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7faff000d620 tx=0x7faff000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:30.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.119+0000 7faff7400640 1 -- 172.21.15.96:0/864258521 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faff000b620 con 0x7fb00006dd10 2024-05-28T06:20:30.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.119+0000 7faff7400640 1 -- 172.21.15.96:0/864258521 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faff0019070 con 0x7fb00006dd10 2024-05-28T06:20:30.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.119+0000 7faff7400640 1 -- 172.21.15.96:0/864258521 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faff000ba00 con 0x7fb00006dd10 2024-05-28T06:20:30.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.119+0000 7fb006800640 1 -- 172.21.15.96:0/864258521 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb00015e340 con 0x7fb00006dd10 2024-05-28T06:20:30.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.119+0000 7fb006800640 1 -- 172.21.15.96:0/864258521 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb00015e900 con 0x7fb00006dd10 2024-05-28T06:20:30.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.120+0000 7fb006800640 1 -- 172.21.15.96:0/864258521 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0000757d0 con 0x7fb00006dd10 2024-05-28T06:20:30.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.121+0000 7faff7400640 1 -- 172.21.15.96:0/864258521 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 34) ==== 92927+0+0 (secure 0 0 0) 0x7faff000bba0 con 0x7fb00006dd10 2024-05-28T06:20:30.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.122+0000 7faff7400640 1 --2- 172.21.15.96:0/864258521 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fafe006ddc0 0x7fafe0070280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:30.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.122+0000 7faff7400640 1 -- 172.21.15.96:0/864258521 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 5765+0+0 (secure 0 0 0) 0x7faff008b030 con 0x7fb00006dd10 2024-05-28T06:20:30.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.122+0000 7faffea00640 1 --2- 172.21.15.96:0/864258521 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fafe006ddc0 0x7fafe0070280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:30.119 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.124+0000 7faffea00640 1 --2- 172.21.15.96:0/864258521 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fafe006ddc0 0x7fafe0070280 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fafec034000 tx=0x7fafec02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:30.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.126+0000 7faff7400640 1 -- 172.21.15.96:0/864258521 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7faff005aad0 con 0x7fb00006dd10 2024-05-28T06:20:30.351 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.355+0000 7fb006800640 1 -- 172.21.15.96:0/864258521 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv1", "mode": "777", "target": ["mon-mgr", ""]}) -- 0x7fb0000689e0 con 0x7fafe006ddc0 2024-05-28T06:20:30.367 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:20:30.372+0000 7fea59e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:20:30.368 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:20:30.372+0000 7fea59e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:20:30.368 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:20:30.372+0000 7fea59e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:20:30.368 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:20:30.372+0000 7fea59e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:20:30.368 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:20:30.372+0000 7fea59e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:20:30.436 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.441+0000 7faff7400640 1 -- 172.21.15.96:0/864258521 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fb0000689e0 con 0x7fafe006ddc0 2024-05-28T06:20:30.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.444+0000 7fb006800640 1 -- 172.21.15.96:0/864258521 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fafe006ddc0 msgr2=0x7fafe0070280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:30.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.444+0000 7fb006800640 1 --2- 172.21.15.96:0/864258521 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fafe006ddc0 0x7fafe0070280 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fafec034000 tx=0x7fafec02e000 comp rx=0 tx=0).stop 2024-05-28T06:20:30.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.444+0000 7fb006800640 1 -- 172.21.15.96:0/864258521 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb00006dd10 msgr2=0x7fb000099a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:30.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.444+0000 7fb006800640 1 --2- 172.21.15.96:0/864258521 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb00006dd10 0x7fb000099a10 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7faff000d620 tx=0x7faff000dae0 comp rx=0 tx=0).stop 2024-05-28T06:20:30.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.444+0000 7fb006800640 1 -- 172.21.15.96:0/864258521 shutdown_connections 2024-05-28T06:20:30.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.444+0000 7fb006800640 1 --2- 172.21.15.96:0/864258521 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fafe006ddc0 0x7fafe0070280 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:30.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.444+0000 7fb006800640 1 --2- 172.21.15.96:0/864258521 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb0000b31c0 0x7fb00015ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:30.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.444+0000 7fb006800640 1 --2- 172.21.15.96:0/864258521 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb00006ea70 0x7fb00015a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:30.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.444+0000 7fb006800640 1 --2- 172.21.15.96:0/864258521 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb00006dd10 0x7fb000099a10 unknown :-1 s=CLOSED pgs=1658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:30.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.444+0000 7fb006800640 1 -- 172.21.15.96:0/864258521 >> 172.21.15.96:0/864258521 conn(0x7fb0000b20a0 msgr2=0x7fb000071b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:30.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.444+0000 7fb006800640 1 -- 172.21.15.96:0/864258521 shutdown_connections 2024-05-28T06:20:30.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.444+0000 7fb006800640 1 -- 172.21.15.96:0/864258521 wait complete. 2024-05-28T06:20:30.461 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-05-28T06:20:30.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.672+0000 7f483fa00640 1 Processor -- start 2024-05-28T06:20:30.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.673+0000 7f483fa00640 1 -- start start 2024-05-28T06:20:30.670 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.674+0000 7f483fa00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f48380d0640 0x7f48380d0a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:30.670 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.674+0000 7f483fa00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f48380725e0 0x7f48380ae650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:30.670 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.674+0000 7f483fa00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48380ade90 con 0x7f483815b6f0 2024-05-28T06:20:30.670 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.674+0000 7f483fa00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f4838071c40 con 0x7f48380725e0 2024-05-28T06:20:30.670 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.674+0000 7f483fa00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f48380d0f60 con 0x7f48380d0640 2024-05-28T06:20:30.671 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.675+0000 7f483da00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f48380725e0 0x7f48380ae650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:30.671 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.675+0000 7f483d000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f48380d0640 0x7f48380d0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:30.671 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.675+0000 7f483d000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f48380d0640 0x7f48380d0a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.96:40672/0 (socket says 172.21.15.96:40672) 2024-05-28T06:20:30.672 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.675+0000 7f483da00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f48380725e0 0x7f48380ae650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.96:39256/0 (socket says 172.21.15.96:39256) 2024-05-28T06:20:30.672 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.675+0000 7f483d000640 1 -- 172.21.15.96:0/3481427981 learned_addr learned my addr 172.21.15.96:0/3481427981 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:20:30.672 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.676+0000 7f483da00640 1 -- 172.21.15.96:0/3481427981 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f48380d0640 msgr2=0x7f48380d0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:30.672 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.676+0000 7f483da00640 1 --2- 172.21.15.96:0/3481427981 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f48380d0640 0x7f48380d0a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:30.672 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.676+0000 7f483da00640 1 -- 172.21.15.96:0/3481427981 >> v1:172.21.15.123:6789/0 conn(0x7f483815b6f0 legacy=0x7f483806e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:30.672 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.676+0000 7f483da00640 1 -- 172.21.15.96:0/3481427981 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4838071e70 con 0x7f48380725e0 2024-05-28T06:20:30.672 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.677+0000 7f483da00640 1 --2- 172.21.15.96:0/3481427981 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f48380725e0 0x7f48380ae650 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f4830009d90 tx=0x7f483002c870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cfd0bb6be24153a server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:30.673 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.678+0000 7f4837400640 1 -- 172.21.15.96:0/3481427981 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4830032770 con 0x7f48380725e0 2024-05-28T06:20:30.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.678+0000 7f4837400640 1 -- 172.21.15.96:0/3481427981 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4830038070 con 0x7f48380725e0 2024-05-28T06:20:30.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.678+0000 7f4837400640 1 -- 172.21.15.96:0/3481427981 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4830032b30 con 0x7f48380725e0 2024-05-28T06:20:30.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.679+0000 7f483fa00640 1 -- 172.21.15.96:0/3481427981 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f48380725e0 msgr2=0x7f48380ae650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:30.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.679+0000 7f483fa00640 1 --2- 172.21.15.96:0/3481427981 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f48380725e0 0x7f48380ae650 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f4830009d90 tx=0x7f483002c870 comp rx=0 tx=0).stop 2024-05-28T06:20:30.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.679+0000 7f483fa00640 1 -- 172.21.15.96:0/3481427981 shutdown_connections 2024-05-28T06:20:30.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.679+0000 7f483fa00640 1 --2- 172.21.15.96:0/3481427981 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f48380725e0 0x7f48380ae650 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:30.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.679+0000 7f483fa00640 1 --2- 172.21.15.96:0/3481427981 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f48380d0640 0x7f48380d0a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:30.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.679+0000 7f483fa00640 1 -- 172.21.15.96:0/3481427981 >> 172.21.15.96:0/3481427981 conn(0x7f483806bd20 msgr2=0x7f483806c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:30.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.679+0000 7f483fa00640 1 -- 172.21.15.96:0/3481427981 shutdown_connections 2024-05-28T06:20:30.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.679+0000 7f483fa00640 1 -- 172.21.15.96:0/3481427981 wait complete. 2024-05-28T06:20:30.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.681+0000 7f483fa00640 1 Processor -- start 2024-05-28T06:20:30.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.681+0000 7f483fa00640 1 -- start start 2024-05-28T06:20:30.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.682+0000 7f483fa00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f48380725e0 0x7f4838147910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:30.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.683+0000 7f483fa00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f48380d0640 0x7f48380b4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:30.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.683+0000 7f483fa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f483815b6f0 0x7f48380b0b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:30.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.683+0000 7f483fa00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f48380aeb90 con 0x7f483815b6f0 2024-05-28T06:20:30.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.683+0000 7f483fa00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f48380ae130 con 0x7f48380d0640 2024-05-28T06:20:30.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.683+0000 7f483fa00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f48380aed10 con 0x7f48380725e0 2024-05-28T06:20:30.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.683+0000 7f483da00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f483815b6f0 0x7f48380b0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:30.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.683+0000 7f483da00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f483815b6f0 0x7f48380b0b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51578/0 (socket says 172.21.15.96:51578) 2024-05-28T06:20:30.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.683+0000 7f483da00640 1 -- 172.21.15.96:0/2084456030 learned_addr learned my addr 172.21.15.96:0/2084456030 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:20:30.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.683+0000 7f483d000640 1 --2- 172.21.15.96:0/2084456030 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f48380725e0 0x7f4838147910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:30.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.683+0000 7f4837e00640 1 --2- 172.21.15.96:0/2084456030 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f48380d0640 0x7f48380b4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:30.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.684+0000 7f483da00640 1 -- 172.21.15.96:0/2084456030 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f48380725e0 msgr2=0x7f4838147910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:30.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.684+0000 7f483da00640 1 --2- 172.21.15.96:0/2084456030 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f48380725e0 0x7f4838147910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:30.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.684+0000 7f483da00640 1 -- 172.21.15.96:0/2084456030 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f48380d0640 msgr2=0x7f48380b4ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:30.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.684+0000 7f483da00640 1 --2- 172.21.15.96:0/2084456030 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f48380d0640 0x7f48380b4ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:30.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.684+0000 7f483da00640 1 -- 172.21.15.96:0/2084456030 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4838147e50 con 0x7f483815b6f0 2024-05-28T06:20:30.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.684+0000 7f483da00640 1 --2- 172.21.15.96:0/2084456030 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f483815b6f0 0x7f48380b0b40 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7f4830009d90 tx=0x7f4830004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:30.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.684+0000 7f4834c00640 1 -- 172.21.15.96:0/2084456030 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f483004a070 con 0x7f483815b6f0 2024-05-28T06:20:30.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.684+0000 7f4834c00640 1 -- 172.21.15.96:0/2084456030 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4830045410 con 0x7f483815b6f0 2024-05-28T06:20:30.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.684+0000 7f483fa00640 1 -- 172.21.15.96:0/2084456030 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48380b38a0 con 0x7f483815b6f0 2024-05-28T06:20:30.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.684+0000 7f4834c00640 1 -- 172.21.15.96:0/2084456030 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4830045800 con 0x7f483815b6f0 2024-05-28T06:20:30.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.685+0000 7f483fa00640 1 -- 172.21.15.96:0/2084456030 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48380b3e60 con 0x7f483815b6f0 2024-05-28T06:20:30.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.686+0000 7f483fa00640 1 -- 172.21.15.96:0/2084456030 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48380689e0 con 0x7f483815b6f0 2024-05-28T06:20:30.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.687+0000 7f4834c00640 1 -- 172.21.15.96:0/2084456030 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 34) ==== 92927+0+0 (secure 0 0 0) 0x7f483004d070 con 0x7f483815b6f0 2024-05-28T06:20:30.683 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.688+0000 7f4834c00640 1 --2- 172.21.15.96:0/2084456030 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f481806ddf0 0x7f48180702b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:30.683 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.688+0000 7f4834c00640 1 -- 172.21.15.96:0/2084456030 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 5765+0+0 (secure 0 0 0) 0x7f48300ad0f0 con 0x7f483815b6f0 2024-05-28T06:20:30.684 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.688+0000 7f483d000640 1 --2- 172.21.15.96:0/2084456030 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f481806ddf0 0x7f48180702b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:30.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.690+0000 7f483d000640 1 --2- 172.21.15.96:0/2084456030 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f481806ddf0 0x7f48180702b0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f483814e730 tx=0x7f4828008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:30.689 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.694+0000 7f4834c00640 1 -- 172.21.15.96:0/2084456030 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f483007d090 con 0x7f483815b6f0 2024-05-28T06:20:30.921 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.926+0000 7f483fa00640 1 -- 172.21.15.96:0/2084456030 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) -- 0x7f483806f810 con 0x7f481806ddf0 2024-05-28T06:20:30.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.937+0000 7f4834c00640 1 -- 172.21.15.96:0/2084456030 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+59 (secure 0 0 0) 0x7f483806f810 con 0x7f481806ddf0 2024-05-28T06:20:30.933 INFO:teuthology.orchestra.run.smithi096.stdout:/volumes/_nogroup/sv1/df1f0247-87ab-4f07-abc5-b5b9370c3a00 2024-05-28T06:20:30.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.940+0000 7f483fa00640 1 -- 172.21.15.96:0/2084456030 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f481806ddf0 msgr2=0x7f48180702b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:30.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.940+0000 7f483fa00640 1 --2- 172.21.15.96:0/2084456030 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f481806ddf0 0x7f48180702b0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f483814e730 tx=0x7f4828008040 comp rx=0 tx=0).stop 2024-05-28T06:20:30.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.940+0000 7f483fa00640 1 -- 172.21.15.96:0/2084456030 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f483815b6f0 msgr2=0x7f48380b0b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:30.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.940+0000 7f483fa00640 1 --2- 172.21.15.96:0/2084456030 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f483815b6f0 0x7f48380b0b40 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7f4830009d90 tx=0x7f4830004a40 comp rx=0 tx=0).stop 2024-05-28T06:20:30.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.940+0000 7f483fa00640 1 -- 172.21.15.96:0/2084456030 shutdown_connections 2024-05-28T06:20:30.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.940+0000 7f483fa00640 1 --2- 172.21.15.96:0/2084456030 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f481806ddf0 0x7f48180702b0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:30.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.940+0000 7f483fa00640 1 --2- 172.21.15.96:0/2084456030 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f483815b6f0 0x7f48380b0b40 unknown :-1 s=CLOSED pgs=1659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:30.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.940+0000 7f483fa00640 1 --2- 172.21.15.96:0/2084456030 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f48380d0640 0x7f48380b4ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:30.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.940+0000 7f483fa00640 1 --2- 172.21.15.96:0/2084456030 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f48380725e0 0x7f4838147910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:30.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.940+0000 7f483fa00640 1 -- 172.21.15.96:0/2084456030 >> 172.21.15.96:0/2084456030 conn(0x7f483806bd20 msgr2=0x7f48380afe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:30.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.941+0000 7f483fa00640 1 -- 172.21.15.96:0/2084456030 shutdown_connections 2024-05-28T06:20:30.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:30.941+0000 7f483fa00640 1 -- 172.21.15.96:0/2084456030 wait complete. 2024-05-28T06:20:30.955 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-05-28T06:20:31.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:31.159+0000 7ff3a6800640 1 Processor -- start 2024-05-28T06:20:31.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:31.159+0000 7ff3a6800640 1 -- start start 2024-05-28T06:20:31.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:31.160+0000 7ff3a6800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff3a015c3f0 0x7ff3a015ee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:31.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:31.160+0000 7ff3a6800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7ff3a00ca6a0 0x7ff3a00caaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:31.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:31.160+0000 7ff3a6800640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3a015f950 con 0x7ff3a015f3d0 2024-05-28T06:20:31.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:31.160+0000 7ff3a6800640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7ff3a00cb030 con 0x7ff3a00ca6a0 2024-05-28T06:20:31.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:31.160+0000 7ff3a6800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ff3a00b1020 con 0x7ff3a015c3f0 2024-05-28T06:20:31.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:31.161+0000 7ff39f400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff3a015c3f0 0x7ff3a015ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:31.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:31.161+0000 7ff39fe00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7ff3a00ca6a0 0x7ff3a00caaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:31.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:31.161+0000 7ff39fe00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7ff3a00ca6a0 0x7ff3a00caaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.96:39276/0 (socket says 172.21.15.96:39276) 2024-05-28T06:20:31.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:31.161+0000 7ff39f400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff3a015c3f0 0x7ff3a015ee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.96:40684/0 (socket says 172.21.15.96:40684) 2024-05-28T06:20:31.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:31.161+0000 7ff39fe00640 1 -- 172.21.15.96:0/3656010621 learned_addr learned my addr 172.21.15.96:0/3656010621 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:20:31.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:31.162+0000 7ff39fe00640 1 -- 172.21.15.96:0/3656010621 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff3a015c3f0 msgr2=0x7ff3a015ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:20:31.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:31.162+0000 7ff39fe00640 1 --2- 172.21.15.96:0/3656010621 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff3a015c3f0 0x7ff3a015ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:31.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:31.162+0000 7ff39fe00640 1 -- 172.21.15.96:0/3656010621 >> v1:172.21.15.123:6789/0 conn(0x7ff3a015f3d0 legacy=0x7ff3a00b14b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:31.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:31.162+0000 7ff39fe00640 1 -- 172.21.15.96:0/3656010621 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3a015fc30 con 0x7ff3a00ca6a0 2024-05-28T06:20:31.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:31.162+0000 7ff39f400640 1 --2- 172.21.15.96:0/3656010621 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff3a015c3f0 0x7ff3a015ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-28T06:20:31.159 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:31.163+0000 7ff39fe00640 1 --2- 172.21.15.96:0/3656010621 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3a00ca6a0 0x7ff3a00caaf0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7ff398009e50 tx=0x7ff39802a7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8534d32b8956a141 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:31.159 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:31.164+0000 7ff39e000640 1 -- 172.21.15.96:0/3656010621 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff398036070 con 0x7ff3a00ca6a0 2024-05-28T06:20:31.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:31.164+0000 7ff39e000640 1 -- 172.21.15.96:0/3656010621 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff398031600 con 0x7ff3a00ca6a0 2024-05-28T06:20:31.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:31.164+0000 7ff39e000640 1 -- 172.21.15.96:0/3656010621 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3980319d0 con 0x7ff3a00ca6a0 2024-05-28T06:20:31.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:31.164+0000 7ff3a6800640 1 -- 172.21.15.96:0/3656010621 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3a00ca6a0 msgr2=0x7ff3a00caaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:31.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:31.165+0000 7ff3a6800640 1 --2- 172.21.15.96:0/3656010621 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3a00ca6a0 0x7ff3a00caaf0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7ff398009e50 tx=0x7ff39802a7a0 comp rx=0 tx=0).stop 2024-05-28T06:20:31.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:31.165+0000 7ff3a6800640 1 -- 172.21.15.96:0/3656010621 shutdown_connections 2024-05-28T06:20:31.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:31.165+0000 7ff3a6800640 1 --2- 172.21.15.96:0/3656010621 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3a00ca6a0 0x7ff3a00caaf0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:31.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:31.165+0000 7ff3a6800640 1 --2- 172.21.15.96:0/3656010621 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff3a015c3f0 0x7ff3a015ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:31.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:31.165+0000 7ff3a6800640 1 -- 172.21.15.96:0/3656010621 >> 172.21.15.96:0/3656010621 conn(0x7ff3a009aa30 msgr2=0x7ff3a00b3020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:31.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:31.165+0000 7ff3a6800640 1 -- 172.21.15.96:0/3656010621 shutdown_connections 2024-05-28T06:20:31.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:31.165+0000 7ff3a6800640 1 -- 172.21.15.96:0/3656010621 wait complete. 2024-05-28T06:20:31.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:31.166+0000 7ff3a6800640 1 Processor -- start 2024-05-28T06:20:31.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:31.167+0000 7ff3a6800640 1 -- start start 2024-05-28T06:20:31.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:31.168+0000 7ff3a6800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3a00ca6a0 0x7ff3a0159300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:31.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:31.168+0000 7ff3a6800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff3a015c3f0 0x7ff3a015b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:31.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:31.168+0000 7ff3a6800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff3a015f3d0 0x7ff3a00b4110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:31.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:31.168+0000 7ff3a6800640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff3a00b38e0 con 0x7ff3a015c3f0 2024-05-28T06:20:31.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:31.168+0000 7ff3a6800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff3a015c7d0 con 0x7ff3a00ca6a0 2024-05-28T06:20:31.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:31.168+0000 7ff3a6800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ff3a00b1bc0 con 0x7ff3a015f3d0 2024-05-28T06:20:31.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:31.168+0000 7ff39ea00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff3a015c3f0 0x7ff3a015b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:31.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:31.168+0000 7ff39ea00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff3a015c3f0 0x7ff3a015b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:51594/0 (socket says 172.21.15.96:51594) 2024-05-28T06:20:31.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:31.168+0000 7ff39ea00640 1 -- 172.21.15.96:0/369467810 learned_addr learned my addr 172.21.15.96:0/369467810 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:20:31.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:31.168+0000 7ff39f400640 1 --2- 172.21.15.96:0/369467810 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3a00ca6a0 0x7ff3a0159300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:31.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:31.168+0000 7ff39fe00640 1 --2- 172.21.15.96:0/369467810 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff3a015f3d0 0x7ff3a00b4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:31.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:31.168+0000 7ff39ea00640 1 -- 172.21.15.96:0/369467810 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff3a015f3d0 msgr2=0x7ff3a00b4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:31.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:31.168+0000 7ff39ea00640 1 --2- 172.21.15.96:0/369467810 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff3a015f3d0 0x7ff3a00b4110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:31.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:31.168+0000 7ff39ea00640 1 -- 172.21.15.96:0/369467810 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3a00ca6a0 msgr2=0x7ff3a0159300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:31.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:31.168+0000 7ff39ea00640 1 --2- 172.21.15.96:0/369467810 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3a00ca6a0 0x7ff3a0159300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:31.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:31.168+0000 7ff39ea00640 1 -- 172.21.15.96:0/369467810 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3a01598e0 con 0x7ff3a015c3f0 2024-05-28T06:20:31.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:31.169+0000 7ff39ea00640 1 --2- 172.21.15.96:0/369467810 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff3a015c3f0 0x7ff3a015b0c0 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7ff38c00c400 tx=0x7ff38c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:31.166 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:31.169+0000 7ff397400640 1 -- 172.21.15.96:0/369467810 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff38c0052d0 con 0x7ff3a015c3f0 2024-05-28T06:20:31.166 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:31.169+0000 7ff397400640 1 -- 172.21.15.96:0/369467810 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff38c005490 con 0x7ff3a015c3f0 2024-05-28T06:20:31.166 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:31.169+0000 7ff3a6800640 1 -- 172.21.15.96:0/369467810 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3a0159bc0 con 0x7ff3a015c3f0 2024-05-28T06:20:31.166 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:31.169+0000 7ff397400640 1 -- 172.21.15.96:0/369467810 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff38c007260 con 0x7ff3a015c3f0 2024-05-28T06:20:31.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:31.169+0000 7ff3a6800640 1 -- 172.21.15.96:0/369467810 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3a015a180 con 0x7ff3a015c3f0 2024-05-28T06:20:31.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:31.170+0000 7ff3a6800640 1 -- 172.21.15.96:0/369467810 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3a00690f0 con 0x7ff3a015c3f0 2024-05-28T06:20:31.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:31.171+0000 7ff397400640 1 -- 172.21.15.96:0/369467810 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 34) ==== 92927+0+0 (secure 0 0 0) 0x7ff38c007400 con 0x7ff3a015c3f0 2024-05-28T06:20:31.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:31.172+0000 7ff397400640 1 --2- 172.21.15.96:0/369467810 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff38006ddc0 0x7ff380070280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:20:31.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:31.172+0000 7ff397400640 1 -- 172.21.15.96:0/369467810 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 5765+0+0 (secure 0 0 0) 0x7ff38c08a1f0 con 0x7ff3a015c3f0 2024-05-28T06:20:31.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:31.173+0000 7ff39f400640 1 --2- 172.21.15.96:0/369467810 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff38006ddc0 0x7ff380070280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:20:31.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:31.174+0000 7ff39f400640 1 --2- 172.21.15.96:0/369467810 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff38006ddc0 0x7ff380070280 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7ff390009560 tx=0x7ff390008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:20:31.172 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:31.176+0000 7ff397400640 1 -- 172.21.15.96:0/369467810 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff38c059c90 con 0x7ff3a015c3f0 2024-05-28T06:20:31.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:31.409+0000 7ff3a6800640 1 -- 172.21.15.96:0/369467810 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) -- 0x7ff3a00b1790 con 0x7ff38006ddc0 2024-05-28T06:20:31.420 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:31.424+0000 7ff397400640 1 -- 172.21.15.96:0/369467810 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+59 (secure 0 0 0) 0x7ff3a00b1790 con 0x7ff38006ddc0 2024-05-28T06:20:31.420 INFO:teuthology.orchestra.run.smithi096.stdout:/volumes/_nogroup/sv1/df1f0247-87ab-4f07-abc5-b5b9370c3a00 2024-05-28T06:20:31.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:31.426+0000 7ff3a6800640 1 -- 172.21.15.96:0/369467810 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff38006ddc0 msgr2=0x7ff380070280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:31.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:31.426+0000 7ff3a6800640 1 --2- 172.21.15.96:0/369467810 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff38006ddc0 0x7ff380070280 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7ff390009560 tx=0x7ff390008040 comp rx=0 tx=0).stop 2024-05-28T06:20:31.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:31.427+0000 7ff3a6800640 1 -- 172.21.15.96:0/369467810 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff3a015c3f0 msgr2=0x7ff3a015b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:20:31.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:31.427+0000 7ff3a6800640 1 --2- 172.21.15.96:0/369467810 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff3a015c3f0 0x7ff3a015b0c0 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7ff38c00c400 tx=0x7ff38c00c8c0 comp rx=0 tx=0).stop 2024-05-28T06:20:31.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:31.427+0000 7ff3a6800640 1 -- 172.21.15.96:0/369467810 shutdown_connections 2024-05-28T06:20:31.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:31.427+0000 7ff3a6800640 1 --2- 172.21.15.96:0/369467810 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff38006ddc0 0x7ff380070280 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:31.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:31.427+0000 7ff3a6800640 1 --2- 172.21.15.96:0/369467810 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff3a015f3d0 0x7ff3a00b4110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:31.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:31.427+0000 7ff3a6800640 1 --2- 172.21.15.96:0/369467810 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff3a015c3f0 0x7ff3a015b0c0 unknown :-1 s=CLOSED pgs=1660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:31.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:31.427+0000 7ff3a6800640 1 --2- 172.21.15.96:0/369467810 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3a00ca6a0 0x7ff3a0159300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:20:31.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:31.427+0000 7ff3a6800640 1 -- 172.21.15.96:0/369467810 >> 172.21.15.96:0/369467810 conn(0x7ff3a009aa30 msgr2=0x7ff3a006b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:20:31.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:31.427+0000 7ff3a6800640 1 -- 172.21.15.96:0/369467810 shutdown_connections 2024-05-28T06:20:31.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:20:31.427+0000 7ff3a6800640 1 -- 172.21.15.96:0/369467810 wait complete. 2024-05-28T06:20:31.440 DEBUG:tasks.cephfs.test_volumes:filling subvolume sv1 with 3 files each 1024MB size under directory volumes/_nogroup/sv1/df1f0247-87ab-4f07-abc5-b5b9370c3a00 2024-05-28T06:20:31.440 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:20:31.440 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:20:31.490 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:20:31.490 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/df1f0247-87ab-4f07-abc5-b5b9370c3a00/subvolume_file.0 bs=1M conv=fdatasync count=1024 seek=0) 2024-05-28T06:20:33.752 DEBUG:teuthology.orchestra.run.smithi096:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:20:33.761 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:20:45.608 INFO:teuthology.orchestra.run.smithi096.stderr:1024+0 records in 2024-05-28T06:20:45.608 INFO:teuthology.orchestra.run.smithi096.stderr:1024+0 records out 2024-05-28T06:20:45.608 INFO:teuthology.orchestra.run.smithi096.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 13.9877 s, 76.8 MB/s 2024-05-28T06:20:45.610 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:20:45.610 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:20:45.665 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:20:45.665 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/df1f0247-87ab-4f07-abc5-b5b9370c3a00/subvolume_file.1 bs=1M conv=fdatasync count=1024 seek=0) 2024-05-28T06:20:56.923 INFO:teuthology.orchestra.run.smithi096.stderr:1024+0 records in 2024-05-28T06:20:56.923 INFO:teuthology.orchestra.run.smithi096.stderr:1024+0 records out 2024-05-28T06:20:56.923 INFO:teuthology.orchestra.run.smithi096.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 11.1971 s, 95.9 MB/s 2024-05-28T06:20:56.925 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:20:56.925 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:20:56.980 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:20:56.980 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/df1f0247-87ab-4f07-abc5-b5b9370c3a00/subvolume_file.2 bs=1M conv=fdatasync count=1024 seek=0) 2024-05-28T06:21:03.891 DEBUG:teuthology.orchestra.run.smithi096:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:21:03.899 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:21:08.409 INFO:teuthology.orchestra.run.smithi096.stderr:1024+0 records in 2024-05-28T06:21:08.409 INFO:teuthology.orchestra.run.smithi096.stderr:1024+0 records out 2024-05-28T06:21:08.409 INFO:teuthology.orchestra.run.smithi096.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 11.3685 s, 94.4 MB/s 2024-05-28T06:21:08.411 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot create cephfs sv1 ss1 2024-05-28T06:21:08.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.623+0000 7f76a5400640 1 Processor -- start 2024-05-28T06:21:08.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.623+0000 7f76a5400640 1 -- start start 2024-05-28T06:21:08.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.624+0000 7f76a5400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f76a015b340 0x7f76a006f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:08.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.624+0000 7f76a5400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f76a015b720 0x7f76a015bb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:08.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.624+0000 7f76a5400640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76a006fa90 con 0x7f76a015c0c0 2024-05-28T06:21:08.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.624+0000 7f76a5400640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f76a015c510 con 0x7f76a015b340 2024-05-28T06:21:08.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.624+0000 7f76a5400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f76a00af9b0 con 0x7f76a015b720 2024-05-28T06:21:08.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.625+0000 7f769f200640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f76a015c0c0 0x7f76a015f0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:40686/0 (socket says 172.21.15.96:40686) 2024-05-28T06:21:08.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.625+0000 7f769f200640 1 -- 172.21.15.96:0/3554892563 learned_addr learned my addr 172.21.15.96:0/3554892563 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:21:08.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.625+0000 7f769e800640 1 --2- 172.21.15.96:0/3554892563 >> v2:172.21.15.123:3300/0 conn(0x7f76a015b340 0x7f76a006f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:08.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.625+0000 7f769de00640 1 --2- 172.21.15.96:0/3554892563 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f76a015b720 0x7f76a015bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:08.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.625+0000 7f769d400640 1 -- 172.21.15.96:0/3554892563 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 695486036 0 0) 0x7f76a006fa90 con 0x7f76a015c0c0 2024-05-28T06:21:08.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.625+0000 7f769d400640 1 -- 172.21.15.96:0/3554892563 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7684003520 con 0x7f76a015c0c0 2024-05-28T06:21:08.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.625+0000 7f769d400640 1 -- 172.21.15.96:0/3554892563 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2869385096 0 0) 0x7f7684003520 con 0x7f76a015c0c0 2024-05-28T06:21:08.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.625+0000 7f769d400640 1 -- 172.21.15.96:0/3554892563 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f76a015b720 msgr2=0x7f76a015bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:21:08.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.625+0000 7f769d400640 1 --2- 172.21.15.96:0/3554892563 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f76a015b720 0x7f76a015bb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:08.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.625+0000 7f769d400640 1 -- 172.21.15.96:0/3554892563 >> v2:172.21.15.123:3300/0 conn(0x7f76a015b340 msgr2=0x7f76a006f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:21:08.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.625+0000 7f769d400640 1 --2- 172.21.15.96:0/3554892563 >> v2:172.21.15.123:3300/0 conn(0x7f76a015b340 0x7f76a006f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:08.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.625+0000 7f769d400640 1 -- 172.21.15.96:0/3554892563 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76a00aeda0 con 0x7f76a015c0c0 2024-05-28T06:21:08.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.625+0000 7f769d400640 1 -- 172.21.15.96:0/3554892563 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f76980019f0 con 0x7f76a015c0c0 2024-05-28T06:21:08.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.625+0000 7f769d400640 1 -- 172.21.15.96:0/3554892563 >> v1:172.21.15.96:6789/0 conn(0x7f76a015c0c0 legacy=0x7f76a015f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:08.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.626+0000 7f769d400640 1 --2- 172.21.15.96:0/3554892563 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f76840039b0 0x7f7684003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:08.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.626+0000 7f769d400640 1 --2- 172.21.15.96:0/3554892563 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7684004800 0x7f7684004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:08.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.626+0000 7f769d400640 1 --2- 172.21.15.96:0/3554892563 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f76a015b720 0x7f76840051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:08.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.626+0000 7f769d400640 1 -- 172.21.15.96:0/3554892563 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f76a00af9b0 con 0x7f7684004800 2024-05-28T06:21:08.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.626+0000 7f769d400640 1 -- 172.21.15.96:0/3554892563 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f768400b9c0 con 0x7f76840039b0 2024-05-28T06:21:08.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.626+0000 7f769d400640 1 -- 172.21.15.96:0/3554892563 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f768400bb70 con 0x7f76a015b720 2024-05-28T06:21:08.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.626+0000 7f769de00640 1 --2- 172.21.15.96:0/3554892563 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7684004800 0x7f7684004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:08.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.626+0000 7f769f200640 1 --2- 172.21.15.96:0/3554892563 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f76a015b720 0x7f76840051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:08.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.626+0000 7f769e800640 1 --2- 172.21.15.96:0/3554892563 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f76840039b0 0x7f7684003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:08.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.626+0000 7f769de00640 1 -- 172.21.15.96:0/3554892563 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f76a015b720 msgr2=0x7f76840051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:21:08.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.626+0000 7f769de00640 1 --2- 172.21.15.96:0/3554892563 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f76a015b720 0x7f76840051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:08.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.626+0000 7f769de00640 1 -- 172.21.15.96:0/3554892563 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f76840039b0 msgr2=0x7f7684003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:21:08.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.626+0000 7f769de00640 1 --2- 172.21.15.96:0/3554892563 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f76840039b0 0x7f7684003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:08.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.626+0000 7f769de00640 1 -- 172.21.15.96:0/3554892563 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7684003520 con 0x7f7684004800 2024-05-28T06:21:08.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.626+0000 7f769e800640 1 --2- 172.21.15.96:0/3554892563 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f76840039b0 0x7f7684003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:21:08.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.627+0000 7f769de00640 1 --2- 172.21.15.96:0/3554892563 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7684004800 0x7f7684004c80 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7f768c009ac0 tx=0x7f7684005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ea7a2188468bbdc server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:21:08.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.628+0000 7f769d400640 1 -- 172.21.15.96:0/3554892563 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f768c036070 con 0x7f7684004800 2024-05-28T06:21:08.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.628+0000 7f769d400640 1 -- 172.21.15.96:0/3554892563 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f768c031400 con 0x7f7684004800 2024-05-28T06:21:08.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.628+0000 7f769d400640 1 -- 172.21.15.96:0/3554892563 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f768c009690 con 0x7f7684004800 2024-05-28T06:21:08.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.628+0000 7f76a5400640 1 -- 172.21.15.96:0/3554892563 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7684004800 msgr2=0x7f7684004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:08.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.628+0000 7f76a5400640 1 --2- 172.21.15.96:0/3554892563 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7684004800 0x7f7684004c80 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7f768c009ac0 tx=0x7f7684005830 comp rx=0 tx=0).stop 2024-05-28T06:21:08.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.628+0000 7f769f200640 1 -- 172.21.15.96:0/3554892563 reap_dead start 2024-05-28T06:21:08.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.628+0000 7f76a5400640 1 -- 172.21.15.96:0/3554892563 shutdown_connections 2024-05-28T06:21:08.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.628+0000 7f76a5400640 1 -- 172.21.15.96:0/3554892563 >> 172.21.15.96:0/3554892563 conn(0x7f76a00b1e50 msgr2=0x7f76a00b2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:21:08.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.628+0000 7f76a5400640 1 -- 172.21.15.96:0/3554892563 shutdown_connections 2024-05-28T06:21:08.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.628+0000 7f76a5400640 1 -- 172.21.15.96:0/3554892563 wait complete. 2024-05-28T06:21:08.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.630+0000 7f76a5400640 1 Processor -- start 2024-05-28T06:21:08.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.630+0000 7f76a5400640 1 -- start start 2024-05-28T06:21:08.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.631+0000 7f76a5400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7684004800 0x7f76a014fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:08.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.631+0000 7f76a5400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f76a015dea0 0x7f76a0143470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:08.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.631+0000 7f76a5400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f76a006e440 0x7f76a006e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:08.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.631+0000 7f76a5400640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f76a00af640 con 0x7f76a006e440 2024-05-28T06:21:08.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.631+0000 7f76a5400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f76a00af360 con 0x7f76a015dea0 2024-05-28T06:21:08.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.631+0000 7f76a5400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f76a00aea30 con 0x7f7684004800 2024-05-28T06:21:08.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.631+0000 7f769f200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f76a006e440 0x7f76a006e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:08.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.631+0000 7f769f200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f76a006e440 0x7f76a006e8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:45502/0 (socket says 172.21.15.96:45502) 2024-05-28T06:21:08.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.631+0000 7f769f200640 1 -- 172.21.15.96:0/788295272 learned_addr learned my addr 172.21.15.96:0/788295272 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:21:08.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.632+0000 7f769de00640 1 --2- 172.21.15.96:0/788295272 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f76a015dea0 0x7f76a0143470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:08.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.632+0000 7f769e800640 1 --2- 172.21.15.96:0/788295272 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7684004800 0x7f76a014fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:08.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.632+0000 7f769f200640 1 -- 172.21.15.96:0/788295272 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7684004800 msgr2=0x7f76a014fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:08.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.632+0000 7f769f200640 1 --2- 172.21.15.96:0/788295272 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7684004800 0x7f76a014fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:08.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.632+0000 7f769f200640 1 -- 172.21.15.96:0/788295272 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f76a015dea0 msgr2=0x7f76a0143470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:08.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.632+0000 7f769f200640 1 --2- 172.21.15.96:0/788295272 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f76a015dea0 0x7f76a0143470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:08.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.632+0000 7f769f200640 1 -- 172.21.15.96:0/788295272 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76a01502d0 con 0x7f76a006e440 2024-05-28T06:21:08.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.632+0000 7f769e800640 1 --2- 172.21.15.96:0/788295272 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7684004800 0x7f76a014fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:21:08.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.632+0000 7f769f200640 1 --2- 172.21.15.96:0/788295272 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f76a006e440 0x7f76a006e8c0 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7f76a00af920 tx=0x7f769800b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:21:08.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.632+0000 7f7696a00640 1 -- 172.21.15.96:0/788295272 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7698004d60 con 0x7f76a006e440 2024-05-28T06:21:08.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.633+0000 7f7696a00640 1 -- 172.21.15.96:0/788295272 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f769800bb80 con 0x7f76a006e440 2024-05-28T06:21:08.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.633+0000 7f7696a00640 1 -- 172.21.15.96:0/788295272 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7698007110 con 0x7f76a006e440 2024-05-28T06:21:08.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.633+0000 7f76a5400640 1 -- 172.21.15.96:0/788295272 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76a01505b0 con 0x7f76a006e440 2024-05-28T06:21:08.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.633+0000 7f76a5400640 1 -- 172.21.15.96:0/788295272 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76a0150b70 con 0x7f76a006e440 2024-05-28T06:21:08.631 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.634+0000 7f76a5400640 1 -- 172.21.15.96:0/788295272 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76a00689e0 con 0x7f76a006e440 2024-05-28T06:21:08.631 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.635+0000 7f7696a00640 1 -- 172.21.15.96:0/788295272 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 35) ==== 92978+0+0 (secure 0 0 0) 0x7f769801b020 con 0x7f76a006e440 2024-05-28T06:21:08.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.636+0000 7f7696a00640 1 --2- 172.21.15.96:0/788295272 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f768006de50 0x7f7680070310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:08.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.636+0000 7f7696a00640 1 -- 172.21.15.96:0/788295272 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 5765+0+0 (secure 0 0 0) 0x7f769808aae0 con 0x7f76a006e440 2024-05-28T06:21:08.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.637+0000 7f769e800640 1 --2- 172.21.15.96:0/788295272 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f768006de50 0x7f7680070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:08.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.638+0000 7f769e800640 1 --2- 172.21.15.96:0/788295272 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f768006de50 0x7f7680070310 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f7690006fd0 tx=0x7f769000b3d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:21:08.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.641+0000 7f7696a00640 1 -- 172.21.15.96:0/788295272 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f769805a960 con 0x7f76a006e440 2024-05-28T06:21:08.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.877+0000 7f76a5400640 1 -- 172.21.15.96:0/788295272 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""]}) -- 0x7f76a0157d60 con 0x7f768006de50 2024-05-28T06:21:08.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.913+0000 7f7696a00640 1 -- 172.21.15.96:0/788295272 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f76a0157d60 con 0x7f768006de50 2024-05-28T06:21:08.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.916+0000 7f76a5400640 1 -- 172.21.15.96:0/788295272 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f768006de50 msgr2=0x7f7680070310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:08.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.916+0000 7f76a5400640 1 --2- 172.21.15.96:0/788295272 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f768006de50 0x7f7680070310 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f7690006fd0 tx=0x7f769000b3d0 comp rx=0 tx=0).stop 2024-05-28T06:21:08.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.916+0000 7f76a5400640 1 -- 172.21.15.96:0/788295272 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f76a006e440 msgr2=0x7f76a006e8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:08.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.916+0000 7f76a5400640 1 --2- 172.21.15.96:0/788295272 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f76a006e440 0x7f76a006e8c0 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7f76a00af920 tx=0x7f769800b4a0 comp rx=0 tx=0).stop 2024-05-28T06:21:08.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.916+0000 7f76a5400640 1 -- 172.21.15.96:0/788295272 shutdown_connections 2024-05-28T06:21:08.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.916+0000 7f76a5400640 1 --2- 172.21.15.96:0/788295272 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f768006de50 0x7f7680070310 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:08.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.916+0000 7f76a5400640 1 --2- 172.21.15.96:0/788295272 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f76a006e440 0x7f76a006e8c0 unknown :-1 s=CLOSED pgs=1663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:08.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.916+0000 7f76a5400640 1 --2- 172.21.15.96:0/788295272 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f76a015dea0 0x7f76a0143470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:08.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.916+0000 7f76a5400640 1 --2- 172.21.15.96:0/788295272 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7684004800 0x7f76a014fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:08.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.916+0000 7f76a5400640 1 -- 172.21.15.96:0/788295272 >> 172.21.15.96:0/788295272 conn(0x7f76a00b1e50 msgr2=0x7f76a00b45f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:21:08.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.916+0000 7f76a5400640 1 -- 172.21.15.96:0/788295272 shutdown_connections 2024-05-28T06:21:08.914 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:08.916+0000 7f76a5400640 1 -- 172.21.15.96:0/788295272 wait complete. 2024-05-28T06:21:08.933 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-05-28T06:21:09.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:09.142+0000 7f4f73e00640 1 Processor -- start 2024-05-28T06:21:09.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:09.142+0000 7f4f73e00640 1 -- start start 2024-05-28T06:21:09.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:09.144+0000 7f4f73e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4f740caa40 0x7f4f740ce960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:09.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:09.144+0000 7f4f73e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f4f741524c0 0x7f4f740cf710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:09.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:09.144+0000 7f4f73e00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f740b3e80 con 0x7f4f740cae20 2024-05-28T06:21:09.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:09.144+0000 7f4f73e00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f4f740d0b30 con 0x7f4f741524c0 2024-05-28T06:21:09.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:09.144+0000 7f4f73e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f4f740b4160 con 0x7f4f740caa40 2024-05-28T06:21:09.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:09.144+0000 7f4f73400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f4f741524c0 0x7f4f740cf710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:09.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:09.144+0000 7f4f72000640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f4f740cae20 0x7f4f74151db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:52888/0 (socket says 172.21.15.96:52888) 2024-05-28T06:21:09.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:09.144+0000 7f4f72000640 1 -- 172.21.15.96:0/1642226011 learned_addr learned my addr 172.21.15.96:0/1642226011 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:21:09.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:09.144+0000 7f4f72a00640 1 --2- 172.21.15.96:0/1642226011 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4f740caa40 0x7f4f740ce960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:09.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:09.145+0000 7f4f73400640 1 -- 172.21.15.96:0/1642226011 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4f740caa40 msgr2=0x7f4f740ce960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:21:09.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:09.145+0000 7f4f73400640 1 --2- 172.21.15.96:0/1642226011 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4f740caa40 0x7f4f740ce960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:09.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:09.145+0000 7f4f73400640 1 -- 172.21.15.96:0/1642226011 >> v1:172.21.15.96:6789/0 conn(0x7f4f740cae20 legacy=0x7f4f74151db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:09.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:09.145+0000 7f4f73400640 1 -- 172.21.15.96:0/1642226011 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f740cde80 con 0x7f4f741524c0 2024-05-28T06:21:09.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:09.145+0000 7f4f71600640 1 -- 172.21.15.96:0/1642226011 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 245827566 0 0) 0x7f4f740b3e80 con 0x7f4f740cae20 2024-05-28T06:21:09.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:09.146+0000 7f4f73400640 1 --2- 172.21.15.96:0/1642226011 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f741524c0 0x7f4f740cf710 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7f4f6c009d90 tx=0x7f4f6c02a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f50bbac255f95137 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:21:09.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:09.146+0000 7f4f71600640 1 -- 172.21.15.96:0/1642226011 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f6c031600 con 0x7f4f741524c0 2024-05-28T06:21:09.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:09.146+0000 7f4f71600640 1 -- 172.21.15.96:0/1642226011 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f6c0317a0 con 0x7f4f741524c0 2024-05-28T06:21:09.143 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:09.146+0000 7f4f71600640 1 -- 172.21.15.96:0/1642226011 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f6c031b70 con 0x7f4f741524c0 2024-05-28T06:21:09.143 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:09.147+0000 7f4f73e00640 1 -- 172.21.15.96:0/1642226011 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f741524c0 msgr2=0x7f4f740cf710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:09.143 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:09.147+0000 7f4f73e00640 1 --2- 172.21.15.96:0/1642226011 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f741524c0 0x7f4f740cf710 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7f4f6c009d90 tx=0x7f4f6c02a5d0 comp rx=0 tx=0).stop 2024-05-28T06:21:09.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:09.147+0000 7f4f73e00640 1 -- 172.21.15.96:0/1642226011 shutdown_connections 2024-05-28T06:21:09.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:09.147+0000 7f4f73e00640 1 --2- 172.21.15.96:0/1642226011 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f741524c0 0x7f4f740cf710 unknown :-1 s=CLOSED pgs=1665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:09.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:09.147+0000 7f4f73e00640 1 --2- 172.21.15.96:0/1642226011 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4f740caa40 0x7f4f740ce960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:09.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:09.147+0000 7f4f73e00640 1 -- 172.21.15.96:0/1642226011 >> 172.21.15.96:0/1642226011 conn(0x7f4f7406b900 msgr2=0x7f4f7406bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:21:09.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:09.147+0000 7f4f73e00640 1 -- 172.21.15.96:0/1642226011 shutdown_connections 2024-05-28T06:21:09.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:09.147+0000 7f4f73e00640 1 -- 172.21.15.96:0/1642226011 wait complete. 2024-05-28T06:21:09.145 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:09.149+0000 7f4f73e00640 1 Processor -- start 2024-05-28T06:21:09.145 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:09.149+0000 7f4f73e00640 1 -- start start 2024-05-28T06:21:09.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:09.150+0000 7f4f73e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4f740caa40 0x7f4f74145480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:09.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:09.150+0000 7f4f73e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4f740cae20 0x7f4f7409a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:09.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:09.150+0000 7f4f73e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f741524c0 0x7f4f74072400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:09.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:09.150+0000 7f4f73e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4f740ceea0 con 0x7f4f741524c0 2024-05-28T06:21:09.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:09.150+0000 7f4f73e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f4f740ce440 con 0x7f4f740cae20 2024-05-28T06:21:09.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:09.150+0000 7f4f73e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f4f740cf020 con 0x7f4f740caa40 2024-05-28T06:21:09.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:09.150+0000 7f4f73400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f741524c0 0x7f4f74072400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:09.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:09.150+0000 7f4f73400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f741524c0 0x7f4f74072400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58672/0 (socket says 172.21.15.96:58672) 2024-05-28T06:21:09.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:09.150+0000 7f4f73400640 1 -- 172.21.15.96:0/769381723 learned_addr learned my addr 172.21.15.96:0/769381723 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:21:09.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:09.150+0000 7f4f72a00640 1 --2- 172.21.15.96:0/769381723 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4f740caa40 0x7f4f74145480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:09.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:09.151+0000 7f4f72000640 1 --2- 172.21.15.96:0/769381723 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4f740cae20 0x7f4f7409a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:09.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:09.151+0000 7f4f73400640 1 -- 172.21.15.96:0/769381723 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4f740caa40 msgr2=0x7f4f74145480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:09.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:09.151+0000 7f4f73400640 1 --2- 172.21.15.96:0/769381723 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4f740caa40 0x7f4f74145480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:09.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:09.151+0000 7f4f73400640 1 -- 172.21.15.96:0/769381723 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4f740cae20 msgr2=0x7f4f7409a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:09.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:09.151+0000 7f4f73400640 1 --2- 172.21.15.96:0/769381723 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4f740cae20 0x7f4f7409a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:09.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:09.151+0000 7f4f73400640 1 -- 172.21.15.96:0/769381723 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f7409ad70 con 0x7f4f741524c0 2024-05-28T06:21:09.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:09.151+0000 7f4f72000640 1 --2- 172.21.15.96:0/769381723 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4f740cae20 0x7f4f7409a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:21:09.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:09.151+0000 7f4f73400640 1 --2- 172.21.15.96:0/769381723 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f741524c0 0x7f4f74072400 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7f4f6c009eb0 tx=0x7f4f6c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:21:09.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:09.151+0000 7f4f6aa00640 1 -- 172.21.15.96:0/769381723 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f6c03f070 con 0x7f4f741524c0 2024-05-28T06:21:09.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:09.151+0000 7f4f6aa00640 1 -- 172.21.15.96:0/769381723 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f6c030be0 con 0x7f4f741524c0 2024-05-28T06:21:09.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:09.151+0000 7f4f6aa00640 1 -- 172.21.15.96:0/769381723 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f6c009450 con 0x7f4f741524c0 2024-05-28T06:21:09.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:09.151+0000 7f4f73e00640 1 -- 172.21.15.96:0/769381723 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f7409b050 con 0x7f4f741524c0 2024-05-28T06:21:09.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:09.151+0000 7f4f73e00640 1 -- 172.21.15.96:0/769381723 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f7409b610 con 0x7f4f741524c0 2024-05-28T06:21:09.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:09.153+0000 7f4f73e00640 1 -- 172.21.15.96:0/769381723 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f740689e0 con 0x7f4f741524c0 2024-05-28T06:21:09.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:09.154+0000 7f4f6aa00640 1 -- 172.21.15.96:0/769381723 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 35) ==== 92978+0+0 (secure 0 0 0) 0x7f4f6c003990 con 0x7f4f741524c0 2024-05-28T06:21:09.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:09.155+0000 7f4f6aa00640 1 --2- 172.21.15.96:0/769381723 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4f5406de50 0x7f4f54070310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:09.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:09.155+0000 7f4f6aa00640 1 -- 172.21.15.96:0/769381723 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 5765+0+0 (secure 0 0 0) 0x7f4f6c0ab890 con 0x7f4f741524c0 2024-05-28T06:21:09.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:09.155+0000 7f4f72a00640 1 --2- 172.21.15.96:0/769381723 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4f5406de50 0x7f4f54070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:09.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:09.157+0000 7f4f72a00640 1 --2- 172.21.15.96:0/769381723 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4f5406de50 0x7f4f54070310 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f4f64006fd0 tx=0x7f4f64008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:21:09.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:09.159+0000 7f4f6aa00640 1 -- 172.21.15.96:0/769381723 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4f6c07b310 con 0x7f4f741524c0 2024-05-28T06:21:09.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:09.395+0000 7f4f73e00640 1 -- 172.21.15.96:0/769381723 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) -- 0x7f4f740b4820 con 0x7f4f5406de50 2024-05-28T06:21:09.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:09.408+0000 7f4f6aa00640 1 -- 172.21.15.96:0/769381723 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+59 (secure 0 0 0) 0x7f4f740b4820 con 0x7f4f5406de50 2024-05-28T06:21:09.405 INFO:teuthology.orchestra.run.smithi096.stdout:/volumes/_nogroup/sv1/df1f0247-87ab-4f07-abc5-b5b9370c3a00 2024-05-28T06:21:09.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:09.410+0000 7f4f73e00640 1 -- 172.21.15.96:0/769381723 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4f5406de50 msgr2=0x7f4f54070310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:09.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:09.410+0000 7f4f73e00640 1 --2- 172.21.15.96:0/769381723 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4f5406de50 0x7f4f54070310 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f4f64006fd0 tx=0x7f4f64008040 comp rx=0 tx=0).stop 2024-05-28T06:21:09.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:09.410+0000 7f4f73e00640 1 -- 172.21.15.96:0/769381723 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f741524c0 msgr2=0x7f4f74072400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:09.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:09.410+0000 7f4f73e00640 1 --2- 172.21.15.96:0/769381723 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f741524c0 0x7f4f74072400 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7f4f6c009eb0 tx=0x7f4f6c004a40 comp rx=0 tx=0).stop 2024-05-28T06:21:09.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:09.410+0000 7f4f73e00640 1 -- 172.21.15.96:0/769381723 shutdown_connections 2024-05-28T06:21:09.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:09.410+0000 7f4f73e00640 1 --2- 172.21.15.96:0/769381723 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4f5406de50 0x7f4f54070310 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:09.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:09.410+0000 7f4f73e00640 1 --2- 172.21.15.96:0/769381723 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f741524c0 0x7f4f74072400 unknown :-1 s=CLOSED pgs=1666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:09.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:09.410+0000 7f4f73e00640 1 --2- 172.21.15.96:0/769381723 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4f740cae20 0x7f4f7409a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:09.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:09.410+0000 7f4f73e00640 1 --2- 172.21.15.96:0/769381723 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4f740caa40 0x7f4f74145480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:09.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:09.410+0000 7f4f73e00640 1 -- 172.21.15.96:0/769381723 >> 172.21.15.96:0/769381723 conn(0x7f4f7406b900 msgr2=0x7f4f740738b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:21:09.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:09.410+0000 7f4f73e00640 1 -- 172.21.15.96:0/769381723 shutdown_connections 2024-05-28T06:21:09.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:09.411+0000 7f4f73e00640 1 -- 172.21.15.96:0/769381723 wait complete. 2024-05-28T06:21:09.425 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:21:09.425 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/df1f0247-87ab-4f07-abc5-b5b9370c3a00 2024-05-28T06:21:09.426 DEBUG:teuthology.orchestra.run.smithi096:> ) 2024-05-28T06:21:09.489 INFO:teuthology.orchestra.run.smithi096.stdout:# file: volumes/_nogroup/sv1/df1f0247-87ab-4f07-abc5-b5b9370c3a00 2024-05-28T06:21:09.489 INFO:teuthology.orchestra.run.smithi096.stdout:ceph.dir.rbytes="3138256896" 2024-05-28T06:21:09.489 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:21:09.490 INFO:tasks.cephfs.test_volumes:r_size = 3138256896 exp_size = 3221225472 2024-05-28T06:21:11.491 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:21:11.492 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/df1f0247-87ab-4f07-abc5-b5b9370c3a00 2024-05-28T06:21:11.492 DEBUG:teuthology.orchestra.run.smithi096:> ) 2024-05-28T06:21:11.558 INFO:teuthology.orchestra.run.smithi096.stdout:# file: volumes/_nogroup/sv1/df1f0247-87ab-4f07-abc5-b5b9370c3a00 2024-05-28T06:21:11.559 INFO:teuthology.orchestra.run.smithi096.stdout:ceph.dir.rbytes="3138256896" 2024-05-28T06:21:11.559 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:21:11.560 INFO:tasks.cephfs.test_volumes:r_size = 3138256896 exp_size = 3221225472 2024-05-28T06:21:13.560 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:21:13.560 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/df1f0247-87ab-4f07-abc5-b5b9370c3a00 2024-05-28T06:21:13.560 DEBUG:teuthology.orchestra.run.smithi096:> ) 2024-05-28T06:21:13.626 INFO:teuthology.orchestra.run.smithi096.stdout:# file: volumes/_nogroup/sv1/df1f0247-87ab-4f07-abc5-b5b9370c3a00 2024-05-28T06:21:13.626 INFO:teuthology.orchestra.run.smithi096.stdout:ceph.dir.rbytes="3221225472" 2024-05-28T06:21:13.626 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:21:13.627 INFO:tasks.cephfs.test_volumes:r_size = 3221225472 exp_size = 3221225472 2024-05-28T06:21:13.628 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 ss1clone1 2024-05-28T06:21:13.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:13.845+0000 7f087ac00640 1 Processor -- start 2024-05-28T06:21:13.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:13.845+0000 7f087ac00640 1 -- start start 2024-05-28T06:21:13.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:13.847+0000 7f087ac00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f0874161a90 0x7f087406e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:13.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:13.847+0000 7f087ac00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0874161e70 0x7f08740b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:13.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:13.847+0000 7f087ac00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f087406e9b0 0x7f08740ad2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:13.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:13.847+0000 7f087ac00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f08740b3d40 con 0x7f0874161a90 2024-05-28T06:21:13.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:13.847+0000 7f087ac00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f08740b4020 con 0x7f087406e9b0 2024-05-28T06:21:13.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:13.847+0000 7f087ac00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f08740b4300 con 0x7f0874161e70 2024-05-28T06:21:13.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:13.847+0000 7f0873e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f0874161a90 0x7f087406e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:13.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:13.847+0000 7f0873e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f0874161a90 0x7f087406e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58676/0 (socket says 172.21.15.96:58676) 2024-05-28T06:21:13.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:13.847+0000 7f0873e00640 1 -- 172.21.15.96:0/106863952 learned_addr learned my addr 172.21.15.96:0/106863952 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:21:13.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:13.848+0000 7f0878c00640 1 --2- 172.21.15.96:0/106863952 >> v2:172.21.15.123:3300/0 conn(0x7f087406e9b0 0x7f08740ad2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:13.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:13.848+0000 7f0873e00640 1 -- 172.21.15.96:0/106863952 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0874161e70 msgr2=0x7f08740b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:21:13.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:13.848+0000 7f0873400640 1 --2- 172.21.15.96:0/106863952 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0874161e70 0x7f08740b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:13.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:13.848+0000 7f0873e00640 1 --2- 172.21.15.96:0/106863952 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0874161e70 0x7f08740b32c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:13.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:13.848+0000 7f0873e00640 1 -- 172.21.15.96:0/106863952 >> v2:172.21.15.123:3300/0 conn(0x7f087406e9b0 msgr2=0x7f08740ad2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:21:13.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:13.848+0000 7f0873e00640 1 --2- 172.21.15.96:0/106863952 >> v2:172.21.15.123:3300/0 conn(0x7f087406e9b0 0x7f08740ad2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:13.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:13.848+0000 7f0873e00640 1 -- 172.21.15.96:0/106863952 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f087406dc70 con 0x7f0874161a90 2024-05-28T06:21:13.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:13.848+0000 7f0878c00640 1 --2- 172.21.15.96:0/106863952 >> v2:172.21.15.123:3300/0 conn(0x7f087406e9b0 0x7f08740ad2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:21:13.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:13.849+0000 7f0873e00640 1 --2- 172.21.15.96:0/106863952 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0874161a90 0x7f087406e470 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7f08740d1320 tx=0x7f086402a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de2a6a42eca674be server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:21:13.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:13.849+0000 7f0872a00640 1 -- 172.21.15.96:0/106863952 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0864036070 con 0x7f0874161a90 2024-05-28T06:21:13.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:13.849+0000 7f0872a00640 1 -- 172.21.15.96:0/106863952 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0864031600 con 0x7f0874161a90 2024-05-28T06:21:13.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:13.849+0000 7f0872a00640 1 -- 172.21.15.96:0/106863952 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08640319d0 con 0x7f0874161a90 2024-05-28T06:21:13.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:13.850+0000 7f087ac00640 1 -- 172.21.15.96:0/106863952 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0874161a90 msgr2=0x7f087406e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:13.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:13.850+0000 7f087ac00640 1 --2- 172.21.15.96:0/106863952 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0874161a90 0x7f087406e470 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7f08740d1320 tx=0x7f086402a6a0 comp rx=0 tx=0).stop 2024-05-28T06:21:13.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:13.850+0000 7f087ac00640 1 -- 172.21.15.96:0/106863952 shutdown_connections 2024-05-28T06:21:13.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:13.850+0000 7f087ac00640 1 --2- 172.21.15.96:0/106863952 >> v2:172.21.15.123:3300/0 conn(0x7f087406e9b0 0x7f08740ad2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:13.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:13.850+0000 7f087ac00640 1 --2- 172.21.15.96:0/106863952 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0874161e70 0x7f08740b32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:13.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:13.850+0000 7f087ac00640 1 --2- 172.21.15.96:0/106863952 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0874161a90 0x7f087406e470 unknown :-1 s=CLOSED pgs=1667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:13.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:13.850+0000 7f087ac00640 1 -- 172.21.15.96:0/106863952 >> 172.21.15.96:0/106863952 conn(0x7f087406bd20 msgr2=0x7f087406c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:21:13.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:13.850+0000 7f087ac00640 1 -- 172.21.15.96:0/106863952 shutdown_connections 2024-05-28T06:21:13.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:13.850+0000 7f087ac00640 1 -- 172.21.15.96:0/106863952 wait complete. 2024-05-28T06:21:13.847 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:13.852+0000 7f087ac00640 1 Processor -- start 2024-05-28T06:21:13.847 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:13.852+0000 7f087ac00640 1 -- start start 2024-05-28T06:21:13.850 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:13.853+0000 7f087ac00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f087406e9b0 0x7f0874160530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:13.850 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:13.853+0000 7f087ac00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0874161a90 0x7f0874160a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:13.850 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:13.853+0000 7f087ac00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0874161e70 0x7f087414c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:13.850 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:13.853+0000 7f087ac00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f08740b3800 con 0x7f0874161e70 2024-05-28T06:21:13.850 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:13.853+0000 7f087ac00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f08740b5020 con 0x7f0874161a90 2024-05-28T06:21:13.850 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:13.853+0000 7f087ac00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f08740b3980 con 0x7f087406e9b0 2024-05-28T06:21:13.850 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:13.853+0000 7f0878c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0874161e70 0x7f087414c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:13.850 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:13.854+0000 7f0878c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0874161e70 0x7f087414c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58688/0 (socket says 172.21.15.96:58688) 2024-05-28T06:21:13.850 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:13.854+0000 7f0878c00640 1 -- 172.21.15.96:0/223269495 learned_addr learned my addr 172.21.15.96:0/223269495 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:21:13.850 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:13.854+0000 7f0873e00640 1 --2- 172.21.15.96:0/223269495 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f087406e9b0 0x7f0874160530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:13.850 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:13.854+0000 7f0873400640 1 --2- 172.21.15.96:0/223269495 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0874161a90 0x7f0874160a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:13.851 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:13.854+0000 7f0878c00640 1 -- 172.21.15.96:0/223269495 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f087406e9b0 msgr2=0x7f0874160530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:13.851 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:13.854+0000 7f0878c00640 1 --2- 172.21.15.96:0/223269495 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f087406e9b0 0x7f0874160530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:13.851 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:13.854+0000 7f0878c00640 1 -- 172.21.15.96:0/223269495 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0874161a90 msgr2=0x7f0874160a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:13.851 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:13.854+0000 7f0878c00640 1 --2- 172.21.15.96:0/223269495 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0874161a90 0x7f0874160a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:13.851 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:13.854+0000 7f0878c00640 1 -- 172.21.15.96:0/223269495 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f087415e840 con 0x7f0874161e70 2024-05-28T06:21:13.851 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:13.854+0000 7f0873400640 1 --2- 172.21.15.96:0/223269495 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0874161a90 0x7f0874160a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:21:13.851 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:13.854+0000 7f0878c00640 1 --2- 172.21.15.96:0/223269495 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0874161e70 0x7f087414c5f0 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7f086c00afd0 tx=0x7f086c00ca20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:21:13.851 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:13.855+0000 7f086be00640 1 -- 172.21.15.96:0/223269495 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f086c015020 con 0x7f0874161e70 2024-05-28T06:21:13.851 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:13.855+0000 7f086be00640 1 -- 172.21.15.96:0/223269495 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f086c004440 con 0x7f0874161e70 2024-05-28T06:21:13.851 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:13.855+0000 7f086be00640 1 -- 172.21.15.96:0/223269495 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f086c003850 con 0x7f0874161e70 2024-05-28T06:21:13.851 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:13.855+0000 7f087ac00640 1 -- 172.21.15.96:0/223269495 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f087415eb00 con 0x7f0874161e70 2024-05-28T06:21:13.851 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:13.855+0000 7f087ac00640 1 -- 172.21.15.96:0/223269495 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0874152f40 con 0x7f0874161e70 2024-05-28T06:21:13.852 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:13.856+0000 7f087ac00640 1 -- 172.21.15.96:0/223269495 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08740689e0 con 0x7f0874161e70 2024-05-28T06:21:13.852 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:13.857+0000 7f086be00640 1 -- 172.21.15.96:0/223269495 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 35) ==== 92978+0+0 (secure 0 0 0) 0x7f086c018020 con 0x7f0874161e70 2024-05-28T06:21:13.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:13.858+0000 7f086be00640 1 --2- 172.21.15.96:0/223269495 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f085006de50 0x7f0850070310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:13.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:13.858+0000 7f086be00640 1 -- 172.21.15.96:0/223269495 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 5765+0+0 (secure 0 0 0) 0x7f086c08acc0 con 0x7f0874161e70 2024-05-28T06:21:13.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:13.859+0000 7f0873e00640 1 --2- 172.21.15.96:0/223269495 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f085006de50 0x7f0850070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:13.854 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:13.860+0000 7f0873e00640 1 --2- 172.21.15.96:0/223269495 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f085006de50 0x7f0850070310 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f0864033000 tx=0x7f0864009000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:21:13.857 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:13.863+0000 7f086be00640 1 -- 172.21.15.96:0/223269495 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f086c01f020 con 0x7f0874161e70 2024-05-28T06:21:14.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:14.101+0000 7f087ac00640 1 -- 172.21.15.96:0/223269495 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "ss1clone1", "target": ["mon-mgr", ""]}) -- 0x7f08740af9c0 con 0x7f085006de50 2024-05-28T06:21:14.253 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:14.258+0000 7f086be00640 1 -- 172.21.15.96:0/223269495 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f08740af9c0 con 0x7f085006de50 2024-05-28T06:21:14.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:14.261+0000 7f087ac00640 1 -- 172.21.15.96:0/223269495 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f085006de50 msgr2=0x7f0850070310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:14.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:14.261+0000 7f087ac00640 1 --2- 172.21.15.96:0/223269495 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f085006de50 0x7f0850070310 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f0864033000 tx=0x7f0864009000 comp rx=0 tx=0).stop 2024-05-28T06:21:14.257 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:14.261+0000 7f087ac00640 1 -- 172.21.15.96:0/223269495 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0874161e70 msgr2=0x7f087414c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:14.257 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:14.261+0000 7f087ac00640 1 --2- 172.21.15.96:0/223269495 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0874161e70 0x7f087414c5f0 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7f086c00afd0 tx=0x7f086c00ca20 comp rx=0 tx=0).stop 2024-05-28T06:21:14.257 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:14.261+0000 7f087ac00640 1 -- 172.21.15.96:0/223269495 shutdown_connections 2024-05-28T06:21:14.257 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:14.261+0000 7f087ac00640 1 --2- 172.21.15.96:0/223269495 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f085006de50 0x7f0850070310 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:14.257 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:14.261+0000 7f087ac00640 1 --2- 172.21.15.96:0/223269495 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0874161e70 0x7f087414c5f0 unknown :-1 s=CLOSED pgs=1668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:14.257 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:14.261+0000 7f087ac00640 1 --2- 172.21.15.96:0/223269495 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0874161a90 0x7f0874160a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:14.257 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:14.261+0000 7f087ac00640 1 --2- 172.21.15.96:0/223269495 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f087406e9b0 0x7f0874160530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:14.257 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:14.261+0000 7f087ac00640 1 -- 172.21.15.96:0/223269495 >> 172.21.15.96:0/223269495 conn(0x7f087406bd20 msgr2=0x7f08740afe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:21:14.257 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:14.261+0000 7f087ac00640 1 -- 172.21.15.96:0/223269495 shutdown_connections 2024-05-28T06:21:14.257 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:14.262+0000 7f087ac00640 1 -- 172.21.15.96:0/223269495 wait complete. 2024-05-28T06:21:14.258 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:21:14.263+0000 7fea61600640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:21:14.258 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:21:14.263+0000 7fea61600640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:21:14.258 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:21:14.263+0000 7fea61600640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:21:14.258 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:21:14.263+0000 7fea61600640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:21:14.258 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:21:14.263+0000 7fea61600640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:21:14.283 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:21:14.288+0000 7fea60c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:21:14.283 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:21:14.288+0000 7fea60c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:21:14.283 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:21:14.288+0000 7fea60c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:21:14.283 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:21:14.288+0000 7fea60c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:21:14.283 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:21:14.288+0000 7fea60c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:21:15.279 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs ss1clone1 2024-05-28T06:21:15.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:15.497+0000 7fa72b800640 1 Processor -- start 2024-05-28T06:21:15.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:15.498+0000 7fa72b800640 1 -- start start 2024-05-28T06:21:15.495 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:15.499+0000 7fa72b800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fa7240d0640 0x7fa7240d0a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:15.495 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:15.499+0000 7fa72b800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa72415b6f0 0x7fa72406e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:15.496 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:15.499+0000 7fa72b800640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7240ade90 con 0x7fa7240725e0 2024-05-28T06:21:15.496 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:15.499+0000 7fa72b800640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fa724071c40 con 0x7fa7240d0640 2024-05-28T06:21:15.496 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:15.499+0000 7fa72b800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fa7240d0f60 con 0x7fa72415b6f0 2024-05-28T06:21:15.496 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:15.499+0000 7fa728e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fa7240d0640 0x7fa7240d0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:15.496 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:15.499+0000 7fa729800640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fa7240725e0 0x7fa7240b44d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:52900/0 (socket says 172.21.15.96:52900) 2024-05-28T06:21:15.496 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:15.499+0000 7fa729800640 1 -- 172.21.15.96:0/2816811198 learned_addr learned my addr 172.21.15.96:0/2816811198 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:21:15.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:15.500+0000 7fa723e00640 1 --2- 172.21.15.96:0/2816811198 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa72415b6f0 0x7fa72406e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:15.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:15.500+0000 7fa728e00640 1 -- 172.21.15.96:0/2816811198 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa72415b6f0 msgr2=0x7fa72406e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:21:15.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:15.500+0000 7fa728e00640 1 --2- 172.21.15.96:0/2816811198 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa72415b6f0 0x7fa72406e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:15.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:15.500+0000 7fa728e00640 1 -- 172.21.15.96:0/2816811198 >> v1:172.21.15.96:6789/0 conn(0x7fa7240725e0 legacy=0x7fa7240b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:15.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:15.500+0000 7fa728e00640 1 -- 172.21.15.96:0/2816811198 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa724071e70 con 0x7fa7240d0640 2024-05-28T06:21:15.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:15.500+0000 7fa723400640 1 -- 172.21.15.96:0/2816811198 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 170286864 0 0) 0x7fa7240ade90 con 0x7fa7240725e0 2024-05-28T06:21:15.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:15.501+0000 7fa728e00640 1 --2- 172.21.15.96:0/2816811198 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa7240d0640 0x7fa7240d0a20 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7fa7180097a0 tx=0x7fa71802a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c49569c8c8558797 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:21:15.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:15.501+0000 7fa723400640 1 -- 172.21.15.96:0/2816811198 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa718037070 con 0x7fa7240d0640 2024-05-28T06:21:15.501 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:15.501+0000 7fa723400640 1 -- 172.21.15.96:0/2816811198 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa718032440 con 0x7fa7240d0640 2024-05-28T06:21:15.501 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:15.501+0000 7fa723400640 1 -- 172.21.15.96:0/2816811198 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa718032810 con 0x7fa7240d0640 2024-05-28T06:21:15.501 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:15.502+0000 7fa72b800640 1 -- 172.21.15.96:0/2816811198 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa7240d0640 msgr2=0x7fa7240d0a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:15.501 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:15.502+0000 7fa72b800640 1 --2- 172.21.15.96:0/2816811198 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa7240d0640 0x7fa7240d0a20 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7fa7180097a0 tx=0x7fa71802a180 comp rx=0 tx=0).stop 2024-05-28T06:21:15.501 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:15.503+0000 7fa72b800640 1 -- 172.21.15.96:0/2816811198 shutdown_connections 2024-05-28T06:21:15.502 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:15.503+0000 7fa72b800640 1 --2- 172.21.15.96:0/2816811198 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa72415b6f0 0x7fa72406e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:15.502 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:15.503+0000 7fa72b800640 1 --2- 172.21.15.96:0/2816811198 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa7240d0640 0x7fa7240d0a20 unknown :-1 s=CLOSED pgs=1670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:15.502 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:15.503+0000 7fa72b800640 1 -- 172.21.15.96:0/2816811198 >> 172.21.15.96:0/2816811198 conn(0x7fa72406bd20 msgr2=0x7fa72406c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:21:15.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:15.503+0000 7fa72b800640 1 -- 172.21.15.96:0/2816811198 shutdown_connections 2024-05-28T06:21:15.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:15.503+0000 7fa72b800640 1 -- 172.21.15.96:0/2816811198 wait complete. 2024-05-28T06:21:15.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:15.505+0000 7fa72b800640 1 Processor -- start 2024-05-28T06:21:15.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:15.505+0000 7fa72b800640 1 -- start start 2024-05-28T06:21:15.507 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:15.506+0000 7fa72b800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa7240725e0 0x7fa7240cf420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:15.508 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:15.506+0000 7fa72b800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa7240d0640 0x7fa724098ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:15.508 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:15.506+0000 7fa72b800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa72415b6f0 0x7fa724099020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:15.508 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:15.506+0000 7fa72b800640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa72406e890 con 0x7fa72415b6f0 2024-05-28T06:21:15.508 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:15.506+0000 7fa72b800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa7240ae130 con 0x7fa7240725e0 2024-05-28T06:21:15.508 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:15.506+0000 7fa72b800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fa7240af660 con 0x7fa7240d0640 2024-05-28T06:21:15.508 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:15.507+0000 7fa729800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa72415b6f0 0x7fa724099020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:15.508 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:15.507+0000 7fa729800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa72415b6f0 0x7fa724099020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58700/0 (socket says 172.21.15.96:58700) 2024-05-28T06:21:15.508 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:15.507+0000 7fa729800640 1 -- 172.21.15.96:0/1107476632 learned_addr learned my addr 172.21.15.96:0/1107476632 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:21:15.509 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:15.507+0000 7fa728e00640 1 --2- 172.21.15.96:0/1107476632 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa7240725e0 0x7fa7240cf420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:15.509 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:15.507+0000 7fa723e00640 1 --2- 172.21.15.96:0/1107476632 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa7240d0640 0x7fa724098ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:15.509 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:15.507+0000 7fa729800640 1 -- 172.21.15.96:0/1107476632 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa7240d0640 msgr2=0x7fa724098ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:15.509 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:15.507+0000 7fa729800640 1 --2- 172.21.15.96:0/1107476632 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa7240d0640 0x7fa724098ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:15.509 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:15.507+0000 7fa729800640 1 -- 172.21.15.96:0/1107476632 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa7240725e0 msgr2=0x7fa7240cf420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:15.509 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:15.507+0000 7fa729800640 1 --2- 172.21.15.96:0/1107476632 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa7240725e0 0x7fa7240cf420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:15.509 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:15.507+0000 7fa729800640 1 -- 172.21.15.96:0/1107476632 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa72409b9b0 con 0x7fa72415b6f0 2024-05-28T06:21:15.509 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:15.508+0000 7fa729800640 1 --2- 172.21.15.96:0/1107476632 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa72415b6f0 0x7fa724099020 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7fa71c00c7a0 tx=0x7fa71c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:21:15.510 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:15.508+0000 7fa720c00640 1 -- 172.21.15.96:0/1107476632 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa71c018070 con 0x7fa72415b6f0 2024-05-28T06:21:15.510 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:15.508+0000 7fa720c00640 1 -- 172.21.15.96:0/1107476632 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa71c00a640 con 0x7fa72415b6f0 2024-05-28T06:21:15.510 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:15.508+0000 7fa72b800640 1 -- 172.21.15.96:0/1107476632 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa72409bc90 con 0x7fa72415b6f0 2024-05-28T06:21:15.510 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:15.508+0000 7fa720c00640 1 -- 172.21.15.96:0/1107476632 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa71c00aa70 con 0x7fa72415b6f0 2024-05-28T06:21:15.510 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:15.508+0000 7fa72b800640 1 -- 172.21.15.96:0/1107476632 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa72409c250 con 0x7fa72415b6f0 2024-05-28T06:21:15.510 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:15.509+0000 7fa72b800640 1 -- 172.21.15.96:0/1107476632 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7240689e0 con 0x7fa72415b6f0 2024-05-28T06:21:15.510 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:15.510+0000 7fa720c00640 1 -- 172.21.15.96:0/1107476632 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 35) ==== 92978+0+0 (secure 0 0 0) 0x7fa71c01b070 con 0x7fa72415b6f0 2024-05-28T06:21:15.510 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:15.511+0000 7fa720c00640 1 --2- 172.21.15.96:0/1107476632 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa70406de50 0x7fa704070310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:15.511 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:15.511+0000 7fa720c00640 1 -- 172.21.15.96:0/1107476632 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 5765+0+0 (secure 0 0 0) 0x7fa71c08a2a0 con 0x7fa72415b6f0 2024-05-28T06:21:15.511 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:15.512+0000 7fa728e00640 1 --2- 172.21.15.96:0/1107476632 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa70406de50 0x7fa704070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:15.511 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:15.513+0000 7fa728e00640 1 --2- 172.21.15.96:0/1107476632 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa70406de50 0x7fa704070310 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fa718034000 tx=0x7fa71802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:21:15.511 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:15.515+0000 7fa720c00640 1 -- 172.21.15.96:0/1107476632 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa71c059d20 con 0x7fa72415b6f0 2024-05-28T06:21:15.750 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:15.755+0000 7fa72b800640 1 -- 172.21.15.96:0/1107476632 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) -- 0x7fa7240739a0 con 0x7fa70406de50 2024-05-28T06:21:16.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:16.641+0000 7fa720c00640 1 -- 172.21.15.96:0/1107476632 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mgrmap(e 36) ==== 93080+0+0 (secure 0 0 0) 0x7fa71c055c90 con 0x7fa72415b6f0 2024-05-28T06:21:18.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:18.712+0000 7fa720c00640 1 -- 172.21.15.96:0/1107476632 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fa7240739a0 con 0x7fa70406de50 2024-05-28T06:21:18.711 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:18.716+0000 7fa72b800640 1 -- 172.21.15.96:0/1107476632 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa70406de50 msgr2=0x7fa704070310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:18.711 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:18.716+0000 7fa72b800640 1 --2- 172.21.15.96:0/1107476632 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa70406de50 0x7fa704070310 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fa718034000 tx=0x7fa71802e000 comp rx=0 tx=0).stop 2024-05-28T06:21:18.711 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:18.716+0000 7fa72b800640 1 -- 172.21.15.96:0/1107476632 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa72415b6f0 msgr2=0x7fa724099020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:18.711 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:18.716+0000 7fa72b800640 1 --2- 172.21.15.96:0/1107476632 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa72415b6f0 0x7fa724099020 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7fa71c00c7a0 tx=0x7fa71c00cc60 comp rx=0 tx=0).stop 2024-05-28T06:21:18.711 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:18.716+0000 7fa72b800640 1 -- 172.21.15.96:0/1107476632 shutdown_connections 2024-05-28T06:21:18.711 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:18.716+0000 7fa72b800640 1 --2- 172.21.15.96:0/1107476632 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa70406de50 0x7fa704070310 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:18.711 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:18.716+0000 7fa72b800640 1 --2- 172.21.15.96:0/1107476632 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa72415b6f0 0x7fa724099020 unknown :-1 s=CLOSED pgs=1671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:18.712 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:18.716+0000 7fa72b800640 1 --2- 172.21.15.96:0/1107476632 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa7240d0640 0x7fa724098ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:18.712 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:18.716+0000 7fa72b800640 1 --2- 172.21.15.96:0/1107476632 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa7240725e0 0x7fa7240cf420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:18.712 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:18.716+0000 7fa72b800640 1 -- 172.21.15.96:0/1107476632 >> 172.21.15.96:0/1107476632 conn(0x7fa72406bd20 msgr2=0x7fa7240b3500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:21:18.712 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:18.716+0000 7fa72b800640 1 -- 172.21.15.96:0/1107476632 shutdown_connections 2024-05-28T06:21:18.712 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:18.716+0000 7fa72b800640 1 -- 172.21.15.96:0/1107476632 wait complete. 2024-05-28T06:21:23.734 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-05-28T06:21:23.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:23.947+0000 7fdd6a600640 1 Processor -- start 2024-05-28T06:21:23.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:23.947+0000 7fdd6a600640 1 -- start start 2024-05-28T06:21:23.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:23.949+0000 7fdd6a600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fdd64155a00 0x7fdd6415baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:23.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:23.949+0000 7fdd6a600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fdd640d0970 0x7fdd6406dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:23.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:23.949+0000 7fdd6a600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd6415c0b0 con 0x7fdd6406f390 2024-05-28T06:21:23.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:23.949+0000 7fdd6a600640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fdd640ccf20 con 0x7fdd64155a00 2024-05-28T06:21:23.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:23.949+0000 7fdd6a600640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fdd640d0490 con 0x7fdd640d0970 2024-05-28T06:21:23.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:23.949+0000 7fdd63400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fdd64155a00 0x7fdd6415baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:23.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:23.949+0000 7fdd63400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fdd64155a00 0x7fdd6415baa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:40810/0 (socket says 172.21.15.96:40810) 2024-05-28T06:21:23.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:23.949+0000 7fdd63400640 1 -- 172.21.15.96:0/1536506777 learned_addr learned my addr 172.21.15.96:0/1536506777 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:21:23.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:23.950+0000 7fdd63400640 1 -- 172.21.15.96:0/1536506777 >> v2:172.21.15.123:3300/0 conn(0x7fdd640d0970 msgr2=0x7fdd6406dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:21:23.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:23.950+0000 7fdd63400640 1 --2- 172.21.15.96:0/1536506777 >> v2:172.21.15.123:3300/0 conn(0x7fdd640d0970 0x7fdd6406dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:23.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:23.950+0000 7fdd63400640 1 -- 172.21.15.96:0/1536506777 >> v1:172.21.15.123:6789/0 conn(0x7fdd6406f390 legacy=0x7fdd6415f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:21:23.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:23.950+0000 7fdd63400640 1 -- 172.21.15.96:0/1536506777 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd6415c390 con 0x7fdd64155a00 2024-05-28T06:21:23.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:23.951+0000 7fdd63400640 1 --2- 172.21.15.96:0/1536506777 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdd64155a00 0x7fdd6415baa0 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7fdd6415bfe0 tx=0x7fdd5402a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=620085c7189d2d4b server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:21:23.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:23.951+0000 7fdd62000640 1 -- 172.21.15.96:0/1536506777 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd54037070 con 0x7fdd64155a00 2024-05-28T06:21:23.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:23.951+0000 7fdd62000640 1 -- 172.21.15.96:0/1536506777 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd54032440 con 0x7fdd64155a00 2024-05-28T06:21:23.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:23.951+0000 7fdd62000640 1 -- 172.21.15.96:0/1536506777 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd54032810 con 0x7fdd64155a00 2024-05-28T06:21:23.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:23.952+0000 7fdd6a600640 1 -- 172.21.15.96:0/1536506777 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdd64155a00 msgr2=0x7fdd6415baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:23.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:23.952+0000 7fdd6a600640 1 --2- 172.21.15.96:0/1536506777 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdd64155a00 0x7fdd6415baa0 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7fdd6415bfe0 tx=0x7fdd5402a230 comp rx=0 tx=0).stop 2024-05-28T06:21:23.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:23.952+0000 7fdd6a600640 1 -- 172.21.15.96:0/1536506777 shutdown_connections 2024-05-28T06:21:23.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:23.952+0000 7fdd6a600640 1 --2- 172.21.15.96:0/1536506777 >> v2:172.21.15.123:3300/0 conn(0x7fdd640d0970 0x7fdd6406dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:23.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:23.952+0000 7fdd6a600640 1 --2- 172.21.15.96:0/1536506777 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdd64155a00 0x7fdd6415baa0 unknown :-1 s=CLOSED pgs=1672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:23.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:23.952+0000 7fdd6a600640 1 -- 172.21.15.96:0/1536506777 >> 172.21.15.96:0/1536506777 conn(0x7fdd64067ce0 msgr2=0x7fdd6406b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:21:23.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:23.952+0000 7fdd6a600640 1 -- 172.21.15.96:0/1536506777 shutdown_connections 2024-05-28T06:21:23.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:23.952+0000 7fdd6a600640 1 -- 172.21.15.96:0/1536506777 wait complete. 2024-05-28T06:21:23.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:23.954+0000 7fdd6a600640 1 Processor -- start 2024-05-28T06:21:23.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:23.954+0000 7fdd6a600640 1 -- start start 2024-05-28T06:21:23.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:23.955+0000 7fdd6a600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdd6406f390 0x7fdd64152ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:23.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:23.955+0000 7fdd6a600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdd640d0970 0x7fdd6406e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:23.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:23.955+0000 7fdd6a600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdd64155a00 0x7fdd6414bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:23.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:23.955+0000 7fdd6a600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fdd6415f090 con 0x7fdd6406f390 2024-05-28T06:21:23.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:23.955+0000 7fdd6a600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fdd6415c950 con 0x7fdd640d0970 2024-05-28T06:21:23.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:23.955+0000 7fdd6a600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fdd64155de0 con 0x7fdd64155a00 2024-05-28T06:21:23.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:23.955+0000 7fdd63400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdd6406f390 0x7fdd64152ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:23.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:23.955+0000 7fdd63400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdd6406f390 0x7fdd64152ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:40822/0 (socket says 172.21.15.96:40822) 2024-05-28T06:21:23.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:23.955+0000 7fdd63400640 1 -- 172.21.15.96:0/1099319353 learned_addr learned my addr 172.21.15.96:0/1099319353 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:21:23.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:23.956+0000 7fdd63400640 1 -- 172.21.15.96:0/1099319353 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdd64155a00 msgr2=0x7fdd6414bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:23.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:23.956+0000 7fdd63400640 1 --2- 172.21.15.96:0/1099319353 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdd64155a00 0x7fdd6414bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:23.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:23.956+0000 7fdd63400640 1 -- 172.21.15.96:0/1099319353 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdd640d0970 msgr2=0x7fdd6406e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:23.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:23.956+0000 7fdd63400640 1 --2- 172.21.15.96:0/1099319353 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdd640d0970 0x7fdd6406e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:23.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:23.956+0000 7fdd63400640 1 -- 172.21.15.96:0/1099319353 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd6414f140 con 0x7fdd6406f390 2024-05-28T06:21:23.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:23.956+0000 7fdd63400640 1 --2- 172.21.15.96:0/1099319353 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdd6406f390 0x7fdd64152ea0 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7fdd54000c00 tx=0x7fdd540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:21:23.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:23.956+0000 7fdd5b400640 1 -- 172.21.15.96:0/1099319353 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd54041070 con 0x7fdd6406f390 2024-05-28T06:21:23.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:23.956+0000 7fdd5b400640 1 -- 172.21.15.96:0/1099319353 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd5403c870 con 0x7fdd6406f390 2024-05-28T06:21:23.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:23.956+0000 7fdd6a600640 1 -- 172.21.15.96:0/1099319353 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd6414ee60 con 0x7fdd6406f390 2024-05-28T06:21:23.953 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:23.956+0000 7fdd5b400640 1 -- 172.21.15.96:0/1099319353 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd54037040 con 0x7fdd6406f390 2024-05-28T06:21:23.953 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:23.956+0000 7fdd6a600640 1 -- 172.21.15.96:0/1099319353 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd641526a0 con 0x7fdd6406f390 2024-05-28T06:21:23.954 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:23.958+0000 7fdd6a600640 1 -- 172.21.15.96:0/1099319353 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd6415b420 con 0x7fdd6406f390 2024-05-28T06:21:23.954 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:23.958+0000 7fdd5b400640 1 -- 172.21.15.96:0/1099319353 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 36) ==== 93080+0+0 (secure 0 0 0) 0x7fdd54044070 con 0x7fdd6406f390 2024-05-28T06:21:23.954 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:23.959+0000 7fdd5b400640 1 --2- 172.21.15.96:0/1099319353 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdd4406df70 0x7fdd44070430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:23.954 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:23.959+0000 7fdd5b400640 1 -- 172.21.15.96:0/1099319353 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 5765+0+0 (secure 0 0 0) 0x7fdd540ab570 con 0x7fdd6406f390 2024-05-28T06:21:23.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:23.960+0000 7fdd62a00640 1 --2- 172.21.15.96:0/1099319353 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdd4406df70 0x7fdd44070430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:23.956 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:23.961+0000 7fdd62a00640 1 --2- 172.21.15.96:0/1099319353 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdd4406df70 0x7fdd44070430 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fdd6406dd30 tx=0x7fdd50006c90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:21:23.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:23.964+0000 7fdd5b400640 1 -- 172.21.15.96:0/1099319353 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fdd6415b420 con 0x7fdd6406f390 2024-05-28T06:21:24.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.236+0000 7fdd6a600640 1 -- 172.21.15.96:0/1099319353 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fdd640689e0 con 0x7fdd6406f390 2024-05-28T06:21:24.233 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.237+0000 7fdd5b400640 1 -- 172.21.15.96:0/1099319353 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+2566 (secure 0 0 0) 0x7fdd5407b520 con 0x7fdd6406f390 2024-05-28T06:21:24.233 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:21:24.233 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:21:24.234 INFO:teuthology.orchestra.run.smithi096.stdout: "fsid": "e4a831b5-e6d1-47a7-96cb-bf85154ee0d2", 2024-05-28T06:21:24.234 INFO:teuthology.orchestra.run.smithi096.stdout: "health": { 2024-05-28T06:21:24.234 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "HEALTH_OK", 2024-05-28T06:21:24.234 INFO:teuthology.orchestra.run.smithi096.stdout: "checks": {}, 2024-05-28T06:21:24.234 INFO:teuthology.orchestra.run.smithi096.stdout: "mutes": [] 2024-05-28T06:21:24.234 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:21:24.234 INFO:teuthology.orchestra.run.smithi096.stdout: "election_epoch": 8, 2024-05-28T06:21:24.234 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum": [ 2024-05-28T06:21:24.234 INFO:teuthology.orchestra.run.smithi096.stdout: 0, 2024-05-28T06:21:24.234 INFO:teuthology.orchestra.run.smithi096.stdout: 1, 2024-05-28T06:21:24.234 INFO:teuthology.orchestra.run.smithi096.stdout: 2 2024-05-28T06:21:24.234 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:21:24.234 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_names": [ 2024-05-28T06:21:24.235 INFO:teuthology.orchestra.run.smithi096.stdout: "a", 2024-05-28T06:21:24.235 INFO:teuthology.orchestra.run.smithi096.stdout: "b", 2024-05-28T06:21:24.235 INFO:teuthology.orchestra.run.smithi096.stdout: "c" 2024-05-28T06:21:24.235 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:21:24.235 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_age": 1057, 2024-05-28T06:21:24.235 INFO:teuthology.orchestra.run.smithi096.stdout: "monmap": { 2024-05-28T06:21:24.235 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 1, 2024-05-28T06:21:24.235 INFO:teuthology.orchestra.run.smithi096.stdout: "min_mon_release_name": "squid", 2024-05-28T06:21:24.235 INFO:teuthology.orchestra.run.smithi096.stdout: "num_mons": 3 2024-05-28T06:21:24.235 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:21:24.235 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap": { 2024-05-28T06:21:24.235 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 103, 2024-05-28T06:21:24.235 INFO:teuthology.orchestra.run.smithi096.stdout: "num_osds": 8, 2024-05-28T06:21:24.235 INFO:teuthology.orchestra.run.smithi096.stdout: "num_up_osds": 8, 2024-05-28T06:21:24.236 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_up_since": 1716876238, 2024-05-28T06:21:24.236 INFO:teuthology.orchestra.run.smithi096.stdout: "num_in_osds": 8, 2024-05-28T06:21:24.236 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_in_since": 1716876229, 2024-05-28T06:21:24.236 INFO:teuthology.orchestra.run.smithi096.stdout: "num_remapped_pgs": 0 2024-05-28T06:21:24.236 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:21:24.236 INFO:teuthology.orchestra.run.smithi096.stdout: "pgmap": { 2024-05-28T06:21:24.236 INFO:teuthology.orchestra.run.smithi096.stdout: "pgs_by_state": [ 2024-05-28T06:21:24.236 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:21:24.236 INFO:teuthology.orchestra.run.smithi096.stdout: "state_name": "active+clean", 2024-05-28T06:21:24.236 INFO:teuthology.orchestra.run.smithi096.stdout: "count": 97 2024-05-28T06:21:24.236 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:21:24.236 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:21:24.236 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pgs": 97, 2024-05-28T06:21:24.236 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pools": 3, 2024-05-28T06:21:24.237 INFO:teuthology.orchestra.run.smithi096.stdout: "num_objects": 800, 2024-05-28T06:21:24.237 INFO:teuthology.orchestra.run.smithi096.stdout: "data_bytes": 3222142595, 2024-05-28T06:21:24.237 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_used": 7344906240, 2024-05-28T06:21:24.237 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_avail": 765749207040, 2024-05-28T06:21:24.237 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_total": 773094113280, 2024-05-28T06:21:24.237 INFO:teuthology.orchestra.run.smithi096.stdout: "read_bytes_sec": 511, 2024-05-28T06:21:24.237 INFO:teuthology.orchestra.run.smithi096.stdout: "write_bytes_sec": 11542205, 2024-05-28T06:21:24.237 INFO:teuthology.orchestra.run.smithi096.stdout: "read_op_per_sec": 0, 2024-05-28T06:21:24.237 INFO:teuthology.orchestra.run.smithi096.stdout: "write_op_per_sec": 6 2024-05-28T06:21:24.237 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:21:24.237 INFO:teuthology.orchestra.run.smithi096.stdout: "fsmap": { 2024-05-28T06:21:24.237 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 63, 2024-05-28T06:21:24.237 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 7, 2024-05-28T06:21:24.237 INFO:teuthology.orchestra.run.smithi096.stdout: "up": 1, 2024-05-28T06:21:24.238 INFO:teuthology.orchestra.run.smithi096.stdout: "in": 1, 2024-05-28T06:21:24.238 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 1, 2024-05-28T06:21:24.238 INFO:teuthology.orchestra.run.smithi096.stdout: "by_rank": [ 2024-05-28T06:21:24.238 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:21:24.238 INFO:teuthology.orchestra.run.smithi096.stdout: "filesystem_id": 7, 2024-05-28T06:21:24.238 INFO:teuthology.orchestra.run.smithi096.stdout: "rank": 0, 2024-05-28T06:21:24.238 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "b", 2024-05-28T06:21:24.238 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "up:active", 2024-05-28T06:21:24.238 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 8739 2024-05-28T06:21:24.238 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:21:24.238 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:21:24.238 INFO:teuthology.orchestra.run.smithi096.stdout: "up:standby": 3 2024-05-28T06:21:24.238 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:21:24.239 INFO:teuthology.orchestra.run.smithi096.stdout: "mgrmap": { 2024-05-28T06:21:24.239 INFO:teuthology.orchestra.run.smithi096.stdout: "available": true, 2024-05-28T06:21:24.239 INFO:teuthology.orchestra.run.smithi096.stdout: "num_standbys": 1, 2024-05-28T06:21:24.239 INFO:teuthology.orchestra.run.smithi096.stdout: "modules": [ 2024-05-28T06:21:24.239 INFO:teuthology.orchestra.run.smithi096.stdout: "iostat", 2024-05-28T06:21:24.239 INFO:teuthology.orchestra.run.smithi096.stdout: "nfs", 2024-05-28T06:21:24.239 INFO:teuthology.orchestra.run.smithi096.stdout: "restful" 2024-05-28T06:21:24.239 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:21:24.239 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:21:24.239 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:21:24.239 INFO:teuthology.orchestra.run.smithi096.stdout: "servicemap": { 2024-05-28T06:21:24.239 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 19, 2024-05-28T06:21:24.239 INFO:teuthology.orchestra.run.smithi096.stdout: "modified": "2024-05-28T06:21:15.939159+0000", 2024-05-28T06:21:24.239 INFO:teuthology.orchestra.run.smithi096.stdout: "services": { 2024-05-28T06:21:24.240 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-05-28T06:21:24.240 INFO:teuthology.orchestra.run.smithi096.stdout: "daemons": { 2024-05-28T06:21:24.240 INFO:teuthology.orchestra.run.smithi096.stdout: "summary": "", 2024-05-28T06:21:24.240 INFO:teuthology.orchestra.run.smithi096.stdout: "b": { 2024-05-28T06:21:24.240 INFO:teuthology.orchestra.run.smithi096.stdout: "start_epoch": 0, 2024-05-28T06:21:24.240 INFO:teuthology.orchestra.run.smithi096.stdout: "start_stamp": "0.000000", 2024-05-28T06:21:24.240 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 0, 2024-05-28T06:21:24.240 INFO:teuthology.orchestra.run.smithi096.stdout: "addr": "(unrecognized address family 0)/0", 2024-05-28T06:21:24.240 INFO:teuthology.orchestra.run.smithi096.stdout: "metadata": {}, 2024-05-28T06:21:24.240 INFO:teuthology.orchestra.run.smithi096.stdout: "task_status": {} 2024-05-28T06:21:24.240 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:21:24.240 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:21:24.240 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:21:24.240 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:21:24.241 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:21:24.241 INFO:teuthology.orchestra.run.smithi096.stdout: "progress_events": { 2024-05-28T06:21:24.241 INFO:teuthology.orchestra.run.smithi096.stdout: "885b6d9d-736d-45a8-ac79-be8f5b8ac230": { 2024-05-28T06:21:24.241 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "All cloning finished, waiting 28.0 seconds before exiting (3s)\n [============================] ", 2024-05-28T06:21:24.241 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": 1, 2024-05-28T06:21:24.241 INFO:teuthology.orchestra.run.smithi096.stdout: "add_to_ceph_s": true 2024-05-28T06:21:24.241 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:21:24.241 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:21:24.241 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:21:24.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.240+0000 7fdd6a600640 1 -- 172.21.15.96:0/1099319353 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdd4406df70 msgr2=0x7fdd44070430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:24.243 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.240+0000 7fdd6a600640 1 --2- 172.21.15.96:0/1099319353 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdd4406df70 0x7fdd44070430 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fdd6406dd30 tx=0x7fdd50006c90 comp rx=0 tx=0).stop 2024-05-28T06:21:24.243 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.240+0000 7fdd6a600640 1 -- 172.21.15.96:0/1099319353 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdd6406f390 msgr2=0x7fdd64152ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:24.243 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.240+0000 7fdd6a600640 1 --2- 172.21.15.96:0/1099319353 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdd6406f390 0x7fdd64152ea0 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7fdd54000c00 tx=0x7fdd540046c0 comp rx=0 tx=0).stop 2024-05-28T06:21:24.243 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.240+0000 7fdd6a600640 1 -- 172.21.15.96:0/1099319353 shutdown_connections 2024-05-28T06:21:24.243 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.240+0000 7fdd6a600640 1 --2- 172.21.15.96:0/1099319353 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdd4406df70 0x7fdd44070430 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:24.243 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.240+0000 7fdd6a600640 1 --2- 172.21.15.96:0/1099319353 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdd64155a00 0x7fdd6414bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:24.243 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.240+0000 7fdd6a600640 1 --2- 172.21.15.96:0/1099319353 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdd640d0970 0x7fdd6406e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:24.243 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.240+0000 7fdd6a600640 1 --2- 172.21.15.96:0/1099319353 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdd6406f390 0x7fdd64152ea0 unknown :-1 s=CLOSED pgs=1673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:24.243 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.240+0000 7fdd6a600640 1 -- 172.21.15.96:0/1099319353 >> 172.21.15.96:0/1099319353 conn(0x7fdd64067ce0 msgr2=0x7fdd64153540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:21:24.243 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.240+0000 7fdd6a600640 1 -- 172.21.15.96:0/1099319353 shutdown_connections 2024-05-28T06:21:24.243 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.240+0000 7fdd6a600640 1 -- 172.21.15.96:0/1099319353 wait complete. 2024-05-28T06:21:24.255 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:21:24.255 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls -lh volumes/_nogroup/ss1clone1/df1f0247-87ab-4f07-abc5-b5b9370c3a00 2024-05-28T06:21:24.255 DEBUG:teuthology.orchestra.run.smithi096:> ) 2024-05-28T06:21:24.315 INFO:teuthology.orchestra.run.smithi096.stderr:ls: cannot access 'volumes/_nogroup/ss1clone1/df1f0247-87ab-4f07-abc5-b5b9370c3a00': No such file or directory 2024-05-28T06:21:24.316 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-05-28T06:21:24.317 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-05-28T06:21:24.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.523+0000 7f2e74a00640 1 Processor -- start 2024-05-28T06:21:24.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.524+0000 7f2e74a00640 1 -- start start 2024-05-28T06:21:24.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.525+0000 7f2e74a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2e70156b30 0x7f2e700af6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:24.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.525+0000 7f2e74a00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e70156f10 con 0x7f2e700ad940 2024-05-28T06:21:24.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.525+0000 7f2e74a00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e7006dda0 con 0x7f2e7006ee40 2024-05-28T06:21:24.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.525+0000 7f2e74a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f2e7006e080 con 0x7f2e70156b30 2024-05-28T06:21:24.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.525+0000 7f2e6fe00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f2e700ad940 0x7f2e700b32c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:59150/0 (socket says 172.21.15.96:59150) 2024-05-28T06:21:24.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.525+0000 7f2e6fe00640 1 -- 172.21.15.96:0/3626699061 learned_addr learned my addr 172.21.15.96:0/3626699061 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:21:24.523 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.525+0000 7f2e6ea00640 1 -- 172.21.15.96:0/3626699061 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1764161045 0 0) 0x7f2e70156f10 con 0x7f2e700ad940 2024-05-28T06:21:24.523 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.526+0000 7f2e6f400640 1 --2- 172.21.15.96:0/3626699061 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2e70156b30 0x7f2e700af6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:24.523 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.526+0000 7f2e6ea00640 1 -- 172.21.15.96:0/3626699061 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2e54003520 con 0x7f2e700ad940 2024-05-28T06:21:24.523 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.526+0000 7f2e6ea00640 1 -- 172.21.15.96:0/3626699061 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3384362781 0 0) 0x7f2e54003520 con 0x7f2e700ad940 2024-05-28T06:21:24.523 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.526+0000 7f2e6ea00640 1 -- 172.21.15.96:0/3626699061 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2e70156b30 msgr2=0x7f2e700af6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:21:24.523 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.526+0000 7f2e6ea00640 1 --2- 172.21.15.96:0/3626699061 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2e70156b30 0x7f2e700af6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:24.523 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.526+0000 7f2e6ea00640 1 -- 172.21.15.96:0/3626699061 >> v1:172.21.15.123:6789/0 conn(0x7f2e7006ee40 legacy=0x7f2e700ad230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:24.523 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.526+0000 7f2e6ea00640 1 -- 172.21.15.96:0/3626699061 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e7006e360 con 0x7f2e700ad940 2024-05-28T06:21:24.524 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.526+0000 7f2e6ea00640 1 -- 172.21.15.96:0/3626699061 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f2e680019f0 con 0x7f2e700ad940 2024-05-28T06:21:24.524 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.526+0000 7f2e6ea00640 1 -- 172.21.15.96:0/3626699061 >> v1:172.21.15.96:6789/0 conn(0x7f2e700ad940 legacy=0x7f2e700b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:24.524 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.526+0000 7f2e6ea00640 1 --2- 172.21.15.96:0/3626699061 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2e540039e0 0x7f2e54003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:24.524 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.526+0000 7f2e6ea00640 1 --2- 172.21.15.96:0/3626699061 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e54004800 0x7f2e54004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:24.524 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.526+0000 7f2e6ea00640 1 --2- 172.21.15.96:0/3626699061 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2e70156b30 0x7f2e5400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:24.524 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.526+0000 7f2e6ea00640 1 -- 172.21.15.96:0/3626699061 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f2e7006e080 con 0x7f2e540039e0 2024-05-28T06:21:24.524 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.526+0000 7f2e6ea00640 1 -- 172.21.15.96:0/3626699061 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f2e54005520 con 0x7f2e54004800 2024-05-28T06:21:24.524 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.526+0000 7f2e6ea00640 1 -- 172.21.15.96:0/3626699061 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f2e5400bf00 con 0x7f2e70156b30 2024-05-28T06:21:24.524 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.526+0000 7f2e6f400640 1 --2- 172.21.15.96:0/3626699061 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2e540039e0 0x7f2e54003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:24.524 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.527+0000 7f2e6fe00640 1 --2- 172.21.15.96:0/3626699061 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2e70156b30 0x7f2e5400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:24.524 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.527+0000 7f2e6f400640 1 -- 172.21.15.96:0/3626699061 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2e70156b30 msgr2=0x7f2e5400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:21:24.524 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.527+0000 7f2e6f400640 1 --2- 172.21.15.96:0/3626699061 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2e70156b30 0x7f2e5400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:24.524 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.527+0000 7f2e6f400640 1 -- 172.21.15.96:0/3626699061 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e54004800 msgr2=0x7f2e54004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:21:24.525 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.527+0000 7f2e6f400640 1 --2- 172.21.15.96:0/3626699061 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e54004800 0x7f2e54004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:24.525 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.527+0000 7f2e6f400640 1 -- 172.21.15.96:0/3626699061 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e7006e360 con 0x7f2e540039e0 2024-05-28T06:21:24.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.528+0000 7f2e6f400640 1 --2- 172.21.15.96:0/3626699061 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2e540039e0 0x7f2e54003e00 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7f2e6000ba30 tx=0x7f2e54005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d3ef79c2cea0efc server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:21:24.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.528+0000 7f2e6ea00640 1 -- 172.21.15.96:0/3626699061 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e60030070 con 0x7f2e540039e0 2024-05-28T06:21:24.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.528+0000 7f2e6ea00640 1 -- 172.21.15.96:0/3626699061 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e6002ade0 con 0x7f2e540039e0 2024-05-28T06:21:24.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.528+0000 7f2e6ea00640 1 -- 172.21.15.96:0/3626699061 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e6000b600 con 0x7f2e540039e0 2024-05-28T06:21:24.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.529+0000 7f2e74a00640 1 -- 172.21.15.96:0/3626699061 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2e540039e0 msgr2=0x7f2e54003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:24.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.529+0000 7f2e74a00640 1 --2- 172.21.15.96:0/3626699061 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2e540039e0 0x7f2e54003e00 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7f2e6000ba30 tx=0x7f2e54005830 comp rx=0 tx=0).stop 2024-05-28T06:21:24.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.529+0000 7f2e6fe00640 1 -- 172.21.15.96:0/3626699061 reap_dead start 2024-05-28T06:21:24.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.529+0000 7f2e74a00640 1 -- 172.21.15.96:0/3626699061 shutdown_connections 2024-05-28T06:21:24.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.529+0000 7f2e74a00640 1 -- 172.21.15.96:0/3626699061 >> 172.21.15.96:0/3626699061 conn(0x7f2e700b1fc0 msgr2=0x7f2e700b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:21:24.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.529+0000 7f2e74a00640 1 -- 172.21.15.96:0/3626699061 shutdown_connections 2024-05-28T06:21:24.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.529+0000 7f2e74a00640 1 -- 172.21.15.96:0/3626699061 wait complete. 2024-05-28T06:21:24.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.531+0000 7f2e74a00640 1 Processor -- start 2024-05-28T06:21:24.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.531+0000 7f2e74a00640 1 -- start start 2024-05-28T06:21:24.529 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.531+0000 7f2e74a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2e540039e0 0x7f2e7014d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:24.530 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.531+0000 7f2e74a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e7015de40 0x7f2e70157970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:24.530 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.532+0000 7f2e74a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2e7009a540 0x7f2e7009a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:24.530 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.532+0000 7f2e74a00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f2e700b2db0 con 0x7f2e540039e0 2024-05-28T06:21:24.530 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.532+0000 7f2e74a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f2e700b2c30 con 0x7f2e7015de40 2024-05-28T06:21:24.530 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.532+0000 7f2e74a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f2e700b39d0 con 0x7f2e7009a540 2024-05-28T06:21:24.530 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.532+0000 7f2e6f400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2e540039e0 0x7f2e7014d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:24.530 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.532+0000 7f2e6f400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2e540039e0 0x7f2e7014d6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:40836/0 (socket says 172.21.15.96:40836) 2024-05-28T06:21:24.530 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.532+0000 7f2e6f400640 1 -- 172.21.15.96:0/3010008238 learned_addr learned my addr 172.21.15.96:0/3010008238 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:21:24.530 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.532+0000 7f2e6f400640 1 -- 172.21.15.96:0/3010008238 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2e7009a540 msgr2=0x7f2e7009a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:24.530 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.532+0000 7f2e6f400640 1 --2- 172.21.15.96:0/3010008238 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2e7009a540 0x7f2e7009a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:24.530 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.532+0000 7f2e6f400640 1 -- 172.21.15.96:0/3010008238 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e7015de40 msgr2=0x7f2e70157970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:24.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.532+0000 7f2e6f400640 1 --2- 172.21.15.96:0/3010008238 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e7015de40 0x7f2e70157970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:24.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.532+0000 7f2e6f400640 1 -- 172.21.15.96:0/3010008238 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e7014dbe0 con 0x7f2e540039e0 2024-05-28T06:21:24.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.533+0000 7f2e6f400640 1 --2- 172.21.15.96:0/3010008238 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2e540039e0 0x7f2e7014d6a0 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7f2e6000bb20 tx=0x7f2e600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:21:24.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.533+0000 7f2e67400640 1 -- 172.21.15.96:0/3010008238 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e60043020 con 0x7f2e540039e0 2024-05-28T06:21:24.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.533+0000 7f2e67400640 1 -- 172.21.15.96:0/3010008238 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e6003f680 con 0x7f2e540039e0 2024-05-28T06:21:24.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.533+0000 7f2e67400640 1 -- 172.21.15.96:0/3010008238 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e6000b600 con 0x7f2e540039e0 2024-05-28T06:21:24.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.533+0000 7f2e74a00640 1 -- 172.21.15.96:0/3010008238 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e7009c640 con 0x7f2e540039e0 2024-05-28T06:21:24.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.533+0000 7f2e74a00640 1 -- 172.21.15.96:0/3010008238 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e7015e6e0 con 0x7f2e540039e0 2024-05-28T06:21:24.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.534+0000 7f2e74a00640 1 -- 172.21.15.96:0/3010008238 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e700b4cf0 con 0x7f2e540039e0 2024-05-28T06:21:24.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.535+0000 7f2e67400640 1 -- 172.21.15.96:0/3010008238 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 36) ==== 93080+0+0 (secure 0 0 0) 0x7f2e60046020 con 0x7f2e540039e0 2024-05-28T06:21:24.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.536+0000 7f2e67400640 1 --2- 172.21.15.96:0/3010008238 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2e4c06dfa0 0x7f2e4c070460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:24.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.536+0000 7f2e67400640 1 -- 172.21.15.96:0/3010008238 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 5765+0+0 (secure 0 0 0) 0x7f2e600aaff0 con 0x7f2e540039e0 2024-05-28T06:21:24.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.536+0000 7f2e67e00640 1 --2- 172.21.15.96:0/3010008238 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2e4c06dfa0 0x7f2e4c070460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:24.533 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.538+0000 7f2e67e00640 1 --2- 172.21.15.96:0/3010008238 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2e4c06dfa0 0x7f2e4c070460 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f2e7015f2b0 tx=0x7f2e580079d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:21:24.536 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.540+0000 7f2e67400640 1 -- 172.21.15.96:0/3010008238 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2e700b4cf0 con 0x7f2e540039e0 2024-05-28T06:21:24.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.769+0000 7f2e74a00640 1 -- 172.21.15.96:0/3010008238 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7f2e700689e0 con 0x7f2e4c06dfa0 2024-05-28T06:21:24.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.773+0000 7f2e67400640 1 -- 172.21.15.96:0/3010008238 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7f2e700689e0 con 0x7f2e4c06dfa0 2024-05-28T06:21:24.769 INFO:teuthology.orchestra.run.smithi096.stdout:[ 2024-05-28T06:21:24.769 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:21:24.769 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "cephfs" 2024-05-28T06:21:24.770 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:21:24.770 INFO:teuthology.orchestra.run.smithi096.stdout:] 2024-05-28T06:21:24.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.775+0000 7f2e74a00640 1 -- 172.21.15.96:0/3010008238 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2e4c06dfa0 msgr2=0x7f2e4c070460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:24.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.775+0000 7f2e74a00640 1 --2- 172.21.15.96:0/3010008238 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2e4c06dfa0 0x7f2e4c070460 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f2e7015f2b0 tx=0x7f2e580079d0 comp rx=0 tx=0).stop 2024-05-28T06:21:24.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.775+0000 7f2e74a00640 1 -- 172.21.15.96:0/3010008238 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2e540039e0 msgr2=0x7f2e7014d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:24.772 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.775+0000 7f2e74a00640 1 --2- 172.21.15.96:0/3010008238 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2e540039e0 0x7f2e7014d6a0 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7f2e6000bb20 tx=0x7f2e600046c0 comp rx=0 tx=0).stop 2024-05-28T06:21:24.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.775+0000 7f2e74a00640 1 -- 172.21.15.96:0/3010008238 shutdown_connections 2024-05-28T06:21:24.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.775+0000 7f2e74a00640 1 --2- 172.21.15.96:0/3010008238 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2e4c06dfa0 0x7f2e4c070460 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:24.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.775+0000 7f2e74a00640 1 --2- 172.21.15.96:0/3010008238 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2e7009a540 0x7f2e7009a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:24.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.775+0000 7f2e74a00640 1 --2- 172.21.15.96:0/3010008238 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e7015de40 0x7f2e70157970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:24.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.775+0000 7f2e74a00640 1 --2- 172.21.15.96:0/3010008238 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2e540039e0 0x7f2e7014d6a0 unknown :-1 s=CLOSED pgs=1676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:24.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.775+0000 7f2e74a00640 1 -- 172.21.15.96:0/3010008238 >> 172.21.15.96:0/3010008238 conn(0x7f2e700b1fc0 msgr2=0x7f2e70071c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:21:24.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.775+0000 7f2e74a00640 1 -- 172.21.15.96:0/3010008238 shutdown_connections 2024-05-28T06:21:24.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.776+0000 7f2e74a00640 1 -- 172.21.15.96:0/3010008238 wait complete. 2024-05-28T06:21:24.789 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-05-28T06:21:24.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.989+0000 7ff0ad600640 1 Processor -- start 2024-05-28T06:21:24.985 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.989+0000 7ff0ad600640 1 -- start start 2024-05-28T06:21:24.987 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.990+0000 7ff0ad600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7ff0a815ce80 0x7ff0a80d09f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:24.987 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.990+0000 7ff0ad600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff0a80d0f30 0x7ff0a8160820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:24.987 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.990+0000 7ff0ad600640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0a815da40 con 0x7ff0a815d260 2024-05-28T06:21:24.987 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.990+0000 7ff0ad600640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7ff0a80cceb0 con 0x7ff0a815ce80 2024-05-28T06:21:24.987 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.990+0000 7ff0ad600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ff0a80d1360 con 0x7ff0a80d0f30 2024-05-28T06:21:24.987 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.990+0000 7ff0a7e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7ff0a815ce80 0x7ff0a80d09f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:24.988 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.990+0000 7ff0a7400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7ff0a815d260 0x7ff0a8160110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:59164/0 (socket says 172.21.15.96:59164) 2024-05-28T06:21:24.988 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.990+0000 7ff0a7400640 1 -- 172.21.15.96:0/1511644661 learned_addr learned my addr 172.21.15.96:0/1511644661 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:21:24.988 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.991+0000 7ff0acc00640 1 --2- 172.21.15.96:0/1511644661 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff0a80d0f30 0x7ff0a8160820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:24.988 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.991+0000 7ff0a7e00640 1 -- 172.21.15.96:0/1511644661 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff0a80d0f30 msgr2=0x7ff0a8160820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:21:24.988 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.991+0000 7ff0a7e00640 1 --2- 172.21.15.96:0/1511644661 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff0a80d0f30 0x7ff0a8160820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:24.988 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.991+0000 7ff0a7e00640 1 -- 172.21.15.96:0/1511644661 >> v1:172.21.15.96:6789/0 conn(0x7ff0a815d260 legacy=0x7ff0a8160110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:24.988 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.991+0000 7ff0a7e00640 1 -- 172.21.15.96:0/1511644661 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0a809ba40 con 0x7ff0a815ce80 2024-05-28T06:21:24.988 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.991+0000 7ff0a6a00640 1 -- 172.21.15.96:0/1511644661 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 923597857 0 0) 0x7ff0a815da40 con 0x7ff0a815d260 2024-05-28T06:21:24.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.992+0000 7ff0a7e00640 1 --2- 172.21.15.96:0/1511644661 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff0a815ce80 0x7ff0a80d09f0 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7ff0980097a0 tx=0x7ff09802a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d13a9541f553522f server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:21:24.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.992+0000 7ff0a6a00640 1 -- 172.21.15.96:0/1511644661 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff098037070 con 0x7ff0a815ce80 2024-05-28T06:21:24.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.992+0000 7ff0a6a00640 1 -- 172.21.15.96:0/1511644661 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff098032440 con 0x7ff0a815ce80 2024-05-28T06:21:24.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.992+0000 7ff0a6a00640 1 -- 172.21.15.96:0/1511644661 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff098032810 con 0x7ff0a815ce80 2024-05-28T06:21:24.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.993+0000 7ff0ad600640 1 -- 172.21.15.96:0/1511644661 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff0a815ce80 msgr2=0x7ff0a80d09f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:24.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.993+0000 7ff0ad600640 1 --2- 172.21.15.96:0/1511644661 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff0a815ce80 0x7ff0a80d09f0 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7ff0980097a0 tx=0x7ff09802a180 comp rx=0 tx=0).stop 2024-05-28T06:21:24.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.993+0000 7ff0ad600640 1 -- 172.21.15.96:0/1511644661 shutdown_connections 2024-05-28T06:21:24.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.993+0000 7ff0ad600640 1 --2- 172.21.15.96:0/1511644661 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff0a80d0f30 0x7ff0a8160820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:24.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.993+0000 7ff0ad600640 1 --2- 172.21.15.96:0/1511644661 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff0a815ce80 0x7ff0a80d09f0 unknown :-1 s=CLOSED pgs=1678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:24.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.993+0000 7ff0ad600640 1 -- 172.21.15.96:0/1511644661 >> 172.21.15.96:0/1511644661 conn(0x7ff0a80b3f30 msgr2=0x7ff0a80b4310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:21:24.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.993+0000 7ff0ad600640 1 -- 172.21.15.96:0/1511644661 shutdown_connections 2024-05-28T06:21:24.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.993+0000 7ff0ad600640 1 -- 172.21.15.96:0/1511644661 wait complete. 2024-05-28T06:21:24.995 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.995+0000 7ff0ad600640 1 Processor -- start 2024-05-28T06:21:24.995 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.995+0000 7ff0ad600640 1 -- start start 2024-05-28T06:21:24.995 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.996+0000 7ff0ad600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0a80d0f30 0x7ff0a815dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:24.995 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.996+0000 7ff0ad600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff0a815ce80 0x7ff0a815e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:24.995 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.996+0000 7ff0ad600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff0a815d260 0x7ff0a8070b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:24.995 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.996+0000 7ff0ad600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff0a80cd6a0 con 0x7ff0a815ce80 2024-05-28T06:21:24.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.996+0000 7ff0ad600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff0a80cd3c0 con 0x7ff0a80d0f30 2024-05-28T06:21:24.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.996+0000 7ff0ad600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ff0a80cd980 con 0x7ff0a815d260 2024-05-28T06:21:24.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.996+0000 7ff0a7400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff0a815ce80 0x7ff0a815e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:24.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.996+0000 7ff0a7400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff0a815ce80 0x7ff0a815e6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:40852/0 (socket says 172.21.15.96:40852) 2024-05-28T06:21:24.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.996+0000 7ff0a7400640 1 -- 172.21.15.96:0/1389753252 learned_addr learned my addr 172.21.15.96:0/1389753252 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:21:24.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.996+0000 7ff0a7400640 1 -- 172.21.15.96:0/1389753252 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff0a815d260 msgr2=0x7ff0a8070b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:24.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.996+0000 7ff0a7400640 1 --2- 172.21.15.96:0/1389753252 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff0a815d260 0x7ff0a8070b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:24.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.996+0000 7ff0a7400640 1 -- 172.21.15.96:0/1389753252 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0a80d0f30 msgr2=0x7ff0a815dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:24.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.996+0000 7ff0a7400640 1 --2- 172.21.15.96:0/1389753252 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0a80d0f30 0x7ff0a815dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:24.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.996+0000 7ff0a7400640 1 -- 172.21.15.96:0/1389753252 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0a815e2a0 con 0x7ff0a815ce80 2024-05-28T06:21:24.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.997+0000 7ff0a7400640 1 --2- 172.21.15.96:0/1389753252 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff0a815ce80 0x7ff0a815e6f0 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7ff09000c7c0 tx=0x7ff09000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:21:24.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.997+0000 7ff09fe00640 1 -- 172.21.15.96:0/1389753252 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff090018070 con 0x7ff0a815ce80 2024-05-28T06:21:24.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.997+0000 7ff09fe00640 1 -- 172.21.15.96:0/1389753252 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff09000a640 con 0x7ff0a815ce80 2024-05-28T06:21:24.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.997+0000 7ff0ad600640 1 -- 172.21.15.96:0/1389753252 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0a81f8480 con 0x7ff0a815ce80 2024-05-28T06:21:24.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.997+0000 7ff09fe00640 1 -- 172.21.15.96:0/1389753252 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff09000aa30 con 0x7ff0a815ce80 2024-05-28T06:21:24.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.997+0000 7ff0ad600640 1 -- 172.21.15.96:0/1389753252 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0a81f8a40 con 0x7ff0a815ce80 2024-05-28T06:21:24.998 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.998+0000 7ff0ad600640 1 -- 172.21.15.96:0/1389753252 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0a80690f0 con 0x7ff0a815ce80 2024-05-28T06:21:24.998 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:24.999+0000 7ff09fe00640 1 -- 172.21.15.96:0/1389753252 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 36) ==== 93080+0+0 (secure 0 0 0) 0x7ff09001b070 con 0x7ff0a815ce80 2024-05-28T06:21:24.999 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:25.000+0000 7ff09fe00640 1 --2- 172.21.15.96:0/1389753252 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff08406df70 0x7ff084070430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:24.999 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:25.000+0000 7ff09fe00640 1 -- 172.21.15.96:0/1389753252 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 5765+0+0 (secure 0 0 0) 0x7ff090089f60 con 0x7ff0a815ce80 2024-05-28T06:21:24.999 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:25.001+0000 7ff0a7e00640 1 --2- 172.21.15.96:0/1389753252 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff08406df70 0x7ff084070430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:24.999 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:25.002+0000 7ff0a7e00640 1 --2- 172.21.15.96:0/1389753252 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff08406df70 0x7ff084070430 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7ff098034000 tx=0x7ff09802c000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:21:25.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:25.005+0000 7ff09fe00640 1 -- 172.21.15.96:0/1389753252 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff090059df0 con 0x7ff0a815ce80 2024-05-28T06:21:25.273 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:25.277+0000 7ff0ad600640 1 -- 172.21.15.96:0/1389753252 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7ff0a80b1b80 con 0x7ff0a815ce80 2024-05-28T06:21:25.273 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:25.277+0000 7ff09fe00640 1 -- 172.21.15.96:0/1389753252 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+2563 (secure 0 0 0) 0x7ff0900596f0 con 0x7ff0a815ce80 2024-05-28T06:21:25.273 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:21:25.274 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:21:25.274 INFO:teuthology.orchestra.run.smithi096.stdout: "fsid": "e4a831b5-e6d1-47a7-96cb-bf85154ee0d2", 2024-05-28T06:21:25.274 INFO:teuthology.orchestra.run.smithi096.stdout: "health": { 2024-05-28T06:21:25.274 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "HEALTH_OK", 2024-05-28T06:21:25.274 INFO:teuthology.orchestra.run.smithi096.stdout: "checks": {}, 2024-05-28T06:21:25.274 INFO:teuthology.orchestra.run.smithi096.stdout: "mutes": [] 2024-05-28T06:21:25.274 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:21:25.274 INFO:teuthology.orchestra.run.smithi096.stdout: "election_epoch": 8, 2024-05-28T06:21:25.274 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum": [ 2024-05-28T06:21:25.274 INFO:teuthology.orchestra.run.smithi096.stdout: 0, 2024-05-28T06:21:25.274 INFO:teuthology.orchestra.run.smithi096.stdout: 1, 2024-05-28T06:21:25.274 INFO:teuthology.orchestra.run.smithi096.stdout: 2 2024-05-28T06:21:25.274 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:21:25.275 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_names": [ 2024-05-28T06:21:25.275 INFO:teuthology.orchestra.run.smithi096.stdout: "a", 2024-05-28T06:21:25.275 INFO:teuthology.orchestra.run.smithi096.stdout: "b", 2024-05-28T06:21:25.275 INFO:teuthology.orchestra.run.smithi096.stdout: "c" 2024-05-28T06:21:25.275 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:21:25.275 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_age": 1058, 2024-05-28T06:21:25.275 INFO:teuthology.orchestra.run.smithi096.stdout: "monmap": { 2024-05-28T06:21:25.275 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 1, 2024-05-28T06:21:25.275 INFO:teuthology.orchestra.run.smithi096.stdout: "min_mon_release_name": "squid", 2024-05-28T06:21:25.275 INFO:teuthology.orchestra.run.smithi096.stdout: "num_mons": 3 2024-05-28T06:21:25.275 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:21:25.275 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap": { 2024-05-28T06:21:25.275 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 103, 2024-05-28T06:21:25.275 INFO:teuthology.orchestra.run.smithi096.stdout: "num_osds": 8, 2024-05-28T06:21:25.276 INFO:teuthology.orchestra.run.smithi096.stdout: "num_up_osds": 8, 2024-05-28T06:21:25.276 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_up_since": 1716876238, 2024-05-28T06:21:25.276 INFO:teuthology.orchestra.run.smithi096.stdout: "num_in_osds": 8, 2024-05-28T06:21:25.276 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_in_since": 1716876229, 2024-05-28T06:21:25.276 INFO:teuthology.orchestra.run.smithi096.stdout: "num_remapped_pgs": 0 2024-05-28T06:21:25.276 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:21:25.276 INFO:teuthology.orchestra.run.smithi096.stdout: "pgmap": { 2024-05-28T06:21:25.276 INFO:teuthology.orchestra.run.smithi096.stdout: "pgs_by_state": [ 2024-05-28T06:21:25.276 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:21:25.276 INFO:teuthology.orchestra.run.smithi096.stdout: "state_name": "active+clean", 2024-05-28T06:21:25.276 INFO:teuthology.orchestra.run.smithi096.stdout: "count": 97 2024-05-28T06:21:25.277 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:21:25.277 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:21:25.277 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pgs": 97, 2024-05-28T06:21:25.277 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pools": 3, 2024-05-28T06:21:25.277 INFO:teuthology.orchestra.run.smithi096.stdout: "num_objects": 800, 2024-05-28T06:21:25.277 INFO:teuthology.orchestra.run.smithi096.stdout: "data_bytes": 3222142595, 2024-05-28T06:21:25.277 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_used": 7344906240, 2024-05-28T06:21:25.277 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_avail": 765749207040, 2024-05-28T06:21:25.277 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_total": 773094113280, 2024-05-28T06:21:25.277 INFO:teuthology.orchestra.run.smithi096.stdout: "read_bytes_sec": 511, 2024-05-28T06:21:25.277 INFO:teuthology.orchestra.run.smithi096.stdout: "write_bytes_sec": 11685, 2024-05-28T06:21:25.278 INFO:teuthology.orchestra.run.smithi096.stdout: "read_op_per_sec": 0, 2024-05-28T06:21:25.278 INFO:teuthology.orchestra.run.smithi096.stdout: "write_op_per_sec": 3 2024-05-28T06:21:25.278 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:21:25.278 INFO:teuthology.orchestra.run.smithi096.stdout: "fsmap": { 2024-05-28T06:21:25.278 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 63, 2024-05-28T06:21:25.278 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 7, 2024-05-28T06:21:25.278 INFO:teuthology.orchestra.run.smithi096.stdout: "up": 1, 2024-05-28T06:21:25.278 INFO:teuthology.orchestra.run.smithi096.stdout: "in": 1, 2024-05-28T06:21:25.278 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 1, 2024-05-28T06:21:25.278 INFO:teuthology.orchestra.run.smithi096.stdout: "by_rank": [ 2024-05-28T06:21:25.278 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:21:25.278 INFO:teuthology.orchestra.run.smithi096.stdout: "filesystem_id": 7, 2024-05-28T06:21:25.278 INFO:teuthology.orchestra.run.smithi096.stdout: "rank": 0, 2024-05-28T06:21:25.279 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "b", 2024-05-28T06:21:25.279 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "up:active", 2024-05-28T06:21:25.279 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 8739 2024-05-28T06:21:25.279 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:21:25.279 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:21:25.279 INFO:teuthology.orchestra.run.smithi096.stdout: "up:standby": 3 2024-05-28T06:21:25.279 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:21:25.279 INFO:teuthology.orchestra.run.smithi096.stdout: "mgrmap": { 2024-05-28T06:21:25.279 INFO:teuthology.orchestra.run.smithi096.stdout: "available": true, 2024-05-28T06:21:25.279 INFO:teuthology.orchestra.run.smithi096.stdout: "num_standbys": 1, 2024-05-28T06:21:25.279 INFO:teuthology.orchestra.run.smithi096.stdout: "modules": [ 2024-05-28T06:21:25.279 INFO:teuthology.orchestra.run.smithi096.stdout: "iostat", 2024-05-28T06:21:25.279 INFO:teuthology.orchestra.run.smithi096.stdout: "nfs", 2024-05-28T06:21:25.279 INFO:teuthology.orchestra.run.smithi096.stdout: "restful" 2024-05-28T06:21:25.280 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:21:25.280 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:21:25.280 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:21:25.280 INFO:teuthology.orchestra.run.smithi096.stdout: "servicemap": { 2024-05-28T06:21:25.280 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 19, 2024-05-28T06:21:25.280 INFO:teuthology.orchestra.run.smithi096.stdout: "modified": "2024-05-28T06:21:15.939159+0000", 2024-05-28T06:21:25.280 INFO:teuthology.orchestra.run.smithi096.stdout: "services": { 2024-05-28T06:21:25.280 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-05-28T06:21:25.280 INFO:teuthology.orchestra.run.smithi096.stdout: "daemons": { 2024-05-28T06:21:25.280 INFO:teuthology.orchestra.run.smithi096.stdout: "summary": "", 2024-05-28T06:21:25.280 INFO:teuthology.orchestra.run.smithi096.stdout: "b": { 2024-05-28T06:21:25.280 INFO:teuthology.orchestra.run.smithi096.stdout: "start_epoch": 0, 2024-05-28T06:21:25.280 INFO:teuthology.orchestra.run.smithi096.stdout: "start_stamp": "0.000000", 2024-05-28T06:21:25.281 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 0, 2024-05-28T06:21:25.281 INFO:teuthology.orchestra.run.smithi096.stdout: "addr": "(unrecognized address family 0)/0", 2024-05-28T06:21:25.281 INFO:teuthology.orchestra.run.smithi096.stdout: "metadata": {}, 2024-05-28T06:21:25.281 INFO:teuthology.orchestra.run.smithi096.stdout: "task_status": {} 2024-05-28T06:21:25.281 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:21:25.281 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:21:25.281 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:21:25.281 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:21:25.281 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:21:25.281 INFO:teuthology.orchestra.run.smithi096.stdout: "progress_events": { 2024-05-28T06:21:25.281 INFO:teuthology.orchestra.run.smithi096.stdout: "885b6d9d-736d-45a8-ac79-be8f5b8ac230": { 2024-05-28T06:21:25.281 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "All cloning finished, waiting 26.0 seconds before exiting (5s)\n [============================] ", 2024-05-28T06:21:25.281 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": 1, 2024-05-28T06:21:25.281 INFO:teuthology.orchestra.run.smithi096.stdout: "add_to_ceph_s": true 2024-05-28T06:21:25.282 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:21:25.282 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:21:25.282 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:21:25.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:25.280+0000 7ff0ad600640 1 -- 172.21.15.96:0/1389753252 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff08406df70 msgr2=0x7ff084070430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:25.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:25.280+0000 7ff0ad600640 1 --2- 172.21.15.96:0/1389753252 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff08406df70 0x7ff084070430 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7ff098034000 tx=0x7ff09802c000 comp rx=0 tx=0).stop 2024-05-28T06:21:25.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:25.280+0000 7ff0ad600640 1 -- 172.21.15.96:0/1389753252 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff0a815ce80 msgr2=0x7ff0a815e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:25.283 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:25.280+0000 7ff0ad600640 1 --2- 172.21.15.96:0/1389753252 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff0a815ce80 0x7ff0a815e6f0 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7ff09000c7c0 tx=0x7ff09000cc80 comp rx=0 tx=0).stop 2024-05-28T06:21:25.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:25.280+0000 7ff0ad600640 1 -- 172.21.15.96:0/1389753252 shutdown_connections 2024-05-28T06:21:25.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:25.280+0000 7ff0ad600640 1 --2- 172.21.15.96:0/1389753252 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff08406df70 0x7ff084070430 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:25.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:25.280+0000 7ff0ad600640 1 --2- 172.21.15.96:0/1389753252 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff0a815d260 0x7ff0a8070b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:25.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:25.280+0000 7ff0ad600640 1 --2- 172.21.15.96:0/1389753252 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff0a815ce80 0x7ff0a815e6f0 unknown :-1 s=CLOSED pgs=1679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:25.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:25.280+0000 7ff0ad600640 1 --2- 172.21.15.96:0/1389753252 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0a80d0f30 0x7ff0a815dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:25.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:25.280+0000 7ff0ad600640 1 -- 172.21.15.96:0/1389753252 >> 172.21.15.96:0/1389753252 conn(0x7ff0a80b3f30 msgr2=0x7ff0a80cfbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:21:25.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:25.280+0000 7ff0ad600640 1 -- 172.21.15.96:0/1389753252 shutdown_connections 2024-05-28T06:21:25.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:25.280+0000 7ff0ad600640 1 -- 172.21.15.96:0/1389753252 wait complete. 2024-05-28T06:21:30.296 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-05-28T06:21:30.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:30.501+0000 7ffa6fe00640 1 Processor -- start 2024-05-28T06:21:30.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:30.502+0000 7ffa6fe00640 1 -- start start 2024-05-28T06:21:30.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:30.503+0000 7ffa6fe00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffa680723d0 0x7ffa680b3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:30.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:30.503+0000 7ffa6fe00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7ffa680b41e0 0x7ffa680cf880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:30.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:30.503+0000 7ffa6fe00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa680cf0c0 con 0x7ffa68145320 2024-05-28T06:21:30.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:30.503+0000 7ffa6fe00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7ffa680b4a20 con 0x7ffa680b41e0 2024-05-28T06:21:30.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:30.503+0000 7ffa6fe00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ffa680b4d00 con 0x7ffa680723d0 2024-05-28T06:21:30.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:30.504+0000 7ffa6ca00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffa680723d0 0x7ffa680b3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:30.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:30.504+0000 7ffa6de00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7ffa680b41e0 0x7ffa680cf880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:30.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:30.504+0000 7ffa6ca00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffa680723d0 0x7ffa680b3ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.96:41078/0 (socket says 172.21.15.96:41078) 2024-05-28T06:21:30.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:30.504+0000 7ffa6ca00640 1 -- 172.21.15.96:0/3499733341 learned_addr learned my addr 172.21.15.96:0/3499733341 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:21:30.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:30.504+0000 7ffa6d400640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7ffa68145320 0x7ffa68071cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.96:58016/0 (socket says 172.21.15.96:58016) 2024-05-28T06:21:30.501 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:30.505+0000 7ffa6ca00640 1 -- 172.21.15.96:0/3499733341 >> v2:172.21.15.123:3300/0 conn(0x7ffa680b41e0 msgr2=0x7ffa680cf880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:21:30.501 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:30.505+0000 7ffa6ca00640 1 --2- 172.21.15.96:0/3499733341 >> v2:172.21.15.123:3300/0 conn(0x7ffa680b41e0 0x7ffa680cf880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:30.501 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:30.505+0000 7ffa6ca00640 1 -- 172.21.15.96:0/3499733341 >> v1:172.21.15.123:6789/0 conn(0x7ffa68145320 legacy=0x7ffa68071cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:30.501 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:30.505+0000 7ffa6ca00640 1 -- 172.21.15.96:0/3499733341 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa680b4fe0 con 0x7ffa680723d0 2024-05-28T06:21:30.501 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:30.505+0000 7ffa6de00640 1 --2- 172.21.15.96:0/3499733341 >> v2:172.21.15.123:3300/0 conn(0x7ffa680b41e0 0x7ffa680cf880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-28T06:21:30.502 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:30.506+0000 7ffa6ca00640 1 --2- 172.21.15.96:0/3499733341 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffa680723d0 0x7ffa680b3ca0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ffa54009870 tx=0x7ffa5402a250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=beb1e73428bf41e server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:21:30.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:30.507+0000 7ffa67e00640 1 -- 172.21.15.96:0/3499733341 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa54037070 con 0x7ffa680723d0 2024-05-28T06:21:30.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:30.507+0000 7ffa67e00640 1 -- 172.21.15.96:0/3499733341 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa54032440 con 0x7ffa680723d0 2024-05-28T06:21:30.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:30.507+0000 7ffa67e00640 1 -- 172.21.15.96:0/3499733341 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa54032810 con 0x7ffa680723d0 2024-05-28T06:21:30.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:30.508+0000 7ffa6fe00640 1 -- 172.21.15.96:0/3499733341 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffa680723d0 msgr2=0x7ffa680b3ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:30.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:30.508+0000 7ffa6fe00640 1 --2- 172.21.15.96:0/3499733341 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffa680723d0 0x7ffa680b3ca0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ffa54009870 tx=0x7ffa5402a250 comp rx=0 tx=0).stop 2024-05-28T06:21:30.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:30.508+0000 7ffa6fe00640 1 -- 172.21.15.96:0/3499733341 shutdown_connections 2024-05-28T06:21:30.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:30.508+0000 7ffa6fe00640 1 --2- 172.21.15.96:0/3499733341 >> v2:172.21.15.123:3300/0 conn(0x7ffa680b41e0 0x7ffa680cf880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:30.505 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:30.508+0000 7ffa6fe00640 1 --2- 172.21.15.96:0/3499733341 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffa680723d0 0x7ffa680b3ca0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:30.505 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:30.508+0000 7ffa6fe00640 1 -- 172.21.15.96:0/3499733341 >> 172.21.15.96:0/3499733341 conn(0x7ffa68067ce0 msgr2=0x7ffa6806b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:21:30.505 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:30.508+0000 7ffa6fe00640 1 -- 172.21.15.96:0/3499733341 shutdown_connections 2024-05-28T06:21:30.505 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:30.508+0000 7ffa6fe00640 1 -- 172.21.15.96:0/3499733341 wait complete. 2024-05-28T06:21:30.506 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:30.510+0000 7ffa6fe00640 1 Processor -- start 2024-05-28T06:21:30.506 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:30.510+0000 7ffa6fe00640 1 -- start start 2024-05-28T06:21:30.509 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:30.511+0000 7ffa6fe00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffa680723d0 0x7ffa681e4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:30.509 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:30.511+0000 7ffa6fe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa680b41e0 0x7ffa68143f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:30.509 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:30.511+0000 7ffa6fe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa68145320 0x7ffa6815b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:30.509 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:30.511+0000 7ffa6fe00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ffa680b4670 con 0x7ffa68145320 2024-05-28T06:21:30.510 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:30.511+0000 7ffa6fe00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ffa68145700 con 0x7ffa680b41e0 2024-05-28T06:21:30.510 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:30.511+0000 7ffa6fe00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ffa681444b0 con 0x7ffa680723d0 2024-05-28T06:21:30.510 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:30.511+0000 7ffa6de00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa68145320 0x7ffa6815b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:30.510 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:30.511+0000 7ffa6de00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa68145320 0x7ffa6815b940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48790/0 (socket says 172.21.15.96:48790) 2024-05-28T06:21:30.510 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:30.511+0000 7ffa6de00640 1 -- 172.21.15.96:0/2982694290 learned_addr learned my addr 172.21.15.96:0/2982694290 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:21:30.510 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:30.511+0000 7ffa6d400640 1 --2- 172.21.15.96:0/2982694290 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffa680723d0 0x7ffa681e4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:30.510 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:30.512+0000 7ffa6ca00640 1 --2- 172.21.15.96:0/2982694290 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa680b41e0 0x7ffa68143f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:30.510 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:30.512+0000 7ffa6de00640 1 -- 172.21.15.96:0/2982694290 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffa680723d0 msgr2=0x7ffa681e4ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:30.510 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:30.512+0000 7ffa6de00640 1 --2- 172.21.15.96:0/2982694290 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffa680723d0 0x7ffa681e4ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:30.511 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:30.512+0000 7ffa6de00640 1 -- 172.21.15.96:0/2982694290 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa680b41e0 msgr2=0x7ffa68143f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:30.511 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:30.512+0000 7ffa6de00640 1 --2- 172.21.15.96:0/2982694290 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa680b41e0 0x7ffa68143f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:30.511 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:30.512+0000 7ffa6de00640 1 -- 172.21.15.96:0/2982694290 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa681e54c0 con 0x7ffa68145320 2024-05-28T06:21:30.511 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:30.512+0000 7ffa6de00640 1 --2- 172.21.15.96:0/2982694290 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa68145320 0x7ffa6815b940 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7ffa6000efc0 tx=0x7ffa60009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:21:30.511 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:30.512+0000 7ffa65600640 1 -- 172.21.15.96:0/2982694290 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa60018030 con 0x7ffa68145320 2024-05-28T06:21:30.511 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:30.512+0000 7ffa65600640 1 -- 172.21.15.96:0/2982694290 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa6000cde0 con 0x7ffa68145320 2024-05-28T06:21:30.511 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:30.512+0000 7ffa6fe00640 1 -- 172.21.15.96:0/2982694290 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa681e57a0 con 0x7ffa68145320 2024-05-28T06:21:30.511 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:30.512+0000 7ffa65600640 1 -- 172.21.15.96:0/2982694290 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa60003bb0 con 0x7ffa68145320 2024-05-28T06:21:30.512 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:30.512+0000 7ffa6fe00640 1 -- 172.21.15.96:0/2982694290 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa681e5d60 con 0x7ffa68145320 2024-05-28T06:21:30.512 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:30.514+0000 7ffa6fe00640 1 -- 172.21.15.96:0/2982694290 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa680689e0 con 0x7ffa68145320 2024-05-28T06:21:30.512 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:30.515+0000 7ffa65600640 1 -- 172.21.15.96:0/2982694290 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 36) ==== 93080+0+0 (secure 0 0 0) 0x7ffa6001b070 con 0x7ffa68145320 2024-05-28T06:21:30.513 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:30.516+0000 7ffa65600640 1 --2- 172.21.15.96:0/2982694290 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ffa4806df70 0x7ffa48070430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:30.513 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:30.516+0000 7ffa65600640 1 -- 172.21.15.96:0/2982694290 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 5765+0+0 (secure 0 0 0) 0x7ffa6008b2e0 con 0x7ffa68145320 2024-05-28T06:21:30.513 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:30.516+0000 7ffa6d400640 1 --2- 172.21.15.96:0/2982694290 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ffa4806df70 0x7ffa48070430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:30.513 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:30.517+0000 7ffa6d400640 1 --2- 172.21.15.96:0/2982694290 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ffa4806df70 0x7ffa48070430 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7ffa68140010 tx=0x7ffa5c007ce0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:21:30.516 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:30.520+0000 7ffa65600640 1 -- 172.21.15.96:0/2982694290 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ffa60010040 con 0x7ffa68145320 2024-05-28T06:21:30.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:30.799+0000 7ffa6fe00640 1 -- 172.21.15.96:0/2982694290 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7ffa680b34e0 con 0x7ffa68145320 2024-05-28T06:21:30.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:30.800+0000 7ffa65600640 1 -- 172.21.15.96:0/2982694290 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+2110 (secure 0 0 0) 0x7ffa60012340 con 0x7ffa68145320 2024-05-28T06:21:30.796 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:21:30.796 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:21:30.797 INFO:teuthology.orchestra.run.smithi096.stdout: "fsid": "e4a831b5-e6d1-47a7-96cb-bf85154ee0d2", 2024-05-28T06:21:30.797 INFO:teuthology.orchestra.run.smithi096.stdout: "health": { 2024-05-28T06:21:30.797 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "HEALTH_OK", 2024-05-28T06:21:30.797 INFO:teuthology.orchestra.run.smithi096.stdout: "checks": {}, 2024-05-28T06:21:30.797 INFO:teuthology.orchestra.run.smithi096.stdout: "mutes": [] 2024-05-28T06:21:30.797 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:21:30.797 INFO:teuthology.orchestra.run.smithi096.stdout: "election_epoch": 8, 2024-05-28T06:21:30.797 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum": [ 2024-05-28T06:21:30.797 INFO:teuthology.orchestra.run.smithi096.stdout: 0, 2024-05-28T06:21:30.797 INFO:teuthology.orchestra.run.smithi096.stdout: 1, 2024-05-28T06:21:30.797 INFO:teuthology.orchestra.run.smithi096.stdout: 2 2024-05-28T06:21:30.797 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:21:30.797 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_names": [ 2024-05-28T06:21:30.797 INFO:teuthology.orchestra.run.smithi096.stdout: "a", 2024-05-28T06:21:30.798 INFO:teuthology.orchestra.run.smithi096.stdout: "b", 2024-05-28T06:21:30.798 INFO:teuthology.orchestra.run.smithi096.stdout: "c" 2024-05-28T06:21:30.798 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:21:30.798 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_age": 1064, 2024-05-28T06:21:30.798 INFO:teuthology.orchestra.run.smithi096.stdout: "monmap": { 2024-05-28T06:21:30.798 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 1, 2024-05-28T06:21:30.798 INFO:teuthology.orchestra.run.smithi096.stdout: "min_mon_release_name": "squid", 2024-05-28T06:21:30.798 INFO:teuthology.orchestra.run.smithi096.stdout: "num_mons": 3 2024-05-28T06:21:30.798 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:21:30.798 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap": { 2024-05-28T06:21:30.798 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 103, 2024-05-28T06:21:30.798 INFO:teuthology.orchestra.run.smithi096.stdout: "num_osds": 8, 2024-05-28T06:21:30.798 INFO:teuthology.orchestra.run.smithi096.stdout: "num_up_osds": 8, 2024-05-28T06:21:30.799 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_up_since": 1716876238, 2024-05-28T06:21:30.799 INFO:teuthology.orchestra.run.smithi096.stdout: "num_in_osds": 8, 2024-05-28T06:21:30.799 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_in_since": 1716876229, 2024-05-28T06:21:30.799 INFO:teuthology.orchestra.run.smithi096.stdout: "num_remapped_pgs": 0 2024-05-28T06:21:30.799 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:21:30.799 INFO:teuthology.orchestra.run.smithi096.stdout: "pgmap": { 2024-05-28T06:21:30.799 INFO:teuthology.orchestra.run.smithi096.stdout: "pgs_by_state": [ 2024-05-28T06:21:30.799 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:21:30.799 INFO:teuthology.orchestra.run.smithi096.stdout: "state_name": "active+clean", 2024-05-28T06:21:30.799 INFO:teuthology.orchestra.run.smithi096.stdout: "count": 97 2024-05-28T06:21:30.799 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:21:30.799 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:21:30.799 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pgs": 97, 2024-05-28T06:21:30.799 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pools": 3, 2024-05-28T06:21:30.800 INFO:teuthology.orchestra.run.smithi096.stdout: "num_objects": 798, 2024-05-28T06:21:30.800 INFO:teuthology.orchestra.run.smithi096.stdout: "data_bytes": 3222170312, 2024-05-28T06:21:30.800 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_used": 7344947200, 2024-05-28T06:21:30.800 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_avail": 765749166080, 2024-05-28T06:21:30.800 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_total": 773094113280, 2024-05-28T06:21:30.800 INFO:teuthology.orchestra.run.smithi096.stdout: "read_bytes_sec": 682, 2024-05-28T06:21:30.800 INFO:teuthology.orchestra.run.smithi096.stdout: "write_bytes_sec": 8018, 2024-05-28T06:21:30.800 INFO:teuthology.orchestra.run.smithi096.stdout: "read_op_per_sec": 0, 2024-05-28T06:21:30.800 INFO:teuthology.orchestra.run.smithi096.stdout: "write_op_per_sec": 3 2024-05-28T06:21:30.800 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:21:30.800 INFO:teuthology.orchestra.run.smithi096.stdout: "fsmap": { 2024-05-28T06:21:30.800 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 63, 2024-05-28T06:21:30.800 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 7, 2024-05-28T06:21:30.801 INFO:teuthology.orchestra.run.smithi096.stdout: "up": 1, 2024-05-28T06:21:30.801 INFO:teuthology.orchestra.run.smithi096.stdout: "in": 1, 2024-05-28T06:21:30.801 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 1, 2024-05-28T06:21:30.801 INFO:teuthology.orchestra.run.smithi096.stdout: "by_rank": [ 2024-05-28T06:21:30.801 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:21:30.801 INFO:teuthology.orchestra.run.smithi096.stdout: "filesystem_id": 7, 2024-05-28T06:21:30.801 INFO:teuthology.orchestra.run.smithi096.stdout: "rank": 0, 2024-05-28T06:21:30.801 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "b", 2024-05-28T06:21:30.801 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "up:active", 2024-05-28T06:21:30.801 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 8739 2024-05-28T06:21:30.801 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:21:30.801 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:21:30.801 INFO:teuthology.orchestra.run.smithi096.stdout: "up:standby": 3 2024-05-28T06:21:30.801 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:21:30.802 INFO:teuthology.orchestra.run.smithi096.stdout: "mgrmap": { 2024-05-28T06:21:30.802 INFO:teuthology.orchestra.run.smithi096.stdout: "available": true, 2024-05-28T06:21:30.802 INFO:teuthology.orchestra.run.smithi096.stdout: "num_standbys": 1, 2024-05-28T06:21:30.802 INFO:teuthology.orchestra.run.smithi096.stdout: "modules": [ 2024-05-28T06:21:30.802 INFO:teuthology.orchestra.run.smithi096.stdout: "iostat", 2024-05-28T06:21:30.802 INFO:teuthology.orchestra.run.smithi096.stdout: "nfs", 2024-05-28T06:21:30.802 INFO:teuthology.orchestra.run.smithi096.stdout: "restful" 2024-05-28T06:21:30.802 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:21:30.802 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:21:30.802 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:21:30.802 INFO:teuthology.orchestra.run.smithi096.stdout: "servicemap": { 2024-05-28T06:21:30.802 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 20, 2024-05-28T06:21:30.802 INFO:teuthology.orchestra.run.smithi096.stdout: "modified": "2024-05-28T06:21:25.943324+0000", 2024-05-28T06:21:30.802 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:21:30.803 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:21:30.803 INFO:teuthology.orchestra.run.smithi096.stdout: "progress_events": { 2024-05-28T06:21:30.803 INFO:teuthology.orchestra.run.smithi096.stdout: "885b6d9d-736d-45a8-ac79-be8f5b8ac230": { 2024-05-28T06:21:30.803 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "All cloning finished, waiting 20.0 seconds before exiting (11s)\n [============================] ", 2024-05-28T06:21:30.803 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": 1, 2024-05-28T06:21:30.803 INFO:teuthology.orchestra.run.smithi096.stdout: "add_to_ceph_s": true 2024-05-28T06:21:30.803 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:21:30.803 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:21:30.803 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:21:30.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:30.803+0000 7ffa6fe00640 1 -- 172.21.15.96:0/2982694290 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ffa4806df70 msgr2=0x7ffa48070430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:30.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:30.803+0000 7ffa6fe00640 1 --2- 172.21.15.96:0/2982694290 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ffa4806df70 0x7ffa48070430 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7ffa68140010 tx=0x7ffa5c007ce0 comp rx=0 tx=0).stop 2024-05-28T06:21:30.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:30.803+0000 7ffa6fe00640 1 -- 172.21.15.96:0/2982694290 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa68145320 msgr2=0x7ffa6815b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:30.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:30.803+0000 7ffa6fe00640 1 --2- 172.21.15.96:0/2982694290 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa68145320 0x7ffa6815b940 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7ffa6000efc0 tx=0x7ffa60009f90 comp rx=0 tx=0).stop 2024-05-28T06:21:30.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:30.804+0000 7ffa6fe00640 1 -- 172.21.15.96:0/2982694290 shutdown_connections 2024-05-28T06:21:30.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:30.804+0000 7ffa6fe00640 1 --2- 172.21.15.96:0/2982694290 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ffa4806df70 0x7ffa48070430 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:30.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:30.804+0000 7ffa6fe00640 1 --2- 172.21.15.96:0/2982694290 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa68145320 0x7ffa6815b940 unknown :-1 s=CLOSED pgs=1680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:30.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:30.804+0000 7ffa6fe00640 1 --2- 172.21.15.96:0/2982694290 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa680b41e0 0x7ffa68143f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:30.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:30.804+0000 7ffa6fe00640 1 --2- 172.21.15.96:0/2982694290 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffa680723d0 0x7ffa681e4ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:30.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:30.804+0000 7ffa6fe00640 1 -- 172.21.15.96:0/2982694290 >> 172.21.15.96:0/2982694290 conn(0x7ffa68067ce0 msgr2=0x7ffa6815cd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:21:30.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:30.804+0000 7ffa6fe00640 1 -- 172.21.15.96:0/2982694290 shutdown_connections 2024-05-28T06:21:30.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:30.804+0000 7ffa6fe00640 1 -- 172.21.15.96:0/2982694290 wait complete. 2024-05-28T06:21:34.012 DEBUG:teuthology.orchestra.run.smithi096:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:21:34.021 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:21:35.821 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-05-28T06:21:36.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.036+0000 7f61ab000640 1 Processor -- start 2024-05-28T06:21:36.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.036+0000 7f61ab000640 1 -- start start 2024-05-28T06:21:36.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.037+0000 7f61ab000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f61a4155a00 0x7f61a415baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:36.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.038+0000 7f61ab000640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61a406e3d0 con 0x7f61a406f390 2024-05-28T06:21:36.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.038+0000 7f61ab000640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61a415c0b0 con 0x7f61a40d0970 2024-05-28T06:21:36.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.038+0000 7f61ab000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f61a40ccf20 con 0x7f61a4155a00 2024-05-28T06:21:36.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.038+0000 7f61a9000640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f61a406f390 0x7f61a415f5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:47466/0 (socket says 172.21.15.96:47466) 2024-05-28T06:21:36.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.038+0000 7f61a9000640 1 -- 172.21.15.96:0/1123772675 learned_addr learned my addr 172.21.15.96:0/1123772675 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:21:36.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.038+0000 7f61a3e00640 1 --2- 172.21.15.96:0/1123772675 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f61a4155a00 0x7f61a415baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:36.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.038+0000 7f61a2a00640 1 -- 172.21.15.96:0/1123772675 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 785523190 0 0) 0x7f61a406e3d0 con 0x7f61a406f390 2024-05-28T06:21:36.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.039+0000 7f61a2a00640 1 -- 172.21.15.96:0/1123772675 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6188005730 con 0x7f61a406f390 2024-05-28T06:21:36.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.039+0000 7f61a2a00640 1 -- 172.21.15.96:0/1123772675 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3514585235 0 0) 0x7f6188005730 con 0x7f61a406f390 2024-05-28T06:21:36.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.039+0000 7f61a2a00640 1 -- 172.21.15.96:0/1123772675 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f61a4155a00 msgr2=0x7f61a415baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:21:36.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.039+0000 7f61a2a00640 1 --2- 172.21.15.96:0/1123772675 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f61a4155a00 0x7f61a415baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:36.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.039+0000 7f61a2a00640 1 -- 172.21.15.96:0/1123772675 >> v1:172.21.15.123:6789/0 conn(0x7f61a40d0970 legacy=0x7f61a406dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:36.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.039+0000 7f61a2a00640 1 -- 172.21.15.96:0/1123772675 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61a415c390 con 0x7f61a406f390 2024-05-28T06:21:36.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.039+0000 7f61a2a00640 1 -- 172.21.15.96:0/1123772675 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f619c0019f0 con 0x7f61a406f390 2024-05-28T06:21:36.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.039+0000 7f61a2a00640 1 -- 172.21.15.96:0/1123772675 >> v1:172.21.15.96:6789/0 conn(0x7f61a406f390 legacy=0x7f61a415f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:36.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.039+0000 7f61a2a00640 1 --2- 172.21.15.96:0/1123772675 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f61a4155a00 0x7f6188005bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:36.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.039+0000 7f61a2a00640 1 --2- 172.21.15.96:0/1123772675 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6188006130 0x7f6188006a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:36.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.039+0000 7f61a2a00640 1 --2- 172.21.15.96:0/1123772675 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6188006f50 0x7f61880073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:36.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.039+0000 7f61a2a00640 1 -- 172.21.15.96:0/1123772675 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f618800ffd0 con 0x7f6188006130 2024-05-28T06:21:36.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.039+0000 7f61a2a00640 1 -- 172.21.15.96:0/1123772675 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f6188010150 con 0x7f6188006f50 2024-05-28T06:21:36.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.039+0000 7f61a2a00640 1 -- 172.21.15.96:0/1123772675 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f61880102d0 con 0x7f61a4155a00 2024-05-28T06:21:36.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.039+0000 7f61a3400640 1 --2- 172.21.15.96:0/1123772675 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6188006130 0x7f6188006a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:36.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.040+0000 7f61a3e00640 1 --2- 172.21.15.96:0/1123772675 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f61a4155a00 0x7f6188005bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:36.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.040+0000 7f61a9000640 1 --2- 172.21.15.96:0/1123772675 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6188006f50 0x7f61880073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:36.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.040+0000 7f61a3400640 1 -- 172.21.15.96:0/1123772675 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f61a4155a00 msgr2=0x7f6188005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:21:36.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.040+0000 7f61a3400640 1 --2- 172.21.15.96:0/1123772675 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f61a4155a00 0x7f6188005bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:36.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.040+0000 7f61a3400640 1 -- 172.21.15.96:0/1123772675 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6188006f50 msgr2=0x7f61880073d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:21:36.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.040+0000 7f61a3400640 1 --2- 172.21.15.96:0/1123772675 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6188006f50 0x7f61880073d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:36.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.040+0000 7f61a3400640 1 -- 172.21.15.96:0/1123772675 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6188005730 con 0x7f6188006130 2024-05-28T06:21:36.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.041+0000 7f61a3400640 1 --2- 172.21.15.96:0/1123772675 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6188006130 0x7f6188006a10 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7f618c001750 tx=0x7f6188007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18727e69a7cbf59 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:21:36.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.041+0000 7f61a2a00640 1 -- 172.21.15.96:0/1123772675 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f618c001370 con 0x7f6188006130 2024-05-28T06:21:36.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.041+0000 7f61a2a00640 1 -- 172.21.15.96:0/1123772675 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f618c0052d0 con 0x7f6188006130 2024-05-28T06:21:36.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.041+0000 7f61a2a00640 1 -- 172.21.15.96:0/1123772675 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f618c02abe0 con 0x7f6188006130 2024-05-28T06:21:36.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.042+0000 7f61ab000640 1 -- 172.21.15.96:0/1123772675 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6188006130 msgr2=0x7f6188006a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:36.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.042+0000 7f61ab000640 1 --2- 172.21.15.96:0/1123772675 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6188006130 0x7f6188006a10 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7f618c001750 tx=0x7f6188007a40 comp rx=0 tx=0).stop 2024-05-28T06:21:36.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.042+0000 7f61a9000640 1 -- 172.21.15.96:0/1123772675 reap_dead start 2024-05-28T06:21:36.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.042+0000 7f61ab000640 1 -- 172.21.15.96:0/1123772675 shutdown_connections 2024-05-28T06:21:36.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.042+0000 7f61ab000640 1 -- 172.21.15.96:0/1123772675 >> 172.21.15.96:0/1123772675 conn(0x7f61a4067ce0 msgr2=0x7f61a406b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:21:36.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.042+0000 7f61ab000640 1 -- 172.21.15.96:0/1123772675 shutdown_connections 2024-05-28T06:21:36.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.042+0000 7f61ab000640 1 -- 172.21.15.96:0/1123772675 wait complete. 2024-05-28T06:21:36.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.044+0000 7f61ab000640 1 Processor -- start 2024-05-28T06:21:36.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.044+0000 7f61ab000640 1 -- start start 2024-05-28T06:21:36.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.045+0000 7f61ab000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6188006130 0x7f61a414cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:36.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.045+0000 7f61ab000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61a4152910 0x7f61a4142e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:36.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.045+0000 7f61ab000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f61a415dc90 0x7f61a415e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:36.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.045+0000 7f61ab000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f61a415c950 con 0x7f6188006130 2024-05-28T06:21:36.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.045+0000 7f61ab000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f61a415f090 con 0x7f61a4152910 2024-05-28T06:21:36.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.045+0000 7f61ab000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f61a4155de0 con 0x7f61a415dc90 2024-05-28T06:21:36.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.045+0000 7f61a3e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6188006130 0x7f61a414cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:36.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.045+0000 7f61a3e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6188006130 0x7f61a414cca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:48814/0 (socket says 172.21.15.96:48814) 2024-05-28T06:21:36.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.045+0000 7f61a3e00640 1 -- 172.21.15.96:0/1409047179 learned_addr learned my addr 172.21.15.96:0/1409047179 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:21:36.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.046+0000 7f61a3e00640 1 -- 172.21.15.96:0/1409047179 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f61a415dc90 msgr2=0x7f61a415e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:36.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.046+0000 7f61a3400640 1 --2- 172.21.15.96:0/1409047179 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61a4152910 0x7f61a4142e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:36.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.046+0000 7f61a9000640 1 --2- 172.21.15.96:0/1409047179 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f61a415dc90 0x7f61a415e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:36.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.046+0000 7f61a3e00640 1 --2- 172.21.15.96:0/1409047179 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f61a415dc90 0x7f61a415e120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:36.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.046+0000 7f61a3e00640 1 -- 172.21.15.96:0/1409047179 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61a4152910 msgr2=0x7f61a4142e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:36.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.046+0000 7f61a3e00640 1 --2- 172.21.15.96:0/1409047179 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61a4152910 0x7f61a4142e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:36.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.046+0000 7f61a3e00640 1 -- 172.21.15.96:0/1409047179 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61a4151680 con 0x7f6188006130 2024-05-28T06:21:36.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.046+0000 7f61a3400640 1 --2- 172.21.15.96:0/1409047179 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61a4152910 0x7f61a4142e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:21:36.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.046+0000 7f61a3e00640 1 --2- 172.21.15.96:0/1409047179 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6188006130 0x7f61a414cca0 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7f6194011630 tx=0x7f6194011af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:21:36.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.046+0000 7f619be00640 1 -- 172.21.15.96:0/1409047179 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f619400f920 con 0x7f6188006130 2024-05-28T06:21:36.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.046+0000 7f619be00640 1 -- 172.21.15.96:0/1409047179 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f619400fac0 con 0x7f6188006130 2024-05-28T06:21:36.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.046+0000 7f61ab000640 1 -- 172.21.15.96:0/1409047179 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61a4151960 con 0x7f6188006130 2024-05-28T06:21:36.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.046+0000 7f619be00640 1 -- 172.21.15.96:0/1409047179 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f619400fe70 con 0x7f6188006130 2024-05-28T06:21:36.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.046+0000 7f61ab000640 1 -- 172.21.15.96:0/1409047179 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61a4151f20 con 0x7f6188006130 2024-05-28T06:21:36.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.048+0000 7f61ab000640 1 -- 172.21.15.96:0/1409047179 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61a40689e0 con 0x7f6188006130 2024-05-28T06:21:36.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.048+0000 7f619be00640 1 -- 172.21.15.96:0/1409047179 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 36) ==== 93080+0+0 (secure 0 0 0) 0x7f6194015460 con 0x7f6188006130 2024-05-28T06:21:36.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.049+0000 7f619be00640 1 --2- 172.21.15.96:0/1409047179 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f618006df70 0x7f6180070430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:36.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.050+0000 7f619be00640 1 -- 172.21.15.96:0/1409047179 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 5765+0+0 (secure 0 0 0) 0x7f619408b310 con 0x7f6188006130 2024-05-28T06:21:36.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.050+0000 7f61a3400640 1 --2- 172.21.15.96:0/1409047179 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f618006df70 0x7f6180070430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:36.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.051+0000 7f61a3400640 1 --2- 172.21.15.96:0/1409047179 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f618006df70 0x7f6180070430 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f618c032000 tx=0x7f618c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:21:36.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.054+0000 7f619be00640 1 -- 172.21.15.96:0/1409047179 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6194019030 con 0x7f6188006130 2024-05-28T06:21:36.331 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.335+0000 7f61ab000640 1 -- 172.21.15.96:0/1409047179 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f61a406f170 con 0x7f6188006130 2024-05-28T06:21:36.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.336+0000 7f619be00640 1 -- 172.21.15.96:0/1409047179 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+2110 (secure 0 0 0) 0x7f619405b1a0 con 0x7f6188006130 2024-05-28T06:21:36.332 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:21:36.332 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:21:36.332 INFO:teuthology.orchestra.run.smithi096.stdout: "fsid": "e4a831b5-e6d1-47a7-96cb-bf85154ee0d2", 2024-05-28T06:21:36.332 INFO:teuthology.orchestra.run.smithi096.stdout: "health": { 2024-05-28T06:21:36.332 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "HEALTH_OK", 2024-05-28T06:21:36.333 INFO:teuthology.orchestra.run.smithi096.stdout: "checks": {}, 2024-05-28T06:21:36.333 INFO:teuthology.orchestra.run.smithi096.stdout: "mutes": [] 2024-05-28T06:21:36.333 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:21:36.333 INFO:teuthology.orchestra.run.smithi096.stdout: "election_epoch": 8, 2024-05-28T06:21:36.333 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum": [ 2024-05-28T06:21:36.333 INFO:teuthology.orchestra.run.smithi096.stdout: 0, 2024-05-28T06:21:36.333 INFO:teuthology.orchestra.run.smithi096.stdout: 1, 2024-05-28T06:21:36.333 INFO:teuthology.orchestra.run.smithi096.stdout: 2 2024-05-28T06:21:36.333 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:21:36.333 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_names": [ 2024-05-28T06:21:36.333 INFO:teuthology.orchestra.run.smithi096.stdout: "a", 2024-05-28T06:21:36.333 INFO:teuthology.orchestra.run.smithi096.stdout: "b", 2024-05-28T06:21:36.333 INFO:teuthology.orchestra.run.smithi096.stdout: "c" 2024-05-28T06:21:36.334 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:21:36.334 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_age": 1069, 2024-05-28T06:21:36.334 INFO:teuthology.orchestra.run.smithi096.stdout: "monmap": { 2024-05-28T06:21:36.334 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 1, 2024-05-28T06:21:36.334 INFO:teuthology.orchestra.run.smithi096.stdout: "min_mon_release_name": "squid", 2024-05-28T06:21:36.334 INFO:teuthology.orchestra.run.smithi096.stdout: "num_mons": 3 2024-05-28T06:21:36.334 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:21:36.334 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap": { 2024-05-28T06:21:36.334 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 103, 2024-05-28T06:21:36.334 INFO:teuthology.orchestra.run.smithi096.stdout: "num_osds": 8, 2024-05-28T06:21:36.334 INFO:teuthology.orchestra.run.smithi096.stdout: "num_up_osds": 8, 2024-05-28T06:21:36.334 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_up_since": 1716876238, 2024-05-28T06:21:36.334 INFO:teuthology.orchestra.run.smithi096.stdout: "num_in_osds": 8, 2024-05-28T06:21:36.334 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_in_since": 1716876229, 2024-05-28T06:21:36.335 INFO:teuthology.orchestra.run.smithi096.stdout: "num_remapped_pgs": 0 2024-05-28T06:21:36.335 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:21:36.335 INFO:teuthology.orchestra.run.smithi096.stdout: "pgmap": { 2024-05-28T06:21:36.335 INFO:teuthology.orchestra.run.smithi096.stdout: "pgs_by_state": [ 2024-05-28T06:21:36.335 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:21:36.335 INFO:teuthology.orchestra.run.smithi096.stdout: "state_name": "active+clean", 2024-05-28T06:21:36.335 INFO:teuthology.orchestra.run.smithi096.stdout: "count": 97 2024-05-28T06:21:36.335 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:21:36.335 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:21:36.335 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pgs": 97, 2024-05-28T06:21:36.335 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pools": 3, 2024-05-28T06:21:36.336 INFO:teuthology.orchestra.run.smithi096.stdout: "num_objects": 797, 2024-05-28T06:21:36.336 INFO:teuthology.orchestra.run.smithi096.stdout: "data_bytes": 3222175368, 2024-05-28T06:21:36.336 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_used": 7344947200, 2024-05-28T06:21:36.336 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_avail": 765749166080, 2024-05-28T06:21:36.336 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_total": 773094113280, 2024-05-28T06:21:36.336 INFO:teuthology.orchestra.run.smithi096.stdout: "read_bytes_sec": 255, 2024-05-28T06:21:36.336 INFO:teuthology.orchestra.run.smithi096.stdout: "write_bytes_sec": 3667, 2024-05-28T06:21:36.336 INFO:teuthology.orchestra.run.smithi096.stdout: "read_op_per_sec": 0, 2024-05-28T06:21:36.336 INFO:teuthology.orchestra.run.smithi096.stdout: "write_op_per_sec": 2 2024-05-28T06:21:36.336 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:21:36.336 INFO:teuthology.orchestra.run.smithi096.stdout: "fsmap": { 2024-05-28T06:21:36.336 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 63, 2024-05-28T06:21:36.336 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 7, 2024-05-28T06:21:36.337 INFO:teuthology.orchestra.run.smithi096.stdout: "up": 1, 2024-05-28T06:21:36.337 INFO:teuthology.orchestra.run.smithi096.stdout: "in": 1, 2024-05-28T06:21:36.337 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 1, 2024-05-28T06:21:36.337 INFO:teuthology.orchestra.run.smithi096.stdout: "by_rank": [ 2024-05-28T06:21:36.337 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:21:36.337 INFO:teuthology.orchestra.run.smithi096.stdout: "filesystem_id": 7, 2024-05-28T06:21:36.337 INFO:teuthology.orchestra.run.smithi096.stdout: "rank": 0, 2024-05-28T06:21:36.337 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "b", 2024-05-28T06:21:36.337 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "up:active", 2024-05-28T06:21:36.337 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 8739 2024-05-28T06:21:36.337 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:21:36.337 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:21:36.337 INFO:teuthology.orchestra.run.smithi096.stdout: "up:standby": 3 2024-05-28T06:21:36.337 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:21:36.338 INFO:teuthology.orchestra.run.smithi096.stdout: "mgrmap": { 2024-05-28T06:21:36.338 INFO:teuthology.orchestra.run.smithi096.stdout: "available": true, 2024-05-28T06:21:36.338 INFO:teuthology.orchestra.run.smithi096.stdout: "num_standbys": 1, 2024-05-28T06:21:36.338 INFO:teuthology.orchestra.run.smithi096.stdout: "modules": [ 2024-05-28T06:21:36.338 INFO:teuthology.orchestra.run.smithi096.stdout: "iostat", 2024-05-28T06:21:36.338 INFO:teuthology.orchestra.run.smithi096.stdout: "nfs", 2024-05-28T06:21:36.338 INFO:teuthology.orchestra.run.smithi096.stdout: "restful" 2024-05-28T06:21:36.338 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:21:36.338 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:21:36.338 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:21:36.338 INFO:teuthology.orchestra.run.smithi096.stdout: "servicemap": { 2024-05-28T06:21:36.338 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 20, 2024-05-28T06:21:36.338 INFO:teuthology.orchestra.run.smithi096.stdout: "modified": "2024-05-28T06:21:25.943324+0000", 2024-05-28T06:21:36.338 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:21:36.339 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:21:36.339 INFO:teuthology.orchestra.run.smithi096.stdout: "progress_events": { 2024-05-28T06:21:36.339 INFO:teuthology.orchestra.run.smithi096.stdout: "885b6d9d-736d-45a8-ac79-be8f5b8ac230": { 2024-05-28T06:21:36.339 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "All cloning finished, waiting 16.0 seconds before exiting (15s)\n [============================] ", 2024-05-28T06:21:36.339 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": 1, 2024-05-28T06:21:36.339 INFO:teuthology.orchestra.run.smithi096.stdout: "add_to_ceph_s": true 2024-05-28T06:21:36.339 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:21:36.339 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:21:36.339 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:21:36.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.339+0000 7f61ab000640 1 -- 172.21.15.96:0/1409047179 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f618006df70 msgr2=0x7f6180070430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:36.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.339+0000 7f61ab000640 1 --2- 172.21.15.96:0/1409047179 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f618006df70 0x7f6180070430 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f618c032000 tx=0x7f618c02e000 comp rx=0 tx=0).stop 2024-05-28T06:21:36.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.339+0000 7f61ab000640 1 -- 172.21.15.96:0/1409047179 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6188006130 msgr2=0x7f61a414cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:36.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.339+0000 7f61ab000640 1 --2- 172.21.15.96:0/1409047179 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6188006130 0x7f61a414cca0 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7f6194011630 tx=0x7f6194011af0 comp rx=0 tx=0).stop 2024-05-28T06:21:36.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.339+0000 7f61ab000640 1 -- 172.21.15.96:0/1409047179 shutdown_connections 2024-05-28T06:21:36.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.339+0000 7f61ab000640 1 --2- 172.21.15.96:0/1409047179 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f618006df70 0x7f6180070430 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:36.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.339+0000 7f61ab000640 1 --2- 172.21.15.96:0/1409047179 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f61a415dc90 0x7f61a415e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:36.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.339+0000 7f61ab000640 1 --2- 172.21.15.96:0/1409047179 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61a4152910 0x7f61a4142e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:36.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.339+0000 7f61ab000640 1 --2- 172.21.15.96:0/1409047179 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6188006130 0x7f61a414cca0 unknown :-1 s=CLOSED pgs=1683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:36.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.339+0000 7f61ab000640 1 -- 172.21.15.96:0/1409047179 >> 172.21.15.96:0/1409047179 conn(0x7f61a4067ce0 msgr2=0x7f61a40b3d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:21:36.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.339+0000 7f61ab000640 1 -- 172.21.15.96:0/1409047179 shutdown_connections 2024-05-28T06:21:36.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:36.339+0000 7f61ab000640 1 -- 172.21.15.96:0/1409047179 wait complete. 2024-05-28T06:21:41.354 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-05-28T06:21:41.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:41.572+0000 7f291fe00640 1 Processor -- start 2024-05-28T06:21:41.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:41.573+0000 7f291fe00640 1 -- start start 2024-05-28T06:21:41.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:41.574+0000 7f291fe00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f292015c6b0 0x7f29200b3db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:41.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:41.574+0000 7f291fe00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f29200d0460 0x7f29200d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:41.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:41.574+0000 7f291fe00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29200cd1a0 con 0x7f29200ccc30 2024-05-28T06:21:41.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:41.574+0000 7f291fe00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f292006e3a0 con 0x7f29200d0460 2024-05-28T06:21:41.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:41.574+0000 7f291fe00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f29200d0df0 con 0x7f292015c6b0 2024-05-28T06:21:41.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:41.575+0000 7f291f400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f29200d0460 0x7f29200d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:41.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:41.575+0000 7f291e000640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f29200ccc30 0x7f292006dc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:43070/0 (socket says 172.21.15.96:43070) 2024-05-28T06:21:41.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:41.575+0000 7f291e000640 1 -- 172.21.15.96:0/3078193116 learned_addr learned my addr 172.21.15.96:0/3078193116 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:21:41.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:41.575+0000 7f291ea00640 1 --2- 172.21.15.96:0/3078193116 >> v2:172.21.15.123:3300/0 conn(0x7f292015c6b0 0x7f29200b3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:41.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:41.575+0000 7f291f400640 1 -- 172.21.15.96:0/3078193116 >> v2:172.21.15.123:3300/0 conn(0x7f292015c6b0 msgr2=0x7f29200b3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:21:41.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:41.575+0000 7f291f400640 1 --2- 172.21.15.96:0/3078193116 >> v2:172.21.15.123:3300/0 conn(0x7f292015c6b0 0x7f29200b3db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:41.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:41.575+0000 7f291f400640 1 -- 172.21.15.96:0/3078193116 >> v1:172.21.15.96:6789/0 conn(0x7f29200ccc30 legacy=0x7f292006dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:41.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:41.575+0000 7f291f400640 1 -- 172.21.15.96:0/3078193116 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29200cd480 con 0x7f29200d0460 2024-05-28T06:21:41.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:41.575+0000 7f291d600640 1 -- 172.21.15.96:0/3078193116 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3187442881 0 0) 0x7f29200cd1a0 con 0x7f29200ccc30 2024-05-28T06:21:41.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:41.576+0000 7f291f400640 1 --2- 172.21.15.96:0/3078193116 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f29200d0460 0x7f29200d08b0 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7f2918009d90 tx=0x7f291802a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51b6eca639faf60c server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:21:41.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:41.577+0000 7f291d600640 1 -- 172.21.15.96:0/3078193116 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2918007740 con 0x7f29200d0460 2024-05-28T06:21:41.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:41.577+0000 7f291d600640 1 -- 172.21.15.96:0/3078193116 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29180078e0 con 0x7f29200d0460 2024-05-28T06:21:41.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:41.577+0000 7f291d600640 1 -- 172.21.15.96:0/3078193116 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2918007cb0 con 0x7f29200d0460 2024-05-28T06:21:41.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:41.577+0000 7f291fe00640 1 -- 172.21.15.96:0/3078193116 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f29200d0460 msgr2=0x7f29200d08b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:41.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:41.577+0000 7f291fe00640 1 --2- 172.21.15.96:0/3078193116 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f29200d0460 0x7f29200d08b0 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7f2918009d90 tx=0x7f291802a6a0 comp rx=0 tx=0).stop 2024-05-28T06:21:41.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:41.577+0000 7f291fe00640 1 -- 172.21.15.96:0/3078193116 shutdown_connections 2024-05-28T06:21:41.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:41.577+0000 7f291fe00640 1 --2- 172.21.15.96:0/3078193116 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f29200d0460 0x7f29200d08b0 unknown :-1 s=CLOSED pgs=1685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:41.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:41.577+0000 7f291fe00640 1 --2- 172.21.15.96:0/3078193116 >> v2:172.21.15.123:3300/0 conn(0x7f292015c6b0 0x7f29200b3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:41.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:41.577+0000 7f291fe00640 1 -- 172.21.15.96:0/3078193116 >> 172.21.15.96:0/3078193116 conn(0x7f292006bae0 msgr2=0x7f292006bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:21:41.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:41.577+0000 7f291fe00640 1 -- 172.21.15.96:0/3078193116 shutdown_connections 2024-05-28T06:21:41.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:41.577+0000 7f291fe00640 1 -- 172.21.15.96:0/3078193116 wait complete. 2024-05-28T06:21:41.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:41.579+0000 7f291fe00640 1 Processor -- start 2024-05-28T06:21:41.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:41.579+0000 7f291fe00640 1 -- start start 2024-05-28T06:21:41.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:41.580+0000 7f291fe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f29200ccc30 0x7f29200b2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:41.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:41.580+0000 7f291fe00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f29200d0460 0x7f29200c9300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:41.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:41.580+0000 7f291fe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f292015c6b0 0x7f2920150420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:41.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:41.580+0000 7f291fe00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f29200cba10 con 0x7f29200ccc30 2024-05-28T06:21:41.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:41.580+0000 7f291fe00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f29200cda40 con 0x7f292015c6b0 2024-05-28T06:21:41.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:41.580+0000 7f291fe00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f29200b42f0 con 0x7f29200d0460 2024-05-28T06:21:41.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:41.580+0000 7f291ea00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f29200ccc30 0x7f29200b2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:41.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:41.580+0000 7f291ea00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f29200ccc30 0x7f29200b2110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:33828/0 (socket says 172.21.15.96:33828) 2024-05-28T06:21:41.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:41.580+0000 7f291ea00640 1 -- 172.21.15.96:0/2942780708 learned_addr learned my addr 172.21.15.96:0/2942780708 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:21:41.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:41.581+0000 7f291f400640 1 --2- 172.21.15.96:0/2942780708 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f292015c6b0 0x7f2920150420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:41.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:41.581+0000 7f291ea00640 1 -- 172.21.15.96:0/2942780708 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f29200d0460 msgr2=0x7f29200c9300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:41.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:41.581+0000 7f291ea00640 1 --2- 172.21.15.96:0/2942780708 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f29200d0460 0x7f29200c9300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:41.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:41.581+0000 7f291ea00640 1 -- 172.21.15.96:0/2942780708 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f292015c6b0 msgr2=0x7f2920150420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:41.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:41.581+0000 7f291ea00640 1 --2- 172.21.15.96:0/2942780708 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f292015c6b0 0x7f2920150420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:41.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:41.581+0000 7f291ea00640 1 -- 172.21.15.96:0/2942780708 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29200c64c0 con 0x7f29200ccc30 2024-05-28T06:21:41.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:41.581+0000 7f291ea00640 1 --2- 172.21.15.96:0/2942780708 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f29200ccc30 0x7f29200b2110 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7f291000d5f0 tx=0x7f291000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:21:41.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:41.581+0000 7f2916a00640 1 -- 172.21.15.96:0/2942780708 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29100125f0 con 0x7f29200ccc30 2024-05-28T06:21:41.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:41.581+0000 7f2916a00640 1 -- 172.21.15.96:0/2942780708 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2910019070 con 0x7f29200ccc30 2024-05-28T06:21:41.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:41.581+0000 7f2916a00640 1 -- 172.21.15.96:0/2942780708 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29100129d0 con 0x7f29200ccc30 2024-05-28T06:21:41.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:41.581+0000 7f291fe00640 1 -- 172.21.15.96:0/2942780708 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2920073f40 con 0x7f29200ccc30 2024-05-28T06:21:41.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:41.582+0000 7f291fe00640 1 -- 172.21.15.96:0/2942780708 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29200b1950 con 0x7f29200ccc30 2024-05-28T06:21:41.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:41.583+0000 7f291fe00640 1 -- 172.21.15.96:0/2942780708 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29200689e0 con 0x7f29200ccc30 2024-05-28T06:21:41.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:41.584+0000 7f2916a00640 1 -- 172.21.15.96:0/2942780708 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 36) ==== 93080+0+0 (secure 0 0 0) 0x7f2910012b70 con 0x7f29200ccc30 2024-05-28T06:21:41.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:41.585+0000 7f2916a00640 1 --2- 172.21.15.96:0/2942780708 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f28fc06df70 0x7f28fc070430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:41.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:41.585+0000 7f2916a00640 1 -- 172.21.15.96:0/2942780708 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 5765+0+0 (secure 0 0 0) 0x7f291008ad40 con 0x7f29200ccc30 2024-05-28T06:21:41.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:41.585+0000 7f291e000640 1 --2- 172.21.15.96:0/2942780708 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f28fc06df70 0x7f28fc070430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:41.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:41.587+0000 7f291e000640 1 --2- 172.21.15.96:0/2942780708 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f28fc06df70 0x7f28fc070430 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f2908007fd0 tx=0x7f2908007920 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:21:41.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:41.592+0000 7f2916a00640 1 -- 172.21.15.96:0/2942780708 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f291005abd0 con 0x7f29200ccc30 2024-05-28T06:21:41.856 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:41.860+0000 7f291fe00640 1 -- 172.21.15.96:0/2942780708 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f292006f740 con 0x7f29200ccc30 2024-05-28T06:21:41.857 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:41.861+0000 7f2916a00640 1 -- 172.21.15.96:0/2942780708 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+2108 (secure 0 0 0) 0x7f291005a4d0 con 0x7f29200ccc30 2024-05-28T06:21:41.857 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:21:41.857 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:21:41.857 INFO:teuthology.orchestra.run.smithi096.stdout: "fsid": "e4a831b5-e6d1-47a7-96cb-bf85154ee0d2", 2024-05-28T06:21:41.857 INFO:teuthology.orchestra.run.smithi096.stdout: "health": { 2024-05-28T06:21:41.858 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "HEALTH_OK", 2024-05-28T06:21:41.858 INFO:teuthology.orchestra.run.smithi096.stdout: "checks": {}, 2024-05-28T06:21:41.858 INFO:teuthology.orchestra.run.smithi096.stdout: "mutes": [] 2024-05-28T06:21:41.858 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:21:41.858 INFO:teuthology.orchestra.run.smithi096.stdout: "election_epoch": 8, 2024-05-28T06:21:41.858 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum": [ 2024-05-28T06:21:41.858 INFO:teuthology.orchestra.run.smithi096.stdout: 0, 2024-05-28T06:21:41.858 INFO:teuthology.orchestra.run.smithi096.stdout: 1, 2024-05-28T06:21:41.858 INFO:teuthology.orchestra.run.smithi096.stdout: 2 2024-05-28T06:21:41.858 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:21:41.858 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_names": [ 2024-05-28T06:21:41.858 INFO:teuthology.orchestra.run.smithi096.stdout: "a", 2024-05-28T06:21:41.858 INFO:teuthology.orchestra.run.smithi096.stdout: "b", 2024-05-28T06:21:41.859 INFO:teuthology.orchestra.run.smithi096.stdout: "c" 2024-05-28T06:21:41.859 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:21:41.859 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_age": 1075, 2024-05-28T06:21:41.859 INFO:teuthology.orchestra.run.smithi096.stdout: "monmap": { 2024-05-28T06:21:41.859 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 1, 2024-05-28T06:21:41.859 INFO:teuthology.orchestra.run.smithi096.stdout: "min_mon_release_name": "squid", 2024-05-28T06:21:41.859 INFO:teuthology.orchestra.run.smithi096.stdout: "num_mons": 3 2024-05-28T06:21:41.859 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:21:41.859 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap": { 2024-05-28T06:21:41.859 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 103, 2024-05-28T06:21:41.859 INFO:teuthology.orchestra.run.smithi096.stdout: "num_osds": 8, 2024-05-28T06:21:41.859 INFO:teuthology.orchestra.run.smithi096.stdout: "num_up_osds": 8, 2024-05-28T06:21:41.859 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_up_since": 1716876238, 2024-05-28T06:21:41.860 INFO:teuthology.orchestra.run.smithi096.stdout: "num_in_osds": 8, 2024-05-28T06:21:41.860 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_in_since": 1716876229, 2024-05-28T06:21:41.860 INFO:teuthology.orchestra.run.smithi096.stdout: "num_remapped_pgs": 0 2024-05-28T06:21:41.860 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:21:41.860 INFO:teuthology.orchestra.run.smithi096.stdout: "pgmap": { 2024-05-28T06:21:41.860 INFO:teuthology.orchestra.run.smithi096.stdout: "pgs_by_state": [ 2024-05-28T06:21:41.860 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:21:41.860 INFO:teuthology.orchestra.run.smithi096.stdout: "state_name": "active+clean", 2024-05-28T06:21:41.860 INFO:teuthology.orchestra.run.smithi096.stdout: "count": 97 2024-05-28T06:21:41.860 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:21:41.860 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:21:41.861 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pgs": 97, 2024-05-28T06:21:41.861 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pools": 3, 2024-05-28T06:21:41.861 INFO:teuthology.orchestra.run.smithi096.stdout: "num_objects": 797, 2024-05-28T06:21:41.861 INFO:teuthology.orchestra.run.smithi096.stdout: "data_bytes": 3222176983, 2024-05-28T06:21:41.861 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_used": 7344947200, 2024-05-28T06:21:41.861 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_avail": 765749166080, 2024-05-28T06:21:41.861 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_total": 773094113280, 2024-05-28T06:21:41.862 INFO:teuthology.orchestra.run.smithi096.stdout: "read_bytes_sec": 85, 2024-05-28T06:21:41.862 INFO:teuthology.orchestra.run.smithi096.stdout: "write_bytes_sec": 767, 2024-05-28T06:21:41.862 INFO:teuthology.orchestra.run.smithi096.stdout: "read_op_per_sec": 0, 2024-05-28T06:21:41.862 INFO:teuthology.orchestra.run.smithi096.stdout: "write_op_per_sec": 0 2024-05-28T06:21:41.862 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:21:41.862 INFO:teuthology.orchestra.run.smithi096.stdout: "fsmap": { 2024-05-28T06:21:41.862 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 63, 2024-05-28T06:21:41.862 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 7, 2024-05-28T06:21:41.862 INFO:teuthology.orchestra.run.smithi096.stdout: "up": 1, 2024-05-28T06:21:41.862 INFO:teuthology.orchestra.run.smithi096.stdout: "in": 1, 2024-05-28T06:21:41.862 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 1, 2024-05-28T06:21:41.862 INFO:teuthology.orchestra.run.smithi096.stdout: "by_rank": [ 2024-05-28T06:21:41.863 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:21:41.863 INFO:teuthology.orchestra.run.smithi096.stdout: "filesystem_id": 7, 2024-05-28T06:21:41.863 INFO:teuthology.orchestra.run.smithi096.stdout: "rank": 0, 2024-05-28T06:21:41.863 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "b", 2024-05-28T06:21:41.863 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "up:active", 2024-05-28T06:21:41.863 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 8739 2024-05-28T06:21:41.863 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:21:41.863 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:21:41.863 INFO:teuthology.orchestra.run.smithi096.stdout: "up:standby": 3 2024-05-28T06:21:41.863 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:21:41.863 INFO:teuthology.orchestra.run.smithi096.stdout: "mgrmap": { 2024-05-28T06:21:41.863 INFO:teuthology.orchestra.run.smithi096.stdout: "available": true, 2024-05-28T06:21:41.863 INFO:teuthology.orchestra.run.smithi096.stdout: "num_standbys": 1, 2024-05-28T06:21:41.864 INFO:teuthology.orchestra.run.smithi096.stdout: "modules": [ 2024-05-28T06:21:41.864 INFO:teuthology.orchestra.run.smithi096.stdout: "iostat", 2024-05-28T06:21:41.864 INFO:teuthology.orchestra.run.smithi096.stdout: "nfs", 2024-05-28T06:21:41.864 INFO:teuthology.orchestra.run.smithi096.stdout: "restful" 2024-05-28T06:21:41.864 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:21:41.864 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:21:41.864 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:21:41.864 INFO:teuthology.orchestra.run.smithi096.stdout: "servicemap": { 2024-05-28T06:21:41.864 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 20, 2024-05-28T06:21:41.864 INFO:teuthology.orchestra.run.smithi096.stdout: "modified": "2024-05-28T06:21:25.943324+0000", 2024-05-28T06:21:41.864 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:21:41.864 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:21:41.864 INFO:teuthology.orchestra.run.smithi096.stdout: "progress_events": { 2024-05-28T06:21:41.864 INFO:teuthology.orchestra.run.smithi096.stdout: "885b6d9d-736d-45a8-ac79-be8f5b8ac230": { 2024-05-28T06:21:41.865 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "All cloning finished, waiting 10.0 seconds before exiting (21s)\n [============================] ", 2024-05-28T06:21:41.865 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": 1, 2024-05-28T06:21:41.865 INFO:teuthology.orchestra.run.smithi096.stdout: "add_to_ceph_s": true 2024-05-28T06:21:41.865 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:21:41.865 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:21:41.865 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:21:41.866 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:41.864+0000 7f291fe00640 1 -- 172.21.15.96:0/2942780708 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f28fc06df70 msgr2=0x7f28fc070430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:41.866 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:41.864+0000 7f291fe00640 1 --2- 172.21.15.96:0/2942780708 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f28fc06df70 0x7f28fc070430 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f2908007fd0 tx=0x7f2908007920 comp rx=0 tx=0).stop 2024-05-28T06:21:41.866 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:41.864+0000 7f291fe00640 1 -- 172.21.15.96:0/2942780708 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f29200ccc30 msgr2=0x7f29200b2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:41.866 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:41.864+0000 7f291fe00640 1 --2- 172.21.15.96:0/2942780708 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f29200ccc30 0x7f29200b2110 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7f291000d5f0 tx=0x7f291000dab0 comp rx=0 tx=0).stop 2024-05-28T06:21:41.866 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:41.864+0000 7f291fe00640 1 -- 172.21.15.96:0/2942780708 shutdown_connections 2024-05-28T06:21:41.866 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:41.864+0000 7f291fe00640 1 --2- 172.21.15.96:0/2942780708 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f28fc06df70 0x7f28fc070430 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:41.866 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:41.864+0000 7f291fe00640 1 --2- 172.21.15.96:0/2942780708 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f292015c6b0 0x7f2920150420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:41.866 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:41.864+0000 7f291fe00640 1 --2- 172.21.15.96:0/2942780708 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f29200d0460 0x7f29200c9300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:41.867 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:41.864+0000 7f291fe00640 1 --2- 172.21.15.96:0/2942780708 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f29200ccc30 0x7f29200b2110 unknown :-1 s=CLOSED pgs=1686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:41.867 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:41.864+0000 7f291fe00640 1 -- 172.21.15.96:0/2942780708 >> 172.21.15.96:0/2942780708 conn(0x7f292006bae0 msgr2=0x7f292006dc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:21:41.867 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:41.864+0000 7f291fe00640 1 -- 172.21.15.96:0/2942780708 shutdown_connections 2024-05-28T06:21:41.867 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:41.864+0000 7f291fe00640 1 -- 172.21.15.96:0/2942780708 wait complete. 2024-05-28T06:21:46.878 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-05-28T06:21:47.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.092+0000 7fcd47c00640 1 Processor -- start 2024-05-28T06:21:47.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.093+0000 7fcd47c00640 1 -- start start 2024-05-28T06:21:47.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.094+0000 7fcd47c00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fcd4015baa0 0x7fcd400d0970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:47.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.094+0000 7fcd47c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcd4006f390 0x7fcd4006f800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:47.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.094+0000 7fcd47c00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd4006e3d0 con 0x7fcd40155a00 2024-05-28T06:21:47.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.094+0000 7fcd47c00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fcd4006fd40 con 0x7fcd4015baa0 2024-05-28T06:21:47.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.094+0000 7fcd47c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fcd400ccf20 con 0x7fcd4006f390 2024-05-28T06:21:47.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.095+0000 7fcd45200640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fcd40155a00 0x7fcd4006dcc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:43078/0 (socket says 172.21.15.96:43078) 2024-05-28T06:21:47.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.095+0000 7fcd45200640 1 -- 172.21.15.96:0/2895105584 learned_addr learned my addr 172.21.15.96:0/2895105584 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:21:47.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.095+0000 7fcd37e00640 1 --2- 172.21.15.96:0/2895105584 >> v2:172.21.15.123:3300/0 conn(0x7fcd4015baa0 0x7fcd400d0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:47.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.095+0000 7fcd45c00640 1 --2- 172.21.15.96:0/2895105584 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcd4006f390 0x7fcd4006f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:47.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.095+0000 7fcd3fe00640 1 -- 172.21.15.96:0/2895105584 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1367713867 0 0) 0x7fcd4006e3d0 con 0x7fcd40155a00 2024-05-28T06:21:47.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.095+0000 7fcd3fe00640 1 -- 172.21.15.96:0/2895105584 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcd24005730 con 0x7fcd40155a00 2024-05-28T06:21:47.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.095+0000 7fcd3fe00640 1 -- 172.21.15.96:0/2895105584 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2776193217 0 0) 0x7fcd24005730 con 0x7fcd40155a00 2024-05-28T06:21:47.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.095+0000 7fcd3fe00640 1 -- 172.21.15.96:0/2895105584 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcd4006f390 msgr2=0x7fcd4006f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:21:47.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.095+0000 7fcd3fe00640 1 --2- 172.21.15.96:0/2895105584 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcd4006f390 0x7fcd4006f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:47.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.095+0000 7fcd3fe00640 1 -- 172.21.15.96:0/2895105584 >> v2:172.21.15.123:3300/0 conn(0x7fcd4015baa0 msgr2=0x7fcd400d0970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:21:47.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.095+0000 7fcd3fe00640 1 --2- 172.21.15.96:0/2895105584 >> v2:172.21.15.123:3300/0 conn(0x7fcd4015baa0 0x7fcd400d0970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:47.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.095+0000 7fcd3fe00640 1 -- 172.21.15.96:0/2895105584 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd4015c0b0 con 0x7fcd40155a00 2024-05-28T06:21:47.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.095+0000 7fcd3fe00640 1 -- 172.21.15.96:0/2895105584 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7fcd30001670 con 0x7fcd40155a00 2024-05-28T06:21:47.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.096+0000 7fcd3fe00640 1 -- 172.21.15.96:0/2895105584 >> v1:172.21.15.96:6789/0 conn(0x7fcd40155a00 legacy=0x7fcd4006dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:47.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.096+0000 7fcd3fe00640 1 --2- 172.21.15.96:0/2895105584 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcd24005bc0 0x7fcd24005fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:47.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.096+0000 7fcd3fe00640 1 --2- 172.21.15.96:0/2895105584 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcd4006f390 0x7fcd24006a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:47.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.096+0000 7fcd3fe00640 1 --2- 172.21.15.96:0/2895105584 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcd24006f50 0x7fcd240073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:47.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.096+0000 7fcd3fe00640 1 -- 172.21.15.96:0/2895105584 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fcd4006fd40 con 0x7fcd24006f50 2024-05-28T06:21:47.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.096+0000 7fcd3fe00640 1 -- 172.21.15.96:0/2895105584 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fcd2400dbd0 con 0x7fcd24005bc0 2024-05-28T06:21:47.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.096+0000 7fcd3fe00640 1 -- 172.21.15.96:0/2895105584 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fcd2400dd80 con 0x7fcd4006f390 2024-05-28T06:21:47.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.096+0000 7fcd45c00640 1 --2- 172.21.15.96:0/2895105584 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcd24006f50 0x7fcd240073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:47.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.096+0000 7fcd45200640 1 --2- 172.21.15.96:0/2895105584 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcd24005bc0 0x7fcd24005fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:47.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.096+0000 7fcd37e00640 1 --2- 172.21.15.96:0/2895105584 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcd4006f390 0x7fcd24006a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:47.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.096+0000 7fcd45c00640 1 -- 172.21.15.96:0/2895105584 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcd4006f390 msgr2=0x7fcd24006a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:21:47.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.096+0000 7fcd45c00640 1 --2- 172.21.15.96:0/2895105584 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcd4006f390 0x7fcd24006a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:47.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.096+0000 7fcd45c00640 1 -- 172.21.15.96:0/2895105584 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcd24005bc0 msgr2=0x7fcd24005fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:21:47.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.096+0000 7fcd45c00640 1 --2- 172.21.15.96:0/2895105584 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcd24005bc0 0x7fcd24005fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:47.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.096+0000 7fcd45c00640 1 -- 172.21.15.96:0/2895105584 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd24005730 con 0x7fcd24006f50 2024-05-28T06:21:47.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.097+0000 7fcd45c00640 1 --2- 172.21.15.96:0/2895105584 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcd24006f50 0x7fcd240073d0 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7fcd38007360 tx=0x7fcd24007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=290e9fe0da0ac5ce server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:21:47.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.098+0000 7fcd3fe00640 1 -- 172.21.15.96:0/2895105584 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd38033070 con 0x7fcd24006f50 2024-05-28T06:21:47.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.098+0000 7fcd3fe00640 1 -- 172.21.15.96:0/2895105584 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd3802f670 con 0x7fcd24006f50 2024-05-28T06:21:47.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.098+0000 7fcd3fe00640 1 -- 172.21.15.96:0/2895105584 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd38009be0 con 0x7fcd24006f50 2024-05-28T06:21:47.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.098+0000 7fcd47c00640 1 -- 172.21.15.96:0/2895105584 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcd24006f50 msgr2=0x7fcd240073d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:47.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.098+0000 7fcd47c00640 1 --2- 172.21.15.96:0/2895105584 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcd24006f50 0x7fcd240073d0 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7fcd38007360 tx=0x7fcd24007a40 comp rx=0 tx=0).stop 2024-05-28T06:21:47.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.098+0000 7fcd45c00640 1 -- 172.21.15.96:0/2895105584 reap_dead start 2024-05-28T06:21:47.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.098+0000 7fcd47c00640 1 -- 172.21.15.96:0/2895105584 shutdown_connections 2024-05-28T06:21:47.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.098+0000 7fcd47c00640 1 -- 172.21.15.96:0/2895105584 >> 172.21.15.96:0/2895105584 conn(0x7fcd40067ce0 msgr2=0x7fcd4006b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:21:47.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.098+0000 7fcd47c00640 1 -- 172.21.15.96:0/2895105584 shutdown_connections 2024-05-28T06:21:47.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.098+0000 7fcd47c00640 1 -- 172.21.15.96:0/2895105584 wait complete. 2024-05-28T06:21:47.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.100+0000 7fcd47c00640 1 Processor -- start 2024-05-28T06:21:47.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.100+0000 7fcd47c00640 1 -- start start 2024-05-28T06:21:47.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.101+0000 7fcd47c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcd4013f630 0x7fcd4013fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:47.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.101+0000 7fcd47c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcd4014efb0 0x7fcd4014f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:47.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.101+0000 7fcd47c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcd40072100 0x7fcd40072580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:47.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.101+0000 7fcd47c00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fcd4015c670 con 0x7fcd4013f630 2024-05-28T06:21:47.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.101+0000 7fcd47c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fcd4015c950 con 0x7fcd4014efb0 2024-05-28T06:21:47.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.101+0000 7fcd47c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fcd4015f090 con 0x7fcd40072100 2024-05-28T06:21:47.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.102+0000 7fcd45200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcd4013f630 0x7fcd4013fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:47.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.102+0000 7fcd45200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcd4013f630 0x7fcd4013fa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:33848/0 (socket says 172.21.15.96:33848) 2024-05-28T06:21:47.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.102+0000 7fcd45200640 1 -- 172.21.15.96:0/4163305816 learned_addr learned my addr 172.21.15.96:0/4163305816 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:21:47.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.102+0000 7fcd37e00640 1 --2- 172.21.15.96:0/4163305816 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcd4014efb0 0x7fcd4014f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:47.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.102+0000 7fcd45c00640 1 --2- 172.21.15.96:0/4163305816 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcd40072100 0x7fcd40072580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:47.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.102+0000 7fcd45200640 1 -- 172.21.15.96:0/4163305816 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcd40072100 msgr2=0x7fcd40072580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:47.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.102+0000 7fcd45200640 1 --2- 172.21.15.96:0/4163305816 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcd40072100 0x7fcd40072580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:47.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.102+0000 7fcd45200640 1 -- 172.21.15.96:0/4163305816 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcd4014efb0 msgr2=0x7fcd4014f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:47.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.102+0000 7fcd45200640 1 --2- 172.21.15.96:0/4163305816 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcd4014efb0 0x7fcd4014f410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:47.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.102+0000 7fcd45200640 1 -- 172.21.15.96:0/4163305816 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd4014fb00 con 0x7fcd4013f630 2024-05-28T06:21:47.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.102+0000 7fcd45200640 1 --2- 172.21.15.96:0/4163305816 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcd4013f630 0x7fcd4013fa10 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7fcd30004840 tx=0x7fcd3000ef80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:21:47.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.103+0000 7fcd3d600640 1 -- 172.21.15.96:0/4163305816 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd3000c8d0 con 0x7fcd4013f630 2024-05-28T06:21:47.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.103+0000 7fcd3d600640 1 -- 172.21.15.96:0/4163305816 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd30018070 con 0x7fcd4013f630 2024-05-28T06:21:47.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.103+0000 7fcd3d600640 1 -- 172.21.15.96:0/4163305816 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd3000ccb0 con 0x7fcd4013f630 2024-05-28T06:21:47.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.103+0000 7fcd47c00640 1 -- 172.21.15.96:0/4163305816 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd4014fde0 con 0x7fcd4013f630 2024-05-28T06:21:47.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.103+0000 7fcd47c00640 1 -- 172.21.15.96:0/4163305816 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd401503a0 con 0x7fcd4013f630 2024-05-28T06:21:47.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.104+0000 7fcd47c00640 1 -- 172.21.15.96:0/4163305816 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd400689e0 con 0x7fcd4013f630 2024-05-28T06:21:47.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.105+0000 7fcd3d600640 1 -- 172.21.15.96:0/4163305816 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 36) ==== 93080+0+0 (secure 0 0 0) 0x7fcd3001d460 con 0x7fcd4013f630 2024-05-28T06:21:47.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.106+0000 7fcd3d600640 1 --2- 172.21.15.96:0/4163305816 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fcd1c06df70 0x7fcd1c070430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:47.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.106+0000 7fcd3d600640 1 -- 172.21.15.96:0/4163305816 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 5765+0+0 (secure 0 0 0) 0x7fcd3008a8e0 con 0x7fcd4013f630 2024-05-28T06:21:47.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.107+0000 7fcd37e00640 1 --2- 172.21.15.96:0/4163305816 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fcd1c06df70 0x7fcd1c070430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:47.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.108+0000 7fcd37e00640 1 --2- 172.21.15.96:0/4163305816 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fcd1c06df70 0x7fcd1c070430 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fcd4014cf60 tx=0x7fcd28008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:21:47.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.111+0000 7fcd3d600640 1 -- 172.21.15.96:0/4163305816 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fcd3005a770 con 0x7fcd4013f630 2024-05-28T06:21:47.383 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.387+0000 7fcd47c00640 1 -- 172.21.15.96:0/4163305816 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fcd4006f170 con 0x7fcd4013f630 2024-05-28T06:21:47.384 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.388+0000 7fcd3d600640 1 -- 172.21.15.96:0/4163305816 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+2077 (secure 0 0 0) 0x7fcd3005a070 con 0x7fcd4013f630 2024-05-28T06:21:47.384 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:21:47.384 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:21:47.385 INFO:teuthology.orchestra.run.smithi096.stdout: "fsid": "e4a831b5-e6d1-47a7-96cb-bf85154ee0d2", 2024-05-28T06:21:47.385 INFO:teuthology.orchestra.run.smithi096.stdout: "health": { 2024-05-28T06:21:47.385 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "HEALTH_OK", 2024-05-28T06:21:47.385 INFO:teuthology.orchestra.run.smithi096.stdout: "checks": {}, 2024-05-28T06:21:47.385 INFO:teuthology.orchestra.run.smithi096.stdout: "mutes": [] 2024-05-28T06:21:47.385 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:21:47.385 INFO:teuthology.orchestra.run.smithi096.stdout: "election_epoch": 8, 2024-05-28T06:21:47.385 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum": [ 2024-05-28T06:21:47.385 INFO:teuthology.orchestra.run.smithi096.stdout: 0, 2024-05-28T06:21:47.385 INFO:teuthology.orchestra.run.smithi096.stdout: 1, 2024-05-28T06:21:47.385 INFO:teuthology.orchestra.run.smithi096.stdout: 2 2024-05-28T06:21:47.385 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:21:47.385 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_names": [ 2024-05-28T06:21:47.386 INFO:teuthology.orchestra.run.smithi096.stdout: "a", 2024-05-28T06:21:47.386 INFO:teuthology.orchestra.run.smithi096.stdout: "b", 2024-05-28T06:21:47.386 INFO:teuthology.orchestra.run.smithi096.stdout: "c" 2024-05-28T06:21:47.386 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:21:47.386 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_age": 1080, 2024-05-28T06:21:47.386 INFO:teuthology.orchestra.run.smithi096.stdout: "monmap": { 2024-05-28T06:21:47.386 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 1, 2024-05-28T06:21:47.386 INFO:teuthology.orchestra.run.smithi096.stdout: "min_mon_release_name": "squid", 2024-05-28T06:21:47.386 INFO:teuthology.orchestra.run.smithi096.stdout: "num_mons": 3 2024-05-28T06:21:47.386 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:21:47.386 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap": { 2024-05-28T06:21:47.386 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 103, 2024-05-28T06:21:47.386 INFO:teuthology.orchestra.run.smithi096.stdout: "num_osds": 8, 2024-05-28T06:21:47.387 INFO:teuthology.orchestra.run.smithi096.stdout: "num_up_osds": 8, 2024-05-28T06:21:47.387 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_up_since": 1716876238, 2024-05-28T06:21:47.387 INFO:teuthology.orchestra.run.smithi096.stdout: "num_in_osds": 8, 2024-05-28T06:21:47.387 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_in_since": 1716876229, 2024-05-28T06:21:47.387 INFO:teuthology.orchestra.run.smithi096.stdout: "num_remapped_pgs": 0 2024-05-28T06:21:47.387 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:21:47.387 INFO:teuthology.orchestra.run.smithi096.stdout: "pgmap": { 2024-05-28T06:21:47.387 INFO:teuthology.orchestra.run.smithi096.stdout: "pgs_by_state": [ 2024-05-28T06:21:47.387 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:21:47.387 INFO:teuthology.orchestra.run.smithi096.stdout: "state_name": "active+clean", 2024-05-28T06:21:47.387 INFO:teuthology.orchestra.run.smithi096.stdout: "count": 97 2024-05-28T06:21:47.387 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:21:47.387 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:21:47.387 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pgs": 97, 2024-05-28T06:21:47.388 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pools": 3, 2024-05-28T06:21:47.388 INFO:teuthology.orchestra.run.smithi096.stdout: "num_objects": 797, 2024-05-28T06:21:47.388 INFO:teuthology.orchestra.run.smithi096.stdout: "data_bytes": 3222176983, 2024-05-28T06:21:47.388 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_used": 7344947200, 2024-05-28T06:21:47.388 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_avail": 765749166080, 2024-05-28T06:21:47.388 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_total": 773094113280, 2024-05-28T06:21:47.388 INFO:teuthology.orchestra.run.smithi096.stdout: "write_bytes_sec": 170, 2024-05-28T06:21:47.388 INFO:teuthology.orchestra.run.smithi096.stdout: "read_op_per_sec": 0, 2024-05-28T06:21:47.388 INFO:teuthology.orchestra.run.smithi096.stdout: "write_op_per_sec": 0 2024-05-28T06:21:47.388 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:21:47.388 INFO:teuthology.orchestra.run.smithi096.stdout: "fsmap": { 2024-05-28T06:21:47.388 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 63, 2024-05-28T06:21:47.388 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 7, 2024-05-28T06:21:47.388 INFO:teuthology.orchestra.run.smithi096.stdout: "up": 1, 2024-05-28T06:21:47.390 INFO:teuthology.orchestra.run.smithi096.stdout: "in": 1, 2024-05-28T06:21:47.390 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 1, 2024-05-28T06:21:47.390 INFO:teuthology.orchestra.run.smithi096.stdout: "by_rank": [ 2024-05-28T06:21:47.390 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:21:47.390 INFO:teuthology.orchestra.run.smithi096.stdout: "filesystem_id": 7, 2024-05-28T06:21:47.390 INFO:teuthology.orchestra.run.smithi096.stdout: "rank": 0, 2024-05-28T06:21:47.390 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "b", 2024-05-28T06:21:47.391 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "up:active", 2024-05-28T06:21:47.391 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 8739 2024-05-28T06:21:47.391 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:21:47.391 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:21:47.391 INFO:teuthology.orchestra.run.smithi096.stdout: "up:standby": 3 2024-05-28T06:21:47.391 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:21:47.391 INFO:teuthology.orchestra.run.smithi096.stdout: "mgrmap": { 2024-05-28T06:21:47.391 INFO:teuthology.orchestra.run.smithi096.stdout: "available": true, 2024-05-28T06:21:47.391 INFO:teuthology.orchestra.run.smithi096.stdout: "num_standbys": 1, 2024-05-28T06:21:47.391 INFO:teuthology.orchestra.run.smithi096.stdout: "modules": [ 2024-05-28T06:21:47.391 INFO:teuthology.orchestra.run.smithi096.stdout: "iostat", 2024-05-28T06:21:47.391 INFO:teuthology.orchestra.run.smithi096.stdout: "nfs", 2024-05-28T06:21:47.391 INFO:teuthology.orchestra.run.smithi096.stdout: "restful" 2024-05-28T06:21:47.392 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:21:47.392 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:21:47.392 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:21:47.392 INFO:teuthology.orchestra.run.smithi096.stdout: "servicemap": { 2024-05-28T06:21:47.392 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 20, 2024-05-28T06:21:47.392 INFO:teuthology.orchestra.run.smithi096.stdout: "modified": "2024-05-28T06:21:25.943324+0000", 2024-05-28T06:21:47.392 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:21:47.392 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:21:47.392 INFO:teuthology.orchestra.run.smithi096.stdout: "progress_events": { 2024-05-28T06:21:47.392 INFO:teuthology.orchestra.run.smithi096.stdout: "885b6d9d-736d-45a8-ac79-be8f5b8ac230": { 2024-05-28T06:21:47.392 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "All cloning finished, waiting 4.0 seconds before exiting (27s)\n [============================] ", 2024-05-28T06:21:47.392 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": 1, 2024-05-28T06:21:47.392 INFO:teuthology.orchestra.run.smithi096.stdout: "add_to_ceph_s": true 2024-05-28T06:21:47.392 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:21:47.393 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:21:47.393 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:21:47.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.391+0000 7fcd47c00640 1 -- 172.21.15.96:0/4163305816 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fcd1c06df70 msgr2=0x7fcd1c070430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:47.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.391+0000 7fcd47c00640 1 --2- 172.21.15.96:0/4163305816 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fcd1c06df70 0x7fcd1c070430 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fcd4014cf60 tx=0x7fcd28008040 comp rx=0 tx=0).stop 2024-05-28T06:21:47.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.391+0000 7fcd47c00640 1 -- 172.21.15.96:0/4163305816 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcd4013f630 msgr2=0x7fcd4013fa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:47.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.391+0000 7fcd47c00640 1 --2- 172.21.15.96:0/4163305816 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcd4013f630 0x7fcd4013fa10 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7fcd30004840 tx=0x7fcd3000ef80 comp rx=0 tx=0).stop 2024-05-28T06:21:47.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.391+0000 7fcd47c00640 1 -- 172.21.15.96:0/4163305816 shutdown_connections 2024-05-28T06:21:47.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.391+0000 7fcd47c00640 1 --2- 172.21.15.96:0/4163305816 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fcd1c06df70 0x7fcd1c070430 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:47.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.391+0000 7fcd47c00640 1 --2- 172.21.15.96:0/4163305816 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcd40072100 0x7fcd40072580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:47.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.391+0000 7fcd47c00640 1 --2- 172.21.15.96:0/4163305816 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcd4014efb0 0x7fcd4014f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:47.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.391+0000 7fcd47c00640 1 --2- 172.21.15.96:0/4163305816 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcd4013f630 0x7fcd4013fa10 unknown :-1 s=CLOSED pgs=1689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:47.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.391+0000 7fcd47c00640 1 -- 172.21.15.96:0/4163305816 >> 172.21.15.96:0/4163305816 conn(0x7fcd40067ce0 msgr2=0x7fcd4006b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:21:47.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.391+0000 7fcd47c00640 1 -- 172.21.15.96:0/4163305816 shutdown_connections 2024-05-28T06:21:47.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:47.391+0000 7fcd47c00640 1 -- 172.21.15.96:0/4163305816 wait complete. 2024-05-28T06:21:52.405 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-05-28T06:21:52.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:52.614+0000 7f5381600640 1 Processor -- start 2024-05-28T06:21:52.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:52.614+0000 7f5381600640 1 -- start start 2024-05-28T06:21:52.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:52.615+0000 7f5381600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f537c06f1a0 0x7f537c0d02d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:52.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:52.616+0000 7f5381600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f537c0b41c0 0x7f537c071cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:52.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:52.616+0000 7f5381600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f537c06f6f0 con 0x7f537c15ae10 2024-05-28T06:21:52.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:52.616+0000 7f5381600640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f537c06f9d0 con 0x7f537c06f1a0 2024-05-28T06:21:52.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:52.616+0000 7f5381600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f537c06fcb0 con 0x7f537c0b41c0 2024-05-28T06:21:52.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:52.616+0000 7f537b400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f537c0b41c0 0x7f537c071cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:52.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:52.616+0000 7f537b400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f537c0b41c0 0x7f537c071cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.96:52746/0 (socket says 172.21.15.96:52746) 2024-05-28T06:21:52.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:52.616+0000 7f537b400640 1 -- 172.21.15.96:0/3796114899 learned_addr learned my addr 172.21.15.96:0/3796114899 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:21:52.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:52.617+0000 7f537aa00640 1 --2- 172.21.15.96:0/3796114899 >> v2:172.21.15.123:3300/0 conn(0x7f537c06f1a0 0x7f537c0d02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:52.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:52.617+0000 7f537b400640 1 -- 172.21.15.96:0/3796114899 >> v2:172.21.15.123:3300/0 conn(0x7f537c06f1a0 msgr2=0x7f537c0d02d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:21:52.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:52.617+0000 7f537b400640 1 --2- 172.21.15.96:0/3796114899 >> v2:172.21.15.123:3300/0 conn(0x7f537c06f1a0 0x7f537c0d02d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:52.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:52.617+0000 7f537b400640 1 -- 172.21.15.96:0/3796114899 >> v1:172.21.15.123:6789/0 conn(0x7f537c15ae10 legacy=0x7f537c0cd7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:52.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:52.617+0000 7f537b400640 1 -- 172.21.15.96:0/3796114899 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f537c06ff90 con 0x7f537c0b41c0 2024-05-28T06:21:52.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:52.618+0000 7f537b400640 1 --2- 172.21.15.96:0/3796114899 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f537c0b41c0 0x7f537c071cc0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f5374009870 tx=0x7f537402a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d687889d5e1c54c server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:21:52.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:52.619+0000 7f537a000640 1 -- 172.21.15.96:0/3796114899 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5374037070 con 0x7f537c0b41c0 2024-05-28T06:21:52.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:52.619+0000 7f537a000640 1 -- 172.21.15.96:0/3796114899 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5374032440 con 0x7f537c0b41c0 2024-05-28T06:21:52.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:52.619+0000 7f537a000640 1 -- 172.21.15.96:0/3796114899 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5374032810 con 0x7f537c0b41c0 2024-05-28T06:21:52.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:52.620+0000 7f5381600640 1 -- 172.21.15.96:0/3796114899 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f537c0b41c0 msgr2=0x7f537c071cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:52.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:52.620+0000 7f5381600640 1 --2- 172.21.15.96:0/3796114899 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f537c0b41c0 0x7f537c071cc0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f5374009870 tx=0x7f537402a270 comp rx=0 tx=0).stop 2024-05-28T06:21:52.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:52.620+0000 7f5381600640 1 -- 172.21.15.96:0/3796114899 shutdown_connections 2024-05-28T06:21:52.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:52.620+0000 7f5381600640 1 --2- 172.21.15.96:0/3796114899 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f537c0b41c0 0x7f537c071cc0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:52.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:52.620+0000 7f5381600640 1 --2- 172.21.15.96:0/3796114899 >> v2:172.21.15.123:3300/0 conn(0x7f537c06f1a0 0x7f537c0d02d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:52.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:52.620+0000 7f5381600640 1 -- 172.21.15.96:0/3796114899 >> 172.21.15.96:0/3796114899 conn(0x7f537c067ce0 msgr2=0x7f537c06b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:21:52.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:52.620+0000 7f5381600640 1 -- 172.21.15.96:0/3796114899 shutdown_connections 2024-05-28T06:21:52.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:52.620+0000 7f5381600640 1 -- 172.21.15.96:0/3796114899 wait complete. 2024-05-28T06:21:52.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:52.622+0000 7f5381600640 1 Processor -- start 2024-05-28T06:21:52.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:52.622+0000 7f5381600640 1 -- start start 2024-05-28T06:21:52.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:52.623+0000 7f5381600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f537c06f1a0 0x7f537c14c160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:52.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:52.623+0000 7f5381600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f537c0b41c0 0x7f537c15bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:52.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:52.623+0000 7f5381600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f537c15ae10 0x7f537c0b2030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:52.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:52.623+0000 7f5381600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f537c0b3fd0 con 0x7f537c06f1a0 2024-05-28T06:21:52.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:52.623+0000 7f5381600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f537c0b3e50 con 0x7f537c0b41c0 2024-05-28T06:21:52.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:52.623+0000 7f5381600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f537c0d0810 con 0x7f537c15ae10 2024-05-28T06:21:52.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:52.624+0000 7f537aa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f537c06f1a0 0x7f537c14c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:52.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:52.624+0000 7f537aa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f537c06f1a0 0x7f537c14c160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47834/0 (socket says 172.21.15.96:47834) 2024-05-28T06:21:52.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:52.624+0000 7f537aa00640 1 -- 172.21.15.96:0/2177937519 learned_addr learned my addr 172.21.15.96:0/2177937519 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:21:52.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:52.624+0000 7f5372000640 1 --2- 172.21.15.96:0/2177937519 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f537c0b41c0 0x7f537c15bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:52.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:52.624+0000 7f537b400640 1 --2- 172.21.15.96:0/2177937519 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f537c15ae10 0x7f537c0b2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:52.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:52.624+0000 7f537aa00640 1 -- 172.21.15.96:0/2177937519 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f537c15ae10 msgr2=0x7f537c0b2030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:52.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:52.624+0000 7f537aa00640 1 --2- 172.21.15.96:0/2177937519 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f537c15ae10 0x7f537c0b2030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:52.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:52.624+0000 7f537aa00640 1 -- 172.21.15.96:0/2177937519 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f537c0b41c0 msgr2=0x7f537c15bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:52.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:52.624+0000 7f537aa00640 1 --2- 172.21.15.96:0/2177937519 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f537c0b41c0 0x7f537c15bcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:52.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:52.624+0000 7f537aa00640 1 -- 172.21.15.96:0/2177937519 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f537c15b5f0 con 0x7f537c06f1a0 2024-05-28T06:21:52.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:52.624+0000 7f5372000640 1 --2- 172.21.15.96:0/2177937519 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f537c0b41c0 0x7f537c15bcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-28T06:21:52.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:52.625+0000 7f537aa00640 1 --2- 172.21.15.96:0/2177937519 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f537c06f1a0 0x7f537c14c160 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7f536c00efc0 tx=0x7f536c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:21:52.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:52.625+0000 7f5373400640 1 -- 172.21.15.96:0/2177937519 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f536c018020 con 0x7f537c06f1a0 2024-05-28T06:21:52.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:52.625+0000 7f5373400640 1 -- 172.21.15.96:0/2177937519 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f536c014480 con 0x7f537c06f1a0 2024-05-28T06:21:52.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:52.625+0000 7f5373400640 1 -- 172.21.15.96:0/2177937519 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f536c014850 con 0x7f537c06f1a0 2024-05-28T06:21:52.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:52.625+0000 7f5381600640 1 -- 172.21.15.96:0/2177937519 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f537c15b8d0 con 0x7f537c06f1a0 2024-05-28T06:21:52.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:52.625+0000 7f5381600640 1 -- 172.21.15.96:0/2177937519 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f537c150010 con 0x7f537c06f1a0 2024-05-28T06:21:52.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:52.627+0000 7f5381600640 1 -- 172.21.15.96:0/2177937519 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f537c0681e0 con 0x7f537c06f1a0 2024-05-28T06:21:52.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:52.628+0000 7f5373400640 1 -- 172.21.15.96:0/2177937519 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 36) ==== 93080+0+0 (secure 0 0 0) 0x7f536c005d10 con 0x7f537c06f1a0 2024-05-28T06:21:52.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:52.629+0000 7f5373400640 1 --2- 172.21.15.96:0/2177937519 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f535806df70 0x7f5358070430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:52.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:52.629+0000 7f5373400640 1 -- 172.21.15.96:0/2177937519 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 5765+0+0 (secure 0 0 0) 0x7f536c08b380 con 0x7f537c06f1a0 2024-05-28T06:21:52.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:52.630+0000 7f5372000640 1 --2- 172.21.15.96:0/2177937519 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f535806df70 0x7f5358070430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:52.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:52.631+0000 7f5372000640 1 --2- 172.21.15.96:0/2177937519 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f535806df70 0x7f5358070430 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f537c159cf0 tx=0x7f5364009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:21:52.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:52.634+0000 7f5373400640 1 -- 172.21.15.96:0/2177937519 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f537c0681e0 con 0x7f537c06f1a0 2024-05-28T06:21:52.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:52.912+0000 7f5381600640 1 -- 172.21.15.96:0/2177937519 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f537c0cf440 con 0x7f537c06f1a0 2024-05-28T06:21:52.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:52.913+0000 7f5373400640 1 -- 172.21.15.96:0/2177937519 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1730 (secure 0 0 0) 0x7f537c0cf440 con 0x7f537c06f1a0 2024-05-28T06:21:52.909 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:21:52.909 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:21:52.909 INFO:teuthology.orchestra.run.smithi096.stdout: "fsid": "e4a831b5-e6d1-47a7-96cb-bf85154ee0d2", 2024-05-28T06:21:52.909 INFO:teuthology.orchestra.run.smithi096.stdout: "health": { 2024-05-28T06:21:52.909 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "HEALTH_OK", 2024-05-28T06:21:52.909 INFO:teuthology.orchestra.run.smithi096.stdout: "checks": {}, 2024-05-28T06:21:52.909 INFO:teuthology.orchestra.run.smithi096.stdout: "mutes": [] 2024-05-28T06:21:52.909 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:21:52.909 INFO:teuthology.orchestra.run.smithi096.stdout: "election_epoch": 8, 2024-05-28T06:21:52.910 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum": [ 2024-05-28T06:21:52.910 INFO:teuthology.orchestra.run.smithi096.stdout: 0, 2024-05-28T06:21:52.910 INFO:teuthology.orchestra.run.smithi096.stdout: 1, 2024-05-28T06:21:52.910 INFO:teuthology.orchestra.run.smithi096.stdout: 2 2024-05-28T06:21:52.910 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:21:52.910 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_names": [ 2024-05-28T06:21:52.910 INFO:teuthology.orchestra.run.smithi096.stdout: "a", 2024-05-28T06:21:52.910 INFO:teuthology.orchestra.run.smithi096.stdout: "b", 2024-05-28T06:21:52.911 INFO:teuthology.orchestra.run.smithi096.stdout: "c" 2024-05-28T06:21:52.911 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:21:52.911 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_age": 1086, 2024-05-28T06:21:52.911 INFO:teuthology.orchestra.run.smithi096.stdout: "monmap": { 2024-05-28T06:21:52.911 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 1, 2024-05-28T06:21:52.911 INFO:teuthology.orchestra.run.smithi096.stdout: "min_mon_release_name": "squid", 2024-05-28T06:21:52.911 INFO:teuthology.orchestra.run.smithi096.stdout: "num_mons": 3 2024-05-28T06:21:52.911 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:21:52.911 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap": { 2024-05-28T06:21:52.911 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 103, 2024-05-28T06:21:52.911 INFO:teuthology.orchestra.run.smithi096.stdout: "num_osds": 8, 2024-05-28T06:21:52.911 INFO:teuthology.orchestra.run.smithi096.stdout: "num_up_osds": 8, 2024-05-28T06:21:52.911 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_up_since": 1716876238, 2024-05-28T06:21:52.912 INFO:teuthology.orchestra.run.smithi096.stdout: "num_in_osds": 8, 2024-05-28T06:21:52.912 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_in_since": 1716876229, 2024-05-28T06:21:52.912 INFO:teuthology.orchestra.run.smithi096.stdout: "num_remapped_pgs": 0 2024-05-28T06:21:52.912 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:21:52.912 INFO:teuthology.orchestra.run.smithi096.stdout: "pgmap": { 2024-05-28T06:21:52.912 INFO:teuthology.orchestra.run.smithi096.stdout: "pgs_by_state": [ 2024-05-28T06:21:52.912 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:21:52.912 INFO:teuthology.orchestra.run.smithi096.stdout: "state_name": "active+clean", 2024-05-28T06:21:52.912 INFO:teuthology.orchestra.run.smithi096.stdout: "count": 97 2024-05-28T06:21:52.912 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:21:52.913 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:21:52.913 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pgs": 97, 2024-05-28T06:21:52.913 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pools": 3, 2024-05-28T06:21:52.913 INFO:teuthology.orchestra.run.smithi096.stdout: "num_objects": 797, 2024-05-28T06:21:52.913 INFO:teuthology.orchestra.run.smithi096.stdout: "data_bytes": 3222176983, 2024-05-28T06:21:52.913 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_used": 7344947200, 2024-05-28T06:21:52.913 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_avail": 765749166080, 2024-05-28T06:21:52.913 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_total": 773094113280 2024-05-28T06:21:52.913 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:21:52.913 INFO:teuthology.orchestra.run.smithi096.stdout: "fsmap": { 2024-05-28T06:21:52.913 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 63, 2024-05-28T06:21:52.913 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 7, 2024-05-28T06:21:52.914 INFO:teuthology.orchestra.run.smithi096.stdout: "up": 1, 2024-05-28T06:21:52.914 INFO:teuthology.orchestra.run.smithi096.stdout: "in": 1, 2024-05-28T06:21:52.914 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 1, 2024-05-28T06:21:52.914 INFO:teuthology.orchestra.run.smithi096.stdout: "by_rank": [ 2024-05-28T06:21:52.914 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:21:52.914 INFO:teuthology.orchestra.run.smithi096.stdout: "filesystem_id": 7, 2024-05-28T06:21:52.914 INFO:teuthology.orchestra.run.smithi096.stdout: "rank": 0, 2024-05-28T06:21:52.914 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "b", 2024-05-28T06:21:52.914 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "up:active", 2024-05-28T06:21:52.914 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 8739 2024-05-28T06:21:52.914 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:21:52.914 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:21:52.914 INFO:teuthology.orchestra.run.smithi096.stdout: "up:standby": 3 2024-05-28T06:21:52.915 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:21:52.915 INFO:teuthology.orchestra.run.smithi096.stdout: "mgrmap": { 2024-05-28T06:21:52.915 INFO:teuthology.orchestra.run.smithi096.stdout: "available": true, 2024-05-28T06:21:52.915 INFO:teuthology.orchestra.run.smithi096.stdout: "num_standbys": 1, 2024-05-28T06:21:52.915 INFO:teuthology.orchestra.run.smithi096.stdout: "modules": [ 2024-05-28T06:21:52.915 INFO:teuthology.orchestra.run.smithi096.stdout: "iostat", 2024-05-28T06:21:52.915 INFO:teuthology.orchestra.run.smithi096.stdout: "nfs", 2024-05-28T06:21:52.915 INFO:teuthology.orchestra.run.smithi096.stdout: "restful" 2024-05-28T06:21:52.915 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:21:52.915 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:21:52.915 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:21:52.915 INFO:teuthology.orchestra.run.smithi096.stdout: "servicemap": { 2024-05-28T06:21:52.915 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 20, 2024-05-28T06:21:52.916 INFO:teuthology.orchestra.run.smithi096.stdout: "modified": "2024-05-28T06:21:25.943324+0000", 2024-05-28T06:21:52.916 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:21:52.916 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:21:52.916 INFO:teuthology.orchestra.run.smithi096.stdout: "progress_events": {} 2024-05-28T06:21:52.916 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:21:52.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:52.915+0000 7f5381600640 1 -- 172.21.15.96:0/2177937519 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f535806df70 msgr2=0x7f5358070430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:52.917 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:52.915+0000 7f5381600640 1 --2- 172.21.15.96:0/2177937519 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f535806df70 0x7f5358070430 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f537c159cf0 tx=0x7f5364009040 comp rx=0 tx=0).stop 2024-05-28T06:21:52.917 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:52.915+0000 7f5381600640 1 -- 172.21.15.96:0/2177937519 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f537c06f1a0 msgr2=0x7f537c14c160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:52.917 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:52.915+0000 7f5381600640 1 --2- 172.21.15.96:0/2177937519 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f537c06f1a0 0x7f537c14c160 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7f536c00efc0 tx=0x7f536c00c490 comp rx=0 tx=0).stop 2024-05-28T06:21:52.917 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:52.916+0000 7f5381600640 1 -- 172.21.15.96:0/2177937519 shutdown_connections 2024-05-28T06:21:52.917 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:52.916+0000 7f5381600640 1 --2- 172.21.15.96:0/2177937519 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f535806df70 0x7f5358070430 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:52.917 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:52.916+0000 7f5381600640 1 --2- 172.21.15.96:0/2177937519 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f537c15ae10 0x7f537c0b2030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:52.917 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:52.916+0000 7f5381600640 1 --2- 172.21.15.96:0/2177937519 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f537c0b41c0 0x7f537c15bcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:52.917 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:52.916+0000 7f5381600640 1 --2- 172.21.15.96:0/2177937519 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f537c06f1a0 0x7f537c14c160 unknown :-1 s=CLOSED pgs=1690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:52.917 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:52.916+0000 7f5381600640 1 -- 172.21.15.96:0/2177937519 >> 172.21.15.96:0/2177937519 conn(0x7f537c067ce0 msgr2=0x7f537c0b0390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:21:52.917 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:52.916+0000 7f5381600640 1 -- 172.21.15.96:0/2177937519 shutdown_connections 2024-05-28T06:21:52.917 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:52.916+0000 7f5381600640 1 -- 172.21.15.96:0/2177937519 wait complete. 2024-05-28T06:21:52.932 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume ls cephfs --format json 2024-05-28T06:21:53.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.142+0000 7f1439c00640 1 Processor -- start 2024-05-28T06:21:53.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.142+0000 7f1439c00640 1 -- start start 2024-05-28T06:21:53.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.143+0000 7f1439c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f14340caa40 0x7f14340ce960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:53.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.143+0000 7f1439c00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f14340cae20 0x7f1434151db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:53.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.143+0000 7f1439c00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1434152780 con 0x7f14341522f0 2024-05-28T06:21:53.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.144+0000 7f1439c00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f14340d0b30 con 0x7f14340cae20 2024-05-28T06:21:53.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.144+0000 7f1439c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f14340b4160 con 0x7f14340caa40 2024-05-28T06:21:53.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.144+0000 7f1433400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f14340cae20 0x7f1434151db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:53.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.144+0000 7f1439200640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f14341522f0 0x7f14340cf710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:59616/0 (socket says 172.21.15.96:59616) 2024-05-28T06:21:53.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.144+0000 7f1439200640 1 -- 172.21.15.96:0/2265503737 learned_addr learned my addr 172.21.15.96:0/2265503737 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:21:53.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.144+0000 7f1433e00640 1 --2- 172.21.15.96:0/2265503737 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f14340caa40 0x7f14340ce960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:53.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.144+0000 7f1433400640 1 -- 172.21.15.96:0/2265503737 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f14340caa40 msgr2=0x7f14340ce960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:21:53.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.144+0000 7f1433400640 1 --2- 172.21.15.96:0/2265503737 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f14340caa40 0x7f14340ce960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:53.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.145+0000 7f1433400640 1 -- 172.21.15.96:0/2265503737 >> v1:172.21.15.96:6789/0 conn(0x7f14341522f0 legacy=0x7f14340cf710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:53.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.145+0000 7f1433400640 1 -- 172.21.15.96:0/2265503737 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14340b3e80 con 0x7f14340cae20 2024-05-28T06:21:53.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.145+0000 7f1432a00640 1 -- 172.21.15.96:0/2265503737 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1605974494 0 0) 0x7f1434152780 con 0x7f14341522f0 2024-05-28T06:21:53.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.146+0000 7f1433400640 1 --2- 172.21.15.96:0/2265503737 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f14340cae20 0x7f1434151db0 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7f14340b42e0 tx=0x7f142002a0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df18a037971e6d85 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:21:53.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.146+0000 7f1432a00640 1 -- 172.21.15.96:0/2265503737 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1420037070 con 0x7f14340cae20 2024-05-28T06:21:53.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.146+0000 7f1432a00640 1 -- 172.21.15.96:0/2265503737 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f142002ae30 con 0x7f14340cae20 2024-05-28T06:21:53.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.146+0000 7f1432a00640 1 -- 172.21.15.96:0/2265503737 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1420032640 con 0x7f14340cae20 2024-05-28T06:21:53.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.146+0000 7f1439c00640 1 -- 172.21.15.96:0/2265503737 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f14340cae20 msgr2=0x7f1434151db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:53.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.146+0000 7f1439c00640 1 --2- 172.21.15.96:0/2265503737 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f14340cae20 0x7f1434151db0 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7f14340b42e0 tx=0x7f142002a0e0 comp rx=0 tx=0).stop 2024-05-28T06:21:53.143 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.146+0000 7f1439c00640 1 -- 172.21.15.96:0/2265503737 shutdown_connections 2024-05-28T06:21:53.143 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.146+0000 7f1439c00640 1 --2- 172.21.15.96:0/2265503737 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f14340cae20 0x7f1434151db0 unknown :-1 s=CLOSED pgs=1692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:53.143 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.146+0000 7f1439c00640 1 --2- 172.21.15.96:0/2265503737 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f14340caa40 0x7f14340ce960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:53.143 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.147+0000 7f1439c00640 1 -- 172.21.15.96:0/2265503737 >> 172.21.15.96:0/2265503737 conn(0x7f143406b900 msgr2=0x7f143406bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:21:53.143 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.147+0000 7f1439c00640 1 -- 172.21.15.96:0/2265503737 shutdown_connections 2024-05-28T06:21:53.143 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.147+0000 7f1439c00640 1 -- 172.21.15.96:0/2265503737 wait complete. 2024-05-28T06:21:53.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.148+0000 7f1439c00640 1 Processor -- start 2024-05-28T06:21:53.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.148+0000 7f1439c00640 1 -- start start 2024-05-28T06:21:53.145 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.149+0000 7f1439c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f14340caa40 0x7f1434073760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:53.145 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.149+0000 7f1439c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f14340cae20 0x7f1434148490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:53.145 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.149+0000 7f1439c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f14341522f0 0x7f14341489d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:53.145 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.149+0000 7f1439c00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f14340ce440 con 0x7f14340caa40 2024-05-28T06:21:53.145 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.149+0000 7f1439c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f14340ce160 con 0x7f14340cae20 2024-05-28T06:21:53.145 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.149+0000 7f1439c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f14340ceea0 con 0x7f14341522f0 2024-05-28T06:21:53.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.150+0000 7f1433e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f14340caa40 0x7f1434073760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:53.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.150+0000 7f1433e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f14340caa40 0x7f1434073760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47854/0 (socket says 172.21.15.96:47854) 2024-05-28T06:21:53.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.150+0000 7f1433e00640 1 -- 172.21.15.96:0/2387641379 learned_addr learned my addr 172.21.15.96:0/2387641379 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:21:53.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.150+0000 7f1433e00640 1 -- 172.21.15.96:0/2387641379 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f14341522f0 msgr2=0x7f14341489d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:53.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.150+0000 7f1433e00640 1 --2- 172.21.15.96:0/2387641379 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f14341522f0 0x7f14341489d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:53.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.150+0000 7f1433e00640 1 -- 172.21.15.96:0/2387641379 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f14340cae20 msgr2=0x7f1434148490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:53.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.150+0000 7f1433e00640 1 --2- 172.21.15.96:0/2387641379 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f14340cae20 0x7f1434148490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:53.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.150+0000 7f1433e00640 1 -- 172.21.15.96:0/2387641379 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1434147110 con 0x7f14340caa40 2024-05-28T06:21:53.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.150+0000 7f1433e00640 1 --2- 172.21.15.96:0/2387641379 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f14340caa40 0x7f1434073760 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7f142400d620 tx=0x7f142400dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:21:53.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.151+0000 7f142be00640 1 -- 172.21.15.96:0/2387641379 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f142400b620 con 0x7f14340caa40 2024-05-28T06:21:53.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.151+0000 7f142be00640 1 -- 172.21.15.96:0/2387641379 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1424019070 con 0x7f14340caa40 2024-05-28T06:21:53.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.151+0000 7f142be00640 1 -- 172.21.15.96:0/2387641379 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f142400ba00 con 0x7f14340caa40 2024-05-28T06:21:53.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.151+0000 7f1439c00640 1 -- 172.21.15.96:0/2387641379 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14341473f0 con 0x7f14340caa40 2024-05-28T06:21:53.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.151+0000 7f1439c00640 1 -- 172.21.15.96:0/2387641379 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14341479b0 con 0x7f14340caa40 2024-05-28T06:21:53.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.152+0000 7f1439c00640 1 -- 172.21.15.96:0/2387641379 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14340689e0 con 0x7f14340caa40 2024-05-28T06:21:53.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.153+0000 7f142be00640 1 -- 172.21.15.96:0/2387641379 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 36) ==== 93080+0+0 (secure 0 0 0) 0x7f142400bba0 con 0x7f14340caa40 2024-05-28T06:21:53.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.154+0000 7f142be00640 1 --2- 172.21.15.96:0/2387641379 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f141406df70 0x7f1414070430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:53.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.154+0000 7f142be00640 1 -- 172.21.15.96:0/2387641379 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 5765+0+0 (secure 0 0 0) 0x7f142408ad00 con 0x7f14340caa40 2024-05-28T06:21:53.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.155+0000 7f1433400640 1 --2- 172.21.15.96:0/2387641379 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f141406df70 0x7f1414070430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:53.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.156+0000 7f1433400640 1 --2- 172.21.15.96:0/2387641379 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f141406df70 0x7f1414070430 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f1420034000 tx=0x7f142002c000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:21:53.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.158+0000 7f142be00640 1 -- 172.21.15.96:0/2387641379 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f142405ab90 con 0x7f14340caa40 2024-05-28T06:21:53.376 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.381+0000 7f1439c00640 1 -- 172.21.15.96:0/2387641379 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume ls", "vol_name": "cephfs", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f14340b4ee0 con 0x7f141406df70 2024-05-28T06:21:53.390 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.395+0000 7f142be00640 1 -- 172.21.15.96:0/2387641379 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+79 (secure 0 0 0) 0x7f14340b4ee0 con 0x7f141406df70 2024-05-28T06:21:53.390 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:21:53.390 INFO:teuthology.orchestra.run.smithi096.stdout:[ 2024-05-28T06:21:53.390 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:21:53.390 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "sv1" 2024-05-28T06:21:53.390 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:21:53.391 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:21:53.391 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "ss1clone1" 2024-05-28T06:21:53.391 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:21:53.391 INFO:teuthology.orchestra.run.smithi096.stdout:] 2024-05-28T06:21:53.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.396+0000 7f1439c00640 1 -- 172.21.15.96:0/2387641379 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f141406df70 msgr2=0x7f1414070430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:53.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.396+0000 7f1439c00640 1 --2- 172.21.15.96:0/2387641379 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f141406df70 0x7f1414070430 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f1420034000 tx=0x7f142002c000 comp rx=0 tx=0).stop 2024-05-28T06:21:53.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.396+0000 7f1439c00640 1 -- 172.21.15.96:0/2387641379 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f14340caa40 msgr2=0x7f1434073760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:53.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.396+0000 7f1439c00640 1 --2- 172.21.15.96:0/2387641379 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f14340caa40 0x7f1434073760 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7f142400d620 tx=0x7f142400dae0 comp rx=0 tx=0).stop 2024-05-28T06:21:53.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.396+0000 7f1439c00640 1 -- 172.21.15.96:0/2387641379 shutdown_connections 2024-05-28T06:21:53.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.396+0000 7f1439c00640 1 --2- 172.21.15.96:0/2387641379 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f141406df70 0x7f1414070430 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:53.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.396+0000 7f1439c00640 1 --2- 172.21.15.96:0/2387641379 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f14341522f0 0x7f14341489d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:53.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.396+0000 7f1439c00640 1 --2- 172.21.15.96:0/2387641379 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f14340cae20 0x7f1434148490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:53.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.396+0000 7f1439c00640 1 --2- 172.21.15.96:0/2387641379 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f14340caa40 0x7f1434073760 unknown :-1 s=CLOSED pgs=1693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:53.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.396+0000 7f1439c00640 1 -- 172.21.15.96:0/2387641379 >> 172.21.15.96:0/2387641379 conn(0x7f143406b900 msgr2=0x7f1434159fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:21:53.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.397+0000 7f1439c00640 1 -- 172.21.15.96:0/2387641379 shutdown_connections 2024-05-28T06:21:53.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.397+0000 7f1439c00640 1 -- 172.21.15.96:0/2387641379 wait complete. 2024-05-28T06:21:53.409 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot ls cephfs sv1 --format json 2024-05-28T06:21:53.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.614+0000 7f9d95c00640 1 Processor -- start 2024-05-28T06:21:53.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.615+0000 7f9d95c00640 1 -- start start 2024-05-28T06:21:53.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.616+0000 7f9d95c00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f9d900cfb40 0x7f9d900d0d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:53.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.616+0000 7f9d95c00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d900d2200 con 0x7f9d9015e5a0 2024-05-28T06:21:53.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.616+0000 7f9d95c00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d900d24e0 con 0x7f9d900cffd0 2024-05-28T06:21:53.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.616+0000 7f9d95c00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f9d900d2b30 con 0x7f9d900cfb40 2024-05-28T06:21:53.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.617+0000 7f9d87e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f9d900cfb40 0x7f9d900d0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:53.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.617+0000 7f9d8fe00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f9d9015e5a0 0x7f9d900d2fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:59626/0 (socket says 172.21.15.96:59626) 2024-05-28T06:21:53.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.617+0000 7f9d8fe00640 1 -- 172.21.15.96:0/3803098391 learned_addr learned my addr 172.21.15.96:0/3803098391 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:21:53.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.617+0000 7f9d87e00640 1 -- 172.21.15.96:0/3803098391 >> v1:172.21.15.123:6789/0 conn(0x7f9d900cffd0 legacy=0x7f9d9006c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:21:53.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.617+0000 7f9d87e00640 1 -- 172.21.15.96:0/3803098391 >> v1:172.21.15.96:6789/0 conn(0x7f9d9015e5a0 legacy=0x7f9d900d2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:53.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.617+0000 7f9d87e00640 1 -- 172.21.15.96:0/3803098391 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d900d27c0 con 0x7f9d900cfb40 2024-05-28T06:21:53.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.618+0000 7f9d87e00640 1 --2- 172.21.15.96:0/3803098391 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9d900cfb40 0x7f9d900d0d10 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7f9d780097a0 tx=0x7f9d7802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2736daece7e33b08 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:21:53.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.618+0000 7f9d8f400640 1 -- 172.21.15.96:0/3803098391 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d78037070 con 0x7f9d900cfb40 2024-05-28T06:21:53.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.618+0000 7f9d8f400640 1 -- 172.21.15.96:0/3803098391 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d78032440 con 0x7f9d900cfb40 2024-05-28T06:21:53.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.618+0000 7f9d8f400640 1 -- 172.21.15.96:0/3803098391 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d78032810 con 0x7f9d900cfb40 2024-05-28T06:21:53.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.619+0000 7f9d95c00640 1 -- 172.21.15.96:0/3803098391 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9d900cfb40 msgr2=0x7f9d900d0d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:53.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.619+0000 7f9d95c00640 1 --2- 172.21.15.96:0/3803098391 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9d900cfb40 0x7f9d900d0d10 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7f9d780097a0 tx=0x7f9d7802a1e0 comp rx=0 tx=0).stop 2024-05-28T06:21:53.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.619+0000 7f9d95c00640 1 -- 172.21.15.96:0/3803098391 shutdown_connections 2024-05-28T06:21:53.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.619+0000 7f9d95c00640 1 --2- 172.21.15.96:0/3803098391 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9d900cfb40 0x7f9d900d0d10 unknown :-1 s=CLOSED pgs=1695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:53.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.619+0000 7f9d95c00640 1 -- 172.21.15.96:0/3803098391 >> 172.21.15.96:0/3803098391 conn(0x7f9d9008bfc0 msgr2=0x7f9d900d44b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:21:53.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.619+0000 7f9d95c00640 1 -- 172.21.15.96:0/3803098391 shutdown_connections 2024-05-28T06:21:53.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.619+0000 7f9d95c00640 1 -- 172.21.15.96:0/3803098391 wait complete. 2024-05-28T06:21:53.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.621+0000 7f9d95c00640 1 Processor -- start 2024-05-28T06:21:53.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.621+0000 7f9d95c00640 1 -- start start 2024-05-28T06:21:53.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.622+0000 7f9d95c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9d900cfb40 0x7f9d90151370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:53.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.622+0000 7f9d95c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9d900cffd0 0x7f9d9015ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:53.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.622+0000 7f9d95c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9d9015e5a0 0x7f9d900cad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:53.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.622+0000 7f9d95c00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f9d900d53a0 con 0x7f9d900cfb40 2024-05-28T06:21:53.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.622+0000 7f9d95c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f9d900d5220 con 0x7f9d900cffd0 2024-05-28T06:21:53.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.622+0000 7f9d95c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f9d900d1250 con 0x7f9d9015e5a0 2024-05-28T06:21:53.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.622+0000 7f9d8fe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9d900cfb40 0x7f9d90151370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:53.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.622+0000 7f9d8fe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9d900cfb40 0x7f9d90151370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47868/0 (socket says 172.21.15.96:47868) 2024-05-28T06:21:53.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.622+0000 7f9d8fe00640 1 -- 172.21.15.96:0/1678752829 learned_addr learned my addr 172.21.15.96:0/1678752829 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:21:53.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.623+0000 7f9d8fe00640 1 -- 172.21.15.96:0/1678752829 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9d9015e5a0 msgr2=0x7f9d900cad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:53.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.623+0000 7f9d87e00640 1 --2- 172.21.15.96:0/1678752829 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9d900cffd0 0x7f9d9015ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:53.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.623+0000 7f9d95200640 1 --2- 172.21.15.96:0/1678752829 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9d9015e5a0 0x7f9d900cad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:53.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.623+0000 7f9d8fe00640 1 --2- 172.21.15.96:0/1678752829 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9d9015e5a0 0x7f9d900cad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:53.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.623+0000 7f9d8fe00640 1 -- 172.21.15.96:0/1678752829 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9d900cffd0 msgr2=0x7f9d9015ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:53.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.623+0000 7f9d8fe00640 1 --2- 172.21.15.96:0/1678752829 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9d900cffd0 0x7f9d9015ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:53.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.623+0000 7f9d8fe00640 1 -- 172.21.15.96:0/1678752829 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d9015f370 con 0x7f9d900cfb40 2024-05-28T06:21:53.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.623+0000 7f9d87e00640 1 --2- 172.21.15.96:0/1678752829 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9d900cffd0 0x7f9d9015ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:21:53.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.623+0000 7f9d95200640 1 --2- 172.21.15.96:0/1678752829 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9d9015e5a0 0x7f9d900cad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:21:53.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.623+0000 7f9d8fe00640 1 --2- 172.21.15.96:0/1678752829 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9d900cfb40 0x7f9d90151370 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f9d8000c3f0 tx=0x7f9d8000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:21:53.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.624+0000 7f9d8cc00640 1 -- 172.21.15.96:0/1678752829 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d80017070 con 0x7f9d900cfb40 2024-05-28T06:21:53.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.624+0000 7f9d8cc00640 1 -- 172.21.15.96:0/1678752829 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d80005ce0 con 0x7f9d900cfb40 2024-05-28T06:21:53.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.624+0000 7f9d95c00640 1 -- 172.21.15.96:0/1678752829 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d9015f650 con 0x7f9d900cfb40 2024-05-28T06:21:53.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.624+0000 7f9d8cc00640 1 -- 172.21.15.96:0/1678752829 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d800072f0 con 0x7f9d900cfb40 2024-05-28T06:21:53.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.624+0000 7f9d95c00640 1 -- 172.21.15.96:0/1678752829 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d9015fc10 con 0x7f9d900cfb40 2024-05-28T06:21:53.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.625+0000 7f9d95c00640 1 -- 172.21.15.96:0/1678752829 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d900689e0 con 0x7f9d900cfb40 2024-05-28T06:21:53.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.626+0000 7f9d8cc00640 1 -- 172.21.15.96:0/1678752829 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 36) ==== 93080+0+0 (secure 0 0 0) 0x7f9d8001a070 con 0x7f9d900cfb40 2024-05-28T06:21:53.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.627+0000 7f9d8cc00640 1 --2- 172.21.15.96:0/1678752829 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9d6c06dfa0 0x7f9d6c070460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:53.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.627+0000 7f9d8cc00640 1 -- 172.21.15.96:0/1678752829 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 5765+0+0 (secure 0 0 0) 0x7f9d80089c80 con 0x7f9d900cfb40 2024-05-28T06:21:53.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.628+0000 7f9d87e00640 1 --2- 172.21.15.96:0/1678752829 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9d6c06dfa0 0x7f9d6c070460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:53.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.629+0000 7f9d87e00640 1 --2- 172.21.15.96:0/1678752829 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9d6c06dfa0 0x7f9d6c070460 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f9d78034000 tx=0x7f9d7802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:21:53.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.631+0000 7f9d8cc00640 1 -- 172.21.15.96:0/1678752829 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9d80059b10 con 0x7f9d900cfb40 2024-05-28T06:21:53.854 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.858+0000 7f9d95c00640 1 -- 172.21.15.96:0/1678752829 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot ls", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9d900d1d30 con 0x7f9d6c06dfa0 2024-05-28T06:21:53.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.889+0000 7f9d8cc00640 1 -- 172.21.15.96:0/1678752829 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f9d900d1d30 con 0x7f9d6c06dfa0 2024-05-28T06:21:53.884 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:21:53.884 INFO:teuthology.orchestra.run.smithi096.stdout:[ 2024-05-28T06:21:53.884 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:21:53.884 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "ss1" 2024-05-28T06:21:53.884 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:21:53.885 INFO:teuthology.orchestra.run.smithi096.stdout:] 2024-05-28T06:21:53.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.891+0000 7f9d95c00640 1 -- 172.21.15.96:0/1678752829 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9d6c06dfa0 msgr2=0x7f9d6c070460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:53.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.891+0000 7f9d95c00640 1 --2- 172.21.15.96:0/1678752829 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9d6c06dfa0 0x7f9d6c070460 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f9d78034000 tx=0x7f9d7802e000 comp rx=0 tx=0).stop 2024-05-28T06:21:53.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.891+0000 7f9d95c00640 1 -- 172.21.15.96:0/1678752829 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9d900cfb40 msgr2=0x7f9d90151370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:53.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.891+0000 7f9d95c00640 1 --2- 172.21.15.96:0/1678752829 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9d900cfb40 0x7f9d90151370 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f9d8000c3f0 tx=0x7f9d8000c8b0 comp rx=0 tx=0).stop 2024-05-28T06:21:53.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.891+0000 7f9d95c00640 1 -- 172.21.15.96:0/1678752829 shutdown_connections 2024-05-28T06:21:53.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.891+0000 7f9d95c00640 1 --2- 172.21.15.96:0/1678752829 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9d6c06dfa0 0x7f9d6c070460 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:53.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.891+0000 7f9d95c00640 1 --2- 172.21.15.96:0/1678752829 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9d9015e5a0 0x7f9d900cad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:53.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.891+0000 7f9d95c00640 1 --2- 172.21.15.96:0/1678752829 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9d900cffd0 0x7f9d9015ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:53.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.891+0000 7f9d95c00640 1 --2- 172.21.15.96:0/1678752829 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9d900cfb40 0x7f9d90151370 unknown :-1 s=CLOSED pgs=1696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:53.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.891+0000 7f9d95c00640 1 -- 172.21.15.96:0/1678752829 >> 172.21.15.96:0/1678752829 conn(0x7f9d9008bfc0 msgr2=0x7f9d9006d5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:21:53.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.891+0000 7f9d95c00640 1 -- 172.21.15.96:0/1678752829 shutdown_connections 2024-05-28T06:21:53.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:53.891+0000 7f9d95c00640 1 -- 172.21.15.96:0/1678752829 wait complete. 2024-05-28T06:21:53.904 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-05-28T06:21:54.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.108+0000 7fa712800640 1 Processor -- start 2024-05-28T06:21:54.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.109+0000 7fa712800640 1 -- start start 2024-05-28T06:21:54.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.110+0000 7fa712800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa70c0cd2a0 0x7fa70c0b4d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:54.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.110+0000 7fa712800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fa70c15cb60 0x7fa70c15cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:54.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.110+0000 7fa712800640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa70c15d900 con 0x7fa70c153510 2024-05-28T06:21:54.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.110+0000 7fa712800640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fa70c1538f0 con 0x7fa70c15cb60 2024-05-28T06:21:54.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.110+0000 7fa712800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fa70c0b48c0 con 0x7fa70c0cd2a0 2024-05-28T06:21:54.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.111+0000 7fa710a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa70c0cd2a0 0x7fa70c0b4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:54.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.111+0000 7fa711e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fa70c15cb60 0x7fa70c15cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:54.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.111+0000 7fa711400640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7fa70c153510 0x7fa70c0ccb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.96:42168/0 (socket says 172.21.15.96:42168) 2024-05-28T06:21:54.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.111+0000 7fa711400640 1 -- 172.21.15.96:0/1654858507 learned_addr learned my addr 172.21.15.96:0/1654858507 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:21:54.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.112+0000 7fa710a00640 1 -- 172.21.15.96:0/1654858507 >> v2:172.21.15.123:3300/0 conn(0x7fa70c15cb60 msgr2=0x7fa70c15cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:21:54.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.112+0000 7fa710a00640 1 --2- 172.21.15.96:0/1654858507 >> v2:172.21.15.123:3300/0 conn(0x7fa70c15cb60 0x7fa70c15cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:54.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.112+0000 7fa710a00640 1 -- 172.21.15.96:0/1654858507 >> v1:172.21.15.123:6789/0 conn(0x7fa70c153510 legacy=0x7fa70c0ccb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:54.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.112+0000 7fa710a00640 1 -- 172.21.15.96:0/1654858507 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa70c0b3f90 con 0x7fa70c0cd2a0 2024-05-28T06:21:54.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.113+0000 7fa710a00640 1 --2- 172.21.15.96:0/1654858507 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa70c0cd2a0 0x7fa70c0b4d50 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fa6f80097a0 tx=0x7fa6f802a100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=53703e798e6b97ce server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:21:54.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.114+0000 7fa70be00640 1 -- 172.21.15.96:0/1654858507 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6f8037070 con 0x7fa70c0cd2a0 2024-05-28T06:21:54.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.114+0000 7fa70be00640 1 -- 172.21.15.96:0/1654858507 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6f802ae50 con 0x7fa70c0cd2a0 2024-05-28T06:21:54.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.114+0000 7fa70be00640 1 -- 172.21.15.96:0/1654858507 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6f8032660 con 0x7fa70c0cd2a0 2024-05-28T06:21:54.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.114+0000 7fa712800640 1 -- 172.21.15.96:0/1654858507 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa70c0cd2a0 msgr2=0x7fa70c0b4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:54.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.114+0000 7fa712800640 1 --2- 172.21.15.96:0/1654858507 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa70c0cd2a0 0x7fa70c0b4d50 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fa6f80097a0 tx=0x7fa6f802a100 comp rx=0 tx=0).stop 2024-05-28T06:21:54.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.114+0000 7fa712800640 1 -- 172.21.15.96:0/1654858507 shutdown_connections 2024-05-28T06:21:54.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.114+0000 7fa712800640 1 --2- 172.21.15.96:0/1654858507 >> v2:172.21.15.123:3300/0 conn(0x7fa70c15cb60 0x7fa70c15cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:54.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.114+0000 7fa712800640 1 --2- 172.21.15.96:0/1654858507 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa70c0cd2a0 0x7fa70c0b4d50 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:54.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.114+0000 7fa712800640 1 -- 172.21.15.96:0/1654858507 >> 172.21.15.96:0/1654858507 conn(0x7fa70c06b900 msgr2=0x7fa70c06bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:21:54.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.114+0000 7fa712800640 1 -- 172.21.15.96:0/1654858507 shutdown_connections 2024-05-28T06:21:54.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.115+0000 7fa712800640 1 -- 172.21.15.96:0/1654858507 wait complete. 2024-05-28T06:21:54.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.116+0000 7fa712800640 1 Processor -- start 2024-05-28T06:21:54.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.116+0000 7fa712800640 1 -- start start 2024-05-28T06:21:54.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.117+0000 7fa712800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa70c0cd2a0 0x7fa70c0ad130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:54.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.117+0000 7fa712800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa70c153510 0x7fa70c1f3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:54.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.117+0000 7fa712800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa70c15cb60 0x7fa70c0aec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:54.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.117+0000 7fa712800640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa70c15d510 con 0x7fa70c15cb60 2024-05-28T06:21:54.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.117+0000 7fa712800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa70c0b4550 con 0x7fa70c0cd2a0 2024-05-28T06:21:54.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.117+0000 7fa712800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fa70c15d690 con 0x7fa70c153510 2024-05-28T06:21:54.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.117+0000 7fa711e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa70c15cb60 0x7fa70c0aec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:54.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.118+0000 7fa711e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa70c15cb60 0x7fa70c0aec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47884/0 (socket says 172.21.15.96:47884) 2024-05-28T06:21:54.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.118+0000 7fa711e00640 1 -- 172.21.15.96:0/3989414755 learned_addr learned my addr 172.21.15.96:0/3989414755 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:21:54.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.118+0000 7fa711400640 1 --2- 172.21.15.96:0/3989414755 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa70c0cd2a0 0x7fa70c0ad130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:54.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.118+0000 7fa710a00640 1 --2- 172.21.15.96:0/3989414755 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa70c153510 0x7fa70c1f3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:54.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.118+0000 7fa711e00640 1 -- 172.21.15.96:0/3989414755 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa70c153510 msgr2=0x7fa70c1f3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:54.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.118+0000 7fa711e00640 1 --2- 172.21.15.96:0/3989414755 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa70c153510 0x7fa70c1f3fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:54.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.118+0000 7fa711e00640 1 -- 172.21.15.96:0/3989414755 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa70c0cd2a0 msgr2=0x7fa70c0ad130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:54.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.118+0000 7fa711e00640 1 --2- 172.21.15.96:0/3989414755 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa70c0cd2a0 0x7fa70c0ad130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:54.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.118+0000 7fa711e00640 1 -- 172.21.15.96:0/3989414755 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa70c0ad7c0 con 0x7fa70c15cb60 2024-05-28T06:21:54.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.118+0000 7fa711e00640 1 --2- 172.21.15.96:0/3989414755 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa70c15cb60 0x7fa70c0aec20 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7fa704009d30 tx=0x7fa70400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:21:54.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.119+0000 7fa709600640 1 -- 172.21.15.96:0/3989414755 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa704018020 con 0x7fa70c15cb60 2024-05-28T06:21:54.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.119+0000 7fa709600640 1 -- 172.21.15.96:0/3989414755 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa704014480 con 0x7fa70c15cb60 2024-05-28T06:21:54.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.119+0000 7fa712800640 1 -- 172.21.15.96:0/3989414755 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa70c1f44e0 con 0x7fa70c15cb60 2024-05-28T06:21:54.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.119+0000 7fa709600640 1 -- 172.21.15.96:0/3989414755 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa704014870 con 0x7fa70c15cb60 2024-05-28T06:21:54.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.119+0000 7fa712800640 1 -- 172.21.15.96:0/3989414755 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa70c0ce1b0 con 0x7fa70c15cb60 2024-05-28T06:21:54.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.120+0000 7fa712800640 1 -- 172.21.15.96:0/3989414755 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa70c0689e0 con 0x7fa70c15cb60 2024-05-28T06:21:54.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.121+0000 7fa709600640 1 -- 172.21.15.96:0/3989414755 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 36) ==== 93080+0+0 (secure 0 0 0) 0x7fa704005d10 con 0x7fa70c15cb60 2024-05-28T06:21:54.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.122+0000 7fa709600640 1 --2- 172.21.15.96:0/3989414755 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa6ec06df70 0x7fa6ec070430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:54.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.122+0000 7fa709600640 1 -- 172.21.15.96:0/3989414755 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 5765+0+0 (secure 0 0 0) 0x7fa70408b360 con 0x7fa70c15cb60 2024-05-28T06:21:54.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.123+0000 7fa711400640 1 --2- 172.21.15.96:0/3989414755 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa6ec06df70 0x7fa6ec070430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:54.120 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.124+0000 7fa711400640 1 --2- 172.21.15.96:0/3989414755 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa6ec06df70 0x7fa6ec070430 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fa70c151110 tx=0x7fa700009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:21:54.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.127+0000 7fa709600640 1 -- 172.21.15.96:0/3989414755 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa704022070 con 0x7fa70c15cb60 2024-05-28T06:21:54.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.363+0000 7fa712800640 1 -- 172.21.15.96:0/3989414755 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa70c0b3c40 con 0x7fa6ec06df70 2024-05-28T06:21:54.433 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.437+0000 7fa709600640 1 -- 172.21.15.96:0/3989414755 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fa70c0b3c40 con 0x7fa6ec06df70 2024-05-28T06:21:54.433 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:21:54.436 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.440+0000 7fa712800640 1 -- 172.21.15.96:0/3989414755 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa6ec06df70 msgr2=0x7fa6ec070430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:54.436 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.440+0000 7fa712800640 1 --2- 172.21.15.96:0/3989414755 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa6ec06df70 0x7fa6ec070430 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fa70c151110 tx=0x7fa700009040 comp rx=0 tx=0).stop 2024-05-28T06:21:54.436 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.440+0000 7fa712800640 1 -- 172.21.15.96:0/3989414755 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa70c15cb60 msgr2=0x7fa70c0aec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:54.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.440+0000 7fa712800640 1 --2- 172.21.15.96:0/3989414755 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa70c15cb60 0x7fa70c0aec20 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7fa704009d30 tx=0x7fa70400c490 comp rx=0 tx=0).stop 2024-05-28T06:21:54.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.440+0000 7fa712800640 1 -- 172.21.15.96:0/3989414755 shutdown_connections 2024-05-28T06:21:54.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.440+0000 7fa712800640 1 --2- 172.21.15.96:0/3989414755 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa6ec06df70 0x7fa6ec070430 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:54.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.440+0000 7fa712800640 1 --2- 172.21.15.96:0/3989414755 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa70c15cb60 0x7fa70c0aec20 unknown :-1 s=CLOSED pgs=1697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:54.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.440+0000 7fa712800640 1 --2- 172.21.15.96:0/3989414755 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa70c153510 0x7fa70c1f3fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:54.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.440+0000 7fa712800640 1 --2- 172.21.15.96:0/3989414755 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa70c0cd2a0 0x7fa70c0ad130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:54.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.440+0000 7fa712800640 1 -- 172.21.15.96:0/3989414755 >> 172.21.15.96:0/3989414755 conn(0x7fa70c06b900 msgr2=0x7fa70c15bcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:21:54.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.440+0000 7fa712800640 1 -- 172.21.15.96:0/3989414755 shutdown_connections 2024-05-28T06:21:54.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.440+0000 7fa712800640 1 -- 172.21.15.96:0/3989414755 wait complete. 2024-05-28T06:21:54.456 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs sv1 2024-05-28T06:21:54.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.650+0000 7f2543200640 1 Processor -- start 2024-05-28T06:21:54.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.650+0000 7f2543200640 1 -- start start 2024-05-28T06:21:54.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.651+0000 7f2543200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f253c15c5a0 0x7f253c09b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:54.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.651+0000 7f2543200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f253c15c980 0x7f253c160290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:54.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.651+0000 7f2543200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f253c160c30 con 0x7f253c0ceec0 2024-05-28T06:21:54.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.651+0000 7f2543200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f253c09af40 con 0x7f253c15c5a0 2024-05-28T06:21:54.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.651+0000 7f2543200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f253c160f10 con 0x7f253c15c980 2024-05-28T06:21:54.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.652+0000 7f2541200640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f253c0ceec0 0x7f253c0cce20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:59628/0 (socket says 172.21.15.96:59628) 2024-05-28T06:21:54.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.652+0000 7f2541200640 1 -- 172.21.15.96:0/693418663 learned_addr learned my addr 172.21.15.96:0/693418663 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:21:54.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.652+0000 7f253be00640 1 --2- 172.21.15.96:0/693418663 >> v2:172.21.15.96:3300/0 conn(0x7f253c15c5a0 0x7f253c09b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:54.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.652+0000 7f2533400640 1 --2- 172.21.15.96:0/693418663 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f253c15c980 0x7f253c160290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:54.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.652+0000 7f253b400640 1 -- 172.21.15.96:0/693418663 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3163133768 0 0) 0x7f253c160c30 con 0x7f253c0ceec0 2024-05-28T06:21:54.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.652+0000 7f253b400640 1 -- 172.21.15.96:0/693418663 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2520003520 con 0x7f253c0ceec0 2024-05-28T06:21:54.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.652+0000 7f253be00640 1 -- 172.21.15.96:0/693418663 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f253c15c980 msgr2=0x7f253c160290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:21:54.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.652+0000 7f253be00640 1 --2- 172.21.15.96:0/693418663 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f253c15c980 0x7f253c160290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:54.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.652+0000 7f253be00640 1 -- 172.21.15.96:0/693418663 >> v1:172.21.15.96:6789/0 conn(0x7f253c0ceec0 legacy=0x7f253c0cce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:54.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.652+0000 7f253be00640 1 -- 172.21.15.96:0/693418663 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f253c09abd0 con 0x7f253c15c5a0 2024-05-28T06:21:54.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.653+0000 7f253be00640 1 --2- 172.21.15.96:0/693418663 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f253c15c5a0 0x7f253c09b3d0 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7f252c009d90 tx=0x7f252c02a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b738951a6aaa724a server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:21:54.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.654+0000 7f253b400640 1 -- 172.21.15.96:0/693418663 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f252c007740 con 0x7f253c15c5a0 2024-05-28T06:21:54.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.654+0000 7f253b400640 1 -- 172.21.15.96:0/693418663 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f252c0078e0 con 0x7f253c15c5a0 2024-05-28T06:21:54.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.654+0000 7f253b400640 1 -- 172.21.15.96:0/693418663 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f252c007c90 con 0x7f253c15c5a0 2024-05-28T06:21:54.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.654+0000 7f2543200640 1 -- 172.21.15.96:0/693418663 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f253c15c5a0 msgr2=0x7f253c09b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:54.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.654+0000 7f2543200640 1 --2- 172.21.15.96:0/693418663 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f253c15c5a0 0x7f253c09b3d0 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7f252c009d90 tx=0x7f252c02a660 comp rx=0 tx=0).stop 2024-05-28T06:21:54.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.654+0000 7f2543200640 1 -- 172.21.15.96:0/693418663 shutdown_connections 2024-05-28T06:21:54.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.654+0000 7f2543200640 1 --2- 172.21.15.96:0/693418663 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f253c15c980 0x7f253c160290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:54.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.654+0000 7f2543200640 1 --2- 172.21.15.96:0/693418663 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f253c15c5a0 0x7f253c09b3d0 unknown :-1 s=CLOSED pgs=1699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:54.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.654+0000 7f2543200640 1 -- 172.21.15.96:0/693418663 >> 172.21.15.96:0/693418663 conn(0x7f253c0b4040 msgr2=0x7f253c0b4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:21:54.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.654+0000 7f2543200640 1 -- 172.21.15.96:0/693418663 shutdown_connections 2024-05-28T06:21:54.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.654+0000 7f2543200640 1 -- 172.21.15.96:0/693418663 wait complete. 2024-05-28T06:21:54.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.656+0000 7f2543200640 1 Processor -- start 2024-05-28T06:21:54.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.656+0000 7f2543200640 1 -- start start 2024-05-28T06:21:54.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.657+0000 7f2543200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f253c0ceec0 0x7f253c159200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:54.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.657+0000 7f2543200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f253c15c5a0 0x7f253c0b04a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:54.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.657+0000 7f2543200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f253c15c980 0x7f253c0b09e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:54.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.657+0000 7f2543200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f253c071de0 con 0x7f253c15c980 2024-05-28T06:21:54.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.657+0000 7f2543200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f253c071c60 con 0x7f253c15c5a0 2024-05-28T06:21:54.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.657+0000 7f2543200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f253c1607d0 con 0x7f253c0ceec0 2024-05-28T06:21:54.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.657+0000 7f2541200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f253c15c980 0x7f253c0b09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:54.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.657+0000 7f2541200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f253c15c980 0x7f253c0b09e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47896/0 (socket says 172.21.15.96:47896) 2024-05-28T06:21:54.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.657+0000 7f2541200640 1 -- 172.21.15.96:0/3208635976 learned_addr learned my addr 172.21.15.96:0/3208635976 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:21:54.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.658+0000 7f2541200640 1 -- 172.21.15.96:0/3208635976 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f253c0ceec0 msgr2=0x7f253c159200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:54.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.658+0000 7f2541200640 1 --2- 172.21.15.96:0/3208635976 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f253c0ceec0 0x7f253c159200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:54.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.658+0000 7f2541200640 1 -- 172.21.15.96:0/3208635976 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f253c15c5a0 msgr2=0x7f253c0b04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:54.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.658+0000 7f2541200640 1 --2- 172.21.15.96:0/3208635976 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f253c15c5a0 0x7f253c0b04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:54.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.658+0000 7f2541200640 1 -- 172.21.15.96:0/3208635976 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f253c159a30 con 0x7f253c15c980 2024-05-28T06:21:54.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.658+0000 7f2541200640 1 --2- 172.21.15.96:0/3208635976 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f253c15c980 0x7f253c0b09e0 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7f253400c460 tx=0x7f253400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:21:54.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.658+0000 7f2538c00640 1 -- 172.21.15.96:0/3208635976 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2534017070 con 0x7f253c15c980 2024-05-28T06:21:54.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.658+0000 7f2538c00640 1 -- 172.21.15.96:0/3208635976 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2534005ce0 con 0x7f253c15c980 2024-05-28T06:21:54.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.658+0000 7f2538c00640 1 -- 172.21.15.96:0/3208635976 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2534007290 con 0x7f253c15c980 2024-05-28T06:21:54.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.658+0000 7f2543200640 1 -- 172.21.15.96:0/3208635976 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f253c159d10 con 0x7f253c15c980 2024-05-28T06:21:54.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.659+0000 7f2543200640 1 -- 172.21.15.96:0/3208635976 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f253c159740 con 0x7f253c15c980 2024-05-28T06:21:54.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.659+0000 7f2543200640 1 -- 172.21.15.96:0/3208635976 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f253c0690f0 con 0x7f253c15c980 2024-05-28T06:21:54.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.660+0000 7f2538c00640 1 -- 172.21.15.96:0/3208635976 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 36) ==== 93080+0+0 (secure 0 0 0) 0x7f253401a070 con 0x7f253c15c980 2024-05-28T06:21:54.657 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.661+0000 7f2538c00640 1 --2- 172.21.15.96:0/3208635976 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f251806df70 0x7f2518070430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:54.657 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.661+0000 7f2538c00640 1 -- 172.21.15.96:0/3208635976 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 5765+0+0 (secure 0 0 0) 0x7f2534089c30 con 0x7f253c15c980 2024-05-28T06:21:54.657 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.662+0000 7f253be00640 1 --2- 172.21.15.96:0/3208635976 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f251806df70 0x7f2518070430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:54.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.663+0000 7f253be00640 1 --2- 172.21.15.96:0/3208635976 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f251806df70 0x7f2518070430 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f252c034000 tx=0x7f252c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:21:54.661 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.665+0000 7f2538c00640 1 -- 172.21.15.96:0/3208635976 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2534059ac0 con 0x7f253c15c980 2024-05-28T06:21:54.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.893+0000 7f2543200640 1 -- 172.21.15.96:0/3208635976 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) -- 0x7f253c0b3bd0 con 0x7f251806df70 2024-05-28T06:21:54.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.941+0000 7f2538c00640 1 -- 172.21.15.96:0/3208635976 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f253c0b3bd0 con 0x7f251806df70 2024-05-28T06:21:54.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.945+0000 7f2543200640 1 -- 172.21.15.96:0/3208635976 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f251806df70 msgr2=0x7f2518070430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:54.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.945+0000 7f2543200640 1 --2- 172.21.15.96:0/3208635976 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f251806df70 0x7f2518070430 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f252c034000 tx=0x7f252c02e000 comp rx=0 tx=0).stop 2024-05-28T06:21:54.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.945+0000 7f2543200640 1 -- 172.21.15.96:0/3208635976 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f253c15c980 msgr2=0x7f253c0b09e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:54.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.945+0000 7f2543200640 1 --2- 172.21.15.96:0/3208635976 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f253c15c980 0x7f253c0b09e0 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7f253400c460 tx=0x7f253400c920 comp rx=0 tx=0).stop 2024-05-28T06:21:54.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.945+0000 7f2543200640 1 -- 172.21.15.96:0/3208635976 shutdown_connections 2024-05-28T06:21:54.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.945+0000 7f2543200640 1 --2- 172.21.15.96:0/3208635976 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f251806df70 0x7f2518070430 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:54.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.945+0000 7f2543200640 1 --2- 172.21.15.96:0/3208635976 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f253c15c980 0x7f253c0b09e0 unknown :-1 s=CLOSED pgs=1700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:54.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.945+0000 7f2543200640 1 --2- 172.21.15.96:0/3208635976 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f253c15c5a0 0x7f253c0b04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:54.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.945+0000 7f2543200640 1 --2- 172.21.15.96:0/3208635976 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f253c0ceec0 0x7f253c159200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:54.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.945+0000 7f2543200640 1 -- 172.21.15.96:0/3208635976 >> 172.21.15.96:0/3208635976 conn(0x7f253c0b4040 msgr2=0x7f253c0b1ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:21:54.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.945+0000 7f2543200640 1 -- 172.21.15.96:0/3208635976 shutdown_connections 2024-05-28T06:21:54.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:54.945+0000 7f2543200640 1 -- 172.21.15.96:0/3208635976 wait complete. 2024-05-28T06:21:54.948 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:21:54.953+0000 7fea5da00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:21:54.948 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:21:54.953+0000 7fea5da00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:21:54.948 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:21:54.953+0000 7fea5da00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:21:54.949 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:21:54.953+0000 7fea5da00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:21:54.949 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:21:54.953+0000 7fea5da00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:21:54.961 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs ss1clone1 2024-05-28T06:21:54.970 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:21:54.975+0000 7fea5bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:21:54.970 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:21:54.975+0000 7fea5bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:21:54.971 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:21:54.975+0000 7fea5bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:21:54.971 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:21:54.975+0000 7fea5bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:21:54.971 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:21:54.975+0000 7fea5bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:21:55.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:55.160+0000 7f8176200640 1 Processor -- start 2024-05-28T06:21:55.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:55.160+0000 7f8176200640 1 -- start start 2024-05-28T06:21:55.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:55.161+0000 7f8176200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f8170144f50 0x7f81701455e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:55.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:55.161+0000 7f8176200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f817006b900 0x7f817006bd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:55.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:55.161+0000 7f8176200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81700b48e0 con 0x7f81700b4320 2024-05-28T06:21:55.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:55.161+0000 7f8176200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f8170145330 con 0x7f8170144f50 2024-05-28T06:21:55.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:55.161+0000 7f8176200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f8170145b20 con 0x7f817006b900 2024-05-28T06:21:55.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:55.162+0000 7f816f400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f8170144f50 0x7f81701455e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:55.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:55.162+0000 7f816f400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f8170144f50 0x7f81701455e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.96:47084/0 (socket says 172.21.15.96:47084) 2024-05-28T06:21:55.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:55.162+0000 7f816f400640 1 -- 172.21.15.96:0/1061353437 learned_addr learned my addr 172.21.15.96:0/1061353437 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:21:55.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:55.162+0000 7f816fe00640 1 --2- 172.21.15.96:0/1061353437 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f817006b900 0x7f817006bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:55.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:55.162+0000 7f816f400640 1 -- 172.21.15.96:0/1061353437 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f817006b900 msgr2=0x7f817006bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:21:55.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:55.163+0000 7f816f400640 1 --2- 172.21.15.96:0/1061353437 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f817006b900 0x7f817006bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:55.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:55.163+0000 7f816f400640 1 -- 172.21.15.96:0/1061353437 >> v1:172.21.15.123:6789/0 conn(0x7f81700b4320 legacy=0x7f81700b28b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:55.159 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:55.163+0000 7f816f400640 1 -- 172.21.15.96:0/1061353437 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81700b62d0 con 0x7f8170144f50 2024-05-28T06:21:55.159 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:55.164+0000 7f816f400640 1 --2- 172.21.15.96:0/1061353437 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8170144f50 0x7f81701455e0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f8160009870 tx=0x7f816002a1d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7e0127b983ab319d server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:21:55.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:55.164+0000 7f816e000640 1 -- 172.21.15.96:0/1061353437 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8160037070 con 0x7f8170144f50 2024-05-28T06:21:55.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:55.164+0000 7f816e000640 1 -- 172.21.15.96:0/1061353437 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8160032440 con 0x7f8170144f50 2024-05-28T06:21:55.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:55.164+0000 7f816e000640 1 -- 172.21.15.96:0/1061353437 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8160032810 con 0x7f8170144f50 2024-05-28T06:21:55.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:55.165+0000 7f8176200640 1 -- 172.21.15.96:0/1061353437 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8170144f50 msgr2=0x7f81701455e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:55.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:55.165+0000 7f8176200640 1 --2- 172.21.15.96:0/1061353437 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8170144f50 0x7f81701455e0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f8160009870 tx=0x7f816002a1d0 comp rx=0 tx=0).stop 2024-05-28T06:21:55.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:55.165+0000 7f8176200640 1 -- 172.21.15.96:0/1061353437 shutdown_connections 2024-05-28T06:21:55.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:55.165+0000 7f8176200640 1 --2- 172.21.15.96:0/1061353437 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f817006b900 0x7f817006bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:55.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:55.165+0000 7f8176200640 1 --2- 172.21.15.96:0/1061353437 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8170144f50 0x7f81701455e0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:55.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:55.165+0000 7f8176200640 1 -- 172.21.15.96:0/1061353437 >> 172.21.15.96:0/1061353437 conn(0x7f81700b6a60 msgr2=0x7f817009aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:21:55.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:55.165+0000 7f8176200640 1 -- 172.21.15.96:0/1061353437 shutdown_connections 2024-05-28T06:21:55.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:55.165+0000 7f8176200640 1 -- 172.21.15.96:0/1061353437 wait complete. 2024-05-28T06:21:55.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:55.166+0000 7f8176200640 1 Processor -- start 2024-05-28T06:21:55.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:55.167+0000 7f8176200640 1 -- start start 2024-05-28T06:21:55.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:55.167+0000 7f8176200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f817006b900 0x7f8170161480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:55.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:55.167+0000 7f8176200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f81700b4320 0x7f81701619c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:55.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:55.168+0000 7f8176200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8170144f50 0x7f817013ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:55.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:55.168+0000 7f8176200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f81700b5ef0 con 0x7f81700b4320 2024-05-28T06:21:55.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:55.168+0000 7f8176200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f817006c270 con 0x7f8170144f50 2024-05-28T06:21:55.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:55.168+0000 7f8176200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f81700b6070 con 0x7f817006b900 2024-05-28T06:21:55.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:55.168+0000 7f816ea00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f81700b4320 0x7f81701619c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:55.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:55.168+0000 7f816ea00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f81700b4320 0x7f81701619c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47900/0 (socket says 172.21.15.96:47900) 2024-05-28T06:21:55.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:55.168+0000 7f816ea00640 1 -- 172.21.15.96:0/844199598 learned_addr learned my addr 172.21.15.96:0/844199598 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:21:55.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:55.168+0000 7f816f400640 1 --2- 172.21.15.96:0/844199598 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f817006b900 0x7f8170161480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:55.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:55.168+0000 7f816fe00640 1 --2- 172.21.15.96:0/844199598 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8170144f50 0x7f817013ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:55.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:55.168+0000 7f816ea00640 1 -- 172.21.15.96:0/844199598 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f817006b900 msgr2=0x7f8170161480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:55.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:55.168+0000 7f816ea00640 1 --2- 172.21.15.96:0/844199598 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f817006b900 0x7f8170161480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:55.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:55.168+0000 7f816ea00640 1 -- 172.21.15.96:0/844199598 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8170144f50 msgr2=0x7f817013ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:55.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:55.168+0000 7f816ea00640 1 --2- 172.21.15.96:0/844199598 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8170144f50 0x7f817013ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:55.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:55.169+0000 7f816ea00640 1 -- 172.21.15.96:0/844199598 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8170140830 con 0x7f81700b4320 2024-05-28T06:21:55.166 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:55.169+0000 7f816f400640 1 --2- 172.21.15.96:0/844199598 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f817006b900 0x7f8170161480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-28T06:21:55.166 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:55.169+0000 7f816ea00640 1 --2- 172.21.15.96:0/844199598 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f81700b4320 0x7f81701619c0 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7f815c00c400 tx=0x7f815c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:21:55.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:55.169+0000 7f8167400640 1 -- 172.21.15.96:0/844199598 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f815c017070 con 0x7f81700b4320 2024-05-28T06:21:55.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:55.169+0000 7f8167400640 1 -- 172.21.15.96:0/844199598 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f815c0052d0 con 0x7f81700b4320 2024-05-28T06:21:55.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:55.169+0000 7f8176200640 1 -- 172.21.15.96:0/844199598 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8170140b10 con 0x7f81700b4320 2024-05-28T06:21:55.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:55.169+0000 7f8167400640 1 -- 172.21.15.96:0/844199598 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f815c003830 con 0x7f81700b4320 2024-05-28T06:21:55.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:55.169+0000 7f8176200640 1 -- 172.21.15.96:0/844199598 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81701410d0 con 0x7f81700b4320 2024-05-28T06:21:55.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:55.170+0000 7f8176200640 1 -- 172.21.15.96:0/844199598 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81700690f0 con 0x7f81700b4320 2024-05-28T06:21:55.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:55.171+0000 7f8167400640 1 -- 172.21.15.96:0/844199598 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 36) ==== 93080+0+0 (secure 0 0 0) 0x7f815c01a070 con 0x7f81700b4320 2024-05-28T06:21:55.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:55.173+0000 7f8167400640 1 --2- 172.21.15.96:0/844199598 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f815006df70 0x7f8150070430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:55.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:55.173+0000 7f8167400640 1 -- 172.21.15.96:0/844199598 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 5765+0+0 (secure 0 0 0) 0x7f815c089c60 con 0x7f81700b4320 2024-05-28T06:21:55.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:55.174+0000 7f816f400640 1 --2- 172.21.15.96:0/844199598 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f815006df70 0x7f8150070430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:55.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:55.175+0000 7f816f400640 1 --2- 172.21.15.96:0/844199598 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f815006df70 0x7f8150070430 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f8160034000 tx=0x7f816002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:21:55.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:55.177+0000 7f8167400640 1 -- 172.21.15.96:0/844199598 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f815c059af0 con 0x7f81700b4320 2024-05-28T06:21:55.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:55.406+0000 7f8176200640 1 -- 172.21.15.96:0/844199598 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "ss1clone1", "target": ["mon-mgr", ""]}) -- 0x7f81700d0930 con 0x7f815006df70 2024-05-28T06:21:55.452 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:55.456+0000 7f8167400640 1 -- 172.21.15.96:0/844199598 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f81700d0930 con 0x7f815006df70 2024-05-28T06:21:55.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:55.459+0000 7f8176200640 1 -- 172.21.15.96:0/844199598 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f815006df70 msgr2=0x7f8150070430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:55.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:55.459+0000 7f8176200640 1 --2- 172.21.15.96:0/844199598 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f815006df70 0x7f8150070430 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f8160034000 tx=0x7f816002e000 comp rx=0 tx=0).stop 2024-05-28T06:21:55.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:55.459+0000 7f8176200640 1 -- 172.21.15.96:0/844199598 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f81700b4320 msgr2=0x7f81701619c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:55.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:55.459+0000 7f8176200640 1 --2- 172.21.15.96:0/844199598 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f81700b4320 0x7f81701619c0 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7f815c00c400 tx=0x7f815c00c8c0 comp rx=0 tx=0).stop 2024-05-28T06:21:55.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:55.459+0000 7f8176200640 1 -- 172.21.15.96:0/844199598 shutdown_connections 2024-05-28T06:21:55.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:55.459+0000 7f8176200640 1 --2- 172.21.15.96:0/844199598 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f815006df70 0x7f8150070430 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:55.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:55.459+0000 7f8176200640 1 --2- 172.21.15.96:0/844199598 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8170144f50 0x7f817013ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:55.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:55.459+0000 7f8176200640 1 --2- 172.21.15.96:0/844199598 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f81700b4320 0x7f81701619c0 unknown :-1 s=CLOSED pgs=1701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:55.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:55.459+0000 7f8176200640 1 --2- 172.21.15.96:0/844199598 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f817006b900 0x7f8170161480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:55.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:55.459+0000 7f8176200640 1 -- 172.21.15.96:0/844199598 >> 172.21.15.96:0/844199598 conn(0x7f81700b6a60 msgr2=0x7f817006f690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:21:55.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:55.459+0000 7f8176200640 1 -- 172.21.15.96:0/844199598 shutdown_connections 2024-05-28T06:21:55.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:55.459+0000 7f8176200640 1 -- 172.21.15.96:0/844199598 wait complete. 2024-05-28T06:21:55.472 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2024-05-28T06:21:55.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:55 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-28T06:21:55.580 INFO:teuthology.orchestra.run.smithi096.stdout:2 2024-05-28T06:21:55.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-28T06:21:55.589 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-05-28T06:21:55.589 DEBUG:teuthology.orchestra.run.smithi096:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-05-28T06:21:55.685 DEBUG:teuthology.orchestra.run.smithi123:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-05-28T06:21:55.792 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-05-28T06:21:55.792 INFO:tasks.cephfs.mount:Terminating background process 2024-05-28T06:21:55.792 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:21:55.792 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:21:55.846 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:21:56.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.051+0000 7f83e5600640 1 Processor -- start 2024-05-28T06:21:56.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.051+0000 7f83e5600640 1 -- start start 2024-05-28T06:21:56.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.052+0000 7f83e5600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f83e015e920 0x7f83e00b42d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:56.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.052+0000 7f83e5600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f83e009aac0 0x7f83e006b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:56.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.052+0000 7f83e5600640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83e00b2860 con 0x7f83e006be40 2024-05-28T06:21:56.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.052+0000 7f83e5600640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f83e00b5ea0 con 0x7f83e009aac0 2024-05-28T06:21:56.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.052+0000 7f83e5600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f83e015ed00 con 0x7f83e015e920 2024-05-28T06:21:56.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.053+0000 7f83d7400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f83e009aac0 0x7f83e006b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:56.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.053+0000 7f83e4c00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f83e006be40 0x7f83e0162420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:59642/0 (socket says 172.21.15.96:59642) 2024-05-28T06:21:56.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.053+0000 7f83e4c00640 1 -- 172.21.15.96:0/3958386128 learned_addr learned my addr 172.21.15.96:0/3958386128 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:21:56.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.053+0000 7f83dfe00640 1 --2- 172.21.15.96:0/3958386128 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f83e015e920 0x7f83e00b42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:56.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.053+0000 7f83df400640 1 -- 172.21.15.96:0/3958386128 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3497337221 0 0) 0x7f83e00b2860 con 0x7f83e006be40 2024-05-28T06:21:56.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.053+0000 7f83df400640 1 -- 172.21.15.96:0/3958386128 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f83c4003520 con 0x7f83e006be40 2024-05-28T06:21:56.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.053+0000 7f83d7400640 1 -- 172.21.15.96:0/3958386128 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f83e015e920 msgr2=0x7f83e00b42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:21:56.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.053+0000 7f83d7400640 1 --2- 172.21.15.96:0/3958386128 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f83e015e920 0x7f83e00b42d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:56.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.053+0000 7f83d7400640 1 -- 172.21.15.96:0/3958386128 >> v1:172.21.15.96:6789/0 conn(0x7f83e006be40 legacy=0x7f83e0162420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:56.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.053+0000 7f83d7400640 1 -- 172.21.15.96:0/3958386128 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83e00b2b40 con 0x7f83e009aac0 2024-05-28T06:21:56.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.055+0000 7f83d7400640 1 --2- 172.21.15.96:0/3958386128 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83e009aac0 0x7f83e006b900 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7f83c80097a0 tx=0x7f83c802a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50709f1161f548ab server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:21:56.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.055+0000 7f83df400640 1 -- 172.21.15.96:0/3958386128 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83c8037070 con 0x7f83e009aac0 2024-05-28T06:21:56.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.056+0000 7f83df400640 1 -- 172.21.15.96:0/3958386128 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83c802ae50 con 0x7f83e009aac0 2024-05-28T06:21:56.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.056+0000 7f83df400640 1 -- 172.21.15.96:0/3958386128 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83c8032660 con 0x7f83e009aac0 2024-05-28T06:21:56.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.056+0000 7f83e5600640 1 -- 172.21.15.96:0/3958386128 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83e009aac0 msgr2=0x7f83e006b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:56.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.056+0000 7f83e5600640 1 --2- 172.21.15.96:0/3958386128 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83e009aac0 0x7f83e006b900 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7f83c80097a0 tx=0x7f83c802a100 comp rx=0 tx=0).stop 2024-05-28T06:21:56.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.056+0000 7f83e5600640 1 -- 172.21.15.96:0/3958386128 shutdown_connections 2024-05-28T06:21:56.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.056+0000 7f83e5600640 1 --2- 172.21.15.96:0/3958386128 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83e009aac0 0x7f83e006b900 unknown :-1 s=CLOSED pgs=1703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:56.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.056+0000 7f83e5600640 1 --2- 172.21.15.96:0/3958386128 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f83e015e920 0x7f83e00b42d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:56.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.056+0000 7f83e5600640 1 -- 172.21.15.96:0/3958386128 >> 172.21.15.96:0/3958386128 conn(0x7f83e00b6a10 msgr2=0x7f83e00b2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:21:56.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.056+0000 7f83e5600640 1 -- 172.21.15.96:0/3958386128 shutdown_connections 2024-05-28T06:21:56.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.056+0000 7f83e5600640 1 -- 172.21.15.96:0/3958386128 wait complete. 2024-05-28T06:21:56.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.058+0000 7f83e5600640 1 Processor -- start 2024-05-28T06:21:56.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.058+0000 7f83e5600640 1 -- start start 2024-05-28T06:21:56.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.059+0000 7f83e5600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f83e006be40 0x7f83e0155f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:56.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.059+0000 7f83e5600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83e009aac0 0x7f83e01f52b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:56.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.059+0000 7f83e5600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f83e015e920 0x7f83e01514d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:56.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.059+0000 7f83e5600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f83e00b2e20 con 0x7f83e009aac0 2024-05-28T06:21:56.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.059+0000 7f83e5600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f83e00b2610 con 0x7f83e015e920 2024-05-28T06:21:56.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.059+0000 7f83e5600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f83e00b4810 con 0x7f83e006be40 2024-05-28T06:21:56.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.059+0000 7f83d7400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83e009aac0 0x7f83e01f52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:56.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.059+0000 7f83d7400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83e009aac0 0x7f83e01f52b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47922/0 (socket says 172.21.15.96:47922) 2024-05-28T06:21:56.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.059+0000 7f83d7400640 1 -- 172.21.15.96:0/2905489702 learned_addr learned my addr 172.21.15.96:0/2905489702 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:21:56.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.060+0000 7f83dfe00640 1 --2- 172.21.15.96:0/2905489702 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f83e006be40 0x7f83e0155f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:56.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.060+0000 7f83d7400640 1 -- 172.21.15.96:0/2905489702 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f83e006be40 msgr2=0x7f83e0155f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:56.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.060+0000 7f83d7400640 1 --2- 172.21.15.96:0/2905489702 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f83e006be40 0x7f83e0155f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:56.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.060+0000 7f83e4c00640 1 --2- 172.21.15.96:0/2905489702 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f83e015e920 0x7f83e01514d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:56.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.060+0000 7f83d7400640 1 -- 172.21.15.96:0/2905489702 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f83e015e920 msgr2=0x7f83e01514d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:56.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.060+0000 7f83d7400640 1 --2- 172.21.15.96:0/2905489702 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f83e015e920 0x7f83e01514d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:56.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.060+0000 7f83d7400640 1 -- 172.21.15.96:0/2905489702 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83e01565e0 con 0x7f83e009aac0 2024-05-28T06:21:56.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.060+0000 7f83e4c00640 1 --2- 172.21.15.96:0/2905489702 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f83e015e920 0x7f83e01514d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:21:56.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.060+0000 7f83d7400640 1 --2- 172.21.15.96:0/2905489702 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83e009aac0 0x7f83e01f52b0 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7f83c8035040 tx=0x7f83c80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:21:56.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.061+0000 7f83dcc00640 1 -- 172.21.15.96:0/2905489702 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83c8041070 con 0x7f83e009aac0 2024-05-28T06:21:56.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.061+0000 7f83dcc00640 1 -- 172.21.15.96:0/2905489702 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83c803c460 con 0x7f83e009aac0 2024-05-28T06:21:56.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.061+0000 7f83e5600640 1 -- 172.21.15.96:0/2905489702 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83e01568c0 con 0x7f83e009aac0 2024-05-28T06:21:56.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.061+0000 7f83dcc00640 1 -- 172.21.15.96:0/2905489702 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83c8037040 con 0x7f83e009aac0 2024-05-28T06:21:56.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.061+0000 7f83e5600640 1 -- 172.21.15.96:0/2905489702 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83e0156e80 con 0x7f83e009aac0 2024-05-28T06:21:56.061 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.062+0000 7f83e5600640 1 -- 172.21.15.96:0/2905489702 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83e00690f0 con 0x7f83e009aac0 2024-05-28T06:21:56.061 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.063+0000 7f83dcc00640 1 -- 172.21.15.96:0/2905489702 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 36) ==== 93080+0+0 (secure 0 0 0) 0x7f83c803a070 con 0x7f83e009aac0 2024-05-28T06:21:56.061 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.064+0000 7f83dcc00640 1 --2- 172.21.15.96:0/2905489702 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f83bc06df70 0x7f83bc070430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:56.061 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.064+0000 7f83dcc00640 1 -- 172.21.15.96:0/2905489702 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 5765+0+0 (secure 0 0 0) 0x7f83c80aab60 con 0x7f83e009aac0 2024-05-28T06:21:56.061 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.064+0000 7f83dfe00640 1 --2- 172.21.15.96:0/2905489702 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f83bc06df70 0x7f83bc070430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:56.062 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.065+0000 7f83dfe00640 1 --2- 172.21.15.96:0/2905489702 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f83bc06df70 0x7f83bc070430 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f83e0155ed0 tx=0x7f83d000b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:21:56.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.068+0000 7f83dcc00640 1 -- 172.21.15.96:0/2905489702 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f83c807a9f0 con 0x7f83e009aac0 2024-05-28T06:21:56.259 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.263+0000 7f83e5600640 1 -- 172.21.15.96:0/2905489702 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f83e00b3b10 con 0x7f83e009aac0 2024-05-28T06:21:56.261 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.265+0000 7f83dcc00640 1 -- 172.21.15.96:0/2905489702 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v103) ==== 74+0+16719 (secure 0 0 0) 0x7f83c807a2f0 con 0x7f83e009aac0 2024-05-28T06:21:56.261 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:21:56.261 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":103,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:20:19.907412+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-05-28T06:20:16.097747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-05-28T06:20:17.338574+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"103","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":13,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:21:56.263 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.268+0000 7f83e5600640 1 -- 172.21.15.96:0/2905489702 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f83bc06df70 msgr2=0x7f83bc070430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:56.264 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.268+0000 7f83e5600640 1 --2- 172.21.15.96:0/2905489702 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f83bc06df70 0x7f83bc070430 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f83e0155ed0 tx=0x7f83d000b040 comp rx=0 tx=0).stop 2024-05-28T06:21:56.264 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.268+0000 7f83e5600640 1 -- 172.21.15.96:0/2905489702 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83e009aac0 msgr2=0x7f83e01f52b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:56.264 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.268+0000 7f83e5600640 1 --2- 172.21.15.96:0/2905489702 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83e009aac0 0x7f83e01f52b0 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7f83c8035040 tx=0x7f83c80046c0 comp rx=0 tx=0).stop 2024-05-28T06:21:56.264 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.268+0000 7f83e5600640 1 -- 172.21.15.96:0/2905489702 shutdown_connections 2024-05-28T06:21:56.264 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.268+0000 7f83e5600640 1 --2- 172.21.15.96:0/2905489702 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f83bc06df70 0x7f83bc070430 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:56.264 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.268+0000 7f83e5600640 1 --2- 172.21.15.96:0/2905489702 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f83e015e920 0x7f83e01514d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:56.264 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.268+0000 7f83e5600640 1 --2- 172.21.15.96:0/2905489702 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83e009aac0 0x7f83e01f52b0 unknown :-1 s=CLOSED pgs=1704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:56.264 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.268+0000 7f83e5600640 1 --2- 172.21.15.96:0/2905489702 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f83e006be40 0x7f83e0155f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:56.265 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.268+0000 7f83e5600640 1 -- 172.21.15.96:0/2905489702 >> 172.21.15.96:0/2905489702 conn(0x7f83e00b6a10 msgr2=0x7f83e0070420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:21:56.265 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.268+0000 7f83e5600640 1 -- 172.21.15.96:0/2905489702 shutdown_connections 2024-05-28T06:21:56.265 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.268+0000 7f83e5600640 1 -- 172.21.15.96:0/2905489702 wait complete. 2024-05-28T06:21:56.282 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-28T06:03:59.517102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-28T06:20:16.097747+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '102', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2024-05-28T06:20:17.338574+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '103', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-28T06:21:56.282 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-28T06:21:56.474 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.479+0000 7f4c71a00640 1 Processor -- start 2024-05-28T06:21:56.474 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.479+0000 7f4c71a00640 1 -- start start 2024-05-28T06:21:56.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.480+0000 7f4c71a00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f4c6c15d5d0 0x7f4c6c0cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:56.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.480+0000 7f4c71a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f4c6c0b32c0 0x7f4c6c0b3700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:56.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.480+0000 7f4c71a00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c6c15cb10 con 0x7f4c6c0d0640 2024-05-28T06:21:56.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.480+0000 7f4c71a00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f4c6c0adbb0 con 0x7f4c6c0b32c0 2024-05-28T06:21:56.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.480+0000 7f4c71a00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f4c6c0ad5f0 con 0x7f4c6c15d5d0 2024-05-28T06:21:56.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.482+0000 7f4c6b400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f4c6c0b32c0 0x7f4c6c0b3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:56.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.482+0000 7f4c6b400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f4c6c0b32c0 0x7f4c6c0b3700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47938/0 (socket says 172.21.15.96:47938) 2024-05-28T06:21:56.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.482+0000 7f4c6b400640 1 -- 172.21.15.96:0/2794126883 learned_addr learned my addr 172.21.15.96:0/2794126883 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:21:56.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.482+0000 7f4c6a000640 1 --2- 172.21.15.96:0/2794126883 >> v2:172.21.15.123:3300/0 conn(0x7f4c6c15d5d0 0x7f4c6c0cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:56.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.482+0000 7f4c6b400640 1 -- 172.21.15.96:0/2794126883 >> v2:172.21.15.123:3300/0 conn(0x7f4c6c15d5d0 msgr2=0x7f4c6c0cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:21:56.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.482+0000 7f4c6b400640 1 --2- 172.21.15.96:0/2794126883 >> v2:172.21.15.123:3300/0 conn(0x7f4c6c15d5d0 0x7f4c6c0cf950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:56.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.482+0000 7f4c6b400640 1 -- 172.21.15.96:0/2794126883 >> v1:172.21.15.123:6789/0 conn(0x7f4c6c0d0640 legacy=0x7f4c6c0d0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:21:56.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.482+0000 7f4c6b400640 1 -- 172.21.15.96:0/2794126883 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c6c0ad8d0 con 0x7f4c6c0b32c0 2024-05-28T06:21:56.478 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.483+0000 7f4c6b400640 1 --2- 172.21.15.96:0/2794126883 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4c6c0b32c0 0x7f4c6c0b3700 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7f4c6c074120 tx=0x7f4c6402a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fc50f29a5b952cb server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:21:56.478 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.483+0000 7f4c69600640 1 -- 172.21.15.96:0/2794126883 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c64036070 con 0x7f4c6c0b32c0 2024-05-28T06:21:56.478 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.483+0000 7f4c69600640 1 -- 172.21.15.96:0/2794126883 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c64031600 con 0x7f4c6c0b32c0 2024-05-28T06:21:56.478 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.483+0000 7f4c69600640 1 -- 172.21.15.96:0/2794126883 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c640319d0 con 0x7f4c6c0b32c0 2024-05-28T06:21:56.479 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.483+0000 7f4c71a00640 1 -- 172.21.15.96:0/2794126883 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4c6c0b32c0 msgr2=0x7f4c6c0b3700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:56.479 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.483+0000 7f4c71a00640 1 --2- 172.21.15.96:0/2794126883 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4c6c0b32c0 0x7f4c6c0b3700 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7f4c6c074120 tx=0x7f4c6402a660 comp rx=0 tx=0).stop 2024-05-28T06:21:56.479 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.483+0000 7f4c71a00640 1 -- 172.21.15.96:0/2794126883 shutdown_connections 2024-05-28T06:21:56.480 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.483+0000 7f4c71a00640 1 --2- 172.21.15.96:0/2794126883 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4c6c0b32c0 0x7f4c6c0b3700 unknown :-1 s=CLOSED pgs=1705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:56.480 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.483+0000 7f4c71a00640 1 --2- 172.21.15.96:0/2794126883 >> v2:172.21.15.123:3300/0 conn(0x7f4c6c15d5d0 0x7f4c6c0cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:56.480 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.483+0000 7f4c71a00640 1 -- 172.21.15.96:0/2794126883 >> 172.21.15.96:0/2794126883 conn(0x7f4c6c06bd20 msgr2=0x7f4c6c06c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:21:56.480 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.483+0000 7f4c71a00640 1 -- 172.21.15.96:0/2794126883 shutdown_connections 2024-05-28T06:21:56.480 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.483+0000 7f4c71a00640 1 -- 172.21.15.96:0/2794126883 wait complete. 2024-05-28T06:21:56.480 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.486+0000 7f4c71a00640 1 Processor -- start 2024-05-28T06:21:56.481 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.486+0000 7f4c71a00640 1 -- start start 2024-05-28T06:21:56.483 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.487+0000 7f4c71a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4c6c0b32c0 0x7f4c6c14e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:56.483 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.487+0000 7f4c71a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4c6c0d0640 0x7f4c6c14c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:56.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.487+0000 7f4c71a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4c6c15d5d0 0x7f4c6c1402f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:56.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.487+0000 7f4c71a00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4c6c15da50 con 0x7f4c6c0b32c0 2024-05-28T06:21:56.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.487+0000 7f4c71a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f4c6c15d0b0 con 0x7f4c6c0d0640 2024-05-28T06:21:56.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.487+0000 7f4c71a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f4c6c0ad2a0 con 0x7f4c6c15d5d0 2024-05-28T06:21:56.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.487+0000 7f4c6aa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4c6c0b32c0 0x7f4c6c14e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:56.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.487+0000 7f4c6aa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4c6c0b32c0 0x7f4c6c14e6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47952/0 (socket says 172.21.15.96:47952) 2024-05-28T06:21:56.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.488+0000 7f4c6aa00640 1 -- 172.21.15.96:0/3517956891 learned_addr learned my addr 172.21.15.96:0/3517956891 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:21:56.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.488+0000 7f4c6b400640 1 --2- 172.21.15.96:0/3517956891 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4c6c15d5d0 0x7f4c6c1402f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:56.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.488+0000 7f4c6a000640 1 --2- 172.21.15.96:0/3517956891 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4c6c0d0640 0x7f4c6c14c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:56.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.488+0000 7f4c6aa00640 1 -- 172.21.15.96:0/3517956891 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4c6c15d5d0 msgr2=0x7f4c6c1402f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:56.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.488+0000 7f4c6aa00640 1 --2- 172.21.15.96:0/3517956891 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4c6c15d5d0 0x7f4c6c1402f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:56.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.488+0000 7f4c6aa00640 1 -- 172.21.15.96:0/3517956891 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4c6c0d0640 msgr2=0x7f4c6c14c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:56.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.488+0000 7f4c6aa00640 1 --2- 172.21.15.96:0/3517956891 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4c6c0d0640 0x7f4c6c14c0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:56.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.488+0000 7f4c6aa00640 1 -- 172.21.15.96:0/3517956891 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c6c14ca30 con 0x7f4c6c0b32c0 2024-05-28T06:21:56.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.488+0000 7f4c6a000640 1 --2- 172.21.15.96:0/3517956891 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4c6c0d0640 0x7f4c6c14c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:21:56.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.488+0000 7f4c6aa00640 1 --2- 172.21.15.96:0/3517956891 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4c6c0b32c0 0x7f4c6c14e6d0 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7f4c5c00b3f0 tx=0x7f4c5c00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:21:56.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.489+0000 7f4c62a00640 1 -- 172.21.15.96:0/3517956891 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c5c015020 con 0x7f4c6c0b32c0 2024-05-28T06:21:56.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.489+0000 7f4c71a00640 1 -- 172.21.15.96:0/3517956891 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c6c14cd10 con 0x7f4c6c0b32c0 2024-05-28T06:21:56.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.489+0000 7f4c62a00640 1 -- 172.21.15.96:0/3517956891 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c5c011830 con 0x7f4c6c0b32c0 2024-05-28T06:21:56.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.489+0000 7f4c62a00640 1 -- 172.21.15.96:0/3517956891 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c5c011c20 con 0x7f4c6c0b32c0 2024-05-28T06:21:56.486 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.489+0000 7f4c71a00640 1 -- 172.21.15.96:0/3517956891 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c6c14d2d0 con 0x7f4c6c0b32c0 2024-05-28T06:21:56.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.490+0000 7f4c71a00640 1 -- 172.21.15.96:0/3517956891 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c6c0689e0 con 0x7f4c6c0b32c0 2024-05-28T06:21:56.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.491+0000 7f4c62a00640 1 -- 172.21.15.96:0/3517956891 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 36) ==== 93080+0+0 (secure 0 0 0) 0x7f4c5c018020 con 0x7f4c6c0b32c0 2024-05-28T06:21:56.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.492+0000 7f4c62a00640 1 --2- 172.21.15.96:0/3517956891 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4c4806df70 0x7f4c48070430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:56.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.492+0000 7f4c62a00640 1 -- 172.21.15.96:0/3517956891 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 5765+0+0 (secure 0 0 0) 0x7f4c5c08a4b0 con 0x7f4c6c0b32c0 2024-05-28T06:21:56.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.493+0000 7f4c6a000640 1 --2- 172.21.15.96:0/3517956891 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4c4806df70 0x7f4c48070430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:56.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.493+0000 7f4c6a000640 1 --2- 172.21.15.96:0/3517956891 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4c4806df70 0x7f4c48070430 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f4c6c0cf410 tx=0x7f4c54008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:21:56.490 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.496+0000 7f4c62a00640 1 -- 172.21.15.96:0/3517956891 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4c5c01f030 con 0x7f4c6c0b32c0 2024-05-28T06:21:56.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.697+0000 7f4c71a00640 1 -- 172.21.15.96:0/3517956891 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f4c6c071ef0 con 0x7f4c6c0b32c0 2024-05-28T06:21:56.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.697+0000 7f4c62a00640 1 -- 172.21.15.96:0/3517956891 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v103) ==== 93+0+10 (secure 0 0 0) 0x7f4c5c0172c0 con 0x7f4c6c0b32c0 2024-05-28T06:21:56.692 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 1 2024-05-28T06:21:56.696 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.700+0000 7f4c71a00640 1 -- 172.21.15.96:0/3517956891 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4c4806df70 msgr2=0x7f4c48070430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:56.696 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.700+0000 7f4c71a00640 1 --2- 172.21.15.96:0/3517956891 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4c4806df70 0x7f4c48070430 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f4c6c0cf410 tx=0x7f4c54008040 comp rx=0 tx=0).stop 2024-05-28T06:21:56.696 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.700+0000 7f4c71a00640 1 -- 172.21.15.96:0/3517956891 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4c6c0b32c0 msgr2=0x7f4c6c14e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:56.696 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.700+0000 7f4c71a00640 1 --2- 172.21.15.96:0/3517956891 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4c6c0b32c0 0x7f4c6c14e6d0 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7f4c5c00b3f0 tx=0x7f4c5c00b8b0 comp rx=0 tx=0).stop 2024-05-28T06:21:56.696 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.700+0000 7f4c71a00640 1 -- 172.21.15.96:0/3517956891 shutdown_connections 2024-05-28T06:21:56.696 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.700+0000 7f4c71a00640 1 --2- 172.21.15.96:0/3517956891 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4c4806df70 0x7f4c48070430 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:56.696 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.700+0000 7f4c71a00640 1 --2- 172.21.15.96:0/3517956891 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4c6c15d5d0 0x7f4c6c1402f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:56.696 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.700+0000 7f4c71a00640 1 --2- 172.21.15.96:0/3517956891 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4c6c0d0640 0x7f4c6c14c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:56.696 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.700+0000 7f4c71a00640 1 --2- 172.21.15.96:0/3517956891 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4c6c0b32c0 0x7f4c6c14e6d0 unknown :-1 s=CLOSED pgs=1706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:56.696 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.701+0000 7f4c71a00640 1 -- 172.21.15.96:0/3517956891 >> 172.21.15.96:0/3517956891 conn(0x7f4c6c06bd20 msgr2=0x7f4c6c154b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:21:56.696 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.701+0000 7f4c71a00640 1 -- 172.21.15.96:0/3517956891 shutdown_connections 2024-05-28T06:21:56.696 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.701+0000 7f4c71a00640 1 -- 172.21.15.96:0/3517956891 wait complete. 2024-05-28T06:21:56.714 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-28T06:21:56.919 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.923+0000 7f844dc00640 1 Processor -- start 2024-05-28T06:21:56.919 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.924+0000 7f844dc00640 1 -- start start 2024-05-28T06:21:56.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.925+0000 7f844dc00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8448162480 0x7f8448160280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:56.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.925+0000 7f844dc00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f844806ca10 0x7f8448162860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:56.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.925+0000 7f844dc00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8448162da0 con 0x7f844809b5d0 2024-05-28T06:21:56.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.925+0000 7f844dc00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f844806d1f0 con 0x7f844806ca10 2024-05-28T06:21:56.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.925+0000 7f844dc00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f844806d4d0 con 0x7f8448162480 2024-05-28T06:21:56.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.926+0000 7f8447400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f844809b5d0 0x7f84480d0dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:59644/0 (socket says 172.21.15.96:59644) 2024-05-28T06:21:56.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.926+0000 7f8447400640 1 -- 172.21.15.96:0/2952897393 learned_addr learned my addr 172.21.15.96:0/2952897393 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:21:56.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.926+0000 7f8446a00640 1 --2- 172.21.15.96:0/2952897393 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8448162480 0x7f8448160280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:56.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.926+0000 7f843e000640 1 --2- 172.21.15.96:0/2952897393 >> v2:172.21.15.123:3300/0 conn(0x7f844806ca10 0x7f8448162860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:56.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.926+0000 7f8446000640 1 -- 172.21.15.96:0/2952897393 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2432111990 0 0) 0x7f8448162da0 con 0x7f844809b5d0 2024-05-28T06:21:56.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.926+0000 7f8446000640 1 -- 172.21.15.96:0/2952897393 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f842c003520 con 0x7f844809b5d0 2024-05-28T06:21:56.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.926+0000 7f8446000640 1 -- 172.21.15.96:0/2952897393 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1307093475 0 0) 0x7f842c003520 con 0x7f844809b5d0 2024-05-28T06:21:56.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.926+0000 7f8446000640 1 -- 172.21.15.96:0/2952897393 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8448162480 msgr2=0x7f8448160280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:21:56.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.926+0000 7f8446000640 1 --2- 172.21.15.96:0/2952897393 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8448162480 0x7f8448160280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:56.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.926+0000 7f8446000640 1 -- 172.21.15.96:0/2952897393 >> v2:172.21.15.123:3300/0 conn(0x7f844806ca10 msgr2=0x7f8448162860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:21:56.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.926+0000 7f8446000640 1 --2- 172.21.15.96:0/2952897393 >> v2:172.21.15.123:3300/0 conn(0x7f844806ca10 0x7f8448162860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:56.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.926+0000 7f8446000640 1 -- 172.21.15.96:0/2952897393 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f844806d7b0 con 0x7f844809b5d0 2024-05-28T06:21:56.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.926+0000 7f8446000640 1 -- 172.21.15.96:0/2952897393 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f84400019f0 con 0x7f844809b5d0 2024-05-28T06:21:56.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.926+0000 7f8446000640 1 -- 172.21.15.96:0/2952897393 >> v1:172.21.15.96:6789/0 conn(0x7f844809b5d0 legacy=0x7f84480d0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:56.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.926+0000 7f8446000640 1 --2- 172.21.15.96:0/2952897393 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f842c0039b0 0x7f842c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:56.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.926+0000 7f8446000640 1 --2- 172.21.15.96:0/2952897393 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f842c004800 0x7f842c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:56.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.927+0000 7f8446000640 1 --2- 172.21.15.96:0/2952897393 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8448162480 0x7f842c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:56.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.927+0000 7f8446000640 1 -- 172.21.15.96:0/2952897393 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f844806d1f0 con 0x7f842c0039b0 2024-05-28T06:21:56.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.927+0000 7f8446000640 1 -- 172.21.15.96:0/2952897393 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f844806d4d0 con 0x7f842c004800 2024-05-28T06:21:56.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.927+0000 7f8446000640 1 -- 172.21.15.96:0/2952897393 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f842c00bac0 con 0x7f8448162480 2024-05-28T06:21:56.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.927+0000 7f8446a00640 1 --2- 172.21.15.96:0/2952897393 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f842c0039b0 0x7f842c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:56.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.927+0000 7f843e000640 1 --2- 172.21.15.96:0/2952897393 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f842c004800 0x7f842c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:56.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.927+0000 7f8447400640 1 --2- 172.21.15.96:0/2952897393 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8448162480 0x7f842c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:56.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.927+0000 7f8446a00640 1 -- 172.21.15.96:0/2952897393 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8448162480 msgr2=0x7f842c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:21:56.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.927+0000 7f8446a00640 1 --2- 172.21.15.96:0/2952897393 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8448162480 0x7f842c0051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:56.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.927+0000 7f8446a00640 1 -- 172.21.15.96:0/2952897393 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f842c004800 msgr2=0x7f842c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:21:56.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.927+0000 7f8446a00640 1 --2- 172.21.15.96:0/2952897393 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f842c004800 0x7f842c004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:56.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.927+0000 7f8446a00640 1 -- 172.21.15.96:0/2952897393 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f842c003520 con 0x7f842c0039b0 2024-05-28T06:21:56.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.928+0000 7f8446a00640 1 --2- 172.21.15.96:0/2952897393 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f842c0039b0 0x7f842c003dd0 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7f843800ba00 tx=0x7f842c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6e1ff62230654d2 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:21:56.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.929+0000 7f8446000640 1 -- 172.21.15.96:0/2952897393 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8438030070 con 0x7f842c0039b0 2024-05-28T06:21:56.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.929+0000 7f8446000640 1 -- 172.21.15.96:0/2952897393 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f843802ad80 con 0x7f842c0039b0 2024-05-28T06:21:56.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.929+0000 7f8446000640 1 -- 172.21.15.96:0/2952897393 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f843800b600 con 0x7f842c0039b0 2024-05-28T06:21:56.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.929+0000 7f844dc00640 1 -- 172.21.15.96:0/2952897393 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f842c0039b0 msgr2=0x7f842c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:56.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.929+0000 7f844dc00640 1 --2- 172.21.15.96:0/2952897393 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f842c0039b0 0x7f842c003dd0 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7f843800ba00 tx=0x7f842c005830 comp rx=0 tx=0).stop 2024-05-28T06:21:56.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.929+0000 7f8447400640 1 -- 172.21.15.96:0/2952897393 reap_dead start 2024-05-28T06:21:56.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.929+0000 7f844dc00640 1 -- 172.21.15.96:0/2952897393 shutdown_connections 2024-05-28T06:21:56.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.929+0000 7f844dc00640 1 -- 172.21.15.96:0/2952897393 >> 172.21.15.96:0/2952897393 conn(0x7f84480d2430 msgr2=0x7f84480d2840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:21:56.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.929+0000 7f844dc00640 1 -- 172.21.15.96:0/2952897393 shutdown_connections 2024-05-28T06:21:56.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.929+0000 7f844dc00640 1 -- 172.21.15.96:0/2952897393 wait complete. 2024-05-28T06:21:56.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.931+0000 7f844dc00640 1 Processor -- start 2024-05-28T06:21:56.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.931+0000 7f844dc00640 1 -- start start 2024-05-28T06:21:56.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.932+0000 7f844dc00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f842c0039b0 0x7f844815ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:56.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.932+0000 7f844dc00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f844809de80 0x7f844809e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:56.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.932+0000 7f844dc00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f84480b7070 0x7f84480b74f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:56.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.932+0000 7f844dc00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f844809ba60 con 0x7f844809de80 2024-05-28T06:21:56.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.932+0000 7f844dc00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f84480d08a0 con 0x7f842c0039b0 2024-05-28T06:21:56.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.932+0000 7f844dc00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f844806ce70 con 0x7f84480b7070 2024-05-28T06:21:56.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.933+0000 7f843e000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f844809de80 0x7f844809e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:56.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.933+0000 7f843e000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f844809de80 0x7f844809e300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47980/0 (socket says 172.21.15.96:47980) 2024-05-28T06:21:56.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.933+0000 7f843e000640 1 -- 172.21.15.96:0/1578357769 learned_addr learned my addr 172.21.15.96:0/1578357769 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:21:56.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.933+0000 7f8446a00640 1 --2- 172.21.15.96:0/1578357769 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f842c0039b0 0x7f844815ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:56.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.933+0000 7f8447400640 1 --2- 172.21.15.96:0/1578357769 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f84480b7070 0x7f84480b74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:56.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.933+0000 7f843e000640 1 -- 172.21.15.96:0/1578357769 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f84480b7070 msgr2=0x7f84480b74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:56.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.933+0000 7f843e000640 1 --2- 172.21.15.96:0/1578357769 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f84480b7070 0x7f84480b74f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:56.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.933+0000 7f843e000640 1 -- 172.21.15.96:0/1578357769 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f842c0039b0 msgr2=0x7f844815ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:56.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.933+0000 7f843e000640 1 --2- 172.21.15.96:0/1578357769 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f842c0039b0 0x7f844815ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:56.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.933+0000 7f843e000640 1 -- 172.21.15.96:0/1578357769 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f844809e920 con 0x7f844809de80 2024-05-28T06:21:56.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.933+0000 7f843e000640 1 --2- 172.21.15.96:0/1578357769 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f844809de80 0x7f844809e300 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7f843000e810 tx=0x7f843000ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:21:56.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.933+0000 7f843f400640 1 -- 172.21.15.96:0/1578357769 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8430017070 con 0x7f844809de80 2024-05-28T06:21:56.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.934+0000 7f843f400640 1 -- 172.21.15.96:0/1578357769 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f843000c6d0 con 0x7f844809de80 2024-05-28T06:21:56.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.934+0000 7f843f400640 1 -- 172.21.15.96:0/1578357769 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f843000f040 con 0x7f844809de80 2024-05-28T06:21:56.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.934+0000 7f844dc00640 1 -- 172.21.15.96:0/1578357769 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f844809ec00 con 0x7f844809de80 2024-05-28T06:21:56.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.934+0000 7f844dc00640 1 -- 172.21.15.96:0/1578357769 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f844809f1c0 con 0x7f844809de80 2024-05-28T06:21:56.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.935+0000 7f844dc00640 1 -- 172.21.15.96:0/1578357769 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84480b15b0 con 0x7f844809de80 2024-05-28T06:21:56.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.936+0000 7f843f400640 1 -- 172.21.15.96:0/1578357769 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 37) ==== 93182+0+0 (secure 0 0 0) 0x7f843001a070 con 0x7f844809de80 2024-05-28T06:21:56.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.937+0000 7f843f400640 1 --2- 172.21.15.96:0/1578357769 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f842406e0c0 0x7f8424070580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:56.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.937+0000 7f843f400640 1 -- 172.21.15.96:0/1578357769 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 5765+0+0 (secure 0 0 0) 0x7f843008a4e0 con 0x7f844809de80 2024-05-28T06:21:56.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.938+0000 7f8446a00640 1 --2- 172.21.15.96:0/1578357769 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f842406e0c0 0x7f8424070580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:56.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.939+0000 7f8446a00640 1 --2- 172.21.15.96:0/1578357769 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f842406e0c0 0x7f8424070580 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f842c0057e0 tx=0x7f8438030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:21:56.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:56.942+0000 7f843f400640 1 -- 172.21.15.96:0/1578357769 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f84480b15b0 con 0x7f844809de80 2024-05-28T06:21:57.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.173+0000 7f844dc00640 1 -- 172.21.15.96:0/1578357769 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f84480689e0 con 0x7f844809de80 2024-05-28T06:21:57.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.174+0000 7f843f400640 1 -- 172.21.15.96:0/1578357769 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v103) ==== 104+0+11 (secure 0 0 0) 0x7f8430060030 con 0x7f844809de80 2024-05-28T06:21:57.171 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 32 2024-05-28T06:21:57.174 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.179+0000 7f844dc00640 1 -- 172.21.15.96:0/1578357769 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f842406e0c0 msgr2=0x7f8424070580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:57.174 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.179+0000 7f844dc00640 1 --2- 172.21.15.96:0/1578357769 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f842406e0c0 0x7f8424070580 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f842c0057e0 tx=0x7f8438030000 comp rx=0 tx=0).stop 2024-05-28T06:21:57.174 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.179+0000 7f844dc00640 1 -- 172.21.15.96:0/1578357769 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f844809de80 msgr2=0x7f844809e300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:57.174 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.179+0000 7f844dc00640 1 --2- 172.21.15.96:0/1578357769 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f844809de80 0x7f844809e300 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7f843000e810 tx=0x7f843000ecd0 comp rx=0 tx=0).stop 2024-05-28T06:21:57.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.179+0000 7f844dc00640 1 -- 172.21.15.96:0/1578357769 shutdown_connections 2024-05-28T06:21:57.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.179+0000 7f844dc00640 1 --2- 172.21.15.96:0/1578357769 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f842406e0c0 0x7f8424070580 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:57.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.179+0000 7f844dc00640 1 --2- 172.21.15.96:0/1578357769 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f84480b7070 0x7f84480b74f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:57.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.179+0000 7f844dc00640 1 --2- 172.21.15.96:0/1578357769 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f844809de80 0x7f844809e300 secure :-1 s=CLOSED pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7f843000e810 tx=0x7f843000ecd0 comp rx=0 tx=0).stop 2024-05-28T06:21:57.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.179+0000 7f844dc00640 1 --2- 172.21.15.96:0/1578357769 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f842c0039b0 0x7f844815ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:57.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.179+0000 7f844dc00640 1 -- 172.21.15.96:0/1578357769 >> 172.21.15.96:0/1578357769 conn(0x7f84480d2430 msgr2=0x7f84480c8350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:21:57.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.180+0000 7f844dc00640 1 -- 172.21.15.96:0/1578357769 shutdown_connections 2024-05-28T06:21:57.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.180+0000 7f844dc00640 1 -- 172.21.15.96:0/1578357769 wait complete. 2024-05-28T06:21:57.202 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-28T06:21:57.473 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.478+0000 7ff15d600640 1 Processor -- start 2024-05-28T06:21:57.474 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.479+0000 7ff15d600640 1 -- start start 2024-05-28T06:21:57.475 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.481+0000 7ff15d600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7ff158162130 0x7ff1580d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:57.475 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.481+0000 7ff15d600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff1580d02e0 0x7ff1580d0770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:57.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.481+0000 7ff15d600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1580b13d0 con 0x7ff1580d15e0 2024-05-28T06:21:57.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.481+0000 7ff15d600640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7ff1580d33b0 con 0x7ff158162130 2024-05-28T06:21:57.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.481+0000 7ff15d600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ff1580b1640 con 0x7ff1580d02e0 2024-05-28T06:21:57.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.481+0000 7ff157400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff1580d02e0 0x7ff1580d0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:57.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.481+0000 7ff156a00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7ff158162130 0x7ff1580d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:57.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.481+0000 7ff157400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff1580d02e0 0x7ff1580d0770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.96:52888/0 (socket says 172.21.15.96:52888) 2024-05-28T06:21:57.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.481+0000 7ff157400640 1 -- 172.21.15.96:0/5714374 learned_addr learned my addr 172.21.15.96:0/5714374 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:21:57.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.482+0000 7ff155600640 1 -- 172.21.15.96:0/5714374 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 217710416 0 0) 0x7ff1580b13d0 con 0x7ff1580d15e0 2024-05-28T06:21:57.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.482+0000 7ff155600640 1 -- 172.21.15.96:0/5714374 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff13c003520 con 0x7ff1580d15e0 2024-05-28T06:21:57.478 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.482+0000 7ff156a00640 1 -- 172.21.15.96:0/5714374 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff1580d02e0 msgr2=0x7ff1580d0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:21:57.478 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.483+0000 7ff156a00640 1 --2- 172.21.15.96:0/5714374 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff1580d02e0 0x7ff1580d0770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:57.478 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.483+0000 7ff156a00640 1 -- 172.21.15.96:0/5714374 >> v1:172.21.15.123:6789/0 conn(0x7ff1580d15e0 legacy=0x7ff1580b1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:57.478 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.483+0000 7ff156a00640 1 -- 172.21.15.96:0/5714374 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1580b1920 con 0x7ff158162130 2024-05-28T06:21:57.478 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.483+0000 7ff157400640 1 --2- 172.21.15.96:0/5714374 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff1580d02e0 0x7ff1580d0770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-28T06:21:57.479 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.485+0000 7ff156a00640 1 --2- 172.21.15.96:0/5714374 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff158162130 0x7ff1580d10a0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7ff148009d90 tx=0x7ff14802a680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=da027d0e4aba01ce server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:21:57.480 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.485+0000 7ff155600640 1 -- 172.21.15.96:0/5714374 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff148007740 con 0x7ff158162130 2024-05-28T06:21:57.480 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.485+0000 7ff155600640 1 -- 172.21.15.96:0/5714374 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1480078e0 con 0x7ff158162130 2024-05-28T06:21:57.480 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.485+0000 7ff155600640 1 -- 172.21.15.96:0/5714374 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff148007cf0 con 0x7ff158162130 2024-05-28T06:21:57.481 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.486+0000 7ff15d600640 1 -- 172.21.15.96:0/5714374 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff158162130 msgr2=0x7ff1580d10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:57.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.486+0000 7ff15d600640 1 --2- 172.21.15.96:0/5714374 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff158162130 0x7ff1580d10a0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7ff148009d90 tx=0x7ff14802a680 comp rx=0 tx=0).stop 2024-05-28T06:21:57.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.487+0000 7ff15d600640 1 -- 172.21.15.96:0/5714374 shutdown_connections 2024-05-28T06:21:57.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.487+0000 7ff15d600640 1 --2- 172.21.15.96:0/5714374 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff1580d02e0 0x7ff1580d0770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:57.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.487+0000 7ff15d600640 1 --2- 172.21.15.96:0/5714374 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff158162130 0x7ff1580d10a0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:57.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.487+0000 7ff15d600640 1 -- 172.21.15.96:0/5714374 >> 172.21.15.96:0/5714374 conn(0x7ff15808bfc0 msgr2=0x7ff1580d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:21:57.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.487+0000 7ff15d600640 1 -- 172.21.15.96:0/5714374 shutdown_connections 2024-05-28T06:21:57.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.487+0000 7ff15d600640 1 -- 172.21.15.96:0/5714374 wait complete. 2024-05-28T06:21:57.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.490+0000 7ff15d600640 1 Processor -- start 2024-05-28T06:21:57.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.490+0000 7ff15d600640 1 -- start start 2024-05-28T06:21:57.486 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.491+0000 7ff15d600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff1580d02e0 0x7ff15815bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:57.486 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.492+0000 7ff15d600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff1580d15e0 0x7ff1580b4560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:57.486 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.492+0000 7ff15d600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff158162130 0x7ff1580cc370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:57.486 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.492+0000 7ff15d600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff158162690 con 0x7ff1580d02e0 2024-05-28T06:21:57.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.492+0000 7ff15d600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff158162510 con 0x7ff158162130 2024-05-28T06:21:57.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.492+0000 7ff15d600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ff1580d3690 con 0x7ff1580d15e0 2024-05-28T06:21:57.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.492+0000 7ff156a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff1580d02e0 0x7ff15815bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:57.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.492+0000 7ff156a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff1580d02e0 0x7ff15815bc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47984/0 (socket says 172.21.15.96:47984) 2024-05-28T06:21:57.487 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.492+0000 7ff156a00640 1 -- 172.21.15.96:0/976277830 learned_addr learned my addr 172.21.15.96:0/976277830 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:21:57.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.492+0000 7ff157400640 1 --2- 172.21.15.96:0/976277830 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff158162130 0x7ff1580cc370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:57.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.493+0000 7ff156a00640 1 -- 172.21.15.96:0/976277830 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff1580d15e0 msgr2=0x7ff1580b4560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:57.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.493+0000 7ff156a00640 1 --2- 172.21.15.96:0/976277830 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff1580d15e0 0x7ff1580b4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:57.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.493+0000 7ff156a00640 1 -- 172.21.15.96:0/976277830 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff158162130 msgr2=0x7ff1580cc370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:57.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.493+0000 7ff156a00640 1 --2- 172.21.15.96:0/976277830 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff158162130 0x7ff1580cc370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:57.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.493+0000 7ff156a00640 1 -- 172.21.15.96:0/976277830 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff15815c1a0 con 0x7ff1580d02e0 2024-05-28T06:21:57.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.493+0000 7ff156a00640 1 --2- 172.21.15.96:0/976277830 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff1580d02e0 0x7ff15815bc60 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7ff148009d90 tx=0x7ff148004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:21:57.489 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.493+0000 7ff14ea00640 1 -- 172.21.15.96:0/976277830 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff14803f070 con 0x7ff1580d02e0 2024-05-28T06:21:57.489 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.494+0000 7ff15d600640 1 -- 172.21.15.96:0/976277830 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1580b5000 con 0x7ff1580d02e0 2024-05-28T06:21:57.489 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.494+0000 7ff15d600640 1 -- 172.21.15.96:0/976277830 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1580b55c0 con 0x7ff1580d02e0 2024-05-28T06:21:57.489 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.494+0000 7ff14ea00640 1 -- 172.21.15.96:0/976277830 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff148031b40 con 0x7ff1580d02e0 2024-05-28T06:21:57.489 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.494+0000 7ff14ea00640 1 -- 172.21.15.96:0/976277830 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff148044460 con 0x7ff1580d02e0 2024-05-28T06:21:57.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.496+0000 7ff15d600640 1 -- 172.21.15.96:0/976277830 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff130005290 con 0x7ff1580d02e0 2024-05-28T06:21:57.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.497+0000 7ff14ea00640 1 -- 172.21.15.96:0/976277830 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 37) ==== 93182+0+0 (secure 0 0 0) 0x7ff14803b410 con 0x7ff1580d02e0 2024-05-28T06:21:57.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.499+0000 7ff14ea00640 1 --2- 172.21.15.96:0/976277830 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff13406e090 0x7ff134070550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:57.494 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.499+0000 7ff14ea00640 1 -- 172.21.15.96:0/976277830 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 5765+0+0 (secure 0 0 0) 0x7ff1480abff0 con 0x7ff1580d02e0 2024-05-28T06:21:57.495 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.500+0000 7ff156000640 1 --2- 172.21.15.96:0/976277830 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff13406e090 0x7ff134070550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:57.496 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.502+0000 7ff156000640 1 --2- 172.21.15.96:0/976277830 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff13406e090 0x7ff134070550 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7ff1400010e0 tx=0x7ff140007950 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:21:57.501 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.506+0000 7ff14ea00640 1 -- 172.21.15.96:0/976277830 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff15815c1a0 con 0x7ff1580d02e0 2024-05-28T06:21:57.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.706+0000 7ff15d600640 1 -- 172.21.15.96:0/976277830 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7ff130005850 con 0x7ff1580d02e0 2024-05-28T06:21:57.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.707+0000 7ff14ea00640 1 -- 172.21.15.96:0/976277830 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v103) ==== 100+0+11 (secure 0 0 0) 0x7ff1480820f0 con 0x7ff1580d02e0 2024-05-28T06:21:57.702 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 64 2024-05-28T06:21:57.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.709+0000 7ff15d600640 1 -- 172.21.15.96:0/976277830 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff13406e090 msgr2=0x7ff134070550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:57.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.709+0000 7ff15d600640 1 --2- 172.21.15.96:0/976277830 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff13406e090 0x7ff134070550 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7ff1400010e0 tx=0x7ff140007950 comp rx=0 tx=0).stop 2024-05-28T06:21:57.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.709+0000 7ff15d600640 1 -- 172.21.15.96:0/976277830 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff1580d02e0 msgr2=0x7ff15815bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:57.705 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.709+0000 7ff15d600640 1 --2- 172.21.15.96:0/976277830 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff1580d02e0 0x7ff15815bc60 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7ff148009d90 tx=0x7ff148004a40 comp rx=0 tx=0).stop 2024-05-28T06:21:57.705 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.709+0000 7ff15d600640 1 -- 172.21.15.96:0/976277830 shutdown_connections 2024-05-28T06:21:57.705 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.709+0000 7ff15d600640 1 --2- 172.21.15.96:0/976277830 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff13406e090 0x7ff134070550 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:57.705 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.709+0000 7ff15d600640 1 --2- 172.21.15.96:0/976277830 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff158162130 0x7ff1580cc370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:57.705 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.709+0000 7ff15d600640 1 --2- 172.21.15.96:0/976277830 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff1580d15e0 0x7ff1580b4560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:57.705 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.709+0000 7ff15d600640 1 --2- 172.21.15.96:0/976277830 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff1580d02e0 0x7ff15815bc60 unknown :-1 s=CLOSED pgs=1710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:57.705 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.709+0000 7ff15d600640 1 -- 172.21.15.96:0/976277830 >> 172.21.15.96:0/976277830 conn(0x7ff15808bfc0 msgr2=0x7ff15815e3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:21:57.705 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.710+0000 7ff15d600640 1 -- 172.21.15.96:0/976277830 shutdown_connections 2024-05-28T06:21:57.705 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.710+0000 7ff15d600640 1 -- 172.21.15.96:0/976277830 wait complete. 2024-05-28T06:21:57.722 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-28T06:21:57.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.918+0000 7fe9b8c00640 1 Processor -- start 2024-05-28T06:21:57.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.918+0000 7fe9b8c00640 1 -- start start 2024-05-28T06:21:57.914 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.920+0000 7fe9b8c00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fe9b406ee40 0x7fe9b40ad230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:57.914 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.920+0000 7fe9b8c00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9b4156f10 con 0x7fe9b4156b30 2024-05-28T06:21:57.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.920+0000 7fe9b8c00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9b406dda0 con 0x7fe9b406f2f0 2024-05-28T06:21:57.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.920+0000 7fe9b8c00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fe9b406e080 con 0x7fe9b406ee40 2024-05-28T06:21:57.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.920+0000 7fe9b3400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fe9b4156b30 0x7fe9b40af6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:59656/0 (socket says 172.21.15.96:59656) 2024-05-28T06:21:57.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.920+0000 7fe9b3400640 1 -- 172.21.15.96:0/1486559715 learned_addr learned my addr 172.21.15.96:0/1486559715 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:21:57.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.920+0000 7fe9abe00640 1 --2- 172.21.15.96:0/1486559715 >> v2:172.21.15.96:3300/0 conn(0x7fe9b406ee40 0x7fe9b40ad230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:57.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.920+0000 7fe9b2a00640 1 -- 172.21.15.96:0/1486559715 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 408335545 0 0) 0x7fe9b4156f10 con 0x7fe9b4156b30 2024-05-28T06:21:57.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.920+0000 7fe9b2a00640 1 -- 172.21.15.96:0/1486559715 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe998003520 con 0x7fe9b4156b30 2024-05-28T06:21:57.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.921+0000 7fe9abe00640 1 -- 172.21.15.96:0/1486559715 >> v1:172.21.15.123:6789/0 conn(0x7fe9b406f2f0 legacy=0x7fe9b40b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:21:57.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.921+0000 7fe9abe00640 1 -- 172.21.15.96:0/1486559715 >> v1:172.21.15.96:6789/0 conn(0x7fe9b4156b30 legacy=0x7fe9b40af6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:57.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.921+0000 7fe9abe00640 1 -- 172.21.15.96:0/1486559715 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9b406e360 con 0x7fe9b406ee40 2024-05-28T06:21:57.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.922+0000 7fe9abe00640 1 --2- 172.21.15.96:0/1486559715 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe9b406ee40 0x7fe9b40ad230 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7fe99c0097a0 tx=0x7fe99c02a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff36e11e5b13863d server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:21:57.917 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.922+0000 7fe9b2a00640 1 -- 172.21.15.96:0/1486559715 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe99c037070 con 0x7fe9b406ee40 2024-05-28T06:21:57.917 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.922+0000 7fe9b2a00640 1 -- 172.21.15.96:0/1486559715 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe99c02ae50 con 0x7fe9b406ee40 2024-05-28T06:21:57.917 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.922+0000 7fe9b2a00640 1 -- 172.21.15.96:0/1486559715 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe99c032640 con 0x7fe9b406ee40 2024-05-28T06:21:57.917 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.922+0000 7fe9b8c00640 1 -- 172.21.15.96:0/1486559715 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe9b406ee40 msgr2=0x7fe9b40ad230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:57.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.922+0000 7fe9b8c00640 1 --2- 172.21.15.96:0/1486559715 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe9b406ee40 0x7fe9b40ad230 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7fe99c0097a0 tx=0x7fe99c02a140 comp rx=0 tx=0).stop 2024-05-28T06:21:57.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.923+0000 7fe9b8c00640 1 -- 172.21.15.96:0/1486559715 shutdown_connections 2024-05-28T06:21:57.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.923+0000 7fe9b8c00640 1 --2- 172.21.15.96:0/1486559715 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe9b406ee40 0x7fe9b40ad230 unknown :-1 s=CLOSED pgs=1712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:57.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.923+0000 7fe9b8c00640 1 -- 172.21.15.96:0/1486559715 >> 172.21.15.96:0/1486559715 conn(0x7fe9b40b1fc0 msgr2=0x7fe9b40b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:21:57.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.923+0000 7fe9b8c00640 1 -- 172.21.15.96:0/1486559715 shutdown_connections 2024-05-28T06:21:57.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.923+0000 7fe9b8c00640 1 -- 172.21.15.96:0/1486559715 wait complete. 2024-05-28T06:21:57.919 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.924+0000 7fe9b8c00640 1 Processor -- start 2024-05-28T06:21:57.919 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.925+0000 7fe9b8c00640 1 -- start start 2024-05-28T06:21:57.920 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.925+0000 7fe9b8c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe9b406ee40 0x7fe9b414c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:57.920 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.925+0000 7fe9b8c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe9b406f2f0 0x7fe9b41599a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:57.920 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.926+0000 7fe9b8c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9b4156b30 0x7fe9b414faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:57.921 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.926+0000 7fe9b8c00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe9b40b2db0 con 0x7fe9b406f2f0 2024-05-28T06:21:57.921 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.926+0000 7fe9b8c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe9b40b2c30 con 0x7fe9b4156b30 2024-05-28T06:21:57.921 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.926+0000 7fe9b8c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fe9b40b39d0 con 0x7fe9b406ee40 2024-05-28T06:21:57.921 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.926+0000 7fe9abe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe9b406f2f0 0x7fe9b41599a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:57.921 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.926+0000 7fe9abe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe9b406f2f0 0x7fe9b41599a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47996/0 (socket says 172.21.15.96:47996) 2024-05-28T06:21:57.921 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.926+0000 7fe9abe00640 1 -- 172.21.15.96:0/3473045966 learned_addr learned my addr 172.21.15.96:0/3473045966 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:21:57.922 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.926+0000 7fe9b3400640 1 --2- 172.21.15.96:0/3473045966 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe9b406ee40 0x7fe9b414c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:57.922 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.926+0000 7fe9abe00640 1 -- 172.21.15.96:0/3473045966 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe9b406ee40 msgr2=0x7fe9b414c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:57.922 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.926+0000 7fe9abe00640 1 --2- 172.21.15.96:0/3473045966 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe9b406ee40 0x7fe9b414c090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:57.922 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.926+0000 7fe9abe00640 1 -- 172.21.15.96:0/3473045966 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9b4156b30 msgr2=0x7fe9b414faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:57.922 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.926+0000 7fe9abe00640 1 --2- 172.21.15.96:0/3473045966 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9b4156b30 0x7fe9b414faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:57.922 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.926+0000 7fe9abe00640 1 -- 172.21.15.96:0/3473045966 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9b414c5d0 con 0x7fe9b406f2f0 2024-05-28T06:21:57.922 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.927+0000 7fe9abe00640 1 --2- 172.21.15.96:0/3473045966 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe9b406f2f0 0x7fe9b41599a0 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7fe99c000c00 tx=0x7fe99c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:21:57.922 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.927+0000 7fe9ab400640 1 -- 172.21.15.96:0/3473045966 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe99c041070 con 0x7fe9b406f2f0 2024-05-28T06:21:57.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.927+0000 7fe9ab400640 1 -- 172.21.15.96:0/3473045966 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe99c03c460 con 0x7fe9b406f2f0 2024-05-28T06:21:57.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.927+0000 7fe9b8c00640 1 -- 172.21.15.96:0/3473045966 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9b4158900 con 0x7fe9b406f2f0 2024-05-28T06:21:57.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.927+0000 7fe9ab400640 1 -- 172.21.15.96:0/3473045966 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe99c037040 con 0x7fe9b406f2f0 2024-05-28T06:21:57.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.927+0000 7fe9b8c00640 1 -- 172.21.15.96:0/3473045966 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9b4158ec0 con 0x7fe9b406f2f0 2024-05-28T06:21:57.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.928+0000 7fe9b8c00640 1 -- 172.21.15.96:0/3473045966 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9b40689e0 con 0x7fe9b406f2f0 2024-05-28T06:21:57.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.929+0000 7fe9ab400640 1 -- 172.21.15.96:0/3473045966 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 37) ==== 93182+0+0 (secure 0 0 0) 0x7fe99c044070 con 0x7fe9b406f2f0 2024-05-28T06:21:57.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.930+0000 7fe9ab400640 1 --2- 172.21.15.96:0/3473045966 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe99006e090 0x7fe990070550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:57.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.930+0000 7fe9ab400640 1 -- 172.21.15.96:0/3473045966 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 5765+0+0 (secure 0 0 0) 0x7fe99c0ab1e0 con 0x7fe9b406f2f0 2024-05-28T06:21:57.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.931+0000 7fe9b3400640 1 --2- 172.21.15.96:0/3473045966 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe99006e090 0x7fe990070550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:57.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.932+0000 7fe9b3400640 1 --2- 172.21.15.96:0/3473045966 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe99006e090 0x7fe990070550 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fe9b40cf930 tx=0x7fe9a4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:21:57.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:57.935+0000 7fe9ab400640 1 -- 172.21.15.96:0/3473045966 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe9b414c5d0 con 0x7fe9b406f2f0 2024-05-28T06:21:58.132 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:58.137+0000 7fe9b8c00640 1 -- 172.21.15.96:0/3473045966 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fe9b415fb30 con 0x7fe9b406f2f0 2024-05-28T06:21:58.132 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:58.138+0000 7fe9ab400640 1 -- 172.21.15.96:0/3473045966 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v103) ==== 80+0+0 (secure 0 0 0) 0x7fe99c07b190 con 0x7fe9b406f2f0 2024-05-28T06:21:58.133 INFO:teuthology.orchestra.run.smithi096.stderr:listed 0 entries 2024-05-28T06:21:58.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:58.140+0000 7fe9b8c00640 1 -- 172.21.15.96:0/3473045966 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe99006e090 msgr2=0x7fe990070550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:58.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:58.140+0000 7fe9b8c00640 1 --2- 172.21.15.96:0/3473045966 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe99006e090 0x7fe990070550 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fe9b40cf930 tx=0x7fe9a4009040 comp rx=0 tx=0).stop 2024-05-28T06:21:58.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:58.140+0000 7fe9b8c00640 1 -- 172.21.15.96:0/3473045966 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe9b406f2f0 msgr2=0x7fe9b41599a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:58.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:58.140+0000 7fe9b8c00640 1 --2- 172.21.15.96:0/3473045966 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe9b406f2f0 0x7fe9b41599a0 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7fe99c000c00 tx=0x7fe99c0046c0 comp rx=0 tx=0).stop 2024-05-28T06:21:58.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:58.141+0000 7fe9b8c00640 1 -- 172.21.15.96:0/3473045966 shutdown_connections 2024-05-28T06:21:58.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:58.141+0000 7fe9b8c00640 1 --2- 172.21.15.96:0/3473045966 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe99006e090 0x7fe990070550 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:58.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:58.141+0000 7fe9b8c00640 1 --2- 172.21.15.96:0/3473045966 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9b4156b30 0x7fe9b414faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:58.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:58.141+0000 7fe9b8c00640 1 --2- 172.21.15.96:0/3473045966 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe9b406f2f0 0x7fe9b41599a0 unknown :-1 s=CLOSED pgs=1713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:58.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:58.141+0000 7fe9b8c00640 1 --2- 172.21.15.96:0/3473045966 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe9b406ee40 0x7fe9b414c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:58.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:58.141+0000 7fe9b8c00640 1 -- 172.21.15.96:0/3473045966 >> 172.21.15.96:0/3473045966 conn(0x7fe9b40b1fc0 msgr2=0x7fe9b40b3ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:21:58.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:58.141+0000 7fe9b8c00640 1 -- 172.21.15.96:0/3473045966 shutdown_connections 2024-05-28T06:21:58.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:58.141+0000 7fe9b8c00640 1 -- 172.21.15.96:0/3473045966 wait complete. 2024-05-28T06:21:58.155 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi096.front.sepia.ceph.com... 2024-05-28T06:21:58.155 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:21:58.155 DEBUG:teuthology.orchestra.run.smithi096:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-05-28T06:21:58.231 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi096.front.sepia.ceph.com 2024-05-28T06:21:58.232 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:21:58.232 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-28T06:21:58.237 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi096.stderr:ceph-fuse[75861]: fuse finished with error 0 and tester_r 0 2024-05-28T06:21:58.329 INFO:tasks.cephfs.mount:Terminating background process 2024-05-28T06:21:58.371 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-05-28T06:21:58.371 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:21:58.414 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052731,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248996k,nr_inodes=812249,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-05-28T06:21:58.414 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi123.front.sepia.ceph.com 2024-05-28T06:21:58.414 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:21:58.414 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-05-28T06:21:58.505 INFO:teuthology.orchestra.run.smithi123.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-05-28T06:21:58.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-28T06:21:58.506 INFO:tasks.cephfs.mount:Terminating background process 2024-05-28T06:21:58.506 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:21:58.507 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:21:58.560 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052731,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248996k,nr_inodes=812249,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-28T06:21:58.561 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi096.front.sepia.ceph.com 2024-05-28T06:21:58.561 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:21:58.561 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-05-28T06:21:58.616 INFO:teuthology.orchestra.run.smithi096.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-05-28T06:21:58.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-28T06:21:58.617 INFO:tasks.cephfs.mount:Terminating background process 2024-05-28T06:21:58.617 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-05-28T06:21:58.618 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:21:58.676 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052731,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248996k,nr_inodes=812249,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-05-28T06:21:58.676 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi123.front.sepia.ceph.com 2024-05-28T06:21:58.676 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:21:58.676 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-05-28T06:21:58.731 INFO:teuthology.orchestra.run.smithi123.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-05-28T06:21:58.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-28T06:21:58.732 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-28T06:21:58.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:58.946+0000 7fa77e200640 1 Processor -- start 2024-05-28T06:21:58.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:58.946+0000 7fa77e200640 1 -- start start 2024-05-28T06:21:58.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:58.948+0000 7fa77e200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fa778156b30 0x7fa7780af6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:58.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:58.948+0000 7fa77e200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa77806ee40 0x7fa7780ad230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:58.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:58.948+0000 7fa77e200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fa77806f2c0 0x7fa7780ad770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:58.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:58.948+0000 7fa77e200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fa778156f10 con 0x7fa778156b30 2024-05-28T06:21:58.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:58.948+0000 7fa77e200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fa77806dda0 con 0x7fa77806f2c0 2024-05-28T06:21:58.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:58.948+0000 7fa77e200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fa77806e080 con 0x7fa77806ee40 2024-05-28T06:21:58.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:58.948+0000 7fa77ce00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fa778156b30 0x7fa7780af6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:58.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:58.948+0000 7fa77ce00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fa778156b30 0x7fa7780af6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35798/0 (socket says 172.21.15.96:35798) 2024-05-28T06:21:58.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:58.948+0000 7fa77ce00640 1 -- 172.21.15.96:0/2255456846 learned_addr learned my addr 172.21.15.96:0/2255456846 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:21:58.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:58.948+0000 7fa77d800640 1 --2- 172.21.15.96:0/2255456846 >> v2:172.21.15.123:3300/0 conn(0x7fa77806f2c0 0x7fa7780ad770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:58.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:58.948+0000 7fa777e00640 1 --2- 172.21.15.96:0/2255456846 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa77806ee40 0x7fa7780ad230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:58.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:58.948+0000 7fa77ce00640 1 -- 172.21.15.96:0/2255456846 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa77806ee40 msgr2=0x7fa7780ad230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:21:58.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:58.949+0000 7fa77ce00640 1 --2- 172.21.15.96:0/2255456846 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa77806ee40 0x7fa7780ad230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:58.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:58.949+0000 7fa77ce00640 1 -- 172.21.15.96:0/2255456846 >> v2:172.21.15.123:3300/0 conn(0x7fa77806f2c0 msgr2=0x7fa7780ad770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:21:58.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:58.949+0000 7fa77ce00640 1 --2- 172.21.15.96:0/2255456846 >> v2:172.21.15.123:3300/0 conn(0x7fa77806f2c0 0x7fa7780ad770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:58.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:58.949+0000 7fa77ce00640 1 -- 172.21.15.96:0/2255456846 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa77806e360 con 0x7fa778156b30 2024-05-28T06:21:58.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:58.949+0000 7fa77d800640 1 --2- 172.21.15.96:0/2255456846 >> v2:172.21.15.123:3300/0 conn(0x7fa77806f2c0 0x7fa7780ad770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:21:58.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:58.950+0000 7fa77ce00640 1 --2- 172.21.15.96:0/2255456846 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa778156b30 0x7fa7780af6e0 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7fa778071410 tx=0x7fa76c02a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18d6aa4b0731f5cf server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:21:58.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:58.950+0000 7fa777400640 1 -- 172.21.15.96:0/2255456846 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa76c037070 con 0x7fa778156b30 2024-05-28T06:21:58.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:58.950+0000 7fa777400640 1 -- 172.21.15.96:0/2255456846 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa76c032440 con 0x7fa778156b30 2024-05-28T06:21:58.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:58.950+0000 7fa777400640 1 -- 172.21.15.96:0/2255456846 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa76c032810 con 0x7fa778156b30 2024-05-28T06:21:58.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:58.951+0000 7fa77e200640 1 -- 172.21.15.96:0/2255456846 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa778156b30 msgr2=0x7fa7780af6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:58.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:58.951+0000 7fa77e200640 1 --2- 172.21.15.96:0/2255456846 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa778156b30 0x7fa7780af6e0 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7fa778071410 tx=0x7fa76c02a140 comp rx=0 tx=0).stop 2024-05-28T06:21:58.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:58.951+0000 7fa77e200640 1 -- 172.21.15.96:0/2255456846 shutdown_connections 2024-05-28T06:21:58.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:58.951+0000 7fa77e200640 1 --2- 172.21.15.96:0/2255456846 >> v2:172.21.15.123:3300/0 conn(0x7fa77806f2c0 0x7fa7780ad770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:58.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:58.951+0000 7fa77e200640 1 --2- 172.21.15.96:0/2255456846 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa77806ee40 0x7fa7780ad230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:58.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:58.951+0000 7fa77e200640 1 --2- 172.21.15.96:0/2255456846 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa778156b30 0x7fa7780af6e0 unknown :-1 s=CLOSED pgs=1714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:58.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:58.951+0000 7fa77e200640 1 -- 172.21.15.96:0/2255456846 >> 172.21.15.96:0/2255456846 conn(0x7fa7780b1fc0 msgr2=0x7fa7780b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:21:58.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:58.951+0000 7fa77e200640 1 -- 172.21.15.96:0/2255456846 shutdown_connections 2024-05-28T06:21:58.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:58.951+0000 7fa77e200640 1 -- 172.21.15.96:0/2255456846 wait complete. 2024-05-28T06:21:58.953 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:58.953+0000 7fa77e200640 1 Processor -- start 2024-05-28T06:21:58.953 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:58.953+0000 7fa77e200640 1 -- start start 2024-05-28T06:21:58.953 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:58.954+0000 7fa77e200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa77806ee40 0x7fa778073890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:58.953 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:58.954+0000 7fa77e200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa77806f2c0 0x7fa7780b3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:58.953 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:58.954+0000 7fa77e200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa778156b30 0x7fa778158f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:58.954 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:58.954+0000 7fa77e200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa7780afc20 con 0x7fa778156b30 2024-05-28T06:21:58.954 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:58.954+0000 7fa77e200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa7780b2c30 con 0x7fa77806ee40 2024-05-28T06:21:58.954 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:58.954+0000 7fa77e200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fa77815f380 con 0x7fa77806f2c0 2024-05-28T06:21:58.954 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:58.955+0000 7fa77d800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa778156b30 0x7fa778158f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:58.954 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:58.955+0000 7fa77d800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa778156b30 0x7fa778158f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35802/0 (socket says 172.21.15.96:35802) 2024-05-28T06:21:58.954 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:58.955+0000 7fa77d800640 1 -- 172.21.15.96:0/1662111247 learned_addr learned my addr 172.21.15.96:0/1662111247 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:21:58.954 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:58.955+0000 7fa77ce00640 1 --2- 172.21.15.96:0/1662111247 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa77806ee40 0x7fa778073890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:58.954 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:58.955+0000 7fa777e00640 1 --2- 172.21.15.96:0/1662111247 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa77806f2c0 0x7fa7780b3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:58.954 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:58.955+0000 7fa77d800640 1 -- 172.21.15.96:0/1662111247 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa77806f2c0 msgr2=0x7fa7780b3650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:58.954 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:58.955+0000 7fa77d800640 1 --2- 172.21.15.96:0/1662111247 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa77806f2c0 0x7fa7780b3650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:58.954 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:58.955+0000 7fa77d800640 1 -- 172.21.15.96:0/1662111247 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa77806ee40 msgr2=0x7fa778073890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:58.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:58.955+0000 7fa77d800640 1 --2- 172.21.15.96:0/1662111247 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa77806ee40 0x7fa778073890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:58.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:58.955+0000 7fa77d800640 1 -- 172.21.15.96:0/1662111247 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa77815e7a0 con 0x7fa778156b30 2024-05-28T06:21:58.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:58.955+0000 7fa777e00640 1 --2- 172.21.15.96:0/1662111247 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa77806f2c0 0x7fa7780b3650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:21:58.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:58.955+0000 7fa77ce00640 1 --2- 172.21.15.96:0/1662111247 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa77806ee40 0x7fa778073890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-28T06:21:58.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:58.956+0000 7fa77d800640 1 --2- 172.21.15.96:0/1662111247 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa778156b30 0x7fa778158f60 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7fa77000ccd0 tx=0x7fa77000a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:21:58.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:58.956+0000 7fa774c00640 1 -- 172.21.15.96:0/1662111247 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa77000f680 con 0x7fa778156b30 2024-05-28T06:21:58.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:58.956+0000 7fa774c00640 1 -- 172.21.15.96:0/1662111247 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa77000f840 con 0x7fa778156b30 2024-05-28T06:21:58.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:58.956+0000 7fa77e200640 1 -- 172.21.15.96:0/1662111247 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa77815ea80 con 0x7fa778156b30 2024-05-28T06:21:58.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:58.956+0000 7fa774c00640 1 -- 172.21.15.96:0/1662111247 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa77000fc30 con 0x7fa778156b30 2024-05-28T06:21:58.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:58.956+0000 7fa77e200640 1 -- 172.21.15.96:0/1662111247 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa778157130 con 0x7fa778156b30 2024-05-28T06:21:58.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:58.958+0000 7fa77e200640 1 -- 172.21.15.96:0/1662111247 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7780689e0 con 0x7fa778156b30 2024-05-28T06:21:58.956 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:58.961+0000 7fa774c00640 1 -- 172.21.15.96:0/1662111247 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 37) ==== 93182+0+0 (secure 0 0 0) 0x7fa7700213f0 con 0x7fa778156b30 2024-05-28T06:21:58.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:58.962+0000 7fa774c00640 1 --2- 172.21.15.96:0/1662111247 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa75806e090 0x7fa758070550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:58.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:58.963+0000 7fa774c00640 1 -- 172.21.15.96:0/1662111247 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 5821+0+0 (secure 0 0 0) 0x7fa77008ba10 con 0x7fa778156b30 2024-05-28T06:21:58.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:58.963+0000 7fa77ce00640 1 --2- 172.21.15.96:0/1662111247 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa75806e090 0x7fa758070550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:58.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:58.964+0000 7fa77ce00640 1 --2- 172.21.15.96:0/1662111247 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa75806e090 0x7fa758070550 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fa76c034000 tx=0x7fa76c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:21:58.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:58.965+0000 7fa774c00640 1 -- 172.21.15.96:0/1662111247 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa770014030 con 0x7fa778156b30 2024-05-28T06:21:59.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.209+0000 7fa77e200640 1 -- 172.21.15.96:0/1662111247 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa778157300 con 0x7fa778156b30 2024-05-28T06:21:59.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.210+0000 7fa774c00640 1 -- 172.21.15.96:0/1662111247 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) ==== 94+0+5189 (secure 0 0 0) 0x7fa77005b920 con 0x7fa778156b30 2024-05-28T06:21:59.205 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:21:59.205 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":63,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5861,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/1001966057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6833","nonce":1001966057},{"type":"v1","addr":"172.21.15.96:6835","nonce":1001966057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":57},{"gid":5864,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6836/4285246962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":4285246962},{"type":"v1","addr":"172.21.15.123:6836","nonce":4285246962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":57},{"gid":6445,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6834/3302807355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":3302807355},{"type":"v1","addr":"172.21.15.96:6834","nonce":3302807355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-28T06:20:18.363024+0000","modified":"2024-05-28T06:20:21.426136+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8739},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8739":{"gid":8739,"name":"b","rank":0,"incarnation":58,"state":"up:active","state_seq":3,"addr":"172.21.15.123:6837/3372660768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6835","nonce":3372660768},{"type":"v1","addr":"172.21.15.123:6837","nonce":3372660768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8739,"qdb_cluster":[8739]},"id":7}]} 2024-05-28T06:21:59.205 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 63 2024-05-28T06:21:59.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.212+0000 7fa77e200640 1 -- 172.21.15.96:0/1662111247 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa75806e090 msgr2=0x7fa758070550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:59.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.212+0000 7fa77e200640 1 --2- 172.21.15.96:0/1662111247 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa75806e090 0x7fa758070550 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fa76c034000 tx=0x7fa76c02e000 comp rx=0 tx=0).stop 2024-05-28T06:21:59.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.212+0000 7fa77e200640 1 -- 172.21.15.96:0/1662111247 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa778156b30 msgr2=0x7fa778158f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:59.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.213+0000 7fa77e200640 1 --2- 172.21.15.96:0/1662111247 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa778156b30 0x7fa778158f60 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7fa77000ccd0 tx=0x7fa77000a650 comp rx=0 tx=0).stop 2024-05-28T06:21:59.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.213+0000 7fa77e200640 1 -- 172.21.15.96:0/1662111247 shutdown_connections 2024-05-28T06:21:59.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.213+0000 7fa77e200640 1 --2- 172.21.15.96:0/1662111247 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa75806e090 0x7fa758070550 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:59.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.213+0000 7fa77e200640 1 --2- 172.21.15.96:0/1662111247 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa778156b30 0x7fa778158f60 unknown :-1 s=CLOSED pgs=1715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:59.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.213+0000 7fa77e200640 1 --2- 172.21.15.96:0/1662111247 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa77806f2c0 0x7fa7780b3650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:59.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.213+0000 7fa77e200640 1 --2- 172.21.15.96:0/1662111247 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa77806ee40 0x7fa778073890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:59.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.213+0000 7fa77e200640 1 -- 172.21.15.96:0/1662111247 >> 172.21.15.96:0/1662111247 conn(0x7fa7780b1fc0 msgr2=0x7fa7780b1370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:21:59.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.213+0000 7fa77e200640 1 -- 172.21.15.96:0/1662111247 shutdown_connections 2024-05-28T06:21:59.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.213+0000 7fa77e200640 1 -- 172.21.15.96:0/1662111247 wait complete. 2024-05-28T06:21:59.227 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:21:59.430 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.436+0000 7f90bb800640 1 Processor -- start 2024-05-28T06:21:59.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.436+0000 7f90bb800640 1 -- start start 2024-05-28T06:21:59.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.437+0000 7f90bb800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f90b40d0a20 0x7f90b40997a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:59.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.437+0000 7f90bb800640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f90b40b51f0 con 0x7f90b40d0590 2024-05-28T06:21:59.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.437+0000 7f90bb800640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f90b40aef60 con 0x7f90b40b4e10 2024-05-28T06:21:59.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.437+0000 7f90bb800640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f90b40ad1d0 con 0x7f90b40d0a20 2024-05-28T06:21:59.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.437+0000 7f90b3e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f90b40d0590 0x7f90b4099090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:37934/0 (socket says 172.21.15.96:37934) 2024-05-28T06:21:59.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.437+0000 7f90b3e00640 1 -- 172.21.15.96:0/2265160335 learned_addr learned my addr 172.21.15.96:0/2265160335 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:21:59.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.438+0000 7f90b9800640 1 --2- 172.21.15.96:0/2265160335 >> v2:172.21.15.123:3300/0 conn(0x7f90b40d0a20 0x7f90b40997a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:59.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.438+0000 7f90b3400640 1 -- 172.21.15.96:0/2265160335 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 59676501 0 0) 0x7f90b40b51f0 con 0x7f90b40d0590 2024-05-28T06:21:59.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.438+0000 7f90b3400640 1 -- 172.21.15.96:0/2265160335 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f909c005730 con 0x7f90b40d0590 2024-05-28T06:21:59.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.438+0000 7f90b3400640 1 -- 172.21.15.96:0/2265160335 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2197197863 0 0) 0x7f909c005730 con 0x7f90b40d0590 2024-05-28T06:21:59.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.438+0000 7f90b3400640 1 -- 172.21.15.96:0/2265160335 >> v2:172.21.15.123:3300/0 conn(0x7f90b40d0a20 msgr2=0x7f90b40997a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:21:59.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.438+0000 7f90b3400640 1 --2- 172.21.15.96:0/2265160335 >> v2:172.21.15.123:3300/0 conn(0x7f90b40d0a20 0x7f90b40997a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:59.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.438+0000 7f90b3400640 1 -- 172.21.15.96:0/2265160335 >> v1:172.21.15.123:6789/0 conn(0x7f90b40b4e10 legacy=0x7f90b40cde30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:21:59.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.438+0000 7f90b3400640 1 -- 172.21.15.96:0/2265160335 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90b40ad470 con 0x7f90b40d0590 2024-05-28T06:21:59.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.438+0000 7f90b3400640 1 -- 172.21.15.96:0/2265160335 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f90a0001670 con 0x7f90b40d0590 2024-05-28T06:21:59.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.438+0000 7f90b3400640 1 -- 172.21.15.96:0/2265160335 >> v1:172.21.15.96:6789/0 conn(0x7f90b40d0590 legacy=0x7f90b4099090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:59.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.438+0000 7f90b3400640 1 --2- 172.21.15.96:0/2265160335 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f909c005bc0 0x7f909c005fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:59.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.438+0000 7f90b3400640 1 --2- 172.21.15.96:0/2265160335 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f909c006a10 0x7f909c006e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:59.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.438+0000 7f90b3400640 1 --2- 172.21.15.96:0/2265160335 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f909c0074f0 0x7f909c00dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:59.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.438+0000 7f90b3400640 1 -- 172.21.15.96:0/2265160335 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f90b40ad1d0 con 0x7f909c005bc0 2024-05-28T06:21:59.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.438+0000 7f90b3400640 1 -- 172.21.15.96:0/2265160335 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f909c00e3a0 con 0x7f909c0074f0 2024-05-28T06:21:59.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.438+0000 7f90b3400640 1 -- 172.21.15.96:0/2265160335 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f909c00e5a0 con 0x7f909c006a10 2024-05-28T06:21:59.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.439+0000 7f90b8e00640 1 --2- 172.21.15.96:0/2265160335 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f909c005bc0 0x7f909c005fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:59.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.439+0000 7f90b9800640 1 --2- 172.21.15.96:0/2265160335 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f909c0074f0 0x7f909c00dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:59.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.439+0000 7f90b8e00640 1 -- 172.21.15.96:0/2265160335 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f909c006a10 msgr2=0x7f909c006e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:21:59.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.439+0000 7f90b8e00640 1 --2- 172.21.15.96:0/2265160335 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f909c006a10 0x7f909c006e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:59.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.439+0000 7f90b8e00640 1 -- 172.21.15.96:0/2265160335 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f909c0074f0 msgr2=0x7f909c00dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:21:59.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.439+0000 7f90b8e00640 1 --2- 172.21.15.96:0/2265160335 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f909c0074f0 0x7f909c00dbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:59.436 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.439+0000 7f90b9800640 1 -- 172.21.15.96:0/2265160335 reap_dead start 2024-05-28T06:21:59.436 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.439+0000 7f90b8e00640 1 -- 172.21.15.96:0/2265160335 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f909c005730 con 0x7f909c005bc0 2024-05-28T06:21:59.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.440+0000 7f90b8e00640 1 --2- 172.21.15.96:0/2265160335 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f909c005bc0 0x7f909c005fc0 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7f90a8003d80 tx=0x7f909c007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acbf6c71e3d66d57 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:21:59.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.441+0000 7f90b3400640 1 -- 172.21.15.96:0/2265160335 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90a8037070 con 0x7f909c005bc0 2024-05-28T06:21:59.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.441+0000 7f90b3400640 1 -- 172.21.15.96:0/2265160335 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90a802a9c0 con 0x7f909c005bc0 2024-05-28T06:21:59.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.441+0000 7f90b3400640 1 -- 172.21.15.96:0/2265160335 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90a8003980 con 0x7f909c005bc0 2024-05-28T06:21:59.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.441+0000 7f90bb800640 1 -- 172.21.15.96:0/2265160335 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f909c005bc0 msgr2=0x7f909c005fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:59.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.441+0000 7f90bb800640 1 --2- 172.21.15.96:0/2265160335 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f909c005bc0 0x7f909c005fc0 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7f90a8003d80 tx=0x7f909c007a40 comp rx=0 tx=0).stop 2024-05-28T06:21:59.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.441+0000 7f90bb800640 1 -- 172.21.15.96:0/2265160335 shutdown_connections 2024-05-28T06:21:59.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.441+0000 7f90bb800640 1 --2- 172.21.15.96:0/2265160335 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f909c005bc0 0x7f909c005fc0 unknown :-1 s=CLOSED pgs=1717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:59.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.441+0000 7f90bb800640 1 -- 172.21.15.96:0/2265160335 >> 172.21.15.96:0/2265160335 conn(0x7f90b406bae0 msgr2=0x7f90b406bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:21:59.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.441+0000 7f90bb800640 1 -- 172.21.15.96:0/2265160335 shutdown_connections 2024-05-28T06:21:59.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.441+0000 7f90bb800640 1 -- 172.21.15.96:0/2265160335 wait complete. 2024-05-28T06:21:59.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.443+0000 7f90bb800640 1 Processor -- start 2024-05-28T06:21:59.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.443+0000 7f90bb800640 1 -- start start 2024-05-28T06:21:59.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.444+0000 7f90bb800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f909c005bc0 0x7f90b40c85e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:59.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.444+0000 7f90bb800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f90b40b16f0 0x7f90b41f7ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:59.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.444+0000 7f90bb800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f90b40b1b50 0x7f90b41f8220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:59.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.444+0000 7f90bb800640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f90b40ce880 con 0x7f909c005bc0 2024-05-28T06:21:59.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.444+0000 7f90bb800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f90b40d0e30 con 0x7f90b40b16f0 2024-05-28T06:21:59.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.444+0000 7f90bb800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f90b40cea00 con 0x7f90b40b1b50 2024-05-28T06:21:59.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.445+0000 7f90b8e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f909c005bc0 0x7f90b40c85e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:59.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.445+0000 7f90b8e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f909c005bc0 0x7f90b40c85e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35824/0 (socket says 172.21.15.96:35824) 2024-05-28T06:21:59.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.445+0000 7f90b8e00640 1 -- 172.21.15.96:0/2614459275 learned_addr learned my addr 172.21.15.96:0/2614459275 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:21:59.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.445+0000 7f90b3e00640 1 --2- 172.21.15.96:0/2614459275 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f90b40b16f0 0x7f90b41f7ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:59.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.445+0000 7f90b9800640 1 --2- 172.21.15.96:0/2614459275 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f90b40b1b50 0x7f90b41f8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:59.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.445+0000 7f90b8e00640 1 -- 172.21.15.96:0/2614459275 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f90b40b1b50 msgr2=0x7f90b41f8220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:59.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.445+0000 7f90b8e00640 1 --2- 172.21.15.96:0/2614459275 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f90b40b1b50 0x7f90b41f8220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:59.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.445+0000 7f90b8e00640 1 -- 172.21.15.96:0/2614459275 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f90b40b16f0 msgr2=0x7f90b41f7ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:59.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.445+0000 7f90b8e00640 1 --2- 172.21.15.96:0/2614459275 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f90b40b16f0 0x7f90b41f7ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:59.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.445+0000 7f90b8e00640 1 -- 172.21.15.96:0/2614459275 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90b40c8b20 con 0x7f909c005bc0 2024-05-28T06:21:59.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.445+0000 7f90b3e00640 1 --2- 172.21.15.96:0/2614459275 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f90b40b16f0 0x7f90b41f7ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:21:59.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.445+0000 7f90b8e00640 1 --2- 172.21.15.96:0/2614459275 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f909c005bc0 0x7f90b40c85e0 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7f90a8035040 tx=0x7f90a80036f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:21:59.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.446+0000 7f90b0c00640 1 -- 172.21.15.96:0/2614459275 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90a8037050 con 0x7f909c005bc0 2024-05-28T06:21:59.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.446+0000 7f90b0c00640 1 -- 172.21.15.96:0/2614459275 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90a8038040 con 0x7f909c005bc0 2024-05-28T06:21:59.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.446+0000 7f90b0c00640 1 -- 172.21.15.96:0/2614459275 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90a8007a70 con 0x7f909c005bc0 2024-05-28T06:21:59.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.446+0000 7f90bb800640 1 -- 172.21.15.96:0/2614459275 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90b40b2960 con 0x7f909c005bc0 2024-05-28T06:21:59.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.446+0000 7f90bb800640 1 -- 172.21.15.96:0/2614459275 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90b41f8af0 con 0x7f909c005bc0 2024-05-28T06:21:59.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.447+0000 7f90bb800640 1 -- 172.21.15.96:0/2614459275 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90b40689e0 con 0x7f909c005bc0 2024-05-28T06:21:59.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.448+0000 7f90b0c00640 1 -- 172.21.15.96:0/2614459275 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 37) ==== 93182+0+0 (secure 0 0 0) 0x7f90a8045020 con 0x7f909c005bc0 2024-05-28T06:21:59.444 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.449+0000 7f90b0c00640 1 --2- 172.21.15.96:0/2614459275 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f909406e0c0 0x7f9094070580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:59.444 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.449+0000 7f90b0c00640 1 -- 172.21.15.96:0/2614459275 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 5821+0+0 (secure 0 0 0) 0x7f90a80aa630 con 0x7f909c005bc0 2024-05-28T06:21:59.444 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.450+0000 7f90b3e00640 1 --2- 172.21.15.96:0/2614459275 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f909406e0c0 0x7f9094070580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:59.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.451+0000 7f90b3e00640 1 --2- 172.21.15.96:0/2614459275 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f909406e0c0 0x7f9094070580 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f90a0001eb0 tx=0x7f90a0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:21:59.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.454+0000 7f90b0c00640 1 -- 172.21.15.96:0/2614459275 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f90b40c8b20 con 0x7f909c005bc0 2024-05-28T06:21:59.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.656+0000 7f90bb800640 1 -- 172.21.15.96:0/2614459275 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f90b406f540 con 0x7f909c005bc0 2024-05-28T06:21:59.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.658+0000 7f90b0c00640 1 -- 172.21.15.96:0/2614459275 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v104) ==== 74+0+16831 (secure 0 0 0) 0x7f90a807a4c0 con 0x7f909c005bc0 2024-05-28T06:21:59.653 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:21:59.653 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":104,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:21:58.794459+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-05-28T06:20:16.097747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-05-28T06:20:17.338574+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"104","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":104,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":13,"snaps":[{"begin":2,"length":2}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:21:59.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.660+0000 7f90bb800640 1 -- 172.21.15.96:0/2614459275 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f909406e0c0 msgr2=0x7f9094070580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:59.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.660+0000 7f90bb800640 1 --2- 172.21.15.96:0/2614459275 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f909406e0c0 0x7f9094070580 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f90a0001eb0 tx=0x7f90a0009040 comp rx=0 tx=0).stop 2024-05-28T06:21:59.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.660+0000 7f90bb800640 1 -- 172.21.15.96:0/2614459275 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f909c005bc0 msgr2=0x7f90b40c85e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:59.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.660+0000 7f90bb800640 1 --2- 172.21.15.96:0/2614459275 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f909c005bc0 0x7f90b40c85e0 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7f90a8035040 tx=0x7f90a80036f0 comp rx=0 tx=0).stop 2024-05-28T06:21:59.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.660+0000 7f90bb800640 1 -- 172.21.15.96:0/2614459275 shutdown_connections 2024-05-28T06:21:59.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.660+0000 7f90bb800640 1 --2- 172.21.15.96:0/2614459275 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f909406e0c0 0x7f9094070580 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:59.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.660+0000 7f90bb800640 1 --2- 172.21.15.96:0/2614459275 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f90b40b1b50 0x7f90b41f8220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:59.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.660+0000 7f90bb800640 1 --2- 172.21.15.96:0/2614459275 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f90b40b16f0 0x7f90b41f7ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:59.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.660+0000 7f90bb800640 1 --2- 172.21.15.96:0/2614459275 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f909c005bc0 0x7f90b40c85e0 unknown :-1 s=CLOSED pgs=1718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:59.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.660+0000 7f90bb800640 1 -- 172.21.15.96:0/2614459275 >> 172.21.15.96:0/2614459275 conn(0x7f90b406bae0 msgr2=0x7f90b40ca8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:21:59.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.661+0000 7f90bb800640 1 -- 172.21.15.96:0/2614459275 shutdown_connections 2024-05-28T06:21:59.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.661+0000 7f90bb800640 1 -- 172.21.15.96:0/2614459275 wait complete. 2024-05-28T06:21:59.675 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-28T06:03:59.517102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-28T06:20:16.097747+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '102', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2024-05-28T06:20:17.338574+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '104', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 104, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-28T06:21:59.675 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-28T06:21:59.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.873+0000 7f0be0c00640 1 Processor -- start 2024-05-28T06:21:59.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.873+0000 7f0be0c00640 1 -- start start 2024-05-28T06:21:59.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.875+0000 7f0be0c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0bdc0b45d0 0x7f0bdc0b4a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:59.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.875+0000 7f0be0c00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0bdc06e2e0 con 0x7f0bdc15abe0 2024-05-28T06:21:59.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.875+0000 7f0be0c00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0bdc06e5c0 con 0x7f0bdc0725c0 2024-05-28T06:21:59.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.875+0000 7f0be0c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f0bdc06e8a0 con 0x7f0bdc0b45d0 2024-05-28T06:21:59.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.875+0000 7f0bd9e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f0bdc15abe0 0x7f0bdc06f380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:37946/0 (socket says 172.21.15.96:37946) 2024-05-28T06:21:59.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.875+0000 7f0bd9e00640 1 -- 172.21.15.96:0/777919045 learned_addr learned my addr 172.21.15.96:0/777919045 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:21:59.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.875+0000 7f0bd1400640 1 --2- 172.21.15.96:0/777919045 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0bdc0b45d0 0x7f0bdc0b4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:59.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.876+0000 7f0bd9400640 1 -- 172.21.15.96:0/777919045 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3915716899 0 0) 0x7f0bdc06e2e0 con 0x7f0bdc15abe0 2024-05-28T06:21:59.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.876+0000 7f0bd9400640 1 -- 172.21.15.96:0/777919045 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0bc0003520 con 0x7f0bdc15abe0 2024-05-28T06:21:59.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.876+0000 7f0bd9400640 1 -- 172.21.15.96:0/777919045 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1860903213 0 0) 0x7f0bc0003520 con 0x7f0bdc15abe0 2024-05-28T06:21:59.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.876+0000 7f0bd9400640 1 -- 172.21.15.96:0/777919045 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0bdc0b45d0 msgr2=0x7f0bdc0b4a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:21:59.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.876+0000 7f0bd9400640 1 --2- 172.21.15.96:0/777919045 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0bdc0b45d0 0x7f0bdc0b4a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:59.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.876+0000 7f0bd9400640 1 -- 172.21.15.96:0/777919045 >> v1:172.21.15.123:6789/0 conn(0x7f0bdc0725c0 legacy=0x7f0bdc0ad680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:59.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.876+0000 7f0bd9400640 1 -- 172.21.15.96:0/777919045 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bdc06eb80 con 0x7f0bdc15abe0 2024-05-28T06:21:59.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.876+0000 7f0bd9400640 1 -- 172.21.15.96:0/777919045 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f0bcc001670 con 0x7f0bdc15abe0 2024-05-28T06:21:59.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.876+0000 7f0bd9400640 1 -- 172.21.15.96:0/777919045 >> v1:172.21.15.96:6789/0 conn(0x7f0bdc15abe0 legacy=0x7f0bdc06f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:59.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.876+0000 7f0bd9400640 1 --2- 172.21.15.96:0/777919045 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0bc00039e0 0x7f0bc0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:59.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.876+0000 7f0bd9400640 1 --2- 172.21.15.96:0/777919045 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0bdc0b45d0 0x7f0bc0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:59.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.876+0000 7f0bd9400640 1 --2- 172.21.15.96:0/777919045 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0bc0004d40 0x7f0bc00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:59.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.876+0000 7f0bd9400640 1 -- 172.21.15.96:0/777919045 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f0bdc06e8a0 con 0x7f0bc00039e0 2024-05-28T06:21:59.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.876+0000 7f0bd9400640 1 -- 172.21.15.96:0/777919045 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f0bc000cbb0 con 0x7f0bc0004d40 2024-05-28T06:21:59.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.876+0000 7f0bd9400640 1 -- 172.21.15.96:0/777919045 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f0bc000cdb0 con 0x7f0bdc0b45d0 2024-05-28T06:21:59.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.876+0000 7f0bd9e00640 1 --2- 172.21.15.96:0/777919045 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0bc00039e0 0x7f0bc0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:59.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.877+0000 7f0bd1400640 1 --2- 172.21.15.96:0/777919045 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0bdc0b45d0 0x7f0bc0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:59.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.877+0000 7f0bd9e00640 1 -- 172.21.15.96:0/777919045 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0bdc0b45d0 msgr2=0x7f0bc0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:21:59.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.877+0000 7f0bd9e00640 1 --2- 172.21.15.96:0/777919045 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0bdc0b45d0 0x7f0bc0004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:59.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.877+0000 7f0bd9e00640 1 -- 172.21.15.96:0/777919045 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0bc0004d40 msgr2=0x7f0bc00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:21:59.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.877+0000 7f0bda800640 1 --2- 172.21.15.96:0/777919045 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0bc0004d40 0x7f0bc00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:59.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.877+0000 7f0bd9e00640 1 --2- 172.21.15.96:0/777919045 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0bc0004d40 0x7f0bc00051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:59.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.877+0000 7f0bd9e00640 1 -- 172.21.15.96:0/777919045 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bc0003520 con 0x7f0bc00039e0 2024-05-28T06:21:59.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.878+0000 7f0bd9e00640 1 --2- 172.21.15.96:0/777919045 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0bc00039e0 0x7f0bc0003e00 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7f0bcc007fd0 tx=0x7f0bc0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=475bec242a69fe33 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:21:59.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.878+0000 7f0bd9400640 1 -- 172.21.15.96:0/777919045 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bcc007c20 con 0x7f0bc00039e0 2024-05-28T06:21:59.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.878+0000 7f0bd9400640 1 -- 172.21.15.96:0/777919045 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0bcc005af0 con 0x7f0bc00039e0 2024-05-28T06:21:59.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.878+0000 7f0bd9400640 1 -- 172.21.15.96:0/777919045 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bcc007070 con 0x7f0bc00039e0 2024-05-28T06:21:59.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.879+0000 7f0be0c00640 1 -- 172.21.15.96:0/777919045 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0bc00039e0 msgr2=0x7f0bc0003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:59.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.879+0000 7f0be0c00640 1 --2- 172.21.15.96:0/777919045 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0bc00039e0 0x7f0bc0003e00 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7f0bcc007fd0 tx=0x7f0bc0005830 comp rx=0 tx=0).stop 2024-05-28T06:21:59.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.879+0000 7f0bda800640 1 -- 172.21.15.96:0/777919045 reap_dead start 2024-05-28T06:21:59.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.879+0000 7f0be0c00640 1 -- 172.21.15.96:0/777919045 shutdown_connections 2024-05-28T06:21:59.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.879+0000 7f0be0c00640 1 -- 172.21.15.96:0/777919045 >> 172.21.15.96:0/777919045 conn(0x7f0bdc06bd20 msgr2=0x7f0bdc06c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:21:59.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.879+0000 7f0be0c00640 1 -- 172.21.15.96:0/777919045 shutdown_connections 2024-05-28T06:21:59.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.879+0000 7f0be0c00640 1 -- 172.21.15.96:0/777919045 wait complete. 2024-05-28T06:21:59.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.881+0000 7f0be0c00640 1 Processor -- start 2024-05-28T06:21:59.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.881+0000 7f0be0c00640 1 -- start start 2024-05-28T06:21:59.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.882+0000 7f0be0c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0bc00039e0 0x7f0bdc09dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:59.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.882+0000 7f0be0c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0bdc0d0680 0x7f0bdc1f14e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:59.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.882+0000 7f0be0c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0bdc0b0da0 0x7f0bdc159650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:59.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.882+0000 7f0be0c00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f0bdc06fa90 con 0x7f0bdc0b0da0 2024-05-28T06:21:59.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.882+0000 7f0be0c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f0bdc15afc0 con 0x7f0bc00039e0 2024-05-28T06:21:59.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.882+0000 7f0be0c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f0bdc06df60 con 0x7f0bdc0d0680 2024-05-28T06:21:59.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.882+0000 7f0bda800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0bdc0b0da0 0x7f0bdc159650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:59.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.882+0000 7f0bda800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0bdc0b0da0 0x7f0bdc159650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35846/0 (socket says 172.21.15.96:35846) 2024-05-28T06:21:59.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.882+0000 7f0bda800640 1 -- 172.21.15.96:0/177863903 learned_addr learned my addr 172.21.15.96:0/177863903 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:21:59.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.883+0000 7f0bd1400640 1 --2- 172.21.15.96:0/177863903 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0bdc0d0680 0x7f0bdc1f14e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:59.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.883+0000 7f0bd9e00640 1 --2- 172.21.15.96:0/177863903 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0bc00039e0 0x7f0bdc09dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:59.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.883+0000 7f0bda800640 1 -- 172.21.15.96:0/177863903 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0bdc0d0680 msgr2=0x7f0bdc1f14e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:59.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.883+0000 7f0bda800640 1 --2- 172.21.15.96:0/177863903 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0bdc0d0680 0x7f0bdc1f14e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:59.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.883+0000 7f0bda800640 1 -- 172.21.15.96:0/177863903 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0bc00039e0 msgr2=0x7f0bdc09dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:21:59.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.883+0000 7f0bda800640 1 --2- 172.21.15.96:0/177863903 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0bc00039e0 0x7f0bdc09dd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:21:59.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.883+0000 7f0bda800640 1 -- 172.21.15.96:0/177863903 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bdc0b1880 con 0x7f0bdc0b0da0 2024-05-28T06:21:59.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.883+0000 7f0bda800640 1 --2- 172.21.15.96:0/177863903 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0bdc0b0da0 0x7f0bdc159650 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7f0bdc06dd10 tx=0x7f0bd400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:21:59.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.883+0000 7f0bd2a00640 1 -- 172.21.15.96:0/177863903 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bd4018070 con 0x7f0bdc0b0da0 2024-05-28T06:21:59.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.883+0000 7f0bd2a00640 1 -- 172.21.15.96:0/177863903 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0bd4003b70 con 0x7f0bdc0b0da0 2024-05-28T06:21:59.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.883+0000 7f0bd2a00640 1 -- 172.21.15.96:0/177863903 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bd4005070 con 0x7f0bdc0b0da0 2024-05-28T06:21:59.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.883+0000 7f0be0c00640 1 -- 172.21.15.96:0/177863903 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0bdc0b1b60 con 0x7f0bdc0b0da0 2024-05-28T06:21:59.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.884+0000 7f0be0c00640 1 -- 172.21.15.96:0/177863903 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0bdc1f1a20 con 0x7f0bdc0b0da0 2024-05-28T06:21:59.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.885+0000 7f0be0c00640 1 -- 172.21.15.96:0/177863903 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0bdc0689e0 con 0x7f0bdc0b0da0 2024-05-28T06:21:59.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.886+0000 7f0bd2a00640 1 -- 172.21.15.96:0/177863903 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 37) ==== 93182+0+0 (secure 0 0 0) 0x7f0bd401b070 con 0x7f0bdc0b0da0 2024-05-28T06:21:59.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.887+0000 7f0bd2a00640 1 --2- 172.21.15.96:0/177863903 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0bb806e030 0x7f0bb80704f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:21:59.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.887+0000 7f0bd2a00640 1 -- 172.21.15.96:0/177863903 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 5821+0+0 (secure 0 0 0) 0x7f0bd408a190 con 0x7f0bdc0b0da0 2024-05-28T06:21:59.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.887+0000 7f0bd9e00640 1 --2- 172.21.15.96:0/177863903 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0bb806e030 0x7f0bb80704f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:21:59.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.889+0000 7f0bd9e00640 1 --2- 172.21.15.96:0/177863903 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0bb806e030 0x7f0bb80704f0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f0bcc032000 tx=0x7f0bcc009030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:21:59.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:21:59.891+0000 7f0bd2a00640 1 -- 172.21.15.96:0/177863903 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0bd405a020 con 0x7f0bdc0b0da0 2024-05-28T06:22:00.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.085+0000 7f0be0c00640 1 -- 172.21.15.96:0/177863903 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f0bdc06dd40 con 0x7f0bdc0b0da0 2024-05-28T06:22:00.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.086+0000 7f0bd2a00640 1 -- 172.21.15.96:0/177863903 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v104) ==== 93+0+10 (secure 0 0 0) 0x7f0bd4059920 con 0x7f0bdc0b0da0 2024-05-28T06:22:00.081 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 1 2024-05-28T06:22:00.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.088+0000 7f0be0c00640 1 -- 172.21.15.96:0/177863903 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0bb806e030 msgr2=0x7f0bb80704f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:00.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.088+0000 7f0be0c00640 1 --2- 172.21.15.96:0/177863903 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0bb806e030 0x7f0bb80704f0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f0bcc032000 tx=0x7f0bcc009030 comp rx=0 tx=0).stop 2024-05-28T06:22:00.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.088+0000 7f0be0c00640 1 -- 172.21.15.96:0/177863903 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0bdc0b0da0 msgr2=0x7f0bdc159650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:00.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.088+0000 7f0be0c00640 1 --2- 172.21.15.96:0/177863903 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0bdc0b0da0 0x7f0bdc159650 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7f0bdc06dd10 tx=0x7f0bd400dc00 comp rx=0 tx=0).stop 2024-05-28T06:22:00.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.088+0000 7f0be0c00640 1 -- 172.21.15.96:0/177863903 shutdown_connections 2024-05-28T06:22:00.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.088+0000 7f0be0c00640 1 --2- 172.21.15.96:0/177863903 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0bb806e030 0x7f0bb80704f0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:00.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.088+0000 7f0be0c00640 1 --2- 172.21.15.96:0/177863903 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0bdc0b0da0 0x7f0bdc159650 unknown :-1 s=CLOSED pgs=1721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:00.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.088+0000 7f0be0c00640 1 --2- 172.21.15.96:0/177863903 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0bdc0d0680 0x7f0bdc1f14e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:00.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.088+0000 7f0be0c00640 1 --2- 172.21.15.96:0/177863903 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0bc00039e0 0x7f0bdc09dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:00.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.088+0000 7f0be0c00640 1 -- 172.21.15.96:0/177863903 >> 172.21.15.96:0/177863903 conn(0x7f0bdc06bd20 msgr2=0x7f0bdc15d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:00.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.088+0000 7f0be0c00640 1 -- 172.21.15.96:0/177863903 shutdown_connections 2024-05-28T06:22:00.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.088+0000 7f0be0c00640 1 -- 172.21.15.96:0/177863903 wait complete. 2024-05-28T06:22:00.102 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-28T06:22:00.299 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.304+0000 7f7094c00640 1 Processor -- start 2024-05-28T06:22:00.299 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.304+0000 7f7094c00640 1 -- start start 2024-05-28T06:22:00.300 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.306+0000 7f7094c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7090153510 0x7f70900b4d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:00.301 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.306+0000 7f7094c00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f709015d270 0x7f70900ccff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:00.301 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.306+0000 7f7094c00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f709015d900 con 0x7f70900ccb90 2024-05-28T06:22:00.301 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.306+0000 7f7094c00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f70901538f0 con 0x7f709015d270 2024-05-28T06:22:00.301 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.306+0000 7f7094c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f70900cd530 con 0x7f7090153510 2024-05-28T06:22:00.301 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.306+0000 7f708ea00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f70900ccb90 0x7f709015cb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:37954/0 (socket says 172.21.15.96:37954) 2024-05-28T06:22:00.301 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.306+0000 7f708ea00640 1 -- 172.21.15.96:0/2798342280 learned_addr learned my addr 172.21.15.96:0/2798342280 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:22:00.301 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.306+0000 7f708fe00640 1 --2- 172.21.15.96:0/2798342280 >> v2:172.21.15.96:3300/0 conn(0x7f709015d270 0x7f70900ccff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:00.302 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.306+0000 7f708f400640 1 --2- 172.21.15.96:0/2798342280 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7090153510 0x7f70900b4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:00.302 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.307+0000 7f708e000640 1 -- 172.21.15.96:0/2798342280 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3332337710 0 0) 0x7f709015d900 con 0x7f70900ccb90 2024-05-28T06:22:00.302 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.307+0000 7f708e000640 1 -- 172.21.15.96:0/2798342280 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7074003520 con 0x7f70900ccb90 2024-05-28T06:22:00.302 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.307+0000 7f708fe00640 1 -- 172.21.15.96:0/2798342280 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7090153510 msgr2=0x7f70900b4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:00.302 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.307+0000 7f708fe00640 1 --2- 172.21.15.96:0/2798342280 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7090153510 0x7f70900b4d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:00.302 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.307+0000 7f708fe00640 1 -- 172.21.15.96:0/2798342280 >> v1:172.21.15.96:6789/0 conn(0x7f70900ccb90 legacy=0x7f709015cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:00.303 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.307+0000 7f708fe00640 1 -- 172.21.15.96:0/2798342280 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70900b3f90 con 0x7f709015d270 2024-05-28T06:22:00.303 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.308+0000 7f708fe00640 1 --2- 172.21.15.96:0/2798342280 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f709015d270 0x7f70900ccff0 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7f7088009d90 tx=0x7f708802a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0a35e69e88cef24 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:00.303 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.308+0000 7f708e000640 1 -- 172.21.15.96:0/2798342280 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7088031600 con 0x7f709015d270 2024-05-28T06:22:00.303 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.308+0000 7f708e000640 1 -- 172.21.15.96:0/2798342280 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70880317a0 con 0x7f709015d270 2024-05-28T06:22:00.303 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.308+0000 7f708e000640 1 -- 172.21.15.96:0/2798342280 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7088031b50 con 0x7f709015d270 2024-05-28T06:22:00.304 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.309+0000 7f7094c00640 1 -- 172.21.15.96:0/2798342280 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f709015d270 msgr2=0x7f70900ccff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:00.304 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.309+0000 7f7094c00640 1 --2- 172.21.15.96:0/2798342280 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f709015d270 0x7f70900ccff0 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7f7088009d90 tx=0x7f708802a5d0 comp rx=0 tx=0).stop 2024-05-28T06:22:00.304 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.309+0000 7f7094c00640 1 -- 172.21.15.96:0/2798342280 shutdown_connections 2024-05-28T06:22:00.304 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.309+0000 7f7094c00640 1 --2- 172.21.15.96:0/2798342280 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f709015d270 0x7f70900ccff0 unknown :-1 s=CLOSED pgs=1723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:00.305 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.309+0000 7f7094c00640 1 --2- 172.21.15.96:0/2798342280 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7090153510 0x7f70900b4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:00.305 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.309+0000 7f7094c00640 1 -- 172.21.15.96:0/2798342280 >> 172.21.15.96:0/2798342280 conn(0x7f709006b900 msgr2=0x7f709006bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:00.305 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.309+0000 7f7094c00640 1 -- 172.21.15.96:0/2798342280 shutdown_connections 2024-05-28T06:22:00.305 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.309+0000 7f7094c00640 1 -- 172.21.15.96:0/2798342280 wait complete. 2024-05-28T06:22:00.307 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.311+0000 7f7094c00640 1 Processor -- start 2024-05-28T06:22:00.307 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.311+0000 7f7094c00640 1 -- start start 2024-05-28T06:22:00.307 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.312+0000 7f7094c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f70900ccb90 0x7f70901f1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:00.307 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.312+0000 7f7094c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7090153510 0x7f70901475f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:00.307 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.312+0000 7f7094c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f709015d270 0x7f70900b1da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:00.308 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.312+0000 7f7094c00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f70900b4550 con 0x7f7090153510 2024-05-28T06:22:00.308 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.312+0000 7f7094c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f70900b48c0 con 0x7f709015d270 2024-05-28T06:22:00.308 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.312+0000 7f7094c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f709006e940 con 0x7f70900ccb90 2024-05-28T06:22:00.308 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.313+0000 7f708ea00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7090153510 0x7f70901475f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:00.308 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.313+0000 7f708ea00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7090153510 0x7f70901475f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35858/0 (socket says 172.21.15.96:35858) 2024-05-28T06:22:00.308 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.313+0000 7f708ea00640 1 -- 172.21.15.96:0/1245912561 learned_addr learned my addr 172.21.15.96:0/1245912561 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:22:00.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.313+0000 7f708ea00640 1 -- 172.21.15.96:0/1245912561 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f70900ccb90 msgr2=0x7f70901f1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:00.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.313+0000 7f708f400640 1 --2- 172.21.15.96:0/1245912561 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f70900ccb90 0x7f70901f1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:00.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.313+0000 7f708fe00640 1 --2- 172.21.15.96:0/1245912561 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f709015d270 0x7f70900b1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:00.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.313+0000 7f708ea00640 1 --2- 172.21.15.96:0/1245912561 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f70900ccb90 0x7f70901f1fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:00.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.313+0000 7f708ea00640 1 -- 172.21.15.96:0/1245912561 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f709015d270 msgr2=0x7f70900b1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:00.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.313+0000 7f708ea00640 1 --2- 172.21.15.96:0/1245912561 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f709015d270 0x7f70900b1da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:00.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.313+0000 7f708ea00640 1 -- 172.21.15.96:0/1245912561 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7090147b30 con 0x7f7090153510 2024-05-28T06:22:00.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.313+0000 7f708ea00640 1 --2- 172.21.15.96:0/1245912561 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7090153510 0x7f70901475f0 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7f707c00c430 tx=0x7f707c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:00.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.314+0000 7f7087400640 1 -- 172.21.15.96:0/1245912561 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f707c017070 con 0x7f7090153510 2024-05-28T06:22:00.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.314+0000 7f7087400640 1 -- 172.21.15.96:0/1245912561 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f707c005ce0 con 0x7f7090153510 2024-05-28T06:22:00.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.314+0000 7f7087400640 1 -- 172.21.15.96:0/1245912561 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f707c0072f0 con 0x7f7090153510 2024-05-28T06:22:00.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.314+0000 7f7094c00640 1 -- 172.21.15.96:0/1245912561 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7090155050 con 0x7f7090153510 2024-05-28T06:22:00.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.314+0000 7f7094c00640 1 -- 172.21.15.96:0/1245912561 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70901552f0 con 0x7f7090153510 2024-05-28T06:22:00.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.315+0000 7f7094c00640 1 -- 172.21.15.96:0/1245912561 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70900689e0 con 0x7f7090153510 2024-05-28T06:22:00.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.316+0000 7f7087400640 1 -- 172.21.15.96:0/1245912561 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 37) ==== 93182+0+0 (secure 0 0 0) 0x7f707c01a070 con 0x7f7090153510 2024-05-28T06:22:00.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.317+0000 7f7087400640 1 --2- 172.21.15.96:0/1245912561 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f707006e090 0x7f7070070550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:00.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.317+0000 7f7087400640 1 -- 172.21.15.96:0/1245912561 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 5821+0+0 (secure 0 0 0) 0x7f707c089e60 con 0x7f7090153510 2024-05-28T06:22:00.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.317+0000 7f708f400640 1 --2- 172.21.15.96:0/1245912561 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f707006e090 0x7f7070070550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:00.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.318+0000 7f708f400640 1 --2- 172.21.15.96:0/1245912561 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f707006e090 0x7f7070070550 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f7080006fd0 tx=0x7f7080008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:00.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.321+0000 7f7087400640 1 -- 172.21.15.96:0/1245912561 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7090147b30 con 0x7f7090153510 2024-05-28T06:22:00.513 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.518+0000 7f7094c00640 1 -- 172.21.15.96:0/1245912561 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f70900b3c40 con 0x7f7090153510 2024-05-28T06:22:00.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.519+0000 7f7087400640 1 -- 172.21.15.96:0/1245912561 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v104) ==== 104+0+11 (secure 0 0 0) 0x7f707c059cf0 con 0x7f7090153510 2024-05-28T06:22:00.515 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 32 2024-05-28T06:22:00.517 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.522+0000 7f7094c00640 1 -- 172.21.15.96:0/1245912561 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f707006e090 msgr2=0x7f7070070550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:00.517 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.522+0000 7f7094c00640 1 --2- 172.21.15.96:0/1245912561 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f707006e090 0x7f7070070550 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f7080006fd0 tx=0x7f7080008040 comp rx=0 tx=0).stop 2024-05-28T06:22:00.517 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.522+0000 7f7094c00640 1 -- 172.21.15.96:0/1245912561 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7090153510 msgr2=0x7f70901475f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:00.518 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.522+0000 7f7094c00640 1 --2- 172.21.15.96:0/1245912561 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7090153510 0x7f70901475f0 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7f707c00c430 tx=0x7f707c00c8f0 comp rx=0 tx=0).stop 2024-05-28T06:22:00.518 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.522+0000 7f7094c00640 1 -- 172.21.15.96:0/1245912561 shutdown_connections 2024-05-28T06:22:00.518 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.522+0000 7f7094c00640 1 --2- 172.21.15.96:0/1245912561 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f707006e090 0x7f7070070550 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:00.518 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.522+0000 7f7094c00640 1 --2- 172.21.15.96:0/1245912561 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f709015d270 0x7f70900b1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:00.518 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.522+0000 7f7094c00640 1 --2- 172.21.15.96:0/1245912561 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7090153510 0x7f70901475f0 unknown :-1 s=CLOSED pgs=1724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:00.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.522+0000 7f7094c00640 1 --2- 172.21.15.96:0/1245912561 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f70900ccb90 0x7f70901f1fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:00.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.522+0000 7f7094c00640 1 -- 172.21.15.96:0/1245912561 >> 172.21.15.96:0/1245912561 conn(0x7f709006b900 msgr2=0x7f70900cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:00.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.522+0000 7f7094c00640 1 -- 172.21.15.96:0/1245912561 shutdown_connections 2024-05-28T06:22:00.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.522+0000 7f7094c00640 1 -- 172.21.15.96:0/1245912561 wait complete. 2024-05-28T06:22:00.537 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-28T06:22:00.745 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.750+0000 7fee7b000640 1 Processor -- start 2024-05-28T06:22:00.746 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.750+0000 7fee7b000640 1 -- start start 2024-05-28T06:22:00.747 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.752+0000 7fee7b000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fee740d16a0 0x7fee740cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:00.747 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.752+0000 7fee7b000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fee740cf870 0x7fee740d25e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:00.747 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.752+0000 7fee7b000640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee7415ec50 con 0x7fee7415e870 2024-05-28T06:22:00.747 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.752+0000 7fee7b000640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fee740d2150 con 0x7fee740cf870 2024-05-28T06:22:00.747 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.752+0000 7fee7b000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fee740cfd20 con 0x7fee740d16a0 2024-05-28T06:22:00.747 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.752+0000 7fee79c00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fee7415e870 0x7fee740d0f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:37964/0 (socket says 172.21.15.96:37964) 2024-05-28T06:22:00.747 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.752+0000 7fee79c00640 1 -- 172.21.15.96:0/1549832060 learned_addr learned my addr 172.21.15.96:0/1549832060 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:22:00.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.753+0000 7fee79200640 1 --2- 172.21.15.96:0/1549832060 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fee740d16a0 0x7fee740cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:00.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.753+0000 7fee73e00640 1 -- 172.21.15.96:0/1549832060 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1321002748 0 0) 0x7fee7415ec50 con 0x7fee7415e870 2024-05-28T06:22:00.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.753+0000 7fee73e00640 1 -- 172.21.15.96:0/1549832060 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fee5c003520 con 0x7fee7415e870 2024-05-28T06:22:00.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.753+0000 7fee7a600640 1 --2- 172.21.15.96:0/1549832060 >> v2:172.21.15.123:3300/0 conn(0x7fee740cf870 0x7fee740d25e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:00.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.753+0000 7fee73e00640 1 -- 172.21.15.96:0/1549832060 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3993094214 0 0) 0x7fee5c003520 con 0x7fee7415e870 2024-05-28T06:22:00.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.753+0000 7fee73e00640 1 -- 172.21.15.96:0/1549832060 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fee740d16a0 msgr2=0x7fee740cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:00.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.753+0000 7fee73e00640 1 --2- 172.21.15.96:0/1549832060 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fee740d16a0 0x7fee740cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:00.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.753+0000 7fee73e00640 1 -- 172.21.15.96:0/1549832060 >> v2:172.21.15.123:3300/0 conn(0x7fee740cf870 msgr2=0x7fee740d25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:00.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.753+0000 7fee73e00640 1 --2- 172.21.15.96:0/1549832060 >> v2:172.21.15.123:3300/0 conn(0x7fee740cf870 0x7fee740d25e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:00.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.753+0000 7fee73e00640 1 -- 172.21.15.96:0/1549832060 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee740cff80 con 0x7fee7415e870 2024-05-28T06:22:00.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.753+0000 7fee73e00640 1 -- 172.21.15.96:0/1549832060 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7fee68001670 con 0x7fee7415e870 2024-05-28T06:22:00.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.753+0000 7fee73e00640 1 -- 172.21.15.96:0/1549832060 >> v1:172.21.15.96:6789/0 conn(0x7fee7415e870 legacy=0x7fee740d0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:00.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.753+0000 7fee73e00640 1 --2- 172.21.15.96:0/1549832060 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee5c0039b0 0x7fee5c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:00.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.753+0000 7fee73e00640 1 --2- 172.21.15.96:0/1549832060 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fee740d16a0 0x7fee5c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:00.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.754+0000 7fee73e00640 1 --2- 172.21.15.96:0/1549832060 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fee5c004d40 0x7fee5c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:00.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.754+0000 7fee73e00640 1 -- 172.21.15.96:0/1549832060 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fee740d2150 con 0x7fee5c004d40 2024-05-28T06:22:00.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.754+0000 7fee73e00640 1 -- 172.21.15.96:0/1549832060 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fee740cfd20 con 0x7fee5c0039b0 2024-05-28T06:22:00.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.754+0000 7fee73e00640 1 -- 172.21.15.96:0/1549832060 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fee5c00b9c0 con 0x7fee740d16a0 2024-05-28T06:22:00.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.754+0000 7fee7a600640 1 --2- 172.21.15.96:0/1549832060 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fee5c004d40 0x7fee5c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:00.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.754+0000 7fee79c00640 1 --2- 172.21.15.96:0/1549832060 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee5c0039b0 0x7fee5c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:00.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.754+0000 7fee79200640 1 --2- 172.21.15.96:0/1549832060 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fee740d16a0 0x7fee5c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:00.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.754+0000 7fee7a600640 1 -- 172.21.15.96:0/1549832060 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fee740d16a0 msgr2=0x7fee5c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:00.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.754+0000 7fee7a600640 1 --2- 172.21.15.96:0/1549832060 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fee740d16a0 0x7fee5c004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:00.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.754+0000 7fee7a600640 1 -- 172.21.15.96:0/1549832060 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee5c0039b0 msgr2=0x7fee5c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:00.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.754+0000 7fee7a600640 1 --2- 172.21.15.96:0/1549832060 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee5c0039b0 0x7fee5c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:00.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.754+0000 7fee7a600640 1 -- 172.21.15.96:0/1549832060 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee5c003520 con 0x7fee5c004d40 2024-05-28T06:22:00.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.755+0000 7fee7a600640 1 --2- 172.21.15.96:0/1549832060 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fee5c004d40 0x7fee5c0051c0 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7fee6c009fc0 tx=0x7fee5c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=475162f9c32d985f server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:00.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.756+0000 7fee73e00640 1 -- 172.21.15.96:0/1549832060 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee6c009c10 con 0x7fee5c004d40 2024-05-28T06:22:00.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.756+0000 7fee73e00640 1 -- 172.21.15.96:0/1549832060 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee6c036070 con 0x7fee5c004d40 2024-05-28T06:22:00.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.756+0000 7fee73e00640 1 -- 172.21.15.96:0/1549832060 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee6c02d070 con 0x7fee5c004d40 2024-05-28T06:22:00.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.756+0000 7fee7b000640 1 -- 172.21.15.96:0/1549832060 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fee5c004d40 msgr2=0x7fee5c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:00.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.756+0000 7fee7b000640 1 --2- 172.21.15.96:0/1549832060 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fee5c004d40 0x7fee5c0051c0 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7fee6c009fc0 tx=0x7fee5c005830 comp rx=0 tx=0).stop 2024-05-28T06:22:00.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.756+0000 7fee7a600640 1 -- 172.21.15.96:0/1549832060 reap_dead start 2024-05-28T06:22:00.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.756+0000 7fee7b000640 1 -- 172.21.15.96:0/1549832060 shutdown_connections 2024-05-28T06:22:00.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.756+0000 7fee7b000640 1 -- 172.21.15.96:0/1549832060 >> 172.21.15.96:0/1549832060 conn(0x7fee7408bfc0 msgr2=0x7fee740d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:00.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.757+0000 7fee7b000640 1 -- 172.21.15.96:0/1549832060 shutdown_connections 2024-05-28T06:22:00.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.757+0000 7fee7b000640 1 -- 172.21.15.96:0/1549832060 wait complete. 2024-05-28T06:22:00.757 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.758+0000 7fee7b000640 1 Processor -- start 2024-05-28T06:22:00.757 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.758+0000 7fee7b000640 1 -- start start 2024-05-28T06:22:00.757 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.759+0000 7fee7b000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fee5c004d40 0x7fee740cc060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:00.757 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.759+0000 7fee7b000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee74157a30 0x7fee740ca130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:00.757 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.759+0000 7fee7b000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fee740cd910 0x7fee740cdd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:00.757 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.759+0000 7fee7b000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fee740d0820 con 0x7fee740cd910 2024-05-28T06:22:00.757 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.759+0000 7fee7b000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fee740d0540 con 0x7fee74157a30 2024-05-28T06:22:00.757 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.759+0000 7fee7b000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fee740d4fb0 con 0x7fee5c004d40 2024-05-28T06:22:00.757 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.760+0000 7fee7a600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fee740cd910 0x7fee740cdd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:00.757 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.760+0000 7fee7a600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fee740cd910 0x7fee740cdd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35876/0 (socket says 172.21.15.96:35876) 2024-05-28T06:22:00.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.760+0000 7fee7a600640 1 -- 172.21.15.96:0/1048914438 learned_addr learned my addr 172.21.15.96:0/1048914438 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:22:00.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.760+0000 7fee79c00640 1 --2- 172.21.15.96:0/1048914438 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fee5c004d40 0x7fee740cc060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:00.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.760+0000 7fee79200640 1 --2- 172.21.15.96:0/1048914438 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee74157a30 0x7fee740ca130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:00.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.760+0000 7fee7a600640 1 -- 172.21.15.96:0/1048914438 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fee5c004d40 msgr2=0x7fee740cc060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:00.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.760+0000 7fee7a600640 1 --2- 172.21.15.96:0/1048914438 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fee5c004d40 0x7fee740cc060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:00.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.760+0000 7fee7a600640 1 -- 172.21.15.96:0/1048914438 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee74157a30 msgr2=0x7fee740ca130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:00.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.760+0000 7fee7a600640 1 --2- 172.21.15.96:0/1048914438 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee74157a30 0x7fee740ca130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:00.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.760+0000 7fee7a600640 1 -- 172.21.15.96:0/1048914438 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee740cc640 con 0x7fee740cd910 2024-05-28T06:22:00.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.760+0000 7fee7a600640 1 --2- 172.21.15.96:0/1048914438 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fee740cd910 0x7fee740cdd90 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7fee6c02af10 tx=0x7fee6c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:00.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.761+0000 7fee71600640 1 -- 172.21.15.96:0/1048914438 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee6c009c10 con 0x7fee740cd910 2024-05-28T06:22:00.759 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.761+0000 7fee71600640 1 -- 172.21.15.96:0/1048914438 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee6c003b90 con 0x7fee740cd910 2024-05-28T06:22:00.759 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.761+0000 7fee71600640 1 -- 172.21.15.96:0/1048914438 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee6c004020 con 0x7fee740cd910 2024-05-28T06:22:00.759 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.761+0000 7fee7b000640 1 -- 172.21.15.96:0/1048914438 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee740cc920 con 0x7fee740cd910 2024-05-28T06:22:00.759 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.761+0000 7fee7b000640 1 -- 172.21.15.96:0/1048914438 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee740ccee0 con 0x7fee740cd910 2024-05-28T06:22:00.759 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.762+0000 7fee7b000640 1 -- 172.21.15.96:0/1048914438 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee740689e0 con 0x7fee740cd910 2024-05-28T06:22:00.760 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.763+0000 7fee71600640 1 -- 172.21.15.96:0/1048914438 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 37) ==== 93182+0+0 (secure 0 0 0) 0x7fee6c005d10 con 0x7fee740cd910 2024-05-28T06:22:00.760 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.764+0000 7fee71600640 1 --2- 172.21.15.96:0/1048914438 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fee5406e090 0x7fee54070550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:00.760 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.765+0000 7fee71600640 1 -- 172.21.15.96:0/1048914438 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 5821+0+0 (secure 0 0 0) 0x7fee6c0abb50 con 0x7fee740cd910 2024-05-28T06:22:00.760 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.765+0000 7fee79c00640 1 --2- 172.21.15.96:0/1048914438 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fee5406e090 0x7fee54070550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:00.761 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.766+0000 7fee79c00640 1 --2- 172.21.15.96:0/1048914438 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fee5406e090 0x7fee54070550 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fee68000c00 tx=0x7fee68009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:00.763 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.769+0000 7fee71600640 1 -- 172.21.15.96:0/1048914438 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fee6c07b9e0 con 0x7fee740cd910 2024-05-28T06:22:00.964 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.970+0000 7fee7b000640 1 -- 172.21.15.96:0/1048914438 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fee740ba500 con 0x7fee740cd910 2024-05-28T06:22:00.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.971+0000 7fee71600640 1 -- 172.21.15.96:0/1048914438 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v104) ==== 100+0+11 (secure 0 0 0) 0x7fee6c07b2e0 con 0x7fee740cd910 2024-05-28T06:22:00.965 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 64 2024-05-28T06:22:00.968 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.973+0000 7fee7b000640 1 -- 172.21.15.96:0/1048914438 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fee5406e090 msgr2=0x7fee54070550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:00.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.973+0000 7fee7b000640 1 --2- 172.21.15.96:0/1048914438 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fee5406e090 0x7fee54070550 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fee68000c00 tx=0x7fee68009040 comp rx=0 tx=0).stop 2024-05-28T06:22:00.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.973+0000 7fee7b000640 1 -- 172.21.15.96:0/1048914438 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fee740cd910 msgr2=0x7fee740cdd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:00.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.973+0000 7fee7b000640 1 --2- 172.21.15.96:0/1048914438 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fee740cd910 0x7fee740cdd90 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7fee6c02af10 tx=0x7fee6c009750 comp rx=0 tx=0).stop 2024-05-28T06:22:00.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.974+0000 7fee7b000640 1 -- 172.21.15.96:0/1048914438 shutdown_connections 2024-05-28T06:22:00.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.974+0000 7fee7b000640 1 --2- 172.21.15.96:0/1048914438 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fee5406e090 0x7fee54070550 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:00.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.974+0000 7fee7b000640 1 --2- 172.21.15.96:0/1048914438 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fee740cd910 0x7fee740cdd90 unknown :-1 s=CLOSED pgs=1727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:00.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.974+0000 7fee7b000640 1 --2- 172.21.15.96:0/1048914438 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee74157a30 0x7fee740ca130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:00.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.974+0000 7fee7b000640 1 --2- 172.21.15.96:0/1048914438 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fee5c004d40 0x7fee740cc060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:00.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.974+0000 7fee7b000640 1 -- 172.21.15.96:0/1048914438 >> 172.21.15.96:0/1048914438 conn(0x7fee7408bfc0 msgr2=0x7fee740d2cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:00.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.974+0000 7fee7b000640 1 -- 172.21.15.96:0/1048914438 shutdown_connections 2024-05-28T06:22:00.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:00.974+0000 7fee7b000640 1 -- 172.21.15.96:0/1048914438 wait complete. 2024-05-28T06:22:00.987 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-28T06:22:01.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.184+0000 7f6b47e00640 1 Processor -- start 2024-05-28T06:22:01.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.184+0000 7f6b47e00640 1 -- start start 2024-05-28T06:22:01.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.186+0000 7f6b47e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f6b4806ee40 0x7f6b480ad230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:01.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.186+0000 7f6b47e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6b4806f2f0 0x7f6b480ad770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:01.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.186+0000 7f6b47e00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b48156f10 con 0x7f6b48156b30 2024-05-28T06:22:01.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.186+0000 7f6b47e00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f6b4806dda0 con 0x7f6b4806ee40 2024-05-28T06:22:01.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.186+0000 7f6b47e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f6b4806e080 con 0x7f6b4806f2f0 2024-05-28T06:22:01.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.186+0000 7f6b46a00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f6b48156b30 0x7f6b480af6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:37978/0 (socket says 172.21.15.96:37978) 2024-05-28T06:22:01.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.186+0000 7f6b46a00640 1 -- 172.21.15.96:0/219745529 learned_addr learned my addr 172.21.15.96:0/219745529 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:22:01.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.187+0000 7f6b45600640 1 -- 172.21.15.96:0/219745529 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3866678896 0 0) 0x7f6b48156f10 con 0x7f6b48156b30 2024-05-28T06:22:01.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.187+0000 7f6b46000640 1 --2- 172.21.15.96:0/219745529 >> v2:172.21.15.123:3300/0 conn(0x7f6b4806ee40 0x7f6b480ad230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:01.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.187+0000 7f6b47400640 1 --2- 172.21.15.96:0/219745529 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6b4806f2f0 0x7f6b480ad770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:01.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.187+0000 7f6b45600640 1 -- 172.21.15.96:0/219745529 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6b2c003520 con 0x7f6b48156b30 2024-05-28T06:22:01.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.187+0000 7f6b45600640 1 -- 172.21.15.96:0/219745529 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4172793332 0 0) 0x7f6b2c003520 con 0x7f6b48156b30 2024-05-28T06:22:01.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.187+0000 7f6b45600640 1 -- 172.21.15.96:0/219745529 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6b4806f2f0 msgr2=0x7f6b480ad770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:01.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.187+0000 7f6b45600640 1 --2- 172.21.15.96:0/219745529 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6b4806f2f0 0x7f6b480ad770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:01.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.187+0000 7f6b45600640 1 -- 172.21.15.96:0/219745529 >> v2:172.21.15.123:3300/0 conn(0x7f6b4806ee40 msgr2=0x7f6b480ad230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:01.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.187+0000 7f6b45600640 1 --2- 172.21.15.96:0/219745529 >> v2:172.21.15.123:3300/0 conn(0x7f6b4806ee40 0x7f6b480ad230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:01.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.187+0000 7f6b45600640 1 -- 172.21.15.96:0/219745529 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b4806e360 con 0x7f6b48156b30 2024-05-28T06:22:01.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.187+0000 7f6b45600640 1 -- 172.21.15.96:0/219745529 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f6b38001670 con 0x7f6b48156b30 2024-05-28T06:22:01.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.187+0000 7f6b45600640 1 -- 172.21.15.96:0/219745529 >> v1:172.21.15.96:6789/0 conn(0x7f6b48156b30 legacy=0x7f6b480af6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:01.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.187+0000 7f6b45600640 1 --2- 172.21.15.96:0/219745529 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6b2c0039b0 0x7f6b2c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:01.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.187+0000 7f6b45600640 1 --2- 172.21.15.96:0/219745529 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6b4806f2f0 0x7f6b2c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:01.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.187+0000 7f6b45600640 1 --2- 172.21.15.96:0/219745529 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6b2c004d40 0x7f6b2c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:01.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.187+0000 7f6b45600640 1 -- 172.21.15.96:0/219745529 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f6b4806dda0 con 0x7f6b2c0039b0 2024-05-28T06:22:01.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.187+0000 7f6b45600640 1 -- 172.21.15.96:0/219745529 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f6b4806e080 con 0x7f6b2c004d40 2024-05-28T06:22:01.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.187+0000 7f6b45600640 1 -- 172.21.15.96:0/219745529 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f6b2c00ba70 con 0x7f6b4806f2f0 2024-05-28T06:22:01.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.188+0000 7f6b46a00640 1 --2- 172.21.15.96:0/219745529 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6b2c0039b0 0x7f6b2c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:01.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.188+0000 7f6b46a00640 1 -- 172.21.15.96:0/219745529 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6b4806f2f0 msgr2=0x7f6b2c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:01.185 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.188+0000 7f6b46a00640 1 --2- 172.21.15.96:0/219745529 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6b4806f2f0 0x7f6b2c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:01.185 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.188+0000 7f6b46a00640 1 -- 172.21.15.96:0/219745529 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6b2c004d40 msgr2=0x7f6b2c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:01.185 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.188+0000 7f6b46a00640 1 --2- 172.21.15.96:0/219745529 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6b2c004d40 0x7f6b2c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:01.185 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.188+0000 7f6b46a00640 1 -- 172.21.15.96:0/219745529 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b2c003520 con 0x7f6b2c0039b0 2024-05-28T06:22:01.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.189+0000 7f6b46a00640 1 --2- 172.21.15.96:0/219745529 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6b2c0039b0 0x7f6b2c003dd0 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7f6b3800c210 tx=0x7f6b2c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcd0eadb7c960b69 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:01.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.189+0000 7f6b45600640 1 -- 172.21.15.96:0/219745529 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b38007d50 con 0x7f6b2c0039b0 2024-05-28T06:22:01.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.189+0000 7f6b45600640 1 -- 172.21.15.96:0/219745529 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b38005af0 con 0x7f6b2c0039b0 2024-05-28T06:22:01.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.189+0000 7f6b45600640 1 -- 172.21.15.96:0/219745529 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b38007070 con 0x7f6b2c0039b0 2024-05-28T06:22:01.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.190+0000 7f6b47e00640 1 -- 172.21.15.96:0/219745529 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6b2c0039b0 msgr2=0x7f6b2c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:01.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.190+0000 7f6b47e00640 1 --2- 172.21.15.96:0/219745529 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6b2c0039b0 0x7f6b2c003dd0 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7f6b3800c210 tx=0x7f6b2c005830 comp rx=0 tx=0).stop 2024-05-28T06:22:01.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.190+0000 7f6b47400640 1 -- 172.21.15.96:0/219745529 reap_dead start 2024-05-28T06:22:01.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.190+0000 7f6b47e00640 1 -- 172.21.15.96:0/219745529 shutdown_connections 2024-05-28T06:22:01.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.190+0000 7f6b47e00640 1 -- 172.21.15.96:0/219745529 >> 172.21.15.96:0/219745529 conn(0x7f6b480b1fc0 msgr2=0x7f6b480b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:01.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.190+0000 7f6b47e00640 1 -- 172.21.15.96:0/219745529 shutdown_connections 2024-05-28T06:22:01.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.190+0000 7f6b47e00640 1 -- 172.21.15.96:0/219745529 wait complete. 2024-05-28T06:22:01.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.192+0000 7f6b47e00640 1 Processor -- start 2024-05-28T06:22:01.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.192+0000 7f6b47e00640 1 -- start start 2024-05-28T06:22:01.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.193+0000 7f6b47e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6b2c0039b0 0x7f6b48099a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:01.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.193+0000 7f6b47e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6b4815bba0 0x7f6b4815d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:01.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.193+0000 7f6b47e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6b480b1af0 0x7f6b4815d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:01.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.193+0000 7f6b47e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f6b480b2db0 con 0x7f6b480b1af0 2024-05-28T06:22:01.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.193+0000 7f6b47e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f6b480b2c30 con 0x7f6b4815bba0 2024-05-28T06:22:01.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.193+0000 7f6b47e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f6b480ad770 con 0x7f6b2c0039b0 2024-05-28T06:22:01.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.193+0000 7f6b47400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6b480b1af0 0x7f6b4815d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:01.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.193+0000 7f6b47400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6b480b1af0 0x7f6b4815d870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35898/0 (socket says 172.21.15.96:35898) 2024-05-28T06:22:01.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.193+0000 7f6b47400640 1 -- 172.21.15.96:0/3045333318 learned_addr learned my addr 172.21.15.96:0/3045333318 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:22:01.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.193+0000 7f6b47400640 1 -- 172.21.15.96:0/3045333318 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6b2c0039b0 msgr2=0x7f6b48099a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:01.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.193+0000 7f6b46a00640 1 --2- 172.21.15.96:0/3045333318 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6b2c0039b0 0x7f6b48099a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:01.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.193+0000 7f6b46000640 1 --2- 172.21.15.96:0/3045333318 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6b4815bba0 0x7f6b4815d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:01.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.193+0000 7f6b47400640 1 --2- 172.21.15.96:0/3045333318 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6b2c0039b0 0x7f6b48099a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:01.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.193+0000 7f6b47400640 1 -- 172.21.15.96:0/3045333318 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6b4815bba0 msgr2=0x7f6b4815d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:01.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.193+0000 7f6b47400640 1 --2- 172.21.15.96:0/3045333318 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6b4815bba0 0x7f6b4815d330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:01.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.193+0000 7f6b47400640 1 -- 172.21.15.96:0/3045333318 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b481e9960 con 0x7f6b480b1af0 2024-05-28T06:22:01.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.194+0000 7f6b47400640 1 --2- 172.21.15.96:0/3045333318 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6b480b1af0 0x7f6b4815d870 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7f6b48071410 tx=0x7f6b40007f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:01.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.194+0000 7f6b3ea00640 1 -- 172.21.15.96:0/3045333318 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b4000ec50 con 0x7f6b480b1af0 2024-05-28T06:22:01.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.194+0000 7f6b3ea00640 1 -- 172.21.15.96:0/3045333318 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b40019070 con 0x7f6b480b1af0 2024-05-28T06:22:01.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.194+0000 7f6b3ea00640 1 -- 172.21.15.96:0/3045333318 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b40014420 con 0x7f6b480b1af0 2024-05-28T06:22:01.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.194+0000 7f6b47e00640 1 -- 172.21.15.96:0/3045333318 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b481e5f60 con 0x7f6b480b1af0 2024-05-28T06:22:01.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.194+0000 7f6b47e00640 1 -- 172.21.15.96:0/3045333318 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b481e9c40 con 0x7f6b480b1af0 2024-05-28T06:22:01.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.195+0000 7f6b47e00640 1 -- 172.21.15.96:0/3045333318 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b480689e0 con 0x7f6b480b1af0 2024-05-28T06:22:01.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.196+0000 7f6b3ea00640 1 -- 172.21.15.96:0/3045333318 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 37) ==== 93182+0+0 (secure 0 0 0) 0x7f6b400145e0 con 0x7f6b480b1af0 2024-05-28T06:22:01.192 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.197+0000 7f6b3ea00640 1 --2- 172.21.15.96:0/3045333318 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6b2806e090 0x7f6b28070550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:01.192 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.197+0000 7f6b3ea00640 1 -- 172.21.15.96:0/3045333318 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 5821+0+0 (secure 0 0 0) 0x7f6b4008ad10 con 0x7f6b480b1af0 2024-05-28T06:22:01.192 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.198+0000 7f6b46a00640 1 --2- 172.21.15.96:0/3045333318 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6b2806e090 0x7f6b28070550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:01.193 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.199+0000 7f6b46a00640 1 --2- 172.21.15.96:0/3045333318 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6b2806e090 0x7f6b28070550 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f6b38033000 tx=0x7f6b38009030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:01.196 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.201+0000 7f6b3ea00640 1 -- 172.21.15.96:0/3045333318 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6b4005aba0 con 0x7f6b480b1af0 2024-05-28T06:22:01.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.442+0000 7f6b47e00640 1 -- 172.21.15.96:0/3045333318 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6b48155830 con 0x7f6b480b1af0 2024-05-28T06:22:01.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.444+0000 7f6b3ea00640 1 -- 172.21.15.96:0/3045333318 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) ==== 94+0+5189 (secure 0 0 0) 0x7f6b4005a4a0 con 0x7f6b480b1af0 2024-05-28T06:22:01.438 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:22:01.439 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":63,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5861,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/1001966057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6833","nonce":1001966057},{"type":"v1","addr":"172.21.15.96:6835","nonce":1001966057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":57},{"gid":5864,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6836/4285246962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":4285246962},{"type":"v1","addr":"172.21.15.123:6836","nonce":4285246962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":57},{"gid":6445,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6834/3302807355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":3302807355},{"type":"v1","addr":"172.21.15.96:6834","nonce":3302807355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-28T06:20:18.363024+0000","modified":"2024-05-28T06:20:21.426136+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8739},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8739":{"gid":8739,"name":"b","rank":0,"incarnation":58,"state":"up:active","state_seq":3,"addr":"172.21.15.123:6837/3372660768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6835","nonce":3372660768},{"type":"v1","addr":"172.21.15.123:6837","nonce":3372660768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8739,"qdb_cluster":[8739]},"id":7}]} 2024-05-28T06:22:01.439 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 63 2024-05-28T06:22:01.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.446+0000 7f6b47e00640 1 -- 172.21.15.96:0/3045333318 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6b2806e090 msgr2=0x7f6b28070550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:01.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.446+0000 7f6b47e00640 1 --2- 172.21.15.96:0/3045333318 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6b2806e090 0x7f6b28070550 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f6b38033000 tx=0x7f6b38009030 comp rx=0 tx=0).stop 2024-05-28T06:22:01.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.446+0000 7f6b47e00640 1 -- 172.21.15.96:0/3045333318 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6b480b1af0 msgr2=0x7f6b4815d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:01.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.446+0000 7f6b47e00640 1 --2- 172.21.15.96:0/3045333318 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6b480b1af0 0x7f6b4815d870 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7f6b48071410 tx=0x7f6b40007f20 comp rx=0 tx=0).stop 2024-05-28T06:22:01.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.446+0000 7f6b47e00640 1 -- 172.21.15.96:0/3045333318 shutdown_connections 2024-05-28T06:22:01.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.446+0000 7f6b47e00640 1 --2- 172.21.15.96:0/3045333318 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6b2806e090 0x7f6b28070550 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:01.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.446+0000 7f6b47e00640 1 --2- 172.21.15.96:0/3045333318 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6b480b1af0 0x7f6b4815d870 unknown :-1 s=CLOSED pgs=1730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:01.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.446+0000 7f6b47e00640 1 --2- 172.21.15.96:0/3045333318 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6b4815bba0 0x7f6b4815d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:01.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.446+0000 7f6b47e00640 1 --2- 172.21.15.96:0/3045333318 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6b2c0039b0 0x7f6b48099a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:01.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.446+0000 7f6b47e00640 1 -- 172.21.15.96:0/3045333318 >> 172.21.15.96:0/3045333318 conn(0x7f6b480b1fc0 msgr2=0x7f6b480adde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:01.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.447+0000 7f6b47e00640 1 -- 172.21.15.96:0/3045333318 shutdown_connections 2024-05-28T06:22:01.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.447+0000 7f6b47e00640 1 -- 172.21.15.96:0/3045333318 wait complete. 2024-05-28T06:22:01.460 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:22:01.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.660+0000 7f320e400640 1 Processor -- start 2024-05-28T06:22:01.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.660+0000 7f320e400640 1 -- start start 2024-05-28T06:22:01.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.661+0000 7f320e400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f3208073310 0x7f32080b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:01.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.661+0000 7f320e400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f320806dc20 0x7f320815f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:01.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.661+0000 7f320e400640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32080afd60 con 0x7f320815b3a0 2024-05-28T06:22:01.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.661+0000 7f320e400640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f320806f980 con 0x7f3208073310 2024-05-28T06:22:01.657 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.661+0000 7f320e400640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f32080ae0c0 con 0x7f320806dc20 2024-05-28T06:22:01.657 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.662+0000 7f31ffe00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f3208073310 0x7f32080b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:01.657 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.662+0000 7f31ffe00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f3208073310 0x7f32080b32c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35912/0 (socket says 172.21.15.96:35912) 2024-05-28T06:22:01.657 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.662+0000 7f31ffe00640 1 -- 172.21.15.96:0/822789859 learned_addr learned my addr 172.21.15.96:0/822789859 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:22:01.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.662+0000 7f320da00640 1 --2- 172.21.15.96:0/822789859 >> v2:172.21.15.123:3300/0 conn(0x7f320806dc20 0x7f320815f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:01.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.662+0000 7f31ffe00640 1 -- 172.21.15.96:0/822789859 >> v2:172.21.15.123:3300/0 conn(0x7f320806dc20 msgr2=0x7f320815f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:01.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.662+0000 7f31ffe00640 1 --2- 172.21.15.96:0/822789859 >> v2:172.21.15.123:3300/0 conn(0x7f320806dc20 0x7f320815f9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:01.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.662+0000 7f31ffe00640 1 -- 172.21.15.96:0/822789859 >> v1:172.21.15.123:6789/0 conn(0x7f320815b3a0 legacy=0x7f3208072c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:01.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.662+0000 7f31ffe00640 1 -- 172.21.15.96:0/822789859 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32080af7a0 con 0x7f3208073310 2024-05-28T06:22:01.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.663+0000 7f31ffe00640 1 --2- 172.21.15.96:0/822789859 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3208073310 0x7f32080b32c0 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7f320806d6e0 tx=0x7f31f002c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d189d34e18fb65f8 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:01.659 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.664+0000 7f3207e00640 1 -- 172.21.15.96:0/822789859 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31f0038070 con 0x7f3208073310 2024-05-28T06:22:01.659 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.664+0000 7f3207e00640 1 -- 172.21.15.96:0/822789859 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31f0031710 con 0x7f3208073310 2024-05-28T06:22:01.659 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.664+0000 7f3207e00640 1 -- 172.21.15.96:0/822789859 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31f0031ae0 con 0x7f3208073310 2024-05-28T06:22:01.659 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.664+0000 7f320e400640 1 -- 172.21.15.96:0/822789859 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3208073310 msgr2=0x7f32080b32c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:01.659 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.664+0000 7f320e400640 1 --2- 172.21.15.96:0/822789859 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3208073310 0x7f32080b32c0 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7f320806d6e0 tx=0x7f31f002c440 comp rx=0 tx=0).stop 2024-05-28T06:22:01.659 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.664+0000 7f320e400640 1 -- 172.21.15.96:0/822789859 shutdown_connections 2024-05-28T06:22:01.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.664+0000 7f320e400640 1 --2- 172.21.15.96:0/822789859 >> v2:172.21.15.123:3300/0 conn(0x7f320806dc20 0x7f320815f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:01.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.664+0000 7f320e400640 1 --2- 172.21.15.96:0/822789859 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3208073310 0x7f32080b32c0 unknown :-1 s=CLOSED pgs=1731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:01.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.664+0000 7f320e400640 1 -- 172.21.15.96:0/822789859 >> 172.21.15.96:0/822789859 conn(0x7f32080b1fc0 msgr2=0x7f32080b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:01.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.664+0000 7f320e400640 1 -- 172.21.15.96:0/822789859 shutdown_connections 2024-05-28T06:22:01.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.665+0000 7f320e400640 1 -- 172.21.15.96:0/822789859 wait complete. 2024-05-28T06:22:01.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.666+0000 7f320e400640 1 Processor -- start 2024-05-28T06:22:01.661 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.666+0000 7f320e400640 1 -- start start 2024-05-28T06:22:01.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.667+0000 7f320e400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f320806dc20 0x7f32080cefe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:01.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.667+0000 7f320e400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3208073310 0x7f32081566a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:01.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.667+0000 7f320e400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f320815b3a0 0x7f3208157bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:01.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.667+0000 7f320e400640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f320815b780 con 0x7f320806dc20 2024-05-28T06:22:01.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.667+0000 7f320e400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f320815fee0 con 0x7f320815b3a0 2024-05-28T06:22:01.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.667+0000 7f320e400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f32080ae550 con 0x7f3208073310 2024-05-28T06:22:01.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.667+0000 7f320d000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f320806dc20 0x7f32080cefe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:01.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.667+0000 7f320d000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f320806dc20 0x7f32080cefe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35914/0 (socket says 172.21.15.96:35914) 2024-05-28T06:22:01.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.667+0000 7f320d000640 1 -- 172.21.15.96:0/2141664657 learned_addr learned my addr 172.21.15.96:0/2141664657 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:22:01.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.668+0000 7f320da00640 1 --2- 172.21.15.96:0/2141664657 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f320815b3a0 0x7f3208157bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:01.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.668+0000 7f31ffe00640 1 --2- 172.21.15.96:0/2141664657 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3208073310 0x7f32081566a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:01.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.668+0000 7f320d000640 1 -- 172.21.15.96:0/2141664657 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3208073310 msgr2=0x7f32081566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:01.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.668+0000 7f320d000640 1 --2- 172.21.15.96:0/2141664657 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3208073310 0x7f32081566a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:01.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.668+0000 7f320d000640 1 -- 172.21.15.96:0/2141664657 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f320815b3a0 msgr2=0x7f3208157bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:01.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.668+0000 7f320d000640 1 --2- 172.21.15.96:0/2141664657 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f320815b3a0 0x7f3208157bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:01.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.668+0000 7f320d000640 1 -- 172.21.15.96:0/2141664657 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32080cfad0 con 0x7f320806dc20 2024-05-28T06:22:01.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.668+0000 7f320d000640 1 --2- 172.21.15.96:0/2141664657 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f320806dc20 0x7f32080cefe0 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7f31f800b3e0 tx=0x7f31f800b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:01.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.668+0000 7f3205600640 1 -- 172.21.15.96:0/2141664657 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31f8015020 con 0x7f320806dc20 2024-05-28T06:22:01.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.668+0000 7f3205600640 1 -- 172.21.15.96:0/2141664657 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31f8011850 con 0x7f320806dc20 2024-05-28T06:22:01.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.668+0000 7f3205600640 1 -- 172.21.15.96:0/2141664657 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31f8011c40 con 0x7f320806dc20 2024-05-28T06:22:01.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.668+0000 7f320e400640 1 -- 172.21.15.96:0/2141664657 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32080ce140 con 0x7f320806dc20 2024-05-28T06:22:01.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.669+0000 7f320e400640 1 -- 172.21.15.96:0/2141664657 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3208157640 con 0x7f320806dc20 2024-05-28T06:22:01.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.670+0000 7f320e400640 1 -- 172.21.15.96:0/2141664657 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32080689e0 con 0x7f320806dc20 2024-05-28T06:22:01.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.671+0000 7f3205600640 1 -- 172.21.15.96:0/2141664657 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 37) ==== 93182+0+0 (secure 0 0 0) 0x7f31f8018020 con 0x7f320806dc20 2024-05-28T06:22:01.666 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.672+0000 7f3205600640 1 --2- 172.21.15.96:0/2141664657 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f31e406e090 0x7f31e4070550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:01.666 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.672+0000 7f3205600640 1 -- 172.21.15.96:0/2141664657 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 5821+0+0 (secure 0 0 0) 0x7f31f808a920 con 0x7f320806dc20 2024-05-28T06:22:01.667 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.672+0000 7f31ffe00640 1 --2- 172.21.15.96:0/2141664657 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f31e406e090 0x7f31e4070550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:01.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.673+0000 7f31ffe00640 1 --2- 172.21.15.96:0/2141664657 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f31e406e090 0x7f31e4070550 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f31f0035000 tx=0x7f31f0008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:01.671 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.676+0000 7f3205600640 1 -- 172.21.15.96:0/2141664657 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f31f801f020 con 0x7f320806dc20 2024-05-28T06:22:01.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.879+0000 7f320e400640 1 -- 172.21.15.96:0/2141664657 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f32080b2e90 con 0x7f320806dc20 2024-05-28T06:22:01.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.882+0000 7f3205600640 1 -- 172.21.15.96:0/2141664657 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v104) ==== 74+0+16831 (secure 0 0 0) 0x7f31f80172c0 con 0x7f320806dc20 2024-05-28T06:22:01.877 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:22:01.877 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":104,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:21:58.794459+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-05-28T06:20:16.097747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-05-28T06:20:17.338574+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"104","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":104,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":13,"snaps":[{"begin":2,"length":2}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:22:01.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.885+0000 7f320e400640 1 -- 172.21.15.96:0/2141664657 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f31e406e090 msgr2=0x7f31e4070550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:01.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.885+0000 7f320e400640 1 --2- 172.21.15.96:0/2141664657 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f31e406e090 0x7f31e4070550 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f31f0035000 tx=0x7f31f0008030 comp rx=0 tx=0).stop 2024-05-28T06:22:01.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.885+0000 7f320e400640 1 -- 172.21.15.96:0/2141664657 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f320806dc20 msgr2=0x7f32080cefe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:01.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.885+0000 7f320e400640 1 --2- 172.21.15.96:0/2141664657 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f320806dc20 0x7f32080cefe0 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7f31f800b3e0 tx=0x7f31f800b8a0 comp rx=0 tx=0).stop 2024-05-28T06:22:01.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.885+0000 7f320e400640 1 -- 172.21.15.96:0/2141664657 shutdown_connections 2024-05-28T06:22:01.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.885+0000 7f320e400640 1 --2- 172.21.15.96:0/2141664657 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f31e406e090 0x7f31e4070550 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:01.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.885+0000 7f320e400640 1 --2- 172.21.15.96:0/2141664657 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f320815b3a0 0x7f3208157bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:01.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.885+0000 7f320e400640 1 --2- 172.21.15.96:0/2141664657 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3208073310 0x7f32081566a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:01.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.885+0000 7f320e400640 1 --2- 172.21.15.96:0/2141664657 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f320806dc20 0x7f32080cefe0 unknown :-1 s=CLOSED pgs=1732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:01.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.885+0000 7f320e400640 1 -- 172.21.15.96:0/2141664657 >> 172.21.15.96:0/2141664657 conn(0x7f32080b1fc0 msgr2=0x7f32080b33f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:01.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.885+0000 7f320e400640 1 -- 172.21.15.96:0/2141664657 shutdown_connections 2024-05-28T06:22:01.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:01.885+0000 7f320e400640 1 -- 172.21.15.96:0/2141664657 wait complete. 2024-05-28T06:22:01.900 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-05-28T06:22:01.900 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-28T06:22:02.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.106+0000 7f6423200640 1 Processor -- start 2024-05-28T06:22:02.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.106+0000 7f6423200640 1 -- start start 2024-05-28T06:22:02.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.107+0000 7f6423200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f642415b3a0 0x7f64240b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:02.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.107+0000 7f6423200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f642406dc20 0x7f642415f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:02.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.108+0000 7f6423200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64240afd60 con 0x7f6424072c00 2024-05-28T06:22:02.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.108+0000 7f6423200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f642406f980 con 0x7f642415b3a0 2024-05-28T06:22:02.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.108+0000 7f6423200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f64240ae0c0 con 0x7f642406dc20 2024-05-28T06:22:02.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.108+0000 7f6421e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f642415b3a0 0x7f64240b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:02.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.108+0000 7f6422800640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f6424072c00 0x7f64240ae550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:37986/0 (socket says 172.21.15.96:37986) 2024-05-28T06:22:02.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.108+0000 7f6422800640 1 -- 172.21.15.96:0/434665948 learned_addr learned my addr 172.21.15.96:0/434665948 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:22:02.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.108+0000 7f641be00640 1 --2- 172.21.15.96:0/434665948 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f642406dc20 0x7f642415f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:02.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.108+0000 7f6421e00640 1 -- 172.21.15.96:0/434665948 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f642406dc20 msgr2=0x7f642415f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:02.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.108+0000 7f6421e00640 1 --2- 172.21.15.96:0/434665948 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f642406dc20 0x7f642415f9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:02.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.108+0000 7f6421e00640 1 -- 172.21.15.96:0/434665948 >> v1:172.21.15.96:6789/0 conn(0x7f6424072c00 legacy=0x7f64240ae550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:02.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.108+0000 7f6421400640 1 -- 172.21.15.96:0/434665948 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 915567697 0 0) 0x7f64240afd60 con 0x7f6424072c00 2024-05-28T06:22:02.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.108+0000 7f6421e00640 1 -- 172.21.15.96:0/434665948 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64240af7a0 con 0x7f642415b3a0 2024-05-28T06:22:02.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.109+0000 7f6421e00640 1 --2- 172.21.15.96:0/434665948 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f642415b3a0 0x7f64240b32c0 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7f64140097a0 tx=0x7f641402a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8611d7b8e28e19e5 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:02.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.110+0000 7f6421400640 1 -- 172.21.15.96:0/434665948 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6414037070 con 0x7f642415b3a0 2024-05-28T06:22:02.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.110+0000 7f6421400640 1 -- 172.21.15.96:0/434665948 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6414032440 con 0x7f642415b3a0 2024-05-28T06:22:02.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.110+0000 7f6421400640 1 -- 172.21.15.96:0/434665948 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6414032810 con 0x7f642415b3a0 2024-05-28T06:22:02.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.110+0000 7f6423200640 1 -- 172.21.15.96:0/434665948 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f642415b3a0 msgr2=0x7f64240b32c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:02.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.110+0000 7f6423200640 1 --2- 172.21.15.96:0/434665948 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f642415b3a0 0x7f64240b32c0 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7f64140097a0 tx=0x7f641402a180 comp rx=0 tx=0).stop 2024-05-28T06:22:02.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.110+0000 7f6423200640 1 -- 172.21.15.96:0/434665948 shutdown_connections 2024-05-28T06:22:02.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.110+0000 7f6423200640 1 --2- 172.21.15.96:0/434665948 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f642406dc20 0x7f642415f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:02.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.110+0000 7f6423200640 1 --2- 172.21.15.96:0/434665948 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f642415b3a0 0x7f64240b32c0 unknown :-1 s=CLOSED pgs=1734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:02.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.110+0000 7f6423200640 1 -- 172.21.15.96:0/434665948 >> 172.21.15.96:0/434665948 conn(0x7f64240b1fc0 msgr2=0x7f64240b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:02.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.111+0000 7f6423200640 1 -- 172.21.15.96:0/434665948 shutdown_connections 2024-05-28T06:22:02.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.111+0000 7f6423200640 1 -- 172.21.15.96:0/434665948 wait complete. 2024-05-28T06:22:02.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.112+0000 7f6423200640 1 Processor -- start 2024-05-28T06:22:02.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.112+0000 7f6423200640 1 -- start start 2024-05-28T06:22:02.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.113+0000 7f6423200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f642406dc20 0x7f6424146a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:02.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.113+0000 7f6423200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6424072c00 0x7f6424146fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:02.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.113+0000 7f6423200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f642415b3a0 0x7f64240993a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:02.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.113+0000 7f6423200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f642415b780 con 0x7f642415b3a0 2024-05-28T06:22:02.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.113+0000 7f6423200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f642415fee0 con 0x7f642406dc20 2024-05-28T06:22:02.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.113+0000 7f6423200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f6424073010 con 0x7f6424072c00 2024-05-28T06:22:02.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.114+0000 7f6422800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f642415b3a0 0x7f64240993a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:02.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.114+0000 7f6422800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f642415b3a0 0x7f64240993a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35940/0 (socket says 172.21.15.96:35940) 2024-05-28T06:22:02.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.114+0000 7f6422800640 1 -- 172.21.15.96:0/2426004411 learned_addr learned my addr 172.21.15.96:0/2426004411 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:22:02.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.114+0000 7f641be00640 1 --2- 172.21.15.96:0/2426004411 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6424072c00 0x7f6424146fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:02.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.114+0000 7f6421e00640 1 --2- 172.21.15.96:0/2426004411 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f642406dc20 0x7f6424146a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:02.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.114+0000 7f6422800640 1 -- 172.21.15.96:0/2426004411 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6424072c00 msgr2=0x7f6424146fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:02.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.114+0000 7f6422800640 1 --2- 172.21.15.96:0/2426004411 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6424072c00 0x7f6424146fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:02.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.114+0000 7f6422800640 1 -- 172.21.15.96:0/2426004411 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f642406dc20 msgr2=0x7f6424146a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:02.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.114+0000 7f6422800640 1 --2- 172.21.15.96:0/2426004411 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f642406dc20 0x7f6424146a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:02.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.114+0000 7f6422800640 1 -- 172.21.15.96:0/2426004411 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6424146750 con 0x7f642415b3a0 2024-05-28T06:22:02.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.114+0000 7f641be00640 1 --2- 172.21.15.96:0/2426004411 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6424072c00 0x7f6424146fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:22:02.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.114+0000 7f6421e00640 1 --2- 172.21.15.96:0/2426004411 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f642406dc20 0x7f6424146a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:22:02.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.114+0000 7f6422800640 1 --2- 172.21.15.96:0/2426004411 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f642415b3a0 0x7f64240993a0 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7f641c00c800 tx=0x7f641c00ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:02.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.115+0000 7f641a000640 1 -- 172.21.15.96:0/2426004411 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f641c018070 con 0x7f642415b3a0 2024-05-28T06:22:02.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.115+0000 7f641a000640 1 -- 172.21.15.96:0/2426004411 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f641c004d80 con 0x7f642415b3a0 2024-05-28T06:22:02.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.115+0000 7f641a000640 1 -- 172.21.15.96:0/2426004411 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f641c00a880 con 0x7f642415b3a0 2024-05-28T06:22:02.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.115+0000 7f6423200640 1 -- 172.21.15.96:0/2426004411 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6424146470 con 0x7f642415b3a0 2024-05-28T06:22:02.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.115+0000 7f6423200640 1 -- 172.21.15.96:0/2426004411 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f642414fb60 con 0x7f642415b3a0 2024-05-28T06:22:02.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.116+0000 7f6423200640 1 -- 172.21.15.96:0/2426004411 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64240689e0 con 0x7f642415b3a0 2024-05-28T06:22:02.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.117+0000 7f641a000640 1 -- 172.21.15.96:0/2426004411 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 37) ==== 93182+0+0 (secure 0 0 0) 0x7f641c00aa20 con 0x7f642415b3a0 2024-05-28T06:22:02.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.119+0000 7f641a000640 1 --2- 172.21.15.96:0/2426004411 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f640006e090 0x7f6400070550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:02.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.119+0000 7f641a000640 1 -- 172.21.15.96:0/2426004411 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 5821+0+0 (secure 0 0 0) 0x7f641c089f60 con 0x7f642415b3a0 2024-05-28T06:22:02.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.119+0000 7f6421e00640 1 --2- 172.21.15.96:0/2426004411 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f640006e090 0x7f6400070550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:02.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.121+0000 7f6421e00640 1 --2- 172.21.15.96:0/2426004411 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f640006e090 0x7f6400070550 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f6414034000 tx=0x7f641402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:02.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.123+0000 7f641a000640 1 -- 172.21.15.96:0/2426004411 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f641c059eb0 con 0x7f642415b3a0 2024-05-28T06:22:02.355 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.360+0000 7f6423200640 1 -- 172.21.15.96:0/2426004411 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f64240b4010 con 0x7f642415b3a0 2024-05-28T06:22:02.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.361+0000 7f641a000640 1 -- 172.21.15.96:0/2426004411 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) ==== 94+0+5189 (secure 0 0 0) 0x7f641c060080 con 0x7f642415b3a0 2024-05-28T06:22:02.360 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:22:02.361 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":63,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5861,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/1001966057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6833","nonce":1001966057},{"type":"v1","addr":"172.21.15.96:6835","nonce":1001966057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":57},{"gid":5864,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6836/4285246962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":4285246962},{"type":"v1","addr":"172.21.15.123:6836","nonce":4285246962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":57},{"gid":6445,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6834/3302807355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":3302807355},{"type":"v1","addr":"172.21.15.96:6834","nonce":3302807355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-28T06:20:18.363024+0000","modified":"2024-05-28T06:20:21.426136+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8739},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8739":{"gid":8739,"name":"b","rank":0,"incarnation":58,"state":"up:active","state_seq":3,"addr":"172.21.15.123:6837/3372660768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6835","nonce":3372660768},{"type":"v1","addr":"172.21.15.123:6837","nonce":3372660768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8739,"qdb_cluster":[8739]},"id":7}]} 2024-05-28T06:22:02.361 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 63 2024-05-28T06:22:02.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.364+0000 7f6423200640 1 -- 172.21.15.96:0/2426004411 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f640006e090 msgr2=0x7f6400070550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:02.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.364+0000 7f6423200640 1 --2- 172.21.15.96:0/2426004411 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f640006e090 0x7f6400070550 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f6414034000 tx=0x7f641402e000 comp rx=0 tx=0).stop 2024-05-28T06:22:02.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.364+0000 7f6423200640 1 -- 172.21.15.96:0/2426004411 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f642415b3a0 msgr2=0x7f64240993a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:02.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.364+0000 7f6423200640 1 --2- 172.21.15.96:0/2426004411 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f642415b3a0 0x7f64240993a0 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7f641c00c800 tx=0x7f641c00ccc0 comp rx=0 tx=0).stop 2024-05-28T06:22:02.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.364+0000 7f6423200640 1 -- 172.21.15.96:0/2426004411 shutdown_connections 2024-05-28T06:22:02.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.364+0000 7f6423200640 1 --2- 172.21.15.96:0/2426004411 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f640006e090 0x7f6400070550 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:02.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.364+0000 7f6423200640 1 --2- 172.21.15.96:0/2426004411 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f642415b3a0 0x7f64240993a0 unknown :-1 s=CLOSED pgs=1735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:02.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.364+0000 7f6423200640 1 --2- 172.21.15.96:0/2426004411 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6424072c00 0x7f6424146fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:02.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.364+0000 7f6423200640 1 --2- 172.21.15.96:0/2426004411 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f642406dc20 0x7f6424146a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:02.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.364+0000 7f6423200640 1 -- 172.21.15.96:0/2426004411 >> 172.21.15.96:0/2426004411 conn(0x7f64240b1fc0 msgr2=0x7f64240d0250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:02.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.364+0000 7f6423200640 1 -- 172.21.15.96:0/2426004411 shutdown_connections 2024-05-28T06:22:02.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.364+0000 7f6423200640 1 -- 172.21.15.96:0/2426004411 wait complete. 2024-05-28T06:22:02.377 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-28T06:22:02.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.603+0000 7ff08a000640 1 Processor -- start 2024-05-28T06:22:02.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.603+0000 7ff08a000640 1 -- start start 2024-05-28T06:22:02.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.605+0000 7ff08a000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff08415e9d0 0x7ff0840d24d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:02.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.605+0000 7ff08a000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7ff0840d52a0 0x7ff0840d3c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:02.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.605+0000 7ff08a000640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0840d5730 con 0x7ff0840d4150 2024-05-28T06:22:02.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.605+0000 7ff08a000640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7ff08415edb0 con 0x7ff0840d52a0 2024-05-28T06:22:02.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.605+0000 7ff08a000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ff0840d2b70 con 0x7ff08415e9d0 2024-05-28T06:22:02.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.605+0000 7ff083e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7ff0840d4150 0x7ff0840d1650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:37994/0 (socket says 172.21.15.96:37994) 2024-05-28T06:22:02.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.605+0000 7ff083e00640 1 -- 172.21.15.96:0/3466245178 learned_addr learned my addr 172.21.15.96:0/3466245178 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:22:02.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.605+0000 7ff082000640 1 -- 172.21.15.96:0/3466245178 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3087666888 0 0) 0x7ff07c001110 con 0x7ff0840d4150 2024-05-28T06:22:02.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.606+0000 7ff082a00640 1 --2- 172.21.15.96:0/3466245178 >> v2:172.21.15.123:3300/0 conn(0x7ff0840d52a0 0x7ff0840d3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:02.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.606+0000 7ff083400640 1 --2- 172.21.15.96:0/3466245178 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff08415e9d0 0x7ff0840d24d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:02.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.606+0000 7ff082000640 1 -- 172.21.15.96:0/3466245178 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff068003520 con 0x7ff0840d4150 2024-05-28T06:22:02.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.606+0000 7ff082000640 1 -- 172.21.15.96:0/3466245178 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3618493153 0 0) 0x7ff068003520 con 0x7ff0840d4150 2024-05-28T06:22:02.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.606+0000 7ff082000640 1 -- 172.21.15.96:0/3466245178 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff08415e9d0 msgr2=0x7ff0840d24d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:02.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.606+0000 7ff082000640 1 --2- 172.21.15.96:0/3466245178 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff08415e9d0 0x7ff0840d24d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:02.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.606+0000 7ff082000640 1 -- 172.21.15.96:0/3466245178 >> v2:172.21.15.123:3300/0 conn(0x7ff0840d52a0 msgr2=0x7ff0840d3c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:02.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.606+0000 7ff082000640 1 --2- 172.21.15.96:0/3466245178 >> v2:172.21.15.123:3300/0 conn(0x7ff0840d52a0 0x7ff0840d3c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:02.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.606+0000 7ff082000640 1 -- 172.21.15.96:0/3466245178 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0840d2e50 con 0x7ff0840d4150 2024-05-28T06:22:02.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.606+0000 7ff082000640 1 -- 172.21.15.96:0/3466245178 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7ff07c001840 con 0x7ff0840d4150 2024-05-28T06:22:02.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.606+0000 7ff082000640 1 -- 172.21.15.96:0/3466245178 >> v1:172.21.15.96:6789/0 conn(0x7ff0840d4150 legacy=0x7ff0840d1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:02.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.606+0000 7ff082000640 1 --2- 172.21.15.96:0/3466245178 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff08415e9d0 0x7ff0680039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:02.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.606+0000 7ff082000640 1 --2- 172.21.15.96:0/3466245178 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff068003fe0 0x7ff068004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:02.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.606+0000 7ff082000640 1 --2- 172.21.15.96:0/3466245178 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff068004d40 0x7ff0680051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:02.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.606+0000 7ff082000640 1 -- 172.21.15.96:0/3466245178 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff08415edb0 con 0x7ff068003fe0 2024-05-28T06:22:02.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.606+0000 7ff082000640 1 -- 172.21.15.96:0/3466245178 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff0840d2b70 con 0x7ff068004d40 2024-05-28T06:22:02.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.606+0000 7ff082000640 1 -- 172.21.15.96:0/3466245178 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ff06800bb80 con 0x7ff08415e9d0 2024-05-28T06:22:02.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.606+0000 7ff082a00640 1 --2- 172.21.15.96:0/3466245178 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff068003fe0 0x7ff068004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:02.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.607+0000 7ff082a00640 1 -- 172.21.15.96:0/3466245178 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff08415e9d0 msgr2=0x7ff0680039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:02.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.607+0000 7ff082a00640 1 --2- 172.21.15.96:0/3466245178 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff08415e9d0 0x7ff0680039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:02.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.607+0000 7ff082a00640 1 -- 172.21.15.96:0/3466245178 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff068004d40 msgr2=0x7ff0680051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:02.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.607+0000 7ff082a00640 1 --2- 172.21.15.96:0/3466245178 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff068004d40 0x7ff0680051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:02.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.607+0000 7ff082a00640 1 -- 172.21.15.96:0/3466245178 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff068003520 con 0x7ff068003fe0 2024-05-28T06:22:02.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.608+0000 7ff082a00640 1 --2- 172.21.15.96:0/3466245178 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff068003fe0 0x7ff068004800 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7ff070007360 tx=0x7ff068005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f2b59673148300b server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:02.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.608+0000 7ff082000640 1 -- 172.21.15.96:0/3466245178 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff070009be0 con 0x7ff068003fe0 2024-05-28T06:22:02.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.608+0000 7ff082000640 1 -- 172.21.15.96:0/3466245178 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff07002e620 con 0x7ff068003fe0 2024-05-28T06:22:02.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.608+0000 7ff082000640 1 -- 172.21.15.96:0/3466245178 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff07002e9f0 con 0x7ff068003fe0 2024-05-28T06:22:02.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.608+0000 7ff08a000640 1 -- 172.21.15.96:0/3466245178 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff068003fe0 msgr2=0x7ff068004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:02.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.608+0000 7ff08a000640 1 --2- 172.21.15.96:0/3466245178 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff068003fe0 0x7ff068004800 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7ff070007360 tx=0x7ff068005830 comp rx=0 tx=0).stop 2024-05-28T06:22:02.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.608+0000 7ff083e00640 1 -- 172.21.15.96:0/3466245178 reap_dead start 2024-05-28T06:22:02.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.608+0000 7ff08a000640 1 -- 172.21.15.96:0/3466245178 shutdown_connections 2024-05-28T06:22:02.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.608+0000 7ff08a000640 1 -- 172.21.15.96:0/3466245178 >> 172.21.15.96:0/3466245178 conn(0x7ff084067890 msgr2=0x7ff084067ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:02.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.609+0000 7ff08a000640 1 -- 172.21.15.96:0/3466245178 shutdown_connections 2024-05-28T06:22:02.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.609+0000 7ff08a000640 1 -- 172.21.15.96:0/3466245178 wait complete. 2024-05-28T06:22:02.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.610+0000 7ff08a000640 1 Processor -- start 2024-05-28T06:22:02.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.610+0000 7ff08a000640 1 -- start start 2024-05-28T06:22:02.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.611+0000 7ff08a000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff068003fe0 0x7ff084156780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:02.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.611+0000 7ff08a000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff0841f5290 0x7ff084155eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:02.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.611+0000 7ff08a000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff084158e00 0x7ff084159280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:02.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.611+0000 7ff08a000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff0840d36f0 con 0x7ff0841f5290 2024-05-28T06:22:02.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.611+0000 7ff08a000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff0840d3410 con 0x7ff084158e00 2024-05-28T06:22:02.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.611+0000 7ff08a000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ff0840d1d60 con 0x7ff068003fe0 2024-05-28T06:22:02.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.612+0000 7ff082a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff0841f5290 0x7ff084155eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:02.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.612+0000 7ff082a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff0841f5290 0x7ff084155eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35958/0 (socket says 172.21.15.96:35958) 2024-05-28T06:22:02.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.612+0000 7ff082a00640 1 -- 172.21.15.96:0/2567510982 learned_addr learned my addr 172.21.15.96:0/2567510982 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:22:02.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.612+0000 7ff083400640 1 --2- 172.21.15.96:0/2567510982 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff068003fe0 0x7ff084156780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:02.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.612+0000 7ff083e00640 1 --2- 172.21.15.96:0/2567510982 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff084158e00 0x7ff084159280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:02.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.612+0000 7ff082a00640 1 -- 172.21.15.96:0/2567510982 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff068003fe0 msgr2=0x7ff084156780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:02.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.612+0000 7ff082a00640 1 --2- 172.21.15.96:0/2567510982 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff068003fe0 0x7ff084156780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:02.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.612+0000 7ff082a00640 1 -- 172.21.15.96:0/2567510982 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff084158e00 msgr2=0x7ff084159280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:02.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.612+0000 7ff082a00640 1 --2- 172.21.15.96:0/2567510982 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff084158e00 0x7ff084159280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:02.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.612+0000 7ff082a00640 1 -- 172.21.15.96:0/2567510982 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0840b8480 con 0x7ff0841f5290 2024-05-28T06:22:02.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.612+0000 7ff082a00640 1 --2- 172.21.15.96:0/2567510982 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff0841f5290 0x7ff084155eb0 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7ff07002b0a0 tx=0x7ff070004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:02.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.613+0000 7ff07b400640 1 -- 172.21.15.96:0/2567510982 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff070045070 con 0x7ff0841f5290 2024-05-28T06:22:02.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.613+0000 7ff07b400640 1 -- 172.21.15.96:0/2567510982 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0700369c0 con 0x7ff0841f5290 2024-05-28T06:22:02.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.613+0000 7ff08a000640 1 -- 172.21.15.96:0/2567510982 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0841f5710 con 0x7ff0841f5290 2024-05-28T06:22:02.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.613+0000 7ff07b400640 1 -- 172.21.15.96:0/2567510982 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff070009be0 con 0x7ff0841f5290 2024-05-28T06:22:02.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.613+0000 7ff08a000640 1 -- 172.21.15.96:0/2567510982 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff084157610 con 0x7ff0841f5290 2024-05-28T06:22:02.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.614+0000 7ff08a000640 1 -- 172.21.15.96:0/2567510982 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0840689e0 con 0x7ff0841f5290 2024-05-28T06:22:02.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.615+0000 7ff07b400640 1 -- 172.21.15.96:0/2567510982 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 37) ==== 93182+0+0 (secure 0 0 0) 0x7ff070005d10 con 0x7ff0841f5290 2024-05-28T06:22:02.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.616+0000 7ff07b400640 1 --2- 172.21.15.96:0/2567510982 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff06406e0c0 0x7ff064070580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:02.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.616+0000 7ff07b400640 1 -- 172.21.15.96:0/2567510982 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 5821+0+0 (secure 0 0 0) 0x7ff0700ab710 con 0x7ff0841f5290 2024-05-28T06:22:02.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.617+0000 7ff083400640 1 --2- 172.21.15.96:0/2567510982 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff06406e0c0 0x7ff064070580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:02.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.618+0000 7ff083400640 1 --2- 172.21.15.96:0/2567510982 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff06406e0c0 0x7ff064070580 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7ff074006a90 tx=0x7ff074008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:02.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.619+0000 7ff07b400640 1 -- 172.21.15.96:0/2567510982 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff07004f080 con 0x7ff0841f5290 2024-05-28T06:22:02.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.841+0000 7ff08a000640 1 -- 172.21.15.96:0/2567510982 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff08415cdd0 con 0x7ff0841f5290 2024-05-28T06:22:02.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.843+0000 7ff07b400640 1 -- 172.21.15.96:0/2567510982 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) ==== 94+0+5189 (secure 0 0 0) 0x7ff07003d070 con 0x7ff0841f5290 2024-05-28T06:22:02.838 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:22:02.838 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":63,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5861,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/1001966057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6833","nonce":1001966057},{"type":"v1","addr":"172.21.15.96:6835","nonce":1001966057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":57},{"gid":5864,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6836/4285246962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":4285246962},{"type":"v1","addr":"172.21.15.123:6836","nonce":4285246962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":57},{"gid":6445,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6834/3302807355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":3302807355},{"type":"v1","addr":"172.21.15.96:6834","nonce":3302807355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-28T06:20:18.363024+0000","modified":"2024-05-28T06:20:21.426136+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8739},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8739":{"gid":8739,"name":"b","rank":0,"incarnation":58,"state":"up:active","state_seq":3,"addr":"172.21.15.123:6837/3372660768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6835","nonce":3372660768},{"type":"v1","addr":"172.21.15.123:6837","nonce":3372660768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8739,"qdb_cluster":[8739]},"id":7}]} 2024-05-28T06:22:02.838 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 63 2024-05-28T06:22:02.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.845+0000 7ff08a000640 1 -- 172.21.15.96:0/2567510982 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff06406e0c0 msgr2=0x7ff064070580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:02.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.845+0000 7ff08a000640 1 --2- 172.21.15.96:0/2567510982 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff06406e0c0 0x7ff064070580 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7ff074006a90 tx=0x7ff074008040 comp rx=0 tx=0).stop 2024-05-28T06:22:02.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.845+0000 7ff08a000640 1 -- 172.21.15.96:0/2567510982 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff0841f5290 msgr2=0x7ff084155eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:02.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.845+0000 7ff08a000640 1 --2- 172.21.15.96:0/2567510982 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff0841f5290 0x7ff084155eb0 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7ff07002b0a0 tx=0x7ff070004ab0 comp rx=0 tx=0).stop 2024-05-28T06:22:02.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.846+0000 7ff08a000640 1 -- 172.21.15.96:0/2567510982 shutdown_connections 2024-05-28T06:22:02.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.846+0000 7ff08a000640 1 --2- 172.21.15.96:0/2567510982 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff06406e0c0 0x7ff064070580 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:02.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.846+0000 7ff08a000640 1 --2- 172.21.15.96:0/2567510982 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff084158e00 0x7ff084159280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:02.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.846+0000 7ff08a000640 1 --2- 172.21.15.96:0/2567510982 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff0841f5290 0x7ff084155eb0 unknown :-1 s=CLOSED pgs=1738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:02.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.846+0000 7ff08a000640 1 --2- 172.21.15.96:0/2567510982 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff068003fe0 0x7ff084156780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:02.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.846+0000 7ff08a000640 1 -- 172.21.15.96:0/2567510982 >> 172.21.15.96:0/2567510982 conn(0x7ff084067890 msgr2=0x7ff08415d5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:02.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.846+0000 7ff08a000640 1 -- 172.21.15.96:0/2567510982 shutdown_connections 2024-05-28T06:22:02.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:02.846+0000 7ff08a000640 1 -- 172.21.15.96:0/2567510982 wait complete. 2024-05-28T06:22:02.858 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:22:03.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.050+0000 7fe0cb200640 1 Processor -- start 2024-05-28T06:22:03.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.050+0000 7fe0cb200640 1 -- start start 2024-05-28T06:22:03.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.051+0000 7fe0cb200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fe0cc15e870 0x7fe0cc0d0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:03.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.051+0000 7fe0cb200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe0cc0d14d0 0x7fe0cc0cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:03.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.052+0000 7fe0cb200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0cc0cfd00 con 0x7fe0cc0cf870 2024-05-28T06:22:03.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.052+0000 7fe0cb200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fe0cc0d2150 con 0x7fe0cc15e870 2024-05-28T06:22:03.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.052+0000 7fe0cb200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fe0cc15ec50 con 0x7fe0cc0d14d0 2024-05-28T06:22:03.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.052+0000 7fe0c9e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fe0cc15e870 0x7fe0cc0d0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:03.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.052+0000 7fe0c9e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fe0cc15e870 0x7fe0cc0d0f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35970/0 (socket says 172.21.15.96:35970) 2024-05-28T06:22:03.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.052+0000 7fe0c9e00640 1 -- 172.21.15.96:0/450082694 learned_addr learned my addr 172.21.15.96:0/450082694 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:22:03.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.052+0000 7fe0c3e00640 1 --2- 172.21.15.96:0/450082694 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe0cc0d14d0 0x7fe0cc0cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:03.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.052+0000 7fe0c9e00640 1 -- 172.21.15.96:0/450082694 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe0cc0d14d0 msgr2=0x7fe0cc0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:03.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.052+0000 7fe0c9e00640 1 --2- 172.21.15.96:0/450082694 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe0cc0d14d0 0x7fe0cc0cf330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:03.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.052+0000 7fe0c9e00640 1 -- 172.21.15.96:0/450082694 >> v1:172.21.15.123:6789/0 conn(0x7fe0cc0cf870 legacy=0x7fe0cc162360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:03.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.053+0000 7fe0c9e00640 1 -- 172.21.15.96:0/450082694 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0cc0cff80 con 0x7fe0cc15e870 2024-05-28T06:22:03.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.054+0000 7fe0c9e00640 1 --2- 172.21.15.96:0/450082694 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe0cc15e870 0x7fe0cc0d0f90 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7fe0bc009870 tx=0x7fe0bc02a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8f768b2721b5c33 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:03.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.054+0000 7fe0c9400640 1 -- 172.21.15.96:0/450082694 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0bc037070 con 0x7fe0cc15e870 2024-05-28T06:22:03.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.054+0000 7fe0c9400640 1 -- 172.21.15.96:0/450082694 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0bc032440 con 0x7fe0cc15e870 2024-05-28T06:22:03.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.054+0000 7fe0c9400640 1 -- 172.21.15.96:0/450082694 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0bc032810 con 0x7fe0cc15e870 2024-05-28T06:22:03.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.055+0000 7fe0cb200640 1 -- 172.21.15.96:0/450082694 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe0cc15e870 msgr2=0x7fe0cc0d0f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:03.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.055+0000 7fe0cb200640 1 --2- 172.21.15.96:0/450082694 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe0cc15e870 0x7fe0cc0d0f90 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7fe0bc009870 tx=0x7fe0bc02a210 comp rx=0 tx=0).stop 2024-05-28T06:22:03.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.055+0000 7fe0cb200640 1 -- 172.21.15.96:0/450082694 shutdown_connections 2024-05-28T06:22:03.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.055+0000 7fe0cb200640 1 --2- 172.21.15.96:0/450082694 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe0cc0d14d0 0x7fe0cc0cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:03.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.055+0000 7fe0cb200640 1 --2- 172.21.15.96:0/450082694 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe0cc15e870 0x7fe0cc0d0f90 unknown :-1 s=CLOSED pgs=1739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:03.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.055+0000 7fe0cb200640 1 -- 172.21.15.96:0/450082694 >> 172.21.15.96:0/450082694 conn(0x7fe0cc08bfc0 msgr2=0x7fe0cc0d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:03.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.055+0000 7fe0cb200640 1 -- 172.21.15.96:0/450082694 shutdown_connections 2024-05-28T06:22:03.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.055+0000 7fe0cb200640 1 -- 172.21.15.96:0/450082694 wait complete. 2024-05-28T06:22:03.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.057+0000 7fe0cb200640 1 Processor -- start 2024-05-28T06:22:03.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.057+0000 7fe0cb200640 1 -- start start 2024-05-28T06:22:03.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.058+0000 7fe0cb200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe0cc0cf870 0x7fe0cc0b71a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:03.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.058+0000 7fe0cb200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe0cc0d14d0 0x7fe0cc0b9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:03.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.058+0000 7fe0cb200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe0cc15e870 0x7fe0cc14f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:03.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.058+0000 7fe0cb200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe0cc0d0820 con 0x7fe0cc15e870 2024-05-28T06:22:03.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.058+0000 7fe0cb200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe0cc0d0540 con 0x7fe0cc0cf870 2024-05-28T06:22:03.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.058+0000 7fe0cb200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fe0cc0d4fb0 con 0x7fe0cc0d14d0 2024-05-28T06:22:03.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.058+0000 7fe0ca800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe0cc15e870 0x7fe0cc14f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:03.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.058+0000 7fe0ca800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe0cc15e870 0x7fe0cc14f9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35974/0 (socket says 172.21.15.96:35974) 2024-05-28T06:22:03.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.058+0000 7fe0ca800640 1 -- 172.21.15.96:0/3920382851 learned_addr learned my addr 172.21.15.96:0/3920382851 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:22:03.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.058+0000 7fe0c9e00640 1 --2- 172.21.15.96:0/3920382851 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe0cc0cf870 0x7fe0cc0b71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:03.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.058+0000 7fe0c3e00640 1 --2- 172.21.15.96:0/3920382851 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe0cc0d14d0 0x7fe0cc0b9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:03.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.058+0000 7fe0ca800640 1 -- 172.21.15.96:0/3920382851 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe0cc0d14d0 msgr2=0x7fe0cc0b9ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:03.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.058+0000 7fe0ca800640 1 --2- 172.21.15.96:0/3920382851 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe0cc0d14d0 0x7fe0cc0b9ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:03.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.058+0000 7fe0ca800640 1 -- 172.21.15.96:0/3920382851 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe0cc0cf870 msgr2=0x7fe0cc0b71a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:03.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.058+0000 7fe0ca800640 1 --2- 172.21.15.96:0/3920382851 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe0cc0cf870 0x7fe0cc0b71a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:03.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.058+0000 7fe0ca800640 1 -- 172.21.15.96:0/3920382851 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0cc159e60 con 0x7fe0cc15e870 2024-05-28T06:22:03.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.058+0000 7fe0c3e00640 1 --2- 172.21.15.96:0/3920382851 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe0cc0d14d0 0x7fe0cc0b9ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:22:03.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.059+0000 7fe0ca800640 1 --2- 172.21.15.96:0/3920382851 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe0cc15e870 0x7fe0cc14f9d0 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7fe0c400b9a0 tx=0x7fe0c400be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:03.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.059+0000 7fe0c2000640 1 -- 172.21.15.96:0/3920382851 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0c4019070 con 0x7fe0cc15e870 2024-05-28T06:22:03.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.059+0000 7fe0c2000640 1 -- 172.21.15.96:0/3920382851 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0c400cba0 con 0x7fe0cc15e870 2024-05-28T06:22:03.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.059+0000 7fe0c2000640 1 -- 172.21.15.96:0/3920382851 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0c4014440 con 0x7fe0cc15e870 2024-05-28T06:22:03.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.059+0000 7fe0cb200640 1 -- 172.21.15.96:0/3920382851 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0cc0cccf0 con 0x7fe0cc15e870 2024-05-28T06:22:03.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.059+0000 7fe0cb200640 1 -- 172.21.15.96:0/3920382851 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0cc0cd2b0 con 0x7fe0cc15e870 2024-05-28T06:22:03.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.060+0000 7fe0cb200640 1 -- 172.21.15.96:0/3920382851 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0cc0689e0 con 0x7fe0cc15e870 2024-05-28T06:22:03.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.061+0000 7fe0c2000640 1 -- 172.21.15.96:0/3920382851 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 37) ==== 93182+0+0 (secure 0 0 0) 0x7fe0c401c070 con 0x7fe0cc15e870 2024-05-28T06:22:03.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.062+0000 7fe0c2000640 1 --2- 172.21.15.96:0/3920382851 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe0a806e090 0x7fe0a8070550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:03.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.062+0000 7fe0c2000640 1 -- 172.21.15.96:0/3920382851 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 5821+0+0 (secure 0 0 0) 0x7fe0c408a570 con 0x7fe0cc15e870 2024-05-28T06:22:03.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.063+0000 7fe0c9e00640 1 --2- 172.21.15.96:0/3920382851 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe0a806e090 0x7fe0a8070550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:03.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.064+0000 7fe0c9e00640 1 --2- 172.21.15.96:0/3920382851 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe0a806e090 0x7fe0a8070550 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fe0bc034000 tx=0x7fe0bc02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:03.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.066+0000 7fe0c2000640 1 -- 172.21.15.96:0/3920382851 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe0cc159e60 con 0x7fe0cc15e870 2024-05-28T06:22:03.243 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.248+0000 7fe0cb200640 1 -- 172.21.15.96:0/3920382851 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe0cc0cb3e0 con 0x7fe0cc15e870 2024-05-28T06:22:03.245 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.250+0000 7fe0c2000640 1 -- 172.21.15.96:0/3920382851 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v104) ==== 74+0+16831 (secure 0 0 0) 0x7fe0c405a400 con 0x7fe0cc15e870 2024-05-28T06:22:03.245 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:22:03.246 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":104,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:21:58.794459+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-05-28T06:20:16.097747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-05-28T06:20:17.338574+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"104","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":104,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":13,"snaps":[{"begin":2,"length":2}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:22:03.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.253+0000 7fe0cb200640 1 -- 172.21.15.96:0/3920382851 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe0a806e090 msgr2=0x7fe0a8070550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:03.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.253+0000 7fe0cb200640 1 --2- 172.21.15.96:0/3920382851 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe0a806e090 0x7fe0a8070550 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fe0bc034000 tx=0x7fe0bc02e000 comp rx=0 tx=0).stop 2024-05-28T06:22:03.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.253+0000 7fe0cb200640 1 -- 172.21.15.96:0/3920382851 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe0cc15e870 msgr2=0x7fe0cc14f9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:03.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.253+0000 7fe0cb200640 1 --2- 172.21.15.96:0/3920382851 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe0cc15e870 0x7fe0cc14f9d0 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7fe0c400b9a0 tx=0x7fe0c400be60 comp rx=0 tx=0).stop 2024-05-28T06:22:03.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.253+0000 7fe0cb200640 1 -- 172.21.15.96:0/3920382851 shutdown_connections 2024-05-28T06:22:03.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.253+0000 7fe0cb200640 1 --2- 172.21.15.96:0/3920382851 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe0a806e090 0x7fe0a8070550 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:03.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.253+0000 7fe0cb200640 1 --2- 172.21.15.96:0/3920382851 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe0cc15e870 0x7fe0cc14f9d0 unknown :-1 s=CLOSED pgs=1740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:03.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.253+0000 7fe0cb200640 1 --2- 172.21.15.96:0/3920382851 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe0cc0d14d0 0x7fe0cc0b9ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:03.249 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.253+0000 7fe0cb200640 1 --2- 172.21.15.96:0/3920382851 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe0cc0cf870 0x7fe0cc0b71a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:03.249 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.253+0000 7fe0cb200640 1 -- 172.21.15.96:0/3920382851 >> 172.21.15.96:0/3920382851 conn(0x7fe0cc08bfc0 msgr2=0x7fe0cc0b6280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:03.249 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.253+0000 7fe0cb200640 1 -- 172.21.15.96:0/3920382851 shutdown_connections 2024-05-28T06:22:03.249 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.253+0000 7fe0cb200640 1 -- 172.21.15.96:0/3920382851 wait complete. 2024-05-28T06:22:03.265 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-05-28T06:22:03.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.465+0000 7f9acae00640 1 Processor -- start 2024-05-28T06:22:03.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.466+0000 7f9acae00640 1 -- start start 2024-05-28T06:22:03.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.467+0000 7f9acae00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9ac40b4b60 0x7f9ac40cfaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:03.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.467+0000 7f9acae00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f9ac40d04a0 0x7f9ac40d0900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:03.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.467+0000 7f9acae00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f9ac40b3460 0x7f9ac40b38d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:03.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.467+0000 7f9acae00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f9ac40d0e40 con 0x7f9ac40b3460 2024-05-28T06:22:03.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.467+0000 7f9acae00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f9ac40ced40 con 0x7f9ac40d04a0 2024-05-28T06:22:03.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.467+0000 7f9acae00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f9ac415fbb0 con 0x7f9ac40b4b60 2024-05-28T06:22:03.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.467+0000 7f9ac8e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f9ac40b3460 0x7f9ac40b38d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:03.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.467+0000 7f9ac8e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f9ac40b3460 0x7f9ac40b38d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35978/0 (socket says 172.21.15.96:35978) 2024-05-28T06:22:03.462 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.467+0000 7f9ac8e00640 1 -- 172.21.15.96:0/3550086750 learned_addr learned my addr 172.21.15.96:0/3550086750 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:22:03.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.468+0000 7f9ac3e00640 1 --2- 172.21.15.96:0/3550086750 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9ac40b4b60 0x7f9ac40cfaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:03.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.468+0000 7f9abb400640 1 --2- 172.21.15.96:0/3550086750 >> v2:172.21.15.123:3300/0 conn(0x7f9ac40d04a0 0x7f9ac40d0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:03.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.468+0000 7f9ac8e00640 1 -- 172.21.15.96:0/3550086750 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9ac40b4b60 msgr2=0x7f9ac40cfaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:03.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.468+0000 7f9ac8e00640 1 --2- 172.21.15.96:0/3550086750 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9ac40b4b60 0x7f9ac40cfaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:03.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.468+0000 7f9ac8e00640 1 -- 172.21.15.96:0/3550086750 >> v2:172.21.15.123:3300/0 conn(0x7f9ac40d04a0 msgr2=0x7f9ac40d0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:03.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.468+0000 7f9ac8e00640 1 --2- 172.21.15.96:0/3550086750 >> v2:172.21.15.123:3300/0 conn(0x7f9ac40d04a0 0x7f9ac40d0900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:03.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.468+0000 7f9ac8e00640 1 -- 172.21.15.96:0/3550086750 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ac40cefc0 con 0x7f9ac40b3460 2024-05-28T06:22:03.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.469+0000 7f9ac8e00640 1 --2- 172.21.15.96:0/3550086750 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ac40b3460 0x7f9ac40b38d0 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7f9ac406c7e0 tx=0x7f9abc02a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b8f20ebf53a0f34 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:03.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.469+0000 7f9ac3400640 1 -- 172.21.15.96:0/3550086750 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9abc036070 con 0x7f9ac40b3460 2024-05-28T06:22:03.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.469+0000 7f9ac3400640 1 -- 172.21.15.96:0/3550086750 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9abc031600 con 0x7f9ac40b3460 2024-05-28T06:22:03.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.469+0000 7f9ac3400640 1 -- 172.21.15.96:0/3550086750 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9abc0319d0 con 0x7f9ac40b3460 2024-05-28T06:22:03.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.470+0000 7f9acae00640 1 -- 172.21.15.96:0/3550086750 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ac40b3460 msgr2=0x7f9ac40b38d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:03.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.470+0000 7f9acae00640 1 --2- 172.21.15.96:0/3550086750 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ac40b3460 0x7f9ac40b38d0 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7f9ac406c7e0 tx=0x7f9abc02a6a0 comp rx=0 tx=0).stop 2024-05-28T06:22:03.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.470+0000 7f9acae00640 1 -- 172.21.15.96:0/3550086750 shutdown_connections 2024-05-28T06:22:03.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.470+0000 7f9acae00640 1 --2- 172.21.15.96:0/3550086750 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ac40b3460 0x7f9ac40b38d0 unknown :-1 s=CLOSED pgs=1741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:03.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.470+0000 7f9acae00640 1 --2- 172.21.15.96:0/3550086750 >> v2:172.21.15.123:3300/0 conn(0x7f9ac40d04a0 0x7f9ac40d0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:03.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.470+0000 7f9acae00640 1 --2- 172.21.15.96:0/3550086750 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9ac40b4b60 0x7f9ac40cfaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:03.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.470+0000 7f9acae00640 1 -- 172.21.15.96:0/3550086750 >> 172.21.15.96:0/3550086750 conn(0x7f9ac406bae0 msgr2=0x7f9ac406bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:03.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.470+0000 7f9acae00640 1 -- 172.21.15.96:0/3550086750 shutdown_connections 2024-05-28T06:22:03.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.470+0000 7f9acae00640 1 -- 172.21.15.96:0/3550086750 wait complete. 2024-05-28T06:22:03.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.472+0000 7f9acae00640 1 Processor -- start 2024-05-28T06:22:03.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.472+0000 7f9acae00640 1 -- start start 2024-05-28T06:22:03.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.473+0000 7f9acae00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ac40b3460 0x7f9ac40c6990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:03.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.473+0000 7f9acae00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9ac40b4b60 0x7f9ac414f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:03.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.473+0000 7f9acae00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ac40d04a0 0x7f9ac406ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:03.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.473+0000 7f9acae00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f9ac40cffe0 con 0x7f9ac40b3460 2024-05-28T06:22:03.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.473+0000 7f9acae00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f9ac40cf580 con 0x7f9ac40d04a0 2024-05-28T06:22:03.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.473+0000 7f9acae00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f9ac40b3e10 con 0x7f9ac40b4b60 2024-05-28T06:22:03.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.473+0000 7f9ac3e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ac40b3460 0x7f9ac40c6990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:03.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.473+0000 7f9ac3e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ac40b3460 0x7f9ac40c6990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35990/0 (socket says 172.21.15.96:35990) 2024-05-28T06:22:03.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.473+0000 7f9ac3e00640 1 -- 172.21.15.96:0/1093554080 learned_addr learned my addr 172.21.15.96:0/1093554080 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:22:03.469 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.474+0000 7f9ac3e00640 1 -- 172.21.15.96:0/1093554080 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9ac40b4b60 msgr2=0x7f9ac414f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:03.469 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.474+0000 7f9ac3e00640 1 --2- 172.21.15.96:0/1093554080 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9ac40b4b60 0x7f9ac414f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:03.470 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.474+0000 7f9ac3e00640 1 -- 172.21.15.96:0/1093554080 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ac40d04a0 msgr2=0x7f9ac406ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:03.470 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.474+0000 7f9ac3e00640 1 --2- 172.21.15.96:0/1093554080 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ac40d04a0 0x7f9ac406ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:03.470 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.474+0000 7f9ac3e00640 1 -- 172.21.15.96:0/1093554080 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ac414f880 con 0x7f9ac40b3460 2024-05-28T06:22:03.470 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.474+0000 7f9ac3e00640 1 --2- 172.21.15.96:0/1093554080 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ac40b3460 0x7f9ac40c6990 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7f9ab400d800 tx=0x7f9ab400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:03.470 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.474+0000 7f9ac0c00640 1 -- 172.21.15.96:0/1093554080 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ab4018020 con 0x7f9ac40b3460 2024-05-28T06:22:03.470 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.474+0000 7f9ac0c00640 1 -- 172.21.15.96:0/1093554080 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ab40096a0 con 0x7f9ac40b3460 2024-05-28T06:22:03.470 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.474+0000 7f9acae00640 1 -- 172.21.15.96:0/1093554080 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ac414fb60 con 0x7f9ac40b3460 2024-05-28T06:22:03.470 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.474+0000 7f9ac0c00640 1 -- 172.21.15.96:0/1093554080 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ab4009a70 con 0x7f9ac40b3460 2024-05-28T06:22:03.470 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.475+0000 7f9acae00640 1 -- 172.21.15.96:0/1093554080 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ac414d200 con 0x7f9ac40b3460 2024-05-28T06:22:03.471 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.476+0000 7f9acae00640 1 -- 172.21.15.96:0/1093554080 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ac41533f0 con 0x7f9ac40b3460 2024-05-28T06:22:03.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.477+0000 7f9ac0c00640 1 -- 172.21.15.96:0/1093554080 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 37) ==== 93182+0+0 (secure 0 0 0) 0x7f9ab401b020 con 0x7f9ac40b3460 2024-05-28T06:22:03.473 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.478+0000 7f9ac0c00640 1 --2- 172.21.15.96:0/1093554080 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9aa006e0e0 0x7f9aa00705a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:03.473 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.479+0000 7f9ac0c00640 1 -- 172.21.15.96:0/1093554080 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 5821+0+0 (secure 0 0 0) 0x7f9ab408b2c0 con 0x7f9ac40b3460 2024-05-28T06:22:03.473 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.479+0000 7f9abb400640 1 --2- 172.21.15.96:0/1093554080 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9aa006e0e0 0x7f9aa00705a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:03.475 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.481+0000 7f9abb400640 1 --2- 172.21.15.96:0/1093554080 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9aa006e0e0 0x7f9aa00705a0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f9aac006fd0 tx=0x7f9aac008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:03.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.482+0000 7f9ac0c00640 1 -- 172.21.15.96:0/1093554080 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9ac41533f0 con 0x7f9ac40b3460 2024-05-28T06:22:03.699 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.705+0000 7f9acae00640 1 -- 172.21.15.96:0/1093554080 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f9ac40689e0 con 0x7f9ac40b3460 2024-05-28T06:22:03.701 INFO:tasks.ceph.mon.a.smithi096.stderr:2024-05-28T06:22:03.706+0000 7fb2fca00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-28T06:22:03.705 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.710+0000 7f9ac0c00640 1 -- 172.21.15.96:0/1093554080 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v64) ==== 161+0+0 (secure 0 0 0) 0x7f9ab4022030 con 0x7f9ac40b3460 2024-05-28T06:22:03.705 INFO:teuthology.orchestra.run.smithi096.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-28T06:22:03.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.713+0000 7f9acae00640 1 -- 172.21.15.96:0/1093554080 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9aa006e0e0 msgr2=0x7f9aa00705a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:03.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.713+0000 7f9acae00640 1 --2- 172.21.15.96:0/1093554080 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9aa006e0e0 0x7f9aa00705a0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f9aac006fd0 tx=0x7f9aac008040 comp rx=0 tx=0).stop 2024-05-28T06:22:03.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.713+0000 7f9acae00640 1 -- 172.21.15.96:0/1093554080 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ac40b3460 msgr2=0x7f9ac40c6990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:03.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.713+0000 7f9acae00640 1 --2- 172.21.15.96:0/1093554080 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ac40b3460 0x7f9ac40c6990 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7f9ab400d800 tx=0x7f9ab400dcc0 comp rx=0 tx=0).stop 2024-05-28T06:22:03.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.713+0000 7f9acae00640 1 -- 172.21.15.96:0/1093554080 shutdown_connections 2024-05-28T06:22:03.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.713+0000 7f9acae00640 1 --2- 172.21.15.96:0/1093554080 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9aa006e0e0 0x7f9aa00705a0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:03.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.713+0000 7f9acae00640 1 --2- 172.21.15.96:0/1093554080 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ac40d04a0 0x7f9ac406ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:03.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.713+0000 7f9acae00640 1 --2- 172.21.15.96:0/1093554080 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9ac40b4b60 0x7f9ac414f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:03.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.713+0000 7f9acae00640 1 --2- 172.21.15.96:0/1093554080 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ac40b3460 0x7f9ac40c6990 unknown :-1 s=CLOSED pgs=1742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:03.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.713+0000 7f9acae00640 1 -- 172.21.15.96:0/1093554080 >> 172.21.15.96:0/1093554080 conn(0x7f9ac406bae0 msgr2=0x7f9ac406bec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:03.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.713+0000 7f9acae00640 1 -- 172.21.15.96:0/1093554080 shutdown_connections 2024-05-28T06:22:03.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.713+0000 7f9acae00640 1 -- 172.21.15.96:0/1093554080 wait complete. 2024-05-28T06:22:03.728 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-05-28T06:22:03.831 INFO:tasks.ceph.mds.b.smithi123.stdout:starting mds.b at 2024-05-28T06:22:03.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.921+0000 7f032c200640 1 Processor -- start 2024-05-28T06:22:03.917 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.922+0000 7f032c200640 1 -- start start 2024-05-28T06:22:03.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.923+0000 7f032c200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f032415d5d0 0x7f03240cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:03.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.923+0000 7f032c200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f03240b32c0 0x7f03240b3750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:03.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.923+0000 7f032c200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f032415cb10 con 0x7f03240d0640 2024-05-28T06:22:03.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.923+0000 7f032c200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f03240adbb0 con 0x7f032415d5d0 2024-05-28T06:22:03.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.923+0000 7f032c200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f03240ad5f0 con 0x7f03240b32c0 2024-05-28T06:22:03.919 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.924+0000 7f0328e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f032415d5d0 0x7f03240cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:03.919 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.924+0000 7f032a200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f03240b32c0 0x7f03240b3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:03.919 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.924+0000 7f0328e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f032415d5d0 0x7f03240cf950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.96:39496/0 (socket says 172.21.15.96:39496) 2024-05-28T06:22:03.919 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.924+0000 7f0328e00640 1 -- 172.21.15.96:0/2019788647 learned_addr learned my addr 172.21.15.96:0/2019788647 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:22:03.919 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.925+0000 7f032a200640 1 -- 172.21.15.96:0/2019788647 >> v2:172.21.15.123:3300/0 conn(0x7f032415d5d0 msgr2=0x7f03240cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:03.920 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.925+0000 7f032a200640 1 --2- 172.21.15.96:0/2019788647 >> v2:172.21.15.123:3300/0 conn(0x7f032415d5d0 0x7f03240cf950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:03.920 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.925+0000 7f032a200640 1 -- 172.21.15.96:0/2019788647 >> v1:172.21.15.123:6789/0 conn(0x7f03240d0640 legacy=0x7f03240d0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:03.920 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.925+0000 7f032a200640 1 -- 172.21.15.96:0/2019788647 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03240ad8d0 con 0x7f03240b32c0 2024-05-28T06:22:03.920 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.925+0000 7f0328e00640 1 --2- 172.21.15.96:0/2019788647 >> v2:172.21.15.123:3300/0 conn(0x7f032415d5d0 0x7f03240cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-28T06:22:03.920 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.926+0000 7f032a200640 1 --2- 172.21.15.96:0/2019788647 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f03240b32c0 0x7f03240b3750 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f031c009d90 tx=0x7f031c02a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=63e390d7d2e7d37f server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:03.921 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.926+0000 7f0323e00640 1 -- 172.21.15.96:0/2019788647 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f031c007740 con 0x7f03240b32c0 2024-05-28T06:22:03.921 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.926+0000 7f0323e00640 1 -- 172.21.15.96:0/2019788647 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f031c0078e0 con 0x7f03240b32c0 2024-05-28T06:22:03.921 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.926+0000 7f0323e00640 1 -- 172.21.15.96:0/2019788647 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f031c007cb0 con 0x7f03240b32c0 2024-05-28T06:22:03.922 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.927+0000 7f032c200640 1 -- 172.21.15.96:0/2019788647 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f03240b32c0 msgr2=0x7f03240b3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:03.922 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.927+0000 7f032c200640 1 --2- 172.21.15.96:0/2019788647 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f03240b32c0 0x7f03240b3750 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f031c009d90 tx=0x7f031c02a660 comp rx=0 tx=0).stop 2024-05-28T06:22:03.922 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.927+0000 7f032c200640 1 -- 172.21.15.96:0/2019788647 shutdown_connections 2024-05-28T06:22:03.922 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.927+0000 7f032c200640 1 --2- 172.21.15.96:0/2019788647 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f03240b32c0 0x7f03240b3750 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:03.922 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.927+0000 7f032c200640 1 --2- 172.21.15.96:0/2019788647 >> v2:172.21.15.123:3300/0 conn(0x7f032415d5d0 0x7f03240cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:03.922 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.927+0000 7f032c200640 1 -- 172.21.15.96:0/2019788647 >> 172.21.15.96:0/2019788647 conn(0x7f032406bd20 msgr2=0x7f032406c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:03.922 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.927+0000 7f032c200640 1 -- 172.21.15.96:0/2019788647 shutdown_connections 2024-05-28T06:22:03.922 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.927+0000 7f032c200640 1 -- 172.21.15.96:0/2019788647 wait complete. 2024-05-28T06:22:03.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.929+0000 7f032c200640 1 Processor -- start 2024-05-28T06:22:03.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.929+0000 7f032c200640 1 -- start start 2024-05-28T06:22:03.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.930+0000 7f032c200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f03240b32c0 0x7f032406edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:03.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.930+0000 7f032c200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f03240d0640 0x7f032415b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:03.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.930+0000 7f032c200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f032415d5d0 0x7f032415b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:03.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.930+0000 7f032c200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f032415da50 con 0x7f03240d0640 2024-05-28T06:22:03.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.930+0000 7f032c200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f032415d0b0 con 0x7f032415d5d0 2024-05-28T06:22:03.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.930+0000 7f032c200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f03240ad2a0 con 0x7f03240b32c0 2024-05-28T06:22:03.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.930+0000 7f0328e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f03240d0640 0x7f032415b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:03.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.930+0000 7f0328e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f03240d0640 0x7f032415b240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:36004/0 (socket says 172.21.15.96:36004) 2024-05-28T06:22:03.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.930+0000 7f0328e00640 1 -- 172.21.15.96:0/1726213344 learned_addr learned my addr 172.21.15.96:0/1726213344 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:22:03.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.931+0000 7f032a200640 1 --2- 172.21.15.96:0/1726213344 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f032415d5d0 0x7f032415b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:03.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.931+0000 7f0329800640 1 --2- 172.21.15.96:0/1726213344 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f03240b32c0 0x7f032406edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:03.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.931+0000 7f0328e00640 1 -- 172.21.15.96:0/1726213344 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f03240b32c0 msgr2=0x7f032406edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:03.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.931+0000 7f0328e00640 1 --2- 172.21.15.96:0/1726213344 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f03240b32c0 0x7f032406edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:03.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.931+0000 7f0328e00640 1 -- 172.21.15.96:0/1726213344 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f032415d5d0 msgr2=0x7f032415b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:03.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.931+0000 7f0328e00640 1 --2- 172.21.15.96:0/1726213344 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f032415d5d0 0x7f032415b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:03.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.931+0000 7f0328e00640 1 -- 172.21.15.96:0/1726213344 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f032406f300 con 0x7f03240d0640 2024-05-28T06:22:03.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.931+0000 7f032a200640 1 --2- 172.21.15.96:0/1726213344 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f032415d5d0 0x7f032415b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:22:03.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.931+0000 7f0329800640 1 --2- 172.21.15.96:0/1726213344 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f03240b32c0 0x7f032406edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:22:03.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.931+0000 7f0328e00640 1 --2- 172.21.15.96:0/1726213344 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f03240d0640 0x7f032415b240 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7f031000e9e0 tx=0x7f031000eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:03.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.931+0000 7f0321600640 1 -- 172.21.15.96:0/1726213344 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0310017070 con 0x7f03240d0640 2024-05-28T06:22:03.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.931+0000 7f0321600640 1 -- 172.21.15.96:0/1726213344 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f031000c9d0 con 0x7f03240d0640 2024-05-28T06:22:03.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.931+0000 7f0321600640 1 -- 172.21.15.96:0/1726213344 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f031000cd80 con 0x7f03240d0640 2024-05-28T06:22:03.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.931+0000 7f032c200640 1 -- 172.21.15.96:0/1726213344 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03240b0430 con 0x7f03240d0640 2024-05-28T06:22:03.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.932+0000 7f032c200640 1 -- 172.21.15.96:0/1726213344 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03240b09f0 con 0x7f03240d0640 2024-05-28T06:22:03.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.933+0000 7f032c200640 1 -- 172.21.15.96:0/1726213344 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03240689e0 con 0x7f03240d0640 2024-05-28T06:22:03.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.934+0000 7f0321600640 1 -- 172.21.15.96:0/1726213344 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 37) ==== 93182+0+0 (secure 0 0 0) 0x7f031001a070 con 0x7f03240d0640 2024-05-28T06:22:03.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.934+0000 7f0321600640 1 --2- 172.21.15.96:0/1726213344 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f030406e090 0x7f0304070550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:03.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.935+0000 7f0321600640 1 -- 172.21.15.96:0/1726213344 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 5879+0+0 (secure 0 0 0) 0x7f031008a800 con 0x7f03240d0640 2024-05-28T06:22:03.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.935+0000 7f0329800640 1 --2- 172.21.15.96:0/1726213344 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f030406e090 0x7f0304070550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:03.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.936+0000 7f0329800640 1 --2- 172.21.15.96:0/1726213344 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f030406e090 0x7f0304070550 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f0318000f60 tx=0x7f0318009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:03.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:03.939+0000 7f0321600640 1 -- 172.21.15.96:0/1726213344 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f031005a690 con 0x7f03240d0640 2024-05-28T06:22:04.133 DEBUG:teuthology.orchestra.run.smithi096:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:22:04.141 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:22:04.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:04.178+0000 7f032c200640 1 -- 172.21.15.96:0/1726213344 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f0324071ef0 con 0x7f03240d0640 2024-05-28T06:22:04.849 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:04.854+0000 7f0321600640 1 -- 172.21.15.96:0/1726213344 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v65) ==== 104+0+0 (secure 0 0 0) 0x7f0310059f90 con 0x7f03240d0640 2024-05-28T06:22:04.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:04.858+0000 7f032c200640 1 -- 172.21.15.96:0/1726213344 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f030406e090 msgr2=0x7f0304070550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:04.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:04.858+0000 7f032c200640 1 --2- 172.21.15.96:0/1726213344 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f030406e090 0x7f0304070550 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f0318000f60 tx=0x7f0318009040 comp rx=0 tx=0).stop 2024-05-28T06:22:04.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:04.858+0000 7f032c200640 1 -- 172.21.15.96:0/1726213344 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f03240d0640 msgr2=0x7f032415b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:04.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:04.858+0000 7f032c200640 1 --2- 172.21.15.96:0/1726213344 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f03240d0640 0x7f032415b240 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7f031000e9e0 tx=0x7f031000eea0 comp rx=0 tx=0).stop 2024-05-28T06:22:04.854 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:04.859+0000 7f032c200640 1 -- 172.21.15.96:0/1726213344 shutdown_connections 2024-05-28T06:22:04.854 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:04.859+0000 7f032c200640 1 --2- 172.21.15.96:0/1726213344 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f030406e090 0x7f0304070550 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:04.854 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:04.859+0000 7f032c200640 1 --2- 172.21.15.96:0/1726213344 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f032415d5d0 0x7f032415b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:04.854 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:04.859+0000 7f032c200640 1 --2- 172.21.15.96:0/1726213344 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f03240d0640 0x7f032415b240 unknown :-1 s=CLOSED pgs=1744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:04.854 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:04.859+0000 7f032c200640 1 --2- 172.21.15.96:0/1726213344 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f03240b32c0 0x7f032406edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:04.854 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:04.859+0000 7f032c200640 1 -- 172.21.15.96:0/1726213344 >> 172.21.15.96:0/1726213344 conn(0x7f032406bd20 msgr2=0x7f03240b3e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:04.854 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:04.859+0000 7f032c200640 1 -- 172.21.15.96:0/1726213344 shutdown_connections 2024-05-28T06:22:04.854 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:04.859+0000 7f032c200640 1 -- 172.21.15.96:0/1726213344 wait complete. 2024-05-28T06:22:04.877 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-05-28T06:22:05.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.095+0000 7f2398a00640 1 Processor -- start 2024-05-28T06:22:05.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.095+0000 7f2398a00640 1 -- start start 2024-05-28T06:22:05.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.097+0000 7f2398a00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f239415e5a0 0x7f23940d2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:05.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.097+0000 7f2398a00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23940d3500 con 0x7f23940d1420 2024-05-28T06:22:05.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.097+0000 7f2398a00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23940d2200 con 0x7f23940cfb40 2024-05-28T06:22:05.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.097+0000 7f2398a00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f23940d2b30 con 0x7f239415e5a0 2024-05-28T06:22:05.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.097+0000 7f2393e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f23940d1420 0x7f239406c740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:38006/0 (socket says 172.21.15.96:38006) 2024-05-28T06:22:05.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.097+0000 7f2393e00640 1 -- 172.21.15.96:0/3500293294 learned_addr learned my addr 172.21.15.96:0/3500293294 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:22:05.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.097+0000 7f2393400640 1 --2- 172.21.15.96:0/3500293294 >> v2:172.21.15.123:3300/0 conn(0x7f239415e5a0 0x7f23940d2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:05.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.097+0000 7f2392000640 1 -- 172.21.15.96:0/3500293294 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1296076974 0 0) 0x7f23940d3500 con 0x7f23940d1420 2024-05-28T06:22:05.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.098+0000 7f2392000640 1 -- 172.21.15.96:0/3500293294 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2378003520 con 0x7f23940d1420 2024-05-28T06:22:05.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.098+0000 7f2392000640 1 -- 172.21.15.96:0/3500293294 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 85486234 0 0) 0x7f2378003520 con 0x7f23940d1420 2024-05-28T06:22:05.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.098+0000 7f2392000640 1 -- 172.21.15.96:0/3500293294 >> v2:172.21.15.123:3300/0 conn(0x7f239415e5a0 msgr2=0x7f23940d2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:05.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.098+0000 7f2392000640 1 --2- 172.21.15.96:0/3500293294 >> v2:172.21.15.123:3300/0 conn(0x7f239415e5a0 0x7f23940d2fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:05.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.098+0000 7f2392000640 1 -- 172.21.15.96:0/3500293294 >> v1:172.21.15.123:6789/0 conn(0x7f23940cfb40 legacy=0x7f23940d0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:05.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.098+0000 7f2392000640 1 -- 172.21.15.96:0/3500293294 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23940d24e0 con 0x7f23940d1420 2024-05-28T06:22:05.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.098+0000 7f2392000640 1 -- 172.21.15.96:0/3500293294 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f238c0019f0 con 0x7f23940d1420 2024-05-28T06:22:05.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.098+0000 7f2392000640 1 -- 172.21.15.96:0/3500293294 >> v1:172.21.15.96:6789/0 conn(0x7f23940d1420 legacy=0x7f239406c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:05.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.098+0000 7f2392000640 1 --2- 172.21.15.96:0/3500293294 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f23780039b0 0x7f2378003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:05.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.098+0000 7f2392000640 1 --2- 172.21.15.96:0/3500293294 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2378004800 0x7f2378004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:05.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.098+0000 7f2392000640 1 --2- 172.21.15.96:0/3500293294 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f23780051c0 0x7f237800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:05.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.098+0000 7f2392000640 1 -- 172.21.15.96:0/3500293294 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f237800c0e0 con 0x7f23780051c0 2024-05-28T06:22:05.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.098+0000 7f2392000640 1 -- 172.21.15.96:0/3500293294 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f237800c290 con 0x7f2378004800 2024-05-28T06:22:05.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.098+0000 7f2392000640 1 -- 172.21.15.96:0/3500293294 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f237800c490 con 0x7f23780039b0 2024-05-28T06:22:05.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.098+0000 7f2393e00640 1 --2- 172.21.15.96:0/3500293294 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f23780051c0 0x7f237800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:05.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.099+0000 7f2392a00640 1 --2- 172.21.15.96:0/3500293294 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2378004800 0x7f2378004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:05.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.099+0000 7f2393400640 1 --2- 172.21.15.96:0/3500293294 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f23780039b0 0x7f2378003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:05.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.099+0000 7f2393e00640 1 -- 172.21.15.96:0/3500293294 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f23780039b0 msgr2=0x7f2378003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:05.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.099+0000 7f2393e00640 1 --2- 172.21.15.96:0/3500293294 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f23780039b0 0x7f2378003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:05.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.099+0000 7f2393e00640 1 -- 172.21.15.96:0/3500293294 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2378004800 msgr2=0x7f2378004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:05.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.099+0000 7f2393e00640 1 --2- 172.21.15.96:0/3500293294 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2378004800 0x7f2378004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:05.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.099+0000 7f2393e00640 1 -- 172.21.15.96:0/3500293294 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2378003520 con 0x7f23780051c0 2024-05-28T06:22:05.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.100+0000 7f2393e00640 1 -- 172.21.15.96:0/3500293294 reap_dead start 2024-05-28T06:22:05.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.100+0000 7f2393e00640 1 --2- 172.21.15.96:0/3500293294 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f23780051c0 0x7f237800b9c0 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7f238c00a830 tx=0x7f2378005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=faa0c436b6e4b01e server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:05.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.100+0000 7f2392000640 1 -- 172.21.15.96:0/3500293294 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f238c00a480 con 0x7f23780051c0 2024-05-28T06:22:05.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.100+0000 7f2392000640 1 -- 172.21.15.96:0/3500293294 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f238c02f070 con 0x7f23780051c0 2024-05-28T06:22:05.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.100+0000 7f2392000640 1 -- 172.21.15.96:0/3500293294 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f238c02bc70 con 0x7f23780051c0 2024-05-28T06:22:05.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.101+0000 7f2398a00640 1 -- 172.21.15.96:0/3500293294 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f23780051c0 msgr2=0x7f237800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:05.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.101+0000 7f2398a00640 1 --2- 172.21.15.96:0/3500293294 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f23780051c0 0x7f237800b9c0 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7f238c00a830 tx=0x7f2378005830 comp rx=0 tx=0).stop 2024-05-28T06:22:05.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.101+0000 7f2398a00640 1 -- 172.21.15.96:0/3500293294 shutdown_connections 2024-05-28T06:22:05.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.101+0000 7f2398a00640 1 --2- 172.21.15.96:0/3500293294 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f23780051c0 0x7f237800b9c0 unknown :-1 s=CLOSED pgs=1746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:05.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.101+0000 7f2398a00640 1 -- 172.21.15.96:0/3500293294 >> 172.21.15.96:0/3500293294 conn(0x7f239408bfc0 msgr2=0x7f23940d44b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:05.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.101+0000 7f2398a00640 1 -- 172.21.15.96:0/3500293294 shutdown_connections 2024-05-28T06:22:05.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.101+0000 7f2398a00640 1 -- 172.21.15.96:0/3500293294 wait complete. 2024-05-28T06:22:05.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.102+0000 7f2398a00640 1 Processor -- start 2024-05-28T06:22:05.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.103+0000 7f2398a00640 1 -- start start 2024-05-28T06:22:05.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.103+0000 7f2398a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f23780051c0 0x7f23940b69f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:05.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.103+0000 7f2398a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2394155960 0x7f23940b7a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:05.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.103+0000 7f2398a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f239415c1d0 0x7f239415c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:05.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.103+0000 7f2398a00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f239415eb00 con 0x7f239415c1d0 2024-05-28T06:22:05.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.103+0000 7f2398a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f239415e980 con 0x7f23780051c0 2024-05-28T06:22:05.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.103+0000 7f2398a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f23940d5220 con 0x7f2394155960 2024-05-28T06:22:05.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.104+0000 7f2393e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f239415c1d0 0x7f239415c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:05.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.104+0000 7f2393e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f239415c1d0 0x7f239415c650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:36016/0 (socket says 172.21.15.96:36016) 2024-05-28T06:22:05.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.104+0000 7f2393e00640 1 -- 172.21.15.96:0/1457655039 learned_addr learned my addr 172.21.15.96:0/1457655039 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:22:05.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.104+0000 7f2393400640 1 --2- 172.21.15.96:0/1457655039 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f23780051c0 0x7f23940b69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:05.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.104+0000 7f2392a00640 1 --2- 172.21.15.96:0/1457655039 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2394155960 0x7f23940b7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:05.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.104+0000 7f2393e00640 1 -- 172.21.15.96:0/1457655039 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2394155960 msgr2=0x7f23940b7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:05.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.104+0000 7f2393e00640 1 --2- 172.21.15.96:0/1457655039 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2394155960 0x7f23940b7a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:05.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.104+0000 7f2393e00640 1 -- 172.21.15.96:0/1457655039 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f23780051c0 msgr2=0x7f23940b69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:05.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.104+0000 7f2393e00640 1 --2- 172.21.15.96:0/1457655039 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f23780051c0 0x7f23940b69f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:05.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.104+0000 7f2393e00640 1 -- 172.21.15.96:0/1457655039 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23940b7f50 con 0x7f239415c1d0 2024-05-28T06:22:05.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.104+0000 7f2392a00640 1 --2- 172.21.15.96:0/1457655039 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2394155960 0x7f23940b7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:22:05.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.104+0000 7f2393e00640 1 --2- 172.21.15.96:0/1457655039 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f239415c1d0 0x7f239415c650 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7f238c00a950 tx=0x7f238c02d070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:05.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.105+0000 7f238b400640 1 -- 172.21.15.96:0/1457655039 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f238c02b850 con 0x7f239415c1d0 2024-05-28T06:22:05.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.105+0000 7f238b400640 1 -- 172.21.15.96:0/1457655039 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f238c02b9f0 con 0x7f239415c1d0 2024-05-28T06:22:05.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.105+0000 7f238b400640 1 -- 172.21.15.96:0/1457655039 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f238c02bde0 con 0x7f239415c1d0 2024-05-28T06:22:05.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.105+0000 7f2398a00640 1 -- 172.21.15.96:0/1457655039 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23940b81f0 con 0x7f239415c1d0 2024-05-28T06:22:05.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.105+0000 7f2398a00640 1 -- 172.21.15.96:0/1457655039 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23940b87b0 con 0x7f239415c1d0 2024-05-28T06:22:05.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.106+0000 7f2398a00640 1 -- 172.21.15.96:0/1457655039 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23940689e0 con 0x7f239415c1d0 2024-05-28T06:22:05.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.107+0000 7f238b400640 1 -- 172.21.15.96:0/1457655039 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 37) ==== 93182+0+0 (secure 0 0 0) 0x7f238c034470 con 0x7f239415c1d0 2024-05-28T06:22:05.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.108+0000 7f238b400640 1 --2- 172.21.15.96:0/1457655039 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f237406e090 0x7f2374070550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:05.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.108+0000 7f238b400640 1 -- 172.21.15.96:0/1457655039 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 5879+0+0 (secure 0 0 0) 0x7f238c0aa6b0 con 0x7f239415c1d0 2024-05-28T06:22:05.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.109+0000 7f2393400640 1 --2- 172.21.15.96:0/1457655039 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f237406e090 0x7f2374070550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:05.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.110+0000 7f2393400640 1 --2- 172.21.15.96:0/1457655039 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f237406e090 0x7f2374070550 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f2384006fd0 tx=0x7f2384008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:05.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.112+0000 7f238b400640 1 -- 172.21.15.96:0/1457655039 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f238c07a540 con 0x7f239415c1d0 2024-05-28T06:22:05.308 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.313+0000 7f2398a00640 1 -- 172.21.15.96:0/1457655039 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f239415b040 con 0x7f239415c1d0 2024-05-28T06:22:05.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.867+0000 7f238b400640 1 -- 172.21.15.96:0/1457655039 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v106) ==== 181+0+0 (secure 0 0 0) 0x7f238c079e40 con 0x7f239415c1d0 2024-05-28T06:22:05.862 INFO:teuthology.orchestra.run.smithi096.stderr:pool 'cephfs_metadata' removed 2024-05-28T06:22:05.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.872+0000 7f2398a00640 1 -- 172.21.15.96:0/1457655039 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f237406e090 msgr2=0x7f2374070550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:05.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.872+0000 7f2398a00640 1 --2- 172.21.15.96:0/1457655039 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f237406e090 0x7f2374070550 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f2384006fd0 tx=0x7f2384008040 comp rx=0 tx=0).stop 2024-05-28T06:22:05.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.872+0000 7f2398a00640 1 -- 172.21.15.96:0/1457655039 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f239415c1d0 msgr2=0x7f239415c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:05.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.872+0000 7f2398a00640 1 --2- 172.21.15.96:0/1457655039 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f239415c1d0 0x7f239415c650 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7f238c00a950 tx=0x7f238c02d070 comp rx=0 tx=0).stop 2024-05-28T06:22:05.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.873+0000 7f2398a00640 1 -- 172.21.15.96:0/1457655039 shutdown_connections 2024-05-28T06:22:05.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.873+0000 7f2398a00640 1 --2- 172.21.15.96:0/1457655039 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f237406e090 0x7f2374070550 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:05.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.873+0000 7f2398a00640 1 --2- 172.21.15.96:0/1457655039 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f239415c1d0 0x7f239415c650 unknown :-1 s=CLOSED pgs=1747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:05.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.873+0000 7f2398a00640 1 --2- 172.21.15.96:0/1457655039 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2394155960 0x7f23940b7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:05.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.873+0000 7f2398a00640 1 --2- 172.21.15.96:0/1457655039 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f23780051c0 0x7f23940b69f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:05.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.873+0000 7f2398a00640 1 -- 172.21.15.96:0/1457655039 >> 172.21.15.96:0/1457655039 conn(0x7f239408bfc0 msgr2=0x7f23940b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:05.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.874+0000 7f2398a00640 1 -- 172.21.15.96:0/1457655039 shutdown_connections 2024-05-28T06:22:05.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:05.874+0000 7f2398a00640 1 -- 172.21.15.96:0/1457655039 wait complete. 2024-05-28T06:22:05.893 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-05-28T06:22:06.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.102+0000 7f6837600640 1 Processor -- start 2024-05-28T06:22:06.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.102+0000 7f6837600640 1 -- start start 2024-05-28T06:22:06.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.103+0000 7f6837600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f68300cf2f0 0x7f68300cf910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:06.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.103+0000 7f6837600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f683006edf0 0x7f683013fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:06.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.103+0000 7f6837600640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f683006dd50 con 0x7f683006f250 2024-05-28T06:22:06.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.103+0000 7f6837600640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f683006e030 con 0x7f68300cf2f0 2024-05-28T06:22:06.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.103+0000 7f6837600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f683006e310 con 0x7f683006edf0 2024-05-28T06:22:06.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.104+0000 7f6835600640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f683006f250 0x7f68300ad130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:38016/0 (socket says 172.21.15.96:38016) 2024-05-28T06:22:06.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.104+0000 7f6834c00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f68300cf2f0 0x7f68300cf910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:06.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.104+0000 7f6835600640 1 -- 172.21.15.96:0/2920341988 learned_addr learned my addr 172.21.15.96:0/2920341988 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:22:06.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.104+0000 7f6827e00640 1 --2- 172.21.15.96:0/2920341988 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f683006edf0 0x7f683013fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:06.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.104+0000 7f682fe00640 1 -- 172.21.15.96:0/2920341988 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 817602137 0 0) 0x7f683006dd50 con 0x7f683006f250 2024-05-28T06:22:06.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.104+0000 7f682fe00640 1 -- 172.21.15.96:0/2920341988 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6814003520 con 0x7f683006f250 2024-05-28T06:22:06.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.105+0000 7f6834c00640 1 -- 172.21.15.96:0/2920341988 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f683006edf0 msgr2=0x7f683013fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:06.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.105+0000 7f6834c00640 1 --2- 172.21.15.96:0/2920341988 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f683006edf0 0x7f683013fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:06.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.105+0000 7f6834c00640 1 -- 172.21.15.96:0/2920341988 >> v1:172.21.15.96:6789/0 conn(0x7f683006f250 legacy=0x7f68300ad130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:06.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.105+0000 7f6834c00640 1 -- 172.21.15.96:0/2920341988 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f683006e5f0 con 0x7f68300cf2f0 2024-05-28T06:22:06.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.106+0000 7f6834c00640 1 --2- 172.21.15.96:0/2920341988 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f68300cf2f0 0x7f68300cf910 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7f68200097a0 tx=0x7f682002a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7714b37a03d511a8 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:06.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.107+0000 7f682fe00640 1 -- 172.21.15.96:0/2920341988 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6820037070 con 0x7f68300cf2f0 2024-05-28T06:22:06.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.107+0000 7f682fe00640 1 -- 172.21.15.96:0/2920341988 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f682002ae50 con 0x7f68300cf2f0 2024-05-28T06:22:06.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.107+0000 7f682fe00640 1 -- 172.21.15.96:0/2920341988 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6820032640 con 0x7f68300cf2f0 2024-05-28T06:22:06.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.107+0000 7f6837600640 1 -- 172.21.15.96:0/2920341988 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f68300cf2f0 msgr2=0x7f68300cf910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:06.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.107+0000 7f6837600640 1 --2- 172.21.15.96:0/2920341988 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f68300cf2f0 0x7f68300cf910 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7f68200097a0 tx=0x7f682002a100 comp rx=0 tx=0).stop 2024-05-28T06:22:06.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.108+0000 7f6837600640 1 -- 172.21.15.96:0/2920341988 shutdown_connections 2024-05-28T06:22:06.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.108+0000 7f6837600640 1 --2- 172.21.15.96:0/2920341988 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f683006edf0 0x7f683013fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:06.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.108+0000 7f6837600640 1 --2- 172.21.15.96:0/2920341988 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f68300cf2f0 0x7f68300cf910 unknown :-1 s=CLOSED pgs=1749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:06.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.108+0000 7f6837600640 1 -- 172.21.15.96:0/2920341988 >> 172.21.15.96:0/2920341988 conn(0x7f68300b1e50 msgr2=0x7f68300b2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:06.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.108+0000 7f6837600640 1 -- 172.21.15.96:0/2920341988 shutdown_connections 2024-05-28T06:22:06.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.108+0000 7f6837600640 1 -- 172.21.15.96:0/2920341988 wait complete. 2024-05-28T06:22:06.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.109+0000 7f6837600640 1 Processor -- start 2024-05-28T06:22:06.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.110+0000 7f6837600640 1 -- start start 2024-05-28T06:22:06.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.110+0000 7f6837600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f683006edf0 0x7f68301548a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:06.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.110+0000 7f6837600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f683006f250 0x7f6830158d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:06.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.110+0000 7f6837600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f68300cf2f0 0x7f6830159290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:06.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.110+0000 7f6837600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f68301461a0 con 0x7f683006edf0 2024-05-28T06:22:06.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.110+0000 7f6837600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f6830146020 con 0x7f68300cf2f0 2024-05-28T06:22:06.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.110+0000 7f6837600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f68300cfe50 con 0x7f683006f250 2024-05-28T06:22:06.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.111+0000 7f6834c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f683006edf0 0x7f68301548a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:06.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.111+0000 7f6834c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f683006edf0 0x7f68301548a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:36032/0 (socket says 172.21.15.96:36032) 2024-05-28T06:22:06.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.111+0000 7f6834c00640 1 -- 172.21.15.96:0/2252536007 learned_addr learned my addr 172.21.15.96:0/2252536007 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:22:06.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.111+0000 7f6827e00640 1 --2- 172.21.15.96:0/2252536007 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f683006f250 0x7f6830158d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:06.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.111+0000 7f6834c00640 1 -- 172.21.15.96:0/2252536007 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f683006f250 msgr2=0x7f6830158d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:06.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.111+0000 7f6834c00640 1 --2- 172.21.15.96:0/2252536007 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f683006f250 0x7f6830158d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:06.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.111+0000 7f6835600640 1 --2- 172.21.15.96:0/2252536007 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f68300cf2f0 0x7f6830159290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:06.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.111+0000 7f6834c00640 1 -- 172.21.15.96:0/2252536007 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f68300cf2f0 msgr2=0x7f6830159290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:06.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.111+0000 7f6834c00640 1 --2- 172.21.15.96:0/2252536007 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f68300cf2f0 0x7f6830159290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:06.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.111+0000 7f6834c00640 1 -- 172.21.15.96:0/2252536007 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68300ce760 con 0x7f683006edf0 2024-05-28T06:22:06.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.112+0000 7f6834c00640 1 --2- 172.21.15.96:0/2252536007 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f683006edf0 0x7f68301548a0 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7f682002a140 tx=0x7f68200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:06.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.112+0000 7f682d600640 1 -- 172.21.15.96:0/2252536007 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6820041070 con 0x7f683006edf0 2024-05-28T06:22:06.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.112+0000 7f682d600640 1 -- 172.21.15.96:0/2252536007 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f682003c670 con 0x7f683006edf0 2024-05-28T06:22:06.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.112+0000 7f6837600640 1 -- 172.21.15.96:0/2252536007 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6830156a30 con 0x7f683006edf0 2024-05-28T06:22:06.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.112+0000 7f682d600640 1 -- 172.21.15.96:0/2252536007 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6820037040 con 0x7f683006edf0 2024-05-28T06:22:06.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.112+0000 7f6837600640 1 -- 172.21.15.96:0/2252536007 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68300cd500 con 0x7f683006edf0 2024-05-28T06:22:06.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.114+0000 7f6837600640 1 -- 172.21.15.96:0/2252536007 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68300689e0 con 0x7f683006edf0 2024-05-28T06:22:06.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.115+0000 7f682d600640 1 -- 172.21.15.96:0/2252536007 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 37) ==== 93182+0+0 (secure 0 0 0) 0x7f6820044070 con 0x7f683006edf0 2024-05-28T06:22:06.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.115+0000 7f682d600640 1 --2- 172.21.15.96:0/2252536007 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f680c06e090 0x7f680c070550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:06.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.116+0000 7f682d600640 1 -- 172.21.15.96:0/2252536007 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 5441+0+0 (secure 0 0 0) 0x7f68200ab0d0 con 0x7f683006edf0 2024-05-28T06:22:06.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.116+0000 7f6827e00640 1 --2- 172.21.15.96:0/2252536007 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f680c06e090 0x7f680c070550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:06.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.117+0000 7f6827e00640 1 --2- 172.21.15.96:0/2252536007 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f680c06e090 0x7f680c070550 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f6818006fd0 tx=0x7f6818008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:06.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.120+0000 7f682d600640 1 -- 172.21.15.96:0/2252536007 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f682007aed0 con 0x7f683006edf0 2024-05-28T06:22:06.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.319+0000 7f6837600640 1 -- 172.21.15.96:0/2252536007 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f6830073f30 con 0x7f683006edf0 2024-05-28T06:22:06.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.555+0000 7f682d600640 1 -- 172.21.15.96:0/2252536007 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v107) ==== 169+0+0 (secure 0 0 0) 0x7f682007a7d0 con 0x7f683006edf0 2024-05-28T06:22:06.550 INFO:teuthology.orchestra.run.smithi096.stderr:pool 'cephfs_data' removed 2024-05-28T06:22:06.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.559+0000 7f6837600640 1 -- 172.21.15.96:0/2252536007 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f680c06e090 msgr2=0x7f680c070550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:06.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.559+0000 7f6837600640 1 --2- 172.21.15.96:0/2252536007 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f680c06e090 0x7f680c070550 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f6818006fd0 tx=0x7f6818008040 comp rx=0 tx=0).stop 2024-05-28T06:22:06.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.559+0000 7f6837600640 1 -- 172.21.15.96:0/2252536007 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f683006edf0 msgr2=0x7f68301548a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:06.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.559+0000 7f6837600640 1 --2- 172.21.15.96:0/2252536007 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f683006edf0 0x7f68301548a0 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7f682002a140 tx=0x7f68200046c0 comp rx=0 tx=0).stop 2024-05-28T06:22:06.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.559+0000 7f6837600640 1 -- 172.21.15.96:0/2252536007 shutdown_connections 2024-05-28T06:22:06.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.559+0000 7f6837600640 1 --2- 172.21.15.96:0/2252536007 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f680c06e090 0x7f680c070550 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:06.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.559+0000 7f6837600640 1 --2- 172.21.15.96:0/2252536007 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f68300cf2f0 0x7f6830159290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:06.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.559+0000 7f6837600640 1 --2- 172.21.15.96:0/2252536007 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f683006f250 0x7f6830158d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:06.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.559+0000 7f6837600640 1 --2- 172.21.15.96:0/2252536007 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f683006edf0 0x7f68301548a0 unknown :-1 s=CLOSED pgs=1750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:06.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.559+0000 7f6837600640 1 -- 172.21.15.96:0/2252536007 >> 172.21.15.96:0/2252536007 conn(0x7f68300b1e50 msgr2=0x7f6830156040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:06.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.560+0000 7f6837600640 1 -- 172.21.15.96:0/2252536007 shutdown_connections 2024-05-28T06:22:06.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.560+0000 7f6837600640 1 -- 172.21.15.96:0/2252536007 wait complete. 2024-05-28T06:22:06.580 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-05-28T06:22:06.774 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.780+0000 7fc7d7000640 1 Processor -- start 2024-05-28T06:22:06.775 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.780+0000 7fc7d7000640 1 -- start start 2024-05-28T06:22:06.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.781+0000 7fc7d7000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fc7d0153510 0x7fc7d00b4d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:06.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.781+0000 7fc7d7000640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7d015d900 con 0x7fc7d00ccb90 2024-05-28T06:22:06.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.781+0000 7fc7d7000640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7d00b3f90 con 0x7fc7d015d270 2024-05-28T06:22:06.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.781+0000 7fc7d7000640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fc7d01538f0 con 0x7fc7d0153510 2024-05-28T06:22:06.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.781+0000 7fc7d5200640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fc7d00ccb90 0x7fc7d015cb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:38032/0 (socket says 172.21.15.96:38032) 2024-05-28T06:22:06.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.781+0000 7fc7d5200640 1 -- 172.21.15.96:0/4205277666 learned_addr learned my addr 172.21.15.96:0/4205277666 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:22:06.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.782+0000 7fc7cfe00640 1 -- 172.21.15.96:0/4205277666 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 441595455 0 0) 0x7fc7d015d900 con 0x7fc7d00ccb90 2024-05-28T06:22:06.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.782+0000 7fc7d5c00640 1 --2- 172.21.15.96:0/4205277666 >> v2:172.21.15.123:3300/0 conn(0x7fc7d0153510 0x7fc7d00b4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:06.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.782+0000 7fc7cfe00640 1 -- 172.21.15.96:0/4205277666 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc7b8003520 con 0x7fc7d00ccb90 2024-05-28T06:22:06.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.782+0000 7fc7cfe00640 1 -- 172.21.15.96:0/4205277666 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1611875022 0 0) 0x7fc7b8003520 con 0x7fc7d00ccb90 2024-05-28T06:22:06.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.782+0000 7fc7cfe00640 1 -- 172.21.15.96:0/4205277666 >> v2:172.21.15.123:3300/0 conn(0x7fc7d0153510 msgr2=0x7fc7d00b4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:06.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.782+0000 7fc7cfe00640 1 --2- 172.21.15.96:0/4205277666 >> v2:172.21.15.123:3300/0 conn(0x7fc7d0153510 0x7fc7d00b4d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:06.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.782+0000 7fc7cfe00640 1 -- 172.21.15.96:0/4205277666 >> v1:172.21.15.123:6789/0 conn(0x7fc7d015d270 legacy=0x7fc7d006e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:06.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.782+0000 7fc7cfe00640 1 -- 172.21.15.96:0/4205277666 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7d00b4270 con 0x7fc7d00ccb90 2024-05-28T06:22:06.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.782+0000 7fc7cfe00640 1 -- 172.21.15.96:0/4205277666 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7fc7c0001670 con 0x7fc7d00ccb90 2024-05-28T06:22:06.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.782+0000 7fc7d5c00640 1 --2- 172.21.15.96:0/4205277666 >> v2:172.21.15.123:3300/0 conn(0x7fc7d0153510 0x7fc7d00b4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-28T06:22:06.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.782+0000 7fc7cfe00640 1 -- 172.21.15.96:0/4205277666 >> v1:172.21.15.96:6789/0 conn(0x7fc7d00ccb90 legacy=0x7fc7d015cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:06.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.782+0000 7fc7cfe00640 1 --2- 172.21.15.96:0/4205277666 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc7b80039b0 0x7fc7b8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:06.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.782+0000 7fc7cfe00640 1 --2- 172.21.15.96:0/4205277666 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc7b8004800 0x7fc7b8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:06.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.782+0000 7fc7cfe00640 1 --2- 172.21.15.96:0/4205277666 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc7b80052e0 0x7fc7b800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:06.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.782+0000 7fc7cfe00640 1 -- 172.21.15.96:0/4205277666 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc7b800c180 con 0x7fc7b80039b0 2024-05-28T06:22:06.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.782+0000 7fc7cfe00640 1 -- 172.21.15.96:0/4205277666 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc7b800c330 con 0x7fc7b80052e0 2024-05-28T06:22:06.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.782+0000 7fc7cfe00640 1 -- 172.21.15.96:0/4205277666 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fc7b800c530 con 0x7fc7b8004800 2024-05-28T06:22:06.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.783+0000 7fc7d5c00640 1 --2- 172.21.15.96:0/4205277666 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc7b80039b0 0x7fc7b8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:06.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.783+0000 7fc7d5200640 1 --2- 172.21.15.96:0/4205277666 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc7b8004800 0x7fc7b8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:06.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.783+0000 7fc7d5c00640 1 -- 172.21.15.96:0/4205277666 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc7b8004800 msgr2=0x7fc7b8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:06.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.783+0000 7fc7d5c00640 1 --2- 172.21.15.96:0/4205277666 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc7b8004800 0x7fc7b8004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:06.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.783+0000 7fc7d5c00640 1 -- 172.21.15.96:0/4205277666 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc7b80052e0 msgr2=0x7fc7b800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:06.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.783+0000 7fc7d5c00640 1 --2- 172.21.15.96:0/4205277666 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc7b80052e0 0x7fc7b800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:06.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.783+0000 7fc7d5c00640 1 -- 172.21.15.96:0/4205277666 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7b8003520 con 0x7fc7b80039b0 2024-05-28T06:22:06.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.783+0000 7fc7d6600640 1 -- 172.21.15.96:0/4205277666 reap_dead start 2024-05-28T06:22:06.781 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.784+0000 7fc7d5c00640 1 --2- 172.21.15.96:0/4205277666 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc7b80039b0 0x7fc7b8003dd0 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7fc7c4006fd0 tx=0x7fc7b8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=872b083317a4cdef server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:06.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.784+0000 7fc7cfe00640 1 -- 172.21.15.96:0/4205277666 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7c400f040 con 0x7fc7b80039b0 2024-05-28T06:22:06.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.784+0000 7fc7cfe00640 1 -- 172.21.15.96:0/4205277666 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7c402ace0 con 0x7fc7b80039b0 2024-05-28T06:22:06.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.784+0000 7fc7cfe00640 1 -- 172.21.15.96:0/4205277666 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7c4004250 con 0x7fc7b80039b0 2024-05-28T06:22:06.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.785+0000 7fc7d7000640 1 -- 172.21.15.96:0/4205277666 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc7b80039b0 msgr2=0x7fc7b8003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:06.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.785+0000 7fc7d7000640 1 --2- 172.21.15.96:0/4205277666 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc7b80039b0 0x7fc7b8003dd0 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7fc7c4006fd0 tx=0x7fc7b8005830 comp rx=0 tx=0).stop 2024-05-28T06:22:06.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.785+0000 7fc7d7000640 1 -- 172.21.15.96:0/4205277666 shutdown_connections 2024-05-28T06:22:06.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.785+0000 7fc7d7000640 1 --2- 172.21.15.96:0/4205277666 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc7b80039b0 0x7fc7b8003dd0 unknown :-1 s=CLOSED pgs=1752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:06.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.785+0000 7fc7d7000640 1 -- 172.21.15.96:0/4205277666 >> 172.21.15.96:0/4205277666 conn(0x7fc7d006b900 msgr2=0x7fc7d006bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:06.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.785+0000 7fc7d7000640 1 -- 172.21.15.96:0/4205277666 shutdown_connections 2024-05-28T06:22:06.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.785+0000 7fc7d7000640 1 -- 172.21.15.96:0/4205277666 wait complete. 2024-05-28T06:22:06.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.787+0000 7fc7d7000640 1 Processor -- start 2024-05-28T06:22:06.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.787+0000 7fc7d7000640 1 -- start start 2024-05-28T06:22:06.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.788+0000 7fc7d7000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc7b80039b0 0x7fc7d00ad9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:06.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.788+0000 7fc7d7000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc7d00af0c0 0x7fc7d01f3950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:06.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.788+0000 7fc7d7000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc7d00733a0 0x7fc7d015b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:06.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.788+0000 7fc7d7000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc7d015d6a0 con 0x7fc7d00733a0 2024-05-28T06:22:06.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.788+0000 7fc7d7000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc7d00b48c0 con 0x7fc7d00af0c0 2024-05-28T06:22:06.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.788+0000 7fc7d7000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fc7d00b4d50 con 0x7fc7b80039b0 2024-05-28T06:22:06.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.788+0000 7fc7d6600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc7d00733a0 0x7fc7d015b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:06.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.788+0000 7fc7d6600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc7d00733a0 0x7fc7d015b430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:36054/0 (socket says 172.21.15.96:36054) 2024-05-28T06:22:06.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.788+0000 7fc7d6600640 1 -- 172.21.15.96:0/914503311 learned_addr learned my addr 172.21.15.96:0/914503311 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:22:06.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.788+0000 7fc7d5c00640 1 --2- 172.21.15.96:0/914503311 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc7b80039b0 0x7fc7d00ad9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:06.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.788+0000 7fc7d6600640 1 -- 172.21.15.96:0/914503311 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc7b80039b0 msgr2=0x7fc7d00ad9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:06.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.788+0000 7fc7d6600640 1 --2- 172.21.15.96:0/914503311 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc7b80039b0 0x7fc7d00ad9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:06.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.788+0000 7fc7d6600640 1 -- 172.21.15.96:0/914503311 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc7d00af0c0 msgr2=0x7fc7d01f3950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:06.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.788+0000 7fc7d6600640 1 --2- 172.21.15.96:0/914503311 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc7d00af0c0 0x7fc7d01f3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:06.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.788+0000 7fc7d6600640 1 -- 172.21.15.96:0/914503311 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7d00ae020 con 0x7fc7d00733a0 2024-05-28T06:22:06.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.789+0000 7fc7d6600640 1 --2- 172.21.15.96:0/914503311 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc7d00733a0 0x7fc7d015b430 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7fc7d00b3c10 tx=0x7fc7c800c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:06.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.789+0000 7fc7cd600640 1 -- 172.21.15.96:0/914503311 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7c8017070 con 0x7fc7d00733a0 2024-05-28T06:22:06.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.789+0000 7fc7cd600640 1 -- 172.21.15.96:0/914503311 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7c8007070 con 0x7fc7d00733a0 2024-05-28T06:22:06.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.789+0000 7fc7cd600640 1 -- 172.21.15.96:0/914503311 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7c8007440 con 0x7fc7d00733a0 2024-05-28T06:22:06.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.789+0000 7fc7d7000640 1 -- 172.21.15.96:0/914503311 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7d00ae300 con 0x7fc7d00733a0 2024-05-28T06:22:06.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.789+0000 7fc7d7000640 1 -- 172.21.15.96:0/914503311 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7d00ae8c0 con 0x7fc7d00733a0 2024-05-28T06:22:06.786 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.790+0000 7fc7d7000640 1 -- 172.21.15.96:0/914503311 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7d00689e0 con 0x7fc7d00733a0 2024-05-28T06:22:06.786 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.791+0000 7fc7cd600640 1 -- 172.21.15.96:0/914503311 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 37) ==== 93182+0+0 (secure 0 0 0) 0x7fc7c801a070 con 0x7fc7d00733a0 2024-05-28T06:22:06.787 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.792+0000 7fc7cd600640 1 --2- 172.21.15.96:0/914503311 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc7b406e090 0x7fc7b4070550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:06.787 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.792+0000 7fc7cd600640 1 -- 172.21.15.96:0/914503311 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 5027+0+0 (secure 0 0 0) 0x7fc7c8089bf0 con 0x7fc7d00733a0 2024-05-28T06:22:06.787 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.793+0000 7fc7d5c00640 1 --2- 172.21.15.96:0/914503311 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc7b406e090 0x7fc7b4070550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:06.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.794+0000 7fc7d5c00640 1 --2- 172.21.15.96:0/914503311 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc7b406e090 0x7fc7b4070550 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fc7c4033000 tx=0x7fc7c402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:06.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.797+0000 7fc7cd600640 1 -- 172.21.15.96:0/914503311 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc7c8059a80 con 0x7fc7d00733a0 2024-05-28T06:22:06.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:06.996+0000 7fc7d7000640 1 -- 172.21.15.96:0/914503311 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7fc7d006cb80 con 0x7fc7d00733a0 2024-05-28T06:22:06.999 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.004+0000 7fc7cd600640 1 -- 172.21.15.96:0/914503311 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v25) ==== 104+0+0 (secure 0 0 0) 0x7fc7c8059380 con 0x7fc7d00733a0 2024-05-28T06:22:07.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.006+0000 7fc7d7000640 1 -- 172.21.15.96:0/914503311 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc7b406e090 msgr2=0x7fc7b4070550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:07.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.006+0000 7fc7d7000640 1 --2- 172.21.15.96:0/914503311 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc7b406e090 0x7fc7b4070550 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fc7c4033000 tx=0x7fc7c402e000 comp rx=0 tx=0).stop 2024-05-28T06:22:07.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.007+0000 7fc7d7000640 1 -- 172.21.15.96:0/914503311 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc7d00733a0 msgr2=0x7fc7d015b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:07.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.007+0000 7fc7d7000640 1 --2- 172.21.15.96:0/914503311 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc7d00733a0 0x7fc7d015b430 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7fc7d00b3c10 tx=0x7fc7c800c960 comp rx=0 tx=0).stop 2024-05-28T06:22:07.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.007+0000 7fc7d7000640 1 -- 172.21.15.96:0/914503311 shutdown_connections 2024-05-28T06:22:07.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.007+0000 7fc7d7000640 1 --2- 172.21.15.96:0/914503311 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc7b406e090 0x7fc7b4070550 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:07.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.007+0000 7fc7d7000640 1 --2- 172.21.15.96:0/914503311 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc7d00733a0 0x7fc7d015b430 unknown :-1 s=CLOSED pgs=1753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:07.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.007+0000 7fc7d7000640 1 --2- 172.21.15.96:0/914503311 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc7d00af0c0 0x7fc7d01f3950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:07.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.007+0000 7fc7d7000640 1 --2- 172.21.15.96:0/914503311 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc7b80039b0 0x7fc7d00ad9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:07.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.007+0000 7fc7d7000640 1 -- 172.21.15.96:0/914503311 >> 172.21.15.96:0/914503311 conn(0x7fc7d006b900 msgr2=0x7fc7d006f170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:07.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.007+0000 7fc7d7000640 1 -- 172.21.15.96:0/914503311 shutdown_connections 2024-05-28T06:22:07.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.007+0000 7fc7d7000640 1 -- 172.21.15.96:0/914503311 wait complete. 2024-05-28T06:22:07.021 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestCloneProgressReporter.test_when_1_clone_is_cancelled' 2024-05-28T06:22:07.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.222+0000 7fb85b200640 1 Processor -- start 2024-05-28T06:22:07.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.223+0000 7fb85b200640 1 -- start start 2024-05-28T06:22:07.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.224+0000 7fb85b200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fb85c15e870 0x7fb85c0d0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:07.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.224+0000 7fb85b200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb85c0d14d0 0x7fb85c0cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:07.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.224+0000 7fb85b200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb85c0cfd00 con 0x7fb85c0cf870 2024-05-28T06:22:07.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.224+0000 7fb85b200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fb85c0d2150 con 0x7fb85c15e870 2024-05-28T06:22:07.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.224+0000 7fb85b200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fb85c15ec50 con 0x7fb85c0d14d0 2024-05-28T06:22:07.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.224+0000 7fb859e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fb85c15e870 0x7fb85c0d0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:07.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.224+0000 7fb859e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fb85c15e870 0x7fb85c0d0f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:36066/0 (socket says 172.21.15.96:36066) 2024-05-28T06:22:07.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.224+0000 7fb859e00640 1 -- 172.21.15.96:0/2478115634 learned_addr learned my addr 172.21.15.96:0/2478115634 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:22:07.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.225+0000 7fb859400640 1 --2- 172.21.15.96:0/2478115634 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb85c0d14d0 0x7fb85c0cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:07.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.225+0000 7fb859e00640 1 -- 172.21.15.96:0/2478115634 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb85c0d14d0 msgr2=0x7fb85c0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:07.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.225+0000 7fb859e00640 1 --2- 172.21.15.96:0/2478115634 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb85c0d14d0 0x7fb85c0cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:07.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.225+0000 7fb859e00640 1 -- 172.21.15.96:0/2478115634 >> v1:172.21.15.123:6789/0 conn(0x7fb85c0cf870 legacy=0x7fb85c162360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:07.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.225+0000 7fb859e00640 1 -- 172.21.15.96:0/2478115634 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb85c0cff80 con 0x7fb85c15e870 2024-05-28T06:22:07.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.225+0000 7fb859400640 1 --2- 172.21.15.96:0/2478115634 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb85c0d14d0 0x7fb85c0cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:22:07.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.226+0000 7fb859e00640 1 --2- 172.21.15.96:0/2478115634 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb85c15e870 0x7fb85c0d0f90 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7fb84c009870 tx=0x7fb84c02a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=969b718f940da802 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:07.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.226+0000 7fb858a00640 1 -- 172.21.15.96:0/2478115634 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb84c037070 con 0x7fb85c15e870 2024-05-28T06:22:07.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.226+0000 7fb858a00640 1 -- 172.21.15.96:0/2478115634 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb84c032440 con 0x7fb85c15e870 2024-05-28T06:22:07.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.226+0000 7fb858a00640 1 -- 172.21.15.96:0/2478115634 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb84c032810 con 0x7fb85c15e870 2024-05-28T06:22:07.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.227+0000 7fb85b200640 1 -- 172.21.15.96:0/2478115634 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb85c15e870 msgr2=0x7fb85c0d0f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:07.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.227+0000 7fb85b200640 1 --2- 172.21.15.96:0/2478115634 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb85c15e870 0x7fb85c0d0f90 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7fb84c009870 tx=0x7fb84c02a210 comp rx=0 tx=0).stop 2024-05-28T06:22:07.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.227+0000 7fb85b200640 1 -- 172.21.15.96:0/2478115634 shutdown_connections 2024-05-28T06:22:07.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.227+0000 7fb85b200640 1 --2- 172.21.15.96:0/2478115634 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb85c0d14d0 0x7fb85c0cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:07.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.227+0000 7fb85b200640 1 --2- 172.21.15.96:0/2478115634 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb85c15e870 0x7fb85c0d0f90 unknown :-1 s=CLOSED pgs=1754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:07.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.227+0000 7fb85b200640 1 -- 172.21.15.96:0/2478115634 >> 172.21.15.96:0/2478115634 conn(0x7fb85c08bfc0 msgr2=0x7fb85c0d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:07.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.227+0000 7fb85b200640 1 -- 172.21.15.96:0/2478115634 shutdown_connections 2024-05-28T06:22:07.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.227+0000 7fb85b200640 1 -- 172.21.15.96:0/2478115634 wait complete. 2024-05-28T06:22:07.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.229+0000 7fb85b200640 1 Processor -- start 2024-05-28T06:22:07.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.229+0000 7fb85b200640 1 -- start start 2024-05-28T06:22:07.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.230+0000 7fb85b200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb85c0cf870 0x7fb85c1617e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:07.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.230+0000 7fb85b200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb85c0d14d0 0x7fb85c159ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:07.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.230+0000 7fb85b200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb85c15e870 0x7fb85c0b72f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:07.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.230+0000 7fb85b200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb85c0d0820 con 0x7fb85c0cf870 2024-05-28T06:22:07.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.230+0000 7fb85b200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fb85c0d0540 con 0x7fb85c15e870 2024-05-28T06:22:07.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.230+0000 7fb85b200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fb85c0d4fb0 con 0x7fb85c0d14d0 2024-05-28T06:22:07.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.231+0000 7fb859e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb85c0cf870 0x7fb85c1617e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:07.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.231+0000 7fb859e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb85c0cf870 0x7fb85c1617e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:36080/0 (socket says 172.21.15.96:36080) 2024-05-28T06:22:07.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.231+0000 7fb859e00640 1 -- 172.21.15.96:0/1666193449 learned_addr learned my addr 172.21.15.96:0/1666193449 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:22:07.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.231+0000 7fb859400640 1 --2- 172.21.15.96:0/1666193449 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb85c0d14d0 0x7fb85c159ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:07.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.231+0000 7fb859e00640 1 -- 172.21.15.96:0/1666193449 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb85c0d14d0 msgr2=0x7fb85c159ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:07.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.231+0000 7fb859e00640 1 --2- 172.21.15.96:0/1666193449 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb85c0d14d0 0x7fb85c159ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:07.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.231+0000 7fb859e00640 1 -- 172.21.15.96:0/1666193449 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb85c15e870 msgr2=0x7fb85c0b72f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:07.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.231+0000 7fb85a800640 1 --2- 172.21.15.96:0/1666193449 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb85c15e870 0x7fb85c0b72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:07.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.231+0000 7fb859e00640 1 --2- 172.21.15.96:0/1666193449 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb85c15e870 0x7fb85c0b72f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:07.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.231+0000 7fb859e00640 1 -- 172.21.15.96:0/1666193449 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb85c15a020 con 0x7fb85c0cf870 2024-05-28T06:22:07.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.231+0000 7fb859400640 1 --2- 172.21.15.96:0/1666193449 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb85c0d14d0 0x7fb85c159ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:22:07.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.231+0000 7fb859e00640 1 --2- 172.21.15.96:0/1666193449 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb85c0cf870 0x7fb85c1617e0 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7fb84c009870 tx=0x7fb84c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:07.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.232+0000 7fb852000640 1 -- 172.21.15.96:0/1666193449 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb84c041070 con 0x7fb85c0cf870 2024-05-28T06:22:07.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.232+0000 7fb852000640 1 -- 172.21.15.96:0/1666193449 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb84c03c870 con 0x7fb85c0cf870 2024-05-28T06:22:07.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.232+0000 7fb852000640 1 -- 172.21.15.96:0/1666193449 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb84c037040 con 0x7fb85c0cf870 2024-05-28T06:22:07.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.232+0000 7fb85b200640 1 -- 172.21.15.96:0/1666193449 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb85c0b8090 con 0x7fb85c0cf870 2024-05-28T06:22:07.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.232+0000 7fb85b200640 1 -- 172.21.15.96:0/1666193449 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb85c160fe0 con 0x7fb85c0cf870 2024-05-28T06:22:07.228 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.233+0000 7fb85b200640 1 -- 172.21.15.96:0/1666193449 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb85c160b30 con 0x7fb85c0cf870 2024-05-28T06:22:07.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.234+0000 7fb852000640 1 -- 172.21.15.96:0/1666193449 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 37) ==== 93182+0+0 (secure 0 0 0) 0x7fb84c044070 con 0x7fb85c0cf870 2024-05-28T06:22:07.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.235+0000 7fb852000640 1 --2- 172.21.15.96:0/1666193449 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb83806e090 0x7fb838070550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:07.229 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.235+0000 7fb852000640 1 -- 172.21.15.96:0/1666193449 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 5027+0+0 (secure 0 0 0) 0x7fb84c0ab4c0 con 0x7fb85c0cf870 2024-05-28T06:22:07.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.236+0000 7fb859400640 1 --2- 172.21.15.96:0/1666193449 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb83806e090 0x7fb838070550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:07.233 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.238+0000 7fb859400640 1 --2- 172.21.15.96:0/1666193449 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb83806e090 0x7fb838070550 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fb85c15cfb0 tx=0x7fb844007580 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:07.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.243+0000 7fb852000640 1 -- 172.21.15.96:0/1666193449 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb85c15a020 con 0x7fb85c0cf870 2024-05-28T06:22:07.525 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.531+0000 7fb85b200640 1 -- 172.21.15.96:0/1666193449 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgressReporter.test_when_1_clone_is_cancelled"]} v 0) -- 0x7fb85c0689e0 con 0x7fb85c0cf870 2024-05-28T06:22:07.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.883+0000 7fb852000640 1 -- 172.21.15.96:0/1666193449 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgressReporter.test_when_1_clone_is_cancelled"]}]=0 v788) ==== 161+0+0 (secure 0 0 0) 0x7fb85c0cfd00 con 0x7fb85c0cf870 2024-05-28T06:22:07.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.887+0000 7fb85b200640 1 -- 172.21.15.96:0/1666193449 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb83806e090 msgr2=0x7fb838070550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:07.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.887+0000 7fb85b200640 1 --2- 172.21.15.96:0/1666193449 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb83806e090 0x7fb838070550 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fb85c15cfb0 tx=0x7fb844007580 comp rx=0 tx=0).stop 2024-05-28T06:22:07.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.887+0000 7fb85b200640 1 -- 172.21.15.96:0/1666193449 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb85c0cf870 msgr2=0x7fb85c1617e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:07.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.887+0000 7fb85b200640 1 --2- 172.21.15.96:0/1666193449 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb85c0cf870 0x7fb85c1617e0 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7fb84c009870 tx=0x7fb84c0046c0 comp rx=0 tx=0).stop 2024-05-28T06:22:07.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.887+0000 7fb85b200640 1 -- 172.21.15.96:0/1666193449 shutdown_connections 2024-05-28T06:22:07.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.887+0000 7fb85b200640 1 --2- 172.21.15.96:0/1666193449 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb83806e090 0x7fb838070550 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:07.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.887+0000 7fb85b200640 1 --2- 172.21.15.96:0/1666193449 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb85c15e870 0x7fb85c0b72f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:07.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.887+0000 7fb85b200640 1 --2- 172.21.15.96:0/1666193449 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb85c0d14d0 0x7fb85c159ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:07.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.887+0000 7fb85b200640 1 --2- 172.21.15.96:0/1666193449 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb85c0cf870 0x7fb85c1617e0 unknown :-1 s=CLOSED pgs=1755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:07.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.887+0000 7fb85b200640 1 -- 172.21.15.96:0/1666193449 >> 172.21.15.96:0/1666193449 conn(0x7fb85c08bfc0 msgr2=0x7fb85c06e460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:07.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.887+0000 7fb85b200640 1 -- 172.21.15.96:0/1666193449 shutdown_connections 2024-05-28T06:22:07.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:07.887+0000 7fb85b200640 1 -- 172.21.15.96:0/1666193449 wait complete. 2024-05-28T06:22:07.906 INFO:tasks.cephfs_test_runner:Test that the progress bar, that is printed for 1 ongoing clone job, is ... ok 2024-05-28T06:22:07.907 INFO:tasks.cephfs_test_runner:Starting test: test_when_3_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgressReporter) Test that progress bars, that printed for 3 ongoing clone jobs, are 2024-05-28T06:22:07.907 INFO:tasks.cephfs_test_runner:test_when_3_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgressReporter) 2024-05-28T06:22:07.907 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:22:08.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.114+0000 7fd9d6c00640 1 Processor -- start 2024-05-28T06:22:08.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.114+0000 7fd9d6c00640 1 -- start start 2024-05-28T06:22:08.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.115+0000 7fd9d6c00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fd9d015b3a0 0x7fd9d00b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:08.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.115+0000 7fd9d6c00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fd9d0073310 0x7fd9d015f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:08.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.115+0000 7fd9d6c00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9d00afd60 con 0x7fd9d006dc20 2024-05-28T06:22:08.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.115+0000 7fd9d6c00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fd9d006f980 con 0x7fd9d0073310 2024-05-28T06:22:08.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.115+0000 7fd9d6c00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fd9d00ae0c0 con 0x7fd9d015b3a0 2024-05-28T06:22:08.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.116+0000 7fd9d4e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fd9d006dc20 0x7fd9d0072c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:38040/0 (socket says 172.21.15.96:38040) 2024-05-28T06:22:08.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.116+0000 7fd9d4e00640 1 -- 172.21.15.96:0/1383269528 learned_addr learned my addr 172.21.15.96:0/1383269528 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:22:08.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.116+0000 7fd9d6200640 1 --2- 172.21.15.96:0/1383269528 >> v2:172.21.15.96:3300/0 conn(0x7fd9d0073310 0x7fd9d015f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:08.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.116+0000 7fd9d5800640 1 --2- 172.21.15.96:0/1383269528 >> v2:172.21.15.123:3300/0 conn(0x7fd9d015b3a0 0x7fd9d00b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:08.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.116+0000 7fd9cfe00640 1 -- 172.21.15.96:0/1383269528 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2867223155 0 0) 0x7fd9d00afd60 con 0x7fd9d006dc20 2024-05-28T06:22:08.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.117+0000 7fd9cfe00640 1 -- 172.21.15.96:0/1383269528 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd9b8005730 con 0x7fd9d006dc20 2024-05-28T06:22:08.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.117+0000 7fd9d6200640 1 -- 172.21.15.96:0/1383269528 >> v2:172.21.15.123:3300/0 conn(0x7fd9d015b3a0 msgr2=0x7fd9d00b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:08.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.117+0000 7fd9d6200640 1 --2- 172.21.15.96:0/1383269528 >> v2:172.21.15.123:3300/0 conn(0x7fd9d015b3a0 0x7fd9d00b32c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:08.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.117+0000 7fd9d6200640 1 -- 172.21.15.96:0/1383269528 >> v1:172.21.15.96:6789/0 conn(0x7fd9d006dc20 legacy=0x7fd9d0072c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:08.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.117+0000 7fd9d6200640 1 -- 172.21.15.96:0/1383269528 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9d00af7a0 con 0x7fd9d0073310 2024-05-28T06:22:08.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.117+0000 7fd9d5800640 1 --2- 172.21.15.96:0/1383269528 >> v2:172.21.15.123:3300/0 conn(0x7fd9d015b3a0 0x7fd9d00b32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-28T06:22:08.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.118+0000 7fd9d6200640 1 --2- 172.21.15.96:0/1383269528 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9d0073310 0x7fd9d015f9a0 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7fd9d0072bd0 tx=0x7fd9c802a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a7c7e4c3c9eb8f8 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:08.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.118+0000 7fd9cfe00640 1 -- 172.21.15.96:0/1383269528 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9c8007740 con 0x7fd9d0073310 2024-05-28T06:22:08.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.118+0000 7fd9cfe00640 1 -- 172.21.15.96:0/1383269528 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9c80078e0 con 0x7fd9d0073310 2024-05-28T06:22:08.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.118+0000 7fd9cfe00640 1 -- 172.21.15.96:0/1383269528 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9c8007c90 con 0x7fd9d0073310 2024-05-28T06:22:08.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.119+0000 7fd9d6c00640 1 -- 172.21.15.96:0/1383269528 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9d0073310 msgr2=0x7fd9d015f9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:08.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.119+0000 7fd9d6c00640 1 --2- 172.21.15.96:0/1383269528 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9d0073310 0x7fd9d015f9a0 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7fd9d0072bd0 tx=0x7fd9c802a680 comp rx=0 tx=0).stop 2024-05-28T06:22:08.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.119+0000 7fd9d6c00640 1 -- 172.21.15.96:0/1383269528 shutdown_connections 2024-05-28T06:22:08.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.119+0000 7fd9d6c00640 1 --2- 172.21.15.96:0/1383269528 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9d0073310 0x7fd9d015f9a0 unknown :-1 s=CLOSED pgs=1757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:08.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.119+0000 7fd9d6c00640 1 --2- 172.21.15.96:0/1383269528 >> v2:172.21.15.123:3300/0 conn(0x7fd9d015b3a0 0x7fd9d00b32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:08.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.119+0000 7fd9d6c00640 1 -- 172.21.15.96:0/1383269528 >> 172.21.15.96:0/1383269528 conn(0x7fd9d00b1fc0 msgr2=0x7fd9d00b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:08.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.119+0000 7fd9d6c00640 1 -- 172.21.15.96:0/1383269528 shutdown_connections 2024-05-28T06:22:08.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.119+0000 7fd9d6c00640 1 -- 172.21.15.96:0/1383269528 wait complete. 2024-05-28T06:22:08.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.120+0000 7fd9d6c00640 1 Processor -- start 2024-05-28T06:22:08.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.121+0000 7fd9d6c00640 1 -- start start 2024-05-28T06:22:08.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.122+0000 7fd9d6c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9d006dc20 0x7fd9d014c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:08.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.122+0000 7fd9d6c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd9d0073310 0x7fd9d015c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:08.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.122+0000 7fd9d6c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9d015b3a0 0x7fd9d01510e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:08.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.122+0000 7fd9d6c00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd9d015b780 con 0x7fd9d006dc20 2024-05-28T06:22:08.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.122+0000 7fd9d6c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd9d015fee0 con 0x7fd9d015b3a0 2024-05-28T06:22:08.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.122+0000 7fd9d6c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fd9d00ae550 con 0x7fd9d0073310 2024-05-28T06:22:08.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.122+0000 7fd9d5800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9d006dc20 0x7fd9d014c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:08.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.122+0000 7fd9d5800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9d006dc20 0x7fd9d014c0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:36096/0 (socket says 172.21.15.96:36096) 2024-05-28T06:22:08.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.122+0000 7fd9d5800640 1 -- 172.21.15.96:0/643438932 learned_addr learned my addr 172.21.15.96:0/643438932 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:22:08.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.122+0000 7fd9d5800640 1 -- 172.21.15.96:0/643438932 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd9d0073310 msgr2=0x7fd9d015c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:08.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.122+0000 7fd9d5800640 1 --2- 172.21.15.96:0/643438932 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd9d0073310 0x7fd9d015c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:08.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.122+0000 7fd9d5800640 1 -- 172.21.15.96:0/643438932 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9d015b3a0 msgr2=0x7fd9d01510e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:08.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.122+0000 7fd9d5800640 1 --2- 172.21.15.96:0/643438932 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9d015b3a0 0x7fd9d01510e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:08.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.122+0000 7fd9d5800640 1 -- 172.21.15.96:0/643438932 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9d015e810 con 0x7fd9d006dc20 2024-05-28T06:22:08.119 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.123+0000 7fd9d5800640 1 --2- 172.21.15.96:0/643438932 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9d006dc20 0x7fd9d014c0b0 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7fd9c400d800 tx=0x7fd9c400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:08.119 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.123+0000 7fd9cd600640 1 -- 172.21.15.96:0/643438932 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9c4004030 con 0x7fd9d006dc20 2024-05-28T06:22:08.119 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.123+0000 7fd9cd600640 1 -- 172.21.15.96:0/643438932 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9c40041d0 con 0x7fd9d006dc20 2024-05-28T06:22:08.119 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.123+0000 7fd9cd600640 1 -- 172.21.15.96:0/643438932 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9c400b6c0 con 0x7fd9d006dc20 2024-05-28T06:22:08.119 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.123+0000 7fd9d6c00640 1 -- 172.21.15.96:0/643438932 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9d015cde0 con 0x7fd9d006dc20 2024-05-28T06:22:08.119 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.123+0000 7fd9d6c00640 1 -- 172.21.15.96:0/643438932 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9d00ce030 con 0x7fd9d006dc20 2024-05-28T06:22:08.119 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.124+0000 7fd9d6c00640 1 -- 172.21.15.96:0/643438932 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9d00689e0 con 0x7fd9d006dc20 2024-05-28T06:22:08.119 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.125+0000 7fd9cd600640 1 -- 172.21.15.96:0/643438932 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 37) ==== 93182+0+0 (secure 0 0 0) 0x7fd9c400b860 con 0x7fd9d006dc20 2024-05-28T06:22:08.120 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.126+0000 7fd9cd600640 1 --2- 172.21.15.96:0/643438932 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd9b006e090 0x7fd9b0070550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:08.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.126+0000 7fd9cd600640 1 -- 172.21.15.96:0/643438932 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 5027+0+0 (secure 0 0 0) 0x7fd9c408ac20 con 0x7fd9d006dc20 2024-05-28T06:22:08.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.127+0000 7fd9d4e00640 1 --2- 172.21.15.96:0/643438932 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd9b006e090 0x7fd9b0070550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:08.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.128+0000 7fd9d4e00640 1 --2- 172.21.15.96:0/643438932 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd9b006e090 0x7fd9b0070550 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc000f60 tx=0x7fd9bc0078f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:08.125 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.130+0000 7fd9cd600640 1 -- 172.21.15.96:0/643438932 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd9c405aab0 con 0x7fd9d006dc20 2024-05-28T06:22:08.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.330+0000 7fd9d6c00640 1 -- 172.21.15.96:0/643438932 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd9d00cfd60 con 0x7fd9d006dc20 2024-05-28T06:22:08.327 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.333+0000 7fd9cd600640 1 -- 172.21.15.96:0/643438932 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v107) ==== 74+0+13176 (secure 0 0 0) 0x7fd9c405a3b0 con 0x7fd9d006dc20 2024-05-28T06:22:08.328 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:22:08.328 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":107,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:22:06.551477+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.123:6837/3372660768":"2024-05-29T06:22:03.706066+0000","172.21.15.123:6835/3372660768":"2024-05-29T06:22:03.706066+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":13,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:22:08.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.335+0000 7fd9d6c00640 1 -- 172.21.15.96:0/643438932 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd9b006e090 msgr2=0x7fd9b0070550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:08.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.335+0000 7fd9d6c00640 1 --2- 172.21.15.96:0/643438932 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd9b006e090 0x7fd9b0070550 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc000f60 tx=0x7fd9bc0078f0 comp rx=0 tx=0).stop 2024-05-28T06:22:08.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.335+0000 7fd9d6c00640 1 -- 172.21.15.96:0/643438932 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9d006dc20 msgr2=0x7fd9d014c0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:08.331 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.335+0000 7fd9d6c00640 1 --2- 172.21.15.96:0/643438932 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9d006dc20 0x7fd9d014c0b0 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7fd9c400d800 tx=0x7fd9c400dcc0 comp rx=0 tx=0).stop 2024-05-28T06:22:08.331 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.336+0000 7fd9d6c00640 1 -- 172.21.15.96:0/643438932 shutdown_connections 2024-05-28T06:22:08.331 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.336+0000 7fd9d6c00640 1 --2- 172.21.15.96:0/643438932 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd9b006e090 0x7fd9b0070550 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:08.331 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.336+0000 7fd9d6c00640 1 --2- 172.21.15.96:0/643438932 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9d015b3a0 0x7fd9d01510e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:08.331 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.336+0000 7fd9d6c00640 1 --2- 172.21.15.96:0/643438932 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd9d0073310 0x7fd9d015c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:08.331 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.336+0000 7fd9d6c00640 1 --2- 172.21.15.96:0/643438932 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9d006dc20 0x7fd9d014c0b0 unknown :-1 s=CLOSED pgs=1758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:08.331 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.336+0000 7fd9d6c00640 1 -- 172.21.15.96:0/643438932 >> 172.21.15.96:0/643438932 conn(0x7fd9d00b1fc0 msgr2=0x7fd9d00d0250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:08.331 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.336+0000 7fd9d6c00640 1 -- 172.21.15.96:0/643438932 shutdown_connections 2024-05-28T06:22:08.331 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.336+0000 7fd9d6c00640 1 -- 172.21.15.96:0/643438932 wait complete. 2024-05-28T06:22:08.350 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-28T06:03:59.517102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-28T06:22:08.350 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-28T06:22:08.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.561+0000 7f813ea00640 1 Processor -- start 2024-05-28T06:22:08.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.562+0000 7f813ea00640 1 -- start start 2024-05-28T06:22:08.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.563+0000 7f813ea00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f81380cef30 0x7f81380d1640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:08.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.563+0000 7f813ea00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81380d1f60 con 0x7f813815e850 2024-05-28T06:22:08.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.563+0000 7f813ea00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81380b8000 con 0x7f81380d1b80 2024-05-28T06:22:08.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.563+0000 7f813ea00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f813815ec30 con 0x7f81380cef30 2024-05-28T06:22:08.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.563+0000 7f812fe00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f81380cef30 0x7f81380d1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:08.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.563+0000 7f8137e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f813815e850 0x7f81380b77b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:38048/0 (socket says 172.21.15.96:38048) 2024-05-28T06:22:08.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.564+0000 7f8137e00640 1 -- 172.21.15.96:0/143312449 learned_addr learned my addr 172.21.15.96:0/143312449 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:22:08.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.564+0000 7f812fe00640 1 -- 172.21.15.96:0/143312449 >> v1:172.21.15.123:6789/0 conn(0x7f81380d1b80 legacy=0x7f81380b90a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:08.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.564+0000 7f812fe00640 1 -- 172.21.15.96:0/143312449 >> v1:172.21.15.96:6789/0 conn(0x7f813815e850 legacy=0x7f81380b77b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:08.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.564+0000 7f812fe00640 1 -- 172.21.15.96:0/143312449 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81380b82e0 con 0x7f81380cef30 2024-05-28T06:22:08.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.565+0000 7f812fe00640 1 --2- 172.21.15.96:0/143312449 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f81380cef30 0x7f81380d1640 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7f81200097a0 tx=0x7f812002a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47edbfbdf3d8f74d server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:08.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.565+0000 7f8137400640 1 -- 172.21.15.96:0/143312449 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8120037070 con 0x7f81380cef30 2024-05-28T06:22:08.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.566+0000 7f8137400640 1 -- 172.21.15.96:0/143312449 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f812002ae50 con 0x7f81380cef30 2024-05-28T06:22:08.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.566+0000 7f8137400640 1 -- 172.21.15.96:0/143312449 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8120032660 con 0x7f81380cef30 2024-05-28T06:22:08.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.566+0000 7f813ea00640 1 -- 172.21.15.96:0/143312449 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f81380cef30 msgr2=0x7f81380d1640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:08.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.566+0000 7f813ea00640 1 --2- 172.21.15.96:0/143312449 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f81380cef30 0x7f81380d1640 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7f81200097a0 tx=0x7f812002a140 comp rx=0 tx=0).stop 2024-05-28T06:22:08.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.566+0000 7f813ea00640 1 -- 172.21.15.96:0/143312449 shutdown_connections 2024-05-28T06:22:08.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.566+0000 7f813ea00640 1 --2- 172.21.15.96:0/143312449 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f81380cef30 0x7f81380d1640 unknown :-1 s=CLOSED pgs=1760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:08.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.566+0000 7f813ea00640 1 -- 172.21.15.96:0/143312449 >> 172.21.15.96:0/143312449 conn(0x7f8138067aa0 msgr2=0x7f8138067eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:08.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.566+0000 7f813ea00640 1 -- 172.21.15.96:0/143312449 shutdown_connections 2024-05-28T06:22:08.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.566+0000 7f813ea00640 1 -- 172.21.15.96:0/143312449 wait complete. 2024-05-28T06:22:08.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.568+0000 7f813ea00640 1 Processor -- start 2024-05-28T06:22:08.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.568+0000 7f813ea00640 1 -- start start 2024-05-28T06:22:08.564 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.569+0000 7f813ea00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f81380cef30 0x7f81380d2150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:08.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.569+0000 7f813ea00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f81380d1b80 0x7f81380d4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:08.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.569+0000 7f813ea00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f813815e850 0x7f81380cd1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:08.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.569+0000 7f813ea00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f81380b97b0 con 0x7f81380d1b80 2024-05-28T06:22:08.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.569+0000 7f813ea00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f81380b88a0 con 0x7f81380cef30 2024-05-28T06:22:08.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.569+0000 7f813ea00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f81380b9930 con 0x7f813815e850 2024-05-28T06:22:08.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.569+0000 7f812fe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f81380d1b80 0x7f81380d4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:08.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.569+0000 7f812fe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f81380d1b80 0x7f81380d4aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:36124/0 (socket says 172.21.15.96:36124) 2024-05-28T06:22:08.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.569+0000 7f812fe00640 1 -- 172.21.15.96:0/3484785622 learned_addr learned my addr 172.21.15.96:0/3484785622 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:22:08.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.570+0000 7f8137e00640 1 --2- 172.21.15.96:0/3484785622 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f81380cef30 0x7f81380d2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:08.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.570+0000 7f812fe00640 1 -- 172.21.15.96:0/3484785622 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f813815e850 msgr2=0x7f81380cd1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:08.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.570+0000 7f812fe00640 1 --2- 172.21.15.96:0/3484785622 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f813815e850 0x7f81380cd1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:08.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.570+0000 7f812fe00640 1 -- 172.21.15.96:0/3484785622 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f81380cef30 msgr2=0x7f81380d2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:08.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.570+0000 7f812fe00640 1 --2- 172.21.15.96:0/3484785622 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f81380cef30 0x7f81380d2150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:08.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.570+0000 7f812fe00640 1 -- 172.21.15.96:0/3484785622 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81380d4fe0 con 0x7f81380d1b80 2024-05-28T06:22:08.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.570+0000 7f812fe00640 1 --2- 172.21.15.96:0/3484785622 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f81380d1b80 0x7f81380d4aa0 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7f8120000c00 tx=0x7f81200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:08.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.571+0000 7f8134c00640 1 -- 172.21.15.96:0/3484785622 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8120041070 con 0x7f81380d1b80 2024-05-28T06:22:08.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.571+0000 7f8134c00640 1 -- 172.21.15.96:0/3484785622 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f812003c670 con 0x7f81380d1b80 2024-05-28T06:22:08.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.571+0000 7f8134c00640 1 -- 172.21.15.96:0/3484785622 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8120037040 con 0x7f81380d1b80 2024-05-28T06:22:08.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.571+0000 7f813ea00640 1 -- 172.21.15.96:0/3484785622 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81380cd710 con 0x7f81380d1b80 2024-05-28T06:22:08.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.571+0000 7f813ea00640 1 -- 172.21.15.96:0/3484785622 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f813814d820 con 0x7f81380d1b80 2024-05-28T06:22:08.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.572+0000 7f813ea00640 1 -- 172.21.15.96:0/3484785622 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81380689e0 con 0x7f81380d1b80 2024-05-28T06:22:08.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.573+0000 7f8134c00640 1 -- 172.21.15.96:0/3484785622 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 37) ==== 93182+0+0 (secure 0 0 0) 0x7f8120044070 con 0x7f81380d1b80 2024-05-28T06:22:08.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.574+0000 7f8134c00640 1 --2- 172.21.15.96:0/3484785622 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f811406e090 0x7f8114070550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:08.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.574+0000 7f8134c00640 1 -- 172.21.15.96:0/3484785622 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 5027+0+0 (secure 0 0 0) 0x7f81200ab240 con 0x7f81380d1b80 2024-05-28T06:22:08.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.575+0000 7f8137e00640 1 --2- 172.21.15.96:0/3484785622 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f811406e090 0x7f8114070550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:08.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.576+0000 7f8137e00640 1 --2- 172.21.15.96:0/3484785622 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f811406e090 0x7f8114070550 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f8128000f60 tx=0x7f8128009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:08.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.578+0000 7f8134c00640 1 -- 172.21.15.96:0/3484785622 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f81380d4fe0 con 0x7f81380d1b80 2024-05-28T06:22:08.770 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.774+0000 7f813ea00640 1 -- 172.21.15.96:0/3484785622 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f81380c7be0 con 0x7f81380d1b80 2024-05-28T06:22:08.770 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.775+0000 7f8134c00640 1 -- 172.21.15.96:0/3484785622 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v107) ==== 93+0+10 (secure 0 0 0) 0x7f812007b1f0 con 0x7f81380d1b80 2024-05-28T06:22:08.771 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 1 2024-05-28T06:22:08.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.777+0000 7f813ea00640 1 -- 172.21.15.96:0/3484785622 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f811406e090 msgr2=0x7f8114070550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:08.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.777+0000 7f813ea00640 1 --2- 172.21.15.96:0/3484785622 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f811406e090 0x7f8114070550 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f8128000f60 tx=0x7f8128009040 comp rx=0 tx=0).stop 2024-05-28T06:22:08.774 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.777+0000 7f813ea00640 1 -- 172.21.15.96:0/3484785622 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f81380d1b80 msgr2=0x7f81380d4aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:08.774 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.777+0000 7f813ea00640 1 --2- 172.21.15.96:0/3484785622 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f81380d1b80 0x7f81380d4aa0 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7f8120000c00 tx=0x7f81200046c0 comp rx=0 tx=0).stop 2024-05-28T06:22:08.774 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.778+0000 7f813ea00640 1 -- 172.21.15.96:0/3484785622 shutdown_connections 2024-05-28T06:22:08.774 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.778+0000 7f813ea00640 1 --2- 172.21.15.96:0/3484785622 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f811406e090 0x7f8114070550 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:08.774 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.778+0000 7f813ea00640 1 --2- 172.21.15.96:0/3484785622 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f813815e850 0x7f81380cd1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:08.774 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.778+0000 7f813ea00640 1 --2- 172.21.15.96:0/3484785622 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f81380d1b80 0x7f81380d4aa0 unknown :-1 s=CLOSED pgs=1761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:08.774 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.778+0000 7f813ea00640 1 --2- 172.21.15.96:0/3484785622 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f81380cef30 0x7f81380d2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:08.774 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.778+0000 7f813ea00640 1 -- 172.21.15.96:0/3484785622 >> 172.21.15.96:0/3484785622 conn(0x7f8138067aa0 msgr2=0x7f81380b5900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:08.774 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.778+0000 7f813ea00640 1 -- 172.21.15.96:0/3484785622 shutdown_connections 2024-05-28T06:22:08.774 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.778+0000 7f813ea00640 1 -- 172.21.15.96:0/3484785622 wait complete. 2024-05-28T06:22:08.793 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestCloneProgressReporter.test_when_3_clones_are_cancelled' 2024-05-28T06:22:08.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.996+0000 7fc926400640 1 Processor -- start 2024-05-28T06:22:08.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.996+0000 7fc926400640 1 -- start start 2024-05-28T06:22:08.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.998+0000 7fc926400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fc92015e870 0x7fc9200d0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:08.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.998+0000 7fc926400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fc9200d14d0 0x7fc9200cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:08.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.998+0000 7fc926400640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9200cfce0 con 0x7fc9200cf870 2024-05-28T06:22:08.995 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.998+0000 7fc926400640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fc9200d2150 con 0x7fc92015e870 2024-05-28T06:22:08.995 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.998+0000 7fc926400640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fc92015ec50 con 0x7fc9200d14d0 2024-05-28T06:22:08.995 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.998+0000 7fc925000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fc92015e870 0x7fc9200d0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:08.995 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.998+0000 7fc925000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fc92015e870 0x7fc9200d0f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:59758/0 (socket says 172.21.15.96:59758) 2024-05-28T06:22:08.995 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.998+0000 7fc925000640 1 -- 172.21.15.96:0/1762159158 learned_addr learned my addr 172.21.15.96:0/1762159158 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:22:08.995 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.999+0000 7fc91fe00640 1 --2- 172.21.15.96:0/1762159158 >> v2:172.21.15.123:3300/0 conn(0x7fc9200d14d0 0x7fc9200cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:08.995 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.999+0000 7fc925000640 1 -- 172.21.15.96:0/1762159158 >> v2:172.21.15.123:3300/0 conn(0x7fc9200d14d0 msgr2=0x7fc9200cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:08.995 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.999+0000 7fc925000640 1 --2- 172.21.15.96:0/1762159158 >> v2:172.21.15.123:3300/0 conn(0x7fc9200d14d0 0x7fc9200cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:08.995 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.999+0000 7fc925000640 1 -- 172.21.15.96:0/1762159158 >> v1:172.21.15.123:6789/0 conn(0x7fc9200cf870 legacy=0x7fc920162360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:08.995 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:08.999+0000 7fc925000640 1 -- 172.21.15.96:0/1762159158 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9200cff80 con 0x7fc92015e870 2024-05-28T06:22:08.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:09.000+0000 7fc925000640 1 --2- 172.21.15.96:0/1762159158 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc92015e870 0x7fc9200d0f90 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7fc92008e520 tx=0x7fc91002a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5583b4399d4ce36f server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:08.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:09.001+0000 7fc91f400640 1 -- 172.21.15.96:0/1762159158 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc910036070 con 0x7fc92015e870 2024-05-28T06:22:08.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:09.001+0000 7fc91f400640 1 -- 172.21.15.96:0/1762159158 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc910031600 con 0x7fc92015e870 2024-05-28T06:22:08.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:09.001+0000 7fc91f400640 1 -- 172.21.15.96:0/1762159158 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9100319d0 con 0x7fc92015e870 2024-05-28T06:22:08.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:09.001+0000 7fc926400640 1 -- 172.21.15.96:0/1762159158 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc92015e870 msgr2=0x7fc9200d0f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:08.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:09.001+0000 7fc926400640 1 --2- 172.21.15.96:0/1762159158 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc92015e870 0x7fc9200d0f90 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7fc92008e520 tx=0x7fc91002a6f0 comp rx=0 tx=0).stop 2024-05-28T06:22:08.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:09.001+0000 7fc926400640 1 -- 172.21.15.96:0/1762159158 shutdown_connections 2024-05-28T06:22:08.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:09.001+0000 7fc926400640 1 --2- 172.21.15.96:0/1762159158 >> v2:172.21.15.123:3300/0 conn(0x7fc9200d14d0 0x7fc9200cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:08.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:09.001+0000 7fc926400640 1 --2- 172.21.15.96:0/1762159158 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc92015e870 0x7fc9200d0f90 unknown :-1 s=CLOSED pgs=1762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:08.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:09.001+0000 7fc926400640 1 -- 172.21.15.96:0/1762159158 >> 172.21.15.96:0/1762159158 conn(0x7fc92008bfc0 msgr2=0x7fc9200d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:08.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:09.001+0000 7fc926400640 1 -- 172.21.15.96:0/1762159158 shutdown_connections 2024-05-28T06:22:08.998 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:09.002+0000 7fc926400640 1 -- 172.21.15.96:0/1762159158 wait complete. 2024-05-28T06:22:08.999 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:09.003+0000 7fc926400640 1 Processor -- start 2024-05-28T06:22:08.999 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:09.003+0000 7fc926400640 1 -- start start 2024-05-28T06:22:09.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:09.004+0000 7fc926400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc9200cf870 0x7fc920159a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:09.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:09.004+0000 7fc926400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc9200d14d0 0x7fc9200b8c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:09.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:09.004+0000 7fc926400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc92015e870 0x7fc9200cd820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:09.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:09.004+0000 7fc926400640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc9200d0820 con 0x7fc9200d14d0 2024-05-28T06:22:09.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:09.004+0000 7fc926400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc9200d0540 con 0x7fc92015e870 2024-05-28T06:22:09.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:09.004+0000 7fc926400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fc9200d4fb0 con 0x7fc9200cf870 2024-05-28T06:22:09.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:09.004+0000 7fc91fe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc9200d14d0 0x7fc9200b8c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:09.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:09.005+0000 7fc91fe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc9200d14d0 0x7fc9200b8c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:59770/0 (socket says 172.21.15.96:59770) 2024-05-28T06:22:09.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:09.005+0000 7fc91fe00640 1 -- 172.21.15.96:0/1770338331 learned_addr learned my addr 172.21.15.96:0/1770338331 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:22:09.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:09.005+0000 7fc925a00640 1 --2- 172.21.15.96:0/1770338331 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc92015e870 0x7fc9200cd820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:09.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:09.005+0000 7fc925000640 1 --2- 172.21.15.96:0/1770338331 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc9200cf870 0x7fc920159a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:09.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:09.005+0000 7fc91fe00640 1 -- 172.21.15.96:0/1770338331 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc9200cf870 msgr2=0x7fc920159a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:09.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:09.005+0000 7fc91fe00640 1 --2- 172.21.15.96:0/1770338331 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc9200cf870 0x7fc920159a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:09.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:09.005+0000 7fc91fe00640 1 -- 172.21.15.96:0/1770338331 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc92015e870 msgr2=0x7fc9200cd820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:09.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:09.005+0000 7fc91fe00640 1 --2- 172.21.15.96:0/1770338331 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc92015e870 0x7fc9200cd820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:09.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:09.005+0000 7fc91fe00640 1 -- 172.21.15.96:0/1770338331 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9201f43d0 con 0x7fc9200d14d0 2024-05-28T06:22:09.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:09.005+0000 7fc91fe00640 1 --2- 172.21.15.96:0/1770338331 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc9200d14d0 0x7fc9200b8c30 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7fc90c00d800 tx=0x7fc90c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:09.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:09.005+0000 7fc91cc00640 1 -- 172.21.15.96:0/1770338331 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc90c00b6c0 con 0x7fc9200d14d0 2024-05-28T06:22:09.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:09.006+0000 7fc91cc00640 1 -- 172.21.15.96:0/1770338331 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc90c019070 con 0x7fc9200d14d0 2024-05-28T06:22:09.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:09.006+0000 7fc91cc00640 1 -- 172.21.15.96:0/1770338331 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc90c00baa0 con 0x7fc9200d14d0 2024-05-28T06:22:09.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:09.006+0000 7fc926400640 1 -- 172.21.15.96:0/1770338331 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9201f46b0 con 0x7fc9200d14d0 2024-05-28T06:22:09.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:09.006+0000 7fc926400640 1 -- 172.21.15.96:0/1770338331 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9201f4c70 con 0x7fc9200d14d0 2024-05-28T06:22:09.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:09.007+0000 7fc926400640 1 -- 172.21.15.96:0/1770338331 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9200689e0 con 0x7fc9200d14d0 2024-05-28T06:22:09.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:09.008+0000 7fc91cc00640 1 -- 172.21.15.96:0/1770338331 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 37) ==== 93182+0+0 (secure 0 0 0) 0x7fc90c00bc40 con 0x7fc9200d14d0 2024-05-28T06:22:09.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:09.009+0000 7fc91cc00640 1 --2- 172.21.15.96:0/1770338331 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc90006e090 0x7fc900070550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:09.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:09.009+0000 7fc91cc00640 1 -- 172.21.15.96:0/1770338331 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 5027+0+0 (secure 0 0 0) 0x7fc90c08ade0 con 0x7fc9200d14d0 2024-05-28T06:22:09.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:09.009+0000 7fc925000640 1 --2- 172.21.15.96:0/1770338331 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc90006e090 0x7fc900070550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:09.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:09.010+0000 7fc925000640 1 --2- 172.21.15.96:0/1770338331 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc90006e090 0x7fc900070550 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fc910033000 tx=0x7fc910009000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:09.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:09.013+0000 7fc91cc00640 1 -- 172.21.15.96:0/1770338331 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc90c05ac70 con 0x7fc9200d14d0 2024-05-28T06:22:09.281 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:09.286+0000 7fc926400640 1 -- 172.21.15.96:0/1770338331 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgressReporter.test_when_3_clones_are_cancelled"]} v 0) -- 0x7fc92006e670 con 0x7fc9200d14d0 2024-05-28T06:22:09.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:09.895+0000 7fc91cc00640 1 -- 172.21.15.96:0/1770338331 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgressReporter.test_when_3_clones_are_cancelled"]}]=0 v790) ==== 166+0+0 (secure 0 0 0) 0x7fc90c05a570 con 0x7fc9200d14d0 2024-05-28T06:22:09.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:09.898+0000 7fc926400640 1 -- 172.21.15.96:0/1770338331 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc90006e090 msgr2=0x7fc900070550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:09.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:09.898+0000 7fc926400640 1 --2- 172.21.15.96:0/1770338331 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc90006e090 0x7fc900070550 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fc910033000 tx=0x7fc910009000 comp rx=0 tx=0).stop 2024-05-28T06:22:09.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:09.898+0000 7fc926400640 1 -- 172.21.15.96:0/1770338331 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc9200d14d0 msgr2=0x7fc9200b8c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:09.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:09.898+0000 7fc926400640 1 --2- 172.21.15.96:0/1770338331 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc9200d14d0 0x7fc9200b8c30 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7fc90c00d800 tx=0x7fc90c00dcc0 comp rx=0 tx=0).stop 2024-05-28T06:22:09.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:09.898+0000 7fc926400640 1 -- 172.21.15.96:0/1770338331 shutdown_connections 2024-05-28T06:22:09.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:09.898+0000 7fc926400640 1 --2- 172.21.15.96:0/1770338331 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc90006e090 0x7fc900070550 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:09.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:09.898+0000 7fc926400640 1 --2- 172.21.15.96:0/1770338331 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc92015e870 0x7fc9200cd820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:09.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:09.898+0000 7fc926400640 1 --2- 172.21.15.96:0/1770338331 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc9200d14d0 0x7fc9200b8c30 unknown :-1 s=CLOSED pgs=1763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:09.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:09.898+0000 7fc926400640 1 --2- 172.21.15.96:0/1770338331 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc9200cf870 0x7fc920159a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:09.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:09.898+0000 7fc926400640 1 -- 172.21.15.96:0/1770338331 >> 172.21.15.96:0/1770338331 conn(0x7fc92008bfc0 msgr2=0x7fc9200b2ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:09.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:09.899+0000 7fc926400640 1 -- 172.21.15.96:0/1770338331 shutdown_connections 2024-05-28T06:22:09.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:09.899+0000 7fc926400640 1 -- 172.21.15.96:0/1770338331 wait complete. 2024-05-28T06:22:09.916 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-05-28T06:22:10.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.122+0000 7fe1b1a00640 1 Processor -- start 2024-05-28T06:22:10.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.122+0000 7fe1b1a00640 1 -- start start 2024-05-28T06:22:10.119 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.123+0000 7fe1b1a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe1ac0cd2a0 0x7fe1ac0b4d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:10.119 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.123+0000 7fe1b1a00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1ac15d900 con 0x7fe1ac153510 2024-05-28T06:22:10.119 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.123+0000 7fe1b1a00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1ac0b3f90 con 0x7fe1ac15cb60 2024-05-28T06:22:10.119 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.123+0000 7fe1b1a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fe1ac15d6e0 con 0x7fe1ac0cd2a0 2024-05-28T06:22:10.119 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.124+0000 7fe1abe00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fe1ac153510 0x7fe1ac0ccb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:33100/0 (socket says 172.21.15.96:33100) 2024-05-28T06:22:10.119 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.124+0000 7fe1abe00640 1 -- 172.21.15.96:0/2367920063 learned_addr learned my addr 172.21.15.96:0/2367920063 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:22:10.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.124+0000 7fe1aaa00640 1 -- 172.21.15.96:0/2367920063 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1373882316 0 0) 0x7fe1ac15d900 con 0x7fe1ac153510 2024-05-28T06:22:10.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.124+0000 7fe1aaa00640 1 -- 172.21.15.96:0/2367920063 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe190003520 con 0x7fe1ac153510 2024-05-28T06:22:10.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.124+0000 7fe1ab400640 1 --2- 172.21.15.96:0/2367920063 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe1ac0cd2a0 0x7fe1ac0b4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:10.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.124+0000 7fe1aaa00640 1 -- 172.21.15.96:0/2367920063 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3539793952 0 0) 0x7fe190003520 con 0x7fe1ac153510 2024-05-28T06:22:10.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.124+0000 7fe1aaa00640 1 -- 172.21.15.96:0/2367920063 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe1ac0cd2a0 msgr2=0x7fe1ac0b4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:10.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.124+0000 7fe1aaa00640 1 --2- 172.21.15.96:0/2367920063 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe1ac0cd2a0 0x7fe1ac0b4d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:10.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.124+0000 7fe1aaa00640 1 -- 172.21.15.96:0/2367920063 >> v1:172.21.15.123:6789/0 conn(0x7fe1ac15cb60 legacy=0x7fe1ac15cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:10.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.124+0000 7fe1aaa00640 1 -- 172.21.15.96:0/2367920063 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1ac0b4270 con 0x7fe1ac153510 2024-05-28T06:22:10.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.125+0000 7fe1aaa00640 1 -- 172.21.15.96:0/2367920063 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7fe19c001670 con 0x7fe1ac153510 2024-05-28T06:22:10.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.125+0000 7fe1aaa00640 1 -- 172.21.15.96:0/2367920063 >> v1:172.21.15.96:6789/0 conn(0x7fe1ac153510 legacy=0x7fe1ac0ccb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:10.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.125+0000 7fe1aaa00640 1 --2- 172.21.15.96:0/2367920063 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe1900039e0 0x7fe190003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:10.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.125+0000 7fe1aaa00640 1 --2- 172.21.15.96:0/2367920063 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe190004800 0x7fe190004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:10.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.125+0000 7fe1aaa00640 1 --2- 172.21.15.96:0/2367920063 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe1ac0cd2a0 0x7fe19000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:10.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.125+0000 7fe1aaa00640 1 -- 172.21.15.96:0/2367920063 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe1ac15d6e0 con 0x7fe1900039e0 2024-05-28T06:22:10.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.125+0000 7fe1aaa00640 1 -- 172.21.15.96:0/2367920063 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe1900054f0 con 0x7fe190004800 2024-05-28T06:22:10.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.125+0000 7fe1aaa00640 1 -- 172.21.15.96:0/2367920063 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fe19000bf00 con 0x7fe1ac0cd2a0 2024-05-28T06:22:10.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.125+0000 7fe1abe00640 1 --2- 172.21.15.96:0/2367920063 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe1900039e0 0x7fe190003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:10.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.125+0000 7fe1ab400640 1 --2- 172.21.15.96:0/2367920063 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe190004800 0x7fe190004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:10.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.125+0000 7fe1abe00640 1 -- 172.21.15.96:0/2367920063 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe1ac0cd2a0 msgr2=0x7fe19000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:10.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.125+0000 7fe1abe00640 1 --2- 172.21.15.96:0/2367920063 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe1ac0cd2a0 0x7fe19000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:10.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.125+0000 7fe1abe00640 1 -- 172.21.15.96:0/2367920063 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe190004800 msgr2=0x7fe190004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:10.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.125+0000 7fe1abe00640 1 --2- 172.21.15.96:0/2367920063 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe190004800 0x7fe190004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:10.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.125+0000 7fe1abe00640 1 -- 172.21.15.96:0/2367920063 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe190003520 con 0x7fe1900039e0 2024-05-28T06:22:10.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.126+0000 7fe1abe00640 1 --2- 172.21.15.96:0/2367920063 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe1900039e0 0x7fe190003e00 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7fe19c00c110 tx=0x7fe190005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8e223d089b4c631 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:10.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.127+0000 7fe1aaa00640 1 -- 172.21.15.96:0/2367920063 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe19c007c40 con 0x7fe1900039e0 2024-05-28T06:22:10.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.127+0000 7fe1aaa00640 1 -- 172.21.15.96:0/2367920063 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe19c005af0 con 0x7fe1900039e0 2024-05-28T06:22:10.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.127+0000 7fe1aaa00640 1 -- 172.21.15.96:0/2367920063 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe19c007070 con 0x7fe1900039e0 2024-05-28T06:22:10.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.127+0000 7fe1b1a00640 1 -- 172.21.15.96:0/2367920063 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe1900039e0 msgr2=0x7fe190003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:10.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.127+0000 7fe1b1a00640 1 --2- 172.21.15.96:0/2367920063 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe1900039e0 0x7fe190003e00 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7fe19c00c110 tx=0x7fe190005830 comp rx=0 tx=0).stop 2024-05-28T06:22:10.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.127+0000 7fe1b1000640 1 -- 172.21.15.96:0/2367920063 reap_dead start 2024-05-28T06:22:10.125 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.127+0000 7fe1b1a00640 1 -- 172.21.15.96:0/2367920063 shutdown_connections 2024-05-28T06:22:10.125 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.127+0000 7fe1b1a00640 1 -- 172.21.15.96:0/2367920063 >> 172.21.15.96:0/2367920063 conn(0x7fe1ac06b900 msgr2=0x7fe1ac06bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:10.125 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.127+0000 7fe1b1a00640 1 -- 172.21.15.96:0/2367920063 shutdown_connections 2024-05-28T06:22:10.125 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.127+0000 7fe1b1a00640 1 -- 172.21.15.96:0/2367920063 wait complete. 2024-05-28T06:22:10.125 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.129+0000 7fe1b1a00640 1 Processor -- start 2024-05-28T06:22:10.125 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.129+0000 7fe1b1a00640 1 -- start start 2024-05-28T06:22:10.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.130+0000 7fe1b1a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe1900039e0 0x7fe1ac13cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:10.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.130+0000 7fe1b1a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe1ac0adad0 0x7fe1ac0adf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:10.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.130+0000 7fe1b1a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe1ac072e70 0x7fe1ac073320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:10.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.130+0000 7fe1b1a00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe1ac0b48c0 con 0x7fe1900039e0 2024-05-28T06:22:10.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.130+0000 7fe1b1a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe1ac0b4d50 con 0x7fe1ac0adad0 2024-05-28T06:22:10.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.130+0000 7fe1b1a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fe1ac0b4ed0 con 0x7fe1ac072e70 2024-05-28T06:22:10.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.131+0000 7fe1abe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe1900039e0 0x7fe1ac13cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:10.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.131+0000 7fe1abe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe1900039e0 0x7fe1ac13cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:59798/0 (socket says 172.21.15.96:59798) 2024-05-28T06:22:10.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.131+0000 7fe1abe00640 1 -- 172.21.15.96:0/1864837034 learned_addr learned my addr 172.21.15.96:0/1864837034 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:22:10.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.131+0000 7fe1ab400640 1 --2- 172.21.15.96:0/1864837034 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe1ac0adad0 0x7fe1ac0adf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:10.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.131+0000 7fe1b1000640 1 --2- 172.21.15.96:0/1864837034 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe1ac072e70 0x7fe1ac073320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:10.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.131+0000 7fe1abe00640 1 -- 172.21.15.96:0/1864837034 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe1ac072e70 msgr2=0x7fe1ac073320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:10.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.131+0000 7fe1abe00640 1 --2- 172.21.15.96:0/1864837034 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe1ac072e70 0x7fe1ac073320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:10.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.131+0000 7fe1abe00640 1 -- 172.21.15.96:0/1864837034 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe1ac0adad0 msgr2=0x7fe1ac0adf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:10.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.131+0000 7fe1abe00640 1 --2- 172.21.15.96:0/1864837034 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe1ac0adad0 0x7fe1ac0adf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:10.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.131+0000 7fe1abe00640 1 -- 172.21.15.96:0/1864837034 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1ac13d2a0 con 0x7fe1900039e0 2024-05-28T06:22:10.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.131+0000 7fe1ab400640 1 --2- 172.21.15.96:0/1864837034 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe1ac0adad0 0x7fe1ac0adf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:22:10.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.131+0000 7fe1abe00640 1 --2- 172.21.15.96:0/1864837034 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe1900039e0 0x7fe1ac13cd60 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7fe19c02ad60 tx=0x7fe1ac15c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:10.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.132+0000 7fe1a3e00640 1 -- 172.21.15.96:0/1864837034 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe19c043070 con 0x7fe1900039e0 2024-05-28T06:22:10.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.132+0000 7fe1a3e00640 1 -- 172.21.15.96:0/1864837034 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe19c038040 con 0x7fe1900039e0 2024-05-28T06:22:10.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.132+0000 7fe1a3e00640 1 -- 172.21.15.96:0/1864837034 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe19c007070 con 0x7fe1900039e0 2024-05-28T06:22:10.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.132+0000 7fe1b1a00640 1 -- 172.21.15.96:0/1864837034 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1ac071dd0 con 0x7fe1900039e0 2024-05-28T06:22:10.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.132+0000 7fe1b1a00640 1 -- 172.21.15.96:0/1864837034 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1ac072390 con 0x7fe1900039e0 2024-05-28T06:22:10.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.133+0000 7fe1b1a00640 1 -- 172.21.15.96:0/1864837034 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1ac0689e0 con 0x7fe1900039e0 2024-05-28T06:22:10.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.134+0000 7fe1a3e00640 1 -- 172.21.15.96:0/1864837034 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 37) ==== 93182+0+0 (secure 0 0 0) 0x7fe19c007210 con 0x7fe1900039e0 2024-05-28T06:22:10.131 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.135+0000 7fe1a3e00640 1 --2- 172.21.15.96:0/1864837034 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe18c06e090 0x7fe18c070550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:10.131 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.135+0000 7fe1a3e00640 1 -- 172.21.15.96:0/1864837034 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 5027+0+0 (secure 0 0 0) 0x7fe19c0aa5f0 con 0x7fe1900039e0 2024-05-28T06:22:10.131 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.136+0000 7fe1ab400640 1 --2- 172.21.15.96:0/1864837034 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe18c06e090 0x7fe18c070550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:10.132 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.137+0000 7fe1ab400640 1 --2- 172.21.15.96:0/1864837034 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe18c06e090 0x7fe18c070550 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fe1ac150d60 tx=0x7fe198008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:10.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.139+0000 7fe1a3e00640 1 -- 172.21.15.96:0/1864837034 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe19c07a480 con 0x7fe1900039e0 2024-05-28T06:22:10.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.341+0000 7fe1b1a00640 1 -- 172.21.15.96:0/1864837034 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7fe1ac0b3c40 con 0x7fe1900039e0 2024-05-28T06:22:10.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.347+0000 7fe1a3e00640 1 -- 172.21.15.96:0/1864837034 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v26)=0 v26) ==== 122+0+0 (secure 0 0 0) 0x7fe19c079d80 con 0x7fe1900039e0 2024-05-28T06:22:10.346 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.350+0000 7fe1b1a00640 1 -- 172.21.15.96:0/1864837034 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe18c06e090 msgr2=0x7fe18c070550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:10.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.350+0000 7fe1b1a00640 1 --2- 172.21.15.96:0/1864837034 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe18c06e090 0x7fe18c070550 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fe1ac150d60 tx=0x7fe198008040 comp rx=0 tx=0).stop 2024-05-28T06:22:10.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.350+0000 7fe1b1a00640 1 -- 172.21.15.96:0/1864837034 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe1900039e0 msgr2=0x7fe1ac13cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:10.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.350+0000 7fe1b1a00640 1 --2- 172.21.15.96:0/1864837034 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe1900039e0 0x7fe1ac13cd60 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7fe19c02ad60 tx=0x7fe1ac15c960 comp rx=0 tx=0).stop 2024-05-28T06:22:10.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.350+0000 7fe1b1a00640 1 -- 172.21.15.96:0/1864837034 shutdown_connections 2024-05-28T06:22:10.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.350+0000 7fe1b1a00640 1 --2- 172.21.15.96:0/1864837034 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe18c06e090 0x7fe18c070550 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:10.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.350+0000 7fe1b1a00640 1 --2- 172.21.15.96:0/1864837034 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe1ac072e70 0x7fe1ac073320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:10.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.350+0000 7fe1b1a00640 1 --2- 172.21.15.96:0/1864837034 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe1ac0adad0 0x7fe1ac0adf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:10.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.350+0000 7fe1b1a00640 1 --2- 172.21.15.96:0/1864837034 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe1900039e0 0x7fe1ac13cd60 unknown :-1 s=CLOSED pgs=1766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:10.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.350+0000 7fe1b1a00640 1 -- 172.21.15.96:0/1864837034 >> 172.21.15.96:0/1864837034 conn(0x7fe1ac06b900 msgr2=0x7fe1ac0cb630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:10.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.350+0000 7fe1b1a00640 1 -- 172.21.15.96:0/1864837034 shutdown_connections 2024-05-28T06:22:10.348 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:10.350+0000 7fe1b1a00640 1 -- 172.21.15.96:0/1864837034 wait complete. 2024-05-28T06:22:10.366 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-05-28T06:22:10.366 DEBUG:teuthology.orchestra.run.smithi096:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-05-28T06:22:10.465 DEBUG:teuthology.orchestra.run.smithi123:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-05-28T06:22:10.579 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-05-28T06:22:10.579 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:22:10.581 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:22:10.640 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052731,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248996k,nr_inodes=812249,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-28T06:22:10.640 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-05-28T06:22:10.640 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:22:10.697 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052731,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248996k,nr_inodes=812249,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-05-28T06:22:10.697 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:22:10.697 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:22:10.754 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052731,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248996k,nr_inodes=812249,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-28T06:22:10.754 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-05-28T06:22:10.754 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:22:10.811 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052731,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248996k,nr_inodes=812249,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-05-28T06:22:10.811 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-05-28T06:22:10.812 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-28T06:22:11.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.023+0000 7f54c4000640 1 Processor -- start 2024-05-28T06:22:11.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.023+0000 7f54c4000640 1 -- start start 2024-05-28T06:22:11.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.025+0000 7f54c4000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f54bc0d04a0 0x7f54bc0d0930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:11.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.025+0000 7f54c4000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f54bc0ad130 0x7f54bc0ad540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:11.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.025+0000 7f54c4000640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f54bc06ed10 con 0x7f54bc06f860 2024-05-28T06:22:11.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.025+0000 7f54c4000640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f54bc15cb00 con 0x7f54bc0ad130 2024-05-28T06:22:11.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.025+0000 7f54c4000640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f54bc0b3cc0 con 0x7f54bc0d04a0 2024-05-28T06:22:11.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.025+0000 7f54c2000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f54bc0ad130 0x7f54bc0ad540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:11.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.025+0000 7f54c2000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f54bc0ad130 0x7f54bc0ad540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:59804/0 (socket says 172.21.15.96:59804) 2024-05-28T06:22:11.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.025+0000 7f54c2000640 1 -- 172.21.15.96:0/2035521500 learned_addr learned my addr 172.21.15.96:0/2035521500 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:22:11.022 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.026+0000 7f54c0c00640 1 --2- 172.21.15.96:0/2035521500 >> v2:172.21.15.123:3300/0 conn(0x7f54bc0d04a0 0x7f54bc0d0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:11.022 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.026+0000 7f54c2000640 1 -- 172.21.15.96:0/2035521500 >> v2:172.21.15.123:3300/0 conn(0x7f54bc0d04a0 msgr2=0x7f54bc0d0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:11.022 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.026+0000 7f54c2000640 1 --2- 172.21.15.96:0/2035521500 >> v2:172.21.15.123:3300/0 conn(0x7f54bc0d04a0 0x7f54bc0d0930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:11.022 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.026+0000 7f54c2000640 1 -- 172.21.15.96:0/2035521500 >> v1:172.21.15.123:6789/0 conn(0x7f54bc06f860 legacy=0x7f54bc0b3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:11.022 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.026+0000 7f54c2000640 1 -- 172.21.15.96:0/2035521500 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54bc0b3fa0 con 0x7f54bc0ad130 2024-05-28T06:22:11.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.027+0000 7f54c2000640 1 --2- 172.21.15.96:0/2035521500 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f54bc0ad130 0x7f54bc0ad540 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7f54bc070120 tx=0x7f54b402a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3629e6dd197e196 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:11.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.027+0000 7f54bbe00640 1 -- 172.21.15.96:0/2035521500 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54b4036070 con 0x7f54bc0ad130 2024-05-28T06:22:11.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.027+0000 7f54bbe00640 1 -- 172.21.15.96:0/2035521500 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54b4031600 con 0x7f54bc0ad130 2024-05-28T06:22:11.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.027+0000 7f54bbe00640 1 -- 172.21.15.96:0/2035521500 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54b40319d0 con 0x7f54bc0ad130 2024-05-28T06:22:11.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.028+0000 7f54c4000640 1 -- 172.21.15.96:0/2035521500 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f54bc0ad130 msgr2=0x7f54bc0ad540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:11.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.028+0000 7f54c4000640 1 --2- 172.21.15.96:0/2035521500 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f54bc0ad130 0x7f54bc0ad540 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7f54bc070120 tx=0x7f54b402a6f0 comp rx=0 tx=0).stop 2024-05-28T06:22:11.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.028+0000 7f54c4000640 1 -- 172.21.15.96:0/2035521500 shutdown_connections 2024-05-28T06:22:11.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.028+0000 7f54c4000640 1 --2- 172.21.15.96:0/2035521500 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f54bc0ad130 0x7f54bc0ad540 unknown :-1 s=CLOSED pgs=1767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:11.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.028+0000 7f54c4000640 1 --2- 172.21.15.96:0/2035521500 >> v2:172.21.15.123:3300/0 conn(0x7f54bc0d04a0 0x7f54bc0d0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:11.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.028+0000 7f54c4000640 1 -- 172.21.15.96:0/2035521500 >> 172.21.15.96:0/2035521500 conn(0x7f54bc06bae0 msgr2=0x7f54bc06bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:11.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.028+0000 7f54c4000640 1 -- 172.21.15.96:0/2035521500 shutdown_connections 2024-05-28T06:22:11.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.028+0000 7f54c4000640 1 -- 172.21.15.96:0/2035521500 wait complete. 2024-05-28T06:22:11.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.030+0000 7f54c4000640 1 Processor -- start 2024-05-28T06:22:11.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.031+0000 7f54c4000640 1 -- start start 2024-05-28T06:22:11.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.032+0000 7f54c4000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f54bc06f860 0x7f54bc0c4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:11.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.032+0000 7f54c4000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f54bc0ad130 0x7f54bc0c88d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:11.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.032+0000 7f54c4000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f54bc0d04a0 0x7f54bc0c8e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:11.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.032+0000 7f54c4000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f54bc0b4840 con 0x7f54bc06f860 2024-05-28T06:22:11.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.032+0000 7f54c4000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f54bc0b4560 con 0x7f54bc0d04a0 2024-05-28T06:22:11.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.032+0000 7f54c4000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f54bc0b3860 con 0x7f54bc0ad130 2024-05-28T06:22:11.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.032+0000 7f54c1600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f54bc06f860 0x7f54bc0c4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:11.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.032+0000 7f54c1600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f54bc06f860 0x7f54bc0c4aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:59812/0 (socket says 172.21.15.96:59812) 2024-05-28T06:22:11.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.032+0000 7f54c1600640 1 -- 172.21.15.96:0/566812892 learned_addr learned my addr 172.21.15.96:0/566812892 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:22:11.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.033+0000 7f54c2000640 1 --2- 172.21.15.96:0/566812892 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f54bc0d04a0 0x7f54bc0c8e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:11.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.033+0000 7f54c0c00640 1 --2- 172.21.15.96:0/566812892 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f54bc0ad130 0x7f54bc0c88d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:11.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.033+0000 7f54c1600640 1 -- 172.21.15.96:0/566812892 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f54bc0ad130 msgr2=0x7f54bc0c88d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:11.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.033+0000 7f54c1600640 1 --2- 172.21.15.96:0/566812892 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f54bc0ad130 0x7f54bc0c88d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:11.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.033+0000 7f54c1600640 1 -- 172.21.15.96:0/566812892 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f54bc0d04a0 msgr2=0x7f54bc0c8e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:11.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.033+0000 7f54c1600640 1 --2- 172.21.15.96:0/566812892 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f54bc0d04a0 0x7f54bc0c8e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:11.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.033+0000 7f54c1600640 1 -- 172.21.15.96:0/566812892 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54bc0c6600 con 0x7f54bc06f860 2024-05-28T06:22:11.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.033+0000 7f54c0c00640 1 --2- 172.21.15.96:0/566812892 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f54bc0ad130 0x7f54bc0c88d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:22:11.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.033+0000 7f54c1600640 1 --2- 172.21.15.96:0/566812892 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f54bc06f860 0x7f54bc0c4aa0 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7f54b000c430 tx=0x7f54b000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:11.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.033+0000 7f54b9600640 1 -- 172.21.15.96:0/566812892 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54b0011790 con 0x7f54bc06f860 2024-05-28T06:22:11.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.033+0000 7f54b9600640 1 -- 172.21.15.96:0/566812892 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54b0017070 con 0x7f54bc06f860 2024-05-28T06:22:11.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.033+0000 7f54b9600640 1 -- 172.21.15.96:0/566812892 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54b0011b70 con 0x7f54bc06f860 2024-05-28T06:22:11.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.033+0000 7f54c4000640 1 -- 172.21.15.96:0/566812892 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54bc0c5560 con 0x7f54bc06f860 2024-05-28T06:22:11.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.034+0000 7f54c4000640 1 -- 172.21.15.96:0/566812892 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54bc0c5b20 con 0x7f54bc06f860 2024-05-28T06:22:11.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.035+0000 7f54c4000640 1 -- 172.21.15.96:0/566812892 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54bc0d0e10 con 0x7f54bc06f860 2024-05-28T06:22:11.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.036+0000 7f54b9600640 1 -- 172.21.15.96:0/566812892 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 37) ==== 93182+0+0 (secure 0 0 0) 0x7f54b0011d10 con 0x7f54bc06f860 2024-05-28T06:22:11.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.037+0000 7f54b9600640 1 --2- 172.21.15.96:0/566812892 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f549c06e090 0x7f549c070550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:11.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.037+0000 7f54b9600640 1 -- 172.21.15.96:0/566812892 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 5027+0+0 (secure 0 0 0) 0x7f54b008a850 con 0x7f54bc06f860 2024-05-28T06:22:11.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.037+0000 7f54c0c00640 1 --2- 172.21.15.96:0/566812892 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f549c06e090 0x7f549c070550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:11.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.038+0000 7f54c0c00640 1 --2- 172.21.15.96:0/566812892 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f549c06e090 0x7f549c070550 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f54a8006fd0 tx=0x7f54a8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:11.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.041+0000 7f54b9600640 1 -- 172.21.15.96:0/566812892 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f54bc0d0e10 con 0x7f54bc06f860 2024-05-28T06:22:11.273 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.277+0000 7f54c4000640 1 -- 172.21.15.96:0/566812892 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f54bc06e2f0 con 0x7f54bc06f860 2024-05-28T06:22:11.274 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.279+0000 7f54b9600640 1 -- 172.21.15.96:0/566812892 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) ==== 94+0+3872 (secure 0 0 0) 0x7f54b005a6e0 con 0x7f54bc06f860 2024-05-28T06:22:11.274 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:22:11.275 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":65,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5861,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/1001966057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6833","nonce":1001966057},{"type":"v1","addr":"172.21.15.96:6835","nonce":1001966057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":57},{"gid":5864,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6836/4285246962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":4285246962},{"type":"v1","addr":"172.21.15.123:6836","nonce":4285246962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":57},{"gid":6445,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6834/3302807355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":3302807355},{"type":"v1","addr":"172.21.15.96:6834","nonce":3302807355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":6670,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/864180437","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6835","nonce":864180437},{"type":"v1","addr":"172.21.15.123:6837","nonce":864180437}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":65}],"filesystems":[]} 2024-05-28T06:22:11.275 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 65 2024-05-28T06:22:11.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.282+0000 7f54c4000640 1 -- 172.21.15.96:0/566812892 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f549c06e090 msgr2=0x7f549c070550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:11.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.282+0000 7f54c4000640 1 --2- 172.21.15.96:0/566812892 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f549c06e090 0x7f549c070550 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f54a8006fd0 tx=0x7f54a8008040 comp rx=0 tx=0).stop 2024-05-28T06:22:11.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.282+0000 7f54c4000640 1 -- 172.21.15.96:0/566812892 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f54bc06f860 msgr2=0x7f54bc0c4aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:11.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.282+0000 7f54c4000640 1 --2- 172.21.15.96:0/566812892 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f54bc06f860 0x7f54bc0c4aa0 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7f54b000c430 tx=0x7f54b000c8f0 comp rx=0 tx=0).stop 2024-05-28T06:22:11.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.282+0000 7f54c4000640 1 -- 172.21.15.96:0/566812892 shutdown_connections 2024-05-28T06:22:11.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.282+0000 7f54c4000640 1 --2- 172.21.15.96:0/566812892 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f549c06e090 0x7f549c070550 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:11.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.282+0000 7f54c4000640 1 --2- 172.21.15.96:0/566812892 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f54bc0d04a0 0x7f54bc0c8e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:11.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.282+0000 7f54c4000640 1 --2- 172.21.15.96:0/566812892 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f54bc0ad130 0x7f54bc0c88d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:11.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.282+0000 7f54c4000640 1 --2- 172.21.15.96:0/566812892 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f54bc06f860 0x7f54bc0c4aa0 unknown :-1 s=CLOSED pgs=1768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:11.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.282+0000 7f54c4000640 1 -- 172.21.15.96:0/566812892 >> 172.21.15.96:0/566812892 conn(0x7f54bc06bae0 msgr2=0x7f54bc073610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:11.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.283+0000 7f54c4000640 1 -- 172.21.15.96:0/566812892 shutdown_connections 2024-05-28T06:22:11.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.283+0000 7f54c4000640 1 -- 172.21.15.96:0/566812892 wait complete. 2024-05-28T06:22:11.298 INFO:tasks.ceph.mds.a:Restarting daemon 2024-05-28T06:22:11.298 INFO:tasks.ceph.mds.a:Stopping old one... 2024-05-28T06:22:11.298 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-05-28T06:22:11.299 INFO:teuthology.orchestra.run:waiting for 300 2024-05-28T06:22:11.299 INFO:tasks.ceph.mds.c:Restarting daemon 2024-05-28T06:22:11.299 INFO:tasks.ceph.mds.c:Stopping old one... 2024-05-28T06:22:11.299 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-05-28T06:22:11.299 INFO:teuthology.orchestra.run:waiting for 300 2024-05-28T06:22:11.299 INFO:tasks.ceph.mds.b:Restarting daemon 2024-05-28T06:22:11.299 INFO:tasks.ceph.mds.b:Stopping old one... 2024-05-28T06:22:11.300 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-05-28T06:22:11.300 INFO:teuthology.orchestra.run:waiting for 300 2024-05-28T06:22:11.300 INFO:tasks.ceph.mds.d:Restarting daemon 2024-05-28T06:22:11.300 INFO:tasks.ceph.mds.d:Stopping old one... 2024-05-28T06:22:11.300 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-05-28T06:22:11.300 INFO:teuthology.orchestra.run:waiting for 300 2024-05-28T06:22:11.316 INFO:tasks.ceph.mds.a:Stopped 2024-05-28T06:22:11.316 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-05-28T06:22:11.317 INFO:tasks.ceph.mds.c:Stopped 2024-05-28T06:22:11.317 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-05-28T06:22:11.322 INFO:tasks.ceph.mds.b:Stopped 2024-05-28T06:22:11.322 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-05-28T06:22:11.323 INFO:tasks.ceph.mds.d:Stopped 2024-05-28T06:22:11.323 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-05-28T06:22:11.362 INFO:tasks.ceph.mds.a:Started 2024-05-28T06:22:11.363 DEBUG:teuthology.parallel:result is None 2024-05-28T06:22:11.366 INFO:tasks.ceph.mds.c:Started 2024-05-28T06:22:11.366 DEBUG:teuthology.parallel:result is None 2024-05-28T06:22:11.370 INFO:tasks.ceph.mds.b:Started 2024-05-28T06:22:11.370 DEBUG:teuthology.parallel:result is None 2024-05-28T06:22:11.375 INFO:tasks.ceph.mds.d:Started 2024-05-28T06:22:11.375 DEBUG:teuthology.parallel:result is None 2024-05-28T06:22:11.376 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-05-28T06:22:11.506 INFO:tasks.ceph.mds.c.smithi096.stdout:starting mds.c at 2024-05-28T06:22:11.506 INFO:tasks.ceph.mds.a.smithi096.stdout:starting mds.a at 2024-05-28T06:22:11.522 INFO:tasks.ceph.mds.b.smithi123.stdout:starting mds.b at 2024-05-28T06:22:11.523 INFO:tasks.ceph.mds.d.smithi123.stdout:starting mds.d at 2024-05-28T06:22:11.549 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.554+0000 7f3293c00640 1 Processor -- start 2024-05-28T06:22:11.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.554+0000 7f3293c00640 1 -- start start 2024-05-28T06:22:11.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.555+0000 7f3293c00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f328c0d2db0 0x7f328c0b3b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:11.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.555+0000 7f3293c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f328c0d4d60 0x7f328c0d3190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:11.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.555+0000 7f3293c00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f328c0cf490 0x7f328c0cf920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:11.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.555+0000 7f3293c00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f328c0d5540 con 0x7f328c0cf490 2024-05-28T06:22:11.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.555+0000 7f3293c00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f328c0d36d0 con 0x7f328c0d2db0 2024-05-28T06:22:11.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.555+0000 7f3293c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f328c163500 con 0x7f328c0d4d60 2024-05-28T06:22:11.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.556+0000 7f3291c00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f328c0cf490 0x7f328c0cf920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:11.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.556+0000 7f3291200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f328c0d2db0 0x7f328c0b3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:11.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.556+0000 7f3291c00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f328c0cf490 0x7f328c0cf920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:59852/0 (socket says 172.21.15.96:59852) 2024-05-28T06:22:11.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.556+0000 7f3291c00640 1 -- 172.21.15.96:0/1396157664 learned_addr learned my addr 172.21.15.96:0/1396157664 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:22:11.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.556+0000 7f328be00640 1 --2- 172.21.15.96:0/1396157664 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f328c0d4d60 0x7f328c0d3190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:11.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.556+0000 7f3291c00640 1 -- 172.21.15.96:0/1396157664 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f328c0d4d60 msgr2=0x7f328c0d3190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:11.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.556+0000 7f3291c00640 1 --2- 172.21.15.96:0/1396157664 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f328c0d4d60 0x7f328c0d3190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:11.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.556+0000 7f3291c00640 1 -- 172.21.15.96:0/1396157664 >> v2:172.21.15.123:3300/0 conn(0x7f328c0d2db0 msgr2=0x7f328c0b3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:11.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.556+0000 7f3291c00640 1 --2- 172.21.15.96:0/1396157664 >> v2:172.21.15.123:3300/0 conn(0x7f328c0d2db0 0x7f328c0b3b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:11.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.556+0000 7f3291c00640 1 -- 172.21.15.96:0/1396157664 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f328c162740 con 0x7f328c0cf490 2024-05-28T06:22:11.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.556+0000 7f3291200640 1 --2- 172.21.15.96:0/1396157664 >> v2:172.21.15.123:3300/0 conn(0x7f328c0d2db0 0x7f328c0b3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-28T06:22:11.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.557+0000 7f3291c00640 1 --2- 172.21.15.96:0/1396157664 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f328c0cf490 0x7f328c0cf920 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7f328c075740 tx=0x7f328402a600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aea637668a683777 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:11.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.557+0000 7f328b400640 1 -- 172.21.15.96:0/1396157664 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3284036070 con 0x7f328c0cf490 2024-05-28T06:22:11.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.557+0000 7f328b400640 1 -- 172.21.15.96:0/1396157664 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3284031600 con 0x7f328c0cf490 2024-05-28T06:22:11.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.557+0000 7f328b400640 1 -- 172.21.15.96:0/1396157664 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32840319d0 con 0x7f328c0cf490 2024-05-28T06:22:11.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.558+0000 7f3293c00640 1 -- 172.21.15.96:0/1396157664 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f328c0cf490 msgr2=0x7f328c0cf920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:11.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.558+0000 7f3293c00640 1 --2- 172.21.15.96:0/1396157664 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f328c0cf490 0x7f328c0cf920 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7f328c075740 tx=0x7f328402a600 comp rx=0 tx=0).stop 2024-05-28T06:22:11.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.558+0000 7f3293c00640 1 -- 172.21.15.96:0/1396157664 shutdown_connections 2024-05-28T06:22:11.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.558+0000 7f3293c00640 1 --2- 172.21.15.96:0/1396157664 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f328c0cf490 0x7f328c0cf920 unknown :-1 s=CLOSED pgs=1777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:11.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.558+0000 7f3293c00640 1 --2- 172.21.15.96:0/1396157664 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f328c0d4d60 0x7f328c0d3190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:11.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.558+0000 7f3293c00640 1 --2- 172.21.15.96:0/1396157664 >> v2:172.21.15.123:3300/0 conn(0x7f328c0d2db0 0x7f328c0b3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:11.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.558+0000 7f3293c00640 1 -- 172.21.15.96:0/1396157664 >> 172.21.15.96:0/1396157664 conn(0x7f328c08bfc0 msgr2=0x7f328c0d4650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:11.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.558+0000 7f3293c00640 1 -- 172.21.15.96:0/1396157664 shutdown_connections 2024-05-28T06:22:11.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.558+0000 7f3293c00640 1 -- 172.21.15.96:0/1396157664 wait complete. 2024-05-28T06:22:11.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.560+0000 7f3293c00640 1 Processor -- start 2024-05-28T06:22:11.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.560+0000 7f3293c00640 1 -- start start 2024-05-28T06:22:11.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.561+0000 7f3293c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f328c0cf490 0x7f328c1f7d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:11.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.561+0000 7f3293c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f328c0d2db0 0x7f328c0c8180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:11.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.561+0000 7f3293c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f328c0d4d60 0x7f328c0cd700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:11.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.561+0000 7f3293c00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f328c162fe0 con 0x7f328c0d4d60 2024-05-28T06:22:11.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.561+0000 7f3293c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f328c162d00 con 0x7f328c0d2db0 2024-05-28T06:22:11.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.561+0000 7f3293c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f328c0d51a0 con 0x7f328c0cf490 2024-05-28T06:22:11.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.561+0000 7f3291c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f328c0d4d60 0x7f328c0cd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:11.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.561+0000 7f3291c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f328c0d4d60 0x7f328c0cd700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:59854/0 (socket says 172.21.15.96:59854) 2024-05-28T06:22:11.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.561+0000 7f3291c00640 1 -- 172.21.15.96:0/2593526976 learned_addr learned my addr 172.21.15.96:0/2593526976 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:22:11.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.561+0000 7f3291200640 1 --2- 172.21.15.96:0/2593526976 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f328c0cf490 0x7f328c1f7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:11.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.561+0000 7f328be00640 1 --2- 172.21.15.96:0/2593526976 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f328c0d2db0 0x7f328c0c8180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:11.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.562+0000 7f3291c00640 1 -- 172.21.15.96:0/2593526976 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f328c0cf490 msgr2=0x7f328c1f7d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:11.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.562+0000 7f3291c00640 1 --2- 172.21.15.96:0/2593526976 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f328c0cf490 0x7f328c1f7d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:11.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.562+0000 7f3291c00640 1 -- 172.21.15.96:0/2593526976 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f328c0d2db0 msgr2=0x7f328c0c8180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:11.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.562+0000 7f3291c00640 1 --2- 172.21.15.96:0/2593526976 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f328c0d2db0 0x7f328c0c8180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:11.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.562+0000 7f3291c00640 1 -- 172.21.15.96:0/2593526976 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f328c0ca920 con 0x7f328c0d4d60 2024-05-28T06:22:11.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.562+0000 7f3291200640 1 --2- 172.21.15.96:0/2593526976 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f328c0cf490 0x7f328c1f7d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-28T06:22:11.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.562+0000 7f3291c00640 1 --2- 172.21.15.96:0/2593526976 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f328c0d4d60 0x7f328c0cd700 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7f3284009f40 tx=0x7f3284004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:11.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.562+0000 7f3288c00640 1 -- 172.21.15.96:0/2593526976 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3284041020 con 0x7f328c0d4d60 2024-05-28T06:22:11.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.562+0000 7f3288c00640 1 -- 172.21.15.96:0/2593526976 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32840077a0 con 0x7f328c0d4d60 2024-05-28T06:22:11.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.562+0000 7f3288c00640 1 -- 172.21.15.96:0/2593526976 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3284036040 con 0x7f328c0d4d60 2024-05-28T06:22:11.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.562+0000 7f3293c00640 1 -- 172.21.15.96:0/2593526976 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f328c0cac00 con 0x7f328c0d4d60 2024-05-28T06:22:11.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.563+0000 7f3293c00640 1 -- 172.21.15.96:0/2593526976 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f328c0cb720 con 0x7f328c0d4d60 2024-05-28T06:22:11.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.564+0000 7f3293c00640 1 -- 172.21.15.96:0/2593526976 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f328c0689e0 con 0x7f328c0d4d60 2024-05-28T06:22:11.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.565+0000 7f3288c00640 1 -- 172.21.15.96:0/2593526976 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 37) ==== 93182+0+0 (secure 0 0 0) 0x7f3284044020 con 0x7f328c0d4d60 2024-05-28T06:22:11.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.565+0000 7f3288c00640 1 --2- 172.21.15.96:0/2593526976 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f326c06e0c0 0x7f326c070580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:11.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.566+0000 7f3288c00640 1 -- 172.21.15.96:0/2593526976 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 5027+0+0 (secure 0 0 0) 0x7f32840ab7a0 con 0x7f328c0d4d60 2024-05-28T06:22:11.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.566+0000 7f3291200640 1 --2- 172.21.15.96:0/2593526976 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f326c06e0c0 0x7f326c070580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:11.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.567+0000 7f3291200640 1 --2- 172.21.15.96:0/2593526976 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f326c06e0c0 0x7f326c070580 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f3280009b20 tx=0x7f3280008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:11.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.570+0000 7f3288c00640 1 -- 172.21.15.96:0/2593526976 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f328404b020 con 0x7f328c0d4d60 2024-05-28T06:22:11.774 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.778+0000 7f3293c00640 1 -- 172.21.15.96:0/2593526976 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f328c0c7ea0 con 0x7f328c0d4d60 2024-05-28T06:22:11.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.931+0000 7f3288c00640 1 -- 172.21.15.96:0/2593526976 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v109) ==== 97+0+0 (secure 0 0 0) 0x7f328402ae20 con 0x7f328c0d4d60 2024-05-28T06:22:11.928 INFO:teuthology.orchestra.run.smithi096.stderr: removed all blocklist entries 2024-05-28T06:22:11.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.934+0000 7f3293c00640 1 -- 172.21.15.96:0/2593526976 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f326c06e0c0 msgr2=0x7f326c070580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:11.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.934+0000 7f3293c00640 1 --2- 172.21.15.96:0/2593526976 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f326c06e0c0 0x7f326c070580 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f3280009b20 tx=0x7f3280008040 comp rx=0 tx=0).stop 2024-05-28T06:22:11.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.934+0000 7f3293c00640 1 -- 172.21.15.96:0/2593526976 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f328c0d4d60 msgr2=0x7f328c0cd700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:11.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.934+0000 7f3293c00640 1 --2- 172.21.15.96:0/2593526976 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f328c0d4d60 0x7f328c0cd700 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7f3284009f40 tx=0x7f3284004a40 comp rx=0 tx=0).stop 2024-05-28T06:22:11.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.935+0000 7f3293c00640 1 -- 172.21.15.96:0/2593526976 shutdown_connections 2024-05-28T06:22:11.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.935+0000 7f3293c00640 1 --2- 172.21.15.96:0/2593526976 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f326c06e0c0 0x7f326c070580 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:11.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.935+0000 7f3293c00640 1 --2- 172.21.15.96:0/2593526976 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f328c0d4d60 0x7f328c0cd700 unknown :-1 s=CLOSED pgs=1778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:11.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.935+0000 7f3293c00640 1 --2- 172.21.15.96:0/2593526976 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f328c0d2db0 0x7f328c0c8180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:11.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.935+0000 7f3293c00640 1 --2- 172.21.15.96:0/2593526976 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f328c0cf490 0x7f328c1f7d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:11.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.935+0000 7f3293c00640 1 -- 172.21.15.96:0/2593526976 >> 172.21.15.96:0/2593526976 conn(0x7f328c08bfc0 msgr2=0x7f328c0d0e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:11.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.935+0000 7f3293c00640 1 -- 172.21.15.96:0/2593526976 shutdown_connections 2024-05-28T06:22:11.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:11.935+0000 7f3293c00640 1 -- 172.21.15.96:0/2593526976 wait complete. 2024-05-28T06:22:11.951 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-05-28T06:22:12.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.143+0000 7f0bb7c00640 1 Processor -- start 2024-05-28T06:22:12.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.143+0000 7f0bb7c00640 1 -- start start 2024-05-28T06:22:12.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.144+0000 7f0bb7c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0bb006f860 0x7f0bb00b3150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:12.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.144+0000 7f0bb7c00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f0bb00b3690 0x7f0bb00d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:12.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.144+0000 7f0bb7c00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f0bb00d09e0 0x7f0bb0160210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:12.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.144+0000 7f0bb7c00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f0bb00d0e50 con 0x7f0bb00b3690 2024-05-28T06:22:12.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.144+0000 7f0bb7c00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f0bb006ed10 con 0x7f0bb00d09e0 2024-05-28T06:22:12.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.144+0000 7f0bb7c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f0bb015cb00 con 0x7f0bb006f860 2024-05-28T06:22:12.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.145+0000 7f0bafe00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f0bb00b3690 0x7f0bb00d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:12.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.145+0000 7f0bafe00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f0bb00b3690 0x7f0bb00d04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:59864/0 (socket says 172.21.15.96:59864) 2024-05-28T06:22:12.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.145+0000 7f0bafe00640 1 -- 172.21.15.96:0/2406624611 learned_addr learned my addr 172.21.15.96:0/2406624611 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:22:12.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.145+0000 7f0bb5200640 1 --2- 172.21.15.96:0/2406624611 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0bb006f860 0x7f0bb00b3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:12.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.145+0000 7f0bb5c00640 1 --2- 172.21.15.96:0/2406624611 >> v2:172.21.15.123:3300/0 conn(0x7f0bb00d09e0 0x7f0bb0160210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:12.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.145+0000 7f0bafe00640 1 -- 172.21.15.96:0/2406624611 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0bb006f860 msgr2=0x7f0bb00b3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:12.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.145+0000 7f0bafe00640 1 --2- 172.21.15.96:0/2406624611 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0bb006f860 0x7f0bb00b3150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:12.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.145+0000 7f0bafe00640 1 -- 172.21.15.96:0/2406624611 >> v2:172.21.15.123:3300/0 conn(0x7f0bb00d09e0 msgr2=0x7f0bb0160210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:12.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.145+0000 7f0bafe00640 1 --2- 172.21.15.96:0/2406624611 >> v2:172.21.15.123:3300/0 conn(0x7f0bb00d09e0 0x7f0bb0160210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:12.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.145+0000 7f0bafe00640 1 -- 172.21.15.96:0/2406624611 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bb00b3cc0 con 0x7f0bb00b3690 2024-05-28T06:22:12.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.146+0000 7f0bafe00640 1 --2- 172.21.15.96:0/2406624611 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0bb00b3690 0x7f0bb00d04a0 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7f0bb006c900 tx=0x7f0b9c02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f763362fc76c09c5 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:12.143 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.147+0000 7f0baf400640 1 -- 172.21.15.96:0/2406624611 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b9c037070 con 0x7f0bb00b3690 2024-05-28T06:22:12.143 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.147+0000 7f0baf400640 1 -- 172.21.15.96:0/2406624611 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b9c032440 con 0x7f0bb00b3690 2024-05-28T06:22:12.143 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.147+0000 7f0baf400640 1 -- 172.21.15.96:0/2406624611 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b9c032810 con 0x7f0bb00b3690 2024-05-28T06:22:12.143 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.147+0000 7f0bb7c00640 1 -- 172.21.15.96:0/2406624611 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0bb00b3690 msgr2=0x7f0bb00d04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:12.143 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.147+0000 7f0bb7c00640 1 --2- 172.21.15.96:0/2406624611 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0bb00b3690 0x7f0bb00d04a0 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7f0bb006c900 tx=0x7f0b9c02a1e0 comp rx=0 tx=0).stop 2024-05-28T06:22:12.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.147+0000 7f0bb7c00640 1 -- 172.21.15.96:0/2406624611 shutdown_connections 2024-05-28T06:22:12.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.147+0000 7f0bb7c00640 1 --2- 172.21.15.96:0/2406624611 >> v2:172.21.15.123:3300/0 conn(0x7f0bb00d09e0 0x7f0bb0160210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:12.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.147+0000 7f0bb7c00640 1 --2- 172.21.15.96:0/2406624611 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0bb00b3690 0x7f0bb00d04a0 unknown :-1 s=CLOSED pgs=1779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:12.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.147+0000 7f0bb7c00640 1 --2- 172.21.15.96:0/2406624611 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0bb006f860 0x7f0bb00b3150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:12.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.147+0000 7f0bb7c00640 1 -- 172.21.15.96:0/2406624611 >> 172.21.15.96:0/2406624611 conn(0x7f0bb006bae0 msgr2=0x7f0bb006bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:12.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.147+0000 7f0bb7c00640 1 -- 172.21.15.96:0/2406624611 shutdown_connections 2024-05-28T06:22:12.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.147+0000 7f0bb7c00640 1 -- 172.21.15.96:0/2406624611 wait complete. 2024-05-28T06:22:12.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.149+0000 7f0bb7c00640 1 Processor -- start 2024-05-28T06:22:12.145 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.149+0000 7f0bb7c00640 1 -- start start 2024-05-28T06:22:12.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.150+0000 7f0bb7c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0bb006f860 0x7f0bb01f4050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:12.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.150+0000 7f0bb7c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0bb00b3690 0x7f0bb00cde30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:12.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.150+0000 7f0bb7c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0bb00d09e0 0x7f0bb00ce370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:12.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.150+0000 7f0bb7c00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f0bb00b4560 con 0x7f0bb006f860 2024-05-28T06:22:12.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.150+0000 7f0bb7c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f0bb00b4280 con 0x7f0bb00d09e0 2024-05-28T06:22:12.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.150+0000 7f0bb7c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f0bb00b4840 con 0x7f0bb00b3690 2024-05-28T06:22:12.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.150+0000 7f0bb5200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0bb006f860 0x7f0bb01f4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:12.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.151+0000 7f0bb5200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0bb006f860 0x7f0bb01f4050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:59880/0 (socket says 172.21.15.96:59880) 2024-05-28T06:22:12.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.151+0000 7f0bb5200640 1 -- 172.21.15.96:0/2283890903 learned_addr learned my addr 172.21.15.96:0/2283890903 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:22:12.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.151+0000 7f0bafe00640 1 --2- 172.21.15.96:0/2283890903 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0bb00b3690 0x7f0bb00cde30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:12.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.151+0000 7f0bb5c00640 1 --2- 172.21.15.96:0/2283890903 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0bb00d09e0 0x7f0bb00ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:12.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.151+0000 7f0bb5200640 1 -- 172.21.15.96:0/2283890903 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0bb00b3690 msgr2=0x7f0bb00cde30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:12.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.151+0000 7f0bb5200640 1 --2- 172.21.15.96:0/2283890903 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0bb00b3690 0x7f0bb00cde30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:12.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.151+0000 7f0bb5200640 1 -- 172.21.15.96:0/2283890903 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0bb00d09e0 msgr2=0x7f0bb00ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:12.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.151+0000 7f0bb5200640 1 --2- 172.21.15.96:0/2283890903 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0bb00d09e0 0x7f0bb00ce370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:12.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.151+0000 7f0bb5200640 1 -- 172.21.15.96:0/2283890903 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bb015d1f0 con 0x7f0bb006f860 2024-05-28T06:22:12.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.151+0000 7f0bb5200640 1 --2- 172.21.15.96:0/2283890903 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0bb006f860 0x7f0bb01f4050 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7f0ba400c820 tx=0x7f0ba400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:12.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.151+0000 7f0bacc00640 1 -- 172.21.15.96:0/2283890903 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ba4015070 con 0x7f0bb006f860 2024-05-28T06:22:12.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.152+0000 7f0bacc00640 1 -- 172.21.15.96:0/2283890903 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ba4004240 con 0x7f0bb006f860 2024-05-28T06:22:12.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.152+0000 7f0bb7c00640 1 -- 172.21.15.96:0/2283890903 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0bb015d4d0 con 0x7f0bb006f860 2024-05-28T06:22:12.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.152+0000 7f0bacc00640 1 -- 172.21.15.96:0/2283890903 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ba400ce10 con 0x7f0bb006f860 2024-05-28T06:22:12.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.152+0000 7f0bb7c00640 1 -- 172.21.15.96:0/2283890903 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0bb015da90 con 0x7f0bb006f860 2024-05-28T06:22:12.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.153+0000 7f0bb7c00640 1 -- 172.21.15.96:0/2283890903 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0bb00689e0 con 0x7f0bb006f860 2024-05-28T06:22:12.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.154+0000 7f0bacc00640 1 -- 172.21.15.96:0/2283890903 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 37) ==== 93182+0+0 (secure 0 0 0) 0x7f0ba4007830 con 0x7f0bb006f860 2024-05-28T06:22:12.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.155+0000 7f0bacc00640 1 --2- 172.21.15.96:0/2283890903 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0b9006e090 0x7f0b90070550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:12.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.155+0000 7f0bacc00640 1 -- 172.21.15.96:0/2283890903 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 4913+0+0 (secure 0 0 0) 0x7f0ba408a7d0 con 0x7f0bb006f860 2024-05-28T06:22:12.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.155+0000 7f0bafe00640 1 --2- 172.21.15.96:0/2283890903 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0b9006e090 0x7f0b90070550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:12.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.156+0000 7f0bafe00640 1 --2- 172.21.15.96:0/2283890903 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0b9006e090 0x7f0b90070550 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f0b9c034000 tx=0x7f0b9c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:12.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.159+0000 7f0bacc00640 1 -- 172.21.15.96:0/2283890903 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0ba405a700 con 0x7f0bb006f860 2024-05-28T06:22:12.410 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.414+0000 7f0bb7c00640 1 -- 172.21.15.96:0/2283890903 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f0bb006e2f0 con 0x7f0bb006f860 2024-05-28T06:22:12.410 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.415+0000 7f0bacc00640 1 -- 172.21.15.96:0/2283890903 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v21) ==== 80+0+5304 (secure 0 0 0) 0x7f0ba405a000 con 0x7f0bb006f860 2024-05-28T06:22:12.411 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:22:12.411 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:22:12.411 INFO:teuthology.orchestra.run.smithi096.stdout: "auth_dump": [ 2024-05-28T06:22:12.411 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:22:12.411 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "mds.a", 2024-05-28T06:22:12.411 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCWc1VmyA9HHBAANJCF8I/RP5ZYviyOzo0JCQ==", 2024-05-28T06:22:12.411 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:22:12.411 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow", 2024-05-28T06:22:12.411 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow *", 2024-05-28T06:22:12.412 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow *", 2024-05-28T06:22:12.412 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:22:12.412 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:22:12.412 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:22:12.412 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:22:12.412 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "mds.b", 2024-05-28T06:22:12.412 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCWc1VmtR0EOBAA296qGHTmB/2sMztDbDJ56g==", 2024-05-28T06:22:12.412 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:22:12.412 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow", 2024-05-28T06:22:12.413 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow *", 2024-05-28T06:22:12.413 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow *", 2024-05-28T06:22:12.413 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:22:12.413 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:22:12.413 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:22:12.413 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:22:12.413 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "mds.c", 2024-05-28T06:22:12.413 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCWc1VmrLxHKhAAhkTRLDKL6RUJQGetB8JQQA==", 2024-05-28T06:22:12.413 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:22:12.413 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow", 2024-05-28T06:22:12.413 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow *", 2024-05-28T06:22:12.413 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow *", 2024-05-28T06:22:12.414 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:22:12.414 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:22:12.414 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:22:12.414 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:22:12.414 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "mds.d", 2024-05-28T06:22:12.414 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCXc1VmW2aQCxAAnkDwQzlgy39vFe2O9Wwf7Q==", 2024-05-28T06:22:12.414 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:22:12.414 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow", 2024-05-28T06:22:12.414 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow *", 2024-05-28T06:22:12.414 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow *", 2024-05-28T06:22:12.414 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:22:12.414 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:22:12.414 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:22:12.415 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:22:12.415 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "osd.0", 2024-05-28T06:22:12.415 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCbc1VmEcNAAxAAiB5AsfY2uRY1kc6h80chuw==", 2024-05-28T06:22:12.415 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:22:12.415 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow profile osd", 2024-05-28T06:22:12.415 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile osd", 2024-05-28T06:22:12.415 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:22:12.415 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:22:12.415 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:22:12.415 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:22:12.415 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "osd.1", 2024-05-28T06:22:12.415 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCec1Vm6CAaDhAA1NkYM2q4fTsgSDSwJZL3Qg==", 2024-05-28T06:22:12.415 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:22:12.415 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow profile osd", 2024-05-28T06:22:12.416 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile osd", 2024-05-28T06:22:12.416 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:22:12.416 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:22:12.416 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:22:12.416 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:22:12.416 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "osd.2", 2024-05-28T06:22:12.416 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQChc1VmKn1jFxAA+dM4gM7FR8NwBpyd833A/Q==", 2024-05-28T06:22:12.416 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:22:12.416 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow profile osd", 2024-05-28T06:22:12.416 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile osd", 2024-05-28T06:22:12.416 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:22:12.416 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:22:12.416 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:22:12.417 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:22:12.417 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "osd.3", 2024-05-28T06:22:12.417 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCkc1VmV07MIRAAHIXQGDVbL9V5QqexQ89IJA==", 2024-05-28T06:22:12.417 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:22:12.417 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow profile osd", 2024-05-28T06:22:12.417 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile osd", 2024-05-28T06:22:12.417 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:22:12.417 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:22:12.417 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:22:12.417 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:22:12.417 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "osd.4", 2024-05-28T06:22:12.417 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCqc1VmobuZMxAAlY8xhuzqlkuFrcbTouF5rA==", 2024-05-28T06:22:12.417 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:22:12.417 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow profile osd", 2024-05-28T06:22:12.418 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile osd", 2024-05-28T06:22:12.418 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:22:12.418 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:22:12.418 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:22:12.418 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:22:12.418 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "osd.5", 2024-05-28T06:22:12.418 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCuc1Vmwt3QAxAAUFj/e/wJvxljgAbcTb32Ug==", 2024-05-28T06:22:12.418 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:22:12.418 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow profile osd", 2024-05-28T06:22:12.418 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile osd", 2024-05-28T06:22:12.418 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:22:12.418 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:22:12.418 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:22:12.419 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:22:12.419 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "osd.6", 2024-05-28T06:22:12.419 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCxc1VmscerDxAAqEDAqy6/OvdfSJ+ALZ7WeQ==", 2024-05-28T06:22:12.419 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:22:12.419 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow profile osd", 2024-05-28T06:22:12.419 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile osd", 2024-05-28T06:22:12.419 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:22:12.419 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:22:12.419 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:22:12.419 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:22:12.419 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "osd.7", 2024-05-28T06:22:12.419 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQC0c1VmgxkbKhAA7dFzDTBKFmKiFFxPsyiT4Q==", 2024-05-28T06:22:12.419 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:22:12.420 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow profile osd", 2024-05-28T06:22:12.420 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile osd", 2024-05-28T06:22:12.420 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:22:12.420 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:22:12.420 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:22:12.420 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:22:12.420 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "client.0", 2024-05-28T06:22:12.420 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCXc1Vm91hbGBAASTL3HQR8XOs7wVQEHbPl/g==", 2024-05-28T06:22:12.420 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:22:12.420 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow", 2024-05-28T06:22:12.420 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow r", 2024-05-28T06:22:12.420 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-05-28T06:22:12.420 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:22:12.420 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:22:12.421 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:22:12.421 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "client.1", 2024-05-28T06:22:12.421 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCXc1VmIc77KhAAw2e0alTmVROhyCIFW4H3Sg==", 2024-05-28T06:22:12.421 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:22:12.421 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow", 2024-05-28T06:22:12.421 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow r", 2024-05-28T06:22:12.421 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow rw", 2024-05-28T06:22:12.421 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow rwx" 2024-05-28T06:22:12.421 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:22:12.421 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:22:12.421 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:22:12.421 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "client.2", 2024-05-28T06:22:12.421 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCXc1VmCS+OIRAAMV3Au6KuchLhA59QTGbv1w==", 2024-05-28T06:22:12.421 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:22:12.422 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow", 2024-05-28T06:22:12.422 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow r", 2024-05-28T06:22:12.422 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow rw", 2024-05-28T06:22:12.422 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow rwx" 2024-05-28T06:22:12.422 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:22:12.422 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:22:12.422 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:22:12.422 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "client.3", 2024-05-28T06:22:12.423 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCXc1Vm5r3pNBAAkVZ3arjywHQkEP5lmS8xqg==", 2024-05-28T06:22:12.423 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:22:12.423 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow", 2024-05-28T06:22:12.423 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow r", 2024-05-28T06:22:12.423 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow rw", 2024-05-28T06:22:12.423 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow rwx" 2024-05-28T06:22:12.425 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:22:12.425 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:22:12.425 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:22:12.425 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "client.admin", 2024-05-28T06:22:12.425 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCVc1Vm5QaXFhAAQFGKgbZ/kpGisNm2lIw0Uw==", 2024-05-28T06:22:12.426 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:22:12.426 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow *", 2024-05-28T06:22:12.426 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow *", 2024-05-28T06:22:12.426 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow *", 2024-05-28T06:22:12.426 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:22:12.426 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:22:12.426 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:22:12.426 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:22:12.426 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "mgr.x", 2024-05-28T06:22:12.426 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCWc1VmVMx/EhAAylvi37aq3hbMOhUSQWLVSg==", 2024-05-28T06:22:12.426 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:22:12.426 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow *", 2024-05-28T06:22:12.426 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile mgr", 2024-05-28T06:22:12.427 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:22:12.427 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:22:12.427 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:22:12.427 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:22:12.427 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "mgr.y", 2024-05-28T06:22:12.427 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCWc1Vmh5r7BxAAA74gn+FozehM+34eFik8kw==", 2024-05-28T06:22:12.427 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:22:12.427 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow *", 2024-05-28T06:22:12.427 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile mgr", 2024-05-28T06:22:12.427 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:22:12.427 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:22:12.427 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:22:12.427 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-05-28T06:22:12.428 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:22:12.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.417+0000 7f0bb7c00640 1 -- 172.21.15.96:0/2283890903 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0b9006e090 msgr2=0x7f0b90070550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:12.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.417+0000 7f0bb7c00640 1 --2- 172.21.15.96:0/2283890903 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0b9006e090 0x7f0b90070550 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f0b9c034000 tx=0x7f0b9c02e000 comp rx=0 tx=0).stop 2024-05-28T06:22:12.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.417+0000 7f0bb7c00640 1 -- 172.21.15.96:0/2283890903 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0bb006f860 msgr2=0x7f0bb01f4050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:12.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.417+0000 7f0bb7c00640 1 --2- 172.21.15.96:0/2283890903 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0bb006f860 0x7f0bb01f4050 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7f0ba400c820 tx=0x7f0ba400cce0 comp rx=0 tx=0).stop 2024-05-28T06:22:12.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.418+0000 7f0bb7c00640 1 -- 172.21.15.96:0/2283890903 shutdown_connections 2024-05-28T06:22:12.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.418+0000 7f0bb7c00640 1 --2- 172.21.15.96:0/2283890903 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0b9006e090 0x7f0b90070550 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:12.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.418+0000 7f0bb7c00640 1 --2- 172.21.15.96:0/2283890903 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0bb00d09e0 0x7f0bb00ce370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:12.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.418+0000 7f0bb7c00640 1 --2- 172.21.15.96:0/2283890903 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0bb00b3690 0x7f0bb00cde30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:12.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.418+0000 7f0bb7c00640 1 --2- 172.21.15.96:0/2283890903 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0bb006f860 0x7f0bb01f4050 unknown :-1 s=CLOSED pgs=1780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:12.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.418+0000 7f0bb7c00640 1 -- 172.21.15.96:0/2283890903 >> 172.21.15.96:0/2283890903 conn(0x7f0bb006bae0 msgr2=0x7f0bb00adfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:12.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.418+0000 7f0bb7c00640 1 -- 172.21.15.96:0/2283890903 shutdown_connections 2024-05-28T06:22:12.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.418+0000 7f0bb7c00640 1 -- 172.21.15.96:0/2283890903 wait complete. 2024-05-28T06:22:12.431 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:22:12.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.626+0000 7fca8e600640 1 Processor -- start 2024-05-28T06:22:12.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.626+0000 7fca8e600640 1 -- start start 2024-05-28T06:22:12.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.628+0000 7fca8e600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fca8815e870 0x7fca880d0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:12.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.628+0000 7fca8e600640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca8815ec50 con 0x7fca880cfa40 2024-05-28T06:22:12.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.628+0000 7fca8e600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca880cff80 con 0x7fca880d14d0 2024-05-28T06:22:12.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.628+0000 7fca8e600640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fca880d2150 con 0x7fca8815e870 2024-05-28T06:22:12.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.628+0000 7fca8d200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fca8815e870 0x7fca880d0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:12.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.628+0000 7fca8dc00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fca880cfa40 0x7fca880d25e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:33128/0 (socket says 172.21.15.96:33128) 2024-05-28T06:22:12.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.628+0000 7fca8dc00640 1 -- 172.21.15.96:0/2550937880 learned_addr learned my addr 172.21.15.96:0/2550937880 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:22:12.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.629+0000 7fca8d200640 1 -- 172.21.15.96:0/2550937880 >> v1:172.21.15.123:6789/0 conn(0x7fca880d14d0 legacy=0x7fca880cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:12.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.629+0000 7fca8d200640 1 -- 172.21.15.96:0/2550937880 >> v1:172.21.15.96:6789/0 conn(0x7fca880cfa40 legacy=0x7fca880d25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:12.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.629+0000 7fca8d200640 1 -- 172.21.15.96:0/2550937880 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca880d0260 con 0x7fca8815e870 2024-05-28T06:22:12.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.629+0000 7fca87e00640 1 -- 172.21.15.96:0/2550937880 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1739546127 0 0) 0x7fca8815ec50 con 0x7fca880cfa40 2024-05-28T06:22:12.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.630+0000 7fca8d200640 1 --2- 172.21.15.96:0/2550937880 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fca8815e870 0x7fca880d0f90 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7fca780097a0 tx=0x7fca7802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61bfeae1f6acf631 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:12.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.630+0000 7fca87e00640 1 -- 172.21.15.96:0/2550937880 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca78037070 con 0x7fca8815e870 2024-05-28T06:22:12.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.630+0000 7fca87e00640 1 -- 172.21.15.96:0/2550937880 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca78032440 con 0x7fca8815e870 2024-05-28T06:22:12.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.630+0000 7fca87e00640 1 -- 172.21.15.96:0/2550937880 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca78032810 con 0x7fca8815e870 2024-05-28T06:22:12.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.631+0000 7fca8e600640 1 -- 172.21.15.96:0/2550937880 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fca8815e870 msgr2=0x7fca880d0f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:12.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.631+0000 7fca8e600640 1 --2- 172.21.15.96:0/2550937880 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fca8815e870 0x7fca880d0f90 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7fca780097a0 tx=0x7fca7802a1e0 comp rx=0 tx=0).stop 2024-05-28T06:22:12.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.631+0000 7fca8e600640 1 -- 172.21.15.96:0/2550937880 shutdown_connections 2024-05-28T06:22:12.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.631+0000 7fca8e600640 1 --2- 172.21.15.96:0/2550937880 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fca8815e870 0x7fca880d0f90 unknown :-1 s=CLOSED pgs=1782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:12.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.631+0000 7fca8e600640 1 -- 172.21.15.96:0/2550937880 >> 172.21.15.96:0/2550937880 conn(0x7fca8808bfc0 msgr2=0x7fca880d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:12.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.631+0000 7fca8e600640 1 -- 172.21.15.96:0/2550937880 shutdown_connections 2024-05-28T06:22:12.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.631+0000 7fca8e600640 1 -- 172.21.15.96:0/2550937880 wait complete. 2024-05-28T06:22:12.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.632+0000 7fca8e600640 1 Processor -- start 2024-05-28T06:22:12.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.633+0000 7fca8e600640 1 -- start start 2024-05-28T06:22:12.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.633+0000 7fca8e600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fca880cfa40 0x7fca880b4740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:12.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.633+0000 7fca8e600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fca880d14d0 0x7fca880b5510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:12.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.634+0000 7fca8e600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fca8815e870 0x7fca881f4330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:12.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.634+0000 7fca8e600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fca880d4fb0 con 0x7fca8815e870 2024-05-28T06:22:12.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.634+0000 7fca8e600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fca880d0820 con 0x7fca880cfa40 2024-05-28T06:22:12.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.634+0000 7fca8e600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fca880d2cf0 con 0x7fca880d14d0 2024-05-28T06:22:12.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.634+0000 7fca8dc00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fca8815e870 0x7fca881f4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:12.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.634+0000 7fca8dc00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fca8815e870 0x7fca881f4330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:59898/0 (socket says 172.21.15.96:59898) 2024-05-28T06:22:12.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.634+0000 7fca8dc00640 1 -- 172.21.15.96:0/297298704 learned_addr learned my addr 172.21.15.96:0/297298704 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:22:12.631 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.634+0000 7fca7fe00640 1 --2- 172.21.15.96:0/297298704 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fca880d14d0 0x7fca880b5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:12.631 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.634+0000 7fca8d200640 1 --2- 172.21.15.96:0/297298704 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fca880cfa40 0x7fca880b4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:12.631 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.634+0000 7fca8dc00640 1 -- 172.21.15.96:0/297298704 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fca880d14d0 msgr2=0x7fca880b5510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:12.631 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.634+0000 7fca8dc00640 1 --2- 172.21.15.96:0/297298704 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fca880d14d0 0x7fca880b5510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:12.631 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.634+0000 7fca8dc00640 1 -- 172.21.15.96:0/297298704 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fca880cfa40 msgr2=0x7fca880b4740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:12.631 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.634+0000 7fca8dc00640 1 --2- 172.21.15.96:0/297298704 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fca880cfa40 0x7fca880b4740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:12.631 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.634+0000 7fca8dc00640 1 -- 172.21.15.96:0/297298704 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca880b3c80 con 0x7fca8815e870 2024-05-28T06:22:12.631 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.635+0000 7fca8dc00640 1 --2- 172.21.15.96:0/297298704 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fca8815e870 0x7fca881f4330 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7fca8000c780 tx=0x7fca8000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:12.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.635+0000 7fca85600640 1 -- 172.21.15.96:0/297298704 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca80018070 con 0x7fca8815e870 2024-05-28T06:22:12.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.635+0000 7fca85600640 1 -- 172.21.15.96:0/297298704 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca8000a640 con 0x7fca8815e870 2024-05-28T06:22:12.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.635+0000 7fca85600640 1 -- 172.21.15.96:0/297298704 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca8000aab0 con 0x7fca8815e870 2024-05-28T06:22:12.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.635+0000 7fca8e600640 1 -- 172.21.15.96:0/297298704 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca881f4ef0 con 0x7fca8815e870 2024-05-28T06:22:12.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.635+0000 7fca8e600640 1 -- 172.21.15.96:0/297298704 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca880b4220 con 0x7fca8815e870 2024-05-28T06:22:12.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.636+0000 7fca8e600640 1 -- 172.21.15.96:0/297298704 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca880b4420 con 0x7fca8815e870 2024-05-28T06:22:12.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.637+0000 7fca85600640 1 -- 172.21.15.96:0/297298704 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 37) ==== 93182+0+0 (secure 0 0 0) 0x7fca8000ac50 con 0x7fca8815e870 2024-05-28T06:22:12.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.638+0000 7fca85600640 1 --2- 172.21.15.96:0/297298704 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fca6406e090 0x7fca64070550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:12.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.638+0000 7fca85600640 1 -- 172.21.15.96:0/297298704 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 4913+0+0 (secure 0 0 0) 0x7fca8008a090 con 0x7fca8815e870 2024-05-28T06:22:12.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.639+0000 7fca8d200640 1 --2- 172.21.15.96:0/297298704 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fca6406e090 0x7fca64070550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:12.635 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.640+0000 7fca8d200640 1 --2- 172.21.15.96:0/297298704 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fca6406e090 0x7fca64070550 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fca78034000 tx=0x7fca7802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:12.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.642+0000 7fca85600640 1 -- 172.21.15.96:0/297298704 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fca880b4420 con 0x7fca8815e870 2024-05-28T06:22:12.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.848+0000 7fca8e600640 1 -- 172.21.15.96:0/297298704 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fca880689e0 con 0x7fca8815e870 2024-05-28T06:22:12.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.850+0000 7fca85600640 1 -- 172.21.15.96:0/297298704 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v111) ==== 74+0+13001 (secure 0 0 0) 0x7fca881f4ef0 con 0x7fca8815e870 2024-05-28T06:22:12.846 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:22:12.846 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":111,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:22:11.948783+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":13,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:22:12.849 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.853+0000 7fca8e600640 1 -- 172.21.15.96:0/297298704 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fca6406e090 msgr2=0x7fca64070550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:12.849 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.853+0000 7fca8e600640 1 --2- 172.21.15.96:0/297298704 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fca6406e090 0x7fca64070550 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fca78034000 tx=0x7fca7802e000 comp rx=0 tx=0).stop 2024-05-28T06:22:12.849 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.853+0000 7fca8e600640 1 -- 172.21.15.96:0/297298704 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fca8815e870 msgr2=0x7fca881f4330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:12.849 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.853+0000 7fca8e600640 1 --2- 172.21.15.96:0/297298704 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fca8815e870 0x7fca881f4330 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7fca8000c780 tx=0x7fca8000cc40 comp rx=0 tx=0).stop 2024-05-28T06:22:12.849 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.853+0000 7fca8e600640 1 -- 172.21.15.96:0/297298704 shutdown_connections 2024-05-28T06:22:12.849 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.853+0000 7fca8e600640 1 --2- 172.21.15.96:0/297298704 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fca6406e090 0x7fca64070550 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:12.849 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.853+0000 7fca8e600640 1 --2- 172.21.15.96:0/297298704 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fca8815e870 0x7fca881f4330 unknown :-1 s=CLOSED pgs=1783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:12.849 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.853+0000 7fca8e600640 1 --2- 172.21.15.96:0/297298704 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fca880d14d0 0x7fca880b5510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:12.849 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.853+0000 7fca8e600640 1 --2- 172.21.15.96:0/297298704 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fca880cfa40 0x7fca880b4740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:12.849 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.853+0000 7fca8e600640 1 -- 172.21.15.96:0/297298704 >> 172.21.15.96:0/297298704 conn(0x7fca8808bfc0 msgr2=0x7fca880b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:12.849 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.853+0000 7fca8e600640 1 -- 172.21.15.96:0/297298704 shutdown_connections 2024-05-28T06:22:12.850 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:12.853+0000 7fca8e600640 1 -- 172.21.15.96:0/297298704 wait complete. 2024-05-28T06:22:12.867 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-28T06:03:59.517102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-28T06:22:12.868 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-28T06:22:13.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.073+0000 7fbea6e00640 1 Processor -- start 2024-05-28T06:22:13.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.073+0000 7fbea6e00640 1 -- start start 2024-05-28T06:22:13.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.074+0000 7fbea6e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbea01524c0 0x7fbea00cf710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:13.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.074+0000 7fbea6e00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbea00b3e80 con 0x7fbea00caa40 2024-05-28T06:22:13.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.074+0000 7fbea6e00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbea00cde80 con 0x7fbea00cae20 2024-05-28T06:22:13.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.074+0000 7fbea6e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fbea00d0b30 con 0x7fbea01524c0 2024-05-28T06:22:13.071 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.075+0000 7fbea5a00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fbea00caa40 0x7fbea00ce960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:33144/0 (socket says 172.21.15.96:33144) 2024-05-28T06:22:13.071 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.075+0000 7fbea5a00640 1 -- 172.21.15.96:0/2131980863 learned_addr learned my addr 172.21.15.96:0/2131980863 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:22:13.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.075+0000 7fbe9fe00640 1 -- 172.21.15.96:0/2131980863 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3692242984 0 0) 0x7fbea00b3e80 con 0x7fbea00caa40 2024-05-28T06:22:13.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.075+0000 7fbe9fe00640 1 -- 172.21.15.96:0/2131980863 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbe88003520 con 0x7fbea00caa40 2024-05-28T06:22:13.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.075+0000 7fbea6400640 1 --2- 172.21.15.96:0/2131980863 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbea01524c0 0x7fbea00cf710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:13.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.075+0000 7fbe9fe00640 1 -- 172.21.15.96:0/2131980863 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2791542937 0 0) 0x7fbe88003520 con 0x7fbea00caa40 2024-05-28T06:22:13.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.075+0000 7fbe9fe00640 1 -- 172.21.15.96:0/2131980863 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbea01524c0 msgr2=0x7fbea00cf710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:13.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.075+0000 7fbe9fe00640 1 --2- 172.21.15.96:0/2131980863 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbea01524c0 0x7fbea00cf710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:13.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.075+0000 7fbe9fe00640 1 -- 172.21.15.96:0/2131980863 >> v1:172.21.15.123:6789/0 conn(0x7fbea00cae20 legacy=0x7fbea0151db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:13.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.075+0000 7fbe9fe00640 1 -- 172.21.15.96:0/2131980863 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbea00ce160 con 0x7fbea00caa40 2024-05-28T06:22:13.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.075+0000 7fbe9fe00640 1 -- 172.21.15.96:0/2131980863 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7fbe94001690 con 0x7fbea00caa40 2024-05-28T06:22:13.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.076+0000 7fbe9fe00640 1 -- 172.21.15.96:0/2131980863 >> v1:172.21.15.96:6789/0 conn(0x7fbea00caa40 legacy=0x7fbea00ce960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:13.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.076+0000 7fbe9fe00640 1 --2- 172.21.15.96:0/2131980863 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbea01524c0 0x7fbe880039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:13.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.076+0000 7fbe9fe00640 1 --2- 172.21.15.96:0/2131980863 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbe88004060 0x7fbe88004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:13.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.076+0000 7fbe9fe00640 1 --2- 172.21.15.96:0/2131980863 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe88004d40 0x7fbe880051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:13.075 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.076+0000 7fbe9fe00640 1 -- 172.21.15.96:0/2131980863 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fbea00d0b30 con 0x7fbe88004d40 2024-05-28T06:22:13.075 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.076+0000 7fbe9fe00640 1 -- 172.21.15.96:0/2131980863 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fbe8800bca0 con 0x7fbe88004060 2024-05-28T06:22:13.075 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.076+0000 7fbe9fe00640 1 -- 172.21.15.96:0/2131980863 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fbe8800bea0 con 0x7fbea01524c0 2024-05-28T06:22:13.075 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.076+0000 7fbea6400640 1 --2- 172.21.15.96:0/2131980863 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe88004d40 0x7fbe880051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:13.075 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.076+0000 7fbea5a00640 1 --2- 172.21.15.96:0/2131980863 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbea01524c0 0x7fbe880039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:13.075 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.076+0000 7fbea5000640 1 --2- 172.21.15.96:0/2131980863 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbe88004060 0x7fbe88004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:13.075 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.076+0000 7fbea6400640 1 -- 172.21.15.96:0/2131980863 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbea01524c0 msgr2=0x7fbe880039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:13.075 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.076+0000 7fbea6400640 1 --2- 172.21.15.96:0/2131980863 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbea01524c0 0x7fbe880039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:13.075 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.076+0000 7fbea6400640 1 -- 172.21.15.96:0/2131980863 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbe88004060 msgr2=0x7fbe88004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:13.075 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.076+0000 7fbea6400640 1 --2- 172.21.15.96:0/2131980863 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbe88004060 0x7fbe88004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:13.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.076+0000 7fbea6400640 1 -- 172.21.15.96:0/2131980863 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe88003520 con 0x7fbe88004d40 2024-05-28T06:22:13.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.076+0000 7fbea5000640 1 --2- 172.21.15.96:0/2131980863 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbe88004060 0x7fbe88004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:22:13.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.078+0000 7fbea6400640 1 --2- 172.21.15.96:0/2131980863 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe88004d40 0x7fbe880051c0 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7fbe98007360 tx=0x7fbe88005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49e8935ce0a4824b server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:13.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.078+0000 7fbe9fe00640 1 -- 172.21.15.96:0/2131980863 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe98009be0 con 0x7fbe88004d40 2024-05-28T06:22:13.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.078+0000 7fbe9fe00640 1 -- 172.21.15.96:0/2131980863 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe98036600 con 0x7fbe88004d40 2024-05-28T06:22:13.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.078+0000 7fbe9fe00640 1 -- 172.21.15.96:0/2131980863 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe980369d0 con 0x7fbe88004d40 2024-05-28T06:22:13.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.078+0000 7fbea6e00640 1 -- 172.21.15.96:0/2131980863 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe88004d40 msgr2=0x7fbe880051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:13.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.078+0000 7fbea6e00640 1 --2- 172.21.15.96:0/2131980863 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe88004d40 0x7fbe880051c0 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7fbe98007360 tx=0x7fbe88005830 comp rx=0 tx=0).stop 2024-05-28T06:22:13.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.078+0000 7fbea6400640 1 -- 172.21.15.96:0/2131980863 reap_dead start 2024-05-28T06:22:13.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.079+0000 7fbea6e00640 1 -- 172.21.15.96:0/2131980863 shutdown_connections 2024-05-28T06:22:13.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.079+0000 7fbea6e00640 1 -- 172.21.15.96:0/2131980863 >> 172.21.15.96:0/2131980863 conn(0x7fbea006b900 msgr2=0x7fbea006bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:13.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.079+0000 7fbea6e00640 1 -- 172.21.15.96:0/2131980863 shutdown_connections 2024-05-28T06:22:13.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.079+0000 7fbea6e00640 1 -- 172.21.15.96:0/2131980863 wait complete. 2024-05-28T06:22:13.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.080+0000 7fbea6e00640 1 Processor -- start 2024-05-28T06:22:13.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.081+0000 7fbea6e00640 1 -- start start 2024-05-28T06:22:13.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.081+0000 7fbea6e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbe88004d40 0x7fbea006e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:13.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.081+0000 7fbea6e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbea006f310 0x7fbea014b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:13.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.081+0000 7fbea6e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbea0071a80 0x7fbea00c94d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:13.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.082+0000 7fbea6e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fbea00cf070 con 0x7fbea0071a80 2024-05-28T06:22:13.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.082+0000 7fbea6e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fbea00b4160 con 0x7fbe88004d40 2024-05-28T06:22:13.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.082+0000 7fbea6e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fbea00cf710 con 0x7fbea006f310 2024-05-28T06:22:13.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.082+0000 7fbea6400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbea0071a80 0x7fbea00c94d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:13.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.082+0000 7fbea6400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbea0071a80 0x7fbea00c94d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:59920/0 (socket says 172.21.15.96:59920) 2024-05-28T06:22:13.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.082+0000 7fbea6400640 1 -- 172.21.15.96:0/3364420029 learned_addr learned my addr 172.21.15.96:0/3364420029 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:22:13.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.082+0000 7fbea5a00640 1 --2- 172.21.15.96:0/3364420029 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbe88004d40 0x7fbea006e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:13.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.082+0000 7fbea6400640 1 -- 172.21.15.96:0/3364420029 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbea006f310 msgr2=0x7fbea014b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:13.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.082+0000 7fbea6400640 1 --2- 172.21.15.96:0/3364420029 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbea006f310 0x7fbea014b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:13.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.082+0000 7fbea6400640 1 -- 172.21.15.96:0/3364420029 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbe88004d40 msgr2=0x7fbea006e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:13.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.082+0000 7fbea6400640 1 --2- 172.21.15.96:0/3364420029 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbe88004d40 0x7fbea006e8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:13.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.082+0000 7fbea6400640 1 -- 172.21.15.96:0/3364420029 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbea006f8d0 con 0x7fbea0071a80 2024-05-28T06:22:13.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.082+0000 7fbea6400640 1 --2- 172.21.15.96:0/3364420029 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbea0071a80 0x7fbea00c94d0 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7fbe98007450 tx=0x7fbe98004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:13.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.083+0000 7fbe9d600640 1 -- 172.21.15.96:0/3364420029 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe98036e60 con 0x7fbea0071a80 2024-05-28T06:22:13.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.083+0000 7fbe9d600640 1 -- 172.21.15.96:0/3364420029 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe98003990 con 0x7fbea0071a80 2024-05-28T06:22:13.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.083+0000 7fbea6e00640 1 -- 172.21.15.96:0/3364420029 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbea006fbb0 con 0x7fbea0071a80 2024-05-28T06:22:13.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.083+0000 7fbe9d600640 1 -- 172.21.15.96:0/3364420029 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe98009be0 con 0x7fbea0071a80 2024-05-28T06:22:13.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.083+0000 7fbea6e00640 1 -- 172.21.15.96:0/3364420029 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbea0153be0 con 0x7fbea0071a80 2024-05-28T06:22:13.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.084+0000 7fbea6e00640 1 -- 172.21.15.96:0/3364420029 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbea00689e0 con 0x7fbea0071a80 2024-05-28T06:22:13.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.085+0000 7fbe9d600640 1 -- 172.21.15.96:0/3364420029 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 37) ==== 93182+0+0 (secure 0 0 0) 0x7fbe98048020 con 0x7fbea0071a80 2024-05-28T06:22:13.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.086+0000 7fbe9d600640 1 --2- 172.21.15.96:0/3364420029 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbe8006e090 0x7fbe80070550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:13.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.086+0000 7fbe9d600640 1 -- 172.21.15.96:0/3364420029 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 4913+0+0 (secure 0 0 0) 0x7fbe980aafa0 con 0x7fbea0071a80 2024-05-28T06:22:13.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.087+0000 7fbea5a00640 1 --2- 172.21.15.96:0/3364420029 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbe8006e090 0x7fbe80070550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:13.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.088+0000 7fbea5a00640 1 --2- 172.21.15.96:0/3364420029 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbe8006e090 0x7fbe80070550 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fbe94001c10 tx=0x7fbe94000fb0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:13.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.090+0000 7fbe9d600640 1 -- 172.21.15.96:0/3364420029 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbe9803d070 con 0x7fbea0071a80 2024-05-28T06:22:13.268 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.273+0000 7fbea6e00640 1 -- 172.21.15.96:0/3364420029 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fbea00b4ee0 con 0x7fbea0071a80 2024-05-28T06:22:13.269 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.273+0000 7fbe9d600640 1 -- 172.21.15.96:0/3364420029 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v111) ==== 93+0+10 (secure 0 0 0) 0x7fbe98032030 con 0x7fbea0071a80 2024-05-28T06:22:13.269 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 1 2024-05-28T06:22:13.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.276+0000 7fbea6e00640 1 -- 172.21.15.96:0/3364420029 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbe8006e090 msgr2=0x7fbe80070550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:13.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.276+0000 7fbea6e00640 1 --2- 172.21.15.96:0/3364420029 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbe8006e090 0x7fbe80070550 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fbe94001c10 tx=0x7fbe94000fb0 comp rx=0 tx=0).stop 2024-05-28T06:22:13.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.276+0000 7fbea6e00640 1 -- 172.21.15.96:0/3364420029 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbea0071a80 msgr2=0x7fbea00c94d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:13.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.276+0000 7fbea6e00640 1 --2- 172.21.15.96:0/3364420029 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbea0071a80 0x7fbea00c94d0 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7fbe98007450 tx=0x7fbe98004ab0 comp rx=0 tx=0).stop 2024-05-28T06:22:13.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.276+0000 7fbea6e00640 1 -- 172.21.15.96:0/3364420029 shutdown_connections 2024-05-28T06:22:13.273 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.276+0000 7fbea6e00640 1 --2- 172.21.15.96:0/3364420029 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbe8006e090 0x7fbe80070550 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:13.273 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.276+0000 7fbea6e00640 1 --2- 172.21.15.96:0/3364420029 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbea0071a80 0x7fbea00c94d0 unknown :-1 s=CLOSED pgs=1786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:13.273 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.276+0000 7fbea6e00640 1 --2- 172.21.15.96:0/3364420029 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbea006f310 0x7fbea014b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:13.273 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.276+0000 7fbea6e00640 1 --2- 172.21.15.96:0/3364420029 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbe88004d40 0x7fbea006e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:13.273 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.276+0000 7fbea6e00640 1 -- 172.21.15.96:0/3364420029 >> 172.21.15.96:0/3364420029 conn(0x7fbea006b900 msgr2=0x7fbea00ad5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:13.273 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.276+0000 7fbea6e00640 1 -- 172.21.15.96:0/3364420029 shutdown_connections 2024-05-28T06:22:13.273 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.276+0000 7fbea6e00640 1 -- 172.21.15.96:0/3364420029 wait complete. 2024-05-28T06:22:13.289 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-05-28T06:22:13.486 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.490+0000 7f7cdb200640 1 Processor -- start 2024-05-28T06:22:13.486 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.491+0000 7f7cdb200640 1 -- start start 2024-05-28T06:22:13.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.492+0000 7f7cdb200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f7cdc162d20 0x7f7cdc071bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:13.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.492+0000 7f7cdb200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7cdc06f590 con 0x7f7cdc162940 2024-05-28T06:22:13.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.492+0000 7f7cdb200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7cdc06f870 con 0x7f7cdc072100 2024-05-28T06:22:13.488 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.492+0000 7f7cdb200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f7cdc072530 con 0x7f7cdc162d20 2024-05-28T06:22:13.490 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.493+0000 7f7cd9e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f7cdc162940 0x7f7cdc06eb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:33150/0 (socket says 172.21.15.96:33150) 2024-05-28T06:22:13.490 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.493+0000 7f7cd9e00640 1 -- 172.21.15.96:0/774306649 learned_addr learned my addr 172.21.15.96:0/774306649 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:22:13.490 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.493+0000 7f7cd9400640 1 --2- 172.21.15.96:0/774306649 >> v2:172.21.15.123:3300/0 conn(0x7f7cdc162d20 0x7f7cdc071bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:13.490 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.493+0000 7f7cd8a00640 1 -- 172.21.15.96:0/774306649 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3309155327 0 0) 0x7f7cdc06f590 con 0x7f7cdc162940 2024-05-28T06:22:13.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.493+0000 7f7cd8a00640 1 -- 172.21.15.96:0/774306649 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7cc0003520 con 0x7f7cdc162940 2024-05-28T06:22:13.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.493+0000 7f7cd8a00640 1 -- 172.21.15.96:0/774306649 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 696558738 0 0) 0x7f7cc0003520 con 0x7f7cdc162940 2024-05-28T06:22:13.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.493+0000 7f7cd8a00640 1 -- 172.21.15.96:0/774306649 >> v2:172.21.15.123:3300/0 conn(0x7f7cdc162d20 msgr2=0x7f7cdc071bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:13.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.493+0000 7f7cd8a00640 1 --2- 172.21.15.96:0/774306649 >> v2:172.21.15.123:3300/0 conn(0x7f7cdc162d20 0x7f7cdc071bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:13.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.493+0000 7f7cd8a00640 1 -- 172.21.15.96:0/774306649 >> v1:172.21.15.123:6789/0 conn(0x7f7cdc072100 legacy=0x7f7cdc0adbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:13.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.493+0000 7f7cd8a00640 1 -- 172.21.15.96:0/774306649 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cdc0ad3d0 con 0x7f7cdc162940 2024-05-28T06:22:13.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.493+0000 7f7cd8a00640 1 -- 172.21.15.96:0/774306649 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f7ccc001670 con 0x7f7cdc162940 2024-05-28T06:22:13.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.494+0000 7f7cd8a00640 1 -- 172.21.15.96:0/774306649 >> v1:172.21.15.96:6789/0 conn(0x7f7cdc162940 legacy=0x7f7cdc06eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:13.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.494+0000 7f7cd8a00640 1 --2- 172.21.15.96:0/774306649 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7cc00039b0 0x7f7cc0003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:13.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.494+0000 7f7cd8a00640 1 --2- 172.21.15.96:0/774306649 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7cc0004800 0x7f7cc0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:13.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.494+0000 7f7cd8a00640 1 --2- 172.21.15.96:0/774306649 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7cc0005230 0x7f7cc000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:13.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.494+0000 7f7cd8a00640 1 -- 172.21.15.96:0/774306649 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f7cdc072530 con 0x7f7cc0004800 2024-05-28T06:22:13.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.494+0000 7f7cd8a00640 1 -- 172.21.15.96:0/774306649 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f7cc000c190 con 0x7f7cc0005230 2024-05-28T06:22:13.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.494+0000 7f7cd8a00640 1 -- 172.21.15.96:0/774306649 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f7cc000c390 con 0x7f7cc00039b0 2024-05-28T06:22:13.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.494+0000 7f7cd9400640 1 --2- 172.21.15.96:0/774306649 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7cc0004800 0x7f7cc0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:13.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.494+0000 7f7cd9e00640 1 --2- 172.21.15.96:0/774306649 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7cc00039b0 0x7f7cc0003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:13.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.494+0000 7f7cd9400640 1 -- 172.21.15.96:0/774306649 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7cc00039b0 msgr2=0x7f7cc0003db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:13.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.494+0000 7f7cd9400640 1 --2- 172.21.15.96:0/774306649 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7cc00039b0 0x7f7cc0003db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:13.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.494+0000 7f7cda800640 1 --2- 172.21.15.96:0/774306649 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7cc0005230 0x7f7cc000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:13.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.494+0000 7f7cd9400640 1 -- 172.21.15.96:0/774306649 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7cc0005230 msgr2=0x7f7cc000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:13.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.494+0000 7f7cd9400640 1 --2- 172.21.15.96:0/774306649 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7cc0005230 0x7f7cc000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:13.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.494+0000 7f7cd9400640 1 -- 172.21.15.96:0/774306649 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cc0003520 con 0x7f7cc0004800 2024-05-28T06:22:13.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.495+0000 7f7cda800640 1 --2- 172.21.15.96:0/774306649 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7cc0005230 0x7f7cc000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:22:13.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.495+0000 7f7cda800640 1 -- 172.21.15.96:0/774306649 reap_dead start 2024-05-28T06:22:13.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.495+0000 7f7cd9400640 1 --2- 172.21.15.96:0/774306649 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7cc0004800 0x7f7cc0004c80 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7f7cdc072960 tx=0x7f7cc0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf2a9fa19204dc1c server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:13.494 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.496+0000 7f7cd8a00640 1 -- 172.21.15.96:0/774306649 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cc8036070 con 0x7f7cc0004800 2024-05-28T06:22:13.494 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.496+0000 7f7cd8a00640 1 -- 172.21.15.96:0/774306649 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7cc8031400 con 0x7f7cc0004800 2024-05-28T06:22:13.494 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.496+0000 7f7cd8a00640 1 -- 172.21.15.96:0/774306649 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cc8009690 con 0x7f7cc0004800 2024-05-28T06:22:13.494 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.496+0000 7f7cdb200640 1 -- 172.21.15.96:0/774306649 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7cc0004800 msgr2=0x7f7cc0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:13.494 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.496+0000 7f7cdb200640 1 --2- 172.21.15.96:0/774306649 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7cc0004800 0x7f7cc0004c80 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7f7cdc072960 tx=0x7f7cc0005830 comp rx=0 tx=0).stop 2024-05-28T06:22:13.494 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.496+0000 7f7cdb200640 1 -- 172.21.15.96:0/774306649 shutdown_connections 2024-05-28T06:22:13.494 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.496+0000 7f7cdb200640 1 --2- 172.21.15.96:0/774306649 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7cc0004800 0x7f7cc0004c80 unknown :-1 s=CLOSED pgs=1788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:13.494 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.496+0000 7f7cdb200640 1 -- 172.21.15.96:0/774306649 >> 172.21.15.96:0/774306649 conn(0x7f7cdc0b1fc0 msgr2=0x7f7cdc0b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:13.494 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.496+0000 7f7cdb200640 1 -- 172.21.15.96:0/774306649 shutdown_connections 2024-05-28T06:22:13.494 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.497+0000 7f7cdb200640 1 -- 172.21.15.96:0/774306649 wait complete. 2024-05-28T06:22:13.494 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.498+0000 7f7cdb200640 1 Processor -- start 2024-05-28T06:22:13.494 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.498+0000 7f7cdb200640 1 -- start start 2024-05-28T06:22:13.496 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.499+0000 7f7cdb200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7cc0004800 0x7f7cdc14f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:13.496 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.499+0000 7f7cdb200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7cdc0cdea0 0x7f7cdc1652d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:13.496 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.499+0000 7f7cdb200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7cdc165810 0x7f7cdc09ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:13.496 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.499+0000 7f7cdb200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f7cdc071d40 con 0x7f7cc0004800 2024-05-28T06:22:13.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.499+0000 7f7cdb200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f7cdc071bc0 con 0x7f7cdc165810 2024-05-28T06:22:13.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.499+0000 7f7cdb200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f7cdc071ec0 con 0x7f7cdc0cdea0 2024-05-28T06:22:13.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.499+0000 7f7cd9e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7cc0004800 0x7f7cdc14f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:13.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.500+0000 7f7cd9e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7cc0004800 0x7f7cdc14f570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:59940/0 (socket says 172.21.15.96:59940) 2024-05-28T06:22:13.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.500+0000 7f7cd9e00640 1 -- 172.21.15.96:0/3615810480 learned_addr learned my addr 172.21.15.96:0/3615810480 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:22:13.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.500+0000 7f7cd9e00640 1 -- 172.21.15.96:0/3615810480 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7cdc0cdea0 msgr2=0x7f7cdc1652d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:13.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.500+0000 7f7cd9e00640 1 --2- 172.21.15.96:0/3615810480 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7cdc0cdea0 0x7f7cdc1652d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:13.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.500+0000 7f7cd9e00640 1 -- 172.21.15.96:0/3615810480 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7cdc165810 msgr2=0x7f7cdc09ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:13.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.500+0000 7f7cda800640 1 --2- 172.21.15.96:0/3615810480 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7cdc165810 0x7f7cdc09ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:13.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.500+0000 7f7cd9e00640 1 --2- 172.21.15.96:0/3615810480 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7cdc165810 0x7f7cdc09ce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:13.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.500+0000 7f7cd9e00640 1 -- 172.21.15.96:0/3615810480 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cdc09dbe0 con 0x7f7cc0004800 2024-05-28T06:22:13.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.500+0000 7f7cd9e00640 1 --2- 172.21.15.96:0/3615810480 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7cc0004800 0x7f7cdc14f570 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc00d840 tx=0x7f7ccc00dd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:13.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.500+0000 7f7cd2000640 1 -- 172.21.15.96:0/3615810480 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ccc019070 con 0x7f7cc0004800 2024-05-28T06:22:13.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.501+0000 7f7cd2000640 1 -- 172.21.15.96:0/3615810480 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ccc0074e0 con 0x7f7cc0004800 2024-05-28T06:22:13.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.501+0000 7f7cd2000640 1 -- 172.21.15.96:0/3615810480 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ccc004a20 con 0x7f7cc0004800 2024-05-28T06:22:13.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.501+0000 7f7cdb200640 1 -- 172.21.15.96:0/3615810480 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cdc09dec0 con 0x7f7cc0004800 2024-05-28T06:22:13.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.501+0000 7f7cdb200640 1 -- 172.21.15.96:0/3615810480 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cdc09a600 con 0x7f7cc0004800 2024-05-28T06:22:13.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.502+0000 7f7cdb200640 1 -- 172.21.15.96:0/3615810480 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7cdc0689e0 con 0x7f7cc0004800 2024-05-28T06:22:13.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.503+0000 7f7cd2000640 1 -- 172.21.15.96:0/3615810480 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 37) ==== 93182+0+0 (secure 0 0 0) 0x7f7ccc004bc0 con 0x7f7cc0004800 2024-05-28T06:22:13.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.504+0000 7f7cd2000640 1 --2- 172.21.15.96:0/3615810480 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7cbc06e0c0 0x7f7cbc070580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:13.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.504+0000 7f7cd2000640 1 -- 172.21.15.96:0/3615810480 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 4913+0+0 (secure 0 0 0) 0x7f7ccc08a380 con 0x7f7cc0004800 2024-05-28T06:22:13.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.505+0000 7f7cd9400640 1 --2- 172.21.15.96:0/3615810480 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7cbc06e0c0 0x7f7cbc070580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:13.501 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.506+0000 7f7cd9400640 1 --2- 172.21.15.96:0/3615810480 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7cbc06e0c0 0x7f7cbc070580 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f7cc8033000 tx=0x7f7cc802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:13.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.509+0000 7f7cd2000640 1 -- 172.21.15.96:0/3615810480 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7ccc05a2b0 con 0x7f7cc0004800 2024-05-28T06:22:13.689 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.693+0000 7f7cdb200640 1 -- 172.21.15.96:0/3615810480 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f7cdc09a7d0 con 0x7f7cc0004800 2024-05-28T06:22:13.689 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.694+0000 7f7cd2000640 1 -- 172.21.15.96:0/3615810480 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v111) ==== 84+0+65 (secure 0 0 0) 0x7f7ccc059bb0 con 0x7f7cc0004800 2024-05-28T06:22:13.690 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:22:13.690 INFO:teuthology.orchestra.run.smithi096.stdout:[ 2024-05-28T06:22:13.690 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:22:13.690 INFO:teuthology.orchestra.run.smithi096.stdout: "poolnum": 1, 2024-05-28T06:22:13.690 INFO:teuthology.orchestra.run.smithi096.stdout: "poolname": ".mgr" 2024-05-28T06:22:13.690 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:22:13.690 INFO:teuthology.orchestra.run.smithi096.stdout:] 2024-05-28T06:22:13.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.697+0000 7f7cdb200640 1 -- 172.21.15.96:0/3615810480 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7cbc06e0c0 msgr2=0x7f7cbc070580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:13.693 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.697+0000 7f7cdb200640 1 --2- 172.21.15.96:0/3615810480 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7cbc06e0c0 0x7f7cbc070580 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f7cc8033000 tx=0x7f7cc802e000 comp rx=0 tx=0).stop 2024-05-28T06:22:13.693 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.697+0000 7f7cdb200640 1 -- 172.21.15.96:0/3615810480 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7cc0004800 msgr2=0x7f7cdc14f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:13.693 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.697+0000 7f7cdb200640 1 --2- 172.21.15.96:0/3615810480 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7cc0004800 0x7f7cdc14f570 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc00d840 tx=0x7f7ccc00dd00 comp rx=0 tx=0).stop 2024-05-28T06:22:13.693 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.697+0000 7f7cdb200640 1 -- 172.21.15.96:0/3615810480 shutdown_connections 2024-05-28T06:22:13.693 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.697+0000 7f7cdb200640 1 --2- 172.21.15.96:0/3615810480 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7cbc06e0c0 0x7f7cbc070580 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:13.693 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.697+0000 7f7cdb200640 1 --2- 172.21.15.96:0/3615810480 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7cdc165810 0x7f7cdc09ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:13.693 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.697+0000 7f7cdb200640 1 --2- 172.21.15.96:0/3615810480 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7cdc0cdea0 0x7f7cdc1652d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:13.693 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.697+0000 7f7cdb200640 1 --2- 172.21.15.96:0/3615810480 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7cc0004800 0x7f7cdc14f570 unknown :-1 s=CLOSED pgs=1789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:13.693 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.697+0000 7f7cdb200640 1 -- 172.21.15.96:0/3615810480 >> 172.21.15.96:0/3615810480 conn(0x7f7cdc0b1fc0 msgr2=0x7f7cdc0ad8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:13.693 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.697+0000 7f7cdb200640 1 -- 172.21.15.96:0/3615810480 shutdown_connections 2024-05-28T06:22:13.693 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.697+0000 7f7cdb200640 1 -- 172.21.15.96:0/3615810480 wait complete. 2024-05-28T06:22:13.711 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-05-28T06:22:13.711 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-05-28T06:22:13.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.917+0000 7f469d200640 1 Processor -- start 2024-05-28T06:22:13.914 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.918+0000 7f469d200640 1 -- start start 2024-05-28T06:22:13.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.919+0000 7f469d200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f469815c830 0x7f46980ce230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:13.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.919+0000 7f469d200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f46980d08e0 0x7f46980d0d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:13.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.919+0000 7f469d200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f469815cc10 con 0x7f46980ce770 2024-05-28T06:22:13.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.919+0000 7f469d200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f46980b4b90 con 0x7f46980d08e0 2024-05-28T06:22:13.915 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.919+0000 7f469d200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f46980b4e70 con 0x7f469815c830 2024-05-28T06:22:13.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.919+0000 7f4696800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f46980d08e0 0x7f46980d0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:13.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.919+0000 7f4696800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f46980d08e0 0x7f46980d0d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:59952/0 (socket says 172.21.15.96:59952) 2024-05-28T06:22:13.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.919+0000 7f4696800640 1 -- 172.21.15.96:0/1567246547 learned_addr learned my addr 172.21.15.96:0/1567246547 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:22:13.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.919+0000 7f4695e00640 1 --2- 172.21.15.96:0/1567246547 >> v2:172.21.15.123:3300/0 conn(0x7f469815c830 0x7f46980ce230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:13.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.919+0000 7f4696800640 1 -- 172.21.15.96:0/1567246547 >> v2:172.21.15.123:3300/0 conn(0x7f469815c830 msgr2=0x7f46980ce230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:13.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.919+0000 7f4696800640 1 --2- 172.21.15.96:0/1567246547 >> v2:172.21.15.123:3300/0 conn(0x7f469815c830 0x7f46980ce230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:13.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.919+0000 7f4696800640 1 -- 172.21.15.96:0/1567246547 >> v1:172.21.15.123:6789/0 conn(0x7f46980ce770 legacy=0x7f46980b40b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:13.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.920+0000 7f4696800640 1 -- 172.21.15.96:0/1567246547 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46980b5150 con 0x7f46980d08e0 2024-05-28T06:22:13.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.920+0000 7f4695e00640 1 --2- 172.21.15.96:0/1567246547 >> v2:172.21.15.123:3300/0 conn(0x7f469815c830 0x7f46980ce230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:22:13.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.921+0000 7f4696800640 1 --2- 172.21.15.96:0/1567246547 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f46980d08e0 0x7f46980d0d10 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f469806df90 tx=0x7f469002a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e69fdcc9f834ef17 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:13.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.921+0000 7f4694a00640 1 -- 172.21.15.96:0/1567246547 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4690036070 con 0x7f46980d08e0 2024-05-28T06:22:13.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.921+0000 7f4694a00640 1 -- 172.21.15.96:0/1567246547 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4690031600 con 0x7f46980d08e0 2024-05-28T06:22:13.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.921+0000 7f4694a00640 1 -- 172.21.15.96:0/1567246547 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46900319d0 con 0x7f46980d08e0 2024-05-28T06:22:13.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.921+0000 7f469d200640 1 -- 172.21.15.96:0/1567246547 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f46980d08e0 msgr2=0x7f46980d0d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:13.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.921+0000 7f469d200640 1 --2- 172.21.15.96:0/1567246547 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f46980d08e0 0x7f46980d0d10 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f469806df90 tx=0x7f469002a6f0 comp rx=0 tx=0).stop 2024-05-28T06:22:13.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.922+0000 7f469d200640 1 -- 172.21.15.96:0/1567246547 shutdown_connections 2024-05-28T06:22:13.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.922+0000 7f469d200640 1 --2- 172.21.15.96:0/1567246547 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f46980d08e0 0x7f46980d0d10 unknown :-1 s=CLOSED pgs=1790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:13.919 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.922+0000 7f469d200640 1 --2- 172.21.15.96:0/1567246547 >> v2:172.21.15.123:3300/0 conn(0x7f469815c830 0x7f46980ce230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:13.919 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.922+0000 7f469d200640 1 -- 172.21.15.96:0/1567246547 >> 172.21.15.96:0/1567246547 conn(0x7f46980b20a0 msgr2=0x7f46980b24b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:13.919 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.922+0000 7f469d200640 1 -- 172.21.15.96:0/1567246547 shutdown_connections 2024-05-28T06:22:13.919 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.922+0000 7f469d200640 1 -- 172.21.15.96:0/1567246547 wait complete. 2024-05-28T06:22:13.922 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.923+0000 7f469d200640 1 Processor -- start 2024-05-28T06:22:13.922 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.924+0000 7f469d200640 1 -- start start 2024-05-28T06:22:13.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.924+0000 7f469d200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f46980ce770 0x7f46980999f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:13.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.924+0000 7f469d200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f46980d08e0 0x7f4698159e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:13.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.924+0000 7f469d200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f469815c830 0x7f4698155c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:13.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.925+0000 7f469d200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f46980b4940 con 0x7f469815c830 2024-05-28T06:22:13.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.925+0000 7f469d200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f46980b47c0 con 0x7f46980ce770 2024-05-28T06:22:13.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.925+0000 7f469d200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f46980cffe0 con 0x7f46980d08e0 2024-05-28T06:22:13.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.925+0000 7f4696800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f469815c830 0x7f4698155c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:13.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.925+0000 7f4696800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f469815c830 0x7f4698155c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:59962/0 (socket says 172.21.15.96:59962) 2024-05-28T06:22:13.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.925+0000 7f4696800640 1 -- 172.21.15.96:0/3278163379 learned_addr learned my addr 172.21.15.96:0/3278163379 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:22:13.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.925+0000 7f4695e00640 1 --2- 172.21.15.96:0/3278163379 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f46980ce770 0x7f46980999f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:13.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.925+0000 7f4695400640 1 --2- 172.21.15.96:0/3278163379 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f46980d08e0 0x7f4698159e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:13.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.925+0000 7f4696800640 1 -- 172.21.15.96:0/3278163379 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f46980d08e0 msgr2=0x7f4698159e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:13.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.925+0000 7f4696800640 1 --2- 172.21.15.96:0/3278163379 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f46980d08e0 0x7f4698159e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:13.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.925+0000 7f4696800640 1 -- 172.21.15.96:0/3278163379 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f46980ce770 msgr2=0x7f46980999f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:13.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.925+0000 7f4696800640 1 --2- 172.21.15.96:0/3278163379 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f46980ce770 0x7f46980999f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:13.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.925+0000 7f4696800640 1 -- 172.21.15.96:0/3278163379 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4698154680 con 0x7f469815c830 2024-05-28T06:22:13.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.925+0000 7f4695e00640 1 --2- 172.21.15.96:0/3278163379 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f46980ce770 0x7f46980999f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:22:13.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.926+0000 7f4696800640 1 --2- 172.21.15.96:0/3278163379 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f469815c830 0x7f4698155c80 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7f4690009f40 tx=0x7f4690004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:13.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.926+0000 7f468e000640 1 -- 172.21.15.96:0/3278163379 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4690036050 con 0x7f469815c830 2024-05-28T06:22:13.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.926+0000 7f468e000640 1 -- 172.21.15.96:0/3278163379 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4690042070 con 0x7f469815c830 2024-05-28T06:22:13.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.926+0000 7f468e000640 1 -- 172.21.15.96:0/3278163379 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4690009510 con 0x7f469815c830 2024-05-28T06:22:13.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.926+0000 7f469d200640 1 -- 172.21.15.96:0/3278163379 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46980aedd0 con 0x7f469815c830 2024-05-28T06:22:13.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.926+0000 7f469d200640 1 -- 172.21.15.96:0/3278163379 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46980af630 con 0x7f469815c830 2024-05-28T06:22:13.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.927+0000 7f469d200640 1 -- 172.21.15.96:0/3278163379 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46980757d0 con 0x7f469815c830 2024-05-28T06:22:13.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.928+0000 7f468e000640 1 -- 172.21.15.96:0/3278163379 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 37) ==== 93182+0+0 (secure 0 0 0) 0x7f4690007bf0 con 0x7f469815c830 2024-05-28T06:22:13.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.930+0000 7f468e000640 1 --2- 172.21.15.96:0/3278163379 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f467806e090 0x7f4678070550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:13.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.930+0000 7f468e000640 1 -- 172.21.15.96:0/3278163379 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 4913+0+0 (secure 0 0 0) 0x7f46900ab840 con 0x7f469815c830 2024-05-28T06:22:13.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.930+0000 7f4695e00640 1 --2- 172.21.15.96:0/3278163379 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f467806e090 0x7f4678070550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:13.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.931+0000 7f4695e00640 1 --2- 172.21.15.96:0/3278163379 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f467806e090 0x7f4678070550 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f4688006fd0 tx=0x7f4688008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:13.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:13.934+0000 7f468e000640 1 -- 172.21.15.96:0/3278163379 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f469007b770 con 0x7f469815c830 2024-05-28T06:22:14.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:14.119+0000 7f469d200640 1 -- 172.21.15.96:0/3278163379 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f46980689e0 con 0x7f469815c830 2024-05-28T06:22:14.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:14.949+0000 7f468e000640 1 -- 172.21.15.96:0/3278163379 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v112) ==== 138+0+0 (secure 0 0 0) 0x7f469007b070 con 0x7f469815c830 2024-05-28T06:22:14.945 INFO:teuthology.orchestra.run.smithi096.stderr:pool 'cephfs_metadata' created 2024-05-28T06:22:14.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:14.954+0000 7f469d200640 1 -- 172.21.15.96:0/3278163379 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f467806e090 msgr2=0x7f4678070550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:14.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:14.954+0000 7f469d200640 1 --2- 172.21.15.96:0/3278163379 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f467806e090 0x7f4678070550 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f4688006fd0 tx=0x7f4688008040 comp rx=0 tx=0).stop 2024-05-28T06:22:14.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:14.955+0000 7f469d200640 1 -- 172.21.15.96:0/3278163379 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f469815c830 msgr2=0x7f4698155c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:14.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:14.955+0000 7f469d200640 1 --2- 172.21.15.96:0/3278163379 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f469815c830 0x7f4698155c80 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7f4690009f40 tx=0x7f4690004a40 comp rx=0 tx=0).stop 2024-05-28T06:22:14.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:14.955+0000 7f469d200640 1 -- 172.21.15.96:0/3278163379 shutdown_connections 2024-05-28T06:22:14.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:14.955+0000 7f469d200640 1 --2- 172.21.15.96:0/3278163379 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f467806e090 0x7f4678070550 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:14.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:14.955+0000 7f469d200640 1 --2- 172.21.15.96:0/3278163379 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f469815c830 0x7f4698155c80 unknown :-1 s=CLOSED pgs=1791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:14.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:14.955+0000 7f469d200640 1 --2- 172.21.15.96:0/3278163379 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f46980d08e0 0x7f4698159e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:14.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:14.955+0000 7f469d200640 1 --2- 172.21.15.96:0/3278163379 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f46980ce770 0x7f46980999f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:14.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:14.955+0000 7f469d200640 1 -- 172.21.15.96:0/3278163379 >> 172.21.15.96:0/3278163379 conn(0x7f46980b20a0 msgr2=0x7f46980cf7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:14.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:14.955+0000 7f469d200640 1 -- 172.21.15.96:0/3278163379 shutdown_connections 2024-05-28T06:22:14.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:14.955+0000 7f469d200640 1 -- 172.21.15.96:0/3278163379 wait complete. 2024-05-28T06:22:14.976 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-05-28T06:22:15.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:15.174+0000 7fe343a00640 1 Processor -- start 2024-05-28T06:22:15.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:15.174+0000 7fe343a00640 1 -- start start 2024-05-28T06:22:15.171 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:15.176+0000 7fe343a00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fe33c15e850 0x7fe33c0b77b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:15.171 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:15.176+0000 7fe343a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fe33c0d1d50 0x7fe33c0b90a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:15.171 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:15.176+0000 7fe343a00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe33c0b8000 con 0x7fe33c0cef30 2024-05-28T06:22:15.172 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:15.176+0000 7fe343a00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fe33c15ec30 con 0x7fe33c0d1d50 2024-05-28T06:22:15.172 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:15.176+0000 7fe343a00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fe33c0b82e0 con 0x7fe33c15e850 2024-05-28T06:22:15.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:15.176+0000 7fe341a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fe33c0d1d50 0x7fe33c0b90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:15.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:15.176+0000 7fe33be00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fe33c0cef30 0x7fe33c0d1640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:33156/0 (socket says 172.21.15.96:33156) 2024-05-28T06:22:15.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:15.176+0000 7fe33be00640 1 -- 172.21.15.96:0/1154528586 learned_addr learned my addr 172.21.15.96:0/1154528586 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:22:15.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:15.177+0000 7fe341000640 1 --2- 172.21.15.96:0/1154528586 >> v2:172.21.15.123:3300/0 conn(0x7fe33c15e850 0x7fe33c0b77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:15.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:15.177+0000 7fe341a00640 1 -- 172.21.15.96:0/1154528586 >> v2:172.21.15.123:3300/0 conn(0x7fe33c15e850 msgr2=0x7fe33c0b77b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:15.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:15.177+0000 7fe341a00640 1 --2- 172.21.15.96:0/1154528586 >> v2:172.21.15.123:3300/0 conn(0x7fe33c15e850 0x7fe33c0b77b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:15.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:15.177+0000 7fe341a00640 1 -- 172.21.15.96:0/1154528586 >> v1:172.21.15.96:6789/0 conn(0x7fe33c0cef30 legacy=0x7fe33c0d1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:15.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:15.177+0000 7fe341a00640 1 -- 172.21.15.96:0/1154528586 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe33c0b85c0 con 0x7fe33c0d1d50 2024-05-28T06:22:15.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:15.177+0000 7fe33b400640 1 -- 172.21.15.96:0/1154528586 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2904321574 0 0) 0x7fe33c0b8000 con 0x7fe33c0cef30 2024-05-28T06:22:15.174 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:15.178+0000 7fe341a00640 1 --2- 172.21.15.96:0/1154528586 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe33c0d1d50 0x7fe33c0b90a0 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7fe33c06dba0 tx=0x7fe33402a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8113330c22e4d49 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:15.174 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:15.178+0000 7fe33b400640 1 -- 172.21.15.96:0/1154528586 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe334007740 con 0x7fe33c0d1d50 2024-05-28T06:22:15.174 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:15.178+0000 7fe33b400640 1 -- 172.21.15.96:0/1154528586 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3340078e0 con 0x7fe33c0d1d50 2024-05-28T06:22:15.174 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:15.178+0000 7fe33b400640 1 -- 172.21.15.96:0/1154528586 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe334007cb0 con 0x7fe33c0d1d50 2024-05-28T06:22:15.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:15.179+0000 7fe343a00640 1 -- 172.21.15.96:0/1154528586 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe33c0d1d50 msgr2=0x7fe33c0b90a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:15.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:15.179+0000 7fe343a00640 1 --2- 172.21.15.96:0/1154528586 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe33c0d1d50 0x7fe33c0b90a0 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7fe33c06dba0 tx=0x7fe33402a5f0 comp rx=0 tx=0).stop 2024-05-28T06:22:15.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:15.179+0000 7fe343a00640 1 -- 172.21.15.96:0/1154528586 shutdown_connections 2024-05-28T06:22:15.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:15.179+0000 7fe343a00640 1 --2- 172.21.15.96:0/1154528586 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe33c0d1d50 0x7fe33c0b90a0 unknown :-1 s=CLOSED pgs=1793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:15.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:15.179+0000 7fe343a00640 1 --2- 172.21.15.96:0/1154528586 >> v2:172.21.15.123:3300/0 conn(0x7fe33c15e850 0x7fe33c0b77b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:15.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:15.179+0000 7fe343a00640 1 -- 172.21.15.96:0/1154528586 >> 172.21.15.96:0/1154528586 conn(0x7fe33c067aa0 msgr2=0x7fe33c067eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:15.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:15.179+0000 7fe343a00640 1 -- 172.21.15.96:0/1154528586 shutdown_connections 2024-05-28T06:22:15.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:15.179+0000 7fe343a00640 1 -- 172.21.15.96:0/1154528586 wait complete. 2024-05-28T06:22:15.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:15.181+0000 7fe343a00640 1 Processor -- start 2024-05-28T06:22:15.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:15.181+0000 7fe343a00640 1 -- start start 2024-05-28T06:22:15.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:15.182+0000 7fe343a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe33c0cef30 0x7fe33c0ce520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:15.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:15.182+0000 7fe343a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe33c0d1d50 0x7fe33c150450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:15.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:15.182+0000 7fe343a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe33c15e850 0x7fe33c0abad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:15.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:15.182+0000 7fe343a00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe33c0b95e0 con 0x7fe33c0d1d50 2024-05-28T06:22:15.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:15.182+0000 7fe343a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe33c0b7cf0 con 0x7fe33c0cef30 2024-05-28T06:22:15.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:15.182+0000 7fe343a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fe33c0b9760 con 0x7fe33c15e850 2024-05-28T06:22:15.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:15.182+0000 7fe33be00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe33c0d1d50 0x7fe33c150450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:15.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:15.182+0000 7fe33be00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe33c0d1d50 0x7fe33c150450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:59972/0 (socket says 172.21.15.96:59972) 2024-05-28T06:22:15.180 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:15.182+0000 7fe33be00640 1 -- 172.21.15.96:0/826860534 learned_addr learned my addr 172.21.15.96:0/826860534 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:22:15.180 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:15.182+0000 7fe341000640 1 --2- 172.21.15.96:0/826860534 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe33c0cef30 0x7fe33c0ce520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:15.180 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:15.183+0000 7fe33be00640 1 -- 172.21.15.96:0/826860534 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe33c15e850 msgr2=0x7fe33c0abad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:15.180 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:15.183+0000 7fe33be00640 1 --2- 172.21.15.96:0/826860534 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe33c15e850 0x7fe33c0abad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:15.180 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:15.183+0000 7fe33be00640 1 -- 172.21.15.96:0/826860534 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe33c0cef30 msgr2=0x7fe33c0ce520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:15.180 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:15.183+0000 7fe33be00640 1 --2- 172.21.15.96:0/826860534 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe33c0cef30 0x7fe33c0ce520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:15.180 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:15.183+0000 7fe33be00640 1 -- 172.21.15.96:0/826860534 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe33c0ac300 con 0x7fe33c0d1d50 2024-05-28T06:22:15.180 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:15.183+0000 7fe33be00640 1 --2- 172.21.15.96:0/826860534 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe33c0d1d50 0x7fe33c150450 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7fe32400c4d0 tx=0x7fe32400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:15.180 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:15.183+0000 7fe338c00640 1 -- 172.21.15.96:0/826860534 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe324018070 con 0x7fe33c0d1d50 2024-05-28T06:22:15.180 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:15.183+0000 7fe338c00640 1 -- 172.21.15.96:0/826860534 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe324007500 con 0x7fe33c0d1d50 2024-05-28T06:22:15.180 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:15.183+0000 7fe338c00640 1 -- 172.21.15.96:0/826860534 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe324004a20 con 0x7fe33c0d1d50 2024-05-28T06:22:15.180 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:15.183+0000 7fe343a00640 1 -- 172.21.15.96:0/826860534 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe33c0cb8d0 con 0x7fe33c0d1d50 2024-05-28T06:22:15.180 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:15.183+0000 7fe343a00640 1 -- 172.21.15.96:0/826860534 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe33c0ac010 con 0x7fe33c0d1d50 2024-05-28T06:22:15.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:15.185+0000 7fe343a00640 1 -- 172.21.15.96:0/826860534 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe33c0689e0 con 0x7fe33c0d1d50 2024-05-28T06:22:15.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:15.186+0000 7fe338c00640 1 -- 172.21.15.96:0/826860534 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 37) ==== 93182+0+0 (secure 0 0 0) 0x7fe32401b070 con 0x7fe33c0d1d50 2024-05-28T06:22:15.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:15.187+0000 7fe338c00640 1 --2- 172.21.15.96:0/826860534 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe31806e090 0x7fe318070550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:15.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:15.187+0000 7fe338c00640 1 -- 172.21.15.96:0/826860534 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 5283+0+0 (secure 0 0 0) 0x7fe324089ca0 con 0x7fe33c0d1d50 2024-05-28T06:22:15.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:15.188+0000 7fe341000640 1 --2- 172.21.15.96:0/826860534 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe31806e090 0x7fe318070550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:15.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:15.189+0000 7fe341000640 1 --2- 172.21.15.96:0/826860534 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe31806e090 0x7fe318070550 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fe32c006fd0 tx=0x7fe32c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:15.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:15.192+0000 7fe338c00640 1 -- 172.21.15.96:0/826860534 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe324059b30 con 0x7fe33c0d1d50 2024-05-28T06:22:15.375 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:15.380+0000 7fe343a00640 1 -- 172.21.15.96:0/826860534 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fe33c06bb00 con 0x7fe33c0d1d50 2024-05-28T06:22:15.953 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:15.957+0000 7fe338c00640 1 -- 172.21.15.96:0/826860534 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v113) ==== 170+0+0 (secure 0 0 0) 0x7fe324059430 con 0x7fe33c0d1d50 2024-05-28T06:22:15.953 INFO:teuthology.orchestra.run.smithi096.stderr:pool 'cephfs_data' created 2024-05-28T06:22:15.956 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:15.961+0000 7fe343a00640 1 -- 172.21.15.96:0/826860534 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe31806e090 msgr2=0x7fe318070550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:15.956 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:15.961+0000 7fe343a00640 1 --2- 172.21.15.96:0/826860534 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe31806e090 0x7fe318070550 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fe32c006fd0 tx=0x7fe32c008040 comp rx=0 tx=0).stop 2024-05-28T06:22:15.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:15.961+0000 7fe343a00640 1 -- 172.21.15.96:0/826860534 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe33c0d1d50 msgr2=0x7fe33c150450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:15.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:15.961+0000 7fe343a00640 1 --2- 172.21.15.96:0/826860534 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe33c0d1d50 0x7fe33c150450 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7fe32400c4d0 tx=0x7fe32400c990 comp rx=0 tx=0).stop 2024-05-28T06:22:15.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:15.961+0000 7fe343a00640 1 -- 172.21.15.96:0/826860534 shutdown_connections 2024-05-28T06:22:15.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:15.961+0000 7fe343a00640 1 --2- 172.21.15.96:0/826860534 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe31806e090 0x7fe318070550 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:15.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:15.961+0000 7fe343a00640 1 --2- 172.21.15.96:0/826860534 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe33c15e850 0x7fe33c0abad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:15.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:15.961+0000 7fe343a00640 1 --2- 172.21.15.96:0/826860534 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe33c0d1d50 0x7fe33c150450 unknown :-1 s=CLOSED pgs=1794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:15.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:15.961+0000 7fe343a00640 1 --2- 172.21.15.96:0/826860534 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe33c0cef30 0x7fe33c0ce520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:15.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:15.961+0000 7fe343a00640 1 -- 172.21.15.96:0/826860534 >> 172.21.15.96:0/826860534 conn(0x7fe33c067aa0 msgr2=0x7fe33c15eea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:15.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:15.961+0000 7fe343a00640 1 -- 172.21.15.96:0/826860534 shutdown_connections 2024-05-28T06:22:15.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:15.961+0000 7fe343a00640 1 -- 172.21.15.96:0/826860534 wait complete. 2024-05-28T06:22:15.992 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-05-28T06:22:16.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:16.171+0000 7fad99000640 1 Processor -- start 2024-05-28T06:22:16.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:16.172+0000 7fad99000640 1 -- start start 2024-05-28T06:22:16.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:16.173+0000 7fad99000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fad940cae20 0x7fad94151db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:16.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:16.173+0000 7fad99000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fad941522f0 0x7fad940cf710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:16.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:16.173+0000 7fad99000640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad940b3e80 con 0x7fad940caa40 2024-05-28T06:22:16.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:16.173+0000 7fad99000640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fad940d0b30 con 0x7fad941522f0 2024-05-28T06:22:16.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:16.173+0000 7fad99000640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fad940b4160 con 0x7fad940cae20 2024-05-28T06:22:16.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:16.173+0000 7fad93e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fad941522f0 0x7fad940cf710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:16.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:16.173+0000 7fad93e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fad941522f0 0x7fad940cf710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:59978/0 (socket says 172.21.15.96:59978) 2024-05-28T06:22:16.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:16.173+0000 7fad93e00640 1 -- 172.21.15.96:0/3870604246 learned_addr learned my addr 172.21.15.96:0/3870604246 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:22:16.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:16.174+0000 7fad92a00640 1 --2- 172.21.15.96:0/3870604246 >> v2:172.21.15.123:3300/0 conn(0x7fad940cae20 0x7fad94151db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:16.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:16.174+0000 7fad93e00640 1 -- 172.21.15.96:0/3870604246 >> v2:172.21.15.123:3300/0 conn(0x7fad940cae20 msgr2=0x7fad94151db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:16.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:16.174+0000 7fad93e00640 1 --2- 172.21.15.96:0/3870604246 >> v2:172.21.15.123:3300/0 conn(0x7fad940cae20 0x7fad94151db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:16.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:16.174+0000 7fad93e00640 1 -- 172.21.15.96:0/3870604246 >> v1:172.21.15.123:6789/0 conn(0x7fad940caa40 legacy=0x7fad940ce960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:16.171 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:16.174+0000 7fad93e00640 1 -- 172.21.15.96:0/3870604246 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad940cde80 con 0x7fad941522f0 2024-05-28T06:22:16.171 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:16.174+0000 7fad92a00640 1 --2- 172.21.15.96:0/3870604246 >> v2:172.21.15.123:3300/0 conn(0x7fad940cae20 0x7fad94151db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:22:16.171 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:16.175+0000 7fad93e00640 1 --2- 172.21.15.96:0/3870604246 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad941522f0 0x7fad940cf710 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7fad940d0fb0 tx=0x7fad8c02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=913a949ef4feb379 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:16.171 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:16.175+0000 7fad92000640 1 -- 172.21.15.96:0/3870604246 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad8c037070 con 0x7fad941522f0 2024-05-28T06:22:16.171 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:16.175+0000 7fad92000640 1 -- 172.21.15.96:0/3870604246 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad8c032440 con 0x7fad941522f0 2024-05-28T06:22:16.171 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:16.175+0000 7fad92000640 1 -- 172.21.15.96:0/3870604246 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad8c032810 con 0x7fad941522f0 2024-05-28T06:22:16.172 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:16.175+0000 7fad99000640 1 -- 172.21.15.96:0/3870604246 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad941522f0 msgr2=0x7fad940cf710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:16.172 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:16.175+0000 7fad99000640 1 --2- 172.21.15.96:0/3870604246 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad941522f0 0x7fad940cf710 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7fad940d0fb0 tx=0x7fad8c02a1a0 comp rx=0 tx=0).stop 2024-05-28T06:22:16.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:16.176+0000 7fad99000640 1 -- 172.21.15.96:0/3870604246 shutdown_connections 2024-05-28T06:22:16.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:16.176+0000 7fad99000640 1 --2- 172.21.15.96:0/3870604246 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad941522f0 0x7fad940cf710 unknown :-1 s=CLOSED pgs=1795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:16.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:16.176+0000 7fad99000640 1 --2- 172.21.15.96:0/3870604246 >> v2:172.21.15.123:3300/0 conn(0x7fad940cae20 0x7fad94151db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:16.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:16.176+0000 7fad99000640 1 -- 172.21.15.96:0/3870604246 >> 172.21.15.96:0/3870604246 conn(0x7fad9406b900 msgr2=0x7fad9406bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:16.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:16.176+0000 7fad99000640 1 -- 172.21.15.96:0/3870604246 shutdown_connections 2024-05-28T06:22:16.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:16.176+0000 7fad99000640 1 -- 172.21.15.96:0/3870604246 wait complete. 2024-05-28T06:22:16.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:16.177+0000 7fad99000640 1 Processor -- start 2024-05-28T06:22:16.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:16.178+0000 7fad99000640 1 -- start start 2024-05-28T06:22:16.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:16.178+0000 7fad99000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fad940caa40 0x7fad940aff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:16.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:16.178+0000 7fad99000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad940cae20 0x7fad940b04a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:16.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:16.178+0000 7fad99000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fad941522f0 0x7fad940b09e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:16.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:16.179+0000 7fad99000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fad940ce440 con 0x7fad940cae20 2024-05-28T06:22:16.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:16.179+0000 7fad99000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fad940cf070 con 0x7fad940caa40 2024-05-28T06:22:16.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:16.179+0000 7fad99000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fad94152710 con 0x7fad941522f0 2024-05-28T06:22:16.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:16.179+0000 7fad92a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad940cae20 0x7fad940b04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:16.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:16.179+0000 7fad92a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad940cae20 0x7fad940b04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:59992/0 (socket says 172.21.15.96:59992) 2024-05-28T06:22:16.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:16.179+0000 7fad92a00640 1 -- 172.21.15.96:0/2836889859 learned_addr learned my addr 172.21.15.96:0/2836889859 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:22:16.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:16.179+0000 7fad93400640 1 --2- 172.21.15.96:0/2836889859 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fad940caa40 0x7fad940aff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:16.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:16.179+0000 7fad93e00640 1 --2- 172.21.15.96:0/2836889859 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fad941522f0 0x7fad940b09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:16.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:16.179+0000 7fad92a00640 1 -- 172.21.15.96:0/2836889859 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fad941522f0 msgr2=0x7fad940b09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:16.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:16.179+0000 7fad92a00640 1 --2- 172.21.15.96:0/2836889859 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fad941522f0 0x7fad940b09e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:16.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:16.179+0000 7fad92a00640 1 -- 172.21.15.96:0/2836889859 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fad940caa40 msgr2=0x7fad940aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:16.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:16.179+0000 7fad92a00640 1 --2- 172.21.15.96:0/2836889859 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fad940caa40 0x7fad940aff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:16.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:16.179+0000 7fad92a00640 1 -- 172.21.15.96:0/2836889859 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad94146fc0 con 0x7fad940cae20 2024-05-28T06:22:16.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:16.179+0000 7fad93400640 1 --2- 172.21.15.96:0/2836889859 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fad940caa40 0x7fad940aff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:22:16.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:16.180+0000 7fad92a00640 1 --2- 172.21.15.96:0/2836889859 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad940cae20 0x7fad940b04a0 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7fad7c00cf60 tx=0x7fad7c00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:16.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:16.180+0000 7fad8b400640 1 -- 172.21.15.96:0/2836889859 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad7c01a070 con 0x7fad940cae20 2024-05-28T06:22:16.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:16.180+0000 7fad8b400640 1 -- 172.21.15.96:0/2836889859 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad7c00d840 con 0x7fad940cae20 2024-05-28T06:22:16.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:16.180+0000 7fad8b400640 1 -- 172.21.15.96:0/2836889859 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad7c00dc10 con 0x7fad940cae20 2024-05-28T06:22:16.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:16.180+0000 7fad99000640 1 -- 172.21.15.96:0/2836889859 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad941472a0 con 0x7fad940cae20 2024-05-28T06:22:16.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:16.180+0000 7fad99000640 1 -- 172.21.15.96:0/2836889859 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad94147860 con 0x7fad940cae20 2024-05-28T06:22:16.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:16.181+0000 7fad99000640 1 -- 172.21.15.96:0/2836889859 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad940689e0 con 0x7fad940cae20 2024-05-28T06:22:16.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:16.183+0000 7fad8b400640 1 -- 172.21.15.96:0/2836889859 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 37) ==== 93182+0+0 (secure 0 0 0) 0x7fad7c01d070 con 0x7fad940cae20 2024-05-28T06:22:16.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:16.183+0000 7fad8b400640 1 --2- 172.21.15.96:0/2836889859 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fad7006e090 0x7fad70070550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:16.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:16.184+0000 7fad8b400640 1 -- 172.21.15.96:0/2836889859 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 5665+0+0 (secure 0 0 0) 0x7fad7c08b970 con 0x7fad940cae20 2024-05-28T06:22:16.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:16.184+0000 7fad93400640 1 --2- 172.21.15.96:0/2836889859 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fad7006e090 0x7fad70070550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:16.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:16.185+0000 7fad93400640 1 --2- 172.21.15.96:0/2836889859 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fad7006e090 0x7fad70070550 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fad84007fd0 tx=0x7fad84009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:16.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:16.187+0000 7fad8b400640 1 -- 172.21.15.96:0/2836889859 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fad7c05b230 con 0x7fad940cae20 2024-05-28T06:22:16.417 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:16.422+0000 7fad99000640 1 -- 172.21.15.96:0/2836889859 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fad941494b0 con 0x7fad940cae20 2024-05-28T06:22:17.000 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.004+0000 7fad8b400640 1 -- 172.21.15.96:0/2836889859 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 16 and data pool 17 v68) ==== 174+0+0 (secure 0 0 0) 0x7fad7c05f530 con 0x7fad940cae20 2024-05-28T06:22:17.000 INFO:teuthology.orchestra.run.smithi096.stderr:new fs with metadata pool 16 and data pool 17 2024-05-28T06:22:17.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.008+0000 7fad99000640 1 -- 172.21.15.96:0/2836889859 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fad7006e090 msgr2=0x7fad70070550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:17.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.008+0000 7fad99000640 1 --2- 172.21.15.96:0/2836889859 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fad7006e090 0x7fad70070550 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fad84007fd0 tx=0x7fad84009040 comp rx=0 tx=0).stop 2024-05-28T06:22:17.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.008+0000 7fad99000640 1 -- 172.21.15.96:0/2836889859 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad940cae20 msgr2=0x7fad940b04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:17.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.008+0000 7fad99000640 1 --2- 172.21.15.96:0/2836889859 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad940cae20 0x7fad940b04a0 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7fad7c00cf60 tx=0x7fad7c00cf90 comp rx=0 tx=0).stop 2024-05-28T06:22:17.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.008+0000 7fad99000640 1 -- 172.21.15.96:0/2836889859 shutdown_connections 2024-05-28T06:22:17.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.008+0000 7fad99000640 1 --2- 172.21.15.96:0/2836889859 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fad7006e090 0x7fad70070550 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:17.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.008+0000 7fad99000640 1 --2- 172.21.15.96:0/2836889859 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fad941522f0 0x7fad940b09e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:17.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.008+0000 7fad99000640 1 --2- 172.21.15.96:0/2836889859 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad940cae20 0x7fad940b04a0 unknown :-1 s=CLOSED pgs=1796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:17.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.008+0000 7fad99000640 1 --2- 172.21.15.96:0/2836889859 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fad940caa40 0x7fad940aff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:17.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.008+0000 7fad99000640 1 -- 172.21.15.96:0/2836889859 >> 172.21.15.96:0/2836889859 conn(0x7fad9406b900 msgr2=0x7fad940ccb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:17.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.008+0000 7fad99000640 1 -- 172.21.15.96:0/2836889859 shutdown_connections 2024-05-28T06:22:17.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.008+0000 7fad99000640 1 -- 172.21.15.96:0/2836889859 wait complete. 2024-05-28T06:22:17.029 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:22:17.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.196+0000 7f6525400640 1 Processor -- start 2024-05-28T06:22:17.192 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.196+0000 7f6525400640 1 -- start start 2024-05-28T06:22:17.193 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.197+0000 7f6525400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f652006ee40 0x7f65200ad230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:17.194 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.197+0000 7f6525400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f652006f2c0 0x7f65200ad770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:17.194 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.197+0000 7f6525400640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6520156f10 con 0x7f6520156b30 2024-05-28T06:22:17.194 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.197+0000 7f6525400640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f652006dda0 con 0x7f652006f2c0 2024-05-28T06:22:17.194 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.197+0000 7f6525400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f652006e080 con 0x7f652006ee40 2024-05-28T06:22:17.194 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.198+0000 7f6524a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f652006f2c0 0x7f65200ad770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:17.194 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.198+0000 7f6524a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f652006f2c0 0x7f65200ad770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:59998/0 (socket says 172.21.15.96:59998) 2024-05-28T06:22:17.194 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.198+0000 7f6524a00640 1 -- 172.21.15.96:0/3708605904 learned_addr learned my addr 172.21.15.96:0/3708605904 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:22:17.194 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.198+0000 7f651f400640 1 --2- 172.21.15.96:0/3708605904 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f652006ee40 0x7f65200ad230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:17.194 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.198+0000 7f6524a00640 1 -- 172.21.15.96:0/3708605904 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f652006ee40 msgr2=0x7f65200ad230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:17.195 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.198+0000 7f6524a00640 1 --2- 172.21.15.96:0/3708605904 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f652006ee40 0x7f65200ad230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:17.195 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.198+0000 7f6524a00640 1 -- 172.21.15.96:0/3708605904 >> v1:172.21.15.123:6789/0 conn(0x7f6520156b30 legacy=0x7f65200af6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:17.195 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.198+0000 7f6524a00640 1 -- 172.21.15.96:0/3708605904 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f652006e360 con 0x7f652006f2c0 2024-05-28T06:22:17.195 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.198+0000 7f651f400640 1 --2- 172.21.15.96:0/3708605904 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f652006ee40 0x7f65200ad230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:22:17.196 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.199+0000 7f6524a00640 1 --2- 172.21.15.96:0/3708605904 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f652006f2c0 0x7f65200ad770 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7f6518009e50 tx=0x7f651802a630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c85a477d9cb140c7 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:17.196 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.200+0000 7f651ea00640 1 -- 172.21.15.96:0/3708605904 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6518036070 con 0x7f652006f2c0 2024-05-28T06:22:17.196 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.200+0000 7f651ea00640 1 -- 172.21.15.96:0/3708605904 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6518031600 con 0x7f652006f2c0 2024-05-28T06:22:17.196 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.200+0000 7f651ea00640 1 -- 172.21.15.96:0/3708605904 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65180319d0 con 0x7f652006f2c0 2024-05-28T06:22:17.196 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.200+0000 7f6525400640 1 -- 172.21.15.96:0/3708605904 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f652006f2c0 msgr2=0x7f65200ad770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:17.197 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.200+0000 7f6525400640 1 --2- 172.21.15.96:0/3708605904 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f652006f2c0 0x7f65200ad770 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7f6518009e50 tx=0x7f651802a630 comp rx=0 tx=0).stop 2024-05-28T06:22:17.197 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.200+0000 7f6525400640 1 -- 172.21.15.96:0/3708605904 shutdown_connections 2024-05-28T06:22:17.197 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.200+0000 7f6525400640 1 --2- 172.21.15.96:0/3708605904 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f652006f2c0 0x7f65200ad770 unknown :-1 s=CLOSED pgs=1797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:17.197 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.200+0000 7f6525400640 1 --2- 172.21.15.96:0/3708605904 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f652006ee40 0x7f65200ad230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:17.197 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.200+0000 7f6525400640 1 -- 172.21.15.96:0/3708605904 >> 172.21.15.96:0/3708605904 conn(0x7f65200b1fc0 msgr2=0x7f65200b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:17.197 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.200+0000 7f6525400640 1 -- 172.21.15.96:0/3708605904 shutdown_connections 2024-05-28T06:22:17.197 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.200+0000 7f6525400640 1 -- 172.21.15.96:0/3708605904 wait complete. 2024-05-28T06:22:17.197 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.202+0000 7f6525400640 1 Processor -- start 2024-05-28T06:22:17.198 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.202+0000 7f6525400640 1 -- start start 2024-05-28T06:22:17.199 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.203+0000 7f6525400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f652006ee40 0x7f65201452f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:17.200 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.203+0000 7f6525400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f652006f2c0 0x7f6520150ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:17.200 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.203+0000 7f6525400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6520156b30 0x7f65201510e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:17.200 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.203+0000 7f6525400640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f65200b2db0 con 0x7f6520156b30 2024-05-28T06:22:17.200 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.203+0000 7f6525400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f65200b2c30 con 0x7f652006ee40 2024-05-28T06:22:17.200 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.203+0000 7f6525400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f65200b32c0 con 0x7f652006f2c0 2024-05-28T06:22:17.200 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.203+0000 7f6524a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6520156b30 0x7f65201510e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:17.200 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.203+0000 7f6524a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6520156b30 0x7f65201510e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:60002/0 (socket says 172.21.15.96:60002) 2024-05-28T06:22:17.200 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.203+0000 7f6524a00640 1 -- 172.21.15.96:0/875846827 learned_addr learned my addr 172.21.15.96:0/875846827 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:22:17.200 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.204+0000 7f651fe00640 1 --2- 172.21.15.96:0/875846827 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f652006ee40 0x7f65201452f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:17.200 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.204+0000 7f6524a00640 1 -- 172.21.15.96:0/875846827 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f652006f2c0 msgr2=0x7f6520150ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:17.201 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.204+0000 7f6524a00640 1 --2- 172.21.15.96:0/875846827 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f652006f2c0 0x7f6520150ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:17.201 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.204+0000 7f6524a00640 1 -- 172.21.15.96:0/875846827 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f652006ee40 msgr2=0x7f65201452f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:17.201 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.204+0000 7f6524a00640 1 --2- 172.21.15.96:0/875846827 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f652006ee40 0x7f65201452f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:17.201 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.204+0000 7f6524a00640 1 -- 172.21.15.96:0/875846827 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65200cf930 con 0x7f6520156b30 2024-05-28T06:22:17.201 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.204+0000 7f6524a00640 1 --2- 172.21.15.96:0/875846827 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6520156b30 0x7f65201510e0 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7f6518009f70 tx=0x7f6518004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:17.202 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.204+0000 7f6517e00640 1 -- 172.21.15.96:0/875846827 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6518036050 con 0x7f6520156b30 2024-05-28T06:22:17.202 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.204+0000 7f6517e00640 1 -- 172.21.15.96:0/875846827 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6518042070 con 0x7f6520156b30 2024-05-28T06:22:17.202 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.204+0000 7f6525400640 1 -- 172.21.15.96:0/875846827 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f652015f1a0 con 0x7f6520156b30 2024-05-28T06:22:17.202 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.205+0000 7f6517e00640 1 -- 172.21.15.96:0/875846827 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6518009510 con 0x7f6520156b30 2024-05-28T06:22:17.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.205+0000 7f6525400640 1 -- 172.21.15.96:0/875846827 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f652015f440 con 0x7f6520156b30 2024-05-28T06:22:17.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.206+0000 7f6525400640 1 -- 172.21.15.96:0/875846827 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65200689e0 con 0x7f6520156b30 2024-05-28T06:22:17.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.207+0000 7f6517e00640 1 -- 172.21.15.96:0/875846827 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 37) ==== 93182+0+0 (secure 0 0 0) 0x7f6518007c70 con 0x7f6520156b30 2024-05-28T06:22:17.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.207+0000 7f6517e00640 1 --2- 172.21.15.96:0/875846827 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f650006e090 0x7f6500070550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:17.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.208+0000 7f6517e00640 1 -- 172.21.15.96:0/875846827 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 5765+0+0 (secure 0 0 0) 0x7f65180ab950 con 0x7f6520156b30 2024-05-28T06:22:17.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.208+0000 7f651fe00640 1 --2- 172.21.15.96:0/875846827 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f650006e090 0x7f6500070550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:17.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.209+0000 7f651fe00640 1 --2- 172.21.15.96:0/875846827 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f650006e090 0x7f6500070550 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f6510006fd0 tx=0x7f6510008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:17.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.212+0000 7f6517e00640 1 -- 172.21.15.96:0/875846827 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6518045360 con 0x7f6520156b30 2024-05-28T06:22:17.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.411+0000 7f6525400640 1 -- 172.21.15.96:0/875846827 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f65201553c0 con 0x7f6520156b30 2024-05-28T06:22:17.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.413+0000 7f6517e00640 1 -- 172.21.15.96:0/875846827 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v114) ==== 74+0+16740 (secure 0 0 0) 0x7f651807b7e0 con 0x7f6520156b30 2024-05-28T06:22:17.409 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:22:17.409 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":114,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:22:16.424853+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-05-28T06:22:14.138685+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"114","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-05-28T06:22:15.394361+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"114","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":13,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:22:17.412 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.416+0000 7f6525400640 1 -- 172.21.15.96:0/875846827 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f650006e090 msgr2=0x7f6500070550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:17.412 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.416+0000 7f6525400640 1 --2- 172.21.15.96:0/875846827 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f650006e090 0x7f6500070550 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f6510006fd0 tx=0x7f6510008040 comp rx=0 tx=0).stop 2024-05-28T06:22:17.412 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.416+0000 7f6525400640 1 -- 172.21.15.96:0/875846827 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6520156b30 msgr2=0x7f65201510e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:17.412 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.416+0000 7f6525400640 1 --2- 172.21.15.96:0/875846827 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6520156b30 0x7f65201510e0 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7f6518009f70 tx=0x7f6518004a40 comp rx=0 tx=0).stop 2024-05-28T06:22:17.412 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.416+0000 7f6525400640 1 -- 172.21.15.96:0/875846827 shutdown_connections 2024-05-28T06:22:17.412 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.416+0000 7f6525400640 1 --2- 172.21.15.96:0/875846827 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f650006e090 0x7f6500070550 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:17.412 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.416+0000 7f6525400640 1 --2- 172.21.15.96:0/875846827 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6520156b30 0x7f65201510e0 unknown :-1 s=CLOSED pgs=1798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:17.412 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.416+0000 7f6525400640 1 --2- 172.21.15.96:0/875846827 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f652006f2c0 0x7f6520150ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:17.412 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.416+0000 7f6525400640 1 --2- 172.21.15.96:0/875846827 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f652006ee40 0x7f65201452f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:17.412 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.416+0000 7f6525400640 1 -- 172.21.15.96:0/875846827 >> 172.21.15.96:0/875846827 conn(0x7f65200b1fc0 msgr2=0x7f6520154120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:17.412 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.416+0000 7f6525400640 1 -- 172.21.15.96:0/875846827 shutdown_connections 2024-05-28T06:22:17.412 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.416+0000 7f6525400640 1 -- 172.21.15.96:0/875846827 wait complete. 2024-05-28T06:22:17.430 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:22:17.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.630+0000 7fb08de00640 1 Processor -- start 2024-05-28T06:22:17.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.630+0000 7fb08de00640 1 -- start start 2024-05-28T06:22:17.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.632+0000 7fb08de00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fb0880d04a0 0x7fb0880d0930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:17.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.632+0000 7fb08de00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb0880ad130 0x7fb0880ad580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:17.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.632+0000 7fb08de00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb08806ed10 con 0x7fb08806f860 2024-05-28T06:22:17.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.632+0000 7fb08de00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fb08815cb00 con 0x7fb0880d04a0 2024-05-28T06:22:17.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.632+0000 7fb08de00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fb0880b3cc0 con 0x7fb0880ad130 2024-05-28T06:22:17.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.632+0000 7fb07fe00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fb0880d04a0 0x7fb0880d0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:17.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.633+0000 7fb07fe00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fb0880d04a0 0x7fb0880d0930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.96:47588/0 (socket says 172.21.15.96:47588) 2024-05-28T06:22:17.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.633+0000 7fb07fe00640 1 -- 172.21.15.96:0/3411412539 learned_addr learned my addr 172.21.15.96:0/3411412539 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:22:17.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.633+0000 7fb087400640 1 --2- 172.21.15.96:0/3411412539 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb0880ad130 0x7fb0880ad580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:17.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.633+0000 7fb07fe00640 1 -- 172.21.15.96:0/3411412539 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb0880ad130 msgr2=0x7fb0880ad580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:17.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.633+0000 7fb07fe00640 1 --2- 172.21.15.96:0/3411412539 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb0880ad130 0x7fb0880ad580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:17.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.633+0000 7fb07fe00640 1 -- 172.21.15.96:0/3411412539 >> v1:172.21.15.123:6789/0 conn(0x7fb08806f860 legacy=0x7fb0880b3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:17.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.633+0000 7fb07fe00640 1 -- 172.21.15.96:0/3411412539 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0880b3fa0 con 0x7fb0880d04a0 2024-05-28T06:22:17.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.635+0000 7fb07fe00640 1 --2- 172.21.15.96:0/3411412539 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb0880d04a0 0x7fb0880d0930 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fb070009870 tx=0x7fb07002a250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2867054c657e5a5b server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:17.631 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.635+0000 7fb086000640 1 -- 172.21.15.96:0/3411412539 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb070037070 con 0x7fb0880d04a0 2024-05-28T06:22:17.631 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.635+0000 7fb086000640 1 -- 172.21.15.96:0/3411412539 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb070032440 con 0x7fb0880d04a0 2024-05-28T06:22:17.631 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.635+0000 7fb086000640 1 -- 172.21.15.96:0/3411412539 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb070032810 con 0x7fb0880d04a0 2024-05-28T06:22:17.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.636+0000 7fb08de00640 1 -- 172.21.15.96:0/3411412539 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb0880d04a0 msgr2=0x7fb0880d0930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:17.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.636+0000 7fb08de00640 1 --2- 172.21.15.96:0/3411412539 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb0880d04a0 0x7fb0880d0930 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fb070009870 tx=0x7fb07002a250 comp rx=0 tx=0).stop 2024-05-28T06:22:17.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.636+0000 7fb08de00640 1 -- 172.21.15.96:0/3411412539 shutdown_connections 2024-05-28T06:22:17.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.636+0000 7fb08de00640 1 --2- 172.21.15.96:0/3411412539 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb0880ad130 0x7fb0880ad580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:17.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.636+0000 7fb08de00640 1 --2- 172.21.15.96:0/3411412539 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb0880d04a0 0x7fb0880d0930 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:17.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.636+0000 7fb08de00640 1 -- 172.21.15.96:0/3411412539 >> 172.21.15.96:0/3411412539 conn(0x7fb08806bae0 msgr2=0x7fb08806bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:17.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.636+0000 7fb08de00640 1 -- 172.21.15.96:0/3411412539 shutdown_connections 2024-05-28T06:22:17.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.636+0000 7fb08de00640 1 -- 172.21.15.96:0/3411412539 wait complete. 2024-05-28T06:22:17.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.638+0000 7fb08de00640 1 Processor -- start 2024-05-28T06:22:17.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.638+0000 7fb08de00640 1 -- start start 2024-05-28T06:22:17.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.640+0000 7fb08de00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb08806f860 0x7fb0880cde30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:17.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.640+0000 7fb08de00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb0880ad130 0x7fb0880cf5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:17.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.640+0000 7fb08de00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb0880d04a0 0x7fb08815e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:17.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.640+0000 7fb08de00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb0880b4840 con 0x7fb0880d04a0 2024-05-28T06:22:17.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.640+0000 7fb08de00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fb0880b4560 con 0x7fb0880ad130 2024-05-28T06:22:17.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.640+0000 7fb08de00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fb0880b3860 con 0x7fb08806f860 2024-05-28T06:22:17.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.640+0000 7fb087400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb0880d04a0 0x7fb08815e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:17.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.640+0000 7fb087400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb0880d04a0 0x7fb08815e510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:60004/0 (socket says 172.21.15.96:60004) 2024-05-28T06:22:17.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.640+0000 7fb087400640 1 -- 172.21.15.96:0/295578416 learned_addr learned my addr 172.21.15.96:0/295578416 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:22:17.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.640+0000 7fb086a00640 1 --2- 172.21.15.96:0/295578416 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb08806f860 0x7fb0880cde30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:17.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.640+0000 7fb087400640 1 -- 172.21.15.96:0/295578416 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb08806f860 msgr2=0x7fb0880cde30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:17.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.640+0000 7fb087400640 1 --2- 172.21.15.96:0/295578416 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb08806f860 0x7fb0880cde30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:17.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.640+0000 7fb087400640 1 -- 172.21.15.96:0/295578416 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb0880ad130 msgr2=0x7fb0880cf5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:17.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.640+0000 7fb087400640 1 --2- 172.21.15.96:0/295578416 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb0880ad130 0x7fb0880cf5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:17.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.640+0000 7fb087400640 1 -- 172.21.15.96:0/295578416 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0881597a0 con 0x7fb0880d04a0 2024-05-28T06:22:17.638 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.641+0000 7fb087400640 1 --2- 172.21.15.96:0/295578416 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb0880d04a0 0x7fb08815e510 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7fb08000efc0 tx=0x7fb08000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:17.638 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.641+0000 7fb07f400640 1 -- 172.21.15.96:0/295578416 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb080018020 con 0x7fb0880d04a0 2024-05-28T06:22:17.638 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.641+0000 7fb07f400640 1 -- 172.21.15.96:0/295578416 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb080014480 con 0x7fb0880d04a0 2024-05-28T06:22:17.638 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.641+0000 7fb07f400640 1 -- 172.21.15.96:0/295578416 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb080014870 con 0x7fb0880d04a0 2024-05-28T06:22:17.638 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.641+0000 7fb08de00640 1 -- 172.21.15.96:0/295578416 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0880ce530 con 0x7fb0880d04a0 2024-05-28T06:22:17.638 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.641+0000 7fb08de00640 1 -- 172.21.15.96:0/295578416 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0880ceaf0 con 0x7fb0880d04a0 2024-05-28T06:22:17.638 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.642+0000 7fb08de00640 1 -- 172.21.15.96:0/295578416 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0880689e0 con 0x7fb0880d04a0 2024-05-28T06:22:17.638 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.643+0000 7fb07f400640 1 -- 172.21.15.96:0/295578416 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 37) ==== 93182+0+0 (secure 0 0 0) 0x7fb080005d10 con 0x7fb0880d04a0 2024-05-28T06:22:17.639 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.644+0000 7fb07f400640 1 --2- 172.21.15.96:0/295578416 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb06406e090 0x7fb064070550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:17.640 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.644+0000 7fb07f400640 1 -- 172.21.15.96:0/295578416 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 5765+0+0 (secure 0 0 0) 0x7fb08008b3f0 con 0x7fb0880d04a0 2024-05-28T06:22:17.640 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.645+0000 7fb086a00640 1 --2- 172.21.15.96:0/295578416 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb06406e090 0x7fb064070550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:17.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.646+0000 7fb086a00640 1 --2- 172.21.15.96:0/295578416 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb06406e090 0x7fb064070550 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fb0780015a0 tx=0x7fb078009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:17.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.648+0000 7fb07f400640 1 -- 172.21.15.96:0/295578416 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb080022070 con 0x7fb0880d04a0 2024-05-28T06:22:17.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.849+0000 7fb08de00640 1 -- 172.21.15.96:0/295578416 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb08806de30 con 0x7fb0880d04a0 2024-05-28T06:22:17.847 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.852+0000 7fb07f400640 1 -- 172.21.15.96:0/295578416 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v114) ==== 74+0+16740 (secure 0 0 0) 0x7fb080010040 con 0x7fb0880d04a0 2024-05-28T06:22:17.848 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:22:17.848 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":114,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:22:16.424853+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-05-28T06:22:14.138685+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"114","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-05-28T06:22:15.394361+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"114","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":13,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:22:17.850 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.855+0000 7fb08de00640 1 -- 172.21.15.96:0/295578416 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb06406e090 msgr2=0x7fb064070550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:17.851 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.855+0000 7fb08de00640 1 --2- 172.21.15.96:0/295578416 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb06406e090 0x7fb064070550 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fb0780015a0 tx=0x7fb078009040 comp rx=0 tx=0).stop 2024-05-28T06:22:17.851 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.855+0000 7fb08de00640 1 -- 172.21.15.96:0/295578416 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb0880d04a0 msgr2=0x7fb08815e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:17.851 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.855+0000 7fb08de00640 1 --2- 172.21.15.96:0/295578416 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb0880d04a0 0x7fb08815e510 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7fb08000efc0 tx=0x7fb08000c490 comp rx=0 tx=0).stop 2024-05-28T06:22:17.851 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.855+0000 7fb08de00640 1 -- 172.21.15.96:0/295578416 shutdown_connections 2024-05-28T06:22:17.851 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.855+0000 7fb08de00640 1 --2- 172.21.15.96:0/295578416 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb06406e090 0x7fb064070550 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:17.851 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.855+0000 7fb08de00640 1 --2- 172.21.15.96:0/295578416 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb0880d04a0 0x7fb08815e510 unknown :-1 s=CLOSED pgs=1799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:17.851 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.855+0000 7fb08de00640 1 --2- 172.21.15.96:0/295578416 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb0880ad130 0x7fb0880cf5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:17.851 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.855+0000 7fb08de00640 1 --2- 172.21.15.96:0/295578416 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb08806f860 0x7fb0880cde30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:17.851 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.855+0000 7fb08de00640 1 -- 172.21.15.96:0/295578416 >> 172.21.15.96:0/295578416 conn(0x7fb08806bae0 msgr2=0x7fb088072fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:17.851 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.855+0000 7fb08de00640 1 -- 172.21.15.96:0/295578416 shutdown_connections 2024-05-28T06:22:17.851 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:17.855+0000 7fb08de00640 1 -- 172.21.15.96:0/295578416 wait complete. 2024-05-28T06:22:17.872 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-05-28T06:22:18.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:18.082+0000 7efd32400640 1 Processor -- start 2024-05-28T06:22:18.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:18.082+0000 7efd32400640 1 -- start start 2024-05-28T06:22:18.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:18.084+0000 7efd32400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7efd2c153510 0x7efd2c0b4d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:18.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:18.084+0000 7efd32400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7efd2c15d270 0x7efd2c0ccff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:18.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:18.084+0000 7efd32400640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd2c15d900 con 0x7efd2c0ccb90 2024-05-28T06:22:18.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:18.084+0000 7efd32400640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7efd2c1538f0 con 0x7efd2c15d270 2024-05-28T06:22:18.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:18.084+0000 7efd32400640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7efd2c0cd530 con 0x7efd2c153510 2024-05-28T06:22:18.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:18.084+0000 7efd31a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7efd2c15d270 0x7efd2c0ccff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:18.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:18.084+0000 7efd31a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7efd2c15d270 0x7efd2c0ccff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:60014/0 (socket says 172.21.15.96:60014) 2024-05-28T06:22:18.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:18.084+0000 7efd31a00640 1 -- 172.21.15.96:0/3653359270 learned_addr learned my addr 172.21.15.96:0/3653359270 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:22:18.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:18.084+0000 7efd2be00640 1 --1- 172.21.15.96:0/3653359270 >> v1:172.21.15.96:6789/0 conn(0x7efd2c0ccb90 0x7efd2c15cb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:33164/0 (socket says 172.21.15.96:33164) 2024-05-28T06:22:18.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:18.085+0000 7efd31000640 1 --2- 172.21.15.96:0/3653359270 >> v2:172.21.15.123:3300/0 conn(0x7efd2c153510 0x7efd2c0b4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:18.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:18.085+0000 7efd31a00640 1 -- 172.21.15.96:0/3653359270 >> v2:172.21.15.123:3300/0 conn(0x7efd2c153510 msgr2=0x7efd2c0b4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:18.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:18.085+0000 7efd31a00640 1 --2- 172.21.15.96:0/3653359270 >> v2:172.21.15.123:3300/0 conn(0x7efd2c153510 0x7efd2c0b4d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:18.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:18.085+0000 7efd31a00640 1 -- 172.21.15.96:0/3653359270 >> v1:172.21.15.96:6789/0 conn(0x7efd2c0ccb90 legacy=0x7efd2c15cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:18.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:18.085+0000 7efd31a00640 1 -- 172.21.15.96:0/3653359270 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd2c0b3f90 con 0x7efd2c15d270 2024-05-28T06:22:18.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:18.085+0000 7efd31000640 1 --2- 172.21.15.96:0/3653359270 >> v2:172.21.15.123:3300/0 conn(0x7efd2c153510 0x7efd2c0b4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:22:18.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:18.086+0000 7efd31a00640 1 --2- 172.21.15.96:0/3653359270 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efd2c15d270 0x7efd2c0ccff0 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7efd2c0b3970 tx=0x7efd2402a6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a05b4dcd133715ef server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:18.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:18.086+0000 7efd2b400640 1 -- 172.21.15.96:0/3653359270 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd24036070 con 0x7efd2c15d270 2024-05-28T06:22:18.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:18.086+0000 7efd2b400640 1 -- 172.21.15.96:0/3653359270 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd24031600 con 0x7efd2c15d270 2024-05-28T06:22:18.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:18.086+0000 7efd2b400640 1 -- 172.21.15.96:0/3653359270 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd240319d0 con 0x7efd2c15d270 2024-05-28T06:22:18.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:18.087+0000 7efd32400640 1 -- 172.21.15.96:0/3653359270 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efd2c15d270 msgr2=0x7efd2c0ccff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:18.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:18.087+0000 7efd32400640 1 --2- 172.21.15.96:0/3653359270 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efd2c15d270 0x7efd2c0ccff0 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7efd2c0b3970 tx=0x7efd2402a6b0 comp rx=0 tx=0).stop 2024-05-28T06:22:18.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:18.087+0000 7efd32400640 1 -- 172.21.15.96:0/3653359270 shutdown_connections 2024-05-28T06:22:18.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:18.087+0000 7efd32400640 1 --2- 172.21.15.96:0/3653359270 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efd2c15d270 0x7efd2c0ccff0 unknown :-1 s=CLOSED pgs=1801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:18.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:18.087+0000 7efd32400640 1 --2- 172.21.15.96:0/3653359270 >> v2:172.21.15.123:3300/0 conn(0x7efd2c153510 0x7efd2c0b4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:18.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:18.087+0000 7efd32400640 1 -- 172.21.15.96:0/3653359270 >> 172.21.15.96:0/3653359270 conn(0x7efd2c06b900 msgr2=0x7efd2c06bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:18.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:18.087+0000 7efd32400640 1 -- 172.21.15.96:0/3653359270 shutdown_connections 2024-05-28T06:22:18.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:18.087+0000 7efd32400640 1 -- 172.21.15.96:0/3653359270 wait complete. 2024-05-28T06:22:18.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:18.089+0000 7efd32400640 1 Processor -- start 2024-05-28T06:22:18.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:18.089+0000 7efd32400640 1 -- start start 2024-05-28T06:22:18.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:18.090+0000 7efd32400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efd2c0ccb90 0x7efd2c072fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:18.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:18.090+0000 7efd32400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd2c153510 0x7efd2c15e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:18.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:18.090+0000 7efd32400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efd2c15d270 0x7efd2c1465e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:18.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:18.090+0000 7efd32400640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7efd2c15d6a0 con 0x7efd2c0ccb90 2024-05-28T06:22:18.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:18.090+0000 7efd32400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7efd2c0b48c0 con 0x7efd2c153510 2024-05-28T06:22:18.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:18.090+0000 7efd32400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7efd2c0b4550 con 0x7efd2c15d270 2024-05-28T06:22:18.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:18.090+0000 7efd31000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efd2c0ccb90 0x7efd2c072fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:18.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:18.090+0000 7efd31000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efd2c0ccb90 0x7efd2c072fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:60030/0 (socket says 172.21.15.96:60030) 2024-05-28T06:22:18.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:18.090+0000 7efd31000640 1 -- 172.21.15.96:0/601596297 learned_addr learned my addr 172.21.15.96:0/601596297 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:22:18.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:18.090+0000 7efd2be00640 1 --2- 172.21.15.96:0/601596297 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd2c153510 0x7efd2c15e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:18.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:18.090+0000 7efd31000640 1 -- 172.21.15.96:0/601596297 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efd2c15d270 msgr2=0x7efd2c1465e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:18.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:18.090+0000 7efd31000640 1 --2- 172.21.15.96:0/601596297 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efd2c15d270 0x7efd2c1465e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:18.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:18.090+0000 7efd31000640 1 -- 172.21.15.96:0/601596297 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd2c153510 msgr2=0x7efd2c15e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:18.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:18.090+0000 7efd31000640 1 --2- 172.21.15.96:0/601596297 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd2c153510 0x7efd2c15e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:18.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:18.090+0000 7efd31000640 1 -- 172.21.15.96:0/601596297 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd2c146b20 con 0x7efd2c0ccb90 2024-05-28T06:22:18.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:18.091+0000 7efd2be00640 1 --2- 172.21.15.96:0/601596297 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd2c153510 0x7efd2c15e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:22:18.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:18.091+0000 7efd31000640 1 --2- 172.21.15.96:0/601596297 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efd2c0ccb90 0x7efd2c072fd0 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7efd2000c820 tx=0x7efd2000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:18.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:18.091+0000 7efd28c00640 1 -- 172.21.15.96:0/601596297 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd20015070 con 0x7efd2c0ccb90 2024-05-28T06:22:18.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:18.091+0000 7efd28c00640 1 -- 172.21.15.96:0/601596297 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd20007a30 con 0x7efd2c0ccb90 2024-05-28T06:22:18.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:18.091+0000 7efd32400640 1 -- 172.21.15.96:0/601596297 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd2c15ed10 con 0x7efd2c0ccb90 2024-05-28T06:22:18.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:18.091+0000 7efd28c00640 1 -- 172.21.15.96:0/601596297 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd20007e00 con 0x7efd2c0ccb90 2024-05-28T06:22:18.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:18.091+0000 7efd32400640 1 -- 172.21.15.96:0/601596297 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd2c15f2d0 con 0x7efd2c0ccb90 2024-05-28T06:22:18.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:18.093+0000 7efd32400640 1 -- 172.21.15.96:0/601596297 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd2c0689e0 con 0x7efd2c0ccb90 2024-05-28T06:22:18.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:18.094+0000 7efd28c00640 1 -- 172.21.15.96:0/601596297 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 37) ==== 93182+0+0 (secure 0 0 0) 0x7efd20018070 con 0x7efd2c0ccb90 2024-05-28T06:22:18.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:18.094+0000 7efd28c00640 1 --2- 172.21.15.96:0/601596297 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7efd0c06e090 0x7efd0c070550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:18.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:18.095+0000 7efd28c00640 1 -- 172.21.15.96:0/601596297 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 5765+0+0 (secure 0 0 0) 0x7efd2008a8d0 con 0x7efd2c0ccb90 2024-05-28T06:22:18.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:18.095+0000 7efd2be00640 1 --2- 172.21.15.96:0/601596297 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7efd0c06e090 0x7efd0c070550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:18.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:18.096+0000 7efd2be00640 1 --2- 172.21.15.96:0/601596297 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7efd0c06e090 0x7efd0c070550 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7efd2c1f7c50 tx=0x7efd18009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:18.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:18.099+0000 7efd28c00640 1 -- 172.21.15.96:0/601596297 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7efd2c146b20 con 0x7efd2c0ccb90 2024-05-28T06:22:18.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:18.344+0000 7efd32400640 1 -- 172.21.15.96:0/601596297 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7efd2c0af380 con 0x7efd2c0ccb90 2024-05-28T06:22:19.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:19.029+0000 7efd28c00640 1 -- 172.21.15.96:0/601596297 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v71) ==== 118+0+0 (secure 0 0 0) 0x7efd2005a770 con 0x7efd2c0ccb90 2024-05-28T06:22:19.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:19.033+0000 7efd32400640 1 -- 172.21.15.96:0/601596297 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7efd0c06e090 msgr2=0x7efd0c070550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:19.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:19.033+0000 7efd32400640 1 --2- 172.21.15.96:0/601596297 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7efd0c06e090 0x7efd0c070550 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7efd2c1f7c50 tx=0x7efd18009040 comp rx=0 tx=0).stop 2024-05-28T06:22:19.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:19.033+0000 7efd32400640 1 -- 172.21.15.96:0/601596297 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efd2c0ccb90 msgr2=0x7efd2c072fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:19.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:19.033+0000 7efd32400640 1 --2- 172.21.15.96:0/601596297 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efd2c0ccb90 0x7efd2c072fd0 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7efd2000c820 tx=0x7efd2000cc20 comp rx=0 tx=0).stop 2024-05-28T06:22:19.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:19.033+0000 7efd32400640 1 -- 172.21.15.96:0/601596297 shutdown_connections 2024-05-28T06:22:19.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:19.033+0000 7efd32400640 1 --2- 172.21.15.96:0/601596297 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7efd0c06e090 0x7efd0c070550 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:19.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:19.033+0000 7efd32400640 1 --2- 172.21.15.96:0/601596297 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efd2c15d270 0x7efd2c1465e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:19.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:19.033+0000 7efd32400640 1 --2- 172.21.15.96:0/601596297 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd2c153510 0x7efd2c15e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:19.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:19.033+0000 7efd32400640 1 --2- 172.21.15.96:0/601596297 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efd2c0ccb90 0x7efd2c072fd0 unknown :-1 s=CLOSED pgs=1802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:19.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:19.033+0000 7efd32400640 1 -- 172.21.15.96:0/601596297 >> 172.21.15.96:0/601596297 conn(0x7efd2c06b900 msgr2=0x7efd2c15b430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:19.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:19.034+0000 7efd32400640 1 -- 172.21.15.96:0/601596297 shutdown_connections 2024-05-28T06:22:19.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:19.034+0000 7efd32400640 1 -- 172.21.15.96:0/601596297 wait complete. 2024-05-28T06:22:19.052 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-05-28T06:22:19.053 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-05-28T06:22:19.258 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:19.262+0000 7f66f1e00640 1 Processor -- start 2024-05-28T06:22:19.258 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:19.262+0000 7f66f1e00640 1 -- start start 2024-05-28T06:22:19.260 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:19.264+0000 7f66f1e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f66ec162ac0 0x7f66ec0d0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:19.260 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:19.264+0000 7f66f1e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f66ec163ac0 0x7f66ec0d0b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:19.260 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:19.264+0000 7f66f1e00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66ec1640e0 con 0x7f66ec0b4120 2024-05-28T06:22:19.261 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:19.264+0000 7f66f1e00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f66ec0b3c90 con 0x7f66ec162ac0 2024-05-28T06:22:19.261 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:19.264+0000 7f66f1e00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f66ec0b3360 con 0x7f66ec163ac0 2024-05-28T06:22:19.261 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:19.264+0000 7f66eaa00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f66ec162ac0 0x7f66ec0d0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:19.261 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:19.264+0000 7f66eaa00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f66ec162ac0 0x7f66ec0d0640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:60134/0 (socket says 172.21.15.96:60134) 2024-05-28T06:22:19.261 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:19.264+0000 7f66eaa00640 1 -- 172.21.15.96:0/3022645705 learned_addr learned my addr 172.21.15.96:0/3022645705 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:22:19.261 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:19.265+0000 7f66eb400640 1 --2- 172.21.15.96:0/3022645705 >> v2:172.21.15.123:3300/0 conn(0x7f66ec163ac0 0x7f66ec0d0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:19.261 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:19.265+0000 7f66eaa00640 1 -- 172.21.15.96:0/3022645705 >> v2:172.21.15.123:3300/0 conn(0x7f66ec163ac0 msgr2=0x7f66ec0d0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:19.261 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:19.265+0000 7f66eaa00640 1 --2- 172.21.15.96:0/3022645705 >> v2:172.21.15.123:3300/0 conn(0x7f66ec163ac0 0x7f66ec0d0b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:19.261 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:19.265+0000 7f66eaa00640 1 -- 172.21.15.96:0/3022645705 >> v1:172.21.15.123:6789/0 conn(0x7f66ec0b4120 legacy=0x7f66ec1633b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:19.261 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:19.265+0000 7f66eaa00640 1 -- 172.21.15.96:0/3022645705 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66ec0b3640 con 0x7f66ec162ac0 2024-05-28T06:22:19.263 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:19.266+0000 7f66eaa00640 1 --2- 172.21.15.96:0/3022645705 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f66ec162ac0 0x7f66ec0d0640 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7f66ec0740c0 tx=0x7f66dc02a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0d0c1d74f8cc5e3 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:19.263 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:19.266+0000 7f66ea000640 1 -- 172.21.15.96:0/3022645705 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66dc037070 con 0x7f66ec162ac0 2024-05-28T06:22:19.263 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:19.266+0000 7f66ea000640 1 -- 172.21.15.96:0/3022645705 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66dc032440 con 0x7f66ec162ac0 2024-05-28T06:22:19.263 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:19.266+0000 7f66ea000640 1 -- 172.21.15.96:0/3022645705 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66dc032810 con 0x7f66ec162ac0 2024-05-28T06:22:19.263 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:19.267+0000 7f66f1e00640 1 -- 172.21.15.96:0/3022645705 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f66ec162ac0 msgr2=0x7f66ec0d0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:19.263 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:19.267+0000 7f66f1e00640 1 --2- 172.21.15.96:0/3022645705 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f66ec162ac0 0x7f66ec0d0640 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7f66ec0740c0 tx=0x7f66dc02a230 comp rx=0 tx=0).stop 2024-05-28T06:22:19.263 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:19.267+0000 7f66f1e00640 1 -- 172.21.15.96:0/3022645705 shutdown_connections 2024-05-28T06:22:19.263 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:19.267+0000 7f66f1e00640 1 --2- 172.21.15.96:0/3022645705 >> v2:172.21.15.123:3300/0 conn(0x7f66ec163ac0 0x7f66ec0d0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:19.263 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:19.267+0000 7f66f1e00640 1 --2- 172.21.15.96:0/3022645705 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f66ec162ac0 0x7f66ec0d0640 unknown :-1 s=CLOSED pgs=1803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:19.264 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:19.267+0000 7f66f1e00640 1 -- 172.21.15.96:0/3022645705 >> 172.21.15.96:0/3022645705 conn(0x7f66ec06bd20 msgr2=0x7f66ec06c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:19.264 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:19.267+0000 7f66f1e00640 1 -- 172.21.15.96:0/3022645705 shutdown_connections 2024-05-28T06:22:19.264 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:19.267+0000 7f66f1e00640 1 -- 172.21.15.96:0/3022645705 wait complete. 2024-05-28T06:22:19.264 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:19.269+0000 7f66f1e00640 1 Processor -- start 2024-05-28T06:22:19.264 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:19.269+0000 7f66f1e00640 1 -- start start 2024-05-28T06:22:19.265 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:19.270+0000 7f66f1e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66ec0b4120 0x7f66ec15f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:19.266 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:19.270+0000 7f66f1e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f66ec162ac0 0x7f66ec15e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:19.266 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:19.270+0000 7f66f1e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f66ec163ac0 0x7f66ec15a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:19.266 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:19.270+0000 7f66f1e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f66ec163020 con 0x7f66ec163ac0 2024-05-28T06:22:19.266 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:19.270+0000 7f66f1e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f66ec162ea0 con 0x7f66ec0b4120 2024-05-28T06:22:19.266 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:19.270+0000 7f66f1e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f66ec06dbd0 con 0x7f66ec162ac0 2024-05-28T06:22:19.266 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:19.270+0000 7f66eb400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f66ec163ac0 0x7f66ec15a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:19.266 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:19.270+0000 7f66eb400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f66ec163ac0 0x7f66ec15a8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:60140/0 (socket says 172.21.15.96:60140) 2024-05-28T06:22:19.267 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:19.270+0000 7f66eb400640 1 -- 172.21.15.96:0/3261428732 learned_addr learned my addr 172.21.15.96:0/3261428732 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:22:19.269 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:19.271+0000 7f66eaa00640 1 --2- 172.21.15.96:0/3261428732 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66ec0b4120 0x7f66ec15f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:19.269 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:19.271+0000 7f66e3e00640 1 --2- 172.21.15.96:0/3261428732 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f66ec162ac0 0x7f66ec15e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:19.269 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:19.271+0000 7f66eb400640 1 -- 172.21.15.96:0/3261428732 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f66ec162ac0 msgr2=0x7f66ec15e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:19.269 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:19.271+0000 7f66eb400640 1 --2- 172.21.15.96:0/3261428732 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f66ec162ac0 0x7f66ec15e020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:19.269 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:19.271+0000 7f66eb400640 1 -- 172.21.15.96:0/3261428732 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66ec0b4120 msgr2=0x7f66ec15f280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:19.269 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:19.271+0000 7f66eb400640 1 --2- 172.21.15.96:0/3261428732 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66ec0b4120 0x7f66ec15f280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:19.269 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:19.271+0000 7f66eb400640 1 -- 172.21.15.96:0/3261428732 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66ec0cca00 con 0x7f66ec163ac0 2024-05-28T06:22:19.269 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:19.271+0000 7f66eaa00640 1 --2- 172.21.15.96:0/3261428732 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66ec0b4120 0x7f66ec15f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:22:19.270 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:19.271+0000 7f66eb400640 1 --2- 172.21.15.96:0/3261428732 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f66ec163ac0 0x7f66ec15a8f0 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7f66e4007e60 tx=0x7f66e400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:19.270 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:19.271+0000 7f66e3400640 1 -- 172.21.15.96:0/3261428732 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66e4014420 con 0x7f66ec163ac0 2024-05-28T06:22:19.270 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:19.271+0000 7f66e3400640 1 -- 172.21.15.96:0/3261428732 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66e4019070 con 0x7f66ec163ac0 2024-05-28T06:22:19.270 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:19.271+0000 7f66f1e00640 1 -- 172.21.15.96:0/3261428732 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66ec0ccce0 con 0x7f66ec163ac0 2024-05-28T06:22:19.270 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:19.271+0000 7f66e3400640 1 -- 172.21.15.96:0/3261428732 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66e4014820 con 0x7f66ec163ac0 2024-05-28T06:22:19.270 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:19.271+0000 7f66f1e00640 1 -- 172.21.15.96:0/3261428732 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66ec15f7c0 con 0x7f66ec163ac0 2024-05-28T06:22:19.270 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:19.273+0000 7f66f1e00640 1 -- 172.21.15.96:0/3261428732 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66ec0689e0 con 0x7f66ec163ac0 2024-05-28T06:22:19.270 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:19.274+0000 7f66e3400640 1 -- 172.21.15.96:0/3261428732 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 37) ==== 93182+0+0 (secure 0 0 0) 0x7f66e40149c0 con 0x7f66ec163ac0 2024-05-28T06:22:19.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:19.275+0000 7f66e3400640 1 --2- 172.21.15.96:0/3261428732 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f66c806e090 0x7f66c8070550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:19.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:19.275+0000 7f66e3400640 1 -- 172.21.15.96:0/3261428732 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 5765+0+0 (secure 0 0 0) 0x7f66e408b1b0 con 0x7f66ec163ac0 2024-05-28T06:22:19.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:19.276+0000 7f66eaa00640 1 --2- 172.21.15.96:0/3261428732 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f66c806e090 0x7f66c8070550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:19.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:19.277+0000 7f66eaa00640 1 --2- 172.21.15.96:0/3261428732 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f66c806e090 0x7f66c8070550 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f66dc034000 tx=0x7f66dc02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:19.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:19.279+0000 7f66e3400640 1 -- 172.21.15.96:0/3261428732 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f66e405b040 con 0x7f66ec163ac0 2024-05-28T06:22:19.518 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:19.522+0000 7f66f1e00640 1 -- 172.21.15.96:0/3261428732 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f66ec0cd070 con 0x7f66ec163ac0 2024-05-28T06:22:20.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.043+0000 7f66e3400640 1 -- 172.21.15.96:0/3261428732 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v72) ==== 122+0+0 (secure 0 0 0) 0x7f66e405a940 con 0x7f66ec163ac0 2024-05-28T06:22:20.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.046+0000 7f66f1e00640 1 -- 172.21.15.96:0/3261428732 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f66c806e090 msgr2=0x7f66c8070550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:20.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.046+0000 7f66f1e00640 1 --2- 172.21.15.96:0/3261428732 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f66c806e090 0x7f66c8070550 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f66dc034000 tx=0x7f66dc02e000 comp rx=0 tx=0).stop 2024-05-28T06:22:20.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.046+0000 7f66f1e00640 1 -- 172.21.15.96:0/3261428732 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f66ec163ac0 msgr2=0x7f66ec15a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:20.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.046+0000 7f66f1e00640 1 --2- 172.21.15.96:0/3261428732 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f66ec163ac0 0x7f66ec15a8f0 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7f66e4007e60 tx=0x7f66e400e4b0 comp rx=0 tx=0).stop 2024-05-28T06:22:20.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.046+0000 7f66f1e00640 1 -- 172.21.15.96:0/3261428732 shutdown_connections 2024-05-28T06:22:20.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.046+0000 7f66f1e00640 1 --2- 172.21.15.96:0/3261428732 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f66c806e090 0x7f66c8070550 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:20.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.046+0000 7f66f1e00640 1 --2- 172.21.15.96:0/3261428732 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f66ec163ac0 0x7f66ec15a8f0 unknown :-1 s=CLOSED pgs=1804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:20.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.046+0000 7f66f1e00640 1 --2- 172.21.15.96:0/3261428732 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f66ec162ac0 0x7f66ec15e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:20.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.046+0000 7f66f1e00640 1 --2- 172.21.15.96:0/3261428732 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66ec0b4120 0x7f66ec15f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:20.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.046+0000 7f66f1e00640 1 -- 172.21.15.96:0/3261428732 >> 172.21.15.96:0/3261428732 conn(0x7f66ec06bd20 msgr2=0x7f66ec072220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:20.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.047+0000 7f66f1e00640 1 -- 172.21.15.96:0/3261428732 shutdown_connections 2024-05-28T06:22:20.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.047+0000 7f66f1e00640 1 -- 172.21.15.96:0/3261428732 wait complete. 2024-05-28T06:22:20.065 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-05-28T06:22:20.065 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-28T06:22:20.267 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.271+0000 7fd698a00640 1 Processor -- start 2024-05-28T06:22:20.267 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.271+0000 7fd698a00640 1 -- start start 2024-05-28T06:22:20.268 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.273+0000 7fd698a00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fd694162d20 0x7fd694071bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:20.268 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.273+0000 7fd698a00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd69406f590 con 0x7fd694162940 2024-05-28T06:22:20.268 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.273+0000 7fd698a00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd69406f870 con 0x7fd694072100 2024-05-28T06:22:20.269 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.273+0000 7fd698a00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fd694072530 con 0x7fd694162d20 2024-05-28T06:22:20.269 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.273+0000 7fd693400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fd694162940 0x7fd69406eb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:35076/0 (socket says 172.21.15.96:35076) 2024-05-28T06:22:20.269 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.273+0000 7fd693400640 1 -- 172.21.15.96:0/2386035676 learned_addr learned my addr 172.21.15.96:0/2386035676 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:22:20.270 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.273+0000 7fd692a00640 1 -- 172.21.15.96:0/2386035676 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 45816410 0 0) 0x7fd69406f590 con 0x7fd694162940 2024-05-28T06:22:20.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.274+0000 7fd692a00640 1 -- 172.21.15.96:0/2386035676 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd678003520 con 0x7fd694162940 2024-05-28T06:22:20.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.274+0000 7fd68aa00640 1 --2- 172.21.15.96:0/2386035676 >> v2:172.21.15.123:3300/0 conn(0x7fd694162d20 0x7fd694071bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:20.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.274+0000 7fd692a00640 1 -- 172.21.15.96:0/2386035676 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1786117689 0 0) 0x7fd678003520 con 0x7fd694162940 2024-05-28T06:22:20.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.274+0000 7fd692a00640 1 -- 172.21.15.96:0/2386035676 >> v2:172.21.15.123:3300/0 conn(0x7fd694162d20 msgr2=0x7fd694071bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:20.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.274+0000 7fd692a00640 1 --2- 172.21.15.96:0/2386035676 >> v2:172.21.15.123:3300/0 conn(0x7fd694162d20 0x7fd694071bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:20.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.274+0000 7fd692a00640 1 -- 172.21.15.96:0/2386035676 >> v1:172.21.15.123:6789/0 conn(0x7fd694072100 legacy=0x7fd6940adbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:20.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.274+0000 7fd692a00640 1 -- 172.21.15.96:0/2386035676 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6940ad3d0 con 0x7fd694162940 2024-05-28T06:22:20.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.274+0000 7fd692a00640 1 -- 172.21.15.96:0/2386035676 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7fd684001670 con 0x7fd694162940 2024-05-28T06:22:20.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.274+0000 7fd692a00640 1 -- 172.21.15.96:0/2386035676 >> v1:172.21.15.96:6789/0 conn(0x7fd694162940 legacy=0x7fd69406eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:20.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.274+0000 7fd692a00640 1 --2- 172.21.15.96:0/2386035676 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd6780039b0 0x7fd678003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:20.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.274+0000 7fd692a00640 1 --2- 172.21.15.96:0/2386035676 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd678004800 0x7fd678004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:20.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.274+0000 7fd692a00640 1 --2- 172.21.15.96:0/2386035676 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd6780052e0 0x7fd67800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:20.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.274+0000 7fd692a00640 1 -- 172.21.15.96:0/2386035676 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd694072530 con 0x7fd678004800 2024-05-28T06:22:20.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.274+0000 7fd692a00640 1 -- 172.21.15.96:0/2386035676 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd67800c230 con 0x7fd6780052e0 2024-05-28T06:22:20.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.274+0000 7fd692a00640 1 -- 172.21.15.96:0/2386035676 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fd67800c430 con 0x7fd6780039b0 2024-05-28T06:22:20.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.274+0000 7fd68aa00640 1 --2- 172.21.15.96:0/2386035676 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd678004800 0x7fd678004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:20.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.275+0000 7fd693400640 1 --2- 172.21.15.96:0/2386035676 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd6780039b0 0x7fd678003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:20.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.275+0000 7fd693e00640 1 --2- 172.21.15.96:0/2386035676 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd6780052e0 0x7fd67800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:20.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.275+0000 7fd68aa00640 1 -- 172.21.15.96:0/2386035676 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd6780039b0 msgr2=0x7fd678003db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:20.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.275+0000 7fd68aa00640 1 --2- 172.21.15.96:0/2386035676 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd6780039b0 0x7fd678003db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:20.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.275+0000 7fd68aa00640 1 -- 172.21.15.96:0/2386035676 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd6780052e0 msgr2=0x7fd67800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:20.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.275+0000 7fd68aa00640 1 --2- 172.21.15.96:0/2386035676 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd6780052e0 0x7fd67800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:20.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.275+0000 7fd68aa00640 1 -- 172.21.15.96:0/2386035676 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd678003520 con 0x7fd678004800 2024-05-28T06:22:20.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.275+0000 7fd693e00640 1 --2- 172.21.15.96:0/2386035676 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd6780052e0 0x7fd67800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:22:20.273 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.275+0000 7fd693e00640 1 -- 172.21.15.96:0/2386035676 reap_dead start 2024-05-28T06:22:20.274 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.276+0000 7fd68aa00640 1 --2- 172.21.15.96:0/2386035676 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd678004800 0x7fd678004c80 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7fd6940af970 tx=0x7fd678005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd9b515bc98b6ede server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:20.274 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.276+0000 7fd692a00640 1 -- 172.21.15.96:0/2386035676 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd67c030070 con 0x7fd678004800 2024-05-28T06:22:20.274 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.276+0000 7fd692a00640 1 -- 172.21.15.96:0/2386035676 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd67c02ad70 con 0x7fd678004800 2024-05-28T06:22:20.274 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.276+0000 7fd692a00640 1 -- 172.21.15.96:0/2386035676 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd67c00b610 con 0x7fd678004800 2024-05-28T06:22:20.274 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.277+0000 7fd698a00640 1 -- 172.21.15.96:0/2386035676 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd678004800 msgr2=0x7fd678004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:20.274 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.277+0000 7fd698a00640 1 --2- 172.21.15.96:0/2386035676 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd678004800 0x7fd678004c80 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7fd6940af970 tx=0x7fd678005830 comp rx=0 tx=0).stop 2024-05-28T06:22:20.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.277+0000 7fd698a00640 1 -- 172.21.15.96:0/2386035676 shutdown_connections 2024-05-28T06:22:20.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.277+0000 7fd698a00640 1 --2- 172.21.15.96:0/2386035676 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd678004800 0x7fd678004c80 unknown :-1 s=CLOSED pgs=1806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:20.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.277+0000 7fd698a00640 1 -- 172.21.15.96:0/2386035676 >> 172.21.15.96:0/2386035676 conn(0x7fd6940b1fc0 msgr2=0x7fd6940b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:20.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.277+0000 7fd698a00640 1 -- 172.21.15.96:0/2386035676 shutdown_connections 2024-05-28T06:22:20.275 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.277+0000 7fd698a00640 1 -- 172.21.15.96:0/2386035676 wait complete. 2024-05-28T06:22:20.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.279+0000 7fd698a00640 1 Processor -- start 2024-05-28T06:22:20.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.279+0000 7fd698a00640 1 -- start start 2024-05-28T06:22:20.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.280+0000 7fd698a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd678004800 0x7fd6940ce800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:20.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.280+0000 7fd698a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd6940ae520 0x7fd6940ccca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:20.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.280+0000 7fd698a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd6940ae9a0 0x7fd694150230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:20.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.280+0000 7fd698a00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd694071d40 con 0x7fd6940ae9a0 2024-05-28T06:22:20.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.280+0000 7fd698a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd694071bc0 con 0x7fd6940ae520 2024-05-28T06:22:20.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.280+0000 7fd698a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fd694071ec0 con 0x7fd678004800 2024-05-28T06:22:20.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.280+0000 7fd693e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd6940ae9a0 0x7fd694150230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:20.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.280+0000 7fd693e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd6940ae9a0 0x7fd694150230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:60152/0 (socket says 172.21.15.96:60152) 2024-05-28T06:22:20.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.280+0000 7fd693e00640 1 -- 172.21.15.96:0/287765366 learned_addr learned my addr 172.21.15.96:0/287765366 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:22:20.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.281+0000 7fd693e00640 1 -- 172.21.15.96:0/287765366 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd678004800 msgr2=0x7fd6940ce800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:20.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.281+0000 7fd693400640 1 --2- 172.21.15.96:0/287765366 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd678004800 0x7fd6940ce800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:20.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.281+0000 7fd693e00640 1 --2- 172.21.15.96:0/287765366 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd678004800 0x7fd6940ce800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:20.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.281+0000 7fd693e00640 1 -- 172.21.15.96:0/287765366 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd6940ae520 msgr2=0x7fd6940ccca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:20.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.281+0000 7fd693e00640 1 --2- 172.21.15.96:0/287765366 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd6940ae520 0x7fd6940ccca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:20.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.281+0000 7fd693e00640 1 -- 172.21.15.96:0/287765366 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6940cd760 con 0x7fd6940ae9a0 2024-05-28T06:22:20.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.281+0000 7fd693e00640 1 --2- 172.21.15.96:0/287765366 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd6940ae9a0 0x7fd694150230 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7fd6940711d0 tx=0x7fd68c00d920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:20.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.281+0000 7fd68be00640 1 -- 172.21.15.96:0/287765366 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd68c016030 con 0x7fd6940ae9a0 2024-05-28T06:22:20.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.281+0000 7fd68be00640 1 -- 172.21.15.96:0/287765366 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd68c006b50 con 0x7fd6940ae9a0 2024-05-28T06:22:20.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.281+0000 7fd698a00640 1 -- 172.21.15.96:0/287765366 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6940cda40 con 0x7fd6940ae9a0 2024-05-28T06:22:20.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.281+0000 7fd68be00640 1 -- 172.21.15.96:0/287765366 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd68c012600 con 0x7fd6940ae9a0 2024-05-28T06:22:20.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.281+0000 7fd698a00640 1 -- 172.21.15.96:0/287765366 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6940ce2e0 con 0x7fd6940ae9a0 2024-05-28T06:22:20.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.283+0000 7fd698a00640 1 -- 172.21.15.96:0/287765366 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6940cde30 con 0x7fd6940ae9a0 2024-05-28T06:22:20.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.284+0000 7fd68be00640 1 -- 172.21.15.96:0/287765366 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 37) ==== 93182+0+0 (secure 0 0 0) 0x7fd68c019070 con 0x7fd6940ae9a0 2024-05-28T06:22:20.280 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.285+0000 7fd68be00640 1 --2- 172.21.15.96:0/287765366 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd67006e090 0x7fd670070550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:20.280 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.285+0000 7fd68be00640 1 -- 172.21.15.96:0/287765366 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 5765+0+0 (secure 0 0 0) 0x7fd68c08a520 con 0x7fd6940ae9a0 2024-05-28T06:22:20.281 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.285+0000 7fd693400640 1 --2- 172.21.15.96:0/287765366 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd67006e090 0x7fd670070550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:20.282 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.287+0000 7fd693400640 1 --2- 172.21.15.96:0/287765366 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd67006e090 0x7fd670070550 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fd6840017e0 tx=0x7fd684009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:20.285 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.289+0000 7fd68be00640 1 -- 172.21.15.96:0/287765366 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd68c05a450 con 0x7fd6940ae9a0 2024-05-28T06:22:20.523 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.527+0000 7fd698a00640 1 -- 172.21.15.96:0/287765366 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd6940689e0 con 0x7fd6940ae9a0 2024-05-28T06:22:20.524 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.529+0000 7fd68be00640 1 -- 172.21.15.96:0/287765366 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) ==== 94+0+5189 (secure 0 0 0) 0x7fd68c059d50 con 0x7fd6940ae9a0 2024-05-28T06:22:20.524 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:22:20.524 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":72,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6724,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/2148532848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6835","nonce":2148532848},{"type":"v1","addr":"172.21.15.123:6837","nonce":2148532848}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":9252,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6834/2173534467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2173534467},{"type":"v1","addr":"172.21.15.96:6834","nonce":2173534467}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":9255,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/2552082898","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6833","nonce":2552082898},{"type":"v1","addr":"172.21.15.96:6835","nonce":2552082898}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-28T06:22:16.424757+0000","modified":"2024-05-28T06:22:19.524390+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"b","rank":0,"incarnation":68,"state":"up:active","state_seq":3,"addr":"172.21.15.123:6836/3192559785","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":3192559785},{"type":"v1","addr":"172.21.15.123:6836","nonce":3192559785}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9270,"qdb_cluster":[9270]},"id":8}]} 2024-05-28T06:22:20.525 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 72 2024-05-28T06:22:20.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.531+0000 7fd698a00640 1 -- 172.21.15.96:0/287765366 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd67006e090 msgr2=0x7fd670070550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:20.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.531+0000 7fd698a00640 1 --2- 172.21.15.96:0/287765366 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd67006e090 0x7fd670070550 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fd6840017e0 tx=0x7fd684009040 comp rx=0 tx=0).stop 2024-05-28T06:22:20.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.531+0000 7fd698a00640 1 -- 172.21.15.96:0/287765366 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd6940ae9a0 msgr2=0x7fd694150230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:20.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.531+0000 7fd698a00640 1 --2- 172.21.15.96:0/287765366 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd6940ae9a0 0x7fd694150230 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7fd6940711d0 tx=0x7fd68c00d920 comp rx=0 tx=0).stop 2024-05-28T06:22:20.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.532+0000 7fd698a00640 1 -- 172.21.15.96:0/287765366 shutdown_connections 2024-05-28T06:22:20.528 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.532+0000 7fd698a00640 1 --2- 172.21.15.96:0/287765366 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd67006e090 0x7fd670070550 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:20.528 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.532+0000 7fd698a00640 1 --2- 172.21.15.96:0/287765366 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd6940ae9a0 0x7fd694150230 unknown :-1 s=CLOSED pgs=1807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:20.528 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.532+0000 7fd698a00640 1 --2- 172.21.15.96:0/287765366 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd6940ae520 0x7fd6940ccca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:20.528 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.532+0000 7fd698a00640 1 --2- 172.21.15.96:0/287765366 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd678004800 0x7fd6940ce800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:20.528 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.532+0000 7fd698a00640 1 -- 172.21.15.96:0/287765366 >> 172.21.15.96:0/287765366 conn(0x7fd6940b1fc0 msgr2=0x7fd6940af510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:20.528 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.532+0000 7fd698a00640 1 -- 172.21.15.96:0/287765366 shutdown_connections 2024-05-28T06:22:20.528 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.532+0000 7fd698a00640 1 -- 172.21.15.96:0/287765366 wait complete. 2024-05-28T06:22:20.546 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:22:20.737 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.742+0000 7fde21c00640 1 Processor -- start 2024-05-28T06:22:20.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.742+0000 7fde21c00640 1 -- start start 2024-05-28T06:22:20.743 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.743+0000 7fde21c00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fde1c0b4b60 0x7fde1c0cfaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:20.743 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.743+0000 7fde21c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fde1c0d04a0 0x7fde1c0d0900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:20.743 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.743+0000 7fde21c00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde1c0ced40 con 0x7fde1c0b3460 2024-05-28T06:22:20.743 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.743+0000 7fde21c00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fde1c0d0e40 con 0x7fde1c0b4b60 2024-05-28T06:22:20.743 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.743+0000 7fde21c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fde1c0b3fe0 con 0x7fde1c0d04a0 2024-05-28T06:22:20.743 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.744+0000 7fde1aa00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fde1c0b4b60 0x7fde1c0cfaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:20.744 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.744+0000 7fde1aa00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fde1c0b4b60 0x7fde1c0cfaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:60164/0 (socket says 172.21.15.96:60164) 2024-05-28T06:22:20.744 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.744+0000 7fde1b400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fde1c0b3460 0x7fde1c0b38d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:35092/0 (socket says 172.21.15.96:35092) 2024-05-28T06:22:20.744 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.744+0000 7fde1aa00640 1 -- 172.21.15.96:0/3467640554 learned_addr learned my addr 172.21.15.96:0/3467640554 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:22:20.744 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.744+0000 7fde1a000640 1 --2- 172.21.15.96:0/3467640554 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fde1c0d04a0 0x7fde1c0d0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:20.744 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.744+0000 7fde1aa00640 1 -- 172.21.15.96:0/3467640554 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fde1c0d04a0 msgr2=0x7fde1c0d0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:20.744 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.744+0000 7fde1aa00640 1 --2- 172.21.15.96:0/3467640554 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fde1c0d04a0 0x7fde1c0d0900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:20.744 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.744+0000 7fde1aa00640 1 -- 172.21.15.96:0/3467640554 >> v1:172.21.15.96:6789/0 conn(0x7fde1c0b3460 legacy=0x7fde1c0b38d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:20.744 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.744+0000 7fde1aa00640 1 -- 172.21.15.96:0/3467640554 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde1c15fbb0 con 0x7fde1c0b4b60 2024-05-28T06:22:20.744 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.744+0000 7fde19600640 1 -- 172.21.15.96:0/3467640554 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 254714152 0 0) 0x7fde14001490 con 0x7fde1c0b3460 2024-05-28T06:22:20.745 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.746+0000 7fde1aa00640 1 --2- 172.21.15.96:0/3467640554 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fde1c0b4b60 0x7fde1c0cfaa0 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7fde0c009870 tx=0x7fde0c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf34d88d5019d2b9 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:20.745 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.746+0000 7fde19600640 1 -- 172.21.15.96:0/3467640554 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde0c037070 con 0x7fde1c0b4b60 2024-05-28T06:22:20.745 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.746+0000 7fde19600640 1 -- 172.21.15.96:0/3467640554 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde0c032440 con 0x7fde1c0b4b60 2024-05-28T06:22:20.745 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.746+0000 7fde19600640 1 -- 172.21.15.96:0/3467640554 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde0c032810 con 0x7fde1c0b4b60 2024-05-28T06:22:20.745 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.746+0000 7fde21c00640 1 -- 172.21.15.96:0/3467640554 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fde1c0b4b60 msgr2=0x7fde1c0cfaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:20.745 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.746+0000 7fde21c00640 1 --2- 172.21.15.96:0/3467640554 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fde1c0b4b60 0x7fde1c0cfaa0 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7fde0c009870 tx=0x7fde0c02a270 comp rx=0 tx=0).stop 2024-05-28T06:22:20.745 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.746+0000 7fde21c00640 1 -- 172.21.15.96:0/3467640554 shutdown_connections 2024-05-28T06:22:20.745 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.746+0000 7fde21c00640 1 --2- 172.21.15.96:0/3467640554 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fde1c0d04a0 0x7fde1c0d0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:20.745 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.746+0000 7fde21c00640 1 --2- 172.21.15.96:0/3467640554 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fde1c0b4b60 0x7fde1c0cfaa0 unknown :-1 s=CLOSED pgs=1809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:20.745 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.746+0000 7fde21c00640 1 -- 172.21.15.96:0/3467640554 >> 172.21.15.96:0/3467640554 conn(0x7fde1c06bae0 msgr2=0x7fde1c06bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:20.745 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.747+0000 7fde21c00640 1 -- 172.21.15.96:0/3467640554 shutdown_connections 2024-05-28T06:22:20.745 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.747+0000 7fde21c00640 1 -- 172.21.15.96:0/3467640554 wait complete. 2024-05-28T06:22:20.747 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.748+0000 7fde21c00640 1 Processor -- start 2024-05-28T06:22:20.747 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.748+0000 7fde21c00640 1 -- start start 2024-05-28T06:22:20.747 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.749+0000 7fde21c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fde1c0b3460 0x7fde1c09da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:20.748 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.749+0000 7fde21c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fde1c0b4b60 0x7fde1c15d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:20.748 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.749+0000 7fde21c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde1c0d04a0 0x7fde1c0c3d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:20.748 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.749+0000 7fde21c00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fde1c0cf580 con 0x7fde1c0b4b60 2024-05-28T06:22:20.748 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.749+0000 7fde21c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fde1c0cf2a0 con 0x7fde1c0d04a0 2024-05-28T06:22:20.748 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.749+0000 7fde21c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fde1c0cffe0 con 0x7fde1c0b3460 2024-05-28T06:22:20.748 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.750+0000 7fde1a000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fde1c0b4b60 0x7fde1c15d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:20.748 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.750+0000 7fde1a000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fde1c0b4b60 0x7fde1c15d360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:60180/0 (socket says 172.21.15.96:60180) 2024-05-28T06:22:20.748 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.750+0000 7fde1a000640 1 -- 172.21.15.96:0/3758257385 learned_addr learned my addr 172.21.15.96:0/3758257385 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:22:20.748 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.750+0000 7fde1aa00640 1 --2- 172.21.15.96:0/3758257385 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fde1c0b3460 0x7fde1c09da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:20.748 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.750+0000 7fde1b400640 1 --2- 172.21.15.96:0/3758257385 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde1c0d04a0 0x7fde1c0c3d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:20.748 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.750+0000 7fde1a000640 1 -- 172.21.15.96:0/3758257385 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fde1c0b3460 msgr2=0x7fde1c09da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:20.748 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.750+0000 7fde1a000640 1 --2- 172.21.15.96:0/3758257385 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fde1c0b3460 0x7fde1c09da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:20.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.750+0000 7fde1a000640 1 -- 172.21.15.96:0/3758257385 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde1c0d04a0 msgr2=0x7fde1c0c3d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:20.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.750+0000 7fde1a000640 1 --2- 172.21.15.96:0/3758257385 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde1c0d04a0 0x7fde1c0c3d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:20.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.750+0000 7fde1a000640 1 -- 172.21.15.96:0/3758257385 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde1c15deb0 con 0x7fde1c0b4b60 2024-05-28T06:22:20.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.751+0000 7fde1a000640 1 --2- 172.21.15.96:0/3758257385 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fde1c0b4b60 0x7fde1c15d360 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7fde0800d800 tx=0x7fde0800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:20.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.751+0000 7fde12a00640 1 -- 172.21.15.96:0/3758257385 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde0800b6c0 con 0x7fde1c0b4b60 2024-05-28T06:22:20.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.751+0000 7fde12a00640 1 -- 172.21.15.96:0/3758257385 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde08019070 con 0x7fde1c0b4b60 2024-05-28T06:22:20.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.751+0000 7fde21c00640 1 -- 172.21.15.96:0/3758257385 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde1c09a2b0 con 0x7fde1c0b4b60 2024-05-28T06:22:20.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.751+0000 7fde12a00640 1 -- 172.21.15.96:0/3758257385 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde0800bac0 con 0x7fde1c0b4b60 2024-05-28T06:22:20.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.751+0000 7fde21c00640 1 -- 172.21.15.96:0/3758257385 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde1c09a550 con 0x7fde1c0b4b60 2024-05-28T06:22:20.750 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.752+0000 7fde21c00640 1 -- 172.21.15.96:0/3758257385 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde1c0689e0 con 0x7fde1c0b4b60 2024-05-28T06:22:20.750 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.753+0000 7fde12a00640 1 -- 172.21.15.96:0/3758257385 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 37) ==== 93182+0+0 (secure 0 0 0) 0x7fde0800bc60 con 0x7fde1c0b4b60 2024-05-28T06:22:20.750 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.755+0000 7fde12a00640 1 --2- 172.21.15.96:0/3758257385 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fddfc06e090 0x7fddfc070550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:20.750 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.755+0000 7fde12a00640 1 -- 172.21.15.96:0/3758257385 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 5765+0+0 (secure 0 0 0) 0x7fde0808ade0 con 0x7fde1c0b4b60 2024-05-28T06:22:20.750 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.755+0000 7fde1aa00640 1 --2- 172.21.15.96:0/3758257385 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fddfc06e090 0x7fddfc070550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:20.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.756+0000 7fde1aa00640 1 --2- 172.21.15.96:0/3758257385 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fddfc06e090 0x7fddfc070550 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fde0c034000 tx=0x7fde0c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:20.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.759+0000 7fde12a00640 1 -- 172.21.15.96:0/3758257385 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fde1c15deb0 con 0x7fde1c0b4b60 2024-05-28T06:22:20.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.962+0000 7fde21c00640 1 -- 172.21.15.96:0/3758257385 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fde1c0b3b70 con 0x7fde1c0b4b60 2024-05-28T06:22:20.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.965+0000 7fde12a00640 1 -- 172.21.15.96:0/3758257385 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v115) ==== 74+0+16727 (secure 0 0 0) 0x7fde0805ad10 con 0x7fde1c0b4b60 2024-05-28T06:22:20.961 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:22:20.961 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":115,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:22:17.989923+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-05-28T06:22:14.138685+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"114","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-05-28T06:22:15.394361+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"115","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":13,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:22:20.963 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.967+0000 7fde21c00640 1 -- 172.21.15.96:0/3758257385 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fddfc06e090 msgr2=0x7fddfc070550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:20.963 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.967+0000 7fde21c00640 1 --2- 172.21.15.96:0/3758257385 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fddfc06e090 0x7fddfc070550 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fde0c034000 tx=0x7fde0c02e000 comp rx=0 tx=0).stop 2024-05-28T06:22:20.963 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.967+0000 7fde21c00640 1 -- 172.21.15.96:0/3758257385 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fde1c0b4b60 msgr2=0x7fde1c15d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:20.964 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.967+0000 7fde21c00640 1 --2- 172.21.15.96:0/3758257385 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fde1c0b4b60 0x7fde1c15d360 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7fde0800d800 tx=0x7fde0800dcc0 comp rx=0 tx=0).stop 2024-05-28T06:22:20.964 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.968+0000 7fde21c00640 1 -- 172.21.15.96:0/3758257385 shutdown_connections 2024-05-28T06:22:20.964 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.968+0000 7fde21c00640 1 --2- 172.21.15.96:0/3758257385 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fddfc06e090 0x7fddfc070550 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:20.964 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.968+0000 7fde21c00640 1 --2- 172.21.15.96:0/3758257385 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde1c0d04a0 0x7fde1c0c3d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:20.964 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.968+0000 7fde21c00640 1 --2- 172.21.15.96:0/3758257385 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fde1c0b4b60 0x7fde1c15d360 unknown :-1 s=CLOSED pgs=1810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:20.964 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.968+0000 7fde21c00640 1 --2- 172.21.15.96:0/3758257385 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fde1c0b3460 0x7fde1c09da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:20.964 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.968+0000 7fde21c00640 1 -- 172.21.15.96:0/3758257385 >> 172.21.15.96:0/3758257385 conn(0x7fde1c06bae0 msgr2=0x7fde1c0b1e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:20.964 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.968+0000 7fde21c00640 1 -- 172.21.15.96:0/3758257385 shutdown_connections 2024-05-28T06:22:20.964 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:20.968+0000 7fde21c00640 1 -- 172.21.15.96:0/3758257385 wait complete. 2024-05-28T06:22:20.983 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-05-28T06:22:20.983 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-28T06:22:21.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.188+0000 7fbdfb200640 1 Processor -- start 2024-05-28T06:22:21.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.189+0000 7fbdfb200640 1 -- start start 2024-05-28T06:22:21.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.190+0000 7fbdfb200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fbdfc09aac0 0x7fbdfc06b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:21.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.190+0000 7fbdfb200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fbdfc06be40 0x7fbdfc15f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:21.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.190+0000 7fbdfb200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbdfc0b2860 con 0x7fbdfc15e920 2024-05-28T06:22:21.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.190+0000 7fbdfb200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fbdfc0b5ea0 con 0x7fbdfc06be40 2024-05-28T06:22:21.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.190+0000 7fbdfb200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fbdfc15ed00 con 0x7fbdfc09aac0 2024-05-28T06:22:21.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.191+0000 7fbdfa800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fbdfc06be40 0x7fbdfc15f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:21.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.191+0000 7fbdfa800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fbdfc06be40 0x7fbdfc15f8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:60186/0 (socket says 172.21.15.96:60186) 2024-05-28T06:22:21.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.191+0000 7fbdfa800640 1 -- 172.21.15.96:0/2360458978 learned_addr learned my addr 172.21.15.96:0/2360458978 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:22:21.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.191+0000 7fbdfa800640 1 -- 172.21.15.96:0/2360458978 >> v2:172.21.15.123:3300/0 conn(0x7fbdfc09aac0 msgr2=0x7fbdfc06b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:21.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.191+0000 7fbdf9400640 1 --2- 172.21.15.96:0/2360458978 >> v2:172.21.15.123:3300/0 conn(0x7fbdfc09aac0 0x7fbdfc06b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:21.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.191+0000 7fbdfa800640 1 --2- 172.21.15.96:0/2360458978 >> v2:172.21.15.123:3300/0 conn(0x7fbdfc09aac0 0x7fbdfc06b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:21.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.191+0000 7fbdfa800640 1 -- 172.21.15.96:0/2360458978 >> v1:172.21.15.123:6789/0 conn(0x7fbdfc15e920 legacy=0x7fbdfc0b42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:21.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.191+0000 7fbdfa800640 1 -- 172.21.15.96:0/2360458978 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdfc0b2b40 con 0x7fbdfc06be40 2024-05-28T06:22:21.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.192+0000 7fbdfa800640 1 --2- 172.21.15.96:0/2360458978 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbdfc06be40 0x7fbdfc15f8b0 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7fbdfc0757c0 tx=0x7fbdec02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ea4689efd103dc server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:21.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.192+0000 7fbdf8a00640 1 -- 172.21.15.96:0/2360458978 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdec037070 con 0x7fbdfc06be40 2024-05-28T06:22:21.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.192+0000 7fbdf8a00640 1 -- 172.21.15.96:0/2360458978 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbdec032440 con 0x7fbdfc06be40 2024-05-28T06:22:21.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.192+0000 7fbdf8a00640 1 -- 172.21.15.96:0/2360458978 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdec032810 con 0x7fbdfc06be40 2024-05-28T06:22:21.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.193+0000 7fbdfb200640 1 -- 172.21.15.96:0/2360458978 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbdfc06be40 msgr2=0x7fbdfc15f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:21.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.193+0000 7fbdfb200640 1 --2- 172.21.15.96:0/2360458978 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbdfc06be40 0x7fbdfc15f8b0 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7fbdfc0757c0 tx=0x7fbdec02a1a0 comp rx=0 tx=0).stop 2024-05-28T06:22:21.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.193+0000 7fbdfb200640 1 -- 172.21.15.96:0/2360458978 shutdown_connections 2024-05-28T06:22:21.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.193+0000 7fbdfb200640 1 --2- 172.21.15.96:0/2360458978 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbdfc06be40 0x7fbdfc15f8b0 unknown :-1 s=CLOSED pgs=1811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:21.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.193+0000 7fbdfb200640 1 --2- 172.21.15.96:0/2360458978 >> v2:172.21.15.123:3300/0 conn(0x7fbdfc09aac0 0x7fbdfc06b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:21.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.193+0000 7fbdfb200640 1 -- 172.21.15.96:0/2360458978 >> 172.21.15.96:0/2360458978 conn(0x7fbdfc0b6a10 msgr2=0x7fbdfc0b2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:21.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.193+0000 7fbdfb200640 1 -- 172.21.15.96:0/2360458978 shutdown_connections 2024-05-28T06:22:21.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.193+0000 7fbdfb200640 1 -- 172.21.15.96:0/2360458978 wait complete. 2024-05-28T06:22:21.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.195+0000 7fbdfb200640 1 Processor -- start 2024-05-28T06:22:21.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.196+0000 7fbdfb200640 1 -- start start 2024-05-28T06:22:21.193 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.197+0000 7fbdfb200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbdfc06be40 0x7fbdfc147d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:21.194 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.197+0000 7fbdfb200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbdfc09aac0 0x7fbdfc15b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:21.194 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.197+0000 7fbdfb200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbdfc15e920 0x7fbdfc153ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:21.194 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.197+0000 7fbdfb200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fbdfc0b2e20 con 0x7fbdfc06be40 2024-05-28T06:22:21.194 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.197+0000 7fbdfb200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fbdfc0b2610 con 0x7fbdfc15e920 2024-05-28T06:22:21.194 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.197+0000 7fbdfb200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fbdfc0b65f0 con 0x7fbdfc09aac0 2024-05-28T06:22:21.194 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.197+0000 7fbdf9e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbdfc06be40 0x7fbdfc147d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:21.194 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.197+0000 7fbdf9e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbdfc06be40 0x7fbdfc147d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:60198/0 (socket says 172.21.15.96:60198) 2024-05-28T06:22:21.194 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.197+0000 7fbdf9e00640 1 -- 172.21.15.96:0/1322761713 learned_addr learned my addr 172.21.15.96:0/1322761713 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:22:21.194 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.197+0000 7fbdfa800640 1 --2- 172.21.15.96:0/1322761713 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbdfc15e920 0x7fbdfc153ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:21.194 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.197+0000 7fbdf9e00640 1 -- 172.21.15.96:0/1322761713 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbdfc09aac0 msgr2=0x7fbdfc15b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:21.195 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.197+0000 7fbdf9400640 1 --2- 172.21.15.96:0/1322761713 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbdfc09aac0 0x7fbdfc15b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:21.195 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.198+0000 7fbdf9e00640 1 --2- 172.21.15.96:0/1322761713 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbdfc09aac0 0x7fbdfc15b220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:21.195 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.198+0000 7fbdf9e00640 1 -- 172.21.15.96:0/1322761713 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbdfc15e920 msgr2=0x7fbdfc153ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:21.195 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.198+0000 7fbdf9e00640 1 --2- 172.21.15.96:0/1322761713 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbdfc15e920 0x7fbdfc153ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:21.195 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.198+0000 7fbdf9e00640 1 -- 172.21.15.96:0/1322761713 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdfc159540 con 0x7fbdfc06be40 2024-05-28T06:22:21.195 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.198+0000 7fbdfa800640 1 --2- 172.21.15.96:0/1322761713 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbdfc15e920 0x7fbdfc153ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:22:21.195 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.198+0000 7fbdf9e00640 1 --2- 172.21.15.96:0/1322761713 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbdfc06be40 0x7fbdfc147d20 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7fbdf400b410 tx=0x7fbdf400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:21.195 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.198+0000 7fbdf2000640 1 -- 172.21.15.96:0/1322761713 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdf4015020 con 0x7fbdfc06be40 2024-05-28T06:22:21.195 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.198+0000 7fbdf2000640 1 -- 172.21.15.96:0/1322761713 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbdf4011830 con 0x7fbdfc06be40 2024-05-28T06:22:21.195 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.198+0000 7fbdf2000640 1 -- 172.21.15.96:0/1322761713 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdf4011c20 con 0x7fbdfc06be40 2024-05-28T06:22:21.195 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.198+0000 7fbdfb200640 1 -- 172.21.15.96:0/1322761713 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbdfc1597e0 con 0x7fbdfc06be40 2024-05-28T06:22:21.195 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.198+0000 7fbdfb200640 1 -- 172.21.15.96:0/1322761713 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbdfc159da0 con 0x7fbdfc06be40 2024-05-28T06:22:21.196 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.200+0000 7fbdfb200640 1 -- 172.21.15.96:0/1322761713 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbdfc0690f0 con 0x7fbdfc06be40 2024-05-28T06:22:21.196 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.201+0000 7fbdf2000640 1 -- 172.21.15.96:0/1322761713 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 37) ==== 93182+0+0 (secure 0 0 0) 0x7fbdf4018020 con 0x7fbdfc06be40 2024-05-28T06:22:21.197 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.202+0000 7fbdf2000640 1 --2- 172.21.15.96:0/1322761713 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbdd806e090 0x7fbdd8070550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:21.197 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.202+0000 7fbdf2000640 1 -- 172.21.15.96:0/1322761713 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 5765+0+0 (secure 0 0 0) 0x7fbdf408a8a0 con 0x7fbdfc06be40 2024-05-28T06:22:21.198 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.203+0000 7fbdf9400640 1 --2- 172.21.15.96:0/1322761713 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbdd806e090 0x7fbdd8070550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:21.199 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.204+0000 7fbdf9400640 1 --2- 172.21.15.96:0/1322761713 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbdd806e090 0x7fbdd8070550 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fbdfc06bb00 tx=0x7fbde4008440 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:21.201 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.206+0000 7fbdf2000640 1 -- 172.21.15.96:0/1322761713 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbdf401f020 con 0x7fbdfc06be40 2024-05-28T06:22:21.382 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.387+0000 7fbdfb200640 1 -- 172.21.15.96:0/1322761713 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fbdfc1586c0 con 0x7fbdd806e090 2024-05-28T06:22:21.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.406+0000 7fbdf2000640 1 -- 172.21.15.96:0/1322761713 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+247562 (secure 0 0 0) 0x7fbdfc1586c0 con 0x7fbdd806e090 2024-05-28T06:22:21.406 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:22:21.407 INFO:teuthology.orchestra.run.smithi096.stderr:dumped all 2024-05-28T06:22:21.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.411+0000 7fbdfb200640 1 -- 172.21.15.96:0/1322761713 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbdd806e090 msgr2=0x7fbdd8070550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:21.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.411+0000 7fbdfb200640 1 --2- 172.21.15.96:0/1322761713 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbdd806e090 0x7fbdd8070550 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fbdfc06bb00 tx=0x7fbde4008440 comp rx=0 tx=0).stop 2024-05-28T06:22:21.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.411+0000 7fbdfb200640 1 -- 172.21.15.96:0/1322761713 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbdfc06be40 msgr2=0x7fbdfc147d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:21.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.411+0000 7fbdfb200640 1 --2- 172.21.15.96:0/1322761713 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbdfc06be40 0x7fbdfc147d20 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7fbdf400b410 tx=0x7fbdf400b8d0 comp rx=0 tx=0).stop 2024-05-28T06:22:21.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.411+0000 7fbdfb200640 1 -- 172.21.15.96:0/1322761713 shutdown_connections 2024-05-28T06:22:21.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.411+0000 7fbdfb200640 1 --2- 172.21.15.96:0/1322761713 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbdd806e090 0x7fbdd8070550 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:21.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.411+0000 7fbdfb200640 1 --2- 172.21.15.96:0/1322761713 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbdfc15e920 0x7fbdfc153ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:21.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.411+0000 7fbdfb200640 1 --2- 172.21.15.96:0/1322761713 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbdfc09aac0 0x7fbdfc15b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:21.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.411+0000 7fbdfb200640 1 --2- 172.21.15.96:0/1322761713 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbdfc06be40 0x7fbdfc147d20 unknown :-1 s=CLOSED pgs=1812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:21.410 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.411+0000 7fbdfb200640 1 -- 172.21.15.96:0/1322761713 >> 172.21.15.96:0/1322761713 conn(0x7fbdfc0b6a10 msgr2=0x7fbdfc15d1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:21.410 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.411+0000 7fbdfb200640 1 -- 172.21.15.96:0/1322761713 shutdown_connections 2024-05-28T06:22:21.410 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.411+0000 7fbdfb200640 1 -- 172.21.15.96:0/1322761713 wait complete. 2024-05-28T06:22:21.426 INFO:teuthology.orchestra.run.smithi096.stdout:{"pg_ready":true,"pg_map":{"version":658,"stamp":"2024-05-28T06:22:19.964198+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592126,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":198,"num_write_kb":2428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148,"ondisk_log_size":148,"up":152,"acting":152,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":104,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":912220,"kb_used_data":12192,"kb_used_omap":12,"kb_used_meta":899891,"kb_avail":754062500,"statfs":{"total":773094113280,"available":772160000000,"internally_reserved":0,"allocated":12484608,"data_stored":6098743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":921488944},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1758,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":39,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.008041"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.242686+0000","last_change":"2024-05-28T06:22:17.004588+0000","last_active":"2024-05-28T06:22:18.242686+0000","last_peered":"2024-05-28T06:22:18.242686+0000","last_clean":"2024-05-28T06:22:18.242686+0000","last_became_active":"2024-05-28T06:22:17.003523+0000","last_became_peered":"2024-05-28T06:22:17.003523+0000","last_unstale":"2024-05-28T06:22:18.242686+0000","last_undegraded":"2024-05-28T06:22:18.242686+0000","last_fullsized":"2024-05-28T06:22:18.242686+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:04:35.674926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-28T06:22:15.950581+0000","last_change":"2024-05-28T06:22:15.950581+0000","last_active":"2024-05-28T06:22:15.950581+0000","last_peered":"2024-05-28T06:22:15.950581+0000","last_clean":"2024-05-28T06:22:15.950581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:22:15.950581+0000","last_undegraded":"2024-05-28T06:22:15.950581+0000","last_fullsized":"2024-05-28T06:22:15.950581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":1,"reported_epoch":113,"state":"creating+peering","last_fresh":"2024-05-28T06:22:15.985954+0000","last_change":"2024-05-28T06:22:15.985954+0000","last_active":"2024-05-28T06:22:15.950581+0000","last_peered":"2024-05-28T06:22:15.950581+0000","last_clean":"2024-05-28T06:22:15.950581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:22:15.985954+0000","last_undegraded":"2024-05-28T06:22:15.985954+0000","last_fullsized":"2024-05-28T06:22:15.985954+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.242843+0000","last_change":"2024-05-28T06:22:16.994809+0000","last_active":"2024-05-28T06:22:18.242843+0000","last_peered":"2024-05-28T06:22:18.242843+0000","last_clean":"2024-05-28T06:22:18.242843+0000","last_became_active":"2024-05-28T06:22:16.994548+0000","last_became_peered":"2024-05-28T06:22:16.994548+0000","last_unstale":"2024-05-28T06:22:18.242843+0000","last_undegraded":"2024-05-28T06:22:18.242843+0000","last_fullsized":"2024-05-28T06:22:18.242843+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:59:36.297627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:17.999386+0000","last_change":"2024-05-28T06:22:16.996834+0000","last_active":"2024-05-28T06:22:17.999386+0000","last_peered":"2024-05-28T06:22:17.999386+0000","last_clean":"2024-05-28T06:22:17.999386+0000","last_became_active":"2024-05-28T06:22:16.996543+0000","last_became_peered":"2024-05-28T06:22:16.996543+0000","last_unstale":"2024-05-28T06:22:17.999386+0000","last_undegraded":"2024-05-28T06:22:17.999386+0000","last_fullsized":"2024-05-28T06:22:17.999386+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:23:11.730784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.528701+0000","last_change":"2024-05-28T06:22:16.995996+0000","last_active":"2024-05-28T06:22:18.528701+0000","last_peered":"2024-05-28T06:22:18.528701+0000","last_clean":"2024-05-28T06:22:18.528701+0000","last_became_active":"2024-05-28T06:22:16.995714+0000","last_became_peered":"2024-05-28T06:22:16.995714+0000","last_unstale":"2024-05-28T06:22:18.528701+0000","last_undegraded":"2024-05-28T06:22:18.528701+0000","last_fullsized":"2024-05-28T06:22:18.528701+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:04:42.043002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-28T06:22:15.950581+0000","last_change":"2024-05-28T06:22:15.950581+0000","last_active":"2024-05-28T06:22:15.950581+0000","last_peered":"2024-05-28T06:22:15.950581+0000","last_clean":"2024-05-28T06:22:15.950581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:22:15.950581+0000","last_undegraded":"2024-05-28T06:22:15.950581+0000","last_fullsized":"2024-05-28T06:22:15.950581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":1,"reported_epoch":113,"state":"creating+peering","last_fresh":"2024-05-28T06:22:15.981127+0000","last_change":"2024-05-28T06:22:15.981127+0000","last_active":"2024-05-28T06:22:15.950581+0000","last_peered":"2024-05-28T06:22:15.950581+0000","last_clean":"2024-05-28T06:22:15.950581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:22:15.981127+0000","last_undegraded":"2024-05-28T06:22:15.981127+0000","last_fullsized":"2024-05-28T06:22:15.981127+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.242476+0000","last_change":"2024-05-28T06:22:17.003545+0000","last_active":"2024-05-28T06:22:18.242476+0000","last_peered":"2024-05-28T06:22:18.242476+0000","last_clean":"2024-05-28T06:22:18.242476+0000","last_became_active":"2024-05-28T06:22:17.001251+0000","last_became_peered":"2024-05-28T06:22:17.001251+0000","last_unstale":"2024-05-28T06:22:18.242476+0000","last_undegraded":"2024-05-28T06:22:18.242476+0000","last_fullsized":"2024-05-28T06:22:18.242476+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:38:50.761814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.243297+0000","last_change":"2024-05-28T06:22:17.001134+0000","last_active":"2024-05-28T06:22:18.243297+0000","last_peered":"2024-05-28T06:22:18.243297+0000","last_clean":"2024-05-28T06:22:18.243297+0000","last_became_active":"2024-05-28T06:22:17.000345+0000","last_became_peered":"2024-05-28T06:22:17.000345+0000","last_unstale":"2024-05-28T06:22:18.243297+0000","last_undegraded":"2024-05-28T06:22:18.243297+0000","last_fullsized":"2024-05-28T06:22:18.243297+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:22:46.981586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-28T06:22:15.950581+0000","last_change":"2024-05-28T06:22:15.950581+0000","last_active":"2024-05-28T06:22:15.950581+0000","last_peered":"2024-05-28T06:22:15.950581+0000","last_clean":"2024-05-28T06:22:15.950581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:22:15.950581+0000","last_undegraded":"2024-05-28T06:22:15.950581+0000","last_fullsized":"2024-05-28T06:22:15.950581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.243231+0000","last_change":"2024-05-28T06:22:16.994222+0000","last_active":"2024-05-28T06:22:18.243231+0000","last_peered":"2024-05-28T06:22:18.243231+0000","last_clean":"2024-05-28T06:22:18.243231+0000","last_became_active":"2024-05-28T06:22:16.993674+0000","last_became_peered":"2024-05-28T06:22:16.993674+0000","last_unstale":"2024-05-28T06:22:18.243231+0000","last_undegraded":"2024-05-28T06:22:18.243231+0000","last_fullsized":"2024-05-28T06:22:18.243231+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:40:55.388301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":1,"reported_epoch":113,"state":"creating+peering","last_fresh":"2024-05-28T06:22:15.979372+0000","last_change":"2024-05-28T06:22:15.979372+0000","last_active":"2024-05-28T06:22:15.950581+0000","last_peered":"2024-05-28T06:22:15.950581+0000","last_clean":"2024-05-28T06:22:15.950581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:22:15.979372+0000","last_undegraded":"2024-05-28T06:22:15.979372+0000","last_fullsized":"2024-05-28T06:22:15.979372+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":1,"reported_epoch":113,"state":"creating+peering","last_fresh":"2024-05-28T06:22:15.979017+0000","last_change":"2024-05-28T06:22:15.979017+0000","last_active":"2024-05-28T06:22:15.950581+0000","last_peered":"2024-05-28T06:22:15.950581+0000","last_clean":"2024-05-28T06:22:15.950581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:22:15.979017+0000","last_undegraded":"2024-05-28T06:22:15.979017+0000","last_fullsized":"2024-05-28T06:22:15.979017+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.527922+0000","last_change":"2024-05-28T06:22:16.997664+0000","last_active":"2024-05-28T06:22:18.527922+0000","last_peered":"2024-05-28T06:22:18.527922+0000","last_clean":"2024-05-28T06:22:18.527922+0000","last_became_active":"2024-05-28T06:22:16.997387+0000","last_became_peered":"2024-05-28T06:22:16.997387+0000","last_unstale":"2024-05-28T06:22:18.527922+0000","last_undegraded":"2024-05-28T06:22:18.527922+0000","last_fullsized":"2024-05-28T06:22:18.527922+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:37:05.453277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.242046+0000","last_change":"2024-05-28T06:22:17.005071+0000","last_active":"2024-05-28T06:22:18.242046+0000","last_peered":"2024-05-28T06:22:18.242046+0000","last_clean":"2024-05-28T06:22:18.242046+0000","last_became_active":"2024-05-28T06:22:17.004078+0000","last_became_peered":"2024-05-28T06:22:17.004078+0000","last_unstale":"2024-05-28T06:22:18.242046+0000","last_undegraded":"2024-05-28T06:22:18.242046+0000","last_fullsized":"2024-05-28T06:22:18.242046+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:26:19.869612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-28T06:22:15.950581+0000","last_change":"2024-05-28T06:22:15.950581+0000","last_active":"2024-05-28T06:22:15.950581+0000","last_peered":"2024-05-28T06:22:15.950581+0000","last_clean":"2024-05-28T06:22:15.950581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:22:15.950581+0000","last_undegraded":"2024-05-28T06:22:15.950581+0000","last_fullsized":"2024-05-28T06:22:15.950581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-28T06:22:15.950581+0000","last_change":"2024-05-28T06:22:15.950581+0000","last_active":"2024-05-28T06:22:15.950581+0000","last_peered":"2024-05-28T06:22:15.950581+0000","last_clean":"2024-05-28T06:22:15.950581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:22:15.950581+0000","last_undegraded":"2024-05-28T06:22:15.950581+0000","last_fullsized":"2024-05-28T06:22:15.950581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":1,"reported_epoch":113,"state":"creating+peering","last_fresh":"2024-05-28T06:22:15.983434+0000","last_change":"2024-05-28T06:22:15.983434+0000","last_active":"2024-05-28T06:22:15.950581+0000","last_peered":"2024-05-28T06:22:15.950581+0000","last_clean":"2024-05-28T06:22:15.950581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:22:15.983434+0000","last_undegraded":"2024-05-28T06:22:15.983434+0000","last_fullsized":"2024-05-28T06:22:15.983434+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-28T06:22:15.950581+0000","last_change":"2024-05-28T06:22:15.950581+0000","last_active":"2024-05-28T06:22:15.950581+0000","last_peered":"2024-05-28T06:22:15.950581+0000","last_clean":"2024-05-28T06:22:15.950581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:22:15.950581+0000","last_undegraded":"2024-05-28T06:22:15.950581+0000","last_fullsized":"2024-05-28T06:22:15.950581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-28T06:22:15.950581+0000","last_change":"2024-05-28T06:22:15.950581+0000","last_active":"2024-05-28T06:22:15.950581+0000","last_peered":"2024-05-28T06:22:15.950581+0000","last_clean":"2024-05-28T06:22:15.950581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:22:15.950581+0000","last_undegraded":"2024-05-28T06:22:15.950581+0000","last_fullsized":"2024-05-28T06:22:15.950581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.528790+0000","last_change":"2024-05-28T06:22:17.004708+0000","last_active":"2024-05-28T06:22:18.528790+0000","last_peered":"2024-05-28T06:22:18.528790+0000","last_clean":"2024-05-28T06:22:18.528790+0000","last_became_active":"2024-05-28T06:22:17.003676+0000","last_became_peered":"2024-05-28T06:22:17.003676+0000","last_unstale":"2024-05-28T06:22:18.528790+0000","last_undegraded":"2024-05-28T06:22:18.528790+0000","last_fullsized":"2024-05-28T06:22:18.528790+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:38:34.851302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-28T06:22:15.950581+0000","last_change":"2024-05-28T06:22:15.950581+0000","last_active":"2024-05-28T06:22:15.950581+0000","last_peered":"2024-05-28T06:22:15.950581+0000","last_clean":"2024-05-28T06:22:15.950581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:22:15.950581+0000","last_undegraded":"2024-05-28T06:22:15.950581+0000","last_fullsized":"2024-05-28T06:22:15.950581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.526910+0000","last_change":"2024-05-28T06:22:17.008315+0000","last_active":"2024-05-28T06:22:18.526910+0000","last_peered":"2024-05-28T06:22:18.526910+0000","last_clean":"2024-05-28T06:22:18.526910+0000","last_became_active":"2024-05-28T06:22:17.007995+0000","last_became_peered":"2024-05-28T06:22:17.007995+0000","last_unstale":"2024-05-28T06:22:18.526910+0000","last_undegraded":"2024-05-28T06:22:18.526910+0000","last_fullsized":"2024-05-28T06:22:18.526910+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T18:02:32.237669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-28T06:22:15.950581+0000","last_change":"2024-05-28T06:22:15.950581+0000","last_active":"2024-05-28T06:22:15.950581+0000","last_peered":"2024-05-28T06:22:15.950581+0000","last_clean":"2024-05-28T06:22:15.950581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:22:15.950581+0000","last_undegraded":"2024-05-28T06:22:15.950581+0000","last_fullsized":"2024-05-28T06:22:15.950581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.243265+0000","last_change":"2024-05-28T06:22:17.006855+0000","last_active":"2024-05-28T06:22:18.243265+0000","last_peered":"2024-05-28T06:22:18.243265+0000","last_clean":"2024-05-28T06:22:18.243265+0000","last_became_active":"2024-05-28T06:22:17.006649+0000","last_became_peered":"2024-05-28T06:22:17.006649+0000","last_unstale":"2024-05-28T06:22:18.243265+0000","last_undegraded":"2024-05-28T06:22:18.243265+0000","last_fullsized":"2024-05-28T06:22:18.243265+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:25:34.876194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.529987+0000","last_change":"2024-05-28T06:22:17.000165+0000","last_active":"2024-05-28T06:22:18.529987+0000","last_peered":"2024-05-28T06:22:18.529987+0000","last_clean":"2024-05-28T06:22:18.529987+0000","last_became_active":"2024-05-28T06:22:16.999904+0000","last_became_peered":"2024-05-28T06:22:16.999904+0000","last_unstale":"2024-05-28T06:22:18.529987+0000","last_undegraded":"2024-05-28T06:22:18.529987+0000","last_fullsized":"2024-05-28T06:22:18.529987+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:58:15.396668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.529632+0000","last_change":"2024-05-28T06:22:16.993958+0000","last_active":"2024-05-28T06:22:18.529632+0000","last_peered":"2024-05-28T06:22:18.529632+0000","last_clean":"2024-05-28T06:22:18.529632+0000","last_became_active":"2024-05-28T06:22:16.993327+0000","last_became_peered":"2024-05-28T06:22:16.993327+0000","last_unstale":"2024-05-28T06:22:18.529632+0000","last_undegraded":"2024-05-28T06:22:18.529632+0000","last_fullsized":"2024-05-28T06:22:18.529632+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:26:18.638055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-28T06:22:15.950581+0000","last_change":"2024-05-28T06:22:15.950581+0000","last_active":"2024-05-28T06:22:15.950581+0000","last_peered":"2024-05-28T06:22:15.950581+0000","last_clean":"2024-05-28T06:22:15.950581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:22:15.950581+0000","last_undegraded":"2024-05-28T06:22:15.950581+0000","last_fullsized":"2024-05-28T06:22:15.950581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-28T06:22:15.950581+0000","last_change":"2024-05-28T06:22:15.950581+0000","last_active":"2024-05-28T06:22:15.950581+0000","last_peered":"2024-05-28T06:22:15.950581+0000","last_clean":"2024-05-28T06:22:15.950581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:22:15.950581+0000","last_undegraded":"2024-05-28T06:22:15.950581+0000","last_fullsized":"2024-05-28T06:22:15.950581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.242168+0000","last_change":"2024-05-28T06:22:17.001908+0000","last_active":"2024-05-28T06:22:18.242168+0000","last_peered":"2024-05-28T06:22:18.242168+0000","last_clean":"2024-05-28T06:22:18.242168+0000","last_became_active":"2024-05-28T06:22:17.000543+0000","last_became_peered":"2024-05-28T06:22:17.000543+0000","last_unstale":"2024-05-28T06:22:18.242168+0000","last_undegraded":"2024-05-28T06:22:18.242168+0000","last_fullsized":"2024-05-28T06:22:18.242168+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:12:55.159821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.527587+0000","last_change":"2024-05-28T06:22:16.992715+0000","last_active":"2024-05-28T06:22:18.527587+0000","last_peered":"2024-05-28T06:22:18.527587+0000","last_clean":"2024-05-28T06:22:18.527587+0000","last_became_active":"2024-05-28T06:22:16.992094+0000","last_became_peered":"2024-05-28T06:22:16.992094+0000","last_unstale":"2024-05-28T06:22:18.527587+0000","last_undegraded":"2024-05-28T06:22:18.527587+0000","last_fullsized":"2024-05-28T06:22:18.527587+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:05:25.661249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.529840+0000","last_change":"2024-05-28T06:22:17.004255+0000","last_active":"2024-05-28T06:22:18.529840+0000","last_peered":"2024-05-28T06:22:18.529840+0000","last_clean":"2024-05-28T06:22:18.529840+0000","last_became_active":"2024-05-28T06:22:17.001691+0000","last_became_peered":"2024-05-28T06:22:17.001691+0000","last_unstale":"2024-05-28T06:22:18.529840+0000","last_undegraded":"2024-05-28T06:22:18.529840+0000","last_fullsized":"2024-05-28T06:22:18.529840+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:40:20.104894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":1,"reported_epoch":112,"state":"creating+peering","last_fresh":"2024-05-28T06:22:14.968914+0000","last_change":"2024-05-28T06:22:14.968914+0000","last_active":"2024-05-28T06:22:14.938772+0000","last_peered":"2024-05-28T06:22:14.938772+0000","last_clean":"2024-05-28T06:22:14.938772+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:22:14.968914+0000","last_undegraded":"2024-05-28T06:22:14.968914+0000","last_fullsized":"2024-05-28T06:22:14.968914+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-28T06:22:15.950581+0000","last_change":"2024-05-28T06:22:15.950581+0000","last_active":"2024-05-28T06:22:15.950581+0000","last_peered":"2024-05-28T06:22:15.950581+0000","last_clean":"2024-05-28T06:22:15.950581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:22:15.950581+0000","last_undegraded":"2024-05-28T06:22:15.950581+0000","last_fullsized":"2024-05-28T06:22:15.950581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":9,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-05-28T06:22:15.998589+0000","last_change":"2024-05-28T06:22:15.996401+0000","last_active":"2024-05-28T06:22:15.998589+0000","last_peered":"2024-05-28T06:22:15.998589+0000","last_clean":"2024-05-28T06:22:15.998589+0000","last_became_active":"2024-05-28T06:22:15.996039+0000","last_became_peered":"2024-05-28T06:22:15.996039+0000","last_unstale":"2024-05-28T06:22:15.998589+0000","last_undegraded":"2024-05-28T06:22:15.998589+0000","last_fullsized":"2024-05-28T06:22:15.998589+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:41:39.601468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.528998+0000","last_change":"2024-05-28T06:22:16.999053+0000","last_active":"2024-05-28T06:22:18.528998+0000","last_peered":"2024-05-28T06:22:18.528998+0000","last_clean":"2024-05-28T06:22:18.528998+0000","last_became_active":"2024-05-28T06:22:16.995589+0000","last_became_peered":"2024-05-28T06:22:16.995589+0000","last_unstale":"2024-05-28T06:22:18.528998+0000","last_undegraded":"2024-05-28T06:22:18.528998+0000","last_fullsized":"2024-05-28T06:22:18.528998+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:58:42.572818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"114'2","reported_seq":16,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:17.999806+0000","last_change":"2024-05-28T06:22:15.989244+0000","last_active":"2024-05-28T06:22:17.999806+0000","last_peered":"2024-05-28T06:22:17.999806+0000","last_clean":"2024-05-28T06:22:17.999806+0000","last_became_active":"2024-05-28T06:22:15.988137+0000","last_became_peered":"2024-05-28T06:22:15.988137+0000","last_unstale":"2024-05-28T06:22:17.999806+0000","last_undegraded":"2024-05-28T06:22:17.999806+0000","last_fullsized":"2024-05-28T06:22:17.999806+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T18:10:18.669151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":1,"reported_epoch":113,"state":"creating+peering","last_fresh":"2024-05-28T06:22:15.979253+0000","last_change":"2024-05-28T06:22:15.979253+0000","last_active":"2024-05-28T06:22:15.950581+0000","last_peered":"2024-05-28T06:22:15.950581+0000","last_clean":"2024-05-28T06:22:15.950581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:22:15.979253+0000","last_undegraded":"2024-05-28T06:22:15.979253+0000","last_fullsized":"2024-05-28T06:22:15.979253+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"114'2","reported_seq":16,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:17.999369+0000","last_change":"2024-05-28T06:22:15.992414+0000","last_active":"2024-05-28T06:22:17.999369+0000","last_peered":"2024-05-28T06:22:17.999369+0000","last_clean":"2024-05-28T06:22:17.999369+0000","last_became_active":"2024-05-28T06:22:15.991608+0000","last_became_peered":"2024-05-28T06:22:15.991608+0000","last_unstale":"2024-05-28T06:22:17.999369+0000","last_undegraded":"2024-05-28T06:22:17.999369+0000","last_fullsized":"2024-05-28T06:22:17.999369+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:42:21.601868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.241943+0000","last_change":"2024-05-28T06:22:16.995582+0000","last_active":"2024-05-28T06:22:18.241943+0000","last_peered":"2024-05-28T06:22:18.241943+0000","last_clean":"2024-05-28T06:22:18.241943+0000","last_became_active":"2024-05-28T06:22:16.995242+0000","last_became_peered":"2024-05-28T06:22:16.995242+0000","last_unstale":"2024-05-28T06:22:18.241943+0000","last_undegraded":"2024-05-28T06:22:18.241943+0000","last_fullsized":"2024-05-28T06:22:18.241943+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:46:21.232752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":1,"reported_epoch":112,"state":"creating+peering","last_fresh":"2024-05-28T06:22:14.970332+0000","last_change":"2024-05-28T06:22:14.970332+0000","last_active":"2024-05-28T06:22:14.938772+0000","last_peered":"2024-05-28T06:22:14.938772+0000","last_clean":"2024-05-28T06:22:14.938772+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:22:14.970332+0000","last_undegraded":"2024-05-28T06:22:14.970332+0000","last_fullsized":"2024-05-28T06:22:14.970332+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-28T06:22:15.950581+0000","last_change":"2024-05-28T06:22:15.950581+0000","last_active":"2024-05-28T06:22:15.950581+0000","last_peered":"2024-05-28T06:22:15.950581+0000","last_clean":"2024-05-28T06:22:15.950581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:22:15.950581+0000","last_undegraded":"2024-05-28T06:22:15.950581+0000","last_fullsized":"2024-05-28T06:22:15.950581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1a","version":"114'1","reported_seq":15,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.529094+0000","last_change":"2024-05-28T06:22:15.990828+0000","last_active":"2024-05-28T06:22:18.529094+0000","last_peered":"2024-05-28T06:22:18.529094+0000","last_clean":"2024-05-28T06:22:18.529094+0000","last_became_active":"2024-05-28T06:22:15.990537+0000","last_became_peered":"2024-05-28T06:22:15.990537+0000","last_unstale":"2024-05-28T06:22:18.529094+0000","last_undegraded":"2024-05-28T06:22:18.529094+0000","last_fullsized":"2024-05-28T06:22:18.529094+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:51:43.927844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.241796+0000","last_change":"2024-05-28T06:22:17.000804+0000","last_active":"2024-05-28T06:22:18.241796+0000","last_peered":"2024-05-28T06:22:18.241796+0000","last_clean":"2024-05-28T06:22:18.241796+0000","last_became_active":"2024-05-28T06:22:16.998428+0000","last_became_peered":"2024-05-28T06:22:16.998428+0000","last_unstale":"2024-05-28T06:22:18.241796+0000","last_undegraded":"2024-05-28T06:22:18.241796+0000","last_fullsized":"2024-05-28T06:22:18.241796+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:43:26.170749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":1,"reported_epoch":112,"state":"creating+peering","last_fresh":"2024-05-28T06:22:14.958623+0000","last_change":"2024-05-28T06:22:14.958623+0000","last_active":"2024-05-28T06:22:14.938772+0000","last_peered":"2024-05-28T06:22:14.938772+0000","last_clean":"2024-05-28T06:22:14.938772+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:22:14.958623+0000","last_undegraded":"2024-05-28T06:22:14.958623+0000","last_fullsized":"2024-05-28T06:22:14.958623+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-28T06:22:15.950581+0000","last_change":"2024-05-28T06:22:15.950581+0000","last_active":"2024-05-28T06:22:15.950581+0000","last_peered":"2024-05-28T06:22:15.950581+0000","last_clean":"2024-05-28T06:22:15.950581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:22:15.950581+0000","last_undegraded":"2024-05-28T06:22:15.950581+0000","last_fullsized":"2024-05-28T06:22:15.950581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.242254+0000","last_change":"2024-05-28T06:22:15.988357+0000","last_active":"2024-05-28T06:22:18.242254+0000","last_peered":"2024-05-28T06:22:18.242254+0000","last_clean":"2024-05-28T06:22:18.242254+0000","last_became_active":"2024-05-28T06:22:15.988053+0000","last_became_peered":"2024-05-28T06:22:15.988053+0000","last_unstale":"2024-05-28T06:22:18.242254+0000","last_undegraded":"2024-05-28T06:22:18.242254+0000","last_fullsized":"2024-05-28T06:22:18.242254+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:24:37.584832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-28T06:22:15.950581+0000","last_change":"2024-05-28T06:22:15.950581+0000","last_active":"2024-05-28T06:22:15.950581+0000","last_peered":"2024-05-28T06:22:15.950581+0000","last_clean":"2024-05-28T06:22:15.950581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:22:15.950581+0000","last_undegraded":"2024-05-28T06:22:15.950581+0000","last_fullsized":"2024-05-28T06:22:15.950581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:17.999693+0000","last_change":"2024-05-28T06:22:15.984544+0000","last_active":"2024-05-28T06:22:17.999693+0000","last_peered":"2024-05-28T06:22:17.999693+0000","last_clean":"2024-05-28T06:22:17.999693+0000","last_became_active":"2024-05-28T06:22:15.984287+0000","last_became_peered":"2024-05-28T06:22:15.984287+0000","last_unstale":"2024-05-28T06:22:17.999693+0000","last_undegraded":"2024-05-28T06:22:17.999693+0000","last_fullsized":"2024-05-28T06:22:17.999693+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T18:04:33.437207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-28T06:22:15.950581+0000","last_change":"2024-05-28T06:22:15.950581+0000","last_active":"2024-05-28T06:22:15.950581+0000","last_peered":"2024-05-28T06:22:15.950581+0000","last_clean":"2024-05-28T06:22:15.950581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:22:15.950581+0000","last_undegraded":"2024-05-28T06:22:15.950581+0000","last_fullsized":"2024-05-28T06:22:15.950581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.529413+0000","last_change":"2024-05-28T06:22:15.989398+0000","last_active":"2024-05-28T06:22:18.529413+0000","last_peered":"2024-05-28T06:22:18.529413+0000","last_clean":"2024-05-28T06:22:18.529413+0000","last_became_active":"2024-05-28T06:22:15.989101+0000","last_became_peered":"2024-05-28T06:22:15.989101+0000","last_unstale":"2024-05-28T06:22:18.529413+0000","last_undegraded":"2024-05-28T06:22:18.529413+0000","last_fullsized":"2024-05-28T06:22:18.529413+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:43:46.372846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.529711+0000","last_change":"2024-05-28T06:22:16.993645+0000","last_active":"2024-05-28T06:22:18.529711+0000","last_peered":"2024-05-28T06:22:18.529711+0000","last_clean":"2024-05-28T06:22:18.529711+0000","last_became_active":"2024-05-28T06:22:16.992912+0000","last_became_peered":"2024-05-28T06:22:16.992912+0000","last_unstale":"2024-05-28T06:22:18.529711+0000","last_undegraded":"2024-05-28T06:22:18.529711+0000","last_fullsized":"2024-05-28T06:22:18.529711+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:16:11.794755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"114'3","reported_seq":17,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.242908+0000","last_change":"2024-05-28T06:22:15.998469+0000","last_active":"2024-05-28T06:22:18.242908+0000","last_peered":"2024-05-28T06:22:18.242908+0000","last_clean":"2024-05-28T06:22:18.242908+0000","last_became_active":"2024-05-28T06:22:15.998186+0000","last_became_peered":"2024-05-28T06:22:15.998186+0000","last_unstale":"2024-05-28T06:22:18.242908+0000","last_undegraded":"2024-05-28T06:22:18.242908+0000","last_fullsized":"2024-05-28T06:22:18.242908+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:23:41.228363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-28T06:22:15.950581+0000","last_change":"2024-05-28T06:22:15.950581+0000","last_active":"2024-05-28T06:22:15.950581+0000","last_peered":"2024-05-28T06:22:15.950581+0000","last_clean":"2024-05-28T06:22:15.950581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:22:15.950581+0000","last_undegraded":"2024-05-28T06:22:15.950581+0000","last_fullsized":"2024-05-28T06:22:15.950581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.14","version":"114'6","reported_seq":20,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.243514+0000","last_change":"2024-05-28T06:22:15.997183+0000","last_active":"2024-05-28T06:22:18.243514+0000","last_peered":"2024-05-28T06:22:18.243514+0000","last_clean":"2024-05-28T06:22:18.243514+0000","last_became_active":"2024-05-28T06:22:15.996809+0000","last_became_peered":"2024-05-28T06:22:15.996809+0000","last_unstale":"2024-05-28T06:22:18.243514+0000","last_undegraded":"2024-05-28T06:22:18.243514+0000","last_fullsized":"2024-05-28T06:22:18.243514+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:38:33.839493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.528917+0000","last_change":"2024-05-28T06:22:17.000291+0000","last_active":"2024-05-28T06:22:18.528917+0000","last_peered":"2024-05-28T06:22:18.528917+0000","last_clean":"2024-05-28T06:22:18.528917+0000","last_became_active":"2024-05-28T06:22:16.999910+0000","last_became_peered":"2024-05-28T06:22:16.999910+0000","last_unstale":"2024-05-28T06:22:18.528917+0000","last_undegraded":"2024-05-28T06:22:18.528917+0000","last_fullsized":"2024-05-28T06:22:18.528917+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:23:42.240172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"114'2","reported_seq":16,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.528553+0000","last_change":"2024-05-28T06:22:15.996773+0000","last_active":"2024-05-28T06:22:18.528553+0000","last_peered":"2024-05-28T06:22:18.528553+0000","last_clean":"2024-05-28T06:22:18.528553+0000","last_became_active":"2024-05-28T06:22:15.995681+0000","last_became_peered":"2024-05-28T06:22:15.995681+0000","last_unstale":"2024-05-28T06:22:18.528553+0000","last_undegraded":"2024-05-28T06:22:18.528553+0000","last_fullsized":"2024-05-28T06:22:18.528553+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:48:09.956343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.528770+0000","last_change":"2024-05-28T06:22:17.003164+0000","last_active":"2024-05-28T06:22:18.528770+0000","last_peered":"2024-05-28T06:22:18.528770+0000","last_clean":"2024-05-28T06:22:18.528770+0000","last_became_active":"2024-05-28T06:22:17.002906+0000","last_became_peered":"2024-05-28T06:22:17.002906+0000","last_unstale":"2024-05-28T06:22:18.528770+0000","last_undegraded":"2024-05-28T06:22:18.528770+0000","last_fullsized":"2024-05-28T06:22:18.528770+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:13:14.889721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":1,"reported_epoch":112,"state":"creating+peering","last_fresh":"2024-05-28T06:22:14.963301+0000","last_change":"2024-05-28T06:22:14.963301+0000","last_active":"2024-05-28T06:22:14.938772+0000","last_peered":"2024-05-28T06:22:14.938772+0000","last_clean":"2024-05-28T06:22:14.938772+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:22:14.963301+0000","last_undegraded":"2024-05-28T06:22:14.963301+0000","last_fullsized":"2024-05-28T06:22:14.963301+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.527440+0000","last_change":"2024-05-28T06:22:16.991808+0000","last_active":"2024-05-28T06:22:18.527440+0000","last_peered":"2024-05-28T06:22:18.527440+0000","last_clean":"2024-05-28T06:22:18.527440+0000","last_became_active":"2024-05-28T06:22:16.991494+0000","last_became_peered":"2024-05-28T06:22:16.991494+0000","last_unstale":"2024-05-28T06:22:18.527440+0000","last_undegraded":"2024-05-28T06:22:18.527440+0000","last_fullsized":"2024-05-28T06:22:18.527440+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:47:18.486697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"114'8","reported_seq":22,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:17.999084+0000","last_change":"2024-05-28T06:22:15.991378+0000","last_active":"2024-05-28T06:22:17.999084+0000","last_peered":"2024-05-28T06:22:17.999084+0000","last_clean":"2024-05-28T06:22:17.999084+0000","last_became_active":"2024-05-28T06:22:15.990388+0000","last_became_peered":"2024-05-28T06:22:15.990388+0000","last_unstale":"2024-05-28T06:22:17.999084+0000","last_undegraded":"2024-05-28T06:22:17.999084+0000","last_fullsized":"2024-05-28T06:22:17.999084+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:49:10.819709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:17.999777+0000","last_change":"2024-05-28T06:22:16.991114+0000","last_active":"2024-05-28T06:22:17.999777+0000","last_peered":"2024-05-28T06:22:17.999777+0000","last_clean":"2024-05-28T06:22:17.999777+0000","last_became_active":"2024-05-28T06:22:16.990159+0000","last_became_peered":"2024-05-28T06:22:16.990159+0000","last_unstale":"2024-05-28T06:22:17.999777+0000","last_undegraded":"2024-05-28T06:22:17.999777+0000","last_fullsized":"2024-05-28T06:22:17.999777+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:18:01.989158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"114'1","reported_seq":15,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.242742+0000","last_change":"2024-05-28T06:22:15.997418+0000","last_active":"2024-05-28T06:22:18.242742+0000","last_peered":"2024-05-28T06:22:18.242742+0000","last_clean":"2024-05-28T06:22:18.242742+0000","last_became_active":"2024-05-28T06:22:15.995786+0000","last_became_peered":"2024-05-28T06:22:15.995786+0000","last_unstale":"2024-05-28T06:22:18.242742+0000","last_undegraded":"2024-05-28T06:22:18.242742+0000","last_fullsized":"2024-05-28T06:22:18.242742+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:50:11.148336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.530295+0000","last_change":"2024-05-28T06:22:16.993861+0000","last_active":"2024-05-28T06:22:18.530295+0000","last_peered":"2024-05-28T06:22:18.530295+0000","last_clean":"2024-05-28T06:22:18.530295+0000","last_became_active":"2024-05-28T06:22:16.993602+0000","last_became_peered":"2024-05-28T06:22:16.993602+0000","last_unstale":"2024-05-28T06:22:18.530295+0000","last_undegraded":"2024-05-28T06:22:18.530295+0000","last_fullsized":"2024-05-28T06:22:18.530295+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:28:17.141535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"114'2","reported_seq":16,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:17.999975+0000","last_change":"2024-05-28T06:22:15.980689+0000","last_active":"2024-05-28T06:22:17.999975+0000","last_peered":"2024-05-28T06:22:17.999975+0000","last_clean":"2024-05-28T06:22:17.999975+0000","last_became_active":"2024-05-28T06:22:15.980139+0000","last_became_peered":"2024-05-28T06:22:15.980139+0000","last_unstale":"2024-05-28T06:22:17.999975+0000","last_undegraded":"2024-05-28T06:22:17.999975+0000","last_fullsized":"2024-05-28T06:22:17.999975+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:19:21.410923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":1,"reported_epoch":113,"state":"creating+peering","last_fresh":"2024-05-28T06:22:15.977950+0000","last_change":"2024-05-28T06:22:15.977950+0000","last_active":"2024-05-28T06:22:15.950581+0000","last_peered":"2024-05-28T06:22:15.950581+0000","last_clean":"2024-05-28T06:22:15.950581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:22:15.977950+0000","last_undegraded":"2024-05-28T06:22:15.977950+0000","last_fullsized":"2024-05-28T06:22:15.977950+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":1,"reported_epoch":112,"state":"creating+peering","last_fresh":"2024-05-28T06:22:14.964473+0000","last_change":"2024-05-28T06:22:14.964473+0000","last_active":"2024-05-28T06:22:14.938772+0000","last_peered":"2024-05-28T06:22:14.938772+0000","last_clean":"2024-05-28T06:22:14.938772+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:22:14.964473+0000","last_undegraded":"2024-05-28T06:22:14.964473+0000","last_fullsized":"2024-05-28T06:22:14.964473+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":1,"reported_epoch":113,"state":"creating+peering","last_fresh":"2024-05-28T06:22:15.978074+0000","last_change":"2024-05-28T06:22:15.978074+0000","last_active":"2024-05-28T06:22:15.950581+0000","last_peered":"2024-05-28T06:22:15.950581+0000","last_clean":"2024-05-28T06:22:15.950581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:22:15.978074+0000","last_undegraded":"2024-05-28T06:22:15.978074+0000","last_fullsized":"2024-05-28T06:22:15.978074+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":1,"reported_epoch":112,"state":"creating+peering","last_fresh":"2024-05-28T06:22:14.968146+0000","last_change":"2024-05-28T06:22:14.968146+0000","last_active":"2024-05-28T06:22:14.938772+0000","last_peered":"2024-05-28T06:22:14.938772+0000","last_clean":"2024-05-28T06:22:14.938772+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:22:14.968146+0000","last_undegraded":"2024-05-28T06:22:14.968146+0000","last_fullsized":"2024-05-28T06:22:14.968146+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":1,"reported_epoch":113,"state":"creating+peering","last_fresh":"2024-05-28T06:22:15.977889+0000","last_change":"2024-05-28T06:22:15.977889+0000","last_active":"2024-05-28T06:22:15.950581+0000","last_peered":"2024-05-28T06:22:15.950581+0000","last_clean":"2024-05-28T06:22:15.950581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:22:15.977889+0000","last_undegraded":"2024-05-28T06:22:15.977889+0000","last_fullsized":"2024-05-28T06:22:15.977889+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"114'2","reported_seq":16,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.528872+0000","last_change":"2024-05-28T06:22:15.991711+0000","last_active":"2024-05-28T06:22:18.528872+0000","last_peered":"2024-05-28T06:22:18.528872+0000","last_clean":"2024-05-28T06:22:18.528872+0000","last_became_active":"2024-05-28T06:22:15.991323+0000","last_became_peered":"2024-05-28T06:22:15.991323+0000","last_unstale":"2024-05-28T06:22:18.528872+0000","last_undegraded":"2024-05-28T06:22:18.528872+0000","last_fullsized":"2024-05-28T06:22:18.528872+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:02:22.717474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.242729+0000","last_change":"2024-05-28T06:22:17.003257+0000","last_active":"2024-05-28T06:22:18.242729+0000","last_peered":"2024-05-28T06:22:18.242729+0000","last_clean":"2024-05-28T06:22:18.242729+0000","last_became_active":"2024-05-28T06:22:17.000808+0000","last_became_peered":"2024-05-28T06:22:17.000808+0000","last_unstale":"2024-05-28T06:22:18.242729+0000","last_undegraded":"2024-05-28T06:22:18.242729+0000","last_fullsized":"2024-05-28T06:22:18.242729+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:27:48.276213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:17.999010+0000","last_change":"2024-05-28T06:22:15.994076+0000","last_active":"2024-05-28T06:22:17.999010+0000","last_peered":"2024-05-28T06:22:17.999010+0000","last_clean":"2024-05-28T06:22:17.999010+0000","last_became_active":"2024-05-28T06:22:15.993488+0000","last_became_peered":"2024-05-28T06:22:15.993488+0000","last_unstale":"2024-05-28T06:22:17.999010+0000","last_undegraded":"2024-05-28T06:22:17.999010+0000","last_fullsized":"2024-05-28T06:22:17.999010+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:08:35.517640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":1,"reported_epoch":113,"state":"creating+peering","last_fresh":"2024-05-28T06:22:15.977812+0000","last_change":"2024-05-28T06:22:15.977812+0000","last_active":"2024-05-28T06:22:15.950581+0000","last_peered":"2024-05-28T06:22:15.950581+0000","last_clean":"2024-05-28T06:22:15.950581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:22:15.977812+0000","last_undegraded":"2024-05-28T06:22:15.977812+0000","last_fullsized":"2024-05-28T06:22:15.977812+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.527971+0000","last_change":"2024-05-28T06:22:15.989588+0000","last_active":"2024-05-28T06:22:18.527971+0000","last_peered":"2024-05-28T06:22:18.527971+0000","last_clean":"2024-05-28T06:22:18.527971+0000","last_became_active":"2024-05-28T06:22:15.989331+0000","last_became_peered":"2024-05-28T06:22:15.989331+0000","last_unstale":"2024-05-28T06:22:18.527971+0000","last_undegraded":"2024-05-28T06:22:18.527971+0000","last_fullsized":"2024-05-28T06:22:18.527971+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:19:21.410923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.527956+0000","last_change":"2024-05-28T06:22:16.993785+0000","last_active":"2024-05-28T06:22:18.527956+0000","last_peered":"2024-05-28T06:22:18.527956+0000","last_clean":"2024-05-28T06:22:18.527956+0000","last_became_active":"2024-05-28T06:22:16.993402+0000","last_became_peered":"2024-05-28T06:22:16.993402+0000","last_unstale":"2024-05-28T06:22:18.527956+0000","last_undegraded":"2024-05-28T06:22:18.527956+0000","last_fullsized":"2024-05-28T06:22:18.527956+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:06:13.719079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.242342+0000","last_change":"2024-05-28T06:22:15.998948+0000","last_active":"2024-05-28T06:22:18.242342+0000","last_peered":"2024-05-28T06:22:18.242342+0000","last_clean":"2024-05-28T06:22:18.242342+0000","last_became_active":"2024-05-28T06:22:15.998205+0000","last_became_peered":"2024-05-28T06:22:15.998205+0000","last_unstale":"2024-05-28T06:22:18.242342+0000","last_undegraded":"2024-05-28T06:22:18.242342+0000","last_fullsized":"2024-05-28T06:22:18.242342+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:58:34.652147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-28T06:22:15.950581+0000","last_change":"2024-05-28T06:22:15.950581+0000","last_active":"2024-05-28T06:22:15.950581+0000","last_peered":"2024-05-28T06:22:15.950581+0000","last_clean":"2024-05-28T06:22:15.950581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:22:15.950581+0000","last_undegraded":"2024-05-28T06:22:15.950581+0000","last_fullsized":"2024-05-28T06:22:15.950581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.7","version":"114'2","reported_seq":16,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.528934+0000","last_change":"2024-05-28T06:22:15.990864+0000","last_active":"2024-05-28T06:22:18.528934+0000","last_peered":"2024-05-28T06:22:18.528934+0000","last_clean":"2024-05-28T06:22:18.528934+0000","last_became_active":"2024-05-28T06:22:15.990559+0000","last_became_peered":"2024-05-28T06:22:15.990559+0000","last_unstale":"2024-05-28T06:22:18.528934+0000","last_undegraded":"2024-05-28T06:22:18.528934+0000","last_fullsized":"2024-05-28T06:22:18.528934+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:51:36.120307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.242633+0000","last_change":"2024-05-28T06:22:16.992471+0000","last_active":"2024-05-28T06:22:18.242633+0000","last_peered":"2024-05-28T06:22:18.242633+0000","last_clean":"2024-05-28T06:22:18.242633+0000","last_became_active":"2024-05-28T06:22:16.992135+0000","last_became_peered":"2024-05-28T06:22:16.992135+0000","last_unstale":"2024-05-28T06:22:18.242633+0000","last_undegraded":"2024-05-28T06:22:18.242633+0000","last_fullsized":"2024-05-28T06:22:18.242633+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:57:13.651588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":1,"reported_epoch":112,"state":"creating+peering","last_fresh":"2024-05-28T06:22:14.963417+0000","last_change":"2024-05-28T06:22:14.963417+0000","last_active":"2024-05-28T06:22:14.938772+0000","last_peered":"2024-05-28T06:22:14.938772+0000","last_clean":"2024-05-28T06:22:14.938772+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:22:14.963417+0000","last_undegraded":"2024-05-28T06:22:14.963417+0000","last_fullsized":"2024-05-28T06:22:14.963417+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.243620+0000","last_change":"2024-05-28T06:22:16.993934+0000","last_active":"2024-05-28T06:22:18.243620+0000","last_peered":"2024-05-28T06:22:18.243620+0000","last_clean":"2024-05-28T06:22:18.243620+0000","last_became_active":"2024-05-28T06:22:16.993249+0000","last_became_peered":"2024-05-28T06:22:16.993249+0000","last_unstale":"2024-05-28T06:22:18.243620+0000","last_undegraded":"2024-05-28T06:22:18.243620+0000","last_fullsized":"2024-05-28T06:22:18.243620+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:21:11.675368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.242990+0000","last_change":"2024-05-28T06:22:15.998999+0000","last_active":"2024-05-28T06:22:18.242990+0000","last_peered":"2024-05-28T06:22:18.242990+0000","last_clean":"2024-05-28T06:22:18.242990+0000","last_became_active":"2024-05-28T06:22:15.998631+0000","last_became_peered":"2024-05-28T06:22:15.998631+0000","last_unstale":"2024-05-28T06:22:18.242990+0000","last_undegraded":"2024-05-28T06:22:18.242990+0000","last_fullsized":"2024-05-28T06:22:18.242990+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:26:17.626246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-28T06:22:15.950581+0000","last_change":"2024-05-28T06:22:15.950581+0000","last_active":"2024-05-28T06:22:15.950581+0000","last_peered":"2024-05-28T06:22:15.950581+0000","last_clean":"2024-05-28T06:22:15.950581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:22:15.950581+0000","last_undegraded":"2024-05-28T06:22:15.950581+0000","last_fullsized":"2024-05-28T06:22:15.950581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.529805+0000","last_change":"2024-05-28T06:22:15.995193+0000","last_active":"2024-05-28T06:22:18.529805+0000","last_peered":"2024-05-28T06:22:18.529805+0000","last_clean":"2024-05-28T06:22:18.529805+0000","last_became_active":"2024-05-28T06:22:15.994748+0000","last_became_peered":"2024-05-28T06:22:15.994748+0000","last_unstale":"2024-05-28T06:22:18.529805+0000","last_undegraded":"2024-05-28T06:22:18.529805+0000","last_fullsized":"2024-05-28T06:22:18.529805+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:08:38.270230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-28T06:22:15.950581+0000","last_change":"2024-05-28T06:22:15.950581+0000","last_active":"2024-05-28T06:22:15.950581+0000","last_peered":"2024-05-28T06:22:15.950581+0000","last_clean":"2024-05-28T06:22:15.950581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:22:15.950581+0000","last_undegraded":"2024-05-28T06:22:15.950581+0000","last_fullsized":"2024-05-28T06:22:15.950581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":1,"reported_epoch":112,"state":"creating+peering","last_fresh":"2024-05-28T06:22:14.968289+0000","last_change":"2024-05-28T06:22:14.968289+0000","last_active":"2024-05-28T06:22:14.938772+0000","last_peered":"2024-05-28T06:22:14.938772+0000","last_clean":"2024-05-28T06:22:14.938772+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:22:14.968289+0000","last_undegraded":"2024-05-28T06:22:14.968289+0000","last_fullsized":"2024-05-28T06:22:14.968289+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-28T06:22:15.950581+0000","last_change":"2024-05-28T06:22:15.950581+0000","last_active":"2024-05-28T06:22:15.950581+0000","last_peered":"2024-05-28T06:22:15.950581+0000","last_clean":"2024-05-28T06:22:15.950581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:22:15.950581+0000","last_undegraded":"2024-05-28T06:22:15.950581+0000","last_fullsized":"2024-05-28T06:22:15.950581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":1,"reported_epoch":112,"state":"creating+peering","last_fresh":"2024-05-28T06:22:14.958443+0000","last_change":"2024-05-28T06:22:14.958443+0000","last_active":"2024-05-28T06:22:14.938772+0000","last_peered":"2024-05-28T06:22:14.938772+0000","last_clean":"2024-05-28T06:22:14.938772+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:22:14.958443+0000","last_undegraded":"2024-05-28T06:22:14.958443+0000","last_fullsized":"2024-05-28T06:22:14.958443+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":1,"reported_epoch":113,"state":"creating+peering","last_fresh":"2024-05-28T06:22:15.976482+0000","last_change":"2024-05-28T06:22:15.976482+0000","last_active":"2024-05-28T06:22:15.950581+0000","last_peered":"2024-05-28T06:22:15.950581+0000","last_clean":"2024-05-28T06:22:15.950581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:22:15.976482+0000","last_undegraded":"2024-05-28T06:22:15.976482+0000","last_fullsized":"2024-05-28T06:22:15.976482+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:17.999121+0000","last_change":"2024-05-28T06:22:15.984208+0000","last_active":"2024-05-28T06:22:17.999121+0000","last_peered":"2024-05-28T06:22:17.999121+0000","last_clean":"2024-05-28T06:22:17.999121+0000","last_became_active":"2024-05-28T06:22:15.983816+0000","last_became_peered":"2024-05-28T06:22:15.983816+0000","last_unstale":"2024-05-28T06:22:17.999121+0000","last_undegraded":"2024-05-28T06:22:17.999121+0000","last_fullsized":"2024-05-28T06:22:17.999121+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:30:43.782132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.242161+0000","last_change":"2024-05-28T06:22:16.996830+0000","last_active":"2024-05-28T06:22:18.242161+0000","last_peered":"2024-05-28T06:22:18.242161+0000","last_clean":"2024-05-28T06:22:18.242161+0000","last_became_active":"2024-05-28T06:22:16.990102+0000","last_became_peered":"2024-05-28T06:22:16.990102+0000","last_unstale":"2024-05-28T06:22:18.242161+0000","last_undegraded":"2024-05-28T06:22:18.242161+0000","last_fullsized":"2024-05-28T06:22:18.242161+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T18:04:15.407639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.242532+0000","last_change":"2024-05-28T06:22:15.998500+0000","last_active":"2024-05-28T06:22:18.242532+0000","last_peered":"2024-05-28T06:22:18.242532+0000","last_clean":"2024-05-28T06:22:18.242532+0000","last_became_active":"2024-05-28T06:22:15.998176+0000","last_became_peered":"2024-05-28T06:22:15.998176+0000","last_unstale":"2024-05-28T06:22:18.242532+0000","last_undegraded":"2024-05-28T06:22:18.242532+0000","last_fullsized":"2024-05-28T06:22:18.242532+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:17:13.644125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.527150+0000","last_change":"2024-05-28T06:22:17.008104+0000","last_active":"2024-05-28T06:22:18.527150+0000","last_peered":"2024-05-28T06:22:18.527150+0000","last_clean":"2024-05-28T06:22:18.527150+0000","last_became_active":"2024-05-28T06:22:17.007752+0000","last_became_peered":"2024-05-28T06:22:17.007752+0000","last_unstale":"2024-05-28T06:22:18.527150+0000","last_undegraded":"2024-05-28T06:22:18.527150+0000","last_fullsized":"2024-05-28T06:22:18.527150+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:08:39.984746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":1,"reported_epoch":112,"state":"creating+peering","last_fresh":"2024-05-28T06:22:14.968070+0000","last_change":"2024-05-28T06:22:14.968070+0000","last_active":"2024-05-28T06:22:14.938772+0000","last_peered":"2024-05-28T06:22:14.938772+0000","last_clean":"2024-05-28T06:22:14.938772+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:22:14.968070+0000","last_undegraded":"2024-05-28T06:22:14.968070+0000","last_fullsized":"2024-05-28T06:22:14.968070+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":350,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-05-28T06:22:15.989329+0000","last_change":"2024-05-28T06:04:01.039129+0000","last_active":"2024-05-28T06:22:15.989329+0000","last_peered":"2024-05-28T06:22:15.989329+0000","last_clean":"2024-05-28T06:22:15.989329+0000","last_became_active":"2024-05-28T06:04:01.033906+0000","last_became_peered":"2024-05-28T06:04:01.033906+0000","last_unstale":"2024-05-28T06:22:15.989329+0000","last_undegraded":"2024-05-28T06:22:15.989329+0000","last_fullsized":"2024-05-28T06:22:15.989329+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:03:59.864237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:03:59.864237+0000","last_clean_scrub_stamp":"2024-05-28T06:03:59.864237+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:17:36.440245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":86,"acting":86,"num_store_stats":5},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":1758,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":39,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31,"ondisk_log_size":31,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575070,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":127460,"kb_used_data":1940,"kb_used_omap":1,"kb_used_meta":125502,"kb_avail":94244380,"statfs":{"total":96636764160,"available":96506245120,"internally_reserved":0,"allocated":1986560,"data_stored":1197121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":128514506},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575072,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":126868,"kb_used_data":1348,"kb_used_omap":1,"kb_used_meta":125502,"kb_avail":94244972,"statfs":{"total":96636764160,"available":96506851328,"internally_reserved":0,"allocated":1380352,"data_stored":600880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":128514502},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575070,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81040,"kb_used_data":1348,"kb_used_omap":1,"kb_used_meta":79678,"kb_avail":94290800,"statfs":{"total":96636764160,"available":96553779200,"internally_reserved":0,"allocated":1380352,"data_stored":600880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":81590725},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575070,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":86616,"kb_used_data":1348,"kb_used_omap":1,"kb_used_meta":85246,"kb_avail":94285224,"statfs":{"total":96636764160,"available":96548069376,"internally_reserved":0,"allocated":1380352,"data_stored":600880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87292357},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575073,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":131096,"kb_used_data":1412,"kb_used_omap":1,"kb_used_meta":129662,"kb_avail":94240744,"statfs":{"total":96636764160,"available":96502521856,"internally_reserved":0,"allocated":1445888,"data_stored":627953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":132774343},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575072,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":144960,"kb_used_data":1396,"kb_used_omap":1,"kb_used_meta":143550,"kb_avail":94226880,"statfs":{"total":96636764160,"available":96488325120,"internally_reserved":0,"allocated":1429504,"data_stored":626271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":146995653},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575072,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":126920,"kb_used_data":1404,"kb_used_omap":1,"kb_used_meta":125502,"kb_avail":94244920,"statfs":{"total":96636764160,"available":96506798080,"internally_reserved":0,"allocated":1437696,"data_stored":626395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":128514501},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575071,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":87260,"kb_used_data":1996,"kb_used_omap":1,"kb_used_meta":85246,"kb_avail":94284580,"statfs":{"total":96636764160,"available":96547409920,"internally_reserved":0,"allocated":2043904,"data_stored":1218363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87292357},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-28T06:22:21.430 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-28T06:22:21.670 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.674+0000 7fef08800640 1 Processor -- start 2024-05-28T06:22:21.670 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.674+0000 7fef08800640 1 -- start start 2024-05-28T06:22:21.671 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.675+0000 7fef08800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fef0006f860 0x7fef000b3150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:21.671 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.676+0000 7fef08800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fef000d0bb0 0x7fef000ad130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:21.671 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.676+0000 7fef08800640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef0006ed10 con 0x7fef000b3690 2024-05-28T06:22:21.671 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.676+0000 7fef08800640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fef0015cb00 con 0x7fef000d0bb0 2024-05-28T06:22:21.671 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.676+0000 7fef08800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fef000b3cc0 con 0x7fef0006f860 2024-05-28T06:22:21.672 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.676+0000 7fef05400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fef000b3690 0x7fef000d04a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:35102/0 (socket says 172.21.15.96:35102) 2024-05-28T06:22:21.672 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.676+0000 7fef05400640 1 -- 172.21.15.96:0/360219462 learned_addr learned my addr 172.21.15.96:0/360219462 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:22:21.673 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.676+0000 7fef05e00640 1 --2- 172.21.15.96:0/360219462 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fef0006f860 0x7fef000b3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:21.673 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.676+0000 7fef06800640 1 --2- 172.21.15.96:0/360219462 >> v2:172.21.15.123:3300/0 conn(0x7fef000d0bb0 0x7fef000ad130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:21.673 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.676+0000 7fef04a00640 1 -- 172.21.15.96:0/360219462 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4054739059 0 0) 0x7feef0001110 con 0x7fef000b3690 2024-05-28T06:22:21.673 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.676+0000 7fef04a00640 1 -- 172.21.15.96:0/360219462 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feee8003520 con 0x7fef000b3690 2024-05-28T06:22:21.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.677+0000 7fef04a00640 1 -- 172.21.15.96:0/360219462 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 331868107 0 0) 0x7feee8003520 con 0x7fef000b3690 2024-05-28T06:22:21.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.677+0000 7fef04a00640 1 -- 172.21.15.96:0/360219462 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fef0006f860 msgr2=0x7fef000b3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:21.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.677+0000 7fef04a00640 1 --2- 172.21.15.96:0/360219462 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fef0006f860 0x7fef000b3150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:21.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.677+0000 7fef04a00640 1 -- 172.21.15.96:0/360219462 >> v2:172.21.15.123:3300/0 conn(0x7fef000d0bb0 msgr2=0x7fef000ad130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:21.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.677+0000 7fef04a00640 1 --2- 172.21.15.96:0/360219462 >> v2:172.21.15.123:3300/0 conn(0x7fef000d0bb0 0x7fef000ad130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:21.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.677+0000 7fef04a00640 1 -- 172.21.15.96:0/360219462 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef000b3fa0 con 0x7fef000b3690 2024-05-28T06:22:21.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.677+0000 7fef04a00640 1 -- 172.21.15.96:0/360219462 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7feef0001840 con 0x7fef000b3690 2024-05-28T06:22:21.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.677+0000 7fef04a00640 1 -- 172.21.15.96:0/360219462 >> v1:172.21.15.96:6789/0 conn(0x7fef000b3690 legacy=0x7fef000d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:21.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.677+0000 7fef04a00640 1 --2- 172.21.15.96:0/360219462 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7feee80039b0 0x7feee8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:21.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.677+0000 7fef04a00640 1 --2- 172.21.15.96:0/360219462 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feee8004800 0x7feee8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:21.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.677+0000 7fef04a00640 1 --2- 172.21.15.96:0/360219462 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feee80051c0 0x7feee800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:21.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.677+0000 7fef04a00640 1 -- 172.21.15.96:0/360219462 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fef0015cb00 con 0x7feee80051c0 2024-05-28T06:22:21.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.677+0000 7fef04a00640 1 -- 172.21.15.96:0/360219462 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fef000b3cc0 con 0x7feee8004800 2024-05-28T06:22:21.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.677+0000 7fef04a00640 1 -- 172.21.15.96:0/360219462 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7feee800bf80 con 0x7feee80039b0 2024-05-28T06:22:21.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.677+0000 7fef05e00640 1 --2- 172.21.15.96:0/360219462 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fef0006f860 0x7fef000b3150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-28T06:22:21.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.677+0000 7fef06800640 1 --2- 172.21.15.96:0/360219462 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feee80051c0 0x7feee800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:21.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.678+0000 7fef06800640 1 -- 172.21.15.96:0/360219462 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7feee80039b0 msgr2=0x7feee8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:21.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.678+0000 7fef06800640 1 --2- 172.21.15.96:0/360219462 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7feee80039b0 0x7feee8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:21.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.678+0000 7fef06800640 1 -- 172.21.15.96:0/360219462 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feee8004800 msgr2=0x7feee8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:21.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.678+0000 7fef06800640 1 --2- 172.21.15.96:0/360219462 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feee8004800 0x7feee8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:21.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.678+0000 7fef06800640 1 -- 172.21.15.96:0/360219462 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feee8003520 con 0x7feee80051c0 2024-05-28T06:22:21.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.679+0000 7fef06800640 1 --2- 172.21.15.96:0/360219462 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feee80051c0 0x7feee800b9c0 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7fef000d1360 tx=0x7feee8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8568e4fd3db8186 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:21.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.679+0000 7fef04a00640 1 -- 172.21.15.96:0/360219462 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feefc009c10 con 0x7feee80051c0 2024-05-28T06:22:21.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.679+0000 7fef04a00640 1 -- 172.21.15.96:0/360219462 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feefc036070 con 0x7feee80051c0 2024-05-28T06:22:21.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.679+0000 7fef04a00640 1 -- 172.21.15.96:0/360219462 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feefc031810 con 0x7feee80051c0 2024-05-28T06:22:21.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.679+0000 7fef08800640 1 -- 172.21.15.96:0/360219462 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feee80051c0 msgr2=0x7feee800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:21.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.679+0000 7fef08800640 1 --2- 172.21.15.96:0/360219462 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feee80051c0 0x7feee800b9c0 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7fef000d1360 tx=0x7feee8005830 comp rx=0 tx=0).stop 2024-05-28T06:22:21.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.679+0000 7fef06800640 1 -- 172.21.15.96:0/360219462 reap_dead start 2024-05-28T06:22:21.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.680+0000 7fef08800640 1 -- 172.21.15.96:0/360219462 shutdown_connections 2024-05-28T06:22:21.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.680+0000 7fef08800640 1 -- 172.21.15.96:0/360219462 >> 172.21.15.96:0/360219462 conn(0x7fef0006bae0 msgr2=0x7fef0006bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:21.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.680+0000 7fef08800640 1 -- 172.21.15.96:0/360219462 shutdown_connections 2024-05-28T06:22:21.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.680+0000 7fef08800640 1 -- 172.21.15.96:0/360219462 wait complete. 2024-05-28T06:22:21.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.681+0000 7fef08800640 1 Processor -- start 2024-05-28T06:22:21.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.682+0000 7fef08800640 1 -- start start 2024-05-28T06:22:21.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.682+0000 7fef08800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7feee80051c0 0x7fef001f3f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:21.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.683+0000 7fef08800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef000cbb50 0x7fef000cbfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:21.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.683+0000 7fef08800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef0015e5c0 0x7fef0015ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:21.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.683+0000 7fef08800640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fef000b4840 con 0x7fef0015e5c0 2024-05-28T06:22:21.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.683+0000 7fef08800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fef000b4560 con 0x7fef000cbb50 2024-05-28T06:22:21.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.683+0000 7fef08800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fef000b3150 con 0x7feee80051c0 2024-05-28T06:22:21.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.683+0000 7fef06800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef0015e5c0 0x7fef0015ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:21.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.683+0000 7fef06800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef0015e5c0 0x7fef0015ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:60214/0 (socket says 172.21.15.96:60214) 2024-05-28T06:22:21.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.683+0000 7fef06800640 1 -- 172.21.15.96:0/764921383 learned_addr learned my addr 172.21.15.96:0/764921383 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:22:21.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.683+0000 7fef05e00640 1 --2- 172.21.15.96:0/764921383 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7feee80051c0 0x7fef001f3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:21.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.683+0000 7fef05400640 1 --2- 172.21.15.96:0/764921383 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef000cbb50 0x7fef000cbfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:21.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.683+0000 7fef06800640 1 -- 172.21.15.96:0/764921383 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7feee80051c0 msgr2=0x7fef001f3f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:21.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.683+0000 7fef06800640 1 --2- 172.21.15.96:0/764921383 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7feee80051c0 0x7fef001f3f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:21.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.683+0000 7fef06800640 1 -- 172.21.15.96:0/764921383 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef000cbb50 msgr2=0x7fef000cbfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:21.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.683+0000 7fef06800640 1 --2- 172.21.15.96:0/764921383 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef000cbb50 0x7fef000cbfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:21.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.683+0000 7fef06800640 1 -- 172.21.15.96:0/764921383 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef0015cd10 con 0x7fef0015e5c0 2024-05-28T06:22:21.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.684+0000 7fef06800640 1 --2- 172.21.15.96:0/764921383 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef0015e5c0 0x7fef0015ea40 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7feefc00e230 tx=0x7feefc009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:21.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.684+0000 7feefa000640 1 -- 172.21.15.96:0/764921383 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feefc009c10 con 0x7fef0015e5c0 2024-05-28T06:22:21.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.684+0000 7feefa000640 1 -- 172.21.15.96:0/764921383 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feefc045650 con 0x7fef0015e5c0 2024-05-28T06:22:21.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.684+0000 7feefa000640 1 -- 172.21.15.96:0/764921383 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feefc045a40 con 0x7fef0015e5c0 2024-05-28T06:22:21.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.684+0000 7fef08800640 1 -- 172.21.15.96:0/764921383 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef0015cfb0 con 0x7fef0015e5c0 2024-05-28T06:22:21.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.684+0000 7fef08800640 1 -- 172.21.15.96:0/764921383 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef0015d570 con 0x7fef0015e5c0 2024-05-28T06:22:21.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.685+0000 7fef08800640 1 -- 172.21.15.96:0/764921383 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef000689e0 con 0x7fef0015e5c0 2024-05-28T06:22:21.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.686+0000 7feefa000640 1 -- 172.21.15.96:0/764921383 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 37) ==== 93182+0+0 (secure 0 0 0) 0x7feefc005d10 con 0x7fef0015e5c0 2024-05-28T06:22:21.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.687+0000 7feefa000640 1 --2- 172.21.15.96:0/764921383 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7feee406e090 0x7feee4070550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:21.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.687+0000 7feefa000640 1 -- 172.21.15.96:0/764921383 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 5765+0+0 (secure 0 0 0) 0x7feefc0ab700 con 0x7fef0015e5c0 2024-05-28T06:22:21.683 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.688+0000 7fef05e00640 1 --2- 172.21.15.96:0/764921383 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7feee406e090 0x7feee4070550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:21.684 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.689+0000 7fef05e00640 1 --2- 172.21.15.96:0/764921383 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7feee406e090 0x7feee4070550 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7feef4006fd0 tx=0x7feef4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:21.687 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.692+0000 7feefa000640 1 -- 172.21.15.96:0/764921383 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7feefc02e050 con 0x7fef0015e5c0 2024-05-28T06:22:21.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.888+0000 7fef08800640 1 -- 172.21.15.96:0/764921383 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fef000ae060 con 0x7feee406e090 2024-05-28T06:22:21.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.907+0000 7feefa000640 1 -- 172.21.15.96:0/764921383 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+247562 (secure 0 0 0) 0x7fef000ae060 con 0x7feee406e090 2024-05-28T06:22:21.906 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:22:21.907 INFO:teuthology.orchestra.run.smithi096.stderr:dumped all 2024-05-28T06:22:21.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.912+0000 7fef08800640 1 -- 172.21.15.96:0/764921383 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7feee406e090 msgr2=0x7feee4070550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:21.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.912+0000 7fef08800640 1 --2- 172.21.15.96:0/764921383 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7feee406e090 0x7feee4070550 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7feef4006fd0 tx=0x7feef4008040 comp rx=0 tx=0).stop 2024-05-28T06:22:21.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.912+0000 7fef08800640 1 -- 172.21.15.96:0/764921383 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef0015e5c0 msgr2=0x7fef0015ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:21.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.912+0000 7fef08800640 1 --2- 172.21.15.96:0/764921383 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef0015e5c0 0x7fef0015ea40 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7feefc00e230 tx=0x7feefc009750 comp rx=0 tx=0).stop 2024-05-28T06:22:21.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.912+0000 7fef08800640 1 -- 172.21.15.96:0/764921383 shutdown_connections 2024-05-28T06:22:21.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.912+0000 7fef08800640 1 --2- 172.21.15.96:0/764921383 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7feee406e090 0x7feee4070550 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:21.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.912+0000 7fef08800640 1 --2- 172.21.15.96:0/764921383 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef0015e5c0 0x7fef0015ea40 unknown :-1 s=CLOSED pgs=1815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:21.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.912+0000 7fef08800640 1 --2- 172.21.15.96:0/764921383 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef000cbb50 0x7fef000cbfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:21.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.912+0000 7fef08800640 1 --2- 172.21.15.96:0/764921383 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7feee80051c0 0x7fef001f3f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:21.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.912+0000 7fef08800640 1 -- 172.21.15.96:0/764921383 >> 172.21.15.96:0/764921383 conn(0x7fef0006bae0 msgr2=0x7fef000aefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:21.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.912+0000 7fef08800640 1 -- 172.21.15.96:0/764921383 shutdown_connections 2024-05-28T06:22:21.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:21.913+0000 7fef08800640 1 -- 172.21.15.96:0/764921383 wait complete. 2024-05-28T06:22:21.927 INFO:teuthology.orchestra.run.smithi096.stdout:{"pg_ready":true,"pg_map":{"version":658,"stamp":"2024-05-28T06:22:19.964198+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592126,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":198,"num_write_kb":2428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148,"ondisk_log_size":148,"up":152,"acting":152,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":104,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":912220,"kb_used_data":12192,"kb_used_omap":12,"kb_used_meta":899891,"kb_avail":754062500,"statfs":{"total":773094113280,"available":772160000000,"internally_reserved":0,"allocated":12484608,"data_stored":6098743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":921488944},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1758,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":39,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.008041"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.242686+0000","last_change":"2024-05-28T06:22:17.004588+0000","last_active":"2024-05-28T06:22:18.242686+0000","last_peered":"2024-05-28T06:22:18.242686+0000","last_clean":"2024-05-28T06:22:18.242686+0000","last_became_active":"2024-05-28T06:22:17.003523+0000","last_became_peered":"2024-05-28T06:22:17.003523+0000","last_unstale":"2024-05-28T06:22:18.242686+0000","last_undegraded":"2024-05-28T06:22:18.242686+0000","last_fullsized":"2024-05-28T06:22:18.242686+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:04:35.674926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-28T06:22:15.950581+0000","last_change":"2024-05-28T06:22:15.950581+0000","last_active":"2024-05-28T06:22:15.950581+0000","last_peered":"2024-05-28T06:22:15.950581+0000","last_clean":"2024-05-28T06:22:15.950581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:22:15.950581+0000","last_undegraded":"2024-05-28T06:22:15.950581+0000","last_fullsized":"2024-05-28T06:22:15.950581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":1,"reported_epoch":113,"state":"creating+peering","last_fresh":"2024-05-28T06:22:15.985954+0000","last_change":"2024-05-28T06:22:15.985954+0000","last_active":"2024-05-28T06:22:15.950581+0000","last_peered":"2024-05-28T06:22:15.950581+0000","last_clean":"2024-05-28T06:22:15.950581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:22:15.985954+0000","last_undegraded":"2024-05-28T06:22:15.985954+0000","last_fullsized":"2024-05-28T06:22:15.985954+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.242843+0000","last_change":"2024-05-28T06:22:16.994809+0000","last_active":"2024-05-28T06:22:18.242843+0000","last_peered":"2024-05-28T06:22:18.242843+0000","last_clean":"2024-05-28T06:22:18.242843+0000","last_became_active":"2024-05-28T06:22:16.994548+0000","last_became_peered":"2024-05-28T06:22:16.994548+0000","last_unstale":"2024-05-28T06:22:18.242843+0000","last_undegraded":"2024-05-28T06:22:18.242843+0000","last_fullsized":"2024-05-28T06:22:18.242843+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:59:36.297627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:17.999386+0000","last_change":"2024-05-28T06:22:16.996834+0000","last_active":"2024-05-28T06:22:17.999386+0000","last_peered":"2024-05-28T06:22:17.999386+0000","last_clean":"2024-05-28T06:22:17.999386+0000","last_became_active":"2024-05-28T06:22:16.996543+0000","last_became_peered":"2024-05-28T06:22:16.996543+0000","last_unstale":"2024-05-28T06:22:17.999386+0000","last_undegraded":"2024-05-28T06:22:17.999386+0000","last_fullsized":"2024-05-28T06:22:17.999386+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:23:11.730784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.528701+0000","last_change":"2024-05-28T06:22:16.995996+0000","last_active":"2024-05-28T06:22:18.528701+0000","last_peered":"2024-05-28T06:22:18.528701+0000","last_clean":"2024-05-28T06:22:18.528701+0000","last_became_active":"2024-05-28T06:22:16.995714+0000","last_became_peered":"2024-05-28T06:22:16.995714+0000","last_unstale":"2024-05-28T06:22:18.528701+0000","last_undegraded":"2024-05-28T06:22:18.528701+0000","last_fullsized":"2024-05-28T06:22:18.528701+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:04:42.043002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-28T06:22:15.950581+0000","last_change":"2024-05-28T06:22:15.950581+0000","last_active":"2024-05-28T06:22:15.950581+0000","last_peered":"2024-05-28T06:22:15.950581+0000","last_clean":"2024-05-28T06:22:15.950581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:22:15.950581+0000","last_undegraded":"2024-05-28T06:22:15.950581+0000","last_fullsized":"2024-05-28T06:22:15.950581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":1,"reported_epoch":113,"state":"creating+peering","last_fresh":"2024-05-28T06:22:15.981127+0000","last_change":"2024-05-28T06:22:15.981127+0000","last_active":"2024-05-28T06:22:15.950581+0000","last_peered":"2024-05-28T06:22:15.950581+0000","last_clean":"2024-05-28T06:22:15.950581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:22:15.981127+0000","last_undegraded":"2024-05-28T06:22:15.981127+0000","last_fullsized":"2024-05-28T06:22:15.981127+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.242476+0000","last_change":"2024-05-28T06:22:17.003545+0000","last_active":"2024-05-28T06:22:18.242476+0000","last_peered":"2024-05-28T06:22:18.242476+0000","last_clean":"2024-05-28T06:22:18.242476+0000","last_became_active":"2024-05-28T06:22:17.001251+0000","last_became_peered":"2024-05-28T06:22:17.001251+0000","last_unstale":"2024-05-28T06:22:18.242476+0000","last_undegraded":"2024-05-28T06:22:18.242476+0000","last_fullsized":"2024-05-28T06:22:18.242476+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:38:50.761814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.243297+0000","last_change":"2024-05-28T06:22:17.001134+0000","last_active":"2024-05-28T06:22:18.243297+0000","last_peered":"2024-05-28T06:22:18.243297+0000","last_clean":"2024-05-28T06:22:18.243297+0000","last_became_active":"2024-05-28T06:22:17.000345+0000","last_became_peered":"2024-05-28T06:22:17.000345+0000","last_unstale":"2024-05-28T06:22:18.243297+0000","last_undegraded":"2024-05-28T06:22:18.243297+0000","last_fullsized":"2024-05-28T06:22:18.243297+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:22:46.981586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-28T06:22:15.950581+0000","last_change":"2024-05-28T06:22:15.950581+0000","last_active":"2024-05-28T06:22:15.950581+0000","last_peered":"2024-05-28T06:22:15.950581+0000","last_clean":"2024-05-28T06:22:15.950581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:22:15.950581+0000","last_undegraded":"2024-05-28T06:22:15.950581+0000","last_fullsized":"2024-05-28T06:22:15.950581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.243231+0000","last_change":"2024-05-28T06:22:16.994222+0000","last_active":"2024-05-28T06:22:18.243231+0000","last_peered":"2024-05-28T06:22:18.243231+0000","last_clean":"2024-05-28T06:22:18.243231+0000","last_became_active":"2024-05-28T06:22:16.993674+0000","last_became_peered":"2024-05-28T06:22:16.993674+0000","last_unstale":"2024-05-28T06:22:18.243231+0000","last_undegraded":"2024-05-28T06:22:18.243231+0000","last_fullsized":"2024-05-28T06:22:18.243231+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:40:55.388301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":1,"reported_epoch":113,"state":"creating+peering","last_fresh":"2024-05-28T06:22:15.979372+0000","last_change":"2024-05-28T06:22:15.979372+0000","last_active":"2024-05-28T06:22:15.950581+0000","last_peered":"2024-05-28T06:22:15.950581+0000","last_clean":"2024-05-28T06:22:15.950581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:22:15.979372+0000","last_undegraded":"2024-05-28T06:22:15.979372+0000","last_fullsized":"2024-05-28T06:22:15.979372+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":1,"reported_epoch":113,"state":"creating+peering","last_fresh":"2024-05-28T06:22:15.979017+0000","last_change":"2024-05-28T06:22:15.979017+0000","last_active":"2024-05-28T06:22:15.950581+0000","last_peered":"2024-05-28T06:22:15.950581+0000","last_clean":"2024-05-28T06:22:15.950581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:22:15.979017+0000","last_undegraded":"2024-05-28T06:22:15.979017+0000","last_fullsized":"2024-05-28T06:22:15.979017+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.527922+0000","last_change":"2024-05-28T06:22:16.997664+0000","last_active":"2024-05-28T06:22:18.527922+0000","last_peered":"2024-05-28T06:22:18.527922+0000","last_clean":"2024-05-28T06:22:18.527922+0000","last_became_active":"2024-05-28T06:22:16.997387+0000","last_became_peered":"2024-05-28T06:22:16.997387+0000","last_unstale":"2024-05-28T06:22:18.527922+0000","last_undegraded":"2024-05-28T06:22:18.527922+0000","last_fullsized":"2024-05-28T06:22:18.527922+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:37:05.453277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.242046+0000","last_change":"2024-05-28T06:22:17.005071+0000","last_active":"2024-05-28T06:22:18.242046+0000","last_peered":"2024-05-28T06:22:18.242046+0000","last_clean":"2024-05-28T06:22:18.242046+0000","last_became_active":"2024-05-28T06:22:17.004078+0000","last_became_peered":"2024-05-28T06:22:17.004078+0000","last_unstale":"2024-05-28T06:22:18.242046+0000","last_undegraded":"2024-05-28T06:22:18.242046+0000","last_fullsized":"2024-05-28T06:22:18.242046+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:26:19.869612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-28T06:22:15.950581+0000","last_change":"2024-05-28T06:22:15.950581+0000","last_active":"2024-05-28T06:22:15.950581+0000","last_peered":"2024-05-28T06:22:15.950581+0000","last_clean":"2024-05-28T06:22:15.950581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:22:15.950581+0000","last_undegraded":"2024-05-28T06:22:15.950581+0000","last_fullsized":"2024-05-28T06:22:15.950581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-28T06:22:15.950581+0000","last_change":"2024-05-28T06:22:15.950581+0000","last_active":"2024-05-28T06:22:15.950581+0000","last_peered":"2024-05-28T06:22:15.950581+0000","last_clean":"2024-05-28T06:22:15.950581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:22:15.950581+0000","last_undegraded":"2024-05-28T06:22:15.950581+0000","last_fullsized":"2024-05-28T06:22:15.950581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":1,"reported_epoch":113,"state":"creating+peering","last_fresh":"2024-05-28T06:22:15.983434+0000","last_change":"2024-05-28T06:22:15.983434+0000","last_active":"2024-05-28T06:22:15.950581+0000","last_peered":"2024-05-28T06:22:15.950581+0000","last_clean":"2024-05-28T06:22:15.950581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:22:15.983434+0000","last_undegraded":"2024-05-28T06:22:15.983434+0000","last_fullsized":"2024-05-28T06:22:15.983434+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-28T06:22:15.950581+0000","last_change":"2024-05-28T06:22:15.950581+0000","last_active":"2024-05-28T06:22:15.950581+0000","last_peered":"2024-05-28T06:22:15.950581+0000","last_clean":"2024-05-28T06:22:15.950581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:22:15.950581+0000","last_undegraded":"2024-05-28T06:22:15.950581+0000","last_fullsized":"2024-05-28T06:22:15.950581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-28T06:22:15.950581+0000","last_change":"2024-05-28T06:22:15.950581+0000","last_active":"2024-05-28T06:22:15.950581+0000","last_peered":"2024-05-28T06:22:15.950581+0000","last_clean":"2024-05-28T06:22:15.950581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:22:15.950581+0000","last_undegraded":"2024-05-28T06:22:15.950581+0000","last_fullsized":"2024-05-28T06:22:15.950581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.528790+0000","last_change":"2024-05-28T06:22:17.004708+0000","last_active":"2024-05-28T06:22:18.528790+0000","last_peered":"2024-05-28T06:22:18.528790+0000","last_clean":"2024-05-28T06:22:18.528790+0000","last_became_active":"2024-05-28T06:22:17.003676+0000","last_became_peered":"2024-05-28T06:22:17.003676+0000","last_unstale":"2024-05-28T06:22:18.528790+0000","last_undegraded":"2024-05-28T06:22:18.528790+0000","last_fullsized":"2024-05-28T06:22:18.528790+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:38:34.851302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-28T06:22:15.950581+0000","last_change":"2024-05-28T06:22:15.950581+0000","last_active":"2024-05-28T06:22:15.950581+0000","last_peered":"2024-05-28T06:22:15.950581+0000","last_clean":"2024-05-28T06:22:15.950581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:22:15.950581+0000","last_undegraded":"2024-05-28T06:22:15.950581+0000","last_fullsized":"2024-05-28T06:22:15.950581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.526910+0000","last_change":"2024-05-28T06:22:17.008315+0000","last_active":"2024-05-28T06:22:18.526910+0000","last_peered":"2024-05-28T06:22:18.526910+0000","last_clean":"2024-05-28T06:22:18.526910+0000","last_became_active":"2024-05-28T06:22:17.007995+0000","last_became_peered":"2024-05-28T06:22:17.007995+0000","last_unstale":"2024-05-28T06:22:18.526910+0000","last_undegraded":"2024-05-28T06:22:18.526910+0000","last_fullsized":"2024-05-28T06:22:18.526910+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T18:02:32.237669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-28T06:22:15.950581+0000","last_change":"2024-05-28T06:22:15.950581+0000","last_active":"2024-05-28T06:22:15.950581+0000","last_peered":"2024-05-28T06:22:15.950581+0000","last_clean":"2024-05-28T06:22:15.950581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:22:15.950581+0000","last_undegraded":"2024-05-28T06:22:15.950581+0000","last_fullsized":"2024-05-28T06:22:15.950581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.243265+0000","last_change":"2024-05-28T06:22:17.006855+0000","last_active":"2024-05-28T06:22:18.243265+0000","last_peered":"2024-05-28T06:22:18.243265+0000","last_clean":"2024-05-28T06:22:18.243265+0000","last_became_active":"2024-05-28T06:22:17.006649+0000","last_became_peered":"2024-05-28T06:22:17.006649+0000","last_unstale":"2024-05-28T06:22:18.243265+0000","last_undegraded":"2024-05-28T06:22:18.243265+0000","last_fullsized":"2024-05-28T06:22:18.243265+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:25:34.876194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.529987+0000","last_change":"2024-05-28T06:22:17.000165+0000","last_active":"2024-05-28T06:22:18.529987+0000","last_peered":"2024-05-28T06:22:18.529987+0000","last_clean":"2024-05-28T06:22:18.529987+0000","last_became_active":"2024-05-28T06:22:16.999904+0000","last_became_peered":"2024-05-28T06:22:16.999904+0000","last_unstale":"2024-05-28T06:22:18.529987+0000","last_undegraded":"2024-05-28T06:22:18.529987+0000","last_fullsized":"2024-05-28T06:22:18.529987+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:58:15.396668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.529632+0000","last_change":"2024-05-28T06:22:16.993958+0000","last_active":"2024-05-28T06:22:18.529632+0000","last_peered":"2024-05-28T06:22:18.529632+0000","last_clean":"2024-05-28T06:22:18.529632+0000","last_became_active":"2024-05-28T06:22:16.993327+0000","last_became_peered":"2024-05-28T06:22:16.993327+0000","last_unstale":"2024-05-28T06:22:18.529632+0000","last_undegraded":"2024-05-28T06:22:18.529632+0000","last_fullsized":"2024-05-28T06:22:18.529632+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:26:18.638055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-28T06:22:15.950581+0000","last_change":"2024-05-28T06:22:15.950581+0000","last_active":"2024-05-28T06:22:15.950581+0000","last_peered":"2024-05-28T06:22:15.950581+0000","last_clean":"2024-05-28T06:22:15.950581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:22:15.950581+0000","last_undegraded":"2024-05-28T06:22:15.950581+0000","last_fullsized":"2024-05-28T06:22:15.950581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-28T06:22:15.950581+0000","last_change":"2024-05-28T06:22:15.950581+0000","last_active":"2024-05-28T06:22:15.950581+0000","last_peered":"2024-05-28T06:22:15.950581+0000","last_clean":"2024-05-28T06:22:15.950581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:22:15.950581+0000","last_undegraded":"2024-05-28T06:22:15.950581+0000","last_fullsized":"2024-05-28T06:22:15.950581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.242168+0000","last_change":"2024-05-28T06:22:17.001908+0000","last_active":"2024-05-28T06:22:18.242168+0000","last_peered":"2024-05-28T06:22:18.242168+0000","last_clean":"2024-05-28T06:22:18.242168+0000","last_became_active":"2024-05-28T06:22:17.000543+0000","last_became_peered":"2024-05-28T06:22:17.000543+0000","last_unstale":"2024-05-28T06:22:18.242168+0000","last_undegraded":"2024-05-28T06:22:18.242168+0000","last_fullsized":"2024-05-28T06:22:18.242168+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:12:55.159821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.527587+0000","last_change":"2024-05-28T06:22:16.992715+0000","last_active":"2024-05-28T06:22:18.527587+0000","last_peered":"2024-05-28T06:22:18.527587+0000","last_clean":"2024-05-28T06:22:18.527587+0000","last_became_active":"2024-05-28T06:22:16.992094+0000","last_became_peered":"2024-05-28T06:22:16.992094+0000","last_unstale":"2024-05-28T06:22:18.527587+0000","last_undegraded":"2024-05-28T06:22:18.527587+0000","last_fullsized":"2024-05-28T06:22:18.527587+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:05:25.661249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.529840+0000","last_change":"2024-05-28T06:22:17.004255+0000","last_active":"2024-05-28T06:22:18.529840+0000","last_peered":"2024-05-28T06:22:18.529840+0000","last_clean":"2024-05-28T06:22:18.529840+0000","last_became_active":"2024-05-28T06:22:17.001691+0000","last_became_peered":"2024-05-28T06:22:17.001691+0000","last_unstale":"2024-05-28T06:22:18.529840+0000","last_undegraded":"2024-05-28T06:22:18.529840+0000","last_fullsized":"2024-05-28T06:22:18.529840+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:40:20.104894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":1,"reported_epoch":112,"state":"creating+peering","last_fresh":"2024-05-28T06:22:14.968914+0000","last_change":"2024-05-28T06:22:14.968914+0000","last_active":"2024-05-28T06:22:14.938772+0000","last_peered":"2024-05-28T06:22:14.938772+0000","last_clean":"2024-05-28T06:22:14.938772+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:22:14.968914+0000","last_undegraded":"2024-05-28T06:22:14.968914+0000","last_fullsized":"2024-05-28T06:22:14.968914+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-28T06:22:15.950581+0000","last_change":"2024-05-28T06:22:15.950581+0000","last_active":"2024-05-28T06:22:15.950581+0000","last_peered":"2024-05-28T06:22:15.950581+0000","last_clean":"2024-05-28T06:22:15.950581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:22:15.950581+0000","last_undegraded":"2024-05-28T06:22:15.950581+0000","last_fullsized":"2024-05-28T06:22:15.950581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":9,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-05-28T06:22:15.998589+0000","last_change":"2024-05-28T06:22:15.996401+0000","last_active":"2024-05-28T06:22:15.998589+0000","last_peered":"2024-05-28T06:22:15.998589+0000","last_clean":"2024-05-28T06:22:15.998589+0000","last_became_active":"2024-05-28T06:22:15.996039+0000","last_became_peered":"2024-05-28T06:22:15.996039+0000","last_unstale":"2024-05-28T06:22:15.998589+0000","last_undegraded":"2024-05-28T06:22:15.998589+0000","last_fullsized":"2024-05-28T06:22:15.998589+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:41:39.601468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.528998+0000","last_change":"2024-05-28T06:22:16.999053+0000","last_active":"2024-05-28T06:22:18.528998+0000","last_peered":"2024-05-28T06:22:18.528998+0000","last_clean":"2024-05-28T06:22:18.528998+0000","last_became_active":"2024-05-28T06:22:16.995589+0000","last_became_peered":"2024-05-28T06:22:16.995589+0000","last_unstale":"2024-05-28T06:22:18.528998+0000","last_undegraded":"2024-05-28T06:22:18.528998+0000","last_fullsized":"2024-05-28T06:22:18.528998+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:58:42.572818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"114'2","reported_seq":16,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:17.999806+0000","last_change":"2024-05-28T06:22:15.989244+0000","last_active":"2024-05-28T06:22:17.999806+0000","last_peered":"2024-05-28T06:22:17.999806+0000","last_clean":"2024-05-28T06:22:17.999806+0000","last_became_active":"2024-05-28T06:22:15.988137+0000","last_became_peered":"2024-05-28T06:22:15.988137+0000","last_unstale":"2024-05-28T06:22:17.999806+0000","last_undegraded":"2024-05-28T06:22:17.999806+0000","last_fullsized":"2024-05-28T06:22:17.999806+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T18:10:18.669151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":1,"reported_epoch":113,"state":"creating+peering","last_fresh":"2024-05-28T06:22:15.979253+0000","last_change":"2024-05-28T06:22:15.979253+0000","last_active":"2024-05-28T06:22:15.950581+0000","last_peered":"2024-05-28T06:22:15.950581+0000","last_clean":"2024-05-28T06:22:15.950581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:22:15.979253+0000","last_undegraded":"2024-05-28T06:22:15.979253+0000","last_fullsized":"2024-05-28T06:22:15.979253+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"114'2","reported_seq":16,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:17.999369+0000","last_change":"2024-05-28T06:22:15.992414+0000","last_active":"2024-05-28T06:22:17.999369+0000","last_peered":"2024-05-28T06:22:17.999369+0000","last_clean":"2024-05-28T06:22:17.999369+0000","last_became_active":"2024-05-28T06:22:15.991608+0000","last_became_peered":"2024-05-28T06:22:15.991608+0000","last_unstale":"2024-05-28T06:22:17.999369+0000","last_undegraded":"2024-05-28T06:22:17.999369+0000","last_fullsized":"2024-05-28T06:22:17.999369+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:42:21.601868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.241943+0000","last_change":"2024-05-28T06:22:16.995582+0000","last_active":"2024-05-28T06:22:18.241943+0000","last_peered":"2024-05-28T06:22:18.241943+0000","last_clean":"2024-05-28T06:22:18.241943+0000","last_became_active":"2024-05-28T06:22:16.995242+0000","last_became_peered":"2024-05-28T06:22:16.995242+0000","last_unstale":"2024-05-28T06:22:18.241943+0000","last_undegraded":"2024-05-28T06:22:18.241943+0000","last_fullsized":"2024-05-28T06:22:18.241943+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:46:21.232752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":1,"reported_epoch":112,"state":"creating+peering","last_fresh":"2024-05-28T06:22:14.970332+0000","last_change":"2024-05-28T06:22:14.970332+0000","last_active":"2024-05-28T06:22:14.938772+0000","last_peered":"2024-05-28T06:22:14.938772+0000","last_clean":"2024-05-28T06:22:14.938772+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:22:14.970332+0000","last_undegraded":"2024-05-28T06:22:14.970332+0000","last_fullsized":"2024-05-28T06:22:14.970332+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-28T06:22:15.950581+0000","last_change":"2024-05-28T06:22:15.950581+0000","last_active":"2024-05-28T06:22:15.950581+0000","last_peered":"2024-05-28T06:22:15.950581+0000","last_clean":"2024-05-28T06:22:15.950581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:22:15.950581+0000","last_undegraded":"2024-05-28T06:22:15.950581+0000","last_fullsized":"2024-05-28T06:22:15.950581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1a","version":"114'1","reported_seq":15,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.529094+0000","last_change":"2024-05-28T06:22:15.990828+0000","last_active":"2024-05-28T06:22:18.529094+0000","last_peered":"2024-05-28T06:22:18.529094+0000","last_clean":"2024-05-28T06:22:18.529094+0000","last_became_active":"2024-05-28T06:22:15.990537+0000","last_became_peered":"2024-05-28T06:22:15.990537+0000","last_unstale":"2024-05-28T06:22:18.529094+0000","last_undegraded":"2024-05-28T06:22:18.529094+0000","last_fullsized":"2024-05-28T06:22:18.529094+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:51:43.927844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.241796+0000","last_change":"2024-05-28T06:22:17.000804+0000","last_active":"2024-05-28T06:22:18.241796+0000","last_peered":"2024-05-28T06:22:18.241796+0000","last_clean":"2024-05-28T06:22:18.241796+0000","last_became_active":"2024-05-28T06:22:16.998428+0000","last_became_peered":"2024-05-28T06:22:16.998428+0000","last_unstale":"2024-05-28T06:22:18.241796+0000","last_undegraded":"2024-05-28T06:22:18.241796+0000","last_fullsized":"2024-05-28T06:22:18.241796+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:43:26.170749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":1,"reported_epoch":112,"state":"creating+peering","last_fresh":"2024-05-28T06:22:14.958623+0000","last_change":"2024-05-28T06:22:14.958623+0000","last_active":"2024-05-28T06:22:14.938772+0000","last_peered":"2024-05-28T06:22:14.938772+0000","last_clean":"2024-05-28T06:22:14.938772+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:22:14.958623+0000","last_undegraded":"2024-05-28T06:22:14.958623+0000","last_fullsized":"2024-05-28T06:22:14.958623+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-28T06:22:15.950581+0000","last_change":"2024-05-28T06:22:15.950581+0000","last_active":"2024-05-28T06:22:15.950581+0000","last_peered":"2024-05-28T06:22:15.950581+0000","last_clean":"2024-05-28T06:22:15.950581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:22:15.950581+0000","last_undegraded":"2024-05-28T06:22:15.950581+0000","last_fullsized":"2024-05-28T06:22:15.950581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.242254+0000","last_change":"2024-05-28T06:22:15.988357+0000","last_active":"2024-05-28T06:22:18.242254+0000","last_peered":"2024-05-28T06:22:18.242254+0000","last_clean":"2024-05-28T06:22:18.242254+0000","last_became_active":"2024-05-28T06:22:15.988053+0000","last_became_peered":"2024-05-28T06:22:15.988053+0000","last_unstale":"2024-05-28T06:22:18.242254+0000","last_undegraded":"2024-05-28T06:22:18.242254+0000","last_fullsized":"2024-05-28T06:22:18.242254+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:24:37.584832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-28T06:22:15.950581+0000","last_change":"2024-05-28T06:22:15.950581+0000","last_active":"2024-05-28T06:22:15.950581+0000","last_peered":"2024-05-28T06:22:15.950581+0000","last_clean":"2024-05-28T06:22:15.950581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:22:15.950581+0000","last_undegraded":"2024-05-28T06:22:15.950581+0000","last_fullsized":"2024-05-28T06:22:15.950581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:17.999693+0000","last_change":"2024-05-28T06:22:15.984544+0000","last_active":"2024-05-28T06:22:17.999693+0000","last_peered":"2024-05-28T06:22:17.999693+0000","last_clean":"2024-05-28T06:22:17.999693+0000","last_became_active":"2024-05-28T06:22:15.984287+0000","last_became_peered":"2024-05-28T06:22:15.984287+0000","last_unstale":"2024-05-28T06:22:17.999693+0000","last_undegraded":"2024-05-28T06:22:17.999693+0000","last_fullsized":"2024-05-28T06:22:17.999693+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T18:04:33.437207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-28T06:22:15.950581+0000","last_change":"2024-05-28T06:22:15.950581+0000","last_active":"2024-05-28T06:22:15.950581+0000","last_peered":"2024-05-28T06:22:15.950581+0000","last_clean":"2024-05-28T06:22:15.950581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:22:15.950581+0000","last_undegraded":"2024-05-28T06:22:15.950581+0000","last_fullsized":"2024-05-28T06:22:15.950581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.529413+0000","last_change":"2024-05-28T06:22:15.989398+0000","last_active":"2024-05-28T06:22:18.529413+0000","last_peered":"2024-05-28T06:22:18.529413+0000","last_clean":"2024-05-28T06:22:18.529413+0000","last_became_active":"2024-05-28T06:22:15.989101+0000","last_became_peered":"2024-05-28T06:22:15.989101+0000","last_unstale":"2024-05-28T06:22:18.529413+0000","last_undegraded":"2024-05-28T06:22:18.529413+0000","last_fullsized":"2024-05-28T06:22:18.529413+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:43:46.372846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.529711+0000","last_change":"2024-05-28T06:22:16.993645+0000","last_active":"2024-05-28T06:22:18.529711+0000","last_peered":"2024-05-28T06:22:18.529711+0000","last_clean":"2024-05-28T06:22:18.529711+0000","last_became_active":"2024-05-28T06:22:16.992912+0000","last_became_peered":"2024-05-28T06:22:16.992912+0000","last_unstale":"2024-05-28T06:22:18.529711+0000","last_undegraded":"2024-05-28T06:22:18.529711+0000","last_fullsized":"2024-05-28T06:22:18.529711+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:16:11.794755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"114'3","reported_seq":17,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.242908+0000","last_change":"2024-05-28T06:22:15.998469+0000","last_active":"2024-05-28T06:22:18.242908+0000","last_peered":"2024-05-28T06:22:18.242908+0000","last_clean":"2024-05-28T06:22:18.242908+0000","last_became_active":"2024-05-28T06:22:15.998186+0000","last_became_peered":"2024-05-28T06:22:15.998186+0000","last_unstale":"2024-05-28T06:22:18.242908+0000","last_undegraded":"2024-05-28T06:22:18.242908+0000","last_fullsized":"2024-05-28T06:22:18.242908+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:23:41.228363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-28T06:22:15.950581+0000","last_change":"2024-05-28T06:22:15.950581+0000","last_active":"2024-05-28T06:22:15.950581+0000","last_peered":"2024-05-28T06:22:15.950581+0000","last_clean":"2024-05-28T06:22:15.950581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:22:15.950581+0000","last_undegraded":"2024-05-28T06:22:15.950581+0000","last_fullsized":"2024-05-28T06:22:15.950581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.14","version":"114'6","reported_seq":20,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.243514+0000","last_change":"2024-05-28T06:22:15.997183+0000","last_active":"2024-05-28T06:22:18.243514+0000","last_peered":"2024-05-28T06:22:18.243514+0000","last_clean":"2024-05-28T06:22:18.243514+0000","last_became_active":"2024-05-28T06:22:15.996809+0000","last_became_peered":"2024-05-28T06:22:15.996809+0000","last_unstale":"2024-05-28T06:22:18.243514+0000","last_undegraded":"2024-05-28T06:22:18.243514+0000","last_fullsized":"2024-05-28T06:22:18.243514+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:38:33.839493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.528917+0000","last_change":"2024-05-28T06:22:17.000291+0000","last_active":"2024-05-28T06:22:18.528917+0000","last_peered":"2024-05-28T06:22:18.528917+0000","last_clean":"2024-05-28T06:22:18.528917+0000","last_became_active":"2024-05-28T06:22:16.999910+0000","last_became_peered":"2024-05-28T06:22:16.999910+0000","last_unstale":"2024-05-28T06:22:18.528917+0000","last_undegraded":"2024-05-28T06:22:18.528917+0000","last_fullsized":"2024-05-28T06:22:18.528917+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:23:42.240172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"114'2","reported_seq":16,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.528553+0000","last_change":"2024-05-28T06:22:15.996773+0000","last_active":"2024-05-28T06:22:18.528553+0000","last_peered":"2024-05-28T06:22:18.528553+0000","last_clean":"2024-05-28T06:22:18.528553+0000","last_became_active":"2024-05-28T06:22:15.995681+0000","last_became_peered":"2024-05-28T06:22:15.995681+0000","last_unstale":"2024-05-28T06:22:18.528553+0000","last_undegraded":"2024-05-28T06:22:18.528553+0000","last_fullsized":"2024-05-28T06:22:18.528553+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:48:09.956343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.528770+0000","last_change":"2024-05-28T06:22:17.003164+0000","last_active":"2024-05-28T06:22:18.528770+0000","last_peered":"2024-05-28T06:22:18.528770+0000","last_clean":"2024-05-28T06:22:18.528770+0000","last_became_active":"2024-05-28T06:22:17.002906+0000","last_became_peered":"2024-05-28T06:22:17.002906+0000","last_unstale":"2024-05-28T06:22:18.528770+0000","last_undegraded":"2024-05-28T06:22:18.528770+0000","last_fullsized":"2024-05-28T06:22:18.528770+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:13:14.889721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":1,"reported_epoch":112,"state":"creating+peering","last_fresh":"2024-05-28T06:22:14.963301+0000","last_change":"2024-05-28T06:22:14.963301+0000","last_active":"2024-05-28T06:22:14.938772+0000","last_peered":"2024-05-28T06:22:14.938772+0000","last_clean":"2024-05-28T06:22:14.938772+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:22:14.963301+0000","last_undegraded":"2024-05-28T06:22:14.963301+0000","last_fullsized":"2024-05-28T06:22:14.963301+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.527440+0000","last_change":"2024-05-28T06:22:16.991808+0000","last_active":"2024-05-28T06:22:18.527440+0000","last_peered":"2024-05-28T06:22:18.527440+0000","last_clean":"2024-05-28T06:22:18.527440+0000","last_became_active":"2024-05-28T06:22:16.991494+0000","last_became_peered":"2024-05-28T06:22:16.991494+0000","last_unstale":"2024-05-28T06:22:18.527440+0000","last_undegraded":"2024-05-28T06:22:18.527440+0000","last_fullsized":"2024-05-28T06:22:18.527440+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:47:18.486697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"114'8","reported_seq":22,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:17.999084+0000","last_change":"2024-05-28T06:22:15.991378+0000","last_active":"2024-05-28T06:22:17.999084+0000","last_peered":"2024-05-28T06:22:17.999084+0000","last_clean":"2024-05-28T06:22:17.999084+0000","last_became_active":"2024-05-28T06:22:15.990388+0000","last_became_peered":"2024-05-28T06:22:15.990388+0000","last_unstale":"2024-05-28T06:22:17.999084+0000","last_undegraded":"2024-05-28T06:22:17.999084+0000","last_fullsized":"2024-05-28T06:22:17.999084+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:49:10.819709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:17.999777+0000","last_change":"2024-05-28T06:22:16.991114+0000","last_active":"2024-05-28T06:22:17.999777+0000","last_peered":"2024-05-28T06:22:17.999777+0000","last_clean":"2024-05-28T06:22:17.999777+0000","last_became_active":"2024-05-28T06:22:16.990159+0000","last_became_peered":"2024-05-28T06:22:16.990159+0000","last_unstale":"2024-05-28T06:22:17.999777+0000","last_undegraded":"2024-05-28T06:22:17.999777+0000","last_fullsized":"2024-05-28T06:22:17.999777+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:18:01.989158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"114'1","reported_seq":15,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.242742+0000","last_change":"2024-05-28T06:22:15.997418+0000","last_active":"2024-05-28T06:22:18.242742+0000","last_peered":"2024-05-28T06:22:18.242742+0000","last_clean":"2024-05-28T06:22:18.242742+0000","last_became_active":"2024-05-28T06:22:15.995786+0000","last_became_peered":"2024-05-28T06:22:15.995786+0000","last_unstale":"2024-05-28T06:22:18.242742+0000","last_undegraded":"2024-05-28T06:22:18.242742+0000","last_fullsized":"2024-05-28T06:22:18.242742+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:50:11.148336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.530295+0000","last_change":"2024-05-28T06:22:16.993861+0000","last_active":"2024-05-28T06:22:18.530295+0000","last_peered":"2024-05-28T06:22:18.530295+0000","last_clean":"2024-05-28T06:22:18.530295+0000","last_became_active":"2024-05-28T06:22:16.993602+0000","last_became_peered":"2024-05-28T06:22:16.993602+0000","last_unstale":"2024-05-28T06:22:18.530295+0000","last_undegraded":"2024-05-28T06:22:18.530295+0000","last_fullsized":"2024-05-28T06:22:18.530295+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:28:17.141535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"114'2","reported_seq":16,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:17.999975+0000","last_change":"2024-05-28T06:22:15.980689+0000","last_active":"2024-05-28T06:22:17.999975+0000","last_peered":"2024-05-28T06:22:17.999975+0000","last_clean":"2024-05-28T06:22:17.999975+0000","last_became_active":"2024-05-28T06:22:15.980139+0000","last_became_peered":"2024-05-28T06:22:15.980139+0000","last_unstale":"2024-05-28T06:22:17.999975+0000","last_undegraded":"2024-05-28T06:22:17.999975+0000","last_fullsized":"2024-05-28T06:22:17.999975+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:19:21.410923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":1,"reported_epoch":113,"state":"creating+peering","last_fresh":"2024-05-28T06:22:15.977950+0000","last_change":"2024-05-28T06:22:15.977950+0000","last_active":"2024-05-28T06:22:15.950581+0000","last_peered":"2024-05-28T06:22:15.950581+0000","last_clean":"2024-05-28T06:22:15.950581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:22:15.977950+0000","last_undegraded":"2024-05-28T06:22:15.977950+0000","last_fullsized":"2024-05-28T06:22:15.977950+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":1,"reported_epoch":112,"state":"creating+peering","last_fresh":"2024-05-28T06:22:14.964473+0000","last_change":"2024-05-28T06:22:14.964473+0000","last_active":"2024-05-28T06:22:14.938772+0000","last_peered":"2024-05-28T06:22:14.938772+0000","last_clean":"2024-05-28T06:22:14.938772+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:22:14.964473+0000","last_undegraded":"2024-05-28T06:22:14.964473+0000","last_fullsized":"2024-05-28T06:22:14.964473+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":1,"reported_epoch":113,"state":"creating+peering","last_fresh":"2024-05-28T06:22:15.978074+0000","last_change":"2024-05-28T06:22:15.978074+0000","last_active":"2024-05-28T06:22:15.950581+0000","last_peered":"2024-05-28T06:22:15.950581+0000","last_clean":"2024-05-28T06:22:15.950581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:22:15.978074+0000","last_undegraded":"2024-05-28T06:22:15.978074+0000","last_fullsized":"2024-05-28T06:22:15.978074+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":1,"reported_epoch":112,"state":"creating+peering","last_fresh":"2024-05-28T06:22:14.968146+0000","last_change":"2024-05-28T06:22:14.968146+0000","last_active":"2024-05-28T06:22:14.938772+0000","last_peered":"2024-05-28T06:22:14.938772+0000","last_clean":"2024-05-28T06:22:14.938772+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:22:14.968146+0000","last_undegraded":"2024-05-28T06:22:14.968146+0000","last_fullsized":"2024-05-28T06:22:14.968146+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":1,"reported_epoch":113,"state":"creating+peering","last_fresh":"2024-05-28T06:22:15.977889+0000","last_change":"2024-05-28T06:22:15.977889+0000","last_active":"2024-05-28T06:22:15.950581+0000","last_peered":"2024-05-28T06:22:15.950581+0000","last_clean":"2024-05-28T06:22:15.950581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:22:15.977889+0000","last_undegraded":"2024-05-28T06:22:15.977889+0000","last_fullsized":"2024-05-28T06:22:15.977889+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"114'2","reported_seq":16,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.528872+0000","last_change":"2024-05-28T06:22:15.991711+0000","last_active":"2024-05-28T06:22:18.528872+0000","last_peered":"2024-05-28T06:22:18.528872+0000","last_clean":"2024-05-28T06:22:18.528872+0000","last_became_active":"2024-05-28T06:22:15.991323+0000","last_became_peered":"2024-05-28T06:22:15.991323+0000","last_unstale":"2024-05-28T06:22:18.528872+0000","last_undegraded":"2024-05-28T06:22:18.528872+0000","last_fullsized":"2024-05-28T06:22:18.528872+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:02:22.717474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.242729+0000","last_change":"2024-05-28T06:22:17.003257+0000","last_active":"2024-05-28T06:22:18.242729+0000","last_peered":"2024-05-28T06:22:18.242729+0000","last_clean":"2024-05-28T06:22:18.242729+0000","last_became_active":"2024-05-28T06:22:17.000808+0000","last_became_peered":"2024-05-28T06:22:17.000808+0000","last_unstale":"2024-05-28T06:22:18.242729+0000","last_undegraded":"2024-05-28T06:22:18.242729+0000","last_fullsized":"2024-05-28T06:22:18.242729+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:27:48.276213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:17.999010+0000","last_change":"2024-05-28T06:22:15.994076+0000","last_active":"2024-05-28T06:22:17.999010+0000","last_peered":"2024-05-28T06:22:17.999010+0000","last_clean":"2024-05-28T06:22:17.999010+0000","last_became_active":"2024-05-28T06:22:15.993488+0000","last_became_peered":"2024-05-28T06:22:15.993488+0000","last_unstale":"2024-05-28T06:22:17.999010+0000","last_undegraded":"2024-05-28T06:22:17.999010+0000","last_fullsized":"2024-05-28T06:22:17.999010+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:08:35.517640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":1,"reported_epoch":113,"state":"creating+peering","last_fresh":"2024-05-28T06:22:15.977812+0000","last_change":"2024-05-28T06:22:15.977812+0000","last_active":"2024-05-28T06:22:15.950581+0000","last_peered":"2024-05-28T06:22:15.950581+0000","last_clean":"2024-05-28T06:22:15.950581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:22:15.977812+0000","last_undegraded":"2024-05-28T06:22:15.977812+0000","last_fullsized":"2024-05-28T06:22:15.977812+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.527971+0000","last_change":"2024-05-28T06:22:15.989588+0000","last_active":"2024-05-28T06:22:18.527971+0000","last_peered":"2024-05-28T06:22:18.527971+0000","last_clean":"2024-05-28T06:22:18.527971+0000","last_became_active":"2024-05-28T06:22:15.989331+0000","last_became_peered":"2024-05-28T06:22:15.989331+0000","last_unstale":"2024-05-28T06:22:18.527971+0000","last_undegraded":"2024-05-28T06:22:18.527971+0000","last_fullsized":"2024-05-28T06:22:18.527971+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:19:21.410923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.527956+0000","last_change":"2024-05-28T06:22:16.993785+0000","last_active":"2024-05-28T06:22:18.527956+0000","last_peered":"2024-05-28T06:22:18.527956+0000","last_clean":"2024-05-28T06:22:18.527956+0000","last_became_active":"2024-05-28T06:22:16.993402+0000","last_became_peered":"2024-05-28T06:22:16.993402+0000","last_unstale":"2024-05-28T06:22:18.527956+0000","last_undegraded":"2024-05-28T06:22:18.527956+0000","last_fullsized":"2024-05-28T06:22:18.527956+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:06:13.719079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.242342+0000","last_change":"2024-05-28T06:22:15.998948+0000","last_active":"2024-05-28T06:22:18.242342+0000","last_peered":"2024-05-28T06:22:18.242342+0000","last_clean":"2024-05-28T06:22:18.242342+0000","last_became_active":"2024-05-28T06:22:15.998205+0000","last_became_peered":"2024-05-28T06:22:15.998205+0000","last_unstale":"2024-05-28T06:22:18.242342+0000","last_undegraded":"2024-05-28T06:22:18.242342+0000","last_fullsized":"2024-05-28T06:22:18.242342+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:58:34.652147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-28T06:22:15.950581+0000","last_change":"2024-05-28T06:22:15.950581+0000","last_active":"2024-05-28T06:22:15.950581+0000","last_peered":"2024-05-28T06:22:15.950581+0000","last_clean":"2024-05-28T06:22:15.950581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:22:15.950581+0000","last_undegraded":"2024-05-28T06:22:15.950581+0000","last_fullsized":"2024-05-28T06:22:15.950581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.7","version":"114'2","reported_seq":16,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.528934+0000","last_change":"2024-05-28T06:22:15.990864+0000","last_active":"2024-05-28T06:22:18.528934+0000","last_peered":"2024-05-28T06:22:18.528934+0000","last_clean":"2024-05-28T06:22:18.528934+0000","last_became_active":"2024-05-28T06:22:15.990559+0000","last_became_peered":"2024-05-28T06:22:15.990559+0000","last_unstale":"2024-05-28T06:22:18.528934+0000","last_undegraded":"2024-05-28T06:22:18.528934+0000","last_fullsized":"2024-05-28T06:22:18.528934+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:51:36.120307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.242633+0000","last_change":"2024-05-28T06:22:16.992471+0000","last_active":"2024-05-28T06:22:18.242633+0000","last_peered":"2024-05-28T06:22:18.242633+0000","last_clean":"2024-05-28T06:22:18.242633+0000","last_became_active":"2024-05-28T06:22:16.992135+0000","last_became_peered":"2024-05-28T06:22:16.992135+0000","last_unstale":"2024-05-28T06:22:18.242633+0000","last_undegraded":"2024-05-28T06:22:18.242633+0000","last_fullsized":"2024-05-28T06:22:18.242633+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:57:13.651588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":1,"reported_epoch":112,"state":"creating+peering","last_fresh":"2024-05-28T06:22:14.963417+0000","last_change":"2024-05-28T06:22:14.963417+0000","last_active":"2024-05-28T06:22:14.938772+0000","last_peered":"2024-05-28T06:22:14.938772+0000","last_clean":"2024-05-28T06:22:14.938772+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:22:14.963417+0000","last_undegraded":"2024-05-28T06:22:14.963417+0000","last_fullsized":"2024-05-28T06:22:14.963417+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.243620+0000","last_change":"2024-05-28T06:22:16.993934+0000","last_active":"2024-05-28T06:22:18.243620+0000","last_peered":"2024-05-28T06:22:18.243620+0000","last_clean":"2024-05-28T06:22:18.243620+0000","last_became_active":"2024-05-28T06:22:16.993249+0000","last_became_peered":"2024-05-28T06:22:16.993249+0000","last_unstale":"2024-05-28T06:22:18.243620+0000","last_undegraded":"2024-05-28T06:22:18.243620+0000","last_fullsized":"2024-05-28T06:22:18.243620+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:21:11.675368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.242990+0000","last_change":"2024-05-28T06:22:15.998999+0000","last_active":"2024-05-28T06:22:18.242990+0000","last_peered":"2024-05-28T06:22:18.242990+0000","last_clean":"2024-05-28T06:22:18.242990+0000","last_became_active":"2024-05-28T06:22:15.998631+0000","last_became_peered":"2024-05-28T06:22:15.998631+0000","last_unstale":"2024-05-28T06:22:18.242990+0000","last_undegraded":"2024-05-28T06:22:18.242990+0000","last_fullsized":"2024-05-28T06:22:18.242990+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:26:17.626246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-28T06:22:15.950581+0000","last_change":"2024-05-28T06:22:15.950581+0000","last_active":"2024-05-28T06:22:15.950581+0000","last_peered":"2024-05-28T06:22:15.950581+0000","last_clean":"2024-05-28T06:22:15.950581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:22:15.950581+0000","last_undegraded":"2024-05-28T06:22:15.950581+0000","last_fullsized":"2024-05-28T06:22:15.950581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.529805+0000","last_change":"2024-05-28T06:22:15.995193+0000","last_active":"2024-05-28T06:22:18.529805+0000","last_peered":"2024-05-28T06:22:18.529805+0000","last_clean":"2024-05-28T06:22:18.529805+0000","last_became_active":"2024-05-28T06:22:15.994748+0000","last_became_peered":"2024-05-28T06:22:15.994748+0000","last_unstale":"2024-05-28T06:22:18.529805+0000","last_undegraded":"2024-05-28T06:22:18.529805+0000","last_fullsized":"2024-05-28T06:22:18.529805+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:08:38.270230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-28T06:22:15.950581+0000","last_change":"2024-05-28T06:22:15.950581+0000","last_active":"2024-05-28T06:22:15.950581+0000","last_peered":"2024-05-28T06:22:15.950581+0000","last_clean":"2024-05-28T06:22:15.950581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:22:15.950581+0000","last_undegraded":"2024-05-28T06:22:15.950581+0000","last_fullsized":"2024-05-28T06:22:15.950581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":1,"reported_epoch":112,"state":"creating+peering","last_fresh":"2024-05-28T06:22:14.968289+0000","last_change":"2024-05-28T06:22:14.968289+0000","last_active":"2024-05-28T06:22:14.938772+0000","last_peered":"2024-05-28T06:22:14.938772+0000","last_clean":"2024-05-28T06:22:14.938772+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:22:14.968289+0000","last_undegraded":"2024-05-28T06:22:14.968289+0000","last_fullsized":"2024-05-28T06:22:14.968289+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-28T06:22:15.950581+0000","last_change":"2024-05-28T06:22:15.950581+0000","last_active":"2024-05-28T06:22:15.950581+0000","last_peered":"2024-05-28T06:22:15.950581+0000","last_clean":"2024-05-28T06:22:15.950581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:22:15.950581+0000","last_undegraded":"2024-05-28T06:22:15.950581+0000","last_fullsized":"2024-05-28T06:22:15.950581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":1,"reported_epoch":112,"state":"creating+peering","last_fresh":"2024-05-28T06:22:14.958443+0000","last_change":"2024-05-28T06:22:14.958443+0000","last_active":"2024-05-28T06:22:14.938772+0000","last_peered":"2024-05-28T06:22:14.938772+0000","last_clean":"2024-05-28T06:22:14.938772+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:22:14.958443+0000","last_undegraded":"2024-05-28T06:22:14.958443+0000","last_fullsized":"2024-05-28T06:22:14.958443+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":1,"reported_epoch":113,"state":"creating+peering","last_fresh":"2024-05-28T06:22:15.976482+0000","last_change":"2024-05-28T06:22:15.976482+0000","last_active":"2024-05-28T06:22:15.950581+0000","last_peered":"2024-05-28T06:22:15.950581+0000","last_clean":"2024-05-28T06:22:15.950581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:22:15.976482+0000","last_undegraded":"2024-05-28T06:22:15.976482+0000","last_fullsized":"2024-05-28T06:22:15.976482+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:17.999121+0000","last_change":"2024-05-28T06:22:15.984208+0000","last_active":"2024-05-28T06:22:17.999121+0000","last_peered":"2024-05-28T06:22:17.999121+0000","last_clean":"2024-05-28T06:22:17.999121+0000","last_became_active":"2024-05-28T06:22:15.983816+0000","last_became_peered":"2024-05-28T06:22:15.983816+0000","last_unstale":"2024-05-28T06:22:17.999121+0000","last_undegraded":"2024-05-28T06:22:17.999121+0000","last_fullsized":"2024-05-28T06:22:17.999121+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:30:43.782132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.242161+0000","last_change":"2024-05-28T06:22:16.996830+0000","last_active":"2024-05-28T06:22:18.242161+0000","last_peered":"2024-05-28T06:22:18.242161+0000","last_clean":"2024-05-28T06:22:18.242161+0000","last_became_active":"2024-05-28T06:22:16.990102+0000","last_became_peered":"2024-05-28T06:22:16.990102+0000","last_unstale":"2024-05-28T06:22:18.242161+0000","last_undegraded":"2024-05-28T06:22:18.242161+0000","last_fullsized":"2024-05-28T06:22:18.242161+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T18:04:15.407639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.242532+0000","last_change":"2024-05-28T06:22:15.998500+0000","last_active":"2024-05-28T06:22:18.242532+0000","last_peered":"2024-05-28T06:22:18.242532+0000","last_clean":"2024-05-28T06:22:18.242532+0000","last_became_active":"2024-05-28T06:22:15.998176+0000","last_became_peered":"2024-05-28T06:22:15.998176+0000","last_unstale":"2024-05-28T06:22:18.242532+0000","last_undegraded":"2024-05-28T06:22:18.242532+0000","last_fullsized":"2024-05-28T06:22:18.242532+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:17:13.644125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.527150+0000","last_change":"2024-05-28T06:22:17.008104+0000","last_active":"2024-05-28T06:22:18.527150+0000","last_peered":"2024-05-28T06:22:18.527150+0000","last_clean":"2024-05-28T06:22:18.527150+0000","last_became_active":"2024-05-28T06:22:17.007752+0000","last_became_peered":"2024-05-28T06:22:17.007752+0000","last_unstale":"2024-05-28T06:22:18.527150+0000","last_undegraded":"2024-05-28T06:22:18.527150+0000","last_fullsized":"2024-05-28T06:22:18.527150+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:08:39.984746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":1,"reported_epoch":112,"state":"creating+peering","last_fresh":"2024-05-28T06:22:14.968070+0000","last_change":"2024-05-28T06:22:14.968070+0000","last_active":"2024-05-28T06:22:14.938772+0000","last_peered":"2024-05-28T06:22:14.938772+0000","last_clean":"2024-05-28T06:22:14.938772+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-28T06:22:14.968070+0000","last_undegraded":"2024-05-28T06:22:14.968070+0000","last_fullsized":"2024-05-28T06:22:14.968070+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":350,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-05-28T06:22:15.989329+0000","last_change":"2024-05-28T06:04:01.039129+0000","last_active":"2024-05-28T06:22:15.989329+0000","last_peered":"2024-05-28T06:22:15.989329+0000","last_clean":"2024-05-28T06:22:15.989329+0000","last_became_active":"2024-05-28T06:04:01.033906+0000","last_became_peered":"2024-05-28T06:04:01.033906+0000","last_unstale":"2024-05-28T06:22:15.989329+0000","last_undegraded":"2024-05-28T06:22:15.989329+0000","last_fullsized":"2024-05-28T06:22:15.989329+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:03:59.864237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:03:59.864237+0000","last_clean_scrub_stamp":"2024-05-28T06:03:59.864237+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:17:36.440245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":86,"acting":86,"num_store_stats":5},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":1758,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":39,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31,"ondisk_log_size":31,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575070,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":127460,"kb_used_data":1940,"kb_used_omap":1,"kb_used_meta":125502,"kb_avail":94244380,"statfs":{"total":96636764160,"available":96506245120,"internally_reserved":0,"allocated":1986560,"data_stored":1197121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":128514506},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575072,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":126868,"kb_used_data":1348,"kb_used_omap":1,"kb_used_meta":125502,"kb_avail":94244972,"statfs":{"total":96636764160,"available":96506851328,"internally_reserved":0,"allocated":1380352,"data_stored":600880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":128514502},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575070,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81040,"kb_used_data":1348,"kb_used_omap":1,"kb_used_meta":79678,"kb_avail":94290800,"statfs":{"total":96636764160,"available":96553779200,"internally_reserved":0,"allocated":1380352,"data_stored":600880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":81590725},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575070,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":86616,"kb_used_data":1348,"kb_used_omap":1,"kb_used_meta":85246,"kb_avail":94285224,"statfs":{"total":96636764160,"available":96548069376,"internally_reserved":0,"allocated":1380352,"data_stored":600880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87292357},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575073,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":131096,"kb_used_data":1412,"kb_used_omap":1,"kb_used_meta":129662,"kb_avail":94240744,"statfs":{"total":96636764160,"available":96502521856,"internally_reserved":0,"allocated":1445888,"data_stored":627953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":132774343},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575072,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":144960,"kb_used_data":1396,"kb_used_omap":1,"kb_used_meta":143550,"kb_avail":94226880,"statfs":{"total":96636764160,"available":96488325120,"internally_reserved":0,"allocated":1429504,"data_stored":626271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":146995653},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575072,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":126920,"kb_used_data":1404,"kb_used_omap":1,"kb_used_meta":125502,"kb_avail":94244920,"statfs":{"total":96636764160,"available":96506798080,"internally_reserved":0,"allocated":1437696,"data_stored":626395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":128514501},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575071,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":87260,"kb_used_data":1996,"kb_used_omap":1,"kb_used_meta":85246,"kb_avail":94284580,"statfs":{"total":96636764160,"available":96547409920,"internally_reserved":0,"allocated":2043904,"data_stored":1218363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87292357},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-28T06:22:21.931 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-28T06:22:22.174 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.179+0000 7fd256800640 1 Processor -- start 2024-05-28T06:22:22.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.179+0000 7fd256800640 1 -- start start 2024-05-28T06:22:22.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.181+0000 7fd256800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fd2500d04a0 0x7fd2500d0930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:22.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.181+0000 7fd256800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fd2500ad130 0x7fd2500ad540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:22.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.181+0000 7fd256800640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd25006ed10 con 0x7fd25006f860 2024-05-28T06:22:22.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.181+0000 7fd256800640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fd25015cb00 con 0x7fd2500d04a0 2024-05-28T06:22:22.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.181+0000 7fd256800640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fd2500b3cc0 con 0x7fd2500ad130 2024-05-28T06:22:22.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.181+0000 7fd24ea00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fd2500d04a0 0x7fd2500d0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:22.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.181+0000 7fd24ea00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fd2500d04a0 0x7fd2500d0930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:60218/0 (socket says 172.21.15.96:60218) 2024-05-28T06:22:22.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.181+0000 7fd24ea00640 1 -- 172.21.15.96:0/1335694322 learned_addr learned my addr 172.21.15.96:0/1335694322 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:22:22.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.182+0000 7fd24fe00640 1 --2- 172.21.15.96:0/1335694322 >> v2:172.21.15.123:3300/0 conn(0x7fd2500ad130 0x7fd2500ad540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:22.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.182+0000 7fd24ea00640 1 -- 172.21.15.96:0/1335694322 >> v2:172.21.15.123:3300/0 conn(0x7fd2500ad130 msgr2=0x7fd2500ad540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:22.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.182+0000 7fd24ea00640 1 --2- 172.21.15.96:0/1335694322 >> v2:172.21.15.123:3300/0 conn(0x7fd2500ad130 0x7fd2500ad540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:22.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.182+0000 7fd24ea00640 1 -- 172.21.15.96:0/1335694322 >> v1:172.21.15.123:6789/0 conn(0x7fd25006f860 legacy=0x7fd2500b3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:22.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.182+0000 7fd24ea00640 1 -- 172.21.15.96:0/1335694322 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2500b3fa0 con 0x7fd2500d04a0 2024-05-28T06:22:22.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.183+0000 7fd24ea00640 1 --2- 172.21.15.96:0/1335694322 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd2500d04a0 0x7fd2500d0930 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7fd2500af6e0 tx=0x7fd23802a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4aada08def5a663f server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:22.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.183+0000 7fd24e000640 1 -- 172.21.15.96:0/1335694322 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd238037070 con 0x7fd2500d04a0 2024-05-28T06:22:22.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.183+0000 7fd24e000640 1 -- 172.21.15.96:0/1335694322 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd238032440 con 0x7fd2500d04a0 2024-05-28T06:22:22.180 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.183+0000 7fd24e000640 1 -- 172.21.15.96:0/1335694322 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd238032810 con 0x7fd2500d04a0 2024-05-28T06:22:22.180 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.184+0000 7fd256800640 1 -- 172.21.15.96:0/1335694322 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd2500d04a0 msgr2=0x7fd2500d0930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:22.180 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.184+0000 7fd256800640 1 --2- 172.21.15.96:0/1335694322 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd2500d04a0 0x7fd2500d0930 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7fd2500af6e0 tx=0x7fd23802a230 comp rx=0 tx=0).stop 2024-05-28T06:22:22.180 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.184+0000 7fd256800640 1 -- 172.21.15.96:0/1335694322 shutdown_connections 2024-05-28T06:22:22.180 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.184+0000 7fd256800640 1 --2- 172.21.15.96:0/1335694322 >> v2:172.21.15.123:3300/0 conn(0x7fd2500ad130 0x7fd2500ad540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:22.180 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.184+0000 7fd256800640 1 --2- 172.21.15.96:0/1335694322 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd2500d04a0 0x7fd2500d0930 unknown :-1 s=CLOSED pgs=1816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:22.180 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.184+0000 7fd256800640 1 -- 172.21.15.96:0/1335694322 >> 172.21.15.96:0/1335694322 conn(0x7fd25006bae0 msgr2=0x7fd25006bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:22.180 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.184+0000 7fd256800640 1 -- 172.21.15.96:0/1335694322 shutdown_connections 2024-05-28T06:22:22.180 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.184+0000 7fd256800640 1 -- 172.21.15.96:0/1335694322 wait complete. 2024-05-28T06:22:22.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.186+0000 7fd256800640 1 Processor -- start 2024-05-28T06:22:22.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.186+0000 7fd256800640 1 -- start start 2024-05-28T06:22:22.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.187+0000 7fd256800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd25006f860 0x7fd2501f4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:22.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.187+0000 7fd256800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd2500ad130 0x7fd250157060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:22.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.187+0000 7fd256800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd2500d04a0 0x7fd2500c7330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:22.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.187+0000 7fd256800640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd2500b4840 con 0x7fd2500ad130 2024-05-28T06:22:22.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.187+0000 7fd256800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd2500b4560 con 0x7fd25006f860 2024-05-28T06:22:22.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.187+0000 7fd256800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fd2500b3860 con 0x7fd2500d04a0 2024-05-28T06:22:22.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.187+0000 7fd24ea00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd2500ad130 0x7fd250157060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:22.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.187+0000 7fd24ea00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd2500ad130 0x7fd250157060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:60232/0 (socket says 172.21.15.96:60232) 2024-05-28T06:22:22.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.187+0000 7fd24ea00640 1 -- 172.21.15.96:0/4236627129 learned_addr learned my addr 172.21.15.96:0/4236627129 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:22:22.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.187+0000 7fd24f400640 1 --2- 172.21.15.96:0/4236627129 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd25006f860 0x7fd2501f4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:22.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.188+0000 7fd24fe00640 1 --2- 172.21.15.96:0/4236627129 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd2500d04a0 0x7fd2500c7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:22.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.188+0000 7fd24ea00640 1 -- 172.21.15.96:0/4236627129 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd2500d04a0 msgr2=0x7fd2500c7330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:22.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.188+0000 7fd24ea00640 1 --2- 172.21.15.96:0/4236627129 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd2500d04a0 0x7fd2500c7330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:22.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.188+0000 7fd24ea00640 1 -- 172.21.15.96:0/4236627129 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd25006f860 msgr2=0x7fd2501f4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:22.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.188+0000 7fd24ea00640 1 --2- 172.21.15.96:0/4236627129 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd25006f860 0x7fd2501f4440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:22.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.188+0000 7fd24ea00640 1 -- 172.21.15.96:0/4236627129 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd25015efa0 con 0x7fd2500ad130 2024-05-28T06:22:22.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.188+0000 7fd24f400640 1 --2- 172.21.15.96:0/4236627129 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd25006f860 0x7fd2501f4440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:22:22.185 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.188+0000 7fd24ea00640 1 --2- 172.21.15.96:0/4236627129 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd2500ad130 0x7fd250157060 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7fd238000c00 tx=0x7fd2380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:22.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.188+0000 7fd247400640 1 -- 172.21.15.96:0/4236627129 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd238041020 con 0x7fd2500ad130 2024-05-28T06:22:22.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.188+0000 7fd247400640 1 -- 172.21.15.96:0/4236627129 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2380306c0 con 0x7fd2500ad130 2024-05-28T06:22:22.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.188+0000 7fd247400640 1 -- 172.21.15.96:0/4236627129 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd238037040 con 0x7fd2500ad130 2024-05-28T06:22:22.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.188+0000 7fd256800640 1 -- 172.21.15.96:0/4236627129 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd25015f280 con 0x7fd2500ad130 2024-05-28T06:22:22.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.189+0000 7fd256800640 1 -- 172.21.15.96:0/4236627129 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd25015f840 con 0x7fd2500ad130 2024-05-28T06:22:22.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.190+0000 7fd256800640 1 -- 172.21.15.96:0/4236627129 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2500689e0 con 0x7fd2500ad130 2024-05-28T06:22:22.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.191+0000 7fd247400640 1 -- 172.21.15.96:0/4236627129 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 37) ==== 93182+0+0 (secure 0 0 0) 0x7fd238044020 con 0x7fd2500ad130 2024-05-28T06:22:22.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.192+0000 7fd247400640 1 --2- 172.21.15.96:0/4236627129 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd22c06e090 0x7fd22c070550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:22.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.192+0000 7fd247400640 1 -- 172.21.15.96:0/4236627129 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 5765+0+0 (secure 0 0 0) 0x7fd2380ab130 con 0x7fd2500ad130 2024-05-28T06:22:22.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.192+0000 7fd24f400640 1 --2- 172.21.15.96:0/4236627129 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd22c06e090 0x7fd22c070550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:22.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.193+0000 7fd24f400640 1 --2- 172.21.15.96:0/4236627129 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd22c06e090 0x7fd22c070550 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fd240007fd0 tx=0x7fd240009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:22.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.196+0000 7fd247400640 1 -- 172.21.15.96:0/4236627129 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd23804b030 con 0x7fd2500ad130 2024-05-28T06:22:22.387 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.392+0000 7fd256800640 1 -- 172.21.15.96:0/4236627129 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd25006e2f0 con 0x7fd22c06e090 2024-05-28T06:22:22.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.412+0000 7fd247400640 1 -- 172.21.15.96:0/4236627129 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252497 (secure 0 0 0) 0x7fd25006e2f0 con 0x7fd22c06e090 2024-05-28T06:22:22.411 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:22:22.412 INFO:teuthology.orchestra.run.smithi096.stderr:dumped all 2024-05-28T06:22:22.414 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.417+0000 7fd256800640 1 -- 172.21.15.96:0/4236627129 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd22c06e090 msgr2=0x7fd22c070550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:22.414 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.417+0000 7fd256800640 1 --2- 172.21.15.96:0/4236627129 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd22c06e090 0x7fd22c070550 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fd240007fd0 tx=0x7fd240009040 comp rx=0 tx=0).stop 2024-05-28T06:22:22.414 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.417+0000 7fd256800640 1 -- 172.21.15.96:0/4236627129 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd2500ad130 msgr2=0x7fd250157060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:22.414 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.417+0000 7fd256800640 1 --2- 172.21.15.96:0/4236627129 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd2500ad130 0x7fd250157060 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7fd238000c00 tx=0x7fd2380046c0 comp rx=0 tx=0).stop 2024-05-28T06:22:22.414 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.417+0000 7fd256800640 1 -- 172.21.15.96:0/4236627129 shutdown_connections 2024-05-28T06:22:22.414 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.417+0000 7fd256800640 1 --2- 172.21.15.96:0/4236627129 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd22c06e090 0x7fd22c070550 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:22.414 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.417+0000 7fd256800640 1 --2- 172.21.15.96:0/4236627129 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd2500d04a0 0x7fd2500c7330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:22.414 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.417+0000 7fd256800640 1 --2- 172.21.15.96:0/4236627129 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd2500ad130 0x7fd250157060 unknown :-1 s=CLOSED pgs=1817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:22.415 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.417+0000 7fd256800640 1 --2- 172.21.15.96:0/4236627129 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd25006f860 0x7fd2501f4440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:22.415 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.417+0000 7fd256800640 1 -- 172.21.15.96:0/4236627129 >> 172.21.15.96:0/4236627129 conn(0x7fd25006bae0 msgr2=0x7fd250071f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:22.415 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.417+0000 7fd256800640 1 -- 172.21.15.96:0/4236627129 shutdown_connections 2024-05-28T06:22:22.415 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.417+0000 7fd256800640 1 -- 172.21.15.96:0/4236627129 wait complete. 2024-05-28T06:22:22.432 INFO:teuthology.orchestra.run.smithi096.stdout:{"pg_ready":true,"pg_map":{"version":659,"stamp":"2024-05-28T06:22:21.965007+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":165,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":895772,"kb_used_data":12392,"kb_used_omap":12,"kb_used_meta":883251,"kb_avail":754078948,"statfs":{"total":773094113280,"available":772176842752,"internally_reserved":0,"allocated":12689408,"data_stored":6195724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":904449584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002310"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.242686+0000","last_change":"2024-05-28T06:22:17.004588+0000","last_active":"2024-05-28T06:22:18.242686+0000","last_peered":"2024-05-28T06:22:18.242686+0000","last_clean":"2024-05-28T06:22:18.242686+0000","last_became_active":"2024-05-28T06:22:17.003523+0000","last_became_peered":"2024-05-28T06:22:17.003523+0000","last_unstale":"2024-05-28T06:22:18.242686+0000","last_undegraded":"2024-05-28T06:22:18.242686+0000","last_fullsized":"2024-05-28T06:22:18.242686+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:04:35.674926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.001575+0000","last_change":"2024-05-28T06:22:16.997146+0000","last_active":"2024-05-28T06:22:18.001575+0000","last_peered":"2024-05-28T06:22:18.001575+0000","last_clean":"2024-05-28T06:22:18.001575+0000","last_became_active":"2024-05-28T06:22:16.995737+0000","last_became_peered":"2024-05-28T06:22:16.995737+0000","last_unstale":"2024-05-28T06:22:18.001575+0000","last_undegraded":"2024-05-28T06:22:18.001575+0000","last_fullsized":"2024-05-28T06:22:18.001575+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:38:34.851302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.528142+0000","last_change":"2024-05-28T06:22:17.006833+0000","last_active":"2024-05-28T06:22:18.528142+0000","last_peered":"2024-05-28T06:22:18.528142+0000","last_clean":"2024-05-28T06:22:18.528142+0000","last_became_active":"2024-05-28T06:22:17.006501+0000","last_became_peered":"2024-05-28T06:22:17.006501+0000","last_unstale":"2024-05-28T06:22:18.528142+0000","last_undegraded":"2024-05-28T06:22:18.528142+0000","last_fullsized":"2024-05-28T06:22:18.528142+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:23:42.240172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.242843+0000","last_change":"2024-05-28T06:22:16.994809+0000","last_active":"2024-05-28T06:22:18.242843+0000","last_peered":"2024-05-28T06:22:18.242843+0000","last_clean":"2024-05-28T06:22:18.242843+0000","last_became_active":"2024-05-28T06:22:16.994548+0000","last_became_peered":"2024-05-28T06:22:16.994548+0000","last_unstale":"2024-05-28T06:22:18.242843+0000","last_undegraded":"2024-05-28T06:22:18.242843+0000","last_fullsized":"2024-05-28T06:22:18.242843+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:59:36.297627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:17.999386+0000","last_change":"2024-05-28T06:22:16.996834+0000","last_active":"2024-05-28T06:22:17.999386+0000","last_peered":"2024-05-28T06:22:17.999386+0000","last_clean":"2024-05-28T06:22:17.999386+0000","last_became_active":"2024-05-28T06:22:16.996543+0000","last_became_peered":"2024-05-28T06:22:16.996543+0000","last_unstale":"2024-05-28T06:22:17.999386+0000","last_undegraded":"2024-05-28T06:22:17.999386+0000","last_fullsized":"2024-05-28T06:22:17.999386+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:23:11.730784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.528701+0000","last_change":"2024-05-28T06:22:16.995996+0000","last_active":"2024-05-28T06:22:18.528701+0000","last_peered":"2024-05-28T06:22:18.528701+0000","last_clean":"2024-05-28T06:22:18.528701+0000","last_became_active":"2024-05-28T06:22:16.995714+0000","last_became_peered":"2024-05-28T06:22:16.995714+0000","last_unstale":"2024-05-28T06:22:18.528701+0000","last_undegraded":"2024-05-28T06:22:18.528701+0000","last_fullsized":"2024-05-28T06:22:18.528701+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:04:42.043002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.024311+0000","last_change":"2024-05-28T06:22:17.006351+0000","last_active":"2024-05-28T06:22:18.024311+0000","last_peered":"2024-05-28T06:22:18.024311+0000","last_clean":"2024-05-28T06:22:18.024311+0000","last_became_active":"2024-05-28T06:22:17.005644+0000","last_became_peered":"2024-05-28T06:22:17.005644+0000","last_unstale":"2024-05-28T06:22:18.024311+0000","last_undegraded":"2024-05-28T06:22:18.024311+0000","last_fullsized":"2024-05-28T06:22:18.024311+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:21:19.908131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.527537+0000","last_change":"2024-05-28T06:22:16.998140+0000","last_active":"2024-05-28T06:22:18.527537+0000","last_peered":"2024-05-28T06:22:18.527537+0000","last_clean":"2024-05-28T06:22:18.527537+0000","last_became_active":"2024-05-28T06:22:16.997796+0000","last_became_peered":"2024-05-28T06:22:16.997796+0000","last_unstale":"2024-05-28T06:22:18.527537+0000","last_undegraded":"2024-05-28T06:22:18.527537+0000","last_fullsized":"2024-05-28T06:22:18.527537+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:06:13.719079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.242476+0000","last_change":"2024-05-28T06:22:17.003545+0000","last_active":"2024-05-28T06:22:18.242476+0000","last_peered":"2024-05-28T06:22:18.242476+0000","last_clean":"2024-05-28T06:22:18.242476+0000","last_became_active":"2024-05-28T06:22:17.001251+0000","last_became_peered":"2024-05-28T06:22:17.001251+0000","last_unstale":"2024-05-28T06:22:18.242476+0000","last_undegraded":"2024-05-28T06:22:18.242476+0000","last_fullsized":"2024-05-28T06:22:18.242476+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:38:50.761814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.243297+0000","last_change":"2024-05-28T06:22:17.001134+0000","last_active":"2024-05-28T06:22:18.243297+0000","last_peered":"2024-05-28T06:22:18.243297+0000","last_clean":"2024-05-28T06:22:18.243297+0000","last_became_active":"2024-05-28T06:22:17.000345+0000","last_became_peered":"2024-05-28T06:22:17.000345+0000","last_unstale":"2024-05-28T06:22:18.243297+0000","last_undegraded":"2024-05-28T06:22:18.243297+0000","last_fullsized":"2024-05-28T06:22:18.243297+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:22:46.981586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.001735+0000","last_change":"2024-05-28T06:22:17.002601+0000","last_active":"2024-05-28T06:22:18.001735+0000","last_peered":"2024-05-28T06:22:18.001735+0000","last_clean":"2024-05-28T06:22:18.001735+0000","last_became_active":"2024-05-28T06:22:17.002340+0000","last_became_peered":"2024-05-28T06:22:17.002340+0000","last_unstale":"2024-05-28T06:22:18.001735+0000","last_undegraded":"2024-05-28T06:22:18.001735+0000","last_fullsized":"2024-05-28T06:22:18.001735+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:26:18.638055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.243231+0000","last_change":"2024-05-28T06:22:16.994222+0000","last_active":"2024-05-28T06:22:18.243231+0000","last_peered":"2024-05-28T06:22:18.243231+0000","last_clean":"2024-05-28T06:22:18.243231+0000","last_became_active":"2024-05-28T06:22:16.993674+0000","last_became_peered":"2024-05-28T06:22:16.993674+0000","last_unstale":"2024-05-28T06:22:18.243231+0000","last_undegraded":"2024-05-28T06:22:18.243231+0000","last_fullsized":"2024-05-28T06:22:18.243231+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:40:55.388301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.528122+0000","last_change":"2024-05-28T06:22:16.998228+0000","last_active":"2024-05-28T06:22:18.528122+0000","last_peered":"2024-05-28T06:22:18.528122+0000","last_clean":"2024-05-28T06:22:18.528122+0000","last_became_active":"2024-05-28T06:22:16.996238+0000","last_became_peered":"2024-05-28T06:22:16.996238+0000","last_unstale":"2024-05-28T06:22:18.528122+0000","last_undegraded":"2024-05-28T06:22:18.528122+0000","last_fullsized":"2024-05-28T06:22:18.528122+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:38:34.851302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.527501+0000","last_change":"2024-05-28T06:22:17.001665+0000","last_active":"2024-05-28T06:22:18.527501+0000","last_peered":"2024-05-28T06:22:18.527501+0000","last_clean":"2024-05-28T06:22:18.527501+0000","last_became_active":"2024-05-28T06:22:17.001304+0000","last_became_peered":"2024-05-28T06:22:17.001304+0000","last_unstale":"2024-05-28T06:22:18.527501+0000","last_undegraded":"2024-05-28T06:22:18.527501+0000","last_fullsized":"2024-05-28T06:22:18.527501+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:47:18.486697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.527922+0000","last_change":"2024-05-28T06:22:16.997664+0000","last_active":"2024-05-28T06:22:18.527922+0000","last_peered":"2024-05-28T06:22:18.527922+0000","last_clean":"2024-05-28T06:22:18.527922+0000","last_became_active":"2024-05-28T06:22:16.997387+0000","last_became_peered":"2024-05-28T06:22:16.997387+0000","last_unstale":"2024-05-28T06:22:18.527922+0000","last_undegraded":"2024-05-28T06:22:18.527922+0000","last_fullsized":"2024-05-28T06:22:18.527922+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:37:05.453277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.242046+0000","last_change":"2024-05-28T06:22:17.005071+0000","last_active":"2024-05-28T06:22:18.242046+0000","last_peered":"2024-05-28T06:22:18.242046+0000","last_clean":"2024-05-28T06:22:18.242046+0000","last_became_active":"2024-05-28T06:22:17.004078+0000","last_became_peered":"2024-05-28T06:22:17.004078+0000","last_unstale":"2024-05-28T06:22:18.242046+0000","last_undegraded":"2024-05-28T06:22:18.242046+0000","last_fullsized":"2024-05-28T06:22:18.242046+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:26:19.869612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.001928+0000","last_change":"2024-05-28T06:22:17.010577+0000","last_active":"2024-05-28T06:22:18.001928+0000","last_peered":"2024-05-28T06:22:18.001928+0000","last_clean":"2024-05-28T06:22:18.001928+0000","last_became_active":"2024-05-28T06:22:17.010281+0000","last_became_peered":"2024-05-28T06:22:17.010281+0000","last_unstale":"2024-05-28T06:22:18.001928+0000","last_undegraded":"2024-05-28T06:22:18.001928+0000","last_fullsized":"2024-05-28T06:22:18.001928+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:23:42.240172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.020188+0000","last_change":"2024-05-28T06:22:17.011422+0000","last_active":"2024-05-28T06:22:18.020188+0000","last_peered":"2024-05-28T06:22:18.020188+0000","last_clean":"2024-05-28T06:22:18.020188+0000","last_became_active":"2024-05-28T06:22:17.010224+0000","last_became_peered":"2024-05-28T06:22:17.010224+0000","last_unstale":"2024-05-28T06:22:18.020188+0000","last_undegraded":"2024-05-28T06:22:18.020188+0000","last_fullsized":"2024-05-28T06:22:18.020188+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:11:13.465001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.528617+0000","last_change":"2024-05-28T06:22:17.002306+0000","last_active":"2024-05-28T06:22:18.528617+0000","last_peered":"2024-05-28T06:22:18.528617+0000","last_clean":"2024-05-28T06:22:18.528617+0000","last_became_active":"2024-05-28T06:22:17.001965+0000","last_became_peered":"2024-05-28T06:22:17.001965+0000","last_unstale":"2024-05-28T06:22:18.528617+0000","last_undegraded":"2024-05-28T06:22:18.528617+0000","last_fullsized":"2024-05-28T06:22:18.528617+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:16:11.794755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.000927+0000","last_change":"2024-05-28T06:22:16.998168+0000","last_active":"2024-05-28T06:22:18.000927+0000","last_peered":"2024-05-28T06:22:18.000927+0000","last_clean":"2024-05-28T06:22:18.000927+0000","last_became_active":"2024-05-28T06:22:16.996215+0000","last_became_peered":"2024-05-28T06:22:16.996215+0000","last_unstale":"2024-05-28T06:22:18.000927+0000","last_undegraded":"2024-05-28T06:22:18.000927+0000","last_fullsized":"2024-05-28T06:22:18.000927+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:06:13.719079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.001186+0000","last_change":"2024-05-28T06:22:16.998348+0000","last_active":"2024-05-28T06:22:18.001186+0000","last_peered":"2024-05-28T06:22:18.001186+0000","last_clean":"2024-05-28T06:22:18.001186+0000","last_became_active":"2024-05-28T06:22:16.998024+0000","last_became_peered":"2024-05-28T06:22:16.998024+0000","last_unstale":"2024-05-28T06:22:18.001186+0000","last_undegraded":"2024-05-28T06:22:18.001186+0000","last_fullsized":"2024-05-28T06:22:18.001186+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:13:14.889721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.528790+0000","last_change":"2024-05-28T06:22:17.004708+0000","last_active":"2024-05-28T06:22:18.528790+0000","last_peered":"2024-05-28T06:22:18.528790+0000","last_clean":"2024-05-28T06:22:18.528790+0000","last_became_active":"2024-05-28T06:22:17.003676+0000","last_became_peered":"2024-05-28T06:22:17.003676+0000","last_unstale":"2024-05-28T06:22:18.528790+0000","last_undegraded":"2024-05-28T06:22:18.528790+0000","last_fullsized":"2024-05-28T06:22:18.528790+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:38:34.851302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.000139+0000","last_change":"2024-05-28T06:22:16.999079+0000","last_active":"2024-05-28T06:22:18.000139+0000","last_peered":"2024-05-28T06:22:18.000139+0000","last_clean":"2024-05-28T06:22:18.000139+0000","last_became_active":"2024-05-28T06:22:16.998795+0000","last_became_peered":"2024-05-28T06:22:16.998795+0000","last_unstale":"2024-05-28T06:22:18.000139+0000","last_undegraded":"2024-05-28T06:22:18.000139+0000","last_fullsized":"2024-05-28T06:22:18.000139+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:17:14.655934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.526910+0000","last_change":"2024-05-28T06:22:17.008315+0000","last_active":"2024-05-28T06:22:18.526910+0000","last_peered":"2024-05-28T06:22:18.526910+0000","last_clean":"2024-05-28T06:22:18.526910+0000","last_became_active":"2024-05-28T06:22:17.007995+0000","last_became_peered":"2024-05-28T06:22:17.007995+0000","last_unstale":"2024-05-28T06:22:18.526910+0000","last_undegraded":"2024-05-28T06:22:18.526910+0000","last_fullsized":"2024-05-28T06:22:18.526910+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T18:02:32.237669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.002108+0000","last_change":"2024-05-28T06:22:17.003291+0000","last_active":"2024-05-28T06:22:18.002108+0000","last_peered":"2024-05-28T06:22:18.002108+0000","last_clean":"2024-05-28T06:22:18.002108+0000","last_became_active":"2024-05-28T06:22:17.003018+0000","last_became_peered":"2024-05-28T06:22:17.003018+0000","last_unstale":"2024-05-28T06:22:18.002108+0000","last_undegraded":"2024-05-28T06:22:18.002108+0000","last_fullsized":"2024-05-28T06:22:18.002108+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:16:11.794755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.243265+0000","last_change":"2024-05-28T06:22:17.006855+0000","last_active":"2024-05-28T06:22:18.243265+0000","last_peered":"2024-05-28T06:22:18.243265+0000","last_clean":"2024-05-28T06:22:18.243265+0000","last_became_active":"2024-05-28T06:22:17.006649+0000","last_became_peered":"2024-05-28T06:22:17.006649+0000","last_unstale":"2024-05-28T06:22:18.243265+0000","last_undegraded":"2024-05-28T06:22:18.243265+0000","last_fullsized":"2024-05-28T06:22:18.243265+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:25:34.876194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.529987+0000","last_change":"2024-05-28T06:22:17.000165+0000","last_active":"2024-05-28T06:22:18.529987+0000","last_peered":"2024-05-28T06:22:18.529987+0000","last_clean":"2024-05-28T06:22:18.529987+0000","last_became_active":"2024-05-28T06:22:16.999904+0000","last_became_peered":"2024-05-28T06:22:16.999904+0000","last_unstale":"2024-05-28T06:22:18.529987+0000","last_undegraded":"2024-05-28T06:22:18.529987+0000","last_fullsized":"2024-05-28T06:22:18.529987+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:58:15.396668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.529632+0000","last_change":"2024-05-28T06:22:16.993958+0000","last_active":"2024-05-28T06:22:18.529632+0000","last_peered":"2024-05-28T06:22:18.529632+0000","last_clean":"2024-05-28T06:22:18.529632+0000","last_became_active":"2024-05-28T06:22:16.993327+0000","last_became_peered":"2024-05-28T06:22:16.993327+0000","last_unstale":"2024-05-28T06:22:18.529632+0000","last_undegraded":"2024-05-28T06:22:18.529632+0000","last_fullsized":"2024-05-28T06:22:18.529632+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:26:18.638055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.000269+0000","last_change":"2024-05-28T06:22:17.001399+0000","last_active":"2024-05-28T06:22:18.000269+0000","last_peered":"2024-05-28T06:22:18.000269+0000","last_clean":"2024-05-28T06:22:18.000269+0000","last_became_active":"2024-05-28T06:22:17.000579+0000","last_became_peered":"2024-05-28T06:22:17.000579+0000","last_unstale":"2024-05-28T06:22:18.000269+0000","last_undegraded":"2024-05-28T06:22:18.000269+0000","last_fullsized":"2024-05-28T06:22:18.000269+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:48:14.529168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.024533+0000","last_change":"2024-05-28T06:22:17.004330+0000","last_active":"2024-05-28T06:22:18.024533+0000","last_peered":"2024-05-28T06:22:18.024533+0000","last_clean":"2024-05-28T06:22:18.024533+0000","last_became_active":"2024-05-28T06:22:16.996955+0000","last_became_peered":"2024-05-28T06:22:16.996955+0000","last_unstale":"2024-05-28T06:22:18.024533+0000","last_undegraded":"2024-05-28T06:22:18.024533+0000","last_fullsized":"2024-05-28T06:22:18.024533+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:25:57.432011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.242168+0000","last_change":"2024-05-28T06:22:17.001908+0000","last_active":"2024-05-28T06:22:18.242168+0000","last_peered":"2024-05-28T06:22:18.242168+0000","last_clean":"2024-05-28T06:22:18.242168+0000","last_became_active":"2024-05-28T06:22:17.000543+0000","last_became_peered":"2024-05-28T06:22:17.000543+0000","last_unstale":"2024-05-28T06:22:18.242168+0000","last_undegraded":"2024-05-28T06:22:18.242168+0000","last_fullsized":"2024-05-28T06:22:18.242168+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:12:55.159821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.527587+0000","last_change":"2024-05-28T06:22:16.992715+0000","last_active":"2024-05-28T06:22:18.527587+0000","last_peered":"2024-05-28T06:22:18.527587+0000","last_clean":"2024-05-28T06:22:18.527587+0000","last_became_active":"2024-05-28T06:22:16.992094+0000","last_became_peered":"2024-05-28T06:22:16.992094+0000","last_unstale":"2024-05-28T06:22:18.527587+0000","last_undegraded":"2024-05-28T06:22:18.527587+0000","last_fullsized":"2024-05-28T06:22:18.527587+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:05:25.661249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.529840+0000","last_change":"2024-05-28T06:22:17.004255+0000","last_active":"2024-05-28T06:22:18.529840+0000","last_peered":"2024-05-28T06:22:18.529840+0000","last_clean":"2024-05-28T06:22:18.529840+0000","last_became_active":"2024-05-28T06:22:17.001691+0000","last_became_peered":"2024-05-28T06:22:17.001691+0000","last_unstale":"2024-05-28T06:22:18.529840+0000","last_undegraded":"2024-05-28T06:22:18.529840+0000","last_fullsized":"2024-05-28T06:22:18.529840+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:40:20.104894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"114'4","reported_seq":18,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.024524+0000","last_change":"2024-05-28T06:22:15.992751+0000","last_active":"2024-05-28T06:22:18.024524+0000","last_peered":"2024-05-28T06:22:18.024524+0000","last_clean":"2024-05-28T06:22:18.024524+0000","last_became_active":"2024-05-28T06:22:15.992113+0000","last_became_peered":"2024-05-28T06:22:15.992113+0000","last_unstale":"2024-05-28T06:22:18.024524+0000","last_undegraded":"2024-05-28T06:22:18.024524+0000","last_fullsized":"2024-05-28T06:22:18.024524+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:28:16.129726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.020384+0000","last_change":"2024-05-28T06:22:17.003592+0000","last_active":"2024-05-28T06:22:18.020384+0000","last_peered":"2024-05-28T06:22:18.020384+0000","last_clean":"2024-05-28T06:22:18.020384+0000","last_became_active":"2024-05-28T06:22:17.002423+0000","last_became_peered":"2024-05-28T06:22:17.002423+0000","last_unstale":"2024-05-28T06:22:18.020384+0000","last_undegraded":"2024-05-28T06:22:18.020384+0000","last_fullsized":"2024-05-28T06:22:18.020384+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:58:35.663956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"114'1","reported_seq":15,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.528549+0000","last_change":"2024-05-28T06:22:15.996401+0000","last_active":"2024-05-28T06:22:18.528549+0000","last_peered":"2024-05-28T06:22:18.528549+0000","last_clean":"2024-05-28T06:22:18.528549+0000","last_became_active":"2024-05-28T06:22:15.996039+0000","last_became_peered":"2024-05-28T06:22:15.996039+0000","last_unstale":"2024-05-28T06:22:18.528549+0000","last_undegraded":"2024-05-28T06:22:18.528549+0000","last_fullsized":"2024-05-28T06:22:18.528549+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:43:46.372846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.528998+0000","last_change":"2024-05-28T06:22:16.999053+0000","last_active":"2024-05-28T06:22:18.528998+0000","last_peered":"2024-05-28T06:22:18.528998+0000","last_clean":"2024-05-28T06:22:18.528998+0000","last_became_active":"2024-05-28T06:22:16.995589+0000","last_became_peered":"2024-05-28T06:22:16.995589+0000","last_unstale":"2024-05-28T06:22:18.528998+0000","last_undegraded":"2024-05-28T06:22:18.528998+0000","last_fullsized":"2024-05-28T06:22:18.528998+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:58:42.572818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"114'2","reported_seq":16,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:17.999806+0000","last_change":"2024-05-28T06:22:15.989244+0000","last_active":"2024-05-28T06:22:17.999806+0000","last_peered":"2024-05-28T06:22:17.999806+0000","last_clean":"2024-05-28T06:22:17.999806+0000","last_became_active":"2024-05-28T06:22:15.988137+0000","last_became_peered":"2024-05-28T06:22:15.988137+0000","last_unstale":"2024-05-28T06:22:17.999806+0000","last_undegraded":"2024-05-28T06:22:17.999806+0000","last_fullsized":"2024-05-28T06:22:17.999806+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T18:10:18.669151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.528637+0000","last_change":"2024-05-28T06:22:17.001702+0000","last_active":"2024-05-28T06:22:18.528637+0000","last_peered":"2024-05-28T06:22:18.528637+0000","last_clean":"2024-05-28T06:22:18.528637+0000","last_became_active":"2024-05-28T06:22:16.999756+0000","last_became_peered":"2024-05-28T06:22:16.999756+0000","last_unstale":"2024-05-28T06:22:18.528637+0000","last_undegraded":"2024-05-28T06:22:18.528637+0000","last_fullsized":"2024-05-28T06:22:18.528637+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:23:29.259681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"114'2","reported_seq":16,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:17.999369+0000","last_change":"2024-05-28T06:22:15.992414+0000","last_active":"2024-05-28T06:22:17.999369+0000","last_peered":"2024-05-28T06:22:17.999369+0000","last_clean":"2024-05-28T06:22:17.999369+0000","last_became_active":"2024-05-28T06:22:15.991608+0000","last_became_peered":"2024-05-28T06:22:15.991608+0000","last_unstale":"2024-05-28T06:22:17.999369+0000","last_undegraded":"2024-05-28T06:22:17.999369+0000","last_fullsized":"2024-05-28T06:22:17.999369+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:42:21.601868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.241943+0000","last_change":"2024-05-28T06:22:16.995582+0000","last_active":"2024-05-28T06:22:18.241943+0000","last_peered":"2024-05-28T06:22:18.241943+0000","last_clean":"2024-05-28T06:22:18.241943+0000","last_became_active":"2024-05-28T06:22:16.995242+0000","last_became_peered":"2024-05-28T06:22:16.995242+0000","last_unstale":"2024-05-28T06:22:18.241943+0000","last_undegraded":"2024-05-28T06:22:18.241943+0000","last_fullsized":"2024-05-28T06:22:18.241943+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:46:21.232752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"114'1","reported_seq":15,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.024104+0000","last_change":"2024-05-28T06:22:15.994783+0000","last_active":"2024-05-28T06:22:18.024104+0000","last_peered":"2024-05-28T06:22:18.024104+0000","last_clean":"2024-05-28T06:22:18.024104+0000","last_became_active":"2024-05-28T06:22:15.994208+0000","last_became_peered":"2024-05-28T06:22:15.994208+0000","last_unstale":"2024-05-28T06:22:18.024104+0000","last_undegraded":"2024-05-28T06:22:18.024104+0000","last_fullsized":"2024-05-28T06:22:18.024104+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:04:41.031193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.000748+0000","last_change":"2024-05-28T06:22:16.992756+0000","last_active":"2024-05-28T06:22:18.000748+0000","last_peered":"2024-05-28T06:22:18.000748+0000","last_clean":"2024-05-28T06:22:18.000748+0000","last_became_active":"2024-05-28T06:22:16.992331+0000","last_became_peered":"2024-05-28T06:22:16.992331+0000","last_unstale":"2024-05-28T06:22:18.000748+0000","last_undegraded":"2024-05-28T06:22:18.000748+0000","last_fullsized":"2024-05-28T06:22:18.000748+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:08:39.984746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"114'1","reported_seq":15,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.529094+0000","last_change":"2024-05-28T06:22:15.990828+0000","last_active":"2024-05-28T06:22:18.529094+0000","last_peered":"2024-05-28T06:22:18.529094+0000","last_clean":"2024-05-28T06:22:18.529094+0000","last_became_active":"2024-05-28T06:22:15.990537+0000","last_became_peered":"2024-05-28T06:22:15.990537+0000","last_unstale":"2024-05-28T06:22:18.529094+0000","last_undegraded":"2024-05-28T06:22:18.529094+0000","last_fullsized":"2024-05-28T06:22:18.529094+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:51:43.927844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.241796+0000","last_change":"2024-05-28T06:22:17.000804+0000","last_active":"2024-05-28T06:22:18.241796+0000","last_peered":"2024-05-28T06:22:18.241796+0000","last_clean":"2024-05-28T06:22:18.241796+0000","last_became_active":"2024-05-28T06:22:16.998428+0000","last_became_peered":"2024-05-28T06:22:16.998428+0000","last_unstale":"2024-05-28T06:22:18.241796+0000","last_undegraded":"2024-05-28T06:22:18.241796+0000","last_fullsized":"2024-05-28T06:22:18.241796+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:43:26.170749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.000006+0000","last_change":"2024-05-28T06:22:15.981969+0000","last_active":"2024-05-28T06:22:18.000006+0000","last_peered":"2024-05-28T06:22:18.000006+0000","last_clean":"2024-05-28T06:22:18.000006+0000","last_became_active":"2024-05-28T06:22:15.979073+0000","last_became_peered":"2024-05-28T06:22:15.979073+0000","last_unstale":"2024-05-28T06:22:18.000006+0000","last_undegraded":"2024-05-28T06:22:18.000006+0000","last_fullsized":"2024-05-28T06:22:18.000006+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:41:39.601468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.024757+0000","last_change":"2024-05-28T06:22:17.006594+0000","last_active":"2024-05-28T06:22:18.024757+0000","last_peered":"2024-05-28T06:22:18.024757+0000","last_clean":"2024-05-28T06:22:18.024757+0000","last_became_active":"2024-05-28T06:22:17.006057+0000","last_became_peered":"2024-05-28T06:22:17.006057+0000","last_unstale":"2024-05-28T06:22:18.024757+0000","last_undegraded":"2024-05-28T06:22:18.024757+0000","last_fullsized":"2024-05-28T06:22:18.024757+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:39:44.471675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.242254+0000","last_change":"2024-05-28T06:22:15.988357+0000","last_active":"2024-05-28T06:22:18.242254+0000","last_peered":"2024-05-28T06:22:18.242254+0000","last_clean":"2024-05-28T06:22:18.242254+0000","last_became_active":"2024-05-28T06:22:15.988053+0000","last_became_peered":"2024-05-28T06:22:15.988053+0000","last_unstale":"2024-05-28T06:22:18.242254+0000","last_undegraded":"2024-05-28T06:22:18.242254+0000","last_fullsized":"2024-05-28T06:22:18.242254+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:24:37.584832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.024688+0000","last_change":"2024-05-28T06:22:17.003420+0000","last_active":"2024-05-28T06:22:18.024688+0000","last_peered":"2024-05-28T06:22:18.024688+0000","last_clean":"2024-05-28T06:22:18.024688+0000","last_became_active":"2024-05-28T06:22:17.003005+0000","last_became_peered":"2024-05-28T06:22:17.003005+0000","last_unstale":"2024-05-28T06:22:18.024688+0000","last_undegraded":"2024-05-28T06:22:18.024688+0000","last_fullsized":"2024-05-28T06:22:18.024688+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:26:50.855556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:17.999693+0000","last_change":"2024-05-28T06:22:15.984544+0000","last_active":"2024-05-28T06:22:17.999693+0000","last_peered":"2024-05-28T06:22:17.999693+0000","last_clean":"2024-05-28T06:22:17.999693+0000","last_became_active":"2024-05-28T06:22:15.984287+0000","last_became_peered":"2024-05-28T06:22:15.984287+0000","last_unstale":"2024-05-28T06:22:17.999693+0000","last_undegraded":"2024-05-28T06:22:17.999693+0000","last_fullsized":"2024-05-28T06:22:17.999693+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T18:04:33.437207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.020450+0000","last_change":"2024-05-28T06:22:17.012046+0000","last_active":"2024-05-28T06:22:18.020450+0000","last_peered":"2024-05-28T06:22:18.020450+0000","last_clean":"2024-05-28T06:22:18.020450+0000","last_became_active":"2024-05-28T06:22:17.007050+0000","last_became_peered":"2024-05-28T06:22:17.007050+0000","last_unstale":"2024-05-28T06:22:18.020450+0000","last_undegraded":"2024-05-28T06:22:18.020450+0000","last_fullsized":"2024-05-28T06:22:18.020450+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:48:14.529168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.529413+0000","last_change":"2024-05-28T06:22:15.989398+0000","last_active":"2024-05-28T06:22:18.529413+0000","last_peered":"2024-05-28T06:22:18.529413+0000","last_clean":"2024-05-28T06:22:18.529413+0000","last_became_active":"2024-05-28T06:22:15.989101+0000","last_became_peered":"2024-05-28T06:22:15.989101+0000","last_unstale":"2024-05-28T06:22:18.529413+0000","last_undegraded":"2024-05-28T06:22:18.529413+0000","last_fullsized":"2024-05-28T06:22:18.529413+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:43:46.372846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.529711+0000","last_change":"2024-05-28T06:22:16.993645+0000","last_active":"2024-05-28T06:22:18.529711+0000","last_peered":"2024-05-28T06:22:18.529711+0000","last_clean":"2024-05-28T06:22:18.529711+0000","last_became_active":"2024-05-28T06:22:16.992912+0000","last_became_peered":"2024-05-28T06:22:16.992912+0000","last_unstale":"2024-05-28T06:22:18.529711+0000","last_undegraded":"2024-05-28T06:22:18.529711+0000","last_fullsized":"2024-05-28T06:22:18.529711+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:16:11.794755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"114'3","reported_seq":17,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.242908+0000","last_change":"2024-05-28T06:22:15.998469+0000","last_active":"2024-05-28T06:22:18.242908+0000","last_peered":"2024-05-28T06:22:18.242908+0000","last_clean":"2024-05-28T06:22:18.242908+0000","last_became_active":"2024-05-28T06:22:15.998186+0000","last_became_peered":"2024-05-28T06:22:15.998186+0000","last_unstale":"2024-05-28T06:22:18.242908+0000","last_undegraded":"2024-05-28T06:22:18.242908+0000","last_fullsized":"2024-05-28T06:22:18.242908+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:23:41.228363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.024744+0000","last_change":"2024-05-28T06:22:17.009032+0000","last_active":"2024-05-28T06:22:18.024744+0000","last_peered":"2024-05-28T06:22:18.024744+0000","last_clean":"2024-05-28T06:22:18.024744+0000","last_became_active":"2024-05-28T06:22:17.007965+0000","last_became_peered":"2024-05-28T06:22:17.007965+0000","last_unstale":"2024-05-28T06:22:18.024744+0000","last_undegraded":"2024-05-28T06:22:18.024744+0000","last_fullsized":"2024-05-28T06:22:18.024744+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:13:32.315186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"114'6","reported_seq":20,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.243514+0000","last_change":"2024-05-28T06:22:15.997183+0000","last_active":"2024-05-28T06:22:18.243514+0000","last_peered":"2024-05-28T06:22:18.243514+0000","last_clean":"2024-05-28T06:22:18.243514+0000","last_became_active":"2024-05-28T06:22:15.996809+0000","last_became_peered":"2024-05-28T06:22:15.996809+0000","last_unstale":"2024-05-28T06:22:18.243514+0000","last_undegraded":"2024-05-28T06:22:18.243514+0000","last_fullsized":"2024-05-28T06:22:18.243514+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:38:33.839493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.528917+0000","last_change":"2024-05-28T06:22:17.000291+0000","last_active":"2024-05-28T06:22:18.528917+0000","last_peered":"2024-05-28T06:22:18.528917+0000","last_clean":"2024-05-28T06:22:18.528917+0000","last_became_active":"2024-05-28T06:22:16.999910+0000","last_became_peered":"2024-05-28T06:22:16.999910+0000","last_unstale":"2024-05-28T06:22:18.528917+0000","last_undegraded":"2024-05-28T06:22:18.528917+0000","last_fullsized":"2024-05-28T06:22:18.528917+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:23:42.240172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"114'2","reported_seq":16,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.528553+0000","last_change":"2024-05-28T06:22:15.996773+0000","last_active":"2024-05-28T06:22:18.528553+0000","last_peered":"2024-05-28T06:22:18.528553+0000","last_clean":"2024-05-28T06:22:18.528553+0000","last_became_active":"2024-05-28T06:22:15.995681+0000","last_became_peered":"2024-05-28T06:22:15.995681+0000","last_unstale":"2024-05-28T06:22:18.528553+0000","last_undegraded":"2024-05-28T06:22:18.528553+0000","last_fullsized":"2024-05-28T06:22:18.528553+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:48:09.956343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.528770+0000","last_change":"2024-05-28T06:22:17.003164+0000","last_active":"2024-05-28T06:22:18.528770+0000","last_peered":"2024-05-28T06:22:18.528770+0000","last_clean":"2024-05-28T06:22:18.528770+0000","last_became_active":"2024-05-28T06:22:17.002906+0000","last_became_peered":"2024-05-28T06:22:17.002906+0000","last_unstale":"2024-05-28T06:22:18.528770+0000","last_undegraded":"2024-05-28T06:22:18.528770+0000","last_fullsized":"2024-05-28T06:22:18.528770+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:13:14.889721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.020651+0000","last_change":"2024-05-28T06:22:15.993511+0000","last_active":"2024-05-28T06:22:18.020651+0000","last_peered":"2024-05-28T06:22:18.020651+0000","last_clean":"2024-05-28T06:22:18.020651+0000","last_became_active":"2024-05-28T06:22:15.992788+0000","last_became_peered":"2024-05-28T06:22:15.992788+0000","last_unstale":"2024-05-28T06:22:18.020651+0000","last_undegraded":"2024-05-28T06:22:18.020651+0000","last_fullsized":"2024-05-28T06:22:18.020651+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:43:46.372846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.527440+0000","last_change":"2024-05-28T06:22:16.991808+0000","last_active":"2024-05-28T06:22:18.527440+0000","last_peered":"2024-05-28T06:22:18.527440+0000","last_clean":"2024-05-28T06:22:18.527440+0000","last_became_active":"2024-05-28T06:22:16.991494+0000","last_became_peered":"2024-05-28T06:22:16.991494+0000","last_unstale":"2024-05-28T06:22:18.527440+0000","last_undegraded":"2024-05-28T06:22:18.527440+0000","last_fullsized":"2024-05-28T06:22:18.527440+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:47:18.486697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"114'8","reported_seq":22,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:17.999084+0000","last_change":"2024-05-28T06:22:15.991378+0000","last_active":"2024-05-28T06:22:17.999084+0000","last_peered":"2024-05-28T06:22:17.999084+0000","last_clean":"2024-05-28T06:22:17.999084+0000","last_became_active":"2024-05-28T06:22:15.990388+0000","last_became_peered":"2024-05-28T06:22:15.990388+0000","last_unstale":"2024-05-28T06:22:17.999084+0000","last_undegraded":"2024-05-28T06:22:17.999084+0000","last_fullsized":"2024-05-28T06:22:17.999084+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:49:10.819709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:17.999777+0000","last_change":"2024-05-28T06:22:16.991114+0000","last_active":"2024-05-28T06:22:17.999777+0000","last_peered":"2024-05-28T06:22:17.999777+0000","last_clean":"2024-05-28T06:22:17.999777+0000","last_became_active":"2024-05-28T06:22:16.990159+0000","last_became_peered":"2024-05-28T06:22:16.990159+0000","last_unstale":"2024-05-28T06:22:17.999777+0000","last_undegraded":"2024-05-28T06:22:17.999777+0000","last_fullsized":"2024-05-28T06:22:17.999777+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:18:01.989158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"114'1","reported_seq":15,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.242742+0000","last_change":"2024-05-28T06:22:15.997418+0000","last_active":"2024-05-28T06:22:18.242742+0000","last_peered":"2024-05-28T06:22:18.242742+0000","last_clean":"2024-05-28T06:22:18.242742+0000","last_became_active":"2024-05-28T06:22:15.995786+0000","last_became_peered":"2024-05-28T06:22:15.995786+0000","last_unstale":"2024-05-28T06:22:18.242742+0000","last_undegraded":"2024-05-28T06:22:18.242742+0000","last_fullsized":"2024-05-28T06:22:18.242742+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:50:11.148336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.530295+0000","last_change":"2024-05-28T06:22:16.993861+0000","last_active":"2024-05-28T06:22:18.530295+0000","last_peered":"2024-05-28T06:22:18.530295+0000","last_clean":"2024-05-28T06:22:18.530295+0000","last_became_active":"2024-05-28T06:22:16.993602+0000","last_became_peered":"2024-05-28T06:22:16.993602+0000","last_unstale":"2024-05-28T06:22:18.530295+0000","last_undegraded":"2024-05-28T06:22:18.530295+0000","last_fullsized":"2024-05-28T06:22:18.530295+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:28:17.141535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"114'2","reported_seq":16,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:17.999975+0000","last_change":"2024-05-28T06:22:15.980689+0000","last_active":"2024-05-28T06:22:17.999975+0000","last_peered":"2024-05-28T06:22:17.999975+0000","last_clean":"2024-05-28T06:22:17.999975+0000","last_became_active":"2024-05-28T06:22:15.980139+0000","last_became_peered":"2024-05-28T06:22:15.980139+0000","last_unstale":"2024-05-28T06:22:17.999975+0000","last_undegraded":"2024-05-28T06:22:17.999975+0000","last_fullsized":"2024-05-28T06:22:17.999975+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:19:21.410923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.528248+0000","last_change":"2024-05-28T06:22:16.999644+0000","last_active":"2024-05-28T06:22:18.528248+0000","last_peered":"2024-05-28T06:22:18.528248+0000","last_clean":"2024-05-28T06:22:18.528248+0000","last_became_active":"2024-05-28T06:22:16.998935+0000","last_became_peered":"2024-05-28T06:22:16.998935+0000","last_unstale":"2024-05-28T06:22:18.528248+0000","last_undegraded":"2024-05-28T06:22:18.528248+0000","last_fullsized":"2024-05-28T06:22:18.528248+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:26:18.638055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"114'1","reported_seq":15,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.019924+0000","last_change":"2024-05-28T06:22:15.996123+0000","last_active":"2024-05-28T06:22:18.019924+0000","last_peered":"2024-05-28T06:22:18.019924+0000","last_clean":"2024-05-28T06:22:18.019924+0000","last_became_active":"2024-05-28T06:22:15.995728+0000","last_became_peered":"2024-05-28T06:22:15.995728+0000","last_unstale":"2024-05-28T06:22:18.019924+0000","last_undegraded":"2024-05-28T06:22:18.019924+0000","last_fullsized":"2024-05-28T06:22:18.019924+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:08:38.270230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.528924+0000","last_change":"2024-05-28T06:22:17.003252+0000","last_active":"2024-05-28T06:22:18.528924+0000","last_peered":"2024-05-28T06:22:18.528924+0000","last_clean":"2024-05-28T06:22:18.528924+0000","last_became_active":"2024-05-28T06:22:17.001195+0000","last_became_peered":"2024-05-28T06:22:17.001195+0000","last_unstale":"2024-05-28T06:22:18.528924+0000","last_undegraded":"2024-05-28T06:22:18.528924+0000","last_fullsized":"2024-05-28T06:22:18.528924+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:27:26.116768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.024177+0000","last_change":"2024-05-28T06:22:15.998545+0000","last_active":"2024-05-28T06:22:18.024177+0000","last_peered":"2024-05-28T06:22:18.024177+0000","last_clean":"2024-05-28T06:22:18.024177+0000","last_became_active":"2024-05-28T06:22:15.997754+0000","last_became_peered":"2024-05-28T06:22:15.997754+0000","last_unstale":"2024-05-28T06:22:18.024177+0000","last_undegraded":"2024-05-28T06:22:18.024177+0000","last_fullsized":"2024-05-28T06:22:18.024177+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:58:41.561009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.527487+0000","last_change":"2024-05-28T06:22:17.000185+0000","last_active":"2024-05-28T06:22:18.527487+0000","last_peered":"2024-05-28T06:22:18.527487+0000","last_clean":"2024-05-28T06:22:18.527487+0000","last_became_active":"2024-05-28T06:22:16.997369+0000","last_became_peered":"2024-05-28T06:22:16.997369+0000","last_unstale":"2024-05-28T06:22:18.527487+0000","last_undegraded":"2024-05-28T06:22:18.527487+0000","last_fullsized":"2024-05-28T06:22:18.527487+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:08:39.984746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"114'2","reported_seq":16,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.528872+0000","last_change":"2024-05-28T06:22:15.991711+0000","last_active":"2024-05-28T06:22:18.528872+0000","last_peered":"2024-05-28T06:22:18.528872+0000","last_clean":"2024-05-28T06:22:18.528872+0000","last_became_active":"2024-05-28T06:22:15.991323+0000","last_became_peered":"2024-05-28T06:22:15.991323+0000","last_unstale":"2024-05-28T06:22:18.528872+0000","last_undegraded":"2024-05-28T06:22:18.528872+0000","last_fullsized":"2024-05-28T06:22:18.528872+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:02:22.717474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.242729+0000","last_change":"2024-05-28T06:22:17.003257+0000","last_active":"2024-05-28T06:22:18.242729+0000","last_peered":"2024-05-28T06:22:18.242729+0000","last_clean":"2024-05-28T06:22:18.242729+0000","last_became_active":"2024-05-28T06:22:17.000808+0000","last_became_peered":"2024-05-28T06:22:17.000808+0000","last_unstale":"2024-05-28T06:22:18.242729+0000","last_undegraded":"2024-05-28T06:22:18.242729+0000","last_fullsized":"2024-05-28T06:22:18.242729+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:27:48.276213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:17.999010+0000","last_change":"2024-05-28T06:22:15.994076+0000","last_active":"2024-05-28T06:22:17.999010+0000","last_peered":"2024-05-28T06:22:17.999010+0000","last_clean":"2024-05-28T06:22:17.999010+0000","last_became_active":"2024-05-28T06:22:15.993488+0000","last_became_peered":"2024-05-28T06:22:15.993488+0000","last_unstale":"2024-05-28T06:22:17.999010+0000","last_undegraded":"2024-05-28T06:22:17.999010+0000","last_fullsized":"2024-05-28T06:22:17.999010+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:08:35.517640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.527460+0000","last_change":"2024-05-28T06:22:16.995803+0000","last_active":"2024-05-28T06:22:18.527460+0000","last_peered":"2024-05-28T06:22:18.527460+0000","last_clean":"2024-05-28T06:22:18.527460+0000","last_became_active":"2024-05-28T06:22:16.995475+0000","last_became_peered":"2024-05-28T06:22:16.995475+0000","last_unstale":"2024-05-28T06:22:18.527460+0000","last_undegraded":"2024-05-28T06:22:18.527460+0000","last_fullsized":"2024-05-28T06:22:18.527460+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T18:02:32.237669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.527971+0000","last_change":"2024-05-28T06:22:15.989588+0000","last_active":"2024-05-28T06:22:18.527971+0000","last_peered":"2024-05-28T06:22:18.527971+0000","last_clean":"2024-05-28T06:22:18.527971+0000","last_became_active":"2024-05-28T06:22:15.989331+0000","last_became_peered":"2024-05-28T06:22:15.989331+0000","last_unstale":"2024-05-28T06:22:18.527971+0000","last_undegraded":"2024-05-28T06:22:18.527971+0000","last_fullsized":"2024-05-28T06:22:18.527971+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:19:21.410923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.527956+0000","last_change":"2024-05-28T06:22:16.993785+0000","last_active":"2024-05-28T06:22:18.527956+0000","last_peered":"2024-05-28T06:22:18.527956+0000","last_clean":"2024-05-28T06:22:18.527956+0000","last_became_active":"2024-05-28T06:22:16.993402+0000","last_became_peered":"2024-05-28T06:22:16.993402+0000","last_unstale":"2024-05-28T06:22:18.527956+0000","last_undegraded":"2024-05-28T06:22:18.527956+0000","last_fullsized":"2024-05-28T06:22:18.527956+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:06:13.719079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.242342+0000","last_change":"2024-05-28T06:22:15.998948+0000","last_active":"2024-05-28T06:22:18.242342+0000","last_peered":"2024-05-28T06:22:18.242342+0000","last_clean":"2024-05-28T06:22:18.242342+0000","last_became_active":"2024-05-28T06:22:15.998205+0000","last_became_peered":"2024-05-28T06:22:15.998205+0000","last_unstale":"2024-05-28T06:22:18.242342+0000","last_undegraded":"2024-05-28T06:22:18.242342+0000","last_fullsized":"2024-05-28T06:22:18.242342+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:58:34.652147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.020433+0000","last_change":"2024-05-28T06:22:17.005113+0000","last_active":"2024-05-28T06:22:18.020433+0000","last_peered":"2024-05-28T06:22:18.020433+0000","last_clean":"2024-05-28T06:22:18.020433+0000","last_became_active":"2024-05-28T06:22:17.003020+0000","last_became_peered":"2024-05-28T06:22:17.003020+0000","last_unstale":"2024-05-28T06:22:18.020433+0000","last_undegraded":"2024-05-28T06:22:18.020433+0000","last_fullsized":"2024-05-28T06:22:18.020433+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:17:14.655934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"114'2","reported_seq":16,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.528934+0000","last_change":"2024-05-28T06:22:15.990864+0000","last_active":"2024-05-28T06:22:18.528934+0000","last_peered":"2024-05-28T06:22:18.528934+0000","last_clean":"2024-05-28T06:22:18.528934+0000","last_became_active":"2024-05-28T06:22:15.990559+0000","last_became_peered":"2024-05-28T06:22:15.990559+0000","last_unstale":"2024-05-28T06:22:18.528934+0000","last_undegraded":"2024-05-28T06:22:18.528934+0000","last_fullsized":"2024-05-28T06:22:18.528934+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:51:36.120307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.242633+0000","last_change":"2024-05-28T06:22:16.992471+0000","last_active":"2024-05-28T06:22:18.242633+0000","last_peered":"2024-05-28T06:22:18.242633+0000","last_clean":"2024-05-28T06:22:18.242633+0000","last_became_active":"2024-05-28T06:22:16.992135+0000","last_became_peered":"2024-05-28T06:22:16.992135+0000","last_unstale":"2024-05-28T06:22:18.242633+0000","last_undegraded":"2024-05-28T06:22:18.242633+0000","last_fullsized":"2024-05-28T06:22:18.242633+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:57:13.651588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"115'1","reported_seq":15,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.020608+0000","last_change":"2024-05-28T06:22:15.994161+0000","last_active":"2024-05-28T06:22:18.020608+0000","last_peered":"2024-05-28T06:22:18.020608+0000","last_clean":"2024-05-28T06:22:18.020608+0000","last_became_active":"2024-05-28T06:22:15.993895+0000","last_became_peered":"2024-05-28T06:22:15.993895+0000","last_unstale":"2024-05-28T06:22:18.020608+0000","last_undegraded":"2024-05-28T06:22:18.020608+0000","last_fullsized":"2024-05-28T06:22:18.020608+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:51:36.120307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.243620+0000","last_change":"2024-05-28T06:22:16.993934+0000","last_active":"2024-05-28T06:22:18.243620+0000","last_peered":"2024-05-28T06:22:18.243620+0000","last_clean":"2024-05-28T06:22:18.243620+0000","last_became_active":"2024-05-28T06:22:16.993249+0000","last_became_peered":"2024-05-28T06:22:16.993249+0000","last_unstale":"2024-05-28T06:22:18.243620+0000","last_undegraded":"2024-05-28T06:22:18.243620+0000","last_fullsized":"2024-05-28T06:22:18.243620+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:21:11.675368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.242990+0000","last_change":"2024-05-28T06:22:15.998999+0000","last_active":"2024-05-28T06:22:18.242990+0000","last_peered":"2024-05-28T06:22:18.242990+0000","last_clean":"2024-05-28T06:22:18.242990+0000","last_became_active":"2024-05-28T06:22:15.998631+0000","last_became_peered":"2024-05-28T06:22:15.998631+0000","last_unstale":"2024-05-28T06:22:18.242990+0000","last_undegraded":"2024-05-28T06:22:18.242990+0000","last_fullsized":"2024-05-28T06:22:18.242990+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:26:17.626246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.020122+0000","last_change":"2024-05-28T06:22:17.002213+0000","last_active":"2024-05-28T06:22:18.020122+0000","last_peered":"2024-05-28T06:22:18.020122+0000","last_clean":"2024-05-28T06:22:18.020122+0000","last_became_active":"2024-05-28T06:22:17.001843+0000","last_became_peered":"2024-05-28T06:22:17.001843+0000","last_unstale":"2024-05-28T06:22:18.020122+0000","last_undegraded":"2024-05-28T06:22:18.020122+0000","last_fullsized":"2024-05-28T06:22:18.020122+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:24:38.596641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.529805+0000","last_change":"2024-05-28T06:22:15.995193+0000","last_active":"2024-05-28T06:22:18.529805+0000","last_peered":"2024-05-28T06:22:18.529805+0000","last_clean":"2024-05-28T06:22:18.529805+0000","last_became_active":"2024-05-28T06:22:15.994748+0000","last_became_peered":"2024-05-28T06:22:15.994748+0000","last_unstale":"2024-05-28T06:22:18.529805+0000","last_undegraded":"2024-05-28T06:22:18.529805+0000","last_fullsized":"2024-05-28T06:22:18.529805+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:08:38.270230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.000212+0000","last_change":"2024-05-28T06:22:17.005154+0000","last_active":"2024-05-28T06:22:18.000212+0000","last_peered":"2024-05-28T06:22:18.000212+0000","last_clean":"2024-05-28T06:22:18.000212+0000","last_became_active":"2024-05-28T06:22:17.004836+0000","last_became_peered":"2024-05-28T06:22:17.004836+0000","last_unstale":"2024-05-28T06:22:18.000212+0000","last_undegraded":"2024-05-28T06:22:18.000212+0000","last_fullsized":"2024-05-28T06:22:18.000212+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T18:02:32.237669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.024271+0000","last_change":"2024-05-28T06:22:15.996930+0000","last_active":"2024-05-28T06:22:18.024271+0000","last_peered":"2024-05-28T06:22:18.024271+0000","last_clean":"2024-05-28T06:22:18.024271+0000","last_became_active":"2024-05-28T06:22:15.996598+0000","last_became_peered":"2024-05-28T06:22:15.996598+0000","last_unstale":"2024-05-28T06:22:18.024271+0000","last_undegraded":"2024-05-28T06:22:18.024271+0000","last_fullsized":"2024-05-28T06:22:18.024271+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:37:04.441468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.000755+0000","last_change":"2024-05-28T06:22:16.983951+0000","last_active":"2024-05-28T06:22:18.000755+0000","last_peered":"2024-05-28T06:22:18.000755+0000","last_clean":"2024-05-28T06:22:18.000755+0000","last_became_active":"2024-05-28T06:22:16.983201+0000","last_became_peered":"2024-05-28T06:22:16.983201+0000","last_unstale":"2024-05-28T06:22:18.000755+0000","last_undegraded":"2024-05-28T06:22:18.000755+0000","last_fullsized":"2024-05-28T06:22:18.000755+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:47:18.486697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.000007+0000","last_change":"2024-05-28T06:22:15.985622+0000","last_active":"2024-05-28T06:22:18.000007+0000","last_peered":"2024-05-28T06:22:18.000007+0000","last_clean":"2024-05-28T06:22:18.000007+0000","last_became_active":"2024-05-28T06:22:15.984560+0000","last_became_peered":"2024-05-28T06:22:15.984560+0000","last_unstale":"2024-05-28T06:22:18.000007+0000","last_undegraded":"2024-05-28T06:22:18.000007+0000","last_fullsized":"2024-05-28T06:22:18.000007+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:21:24.669839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.527773+0000","last_change":"2024-05-28T06:22:17.005937+0000","last_active":"2024-05-28T06:22:18.527773+0000","last_peered":"2024-05-28T06:22:18.527773+0000","last_clean":"2024-05-28T06:22:18.527773+0000","last_became_active":"2024-05-28T06:22:17.004354+0000","last_became_peered":"2024-05-28T06:22:17.004354+0000","last_unstale":"2024-05-28T06:22:18.527773+0000","last_undegraded":"2024-05-28T06:22:18.527773+0000","last_fullsized":"2024-05-28T06:22:18.527773+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:13:14.889721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:17.999121+0000","last_change":"2024-05-28T06:22:15.984208+0000","last_active":"2024-05-28T06:22:17.999121+0000","last_peered":"2024-05-28T06:22:17.999121+0000","last_clean":"2024-05-28T06:22:17.999121+0000","last_became_active":"2024-05-28T06:22:15.983816+0000","last_became_peered":"2024-05-28T06:22:15.983816+0000","last_unstale":"2024-05-28T06:22:17.999121+0000","last_undegraded":"2024-05-28T06:22:17.999121+0000","last_fullsized":"2024-05-28T06:22:17.999121+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:30:43.782132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.242161+0000","last_change":"2024-05-28T06:22:16.996830+0000","last_active":"2024-05-28T06:22:18.242161+0000","last_peered":"2024-05-28T06:22:18.242161+0000","last_clean":"2024-05-28T06:22:18.242161+0000","last_became_active":"2024-05-28T06:22:16.990102+0000","last_became_peered":"2024-05-28T06:22:16.990102+0000","last_unstale":"2024-05-28T06:22:18.242161+0000","last_undegraded":"2024-05-28T06:22:18.242161+0000","last_fullsized":"2024-05-28T06:22:18.242161+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T18:04:15.407639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.242532+0000","last_change":"2024-05-28T06:22:15.998500+0000","last_active":"2024-05-28T06:22:18.242532+0000","last_peered":"2024-05-28T06:22:18.242532+0000","last_clean":"2024-05-28T06:22:18.242532+0000","last_became_active":"2024-05-28T06:22:15.998176+0000","last_became_peered":"2024-05-28T06:22:15.998176+0000","last_unstale":"2024-05-28T06:22:18.242532+0000","last_undegraded":"2024-05-28T06:22:18.242532+0000","last_fullsized":"2024-05-28T06:22:18.242532+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:17:13.644125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.527150+0000","last_change":"2024-05-28T06:22:17.008104+0000","last_active":"2024-05-28T06:22:18.527150+0000","last_peered":"2024-05-28T06:22:18.527150+0000","last_clean":"2024-05-28T06:22:18.527150+0000","last_became_active":"2024-05-28T06:22:17.007752+0000","last_became_peered":"2024-05-28T06:22:17.007752+0000","last_unstale":"2024-05-28T06:22:18.527150+0000","last_undegraded":"2024-05-28T06:22:18.527150+0000","last_fullsized":"2024-05-28T06:22:18.527150+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:08:39.984746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.024388+0000","last_change":"2024-05-28T06:22:15.998356+0000","last_active":"2024-05-28T06:22:18.024388+0000","last_peered":"2024-05-28T06:22:18.024388+0000","last_clean":"2024-05-28T06:22:18.024388+0000","last_became_active":"2024-05-28T06:22:15.998063+0000","last_became_peered":"2024-05-28T06:22:15.998063+0000","last_unstale":"2024-05-28T06:22:18.024388+0000","last_undegraded":"2024-05-28T06:22:18.024388+0000","last_fullsized":"2024-05-28T06:22:18.024388+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:05:24.649440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":354,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.528168+0000","last_change":"2024-05-28T06:04:01.039129+0000","last_active":"2024-05-28T06:22:18.528168+0000","last_peered":"2024-05-28T06:22:18.528168+0000","last_clean":"2024-05-28T06:22:18.528168+0000","last_became_active":"2024-05-28T06:04:01.033906+0000","last_became_peered":"2024-05-28T06:04:01.033906+0000","last_unstale":"2024-05-28T06:22:18.528168+0000","last_undegraded":"2024-05-28T06:22:18.528168+0000","last_fullsized":"2024-05-28T06:22:18.528168+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:03:59.864237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:03:59.864237+0000","last_clean_scrub_stamp":"2024-05-28T06:03:59.864237+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:17:36.440245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575071,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":127500,"kb_used_data":1984,"kb_used_omap":1,"kb_used_meta":125502,"kb_avail":94244340,"statfs":{"total":96636764160,"available":96506204160,"internally_reserved":0,"allocated":2031616,"data_stored":1217255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":128514506},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575073,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":126916,"kb_used_data":1400,"kb_used_omap":1,"kb_used_meta":125502,"kb_avail":94244924,"statfs":{"total":96636764160,"available":96506802176,"internally_reserved":0,"allocated":1433600,"data_stored":626341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":128514502},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575071,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64464,"kb_used_data":1404,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94307376,"statfs":{"total":96636764160,"available":96570753024,"internally_reserved":0,"allocated":1437696,"data_stored":626875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575071,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":86656,"kb_used_data":1396,"kb_used_omap":1,"kb_used_meta":85246,"kb_avail":94285184,"statfs":{"total":96636764160,"available":96548028416,"internally_reserved":0,"allocated":1429504,"data_stored":626271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87292357},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575073,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":131096,"kb_used_data":1412,"kb_used_omap":1,"kb_used_meta":129662,"kb_avail":94240744,"statfs":{"total":96636764160,"available":96502521856,"internally_reserved":0,"allocated":1445888,"data_stored":627953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":132774343},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575072,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":144960,"kb_used_data":1396,"kb_used_omap":1,"kb_used_meta":143550,"kb_avail":94226880,"statfs":{"total":96636764160,"available":96488325120,"internally_reserved":0,"allocated":1429504,"data_stored":626271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":146995653},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575072,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":126920,"kb_used_data":1404,"kb_used_omap":1,"kb_used_meta":125502,"kb_avail":94244920,"statfs":{"total":96636764160,"available":96506798080,"internally_reserved":0,"allocated":1437696,"data_stored":626395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":128514501},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575071,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":87260,"kb_used_data":1996,"kb_used_omap":1,"kb_used_meta":85246,"kb_avail":94284580,"statfs":{"total":96636764160,"available":96547409920,"internally_reserved":0,"allocated":2043904,"data_stored":1218363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87292357},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-28T06:22:22.436 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-05-28T06:22:22.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.679+0000 7fb43ac00640 1 Processor -- start 2024-05-28T06:22:22.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.679+0000 7fb43ac00640 1 -- start start 2024-05-28T06:22:22.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.680+0000 7fb43ac00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb43415ae10 0x7fb4340cd7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:22.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.680+0000 7fb43ac00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fb43406f1a0 0x7fb434071cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:22.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.680+0000 7fb43ac00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb43406f6f0 con 0x7fb4340b41c0 2024-05-28T06:22:22.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.680+0000 7fb43ac00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fb43406f9d0 con 0x7fb43406f1a0 2024-05-28T06:22:22.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.680+0000 7fb43ac00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fb43406fcb0 con 0x7fb43415ae10 2024-05-28T06:22:22.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.681+0000 7fb42b400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fb43406f1a0 0x7fb434071cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:22.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.681+0000 7fb433e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb43415ae10 0x7fb4340cd7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:22.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.681+0000 7fb433e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb43415ae10 0x7fb4340cd7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.96:38382/0 (socket says 172.21.15.96:38382) 2024-05-28T06:22:22.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.681+0000 7fb433e00640 1 -- 172.21.15.96:0/857929930 learned_addr learned my addr 172.21.15.96:0/857929930 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:22:22.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.681+0000 7fb42b400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fb43406f1a0 0x7fb434071cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.96:58774/0 (socket says 172.21.15.96:58774) 2024-05-28T06:22:22.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.682+0000 7fb42b400640 1 -- 172.21.15.96:0/857929930 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb43415ae10 msgr2=0x7fb4340cd7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:22.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.682+0000 7fb42b400640 1 --2- 172.21.15.96:0/857929930 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb43415ae10 0x7fb4340cd7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:22.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.682+0000 7fb42b400640 1 -- 172.21.15.96:0/857929930 >> v1:172.21.15.123:6789/0 conn(0x7fb4340b41c0 legacy=0x7fb43415ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:22.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.682+0000 7fb42b400640 1 -- 172.21.15.96:0/857929930 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb43406ff90 con 0x7fb43406f1a0 2024-05-28T06:22:22.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.683+0000 7fb42b400640 1 --2- 172.21.15.96:0/857929930 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb43406f1a0 0x7fb434071cc0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fb41c0097a0 tx=0x7fb41c02a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=58117755651ea026 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:22.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.684+0000 7fb433400640 1 -- 172.21.15.96:0/857929930 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb41c037070 con 0x7fb43406f1a0 2024-05-28T06:22:22.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.684+0000 7fb433400640 1 -- 172.21.15.96:0/857929930 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb41c032440 con 0x7fb43406f1a0 2024-05-28T06:22:22.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.684+0000 7fb433400640 1 -- 172.21.15.96:0/857929930 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb41c032810 con 0x7fb43406f1a0 2024-05-28T06:22:22.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.685+0000 7fb43ac00640 1 -- 172.21.15.96:0/857929930 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb43406f1a0 msgr2=0x7fb434071cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:22.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.685+0000 7fb43ac00640 1 --2- 172.21.15.96:0/857929930 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb43406f1a0 0x7fb434071cc0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fb41c0097a0 tx=0x7fb41c02a180 comp rx=0 tx=0).stop 2024-05-28T06:22:22.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.685+0000 7fb43ac00640 1 -- 172.21.15.96:0/857929930 shutdown_connections 2024-05-28T06:22:22.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.685+0000 7fb43ac00640 1 --2- 172.21.15.96:0/857929930 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb43406f1a0 0x7fb434071cc0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:22.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.685+0000 7fb43ac00640 1 --2- 172.21.15.96:0/857929930 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb43415ae10 0x7fb4340cd7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:22.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.685+0000 7fb43ac00640 1 -- 172.21.15.96:0/857929930 >> 172.21.15.96:0/857929930 conn(0x7fb434067ce0 msgr2=0x7fb43406b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:22.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.685+0000 7fb43ac00640 1 -- 172.21.15.96:0/857929930 shutdown_connections 2024-05-28T06:22:22.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.685+0000 7fb43ac00640 1 -- 172.21.15.96:0/857929930 wait complete. 2024-05-28T06:22:22.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.686+0000 7fb43ac00640 1 Processor -- start 2024-05-28T06:22:22.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.687+0000 7fb43ac00640 1 -- start start 2024-05-28T06:22:22.684 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.687+0000 7fb43ac00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb43406f1a0 0x7fb434168d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:22.684 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.687+0000 7fb43ac00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb4340b41c0 0x7fb43415a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:22.684 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.687+0000 7fb43ac00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb43415ae10 0x7fb434146ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:22.684 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.687+0000 7fb43ac00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb4340b3fd0 con 0x7fb43415ae10 2024-05-28T06:22:22.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.687+0000 7fb43ac00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fb4340b3e50 con 0x7fb4340b41c0 2024-05-28T06:22:22.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.687+0000 7fb43ac00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fb43415f510 con 0x7fb43406f1a0 2024-05-28T06:22:22.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.688+0000 7fb438c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb43415ae10 0x7fb434146ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:22.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.688+0000 7fb438c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb43415ae10 0x7fb434146ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:60242/0 (socket says 172.21.15.96:60242) 2024-05-28T06:22:22.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.688+0000 7fb438c00640 1 -- 172.21.15.96:0/3960673830 learned_addr learned my addr 172.21.15.96:0/3960673830 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:22:22.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.688+0000 7fb42b400640 1 --2- 172.21.15.96:0/3960673830 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb4340b41c0 0x7fb43415a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:22.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.688+0000 7fb433e00640 1 --2- 172.21.15.96:0/3960673830 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb43406f1a0 0x7fb434168d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:22.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.688+0000 7fb438c00640 1 -- 172.21.15.96:0/3960673830 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb43406f1a0 msgr2=0x7fb434168d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:22.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.688+0000 7fb438c00640 1 --2- 172.21.15.96:0/3960673830 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb43406f1a0 0x7fb434168d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:22.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.688+0000 7fb438c00640 1 -- 172.21.15.96:0/3960673830 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb4340b41c0 msgr2=0x7fb43415a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:22.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.688+0000 7fb438c00640 1 --2- 172.21.15.96:0/3960673830 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb4340b41c0 0x7fb43415a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:22.686 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.688+0000 7fb438c00640 1 -- 172.21.15.96:0/3960673830 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb434147430 con 0x7fb43415ae10 2024-05-28T06:22:22.686 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.688+0000 7fb433e00640 1 --2- 172.21.15.96:0/3960673830 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb43406f1a0 0x7fb434168d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:22:22.686 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.689+0000 7fb438c00640 1 --2- 172.21.15.96:0/3960673830 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb43415ae10 0x7fb434146ef0 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7fb42c00c7a0 tx=0x7fb42c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:22.686 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.689+0000 7fb430c00640 1 -- 172.21.15.96:0/3960673830 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb42c003b70 con 0x7fb43415ae10 2024-05-28T06:22:22.686 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.689+0000 7fb430c00640 1 -- 172.21.15.96:0/3960673830 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb42c003d30 con 0x7fb43415ae10 2024-05-28T06:22:22.686 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.689+0000 7fb430c00640 1 -- 172.21.15.96:0/3960673830 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb42c00aa90 con 0x7fb43415ae10 2024-05-28T06:22:22.686 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.689+0000 7fb43ac00640 1 -- 172.21.15.96:0/3960673830 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb434167f40 con 0x7fb43415ae10 2024-05-28T06:22:22.686 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.689+0000 7fb43ac00640 1 -- 172.21.15.96:0/3960673830 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb434168500 con 0x7fb43415ae10 2024-05-28T06:22:22.687 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.690+0000 7fb43ac00640 1 -- 172.21.15.96:0/3960673830 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4340689e0 con 0x7fb43415ae10 2024-05-28T06:22:22.687 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.691+0000 7fb430c00640 1 -- 172.21.15.96:0/3960673830 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 37) ==== 93182+0+0 (secure 0 0 0) 0x7fb42c00ac30 con 0x7fb43415ae10 2024-05-28T06:22:22.688 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.692+0000 7fb430c00640 1 --2- 172.21.15.96:0/3960673830 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb41006e090 0x7fb410070550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:22.688 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.692+0000 7fb430c00640 1 -- 172.21.15.96:0/3960673830 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 5765+0+0 (secure 0 0 0) 0x7fb42c08a860 con 0x7fb43415ae10 2024-05-28T06:22:22.688 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.693+0000 7fb433e00640 1 --2- 172.21.15.96:0/3960673830 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb41006e090 0x7fb410070550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:22.689 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.694+0000 7fb433e00640 1 --2- 172.21.15.96:0/3960673830 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb41006e090 0x7fb410070550 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fb434072630 tx=0x7fb424008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:22.691 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.696+0000 7fb430c00640 1 -- 172.21.15.96:0/3960673830 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb42c05a790 con 0x7fb43415ae10 2024-05-28T06:22:22.944 INFO:tasks.ceph.mds.b.smithi123.stderr:2024-05-28T06:22:22.949+0000 7ff4b8c00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-28T06:22:22.973 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.978+0000 7fb43ac00640 1 -- 172.21.15.96:0/3960673830 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7fb434072db0 con 0x7fb43415ae10 2024-05-28T06:22:22.974 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.979+0000 7fb430c00640 1 -- 172.21.15.96:0/3960673830 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1016 (secure 0 0 0) 0x7fb42c05a090 con 0x7fb43415ae10 2024-05-28T06:22:22.974 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:22:22.974 INFO:teuthology.orchestra.run.smithi096.stdout:{"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1116,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":115,"num_osds":8,"num_up_osds":8,"osd_up_since":1716876238,"num_in_osds":8,"osd_in_since":1716876229,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":24,"data_bytes":592778,"bytes_used":917270528,"bytes_avail":772176842752,"bytes_total":773094113280,"write_bytes_sec":2388,"read_op_per_sec":0,"write_op_per_sec":7},"fsmap":{"epoch":72,"id":8,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":8,"rank":0,"name":"b","status":"up:active","gid":9270}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":20,"modified":"2024-05-28T06:21:25.943324+0000","services":{}},"progress_events":{}} 2024-05-28T06:22:22.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.981+0000 7fb43ac00640 1 -- 172.21.15.96:0/3960673830 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb41006e090 msgr2=0x7fb410070550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:22.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.981+0000 7fb43ac00640 1 --2- 172.21.15.96:0/3960673830 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb41006e090 0x7fb410070550 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fb434072630 tx=0x7fb424008040 comp rx=0 tx=0).stop 2024-05-28T06:22:22.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.981+0000 7fb43ac00640 1 -- 172.21.15.96:0/3960673830 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb43415ae10 msgr2=0x7fb434146ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:22.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.981+0000 7fb43ac00640 1 --2- 172.21.15.96:0/3960673830 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb43415ae10 0x7fb434146ef0 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7fb42c00c7a0 tx=0x7fb42c00cc60 comp rx=0 tx=0).stop 2024-05-28T06:22:22.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.981+0000 7fb43ac00640 1 -- 172.21.15.96:0/3960673830 shutdown_connections 2024-05-28T06:22:22.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.981+0000 7fb43ac00640 1 --2- 172.21.15.96:0/3960673830 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb41006e090 0x7fb410070550 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:22.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.981+0000 7fb43ac00640 1 --2- 172.21.15.96:0/3960673830 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb43415ae10 0x7fb434146ef0 unknown :-1 s=CLOSED pgs=1818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:22.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.981+0000 7fb43ac00640 1 --2- 172.21.15.96:0/3960673830 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb4340b41c0 0x7fb43415a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:22.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.981+0000 7fb43ac00640 1 --2- 172.21.15.96:0/3960673830 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb43406f1a0 0x7fb434168d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:22.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.981+0000 7fb43ac00640 1 -- 172.21.15.96:0/3960673830 >> 172.21.15.96:0/3960673830 conn(0x7fb434067ce0 msgr2=0x7fb4340b3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:22.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.981+0000 7fb43ac00640 1 -- 172.21.15.96:0/3960673830 shutdown_connections 2024-05-28T06:22:22.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:22.981+0000 7fb43ac00640 1 -- 172.21.15.96:0/3960673830 wait complete. 2024-05-28T06:22:22.996 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-05-28T06:22:22.996 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-28T06:22:23.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:23.192+0000 7f302ac00640 1 Processor -- start 2024-05-28T06:22:23.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:23.192+0000 7f302ac00640 1 -- start start 2024-05-28T06:22:23.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:23.194+0000 7f302ac00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f30240b4e10 0x7f30240cde30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:23.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:23.194+0000 7f302ac00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f30240ce370 0x7f30240d0590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:23.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:23.194+0000 7f302ac00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f30240b51f0 con 0x7f30240d0ad0 2024-05-28T06:22:23.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:23.194+0000 7f302ac00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f30240d0f20 con 0x7f30240b4e10 2024-05-28T06:22:23.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:23.194+0000 7f302ac00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f30240aef60 con 0x7f30240ce370 2024-05-28T06:22:23.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:23.194+0000 7f3023e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f30240b4e10 0x7f30240cde30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:23.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:23.194+0000 7f3023e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f30240b4e10 0x7f30240cde30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:60250/0 (socket says 172.21.15.96:60250) 2024-05-28T06:22:23.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:23.194+0000 7f3023e00640 1 -- 172.21.15.96:0/553993868 learned_addr learned my addr 172.21.15.96:0/553993868 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:22:23.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:23.195+0000 7f3023400640 1 --2- 172.21.15.96:0/553993868 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f30240ce370 0x7f30240d0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:23.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:23.195+0000 7f3023e00640 1 -- 172.21.15.96:0/553993868 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f30240ce370 msgr2=0x7f30240d0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:23.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:23.195+0000 7f3023e00640 1 --2- 172.21.15.96:0/553993868 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f30240ce370 0x7f30240d0590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:23.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:23.195+0000 7f3023e00640 1 -- 172.21.15.96:0/553993868 >> v1:172.21.15.96:6789/0 conn(0x7f30240d0ad0 legacy=0x7f3024099090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:23.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:23.195+0000 7f3023e00640 1 -- 172.21.15.96:0/553993868 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30240ad1d0 con 0x7f30240b4e10 2024-05-28T06:22:23.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:23.195+0000 7f3022a00640 1 -- 172.21.15.96:0/553993868 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3070655595 0 0) 0x7f30240b51f0 con 0x7f30240d0ad0 2024-05-28T06:22:23.192 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:23.195+0000 7f3023400640 1 --2- 172.21.15.96:0/553993868 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f30240ce370 0x7f30240d0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:22:23.192 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:23.196+0000 7f3023e00640 1 --2- 172.21.15.96:0/553993868 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f30240b4e10 0x7f30240cde30 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7f3014009870 tx=0x7f301402a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef12340febeb03fd server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:23.193 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:23.196+0000 7f3022a00640 1 -- 172.21.15.96:0/553993868 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3014037070 con 0x7f30240b4e10 2024-05-28T06:22:23.193 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:23.196+0000 7f3022a00640 1 -- 172.21.15.96:0/553993868 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3014032440 con 0x7f30240b4e10 2024-05-28T06:22:23.193 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:23.196+0000 7f3022a00640 1 -- 172.21.15.96:0/553993868 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3014032810 con 0x7f30240b4e10 2024-05-28T06:22:23.193 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:23.197+0000 7f302ac00640 1 -- 172.21.15.96:0/553993868 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f30240b4e10 msgr2=0x7f30240cde30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:23.193 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:23.197+0000 7f302ac00640 1 --2- 172.21.15.96:0/553993868 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f30240b4e10 0x7f30240cde30 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7f3014009870 tx=0x7f301402a1c0 comp rx=0 tx=0).stop 2024-05-28T06:22:23.193 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:23.197+0000 7f302ac00640 1 -- 172.21.15.96:0/553993868 shutdown_connections 2024-05-28T06:22:23.193 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:23.197+0000 7f302ac00640 1 --2- 172.21.15.96:0/553993868 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f30240ce370 0x7f30240d0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:23.193 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:23.197+0000 7f302ac00640 1 --2- 172.21.15.96:0/553993868 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f30240b4e10 0x7f30240cde30 unknown :-1 s=CLOSED pgs=1820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:23.193 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:23.197+0000 7f302ac00640 1 -- 172.21.15.96:0/553993868 >> 172.21.15.96:0/553993868 conn(0x7f302406bae0 msgr2=0x7f302406bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:23.193 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:23.197+0000 7f302ac00640 1 -- 172.21.15.96:0/553993868 shutdown_connections 2024-05-28T06:22:23.193 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:23.197+0000 7f302ac00640 1 -- 172.21.15.96:0/553993868 wait complete. 2024-05-28T06:22:23.194 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:23.199+0000 7f302ac00640 1 Processor -- start 2024-05-28T06:22:23.194 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:23.199+0000 7f302ac00640 1 -- start start 2024-05-28T06:22:23.195 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:23.200+0000 7f302ac00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f30240b4e10 0x7f302415e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:23.195 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:23.200+0000 7f302ac00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f30240ce370 0x7f30240b11c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:23.195 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:23.200+0000 7f302ac00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f30240d0ad0 0x7f30240b1700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:23.195 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:23.200+0000 7f302ac00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f30240ce880 con 0x7f30240d0ad0 2024-05-28T06:22:23.196 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:23.200+0000 7f302ac00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f30240ad750 con 0x7f30240ce370 2024-05-28T06:22:23.196 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:23.200+0000 7f302ac00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f30240cea00 con 0x7f30240b4e10 2024-05-28T06:22:23.196 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:23.200+0000 7f3028c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f30240d0ad0 0x7f30240b1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:23.196 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:23.200+0000 7f3028c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f30240d0ad0 0x7f30240b1700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:60258/0 (socket says 172.21.15.96:60258) 2024-05-28T06:22:23.196 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:23.200+0000 7f3028c00640 1 -- 172.21.15.96:0/1142692083 learned_addr learned my addr 172.21.15.96:0/1142692083 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:22:23.197 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:23.200+0000 7f3023e00640 1 --2- 172.21.15.96:0/1142692083 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f30240b4e10 0x7f302415e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:23.197 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:23.200+0000 7f3023400640 1 --2- 172.21.15.96:0/1142692083 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f30240ce370 0x7f30240b11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:23.198 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:23.200+0000 7f3028c00640 1 -- 172.21.15.96:0/1142692083 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f30240b4e10 msgr2=0x7f302415e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:23.198 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:23.200+0000 7f3028c00640 1 --2- 172.21.15.96:0/1142692083 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f30240b4e10 0x7f302415e1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:23.198 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:23.200+0000 7f3028c00640 1 -- 172.21.15.96:0/1142692083 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f30240ce370 msgr2=0x7f30240b11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:23.198 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:23.200+0000 7f3028c00640 1 --2- 172.21.15.96:0/1142692083 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f30240ce370 0x7f30240b11c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:23.198 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:23.200+0000 7f3028c00640 1 -- 172.21.15.96:0/1142692083 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30240b39b0 con 0x7f30240d0ad0 2024-05-28T06:22:23.198 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:23.201+0000 7f3028c00640 1 --2- 172.21.15.96:0/1142692083 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f30240d0ad0 0x7f30240b1700 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7f301c00c810 tx=0x7f301c00ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:23.198 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:23.201+0000 7f301be00640 1 -- 172.21.15.96:0/1142692083 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f301c017070 con 0x7f30240d0ad0 2024-05-28T06:22:23.198 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:23.201+0000 7f301be00640 1 -- 172.21.15.96:0/1142692083 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f301c003dc0 con 0x7f30240d0ad0 2024-05-28T06:22:23.198 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:23.201+0000 7f301be00640 1 -- 172.21.15.96:0/1142692083 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f301c0052b0 con 0x7f30240d0ad0 2024-05-28T06:22:23.198 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:23.201+0000 7f302ac00640 1 -- 172.21.15.96:0/1142692083 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f302415d710 con 0x7f30240d0ad0 2024-05-28T06:22:23.198 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:23.201+0000 7f302ac00640 1 -- 172.21.15.96:0/1142692083 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f302415dcd0 con 0x7f30240d0ad0 2024-05-28T06:22:23.199 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:23.202+0000 7f302ac00640 1 -- 172.21.15.96:0/1142692083 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30240689e0 con 0x7f30240d0ad0 2024-05-28T06:22:23.199 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:23.203+0000 7f301be00640 1 -- 172.21.15.96:0/1142692083 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 37) ==== 93182+0+0 (secure 0 0 0) 0x7f301c01a070 con 0x7f30240d0ad0 2024-05-28T06:22:23.200 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:23.205+0000 7f301be00640 1 --2- 172.21.15.96:0/1142692083 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f300006e0c0 0x7f3000070580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:23.200 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:23.205+0000 7f301be00640 1 -- 172.21.15.96:0/1142692083 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 5765+0+0 (secure 0 0 0) 0x7f301c089fa0 con 0x7f30240d0ad0 2024-05-28T06:22:23.201 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:23.206+0000 7f3023e00640 1 --2- 172.21.15.96:0/1142692083 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f300006e0c0 0x7f3000070580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:23.202 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:23.207+0000 7f3023e00640 1 --2- 172.21.15.96:0/1142692083 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f300006e0c0 0x7f3000070580 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f3014034000 tx=0x7f301402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:23.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:23.209+0000 7f301be00640 1 -- 172.21.15.96:0/1142692083 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f301c059e30 con 0x7f30240d0ad0 2024-05-28T06:22:23.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:23.396+0000 7f302ac00640 1 -- 172.21.15.96:0/1142692083 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f30240b3c90 con 0x7f300006e0c0 2024-05-28T06:22:23.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:23.415+0000 7f301be00640 1 -- 172.21.15.96:0/1142692083 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252497 (secure 0 0 0) 0x7f30240b3c90 con 0x7f300006e0c0 2024-05-28T06:22:23.413 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:22:23.415 INFO:teuthology.orchestra.run.smithi096.stderr:dumped all 2024-05-28T06:22:23.417 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:23.421+0000 7f302ac00640 1 -- 172.21.15.96:0/1142692083 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f300006e0c0 msgr2=0x7f3000070580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:23.417 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:23.421+0000 7f302ac00640 1 --2- 172.21.15.96:0/1142692083 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f300006e0c0 0x7f3000070580 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f3014034000 tx=0x7f301402e000 comp rx=0 tx=0).stop 2024-05-28T06:22:23.417 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:23.421+0000 7f302ac00640 1 -- 172.21.15.96:0/1142692083 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f30240d0ad0 msgr2=0x7f30240b1700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:23.417 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:23.421+0000 7f302ac00640 1 --2- 172.21.15.96:0/1142692083 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f30240d0ad0 0x7f30240b1700 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7f301c00c810 tx=0x7f301c00ccd0 comp rx=0 tx=0).stop 2024-05-28T06:22:23.417 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:23.421+0000 7f302ac00640 1 -- 172.21.15.96:0/1142692083 shutdown_connections 2024-05-28T06:22:23.417 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:23.421+0000 7f302ac00640 1 --2- 172.21.15.96:0/1142692083 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f300006e0c0 0x7f3000070580 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:23.417 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:23.421+0000 7f302ac00640 1 --2- 172.21.15.96:0/1142692083 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f30240d0ad0 0x7f30240b1700 unknown :-1 s=CLOSED pgs=1821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:23.417 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:23.421+0000 7f302ac00640 1 --2- 172.21.15.96:0/1142692083 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f30240ce370 0x7f30240b11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:23.417 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:23.421+0000 7f302ac00640 1 --2- 172.21.15.96:0/1142692083 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f30240b4e10 0x7f302415e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:23.418 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:23.421+0000 7f302ac00640 1 -- 172.21.15.96:0/1142692083 >> 172.21.15.96:0/1142692083 conn(0x7f302406bae0 msgr2=0x7f3024071b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:23.418 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:23.421+0000 7f302ac00640 1 -- 172.21.15.96:0/1142692083 shutdown_connections 2024-05-28T06:22:23.418 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:23.421+0000 7f302ac00640 1 -- 172.21.15.96:0/1142692083 wait complete. 2024-05-28T06:22:23.435 INFO:teuthology.orchestra.run.smithi096.stdout:{"pg_ready":true,"pg_map":{"version":659,"stamp":"2024-05-28T06:22:21.965007+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":165,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":895772,"kb_used_data":12392,"kb_used_omap":12,"kb_used_meta":883251,"kb_avail":754078948,"statfs":{"total":773094113280,"available":772176842752,"internally_reserved":0,"allocated":12689408,"data_stored":6195724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":904449584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002310"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.242686+0000","last_change":"2024-05-28T06:22:17.004588+0000","last_active":"2024-05-28T06:22:18.242686+0000","last_peered":"2024-05-28T06:22:18.242686+0000","last_clean":"2024-05-28T06:22:18.242686+0000","last_became_active":"2024-05-28T06:22:17.003523+0000","last_became_peered":"2024-05-28T06:22:17.003523+0000","last_unstale":"2024-05-28T06:22:18.242686+0000","last_undegraded":"2024-05-28T06:22:18.242686+0000","last_fullsized":"2024-05-28T06:22:18.242686+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:04:35.674926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.001575+0000","last_change":"2024-05-28T06:22:16.997146+0000","last_active":"2024-05-28T06:22:18.001575+0000","last_peered":"2024-05-28T06:22:18.001575+0000","last_clean":"2024-05-28T06:22:18.001575+0000","last_became_active":"2024-05-28T06:22:16.995737+0000","last_became_peered":"2024-05-28T06:22:16.995737+0000","last_unstale":"2024-05-28T06:22:18.001575+0000","last_undegraded":"2024-05-28T06:22:18.001575+0000","last_fullsized":"2024-05-28T06:22:18.001575+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:38:34.851302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.528142+0000","last_change":"2024-05-28T06:22:17.006833+0000","last_active":"2024-05-28T06:22:18.528142+0000","last_peered":"2024-05-28T06:22:18.528142+0000","last_clean":"2024-05-28T06:22:18.528142+0000","last_became_active":"2024-05-28T06:22:17.006501+0000","last_became_peered":"2024-05-28T06:22:17.006501+0000","last_unstale":"2024-05-28T06:22:18.528142+0000","last_undegraded":"2024-05-28T06:22:18.528142+0000","last_fullsized":"2024-05-28T06:22:18.528142+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:23:42.240172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.242843+0000","last_change":"2024-05-28T06:22:16.994809+0000","last_active":"2024-05-28T06:22:18.242843+0000","last_peered":"2024-05-28T06:22:18.242843+0000","last_clean":"2024-05-28T06:22:18.242843+0000","last_became_active":"2024-05-28T06:22:16.994548+0000","last_became_peered":"2024-05-28T06:22:16.994548+0000","last_unstale":"2024-05-28T06:22:18.242843+0000","last_undegraded":"2024-05-28T06:22:18.242843+0000","last_fullsized":"2024-05-28T06:22:18.242843+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:59:36.297627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:17.999386+0000","last_change":"2024-05-28T06:22:16.996834+0000","last_active":"2024-05-28T06:22:17.999386+0000","last_peered":"2024-05-28T06:22:17.999386+0000","last_clean":"2024-05-28T06:22:17.999386+0000","last_became_active":"2024-05-28T06:22:16.996543+0000","last_became_peered":"2024-05-28T06:22:16.996543+0000","last_unstale":"2024-05-28T06:22:17.999386+0000","last_undegraded":"2024-05-28T06:22:17.999386+0000","last_fullsized":"2024-05-28T06:22:17.999386+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:23:11.730784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.528701+0000","last_change":"2024-05-28T06:22:16.995996+0000","last_active":"2024-05-28T06:22:18.528701+0000","last_peered":"2024-05-28T06:22:18.528701+0000","last_clean":"2024-05-28T06:22:18.528701+0000","last_became_active":"2024-05-28T06:22:16.995714+0000","last_became_peered":"2024-05-28T06:22:16.995714+0000","last_unstale":"2024-05-28T06:22:18.528701+0000","last_undegraded":"2024-05-28T06:22:18.528701+0000","last_fullsized":"2024-05-28T06:22:18.528701+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:04:42.043002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.024311+0000","last_change":"2024-05-28T06:22:17.006351+0000","last_active":"2024-05-28T06:22:18.024311+0000","last_peered":"2024-05-28T06:22:18.024311+0000","last_clean":"2024-05-28T06:22:18.024311+0000","last_became_active":"2024-05-28T06:22:17.005644+0000","last_became_peered":"2024-05-28T06:22:17.005644+0000","last_unstale":"2024-05-28T06:22:18.024311+0000","last_undegraded":"2024-05-28T06:22:18.024311+0000","last_fullsized":"2024-05-28T06:22:18.024311+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:21:19.908131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.527537+0000","last_change":"2024-05-28T06:22:16.998140+0000","last_active":"2024-05-28T06:22:18.527537+0000","last_peered":"2024-05-28T06:22:18.527537+0000","last_clean":"2024-05-28T06:22:18.527537+0000","last_became_active":"2024-05-28T06:22:16.997796+0000","last_became_peered":"2024-05-28T06:22:16.997796+0000","last_unstale":"2024-05-28T06:22:18.527537+0000","last_undegraded":"2024-05-28T06:22:18.527537+0000","last_fullsized":"2024-05-28T06:22:18.527537+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:06:13.719079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.242476+0000","last_change":"2024-05-28T06:22:17.003545+0000","last_active":"2024-05-28T06:22:18.242476+0000","last_peered":"2024-05-28T06:22:18.242476+0000","last_clean":"2024-05-28T06:22:18.242476+0000","last_became_active":"2024-05-28T06:22:17.001251+0000","last_became_peered":"2024-05-28T06:22:17.001251+0000","last_unstale":"2024-05-28T06:22:18.242476+0000","last_undegraded":"2024-05-28T06:22:18.242476+0000","last_fullsized":"2024-05-28T06:22:18.242476+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:38:50.761814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.243297+0000","last_change":"2024-05-28T06:22:17.001134+0000","last_active":"2024-05-28T06:22:18.243297+0000","last_peered":"2024-05-28T06:22:18.243297+0000","last_clean":"2024-05-28T06:22:18.243297+0000","last_became_active":"2024-05-28T06:22:17.000345+0000","last_became_peered":"2024-05-28T06:22:17.000345+0000","last_unstale":"2024-05-28T06:22:18.243297+0000","last_undegraded":"2024-05-28T06:22:18.243297+0000","last_fullsized":"2024-05-28T06:22:18.243297+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:22:46.981586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.001735+0000","last_change":"2024-05-28T06:22:17.002601+0000","last_active":"2024-05-28T06:22:18.001735+0000","last_peered":"2024-05-28T06:22:18.001735+0000","last_clean":"2024-05-28T06:22:18.001735+0000","last_became_active":"2024-05-28T06:22:17.002340+0000","last_became_peered":"2024-05-28T06:22:17.002340+0000","last_unstale":"2024-05-28T06:22:18.001735+0000","last_undegraded":"2024-05-28T06:22:18.001735+0000","last_fullsized":"2024-05-28T06:22:18.001735+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:26:18.638055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.243231+0000","last_change":"2024-05-28T06:22:16.994222+0000","last_active":"2024-05-28T06:22:18.243231+0000","last_peered":"2024-05-28T06:22:18.243231+0000","last_clean":"2024-05-28T06:22:18.243231+0000","last_became_active":"2024-05-28T06:22:16.993674+0000","last_became_peered":"2024-05-28T06:22:16.993674+0000","last_unstale":"2024-05-28T06:22:18.243231+0000","last_undegraded":"2024-05-28T06:22:18.243231+0000","last_fullsized":"2024-05-28T06:22:18.243231+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:40:55.388301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.528122+0000","last_change":"2024-05-28T06:22:16.998228+0000","last_active":"2024-05-28T06:22:18.528122+0000","last_peered":"2024-05-28T06:22:18.528122+0000","last_clean":"2024-05-28T06:22:18.528122+0000","last_became_active":"2024-05-28T06:22:16.996238+0000","last_became_peered":"2024-05-28T06:22:16.996238+0000","last_unstale":"2024-05-28T06:22:18.528122+0000","last_undegraded":"2024-05-28T06:22:18.528122+0000","last_fullsized":"2024-05-28T06:22:18.528122+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:38:34.851302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.527501+0000","last_change":"2024-05-28T06:22:17.001665+0000","last_active":"2024-05-28T06:22:18.527501+0000","last_peered":"2024-05-28T06:22:18.527501+0000","last_clean":"2024-05-28T06:22:18.527501+0000","last_became_active":"2024-05-28T06:22:17.001304+0000","last_became_peered":"2024-05-28T06:22:17.001304+0000","last_unstale":"2024-05-28T06:22:18.527501+0000","last_undegraded":"2024-05-28T06:22:18.527501+0000","last_fullsized":"2024-05-28T06:22:18.527501+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:47:18.486697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.527922+0000","last_change":"2024-05-28T06:22:16.997664+0000","last_active":"2024-05-28T06:22:18.527922+0000","last_peered":"2024-05-28T06:22:18.527922+0000","last_clean":"2024-05-28T06:22:18.527922+0000","last_became_active":"2024-05-28T06:22:16.997387+0000","last_became_peered":"2024-05-28T06:22:16.997387+0000","last_unstale":"2024-05-28T06:22:18.527922+0000","last_undegraded":"2024-05-28T06:22:18.527922+0000","last_fullsized":"2024-05-28T06:22:18.527922+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:37:05.453277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.242046+0000","last_change":"2024-05-28T06:22:17.005071+0000","last_active":"2024-05-28T06:22:18.242046+0000","last_peered":"2024-05-28T06:22:18.242046+0000","last_clean":"2024-05-28T06:22:18.242046+0000","last_became_active":"2024-05-28T06:22:17.004078+0000","last_became_peered":"2024-05-28T06:22:17.004078+0000","last_unstale":"2024-05-28T06:22:18.242046+0000","last_undegraded":"2024-05-28T06:22:18.242046+0000","last_fullsized":"2024-05-28T06:22:18.242046+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:26:19.869612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.001928+0000","last_change":"2024-05-28T06:22:17.010577+0000","last_active":"2024-05-28T06:22:18.001928+0000","last_peered":"2024-05-28T06:22:18.001928+0000","last_clean":"2024-05-28T06:22:18.001928+0000","last_became_active":"2024-05-28T06:22:17.010281+0000","last_became_peered":"2024-05-28T06:22:17.010281+0000","last_unstale":"2024-05-28T06:22:18.001928+0000","last_undegraded":"2024-05-28T06:22:18.001928+0000","last_fullsized":"2024-05-28T06:22:18.001928+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:23:42.240172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.020188+0000","last_change":"2024-05-28T06:22:17.011422+0000","last_active":"2024-05-28T06:22:18.020188+0000","last_peered":"2024-05-28T06:22:18.020188+0000","last_clean":"2024-05-28T06:22:18.020188+0000","last_became_active":"2024-05-28T06:22:17.010224+0000","last_became_peered":"2024-05-28T06:22:17.010224+0000","last_unstale":"2024-05-28T06:22:18.020188+0000","last_undegraded":"2024-05-28T06:22:18.020188+0000","last_fullsized":"2024-05-28T06:22:18.020188+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:11:13.465001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.528617+0000","last_change":"2024-05-28T06:22:17.002306+0000","last_active":"2024-05-28T06:22:18.528617+0000","last_peered":"2024-05-28T06:22:18.528617+0000","last_clean":"2024-05-28T06:22:18.528617+0000","last_became_active":"2024-05-28T06:22:17.001965+0000","last_became_peered":"2024-05-28T06:22:17.001965+0000","last_unstale":"2024-05-28T06:22:18.528617+0000","last_undegraded":"2024-05-28T06:22:18.528617+0000","last_fullsized":"2024-05-28T06:22:18.528617+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:16:11.794755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.000927+0000","last_change":"2024-05-28T06:22:16.998168+0000","last_active":"2024-05-28T06:22:18.000927+0000","last_peered":"2024-05-28T06:22:18.000927+0000","last_clean":"2024-05-28T06:22:18.000927+0000","last_became_active":"2024-05-28T06:22:16.996215+0000","last_became_peered":"2024-05-28T06:22:16.996215+0000","last_unstale":"2024-05-28T06:22:18.000927+0000","last_undegraded":"2024-05-28T06:22:18.000927+0000","last_fullsized":"2024-05-28T06:22:18.000927+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:06:13.719079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.001186+0000","last_change":"2024-05-28T06:22:16.998348+0000","last_active":"2024-05-28T06:22:18.001186+0000","last_peered":"2024-05-28T06:22:18.001186+0000","last_clean":"2024-05-28T06:22:18.001186+0000","last_became_active":"2024-05-28T06:22:16.998024+0000","last_became_peered":"2024-05-28T06:22:16.998024+0000","last_unstale":"2024-05-28T06:22:18.001186+0000","last_undegraded":"2024-05-28T06:22:18.001186+0000","last_fullsized":"2024-05-28T06:22:18.001186+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:13:14.889721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.528790+0000","last_change":"2024-05-28T06:22:17.004708+0000","last_active":"2024-05-28T06:22:18.528790+0000","last_peered":"2024-05-28T06:22:18.528790+0000","last_clean":"2024-05-28T06:22:18.528790+0000","last_became_active":"2024-05-28T06:22:17.003676+0000","last_became_peered":"2024-05-28T06:22:17.003676+0000","last_unstale":"2024-05-28T06:22:18.528790+0000","last_undegraded":"2024-05-28T06:22:18.528790+0000","last_fullsized":"2024-05-28T06:22:18.528790+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:38:34.851302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.000139+0000","last_change":"2024-05-28T06:22:16.999079+0000","last_active":"2024-05-28T06:22:18.000139+0000","last_peered":"2024-05-28T06:22:18.000139+0000","last_clean":"2024-05-28T06:22:18.000139+0000","last_became_active":"2024-05-28T06:22:16.998795+0000","last_became_peered":"2024-05-28T06:22:16.998795+0000","last_unstale":"2024-05-28T06:22:18.000139+0000","last_undegraded":"2024-05-28T06:22:18.000139+0000","last_fullsized":"2024-05-28T06:22:18.000139+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:17:14.655934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.526910+0000","last_change":"2024-05-28T06:22:17.008315+0000","last_active":"2024-05-28T06:22:18.526910+0000","last_peered":"2024-05-28T06:22:18.526910+0000","last_clean":"2024-05-28T06:22:18.526910+0000","last_became_active":"2024-05-28T06:22:17.007995+0000","last_became_peered":"2024-05-28T06:22:17.007995+0000","last_unstale":"2024-05-28T06:22:18.526910+0000","last_undegraded":"2024-05-28T06:22:18.526910+0000","last_fullsized":"2024-05-28T06:22:18.526910+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T18:02:32.237669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.002108+0000","last_change":"2024-05-28T06:22:17.003291+0000","last_active":"2024-05-28T06:22:18.002108+0000","last_peered":"2024-05-28T06:22:18.002108+0000","last_clean":"2024-05-28T06:22:18.002108+0000","last_became_active":"2024-05-28T06:22:17.003018+0000","last_became_peered":"2024-05-28T06:22:17.003018+0000","last_unstale":"2024-05-28T06:22:18.002108+0000","last_undegraded":"2024-05-28T06:22:18.002108+0000","last_fullsized":"2024-05-28T06:22:18.002108+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:16:11.794755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.243265+0000","last_change":"2024-05-28T06:22:17.006855+0000","last_active":"2024-05-28T06:22:18.243265+0000","last_peered":"2024-05-28T06:22:18.243265+0000","last_clean":"2024-05-28T06:22:18.243265+0000","last_became_active":"2024-05-28T06:22:17.006649+0000","last_became_peered":"2024-05-28T06:22:17.006649+0000","last_unstale":"2024-05-28T06:22:18.243265+0000","last_undegraded":"2024-05-28T06:22:18.243265+0000","last_fullsized":"2024-05-28T06:22:18.243265+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:25:34.876194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.529987+0000","last_change":"2024-05-28T06:22:17.000165+0000","last_active":"2024-05-28T06:22:18.529987+0000","last_peered":"2024-05-28T06:22:18.529987+0000","last_clean":"2024-05-28T06:22:18.529987+0000","last_became_active":"2024-05-28T06:22:16.999904+0000","last_became_peered":"2024-05-28T06:22:16.999904+0000","last_unstale":"2024-05-28T06:22:18.529987+0000","last_undegraded":"2024-05-28T06:22:18.529987+0000","last_fullsized":"2024-05-28T06:22:18.529987+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:58:15.396668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.529632+0000","last_change":"2024-05-28T06:22:16.993958+0000","last_active":"2024-05-28T06:22:18.529632+0000","last_peered":"2024-05-28T06:22:18.529632+0000","last_clean":"2024-05-28T06:22:18.529632+0000","last_became_active":"2024-05-28T06:22:16.993327+0000","last_became_peered":"2024-05-28T06:22:16.993327+0000","last_unstale":"2024-05-28T06:22:18.529632+0000","last_undegraded":"2024-05-28T06:22:18.529632+0000","last_fullsized":"2024-05-28T06:22:18.529632+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:26:18.638055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.000269+0000","last_change":"2024-05-28T06:22:17.001399+0000","last_active":"2024-05-28T06:22:18.000269+0000","last_peered":"2024-05-28T06:22:18.000269+0000","last_clean":"2024-05-28T06:22:18.000269+0000","last_became_active":"2024-05-28T06:22:17.000579+0000","last_became_peered":"2024-05-28T06:22:17.000579+0000","last_unstale":"2024-05-28T06:22:18.000269+0000","last_undegraded":"2024-05-28T06:22:18.000269+0000","last_fullsized":"2024-05-28T06:22:18.000269+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:48:14.529168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.024533+0000","last_change":"2024-05-28T06:22:17.004330+0000","last_active":"2024-05-28T06:22:18.024533+0000","last_peered":"2024-05-28T06:22:18.024533+0000","last_clean":"2024-05-28T06:22:18.024533+0000","last_became_active":"2024-05-28T06:22:16.996955+0000","last_became_peered":"2024-05-28T06:22:16.996955+0000","last_unstale":"2024-05-28T06:22:18.024533+0000","last_undegraded":"2024-05-28T06:22:18.024533+0000","last_fullsized":"2024-05-28T06:22:18.024533+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:25:57.432011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.242168+0000","last_change":"2024-05-28T06:22:17.001908+0000","last_active":"2024-05-28T06:22:18.242168+0000","last_peered":"2024-05-28T06:22:18.242168+0000","last_clean":"2024-05-28T06:22:18.242168+0000","last_became_active":"2024-05-28T06:22:17.000543+0000","last_became_peered":"2024-05-28T06:22:17.000543+0000","last_unstale":"2024-05-28T06:22:18.242168+0000","last_undegraded":"2024-05-28T06:22:18.242168+0000","last_fullsized":"2024-05-28T06:22:18.242168+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:12:55.159821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.527587+0000","last_change":"2024-05-28T06:22:16.992715+0000","last_active":"2024-05-28T06:22:18.527587+0000","last_peered":"2024-05-28T06:22:18.527587+0000","last_clean":"2024-05-28T06:22:18.527587+0000","last_became_active":"2024-05-28T06:22:16.992094+0000","last_became_peered":"2024-05-28T06:22:16.992094+0000","last_unstale":"2024-05-28T06:22:18.527587+0000","last_undegraded":"2024-05-28T06:22:18.527587+0000","last_fullsized":"2024-05-28T06:22:18.527587+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:05:25.661249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.529840+0000","last_change":"2024-05-28T06:22:17.004255+0000","last_active":"2024-05-28T06:22:18.529840+0000","last_peered":"2024-05-28T06:22:18.529840+0000","last_clean":"2024-05-28T06:22:18.529840+0000","last_became_active":"2024-05-28T06:22:17.001691+0000","last_became_peered":"2024-05-28T06:22:17.001691+0000","last_unstale":"2024-05-28T06:22:18.529840+0000","last_undegraded":"2024-05-28T06:22:18.529840+0000","last_fullsized":"2024-05-28T06:22:18.529840+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:40:20.104894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"114'4","reported_seq":18,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.024524+0000","last_change":"2024-05-28T06:22:15.992751+0000","last_active":"2024-05-28T06:22:18.024524+0000","last_peered":"2024-05-28T06:22:18.024524+0000","last_clean":"2024-05-28T06:22:18.024524+0000","last_became_active":"2024-05-28T06:22:15.992113+0000","last_became_peered":"2024-05-28T06:22:15.992113+0000","last_unstale":"2024-05-28T06:22:18.024524+0000","last_undegraded":"2024-05-28T06:22:18.024524+0000","last_fullsized":"2024-05-28T06:22:18.024524+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:28:16.129726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.020384+0000","last_change":"2024-05-28T06:22:17.003592+0000","last_active":"2024-05-28T06:22:18.020384+0000","last_peered":"2024-05-28T06:22:18.020384+0000","last_clean":"2024-05-28T06:22:18.020384+0000","last_became_active":"2024-05-28T06:22:17.002423+0000","last_became_peered":"2024-05-28T06:22:17.002423+0000","last_unstale":"2024-05-28T06:22:18.020384+0000","last_undegraded":"2024-05-28T06:22:18.020384+0000","last_fullsized":"2024-05-28T06:22:18.020384+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:58:35.663956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"114'1","reported_seq":15,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.528549+0000","last_change":"2024-05-28T06:22:15.996401+0000","last_active":"2024-05-28T06:22:18.528549+0000","last_peered":"2024-05-28T06:22:18.528549+0000","last_clean":"2024-05-28T06:22:18.528549+0000","last_became_active":"2024-05-28T06:22:15.996039+0000","last_became_peered":"2024-05-28T06:22:15.996039+0000","last_unstale":"2024-05-28T06:22:18.528549+0000","last_undegraded":"2024-05-28T06:22:18.528549+0000","last_fullsized":"2024-05-28T06:22:18.528549+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:43:46.372846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.528998+0000","last_change":"2024-05-28T06:22:16.999053+0000","last_active":"2024-05-28T06:22:18.528998+0000","last_peered":"2024-05-28T06:22:18.528998+0000","last_clean":"2024-05-28T06:22:18.528998+0000","last_became_active":"2024-05-28T06:22:16.995589+0000","last_became_peered":"2024-05-28T06:22:16.995589+0000","last_unstale":"2024-05-28T06:22:18.528998+0000","last_undegraded":"2024-05-28T06:22:18.528998+0000","last_fullsized":"2024-05-28T06:22:18.528998+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:58:42.572818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"114'2","reported_seq":16,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:17.999806+0000","last_change":"2024-05-28T06:22:15.989244+0000","last_active":"2024-05-28T06:22:17.999806+0000","last_peered":"2024-05-28T06:22:17.999806+0000","last_clean":"2024-05-28T06:22:17.999806+0000","last_became_active":"2024-05-28T06:22:15.988137+0000","last_became_peered":"2024-05-28T06:22:15.988137+0000","last_unstale":"2024-05-28T06:22:17.999806+0000","last_undegraded":"2024-05-28T06:22:17.999806+0000","last_fullsized":"2024-05-28T06:22:17.999806+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T18:10:18.669151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.528637+0000","last_change":"2024-05-28T06:22:17.001702+0000","last_active":"2024-05-28T06:22:18.528637+0000","last_peered":"2024-05-28T06:22:18.528637+0000","last_clean":"2024-05-28T06:22:18.528637+0000","last_became_active":"2024-05-28T06:22:16.999756+0000","last_became_peered":"2024-05-28T06:22:16.999756+0000","last_unstale":"2024-05-28T06:22:18.528637+0000","last_undegraded":"2024-05-28T06:22:18.528637+0000","last_fullsized":"2024-05-28T06:22:18.528637+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:23:29.259681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"114'2","reported_seq":16,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:17.999369+0000","last_change":"2024-05-28T06:22:15.992414+0000","last_active":"2024-05-28T06:22:17.999369+0000","last_peered":"2024-05-28T06:22:17.999369+0000","last_clean":"2024-05-28T06:22:17.999369+0000","last_became_active":"2024-05-28T06:22:15.991608+0000","last_became_peered":"2024-05-28T06:22:15.991608+0000","last_unstale":"2024-05-28T06:22:17.999369+0000","last_undegraded":"2024-05-28T06:22:17.999369+0000","last_fullsized":"2024-05-28T06:22:17.999369+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:42:21.601868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.241943+0000","last_change":"2024-05-28T06:22:16.995582+0000","last_active":"2024-05-28T06:22:18.241943+0000","last_peered":"2024-05-28T06:22:18.241943+0000","last_clean":"2024-05-28T06:22:18.241943+0000","last_became_active":"2024-05-28T06:22:16.995242+0000","last_became_peered":"2024-05-28T06:22:16.995242+0000","last_unstale":"2024-05-28T06:22:18.241943+0000","last_undegraded":"2024-05-28T06:22:18.241943+0000","last_fullsized":"2024-05-28T06:22:18.241943+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:46:21.232752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"114'1","reported_seq":15,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.024104+0000","last_change":"2024-05-28T06:22:15.994783+0000","last_active":"2024-05-28T06:22:18.024104+0000","last_peered":"2024-05-28T06:22:18.024104+0000","last_clean":"2024-05-28T06:22:18.024104+0000","last_became_active":"2024-05-28T06:22:15.994208+0000","last_became_peered":"2024-05-28T06:22:15.994208+0000","last_unstale":"2024-05-28T06:22:18.024104+0000","last_undegraded":"2024-05-28T06:22:18.024104+0000","last_fullsized":"2024-05-28T06:22:18.024104+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:04:41.031193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.000748+0000","last_change":"2024-05-28T06:22:16.992756+0000","last_active":"2024-05-28T06:22:18.000748+0000","last_peered":"2024-05-28T06:22:18.000748+0000","last_clean":"2024-05-28T06:22:18.000748+0000","last_became_active":"2024-05-28T06:22:16.992331+0000","last_became_peered":"2024-05-28T06:22:16.992331+0000","last_unstale":"2024-05-28T06:22:18.000748+0000","last_undegraded":"2024-05-28T06:22:18.000748+0000","last_fullsized":"2024-05-28T06:22:18.000748+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:08:39.984746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"114'1","reported_seq":15,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.529094+0000","last_change":"2024-05-28T06:22:15.990828+0000","last_active":"2024-05-28T06:22:18.529094+0000","last_peered":"2024-05-28T06:22:18.529094+0000","last_clean":"2024-05-28T06:22:18.529094+0000","last_became_active":"2024-05-28T06:22:15.990537+0000","last_became_peered":"2024-05-28T06:22:15.990537+0000","last_unstale":"2024-05-28T06:22:18.529094+0000","last_undegraded":"2024-05-28T06:22:18.529094+0000","last_fullsized":"2024-05-28T06:22:18.529094+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:51:43.927844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.241796+0000","last_change":"2024-05-28T06:22:17.000804+0000","last_active":"2024-05-28T06:22:18.241796+0000","last_peered":"2024-05-28T06:22:18.241796+0000","last_clean":"2024-05-28T06:22:18.241796+0000","last_became_active":"2024-05-28T06:22:16.998428+0000","last_became_peered":"2024-05-28T06:22:16.998428+0000","last_unstale":"2024-05-28T06:22:18.241796+0000","last_undegraded":"2024-05-28T06:22:18.241796+0000","last_fullsized":"2024-05-28T06:22:18.241796+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:43:26.170749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.000006+0000","last_change":"2024-05-28T06:22:15.981969+0000","last_active":"2024-05-28T06:22:18.000006+0000","last_peered":"2024-05-28T06:22:18.000006+0000","last_clean":"2024-05-28T06:22:18.000006+0000","last_became_active":"2024-05-28T06:22:15.979073+0000","last_became_peered":"2024-05-28T06:22:15.979073+0000","last_unstale":"2024-05-28T06:22:18.000006+0000","last_undegraded":"2024-05-28T06:22:18.000006+0000","last_fullsized":"2024-05-28T06:22:18.000006+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:41:39.601468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.024757+0000","last_change":"2024-05-28T06:22:17.006594+0000","last_active":"2024-05-28T06:22:18.024757+0000","last_peered":"2024-05-28T06:22:18.024757+0000","last_clean":"2024-05-28T06:22:18.024757+0000","last_became_active":"2024-05-28T06:22:17.006057+0000","last_became_peered":"2024-05-28T06:22:17.006057+0000","last_unstale":"2024-05-28T06:22:18.024757+0000","last_undegraded":"2024-05-28T06:22:18.024757+0000","last_fullsized":"2024-05-28T06:22:18.024757+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:39:44.471675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.242254+0000","last_change":"2024-05-28T06:22:15.988357+0000","last_active":"2024-05-28T06:22:18.242254+0000","last_peered":"2024-05-28T06:22:18.242254+0000","last_clean":"2024-05-28T06:22:18.242254+0000","last_became_active":"2024-05-28T06:22:15.988053+0000","last_became_peered":"2024-05-28T06:22:15.988053+0000","last_unstale":"2024-05-28T06:22:18.242254+0000","last_undegraded":"2024-05-28T06:22:18.242254+0000","last_fullsized":"2024-05-28T06:22:18.242254+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:24:37.584832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.024688+0000","last_change":"2024-05-28T06:22:17.003420+0000","last_active":"2024-05-28T06:22:18.024688+0000","last_peered":"2024-05-28T06:22:18.024688+0000","last_clean":"2024-05-28T06:22:18.024688+0000","last_became_active":"2024-05-28T06:22:17.003005+0000","last_became_peered":"2024-05-28T06:22:17.003005+0000","last_unstale":"2024-05-28T06:22:18.024688+0000","last_undegraded":"2024-05-28T06:22:18.024688+0000","last_fullsized":"2024-05-28T06:22:18.024688+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:26:50.855556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:17.999693+0000","last_change":"2024-05-28T06:22:15.984544+0000","last_active":"2024-05-28T06:22:17.999693+0000","last_peered":"2024-05-28T06:22:17.999693+0000","last_clean":"2024-05-28T06:22:17.999693+0000","last_became_active":"2024-05-28T06:22:15.984287+0000","last_became_peered":"2024-05-28T06:22:15.984287+0000","last_unstale":"2024-05-28T06:22:17.999693+0000","last_undegraded":"2024-05-28T06:22:17.999693+0000","last_fullsized":"2024-05-28T06:22:17.999693+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T18:04:33.437207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.020450+0000","last_change":"2024-05-28T06:22:17.012046+0000","last_active":"2024-05-28T06:22:18.020450+0000","last_peered":"2024-05-28T06:22:18.020450+0000","last_clean":"2024-05-28T06:22:18.020450+0000","last_became_active":"2024-05-28T06:22:17.007050+0000","last_became_peered":"2024-05-28T06:22:17.007050+0000","last_unstale":"2024-05-28T06:22:18.020450+0000","last_undegraded":"2024-05-28T06:22:18.020450+0000","last_fullsized":"2024-05-28T06:22:18.020450+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:48:14.529168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.529413+0000","last_change":"2024-05-28T06:22:15.989398+0000","last_active":"2024-05-28T06:22:18.529413+0000","last_peered":"2024-05-28T06:22:18.529413+0000","last_clean":"2024-05-28T06:22:18.529413+0000","last_became_active":"2024-05-28T06:22:15.989101+0000","last_became_peered":"2024-05-28T06:22:15.989101+0000","last_unstale":"2024-05-28T06:22:18.529413+0000","last_undegraded":"2024-05-28T06:22:18.529413+0000","last_fullsized":"2024-05-28T06:22:18.529413+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:43:46.372846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.529711+0000","last_change":"2024-05-28T06:22:16.993645+0000","last_active":"2024-05-28T06:22:18.529711+0000","last_peered":"2024-05-28T06:22:18.529711+0000","last_clean":"2024-05-28T06:22:18.529711+0000","last_became_active":"2024-05-28T06:22:16.992912+0000","last_became_peered":"2024-05-28T06:22:16.992912+0000","last_unstale":"2024-05-28T06:22:18.529711+0000","last_undegraded":"2024-05-28T06:22:18.529711+0000","last_fullsized":"2024-05-28T06:22:18.529711+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:16:11.794755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"114'3","reported_seq":17,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.242908+0000","last_change":"2024-05-28T06:22:15.998469+0000","last_active":"2024-05-28T06:22:18.242908+0000","last_peered":"2024-05-28T06:22:18.242908+0000","last_clean":"2024-05-28T06:22:18.242908+0000","last_became_active":"2024-05-28T06:22:15.998186+0000","last_became_peered":"2024-05-28T06:22:15.998186+0000","last_unstale":"2024-05-28T06:22:18.242908+0000","last_undegraded":"2024-05-28T06:22:18.242908+0000","last_fullsized":"2024-05-28T06:22:18.242908+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:23:41.228363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.024744+0000","last_change":"2024-05-28T06:22:17.009032+0000","last_active":"2024-05-28T06:22:18.024744+0000","last_peered":"2024-05-28T06:22:18.024744+0000","last_clean":"2024-05-28T06:22:18.024744+0000","last_became_active":"2024-05-28T06:22:17.007965+0000","last_became_peered":"2024-05-28T06:22:17.007965+0000","last_unstale":"2024-05-28T06:22:18.024744+0000","last_undegraded":"2024-05-28T06:22:18.024744+0000","last_fullsized":"2024-05-28T06:22:18.024744+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:13:32.315186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"114'6","reported_seq":20,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.243514+0000","last_change":"2024-05-28T06:22:15.997183+0000","last_active":"2024-05-28T06:22:18.243514+0000","last_peered":"2024-05-28T06:22:18.243514+0000","last_clean":"2024-05-28T06:22:18.243514+0000","last_became_active":"2024-05-28T06:22:15.996809+0000","last_became_peered":"2024-05-28T06:22:15.996809+0000","last_unstale":"2024-05-28T06:22:18.243514+0000","last_undegraded":"2024-05-28T06:22:18.243514+0000","last_fullsized":"2024-05-28T06:22:18.243514+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:38:33.839493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.528917+0000","last_change":"2024-05-28T06:22:17.000291+0000","last_active":"2024-05-28T06:22:18.528917+0000","last_peered":"2024-05-28T06:22:18.528917+0000","last_clean":"2024-05-28T06:22:18.528917+0000","last_became_active":"2024-05-28T06:22:16.999910+0000","last_became_peered":"2024-05-28T06:22:16.999910+0000","last_unstale":"2024-05-28T06:22:18.528917+0000","last_undegraded":"2024-05-28T06:22:18.528917+0000","last_fullsized":"2024-05-28T06:22:18.528917+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:23:42.240172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"114'2","reported_seq":16,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.528553+0000","last_change":"2024-05-28T06:22:15.996773+0000","last_active":"2024-05-28T06:22:18.528553+0000","last_peered":"2024-05-28T06:22:18.528553+0000","last_clean":"2024-05-28T06:22:18.528553+0000","last_became_active":"2024-05-28T06:22:15.995681+0000","last_became_peered":"2024-05-28T06:22:15.995681+0000","last_unstale":"2024-05-28T06:22:18.528553+0000","last_undegraded":"2024-05-28T06:22:18.528553+0000","last_fullsized":"2024-05-28T06:22:18.528553+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:48:09.956343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.528770+0000","last_change":"2024-05-28T06:22:17.003164+0000","last_active":"2024-05-28T06:22:18.528770+0000","last_peered":"2024-05-28T06:22:18.528770+0000","last_clean":"2024-05-28T06:22:18.528770+0000","last_became_active":"2024-05-28T06:22:17.002906+0000","last_became_peered":"2024-05-28T06:22:17.002906+0000","last_unstale":"2024-05-28T06:22:18.528770+0000","last_undegraded":"2024-05-28T06:22:18.528770+0000","last_fullsized":"2024-05-28T06:22:18.528770+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:13:14.889721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.020651+0000","last_change":"2024-05-28T06:22:15.993511+0000","last_active":"2024-05-28T06:22:18.020651+0000","last_peered":"2024-05-28T06:22:18.020651+0000","last_clean":"2024-05-28T06:22:18.020651+0000","last_became_active":"2024-05-28T06:22:15.992788+0000","last_became_peered":"2024-05-28T06:22:15.992788+0000","last_unstale":"2024-05-28T06:22:18.020651+0000","last_undegraded":"2024-05-28T06:22:18.020651+0000","last_fullsized":"2024-05-28T06:22:18.020651+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:43:46.372846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.527440+0000","last_change":"2024-05-28T06:22:16.991808+0000","last_active":"2024-05-28T06:22:18.527440+0000","last_peered":"2024-05-28T06:22:18.527440+0000","last_clean":"2024-05-28T06:22:18.527440+0000","last_became_active":"2024-05-28T06:22:16.991494+0000","last_became_peered":"2024-05-28T06:22:16.991494+0000","last_unstale":"2024-05-28T06:22:18.527440+0000","last_undegraded":"2024-05-28T06:22:18.527440+0000","last_fullsized":"2024-05-28T06:22:18.527440+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:47:18.486697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"114'8","reported_seq":22,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:17.999084+0000","last_change":"2024-05-28T06:22:15.991378+0000","last_active":"2024-05-28T06:22:17.999084+0000","last_peered":"2024-05-28T06:22:17.999084+0000","last_clean":"2024-05-28T06:22:17.999084+0000","last_became_active":"2024-05-28T06:22:15.990388+0000","last_became_peered":"2024-05-28T06:22:15.990388+0000","last_unstale":"2024-05-28T06:22:17.999084+0000","last_undegraded":"2024-05-28T06:22:17.999084+0000","last_fullsized":"2024-05-28T06:22:17.999084+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:49:10.819709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:17.999777+0000","last_change":"2024-05-28T06:22:16.991114+0000","last_active":"2024-05-28T06:22:17.999777+0000","last_peered":"2024-05-28T06:22:17.999777+0000","last_clean":"2024-05-28T06:22:17.999777+0000","last_became_active":"2024-05-28T06:22:16.990159+0000","last_became_peered":"2024-05-28T06:22:16.990159+0000","last_unstale":"2024-05-28T06:22:17.999777+0000","last_undegraded":"2024-05-28T06:22:17.999777+0000","last_fullsized":"2024-05-28T06:22:17.999777+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:18:01.989158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"114'1","reported_seq":15,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.242742+0000","last_change":"2024-05-28T06:22:15.997418+0000","last_active":"2024-05-28T06:22:18.242742+0000","last_peered":"2024-05-28T06:22:18.242742+0000","last_clean":"2024-05-28T06:22:18.242742+0000","last_became_active":"2024-05-28T06:22:15.995786+0000","last_became_peered":"2024-05-28T06:22:15.995786+0000","last_unstale":"2024-05-28T06:22:18.242742+0000","last_undegraded":"2024-05-28T06:22:18.242742+0000","last_fullsized":"2024-05-28T06:22:18.242742+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:50:11.148336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.530295+0000","last_change":"2024-05-28T06:22:16.993861+0000","last_active":"2024-05-28T06:22:18.530295+0000","last_peered":"2024-05-28T06:22:18.530295+0000","last_clean":"2024-05-28T06:22:18.530295+0000","last_became_active":"2024-05-28T06:22:16.993602+0000","last_became_peered":"2024-05-28T06:22:16.993602+0000","last_unstale":"2024-05-28T06:22:18.530295+0000","last_undegraded":"2024-05-28T06:22:18.530295+0000","last_fullsized":"2024-05-28T06:22:18.530295+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:28:17.141535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"114'2","reported_seq":16,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:17.999975+0000","last_change":"2024-05-28T06:22:15.980689+0000","last_active":"2024-05-28T06:22:17.999975+0000","last_peered":"2024-05-28T06:22:17.999975+0000","last_clean":"2024-05-28T06:22:17.999975+0000","last_became_active":"2024-05-28T06:22:15.980139+0000","last_became_peered":"2024-05-28T06:22:15.980139+0000","last_unstale":"2024-05-28T06:22:17.999975+0000","last_undegraded":"2024-05-28T06:22:17.999975+0000","last_fullsized":"2024-05-28T06:22:17.999975+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:19:21.410923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.528248+0000","last_change":"2024-05-28T06:22:16.999644+0000","last_active":"2024-05-28T06:22:18.528248+0000","last_peered":"2024-05-28T06:22:18.528248+0000","last_clean":"2024-05-28T06:22:18.528248+0000","last_became_active":"2024-05-28T06:22:16.998935+0000","last_became_peered":"2024-05-28T06:22:16.998935+0000","last_unstale":"2024-05-28T06:22:18.528248+0000","last_undegraded":"2024-05-28T06:22:18.528248+0000","last_fullsized":"2024-05-28T06:22:18.528248+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:26:18.638055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"114'1","reported_seq":15,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.019924+0000","last_change":"2024-05-28T06:22:15.996123+0000","last_active":"2024-05-28T06:22:18.019924+0000","last_peered":"2024-05-28T06:22:18.019924+0000","last_clean":"2024-05-28T06:22:18.019924+0000","last_became_active":"2024-05-28T06:22:15.995728+0000","last_became_peered":"2024-05-28T06:22:15.995728+0000","last_unstale":"2024-05-28T06:22:18.019924+0000","last_undegraded":"2024-05-28T06:22:18.019924+0000","last_fullsized":"2024-05-28T06:22:18.019924+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:08:38.270230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.528924+0000","last_change":"2024-05-28T06:22:17.003252+0000","last_active":"2024-05-28T06:22:18.528924+0000","last_peered":"2024-05-28T06:22:18.528924+0000","last_clean":"2024-05-28T06:22:18.528924+0000","last_became_active":"2024-05-28T06:22:17.001195+0000","last_became_peered":"2024-05-28T06:22:17.001195+0000","last_unstale":"2024-05-28T06:22:18.528924+0000","last_undegraded":"2024-05-28T06:22:18.528924+0000","last_fullsized":"2024-05-28T06:22:18.528924+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:27:26.116768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.024177+0000","last_change":"2024-05-28T06:22:15.998545+0000","last_active":"2024-05-28T06:22:18.024177+0000","last_peered":"2024-05-28T06:22:18.024177+0000","last_clean":"2024-05-28T06:22:18.024177+0000","last_became_active":"2024-05-28T06:22:15.997754+0000","last_became_peered":"2024-05-28T06:22:15.997754+0000","last_unstale":"2024-05-28T06:22:18.024177+0000","last_undegraded":"2024-05-28T06:22:18.024177+0000","last_fullsized":"2024-05-28T06:22:18.024177+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:58:41.561009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.527487+0000","last_change":"2024-05-28T06:22:17.000185+0000","last_active":"2024-05-28T06:22:18.527487+0000","last_peered":"2024-05-28T06:22:18.527487+0000","last_clean":"2024-05-28T06:22:18.527487+0000","last_became_active":"2024-05-28T06:22:16.997369+0000","last_became_peered":"2024-05-28T06:22:16.997369+0000","last_unstale":"2024-05-28T06:22:18.527487+0000","last_undegraded":"2024-05-28T06:22:18.527487+0000","last_fullsized":"2024-05-28T06:22:18.527487+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:08:39.984746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"114'2","reported_seq":16,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.528872+0000","last_change":"2024-05-28T06:22:15.991711+0000","last_active":"2024-05-28T06:22:18.528872+0000","last_peered":"2024-05-28T06:22:18.528872+0000","last_clean":"2024-05-28T06:22:18.528872+0000","last_became_active":"2024-05-28T06:22:15.991323+0000","last_became_peered":"2024-05-28T06:22:15.991323+0000","last_unstale":"2024-05-28T06:22:18.528872+0000","last_undegraded":"2024-05-28T06:22:18.528872+0000","last_fullsized":"2024-05-28T06:22:18.528872+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:02:22.717474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.242729+0000","last_change":"2024-05-28T06:22:17.003257+0000","last_active":"2024-05-28T06:22:18.242729+0000","last_peered":"2024-05-28T06:22:18.242729+0000","last_clean":"2024-05-28T06:22:18.242729+0000","last_became_active":"2024-05-28T06:22:17.000808+0000","last_became_peered":"2024-05-28T06:22:17.000808+0000","last_unstale":"2024-05-28T06:22:18.242729+0000","last_undegraded":"2024-05-28T06:22:18.242729+0000","last_fullsized":"2024-05-28T06:22:18.242729+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:27:48.276213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:17.999010+0000","last_change":"2024-05-28T06:22:15.994076+0000","last_active":"2024-05-28T06:22:17.999010+0000","last_peered":"2024-05-28T06:22:17.999010+0000","last_clean":"2024-05-28T06:22:17.999010+0000","last_became_active":"2024-05-28T06:22:15.993488+0000","last_became_peered":"2024-05-28T06:22:15.993488+0000","last_unstale":"2024-05-28T06:22:17.999010+0000","last_undegraded":"2024-05-28T06:22:17.999010+0000","last_fullsized":"2024-05-28T06:22:17.999010+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:08:35.517640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.527460+0000","last_change":"2024-05-28T06:22:16.995803+0000","last_active":"2024-05-28T06:22:18.527460+0000","last_peered":"2024-05-28T06:22:18.527460+0000","last_clean":"2024-05-28T06:22:18.527460+0000","last_became_active":"2024-05-28T06:22:16.995475+0000","last_became_peered":"2024-05-28T06:22:16.995475+0000","last_unstale":"2024-05-28T06:22:18.527460+0000","last_undegraded":"2024-05-28T06:22:18.527460+0000","last_fullsized":"2024-05-28T06:22:18.527460+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T18:02:32.237669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.527971+0000","last_change":"2024-05-28T06:22:15.989588+0000","last_active":"2024-05-28T06:22:18.527971+0000","last_peered":"2024-05-28T06:22:18.527971+0000","last_clean":"2024-05-28T06:22:18.527971+0000","last_became_active":"2024-05-28T06:22:15.989331+0000","last_became_peered":"2024-05-28T06:22:15.989331+0000","last_unstale":"2024-05-28T06:22:18.527971+0000","last_undegraded":"2024-05-28T06:22:18.527971+0000","last_fullsized":"2024-05-28T06:22:18.527971+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:19:21.410923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.527956+0000","last_change":"2024-05-28T06:22:16.993785+0000","last_active":"2024-05-28T06:22:18.527956+0000","last_peered":"2024-05-28T06:22:18.527956+0000","last_clean":"2024-05-28T06:22:18.527956+0000","last_became_active":"2024-05-28T06:22:16.993402+0000","last_became_peered":"2024-05-28T06:22:16.993402+0000","last_unstale":"2024-05-28T06:22:18.527956+0000","last_undegraded":"2024-05-28T06:22:18.527956+0000","last_fullsized":"2024-05-28T06:22:18.527956+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:06:13.719079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.242342+0000","last_change":"2024-05-28T06:22:15.998948+0000","last_active":"2024-05-28T06:22:18.242342+0000","last_peered":"2024-05-28T06:22:18.242342+0000","last_clean":"2024-05-28T06:22:18.242342+0000","last_became_active":"2024-05-28T06:22:15.998205+0000","last_became_peered":"2024-05-28T06:22:15.998205+0000","last_unstale":"2024-05-28T06:22:18.242342+0000","last_undegraded":"2024-05-28T06:22:18.242342+0000","last_fullsized":"2024-05-28T06:22:18.242342+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:58:34.652147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.020433+0000","last_change":"2024-05-28T06:22:17.005113+0000","last_active":"2024-05-28T06:22:18.020433+0000","last_peered":"2024-05-28T06:22:18.020433+0000","last_clean":"2024-05-28T06:22:18.020433+0000","last_became_active":"2024-05-28T06:22:17.003020+0000","last_became_peered":"2024-05-28T06:22:17.003020+0000","last_unstale":"2024-05-28T06:22:18.020433+0000","last_undegraded":"2024-05-28T06:22:18.020433+0000","last_fullsized":"2024-05-28T06:22:18.020433+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:17:14.655934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"114'2","reported_seq":16,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.528934+0000","last_change":"2024-05-28T06:22:15.990864+0000","last_active":"2024-05-28T06:22:18.528934+0000","last_peered":"2024-05-28T06:22:18.528934+0000","last_clean":"2024-05-28T06:22:18.528934+0000","last_became_active":"2024-05-28T06:22:15.990559+0000","last_became_peered":"2024-05-28T06:22:15.990559+0000","last_unstale":"2024-05-28T06:22:18.528934+0000","last_undegraded":"2024-05-28T06:22:18.528934+0000","last_fullsized":"2024-05-28T06:22:18.528934+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:51:36.120307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.242633+0000","last_change":"2024-05-28T06:22:16.992471+0000","last_active":"2024-05-28T06:22:18.242633+0000","last_peered":"2024-05-28T06:22:18.242633+0000","last_clean":"2024-05-28T06:22:18.242633+0000","last_became_active":"2024-05-28T06:22:16.992135+0000","last_became_peered":"2024-05-28T06:22:16.992135+0000","last_unstale":"2024-05-28T06:22:18.242633+0000","last_undegraded":"2024-05-28T06:22:18.242633+0000","last_fullsized":"2024-05-28T06:22:18.242633+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:57:13.651588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"115'1","reported_seq":15,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.020608+0000","last_change":"2024-05-28T06:22:15.994161+0000","last_active":"2024-05-28T06:22:18.020608+0000","last_peered":"2024-05-28T06:22:18.020608+0000","last_clean":"2024-05-28T06:22:18.020608+0000","last_became_active":"2024-05-28T06:22:15.993895+0000","last_became_peered":"2024-05-28T06:22:15.993895+0000","last_unstale":"2024-05-28T06:22:18.020608+0000","last_undegraded":"2024-05-28T06:22:18.020608+0000","last_fullsized":"2024-05-28T06:22:18.020608+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:51:36.120307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.243620+0000","last_change":"2024-05-28T06:22:16.993934+0000","last_active":"2024-05-28T06:22:18.243620+0000","last_peered":"2024-05-28T06:22:18.243620+0000","last_clean":"2024-05-28T06:22:18.243620+0000","last_became_active":"2024-05-28T06:22:16.993249+0000","last_became_peered":"2024-05-28T06:22:16.993249+0000","last_unstale":"2024-05-28T06:22:18.243620+0000","last_undegraded":"2024-05-28T06:22:18.243620+0000","last_fullsized":"2024-05-28T06:22:18.243620+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:21:11.675368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.242990+0000","last_change":"2024-05-28T06:22:15.998999+0000","last_active":"2024-05-28T06:22:18.242990+0000","last_peered":"2024-05-28T06:22:18.242990+0000","last_clean":"2024-05-28T06:22:18.242990+0000","last_became_active":"2024-05-28T06:22:15.998631+0000","last_became_peered":"2024-05-28T06:22:15.998631+0000","last_unstale":"2024-05-28T06:22:18.242990+0000","last_undegraded":"2024-05-28T06:22:18.242990+0000","last_fullsized":"2024-05-28T06:22:18.242990+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:26:17.626246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.020122+0000","last_change":"2024-05-28T06:22:17.002213+0000","last_active":"2024-05-28T06:22:18.020122+0000","last_peered":"2024-05-28T06:22:18.020122+0000","last_clean":"2024-05-28T06:22:18.020122+0000","last_became_active":"2024-05-28T06:22:17.001843+0000","last_became_peered":"2024-05-28T06:22:17.001843+0000","last_unstale":"2024-05-28T06:22:18.020122+0000","last_undegraded":"2024-05-28T06:22:18.020122+0000","last_fullsized":"2024-05-28T06:22:18.020122+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:24:38.596641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.529805+0000","last_change":"2024-05-28T06:22:15.995193+0000","last_active":"2024-05-28T06:22:18.529805+0000","last_peered":"2024-05-28T06:22:18.529805+0000","last_clean":"2024-05-28T06:22:18.529805+0000","last_became_active":"2024-05-28T06:22:15.994748+0000","last_became_peered":"2024-05-28T06:22:15.994748+0000","last_unstale":"2024-05-28T06:22:18.529805+0000","last_undegraded":"2024-05-28T06:22:18.529805+0000","last_fullsized":"2024-05-28T06:22:18.529805+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:08:38.270230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.000212+0000","last_change":"2024-05-28T06:22:17.005154+0000","last_active":"2024-05-28T06:22:18.000212+0000","last_peered":"2024-05-28T06:22:18.000212+0000","last_clean":"2024-05-28T06:22:18.000212+0000","last_became_active":"2024-05-28T06:22:17.004836+0000","last_became_peered":"2024-05-28T06:22:17.004836+0000","last_unstale":"2024-05-28T06:22:18.000212+0000","last_undegraded":"2024-05-28T06:22:18.000212+0000","last_fullsized":"2024-05-28T06:22:18.000212+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T18:02:32.237669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.024271+0000","last_change":"2024-05-28T06:22:15.996930+0000","last_active":"2024-05-28T06:22:18.024271+0000","last_peered":"2024-05-28T06:22:18.024271+0000","last_clean":"2024-05-28T06:22:18.024271+0000","last_became_active":"2024-05-28T06:22:15.996598+0000","last_became_peered":"2024-05-28T06:22:15.996598+0000","last_unstale":"2024-05-28T06:22:18.024271+0000","last_undegraded":"2024-05-28T06:22:18.024271+0000","last_fullsized":"2024-05-28T06:22:18.024271+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:37:04.441468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.000755+0000","last_change":"2024-05-28T06:22:16.983951+0000","last_active":"2024-05-28T06:22:18.000755+0000","last_peered":"2024-05-28T06:22:18.000755+0000","last_clean":"2024-05-28T06:22:18.000755+0000","last_became_active":"2024-05-28T06:22:16.983201+0000","last_became_peered":"2024-05-28T06:22:16.983201+0000","last_unstale":"2024-05-28T06:22:18.000755+0000","last_undegraded":"2024-05-28T06:22:18.000755+0000","last_fullsized":"2024-05-28T06:22:18.000755+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:47:18.486697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.000007+0000","last_change":"2024-05-28T06:22:15.985622+0000","last_active":"2024-05-28T06:22:18.000007+0000","last_peered":"2024-05-28T06:22:18.000007+0000","last_clean":"2024-05-28T06:22:18.000007+0000","last_became_active":"2024-05-28T06:22:15.984560+0000","last_became_peered":"2024-05-28T06:22:15.984560+0000","last_unstale":"2024-05-28T06:22:18.000007+0000","last_undegraded":"2024-05-28T06:22:18.000007+0000","last_fullsized":"2024-05-28T06:22:18.000007+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:21:24.669839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.527773+0000","last_change":"2024-05-28T06:22:17.005937+0000","last_active":"2024-05-28T06:22:18.527773+0000","last_peered":"2024-05-28T06:22:18.527773+0000","last_clean":"2024-05-28T06:22:18.527773+0000","last_became_active":"2024-05-28T06:22:17.004354+0000","last_became_peered":"2024-05-28T06:22:17.004354+0000","last_unstale":"2024-05-28T06:22:18.527773+0000","last_undegraded":"2024-05-28T06:22:18.527773+0000","last_fullsized":"2024-05-28T06:22:18.527773+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:13:14.889721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:17.999121+0000","last_change":"2024-05-28T06:22:15.984208+0000","last_active":"2024-05-28T06:22:17.999121+0000","last_peered":"2024-05-28T06:22:17.999121+0000","last_clean":"2024-05-28T06:22:17.999121+0000","last_became_active":"2024-05-28T06:22:15.983816+0000","last_became_peered":"2024-05-28T06:22:15.983816+0000","last_unstale":"2024-05-28T06:22:17.999121+0000","last_undegraded":"2024-05-28T06:22:17.999121+0000","last_fullsized":"2024-05-28T06:22:17.999121+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:30:43.782132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.242161+0000","last_change":"2024-05-28T06:22:16.996830+0000","last_active":"2024-05-28T06:22:18.242161+0000","last_peered":"2024-05-28T06:22:18.242161+0000","last_clean":"2024-05-28T06:22:18.242161+0000","last_became_active":"2024-05-28T06:22:16.990102+0000","last_became_peered":"2024-05-28T06:22:16.990102+0000","last_unstale":"2024-05-28T06:22:18.242161+0000","last_undegraded":"2024-05-28T06:22:18.242161+0000","last_fullsized":"2024-05-28T06:22:18.242161+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T18:04:15.407639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.242532+0000","last_change":"2024-05-28T06:22:15.998500+0000","last_active":"2024-05-28T06:22:18.242532+0000","last_peered":"2024-05-28T06:22:18.242532+0000","last_clean":"2024-05-28T06:22:18.242532+0000","last_became_active":"2024-05-28T06:22:15.998176+0000","last_became_peered":"2024-05-28T06:22:15.998176+0000","last_unstale":"2024-05-28T06:22:18.242532+0000","last_undegraded":"2024-05-28T06:22:18.242532+0000","last_fullsized":"2024-05-28T06:22:18.242532+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:17:13.644125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.527150+0000","last_change":"2024-05-28T06:22:17.008104+0000","last_active":"2024-05-28T06:22:18.527150+0000","last_peered":"2024-05-28T06:22:18.527150+0000","last_clean":"2024-05-28T06:22:18.527150+0000","last_became_active":"2024-05-28T06:22:17.007752+0000","last_became_peered":"2024-05-28T06:22:17.007752+0000","last_unstale":"2024-05-28T06:22:18.527150+0000","last_undegraded":"2024-05-28T06:22:18.527150+0000","last_fullsized":"2024-05-28T06:22:18.527150+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:08:39.984746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.024388+0000","last_change":"2024-05-28T06:22:15.998356+0000","last_active":"2024-05-28T06:22:18.024388+0000","last_peered":"2024-05-28T06:22:18.024388+0000","last_clean":"2024-05-28T06:22:18.024388+0000","last_became_active":"2024-05-28T06:22:15.998063+0000","last_became_peered":"2024-05-28T06:22:15.998063+0000","last_unstale":"2024-05-28T06:22:18.024388+0000","last_undegraded":"2024-05-28T06:22:18.024388+0000","last_fullsized":"2024-05-28T06:22:18.024388+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:05:24.649440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":354,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.528168+0000","last_change":"2024-05-28T06:04:01.039129+0000","last_active":"2024-05-28T06:22:18.528168+0000","last_peered":"2024-05-28T06:22:18.528168+0000","last_clean":"2024-05-28T06:22:18.528168+0000","last_became_active":"2024-05-28T06:04:01.033906+0000","last_became_peered":"2024-05-28T06:04:01.033906+0000","last_unstale":"2024-05-28T06:22:18.528168+0000","last_undegraded":"2024-05-28T06:22:18.528168+0000","last_fullsized":"2024-05-28T06:22:18.528168+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:03:59.864237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:03:59.864237+0000","last_clean_scrub_stamp":"2024-05-28T06:03:59.864237+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:17:36.440245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575071,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":127500,"kb_used_data":1984,"kb_used_omap":1,"kb_used_meta":125502,"kb_avail":94244340,"statfs":{"total":96636764160,"available":96506204160,"internally_reserved":0,"allocated":2031616,"data_stored":1217255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":128514506},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575073,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":126916,"kb_used_data":1400,"kb_used_omap":1,"kb_used_meta":125502,"kb_avail":94244924,"statfs":{"total":96636764160,"available":96506802176,"internally_reserved":0,"allocated":1433600,"data_stored":626341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":128514502},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575071,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64464,"kb_used_data":1404,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94307376,"statfs":{"total":96636764160,"available":96570753024,"internally_reserved":0,"allocated":1437696,"data_stored":626875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575071,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":86656,"kb_used_data":1396,"kb_used_omap":1,"kb_used_meta":85246,"kb_avail":94285184,"statfs":{"total":96636764160,"available":96548028416,"internally_reserved":0,"allocated":1429504,"data_stored":626271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87292357},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575073,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":131096,"kb_used_data":1412,"kb_used_omap":1,"kb_used_meta":129662,"kb_avail":94240744,"statfs":{"total":96636764160,"available":96502521856,"internally_reserved":0,"allocated":1445888,"data_stored":627953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":132774343},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575072,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":144960,"kb_used_data":1396,"kb_used_omap":1,"kb_used_meta":143550,"kb_avail":94226880,"statfs":{"total":96636764160,"available":96488325120,"internally_reserved":0,"allocated":1429504,"data_stored":626271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":146995653},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575072,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":126920,"kb_used_data":1404,"kb_used_omap":1,"kb_used_meta":125502,"kb_avail":94244920,"statfs":{"total":96636764160,"available":96506798080,"internally_reserved":0,"allocated":1437696,"data_stored":626395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":128514501},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575071,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":87260,"kb_used_data":1996,"kb_used_omap":1,"kb_used_meta":85246,"kb_avail":94284580,"statfs":{"total":96636764160,"available":96547409920,"internally_reserved":0,"allocated":2043904,"data_stored":1218363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87292357},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-28T06:22:26.439 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-28T06:22:26.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.655+0000 7f47c4c00640 1 Processor -- start 2024-05-28T06:22:26.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.655+0000 7f47c4c00640 1 -- start start 2024-05-28T06:22:26.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.657+0000 7f47c4c00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f47c00af6d0 0x7f47c00b4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:26.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.657+0000 7f47c4c00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47c00b5780 con 0x7f47c015f1e0 2024-05-28T06:22:26.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.657+0000 7f47c4c00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47c00aebf0 con 0x7f47c00b5200 2024-05-28T06:22:26.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.657+0000 7f47c4c00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f47c00af240 con 0x7f47c00af6d0 2024-05-28T06:22:26.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.657+0000 7f47bf400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f47c015f1e0 0x7f47c015f5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:35128/0 (socket says 172.21.15.96:35128) 2024-05-28T06:22:26.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.657+0000 7f47bf400640 1 -- 172.21.15.96:0/974750550 learned_addr learned my addr 172.21.15.96:0/974750550 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:22:26.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.658+0000 7f47b7e00640 1 --2- 172.21.15.96:0/974750550 >> v2:172.21.15.123:3300/0 conn(0x7f47c00af6d0 0x7f47c00b4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:26.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.658+0000 7f47bea00640 1 -- 172.21.15.96:0/974750550 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3936692925 0 0) 0x7f47c00b5780 con 0x7f47c015f1e0 2024-05-28T06:22:26.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.658+0000 7f47bea00640 1 -- 172.21.15.96:0/974750550 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f47a4003520 con 0x7f47c015f1e0 2024-05-28T06:22:26.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.658+0000 7f47bea00640 1 -- 172.21.15.96:0/974750550 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2453416930 0 0) 0x7f47a4003520 con 0x7f47c015f1e0 2024-05-28T06:22:26.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.658+0000 7f47bea00640 1 -- 172.21.15.96:0/974750550 >> v2:172.21.15.123:3300/0 conn(0x7f47c00af6d0 msgr2=0x7f47c00b4cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:26.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.658+0000 7f47bea00640 1 --2- 172.21.15.96:0/974750550 >> v2:172.21.15.123:3300/0 conn(0x7f47c00af6d0 0x7f47c00b4cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:26.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.658+0000 7f47bea00640 1 -- 172.21.15.96:0/974750550 >> v1:172.21.15.123:6789/0 conn(0x7f47c00b5200 legacy=0x7f47c0071f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:26.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.658+0000 7f47bea00640 1 -- 172.21.15.96:0/974750550 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47c00aeed0 con 0x7f47c015f1e0 2024-05-28T06:22:26.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.658+0000 7f47bea00640 1 -- 172.21.15.96:0/974750550 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f47b0001670 con 0x7f47c015f1e0 2024-05-28T06:22:26.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.658+0000 7f47bea00640 1 -- 172.21.15.96:0/974750550 >> v1:172.21.15.96:6789/0 conn(0x7f47c015f1e0 legacy=0x7f47c015f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:26.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.658+0000 7f47bea00640 1 --2- 172.21.15.96:0/974750550 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f47a40039b0 0x7f47a4003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:26.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.658+0000 7f47bea00640 1 --2- 172.21.15.96:0/974750550 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f47a4004800 0x7f47a4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:26.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.658+0000 7f47bea00640 1 --2- 172.21.15.96:0/974750550 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f47a40051c0 0x7f47a400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:26.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.658+0000 7f47bea00640 1 -- 172.21.15.96:0/974750550 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f47c00af240 con 0x7f47a4004800 2024-05-28T06:22:26.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.658+0000 7f47bea00640 1 -- 172.21.15.96:0/974750550 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f47a400c090 con 0x7f47a40051c0 2024-05-28T06:22:26.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.658+0000 7f47bea00640 1 -- 172.21.15.96:0/974750550 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f47a400c290 con 0x7f47a40039b0 2024-05-28T06:22:26.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.659+0000 7f47b7e00640 1 --2- 172.21.15.96:0/974750550 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f47a4004800 0x7f47a4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:26.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.659+0000 7f47bf400640 1 --2- 172.21.15.96:0/974750550 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f47a40039b0 0x7f47a4003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:26.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.659+0000 7f47bfe00640 1 --2- 172.21.15.96:0/974750550 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f47a40051c0 0x7f47a400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:26.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.659+0000 7f47b7e00640 1 -- 172.21.15.96:0/974750550 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f47a40039b0 msgr2=0x7f47a4003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:26.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.659+0000 7f47b7e00640 1 --2- 172.21.15.96:0/974750550 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f47a40039b0 0x7f47a4003df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:26.657 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.659+0000 7f47b7e00640 1 -- 172.21.15.96:0/974750550 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f47a40051c0 msgr2=0x7f47a400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:26.657 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.659+0000 7f47b7e00640 1 --2- 172.21.15.96:0/974750550 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f47a40051c0 0x7f47a400b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:26.657 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.659+0000 7f47b7e00640 1 -- 172.21.15.96:0/974750550 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47a4003520 con 0x7f47a4004800 2024-05-28T06:22:26.657 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.659+0000 7f47bfe00640 1 -- 172.21.15.96:0/974750550 reap_dead start 2024-05-28T06:22:26.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.660+0000 7f47b7e00640 1 --2- 172.21.15.96:0/974750550 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f47a4004800 0x7f47a4004c80 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7f47a8009ac0 tx=0x7f47a4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88f61a4c869b17a5 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:26.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.660+0000 7f47bea00640 1 -- 172.21.15.96:0/974750550 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47a8036070 con 0x7f47a4004800 2024-05-28T06:22:26.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.661+0000 7f47bea00640 1 -- 172.21.15.96:0/974750550 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47a8031400 con 0x7f47a4004800 2024-05-28T06:22:26.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.661+0000 7f47bea00640 1 -- 172.21.15.96:0/974750550 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47a8009690 con 0x7f47a4004800 2024-05-28T06:22:26.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.661+0000 7f47c4c00640 1 -- 172.21.15.96:0/974750550 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f47a4004800 msgr2=0x7f47a4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:26.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.661+0000 7f47c4c00640 1 --2- 172.21.15.96:0/974750550 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f47a4004800 0x7f47a4004c80 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7f47a8009ac0 tx=0x7f47a4005830 comp rx=0 tx=0).stop 2024-05-28T06:22:26.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.661+0000 7f47c4c00640 1 -- 172.21.15.96:0/974750550 shutdown_connections 2024-05-28T06:22:26.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.661+0000 7f47c4c00640 1 --2- 172.21.15.96:0/974750550 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f47a4004800 0x7f47a4004c80 unknown :-1 s=CLOSED pgs=1823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:26.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.661+0000 7f47c4c00640 1 -- 172.21.15.96:0/974750550 >> 172.21.15.96:0/974750550 conn(0x7f47c006b900 msgr2=0x7f47c006bce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:26.659 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.661+0000 7f47c4c00640 1 -- 172.21.15.96:0/974750550 shutdown_connections 2024-05-28T06:22:26.659 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.661+0000 7f47c4c00640 1 -- 172.21.15.96:0/974750550 wait complete. 2024-05-28T06:22:26.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.663+0000 7f47c4c00640 1 Processor -- start 2024-05-28T06:22:26.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.663+0000 7f47c4c00640 1 -- start start 2024-05-28T06:22:26.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.664+0000 7f47c4c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f47a4004800 0x7f47c0147f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:26.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.664+0000 7f47c4c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f47c015a7c0 0x7f47c0156870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:26.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.664+0000 7f47c4c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f47c015b6d0 0x7f47c015bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:26.661 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.664+0000 7f47c4c00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f47c00afcd0 con 0x7f47c015b6d0 2024-05-28T06:22:26.661 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.664+0000 7f47c4c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f47c00afb50 con 0x7f47a4004800 2024-05-28T06:22:26.661 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.664+0000 7f47c4c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f47c00b4cc0 con 0x7f47c015a7c0 2024-05-28T06:22:26.661 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.664+0000 7f47bfe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f47c015b6d0 0x7f47c015bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:26.661 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.664+0000 7f47bfe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f47c015b6d0 0x7f47c015bb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:60268/0 (socket says 172.21.15.96:60268) 2024-05-28T06:22:26.661 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.664+0000 7f47bfe00640 1 -- 172.21.15.96:0/3895667067 learned_addr learned my addr 172.21.15.96:0/3895667067 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:22:26.661 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.665+0000 7f47bf400640 1 --2- 172.21.15.96:0/3895667067 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f47a4004800 0x7f47c0147f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:26.661 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.665+0000 7f47bfe00640 1 -- 172.21.15.96:0/3895667067 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f47c015a7c0 msgr2=0x7f47c0156870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:26.661 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.665+0000 7f47bfe00640 1 --2- 172.21.15.96:0/3895667067 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f47c015a7c0 0x7f47c0156870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:26.661 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.665+0000 7f47bfe00640 1 -- 172.21.15.96:0/3895667067 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f47a4004800 msgr2=0x7f47c0147f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:26.661 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.665+0000 7f47bfe00640 1 --2- 172.21.15.96:0/3895667067 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f47a4004800 0x7f47c0147f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:26.661 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.665+0000 7f47bfe00640 1 -- 172.21.15.96:0/3895667067 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47c0157170 con 0x7f47c015b6d0 2024-05-28T06:22:26.661 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.665+0000 7f47bfe00640 1 --2- 172.21.15.96:0/3895667067 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f47c015b6d0 0x7f47c015bb50 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7f47c01591d0 tx=0x7f47b800d9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:26.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.665+0000 7f47b7400640 1 -- 172.21.15.96:0/3895667067 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47b8018070 con 0x7f47c015b6d0 2024-05-28T06:22:26.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.665+0000 7f47b7400640 1 -- 172.21.15.96:0/3895667067 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47b8003d80 con 0x7f47c015b6d0 2024-05-28T06:22:26.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.665+0000 7f47b7400640 1 -- 172.21.15.96:0/3895667067 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47b80052b0 con 0x7f47c015b6d0 2024-05-28T06:22:26.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.665+0000 7f47c4c00640 1 -- 172.21.15.96:0/3895667067 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47c0157450 con 0x7f47c015b6d0 2024-05-28T06:22:26.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.666+0000 7f47c4c00640 1 -- 172.21.15.96:0/3895667067 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47c0153db0 con 0x7f47c015b6d0 2024-05-28T06:22:26.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.667+0000 7f47c4c00640 1 -- 172.21.15.96:0/3895667067 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47c00689e0 con 0x7f47c015b6d0 2024-05-28T06:22:26.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.668+0000 7f47b7400640 1 -- 172.21.15.96:0/3895667067 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 37) ==== 93182+0+0 (secure 0 0 0) 0x7f47b801b070 con 0x7f47c015b6d0 2024-05-28T06:22:26.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.669+0000 7f47b7400640 1 --2- 172.21.15.96:0/3895667067 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f479c06e090 0x7f479c070550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:26.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.669+0000 7f47b7400640 1 -- 172.21.15.96:0/3895667067 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 5765+0+0 (secure 0 0 0) 0x7f47b808a030 con 0x7f47c015b6d0 2024-05-28T06:22:26.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.669+0000 7f47bf400640 1 --2- 172.21.15.96:0/3895667067 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f479c06e090 0x7f479c070550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:26.666 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.671+0000 7f47bf400640 1 --2- 172.21.15.96:0/3895667067 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f479c06e090 0x7f479c070550 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f47b0001050 tx=0x7f47b0000fe0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:26.669 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.673+0000 7f47b7400640 1 -- 172.21.15.96:0/3895667067 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f47b8059ec0 con 0x7f47c015b6d0 2024-05-28T06:22:26.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.867+0000 7f47c4c00640 1 -- 172.21.15.96:0/3895667067 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f47c00b0130 con 0x7f479c06e090 2024-05-28T06:22:26.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.887+0000 7f47b7400640 1 -- 172.21.15.96:0/3895667067 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252410 (secure 0 0 0) 0x7f47c00b0130 con 0x7f479c06e090 2024-05-28T06:22:26.886 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:22:26.887 INFO:teuthology.orchestra.run.smithi096.stderr:dumped all 2024-05-28T06:22:26.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.893+0000 7f47c4c00640 1 -- 172.21.15.96:0/3895667067 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f479c06e090 msgr2=0x7f479c070550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:26.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.893+0000 7f47c4c00640 1 --2- 172.21.15.96:0/3895667067 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f479c06e090 0x7f479c070550 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f47b0001050 tx=0x7f47b0000fe0 comp rx=0 tx=0).stop 2024-05-28T06:22:26.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.893+0000 7f47c4c00640 1 -- 172.21.15.96:0/3895667067 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f47c015b6d0 msgr2=0x7f47c015bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:26.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.893+0000 7f47c4c00640 1 --2- 172.21.15.96:0/3895667067 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f47c015b6d0 0x7f47c015bb50 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7f47c01591d0 tx=0x7f47b800d9d0 comp rx=0 tx=0).stop 2024-05-28T06:22:26.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.893+0000 7f47c4c00640 1 -- 172.21.15.96:0/3895667067 shutdown_connections 2024-05-28T06:22:26.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.893+0000 7f47c4c00640 1 --2- 172.21.15.96:0/3895667067 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f479c06e090 0x7f479c070550 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:26.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.893+0000 7f47c4c00640 1 --2- 172.21.15.96:0/3895667067 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f47c015b6d0 0x7f47c015bb50 unknown :-1 s=CLOSED pgs=1824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:26.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.893+0000 7f47c4c00640 1 --2- 172.21.15.96:0/3895667067 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f47c015a7c0 0x7f47c0156870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:26.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.893+0000 7f47c4c00640 1 --2- 172.21.15.96:0/3895667067 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f47a4004800 0x7f47c0147f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:26.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.893+0000 7f47c4c00640 1 -- 172.21.15.96:0/3895667067 >> 172.21.15.96:0/3895667067 conn(0x7f47c006b900 msgr2=0x7f47c015d380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:26.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.893+0000 7f47c4c00640 1 -- 172.21.15.96:0/3895667067 shutdown_connections 2024-05-28T06:22:26.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:26.893+0000 7f47c4c00640 1 -- 172.21.15.96:0/3895667067 wait complete. 2024-05-28T06:22:26.908 INFO:teuthology.orchestra.run.smithi096.stdout:{"pg_ready":true,"pg_map":{"version":661,"stamp":"2024-05-28T06:22:25.966467+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":895772,"kb_used_data":12392,"kb_used_omap":12,"kb_used_meta":883251,"kb_avail":754078948,"statfs":{"total":773094113280,"available":772176842752,"internally_reserved":0,"allocated":12689408,"data_stored":6195724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":904449584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.996505"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.242686+0000","last_change":"2024-05-28T06:22:17.004588+0000","last_active":"2024-05-28T06:22:18.242686+0000","last_peered":"2024-05-28T06:22:18.242686+0000","last_clean":"2024-05-28T06:22:18.242686+0000","last_became_active":"2024-05-28T06:22:17.003523+0000","last_became_peered":"2024-05-28T06:22:17.003523+0000","last_unstale":"2024-05-28T06:22:18.242686+0000","last_undegraded":"2024-05-28T06:22:18.242686+0000","last_fullsized":"2024-05-28T06:22:18.242686+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:04:35.674926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.001575+0000","last_change":"2024-05-28T06:22:16.997146+0000","last_active":"2024-05-28T06:22:18.001575+0000","last_peered":"2024-05-28T06:22:18.001575+0000","last_clean":"2024-05-28T06:22:18.001575+0000","last_became_active":"2024-05-28T06:22:16.995737+0000","last_became_peered":"2024-05-28T06:22:16.995737+0000","last_unstale":"2024-05-28T06:22:18.001575+0000","last_undegraded":"2024-05-28T06:22:18.001575+0000","last_fullsized":"2024-05-28T06:22:18.001575+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:38:34.851302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.528142+0000","last_change":"2024-05-28T06:22:17.006833+0000","last_active":"2024-05-28T06:22:18.528142+0000","last_peered":"2024-05-28T06:22:18.528142+0000","last_clean":"2024-05-28T06:22:18.528142+0000","last_became_active":"2024-05-28T06:22:17.006501+0000","last_became_peered":"2024-05-28T06:22:17.006501+0000","last_unstale":"2024-05-28T06:22:18.528142+0000","last_undegraded":"2024-05-28T06:22:18.528142+0000","last_fullsized":"2024-05-28T06:22:18.528142+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:23:42.240172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.242843+0000","last_change":"2024-05-28T06:22:16.994809+0000","last_active":"2024-05-28T06:22:18.242843+0000","last_peered":"2024-05-28T06:22:18.242843+0000","last_clean":"2024-05-28T06:22:18.242843+0000","last_became_active":"2024-05-28T06:22:16.994548+0000","last_became_peered":"2024-05-28T06:22:16.994548+0000","last_unstale":"2024-05-28T06:22:18.242843+0000","last_undegraded":"2024-05-28T06:22:18.242843+0000","last_fullsized":"2024-05-28T06:22:18.242843+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:59:36.297627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:17.999386+0000","last_change":"2024-05-28T06:22:16.996834+0000","last_active":"2024-05-28T06:22:17.999386+0000","last_peered":"2024-05-28T06:22:17.999386+0000","last_clean":"2024-05-28T06:22:17.999386+0000","last_became_active":"2024-05-28T06:22:16.996543+0000","last_became_peered":"2024-05-28T06:22:16.996543+0000","last_unstale":"2024-05-28T06:22:17.999386+0000","last_undegraded":"2024-05-28T06:22:17.999386+0000","last_fullsized":"2024-05-28T06:22:17.999386+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:23:11.730784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.528701+0000","last_change":"2024-05-28T06:22:16.995996+0000","last_active":"2024-05-28T06:22:18.528701+0000","last_peered":"2024-05-28T06:22:18.528701+0000","last_clean":"2024-05-28T06:22:18.528701+0000","last_became_active":"2024-05-28T06:22:16.995714+0000","last_became_peered":"2024-05-28T06:22:16.995714+0000","last_unstale":"2024-05-28T06:22:18.528701+0000","last_undegraded":"2024-05-28T06:22:18.528701+0000","last_fullsized":"2024-05-28T06:22:18.528701+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:04:42.043002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.024311+0000","last_change":"2024-05-28T06:22:17.006351+0000","last_active":"2024-05-28T06:22:18.024311+0000","last_peered":"2024-05-28T06:22:18.024311+0000","last_clean":"2024-05-28T06:22:18.024311+0000","last_became_active":"2024-05-28T06:22:17.005644+0000","last_became_peered":"2024-05-28T06:22:17.005644+0000","last_unstale":"2024-05-28T06:22:18.024311+0000","last_undegraded":"2024-05-28T06:22:18.024311+0000","last_fullsized":"2024-05-28T06:22:18.024311+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:21:19.908131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.527537+0000","last_change":"2024-05-28T06:22:16.998140+0000","last_active":"2024-05-28T06:22:18.527537+0000","last_peered":"2024-05-28T06:22:18.527537+0000","last_clean":"2024-05-28T06:22:18.527537+0000","last_became_active":"2024-05-28T06:22:16.997796+0000","last_became_peered":"2024-05-28T06:22:16.997796+0000","last_unstale":"2024-05-28T06:22:18.527537+0000","last_undegraded":"2024-05-28T06:22:18.527537+0000","last_fullsized":"2024-05-28T06:22:18.527537+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:06:13.719079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.242476+0000","last_change":"2024-05-28T06:22:17.003545+0000","last_active":"2024-05-28T06:22:18.242476+0000","last_peered":"2024-05-28T06:22:18.242476+0000","last_clean":"2024-05-28T06:22:18.242476+0000","last_became_active":"2024-05-28T06:22:17.001251+0000","last_became_peered":"2024-05-28T06:22:17.001251+0000","last_unstale":"2024-05-28T06:22:18.242476+0000","last_undegraded":"2024-05-28T06:22:18.242476+0000","last_fullsized":"2024-05-28T06:22:18.242476+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:38:50.761814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.243297+0000","last_change":"2024-05-28T06:22:17.001134+0000","last_active":"2024-05-28T06:22:18.243297+0000","last_peered":"2024-05-28T06:22:18.243297+0000","last_clean":"2024-05-28T06:22:18.243297+0000","last_became_active":"2024-05-28T06:22:17.000345+0000","last_became_peered":"2024-05-28T06:22:17.000345+0000","last_unstale":"2024-05-28T06:22:18.243297+0000","last_undegraded":"2024-05-28T06:22:18.243297+0000","last_fullsized":"2024-05-28T06:22:18.243297+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:22:46.981586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.001735+0000","last_change":"2024-05-28T06:22:17.002601+0000","last_active":"2024-05-28T06:22:18.001735+0000","last_peered":"2024-05-28T06:22:18.001735+0000","last_clean":"2024-05-28T06:22:18.001735+0000","last_became_active":"2024-05-28T06:22:17.002340+0000","last_became_peered":"2024-05-28T06:22:17.002340+0000","last_unstale":"2024-05-28T06:22:18.001735+0000","last_undegraded":"2024-05-28T06:22:18.001735+0000","last_fullsized":"2024-05-28T06:22:18.001735+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:26:18.638055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.243231+0000","last_change":"2024-05-28T06:22:16.994222+0000","last_active":"2024-05-28T06:22:18.243231+0000","last_peered":"2024-05-28T06:22:18.243231+0000","last_clean":"2024-05-28T06:22:18.243231+0000","last_became_active":"2024-05-28T06:22:16.993674+0000","last_became_peered":"2024-05-28T06:22:16.993674+0000","last_unstale":"2024-05-28T06:22:18.243231+0000","last_undegraded":"2024-05-28T06:22:18.243231+0000","last_fullsized":"2024-05-28T06:22:18.243231+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:40:55.388301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.528122+0000","last_change":"2024-05-28T06:22:16.998228+0000","last_active":"2024-05-28T06:22:18.528122+0000","last_peered":"2024-05-28T06:22:18.528122+0000","last_clean":"2024-05-28T06:22:18.528122+0000","last_became_active":"2024-05-28T06:22:16.996238+0000","last_became_peered":"2024-05-28T06:22:16.996238+0000","last_unstale":"2024-05-28T06:22:18.528122+0000","last_undegraded":"2024-05-28T06:22:18.528122+0000","last_fullsized":"2024-05-28T06:22:18.528122+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:38:34.851302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.527501+0000","last_change":"2024-05-28T06:22:17.001665+0000","last_active":"2024-05-28T06:22:18.527501+0000","last_peered":"2024-05-28T06:22:18.527501+0000","last_clean":"2024-05-28T06:22:18.527501+0000","last_became_active":"2024-05-28T06:22:17.001304+0000","last_became_peered":"2024-05-28T06:22:17.001304+0000","last_unstale":"2024-05-28T06:22:18.527501+0000","last_undegraded":"2024-05-28T06:22:18.527501+0000","last_fullsized":"2024-05-28T06:22:18.527501+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:47:18.486697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.527922+0000","last_change":"2024-05-28T06:22:16.997664+0000","last_active":"2024-05-28T06:22:18.527922+0000","last_peered":"2024-05-28T06:22:18.527922+0000","last_clean":"2024-05-28T06:22:18.527922+0000","last_became_active":"2024-05-28T06:22:16.997387+0000","last_became_peered":"2024-05-28T06:22:16.997387+0000","last_unstale":"2024-05-28T06:22:18.527922+0000","last_undegraded":"2024-05-28T06:22:18.527922+0000","last_fullsized":"2024-05-28T06:22:18.527922+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:37:05.453277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.242046+0000","last_change":"2024-05-28T06:22:17.005071+0000","last_active":"2024-05-28T06:22:18.242046+0000","last_peered":"2024-05-28T06:22:18.242046+0000","last_clean":"2024-05-28T06:22:18.242046+0000","last_became_active":"2024-05-28T06:22:17.004078+0000","last_became_peered":"2024-05-28T06:22:17.004078+0000","last_unstale":"2024-05-28T06:22:18.242046+0000","last_undegraded":"2024-05-28T06:22:18.242046+0000","last_fullsized":"2024-05-28T06:22:18.242046+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:26:19.869612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.001928+0000","last_change":"2024-05-28T06:22:17.010577+0000","last_active":"2024-05-28T06:22:18.001928+0000","last_peered":"2024-05-28T06:22:18.001928+0000","last_clean":"2024-05-28T06:22:18.001928+0000","last_became_active":"2024-05-28T06:22:17.010281+0000","last_became_peered":"2024-05-28T06:22:17.010281+0000","last_unstale":"2024-05-28T06:22:18.001928+0000","last_undegraded":"2024-05-28T06:22:18.001928+0000","last_fullsized":"2024-05-28T06:22:18.001928+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:23:42.240172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.020188+0000","last_change":"2024-05-28T06:22:17.011422+0000","last_active":"2024-05-28T06:22:18.020188+0000","last_peered":"2024-05-28T06:22:18.020188+0000","last_clean":"2024-05-28T06:22:18.020188+0000","last_became_active":"2024-05-28T06:22:17.010224+0000","last_became_peered":"2024-05-28T06:22:17.010224+0000","last_unstale":"2024-05-28T06:22:18.020188+0000","last_undegraded":"2024-05-28T06:22:18.020188+0000","last_fullsized":"2024-05-28T06:22:18.020188+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:11:13.465001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.528617+0000","last_change":"2024-05-28T06:22:17.002306+0000","last_active":"2024-05-28T06:22:18.528617+0000","last_peered":"2024-05-28T06:22:18.528617+0000","last_clean":"2024-05-28T06:22:18.528617+0000","last_became_active":"2024-05-28T06:22:17.001965+0000","last_became_peered":"2024-05-28T06:22:17.001965+0000","last_unstale":"2024-05-28T06:22:18.528617+0000","last_undegraded":"2024-05-28T06:22:18.528617+0000","last_fullsized":"2024-05-28T06:22:18.528617+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:16:11.794755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.000927+0000","last_change":"2024-05-28T06:22:16.998168+0000","last_active":"2024-05-28T06:22:18.000927+0000","last_peered":"2024-05-28T06:22:18.000927+0000","last_clean":"2024-05-28T06:22:18.000927+0000","last_became_active":"2024-05-28T06:22:16.996215+0000","last_became_peered":"2024-05-28T06:22:16.996215+0000","last_unstale":"2024-05-28T06:22:18.000927+0000","last_undegraded":"2024-05-28T06:22:18.000927+0000","last_fullsized":"2024-05-28T06:22:18.000927+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:06:13.719079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.001186+0000","last_change":"2024-05-28T06:22:16.998348+0000","last_active":"2024-05-28T06:22:18.001186+0000","last_peered":"2024-05-28T06:22:18.001186+0000","last_clean":"2024-05-28T06:22:18.001186+0000","last_became_active":"2024-05-28T06:22:16.998024+0000","last_became_peered":"2024-05-28T06:22:16.998024+0000","last_unstale":"2024-05-28T06:22:18.001186+0000","last_undegraded":"2024-05-28T06:22:18.001186+0000","last_fullsized":"2024-05-28T06:22:18.001186+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:13:14.889721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.528790+0000","last_change":"2024-05-28T06:22:17.004708+0000","last_active":"2024-05-28T06:22:18.528790+0000","last_peered":"2024-05-28T06:22:18.528790+0000","last_clean":"2024-05-28T06:22:18.528790+0000","last_became_active":"2024-05-28T06:22:17.003676+0000","last_became_peered":"2024-05-28T06:22:17.003676+0000","last_unstale":"2024-05-28T06:22:18.528790+0000","last_undegraded":"2024-05-28T06:22:18.528790+0000","last_fullsized":"2024-05-28T06:22:18.528790+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:38:34.851302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.000139+0000","last_change":"2024-05-28T06:22:16.999079+0000","last_active":"2024-05-28T06:22:18.000139+0000","last_peered":"2024-05-28T06:22:18.000139+0000","last_clean":"2024-05-28T06:22:18.000139+0000","last_became_active":"2024-05-28T06:22:16.998795+0000","last_became_peered":"2024-05-28T06:22:16.998795+0000","last_unstale":"2024-05-28T06:22:18.000139+0000","last_undegraded":"2024-05-28T06:22:18.000139+0000","last_fullsized":"2024-05-28T06:22:18.000139+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:17:14.655934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.526910+0000","last_change":"2024-05-28T06:22:17.008315+0000","last_active":"2024-05-28T06:22:18.526910+0000","last_peered":"2024-05-28T06:22:18.526910+0000","last_clean":"2024-05-28T06:22:18.526910+0000","last_became_active":"2024-05-28T06:22:17.007995+0000","last_became_peered":"2024-05-28T06:22:17.007995+0000","last_unstale":"2024-05-28T06:22:18.526910+0000","last_undegraded":"2024-05-28T06:22:18.526910+0000","last_fullsized":"2024-05-28T06:22:18.526910+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T18:02:32.237669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.002108+0000","last_change":"2024-05-28T06:22:17.003291+0000","last_active":"2024-05-28T06:22:18.002108+0000","last_peered":"2024-05-28T06:22:18.002108+0000","last_clean":"2024-05-28T06:22:18.002108+0000","last_became_active":"2024-05-28T06:22:17.003018+0000","last_became_peered":"2024-05-28T06:22:17.003018+0000","last_unstale":"2024-05-28T06:22:18.002108+0000","last_undegraded":"2024-05-28T06:22:18.002108+0000","last_fullsized":"2024-05-28T06:22:18.002108+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:16:11.794755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.243265+0000","last_change":"2024-05-28T06:22:17.006855+0000","last_active":"2024-05-28T06:22:18.243265+0000","last_peered":"2024-05-28T06:22:18.243265+0000","last_clean":"2024-05-28T06:22:18.243265+0000","last_became_active":"2024-05-28T06:22:17.006649+0000","last_became_peered":"2024-05-28T06:22:17.006649+0000","last_unstale":"2024-05-28T06:22:18.243265+0000","last_undegraded":"2024-05-28T06:22:18.243265+0000","last_fullsized":"2024-05-28T06:22:18.243265+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:25:34.876194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.529987+0000","last_change":"2024-05-28T06:22:17.000165+0000","last_active":"2024-05-28T06:22:18.529987+0000","last_peered":"2024-05-28T06:22:18.529987+0000","last_clean":"2024-05-28T06:22:18.529987+0000","last_became_active":"2024-05-28T06:22:16.999904+0000","last_became_peered":"2024-05-28T06:22:16.999904+0000","last_unstale":"2024-05-28T06:22:18.529987+0000","last_undegraded":"2024-05-28T06:22:18.529987+0000","last_fullsized":"2024-05-28T06:22:18.529987+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:58:15.396668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.529632+0000","last_change":"2024-05-28T06:22:16.993958+0000","last_active":"2024-05-28T06:22:18.529632+0000","last_peered":"2024-05-28T06:22:18.529632+0000","last_clean":"2024-05-28T06:22:18.529632+0000","last_became_active":"2024-05-28T06:22:16.993327+0000","last_became_peered":"2024-05-28T06:22:16.993327+0000","last_unstale":"2024-05-28T06:22:18.529632+0000","last_undegraded":"2024-05-28T06:22:18.529632+0000","last_fullsized":"2024-05-28T06:22:18.529632+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:26:18.638055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.000269+0000","last_change":"2024-05-28T06:22:17.001399+0000","last_active":"2024-05-28T06:22:18.000269+0000","last_peered":"2024-05-28T06:22:18.000269+0000","last_clean":"2024-05-28T06:22:18.000269+0000","last_became_active":"2024-05-28T06:22:17.000579+0000","last_became_peered":"2024-05-28T06:22:17.000579+0000","last_unstale":"2024-05-28T06:22:18.000269+0000","last_undegraded":"2024-05-28T06:22:18.000269+0000","last_fullsized":"2024-05-28T06:22:18.000269+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:48:14.529168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.024533+0000","last_change":"2024-05-28T06:22:17.004330+0000","last_active":"2024-05-28T06:22:18.024533+0000","last_peered":"2024-05-28T06:22:18.024533+0000","last_clean":"2024-05-28T06:22:18.024533+0000","last_became_active":"2024-05-28T06:22:16.996955+0000","last_became_peered":"2024-05-28T06:22:16.996955+0000","last_unstale":"2024-05-28T06:22:18.024533+0000","last_undegraded":"2024-05-28T06:22:18.024533+0000","last_fullsized":"2024-05-28T06:22:18.024533+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:25:57.432011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.242168+0000","last_change":"2024-05-28T06:22:17.001908+0000","last_active":"2024-05-28T06:22:18.242168+0000","last_peered":"2024-05-28T06:22:18.242168+0000","last_clean":"2024-05-28T06:22:18.242168+0000","last_became_active":"2024-05-28T06:22:17.000543+0000","last_became_peered":"2024-05-28T06:22:17.000543+0000","last_unstale":"2024-05-28T06:22:18.242168+0000","last_undegraded":"2024-05-28T06:22:18.242168+0000","last_fullsized":"2024-05-28T06:22:18.242168+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:12:55.159821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.527587+0000","last_change":"2024-05-28T06:22:16.992715+0000","last_active":"2024-05-28T06:22:18.527587+0000","last_peered":"2024-05-28T06:22:18.527587+0000","last_clean":"2024-05-28T06:22:18.527587+0000","last_became_active":"2024-05-28T06:22:16.992094+0000","last_became_peered":"2024-05-28T06:22:16.992094+0000","last_unstale":"2024-05-28T06:22:18.527587+0000","last_undegraded":"2024-05-28T06:22:18.527587+0000","last_fullsized":"2024-05-28T06:22:18.527587+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:05:25.661249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.529840+0000","last_change":"2024-05-28T06:22:17.004255+0000","last_active":"2024-05-28T06:22:18.529840+0000","last_peered":"2024-05-28T06:22:18.529840+0000","last_clean":"2024-05-28T06:22:18.529840+0000","last_became_active":"2024-05-28T06:22:17.001691+0000","last_became_peered":"2024-05-28T06:22:17.001691+0000","last_unstale":"2024-05-28T06:22:18.529840+0000","last_undegraded":"2024-05-28T06:22:18.529840+0000","last_fullsized":"2024-05-28T06:22:18.529840+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:40:20.104894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"114'4","reported_seq":18,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.024524+0000","last_change":"2024-05-28T06:22:15.992751+0000","last_active":"2024-05-28T06:22:18.024524+0000","last_peered":"2024-05-28T06:22:18.024524+0000","last_clean":"2024-05-28T06:22:18.024524+0000","last_became_active":"2024-05-28T06:22:15.992113+0000","last_became_peered":"2024-05-28T06:22:15.992113+0000","last_unstale":"2024-05-28T06:22:18.024524+0000","last_undegraded":"2024-05-28T06:22:18.024524+0000","last_fullsized":"2024-05-28T06:22:18.024524+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:28:16.129726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.020384+0000","last_change":"2024-05-28T06:22:17.003592+0000","last_active":"2024-05-28T06:22:18.020384+0000","last_peered":"2024-05-28T06:22:18.020384+0000","last_clean":"2024-05-28T06:22:18.020384+0000","last_became_active":"2024-05-28T06:22:17.002423+0000","last_became_peered":"2024-05-28T06:22:17.002423+0000","last_unstale":"2024-05-28T06:22:18.020384+0000","last_undegraded":"2024-05-28T06:22:18.020384+0000","last_fullsized":"2024-05-28T06:22:18.020384+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:58:35.663956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"114'1","reported_seq":15,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.528549+0000","last_change":"2024-05-28T06:22:15.996401+0000","last_active":"2024-05-28T06:22:18.528549+0000","last_peered":"2024-05-28T06:22:18.528549+0000","last_clean":"2024-05-28T06:22:18.528549+0000","last_became_active":"2024-05-28T06:22:15.996039+0000","last_became_peered":"2024-05-28T06:22:15.996039+0000","last_unstale":"2024-05-28T06:22:18.528549+0000","last_undegraded":"2024-05-28T06:22:18.528549+0000","last_fullsized":"2024-05-28T06:22:18.528549+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:43:46.372846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.528998+0000","last_change":"2024-05-28T06:22:16.999053+0000","last_active":"2024-05-28T06:22:18.528998+0000","last_peered":"2024-05-28T06:22:18.528998+0000","last_clean":"2024-05-28T06:22:18.528998+0000","last_became_active":"2024-05-28T06:22:16.995589+0000","last_became_peered":"2024-05-28T06:22:16.995589+0000","last_unstale":"2024-05-28T06:22:18.528998+0000","last_undegraded":"2024-05-28T06:22:18.528998+0000","last_fullsized":"2024-05-28T06:22:18.528998+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:58:42.572818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"114'2","reported_seq":16,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:17.999806+0000","last_change":"2024-05-28T06:22:15.989244+0000","last_active":"2024-05-28T06:22:17.999806+0000","last_peered":"2024-05-28T06:22:17.999806+0000","last_clean":"2024-05-28T06:22:17.999806+0000","last_became_active":"2024-05-28T06:22:15.988137+0000","last_became_peered":"2024-05-28T06:22:15.988137+0000","last_unstale":"2024-05-28T06:22:17.999806+0000","last_undegraded":"2024-05-28T06:22:17.999806+0000","last_fullsized":"2024-05-28T06:22:17.999806+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T18:10:18.669151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.528637+0000","last_change":"2024-05-28T06:22:17.001702+0000","last_active":"2024-05-28T06:22:18.528637+0000","last_peered":"2024-05-28T06:22:18.528637+0000","last_clean":"2024-05-28T06:22:18.528637+0000","last_became_active":"2024-05-28T06:22:16.999756+0000","last_became_peered":"2024-05-28T06:22:16.999756+0000","last_unstale":"2024-05-28T06:22:18.528637+0000","last_undegraded":"2024-05-28T06:22:18.528637+0000","last_fullsized":"2024-05-28T06:22:18.528637+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:23:29.259681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"114'2","reported_seq":16,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:17.999369+0000","last_change":"2024-05-28T06:22:15.992414+0000","last_active":"2024-05-28T06:22:17.999369+0000","last_peered":"2024-05-28T06:22:17.999369+0000","last_clean":"2024-05-28T06:22:17.999369+0000","last_became_active":"2024-05-28T06:22:15.991608+0000","last_became_peered":"2024-05-28T06:22:15.991608+0000","last_unstale":"2024-05-28T06:22:17.999369+0000","last_undegraded":"2024-05-28T06:22:17.999369+0000","last_fullsized":"2024-05-28T06:22:17.999369+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:42:21.601868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.241943+0000","last_change":"2024-05-28T06:22:16.995582+0000","last_active":"2024-05-28T06:22:18.241943+0000","last_peered":"2024-05-28T06:22:18.241943+0000","last_clean":"2024-05-28T06:22:18.241943+0000","last_became_active":"2024-05-28T06:22:16.995242+0000","last_became_peered":"2024-05-28T06:22:16.995242+0000","last_unstale":"2024-05-28T06:22:18.241943+0000","last_undegraded":"2024-05-28T06:22:18.241943+0000","last_fullsized":"2024-05-28T06:22:18.241943+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:46:21.232752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"114'1","reported_seq":15,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.024104+0000","last_change":"2024-05-28T06:22:15.994783+0000","last_active":"2024-05-28T06:22:18.024104+0000","last_peered":"2024-05-28T06:22:18.024104+0000","last_clean":"2024-05-28T06:22:18.024104+0000","last_became_active":"2024-05-28T06:22:15.994208+0000","last_became_peered":"2024-05-28T06:22:15.994208+0000","last_unstale":"2024-05-28T06:22:18.024104+0000","last_undegraded":"2024-05-28T06:22:18.024104+0000","last_fullsized":"2024-05-28T06:22:18.024104+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:04:41.031193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.000748+0000","last_change":"2024-05-28T06:22:16.992756+0000","last_active":"2024-05-28T06:22:18.000748+0000","last_peered":"2024-05-28T06:22:18.000748+0000","last_clean":"2024-05-28T06:22:18.000748+0000","last_became_active":"2024-05-28T06:22:16.992331+0000","last_became_peered":"2024-05-28T06:22:16.992331+0000","last_unstale":"2024-05-28T06:22:18.000748+0000","last_undegraded":"2024-05-28T06:22:18.000748+0000","last_fullsized":"2024-05-28T06:22:18.000748+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:08:39.984746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"114'1","reported_seq":15,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.529094+0000","last_change":"2024-05-28T06:22:15.990828+0000","last_active":"2024-05-28T06:22:18.529094+0000","last_peered":"2024-05-28T06:22:18.529094+0000","last_clean":"2024-05-28T06:22:18.529094+0000","last_became_active":"2024-05-28T06:22:15.990537+0000","last_became_peered":"2024-05-28T06:22:15.990537+0000","last_unstale":"2024-05-28T06:22:18.529094+0000","last_undegraded":"2024-05-28T06:22:18.529094+0000","last_fullsized":"2024-05-28T06:22:18.529094+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:51:43.927844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.241796+0000","last_change":"2024-05-28T06:22:17.000804+0000","last_active":"2024-05-28T06:22:18.241796+0000","last_peered":"2024-05-28T06:22:18.241796+0000","last_clean":"2024-05-28T06:22:18.241796+0000","last_became_active":"2024-05-28T06:22:16.998428+0000","last_became_peered":"2024-05-28T06:22:16.998428+0000","last_unstale":"2024-05-28T06:22:18.241796+0000","last_undegraded":"2024-05-28T06:22:18.241796+0000","last_fullsized":"2024-05-28T06:22:18.241796+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:43:26.170749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.000006+0000","last_change":"2024-05-28T06:22:15.981969+0000","last_active":"2024-05-28T06:22:18.000006+0000","last_peered":"2024-05-28T06:22:18.000006+0000","last_clean":"2024-05-28T06:22:18.000006+0000","last_became_active":"2024-05-28T06:22:15.979073+0000","last_became_peered":"2024-05-28T06:22:15.979073+0000","last_unstale":"2024-05-28T06:22:18.000006+0000","last_undegraded":"2024-05-28T06:22:18.000006+0000","last_fullsized":"2024-05-28T06:22:18.000006+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:41:39.601468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.024757+0000","last_change":"2024-05-28T06:22:17.006594+0000","last_active":"2024-05-28T06:22:18.024757+0000","last_peered":"2024-05-28T06:22:18.024757+0000","last_clean":"2024-05-28T06:22:18.024757+0000","last_became_active":"2024-05-28T06:22:17.006057+0000","last_became_peered":"2024-05-28T06:22:17.006057+0000","last_unstale":"2024-05-28T06:22:18.024757+0000","last_undegraded":"2024-05-28T06:22:18.024757+0000","last_fullsized":"2024-05-28T06:22:18.024757+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:39:44.471675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.242254+0000","last_change":"2024-05-28T06:22:15.988357+0000","last_active":"2024-05-28T06:22:18.242254+0000","last_peered":"2024-05-28T06:22:18.242254+0000","last_clean":"2024-05-28T06:22:18.242254+0000","last_became_active":"2024-05-28T06:22:15.988053+0000","last_became_peered":"2024-05-28T06:22:15.988053+0000","last_unstale":"2024-05-28T06:22:18.242254+0000","last_undegraded":"2024-05-28T06:22:18.242254+0000","last_fullsized":"2024-05-28T06:22:18.242254+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:24:37.584832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.024688+0000","last_change":"2024-05-28T06:22:17.003420+0000","last_active":"2024-05-28T06:22:18.024688+0000","last_peered":"2024-05-28T06:22:18.024688+0000","last_clean":"2024-05-28T06:22:18.024688+0000","last_became_active":"2024-05-28T06:22:17.003005+0000","last_became_peered":"2024-05-28T06:22:17.003005+0000","last_unstale":"2024-05-28T06:22:18.024688+0000","last_undegraded":"2024-05-28T06:22:18.024688+0000","last_fullsized":"2024-05-28T06:22:18.024688+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:26:50.855556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:17.999693+0000","last_change":"2024-05-28T06:22:15.984544+0000","last_active":"2024-05-28T06:22:17.999693+0000","last_peered":"2024-05-28T06:22:17.999693+0000","last_clean":"2024-05-28T06:22:17.999693+0000","last_became_active":"2024-05-28T06:22:15.984287+0000","last_became_peered":"2024-05-28T06:22:15.984287+0000","last_unstale":"2024-05-28T06:22:17.999693+0000","last_undegraded":"2024-05-28T06:22:17.999693+0000","last_fullsized":"2024-05-28T06:22:17.999693+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T18:04:33.437207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.020450+0000","last_change":"2024-05-28T06:22:17.012046+0000","last_active":"2024-05-28T06:22:18.020450+0000","last_peered":"2024-05-28T06:22:18.020450+0000","last_clean":"2024-05-28T06:22:18.020450+0000","last_became_active":"2024-05-28T06:22:17.007050+0000","last_became_peered":"2024-05-28T06:22:17.007050+0000","last_unstale":"2024-05-28T06:22:18.020450+0000","last_undegraded":"2024-05-28T06:22:18.020450+0000","last_fullsized":"2024-05-28T06:22:18.020450+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:48:14.529168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.529413+0000","last_change":"2024-05-28T06:22:15.989398+0000","last_active":"2024-05-28T06:22:18.529413+0000","last_peered":"2024-05-28T06:22:18.529413+0000","last_clean":"2024-05-28T06:22:18.529413+0000","last_became_active":"2024-05-28T06:22:15.989101+0000","last_became_peered":"2024-05-28T06:22:15.989101+0000","last_unstale":"2024-05-28T06:22:18.529413+0000","last_undegraded":"2024-05-28T06:22:18.529413+0000","last_fullsized":"2024-05-28T06:22:18.529413+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:43:46.372846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.529711+0000","last_change":"2024-05-28T06:22:16.993645+0000","last_active":"2024-05-28T06:22:18.529711+0000","last_peered":"2024-05-28T06:22:18.529711+0000","last_clean":"2024-05-28T06:22:18.529711+0000","last_became_active":"2024-05-28T06:22:16.992912+0000","last_became_peered":"2024-05-28T06:22:16.992912+0000","last_unstale":"2024-05-28T06:22:18.529711+0000","last_undegraded":"2024-05-28T06:22:18.529711+0000","last_fullsized":"2024-05-28T06:22:18.529711+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:16:11.794755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"114'3","reported_seq":17,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.242908+0000","last_change":"2024-05-28T06:22:15.998469+0000","last_active":"2024-05-28T06:22:18.242908+0000","last_peered":"2024-05-28T06:22:18.242908+0000","last_clean":"2024-05-28T06:22:18.242908+0000","last_became_active":"2024-05-28T06:22:15.998186+0000","last_became_peered":"2024-05-28T06:22:15.998186+0000","last_unstale":"2024-05-28T06:22:18.242908+0000","last_undegraded":"2024-05-28T06:22:18.242908+0000","last_fullsized":"2024-05-28T06:22:18.242908+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:23:41.228363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.024744+0000","last_change":"2024-05-28T06:22:17.009032+0000","last_active":"2024-05-28T06:22:18.024744+0000","last_peered":"2024-05-28T06:22:18.024744+0000","last_clean":"2024-05-28T06:22:18.024744+0000","last_became_active":"2024-05-28T06:22:17.007965+0000","last_became_peered":"2024-05-28T06:22:17.007965+0000","last_unstale":"2024-05-28T06:22:18.024744+0000","last_undegraded":"2024-05-28T06:22:18.024744+0000","last_fullsized":"2024-05-28T06:22:18.024744+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:13:32.315186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"114'6","reported_seq":20,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.243514+0000","last_change":"2024-05-28T06:22:15.997183+0000","last_active":"2024-05-28T06:22:18.243514+0000","last_peered":"2024-05-28T06:22:18.243514+0000","last_clean":"2024-05-28T06:22:18.243514+0000","last_became_active":"2024-05-28T06:22:15.996809+0000","last_became_peered":"2024-05-28T06:22:15.996809+0000","last_unstale":"2024-05-28T06:22:18.243514+0000","last_undegraded":"2024-05-28T06:22:18.243514+0000","last_fullsized":"2024-05-28T06:22:18.243514+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:38:33.839493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.528917+0000","last_change":"2024-05-28T06:22:17.000291+0000","last_active":"2024-05-28T06:22:18.528917+0000","last_peered":"2024-05-28T06:22:18.528917+0000","last_clean":"2024-05-28T06:22:18.528917+0000","last_became_active":"2024-05-28T06:22:16.999910+0000","last_became_peered":"2024-05-28T06:22:16.999910+0000","last_unstale":"2024-05-28T06:22:18.528917+0000","last_undegraded":"2024-05-28T06:22:18.528917+0000","last_fullsized":"2024-05-28T06:22:18.528917+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:23:42.240172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"114'2","reported_seq":16,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.528553+0000","last_change":"2024-05-28T06:22:15.996773+0000","last_active":"2024-05-28T06:22:18.528553+0000","last_peered":"2024-05-28T06:22:18.528553+0000","last_clean":"2024-05-28T06:22:18.528553+0000","last_became_active":"2024-05-28T06:22:15.995681+0000","last_became_peered":"2024-05-28T06:22:15.995681+0000","last_unstale":"2024-05-28T06:22:18.528553+0000","last_undegraded":"2024-05-28T06:22:18.528553+0000","last_fullsized":"2024-05-28T06:22:18.528553+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:48:09.956343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.528770+0000","last_change":"2024-05-28T06:22:17.003164+0000","last_active":"2024-05-28T06:22:18.528770+0000","last_peered":"2024-05-28T06:22:18.528770+0000","last_clean":"2024-05-28T06:22:18.528770+0000","last_became_active":"2024-05-28T06:22:17.002906+0000","last_became_peered":"2024-05-28T06:22:17.002906+0000","last_unstale":"2024-05-28T06:22:18.528770+0000","last_undegraded":"2024-05-28T06:22:18.528770+0000","last_fullsized":"2024-05-28T06:22:18.528770+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:13:14.889721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.020651+0000","last_change":"2024-05-28T06:22:15.993511+0000","last_active":"2024-05-28T06:22:18.020651+0000","last_peered":"2024-05-28T06:22:18.020651+0000","last_clean":"2024-05-28T06:22:18.020651+0000","last_became_active":"2024-05-28T06:22:15.992788+0000","last_became_peered":"2024-05-28T06:22:15.992788+0000","last_unstale":"2024-05-28T06:22:18.020651+0000","last_undegraded":"2024-05-28T06:22:18.020651+0000","last_fullsized":"2024-05-28T06:22:18.020651+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:43:46.372846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.527440+0000","last_change":"2024-05-28T06:22:16.991808+0000","last_active":"2024-05-28T06:22:18.527440+0000","last_peered":"2024-05-28T06:22:18.527440+0000","last_clean":"2024-05-28T06:22:18.527440+0000","last_became_active":"2024-05-28T06:22:16.991494+0000","last_became_peered":"2024-05-28T06:22:16.991494+0000","last_unstale":"2024-05-28T06:22:18.527440+0000","last_undegraded":"2024-05-28T06:22:18.527440+0000","last_fullsized":"2024-05-28T06:22:18.527440+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:47:18.486697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"114'8","reported_seq":22,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:17.999084+0000","last_change":"2024-05-28T06:22:15.991378+0000","last_active":"2024-05-28T06:22:17.999084+0000","last_peered":"2024-05-28T06:22:17.999084+0000","last_clean":"2024-05-28T06:22:17.999084+0000","last_became_active":"2024-05-28T06:22:15.990388+0000","last_became_peered":"2024-05-28T06:22:15.990388+0000","last_unstale":"2024-05-28T06:22:17.999084+0000","last_undegraded":"2024-05-28T06:22:17.999084+0000","last_fullsized":"2024-05-28T06:22:17.999084+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:49:10.819709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:17.999777+0000","last_change":"2024-05-28T06:22:16.991114+0000","last_active":"2024-05-28T06:22:17.999777+0000","last_peered":"2024-05-28T06:22:17.999777+0000","last_clean":"2024-05-28T06:22:17.999777+0000","last_became_active":"2024-05-28T06:22:16.990159+0000","last_became_peered":"2024-05-28T06:22:16.990159+0000","last_unstale":"2024-05-28T06:22:17.999777+0000","last_undegraded":"2024-05-28T06:22:17.999777+0000","last_fullsized":"2024-05-28T06:22:17.999777+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:18:01.989158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"114'1","reported_seq":15,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.242742+0000","last_change":"2024-05-28T06:22:15.997418+0000","last_active":"2024-05-28T06:22:18.242742+0000","last_peered":"2024-05-28T06:22:18.242742+0000","last_clean":"2024-05-28T06:22:18.242742+0000","last_became_active":"2024-05-28T06:22:15.995786+0000","last_became_peered":"2024-05-28T06:22:15.995786+0000","last_unstale":"2024-05-28T06:22:18.242742+0000","last_undegraded":"2024-05-28T06:22:18.242742+0000","last_fullsized":"2024-05-28T06:22:18.242742+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:50:11.148336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.530295+0000","last_change":"2024-05-28T06:22:16.993861+0000","last_active":"2024-05-28T06:22:18.530295+0000","last_peered":"2024-05-28T06:22:18.530295+0000","last_clean":"2024-05-28T06:22:18.530295+0000","last_became_active":"2024-05-28T06:22:16.993602+0000","last_became_peered":"2024-05-28T06:22:16.993602+0000","last_unstale":"2024-05-28T06:22:18.530295+0000","last_undegraded":"2024-05-28T06:22:18.530295+0000","last_fullsized":"2024-05-28T06:22:18.530295+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:28:17.141535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"114'2","reported_seq":16,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:17.999975+0000","last_change":"2024-05-28T06:22:15.980689+0000","last_active":"2024-05-28T06:22:17.999975+0000","last_peered":"2024-05-28T06:22:17.999975+0000","last_clean":"2024-05-28T06:22:17.999975+0000","last_became_active":"2024-05-28T06:22:15.980139+0000","last_became_peered":"2024-05-28T06:22:15.980139+0000","last_unstale":"2024-05-28T06:22:17.999975+0000","last_undegraded":"2024-05-28T06:22:17.999975+0000","last_fullsized":"2024-05-28T06:22:17.999975+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:19:21.410923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.528248+0000","last_change":"2024-05-28T06:22:16.999644+0000","last_active":"2024-05-28T06:22:18.528248+0000","last_peered":"2024-05-28T06:22:18.528248+0000","last_clean":"2024-05-28T06:22:18.528248+0000","last_became_active":"2024-05-28T06:22:16.998935+0000","last_became_peered":"2024-05-28T06:22:16.998935+0000","last_unstale":"2024-05-28T06:22:18.528248+0000","last_undegraded":"2024-05-28T06:22:18.528248+0000","last_fullsized":"2024-05-28T06:22:18.528248+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:26:18.638055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"114'1","reported_seq":15,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.019924+0000","last_change":"2024-05-28T06:22:15.996123+0000","last_active":"2024-05-28T06:22:18.019924+0000","last_peered":"2024-05-28T06:22:18.019924+0000","last_clean":"2024-05-28T06:22:18.019924+0000","last_became_active":"2024-05-28T06:22:15.995728+0000","last_became_peered":"2024-05-28T06:22:15.995728+0000","last_unstale":"2024-05-28T06:22:18.019924+0000","last_undegraded":"2024-05-28T06:22:18.019924+0000","last_fullsized":"2024-05-28T06:22:18.019924+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:08:38.270230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.528924+0000","last_change":"2024-05-28T06:22:17.003252+0000","last_active":"2024-05-28T06:22:18.528924+0000","last_peered":"2024-05-28T06:22:18.528924+0000","last_clean":"2024-05-28T06:22:18.528924+0000","last_became_active":"2024-05-28T06:22:17.001195+0000","last_became_peered":"2024-05-28T06:22:17.001195+0000","last_unstale":"2024-05-28T06:22:18.528924+0000","last_undegraded":"2024-05-28T06:22:18.528924+0000","last_fullsized":"2024-05-28T06:22:18.528924+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:27:26.116768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.024177+0000","last_change":"2024-05-28T06:22:15.998545+0000","last_active":"2024-05-28T06:22:18.024177+0000","last_peered":"2024-05-28T06:22:18.024177+0000","last_clean":"2024-05-28T06:22:18.024177+0000","last_became_active":"2024-05-28T06:22:15.997754+0000","last_became_peered":"2024-05-28T06:22:15.997754+0000","last_unstale":"2024-05-28T06:22:18.024177+0000","last_undegraded":"2024-05-28T06:22:18.024177+0000","last_fullsized":"2024-05-28T06:22:18.024177+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:58:41.561009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.527487+0000","last_change":"2024-05-28T06:22:17.000185+0000","last_active":"2024-05-28T06:22:18.527487+0000","last_peered":"2024-05-28T06:22:18.527487+0000","last_clean":"2024-05-28T06:22:18.527487+0000","last_became_active":"2024-05-28T06:22:16.997369+0000","last_became_peered":"2024-05-28T06:22:16.997369+0000","last_unstale":"2024-05-28T06:22:18.527487+0000","last_undegraded":"2024-05-28T06:22:18.527487+0000","last_fullsized":"2024-05-28T06:22:18.527487+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:08:39.984746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"114'2","reported_seq":16,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.528872+0000","last_change":"2024-05-28T06:22:15.991711+0000","last_active":"2024-05-28T06:22:18.528872+0000","last_peered":"2024-05-28T06:22:18.528872+0000","last_clean":"2024-05-28T06:22:18.528872+0000","last_became_active":"2024-05-28T06:22:15.991323+0000","last_became_peered":"2024-05-28T06:22:15.991323+0000","last_unstale":"2024-05-28T06:22:18.528872+0000","last_undegraded":"2024-05-28T06:22:18.528872+0000","last_fullsized":"2024-05-28T06:22:18.528872+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:02:22.717474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.242729+0000","last_change":"2024-05-28T06:22:17.003257+0000","last_active":"2024-05-28T06:22:18.242729+0000","last_peered":"2024-05-28T06:22:18.242729+0000","last_clean":"2024-05-28T06:22:18.242729+0000","last_became_active":"2024-05-28T06:22:17.000808+0000","last_became_peered":"2024-05-28T06:22:17.000808+0000","last_unstale":"2024-05-28T06:22:18.242729+0000","last_undegraded":"2024-05-28T06:22:18.242729+0000","last_fullsized":"2024-05-28T06:22:18.242729+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:27:48.276213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:17.999010+0000","last_change":"2024-05-28T06:22:15.994076+0000","last_active":"2024-05-28T06:22:17.999010+0000","last_peered":"2024-05-28T06:22:17.999010+0000","last_clean":"2024-05-28T06:22:17.999010+0000","last_became_active":"2024-05-28T06:22:15.993488+0000","last_became_peered":"2024-05-28T06:22:15.993488+0000","last_unstale":"2024-05-28T06:22:17.999010+0000","last_undegraded":"2024-05-28T06:22:17.999010+0000","last_fullsized":"2024-05-28T06:22:17.999010+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:08:35.517640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.527460+0000","last_change":"2024-05-28T06:22:16.995803+0000","last_active":"2024-05-28T06:22:18.527460+0000","last_peered":"2024-05-28T06:22:18.527460+0000","last_clean":"2024-05-28T06:22:18.527460+0000","last_became_active":"2024-05-28T06:22:16.995475+0000","last_became_peered":"2024-05-28T06:22:16.995475+0000","last_unstale":"2024-05-28T06:22:18.527460+0000","last_undegraded":"2024-05-28T06:22:18.527460+0000","last_fullsized":"2024-05-28T06:22:18.527460+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T18:02:32.237669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.527971+0000","last_change":"2024-05-28T06:22:15.989588+0000","last_active":"2024-05-28T06:22:18.527971+0000","last_peered":"2024-05-28T06:22:18.527971+0000","last_clean":"2024-05-28T06:22:18.527971+0000","last_became_active":"2024-05-28T06:22:15.989331+0000","last_became_peered":"2024-05-28T06:22:15.989331+0000","last_unstale":"2024-05-28T06:22:18.527971+0000","last_undegraded":"2024-05-28T06:22:18.527971+0000","last_fullsized":"2024-05-28T06:22:18.527971+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:19:21.410923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.527956+0000","last_change":"2024-05-28T06:22:16.993785+0000","last_active":"2024-05-28T06:22:18.527956+0000","last_peered":"2024-05-28T06:22:18.527956+0000","last_clean":"2024-05-28T06:22:18.527956+0000","last_became_active":"2024-05-28T06:22:16.993402+0000","last_became_peered":"2024-05-28T06:22:16.993402+0000","last_unstale":"2024-05-28T06:22:18.527956+0000","last_undegraded":"2024-05-28T06:22:18.527956+0000","last_fullsized":"2024-05-28T06:22:18.527956+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:06:13.719079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.242342+0000","last_change":"2024-05-28T06:22:15.998948+0000","last_active":"2024-05-28T06:22:18.242342+0000","last_peered":"2024-05-28T06:22:18.242342+0000","last_clean":"2024-05-28T06:22:18.242342+0000","last_became_active":"2024-05-28T06:22:15.998205+0000","last_became_peered":"2024-05-28T06:22:15.998205+0000","last_unstale":"2024-05-28T06:22:18.242342+0000","last_undegraded":"2024-05-28T06:22:18.242342+0000","last_fullsized":"2024-05-28T06:22:18.242342+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:58:34.652147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.020433+0000","last_change":"2024-05-28T06:22:17.005113+0000","last_active":"2024-05-28T06:22:18.020433+0000","last_peered":"2024-05-28T06:22:18.020433+0000","last_clean":"2024-05-28T06:22:18.020433+0000","last_became_active":"2024-05-28T06:22:17.003020+0000","last_became_peered":"2024-05-28T06:22:17.003020+0000","last_unstale":"2024-05-28T06:22:18.020433+0000","last_undegraded":"2024-05-28T06:22:18.020433+0000","last_fullsized":"2024-05-28T06:22:18.020433+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:17:14.655934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"114'2","reported_seq":16,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.528934+0000","last_change":"2024-05-28T06:22:15.990864+0000","last_active":"2024-05-28T06:22:18.528934+0000","last_peered":"2024-05-28T06:22:18.528934+0000","last_clean":"2024-05-28T06:22:18.528934+0000","last_became_active":"2024-05-28T06:22:15.990559+0000","last_became_peered":"2024-05-28T06:22:15.990559+0000","last_unstale":"2024-05-28T06:22:18.528934+0000","last_undegraded":"2024-05-28T06:22:18.528934+0000","last_fullsized":"2024-05-28T06:22:18.528934+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:51:36.120307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.242633+0000","last_change":"2024-05-28T06:22:16.992471+0000","last_active":"2024-05-28T06:22:18.242633+0000","last_peered":"2024-05-28T06:22:18.242633+0000","last_clean":"2024-05-28T06:22:18.242633+0000","last_became_active":"2024-05-28T06:22:16.992135+0000","last_became_peered":"2024-05-28T06:22:16.992135+0000","last_unstale":"2024-05-28T06:22:18.242633+0000","last_undegraded":"2024-05-28T06:22:18.242633+0000","last_fullsized":"2024-05-28T06:22:18.242633+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:57:13.651588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"115'1","reported_seq":15,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.020608+0000","last_change":"2024-05-28T06:22:15.994161+0000","last_active":"2024-05-28T06:22:18.020608+0000","last_peered":"2024-05-28T06:22:18.020608+0000","last_clean":"2024-05-28T06:22:18.020608+0000","last_became_active":"2024-05-28T06:22:15.993895+0000","last_became_peered":"2024-05-28T06:22:15.993895+0000","last_unstale":"2024-05-28T06:22:18.020608+0000","last_undegraded":"2024-05-28T06:22:18.020608+0000","last_fullsized":"2024-05-28T06:22:18.020608+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:51:36.120307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.243620+0000","last_change":"2024-05-28T06:22:16.993934+0000","last_active":"2024-05-28T06:22:18.243620+0000","last_peered":"2024-05-28T06:22:18.243620+0000","last_clean":"2024-05-28T06:22:18.243620+0000","last_became_active":"2024-05-28T06:22:16.993249+0000","last_became_peered":"2024-05-28T06:22:16.993249+0000","last_unstale":"2024-05-28T06:22:18.243620+0000","last_undegraded":"2024-05-28T06:22:18.243620+0000","last_fullsized":"2024-05-28T06:22:18.243620+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:21:11.675368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.242990+0000","last_change":"2024-05-28T06:22:15.998999+0000","last_active":"2024-05-28T06:22:18.242990+0000","last_peered":"2024-05-28T06:22:18.242990+0000","last_clean":"2024-05-28T06:22:18.242990+0000","last_became_active":"2024-05-28T06:22:15.998631+0000","last_became_peered":"2024-05-28T06:22:15.998631+0000","last_unstale":"2024-05-28T06:22:18.242990+0000","last_undegraded":"2024-05-28T06:22:18.242990+0000","last_fullsized":"2024-05-28T06:22:18.242990+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:26:17.626246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.020122+0000","last_change":"2024-05-28T06:22:17.002213+0000","last_active":"2024-05-28T06:22:18.020122+0000","last_peered":"2024-05-28T06:22:18.020122+0000","last_clean":"2024-05-28T06:22:18.020122+0000","last_became_active":"2024-05-28T06:22:17.001843+0000","last_became_peered":"2024-05-28T06:22:17.001843+0000","last_unstale":"2024-05-28T06:22:18.020122+0000","last_undegraded":"2024-05-28T06:22:18.020122+0000","last_fullsized":"2024-05-28T06:22:18.020122+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:24:38.596641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.529805+0000","last_change":"2024-05-28T06:22:15.995193+0000","last_active":"2024-05-28T06:22:18.529805+0000","last_peered":"2024-05-28T06:22:18.529805+0000","last_clean":"2024-05-28T06:22:18.529805+0000","last_became_active":"2024-05-28T06:22:15.994748+0000","last_became_peered":"2024-05-28T06:22:15.994748+0000","last_unstale":"2024-05-28T06:22:18.529805+0000","last_undegraded":"2024-05-28T06:22:18.529805+0000","last_fullsized":"2024-05-28T06:22:18.529805+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:08:38.270230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.000212+0000","last_change":"2024-05-28T06:22:17.005154+0000","last_active":"2024-05-28T06:22:18.000212+0000","last_peered":"2024-05-28T06:22:18.000212+0000","last_clean":"2024-05-28T06:22:18.000212+0000","last_became_active":"2024-05-28T06:22:17.004836+0000","last_became_peered":"2024-05-28T06:22:17.004836+0000","last_unstale":"2024-05-28T06:22:18.000212+0000","last_undegraded":"2024-05-28T06:22:18.000212+0000","last_fullsized":"2024-05-28T06:22:18.000212+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T18:02:32.237669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.024271+0000","last_change":"2024-05-28T06:22:15.996930+0000","last_active":"2024-05-28T06:22:18.024271+0000","last_peered":"2024-05-28T06:22:18.024271+0000","last_clean":"2024-05-28T06:22:18.024271+0000","last_became_active":"2024-05-28T06:22:15.996598+0000","last_became_peered":"2024-05-28T06:22:15.996598+0000","last_unstale":"2024-05-28T06:22:18.024271+0000","last_undegraded":"2024-05-28T06:22:18.024271+0000","last_fullsized":"2024-05-28T06:22:18.024271+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:37:04.441468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.000755+0000","last_change":"2024-05-28T06:22:16.983951+0000","last_active":"2024-05-28T06:22:18.000755+0000","last_peered":"2024-05-28T06:22:18.000755+0000","last_clean":"2024-05-28T06:22:18.000755+0000","last_became_active":"2024-05-28T06:22:16.983201+0000","last_became_peered":"2024-05-28T06:22:16.983201+0000","last_unstale":"2024-05-28T06:22:18.000755+0000","last_undegraded":"2024-05-28T06:22:18.000755+0000","last_fullsized":"2024-05-28T06:22:18.000755+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:47:18.486697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.000007+0000","last_change":"2024-05-28T06:22:15.985622+0000","last_active":"2024-05-28T06:22:18.000007+0000","last_peered":"2024-05-28T06:22:18.000007+0000","last_clean":"2024-05-28T06:22:18.000007+0000","last_became_active":"2024-05-28T06:22:15.984560+0000","last_became_peered":"2024-05-28T06:22:15.984560+0000","last_unstale":"2024-05-28T06:22:18.000007+0000","last_undegraded":"2024-05-28T06:22:18.000007+0000","last_fullsized":"2024-05-28T06:22:18.000007+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:21:24.669839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.527773+0000","last_change":"2024-05-28T06:22:17.005937+0000","last_active":"2024-05-28T06:22:18.527773+0000","last_peered":"2024-05-28T06:22:18.527773+0000","last_clean":"2024-05-28T06:22:18.527773+0000","last_became_active":"2024-05-28T06:22:17.004354+0000","last_became_peered":"2024-05-28T06:22:17.004354+0000","last_unstale":"2024-05-28T06:22:18.527773+0000","last_undegraded":"2024-05-28T06:22:18.527773+0000","last_fullsized":"2024-05-28T06:22:18.527773+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:13:14.889721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:17.999121+0000","last_change":"2024-05-28T06:22:15.984208+0000","last_active":"2024-05-28T06:22:17.999121+0000","last_peered":"2024-05-28T06:22:17.999121+0000","last_clean":"2024-05-28T06:22:17.999121+0000","last_became_active":"2024-05-28T06:22:15.983816+0000","last_became_peered":"2024-05-28T06:22:15.983816+0000","last_unstale":"2024-05-28T06:22:17.999121+0000","last_undegraded":"2024-05-28T06:22:17.999121+0000","last_fullsized":"2024-05-28T06:22:17.999121+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:30:43.782132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.242161+0000","last_change":"2024-05-28T06:22:16.996830+0000","last_active":"2024-05-28T06:22:18.242161+0000","last_peered":"2024-05-28T06:22:18.242161+0000","last_clean":"2024-05-28T06:22:18.242161+0000","last_became_active":"2024-05-28T06:22:16.990102+0000","last_became_peered":"2024-05-28T06:22:16.990102+0000","last_unstale":"2024-05-28T06:22:18.242161+0000","last_undegraded":"2024-05-28T06:22:18.242161+0000","last_fullsized":"2024-05-28T06:22:18.242161+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T18:04:15.407639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.242532+0000","last_change":"2024-05-28T06:22:15.998500+0000","last_active":"2024-05-28T06:22:18.242532+0000","last_peered":"2024-05-28T06:22:18.242532+0000","last_clean":"2024-05-28T06:22:18.242532+0000","last_became_active":"2024-05-28T06:22:15.998176+0000","last_became_peered":"2024-05-28T06:22:15.998176+0000","last_unstale":"2024-05-28T06:22:18.242532+0000","last_undegraded":"2024-05-28T06:22:18.242532+0000","last_fullsized":"2024-05-28T06:22:18.242532+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:17:13.644125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":12,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.527150+0000","last_change":"2024-05-28T06:22:17.008104+0000","last_active":"2024-05-28T06:22:18.527150+0000","last_peered":"2024-05-28T06:22:18.527150+0000","last_clean":"2024-05-28T06:22:18.527150+0000","last_became_active":"2024-05-28T06:22:17.007752+0000","last_became_peered":"2024-05-28T06:22:17.007752+0000","last_unstale":"2024-05-28T06:22:18.527150+0000","last_undegraded":"2024-05-28T06:22:18.527150+0000","last_fullsized":"2024-05-28T06:22:18.527150+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":113,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:15.950581+0000","last_clean_scrub_stamp":"2024-05-28T06:22:15.950581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:08:39.984746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":14,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.024388+0000","last_change":"2024-05-28T06:22:15.998356+0000","last_active":"2024-05-28T06:22:18.024388+0000","last_peered":"2024-05-28T06:22:18.024388+0000","last_clean":"2024-05-28T06:22:18.024388+0000","last_became_active":"2024-05-28T06:22:15.998063+0000","last_became_peered":"2024-05-28T06:22:15.998063+0000","last_unstale":"2024-05-28T06:22:18.024388+0000","last_undegraded":"2024-05-28T06:22:18.024388+0000","last_fullsized":"2024-05-28T06:22:18.024388+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:22:14.938772+0000","last_clean_scrub_stamp":"2024-05-28T06:22:14.938772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:05:24.649440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":354,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-05-28T06:22:18.528168+0000","last_change":"2024-05-28T06:04:01.039129+0000","last_active":"2024-05-28T06:22:18.528168+0000","last_peered":"2024-05-28T06:22:18.528168+0000","last_clean":"2024-05-28T06:22:18.528168+0000","last_became_active":"2024-05-28T06:04:01.033906+0000","last_became_peered":"2024-05-28T06:04:01.033906+0000","last_unstale":"2024-05-28T06:22:18.528168+0000","last_undegraded":"2024-05-28T06:22:18.528168+0000","last_fullsized":"2024-05-28T06:22:18.528168+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:03:59.864237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:03:59.864237+0000","last_clean_scrub_stamp":"2024-05-28T06:03:59.864237+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:17:36.440245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575071,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":127500,"kb_used_data":1984,"kb_used_omap":1,"kb_used_meta":125502,"kb_avail":94244340,"statfs":{"total":96636764160,"available":96506204160,"internally_reserved":0,"allocated":2031616,"data_stored":1217255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":128514506},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575074,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":126916,"kb_used_data":1400,"kb_used_omap":1,"kb_used_meta":125502,"kb_avail":94244924,"statfs":{"total":96636764160,"available":96506802176,"internally_reserved":0,"allocated":1433600,"data_stored":626341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":128514502},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575072,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64464,"kb_used_data":1404,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94307376,"statfs":{"total":96636764160,"available":96570753024,"internally_reserved":0,"allocated":1437696,"data_stored":626875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575072,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":86656,"kb_used_data":1396,"kb_used_omap":1,"kb_used_meta":85246,"kb_avail":94285184,"statfs":{"total":96636764160,"available":96548028416,"internally_reserved":0,"allocated":1429504,"data_stored":626271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87292357},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575074,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":131096,"kb_used_data":1412,"kb_used_omap":1,"kb_used_meta":129662,"kb_avail":94240744,"statfs":{"total":96636764160,"available":96502521856,"internally_reserved":0,"allocated":1445888,"data_stored":627953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":132774343},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575073,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":144960,"kb_used_data":1396,"kb_used_omap":1,"kb_used_meta":143550,"kb_avail":94226880,"statfs":{"total":96636764160,"available":96488325120,"internally_reserved":0,"allocated":1429504,"data_stored":626271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":146995653},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575073,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":126920,"kb_used_data":1404,"kb_used_omap":1,"kb_used_meta":125502,"kb_avail":94244920,"statfs":{"total":96636764160,"available":96506798080,"internally_reserved":0,"allocated":1437696,"data_stored":626395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":128514501},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575072,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":87260,"kb_used_data":1996,"kb_used_omap":1,"kb_used_meta":85246,"kb_avail":94284580,"statfs":{"total":96636764160,"available":96547409920,"internally_reserved":0,"allocated":2043904,"data_stored":1218363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87292357},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-28T06:22:26.912 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-05-28T06:22:26.913 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-05-28T06:22:27.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.166+0000 7f0272200640 1 Processor -- start 2024-05-28T06:22:27.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.167+0000 7f0272200640 1 -- start start 2024-05-28T06:22:27.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.168+0000 7f0272200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f026c15e850 0x7f026c0b77b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:27.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.168+0000 7f0272200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f026c0d1d50 0x7f026c0b90a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:27.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.168+0000 7f0272200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f026c0b8000 con 0x7f026c0cef30 2024-05-28T06:22:27.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.168+0000 7f0272200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f026c15ec30 con 0x7f026c15e850 2024-05-28T06:22:27.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.168+0000 7f0272200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f026c0b82e0 con 0x7f026c0d1d50 2024-05-28T06:22:27.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.168+0000 7f026b400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f026c15e850 0x7f026c0b77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:27.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.168+0000 7f026aa00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f026c0cef30 0x7f026c0d1640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:35144/0 (socket says 172.21.15.96:35144) 2024-05-28T06:22:27.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.168+0000 7f026aa00640 1 -- 172.21.15.96:0/3982002991 learned_addr learned my addr 172.21.15.96:0/3982002991 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:22:27.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.169+0000 7f026be00640 1 --2- 172.21.15.96:0/3982002991 >> v2:172.21.15.123:3300/0 conn(0x7f026c0d1d50 0x7f026c0b90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:27.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.169+0000 7f026b400640 1 -- 172.21.15.96:0/3982002991 >> v2:172.21.15.123:3300/0 conn(0x7f026c0d1d50 msgr2=0x7f026c0b90a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:27.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.169+0000 7f026b400640 1 --2- 172.21.15.96:0/3982002991 >> v2:172.21.15.123:3300/0 conn(0x7f026c0d1d50 0x7f026c0b90a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:27.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.169+0000 7f026b400640 1 -- 172.21.15.96:0/3982002991 >> v1:172.21.15.96:6789/0 conn(0x7f026c0cef30 legacy=0x7f026c0d1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:27.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.169+0000 7f026b400640 1 -- 172.21.15.96:0/3982002991 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f026c0b85c0 con 0x7f026c15e850 2024-05-28T06:22:27.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.169+0000 7f026a000640 1 -- 172.21.15.96:0/3982002991 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2153182989 0 0) 0x7f026c0b8000 con 0x7f026c0cef30 2024-05-28T06:22:27.166 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.170+0000 7f026b400640 1 --2- 172.21.15.96:0/3982002991 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f026c15e850 0x7f026c0b77b0 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7f026c0cb280 tx=0x7f025c02a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67a8cd57d7e0d45d server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:27.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.170+0000 7f026a000640 1 -- 172.21.15.96:0/3982002991 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f025c007740 con 0x7f026c15e850 2024-05-28T06:22:27.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.170+0000 7f026a000640 1 -- 172.21.15.96:0/3982002991 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f025c0078e0 con 0x7f026c15e850 2024-05-28T06:22:27.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.170+0000 7f026a000640 1 -- 172.21.15.96:0/3982002991 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f025c007cb0 con 0x7f026c15e850 2024-05-28T06:22:27.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.171+0000 7f0272200640 1 -- 172.21.15.96:0/3982002991 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f026c15e850 msgr2=0x7f026c0b77b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:27.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.171+0000 7f0272200640 1 --2- 172.21.15.96:0/3982002991 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f026c15e850 0x7f026c0b77b0 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7f026c0cb280 tx=0x7f025c02a5f0 comp rx=0 tx=0).stop 2024-05-28T06:22:27.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.171+0000 7f0272200640 1 -- 172.21.15.96:0/3982002991 shutdown_connections 2024-05-28T06:22:27.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.171+0000 7f0272200640 1 --2- 172.21.15.96:0/3982002991 >> v2:172.21.15.123:3300/0 conn(0x7f026c0d1d50 0x7f026c0b90a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:27.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.171+0000 7f0272200640 1 --2- 172.21.15.96:0/3982002991 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f026c15e850 0x7f026c0b77b0 unknown :-1 s=CLOSED pgs=1826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:27.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.171+0000 7f0272200640 1 -- 172.21.15.96:0/3982002991 >> 172.21.15.96:0/3982002991 conn(0x7f026c067aa0 msgr2=0x7f026c067eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:27.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.171+0000 7f0272200640 1 -- 172.21.15.96:0/3982002991 shutdown_connections 2024-05-28T06:22:27.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.171+0000 7f0272200640 1 -- 172.21.15.96:0/3982002991 wait complete. 2024-05-28T06:22:27.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.173+0000 7f0272200640 1 Processor -- start 2024-05-28T06:22:27.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.173+0000 7f0272200640 1 -- start start 2024-05-28T06:22:27.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.174+0000 7f0272200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f026c0cef30 0x7f026c160910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:27.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.174+0000 7f0272200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f026c0d1d50 0x7f026c160e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:27.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.174+0000 7f0272200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f026c15e850 0x7f026c1f4290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:27.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.174+0000 7f0272200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f026c0b95e0 con 0x7f026c0cef30 2024-05-28T06:22:27.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.174+0000 7f0272200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f026c0b7cf0 con 0x7f026c0d1d50 2024-05-28T06:22:27.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.174+0000 7f0272200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f026c0b9760 con 0x7f026c15e850 2024-05-28T06:22:27.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.174+0000 7f026b400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f026c0cef30 0x7f026c160910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:27.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.174+0000 7f026b400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f026c0cef30 0x7f026c160910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:60282/0 (socket says 172.21.15.96:60282) 2024-05-28T06:22:27.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.174+0000 7f026b400640 1 -- 172.21.15.96:0/3478766837 learned_addr learned my addr 172.21.15.96:0/3478766837 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:22:27.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.174+0000 7f026b400640 1 -- 172.21.15.96:0/3478766837 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f026c15e850 msgr2=0x7f026c1f4290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:27.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.174+0000 7f026b400640 1 --2- 172.21.15.96:0/3478766837 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f026c15e850 0x7f026c1f4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:27.171 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.174+0000 7f026b400640 1 -- 172.21.15.96:0/3478766837 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f026c0d1d50 msgr2=0x7f026c160e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:27.171 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.174+0000 7f026b400640 1 --2- 172.21.15.96:0/3478766837 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f026c0d1d50 0x7f026c160e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:27.171 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.174+0000 7f026b400640 1 -- 172.21.15.96:0/3478766837 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f026c1f5000 con 0x7f026c0cef30 2024-05-28T06:22:27.172 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.175+0000 7f026b400640 1 --2- 172.21.15.96:0/3478766837 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f026c0cef30 0x7f026c160910 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7f025c000f80 tx=0x7f025c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:27.172 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.175+0000 7f0263400640 1 -- 172.21.15.96:0/3478766837 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f025c03f070 con 0x7f026c0cef30 2024-05-28T06:22:27.172 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.175+0000 7f0263400640 1 -- 172.21.15.96:0/3478766837 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f025c031c40 con 0x7f026c0cef30 2024-05-28T06:22:27.172 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.175+0000 7f0263400640 1 -- 172.21.15.96:0/3478766837 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f025c003990 con 0x7f026c0cef30 2024-05-28T06:22:27.172 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.175+0000 7f0272200640 1 -- 172.21.15.96:0/3478766837 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f026c0cb8d0 con 0x7f026c0cef30 2024-05-28T06:22:27.172 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.175+0000 7f0272200640 1 -- 172.21.15.96:0/3478766837 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f026c0cbe90 con 0x7f026c0cef30 2024-05-28T06:22:27.172 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.176+0000 7f0272200640 1 -- 172.21.15.96:0/3478766837 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f026c0689e0 con 0x7f026c0cef30 2024-05-28T06:22:27.172 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.177+0000 7f0263400640 1 -- 172.21.15.96:0/3478766837 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 37) ==== 93182+0+0 (secure 0 0 0) 0x7f025c007790 con 0x7f026c0cef30 2024-05-28T06:22:27.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.178+0000 7f0263400640 1 --2- 172.21.15.96:0/3478766837 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f024806e090 0x7f0248070550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:27.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.178+0000 7f0263400640 1 -- 172.21.15.96:0/3478766837 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 5765+0+0 (secure 0 0 0) 0x7f025c0abca0 con 0x7f026c0cef30 2024-05-28T06:22:27.174 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.179+0000 7f026aa00640 1 --2- 172.21.15.96:0/3478766837 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f024806e090 0x7f0248070550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:27.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.180+0000 7f026aa00640 1 --2- 172.21.15.96:0/3478766837 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f024806e090 0x7f0248070550 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f0254007fd0 tx=0x7f0254007920 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:27.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.182+0000 7f0263400640 1 -- 172.21.15.96:0/3478766837 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f025c07bb30 con 0x7f026c0cef30 2024-05-28T06:22:27.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.455+0000 7f0272200640 1 -- 172.21.15.96:0/3478766837 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f026c0cf6f0 con 0x7f026c0cef30 2024-05-28T06:22:27.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.459+0000 7f0263400640 1 -- 172.21.15.96:0/3478766837 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v22) ==== 190+0+147 (secure 0 0 0) 0x7f025c07b430 con 0x7f026c0cef30 2024-05-28T06:22:27.455 INFO:teuthology.orchestra.run.smithi096.stdout:[client.0] 2024-05-28T06:22:27.455 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQCXc1Vm91hbGBAASTL3HQR8XOs7wVQEHbPl/g== 2024-05-28T06:22:27.455 INFO:teuthology.orchestra.run.smithi096.stdout: caps mds = "allow" 2024-05-28T06:22:27.456 INFO:teuthology.orchestra.run.smithi096.stdout: caps mon = "allow r" 2024-05-28T06:22:27.456 INFO:teuthology.orchestra.run.smithi096.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-05-28T06:22:27.456 INFO:teuthology.orchestra.run.smithi096.stderr:updated caps for client.0 2024-05-28T06:22:27.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.462+0000 7f0272200640 1 -- 172.21.15.96:0/3478766837 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f024806e090 msgr2=0x7f0248070550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:27.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.462+0000 7f0272200640 1 --2- 172.21.15.96:0/3478766837 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f024806e090 0x7f0248070550 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f0254007fd0 tx=0x7f0254007920 comp rx=0 tx=0).stop 2024-05-28T06:22:27.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.462+0000 7f0272200640 1 -- 172.21.15.96:0/3478766837 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f026c0cef30 msgr2=0x7f026c160910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:27.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.462+0000 7f0272200640 1 --2- 172.21.15.96:0/3478766837 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f026c0cef30 0x7f026c160910 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7f025c000f80 tx=0x7f025c004a40 comp rx=0 tx=0).stop 2024-05-28T06:22:27.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.463+0000 7f0272200640 1 -- 172.21.15.96:0/3478766837 shutdown_connections 2024-05-28T06:22:27.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.463+0000 7f0272200640 1 --2- 172.21.15.96:0/3478766837 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f024806e090 0x7f0248070550 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:27.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.463+0000 7f0272200640 1 --2- 172.21.15.96:0/3478766837 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f026c15e850 0x7f026c1f4290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:27.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.463+0000 7f0272200640 1 --2- 172.21.15.96:0/3478766837 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f026c0d1d50 0x7f026c160e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:27.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.463+0000 7f0272200640 1 --2- 172.21.15.96:0/3478766837 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f026c0cef30 0x7f026c160910 unknown :-1 s=CLOSED pgs=1827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:27.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.463+0000 7f0272200640 1 -- 172.21.15.96:0/3478766837 >> 172.21.15.96:0/3478766837 conn(0x7f026c067aa0 msgr2=0x7f026c0b56e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:27.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.463+0000 7f0272200640 1 -- 172.21.15.96:0/3478766837 shutdown_connections 2024-05-28T06:22:27.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.463+0000 7f0272200640 1 -- 172.21.15.96:0/3478766837 wait complete. 2024-05-28T06:22:27.476 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-28T06:22:27.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.683+0000 7fa6e3200640 1 Processor -- start 2024-05-28T06:22:27.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.683+0000 7fa6e3200640 1 -- start start 2024-05-28T06:22:27.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.685+0000 7fa6e3200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fa6dc0b37a0 0x7fa6dc0d0780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:27.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.685+0000 7fa6e3200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6dc0d0ef0 con 0x7fa6dc06d950 2024-05-28T06:22:27.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.685+0000 7fa6e3200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6dc0d11d0 con 0x7fa6dc09b140 2024-05-28T06:22:27.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.685+0000 7fa6e3200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fa6dc0d0cc0 con 0x7fa6dc0b37a0 2024-05-28T06:22:27.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.685+0000 7fa6dbe00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fa6dc0b37a0 0x7fa6dc0d0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:27.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.685+0000 7fa6db400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fa6dc06d950 0x7fa6dc09aa30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:35156/0 (socket says 172.21.15.96:35156) 2024-05-28T06:22:27.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.685+0000 7fa6db400640 1 -- 172.21.15.96:0/186021288 learned_addr learned my addr 172.21.15.96:0/186021288 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:22:27.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.686+0000 7fa6dbe00640 1 -- 172.21.15.96:0/186021288 >> v1:172.21.15.123:6789/0 conn(0x7fa6dc09b140 legacy=0x7fa6dc15e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:27.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.686+0000 7fa6dbe00640 1 -- 172.21.15.96:0/186021288 >> v1:172.21.15.96:6789/0 conn(0x7fa6dc06d950 legacy=0x7fa6dc09aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:27.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.686+0000 7fa6dbe00640 1 -- 172.21.15.96:0/186021288 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6dc0d14b0 con 0x7fa6dc0b37a0 2024-05-28T06:22:27.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.687+0000 7fa6dbe00640 1 --2- 172.21.15.96:0/186021288 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6dc0b37a0 0x7fa6dc0d0780 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7fa6cc0097a0 tx=0x7fa6cc02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a39e0b96aba01206 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:27.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.687+0000 7fa6daa00640 1 -- 172.21.15.96:0/186021288 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6cc037070 con 0x7fa6dc0b37a0 2024-05-28T06:22:27.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.687+0000 7fa6daa00640 1 -- 172.21.15.96:0/186021288 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6cc032440 con 0x7fa6dc0b37a0 2024-05-28T06:22:27.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.687+0000 7fa6daa00640 1 -- 172.21.15.96:0/186021288 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6cc032810 con 0x7fa6dc0b37a0 2024-05-28T06:22:27.683 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.688+0000 7fa6e3200640 1 -- 172.21.15.96:0/186021288 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6dc0b37a0 msgr2=0x7fa6dc0d0780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:27.683 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.688+0000 7fa6e3200640 1 --2- 172.21.15.96:0/186021288 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6dc0b37a0 0x7fa6dc0d0780 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7fa6cc0097a0 tx=0x7fa6cc02a270 comp rx=0 tx=0).stop 2024-05-28T06:22:27.683 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.688+0000 7fa6e3200640 1 -- 172.21.15.96:0/186021288 shutdown_connections 2024-05-28T06:22:27.683 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.688+0000 7fa6e3200640 1 --2- 172.21.15.96:0/186021288 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6dc0b37a0 0x7fa6dc0d0780 unknown :-1 s=CLOSED pgs=1829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:27.683 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.688+0000 7fa6e3200640 1 -- 172.21.15.96:0/186021288 >> 172.21.15.96:0/186021288 conn(0x7fa6dc0b3f40 msgr2=0x7fa6dc0b4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:27.684 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.688+0000 7fa6e3200640 1 -- 172.21.15.96:0/186021288 shutdown_connections 2024-05-28T06:22:27.684 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.688+0000 7fa6e3200640 1 -- 172.21.15.96:0/186021288 wait complete. 2024-05-28T06:22:27.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.689+0000 7fa6e3200640 1 Processor -- start 2024-05-28T06:22:27.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.690+0000 7fa6e3200640 1 -- start start 2024-05-28T06:22:27.687 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.690+0000 7fa6e3200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6dc06d950 0x7fa6dc152880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:27.687 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.690+0000 7fa6e3200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa6dc09b140 0x7fa6dc0b7b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:27.687 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.690+0000 7fa6e3200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6dc0b37a0 0x7fa6dc0cb3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:27.687 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.691+0000 7fa6e3200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa6dc06ddc0 con 0x7fa6dc0b37a0 2024-05-28T06:22:27.687 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.691+0000 7fa6e3200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa6dc0d17f0 con 0x7fa6dc06d950 2024-05-28T06:22:27.687 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.691+0000 7fa6e3200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fa6dc06df40 con 0x7fa6dc09b140 2024-05-28T06:22:27.687 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.691+0000 7fa6e1200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6dc0b37a0 0x7fa6dc0cb3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:27.687 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.691+0000 7fa6e1200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6dc0b37a0 0x7fa6dc0cb3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:60310/0 (socket says 172.21.15.96:60310) 2024-05-28T06:22:27.687 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.691+0000 7fa6e1200640 1 -- 172.21.15.96:0/1071099876 learned_addr learned my addr 172.21.15.96:0/1071099876 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:22:27.687 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.691+0000 7fa6dbe00640 1 --2- 172.21.15.96:0/1071099876 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6dc06d950 0x7fa6dc152880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:27.687 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.691+0000 7fa6db400640 1 --2- 172.21.15.96:0/1071099876 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa6dc09b140 0x7fa6dc0b7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:27.688 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.691+0000 7fa6e1200640 1 -- 172.21.15.96:0/1071099876 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa6dc09b140 msgr2=0x7fa6dc0b7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:27.688 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.691+0000 7fa6e1200640 1 --2- 172.21.15.96:0/1071099876 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa6dc09b140 0x7fa6dc0b7b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:27.688 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.691+0000 7fa6e1200640 1 -- 172.21.15.96:0/1071099876 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6dc06d950 msgr2=0x7fa6dc152880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:27.688 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.691+0000 7fa6e1200640 1 --2- 172.21.15.96:0/1071099876 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6dc06d950 0x7fa6dc152880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:27.688 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.691+0000 7fa6e1200640 1 -- 172.21.15.96:0/1071099876 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6dc0b6bf0 con 0x7fa6dc0b37a0 2024-05-28T06:22:27.688 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.692+0000 7fa6e1200640 1 --2- 172.21.15.96:0/1071099876 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6dc0b37a0 0x7fa6dc0cb3e0 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7fa6d400c780 tx=0x7fa6d400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:27.689 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.692+0000 7fa6d3e00640 1 -- 172.21.15.96:0/1071099876 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6d4004b60 con 0x7fa6dc0b37a0 2024-05-28T06:22:27.689 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.692+0000 7fa6d3e00640 1 -- 172.21.15.96:0/1071099876 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6d4004d00 con 0x7fa6dc0b37a0 2024-05-28T06:22:27.689 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.692+0000 7fa6e3200640 1 -- 172.21.15.96:0/1071099876 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6dc0b5e30 con 0x7fa6dc0b37a0 2024-05-28T06:22:27.689 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.692+0000 7fa6d3e00640 1 -- 172.21.15.96:0/1071099876 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6d400aa90 con 0x7fa6dc0b37a0 2024-05-28T06:22:27.689 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.692+0000 7fa6e3200640 1 -- 172.21.15.96:0/1071099876 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6dc0b63f0 con 0x7fa6dc0b37a0 2024-05-28T06:22:27.689 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.693+0000 7fa6e3200640 1 -- 172.21.15.96:0/1071099876 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6dc0689e0 con 0x7fa6dc0b37a0 2024-05-28T06:22:27.689 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.694+0000 7fa6d3e00640 1 -- 172.21.15.96:0/1071099876 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 37) ==== 93182+0+0 (secure 0 0 0) 0x7fa6d400ac30 con 0x7fa6dc0b37a0 2024-05-28T06:22:27.691 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.696+0000 7fa6d3e00640 1 --2- 172.21.15.96:0/1071099876 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa6b806e090 0x7fa6b8070550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:27.691 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.696+0000 7fa6d3e00640 1 -- 172.21.15.96:0/1071099876 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 5765+0+0 (secure 0 0 0) 0x7fa6d408ac30 con 0x7fa6dc0b37a0 2024-05-28T06:22:27.691 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.696+0000 7fa6dbe00640 1 --2- 172.21.15.96:0/1071099876 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa6b806e090 0x7fa6b8070550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:27.693 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.697+0000 7fa6dbe00640 1 --2- 172.21.15.96:0/1071099876 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa6b806e090 0x7fa6b8070550 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fa6cc034000 tx=0x7fa6cc02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:27.696 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.700+0000 7fa6d3e00640 1 -- 172.21.15.96:0/1071099876 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa6d4020070 con 0x7fa6dc0b37a0 2024-05-28T06:22:27.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.941+0000 7fa6e3200640 1 -- 172.21.15.96:0/1071099876 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa6dc0b4b30 con 0x7fa6dc0b37a0 2024-05-28T06:22:27.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.942+0000 7fa6d3e00640 1 -- 172.21.15.96:0/1071099876 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) ==== 94+0+5189 (secure 0 0 0) 0x7fa6d405aac0 con 0x7fa6dc0b37a0 2024-05-28T06:22:27.938 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:22:27.938 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":72,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6724,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/2148532848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6835","nonce":2148532848},{"type":"v1","addr":"172.21.15.123:6837","nonce":2148532848}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":9252,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6834/2173534467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2173534467},{"type":"v1","addr":"172.21.15.96:6834","nonce":2173534467}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":9255,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/2552082898","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6833","nonce":2552082898},{"type":"v1","addr":"172.21.15.96:6835","nonce":2552082898}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-28T06:22:16.424757+0000","modified":"2024-05-28T06:22:19.524390+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"b","rank":0,"incarnation":68,"state":"up:active","state_seq":3,"addr":"172.21.15.123:6836/3192559785","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":3192559785},{"type":"v1","addr":"172.21.15.123:6836","nonce":3192559785}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9270,"qdb_cluster":[9270]},"id":8}]} 2024-05-28T06:22:27.938 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 72 2024-05-28T06:22:27.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.945+0000 7fa6e3200640 1 -- 172.21.15.96:0/1071099876 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa6b806e090 msgr2=0x7fa6b8070550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:27.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.945+0000 7fa6e3200640 1 --2- 172.21.15.96:0/1071099876 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa6b806e090 0x7fa6b8070550 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fa6cc034000 tx=0x7fa6cc02e000 comp rx=0 tx=0).stop 2024-05-28T06:22:27.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.945+0000 7fa6e3200640 1 -- 172.21.15.96:0/1071099876 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6dc0b37a0 msgr2=0x7fa6dc0cb3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:27.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.945+0000 7fa6e3200640 1 --2- 172.21.15.96:0/1071099876 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6dc0b37a0 0x7fa6dc0cb3e0 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7fa6d400c780 tx=0x7fa6d400cc40 comp rx=0 tx=0).stop 2024-05-28T06:22:27.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.945+0000 7fa6e3200640 1 -- 172.21.15.96:0/1071099876 shutdown_connections 2024-05-28T06:22:27.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.945+0000 7fa6e3200640 1 --2- 172.21.15.96:0/1071099876 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa6b806e090 0x7fa6b8070550 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:27.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.945+0000 7fa6e3200640 1 --2- 172.21.15.96:0/1071099876 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6dc0b37a0 0x7fa6dc0cb3e0 unknown :-1 s=CLOSED pgs=1830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:27.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.945+0000 7fa6e3200640 1 --2- 172.21.15.96:0/1071099876 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa6dc09b140 0x7fa6dc0b7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:27.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.945+0000 7fa6e3200640 1 --2- 172.21.15.96:0/1071099876 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6dc06d950 0x7fa6dc152880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:27.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.945+0000 7fa6e3200640 1 -- 172.21.15.96:0/1071099876 >> 172.21.15.96:0/1071099876 conn(0x7fa6dc0b3f40 msgr2=0x7fa6dc0ccae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:27.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.945+0000 7fa6e3200640 1 -- 172.21.15.96:0/1071099876 shutdown_connections 2024-05-28T06:22:27.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:27.946+0000 7fa6e3200640 1 -- 172.21.15.96:0/1071099876 wait complete. 2024-05-28T06:22:28.362 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 72, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-05-28T06:22:16.424757+0000', 'modified': '2024-05-28T06:22:19.524390+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'b', 'rank': 0, 'incarnation': 68, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.123:6836/3192559785', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.123:6834', 'nonce': 3192559785}, {'type': 'v1', 'addr': '172.21.15.123:6836', 'nonce': 3192559785}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9270, 'qdb_cluster': [9270]} 2024-05-28T06:22:28.363 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-05-28T06:22:28.363 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-05-28T06:22:28.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.576+0000 7fdea9e00640 1 Processor -- start 2024-05-28T06:22:28.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.576+0000 7fdea9e00640 1 -- start start 2024-05-28T06:22:28.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.578+0000 7fdea9e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fdea40d16a0 0x7fdea40cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:28.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.578+0000 7fdea9e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdea40cf870 0x7fdea40d25e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:28.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.578+0000 7fdea9e00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdea415ec50 con 0x7fdea415e870 2024-05-28T06:22:28.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.578+0000 7fdea9e00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fdea40d2150 con 0x7fdea40d16a0 2024-05-28T06:22:28.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.578+0000 7fdea9e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fdea40cfd20 con 0x7fdea40cf870 2024-05-28T06:22:28.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.578+0000 7fdea3e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fdea40d16a0 0x7fdea40cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:28.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.578+0000 7fdea3e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fdea40d16a0 0x7fdea40cf330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:60312/0 (socket says 172.21.15.96:60312) 2024-05-28T06:22:28.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.578+0000 7fdea3e00640 1 -- 172.21.15.96:0/3960082313 learned_addr learned my addr 172.21.15.96:0/3960082313 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:22:28.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.578+0000 7fdea3e00640 1 -- 172.21.15.96:0/3960082313 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdea40cf870 msgr2=0x7fdea40d25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:28.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.578+0000 7fdea3e00640 1 --2- 172.21.15.96:0/3960082313 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdea40cf870 0x7fdea40d25e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:28.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.579+0000 7fdea3e00640 1 -- 172.21.15.96:0/3960082313 >> v1:172.21.15.123:6789/0 conn(0x7fdea415e870 legacy=0x7fdea40d0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:28.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.579+0000 7fdea3e00640 1 -- 172.21.15.96:0/3960082313 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdea40cff80 con 0x7fdea40d16a0 2024-05-28T06:22:28.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.580+0000 7fdea3e00640 1 --2- 172.21.15.96:0/3960082313 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdea40d16a0 0x7fdea40cf330 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7fde90009850 tx=0x7fde9002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea8db3a1cfe553e9 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:28.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.580+0000 7fdea3400640 1 -- 172.21.15.96:0/3960082313 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde90004030 con 0x7fdea40d16a0 2024-05-28T06:22:28.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.580+0000 7fdea3400640 1 -- 172.21.15.96:0/3960082313 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde900041d0 con 0x7fdea40d16a0 2024-05-28T06:22:28.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.580+0000 7fdea3400640 1 -- 172.21.15.96:0/3960082313 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde90003610 con 0x7fdea40d16a0 2024-05-28T06:22:28.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.580+0000 7fdea9e00640 1 -- 172.21.15.96:0/3960082313 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdea40d16a0 msgr2=0x7fdea40cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:28.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.580+0000 7fdea9e00640 1 --2- 172.21.15.96:0/3960082313 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdea40d16a0 0x7fdea40cf330 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7fde90009850 tx=0x7fde9002a1e0 comp rx=0 tx=0).stop 2024-05-28T06:22:28.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.580+0000 7fdea9e00640 1 -- 172.21.15.96:0/3960082313 shutdown_connections 2024-05-28T06:22:28.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.580+0000 7fdea9e00640 1 --2- 172.21.15.96:0/3960082313 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdea40cf870 0x7fdea40d25e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:28.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.580+0000 7fdea9e00640 1 --2- 172.21.15.96:0/3960082313 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdea40d16a0 0x7fdea40cf330 unknown :-1 s=CLOSED pgs=1831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:28.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.580+0000 7fdea9e00640 1 -- 172.21.15.96:0/3960082313 >> 172.21.15.96:0/3960082313 conn(0x7fdea408bfc0 msgr2=0x7fdea40d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:28.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.581+0000 7fdea9e00640 1 -- 172.21.15.96:0/3960082313 shutdown_connections 2024-05-28T06:22:28.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.581+0000 7fdea9e00640 1 -- 172.21.15.96:0/3960082313 wait complete. 2024-05-28T06:22:28.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.582+0000 7fdea9e00640 1 Processor -- start 2024-05-28T06:22:28.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.582+0000 7fdea9e00640 1 -- start start 2024-05-28T06:22:28.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.583+0000 7fdea9e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdea40cf870 0x7fdea40b8db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:28.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.583+0000 7fdea9e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdea40d16a0 0x7fdea41e9610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:28.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.583+0000 7fdea9e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdea415e870 0x7fdea40b71a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:28.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.583+0000 7fdea9e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fdea40d0820 con 0x7fdea40cf870 2024-05-28T06:22:28.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.583+0000 7fdea9e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fdea40d0540 con 0x7fdea415e870 2024-05-28T06:22:28.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.583+0000 7fdea9e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fdea40d4fb0 con 0x7fdea40d16a0 2024-05-28T06:22:28.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.583+0000 7fdea8a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdea40cf870 0x7fdea40b8db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:28.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.583+0000 7fdea8a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdea40cf870 0x7fdea40b8db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:60328/0 (socket says 172.21.15.96:60328) 2024-05-28T06:22:28.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.583+0000 7fdea8a00640 1 -- 172.21.15.96:0/1501386549 learned_addr learned my addr 172.21.15.96:0/1501386549 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:22:28.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.584+0000 7fdea8a00640 1 -- 172.21.15.96:0/1501386549 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdea40d16a0 msgr2=0x7fdea41e9610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:28.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.584+0000 7fdea8a00640 1 --2- 172.21.15.96:0/1501386549 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdea40d16a0 0x7fdea41e9610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:28.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.584+0000 7fdea8a00640 1 -- 172.21.15.96:0/1501386549 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdea415e870 msgr2=0x7fdea40b71a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:28.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.584+0000 7fdea8a00640 1 --2- 172.21.15.96:0/1501386549 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdea415e870 0x7fdea40b71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:28.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.584+0000 7fdea8a00640 1 -- 172.21.15.96:0/1501386549 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdea41e9b50 con 0x7fdea40cf870 2024-05-28T06:22:28.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.584+0000 7fdea8a00640 1 --2- 172.21.15.96:0/1501386549 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdea40cf870 0x7fdea40b8db0 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7fde9400ba50 tx=0x7fde9400bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:28.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.584+0000 7fdea0c00640 1 -- 172.21.15.96:0/1501386549 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde94019070 con 0x7fdea40cf870 2024-05-28T06:22:28.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.584+0000 7fdea0c00640 1 -- 172.21.15.96:0/1501386549 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde9400cc10 con 0x7fdea40cf870 2024-05-28T06:22:28.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.584+0000 7fdea0c00640 1 -- 172.21.15.96:0/1501386549 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde94014440 con 0x7fdea40cf870 2024-05-28T06:22:28.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.584+0000 7fdea9e00640 1 -- 172.21.15.96:0/1501386549 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdea40b7d10 con 0x7fdea40cf870 2024-05-28T06:22:28.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.585+0000 7fdea9e00640 1 -- 172.21.15.96:0/1501386549 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdea40b82d0 con 0x7fdea40cf870 2024-05-28T06:22:28.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.587+0000 7fdea0c00640 1 -- 172.21.15.96:0/1501386549 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 37) ==== 93182+0+0 (secure 0 0 0) 0x7fde9401c070 con 0x7fdea40cf870 2024-05-28T06:22:28.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.588+0000 7fdea0c00640 1 --2- 172.21.15.96:0/1501386549 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fde8406e090 0x7fde84070550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:28.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.588+0000 7fdea0c00640 1 -- 172.21.15.96:0/1501386549 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 5765+0+0 (secure 0 0 0) 0x7fde9408a790 con 0x7fdea40cf870 2024-05-28T06:22:28.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.588+0000 7fdeae3b8740 1 Processor -- start 2024-05-28T06:22:28.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.588+0000 7fdea3e00640 1 --2- 172.21.15.96:0/1501386549 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fde8406e090 0x7fde84070550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:28.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.588+0000 7fdeae3b8740 1 -- start start 2024-05-28T06:22:28.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.589+0000 7fdeae3b8740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560b31f85490 0x560b31f88c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:28.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.589+0000 7fdeae3b8740 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x560b31f89150 0x560b31fa5090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:28.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.589+0000 7fdeae3b8740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560b31fa55d0 0x560b31fa5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:28.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.589+0000 7fdeae3b8740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x560b31ec2b10 con 0x560b31fa55d0 2024-05-28T06:22:28.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.589+0000 7fdeae3b8740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x560b31e692b0 con 0x560b31f85490 2024-05-28T06:22:28.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.589+0000 7fdeae3b8740 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x560b31ebead0 con 0x560b31f89150 2024-05-28T06:22:28.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.589+0000 7fdea3e00640 1 --2- 172.21.15.96:0/1501386549 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fde8406e090 0x7fde84070550 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fde90033000 tx=0x7fde9002c000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:28.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.590+0000 7fdea9400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560b31fa55d0 0x560b31fa5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:28.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.590+0000 7fdea9400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560b31fa55d0 0x560b31fa5a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:60332/0 (socket says 172.21.15.96:60332) 2024-05-28T06:22:28.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.590+0000 7fdea9400640 1 -- 172.21.15.96:0/2884115400 learned_addr learned my addr 172.21.15.96:0/2884115400 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:22:28.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.590+0000 7fdea3e00640 1 --2- 172.21.15.96:0/2884115400 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x560b31f89150 0x560b31fa5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:28.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.590+0000 7fdea8a00640 1 --2- 172.21.15.96:0/2884115400 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560b31f85490 0x560b31f88c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:28.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.590+0000 7fdea9400640 1 -- 172.21.15.96:0/2884115400 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x560b31f89150 msgr2=0x560b31fa5090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:28.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.590+0000 7fdea9400640 1 --2- 172.21.15.96:0/2884115400 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x560b31f89150 0x560b31fa5090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:28.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.590+0000 7fdea9400640 1 -- 172.21.15.96:0/2884115400 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560b31f85490 msgr2=0x560b31f88c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:28.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.590+0000 7fdea9400640 1 --2- 172.21.15.96:0/2884115400 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560b31f85490 0x560b31f88c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:28.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.590+0000 7fdea9400640 1 -- 172.21.15.96:0/2884115400 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560b31bd52d0 con 0x560b31fa55d0 2024-05-28T06:22:28.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.590+0000 7fdea9400640 1 --2- 172.21.15.96:0/2884115400 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560b31fa55d0 0x560b31fa5a30 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7fde9c0097a0 tx=0x7fde9c009c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c683b930c9fdd78 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:28.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.591+0000 7fde99200640 1 -- 172.21.15.96:0/2884115400 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde9c015020 con 0x560b31fa55d0 2024-05-28T06:22:28.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.591+0000 7fde99200640 1 -- 172.21.15.96:0/2884115400 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde9c00bad0 con 0x560b31fa55d0 2024-05-28T06:22:28.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.591+0000 7fde99200640 1 -- 172.21.15.96:0/2884115400 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde9c011420 con 0x560b31fa55d0 2024-05-28T06:22:28.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.591+0000 7fdeae3b8740 1 -- 172.21.15.96:0/2884115400 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560b31fa55d0 msgr2=0x560b31fa5a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:28.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.591+0000 7fdeae3b8740 1 --2- 172.21.15.96:0/2884115400 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560b31fa55d0 0x560b31fa5a30 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7fde9c0097a0 tx=0x7fde9c009c60 comp rx=0 tx=0).stop 2024-05-28T06:22:28.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.591+0000 7fdeae3b8740 1 -- 172.21.15.96:0/2884115400 shutdown_connections 2024-05-28T06:22:28.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.591+0000 7fdeae3b8740 1 --2- 172.21.15.96:0/2884115400 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560b31fa55d0 0x560b31fa5a30 unknown :-1 s=CLOSED pgs=1833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:28.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.591+0000 7fdeae3b8740 1 --2- 172.21.15.96:0/2884115400 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x560b31f89150 0x560b31fa5090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:28.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.591+0000 7fdeae3b8740 1 --2- 172.21.15.96:0/2884115400 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560b31f85490 0x560b31f88c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:28.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.591+0000 7fdeae3b8740 1 -- 172.21.15.96:0/2884115400 >> 172.21.15.96:0/2884115400 conn(0x560b31f5b470 msgr2=0x560b31f5c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:28.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.592+0000 7fdeae3b8740 1 -- 172.21.15.96:0/2884115400 shutdown_connections 2024-05-28T06:22:28.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.592+0000 7fdeae3b8740 1 -- 172.21.15.96:0/2884115400 wait complete. 2024-05-28T06:22:28.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.592+0000 7fdeae3b8740 1 -- start start 2024-05-28T06:22:28.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.592+0000 7fdeae3b8740 1 Processor -- start 2024-05-28T06:22:28.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.593+0000 7fdeae3b8740 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x560b31f89150 0x560b31f8c820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:28.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.593+0000 7fdeae3b8740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560b31fa55d0 0x560b31f58440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:28.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.593+0000 7fdeae3b8740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560b31f817a0 0x560b31f58980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:28.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.593+0000 7fdeae3b8740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x560b31f592a0 con 0x560b31fa55d0 2024-05-28T06:22:28.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.593+0000 7fdeae3b8740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x560b31e7c570 con 0x560b31f817a0 2024-05-28T06:22:28.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.593+0000 7fdeae3b8740 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x560b31f59580 con 0x560b31f89150 2024-05-28T06:22:28.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.593+0000 7fdeae3b8740 10 client.0 _ll_register_callbacks cb 0x560b31e4afc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-28T06:22:28.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.594+0000 7fdea3e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560b31fa55d0 0x560b31f58440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:28.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.594+0000 7fdea3e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560b31fa55d0 0x560b31f58440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:60348/0 (socket says 172.21.15.96:60348) 2024-05-28T06:22:28.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.594+0000 7fdea3e00640 1 -- 172.21.15.96:0/4241220032 learned_addr learned my addr 172.21.15.96:0/4241220032 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:22:28.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.594+0000 7fdea9400640 1 --2- 172.21.15.96:0/4241220032 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560b31f817a0 0x560b31f58980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:28.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.594+0000 7fdea8a00640 1 --2- 172.21.15.96:0/4241220032 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x560b31f89150 0x560b31f8c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:28.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.594+0000 7fdea3e00640 1 -- 172.21.15.96:0/4241220032 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x560b31f89150 msgr2=0x560b31f8c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:28.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.594+0000 7fdea3e00640 1 --2- 172.21.15.96:0/4241220032 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x560b31f89150 0x560b31f8c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:28.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.594+0000 7fdea3e00640 1 -- 172.21.15.96:0/4241220032 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560b31f817a0 msgr2=0x560b31f58980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:28.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.594+0000 7fdea3e00640 1 --2- 172.21.15.96:0/4241220032 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560b31f817a0 0x560b31f58980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:28.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.594+0000 7fdea3e00640 1 -- 172.21.15.96:0/4241220032 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560b31c9f6b0 con 0x560b31fa55d0 2024-05-28T06:22:28.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.594+0000 7fdea8a00640 1 --2- 172.21.15.96:0/4241220032 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x560b31f89150 0x560b31f8c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:22:28.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.594+0000 7fdea9400640 1 --2- 172.21.15.96:0/4241220032 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560b31f817a0 0x560b31f58980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:22:28.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.594+0000 7fdea3e00640 1 --2- 172.21.15.96:0/4241220032 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560b31fa55d0 0x560b31f58440 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7fde9002a220 tx=0x7fde90004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96f61f4c952b9801 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:28.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.594+0000 7fde8d600640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-05-28T06:22:28.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.595+0000 7fde8d600640 1 -- 172.21.15.96:0/4241220032 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde90003db0 con 0x560b31fa55d0 2024-05-28T06:22:28.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.595+0000 7fde8d600640 1 -- 172.21.15.96:0/4241220032 <== mon.0 v2:172.21.15.96:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 5765+0+0 (secure 0 0 0) 0x7fde90006040 con 0x560b31fa55d0 2024-05-28T06:22:28.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.595+0000 7fdeae3b8740 1 -- 172.21.15.96:0/4241220032 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fde9c00bad0 con 0x560b31fa55d0 2024-05-28T06:22:28.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.595+0000 7fde8d600640 1 -- 172.21.15.96:0/4241220032 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=72) ==== 24+0+0 (secure 0 0 0) 0x7fde90056050 con 0x560b31fa55d0 2024-05-28T06:22:28.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.595+0000 7fdeae3b8740 10 client.9432 fetch_fsmap learned FSMap version 72 2024-05-28T06:22:28.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.595+0000 7fdeae3b8740 1 -- 172.21.15.96:0/4241220032 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=72,monmap=2+}) -- 0x560b31fa0d70 con 0x560b31fa55d0 2024-05-28T06:22:28.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.595+0000 7fde8d600640 1 -- 172.21.15.96:0/4241220032 <== mon.0 v2:172.21.15.96:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde9c00bad0 con 0x560b31fa55d0 2024-05-28T06:22:28.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.595+0000 7fde8d600640 1 -- 172.21.15.96:0/4241220032 <== mon.0 v2:172.21.15.96:3300/0 5 ==== fsmap(e 72) ==== 3494+0+0 (secure 0 0 0) 0x7fde90059020 con 0x560b31fa55d0 2024-05-28T06:22:28.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.595+0000 7fdeae3b8740 10 client.9432 fetch_fsmap finished waiting for FSMap version 72 2024-05-28T06:22:28.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.595+0000 7fdeae3b8740 10 client.9432 resolve_mds: resolved name 'b' to daemon mds.b 2024-05-28T06:22:28.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.595+0000 7fdeae3b8740 20 client.9432 populate_metadata read hostname 'smithi096' 2024-05-28T06:22:28.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.595+0000 7fdeae3b8740 1 --2- 172.21.15.96:0/4241220032 >> [v2:172.21.15.123:6834/3192559785,v1:172.21.15.123:6836/3192559785] conn(0x560b31f83870 0x560b31fc22d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:28.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.595+0000 7fdeae3b8740 4 client.9432 mds_command: new command op to 9270 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-28T06:22:28.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.595+0000 7fdeae3b8740 1 -- 172.21.15.96:0/4241220032 --> [v2:172.21.15.123:6834/3192559785,v1:172.21.15.123:6836/3192559785] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560b31ed0530 con 0x560b31f83870 2024-05-28T06:22:28.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.596+0000 7fdea8a00640 1 --2- 172.21.15.96:0/4241220032 >> [v2:172.21.15.123:6834/3192559785,v1:172.21.15.123:6836/3192559785] conn(0x560b31f83870 0x560b31fc22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:28.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.597+0000 7fdea8a00640 1 --2- 172.21.15.96:0/4241220032 >> [v2:172.21.15.123:6834/3192559785,v1:172.21.15.123:6836/3192559785] conn(0x560b31f83870 0x560b31fc22d0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=77bd4166752e1e32 server_cookie=54ea9d209a3a6a98 in_seq=0 out_seq=0 2024-05-28T06:22:28.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.597+0000 7fde8d600640 10 client.9432 ms_handle_connect on v2:172.21.15.123:6834/3192559785 2024-05-28T06:22:28.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.599+0000 7fde8d600640 1 -- 172.21.15.96:0/4241220032 <== mds.0 v2:172.21.15.123:6834/3192559785 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560b31ed0530 con 0x560b31f83870 2024-05-28T06:22:28.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.599+0000 7fde8d600640 10 client.9432 handle_command_reply: tid=0 2024-05-28T06:22:28.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.599+0000 7fdeae3b8740 1 client.9432 shutdown 2024-05-28T06:22:28.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.600+0000 7fdeae3b8740 1 -- 172.21.15.96:0/4241220032 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560b31fa55d0 msgr2=0x560b31f58440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:28.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.600+0000 7fdeae3b8740 1 --2- 172.21.15.96:0/4241220032 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560b31fa55d0 0x560b31f58440 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7fde9002a220 tx=0x7fde90004730 comp rx=0 tx=0).stop 2024-05-28T06:22:28.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.600+0000 7fdeae3b8740 1 -- 172.21.15.96:0/4241220032 shutdown_connections 2024-05-28T06:22:28.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.600+0000 7fdeae3b8740 1 --2- 172.21.15.96:0/4241220032 >> [v2:172.21.15.123:6834/3192559785,v1:172.21.15.123:6836/3192559785] conn(0x560b31f83870 0x560b31fc22d0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:28.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.600+0000 7fdeae3b8740 1 --2- 172.21.15.96:0/4241220032 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560b31f817a0 0x560b31f58980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:28.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.600+0000 7fdeae3b8740 1 --2- 172.21.15.96:0/4241220032 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560b31fa55d0 0x560b31f58440 unknown :-1 s=CLOSED pgs=1834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:28.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.600+0000 7fdeae3b8740 1 --2- 172.21.15.96:0/4241220032 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x560b31f89150 0x560b31f8c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:28.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.600+0000 7fdeae3b8740 1 -- 172.21.15.96:0/4241220032 >> 172.21.15.96:0/4241220032 conn(0x560b31f85490 msgr2=0x560b31fa3be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:28.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.600+0000 7fde8d600640 0 client.9432 ms_handle_reset on v2:172.21.15.123:6834/3192559785 2024-05-28T06:22:28.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.600+0000 7fdeae3b8740 1 -- 172.21.15.96:0/4241220032 shutdown_connections 2024-05-28T06:22:28.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.600+0000 7fdeae3b8740 1 -- 172.21.15.96:0/4241220032 wait complete. 2024-05-28T06:22:28.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.600+0000 7fdeae3b8740 20 client.9432 trim_cache size 0 max 16384 2024-05-28T06:22:28.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.620+0000 7fdeae3b8740 1 Processor -- start 2024-05-28T06:22:28.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.620+0000 7fdeae3b8740 1 -- start start 2024-05-28T06:22:28.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.620+0000 7fdeae3b8740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560b31f817a0 0x560b31f8b170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:28.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.620+0000 7fdeae3b8740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560b31f89150 0x560b31f804f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:28.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.620+0000 7fdeae3b8740 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x560b31fa55d0 0x560b31f80a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:28.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.620+0000 7fdeae3b8740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fdea415fd90 con 0x560b31f89150 2024-05-28T06:22:28.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.620+0000 7fdeae3b8740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fdea415f0f0 con 0x560b31f817a0 2024-05-28T06:22:28.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.620+0000 7fdeae3b8740 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fdea415f740 con 0x560b31fa55d0 2024-05-28T06:22:28.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.621+0000 7fdea3e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560b31f89150 0x560b31f804f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:28.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.621+0000 7fdea3e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560b31f89150 0x560b31f804f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:60358/0 (socket says 172.21.15.96:60358) 2024-05-28T06:22:28.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.621+0000 7fdea3e00640 1 -- 172.21.15.96:0/3773059722 learned_addr learned my addr 172.21.15.96:0/3773059722 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:22:28.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.621+0000 7fdea3e00640 1 -- 172.21.15.96:0/3773059722 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x560b31fa55d0 msgr2=0x560b31f80a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:28.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.621+0000 7fdea3e00640 1 --2- 172.21.15.96:0/3773059722 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x560b31fa55d0 0x560b31f80a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:28.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.621+0000 7fdea3e00640 1 -- 172.21.15.96:0/3773059722 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560b31f817a0 msgr2=0x560b31f8b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:28.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.621+0000 7fdea3e00640 1 --2- 172.21.15.96:0/3773059722 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560b31f817a0 0x560b31f8b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:28.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.621+0000 7fdea3e00640 1 -- 172.21.15.96:0/3773059722 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560b31f60170 con 0x560b31f89150 2024-05-28T06:22:28.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.621+0000 7fdea3e00640 1 --2- 172.21.15.96:0/3773059722 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560b31f89150 0x560b31f804f0 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x7fde90003e90 tx=0x7fde90041070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3e082e341e3ba7f server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:28.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.622+0000 7fde8fe00640 1 -- 172.21.15.96:0/3773059722 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde90006800 con 0x560b31f89150 2024-05-28T06:22:28.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.622+0000 7fde8fe00640 1 -- 172.21.15.96:0/3773059722 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde9005f070 con 0x560b31f89150 2024-05-28T06:22:28.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.622+0000 7fde8fe00640 1 -- 172.21.15.96:0/3773059722 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde900061e0 con 0x560b31f89150 2024-05-28T06:22:28.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.622+0000 7fdeae3b8740 1 -- 172.21.15.96:0/3773059722 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560b31f89150 msgr2=0x560b31f804f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:28.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.622+0000 7fdeae3b8740 1 --2- 172.21.15.96:0/3773059722 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560b31f89150 0x560b31f804f0 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x7fde90003e90 tx=0x7fde90041070 comp rx=0 tx=0).stop 2024-05-28T06:22:28.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.622+0000 7fdeae3b8740 1 -- 172.21.15.96:0/3773059722 shutdown_connections 2024-05-28T06:22:28.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.622+0000 7fdeae3b8740 1 --2- 172.21.15.96:0/3773059722 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x560b31fa55d0 0x560b31f80a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:28.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.622+0000 7fdeae3b8740 1 --2- 172.21.15.96:0/3773059722 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560b31f89150 0x560b31f804f0 unknown :-1 s=CLOSED pgs=1835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:28.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.622+0000 7fdeae3b8740 1 --2- 172.21.15.96:0/3773059722 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560b31f817a0 0x560b31f8b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:28.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.622+0000 7fdeae3b8740 1 -- 172.21.15.96:0/3773059722 >> 172.21.15.96:0/3773059722 conn(0x560b31f85490 msgr2=0x560b31fa3be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:28.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.622+0000 7fdeae3b8740 1 -- 172.21.15.96:0/3773059722 shutdown_connections 2024-05-28T06:22:28.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.622+0000 7fdeae3b8740 1 -- 172.21.15.96:0/3773059722 wait complete. 2024-05-28T06:22:28.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.623+0000 7fdeae3b8740 1 -- start start 2024-05-28T06:22:28.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.623+0000 7fdeae3b8740 1 Processor -- start 2024-05-28T06:22:28.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.624+0000 7fdeae3b8740 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x560b31f817a0 0x560b31fc6780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:28.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.624+0000 7fdeae3b8740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560b31f89150 0x560b31f5a160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:28.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.624+0000 7fdeae3b8740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560b31fa55d0 0x560b31f5a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:28.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.624+0000 7fdeae3b8740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x560b31f9fcd0 con 0x560b31fa55d0 2024-05-28T06:22:28.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.624+0000 7fdeae3b8740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x560b31f8abe0 con 0x560b31f89150 2024-05-28T06:22:28.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.624+0000 7fdeae3b8740 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x560b31f8a700 con 0x560b31f817a0 2024-05-28T06:22:28.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.624+0000 7fdeae3b8740 10 client.0 _ll_register_callbacks cb 0x7fde90004420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-28T06:22:28.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.624+0000 7fdea9400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560b31fa55d0 0x560b31f5a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:28.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.625+0000 7fdea9400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560b31fa55d0 0x560b31f5a6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:60370/0 (socket says 172.21.15.96:60370) 2024-05-28T06:22:28.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.625+0000 7fdea9400640 1 -- 172.21.15.96:0/2940507624 learned_addr learned my addr 172.21.15.96:0/2940507624 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:22:28.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.625+0000 7fdea8a00640 1 --2- 172.21.15.96:0/2940507624 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x560b31f817a0 0x560b31fc6780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:28.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.625+0000 7fdea3e00640 1 --2- 172.21.15.96:0/2940507624 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560b31f89150 0x560b31f5a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:28.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.625+0000 7fdea9400640 1 -- 172.21.15.96:0/2940507624 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x560b31f817a0 msgr2=0x560b31fc6780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:28.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.625+0000 7fdea9400640 1 --2- 172.21.15.96:0/2940507624 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x560b31f817a0 0x560b31fc6780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:28.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.625+0000 7fdea9400640 1 -- 172.21.15.96:0/2940507624 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560b31f89150 msgr2=0x560b31f5a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:28.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.625+0000 7fdea9400640 1 --2- 172.21.15.96:0/2940507624 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560b31f89150 0x560b31f5a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:28.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.625+0000 7fdea9400640 1 -- 172.21.15.96:0/2940507624 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560b31fc6cc0 con 0x560b31fa55d0 2024-05-28T06:22:28.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.625+0000 7fdea8a00640 1 --2- 172.21.15.96:0/2940507624 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x560b31f817a0 0x560b31fc6780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:22:28.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.625+0000 7fdea9400640 1 --2- 172.21.15.96:0/2940507624 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560b31fa55d0 0x560b31f5a6a0 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7fde9c000c00 tx=0x7fde9c004840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1f2e3d03acc4f7e server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:28.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.625+0000 7fde8d600640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-05-28T06:22:28.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.625+0000 7fde8d600640 1 -- 172.21.15.96:0/2940507624 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde9c015030 con 0x560b31fa55d0 2024-05-28T06:22:28.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.626+0000 7fdeae3b8740 1 -- 172.21.15.96:0/2940507624 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fde9005f070 con 0x560b31fa55d0 2024-05-28T06:22:28.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.626+0000 7fde8d600640 1 -- 172.21.15.96:0/2940507624 <== mon.0 v2:172.21.15.96:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 5765+0+0 (secure 0 0 0) 0x7fde9c00be10 con 0x560b31fa55d0 2024-05-28T06:22:28.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.626+0000 7fde8d600640 1 -- 172.21.15.96:0/2940507624 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=72) ==== 24+0+0 (secure 0 0 0) 0x7fde9c02d050 con 0x560b31fa55d0 2024-05-28T06:22:28.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.626+0000 7fdeae3b8740 10 client.9438 fetch_fsmap learned FSMap version 72 2024-05-28T06:22:28.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.626+0000 7fdeae3b8740 1 -- 172.21.15.96:0/2940507624 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=72,monmap=2+}) -- 0x560b31fb64f0 con 0x560b31fa55d0 2024-05-28T06:22:28.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.626+0000 7fde8d600640 1 -- 172.21.15.96:0/2940507624 <== mon.0 v2:172.21.15.96:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde9005f070 con 0x560b31fa55d0 2024-05-28T06:22:28.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.626+0000 7fde8d600640 1 -- 172.21.15.96:0/2940507624 <== mon.0 v2:172.21.15.96:3300/0 5 ==== fsmap(e 72) ==== 3494+0+0 (secure 0 0 0) 0x7fde9c030020 con 0x560b31fa55d0 2024-05-28T06:22:28.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.626+0000 7fdeae3b8740 10 client.9438 fetch_fsmap finished waiting for FSMap version 72 2024-05-28T06:22:28.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.626+0000 7fdeae3b8740 10 client.9438 resolve_mds: resolved name 'b' to daemon mds.b 2024-05-28T06:22:28.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.626+0000 7fdeae3b8740 20 client.9438 populate_metadata read hostname 'smithi096' 2024-05-28T06:22:28.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.626+0000 7fdeae3b8740 1 --2- 172.21.15.96:0/2940507624 >> [v2:172.21.15.123:6834/3192559785,v1:172.21.15.123:6836/3192559785] conn(0x560b3201d970 0x560b31f581b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:28.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.626+0000 7fdeae3b8740 4 client.9438 mds_command: new command op to 9270 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-28T06:22:28.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.626+0000 7fdeae3b8740 1 -- 172.21.15.96:0/2940507624 --> [v2:172.21.15.123:6834/3192559785,v1:172.21.15.123:6836/3192559785] -- command(tid 0: {"prefix": "status"}) -- 0x560b31fa6650 con 0x560b3201d970 2024-05-28T06:22:28.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.627+0000 7fdea8a00640 1 --2- 172.21.15.96:0/2940507624 >> [v2:172.21.15.123:6834/3192559785,v1:172.21.15.123:6836/3192559785] conn(0x560b3201d970 0x560b31f581b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:28.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.628+0000 7fdea8a00640 1 --2- 172.21.15.96:0/2940507624 >> [v2:172.21.15.123:6834/3192559785,v1:172.21.15.123:6836/3192559785] conn(0x560b3201d970 0x560b31f581b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e3e7c14df8ec8675 server_cookie=e045b029fd4de0ad in_seq=0 out_seq=0 2024-05-28T06:22:28.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.628+0000 7fde8d600640 10 client.9438 ms_handle_connect on v2:172.21.15.123:6834/3192559785 2024-05-28T06:22:28.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.628+0000 7fde8d600640 1 -- 172.21.15.96:0/2940507624 <== mds.0 v2:172.21.15.123:6834/3192559785 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x560b31fa6650 con 0x560b3201d970 2024-05-28T06:22:28.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.628+0000 7fde8d600640 10 client.9438 handle_command_reply: tid=0 2024-05-28T06:22:28.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.628+0000 7fdeae3b8740 1 client.9438 shutdown 2024-05-28T06:22:28.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.629+0000 7fdeae3b8740 1 -- 172.21.15.96:0/2940507624 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560b31fa55d0 msgr2=0x560b31f5a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:28.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.629+0000 7fdeae3b8740 1 --2- 172.21.15.96:0/2940507624 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560b31fa55d0 0x560b31f5a6a0 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7fde9c000c00 tx=0x7fde9c004840 comp rx=0 tx=0).stop 2024-05-28T06:22:28.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.629+0000 7fdeae3b8740 1 -- 172.21.15.96:0/2940507624 shutdown_connections 2024-05-28T06:22:28.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.629+0000 7fdeae3b8740 1 --2- 172.21.15.96:0/2940507624 >> [v2:172.21.15.123:6834/3192559785,v1:172.21.15.123:6836/3192559785] conn(0x560b3201d970 0x560b31f581b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:28.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.629+0000 7fdeae3b8740 1 --2- 172.21.15.96:0/2940507624 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560b31fa55d0 0x560b31f5a6a0 unknown :-1 s=CLOSED pgs=1836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:28.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.629+0000 7fdeae3b8740 1 --2- 172.21.15.96:0/2940507624 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560b31f89150 0x560b31f5a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:28.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.629+0000 7fdeae3b8740 1 --2- 172.21.15.96:0/2940507624 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x560b31f817a0 0x560b31fc6780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:28.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.629+0000 7fde8d600640 0 client.9438 ms_handle_reset on v2:172.21.15.123:6834/3192559785 2024-05-28T06:22:28.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.629+0000 7fdeae3b8740 1 -- 172.21.15.96:0/2940507624 >> 172.21.15.96:0/2940507624 conn(0x560b31f85490 msgr2=0x560b31f5eb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:28.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.629+0000 7fdeae3b8740 1 -- 172.21.15.96:0/2940507624 shutdown_connections 2024-05-28T06:22:28.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.629+0000 7fdeae3b8740 1 -- 172.21.15.96:0/2940507624 wait complete. 2024-05-28T06:22:28.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.629+0000 7fdeae3b8740 20 client.9438 trim_cache size 0 max 16384 2024-05-28T06:22:28.629 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:22:28.629 INFO:teuthology.orchestra.run.smithi096.stdout: "cluster_fsid": "e4a831b5-e6d1-47a7-96cb-bf85154ee0d2", 2024-05-28T06:22:28.629 INFO:teuthology.orchestra.run.smithi096.stdout: "whoami": 0, 2024-05-28T06:22:28.629 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 9270, 2024-05-28T06:22:28.629 INFO:teuthology.orchestra.run.smithi096.stdout: "want_state": "up:active", 2024-05-28T06:22:28.629 INFO:teuthology.orchestra.run.smithi096.stdout: "state": "up:active", 2024-05-28T06:22:28.629 INFO:teuthology.orchestra.run.smithi096.stdout: "fs_name": "cephfs", 2024-05-28T06:22:28.629 INFO:teuthology.orchestra.run.smithi096.stdout: "rank_uptime": 11.623090711, 2024-05-28T06:22:28.629 INFO:teuthology.orchestra.run.smithi096.stdout: "mdsmap_epoch": 72, 2024-05-28T06:22:28.629 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap_epoch": 115, 2024-05-28T06:22:28.629 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap_epoch_barrier": 115, 2024-05-28T06:22:28.629 INFO:teuthology.orchestra.run.smithi096.stdout: "uptime": 17.097130848999999 2024-05-28T06:22:28.630 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:22:28.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.630+0000 7fdea9e00640 1 -- 172.21.15.96:0/1501386549 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fde8406e090 msgr2=0x7fde84070550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:28.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.630+0000 7fdea9e00640 1 --2- 172.21.15.96:0/1501386549 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fde8406e090 0x7fde84070550 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fde90033000 tx=0x7fde9002c000 comp rx=0 tx=0).stop 2024-05-28T06:22:28.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.630+0000 7fdea9e00640 1 -- 172.21.15.96:0/1501386549 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdea40cf870 msgr2=0x7fdea40b8db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:28.631 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.630+0000 7fdea9e00640 1 --2- 172.21.15.96:0/1501386549 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdea40cf870 0x7fdea40b8db0 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7fde9400ba50 tx=0x7fde9400bf10 comp rx=0 tx=0).stop 2024-05-28T06:22:28.631 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.630+0000 7fdea9e00640 1 -- 172.21.15.96:0/1501386549 shutdown_connections 2024-05-28T06:22:28.631 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.630+0000 7fdea9e00640 1 --2- 172.21.15.96:0/1501386549 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fde8406e090 0x7fde84070550 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:28.631 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.630+0000 7fdea9e00640 1 --2- 172.21.15.96:0/1501386549 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdea415e870 0x7fdea40b71a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:28.631 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.630+0000 7fdea9e00640 1 --2- 172.21.15.96:0/1501386549 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdea40d16a0 0x7fdea41e9610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:28.631 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.630+0000 7fdea9e00640 1 --2- 172.21.15.96:0/1501386549 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdea40cf870 0x7fdea40b8db0 unknown :-1 s=CLOSED pgs=1832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:28.631 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.630+0000 7fdea9e00640 1 -- 172.21.15.96:0/1501386549 >> 172.21.15.96:0/1501386549 conn(0x7fdea408bfc0 msgr2=0x7fdea406ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:28.631 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.630+0000 7fdea9e00640 1 -- 172.21.15.96:0/1501386549 shutdown_connections 2024-05-28T06:22:28.631 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.630+0000 7fdea9e00640 1 -- 172.21.15.96:0/1501386549 wait complete. 2024-05-28T06:22:28.642 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-05-28T06:22:28.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.836+0000 7f9e49200640 1 Processor -- start 2024-05-28T06:22:28.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.836+0000 7f9e49200640 1 -- start start 2024-05-28T06:22:28.833 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.838+0000 7f9e49200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f9e44073310 0x7f9e440b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:28.833 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.838+0000 7f9e49200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f9e4406dc20 0x7f9e4415f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:28.833 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.838+0000 7f9e49200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e440afd60 con 0x7f9e4415b3a0 2024-05-28T06:22:28.833 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.838+0000 7f9e49200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f9e4406f980 con 0x7f9e44073310 2024-05-28T06:22:28.833 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.838+0000 7f9e49200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f9e440ae0c0 con 0x7f9e4406dc20 2024-05-28T06:22:28.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.838+0000 7f9e42a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f9e44073310 0x7f9e440b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:28.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.838+0000 7f9e42a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f9e44073310 0x7f9e440b32c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47678/0 (socket says 172.21.15.96:47678) 2024-05-28T06:22:28.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.838+0000 7f9e42a00640 1 -- 172.21.15.96:0/2923514177 learned_addr learned my addr 172.21.15.96:0/2923514177 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:22:28.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.838+0000 7f9e43e00640 1 --2- 172.21.15.96:0/2923514177 >> v2:172.21.15.123:3300/0 conn(0x7f9e4406dc20 0x7f9e4415f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:28.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.839+0000 7f9e42a00640 1 -- 172.21.15.96:0/2923514177 >> v2:172.21.15.123:3300/0 conn(0x7f9e4406dc20 msgr2=0x7f9e4415f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:28.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.839+0000 7f9e42a00640 1 --2- 172.21.15.96:0/2923514177 >> v2:172.21.15.123:3300/0 conn(0x7f9e4406dc20 0x7f9e4415f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:28.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.839+0000 7f9e42a00640 1 -- 172.21.15.96:0/2923514177 >> v1:172.21.15.123:6789/0 conn(0x7f9e4415b3a0 legacy=0x7f9e44072c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:28.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.839+0000 7f9e42a00640 1 -- 172.21.15.96:0/2923514177 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e440af7a0 con 0x7f9e44073310 2024-05-28T06:22:28.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.840+0000 7f9e42a00640 1 --2- 172.21.15.96:0/2923514177 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9e44073310 0x7f9e440b32c0 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7f9e4406d6e0 tx=0x7f9e3002c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=125177017b727f60 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:28.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.840+0000 7f9e42000640 1 -- 172.21.15.96:0/2923514177 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e30038070 con 0x7f9e44073310 2024-05-28T06:22:28.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.840+0000 7f9e42000640 1 -- 172.21.15.96:0/2923514177 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e30031710 con 0x7f9e44073310 2024-05-28T06:22:28.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.840+0000 7f9e42000640 1 -- 172.21.15.96:0/2923514177 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e30031ae0 con 0x7f9e44073310 2024-05-28T06:22:28.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.840+0000 7f9e49200640 1 -- 172.21.15.96:0/2923514177 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9e44073310 msgr2=0x7f9e440b32c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:28.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.840+0000 7f9e49200640 1 --2- 172.21.15.96:0/2923514177 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9e44073310 0x7f9e440b32c0 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7f9e4406d6e0 tx=0x7f9e3002c440 comp rx=0 tx=0).stop 2024-05-28T06:22:28.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.841+0000 7f9e49200640 1 -- 172.21.15.96:0/2923514177 shutdown_connections 2024-05-28T06:22:28.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.841+0000 7f9e49200640 1 --2- 172.21.15.96:0/2923514177 >> v2:172.21.15.123:3300/0 conn(0x7f9e4406dc20 0x7f9e4415f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:28.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.841+0000 7f9e49200640 1 --2- 172.21.15.96:0/2923514177 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9e44073310 0x7f9e440b32c0 unknown :-1 s=CLOSED pgs=1837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:28.836 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.841+0000 7f9e49200640 1 -- 172.21.15.96:0/2923514177 >> 172.21.15.96:0/2923514177 conn(0x7f9e440b1fc0 msgr2=0x7f9e440b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:28.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.841+0000 7f9e49200640 1 -- 172.21.15.96:0/2923514177 shutdown_connections 2024-05-28T06:22:28.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.841+0000 7f9e49200640 1 -- 172.21.15.96:0/2923514177 wait complete. 2024-05-28T06:22:28.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.842+0000 7f9e49200640 1 Processor -- start 2024-05-28T06:22:28.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.843+0000 7f9e49200640 1 -- start start 2024-05-28T06:22:28.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.843+0000 7f9e49200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9e4406dc20 0x7f9e440ce9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:28.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.843+0000 7f9e49200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9e44073310 0x7f9e440b0030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:28.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.843+0000 7f9e49200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9e4415b3a0 0x7f9e441f7a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:28.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.843+0000 7f9e49200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f9e4415b780 con 0x7f9e44073310 2024-05-28T06:22:28.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.843+0000 7f9e49200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f9e4415fee0 con 0x7f9e4415b3a0 2024-05-28T06:22:28.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.843+0000 7f9e49200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f9e440ae550 con 0x7f9e4406dc20 2024-05-28T06:22:28.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.844+0000 7f9e42a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9e44073310 0x7f9e440b0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:28.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.844+0000 7f9e42a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9e44073310 0x7f9e440b0030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47694/0 (socket says 172.21.15.96:47694) 2024-05-28T06:22:28.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.844+0000 7f9e42a00640 1 -- 172.21.15.96:0/3455279340 learned_addr learned my addr 172.21.15.96:0/3455279340 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:22:28.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.844+0000 7f9e43400640 1 --2- 172.21.15.96:0/3455279340 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9e4406dc20 0x7f9e440ce9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:28.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.844+0000 7f9e43e00640 1 --2- 172.21.15.96:0/3455279340 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9e4415b3a0 0x7f9e441f7a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:28.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.844+0000 7f9e42a00640 1 -- 172.21.15.96:0/3455279340 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9e4406dc20 msgr2=0x7f9e440ce9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:28.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.844+0000 7f9e42a00640 1 --2- 172.21.15.96:0/3455279340 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9e4406dc20 0x7f9e440ce9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:28.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.844+0000 7f9e42a00640 1 -- 172.21.15.96:0/3455279340 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9e4415b3a0 msgr2=0x7f9e441f7a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:28.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.844+0000 7f9e42a00640 1 --2- 172.21.15.96:0/3455279340 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9e4415b3a0 0x7f9e441f7a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:28.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.844+0000 7f9e42a00640 1 -- 172.21.15.96:0/3455279340 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e440b0af0 con 0x7f9e44073310 2024-05-28T06:22:28.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.844+0000 7f9e43400640 1 --2- 172.21.15.96:0/3455279340 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9e4406dc20 0x7f9e440ce9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:22:28.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.844+0000 7f9e43e00640 1 --2- 172.21.15.96:0/3455279340 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9e4415b3a0 0x7f9e441f7a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:22:28.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.844+0000 7f9e42a00640 1 --2- 172.21.15.96:0/3455279340 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9e44073310 0x7f9e440b0030 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7f9e30000c00 tx=0x7f9e300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:28.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.845+0000 7f9e3b400640 1 -- 172.21.15.96:0/3455279340 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e30038040 con 0x7f9e44073310 2024-05-28T06:22:28.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.845+0000 7f9e3b400640 1 -- 172.21.15.96:0/3455279340 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e30043070 con 0x7f9e44073310 2024-05-28T06:22:28.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.845+0000 7f9e3b400640 1 -- 172.21.15.96:0/3455279340 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e30033a50 con 0x7f9e44073310 2024-05-28T06:22:28.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.845+0000 7f9e49200640 1 -- 172.21.15.96:0/3455279340 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e440b0dd0 con 0x7f9e44073310 2024-05-28T06:22:28.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.845+0000 7f9e49200640 1 -- 172.21.15.96:0/3455279340 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e440b1390 con 0x7f9e44073310 2024-05-28T06:22:28.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.846+0000 7f9e49200640 1 -- 172.21.15.96:0/3455279340 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e440689e0 con 0x7f9e44073310 2024-05-28T06:22:28.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.847+0000 7f9e3b400640 1 -- 172.21.15.96:0/3455279340 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 37) ==== 93182+0+0 (secure 0 0 0) 0x7f9e30033bf0 con 0x7f9e44073310 2024-05-28T06:22:28.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.848+0000 7f9e3b400640 1 --2- 172.21.15.96:0/3455279340 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9e2406e0c0 0x7f9e24070580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:28.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.848+0000 7f9e3b400640 1 -- 172.21.15.96:0/3455279340 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 5765+0+0 (secure 0 0 0) 0x7f9e300ab560 con 0x7f9e44073310 2024-05-28T06:22:28.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.849+0000 7f9e43400640 1 --2- 172.21.15.96:0/3455279340 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9e2406e0c0 0x7f9e24070580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:28.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.850+0000 7f9e43400640 1 --2- 172.21.15.96:0/3455279340 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9e2406e0c0 0x7f9e24070580 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f9e3400a810 tx=0x7f9e34008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:28.847 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:28.852+0000 7f9e3b400640 1 -- 172.21.15.96:0/3455279340 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9e300b1e20 con 0x7f9e44073310 2024-05-28T06:22:29.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:29.093+0000 7f9e49200640 1 -- 172.21.15.96:0/3455279340 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f9e440b34a0 con 0x7f9e44073310 2024-05-28T06:22:29.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:29.094+0000 7f9e3b400640 1 -- 172.21.15.96:0/3455279340 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v72) ==== 53+0+73 (secure 0 0 0) 0x7f9e3007b510 con 0x7f9e44073310 2024-05-28T06:22:29.089 INFO:teuthology.orchestra.run.smithi096.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-05-28T06:22:29.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:29.096+0000 7f9e49200640 1 -- 172.21.15.96:0/3455279340 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9e2406e0c0 msgr2=0x7f9e24070580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:29.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:29.096+0000 7f9e49200640 1 --2- 172.21.15.96:0/3455279340 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9e2406e0c0 0x7f9e24070580 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f9e3400a810 tx=0x7f9e34008040 comp rx=0 tx=0).stop 2024-05-28T06:22:29.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:29.096+0000 7f9e49200640 1 -- 172.21.15.96:0/3455279340 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9e44073310 msgr2=0x7f9e440b0030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:29.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:29.096+0000 7f9e49200640 1 --2- 172.21.15.96:0/3455279340 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9e44073310 0x7f9e440b0030 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7f9e30000c00 tx=0x7f9e300046c0 comp rx=0 tx=0).stop 2024-05-28T06:22:29.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:29.096+0000 7f9e49200640 1 -- 172.21.15.96:0/3455279340 shutdown_connections 2024-05-28T06:22:29.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:29.096+0000 7f9e49200640 1 --2- 172.21.15.96:0/3455279340 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9e2406e0c0 0x7f9e24070580 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:29.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:29.096+0000 7f9e49200640 1 --2- 172.21.15.96:0/3455279340 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9e4415b3a0 0x7f9e441f7a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:29.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:29.096+0000 7f9e49200640 1 --2- 172.21.15.96:0/3455279340 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9e44073310 0x7f9e440b0030 unknown :-1 s=CLOSED pgs=1838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:29.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:29.096+0000 7f9e49200640 1 --2- 172.21.15.96:0/3455279340 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9e4406dc20 0x7f9e440ce9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:29.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:29.096+0000 7f9e49200640 1 -- 172.21.15.96:0/3455279340 >> 172.21.15.96:0/3455279340 conn(0x7f9e440b1fc0 msgr2=0x7f9e4415ef20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:29.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:29.096+0000 7f9e49200640 1 -- 172.21.15.96:0/3455279340 shutdown_connections 2024-05-28T06:22:29.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:29.096+0000 7f9e49200640 1 -- 172.21.15.96:0/3455279340 wait complete. 2024-05-28T06:22:29.111 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-05-28T06:22:29.111 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-28T06:22:29.111 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi096.front.sepia.ceph.com 2024-05-28T06:22:29.111 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-05-28T06:22:29.111 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-28T06:22:29.111 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-05-28T06:22:29.111 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-05-28T06:22:29.111 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-28T06:22:29.111 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-28T06:22:29.111 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:22:29.112 DEBUG:teuthology.orchestra.run.smithi096:> ip addr 2024-05-28T06:22:29.177 INFO:teuthology.orchestra.run.smithi096.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-28T06:22:29.177 INFO:teuthology.orchestra.run.smithi096.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-28T06:22:29.177 INFO:teuthology.orchestra.run.smithi096.stdout: inet 127.0.0.1/8 scope host lo 2024-05-28T06:22:29.177 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:22:29.177 INFO:teuthology.orchestra.run.smithi096.stdout: inet6 ::1/128 scope host 2024-05-28T06:22:29.177 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:22:29.177 INFO:teuthology.orchestra.run.smithi096.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-28T06:22:29.177 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether 0c:c4:7a:d8:d9:bc brd ff:ff:ff:ff:ff:ff 2024-05-28T06:22:29.177 INFO:teuthology.orchestra.run.smithi096.stdout: altname enp7s0f0 2024-05-28T06:22:29.178 INFO:teuthology.orchestra.run.smithi096.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-28T06:22:29.178 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether 0c:c4:7a:d8:d9:bd brd ff:ff:ff:ff:ff:ff 2024-05-28T06:22:29.178 INFO:teuthology.orchestra.run.smithi096.stdout: altname enp7s0f1 2024-05-28T06:22:29.178 INFO:teuthology.orchestra.run.smithi096.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-28T06:22:29.178 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether 0c:c4:7a:8f:cc:0c brd ff:ff:ff:ff:ff:ff 2024-05-28T06:22:29.178 INFO:teuthology.orchestra.run.smithi096.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-28T06:22:29.178 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether 0c:c4:7a:8f:cc:0d brd ff:ff:ff:ff:ff:ff 2024-05-28T06:22:29.178 INFO:teuthology.orchestra.run.smithi096.stdout: inet 172.21.15.96/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-05-28T06:22:29.178 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft 41488sec preferred_lft 41488sec 2024-05-28T06:22:29.178 INFO:teuthology.orchestra.run.smithi096.stdout: inet6 fe80::ec4:7aff:fe8f:cc0d/64 scope link 2024-05-28T06:22:29.178 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:22:29.178 INFO:teuthology.orchestra.run.smithi096.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-05-28T06:22:29.178 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether ae:d9:f5:b0:4f:f3 brd ff:ff:ff:ff:ff:ff 2024-05-28T06:22:29.179 INFO:teuthology.orchestra.run.smithi096.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-05-28T06:22:29.179 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:22:29.179 INFO:teuthology.orchestra.run.smithi096.stdout: inet6 fe80::604e:fcff:fe60:7892/64 scope link 2024-05-28T06:22:29.179 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:22:29.179 INFO:teuthology.orchestra.run.smithi096.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-28T06:22:29.179 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether ce:6f:19:93:44:f9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-28T06:22:29.179 INFO:teuthology.orchestra.run.smithi096.stdout: inet6 fe80::cc6f:19ff:fe93:44f9/64 scope link 2024-05-28T06:22:29.179 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:22:29.179 INFO:teuthology.orchestra.run.smithi096.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-28T06:22:29.179 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether ae:d9:f5:b0:4f:f3 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-28T06:22:29.179 INFO:teuthology.orchestra.run.smithi096.stdout: inet6 fe80::acd9:f5ff:feb0:4ff3/64 scope link 2024-05-28T06:22:29.179 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:22:29.180 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:22:29.180 DEBUG:teuthology.orchestra.run.smithi096:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-28T06:22:29.308 INFO:teuthology.orchestra.run.smithi096.stdout:1 2024-05-28T06:22:29.312 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:22:29.312 DEBUG:teuthology.orchestra.run.smithi096:> route 2024-05-28T06:22:29.366 INFO:teuthology.orchestra.run.smithi096.stdout:Kernel IP routing table 2024-05-28T06:22:29.367 INFO:teuthology.orchestra.run.smithi096.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-28T06:22:29.367 INFO:teuthology.orchestra.run.smithi096.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-05-28T06:22:29.367 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-05-28T06:22:29.367 INFO:teuthology.orchestra.run.smithi096.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-28T06:22:29.367 DEBUG:teuthology.orchestra.run.smithi096:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-28T06:22:29.367 DEBUG:teuthology.orchestra.run.smithi096:> set -e 2024-05-28T06:22:29.367 DEBUG:teuthology.orchestra.run.smithi096:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-05-28T06:22:29.368 DEBUG:teuthology.orchestra.run.smithi096:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-05-28T06:22:29.368 DEBUG:teuthology.orchestra.run.smithi096:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-05-28T06:22:29.368 DEBUG:teuthology.orchestra.run.smithi096:> ') 2024-05-28T06:22:29.475 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:29 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-28T06:22:29.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-28T06:22:29.577 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:22:29.577 DEBUG:teuthology.orchestra.run.smithi096:> ip netns list 2024-05-28T06:22:29.629 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-05-28T06:22:29.629 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-05-28T06:22:29.630 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-28T06:22:29.630 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:22:29.630 DEBUG:teuthology.orchestra.run.smithi096:> sudo ip link set brx.0 up 2024-05-28T06:22:29.720 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-05-28T06:22:29.720 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-28T06:22:29.720 DEBUG:teuthology.orchestra.run.smithi096:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-05-28T06:22:29.774 INFO:teuthology.orchestra.run.smithi096.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-05-28T06:22:29.775 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-28T06:22:29.775 DEBUG:teuthology.orchestra.run.smithi096:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-05-28T06:22:29.831 DEBUG:teuthology.orchestra.run.smithi096:> sudo modprobe fuse 2024-05-28T06:22:29.921 DEBUG:teuthology.orchestra.run.smithi096:> cat /proc/self/mounts | awk '{print $2}' 2024-05-28T06:22:29.978 INFO:teuthology.orchestra.run.smithi096.stdout:/proc 2024-05-28T06:22:29.978 INFO:teuthology.orchestra.run.smithi096.stdout:/sys 2024-05-28T06:22:29.978 INFO:teuthology.orchestra.run.smithi096.stdout:/dev 2024-05-28T06:22:29.978 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/security 2024-05-28T06:22:29.978 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/shm 2024-05-28T06:22:29.978 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/pts 2024-05-28T06:22:29.978 INFO:teuthology.orchestra.run.smithi096.stdout:/run 2024-05-28T06:22:29.978 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/cgroup 2024-05-28T06:22:29.978 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/pstore 2024-05-28T06:22:29.978 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/bpf 2024-05-28T06:22:29.978 INFO:teuthology.orchestra.run.smithi096.stdout:/ 2024-05-28T06:22:29.979 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/selinux 2024-05-28T06:22:29.979 INFO:teuthology.orchestra.run.smithi096.stdout:/proc/sys/fs/binfmt_misc 2024-05-28T06:22:29.979 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/mqueue 2024-05-28T06:22:29.979 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/debug 2024-05-28T06:22:29.979 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/hugepages 2024-05-28T06:22:29.979 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/tracing 2024-05-28T06:22:29.979 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/fuse/connections 2024-05-28T06:22:29.979 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/config 2024-05-28T06:22:29.979 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-sysctl.service 2024-05-28T06:22:29.979 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-28T06:22:29.979 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-28T06:22:29.979 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/nfs/rpc_pipefs 2024-05-28T06:22:29.980 INFO:teuthology.orchestra.run.smithi096.stdout:/run/user/1000 2024-05-28T06:22:29.980 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph 2024-05-28T06:22:29.980 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-0 2024-05-28T06:22:29.980 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-1 2024-05-28T06:22:29.980 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-2 2024-05-28T06:22:29.980 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-3 2024-05-28T06:22:29.980 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns 2024-05-28T06:22:29.980 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-28T06:22:29.980 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-28T06:22:29.980 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-28T06:22:29.980 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-28T06:22:29.982 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:22:29.982 DEBUG:teuthology.orchestra.run.smithi096:> ls /sys/fs/fuse/connections 2024-05-28T06:22:30.035 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-05-28T06:22:30.035 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-05-28T06:22:30.040 DEBUG:teuthology.orchestra.run.smithi096:> sudo modprobe fuse 2024-05-28T06:22:30.108 DEBUG:teuthology.orchestra.run.smithi096:> cat /proc/self/mounts | awk '{print $2}' 2024-05-28T06:22:30.150 INFO:teuthology.orchestra.run.smithi096.stdout:/proc 2024-05-28T06:22:30.151 INFO:teuthology.orchestra.run.smithi096.stdout:/sys 2024-05-28T06:22:30.151 INFO:teuthology.orchestra.run.smithi096.stdout:/dev 2024-05-28T06:22:30.151 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/security 2024-05-28T06:22:30.151 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/shm 2024-05-28T06:22:30.151 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/pts 2024-05-28T06:22:30.151 INFO:teuthology.orchestra.run.smithi096.stdout:/run 2024-05-28T06:22:30.151 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/cgroup 2024-05-28T06:22:30.151 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/pstore 2024-05-28T06:22:30.152 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/bpf 2024-05-28T06:22:30.152 INFO:teuthology.orchestra.run.smithi096.stdout:/ 2024-05-28T06:22:30.152 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/selinux 2024-05-28T06:22:30.152 INFO:teuthology.orchestra.run.smithi096.stdout:/proc/sys/fs/binfmt_misc 2024-05-28T06:22:30.152 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/mqueue 2024-05-28T06:22:30.153 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/debug 2024-05-28T06:22:30.153 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/hugepages 2024-05-28T06:22:30.153 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/tracing 2024-05-28T06:22:30.153 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/fuse/connections 2024-05-28T06:22:30.153 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/config 2024-05-28T06:22:30.153 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-sysctl.service 2024-05-28T06:22:30.153 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-28T06:22:30.153 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-28T06:22:30.153 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/nfs/rpc_pipefs 2024-05-28T06:22:30.153 INFO:teuthology.orchestra.run.smithi096.stdout:/run/user/1000 2024-05-28T06:22:30.154 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph 2024-05-28T06:22:30.154 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-0 2024-05-28T06:22:30.154 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-1 2024-05-28T06:22:30.154 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-2 2024-05-28T06:22:30.154 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-3 2024-05-28T06:22:30.154 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns 2024-05-28T06:22:30.154 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-28T06:22:30.154 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-28T06:22:30.154 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-28T06:22:30.154 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-28T06:22:30.155 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:22:30.155 DEBUG:teuthology.orchestra.run.smithi096:> ls /sys/fs/fuse/connections 2024-05-28T06:22:30.209 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi096.stderr:2024-05-28T06:22:30.214+0000 7f0440b1e480 -1 init, newargv = 0x55dc356e8c50 newargc=15 2024-05-28T06:22:30.209 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi096.stderr:ceph-fuse[79827]: starting ceph client 2024-05-28T06:22:30.218 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi096.stderr:ceph-fuse[79827]: starting fuse 2024-05-28T06:22:30.242 INFO:teuthology.orchestra.run.smithi096.stdout:40 2024-05-28T06:22:30.242 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40] 2024-05-28T06:22:30.242 DEBUG:teuthology.orchestra.run.smithi096:> sudo stdin-killer -- python3 -c ' 2024-05-28T06:22:30.243 DEBUG:teuthology.orchestra.run.smithi096:> import glob 2024-05-28T06:22:30.243 DEBUG:teuthology.orchestra.run.smithi096:> import re 2024-05-28T06:22:30.243 DEBUG:teuthology.orchestra.run.smithi096:> import os 2024-05-28T06:22:30.243 DEBUG:teuthology.orchestra.run.smithi096:> import subprocess 2024-05-28T06:22:30.243 DEBUG:teuthology.orchestra.run.smithi096:> 2024-05-28T06:22:30.243 DEBUG:teuthology.orchestra.run.smithi096:> def _find_admin_socket(client_name): 2024-05-28T06:22:30.243 DEBUG:teuthology.orchestra.run.smithi096:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-05-28T06:22:30.243 DEBUG:teuthology.orchestra.run.smithi096:> files = glob.glob(asok_path) 2024-05-28T06:22:30.243 DEBUG:teuthology.orchestra.run.smithi096:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-05-28T06:22:30.243 DEBUG:teuthology.orchestra.run.smithi096:> 2024-05-28T06:22:30.243 DEBUG:teuthology.orchestra.run.smithi096:> # Given a non-glob path, it better be there 2024-05-28T06:22:30.243 DEBUG:teuthology.orchestra.run.smithi096:> if "*" not in asok_path: 2024-05-28T06:22:30.243 DEBUG:teuthology.orchestra.run.smithi096:> assert(len(files) == 1) 2024-05-28T06:22:30.243 DEBUG:teuthology.orchestra.run.smithi096:> return files[0] 2024-05-28T06:22:30.244 DEBUG:teuthology.orchestra.run.smithi096:> 2024-05-28T06:22:30.244 DEBUG:teuthology.orchestra.run.smithi096:> for f in files: 2024-05-28T06:22:30.244 DEBUG:teuthology.orchestra.run.smithi096:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-05-28T06:22:30.244 DEBUG:teuthology.orchestra.run.smithi096:> if os.path.exists("/proc/{0}".format(pid)): 2024-05-28T06:22:30.244 DEBUG:teuthology.orchestra.run.smithi096:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-05-28T06:22:30.244 DEBUG:teuthology.orchestra.run.smithi096:> contents = proc_f.read() 2024-05-28T06:22:30.244 DEBUG:teuthology.orchestra.run.smithi096:> if mountpoint in contents: 2024-05-28T06:22:30.244 DEBUG:teuthology.orchestra.run.smithi096:> return f 2024-05-28T06:22:30.244 DEBUG:teuthology.orchestra.run.smithi096:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-05-28T06:22:30.244 DEBUG:teuthology.orchestra.run.smithi096:> 2024-05-28T06:22:30.244 DEBUG:teuthology.orchestra.run.smithi096:> print(_find_admin_socket("client.0")) 2024-05-28T06:22:30.244 DEBUG:teuthology.orchestra.run.smithi096:> ' 2024-05-28T06:22:30.431 INFO:teuthology.orchestra.run.smithi096.stdout:/var/run/ceph/ceph-client.0.79827.asok 2024-05-28T06:22:30.436 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:30 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-28T06:22:30.448 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.79827.asok 2024-05-28T06:22:30.448 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:22:30.448 DEBUG:teuthology.orchestra.run.smithi096:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.79827.asok status 2024-05-28T06:22:30.685 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:22:30.685 INFO:teuthology.orchestra.run.smithi096.stdout: "metadata": { 2024-05-28T06:22:30.685 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph_sha1": "9ed025fbf4406fb54980a0647506b15895b6d863", 2024-05-28T06:22:30.685 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph_version": "ceph version 19.0.0-3154-g9ed025fb (9ed025fbf4406fb54980a0647506b15895b6d863) squid (dev)", 2024-05-28T06:22:30.685 INFO:teuthology.orchestra.run.smithi096.stdout: "entity_id": "0", 2024-05-28T06:22:30.685 INFO:teuthology.orchestra.run.smithi096.stdout: "hostname": "smithi096", 2024-05-28T06:22:30.685 INFO:teuthology.orchestra.run.smithi096.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-05-28T06:22:30.685 INFO:teuthology.orchestra.run.smithi096.stdout: "pid": "79827", 2024-05-28T06:22:30.685 INFO:teuthology.orchestra.run.smithi096.stdout: "root": "/" 2024-05-28T06:22:30.686 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:22:30.686 INFO:teuthology.orchestra.run.smithi096.stdout: "dentry_count": 0, 2024-05-28T06:22:30.686 INFO:teuthology.orchestra.run.smithi096.stdout: "dentry_pinned_count": 0, 2024-05-28T06:22:30.686 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 9447, 2024-05-28T06:22:30.686 INFO:teuthology.orchestra.run.smithi096.stdout: "inst": { 2024-05-28T06:22:30.686 INFO:teuthology.orchestra.run.smithi096.stdout: "name": { 2024-05-28T06:22:30.686 INFO:teuthology.orchestra.run.smithi096.stdout: "type": "client", 2024-05-28T06:22:30.686 INFO:teuthology.orchestra.run.smithi096.stdout: "num": 9447 2024-05-28T06:22:30.686 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:22:30.686 INFO:teuthology.orchestra.run.smithi096.stdout: "addr": { 2024-05-28T06:22:30.686 INFO:teuthology.orchestra.run.smithi096.stdout: "type": "v1", 2024-05-28T06:22:30.686 INFO:teuthology.orchestra.run.smithi096.stdout: "addr": "192.168.0.1:0", 2024-05-28T06:22:30.686 INFO:teuthology.orchestra.run.smithi096.stdout: "nonce": 454469433 2024-05-28T06:22:30.687 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:22:30.687 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:22:30.687 INFO:teuthology.orchestra.run.smithi096.stdout: "addr": { 2024-05-28T06:22:30.687 INFO:teuthology.orchestra.run.smithi096.stdout: "type": "v1", 2024-05-28T06:22:30.687 INFO:teuthology.orchestra.run.smithi096.stdout: "addr": "192.168.0.1:0", 2024-05-28T06:22:30.687 INFO:teuthology.orchestra.run.smithi096.stdout: "nonce": 454469433 2024-05-28T06:22:30.687 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:22:30.687 INFO:teuthology.orchestra.run.smithi096.stdout: "inst_str": "client.9447 192.168.0.1:0/454469433", 2024-05-28T06:22:30.687 INFO:teuthology.orchestra.run.smithi096.stdout: "addr_str": "192.168.0.1:0/454469433", 2024-05-28T06:22:30.687 INFO:teuthology.orchestra.run.smithi096.stdout: "inode_count": 1, 2024-05-28T06:22:30.687 INFO:teuthology.orchestra.run.smithi096.stdout: "mds_epoch": 72, 2024-05-28T06:22:30.687 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_epoch": 115, 2024-05-28T06:22:30.687 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_epoch_barrier": 0, 2024-05-28T06:22:30.687 INFO:teuthology.orchestra.run.smithi096.stdout: "blocklisted": false, 2024-05-28T06:22:30.688 INFO:teuthology.orchestra.run.smithi096.stdout: "fs_name": "cephfs" 2024-05-28T06:22:30.688 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:22:30.700 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:22:30.700 DEBUG:teuthology.orchestra.run.smithi096:> stat --file-system '--printf=%T 2024-05-28T06:22:30.700 DEBUG:teuthology.orchestra.run.smithi096:> ' -- /home/ubuntu/cephtest/mnt.0 2024-05-28T06:22:30.752 INFO:teuthology.orchestra.run.smithi096.stdout:fuseblk 2024-05-28T06:22:30.752 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-05-28T06:22:30.752 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:22:30.752 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-05-28T06:22:30.885 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-05-28T06:22:31.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.105+0000 7fcf23a00640 1 Processor -- start 2024-05-28T06:22:31.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.105+0000 7fcf23a00640 1 -- start start 2024-05-28T06:22:31.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.107+0000 7fcf23a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcf1c15b340 0x7fcf1c06f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:31.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.107+0000 7fcf23a00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fcf1c15b720 0x7fcf1c15bb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:31.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.107+0000 7fcf23a00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf1c06fa90 con 0x7fcf1c15c0c0 2024-05-28T06:22:31.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.107+0000 7fcf23a00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fcf1c15c510 con 0x7fcf1c15b720 2024-05-28T06:22:31.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.107+0000 7fcf23a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fcf1c0af9b0 con 0x7fcf1c15b340 2024-05-28T06:22:31.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.108+0000 7fcf21000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcf1c15b340 0x7fcf1c06f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:31.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.108+0000 7fcf13e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fcf1c15b720 0x7fcf1c15bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:31.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.108+0000 7fcf21000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcf1c15b340 0x7fcf1c06f550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.96:51152/0 (socket says 172.21.15.96:51152) 2024-05-28T06:22:31.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.108+0000 7fcf13e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fcf1c15b720 0x7fcf1c15bb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.96:55930/0 (socket says 172.21.15.96:55930) 2024-05-28T06:22:31.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.108+0000 7fcf21000640 1 -- 172.21.15.96:0/500565402 learned_addr learned my addr 172.21.15.96:0/500565402 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:22:31.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.108+0000 7fcf21000640 1 -- 172.21.15.96:0/500565402 >> v2:172.21.15.123:3300/0 conn(0x7fcf1c15b720 msgr2=0x7fcf1c15bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:31.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.108+0000 7fcf21000640 1 --2- 172.21.15.96:0/500565402 >> v2:172.21.15.123:3300/0 conn(0x7fcf1c15b720 0x7fcf1c15bb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:31.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.108+0000 7fcf21000640 1 -- 172.21.15.96:0/500565402 >> v1:172.21.15.123:6789/0 conn(0x7fcf1c15c0c0 legacy=0x7fcf1c15f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:31.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.108+0000 7fcf21000640 1 -- 172.21.15.96:0/500565402 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf1c0aeda0 con 0x7fcf1c15b340 2024-05-28T06:22:31.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.110+0000 7fcf21000640 1 --2- 172.21.15.96:0/500565402 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcf1c15b340 0x7fcf1c06f550 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c009870 tx=0x7fcf0c02a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f7f09985ca29403 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:31.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.110+0000 7fcf1be00640 1 -- 172.21.15.96:0/500565402 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf0c037070 con 0x7fcf1c15b340 2024-05-28T06:22:31.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.110+0000 7fcf1be00640 1 -- 172.21.15.96:0/500565402 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf0c032440 con 0x7fcf1c15b340 2024-05-28T06:22:31.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.110+0000 7fcf1be00640 1 -- 172.21.15.96:0/500565402 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf0c032810 con 0x7fcf1c15b340 2024-05-28T06:22:31.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.111+0000 7fcf23a00640 1 -- 172.21.15.96:0/500565402 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcf1c15b340 msgr2=0x7fcf1c06f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:31.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.111+0000 7fcf23a00640 1 --2- 172.21.15.96:0/500565402 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcf1c15b340 0x7fcf1c06f550 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c009870 tx=0x7fcf0c02a270 comp rx=0 tx=0).stop 2024-05-28T06:22:31.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.111+0000 7fcf23a00640 1 -- 172.21.15.96:0/500565402 shutdown_connections 2024-05-28T06:22:31.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.111+0000 7fcf23a00640 1 --2- 172.21.15.96:0/500565402 >> v2:172.21.15.123:3300/0 conn(0x7fcf1c15b720 0x7fcf1c15bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:31.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.111+0000 7fcf23a00640 1 --2- 172.21.15.96:0/500565402 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcf1c15b340 0x7fcf1c06f550 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:31.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.111+0000 7fcf23a00640 1 -- 172.21.15.96:0/500565402 >> 172.21.15.96:0/500565402 conn(0x7fcf1c0b1e50 msgr2=0x7fcf1c0b2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:31.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.111+0000 7fcf23a00640 1 -- 172.21.15.96:0/500565402 shutdown_connections 2024-05-28T06:22:31.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.111+0000 7fcf23a00640 1 -- 172.21.15.96:0/500565402 wait complete. 2024-05-28T06:22:31.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.113+0000 7fcf23a00640 1 Processor -- start 2024-05-28T06:22:31.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.113+0000 7fcf23a00640 1 -- start start 2024-05-28T06:22:31.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.114+0000 7fcf23a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcf1c15b340 0x7fcf1c0999f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:31.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.114+0000 7fcf23a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcf1c15b720 0x7fcf1c1f8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:31.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.114+0000 7fcf23a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf1c15c0c0 0x7fcf1c159e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:31.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.114+0000 7fcf23a00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fcf1c0af640 con 0x7fcf1c15b340 2024-05-28T06:22:31.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.114+0000 7fcf23a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fcf1c0af360 con 0x7fcf1c15c0c0 2024-05-28T06:22:31.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.114+0000 7fcf23a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fcf1c0aea30 con 0x7fcf1c15b720 2024-05-28T06:22:31.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.114+0000 7fcf21000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcf1c15b340 0x7fcf1c0999f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:31.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.114+0000 7fcf21000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcf1c15b340 0x7fcf1c0999f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47698/0 (socket says 172.21.15.96:47698) 2024-05-28T06:22:31.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.114+0000 7fcf21000640 1 -- 172.21.15.96:0/219668268 learned_addr learned my addr 172.21.15.96:0/219668268 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:22:31.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.114+0000 7fcf13e00640 1 --2- 172.21.15.96:0/219668268 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcf1c15b720 0x7fcf1c1f8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:31.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.114+0000 7fcf21a00640 1 --2- 172.21.15.96:0/219668268 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf1c15c0c0 0x7fcf1c159e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:31.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.115+0000 7fcf21000640 1 -- 172.21.15.96:0/219668268 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcf1c15b720 msgr2=0x7fcf1c1f8a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:31.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.115+0000 7fcf21000640 1 --2- 172.21.15.96:0/219668268 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcf1c15b720 0x7fcf1c1f8a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:31.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.115+0000 7fcf21000640 1 -- 172.21.15.96:0/219668268 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf1c15c0c0 msgr2=0x7fcf1c159e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:31.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.115+0000 7fcf21000640 1 --2- 172.21.15.96:0/219668268 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf1c15c0c0 0x7fcf1c159e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:31.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.115+0000 7fcf21000640 1 -- 172.21.15.96:0/219668268 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf1c15d700 con 0x7fcf1c15b340 2024-05-28T06:22:31.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.115+0000 7fcf21000640 1 --2- 172.21.15.96:0/219668268 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcf1c15b340 0x7fcf1c0999f0 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c02a2b0 tx=0x7fcf0c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:31.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.115+0000 7fcf19600640 1 -- 172.21.15.96:0/219668268 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf0c037040 con 0x7fcf1c15b340 2024-05-28T06:22:31.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.115+0000 7fcf19600640 1 -- 172.21.15.96:0/219668268 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf0c042070 con 0x7fcf1c15b340 2024-05-28T06:22:31.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.115+0000 7fcf19600640 1 -- 172.21.15.96:0/219668268 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf0c030880 con 0x7fcf1c15b340 2024-05-28T06:22:31.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.115+0000 7fcf23a00640 1 -- 172.21.15.96:0/219668268 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf1c15d9e0 con 0x7fcf1c15b340 2024-05-28T06:22:31.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.115+0000 7fcf23a00640 1 -- 172.21.15.96:0/219668268 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf1c15dfa0 con 0x7fcf1c15b340 2024-05-28T06:22:31.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.116+0000 7fcf23a00640 1 -- 172.21.15.96:0/219668268 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf1c0689e0 con 0x7fcf1c15b340 2024-05-28T06:22:31.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.118+0000 7fcf19600640 1 -- 172.21.15.96:0/219668268 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 37) ==== 93182+0+0 (secure 0 0 0) 0x7fcf0c030a20 con 0x7fcf1c15b340 2024-05-28T06:22:31.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.118+0000 7fcf19600640 1 --2- 172.21.15.96:0/219668268 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fcef806e090 0x7fcef8070550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:31.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.119+0000 7fcf19600640 1 -- 172.21.15.96:0/219668268 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 5765+0+0 (secure 0 0 0) 0x7fcf0c0ab210 con 0x7fcf1c15b340 2024-05-28T06:22:31.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.119+0000 7fcf13e00640 1 --2- 172.21.15.96:0/219668268 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fcef806e090 0x7fcef8070550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:31.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.120+0000 7fcf13e00640 1 --2- 172.21.15.96:0/219668268 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fcef806e090 0x7fcef8070550 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c1f8010 tx=0x7fcf04008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:31.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.123+0000 7fcf19600640 1 -- 172.21.15.96:0/219668268 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fcf0c07b0a0 con 0x7fcf1c15b340 2024-05-28T06:22:31.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.364+0000 7fcf23a00640 1 -- 172.21.15.96:0/219668268 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) -- 0x7fcf1c0b4940 con 0x7fcf1c15b340 2024-05-28T06:22:31.471 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.472+0000 7fcf19600640 1 -- 172.21.15.96:0/219668268 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v73) ==== 136+0+0 (secure 0 0 0) 0x7fcf0c07a9a0 con 0x7fcf1c15b340 2024-05-28T06:22:31.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.475+0000 7fcf23a00640 1 -- 172.21.15.96:0/219668268 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fcef806e090 msgr2=0x7fcef8070550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:31.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.475+0000 7fcf23a00640 1 --2- 172.21.15.96:0/219668268 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fcef806e090 0x7fcef8070550 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c1f8010 tx=0x7fcf04008040 comp rx=0 tx=0).stop 2024-05-28T06:22:31.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.475+0000 7fcf23a00640 1 -- 172.21.15.96:0/219668268 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcf1c15b340 msgr2=0x7fcf1c0999f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:31.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.475+0000 7fcf23a00640 1 --2- 172.21.15.96:0/219668268 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcf1c15b340 0x7fcf1c0999f0 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c02a2b0 tx=0x7fcf0c0046c0 comp rx=0 tx=0).stop 2024-05-28T06:22:31.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.475+0000 7fcf23a00640 1 -- 172.21.15.96:0/219668268 shutdown_connections 2024-05-28T06:22:31.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.475+0000 7fcf23a00640 1 --2- 172.21.15.96:0/219668268 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fcef806e090 0x7fcef8070550 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:31.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.475+0000 7fcf23a00640 1 --2- 172.21.15.96:0/219668268 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf1c15c0c0 0x7fcf1c159e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:31.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.475+0000 7fcf23a00640 1 --2- 172.21.15.96:0/219668268 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcf1c15b720 0x7fcf1c1f8a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:31.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.475+0000 7fcf23a00640 1 --2- 172.21.15.96:0/219668268 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcf1c15b340 0x7fcf1c0999f0 unknown :-1 s=CLOSED pgs=1840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:31.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.475+0000 7fcf23a00640 1 -- 172.21.15.96:0/219668268 >> 172.21.15.96:0/219668268 conn(0x7fcf1c0b1e50 msgr2=0x7fcf1c154dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:31.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.475+0000 7fcf23a00640 1 -- 172.21.15.96:0/219668268 shutdown_connections 2024-05-28T06:22:31.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.475+0000 7fcf23a00640 1 -- 172.21.15.96:0/219668268 wait complete. 2024-05-28T06:22:31.491 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-05-28T06:22:31.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.696+0000 7ff4df000640 1 Processor -- start 2024-05-28T06:22:31.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.697+0000 7ff4df000640 1 -- start start 2024-05-28T06:22:31.694 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.698+0000 7ff4df000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7ff4d80d16a0 0x7ff4d80cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:31.694 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.698+0000 7ff4df000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7ff4d80cf870 0x7ff4d80d25e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:31.694 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.698+0000 7ff4df000640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4d80cfce0 con 0x7ff4d815e870 2024-05-28T06:22:31.694 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.698+0000 7ff4df000640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7ff4d80d2150 con 0x7ff4d80d16a0 2024-05-28T06:22:31.694 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.698+0000 7ff4df000640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7ff4d815ec50 con 0x7ff4d80cf870 2024-05-28T06:22:31.695 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.698+0000 7ff4dd200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7ff4d80d16a0 0x7ff4d80cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:31.696 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.698+0000 7ff4dd200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7ff4d80d16a0 0x7ff4d80cf330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47700/0 (socket says 172.21.15.96:47700) 2024-05-28T06:22:31.696 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.698+0000 7ff4dd200640 1 -- 172.21.15.96:0/1730544827 learned_addr learned my addr 172.21.15.96:0/1730544827 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:22:31.696 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.699+0000 7ff4de600640 1 --2- 172.21.15.96:0/1730544827 >> v2:172.21.15.123:3300/0 conn(0x7ff4d80cf870 0x7ff4d80d25e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:31.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.699+0000 7ff4dd200640 1 -- 172.21.15.96:0/1730544827 >> v2:172.21.15.123:3300/0 conn(0x7ff4d80cf870 msgr2=0x7ff4d80d25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:31.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.699+0000 7ff4dd200640 1 --2- 172.21.15.96:0/1730544827 >> v2:172.21.15.123:3300/0 conn(0x7ff4d80cf870 0x7ff4d80d25e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:31.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.699+0000 7ff4dd200640 1 -- 172.21.15.96:0/1730544827 >> v1:172.21.15.123:6789/0 conn(0x7ff4d815e870 legacy=0x7ff4d80d0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:31.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.699+0000 7ff4dd200640 1 -- 172.21.15.96:0/1730544827 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4d80cff80 con 0x7ff4d80d16a0 2024-05-28T06:22:31.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.699+0000 7ff4de600640 1 --2- 172.21.15.96:0/1730544827 >> v2:172.21.15.123:3300/0 conn(0x7ff4d80cf870 0x7ff4d80d25e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:22:31.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.700+0000 7ff4dd200640 1 --2- 172.21.15.96:0/1730544827 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff4d80d16a0 0x7ff4d80cf330 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7ff4d8163400 tx=0x7ff4c402a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4950c595869d2bc server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:31.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.700+0000 7ff4d7e00640 1 -- 172.21.15.96:0/1730544827 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4c4004030 con 0x7ff4d80d16a0 2024-05-28T06:22:31.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.700+0000 7ff4d7e00640 1 -- 172.21.15.96:0/1730544827 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4c40041d0 con 0x7ff4d80d16a0 2024-05-28T06:22:31.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.700+0000 7ff4d7e00640 1 -- 172.21.15.96:0/1730544827 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4c4003610 con 0x7ff4d80d16a0 2024-05-28T06:22:31.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.701+0000 7ff4df000640 1 -- 172.21.15.96:0/1730544827 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff4d80d16a0 msgr2=0x7ff4d80cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:31.698 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.701+0000 7ff4df000640 1 --2- 172.21.15.96:0/1730544827 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff4d80d16a0 0x7ff4d80cf330 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7ff4d8163400 tx=0x7ff4c402a210 comp rx=0 tx=0).stop 2024-05-28T06:22:31.699 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.701+0000 7ff4df000640 1 -- 172.21.15.96:0/1730544827 shutdown_connections 2024-05-28T06:22:31.699 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.701+0000 7ff4df000640 1 --2- 172.21.15.96:0/1730544827 >> v2:172.21.15.123:3300/0 conn(0x7ff4d80cf870 0x7ff4d80d25e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:31.699 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.701+0000 7ff4df000640 1 --2- 172.21.15.96:0/1730544827 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff4d80d16a0 0x7ff4d80cf330 unknown :-1 s=CLOSED pgs=1841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:31.699 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.701+0000 7ff4df000640 1 -- 172.21.15.96:0/1730544827 >> 172.21.15.96:0/1730544827 conn(0x7ff4d808bfc0 msgr2=0x7ff4d80d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:31.699 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.701+0000 7ff4df000640 1 -- 172.21.15.96:0/1730544827 shutdown_connections 2024-05-28T06:22:31.699 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.701+0000 7ff4df000640 1 -- 172.21.15.96:0/1730544827 wait complete. 2024-05-28T06:22:31.699 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.703+0000 7ff4df000640 1 Processor -- start 2024-05-28T06:22:31.699 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.703+0000 7ff4df000640 1 -- start start 2024-05-28T06:22:31.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.704+0000 7ff4df000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff4d80cf870 0x7ff4d8159c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:31.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.704+0000 7ff4df000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff4d80d16a0 0x7ff4d8160bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:31.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.704+0000 7ff4df000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4d815e870 0x7ff4d815f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:31.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.704+0000 7ff4df000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff4d80d0820 con 0x7ff4d80cf870 2024-05-28T06:22:31.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.704+0000 7ff4df000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff4d80d0540 con 0x7ff4d815e870 2024-05-28T06:22:31.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.704+0000 7ff4df000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ff4d80d4fb0 con 0x7ff4d80d16a0 2024-05-28T06:22:31.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.704+0000 7ff4ddc00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff4d80cf870 0x7ff4d8159c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:31.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.704+0000 7ff4ddc00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff4d80cf870 0x7ff4d8159c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47706/0 (socket says 172.21.15.96:47706) 2024-05-28T06:22:31.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.704+0000 7ff4ddc00640 1 -- 172.21.15.96:0/2147072461 learned_addr learned my addr 172.21.15.96:0/2147072461 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:22:31.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.705+0000 7ff4de600640 1 --2- 172.21.15.96:0/2147072461 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4d815e870 0x7ff4d815f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:31.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.705+0000 7ff4ddc00640 1 -- 172.21.15.96:0/2147072461 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff4d80d16a0 msgr2=0x7ff4d8160bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:31.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.705+0000 7ff4ddc00640 1 --2- 172.21.15.96:0/2147072461 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff4d80d16a0 0x7ff4d8160bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:31.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.705+0000 7ff4ddc00640 1 -- 172.21.15.96:0/2147072461 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4d815e870 msgr2=0x7ff4d815f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:31.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.705+0000 7ff4ddc00640 1 --2- 172.21.15.96:0/2147072461 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4d815e870 0x7ff4d815f900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:31.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.705+0000 7ff4ddc00640 1 -- 172.21.15.96:0/2147072461 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4d815fe40 con 0x7ff4d80cf870 2024-05-28T06:22:31.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.705+0000 7ff4ddc00640 1 --2- 172.21.15.96:0/2147072461 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff4d80cf870 0x7ff4d8159c00 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7ff4c800c460 tx=0x7ff4c800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:31.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.706+0000 7ff4d5600640 1 -- 172.21.15.96:0/2147072461 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4c8016020 con 0x7ff4d80cf870 2024-05-28T06:22:31.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.706+0000 7ff4d5600640 1 -- 172.21.15.96:0/2147072461 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4c8012870 con 0x7ff4d80cf870 2024-05-28T06:22:31.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.706+0000 7ff4d5600640 1 -- 172.21.15.96:0/2147072461 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4c8012c60 con 0x7ff4d80cf870 2024-05-28T06:22:31.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.706+0000 7ff4df000640 1 -- 172.21.15.96:0/2147072461 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4d8161240 con 0x7ff4d80cf870 2024-05-28T06:22:31.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.706+0000 7ff4df000640 1 -- 172.21.15.96:0/2147072461 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4d8161800 con 0x7ff4d80cf870 2024-05-28T06:22:31.705 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.707+0000 7ff4df000640 1 -- 172.21.15.96:0/2147072461 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4d80689e0 con 0x7ff4d80cf870 2024-05-28T06:22:31.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.708+0000 7ff4d5600640 1 -- 172.21.15.96:0/2147072461 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 37) ==== 93182+0+0 (secure 0 0 0) 0x7ff4c8019020 con 0x7ff4d80cf870 2024-05-28T06:22:31.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.709+0000 7ff4d5600640 1 --2- 172.21.15.96:0/2147072461 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff4b806e090 0x7ff4b8070550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:31.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.709+0000 7ff4d5600640 1 -- 172.21.15.96:0/2147072461 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 5765+0+0 (secure 0 0 0) 0x7ff4c808a410 con 0x7ff4d80cf870 2024-05-28T06:22:31.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.710+0000 7ff4dd200640 1 --2- 172.21.15.96:0/2147072461 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff4b806e090 0x7ff4b8070550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:31.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.711+0000 7ff4dd200640 1 --2- 172.21.15.96:0/2147072461 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff4b806e090 0x7ff4b8070550 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7ff4c4033000 tx=0x7ff4c402c000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:31.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.714+0000 7ff4d5600640 1 -- 172.21.15.96:0/2147072461 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff4c8020070 con 0x7ff4d80cf870 2024-05-28T06:22:31.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.938+0000 7ff4df000640 1 -- 172.21.15.96:0/2147072461 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7ff4d80ba500 con 0x7ff4b806e090 2024-05-28T06:22:31.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.942+0000 7ff4d5600640 1 -- 172.21.15.96:0/2147072461 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7ff4d80ba500 con 0x7ff4b806e090 2024-05-28T06:22:31.937 INFO:teuthology.orchestra.run.smithi096.stdout:[ 2024-05-28T06:22:31.938 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:22:31.938 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "cephfs" 2024-05-28T06:22:31.938 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:22:31.938 INFO:teuthology.orchestra.run.smithi096.stdout:] 2024-05-28T06:22:31.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.944+0000 7ff4df000640 1 -- 172.21.15.96:0/2147072461 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff4b806e090 msgr2=0x7ff4b8070550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:31.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.944+0000 7ff4df000640 1 --2- 172.21.15.96:0/2147072461 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff4b806e090 0x7ff4b8070550 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7ff4c4033000 tx=0x7ff4c402c000 comp rx=0 tx=0).stop 2024-05-28T06:22:31.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.944+0000 7ff4df000640 1 -- 172.21.15.96:0/2147072461 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff4d80cf870 msgr2=0x7ff4d8159c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:31.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.944+0000 7ff4df000640 1 --2- 172.21.15.96:0/2147072461 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff4d80cf870 0x7ff4d8159c00 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7ff4c800c460 tx=0x7ff4c800c920 comp rx=0 tx=0).stop 2024-05-28T06:22:31.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.944+0000 7ff4df000640 1 -- 172.21.15.96:0/2147072461 shutdown_connections 2024-05-28T06:22:31.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.944+0000 7ff4df000640 1 --2- 172.21.15.96:0/2147072461 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff4b806e090 0x7ff4b8070550 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:31.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.944+0000 7ff4df000640 1 --2- 172.21.15.96:0/2147072461 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4d815e870 0x7ff4d815f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:31.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.944+0000 7ff4df000640 1 --2- 172.21.15.96:0/2147072461 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff4d80d16a0 0x7ff4d8160bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:31.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.944+0000 7ff4df000640 1 --2- 172.21.15.96:0/2147072461 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff4d80cf870 0x7ff4d8159c00 unknown :-1 s=CLOSED pgs=1842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:31.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.944+0000 7ff4df000640 1 -- 172.21.15.96:0/2147072461 >> 172.21.15.96:0/2147072461 conn(0x7ff4d808bfc0 msgr2=0x7ff4d80b21b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:31.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.945+0000 7ff4df000640 1 -- 172.21.15.96:0/2147072461 shutdown_connections 2024-05-28T06:22:31.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:31.945+0000 7ff4df000640 1 -- 172.21.15.96:0/2147072461 wait complete. 2024-05-28T06:22:31.960 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-05-28T06:22:32.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.178+0000 7f6577a00640 1 Processor -- start 2024-05-28T06:22:32.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.178+0000 7f6577a00640 1 -- start start 2024-05-28T06:22:32.174 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.179+0000 7f6577a00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f65700ce6e0 0x7f65700d0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:32.174 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.179+0000 7f6577a00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65700d16c0 con 0x7f65700d11e0 2024-05-28T06:22:32.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.179+0000 7f6577a00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65700ccbf0 con 0x7f657015e940 2024-05-28T06:22:32.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.179+0000 7f6577a00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f65700cd240 con 0x7f65700ce6e0 2024-05-28T06:22:32.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.179+0000 7f6575a00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f65700d11e0 0x7f657009aa30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:45624/0 (socket says 172.21.15.96:45624) 2024-05-28T06:22:32.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.179+0000 7f6575a00640 1 -- 172.21.15.96:0/3254920507 learned_addr learned my addr 172.21.15.96:0/3254920507 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:22:32.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.180+0000 7f6567e00640 1 --2- 172.21.15.96:0/3254920507 >> v2:172.21.15.123:3300/0 conn(0x7f65700ce6e0 0x7f65700d0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:32.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.180+0000 7f656fe00640 1 -- 172.21.15.96:0/3254920507 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3866510997 0 0) 0x7f65700d16c0 con 0x7f65700d11e0 2024-05-28T06:22:32.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.180+0000 7f656fe00640 1 -- 172.21.15.96:0/3254920507 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6554003520 con 0x7f65700d11e0 2024-05-28T06:22:32.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.180+0000 7f656fe00640 1 -- 172.21.15.96:0/3254920507 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2141463905 0 0) 0x7f6554003520 con 0x7f65700d11e0 2024-05-28T06:22:32.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.180+0000 7f656fe00640 1 -- 172.21.15.96:0/3254920507 >> v2:172.21.15.123:3300/0 conn(0x7f65700ce6e0 msgr2=0x7f65700d0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:32.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.180+0000 7f656fe00640 1 --2- 172.21.15.96:0/3254920507 >> v2:172.21.15.123:3300/0 conn(0x7f65700ce6e0 0x7f65700d0ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:32.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.180+0000 7f656fe00640 1 -- 172.21.15.96:0/3254920507 >> v1:172.21.15.123:6789/0 conn(0x7f657015e940 legacy=0x7f65700d0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:32.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.180+0000 7f656fe00640 1 -- 172.21.15.96:0/3254920507 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65700cced0 con 0x7f65700d11e0 2024-05-28T06:22:32.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.180+0000 7f656fe00640 1 -- 172.21.15.96:0/3254920507 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f6568001670 con 0x7f65700d11e0 2024-05-28T06:22:32.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.180+0000 7f656fe00640 1 -- 172.21.15.96:0/3254920507 >> v1:172.21.15.96:6789/0 conn(0x7f65700d11e0 legacy=0x7f657009aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:32.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.180+0000 7f656fe00640 1 --2- 172.21.15.96:0/3254920507 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f65540039b0 0x7f6554003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:32.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.180+0000 7f656fe00640 1 --2- 172.21.15.96:0/3254920507 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6554004800 0x7f6554004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:32.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.180+0000 7f656fe00640 1 --2- 172.21.15.96:0/3254920507 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f65540051c0 0x7f655400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:32.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.180+0000 7f656fe00640 1 -- 172.21.15.96:0/3254920507 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f65700cd240 con 0x7f65540039b0 2024-05-28T06:22:32.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.180+0000 7f656fe00640 1 -- 172.21.15.96:0/3254920507 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f655400c060 con 0x7f6554004800 2024-05-28T06:22:32.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.180+0000 7f656fe00640 1 -- 172.21.15.96:0/3254920507 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f655400c260 con 0x7f65540051c0 2024-05-28T06:22:32.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.181+0000 7f6575000640 1 --2- 172.21.15.96:0/3254920507 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f65540039b0 0x7f6554003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:32.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.181+0000 7f6567e00640 1 --2- 172.21.15.96:0/3254920507 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6554004800 0x7f6554004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:32.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.181+0000 7f6575a00640 1 --2- 172.21.15.96:0/3254920507 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f65540051c0 0x7f655400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:32.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.181+0000 7f6575000640 1 -- 172.21.15.96:0/3254920507 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f65540051c0 msgr2=0x7f655400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:32.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.181+0000 7f6575000640 1 --2- 172.21.15.96:0/3254920507 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f65540051c0 0x7f655400b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:32.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.181+0000 7f6575000640 1 -- 172.21.15.96:0/3254920507 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6554004800 msgr2=0x7f6554004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:32.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.181+0000 7f6575000640 1 --2- 172.21.15.96:0/3254920507 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6554004800 0x7f6554004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:32.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.181+0000 7f6575000640 1 -- 172.21.15.96:0/3254920507 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6554003520 con 0x7f65540039b0 2024-05-28T06:22:32.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.181+0000 7f6567e00640 1 --2- 172.21.15.96:0/3254920507 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6554004800 0x7f6554004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:22:32.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.181+0000 7f6575a00640 1 -- 172.21.15.96:0/3254920507 reap_dead start 2024-05-28T06:22:32.180 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.183+0000 7f6575000640 1 --2- 172.21.15.96:0/3254920507 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f65540039b0 0x7f6554003dd0 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7f6560001e50 tx=0x7f6554005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccb979fe970de18d server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:32.180 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.183+0000 7f656fe00640 1 -- 172.21.15.96:0/3254920507 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6560001a70 con 0x7f65540039b0 2024-05-28T06:22:32.180 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.183+0000 7f656fe00640 1 -- 172.21.15.96:0/3254920507 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6560003970 con 0x7f65540039b0 2024-05-28T06:22:32.180 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.183+0000 7f656fe00640 1 -- 172.21.15.96:0/3254920507 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6560003d60 con 0x7f65540039b0 2024-05-28T06:22:32.180 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.183+0000 7f6577a00640 1 -- 172.21.15.96:0/3254920507 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f65540039b0 msgr2=0x7f6554003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:32.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.183+0000 7f6577a00640 1 --2- 172.21.15.96:0/3254920507 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f65540039b0 0x7f6554003dd0 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7f6560001e50 tx=0x7f6554005830 comp rx=0 tx=0).stop 2024-05-28T06:22:32.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.184+0000 7f6577a00640 1 -- 172.21.15.96:0/3254920507 shutdown_connections 2024-05-28T06:22:32.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.184+0000 7f6577a00640 1 --2- 172.21.15.96:0/3254920507 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f65540039b0 0x7f6554003dd0 unknown :-1 s=CLOSED pgs=1844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:32.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.184+0000 7f6577a00640 1 -- 172.21.15.96:0/3254920507 >> 172.21.15.96:0/3254920507 conn(0x7f65700b3f60 msgr2=0x7f65700b4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:32.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.184+0000 7f6577a00640 1 -- 172.21.15.96:0/3254920507 shutdown_connections 2024-05-28T06:22:32.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.184+0000 7f6577a00640 1 -- 172.21.15.96:0/3254920507 wait complete. 2024-05-28T06:22:32.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.185+0000 7f6577a00640 1 Processor -- start 2024-05-28T06:22:32.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.186+0000 7f6577a00640 1 -- start start 2024-05-28T06:22:32.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.186+0000 7f6577a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f65540039b0 0x7f657014b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:32.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.187+0000 7f6577a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f65700b1570 0x7f65700b5640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:32.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.187+0000 7f6577a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65700b7670 0x7f65700b7af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:32.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.187+0000 7f6577a00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f657015ed20 con 0x7f65540039b0 2024-05-28T06:22:32.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.187+0000 7f6577a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f65700d0590 con 0x7f65700b7670 2024-05-28T06:22:32.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.187+0000 7f6577a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f65700b6170 con 0x7f65700b1570 2024-05-28T06:22:32.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.187+0000 7f6575000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f65540039b0 0x7f657014b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:32.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.187+0000 7f6575000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f65540039b0 0x7f657014b220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47726/0 (socket says 172.21.15.96:47726) 2024-05-28T06:22:32.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.187+0000 7f6575000640 1 -- 172.21.15.96:0/4031522540 learned_addr learned my addr 172.21.15.96:0/4031522540 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:22:32.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.187+0000 7f6575000640 1 -- 172.21.15.96:0/4031522540 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f65700b1570 msgr2=0x7f65700b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:32.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.187+0000 7f6575000640 1 --2- 172.21.15.96:0/4031522540 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f65700b1570 0x7f65700b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:32.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.187+0000 7f6575000640 1 -- 172.21.15.96:0/4031522540 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65700b7670 msgr2=0x7f65700b7af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:32.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.187+0000 7f6575000640 1 --2- 172.21.15.96:0/4031522540 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65700b7670 0x7f65700b7af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:32.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.187+0000 7f6575000640 1 -- 172.21.15.96:0/4031522540 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f657014b760 con 0x7f65540039b0 2024-05-28T06:22:32.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.188+0000 7f6575000640 1 --2- 172.21.15.96:0/4031522540 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f65540039b0 0x7f657014b220 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7f6560005040 tx=0x7f65600014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:32.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.188+0000 7f656d600640 1 -- 172.21.15.96:0/4031522540 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f656003f020 con 0x7f65540039b0 2024-05-28T06:22:32.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.188+0000 7f656d600640 1 -- 172.21.15.96:0/4031522540 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65600052d0 con 0x7f65540039b0 2024-05-28T06:22:32.185 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.188+0000 7f656d600640 1 -- 172.21.15.96:0/4031522540 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6560037040 con 0x7f65540039b0 2024-05-28T06:22:32.185 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.188+0000 7f6577a00640 1 -- 172.21.15.96:0/4031522540 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65700b5e90 con 0x7f65540039b0 2024-05-28T06:22:32.185 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.188+0000 7f6577a00640 1 -- 172.21.15.96:0/4031522540 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f657014bea0 con 0x7f65540039b0 2024-05-28T06:22:32.185 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.189+0000 7f6577a00640 1 -- 172.21.15.96:0/4031522540 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65700689e0 con 0x7f65540039b0 2024-05-28T06:22:32.185 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.190+0000 7f656d600640 1 -- 172.21.15.96:0/4031522540 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 37) ==== 93182+0+0 (secure 0 0 0) 0x7f6560042020 con 0x7f65540039b0 2024-05-28T06:22:32.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.192+0000 7f656d600640 1 --2- 172.21.15.96:0/4031522540 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f654c06e0c0 0x7f654c070580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:32.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.192+0000 7f656d600640 1 -- 172.21.15.96:0/4031522540 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 5765+0+0 (secure 0 0 0) 0x7f65600aa6d0 con 0x7f65540039b0 2024-05-28T06:22:32.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.192+0000 7f6567e00640 1 --2- 172.21.15.96:0/4031522540 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f654c06e0c0 0x7f654c070580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:32.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.193+0000 7f6567e00640 1 --2- 172.21.15.96:0/4031522540 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f654c06e0c0 0x7f654c070580 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f65700cc540 tx=0x7f6558008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:32.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.196+0000 7f656d600640 1 -- 172.21.15.96:0/4031522540 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6560049030 con 0x7f65540039b0 2024-05-28T06:22:32.395 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.399+0000 7f6577a00640 1 -- 172.21.15.96:0/4031522540 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f65700d2250 con 0x7f65540039b0 2024-05-28T06:22:32.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.400+0000 7f656d600640 1 -- 172.21.15.96:0/4031522540 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v26)=0 v26) ==== 122+0+0 (secure 0 0 0) 0x7f656007a560 con 0x7f65540039b0 2024-05-28T06:22:32.400 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.403+0000 7f6577a00640 1 -- 172.21.15.96:0/4031522540 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f654c06e0c0 msgr2=0x7f654c070580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:32.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.403+0000 7f6577a00640 1 --2- 172.21.15.96:0/4031522540 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f654c06e0c0 0x7f654c070580 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f65700cc540 tx=0x7f6558008040 comp rx=0 tx=0).stop 2024-05-28T06:22:32.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.403+0000 7f6577a00640 1 -- 172.21.15.96:0/4031522540 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f65540039b0 msgr2=0x7f657014b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:32.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.403+0000 7f6577a00640 1 --2- 172.21.15.96:0/4031522540 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f65540039b0 0x7f657014b220 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7f6560005040 tx=0x7f65600014a0 comp rx=0 tx=0).stop 2024-05-28T06:22:32.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.403+0000 7f6577a00640 1 -- 172.21.15.96:0/4031522540 shutdown_connections 2024-05-28T06:22:32.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.403+0000 7f6577a00640 1 --2- 172.21.15.96:0/4031522540 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f654c06e0c0 0x7f654c070580 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:32.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.403+0000 7f6577a00640 1 --2- 172.21.15.96:0/4031522540 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65700b7670 0x7f65700b7af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:32.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.403+0000 7f6577a00640 1 --2- 172.21.15.96:0/4031522540 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f65700b1570 0x7f65700b5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:32.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.403+0000 7f6577a00640 1 --2- 172.21.15.96:0/4031522540 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f65540039b0 0x7f657014b220 unknown :-1 s=CLOSED pgs=1845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:32.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.403+0000 7f6577a00640 1 -- 172.21.15.96:0/4031522540 >> 172.21.15.96:0/4031522540 conn(0x7f65700b3f60 msgr2=0x7f65700b2d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:32.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.403+0000 7f6577a00640 1 -- 172.21.15.96:0/4031522540 shutdown_connections 2024-05-28T06:22:32.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.403+0000 7f6577a00640 1 -- 172.21.15.96:0/4031522540 wait complete. 2024-05-28T06:22:32.419 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv1 --mode=777 2024-05-28T06:22:32.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.626+0000 7f7c23200640 1 Processor -- start 2024-05-28T06:22:32.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.627+0000 7f7c23200640 1 -- start start 2024-05-28T06:22:32.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.628+0000 7f7c23200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f7c2415c3f0 0x7f7c241608d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:32.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.628+0000 7f7c23200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7c2406ef00 0x7f7c240d0320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:32.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.628+0000 7f7c23200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c2415c7d0 con 0x7f7c240b4790 2024-05-28T06:22:32.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.628+0000 7f7c23200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f7c240ad9a0 con 0x7f7c2415c3f0 2024-05-28T06:22:32.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.628+0000 7f7c23200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f7c24160e10 con 0x7f7c2406ef00 2024-05-28T06:22:32.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.629+0000 7f7c21e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f7c2415c3f0 0x7f7c241608d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:32.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.629+0000 7f7c19400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7c2406ef00 0x7f7c240d0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:32.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.629+0000 7f7c19400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7c2406ef00 0x7f7c240d0320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.96:51184/0 (socket says 172.21.15.96:51184) 2024-05-28T06:22:32.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.629+0000 7f7c19400640 1 -- 172.21.15.96:0/1813850497 learned_addr learned my addr 172.21.15.96:0/1813850497 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:22:32.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.630+0000 7f7c19400640 1 -- 172.21.15.96:0/1813850497 >> v2:172.21.15.123:3300/0 conn(0x7f7c2415c3f0 msgr2=0x7f7c241608d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:32.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.630+0000 7f7c19400640 1 --2- 172.21.15.96:0/1813850497 >> v2:172.21.15.123:3300/0 conn(0x7f7c2415c3f0 0x7f7c241608d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:32.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.630+0000 7f7c19400640 1 -- 172.21.15.96:0/1813850497 >> v1:172.21.15.123:6789/0 conn(0x7f7c240b4790 legacy=0x7f7c240b30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:32.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.630+0000 7f7c19400640 1 -- 172.21.15.96:0/1813850497 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c240ad3e0 con 0x7f7c2406ef00 2024-05-28T06:22:32.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.631+0000 7f7c19400640 1 --2- 172.21.15.96:0/1813850497 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7c2406ef00 0x7f7c240d0320 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c009e50 tx=0x7f7c0c02a710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=147b111be5549cc1 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:32.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.632+0000 7f7c21400640 1 -- 172.21.15.96:0/1813850497 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c0c036070 con 0x7f7c2406ef00 2024-05-28T06:22:32.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.632+0000 7f7c21400640 1 -- 172.21.15.96:0/1813850497 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c0c031600 con 0x7f7c2406ef00 2024-05-28T06:22:32.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.632+0000 7f7c21400640 1 -- 172.21.15.96:0/1813850497 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c0c0319d0 con 0x7f7c2406ef00 2024-05-28T06:22:32.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.632+0000 7f7c23200640 1 -- 172.21.15.96:0/1813850497 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7c2406ef00 msgr2=0x7f7c240d0320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:32.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.632+0000 7f7c23200640 1 --2- 172.21.15.96:0/1813850497 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7c2406ef00 0x7f7c240d0320 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c009e50 tx=0x7f7c0c02a710 comp rx=0 tx=0).stop 2024-05-28T06:22:32.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.632+0000 7f7c23200640 1 -- 172.21.15.96:0/1813850497 shutdown_connections 2024-05-28T06:22:32.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.632+0000 7f7c23200640 1 --2- 172.21.15.96:0/1813850497 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7c2406ef00 0x7f7c240d0320 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:32.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.632+0000 7f7c23200640 1 --2- 172.21.15.96:0/1813850497 >> v2:172.21.15.123:3300/0 conn(0x7f7c2415c3f0 0x7f7c241608d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:32.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.632+0000 7f7c23200640 1 -- 172.21.15.96:0/1813850497 >> 172.21.15.96:0/1813850497 conn(0x7f7c2406bae0 msgr2=0x7f7c2406bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:32.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.632+0000 7f7c23200640 1 -- 172.21.15.96:0/1813850497 shutdown_connections 2024-05-28T06:22:32.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.632+0000 7f7c23200640 1 -- 172.21.15.96:0/1813850497 wait complete. 2024-05-28T06:22:32.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.634+0000 7f7c23200640 1 Processor -- start 2024-05-28T06:22:32.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.634+0000 7f7c23200640 1 -- start start 2024-05-28T06:22:32.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.635+0000 7f7c23200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c2406ef00 0x7f7c24158b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:32.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.635+0000 7f7c23200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7c240b4790 0x7f7c2414d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:32.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.635+0000 7f7c23200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c2415c3f0 0x7f7c2415ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:32.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.635+0000 7f7c23200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f7c241603b0 con 0x7f7c2415c3f0 2024-05-28T06:22:32.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.635+0000 7f7c23200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f7c241600d0 con 0x7f7c2406ef00 2024-05-28T06:22:32.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.635+0000 7f7c23200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f7c240ad090 con 0x7f7c240b4790 2024-05-28T06:22:32.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.636+0000 7f7c22800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c2415c3f0 0x7f7c2415ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:32.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.636+0000 7f7c22800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c2415c3f0 0x7f7c2415ea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47734/0 (socket says 172.21.15.96:47734) 2024-05-28T06:22:32.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.636+0000 7f7c22800640 1 -- 172.21.15.96:0/1381375150 learned_addr learned my addr 172.21.15.96:0/1381375150 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:22:32.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.636+0000 7f7c21e00640 1 --2- 172.21.15.96:0/1381375150 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c2406ef00 0x7f7c24158b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:32.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.636+0000 7f7c19400640 1 --2- 172.21.15.96:0/1381375150 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7c240b4790 0x7f7c2414d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:32.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.636+0000 7f7c22800640 1 -- 172.21.15.96:0/1381375150 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7c240b4790 msgr2=0x7f7c2414d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:32.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.636+0000 7f7c22800640 1 --2- 172.21.15.96:0/1381375150 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7c240b4790 0x7f7c2414d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:32.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.636+0000 7f7c22800640 1 -- 172.21.15.96:0/1381375150 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c2406ef00 msgr2=0x7f7c24158b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:32.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.636+0000 7f7c22800640 1 --2- 172.21.15.96:0/1381375150 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c2406ef00 0x7f7c24158b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:32.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.636+0000 7f7c22800640 1 -- 172.21.15.96:0/1381375150 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c24157b20 con 0x7f7c2415c3f0 2024-05-28T06:22:32.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.636+0000 7f7c19400640 1 --2- 172.21.15.96:0/1381375150 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7c240b4790 0x7f7c2414d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:22:32.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.636+0000 7f7c22800640 1 --2- 172.21.15.96:0/1381375150 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c2415c3f0 0x7f7c2415ea80 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7f7c1c00c3d0 tx=0x7f7c1c00c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:32.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.637+0000 7f7c1aa00640 1 -- 172.21.15.96:0/1381375150 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c1c017070 con 0x7f7c2415c3f0 2024-05-28T06:22:32.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.637+0000 7f7c1aa00640 1 -- 172.21.15.96:0/1381375150 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c1c0052d0 con 0x7f7c2415c3f0 2024-05-28T06:22:32.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.637+0000 7f7c1aa00640 1 -- 172.21.15.96:0/1381375150 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c1c003830 con 0x7f7c2415c3f0 2024-05-28T06:22:32.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.637+0000 7f7c23200640 1 -- 172.21.15.96:0/1381375150 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c24157dc0 con 0x7f7c2415c3f0 2024-05-28T06:22:32.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.637+0000 7f7c23200640 1 -- 172.21.15.96:0/1381375150 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c24158380 con 0x7f7c2415c3f0 2024-05-28T06:22:32.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.638+0000 7f7c23200640 1 -- 172.21.15.96:0/1381375150 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c240689e0 con 0x7f7c2415c3f0 2024-05-28T06:22:32.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.639+0000 7f7c1aa00640 1 -- 172.21.15.96:0/1381375150 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 37) ==== 93182+0+0 (secure 0 0 0) 0x7f7c1c01a070 con 0x7f7c2415c3f0 2024-05-28T06:22:32.635 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.640+0000 7f7c1aa00640 1 --2- 172.21.15.96:0/1381375150 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7c0006e090 0x7f7c00070550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:32.635 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.640+0000 7f7c1aa00640 1 -- 172.21.15.96:0/1381375150 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 5765+0+0 (secure 0 0 0) 0x7f7c1c089cd0 con 0x7f7c2415c3f0 2024-05-28T06:22:32.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.641+0000 7f7c21e00640 1 --2- 172.21.15.96:0/1381375150 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7c0006e090 0x7f7c00070550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:32.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.642+0000 7f7c21e00640 1 --2- 172.21.15.96:0/1381375150 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7c0006e090 0x7f7c00070550 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f7c14009ad0 tx=0x7f7c14008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:32.640 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.645+0000 7f7c1aa00640 1 -- 172.21.15.96:0/1381375150 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7c1c059c20 con 0x7f7c2415c3f0 2024-05-28T06:22:32.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.872+0000 7f7c23200640 1 -- 172.21.15.96:0/1381375150 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv1", "mode": "777", "target": ["mon-mgr", ""]}) -- 0x7f7c2414da00 con 0x7f7c0006e090 2024-05-28T06:22:32.882 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:22:32.886+0000 7fea59e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:22:32.883 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:22:32.886+0000 7fea59e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:22:32.883 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:22:32.886+0000 7fea59e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:22:32.883 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:22:32.886+0000 7fea59e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:22:32.883 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:22:32.886+0000 7fea59e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:22:32.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.956+0000 7f7c1aa00640 1 -- 172.21.15.96:0/1381375150 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f7c2414da00 con 0x7f7c0006e090 2024-05-28T06:22:32.954 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.959+0000 7f7c23200640 1 -- 172.21.15.96:0/1381375150 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7c0006e090 msgr2=0x7f7c00070550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:32.954 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.959+0000 7f7c23200640 1 --2- 172.21.15.96:0/1381375150 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7c0006e090 0x7f7c00070550 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f7c14009ad0 tx=0x7f7c14008040 comp rx=0 tx=0).stop 2024-05-28T06:22:32.954 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.959+0000 7f7c23200640 1 -- 172.21.15.96:0/1381375150 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c2415c3f0 msgr2=0x7f7c2415ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:32.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.959+0000 7f7c23200640 1 --2- 172.21.15.96:0/1381375150 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c2415c3f0 0x7f7c2415ea80 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7f7c1c00c3d0 tx=0x7f7c1c00c890 comp rx=0 tx=0).stop 2024-05-28T06:22:32.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.959+0000 7f7c23200640 1 -- 172.21.15.96:0/1381375150 shutdown_connections 2024-05-28T06:22:32.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.959+0000 7f7c23200640 1 --2- 172.21.15.96:0/1381375150 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7c0006e090 0x7f7c00070550 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:32.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.959+0000 7f7c23200640 1 --2- 172.21.15.96:0/1381375150 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c2415c3f0 0x7f7c2415ea80 unknown :-1 s=CLOSED pgs=1846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:32.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.959+0000 7f7c23200640 1 --2- 172.21.15.96:0/1381375150 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7c240b4790 0x7f7c2414d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:32.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.959+0000 7f7c23200640 1 --2- 172.21.15.96:0/1381375150 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c2406ef00 0x7f7c24158b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:32.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.959+0000 7f7c23200640 1 -- 172.21.15.96:0/1381375150 >> 172.21.15.96:0/1381375150 conn(0x7f7c2406bae0 msgr2=0x7f7c240b09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:32.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.959+0000 7f7c23200640 1 -- 172.21.15.96:0/1381375150 shutdown_connections 2024-05-28T06:22:32.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:32.959+0000 7f7c23200640 1 -- 172.21.15.96:0/1381375150 wait complete. 2024-05-28T06:22:32.977 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-05-28T06:22:33.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.183+0000 7fabb3400640 1 Processor -- start 2024-05-28T06:22:33.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.183+0000 7fabb3400640 1 -- start start 2024-05-28T06:22:33.180 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.185+0000 7fabb3400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fabac0cd550 0x7fabac06b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:33.180 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.185+0000 7fabb3400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fabac0cb9a0 0x7fabac0cbe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:33.180 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.185+0000 7fabb3400640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fabac06ef80 con 0x7fabac0ccd20 2024-05-28T06:22:33.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.185+0000 7fabb3400640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fabac06c660 con 0x7fabac0cd550 2024-05-28T06:22:33.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.185+0000 7fabb3400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fabac06f2b0 con 0x7fabac0cb9a0 2024-05-28T06:22:33.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.186+0000 7fabb1400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fabac0cb9a0 0x7fabac0cbe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:33.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.186+0000 7fabb1400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fabac0cb9a0 0x7fabac0cbe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.96:51196/0 (socket says 172.21.15.96:51196) 2024-05-28T06:22:33.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.186+0000 7fabb1400640 1 -- 172.21.15.96:0/292993708 learned_addr learned my addr 172.21.15.96:0/292993708 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:22:33.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.186+0000 7fababe00640 1 --2- 172.21.15.96:0/292993708 >> v2:172.21.15.123:3300/0 conn(0x7fabac0cd550 0x7fabac06b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:33.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.187+0000 7fabb1400640 1 -- 172.21.15.96:0/292993708 >> v2:172.21.15.123:3300/0 conn(0x7fabac0cd550 msgr2=0x7fabac06b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:33.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.187+0000 7fabb1400640 1 --2- 172.21.15.96:0/292993708 >> v2:172.21.15.123:3300/0 conn(0x7fabac0cd550 0x7fabac06b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:33.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.187+0000 7fabb1400640 1 -- 172.21.15.96:0/292993708 >> v1:172.21.15.123:6789/0 conn(0x7fabac0ccd20 legacy=0x7fabac06f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:33.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.187+0000 7fabb1400640 1 -- 172.21.15.96:0/292993708 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabac0cd100 con 0x7fabac0cb9a0 2024-05-28T06:22:33.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.188+0000 7fabb1400640 1 --2- 172.21.15.96:0/292993708 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fabac0cb9a0 0x7fabac0cbe10 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7faba4009e50 tx=0x7faba402a680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c0cc371ebb3fe091 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:33.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.189+0000 7fabab400640 1 -- 172.21.15.96:0/292993708 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faba4036070 con 0x7fabac0cb9a0 2024-05-28T06:22:33.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.189+0000 7fabab400640 1 -- 172.21.15.96:0/292993708 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faba4031600 con 0x7fabac0cb9a0 2024-05-28T06:22:33.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.189+0000 7fabab400640 1 -- 172.21.15.96:0/292993708 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faba40319d0 con 0x7fabac0cb9a0 2024-05-28T06:22:33.185 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.190+0000 7fabb3400640 1 -- 172.21.15.96:0/292993708 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fabac0cb9a0 msgr2=0x7fabac0cbe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:33.185 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.190+0000 7fabb3400640 1 --2- 172.21.15.96:0/292993708 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fabac0cb9a0 0x7fabac0cbe10 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7faba4009e50 tx=0x7faba402a680 comp rx=0 tx=0).stop 2024-05-28T06:22:33.185 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.190+0000 7fabb3400640 1 -- 172.21.15.96:0/292993708 shutdown_connections 2024-05-28T06:22:33.185 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.190+0000 7fabb3400640 1 --2- 172.21.15.96:0/292993708 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fabac0cb9a0 0x7fabac0cbe10 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:33.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.190+0000 7fabb3400640 1 --2- 172.21.15.96:0/292993708 >> v2:172.21.15.123:3300/0 conn(0x7fabac0cd550 0x7fabac06b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:33.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.190+0000 7fabb3400640 1 -- 172.21.15.96:0/292993708 >> 172.21.15.96:0/292993708 conn(0x7fabac0b2f40 msgr2=0x7fabac0b3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:33.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.190+0000 7fabb3400640 1 -- 172.21.15.96:0/292993708 shutdown_connections 2024-05-28T06:22:33.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.190+0000 7fabb3400640 1 -- 172.21.15.96:0/292993708 wait complete. 2024-05-28T06:22:33.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.192+0000 7fabb3400640 1 Processor -- start 2024-05-28T06:22:33.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.192+0000 7fabb3400640 1 -- start start 2024-05-28T06:22:33.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.194+0000 7fabb3400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fabac0cb9a0 0x7fabac0caf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:33.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.194+0000 7fabb3400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fabac0ccd20 0x7fabac0b4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:33.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.194+0000 7fabb3400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fabac0cd550 0x7fabac1526d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:33.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.194+0000 7fabb3400640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fabac06be40 con 0x7fabac0ccd20 2024-05-28T06:22:33.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.194+0000 7fabb3400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fabac0cf140 con 0x7fabac0cb9a0 2024-05-28T06:22:33.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.194+0000 7fabb3400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fabac06c380 con 0x7fabac0cd550 2024-05-28T06:22:33.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.194+0000 7fababe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fabac0ccd20 0x7fabac0b4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:33.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.194+0000 7fababe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fabac0ccd20 0x7fabac0b4000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47738/0 (socket says 172.21.15.96:47738) 2024-05-28T06:22:33.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.194+0000 7fababe00640 1 -- 172.21.15.96:0/1290263049 learned_addr learned my addr 172.21.15.96:0/1290263049 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:22:33.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.195+0000 7fababe00640 1 -- 172.21.15.96:0/1290263049 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fabac0cd550 msgr2=0x7fabac1526d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:33.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.195+0000 7fababe00640 1 --2- 172.21.15.96:0/1290263049 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fabac0cd550 0x7fabac1526d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:33.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.195+0000 7fababe00640 1 -- 172.21.15.96:0/1290263049 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fabac0cb9a0 msgr2=0x7fabac0caf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:33.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.195+0000 7fababe00640 1 --2- 172.21.15.96:0/1290263049 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fabac0cb9a0 0x7fabac0caf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:33.192 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.195+0000 7fababe00640 1 -- 172.21.15.96:0/1290263049 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabac0b6710 con 0x7fabac0ccd20 2024-05-28T06:22:33.192 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.195+0000 7fababe00640 1 --2- 172.21.15.96:0/1290263049 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fabac0ccd20 0x7fabac0b4000 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7fab9800e7f0 tx=0x7fab9800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:33.192 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.195+0000 7faba8c00640 1 -- 172.21.15.96:0/1290263049 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab98017070 con 0x7fabac0ccd20 2024-05-28T06:22:33.192 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.195+0000 7faba8c00640 1 -- 172.21.15.96:0/1290263049 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab9800c940 con 0x7fabac0ccd20 2024-05-28T06:22:33.192 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.195+0000 7faba8c00640 1 -- 172.21.15.96:0/1290263049 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab9800eea0 con 0x7fabac0ccd20 2024-05-28T06:22:33.192 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.195+0000 7fabb3400640 1 -- 172.21.15.96:0/1290263049 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabac0b69f0 con 0x7fabac0ccd20 2024-05-28T06:22:33.192 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.195+0000 7fabb3400640 1 -- 172.21.15.96:0/1290263049 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabac0b6fb0 con 0x7fabac0ccd20 2024-05-28T06:22:33.192 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.197+0000 7fabb3400640 1 -- 172.21.15.96:0/1290263049 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fabac0689e0 con 0x7fabac0ccd20 2024-05-28T06:22:33.193 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.198+0000 7faba8c00640 1 -- 172.21.15.96:0/1290263049 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 37) ==== 93182+0+0 (secure 0 0 0) 0x7fab9801a070 con 0x7fabac0ccd20 2024-05-28T06:22:33.194 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.199+0000 7faba8c00640 1 --2- 172.21.15.96:0/1290263049 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fab8c06e090 0x7fab8c070550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:33.194 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.199+0000 7faba8c00640 1 -- 172.21.15.96:0/1290263049 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 5765+0+0 (secure 0 0 0) 0x7fab9808a7c0 con 0x7fabac0ccd20 2024-05-28T06:22:33.196 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.200+0000 7fabb0a00640 1 --2- 172.21.15.96:0/1290263049 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fab8c06e090 0x7fab8c070550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:33.196 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.201+0000 7fabb0a00640 1 --2- 172.21.15.96:0/1290263049 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fab8c06e090 0x7fab8c070550 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fabac06f150 tx=0x7faba0007970 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:33.199 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.204+0000 7faba8c00640 1 -- 172.21.15.96:0/1290263049 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fab9805a650 con 0x7fabac0ccd20 2024-05-28T06:22:33.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.435+0000 7fabb3400640 1 -- 172.21.15.96:0/1290263049 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) -- 0x7fabac0cb5e0 con 0x7fab8c06e090 2024-05-28T06:22:33.444 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.449+0000 7faba8c00640 1 -- 172.21.15.96:0/1290263049 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+59 (secure 0 0 0) 0x7fabac0cb5e0 con 0x7fab8c06e090 2024-05-28T06:22:33.445 INFO:teuthology.orchestra.run.smithi096.stdout:/volumes/_nogroup/sv1/f0a99b9f-1e23-4f2b-ae5c-c3aa6b024e70 2024-05-28T06:22:33.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.452+0000 7fabb3400640 1 -- 172.21.15.96:0/1290263049 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fab8c06e090 msgr2=0x7fab8c070550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:33.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.452+0000 7fabb3400640 1 --2- 172.21.15.96:0/1290263049 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fab8c06e090 0x7fab8c070550 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fabac06f150 tx=0x7faba0007970 comp rx=0 tx=0).stop 2024-05-28T06:22:33.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.452+0000 7fabb3400640 1 -- 172.21.15.96:0/1290263049 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fabac0ccd20 msgr2=0x7fabac0b4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:33.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.452+0000 7fabb3400640 1 --2- 172.21.15.96:0/1290263049 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fabac0ccd20 0x7fabac0b4000 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7fab9800e7f0 tx=0x7fab9800ecb0 comp rx=0 tx=0).stop 2024-05-28T06:22:33.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.452+0000 7fabb3400640 1 -- 172.21.15.96:0/1290263049 shutdown_connections 2024-05-28T06:22:33.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.452+0000 7fabb3400640 1 --2- 172.21.15.96:0/1290263049 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fab8c06e090 0x7fab8c070550 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:33.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.452+0000 7fabb3400640 1 --2- 172.21.15.96:0/1290263049 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fabac0cd550 0x7fabac1526d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:33.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.452+0000 7fabb3400640 1 --2- 172.21.15.96:0/1290263049 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fabac0ccd20 0x7fabac0b4000 unknown :-1 s=CLOSED pgs=1847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:33.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.452+0000 7fabb3400640 1 --2- 172.21.15.96:0/1290263049 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fabac0cb9a0 0x7fabac0caf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:33.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.452+0000 7fabb3400640 1 -- 172.21.15.96:0/1290263049 >> 172.21.15.96:0/1290263049 conn(0x7fabac0b2f40 msgr2=0x7fabac0cd9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:33.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.452+0000 7fabb3400640 1 -- 172.21.15.96:0/1290263049 shutdown_connections 2024-05-28T06:22:33.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.452+0000 7fabb3400640 1 -- 172.21.15.96:0/1290263049 wait complete. 2024-05-28T06:22:33.465 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-05-28T06:22:33.657 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.659+0000 7f33d7200640 1 Processor -- start 2024-05-28T06:22:33.657 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.660+0000 7f33d7200640 1 -- start start 2024-05-28T06:22:33.657 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.661+0000 7f33d7200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f33d80cfa40 0x7f33d8169690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:33.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.661+0000 7f33d7200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33d815ec50 con 0x7f33d815e870 2024-05-28T06:22:33.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.661+0000 7f33d7200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33d80cff80 con 0x7f33d80d16a0 2024-05-28T06:22:33.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.661+0000 7f33d7200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f33d80d2150 con 0x7f33d80cfa40 2024-05-28T06:22:33.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.661+0000 7f33d6800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f33d80cfa40 0x7f33d8169690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:33.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.661+0000 7f33d6800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f33d80cfa40 0x7f33d8169690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47754/0 (socket says 172.21.15.96:47754) 2024-05-28T06:22:33.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.661+0000 7f33d6800640 1 -- 172.21.15.96:0/4218210024 learned_addr learned my addr 172.21.15.96:0/4218210024 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:22:33.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.661+0000 7f33d5e00640 1 --1- 172.21.15.96:0/4218210024 >> v1:172.21.15.96:6789/0 conn(0x7f33d815e870 0x7f33d80d0f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:45628/0 (socket says 172.21.15.96:45628) 2024-05-28T06:22:33.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.662+0000 7f33d6800640 1 -- 172.21.15.96:0/4218210024 >> v1:172.21.15.123:6789/0 conn(0x7f33d80d16a0 legacy=0x7f33d80cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:22:33.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.662+0000 7f33d6800640 1 -- 172.21.15.96:0/4218210024 >> v1:172.21.15.96:6789/0 conn(0x7f33d815e870 legacy=0x7f33d80d0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:33.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.662+0000 7f33d6800640 1 -- 172.21.15.96:0/4218210024 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33d80d0260 con 0x7f33d80cfa40 2024-05-28T06:22:33.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.662+0000 7f33d4a00640 1 -- 172.21.15.96:0/4218210024 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1832098561 0 0) 0x7f33d815ec50 con 0x7f33d815e870 2024-05-28T06:22:33.659 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.663+0000 7f33d6800640 1 --2- 172.21.15.96:0/4218210024 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f33d80cfa40 0x7f33d8169690 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7f33d0009870 tx=0x7f33d002a2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2ffa681f01d1a51 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:33.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.663+0000 7f33d4a00640 1 -- 172.21.15.96:0/4218210024 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33d0037070 con 0x7f33d80cfa40 2024-05-28T06:22:33.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.663+0000 7f33d4a00640 1 -- 172.21.15.96:0/4218210024 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33d0032440 con 0x7f33d80cfa40 2024-05-28T06:22:33.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.663+0000 7f33d4a00640 1 -- 172.21.15.96:0/4218210024 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33d0032810 con 0x7f33d80cfa40 2024-05-28T06:22:33.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.664+0000 7f33d7200640 1 -- 172.21.15.96:0/4218210024 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f33d80cfa40 msgr2=0x7f33d8169690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:33.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.664+0000 7f33d7200640 1 --2- 172.21.15.96:0/4218210024 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f33d80cfa40 0x7f33d8169690 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7f33d0009870 tx=0x7f33d002a2a0 comp rx=0 tx=0).stop 2024-05-28T06:22:33.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.664+0000 7f33d7200640 1 -- 172.21.15.96:0/4218210024 shutdown_connections 2024-05-28T06:22:33.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.664+0000 7f33d7200640 1 --2- 172.21.15.96:0/4218210024 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f33d80cfa40 0x7f33d8169690 unknown :-1 s=CLOSED pgs=1849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:33.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.664+0000 7f33d7200640 1 -- 172.21.15.96:0/4218210024 >> 172.21.15.96:0/4218210024 conn(0x7f33d808bfc0 msgr2=0x7f33d80d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:33.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.664+0000 7f33d7200640 1 -- 172.21.15.96:0/4218210024 shutdown_connections 2024-05-28T06:22:33.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.664+0000 7f33d7200640 1 -- 172.21.15.96:0/4218210024 wait complete. 2024-05-28T06:22:33.661 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.666+0000 7f33d7200640 1 Processor -- start 2024-05-28T06:22:33.661 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.666+0000 7f33d7200640 1 -- start start 2024-05-28T06:22:33.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.667+0000 7f33d7200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f33d80cfa40 0x7f33d81593f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:33.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.667+0000 7f33d7200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f33d80d16a0 0x7f33d81f4710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:33.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.667+0000 7f33d7200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f33d815e870 0x7f33d80b8390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:33.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.667+0000 7f33d7200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f33d80d4fb0 con 0x7f33d815e870 2024-05-28T06:22:33.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.667+0000 7f33d7200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f33d80d0820 con 0x7f33d80d16a0 2024-05-28T06:22:33.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.667+0000 7f33d7200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f33d80d35f0 con 0x7f33d80cfa40 2024-05-28T06:22:33.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.667+0000 7f33d6800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f33d815e870 0x7f33d80b8390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:33.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.667+0000 7f33d6800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f33d815e870 0x7f33d80b8390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47756/0 (socket says 172.21.15.96:47756) 2024-05-28T06:22:33.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.667+0000 7f33d6800640 1 -- 172.21.15.96:0/2680956295 learned_addr learned my addr 172.21.15.96:0/2680956295 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:22:33.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.667+0000 7f33d6800640 1 -- 172.21.15.96:0/2680956295 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f33d80cfa40 msgr2=0x7f33d81593f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:33.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.667+0000 7f33d6800640 1 --2- 172.21.15.96:0/2680956295 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f33d80cfa40 0x7f33d81593f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:33.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.667+0000 7f33d6800640 1 -- 172.21.15.96:0/2680956295 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f33d80d16a0 msgr2=0x7f33d81f4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:33.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.667+0000 7f33d6800640 1 --2- 172.21.15.96:0/2680956295 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f33d80d16a0 0x7f33d81f4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:33.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.667+0000 7f33d6800640 1 -- 172.21.15.96:0/2680956295 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33d80c80b0 con 0x7f33d815e870 2024-05-28T06:22:33.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.668+0000 7f33d6800640 1 --2- 172.21.15.96:0/2680956295 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f33d815e870 0x7f33d80b8390 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7f33d00090b0 tx=0x7f33d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:33.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.668+0000 7f33ce000640 1 -- 172.21.15.96:0/2680956295 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33d0037040 con 0x7f33d815e870 2024-05-28T06:22:33.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.668+0000 7f33ce000640 1 -- 172.21.15.96:0/2680956295 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33d0004250 con 0x7f33d815e870 2024-05-28T06:22:33.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.668+0000 7f33ce000640 1 -- 172.21.15.96:0/2680956295 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33d003c670 con 0x7f33d815e870 2024-05-28T06:22:33.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.668+0000 7f33d7200640 1 -- 172.21.15.96:0/2680956295 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33d81f4f60 con 0x7f33d815e870 2024-05-28T06:22:33.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.668+0000 7f33d7200640 1 -- 172.21.15.96:0/2680956295 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33d81f5520 con 0x7f33d815e870 2024-05-28T06:22:33.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.669+0000 7f33d7200640 1 -- 172.21.15.96:0/2680956295 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33d806c980 con 0x7f33d815e870 2024-05-28T06:22:33.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.670+0000 7f33ce000640 1 -- 172.21.15.96:0/2680956295 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 37) ==== 93182+0+0 (secure 0 0 0) 0x7f33d003a070 con 0x7f33d815e870 2024-05-28T06:22:33.666 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.671+0000 7f33ce000640 1 --2- 172.21.15.96:0/2680956295 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f33b406e090 0x7f33b4070550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:22:33.666 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.671+0000 7f33ce000640 1 -- 172.21.15.96:0/2680956295 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 5765+0+0 (secure 0 0 0) 0x7f33d00ab350 con 0x7f33d815e870 2024-05-28T06:22:33.667 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.672+0000 7f33d5e00640 1 --2- 172.21.15.96:0/2680956295 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f33b406e090 0x7f33b4070550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:22:33.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.673+0000 7f33d5e00640 1 --2- 172.21.15.96:0/2680956295 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f33b406e090 0x7f33b4070550 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f33c8007fd0 tx=0x7f33c8007920 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:22:33.671 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.676+0000 7f33ce000640 1 -- 172.21.15.96:0/2680956295 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f33d0041020 con 0x7f33d815e870 2024-05-28T06:22:33.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.903+0000 7f33d7200640 1 -- 172.21.15.96:0/2680956295 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) -- 0x7f33d80689e0 con 0x7f33b406e090 2024-05-28T06:22:33.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.917+0000 7f33ce000640 1 -- 172.21.15.96:0/2680956295 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+59 (secure 0 0 0) 0x7f33d80689e0 con 0x7f33b406e090 2024-05-28T06:22:33.912 INFO:teuthology.orchestra.run.smithi096.stdout:/volumes/_nogroup/sv1/f0a99b9f-1e23-4f2b-ae5c-c3aa6b024e70 2024-05-28T06:22:33.914 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.918+0000 7f33d7200640 1 -- 172.21.15.96:0/2680956295 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f33b406e090 msgr2=0x7f33b4070550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:33.914 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.918+0000 7f33d7200640 1 --2- 172.21.15.96:0/2680956295 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f33b406e090 0x7f33b4070550 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f33c8007fd0 tx=0x7f33c8007920 comp rx=0 tx=0).stop 2024-05-28T06:22:33.914 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.918+0000 7f33d7200640 1 -- 172.21.15.96:0/2680956295 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f33d815e870 msgr2=0x7f33d80b8390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:22:33.917 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.918+0000 7f33d7200640 1 --2- 172.21.15.96:0/2680956295 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f33d815e870 0x7f33d80b8390 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7f33d00090b0 tx=0x7f33d00046c0 comp rx=0 tx=0).stop 2024-05-28T06:22:33.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.919+0000 7f33d7200640 1 -- 172.21.15.96:0/2680956295 shutdown_connections 2024-05-28T06:22:33.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.919+0000 7f33d7200640 1 --2- 172.21.15.96:0/2680956295 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f33b406e090 0x7f33b4070550 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:33.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.919+0000 7f33d7200640 1 --2- 172.21.15.96:0/2680956295 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f33d815e870 0x7f33d80b8390 unknown :-1 s=CLOSED pgs=1850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:33.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.919+0000 7f33d7200640 1 --2- 172.21.15.96:0/2680956295 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f33d80d16a0 0x7f33d81f4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:33.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.919+0000 7f33d7200640 1 --2- 172.21.15.96:0/2680956295 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f33d80cfa40 0x7f33d81593f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:22:33.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.919+0000 7f33d7200640 1 -- 172.21.15.96:0/2680956295 >> 172.21.15.96:0/2680956295 conn(0x7f33d808bfc0 msgr2=0x7f33d80b3fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:22:33.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.919+0000 7f33d7200640 1 -- 172.21.15.96:0/2680956295 shutdown_connections 2024-05-28T06:22:33.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:22:33.919+0000 7f33d7200640 1 -- 172.21.15.96:0/2680956295 wait complete. 2024-05-28T06:22:33.933 DEBUG:tasks.cephfs.test_volumes:filling subvolume sv1 with 3 files each 1024MB size under directory volumes/_nogroup/sv1/f0a99b9f-1e23-4f2b-ae5c-c3aa6b024e70 2024-05-28T06:22:33.933 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:22:33.934 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:22:33.986 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:22:33.986 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/f0a99b9f-1e23-4f2b-ae5c-c3aa6b024e70/subvolume_file.0 bs=1M conv=fdatasync count=1024 seek=0) 2024-05-28T06:22:34.264 DEBUG:teuthology.orchestra.run.smithi096:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:22:34.275 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:22:48.314 INFO:teuthology.orchestra.run.smithi096.stderr:1024+0 records in 2024-05-28T06:22:48.314 INFO:teuthology.orchestra.run.smithi096.stderr:1024+0 records out 2024-05-28T06:22:48.314 INFO:teuthology.orchestra.run.smithi096.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 14.2472 s, 75.4 MB/s 2024-05-28T06:22:48.316 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:22:48.317 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:22:48.367 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:22:48.367 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/f0a99b9f-1e23-4f2b-ae5c-c3aa6b024e70/subvolume_file.1 bs=1M conv=fdatasync count=1024 seek=0) 2024-05-28T06:23:00.457 INFO:teuthology.orchestra.run.smithi096.stderr:1024+0 records in 2024-05-28T06:23:00.458 INFO:teuthology.orchestra.run.smithi096.stderr:1024+0 records out 2024-05-28T06:23:00.458 INFO:teuthology.orchestra.run.smithi096.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 11.9826 s, 89.6 MB/s 2024-05-28T06:23:00.459 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:23:00.459 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:23:00.515 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:23:00.516 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/f0a99b9f-1e23-4f2b-ae5c-c3aa6b024e70/subvolume_file.2 bs=1M conv=fdatasync count=1024 seek=0) 2024-05-28T06:23:04.421 DEBUG:teuthology.orchestra.run.smithi096:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:23:04.427 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:23:11.959 INFO:teuthology.orchestra.run.smithi096.stderr:1024+0 records in 2024-05-28T06:23:11.959 INFO:teuthology.orchestra.run.smithi096.stderr:1024+0 records out 2024-05-28T06:23:11.959 INFO:teuthology.orchestra.run.smithi096.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 11.3826 s, 94.3 MB/s 2024-05-28T06:23:11.961 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot create cephfs sv1 ss1 2024-05-28T06:23:12.171 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.176+0000 7fd9c9e00640 1 Processor -- start 2024-05-28T06:23:12.171 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.176+0000 7fd9c9e00640 1 -- start start 2024-05-28T06:23:12.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.177+0000 7fd9c9e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fd9c40cf2f0 0x7fd9c40cf910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:12.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.177+0000 7fd9c9e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fd9c406edf0 0x7fd9c413fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:12.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.177+0000 7fd9c9e00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9c406dd50 con 0x7fd9c406f250 2024-05-28T06:23:12.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.177+0000 7fd9c9e00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fd9c406e030 con 0x7fd9c406edf0 2024-05-28T06:23:12.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.177+0000 7fd9c9e00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fd9c406e310 con 0x7fd9c40cf2f0 2024-05-28T06:23:12.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.178+0000 7fd9c2000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fd9c406edf0 0x7fd9c413fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:12.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.178+0000 7fd9c3400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fd9c406f250 0x7fd9c40ad130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:50364/0 (socket says 172.21.15.96:50364) 2024-05-28T06:23:12.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.178+0000 7fd9c2000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fd9c406edf0 0x7fd9c413fdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:36912/0 (socket says 172.21.15.96:36912) 2024-05-28T06:23:12.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.178+0000 7fd9c3400640 1 -- 172.21.15.96:0/1192202512 learned_addr learned my addr 172.21.15.96:0/1192202512 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:23:12.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.178+0000 7fd9c2a00640 1 --2- 172.21.15.96:0/1192202512 >> v2:172.21.15.123:3300/0 conn(0x7fd9c40cf2f0 0x7fd9c40cf910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:12.174 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.178+0000 7fd9c2000640 1 -- 172.21.15.96:0/1192202512 >> v2:172.21.15.123:3300/0 conn(0x7fd9c40cf2f0 msgr2=0x7fd9c40cf910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:23:12.174 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.178+0000 7fd9c2000640 1 --2- 172.21.15.96:0/1192202512 >> v2:172.21.15.123:3300/0 conn(0x7fd9c40cf2f0 0x7fd9c40cf910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:12.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.178+0000 7fd9c2000640 1 -- 172.21.15.96:0/1192202512 >> v1:172.21.15.96:6789/0 conn(0x7fd9c406f250 legacy=0x7fd9c40ad130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:12.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.178+0000 7fd9c2000640 1 -- 172.21.15.96:0/1192202512 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9c406e5f0 con 0x7fd9c406edf0 2024-05-28T06:23:12.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.178+0000 7fd9c1600640 1 -- 172.21.15.96:0/1192202512 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1426624572 0 0) 0x7fd9c406dd50 con 0x7fd9c406f250 2024-05-28T06:23:12.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.179+0000 7fd9c2000640 1 --2- 172.21.15.96:0/1192202512 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9c406edf0 0x7fd9c413fdf0 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7fd9c40b6170 tx=0x7fd9b002a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dac6e847065eea6c server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:23:12.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.180+0000 7fd9c1600640 1 -- 172.21.15.96:0/1192202512 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9b0037070 con 0x7fd9c406edf0 2024-05-28T06:23:12.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.180+0000 7fd9c1600640 1 -- 172.21.15.96:0/1192202512 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9b0032440 con 0x7fd9c406edf0 2024-05-28T06:23:12.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.180+0000 7fd9c1600640 1 -- 172.21.15.96:0/1192202512 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9b002ae60 con 0x7fd9c406edf0 2024-05-28T06:23:12.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.180+0000 7fd9c9e00640 1 -- 172.21.15.96:0/1192202512 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9c406edf0 msgr2=0x7fd9c413fdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:12.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.180+0000 7fd9c9e00640 1 --2- 172.21.15.96:0/1192202512 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9c406edf0 0x7fd9c413fdf0 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7fd9c40b6170 tx=0x7fd9b002a110 comp rx=0 tx=0).stop 2024-05-28T06:23:12.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.180+0000 7fd9c9e00640 1 -- 172.21.15.96:0/1192202512 shutdown_connections 2024-05-28T06:23:12.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.180+0000 7fd9c9e00640 1 --2- 172.21.15.96:0/1192202512 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9c406edf0 0x7fd9c413fdf0 unknown :-1 s=CLOSED pgs=1852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:12.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.180+0000 7fd9c9e00640 1 --2- 172.21.15.96:0/1192202512 >> v2:172.21.15.123:3300/0 conn(0x7fd9c40cf2f0 0x7fd9c40cf910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:12.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.180+0000 7fd9c9e00640 1 -- 172.21.15.96:0/1192202512 >> 172.21.15.96:0/1192202512 conn(0x7fd9c40b1e50 msgr2=0x7fd9c40b2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:23:12.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.180+0000 7fd9c9e00640 1 -- 172.21.15.96:0/1192202512 shutdown_connections 2024-05-28T06:23:12.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.180+0000 7fd9c9e00640 1 -- 172.21.15.96:0/1192202512 wait complete. 2024-05-28T06:23:12.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.182+0000 7fd9c9e00640 1 Processor -- start 2024-05-28T06:23:12.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.182+0000 7fd9c9e00640 1 -- start start 2024-05-28T06:23:12.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.183+0000 7fd9c9e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd9c406edf0 0x7fd9c415a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:12.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.183+0000 7fd9c9e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9c406f250 0x7fd9c415cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:12.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.183+0000 7fd9c9e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9c40cf2f0 0x7fd9c40b45d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:12.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.183+0000 7fd9c9e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd9c41461a0 con 0x7fd9c40cf2f0 2024-05-28T06:23:12.180 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.183+0000 7fd9c9e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd9c4146020 con 0x7fd9c406f250 2024-05-28T06:23:12.180 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.183+0000 7fd9c9e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fd9c40cfe50 con 0x7fd9c406edf0 2024-05-28T06:23:12.180 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.184+0000 7fd9c3400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9c40cf2f0 0x7fd9c40b45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:12.180 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.184+0000 7fd9c3400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9c40cf2f0 0x7fd9c40b45d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:36916/0 (socket says 172.21.15.96:36916) 2024-05-28T06:23:12.180 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.184+0000 7fd9c3400640 1 -- 172.21.15.96:0/1337911220 learned_addr learned my addr 172.21.15.96:0/1337911220 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:23:12.180 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.184+0000 7fd9c2a00640 1 --2- 172.21.15.96:0/1337911220 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd9c406edf0 0x7fd9c415a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:12.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.184+0000 7fd9c3400640 1 -- 172.21.15.96:0/1337911220 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd9c406edf0 msgr2=0x7fd9c415a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:12.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.184+0000 7fd9c3400640 1 --2- 172.21.15.96:0/1337911220 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd9c406edf0 0x7fd9c415a600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:12.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.184+0000 7fd9c3400640 1 -- 172.21.15.96:0/1337911220 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9c406f250 msgr2=0x7fd9c415cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:12.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.184+0000 7fd9c3400640 1 --2- 172.21.15.96:0/1337911220 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9c406f250 0x7fd9c415cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:12.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.184+0000 7fd9c3400640 1 -- 172.21.15.96:0/1337911220 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9c415d1b0 con 0x7fd9c40cf2f0 2024-05-28T06:23:12.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.184+0000 7fd9c3400640 1 --2- 172.21.15.96:0/1337911220 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9c40cf2f0 0x7fd9c40b45d0 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc00c410 tx=0x7fd9bc00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:23:12.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.185+0000 7fd9bb400640 1 -- 172.21.15.96:0/1337911220 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9bc0074e0 con 0x7fd9c40cf2f0 2024-05-28T06:23:12.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.185+0000 7fd9bb400640 1 -- 172.21.15.96:0/1337911220 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9bc0047e0 con 0x7fd9c40cf2f0 2024-05-28T06:23:12.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.185+0000 7fd9bb400640 1 -- 172.21.15.96:0/1337911220 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9bc004bd0 con 0x7fd9c40cf2f0 2024-05-28T06:23:12.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.185+0000 7fd9c9e00640 1 -- 172.21.15.96:0/1337911220 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9c4141170 con 0x7fd9c40cf2f0 2024-05-28T06:23:12.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.185+0000 7fd9c9e00640 1 -- 172.21.15.96:0/1337911220 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9c40b3af0 con 0x7fd9c40cf2f0 2024-05-28T06:23:12.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.186+0000 7fd9c9e00640 1 -- 172.21.15.96:0/1337911220 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9c40689e0 con 0x7fd9c40cf2f0 2024-05-28T06:23:12.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.187+0000 7fd9bb400640 1 -- 172.21.15.96:0/1337911220 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) ==== 93233+0+0 (secure 0 0 0) 0x7fd9bc011070 con 0x7fd9c40cf2f0 2024-05-28T06:23:12.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.188+0000 7fd9bb400640 1 --2- 172.21.15.96:0/1337911220 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd9a406e120 0x7fd9a40705e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:12.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.188+0000 7fd9bb400640 1 -- 172.21.15.96:0/1337911220 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 5823+0+0 (secure 0 0 0) 0x7fd9bc08a4f0 con 0x7fd9c40cf2f0 2024-05-28T06:23:12.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.189+0000 7fd9c2a00640 1 --2- 172.21.15.96:0/1337911220 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd9a406e120 0x7fd9a40705e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:12.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.190+0000 7fd9c2a00640 1 --2- 172.21.15.96:0/1337911220 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd9a406e120 0x7fd9a40705e0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fd9b4009f30 tx=0x7fd9b400b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:23:12.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.193+0000 7fd9bb400640 1 -- 172.21.15.96:0/1337911220 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd9bc01a2b0 con 0x7fd9c40cf2f0 2024-05-28T06:23:12.418 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.423+0000 7fd9c9e00640 1 -- 172.21.15.96:0/1337911220 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""]}) -- 0x7fd9c4073f30 con 0x7fd9a406e120 2024-05-28T06:23:12.451 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.456+0000 7fd9bb400640 1 -- 172.21.15.96:0/1337911220 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fd9c4073f30 con 0x7fd9a406e120 2024-05-28T06:23:12.454 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.458+0000 7fd9c9e00640 1 -- 172.21.15.96:0/1337911220 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd9a406e120 msgr2=0x7fd9a40705e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:12.454 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.458+0000 7fd9c9e00640 1 --2- 172.21.15.96:0/1337911220 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd9a406e120 0x7fd9a40705e0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fd9b4009f30 tx=0x7fd9b400b040 comp rx=0 tx=0).stop 2024-05-28T06:23:12.454 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.458+0000 7fd9c9e00640 1 -- 172.21.15.96:0/1337911220 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9c40cf2f0 msgr2=0x7fd9c40b45d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:12.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.458+0000 7fd9c9e00640 1 --2- 172.21.15.96:0/1337911220 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9c40cf2f0 0x7fd9c40b45d0 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc00c410 tx=0x7fd9bc00c8d0 comp rx=0 tx=0).stop 2024-05-28T06:23:12.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.459+0000 7fd9c9e00640 1 -- 172.21.15.96:0/1337911220 shutdown_connections 2024-05-28T06:23:12.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.459+0000 7fd9c9e00640 1 --2- 172.21.15.96:0/1337911220 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd9a406e120 0x7fd9a40705e0 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:12.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.459+0000 7fd9c9e00640 1 --2- 172.21.15.96:0/1337911220 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9c40cf2f0 0x7fd9c40b45d0 unknown :-1 s=CLOSED pgs=1853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:12.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.459+0000 7fd9c9e00640 1 --2- 172.21.15.96:0/1337911220 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9c406f250 0x7fd9c415cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:12.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.459+0000 7fd9c9e00640 1 --2- 172.21.15.96:0/1337911220 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd9c406edf0 0x7fd9c415a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:12.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.459+0000 7fd9c9e00640 1 -- 172.21.15.96:0/1337911220 >> 172.21.15.96:0/1337911220 conn(0x7fd9c40b1e50 msgr2=0x7fd9c40ccc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:23:12.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.459+0000 7fd9c9e00640 1 -- 172.21.15.96:0/1337911220 shutdown_connections 2024-05-28T06:23:12.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.459+0000 7fd9c9e00640 1 -- 172.21.15.96:0/1337911220 wait complete. 2024-05-28T06:23:12.474 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-05-28T06:23:12.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.684+0000 7f4df6e00640 1 Processor -- start 2024-05-28T06:23:12.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.684+0000 7f4df6e00640 1 -- start start 2024-05-28T06:23:12.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.686+0000 7f4df6e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4df015e870 0x7f4df00d0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:12.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.686+0000 7f4df6e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f4df00d14d0 0x7f4df00cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:12.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.686+0000 7f4df6e00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4df00cfd00 con 0x7f4df00cf870 2024-05-28T06:23:12.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.686+0000 7f4df6e00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f4df00d2150 con 0x7f4df00d14d0 2024-05-28T06:23:12.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.686+0000 7f4df6e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f4df015ec50 con 0x7f4df015e870 2024-05-28T06:23:12.683 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.686+0000 7f4df6400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f4df00cf870 0x7f4df0162360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:50370/0 (socket says 172.21.15.96:50370) 2024-05-28T06:23:12.683 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.686+0000 7f4df6400640 1 -- 172.21.15.96:0/1228038337 learned_addr learned my addr 172.21.15.96:0/1228038337 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:23:12.683 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.687+0000 7f4df5a00640 1 --2- 172.21.15.96:0/1228038337 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4df015e870 0x7f4df00d0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:12.683 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.687+0000 7f4df5000640 1 --2- 172.21.15.96:0/1228038337 >> v2:172.21.15.123:3300/0 conn(0x7f4df00d14d0 0x7f4df00cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:12.683 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.687+0000 7f4defe00640 1 -- 172.21.15.96:0/1228038337 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2467528910 0 0) 0x7f4de8001110 con 0x7f4df00cf870 2024-05-28T06:23:12.684 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.687+0000 7f4defe00640 1 -- 172.21.15.96:0/1228038337 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4dd8003520 con 0x7f4df00cf870 2024-05-28T06:23:12.684 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.687+0000 7f4defe00640 1 -- 172.21.15.96:0/1228038337 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3169100494 0 0) 0x7f4dd8003520 con 0x7f4df00cf870 2024-05-28T06:23:12.684 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.687+0000 7f4defe00640 1 -- 172.21.15.96:0/1228038337 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4df015e870 msgr2=0x7f4df00d0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:23:12.684 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.687+0000 7f4defe00640 1 --2- 172.21.15.96:0/1228038337 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4df015e870 0x7f4df00d0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:12.684 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.687+0000 7f4defe00640 1 -- 172.21.15.96:0/1228038337 >> v2:172.21.15.123:3300/0 conn(0x7f4df00d14d0 msgr2=0x7f4df00cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:23:12.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.687+0000 7f4defe00640 1 --2- 172.21.15.96:0/1228038337 >> v2:172.21.15.123:3300/0 conn(0x7f4df00d14d0 0x7f4df00cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:12.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.687+0000 7f4defe00640 1 -- 172.21.15.96:0/1228038337 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4df00cff80 con 0x7f4df00cf870 2024-05-28T06:23:12.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.687+0000 7f4defe00640 1 -- 172.21.15.96:0/1228038337 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f4de8001840 con 0x7f4df00cf870 2024-05-28T06:23:12.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.687+0000 7f4df5000640 1 --2- 172.21.15.96:0/1228038337 >> v2:172.21.15.123:3300/0 conn(0x7f4df00d14d0 0x7f4df00cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-28T06:23:12.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.687+0000 7f4defe00640 1 -- 172.21.15.96:0/1228038337 >> v1:172.21.15.96:6789/0 conn(0x7f4df00cf870 legacy=0x7f4df0162360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:12.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.687+0000 7f4defe00640 1 --2- 172.21.15.96:0/1228038337 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4dd80039b0 0x7f4dd8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:12.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.688+0000 7f4defe00640 1 --2- 172.21.15.96:0/1228038337 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4df015e870 0x7f4dd8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:12.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.688+0000 7f4defe00640 1 --2- 172.21.15.96:0/1228038337 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4dd8004d40 0x7f4dd80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:12.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.688+0000 7f4defe00640 1 -- 172.21.15.96:0/1228038337 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4df00d2150 con 0x7f4dd80039b0 2024-05-28T06:23:12.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.688+0000 7f4defe00640 1 -- 172.21.15.96:0/1228038337 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f4df015ec50 con 0x7f4dd8004d40 2024-05-28T06:23:12.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.688+0000 7f4defe00640 1 -- 172.21.15.96:0/1228038337 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f4dd800b9c0 con 0x7f4df015e870 2024-05-28T06:23:12.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.688+0000 7f4df5a00640 1 --2- 172.21.15.96:0/1228038337 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4dd80039b0 0x7f4dd8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:12.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.688+0000 7f4df5000640 1 --2- 172.21.15.96:0/1228038337 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4df015e870 0x7f4dd8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:12.686 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.688+0000 7f4df5a00640 1 -- 172.21.15.96:0/1228038337 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4df015e870 msgr2=0x7f4dd8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:23:12.686 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.688+0000 7f4df5a00640 1 --2- 172.21.15.96:0/1228038337 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4df015e870 0x7f4dd8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:12.686 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.688+0000 7f4df5a00640 1 -- 172.21.15.96:0/1228038337 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4dd8004d40 msgr2=0x7f4dd80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:23:12.686 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.688+0000 7f4df6400640 1 --2- 172.21.15.96:0/1228038337 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4dd8004d40 0x7f4dd80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:12.686 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.688+0000 7f4df5a00640 1 --2- 172.21.15.96:0/1228038337 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4dd8004d40 0x7f4dd80051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:12.686 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.688+0000 7f4df5a00640 1 -- 172.21.15.96:0/1228038337 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4dd8003520 con 0x7f4dd80039b0 2024-05-28T06:23:12.690 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.690+0000 7f4df5a00640 1 --2- 172.21.15.96:0/1228038337 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4dd80039b0 0x7f4dd8003dd0 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7f4de4009af0 tx=0x7f4dd8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd239e0f2631124e server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:23:12.690 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.690+0000 7f4defe00640 1 -- 172.21.15.96:0/1228038337 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4de4036070 con 0x7f4dd80039b0 2024-05-28T06:23:12.690 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.690+0000 7f4defe00640 1 -- 172.21.15.96:0/1228038337 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4de4031400 con 0x7f4dd80039b0 2024-05-28T06:23:12.690 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.690+0000 7f4defe00640 1 -- 172.21.15.96:0/1228038337 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4de4009690 con 0x7f4dd80039b0 2024-05-28T06:23:12.691 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.690+0000 7f4df6e00640 1 -- 172.21.15.96:0/1228038337 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4dd80039b0 msgr2=0x7f4dd8003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:12.691 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.690+0000 7f4df6e00640 1 --2- 172.21.15.96:0/1228038337 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4dd80039b0 0x7f4dd8003dd0 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7f4de4009af0 tx=0x7f4dd8005830 comp rx=0 tx=0).stop 2024-05-28T06:23:12.691 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.690+0000 7f4df6400640 1 -- 172.21.15.96:0/1228038337 reap_dead start 2024-05-28T06:23:12.691 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.691+0000 7f4df6e00640 1 -- 172.21.15.96:0/1228038337 shutdown_connections 2024-05-28T06:23:12.691 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.691+0000 7f4df6e00640 1 -- 172.21.15.96:0/1228038337 >> 172.21.15.96:0/1228038337 conn(0x7f4df008bfc0 msgr2=0x7f4df00d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:23:12.691 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.691+0000 7f4df6e00640 1 -- 172.21.15.96:0/1228038337 shutdown_connections 2024-05-28T06:23:12.691 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.691+0000 7f4df6e00640 1 -- 172.21.15.96:0/1228038337 wait complete. 2024-05-28T06:23:12.691 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.692+0000 7f4df6e00640 1 Processor -- start 2024-05-28T06:23:12.691 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.693+0000 7f4df6e00640 1 -- start start 2024-05-28T06:23:12.691 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.694+0000 7f4df6e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4dd80039b0 0x7f4df0159040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:12.691 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.694+0000 7f4df6e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4df0159860 0x7f4df015f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:12.691 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.694+0000 7f4df6e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4df00cd280 0x7f4df00b94e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:12.691 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.694+0000 7f4df6e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4df00d0820 con 0x7f4df0159860 2024-05-28T06:23:12.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.694+0000 7f4df6e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f4df00d0540 con 0x7f4df00cd280 2024-05-28T06:23:12.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.694+0000 7f4df6e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f4df00d4fb0 con 0x7f4dd80039b0 2024-05-28T06:23:12.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.694+0000 7f4df5000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4df0159860 0x7f4df015f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:12.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.694+0000 7f4df5000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4df0159860 0x7f4df015f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:36932/0 (socket says 172.21.15.96:36932) 2024-05-28T06:23:12.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.694+0000 7f4df5000640 1 -- 172.21.15.96:0/3738091668 learned_addr learned my addr 172.21.15.96:0/3738091668 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:23:12.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.694+0000 7f4df5a00640 1 --2- 172.21.15.96:0/3738091668 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4dd80039b0 0x7f4df0159040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:12.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.694+0000 7f4df5000640 1 -- 172.21.15.96:0/3738091668 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4dd80039b0 msgr2=0x7f4df0159040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:12.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.694+0000 7f4df5000640 1 --2- 172.21.15.96:0/3738091668 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4dd80039b0 0x7f4df0159040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:12.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.694+0000 7f4df5000640 1 -- 172.21.15.96:0/3738091668 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4df00cd280 msgr2=0x7f4df00b94e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:12.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.694+0000 7f4df5000640 1 --2- 172.21.15.96:0/3738091668 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4df00cd280 0x7f4df00b94e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:12.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.694+0000 7f4df5000640 1 -- 172.21.15.96:0/3738091668 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4df015fc20 con 0x7f4df0159860 2024-05-28T06:23:12.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.695+0000 7f4df5000640 1 --2- 172.21.15.96:0/3738091668 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4df0159860 0x7f4df015f6e0 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7f4de0004780 tx=0x7f4de0007700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:23:12.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.695+0000 7f4ded600640 1 -- 172.21.15.96:0/3738091668 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4de0013780 con 0x7f4df0159860 2024-05-28T06:23:12.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.695+0000 7f4ded600640 1 -- 172.21.15.96:0/3738091668 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4de0019070 con 0x7f4df0159860 2024-05-28T06:23:12.693 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.695+0000 7f4ded600640 1 -- 172.21.15.96:0/3738091668 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4de0013b60 con 0x7f4df0159860 2024-05-28T06:23:12.693 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.695+0000 7f4df6e00640 1 -- 172.21.15.96:0/3738091668 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4df00cdbd0 con 0x7f4df0159860 2024-05-28T06:23:12.693 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.695+0000 7f4df6e00640 1 -- 172.21.15.96:0/3738091668 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4df00b7420 con 0x7f4df0159860 2024-05-28T06:23:12.693 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.696+0000 7f4df6e00640 1 -- 172.21.15.96:0/3738091668 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4df00689e0 con 0x7f4df0159860 2024-05-28T06:23:12.694 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.697+0000 7f4ded600640 1 -- 172.21.15.96:0/3738091668 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) ==== 93233+0+0 (secure 0 0 0) 0x7f4de0013d00 con 0x7f4df0159860 2024-05-28T06:23:12.694 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.698+0000 7f4ded600640 1 --2- 172.21.15.96:0/3738091668 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4dd406e150 0x7f4dd4070610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:12.694 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.698+0000 7f4ded600640 1 -- 172.21.15.96:0/3738091668 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 5823+0+0 (secure 0 0 0) 0x7f4de008b430 con 0x7f4df0159860 2024-05-28T06:23:12.694 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.699+0000 7f4df5a00640 1 --2- 172.21.15.96:0/3738091668 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4dd406e150 0x7f4dd4070610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:12.695 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.700+0000 7f4df5a00640 1 --2- 172.21.15.96:0/3738091668 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4dd406e150 0x7f4dd4070610 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f4de4033000 tx=0x7f4de4008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:23:12.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.702+0000 7f4ded600640 1 -- 172.21.15.96:0/3738091668 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4de005b3c0 con 0x7f4df0159860 2024-05-28T06:23:12.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.933+0000 7f4df6e00640 1 -- 172.21.15.96:0/3738091668 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) -- 0x7f4df0159580 con 0x7f4dd406e150 2024-05-28T06:23:12.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.945+0000 7f4ded600640 1 -- 172.21.15.96:0/3738091668 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+59 (secure 0 0 0) 0x7f4de4004040 con 0x7f4dd406e150 2024-05-28T06:23:12.940 INFO:teuthology.orchestra.run.smithi096.stdout:/volumes/_nogroup/sv1/f0a99b9f-1e23-4f2b-ae5c-c3aa6b024e70 2024-05-28T06:23:12.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.948+0000 7f4df6e00640 1 -- 172.21.15.96:0/3738091668 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4dd406e150 msgr2=0x7f4dd4070610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:12.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.948+0000 7f4df6e00640 1 --2- 172.21.15.96:0/3738091668 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4dd406e150 0x7f4dd4070610 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f4de4033000 tx=0x7f4de4008030 comp rx=0 tx=0).stop 2024-05-28T06:23:12.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.948+0000 7f4df6e00640 1 -- 172.21.15.96:0/3738091668 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4df0159860 msgr2=0x7f4df015f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:12.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.948+0000 7f4df6e00640 1 --2- 172.21.15.96:0/3738091668 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4df0159860 0x7f4df015f6e0 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7f4de0004780 tx=0x7f4de0007700 comp rx=0 tx=0).stop 2024-05-28T06:23:12.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.948+0000 7f4df6e00640 1 -- 172.21.15.96:0/3738091668 shutdown_connections 2024-05-28T06:23:12.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.948+0000 7f4df6e00640 1 --2- 172.21.15.96:0/3738091668 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f4dd406e150 0x7f4dd4070610 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:12.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.948+0000 7f4df6e00640 1 --2- 172.21.15.96:0/3738091668 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4df00cd280 0x7f4df00b94e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:12.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.948+0000 7f4df6e00640 1 --2- 172.21.15.96:0/3738091668 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4df0159860 0x7f4df015f6e0 unknown :-1 s=CLOSED pgs=1856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:12.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.948+0000 7f4df6e00640 1 --2- 172.21.15.96:0/3738091668 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4dd80039b0 0x7f4df0159040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:12.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.948+0000 7f4df6e00640 1 -- 172.21.15.96:0/3738091668 >> 172.21.15.96:0/3738091668 conn(0x7f4df008bfc0 msgr2=0x7f4df00d24e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:23:12.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.948+0000 7f4df6e00640 1 -- 172.21.15.96:0/3738091668 shutdown_connections 2024-05-28T06:23:12.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:12.948+0000 7f4df6e00640 1 -- 172.21.15.96:0/3738091668 wait complete. 2024-05-28T06:23:12.961 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:23:12.961 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/f0a99b9f-1e23-4f2b-ae5c-c3aa6b024e70 2024-05-28T06:23:12.961 DEBUG:teuthology.orchestra.run.smithi096:> ) 2024-05-28T06:23:13.019 INFO:teuthology.orchestra.run.smithi096.stdout:# file: volumes/_nogroup/sv1/f0a99b9f-1e23-4f2b-ae5c-c3aa6b024e70 2024-05-28T06:23:13.019 INFO:teuthology.orchestra.run.smithi096.stdout:ceph.dir.rbytes="3138256896" 2024-05-28T06:23:13.019 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:23:13.020 INFO:tasks.cephfs.test_volumes:r_size = 3138256896 exp_size = 3221225472 2024-05-28T06:23:15.020 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:23:15.021 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/f0a99b9f-1e23-4f2b-ae5c-c3aa6b024e70 2024-05-28T06:23:15.021 DEBUG:teuthology.orchestra.run.smithi096:> ) 2024-05-28T06:23:15.089 INFO:teuthology.orchestra.run.smithi096.stdout:# file: volumes/_nogroup/sv1/f0a99b9f-1e23-4f2b-ae5c-c3aa6b024e70 2024-05-28T06:23:15.090 INFO:teuthology.orchestra.run.smithi096.stdout:ceph.dir.rbytes="3138256896" 2024-05-28T06:23:15.090 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:23:15.090 INFO:tasks.cephfs.test_volumes:r_size = 3138256896 exp_size = 3221225472 2024-05-28T06:23:17.092 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:23:17.092 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/f0a99b9f-1e23-4f2b-ae5c-c3aa6b024e70 2024-05-28T06:23:17.092 DEBUG:teuthology.orchestra.run.smithi096:> ) 2024-05-28T06:23:17.159 INFO:teuthology.orchestra.run.smithi096.stdout:# file: volumes/_nogroup/sv1/f0a99b9f-1e23-4f2b-ae5c-c3aa6b024e70 2024-05-28T06:23:17.159 INFO:teuthology.orchestra.run.smithi096.stdout:ceph.dir.rbytes="3221225472" 2024-05-28T06:23:17.160 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:23:17.161 INFO:tasks.cephfs.test_volumes:r_size = 3221225472 exp_size = 3221225472 2024-05-28T06:23:17.161 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone36081 2024-05-28T06:23:17.373 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:17.378+0000 7fc88dc00640 1 Processor -- start 2024-05-28T06:23:17.373 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:17.378+0000 7fc88dc00640 1 -- start start 2024-05-28T06:23:17.375 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:17.379+0000 7fc88dc00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc888156bd0 0x7fc8880b3af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:17.375 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:17.379+0000 7fc88dc00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fc8880b4e30 0x7fc88815caf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:17.375 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:17.379+0000 7fc88dc00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc888156fb0 con 0x7fc88815d030 2024-05-28T06:23:17.376 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:17.379+0000 7fc88dc00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fc88815dae0 con 0x7fc8880b4e30 2024-05-28T06:23:17.376 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:17.379+0000 7fc88dc00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fc88815d520 con 0x7fc888156bd0 2024-05-28T06:23:17.376 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:17.379+0000 7fc87e000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fc8880b4e30 0x7fc88815caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:17.376 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:17.380+0000 7fc87e000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fc8880b4e30 0x7fc88815caf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:36942/0 (socket says 172.21.15.96:36942) 2024-05-28T06:23:17.376 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:17.380+0000 7fc87e000640 1 -- 172.21.15.96:0/3229953986 learned_addr learned my addr 172.21.15.96:0/3229953986 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:23:17.376 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:17.380+0000 7fc886a00640 1 --2- 172.21.15.96:0/3229953986 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc888156bd0 0x7fc8880b3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:17.376 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:17.380+0000 7fc87e000640 1 -- 172.21.15.96:0/3229953986 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc888156bd0 msgr2=0x7fc8880b3af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:23:17.376 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:17.380+0000 7fc87e000640 1 --2- 172.21.15.96:0/3229953986 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc888156bd0 0x7fc8880b3af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:17.376 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:17.380+0000 7fc87e000640 1 -- 172.21.15.96:0/3229953986 >> v1:172.21.15.123:6789/0 conn(0x7fc88815d030 legacy=0x7fc8880d06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:23:17.377 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:17.380+0000 7fc87e000640 1 -- 172.21.15.96:0/3229953986 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc88815d800 con 0x7fc8880b4e30 2024-05-28T06:23:17.378 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:17.381+0000 7fc87e000640 1 --2- 172.21.15.96:0/3229953986 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc8880b4e30 0x7fc88815caf0 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7fc888074150 tx=0x7fc87002a160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af20eaac779cdc31 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:23:17.378 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:17.382+0000 7fc886000640 1 -- 172.21.15.96:0/3229953986 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc870037070 con 0x7fc8880b4e30 2024-05-28T06:23:17.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:17.382+0000 7fc886000640 1 -- 172.21.15.96:0/3229953986 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc870032440 con 0x7fc8880b4e30 2024-05-28T06:23:17.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:17.382+0000 7fc886000640 1 -- 172.21.15.96:0/3229953986 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc870032810 con 0x7fc8880b4e30 2024-05-28T06:23:17.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:17.382+0000 7fc88dc00640 1 -- 172.21.15.96:0/3229953986 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc8880b4e30 msgr2=0x7fc88815caf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:17.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:17.382+0000 7fc88dc00640 1 --2- 172.21.15.96:0/3229953986 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc8880b4e30 0x7fc88815caf0 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7fc888074150 tx=0x7fc87002a160 comp rx=0 tx=0).stop 2024-05-28T06:23:17.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:17.383+0000 7fc88dc00640 1 -- 172.21.15.96:0/3229953986 shutdown_connections 2024-05-28T06:23:17.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:17.383+0000 7fc88dc00640 1 --2- 172.21.15.96:0/3229953986 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc8880b4e30 0x7fc88815caf0 unknown :-1 s=CLOSED pgs=1857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:17.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:17.383+0000 7fc88dc00640 1 --2- 172.21.15.96:0/3229953986 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc888156bd0 0x7fc8880b3af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:17.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:17.383+0000 7fc88dc00640 1 -- 172.21.15.96:0/3229953986 >> 172.21.15.96:0/3229953986 conn(0x7fc88806bd20 msgr2=0x7fc88806c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:23:17.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:17.383+0000 7fc88dc00640 1 -- 172.21.15.96:0/3229953986 shutdown_connections 2024-05-28T06:23:17.380 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:17.383+0000 7fc88dc00640 1 -- 172.21.15.96:0/3229953986 wait complete. 2024-05-28T06:23:17.382 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:17.384+0000 7fc88dc00640 1 Processor -- start 2024-05-28T06:23:17.382 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:17.384+0000 7fc88dc00640 1 -- start start 2024-05-28T06:23:17.382 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:17.385+0000 7fc88dc00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc8880b4e30 0x7fc8880cf940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:17.383 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:17.385+0000 7fc88dc00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc888156bd0 0x7fc88815f960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:17.383 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:17.385+0000 7fc88dc00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc88815d030 0x7fc888160680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:17.383 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:17.385+0000 7fc88dc00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc8880b4030 con 0x7fc8880b4e30 2024-05-28T06:23:17.383 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:17.385+0000 7fc88dc00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc8880b35d0 con 0x7fc88815d030 2024-05-28T06:23:17.383 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:17.385+0000 7fc88dc00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fc8880b41b0 con 0x7fc888156bd0 2024-05-28T06:23:17.383 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:17.386+0000 7fc886a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc8880b4e30 0x7fc8880cf940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:17.383 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:17.386+0000 7fc886a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc8880b4e30 0x7fc8880cf940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:36952/0 (socket says 172.21.15.96:36952) 2024-05-28T06:23:17.383 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:17.386+0000 7fc886a00640 1 -- 172.21.15.96:0/210744897 learned_addr learned my addr 172.21.15.96:0/210744897 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:23:17.383 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:17.386+0000 7fc87e000640 1 --2- 172.21.15.96:0/210744897 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc888156bd0 0x7fc88815f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:17.383 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:17.386+0000 7fc886a00640 1 -- 172.21.15.96:0/210744897 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc888156bd0 msgr2=0x7fc88815f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:17.383 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:17.386+0000 7fc886a00640 1 --2- 172.21.15.96:0/210744897 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc888156bd0 0x7fc88815f960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:17.383 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:17.386+0000 7fc886a00640 1 -- 172.21.15.96:0/210744897 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc88815d030 msgr2=0x7fc888160680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:17.383 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:17.386+0000 7fc887400640 1 --2- 172.21.15.96:0/210744897 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc88815d030 0x7fc888160680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:17.384 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:17.386+0000 7fc886a00640 1 --2- 172.21.15.96:0/210744897 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc88815d030 0x7fc888160680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:17.384 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:17.386+0000 7fc886a00640 1 -- 172.21.15.96:0/210744897 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc888160bc0 con 0x7fc8880b4e30 2024-05-28T06:23:17.384 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:17.386+0000 7fc886a00640 1 --2- 172.21.15.96:0/210744897 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc8880b4e30 0x7fc8880cf940 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7fc878007ef0 tx=0x7fc87800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:23:17.384 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:17.387+0000 7fc87f400640 1 -- 172.21.15.96:0/210744897 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc878014620 con 0x7fc8880b4e30 2024-05-28T06:23:17.384 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:17.387+0000 7fc87f400640 1 -- 172.21.15.96:0/210744897 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc878019070 con 0x7fc8880b4e30 2024-05-28T06:23:17.384 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:17.387+0000 7fc87f400640 1 -- 172.21.15.96:0/210744897 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc878014a00 con 0x7fc8880b4e30 2024-05-28T06:23:17.384 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:17.387+0000 7fc88dc00640 1 -- 172.21.15.96:0/210744897 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8881f06c0 con 0x7fc8880b4e30 2024-05-28T06:23:17.384 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:17.387+0000 7fc88dc00640 1 -- 172.21.15.96:0/210744897 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8881f0c80 con 0x7fc8880b4e30 2024-05-28T06:23:17.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:17.388+0000 7fc88dc00640 1 -- 172.21.15.96:0/210744897 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc888068240 con 0x7fc8880b4e30 2024-05-28T06:23:17.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:17.389+0000 7fc87f400640 1 -- 172.21.15.96:0/210744897 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) ==== 93233+0+0 (secure 0 0 0) 0x7fc878014ba0 con 0x7fc8880b4e30 2024-05-28T06:23:17.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:17.390+0000 7fc87f400640 1 --2- 172.21.15.96:0/210744897 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc86406e150 0x7fc864070610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:17.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:17.390+0000 7fc87f400640 1 -- 172.21.15.96:0/210744897 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 5823+0+0 (secure 0 0 0) 0x7fc87808b2d0 con 0x7fc8880b4e30 2024-05-28T06:23:17.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:17.391+0000 7fc87e000640 1 --2- 172.21.15.96:0/210744897 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc86406e150 0x7fc864070610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:17.386 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:17.392+0000 7fc87e000640 1 --2- 172.21.15.96:0/210744897 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc86406e150 0x7fc864070610 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fc870034000 tx=0x7fc87002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:23:17.390 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:17.395+0000 7fc87f400640 1 -- 172.21.15.96:0/210744897 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc888068240 con 0x7fc8880b4e30 2024-05-28T06:23:17.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:17.632+0000 7fc88dc00640 1 -- 172.21.15.96:0/210744897 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone36081", "target": ["mon-mgr", ""]}) -- 0x7fc8880689e0 con 0x7fc86406e150 2024-05-28T06:23:17.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:17.801+0000 7fc87f400640 1 -- 172.21.15.96:0/210744897 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fc8880689e0 con 0x7fc86406e150 2024-05-28T06:23:17.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:17.804+0000 7fc88dc00640 1 -- 172.21.15.96:0/210744897 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc86406e150 msgr2=0x7fc864070610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:17.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:17.804+0000 7fc88dc00640 1 --2- 172.21.15.96:0/210744897 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc86406e150 0x7fc864070610 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fc870034000 tx=0x7fc87002e000 comp rx=0 tx=0).stop 2024-05-28T06:23:17.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:17.804+0000 7fc88dc00640 1 -- 172.21.15.96:0/210744897 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc8880b4e30 msgr2=0x7fc8880cf940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:17.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:17.804+0000 7fc88dc00640 1 --2- 172.21.15.96:0/210744897 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc8880b4e30 0x7fc8880cf940 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7fc878007ef0 tx=0x7fc87800e4b0 comp rx=0 tx=0).stop 2024-05-28T06:23:17.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:17.804+0000 7fc88dc00640 1 -- 172.21.15.96:0/210744897 shutdown_connections 2024-05-28T06:23:17.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:17.804+0000 7fc88dc00640 1 --2- 172.21.15.96:0/210744897 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc86406e150 0x7fc864070610 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:17.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:17.804+0000 7fc88dc00640 1 --2- 172.21.15.96:0/210744897 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc88815d030 0x7fc888160680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:17.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:17.804+0000 7fc88dc00640 1 --2- 172.21.15.96:0/210744897 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc888156bd0 0x7fc88815f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:17.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:17.804+0000 7fc88dc00640 1 --2- 172.21.15.96:0/210744897 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc8880b4e30 0x7fc8880cf940 unknown :-1 s=CLOSED pgs=1858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:17.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:17.804+0000 7fc88dc00640 1 -- 172.21.15.96:0/210744897 >> 172.21.15.96:0/210744897 conn(0x7fc88806bd20 msgr2=0x7fc88815bbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:23:17.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:17.805+0000 7fc88dc00640 1 -- 172.21.15.96:0/210744897 shutdown_connections 2024-05-28T06:23:17.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:17.805+0000 7fc88dc00640 1 -- 172.21.15.96:0/210744897 wait complete. 2024-05-28T06:23:17.803 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:23:17.808+0000 7fea60c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:23:17.803 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:23:17.808+0000 7fea60c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:23:17.803 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:23:17.808+0000 7fea60c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:23:17.803 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:23:17.808+0000 7fea60c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:23:17.803 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:23:17.808+0000 7fea60c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:23:17.821 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone36082 2024-05-28T06:23:17.825 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:23:17.830+0000 7fea5f800640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:23:17.825 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:23:17.830+0000 7fea5f800640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:23:17.825 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:23:17.830+0000 7fea5f800640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:23:17.825 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:23:17.830+0000 7fea5f800640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:23:17.825 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:23:17.830+0000 7fea5f800640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:23:18.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.037+0000 7f883ee00640 1 Processor -- start 2024-05-28T06:23:18.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.037+0000 7f883ee00640 1 -- start start 2024-05-28T06:23:18.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.039+0000 7f883ee00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f88380af840 0x7f8838162ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:18.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.039+0000 7f883ee00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f883806ef10 0x7f88380ad090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:18.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.039+0000 7f883ee00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f883806de70 con 0x7f8838071ac0 2024-05-28T06:23:18.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.039+0000 7f883ee00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f883806e150 con 0x7f88380af840 2024-05-28T06:23:18.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.039+0000 7f883ee00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f883806e430 con 0x7f883806ef10 2024-05-28T06:23:18.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.039+0000 7f883da00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f8838071ac0 0x7f8838071ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:50378/0 (socket says 172.21.15.96:50378) 2024-05-28T06:23:18.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.039+0000 7f883da00640 1 -- 172.21.15.96:0/1874154391 learned_addr learned my addr 172.21.15.96:0/1874154391 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:23:18.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.039+0000 7f883d000640 1 --2- 172.21.15.96:0/1874154391 >> v2:172.21.15.123:3300/0 conn(0x7f88380af840 0x7f8838162ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:18.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.039+0000 7f8837e00640 1 -- 172.21.15.96:0/1874154391 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2189736786 0 0) 0x7f883806de70 con 0x7f8838071ac0 2024-05-28T06:23:18.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.039+0000 7f883e400640 1 --2- 172.21.15.96:0/1874154391 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f883806ef10 0x7f88380ad090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:18.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.039+0000 7f8837e00640 1 -- 172.21.15.96:0/1874154391 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8820003520 con 0x7f8838071ac0 2024-05-28T06:23:18.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.040+0000 7f8837e00640 1 -- 172.21.15.96:0/1874154391 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 617612428 0 0) 0x7f8820003520 con 0x7f8838071ac0 2024-05-28T06:23:18.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.040+0000 7f8837e00640 1 -- 172.21.15.96:0/1874154391 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f883806ef10 msgr2=0x7f88380ad090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:23:18.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.040+0000 7f8837e00640 1 --2- 172.21.15.96:0/1874154391 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f883806ef10 0x7f88380ad090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:18.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.040+0000 7f8837e00640 1 -- 172.21.15.96:0/1874154391 >> v2:172.21.15.123:3300/0 conn(0x7f88380af840 msgr2=0x7f8838162ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:23:18.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.040+0000 7f8837e00640 1 --2- 172.21.15.96:0/1874154391 >> v2:172.21.15.123:3300/0 conn(0x7f88380af840 0x7f8838162ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:18.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.040+0000 7f8837e00640 1 -- 172.21.15.96:0/1874154391 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f883806e710 con 0x7f8838071ac0 2024-05-28T06:23:18.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.040+0000 7f8837e00640 1 -- 172.21.15.96:0/1874154391 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f882c001670 con 0x7f8838071ac0 2024-05-28T06:23:18.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.040+0000 7f8837e00640 1 -- 172.21.15.96:0/1874154391 >> v1:172.21.15.96:6789/0 conn(0x7f8838071ac0 legacy=0x7f8838071ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:18.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.040+0000 7f8837e00640 1 --2- 172.21.15.96:0/1874154391 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f88200039b0 0x7f8820003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:18.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.040+0000 7f8837e00640 1 --2- 172.21.15.96:0/1874154391 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f883806ef10 0x7f8820004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:18.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.040+0000 7f8837e00640 1 --2- 172.21.15.96:0/1874154391 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8820004d40 0x7f88200051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:18.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.040+0000 7f8837e00640 1 -- 172.21.15.96:0/1874154391 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f883806e150 con 0x7f8820004d40 2024-05-28T06:23:18.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.040+0000 7f8837e00640 1 -- 172.21.15.96:0/1874154391 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f883806e430 con 0x7f88200039b0 2024-05-28T06:23:18.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.040+0000 7f8837e00640 1 -- 172.21.15.96:0/1874154391 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f882000b9c0 con 0x7f883806ef10 2024-05-28T06:23:18.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.040+0000 7f883e400640 1 --2- 172.21.15.96:0/1874154391 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8820004d40 0x7f88200051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:18.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.041+0000 7f883da00640 1 --2- 172.21.15.96:0/1874154391 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f88200039b0 0x7f8820003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:18.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.041+0000 7f883d000640 1 --2- 172.21.15.96:0/1874154391 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f883806ef10 0x7f8820004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:18.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.041+0000 7f883e400640 1 -- 172.21.15.96:0/1874154391 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f883806ef10 msgr2=0x7f8820004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:23:18.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.041+0000 7f883e400640 1 --2- 172.21.15.96:0/1874154391 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f883806ef10 0x7f8820004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:18.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.041+0000 7f883e400640 1 -- 172.21.15.96:0/1874154391 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f88200039b0 msgr2=0x7f8820003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:23:18.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.041+0000 7f883e400640 1 --2- 172.21.15.96:0/1874154391 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f88200039b0 0x7f8820003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:18.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.041+0000 7f883e400640 1 -- 172.21.15.96:0/1874154391 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8820003520 con 0x7f8820004d40 2024-05-28T06:23:18.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.042+0000 7f883e400640 1 --2- 172.21.15.96:0/1874154391 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8820004d40 0x7f88200051c0 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7f8830009fc0 tx=0x7f8820005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7f1b6fe5c86f188 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:23:18.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.042+0000 7f8837e00640 1 -- 172.21.15.96:0/1874154391 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8830009c10 con 0x7f8820004d40 2024-05-28T06:23:18.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.042+0000 7f8837e00640 1 -- 172.21.15.96:0/1874154391 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8830036070 con 0x7f8820004d40 2024-05-28T06:23:18.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.042+0000 7f8837e00640 1 -- 172.21.15.96:0/1874154391 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8830031880 con 0x7f8820004d40 2024-05-28T06:23:18.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.043+0000 7f883ee00640 1 -- 172.21.15.96:0/1874154391 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8820004d40 msgr2=0x7f88200051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:18.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.043+0000 7f883ee00640 1 --2- 172.21.15.96:0/1874154391 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8820004d40 0x7f88200051c0 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7f8830009fc0 tx=0x7f8820005830 comp rx=0 tx=0).stop 2024-05-28T06:23:18.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.043+0000 7f883e400640 1 -- 172.21.15.96:0/1874154391 reap_dead start 2024-05-28T06:23:18.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.043+0000 7f883ee00640 1 -- 172.21.15.96:0/1874154391 shutdown_connections 2024-05-28T06:23:18.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.043+0000 7f883ee00640 1 -- 172.21.15.96:0/1874154391 >> 172.21.15.96:0/1874154391 conn(0x7f88380b1db0 msgr2=0x7f88380b21c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:23:18.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.043+0000 7f883ee00640 1 -- 172.21.15.96:0/1874154391 shutdown_connections 2024-05-28T06:23:18.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.043+0000 7f883ee00640 1 -- 172.21.15.96:0/1874154391 wait complete. 2024-05-28T06:23:18.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.045+0000 7f883ee00640 1 Processor -- start 2024-05-28T06:23:18.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.045+0000 7f883ee00640 1 -- start start 2024-05-28T06:23:18.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.046+0000 7f883ee00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8820004d40 0x7f88381f8210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:18.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.046+0000 7f883ee00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f883814bc20 0x7f883809a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:18.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.046+0000 7f883ee00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f883809a6e0 0x7f88380b4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:18.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.046+0000 7f883ee00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f883815d1e0 con 0x7f8820004d40 2024-05-28T06:23:18.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.046+0000 7f883ee00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f883806db20 con 0x7f883814bc20 2024-05-28T06:23:18.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.046+0000 7f883ee00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f883815d360 con 0x7f883809a6e0 2024-05-28T06:23:18.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.046+0000 7f883da00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8820004d40 0x7f88381f8210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:18.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.046+0000 7f883da00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8820004d40 0x7f88381f8210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:36964/0 (socket says 172.21.15.96:36964) 2024-05-28T06:23:18.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.046+0000 7f883da00640 1 -- 172.21.15.96:0/403673915 learned_addr learned my addr 172.21.15.96:0/403673915 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:23:18.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.047+0000 7f883d000640 1 --2- 172.21.15.96:0/403673915 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f883814bc20 0x7f883809a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:18.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.047+0000 7f883e400640 1 --2- 172.21.15.96:0/403673915 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f883809a6e0 0x7f88380b4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:18.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.047+0000 7f883da00640 1 -- 172.21.15.96:0/403673915 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f883809a6e0 msgr2=0x7f88380b4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:18.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.047+0000 7f883da00640 1 --2- 172.21.15.96:0/403673915 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f883809a6e0 0x7f88380b4690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:18.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.047+0000 7f883da00640 1 -- 172.21.15.96:0/403673915 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f883814bc20 msgr2=0x7f883809a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:18.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.047+0000 7f883da00640 1 --2- 172.21.15.96:0/403673915 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f883814bc20 0x7f883809a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:18.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.047+0000 7f883da00640 1 -- 172.21.15.96:0/403673915 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88381f8750 con 0x7f8820004d40 2024-05-28T06:23:18.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.047+0000 7f883e400640 1 --2- 172.21.15.96:0/403673915 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f883809a6e0 0x7f88380b4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:23:18.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.047+0000 7f883d000640 1 --2- 172.21.15.96:0/403673915 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f883814bc20 0x7f883809a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:23:18.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.047+0000 7f883da00640 1 --2- 172.21.15.96:0/403673915 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8820004d40 0x7f88381f8210 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7f882c00e780 tx=0x7f882c00ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:23:18.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.047+0000 7f8835600640 1 -- 172.21.15.96:0/403673915 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f882c00c870 con 0x7f8820004d40 2024-05-28T06:23:18.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.047+0000 7f8835600640 1 -- 172.21.15.96:0/403673915 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f882c018070 con 0x7f8820004d40 2024-05-28T06:23:18.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.047+0000 7f883ee00640 1 -- 172.21.15.96:0/403673915 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88381f89d0 con 0x7f8820004d40 2024-05-28T06:23:18.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.047+0000 7f8835600640 1 -- 172.21.15.96:0/403673915 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f882c00cc50 con 0x7f8820004d40 2024-05-28T06:23:18.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.047+0000 7f883ee00640 1 -- 172.21.15.96:0/403673915 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88381f8f90 con 0x7f8820004d40 2024-05-28T06:23:18.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.049+0000 7f883ee00640 1 -- 172.21.15.96:0/403673915 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88380689e0 con 0x7f8820004d40 2024-05-28T06:23:18.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.050+0000 7f8835600640 1 -- 172.21.15.96:0/403673915 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 38) ==== 93233+0+0 (secure 0 0 0) 0x7f882c01d460 con 0x7f8820004d40 2024-05-28T06:23:18.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.050+0000 7f8835600640 1 --2- 172.21.15.96:0/403673915 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f881806e120 0x7f88180705e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:18.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.051+0000 7f8835600640 1 -- 172.21.15.96:0/403673915 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 5823+0+0 (secure 0 0 0) 0x7f882c08a8c0 con 0x7f8820004d40 2024-05-28T06:23:18.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.051+0000 7f883d000640 1 --2- 172.21.15.96:0/403673915 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f881806e120 0x7f88180705e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:18.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.052+0000 7f883d000640 1 --2- 172.21.15.96:0/403673915 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f881806e120 0x7f88180705e0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f883814c060 tx=0x7f8824008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:23:18.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.055+0000 7f8835600640 1 -- 172.21.15.96:0/403673915 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f88381f8750 con 0x7f8820004d40 2024-05-28T06:23:18.286 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.290+0000 7f883ee00640 1 -- 172.21.15.96:0/403673915 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone36082", "target": ["mon-mgr", ""]}) -- 0x7f88380b5360 con 0x7f881806e120 2024-05-28T06:23:18.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.706+0000 7f8835600640 1 -- 172.21.15.96:0/403673915 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mgrmap(e 39) ==== 93335+0+0 (secure 0 0 0) 0x7f882c0566c0 con 0x7f8820004d40 2024-05-28T06:23:18.804 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:23:18.808+0000 7fea28800640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:23:18.805 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:23:18.808+0000 7fea28800640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:23:18.805 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:23:18.808+0000 7fea28800640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:23:18.805 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:23:18.808+0000 7fea28800640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:23:18.805 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:23:18.808+0000 7fea28800640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:23:18.956 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:23:18.958+0000 7fea60200640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:23:18.956 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:23:18.958+0000 7fea60200640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:23:18.956 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:23:18.958+0000 7fea60200640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:23:18.957 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:23:18.958+0000 7fea60200640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:23:18.957 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:23:18.958+0000 7fea60200640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:23:18.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.945+0000 7f8835600640 1 -- 172.21.15.96:0/403673915 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f88380b5360 con 0x7f881806e120 2024-05-28T06:23:18.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.948+0000 7f883ee00640 1 -- 172.21.15.96:0/403673915 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f881806e120 msgr2=0x7f88180705e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:18.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.948+0000 7f883ee00640 1 --2- 172.21.15.96:0/403673915 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f881806e120 0x7f88180705e0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f883814c060 tx=0x7f8824008040 comp rx=0 tx=0).stop 2024-05-28T06:23:18.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.948+0000 7f883ee00640 1 -- 172.21.15.96:0/403673915 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8820004d40 msgr2=0x7f88381f8210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:18.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.948+0000 7f883ee00640 1 --2- 172.21.15.96:0/403673915 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8820004d40 0x7f88381f8210 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7f882c00e780 tx=0x7f882c00ec40 comp rx=0 tx=0).stop 2024-05-28T06:23:18.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.949+0000 7f883ee00640 1 -- 172.21.15.96:0/403673915 shutdown_connections 2024-05-28T06:23:18.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.949+0000 7f883ee00640 1 --2- 172.21.15.96:0/403673915 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f881806e120 0x7f88180705e0 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:18.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.949+0000 7f883ee00640 1 --2- 172.21.15.96:0/403673915 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f883809a6e0 0x7f88380b4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:18.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.949+0000 7f883ee00640 1 --2- 172.21.15.96:0/403673915 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f883814bc20 0x7f883809a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:18.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.949+0000 7f883ee00640 1 --2- 172.21.15.96:0/403673915 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8820004d40 0x7f88381f8210 unknown :-1 s=CLOSED pgs=1861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:18.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.949+0000 7f883ee00640 1 -- 172.21.15.96:0/403673915 >> 172.21.15.96:0/403673915 conn(0x7f88380b1db0 msgr2=0x7f8838071ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:23:18.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.949+0000 7f883ee00640 1 -- 172.21.15.96:0/403673915 shutdown_connections 2024-05-28T06:23:18.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:18.949+0000 7f883ee00640 1 -- 172.21.15.96:0/403673915 wait complete. 2024-05-28T06:23:18.978 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone36083 2024-05-28T06:23:19.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:19.244+0000 7f0fcfa00640 1 Processor -- start 2024-05-28T06:23:19.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:19.244+0000 7f0fcfa00640 1 -- start start 2024-05-28T06:23:19.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:19.245+0000 7f0fcfa00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f0fc815ee40 0x7f0fc806d970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:19.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:19.245+0000 7f0fcfa00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f0fc815ea30 0x7f0fc80d10e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:19.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:19.245+0000 7f0fcfa00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fc806cbb0 con 0x7f0fc815fa50 2024-05-28T06:23:19.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:19.245+0000 7f0fcfa00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f0fc806ce90 con 0x7f0fc815ee40 2024-05-28T06:23:19.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:19.246+0000 7f0fcfa00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f0fc806d170 con 0x7f0fc815ea30 2024-05-28T06:23:19.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:19.246+0000 7f0fcd000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f0fc815ee40 0x7f0fc806d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:19.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:19.246+0000 7f0fcd000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f0fc815ee40 0x7f0fc806d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:57304/0 (socket says 172.21.15.96:57304) 2024-05-28T06:23:19.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:19.246+0000 7f0fc7e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f0fc815fa50 0x7f0fc815f220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:35080/0 (socket says 172.21.15.96:35080) 2024-05-28T06:23:19.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:19.246+0000 7f0fcd000640 1 -- 172.21.15.96:0/279964901 learned_addr learned my addr 172.21.15.96:0/279964901 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:23:19.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:19.246+0000 7f0fcd000640 1 -- 172.21.15.96:0/279964901 >> v2:172.21.15.123:3300/0 conn(0x7f0fc815ea30 msgr2=0x7f0fc80d10e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-28T06:23:19.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:19.246+0000 7f0fcd000640 1 --2- 172.21.15.96:0/279964901 >> v2:172.21.15.123:3300/0 conn(0x7f0fc815ea30 0x7f0fc80d10e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:19.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:19.246+0000 7f0fcd000640 1 -- 172.21.15.96:0/279964901 >> v1:172.21.15.96:6789/0 conn(0x7f0fc815fa50 legacy=0x7f0fc815f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:19.249 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:19.246+0000 7f0fcd000640 1 -- 172.21.15.96:0/279964901 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fc806d450 con 0x7f0fc815ee40 2024-05-28T06:23:19.249 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:19.247+0000 7f0fcd000640 1 --2- 172.21.15.96:0/279964901 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0fc815ee40 0x7f0fc806d970 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7f0fc80b84f0 tx=0x7f0fbc02a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8a94942016ebdaf server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:23:19.249 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:19.248+0000 7f0fc7400640 1 -- 172.21.15.96:0/279964901 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fbc037070 con 0x7f0fc815ee40 2024-05-28T06:23:19.249 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:19.248+0000 7f0fc7400640 1 -- 172.21.15.96:0/279964901 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0fbc032440 con 0x7f0fc815ee40 2024-05-28T06:23:19.249 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:19.248+0000 7f0fc7400640 1 -- 172.21.15.96:0/279964901 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fbc032810 con 0x7f0fc815ee40 2024-05-28T06:23:19.250 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:19.249+0000 7f0fcfa00640 1 -- 172.21.15.96:0/279964901 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0fc815ee40 msgr2=0x7f0fc806d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:19.250 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:19.249+0000 7f0fcfa00640 1 --2- 172.21.15.96:0/279964901 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0fc815ee40 0x7f0fc806d970 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7f0fc80b84f0 tx=0x7f0fbc02a210 comp rx=0 tx=0).stop 2024-05-28T06:23:19.250 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:19.249+0000 7f0fcfa00640 1 -- 172.21.15.96:0/279964901 shutdown_connections 2024-05-28T06:23:19.250 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:19.249+0000 7f0fcfa00640 1 --2- 172.21.15.96:0/279964901 >> v2:172.21.15.123:3300/0 conn(0x7f0fc815ea30 0x7f0fc80d10e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:19.250 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:19.249+0000 7f0fcfa00640 1 --2- 172.21.15.96:0/279964901 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0fc815ee40 0x7f0fc806d970 unknown :-1 s=CLOSED pgs=1863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:19.250 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:19.249+0000 7f0fcfa00640 1 -- 172.21.15.96:0/279964901 >> 172.21.15.96:0/279964901 conn(0x7f0fc80b3f30 msgr2=0x7f0fc80b4340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:23:19.250 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:19.249+0000 7f0fcfa00640 1 -- 172.21.15.96:0/279964901 shutdown_connections 2024-05-28T06:23:19.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:19.249+0000 7f0fcfa00640 1 -- 172.21.15.96:0/279964901 wait complete. 2024-05-28T06:23:19.255 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:19.252+0000 7f0fcfa00640 1 Processor -- start 2024-05-28T06:23:19.255 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:19.253+0000 7f0fcfa00640 1 -- start start 2024-05-28T06:23:19.255 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:19.254+0000 7f0fcfa00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0fc815ea30 0x7f0fc815b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:19.255 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:19.254+0000 7f0fcfa00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0fc815ee40 0x7f0fc81f4c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:19.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:19.254+0000 7f0fcfa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0fc815fa50 0x7f0fc81f5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:19.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:19.254+0000 7f0fcfa00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f0fc809abb0 con 0x7f0fc815fa50 2024-05-28T06:23:19.256 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:19.254+0000 7f0fcfa00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f0fc809aa30 con 0x7f0fc815ee40 2024-05-28T06:23:19.257 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:19.254+0000 7f0fcfa00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f0fc809ad30 con 0x7f0fc815ea30 2024-05-28T06:23:19.257 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:19.254+0000 7f0fcda00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0fc815fa50 0x7f0fc81f5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:19.257 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:19.254+0000 7f0fcda00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0fc815fa50 0x7f0fc81f5170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:57308/0 (socket says 172.21.15.96:57308) 2024-05-28T06:23:19.257 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:19.254+0000 7f0fcda00640 1 -- 172.21.15.96:0/3007629013 learned_addr learned my addr 172.21.15.96:0/3007629013 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:23:19.257 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:19.255+0000 7f0fcda00640 1 -- 172.21.15.96:0/3007629013 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0fc815ea30 msgr2=0x7f0fc815b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:19.257 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:19.255+0000 7f0fcda00640 1 --2- 172.21.15.96:0/3007629013 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0fc815ea30 0x7f0fc815b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:19.258 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:19.255+0000 7f0fcda00640 1 -- 172.21.15.96:0/3007629013 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0fc815ee40 msgr2=0x7f0fc81f4c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:19.258 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:19.255+0000 7f0fcda00640 1 --2- 172.21.15.96:0/3007629013 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0fc815ee40 0x7f0fc81f4c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:19.258 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:19.255+0000 7f0fcda00640 1 -- 172.21.15.96:0/3007629013 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fc80b10e0 con 0x7f0fc815fa50 2024-05-28T06:23:19.258 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:19.255+0000 7f0fcda00640 1 --2- 172.21.15.96:0/3007629013 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0fc815fa50 0x7f0fc81f5170 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7f0fc000b9c0 tx=0x7f0fc000be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:23:19.258 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:19.255+0000 7f0fc4c00640 1 -- 172.21.15.96:0/3007629013 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fc0019070 con 0x7f0fc815fa50 2024-05-28T06:23:19.258 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:19.255+0000 7f0fc4c00640 1 -- 172.21.15.96:0/3007629013 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0fc000cba0 con 0x7f0fc815fa50 2024-05-28T06:23:19.258 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:19.255+0000 7f0fcfa00640 1 -- 172.21.15.96:0/3007629013 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fc81f5990 con 0x7f0fc815fa50 2024-05-28T06:23:19.258 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:19.255+0000 7f0fc4c00640 1 -- 172.21.15.96:0/3007629013 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fc0014440 con 0x7f0fc815fa50 2024-05-28T06:23:19.258 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:19.255+0000 7f0fcfa00640 1 -- 172.21.15.96:0/3007629013 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fc80b1660 con 0x7f0fc815fa50 2024-05-28T06:23:19.259 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:19.257+0000 7f0fcfa00640 1 -- 172.21.15.96:0/3007629013 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0fc80690f0 con 0x7f0fc815fa50 2024-05-28T06:23:19.259 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:19.258+0000 7f0fc4c00640 1 -- 172.21.15.96:0/3007629013 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 39) ==== 93335+0+0 (secure 0 0 0) 0x7f0fc001c070 con 0x7f0fc815fa50 2024-05-28T06:23:19.259 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:19.259+0000 7f0fc4c00640 1 --2- 172.21.15.96:0/3007629013 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0fa806e240 0x7f0fa8070700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:19.259 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:19.259+0000 7f0fc4c00640 1 -- 172.21.15.96:0/3007629013 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 5823+0+0 (secure 0 0 0) 0x7f0fc008a7c0 con 0x7f0fc815fa50 2024-05-28T06:23:19.259 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:19.259+0000 7f0fcd000640 1 --2- 172.21.15.96:0/3007629013 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0fa806e240 0x7f0fa8070700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:19.259 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:19.262+0000 7f0fcd000640 1 --2- 172.21.15.96:0/3007629013 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0fa806e240 0x7f0fa8070700 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc034000 tx=0x7f0fbc02c000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:23:19.260 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:19.265+0000 7f0fc4c00640 1 -- 172.21.15.96:0/3007629013 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0fc005a650 con 0x7f0fc815fa50 2024-05-28T06:23:19.657 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:19.662+0000 7f0fcfa00640 1 -- 172.21.15.96:0/3007629013 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone36083", "target": ["mon-mgr", ""]}) -- 0x7f0fc809b940 con 0x7f0fa806e240 2024-05-28T06:23:20.293 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:20.294+0000 7f0fc4c00640 1 -- 172.21.15.96:0/3007629013 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f0fc809b940 con 0x7f0fa806e240 2024-05-28T06:23:20.295 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:20.298+0000 7f0fcfa00640 1 -- 172.21.15.96:0/3007629013 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0fa806e240 msgr2=0x7f0fa8070700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:20.295 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:20.298+0000 7f0fcfa00640 1 --2- 172.21.15.96:0/3007629013 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0fa806e240 0x7f0fa8070700 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc034000 tx=0x7f0fbc02c000 comp rx=0 tx=0).stop 2024-05-28T06:23:20.295 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:20.298+0000 7f0fcfa00640 1 -- 172.21.15.96:0/3007629013 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0fc815fa50 msgr2=0x7f0fc81f5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:20.295 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:20.298+0000 7f0fcfa00640 1 --2- 172.21.15.96:0/3007629013 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0fc815fa50 0x7f0fc81f5170 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7f0fc000b9c0 tx=0x7f0fc000be80 comp rx=0 tx=0).stop 2024-05-28T06:23:20.295 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:20.299+0000 7f0fcfa00640 1 -- 172.21.15.96:0/3007629013 shutdown_connections 2024-05-28T06:23:20.295 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:20.299+0000 7f0fcfa00640 1 --2- 172.21.15.96:0/3007629013 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0fa806e240 0x7f0fa8070700 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:20.295 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:20.299+0000 7f0fcfa00640 1 --2- 172.21.15.96:0/3007629013 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0fc815fa50 0x7f0fc81f5170 unknown :-1 s=CLOSED pgs=1864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:20.295 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:20.299+0000 7f0fcfa00640 1 --2- 172.21.15.96:0/3007629013 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0fc815ee40 0x7f0fc81f4c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:20.295 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:20.299+0000 7f0fcfa00640 1 --2- 172.21.15.96:0/3007629013 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0fc815ea30 0x7f0fc815b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:20.296 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:20.299+0000 7f0fcfa00640 1 -- 172.21.15.96:0/3007629013 >> 172.21.15.96:0/3007629013 conn(0x7f0fc80b3f30 msgr2=0x7f0fc809b140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:23:20.296 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:20.299+0000 7f0fcfa00640 1 -- 172.21.15.96:0/3007629013 shutdown_connections 2024-05-28T06:23:20.296 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:20.299+0000 7f0fcfa00640 1 -- 172.21.15.96:0/3007629013 wait complete. 2024-05-28T06:23:20.343 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:23:20.349+0000 7fea60200640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:23:20.343 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:23:20.349+0000 7fea60200640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:23:20.344 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:23:20.349+0000 7fea60200640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:23:20.344 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:23:20.349+0000 7fea60200640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:23:20.344 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:23:20.350+0000 7fea60200640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:23:21.325 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone36081 2024-05-28T06:23:21.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:21.611+0000 7f1f7f400640 1 Processor -- start 2024-05-28T06:23:21.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:21.611+0000 7f1f7f400640 1 -- start start 2024-05-28T06:23:21.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:21.613+0000 7f1f7f400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1f7815e850 0x7f1f780b77b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:21.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:21.613+0000 7f1f7f400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f1f780cef30 0x7f1f780d1640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:21.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:21.613+0000 7f1f7f400640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f780b8000 con 0x7f1f780d1b80 2024-05-28T06:23:21.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:21.613+0000 7f1f7f400640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f1f7815ec30 con 0x7f1f780cef30 2024-05-28T06:23:21.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:21.613+0000 7f1f7f400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f1f780d1fd0 con 0x7f1f7815e850 2024-05-28T06:23:21.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:21.613+0000 7f1f7d400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f1f780d1b80 0x7f1f78162340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:35094/0 (socket says 172.21.15.96:35094) 2024-05-28T06:23:21.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:21.613+0000 7f1f7d400640 1 -- 172.21.15.96:0/3964431672 learned_addr learned my addr 172.21.15.96:0/3964431672 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:23:21.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:21.614+0000 7f1f77400640 1 -- 172.21.15.96:0/3964431672 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4002052064 0 0) 0x7f1f780b8000 con 0x7f1f780d1b80 2024-05-28T06:23:21.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:21.614+0000 7f1f77400640 1 -- 172.21.15.96:0/3964431672 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1f60003520 con 0x7f1f780d1b80 2024-05-28T06:23:21.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:21.614+0000 7f1f7ca00640 1 --2- 172.21.15.96:0/3964431672 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1f7815e850 0x7f1f780b77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:21.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:21.615+0000 7f1f77400640 1 -- 172.21.15.96:0/3964431672 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1711288837 0 0) 0x7f1f60003520 con 0x7f1f780d1b80 2024-05-28T06:23:21.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:21.615+0000 7f1f77400640 1 -- 172.21.15.96:0/3964431672 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1f7815e850 msgr2=0x7f1f780b77b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:23:21.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:21.615+0000 7f1f77400640 1 --2- 172.21.15.96:0/3964431672 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1f7815e850 0x7f1f780b77b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:21.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:21.615+0000 7f1f77400640 1 -- 172.21.15.96:0/3964431672 >> v2:172.21.15.123:3300/0 conn(0x7f1f780cef30 msgr2=0x7f1f780d1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:23:21.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:21.615+0000 7f1f77400640 1 --2- 172.21.15.96:0/3964431672 >> v2:172.21.15.123:3300/0 conn(0x7f1f780cef30 0x7f1f780d1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:21.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:21.615+0000 7f1f77400640 1 -- 172.21.15.96:0/3964431672 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f780b82e0 con 0x7f1f780d1b80 2024-05-28T06:23:21.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:21.615+0000 7f1f77400640 1 -- 172.21.15.96:0/3964431672 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f1f700019f0 con 0x7f1f780d1b80 2024-05-28T06:23:21.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:21.615+0000 7f1f77400640 1 -- 172.21.15.96:0/3964431672 >> v1:172.21.15.96:6789/0 conn(0x7f1f780d1b80 legacy=0x7f1f78162340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:21.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:21.615+0000 7f1f77400640 1 --2- 172.21.15.96:0/3964431672 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1f600039b0 0x7f1f60003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:21.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:21.615+0000 7f1f77400640 1 --2- 172.21.15.96:0/3964431672 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1f60004800 0x7f1f60004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:21.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:21.615+0000 7f1f77400640 1 --2- 172.21.15.96:0/3964431672 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1f600052e0 0x7f1f6000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:21.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:21.615+0000 7f1f77400640 1 -- 172.21.15.96:0/3964431672 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f1f780d1fd0 con 0x7f1f600052e0 2024-05-28T06:23:21.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:21.615+0000 7f1f77400640 1 -- 172.21.15.96:0/3964431672 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f1f6000c1c0 con 0x7f1f600039b0 2024-05-28T06:23:21.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:21.615+0000 7f1f77400640 1 -- 172.21.15.96:0/3964431672 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f1f6000c3c0 con 0x7f1f60004800 2024-05-28T06:23:21.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:21.615+0000 7f1f7ca00640 1 --2- 172.21.15.96:0/3964431672 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1f7815e850 0x7f1f780b77b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:23:21.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:21.616+0000 7f1f7d400640 1 --2- 172.21.15.96:0/3964431672 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1f600052e0 0x7f1f6000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:21.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:21.616+0000 7f1f7ca00640 1 --2- 172.21.15.96:0/3964431672 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1f600039b0 0x7f1f60003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:21.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:21.616+0000 7f1f77e00640 1 --2- 172.21.15.96:0/3964431672 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1f60004800 0x7f1f60004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:21.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:21.617+0000 7f1f7d400640 1 -- 172.21.15.96:0/3964431672 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1f60004800 msgr2=0x7f1f60004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:23:21.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:21.617+0000 7f1f7d400640 1 --2- 172.21.15.96:0/3964431672 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1f60004800 0x7f1f60004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:21.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:21.617+0000 7f1f7d400640 1 -- 172.21.15.96:0/3964431672 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1f600039b0 msgr2=0x7f1f60003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:23:21.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:21.617+0000 7f1f7d400640 1 --2- 172.21.15.96:0/3964431672 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1f600039b0 0x7f1f60003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:21.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:21.617+0000 7f1f7d400640 1 -- 172.21.15.96:0/3964431672 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f780b82e0 con 0x7f1f600052e0 2024-05-28T06:23:21.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:21.619+0000 7f1f7d400640 1 --2- 172.21.15.96:0/3964431672 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1f600052e0 0x7f1f6000b9c0 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7f1f7000a8e0 tx=0x7f1f60005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b650e2321723c0a7 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:23:21.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:21.619+0000 7f1f77400640 1 -- 172.21.15.96:0/3964431672 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f7000a4e0 con 0x7f1f600052e0 2024-05-28T06:23:21.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:21.619+0000 7f1f77400640 1 -- 172.21.15.96:0/3964431672 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f7002f070 con 0x7f1f600052e0 2024-05-28T06:23:21.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:21.619+0000 7f1f77400640 1 -- 172.21.15.96:0/3964431672 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f7002bd60 con 0x7f1f600052e0 2024-05-28T06:23:21.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:21.620+0000 7f1f7f400640 1 -- 172.21.15.96:0/3964431672 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1f600052e0 msgr2=0x7f1f6000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:21.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:21.620+0000 7f1f7f400640 1 --2- 172.21.15.96:0/3964431672 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1f600052e0 0x7f1f6000b9c0 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7f1f7000a8e0 tx=0x7f1f60005830 comp rx=0 tx=0).stop 2024-05-28T06:23:21.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:21.620+0000 7f1f7d400640 1 -- 172.21.15.96:0/3964431672 reap_dead start 2024-05-28T06:23:21.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:21.621+0000 7f1f7f400640 1 -- 172.21.15.96:0/3964431672 shutdown_connections 2024-05-28T06:23:21.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:21.621+0000 7f1f7f400640 1 -- 172.21.15.96:0/3964431672 >> 172.21.15.96:0/3964431672 conn(0x7f1f78067aa0 msgr2=0x7f1f78067eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:23:21.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:21.621+0000 7f1f7f400640 1 -- 172.21.15.96:0/3964431672 shutdown_connections 2024-05-28T06:23:21.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:21.621+0000 7f1f7f400640 1 -- 172.21.15.96:0/3964431672 wait complete. 2024-05-28T06:23:21.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:21.623+0000 7f1f7f400640 1 Processor -- start 2024-05-28T06:23:21.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:21.623+0000 7f1f7f400640 1 -- start start 2024-05-28T06:23:21.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:21.624+0000 7f1f7f400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1f78159870 0x7f1f780d4200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:21.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:21.624+0000 7f1f7f400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1f78159c50 0x7f1f780d4740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:21.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:21.624+0000 7f1f7f400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1f780d2b30 0x7f1f780d4c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:21.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:21.624+0000 7f1f7f400640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f1f780ba0b0 con 0x7f1f78159c50 2024-05-28T06:23:21.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:21.624+0000 7f1f7f400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f1f780b88a0 con 0x7f1f780d2b30 2024-05-28T06:23:21.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:21.624+0000 7f1f7f400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f1f780ba230 con 0x7f1f78159870 2024-05-28T06:23:21.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:21.624+0000 7f1f77e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1f78159c50 0x7f1f780d4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:21.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:21.624+0000 7f1f77e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1f78159c50 0x7f1f780d4740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:57328/0 (socket says 172.21.15.96:57328) 2024-05-28T06:23:21.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:21.624+0000 7f1f77e00640 1 -- 172.21.15.96:0/1090222590 learned_addr learned my addr 172.21.15.96:0/1090222590 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:23:21.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:21.625+0000 7f1f77e00640 1 -- 172.21.15.96:0/1090222590 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1f78159870 msgr2=0x7f1f780d4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:21.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:21.625+0000 7f1f77e00640 1 --2- 172.21.15.96:0/1090222590 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1f78159870 0x7f1f780d4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:21.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:21.625+0000 7f1f77e00640 1 -- 172.21.15.96:0/1090222590 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1f780d2b30 msgr2=0x7f1f780d4c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:21.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:21.625+0000 7f1f77e00640 1 --2- 172.21.15.96:0/1090222590 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1f780d2b30 0x7f1f780d4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:21.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:21.625+0000 7f1f77e00640 1 -- 172.21.15.96:0/1090222590 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f780d38e0 con 0x7f1f78159c50 2024-05-28T06:23:21.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:21.625+0000 7f1f77e00640 1 --2- 172.21.15.96:0/1090222590 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1f78159c50 0x7f1f780d4740 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7f1f6400d5a0 tx=0x7f1f6400da60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:23:21.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:21.626+0000 7f1f74c00640 1 -- 172.21.15.96:0/1090222590 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f64018020 con 0x7f1f78159c50 2024-05-28T06:23:21.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:21.626+0000 7f1f74c00640 1 -- 172.21.15.96:0/1090222590 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f64014620 con 0x7f1f78159c50 2024-05-28T06:23:21.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:21.626+0000 7f1f7f400640 1 -- 172.21.15.96:0/1090222590 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f780d3040 con 0x7f1f78159c50 2024-05-28T06:23:21.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:21.626+0000 7f1f74c00640 1 -- 172.21.15.96:0/1090222590 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f64014a10 con 0x7f1f78159c50 2024-05-28T06:23:21.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:21.626+0000 7f1f7f400640 1 -- 172.21.15.96:0/1090222590 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f780d3600 con 0x7f1f78159c50 2024-05-28T06:23:21.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:21.628+0000 7f1f7f400640 1 -- 172.21.15.96:0/1090222590 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f780689e0 con 0x7f1f78159c50 2024-05-28T06:23:21.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:21.629+0000 7f1f74c00640 1 -- 172.21.15.96:0/1090222590 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 40) ==== 93437+0+0 (secure 0 0 0) 0x7f1f6401b020 con 0x7f1f78159c50 2024-05-28T06:23:21.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:21.630+0000 7f1f74c00640 1 --2- 172.21.15.96:0/1090222590 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1f5806e390 0x7f1f58070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:21.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:21.630+0000 7f1f74c00640 1 -- 172.21.15.96:0/1090222590 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 5823+0+0 (secure 0 0 0) 0x7f1f6408ae10 con 0x7f1f78159c50 2024-05-28T06:23:21.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:21.635+0000 7f1f74c00640 1 -- 172.21.15.96:0/1090222590 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1f64022020 con 0x7f1f78159c50 2024-05-28T06:23:21.640 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:21.646+0000 7f1f7ca00640 1 --2- 172.21.15.96:0/1090222590 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1f5806e390 0x7f1f58070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:21.666 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:21.671+0000 7f1f7ca00640 1 --2- 172.21.15.96:0/1090222590 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1f5806e390 0x7f1f58070850 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f1f7806dba0 tx=0x7f1f6c006db0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:23:22.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:22.036+0000 7f1f7f400640 1 -- 172.21.15.96:0/1090222590 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone36081", "target": ["mon-mgr", ""]}) -- 0x7f1f780b4410 con 0x7f1f5806e390 2024-05-28T06:23:22.689 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:22.693+0000 7f1f74c00640 1 -- 172.21.15.96:0/1090222590 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f1f780b4410 con 0x7f1f5806e390 2024-05-28T06:23:22.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:22.697+0000 7f1f7f400640 1 -- 172.21.15.96:0/1090222590 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1f5806e390 msgr2=0x7f1f58070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:22.693 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:22.697+0000 7f1f7f400640 1 --2- 172.21.15.96:0/1090222590 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1f5806e390 0x7f1f58070850 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f1f7806dba0 tx=0x7f1f6c006db0 comp rx=0 tx=0).stop 2024-05-28T06:23:22.693 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:22.697+0000 7f1f7f400640 1 -- 172.21.15.96:0/1090222590 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1f78159c50 msgr2=0x7f1f780d4740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:22.693 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:22.697+0000 7f1f7f400640 1 --2- 172.21.15.96:0/1090222590 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1f78159c50 0x7f1f780d4740 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7f1f6400d5a0 tx=0x7f1f6400da60 comp rx=0 tx=0).stop 2024-05-28T06:23:22.693 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:22.697+0000 7f1f7f400640 1 -- 172.21.15.96:0/1090222590 shutdown_connections 2024-05-28T06:23:22.693 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:22.697+0000 7f1f7f400640 1 --2- 172.21.15.96:0/1090222590 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1f5806e390 0x7f1f58070850 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:22.693 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:22.697+0000 7f1f7f400640 1 --2- 172.21.15.96:0/1090222590 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1f780d2b30 0x7f1f780d4c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:22.693 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:22.697+0000 7f1f7f400640 1 --2- 172.21.15.96:0/1090222590 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1f78159c50 0x7f1f780d4740 unknown :-1 s=CLOSED pgs=1867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:22.693 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:22.697+0000 7f1f7f400640 1 --2- 172.21.15.96:0/1090222590 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1f78159870 0x7f1f780d4200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:22.693 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:22.697+0000 7f1f7f400640 1 -- 172.21.15.96:0/1090222590 >> 172.21.15.96:0/1090222590 conn(0x7f1f78067aa0 msgr2=0x7f1f780b4f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:23:22.693 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:22.698+0000 7f1f7f400640 1 -- 172.21.15.96:0/1090222590 shutdown_connections 2024-05-28T06:23:22.693 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:22.698+0000 7f1f7f400640 1 -- 172.21.15.96:0/1090222590 wait complete. 2024-05-28T06:23:22.724 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone36082 2024-05-28T06:23:22.987 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:22.992+0000 7f7fe1800640 1 Processor -- start 2024-05-28T06:23:22.987 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:22.992+0000 7f7fe1800640 1 -- start start 2024-05-28T06:23:22.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:22.994+0000 7f7fe1800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7fdc15e870 0x7f7fdc0d0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:22.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:22.994+0000 7f7fe1800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f7fdc0d14d0 0x7f7fdc0cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:22.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:22.994+0000 7f7fe1800640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7fdc0cfd00 con 0x7f7fdc0cf870 2024-05-28T06:23:22.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:22.994+0000 7f7fe1800640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f7fdc0d2150 con 0x7f7fdc0d14d0 2024-05-28T06:23:22.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:22.994+0000 7f7fe1800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f7fdc15ec50 con 0x7f7fdc15e870 2024-05-28T06:23:22.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:22.994+0000 7f7fe0e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f7fdc0cf870 0x7f7fdc162360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:35106/0 (socket says 172.21.15.96:35106) 2024-05-28T06:23:22.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:22.994+0000 7f7fe0e00640 1 -- 172.21.15.96:0/682829081 learned_addr learned my addr 172.21.15.96:0/682829081 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:23:22.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:22.994+0000 7f7fdb400640 1 --2- 172.21.15.96:0/682829081 >> v2:172.21.15.96:3300/0 conn(0x7f7fdc0d14d0 0x7f7fdc0cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:22.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:22.995+0000 7f7fdb400640 1 -- 172.21.15.96:0/682829081 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7fdc15e870 msgr2=0x7f7fdc0d0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:23:22.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:22.995+0000 7f7fdb400640 1 --2- 172.21.15.96:0/682829081 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7fdc15e870 0x7f7fdc0d0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:22.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:22.995+0000 7f7fdb400640 1 -- 172.21.15.96:0/682829081 >> v1:172.21.15.96:6789/0 conn(0x7f7fdc0cf870 legacy=0x7f7fdc162360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:22.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:22.995+0000 7f7fdb400640 1 -- 172.21.15.96:0/682829081 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fdc0cff80 con 0x7f7fdc0d14d0 2024-05-28T06:23:22.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:22.995+0000 7f7fdaa00640 1 -- 172.21.15.96:0/682829081 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2867583667 0 0) 0x7f7fd4001490 con 0x7f7fdc0cf870 2024-05-28T06:23:22.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:22.996+0000 7f7fdb400640 1 --2- 172.21.15.96:0/682829081 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7fdc0d14d0 0x7f7fdc0cf330 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7f7fc8009790 tx=0x7f7fc802a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=351b6611df2770fa server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:23:22.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:22.996+0000 7f7fdaa00640 1 -- 172.21.15.96:0/682829081 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fc8037070 con 0x7f7fdc0d14d0 2024-05-28T06:23:22.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:22.996+0000 7f7fdaa00640 1 -- 172.21.15.96:0/682829081 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7fc8032440 con 0x7f7fdc0d14d0 2024-05-28T06:23:22.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:22.996+0000 7f7fdaa00640 1 -- 172.21.15.96:0/682829081 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fc8032810 con 0x7f7fdc0d14d0 2024-05-28T06:23:22.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:22.997+0000 7f7fe1800640 1 -- 172.21.15.96:0/682829081 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7fdc0d14d0 msgr2=0x7f7fdc0cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:22.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:22.997+0000 7f7fe1800640 1 --2- 172.21.15.96:0/682829081 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7fdc0d14d0 0x7f7fdc0cf330 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7f7fc8009790 tx=0x7f7fc802a180 comp rx=0 tx=0).stop 2024-05-28T06:23:22.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:22.997+0000 7f7fe1800640 1 -- 172.21.15.96:0/682829081 shutdown_connections 2024-05-28T06:23:22.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:22.997+0000 7f7fe1800640 1 --2- 172.21.15.96:0/682829081 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7fdc0d14d0 0x7f7fdc0cf330 unknown :-1 s=CLOSED pgs=1869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:22.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:22.997+0000 7f7fe1800640 1 --2- 172.21.15.96:0/682829081 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7fdc15e870 0x7f7fdc0d0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:22.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:22.997+0000 7f7fe1800640 1 -- 172.21.15.96:0/682829081 >> 172.21.15.96:0/682829081 conn(0x7f7fdc08bfc0 msgr2=0x7f7fdc0d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:23:22.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:22.997+0000 7f7fe1800640 1 -- 172.21.15.96:0/682829081 shutdown_connections 2024-05-28T06:23:22.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:22.997+0000 7f7fe1800640 1 -- 172.21.15.96:0/682829081 wait complete. 2024-05-28T06:23:22.995 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:23.000+0000 7f7fe1800640 1 Processor -- start 2024-05-28T06:23:22.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:23.000+0000 7f7fe1800640 1 -- start start 2024-05-28T06:23:22.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:23.002+0000 7f7fe1800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7fdc0cf870 0x7f7fdc14c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:22.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:23.002+0000 7f7fe1800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7fdc0d14d0 0x7f7fdc1612a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:22.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:23.002+0000 7f7fe1800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7fdc15e870 0x7f7fdc159270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:22.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:23.002+0000 7f7fe1800640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f7fdc0d0820 con 0x7f7fdc0d14d0 2024-05-28T06:23:22.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:23.002+0000 7f7fe1800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f7fdc0d0540 con 0x7f7fdc15e870 2024-05-28T06:23:22.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:23.002+0000 7f7fe1800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f7fdc0d4fb0 con 0x7f7fdc0cf870 2024-05-28T06:23:22.998 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:23.003+0000 7f7fdb400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7fdc0d14d0 0x7f7fdc1612a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:22.998 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:23.003+0000 7f7fdb400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7fdc0d14d0 0x7f7fdc1612a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:57356/0 (socket says 172.21.15.96:57356) 2024-05-28T06:23:22.998 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:23.003+0000 7f7fdb400640 1 -- 172.21.15.96:0/471397381 learned_addr learned my addr 172.21.15.96:0/471397381 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:23:22.998 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:23.003+0000 7f7fe0e00640 1 --2- 172.21.15.96:0/471397381 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7fdc15e870 0x7f7fdc159270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:22.998 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:23.003+0000 7f7fdb400640 1 -- 172.21.15.96:0/471397381 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7fdc0cf870 msgr2=0x7f7fdc14c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:22.998 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:23.003+0000 7f7fdb400640 1 --2- 172.21.15.96:0/471397381 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7fdc0cf870 0x7f7fdc14c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:22.999 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:23.003+0000 7f7fdb400640 1 -- 172.21.15.96:0/471397381 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7fdc15e870 msgr2=0x7f7fdc159270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:22.999 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:23.003+0000 7f7fdb400640 1 --2- 172.21.15.96:0/471397381 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7fdc15e870 0x7f7fdc159270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:22.999 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:23.003+0000 7f7fdb400640 1 -- 172.21.15.96:0/471397381 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fdc0b7450 con 0x7f7fdc0d14d0 2024-05-28T06:23:22.999 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:23.004+0000 7f7fdb400640 1 --2- 172.21.15.96:0/471397381 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7fdc0d14d0 0x7f7fdc1612a0 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7f7fc8009790 tx=0x7f7fc80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:23:23.000 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:23.004+0000 7f7fd3e00640 1 -- 172.21.15.96:0/471397381 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fc8037040 con 0x7f7fdc0d14d0 2024-05-28T06:23:23.000 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:23.004+0000 7f7fd3e00640 1 -- 172.21.15.96:0/471397381 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7fc8004230 con 0x7f7fdc0d14d0 2024-05-28T06:23:23.000 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:23.004+0000 7f7fd3e00640 1 -- 172.21.15.96:0/471397381 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fc8032e40 con 0x7f7fdc0d14d0 2024-05-28T06:23:23.000 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:23.005+0000 7f7fe1800640 1 -- 172.21.15.96:0/471397381 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fdc0b7730 con 0x7f7fdc0d14d0 2024-05-28T06:23:23.000 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:23.005+0000 7f7fe1800640 1 -- 172.21.15.96:0/471397381 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fdc0b7cf0 con 0x7f7fdc0d14d0 2024-05-28T06:23:23.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:23.006+0000 7f7fe1800640 1 -- 172.21.15.96:0/471397381 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7fdc0689e0 con 0x7f7fdc0d14d0 2024-05-28T06:23:23.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:23.007+0000 7f7fd3e00640 1 -- 172.21.15.96:0/471397381 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 41) ==== 93488+0+0 (secure 0 0 0) 0x7f7fc8030770 con 0x7f7fdc0d14d0 2024-05-28T06:23:23.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:23.008+0000 7f7fd3e00640 1 --2- 172.21.15.96:0/471397381 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7fbc06e3f0 0x7f7fbc0708b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:23.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:23.008+0000 7f7fd3e00640 1 -- 172.21.15.96:0/471397381 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 5823+0+0 (secure 0 0 0) 0x7f7fc80abd20 con 0x7f7fdc0d14d0 2024-05-28T06:23:23.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:23.013+0000 7f7fdbe00640 1 --2- 172.21.15.96:0/471397381 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7fbc06e3f0 0x7f7fbc0708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:23.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:23.014+0000 7f7fd3e00640 1 -- 172.21.15.96:0/471397381 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7fc807bcd0 con 0x7f7fdc0d14d0 2024-05-28T06:23:23.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:23.034+0000 7f7fdbe00640 1 --2- 172.21.15.96:0/471397381 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7fbc06e3f0 0x7f7fbc0708b0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f7fdc0b1bd0 tx=0x7f7fcc006c90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:23:23.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:23.429+0000 7f7fe1800640 1 -- 172.21.15.96:0/471397381 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone36082", "target": ["mon-mgr", ""]}) -- 0x7f7fdc06db40 con 0x7f7fbc06e3f0 2024-05-28T06:23:23.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:23.794+0000 7f7fd3e00640 1 -- 172.21.15.96:0/471397381 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f7fdc06db40 con 0x7f7fbc06e3f0 2024-05-28T06:23:23.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:23.797+0000 7f7fe1800640 1 -- 172.21.15.96:0/471397381 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7fbc06e3f0 msgr2=0x7f7fbc0708b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:23.793 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:23.797+0000 7f7fe1800640 1 --2- 172.21.15.96:0/471397381 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7fbc06e3f0 0x7f7fbc0708b0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f7fdc0b1bd0 tx=0x7f7fcc006c90 comp rx=0 tx=0).stop 2024-05-28T06:23:23.793 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:23.797+0000 7f7fe1800640 1 -- 172.21.15.96:0/471397381 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7fdc0d14d0 msgr2=0x7f7fdc1612a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:23.793 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:23.797+0000 7f7fe1800640 1 --2- 172.21.15.96:0/471397381 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7fdc0d14d0 0x7f7fdc1612a0 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7f7fc8009790 tx=0x7f7fc80046c0 comp rx=0 tx=0).stop 2024-05-28T06:23:23.793 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:23.797+0000 7f7fe1800640 1 -- 172.21.15.96:0/471397381 shutdown_connections 2024-05-28T06:23:23.793 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:23.797+0000 7f7fe1800640 1 --2- 172.21.15.96:0/471397381 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7fbc06e3f0 0x7f7fbc0708b0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:23.793 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:23.797+0000 7f7fe1800640 1 --2- 172.21.15.96:0/471397381 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7fdc15e870 0x7f7fdc159270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:23.793 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:23.797+0000 7f7fe1800640 1 --2- 172.21.15.96:0/471397381 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7fdc0d14d0 0x7f7fdc1612a0 unknown :-1 s=CLOSED pgs=1870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:23.793 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:23.797+0000 7f7fe1800640 1 --2- 172.21.15.96:0/471397381 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7fdc0cf870 0x7f7fdc14c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:23.793 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:23.797+0000 7f7fe1800640 1 -- 172.21.15.96:0/471397381 >> 172.21.15.96:0/471397381 conn(0x7f7fdc08bfc0 msgr2=0x7f7fdc0b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:23:23.793 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:23.797+0000 7f7fe1800640 1 -- 172.21.15.96:0/471397381 shutdown_connections 2024-05-28T06:23:23.793 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:23.797+0000 7f7fe1800640 1 -- 172.21.15.96:0/471397381 wait complete. 2024-05-28T06:23:23.816 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone36083 2024-05-28T06:23:24.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.056+0000 7fe193000640 1 Processor -- start 2024-05-28T06:23:24.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.057+0000 7fe193000640 1 -- start start 2024-05-28T06:23:24.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.058+0000 7fe193000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe18c15e850 0x7fe18c0b77b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:24.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.058+0000 7fe193000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fe18c0d1d50 0x7fe18c0b90a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:24.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.058+0000 7fe193000640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe18c0b8000 con 0x7fe18c0cef30 2024-05-28T06:23:24.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.058+0000 7fe193000640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fe18c15ec30 con 0x7fe18c0d1d50 2024-05-28T06:23:24.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.058+0000 7fe193000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fe18c0b82e0 con 0x7fe18c15e850 2024-05-28T06:23:24.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.058+0000 7fe18b400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fe18c0cef30 0x7fe18c0d1640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:35110/0 (socket says 172.21.15.96:35110) 2024-05-28T06:23:24.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.058+0000 7fe18b400640 1 -- 172.21.15.96:0/3307636592 learned_addr learned my addr 172.21.15.96:0/3307636592 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:23:24.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.059+0000 7fe191000640 1 --2- 172.21.15.96:0/3307636592 >> v2:172.21.15.123:3300/0 conn(0x7fe18c0d1d50 0x7fe18c0b90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:24.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.059+0000 7fe18aa00640 1 -- 172.21.15.96:0/3307636592 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1803742798 0 0) 0x7fe18c0b8000 con 0x7fe18c0cef30 2024-05-28T06:23:24.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.059+0000 7fe18be00640 1 --2- 172.21.15.96:0/3307636592 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe18c15e850 0x7fe18c0b77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:24.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.059+0000 7fe18aa00640 1 -- 172.21.15.96:0/3307636592 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe170003520 con 0x7fe18c0cef30 2024-05-28T06:23:24.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.059+0000 7fe18aa00640 1 -- 172.21.15.96:0/3307636592 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3365303418 0 0) 0x7fe170003520 con 0x7fe18c0cef30 2024-05-28T06:23:24.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.059+0000 7fe18aa00640 1 -- 172.21.15.96:0/3307636592 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe18c15e850 msgr2=0x7fe18c0b77b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:23:24.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.059+0000 7fe18aa00640 1 --2- 172.21.15.96:0/3307636592 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe18c15e850 0x7fe18c0b77b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:24.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.059+0000 7fe18aa00640 1 -- 172.21.15.96:0/3307636592 >> v2:172.21.15.123:3300/0 conn(0x7fe18c0d1d50 msgr2=0x7fe18c0b90a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:23:24.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.059+0000 7fe18aa00640 1 --2- 172.21.15.96:0/3307636592 >> v2:172.21.15.123:3300/0 conn(0x7fe18c0d1d50 0x7fe18c0b90a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:24.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.059+0000 7fe18aa00640 1 -- 172.21.15.96:0/3307636592 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe18c0b85c0 con 0x7fe18c0cef30 2024-05-28T06:23:24.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.059+0000 7fe18aa00640 1 -- 172.21.15.96:0/3307636592 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7fe178001670 con 0x7fe18c0cef30 2024-05-28T06:23:24.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.059+0000 7fe18aa00640 1 -- 172.21.15.96:0/3307636592 >> v1:172.21.15.96:6789/0 conn(0x7fe18c0cef30 legacy=0x7fe18c0d1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:24.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.060+0000 7fe18aa00640 1 --2- 172.21.15.96:0/3307636592 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe1700039b0 0x7fe170003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:24.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.060+0000 7fe18aa00640 1 --2- 172.21.15.96:0/3307636592 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe170004800 0x7fe170004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:24.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.060+0000 7fe18aa00640 1 --2- 172.21.15.96:0/3307636592 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe18c15e850 0x7fe1700051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:24.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.060+0000 7fe18aa00640 1 -- 172.21.15.96:0/3307636592 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe18c0b82e0 con 0x7fe170004800 2024-05-28T06:23:24.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.060+0000 7fe18aa00640 1 -- 172.21.15.96:0/3307636592 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe17000bac0 con 0x7fe1700039b0 2024-05-28T06:23:24.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.060+0000 7fe18aa00640 1 -- 172.21.15.96:0/3307636592 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fe17000bcc0 con 0x7fe18c15e850 2024-05-28T06:23:24.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.060+0000 7fe18b400640 1 --2- 172.21.15.96:0/3307636592 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe170004800 0x7fe170004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:24.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.060+0000 7fe18be00640 1 --2- 172.21.15.96:0/3307636592 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe1700039b0 0x7fe170003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:24.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.061+0000 7fe191000640 1 --2- 172.21.15.96:0/3307636592 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe18c15e850 0x7fe1700051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:24.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.061+0000 7fe18b400640 1 -- 172.21.15.96:0/3307636592 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe18c15e850 msgr2=0x7fe1700051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:23:24.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.061+0000 7fe18b400640 1 --2- 172.21.15.96:0/3307636592 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe18c15e850 0x7fe1700051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:24.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.061+0000 7fe18b400640 1 -- 172.21.15.96:0/3307636592 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe1700039b0 msgr2=0x7fe170003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:23:24.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.061+0000 7fe18b400640 1 --2- 172.21.15.96:0/3307636592 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe1700039b0 0x7fe170003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:24.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.061+0000 7fe18b400640 1 -- 172.21.15.96:0/3307636592 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe170003520 con 0x7fe170004800 2024-05-28T06:23:24.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.061+0000 7fe18be00640 1 --2- 172.21.15.96:0/3307636592 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe1700039b0 0x7fe170003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-28T06:23:24.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.062+0000 7fe18b400640 1 --2- 172.21.15.96:0/3307636592 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe170004800 0x7fe170004c80 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7fe17800c240 tx=0x7fe170005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=744e3dcf28408a61 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:23:24.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.063+0000 7fe18aa00640 1 -- 172.21.15.96:0/3307636592 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe17802e070 con 0x7fe170004800 2024-05-28T06:23:24.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.063+0000 7fe18aa00640 1 -- 172.21.15.96:0/3307636592 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe178005af0 con 0x7fe170004800 2024-05-28T06:23:24.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.063+0000 7fe18aa00640 1 -- 172.21.15.96:0/3307636592 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe178007d30 con 0x7fe170004800 2024-05-28T06:23:24.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.064+0000 7fe193000640 1 -- 172.21.15.96:0/3307636592 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe170004800 msgr2=0x7fe170004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:24.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.064+0000 7fe193000640 1 --2- 172.21.15.96:0/3307636592 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe170004800 0x7fe170004c80 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7fe17800c240 tx=0x7fe170005830 comp rx=0 tx=0).stop 2024-05-28T06:23:24.061 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.064+0000 7fe191000640 1 -- 172.21.15.96:0/3307636592 reap_dead start 2024-05-28T06:23:24.061 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.064+0000 7fe193000640 1 -- 172.21.15.96:0/3307636592 shutdown_connections 2024-05-28T06:23:24.061 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.064+0000 7fe193000640 1 -- 172.21.15.96:0/3307636592 >> 172.21.15.96:0/3307636592 conn(0x7fe18c067aa0 msgr2=0x7fe18c067eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:23:24.061 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.064+0000 7fe193000640 1 -- 172.21.15.96:0/3307636592 shutdown_connections 2024-05-28T06:23:24.061 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.064+0000 7fe193000640 1 -- 172.21.15.96:0/3307636592 wait complete. 2024-05-28T06:23:24.061 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.066+0000 7fe193000640 1 Processor -- start 2024-05-28T06:23:24.061 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.066+0000 7fe193000640 1 -- start start 2024-05-28T06:23:24.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.068+0000 7fe193000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe170004800 0x7fe18c1f72a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:24.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.068+0000 7fe193000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe18c0cd2e0 0x7fe18c0cd760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:24.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.068+0000 7fe193000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe18c0ca670 0x7fe18c0caaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:24.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.068+0000 7fe193000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe18c0b7930 con 0x7fe18c0ca670 2024-05-28T06:23:24.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.068+0000 7fe193000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe18c0b77b0 con 0x7fe18c0cd2e0 2024-05-28T06:23:24.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.068+0000 7fe193000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fe18c0b7ab0 con 0x7fe170004800 2024-05-28T06:23:24.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.069+0000 7fe191000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe18c0ca670 0x7fe18c0caaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:24.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.069+0000 7fe191000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe18c0ca670 0x7fe18c0caaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:57382/0 (socket says 172.21.15.96:57382) 2024-05-28T06:23:24.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.069+0000 7fe191000640 1 -- 172.21.15.96:0/2772468011 learned_addr learned my addr 172.21.15.96:0/2772468011 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:23:24.065 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.069+0000 7fe18b400640 1 --2- 172.21.15.96:0/2772468011 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe18c0cd2e0 0x7fe18c0cd760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:24.065 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.069+0000 7fe191000640 1 -- 172.21.15.96:0/2772468011 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe170004800 msgr2=0x7fe18c1f72a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:24.065 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.069+0000 7fe191000640 1 --2- 172.21.15.96:0/2772468011 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe170004800 0x7fe18c1f72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:24.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.069+0000 7fe191000640 1 -- 172.21.15.96:0/2772468011 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe18c0cd2e0 msgr2=0x7fe18c0cd760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:24.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.069+0000 7fe191000640 1 --2- 172.21.15.96:0/2772468011 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe18c0cd2e0 0x7fe18c0cd760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:24.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.069+0000 7fe191000640 1 -- 172.21.15.96:0/2772468011 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe18c1f77e0 con 0x7fe18c0ca670 2024-05-28T06:23:24.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.070+0000 7fe191000640 1 --2- 172.21.15.96:0/2772468011 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe18c0ca670 0x7fe18c0caaf0 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7fe18c0cb750 tx=0x7fe1840115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:23:24.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.070+0000 7fe183e00640 1 -- 172.21.15.96:0/2772468011 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe17c00a570 con 0x7fe18c0ca670 2024-05-28T06:23:24.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.070+0000 7fe183e00640 1 -- 172.21.15.96:0/2772468011 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe18401b070 con 0x7fe18c0ca670 2024-05-28T06:23:24.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.070+0000 7fe183e00640 1 -- 172.21.15.96:0/2772468011 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe184011ea0 con 0x7fe18c0ca670 2024-05-28T06:23:24.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.070+0000 7fe193000640 1 -- 172.21.15.96:0/2772468011 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe18c0cdda0 con 0x7fe18c0ca670 2024-05-28T06:23:24.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.070+0000 7fe193000640 1 -- 172.21.15.96:0/2772468011 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe18c0ce360 con 0x7fe18c0ca670 2024-05-28T06:23:24.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.072+0000 7fe193000640 1 -- 172.21.15.96:0/2772468011 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe18c0689e0 con 0x7fe18c0ca670 2024-05-28T06:23:24.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.073+0000 7fe183e00640 1 -- 172.21.15.96:0/2772468011 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 41) ==== 93488+0+0 (secure 0 0 0) 0x7fe184005d10 con 0x7fe18c0ca670 2024-05-28T06:23:24.071 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.076+0000 7fe183e00640 1 --2- 172.21.15.96:0/2772468011 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe16c06e3f0 0x7fe16c0708b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:24.071 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.076+0000 7fe183e00640 1 -- 172.21.15.96:0/2772468011 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 5823+0+0 (secure 0 0 0) 0x7fe18408acb0 con 0x7fe18c0ca670 2024-05-28T06:23:24.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.081+0000 7fe183e00640 1 -- 172.21.15.96:0/2772468011 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe18c1f77e0 con 0x7fe18c0ca670 2024-05-28T06:23:24.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.085+0000 7fe18be00640 1 --2- 172.21.15.96:0/2772468011 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe16c06e3f0 0x7fe16c0708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:24.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.094+0000 7fe18be00640 1 --2- 172.21.15.96:0/2772468011 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe16c06e3f0 0x7fe16c0708b0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fe17c00ab80 tx=0x7fe17c00a470 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:23:24.436 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.441+0000 7fe193000640 1 -- 172.21.15.96:0/2772468011 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone36083", "target": ["mon-mgr", ""]}) -- 0x7fe18c15f720 con 0x7fe16c06e3f0 2024-05-28T06:23:24.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.613+0000 7fe183e00640 1 -- 172.21.15.96:0/2772468011 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fe18c15f720 con 0x7fe16c06e3f0 2024-05-28T06:23:24.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.615+0000 7fe193000640 1 -- 172.21.15.96:0/2772468011 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe16c06e3f0 msgr2=0x7fe16c0708b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:24.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.615+0000 7fe193000640 1 --2- 172.21.15.96:0/2772468011 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe16c06e3f0 0x7fe16c0708b0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fe17c00ab80 tx=0x7fe17c00a470 comp rx=0 tx=0).stop 2024-05-28T06:23:24.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.615+0000 7fe193000640 1 -- 172.21.15.96:0/2772468011 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe18c0ca670 msgr2=0x7fe18c0caaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:24.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.615+0000 7fe193000640 1 --2- 172.21.15.96:0/2772468011 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe18c0ca670 0x7fe18c0caaf0 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7fe18c0cb750 tx=0x7fe1840115a0 comp rx=0 tx=0).stop 2024-05-28T06:23:24.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.615+0000 7fe193000640 1 -- 172.21.15.96:0/2772468011 shutdown_connections 2024-05-28T06:23:24.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.615+0000 7fe193000640 1 --2- 172.21.15.96:0/2772468011 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe16c06e3f0 0x7fe16c0708b0 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:24.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.615+0000 7fe193000640 1 --2- 172.21.15.96:0/2772468011 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe18c0ca670 0x7fe18c0caaf0 unknown :-1 s=CLOSED pgs=1873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:24.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.615+0000 7fe193000640 1 --2- 172.21.15.96:0/2772468011 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe18c0cd2e0 0x7fe18c0cd760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:24.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.615+0000 7fe193000640 1 --2- 172.21.15.96:0/2772468011 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe170004800 0x7fe18c1f72a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:24.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.615+0000 7fe193000640 1 -- 172.21.15.96:0/2772468011 >> 172.21.15.96:0/2772468011 conn(0x7fe18c067aa0 msgr2=0x7fe18c067eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:23:24.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.615+0000 7fe193000640 1 -- 172.21.15.96:0/2772468011 shutdown_connections 2024-05-28T06:23:24.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:24.615+0000 7fe193000640 1 -- 172.21.15.96:0/2772468011 wait complete. 2024-05-28T06:23:29.633 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-05-28T06:23:29.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:29.846+0000 7fbc13600640 1 Processor -- start 2024-05-28T06:23:29.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:29.846+0000 7fbc13600640 1 -- start start 2024-05-28T06:23:29.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:29.848+0000 7fbc13600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbc0c145320 0x7fbc0c071cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:29.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:29.848+0000 7fbc13600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fbc0c0b43b0 0x7fbc0c0cf880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:29.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:29.848+0000 7fbc13600640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc0c0cf0c0 con 0x7fbc0c072200 2024-05-28T06:23:29.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:29.848+0000 7fbc13600640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fbc0c072660 con 0x7fbc0c0b43b0 2024-05-28T06:23:29.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:29.848+0000 7fbc13600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fbc0c0b4a20 con 0x7fbc0c145320 2024-05-28T06:23:29.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:29.848+0000 7fbc0be00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fbc0c072200 0x7fbc0c0b3ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:33884/0 (socket says 172.21.15.96:33884) 2024-05-28T06:23:29.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:29.849+0000 7fbc0be00640 1 -- 172.21.15.96:0/3168262974 learned_addr learned my addr 172.21.15.96:0/3168262974 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:23:29.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:29.849+0000 7fbc10c00640 1 --2- 172.21.15.96:0/3168262974 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbc0c145320 0x7fbc0c071cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:29.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:29.849+0000 7fbc11600640 1 --2- 172.21.15.96:0/3168262974 >> v2:172.21.15.123:3300/0 conn(0x7fbc0c0b43b0 0x7fbc0c0cf880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:29.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:29.849+0000 7fbc0b400640 1 -- 172.21.15.96:0/3168262974 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2903998733 0 0) 0x7fbc0c0cf0c0 con 0x7fbc0c072200 2024-05-28T06:23:29.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:29.849+0000 7fbc0b400640 1 -- 172.21.15.96:0/3168262974 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbbf0003520 con 0x7fbc0c072200 2024-05-28T06:23:29.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:29.849+0000 7fbc0b400640 1 -- 172.21.15.96:0/3168262974 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2505563533 0 0) 0x7fbbf0003520 con 0x7fbc0c072200 2024-05-28T06:23:29.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:29.849+0000 7fbc0b400640 1 -- 172.21.15.96:0/3168262974 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbc0c145320 msgr2=0x7fbc0c071cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:23:29.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:29.849+0000 7fbc0b400640 1 --2- 172.21.15.96:0/3168262974 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbc0c145320 0x7fbc0c071cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:29.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:29.849+0000 7fbc0b400640 1 -- 172.21.15.96:0/3168262974 >> v2:172.21.15.123:3300/0 conn(0x7fbc0c0b43b0 msgr2=0x7fbc0c0cf880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:23:29.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:29.849+0000 7fbc0b400640 1 --2- 172.21.15.96:0/3168262974 >> v2:172.21.15.123:3300/0 conn(0x7fbc0c0b43b0 0x7fbc0c0cf880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:29.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:29.849+0000 7fbc0b400640 1 -- 172.21.15.96:0/3168262974 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc0c0b4d00 con 0x7fbc0c072200 2024-05-28T06:23:29.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:29.849+0000 7fbc0b400640 1 -- 172.21.15.96:0/3168262974 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7fbbf4001670 con 0x7fbc0c072200 2024-05-28T06:23:29.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:29.850+0000 7fbc0b400640 1 -- 172.21.15.96:0/3168262974 >> v1:172.21.15.96:6789/0 conn(0x7fbc0c072200 legacy=0x7fbc0c0b3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:29.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:29.850+0000 7fbc0b400640 1 --2- 172.21.15.96:0/3168262974 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbbf00039b0 0x7fbbf0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:29.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:29.850+0000 7fbc0b400640 1 --2- 172.21.15.96:0/3168262974 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbc0c145320 0x7fbbf0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:29.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:29.850+0000 7fbc0b400640 1 --2- 172.21.15.96:0/3168262974 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbbf0004d40 0x7fbbf00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:29.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:29.850+0000 7fbc0b400640 1 -- 172.21.15.96:0/3168262974 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fbc0c0b4a20 con 0x7fbbf0004d40 2024-05-28T06:23:29.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:29.850+0000 7fbc0b400640 1 -- 172.21.15.96:0/3168262974 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fbbf000b9c0 con 0x7fbbf00039b0 2024-05-28T06:23:29.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:29.850+0000 7fbc0b400640 1 -- 172.21.15.96:0/3168262974 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fbbf000bb40 con 0x7fbc0c145320 2024-05-28T06:23:29.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:29.850+0000 7fbc11600640 1 --2- 172.21.15.96:0/3168262974 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbbf0004d40 0x7fbbf00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:29.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:29.850+0000 7fbc0be00640 1 --2- 172.21.15.96:0/3168262974 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbc0c145320 0x7fbbf0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:29.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:29.850+0000 7fbc11600640 1 -- 172.21.15.96:0/3168262974 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbc0c145320 msgr2=0x7fbbf0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:23:29.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:29.850+0000 7fbc11600640 1 --2- 172.21.15.96:0/3168262974 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbc0c145320 0x7fbbf0004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:29.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:29.850+0000 7fbc11600640 1 -- 172.21.15.96:0/3168262974 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbbf00039b0 msgr2=0x7fbbf0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:23:29.847 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:29.850+0000 7fbc11600640 1 --2- 172.21.15.96:0/3168262974 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbbf00039b0 0x7fbbf0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:29.847 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:29.850+0000 7fbc11600640 1 -- 172.21.15.96:0/3168262974 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbf0003520 con 0x7fbbf0004d40 2024-05-28T06:23:29.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:29.851+0000 7fbc11600640 1 --2- 172.21.15.96:0/3168262974 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbbf0004d40 0x7fbbf00051c0 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7fbc04009ac0 tx=0x7fbbf0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45b2d3f6a829754 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:23:29.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:29.852+0000 7fbc0b400640 1 -- 172.21.15.96:0/3168262974 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc04036070 con 0x7fbbf0004d40 2024-05-28T06:23:29.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:29.852+0000 7fbc0b400640 1 -- 172.21.15.96:0/3168262974 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc04031400 con 0x7fbbf0004d40 2024-05-28T06:23:29.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:29.852+0000 7fbc0b400640 1 -- 172.21.15.96:0/3168262974 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc04009690 con 0x7fbbf0004d40 2024-05-28T06:23:29.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:29.852+0000 7fbc13600640 1 -- 172.21.15.96:0/3168262974 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbbf0004d40 msgr2=0x7fbbf00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:29.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:29.852+0000 7fbc13600640 1 --2- 172.21.15.96:0/3168262974 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbbf0004d40 0x7fbbf00051c0 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7fbc04009ac0 tx=0x7fbbf0005830 comp rx=0 tx=0).stop 2024-05-28T06:23:29.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:29.852+0000 7fbc11600640 1 -- 172.21.15.96:0/3168262974 reap_dead start 2024-05-28T06:23:29.849 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:29.852+0000 7fbc13600640 1 -- 172.21.15.96:0/3168262974 shutdown_connections 2024-05-28T06:23:29.849 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:29.852+0000 7fbc13600640 1 -- 172.21.15.96:0/3168262974 >> 172.21.15.96:0/3168262974 conn(0x7fbc0c067ce0 msgr2=0x7fbc0c06b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:23:29.849 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:29.852+0000 7fbc13600640 1 -- 172.21.15.96:0/3168262974 shutdown_connections 2024-05-28T06:23:29.849 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:29.852+0000 7fbc13600640 1 -- 172.21.15.96:0/3168262974 wait complete. 2024-05-28T06:23:29.849 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:29.854+0000 7fbc13600640 1 Processor -- start 2024-05-28T06:23:29.849 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:29.854+0000 7fbc13600640 1 -- start start 2024-05-28T06:23:29.851 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:29.855+0000 7fbc13600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbbf0004d40 0x7fbc0c13fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:29.851 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:29.855+0000 7fbc13600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc0c1f4f90 0x7fbc0c06e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:29.851 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:29.855+0000 7fbc13600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc0c0c5e80 0x7fbc0c0c6310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:29.851 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:29.855+0000 7fbc13600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fbc0c145700 con 0x7fbc0c1f4f90 2024-05-28T06:23:29.851 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:29.855+0000 7fbc13600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fbc0c0cf360 con 0x7fbc0c0c5e80 2024-05-28T06:23:29.851 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:29.855+0000 7fbc13600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fbc0c071cc0 con 0x7fbbf0004d40 2024-05-28T06:23:29.851 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:29.856+0000 7fbc0be00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc0c1f4f90 0x7fbc0c06e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:29.851 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:29.856+0000 7fbc0be00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc0c1f4f90 0x7fbc0c06e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:50620/0 (socket says 172.21.15.96:50620) 2024-05-28T06:23:29.852 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:29.856+0000 7fbc0be00640 1 -- 172.21.15.96:0/1624901098 learned_addr learned my addr 172.21.15.96:0/1624901098 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:23:29.852 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:29.856+0000 7fbc10c00640 1 --2- 172.21.15.96:0/1624901098 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbbf0004d40 0x7fbc0c13fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:29.852 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:29.856+0000 7fbc0be00640 1 -- 172.21.15.96:0/1624901098 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbbf0004d40 msgr2=0x7fbc0c13fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:29.852 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:29.856+0000 7fbc0be00640 1 --2- 172.21.15.96:0/1624901098 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbbf0004d40 0x7fbc0c13fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:29.852 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:29.856+0000 7fbc0be00640 1 -- 172.21.15.96:0/1624901098 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc0c0c5e80 msgr2=0x7fbc0c0c6310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:29.852 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:29.856+0000 7fbc0be00640 1 --2- 172.21.15.96:0/1624901098 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc0c0c5e80 0x7fbc0c0c6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:29.852 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:29.856+0000 7fbc0be00640 1 -- 172.21.15.96:0/1624901098 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc0c06eda0 con 0x7fbc0c1f4f90 2024-05-28T06:23:29.852 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:29.856+0000 7fbc10c00640 1 --2- 172.21.15.96:0/1624901098 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbbf0004d40 0x7fbc0c13fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:23:29.852 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:29.856+0000 7fbc0be00640 1 --2- 172.21.15.96:0/1624901098 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc0c1f4f90 0x7fbc0c06e420 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7fbbf400d910 tx=0x7fbbf400ddd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:23:29.852 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:29.856+0000 7fbc08c00640 1 -- 172.21.15.96:0/1624901098 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbf4019070 con 0x7fbc0c1f4f90 2024-05-28T06:23:29.852 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:29.857+0000 7fbc08c00640 1 -- 172.21.15.96:0/1624901098 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbf40074e0 con 0x7fbc0c1f4f90 2024-05-28T06:23:29.852 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:29.857+0000 7fbc13600640 1 -- 172.21.15.96:0/1624901098 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc0c06f080 con 0x7fbc0c1f4f90 2024-05-28T06:23:29.852 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:29.857+0000 7fbc08c00640 1 -- 172.21.15.96:0/1624901098 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbf4004a00 con 0x7fbc0c1f4f90 2024-05-28T06:23:29.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:29.857+0000 7fbc13600640 1 -- 172.21.15.96:0/1624901098 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc0c06f640 con 0x7fbc0c1f4f90 2024-05-28T06:23:29.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:29.858+0000 7fbc13600640 1 -- 172.21.15.96:0/1624901098 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc0c0689e0 con 0x7fbc0c1f4f90 2024-05-28T06:23:29.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:29.859+0000 7fbc08c00640 1 -- 172.21.15.96:0/1624901098 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 41) ==== 93488+0+0 (secure 0 0 0) 0x7fbbf401c070 con 0x7fbc0c1f4f90 2024-05-28T06:23:29.854 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:29.860+0000 7fbc08c00640 1 --2- 172.21.15.96:0/1624901098 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbbe806e420 0x7fbbe80708e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:29.854 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:29.860+0000 7fbc08c00640 1 -- 172.21.15.96:0/1624901098 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 5823+0+0 (secure 0 0 0) 0x7fbbf408a530 con 0x7fbc0c1f4f90 2024-05-28T06:23:29.854 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:29.861+0000 7fbc10c00640 1 --2- 172.21.15.96:0/1624901098 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbbe806e420 0x7fbbe80708e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:29.855 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:29.862+0000 7fbc10c00640 1 --2- 172.21.15.96:0/1624901098 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbbe806e420 0x7fbbe80708e0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc004b30 tx=0x7fbbfc00b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:23:29.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:29.865+0000 7fbc08c00640 1 -- 172.21.15.96:0/1624901098 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbbf405a3c0 con 0x7fbc0c1f4f90 2024-05-28T06:23:30.136 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.142+0000 7fbc13600640 1 -- 172.21.15.96:0/1624901098 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fbc0c0b34e0 con 0x7fbc0c1f4f90 2024-05-28T06:23:30.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.143+0000 7fbc08c00640 1 -- 172.21.15.96:0/1624901098 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+2125 (secure 0 0 0) 0x7fbbf4059cc0 con 0x7fbc0c1f4f90 2024-05-28T06:23:30.137 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:23:30.138 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:23:30.138 INFO:teuthology.orchestra.run.smithi096.stdout: "fsid": "e4a831b5-e6d1-47a7-96cb-bf85154ee0d2", 2024-05-28T06:23:30.138 INFO:teuthology.orchestra.run.smithi096.stdout: "health": { 2024-05-28T06:23:30.138 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "HEALTH_OK", 2024-05-28T06:23:30.138 INFO:teuthology.orchestra.run.smithi096.stdout: "checks": {}, 2024-05-28T06:23:30.138 INFO:teuthology.orchestra.run.smithi096.stdout: "mutes": [] 2024-05-28T06:23:30.138 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:23:30.138 INFO:teuthology.orchestra.run.smithi096.stdout: "election_epoch": 8, 2024-05-28T06:23:30.138 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum": [ 2024-05-28T06:23:30.138 INFO:teuthology.orchestra.run.smithi096.stdout: 0, 2024-05-28T06:23:30.139 INFO:teuthology.orchestra.run.smithi096.stdout: 1, 2024-05-28T06:23:30.139 INFO:teuthology.orchestra.run.smithi096.stdout: 2 2024-05-28T06:23:30.139 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:23:30.139 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_names": [ 2024-05-28T06:23:30.139 INFO:teuthology.orchestra.run.smithi096.stdout: "a", 2024-05-28T06:23:30.139 INFO:teuthology.orchestra.run.smithi096.stdout: "b", 2024-05-28T06:23:30.139 INFO:teuthology.orchestra.run.smithi096.stdout: "c" 2024-05-28T06:23:30.139 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:23:30.140 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_age": 1183, 2024-05-28T06:23:30.140 INFO:teuthology.orchestra.run.smithi096.stdout: "monmap": { 2024-05-28T06:23:30.140 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 1, 2024-05-28T06:23:30.140 INFO:teuthology.orchestra.run.smithi096.stdout: "min_mon_release_name": "squid", 2024-05-28T06:23:30.140 INFO:teuthology.orchestra.run.smithi096.stdout: "num_mons": 3 2024-05-28T06:23:30.140 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:23:30.140 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap": { 2024-05-28T06:23:30.140 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 117, 2024-05-28T06:23:30.140 INFO:teuthology.orchestra.run.smithi096.stdout: "num_osds": 8, 2024-05-28T06:23:30.140 INFO:teuthology.orchestra.run.smithi096.stdout: "num_up_osds": 8, 2024-05-28T06:23:30.140 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_up_since": 1716876238, 2024-05-28T06:23:30.140 INFO:teuthology.orchestra.run.smithi096.stdout: "num_in_osds": 8, 2024-05-28T06:23:30.140 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_in_since": 1716876229, 2024-05-28T06:23:30.141 INFO:teuthology.orchestra.run.smithi096.stdout: "num_remapped_pgs": 0 2024-05-28T06:23:30.141 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:23:30.141 INFO:teuthology.orchestra.run.smithi096.stdout: "pgmap": { 2024-05-28T06:23:30.141 INFO:teuthology.orchestra.run.smithi096.stdout: "pgs_by_state": [ 2024-05-28T06:23:30.141 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:23:30.141 INFO:teuthology.orchestra.run.smithi096.stdout: "state_name": "active+clean", 2024-05-28T06:23:30.141 INFO:teuthology.orchestra.run.smithi096.stdout: "count": 97 2024-05-28T06:23:30.141 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:23:30.141 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:23:30.141 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pgs": 97, 2024-05-28T06:23:30.141 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pools": 3, 2024-05-28T06:23:30.141 INFO:teuthology.orchestra.run.smithi096.stdout: "num_objects": 1748, 2024-05-28T06:23:30.141 INFO:teuthology.orchestra.run.smithi096.stdout: "data_bytes": 7190339917, 2024-05-28T06:23:30.142 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_used": 15374557184, 2024-05-28T06:23:30.142 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_avail": 757719556096, 2024-05-28T06:23:30.142 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_total": 773094113280, 2024-05-28T06:23:30.142 INFO:teuthology.orchestra.run.smithi096.stdout: "read_bytes_sec": 336762769, 2024-05-28T06:23:30.142 INFO:teuthology.orchestra.run.smithi096.stdout: "write_bytes_sec": 330520778, 2024-05-28T06:23:30.142 INFO:teuthology.orchestra.run.smithi096.stdout: "read_op_per_sec": 83, 2024-05-28T06:23:30.142 INFO:teuthology.orchestra.run.smithi096.stdout: "write_op_per_sec": 94 2024-05-28T06:23:30.142 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:23:30.142 INFO:teuthology.orchestra.run.smithi096.stdout: "fsmap": { 2024-05-28T06:23:30.142 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 73, 2024-05-28T06:23:30.142 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 8, 2024-05-28T06:23:30.142 INFO:teuthology.orchestra.run.smithi096.stdout: "up": 1, 2024-05-28T06:23:30.142 INFO:teuthology.orchestra.run.smithi096.stdout: "in": 1, 2024-05-28T06:23:30.143 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 1, 2024-05-28T06:23:30.143 INFO:teuthology.orchestra.run.smithi096.stdout: "by_rank": [ 2024-05-28T06:23:30.143 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:23:30.143 INFO:teuthology.orchestra.run.smithi096.stdout: "filesystem_id": 8, 2024-05-28T06:23:30.143 INFO:teuthology.orchestra.run.smithi096.stdout: "rank": 0, 2024-05-28T06:23:30.143 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "b", 2024-05-28T06:23:30.143 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "up:active", 2024-05-28T06:23:30.143 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 9270 2024-05-28T06:23:30.143 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:23:30.143 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:23:30.143 INFO:teuthology.orchestra.run.smithi096.stdout: "up:standby": 3 2024-05-28T06:23:30.143 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:23:30.143 INFO:teuthology.orchestra.run.smithi096.stdout: "mgrmap": { 2024-05-28T06:23:30.143 INFO:teuthology.orchestra.run.smithi096.stdout: "available": true, 2024-05-28T06:23:30.144 INFO:teuthology.orchestra.run.smithi096.stdout: "num_standbys": 1, 2024-05-28T06:23:30.144 INFO:teuthology.orchestra.run.smithi096.stdout: "modules": [ 2024-05-28T06:23:30.144 INFO:teuthology.orchestra.run.smithi096.stdout: "iostat", 2024-05-28T06:23:30.144 INFO:teuthology.orchestra.run.smithi096.stdout: "nfs", 2024-05-28T06:23:30.144 INFO:teuthology.orchestra.run.smithi096.stdout: "restful" 2024-05-28T06:23:30.144 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:23:30.144 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:23:30.144 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:23:30.144 INFO:teuthology.orchestra.run.smithi096.stdout: "servicemap": { 2024-05-28T06:23:30.144 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 23, 2024-05-28T06:23:30.144 INFO:teuthology.orchestra.run.smithi096.stdout: "modified": "2024-05-28T06:23:23.993668+0000", 2024-05-28T06:23:30.144 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:23:30.145 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:23:30.145 INFO:teuthology.orchestra.run.smithi096.stdout: "progress_events": { 2024-05-28T06:23:30.145 INFO:teuthology.orchestra.run.smithi096.stdout: "d6fb006f-47c2-4c80-b874-17506401c2f6": { 2024-05-28T06:23:30.145 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "All cloning finished, waiting 26.0 seconds before exiting (10s)\n [============================] ", 2024-05-28T06:23:30.145 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": 1, 2024-05-28T06:23:30.145 INFO:teuthology.orchestra.run.smithi096.stdout: "add_to_ceph_s": true 2024-05-28T06:23:30.145 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:23:30.145 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:23:30.145 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:23:30.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.146+0000 7fbc13600640 1 -- 172.21.15.96:0/1624901098 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbbe806e420 msgr2=0x7fbbe80708e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:30.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.146+0000 7fbc13600640 1 --2- 172.21.15.96:0/1624901098 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbbe806e420 0x7fbbe80708e0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc004b30 tx=0x7fbbfc00b040 comp rx=0 tx=0).stop 2024-05-28T06:23:30.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.146+0000 7fbc13600640 1 -- 172.21.15.96:0/1624901098 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc0c1f4f90 msgr2=0x7fbc0c06e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:30.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.146+0000 7fbc13600640 1 --2- 172.21.15.96:0/1624901098 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc0c1f4f90 0x7fbc0c06e420 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7fbbf400d910 tx=0x7fbbf400ddd0 comp rx=0 tx=0).stop 2024-05-28T06:23:30.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.146+0000 7fbc13600640 1 -- 172.21.15.96:0/1624901098 shutdown_connections 2024-05-28T06:23:30.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.146+0000 7fbc13600640 1 --2- 172.21.15.96:0/1624901098 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbbe806e420 0x7fbbe80708e0 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:30.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.146+0000 7fbc13600640 1 --2- 172.21.15.96:0/1624901098 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc0c0c5e80 0x7fbc0c0c6310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:30.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.146+0000 7fbc13600640 1 --2- 172.21.15.96:0/1624901098 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc0c1f4f90 0x7fbc0c06e420 unknown :-1 s=CLOSED pgs=1876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:30.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.146+0000 7fbc13600640 1 --2- 172.21.15.96:0/1624901098 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbbf0004d40 0x7fbc0c13fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:30.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.146+0000 7fbc13600640 1 -- 172.21.15.96:0/1624901098 >> 172.21.15.96:0/1624901098 conn(0x7fbc0c067ce0 msgr2=0x7fbc0c15d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:23:30.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.146+0000 7fbc13600640 1 -- 172.21.15.96:0/1624901098 shutdown_connections 2024-05-28T06:23:30.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.146+0000 7fbc13600640 1 -- 172.21.15.96:0/1624901098 wait complete. 2024-05-28T06:23:30.158 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:23:30.159 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls -lh volumes/_nogroup/subvol_clone36081/f0a99b9f-1e23-4f2b-ae5c-c3aa6b024e70 2024-05-28T06:23:30.159 DEBUG:teuthology.orchestra.run.smithi096:> ) 2024-05-28T06:23:30.218 INFO:teuthology.orchestra.run.smithi096.stderr:ls: cannot access 'volumes/_nogroup/subvol_clone36081/f0a99b9f-1e23-4f2b-ae5c-c3aa6b024e70': No such file or directory 2024-05-28T06:23:30.218 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-05-28T06:23:30.219 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-05-28T06:23:30.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.428+0000 7f8cc1a00640 1 Processor -- start 2024-05-28T06:23:30.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.428+0000 7f8cc1a00640 1 -- start start 2024-05-28T06:23:30.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.430+0000 7f8cc1a00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f8cbc15b3a0 0x7f8cbc0b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:30.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.430+0000 7f8cc1a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f8cbc06dc20 0x7f8cbc15f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:30.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.430+0000 7f8cc1a00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8cbc0afd60 con 0x7f8cbc072c00 2024-05-28T06:23:30.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.430+0000 7f8cc1a00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f8cbc06f980 con 0x7f8cbc06dc20 2024-05-28T06:23:30.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.430+0000 7f8cc1a00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f8cbc0ae0c0 con 0x7f8cbc15b3a0 2024-05-28T06:23:30.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.430+0000 7f8cb3400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f8cbc06dc20 0x7f8cbc15f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:30.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.430+0000 7f8cb3400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f8cbc06dc20 0x7f8cbc15f9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:50622/0 (socket says 172.21.15.96:50622) 2024-05-28T06:23:30.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.430+0000 7f8cb3400640 1 -- 172.21.15.96:0/2700611118 learned_addr learned my addr 172.21.15.96:0/2700611118 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:23:30.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.430+0000 7f8cc1000640 1 --1- 172.21.15.96:0/2700611118 >> v1:172.21.15.96:6789/0 conn(0x7f8cbc072c00 0x7f8cbc073010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:33896/0 (socket says 172.21.15.96:33896) 2024-05-28T06:23:30.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.431+0000 7f8cbbe00640 1 --2- 172.21.15.96:0/2700611118 >> v2:172.21.15.123:3300/0 conn(0x7f8cbc15b3a0 0x7f8cbc0b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:30.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.431+0000 7f8cb3400640 1 -- 172.21.15.96:0/2700611118 >> v2:172.21.15.123:3300/0 conn(0x7f8cbc15b3a0 msgr2=0x7f8cbc0b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:23:30.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.431+0000 7f8cb3400640 1 --2- 172.21.15.96:0/2700611118 >> v2:172.21.15.123:3300/0 conn(0x7f8cbc15b3a0 0x7f8cbc0b32c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:30.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.431+0000 7f8cb3400640 1 -- 172.21.15.96:0/2700611118 >> v1:172.21.15.96:6789/0 conn(0x7f8cbc072c00 legacy=0x7f8cbc073010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:30.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.431+0000 7f8cb3400640 1 -- 172.21.15.96:0/2700611118 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cbc0af7a0 con 0x7f8cbc06dc20 2024-05-28T06:23:30.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.431+0000 7f8cbb400640 1 -- 172.21.15.96:0/2700611118 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3056611394 0 0) 0x7f8cbc0afd60 con 0x7f8cbc072c00 2024-05-28T06:23:30.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.432+0000 7f8cb3400640 1 --2- 172.21.15.96:0/2700611118 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8cbc06dc20 0x7f8cbc15f9a0 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7f8cbc072bd0 tx=0x7f8ca402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14e917c5d28b8856 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:23:30.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.432+0000 7f8cbb400640 1 -- 172.21.15.96:0/2700611118 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ca4037070 con 0x7f8cbc06dc20 2024-05-28T06:23:30.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.432+0000 7f8cbb400640 1 -- 172.21.15.96:0/2700611118 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ca4032440 con 0x7f8cbc06dc20 2024-05-28T06:23:30.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.432+0000 7f8cbb400640 1 -- 172.21.15.96:0/2700611118 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ca4032810 con 0x7f8cbc06dc20 2024-05-28T06:23:30.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.433+0000 7f8cc1a00640 1 -- 172.21.15.96:0/2700611118 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8cbc06dc20 msgr2=0x7f8cbc15f9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:30.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.433+0000 7f8cc1a00640 1 --2- 172.21.15.96:0/2700611118 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8cbc06dc20 0x7f8cbc15f9a0 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7f8cbc072bd0 tx=0x7f8ca402a270 comp rx=0 tx=0).stop 2024-05-28T06:23:30.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.433+0000 7f8cc1a00640 1 -- 172.21.15.96:0/2700611118 shutdown_connections 2024-05-28T06:23:30.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.433+0000 7f8cc1a00640 1 --2- 172.21.15.96:0/2700611118 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8cbc06dc20 0x7f8cbc15f9a0 unknown :-1 s=CLOSED pgs=1878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:30.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.433+0000 7f8cc1a00640 1 --2- 172.21.15.96:0/2700611118 >> v2:172.21.15.123:3300/0 conn(0x7f8cbc15b3a0 0x7f8cbc0b32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:30.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.433+0000 7f8cc1a00640 1 -- 172.21.15.96:0/2700611118 >> 172.21.15.96:0/2700611118 conn(0x7f8cbc0b1fc0 msgr2=0x7f8cbc0b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:23:30.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.433+0000 7f8cc1a00640 1 -- 172.21.15.96:0/2700611118 shutdown_connections 2024-05-28T06:23:30.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.433+0000 7f8cc1a00640 1 -- 172.21.15.96:0/2700611118 wait complete. 2024-05-28T06:23:30.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.435+0000 7f8cc1a00640 1 Processor -- start 2024-05-28T06:23:30.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.435+0000 7f8cc1a00640 1 -- start start 2024-05-28T06:23:30.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.436+0000 7f8cc1a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8cbc06dc20 0x7f8cbc14abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:30.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.436+0000 7f8cc1a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8cbc072c00 0x7f8cbc15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:30.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.436+0000 7f8cc1a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8cbc15b3a0 0x7f8cbc0b1350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:30.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.436+0000 7f8cc1a00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f8cbc15b780 con 0x7f8cbc06dc20 2024-05-28T06:23:30.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.436+0000 7f8cc1a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f8cbc15fee0 con 0x7f8cbc15b3a0 2024-05-28T06:23:30.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.436+0000 7f8cc1a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f8cbc0ae550 con 0x7f8cbc072c00 2024-05-28T06:23:30.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.436+0000 7f8cbbe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8cbc06dc20 0x7f8cbc14abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:30.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.437+0000 7f8cbbe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8cbc06dc20 0x7f8cbc14abc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:50632/0 (socket says 172.21.15.96:50632) 2024-05-28T06:23:30.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.437+0000 7f8cbbe00640 1 -- 172.21.15.96:0/776110012 learned_addr learned my addr 172.21.15.96:0/776110012 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:23:30.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.437+0000 7f8cc1000640 1 --2- 172.21.15.96:0/776110012 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8cbc15b3a0 0x7f8cbc0b1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:30.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.437+0000 7f8cbbe00640 1 -- 172.21.15.96:0/776110012 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8cbc072c00 msgr2=0x7f8cbc15d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:30.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.437+0000 7f8cbbe00640 1 --2- 172.21.15.96:0/776110012 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8cbc072c00 0x7f8cbc15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:30.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.437+0000 7f8cbbe00640 1 -- 172.21.15.96:0/776110012 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8cbc15b3a0 msgr2=0x7f8cbc0b1350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:30.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.437+0000 7f8cbbe00640 1 --2- 172.21.15.96:0/776110012 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8cbc15b3a0 0x7f8cbc0b1350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:30.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.437+0000 7f8cbbe00640 1 -- 172.21.15.96:0/776110012 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cbc15e120 con 0x7f8cbc06dc20 2024-05-28T06:23:30.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.437+0000 7f8cc1000640 1 --2- 172.21.15.96:0/776110012 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8cbc15b3a0 0x7f8cbc0b1350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:23:30.433 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.437+0000 7f8cbbe00640 1 --2- 172.21.15.96:0/776110012 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8cbc06dc20 0x7f8cbc14abc0 secure :-1 s=READY pgs=1879 cs=0 l=1 rev1=1 crypto rx=0x7f8cac00c7f0 tx=0x7f8cac00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:23:30.433 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.438+0000 7f8cb8c00640 1 -- 172.21.15.96:0/776110012 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cac015070 con 0x7f8cbc06dc20 2024-05-28T06:23:30.433 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.438+0000 7f8cb8c00640 1 -- 172.21.15.96:0/776110012 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8cac004220 con 0x7f8cbc06dc20 2024-05-28T06:23:30.433 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.438+0000 7f8cb8c00640 1 -- 172.21.15.96:0/776110012 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cac007620 con 0x7f8cbc06dc20 2024-05-28T06:23:30.433 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.438+0000 7f8cc1a00640 1 -- 172.21.15.96:0/776110012 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8cbc0c97c0 con 0x7f8cbc06dc20 2024-05-28T06:23:30.433 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.438+0000 7f8cc1a00640 1 -- 172.21.15.96:0/776110012 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8cbc0c9d80 con 0x7f8cbc06dc20 2024-05-28T06:23:30.433 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.439+0000 7f8cc1a00640 1 -- 172.21.15.96:0/776110012 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8cbc0689e0 con 0x7f8cbc06dc20 2024-05-28T06:23:30.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.440+0000 7f8cb8c00640 1 -- 172.21.15.96:0/776110012 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 41) ==== 93488+0+0 (secure 0 0 0) 0x7f8cac0079d0 con 0x7f8cbc06dc20 2024-05-28T06:23:30.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.441+0000 7f8cb8c00640 1 --2- 172.21.15.96:0/776110012 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8c9806e3f0 0x7f8c980708b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:30.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.441+0000 7f8cb8c00640 1 -- 172.21.15.96:0/776110012 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 5823+0+0 (secure 0 0 0) 0x7f8cac08ada0 con 0x7f8cbc06dc20 2024-05-28T06:23:30.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.442+0000 7f8cb3400640 1 --2- 172.21.15.96:0/776110012 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8c9806e3f0 0x7f8c980708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:30.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.443+0000 7f8cb3400640 1 --2- 172.21.15.96:0/776110012 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8c9806e3f0 0x7f8c980708b0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f8ca4034000 tx=0x7f8ca402c000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:23:30.439 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.445+0000 7f8cb8c00640 1 -- 172.21.15.96:0/776110012 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8cac05acd0 con 0x7f8cbc06dc20 2024-05-28T06:23:30.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.669+0000 7f8cc1a00640 1 -- 172.21.15.96:0/776110012 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7f8cbc0b2e90 con 0x7f8c9806e3f0 2024-05-28T06:23:30.667 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.673+0000 7f8cb8c00640 1 -- 172.21.15.96:0/776110012 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7f8cbc0b2e90 con 0x7f8c9806e3f0 2024-05-28T06:23:30.667 INFO:teuthology.orchestra.run.smithi096.stdout:[ 2024-05-28T06:23:30.667 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:23:30.667 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "cephfs" 2024-05-28T06:23:30.668 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:23:30.668 INFO:teuthology.orchestra.run.smithi096.stdout:] 2024-05-28T06:23:30.669 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.675+0000 7f8cc1a00640 1 -- 172.21.15.96:0/776110012 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8c9806e3f0 msgr2=0x7f8c980708b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:30.670 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.675+0000 7f8cc1a00640 1 --2- 172.21.15.96:0/776110012 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8c9806e3f0 0x7f8c980708b0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f8ca4034000 tx=0x7f8ca402c000 comp rx=0 tx=0).stop 2024-05-28T06:23:30.670 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.675+0000 7f8cc1a00640 1 -- 172.21.15.96:0/776110012 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8cbc06dc20 msgr2=0x7f8cbc14abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:30.670 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.675+0000 7f8cc1a00640 1 --2- 172.21.15.96:0/776110012 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8cbc06dc20 0x7f8cbc14abc0 secure :-1 s=READY pgs=1879 cs=0 l=1 rev1=1 crypto rx=0x7f8cac00c7f0 tx=0x7f8cac00ccb0 comp rx=0 tx=0).stop 2024-05-28T06:23:30.670 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.675+0000 7f8cc1a00640 1 -- 172.21.15.96:0/776110012 shutdown_connections 2024-05-28T06:23:30.670 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.675+0000 7f8cc1a00640 1 --2- 172.21.15.96:0/776110012 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8c9806e3f0 0x7f8c980708b0 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:30.670 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.675+0000 7f8cc1a00640 1 --2- 172.21.15.96:0/776110012 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8cbc15b3a0 0x7f8cbc0b1350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:30.670 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.675+0000 7f8cc1a00640 1 --2- 172.21.15.96:0/776110012 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8cbc072c00 0x7f8cbc15d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:30.670 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.675+0000 7f8cc1a00640 1 --2- 172.21.15.96:0/776110012 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8cbc06dc20 0x7f8cbc14abc0 unknown :-1 s=CLOSED pgs=1879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:30.670 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.675+0000 7f8cc1a00640 1 -- 172.21.15.96:0/776110012 >> 172.21.15.96:0/776110012 conn(0x7f8cbc0b1fc0 msgr2=0x7f8cbc14de40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:23:30.670 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.676+0000 7f8cc1a00640 1 -- 172.21.15.96:0/776110012 shutdown_connections 2024-05-28T06:23:30.670 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.676+0000 7f8cc1a00640 1 -- 172.21.15.96:0/776110012 wait complete. 2024-05-28T06:23:30.689 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-05-28T06:23:30.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.890+0000 7f65f0800640 1 Processor -- start 2024-05-28T06:23:30.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.890+0000 7f65f0800640 1 -- start start 2024-05-28T06:23:30.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.891+0000 7f65f0800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f65e80d0890 0x7f65e80b43b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:30.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.891+0000 7f65f0800640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65e80cf0c0 con 0x7f65e8145320 2024-05-28T06:23:30.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.891+0000 7f65f0800640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65e80b4a20 con 0x7f65e80723d0 2024-05-28T06:23:30.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.891+0000 7f65f0800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f65e80b4d00 con 0x7f65e80d0890 2024-05-28T06:23:30.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.891+0000 7f65ede00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f65e8145320 0x7f65e8071cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:33908/0 (socket says 172.21.15.96:33908) 2024-05-28T06:23:30.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.891+0000 7f65ede00640 1 -- 172.21.15.96:0/2265390043 learned_addr learned my addr 172.21.15.96:0/2265390043 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:23:30.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.892+0000 7f65eca00640 1 -- 172.21.15.96:0/2265390043 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1239587906 0 0) 0x7f65e80cf0c0 con 0x7f65e8145320 2024-05-28T06:23:30.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.892+0000 7f65eca00640 1 -- 172.21.15.96:0/2265390043 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f65d0003520 con 0x7f65e8145320 2024-05-28T06:23:30.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.892+0000 7f65ee800640 1 --2- 172.21.15.96:0/2265390043 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f65e80d0890 0x7f65e80b43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:30.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.892+0000 7f65eca00640 1 -- 172.21.15.96:0/2265390043 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3198341451 0 0) 0x7f65d0003520 con 0x7f65e8145320 2024-05-28T06:23:30.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.892+0000 7f65eca00640 1 -- 172.21.15.96:0/2265390043 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f65e80d0890 msgr2=0x7f65e80b43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:23:30.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.892+0000 7f65eca00640 1 --2- 172.21.15.96:0/2265390043 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f65e80d0890 0x7f65e80b43b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:30.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.892+0000 7f65eca00640 1 -- 172.21.15.96:0/2265390043 >> v1:172.21.15.123:6789/0 conn(0x7f65e80723d0 legacy=0x7f65e80b3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:30.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.892+0000 7f65eca00640 1 -- 172.21.15.96:0/2265390043 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65e80b4fe0 con 0x7f65e8145320 2024-05-28T06:23:30.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.892+0000 7f65eca00640 1 -- 172.21.15.96:0/2265390043 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f65dc001690 con 0x7f65e8145320 2024-05-28T06:23:30.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.892+0000 7f65eca00640 1 -- 172.21.15.96:0/2265390043 >> v1:172.21.15.96:6789/0 conn(0x7f65e8145320 legacy=0x7f65e8071cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:30.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.892+0000 7f65eca00640 1 --2- 172.21.15.96:0/2265390043 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f65d00039e0 0x7f65d0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:30.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.892+0000 7f65eca00640 1 --2- 172.21.15.96:0/2265390043 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65d0004800 0x7f65d0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:30.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.892+0000 7f65eca00640 1 --2- 172.21.15.96:0/2265390043 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f65e80d0890 0x7f65d000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:30.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.893+0000 7f65eca00640 1 -- 172.21.15.96:0/2265390043 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f65e80b4d00 con 0x7f65d00039e0 2024-05-28T06:23:30.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.893+0000 7f65eca00640 1 -- 172.21.15.96:0/2265390043 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f65d0005520 con 0x7f65d0004800 2024-05-28T06:23:30.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.893+0000 7f65eca00640 1 -- 172.21.15.96:0/2265390043 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f65d000bf00 con 0x7f65e80d0890 2024-05-28T06:23:30.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.893+0000 7f65ede00640 1 --2- 172.21.15.96:0/2265390043 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f65d00039e0 0x7f65d0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:30.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.893+0000 7f65ede00640 1 -- 172.21.15.96:0/2265390043 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f65e80d0890 msgr2=0x7f65d000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:23:30.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.893+0000 7f65ede00640 1 --2- 172.21.15.96:0/2265390043 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f65e80d0890 0x7f65d000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:30.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.893+0000 7f65ede00640 1 -- 172.21.15.96:0/2265390043 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65d0004800 msgr2=0x7f65d0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:23:30.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.893+0000 7f65ed400640 1 --2- 172.21.15.96:0/2265390043 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65d0004800 0x7f65d0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:30.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.893+0000 7f65ede00640 1 --2- 172.21.15.96:0/2265390043 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65d0004800 0x7f65d0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:30.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.893+0000 7f65ede00640 1 -- 172.21.15.96:0/2265390043 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65d0003520 con 0x7f65d00039e0 2024-05-28T06:23:30.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.893+0000 7f65ed400640 1 --2- 172.21.15.96:0/2265390043 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65d0004800 0x7f65d0004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:23:30.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.894+0000 7f65ede00640 1 --2- 172.21.15.96:0/2265390043 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f65d00039e0 0x7f65d0003e00 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7f65dc00c210 tx=0x7f65d0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ac705f995991293 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:23:30.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.894+0000 7f65eca00640 1 -- 172.21.15.96:0/2265390043 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65dc007d10 con 0x7f65d00039e0 2024-05-28T06:23:30.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.894+0000 7f65eca00640 1 -- 172.21.15.96:0/2265390043 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65dc005af0 con 0x7f65d00039e0 2024-05-28T06:23:30.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.894+0000 7f65eca00640 1 -- 172.21.15.96:0/2265390043 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65dc007070 con 0x7f65d00039e0 2024-05-28T06:23:30.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.895+0000 7f65f0800640 1 -- 172.21.15.96:0/2265390043 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f65d00039e0 msgr2=0x7f65d0003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:30.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.895+0000 7f65f0800640 1 --2- 172.21.15.96:0/2265390043 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f65d00039e0 0x7f65d0003e00 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7f65dc00c210 tx=0x7f65d0005830 comp rx=0 tx=0).stop 2024-05-28T06:23:30.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.895+0000 7f65ee800640 1 -- 172.21.15.96:0/2265390043 reap_dead start 2024-05-28T06:23:30.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.895+0000 7f65f0800640 1 -- 172.21.15.96:0/2265390043 shutdown_connections 2024-05-28T06:23:30.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.895+0000 7f65f0800640 1 -- 172.21.15.96:0/2265390043 >> 172.21.15.96:0/2265390043 conn(0x7f65e8067ce0 msgr2=0x7f65e806b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:23:30.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.895+0000 7f65f0800640 1 -- 172.21.15.96:0/2265390043 shutdown_connections 2024-05-28T06:23:30.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.895+0000 7f65f0800640 1 -- 172.21.15.96:0/2265390043 wait complete. 2024-05-28T06:23:30.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.897+0000 7f65f0800640 1 Processor -- start 2024-05-28T06:23:30.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.897+0000 7f65f0800640 1 -- start start 2024-05-28T06:23:30.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.898+0000 7f65f0800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f65d00039e0 0x7f65e80acbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:30.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.898+0000 7f65f0800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65e813fef0 0x7f65e81407a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:30.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.898+0000 7f65f0800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f65e8140370 0x7f65e80aa980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:30.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.898+0000 7f65f0800640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f65e80b43b0 con 0x7f65e8140370 2024-05-28T06:23:30.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.898+0000 7f65f0800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f65e8145700 con 0x7f65e813fef0 2024-05-28T06:23:30.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.898+0000 7f65f0800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f65e80b4530 con 0x7f65d00039e0 2024-05-28T06:23:30.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.898+0000 7f65ee800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f65e8140370 0x7f65e80aa980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:30.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.898+0000 7f65ee800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f65e8140370 0x7f65e80aa980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:50662/0 (socket says 172.21.15.96:50662) 2024-05-28T06:23:30.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.898+0000 7f65ee800640 1 -- 172.21.15.96:0/3875372109 learned_addr learned my addr 172.21.15.96:0/3875372109 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:23:30.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.898+0000 7f65ee800640 1 -- 172.21.15.96:0/3875372109 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f65d00039e0 msgr2=0x7f65e80acbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:30.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.898+0000 7f65ee800640 1 --2- 172.21.15.96:0/3875372109 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f65d00039e0 0x7f65e80acbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:30.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.898+0000 7f65ee800640 1 -- 172.21.15.96:0/3875372109 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65e813fef0 msgr2=0x7f65e81407a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:30.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.898+0000 7f65ee800640 1 --2- 172.21.15.96:0/3875372109 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65e813fef0 0x7f65e81407a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:30.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.898+0000 7f65ee800640 1 -- 172.21.15.96:0/3875372109 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65e815ede0 con 0x7f65e8140370 2024-05-28T06:23:30.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.899+0000 7f65ee800640 1 --2- 172.21.15.96:0/3875372109 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f65e8140370 0x7f65e80aa980 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7f65e806c9c0 tx=0x7f65e400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:23:30.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.899+0000 7f65e2000640 1 -- 172.21.15.96:0/3875372109 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65e400eb90 con 0x7f65e8140370 2024-05-28T06:23:30.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.899+0000 7f65e2000640 1 -- 172.21.15.96:0/3875372109 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65e400ed50 con 0x7f65e8140370 2024-05-28T06:23:30.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.899+0000 7f65f0800640 1 -- 172.21.15.96:0/3875372109 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65e815f0c0 con 0x7f65e8140370 2024-05-28T06:23:30.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.899+0000 7f65e2000640 1 -- 172.21.15.96:0/3875372109 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65e4003bd0 con 0x7f65e8140370 2024-05-28T06:23:30.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.899+0000 7f65f0800640 1 -- 172.21.15.96:0/3875372109 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65e80aa180 con 0x7f65e8140370 2024-05-28T06:23:30.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.900+0000 7f65f0800640 1 -- 172.21.15.96:0/3875372109 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65e80689e0 con 0x7f65e8140370 2024-05-28T06:23:30.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.901+0000 7f65e2000640 1 -- 172.21.15.96:0/3875372109 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 41) ==== 93488+0+0 (secure 0 0 0) 0x7f65e401b020 con 0x7f65e8140370 2024-05-28T06:23:30.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.903+0000 7f65e2000640 1 --2- 172.21.15.96:0/3875372109 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f65cc06e3f0 0x7f65cc0708b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:30.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.903+0000 7f65e2000640 1 -- 172.21.15.96:0/3875372109 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 5823+0+0 (secure 0 0 0) 0x7f65e408a860 con 0x7f65e8140370 2024-05-28T06:23:30.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.903+0000 7f65ede00640 1 --2- 172.21.15.96:0/3875372109 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f65cc06e3f0 0x7f65cc0708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:30.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.904+0000 7f65ede00640 1 --2- 172.21.15.96:0/3875372109 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f65cc06e3f0 0x7f65cc0708b0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f65dc032000 tx=0x7f65dc007000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:23:30.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:30.907+0000 7f65e2000640 1 -- 172.21.15.96:0/3875372109 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f65e405a6f0 con 0x7f65e8140370 2024-05-28T06:23:31.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:31.175+0000 7f65f0800640 1 -- 172.21.15.96:0/3875372109 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f65e80b34e0 con 0x7f65e8140370 2024-05-28T06:23:31.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:31.176+0000 7f65e2000640 1 -- 172.21.15.96:0/3875372109 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+2125 (secure 0 0 0) 0x7f65e4059ff0 con 0x7f65e8140370 2024-05-28T06:23:31.170 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:23:31.170 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:23:31.170 INFO:teuthology.orchestra.run.smithi096.stdout: "fsid": "e4a831b5-e6d1-47a7-96cb-bf85154ee0d2", 2024-05-28T06:23:31.170 INFO:teuthology.orchestra.run.smithi096.stdout: "health": { 2024-05-28T06:23:31.170 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "HEALTH_OK", 2024-05-28T06:23:31.171 INFO:teuthology.orchestra.run.smithi096.stdout: "checks": {}, 2024-05-28T06:23:31.171 INFO:teuthology.orchestra.run.smithi096.stdout: "mutes": [] 2024-05-28T06:23:31.171 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:23:31.171 INFO:teuthology.orchestra.run.smithi096.stdout: "election_epoch": 8, 2024-05-28T06:23:31.171 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum": [ 2024-05-28T06:23:31.171 INFO:teuthology.orchestra.run.smithi096.stdout: 0, 2024-05-28T06:23:31.171 INFO:teuthology.orchestra.run.smithi096.stdout: 1, 2024-05-28T06:23:31.171 INFO:teuthology.orchestra.run.smithi096.stdout: 2 2024-05-28T06:23:31.171 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:23:31.171 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_names": [ 2024-05-28T06:23:31.171 INFO:teuthology.orchestra.run.smithi096.stdout: "a", 2024-05-28T06:23:31.171 INFO:teuthology.orchestra.run.smithi096.stdout: "b", 2024-05-28T06:23:31.171 INFO:teuthology.orchestra.run.smithi096.stdout: "c" 2024-05-28T06:23:31.172 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:23:31.172 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_age": 1184, 2024-05-28T06:23:31.172 INFO:teuthology.orchestra.run.smithi096.stdout: "monmap": { 2024-05-28T06:23:31.172 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 1, 2024-05-28T06:23:31.172 INFO:teuthology.orchestra.run.smithi096.stdout: "min_mon_release_name": "squid", 2024-05-28T06:23:31.172 INFO:teuthology.orchestra.run.smithi096.stdout: "num_mons": 3 2024-05-28T06:23:31.172 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:23:31.172 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap": { 2024-05-28T06:23:31.172 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 117, 2024-05-28T06:23:31.172 INFO:teuthology.orchestra.run.smithi096.stdout: "num_osds": 8, 2024-05-28T06:23:31.172 INFO:teuthology.orchestra.run.smithi096.stdout: "num_up_osds": 8, 2024-05-28T06:23:31.172 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_up_since": 1716876238, 2024-05-28T06:23:31.172 INFO:teuthology.orchestra.run.smithi096.stdout: "num_in_osds": 8, 2024-05-28T06:23:31.173 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_in_since": 1716876229, 2024-05-28T06:23:31.173 INFO:teuthology.orchestra.run.smithi096.stdout: "num_remapped_pgs": 0 2024-05-28T06:23:31.173 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:23:31.173 INFO:teuthology.orchestra.run.smithi096.stdout: "pgmap": { 2024-05-28T06:23:31.173 INFO:teuthology.orchestra.run.smithi096.stdout: "pgs_by_state": [ 2024-05-28T06:23:31.173 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:23:31.173 INFO:teuthology.orchestra.run.smithi096.stdout: "state_name": "active+clean", 2024-05-28T06:23:31.173 INFO:teuthology.orchestra.run.smithi096.stdout: "count": 97 2024-05-28T06:23:31.173 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:23:31.173 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:23:31.173 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pgs": 97, 2024-05-28T06:23:31.173 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pools": 3, 2024-05-28T06:23:31.174 INFO:teuthology.orchestra.run.smithi096.stdout: "num_objects": 1748, 2024-05-28T06:23:31.174 INFO:teuthology.orchestra.run.smithi096.stdout: "data_bytes": 7190339917, 2024-05-28T06:23:31.174 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_used": 15374557184, 2024-05-28T06:23:31.174 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_avail": 757719556096, 2024-05-28T06:23:31.174 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_total": 773094113280, 2024-05-28T06:23:31.174 INFO:teuthology.orchestra.run.smithi096.stdout: "read_bytes_sec": 336762769, 2024-05-28T06:23:31.174 INFO:teuthology.orchestra.run.smithi096.stdout: "write_bytes_sec": 330520778, 2024-05-28T06:23:31.174 INFO:teuthology.orchestra.run.smithi096.stdout: "read_op_per_sec": 83, 2024-05-28T06:23:31.174 INFO:teuthology.orchestra.run.smithi096.stdout: "write_op_per_sec": 94 2024-05-28T06:23:31.174 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:23:31.174 INFO:teuthology.orchestra.run.smithi096.stdout: "fsmap": { 2024-05-28T06:23:31.174 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 73, 2024-05-28T06:23:31.175 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 8, 2024-05-28T06:23:31.175 INFO:teuthology.orchestra.run.smithi096.stdout: "up": 1, 2024-05-28T06:23:31.175 INFO:teuthology.orchestra.run.smithi096.stdout: "in": 1, 2024-05-28T06:23:31.175 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 1, 2024-05-28T06:23:31.175 INFO:teuthology.orchestra.run.smithi096.stdout: "by_rank": [ 2024-05-28T06:23:31.175 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:23:31.175 INFO:teuthology.orchestra.run.smithi096.stdout: "filesystem_id": 8, 2024-05-28T06:23:31.175 INFO:teuthology.orchestra.run.smithi096.stdout: "rank": 0, 2024-05-28T06:23:31.175 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "b", 2024-05-28T06:23:31.175 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "up:active", 2024-05-28T06:23:31.175 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 9270 2024-05-28T06:23:31.176 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:23:31.176 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:23:31.176 INFO:teuthology.orchestra.run.smithi096.stdout: "up:standby": 3 2024-05-28T06:23:31.176 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:23:31.176 INFO:teuthology.orchestra.run.smithi096.stdout: "mgrmap": { 2024-05-28T06:23:31.176 INFO:teuthology.orchestra.run.smithi096.stdout: "available": true, 2024-05-28T06:23:31.176 INFO:teuthology.orchestra.run.smithi096.stdout: "num_standbys": 1, 2024-05-28T06:23:31.176 INFO:teuthology.orchestra.run.smithi096.stdout: "modules": [ 2024-05-28T06:23:31.177 INFO:teuthology.orchestra.run.smithi096.stdout: "iostat", 2024-05-28T06:23:31.177 INFO:teuthology.orchestra.run.smithi096.stdout: "nfs", 2024-05-28T06:23:31.177 INFO:teuthology.orchestra.run.smithi096.stdout: "restful" 2024-05-28T06:23:31.177 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:23:31.177 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:23:31.177 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:23:31.177 INFO:teuthology.orchestra.run.smithi096.stdout: "servicemap": { 2024-05-28T06:23:31.177 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 23, 2024-05-28T06:23:31.177 INFO:teuthology.orchestra.run.smithi096.stdout: "modified": "2024-05-28T06:23:23.993668+0000", 2024-05-28T06:23:31.178 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:23:31.178 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:23:31.178 INFO:teuthology.orchestra.run.smithi096.stdout: "progress_events": { 2024-05-28T06:23:31.178 INFO:teuthology.orchestra.run.smithi096.stdout: "d6fb006f-47c2-4c80-b874-17506401c2f6": { 2024-05-28T06:23:31.178 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "All cloning finished, waiting 26.0 seconds before exiting (10s)\n [============================] ", 2024-05-28T06:23:31.178 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": 1, 2024-05-28T06:23:31.178 INFO:teuthology.orchestra.run.smithi096.stdout: "add_to_ceph_s": true 2024-05-28T06:23:31.178 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:23:31.178 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:23:31.178 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:23:31.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:31.179+0000 7f65f0800640 1 -- 172.21.15.96:0/3875372109 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f65cc06e3f0 msgr2=0x7f65cc0708b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:31.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:31.179+0000 7f65f0800640 1 --2- 172.21.15.96:0/3875372109 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f65cc06e3f0 0x7f65cc0708b0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f65dc032000 tx=0x7f65dc007000 comp rx=0 tx=0).stop 2024-05-28T06:23:31.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:31.179+0000 7f65f0800640 1 -- 172.21.15.96:0/3875372109 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f65e8140370 msgr2=0x7f65e80aa980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:31.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:31.179+0000 7f65f0800640 1 --2- 172.21.15.96:0/3875372109 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f65e8140370 0x7f65e80aa980 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7f65e806c9c0 tx=0x7f65e400e4b0 comp rx=0 tx=0).stop 2024-05-28T06:23:31.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:31.179+0000 7f65f0800640 1 -- 172.21.15.96:0/3875372109 shutdown_connections 2024-05-28T06:23:31.180 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:31.179+0000 7f65f0800640 1 --2- 172.21.15.96:0/3875372109 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f65cc06e3f0 0x7f65cc0708b0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:31.180 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:31.179+0000 7f65f0800640 1 --2- 172.21.15.96:0/3875372109 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f65e8140370 0x7f65e80aa980 unknown :-1 s=CLOSED pgs=1882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:31.180 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:31.179+0000 7f65f0800640 1 --2- 172.21.15.96:0/3875372109 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65e813fef0 0x7f65e81407a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:31.180 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:31.179+0000 7f65f0800640 1 --2- 172.21.15.96:0/3875372109 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f65d00039e0 0x7f65e80acbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:31.180 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:31.179+0000 7f65f0800640 1 -- 172.21.15.96:0/3875372109 >> 172.21.15.96:0/3875372109 conn(0x7f65e8067ce0 msgr2=0x7f65e806e550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:23:31.180 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:31.179+0000 7f65f0800640 1 -- 172.21.15.96:0/3875372109 shutdown_connections 2024-05-28T06:23:31.180 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:31.179+0000 7f65f0800640 1 -- 172.21.15.96:0/3875372109 wait complete. 2024-05-28T06:23:34.566 DEBUG:teuthology.orchestra.run.smithi096:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:23:34.575 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:23:36.192 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-05-28T06:23:36.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:36.404+0000 7ffa38c00640 1 Processor -- start 2024-05-28T06:23:36.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:36.404+0000 7ffa38c00640 1 -- start start 2024-05-28T06:23:36.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:36.405+0000 7ffa38c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffa340ccc30 0x7ffa3406dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:36.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:36.406+0000 7ffa38c00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7ffa340d0460 0x7ffa340d08f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:36.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:36.406+0000 7ffa38c00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa340cd1a0 con 0x7ffa3415c6b0 2024-05-28T06:23:36.400 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:36.406+0000 7ffa38c00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7ffa340cd480 con 0x7ffa340d0460 2024-05-28T06:23:36.400 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:36.406+0000 7ffa38c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ffa340cd760 con 0x7ffa340ccc30 2024-05-28T06:23:36.400 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:36.406+0000 7ffa33e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7ffa340d0460 0x7ffa340d08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:36.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:36.406+0000 7ffa33e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7ffa340d0460 0x7ffa340d08f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:50668/0 (socket says 172.21.15.96:50668) 2024-05-28T06:23:36.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:36.406+0000 7ffa33e00640 1 -- 172.21.15.96:0/3242100023 learned_addr learned my addr 172.21.15.96:0/3242100023 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:23:36.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:36.406+0000 7ffa32a00640 1 --2- 172.21.15.96:0/3242100023 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffa340ccc30 0x7ffa3406dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:36.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:36.407+0000 7ffa33e00640 1 -- 172.21.15.96:0/3242100023 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffa340ccc30 msgr2=0x7ffa3406dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:23:36.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:36.407+0000 7ffa33e00640 1 --2- 172.21.15.96:0/3242100023 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffa340ccc30 0x7ffa3406dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:36.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:36.407+0000 7ffa33e00640 1 -- 172.21.15.96:0/3242100023 >> v1:172.21.15.123:6789/0 conn(0x7ffa3415c6b0 legacy=0x7ffa340b3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:23:36.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:36.407+0000 7ffa33e00640 1 -- 172.21.15.96:0/3242100023 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa340cda40 con 0x7ffa340d0460 2024-05-28T06:23:36.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:36.408+0000 7ffa33e00640 1 --2- 172.21.15.96:0/3242100023 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa340d0460 0x7ffa340d08f0 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7ffa2c009e50 tx=0x7ffa2c02a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c657fcfe5c029f8 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:23:36.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:36.408+0000 7ffa32000640 1 -- 172.21.15.96:0/3242100023 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa2c036070 con 0x7ffa340d0460 2024-05-28T06:23:36.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:36.408+0000 7ffa32000640 1 -- 172.21.15.96:0/3242100023 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa2c031600 con 0x7ffa340d0460 2024-05-28T06:23:36.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:36.408+0000 7ffa32000640 1 -- 172.21.15.96:0/3242100023 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa2c0319d0 con 0x7ffa340d0460 2024-05-28T06:23:36.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:36.408+0000 7ffa38c00640 1 -- 172.21.15.96:0/3242100023 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa340d0460 msgr2=0x7ffa340d08f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:36.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:36.408+0000 7ffa38c00640 1 --2- 172.21.15.96:0/3242100023 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa340d0460 0x7ffa340d08f0 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7ffa2c009e50 tx=0x7ffa2c02a6c0 comp rx=0 tx=0).stop 2024-05-28T06:23:36.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:36.409+0000 7ffa38c00640 1 -- 172.21.15.96:0/3242100023 shutdown_connections 2024-05-28T06:23:36.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:36.409+0000 7ffa38c00640 1 --2- 172.21.15.96:0/3242100023 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa340d0460 0x7ffa340d08f0 unknown :-1 s=CLOSED pgs=1883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:36.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:36.409+0000 7ffa38c00640 1 --2- 172.21.15.96:0/3242100023 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffa340ccc30 0x7ffa3406dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:36.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:36.409+0000 7ffa38c00640 1 -- 172.21.15.96:0/3242100023 >> 172.21.15.96:0/3242100023 conn(0x7ffa3406bae0 msgr2=0x7ffa3406bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:23:36.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:36.409+0000 7ffa38c00640 1 -- 172.21.15.96:0/3242100023 shutdown_connections 2024-05-28T06:23:36.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:36.409+0000 7ffa38c00640 1 -- 172.21.15.96:0/3242100023 wait complete. 2024-05-28T06:23:36.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:36.410+0000 7ffa38c00640 1 Processor -- start 2024-05-28T06:23:36.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:36.411+0000 7ffa38c00640 1 -- start start 2024-05-28T06:23:36.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:36.411+0000 7ffa38c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa340ccc30 0x7ffa341530d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:36.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:36.411+0000 7ffa38c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa340d0460 0x7ffa340b07d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:36.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:36.411+0000 7ffa38c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffa3415c6b0 0x7ffa34099800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:36.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:36.411+0000 7ffa38c00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ffa3406e350 con 0x7ffa340ccc30 2024-05-28T06:23:36.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:36.411+0000 7ffa38c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ffa3406e1d0 con 0x7ffa340d0460 2024-05-28T06:23:36.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:36.411+0000 7ffa38c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ffa340cbf30 con 0x7ffa3415c6b0 2024-05-28T06:23:36.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:36.412+0000 7ffa33400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa340ccc30 0x7ffa341530d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:36.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:36.412+0000 7ffa33400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa340ccc30 0x7ffa341530d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:50672/0 (socket says 172.21.15.96:50672) 2024-05-28T06:23:36.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:36.412+0000 7ffa33400640 1 -- 172.21.15.96:0/1733450260 learned_addr learned my addr 172.21.15.96:0/1733450260 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:23:36.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:36.412+0000 7ffa33e00640 1 --2- 172.21.15.96:0/1733450260 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffa3415c6b0 0x7ffa34099800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:36.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:36.412+0000 7ffa33400640 1 -- 172.21.15.96:0/1733450260 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffa3415c6b0 msgr2=0x7ffa34099800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:36.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:36.412+0000 7ffa33400640 1 --2- 172.21.15.96:0/1733450260 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffa3415c6b0 0x7ffa34099800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:36.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:36.412+0000 7ffa32a00640 1 --2- 172.21.15.96:0/1733450260 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa340d0460 0x7ffa340b07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:36.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:36.412+0000 7ffa33400640 1 -- 172.21.15.96:0/1733450260 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa340d0460 msgr2=0x7ffa340b07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:36.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:36.412+0000 7ffa33400640 1 --2- 172.21.15.96:0/1733450260 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa340d0460 0x7ffa340b07d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:36.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:36.412+0000 7ffa33400640 1 -- 172.21.15.96:0/1733450260 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa340b0d10 con 0x7ffa340ccc30 2024-05-28T06:23:36.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:36.412+0000 7ffa33400640 1 --2- 172.21.15.96:0/1733450260 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa340ccc30 0x7ffa341530d0 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7ffa2400b3f0 tx=0x7ffa2400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:23:36.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:36.413+0000 7ffa2b400640 1 -- 172.21.15.96:0/1733450260 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa24015020 con 0x7ffa340ccc30 2024-05-28T06:23:36.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:36.413+0000 7ffa2b400640 1 -- 172.21.15.96:0/1733450260 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa24011830 con 0x7ffa340ccc30 2024-05-28T06:23:36.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:36.413+0000 7ffa2b400640 1 -- 172.21.15.96:0/1733450260 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa24011c20 con 0x7ffa340ccc30 2024-05-28T06:23:36.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:36.413+0000 7ffa38c00640 1 -- 172.21.15.96:0/1733450260 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa340c6730 con 0x7ffa340ccc30 2024-05-28T06:23:36.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:36.413+0000 7ffa38c00640 1 -- 172.21.15.96:0/1733450260 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa340c6cf0 con 0x7ffa340ccc30 2024-05-28T06:23:36.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:36.414+0000 7ffa38c00640 1 -- 172.21.15.96:0/1733450260 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa340689e0 con 0x7ffa340ccc30 2024-05-28T06:23:36.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:36.415+0000 7ffa2b400640 1 -- 172.21.15.96:0/1733450260 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 41) ==== 93488+0+0 (secure 0 0 0) 0x7ffa24018020 con 0x7ffa340ccc30 2024-05-28T06:23:36.410 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:36.416+0000 7ffa2b400640 1 --2- 172.21.15.96:0/1733450260 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ffa1406e3f0 0x7ffa140708b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:36.410 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:36.416+0000 7ffa2b400640 1 -- 172.21.15.96:0/1733450260 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 5823+0+0 (secure 0 0 0) 0x7ffa2408a630 con 0x7ffa340ccc30 2024-05-28T06:23:36.410 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:36.416+0000 7ffa32a00640 1 --2- 172.21.15.96:0/1733450260 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ffa1406e3f0 0x7ffa140708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:36.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:36.418+0000 7ffa32a00640 1 --2- 172.21.15.96:0/1733450260 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ffa1406e3f0 0x7ffa140708b0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7ffa3415bb30 tx=0x7ffa20008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:23:36.415 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:36.421+0000 7ffa2b400640 1 -- 172.21.15.96:0/1733450260 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ffa2401f030 con 0x7ffa340ccc30 2024-05-28T06:23:36.695 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:36.701+0000 7ffa38c00640 1 -- 172.21.15.96:0/1733450260 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7ffa340cfee0 con 0x7ffa340ccc30 2024-05-28T06:23:36.696 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:36.702+0000 7ffa2b400640 1 -- 172.21.15.96:0/1733450260 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+2125 (secure 0 0 0) 0x7ffa240172c0 con 0x7ffa340ccc30 2024-05-28T06:23:36.696 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:23:36.696 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:23:36.696 INFO:teuthology.orchestra.run.smithi096.stdout: "fsid": "e4a831b5-e6d1-47a7-96cb-bf85154ee0d2", 2024-05-28T06:23:36.696 INFO:teuthology.orchestra.run.smithi096.stdout: "health": { 2024-05-28T06:23:36.696 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "HEALTH_OK", 2024-05-28T06:23:36.696 INFO:teuthology.orchestra.run.smithi096.stdout: "checks": {}, 2024-05-28T06:23:36.696 INFO:teuthology.orchestra.run.smithi096.stdout: "mutes": [] 2024-05-28T06:23:36.697 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:23:36.697 INFO:teuthology.orchestra.run.smithi096.stdout: "election_epoch": 8, 2024-05-28T06:23:36.697 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum": [ 2024-05-28T06:23:36.697 INFO:teuthology.orchestra.run.smithi096.stdout: 0, 2024-05-28T06:23:36.697 INFO:teuthology.orchestra.run.smithi096.stdout: 1, 2024-05-28T06:23:36.697 INFO:teuthology.orchestra.run.smithi096.stdout: 2 2024-05-28T06:23:36.697 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:23:36.697 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_names": [ 2024-05-28T06:23:36.697 INFO:teuthology.orchestra.run.smithi096.stdout: "a", 2024-05-28T06:23:36.697 INFO:teuthology.orchestra.run.smithi096.stdout: "b", 2024-05-28T06:23:36.697 INFO:teuthology.orchestra.run.smithi096.stdout: "c" 2024-05-28T06:23:36.697 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:23:36.697 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_age": 1190, 2024-05-28T06:23:36.698 INFO:teuthology.orchestra.run.smithi096.stdout: "monmap": { 2024-05-28T06:23:36.698 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 1, 2024-05-28T06:23:36.698 INFO:teuthology.orchestra.run.smithi096.stdout: "min_mon_release_name": "squid", 2024-05-28T06:23:36.698 INFO:teuthology.orchestra.run.smithi096.stdout: "num_mons": 3 2024-05-28T06:23:36.698 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:23:36.698 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap": { 2024-05-28T06:23:36.698 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 117, 2024-05-28T06:23:36.698 INFO:teuthology.orchestra.run.smithi096.stdout: "num_osds": 8, 2024-05-28T06:23:36.698 INFO:teuthology.orchestra.run.smithi096.stdout: "num_up_osds": 8, 2024-05-28T06:23:36.698 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_up_since": 1716876238, 2024-05-28T06:23:36.698 INFO:teuthology.orchestra.run.smithi096.stdout: "num_in_osds": 8, 2024-05-28T06:23:36.699 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_in_since": 1716876229, 2024-05-28T06:23:36.699 INFO:teuthology.orchestra.run.smithi096.stdout: "num_remapped_pgs": 0 2024-05-28T06:23:36.699 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:23:36.699 INFO:teuthology.orchestra.run.smithi096.stdout: "pgmap": { 2024-05-28T06:23:36.699 INFO:teuthology.orchestra.run.smithi096.stdout: "pgs_by_state": [ 2024-05-28T06:23:36.699 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:23:36.699 INFO:teuthology.orchestra.run.smithi096.stdout: "state_name": "active+clean", 2024-05-28T06:23:36.699 INFO:teuthology.orchestra.run.smithi096.stdout: "count": 97 2024-05-28T06:23:36.699 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:23:36.700 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:23:36.700 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pgs": 97, 2024-05-28T06:23:36.700 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pools": 3, 2024-05-28T06:23:36.700 INFO:teuthology.orchestra.run.smithi096.stdout: "num_objects": 1745, 2024-05-28T06:23:36.700 INFO:teuthology.orchestra.run.smithi096.stdout: "data_bytes": 7190372216, 2024-05-28T06:23:36.700 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_used": 15424937984, 2024-05-28T06:23:36.700 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_avail": 757669175296, 2024-05-28T06:23:36.700 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_total": 773094113280, 2024-05-28T06:23:36.700 INFO:teuthology.orchestra.run.smithi096.stdout: "read_bytes_sec": 258897130, 2024-05-28T06:23:36.700 INFO:teuthology.orchestra.run.smithi096.stdout: "write_bytes_sec": 260327769, 2024-05-28T06:23:36.700 INFO:teuthology.orchestra.run.smithi096.stdout: "read_op_per_sec": 63, 2024-05-28T06:23:36.700 INFO:teuthology.orchestra.run.smithi096.stdout: "write_op_per_sec": 73 2024-05-28T06:23:36.700 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:23:36.700 INFO:teuthology.orchestra.run.smithi096.stdout: "fsmap": { 2024-05-28T06:23:36.701 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 73, 2024-05-28T06:23:36.701 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 8, 2024-05-28T06:23:36.701 INFO:teuthology.orchestra.run.smithi096.stdout: "up": 1, 2024-05-28T06:23:36.701 INFO:teuthology.orchestra.run.smithi096.stdout: "in": 1, 2024-05-28T06:23:36.701 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 1, 2024-05-28T06:23:36.701 INFO:teuthology.orchestra.run.smithi096.stdout: "by_rank": [ 2024-05-28T06:23:36.701 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:23:36.702 INFO:teuthology.orchestra.run.smithi096.stdout: "filesystem_id": 8, 2024-05-28T06:23:36.702 INFO:teuthology.orchestra.run.smithi096.stdout: "rank": 0, 2024-05-28T06:23:36.702 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "b", 2024-05-28T06:23:36.702 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "up:active", 2024-05-28T06:23:36.702 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 9270 2024-05-28T06:23:36.702 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:23:36.702 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:23:36.702 INFO:teuthology.orchestra.run.smithi096.stdout: "up:standby": 3 2024-05-28T06:23:36.702 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:23:36.702 INFO:teuthology.orchestra.run.smithi096.stdout: "mgrmap": { 2024-05-28T06:23:36.702 INFO:teuthology.orchestra.run.smithi096.stdout: "available": true, 2024-05-28T06:23:36.703 INFO:teuthology.orchestra.run.smithi096.stdout: "num_standbys": 1, 2024-05-28T06:23:36.703 INFO:teuthology.orchestra.run.smithi096.stdout: "modules": [ 2024-05-28T06:23:36.703 INFO:teuthology.orchestra.run.smithi096.stdout: "iostat", 2024-05-28T06:23:36.703 INFO:teuthology.orchestra.run.smithi096.stdout: "nfs", 2024-05-28T06:23:36.703 INFO:teuthology.orchestra.run.smithi096.stdout: "restful" 2024-05-28T06:23:36.703 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:23:36.703 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:23:36.703 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:23:36.703 INFO:teuthology.orchestra.run.smithi096.stdout: "servicemap": { 2024-05-28T06:23:36.703 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 23, 2024-05-28T06:23:36.703 INFO:teuthology.orchestra.run.smithi096.stdout: "modified": "2024-05-28T06:23:23.993668+0000", 2024-05-28T06:23:36.703 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:23:36.704 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:23:36.704 INFO:teuthology.orchestra.run.smithi096.stdout: "progress_events": { 2024-05-28T06:23:36.704 INFO:teuthology.orchestra.run.smithi096.stdout: "d6fb006f-47c2-4c80-b874-17506401c2f6": { 2024-05-28T06:23:36.704 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "All cloning finished, waiting 20.0 seconds before exiting (16s)\n [============================] ", 2024-05-28T06:23:36.704 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": 1, 2024-05-28T06:23:36.704 INFO:teuthology.orchestra.run.smithi096.stdout: "add_to_ceph_s": true 2024-05-28T06:23:36.704 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:23:36.704 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:23:36.704 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:23:36.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:36.704+0000 7ffa38c00640 1 -- 172.21.15.96:0/1733450260 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ffa1406e3f0 msgr2=0x7ffa140708b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:36.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:36.704+0000 7ffa38c00640 1 --2- 172.21.15.96:0/1733450260 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ffa1406e3f0 0x7ffa140708b0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7ffa3415bb30 tx=0x7ffa20008040 comp rx=0 tx=0).stop 2024-05-28T06:23:36.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:36.704+0000 7ffa38c00640 1 -- 172.21.15.96:0/1733450260 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa340ccc30 msgr2=0x7ffa341530d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:36.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:36.704+0000 7ffa38c00640 1 --2- 172.21.15.96:0/1733450260 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa340ccc30 0x7ffa341530d0 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7ffa2400b3f0 tx=0x7ffa2400b8b0 comp rx=0 tx=0).stop 2024-05-28T06:23:36.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:36.704+0000 7ffa38c00640 1 -- 172.21.15.96:0/1733450260 shutdown_connections 2024-05-28T06:23:36.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:36.704+0000 7ffa38c00640 1 --2- 172.21.15.96:0/1733450260 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ffa1406e3f0 0x7ffa140708b0 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:36.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:36.704+0000 7ffa38c00640 1 --2- 172.21.15.96:0/1733450260 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffa3415c6b0 0x7ffa34099800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:36.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:36.704+0000 7ffa38c00640 1 --2- 172.21.15.96:0/1733450260 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa340d0460 0x7ffa340b07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:36.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:36.704+0000 7ffa38c00640 1 --2- 172.21.15.96:0/1733450260 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa340ccc30 0x7ffa341530d0 unknown :-1 s=CLOSED pgs=1884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:36.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:36.704+0000 7ffa38c00640 1 -- 172.21.15.96:0/1733450260 >> 172.21.15.96:0/1733450260 conn(0x7ffa3406bae0 msgr2=0x7ffa3406bec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:23:36.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:36.705+0000 7ffa38c00640 1 -- 172.21.15.96:0/1733450260 shutdown_connections 2024-05-28T06:23:36.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:36.705+0000 7ffa38c00640 1 -- 172.21.15.96:0/1733450260 wait complete. 2024-05-28T06:23:41.720 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-05-28T06:23:41.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:41.935+0000 7f1a35400640 1 Processor -- start 2024-05-28T06:23:41.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:41.935+0000 7f1a35400640 1 -- start start 2024-05-28T06:23:41.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:41.936+0000 7f1a35400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f1a3015ef60 0x7f1a300ce800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:41.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:41.936+0000 7f1a35400640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a3015fe50 con 0x7f1a300d0460 2024-05-28T06:23:41.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:41.936+0000 7f1a35400640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a3015f5b0 con 0x7f1a3015b380 2024-05-28T06:23:41.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:41.936+0000 7f1a35400640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f1a300b3310 con 0x7f1a3015ef60 2024-05-28T06:23:41.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:41.937+0000 7f1a34a00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f1a300d0460 0x7f1a300b37a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:34374/0 (socket says 172.21.15.96:34374) 2024-05-28T06:23:41.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:41.937+0000 7f1a34a00640 1 -- 172.21.15.96:0/3947786896 learned_addr learned my addr 172.21.15.96:0/3947786896 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:23:41.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:41.937+0000 7f1a27400640 1 --2- 172.21.15.96:0/3947786896 >> v2:172.21.15.123:3300/0 conn(0x7f1a3015ef60 0x7f1a300ce800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:41.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:41.937+0000 7f1a2f400640 1 -- 172.21.15.96:0/3947786896 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2935317012 0 0) 0x7f1a3015fe50 con 0x7f1a300d0460 2024-05-28T06:23:41.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:41.937+0000 7f1a2f400640 1 -- 172.21.15.96:0/3947786896 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1a14003520 con 0x7f1a300d0460 2024-05-28T06:23:41.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:41.937+0000 7f1a2f400640 1 -- 172.21.15.96:0/3947786896 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 886890790 0 0) 0x7f1a14003520 con 0x7f1a300d0460 2024-05-28T06:23:41.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:41.937+0000 7f1a2f400640 1 -- 172.21.15.96:0/3947786896 >> v2:172.21.15.123:3300/0 conn(0x7f1a3015ef60 msgr2=0x7f1a300ce800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:23:41.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:41.937+0000 7f1a2f400640 1 --2- 172.21.15.96:0/3947786896 >> v2:172.21.15.123:3300/0 conn(0x7f1a3015ef60 0x7f1a300ce800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:41.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:41.937+0000 7f1a2f400640 1 -- 172.21.15.96:0/3947786896 >> v1:172.21.15.123:6789/0 conn(0x7f1a3015b380 legacy=0x7f1a300ad130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:41.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:41.937+0000 7f1a2f400640 1 -- 172.21.15.96:0/3947786896 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a3015f890 con 0x7f1a300d0460 2024-05-28T06:23:41.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:41.938+0000 7f1a2f400640 1 -- 172.21.15.96:0/3947786896 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f1a280019f0 con 0x7f1a300d0460 2024-05-28T06:23:41.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:41.938+0000 7f1a2f400640 1 -- 172.21.15.96:0/3947786896 >> v1:172.21.15.96:6789/0 conn(0x7f1a300d0460 legacy=0x7f1a300b37a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:41.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:41.938+0000 7f1a2f400640 1 --2- 172.21.15.96:0/3947786896 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a140039b0 0x7f1a14003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:41.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:41.938+0000 7f1a2f400640 1 --2- 172.21.15.96:0/3947786896 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1a14004800 0x7f1a14004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:41.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:41.938+0000 7f1a2f400640 1 --2- 172.21.15.96:0/3947786896 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a140053c0 0x7f1a1400dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:41.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:41.938+0000 7f1a2f400640 1 -- 172.21.15.96:0/3947786896 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f1a1400e430 con 0x7f1a140039b0 2024-05-28T06:23:41.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:41.938+0000 7f1a2f400640 1 -- 172.21.15.96:0/3947786896 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f1a1400e5e0 con 0x7f1a140053c0 2024-05-28T06:23:41.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:41.938+0000 7f1a2f400640 1 -- 172.21.15.96:0/3947786896 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f1a1400e7e0 con 0x7f1a14004800 2024-05-28T06:23:41.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:41.938+0000 7f1a2fe00640 1 --2- 172.21.15.96:0/3947786896 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a140039b0 0x7f1a14003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:41.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:41.938+0000 7f1a27400640 1 --2- 172.21.15.96:0/3947786896 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1a14004800 0x7f1a14004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:41.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:41.938+0000 7f1a34a00640 1 --2- 172.21.15.96:0/3947786896 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a140053c0 0x7f1a1400dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:41.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:41.938+0000 7f1a2fe00640 1 -- 172.21.15.96:0/3947786896 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1a14004800 msgr2=0x7f1a14004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:23:41.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:41.938+0000 7f1a2fe00640 1 --2- 172.21.15.96:0/3947786896 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1a14004800 0x7f1a14004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:41.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:41.938+0000 7f1a2fe00640 1 -- 172.21.15.96:0/3947786896 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a140053c0 msgr2=0x7f1a1400dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:23:41.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:41.938+0000 7f1a2fe00640 1 --2- 172.21.15.96:0/3947786896 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a140053c0 0x7f1a1400dbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:41.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:41.938+0000 7f1a34a00640 1 -- 172.21.15.96:0/3947786896 reap_dead start 2024-05-28T06:23:41.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:41.938+0000 7f1a2fe00640 1 -- 172.21.15.96:0/3947786896 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a14003520 con 0x7f1a140039b0 2024-05-28T06:23:41.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:41.939+0000 7f1a2fe00640 1 --2- 172.21.15.96:0/3947786896 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a140039b0 0x7f1a14003dd0 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7f1a20001750 tx=0x7f1a14005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc8bac1ef9766c8d server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:23:41.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:41.940+0000 7f1a2f400640 1 -- 172.21.15.96:0/3947786896 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a20001370 con 0x7f1a140039b0 2024-05-28T06:23:41.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:41.940+0000 7f1a2f400640 1 -- 172.21.15.96:0/3947786896 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a200052d0 con 0x7f1a140039b0 2024-05-28T06:23:41.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:41.940+0000 7f1a2f400640 1 -- 172.21.15.96:0/3947786896 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a2002abc0 con 0x7f1a140039b0 2024-05-28T06:23:41.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:41.940+0000 7f1a35400640 1 -- 172.21.15.96:0/3947786896 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a140039b0 msgr2=0x7f1a14003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:41.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:41.940+0000 7f1a35400640 1 --2- 172.21.15.96:0/3947786896 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a140039b0 0x7f1a14003dd0 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7f1a20001750 tx=0x7f1a14005830 comp rx=0 tx=0).stop 2024-05-28T06:23:41.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:41.940+0000 7f1a35400640 1 -- 172.21.15.96:0/3947786896 shutdown_connections 2024-05-28T06:23:41.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:41.940+0000 7f1a35400640 1 --2- 172.21.15.96:0/3947786896 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a140039b0 0x7f1a14003dd0 unknown :-1 s=CLOSED pgs=1886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:41.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:41.940+0000 7f1a35400640 1 -- 172.21.15.96:0/3947786896 >> 172.21.15.96:0/3947786896 conn(0x7f1a3006bae0 msgr2=0x7f1a3006bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:23:41.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:41.940+0000 7f1a35400640 1 -- 172.21.15.96:0/3947786896 shutdown_connections 2024-05-28T06:23:41.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:41.941+0000 7f1a35400640 1 -- 172.21.15.96:0/3947786896 wait complete. 2024-05-28T06:23:41.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:41.942+0000 7f1a35400640 1 Processor -- start 2024-05-28T06:23:41.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:41.942+0000 7f1a35400640 1 -- start start 2024-05-28T06:23:41.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:41.943+0000 7f1a35400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a140039b0 0x7f1a30099a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:41.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:41.943+0000 7f1a35400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a30156b70 0x7f1a30156fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:41.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:41.944+0000 7f1a35400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1a30141550 0x7f1a301419d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:41.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:41.944+0000 7f1a35400640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f1a3015b8e0 con 0x7f1a140039b0 2024-05-28T06:23:41.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:41.944+0000 7f1a35400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f1a3015b760 con 0x7f1a30156b70 2024-05-28T06:23:41.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:41.944+0000 7f1a35400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f1a300ce800 con 0x7f1a30141550 2024-05-28T06:23:41.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:41.944+0000 7f1a2fe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a140039b0 0x7f1a30099a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:41.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:41.944+0000 7f1a2fe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a140039b0 0x7f1a30099a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:33180/0 (socket says 172.21.15.96:33180) 2024-05-28T06:23:41.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:41.944+0000 7f1a2fe00640 1 -- 172.21.15.96:0/2047318381 learned_addr learned my addr 172.21.15.96:0/2047318381 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:23:41.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:41.944+0000 7f1a2fe00640 1 -- 172.21.15.96:0/2047318381 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1a30141550 msgr2=0x7f1a301419d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:41.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:41.944+0000 7f1a34a00640 1 --2- 172.21.15.96:0/2047318381 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1a30141550 0x7f1a301419d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:41.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:41.944+0000 7f1a27400640 1 --2- 172.21.15.96:0/2047318381 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a30156b70 0x7f1a30156fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:41.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:41.944+0000 7f1a2fe00640 1 --2- 172.21.15.96:0/2047318381 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1a30141550 0x7f1a301419d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:41.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:41.944+0000 7f1a2fe00640 1 -- 172.21.15.96:0/2047318381 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a30156b70 msgr2=0x7f1a30156fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:41.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:41.944+0000 7f1a2fe00640 1 --2- 172.21.15.96:0/2047318381 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a30156b70 0x7f1a30156fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:41.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:41.944+0000 7f1a2fe00640 1 -- 172.21.15.96:0/2047318381 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a3013f640 con 0x7f1a140039b0 2024-05-28T06:23:41.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:41.945+0000 7f1a2fe00640 1 --2- 172.21.15.96:0/2047318381 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a140039b0 0x7f1a30099a10 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7f1a20005040 tx=0x7f1a20001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:23:41.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:41.945+0000 7f1a2cc00640 1 -- 172.21.15.96:0/2047318381 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a20037040 con 0x7f1a140039b0 2024-05-28T06:23:41.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:41.945+0000 7f1a2cc00640 1 -- 172.21.15.96:0/2047318381 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a20040070 con 0x7f1a140039b0 2024-05-28T06:23:41.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:41.945+0000 7f1a2cc00640 1 -- 172.21.15.96:0/2047318381 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a20007a30 con 0x7f1a140039b0 2024-05-28T06:23:41.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:41.945+0000 7f1a35400640 1 -- 172.21.15.96:0/2047318381 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a30142280 con 0x7f1a140039b0 2024-05-28T06:23:41.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:41.945+0000 7f1a35400640 1 -- 172.21.15.96:0/2047318381 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a30141f10 con 0x7f1a140039b0 2024-05-28T06:23:41.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:41.946+0000 7f1a35400640 1 -- 172.21.15.96:0/2047318381 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a300689e0 con 0x7f1a140039b0 2024-05-28T06:23:41.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:41.947+0000 7f1a2cc00640 1 -- 172.21.15.96:0/2047318381 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 41) ==== 93488+0+0 (secure 0 0 0) 0x7f1a200052d0 con 0x7f1a140039b0 2024-05-28T06:23:41.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:41.948+0000 7f1a2cc00640 1 --2- 172.21.15.96:0/2047318381 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1a0c06e3f0 0x7f1a0c0708b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:41.943 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:41.948+0000 7f1a2cc00640 1 -- 172.21.15.96:0/2047318381 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 5823+0+0 (secure 0 0 0) 0x7f1a200aa820 con 0x7f1a140039b0 2024-05-28T06:23:41.943 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:41.949+0000 7f1a27400640 1 --2- 172.21.15.96:0/2047318381 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1a0c06e3f0 0x7f1a0c0708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:41.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:41.950+0000 7f1a27400640 1 --2- 172.21.15.96:0/2047318381 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1a0c06e3f0 0x7f1a0c0708b0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f1a3013f390 tx=0x7f1a18008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:23:41.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:41.953+0000 7f1a2cc00640 1 -- 172.21.15.96:0/2047318381 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1a2007a620 con 0x7f1a140039b0 2024-05-28T06:23:42.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:42.229+0000 7f1a35400640 1 -- 172.21.15.96:0/2047318381 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f1a300b4fe0 con 0x7f1a140039b0 2024-05-28T06:23:42.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:42.231+0000 7f1a2cc00640 1 -- 172.21.15.96:0/2047318381 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+2081 (secure 0 0 0) 0x7f1a20079f20 con 0x7f1a140039b0 2024-05-28T06:23:42.226 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:23:42.226 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:23:42.226 INFO:teuthology.orchestra.run.smithi096.stdout: "fsid": "e4a831b5-e6d1-47a7-96cb-bf85154ee0d2", 2024-05-28T06:23:42.226 INFO:teuthology.orchestra.run.smithi096.stdout: "health": { 2024-05-28T06:23:42.226 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "HEALTH_OK", 2024-05-28T06:23:42.226 INFO:teuthology.orchestra.run.smithi096.stdout: "checks": {}, 2024-05-28T06:23:42.226 INFO:teuthology.orchestra.run.smithi096.stdout: "mutes": [] 2024-05-28T06:23:42.226 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:23:42.226 INFO:teuthology.orchestra.run.smithi096.stdout: "election_epoch": 8, 2024-05-28T06:23:42.226 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum": [ 2024-05-28T06:23:42.226 INFO:teuthology.orchestra.run.smithi096.stdout: 0, 2024-05-28T06:23:42.227 INFO:teuthology.orchestra.run.smithi096.stdout: 1, 2024-05-28T06:23:42.227 INFO:teuthology.orchestra.run.smithi096.stdout: 2 2024-05-28T06:23:42.227 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:23:42.227 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_names": [ 2024-05-28T06:23:42.227 INFO:teuthology.orchestra.run.smithi096.stdout: "a", 2024-05-28T06:23:42.227 INFO:teuthology.orchestra.run.smithi096.stdout: "b", 2024-05-28T06:23:42.227 INFO:teuthology.orchestra.run.smithi096.stdout: "c" 2024-05-28T06:23:42.227 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:23:42.227 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_age": 1195, 2024-05-28T06:23:42.227 INFO:teuthology.orchestra.run.smithi096.stdout: "monmap": { 2024-05-28T06:23:42.227 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 1, 2024-05-28T06:23:42.227 INFO:teuthology.orchestra.run.smithi096.stdout: "min_mon_release_name": "squid", 2024-05-28T06:23:42.227 INFO:teuthology.orchestra.run.smithi096.stdout: "num_mons": 3 2024-05-28T06:23:42.227 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:23:42.228 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap": { 2024-05-28T06:23:42.228 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 117, 2024-05-28T06:23:42.228 INFO:teuthology.orchestra.run.smithi096.stdout: "num_osds": 8, 2024-05-28T06:23:42.228 INFO:teuthology.orchestra.run.smithi096.stdout: "num_up_osds": 8, 2024-05-28T06:23:42.228 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_up_since": 1716876238, 2024-05-28T06:23:42.228 INFO:teuthology.orchestra.run.smithi096.stdout: "num_in_osds": 8, 2024-05-28T06:23:42.228 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_in_since": 1716876229, 2024-05-28T06:23:42.228 INFO:teuthology.orchestra.run.smithi096.stdout: "num_remapped_pgs": 0 2024-05-28T06:23:42.228 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:23:42.228 INFO:teuthology.orchestra.run.smithi096.stdout: "pgmap": { 2024-05-28T06:23:42.228 INFO:teuthology.orchestra.run.smithi096.stdout: "pgs_by_state": [ 2024-05-28T06:23:42.228 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:23:42.228 INFO:teuthology.orchestra.run.smithi096.stdout: "state_name": "active+clean", 2024-05-28T06:23:42.228 INFO:teuthology.orchestra.run.smithi096.stdout: "count": 97 2024-05-28T06:23:42.229 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:23:42.229 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:23:42.229 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pgs": 97, 2024-05-28T06:23:42.229 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pools": 3, 2024-05-28T06:23:42.229 INFO:teuthology.orchestra.run.smithi096.stdout: "num_objects": 1745, 2024-05-28T06:23:42.229 INFO:teuthology.orchestra.run.smithi096.stdout: "data_bytes": 7190389093, 2024-05-28T06:23:42.229 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_used": 15424970752, 2024-05-28T06:23:42.229 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_avail": 757669142528, 2024-05-28T06:23:42.229 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_total": 773094113280, 2024-05-28T06:23:42.229 INFO:teuthology.orchestra.run.smithi096.stdout: "write_bytes_sec": 4606, 2024-05-28T06:23:42.229 INFO:teuthology.orchestra.run.smithi096.stdout: "read_op_per_sec": 0, 2024-05-28T06:23:42.229 INFO:teuthology.orchestra.run.smithi096.stdout: "write_op_per_sec": 1 2024-05-28T06:23:42.229 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:23:42.229 INFO:teuthology.orchestra.run.smithi096.stdout: "fsmap": { 2024-05-28T06:23:42.230 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 73, 2024-05-28T06:23:42.230 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 8, 2024-05-28T06:23:42.230 INFO:teuthology.orchestra.run.smithi096.stdout: "up": 1, 2024-05-28T06:23:42.230 INFO:teuthology.orchestra.run.smithi096.stdout: "in": 1, 2024-05-28T06:23:42.230 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 1, 2024-05-28T06:23:42.230 INFO:teuthology.orchestra.run.smithi096.stdout: "by_rank": [ 2024-05-28T06:23:42.230 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:23:42.230 INFO:teuthology.orchestra.run.smithi096.stdout: "filesystem_id": 8, 2024-05-28T06:23:42.230 INFO:teuthology.orchestra.run.smithi096.stdout: "rank": 0, 2024-05-28T06:23:42.230 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "b", 2024-05-28T06:23:42.230 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "up:active", 2024-05-28T06:23:42.230 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 9270 2024-05-28T06:23:42.230 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:23:42.231 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:23:42.231 INFO:teuthology.orchestra.run.smithi096.stdout: "up:standby": 3 2024-05-28T06:23:42.231 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:23:42.231 INFO:teuthology.orchestra.run.smithi096.stdout: "mgrmap": { 2024-05-28T06:23:42.231 INFO:teuthology.orchestra.run.smithi096.stdout: "available": true, 2024-05-28T06:23:42.231 INFO:teuthology.orchestra.run.smithi096.stdout: "num_standbys": 1, 2024-05-28T06:23:42.231 INFO:teuthology.orchestra.run.smithi096.stdout: "modules": [ 2024-05-28T06:23:42.231 INFO:teuthology.orchestra.run.smithi096.stdout: "iostat", 2024-05-28T06:23:42.231 INFO:teuthology.orchestra.run.smithi096.stdout: "nfs", 2024-05-28T06:23:42.231 INFO:teuthology.orchestra.run.smithi096.stdout: "restful" 2024-05-28T06:23:42.231 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:23:42.231 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:23:42.231 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:23:42.231 INFO:teuthology.orchestra.run.smithi096.stdout: "servicemap": { 2024-05-28T06:23:42.232 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 23, 2024-05-28T06:23:42.232 INFO:teuthology.orchestra.run.smithi096.stdout: "modified": "2024-05-28T06:23:23.993668+0000", 2024-05-28T06:23:42.232 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:23:42.232 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:23:42.232 INFO:teuthology.orchestra.run.smithi096.stdout: "progress_events": { 2024-05-28T06:23:42.232 INFO:teuthology.orchestra.run.smithi096.stdout: "d6fb006f-47c2-4c80-b874-17506401c2f6": { 2024-05-28T06:23:42.232 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "All cloning finished, waiting 14.0 seconds before exiting (22s)\n [============================] ", 2024-05-28T06:23:42.232 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": 1, 2024-05-28T06:23:42.232 INFO:teuthology.orchestra.run.smithi096.stdout: "add_to_ceph_s": true 2024-05-28T06:23:42.232 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:23:42.232 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:23:42.232 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:23:42.233 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:42.233+0000 7f1a35400640 1 -- 172.21.15.96:0/2047318381 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1a0c06e3f0 msgr2=0x7f1a0c0708b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:42.233 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:42.233+0000 7f1a35400640 1 --2- 172.21.15.96:0/2047318381 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1a0c06e3f0 0x7f1a0c0708b0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f1a3013f390 tx=0x7f1a18008040 comp rx=0 tx=0).stop 2024-05-28T06:23:42.233 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:42.233+0000 7f1a35400640 1 -- 172.21.15.96:0/2047318381 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a140039b0 msgr2=0x7f1a30099a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:42.233 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:42.233+0000 7f1a35400640 1 --2- 172.21.15.96:0/2047318381 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a140039b0 0x7f1a30099a10 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7f1a20005040 tx=0x7f1a20001120 comp rx=0 tx=0).stop 2024-05-28T06:23:42.233 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:42.233+0000 7f1a35400640 1 -- 172.21.15.96:0/2047318381 shutdown_connections 2024-05-28T06:23:42.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:42.233+0000 7f1a35400640 1 --2- 172.21.15.96:0/2047318381 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1a0c06e3f0 0x7f1a0c0708b0 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:42.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:42.233+0000 7f1a35400640 1 --2- 172.21.15.96:0/2047318381 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1a30141550 0x7f1a301419d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:42.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:42.233+0000 7f1a35400640 1 --2- 172.21.15.96:0/2047318381 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a30156b70 0x7f1a30156fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:42.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:42.233+0000 7f1a35400640 1 --2- 172.21.15.96:0/2047318381 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a140039b0 0x7f1a30099a10 unknown :-1 s=CLOSED pgs=1887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:42.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:42.233+0000 7f1a35400640 1 -- 172.21.15.96:0/2047318381 >> 172.21.15.96:0/2047318381 conn(0x7f1a3006bae0 msgr2=0x7f1a300ad280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:23:42.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:42.233+0000 7f1a35400640 1 -- 172.21.15.96:0/2047318381 shutdown_connections 2024-05-28T06:23:42.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:42.233+0000 7f1a35400640 1 -- 172.21.15.96:0/2047318381 wait complete. 2024-05-28T06:23:47.249 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-05-28T06:23:47.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:47.465+0000 7fddff200640 1 Processor -- start 2024-05-28T06:23:47.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:47.466+0000 7fddff200640 1 -- start start 2024-05-28T06:23:47.464 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:47.467+0000 7fddff200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fde000ccc30 0x7fde0006dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:47.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:47.467+0000 7fddff200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde000cd1a0 con 0x7fde000d0460 2024-05-28T06:23:47.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:47.467+0000 7fddff200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde000cd480 con 0x7fde0015c6b0 2024-05-28T06:23:47.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:47.467+0000 7fddff200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fde000cd760 con 0x7fde000ccc30 2024-05-28T06:23:47.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:47.467+0000 7fddf5400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fde000ccc30 0x7fde0006dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:47.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:47.467+0000 7fddfe800640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fde000d0460 0x7fde000cbf30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:34388/0 (socket says 172.21.15.96:34388) 2024-05-28T06:23:47.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:47.467+0000 7fddfe800640 1 -- 172.21.15.96:0/35866645 learned_addr learned my addr 172.21.15.96:0/35866645 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:23:47.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:47.468+0000 7fddf5400640 1 -- 172.21.15.96:0/35866645 >> v1:172.21.15.123:6789/0 conn(0x7fde0015c6b0 legacy=0x7fde000b3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:23:47.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:47.468+0000 7fddf5400640 1 -- 172.21.15.96:0/35866645 >> v1:172.21.15.96:6789/0 conn(0x7fde000d0460 legacy=0x7fde000cbf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:47.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:47.468+0000 7fddf5400640 1 -- 172.21.15.96:0/35866645 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde000cda40 con 0x7fde000ccc30 2024-05-28T06:23:47.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:47.468+0000 7fddfd400640 1 -- 172.21.15.96:0/35866645 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2113940107 0 0) 0x7fde000cd1a0 con 0x7fde000d0460 2024-05-28T06:23:47.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:47.469+0000 7fddf5400640 1 --2- 172.21.15.96:0/35866645 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fde000ccc30 0x7fde0006dc90 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7fdde80097d0 tx=0x7fdde802a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62286c618f26e59d server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:23:47.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:47.469+0000 7fddfd400640 1 -- 172.21.15.96:0/35866645 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdde8037070 con 0x7fde000ccc30 2024-05-28T06:23:47.467 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:47.469+0000 7fddfd400640 1 -- 172.21.15.96:0/35866645 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdde8032440 con 0x7fde000ccc30 2024-05-28T06:23:47.473 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:47.469+0000 7fddfd400640 1 -- 172.21.15.96:0/35866645 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdde8032810 con 0x7fde000ccc30 2024-05-28T06:23:47.473 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:47.470+0000 7fddff200640 1 -- 172.21.15.96:0/35866645 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fde000ccc30 msgr2=0x7fde0006dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:47.474 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:47.470+0000 7fddff200640 1 --2- 172.21.15.96:0/35866645 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fde000ccc30 0x7fde0006dc90 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7fdde80097d0 tx=0x7fdde802a210 comp rx=0 tx=0).stop 2024-05-28T06:23:47.474 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:47.470+0000 7fddff200640 1 -- 172.21.15.96:0/35866645 shutdown_connections 2024-05-28T06:23:47.474 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:47.470+0000 7fddff200640 1 --2- 172.21.15.96:0/35866645 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fde000ccc30 0x7fde0006dc90 unknown :-1 s=CLOSED pgs=1889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:47.474 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:47.470+0000 7fddff200640 1 -- 172.21.15.96:0/35866645 >> 172.21.15.96:0/35866645 conn(0x7fde0006bae0 msgr2=0x7fde0006bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:23:47.474 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:47.470+0000 7fddff200640 1 -- 172.21.15.96:0/35866645 shutdown_connections 2024-05-28T06:23:47.474 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:47.470+0000 7fddff200640 1 -- 172.21.15.96:0/35866645 wait complete. 2024-05-28T06:23:47.475 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:47.472+0000 7fddff200640 1 Processor -- start 2024-05-28T06:23:47.475 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:47.472+0000 7fddff200640 1 -- start start 2024-05-28T06:23:47.475 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:47.473+0000 7fddff200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fde000ccc30 0x7fde000b21d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:47.475 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:47.473+0000 7fddff200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fde000d0460 0x7fde000afdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:47.475 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:47.473+0000 7fddff200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde0015c6b0 0x7fde000ae180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:47.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:47.473+0000 7fddff200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fde0006e350 con 0x7fde000d0460 2024-05-28T06:23:47.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:47.473+0000 7fddff200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fde0006e1d0 con 0x7fde0015c6b0 2024-05-28T06:23:47.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:47.473+0000 7fddff200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fde000cc640 con 0x7fde000ccc30 2024-05-28T06:23:47.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:47.473+0000 7fddf5400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fde000d0460 0x7fde000afdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:47.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:47.473+0000 7fddf5400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fde000d0460 0x7fde000afdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:33204/0 (socket says 172.21.15.96:33204) 2024-05-28T06:23:47.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:47.473+0000 7fddf5400640 1 -- 172.21.15.96:0/2733328726 learned_addr learned my addr 172.21.15.96:0/2733328726 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:23:47.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:47.474+0000 7fddf5400640 1 -- 172.21.15.96:0/2733328726 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fde000ccc30 msgr2=0x7fde000b21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:47.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:47.474+0000 7fddfe800640 1 --2- 172.21.15.96:0/2733328726 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde0015c6b0 0x7fde000ae180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:47.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:47.474+0000 7fddfde00640 1 --2- 172.21.15.96:0/2733328726 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fde000ccc30 0x7fde000b21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:47.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:47.474+0000 7fddf5400640 1 --2- 172.21.15.96:0/2733328726 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fde000ccc30 0x7fde000b21d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:47.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:47.474+0000 7fddf5400640 1 -- 172.21.15.96:0/2733328726 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde0015c6b0 msgr2=0x7fde000ae180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:47.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:47.474+0000 7fddf5400640 1 --2- 172.21.15.96:0/2733328726 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde0015c6b0 0x7fde000ae180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:47.478 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:47.474+0000 7fddf5400640 1 -- 172.21.15.96:0/2733328726 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde000b06c0 con 0x7fde000d0460 2024-05-28T06:23:47.478 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:47.474+0000 7fddfde00640 1 --2- 172.21.15.96:0/2733328726 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fde000ccc30 0x7fde000b21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:23:47.478 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:47.474+0000 7fddfe800640 1 --2- 172.21.15.96:0/2733328726 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde0015c6b0 0x7fde000ae180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:23:47.478 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:47.474+0000 7fddf5400640 1 --2- 172.21.15.96:0/2733328726 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fde000d0460 0x7fde000afdc0 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7fdde8009720 tx=0x7fdde80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:23:47.478 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:47.474+0000 7fddf6a00640 1 -- 172.21.15.96:0/2733328726 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdde8037040 con 0x7fde000d0460 2024-05-28T06:23:47.478 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:47.474+0000 7fddf6a00640 1 -- 172.21.15.96:0/2733328726 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdde8004220 con 0x7fde000d0460 2024-05-28T06:23:47.479 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:47.474+0000 7fddff200640 1 -- 172.21.15.96:0/2733328726 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde000b09a0 con 0x7fde000d0460 2024-05-28T06:23:47.479 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:47.474+0000 7fddf6a00640 1 -- 172.21.15.96:0/2733328726 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdde803c730 con 0x7fde000d0460 2024-05-28T06:23:47.479 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:47.475+0000 7fddff200640 1 -- 172.21.15.96:0/2733328726 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde000b0f60 con 0x7fde000d0460 2024-05-28T06:23:47.479 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:47.476+0000 7fddff200640 1 -- 172.21.15.96:0/2733328726 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde0006dc90 con 0x7fde000d0460 2024-05-28T06:23:47.479 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:47.477+0000 7fddf6a00640 1 -- 172.21.15.96:0/2733328726 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 41) ==== 93488+0+0 (secure 0 0 0) 0x7fdde803a070 con 0x7fde000d0460 2024-05-28T06:23:47.479 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:47.478+0000 7fddf6a00640 1 --2- 172.21.15.96:0/2733328726 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdddc06e3f0 0x7fdddc0708b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:47.480 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:47.478+0000 7fddf6a00640 1 -- 172.21.15.96:0/2733328726 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 5823+0+0 (secure 0 0 0) 0x7fdde80ab3f0 con 0x7fde000d0460 2024-05-28T06:23:47.480 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:47.478+0000 7fddfde00640 1 --2- 172.21.15.96:0/2733328726 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdddc06e3f0 0x7fdddc0708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:47.481 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:47.480+0000 7fddfde00640 1 --2- 172.21.15.96:0/2733328726 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdddc06e3f0 0x7fdddc0708b0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fde000cf0d0 tx=0x7fddf0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:23:47.481 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:47.484+0000 7fddf6a00640 1 -- 172.21.15.96:0/2733328726 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fde0006dc90 con 0x7fde000d0460 2024-05-28T06:23:47.744 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:47.746+0000 7fddff200640 1 -- 172.21.15.96:0/2733328726 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fde000689e0 con 0x7fde000d0460 2024-05-28T06:23:47.744 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:47.747+0000 7fddf6a00640 1 -- 172.21.15.96:0/2733328726 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+2081 (secure 0 0 0) 0x7fdde804a030 con 0x7fde000d0460 2024-05-28T06:23:47.744 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:23:47.745 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:23:47.745 INFO:teuthology.orchestra.run.smithi096.stdout: "fsid": "e4a831b5-e6d1-47a7-96cb-bf85154ee0d2", 2024-05-28T06:23:47.745 INFO:teuthology.orchestra.run.smithi096.stdout: "health": { 2024-05-28T06:23:47.745 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "HEALTH_OK", 2024-05-28T06:23:47.745 INFO:teuthology.orchestra.run.smithi096.stdout: "checks": {}, 2024-05-28T06:23:47.745 INFO:teuthology.orchestra.run.smithi096.stdout: "mutes": [] 2024-05-28T06:23:47.745 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:23:47.745 INFO:teuthology.orchestra.run.smithi096.stdout: "election_epoch": 8, 2024-05-28T06:23:47.745 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum": [ 2024-05-28T06:23:47.745 INFO:teuthology.orchestra.run.smithi096.stdout: 0, 2024-05-28T06:23:47.745 INFO:teuthology.orchestra.run.smithi096.stdout: 1, 2024-05-28T06:23:47.745 INFO:teuthology.orchestra.run.smithi096.stdout: 2 2024-05-28T06:23:47.745 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:23:47.746 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_names": [ 2024-05-28T06:23:47.746 INFO:teuthology.orchestra.run.smithi096.stdout: "a", 2024-05-28T06:23:47.746 INFO:teuthology.orchestra.run.smithi096.stdout: "b", 2024-05-28T06:23:47.746 INFO:teuthology.orchestra.run.smithi096.stdout: "c" 2024-05-28T06:23:47.746 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:23:47.746 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_age": 1201, 2024-05-28T06:23:47.746 INFO:teuthology.orchestra.run.smithi096.stdout: "monmap": { 2024-05-28T06:23:47.746 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 1, 2024-05-28T06:23:47.746 INFO:teuthology.orchestra.run.smithi096.stdout: "min_mon_release_name": "squid", 2024-05-28T06:23:47.746 INFO:teuthology.orchestra.run.smithi096.stdout: "num_mons": 3 2024-05-28T06:23:47.746 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:23:47.746 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap": { 2024-05-28T06:23:47.747 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 117, 2024-05-28T06:23:47.747 INFO:teuthology.orchestra.run.smithi096.stdout: "num_osds": 8, 2024-05-28T06:23:47.747 INFO:teuthology.orchestra.run.smithi096.stdout: "num_up_osds": 8, 2024-05-28T06:23:47.747 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_up_since": 1716876238, 2024-05-28T06:23:47.747 INFO:teuthology.orchestra.run.smithi096.stdout: "num_in_osds": 8, 2024-05-28T06:23:47.747 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_in_since": 1716876229, 2024-05-28T06:23:47.747 INFO:teuthology.orchestra.run.smithi096.stdout: "num_remapped_pgs": 0 2024-05-28T06:23:47.747 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:23:47.747 INFO:teuthology.orchestra.run.smithi096.stdout: "pgmap": { 2024-05-28T06:23:47.747 INFO:teuthology.orchestra.run.smithi096.stdout: "pgs_by_state": [ 2024-05-28T06:23:47.748 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:23:47.748 INFO:teuthology.orchestra.run.smithi096.stdout: "state_name": "active+clean", 2024-05-28T06:23:47.748 INFO:teuthology.orchestra.run.smithi096.stdout: "count": 97 2024-05-28T06:23:47.748 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:23:47.748 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:23:47.748 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pgs": 97, 2024-05-28T06:23:47.748 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pools": 3, 2024-05-28T06:23:47.748 INFO:teuthology.orchestra.run.smithi096.stdout: "num_objects": 1745, 2024-05-28T06:23:47.748 INFO:teuthology.orchestra.run.smithi096.stdout: "data_bytes": 7190389093, 2024-05-28T06:23:47.748 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_used": 15424970752, 2024-05-28T06:23:47.749 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_avail": 757669142528, 2024-05-28T06:23:47.749 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_total": 773094113280, 2024-05-28T06:23:47.749 INFO:teuthology.orchestra.run.smithi096.stdout: "write_bytes_sec": 4520, 2024-05-28T06:23:47.749 INFO:teuthology.orchestra.run.smithi096.stdout: "read_op_per_sec": 0, 2024-05-28T06:23:47.749 INFO:teuthology.orchestra.run.smithi096.stdout: "write_op_per_sec": 0 2024-05-28T06:23:47.749 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:23:47.749 INFO:teuthology.orchestra.run.smithi096.stdout: "fsmap": { 2024-05-28T06:23:47.750 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 73, 2024-05-28T06:23:47.750 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 8, 2024-05-28T06:23:47.750 INFO:teuthology.orchestra.run.smithi096.stdout: "up": 1, 2024-05-28T06:23:47.750 INFO:teuthology.orchestra.run.smithi096.stdout: "in": 1, 2024-05-28T06:23:47.750 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 1, 2024-05-28T06:23:47.750 INFO:teuthology.orchestra.run.smithi096.stdout: "by_rank": [ 2024-05-28T06:23:47.750 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:23:47.750 INFO:teuthology.orchestra.run.smithi096.stdout: "filesystem_id": 8, 2024-05-28T06:23:47.750 INFO:teuthology.orchestra.run.smithi096.stdout: "rank": 0, 2024-05-28T06:23:47.750 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "b", 2024-05-28T06:23:47.750 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "up:active", 2024-05-28T06:23:47.750 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 9270 2024-05-28T06:23:47.750 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:23:47.751 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:23:47.751 INFO:teuthology.orchestra.run.smithi096.stdout: "up:standby": 3 2024-05-28T06:23:47.751 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:23:47.751 INFO:teuthology.orchestra.run.smithi096.stdout: "mgrmap": { 2024-05-28T06:23:47.751 INFO:teuthology.orchestra.run.smithi096.stdout: "available": true, 2024-05-28T06:23:47.751 INFO:teuthology.orchestra.run.smithi096.stdout: "num_standbys": 1, 2024-05-28T06:23:47.751 INFO:teuthology.orchestra.run.smithi096.stdout: "modules": [ 2024-05-28T06:23:47.751 INFO:teuthology.orchestra.run.smithi096.stdout: "iostat", 2024-05-28T06:23:47.751 INFO:teuthology.orchestra.run.smithi096.stdout: "nfs", 2024-05-28T06:23:47.751 INFO:teuthology.orchestra.run.smithi096.stdout: "restful" 2024-05-28T06:23:47.751 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:23:47.751 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:23:47.751 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:23:47.752 INFO:teuthology.orchestra.run.smithi096.stdout: "servicemap": { 2024-05-28T06:23:47.752 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 23, 2024-05-28T06:23:47.752 INFO:teuthology.orchestra.run.smithi096.stdout: "modified": "2024-05-28T06:23:23.993668+0000", 2024-05-28T06:23:47.752 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:23:47.752 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:23:47.752 INFO:teuthology.orchestra.run.smithi096.stdout: "progress_events": { 2024-05-28T06:23:47.752 INFO:teuthology.orchestra.run.smithi096.stdout: "d6fb006f-47c2-4c80-b874-17506401c2f6": { 2024-05-28T06:23:47.752 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "All cloning finished, waiting 10.0 seconds before exiting (26s)\n [============================] ", 2024-05-28T06:23:47.752 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": 1, 2024-05-28T06:23:47.752 INFO:teuthology.orchestra.run.smithi096.stdout: "add_to_ceph_s": true 2024-05-28T06:23:47.752 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:23:47.752 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:23:47.752 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:23:47.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:47.750+0000 7fddff200640 1 -- 172.21.15.96:0/2733328726 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdddc06e3f0 msgr2=0x7fdddc0708b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:47.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:47.750+0000 7fddff200640 1 --2- 172.21.15.96:0/2733328726 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdddc06e3f0 0x7fdddc0708b0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fde000cf0d0 tx=0x7fddf0008040 comp rx=0 tx=0).stop 2024-05-28T06:23:47.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:47.750+0000 7fddff200640 1 -- 172.21.15.96:0/2733328726 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fde000d0460 msgr2=0x7fde000afdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:47.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:47.750+0000 7fddff200640 1 --2- 172.21.15.96:0/2733328726 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fde000d0460 0x7fde000afdc0 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7fdde8009720 tx=0x7fdde80046c0 comp rx=0 tx=0).stop 2024-05-28T06:23:47.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:47.750+0000 7fddff200640 1 -- 172.21.15.96:0/2733328726 shutdown_connections 2024-05-28T06:23:47.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:47.750+0000 7fddff200640 1 --2- 172.21.15.96:0/2733328726 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdddc06e3f0 0x7fdddc0708b0 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:47.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:47.750+0000 7fddff200640 1 --2- 172.21.15.96:0/2733328726 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde0015c6b0 0x7fde000ae180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:47.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:47.750+0000 7fddff200640 1 --2- 172.21.15.96:0/2733328726 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fde000d0460 0x7fde000afdc0 unknown :-1 s=CLOSED pgs=1890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:47.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:47.750+0000 7fddff200640 1 --2- 172.21.15.96:0/2733328726 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fde000ccc30 0x7fde000b21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:47.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:47.750+0000 7fddff200640 1 -- 172.21.15.96:0/2733328726 >> 172.21.15.96:0/2733328726 conn(0x7fde0006bae0 msgr2=0x7fde000cbf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:23:47.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:47.750+0000 7fddff200640 1 -- 172.21.15.96:0/2733328726 shutdown_connections 2024-05-28T06:23:47.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:47.750+0000 7fddff200640 1 -- 172.21.15.96:0/2733328726 wait complete. 2024-05-28T06:23:52.763 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-05-28T06:23:52.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:52.979+0000 7fc55d600640 1 Processor -- start 2024-05-28T06:23:52.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:52.980+0000 7fc55d600640 1 -- start start 2024-05-28T06:23:52.976 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:52.981+0000 7fc55d600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc55815c6b0 0x7fc5580b3db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:52.976 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:52.981+0000 7fc55d600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fc5580d0460 0x7fc5580d08f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:52.976 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:52.981+0000 7fc55d600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5580cd1a0 con 0x7fc5580ccc30 2024-05-28T06:23:52.976 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:52.981+0000 7fc55d600640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fc55806e3a0 con 0x7fc5580d0460 2024-05-28T06:23:52.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:52.981+0000 7fc55d600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fc5580cd480 con 0x7fc55815c6b0 2024-05-28T06:23:52.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:52.982+0000 7fc557e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc55815c6b0 0x7fc5580b3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:52.979 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:52.982+0000 7fc557e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc55815c6b0 0x7fc5580b3db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.96:42216/0 (socket says 172.21.15.96:42216) 2024-05-28T06:23:52.979 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:52.982+0000 7fc557e00640 1 -- 172.21.15.96:0/1430503358 learned_addr learned my addr 172.21.15.96:0/1430503358 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:23:52.979 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:52.982+0000 7fc55cc00640 1 --2- 172.21.15.96:0/1430503358 >> v2:172.21.15.123:3300/0 conn(0x7fc5580d0460 0x7fc5580d08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:52.979 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:52.983+0000 7fc557e00640 1 -- 172.21.15.96:0/1430503358 >> v2:172.21.15.123:3300/0 conn(0x7fc5580d0460 msgr2=0x7fc5580d08f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:23:52.980 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:52.983+0000 7fc557e00640 1 --2- 172.21.15.96:0/1430503358 >> v2:172.21.15.123:3300/0 conn(0x7fc5580d0460 0x7fc5580d08f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:52.980 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:52.983+0000 7fc557e00640 1 -- 172.21.15.96:0/1430503358 >> v1:172.21.15.123:6789/0 conn(0x7fc5580ccc30 legacy=0x7fc55806dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:52.980 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:52.983+0000 7fc557e00640 1 -- 172.21.15.96:0/1430503358 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5580cd760 con 0x7fc55815c6b0 2024-05-28T06:23:52.980 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:52.984+0000 7fc557e00640 1 --2- 172.21.15.96:0/1430503358 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc55815c6b0 0x7fc5580b3db0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc548009870 tx=0x7fc54802a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3e593221aa70a715 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:23:52.982 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:52.985+0000 7fc556a00640 1 -- 172.21.15.96:0/1430503358 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc548037070 con 0x7fc55815c6b0 2024-05-28T06:23:52.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:52.985+0000 7fc556a00640 1 -- 172.21.15.96:0/1430503358 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc548032440 con 0x7fc55815c6b0 2024-05-28T06:23:52.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:52.985+0000 7fc556a00640 1 -- 172.21.15.96:0/1430503358 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc548032810 con 0x7fc55815c6b0 2024-05-28T06:23:52.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:52.985+0000 7fc55d600640 1 -- 172.21.15.96:0/1430503358 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc55815c6b0 msgr2=0x7fc5580b3db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:52.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:52.985+0000 7fc55d600640 1 --2- 172.21.15.96:0/1430503358 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc55815c6b0 0x7fc5580b3db0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc548009870 tx=0x7fc54802a270 comp rx=0 tx=0).stop 2024-05-28T06:23:52.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:52.985+0000 7fc55d600640 1 -- 172.21.15.96:0/1430503358 shutdown_connections 2024-05-28T06:23:52.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:52.985+0000 7fc55d600640 1 --2- 172.21.15.96:0/1430503358 >> v2:172.21.15.123:3300/0 conn(0x7fc5580d0460 0x7fc5580d08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:52.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:52.985+0000 7fc55d600640 1 --2- 172.21.15.96:0/1430503358 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc55815c6b0 0x7fc5580b3db0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:52.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:52.985+0000 7fc55d600640 1 -- 172.21.15.96:0/1430503358 >> 172.21.15.96:0/1430503358 conn(0x7fc55806bae0 msgr2=0x7fc55806bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:23:52.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:52.986+0000 7fc55d600640 1 -- 172.21.15.96:0/1430503358 shutdown_connections 2024-05-28T06:23:52.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:52.986+0000 7fc55d600640 1 -- 172.21.15.96:0/1430503358 wait complete. 2024-05-28T06:23:52.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:52.987+0000 7fc55d600640 1 Processor -- start 2024-05-28T06:23:52.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:52.987+0000 7fc55d600640 1 -- start start 2024-05-28T06:23:52.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:52.988+0000 7fc55d600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc5580ccc30 0x7fc5581537c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:52.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:52.988+0000 7fc55d600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5580d0460 0x7fc5580c5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:52.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:52.988+0000 7fc55d600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc55815c6b0 0x7fc5580c58c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:52.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:52.988+0000 7fc55d600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc5580b42f0 con 0x7fc55815c6b0 2024-05-28T06:23:52.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:52.988+0000 7fc55d600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc5580cba10 con 0x7fc5580d0460 2024-05-28T06:23:52.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:52.988+0000 7fc55d600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fc5580b4470 con 0x7fc5580ccc30 2024-05-28T06:23:52.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:52.989+0000 7fc55cc00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc55815c6b0 0x7fc5580c58c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:52.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:52.989+0000 7fc55cc00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc55815c6b0 0x7fc5580c58c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:57476/0 (socket says 172.21.15.96:57476) 2024-05-28T06:23:52.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:52.989+0000 7fc55cc00640 1 -- 172.21.15.96:0/882633864 learned_addr learned my addr 172.21.15.96:0/882633864 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:23:52.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:52.989+0000 7fc557e00640 1 --2- 172.21.15.96:0/882633864 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc5580ccc30 0x7fc5581537c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:52.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:52.989+0000 7fc557400640 1 --2- 172.21.15.96:0/882633864 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5580d0460 0x7fc5580c5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:52.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:52.989+0000 7fc55cc00640 1 -- 172.21.15.96:0/882633864 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc5580ccc30 msgr2=0x7fc5581537c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:52.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:52.989+0000 7fc55cc00640 1 --2- 172.21.15.96:0/882633864 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc5580ccc30 0x7fc5581537c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:52.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:52.989+0000 7fc55cc00640 1 -- 172.21.15.96:0/882633864 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5580d0460 msgr2=0x7fc5580c5380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:52.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:52.989+0000 7fc55cc00640 1 --2- 172.21.15.96:0/882633864 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5580d0460 0x7fc5580c5380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:52.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:52.989+0000 7fc55cc00640 1 -- 172.21.15.96:0/882633864 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc55815b0d0 con 0x7fc55815c6b0 2024-05-28T06:23:52.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:52.989+0000 7fc557e00640 1 --2- 172.21.15.96:0/882633864 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc5580ccc30 0x7fc5581537c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:23:52.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:52.990+0000 7fc55cc00640 1 --2- 172.21.15.96:0/882633864 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc55815c6b0 0x7fc5580c58c0 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7fc55000d830 tx=0x7fc55000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:23:52.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:52.990+0000 7fc54fe00640 1 -- 172.21.15.96:0/882633864 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc55000b6c0 con 0x7fc55815c6b0 2024-05-28T06:23:52.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:52.990+0000 7fc54fe00640 1 -- 172.21.15.96:0/882633864 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc550019070 con 0x7fc55815c6b0 2024-05-28T06:23:52.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:52.990+0000 7fc55d600640 1 -- 172.21.15.96:0/882633864 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5581f4e70 con 0x7fc55815c6b0 2024-05-28T06:23:52.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:52.990+0000 7fc54fe00640 1 -- 172.21.15.96:0/882633864 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc55000bac0 con 0x7fc55815c6b0 2024-05-28T06:23:52.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:52.990+0000 7fc55d600640 1 -- 172.21.15.96:0/882633864 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5580c84e0 con 0x7fc55815c6b0 2024-05-28T06:23:52.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:52.991+0000 7fc55d600640 1 -- 172.21.15.96:0/882633864 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc55806fc40 con 0x7fc55815c6b0 2024-05-28T06:23:52.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:52.992+0000 7fc54fe00640 1 -- 172.21.15.96:0/882633864 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 41) ==== 93488+0+0 (secure 0 0 0) 0x7fc55001e460 con 0x7fc55815c6b0 2024-05-28T06:23:52.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:52.993+0000 7fc54fe00640 1 --2- 172.21.15.96:0/882633864 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc53806e3f0 0x7fc5380708b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:52.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:52.993+0000 7fc54fe00640 1 -- 172.21.15.96:0/882633864 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 5823+0+0 (secure 0 0 0) 0x7fc55008b140 con 0x7fc55815c6b0 2024-05-28T06:23:52.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:52.994+0000 7fc557e00640 1 --2- 172.21.15.96:0/882633864 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc53806e3f0 0x7fc5380708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:52.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:52.996+0000 7fc557e00640 1 --2- 172.21.15.96:0/882633864 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc53806e3f0 0x7fc5380708b0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fc548034000 tx=0x7fc54802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:23:52.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:52.999+0000 7fc54fe00640 1 -- 172.21.15.96:0/882633864 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc55806fc40 con 0x7fc55815c6b0 2024-05-28T06:23:53.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:53.276+0000 7fc55d600640 1 -- 172.21.15.96:0/882633864 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fc5580689e0 con 0x7fc55815c6b0 2024-05-28T06:23:53.271 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:53.277+0000 7fc54fe00640 1 -- 172.21.15.96:0/882633864 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+2079 (secure 0 0 0) 0x7fc55815b0d0 con 0x7fc55815c6b0 2024-05-28T06:23:53.272 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:23:53.272 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:23:53.272 INFO:teuthology.orchestra.run.smithi096.stdout: "fsid": "e4a831b5-e6d1-47a7-96cb-bf85154ee0d2", 2024-05-28T06:23:53.272 INFO:teuthology.orchestra.run.smithi096.stdout: "health": { 2024-05-28T06:23:53.272 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "HEALTH_OK", 2024-05-28T06:23:53.272 INFO:teuthology.orchestra.run.smithi096.stdout: "checks": {}, 2024-05-28T06:23:53.272 INFO:teuthology.orchestra.run.smithi096.stdout: "mutes": [] 2024-05-28T06:23:53.272 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:23:53.273 INFO:teuthology.orchestra.run.smithi096.stdout: "election_epoch": 8, 2024-05-28T06:23:53.273 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum": [ 2024-05-28T06:23:53.273 INFO:teuthology.orchestra.run.smithi096.stdout: 0, 2024-05-28T06:23:53.273 INFO:teuthology.orchestra.run.smithi096.stdout: 1, 2024-05-28T06:23:53.273 INFO:teuthology.orchestra.run.smithi096.stdout: 2 2024-05-28T06:23:53.273 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:23:53.273 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_names": [ 2024-05-28T06:23:53.274 INFO:teuthology.orchestra.run.smithi096.stdout: "a", 2024-05-28T06:23:53.274 INFO:teuthology.orchestra.run.smithi096.stdout: "b", 2024-05-28T06:23:53.274 INFO:teuthology.orchestra.run.smithi096.stdout: "c" 2024-05-28T06:23:53.274 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:23:53.274 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_age": 1206, 2024-05-28T06:23:53.274 INFO:teuthology.orchestra.run.smithi096.stdout: "monmap": { 2024-05-28T06:23:53.274 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 1, 2024-05-28T06:23:53.274 INFO:teuthology.orchestra.run.smithi096.stdout: "min_mon_release_name": "squid", 2024-05-28T06:23:53.274 INFO:teuthology.orchestra.run.smithi096.stdout: "num_mons": 3 2024-05-28T06:23:53.274 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:23:53.275 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap": { 2024-05-28T06:23:53.275 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 117, 2024-05-28T06:23:53.275 INFO:teuthology.orchestra.run.smithi096.stdout: "num_osds": 8, 2024-05-28T06:23:53.275 INFO:teuthology.orchestra.run.smithi096.stdout: "num_up_osds": 8, 2024-05-28T06:23:53.275 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_up_since": 1716876238, 2024-05-28T06:23:53.275 INFO:teuthology.orchestra.run.smithi096.stdout: "num_in_osds": 8, 2024-05-28T06:23:53.276 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_in_since": 1716876229, 2024-05-28T06:23:53.276 INFO:teuthology.orchestra.run.smithi096.stdout: "num_remapped_pgs": 0 2024-05-28T06:23:53.276 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:23:53.276 INFO:teuthology.orchestra.run.smithi096.stdout: "pgmap": { 2024-05-28T06:23:53.276 INFO:teuthology.orchestra.run.smithi096.stdout: "pgs_by_state": [ 2024-05-28T06:23:53.276 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:23:53.276 INFO:teuthology.orchestra.run.smithi096.stdout: "state_name": "active+clean", 2024-05-28T06:23:53.276 INFO:teuthology.orchestra.run.smithi096.stdout: "count": 97 2024-05-28T06:23:53.276 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:23:53.277 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:23:53.277 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pgs": 97, 2024-05-28T06:23:53.277 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pools": 3, 2024-05-28T06:23:53.277 INFO:teuthology.orchestra.run.smithi096.stdout: "num_objects": 1745, 2024-05-28T06:23:53.278 INFO:teuthology.orchestra.run.smithi096.stdout: "data_bytes": 7190393650, 2024-05-28T06:23:53.278 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_used": 15424987136, 2024-05-28T06:23:53.278 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_avail": 757669126144, 2024-05-28T06:23:53.278 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_total": 773094113280, 2024-05-28T06:23:53.278 INFO:teuthology.orchestra.run.smithi096.stdout: "write_bytes_sec": 426, 2024-05-28T06:23:53.278 INFO:teuthology.orchestra.run.smithi096.stdout: "read_op_per_sec": 0, 2024-05-28T06:23:53.279 INFO:teuthology.orchestra.run.smithi096.stdout: "write_op_per_sec": 0 2024-05-28T06:23:53.279 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:23:53.279 INFO:teuthology.orchestra.run.smithi096.stdout: "fsmap": { 2024-05-28T06:23:53.279 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 73, 2024-05-28T06:23:53.279 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 8, 2024-05-28T06:23:53.279 INFO:teuthology.orchestra.run.smithi096.stdout: "up": 1, 2024-05-28T06:23:53.279 INFO:teuthology.orchestra.run.smithi096.stdout: "in": 1, 2024-05-28T06:23:53.280 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 1, 2024-05-28T06:23:53.280 INFO:teuthology.orchestra.run.smithi096.stdout: "by_rank": [ 2024-05-28T06:23:53.280 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:23:53.280 INFO:teuthology.orchestra.run.smithi096.stdout: "filesystem_id": 8, 2024-05-28T06:23:53.280 INFO:teuthology.orchestra.run.smithi096.stdout: "rank": 0, 2024-05-28T06:23:53.280 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "b", 2024-05-28T06:23:53.280 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "up:active", 2024-05-28T06:23:53.280 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 9270 2024-05-28T06:23:53.280 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:23:53.280 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:23:53.280 INFO:teuthology.orchestra.run.smithi096.stdout: "up:standby": 3 2024-05-28T06:23:53.281 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:23:53.281 INFO:teuthology.orchestra.run.smithi096.stdout: "mgrmap": { 2024-05-28T06:23:53.281 INFO:teuthology.orchestra.run.smithi096.stdout: "available": true, 2024-05-28T06:23:53.281 INFO:teuthology.orchestra.run.smithi096.stdout: "num_standbys": 1, 2024-05-28T06:23:53.281 INFO:teuthology.orchestra.run.smithi096.stdout: "modules": [ 2024-05-28T06:23:53.281 INFO:teuthology.orchestra.run.smithi096.stdout: "iostat", 2024-05-28T06:23:53.281 INFO:teuthology.orchestra.run.smithi096.stdout: "nfs", 2024-05-28T06:23:53.281 INFO:teuthology.orchestra.run.smithi096.stdout: "restful" 2024-05-28T06:23:53.281 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:23:53.281 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:23:53.281 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:23:53.281 INFO:teuthology.orchestra.run.smithi096.stdout: "servicemap": { 2024-05-28T06:23:53.281 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 23, 2024-05-28T06:23:53.282 INFO:teuthology.orchestra.run.smithi096.stdout: "modified": "2024-05-28T06:23:23.993668+0000", 2024-05-28T06:23:53.282 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:23:53.282 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:23:53.282 INFO:teuthology.orchestra.run.smithi096.stdout: "progress_events": { 2024-05-28T06:23:53.282 INFO:teuthology.orchestra.run.smithi096.stdout: "d6fb006f-47c2-4c80-b874-17506401c2f6": { 2024-05-28T06:23:53.282 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "All cloning finished, waiting 4.0 seconds before exiting (32s)\n [============================] ", 2024-05-28T06:23:53.282 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": 1, 2024-05-28T06:23:53.282 INFO:teuthology.orchestra.run.smithi096.stdout: "add_to_ceph_s": true 2024-05-28T06:23:53.282 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:23:53.282 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:23:53.283 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:23:53.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:53.279+0000 7fc55d600640 1 -- 172.21.15.96:0/882633864 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc53806e3f0 msgr2=0x7fc5380708b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:53.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:53.279+0000 7fc55d600640 1 --2- 172.21.15.96:0/882633864 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc53806e3f0 0x7fc5380708b0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fc548034000 tx=0x7fc54802e000 comp rx=0 tx=0).stop 2024-05-28T06:23:53.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:53.279+0000 7fc55d600640 1 -- 172.21.15.96:0/882633864 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc55815c6b0 msgr2=0x7fc5580c58c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:53.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:53.279+0000 7fc55d600640 1 --2- 172.21.15.96:0/882633864 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc55815c6b0 0x7fc5580c58c0 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7fc55000d830 tx=0x7fc55000dcf0 comp rx=0 tx=0).stop 2024-05-28T06:23:53.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:53.279+0000 7fc55d600640 1 -- 172.21.15.96:0/882633864 shutdown_connections 2024-05-28T06:23:53.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:53.279+0000 7fc55d600640 1 --2- 172.21.15.96:0/882633864 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc53806e3f0 0x7fc5380708b0 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:53.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:53.279+0000 7fc55d600640 1 --2- 172.21.15.96:0/882633864 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc55815c6b0 0x7fc5580c58c0 unknown :-1 s=CLOSED pgs=1891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:53.284 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:53.279+0000 7fc55d600640 1 --2- 172.21.15.96:0/882633864 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5580d0460 0x7fc5580c5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:53.285 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:53.279+0000 7fc55d600640 1 --2- 172.21.15.96:0/882633864 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc5580ccc30 0x7fc5581537c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:53.285 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:53.279+0000 7fc55d600640 1 -- 172.21.15.96:0/882633864 >> 172.21.15.96:0/882633864 conn(0x7fc55806bae0 msgr2=0x7fc558073a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:23:53.285 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:53.279+0000 7fc55d600640 1 -- 172.21.15.96:0/882633864 shutdown_connections 2024-05-28T06:23:53.285 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:53.280+0000 7fc55d600640 1 -- 172.21.15.96:0/882633864 wait complete. 2024-05-28T06:23:58.296 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-05-28T06:23:58.505 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:58.510+0000 7f97f6c00640 1 Processor -- start 2024-05-28T06:23:58.505 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:58.511+0000 7f97f6c00640 1 -- start start 2024-05-28T06:23:58.507 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:58.512+0000 7f97f6c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f97f00723d0 0x7f97f00b3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:58.507 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:58.512+0000 7f97f6c00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f97f00b41e0 0x7f97f00cf880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:58.507 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:58.512+0000 7f97f6c00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97f00cf0c0 con 0x7f97f0145320 2024-05-28T06:23:58.507 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:58.512+0000 7f97f6c00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f97f00b4a20 con 0x7f97f00b41e0 2024-05-28T06:23:58.507 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:58.512+0000 7f97f6c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f97f00b4d00 con 0x7f97f00723d0 2024-05-28T06:23:58.507 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:58.512+0000 7f97f4c00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f97f00b41e0 0x7f97f00cf880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:58.507 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:58.512+0000 7f97f4c00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f97f00b41e0 0x7f97f00cf880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:57486/0 (socket says 172.21.15.96:57486) 2024-05-28T06:23:58.507 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:58.512+0000 7f97f4c00640 1 -- 172.21.15.96:0/1288276199 learned_addr learned my addr 172.21.15.96:0/1288276199 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:23:58.508 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:58.513+0000 7f97ef400640 1 --2- 172.21.15.96:0/1288276199 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f97f00723d0 0x7f97f00b3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:58.508 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:58.513+0000 7f97f4c00640 1 -- 172.21.15.96:0/1288276199 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f97f00723d0 msgr2=0x7f97f00b3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:23:58.508 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:58.513+0000 7f97f4c00640 1 --2- 172.21.15.96:0/1288276199 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f97f00723d0 0x7f97f00b3ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:58.509 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:58.513+0000 7f97f4c00640 1 -- 172.21.15.96:0/1288276199 >> v1:172.21.15.123:6789/0 conn(0x7f97f0145320 legacy=0x7f97f0071cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:23:58.509 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:58.513+0000 7f97f4c00640 1 -- 172.21.15.96:0/1288276199 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97f00b4fe0 con 0x7f97f00b41e0 2024-05-28T06:23:58.509 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:58.514+0000 7f97f4c00640 1 --2- 172.21.15.96:0/1288276199 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97f00b41e0 0x7f97f00cf880 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7f97e8009870 tx=0x7f97e802a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=807bd1db003009a4 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:23:58.510 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:58.514+0000 7f97eea00640 1 -- 172.21.15.96:0/1288276199 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97e8037070 con 0x7f97f00b41e0 2024-05-28T06:23:58.510 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:58.514+0000 7f97eea00640 1 -- 172.21.15.96:0/1288276199 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97e8032440 con 0x7f97f00b41e0 2024-05-28T06:23:58.510 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:58.514+0000 7f97eea00640 1 -- 172.21.15.96:0/1288276199 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97e8032810 con 0x7f97f00b41e0 2024-05-28T06:23:58.510 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:58.515+0000 7f97f6c00640 1 -- 172.21.15.96:0/1288276199 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97f00b41e0 msgr2=0x7f97f00cf880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:58.510 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:58.515+0000 7f97f6c00640 1 --2- 172.21.15.96:0/1288276199 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97f00b41e0 0x7f97f00cf880 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7f97e8009870 tx=0x7f97e802a210 comp rx=0 tx=0).stop 2024-05-28T06:23:58.511 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:58.515+0000 7f97f6c00640 1 -- 172.21.15.96:0/1288276199 shutdown_connections 2024-05-28T06:23:58.511 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:58.515+0000 7f97f6c00640 1 --2- 172.21.15.96:0/1288276199 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97f00b41e0 0x7f97f00cf880 unknown :-1 s=CLOSED pgs=1892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:58.511 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:58.515+0000 7f97f6c00640 1 --2- 172.21.15.96:0/1288276199 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f97f00723d0 0x7f97f00b3ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:58.511 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:58.515+0000 7f97f6c00640 1 -- 172.21.15.96:0/1288276199 >> 172.21.15.96:0/1288276199 conn(0x7f97f0067ce0 msgr2=0x7f97f006b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:23:58.511 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:58.515+0000 7f97f6c00640 1 -- 172.21.15.96:0/1288276199 shutdown_connections 2024-05-28T06:23:58.511 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:58.515+0000 7f97f6c00640 1 -- 172.21.15.96:0/1288276199 wait complete. 2024-05-28T06:23:58.512 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:58.517+0000 7f97f6c00640 1 Processor -- start 2024-05-28T06:23:58.512 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:58.517+0000 7f97f6c00640 1 -- start start 2024-05-28T06:23:58.514 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:58.517+0000 7f97f6c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f97f00723d0 0x7f97f01e4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:58.514 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:58.518+0000 7f97f6c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f97f00b41e0 0x7f97f015eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:58.514 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:58.518+0000 7f97f6c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97f0145320 0x7f97f0143f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:58.514 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:58.518+0000 7f97f6c00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f97f00b4670 con 0x7f97f0145320 2024-05-28T06:23:58.514 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:58.518+0000 7f97f6c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f97f0145700 con 0x7f97f00723d0 2024-05-28T06:23:58.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:58.518+0000 7f97f6c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f97f015eff0 con 0x7f97f00b41e0 2024-05-28T06:23:58.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:58.518+0000 7f97f4c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97f0145320 0x7f97f0143f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:58.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:58.518+0000 7f97f4c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97f0145320 0x7f97f0143f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:57492/0 (socket says 172.21.15.96:57492) 2024-05-28T06:23:58.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:58.518+0000 7f97f4c00640 1 -- 172.21.15.96:0/2657312567 learned_addr learned my addr 172.21.15.96:0/2657312567 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:23:58.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:58.518+0000 7f97efe00640 1 --2- 172.21.15.96:0/2657312567 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f97f00723d0 0x7f97f01e4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:58.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:58.518+0000 7f97ef400640 1 --2- 172.21.15.96:0/2657312567 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f97f00b41e0 0x7f97f015eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:58.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:58.518+0000 7f97f4c00640 1 -- 172.21.15.96:0/2657312567 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f97f00b41e0 msgr2=0x7f97f015eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:58.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:58.518+0000 7f97f4c00640 1 --2- 172.21.15.96:0/2657312567 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f97f00b41e0 0x7f97f015eab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:58.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:58.518+0000 7f97f4c00640 1 -- 172.21.15.96:0/2657312567 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f97f00723d0 msgr2=0x7f97f01e4ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:58.516 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:58.518+0000 7f97f4c00640 1 --2- 172.21.15.96:0/2657312567 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f97f00723d0 0x7f97f01e4ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:58.516 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:58.518+0000 7f97f4c00640 1 -- 172.21.15.96:0/2657312567 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97f01e5530 con 0x7f97f0145320 2024-05-28T06:23:58.516 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:58.518+0000 7f97efe00640 1 --2- 172.21.15.96:0/2657312567 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f97f00723d0 0x7f97f01e4ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:23:58.516 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:58.519+0000 7f97f4c00640 1 --2- 172.21.15.96:0/2657312567 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97f0145320 0x7f97f0143f00 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7f97e802a250 tx=0x7f97e80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:23:58.516 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:58.519+0000 7f97e7e00640 1 -- 172.21.15.96:0/2657312567 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97e8041020 con 0x7f97f0145320 2024-05-28T06:23:58.516 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:58.519+0000 7f97e7e00640 1 -- 172.21.15.96:0/2657312567 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97e80306c0 con 0x7f97f0145320 2024-05-28T06:23:58.516 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:58.519+0000 7f97f6c00640 1 -- 172.21.15.96:0/2657312567 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97f01e5810 con 0x7f97f0145320 2024-05-28T06:23:58.516 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:58.519+0000 7f97e7e00640 1 -- 172.21.15.96:0/2657312567 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97e8037040 con 0x7f97f0145320 2024-05-28T06:23:58.516 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:58.519+0000 7f97f6c00640 1 -- 172.21.15.96:0/2657312567 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97f01e5dd0 con 0x7f97f0145320 2024-05-28T06:23:58.517 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:58.520+0000 7f97f6c00640 1 -- 172.21.15.96:0/2657312567 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97f00689e0 con 0x7f97f0145320 2024-05-28T06:23:58.517 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:58.521+0000 7f97e7e00640 1 -- 172.21.15.96:0/2657312567 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 41) ==== 93488+0+0 (secure 0 0 0) 0x7f97e8044020 con 0x7f97f0145320 2024-05-28T06:23:58.518 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:58.522+0000 7f97e7e00640 1 --2- 172.21.15.96:0/2657312567 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f97cc06e3f0 0x7f97cc0708b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:58.518 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:58.523+0000 7f97e7e00640 1 -- 172.21.15.96:0/2657312567 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 5823+0+0 (secure 0 0 0) 0x7f97e80ab220 con 0x7f97f0145320 2024-05-28T06:23:58.518 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:58.523+0000 7f97efe00640 1 --2- 172.21.15.96:0/2657312567 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f97cc06e3f0 0x7f97cc0708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:58.518 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:58.524+0000 7f97efe00640 1 --2- 172.21.15.96:0/2657312567 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f97cc06e3f0 0x7f97cc0708b0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f97e0007e90 tx=0x7f97e00073a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:23:58.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:58.527+0000 7f97e7e00640 1 -- 172.21.15.96:0/2657312567 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f97e804b030 con 0x7f97f0145320 2024-05-28T06:23:58.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:58.801+0000 7f97f6c00640 1 -- 172.21.15.96:0/2657312567 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f97f015f1b0 con 0x7f97f0145320 2024-05-28T06:23:58.797 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:58.802+0000 7f97e7e00640 1 -- 172.21.15.96:0/2657312567 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1825 (secure 0 0 0) 0x7f97e807b0b0 con 0x7f97f0145320 2024-05-28T06:23:58.797 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:23:58.797 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:23:58.797 INFO:teuthology.orchestra.run.smithi096.stdout: "fsid": "e4a831b5-e6d1-47a7-96cb-bf85154ee0d2", 2024-05-28T06:23:58.797 INFO:teuthology.orchestra.run.smithi096.stdout: "health": { 2024-05-28T06:23:58.797 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "HEALTH_OK", 2024-05-28T06:23:58.798 INFO:teuthology.orchestra.run.smithi096.stdout: "checks": {}, 2024-05-28T06:23:58.798 INFO:teuthology.orchestra.run.smithi096.stdout: "mutes": [] 2024-05-28T06:23:58.798 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:23:58.798 INFO:teuthology.orchestra.run.smithi096.stdout: "election_epoch": 8, 2024-05-28T06:23:58.798 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum": [ 2024-05-28T06:23:58.798 INFO:teuthology.orchestra.run.smithi096.stdout: 0, 2024-05-28T06:23:58.798 INFO:teuthology.orchestra.run.smithi096.stdout: 1, 2024-05-28T06:23:58.798 INFO:teuthology.orchestra.run.smithi096.stdout: 2 2024-05-28T06:23:58.798 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:23:58.798 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_names": [ 2024-05-28T06:23:58.798 INFO:teuthology.orchestra.run.smithi096.stdout: "a", 2024-05-28T06:23:58.798 INFO:teuthology.orchestra.run.smithi096.stdout: "b", 2024-05-28T06:23:58.798 INFO:teuthology.orchestra.run.smithi096.stdout: "c" 2024-05-28T06:23:58.799 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:23:58.799 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_age": 1212, 2024-05-28T06:23:58.799 INFO:teuthology.orchestra.run.smithi096.stdout: "monmap": { 2024-05-28T06:23:58.799 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 1, 2024-05-28T06:23:58.799 INFO:teuthology.orchestra.run.smithi096.stdout: "min_mon_release_name": "squid", 2024-05-28T06:23:58.799 INFO:teuthology.orchestra.run.smithi096.stdout: "num_mons": 3 2024-05-28T06:23:58.799 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:23:58.799 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap": { 2024-05-28T06:23:58.799 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 117, 2024-05-28T06:23:58.799 INFO:teuthology.orchestra.run.smithi096.stdout: "num_osds": 8, 2024-05-28T06:23:58.799 INFO:teuthology.orchestra.run.smithi096.stdout: "num_up_osds": 8, 2024-05-28T06:23:58.799 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_up_since": 1716876238, 2024-05-28T06:23:58.799 INFO:teuthology.orchestra.run.smithi096.stdout: "num_in_osds": 8, 2024-05-28T06:23:58.800 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_in_since": 1716876229, 2024-05-28T06:23:58.800 INFO:teuthology.orchestra.run.smithi096.stdout: "num_remapped_pgs": 0 2024-05-28T06:23:58.800 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:23:58.800 INFO:teuthology.orchestra.run.smithi096.stdout: "pgmap": { 2024-05-28T06:23:58.800 INFO:teuthology.orchestra.run.smithi096.stdout: "pgs_by_state": [ 2024-05-28T06:23:58.800 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:23:58.800 INFO:teuthology.orchestra.run.smithi096.stdout: "state_name": "active+clean", 2024-05-28T06:23:58.800 INFO:teuthology.orchestra.run.smithi096.stdout: "count": 97 2024-05-28T06:23:58.800 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:23:58.800 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:23:58.800 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pgs": 97, 2024-05-28T06:23:58.800 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pools": 3, 2024-05-28T06:23:58.800 INFO:teuthology.orchestra.run.smithi096.stdout: "num_objects": 1745, 2024-05-28T06:23:58.801 INFO:teuthology.orchestra.run.smithi096.stdout: "data_bytes": 7190393650, 2024-05-28T06:23:58.801 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_used": 15424987136, 2024-05-28T06:23:58.801 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_avail": 757669126144, 2024-05-28T06:23:58.801 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_total": 773094113280, 2024-05-28T06:23:58.801 INFO:teuthology.orchestra.run.smithi096.stdout: "write_bytes_sec": 426, 2024-05-28T06:23:58.801 INFO:teuthology.orchestra.run.smithi096.stdout: "read_op_per_sec": 0, 2024-05-28T06:23:58.801 INFO:teuthology.orchestra.run.smithi096.stdout: "write_op_per_sec": 0 2024-05-28T06:23:58.801 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:23:58.801 INFO:teuthology.orchestra.run.smithi096.stdout: "fsmap": { 2024-05-28T06:23:58.801 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 73, 2024-05-28T06:23:58.801 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 8, 2024-05-28T06:23:58.801 INFO:teuthology.orchestra.run.smithi096.stdout: "up": 1, 2024-05-28T06:23:58.802 INFO:teuthology.orchestra.run.smithi096.stdout: "in": 1, 2024-05-28T06:23:58.802 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 1, 2024-05-28T06:23:58.802 INFO:teuthology.orchestra.run.smithi096.stdout: "by_rank": [ 2024-05-28T06:23:58.802 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:23:58.802 INFO:teuthology.orchestra.run.smithi096.stdout: "filesystem_id": 8, 2024-05-28T06:23:58.802 INFO:teuthology.orchestra.run.smithi096.stdout: "rank": 0, 2024-05-28T06:23:58.802 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "b", 2024-05-28T06:23:58.802 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "up:active", 2024-05-28T06:23:58.802 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 9270 2024-05-28T06:23:58.802 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:23:58.802 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:23:58.802 INFO:teuthology.orchestra.run.smithi096.stdout: "up:standby": 3 2024-05-28T06:23:58.802 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:23:58.803 INFO:teuthology.orchestra.run.smithi096.stdout: "mgrmap": { 2024-05-28T06:23:58.803 INFO:teuthology.orchestra.run.smithi096.stdout: "available": true, 2024-05-28T06:23:58.803 INFO:teuthology.orchestra.run.smithi096.stdout: "num_standbys": 1, 2024-05-28T06:23:58.803 INFO:teuthology.orchestra.run.smithi096.stdout: "modules": [ 2024-05-28T06:23:58.803 INFO:teuthology.orchestra.run.smithi096.stdout: "iostat", 2024-05-28T06:23:58.803 INFO:teuthology.orchestra.run.smithi096.stdout: "nfs", 2024-05-28T06:23:58.803 INFO:teuthology.orchestra.run.smithi096.stdout: "restful" 2024-05-28T06:23:58.803 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:23:58.803 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:23:58.803 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:23:58.803 INFO:teuthology.orchestra.run.smithi096.stdout: "servicemap": { 2024-05-28T06:23:58.803 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 23, 2024-05-28T06:23:58.803 INFO:teuthology.orchestra.run.smithi096.stdout: "modified": "2024-05-28T06:23:23.993668+0000", 2024-05-28T06:23:58.804 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:23:58.804 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:23:58.804 INFO:teuthology.orchestra.run.smithi096.stdout: "progress_events": {} 2024-05-28T06:23:58.804 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:23:58.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:58.805+0000 7f97f6c00640 1 -- 172.21.15.96:0/2657312567 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f97cc06e3f0 msgr2=0x7f97cc0708b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:58.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:58.805+0000 7f97f6c00640 1 --2- 172.21.15.96:0/2657312567 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f97cc06e3f0 0x7f97cc0708b0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f97e0007e90 tx=0x7f97e00073a0 comp rx=0 tx=0).stop 2024-05-28T06:23:58.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:58.805+0000 7f97f6c00640 1 -- 172.21.15.96:0/2657312567 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97f0145320 msgr2=0x7f97f0143f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:58.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:58.805+0000 7f97f6c00640 1 --2- 172.21.15.96:0/2657312567 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97f0145320 0x7f97f0143f00 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7f97e802a250 tx=0x7f97e80046c0 comp rx=0 tx=0).stop 2024-05-28T06:23:58.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:58.805+0000 7f97f6c00640 1 -- 172.21.15.96:0/2657312567 shutdown_connections 2024-05-28T06:23:58.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:58.805+0000 7f97f6c00640 1 --2- 172.21.15.96:0/2657312567 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f97cc06e3f0 0x7f97cc0708b0 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:58.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:58.805+0000 7f97f6c00640 1 --2- 172.21.15.96:0/2657312567 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97f0145320 0x7f97f0143f00 unknown :-1 s=CLOSED pgs=1893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:58.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:58.805+0000 7f97f6c00640 1 --2- 172.21.15.96:0/2657312567 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f97f00b41e0 0x7f97f015eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:58.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:58.805+0000 7f97f6c00640 1 --2- 172.21.15.96:0/2657312567 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f97f00723d0 0x7f97f01e4ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:58.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:58.805+0000 7f97f6c00640 1 -- 172.21.15.96:0/2657312567 >> 172.21.15.96:0/2657312567 conn(0x7f97f0067ce0 msgr2=0x7f97f006b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:23:58.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:58.805+0000 7f97f6c00640 1 -- 172.21.15.96:0/2657312567 shutdown_connections 2024-05-28T06:23:58.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:58.805+0000 7f97f6c00640 1 -- 172.21.15.96:0/2657312567 wait complete. 2024-05-28T06:23:58.821 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume ls cephfs --format json 2024-05-28T06:23:59.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.028+0000 7f9e1f200640 1 Processor -- start 2024-05-28T06:23:59.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.028+0000 7f9e1f200640 1 -- start start 2024-05-28T06:23:59.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.029+0000 7f9e1f200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f9e200cd2a0 0x7f9e200b4d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:59.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.029+0000 7f9e1f200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f9e2015cb60 0x7f9e2015cf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:59.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.030+0000 7f9e1f200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e2015d900 con 0x7f9e20153510 2024-05-28T06:23:59.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.030+0000 7f9e1f200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f9e201538f0 con 0x7f9e200cd2a0 2024-05-28T06:23:59.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.030+0000 7f9e1f200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f9e200b48c0 con 0x7f9e2015cb60 2024-05-28T06:23:59.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.030+0000 7f9e15400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f9e200cd2a0 0x7f9e200b4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:59.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.030+0000 7f9e15400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f9e200cd2a0 0x7f9e200b4d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:42920/0 (socket says 172.21.15.96:42920) 2024-05-28T06:23:59.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.030+0000 7f9e15400640 1 -- 172.21.15.96:0/3197461650 learned_addr learned my addr 172.21.15.96:0/3197461650 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:23:59.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.030+0000 7f9e15400640 1 -- 172.21.15.96:0/3197461650 >> v2:172.21.15.123:3300/0 conn(0x7f9e2015cb60 msgr2=0x7f9e2015cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:23:59.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.030+0000 7f9e15400640 1 --2- 172.21.15.96:0/3197461650 >> v2:172.21.15.123:3300/0 conn(0x7f9e2015cb60 0x7f9e2015cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:59.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.030+0000 7f9e15400640 1 -- 172.21.15.96:0/3197461650 >> v1:172.21.15.123:6789/0 conn(0x7f9e20153510 legacy=0x7f9e200ccb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:23:59.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.030+0000 7f9e15400640 1 -- 172.21.15.96:0/3197461650 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e200b3f90 con 0x7f9e200cd2a0 2024-05-28T06:23:59.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.031+0000 7f9e15400640 1 --2- 172.21.15.96:0/3197461650 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9e200cd2a0 0x7f9e200b4d50 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 crypto rx=0x7f9e200b4a40 tx=0x7f9e0802a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a386dca3c7e626b0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:23:59.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.032+0000 7f9e1d400640 1 -- 172.21.15.96:0/3197461650 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e08037070 con 0x7f9e200cd2a0 2024-05-28T06:23:59.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.032+0000 7f9e1d400640 1 -- 172.21.15.96:0/3197461650 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e08032440 con 0x7f9e200cd2a0 2024-05-28T06:23:59.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.032+0000 7f9e1d400640 1 -- 172.21.15.96:0/3197461650 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e08032810 con 0x7f9e200cd2a0 2024-05-28T06:23:59.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.032+0000 7f9e1f200640 1 -- 172.21.15.96:0/3197461650 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9e200cd2a0 msgr2=0x7f9e200b4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:59.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.032+0000 7f9e1f200640 1 --2- 172.21.15.96:0/3197461650 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9e200cd2a0 0x7f9e200b4d50 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 crypto rx=0x7f9e200b4a40 tx=0x7f9e0802a180 comp rx=0 tx=0).stop 2024-05-28T06:23:59.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.032+0000 7f9e1f200640 1 -- 172.21.15.96:0/3197461650 shutdown_connections 2024-05-28T06:23:59.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.032+0000 7f9e1f200640 1 --2- 172.21.15.96:0/3197461650 >> v2:172.21.15.123:3300/0 conn(0x7f9e2015cb60 0x7f9e2015cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:59.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.032+0000 7f9e1f200640 1 --2- 172.21.15.96:0/3197461650 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9e200cd2a0 0x7f9e200b4d50 unknown :-1 s=CLOSED pgs=1894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:59.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.032+0000 7f9e1f200640 1 -- 172.21.15.96:0/3197461650 >> 172.21.15.96:0/3197461650 conn(0x7f9e2006b900 msgr2=0x7f9e2006bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:23:59.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.032+0000 7f9e1f200640 1 -- 172.21.15.96:0/3197461650 shutdown_connections 2024-05-28T06:23:59.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.032+0000 7f9e1f200640 1 -- 172.21.15.96:0/3197461650 wait complete. 2024-05-28T06:23:59.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.034+0000 7f9e1f200640 1 Processor -- start 2024-05-28T06:23:59.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.034+0000 7f9e1f200640 1 -- start start 2024-05-28T06:23:59.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.035+0000 7f9e1f200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9e200cd2a0 0x7f9e200c9740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:59.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.035+0000 7f9e1f200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9e20153510 0x7f9e200994e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:59.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.035+0000 7f9e1f200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9e2015cb60 0x7f9e20099a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:59.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.035+0000 7f9e1f200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f9e2015d4b0 con 0x7f9e20153510 2024-05-28T06:23:59.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.035+0000 7f9e1f200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f9e200b4550 con 0x7f9e2015cb60 2024-05-28T06:23:59.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.035+0000 7f9e1f200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f9e2015d630 con 0x7f9e200cd2a0 2024-05-28T06:23:59.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.035+0000 7f9e15400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9e20153510 0x7f9e200994e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:59.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.035+0000 7f9e15400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9e20153510 0x7f9e200994e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:42932/0 (socket says 172.21.15.96:42932) 2024-05-28T06:23:59.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.035+0000 7f9e15400640 1 -- 172.21.15.96:0/4220396604 learned_addr learned my addr 172.21.15.96:0/4220396604 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:23:59.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.036+0000 7f9e1e800640 1 --2- 172.21.15.96:0/4220396604 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9e2015cb60 0x7f9e20099a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:59.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.036+0000 7f9e1de00640 1 --2- 172.21.15.96:0/4220396604 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9e200cd2a0 0x7f9e200c9740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:59.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.036+0000 7f9e15400640 1 -- 172.21.15.96:0/4220396604 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9e200cd2a0 msgr2=0x7f9e200c9740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:59.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.036+0000 7f9e15400640 1 --2- 172.21.15.96:0/4220396604 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9e200cd2a0 0x7f9e200c9740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:59.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.036+0000 7f9e15400640 1 -- 172.21.15.96:0/4220396604 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9e2015cb60 msgr2=0x7f9e20099a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:59.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.036+0000 7f9e15400640 1 --2- 172.21.15.96:0/4220396604 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9e2015cb60 0x7f9e20099a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:59.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.036+0000 7f9e15400640 1 -- 172.21.15.96:0/4220396604 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e20146fa0 con 0x7f9e20153510 2024-05-28T06:23:59.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.036+0000 7f9e1e800640 1 --2- 172.21.15.96:0/4220396604 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9e2015cb60 0x7f9e20099a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:23:59.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.036+0000 7f9e15400640 1 --2- 172.21.15.96:0/4220396604 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9e20153510 0x7f9e200994e0 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7f9e08000c00 tx=0x7f9e080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:23:59.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.036+0000 7f9e16a00640 1 -- 172.21.15.96:0/4220396604 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e08041070 con 0x7f9e20153510 2024-05-28T06:23:59.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.037+0000 7f9e16a00640 1 -- 172.21.15.96:0/4220396604 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e0803c870 con 0x7f9e20153510 2024-05-28T06:23:59.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.037+0000 7f9e1f200640 1 -- 172.21.15.96:0/4220396604 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e201f49f0 con 0x7f9e20153510 2024-05-28T06:23:59.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.037+0000 7f9e16a00640 1 -- 172.21.15.96:0/4220396604 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e08037040 con 0x7f9e20153510 2024-05-28T06:23:59.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.037+0000 7f9e1f200640 1 -- 172.21.15.96:0/4220396604 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e201f4fb0 con 0x7f9e20153510 2024-05-28T06:23:59.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.038+0000 7f9e1f200640 1 -- 172.21.15.96:0/4220396604 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e200689e0 con 0x7f9e20153510 2024-05-28T06:23:59.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.039+0000 7f9e16a00640 1 -- 172.21.15.96:0/4220396604 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 41) ==== 93488+0+0 (secure 0 0 0) 0x7f9e08044070 con 0x7f9e20153510 2024-05-28T06:23:59.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.040+0000 7f9e16a00640 1 --2- 172.21.15.96:0/4220396604 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9dfc06e3f0 0x7f9dfc0708b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:59.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.040+0000 7f9e16a00640 1 -- 172.21.15.96:0/4220396604 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 5823+0+0 (secure 0 0 0) 0x7f9e080ab7a0 con 0x7f9e20153510 2024-05-28T06:23:59.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.040+0000 7f9e1de00640 1 --2- 172.21.15.96:0/4220396604 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9dfc06e3f0 0x7f9dfc0708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:59.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.042+0000 7f9e1de00640 1 --2- 172.21.15.96:0/4220396604 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9dfc06e3f0 0x7f9dfc0708b0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f9e1000a750 tx=0x7f9e10008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:23:59.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.044+0000 7f9e16a00640 1 -- 172.21.15.96:0/4220396604 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9e0807b750 con 0x7f9e20153510 2024-05-28T06:23:59.261 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.266+0000 7f9e1f200640 1 -- 172.21.15.96:0/4220396604 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume ls", "vol_name": "cephfs", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9e200cc300 con 0x7f9dfc06e3f0 2024-05-28T06:23:59.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.282+0000 7f9e16a00640 1 -- 172.21.15.96:0/4220396604 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+185 (secure 0 0 0) 0x7f9e200cc300 con 0x7f9dfc06e3f0 2024-05-28T06:23:59.277 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:23:59.277 INFO:teuthology.orchestra.run.smithi096.stdout:[ 2024-05-28T06:23:59.277 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:23:59.277 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "subvol_clone36081" 2024-05-28T06:23:59.277 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:23:59.278 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:23:59.278 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "subvol_clone36082" 2024-05-28T06:23:59.278 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:23:59.278 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:23:59.278 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "sv1" 2024-05-28T06:23:59.278 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:23:59.278 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:23:59.278 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "subvol_clone36083" 2024-05-28T06:23:59.278 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:23:59.278 INFO:teuthology.orchestra.run.smithi096.stdout:] 2024-05-28T06:23:59.280 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.285+0000 7f9e1f200640 1 -- 172.21.15.96:0/4220396604 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9dfc06e3f0 msgr2=0x7f9dfc0708b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:59.280 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.285+0000 7f9e1f200640 1 --2- 172.21.15.96:0/4220396604 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9dfc06e3f0 0x7f9dfc0708b0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f9e1000a750 tx=0x7f9e10008040 comp rx=0 tx=0).stop 2024-05-28T06:23:59.280 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.285+0000 7f9e1f200640 1 -- 172.21.15.96:0/4220396604 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9e20153510 msgr2=0x7f9e200994e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:59.280 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.285+0000 7f9e1f200640 1 --2- 172.21.15.96:0/4220396604 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9e20153510 0x7f9e200994e0 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7f9e08000c00 tx=0x7f9e080046c0 comp rx=0 tx=0).stop 2024-05-28T06:23:59.280 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.285+0000 7f9e1f200640 1 -- 172.21.15.96:0/4220396604 shutdown_connections 2024-05-28T06:23:59.280 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.285+0000 7f9e1f200640 1 --2- 172.21.15.96:0/4220396604 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9dfc06e3f0 0x7f9dfc0708b0 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:59.280 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.285+0000 7f9e1f200640 1 --2- 172.21.15.96:0/4220396604 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9e2015cb60 0x7f9e20099a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:59.280 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.285+0000 7f9e1f200640 1 --2- 172.21.15.96:0/4220396604 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9e20153510 0x7f9e200994e0 unknown :-1 s=CLOSED pgs=1895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:59.280 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.285+0000 7f9e1f200640 1 --2- 172.21.15.96:0/4220396604 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9e200cd2a0 0x7f9e200c9740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:59.280 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.285+0000 7f9e1f200640 1 -- 172.21.15.96:0/4220396604 >> 172.21.15.96:0/4220396604 conn(0x7f9e2006b900 msgr2=0x7f9e200ccb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:23:59.280 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.285+0000 7f9e1f200640 1 -- 172.21.15.96:0/4220396604 shutdown_connections 2024-05-28T06:23:59.281 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.285+0000 7f9e1f200640 1 -- 172.21.15.96:0/4220396604 wait complete. 2024-05-28T06:23:59.299 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone36081 2024-05-28T06:23:59.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.496+0000 7f5feb200640 1 Processor -- start 2024-05-28T06:23:59.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.496+0000 7f5feb200640 1 -- start start 2024-05-28T06:23:59.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.498+0000 7f5feb200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5fec15fa00 0x7f5fec06e970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:59.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.498+0000 7f5feb200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f5fec1600f0 0x7f5fec0d0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:59.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.498+0000 7f5feb200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5fec0d0d90 con 0x7f5fec0d0970 2024-05-28T06:23:59.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.498+0000 7f5feb200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f5fec06fa00 con 0x7f5fec1600f0 2024-05-28T06:23:59.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.498+0000 7f5feb200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f5fec06eeb0 con 0x7f5fec15fa00 2024-05-28T06:23:59.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.498+0000 7f5fe9400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f5fec1600f0 0x7f5fec0d0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:59.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.498+0000 7f5fe9400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f5fec1600f0 0x7f5fec0d0430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:42934/0 (socket says 172.21.15.96:42934) 2024-05-28T06:23:59.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.498+0000 7f5fe9400640 1 -- 172.21.15.96:0/1054431827 learned_addr learned my addr 172.21.15.96:0/1054431827 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:23:59.494 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.499+0000 7f5fe9e00640 1 --2- 172.21.15.96:0/1054431827 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5fec15fa00 0x7f5fec06e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:59.494 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.499+0000 7f5fe9400640 1 -- 172.21.15.96:0/1054431827 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5fec15fa00 msgr2=0x7f5fec06e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:23:59.494 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.499+0000 7f5fe9400640 1 --2- 172.21.15.96:0/1054431827 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5fec15fa00 0x7f5fec06e970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:59.494 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.499+0000 7f5fe9400640 1 -- 172.21.15.96:0/1054431827 >> v1:172.21.15.123:6789/0 conn(0x7f5fec0d0970 legacy=0x7f5fec0b3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:23:59.494 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.499+0000 7f5fe9400640 1 -- 172.21.15.96:0/1054431827 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fec06f160 con 0x7f5fec1600f0 2024-05-28T06:23:59.495 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.500+0000 7f5fe9400640 1 --2- 172.21.15.96:0/1054431827 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5fec1600f0 0x7f5fec0d0430 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7f5fd8009870 tx=0x7f5fd802a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f52ed12fd4c32d server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:23:59.495 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.500+0000 7f5fe8a00640 1 -- 172.21.15.96:0/1054431827 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fd8037070 con 0x7f5fec1600f0 2024-05-28T06:23:59.496 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.500+0000 7f5fe8a00640 1 -- 172.21.15.96:0/1054431827 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5fd8032440 con 0x7f5fec1600f0 2024-05-28T06:23:59.496 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.500+0000 7f5fe8a00640 1 -- 172.21.15.96:0/1054431827 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fd8032810 con 0x7f5fec1600f0 2024-05-28T06:23:59.496 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.501+0000 7f5feb200640 1 -- 172.21.15.96:0/1054431827 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5fec1600f0 msgr2=0x7f5fec0d0430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:59.496 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.501+0000 7f5feb200640 1 --2- 172.21.15.96:0/1054431827 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5fec1600f0 0x7f5fec0d0430 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7f5fd8009870 tx=0x7f5fd802a180 comp rx=0 tx=0).stop 2024-05-28T06:23:59.496 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.501+0000 7f5feb200640 1 -- 172.21.15.96:0/1054431827 shutdown_connections 2024-05-28T06:23:59.496 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.501+0000 7f5feb200640 1 --2- 172.21.15.96:0/1054431827 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5fec1600f0 0x7f5fec0d0430 unknown :-1 s=CLOSED pgs=1896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:59.496 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.501+0000 7f5feb200640 1 --2- 172.21.15.96:0/1054431827 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5fec15fa00 0x7f5fec06e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:59.496 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.501+0000 7f5feb200640 1 -- 172.21.15.96:0/1054431827 >> 172.21.15.96:0/1054431827 conn(0x7f5fec06bae0 msgr2=0x7f5fec06bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:23:59.496 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.501+0000 7f5feb200640 1 -- 172.21.15.96:0/1054431827 shutdown_connections 2024-05-28T06:23:59.496 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.501+0000 7f5feb200640 1 -- 172.21.15.96:0/1054431827 wait complete. 2024-05-28T06:23:59.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.503+0000 7f5feb200640 1 Processor -- start 2024-05-28T06:23:59.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.503+0000 7f5feb200640 1 -- start start 2024-05-28T06:23:59.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.505+0000 7f5feb200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5fec0d0970 0x7f5fec072ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:59.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.505+0000 7f5feb200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5fec15fa00 0x7f5fec1e65b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:59.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.505+0000 7f5feb200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5fec1600f0 0x7f5fec158350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:59.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.505+0000 7f5feb200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f5fec071c70 con 0x7f5fec1600f0 2024-05-28T06:23:59.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.505+0000 7f5feb200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5fec06f720 con 0x7f5fec15fa00 2024-05-28T06:23:59.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.505+0000 7f5feb200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f5fec15fde0 con 0x7f5fec0d0970 2024-05-28T06:23:59.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.505+0000 7f5fea800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5fec1600f0 0x7f5fec158350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:59.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.505+0000 7f5fea800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5fec1600f0 0x7f5fec158350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:42936/0 (socket says 172.21.15.96:42936) 2024-05-28T06:23:59.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.505+0000 7f5fea800640 1 -- 172.21.15.96:0/1706300376 learned_addr learned my addr 172.21.15.96:0/1706300376 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:23:59.501 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.505+0000 7f5fe9e00640 1 --2- 172.21.15.96:0/1706300376 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5fec0d0970 0x7f5fec072ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:59.501 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.505+0000 7f5fea800640 1 -- 172.21.15.96:0/1706300376 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5fec0d0970 msgr2=0x7f5fec072ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:59.501 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.505+0000 7f5fea800640 1 --2- 172.21.15.96:0/1706300376 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5fec0d0970 0x7f5fec072ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:59.502 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.505+0000 7f5fea800640 1 -- 172.21.15.96:0/1706300376 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5fec15fa00 msgr2=0x7f5fec1e65b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:59.502 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.505+0000 7f5fe9400640 1 --2- 172.21.15.96:0/1706300376 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5fec15fa00 0x7f5fec1e65b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:59.502 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.505+0000 7f5fea800640 1 --2- 172.21.15.96:0/1706300376 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5fec15fa00 0x7f5fec1e65b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:59.502 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.505+0000 7f5fea800640 1 -- 172.21.15.96:0/1706300376 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fec1e7050 con 0x7f5fec1600f0 2024-05-28T06:23:59.502 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.506+0000 7f5fea800640 1 --2- 172.21.15.96:0/1706300376 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5fec1600f0 0x7f5fec158350 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7f5fe400b970 tx=0x7f5fe400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:23:59.502 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.506+0000 7f5fe2000640 1 -- 172.21.15.96:0/1706300376 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fe4019070 con 0x7f5fec1600f0 2024-05-28T06:23:59.502 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.506+0000 7f5fe2000640 1 -- 172.21.15.96:0/1706300376 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5fe400cb40 con 0x7f5fec1600f0 2024-05-28T06:23:59.502 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.506+0000 7f5fe2000640 1 -- 172.21.15.96:0/1706300376 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fe4014440 con 0x7f5fec1600f0 2024-05-28T06:23:59.502 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.506+0000 7f5feb200640 1 -- 172.21.15.96:0/1706300376 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fec0ad850 con 0x7f5fec1600f0 2024-05-28T06:23:59.502 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.506+0000 7f5feb200640 1 -- 172.21.15.96:0/1706300376 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fec0ade10 con 0x7f5fec1600f0 2024-05-28T06:23:59.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.507+0000 7f5feb200640 1 -- 172.21.15.96:0/1706300376 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5fec0689e0 con 0x7f5fec1600f0 2024-05-28T06:23:59.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.508+0000 7f5fe2000640 1 -- 172.21.15.96:0/1706300376 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 41) ==== 93488+0+0 (secure 0 0 0) 0x7f5fe401c070 con 0x7f5fec1600f0 2024-05-28T06:23:59.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.509+0000 7f5fe2000640 1 --2- 172.21.15.96:0/1706300376 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5fcc06e3f0 0x7f5fcc0708b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:23:59.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.509+0000 7f5fe2000640 1 -- 172.21.15.96:0/1706300376 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 5823+0+0 (secure 0 0 0) 0x7f5fe408a6b0 con 0x7f5fec1600f0 2024-05-28T06:23:59.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.510+0000 7f5fe9e00640 1 --2- 172.21.15.96:0/1706300376 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5fcc06e3f0 0x7f5fcc0708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:23:59.505 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.511+0000 7f5fe9e00640 1 --2- 172.21.15.96:0/1706300376 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5fcc06e3f0 0x7f5fcc0708b0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f5fdc00b7c0 tx=0x7f5fdc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:23:59.508 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.514+0000 7f5fe2000640 1 -- 172.21.15.96:0/1706300376 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5fe405a540 con 0x7f5fec1600f0 2024-05-28T06:23:59.732 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.737+0000 7f5feb200640 1 -- 172.21.15.96:0/1706300376 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone36081", "target": ["mon-mgr", ""]}) -- 0x7f5fec0cda60 con 0x7f5fcc06e3f0 2024-05-28T06:23:59.787 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.793+0000 7f5fe2000640 1 -- 172.21.15.96:0/1706300376 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f5fec0cda60 con 0x7f5fcc06e3f0 2024-05-28T06:23:59.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.795+0000 7f5feb200640 1 -- 172.21.15.96:0/1706300376 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5fcc06e3f0 msgr2=0x7f5fcc0708b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:59.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.795+0000 7f5feb200640 1 --2- 172.21.15.96:0/1706300376 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5fcc06e3f0 0x7f5fcc0708b0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f5fdc00b7c0 tx=0x7f5fdc008040 comp rx=0 tx=0).stop 2024-05-28T06:23:59.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.795+0000 7f5feb200640 1 -- 172.21.15.96:0/1706300376 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5fec1600f0 msgr2=0x7f5fec158350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:23:59.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.795+0000 7f5feb200640 1 --2- 172.21.15.96:0/1706300376 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5fec1600f0 0x7f5fec158350 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7f5fe400b970 tx=0x7f5fe400be30 comp rx=0 tx=0).stop 2024-05-28T06:23:59.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.795+0000 7f5feb200640 1 -- 172.21.15.96:0/1706300376 shutdown_connections 2024-05-28T06:23:59.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.795+0000 7f5feb200640 1 --2- 172.21.15.96:0/1706300376 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5fcc06e3f0 0x7f5fcc0708b0 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:59.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.796+0000 7f5feb200640 1 --2- 172.21.15.96:0/1706300376 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5fec1600f0 0x7f5fec158350 unknown :-1 s=CLOSED pgs=1897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:59.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.796+0000 7f5feb200640 1 --2- 172.21.15.96:0/1706300376 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5fec15fa00 0x7f5fec1e65b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:59.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.796+0000 7f5feb200640 1 --2- 172.21.15.96:0/1706300376 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5fec0d0970 0x7f5fec072ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:23:59.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.796+0000 7f5feb200640 1 -- 172.21.15.96:0/1706300376 >> 172.21.15.96:0/1706300376 conn(0x7f5fec06bae0 msgr2=0x7f5fec072090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:23:59.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.796+0000 7f5feb200640 1 -- 172.21.15.96:0/1706300376 shutdown_connections 2024-05-28T06:23:59.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:23:59.796+0000 7f5feb200640 1 -- 172.21.15.96:0/1706300376 wait complete. 2024-05-28T06:23:59.798 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:23:59.803+0000 7fea5d000640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:23:59.798 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:23:59.803+0000 7fea5d000640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:23:59.798 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:23:59.803+0000 7fea5d000640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:23:59.798 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:23:59.803+0000 7fea5d000640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:23:59.798 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:23:59.803+0000 7fea5d000640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:23:59.810 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone36082 2024-05-28T06:23:59.819 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:23:59.825+0000 7fea5bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:23:59.819 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:23:59.825+0000 7fea5bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:23:59.819 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:23:59.825+0000 7fea5bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:23:59.819 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:23:59.825+0000 7fea5bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:23:59.819 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:23:59.825+0000 7fea5bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:23:59.999 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.005+0000 7f2645000640 1 Processor -- start 2024-05-28T06:24:00.000 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.005+0000 7f2645000640 1 -- start start 2024-05-28T06:24:00.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.006+0000 7f2645000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f264015c2e0 0x7f26400b45e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:00.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.006+0000 7f2645000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f26400cef50 0x7f264015c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:00.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.006+0000 7f2645000640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26400b4c50 con 0x7f2640067a70 2024-05-28T06:24:00.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.006+0000 7f2645000640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f26400b4f30 con 0x7f264015c2e0 2024-05-28T06:24:00.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.006+0000 7f2645000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f26400b5210 con 0x7f26400cef50 2024-05-28T06:24:00.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.007+0000 7f2636a00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f2640067a70 0x7f26400ce840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:55874/0 (socket says 172.21.15.96:55874) 2024-05-28T06:24:00.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.007+0000 7f2636a00640 1 -- 172.21.15.96:0/705847511 learned_addr learned my addr 172.21.15.96:0/705847511 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:24:00.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.007+0000 7f263f400640 1 --2- 172.21.15.96:0/705847511 >> v2:172.21.15.123:3300/0 conn(0x7f264015c2e0 0x7f26400b45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:00.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.007+0000 7f263fe00640 1 --2- 172.21.15.96:0/705847511 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f26400cef50 0x7f264015c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:00.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.007+0000 7f263ea00640 1 -- 172.21.15.96:0/705847511 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3986443090 0 0) 0x7f26400b4c50 con 0x7f2640067a70 2024-05-28T06:24:00.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.007+0000 7f263ea00640 1 -- 172.21.15.96:0/705847511 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2624003520 con 0x7f2640067a70 2024-05-28T06:24:00.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.008+0000 7f263ea00640 1 -- 172.21.15.96:0/705847511 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 231185041 0 0) 0x7f2624003520 con 0x7f2640067a70 2024-05-28T06:24:00.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.008+0000 7f263ea00640 1 -- 172.21.15.96:0/705847511 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f26400cef50 msgr2=0x7f264015c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:00.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.008+0000 7f263ea00640 1 --2- 172.21.15.96:0/705847511 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f26400cef50 0x7f264015c6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:00.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.008+0000 7f263ea00640 1 -- 172.21.15.96:0/705847511 >> v2:172.21.15.123:3300/0 conn(0x7f264015c2e0 msgr2=0x7f26400b45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:00.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.008+0000 7f263ea00640 1 --2- 172.21.15.96:0/705847511 >> v2:172.21.15.123:3300/0 conn(0x7f264015c2e0 0x7f26400b45e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:00.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.008+0000 7f263ea00640 1 -- 172.21.15.96:0/705847511 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26400b54f0 con 0x7f2640067a70 2024-05-28T06:24:00.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.008+0000 7f263ea00640 1 -- 172.21.15.96:0/705847511 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f2628001690 con 0x7f2640067a70 2024-05-28T06:24:00.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.008+0000 7f263ea00640 1 -- 172.21.15.96:0/705847511 >> v1:172.21.15.96:6789/0 conn(0x7f2640067a70 legacy=0x7f26400ce840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:00.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.008+0000 7f263ea00640 1 --2- 172.21.15.96:0/705847511 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f26400cef50 0x7f26240039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:00.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.008+0000 7f263ea00640 1 --2- 172.21.15.96:0/705847511 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2624003f30 0x7f2624004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:00.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.008+0000 7f263ea00640 1 --2- 172.21.15.96:0/705847511 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2624004d40 0x7f26240051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:00.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.008+0000 7f263ea00640 1 -- 172.21.15.96:0/705847511 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f26400b4f30 con 0x7f2624003f30 2024-05-28T06:24:00.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.008+0000 7f263ea00640 1 -- 172.21.15.96:0/705847511 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f26400b5210 con 0x7f2624004d40 2024-05-28T06:24:00.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.008+0000 7f263ea00640 1 -- 172.21.15.96:0/705847511 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f262400bac0 con 0x7f26400cef50 2024-05-28T06:24:00.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.008+0000 7f2636a00640 1 --2- 172.21.15.96:0/705847511 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2624003f30 0x7f2624004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:00.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.008+0000 7f263f400640 1 --2- 172.21.15.96:0/705847511 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f26400cef50 0x7f26240039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:00.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.008+0000 7f2636a00640 1 -- 172.21.15.96:0/705847511 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f26400cef50 msgr2=0x7f26240039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:00.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.009+0000 7f2636a00640 1 --2- 172.21.15.96:0/705847511 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f26400cef50 0x7f26240039b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:00.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.009+0000 7f2636a00640 1 -- 172.21.15.96:0/705847511 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2624004d40 msgr2=0x7f26240051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:00.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.009+0000 7f2636a00640 1 --2- 172.21.15.96:0/705847511 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2624004d40 0x7f26240051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:00.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.009+0000 7f2636a00640 1 -- 172.21.15.96:0/705847511 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2624003520 con 0x7f2624003f30 2024-05-28T06:24:00.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.009+0000 7f2636a00640 1 --2- 172.21.15.96:0/705847511 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2624003f30 0x7f2624004800 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7f262800c210 tx=0x7f2624005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5baf6f07c68894f5 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:00.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.010+0000 7f263ea00640 1 -- 172.21.15.96:0/705847511 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2628007d50 con 0x7f2624003f30 2024-05-28T06:24:00.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.010+0000 7f263ea00640 1 -- 172.21.15.96:0/705847511 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26280058d0 con 0x7f2624003f30 2024-05-28T06:24:00.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.010+0000 7f263ea00640 1 -- 172.21.15.96:0/705847511 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2628005d20 con 0x7f2624003f30 2024-05-28T06:24:00.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.010+0000 7f2645000640 1 -- 172.21.15.96:0/705847511 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2624003f30 msgr2=0x7f2624004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:00.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.010+0000 7f2645000640 1 --2- 172.21.15.96:0/705847511 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2624003f30 0x7f2624004800 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7f262800c210 tx=0x7f2624005830 comp rx=0 tx=0).stop 2024-05-28T06:24:00.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.010+0000 7f263fe00640 1 -- 172.21.15.96:0/705847511 reap_dead start 2024-05-28T06:24:00.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.010+0000 7f2645000640 1 -- 172.21.15.96:0/705847511 shutdown_connections 2024-05-28T06:24:00.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.010+0000 7f2645000640 1 -- 172.21.15.96:0/705847511 >> 172.21.15.96:0/705847511 conn(0x7f2640073230 msgr2=0x7f264006b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:00.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.010+0000 7f2645000640 1 -- 172.21.15.96:0/705847511 shutdown_connections 2024-05-28T06:24:00.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.010+0000 7f2645000640 1 -- 172.21.15.96:0/705847511 wait complete. 2024-05-28T06:24:00.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.012+0000 7f2645000640 1 Processor -- start 2024-05-28T06:24:00.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.012+0000 7f2645000640 1 -- start start 2024-05-28T06:24:00.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.013+0000 7f2645000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2624003f30 0x7f26400a96d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:00.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.013+0000 7f2645000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f26401515b0 0x7f2640148350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:00.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.013+0000 7f2645000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26400ab010 0x7f26400ab490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:00.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.013+0000 7f2645000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f264015c6c0 con 0x7f26401515b0 2024-05-28T06:24:00.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.013+0000 7f2645000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f26400b5cf0 con 0x7f26400ab010 2024-05-28T06:24:00.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.013+0000 7f2645000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f264015c840 con 0x7f2624003f30 2024-05-28T06:24:00.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.013+0000 7f2636a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f26401515b0 0x7f2640148350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:00.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.013+0000 7f2636a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f26401515b0 0x7f2640148350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:42952/0 (socket says 172.21.15.96:42952) 2024-05-28T06:24:00.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.013+0000 7f2636a00640 1 -- 172.21.15.96:0/3512504638 learned_addr learned my addr 172.21.15.96:0/3512504638 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:24:00.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.014+0000 7f263fe00640 1 --2- 172.21.15.96:0/3512504638 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26400ab010 0x7f26400ab490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:00.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.014+0000 7f263f400640 1 --2- 172.21.15.96:0/3512504638 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2624003f30 0x7f26400a96d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:00.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.014+0000 7f2636a00640 1 -- 172.21.15.96:0/3512504638 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2624003f30 msgr2=0x7f26400a96d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:00.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.014+0000 7f2636a00640 1 --2- 172.21.15.96:0/3512504638 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2624003f30 0x7f26400a96d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:00.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.014+0000 7f2636a00640 1 -- 172.21.15.96:0/3512504638 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26400ab010 msgr2=0x7f26400ab490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:00.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.014+0000 7f2636a00640 1 --2- 172.21.15.96:0/3512504638 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26400ab010 0x7f26400ab490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:00.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.014+0000 7f2636a00640 1 -- 172.21.15.96:0/3512504638 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26400a9c10 con 0x7f26401515b0 2024-05-28T06:24:00.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.014+0000 7f2636a00640 1 --2- 172.21.15.96:0/3512504638 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f26401515b0 0x7f2640148350 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x7f2628001990 tx=0x7f26400b57d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:00.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.014+0000 7f2637e00640 1 -- 172.21.15.96:0/3512504638 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2628038040 con 0x7f26401515b0 2024-05-28T06:24:00.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.014+0000 7f2637e00640 1 -- 172.21.15.96:0/3512504638 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2628007070 con 0x7f26401515b0 2024-05-28T06:24:00.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.014+0000 7f2637e00640 1 -- 172.21.15.96:0/3512504638 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2628001c90 con 0x7f26401515b0 2024-05-28T06:24:00.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.014+0000 7f2645000640 1 -- 172.21.15.96:0/3512504638 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2640148890 con 0x7f26401515b0 2024-05-28T06:24:00.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.015+0000 7f2645000640 1 -- 172.21.15.96:0/3512504638 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26400aa190 con 0x7f26401515b0 2024-05-28T06:24:00.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.016+0000 7f2645000640 1 -- 172.21.15.96:0/3512504638 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26400690f0 con 0x7f26401515b0 2024-05-28T06:24:00.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.017+0000 7f2637e00640 1 -- 172.21.15.96:0/3512504638 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 41) ==== 93488+0+0 (secure 0 0 0) 0x7f2628007440 con 0x7f26401515b0 2024-05-28T06:24:00.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.018+0000 7f2637e00640 1 --2- 172.21.15.96:0/3512504638 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f261c06e3f0 0x7f261c0708b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:00.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.018+0000 7f2637e00640 1 -- 172.21.15.96:0/3512504638 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 5823+0+0 (secure 0 0 0) 0x7f26280aae20 con 0x7f26401515b0 2024-05-28T06:24:00.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.018+0000 7f263f400640 1 --2- 172.21.15.96:0/3512504638 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f261c06e3f0 0x7f261c0708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:00.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.019+0000 7f263f400640 1 --2- 172.21.15.96:0/3512504638 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f261c06e3f0 0x7f261c0708b0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f2630009990 tx=0x7f2630008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:00.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.022+0000 7f2637e00640 1 -- 172.21.15.96:0/3512504638 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2628031370 con 0x7f26401515b0 2024-05-28T06:24:00.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.251+0000 7f2645000640 1 -- 172.21.15.96:0/3512504638 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone36082", "target": ["mon-mgr", ""]}) -- 0x7f264015e5e0 con 0x7f261c06e3f0 2024-05-28T06:24:00.298 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.303+0000 7f2637e00640 1 -- 172.21.15.96:0/3512504638 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f264015e5e0 con 0x7f261c06e3f0 2024-05-28T06:24:00.300 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.306+0000 7f2645000640 1 -- 172.21.15.96:0/3512504638 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f261c06e3f0 msgr2=0x7f261c0708b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:00.300 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.306+0000 7f2645000640 1 --2- 172.21.15.96:0/3512504638 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f261c06e3f0 0x7f261c0708b0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f2630009990 tx=0x7f2630008040 comp rx=0 tx=0).stop 2024-05-28T06:24:00.301 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.306+0000 7f2645000640 1 -- 172.21.15.96:0/3512504638 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f26401515b0 msgr2=0x7f2640148350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:00.301 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.306+0000 7f2645000640 1 --2- 172.21.15.96:0/3512504638 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f26401515b0 0x7f2640148350 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x7f2628001990 tx=0x7f26400b57d0 comp rx=0 tx=0).stop 2024-05-28T06:24:00.301 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.306+0000 7f2645000640 1 -- 172.21.15.96:0/3512504638 shutdown_connections 2024-05-28T06:24:00.301 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.306+0000 7f2645000640 1 --2- 172.21.15.96:0/3512504638 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f261c06e3f0 0x7f261c0708b0 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:00.301 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.306+0000 7f2645000640 1 --2- 172.21.15.96:0/3512504638 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26400ab010 0x7f26400ab490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:00.302 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.306+0000 7f2645000640 1 --2- 172.21.15.96:0/3512504638 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f26401515b0 0x7f2640148350 unknown :-1 s=CLOSED pgs=1901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:00.302 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.306+0000 7f2645000640 1 --2- 172.21.15.96:0/3512504638 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2624003f30 0x7f26400a96d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:00.302 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.306+0000 7f2645000640 1 -- 172.21.15.96:0/3512504638 >> 172.21.15.96:0/3512504638 conn(0x7f2640073230 msgr2=0x7f264015b5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:00.302 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.306+0000 7f2645000640 1 -- 172.21.15.96:0/3512504638 shutdown_connections 2024-05-28T06:24:00.302 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.307+0000 7f2645000640 1 -- 172.21.15.96:0/3512504638 wait complete. 2024-05-28T06:24:00.321 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot ls cephfs sv1 --format json 2024-05-28T06:24:00.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.526+0000 7f93b9c00640 1 Processor -- start 2024-05-28T06:24:00.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.527+0000 7f93b9c00640 1 -- start start 2024-05-28T06:24:00.523 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.528+0000 7f93b9c00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f93b415fa00 0x7f93b406e970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:00.523 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.528+0000 7f93b9c00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f93b40d0b40 0x7f93b4072190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:00.523 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.528+0000 7f93b9c00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93b406fa00 con 0x7f93b41600f0 2024-05-28T06:24:00.523 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.528+0000 7f93b9c00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f93b406eeb0 con 0x7f93b40d0b40 2024-05-28T06:24:00.523 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.528+0000 7f93b9c00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f93b406f160 con 0x7f93b415fa00 2024-05-28T06:24:00.524 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.528+0000 7f93b9200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f93b40d0b40 0x7f93b4072190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:00.524 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.528+0000 7f93ab400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f93b41600f0 0x7f93b40d0430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:55876/0 (socket says 172.21.15.96:55876) 2024-05-28T06:24:00.524 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.528+0000 7f93ab400640 1 -- 172.21.15.96:0/3343614245 learned_addr learned my addr 172.21.15.96:0/3343614245 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:24:00.525 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.529+0000 7f93b3e00640 1 --2- 172.21.15.96:0/3343614245 >> v2:172.21.15.123:3300/0 conn(0x7f93b415fa00 0x7f93b406e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:00.525 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.529+0000 7f93b9200640 1 -- 172.21.15.96:0/3343614245 >> v2:172.21.15.123:3300/0 conn(0x7f93b415fa00 msgr2=0x7f93b406e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:00.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.529+0000 7f93b9200640 1 --2- 172.21.15.96:0/3343614245 >> v2:172.21.15.123:3300/0 conn(0x7f93b415fa00 0x7f93b406e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:00.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.529+0000 7f93b9200640 1 -- 172.21.15.96:0/3343614245 >> v1:172.21.15.96:6789/0 conn(0x7f93b41600f0 legacy=0x7f93b40d0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:00.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.529+0000 7f93b9200640 1 -- 172.21.15.96:0/3343614245 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93b406f440 con 0x7f93b40d0b40 2024-05-28T06:24:00.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.529+0000 7f93b3400640 1 -- 172.21.15.96:0/3343614245 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2376088937 0 0) 0x7f93b406fa00 con 0x7f93b41600f0 2024-05-28T06:24:00.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.530+0000 7f93b9200640 1 --2- 172.21.15.96:0/3343614245 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f93b40d0b40 0x7f93b4072190 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7f93b40d0130 tx=0x7f93ac02a560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddece864616a0eae server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:00.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.531+0000 7f93b3400640 1 -- 172.21.15.96:0/3343614245 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93ac007740 con 0x7f93b40d0b40 2024-05-28T06:24:00.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.531+0000 7f93b3400640 1 -- 172.21.15.96:0/3343614245 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93ac0078e0 con 0x7f93b40d0b40 2024-05-28T06:24:00.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.531+0000 7f93b3400640 1 -- 172.21.15.96:0/3343614245 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93ac007cb0 con 0x7f93b40d0b40 2024-05-28T06:24:00.528 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.531+0000 7f93b9c00640 1 -- 172.21.15.96:0/3343614245 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f93b40d0b40 msgr2=0x7f93b4072190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:00.528 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.531+0000 7f93b9c00640 1 --2- 172.21.15.96:0/3343614245 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f93b40d0b40 0x7f93b4072190 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7f93b40d0130 tx=0x7f93ac02a560 comp rx=0 tx=0).stop 2024-05-28T06:24:00.528 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.531+0000 7f93b9c00640 1 -- 172.21.15.96:0/3343614245 shutdown_connections 2024-05-28T06:24:00.528 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.531+0000 7f93b9c00640 1 --2- 172.21.15.96:0/3343614245 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f93b40d0b40 0x7f93b4072190 unknown :-1 s=CLOSED pgs=1903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:00.528 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.531+0000 7f93b9c00640 1 --2- 172.21.15.96:0/3343614245 >> v2:172.21.15.123:3300/0 conn(0x7f93b415fa00 0x7f93b406e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:00.528 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.531+0000 7f93b9c00640 1 -- 172.21.15.96:0/3343614245 >> 172.21.15.96:0/3343614245 conn(0x7f93b406bae0 msgr2=0x7f93b406bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:00.528 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.532+0000 7f93b9c00640 1 -- 172.21.15.96:0/3343614245 shutdown_connections 2024-05-28T06:24:00.528 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.532+0000 7f93b9c00640 1 -- 172.21.15.96:0/3343614245 wait complete. 2024-05-28T06:24:00.528 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.533+0000 7f93b9c00640 1 Processor -- start 2024-05-28T06:24:00.529 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.533+0000 7f93b9c00640 1 -- start start 2024-05-28T06:24:00.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.534+0000 7f93b9c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f93b40d0b40 0x7f93b40cd2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:00.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.534+0000 7f93b9c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f93b415fa00 0x7f93b4156a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:00.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.534+0000 7f93b9c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f93b41600f0 0x7f93b40c9540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:00.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.534+0000 7f93b9c00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f93b415fde0 con 0x7f93b415fa00 2024-05-28T06:24:00.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.534+0000 7f93b9c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f93b4071c70 con 0x7f93b41600f0 2024-05-28T06:24:00.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.534+0000 7f93b9c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f93b40726d0 con 0x7f93b40d0b40 2024-05-28T06:24:00.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.535+0000 7f93ab400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f93b415fa00 0x7f93b4156a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:00.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.535+0000 7f93ab400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f93b415fa00 0x7f93b4156a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:42974/0 (socket says 172.21.15.96:42974) 2024-05-28T06:24:00.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.535+0000 7f93ab400640 1 -- 172.21.15.96:0/3435738713 learned_addr learned my addr 172.21.15.96:0/3435738713 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:24:00.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.535+0000 7f93b9200640 1 --2- 172.21.15.96:0/3435738713 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f93b41600f0 0x7f93b40c9540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:00.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.535+0000 7f93b3e00640 1 --2- 172.21.15.96:0/3435738713 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f93b40d0b40 0x7f93b40cd2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:00.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.535+0000 7f93ab400640 1 -- 172.21.15.96:0/3435738713 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f93b40d0b40 msgr2=0x7f93b40cd2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:00.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.535+0000 7f93ab400640 1 --2- 172.21.15.96:0/3435738713 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f93b40d0b40 0x7f93b40cd2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:00.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.535+0000 7f93ab400640 1 -- 172.21.15.96:0/3435738713 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f93b41600f0 msgr2=0x7f93b40c9540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:00.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.535+0000 7f93ab400640 1 --2- 172.21.15.96:0/3435738713 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f93b41600f0 0x7f93b40c9540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:00.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.535+0000 7f93ab400640 1 -- 172.21.15.96:0/3435738713 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93b41f8130 con 0x7f93b415fa00 2024-05-28T06:24:00.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.535+0000 7f93b9200640 1 --2- 172.21.15.96:0/3435738713 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f93b41600f0 0x7f93b40c9540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:24:00.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.535+0000 7f93ab400640 1 --2- 172.21.15.96:0/3435738713 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f93b415fa00 0x7f93b4156a00 secure :-1 s=READY pgs=1904 cs=0 l=1 rev1=1 crypto rx=0x7f939c00c400 tx=0x7f939c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:00.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.536+0000 7f93b0c00640 1 -- 172.21.15.96:0/3435738713 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f939c017070 con 0x7f93b415fa00 2024-05-28T06:24:00.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.536+0000 7f93b0c00640 1 -- 172.21.15.96:0/3435738713 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f939c005ce0 con 0x7f93b415fa00 2024-05-28T06:24:00.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.536+0000 7f93b9c00640 1 -- 172.21.15.96:0/3435738713 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93b40cc7c0 con 0x7f93b415fa00 2024-05-28T06:24:00.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.536+0000 7f93b0c00640 1 -- 172.21.15.96:0/3435738713 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f939c007310 con 0x7f93b415fa00 2024-05-28T06:24:00.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.536+0000 7f93b9c00640 1 -- 172.21.15.96:0/3435738713 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93b4156f40 con 0x7f93b415fa00 2024-05-28T06:24:00.533 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.537+0000 7f93b9c00640 1 -- 172.21.15.96:0/3435738713 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93b40689e0 con 0x7f93b415fa00 2024-05-28T06:24:00.533 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.538+0000 7f93b0c00640 1 -- 172.21.15.96:0/3435738713 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 41) ==== 93488+0+0 (secure 0 0 0) 0x7f939c01a070 con 0x7f93b415fa00 2024-05-28T06:24:00.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.539+0000 7f93b0c00640 1 --2- 172.21.15.96:0/3435738713 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f939006e3f0 0x7f93900708b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:00.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.540+0000 7f93b0c00640 1 -- 172.21.15.96:0/3435738713 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 5823+0+0 (secure 0 0 0) 0x7f939c089e20 con 0x7f93b415fa00 2024-05-28T06:24:00.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.540+0000 7f93b3e00640 1 --2- 172.21.15.96:0/3435738713 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f939006e3f0 0x7f93900708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:00.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.541+0000 7f93b3e00640 1 --2- 172.21.15.96:0/3435738713 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f939006e3f0 0x7f93900708b0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f93a4006fd0 tx=0x7f93a4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:00.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.544+0000 7f93b0c00640 1 -- 172.21.15.96:0/3435738713 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f939c059cb0 con 0x7f93b415fa00 2024-05-28T06:24:00.761 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.766+0000 7f93b0c00640 1 -- 172.21.15.96:0/3435738713 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mgrmap(e 42) ==== 93590+0+0 (secure 0 0 0) 0x7f939c055c20 con 0x7f93b415fa00 2024-05-28T06:24:00.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.784+0000 7f93b9c00640 1 -- 172.21.15.96:0/3435738713 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot ls", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f93b406e1e0 con 0x7f939006e3f0 2024-05-28T06:24:00.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.819+0000 7f93b0c00640 1 -- 172.21.15.96:0/3435738713 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f93b406e1e0 con 0x7f939006e3f0 2024-05-28T06:24:00.815 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:24:00.815 INFO:teuthology.orchestra.run.smithi096.stdout:[ 2024-05-28T06:24:00.815 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:24:00.816 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "ss1" 2024-05-28T06:24:00.816 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:24:00.816 INFO:teuthology.orchestra.run.smithi096.stdout:] 2024-05-28T06:24:00.817 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.822+0000 7f93b9c00640 1 -- 172.21.15.96:0/3435738713 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f939006e3f0 msgr2=0x7f93900708b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:00.817 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.822+0000 7f93b9c00640 1 --2- 172.21.15.96:0/3435738713 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f939006e3f0 0x7f93900708b0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f93a4006fd0 tx=0x7f93a4008040 comp rx=0 tx=0).stop 2024-05-28T06:24:00.817 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.822+0000 7f93b9c00640 1 -- 172.21.15.96:0/3435738713 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f93b415fa00 msgr2=0x7f93b4156a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:00.817 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.822+0000 7f93b9c00640 1 --2- 172.21.15.96:0/3435738713 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f93b415fa00 0x7f93b4156a00 secure :-1 s=READY pgs=1904 cs=0 l=1 rev1=1 crypto rx=0x7f939c00c400 tx=0x7f939c00c8c0 comp rx=0 tx=0).stop 2024-05-28T06:24:00.817 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.822+0000 7f93b9c00640 1 -- 172.21.15.96:0/3435738713 shutdown_connections 2024-05-28T06:24:00.817 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.822+0000 7f93b9c00640 1 --2- 172.21.15.96:0/3435738713 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f939006e3f0 0x7f93900708b0 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:00.817 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.822+0000 7f93b9c00640 1 --2- 172.21.15.96:0/3435738713 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f93b41600f0 0x7f93b40c9540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:00.817 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.822+0000 7f93b9c00640 1 --2- 172.21.15.96:0/3435738713 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f93b415fa00 0x7f93b4156a00 unknown :-1 s=CLOSED pgs=1904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:00.817 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.822+0000 7f93b9c00640 1 --2- 172.21.15.96:0/3435738713 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f93b40d0b40 0x7f93b40cd2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:00.817 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.822+0000 7f93b9c00640 1 -- 172.21.15.96:0/3435738713 >> 172.21.15.96:0/3435738713 conn(0x7f93b406bae0 msgr2=0x7f93b40d0400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:00.818 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.822+0000 7f93b9c00640 1 -- 172.21.15.96:0/3435738713 shutdown_connections 2024-05-28T06:24:00.818 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:00.822+0000 7f93b9c00640 1 -- 172.21.15.96:0/3435738713 wait complete. 2024-05-28T06:24:00.837 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-05-28T06:24:01.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.052+0000 7fbe20600640 1 Processor -- start 2024-05-28T06:24:01.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.052+0000 7fbe20600640 1 -- start start 2024-05-28T06:24:01.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.054+0000 7fbe20600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fbe18161a90 0x7fbe1806e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:01.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.054+0000 7fbe20600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fbe18161e70 0x7fbe180b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:01.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.054+0000 7fbe20600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe180b3d40 con 0x7fbe1806e9b0 2024-05-28T06:24:01.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.054+0000 7fbe20600640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fbe180b4020 con 0x7fbe18161a90 2024-05-28T06:24:01.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.054+0000 7fbe20600640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fbe180b4300 con 0x7fbe18161e70 2024-05-28T06:24:01.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.054+0000 7fbe1dc00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fbe18161a90 0x7fbe1806e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:01.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.054+0000 7fbe1dc00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fbe18161a90 0x7fbe1806e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:42990/0 (socket says 172.21.15.96:42990) 2024-05-28T06:24:01.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.054+0000 7fbe1dc00640 1 -- 172.21.15.96:0/79653141 learned_addr learned my addr 172.21.15.96:0/79653141 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:24:01.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.055+0000 7fbe1dc00640 1 -- 172.21.15.96:0/79653141 >> v2:172.21.15.123:3300/0 conn(0x7fbe18161e70 msgr2=0x7fbe180b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:01.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.055+0000 7fbe1dc00640 1 --2- 172.21.15.96:0/79653141 >> v2:172.21.15.123:3300/0 conn(0x7fbe18161e70 0x7fbe180b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:01.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.055+0000 7fbe1dc00640 1 -- 172.21.15.96:0/79653141 >> v1:172.21.15.123:6789/0 conn(0x7fbe1806e9b0 legacy=0x7fbe180ad2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:01.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.055+0000 7fbe1dc00640 1 -- 172.21.15.96:0/79653141 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe1806dc70 con 0x7fbe18161a90 2024-05-28T06:24:01.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.056+0000 7fbe1dc00640 1 --2- 172.21.15.96:0/79653141 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe18161a90 0x7fbe1806e470 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7fbe180d1350 tx=0x7fbe0c02a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cc9400eadf5450d server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:01.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.056+0000 7fbe17e00640 1 -- 172.21.15.96:0/79653141 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe0c037070 con 0x7fbe18161a90 2024-05-28T06:24:01.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.056+0000 7fbe17e00640 1 -- 172.21.15.96:0/79653141 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe0c032440 con 0x7fbe18161a90 2024-05-28T06:24:01.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.056+0000 7fbe17e00640 1 -- 172.21.15.96:0/79653141 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe0c032810 con 0x7fbe18161a90 2024-05-28T06:24:01.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.057+0000 7fbe20600640 1 -- 172.21.15.96:0/79653141 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe18161a90 msgr2=0x7fbe1806e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:01.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.057+0000 7fbe20600640 1 --2- 172.21.15.96:0/79653141 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe18161a90 0x7fbe1806e470 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7fbe180d1350 tx=0x7fbe0c02a230 comp rx=0 tx=0).stop 2024-05-28T06:24:01.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.057+0000 7fbe20600640 1 -- 172.21.15.96:0/79653141 shutdown_connections 2024-05-28T06:24:01.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.057+0000 7fbe20600640 1 --2- 172.21.15.96:0/79653141 >> v2:172.21.15.123:3300/0 conn(0x7fbe18161e70 0x7fbe180b32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:01.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.057+0000 7fbe20600640 1 --2- 172.21.15.96:0/79653141 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe18161a90 0x7fbe1806e470 unknown :-1 s=CLOSED pgs=1905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:01.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.057+0000 7fbe20600640 1 -- 172.21.15.96:0/79653141 >> 172.21.15.96:0/79653141 conn(0x7fbe1806bd20 msgr2=0x7fbe1806c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:01.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.058+0000 7fbe20600640 1 -- 172.21.15.96:0/79653141 shutdown_connections 2024-05-28T06:24:01.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.058+0000 7fbe20600640 1 -- 172.21.15.96:0/79653141 wait complete. 2024-05-28T06:24:01.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.059+0000 7fbe20600640 1 Processor -- start 2024-05-28T06:24:01.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.059+0000 7fbe20600640 1 -- start start 2024-05-28T06:24:01.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.060+0000 7fbe20600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbe1806e9b0 0x7fbe18099750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:01.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.060+0000 7fbe20600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe18161a90 0x7fbe1814d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:01.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.060+0000 7fbe20600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbe18161e70 0x7fbe180af530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:01.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.060+0000 7fbe20600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fbe180ad9b0 con 0x7fbe18161a90 2024-05-28T06:24:01.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.060+0000 7fbe20600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fbe180b5020 con 0x7fbe18161e70 2024-05-28T06:24:01.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.060+0000 7fbe20600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fbe180adb30 con 0x7fbe1806e9b0 2024-05-28T06:24:01.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.061+0000 7fbe1d200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe18161a90 0x7fbe1814d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:01.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.061+0000 7fbe1d200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe18161a90 0x7fbe1814d680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:43000/0 (socket says 172.21.15.96:43000) 2024-05-28T06:24:01.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.061+0000 7fbe1d200640 1 -- 172.21.15.96:0/3154016780 learned_addr learned my addr 172.21.15.96:0/3154016780 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:24:01.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.061+0000 7fbe1e600640 1 --2- 172.21.15.96:0/3154016780 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbe18161e70 0x7fbe180af530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:01.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.061+0000 7fbe1dc00640 1 --2- 172.21.15.96:0/3154016780 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbe1806e9b0 0x7fbe18099750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:01.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.061+0000 7fbe1d200640 1 -- 172.21.15.96:0/3154016780 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbe1806e9b0 msgr2=0x7fbe18099750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:01.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.061+0000 7fbe1d200640 1 --2- 172.21.15.96:0/3154016780 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbe1806e9b0 0x7fbe18099750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:01.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.061+0000 7fbe1d200640 1 -- 172.21.15.96:0/3154016780 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbe18161e70 msgr2=0x7fbe180af530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:01.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.061+0000 7fbe1d200640 1 --2- 172.21.15.96:0/3154016780 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbe18161e70 0x7fbe180af530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:01.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.061+0000 7fbe1d200640 1 -- 172.21.15.96:0/3154016780 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe1814eb50 con 0x7fbe18161a90 2024-05-28T06:24:01.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.061+0000 7fbe1e600640 1 --2- 172.21.15.96:0/3154016780 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbe18161e70 0x7fbe180af530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:24:01.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.061+0000 7fbe1d200640 1 --2- 172.21.15.96:0/3154016780 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe18161a90 0x7fbe1814d680 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7fbe0400b410 tx=0x7fbe0400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:01.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.061+0000 7fbe15600640 1 -- 172.21.15.96:0/3154016780 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe04015020 con 0x7fbe18161a90 2024-05-28T06:24:01.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.062+0000 7fbe15600640 1 -- 172.21.15.96:0/3154016780 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe04011850 con 0x7fbe18161a90 2024-05-28T06:24:01.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.062+0000 7fbe15600640 1 -- 172.21.15.96:0/3154016780 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe04011c40 con 0x7fbe18161a90 2024-05-28T06:24:01.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.062+0000 7fbe20600640 1 -- 172.21.15.96:0/3154016780 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe1814edf0 con 0x7fbe18161a90 2024-05-28T06:24:01.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.062+0000 7fbe20600640 1 -- 172.21.15.96:0/3154016780 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe1814f3b0 con 0x7fbe18161a90 2024-05-28T06:24:01.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.063+0000 7fbe20600640 1 -- 172.21.15.96:0/3154016780 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe180689e0 con 0x7fbe18161a90 2024-05-28T06:24:01.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.064+0000 7fbe15600640 1 -- 172.21.15.96:0/3154016780 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 42) ==== 93590+0+0 (secure 0 0 0) 0x7fbe04018020 con 0x7fbe18161a90 2024-05-28T06:24:01.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.065+0000 7fbe15600640 1 --2- 172.21.15.96:0/3154016780 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbdf806e510 0x7fbdf80709d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:01.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.066+0000 7fbe15600640 1 -- 172.21.15.96:0/3154016780 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 5823+0+0 (secure 0 0 0) 0x7fbe0408aa70 con 0x7fbe18161a90 2024-05-28T06:24:01.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.066+0000 7fbe1dc00640 1 --2- 172.21.15.96:0/3154016780 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbdf806e510 0x7fbdf80709d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:01.062 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.067+0000 7fbe1dc00640 1 --2- 172.21.15.96:0/3154016780 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbdf806e510 0x7fbdf80709d0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fbe0c034000 tx=0x7fbe0c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:01.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.070+0000 7fbe15600640 1 -- 172.21.15.96:0/3154016780 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbe0401f020 con 0x7fbe18161a90 2024-05-28T06:24:01.297 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.303+0000 7fbe20600640 1 -- 172.21.15.96:0/3154016780 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fbe181611b0 con 0x7fbdf806e510 2024-05-28T06:24:01.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.376+0000 7fbe15600640 1 -- 172.21.15.96:0/3154016780 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fbe181611b0 con 0x7fbdf806e510 2024-05-28T06:24:01.371 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:24:01.373 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.379+0000 7fbe20600640 1 -- 172.21.15.96:0/3154016780 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbdf806e510 msgr2=0x7fbdf80709d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:01.374 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.379+0000 7fbe20600640 1 --2- 172.21.15.96:0/3154016780 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbdf806e510 0x7fbdf80709d0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fbe0c034000 tx=0x7fbe0c02e000 comp rx=0 tx=0).stop 2024-05-28T06:24:01.374 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.379+0000 7fbe20600640 1 -- 172.21.15.96:0/3154016780 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe18161a90 msgr2=0x7fbe1814d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:01.374 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.379+0000 7fbe20600640 1 --2- 172.21.15.96:0/3154016780 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe18161a90 0x7fbe1814d680 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7fbe0400b410 tx=0x7fbe0400b8d0 comp rx=0 tx=0).stop 2024-05-28T06:24:01.374 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.379+0000 7fbe20600640 1 -- 172.21.15.96:0/3154016780 shutdown_connections 2024-05-28T06:24:01.374 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.379+0000 7fbe20600640 1 --2- 172.21.15.96:0/3154016780 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbdf806e510 0x7fbdf80709d0 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:01.374 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.379+0000 7fbe20600640 1 --2- 172.21.15.96:0/3154016780 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbe18161e70 0x7fbe180af530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:01.374 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.379+0000 7fbe20600640 1 --2- 172.21.15.96:0/3154016780 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe18161a90 0x7fbe1814d680 unknown :-1 s=CLOSED pgs=1906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:01.374 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.379+0000 7fbe20600640 1 --2- 172.21.15.96:0/3154016780 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbe1806e9b0 0x7fbe18099750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:01.374 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.379+0000 7fbe20600640 1 -- 172.21.15.96:0/3154016780 >> 172.21.15.96:0/3154016780 conn(0x7fbe1806bd20 msgr2=0x7fbe1815f190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:01.375 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.379+0000 7fbe20600640 1 -- 172.21.15.96:0/3154016780 shutdown_connections 2024-05-28T06:24:01.375 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.379+0000 7fbe20600640 1 -- 172.21.15.96:0/3154016780 wait complete. 2024-05-28T06:24:01.394 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs sv1 2024-05-28T06:24:01.596 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.601+0000 7f1075200640 1 Processor -- start 2024-05-28T06:24:01.596 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.602+0000 7f1075200640 1 -- start start 2024-05-28T06:24:01.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.603+0000 7f1075200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f1070145d80 0x7f10700cedf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:01.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.603+0000 7f1075200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f10701602b0 0x7f10700d0420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:01.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.603+0000 7f1075200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f10700d0960 0x7f107006b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:01.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.603+0000 7f1075200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f1070146160 con 0x7f10700d0960 2024-05-28T06:24:01.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.603+0000 7f1075200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f10700cf700 con 0x7f1070145d80 2024-05-28T06:24:01.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.603+0000 7f1075200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f10700cf9c0 con 0x7f10701602b0 2024-05-28T06:24:01.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.603+0000 7f106fe00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f10700d0960 0x7f107006b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:01.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.603+0000 7f106fe00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f10700d0960 0x7f107006b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:43016/0 (socket says 172.21.15.96:43016) 2024-05-28T06:24:01.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.603+0000 7f106fe00640 1 -- 172.21.15.96:0/1557116785 learned_addr learned my addr 172.21.15.96:0/1557116785 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:24:01.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.604+0000 7f1066a00640 1 --2- 172.21.15.96:0/1557116785 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f10701602b0 0x7f10700d0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:01.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.604+0000 7f106f400640 1 --2- 172.21.15.96:0/1557116785 >> v2:172.21.15.123:3300/0 conn(0x7f1070145d80 0x7f10700cedf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:01.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.604+0000 7f106fe00640 1 -- 172.21.15.96:0/1557116785 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f10701602b0 msgr2=0x7f10700d0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:01.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.604+0000 7f106fe00640 1 --2- 172.21.15.96:0/1557116785 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f10701602b0 0x7f10700d0420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:01.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.604+0000 7f106fe00640 1 -- 172.21.15.96:0/1557116785 >> v2:172.21.15.123:3300/0 conn(0x7f1070145d80 msgr2=0x7f10700cedf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:01.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.604+0000 7f106fe00640 1 --2- 172.21.15.96:0/1557116785 >> v2:172.21.15.123:3300/0 conn(0x7f1070145d80 0x7f10700cedf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:01.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.604+0000 7f106fe00640 1 -- 172.21.15.96:0/1557116785 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10700cfca0 con 0x7f10700d0960 2024-05-28T06:24:01.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.604+0000 7f106f400640 1 --2- 172.21.15.96:0/1557116785 >> v2:172.21.15.123:3300/0 conn(0x7f1070145d80 0x7f10700cedf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:24:01.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.605+0000 7f106fe00640 1 --2- 172.21.15.96:0/1557116785 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f10700d0960 0x7f107006b900 secure :-1 s=READY pgs=1907 cs=0 l=1 rev1=1 crypto rx=0x7f1068009e50 tx=0x7f106802a750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7e392897bc771a6 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:01.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.605+0000 7f106ea00640 1 -- 172.21.15.96:0/1557116785 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1068036070 con 0x7f10700d0960 2024-05-28T06:24:01.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.605+0000 7f106ea00640 1 -- 172.21.15.96:0/1557116785 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1068031600 con 0x7f10700d0960 2024-05-28T06:24:01.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.605+0000 7f106ea00640 1 -- 172.21.15.96:0/1557116785 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10680319d0 con 0x7f10700d0960 2024-05-28T06:24:01.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.606+0000 7f1075200640 1 -- 172.21.15.96:0/1557116785 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f10700d0960 msgr2=0x7f107006b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:01.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.606+0000 7f1075200640 1 --2- 172.21.15.96:0/1557116785 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f10700d0960 0x7f107006b900 secure :-1 s=READY pgs=1907 cs=0 l=1 rev1=1 crypto rx=0x7f1068009e50 tx=0x7f106802a750 comp rx=0 tx=0).stop 2024-05-28T06:24:01.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.606+0000 7f1075200640 1 -- 172.21.15.96:0/1557116785 shutdown_connections 2024-05-28T06:24:01.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.606+0000 7f1075200640 1 --2- 172.21.15.96:0/1557116785 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f10700d0960 0x7f107006b900 unknown :-1 s=CLOSED pgs=1907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:01.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.606+0000 7f1075200640 1 --2- 172.21.15.96:0/1557116785 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f10701602b0 0x7f10700d0420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:01.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.606+0000 7f1075200640 1 --2- 172.21.15.96:0/1557116785 >> v2:172.21.15.123:3300/0 conn(0x7f1070145d80 0x7f10700cedf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:01.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.606+0000 7f1075200640 1 -- 172.21.15.96:0/1557116785 >> 172.21.15.96:0/1557116785 conn(0x7f10700b2ed0 msgr2=0x7f10700b32e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:01.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.606+0000 7f1075200640 1 -- 172.21.15.96:0/1557116785 shutdown_connections 2024-05-28T06:24:01.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.606+0000 7f1075200640 1 -- 172.21.15.96:0/1557116785 wait complete. 2024-05-28T06:24:01.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.608+0000 7f1075200640 1 Processor -- start 2024-05-28T06:24:01.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.608+0000 7f1075200640 1 -- start start 2024-05-28T06:24:01.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.608+0000 7f1075200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f10700d0960 0x7f107015cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:01.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.609+0000 7f1075200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1070145d80 0x7f107015c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:01.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.609+0000 7f1075200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f107009c8a0 0x7f107015e200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:01.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.609+0000 7f1075200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f10700cf4b0 con 0x7f107009c8a0 2024-05-28T06:24:01.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.609+0000 7f1075200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f10700cf330 con 0x7f1070145d80 2024-05-28T06:24:01.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.609+0000 7f1075200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f107006be40 con 0x7f10700d0960 2024-05-28T06:24:01.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.609+0000 7f106fe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f107009c8a0 0x7f107015e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:01.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.609+0000 7f106fe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f107009c8a0 0x7f107015e200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:43018/0 (socket says 172.21.15.96:43018) 2024-05-28T06:24:01.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.609+0000 7f106fe00640 1 -- 172.21.15.96:0/2582027165 learned_addr learned my addr 172.21.15.96:0/2582027165 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:24:01.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.609+0000 7f1066a00640 1 --2- 172.21.15.96:0/2582027165 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1070145d80 0x7f107015c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:01.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.609+0000 7f106f400640 1 --2- 172.21.15.96:0/2582027165 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f10700d0960 0x7f107015cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:01.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.609+0000 7f106fe00640 1 -- 172.21.15.96:0/2582027165 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f10700d0960 msgr2=0x7f107015cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:01.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.609+0000 7f106fe00640 1 --2- 172.21.15.96:0/2582027165 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f10700d0960 0x7f107015cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:01.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.609+0000 7f106fe00640 1 -- 172.21.15.96:0/2582027165 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1070145d80 msgr2=0x7f107015c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:01.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.609+0000 7f106fe00640 1 --2- 172.21.15.96:0/2582027165 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1070145d80 0x7f107015c070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:01.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.609+0000 7f106fe00640 1 -- 172.21.15.96:0/2582027165 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f107015e740 con 0x7f107009c8a0 2024-05-28T06:24:01.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.609+0000 7f106f400640 1 --2- 172.21.15.96:0/2582027165 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f10700d0960 0x7f107015cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:24:01.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.610+0000 7f106fe00640 1 --2- 172.21.15.96:0/2582027165 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f107009c8a0 0x7f107015e200 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7f1068034040 tx=0x7f1068004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:01.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.610+0000 7f1067e00640 1 -- 172.21.15.96:0/2582027165 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f106803f020 con 0x7f107009c8a0 2024-05-28T06:24:01.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.610+0000 7f1067e00640 1 -- 172.21.15.96:0/2582027165 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1068037040 con 0x7f107009c8a0 2024-05-28T06:24:01.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.610+0000 7f1067e00640 1 -- 172.21.15.96:0/2582027165 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1068036070 con 0x7f107009c8a0 2024-05-28T06:24:01.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.610+0000 7f1075200640 1 -- 172.21.15.96:0/2582027165 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f107015e9e0 con 0x7f107009c8a0 2024-05-28T06:24:01.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.610+0000 7f1075200640 1 -- 172.21.15.96:0/2582027165 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10700b50b0 con 0x7f107009c8a0 2024-05-28T06:24:01.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.612+0000 7f1075200640 1 -- 172.21.15.96:0/2582027165 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f107015ef00 con 0x7f107009c8a0 2024-05-28T06:24:01.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.613+0000 7f1067e00640 1 -- 172.21.15.96:0/2582027165 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 42) ==== 93590+0+0 (secure 0 0 0) 0x7f1068042020 con 0x7f107009c8a0 2024-05-28T06:24:01.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.613+0000 7f1067e00640 1 --2- 172.21.15.96:0/2582027165 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f104c06e510 0x7f104c0709d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:01.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.614+0000 7f1067e00640 1 -- 172.21.15.96:0/2582027165 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 5823+0+0 (secure 0 0 0) 0x7f10680aaff0 con 0x7f107009c8a0 2024-05-28T06:24:01.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.614+0000 7f106f400640 1 --2- 172.21.15.96:0/2582027165 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f104c06e510 0x7f104c0709d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:01.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.615+0000 7f106f400640 1 --2- 172.21.15.96:0/2582027165 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f104c06e510 0x7f104c0709d0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f10600088b0 tx=0x7f1060008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:01.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.618+0000 7f1067e00640 1 -- 172.21.15.96:0/2582027165 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f107015ef00 con 0x7f107009c8a0 2024-05-28T06:24:01.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.854+0000 7f1075200640 1 -- 172.21.15.96:0/2582027165 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) -- 0x7f10700690f0 con 0x7f104c06e510 2024-05-28T06:24:01.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.901+0000 7f1067e00640 1 -- 172.21.15.96:0/2582027165 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f10700690f0 con 0x7f104c06e510 2024-05-28T06:24:01.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.903+0000 7f1075200640 1 -- 172.21.15.96:0/2582027165 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f104c06e510 msgr2=0x7f104c0709d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:01.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.903+0000 7f1075200640 1 --2- 172.21.15.96:0/2582027165 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f104c06e510 0x7f104c0709d0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f10600088b0 tx=0x7f1060008040 comp rx=0 tx=0).stop 2024-05-28T06:24:01.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.903+0000 7f1075200640 1 -- 172.21.15.96:0/2582027165 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f107009c8a0 msgr2=0x7f107015e200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:01.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.903+0000 7f1075200640 1 --2- 172.21.15.96:0/2582027165 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f107009c8a0 0x7f107015e200 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7f1068034040 tx=0x7f1068004a40 comp rx=0 tx=0).stop 2024-05-28T06:24:01.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.903+0000 7f1075200640 1 -- 172.21.15.96:0/2582027165 shutdown_connections 2024-05-28T06:24:01.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.903+0000 7f1075200640 1 --2- 172.21.15.96:0/2582027165 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f104c06e510 0x7f104c0709d0 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:01.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.903+0000 7f1075200640 1 --2- 172.21.15.96:0/2582027165 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f107009c8a0 0x7f107015e200 unknown :-1 s=CLOSED pgs=1908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:01.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.903+0000 7f1075200640 1 --2- 172.21.15.96:0/2582027165 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1070145d80 0x7f107015c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:01.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.903+0000 7f1075200640 1 --2- 172.21.15.96:0/2582027165 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f10700d0960 0x7f107015cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:01.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.903+0000 7f1075200640 1 -- 172.21.15.96:0/2582027165 >> 172.21.15.96:0/2582027165 conn(0x7f10700b2ed0 msgr2=0x7f10700cd100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:01.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.903+0000 7f1075200640 1 -- 172.21.15.96:0/2582027165 shutdown_connections 2024-05-28T06:24:01.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:01.903+0000 7f1075200640 1 -- 172.21.15.96:0/2582027165 wait complete. 2024-05-28T06:24:01.923 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone36083 2024-05-28T06:24:02.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.112+0000 7fdfb1000640 1 Processor -- start 2024-05-28T06:24:02.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.112+0000 7fdfb1000640 1 -- start start 2024-05-28T06:24:02.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.114+0000 7fdfb1000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fdfac0cfb40 0x7fdfac0d0d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:02.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.114+0000 7fdfb1000640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdfac0d2200 con 0x7fdfac15e5a0 2024-05-28T06:24:02.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.114+0000 7fdfb1000640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdfac0d24e0 con 0x7fdfac0cffd0 2024-05-28T06:24:02.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.114+0000 7fdfb1000640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fdfac0d2b30 con 0x7fdfac0cfb40 2024-05-28T06:24:02.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.114+0000 7fdfab400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fdfac15e5a0 0x7fdfac0d2fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:55880/0 (socket says 172.21.15.96:55880) 2024-05-28T06:24:02.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.114+0000 7fdfab400640 1 -- 172.21.15.96:0/845385022 learned_addr learned my addr 172.21.15.96:0/845385022 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:24:02.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.114+0000 7fdfaa000640 1 -- 172.21.15.96:0/845385022 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 798148034 0 0) 0x7fdfac0d2200 con 0x7fdfac15e5a0 2024-05-28T06:24:02.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.114+0000 7fdfaa000640 1 -- 172.21.15.96:0/845385022 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdf90003520 con 0x7fdfac15e5a0 2024-05-28T06:24:02.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.115+0000 7fdfaaa00640 1 --2- 172.21.15.96:0/845385022 >> v2:172.21.15.123:3300/0 conn(0x7fdfac0cfb40 0x7fdfac0d0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:02.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.115+0000 7fdfaa000640 1 -- 172.21.15.96:0/845385022 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4253540494 0 0) 0x7fdf90003520 con 0x7fdfac15e5a0 2024-05-28T06:24:02.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.115+0000 7fdfaa000640 1 -- 172.21.15.96:0/845385022 >> v2:172.21.15.123:3300/0 conn(0x7fdfac0cfb40 msgr2=0x7fdfac0d0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:02.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.115+0000 7fdfaa000640 1 --2- 172.21.15.96:0/845385022 >> v2:172.21.15.123:3300/0 conn(0x7fdfac0cfb40 0x7fdfac0d0d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:02.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.115+0000 7fdfaa000640 1 -- 172.21.15.96:0/845385022 >> v1:172.21.15.123:6789/0 conn(0x7fdfac0cffd0 legacy=0x7fdfac06c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:02.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.115+0000 7fdfaa000640 1 -- 172.21.15.96:0/845385022 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfac0d27c0 con 0x7fdfac15e5a0 2024-05-28T06:24:02.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.115+0000 7fdfaa000640 1 -- 172.21.15.96:0/845385022 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7fdf9c0019f0 con 0x7fdfac15e5a0 2024-05-28T06:24:02.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.115+0000 7fdfaaa00640 1 --2- 172.21.15.96:0/845385022 >> v2:172.21.15.123:3300/0 conn(0x7fdfac0cfb40 0x7fdfac0d0d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:24:02.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.115+0000 7fdfaa000640 1 -- 172.21.15.96:0/845385022 >> v1:172.21.15.96:6789/0 conn(0x7fdfac15e5a0 legacy=0x7fdfac0d2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:02.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.115+0000 7fdfaa000640 1 --2- 172.21.15.96:0/845385022 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdf90003a00 0x7fdf90003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:02.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.115+0000 7fdfaa000640 1 --2- 172.21.15.96:0/845385022 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdf90004800 0x7fdf90004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:02.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.115+0000 7fdfaa000640 1 --2- 172.21.15.96:0/845385022 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdf900052e0 0x7fdf9000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:02.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.115+0000 7fdfaa000640 1 -- 172.21.15.96:0/845385022 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fdf9000c200 con 0x7fdf900052e0 2024-05-28T06:24:02.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.115+0000 7fdfaa000640 1 -- 172.21.15.96:0/845385022 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fdf9000c3b0 con 0x7fdf90003a00 2024-05-28T06:24:02.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.115+0000 7fdfaa000640 1 -- 172.21.15.96:0/845385022 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fdf9000c5b0 con 0x7fdf90004800 2024-05-28T06:24:02.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.115+0000 7fdfabe00640 1 --2- 172.21.15.96:0/845385022 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdf900052e0 0x7fdf9000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:02.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.116+0000 7fdfab400640 1 --2- 172.21.15.96:0/845385022 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdf90003a00 0x7fdf90003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:02.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.116+0000 7fdfabe00640 1 -- 172.21.15.96:0/845385022 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdf90004800 msgr2=0x7fdf90004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:02.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.116+0000 7fdfabe00640 1 --2- 172.21.15.96:0/845385022 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdf90004800 0x7fdf90004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:02.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.116+0000 7fdfabe00640 1 -- 172.21.15.96:0/845385022 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdf90003a00 msgr2=0x7fdf90003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:02.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.116+0000 7fdfabe00640 1 --2- 172.21.15.96:0/845385022 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdf90003a00 0x7fdf90003e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:02.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.116+0000 7fdfabe00640 1 -- 172.21.15.96:0/845385022 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf90003520 con 0x7fdf900052e0 2024-05-28T06:24:02.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.117+0000 7fdfabe00640 1 -- 172.21.15.96:0/845385022 reap_dead start 2024-05-28T06:24:02.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.117+0000 7fdfabe00640 1 --2- 172.21.15.96:0/845385022 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdf900052e0 0x7fdf9000b9c0 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 crypto rx=0x7fdfa4003d80 tx=0x7fdf90005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=456468877084db72 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:02.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.117+0000 7fdfaa000640 1 -- 172.21.15.96:0/845385022 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfa4003980 con 0x7fdf900052e0 2024-05-28T06:24:02.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.117+0000 7fdfaa000640 1 -- 172.21.15.96:0/845385022 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdfa4007240 con 0x7fdf900052e0 2024-05-28T06:24:02.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.117+0000 7fdfaa000640 1 -- 172.21.15.96:0/845385022 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfa4007630 con 0x7fdf900052e0 2024-05-28T06:24:02.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.118+0000 7fdfb1000640 1 -- 172.21.15.96:0/845385022 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdf900052e0 msgr2=0x7fdf9000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:02.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.118+0000 7fdfb1000640 1 --2- 172.21.15.96:0/845385022 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdf900052e0 0x7fdf9000b9c0 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 crypto rx=0x7fdfa4003d80 tx=0x7fdf90005830 comp rx=0 tx=0).stop 2024-05-28T06:24:02.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.118+0000 7fdfb1000640 1 -- 172.21.15.96:0/845385022 shutdown_connections 2024-05-28T06:24:02.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.118+0000 7fdfb1000640 1 --2- 172.21.15.96:0/845385022 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdf900052e0 0x7fdf9000b9c0 unknown :-1 s=CLOSED pgs=1910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:02.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.118+0000 7fdfb1000640 1 -- 172.21.15.96:0/845385022 >> 172.21.15.96:0/845385022 conn(0x7fdfac08bfc0 msgr2=0x7fdfac0d44b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:02.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.118+0000 7fdfb1000640 1 -- 172.21.15.96:0/845385022 shutdown_connections 2024-05-28T06:24:02.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.118+0000 7fdfb1000640 1 -- 172.21.15.96:0/845385022 wait complete. 2024-05-28T06:24:02.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.120+0000 7fdfb1000640 1 Processor -- start 2024-05-28T06:24:02.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.120+0000 7fdfb1000640 1 -- start start 2024-05-28T06:24:02.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.121+0000 7fdfb1000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdf900052e0 0x7fdfac0b9690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:02.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.121+0000 7fdfb1000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdfac0b5d90 0x7fdfac0b6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:02.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.121+0000 7fdfb1000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdfac0cd960 0x7fdfac0cdde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:02.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.121+0000 7fdfb1000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fdfac0d53a0 con 0x7fdf900052e0 2024-05-28T06:24:02.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.121+0000 7fdfb1000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fdfac0d5220 con 0x7fdfac0b5d90 2024-05-28T06:24:02.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.121+0000 7fdfb1000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fdfac0d2fc0 con 0x7fdfac0cd960 2024-05-28T06:24:02.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.121+0000 7fdfab400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdf900052e0 0x7fdfac0b9690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:02.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.121+0000 7fdfab400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdf900052e0 0x7fdfac0b9690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:43036/0 (socket says 172.21.15.96:43036) 2024-05-28T06:24:02.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.121+0000 7fdfab400640 1 -- 172.21.15.96:0/3853856587 learned_addr learned my addr 172.21.15.96:0/3853856587 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:24:02.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.121+0000 7fdfab400640 1 -- 172.21.15.96:0/3853856587 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdfac0cd960 msgr2=0x7fdfac0cdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:02.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.121+0000 7fdfab400640 1 --2- 172.21.15.96:0/3853856587 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdfac0cd960 0x7fdfac0cdde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:02.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.121+0000 7fdfab400640 1 -- 172.21.15.96:0/3853856587 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdfac0b5d90 msgr2=0x7fdfac0b6210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:02.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.121+0000 7fdfab400640 1 --2- 172.21.15.96:0/3853856587 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdfac0b5d90 0x7fdfac0b6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:02.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.121+0000 7fdfab400640 1 -- 172.21.15.96:0/3853856587 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfac0ce320 con 0x7fdf900052e0 2024-05-28T06:24:02.119 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.122+0000 7fdfab400640 1 --2- 172.21.15.96:0/3853856587 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdf900052e0 0x7fdfac0b9690 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c002070 tx=0x7fdf9c00b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:02.119 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.122+0000 7fdfa3400640 1 -- 172.21.15.96:0/3853856587 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf9c019070 con 0x7fdf900052e0 2024-05-28T06:24:02.119 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.122+0000 7fdfa3400640 1 -- 172.21.15.96:0/3853856587 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf9c004d60 con 0x7fdf900052e0 2024-05-28T06:24:02.119 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.122+0000 7fdfb1000640 1 -- 172.21.15.96:0/3853856587 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfac0cb720 con 0x7fdf900052e0 2024-05-28T06:24:02.119 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.122+0000 7fdfa3400640 1 -- 172.21.15.96:0/3853856587 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf9c0072f0 con 0x7fdf900052e0 2024-05-28T06:24:02.119 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.122+0000 7fdfb1000640 1 -- 172.21.15.96:0/3853856587 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfac0cbce0 con 0x7fdf900052e0 2024-05-28T06:24:02.119 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.123+0000 7fdfb1000640 1 -- 172.21.15.96:0/3853856587 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdfac0689e0 con 0x7fdf900052e0 2024-05-28T06:24:02.119 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.124+0000 7fdfa3400640 1 -- 172.21.15.96:0/3853856587 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 42) ==== 93590+0+0 (secure 0 0 0) 0x7fdf9c01c070 con 0x7fdf900052e0 2024-05-28T06:24:02.120 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.125+0000 7fdfa3400640 1 --2- 172.21.15.96:0/3853856587 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdf8806e510 0x7fdf880709d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:02.120 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.125+0000 7fdfa3400640 1 -- 172.21.15.96:0/3853856587 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 5879+0+0 (secure 0 0 0) 0x7fdf9c08ac40 con 0x7fdf900052e0 2024-05-28T06:24:02.120 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.126+0000 7fdfaaa00640 1 --2- 172.21.15.96:0/3853856587 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdf8806e510 0x7fdf880709d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:02.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.127+0000 7fdfaaa00640 1 --2- 172.21.15.96:0/3853856587 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdf8806e510 0x7fdf880709d0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fdfac15f1d0 tx=0x7fdf94008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:02.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.130+0000 7fdfa3400640 1 -- 172.21.15.96:0/3853856587 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fdf9c05aad0 con 0x7fdf900052e0 2024-05-28T06:24:02.496 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.501+0000 7fdfb1000640 1 -- 172.21.15.96:0/3853856587 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone36083", "target": ["mon-mgr", ""]}) -- 0x7fdfac0d1d30 con 0x7fdf8806e510 2024-05-28T06:24:02.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.539+0000 7fdfa3400640 1 -- 172.21.15.96:0/3853856587 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fdfac0d1d30 con 0x7fdf8806e510 2024-05-28T06:24:02.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.543+0000 7fdfb1000640 1 -- 172.21.15.96:0/3853856587 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdf8806e510 msgr2=0x7fdf880709d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:02.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.543+0000 7fdfb1000640 1 --2- 172.21.15.96:0/3853856587 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdf8806e510 0x7fdf880709d0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fdfac15f1d0 tx=0x7fdf94008040 comp rx=0 tx=0).stop 2024-05-28T06:24:02.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.543+0000 7fdfb1000640 1 -- 172.21.15.96:0/3853856587 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdf900052e0 msgr2=0x7fdfac0b9690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:02.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.543+0000 7fdfb1000640 1 --2- 172.21.15.96:0/3853856587 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdf900052e0 0x7fdfac0b9690 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c002070 tx=0x7fdf9c00b4a0 comp rx=0 tx=0).stop 2024-05-28T06:24:02.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.544+0000 7fdfb1000640 1 -- 172.21.15.96:0/3853856587 shutdown_connections 2024-05-28T06:24:02.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.544+0000 7fdfb1000640 1 --2- 172.21.15.96:0/3853856587 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fdf8806e510 0x7fdf880709d0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:02.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.544+0000 7fdfb1000640 1 --2- 172.21.15.96:0/3853856587 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fdfac0cd960 0x7fdfac0cdde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:02.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.544+0000 7fdfb1000640 1 --2- 172.21.15.96:0/3853856587 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdfac0b5d90 0x7fdfac0b6210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:02.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.544+0000 7fdfb1000640 1 --2- 172.21.15.96:0/3853856587 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdf900052e0 0x7fdfac0b9690 unknown :-1 s=CLOSED pgs=1911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:02.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.544+0000 7fdfb1000640 1 -- 172.21.15.96:0/3853856587 >> 172.21.15.96:0/3853856587 conn(0x7fdfac08bfc0 msgr2=0x7fdfac06d300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:02.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.544+0000 7fdfb1000640 1 -- 172.21.15.96:0/3853856587 shutdown_connections 2024-05-28T06:24:02.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02.544+0000 7fdfb1000640 1 -- 172.21.15.96:0/3853856587 wait complete. 2024-05-28T06:24:02.564 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2024-05-28T06:24:02.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-28T06:24:02.659 INFO:teuthology.orchestra.run.smithi096.stdout:2 2024-05-28T06:24:02.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-28T06:24:02.668 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-05-28T06:24:02.668 DEBUG:teuthology.orchestra.run.smithi096:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-05-28T06:24:02.743 DEBUG:teuthology.orchestra.run.smithi123:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-05-28T06:24:02.854 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-05-28T06:24:02.854 INFO:tasks.cephfs.mount:Terminating background process 2024-05-28T06:24:02.858 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:24:02.858 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:24:02.913 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:24:03.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.106+0000 7f426ee00640 1 Processor -- start 2024-05-28T06:24:03.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.106+0000 7f426ee00640 1 -- start start 2024-05-28T06:24:03.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.107+0000 7f426ee00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f4268156b30 0x7f42680af6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:03.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.107+0000 7f426ee00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f426806ee40 0x7f42680ad230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:03.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.107+0000 7f426ee00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f426806f2f0 0x7f42680ad770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:03.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.107+0000 7f426ee00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f4268156f10 con 0x7f4268156b30 2024-05-28T06:24:03.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.107+0000 7f426ee00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f426806dda0 con 0x7f426806ee40 2024-05-28T06:24:03.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.108+0000 7f426ee00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f426806e080 con 0x7f426806f2f0 2024-05-28T06:24:03.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.108+0000 7f426da00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f4268156b30 0x7f42680af6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:03.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.108+0000 7f426da00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f4268156b30 0x7f42680af6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:43046/0 (socket says 172.21.15.96:43046) 2024-05-28T06:24:03.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.108+0000 7f426da00640 1 -- 172.21.15.96:0/2804145704 learned_addr learned my addr 172.21.15.96:0/2804145704 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:24:03.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.108+0000 7f426da00640 1 -- 172.21.15.96:0/2804145704 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f426806f2f0 msgr2=0x7f42680ad770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:03.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.108+0000 7f426da00640 1 --2- 172.21.15.96:0/2804145704 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f426806f2f0 0x7f42680ad770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:03.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.108+0000 7f426da00640 1 -- 172.21.15.96:0/2804145704 >> v2:172.21.15.123:3300/0 conn(0x7f426806ee40 msgr2=0x7f42680ad230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:03.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.108+0000 7f426da00640 1 --2- 172.21.15.96:0/2804145704 >> v2:172.21.15.123:3300/0 conn(0x7f426806ee40 0x7f42680ad230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:03.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.108+0000 7f426da00640 1 -- 172.21.15.96:0/2804145704 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f426806e360 con 0x7f4268156b30 2024-05-28T06:24:03.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.109+0000 7f426da00640 1 --2- 172.21.15.96:0/2804145704 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4268156b30 0x7f42680af6e0 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 crypto rx=0x7f426806fdf0 tx=0x7f425c02a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b2c71f8192d860c server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:03.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.109+0000 7f4267e00640 1 -- 172.21.15.96:0/2804145704 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f425c037070 con 0x7f4268156b30 2024-05-28T06:24:03.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.109+0000 7f4267e00640 1 -- 172.21.15.96:0/2804145704 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f425c032440 con 0x7f4268156b30 2024-05-28T06:24:03.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.109+0000 7f4267e00640 1 -- 172.21.15.96:0/2804145704 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f425c032810 con 0x7f4268156b30 2024-05-28T06:24:03.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.110+0000 7f426ee00640 1 -- 172.21.15.96:0/2804145704 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4268156b30 msgr2=0x7f42680af6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:03.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.110+0000 7f426ee00640 1 --2- 172.21.15.96:0/2804145704 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4268156b30 0x7f42680af6e0 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 crypto rx=0x7f426806fdf0 tx=0x7f425c02a140 comp rx=0 tx=0).stop 2024-05-28T06:24:03.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.110+0000 7f426ee00640 1 -- 172.21.15.96:0/2804145704 shutdown_connections 2024-05-28T06:24:03.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.110+0000 7f426ee00640 1 --2- 172.21.15.96:0/2804145704 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f426806f2f0 0x7f42680ad770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:03.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.110+0000 7f426ee00640 1 --2- 172.21.15.96:0/2804145704 >> v2:172.21.15.123:3300/0 conn(0x7f426806ee40 0x7f42680ad230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:03.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.110+0000 7f426ee00640 1 --2- 172.21.15.96:0/2804145704 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4268156b30 0x7f42680af6e0 unknown :-1 s=CLOSED pgs=1912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:03.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.110+0000 7f426ee00640 1 -- 172.21.15.96:0/2804145704 >> 172.21.15.96:0/2804145704 conn(0x7f42680b1fc0 msgr2=0x7f42680b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:03.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.110+0000 7f426ee00640 1 -- 172.21.15.96:0/2804145704 shutdown_connections 2024-05-28T06:24:03.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.110+0000 7f426ee00640 1 -- 172.21.15.96:0/2804145704 wait complete. 2024-05-28T06:24:03.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.112+0000 7f426ee00640 1 Processor -- start 2024-05-28T06:24:03.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.112+0000 7f426ee00640 1 -- start start 2024-05-28T06:24:03.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.113+0000 7f426ee00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f426806ee40 0x7f426814c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:03.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.113+0000 7f426ee00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f426806f2f0 0x7f426815f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:03.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.113+0000 7f426ee00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4268156b30 0x7f4268151bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:03.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.113+0000 7f426ee00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f42680afc20 con 0x7f4268156b30 2024-05-28T06:24:03.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.113+0000 7f426ee00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f42680b2c30 con 0x7f426806f2f0 2024-05-28T06:24:03.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.113+0000 7f426ee00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f4268154170 con 0x7f426806ee40 2024-05-28T06:24:03.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.113+0000 7f426e400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4268156b30 0x7f4268151bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:03.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.113+0000 7f426e400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4268156b30 0x7f4268151bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:43056/0 (socket says 172.21.15.96:43056) 2024-05-28T06:24:03.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.113+0000 7f426e400640 1 -- 172.21.15.96:0/231704249 learned_addr learned my addr 172.21.15.96:0/231704249 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:24:03.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.113+0000 7f426e400640 1 -- 172.21.15.96:0/231704249 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f426806ee40 msgr2=0x7f426814c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:03.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.113+0000 7f426e400640 1 --2- 172.21.15.96:0/231704249 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f426806ee40 0x7f426814c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:03.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.113+0000 7f426e400640 1 -- 172.21.15.96:0/231704249 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f426806f2f0 msgr2=0x7f426815f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:03.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.113+0000 7f426e400640 1 --2- 172.21.15.96:0/231704249 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f426806f2f0 0x7f426815f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:03.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.113+0000 7f426e400640 1 -- 172.21.15.96:0/231704249 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4268154450 con 0x7f4268156b30 2024-05-28T06:24:03.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.114+0000 7f426e400640 1 --2- 172.21.15.96:0/231704249 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4268156b30 0x7f4268151bf0 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7f426000b9e0 tx=0x7f426000bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:03.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.114+0000 7f4265600640 1 -- 172.21.15.96:0/231704249 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4260019070 con 0x7f4268156b30 2024-05-28T06:24:03.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.114+0000 7f4265600640 1 -- 172.21.15.96:0/231704249 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f426000cac0 con 0x7f4268156b30 2024-05-28T06:24:03.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.114+0000 7f426ee00640 1 -- 172.21.15.96:0/231704249 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4268154730 con 0x7f4268156b30 2024-05-28T06:24:03.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.114+0000 7f4265600640 1 -- 172.21.15.96:0/231704249 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4260014440 con 0x7f4268156b30 2024-05-28T06:24:03.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.114+0000 7f426ee00640 1 -- 172.21.15.96:0/231704249 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4268154cf0 con 0x7f4268156b30 2024-05-28T06:24:03.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.115+0000 7f426ee00640 1 -- 172.21.15.96:0/231704249 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4268154ef0 con 0x7f4268156b30 2024-05-28T06:24:03.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.116+0000 7f4265600640 1 -- 172.21.15.96:0/231704249 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 42) ==== 93590+0+0 (secure 0 0 0) 0x7f426001c070 con 0x7f4268156b30 2024-05-28T06:24:03.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.117+0000 7f4265600640 1 --2- 172.21.15.96:0/231704249 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f424c06e540 0x7f424c070a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:03.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.117+0000 7f4265600640 1 -- 172.21.15.96:0/231704249 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 5879+0+0 (secure 0 0 0) 0x7f426008a730 con 0x7f4268156b30 2024-05-28T06:24:03.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.118+0000 7f426da00640 1 --2- 172.21.15.96:0/231704249 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f424c06e540 0x7f424c070a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:03.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.119+0000 7f426da00640 1 --2- 172.21.15.96:0/231704249 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f424c06e540 0x7f424c070a00 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f425c034000 tx=0x7f425c02c000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:03.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.121+0000 7f4265600640 1 -- 172.21.15.96:0/231704249 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f426005a660 con 0x7f4268156b30 2024-05-28T06:24:03.304 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.309+0000 7f426ee00640 1 -- 172.21.15.96:0/231704249 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f42680689e0 con 0x7f4268156b30 2024-05-28T06:24:03.306 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.311+0000 7f4265600640 1 -- 172.21.15.96:0/231704249 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v118) ==== 74+0+16936 (secure 0 0 0) 0x7f4260059f60 con 0x7f4268156b30 2024-05-28T06:24:03.306 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:24:03.307 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":118,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:24:01.754615+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-05-28T06:22:14.138685+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"114","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-05-28T06:22:15.394361+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"118","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":118,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.c","mappings":[{"from":0,"to":2}]},{"pgid":"17.17","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":13,"snaps":[{"begin":2,"length":2}]},{"pool":17,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[{"pool":17,"snaps":[{"begin":2,"length":2}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:24:03.309 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.314+0000 7f426ee00640 1 -- 172.21.15.96:0/231704249 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f424c06e540 msgr2=0x7f424c070a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:03.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.314+0000 7f426ee00640 1 --2- 172.21.15.96:0/231704249 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f424c06e540 0x7f424c070a00 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f425c034000 tx=0x7f425c02c000 comp rx=0 tx=0).stop 2024-05-28T06:24:03.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.314+0000 7f426ee00640 1 -- 172.21.15.96:0/231704249 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4268156b30 msgr2=0x7f4268151bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:03.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.314+0000 7f426ee00640 1 --2- 172.21.15.96:0/231704249 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4268156b30 0x7f4268151bf0 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7f426000b9e0 tx=0x7f426000bea0 comp rx=0 tx=0).stop 2024-05-28T06:24:03.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.315+0000 7f426ee00640 1 -- 172.21.15.96:0/231704249 shutdown_connections 2024-05-28T06:24:03.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.315+0000 7f426ee00640 1 --2- 172.21.15.96:0/231704249 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f424c06e540 0x7f424c070a00 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:03.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.315+0000 7f426ee00640 1 --2- 172.21.15.96:0/231704249 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4268156b30 0x7f4268151bf0 unknown :-1 s=CLOSED pgs=1913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:03.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.315+0000 7f426ee00640 1 --2- 172.21.15.96:0/231704249 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f426806f2f0 0x7f426815f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:03.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.315+0000 7f426ee00640 1 --2- 172.21.15.96:0/231704249 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f426806ee40 0x7f426814c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:03.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.315+0000 7f426ee00640 1 -- 172.21.15.96:0/231704249 >> 172.21.15.96:0/231704249 conn(0x7f42680b1fc0 msgr2=0x7f42680b0080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:03.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.315+0000 7f426ee00640 1 -- 172.21.15.96:0/231704249 shutdown_connections 2024-05-28T06:24:03.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.315+0000 7f426ee00640 1 -- 172.21.15.96:0/231704249 wait complete. 2024-05-28T06:24:03.328 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-28T06:03:59.517102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-28T06:22:14.138685+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '114', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-05-28T06:22:15.394361+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '118', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 118, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-28T06:24:03.329 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-28T06:24:03.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.525+0000 7f3176e00640 1 Processor -- start 2024-05-28T06:24:03.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.526+0000 7f3176e00640 1 -- start start 2024-05-28T06:24:03.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.527+0000 7f3176e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f31700725e0 0x7f3170166050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:03.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.527+0000 7f3176e00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31700ade90 con 0x7f317015b6f0 2024-05-28T06:24:03.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.527+0000 7f3176e00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3170071e70 con 0x7f31700d0640 2024-05-28T06:24:03.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.527+0000 7f3176e00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f3170071c40 con 0x7f31700725e0 2024-05-28T06:24:03.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.528+0000 7f3167400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f317015b6f0 0x7f317006e350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:55888/0 (socket says 172.21.15.96:55888) 2024-05-28T06:24:03.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.528+0000 7f3167400640 1 -- 172.21.15.96:0/2784008188 learned_addr learned my addr 172.21.15.96:0/2784008188 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:24:03.524 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.528+0000 7f316f400640 1 -- 172.21.15.96:0/2784008188 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3963059612 0 0) 0x7f31700ade90 con 0x7f317015b6f0 2024-05-28T06:24:03.524 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.528+0000 7f316f400640 1 -- 172.21.15.96:0/2784008188 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3154005730 con 0x7f317015b6f0 2024-05-28T06:24:03.524 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.528+0000 7f3174e00640 1 --2- 172.21.15.96:0/2784008188 >> v2:172.21.15.123:3300/0 conn(0x7f31700725e0 0x7f3170166050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:03.524 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.528+0000 7f316f400640 1 -- 172.21.15.96:0/2784008188 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2035969429 0 0) 0x7f3154005730 con 0x7f317015b6f0 2024-05-28T06:24:03.524 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.528+0000 7f316f400640 1 -- 172.21.15.96:0/2784008188 >> v2:172.21.15.123:3300/0 conn(0x7f31700725e0 msgr2=0x7f3170166050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:03.524 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.528+0000 7f316f400640 1 --2- 172.21.15.96:0/2784008188 >> v2:172.21.15.123:3300/0 conn(0x7f31700725e0 0x7f3170166050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:03.525 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.528+0000 7f316f400640 1 -- 172.21.15.96:0/2784008188 >> v1:172.21.15.123:6789/0 conn(0x7f31700d0640 legacy=0x7f31700d0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:03.525 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.528+0000 7f316f400640 1 -- 172.21.15.96:0/2784008188 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3170072130 con 0x7f317015b6f0 2024-05-28T06:24:03.525 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.528+0000 7f316f400640 1 -- 172.21.15.96:0/2784008188 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f3158001670 con 0x7f317015b6f0 2024-05-28T06:24:03.525 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.528+0000 7f316f400640 1 -- 172.21.15.96:0/2784008188 >> v1:172.21.15.96:6789/0 conn(0x7f317015b6f0 legacy=0x7f317006e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:03.525 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.529+0000 7f316f400640 1 --2- 172.21.15.96:0/2784008188 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3154005c10 0x7f3154006030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:03.525 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.529+0000 7f316f400640 1 --2- 172.21.15.96:0/2784008188 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3154006a10 0x7f3154006e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:03.525 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.529+0000 7f316f400640 1 --2- 172.21.15.96:0/2784008188 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3154007420 0x7f315400dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:03.525 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.529+0000 7f316f400640 1 -- 172.21.15.96:0/2784008188 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f315400e340 con 0x7f3154007420 2024-05-28T06:24:03.525 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.529+0000 7f316f400640 1 -- 172.21.15.96:0/2784008188 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f315400e4f0 con 0x7f3154005c10 2024-05-28T06:24:03.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.529+0000 7f316f400640 1 -- 172.21.15.96:0/2784008188 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f315400e6f0 con 0x7f3154006a10 2024-05-28T06:24:03.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.529+0000 7f3174e00640 1 --2- 172.21.15.96:0/2784008188 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3154007420 0x7f315400dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:03.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.529+0000 7f316fe00640 1 --2- 172.21.15.96:0/2784008188 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3154005c10 0x7f3154006030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:03.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.529+0000 7f3174e00640 1 -- 172.21.15.96:0/2784008188 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3154006a10 msgr2=0x7f3154006e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:03.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.529+0000 7f3174e00640 1 --2- 172.21.15.96:0/2784008188 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3154006a10 0x7f3154006e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:03.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.529+0000 7f3174e00640 1 -- 172.21.15.96:0/2784008188 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3154005c10 msgr2=0x7f3154006030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:03.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.529+0000 7f3174e00640 1 --2- 172.21.15.96:0/2784008188 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3154005c10 0x7f3154006030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:03.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.529+0000 7f3174e00640 1 -- 172.21.15.96:0/2784008188 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3154005730 con 0x7f3154007420 2024-05-28T06:24:03.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.529+0000 7f316fe00640 1 --2- 172.21.15.96:0/2784008188 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3154005c10 0x7f3154006030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:24:03.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.530+0000 7f3174e00640 1 -- 172.21.15.96:0/2784008188 reap_dead start 2024-05-28T06:24:03.528 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.530+0000 7f3174e00640 1 --2- 172.21.15.96:0/2784008188 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3154007420 0x7f315400dbd0 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7f3168007360 tx=0x7f3154007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11e8aa423c6192bf server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:03.528 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.531+0000 7f316f400640 1 -- 172.21.15.96:0/2784008188 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3168009be0 con 0x7f3154007420 2024-05-28T06:24:03.528 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.531+0000 7f316f400640 1 -- 172.21.15.96:0/2784008188 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3168003bd0 con 0x7f3154007420 2024-05-28T06:24:03.528 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.531+0000 7f316f400640 1 -- 172.21.15.96:0/2784008188 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3168004020 con 0x7f3154007420 2024-05-28T06:24:03.528 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.531+0000 7f3176e00640 1 -- 172.21.15.96:0/2784008188 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3154007420 msgr2=0x7f315400dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:03.528 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.531+0000 7f3176e00640 1 --2- 172.21.15.96:0/2784008188 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3154007420 0x7f315400dbd0 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7f3168007360 tx=0x7f3154007a40 comp rx=0 tx=0).stop 2024-05-28T06:24:03.528 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.531+0000 7f3176e00640 1 -- 172.21.15.96:0/2784008188 shutdown_connections 2024-05-28T06:24:03.528 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.531+0000 7f3176e00640 1 --2- 172.21.15.96:0/2784008188 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3154007420 0x7f315400dbd0 unknown :-1 s=CLOSED pgs=1915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:03.528 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.531+0000 7f3176e00640 1 -- 172.21.15.96:0/2784008188 >> 172.21.15.96:0/2784008188 conn(0x7f317006bd20 msgr2=0x7f317006c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:03.528 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.531+0000 7f3176e00640 1 -- 172.21.15.96:0/2784008188 shutdown_connections 2024-05-28T06:24:03.529 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.531+0000 7f3176e00640 1 -- 172.21.15.96:0/2784008188 wait complete. 2024-05-28T06:24:03.529 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.533+0000 7f3176e00640 1 Processor -- start 2024-05-28T06:24:03.529 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.533+0000 7f3176e00640 1 -- start start 2024-05-28T06:24:03.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.534+0000 7f3176e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3154007420 0x7f317013fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:03.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.534+0000 7f3176e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f317015e6b0 0x7f31701f31b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:03.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.534+0000 7f3176e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f317015eb30 0x7f31701f36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:03.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.534+0000 7f3176e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f31700d0ba0 con 0x7f3154007420 2024-05-28T06:24:03.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.534+0000 7f3176e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f31700d0a20 con 0x7f317015e6b0 2024-05-28T06:24:03.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.534+0000 7f3176e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f31700d0d20 con 0x7f317015eb30 2024-05-28T06:24:03.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.534+0000 7f316fe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3154007420 0x7f317013fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:03.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.534+0000 7f316fe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3154007420 0x7f317013fe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:43064/0 (socket says 172.21.15.96:43064) 2024-05-28T06:24:03.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.535+0000 7f316fe00640 1 -- 172.21.15.96:0/1566252693 learned_addr learned my addr 172.21.15.96:0/1566252693 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:24:03.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.535+0000 7f316fe00640 1 -- 172.21.15.96:0/1566252693 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f317015eb30 msgr2=0x7f31701f36f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:03.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.535+0000 7f316fe00640 1 --2- 172.21.15.96:0/1566252693 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f317015eb30 0x7f31701f36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:03.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.535+0000 7f316fe00640 1 -- 172.21.15.96:0/1566252693 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f317015e6b0 msgr2=0x7f31701f31b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:03.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.535+0000 7f3167400640 1 --2- 172.21.15.96:0/1566252693 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f317015e6b0 0x7f31701f31b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:03.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.535+0000 7f316fe00640 1 --2- 172.21.15.96:0/1566252693 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f317015e6b0 0x7f31701f31b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:03.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.535+0000 7f316fe00640 1 -- 172.21.15.96:0/1566252693 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31701f0a80 con 0x7f3154007420 2024-05-28T06:24:03.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.535+0000 7f316fe00640 1 --2- 172.21.15.96:0/1566252693 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3154007420 0x7f317013fe00 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 crypto rx=0x7f316000d7d0 tx=0x7f316000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:03.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.535+0000 7f316cc00640 1 -- 172.21.15.96:0/1566252693 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3160005270 con 0x7f3154007420 2024-05-28T06:24:03.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.535+0000 7f316cc00640 1 -- 172.21.15.96:0/1566252693 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3160017070 con 0x7f3154007420 2024-05-28T06:24:03.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.536+0000 7f316cc00640 1 -- 172.21.15.96:0/1566252693 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f316000b6b0 con 0x7f3154007420 2024-05-28T06:24:03.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.536+0000 7f3176e00640 1 -- 172.21.15.96:0/1566252693 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31700cd8c0 con 0x7f3154007420 2024-05-28T06:24:03.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.536+0000 7f3176e00640 1 -- 172.21.15.96:0/1566252693 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31700cde80 con 0x7f3154007420 2024-05-28T06:24:03.533 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.537+0000 7f3176e00640 1 -- 172.21.15.96:0/1566252693 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31700689e0 con 0x7f3154007420 2024-05-28T06:24:03.533 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.538+0000 7f316cc00640 1 -- 172.21.15.96:0/1566252693 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 42) ==== 93590+0+0 (secure 0 0 0) 0x7f316000b850 con 0x7f3154007420 2024-05-28T06:24:03.533 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.539+0000 7f316cc00640 1 --2- 172.21.15.96:0/1566252693 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f314c06e510 0x7f314c0709d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:03.533 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.539+0000 7f316cc00640 1 -- 172.21.15.96:0/1566252693 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 5879+0+0 (secure 0 0 0) 0x7f316008a870 con 0x7f3154007420 2024-05-28T06:24:03.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.539+0000 7f3167400640 1 --2- 172.21.15.96:0/1566252693 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f314c06e510 0x7f314c0709d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:03.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.541+0000 7f3167400640 1 --2- 172.21.15.96:0/1566252693 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f314c06e510 0x7f314c0709d0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f3170142f50 tx=0x7f3158009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:03.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.543+0000 7f316cc00640 1 -- 172.21.15.96:0/1566252693 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3160019310 con 0x7f3154007420 2024-05-28T06:24:03.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.728+0000 7f3176e00640 1 -- 172.21.15.96:0/1566252693 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f31700b4410 con 0x7f3154007420 2024-05-28T06:24:03.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.729+0000 7f316cc00640 1 -- 172.21.15.96:0/1566252693 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v118) ==== 93+0+10 (secure 0 0 0) 0x7f3160001230 con 0x7f3154007420 2024-05-28T06:24:03.724 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 1 2024-05-28T06:24:03.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.731+0000 7f3176e00640 1 -- 172.21.15.96:0/1566252693 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f314c06e510 msgr2=0x7f314c0709d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:03.727 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.731+0000 7f3176e00640 1 --2- 172.21.15.96:0/1566252693 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f314c06e510 0x7f314c0709d0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f3170142f50 tx=0x7f3158009040 comp rx=0 tx=0).stop 2024-05-28T06:24:03.727 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.731+0000 7f3176e00640 1 -- 172.21.15.96:0/1566252693 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3154007420 msgr2=0x7f317013fe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:03.727 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.731+0000 7f3176e00640 1 --2- 172.21.15.96:0/1566252693 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3154007420 0x7f317013fe00 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 crypto rx=0x7f316000d7d0 tx=0x7f316000dc90 comp rx=0 tx=0).stop 2024-05-28T06:24:03.727 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.731+0000 7f3176e00640 1 -- 172.21.15.96:0/1566252693 shutdown_connections 2024-05-28T06:24:03.727 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.731+0000 7f3176e00640 1 --2- 172.21.15.96:0/1566252693 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f314c06e510 0x7f314c0709d0 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:03.727 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.732+0000 7f3176e00640 1 --2- 172.21.15.96:0/1566252693 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f317015eb30 0x7f31701f36f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:03.727 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.732+0000 7f3176e00640 1 --2- 172.21.15.96:0/1566252693 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f317015e6b0 0x7f31701f31b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:03.727 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.732+0000 7f3176e00640 1 --2- 172.21.15.96:0/1566252693 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3154007420 0x7f317013fe00 unknown :-1 s=CLOSED pgs=1916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:03.727 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.732+0000 7f3176e00640 1 -- 172.21.15.96:0/1566252693 >> 172.21.15.96:0/1566252693 conn(0x7f317006bd20 msgr2=0x7f31700b4c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:03.727 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.732+0000 7f3176e00640 1 -- 172.21.15.96:0/1566252693 shutdown_connections 2024-05-28T06:24:03.727 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.732+0000 7f3176e00640 1 -- 172.21.15.96:0/1566252693 wait complete. 2024-05-28T06:24:03.745 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-28T06:24:03.943 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.949+0000 7f577d400640 1 Processor -- start 2024-05-28T06:24:03.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.949+0000 7f577d400640 1 -- start start 2024-05-28T06:24:03.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.950+0000 7f577d400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f577815afd0 0x7f57780cdd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:03.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.950+0000 7f577d400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f57780ccb90 0x7f57780cd020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:03.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.950+0000 7f577d400640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57780cc940 con 0x7f577806f5e0 2024-05-28T06:24:03.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.950+0000 7f577d400640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f577806f9c0 con 0x7f577815afd0 2024-05-28T06:24:03.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.950+0000 7f577d400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f577815b520 con 0x7f57780ccb90 2024-05-28T06:24:03.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.951+0000 7f577ca00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f57780ccb90 0x7f57780cd020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:03.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.951+0000 7f577ca00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f57780ccb90 0x7f57780cd020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.96:49990/0 (socket says 172.21.15.96:49990) 2024-05-28T06:24:03.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.951+0000 7f577ca00640 1 -- 172.21.15.96:0/2682557916 learned_addr learned my addr 172.21.15.96:0/2682557916 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:24:03.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.952+0000 7f576f400640 1 --2- 172.21.15.96:0/2682557916 >> v2:172.21.15.123:3300/0 conn(0x7f577815afd0 0x7f57780cdd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:03.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.952+0000 7f577ca00640 1 -- 172.21.15.96:0/2682557916 >> v2:172.21.15.123:3300/0 conn(0x7f577815afd0 msgr2=0x7f57780cdd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:03.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.952+0000 7f577ca00640 1 --2- 172.21.15.96:0/2682557916 >> v2:172.21.15.123:3300/0 conn(0x7f577815afd0 0x7f57780cdd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:03.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.952+0000 7f577ca00640 1 -- 172.21.15.96:0/2682557916 >> v1:172.21.15.123:6789/0 conn(0x7f577806f5e0 legacy=0x7f57780cc230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:03.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.952+0000 7f577ca00640 1 -- 172.21.15.96:0/2682557916 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f577815b800 con 0x7f57780ccb90 2024-05-28T06:24:03.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.953+0000 7f577ca00640 1 --2- 172.21.15.96:0/2682557916 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f57780ccb90 0x7f57780cd020 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f5770009e50 tx=0x7f577002c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db427c3a66cd8b8 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:03.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.954+0000 7f5777400640 1 -- 172.21.15.96:0/2682557916 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5770032610 con 0x7f57780ccb90 2024-05-28T06:24:03.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.954+0000 7f5777400640 1 -- 172.21.15.96:0/2682557916 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5770037070 con 0x7f57780ccb90 2024-05-28T06:24:03.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.954+0000 7f5777400640 1 -- 172.21.15.96:0/2682557916 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57700329d0 con 0x7f57780ccb90 2024-05-28T06:24:03.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.955+0000 7f577d400640 1 -- 172.21.15.96:0/2682557916 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f57780ccb90 msgr2=0x7f57780cd020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:03.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.955+0000 7f577d400640 1 --2- 172.21.15.96:0/2682557916 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f57780ccb90 0x7f57780cd020 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f5770009e50 tx=0x7f577002c920 comp rx=0 tx=0).stop 2024-05-28T06:24:03.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.955+0000 7f577d400640 1 -- 172.21.15.96:0/2682557916 shutdown_connections 2024-05-28T06:24:03.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.955+0000 7f577d400640 1 --2- 172.21.15.96:0/2682557916 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f57780ccb90 0x7f57780cd020 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:03.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.955+0000 7f577d400640 1 --2- 172.21.15.96:0/2682557916 >> v2:172.21.15.123:3300/0 conn(0x7f577815afd0 0x7f57780cdd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:03.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.955+0000 7f577d400640 1 -- 172.21.15.96:0/2682557916 >> 172.21.15.96:0/2682557916 conn(0x7f577806b900 msgr2=0x7f577806bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:03.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.955+0000 7f577d400640 1 -- 172.21.15.96:0/2682557916 shutdown_connections 2024-05-28T06:24:03.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.955+0000 7f577d400640 1 -- 172.21.15.96:0/2682557916 wait complete. 2024-05-28T06:24:03.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.956+0000 7f577d400640 1 Processor -- start 2024-05-28T06:24:03.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.957+0000 7f577d400640 1 -- start start 2024-05-28T06:24:03.953 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.957+0000 7f577d400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f577806f5e0 0x7f5778158b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:03.953 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.957+0000 7f577d400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f57780ccb90 0x7f577806dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:03.953 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.957+0000 7f577d400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f577815afd0 0x7f577806e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:03.953 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.957+0000 7f577d400640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f57780d0180 con 0x7f57780ccb90 2024-05-28T06:24:03.953 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.957+0000 7f577d400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f577815bdc0 con 0x7f577806f5e0 2024-05-28T06:24:03.953 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.957+0000 7f577d400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f57780d0300 con 0x7f577815afd0 2024-05-28T06:24:03.953 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.958+0000 7f576f400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f57780ccb90 0x7f577806dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:03.953 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.958+0000 7f576f400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f57780ccb90 0x7f577806dbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:43066/0 (socket says 172.21.15.96:43066) 2024-05-28T06:24:03.953 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.958+0000 7f576f400640 1 -- 172.21.15.96:0/526189953 learned_addr learned my addr 172.21.15.96:0/526189953 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:24:03.953 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.958+0000 7f5777e00640 1 --2- 172.21.15.96:0/526189953 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f577806f5e0 0x7f5778158b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:03.953 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.958+0000 7f577ca00640 1 --2- 172.21.15.96:0/526189953 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f577815afd0 0x7f577806e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:03.953 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.958+0000 7f576f400640 1 -- 172.21.15.96:0/526189953 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f577815afd0 msgr2=0x7f577806e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:03.953 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.958+0000 7f576f400640 1 --2- 172.21.15.96:0/526189953 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f577815afd0 0x7f577806e120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:03.954 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.958+0000 7f576f400640 1 -- 172.21.15.96:0/526189953 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f577806f5e0 msgr2=0x7f5778158b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:03.954 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.958+0000 7f576f400640 1 --2- 172.21.15.96:0/526189953 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f577806f5e0 0x7f5778158b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:03.954 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.958+0000 7f576f400640 1 -- 172.21.15.96:0/526189953 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57781f4680 con 0x7f57780ccb90 2024-05-28T06:24:03.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.959+0000 7f576f400640 1 --2- 172.21.15.96:0/526189953 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f57780ccb90 0x7f577806dbe0 secure :-1 s=READY pgs=1917 cs=0 l=1 rev1=1 crypto rx=0x7f576000e870 tx=0x7f576000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:03.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.959+0000 7f5774c00640 1 -- 172.21.15.96:0/526189953 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5760017070 con 0x7f57780ccb90 2024-05-28T06:24:03.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.959+0000 7f5774c00640 1 -- 172.21.15.96:0/526189953 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f576000c940 con 0x7f57780ccb90 2024-05-28T06:24:03.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.959+0000 7f5774c00640 1 -- 172.21.15.96:0/526189953 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f576000cd30 con 0x7f57780ccb90 2024-05-28T06:24:03.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.959+0000 7f577d400640 1 -- 172.21.15.96:0/526189953 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57781f4960 con 0x7f57780ccb90 2024-05-28T06:24:03.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.959+0000 7f577d400640 1 -- 172.21.15.96:0/526189953 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57781f4f20 con 0x7f57780ccb90 2024-05-28T06:24:03.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.960+0000 7f577d400640 1 -- 172.21.15.96:0/526189953 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57780689e0 con 0x7f57780ccb90 2024-05-28T06:24:03.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.961+0000 7f5774c00640 1 -- 172.21.15.96:0/526189953 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 42) ==== 93590+0+0 (secure 0 0 0) 0x7f576001a070 con 0x7f57780ccb90 2024-05-28T06:24:03.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.962+0000 7f5774c00640 1 --2- 172.21.15.96:0/526189953 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f575406e510 0x7f57540709d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:03.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.962+0000 7f5774c00640 1 -- 172.21.15.96:0/526189953 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 5879+0+0 (secure 0 0 0) 0x7f576008ad80 con 0x7f57780ccb90 2024-05-28T06:24:03.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.963+0000 7f5777e00640 1 --2- 172.21.15.96:0/526189953 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f575406e510 0x7f57540709d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:03.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.964+0000 7f5777e00640 1 --2- 172.21.15.96:0/526189953 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f575406e510 0x7f57540709d0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f5768000f60 tx=0x7f5768009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:03.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:03.966+0000 7f5774c00640 1 -- 172.21.15.96:0/526189953 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f576005ac10 con 0x7f57780ccb90 2024-05-28T06:24:04.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.163+0000 7f577d400640 1 -- 172.21.15.96:0/526189953 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f577806edc0 con 0x7f57780ccb90 2024-05-28T06:24:04.159 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.164+0000 7f5774c00640 1 -- 172.21.15.96:0/526189953 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v118) ==== 104+0+11 (secure 0 0 0) 0x7f576005a510 con 0x7f57780ccb90 2024-05-28T06:24:04.159 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 32 2024-05-28T06:24:04.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.167+0000 7f577d400640 1 -- 172.21.15.96:0/526189953 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f575406e510 msgr2=0x7f57540709d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:04.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.167+0000 7f577d400640 1 --2- 172.21.15.96:0/526189953 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f575406e510 0x7f57540709d0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f5768000f60 tx=0x7f5768009040 comp rx=0 tx=0).stop 2024-05-28T06:24:04.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.167+0000 7f577d400640 1 -- 172.21.15.96:0/526189953 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f57780ccb90 msgr2=0x7f577806dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:04.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.167+0000 7f577d400640 1 --2- 172.21.15.96:0/526189953 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f57780ccb90 0x7f577806dbe0 secure :-1 s=READY pgs=1917 cs=0 l=1 rev1=1 crypto rx=0x7f576000e870 tx=0x7f576000ed30 comp rx=0 tx=0).stop 2024-05-28T06:24:04.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.167+0000 7f577d400640 1 -- 172.21.15.96:0/526189953 shutdown_connections 2024-05-28T06:24:04.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.167+0000 7f577d400640 1 --2- 172.21.15.96:0/526189953 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f575406e510 0x7f57540709d0 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:04.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.167+0000 7f577d400640 1 --2- 172.21.15.96:0/526189953 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f577815afd0 0x7f577806e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:04.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.167+0000 7f577d400640 1 --2- 172.21.15.96:0/526189953 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f57780ccb90 0x7f577806dbe0 unknown :-1 s=CLOSED pgs=1917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:04.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.167+0000 7f577d400640 1 --2- 172.21.15.96:0/526189953 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f577806f5e0 0x7f5778158b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:04.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.167+0000 7f577d400640 1 -- 172.21.15.96:0/526189953 >> 172.21.15.96:0/526189953 conn(0x7f577806b900 msgr2=0x7f5778159730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:04.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.167+0000 7f577d400640 1 -- 172.21.15.96:0/526189953 shutdown_connections 2024-05-28T06:24:04.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.168+0000 7f577d400640 1 -- 172.21.15.96:0/526189953 wait complete. 2024-05-28T06:24:04.181 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-28T06:24:04.444 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.450+0000 7f40a3200640 1 Processor -- start 2024-05-28T06:24:04.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.450+0000 7f40a3200640 1 -- start start 2024-05-28T06:24:04.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.451+0000 7f40a3200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f40a415e870 0x7f40a40d0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:04.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.451+0000 7f40a3200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f40a40cfa40 0x7f40a40d25e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:04.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.451+0000 7f40a3200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40a415ec50 con 0x7f40a40d14d0 2024-05-28T06:24:04.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.451+0000 7f40a3200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f40a40d2150 con 0x7f40a40cfa40 2024-05-28T06:24:04.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.451+0000 7f40a3200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f40a40cff80 con 0x7f40a415e870 2024-05-28T06:24:04.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.452+0000 7f40a1400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f40a40d14d0 0x7f40a40cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:55902/0 (socket says 172.21.15.96:55902) 2024-05-28T06:24:04.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.452+0000 7f40a1400640 1 -- 172.21.15.96:0/643355619 learned_addr learned my addr 172.21.15.96:0/643355619 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:24:04.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.452+0000 7f40a2800640 1 --2- 172.21.15.96:0/643355619 >> v2:172.21.15.96:3300/0 conn(0x7f40a40cfa40 0x7f40a40d25e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:04.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.452+0000 7f40a1e00640 1 --2- 172.21.15.96:0/643355619 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f40a415e870 0x7f40a40d0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:04.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.453+0000 7f40a0a00640 1 -- 172.21.15.96:0/643355619 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2419850574 0 0) 0x7f40a415ec50 con 0x7f40a40d14d0 2024-05-28T06:24:04.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.453+0000 7f40a0a00640 1 -- 172.21.15.96:0/643355619 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4088003520 con 0x7f40a40d14d0 2024-05-28T06:24:04.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.453+0000 7f40a2800640 1 -- 172.21.15.96:0/643355619 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f40a415e870 msgr2=0x7f40a40d0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:04.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.453+0000 7f40a2800640 1 --2- 172.21.15.96:0/643355619 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f40a415e870 0x7f40a40d0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:04.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.453+0000 7f40a2800640 1 -- 172.21.15.96:0/643355619 >> v1:172.21.15.96:6789/0 conn(0x7f40a40d14d0 legacy=0x7f40a40cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:04.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.453+0000 7f40a2800640 1 -- 172.21.15.96:0/643355619 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40a40d0260 con 0x7f40a40cfa40 2024-05-28T06:24:04.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.454+0000 7f40a2800640 1 --2- 172.21.15.96:0/643355619 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40a40cfa40 0x7f40a40d25e0 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x7f409c009d90 tx=0x7f409c02a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=762a92b0f160bb9e server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:04.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.455+0000 7f40a0a00640 1 -- 172.21.15.96:0/643355619 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f409c031600 con 0x7f40a40cfa40 2024-05-28T06:24:04.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.455+0000 7f40a0a00640 1 -- 172.21.15.96:0/643355619 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f409c0317a0 con 0x7f40a40cfa40 2024-05-28T06:24:04.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.455+0000 7f40a0a00640 1 -- 172.21.15.96:0/643355619 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f409c031600 con 0x7f40a40cfa40 2024-05-28T06:24:04.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.456+0000 7f40a3200640 1 -- 172.21.15.96:0/643355619 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40a40cfa40 msgr2=0x7f40a40d25e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:04.451 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.456+0000 7f40a3200640 1 --2- 172.21.15.96:0/643355619 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40a40cfa40 0x7f40a40d25e0 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x7f409c009d90 tx=0x7f409c02a610 comp rx=0 tx=0).stop 2024-05-28T06:24:04.451 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.456+0000 7f40a3200640 1 -- 172.21.15.96:0/643355619 shutdown_connections 2024-05-28T06:24:04.451 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.456+0000 7f40a3200640 1 --2- 172.21.15.96:0/643355619 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40a40cfa40 0x7f40a40d25e0 unknown :-1 s=CLOSED pgs=1919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:04.451 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.456+0000 7f40a3200640 1 --2- 172.21.15.96:0/643355619 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f40a415e870 0x7f40a40d0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:04.451 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.456+0000 7f40a3200640 1 -- 172.21.15.96:0/643355619 >> 172.21.15.96:0/643355619 conn(0x7f40a408bfc0 msgr2=0x7f40a40d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:04.452 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.457+0000 7f40a3200640 1 -- 172.21.15.96:0/643355619 shutdown_connections 2024-05-28T06:24:04.452 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.457+0000 7f40a3200640 1 -- 172.21.15.96:0/643355619 wait complete. 2024-05-28T06:24:04.454 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.459+0000 7f40a3200640 1 Processor -- start 2024-05-28T06:24:04.454 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.459+0000 7f40a3200640 1 -- start start 2024-05-28T06:24:04.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.461+0000 7f40a3200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40a40cfa40 0x7f40a415fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:04.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.461+0000 7f40a3200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f40a40d14d0 0x7f40a41fba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:04.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.461+0000 7f40a3200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40a415e870 0x7f40a41f58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:04.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.461+0000 7f40a3200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f40a40d4fb0 con 0x7f40a40cfa40 2024-05-28T06:24:04.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.461+0000 7f40a3200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f40a40d0820 con 0x7f40a415e870 2024-05-28T06:24:04.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.461+0000 7f40a3200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f40a40d2b20 con 0x7f40a40d14d0 2024-05-28T06:24:04.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.462+0000 7f40a1e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40a40cfa40 0x7f40a415fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:04.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.462+0000 7f40a2800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40a415e870 0x7f40a41f58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:04.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.462+0000 7f40a2800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40a415e870 0x7f40a41f58f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.96:33232/0 (socket says 172.21.15.96:33232) 2024-05-28T06:24:04.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.462+0000 7f40a2800640 1 -- 172.21.15.96:0/433803574 learned_addr learned my addr 172.21.15.96:0/433803574 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:24:04.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.462+0000 7f40a1400640 1 --2- 172.21.15.96:0/433803574 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f40a40d14d0 0x7f40a41fba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:04.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.463+0000 7f40a1e00640 1 -- 172.21.15.96:0/433803574 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f40a40d14d0 msgr2=0x7f40a41fba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:04.457 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.463+0000 7f40a1e00640 1 --2- 172.21.15.96:0/433803574 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f40a40d14d0 0x7f40a41fba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:04.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.463+0000 7f40a1e00640 1 -- 172.21.15.96:0/433803574 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40a415e870 msgr2=0x7f40a41f58f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:04.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.463+0000 7f40a1e00640 1 --2- 172.21.15.96:0/433803574 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40a415e870 0x7f40a41f58f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:04.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.463+0000 7f40a1e00640 1 -- 172.21.15.96:0/433803574 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40a41f5e30 con 0x7f40a40cfa40 2024-05-28T06:24:04.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.463+0000 7f40a1e00640 1 --2- 172.21.15.96:0/433803574 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40a40cfa40 0x7f40a415fd40 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x7f409400d520 tx=0x7f409400d9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:04.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.464+0000 7f409a000640 1 -- 172.21.15.96:0/433803574 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40940125f0 con 0x7f40a40cfa40 2024-05-28T06:24:04.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.464+0000 7f409a000640 1 -- 172.21.15.96:0/433803574 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4094019070 con 0x7f40a40cfa40 2024-05-28T06:24:04.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.464+0000 7f409a000640 1 -- 172.21.15.96:0/433803574 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40940129b0 con 0x7f40a40cfa40 2024-05-28T06:24:04.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.464+0000 7f40a3200640 1 -- 172.21.15.96:0/433803574 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40a41fc100 con 0x7f40a40cfa40 2024-05-28T06:24:04.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.464+0000 7f40a3200640 1 -- 172.21.15.96:0/433803574 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40a41fc6c0 con 0x7f40a40cfa40 2024-05-28T06:24:04.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.466+0000 7f40a3200640 1 -- 172.21.15.96:0/433803574 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40a41fc900 con 0x7f40a40cfa40 2024-05-28T06:24:04.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.469+0000 7f409a000640 1 -- 172.21.15.96:0/433803574 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 42) ==== 93590+0+0 (secure 0 0 0) 0x7f4094012b50 con 0x7f40a40cfa40 2024-05-28T06:24:04.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.470+0000 7f409a000640 1 --2- 172.21.15.96:0/433803574 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f408406e510 0x7f40840709d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:04.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.470+0000 7f409a000640 1 -- 172.21.15.96:0/433803574 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 5879+0+0 (secure 0 0 0) 0x7f409408ad80 con 0x7f40a40cfa40 2024-05-28T06:24:04.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.471+0000 7f40a1400640 1 --2- 172.21.15.96:0/433803574 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f408406e510 0x7f40840709d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:04.466 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.472+0000 7f40a1400640 1 --2- 172.21.15.96:0/433803574 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f408406e510 0x7f40840709d0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f40a40ce440 tx=0x7f4090009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:04.473 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.478+0000 7f409a000640 1 -- 172.21.15.96:0/433803574 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f40a41fc900 con 0x7f40a40cfa40 2024-05-28T06:24:04.669 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.674+0000 7f40a3200640 1 -- 172.21.15.96:0/433803574 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f40a40689e0 con 0x7f40a40cfa40 2024-05-28T06:24:04.670 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 64 2024-05-28T06:24:04.670 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.675+0000 7f409a000640 1 -- 172.21.15.96:0/433803574 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v118) ==== 100+0+11 (secure 0 0 0) 0x7f409405ac10 con 0x7f40a40cfa40 2024-05-28T06:24:04.673 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.677+0000 7f40a3200640 1 -- 172.21.15.96:0/433803574 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f408406e510 msgr2=0x7f40840709d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:04.673 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.677+0000 7f40a3200640 1 --2- 172.21.15.96:0/433803574 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f408406e510 0x7f40840709d0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f40a40ce440 tx=0x7f4090009040 comp rx=0 tx=0).stop 2024-05-28T06:24:04.673 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.677+0000 7f40a3200640 1 -- 172.21.15.96:0/433803574 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40a40cfa40 msgr2=0x7f40a415fd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:04.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.677+0000 7f40a3200640 1 --2- 172.21.15.96:0/433803574 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40a40cfa40 0x7f40a415fd40 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x7f409400d520 tx=0x7f409400d9e0 comp rx=0 tx=0).stop 2024-05-28T06:24:04.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.677+0000 7f40a3200640 1 -- 172.21.15.96:0/433803574 shutdown_connections 2024-05-28T06:24:04.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.678+0000 7f40a3200640 1 --2- 172.21.15.96:0/433803574 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f408406e510 0x7f40840709d0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:04.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.678+0000 7f40a3200640 1 --2- 172.21.15.96:0/433803574 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40a415e870 0x7f40a41f58f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:04.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.678+0000 7f40a3200640 1 --2- 172.21.15.96:0/433803574 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f40a40d14d0 0x7f40a41fba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:04.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.678+0000 7f40a3200640 1 --2- 172.21.15.96:0/433803574 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40a40cfa40 0x7f40a415fd40 unknown :-1 s=CLOSED pgs=1920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:04.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.678+0000 7f40a3200640 1 -- 172.21.15.96:0/433803574 >> 172.21.15.96:0/433803574 conn(0x7f40a408bfc0 msgr2=0x7f40a40b9a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:04.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.678+0000 7f40a3200640 1 -- 172.21.15.96:0/433803574 shutdown_connections 2024-05-28T06:24:04.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.678+0000 7f40a3200640 1 -- 172.21.15.96:0/433803574 wait complete. 2024-05-28T06:24:04.690 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-28T06:24:04.695 DEBUG:teuthology.orchestra.run.smithi096:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:24:04.741 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:24:04.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.876+0000 7f69d7e00640 1 Processor -- start 2024-05-28T06:24:04.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.876+0000 7f69d7e00640 1 -- start start 2024-05-28T06:24:04.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.878+0000 7f69d7e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f69d00d04a0 0x7f69d00d0880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:04.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.878+0000 7f69d7e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f69d0162e70 0x7f69d00ce230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:04.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.878+0000 7f69d7e00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69d00b4350 con 0x7f69d006f6e0 2024-05-28T06:24:04.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.878+0000 7f69d7e00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f69d00b3ec0 con 0x7f69d00d04a0 2024-05-28T06:24:04.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.878+0000 7f69d7e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f69d00b32d0 con 0x7f69d0162e70 2024-05-28T06:24:04.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.878+0000 7f69d4a00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f69d006f6e0 0x7f69d00cd4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:55906/0 (socket says 172.21.15.96:55906) 2024-05-28T06:24:04.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.878+0000 7f69d4a00640 1 -- 172.21.15.96:0/4212712938 learned_addr learned my addr 172.21.15.96:0/4212712938 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:24:04.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.878+0000 7f69d5400640 1 --2- 172.21.15.96:0/4212712938 >> v2:172.21.15.123:3300/0 conn(0x7f69d00d04a0 0x7f69d00d0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:04.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.878+0000 7f69cfe00640 1 -- 172.21.15.96:0/4212712938 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2979416458 0 0) 0x7f69d00b4350 con 0x7f69d006f6e0 2024-05-28T06:24:04.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.878+0000 7f69d5e00640 1 --2- 172.21.15.96:0/4212712938 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f69d0162e70 0x7f69d00ce230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:04.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.878+0000 7f69cfe00640 1 -- 172.21.15.96:0/4212712938 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f69b4003520 con 0x7f69d006f6e0 2024-05-28T06:24:04.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.879+0000 7f69cfe00640 1 -- 172.21.15.96:0/4212712938 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 373383746 0 0) 0x7f69b4003520 con 0x7f69d006f6e0 2024-05-28T06:24:04.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.879+0000 7f69cfe00640 1 -- 172.21.15.96:0/4212712938 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f69d0162e70 msgr2=0x7f69d00ce230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:04.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.879+0000 7f69cfe00640 1 --2- 172.21.15.96:0/4212712938 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f69d0162e70 0x7f69d00ce230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:04.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.879+0000 7f69cfe00640 1 -- 172.21.15.96:0/4212712938 >> v2:172.21.15.123:3300/0 conn(0x7f69d00d04a0 msgr2=0x7f69d00d0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:04.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.879+0000 7f69cfe00640 1 --2- 172.21.15.96:0/4212712938 >> v2:172.21.15.123:3300/0 conn(0x7f69d00d04a0 0x7f69d00d0880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:04.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.879+0000 7f69cfe00640 1 -- 172.21.15.96:0/4212712938 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69d00b3590 con 0x7f69d006f6e0 2024-05-28T06:24:04.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.879+0000 7f69cfe00640 1 -- 172.21.15.96:0/4212712938 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f69bc001670 con 0x7f69d006f6e0 2024-05-28T06:24:04.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.879+0000 7f69cfe00640 1 -- 172.21.15.96:0/4212712938 >> v1:172.21.15.96:6789/0 conn(0x7f69d006f6e0 legacy=0x7f69d00cd4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:04.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.879+0000 7f69cfe00640 1 --2- 172.21.15.96:0/4212712938 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f69d0162e70 0x7f69b40039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:04.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.879+0000 7f69cfe00640 1 --2- 172.21.15.96:0/4212712938 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69b4003f30 0x7f69b4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:04.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.879+0000 7f69cfe00640 1 --2- 172.21.15.96:0/4212712938 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f69b4004d40 0x7f69b40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:04.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.879+0000 7f69cfe00640 1 -- 172.21.15.96:0/4212712938 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f69d00b3ec0 con 0x7f69b4004d40 2024-05-28T06:24:04.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.879+0000 7f69cfe00640 1 -- 172.21.15.96:0/4212712938 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f69d00b32d0 con 0x7f69b4003f30 2024-05-28T06:24:04.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.879+0000 7f69cfe00640 1 -- 172.21.15.96:0/4212712938 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f69b400ba10 con 0x7f69d0162e70 2024-05-28T06:24:04.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.879+0000 7f69d5e00640 1 --2- 172.21.15.96:0/4212712938 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f69b4004d40 0x7f69b40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:04.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.880+0000 7f69d5400640 1 --2- 172.21.15.96:0/4212712938 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f69d0162e70 0x7f69b40039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:04.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.880+0000 7f69d4a00640 1 --2- 172.21.15.96:0/4212712938 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69b4003f30 0x7f69b4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:04.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.880+0000 7f69d5e00640 1 -- 172.21.15.96:0/4212712938 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f69d0162e70 msgr2=0x7f69b40039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:04.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.880+0000 7f69d5e00640 1 --2- 172.21.15.96:0/4212712938 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f69d0162e70 0x7f69b40039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:04.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.880+0000 7f69d5e00640 1 -- 172.21.15.96:0/4212712938 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69b4003f30 msgr2=0x7f69b4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:04.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.880+0000 7f69d5e00640 1 --2- 172.21.15.96:0/4212712938 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69b4003f30 0x7f69b4004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:04.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.880+0000 7f69d5e00640 1 -- 172.21.15.96:0/4212712938 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69b4003520 con 0x7f69b4004d40 2024-05-28T06:24:04.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.881+0000 7f69d5e00640 1 --2- 172.21.15.96:0/4212712938 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f69b4004d40 0x7f69b40051c0 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x7f69c800ba10 tx=0x7f69b4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5080465569136e63 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:04.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.881+0000 7f69cfe00640 1 -- 172.21.15.96:0/4212712938 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69c8030070 con 0x7f69b4004d40 2024-05-28T06:24:04.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.881+0000 7f69cfe00640 1 -- 172.21.15.96:0/4212712938 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69c802ade0 con 0x7f69b4004d40 2024-05-28T06:24:04.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.881+0000 7f69cfe00640 1 -- 172.21.15.96:0/4212712938 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69c800b610 con 0x7f69b4004d40 2024-05-28T06:24:04.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.882+0000 7f69d7e00640 1 -- 172.21.15.96:0/4212712938 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f69b4004d40 msgr2=0x7f69b40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:04.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.882+0000 7f69d7e00640 1 --2- 172.21.15.96:0/4212712938 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f69b4004d40 0x7f69b40051c0 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x7f69c800ba10 tx=0x7f69b4005830 comp rx=0 tx=0).stop 2024-05-28T06:24:04.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.882+0000 7f69d5e00640 1 -- 172.21.15.96:0/4212712938 reap_dead start 2024-05-28T06:24:04.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.882+0000 7f69d7e00640 1 -- 172.21.15.96:0/4212712938 shutdown_connections 2024-05-28T06:24:04.881 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.882+0000 7f69d7e00640 1 -- 172.21.15.96:0/4212712938 >> 172.21.15.96:0/4212712938 conn(0x7f69d006bae0 msgr2=0x7f69d006bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:04.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.882+0000 7f69d7e00640 1 -- 172.21.15.96:0/4212712938 shutdown_connections 2024-05-28T06:24:04.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.882+0000 7f69d7e00640 1 -- 172.21.15.96:0/4212712938 wait complete. 2024-05-28T06:24:04.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.884+0000 7f69d7e00640 1 Processor -- start 2024-05-28T06:24:04.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.884+0000 7f69d7e00640 1 -- start start 2024-05-28T06:24:04.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.885+0000 7f69d7e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f69b4004d40 0x7f69d01f9010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:04.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.885+0000 7f69d7e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f69d00c3d90 0x7f69d015c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:04.882 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.885+0000 7f69d7e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69d00c5f60 0x7f69d00c63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:04.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.885+0000 7f69d7e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f69d00d0dc0 con 0x7f69b4004d40 2024-05-28T06:24:04.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.885+0000 7f69d7e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f69d00b3b50 con 0x7f69d00c5f60 2024-05-28T06:24:04.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.885+0000 7f69d7e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f69d00ce230 con 0x7f69d00c3d90 2024-05-28T06:24:04.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.885+0000 7f69d5400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f69b4004d40 0x7f69d01f9010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:04.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.885+0000 7f69d5400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f69b4004d40 0x7f69d01f9010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:43090/0 (socket says 172.21.15.96:43090) 2024-05-28T06:24:04.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.885+0000 7f69d5400640 1 -- 172.21.15.96:0/1070725058 learned_addr learned my addr 172.21.15.96:0/1070725058 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:24:04.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.885+0000 7f69d5400640 1 -- 172.21.15.96:0/1070725058 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f69d00c3d90 msgr2=0x7f69d015c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:04.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.885+0000 7f69d5400640 1 --2- 172.21.15.96:0/1070725058 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f69d00c3d90 0x7f69d015c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:04.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.885+0000 7f69d5400640 1 -- 172.21.15.96:0/1070725058 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69d00c5f60 msgr2=0x7f69d00c63e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:04.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.885+0000 7f69d5400640 1 --2- 172.21.15.96:0/1070725058 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69d00c5f60 0x7f69d00c63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:04.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.885+0000 7f69d5400640 1 -- 172.21.15.96:0/1070725058 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69d00c4500 con 0x7f69b4004d40 2024-05-28T06:24:04.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.886+0000 7f69d5400640 1 --2- 172.21.15.96:0/1070725058 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f69b4004d40 0x7f69d01f9010 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 crypto rx=0x7f69c0007f00 tx=0x7f69c00115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:04.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.886+0000 7f69cd600640 1 -- 172.21.15.96:0/1070725058 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69c001a020 con 0x7f69b4004d40 2024-05-28T06:24:04.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.886+0000 7f69cd600640 1 -- 172.21.15.96:0/1070725058 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69c0016470 con 0x7f69b4004d40 2024-05-28T06:24:04.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.886+0000 7f69cd600640 1 -- 172.21.15.96:0/1070725058 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69c0016840 con 0x7f69b4004d40 2024-05-28T06:24:04.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.886+0000 7f69d7e00640 1 -- 172.21.15.96:0/1070725058 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69d00c47e0 con 0x7f69b4004d40 2024-05-28T06:24:04.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.886+0000 7f69d7e00640 1 -- 172.21.15.96:0/1070725058 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69d00c4da0 con 0x7f69b4004d40 2024-05-28T06:24:04.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.887+0000 7f69d7e00640 1 -- 172.21.15.96:0/1070725058 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69d00689e0 con 0x7f69b4004d40 2024-05-28T06:24:04.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.888+0000 7f69cd600640 1 -- 172.21.15.96:0/1070725058 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 42) ==== 93590+0+0 (secure 0 0 0) 0x7f69c0005d10 con 0x7f69b4004d40 2024-05-28T06:24:04.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.889+0000 7f69cd600640 1 --2- 172.21.15.96:0/1070725058 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f69b006e510 0x7f69b00709d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:04.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.889+0000 7f69cd600640 1 -- 172.21.15.96:0/1070725058 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 5879+0+0 (secure 0 0 0) 0x7f69c008bab0 con 0x7f69b4004d40 2024-05-28T06:24:04.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.890+0000 7f69d4a00640 1 --2- 172.21.15.96:0/1070725058 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f69b006e510 0x7f69b00709d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:04.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.891+0000 7f69d4a00640 1 --2- 172.21.15.96:0/1070725058 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f69b006e510 0x7f69b00709d0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f69bc001eb0 tx=0x7f69bc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:04.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:04.893+0000 7f69cd600640 1 -- 172.21.15.96:0/1070725058 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f69c0024030 con 0x7f69b4004d40 2024-05-28T06:24:05.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:05.093+0000 7f69d7e00640 1 -- 172.21.15.96:0/1070725058 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f69d00cf520 con 0x7f69b4004d40 2024-05-28T06:24:05.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:05.094+0000 7f69cd600640 1 -- 172.21.15.96:0/1070725058 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v118) ==== 80+0+0 (secure 0 0 0) 0x7f69c005b940 con 0x7f69b4004d40 2024-05-28T06:24:05.089 INFO:teuthology.orchestra.run.smithi096.stderr:listed 0 entries 2024-05-28T06:24:05.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:05.096+0000 7f69d7e00640 1 -- 172.21.15.96:0/1070725058 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f69b006e510 msgr2=0x7f69b00709d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:05.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:05.096+0000 7f69d7e00640 1 --2- 172.21.15.96:0/1070725058 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f69b006e510 0x7f69b00709d0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f69bc001eb0 tx=0x7f69bc009040 comp rx=0 tx=0).stop 2024-05-28T06:24:05.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:05.097+0000 7f69d7e00640 1 -- 172.21.15.96:0/1070725058 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f69b4004d40 msgr2=0x7f69d01f9010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:05.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:05.097+0000 7f69d7e00640 1 --2- 172.21.15.96:0/1070725058 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f69b4004d40 0x7f69d01f9010 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 crypto rx=0x7f69c0007f00 tx=0x7f69c00115a0 comp rx=0 tx=0).stop 2024-05-28T06:24:05.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:05.097+0000 7f69d7e00640 1 -- 172.21.15.96:0/1070725058 shutdown_connections 2024-05-28T06:24:05.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:05.097+0000 7f69d7e00640 1 --2- 172.21.15.96:0/1070725058 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f69b006e510 0x7f69b00709d0 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:05.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:05.097+0000 7f69d7e00640 1 --2- 172.21.15.96:0/1070725058 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69d00c5f60 0x7f69d00c63e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:05.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:05.097+0000 7f69d7e00640 1 --2- 172.21.15.96:0/1070725058 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f69d00c3d90 0x7f69d015c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:05.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:05.097+0000 7f69d7e00640 1 --2- 172.21.15.96:0/1070725058 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f69b4004d40 0x7f69d01f9010 unknown :-1 s=CLOSED pgs=1923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:05.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:05.097+0000 7f69d7e00640 1 -- 172.21.15.96:0/1070725058 >> 172.21.15.96:0/1070725058 conn(0x7f69d006bae0 msgr2=0x7f69d00afc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:05.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:05.097+0000 7f69d7e00640 1 -- 172.21.15.96:0/1070725058 shutdown_connections 2024-05-28T06:24:05.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:05.097+0000 7f69d7e00640 1 -- 172.21.15.96:0/1070725058 wait complete. 2024-05-28T06:24:05.108 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi096.front.sepia.ceph.com... 2024-05-28T06:24:05.108 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:24:05.108 DEBUG:teuthology.orchestra.run.smithi096:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-05-28T06:24:05.239 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi096.front.sepia.ceph.com 2024-05-28T06:24:05.239 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:24:05.239 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-28T06:24:05.244 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi096.stderr:ceph-fuse[79827]: fuse finished with error 0 and tester_r 0 2024-05-28T06:24:05.431 INFO:tasks.cephfs.mount:Terminating background process 2024-05-28T06:24:05.431 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-05-28T06:24:05.431 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:24:05.497 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052731,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248996k,nr_inodes=812249,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-05-28T06:24:05.497 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi123.front.sepia.ceph.com 2024-05-28T06:24:05.497 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:24:05.497 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-05-28T06:24:05.552 INFO:teuthology.orchestra.run.smithi123.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-05-28T06:24:05.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-28T06:24:05.553 INFO:tasks.cephfs.mount:Terminating background process 2024-05-28T06:24:05.553 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:24:05.553 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:24:05.590 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052731,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248996k,nr_inodes=812249,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-28T06:24:05.591 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi096.front.sepia.ceph.com 2024-05-28T06:24:05.591 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:24:05.591 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-05-28T06:24:05.674 INFO:teuthology.orchestra.run.smithi096.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-05-28T06:24:05.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-28T06:24:05.675 INFO:tasks.cephfs.mount:Terminating background process 2024-05-28T06:24:05.675 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-05-28T06:24:05.675 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:24:05.731 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052731,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248996k,nr_inodes=812249,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-05-28T06:24:05.732 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi123.front.sepia.ceph.com 2024-05-28T06:24:05.732 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:24:05.732 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-05-28T06:24:05.782 INFO:teuthology.orchestra.run.smithi123.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-05-28T06:24:05.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-28T06:24:05.783 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-28T06:24:05.988 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:05.994+0000 7faa3ca00640 1 Processor -- start 2024-05-28T06:24:05.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:05.994+0000 7faa3ca00640 1 -- start start 2024-05-28T06:24:05.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:05.995+0000 7faa3ca00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7faa380d0a20 0x7faa380997a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:05.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:05.995+0000 7faa3ca00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa380b51f0 con 0x7faa380d0590 2024-05-28T06:24:05.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:05.995+0000 7faa3ca00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa380aef60 con 0x7faa380b4e10 2024-05-28T06:24:05.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:05.995+0000 7faa3ca00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7faa380ad1d0 con 0x7faa380d0a20 2024-05-28T06:24:05.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:05.996+0000 7faa35400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7faa380d0590 0x7faa38099090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:55908/0 (socket says 172.21.15.96:55908) 2024-05-28T06:24:05.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:05.996+0000 7faa35400640 1 -- 172.21.15.96:0/274392443 learned_addr learned my addr 172.21.15.96:0/274392443 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:24:05.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:05.996+0000 7faa36800640 1 --2- 172.21.15.96:0/274392443 >> v2:172.21.15.96:3300/0 conn(0x7faa380d0a20 0x7faa380997a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:05.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:05.996+0000 7faa34a00640 1 -- 172.21.15.96:0/274392443 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2932578152 0 0) 0x7faa380b51f0 con 0x7faa380d0590 2024-05-28T06:24:05.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:05.996+0000 7faa34a00640 1 -- 172.21.15.96:0/274392443 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa1c003520 con 0x7faa380d0590 2024-05-28T06:24:05.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:05.997+0000 7faa34a00640 1 -- 172.21.15.96:0/274392443 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1042922481 0 0) 0x7faa380aef60 con 0x7faa380b4e10 2024-05-28T06:24:05.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:05.997+0000 7faa34a00640 1 -- 172.21.15.96:0/274392443 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa380b51f0 con 0x7faa380b4e10 2024-05-28T06:24:05.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:05.997+0000 7faa34a00640 1 -- 172.21.15.96:0/274392443 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3056201529 0 0) 0x7faa1c003520 con 0x7faa380d0590 2024-05-28T06:24:05.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:05.997+0000 7faa34a00640 1 -- 172.21.15.96:0/274392443 >> v2:172.21.15.96:3300/0 conn(0x7faa380d0a20 msgr2=0x7faa380997a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:05.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:05.997+0000 7faa34a00640 1 --2- 172.21.15.96:0/274392443 >> v2:172.21.15.96:3300/0 conn(0x7faa380d0a20 0x7faa380997a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:05.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:05.997+0000 7faa34a00640 1 -- 172.21.15.96:0/274392443 >> v1:172.21.15.123:6789/0 conn(0x7faa380b4e10 legacy=0x7faa380cde30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:05.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:05.997+0000 7faa34a00640 1 -- 172.21.15.96:0/274392443 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa380ad470 con 0x7faa380d0590 2024-05-28T06:24:05.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:05.997+0000 7faa34a00640 1 -- 172.21.15.96:0/274392443 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7faa240015f0 con 0x7faa380d0590 2024-05-28T06:24:05.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:05.997+0000 7faa36800640 1 --2- 172.21.15.96:0/274392443 >> v2:172.21.15.96:3300/0 conn(0x7faa380d0a20 0x7faa380997a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-28T06:24:05.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:05.997+0000 7faa34a00640 1 -- 172.21.15.96:0/274392443 >> v1:172.21.15.96:6789/0 conn(0x7faa380d0590 legacy=0x7faa38099090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:05.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:05.997+0000 7faa34a00640 1 --2- 172.21.15.96:0/274392443 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa1c003b90 0x7faa1c003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:05.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:05.997+0000 7faa34a00640 1 --2- 172.21.15.96:0/274392443 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faa1c004800 0x7faa1c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:05.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:05.997+0000 7faa34a00640 1 --2- 172.21.15.96:0/274392443 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa1c005290 0x7faa1c00dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:05.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:05.997+0000 7faa34a00640 1 -- 172.21.15.96:0/274392443 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7faa380ad1d0 con 0x7faa1c005290 2024-05-28T06:24:05.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:05.997+0000 7faa34a00640 1 -- 172.21.15.96:0/274392443 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7faa1c00e3f0 con 0x7faa1c003b90 2024-05-28T06:24:05.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:05.997+0000 7faa34a00640 1 -- 172.21.15.96:0/274392443 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7faa1c00e5f0 con 0x7faa1c004800 2024-05-28T06:24:05.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:05.997+0000 7faa36800640 1 --2- 172.21.15.96:0/274392443 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa1c005290 0x7faa1c00dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:05.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:05.998+0000 7faa35400640 1 --2- 172.21.15.96:0/274392443 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faa1c004800 0x7faa1c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:05.995 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:05.998+0000 7faa35e00640 1 --2- 172.21.15.96:0/274392443 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa1c003b90 0x7faa1c003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:05.995 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:05.998+0000 7faa36800640 1 -- 172.21.15.96:0/274392443 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faa1c004800 msgr2=0x7faa1c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:05.995 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:05.998+0000 7faa36800640 1 --2- 172.21.15.96:0/274392443 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faa1c004800 0x7faa1c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:05.995 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:05.998+0000 7faa36800640 1 -- 172.21.15.96:0/274392443 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa1c003b90 msgr2=0x7faa1c003f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:05.995 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:05.998+0000 7faa36800640 1 --2- 172.21.15.96:0/274392443 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa1c003b90 0x7faa1c003f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:05.995 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:05.998+0000 7faa36800640 1 -- 172.21.15.96:0/274392443 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa1c003520 con 0x7faa1c005290 2024-05-28T06:24:05.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:05.999+0000 7faa36800640 1 -- 172.21.15.96:0/274392443 reap_dead start 2024-05-28T06:24:05.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.000+0000 7faa36800640 1 --2- 172.21.15.96:0/274392443 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa1c005290 0x7faa1c00dbd0 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7faa300073e0 tx=0x7faa1c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca13b0837d07b048 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:05.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.000+0000 7faa34a00640 1 -- 172.21.15.96:0/274392443 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa30009c80 con 0x7faa1c005290 2024-05-28T06:24:05.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.000+0000 7faa34a00640 1 -- 172.21.15.96:0/274392443 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa3002c680 con 0x7faa1c005290 2024-05-28T06:24:05.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.000+0000 7faa34a00640 1 -- 172.21.15.96:0/274392443 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa3002ca50 con 0x7faa1c005290 2024-05-28T06:24:05.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.001+0000 7faa3ca00640 1 -- 172.21.15.96:0/274392443 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa1c005290 msgr2=0x7faa1c00dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:05.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.001+0000 7faa3ca00640 1 --2- 172.21.15.96:0/274392443 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa1c005290 0x7faa1c00dbd0 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7faa300073e0 tx=0x7faa1c005830 comp rx=0 tx=0).stop 2024-05-28T06:24:05.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.001+0000 7faa3ca00640 1 -- 172.21.15.96:0/274392443 shutdown_connections 2024-05-28T06:24:05.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.001+0000 7faa3ca00640 1 --2- 172.21.15.96:0/274392443 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa1c005290 0x7faa1c00dbd0 unknown :-1 s=CLOSED pgs=1925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:05.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.001+0000 7faa3ca00640 1 -- 172.21.15.96:0/274392443 >> 172.21.15.96:0/274392443 conn(0x7faa3806bae0 msgr2=0x7faa3806bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:05.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.001+0000 7faa3ca00640 1 -- 172.21.15.96:0/274392443 shutdown_connections 2024-05-28T06:24:05.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.001+0000 7faa3ca00640 1 -- 172.21.15.96:0/274392443 wait complete. 2024-05-28T06:24:05.998 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.002+0000 7faa3ca00640 1 Processor -- start 2024-05-28T06:24:05.998 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.003+0000 7faa3ca00640 1 -- start start 2024-05-28T06:24:05.998 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.003+0000 7faa3ca00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faa1c005290 0x7faa38157270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:05.998 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.003+0000 7faa3ca00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa3809a2b0 0x7faa3815d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:05.998 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.003+0000 7faa3ca00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa3815dae0 0x7faa380b0a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:05.998 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.003+0000 7faa3ca00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7faa380ce880 con 0x7faa3809a2b0 2024-05-28T06:24:05.998 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.003+0000 7faa3ca00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7faa380d0e30 con 0x7faa3815dae0 2024-05-28T06:24:05.998 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.003+0000 7faa3ca00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7faa380cea00 con 0x7faa1c005290 2024-05-28T06:24:06.000 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.004+0000 7faa35400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa3809a2b0 0x7faa3815d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:06.000 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.004+0000 7faa35400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa3809a2b0 0x7faa3815d5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:43134/0 (socket says 172.21.15.96:43134) 2024-05-28T06:24:06.000 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.004+0000 7faa35400640 1 -- 172.21.15.96:0/1647689816 learned_addr learned my addr 172.21.15.96:0/1647689816 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:24:06.000 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.004+0000 7faa35e00640 1 --2- 172.21.15.96:0/1647689816 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faa1c005290 0x7faa38157270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:06.000 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.004+0000 7faa36800640 1 --2- 172.21.15.96:0/1647689816 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa3815dae0 0x7faa380b0a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:06.000 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.004+0000 7faa35400640 1 -- 172.21.15.96:0/1647689816 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faa1c005290 msgr2=0x7faa38157270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:06.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.004+0000 7faa35400640 1 --2- 172.21.15.96:0/1647689816 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faa1c005290 0x7faa38157270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:06.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.004+0000 7faa35400640 1 -- 172.21.15.96:0/1647689816 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa3815dae0 msgr2=0x7faa380b0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:06.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.004+0000 7faa35400640 1 --2- 172.21.15.96:0/1647689816 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa3815dae0 0x7faa380b0a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:06.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.004+0000 7faa35400640 1 -- 172.21.15.96:0/1647689816 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa381577b0 con 0x7faa3809a2b0 2024-05-28T06:24:06.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.004+0000 7faa35e00640 1 --2- 172.21.15.96:0/1647689816 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faa1c005290 0x7faa38157270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-28T06:24:06.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.005+0000 7faa35400640 1 --2- 172.21.15.96:0/1647689816 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa3809a2b0 0x7faa3815d5a0 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7faa24004e20 tx=0x7faa24007430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:06.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.005+0000 7faa2e000640 1 -- 172.21.15.96:0/1647689816 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa2401a070 con 0x7faa3809a2b0 2024-05-28T06:24:06.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.005+0000 7faa2e000640 1 -- 172.21.15.96:0/1647689816 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa2400f7b0 con 0x7faa3809a2b0 2024-05-28T06:24:06.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.005+0000 7faa3ca00640 1 -- 172.21.15.96:0/1647689816 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa380c3ef0 con 0x7faa3809a2b0 2024-05-28T06:24:06.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.005+0000 7faa2e000640 1 -- 172.21.15.96:0/1647689816 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa2400fba0 con 0x7faa3809a2b0 2024-05-28T06:24:06.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.005+0000 7faa3ca00640 1 -- 172.21.15.96:0/1647689816 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa380c44b0 con 0x7faa3809a2b0 2024-05-28T06:24:06.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.006+0000 7faa3ca00640 1 -- 172.21.15.96:0/1647689816 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa380689e0 con 0x7faa3809a2b0 2024-05-28T06:24:06.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.009+0000 7faa2e000640 1 -- 172.21.15.96:0/1647689816 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 42) ==== 93590+0+0 (secure 0 0 0) 0x7faa2401d070 con 0x7faa3809a2b0 2024-05-28T06:24:06.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.010+0000 7faa2e000640 1 --2- 172.21.15.96:0/1647689816 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7faa1806e510 0x7faa180709d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:06.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.010+0000 7faa2e000640 1 -- 172.21.15.96:0/1647689816 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 5879+0+0 (secure 0 0 0) 0x7faa2408a2f0 con 0x7faa3809a2b0 2024-05-28T06:24:06.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.011+0000 7faa35e00640 1 --2- 172.21.15.96:0/1647689816 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7faa1806e510 0x7faa180709d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:06.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.012+0000 7faa35e00640 1 --2- 172.21.15.96:0/1647689816 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7faa1806e510 0x7faa180709d0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7faa280035f0 tx=0x7faa28009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:06.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.014+0000 7faa2e000640 1 -- 172.21.15.96:0/1647689816 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7faa381577b0 con 0x7faa3809a2b0 2024-05-28T06:24:06.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.235+0000 7faa3ca00640 1 -- 172.21.15.96:0/1647689816 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faa3806f540 con 0x7faa3809a2b0 2024-05-28T06:24:06.230 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.236+0000 7faa2e000640 1 -- 172.21.15.96:0/1647689816 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) ==== 94+0+5189 (secure 0 0 0) 0x7faa2405a220 con 0x7faa3809a2b0 2024-05-28T06:24:06.231 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:24:06.231 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":73,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6724,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/2148532848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6835","nonce":2148532848},{"type":"v1","addr":"172.21.15.123:6837","nonce":2148532848}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":9252,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6834/2173534467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2173534467},{"type":"v1","addr":"172.21.15.96:6834","nonce":2173534467}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":9255,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/2552082898","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6833","nonce":2552082898},{"type":"v1","addr":"172.21.15.96:6835","nonce":2552082898}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-28T06:22:16.424757+0000","modified":"2024-05-28T06:22:19.524390+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"b","rank":0,"incarnation":68,"state":"up:active","state_seq":3,"addr":"172.21.15.123:6836/3192559785","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":3192559785},{"type":"v1","addr":"172.21.15.123:6836","nonce":3192559785}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9270,"qdb_cluster":[9270]},"id":8}]} 2024-05-28T06:24:06.231 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 73 2024-05-28T06:24:06.233 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.238+0000 7faa3ca00640 1 -- 172.21.15.96:0/1647689816 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7faa1806e510 msgr2=0x7faa180709d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:06.233 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.238+0000 7faa3ca00640 1 --2- 172.21.15.96:0/1647689816 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7faa1806e510 0x7faa180709d0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7faa280035f0 tx=0x7faa28009040 comp rx=0 tx=0).stop 2024-05-28T06:24:06.233 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.238+0000 7faa3ca00640 1 -- 172.21.15.96:0/1647689816 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa3809a2b0 msgr2=0x7faa3815d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:06.233 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.238+0000 7faa3ca00640 1 --2- 172.21.15.96:0/1647689816 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa3809a2b0 0x7faa3815d5a0 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7faa24004e20 tx=0x7faa24007430 comp rx=0 tx=0).stop 2024-05-28T06:24:06.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.239+0000 7faa3ca00640 1 -- 172.21.15.96:0/1647689816 shutdown_connections 2024-05-28T06:24:06.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.239+0000 7faa3ca00640 1 --2- 172.21.15.96:0/1647689816 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7faa1806e510 0x7faa180709d0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:06.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.239+0000 7faa3ca00640 1 --2- 172.21.15.96:0/1647689816 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faa3815dae0 0x7faa380b0a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:06.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.239+0000 7faa3ca00640 1 --2- 172.21.15.96:0/1647689816 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa3809a2b0 0x7faa3815d5a0 unknown :-1 s=CLOSED pgs=1926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:06.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.239+0000 7faa3ca00640 1 --2- 172.21.15.96:0/1647689816 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7faa1c005290 0x7faa38157270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:06.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.239+0000 7faa3ca00640 1 -- 172.21.15.96:0/1647689816 >> 172.21.15.96:0/1647689816 conn(0x7faa3806bae0 msgr2=0x7faa38072ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:06.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.239+0000 7faa3ca00640 1 -- 172.21.15.96:0/1647689816 shutdown_connections 2024-05-28T06:24:06.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.239+0000 7faa3ca00640 1 -- 172.21.15.96:0/1647689816 wait complete. 2024-05-28T06:24:06.253 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:24:06.439 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.444+0000 7fda8b600640 1 Processor -- start 2024-05-28T06:24:06.439 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.444+0000 7fda8b600640 1 -- start start 2024-05-28T06:24:06.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.445+0000 7fda8b600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fda840d0a20 0x7fda840997a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:06.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.445+0000 7fda8b600640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda840b51f0 con 0x7fda840b4e10 2024-05-28T06:24:06.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.445+0000 7fda8b600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda840aef60 con 0x7fda840d0590 2024-05-28T06:24:06.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.445+0000 7fda8b600640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fda840ad1d0 con 0x7fda840d0a20 2024-05-28T06:24:06.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.446+0000 7fda88c00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fda840b4e10 0x7fda840cde30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:55916/0 (socket says 172.21.15.96:55916) 2024-05-28T06:24:06.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.446+0000 7fda88c00640 1 -- 172.21.15.96:0/1469595018 learned_addr learned my addr 172.21.15.96:0/1469595018 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:24:06.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.446+0000 7fda83400640 1 -- 172.21.15.96:0/1469595018 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 159746771 0 0) 0x7fda840b51f0 con 0x7fda840b4e10 2024-05-28T06:24:06.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.446+0000 7fda83400640 1 -- 172.21.15.96:0/1469595018 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fda6c005730 con 0x7fda840b4e10 2024-05-28T06:24:06.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.446+0000 7fda89600640 1 --2- 172.21.15.96:0/1469595018 >> v2:172.21.15.123:3300/0 conn(0x7fda840d0a20 0x7fda840997a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:06.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.446+0000 7fda83400640 1 -- 172.21.15.96:0/1469595018 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3587865834 0 0) 0x7fda6c005730 con 0x7fda840b4e10 2024-05-28T06:24:06.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.447+0000 7fda83400640 1 -- 172.21.15.96:0/1469595018 >> v2:172.21.15.123:3300/0 conn(0x7fda840d0a20 msgr2=0x7fda840997a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:06.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.447+0000 7fda83400640 1 --2- 172.21.15.96:0/1469595018 >> v2:172.21.15.123:3300/0 conn(0x7fda840d0a20 0x7fda840997a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:06.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.447+0000 7fda83400640 1 -- 172.21.15.96:0/1469595018 >> v1:172.21.15.123:6789/0 conn(0x7fda840d0590 legacy=0x7fda84099090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:06.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.447+0000 7fda83400640 1 -- 172.21.15.96:0/1469595018 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda840ad470 con 0x7fda840b4e10 2024-05-28T06:24:06.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.447+0000 7fda83400640 1 -- 172.21.15.96:0/1469595018 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7fda78001670 con 0x7fda840b4e10 2024-05-28T06:24:06.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.447+0000 7fda83400640 1 -- 172.21.15.96:0/1469595018 >> v1:172.21.15.96:6789/0 conn(0x7fda840b4e10 legacy=0x7fda840cde30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:06.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.447+0000 7fda83400640 1 --2- 172.21.15.96:0/1469595018 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fda6c005c10 0x7fda6c006030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:06.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.447+0000 7fda83400640 1 --2- 172.21.15.96:0/1469595018 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fda6c006a10 0x7fda6c006e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:06.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.447+0000 7fda83400640 1 --2- 172.21.15.96:0/1469595018 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fda6c007420 0x7fda6c00dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:06.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.447+0000 7fda83400640 1 -- 172.21.15.96:0/1469595018 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fda840ad1d0 con 0x7fda6c007420 2024-05-28T06:24:06.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.447+0000 7fda83400640 1 -- 172.21.15.96:0/1469595018 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fda6c00e370 con 0x7fda6c006a10 2024-05-28T06:24:06.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.447+0000 7fda83400640 1 -- 172.21.15.96:0/1469595018 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fda6c00e570 con 0x7fda6c005c10 2024-05-28T06:24:06.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.447+0000 7fda89600640 1 --2- 172.21.15.96:0/1469595018 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fda6c007420 0x7fda6c00dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:06.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.447+0000 7fda89600640 1 -- 172.21.15.96:0/1469595018 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fda6c005c10 msgr2=0x7fda6c006030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:06.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.447+0000 7fda89600640 1 --2- 172.21.15.96:0/1469595018 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fda6c005c10 0x7fda6c006030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:06.444 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.447+0000 7fda89600640 1 -- 172.21.15.96:0/1469595018 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fda6c006a10 msgr2=0x7fda6c006e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:06.444 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.447+0000 7fda89600640 1 --2- 172.21.15.96:0/1469595018 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fda6c006a10 0x7fda6c006e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:06.444 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.447+0000 7fda89600640 1 -- 172.21.15.96:0/1469595018 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda6c005730 con 0x7fda6c007420 2024-05-28T06:24:06.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.448+0000 7fda89600640 1 -- 172.21.15.96:0/1469595018 reap_dead start 2024-05-28T06:24:06.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.448+0000 7fda89600640 1 --2- 172.21.15.96:0/1469595018 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fda6c007420 0x7fda6c00dbd0 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 crypto rx=0x7fda7c009f90 tx=0x7fda6c007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=237c4ff15ed8165c server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:06.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.448+0000 7fda83400640 1 -- 172.21.15.96:0/1469595018 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda7c035020 con 0x7fda6c007420 2024-05-28T06:24:06.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.448+0000 7fda83400640 1 -- 172.21.15.96:0/1469595018 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda7c003b90 con 0x7fda6c007420 2024-05-28T06:24:06.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.448+0000 7fda83400640 1 -- 172.21.15.96:0/1469595018 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda7c009c10 con 0x7fda6c007420 2024-05-28T06:24:06.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.449+0000 7fda8b600640 1 -- 172.21.15.96:0/1469595018 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fda6c007420 msgr2=0x7fda6c00dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:06.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.449+0000 7fda8b600640 1 --2- 172.21.15.96:0/1469595018 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fda6c007420 0x7fda6c00dbd0 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 crypto rx=0x7fda7c009f90 tx=0x7fda6c007a40 comp rx=0 tx=0).stop 2024-05-28T06:24:06.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.449+0000 7fda8b600640 1 -- 172.21.15.96:0/1469595018 shutdown_connections 2024-05-28T06:24:06.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.449+0000 7fda8b600640 1 --2- 172.21.15.96:0/1469595018 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fda6c007420 0x7fda6c00dbd0 unknown :-1 s=CLOSED pgs=1928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:06.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.449+0000 7fda8b600640 1 -- 172.21.15.96:0/1469595018 >> 172.21.15.96:0/1469595018 conn(0x7fda8406bae0 msgr2=0x7fda8406bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:06.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.449+0000 7fda8b600640 1 -- 172.21.15.96:0/1469595018 shutdown_connections 2024-05-28T06:24:06.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.449+0000 7fda8b600640 1 -- 172.21.15.96:0/1469595018 wait complete. 2024-05-28T06:24:06.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.451+0000 7fda8b600640 1 Processor -- start 2024-05-28T06:24:06.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.451+0000 7fda8b600640 1 -- start start 2024-05-28T06:24:06.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.452+0000 7fda8b600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fda6c007420 0x7fda84156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:06.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.452+0000 7fda8b600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fda84151790 0x7fda84072b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:06.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.452+0000 7fda8b600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fda840730c0 0x7fda84073540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:06.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.452+0000 7fda8b600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fda840ce880 con 0x7fda84151790 2024-05-28T06:24:06.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.452+0000 7fda8b600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fda840d0e30 con 0x7fda6c007420 2024-05-28T06:24:06.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.452+0000 7fda8b600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fda840cea00 con 0x7fda840730c0 2024-05-28T06:24:06.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.452+0000 7fda83e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fda84151790 0x7fda84072b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:06.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.452+0000 7fda83e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fda84151790 0x7fda84072b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:43154/0 (socket says 172.21.15.96:43154) 2024-05-28T06:24:06.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.452+0000 7fda83e00640 1 -- 172.21.15.96:0/2134829575 learned_addr learned my addr 172.21.15.96:0/2134829575 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:24:06.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.453+0000 7fda83e00640 1 -- 172.21.15.96:0/2134829575 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fda840730c0 msgr2=0x7fda84073540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:06.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.453+0000 7fda83e00640 1 --2- 172.21.15.96:0/2134829575 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fda840730c0 0x7fda84073540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:06.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.453+0000 7fda83e00640 1 -- 172.21.15.96:0/2134829575 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fda6c007420 msgr2=0x7fda84156ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:06.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.453+0000 7fda83e00640 1 --2- 172.21.15.96:0/2134829575 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fda6c007420 0x7fda84156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:06.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.453+0000 7fda83e00640 1 -- 172.21.15.96:0/2134829575 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda84157410 con 0x7fda84151790 2024-05-28T06:24:06.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.453+0000 7fda83e00640 1 --2- 172.21.15.96:0/2134829575 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fda84151790 0x7fda84072b80 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7fda7000c500 tx=0x7fda7000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:06.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.453+0000 7fda80c00640 1 -- 172.21.15.96:0/2134829575 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda700052d0 con 0x7fda84151790 2024-05-28T06:24:06.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.453+0000 7fda80c00640 1 -- 172.21.15.96:0/2134829575 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda70005470 con 0x7fda84151790 2024-05-28T06:24:06.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.453+0000 7fda80c00640 1 -- 172.21.15.96:0/2134829575 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda70012850 con 0x7fda84151790 2024-05-28T06:24:06.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.453+0000 7fda8b600640 1 -- 172.21.15.96:0/2134829575 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda8415ded0 con 0x7fda84151790 2024-05-28T06:24:06.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.453+0000 7fda8b600640 1 -- 172.21.15.96:0/2134829575 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda841e4970 con 0x7fda84151790 2024-05-28T06:24:06.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.455+0000 7fda8b600640 1 -- 172.21.15.96:0/2134829575 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda841e44a0 con 0x7fda84151790 2024-05-28T06:24:06.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.456+0000 7fda80c00640 1 -- 172.21.15.96:0/2134829575 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 42) ==== 93590+0+0 (secure 0 0 0) 0x7fda700129f0 con 0x7fda84151790 2024-05-28T06:24:06.451 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.457+0000 7fda80c00640 1 --2- 172.21.15.96:0/2134829575 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fda6406e510 0x7fda640709d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:06.452 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.457+0000 7fda80c00640 1 -- 172.21.15.96:0/2134829575 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 5879+0+0 (secure 0 0 0) 0x7fda7008a720 con 0x7fda84151790 2024-05-28T06:24:06.452 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.458+0000 7fda88c00640 1 --2- 172.21.15.96:0/2134829575 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fda6406e510 0x7fda640709d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:06.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.459+0000 7fda88c00640 1 --2- 172.21.15.96:0/2134829575 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fda6406e510 0x7fda640709d0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fda780078d0 tx=0x7fda78001500 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:06.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.462+0000 7fda80c00640 1 -- 172.21.15.96:0/2134829575 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fda7005a650 con 0x7fda84151790 2024-05-28T06:24:06.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.669+0000 7fda8b600640 1 -- 172.21.15.96:0/2134829575 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fda840689e0 con 0x7fda84151790 2024-05-28T06:24:06.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.670+0000 7fda80c00640 1 -- 172.21.15.96:0/2134829575 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v118) ==== 74+0+16936 (secure 0 0 0) 0x7fda70059f50 con 0x7fda84151790 2024-05-28T06:24:06.666 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:24:06.666 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":118,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:24:01.754615+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-05-28T06:22:14.138685+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"114","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-05-28T06:22:15.394361+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"118","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":118,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.c","mappings":[{"from":0,"to":2}]},{"pgid":"17.17","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":13,"snaps":[{"begin":2,"length":2}]},{"pool":17,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[{"pool":17,"snaps":[{"begin":2,"length":2}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:24:06.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.673+0000 7fda8b600640 1 -- 172.21.15.96:0/2134829575 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fda6406e510 msgr2=0x7fda640709d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:06.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.673+0000 7fda8b600640 1 --2- 172.21.15.96:0/2134829575 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fda6406e510 0x7fda640709d0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fda780078d0 tx=0x7fda78001500 comp rx=0 tx=0).stop 2024-05-28T06:24:06.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.673+0000 7fda8b600640 1 -- 172.21.15.96:0/2134829575 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fda84151790 msgr2=0x7fda84072b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:06.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.673+0000 7fda8b600640 1 --2- 172.21.15.96:0/2134829575 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fda84151790 0x7fda84072b80 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7fda7000c500 tx=0x7fda7000c9c0 comp rx=0 tx=0).stop 2024-05-28T06:24:06.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.673+0000 7fda8b600640 1 -- 172.21.15.96:0/2134829575 shutdown_connections 2024-05-28T06:24:06.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.673+0000 7fda8b600640 1 --2- 172.21.15.96:0/2134829575 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fda6406e510 0x7fda640709d0 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:06.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.673+0000 7fda8b600640 1 --2- 172.21.15.96:0/2134829575 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fda840730c0 0x7fda84073540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:06.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.673+0000 7fda8b600640 1 --2- 172.21.15.96:0/2134829575 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fda84151790 0x7fda84072b80 unknown :-1 s=CLOSED pgs=1929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:06.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.673+0000 7fda8b600640 1 --2- 172.21.15.96:0/2134829575 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fda6c007420 0x7fda84156ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:06.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.673+0000 7fda8b600640 1 -- 172.21.15.96:0/2134829575 >> 172.21.15.96:0/2134829575 conn(0x7fda8406bae0 msgr2=0x7fda840ca810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:06.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.673+0000 7fda8b600640 1 -- 172.21.15.96:0/2134829575 shutdown_connections 2024-05-28T06:24:06.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.673+0000 7fda8b600640 1 -- 172.21.15.96:0/2134829575 wait complete. 2024-05-28T06:24:06.686 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-28T06:03:59.517102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-28T06:22:14.138685+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '114', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-05-28T06:22:15.394361+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '118', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 118, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-28T06:24:06.686 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-28T06:24:06.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.893+0000 7f3d36a00640 1 Processor -- start 2024-05-28T06:24:06.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.893+0000 7f3d36a00640 1 -- start start 2024-05-28T06:24:06.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.895+0000 7f3d36a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3d3015e870 0x7f3d300d0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:06.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.895+0000 7f3d36a00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d3015ec50 con 0x7f3d300d14d0 2024-05-28T06:24:06.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.895+0000 7f3d36a00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d300cff80 con 0x7f3d300cfa40 2024-05-28T06:24:06.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.895+0000 7f3d36a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f3d300d2150 con 0x7f3d3015e870 2024-05-28T06:24:06.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.895+0000 7f3d34c00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f3d300d14d0 0x7f3d300cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:55924/0 (socket says 172.21.15.96:55924) 2024-05-28T06:24:06.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.895+0000 7f3d34c00640 1 -- 172.21.15.96:0/2543693518 learned_addr learned my addr 172.21.15.96:0/2543693518 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:24:06.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.896+0000 7f3d35600640 1 --2- 172.21.15.96:0/2543693518 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3d3015e870 0x7f3d300d0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:06.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.896+0000 7f3d2fe00640 1 -- 172.21.15.96:0/2543693518 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3760676234 0 0) 0x7f3d3015ec50 con 0x7f3d300d14d0 2024-05-28T06:24:06.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.896+0000 7f3d2fe00640 1 -- 172.21.15.96:0/2543693518 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d18003520 con 0x7f3d300d14d0 2024-05-28T06:24:06.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.896+0000 7f3d2fe00640 1 -- 172.21.15.96:0/2543693518 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 988111184 0 0) 0x7f3d18003520 con 0x7f3d300d14d0 2024-05-28T06:24:06.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.896+0000 7f3d2fe00640 1 -- 172.21.15.96:0/2543693518 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3d3015e870 msgr2=0x7f3d300d0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:06.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.896+0000 7f3d2fe00640 1 --2- 172.21.15.96:0/2543693518 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3d3015e870 0x7f3d300d0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:06.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.896+0000 7f3d2fe00640 1 -- 172.21.15.96:0/2543693518 >> v1:172.21.15.123:6789/0 conn(0x7f3d300cfa40 legacy=0x7f3d300d25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:06.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.896+0000 7f3d2fe00640 1 -- 172.21.15.96:0/2543693518 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d300d0260 con 0x7f3d300d14d0 2024-05-28T06:24:06.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.896+0000 7f3d2fe00640 1 -- 172.21.15.96:0/2543693518 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f3d200017f0 con 0x7f3d300d14d0 2024-05-28T06:24:06.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.896+0000 7f3d2fe00640 1 -- 172.21.15.96:0/2543693518 >> v1:172.21.15.96:6789/0 conn(0x7f3d300d14d0 legacy=0x7f3d300cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:06.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.896+0000 7f3d2fe00640 1 --2- 172.21.15.96:0/2543693518 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3d180039e0 0x7f3d18003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:06.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.896+0000 7f3d2fe00640 1 --2- 172.21.15.96:0/2543693518 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3d3015e870 0x7f3d18004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:06.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.896+0000 7f3d2fe00640 1 --2- 172.21.15.96:0/2543693518 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d18004d40 0x7f3d180051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:06.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.896+0000 7f3d2fe00640 1 -- 172.21.15.96:0/2543693518 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f3d300d2150 con 0x7f3d180039e0 2024-05-28T06:24:06.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.896+0000 7f3d2fe00640 1 -- 172.21.15.96:0/2543693518 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f3d1800ba70 con 0x7f3d18004d40 2024-05-28T06:24:06.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.896+0000 7f3d2fe00640 1 -- 172.21.15.96:0/2543693518 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f3d1800bc70 con 0x7f3d3015e870 2024-05-28T06:24:06.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.897+0000 7f3d35600640 1 --2- 172.21.15.96:0/2543693518 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3d180039e0 0x7f3d18003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:06.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.897+0000 7f3d34c00640 1 --2- 172.21.15.96:0/2543693518 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3d3015e870 0x7f3d18004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:06.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.897+0000 7f3d35600640 1 -- 172.21.15.96:0/2543693518 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3d3015e870 msgr2=0x7f3d18004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:06.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.897+0000 7f3d35600640 1 --2- 172.21.15.96:0/2543693518 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3d3015e870 0x7f3d18004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:06.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.897+0000 7f3d35600640 1 -- 172.21.15.96:0/2543693518 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d18004d40 msgr2=0x7f3d180051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:06.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.897+0000 7f3d36000640 1 --2- 172.21.15.96:0/2543693518 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d18004d40 0x7f3d180051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:06.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.897+0000 7f3d35600640 1 --2- 172.21.15.96:0/2543693518 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d18004d40 0x7f3d180051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:06.893 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.897+0000 7f3d35600640 1 -- 172.21.15.96:0/2543693518 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d18003520 con 0x7f3d180039e0 2024-05-28T06:24:06.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.898+0000 7f3d35600640 1 --2- 172.21.15.96:0/2543693518 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3d180039e0 0x7f3d18003de0 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 crypto rx=0x7f3d24009fc0 tx=0x7f3d18005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7365b5d5e774d079 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:06.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.898+0000 7f3d2fe00640 1 -- 172.21.15.96:0/2543693518 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d24009c10 con 0x7f3d180039e0 2024-05-28T06:24:06.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.898+0000 7f3d2fe00640 1 -- 172.21.15.96:0/2543693518 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d24036070 con 0x7f3d180039e0 2024-05-28T06:24:06.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.898+0000 7f3d2fe00640 1 -- 172.21.15.96:0/2543693518 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d24031880 con 0x7f3d180039e0 2024-05-28T06:24:06.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.899+0000 7f3d36a00640 1 -- 172.21.15.96:0/2543693518 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3d180039e0 msgr2=0x7f3d18003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:06.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.899+0000 7f3d36a00640 1 --2- 172.21.15.96:0/2543693518 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3d180039e0 0x7f3d18003de0 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 crypto rx=0x7f3d24009fc0 tx=0x7f3d18005830 comp rx=0 tx=0).stop 2024-05-28T06:24:06.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.899+0000 7f3d36000640 1 -- 172.21.15.96:0/2543693518 reap_dead start 2024-05-28T06:24:06.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.899+0000 7f3d36a00640 1 -- 172.21.15.96:0/2543693518 shutdown_connections 2024-05-28T06:24:06.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.899+0000 7f3d36a00640 1 -- 172.21.15.96:0/2543693518 >> 172.21.15.96:0/2543693518 conn(0x7f3d3008bfc0 msgr2=0x7f3d300d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:06.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.899+0000 7f3d36a00640 1 -- 172.21.15.96:0/2543693518 shutdown_connections 2024-05-28T06:24:06.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.899+0000 7f3d36a00640 1 -- 172.21.15.96:0/2543693518 wait complete. 2024-05-28T06:24:06.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.901+0000 7f3d36a00640 1 Processor -- start 2024-05-28T06:24:06.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.901+0000 7f3d36a00640 1 -- start start 2024-05-28T06:24:06.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.902+0000 7f3d36a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3d180039e0 0x7f3d3015ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:06.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.902+0000 7f3d36a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d300c9de0 0x7f3d300b47e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:06.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.902+0000 7f3d36a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3d30160570 0x7f3d301596b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:06.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.902+0000 7f3d36a00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f3d300d4fb0 con 0x7f3d180039e0 2024-05-28T06:24:06.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.902+0000 7f3d36a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f3d300d0820 con 0x7f3d300c9de0 2024-05-28T06:24:06.897 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.902+0000 7f3d36a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f3d300d0f90 con 0x7f3d30160570 2024-05-28T06:24:06.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.902+0000 7f3d35600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3d180039e0 0x7f3d3015ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:06.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.902+0000 7f3d35600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3d180039e0 0x7f3d3015ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:43164/0 (socket says 172.21.15.96:43164) 2024-05-28T06:24:06.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.902+0000 7f3d35600640 1 -- 172.21.15.96:0/3651099346 learned_addr learned my addr 172.21.15.96:0/3651099346 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:24:06.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.902+0000 7f3d36000640 1 --2- 172.21.15.96:0/3651099346 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3d30160570 0x7f3d301596b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:06.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.903+0000 7f3d35600640 1 -- 172.21.15.96:0/3651099346 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3d30160570 msgr2=0x7f3d301596b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:06.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.903+0000 7f3d35600640 1 --2- 172.21.15.96:0/3651099346 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3d30160570 0x7f3d301596b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:06.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.903+0000 7f3d35600640 1 -- 172.21.15.96:0/3651099346 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d300c9de0 msgr2=0x7f3d300b47e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:06.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.903+0000 7f3d35600640 1 --2- 172.21.15.96:0/3651099346 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d300c9de0 0x7f3d300b47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:06.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.903+0000 7f3d35600640 1 -- 172.21.15.96:0/3651099346 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d300cafe0 con 0x7f3d180039e0 2024-05-28T06:24:06.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.903+0000 7f3d35600640 1 --2- 172.21.15.96:0/3651099346 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3d180039e0 0x7f3d3015ee60 secure :-1 s=READY pgs=1932 cs=0 l=1 rev1=1 crypto rx=0x7f3d2402aee0 tx=0x7f3d24009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:06.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.903+0000 7f3d2d600640 1 -- 172.21.15.96:0/3651099346 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d24009c10 con 0x7f3d180039e0 2024-05-28T06:24:06.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.903+0000 7f3d2d600640 1 -- 172.21.15.96:0/3651099346 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d24003990 con 0x7f3d180039e0 2024-05-28T06:24:06.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.903+0000 7f3d2d600640 1 -- 172.21.15.96:0/3651099346 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d24003d60 con 0x7f3d180039e0 2024-05-28T06:24:06.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.903+0000 7f3d36a00640 1 -- 172.21.15.96:0/3651099346 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d3015f4d0 con 0x7f3d180039e0 2024-05-28T06:24:06.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.903+0000 7f3d36a00640 1 -- 172.21.15.96:0/3651099346 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d3015fa90 con 0x7f3d180039e0 2024-05-28T06:24:06.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.904+0000 7f3d36a00640 1 -- 172.21.15.96:0/3651099346 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d300689e0 con 0x7f3d180039e0 2024-05-28T06:24:06.900 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.906+0000 7f3d2d600640 1 -- 172.21.15.96:0/3651099346 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 42) ==== 93590+0+0 (secure 0 0 0) 0x7f3d24005d10 con 0x7f3d180039e0 2024-05-28T06:24:06.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.907+0000 7f3d2d600640 1 --2- 172.21.15.96:0/3651099346 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3d1406e510 0x7f3d140709d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:06.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.907+0000 7f3d2d600640 1 -- 172.21.15.96:0/3651099346 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 5879+0+0 (secure 0 0 0) 0x7f3d240abdc0 con 0x7f3d180039e0 2024-05-28T06:24:06.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.908+0000 7f3d34c00640 1 --2- 172.21.15.96:0/3651099346 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3d1406e510 0x7f3d140709d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:06.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.909+0000 7f3d34c00640 1 --2- 172.21.15.96:0/3651099346 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3d1406e510 0x7f3d140709d0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f3d20001020 tx=0x7f3d20009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:06.906 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:06.912+0000 7f3d2d600640 1 -- 172.21.15.96:0/3651099346 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3d2407bc50 con 0x7f3d180039e0 2024-05-28T06:24:07.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.097+0000 7f3d36a00640 1 -- 172.21.15.96:0/3651099346 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f3d300d3ca0 con 0x7f3d180039e0 2024-05-28T06:24:07.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.098+0000 7f3d2d600640 1 -- 172.21.15.96:0/3651099346 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v118) ==== 93+0+10 (secure 0 0 0) 0x7f3d2407b550 con 0x7f3d180039e0 2024-05-28T06:24:07.093 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 1 2024-05-28T06:24:07.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.100+0000 7f3d36a00640 1 -- 172.21.15.96:0/3651099346 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3d1406e510 msgr2=0x7f3d140709d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:07.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.100+0000 7f3d36a00640 1 --2- 172.21.15.96:0/3651099346 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3d1406e510 0x7f3d140709d0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f3d20001020 tx=0x7f3d20009040 comp rx=0 tx=0).stop 2024-05-28T06:24:07.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.101+0000 7f3d36a00640 1 -- 172.21.15.96:0/3651099346 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3d180039e0 msgr2=0x7f3d3015ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:07.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.101+0000 7f3d36a00640 1 --2- 172.21.15.96:0/3651099346 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3d180039e0 0x7f3d3015ee60 secure :-1 s=READY pgs=1932 cs=0 l=1 rev1=1 crypto rx=0x7f3d2402aee0 tx=0x7f3d24009750 comp rx=0 tx=0).stop 2024-05-28T06:24:07.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.101+0000 7f3d36a00640 1 -- 172.21.15.96:0/3651099346 shutdown_connections 2024-05-28T06:24:07.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.101+0000 7f3d36a00640 1 --2- 172.21.15.96:0/3651099346 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f3d1406e510 0x7f3d140709d0 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:07.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.101+0000 7f3d36a00640 1 --2- 172.21.15.96:0/3651099346 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3d30160570 0x7f3d301596b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:07.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.101+0000 7f3d36a00640 1 --2- 172.21.15.96:0/3651099346 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d300c9de0 0x7f3d300b47e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:07.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.101+0000 7f3d36a00640 1 --2- 172.21.15.96:0/3651099346 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3d180039e0 0x7f3d3015ee60 unknown :-1 s=CLOSED pgs=1932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:07.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.101+0000 7f3d36a00640 1 -- 172.21.15.96:0/3651099346 >> 172.21.15.96:0/3651099346 conn(0x7f3d3008bfc0 msgr2=0x7f3d300b10f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:07.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.101+0000 7f3d36a00640 1 -- 172.21.15.96:0/3651099346 shutdown_connections 2024-05-28T06:24:07.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.101+0000 7f3d36a00640 1 -- 172.21.15.96:0/3651099346 wait complete. 2024-05-28T06:24:07.115 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-28T06:24:07.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.326+0000 7fa1de600640 1 Processor -- start 2024-05-28T06:24:07.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.326+0000 7fa1de600640 1 -- start start 2024-05-28T06:24:07.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.328+0000 7fa1de600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fa1d815d270 0x7fa1d80b4d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:07.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.328+0000 7fa1de600640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1d815d680 con 0x7fa1d8153510 2024-05-28T06:24:07.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.328+0000 7fa1de600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1d815d900 con 0x7fa1d80cd2a0 2024-05-28T06:24:07.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.328+0000 7fa1de600640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fa1d81538f0 con 0x7fa1d815d270 2024-05-28T06:24:07.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.328+0000 7fa1dd200640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fa1d8153510 0x7fa1d80ccb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:55926/0 (socket says 172.21.15.96:55926) 2024-05-28T06:24:07.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.328+0000 7fa1dd200640 1 -- 172.21.15.96:0/3854031039 learned_addr learned my addr 172.21.15.96:0/3854031039 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:24:07.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.328+0000 7fa1ddc00640 1 --2- 172.21.15.96:0/3854031039 >> v2:172.21.15.96:3300/0 conn(0x7fa1d815d270 0x7fa1d80b4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:07.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.328+0000 7fa1ddc00640 1 -- 172.21.15.96:0/3854031039 >> v1:172.21.15.123:6789/0 conn(0x7fa1d80cd2a0 legacy=0x7fa1d815cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:07.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.328+0000 7fa1ddc00640 1 -- 172.21.15.96:0/3854031039 >> v1:172.21.15.96:6789/0 conn(0x7fa1d8153510 legacy=0x7fa1d80ccb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:07.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.329+0000 7fa1ddc00640 1 -- 172.21.15.96:0/3854031039 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1d80b3f90 con 0x7fa1d815d270 2024-05-28T06:24:07.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.330+0000 7fa1ddc00640 1 --2- 172.21.15.96:0/3854031039 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa1d815d270 0x7fa1d80b4d50 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x7fa1d0009d90 tx=0x7fa1d002a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4efb987c93c3211a server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:07.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.330+0000 7fa1d7400640 1 -- 172.21.15.96:0/3854031039 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1d0031600 con 0x7fa1d815d270 2024-05-28T06:24:07.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.330+0000 7fa1d7400640 1 -- 172.21.15.96:0/3854031039 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1d00317a0 con 0x7fa1d815d270 2024-05-28T06:24:07.325 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.330+0000 7fa1d7400640 1 -- 172.21.15.96:0/3854031039 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1d0031b70 con 0x7fa1d815d270 2024-05-28T06:24:07.325 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.330+0000 7fa1de600640 1 -- 172.21.15.96:0/3854031039 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa1d815d270 msgr2=0x7fa1d80b4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:07.325 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.330+0000 7fa1de600640 1 --2- 172.21.15.96:0/3854031039 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa1d815d270 0x7fa1d80b4d50 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x7fa1d0009d90 tx=0x7fa1d002a610 comp rx=0 tx=0).stop 2024-05-28T06:24:07.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.331+0000 7fa1de600640 1 -- 172.21.15.96:0/3854031039 shutdown_connections 2024-05-28T06:24:07.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.331+0000 7fa1de600640 1 --2- 172.21.15.96:0/3854031039 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa1d815d270 0x7fa1d80b4d50 unknown :-1 s=CLOSED pgs=1934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:07.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.331+0000 7fa1de600640 1 -- 172.21.15.96:0/3854031039 >> 172.21.15.96:0/3854031039 conn(0x7fa1d806b900 msgr2=0x7fa1d806bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:07.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.331+0000 7fa1de600640 1 -- 172.21.15.96:0/3854031039 shutdown_connections 2024-05-28T06:24:07.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.331+0000 7fa1de600640 1 -- 172.21.15.96:0/3854031039 wait complete. 2024-05-28T06:24:07.327 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.332+0000 7fa1de600640 1 Processor -- start 2024-05-28T06:24:07.327 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.332+0000 7fa1de600640 1 -- start start 2024-05-28T06:24:07.328 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.333+0000 7fa1de600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa1d80cd2a0 0x7fa1d8151d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:07.328 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.333+0000 7fa1de600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa1d8153510 0x7fa1d8153a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:07.328 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.333+0000 7fa1de600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa1d815d270 0x7fa1d8072f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:07.328 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.333+0000 7fa1de600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa1d80b4550 con 0x7fa1d8153510 2024-05-28T06:24:07.328 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.333+0000 7fa1de600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa1d80b48c0 con 0x7fa1d815d270 2024-05-28T06:24:07.328 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.333+0000 7fa1de600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fa1d806e940 con 0x7fa1d80cd2a0 2024-05-28T06:24:07.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.334+0000 7fa1d7e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa1d8153510 0x7fa1d8153a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:07.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.334+0000 7fa1d7e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa1d8153510 0x7fa1d8153a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:43186/0 (socket says 172.21.15.96:43186) 2024-05-28T06:24:07.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.334+0000 7fa1d7e00640 1 -- 172.21.15.96:0/4141231295 learned_addr learned my addr 172.21.15.96:0/4141231295 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:24:07.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.334+0000 7fa1dd200640 1 --2- 172.21.15.96:0/4141231295 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa1d80cd2a0 0x7fa1d8151d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:07.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.334+0000 7fa1d7e00640 1 -- 172.21.15.96:0/4141231295 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa1d80cd2a0 msgr2=0x7fa1d8151d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:07.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.334+0000 7fa1d7e00640 1 --2- 172.21.15.96:0/4141231295 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa1d80cd2a0 0x7fa1d8151d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:07.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.334+0000 7fa1d7e00640 1 -- 172.21.15.96:0/4141231295 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa1d815d270 msgr2=0x7fa1d8072f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:07.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.334+0000 7fa1d7e00640 1 --2- 172.21.15.96:0/4141231295 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa1d815d270 0x7fa1d8072f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:07.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.334+0000 7fa1d7e00640 1 -- 172.21.15.96:0/4141231295 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1d8154f00 con 0x7fa1d8153510 2024-05-28T06:24:07.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.334+0000 7fa1dd200640 1 --2- 172.21.15.96:0/4141231295 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa1d80cd2a0 0x7fa1d8151d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:24:07.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.334+0000 7fa1d7e00640 1 --2- 172.21.15.96:0/4141231295 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa1d8153510 0x7fa1d8153a80 secure :-1 s=READY pgs=1935 cs=0 l=1 rev1=1 crypto rx=0x7fa1c400c430 tx=0x7fa1c400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:07.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.334+0000 7fa1d4c00640 1 -- 172.21.15.96:0/4141231295 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1c4011790 con 0x7fa1d8153510 2024-05-28T06:24:07.331 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.335+0000 7fa1d4c00640 1 -- 172.21.15.96:0/4141231295 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1c4017070 con 0x7fa1d8153510 2024-05-28T06:24:07.331 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.335+0000 7fa1d4c00640 1 -- 172.21.15.96:0/4141231295 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1c4011bb0 con 0x7fa1d8153510 2024-05-28T06:24:07.331 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.335+0000 7fa1de600640 1 -- 172.21.15.96:0/4141231295 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1d8154380 con 0x7fa1d8153510 2024-05-28T06:24:07.331 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.335+0000 7fa1de600640 1 -- 172.21.15.96:0/4141231295 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1d8154940 con 0x7fa1d8153510 2024-05-28T06:24:07.331 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.336+0000 7fa1de600640 1 -- 172.21.15.96:0/4141231295 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1d80689e0 con 0x7fa1d8153510 2024-05-28T06:24:07.331 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.337+0000 7fa1d4c00640 1 -- 172.21.15.96:0/4141231295 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 42) ==== 93590+0+0 (secure 0 0 0) 0x7fa1c401c460 con 0x7fa1d8153510 2024-05-28T06:24:07.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.338+0000 7fa1d4c00640 1 --2- 172.21.15.96:0/4141231295 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa1b806e510 0x7fa1b80709d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:07.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.338+0000 7fa1d4c00640 1 -- 172.21.15.96:0/4141231295 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 5879+0+0 (secure 0 0 0) 0x7fa1c408a970 con 0x7fa1d8153510 2024-05-28T06:24:07.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.339+0000 7fa1dd200640 1 --2- 172.21.15.96:0/4141231295 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa1b806e510 0x7fa1b80709d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:07.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.340+0000 7fa1dd200640 1 --2- 172.21.15.96:0/4141231295 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa1b806e510 0x7fa1b80709d0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fa1d80cac40 tx=0x7fa1cc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:07.337 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.343+0000 7fa1d4c00640 1 -- 172.21.15.96:0/4141231295 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa1c405a800 con 0x7fa1d8153510 2024-05-28T06:24:07.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.540+0000 7fa1de600640 1 -- 172.21.15.96:0/4141231295 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fa1d80b3c40 con 0x7fa1d8153510 2024-05-28T06:24:07.536 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.541+0000 7fa1d4c00640 1 -- 172.21.15.96:0/4141231295 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v118) ==== 104+0+11 (secure 0 0 0) 0x7fa1c405a100 con 0x7fa1d8153510 2024-05-28T06:24:07.536 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 32 2024-05-28T06:24:07.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.545+0000 7fa1de600640 1 -- 172.21.15.96:0/4141231295 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa1b806e510 msgr2=0x7fa1b80709d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:07.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.545+0000 7fa1de600640 1 --2- 172.21.15.96:0/4141231295 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa1b806e510 0x7fa1b80709d0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fa1d80cac40 tx=0x7fa1cc009040 comp rx=0 tx=0).stop 2024-05-28T06:24:07.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.545+0000 7fa1de600640 1 -- 172.21.15.96:0/4141231295 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa1d8153510 msgr2=0x7fa1d8153a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:07.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.545+0000 7fa1de600640 1 --2- 172.21.15.96:0/4141231295 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa1d8153510 0x7fa1d8153a80 secure :-1 s=READY pgs=1935 cs=0 l=1 rev1=1 crypto rx=0x7fa1c400c430 tx=0x7fa1c400c8f0 comp rx=0 tx=0).stop 2024-05-28T06:24:07.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.545+0000 7fa1de600640 1 -- 172.21.15.96:0/4141231295 shutdown_connections 2024-05-28T06:24:07.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.545+0000 7fa1de600640 1 --2- 172.21.15.96:0/4141231295 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa1b806e510 0x7fa1b80709d0 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:07.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.545+0000 7fa1de600640 1 --2- 172.21.15.96:0/4141231295 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa1d815d270 0x7fa1d8072f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:07.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.545+0000 7fa1de600640 1 --2- 172.21.15.96:0/4141231295 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa1d8153510 0x7fa1d8153a80 unknown :-1 s=CLOSED pgs=1935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:07.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.545+0000 7fa1de600640 1 --2- 172.21.15.96:0/4141231295 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa1d80cd2a0 0x7fa1d8151d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:07.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.545+0000 7fa1de600640 1 -- 172.21.15.96:0/4141231295 >> 172.21.15.96:0/4141231295 conn(0x7fa1d806b900 msgr2=0x7fa1d815fee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:07.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.545+0000 7fa1de600640 1 -- 172.21.15.96:0/4141231295 shutdown_connections 2024-05-28T06:24:07.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.545+0000 7fa1de600640 1 -- 172.21.15.96:0/4141231295 wait complete. 2024-05-28T06:24:07.559 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-28T06:24:07.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.759+0000 7fa412000640 1 Processor -- start 2024-05-28T06:24:07.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.759+0000 7fa412000640 1 -- start start 2024-05-28T06:24:07.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.761+0000 7fa412000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fa40c15d5d0 0x7fa40c0cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:07.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.761+0000 7fa412000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fa40c0b32c0 0x7fa40c0b3700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:07.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.761+0000 7fa412000640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa40c15cb10 con 0x7fa40c0d0640 2024-05-28T06:24:07.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.761+0000 7fa412000640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fa40c0adbb0 con 0x7fa40c15d5d0 2024-05-28T06:24:07.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.761+0000 7fa412000640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fa40c0ad5f0 con 0x7fa40c0b32c0 2024-05-28T06:24:07.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.761+0000 7fa40aa00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fa40c15d5d0 0x7fa40c0cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:07.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.761+0000 7fa40aa00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fa40c15d5d0 0x7fa40c0cf950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:43202/0 (socket says 172.21.15.96:43202) 2024-05-28T06:24:07.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.761+0000 7fa40aa00640 1 -- 172.21.15.96:0/1711944470 learned_addr learned my addr 172.21.15.96:0/1711944470 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:24:07.757 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.762+0000 7fa40aa00640 1 -- 172.21.15.96:0/1711944470 >> v2:172.21.15.123:3300/0 conn(0x7fa40c0b32c0 msgr2=0x7fa40c0b3700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:07.757 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.762+0000 7fa40aa00640 1 --2- 172.21.15.96:0/1711944470 >> v2:172.21.15.123:3300/0 conn(0x7fa40c0b32c0 0x7fa40c0b3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:07.757 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.762+0000 7fa40aa00640 1 -- 172.21.15.96:0/1711944470 >> v1:172.21.15.123:6789/0 conn(0x7fa40c0d0640 legacy=0x7fa40c0d0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:07.757 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.762+0000 7fa40aa00640 1 -- 172.21.15.96:0/1711944470 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa40c0ad8d0 con 0x7fa40c15d5d0 2024-05-28T06:24:07.757 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.763+0000 7fa40aa00640 1 --2- 172.21.15.96:0/1711944470 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa40c15d5d0 0x7fa40c0cf950 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 crypto rx=0x7fa40c0d1200 tx=0x7fa3f402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8beebb8a1cfc2ef8 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:07.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.763+0000 7fa40a000640 1 -- 172.21.15.96:0/1711944470 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3f4037070 con 0x7fa40c15d5d0 2024-05-28T06:24:07.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.763+0000 7fa40a000640 1 -- 172.21.15.96:0/1711944470 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3f4032440 con 0x7fa40c15d5d0 2024-05-28T06:24:07.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.763+0000 7fa40a000640 1 -- 172.21.15.96:0/1711944470 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3f4032810 con 0x7fa40c15d5d0 2024-05-28T06:24:07.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.763+0000 7fa412000640 1 -- 172.21.15.96:0/1711944470 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa40c15d5d0 msgr2=0x7fa40c0cf950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:07.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.763+0000 7fa412000640 1 --2- 172.21.15.96:0/1711944470 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa40c15d5d0 0x7fa40c0cf950 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 crypto rx=0x7fa40c0d1200 tx=0x7fa3f402a1a0 comp rx=0 tx=0).stop 2024-05-28T06:24:07.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.763+0000 7fa412000640 1 -- 172.21.15.96:0/1711944470 shutdown_connections 2024-05-28T06:24:07.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.763+0000 7fa412000640 1 --2- 172.21.15.96:0/1711944470 >> v2:172.21.15.123:3300/0 conn(0x7fa40c0b32c0 0x7fa40c0b3700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:07.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.763+0000 7fa412000640 1 --2- 172.21.15.96:0/1711944470 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa40c15d5d0 0x7fa40c0cf950 unknown :-1 s=CLOSED pgs=1936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:07.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.763+0000 7fa412000640 1 -- 172.21.15.96:0/1711944470 >> 172.21.15.96:0/1711944470 conn(0x7fa40c06bd20 msgr2=0x7fa40c06c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:07.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.764+0000 7fa412000640 1 -- 172.21.15.96:0/1711944470 shutdown_connections 2024-05-28T06:24:07.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.764+0000 7fa412000640 1 -- 172.21.15.96:0/1711944470 wait complete. 2024-05-28T06:24:07.764 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.765+0000 7fa412000640 1 Processor -- start 2024-05-28T06:24:07.764 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.765+0000 7fa412000640 1 -- start start 2024-05-28T06:24:07.764 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.766+0000 7fa412000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa40c0b32c0 0x7fa40c09df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:07.764 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.766+0000 7fa412000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa40c0d0640 0x7fa40c13ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:07.764 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.766+0000 7fa412000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa40c15d5d0 0x7fa40c13f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:07.764 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.766+0000 7fa412000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa40c15da50 con 0x7fa40c15d5d0 2024-05-28T06:24:07.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.766+0000 7fa412000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa40c15d0b0 con 0x7fa40c0b32c0 2024-05-28T06:24:07.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.766+0000 7fa412000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fa40c0ad2a0 con 0x7fa40c0d0640 2024-05-28T06:24:07.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.767+0000 7fa40be00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa40c15d5d0 0x7fa40c13f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:07.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.767+0000 7fa40be00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa40c15d5d0 0x7fa40c13f2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:43204/0 (socket says 172.21.15.96:43204) 2024-05-28T06:24:07.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.767+0000 7fa40be00640 1 -- 172.21.15.96:0/2316598142 learned_addr learned my addr 172.21.15.96:0/2316598142 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:24:07.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.767+0000 7fa40aa00640 1 --2- 172.21.15.96:0/2316598142 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa40c0d0640 0x7fa40c13ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:07.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.767+0000 7fa40be00640 1 -- 172.21.15.96:0/2316598142 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa40c0d0640 msgr2=0x7fa40c13ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:07.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.767+0000 7fa40b400640 1 --2- 172.21.15.96:0/2316598142 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa40c0b32c0 0x7fa40c09df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:07.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.767+0000 7fa40be00640 1 --2- 172.21.15.96:0/2316598142 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa40c0d0640 0x7fa40c13ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:07.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.767+0000 7fa40be00640 1 -- 172.21.15.96:0/2316598142 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa40c0b32c0 msgr2=0x7fa40c09df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:07.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.767+0000 7fa40be00640 1 --2- 172.21.15.96:0/2316598142 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa40c0b32c0 0x7fa40c09df20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:07.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.767+0000 7fa40be00640 1 -- 172.21.15.96:0/2316598142 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa40c1e57c0 con 0x7fa40c15d5d0 2024-05-28T06:24:07.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.767+0000 7fa40aa00640 1 --2- 172.21.15.96:0/2316598142 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa40c0d0640 0x7fa40c13ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:24:07.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.767+0000 7fa40be00640 1 --2- 172.21.15.96:0/2316598142 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa40c15d5d0 0x7fa40c13f2a0 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7fa40400b410 tx=0x7fa40400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:07.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.768+0000 7fa403400640 1 -- 172.21.15.96:0/2316598142 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa404015020 con 0x7fa40c15d5d0 2024-05-28T06:24:07.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.768+0000 7fa403400640 1 -- 172.21.15.96:0/2316598142 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa404011850 con 0x7fa40c15d5d0 2024-05-28T06:24:07.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.768+0000 7fa403400640 1 -- 172.21.15.96:0/2316598142 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa404011c40 con 0x7fa40c15d5d0 2024-05-28T06:24:07.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.768+0000 7fa412000640 1 -- 172.21.15.96:0/2316598142 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa40c0b27e0 con 0x7fa40c15d5d0 2024-05-28T06:24:07.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.768+0000 7fa412000640 1 -- 172.21.15.96:0/2316598142 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa40c1f4090 con 0x7fa40c15d5d0 2024-05-28T06:24:07.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.769+0000 7fa412000640 1 -- 172.21.15.96:0/2316598142 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa40c0689e0 con 0x7fa40c15d5d0 2024-05-28T06:24:07.767 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.770+0000 7fa403400640 1 -- 172.21.15.96:0/2316598142 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 42) ==== 93590+0+0 (secure 0 0 0) 0x7fa404018020 con 0x7fa40c15d5d0 2024-05-28T06:24:07.767 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.772+0000 7fa403400640 1 --2- 172.21.15.96:0/2316598142 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa3e806e540 0x7fa3e8070a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:07.767 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.772+0000 7fa403400640 1 -- 172.21.15.96:0/2316598142 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 5879+0+0 (secure 0 0 0) 0x7fa40408aa40 con 0x7fa40c15d5d0 2024-05-28T06:24:07.767 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.772+0000 7fa40b400640 1 --2- 172.21.15.96:0/2316598142 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa3e806e540 0x7fa3e8070a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:07.767 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.773+0000 7fa40b400640 1 --2- 172.21.15.96:0/2316598142 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa3e806e540 0x7fa3e8070a00 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fa3fc007dc0 tx=0x7fa3fc0072f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:07.770 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.776+0000 7fa403400640 1 -- 172.21.15.96:0/2316598142 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa40401f020 con 0x7fa40c15d5d0 2024-05-28T06:24:07.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.962+0000 7fa412000640 1 -- 172.21.15.96:0/2316598142 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fa40c150920 con 0x7fa40c15d5d0 2024-05-28T06:24:07.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.963+0000 7fa403400640 1 -- 172.21.15.96:0/2316598142 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v118) ==== 100+0+11 (secure 0 0 0) 0x7fa4040172c0 con 0x7fa40c15d5d0 2024-05-28T06:24:07.958 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 64 2024-05-28T06:24:07.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.966+0000 7fa412000640 1 -- 172.21.15.96:0/2316598142 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa3e806e540 msgr2=0x7fa3e8070a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:07.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.966+0000 7fa412000640 1 --2- 172.21.15.96:0/2316598142 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa3e806e540 0x7fa3e8070a00 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fa3fc007dc0 tx=0x7fa3fc0072f0 comp rx=0 tx=0).stop 2024-05-28T06:24:07.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.966+0000 7fa412000640 1 -- 172.21.15.96:0/2316598142 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa40c15d5d0 msgr2=0x7fa40c13f2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:07.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.966+0000 7fa412000640 1 --2- 172.21.15.96:0/2316598142 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa40c15d5d0 0x7fa40c13f2a0 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7fa40400b410 tx=0x7fa40400b8d0 comp rx=0 tx=0).stop 2024-05-28T06:24:07.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.966+0000 7fa412000640 1 -- 172.21.15.96:0/2316598142 shutdown_connections 2024-05-28T06:24:07.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.966+0000 7fa412000640 1 --2- 172.21.15.96:0/2316598142 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa3e806e540 0x7fa3e8070a00 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:07.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.966+0000 7fa412000640 1 --2- 172.21.15.96:0/2316598142 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa40c15d5d0 0x7fa40c13f2a0 unknown :-1 s=CLOSED pgs=1937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:07.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.966+0000 7fa412000640 1 --2- 172.21.15.96:0/2316598142 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa40c0d0640 0x7fa40c13ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:07.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.966+0000 7fa412000640 1 --2- 172.21.15.96:0/2316598142 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa40c0b32c0 0x7fa40c09df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:07.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.966+0000 7fa412000640 1 -- 172.21.15.96:0/2316598142 >> 172.21.15.96:0/2316598142 conn(0x7fa40c06bd20 msgr2=0x7fa40c0d0a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:07.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.966+0000 7fa412000640 1 -- 172.21.15.96:0/2316598142 shutdown_connections 2024-05-28T06:24:07.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:07.966+0000 7fa412000640 1 -- 172.21.15.96:0/2316598142 wait complete. 2024-05-28T06:24:07.977 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-28T06:24:08.166 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.171+0000 7fe0b9800640 1 Processor -- start 2024-05-28T06:24:08.166 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.172+0000 7fe0b9800640 1 -- start start 2024-05-28T06:24:08.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.173+0000 7fe0b9800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fe0b415e870 0x7fe0b40d0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:08.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.173+0000 7fe0b9800640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0b415ec50 con 0x7fe0b40d14d0 2024-05-28T06:24:08.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.173+0000 7fe0b9800640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0b40cff80 con 0x7fe0b40cfa40 2024-05-28T06:24:08.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.173+0000 7fe0b9800640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fe0b40d2150 con 0x7fe0b415e870 2024-05-28T06:24:08.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.173+0000 7fe0ab400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fe0b40d14d0 0x7fe0b40cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:55940/0 (socket says 172.21.15.96:55940) 2024-05-28T06:24:08.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.173+0000 7fe0ab400640 1 -- 172.21.15.96:0/636063931 learned_addr learned my addr 172.21.15.96:0/636063931 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:24:08.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.174+0000 7fe0b3e00640 1 --2- 172.21.15.96:0/636063931 >> v2:172.21.15.123:3300/0 conn(0x7fe0b415e870 0x7fe0b40d0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:08.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.174+0000 7fe0b3400640 1 -- 172.21.15.96:0/636063931 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1669156984 0 0) 0x7fe0b415ec50 con 0x7fe0b40d14d0 2024-05-28T06:24:08.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.174+0000 7fe0b3400640 1 -- 172.21.15.96:0/636063931 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe098003520 con 0x7fe0b40d14d0 2024-05-28T06:24:08.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.174+0000 7fe0b3400640 1 -- 172.21.15.96:0/636063931 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 628516966 0 0) 0x7fe098003520 con 0x7fe0b40d14d0 2024-05-28T06:24:08.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.174+0000 7fe0b3400640 1 -- 172.21.15.96:0/636063931 >> v2:172.21.15.123:3300/0 conn(0x7fe0b415e870 msgr2=0x7fe0b40d0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:08.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.174+0000 7fe0b3400640 1 --2- 172.21.15.96:0/636063931 >> v2:172.21.15.123:3300/0 conn(0x7fe0b415e870 0x7fe0b40d0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:08.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.174+0000 7fe0b3400640 1 -- 172.21.15.96:0/636063931 >> v1:172.21.15.123:6789/0 conn(0x7fe0b40cfa40 legacy=0x7fe0b40d25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:08.171 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.174+0000 7fe0b3400640 1 -- 172.21.15.96:0/636063931 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0b40d0260 con 0x7fe0b40d14d0 2024-05-28T06:24:08.171 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.174+0000 7fe0b3400640 1 -- 172.21.15.96:0/636063931 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7fe09c0017f0 con 0x7fe0b40d14d0 2024-05-28T06:24:08.171 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.174+0000 7fe0b3400640 1 -- 172.21.15.96:0/636063931 >> v1:172.21.15.96:6789/0 conn(0x7fe0b40d14d0 legacy=0x7fe0b40cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:08.171 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.174+0000 7fe0b3400640 1 --2- 172.21.15.96:0/636063931 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe0980039b0 0x7fe098003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:08.171 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.174+0000 7fe0b3400640 1 --2- 172.21.15.96:0/636063931 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe098004800 0x7fe098004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:08.171 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.175+0000 7fe0b3400640 1 --2- 172.21.15.96:0/636063931 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe0980051c0 0x7fe09800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:08.171 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.175+0000 7fe0b3400640 1 -- 172.21.15.96:0/636063931 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe0b40d2150 con 0x7fe0980051c0 2024-05-28T06:24:08.171 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.175+0000 7fe0b3400640 1 -- 172.21.15.96:0/636063931 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe09800c110 con 0x7fe098004800 2024-05-28T06:24:08.171 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.175+0000 7fe0b3400640 1 -- 172.21.15.96:0/636063931 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fe09800c310 con 0x7fe0980039b0 2024-05-28T06:24:08.171 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.175+0000 7fe0b8e00640 1 --2- 172.21.15.96:0/636063931 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe0980051c0 0x7fe09800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:08.171 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.175+0000 7fe0b3e00640 1 --2- 172.21.15.96:0/636063931 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe0980039b0 0x7fe098003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:08.171 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.175+0000 7fe0ab400640 1 --2- 172.21.15.96:0/636063931 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe098004800 0x7fe098004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:08.171 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.175+0000 7fe0b8e00640 1 -- 172.21.15.96:0/636063931 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe0980039b0 msgr2=0x7fe098003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:08.172 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.175+0000 7fe0b8e00640 1 --2- 172.21.15.96:0/636063931 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe0980039b0 0x7fe098003dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:08.172 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.175+0000 7fe0b8e00640 1 -- 172.21.15.96:0/636063931 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe098004800 msgr2=0x7fe098004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:08.172 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.175+0000 7fe0b8e00640 1 --2- 172.21.15.96:0/636063931 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe098004800 0x7fe098004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:08.172 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.175+0000 7fe0b8e00640 1 -- 172.21.15.96:0/636063931 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe098003520 con 0x7fe0980051c0 2024-05-28T06:24:08.172 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.175+0000 7fe0ab400640 1 --2- 172.21.15.96:0/636063931 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe098004800 0x7fe098004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:24:08.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.176+0000 7fe0b8e00640 1 -- 172.21.15.96:0/636063931 reap_dead start 2024-05-28T06:24:08.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.176+0000 7fe0b8e00640 1 --2- 172.21.15.96:0/636063931 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe0980051c0 0x7fe09800b9c0 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 crypto rx=0x7fe0ac001e20 tx=0x7fe098005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c74ba67a5211ea2e server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:08.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.176+0000 7fe0b3400640 1 -- 172.21.15.96:0/636063931 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0ac001a70 con 0x7fe0980051c0 2024-05-28T06:24:08.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.177+0000 7fe0b3400640 1 -- 172.21.15.96:0/636063931 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0ac02ae40 con 0x7fe0980051c0 2024-05-28T06:24:08.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.177+0000 7fe0b3400640 1 -- 172.21.15.96:0/636063931 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0ac003bb0 con 0x7fe0980051c0 2024-05-28T06:24:08.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.177+0000 7fe0b9800640 1 -- 172.21.15.96:0/636063931 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe0980051c0 msgr2=0x7fe09800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:08.173 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.177+0000 7fe0b9800640 1 --2- 172.21.15.96:0/636063931 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe0980051c0 0x7fe09800b9c0 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 crypto rx=0x7fe0ac001e20 tx=0x7fe098005830 comp rx=0 tx=0).stop 2024-05-28T06:24:08.174 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.177+0000 7fe0b9800640 1 -- 172.21.15.96:0/636063931 shutdown_connections 2024-05-28T06:24:08.174 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.177+0000 7fe0b9800640 1 --2- 172.21.15.96:0/636063931 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe0980051c0 0x7fe09800b9c0 unknown :-1 s=CLOSED pgs=1939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:08.174 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.177+0000 7fe0b9800640 1 -- 172.21.15.96:0/636063931 >> 172.21.15.96:0/636063931 conn(0x7fe0b408bfc0 msgr2=0x7fe0b40d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:08.174 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.177+0000 7fe0b9800640 1 -- 172.21.15.96:0/636063931 shutdown_connections 2024-05-28T06:24:08.174 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.177+0000 7fe0b9800640 1 -- 172.21.15.96:0/636063931 wait complete. 2024-05-28T06:24:08.174 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.179+0000 7fe0b9800640 1 Processor -- start 2024-05-28T06:24:08.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.179+0000 7fe0b9800640 1 -- start start 2024-05-28T06:24:08.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.180+0000 7fe0b9800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe0980051c0 0x7fe0b40c9640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:08.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.180+0000 7fe0b9800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe0b41528c0 0x7fe0b41540a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:08.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.180+0000 7fe0b9800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe0b40c74f0 0x7fe0b41545e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:08.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.180+0000 7fe0b9800640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe0b40d4fb0 con 0x7fe0b41528c0 2024-05-28T06:24:08.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.180+0000 7fe0b9800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe0b40d0820 con 0x7fe0980051c0 2024-05-28T06:24:08.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.180+0000 7fe0b9800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fe0b40d0f90 con 0x7fe0b40c74f0 2024-05-28T06:24:08.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.180+0000 7fe0ab400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe0b41528c0 0x7fe0b41540a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:08.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.180+0000 7fe0ab400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe0b41528c0 0x7fe0b41540a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:43226/0 (socket says 172.21.15.96:43226) 2024-05-28T06:24:08.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.180+0000 7fe0ab400640 1 -- 172.21.15.96:0/1544999564 learned_addr learned my addr 172.21.15.96:0/1544999564 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:24:08.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.181+0000 7fe0b3e00640 1 --2- 172.21.15.96:0/1544999564 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe0980051c0 0x7fe0b40c9640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:08.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.181+0000 7fe0ab400640 1 -- 172.21.15.96:0/1544999564 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe0b40c74f0 msgr2=0x7fe0b41545e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:08.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.181+0000 7fe0b8e00640 1 --2- 172.21.15.96:0/1544999564 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe0b40c74f0 0x7fe0b41545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:08.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.181+0000 7fe0ab400640 1 --2- 172.21.15.96:0/1544999564 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe0b40c74f0 0x7fe0b41545e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:08.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.181+0000 7fe0ab400640 1 -- 172.21.15.96:0/1544999564 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe0980051c0 msgr2=0x7fe0b40c9640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:08.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.181+0000 7fe0ab400640 1 --2- 172.21.15.96:0/1544999564 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe0980051c0 0x7fe0b40c9640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:08.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.181+0000 7fe0ab400640 1 -- 172.21.15.96:0/1544999564 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0b40cabd0 con 0x7fe0b41528c0 2024-05-28T06:24:08.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.181+0000 7fe0b3e00640 1 --2- 172.21.15.96:0/1544999564 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe0980051c0 0x7fe0b40c9640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:24:08.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.181+0000 7fe0ab400640 1 --2- 172.21.15.96:0/1544999564 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe0b41528c0 0x7fe0b41540a0 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 crypto rx=0x7fe09c00d750 tx=0x7fe09c00dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:08.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.181+0000 7fe0b0c00640 1 -- 172.21.15.96:0/1544999564 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe09c00de60 con 0x7fe0b41528c0 2024-05-28T06:24:08.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.181+0000 7fe0b0c00640 1 -- 172.21.15.96:0/1544999564 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe09c017070 con 0x7fe0b41528c0 2024-05-28T06:24:08.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.181+0000 7fe0b9800640 1 -- 172.21.15.96:0/1544999564 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0b40c8b60 con 0x7fe0b41528c0 2024-05-28T06:24:08.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.181+0000 7fe0b0c00640 1 -- 172.21.15.96:0/1544999564 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe09c0057a0 con 0x7fe0b41528c0 2024-05-28T06:24:08.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.182+0000 7fe0b9800640 1 -- 172.21.15.96:0/1544999564 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0b40caeb0 con 0x7fe0b41528c0 2024-05-28T06:24:08.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.183+0000 7fe0b9800640 1 -- 172.21.15.96:0/1544999564 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0b40689e0 con 0x7fe0b41528c0 2024-05-28T06:24:08.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.184+0000 7fe0b0c00640 1 -- 172.21.15.96:0/1544999564 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 42) ==== 93590+0+0 (secure 0 0 0) 0x7fe09c005940 con 0x7fe0b41528c0 2024-05-28T06:24:08.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.185+0000 7fe0b0c00640 1 --2- 172.21.15.96:0/1544999564 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe09006e510 0x7fe0900709d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:08.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.185+0000 7fe0b0c00640 1 -- 172.21.15.96:0/1544999564 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 5879+0+0 (secure 0 0 0) 0x7fe09c08aba0 con 0x7fe0b41528c0 2024-05-28T06:24:08.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.185+0000 7fe0b3e00640 1 --2- 172.21.15.96:0/1544999564 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe09006e510 0x7fe0900709d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:08.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.187+0000 7fe0b3e00640 1 --2- 172.21.15.96:0/1544999564 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe09006e510 0x7fe0900709d0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fe0b4159ed0 tx=0x7fe0a4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:08.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.189+0000 7fe0b0c00640 1 -- 172.21.15.96:0/1544999564 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe09c05aaf0 con 0x7fe0b41528c0 2024-05-28T06:24:08.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.434+0000 7fe0b9800640 1 -- 172.21.15.96:0/1544999564 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe0b40b18e0 con 0x7fe0b41528c0 2024-05-28T06:24:08.430 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.435+0000 7fe0b0c00640 1 -- 172.21.15.96:0/1544999564 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) ==== 94+0+5189 (secure 0 0 0) 0x7fe09c05a3f0 con 0x7fe0b41528c0 2024-05-28T06:24:08.430 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:24:08.430 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":73,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6724,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/2148532848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6835","nonce":2148532848},{"type":"v1","addr":"172.21.15.123:6837","nonce":2148532848}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":9252,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6834/2173534467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2173534467},{"type":"v1","addr":"172.21.15.96:6834","nonce":2173534467}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":9255,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/2552082898","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6833","nonce":2552082898},{"type":"v1","addr":"172.21.15.96:6835","nonce":2552082898}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-28T06:22:16.424757+0000","modified":"2024-05-28T06:22:19.524390+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"b","rank":0,"incarnation":68,"state":"up:active","state_seq":3,"addr":"172.21.15.123:6836/3192559785","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":3192559785},{"type":"v1","addr":"172.21.15.123:6836","nonce":3192559785}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9270,"qdb_cluster":[9270]},"id":8}]} 2024-05-28T06:24:08.431 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 73 2024-05-28T06:24:08.433 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.438+0000 7fe0b9800640 1 -- 172.21.15.96:0/1544999564 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe09006e510 msgr2=0x7fe0900709d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:08.433 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.438+0000 7fe0b9800640 1 --2- 172.21.15.96:0/1544999564 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe09006e510 0x7fe0900709d0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fe0b4159ed0 tx=0x7fe0a4008040 comp rx=0 tx=0).stop 2024-05-28T06:24:08.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.438+0000 7fe0b9800640 1 -- 172.21.15.96:0/1544999564 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe0b41528c0 msgr2=0x7fe0b41540a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:08.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.438+0000 7fe0b9800640 1 --2- 172.21.15.96:0/1544999564 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe0b41528c0 0x7fe0b41540a0 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 crypto rx=0x7fe09c00d750 tx=0x7fe09c00dc10 comp rx=0 tx=0).stop 2024-05-28T06:24:08.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.439+0000 7fe0b9800640 1 -- 172.21.15.96:0/1544999564 shutdown_connections 2024-05-28T06:24:08.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.439+0000 7fe0b9800640 1 --2- 172.21.15.96:0/1544999564 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe09006e510 0x7fe0900709d0 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:08.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.439+0000 7fe0b9800640 1 --2- 172.21.15.96:0/1544999564 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe0b40c74f0 0x7fe0b41545e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:08.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.439+0000 7fe0b9800640 1 --2- 172.21.15.96:0/1544999564 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe0b41528c0 0x7fe0b41540a0 unknown :-1 s=CLOSED pgs=1940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:08.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.439+0000 7fe0b9800640 1 --2- 172.21.15.96:0/1544999564 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe0980051c0 0x7fe0b40c9640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:08.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.439+0000 7fe0b9800640 1 -- 172.21.15.96:0/1544999564 >> 172.21.15.96:0/1544999564 conn(0x7fe0b408bfc0 msgr2=0x7fe0b4158c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:08.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.439+0000 7fe0b9800640 1 -- 172.21.15.96:0/1544999564 shutdown_connections 2024-05-28T06:24:08.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.439+0000 7fe0b9800640 1 -- 172.21.15.96:0/1544999564 wait complete. 2024-05-28T06:24:08.453 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:24:08.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.660+0000 7fab22800640 1 Processor -- start 2024-05-28T06:24:08.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.660+0000 7fab22800640 1 -- start start 2024-05-28T06:24:08.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.661+0000 7fab22800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fab1c0b4b60 0x7fab1c0cfaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:08.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.661+0000 7fab22800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fab1c0b3b70 0x7fab1c0d0930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:08.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.661+0000 7fab22800640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab1c0ced40 con 0x7fab1c0d04a0 2024-05-28T06:24:08.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.661+0000 7fab22800640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fab1c0b3fe0 con 0x7fab1c0b4b60 2024-05-28T06:24:08.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.661+0000 7fab22800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fab1c15fbb0 con 0x7fab1c0b3b70 2024-05-28T06:24:08.657 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.661+0000 7fab13e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fab1c0d04a0 0x7fab1c0b3460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:55946/0 (socket says 172.21.15.96:55946) 2024-05-28T06:24:08.657 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.661+0000 7fab13e00640 1 -- 172.21.15.96:0/1334536169 learned_addr learned my addr 172.21.15.96:0/1334536169 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:24:08.657 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.661+0000 7fab1be00640 1 --2- 172.21.15.96:0/1334536169 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fab1c0b3b70 0x7fab1c0d0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:08.657 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.661+0000 7fab1b400640 1 --2- 172.21.15.96:0/1334536169 >> v2:172.21.15.123:3300/0 conn(0x7fab1c0b4b60 0x7fab1c0cfaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:08.657 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.662+0000 7fab1aa00640 1 -- 172.21.15.96:0/1334536169 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2464131542 0 0) 0x7fab04001110 con 0x7fab1c0d04a0 2024-05-28T06:24:08.657 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.662+0000 7fab1aa00640 1 -- 172.21.15.96:0/1334536169 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fab00003520 con 0x7fab1c0d04a0 2024-05-28T06:24:08.657 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.662+0000 7fab1aa00640 1 -- 172.21.15.96:0/1334536169 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4251192759 0 0) 0x7fab00003520 con 0x7fab1c0d04a0 2024-05-28T06:24:08.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.662+0000 7fab1aa00640 1 -- 172.21.15.96:0/1334536169 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fab1c0b3b70 msgr2=0x7fab1c0d0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:08.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.662+0000 7fab1aa00640 1 --2- 172.21.15.96:0/1334536169 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fab1c0b3b70 0x7fab1c0d0930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:08.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.662+0000 7fab1aa00640 1 -- 172.21.15.96:0/1334536169 >> v2:172.21.15.123:3300/0 conn(0x7fab1c0b4b60 msgr2=0x7fab1c0cfaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:08.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.662+0000 7fab1aa00640 1 --2- 172.21.15.96:0/1334536169 >> v2:172.21.15.123:3300/0 conn(0x7fab1c0b4b60 0x7fab1c0cfaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:08.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.662+0000 7fab1aa00640 1 -- 172.21.15.96:0/1334536169 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab1c0cefc0 con 0x7fab1c0d04a0 2024-05-28T06:24:08.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.662+0000 7fab1aa00640 1 -- 172.21.15.96:0/1334536169 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7fab04001860 con 0x7fab1c0d04a0 2024-05-28T06:24:08.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.662+0000 7fab1aa00640 1 -- 172.21.15.96:0/1334536169 >> v1:172.21.15.96:6789/0 conn(0x7fab1c0d04a0 legacy=0x7fab1c0b3460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:08.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.662+0000 7fab1aa00640 1 --2- 172.21.15.96:0/1334536169 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fab000039b0 0x7fab00003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:08.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.662+0000 7fab1aa00640 1 --2- 172.21.15.96:0/1334536169 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fab1c0b3b70 0x7fab00004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:08.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.662+0000 7fab1aa00640 1 --2- 172.21.15.96:0/1334536169 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fab00004d40 0x7fab000051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:08.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.662+0000 7fab1aa00640 1 -- 172.21.15.96:0/1334536169 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fab1c15fbb0 con 0x7fab00004d40 2024-05-28T06:24:08.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.662+0000 7fab1aa00640 1 -- 172.21.15.96:0/1334536169 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fab0000b9c0 con 0x7fab000039b0 2024-05-28T06:24:08.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.662+0000 7fab1aa00640 1 -- 172.21.15.96:0/1334536169 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fab0000bb40 con 0x7fab1c0b3b70 2024-05-28T06:24:08.659 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.663+0000 7fab1be00640 1 --2- 172.21.15.96:0/1334536169 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fab00004d40 0x7fab000051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:08.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.663+0000 7fab1b400640 1 --2- 172.21.15.96:0/1334536169 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fab000039b0 0x7fab00003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:08.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.663+0000 7fab1be00640 1 -- 172.21.15.96:0/1334536169 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fab1c0b3b70 msgr2=0x7fab00004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:08.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.663+0000 7fab1be00640 1 --2- 172.21.15.96:0/1334536169 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fab1c0b3b70 0x7fab00004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:08.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.663+0000 7fab1be00640 1 -- 172.21.15.96:0/1334536169 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fab000039b0 msgr2=0x7fab00003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:08.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.663+0000 7fab1be00640 1 --2- 172.21.15.96:0/1334536169 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fab000039b0 0x7fab00003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:08.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.663+0000 7fab1be00640 1 -- 172.21.15.96:0/1334536169 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab00003520 con 0x7fab00004d40 2024-05-28T06:24:08.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.664+0000 7fab1be00640 1 --2- 172.21.15.96:0/1334536169 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fab00004d40 0x7fab000051c0 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 crypto rx=0x7fab14009fc0 tx=0x7fab00005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea3bac383e643296 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:08.661 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.664+0000 7fab1aa00640 1 -- 172.21.15.96:0/1334536169 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab14009c10 con 0x7fab00004d40 2024-05-28T06:24:08.661 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.664+0000 7fab1aa00640 1 -- 172.21.15.96:0/1334536169 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab14036070 con 0x7fab00004d40 2024-05-28T06:24:08.661 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.664+0000 7fab1aa00640 1 -- 172.21.15.96:0/1334536169 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab1402d070 con 0x7fab00004d40 2024-05-28T06:24:08.661 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.665+0000 7fab22800640 1 -- 172.21.15.96:0/1334536169 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fab00004d40 msgr2=0x7fab000051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:08.661 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.665+0000 7fab22800640 1 --2- 172.21.15.96:0/1334536169 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fab00004d40 0x7fab000051c0 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 crypto rx=0x7fab14009fc0 tx=0x7fab00005830 comp rx=0 tx=0).stop 2024-05-28T06:24:08.661 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.665+0000 7fab1be00640 1 -- 172.21.15.96:0/1334536169 reap_dead start 2024-05-28T06:24:08.661 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.665+0000 7fab22800640 1 -- 172.21.15.96:0/1334536169 shutdown_connections 2024-05-28T06:24:08.661 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.665+0000 7fab22800640 1 -- 172.21.15.96:0/1334536169 >> 172.21.15.96:0/1334536169 conn(0x7fab1c06bae0 msgr2=0x7fab1c06bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:08.661 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.665+0000 7fab22800640 1 -- 172.21.15.96:0/1334536169 shutdown_connections 2024-05-28T06:24:08.661 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.665+0000 7fab22800640 1 -- 172.21.15.96:0/1334536169 wait complete. 2024-05-28T06:24:08.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.667+0000 7fab22800640 1 Processor -- start 2024-05-28T06:24:08.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.667+0000 7fab22800640 1 -- start start 2024-05-28T06:24:08.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.668+0000 7fab22800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fab00004d40 0x7fab1c15cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:08.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.668+0000 7fab22800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fab1c1f4450 0x7fab1c1610b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:08.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.668+0000 7fab22800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fab1c15c140 0x7fab1c15c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:08.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.668+0000 7fab22800640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fab1c0cffe0 con 0x7fab00004d40 2024-05-28T06:24:08.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.668+0000 7fab22800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fab1c0cf580 con 0x7fab1c1f4450 2024-05-28T06:24:08.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.668+0000 7fab22800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fab1c0d0930 con 0x7fab1c15c140 2024-05-28T06:24:08.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.668+0000 7fab1b400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fab00004d40 0x7fab1c15cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:08.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.668+0000 7fab1b400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fab00004d40 0x7fab1c15cf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:43248/0 (socket says 172.21.15.96:43248) 2024-05-28T06:24:08.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.668+0000 7fab1b400640 1 -- 172.21.15.96:0/3059172879 learned_addr learned my addr 172.21.15.96:0/3059172879 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:24:08.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.669+0000 7fab13e00640 1 --2- 172.21.15.96:0/3059172879 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fab1c1f4450 0x7fab1c1610b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:08.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.669+0000 7fab1b400640 1 -- 172.21.15.96:0/3059172879 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fab1c15c140 msgr2=0x7fab1c15c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:08.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.669+0000 7fab1b400640 1 --2- 172.21.15.96:0/3059172879 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fab1c15c140 0x7fab1c15c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:08.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.669+0000 7fab1b400640 1 -- 172.21.15.96:0/3059172879 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fab1c1f4450 msgr2=0x7fab1c1610b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:08.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.669+0000 7fab1b400640 1 --2- 172.21.15.96:0/3059172879 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fab1c1f4450 0x7fab1c1610b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:08.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.669+0000 7fab1b400640 1 -- 172.21.15.96:0/3059172879 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab1c1f4c20 con 0x7fab00004d40 2024-05-28T06:24:08.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.669+0000 7fab1b400640 1 --2- 172.21.15.96:0/3059172879 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fab00004d40 0x7fab1c15cf50 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7fab0c0116d0 tx=0x7fab0c011b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:08.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.669+0000 7fab13400640 1 -- 172.21.15.96:0/3059172879 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab0c00f940 con 0x7fab00004d40 2024-05-28T06:24:08.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.669+0000 7fab13400640 1 -- 172.21.15.96:0/3059172879 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab0c00fae0 con 0x7fab00004d40 2024-05-28T06:24:08.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.669+0000 7fab13400640 1 -- 172.21.15.96:0/3059172879 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab0c00fe90 con 0x7fab00004d40 2024-05-28T06:24:08.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.669+0000 7fab22800640 1 -- 172.21.15.96:0/3059172879 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab1c1f4f00 con 0x7fab00004d40 2024-05-28T06:24:08.666 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.670+0000 7fab22800640 1 -- 172.21.15.96:0/3059172879 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab1c1f48d0 con 0x7fab00004d40 2024-05-28T06:24:08.666 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.671+0000 7fab22800640 1 -- 172.21.15.96:0/3059172879 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab1c0689e0 con 0x7fab00004d40 2024-05-28T06:24:08.666 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.672+0000 7fab13400640 1 -- 172.21.15.96:0/3059172879 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 42) ==== 93590+0+0 (secure 0 0 0) 0x7fab0c015460 con 0x7fab00004d40 2024-05-28T06:24:08.667 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.673+0000 7fab13400640 1 --2- 172.21.15.96:0/3059172879 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7faaf806e6e0 0x7faaf8070ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:08.667 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.673+0000 7fab13400640 1 -- 172.21.15.96:0/3059172879 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 5879+0+0 (secure 0 0 0) 0x7fab0c08b480 con 0x7fab00004d40 2024-05-28T06:24:08.667 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.673+0000 7fab13e00640 1 --2- 172.21.15.96:0/3059172879 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7faaf806e6e0 0x7faaf8070ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:08.669 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.674+0000 7fab13e00640 1 --2- 172.21.15.96:0/3059172879 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7faaf806e6e0 0x7faaf8070ba0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fab04001eb0 tx=0x7fab04009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:08.671 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.677+0000 7fab13400640 1 -- 172.21.15.96:0/3059172879 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fab0c096080 con 0x7fab00004d40 2024-05-28T06:24:08.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.879+0000 7fab22800640 1 -- 172.21.15.96:0/3059172879 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fab1c15b9f0 con 0x7fab00004d40 2024-05-28T06:24:08.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.881+0000 7fab13400640 1 -- 172.21.15.96:0/3059172879 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v118) ==== 74+0+16936 (secure 0 0 0) 0x7fab0c019030 con 0x7fab00004d40 2024-05-28T06:24:08.877 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:24:08.877 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":118,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:24:01.754615+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-05-28T06:22:14.138685+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"114","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-05-28T06:22:15.394361+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"118","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":118,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.c","mappings":[{"from":0,"to":2}]},{"pgid":"17.17","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":13,"snaps":[{"begin":2,"length":2}]},{"pool":17,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[{"pool":17,"snaps":[{"begin":2,"length":2}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:24:08.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.884+0000 7fab22800640 1 -- 172.21.15.96:0/3059172879 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7faaf806e6e0 msgr2=0x7faaf8070ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:08.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.884+0000 7fab22800640 1 --2- 172.21.15.96:0/3059172879 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7faaf806e6e0 0x7faaf8070ba0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fab04001eb0 tx=0x7fab04009040 comp rx=0 tx=0).stop 2024-05-28T06:24:08.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.884+0000 7fab22800640 1 -- 172.21.15.96:0/3059172879 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fab00004d40 msgr2=0x7fab1c15cf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:08.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.884+0000 7fab22800640 1 --2- 172.21.15.96:0/3059172879 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fab00004d40 0x7fab1c15cf50 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7fab0c0116d0 tx=0x7fab0c011b90 comp rx=0 tx=0).stop 2024-05-28T06:24:08.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.885+0000 7fab22800640 1 -- 172.21.15.96:0/3059172879 shutdown_connections 2024-05-28T06:24:08.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.885+0000 7fab22800640 1 --2- 172.21.15.96:0/3059172879 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7faaf806e6e0 0x7faaf8070ba0 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:08.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.885+0000 7fab22800640 1 --2- 172.21.15.96:0/3059172879 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fab1c15c140 0x7fab1c15c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:08.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.885+0000 7fab22800640 1 --2- 172.21.15.96:0/3059172879 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fab1c1f4450 0x7fab1c1610b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:08.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.885+0000 7fab22800640 1 --2- 172.21.15.96:0/3059172879 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fab00004d40 0x7fab1c15cf50 unknown :-1 s=CLOSED pgs=1943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:08.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.885+0000 7fab22800640 1 -- 172.21.15.96:0/3059172879 >> 172.21.15.96:0/3059172879 conn(0x7fab1c06bae0 msgr2=0x7fab1c06f3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:08.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.885+0000 7fab22800640 1 -- 172.21.15.96:0/3059172879 shutdown_connections 2024-05-28T06:24:08.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:08.885+0000 7fab22800640 1 -- 172.21.15.96:0/3059172879 wait complete. 2024-05-28T06:24:08.900 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-05-28T06:24:08.900 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-28T06:24:09.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.109+0000 7fa01b000640 1 Processor -- start 2024-05-28T06:24:09.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.109+0000 7fa01b000640 1 -- start start 2024-05-28T06:24:09.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.111+0000 7fa01b000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa01415e870 0x7fa0140d0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:09.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.111+0000 7fa01b000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fa0140d14d0 0x7fa0140cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:09.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.111+0000 7fa01b000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fa0140cf870 0x7fa014162360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:09.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.111+0000 7fa01b000640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fa0140d2150 con 0x7fa0140d14d0 2024-05-28T06:24:09.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.111+0000 7fa01b000640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fa0140cfd00 con 0x7fa0140cf870 2024-05-28T06:24:09.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.111+0000 7fa01b000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fa01415ec50 con 0x7fa01415e870 2024-05-28T06:24:09.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.111+0000 7fa019200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fa0140d14d0 0x7fa0140cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:09.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.111+0000 7fa019200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fa0140d14d0 0x7fa0140cf330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:43518/0 (socket says 172.21.15.96:43518) 2024-05-28T06:24:09.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.111+0000 7fa019200640 1 -- 172.21.15.96:0/4046923311 learned_addr learned my addr 172.21.15.96:0/4046923311 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:24:09.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.111+0000 7fa019200640 1 -- 172.21.15.96:0/4046923311 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa01415e870 msgr2=0x7fa0140d0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:09.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.111+0000 7fa019200640 1 --2- 172.21.15.96:0/4046923311 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa01415e870 0x7fa0140d0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:09.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.111+0000 7fa019200640 1 -- 172.21.15.96:0/4046923311 >> v2:172.21.15.123:3300/0 conn(0x7fa0140cf870 msgr2=0x7fa014162360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:09.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.111+0000 7fa019200640 1 --2- 172.21.15.96:0/4046923311 >> v2:172.21.15.123:3300/0 conn(0x7fa0140cf870 0x7fa014162360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:09.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.111+0000 7fa019200640 1 -- 172.21.15.96:0/4046923311 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0140cff80 con 0x7fa0140d14d0 2024-05-28T06:24:09.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.112+0000 7fa019200640 1 --2- 172.21.15.96:0/4046923311 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa0140d14d0 0x7fa0140cf330 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 crypto rx=0x7fa0140b3f60 tx=0x7fa00802a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=109531de23a556d5 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:09.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.112+0000 7fa013e00640 1 -- 172.21.15.96:0/4046923311 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa008036070 con 0x7fa0140d14d0 2024-05-28T06:24:09.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.112+0000 7fa013e00640 1 -- 172.21.15.96:0/4046923311 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa008031600 con 0x7fa0140d14d0 2024-05-28T06:24:09.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.112+0000 7fa013e00640 1 -- 172.21.15.96:0/4046923311 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0080319d0 con 0x7fa0140d14d0 2024-05-28T06:24:09.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.112+0000 7fa01b000640 1 -- 172.21.15.96:0/4046923311 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa0140d14d0 msgr2=0x7fa0140cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:09.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.112+0000 7fa01b000640 1 --2- 172.21.15.96:0/4046923311 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa0140d14d0 0x7fa0140cf330 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 crypto rx=0x7fa0140b3f60 tx=0x7fa00802a6f0 comp rx=0 tx=0).stop 2024-05-28T06:24:09.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.113+0000 7fa01b000640 1 -- 172.21.15.96:0/4046923311 shutdown_connections 2024-05-28T06:24:09.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.113+0000 7fa01b000640 1 --2- 172.21.15.96:0/4046923311 >> v2:172.21.15.123:3300/0 conn(0x7fa0140cf870 0x7fa014162360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:09.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.113+0000 7fa01b000640 1 --2- 172.21.15.96:0/4046923311 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa0140d14d0 0x7fa0140cf330 unknown :-1 s=CLOSED pgs=1944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:09.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.113+0000 7fa01b000640 1 --2- 172.21.15.96:0/4046923311 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa01415e870 0x7fa0140d0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:09.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.113+0000 7fa01b000640 1 -- 172.21.15.96:0/4046923311 >> 172.21.15.96:0/4046923311 conn(0x7fa01408bfc0 msgr2=0x7fa0140d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:09.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.113+0000 7fa01b000640 1 -- 172.21.15.96:0/4046923311 shutdown_connections 2024-05-28T06:24:09.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.113+0000 7fa01b000640 1 -- 172.21.15.96:0/4046923311 wait complete. 2024-05-28T06:24:09.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.114+0000 7fa01b000640 1 Processor -- start 2024-05-28T06:24:09.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.114+0000 7fa01b000640 1 -- start start 2024-05-28T06:24:09.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.115+0000 7fa01b000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa0140cf870 0x7fa0140cd350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:09.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.115+0000 7fa01b000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa0140d14d0 0x7fa0141619c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:09.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.115+0000 7fa01b000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa01415e870 0x7fa0140b7b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:09.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.115+0000 7fa01b000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa0140d0820 con 0x7fa0140cf870 2024-05-28T06:24:09.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.115+0000 7fa01b000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa0140d0540 con 0x7fa01415e870 2024-05-28T06:24:09.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.115+0000 7fa01b000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fa0140d4fb0 con 0x7fa0140d14d0 2024-05-28T06:24:09.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.116+0000 7fa019c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa0140cf870 0x7fa0140cd350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:09.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.116+0000 7fa019c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa0140cf870 0x7fa0140cd350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:43520/0 (socket says 172.21.15.96:43520) 2024-05-28T06:24:09.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.116+0000 7fa019c00640 1 -- 172.21.15.96:0/3807732739 learned_addr learned my addr 172.21.15.96:0/3807732739 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:24:09.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.116+0000 7fa019200640 1 --2- 172.21.15.96:0/3807732739 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa0140d14d0 0x7fa0141619c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:09.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.116+0000 7fa01a600640 1 --2- 172.21.15.96:0/3807732739 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa01415e870 0x7fa0140b7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:09.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.116+0000 7fa019c00640 1 -- 172.21.15.96:0/3807732739 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa0140d14d0 msgr2=0x7fa0141619c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:09.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.116+0000 7fa019c00640 1 --2- 172.21.15.96:0/3807732739 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa0140d14d0 0x7fa0141619c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:09.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.116+0000 7fa019c00640 1 -- 172.21.15.96:0/3807732739 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa01415e870 msgr2=0x7fa0140b7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:09.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.116+0000 7fa019c00640 1 --2- 172.21.15.96:0/3807732739 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa01415e870 0x7fa0140b7b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:09.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.116+0000 7fa019c00640 1 -- 172.21.15.96:0/3807732739 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa014161f00 con 0x7fa0140cf870 2024-05-28T06:24:09.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.116+0000 7fa019c00640 1 --2- 172.21.15.96:0/3807732739 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa0140cf870 0x7fa0140cd350 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7fa00400b4a0 tx=0x7fa00400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:09.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.117+0000 7fa011600640 1 -- 172.21.15.96:0/3807732739 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa004014070 con 0x7fa0140cf870 2024-05-28T06:24:09.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.117+0000 7fa011600640 1 -- 172.21.15.96:0/3807732739 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa004010610 con 0x7fa0140cf870 2024-05-28T06:24:09.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.117+0000 7fa011600640 1 -- 172.21.15.96:0/3807732739 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa004010a00 con 0x7fa0140cf870 2024-05-28T06:24:09.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.117+0000 7fa01b000640 1 -- 172.21.15.96:0/3807732739 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0140cd930 con 0x7fa0140cf870 2024-05-28T06:24:09.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.117+0000 7fa01b000640 1 -- 172.21.15.96:0/3807732739 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0140cdef0 con 0x7fa0140cf870 2024-05-28T06:24:09.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.118+0000 7fa01b000640 1 -- 172.21.15.96:0/3807732739 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0140689e0 con 0x7fa0140cf870 2024-05-28T06:24:09.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.119+0000 7fa011600640 1 -- 172.21.15.96:0/3807732739 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 42) ==== 93590+0+0 (secure 0 0 0) 0x7fa004017070 con 0x7fa0140cf870 2024-05-28T06:24:09.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.120+0000 7fa011600640 1 --2- 172.21.15.96:0/3807732739 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9ff806e510 0x7f9ff80709d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:09.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.120+0000 7fa011600640 1 -- 172.21.15.96:0/3807732739 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 5879+0+0 (secure 0 0 0) 0x7fa00408a570 con 0x7fa0140cf870 2024-05-28T06:24:09.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.121+0000 7fa019200640 1 --2- 172.21.15.96:0/3807732739 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9ff806e510 0x7f9ff80709d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:09.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.122+0000 7fa019200640 1 --2- 172.21.15.96:0/3807732739 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9ff806e510 0x7f9ff80709d0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fa008033000 tx=0x7fa008009000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:09.119 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.125+0000 7fa011600640 1 -- 172.21.15.96:0/3807732739 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa00405a400 con 0x7fa0140cf870 2024-05-28T06:24:09.358 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.363+0000 7fa01b000640 1 -- 172.21.15.96:0/3807732739 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa0140d2f10 con 0x7fa0140cf870 2024-05-28T06:24:09.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.365+0000 7fa011600640 1 -- 172.21.15.96:0/3807732739 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) ==== 94+0+5189 (secure 0 0 0) 0x7fa004059d00 con 0x7fa0140cf870 2024-05-28T06:24:09.359 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:24:09.359 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":73,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6724,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/2148532848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6835","nonce":2148532848},{"type":"v1","addr":"172.21.15.123:6837","nonce":2148532848}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":9252,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6834/2173534467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2173534467},{"type":"v1","addr":"172.21.15.96:6834","nonce":2173534467}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":9255,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/2552082898","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6833","nonce":2552082898},{"type":"v1","addr":"172.21.15.96:6835","nonce":2552082898}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-28T06:22:16.424757+0000","modified":"2024-05-28T06:22:19.524390+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"b","rank":0,"incarnation":68,"state":"up:active","state_seq":3,"addr":"172.21.15.123:6836/3192559785","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":3192559785},{"type":"v1","addr":"172.21.15.123:6836","nonce":3192559785}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9270,"qdb_cluster":[9270]},"id":8}]} 2024-05-28T06:24:09.360 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 73 2024-05-28T06:24:09.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.367+0000 7fa01b000640 1 -- 172.21.15.96:0/3807732739 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9ff806e510 msgr2=0x7f9ff80709d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:09.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.367+0000 7fa01b000640 1 --2- 172.21.15.96:0/3807732739 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9ff806e510 0x7f9ff80709d0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fa008033000 tx=0x7fa008009000 comp rx=0 tx=0).stop 2024-05-28T06:24:09.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.368+0000 7fa01b000640 1 -- 172.21.15.96:0/3807732739 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa0140cf870 msgr2=0x7fa0140cd350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:09.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.368+0000 7fa01b000640 1 --2- 172.21.15.96:0/3807732739 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa0140cf870 0x7fa0140cd350 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7fa00400b4a0 tx=0x7fa00400b960 comp rx=0 tx=0).stop 2024-05-28T06:24:09.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.368+0000 7fa01b000640 1 -- 172.21.15.96:0/3807732739 shutdown_connections 2024-05-28T06:24:09.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.368+0000 7fa01b000640 1 --2- 172.21.15.96:0/3807732739 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9ff806e510 0x7f9ff80709d0 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:09.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.368+0000 7fa01b000640 1 --2- 172.21.15.96:0/3807732739 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa01415e870 0x7fa0140b7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:09.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.368+0000 7fa01b000640 1 --2- 172.21.15.96:0/3807732739 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa0140d14d0 0x7fa0141619c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:09.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.368+0000 7fa01b000640 1 --2- 172.21.15.96:0/3807732739 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa0140cf870 0x7fa0140cd350 unknown :-1 s=CLOSED pgs=1945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:09.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.368+0000 7fa01b000640 1 -- 172.21.15.96:0/3807732739 >> 172.21.15.96:0/3807732739 conn(0x7fa01408bfc0 msgr2=0x7fa014159280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:09.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.368+0000 7fa01b000640 1 -- 172.21.15.96:0/3807732739 shutdown_connections 2024-05-28T06:24:09.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.368+0000 7fa01b000640 1 -- 172.21.15.96:0/3807732739 wait complete. 2024-05-28T06:24:09.380 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-28T06:24:09.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.587+0000 7fac85200640 1 Processor -- start 2024-05-28T06:24:09.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.587+0000 7fac85200640 1 -- start start 2024-05-28T06:24:09.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.588+0000 7fac85200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fac80156b30 0x7fac800af6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:09.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.589+0000 7fac85200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fac8006ee40 0x7fac800ad230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:09.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.589+0000 7fac85200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fac8006f2c0 0x7fac800ad770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:09.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.589+0000 7fac85200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fac80156f10 con 0x7fac80156b30 2024-05-28T06:24:09.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.589+0000 7fac85200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fac8006dda0 con 0x7fac8006f2c0 2024-05-28T06:24:09.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.589+0000 7fac85200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fac8006e080 con 0x7fac8006ee40 2024-05-28T06:24:09.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.589+0000 7fac7f400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fac80156b30 0x7fac800af6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:09.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.589+0000 7fac7f400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fac80156b30 0x7fac800af6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:43528/0 (socket says 172.21.15.96:43528) 2024-05-28T06:24:09.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.589+0000 7fac7f400640 1 -- 172.21.15.96:0/1220651278 learned_addr learned my addr 172.21.15.96:0/1220651278 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:24:09.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.589+0000 7fac7f400640 1 -- 172.21.15.96:0/1220651278 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fac8006ee40 msgr2=0x7fac800ad230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:09.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.589+0000 7fac7f400640 1 --2- 172.21.15.96:0/1220651278 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fac8006ee40 0x7fac800ad230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:09.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.589+0000 7fac7f400640 1 -- 172.21.15.96:0/1220651278 >> v2:172.21.15.123:3300/0 conn(0x7fac8006f2c0 msgr2=0x7fac800ad770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:09.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.589+0000 7fac7fe00640 1 --2- 172.21.15.96:0/1220651278 >> v2:172.21.15.123:3300/0 conn(0x7fac8006f2c0 0x7fac800ad770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:09.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.589+0000 7fac7f400640 1 --2- 172.21.15.96:0/1220651278 >> v2:172.21.15.123:3300/0 conn(0x7fac8006f2c0 0x7fac800ad770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:09.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.589+0000 7fac7f400640 1 -- 172.21.15.96:0/1220651278 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac8006e360 con 0x7fac80156b30 2024-05-28T06:24:09.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.589+0000 7fac7fe00640 1 --2- 172.21.15.96:0/1220651278 >> v2:172.21.15.123:3300/0 conn(0x7fac8006f2c0 0x7fac800ad770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:24:09.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.590+0000 7fac7f400640 1 --2- 172.21.15.96:0/1220651278 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fac80156b30 0x7fac800af6e0 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 crypto rx=0x7fac80071410 tx=0x7fac7002a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2350c2a488f31893 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:09.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.590+0000 7fac7e000640 1 -- 172.21.15.96:0/1220651278 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac70037070 con 0x7fac80156b30 2024-05-28T06:24:09.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.590+0000 7fac7e000640 1 -- 172.21.15.96:0/1220651278 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac70032440 con 0x7fac80156b30 2024-05-28T06:24:09.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.590+0000 7fac7e000640 1 -- 172.21.15.96:0/1220651278 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac70032810 con 0x7fac80156b30 2024-05-28T06:24:09.586 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.590+0000 7fac85200640 1 -- 172.21.15.96:0/1220651278 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fac80156b30 msgr2=0x7fac800af6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:09.586 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.590+0000 7fac85200640 1 --2- 172.21.15.96:0/1220651278 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fac80156b30 0x7fac800af6e0 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 crypto rx=0x7fac80071410 tx=0x7fac7002a140 comp rx=0 tx=0).stop 2024-05-28T06:24:09.586 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.591+0000 7fac85200640 1 -- 172.21.15.96:0/1220651278 shutdown_connections 2024-05-28T06:24:09.586 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.591+0000 7fac85200640 1 --2- 172.21.15.96:0/1220651278 >> v2:172.21.15.123:3300/0 conn(0x7fac8006f2c0 0x7fac800ad770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:09.586 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.591+0000 7fac85200640 1 --2- 172.21.15.96:0/1220651278 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fac8006ee40 0x7fac800ad230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:09.586 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.591+0000 7fac85200640 1 --2- 172.21.15.96:0/1220651278 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fac80156b30 0x7fac800af6e0 unknown :-1 s=CLOSED pgs=1946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:09.586 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.591+0000 7fac85200640 1 -- 172.21.15.96:0/1220651278 >> 172.21.15.96:0/1220651278 conn(0x7fac800b1fc0 msgr2=0x7fac800b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:09.586 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.591+0000 7fac85200640 1 -- 172.21.15.96:0/1220651278 shutdown_connections 2024-05-28T06:24:09.586 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.591+0000 7fac85200640 1 -- 172.21.15.96:0/1220651278 wait complete. 2024-05-28T06:24:09.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.592+0000 7fac85200640 1 Processor -- start 2024-05-28T06:24:09.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.593+0000 7fac85200640 1 -- start start 2024-05-28T06:24:09.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.594+0000 7fac85200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fac8006ee40 0x7fac801552c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:09.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.594+0000 7fac85200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fac8006f2c0 0x7fac80157250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:09.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.594+0000 7fac85200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac80156b30 0x7fac80158c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:09.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.594+0000 7fac85200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fac800afc20 con 0x7fac8006ee40 2024-05-28T06:24:09.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.594+0000 7fac85200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fac800b2c30 con 0x7fac80156b30 2024-05-28T06:24:09.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.594+0000 7fac85200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fac801f55e0 con 0x7fac8006f2c0 2024-05-28T06:24:09.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.594+0000 7fac7f400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fac8006ee40 0x7fac801552c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:09.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.594+0000 7fac7f400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fac8006ee40 0x7fac801552c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:43530/0 (socket says 172.21.15.96:43530) 2024-05-28T06:24:09.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.594+0000 7fac7f400640 1 -- 172.21.15.96:0/3580579248 learned_addr learned my addr 172.21.15.96:0/3580579248 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:24:09.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.594+0000 7fac7ea00640 1 --2- 172.21.15.96:0/3580579248 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fac8006f2c0 0x7fac80157250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:09.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.594+0000 7fac7f400640 1 -- 172.21.15.96:0/3580579248 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fac8006f2c0 msgr2=0x7fac80157250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:09.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.594+0000 7fac7f400640 1 --2- 172.21.15.96:0/3580579248 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fac8006f2c0 0x7fac80157250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:09.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.594+0000 7fac7f400640 1 -- 172.21.15.96:0/3580579248 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac80156b30 msgr2=0x7fac80158c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:09.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.594+0000 7fac7f400640 1 --2- 172.21.15.96:0/3580579248 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac80156b30 0x7fac80158c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:09.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.594+0000 7fac7f400640 1 -- 172.21.15.96:0/3580579248 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac801f58c0 con 0x7fac8006ee40 2024-05-28T06:24:09.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.595+0000 7fac7f400640 1 --2- 172.21.15.96:0/3580579248 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fac8006ee40 0x7fac801552c0 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x7fac7002a180 tx=0x7fac700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:09.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.595+0000 7fac77400640 1 -- 172.21.15.96:0/3580579248 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac70037040 con 0x7fac8006ee40 2024-05-28T06:24:09.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.595+0000 7fac77400640 1 -- 172.21.15.96:0/3580579248 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac70042070 con 0x7fac8006ee40 2024-05-28T06:24:09.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.595+0000 7fac85200640 1 -- 172.21.15.96:0/3580579248 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac801f5ba0 con 0x7fac8006ee40 2024-05-28T06:24:09.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.595+0000 7fac77400640 1 -- 172.21.15.96:0/3580579248 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac70030bb0 con 0x7fac8006ee40 2024-05-28T06:24:09.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.595+0000 7fac85200640 1 -- 172.21.15.96:0/3580579248 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac80157790 con 0x7fac8006ee40 2024-05-28T06:24:09.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.596+0000 7fac85200640 1 -- 172.21.15.96:0/3580579248 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac800689e0 con 0x7fac8006ee40 2024-05-28T06:24:09.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.597+0000 7fac77400640 1 -- 172.21.15.96:0/3580579248 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 42) ==== 93590+0+0 (secure 0 0 0) 0x7fac70047460 con 0x7fac8006ee40 2024-05-28T06:24:09.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.598+0000 7fac77400640 1 --2- 172.21.15.96:0/3580579248 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fac6006e510 0x7fac600709d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:09.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.598+0000 7fac77400640 1 -- 172.21.15.96:0/3580579248 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 5879+0+0 (secure 0 0 0) 0x7fac700ab760 con 0x7fac8006ee40 2024-05-28T06:24:09.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.599+0000 7fac7ea00640 1 --2- 172.21.15.96:0/3580579248 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fac6006e510 0x7fac600709d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:09.594 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.600+0000 7fac7ea00640 1 --2- 172.21.15.96:0/3580579248 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fac6006e510 0x7fac600709d0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fac8015a150 tx=0x7fac6c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:09.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.603+0000 7fac77400640 1 -- 172.21.15.96:0/3580579248 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fac70045360 con 0x7fac8006ee40 2024-05-28T06:24:09.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.843+0000 7fac85200640 1 -- 172.21.15.96:0/3580579248 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fac8006f7f0 con 0x7fac8006ee40 2024-05-28T06:24:09.838 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.844+0000 7fac77400640 1 -- 172.21.15.96:0/3580579248 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) ==== 94+0+5189 (secure 0 0 0) 0x7fac7007b5f0 con 0x7fac8006ee40 2024-05-28T06:24:09.838 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:24:09.838 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":73,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6724,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/2148532848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6835","nonce":2148532848},{"type":"v1","addr":"172.21.15.123:6837","nonce":2148532848}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":9252,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6834/2173534467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2173534467},{"type":"v1","addr":"172.21.15.96:6834","nonce":2173534467}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":9255,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/2552082898","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6833","nonce":2552082898},{"type":"v1","addr":"172.21.15.96:6835","nonce":2552082898}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-28T06:22:16.424757+0000","modified":"2024-05-28T06:22:19.524390+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"b","rank":0,"incarnation":68,"state":"up:active","state_seq":3,"addr":"172.21.15.123:6836/3192559785","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":3192559785},{"type":"v1","addr":"172.21.15.123:6836","nonce":3192559785}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9270,"qdb_cluster":[9270]},"id":8}]} 2024-05-28T06:24:09.838 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 73 2024-05-28T06:24:09.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.847+0000 7fac85200640 1 -- 172.21.15.96:0/3580579248 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fac6006e510 msgr2=0x7fac600709d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:09.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.847+0000 7fac85200640 1 --2- 172.21.15.96:0/3580579248 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fac6006e510 0x7fac600709d0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fac8015a150 tx=0x7fac6c008040 comp rx=0 tx=0).stop 2024-05-28T06:24:09.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.847+0000 7fac85200640 1 -- 172.21.15.96:0/3580579248 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fac8006ee40 msgr2=0x7fac801552c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:09.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.847+0000 7fac85200640 1 --2- 172.21.15.96:0/3580579248 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fac8006ee40 0x7fac801552c0 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x7fac7002a180 tx=0x7fac700046c0 comp rx=0 tx=0).stop 2024-05-28T06:24:09.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.847+0000 7fac85200640 1 -- 172.21.15.96:0/3580579248 shutdown_connections 2024-05-28T06:24:09.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.847+0000 7fac85200640 1 --2- 172.21.15.96:0/3580579248 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fac6006e510 0x7fac600709d0 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:09.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.847+0000 7fac85200640 1 --2- 172.21.15.96:0/3580579248 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac80156b30 0x7fac80158c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:09.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.847+0000 7fac85200640 1 --2- 172.21.15.96:0/3580579248 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fac8006f2c0 0x7fac80157250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:09.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.847+0000 7fac85200640 1 --2- 172.21.15.96:0/3580579248 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fac8006ee40 0x7fac801552c0 unknown :-1 s=CLOSED pgs=1947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:09.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.847+0000 7fac85200640 1 -- 172.21.15.96:0/3580579248 >> 172.21.15.96:0/3580579248 conn(0x7fac800b1fc0 msgr2=0x7fac800af060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:09.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.847+0000 7fac85200640 1 -- 172.21.15.96:0/3580579248 shutdown_connections 2024-05-28T06:24:09.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:09.847+0000 7fac85200640 1 -- 172.21.15.96:0/3580579248 wait complete. 2024-05-28T06:24:09.860 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:24:10.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.063+0000 7f14efe00640 1 Processor -- start 2024-05-28T06:24:10.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.063+0000 7f14efe00640 1 -- start start 2024-05-28T06:24:10.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.064+0000 7f14efe00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f14f00b3750 0x7f14f00adcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:10.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.064+0000 7f14efe00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14f015e4f0 con 0x7f14f00b32c0 2024-05-28T06:24:10.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.065+0000 7f14efe00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14f00ae210 con 0x7f14f015e110 2024-05-28T06:24:10.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.065+0000 7f14efe00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f14f00b3d40 con 0x7f14f00b3750 2024-05-28T06:24:10.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.065+0000 7f14ee000640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f14f00b32c0 0x7f14f00ad5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:57516/0 (socket says 172.21.15.96:57516) 2024-05-28T06:24:10.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.065+0000 7f14ee000640 1 -- 172.21.15.96:0/985864941 learned_addr learned my addr 172.21.15.96:0/985864941 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:24:10.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.065+0000 7f14ef400640 1 --2- 172.21.15.96:0/985864941 >> v2:172.21.15.96:3300/0 conn(0x7f14f00b3750 0x7f14f00adcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:10.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.065+0000 7f14ef400640 1 -- 172.21.15.96:0/985864941 >> v1:172.21.15.123:6789/0 conn(0x7f14f015e110 legacy=0x7f14f00b4de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:10.059 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.065+0000 7f14ef400640 1 -- 172.21.15.96:0/985864941 >> v1:172.21.15.96:6789/0 conn(0x7f14f00b32c0 legacy=0x7f14f00ad5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:10.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.065+0000 7f14ed600640 1 -- 172.21.15.96:0/985864941 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1854814357 0 0) 0x7f14f015e4f0 con 0x7f14f00b32c0 2024-05-28T06:24:10.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.065+0000 7f14ef400640 1 -- 172.21.15.96:0/985864941 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14f00b4020 con 0x7f14f00b3750 2024-05-28T06:24:10.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.066+0000 7f14ef400640 1 --2- 172.21.15.96:0/985864941 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f14f00b3750 0x7f14f00adcd0 secure :-1 s=READY pgs=1949 cs=0 l=1 rev1=1 crypto rx=0x7f14e8009dc0 tx=0x7f14e802a6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b92b48fc2cdf199e server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:10.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.067+0000 7f14ed600640 1 -- 172.21.15.96:0/985864941 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14e8007740 con 0x7f14f00b3750 2024-05-28T06:24:10.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.067+0000 7f14ed600640 1 -- 172.21.15.96:0/985864941 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14e80078e0 con 0x7f14f00b3750 2024-05-28T06:24:10.061 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.067+0000 7f14ed600640 1 -- 172.21.15.96:0/985864941 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14e8007cb0 con 0x7f14f00b3750 2024-05-28T06:24:10.061 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.067+0000 7f14efe00640 1 -- 172.21.15.96:0/985864941 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f14f00b3750 msgr2=0x7f14f00adcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:10.061 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.067+0000 7f14efe00640 1 --2- 172.21.15.96:0/985864941 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f14f00b3750 0x7f14f00adcd0 secure :-1 s=READY pgs=1949 cs=0 l=1 rev1=1 crypto rx=0x7f14e8009dc0 tx=0x7f14e802a6d0 comp rx=0 tx=0).stop 2024-05-28T06:24:10.061 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.067+0000 7f14efe00640 1 -- 172.21.15.96:0/985864941 shutdown_connections 2024-05-28T06:24:10.061 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.067+0000 7f14efe00640 1 --2- 172.21.15.96:0/985864941 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f14f00b3750 0x7f14f00adcd0 unknown :-1 s=CLOSED pgs=1949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:10.062 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.067+0000 7f14efe00640 1 -- 172.21.15.96:0/985864941 >> 172.21.15.96:0/985864941 conn(0x7f14f00b1fc0 msgr2=0x7f14f00b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:10.062 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.068+0000 7f14efe00640 1 -- 172.21.15.96:0/985864941 shutdown_connections 2024-05-28T06:24:10.062 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.068+0000 7f14efe00640 1 -- 172.21.15.96:0/985864941 wait complete. 2024-05-28T06:24:10.062 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.069+0000 7f14efe00640 1 Processor -- start 2024-05-28T06:24:10.063 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.069+0000 7f14efe00640 1 -- start start 2024-05-28T06:24:10.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.070+0000 7f14efe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f14f00b32c0 0x7f14f0099910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:10.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.070+0000 7f14efe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f14f00b3750 0x7f14f00b0220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:10.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.070+0000 7f14efe00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f14f015e110 0x7f14f0151920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:10.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.070+0000 7f14efe00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f14f00b48c0 con 0x7f14f00b32c0 2024-05-28T06:24:10.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.070+0000 7f14efe00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f14f00b45e0 con 0x7f14f00b3750 2024-05-28T06:24:10.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.070+0000 7f14efe00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f14f006f4d0 con 0x7f14f015e110 2024-05-28T06:24:10.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.070+0000 7f14eea00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f14f00b32c0 0x7f14f0099910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:10.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.071+0000 7f14eea00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f14f00b32c0 0x7f14f0099910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:43556/0 (socket says 172.21.15.96:43556) 2024-05-28T06:24:10.064 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.071+0000 7f14eea00640 1 -- 172.21.15.96:0/4024130487 learned_addr learned my addr 172.21.15.96:0/4024130487 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:24:10.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.071+0000 7f14ef400640 1 --2- 172.21.15.96:0/4024130487 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f14f015e110 0x7f14f0151920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:10.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.071+0000 7f14eea00640 1 -- 172.21.15.96:0/4024130487 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f14f015e110 msgr2=0x7f14f0151920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:10.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.071+0000 7f14eea00640 1 --2- 172.21.15.96:0/4024130487 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f14f015e110 0x7f14f0151920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:10.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.071+0000 7f14eea00640 1 -- 172.21.15.96:0/4024130487 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f14f00b3750 msgr2=0x7f14f00b0220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:10.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.071+0000 7f14eea00640 1 --2- 172.21.15.96:0/4024130487 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f14f00b3750 0x7f14f00b0220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:10.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.071+0000 7f14eea00640 1 -- 172.21.15.96:0/4024130487 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14f0154730 con 0x7f14f00b32c0 2024-05-28T06:24:10.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.071+0000 7f14eea00640 1 --2- 172.21.15.96:0/4024130487 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f14f00b32c0 0x7f14f0099910 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 crypto rx=0x7f14e000b3f0 tx=0x7f14e000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:10.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.071+0000 7f14e6a00640 1 -- 172.21.15.96:0/4024130487 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14e0015020 con 0x7f14f00b32c0 2024-05-28T06:24:10.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.071+0000 7f14e6a00640 1 -- 172.21.15.96:0/4024130487 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14e0011830 con 0x7f14f00b32c0 2024-05-28T06:24:10.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.071+0000 7f14e6a00640 1 -- 172.21.15.96:0/4024130487 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14e0011c20 con 0x7f14f00b32c0 2024-05-28T06:24:10.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.071+0000 7f14efe00640 1 -- 172.21.15.96:0/4024130487 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14f00b0b00 con 0x7f14f00b32c0 2024-05-28T06:24:10.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.072+0000 7f14efe00640 1 -- 172.21.15.96:0/4024130487 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14f01549a0 con 0x7f14f00b32c0 2024-05-28T06:24:10.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.073+0000 7f14efe00640 1 -- 172.21.15.96:0/4024130487 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14f0099eb0 con 0x7f14f00b32c0 2024-05-28T06:24:10.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.074+0000 7f14e6a00640 1 -- 172.21.15.96:0/4024130487 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 42) ==== 93590+0+0 (secure 0 0 0) 0x7f14e0018020 con 0x7f14f00b32c0 2024-05-28T06:24:10.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.075+0000 7f14e6a00640 1 --2- 172.21.15.96:0/4024130487 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f14d006e510 0x7f14d00709d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:10.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.075+0000 7f14e6a00640 1 -- 172.21.15.96:0/4024130487 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 5879+0+0 (secure 0 0 0) 0x7f14e008a650 con 0x7f14f00b32c0 2024-05-28T06:24:10.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.076+0000 7f14ee000640 1 --2- 172.21.15.96:0/4024130487 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f14d006e510 0x7f14d00709d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:10.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.077+0000 7f14ee000640 1 --2- 172.21.15.96:0/4024130487 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f14d006e510 0x7f14d00709d0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f14dc007fd0 tx=0x7f14dc007920 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:10.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.080+0000 7f14e6a00640 1 -- 172.21.15.96:0/4024130487 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f14e001f030 con 0x7f14f00b32c0 2024-05-28T06:24:10.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.286+0000 7f14efe00640 1 -- 172.21.15.96:0/4024130487 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f14f00b2e90 con 0x7f14f00b32c0 2024-05-28T06:24:10.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.288+0000 7f14e6a00640 1 -- 172.21.15.96:0/4024130487 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v118) ==== 74+0+16936 (secure 0 0 0) 0x7f14e00172c0 con 0x7f14f00b32c0 2024-05-28T06:24:10.342 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:24:10.343 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":118,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:24:01.754615+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-05-28T06:22:14.138685+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"114","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-05-28T06:22:15.394361+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"118","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":118,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.c","mappings":[{"from":0,"to":2}]},{"pgid":"17.17","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":13,"snaps":[{"begin":2,"length":2}]},{"pool":17,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[{"pool":17,"snaps":[{"begin":2,"length":2}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:24:10.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.291+0000 7f14efe00640 1 -- 172.21.15.96:0/4024130487 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f14d006e510 msgr2=0x7f14d00709d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:10.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.291+0000 7f14efe00640 1 --2- 172.21.15.96:0/4024130487 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f14d006e510 0x7f14d00709d0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f14dc007fd0 tx=0x7f14dc007920 comp rx=0 tx=0).stop 2024-05-28T06:24:10.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.291+0000 7f14efe00640 1 -- 172.21.15.96:0/4024130487 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f14f00b32c0 msgr2=0x7f14f0099910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:10.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.291+0000 7f14efe00640 1 --2- 172.21.15.96:0/4024130487 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f14f00b32c0 0x7f14f0099910 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 crypto rx=0x7f14e000b3f0 tx=0x7f14e000b8b0 comp rx=0 tx=0).stop 2024-05-28T06:24:10.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.291+0000 7f14efe00640 1 -- 172.21.15.96:0/4024130487 shutdown_connections 2024-05-28T06:24:10.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.291+0000 7f14efe00640 1 --2- 172.21.15.96:0/4024130487 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f14d006e510 0x7f14d00709d0 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:10.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.291+0000 7f14efe00640 1 --2- 172.21.15.96:0/4024130487 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f14f015e110 0x7f14f0151920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:10.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.291+0000 7f14efe00640 1 --2- 172.21.15.96:0/4024130487 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f14f00b3750 0x7f14f00b0220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:10.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.291+0000 7f14efe00640 1 --2- 172.21.15.96:0/4024130487 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f14f00b32c0 0x7f14f0099910 unknown :-1 s=CLOSED pgs=1950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:10.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.291+0000 7f14efe00640 1 -- 172.21.15.96:0/4024130487 >> 172.21.15.96:0/4024130487 conn(0x7f14f00b1fc0 msgr2=0x7f14f00ad580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:10.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.292+0000 7f14efe00640 1 -- 172.21.15.96:0/4024130487 shutdown_connections 2024-05-28T06:24:10.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.292+0000 7f14efe00640 1 -- 172.21.15.96:0/4024130487 wait complete. 2024-05-28T06:24:10.346 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-05-28T06:24:10.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.560+0000 7f017b600640 1 Processor -- start 2024-05-28T06:24:10.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.561+0000 7f017b600640 1 -- start start 2024-05-28T06:24:10.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.562+0000 7f017b600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f017406f860 0x7f01740b3150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:10.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.562+0000 7f017b600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f01740b3690 0x7f01740d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:10.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.562+0000 7f017b600640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f017406ed10 con 0x7f01740d09e0 2024-05-28T06:24:10.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.562+0000 7f017b600640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f01740d0e50 con 0x7f017406f860 2024-05-28T06:24:10.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.562+0000 7f017b600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f017415cb00 con 0x7f01740b3690 2024-05-28T06:24:10.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.562+0000 7f0179600640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f01740d09e0 0x7f0174160210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:57518/0 (socket says 172.21.15.96:57518) 2024-05-28T06:24:10.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.562+0000 7f0179600640 1 -- 172.21.15.96:0/317018922 learned_addr learned my addr 172.21.15.96:0/317018922 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:24:10.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.563+0000 7f0173e00640 1 --2- 172.21.15.96:0/317018922 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f01740b3690 0x7f01740d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:10.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.563+0000 7f0178c00640 1 --2- 172.21.15.96:0/317018922 >> v2:172.21.15.123:3300/0 conn(0x7f017406f860 0x7f01740b3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:10.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.563+0000 7f0173400640 1 -- 172.21.15.96:0/317018922 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3806749259 0 0) 0x7f016c001490 con 0x7f01740d09e0 2024-05-28T06:24:10.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.563+0000 7f0173400640 1 -- 172.21.15.96:0/317018922 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0154003520 con 0x7f01740d09e0 2024-05-28T06:24:10.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.563+0000 7f0173400640 1 -- 172.21.15.96:0/317018922 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 451765897 0 0) 0x7f0154003520 con 0x7f01740d09e0 2024-05-28T06:24:10.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.563+0000 7f0173400640 1 -- 172.21.15.96:0/317018922 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f01740b3690 msgr2=0x7f01740d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:10.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.563+0000 7f0173400640 1 --2- 172.21.15.96:0/317018922 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f01740b3690 0x7f01740d04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:10.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.563+0000 7f0173400640 1 -- 172.21.15.96:0/317018922 >> v2:172.21.15.123:3300/0 conn(0x7f017406f860 msgr2=0x7f01740b3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:10.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.563+0000 7f0173400640 1 --2- 172.21.15.96:0/317018922 >> v2:172.21.15.123:3300/0 conn(0x7f017406f860 0x7f01740b3150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:10.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.563+0000 7f0173400640 1 -- 172.21.15.96:0/317018922 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01740b3cc0 con 0x7f01740d09e0 2024-05-28T06:24:10.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.563+0000 7f0173400640 1 -- 172.21.15.96:0/317018922 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f016c001bc0 con 0x7f01740d09e0 2024-05-28T06:24:10.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.563+0000 7f0173400640 1 -- 172.21.15.96:0/317018922 >> v1:172.21.15.96:6789/0 conn(0x7f01740d09e0 legacy=0x7f0174160210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:10.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.563+0000 7f0173400640 1 --2- 172.21.15.96:0/317018922 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01540039b0 0x7f0154003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:10.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.563+0000 7f0173400640 1 --2- 172.21.15.96:0/317018922 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0154004800 0x7f0154004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:10.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.563+0000 7f0173400640 1 --2- 172.21.15.96:0/317018922 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f01740b3690 0x7f01540051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:10.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.563+0000 7f0173400640 1 -- 172.21.15.96:0/317018922 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f01740d0e50 con 0x7f0154004800 2024-05-28T06:24:10.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.563+0000 7f0173400640 1 -- 172.21.15.96:0/317018922 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f017415cb00 con 0x7f01540039b0 2024-05-28T06:24:10.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.563+0000 7f0173400640 1 -- 172.21.15.96:0/317018922 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f015400b9c0 con 0x7f01740b3690 2024-05-28T06:24:10.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.564+0000 7f0173e00640 1 --2- 172.21.15.96:0/317018922 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0154004800 0x7f0154004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:10.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.564+0000 7f0179600640 1 --2- 172.21.15.96:0/317018922 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f01740b3690 0x7f01540051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:10.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.564+0000 7f0178c00640 1 --2- 172.21.15.96:0/317018922 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01540039b0 0x7f0154003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:10.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.564+0000 7f0173e00640 1 -- 172.21.15.96:0/317018922 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f01740b3690 msgr2=0x7f01540051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:10.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.564+0000 7f0173e00640 1 --2- 172.21.15.96:0/317018922 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f01740b3690 0x7f01540051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:10.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.564+0000 7f0173e00640 1 -- 172.21.15.96:0/317018922 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01540039b0 msgr2=0x7f0154003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:10.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.564+0000 7f0173e00640 1 --2- 172.21.15.96:0/317018922 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01540039b0 0x7f0154003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:10.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.564+0000 7f0173e00640 1 -- 172.21.15.96:0/317018922 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0154003520 con 0x7f0154004800 2024-05-28T06:24:10.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.564+0000 7f0178c00640 1 --2- 172.21.15.96:0/317018922 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01540039b0 0x7f0154003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:24:10.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.565+0000 7f0173e00640 1 --2- 172.21.15.96:0/317018922 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0154004800 0x7f0154004c80 secure :-1 s=READY pgs=1952 cs=0 l=1 rev1=1 crypto rx=0x7f015c009ac0 tx=0x7f0154005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=664fe27cf3ff2053 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:10.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.565+0000 7f0173400640 1 -- 172.21.15.96:0/317018922 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f015c036070 con 0x7f0154004800 2024-05-28T06:24:10.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.565+0000 7f0173400640 1 -- 172.21.15.96:0/317018922 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f015c031400 con 0x7f0154004800 2024-05-28T06:24:10.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.565+0000 7f0173400640 1 -- 172.21.15.96:0/317018922 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f015c009690 con 0x7f0154004800 2024-05-28T06:24:10.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.566+0000 7f017b600640 1 -- 172.21.15.96:0/317018922 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0154004800 msgr2=0x7f0154004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:10.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.566+0000 7f017b600640 1 --2- 172.21.15.96:0/317018922 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0154004800 0x7f0154004c80 secure :-1 s=READY pgs=1952 cs=0 l=1 rev1=1 crypto rx=0x7f015c009ac0 tx=0x7f0154005830 comp rx=0 tx=0).stop 2024-05-28T06:24:10.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.566+0000 7f0179600640 1 -- 172.21.15.96:0/317018922 reap_dead start 2024-05-28T06:24:10.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.566+0000 7f017b600640 1 -- 172.21.15.96:0/317018922 shutdown_connections 2024-05-28T06:24:10.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.566+0000 7f017b600640 1 -- 172.21.15.96:0/317018922 >> 172.21.15.96:0/317018922 conn(0x7f017406bae0 msgr2=0x7f017406bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:10.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.566+0000 7f017b600640 1 -- 172.21.15.96:0/317018922 shutdown_connections 2024-05-28T06:24:10.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.566+0000 7f017b600640 1 -- 172.21.15.96:0/317018922 wait complete. 2024-05-28T06:24:10.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.568+0000 7f017b600640 1 Processor -- start 2024-05-28T06:24:10.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.568+0000 7f017b600640 1 -- start start 2024-05-28T06:24:10.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.569+0000 7f017b600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0154004800 0x7f01740c8760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:10.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.569+0000 7f017b600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f01741f43a0 0x7f01740c7330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:10.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.569+0000 7f017b600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f017415be30 0x7f017415c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:10.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.569+0000 7f017b600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f01740b4560 con 0x7f0154004800 2024-05-28T06:24:10.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.569+0000 7f017b600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f01740b4280 con 0x7f017415be30 2024-05-28T06:24:10.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.569+0000 7f017b600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f01740b4840 con 0x7f01741f43a0 2024-05-28T06:24:10.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.569+0000 7f0178c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0154004800 0x7f01740c8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:10.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.569+0000 7f0178c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0154004800 0x7f01740c8760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:43568/0 (socket says 172.21.15.96:43568) 2024-05-28T06:24:10.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.569+0000 7f0178c00640 1 -- 172.21.15.96:0/2504181069 learned_addr learned my addr 172.21.15.96:0/2504181069 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:24:10.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.570+0000 7f0178c00640 1 -- 172.21.15.96:0/2504181069 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f01741f43a0 msgr2=0x7f01740c7330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:10.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.570+0000 7f0173e00640 1 --2- 172.21.15.96:0/2504181069 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f01741f43a0 0x7f01740c7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:10.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.570+0000 7f0178c00640 1 --2- 172.21.15.96:0/2504181069 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f01741f43a0 0x7f01740c7330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:10.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.570+0000 7f0179600640 1 --2- 172.21.15.96:0/2504181069 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f017415be30 0x7f017415c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:10.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.570+0000 7f0178c00640 1 -- 172.21.15.96:0/2504181069 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f017415be30 msgr2=0x7f017415c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:10.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.570+0000 7f0178c00640 1 --2- 172.21.15.96:0/2504181069 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f017415be30 0x7f017415c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:10.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.570+0000 7f0178c00640 1 -- 172.21.15.96:0/2504181069 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f017415c800 con 0x7f0154004800 2024-05-28T06:24:10.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.570+0000 7f0178c00640 1 --2- 172.21.15.96:0/2504181069 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0154004800 0x7f01740c8760 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 crypto rx=0x7f01640109d0 tx=0x7f0164010e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:10.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.570+0000 7f0170c00640 1 -- 172.21.15.96:0/2504181069 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f016400f040 con 0x7f0154004800 2024-05-28T06:24:10.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.570+0000 7f0170c00640 1 -- 172.21.15.96:0/2504181069 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f016400ebc0 con 0x7f0154004800 2024-05-28T06:24:10.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.570+0000 7f0170c00640 1 -- 172.21.15.96:0/2504181069 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f016401c400 con 0x7f0154004800 2024-05-28T06:24:10.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.570+0000 7f017b600640 1 -- 172.21.15.96:0/2504181069 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01740c7ca0 con 0x7f0154004800 2024-05-28T06:24:10.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.571+0000 7f017b600640 1 -- 172.21.15.96:0/2504181069 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01740c8260 con 0x7f0154004800 2024-05-28T06:24:10.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.572+0000 7f017b600640 1 -- 172.21.15.96:0/2504181069 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01740689e0 con 0x7f0154004800 2024-05-28T06:24:10.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.573+0000 7f0170c00640 1 -- 172.21.15.96:0/2504181069 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 42) ==== 93590+0+0 (secure 0 0 0) 0x7f016401c5a0 con 0x7f0154004800 2024-05-28T06:24:10.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.574+0000 7f0170c00640 1 --2- 172.21.15.96:0/2504181069 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f015006e510 0x7f01500709d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:10.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.574+0000 7f0170c00640 1 -- 172.21.15.96:0/2504181069 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 5879+0+0 (secure 0 0 0) 0x7f016408b380 con 0x7f0154004800 2024-05-28T06:24:10.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.574+0000 7f0173e00640 1 --2- 172.21.15.96:0/2504181069 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f015006e510 0x7f01500709d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:10.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.576+0000 7f0173e00640 1 --2- 172.21.15.96:0/2504181069 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f015006e510 0x7f01500709d0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f015c033000 tx=0x7f015c008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:10.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.579+0000 7f0170c00640 1 -- 172.21.15.96:0/2504181069 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f017415c800 con 0x7f0154004800 2024-05-28T06:24:10.809 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.815+0000 7f017b600640 1 -- 172.21.15.96:0/2504181069 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f017406e2f0 con 0x7f0154004800 2024-05-28T06:24:10.810 INFO:tasks.ceph.mon.a.smithi096.stderr:2024-05-28T06:24:10.816+0000 7fb2fca00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-28T06:24:10.813 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.820+0000 7f0170c00640 1 -- 172.21.15.96:0/2504181069 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v74) ==== 161+0+0 (secure 0 0 0) 0x7f0164018030 con 0x7f0154004800 2024-05-28T06:24:10.814 INFO:teuthology.orchestra.run.smithi096.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-28T06:24:10.816 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.822+0000 7f017b600640 1 -- 172.21.15.96:0/2504181069 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f015006e510 msgr2=0x7f01500709d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:10.816 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.822+0000 7f017b600640 1 --2- 172.21.15.96:0/2504181069 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f015006e510 0x7f01500709d0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f015c033000 tx=0x7f015c008030 comp rx=0 tx=0).stop 2024-05-28T06:24:10.816 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.822+0000 7f017b600640 1 -- 172.21.15.96:0/2504181069 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0154004800 msgr2=0x7f01740c8760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:10.816 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.822+0000 7f017b600640 1 --2- 172.21.15.96:0/2504181069 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0154004800 0x7f01740c8760 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 crypto rx=0x7f01640109d0 tx=0x7f0164010e90 comp rx=0 tx=0).stop 2024-05-28T06:24:10.817 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.823+0000 7f017b600640 1 -- 172.21.15.96:0/2504181069 shutdown_connections 2024-05-28T06:24:10.817 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.823+0000 7f017b600640 1 --2- 172.21.15.96:0/2504181069 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f015006e510 0x7f01500709d0 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:10.817 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.823+0000 7f017b600640 1 --2- 172.21.15.96:0/2504181069 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f017415be30 0x7f017415c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:10.817 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.823+0000 7f017b600640 1 --2- 172.21.15.96:0/2504181069 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f01741f43a0 0x7f01740c7330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:10.818 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.823+0000 7f017b600640 1 --2- 172.21.15.96:0/2504181069 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0154004800 0x7f01740c8760 unknown :-1 s=CLOSED pgs=1953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:10.818 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.823+0000 7f017b600640 1 -- 172.21.15.96:0/2504181069 >> 172.21.15.96:0/2504181069 conn(0x7f017406bae0 msgr2=0x7f017406bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:10.818 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.823+0000 7f017b600640 1 -- 172.21.15.96:0/2504181069 shutdown_connections 2024-05-28T06:24:10.818 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:10.823+0000 7f017b600640 1 -- 172.21.15.96:0/2504181069 wait complete. 2024-05-28T06:24:10.837 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-05-28T06:24:10.955 INFO:tasks.ceph.mds.b.smithi123.stdout:starting mds.b at 2024-05-28T06:24:11.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.036+0000 7fb00da00640 1 Processor -- start 2024-05-28T06:24:11.030 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.036+0000 7fb00da00640 1 -- start start 2024-05-28T06:24:11.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.038+0000 7fb00da00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb0080cfa40 0x7fb008169690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:11.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.038+0000 7fb00da00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb00815ec50 con 0x7fb00815e870 2024-05-28T06:24:11.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.038+0000 7fb00da00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0080cff80 con 0x7fb0080d16a0 2024-05-28T06:24:11.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.038+0000 7fb00da00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fb0080d2150 con 0x7fb0080cfa40 2024-05-28T06:24:11.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.038+0000 7fb007e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fb00815e870 0x7fb0080d0f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:57530/0 (socket says 172.21.15.96:57530) 2024-05-28T06:24:11.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.038+0000 7fb007e00640 1 -- 172.21.15.96:0/1221251784 learned_addr learned my addr 172.21.15.96:0/1221251784 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:24:11.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.039+0000 7fb00d000640 1 --2- 172.21.15.96:0/1221251784 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb0080cfa40 0x7fb008169690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:11.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.039+0000 7fb007400640 1 -- 172.21.15.96:0/1221251784 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 288189296 0 0) 0x7fb00815ec50 con 0x7fb00815e870 2024-05-28T06:24:11.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.039+0000 7fb007400640 1 -- 172.21.15.96:0/1221251784 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fafec003520 con 0x7fb00815e870 2024-05-28T06:24:11.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.039+0000 7fb007400640 1 -- 172.21.15.96:0/1221251784 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2657805975 0 0) 0x7fafec003520 con 0x7fb00815e870 2024-05-28T06:24:11.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.039+0000 7fb007400640 1 -- 172.21.15.96:0/1221251784 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb0080cfa40 msgr2=0x7fb008169690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:11.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.039+0000 7fb007400640 1 --2- 172.21.15.96:0/1221251784 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb0080cfa40 0x7fb008169690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:11.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.039+0000 7fb007400640 1 -- 172.21.15.96:0/1221251784 >> v1:172.21.15.123:6789/0 conn(0x7fb0080d16a0 legacy=0x7fb0080cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:11.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.039+0000 7fb007400640 1 -- 172.21.15.96:0/1221251784 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0080d0260 con 0x7fb00815e870 2024-05-28T06:24:11.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.039+0000 7fb007400640 1 -- 172.21.15.96:0/1221251784 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7faff8001670 con 0x7fb00815e870 2024-05-28T06:24:11.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.039+0000 7fb007400640 1 -- 172.21.15.96:0/1221251784 >> v1:172.21.15.96:6789/0 conn(0x7fb00815e870 legacy=0x7fb0080d0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:11.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.039+0000 7fb007400640 1 --2- 172.21.15.96:0/1221251784 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fafec0039e0 0x7fafec003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:11.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.039+0000 7fb007400640 1 --2- 172.21.15.96:0/1221251784 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb0080cfa40 0x7fafec004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:11.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.039+0000 7fb007400640 1 --2- 172.21.15.96:0/1221251784 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fafec004d40 0x7fafec0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:11.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.039+0000 7fb007400640 1 -- 172.21.15.96:0/1221251784 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb0080d2150 con 0x7fafec0039e0 2024-05-28T06:24:11.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.039+0000 7fb007400640 1 -- 172.21.15.96:0/1221251784 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fafec00bb50 con 0x7fafec004d40 2024-05-28T06:24:11.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.039+0000 7fb007400640 1 -- 172.21.15.96:0/1221251784 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fafec00bd20 con 0x7fb0080cfa40 2024-05-28T06:24:11.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.039+0000 7fb007e00640 1 --2- 172.21.15.96:0/1221251784 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fafec0039e0 0x7fafec003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:11.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.040+0000 7fb007e00640 1 -- 172.21.15.96:0/1221251784 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb0080cfa40 msgr2=0x7fafec004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:11.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.040+0000 7fb007e00640 1 --2- 172.21.15.96:0/1221251784 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb0080cfa40 0x7fafec004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:11.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.040+0000 7fb007e00640 1 -- 172.21.15.96:0/1221251784 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fafec004d40 msgr2=0x7fafec0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:11.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.040+0000 7fb007e00640 1 --2- 172.21.15.96:0/1221251784 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fafec004d40 0x7fafec0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:11.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.040+0000 7fb007e00640 1 -- 172.21.15.96:0/1221251784 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafec003520 con 0x7fafec0039e0 2024-05-28T06:24:11.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.041+0000 7fb007e00640 1 --2- 172.21.15.96:0/1221251784 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fafec0039e0 0x7fafec003e00 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7faff800c240 tx=0x7fafec005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1c074ad3d59fab5 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:11.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.041+0000 7fb007400640 1 -- 172.21.15.96:0/1221251784 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faff8007d30 con 0x7fafec0039e0 2024-05-28T06:24:11.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.041+0000 7fb007400640 1 -- 172.21.15.96:0/1221251784 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faff8005ad0 con 0x7fafec0039e0 2024-05-28T06:24:11.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.041+0000 7fb007400640 1 -- 172.21.15.96:0/1221251784 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faff8007070 con 0x7fafec0039e0 2024-05-28T06:24:11.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.042+0000 7fb00da00640 1 -- 172.21.15.96:0/1221251784 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fafec0039e0 msgr2=0x7fafec003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:11.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.042+0000 7fb00da00640 1 --2- 172.21.15.96:0/1221251784 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fafec0039e0 0x7fafec003e00 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7faff800c240 tx=0x7fafec005830 comp rx=0 tx=0).stop 2024-05-28T06:24:11.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.042+0000 7fb00d000640 1 -- 172.21.15.96:0/1221251784 reap_dead start 2024-05-28T06:24:11.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.042+0000 7fb00da00640 1 -- 172.21.15.96:0/1221251784 shutdown_connections 2024-05-28T06:24:11.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.042+0000 7fb00da00640 1 -- 172.21.15.96:0/1221251784 >> 172.21.15.96:0/1221251784 conn(0x7fb00808bfc0 msgr2=0x7fb0080d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:11.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.042+0000 7fb00da00640 1 -- 172.21.15.96:0/1221251784 shutdown_connections 2024-05-28T06:24:11.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.042+0000 7fb00da00640 1 -- 172.21.15.96:0/1221251784 wait complete. 2024-05-28T06:24:11.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.044+0000 7fb00da00640 1 Processor -- start 2024-05-28T06:24:11.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.044+0000 7fb00da00640 1 -- start start 2024-05-28T06:24:11.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.045+0000 7fb00da00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fafec0039e0 0x7fb0080cd2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:11.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.045+0000 7fb00da00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb0080cb650 0x7fb0080cbad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:11.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.045+0000 7fb00da00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb00815f4e0 0x7fb00815f960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:11.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.045+0000 7fb00da00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb0080d4fb0 con 0x7fb0080cb650 2024-05-28T06:24:11.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.045+0000 7fb00da00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fb0080d0820 con 0x7fafec0039e0 2024-05-28T06:24:11.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.045+0000 7fb00da00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fb0080d35f0 con 0x7fb00815f4e0 2024-05-28T06:24:11.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.045+0000 7fafff400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb0080cb650 0x7fb0080cbad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:11.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.045+0000 7fafff400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb0080cb650 0x7fb0080cbad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:43584/0 (socket says 172.21.15.96:43584) 2024-05-28T06:24:11.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.045+0000 7fafff400640 1 -- 172.21.15.96:0/3624140393 learned_addr learned my addr 172.21.15.96:0/3624140393 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:24:11.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.045+0000 7fb007e00640 1 --2- 172.21.15.96:0/3624140393 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fafec0039e0 0x7fb0080cd2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:11.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.045+0000 7fb00d000640 1 --2- 172.21.15.96:0/3624140393 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb00815f4e0 0x7fb00815f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:11.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.046+0000 7fafff400640 1 -- 172.21.15.96:0/3624140393 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb00815f4e0 msgr2=0x7fb00815f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:11.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.046+0000 7fafff400640 1 --2- 172.21.15.96:0/3624140393 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb00815f4e0 0x7fb00815f960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:11.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.046+0000 7fafff400640 1 -- 172.21.15.96:0/3624140393 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fafec0039e0 msgr2=0x7fb0080cd2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:11.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.046+0000 7fafff400640 1 --2- 172.21.15.96:0/3624140393 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fafec0039e0 0x7fb0080cd2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:11.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.046+0000 7fafff400640 1 -- 172.21.15.96:0/3624140393 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0080b9390 con 0x7fb0080cb650 2024-05-28T06:24:11.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.046+0000 7fb00d000640 1 --2- 172.21.15.96:0/3624140393 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb00815f4e0 0x7fb00815f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:24:11.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.046+0000 7fafff400640 1 --2- 172.21.15.96:0/3624140393 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb0080cb650 0x7fb0080cbad0 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7faff000c500 tx=0x7faff000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:11.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.046+0000 7fb004c00640 1 -- 172.21.15.96:0/3624140393 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faff00052d0 con 0x7fb0080cb650 2024-05-28T06:24:11.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.046+0000 7fb004c00640 1 -- 172.21.15.96:0/3624140393 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faff0005470 con 0x7fb0080cb650 2024-05-28T06:24:11.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.046+0000 7fb00da00640 1 -- 172.21.15.96:0/3624140393 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0080cc230 con 0x7fb0080cb650 2024-05-28T06:24:11.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.046+0000 7fb004c00640 1 -- 172.21.15.96:0/3624140393 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faff0003880 con 0x7fb0080cb650 2024-05-28T06:24:11.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.046+0000 7fb00da00640 1 -- 172.21.15.96:0/3624140393 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0080cc7f0 con 0x7fb0080cb650 2024-05-28T06:24:11.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.048+0000 7fb00da00640 1 -- 172.21.15.96:0/3624140393 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0080689e0 con 0x7fb0080cb650 2024-05-28T06:24:11.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.048+0000 7fb004c00640 1 -- 172.21.15.96:0/3624140393 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 42) ==== 93590+0+0 (secure 0 0 0) 0x7faff0003a20 con 0x7fb0080cb650 2024-05-28T06:24:11.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.049+0000 7fb004c00640 1 --2- 172.21.15.96:0/3624140393 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fafe406e510 0x7fafe40709d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:11.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.049+0000 7fb004c00640 1 -- 172.21.15.96:0/3624140393 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 5937+0+0 (secure 0 0 0) 0x7faff008a800 con 0x7fb0080cb650 2024-05-28T06:24:11.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.050+0000 7fb007e00640 1 --2- 172.21.15.96:0/3624140393 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fafe406e510 0x7fafe40709d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:11.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.051+0000 7fb007e00640 1 --2- 172.21.15.96:0/3624140393 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fafe406e510 0x7fafe40709d0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7faff8033000 tx=0x7faff802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:11.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.054+0000 7fb004c00640 1 -- 172.21.15.96:0/3624140393 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb0080b9390 con 0x7fb0080cb650 2024-05-28T06:24:11.287 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.294+0000 7fb00da00640 1 -- 172.21.15.96:0/3624140393 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fb00806e070 con 0x7fb0080cb650 2024-05-28T06:24:11.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.850+0000 7fb004c00640 1 -- 172.21.15.96:0/3624140393 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v75) ==== 104+0+0 (secure 0 0 0) 0x7faff005a690 con 0x7fb0080cb650 2024-05-28T06:24:11.847 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.853+0000 7fb00da00640 1 -- 172.21.15.96:0/3624140393 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fafe406e510 msgr2=0x7fafe40709d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:11.847 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.853+0000 7fb00da00640 1 --2- 172.21.15.96:0/3624140393 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fafe406e510 0x7fafe40709d0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7faff8033000 tx=0x7faff802e000 comp rx=0 tx=0).stop 2024-05-28T06:24:11.847 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.853+0000 7fb00da00640 1 -- 172.21.15.96:0/3624140393 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb0080cb650 msgr2=0x7fb0080cbad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:11.847 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.853+0000 7fb00da00640 1 --2- 172.21.15.96:0/3624140393 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb0080cb650 0x7fb0080cbad0 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7faff000c500 tx=0x7faff000c9c0 comp rx=0 tx=0).stop 2024-05-28T06:24:11.847 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.853+0000 7fb00da00640 1 -- 172.21.15.96:0/3624140393 shutdown_connections 2024-05-28T06:24:11.847 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.853+0000 7fb00da00640 1 --2- 172.21.15.96:0/3624140393 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fafe406e510 0x7fafe40709d0 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:11.847 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.853+0000 7fb00da00640 1 --2- 172.21.15.96:0/3624140393 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fb00815f4e0 0x7fb00815f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:11.847 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.853+0000 7fb00da00640 1 --2- 172.21.15.96:0/3624140393 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb0080cb650 0x7fb0080cbad0 unknown :-1 s=CLOSED pgs=1957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:11.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.853+0000 7fb00da00640 1 --2- 172.21.15.96:0/3624140393 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fafec0039e0 0x7fb0080cd2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:11.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.853+0000 7fb00da00640 1 -- 172.21.15.96:0/3624140393 >> 172.21.15.96:0/3624140393 conn(0x7fb00808bfc0 msgr2=0x7fb0081622f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:11.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.854+0000 7fb00da00640 1 -- 172.21.15.96:0/3624140393 shutdown_connections 2024-05-28T06:24:11.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:11.854+0000 7fb00da00640 1 -- 172.21.15.96:0/3624140393 wait complete. 2024-05-28T06:24:11.867 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-05-28T06:24:12.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:12.080+0000 7fc57f800640 1 Processor -- start 2024-05-28T06:24:12.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:12.080+0000 7fc57f800640 1 -- start start 2024-05-28T06:24:12.075 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:12.081+0000 7fc57f800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fc57815b340 0x7fc57806f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:12.075 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:12.081+0000 7fc57f800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc57815c2c0 0x7fc5780b3300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:12.075 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:12.081+0000 7fc57f800640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc57806fa90 con 0x7fc57815b720 2024-05-28T06:24:12.075 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:12.081+0000 7fc57f800640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fc5780af9b0 con 0x7fc57815b340 2024-05-28T06:24:12.075 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:12.081+0000 7fc57f800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fc5780aeda0 con 0x7fc57815c2c0 2024-05-28T06:24:12.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:12.082+0000 7fc57ce00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fc57815b340 0x7fc57806f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:12.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:12.082+0000 7fc57ce00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fc57815b340 0x7fc57806f550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:43588/0 (socket says 172.21.15.96:43588) 2024-05-28T06:24:12.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:12.082+0000 7fc57ce00640 1 -- 172.21.15.96:0/3137981776 learned_addr learned my addr 172.21.15.96:0/3137981776 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:24:12.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:12.082+0000 7fc57d800640 1 --2- 172.21.15.96:0/3137981776 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc57815c2c0 0x7fc5780b3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:12.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:12.082+0000 7fc57ce00640 1 -- 172.21.15.96:0/3137981776 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc57815c2c0 msgr2=0x7fc5780b3300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:12.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:12.082+0000 7fc57ce00640 1 --2- 172.21.15.96:0/3137981776 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc57815c2c0 0x7fc5780b3300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:12.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:12.082+0000 7fc57ce00640 1 -- 172.21.15.96:0/3137981776 >> v1:172.21.15.123:6789/0 conn(0x7fc57815b720 legacy=0x7fc57815bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:12.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:12.082+0000 7fc57ce00640 1 -- 172.21.15.96:0/3137981776 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5780af080 con 0x7fc57815b340 2024-05-28T06:24:12.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:12.084+0000 7fc57ce00640 1 --2- 172.21.15.96:0/3137981776 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc57815b340 0x7fc57806f550 secure :-1 s=READY pgs=1958 cs=0 l=1 rev1=1 crypto rx=0x7fc568009870 tx=0x7fc56802a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfaa207195cadacf server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:12.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:12.084+0000 7fc577e00640 1 -- 172.21.15.96:0/3137981776 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc568037070 con 0x7fc57815b340 2024-05-28T06:24:12.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:12.084+0000 7fc577e00640 1 -- 172.21.15.96:0/3137981776 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc568032440 con 0x7fc57815b340 2024-05-28T06:24:12.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:12.084+0000 7fc577e00640 1 -- 172.21.15.96:0/3137981776 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc568032810 con 0x7fc57815b340 2024-05-28T06:24:12.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:12.085+0000 7fc57f800640 1 -- 172.21.15.96:0/3137981776 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc57815b340 msgr2=0x7fc57806f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:12.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:12.085+0000 7fc57f800640 1 --2- 172.21.15.96:0/3137981776 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc57815b340 0x7fc57806f550 secure :-1 s=READY pgs=1958 cs=0 l=1 rev1=1 crypto rx=0x7fc568009870 tx=0x7fc56802a210 comp rx=0 tx=0).stop 2024-05-28T06:24:12.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:12.085+0000 7fc57f800640 1 -- 172.21.15.96:0/3137981776 shutdown_connections 2024-05-28T06:24:12.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:12.085+0000 7fc57f800640 1 --2- 172.21.15.96:0/3137981776 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc57815c2c0 0x7fc5780b3300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:12.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:12.085+0000 7fc57f800640 1 --2- 172.21.15.96:0/3137981776 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc57815b340 0x7fc57806f550 unknown :-1 s=CLOSED pgs=1958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:12.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:12.085+0000 7fc57f800640 1 -- 172.21.15.96:0/3137981776 >> 172.21.15.96:0/3137981776 conn(0x7fc5780b1e50 msgr2=0x7fc5780b2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:12.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:12.085+0000 7fc57f800640 1 -- 172.21.15.96:0/3137981776 shutdown_connections 2024-05-28T06:24:12.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:12.085+0000 7fc57f800640 1 -- 172.21.15.96:0/3137981776 wait complete. 2024-05-28T06:24:12.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:12.087+0000 7fc57f800640 1 Processor -- start 2024-05-28T06:24:12.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:12.087+0000 7fc57f800640 1 -- start start 2024-05-28T06:24:12.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:12.088+0000 7fc57f800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc57815b340 0x7fc57814f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:12.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:12.088+0000 7fc57f800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc57815b720 0x7fc5780b3570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:12.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:12.088+0000 7fc57f800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc57815c2c0 0x7fc57815dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:12.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:12.088+0000 7fc57f800640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc5780aea30 con 0x7fc57815b340 2024-05-28T06:24:12.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:12.088+0000 7fc57f800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc5780af640 con 0x7fc57815c2c0 2024-05-28T06:24:12.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:12.088+0000 7fc57f800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fc57815c8c0 con 0x7fc57815b720 2024-05-28T06:24:12.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:12.088+0000 7fc57ce00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc57815b340 0x7fc57814f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:12.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:12.088+0000 7fc57ce00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc57815b340 0x7fc57814f500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:43594/0 (socket says 172.21.15.96:43594) 2024-05-28T06:24:12.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:12.088+0000 7fc57ce00640 1 -- 172.21.15.96:0/1829183637 learned_addr learned my addr 172.21.15.96:0/1829183637 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:24:12.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:12.089+0000 7fc57ce00640 1 -- 172.21.15.96:0/1829183637 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc57815b720 msgr2=0x7fc5780b3570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:12.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:12.089+0000 7fc57ce00640 1 --2- 172.21.15.96:0/1829183637 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc57815b720 0x7fc5780b3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:12.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:12.089+0000 7fc57ce00640 1 -- 172.21.15.96:0/1829183637 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc57815c2c0 msgr2=0x7fc57815dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:12.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:12.089+0000 7fc57ce00640 1 --2- 172.21.15.96:0/1829183637 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc57815c2c0 0x7fc57815dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:12.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:12.089+0000 7fc57ce00640 1 -- 172.21.15.96:0/1829183637 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc57815ec00 con 0x7fc57815b340 2024-05-28T06:24:12.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:12.089+0000 7fc57ce00640 1 --2- 172.21.15.96:0/1829183637 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc57815b340 0x7fc57814f500 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7fc568009870 tx=0x7fc5680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:12.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:12.089+0000 7fc575600640 1 -- 172.21.15.96:0/1829183637 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc568041070 con 0x7fc57815b340 2024-05-28T06:24:12.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:12.089+0000 7fc575600640 1 -- 172.21.15.96:0/1829183637 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc56803c870 con 0x7fc57815b340 2024-05-28T06:24:12.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:12.089+0000 7fc575600640 1 -- 172.21.15.96:0/1829183637 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc568037040 con 0x7fc57815b340 2024-05-28T06:24:12.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:12.089+0000 7fc57f800640 1 -- 172.21.15.96:0/1829183637 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc57814ed40 con 0x7fc57815b340 2024-05-28T06:24:12.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:12.089+0000 7fc57f800640 1 -- 172.21.15.96:0/1829183637 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc57814efe0 con 0x7fc57815b340 2024-05-28T06:24:12.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:12.091+0000 7fc57f800640 1 -- 172.21.15.96:0/1829183637 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5780689e0 con 0x7fc57815b340 2024-05-28T06:24:12.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:12.092+0000 7fc575600640 1 -- 172.21.15.96:0/1829183637 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 42) ==== 93590+0+0 (secure 0 0 0) 0x7fc568044070 con 0x7fc57815b340 2024-05-28T06:24:12.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:12.093+0000 7fc575600640 1 --2- 172.21.15.96:0/1829183637 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc55406e510 0x7fc5540709d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:12.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:12.093+0000 7fc575600640 1 -- 172.21.15.96:0/1829183637 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 5937+0+0 (secure 0 0 0) 0x7fc5680ab610 con 0x7fc57815b340 2024-05-28T06:24:12.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:12.093+0000 7fc56fe00640 1 --2- 172.21.15.96:0/1829183637 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc55406e510 0x7fc5540709d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:12.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:12.095+0000 7fc56fe00640 1 --2- 172.21.15.96:0/1829183637 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc55406e510 0x7fc5540709d0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fc57806f840 tx=0x7fc5600079b0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:12.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:12.098+0000 7fc575600640 1 -- 172.21.15.96:0/1829183637 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc56807b5c0 con 0x7fc57815b340 2024-05-28T06:24:12.296 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:12.303+0000 7fc57f800640 1 -- 172.21.15.96:0/1829183637 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fc57806f870 con 0x7fc57815b340 2024-05-28T06:24:12.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:12.854+0000 7fc575600640 1 -- 172.21.15.96:0/1829183637 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v121) ==== 181+0+0 (secure 0 0 0) 0x7fc56807aec0 con 0x7fc57815b340 2024-05-28T06:24:12.848 INFO:teuthology.orchestra.run.smithi096.stderr:pool 'cephfs_metadata' removed 2024-05-28T06:24:12.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:12.858+0000 7fc57f800640 1 -- 172.21.15.96:0/1829183637 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc55406e510 msgr2=0x7fc5540709d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:12.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:12.858+0000 7fc57f800640 1 --2- 172.21.15.96:0/1829183637 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc55406e510 0x7fc5540709d0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fc57806f840 tx=0x7fc5600079b0 comp rx=0 tx=0).stop 2024-05-28T06:24:12.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:12.858+0000 7fc57f800640 1 -- 172.21.15.96:0/1829183637 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc57815b340 msgr2=0x7fc57814f500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:12.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:12.858+0000 7fc57f800640 1 --2- 172.21.15.96:0/1829183637 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc57815b340 0x7fc57814f500 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7fc568009870 tx=0x7fc5680046c0 comp rx=0 tx=0).stop 2024-05-28T06:24:12.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:12.858+0000 7fc57f800640 1 -- 172.21.15.96:0/1829183637 shutdown_connections 2024-05-28T06:24:12.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:12.858+0000 7fc57f800640 1 --2- 172.21.15.96:0/1829183637 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc55406e510 0x7fc5540709d0 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:12.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:12.858+0000 7fc57f800640 1 --2- 172.21.15.96:0/1829183637 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc57815c2c0 0x7fc57815dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:12.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:12.858+0000 7fc57f800640 1 --2- 172.21.15.96:0/1829183637 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc57815b720 0x7fc5780b3570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:12.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:12.858+0000 7fc57f800640 1 --2- 172.21.15.96:0/1829183637 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc57815b340 0x7fc57814f500 unknown :-1 s=CLOSED pgs=1959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:12.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:12.858+0000 7fc57f800640 1 -- 172.21.15.96:0/1829183637 >> 172.21.15.96:0/1829183637 conn(0x7fc5780b1e50 msgr2=0x7fc578071c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:12.853 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:12.859+0000 7fc57f800640 1 -- 172.21.15.96:0/1829183637 shutdown_connections 2024-05-28T06:24:12.854 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:12.859+0000 7fc57f800640 1 -- 172.21.15.96:0/1829183637 wait complete. 2024-05-28T06:24:12.881 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-05-28T06:24:13.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:13.081+0000 7f2e35800640 1 Processor -- start 2024-05-28T06:24:13.075 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:13.081+0000 7f2e35800640 1 -- start start 2024-05-28T06:24:13.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:13.082+0000 7f2e35800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f2e3006edf0 0x7f2e3013fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:13.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:13.082+0000 7f2e35800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2e3006f250 0x7f2e30140330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:13.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:13.082+0000 7f2e35800640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e300d0020 con 0x7f2e300cf2f0 2024-05-28T06:24:13.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:13.082+0000 7f2e35800640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f2e3006dd50 con 0x7f2e3006edf0 2024-05-28T06:24:13.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:13.082+0000 7f2e35800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f2e3006e030 con 0x7f2e3006f250 2024-05-28T06:24:13.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:13.083+0000 7f2e2ea00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f2e300cf2f0 0x7f2e300cf910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:57540/0 (socket says 172.21.15.96:57540) 2024-05-28T06:24:13.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:13.083+0000 7f2e2e000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f2e3006edf0 0x7f2e3013fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:13.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:13.083+0000 7f2e2ea00640 1 -- 172.21.15.96:0/2788654358 learned_addr learned my addr 172.21.15.96:0/2788654358 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:24:13.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:13.083+0000 7f2e2f400640 1 --2- 172.21.15.96:0/2788654358 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2e3006f250 0x7f2e30140330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:13.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:13.083+0000 7f2e2e000640 1 -- 172.21.15.96:0/2788654358 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2e3006f250 msgr2=0x7f2e30140330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:13.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:13.083+0000 7f2e2e000640 1 --2- 172.21.15.96:0/2788654358 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2e3006f250 0x7f2e30140330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:13.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:13.083+0000 7f2e2e000640 1 -- 172.21.15.96:0/2788654358 >> v1:172.21.15.96:6789/0 conn(0x7f2e300cf2f0 legacy=0x7f2e300cf910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:13.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:13.083+0000 7f2e2e000640 1 -- 172.21.15.96:0/2788654358 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e3006e310 con 0x7f2e3006edf0 2024-05-28T06:24:13.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:13.083+0000 7f2e2d600640 1 -- 172.21.15.96:0/2788654358 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3774987265 0 0) 0x7f2e300d0020 con 0x7f2e300cf2f0 2024-05-28T06:24:13.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:13.083+0000 7f2e2f400640 1 --2- 172.21.15.96:0/2788654358 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2e3006f250 0x7f2e30140330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:24:13.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:13.084+0000 7f2e2e000640 1 --2- 172.21.15.96:0/2788654358 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2e3006edf0 0x7f2e3013fdf0 secure :-1 s=READY pgs=1961 cs=0 l=1 rev1=1 crypto rx=0x7f2e180097a0 tx=0x7f2e1802a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a705588a0403e90 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:13.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:13.085+0000 7f2e2d600640 1 -- 172.21.15.96:0/2788654358 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e18037070 con 0x7f2e3006edf0 2024-05-28T06:24:13.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:13.085+0000 7f2e2d600640 1 -- 172.21.15.96:0/2788654358 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e1802ae50 con 0x7f2e3006edf0 2024-05-28T06:24:13.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:13.085+0000 7f2e2d600640 1 -- 172.21.15.96:0/2788654358 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e18032660 con 0x7f2e3006edf0 2024-05-28T06:24:13.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:13.085+0000 7f2e35800640 1 -- 172.21.15.96:0/2788654358 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2e3006edf0 msgr2=0x7f2e3013fdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:13.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:13.085+0000 7f2e35800640 1 --2- 172.21.15.96:0/2788654358 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2e3006edf0 0x7f2e3013fdf0 secure :-1 s=READY pgs=1961 cs=0 l=1 rev1=1 crypto rx=0x7f2e180097a0 tx=0x7f2e1802a100 comp rx=0 tx=0).stop 2024-05-28T06:24:13.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:13.085+0000 7f2e35800640 1 -- 172.21.15.96:0/2788654358 shutdown_connections 2024-05-28T06:24:13.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:13.085+0000 7f2e35800640 1 --2- 172.21.15.96:0/2788654358 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2e3006f250 0x7f2e30140330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:13.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:13.085+0000 7f2e35800640 1 --2- 172.21.15.96:0/2788654358 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2e3006edf0 0x7f2e3013fdf0 unknown :-1 s=CLOSED pgs=1961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:13.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:13.085+0000 7f2e35800640 1 -- 172.21.15.96:0/2788654358 >> 172.21.15.96:0/2788654358 conn(0x7f2e300b1e50 msgr2=0x7f2e300b2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:13.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:13.086+0000 7f2e35800640 1 -- 172.21.15.96:0/2788654358 shutdown_connections 2024-05-28T06:24:13.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:13.086+0000 7f2e35800640 1 -- 172.21.15.96:0/2788654358 wait complete. 2024-05-28T06:24:13.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:13.087+0000 7f2e35800640 1 Processor -- start 2024-05-28T06:24:13.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:13.087+0000 7f2e35800640 1 -- start start 2024-05-28T06:24:13.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:13.088+0000 7f2e35800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e3006edf0 0x7f2e3015f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:13.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:13.088+0000 7f2e35800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2e3006f250 0x7f2e300c9e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:13.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:13.088+0000 7f2e35800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2e300cf2f0 0x7f2e30143fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:13.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:13.088+0000 7f2e35800640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f2e30146020 con 0x7f2e3006f250 2024-05-28T06:24:13.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:13.088+0000 7f2e35800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f2e3006e8d0 con 0x7f2e3006edf0 2024-05-28T06:24:13.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:13.088+0000 7f2e35800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f2e301461a0 con 0x7f2e300cf2f0 2024-05-28T06:24:13.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:13.089+0000 7f2e2e000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2e3006f250 0x7f2e300c9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:13.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:13.089+0000 7f2e2e000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2e3006f250 0x7f2e300c9e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:43614/0 (socket says 172.21.15.96:43614) 2024-05-28T06:24:13.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:13.089+0000 7f2e2e000640 1 -- 172.21.15.96:0/2337963353 learned_addr learned my addr 172.21.15.96:0/2337963353 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:24:13.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:13.089+0000 7f2e2e000640 1 -- 172.21.15.96:0/2337963353 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2e300cf2f0 msgr2=0x7f2e30143fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:13.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:13.089+0000 7f2e2f400640 1 --2- 172.21.15.96:0/2337963353 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2e300cf2f0 0x7f2e30143fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:13.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:13.089+0000 7f2e2ea00640 1 --2- 172.21.15.96:0/2337963353 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e3006edf0 0x7f2e3015f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:13.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:13.089+0000 7f2e2e000640 1 --2- 172.21.15.96:0/2337963353 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2e300cf2f0 0x7f2e30143fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:13.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:13.089+0000 7f2e2e000640 1 -- 172.21.15.96:0/2337963353 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e3006edf0 msgr2=0x7f2e3015f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:13.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:13.089+0000 7f2e2e000640 1 --2- 172.21.15.96:0/2337963353 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e3006edf0 0x7f2e3015f720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:13.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:13.089+0000 7f2e2e000640 1 -- 172.21.15.96:0/2337963353 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e300ca8c0 con 0x7f2e3006f250 2024-05-28T06:24:13.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:13.089+0000 7f2e2e000640 1 --2- 172.21.15.96:0/2337963353 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2e3006f250 0x7f2e300c9e90 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 crypto rx=0x7f2e18035040 tx=0x7f2e180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:13.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:13.090+0000 7f2e26a00640 1 -- 172.21.15.96:0/2337963353 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e18041070 con 0x7f2e3006f250 2024-05-28T06:24:13.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:13.090+0000 7f2e26a00640 1 -- 172.21.15.96:0/2337963353 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e1803c670 con 0x7f2e3006f250 2024-05-28T06:24:13.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:13.090+0000 7f2e35800640 1 -- 172.21.15.96:0/2337963353 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e300caba0 con 0x7f2e3006f250 2024-05-28T06:24:13.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:13.090+0000 7f2e26a00640 1 -- 172.21.15.96:0/2337963353 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e18037040 con 0x7f2e3006f250 2024-05-28T06:24:13.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:13.090+0000 7f2e35800640 1 -- 172.21.15.96:0/2337963353 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e3015d3d0 con 0x7f2e3006f250 2024-05-28T06:24:13.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:13.091+0000 7f2e35800640 1 -- 172.21.15.96:0/2337963353 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e300689e0 con 0x7f2e3006f250 2024-05-28T06:24:13.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:13.092+0000 7f2e26a00640 1 -- 172.21.15.96:0/2337963353 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 42) ==== 93590+0+0 (secure 0 0 0) 0x7f2e1803a070 con 0x7f2e3006f250 2024-05-28T06:24:13.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:13.093+0000 7f2e26a00640 1 --2- 172.21.15.96:0/2337963353 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2e0c06e510 0x7f2e0c0709d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:13.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:13.093+0000 7f2e26a00640 1 -- 172.21.15.96:0/2337963353 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 5471+0+0 (secure 0 0 0) 0x7f2e180aad30 con 0x7f2e3006f250 2024-05-28T06:24:13.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:13.094+0000 7f2e2ea00640 1 --2- 172.21.15.96:0/2337963353 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2e0c06e510 0x7f2e0c0709d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:13.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:13.095+0000 7f2e2ea00640 1 --2- 172.21.15.96:0/2337963353 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2e0c06e510 0x7f2e0c0709d0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f2e20007fd0 tx=0x7f2e20009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:13.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:13.098+0000 7f2e26a00640 1 -- 172.21.15.96:0/2337963353 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2e1807ac60 con 0x7f2e3006f250 2024-05-28T06:24:13.301 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:13.308+0000 7f2e35800640 1 -- 172.21.15.96:0/2337963353 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f2e30073f30 con 0x7f2e3006f250 2024-05-28T06:24:13.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:13.868+0000 7f2e26a00640 1 -- 172.21.15.96:0/2337963353 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v122) ==== 169+0+0 (secure 0 0 0) 0x7f2e1807a560 con 0x7f2e3006f250 2024-05-28T06:24:13.862 INFO:teuthology.orchestra.run.smithi096.stderr:pool 'cephfs_data' removed 2024-05-28T06:24:13.865 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:13.872+0000 7f2e35800640 1 -- 172.21.15.96:0/2337963353 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2e0c06e510 msgr2=0x7f2e0c0709d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:13.865 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:13.872+0000 7f2e35800640 1 --2- 172.21.15.96:0/2337963353 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2e0c06e510 0x7f2e0c0709d0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f2e20007fd0 tx=0x7f2e20009040 comp rx=0 tx=0).stop 2024-05-28T06:24:13.866 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:13.872+0000 7f2e35800640 1 -- 172.21.15.96:0/2337963353 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2e3006f250 msgr2=0x7f2e300c9e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:13.866 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:13.872+0000 7f2e35800640 1 --2- 172.21.15.96:0/2337963353 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2e3006f250 0x7f2e300c9e90 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 crypto rx=0x7f2e18035040 tx=0x7f2e180046c0 comp rx=0 tx=0).stop 2024-05-28T06:24:13.866 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:13.872+0000 7f2e35800640 1 -- 172.21.15.96:0/2337963353 shutdown_connections 2024-05-28T06:24:13.866 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:13.872+0000 7f2e35800640 1 --2- 172.21.15.96:0/2337963353 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2e0c06e510 0x7f2e0c0709d0 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:13.866 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:13.872+0000 7f2e35800640 1 --2- 172.21.15.96:0/2337963353 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2e300cf2f0 0x7f2e30143fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:13.866 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:13.872+0000 7f2e35800640 1 --2- 172.21.15.96:0/2337963353 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2e3006f250 0x7f2e300c9e90 unknown :-1 s=CLOSED pgs=1962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:13.867 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:13.872+0000 7f2e35800640 1 --2- 172.21.15.96:0/2337963353 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e3006edf0 0x7f2e3015f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:13.867 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:13.872+0000 7f2e35800640 1 -- 172.21.15.96:0/2337963353 >> 172.21.15.96:0/2337963353 conn(0x7f2e300b1e50 msgr2=0x7f2e3015be80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:13.867 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:13.873+0000 7f2e35800640 1 -- 172.21.15.96:0/2337963353 shutdown_connections 2024-05-28T06:24:13.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:13.874+0000 7f2e35800640 1 -- 172.21.15.96:0/2337963353 wait complete. 2024-05-28T06:24:13.896 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-05-28T06:24:14.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.091+0000 7f4746800640 1 Processor -- start 2024-05-28T06:24:14.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.092+0000 7f4746800640 1 -- start start 2024-05-28T06:24:14.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.093+0000 7f4746800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f4740153510 0x7f47400b4d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:14.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.093+0000 7f4746800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f47400ccb90 0x7f47400ccff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:14.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.093+0000 7f4746800640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f474015d900 con 0x7f474015cb60 2024-05-28T06:24:14.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.093+0000 7f4746800640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f47401538f0 con 0x7f4740153510 2024-05-28T06:24:14.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.093+0000 7f4746800640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f47400cd530 con 0x7f47400ccb90 2024-05-28T06:24:14.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.094+0000 7f4745400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f4740153510 0x7f47400b4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:14.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.094+0000 7f4745400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f4740153510 0x7f47400b4d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:43630/0 (socket says 172.21.15.96:43630) 2024-05-28T06:24:14.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.094+0000 7f4745400640 1 -- 172.21.15.96:0/1369124645 learned_addr learned my addr 172.21.15.96:0/1369124645 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:24:14.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.094+0000 7f4744a00640 1 --2- 172.21.15.96:0/1369124645 >> v2:172.21.15.123:3300/0 conn(0x7f47400ccb90 0x7f47400ccff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:14.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.094+0000 7f4745400640 1 -- 172.21.15.96:0/1369124645 >> v2:172.21.15.123:3300/0 conn(0x7f47400ccb90 msgr2=0x7f47400ccff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:14.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.094+0000 7f4745400640 1 --2- 172.21.15.96:0/1369124645 >> v2:172.21.15.123:3300/0 conn(0x7f47400ccb90 0x7f47400ccff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:14.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.094+0000 7f4745400640 1 -- 172.21.15.96:0/1369124645 >> v1:172.21.15.123:6789/0 conn(0x7f474015cb60 legacy=0x7f474015cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:14.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.094+0000 7f4745400640 1 -- 172.21.15.96:0/1369124645 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47400b3f90 con 0x7f4740153510 2024-05-28T06:24:14.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.094+0000 7f4744a00640 1 --2- 172.21.15.96:0/1369124645 >> v2:172.21.15.123:3300/0 conn(0x7f47400ccb90 0x7f47400ccff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:24:14.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.095+0000 7f4745400640 1 --2- 172.21.15.96:0/1369124645 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4740153510 0x7f47400b4d50 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 crypto rx=0x7f47400cd7a0 tx=0x7f473402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd21b788acad9beb server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:14.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.096+0000 7f473fe00640 1 -- 172.21.15.96:0/1369124645 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4734037070 con 0x7f4740153510 2024-05-28T06:24:14.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.096+0000 7f473fe00640 1 -- 172.21.15.96:0/1369124645 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4734032440 con 0x7f4740153510 2024-05-28T06:24:14.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.096+0000 7f473fe00640 1 -- 172.21.15.96:0/1369124645 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4734032810 con 0x7f4740153510 2024-05-28T06:24:14.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.096+0000 7f4746800640 1 -- 172.21.15.96:0/1369124645 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4740153510 msgr2=0x7f47400b4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:14.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.096+0000 7f4746800640 1 --2- 172.21.15.96:0/1369124645 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4740153510 0x7f47400b4d50 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 crypto rx=0x7f47400cd7a0 tx=0x7f473402a1a0 comp rx=0 tx=0).stop 2024-05-28T06:24:14.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.096+0000 7f4746800640 1 -- 172.21.15.96:0/1369124645 shutdown_connections 2024-05-28T06:24:14.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.096+0000 7f4746800640 1 --2- 172.21.15.96:0/1369124645 >> v2:172.21.15.123:3300/0 conn(0x7f47400ccb90 0x7f47400ccff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:14.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.096+0000 7f4746800640 1 --2- 172.21.15.96:0/1369124645 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4740153510 0x7f47400b4d50 unknown :-1 s=CLOSED pgs=1963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:14.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.096+0000 7f4746800640 1 -- 172.21.15.96:0/1369124645 >> 172.21.15.96:0/1369124645 conn(0x7f474006b900 msgr2=0x7f474006bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:14.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.096+0000 7f4746800640 1 -- 172.21.15.96:0/1369124645 shutdown_connections 2024-05-28T06:24:14.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.096+0000 7f4746800640 1 -- 172.21.15.96:0/1369124645 wait complete. 2024-05-28T06:24:14.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.098+0000 7f4746800640 1 Processor -- start 2024-05-28T06:24:14.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.098+0000 7f4746800640 1 -- start start 2024-05-28T06:24:14.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.099+0000 7f4746800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f47400ccb90 0x7f47401f7ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:14.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.099+0000 7f4746800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4740153510 0x7f4740073480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:14.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.099+0000 7f4746800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f474015cb60 0x7f4740160150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:14.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.099+0000 7f4746800640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f474015d6a0 con 0x7f474015cb60 2024-05-28T06:24:14.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.099+0000 7f4746800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f47400b48c0 con 0x7f4740153510 2024-05-28T06:24:14.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.099+0000 7f4746800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f47400b4550 con 0x7f47400ccb90 2024-05-28T06:24:14.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.100+0000 7f4745e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f474015cb60 0x7f4740160150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:14.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.100+0000 7f4745e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f474015cb60 0x7f4740160150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:43642/0 (socket says 172.21.15.96:43642) 2024-05-28T06:24:14.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.100+0000 7f4745e00640 1 -- 172.21.15.96:0/2622395874 learned_addr learned my addr 172.21.15.96:0/2622395874 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:24:14.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.100+0000 7f4744a00640 1 --2- 172.21.15.96:0/2622395874 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4740153510 0x7f4740073480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:14.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.100+0000 7f4745400640 1 --2- 172.21.15.96:0/2622395874 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f47400ccb90 0x7f47401f7ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:14.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.100+0000 7f4745e00640 1 -- 172.21.15.96:0/2622395874 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f47400ccb90 msgr2=0x7f47401f7ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:14.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.100+0000 7f4745e00640 1 --2- 172.21.15.96:0/2622395874 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f47400ccb90 0x7f47401f7ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:14.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.100+0000 7f4745e00640 1 -- 172.21.15.96:0/2622395874 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4740153510 msgr2=0x7f4740073480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:14.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.100+0000 7f4745e00640 1 --2- 172.21.15.96:0/2622395874 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4740153510 0x7f4740073480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:14.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.100+0000 7f4745e00640 1 -- 172.21.15.96:0/2622395874 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47400739c0 con 0x7f474015cb60 2024-05-28T06:24:14.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.100+0000 7f4745e00640 1 --2- 172.21.15.96:0/2622395874 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f474015cb60 0x7f4740160150 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 crypto rx=0x7f473800c780 tx=0x7f473800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:14.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.101+0000 7f473d600640 1 -- 172.21.15.96:0/2622395874 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4738004b60 con 0x7f474015cb60 2024-05-28T06:24:14.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.101+0000 7f473d600640 1 -- 172.21.15.96:0/2622395874 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4738004d00 con 0x7f474015cb60 2024-05-28T06:24:14.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.101+0000 7f4746800640 1 -- 172.21.15.96:0/2622395874 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47401f8180 con 0x7f474015cb60 2024-05-28T06:24:14.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.101+0000 7f473d600640 1 -- 172.21.15.96:0/2622395874 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f473800aa90 con 0x7f474015cb60 2024-05-28T06:24:14.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.101+0000 7f4746800640 1 -- 172.21.15.96:0/2622395874 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47401f8740 con 0x7f474015cb60 2024-05-28T06:24:14.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.102+0000 7f4746800640 1 -- 172.21.15.96:0/2622395874 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f474015c950 con 0x7f474015cb60 2024-05-28T06:24:14.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.103+0000 7f473d600640 1 -- 172.21.15.96:0/2622395874 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 42) ==== 93590+0+0 (secure 0 0 0) 0x7f473800ac30 con 0x7f474015cb60 2024-05-28T06:24:14.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.104+0000 7f473d600640 1 --2- 172.21.15.96:0/2622395874 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f472406e510 0x7f47240709d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:14.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.104+0000 7f473d600640 1 -- 172.21.15.96:0/2622395874 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 4999+0+0 (secure 0 0 0) 0x7f473808ae30 con 0x7f474015cb60 2024-05-28T06:24:14.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.105+0000 7f4745400640 1 --2- 172.21.15.96:0/2622395874 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f472406e510 0x7f47240709d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:14.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.106+0000 7f4745400640 1 --2- 172.21.15.96:0/2622395874 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f472406e510 0x7f47240709d0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f4734034000 tx=0x7f473402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:14.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.108+0000 7f473d600640 1 -- 172.21.15.96:0/2622395874 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f474015c950 con 0x7f474015cb60 2024-05-28T06:24:14.306 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.313+0000 7f4746800640 1 -- 172.21.15.96:0/2622395874 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7f47400689e0 con 0x7f474015cb60 2024-05-28T06:24:14.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.320+0000 7f473d600640 1 -- 172.21.15.96:0/2622395874 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v27) ==== 104+0+0 (secure 0 0 0) 0x7f4738020070 con 0x7f474015cb60 2024-05-28T06:24:14.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.323+0000 7f4746800640 1 -- 172.21.15.96:0/2622395874 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f472406e510 msgr2=0x7f47240709d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:14.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.323+0000 7f4746800640 1 --2- 172.21.15.96:0/2622395874 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f472406e510 0x7f47240709d0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f4734034000 tx=0x7f473402e000 comp rx=0 tx=0).stop 2024-05-28T06:24:14.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.323+0000 7f4746800640 1 -- 172.21.15.96:0/2622395874 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f474015cb60 msgr2=0x7f4740160150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:14.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.323+0000 7f4746800640 1 --2- 172.21.15.96:0/2622395874 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f474015cb60 0x7f4740160150 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 crypto rx=0x7f473800c780 tx=0x7f473800cc40 comp rx=0 tx=0).stop 2024-05-28T06:24:14.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.323+0000 7f4746800640 1 -- 172.21.15.96:0/2622395874 shutdown_connections 2024-05-28T06:24:14.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.323+0000 7f4746800640 1 --2- 172.21.15.96:0/2622395874 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f472406e510 0x7f47240709d0 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:14.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.323+0000 7f4746800640 1 --2- 172.21.15.96:0/2622395874 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f474015cb60 0x7f4740160150 unknown :-1 s=CLOSED pgs=1964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:14.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.323+0000 7f4746800640 1 --2- 172.21.15.96:0/2622395874 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4740153510 0x7f4740073480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:14.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.323+0000 7f4746800640 1 --2- 172.21.15.96:0/2622395874 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f47400ccb90 0x7f47401f7ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:14.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.323+0000 7f4746800640 1 -- 172.21.15.96:0/2622395874 >> 172.21.15.96:0/2622395874 conn(0x7f474006b900 msgr2=0x7f47400d0980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:14.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.323+0000 7f4746800640 1 -- 172.21.15.96:0/2622395874 shutdown_connections 2024-05-28T06:24:14.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.323+0000 7f4746800640 1 -- 172.21.15.96:0/2622395874 wait complete. 2024-05-28T06:24:14.337 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestCloneProgressReporter.test_when_3_clones_are_cancelled' 2024-05-28T06:24:14.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.526+0000 7f9fb8600640 1 Processor -- start 2024-05-28T06:24:14.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.527+0000 7f9fb8600640 1 -- start start 2024-05-28T06:24:14.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.528+0000 7f9fb8600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9fb006f860 0x7f9fb00b3150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:14.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.528+0000 7f9fb8600640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9fb006ed10 con 0x7f9fb00b3690 2024-05-28T06:24:14.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.528+0000 7f9fb8600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9fb015cb00 con 0x7f9fb00d0bb0 2024-05-28T06:24:14.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.528+0000 7f9fb8600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f9fb00b3cc0 con 0x7f9fb006f860 2024-05-28T06:24:14.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.528+0000 7f9fb5200640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f9fb00b3690 0x7f9fb00d04a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:57554/0 (socket says 172.21.15.96:57554) 2024-05-28T06:24:14.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.528+0000 7f9fb5200640 1 -- 172.21.15.96:0/4051792682 learned_addr learned my addr 172.21.15.96:0/4051792682 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:24:14.523 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.529+0000 7f9fb5c00640 1 --2- 172.21.15.96:0/4051792682 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9fb006f860 0x7f9fb00b3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:14.524 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.529+0000 7f9fafe00640 1 -- 172.21.15.96:0/4051792682 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2411089212 0 0) 0x7f9f9c001110 con 0x7f9fb00b3690 2024-05-28T06:24:14.524 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.529+0000 7f9fafe00640 1 -- 172.21.15.96:0/4051792682 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9f98003520 con 0x7f9fb00b3690 2024-05-28T06:24:14.524 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.529+0000 7f9fafe00640 1 -- 172.21.15.96:0/4051792682 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2975375112 0 0) 0x7f9f98003520 con 0x7f9fb00b3690 2024-05-28T06:24:14.524 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.529+0000 7f9fafe00640 1 -- 172.21.15.96:0/4051792682 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9fb006f860 msgr2=0x7f9fb00b3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:14.524 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.529+0000 7f9fafe00640 1 --2- 172.21.15.96:0/4051792682 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9fb006f860 0x7f9fb00b3150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:14.524 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.529+0000 7f9fafe00640 1 -- 172.21.15.96:0/4051792682 >> v1:172.21.15.123:6789/0 conn(0x7f9fb00d0bb0 legacy=0x7f9fb00ad130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:14.524 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.529+0000 7f9fafe00640 1 -- 172.21.15.96:0/4051792682 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fb00b3fa0 con 0x7f9fb00b3690 2024-05-28T06:24:14.524 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.529+0000 7f9fafe00640 1 -- 172.21.15.96:0/4051792682 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f9f9c001840 con 0x7f9fb00b3690 2024-05-28T06:24:14.524 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.529+0000 7f9fafe00640 1 -- 172.21.15.96:0/4051792682 >> v1:172.21.15.96:6789/0 conn(0x7f9fb00b3690 legacy=0x7f9fb00d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:14.524 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.529+0000 7f9fafe00640 1 --2- 172.21.15.96:0/4051792682 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f980039e0 0x7f9f98003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:14.524 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.529+0000 7f9fafe00640 1 --2- 172.21.15.96:0/4051792682 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9fb006f860 0x7f9f98004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:14.524 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.529+0000 7f9fafe00640 1 --2- 172.21.15.96:0/4051792682 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f98004d40 0x7f9f980051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:14.525 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.529+0000 7f9fafe00640 1 -- 172.21.15.96:0/4051792682 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f9fb00b3cc0 con 0x7f9f98004d40 2024-05-28T06:24:14.525 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.529+0000 7f9fafe00640 1 -- 172.21.15.96:0/4051792682 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f9f9800bb50 con 0x7f9f980039e0 2024-05-28T06:24:14.525 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.529+0000 7f9fafe00640 1 -- 172.21.15.96:0/4051792682 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f9f9800bd20 con 0x7f9fb006f860 2024-05-28T06:24:14.525 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.530+0000 7f9fb6600640 1 --2- 172.21.15.96:0/4051792682 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f98004d40 0x7f9f980051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:14.525 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.530+0000 7f9fb5200640 1 --2- 172.21.15.96:0/4051792682 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9fb006f860 0x7f9f98004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:14.525 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.530+0000 7f9fb5c00640 1 --2- 172.21.15.96:0/4051792682 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f980039e0 0x7f9f98003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:14.525 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.530+0000 7f9fb6600640 1 -- 172.21.15.96:0/4051792682 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9fb006f860 msgr2=0x7f9f98004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:14.525 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.530+0000 7f9fb6600640 1 --2- 172.21.15.96:0/4051792682 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9fb006f860 0x7f9f98004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:14.525 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.530+0000 7f9fb6600640 1 -- 172.21.15.96:0/4051792682 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f980039e0 msgr2=0x7f9f98003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:14.525 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.530+0000 7f9fb6600640 1 --2- 172.21.15.96:0/4051792682 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f980039e0 0x7f9f98003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:14.525 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.530+0000 7f9fb6600640 1 -- 172.21.15.96:0/4051792682 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f98003520 con 0x7f9f98004d40 2024-05-28T06:24:14.525 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.530+0000 7f9fb5200640 1 --2- 172.21.15.96:0/4051792682 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9fb006f860 0x7f9f98004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:24:14.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.531+0000 7f9fb6600640 1 --2- 172.21.15.96:0/4051792682 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f98004d40 0x7f9f980051c0 secure :-1 s=READY pgs=1966 cs=0 l=1 rev1=1 crypto rx=0x7f9fa8001e20 tx=0x7f9f98005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7f6bc9f4fda9c70 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:14.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.531+0000 7f9fafe00640 1 -- 172.21.15.96:0/4051792682 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fa8001a70 con 0x7f9f98004d40 2024-05-28T06:24:14.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.531+0000 7f9fafe00640 1 -- 172.21.15.96:0/4051792682 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9fa802ae00 con 0x7f9f98004d40 2024-05-28T06:24:14.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.531+0000 7f9fafe00640 1 -- 172.21.15.96:0/4051792682 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fa8003bb0 con 0x7f9f98004d40 2024-05-28T06:24:14.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.532+0000 7f9fb8600640 1 -- 172.21.15.96:0/4051792682 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f98004d40 msgr2=0x7f9f980051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:14.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.532+0000 7f9fb8600640 1 --2- 172.21.15.96:0/4051792682 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f98004d40 0x7f9f980051c0 secure :-1 s=READY pgs=1966 cs=0 l=1 rev1=1 crypto rx=0x7f9fa8001e20 tx=0x7f9f98005830 comp rx=0 tx=0).stop 2024-05-28T06:24:14.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.532+0000 7f9fb6600640 1 -- 172.21.15.96:0/4051792682 reap_dead start 2024-05-28T06:24:14.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.532+0000 7f9fb8600640 1 -- 172.21.15.96:0/4051792682 shutdown_connections 2024-05-28T06:24:14.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.532+0000 7f9fb8600640 1 -- 172.21.15.96:0/4051792682 >> 172.21.15.96:0/4051792682 conn(0x7f9fb006bae0 msgr2=0x7f9fb006bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:14.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.532+0000 7f9fb8600640 1 -- 172.21.15.96:0/4051792682 shutdown_connections 2024-05-28T06:24:14.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.532+0000 7f9fb8600640 1 -- 172.21.15.96:0/4051792682 wait complete. 2024-05-28T06:24:14.528 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.534+0000 7f9fb8600640 1 Processor -- start 2024-05-28T06:24:14.528 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.534+0000 7f9fb8600640 1 -- start start 2024-05-28T06:24:14.529 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.535+0000 7f9fb8600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f98004d40 0x7f9fb01f4760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:14.529 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.535+0000 7f9fb8600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9fb00c8c00 0x7f9fb00cccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:14.529 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.535+0000 7f9fb8600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9fb00c9080 0x7f9fb00cd230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:14.529 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.535+0000 7f9fb8600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f9fb00b4840 con 0x7f9fb00c9080 2024-05-28T06:24:14.529 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.535+0000 7f9fb8600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f9fb00b4560 con 0x7f9f98004d40 2024-05-28T06:24:14.529 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.535+0000 7f9fb8600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f9fb00b3150 con 0x7f9fb00c8c00 2024-05-28T06:24:14.529 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.535+0000 7f9fb6600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9fb00c9080 0x7f9fb00cd230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:14.529 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.535+0000 7f9fb6600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9fb00c9080 0x7f9fb00cd230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:43660/0 (socket says 172.21.15.96:43660) 2024-05-28T06:24:14.529 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.535+0000 7f9fb6600640 1 -- 172.21.15.96:0/57151421 learned_addr learned my addr 172.21.15.96:0/57151421 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:24:14.530 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.536+0000 7f9fb5c00640 1 --2- 172.21.15.96:0/57151421 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f98004d40 0x7f9fb01f4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:14.530 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.536+0000 7f9fb6600640 1 -- 172.21.15.96:0/57151421 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9fb00c8c00 msgr2=0x7f9fb00cccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:14.530 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.536+0000 7f9fb6600640 1 --2- 172.21.15.96:0/57151421 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9fb00c8c00 0x7f9fb00cccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:14.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.536+0000 7f9fb6600640 1 -- 172.21.15.96:0/57151421 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f98004d40 msgr2=0x7f9fb01f4760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:14.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.536+0000 7f9fb6600640 1 --2- 172.21.15.96:0/57151421 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f98004d40 0x7f9fb01f4760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:14.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.536+0000 7f9fb6600640 1 -- 172.21.15.96:0/57151421 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fb015f030 con 0x7f9fb00c9080 2024-05-28T06:24:14.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.536+0000 7f9fb6600640 1 --2- 172.21.15.96:0/57151421 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9fb00c9080 0x7f9fb00cd230 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 crypto rx=0x7f9fa8001f10 tx=0x7f9fa80014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:14.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.536+0000 7f9fad600640 1 -- 172.21.15.96:0/57151421 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fa8037040 con 0x7f9fb00c9080 2024-05-28T06:24:14.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.536+0000 7f9fad600640 1 -- 172.21.15.96:0/57151421 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9fa802ac00 con 0x7f9fb00c9080 2024-05-28T06:24:14.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.536+0000 7f9fad600640 1 -- 172.21.15.96:0/57151421 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fa8001a70 con 0x7f9fb00c9080 2024-05-28T06:24:14.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.536+0000 7f9fb8600640 1 -- 172.21.15.96:0/57151421 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fb015f310 con 0x7f9fb00c9080 2024-05-28T06:24:14.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.536+0000 7f9fb8600640 1 -- 172.21.15.96:0/57151421 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fb015f8d0 con 0x7f9fb00c9080 2024-05-28T06:24:14.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.538+0000 7f9fb8600640 1 -- 172.21.15.96:0/57151421 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9fb006fc40 con 0x7f9fb00c9080 2024-05-28T06:24:14.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.539+0000 7f9fad600640 1 -- 172.21.15.96:0/57151421 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 42) ==== 93590+0+0 (secure 0 0 0) 0x7f9fa8042020 con 0x7f9fb00c9080 2024-05-28T06:24:14.533 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.540+0000 7f9fad600640 1 --2- 172.21.15.96:0/57151421 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9f9006e510 0x7f9f900709d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:14.533 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.540+0000 7f9fad600640 1 -- 172.21.15.96:0/57151421 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 4999+0+0 (secure 0 0 0) 0x7f9fa80a9fb0 con 0x7f9fb00c9080 2024-05-28T06:24:14.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.540+0000 7f9fb5c00640 1 --2- 172.21.15.96:0/57151421 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9f9006e510 0x7f9f900709d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:14.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.542+0000 7f9fb5c00640 1 --2- 172.21.15.96:0/57151421 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9f9006e510 0x7f9f900709d0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f9fa4006fd0 tx=0x7f9fa4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:14.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.544+0000 7f9fad600640 1 -- 172.21.15.96:0/57151421 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9fb006fc40 con 0x7f9fb00c9080 2024-05-28T06:24:14.813 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.819+0000 7f9fb8600640 1 -- 172.21.15.96:0/57151421 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgressReporter.test_when_3_clones_are_cancelled"]} v 0) -- 0x7f9fb00689e0 con 0x7f9fb00c9080 2024-05-28T06:24:14.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.881+0000 7f9fad600640 1 -- 172.21.15.96:0/57151421 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgressReporter.test_when_3_clones_are_cancelled"]}]=0 v876) ==== 163+0+0 (secure 0 0 0) 0x7f9fa8049020 con 0x7f9fb00c9080 2024-05-28T06:24:14.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.884+0000 7f9fb8600640 1 -- 172.21.15.96:0/57151421 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9f9006e510 msgr2=0x7f9f900709d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:14.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.884+0000 7f9fb8600640 1 --2- 172.21.15.96:0/57151421 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9f9006e510 0x7f9f900709d0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f9fa4006fd0 tx=0x7f9fa4008040 comp rx=0 tx=0).stop 2024-05-28T06:24:14.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.884+0000 7f9fb8600640 1 -- 172.21.15.96:0/57151421 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9fb00c9080 msgr2=0x7f9fb00cd230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:14.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.884+0000 7f9fb8600640 1 --2- 172.21.15.96:0/57151421 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9fb00c9080 0x7f9fb00cd230 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 crypto rx=0x7f9fa8001f10 tx=0x7f9fa80014a0 comp rx=0 tx=0).stop 2024-05-28T06:24:14.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.884+0000 7f9fb8600640 1 -- 172.21.15.96:0/57151421 shutdown_connections 2024-05-28T06:24:14.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.884+0000 7f9fb8600640 1 --2- 172.21.15.96:0/57151421 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f9f9006e510 0x7f9f900709d0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:14.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.884+0000 7f9fb8600640 1 --2- 172.21.15.96:0/57151421 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9fb00c9080 0x7f9fb00cd230 unknown :-1 s=CLOSED pgs=1967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:14.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.884+0000 7f9fb8600640 1 --2- 172.21.15.96:0/57151421 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9fb00c8c00 0x7f9fb00cccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:14.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.884+0000 7f9fb8600640 1 --2- 172.21.15.96:0/57151421 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f98004d40 0x7f9fb01f4760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:14.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.884+0000 7f9fb8600640 1 -- 172.21.15.96:0/57151421 >> 172.21.15.96:0/57151421 conn(0x7f9fb006bae0 msgr2=0x7f9fb00afc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:14.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.884+0000 7f9fb8600640 1 -- 172.21.15.96:0/57151421 shutdown_connections 2024-05-28T06:24:14.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:14.884+0000 7f9fb8600640 1 -- 172.21.15.96:0/57151421 wait complete. 2024-05-28T06:24:14.900 INFO:tasks.cephfs_test_runner:Test that progress bars, that printed for 3 ongoing clone jobs, are ... ok 2024-05-28T06:24:14.900 INFO:tasks.cephfs_test_runner:Starting test: test_when_7_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgressReporter) Test that both the progress bars, that are printed for all 7 clone 2024-05-28T06:24:14.900 INFO:tasks.cephfs_test_runner:test_when_7_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgressReporter) 2024-05-28T06:24:14.900 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:24:15.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.092+0000 7f4a1d800640 1 Processor -- start 2024-05-28T06:24:15.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.092+0000 7f4a1d800640 1 -- start start 2024-05-28T06:24:15.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.093+0000 7f4a1d800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4a1806f860 0x7f4a180b3150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:15.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.093+0000 7f4a1d800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f4a180b3690 0x7f4a180d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:15.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.093+0000 7f4a1d800640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a1806ed10 con 0x7f4a180d09e0 2024-05-28T06:24:15.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.093+0000 7f4a1d800640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f4a180d0e50 con 0x7f4a180b3690 2024-05-28T06:24:15.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.093+0000 7f4a1d800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f4a1815cb00 con 0x7f4a1806f860 2024-05-28T06:24:15.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.094+0000 7f4a16a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4a1806f860 0x7f4a180b3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:15.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.094+0000 7f4a17400640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f4a180d09e0 0x7f4a18160210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.96:42534/0 (socket says 172.21.15.96:42534) 2024-05-28T06:24:15.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.094+0000 7f4a17400640 1 -- 172.21.15.96:0/2823955514 learned_addr learned my addr 172.21.15.96:0/2823955514 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:24:15.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.094+0000 7f4a0e000640 1 --2- 172.21.15.96:0/2823955514 >> v2:172.21.15.123:3300/0 conn(0x7f4a180b3690 0x7f4a180d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:15.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.095+0000 7f4a16a00640 1 -- 172.21.15.96:0/2823955514 >> v2:172.21.15.123:3300/0 conn(0x7f4a180b3690 msgr2=0x7f4a180d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:15.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.095+0000 7f4a16a00640 1 --2- 172.21.15.96:0/2823955514 >> v2:172.21.15.123:3300/0 conn(0x7f4a180b3690 0x7f4a180d04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:15.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.095+0000 7f4a16a00640 1 -- 172.21.15.96:0/2823955514 >> v1:172.21.15.123:6789/0 conn(0x7f4a180d09e0 legacy=0x7f4a18160210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:15.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.095+0000 7f4a16a00640 1 -- 172.21.15.96:0/2823955514 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a180b3cc0 con 0x7f4a1806f860 2024-05-28T06:24:15.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.097+0000 7f4a16a00640 1 --2- 172.21.15.96:0/2823955514 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4a1806f860 0x7f4a180b3150 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f4a080097a0 tx=0x7f4a0802a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=41ad5f26b3cf7e98 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:15.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.098+0000 7f4a16000640 1 -- 172.21.15.96:0/2823955514 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a08037070 con 0x7f4a1806f860 2024-05-28T06:24:15.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.098+0000 7f4a16000640 1 -- 172.21.15.96:0/2823955514 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a08032440 con 0x7f4a1806f860 2024-05-28T06:24:15.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.098+0000 7f4a16000640 1 -- 172.21.15.96:0/2823955514 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a08032810 con 0x7f4a1806f860 2024-05-28T06:24:15.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.098+0000 7f4a1d800640 1 -- 172.21.15.96:0/2823955514 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4a1806f860 msgr2=0x7f4a180b3150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:15.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.098+0000 7f4a1d800640 1 --2- 172.21.15.96:0/2823955514 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4a1806f860 0x7f4a180b3150 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f4a080097a0 tx=0x7f4a0802a180 comp rx=0 tx=0).stop 2024-05-28T06:24:15.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.099+0000 7f4a1d800640 1 -- 172.21.15.96:0/2823955514 shutdown_connections 2024-05-28T06:24:15.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.099+0000 7f4a1d800640 1 --2- 172.21.15.96:0/2823955514 >> v2:172.21.15.123:3300/0 conn(0x7f4a180b3690 0x7f4a180d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:15.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.099+0000 7f4a1d800640 1 --2- 172.21.15.96:0/2823955514 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4a1806f860 0x7f4a180b3150 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:15.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.099+0000 7f4a1d800640 1 -- 172.21.15.96:0/2823955514 >> 172.21.15.96:0/2823955514 conn(0x7f4a1806bae0 msgr2=0x7f4a1806bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:15.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.099+0000 7f4a1d800640 1 -- 172.21.15.96:0/2823955514 shutdown_connections 2024-05-28T06:24:15.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.099+0000 7f4a1d800640 1 -- 172.21.15.96:0/2823955514 wait complete. 2024-05-28T06:24:15.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.101+0000 7f4a1d800640 1 Processor -- start 2024-05-28T06:24:15.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.101+0000 7f4a1d800640 1 -- start start 2024-05-28T06:24:15.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.102+0000 7f4a1d800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4a1806f860 0x7f4a180c9280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:15.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.102+0000 7f4a1d800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4a180b3690 0x7f4a18099850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:15.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.102+0000 7f4a1d800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4a180d09e0 0x7f4a180ccfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:15.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.102+0000 7f4a1d800640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4a180b4560 con 0x7f4a180b3690 2024-05-28T06:24:15.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.102+0000 7f4a1d800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f4a180b4280 con 0x7f4a180d09e0 2024-05-28T06:24:15.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.102+0000 7f4a1d800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f4a180b4840 con 0x7f4a1806f860 2024-05-28T06:24:15.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.102+0000 7f4a0e000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4a180b3690 0x7f4a18099850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:15.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.102+0000 7f4a0e000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4a180b3690 0x7f4a18099850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:43674/0 (socket says 172.21.15.96:43674) 2024-05-28T06:24:15.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.102+0000 7f4a0e000640 1 -- 172.21.15.96:0/4194350822 learned_addr learned my addr 172.21.15.96:0/4194350822 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:24:15.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.103+0000 7f4a16a00640 1 --2- 172.21.15.96:0/4194350822 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4a1806f860 0x7f4a180c9280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:15.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.103+0000 7f4a17400640 1 --2- 172.21.15.96:0/4194350822 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4a180d09e0 0x7f4a180ccfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:15.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.103+0000 7f4a0e000640 1 -- 172.21.15.96:0/4194350822 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4a1806f860 msgr2=0x7f4a180c9280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:15.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.103+0000 7f4a0e000640 1 --2- 172.21.15.96:0/4194350822 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4a1806f860 0x7f4a180c9280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:15.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.103+0000 7f4a0e000640 1 -- 172.21.15.96:0/4194350822 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4a180d09e0 msgr2=0x7f4a180ccfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:15.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.103+0000 7f4a0e000640 1 --2- 172.21.15.96:0/4194350822 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4a180d09e0 0x7f4a180ccfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:15.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.103+0000 7f4a0e000640 1 -- 172.21.15.96:0/4194350822 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a1815d0f0 con 0x7f4a180b3690 2024-05-28T06:24:15.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.103+0000 7f4a0e000640 1 --2- 172.21.15.96:0/4194350822 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4a180b3690 0x7f4a18099850 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 crypto rx=0x7f4a0000e9e0 tx=0x7f4a0000eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:15.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.103+0000 7f4a0f400640 1 -- 172.21.15.96:0/4194350822 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a00017070 con 0x7f4a180b3690 2024-05-28T06:24:15.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.104+0000 7f4a0f400640 1 -- 172.21.15.96:0/4194350822 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a0000c9d0 con 0x7f4a180b3690 2024-05-28T06:24:15.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.104+0000 7f4a0f400640 1 -- 172.21.15.96:0/4194350822 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a0000cdc0 con 0x7f4a180b3690 2024-05-28T06:24:15.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.104+0000 7f4a1d800640 1 -- 172.21.15.96:0/4194350822 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a1815d3d0 con 0x7f4a180b3690 2024-05-28T06:24:15.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.104+0000 7f4a1d800640 1 -- 172.21.15.96:0/4194350822 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a1815d990 con 0x7f4a180b3690 2024-05-28T06:24:15.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.105+0000 7f4a1d800640 1 -- 172.21.15.96:0/4194350822 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a180689e0 con 0x7f4a180b3690 2024-05-28T06:24:15.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.106+0000 7f4a0f400640 1 -- 172.21.15.96:0/4194350822 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 42) ==== 93590+0+0 (secure 0 0 0) 0x7f4a0001a070 con 0x7f4a180b3690 2024-05-28T06:24:15.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.107+0000 7f4a0f400640 1 --2- 172.21.15.96:0/4194350822 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f49f406e510 0x7f49f40709d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:15.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.108+0000 7f4a0f400640 1 -- 172.21.15.96:0/4194350822 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 4999+0+0 (secure 0 0 0) 0x7f4a0008aec0 con 0x7f4a180b3690 2024-05-28T06:24:15.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.108+0000 7f4a16a00640 1 --2- 172.21.15.96:0/4194350822 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f49f406e510 0x7f49f40709d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:15.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.109+0000 7f4a16a00640 1 --2- 172.21.15.96:0/4194350822 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f49f406e510 0x7f49f40709d0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f4a08034000 tx=0x7f4a0802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:15.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.112+0000 7f4a0f400640 1 -- 172.21.15.96:0/4194350822 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4a0005ad50 con 0x7f4a180b3690 2024-05-28T06:24:15.305 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.311+0000 7f4a1d800640 1 -- 172.21.15.96:0/4194350822 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4a1806e2f0 con 0x7f4a180b3690 2024-05-28T06:24:15.307 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.313+0000 7f4a0f400640 1 -- 172.21.15.96:0/4194350822 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v122) ==== 74+0+13132 (secure 0 0 0) 0x7f4a0005a650 con 0x7f4a180b3690 2024-05-28T06:24:15.308 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:24:15.308 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":122,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:24:13.864870+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.123:6836/3192559785":"2024-05-29T06:24:10.816651+0000","172.21.15.123:6834/3192559785":"2024-05-29T06:24:10.816651+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":13,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:24:15.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.316+0000 7f4a1d800640 1 -- 172.21.15.96:0/4194350822 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f49f406e510 msgr2=0x7f49f40709d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:15.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.316+0000 7f4a1d800640 1 --2- 172.21.15.96:0/4194350822 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f49f406e510 0x7f49f40709d0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f4a08034000 tx=0x7f4a0802e000 comp rx=0 tx=0).stop 2024-05-28T06:24:15.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.316+0000 7f4a1d800640 1 -- 172.21.15.96:0/4194350822 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4a180b3690 msgr2=0x7f4a18099850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:15.310 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.316+0000 7f4a1d800640 1 --2- 172.21.15.96:0/4194350822 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4a180b3690 0x7f4a18099850 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 crypto rx=0x7f4a0000e9e0 tx=0x7f4a0000eea0 comp rx=0 tx=0).stop 2024-05-28T06:24:15.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.316+0000 7f4a1d800640 1 -- 172.21.15.96:0/4194350822 shutdown_connections 2024-05-28T06:24:15.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.316+0000 7f4a1d800640 1 --2- 172.21.15.96:0/4194350822 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f49f406e510 0x7f49f40709d0 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:15.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.316+0000 7f4a1d800640 1 --2- 172.21.15.96:0/4194350822 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4a180d09e0 0x7f4a180ccfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:15.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.316+0000 7f4a1d800640 1 --2- 172.21.15.96:0/4194350822 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4a180b3690 0x7f4a18099850 unknown :-1 s=CLOSED pgs=1968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:15.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.316+0000 7f4a1d800640 1 --2- 172.21.15.96:0/4194350822 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4a1806f860 0x7f4a180c9280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:15.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.316+0000 7f4a1d800640 1 -- 172.21.15.96:0/4194350822 >> 172.21.15.96:0/4194350822 conn(0x7f4a1806bae0 msgr2=0x7f4a180b0cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:15.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.316+0000 7f4a1d800640 1 -- 172.21.15.96:0/4194350822 shutdown_connections 2024-05-28T06:24:15.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.316+0000 7f4a1d800640 1 -- 172.21.15.96:0/4194350822 wait complete. 2024-05-28T06:24:15.332 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-28T06:03:59.517102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-28T06:24:15.332 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-28T06:24:15.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.537+0000 7f304ba00640 1 Processor -- start 2024-05-28T06:24:15.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.537+0000 7f304ba00640 1 -- start start 2024-05-28T06:24:15.536 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.539+0000 7f304ba00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f30440d0640 0x7f304415d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:15.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.539+0000 7f304ba00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f30440b32c0 0x7f30440b3750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:15.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.539+0000 7f304ba00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f30440d0060 con 0x7f30440d0a20 2024-05-28T06:24:15.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.539+0000 7f304ba00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f30440adbb0 con 0x7f30440b32c0 2024-05-28T06:24:15.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.539+0000 7f304ba00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f304415db10 con 0x7f30440d0640 2024-05-28T06:24:15.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.540+0000 7f3049a00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f30440b32c0 0x7f30440b3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:15.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.540+0000 7f3049a00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f30440b32c0 0x7f30440b3750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.96:52072/0 (socket says 172.21.15.96:52072) 2024-05-28T06:24:15.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.540+0000 7f3049a00640 1 -- 172.21.15.96:0/2157567309 learned_addr learned my addr 172.21.15.96:0/2157567309 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:24:15.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.540+0000 7f3049000640 1 --2- 172.21.15.96:0/2157567309 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f30440d0640 0x7f304415d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:15.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.541+0000 7f3049a00640 1 -- 172.21.15.96:0/2157567309 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f30440d0640 msgr2=0x7f304415d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:15.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.541+0000 7f3049a00640 1 --2- 172.21.15.96:0/2157567309 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f30440d0640 0x7f304415d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:15.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.541+0000 7f3049a00640 1 -- 172.21.15.96:0/2157567309 >> v1:172.21.15.123:6789/0 conn(0x7f30440d0a20 legacy=0x7f30440cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:15.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.541+0000 7f3049a00640 1 -- 172.21.15.96:0/2157567309 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f304415cb10 con 0x7f30440b32c0 2024-05-28T06:24:15.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.542+0000 7f3049a00640 1 --2- 172.21.15.96:0/2157567309 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f30440b32c0 0x7f30440b3750 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f303c009e50 tx=0x7f303c02a710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4888f7b70e5b90e6 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:15.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.542+0000 7f3043e00640 1 -- 172.21.15.96:0/2157567309 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f303c036070 con 0x7f30440b32c0 2024-05-28T06:24:15.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.542+0000 7f3043e00640 1 -- 172.21.15.96:0/2157567309 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f303c031600 con 0x7f30440b32c0 2024-05-28T06:24:15.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.542+0000 7f3043e00640 1 -- 172.21.15.96:0/2157567309 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f303c0319d0 con 0x7f30440b32c0 2024-05-28T06:24:15.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.543+0000 7f304ba00640 1 -- 172.21.15.96:0/2157567309 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f30440b32c0 msgr2=0x7f30440b3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:15.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.543+0000 7f304ba00640 1 --2- 172.21.15.96:0/2157567309 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f30440b32c0 0x7f30440b3750 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f303c009e50 tx=0x7f303c02a710 comp rx=0 tx=0).stop 2024-05-28T06:24:15.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.543+0000 7f304ba00640 1 -- 172.21.15.96:0/2157567309 shutdown_connections 2024-05-28T06:24:15.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.543+0000 7f304ba00640 1 --2- 172.21.15.96:0/2157567309 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f30440b32c0 0x7f30440b3750 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:15.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.543+0000 7f304ba00640 1 --2- 172.21.15.96:0/2157567309 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f30440d0640 0x7f304415d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:15.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.543+0000 7f304ba00640 1 -- 172.21.15.96:0/2157567309 >> 172.21.15.96:0/2157567309 conn(0x7f304406bd20 msgr2=0x7f304406c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:15.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.543+0000 7f304ba00640 1 -- 172.21.15.96:0/2157567309 shutdown_connections 2024-05-28T06:24:15.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.543+0000 7f304ba00640 1 -- 172.21.15.96:0/2157567309 wait complete. 2024-05-28T06:24:15.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.546+0000 7f304ba00640 1 Processor -- start 2024-05-28T06:24:15.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.546+0000 7f304ba00640 1 -- start start 2024-05-28T06:24:15.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.547+0000 7f304ba00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f30440b32c0 0x7f304414d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:15.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.547+0000 7f304ba00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f30440d0640 0x7f304414e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:15.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.547+0000 7f304ba00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f30440d0a20 0x7f30440b1630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:15.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.547+0000 7f304ba00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f304415cdd0 con 0x7f30440d0a20 2024-05-28T06:24:15.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.547+0000 7f304ba00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f30440ad8d0 con 0x7f30440d0640 2024-05-28T06:24:15.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.547+0000 7f304ba00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f304415d0b0 con 0x7f30440b32c0 2024-05-28T06:24:15.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.548+0000 7f3049a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f30440d0a20 0x7f30440b1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:15.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.548+0000 7f3049a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f30440d0a20 0x7f30440b1630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:43676/0 (socket says 172.21.15.96:43676) 2024-05-28T06:24:15.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.548+0000 7f3049a00640 1 -- 172.21.15.96:0/875314019 learned_addr learned my addr 172.21.15.96:0/875314019 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:24:15.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.548+0000 7f3049000640 1 --2- 172.21.15.96:0/875314019 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f30440b32c0 0x7f304414d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:15.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.548+0000 7f303be00640 1 --2- 172.21.15.96:0/875314019 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f30440d0640 0x7f304414e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:15.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.548+0000 7f3049a00640 1 -- 172.21.15.96:0/875314019 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f30440b32c0 msgr2=0x7f304414d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:15.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.548+0000 7f3049a00640 1 --2- 172.21.15.96:0/875314019 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f30440b32c0 0x7f304414d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:15.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.548+0000 7f3049a00640 1 -- 172.21.15.96:0/875314019 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f30440d0640 msgr2=0x7f304414e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:15.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.548+0000 7f3049a00640 1 --2- 172.21.15.96:0/875314019 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f30440d0640 0x7f304414e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:15.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.548+0000 7f3049a00640 1 -- 172.21.15.96:0/875314019 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f304414e8e0 con 0x7f30440d0a20 2024-05-28T06:24:15.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.548+0000 7f303be00640 1 --2- 172.21.15.96:0/875314019 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f30440d0640 0x7f304414e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-28T06:24:15.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.548+0000 7f3049000640 1 --2- 172.21.15.96:0/875314019 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f30440b32c0 0x7f304414d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-28T06:24:15.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.549+0000 7f3049a00640 1 --2- 172.21.15.96:0/875314019 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f30440d0a20 0x7f30440b1630 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7f303c009e50 tx=0x7f303c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:15.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.549+0000 7f3041600640 1 -- 172.21.15.96:0/875314019 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f303c036050 con 0x7f30440d0a20 2024-05-28T06:24:15.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.549+0000 7f3041600640 1 -- 172.21.15.96:0/875314019 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f303c042070 con 0x7f30440d0a20 2024-05-28T06:24:15.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.549+0000 7f3041600640 1 -- 172.21.15.96:0/875314019 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f303c009510 con 0x7f30440d0a20 2024-05-28T06:24:15.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.549+0000 7f304ba00640 1 -- 172.21.15.96:0/875314019 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f304414db00 con 0x7f30440d0a20 2024-05-28T06:24:15.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.549+0000 7f304ba00640 1 -- 172.21.15.96:0/875314019 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f304414e0c0 con 0x7f30440d0a20 2024-05-28T06:24:15.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.550+0000 7f304ba00640 1 -- 172.21.15.96:0/875314019 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30440689e0 con 0x7f30440d0a20 2024-05-28T06:24:15.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.551+0000 7f3041600640 1 -- 172.21.15.96:0/875314019 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 42) ==== 93590+0+0 (secure 0 0 0) 0x7f303c007bf0 con 0x7f30440d0a20 2024-05-28T06:24:15.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.553+0000 7f3041600640 1 --2- 172.21.15.96:0/875314019 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f302006e510 0x7f30200709d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:15.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.553+0000 7f3041600640 1 -- 172.21.15.96:0/875314019 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 4999+0+0 (secure 0 0 0) 0x7f303c0ab820 con 0x7f30440d0a20 2024-05-28T06:24:15.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.553+0000 7f3049000640 1 --2- 172.21.15.96:0/875314019 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f302006e510 0x7f30200709d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:15.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.555+0000 7f3049000640 1 --2- 172.21.15.96:0/875314019 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f302006e510 0x7f30200709d0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f3034009b90 tx=0x7f3034008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:15.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.557+0000 7f3041600640 1 -- 172.21.15.96:0/875314019 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f303c07b750 con 0x7f30440d0a20 2024-05-28T06:24:15.751 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.755+0000 7f304ba00640 1 -- 172.21.15.96:0/875314019 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f304414eb40 con 0x7f30440d0a20 2024-05-28T06:24:15.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.756+0000 7f3041600640 1 -- 172.21.15.96:0/875314019 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v122) ==== 93+0+10 (secure 0 0 0) 0x7f303c07b050 con 0x7f30440d0a20 2024-05-28T06:24:15.752 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 1 2024-05-28T06:24:15.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.759+0000 7f304ba00640 1 -- 172.21.15.96:0/875314019 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f302006e510 msgr2=0x7f30200709d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:15.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.759+0000 7f304ba00640 1 --2- 172.21.15.96:0/875314019 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f302006e510 0x7f30200709d0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f3034009b90 tx=0x7f3034008040 comp rx=0 tx=0).stop 2024-05-28T06:24:15.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.759+0000 7f304ba00640 1 -- 172.21.15.96:0/875314019 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f30440d0a20 msgr2=0x7f30440b1630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:15.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.759+0000 7f304ba00640 1 --2- 172.21.15.96:0/875314019 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f30440d0a20 0x7f30440b1630 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7f303c009e50 tx=0x7f303c004a40 comp rx=0 tx=0).stop 2024-05-28T06:24:15.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.759+0000 7f304ba00640 1 -- 172.21.15.96:0/875314019 shutdown_connections 2024-05-28T06:24:15.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.759+0000 7f304ba00640 1 --2- 172.21.15.96:0/875314019 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f302006e510 0x7f30200709d0 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:15.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.759+0000 7f304ba00640 1 --2- 172.21.15.96:0/875314019 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f30440d0a20 0x7f30440b1630 unknown :-1 s=CLOSED pgs=1969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:15.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.759+0000 7f304ba00640 1 --2- 172.21.15.96:0/875314019 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f30440d0640 0x7f304414e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:15.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.759+0000 7f304ba00640 1 --2- 172.21.15.96:0/875314019 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f30440b32c0 0x7f304414d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:15.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.759+0000 7f304ba00640 1 -- 172.21.15.96:0/875314019 >> 172.21.15.96:0/875314019 conn(0x7f304406bd20 msgr2=0x7f304415bad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:15.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.759+0000 7f304ba00640 1 -- 172.21.15.96:0/875314019 shutdown_connections 2024-05-28T06:24:15.755 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.759+0000 7f304ba00640 1 -- 172.21.15.96:0/875314019 wait complete. 2024-05-28T06:24:15.777 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestCloneProgressReporter.test_when_7_clones_are_cancelled' 2024-05-28T06:24:15.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.963+0000 7f7df1e00640 1 Processor -- start 2024-05-28T06:24:15.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.964+0000 7f7df1e00640 1 -- start start 2024-05-28T06:24:15.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.965+0000 7f7df1e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f7dec0d04a0 0x7f7dec0d0930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:15.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.965+0000 7f7df1e00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7dec06ed10 con 0x7f7dec0ad130 2024-05-28T06:24:15.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.965+0000 7f7df1e00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7dec15cb00 con 0x7f7dec06f860 2024-05-28T06:24:15.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.965+0000 7f7df1e00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f7dec0b3cc0 con 0x7f7dec0d04a0 2024-05-28T06:24:15.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.965+0000 7f7deb400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f7dec0ad130 0x7f7dec0ad540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:57566/0 (socket says 172.21.15.96:57566) 2024-05-28T06:24:15.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.965+0000 7f7deb400640 1 -- 172.21.15.96:0/823530101 learned_addr learned my addr 172.21.15.96:0/823530101 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:24:15.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.966+0000 7f7dea000640 1 --2- 172.21.15.96:0/823530101 >> v2:172.21.15.123:3300/0 conn(0x7f7dec0d04a0 0x7f7dec0d0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:15.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.966+0000 7f7de9600640 1 -- 172.21.15.96:0/823530101 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2376889254 0 0) 0x7f7de4001490 con 0x7f7dec0ad130 2024-05-28T06:24:15.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.966+0000 7f7de9600640 1 -- 172.21.15.96:0/823530101 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7dd0003520 con 0x7f7dec0ad130 2024-05-28T06:24:15.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.966+0000 7f7de9600640 1 -- 172.21.15.96:0/823530101 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4170642253 0 0) 0x7f7dd0003520 con 0x7f7dec0ad130 2024-05-28T06:24:15.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.966+0000 7f7de9600640 1 -- 172.21.15.96:0/823530101 >> v2:172.21.15.123:3300/0 conn(0x7f7dec0d04a0 msgr2=0x7f7dec0d0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:15.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.966+0000 7f7de9600640 1 --2- 172.21.15.96:0/823530101 >> v2:172.21.15.123:3300/0 conn(0x7f7dec0d04a0 0x7f7dec0d0930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:15.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.966+0000 7f7de9600640 1 -- 172.21.15.96:0/823530101 >> v1:172.21.15.123:6789/0 conn(0x7f7dec06f860 legacy=0x7f7dec0b3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:15.962 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.966+0000 7f7de9600640 1 -- 172.21.15.96:0/823530101 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7dec0b3fa0 con 0x7f7dec0ad130 2024-05-28T06:24:15.962 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.966+0000 7f7de9600640 1 -- 172.21.15.96:0/823530101 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f7de4001bc0 con 0x7f7dec0ad130 2024-05-28T06:24:15.962 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.966+0000 7f7de9600640 1 -- 172.21.15.96:0/823530101 >> v1:172.21.15.96:6789/0 conn(0x7f7dec0ad130 legacy=0x7f7dec0ad540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:15.962 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.966+0000 7f7de9600640 1 --2- 172.21.15.96:0/823530101 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7dd00039b0 0x7f7dd0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:15.962 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.966+0000 7f7de9600640 1 --2- 172.21.15.96:0/823530101 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7dd0004800 0x7f7dd0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:15.962 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.966+0000 7f7de9600640 1 --2- 172.21.15.96:0/823530101 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7dd00051c0 0x7f7dd000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:15.962 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.966+0000 7f7de9600640 1 -- 172.21.15.96:0/823530101 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f7dec0b3cc0 con 0x7f7dd0004800 2024-05-28T06:24:15.962 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.966+0000 7f7de9600640 1 -- 172.21.15.96:0/823530101 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f7dd000c060 con 0x7f7dd00051c0 2024-05-28T06:24:15.962 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.966+0000 7f7de9600640 1 -- 172.21.15.96:0/823530101 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f7dd000c260 con 0x7f7dd00039b0 2024-05-28T06:24:15.962 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.967+0000 7f7dea000640 1 --2- 172.21.15.96:0/823530101 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7dd0004800 0x7f7dd0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:15.962 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.967+0000 7f7dea000640 1 -- 172.21.15.96:0/823530101 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7dd00039b0 msgr2=0x7f7dd0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:15.962 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.967+0000 7f7dea000640 1 --2- 172.21.15.96:0/823530101 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7dd00039b0 0x7f7dd0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:15.962 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.967+0000 7f7dea000640 1 -- 172.21.15.96:0/823530101 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7dd00051c0 msgr2=0x7f7dd000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:15.963 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.967+0000 7f7dea000640 1 --2- 172.21.15.96:0/823530101 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7dd00051c0 0x7f7dd000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:15.963 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.967+0000 7f7deb400640 1 -- 172.21.15.96:0/823530101 reap_dead start 2024-05-28T06:24:15.963 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.967+0000 7f7dea000640 1 -- 172.21.15.96:0/823530101 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7dd0003520 con 0x7f7dd0004800 2024-05-28T06:24:15.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.968+0000 7f7dea000640 1 --2- 172.21.15.96:0/823530101 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7dd0004800 0x7f7dd0004c80 secure :-1 s=READY pgs=1971 cs=0 l=1 rev1=1 crypto rx=0x7f7dd4009ac0 tx=0x7f7dd0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2350515e4f3b30 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:15.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.968+0000 7f7de9600640 1 -- 172.21.15.96:0/823530101 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dd4036070 con 0x7f7dd0004800 2024-05-28T06:24:15.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.968+0000 7f7de9600640 1 -- 172.21.15.96:0/823530101 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7dd4031400 con 0x7f7dd0004800 2024-05-28T06:24:15.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.968+0000 7f7de9600640 1 -- 172.21.15.96:0/823530101 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dd4009690 con 0x7f7dd0004800 2024-05-28T06:24:15.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.969+0000 7f7df1e00640 1 -- 172.21.15.96:0/823530101 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7dd0004800 msgr2=0x7f7dd0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:15.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.969+0000 7f7df1e00640 1 --2- 172.21.15.96:0/823530101 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7dd0004800 0x7f7dd0004c80 secure :-1 s=READY pgs=1971 cs=0 l=1 rev1=1 crypto rx=0x7f7dd4009ac0 tx=0x7f7dd0005830 comp rx=0 tx=0).stop 2024-05-28T06:24:15.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.969+0000 7f7df1e00640 1 -- 172.21.15.96:0/823530101 shutdown_connections 2024-05-28T06:24:15.966 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.969+0000 7f7df1e00640 1 --2- 172.21.15.96:0/823530101 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7dd0004800 0x7f7dd0004c80 unknown :-1 s=CLOSED pgs=1971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:15.966 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.969+0000 7f7df1e00640 1 -- 172.21.15.96:0/823530101 >> 172.21.15.96:0/823530101 conn(0x7f7dec06bae0 msgr2=0x7f7dec06bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:15.966 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.969+0000 7f7df1e00640 1 -- 172.21.15.96:0/823530101 shutdown_connections 2024-05-28T06:24:15.966 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.969+0000 7f7df1e00640 1 -- 172.21.15.96:0/823530101 wait complete. 2024-05-28T06:24:15.966 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.971+0000 7f7df1e00640 1 Processor -- start 2024-05-28T06:24:15.966 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.971+0000 7f7df1e00640 1 -- start start 2024-05-28T06:24:15.967 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.972+0000 7f7df1e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7dd0004800 0x7f7dec0cf2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:15.967 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.972+0000 7f7df1e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7dec15ce50 0x7f7dec0733a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:15.968 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.972+0000 7f7df1e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7dec0738e0 0x7f7dec0cde30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:15.968 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.972+0000 7f7df1e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f7dec0b4840 con 0x7f7dd0004800 2024-05-28T06:24:15.968 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.972+0000 7f7df1e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f7dec0b4560 con 0x7f7dec15ce50 2024-05-28T06:24:15.968 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.972+0000 7f7df1e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f7dec0d0930 con 0x7f7dec0738e0 2024-05-28T06:24:15.968 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.972+0000 7f7deaa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7dd0004800 0x7f7dec0cf2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:15.968 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.972+0000 7f7deaa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7dd0004800 0x7f7dec0cf2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:43684/0 (socket says 172.21.15.96:43684) 2024-05-28T06:24:15.968 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.972+0000 7f7deaa00640 1 -- 172.21.15.96:0/1898369898 learned_addr learned my addr 172.21.15.96:0/1898369898 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:24:15.968 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.973+0000 7f7deb400640 1 --2- 172.21.15.96:0/1898369898 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7dec0738e0 0x7f7dec0cde30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:15.968 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.973+0000 7f7dea000640 1 --2- 172.21.15.96:0/1898369898 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7dec15ce50 0x7f7dec0733a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:15.968 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.973+0000 7f7deaa00640 1 -- 172.21.15.96:0/1898369898 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7dec0738e0 msgr2=0x7f7dec0cde30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:15.968 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.973+0000 7f7deaa00640 1 --2- 172.21.15.96:0/1898369898 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7dec0738e0 0x7f7dec0cde30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:15.968 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.973+0000 7f7deaa00640 1 -- 172.21.15.96:0/1898369898 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7dec15ce50 msgr2=0x7f7dec0733a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:15.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.973+0000 7f7deaa00640 1 --2- 172.21.15.96:0/1898369898 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7dec15ce50 0x7f7dec0733a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:15.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.973+0000 7f7deaa00640 1 -- 172.21.15.96:0/1898369898 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7dec0c4cb0 con 0x7f7dd0004800 2024-05-28T06:24:15.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.973+0000 7f7dea000640 1 --2- 172.21.15.96:0/1898369898 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7dec15ce50 0x7f7dec0733a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:24:15.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.973+0000 7f7deaa00640 1 --2- 172.21.15.96:0/1898369898 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7dd0004800 0x7f7dec0cf2e0 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7f7ddc00c3c0 tx=0x7f7ddc00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:15.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.973+0000 7f7de3400640 1 -- 172.21.15.96:0/1898369898 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ddc0052d0 con 0x7f7dd0004800 2024-05-28T06:24:15.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.973+0000 7f7de3400640 1 -- 172.21.15.96:0/1898369898 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ddc005490 con 0x7f7dd0004800 2024-05-28T06:24:15.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.973+0000 7f7de3400640 1 -- 172.21.15.96:0/1898369898 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ddc007260 con 0x7f7dd0004800 2024-05-28T06:24:15.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.973+0000 7f7df1e00640 1 -- 172.21.15.96:0/1898369898 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7dec0c4f90 con 0x7f7dd0004800 2024-05-28T06:24:15.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.973+0000 7f7df1e00640 1 -- 172.21.15.96:0/1898369898 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7dec15d2d0 con 0x7f7dd0004800 2024-05-28T06:24:15.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.974+0000 7f7df1e00640 1 -- 172.21.15.96:0/1898369898 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7dec0689e0 con 0x7f7dd0004800 2024-05-28T06:24:15.970 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.976+0000 7f7de3400640 1 -- 172.21.15.96:0/1898369898 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 42) ==== 93590+0+0 (secure 0 0 0) 0x7f7ddc007400 con 0x7f7dd0004800 2024-05-28T06:24:15.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.977+0000 7f7de3400640 1 --2- 172.21.15.96:0/1898369898 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7dc806e540 0x7f7dc8070a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:15.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.977+0000 7f7de3400640 1 -- 172.21.15.96:0/1898369898 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 4999+0+0 (secure 0 0 0) 0x7f7ddc089fd0 con 0x7f7dd0004800 2024-05-28T06:24:15.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.978+0000 7f7dea000640 1 --2- 172.21.15.96:0/1898369898 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7dc806e540 0x7f7dc8070a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:15.973 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.979+0000 7f7dea000640 1 --2- 172.21.15.96:0/1898369898 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7dc806e540 0x7f7dc8070a00 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f7dd4033000 tx=0x7f7dd4008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:15.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:15.981+0000 7f7de3400640 1 -- 172.21.15.96:0/1898369898 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7ddc059e60 con 0x7f7dd0004800 2024-05-28T06:24:16.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:16.253+0000 7f7df1e00640 1 -- 172.21.15.96:0/1898369898 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgressReporter.test_when_7_clones_are_cancelled"]} v 0) -- 0x7f7dec073cc0 con 0x7f7dd0004800 2024-05-28T06:24:16.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:16.894+0000 7f7de3400640 1 -- 172.21.15.96:0/1898369898 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgressReporter.test_when_7_clones_are_cancelled"]}]=0 v878) ==== 166+0+0 (secure 0 0 0) 0x7f7ddc059760 con 0x7f7dd0004800 2024-05-28T06:24:16.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:16.898+0000 7f7df1e00640 1 -- 172.21.15.96:0/1898369898 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7dc806e540 msgr2=0x7f7dc8070a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:16.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:16.898+0000 7f7df1e00640 1 --2- 172.21.15.96:0/1898369898 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7dc806e540 0x7f7dc8070a00 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f7dd4033000 tx=0x7f7dd4008030 comp rx=0 tx=0).stop 2024-05-28T06:24:16.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:16.898+0000 7f7df1e00640 1 -- 172.21.15.96:0/1898369898 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7dd0004800 msgr2=0x7f7dec0cf2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:16.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:16.898+0000 7f7df1e00640 1 --2- 172.21.15.96:0/1898369898 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7dd0004800 0x7f7dec0cf2e0 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7f7ddc00c3c0 tx=0x7f7ddc00c880 comp rx=0 tx=0).stop 2024-05-28T06:24:16.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:16.898+0000 7f7df1e00640 1 -- 172.21.15.96:0/1898369898 shutdown_connections 2024-05-28T06:24:16.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:16.898+0000 7f7df1e00640 1 --2- 172.21.15.96:0/1898369898 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7dc806e540 0x7f7dc8070a00 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:16.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:16.898+0000 7f7df1e00640 1 --2- 172.21.15.96:0/1898369898 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7dec0738e0 0x7f7dec0cde30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:16.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:16.898+0000 7f7df1e00640 1 --2- 172.21.15.96:0/1898369898 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7dec15ce50 0x7f7dec0733a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:16.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:16.898+0000 7f7df1e00640 1 --2- 172.21.15.96:0/1898369898 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7dd0004800 0x7f7dec0cf2e0 unknown :-1 s=CLOSED pgs=1972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:16.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:16.898+0000 7f7df1e00640 1 -- 172.21.15.96:0/1898369898 >> 172.21.15.96:0/1898369898 conn(0x7f7dec06bae0 msgr2=0x7f7dec0b01e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:16.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:16.898+0000 7f7df1e00640 1 -- 172.21.15.96:0/1898369898 shutdown_connections 2024-05-28T06:24:16.892 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:16.898+0000 7f7df1e00640 1 -- 172.21.15.96:0/1898369898 wait complete. 2024-05-28T06:24:16.913 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-05-28T06:24:17.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:17.118+0000 7f9616a00640 1 Processor -- start 2024-05-28T06:24:17.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:17.119+0000 7f9616a00640 1 -- start start 2024-05-28T06:24:17.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:17.120+0000 7f9616a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f9610153510 0x7f96100b4d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:17.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:17.120+0000 7f9616a00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f961015d270 0x7f96100ccff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:17.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:17.120+0000 7f9616a00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f961015d900 con 0x7f96100ccb90 2024-05-28T06:24:17.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:17.120+0000 7f9616a00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f96101538f0 con 0x7f9610153510 2024-05-28T06:24:17.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:17.120+0000 7f9616a00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f96100cd530 con 0x7f961015d270 2024-05-28T06:24:17.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:17.120+0000 7f9615600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f9610153510 0x7f96100b4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:17.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:17.121+0000 7f9615600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f9610153510 0x7f96100b4d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:43686/0 (socket says 172.21.15.96:43686) 2024-05-28T06:24:17.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:17.121+0000 7f9615600640 1 -- 172.21.15.96:0/2582104786 learned_addr learned my addr 172.21.15.96:0/2582104786 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:24:17.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:17.121+0000 7f9616000640 1 --2- 172.21.15.96:0/2582104786 >> v2:172.21.15.123:3300/0 conn(0x7f961015d270 0x7f96100ccff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:17.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:17.121+0000 7f9615600640 1 -- 172.21.15.96:0/2582104786 >> v2:172.21.15.123:3300/0 conn(0x7f961015d270 msgr2=0x7f96100ccff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:17.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:17.121+0000 7f9615600640 1 --2- 172.21.15.96:0/2582104786 >> v2:172.21.15.123:3300/0 conn(0x7f961015d270 0x7f96100ccff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:17.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:17.121+0000 7f9615600640 1 -- 172.21.15.96:0/2582104786 >> v1:172.21.15.123:6789/0 conn(0x7f96100ccb90 legacy=0x7f961015cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:17.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:17.121+0000 7f9615600640 1 -- 172.21.15.96:0/2582104786 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96100b3f90 con 0x7f9610153510 2024-05-28T06:24:17.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:17.122+0000 7f9615600640 1 --2- 172.21.15.96:0/2582104786 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9610153510 0x7f96100b4d50 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7f96100cdce0 tx=0x7f960402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f04d1470e7d8da5 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:17.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:17.122+0000 7f960fe00640 1 -- 172.21.15.96:0/2582104786 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9604037070 con 0x7f9610153510 2024-05-28T06:24:17.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:17.122+0000 7f960fe00640 1 -- 172.21.15.96:0/2582104786 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9604032440 con 0x7f9610153510 2024-05-28T06:24:17.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:17.122+0000 7f960fe00640 1 -- 172.21.15.96:0/2582104786 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9604032810 con 0x7f9610153510 2024-05-28T06:24:17.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:17.123+0000 7f9616a00640 1 -- 172.21.15.96:0/2582104786 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9610153510 msgr2=0x7f96100b4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:17.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:17.123+0000 7f9616a00640 1 --2- 172.21.15.96:0/2582104786 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9610153510 0x7f96100b4d50 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7f96100cdce0 tx=0x7f960402a1a0 comp rx=0 tx=0).stop 2024-05-28T06:24:17.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:17.123+0000 7f9616a00640 1 -- 172.21.15.96:0/2582104786 shutdown_connections 2024-05-28T06:24:17.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:17.123+0000 7f9616a00640 1 --2- 172.21.15.96:0/2582104786 >> v2:172.21.15.123:3300/0 conn(0x7f961015d270 0x7f96100ccff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:17.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:17.123+0000 7f9616a00640 1 --2- 172.21.15.96:0/2582104786 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9610153510 0x7f96100b4d50 unknown :-1 s=CLOSED pgs=1973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:17.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:17.123+0000 7f9616a00640 1 -- 172.21.15.96:0/2582104786 >> 172.21.15.96:0/2582104786 conn(0x7f961006b900 msgr2=0x7f961006bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:17.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:17.123+0000 7f9616a00640 1 -- 172.21.15.96:0/2582104786 shutdown_connections 2024-05-28T06:24:17.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:17.123+0000 7f9616a00640 1 -- 172.21.15.96:0/2582104786 wait complete. 2024-05-28T06:24:17.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:17.125+0000 7f9616a00640 1 Processor -- start 2024-05-28T06:24:17.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:17.125+0000 7f9616a00640 1 -- start start 2024-05-28T06:24:17.120 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:17.126+0000 7f9616a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f96100ccb90 0x7f96100cf910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:17.120 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:17.126+0000 7f9616a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9610153510 0x7f96101433f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:17.120 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:17.126+0000 7f9616a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f961015d270 0x7f96100addb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:17.120 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:17.126+0000 7f9616a00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f961015d6a0 con 0x7f9610153510 2024-05-28T06:24:17.120 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:17.126+0000 7f9616a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f96100b48c0 con 0x7f961015d270 2024-05-28T06:24:17.120 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:17.126+0000 7f9616a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f96100b4550 con 0x7f96100ccb90 2024-05-28T06:24:17.120 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:17.126+0000 7f9614c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9610153510 0x7f96101433f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:17.120 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:17.126+0000 7f9614c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9610153510 0x7f96101433f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:43688/0 (socket says 172.21.15.96:43688) 2024-05-28T06:24:17.120 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:17.126+0000 7f9614c00640 1 -- 172.21.15.96:0/2334785200 learned_addr learned my addr 172.21.15.96:0/2334785200 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:24:17.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:17.127+0000 7f9614c00640 1 -- 172.21.15.96:0/2334785200 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f96100ccb90 msgr2=0x7f96100cf910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:17.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:17.127+0000 7f9614c00640 1 --2- 172.21.15.96:0/2334785200 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f96100ccb90 0x7f96100cf910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:17.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:17.127+0000 7f9614c00640 1 -- 172.21.15.96:0/2334785200 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f961015d270 msgr2=0x7f96100addb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:17.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:17.127+0000 7f9614c00640 1 --2- 172.21.15.96:0/2334785200 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f961015d270 0x7f96100addb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:17.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:17.127+0000 7f9614c00640 1 -- 172.21.15.96:0/2334785200 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96100ce4a0 con 0x7f9610153510 2024-05-28T06:24:17.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:17.127+0000 7f9614c00640 1 --2- 172.21.15.96:0/2334785200 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9610153510 0x7f96101433f0 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 crypto rx=0x7f960000b3e0 tx=0x7f960000b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:17.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:17.127+0000 7f960d600640 1 -- 172.21.15.96:0/2334785200 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9600015020 con 0x7f9610153510 2024-05-28T06:24:17.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:17.127+0000 7f960d600640 1 -- 172.21.15.96:0/2334785200 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9600011850 con 0x7f9610153510 2024-05-28T06:24:17.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:17.127+0000 7f9616a00640 1 -- 172.21.15.96:0/2334785200 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96100ce1c0 con 0x7f9610153510 2024-05-28T06:24:17.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:17.127+0000 7f960d600640 1 -- 172.21.15.96:0/2334785200 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9600011c40 con 0x7f9610153510 2024-05-28T06:24:17.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:17.127+0000 7f9616a00640 1 -- 172.21.15.96:0/2334785200 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96100ad5b0 con 0x7f9610153510 2024-05-28T06:24:17.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:17.129+0000 7f9616a00640 1 -- 172.21.15.96:0/2334785200 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96100689e0 con 0x7f9610153510 2024-05-28T06:24:17.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:17.130+0000 7f960d600640 1 -- 172.21.15.96:0/2334785200 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 42) ==== 93590+0+0 (secure 0 0 0) 0x7f9600018020 con 0x7f9610153510 2024-05-28T06:24:17.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:17.131+0000 7f960d600640 1 --2- 172.21.15.96:0/2334785200 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f95f406e540 0x7f95f4070a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:17.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:17.131+0000 7f960d600640 1 -- 172.21.15.96:0/2334785200 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 4999+0+0 (secure 0 0 0) 0x7f960008a990 con 0x7f9610153510 2024-05-28T06:24:17.125 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:17.131+0000 7f9615600640 1 --2- 172.21.15.96:0/2334785200 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f95f406e540 0x7f95f4070a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:17.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:17.133+0000 7f9615600640 1 --2- 172.21.15.96:0/2334785200 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f95f406e540 0x7f95f4070a00 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f9604034000 tx=0x7f960402c000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:17.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:17.135+0000 7f960d600640 1 -- 172.21.15.96:0/2334785200 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f960001f020 con 0x7f9610153510 2024-05-28T06:24:17.328 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:17.335+0000 7f9616a00640 1 -- 172.21.15.96:0/2334785200 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f96100b3c40 con 0x7f9610153510 2024-05-28T06:24:17.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:17.341+0000 7f960d600640 1 -- 172.21.15.96:0/2334785200 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v28)=0 v28) ==== 122+0+0 (secure 0 0 0) 0x7f96000172c0 con 0x7f9610153510 2024-05-28T06:24:17.337 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:17.343+0000 7f9616a00640 1 -- 172.21.15.96:0/2334785200 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f95f406e540 msgr2=0x7f95f4070a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:17.337 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:17.343+0000 7f9616a00640 1 --2- 172.21.15.96:0/2334785200 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f95f406e540 0x7f95f4070a00 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f9604034000 tx=0x7f960402c000 comp rx=0 tx=0).stop 2024-05-28T06:24:17.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:17.343+0000 7f9616a00640 1 -- 172.21.15.96:0/2334785200 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9610153510 msgr2=0x7f96101433f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:17.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:17.343+0000 7f9616a00640 1 --2- 172.21.15.96:0/2334785200 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9610153510 0x7f96101433f0 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 crypto rx=0x7f960000b3e0 tx=0x7f960000b8a0 comp rx=0 tx=0).stop 2024-05-28T06:24:17.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:17.344+0000 7f9616a00640 1 -- 172.21.15.96:0/2334785200 shutdown_connections 2024-05-28T06:24:17.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:17.344+0000 7f9616a00640 1 --2- 172.21.15.96:0/2334785200 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f95f406e540 0x7f95f4070a00 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:17.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:17.344+0000 7f9616a00640 1 --2- 172.21.15.96:0/2334785200 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f961015d270 0x7f96100addb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:17.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:17.344+0000 7f9616a00640 1 --2- 172.21.15.96:0/2334785200 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9610153510 0x7f96101433f0 unknown :-1 s=CLOSED pgs=1974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:17.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:17.344+0000 7f9616a00640 1 --2- 172.21.15.96:0/2334785200 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f96100ccb90 0x7f96100cf910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:17.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:17.344+0000 7f9616a00640 1 -- 172.21.15.96:0/2334785200 >> 172.21.15.96:0/2334785200 conn(0x7f961006b900 msgr2=0x7f961006f6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:17.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:17.344+0000 7f9616a00640 1 -- 172.21.15.96:0/2334785200 shutdown_connections 2024-05-28T06:24:17.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:17.344+0000 7f9616a00640 1 -- 172.21.15.96:0/2334785200 wait complete. 2024-05-28T06:24:17.358 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-05-28T06:24:17.359 DEBUG:teuthology.orchestra.run.smithi096:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-05-28T06:24:17.460 DEBUG:teuthology.orchestra.run.smithi123:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-05-28T06:24:17.576 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-05-28T06:24:17.576 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:24:17.576 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:24:17.636 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052731,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248996k,nr_inodes=812249,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-28T06:24:17.636 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-05-28T06:24:17.636 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:24:17.691 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052731,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248996k,nr_inodes=812249,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-05-28T06:24:17.692 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:24:17.692 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:24:17.752 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052731,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248996k,nr_inodes=812249,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-28T06:24:17.752 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-05-28T06:24:17.752 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:24:17.809 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052731,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248996k,nr_inodes=812249,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-05-28T06:24:17.809 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-05-28T06:24:17.809 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-28T06:24:18.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.023+0000 7f8a55a00640 1 Processor -- start 2024-05-28T06:24:18.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.024+0000 7f8a55a00640 1 -- start start 2024-05-28T06:24:18.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.025+0000 7f8a55a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8a500d24d0 0x7f8a500d52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:18.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.025+0000 7f8a55a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f8a500d1650 0x7f8a500d1aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:18.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.025+0000 7f8a55a00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a500d1fe0 con 0x7f8a5015e9d0 2024-05-28T06:24:18.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.025+0000 7f8a55a00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f8a5015edb0 con 0x7f8a500d1650 2024-05-28T06:24:18.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.025+0000 7f8a55a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f8a500d4320 con 0x7f8a500d24d0 2024-05-28T06:24:18.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.025+0000 7f8a4ea00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f8a5015e9d0 0x7f8a500d3c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:57572/0 (socket says 172.21.15.96:57572) 2024-05-28T06:24:18.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.025+0000 7f8a4ea00640 1 -- 172.21.15.96:0/1349008455 learned_addr learned my addr 172.21.15.96:0/1349008455 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:24:18.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.025+0000 7f8a4f400640 1 --2- 172.21.15.96:0/1349008455 >> v2:172.21.15.96:3300/0 conn(0x7f8a500d1650 0x7f8a500d1aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:18.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.026+0000 7f8a4e000640 1 --2- 172.21.15.96:0/1349008455 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8a500d24d0 0x7f8a500d52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:18.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.026+0000 7f8a4d600640 1 -- 172.21.15.96:0/1349008455 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4177573966 0 0) 0x7f8a500d1fe0 con 0x7f8a5015e9d0 2024-05-28T06:24:18.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.026+0000 7f8a4d600640 1 -- 172.21.15.96:0/1349008455 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8a34003520 con 0x7f8a5015e9d0 2024-05-28T06:24:18.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.026+0000 7f8a4f400640 1 -- 172.21.15.96:0/1349008455 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8a500d24d0 msgr2=0x7f8a500d52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:18.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.026+0000 7f8a4f400640 1 --2- 172.21.15.96:0/1349008455 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8a500d24d0 0x7f8a500d52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:18.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.026+0000 7f8a4f400640 1 -- 172.21.15.96:0/1349008455 >> v1:172.21.15.96:6789/0 conn(0x7f8a5015e9d0 legacy=0x7f8a500d3c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:18.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.026+0000 7f8a4f400640 1 -- 172.21.15.96:0/1349008455 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a500d2b70 con 0x7f8a500d1650 2024-05-28T06:24:18.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.027+0000 7f8a4f400640 1 --2- 172.21.15.96:0/1349008455 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8a500d1650 0x7f8a500d1aa0 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7f8a48009d90 tx=0x7f8a4802a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cfb0f89598639a0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:18.022 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.027+0000 7f8a4d600640 1 -- 172.21.15.96:0/1349008455 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a48031600 con 0x7f8a500d1650 2024-05-28T06:24:18.022 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.027+0000 7f8a4d600640 1 -- 172.21.15.96:0/1349008455 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a480317a0 con 0x7f8a500d1650 2024-05-28T06:24:18.022 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.027+0000 7f8a4d600640 1 -- 172.21.15.96:0/1349008455 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a48031b70 con 0x7f8a500d1650 2024-05-28T06:24:18.022 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.028+0000 7f8a55a00640 1 -- 172.21.15.96:0/1349008455 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8a500d1650 msgr2=0x7f8a500d1aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:18.022 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.028+0000 7f8a55a00640 1 --2- 172.21.15.96:0/1349008455 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8a500d1650 0x7f8a500d1aa0 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7f8a48009d90 tx=0x7f8a4802a5d0 comp rx=0 tx=0).stop 2024-05-28T06:24:18.022 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.028+0000 7f8a55a00640 1 -- 172.21.15.96:0/1349008455 shutdown_connections 2024-05-28T06:24:18.022 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.028+0000 7f8a55a00640 1 --2- 172.21.15.96:0/1349008455 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8a500d1650 0x7f8a500d1aa0 unknown :-1 s=CLOSED pgs=1976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:18.022 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.028+0000 7f8a55a00640 1 --2- 172.21.15.96:0/1349008455 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8a500d24d0 0x7f8a500d52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:18.022 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.028+0000 7f8a55a00640 1 -- 172.21.15.96:0/1349008455 >> 172.21.15.96:0/1349008455 conn(0x7f8a50067890 msgr2=0x7f8a50067ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:18.022 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.028+0000 7f8a55a00640 1 -- 172.21.15.96:0/1349008455 shutdown_connections 2024-05-28T06:24:18.022 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.028+0000 7f8a55a00640 1 -- 172.21.15.96:0/1349008455 wait complete. 2024-05-28T06:24:18.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.030+0000 7f8a55a00640 1 Processor -- start 2024-05-28T06:24:18.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.030+0000 7f8a55a00640 1 -- start start 2024-05-28T06:24:18.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.031+0000 7f8a55a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8a500d1650 0x7f8a5014a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:18.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.031+0000 7f8a55a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8a500d24d0 0x7f8a5015a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:18.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.031+0000 7f8a55a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8a5015e9d0 0x7f8a5015a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:18.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.031+0000 7f8a55a00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f8a500d3410 con 0x7f8a500d24d0 2024-05-28T06:24:18.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.031+0000 7f8a55a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f8a500d3130 con 0x7f8a5015e9d0 2024-05-28T06:24:18.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.031+0000 7f8a55a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f8a500d36f0 con 0x7f8a500d1650 2024-05-28T06:24:18.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.031+0000 7f8a4e000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8a500d24d0 0x7f8a5015a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:18.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.031+0000 7f8a4e000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8a500d24d0 0x7f8a5015a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:43706/0 (socket says 172.21.15.96:43706) 2024-05-28T06:24:18.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.031+0000 7f8a4e000640 1 -- 172.21.15.96:0/2021818591 learned_addr learned my addr 172.21.15.96:0/2021818591 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:24:18.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.032+0000 7f8a4f400640 1 --2- 172.21.15.96:0/2021818591 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8a5015e9d0 0x7f8a5015a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:18.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.032+0000 7f8a4e000640 1 -- 172.21.15.96:0/2021818591 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8a500d1650 msgr2=0x7f8a5014a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:18.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.032+0000 7f8a4ea00640 1 --2- 172.21.15.96:0/2021818591 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8a500d1650 0x7f8a5014a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:18.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.032+0000 7f8a4e000640 1 --2- 172.21.15.96:0/2021818591 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8a500d1650 0x7f8a5014a5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:18.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.032+0000 7f8a4e000640 1 -- 172.21.15.96:0/2021818591 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8a5015e9d0 msgr2=0x7f8a5015a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:18.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.032+0000 7f8a4e000640 1 --2- 172.21.15.96:0/2021818591 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8a5015e9d0 0x7f8a5015a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:18.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.032+0000 7f8a4e000640 1 -- 172.21.15.96:0/2021818591 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a500abb50 con 0x7f8a500d24d0 2024-05-28T06:24:18.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.032+0000 7f8a4f400640 1 --2- 172.21.15.96:0/2021818591 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8a5015e9d0 0x7f8a5015a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:24:18.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.032+0000 7f8a4e000640 1 --2- 172.21.15.96:0/2021818591 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8a500d24d0 0x7f8a5015a210 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c00d800 tx=0x7f8a3c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:18.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.032+0000 7f8a46a00640 1 -- 172.21.15.96:0/2021818591 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a3c00b6c0 con 0x7f8a500d24d0 2024-05-28T06:24:18.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.032+0000 7f8a46a00640 1 -- 172.21.15.96:0/2021818591 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a3c019070 con 0x7f8a500d24d0 2024-05-28T06:24:18.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.032+0000 7f8a46a00640 1 -- 172.21.15.96:0/2021818591 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a3c00ba80 con 0x7f8a500d24d0 2024-05-28T06:24:18.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.032+0000 7f8a55a00640 1 -- 172.21.15.96:0/2021818591 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a500ab870 con 0x7f8a500d24d0 2024-05-28T06:24:18.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.032+0000 7f8a55a00640 1 -- 172.21.15.96:0/2021818591 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a50148070 con 0x7f8a500d24d0 2024-05-28T06:24:18.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.034+0000 7f8a55a00640 1 -- 172.21.15.96:0/2021818591 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a500689e0 con 0x7f8a500d24d0 2024-05-28T06:24:18.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.035+0000 7f8a46a00640 1 -- 172.21.15.96:0/2021818591 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 42) ==== 93590+0+0 (secure 0 0 0) 0x7f8a3c00bc20 con 0x7f8a500d24d0 2024-05-28T06:24:18.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.035+0000 7f8a46a00640 1 --2- 172.21.15.96:0/2021818591 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8a3006e510 0x7f8a300709d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:18.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.036+0000 7f8a46a00640 1 -- 172.21.15.96:0/2021818591 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 4999+0+0 (secure 0 0 0) 0x7f8a3c08aff0 con 0x7f8a500d24d0 2024-05-28T06:24:18.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.036+0000 7f8a4ea00640 1 --2- 172.21.15.96:0/2021818591 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8a3006e510 0x7f8a300709d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:18.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.037+0000 7f8a4ea00640 1 --2- 172.21.15.96:0/2021818591 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8a3006e510 0x7f8a300709d0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f8a501f7960 tx=0x7f8a40009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:18.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.040+0000 7f8a46a00640 1 -- 172.21.15.96:0/2021818591 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8a500abb50 con 0x7f8a500d24d0 2024-05-28T06:24:18.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.279+0000 7f8a55a00640 1 -- 172.21.15.96:0/2021818591 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8a500cf9e0 con 0x7f8a500d24d0 2024-05-28T06:24:18.273 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.280+0000 7f8a46a00640 1 -- 172.21.15.96:0/2021818591 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) ==== 94+0+3872 (secure 0 0 0) 0x7f8a3c05ae80 con 0x7f8a500d24d0 2024-05-28T06:24:18.274 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:24:18.274 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":75,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6724,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/2148532848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6835","nonce":2148532848},{"type":"v1","addr":"172.21.15.123:6837","nonce":2148532848}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6961,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6836/826857127","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":826857127},{"type":"v1","addr":"172.21.15.123:6836","nonce":826857127}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":75},{"gid":9252,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6834/2173534467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2173534467},{"type":"v1","addr":"172.21.15.96:6834","nonce":2173534467}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":9255,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/2552082898","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6833","nonce":2552082898},{"type":"v1","addr":"172.21.15.96:6835","nonce":2552082898}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[]} 2024-05-28T06:24:18.274 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 75 2024-05-28T06:24:18.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.282+0000 7f8a55a00640 1 -- 172.21.15.96:0/2021818591 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8a3006e510 msgr2=0x7f8a300709d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:18.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.282+0000 7f8a55a00640 1 --2- 172.21.15.96:0/2021818591 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8a3006e510 0x7f8a300709d0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f8a501f7960 tx=0x7f8a40009040 comp rx=0 tx=0).stop 2024-05-28T06:24:18.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.282+0000 7f8a55a00640 1 -- 172.21.15.96:0/2021818591 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8a500d24d0 msgr2=0x7f8a5015a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:18.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.282+0000 7f8a55a00640 1 --2- 172.21.15.96:0/2021818591 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8a500d24d0 0x7f8a5015a210 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c00d800 tx=0x7f8a3c00dcc0 comp rx=0 tx=0).stop 2024-05-28T06:24:18.277 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.283+0000 7f8a55a00640 1 -- 172.21.15.96:0/2021818591 shutdown_connections 2024-05-28T06:24:18.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.283+0000 7f8a55a00640 1 --2- 172.21.15.96:0/2021818591 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8a3006e510 0x7f8a300709d0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:18.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.283+0000 7f8a55a00640 1 --2- 172.21.15.96:0/2021818591 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8a5015e9d0 0x7f8a5015a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:18.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.283+0000 7f8a55a00640 1 --2- 172.21.15.96:0/2021818591 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8a500d24d0 0x7f8a5015a210 unknown :-1 s=CLOSED pgs=1977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:18.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.283+0000 7f8a55a00640 1 --2- 172.21.15.96:0/2021818591 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8a500d1650 0x7f8a5014a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:18.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.283+0000 7f8a55a00640 1 -- 172.21.15.96:0/2021818591 >> 172.21.15.96:0/2021818591 conn(0x7f8a50067890 msgr2=0x7f8a500b6720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:18.278 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.283+0000 7f8a55a00640 1 -- 172.21.15.96:0/2021818591 shutdown_connections 2024-05-28T06:24:18.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.283+0000 7f8a55a00640 1 -- 172.21.15.96:0/2021818591 wait complete. 2024-05-28T06:24:18.297 INFO:tasks.ceph.mds.a:Restarting daemon 2024-05-28T06:24:18.297 INFO:tasks.ceph.mds.a:Stopping old one... 2024-05-28T06:24:18.297 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-05-28T06:24:18.297 INFO:teuthology.orchestra.run:waiting for 300 2024-05-28T06:24:18.298 INFO:tasks.ceph.mds.c:Restarting daemon 2024-05-28T06:24:18.298 INFO:tasks.ceph.mds.c:Stopping old one... 2024-05-28T06:24:18.298 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-05-28T06:24:18.298 INFO:teuthology.orchestra.run:waiting for 300 2024-05-28T06:24:18.299 INFO:tasks.ceph.mds.b:Restarting daemon 2024-05-28T06:24:18.299 INFO:tasks.ceph.mds.b:Stopping old one... 2024-05-28T06:24:18.300 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-05-28T06:24:18.300 INFO:teuthology.orchestra.run:waiting for 300 2024-05-28T06:24:18.300 INFO:tasks.ceph.mds.d:Restarting daemon 2024-05-28T06:24:18.300 INFO:tasks.ceph.mds.d:Stopping old one... 2024-05-28T06:24:18.300 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-05-28T06:24:18.300 INFO:teuthology.orchestra.run:waiting for 300 2024-05-28T06:24:18.318 INFO:tasks.ceph.mds.c:Stopped 2024-05-28T06:24:18.318 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-05-28T06:24:18.323 INFO:tasks.ceph.mds.a:Stopped 2024-05-28T06:24:18.323 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-05-28T06:24:18.332 INFO:tasks.ceph.mds.b:Stopped 2024-05-28T06:24:18.332 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-05-28T06:24:18.333 INFO:tasks.ceph.mds.d:Stopped 2024-05-28T06:24:18.333 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-05-28T06:24:18.374 INFO:tasks.ceph.mds.c:Started 2024-05-28T06:24:18.375 DEBUG:teuthology.parallel:result is None 2024-05-28T06:24:18.380 INFO:tasks.ceph.mds.b:Started 2024-05-28T06:24:18.380 DEBUG:teuthology.parallel:result is None 2024-05-28T06:24:18.381 INFO:tasks.ceph.mds.a:Started 2024-05-28T06:24:18.381 DEBUG:teuthology.parallel:result is None 2024-05-28T06:24:18.386 INFO:tasks.ceph.mds.d:Started 2024-05-28T06:24:18.386 DEBUG:teuthology.parallel:result is None 2024-05-28T06:24:18.387 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-05-28T06:24:18.521 INFO:tasks.ceph.mds.c.smithi096.stdout:starting mds.c at 2024-05-28T06:24:18.525 INFO:tasks.ceph.mds.a.smithi096.stdout:starting mds.a at 2024-05-28T06:24:18.534 INFO:tasks.ceph.mds.b.smithi123.stdout:starting mds.b at 2024-05-28T06:24:18.537 INFO:tasks.ceph.mds.d.smithi123.stdout:starting mds.d at 2024-05-28T06:24:18.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.576+0000 7f8ad8a00640 1 Processor -- start 2024-05-28T06:24:18.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.576+0000 7f8ad8a00640 1 -- start start 2024-05-28T06:24:18.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.577+0000 7f8ad8a00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f8ad415e870 0x7f8ad40d0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:18.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.578+0000 7f8ad8a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f8ad40d14d0 0x7f8ad40cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:18.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.578+0000 7f8ad8a00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ad40cfce0 con 0x7f8ad40cf870 2024-05-28T06:24:18.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.578+0000 7f8ad8a00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f8ad40d2150 con 0x7f8ad40d14d0 2024-05-28T06:24:18.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.578+0000 7f8ad8a00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f8ad415ec50 con 0x7f8ad415e870 2024-05-28T06:24:18.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.578+0000 7f8acbe00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f8ad40d14d0 0x7f8ad40cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:18.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.578+0000 7f8acbe00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f8ad40d14d0 0x7f8ad40cf330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:43732/0 (socket says 172.21.15.96:43732) 2024-05-28T06:24:18.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.578+0000 7f8acbe00640 1 -- 172.21.15.96:0/698459479 learned_addr learned my addr 172.21.15.96:0/698459479 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:24:18.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.578+0000 7f8ad3400640 1 --2- 172.21.15.96:0/698459479 >> v2:172.21.15.123:3300/0 conn(0x7f8ad415e870 0x7f8ad40d0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:18.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.578+0000 7f8acbe00640 1 -- 172.21.15.96:0/698459479 >> v2:172.21.15.123:3300/0 conn(0x7f8ad415e870 msgr2=0x7f8ad40d0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:18.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.578+0000 7f8acbe00640 1 --2- 172.21.15.96:0/698459479 >> v2:172.21.15.123:3300/0 conn(0x7f8ad415e870 0x7f8ad40d0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:18.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.578+0000 7f8acbe00640 1 -- 172.21.15.96:0/698459479 >> v1:172.21.15.123:6789/0 conn(0x7f8ad40cf870 legacy=0x7f8ad4162360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:18.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.578+0000 7f8acbe00640 1 -- 172.21.15.96:0/698459479 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ad40cff80 con 0x7f8ad40d14d0 2024-05-28T06:24:18.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.579+0000 7f8acbe00640 1 --2- 172.21.15.96:0/698459479 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ad40d14d0 0x7f8ad40cf330 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7f8ad408e720 tx=0x7f8abc02a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=176e9411dbbc5ce4 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:18.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.580+0000 7f8ad2a00640 1 -- 172.21.15.96:0/698459479 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8abc004030 con 0x7f8ad40d14d0 2024-05-28T06:24:18.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.580+0000 7f8ad2a00640 1 -- 172.21.15.96:0/698459479 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8abc0041d0 con 0x7f8ad40d14d0 2024-05-28T06:24:18.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.580+0000 7f8ad2a00640 1 -- 172.21.15.96:0/698459479 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8abc003610 con 0x7f8ad40d14d0 2024-05-28T06:24:18.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.580+0000 7f8ad8a00640 1 -- 172.21.15.96:0/698459479 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ad40d14d0 msgr2=0x7f8ad40cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:18.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.580+0000 7f8ad8a00640 1 --2- 172.21.15.96:0/698459479 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ad40d14d0 0x7f8ad40cf330 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7f8ad408e720 tx=0x7f8abc02a250 comp rx=0 tx=0).stop 2024-05-28T06:24:18.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.580+0000 7f8ad8a00640 1 -- 172.21.15.96:0/698459479 shutdown_connections 2024-05-28T06:24:18.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.580+0000 7f8ad8a00640 1 --2- 172.21.15.96:0/698459479 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ad40d14d0 0x7f8ad40cf330 unknown :-1 s=CLOSED pgs=1982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:18.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.580+0000 7f8ad8a00640 1 --2- 172.21.15.96:0/698459479 >> v2:172.21.15.123:3300/0 conn(0x7f8ad415e870 0x7f8ad40d0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:18.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.580+0000 7f8ad8a00640 1 -- 172.21.15.96:0/698459479 >> 172.21.15.96:0/698459479 conn(0x7f8ad408bfc0 msgr2=0x7f8ad40d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:18.575 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.580+0000 7f8ad8a00640 1 -- 172.21.15.96:0/698459479 shutdown_connections 2024-05-28T06:24:18.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.580+0000 7f8ad8a00640 1 -- 172.21.15.96:0/698459479 wait complete. 2024-05-28T06:24:18.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.582+0000 7f8ad8a00640 1 Processor -- start 2024-05-28T06:24:18.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.582+0000 7f8ad8a00640 1 -- start start 2024-05-28T06:24:18.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.583+0000 7f8ad8a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ad40cf870 0x7f8ad40b93b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:18.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.583+0000 7f8ad8a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8ad40d14d0 0x7f8ad40b71a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:18.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.583+0000 7f8ad8a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8ad415e870 0x7f8ad415bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:18.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.583+0000 7f8ad8a00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f8ad40d0820 con 0x7f8ad40cf870 2024-05-28T06:24:18.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.583+0000 7f8ad8a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f8ad40d0540 con 0x7f8ad40d14d0 2024-05-28T06:24:18.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.583+0000 7f8ad8a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f8ad40d4fb0 con 0x7f8ad415e870 2024-05-28T06:24:18.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.583+0000 7f8ad3400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ad40cf870 0x7f8ad40b93b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:18.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.583+0000 7f8ad3400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ad40cf870 0x7f8ad40b93b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:43746/0 (socket says 172.21.15.96:43746) 2024-05-28T06:24:18.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.583+0000 7f8ad3400640 1 -- 172.21.15.96:0/721318750 learned_addr learned my addr 172.21.15.96:0/721318750 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:24:18.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.583+0000 7f8ad3e00640 1 --2- 172.21.15.96:0/721318750 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8ad415e870 0x7f8ad415bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:18.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.583+0000 7f8acbe00640 1 --2- 172.21.15.96:0/721318750 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8ad40d14d0 0x7f8ad40b71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:18.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.584+0000 7f8ad3400640 1 -- 172.21.15.96:0/721318750 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8ad415e870 msgr2=0x7f8ad415bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:18.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.584+0000 7f8ad3400640 1 --2- 172.21.15.96:0/721318750 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8ad415e870 0x7f8ad415bc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:18.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.584+0000 7f8ad3400640 1 -- 172.21.15.96:0/721318750 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8ad40d14d0 msgr2=0x7f8ad40b71a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:18.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.584+0000 7f8ad3400640 1 --2- 172.21.15.96:0/721318750 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8ad40d14d0 0x7f8ad40b71a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:18.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.584+0000 7f8ad3400640 1 -- 172.21.15.96:0/721318750 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ad415c160 con 0x7f8ad40cf870 2024-05-28T06:24:18.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.584+0000 7f8ad3400640 1 --2- 172.21.15.96:0/721318750 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ad40cf870 0x7f8ad40b93b0 secure :-1 s=READY pgs=1983 cs=0 l=1 rev1=1 crypto rx=0x7f8ac400d800 tx=0x7f8ac400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:18.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.584+0000 7f8acb400640 1 -- 172.21.15.96:0/721318750 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ac400b6c0 con 0x7f8ad40cf870 2024-05-28T06:24:18.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.584+0000 7f8acb400640 1 -- 172.21.15.96:0/721318750 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ac4019070 con 0x7f8ad40cf870 2024-05-28T06:24:18.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.584+0000 7f8acb400640 1 -- 172.21.15.96:0/721318750 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ac400ba80 con 0x7f8ad40cf870 2024-05-28T06:24:18.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.584+0000 7f8ad8a00640 1 -- 172.21.15.96:0/721318750 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ad415c400 con 0x7f8ad40cf870 2024-05-28T06:24:18.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.584+0000 7f8ad8a00640 1 -- 172.21.15.96:0/721318750 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ad415c9c0 con 0x7f8ad40cf870 2024-05-28T06:24:18.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.586+0000 7f8ad8a00640 1 -- 172.21.15.96:0/721318750 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ad40689e0 con 0x7f8ad40cf870 2024-05-28T06:24:18.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.587+0000 7f8acb400640 1 -- 172.21.15.96:0/721318750 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 42) ==== 93590+0+0 (secure 0 0 0) 0x7f8ac400bc20 con 0x7f8ad40cf870 2024-05-28T06:24:18.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.588+0000 7f8acb400640 1 --2- 172.21.15.96:0/721318750 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8ab006e510 0x7f8ab00709d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:18.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.588+0000 7f8acb400640 1 -- 172.21.15.96:0/721318750 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 4999+0+0 (secure 0 0 0) 0x7f8ac408afb0 con 0x7f8ad40cf870 2024-05-28T06:24:18.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.588+0000 7f8acbe00640 1 --2- 172.21.15.96:0/721318750 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8ab006e510 0x7f8ab00709d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:18.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.589+0000 7f8acbe00640 1 --2- 172.21.15.96:0/721318750 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8ab006e510 0x7f8ab00709d0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f8abc033000 tx=0x7f8abc02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:18.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.592+0000 7f8acb400640 1 -- 172.21.15.96:0/721318750 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8ac405ae40 con 0x7f8ad40cf870 2024-05-28T06:24:18.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.811+0000 7f8ad8a00640 1 -- 172.21.15.96:0/721318750 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f8ad40ba500 con 0x7f8ad40cf870 2024-05-28T06:24:18.914 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.920+0000 7f8acb400640 1 -- 172.21.15.96:0/721318750 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v124) ==== 97+0+0 (secure 0 0 0) 0x7f8ac405a740 con 0x7f8ad40cf870 2024-05-28T06:24:18.914 INFO:teuthology.orchestra.run.smithi096.stderr: removed all blocklist entries 2024-05-28T06:24:18.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.924+0000 7f8ad8a00640 1 -- 172.21.15.96:0/721318750 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8ab006e510 msgr2=0x7f8ab00709d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:18.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.924+0000 7f8ad8a00640 1 --2- 172.21.15.96:0/721318750 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8ab006e510 0x7f8ab00709d0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f8abc033000 tx=0x7f8abc02e000 comp rx=0 tx=0).stop 2024-05-28T06:24:18.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.924+0000 7f8ad8a00640 1 -- 172.21.15.96:0/721318750 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ad40cf870 msgr2=0x7f8ad40b93b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:18.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.924+0000 7f8ad8a00640 1 --2- 172.21.15.96:0/721318750 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ad40cf870 0x7f8ad40b93b0 secure :-1 s=READY pgs=1983 cs=0 l=1 rev1=1 crypto rx=0x7f8ac400d800 tx=0x7f8ac400dcc0 comp rx=0 tx=0).stop 2024-05-28T06:24:18.919 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.924+0000 7f8ad8a00640 1 -- 172.21.15.96:0/721318750 shutdown_connections 2024-05-28T06:24:18.919 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.924+0000 7f8ad8a00640 1 --2- 172.21.15.96:0/721318750 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8ab006e510 0x7f8ab00709d0 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:18.919 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.924+0000 7f8ad8a00640 1 --2- 172.21.15.96:0/721318750 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8ad415e870 0x7f8ad415bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:18.919 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.924+0000 7f8ad8a00640 1 --2- 172.21.15.96:0/721318750 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8ad40d14d0 0x7f8ad40b71a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:18.919 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.924+0000 7f8ad8a00640 1 --2- 172.21.15.96:0/721318750 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ad40cf870 0x7f8ad40b93b0 unknown :-1 s=CLOSED pgs=1983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:18.919 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.924+0000 7f8ad8a00640 1 -- 172.21.15.96:0/721318750 >> 172.21.15.96:0/721318750 conn(0x7f8ad408bfc0 msgr2=0x7f8ad406d890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:18.919 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.924+0000 7f8ad8a00640 1 -- 172.21.15.96:0/721318750 shutdown_connections 2024-05-28T06:24:18.919 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:18.924+0000 7f8ad8a00640 1 -- 172.21.15.96:0/721318750 wait complete. 2024-05-28T06:24:18.939 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-05-28T06:24:19.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.144+0000 7f564c600640 1 Processor -- start 2024-05-28T06:24:19.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.144+0000 7f564c600640 1 -- start start 2024-05-28T06:24:19.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.146+0000 7f564c600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f56440d04a0 0x7f56440d0930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:19.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.146+0000 7f564c600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f56440ad130 0x7f56440ad540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:19.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.146+0000 7f564c600640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f564406ed10 con 0x7f564406f860 2024-05-28T06:24:19.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.146+0000 7f564c600640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f564415cb00 con 0x7f56440d04a0 2024-05-28T06:24:19.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.146+0000 7f564c600640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f56440b3cc0 con 0x7f56440ad130 2024-05-28T06:24:19.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.146+0000 7f5649200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f56440d04a0 0x7f56440d0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:19.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.146+0000 7f5649c00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f564406f860 0x7f56440b3150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:33730/0 (socket says 172.21.15.96:33730) 2024-05-28T06:24:19.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.146+0000 7f5649c00640 1 -- 172.21.15.96:0/3863975246 learned_addr learned my addr 172.21.15.96:0/3863975246 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:24:19.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.147+0000 7f564a600640 1 --2- 172.21.15.96:0/3863975246 >> v2:172.21.15.123:3300/0 conn(0x7f56440ad130 0x7f56440ad540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:19.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.147+0000 7f5649200640 1 -- 172.21.15.96:0/3863975246 >> v2:172.21.15.123:3300/0 conn(0x7f56440ad130 msgr2=0x7f56440ad540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:19.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.147+0000 7f5649200640 1 --2- 172.21.15.96:0/3863975246 >> v2:172.21.15.123:3300/0 conn(0x7f56440ad130 0x7f56440ad540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:19.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.147+0000 7f5649200640 1 -- 172.21.15.96:0/3863975246 >> v1:172.21.15.96:6789/0 conn(0x7f564406f860 legacy=0x7f56440b3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:19.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.147+0000 7f5649200640 1 -- 172.21.15.96:0/3863975246 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56440b3fa0 con 0x7f56440d04a0 2024-05-28T06:24:19.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.147+0000 7f564a600640 1 --2- 172.21.15.96:0/3863975246 >> v2:172.21.15.123:3300/0 conn(0x7f56440ad130 0x7f56440ad540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:24:19.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.148+0000 7f5649200640 1 --2- 172.21.15.96:0/3863975246 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f56440d04a0 0x7f56440d0930 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7f56440af6e0 tx=0x7f563002a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8902b5adfdf87ef server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:19.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.148+0000 7f5643e00640 1 -- 172.21.15.96:0/3863975246 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5630037070 con 0x7f56440d04a0 2024-05-28T06:24:19.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.148+0000 7f5643e00640 1 -- 172.21.15.96:0/3863975246 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5630032440 con 0x7f56440d04a0 2024-05-28T06:24:19.143 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.148+0000 7f5643e00640 1 -- 172.21.15.96:0/3863975246 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5630032810 con 0x7f56440d04a0 2024-05-28T06:24:19.143 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.149+0000 7f564c600640 1 -- 172.21.15.96:0/3863975246 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f56440d04a0 msgr2=0x7f56440d0930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:19.143 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.149+0000 7f564c600640 1 --2- 172.21.15.96:0/3863975246 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f56440d04a0 0x7f56440d0930 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7f56440af6e0 tx=0x7f563002a1a0 comp rx=0 tx=0).stop 2024-05-28T06:24:19.143 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.149+0000 7f564c600640 1 -- 172.21.15.96:0/3863975246 shutdown_connections 2024-05-28T06:24:19.143 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.149+0000 7f564c600640 1 --2- 172.21.15.96:0/3863975246 >> v2:172.21.15.123:3300/0 conn(0x7f56440ad130 0x7f56440ad540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:19.143 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.149+0000 7f564c600640 1 --2- 172.21.15.96:0/3863975246 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f56440d04a0 0x7f56440d0930 unknown :-1 s=CLOSED pgs=1985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:19.143 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.149+0000 7f564c600640 1 -- 172.21.15.96:0/3863975246 >> 172.21.15.96:0/3863975246 conn(0x7f564406bae0 msgr2=0x7f564406bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:19.143 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.149+0000 7f564c600640 1 -- 172.21.15.96:0/3863975246 shutdown_connections 2024-05-28T06:24:19.143 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.149+0000 7f564c600640 1 -- 172.21.15.96:0/3863975246 wait complete. 2024-05-28T06:24:19.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.150+0000 7f564c600640 1 Processor -- start 2024-05-28T06:24:19.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.151+0000 7f564c600640 1 -- start start 2024-05-28T06:24:19.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.152+0000 7f564c600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f564406f860 0x7f564415cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:19.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.152+0000 7f564c600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f56440ad130 0x7f564415ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:19.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.152+0000 7f564c600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f56440d04a0 0x7f5644099570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:19.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.152+0000 7f564c600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f56440b4840 con 0x7f56440d04a0 2024-05-28T06:24:19.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.152+0000 7f564c600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f56440b4560 con 0x7f564406f860 2024-05-28T06:24:19.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.152+0000 7f564c600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f56440b3860 con 0x7f56440ad130 2024-05-28T06:24:19.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.152+0000 7f564a600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f56440d04a0 0x7f5644099570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:19.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.152+0000 7f564a600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f56440d04a0 0x7f5644099570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:45236/0 (socket says 172.21.15.96:45236) 2024-05-28T06:24:19.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.152+0000 7f564a600640 1 -- 172.21.15.96:0/1422966051 learned_addr learned my addr 172.21.15.96:0/1422966051 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:24:19.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.152+0000 7f564a600640 1 -- 172.21.15.96:0/1422966051 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f56440ad130 msgr2=0x7f564415ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:19.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.152+0000 7f564a600640 1 --2- 172.21.15.96:0/1422966051 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f56440ad130 0x7f564415ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:19.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.152+0000 7f564a600640 1 -- 172.21.15.96:0/1422966051 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f564406f860 msgr2=0x7f564415cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:19.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.152+0000 7f564a600640 1 --2- 172.21.15.96:0/1422966051 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f564406f860 0x7f564415cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:19.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.152+0000 7f564a600640 1 -- 172.21.15.96:0/1422966051 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5644159740 con 0x7f56440d04a0 2024-05-28T06:24:19.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.153+0000 7f564a600640 1 --2- 172.21.15.96:0/1422966051 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f56440d04a0 0x7f5644099570 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 crypto rx=0x7f563c00cf20 tx=0x7f563c00a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:19.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.153+0000 7f5641600640 1 -- 172.21.15.96:0/1422966051 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f563c01a070 con 0x7f56440d04a0 2024-05-28T06:24:19.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.153+0000 7f5641600640 1 -- 172.21.15.96:0/1422966051 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f563c00d840 con 0x7f56440d04a0 2024-05-28T06:24:19.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.153+0000 7f5641600640 1 -- 172.21.15.96:0/1422966051 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f563c00dc30 con 0x7f56440d04a0 2024-05-28T06:24:19.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.153+0000 7f564c600640 1 -- 172.21.15.96:0/1422966051 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56440cca20 con 0x7f56440d04a0 2024-05-28T06:24:19.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.153+0000 7f564c600640 1 -- 172.21.15.96:0/1422966051 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f564415f670 con 0x7f56440d04a0 2024-05-28T06:24:19.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.154+0000 7f564c600640 1 -- 172.21.15.96:0/1422966051 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5644154480 con 0x7f56440d04a0 2024-05-28T06:24:19.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.155+0000 7f5641600640 1 -- 172.21.15.96:0/1422966051 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 42) ==== 93590+0+0 (secure 0 0 0) 0x7f563c01d070 con 0x7f56440d04a0 2024-05-28T06:24:19.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.157+0000 7f5641600640 1 --2- 172.21.15.96:0/1422966051 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f562406e510 0x7f56240709d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:19.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.157+0000 7f5641600640 1 -- 172.21.15.96:0/1422966051 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 4913+0+0 (secure 0 0 0) 0x7f563c08b360 con 0x7f56440d04a0 2024-05-28T06:24:19.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.157+0000 7f5649c00640 1 --2- 172.21.15.96:0/1422966051 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f562406e510 0x7f56240709d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:19.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.159+0000 7f5649c00640 1 --2- 172.21.15.96:0/1422966051 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f562406e510 0x7f56240709d0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f5638000f60 tx=0x7f5638007a80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:19.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.161+0000 7f5641600640 1 -- 172.21.15.96:0/1422966051 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f563c05b1f0 con 0x7f56440d04a0 2024-05-28T06:24:19.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.430+0000 7f564c600640 1 -- 172.21.15.96:0/1422966051 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f564406e2f0 con 0x7f56440d04a0 2024-05-28T06:24:19.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.431+0000 7f5641600640 1 -- 172.21.15.96:0/1422966051 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v23) ==== 80+0+5304 (secure 0 0 0) 0x7f563c05aaf0 con 0x7f56440d04a0 2024-05-28T06:24:19.425 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:24:19.425 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:24:19.425 INFO:teuthology.orchestra.run.smithi096.stdout: "auth_dump": [ 2024-05-28T06:24:19.425 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:24:19.425 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "mds.a", 2024-05-28T06:24:19.425 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCWc1VmyA9HHBAANJCF8I/RP5ZYviyOzo0JCQ==", 2024-05-28T06:24:19.425 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:24:19.425 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow", 2024-05-28T06:24:19.426 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow *", 2024-05-28T06:24:19.426 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow *", 2024-05-28T06:24:19.426 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:24:19.426 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:24:19.426 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:24:19.426 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:24:19.426 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "mds.b", 2024-05-28T06:24:19.426 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCWc1VmtR0EOBAA296qGHTmB/2sMztDbDJ56g==", 2024-05-28T06:24:19.426 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:24:19.426 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow", 2024-05-28T06:24:19.426 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow *", 2024-05-28T06:24:19.426 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow *", 2024-05-28T06:24:19.426 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:24:19.427 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:24:19.427 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:24:19.427 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:24:19.427 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "mds.c", 2024-05-28T06:24:19.427 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCWc1VmrLxHKhAAhkTRLDKL6RUJQGetB8JQQA==", 2024-05-28T06:24:19.427 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:24:19.427 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow", 2024-05-28T06:24:19.427 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow *", 2024-05-28T06:24:19.427 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow *", 2024-05-28T06:24:19.427 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:24:19.427 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:24:19.428 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:24:19.428 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:24:19.428 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "mds.d", 2024-05-28T06:24:19.428 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCXc1VmW2aQCxAAnkDwQzlgy39vFe2O9Wwf7Q==", 2024-05-28T06:24:19.428 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:24:19.428 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow", 2024-05-28T06:24:19.428 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow *", 2024-05-28T06:24:19.428 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow *", 2024-05-28T06:24:19.428 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:24:19.428 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:24:19.428 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:24:19.428 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:24:19.429 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "osd.0", 2024-05-28T06:24:19.429 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCbc1VmEcNAAxAAiB5AsfY2uRY1kc6h80chuw==", 2024-05-28T06:24:19.429 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:24:19.429 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow profile osd", 2024-05-28T06:24:19.429 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile osd", 2024-05-28T06:24:19.429 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:24:19.429 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:24:19.429 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:24:19.429 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:24:19.429 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "osd.1", 2024-05-28T06:24:19.429 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCec1Vm6CAaDhAA1NkYM2q4fTsgSDSwJZL3Qg==", 2024-05-28T06:24:19.429 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:24:19.429 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow profile osd", 2024-05-28T06:24:19.429 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile osd", 2024-05-28T06:24:19.430 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:24:19.430 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:24:19.430 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:24:19.430 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:24:19.430 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "osd.2", 2024-05-28T06:24:19.430 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQChc1VmKn1jFxAA+dM4gM7FR8NwBpyd833A/Q==", 2024-05-28T06:24:19.430 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:24:19.430 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow profile osd", 2024-05-28T06:24:19.430 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile osd", 2024-05-28T06:24:19.430 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:24:19.430 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:24:19.430 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:24:19.430 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:24:19.431 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "osd.3", 2024-05-28T06:24:19.431 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCkc1VmV07MIRAAHIXQGDVbL9V5QqexQ89IJA==", 2024-05-28T06:24:19.431 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:24:19.431 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow profile osd", 2024-05-28T06:24:19.431 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile osd", 2024-05-28T06:24:19.431 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:24:19.431 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:24:19.431 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:24:19.431 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:24:19.431 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "osd.4", 2024-05-28T06:24:19.431 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCqc1VmobuZMxAAlY8xhuzqlkuFrcbTouF5rA==", 2024-05-28T06:24:19.431 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:24:19.431 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow profile osd", 2024-05-28T06:24:19.431 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile osd", 2024-05-28T06:24:19.432 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:24:19.432 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:24:19.432 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:24:19.432 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:24:19.432 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "osd.5", 2024-05-28T06:24:19.432 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCuc1Vmwt3QAxAAUFj/e/wJvxljgAbcTb32Ug==", 2024-05-28T06:24:19.432 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:24:19.432 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow profile osd", 2024-05-28T06:24:19.432 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile osd", 2024-05-28T06:24:19.432 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:24:19.432 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:24:19.432 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:24:19.432 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:24:19.433 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "osd.6", 2024-05-28T06:24:19.433 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCxc1VmscerDxAAqEDAqy6/OvdfSJ+ALZ7WeQ==", 2024-05-28T06:24:19.433 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:24:19.433 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow profile osd", 2024-05-28T06:24:19.433 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile osd", 2024-05-28T06:24:19.433 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:24:19.433 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:24:19.433 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:24:19.433 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:24:19.434 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "osd.7", 2024-05-28T06:24:19.434 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQC0c1VmgxkbKhAA7dFzDTBKFmKiFFxPsyiT4Q==", 2024-05-28T06:24:19.434 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:24:19.434 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow profile osd", 2024-05-28T06:24:19.434 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile osd", 2024-05-28T06:24:19.434 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:24:19.434 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:24:19.434 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:24:19.434 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:24:19.434 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "client.0", 2024-05-28T06:24:19.434 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCXc1Vm91hbGBAASTL3HQR8XOs7wVQEHbPl/g==", 2024-05-28T06:24:19.434 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:24:19.434 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow", 2024-05-28T06:24:19.435 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow r", 2024-05-28T06:24:19.435 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-05-28T06:24:19.435 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:24:19.435 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:24:19.435 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:24:19.435 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "client.1", 2024-05-28T06:24:19.435 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCXc1VmIc77KhAAw2e0alTmVROhyCIFW4H3Sg==", 2024-05-28T06:24:19.435 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:24:19.435 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow", 2024-05-28T06:24:19.435 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow r", 2024-05-28T06:24:19.435 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow rw", 2024-05-28T06:24:19.435 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow rwx" 2024-05-28T06:24:19.436 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:24:19.436 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:24:19.436 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:24:19.436 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "client.2", 2024-05-28T06:24:19.436 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCXc1VmCS+OIRAAMV3Au6KuchLhA59QTGbv1w==", 2024-05-28T06:24:19.436 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:24:19.436 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow", 2024-05-28T06:24:19.436 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow r", 2024-05-28T06:24:19.436 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow rw", 2024-05-28T06:24:19.436 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow rwx" 2024-05-28T06:24:19.436 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:24:19.436 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:24:19.436 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:24:19.436 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "client.3", 2024-05-28T06:24:19.437 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCXc1Vm5r3pNBAAkVZ3arjywHQkEP5lmS8xqg==", 2024-05-28T06:24:19.437 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:24:19.437 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow", 2024-05-28T06:24:19.437 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow r", 2024-05-28T06:24:19.437 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow rw", 2024-05-28T06:24:19.437 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow rwx" 2024-05-28T06:24:19.437 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:24:19.437 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:24:19.437 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:24:19.437 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "client.admin", 2024-05-28T06:24:19.437 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCVc1Vm5QaXFhAAQFGKgbZ/kpGisNm2lIw0Uw==", 2024-05-28T06:24:19.437 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:24:19.437 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow *", 2024-05-28T06:24:19.438 INFO:teuthology.orchestra.run.smithi096.stdout: "mgr": "allow *", 2024-05-28T06:24:19.438 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow *", 2024-05-28T06:24:19.438 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:24:19.438 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:24:19.438 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:24:19.438 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:24:19.438 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "mgr.x", 2024-05-28T06:24:19.438 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCWc1VmVMx/EhAAylvi37aq3hbMOhUSQWLVSg==", 2024-05-28T06:24:19.438 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:24:19.438 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow *", 2024-05-28T06:24:19.438 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile mgr", 2024-05-28T06:24:19.438 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:24:19.438 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:24:19.439 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:24:19.439 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:24:19.439 INFO:teuthology.orchestra.run.smithi096.stdout: "entity": "mgr.y", 2024-05-28T06:24:19.439 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "AQCWc1Vmh5r7BxAAA74gn+FozehM+34eFik8kw==", 2024-05-28T06:24:19.439 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": { 2024-05-28T06:24:19.439 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": "allow *", 2024-05-28T06:24:19.439 INFO:teuthology.orchestra.run.smithi096.stdout: "mon": "allow profile mgr", 2024-05-28T06:24:19.439 INFO:teuthology.orchestra.run.smithi096.stdout: "osd": "allow *" 2024-05-28T06:24:19.439 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:24:19.439 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:24:19.439 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-05-28T06:24:19.439 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:24:19.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.433+0000 7f564c600640 1 -- 172.21.15.96:0/1422966051 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f562406e510 msgr2=0x7f56240709d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:19.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.433+0000 7f564c600640 1 --2- 172.21.15.96:0/1422966051 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f562406e510 0x7f56240709d0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f5638000f60 tx=0x7f5638007a80 comp rx=0 tx=0).stop 2024-05-28T06:24:19.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.433+0000 7f564c600640 1 -- 172.21.15.96:0/1422966051 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f56440d04a0 msgr2=0x7f5644099570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:19.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.433+0000 7f564c600640 1 --2- 172.21.15.96:0/1422966051 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f56440d04a0 0x7f5644099570 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 crypto rx=0x7f563c00cf20 tx=0x7f563c00a7d0 comp rx=0 tx=0).stop 2024-05-28T06:24:19.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.433+0000 7f564c600640 1 -- 172.21.15.96:0/1422966051 shutdown_connections 2024-05-28T06:24:19.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.433+0000 7f564c600640 1 --2- 172.21.15.96:0/1422966051 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f562406e510 0x7f56240709d0 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:19.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.433+0000 7f564c600640 1 --2- 172.21.15.96:0/1422966051 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f56440d04a0 0x7f5644099570 unknown :-1 s=CLOSED pgs=1986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:19.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.433+0000 7f564c600640 1 --2- 172.21.15.96:0/1422966051 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f56440ad130 0x7f564415ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:19.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.433+0000 7f564c600640 1 --2- 172.21.15.96:0/1422966051 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f564406f860 0x7f564415cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:19.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.434+0000 7f564c600640 1 -- 172.21.15.96:0/1422966051 >> 172.21.15.96:0/1422966051 conn(0x7f564406bae0 msgr2=0x7f56440cd010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:19.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.434+0000 7f564c600640 1 -- 172.21.15.96:0/1422966051 shutdown_connections 2024-05-28T06:24:19.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.434+0000 7f564c600640 1 -- 172.21.15.96:0/1422966051 wait complete. 2024-05-28T06:24:19.446 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:24:19.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.642+0000 7fa5b5400640 1 Processor -- start 2024-05-28T06:24:19.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.642+0000 7fa5b5400640 1 -- start start 2024-05-28T06:24:19.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.644+0000 7fa5b5400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fa5b00ad940 0x7fa5b00b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:19.638 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.644+0000 7fa5b5400640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5b0156f10 con 0x7fa5b006ee40 2024-05-28T06:24:19.638 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.644+0000 7fa5b5400640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5b006dda0 con 0x7fa5b0156b30 2024-05-28T06:24:19.638 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.644+0000 7fa5b5400640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fa5b006e080 con 0x7fa5b00ad940 2024-05-28T06:24:19.638 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.644+0000 7fa5a7e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fa5b006ee40 0x7fa5b00ad230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:33734/0 (socket says 172.21.15.96:33734) 2024-05-28T06:24:19.638 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.644+0000 7fa5a7e00640 1 -- 172.21.15.96:0/3928967495 learned_addr learned my addr 172.21.15.96:0/3928967495 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:24:19.640 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.645+0000 7fa5b4a00640 1 --2- 172.21.15.96:0/3928967495 >> v2:172.21.15.123:3300/0 conn(0x7fa5b00ad940 0x7fa5b00b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:19.640 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.645+0000 7fa5af400640 1 -- 172.21.15.96:0/3928967495 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 82451680 0 0) 0x7fa5b0156f10 con 0x7fa5b006ee40 2024-05-28T06:24:19.640 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.645+0000 7fa5af400640 1 -- 172.21.15.96:0/3928967495 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa594003520 con 0x7fa5b006ee40 2024-05-28T06:24:19.640 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.645+0000 7fa5af400640 1 -- 172.21.15.96:0/3928967495 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1541685086 0 0) 0x7fa594003520 con 0x7fa5b006ee40 2024-05-28T06:24:19.640 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.645+0000 7fa5af400640 1 -- 172.21.15.96:0/3928967495 >> v2:172.21.15.123:3300/0 conn(0x7fa5b00ad940 msgr2=0x7fa5b00b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:19.640 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.645+0000 7fa5af400640 1 --2- 172.21.15.96:0/3928967495 >> v2:172.21.15.123:3300/0 conn(0x7fa5b00ad940 0x7fa5b00b32c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:19.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.645+0000 7fa5af400640 1 -- 172.21.15.96:0/3928967495 >> v1:172.21.15.123:6789/0 conn(0x7fa5b0156b30 legacy=0x7fa5b00af6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:19.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.645+0000 7fa5af400640 1 -- 172.21.15.96:0/3928967495 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5b006e360 con 0x7fa5b006ee40 2024-05-28T06:24:19.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.645+0000 7fa5af400640 1 -- 172.21.15.96:0/3928967495 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7fa598001670 con 0x7fa5b006ee40 2024-05-28T06:24:19.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.645+0000 7fa5af400640 1 -- 172.21.15.96:0/3928967495 >> v1:172.21.15.96:6789/0 conn(0x7fa5b006ee40 legacy=0x7fa5b00ad230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:19.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.645+0000 7fa5af400640 1 --2- 172.21.15.96:0/3928967495 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa594003a00 0x7fa594003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:19.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.645+0000 7fa5af400640 1 --2- 172.21.15.96:0/3928967495 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa594004800 0x7fa594004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:19.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.646+0000 7fa5af400640 1 --2- 172.21.15.96:0/3928967495 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa594005210 0x7fa59400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:19.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.646+0000 7fa5af400640 1 -- 172.21.15.96:0/3928967495 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa5b006e080 con 0x7fa594003a00 2024-05-28T06:24:19.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.646+0000 7fa5af400640 1 -- 172.21.15.96:0/3928967495 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa59400c160 con 0x7fa594004800 2024-05-28T06:24:19.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.646+0000 7fa5af400640 1 -- 172.21.15.96:0/3928967495 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fa59400c360 con 0x7fa594005210 2024-05-28T06:24:19.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.646+0000 7fa5afe00640 1 --2- 172.21.15.96:0/3928967495 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa594003a00 0x7fa594003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:19.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.646+0000 7fa5a7e00640 1 --2- 172.21.15.96:0/3928967495 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa594004800 0x7fa594004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:19.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.646+0000 7fa5afe00640 1 -- 172.21.15.96:0/3928967495 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa594005210 msgr2=0x7fa59400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:19.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.646+0000 7fa5afe00640 1 --2- 172.21.15.96:0/3928967495 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa594005210 0x7fa59400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:19.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.646+0000 7fa5afe00640 1 -- 172.21.15.96:0/3928967495 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa594004800 msgr2=0x7fa594004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:19.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.646+0000 7fa5afe00640 1 --2- 172.21.15.96:0/3928967495 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa594004800 0x7fa594004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:19.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.646+0000 7fa5afe00640 1 -- 172.21.15.96:0/3928967495 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa594003520 con 0x7fa594003a00 2024-05-28T06:24:19.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.646+0000 7fa5a7e00640 1 --2- 172.21.15.96:0/3928967495 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa594004800 0x7fa594004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:24:19.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.646+0000 7fa5b4a00640 1 -- 172.21.15.96:0/3928967495 reap_dead start 2024-05-28T06:24:19.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.647+0000 7fa5afe00640 1 --2- 172.21.15.96:0/3928967495 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa594003a00 0x7fa594003e20 secure :-1 s=READY pgs=1988 cs=0 l=1 rev1=1 crypto rx=0x7fa5a0001720 tx=0x7fa594005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d065c383507a34c4 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:19.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.647+0000 7fa5af400640 1 -- 172.21.15.96:0/3928967495 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5a0001370 con 0x7fa594003a00 2024-05-28T06:24:19.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.647+0000 7fa5af400640 1 -- 172.21.15.96:0/3928967495 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5a00052d0 con 0x7fa594003a00 2024-05-28T06:24:19.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.647+0000 7fa5af400640 1 -- 172.21.15.96:0/3928967495 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5a002abc0 con 0x7fa594003a00 2024-05-28T06:24:19.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.648+0000 7fa5b5400640 1 -- 172.21.15.96:0/3928967495 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa594003a00 msgr2=0x7fa594003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:19.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.648+0000 7fa5b5400640 1 --2- 172.21.15.96:0/3928967495 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa594003a00 0x7fa594003e20 secure :-1 s=READY pgs=1988 cs=0 l=1 rev1=1 crypto rx=0x7fa5a0001720 tx=0x7fa594005830 comp rx=0 tx=0).stop 2024-05-28T06:24:19.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.648+0000 7fa5b5400640 1 -- 172.21.15.96:0/3928967495 shutdown_connections 2024-05-28T06:24:19.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.648+0000 7fa5b5400640 1 --2- 172.21.15.96:0/3928967495 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa594003a00 0x7fa594003e20 unknown :-1 s=CLOSED pgs=1988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:19.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.648+0000 7fa5b5400640 1 -- 172.21.15.96:0/3928967495 >> 172.21.15.96:0/3928967495 conn(0x7fa5b00b1fc0 msgr2=0x7fa5b00b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:19.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.648+0000 7fa5b5400640 1 -- 172.21.15.96:0/3928967495 shutdown_connections 2024-05-28T06:24:19.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.648+0000 7fa5b5400640 1 -- 172.21.15.96:0/3928967495 wait complete. 2024-05-28T06:24:19.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.650+0000 7fa5b5400640 1 Processor -- start 2024-05-28T06:24:19.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.650+0000 7fa5b5400640 1 -- start start 2024-05-28T06:24:19.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.651+0000 7fa5b5400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa594003a00 0x7fa5b0073020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:19.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.651+0000 7fa5b5400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5b0099b50 0x7fa5b01f4080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:19.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.651+0000 7fa5b5400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa5b00b4610 0x7fa5b00b4a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:19.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.651+0000 7fa5b5400640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa5b00b2db0 con 0x7fa594003a00 2024-05-28T06:24:19.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.651+0000 7fa5b5400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa5b00b2c30 con 0x7fa5b0099b50 2024-05-28T06:24:19.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.651+0000 7fa5b5400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fa5b006f2f0 con 0x7fa5b00b4610 2024-05-28T06:24:19.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.651+0000 7fa5afe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa594003a00 0x7fa5b0073020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:19.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.651+0000 7fa5afe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa594003a00 0x7fa5b0073020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:45252/0 (socket says 172.21.15.96:45252) 2024-05-28T06:24:19.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.651+0000 7fa5afe00640 1 -- 172.21.15.96:0/3096836691 learned_addr learned my addr 172.21.15.96:0/3096836691 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:24:19.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.652+0000 7fa5afe00640 1 -- 172.21.15.96:0/3096836691 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa5b00b4610 msgr2=0x7fa5b00b4a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:19.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.652+0000 7fa5afe00640 1 --2- 172.21.15.96:0/3096836691 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa5b00b4610 0x7fa5b00b4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:19.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.652+0000 7fa5afe00640 1 -- 172.21.15.96:0/3096836691 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5b0099b50 msgr2=0x7fa5b01f4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:19.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.652+0000 7fa5afe00640 1 --2- 172.21.15.96:0/3096836691 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5b0099b50 0x7fa5b01f4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:19.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.652+0000 7fa5afe00640 1 -- 172.21.15.96:0/3096836691 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5b01f45c0 con 0x7fa594003a00 2024-05-28T06:24:19.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.652+0000 7fa5afe00640 1 --2- 172.21.15.96:0/3096836691 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa594003a00 0x7fa5b0073020 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 crypto rx=0x7fa5a0001840 tx=0x7fa5a0001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:19.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.652+0000 7fa5acc00640 1 -- 172.21.15.96:0/3096836691 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5a003f030 con 0x7fa594003a00 2024-05-28T06:24:19.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.652+0000 7fa5acc00640 1 -- 172.21.15.96:0/3096836691 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5a0005480 con 0x7fa594003a00 2024-05-28T06:24:19.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.652+0000 7fa5acc00640 1 -- 172.21.15.96:0/3096836691 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5a0037040 con 0x7fa594003a00 2024-05-28T06:24:19.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.652+0000 7fa5b5400640 1 -- 172.21.15.96:0/3096836691 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5b0071f80 con 0x7fa594003a00 2024-05-28T06:24:19.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.652+0000 7fa5b5400640 1 -- 172.21.15.96:0/3096836691 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5b0072540 con 0x7fa594003a00 2024-05-28T06:24:19.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.653+0000 7fa5b5400640 1 -- 172.21.15.96:0/3096836691 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5b00689e0 con 0x7fa594003a00 2024-05-28T06:24:19.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.655+0000 7fa5acc00640 1 -- 172.21.15.96:0/3096836691 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 42) ==== 93590+0+0 (secure 0 0 0) 0x7fa5a0042070 con 0x7fa594003a00 2024-05-28T06:24:19.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.655+0000 7fa5acc00640 1 --2- 172.21.15.96:0/3096836691 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa58c06e540 0x7fa58c070a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:19.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.656+0000 7fa5acc00640 1 -- 172.21.15.96:0/3096836691 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 4913+0+0 (secure 0 0 0) 0x7fa5a0007640 con 0x7fa594003a00 2024-05-28T06:24:19.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.656+0000 7fa5a7e00640 1 --2- 172.21.15.96:0/3096836691 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa58c06e540 0x7fa58c070a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:19.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.657+0000 7fa5a7e00640 1 --2- 172.21.15.96:0/3096836691 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa58c06e540 0x7fa58c070a00 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fa598000c00 tx=0x7fa598009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:19.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.659+0000 7fa5acc00640 1 -- 172.21.15.96:0/3096836691 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa5a00433b0 con 0x7fa594003a00 2024-05-28T06:24:19.857 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.864+0000 7fa5b5400640 1 -- 172.21.15.96:0/3096836691 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa5b006f7f0 con 0x7fa594003a00 2024-05-28T06:24:19.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.866+0000 7fa5acc00640 1 -- 172.21.15.96:0/3096836691 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v126) ==== 74+0+13001 (secure 0 0 0) 0x7fa5a007a010 con 0x7fa594003a00 2024-05-28T06:24:19.860 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:24:19.860 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":126,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:24:18.938417+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":13,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:24:19.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.868+0000 7fa5b5400640 1 -- 172.21.15.96:0/3096836691 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa58c06e540 msgr2=0x7fa58c070a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:19.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.869+0000 7fa5b5400640 1 --2- 172.21.15.96:0/3096836691 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa58c06e540 0x7fa58c070a00 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fa598000c00 tx=0x7fa598009040 comp rx=0 tx=0).stop 2024-05-28T06:24:19.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.869+0000 7fa5b5400640 1 -- 172.21.15.96:0/3096836691 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa594003a00 msgr2=0x7fa5b0073020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:19.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.869+0000 7fa5b5400640 1 --2- 172.21.15.96:0/3096836691 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa594003a00 0x7fa5b0073020 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 crypto rx=0x7fa5a0001840 tx=0x7fa5a0001120 comp rx=0 tx=0).stop 2024-05-28T06:24:19.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.869+0000 7fa5b5400640 1 -- 172.21.15.96:0/3096836691 shutdown_connections 2024-05-28T06:24:19.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.869+0000 7fa5b5400640 1 --2- 172.21.15.96:0/3096836691 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa58c06e540 0x7fa58c070a00 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:19.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.869+0000 7fa5b5400640 1 --2- 172.21.15.96:0/3096836691 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa5b00b4610 0x7fa5b00b4a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:19.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.869+0000 7fa5b5400640 1 --2- 172.21.15.96:0/3096836691 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5b0099b50 0x7fa5b01f4080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:19.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.869+0000 7fa5b5400640 1 --2- 172.21.15.96:0/3096836691 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa594003a00 0x7fa5b0073020 unknown :-1 s=CLOSED pgs=1989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:19.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.869+0000 7fa5b5400640 1 -- 172.21.15.96:0/3096836691 >> 172.21.15.96:0/3096836691 conn(0x7fa5b00b1fc0 msgr2=0x7fa5b015b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:19.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.869+0000 7fa5b5400640 1 -- 172.21.15.96:0/3096836691 shutdown_connections 2024-05-28T06:24:19.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:19.869+0000 7fa5b5400640 1 -- 172.21.15.96:0/3096836691 wait complete. 2024-05-28T06:24:19.881 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-28T06:03:59.517102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-28T06:24:19.881 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-28T06:24:20.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.091+0000 7fee0be00640 1 Processor -- start 2024-05-28T06:24:20.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.091+0000 7fee0be00640 1 -- start start 2024-05-28T06:24:20.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.093+0000 7fee0be00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fee0c0cfa40 0x7fee0c169690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:20.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.093+0000 7fee0be00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee0c15ec50 con 0x7fee0c0d16a0 2024-05-28T06:24:20.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.093+0000 7fee0be00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee0c0cff80 con 0x7fee0c15e870 2024-05-28T06:24:20.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.093+0000 7fee0be00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fee0c0d2150 con 0x7fee0c0cfa40 2024-05-28T06:24:20.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.093+0000 7fee03e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fee0c0d16a0 0x7fee0c0cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:33750/0 (socket says 172.21.15.96:33750) 2024-05-28T06:24:20.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.093+0000 7fee03e00640 1 -- 172.21.15.96:0/3518145093 learned_addr learned my addr 172.21.15.96:0/3518145093 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:24:20.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.093+0000 7fee0a000640 1 -- 172.21.15.96:0/3518145093 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1434531422 0 0) 0x7fee0c15ec50 con 0x7fee0c0d16a0 2024-05-28T06:24:20.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.093+0000 7fee0b400640 1 --2- 172.21.15.96:0/3518145093 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fee0c0cfa40 0x7fee0c169690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:20.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.093+0000 7fee0a000640 1 -- 172.21.15.96:0/3518145093 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fedf0003520 con 0x7fee0c0d16a0 2024-05-28T06:24:20.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.094+0000 7fee0a000640 1 -- 172.21.15.96:0/3518145093 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2660372582 0 0) 0x7fedf0003520 con 0x7fee0c0d16a0 2024-05-28T06:24:20.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.094+0000 7fee0a000640 1 -- 172.21.15.96:0/3518145093 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fee0c0cfa40 msgr2=0x7fee0c169690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:20.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.094+0000 7fee0a000640 1 --2- 172.21.15.96:0/3518145093 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fee0c0cfa40 0x7fee0c169690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:20.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.094+0000 7fee0a000640 1 -- 172.21.15.96:0/3518145093 >> v1:172.21.15.123:6789/0 conn(0x7fee0c15e870 legacy=0x7fee0c0d0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:20.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.094+0000 7fee0a000640 1 -- 172.21.15.96:0/3518145093 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee0c0d0260 con 0x7fee0c0d16a0 2024-05-28T06:24:20.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.094+0000 7fee0a000640 1 -- 172.21.15.96:0/3518145093 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7fedf40017f0 con 0x7fee0c0d16a0 2024-05-28T06:24:20.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.094+0000 7fee0a000640 1 -- 172.21.15.96:0/3518145093 >> v1:172.21.15.96:6789/0 conn(0x7fee0c0d16a0 legacy=0x7fee0c0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:20.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.094+0000 7fee0a000640 1 --2- 172.21.15.96:0/3518145093 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fedf00039e0 0x7fedf0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:20.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.094+0000 7fee0a000640 1 --2- 172.21.15.96:0/3518145093 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fedf0004800 0x7fedf0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:20.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.094+0000 7fee0a000640 1 --2- 172.21.15.96:0/3518145093 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fee0c0cfa40 0x7fedf0005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:20.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.094+0000 7fee0a000640 1 -- 172.21.15.96:0/3518145093 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fee0c0d2150 con 0x7fedf00039e0 2024-05-28T06:24:20.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.094+0000 7fee0a000640 1 -- 172.21.15.96:0/3518145093 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fedf000baf0 con 0x7fedf0004800 2024-05-28T06:24:20.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.094+0000 7fee0a000640 1 -- 172.21.15.96:0/3518145093 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fedf000bcf0 con 0x7fee0c0cfa40 2024-05-28T06:24:20.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.094+0000 7fee0aa00640 1 --2- 172.21.15.96:0/3518145093 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fedf00039e0 0x7fedf0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:20.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.094+0000 7fee0b400640 1 --2- 172.21.15.96:0/3518145093 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fee0c0cfa40 0x7fedf0005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:20.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.095+0000 7fee03e00640 1 --2- 172.21.15.96:0/3518145093 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fedf0004800 0x7fedf0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:20.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.095+0000 7fee0aa00640 1 -- 172.21.15.96:0/3518145093 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fee0c0cfa40 msgr2=0x7fedf0005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:20.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.095+0000 7fee0aa00640 1 --2- 172.21.15.96:0/3518145093 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fee0c0cfa40 0x7fedf0005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:20.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.095+0000 7fee0aa00640 1 -- 172.21.15.96:0/3518145093 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fedf0004800 msgr2=0x7fedf0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:20.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.095+0000 7fee0aa00640 1 --2- 172.21.15.96:0/3518145093 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fedf0004800 0x7fedf0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:20.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.095+0000 7fee0aa00640 1 -- 172.21.15.96:0/3518145093 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedf0003520 con 0x7fedf00039e0 2024-05-28T06:24:20.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.095+0000 7fee03e00640 1 --2- 172.21.15.96:0/3518145093 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fedf0004800 0x7fedf0004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:24:20.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.096+0000 7fee0aa00640 1 --2- 172.21.15.96:0/3518145093 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fedf00039e0 0x7fedf0003e00 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 crypto rx=0x7fedfc001750 tx=0x7fedf0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=742e512248e9dae4 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:20.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.096+0000 7fee0a000640 1 -- 172.21.15.96:0/3518145093 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedfc001370 con 0x7fedf00039e0 2024-05-28T06:24:20.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.096+0000 7fee0a000640 1 -- 172.21.15.96:0/3518145093 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fedfc0052d0 con 0x7fedf00039e0 2024-05-28T06:24:20.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.096+0000 7fee0a000640 1 -- 172.21.15.96:0/3518145093 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedfc02abe0 con 0x7fedf00039e0 2024-05-28T06:24:20.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.096+0000 7fee0be00640 1 -- 172.21.15.96:0/3518145093 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fedf00039e0 msgr2=0x7fedf0003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:20.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.096+0000 7fee0be00640 1 --2- 172.21.15.96:0/3518145093 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fedf00039e0 0x7fedf0003e00 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 crypto rx=0x7fedfc001750 tx=0x7fedf0005830 comp rx=0 tx=0).stop 2024-05-28T06:24:20.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.096+0000 7fee0b400640 1 -- 172.21.15.96:0/3518145093 reap_dead start 2024-05-28T06:24:20.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.096+0000 7fee0be00640 1 -- 172.21.15.96:0/3518145093 shutdown_connections 2024-05-28T06:24:20.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.096+0000 7fee0be00640 1 -- 172.21.15.96:0/3518145093 >> 172.21.15.96:0/3518145093 conn(0x7fee0c08bfc0 msgr2=0x7fee0c0d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:20.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.097+0000 7fee0be00640 1 -- 172.21.15.96:0/3518145093 shutdown_connections 2024-05-28T06:24:20.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.097+0000 7fee0be00640 1 -- 172.21.15.96:0/3518145093 wait complete. 2024-05-28T06:24:20.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.098+0000 7fee0be00640 1 Processor -- start 2024-05-28T06:24:20.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.098+0000 7fee0be00640 1 -- start start 2024-05-28T06:24:20.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.099+0000 7fee0be00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fedf00039e0 0x7fee0c159e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:20.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.099+0000 7fee0be00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fee0c161010 0x7fee0c161490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:20.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.099+0000 7fee0be00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fee0c1f7d10 0x7fee0c1f8190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:20.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.099+0000 7fee0be00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fee0c0d4fb0 con 0x7fee0c1f7d10 2024-05-28T06:24:20.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.099+0000 7fee0be00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fee0c0d0820 con 0x7fedf00039e0 2024-05-28T06:24:20.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.099+0000 7fee0be00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fee0c0d35f0 con 0x7fee0c161010 2024-05-28T06:24:20.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.099+0000 7fee0b400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fee0c1f7d10 0x7fee0c1f8190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:20.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.099+0000 7fee0b400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fee0c1f7d10 0x7fee0c1f8190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:45280/0 (socket says 172.21.15.96:45280) 2024-05-28T06:24:20.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.099+0000 7fee0b400640 1 -- 172.21.15.96:0/2511636374 learned_addr learned my addr 172.21.15.96:0/2511636374 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:24:20.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.100+0000 7fee0aa00640 1 --2- 172.21.15.96:0/2511636374 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fedf00039e0 0x7fee0c159e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:20.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.100+0000 7fee0b400640 1 -- 172.21.15.96:0/2511636374 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fee0c161010 msgr2=0x7fee0c161490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:20.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.100+0000 7fee0b400640 1 --2- 172.21.15.96:0/2511636374 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fee0c161010 0x7fee0c161490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:20.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.100+0000 7fee0b400640 1 -- 172.21.15.96:0/2511636374 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fedf00039e0 msgr2=0x7fee0c159e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:20.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.100+0000 7fee0b400640 1 --2- 172.21.15.96:0/2511636374 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fedf00039e0 0x7fee0c159e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:20.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.100+0000 7fee0b400640 1 -- 172.21.15.96:0/2511636374 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee0c15a3d0 con 0x7fee0c1f7d10 2024-05-28T06:24:20.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.100+0000 7fee0aa00640 1 --2- 172.21.15.96:0/2511636374 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fedf00039e0 0x7fee0c159e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:24:20.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.100+0000 7fee0b400640 1 --2- 172.21.15.96:0/2511636374 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fee0c1f7d10 0x7fee0c1f8190 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 crypto rx=0x7fee040098c0 tx=0x7fee04009d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:20.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.100+0000 7fee02a00640 1 -- 172.21.15.96:0/2511636374 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee04007cc0 con 0x7fee0c1f7d10 2024-05-28T06:24:20.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.100+0000 7fee02a00640 1 -- 172.21.15.96:0/2511636374 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee04019070 con 0x7fee0c1f7d10 2024-05-28T06:24:20.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.100+0000 7fee02a00640 1 -- 172.21.15.96:0/2511636374 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee04007e60 con 0x7fee0c1f7d10 2024-05-28T06:24:20.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.100+0000 7fee0be00640 1 -- 172.21.15.96:0/2511636374 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee0c1f8a70 con 0x7fee0c1f7d10 2024-05-28T06:24:20.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.101+0000 7fee0be00640 1 -- 172.21.15.96:0/2511636374 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee0c1e84e0 con 0x7fee0c1f7d10 2024-05-28T06:24:20.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.101+0000 7fee0be00640 1 -- 172.21.15.96:0/2511636374 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee0c0689e0 con 0x7fee0c1f7d10 2024-05-28T06:24:20.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.102+0000 7fee02a00640 1 -- 172.21.15.96:0/2511636374 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 42) ==== 93590+0+0 (secure 0 0 0) 0x7fee04014650 con 0x7fee0c1f7d10 2024-05-28T06:24:20.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.103+0000 7fee02a00640 1 --2- 172.21.15.96:0/2511636374 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fede806e540 0x7fede8070a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:20.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.103+0000 7fee02a00640 1 -- 172.21.15.96:0/2511636374 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 4913+0+0 (secure 0 0 0) 0x7fee0408ae50 con 0x7fee0c1f7d10 2024-05-28T06:24:20.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.104+0000 7fee0aa00640 1 --2- 172.21.15.96:0/2511636374 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fede806e540 0x7fede8070a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:20.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.106+0000 7fee0aa00640 1 --2- 172.21.15.96:0/2511636374 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fede806e540 0x7fede8070a00 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fedfc032000 tx=0x7fedfc02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:20.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.108+0000 7fee02a00640 1 -- 172.21.15.96:0/2511636374 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fee0405ace0 con 0x7fee0c1f7d10 2024-05-28T06:24:20.293 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.300+0000 7fee0be00640 1 -- 172.21.15.96:0/2511636374 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fee0c0ba500 con 0x7fee0c1f7d10 2024-05-28T06:24:20.294 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.300+0000 7fee02a00640 1 -- 172.21.15.96:0/2511636374 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v126) ==== 93+0+10 (secure 0 0 0) 0x7fee0405a5e0 con 0x7fee0c1f7d10 2024-05-28T06:24:20.294 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 1 2024-05-28T06:24:20.297 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.303+0000 7fee0be00640 1 -- 172.21.15.96:0/2511636374 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fede806e540 msgr2=0x7fede8070a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:20.297 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.303+0000 7fee0be00640 1 --2- 172.21.15.96:0/2511636374 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fede806e540 0x7fede8070a00 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fedfc032000 tx=0x7fedfc02e000 comp rx=0 tx=0).stop 2024-05-28T06:24:20.297 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.303+0000 7fee0be00640 1 -- 172.21.15.96:0/2511636374 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fee0c1f7d10 msgr2=0x7fee0c1f8190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:20.297 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.303+0000 7fee0be00640 1 --2- 172.21.15.96:0/2511636374 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fee0c1f7d10 0x7fee0c1f8190 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 crypto rx=0x7fee040098c0 tx=0x7fee04009d80 comp rx=0 tx=0).stop 2024-05-28T06:24:20.297 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.303+0000 7fee0be00640 1 -- 172.21.15.96:0/2511636374 shutdown_connections 2024-05-28T06:24:20.297 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.303+0000 7fee0be00640 1 --2- 172.21.15.96:0/2511636374 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fede806e540 0x7fede8070a00 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:20.297 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.303+0000 7fee0be00640 1 --2- 172.21.15.96:0/2511636374 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fee0c1f7d10 0x7fee0c1f8190 unknown :-1 s=CLOSED pgs=1992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:20.297 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.303+0000 7fee0be00640 1 --2- 172.21.15.96:0/2511636374 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fee0c161010 0x7fee0c161490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:20.297 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.303+0000 7fee0be00640 1 --2- 172.21.15.96:0/2511636374 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fedf00039e0 0x7fee0c159e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:20.297 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.303+0000 7fee0be00640 1 -- 172.21.15.96:0/2511636374 >> 172.21.15.96:0/2511636374 conn(0x7fee0c08bfc0 msgr2=0x7fee0c1622f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:20.297 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.303+0000 7fee0be00640 1 -- 172.21.15.96:0/2511636374 shutdown_connections 2024-05-28T06:24:20.298 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.303+0000 7fee0be00640 1 -- 172.21.15.96:0/2511636374 wait complete. 2024-05-28T06:24:20.315 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-05-28T06:24:20.492 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.499+0000 7f2b8d800640 1 Processor -- start 2024-05-28T06:24:20.493 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.499+0000 7f2b8d800640 1 -- start start 2024-05-28T06:24:20.494 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.500+0000 7f2b8d800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f2b8806f860 0x7f2b880b3150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:20.494 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.500+0000 7f2b8d800640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b8806ed10 con 0x7f2b880d0bb0 2024-05-28T06:24:20.494 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.500+0000 7f2b8d800640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b8815cb00 con 0x7f2b880b3690 2024-05-28T06:24:20.494 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.500+0000 7f2b8d800640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f2b880b3cc0 con 0x7f2b8806f860 2024-05-28T06:24:20.494 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.501+0000 7f2b87400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f2b880d0bb0 0x7f2b880ad130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:33762/0 (socket says 172.21.15.96:33762) 2024-05-28T06:24:20.494 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.501+0000 7f2b87400640 1 -- 172.21.15.96:0/1814030323 learned_addr learned my addr 172.21.15.96:0/1814030323 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:24:20.494 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.501+0000 7f2b86a00640 1 --2- 172.21.15.96:0/1814030323 >> v2:172.21.15.123:3300/0 conn(0x7f2b8806f860 0x7f2b880b3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:20.496 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.501+0000 7f2b86000640 1 -- 172.21.15.96:0/1814030323 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4051905313 0 0) 0x7f2b80001490 con 0x7f2b880d0bb0 2024-05-28T06:24:20.496 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.501+0000 7f2b86000640 1 -- 172.21.15.96:0/1814030323 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2b6c003520 con 0x7f2b880d0bb0 2024-05-28T06:24:20.496 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.501+0000 7f2b86000640 1 -- 172.21.15.96:0/1814030323 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1778923043 0 0) 0x7f2b6c003520 con 0x7f2b880d0bb0 2024-05-28T06:24:20.496 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.501+0000 7f2b86000640 1 -- 172.21.15.96:0/1814030323 >> v2:172.21.15.123:3300/0 conn(0x7f2b8806f860 msgr2=0x7f2b880b3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:20.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.501+0000 7f2b86000640 1 --2- 172.21.15.96:0/1814030323 >> v2:172.21.15.123:3300/0 conn(0x7f2b8806f860 0x7f2b880b3150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:20.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.501+0000 7f2b86000640 1 -- 172.21.15.96:0/1814030323 >> v1:172.21.15.123:6789/0 conn(0x7f2b880b3690 legacy=0x7f2b880d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:20.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.501+0000 7f2b86000640 1 -- 172.21.15.96:0/1814030323 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b880b3fa0 con 0x7f2b880d0bb0 2024-05-28T06:24:20.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.501+0000 7f2b86000640 1 -- 172.21.15.96:0/1814030323 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f2b80001bc0 con 0x7f2b880d0bb0 2024-05-28T06:24:20.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.502+0000 7f2b86000640 1 -- 172.21.15.96:0/1814030323 >> v1:172.21.15.96:6789/0 conn(0x7f2b880d0bb0 legacy=0x7f2b880ad130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:20.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.502+0000 7f2b86000640 1 --2- 172.21.15.96:0/1814030323 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2b6c0039b0 0x7f2b6c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:20.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.502+0000 7f2b86000640 1 --2- 172.21.15.96:0/1814030323 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2b6c004800 0x7f2b6c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:20.497 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.502+0000 7f2b86000640 1 --2- 172.21.15.96:0/1814030323 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2b6c0051c0 0x7f2b6c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:20.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.502+0000 7f2b86000640 1 -- 172.21.15.96:0/1814030323 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f2b880b3cc0 con 0x7f2b6c0039b0 2024-05-28T06:24:20.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.502+0000 7f2b86000640 1 -- 172.21.15.96:0/1814030323 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f2b6c00c060 con 0x7f2b6c004800 2024-05-28T06:24:20.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.502+0000 7f2b86000640 1 -- 172.21.15.96:0/1814030323 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f2b6c00c260 con 0x7f2b6c0051c0 2024-05-28T06:24:20.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.502+0000 7f2b86a00640 1 --2- 172.21.15.96:0/1814030323 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2b6c0039b0 0x7f2b6c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:20.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.502+0000 7f2b86a00640 1 -- 172.21.15.96:0/1814030323 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2b6c0051c0 msgr2=0x7f2b6c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:20.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.502+0000 7f2b86a00640 1 --2- 172.21.15.96:0/1814030323 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2b6c0051c0 0x7f2b6c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:20.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.502+0000 7f2b86a00640 1 -- 172.21.15.96:0/1814030323 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2b6c004800 msgr2=0x7f2b6c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:20.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.502+0000 7f2b86a00640 1 --2- 172.21.15.96:0/1814030323 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2b6c004800 0x7f2b6c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:20.498 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.502+0000 7f2b86a00640 1 -- 172.21.15.96:0/1814030323 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b6c003520 con 0x7f2b6c0039b0 2024-05-28T06:24:20.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.502+0000 7f2b87400640 1 -- 172.21.15.96:0/1814030323 reap_dead start 2024-05-28T06:24:20.499 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.503+0000 7f2b86a00640 1 --2- 172.21.15.96:0/1814030323 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2b6c0039b0 0x7f2b6c003dd0 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 crypto rx=0x7f2b78009ac0 tx=0x7f2b6c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a38533086db92b8 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:20.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.503+0000 7f2b86000640 1 -- 172.21.15.96:0/1814030323 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b78036070 con 0x7f2b6c0039b0 2024-05-28T06:24:20.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.503+0000 7f2b86000640 1 -- 172.21.15.96:0/1814030323 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b78031400 con 0x7f2b6c0039b0 2024-05-28T06:24:20.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.503+0000 7f2b86000640 1 -- 172.21.15.96:0/1814030323 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b78009690 con 0x7f2b6c0039b0 2024-05-28T06:24:20.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.504+0000 7f2b8d800640 1 -- 172.21.15.96:0/1814030323 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2b6c0039b0 msgr2=0x7f2b6c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:20.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.504+0000 7f2b8d800640 1 --2- 172.21.15.96:0/1814030323 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2b6c0039b0 0x7f2b6c003dd0 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 crypto rx=0x7f2b78009ac0 tx=0x7f2b6c005830 comp rx=0 tx=0).stop 2024-05-28T06:24:20.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.504+0000 7f2b8d800640 1 -- 172.21.15.96:0/1814030323 shutdown_connections 2024-05-28T06:24:20.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.504+0000 7f2b8d800640 1 --2- 172.21.15.96:0/1814030323 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2b6c0039b0 0x7f2b6c003dd0 unknown :-1 s=CLOSED pgs=1994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:20.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.504+0000 7f2b8d800640 1 -- 172.21.15.96:0/1814030323 >> 172.21.15.96:0/1814030323 conn(0x7f2b8806bae0 msgr2=0x7f2b8806bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:20.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.504+0000 7f2b8d800640 1 -- 172.21.15.96:0/1814030323 shutdown_connections 2024-05-28T06:24:20.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.504+0000 7f2b8d800640 1 -- 172.21.15.96:0/1814030323 wait complete. 2024-05-28T06:24:20.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.506+0000 7f2b8d800640 1 Processor -- start 2024-05-28T06:24:20.501 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.506+0000 7f2b8d800640 1 -- start start 2024-05-28T06:24:20.502 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.507+0000 7f2b8d800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2b6c0039b0 0x7f2b880c94b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:20.502 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.507+0000 7f2b8d800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2b880cd2a0 0x7f2b880c6750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:20.502 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.507+0000 7f2b8d800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2b880c4760 0x7f2b880c7330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:20.502 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.507+0000 7f2b8d800640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f2b880b4840 con 0x7f2b6c0039b0 2024-05-28T06:24:20.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.507+0000 7f2b8d800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f2b880b4560 con 0x7f2b880cd2a0 2024-05-28T06:24:20.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.507+0000 7f2b8d800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f2b880b3150 con 0x7f2b880c4760 2024-05-28T06:24:20.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.507+0000 7f2b86a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2b6c0039b0 0x7f2b880c94b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:20.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.507+0000 7f2b86a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2b6c0039b0 0x7f2b880c94b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:45306/0 (socket says 172.21.15.96:45306) 2024-05-28T06:24:20.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.507+0000 7f2b86a00640 1 -- 172.21.15.96:0/2369749678 learned_addr learned my addr 172.21.15.96:0/2369749678 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:24:20.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.507+0000 7f2b7e000640 1 --2- 172.21.15.96:0/2369749678 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2b880cd2a0 0x7f2b880c6750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:20.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.507+0000 7f2b86a00640 1 -- 172.21.15.96:0/2369749678 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2b880c4760 msgr2=0x7f2b880c7330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:20.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.507+0000 7f2b86a00640 1 --2- 172.21.15.96:0/2369749678 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2b880c4760 0x7f2b880c7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:20.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.507+0000 7f2b86a00640 1 -- 172.21.15.96:0/2369749678 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2b880cd2a0 msgr2=0x7f2b880c6750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:20.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.508+0000 7f2b86a00640 1 --2- 172.21.15.96:0/2369749678 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2b880cd2a0 0x7f2b880c6750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:20.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.508+0000 7f2b86a00640 1 -- 172.21.15.96:0/2369749678 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b8815de00 con 0x7f2b6c0039b0 2024-05-28T06:24:20.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.508+0000 7f2b86a00640 1 --2- 172.21.15.96:0/2369749678 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2b6c0039b0 0x7f2b880c94b0 secure :-1 s=READY pgs=1995 cs=0 l=1 rev1=1 crypto rx=0x7f2b78034040 tx=0x7f2b78004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:20.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.508+0000 7f2b7f400640 1 -- 172.21.15.96:0/2369749678 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b7803f020 con 0x7f2b6c0039b0 2024-05-28T06:24:20.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.508+0000 7f2b7f400640 1 -- 172.21.15.96:0/2369749678 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b78037040 con 0x7f2b6c0039b0 2024-05-28T06:24:20.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.508+0000 7f2b7f400640 1 -- 172.21.15.96:0/2369749678 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b78036070 con 0x7f2b6c0039b0 2024-05-28T06:24:20.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.508+0000 7f2b8d800640 1 -- 172.21.15.96:0/2369749678 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b8815e0e0 con 0x7f2b6c0039b0 2024-05-28T06:24:20.505 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.508+0000 7f2b8d800640 1 -- 172.21.15.96:0/2369749678 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b8815e6a0 con 0x7f2b6c0039b0 2024-05-28T06:24:20.505 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.510+0000 7f2b8d800640 1 -- 172.21.15.96:0/2369749678 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b880689e0 con 0x7f2b6c0039b0 2024-05-28T06:24:20.506 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.511+0000 7f2b7f400640 1 -- 172.21.15.96:0/2369749678 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 42) ==== 93590+0+0 (secure 0 0 0) 0x7f2b78042020 con 0x7f2b6c0039b0 2024-05-28T06:24:20.506 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.511+0000 7f2b7f400640 1 --2- 172.21.15.96:0/2369749678 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2b6406e510 0x7f2b640709d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:20.506 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.511+0000 7f2b7f400640 1 -- 172.21.15.96:0/2369749678 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 4913+0+0 (secure 0 0 0) 0x7f2b780aab40 con 0x7f2b6c0039b0 2024-05-28T06:24:20.506 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.512+0000 7f2b7e000640 1 --2- 172.21.15.96:0/2369749678 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2b6406e510 0x7f2b640709d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:20.506 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.513+0000 7f2b7e000640 1 --2- 172.21.15.96:0/2369749678 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2b6406e510 0x7f2b640709d0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f2b880c3710 tx=0x7f2b70009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:20.509 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.516+0000 7f2b7f400640 1 -- 172.21.15.96:0/2369749678 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2b78049030 con 0x7f2b6c0039b0 2024-05-28T06:24:20.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.714+0000 7f2b8d800640 1 -- 172.21.15.96:0/2369749678 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f2b8806c170 con 0x7f2b6c0039b0 2024-05-28T06:24:20.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.714+0000 7f2b7f400640 1 -- 172.21.15.96:0/2369749678 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v126) ==== 84+0+65 (secure 0 0 0) 0x7f2b7807aa90 con 0x7f2b6c0039b0 2024-05-28T06:24:20.708 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:24:20.708 INFO:teuthology.orchestra.run.smithi096.stdout:[ 2024-05-28T06:24:20.708 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:24:20.708 INFO:teuthology.orchestra.run.smithi096.stdout: "poolnum": 1, 2024-05-28T06:24:20.709 INFO:teuthology.orchestra.run.smithi096.stdout: "poolname": ".mgr" 2024-05-28T06:24:20.709 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:24:20.709 INFO:teuthology.orchestra.run.smithi096.stdout:] 2024-05-28T06:24:20.711 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.717+0000 7f2b8d800640 1 -- 172.21.15.96:0/2369749678 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2b6406e510 msgr2=0x7f2b640709d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:20.711 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.717+0000 7f2b8d800640 1 --2- 172.21.15.96:0/2369749678 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2b6406e510 0x7f2b640709d0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f2b880c3710 tx=0x7f2b70009040 comp rx=0 tx=0).stop 2024-05-28T06:24:20.711 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.717+0000 7f2b8d800640 1 -- 172.21.15.96:0/2369749678 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2b6c0039b0 msgr2=0x7f2b880c94b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:20.711 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.717+0000 7f2b8d800640 1 --2- 172.21.15.96:0/2369749678 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2b6c0039b0 0x7f2b880c94b0 secure :-1 s=READY pgs=1995 cs=0 l=1 rev1=1 crypto rx=0x7f2b78034040 tx=0x7f2b78004730 comp rx=0 tx=0).stop 2024-05-28T06:24:20.711 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.717+0000 7f2b8d800640 1 -- 172.21.15.96:0/2369749678 shutdown_connections 2024-05-28T06:24:20.711 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.717+0000 7f2b8d800640 1 --2- 172.21.15.96:0/2369749678 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2b6406e510 0x7f2b640709d0 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:20.711 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.717+0000 7f2b8d800640 1 --2- 172.21.15.96:0/2369749678 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2b880c4760 0x7f2b880c7330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:20.711 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.717+0000 7f2b8d800640 1 --2- 172.21.15.96:0/2369749678 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2b880cd2a0 0x7f2b880c6750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:20.711 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.717+0000 7f2b8d800640 1 --2- 172.21.15.96:0/2369749678 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2b6c0039b0 0x7f2b880c94b0 unknown :-1 s=CLOSED pgs=1995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:20.712 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.717+0000 7f2b8d800640 1 -- 172.21.15.96:0/2369749678 >> 172.21.15.96:0/2369749678 conn(0x7f2b8806bae0 msgr2=0x7f2b880af8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:20.712 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.717+0000 7f2b8d800640 1 -- 172.21.15.96:0/2369749678 shutdown_connections 2024-05-28T06:24:20.712 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.717+0000 7f2b8d800640 1 -- 172.21.15.96:0/2369749678 wait complete. 2024-05-28T06:24:20.730 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-05-28T06:24:20.730 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-05-28T06:24:20.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.937+0000 7f9138c00640 1 Processor -- start 2024-05-28T06:24:20.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.937+0000 7f9138c00640 1 -- start start 2024-05-28T06:24:20.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.938+0000 7f9138c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f91340cf7b0 0x7f91340d2380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:20.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.938+0000 7f9138c00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f913406d900 con 0x7f9134162080 2024-05-28T06:24:20.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.938+0000 7f9138c00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f913406dbe0 con 0x7f9134161ca0 2024-05-28T06:24:20.932 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.938+0000 7f9138c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f913406e510 con 0x7f91340cf7b0 2024-05-28T06:24:20.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.939+0000 7f9133e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f9134162080 0x7f91340d28c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:33772/0 (socket says 172.21.15.96:33772) 2024-05-28T06:24:20.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.939+0000 7f9133e00640 1 -- 172.21.15.96:0/1302684047 learned_addr learned my addr 172.21.15.96:0/1302684047 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:24:20.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.939+0000 7f912aa00640 1 --2- 172.21.15.96:0/1302684047 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f91340cf7b0 0x7f91340d2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:20.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.939+0000 7f9132a00640 1 -- 172.21.15.96:0/1302684047 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1578701185 0 0) 0x7f913406d900 con 0x7f9134162080 2024-05-28T06:24:20.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.939+0000 7f9132a00640 1 -- 172.21.15.96:0/1302684047 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9118003520 con 0x7f9134162080 2024-05-28T06:24:20.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.939+0000 7f9132a00640 1 -- 172.21.15.96:0/1302684047 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4068169143 0 0) 0x7f9118003520 con 0x7f9134162080 2024-05-28T06:24:20.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.939+0000 7f9132a00640 1 -- 172.21.15.96:0/1302684047 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f91340cf7b0 msgr2=0x7f91340d2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:20.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.939+0000 7f9132a00640 1 --2- 172.21.15.96:0/1302684047 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f91340cf7b0 0x7f91340d2380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:20.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.939+0000 7f9132a00640 1 -- 172.21.15.96:0/1302684047 >> v1:172.21.15.123:6789/0 conn(0x7f9134161ca0 legacy=0x7f91340cf0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:20.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.939+0000 7f9132a00640 1 -- 172.21.15.96:0/1302684047 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f913406dec0 con 0x7f9134162080 2024-05-28T06:24:20.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.939+0000 7f9132a00640 1 -- 172.21.15.96:0/1302684047 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f912c0019f0 con 0x7f9134162080 2024-05-28T06:24:20.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.939+0000 7f9132a00640 1 -- 172.21.15.96:0/1302684047 >> v1:172.21.15.96:6789/0 conn(0x7f9134162080 legacy=0x7f91340d28c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:20.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.940+0000 7f9132a00640 1 --2- 172.21.15.96:0/1302684047 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91180039e0 0x7f9118003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:20.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.940+0000 7f9132a00640 1 --2- 172.21.15.96:0/1302684047 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9118004800 0x7f9118004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:20.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.940+0000 7f9132a00640 1 --2- 172.21.15.96:0/1302684047 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f91340cf7b0 0x7f9118005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:20.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.940+0000 7f9132a00640 1 -- 172.21.15.96:0/1302684047 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f913406e510 con 0x7f9118004800 2024-05-28T06:24:20.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.940+0000 7f9132a00640 1 -- 172.21.15.96:0/1302684047 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f911800baf0 con 0x7f91180039e0 2024-05-28T06:24:20.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.940+0000 7f9132a00640 1 -- 172.21.15.96:0/1302684047 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f911800bcf0 con 0x7f91340cf7b0 2024-05-28T06:24:20.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.940+0000 7f912aa00640 1 --2- 172.21.15.96:0/1302684047 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9118004800 0x7f9118004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:20.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.940+0000 7f9133400640 1 --2- 172.21.15.96:0/1302684047 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91180039e0 0x7f9118003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:20.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.940+0000 7f912aa00640 1 -- 172.21.15.96:0/1302684047 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f91340cf7b0 msgr2=0x7f9118005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:20.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.940+0000 7f912aa00640 1 --2- 172.21.15.96:0/1302684047 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f91340cf7b0 0x7f9118005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:20.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.940+0000 7f912aa00640 1 -- 172.21.15.96:0/1302684047 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91180039e0 msgr2=0x7f9118003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:20.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.940+0000 7f912aa00640 1 --2- 172.21.15.96:0/1302684047 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91180039e0 0x7f9118003e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:20.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.940+0000 7f912aa00640 1 -- 172.21.15.96:0/1302684047 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9118003520 con 0x7f9118004800 2024-05-28T06:24:20.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.941+0000 7f912aa00640 1 --2- 172.21.15.96:0/1302684047 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9118004800 0x7f9118004c80 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 crypto rx=0x7f911c00ba30 tx=0x7f9118005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f11417baf7f714e server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:20.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.941+0000 7f9132a00640 1 -- 172.21.15.96:0/1302684047 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f911c030070 con 0x7f9118004800 2024-05-28T06:24:20.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.941+0000 7f9132a00640 1 -- 172.21.15.96:0/1302684047 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f911c02ae10 con 0x7f9118004800 2024-05-28T06:24:20.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.941+0000 7f9132a00640 1 -- 172.21.15.96:0/1302684047 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f911c00b600 con 0x7f9118004800 2024-05-28T06:24:20.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.942+0000 7f9138c00640 1 -- 172.21.15.96:0/1302684047 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9118004800 msgr2=0x7f9118004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:20.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.942+0000 7f9138c00640 1 --2- 172.21.15.96:0/1302684047 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9118004800 0x7f9118004c80 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 crypto rx=0x7f911c00ba30 tx=0x7f9118005830 comp rx=0 tx=0).stop 2024-05-28T06:24:20.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.942+0000 7f9133e00640 1 -- 172.21.15.96:0/1302684047 reap_dead start 2024-05-28T06:24:20.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.942+0000 7f9138c00640 1 -- 172.21.15.96:0/1302684047 shutdown_connections 2024-05-28T06:24:20.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.942+0000 7f9138c00640 1 -- 172.21.15.96:0/1302684047 >> 172.21.15.96:0/1302684047 conn(0x7f913409d7d0 msgr2=0x7f91340d4520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:20.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.942+0000 7f9138c00640 1 -- 172.21.15.96:0/1302684047 shutdown_connections 2024-05-28T06:24:20.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.942+0000 7f9138c00640 1 -- 172.21.15.96:0/1302684047 wait complete. 2024-05-28T06:24:20.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.944+0000 7f9138c00640 1 Processor -- start 2024-05-28T06:24:20.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.944+0000 7f9138c00640 1 -- start start 2024-05-28T06:24:20.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.945+0000 7f9138c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9118004800 0x7f9134156d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:20.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.945+0000 7f9138c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f91341f4690 0x7f913415c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:20.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.945+0000 7f9138c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f913415cdc0 0x7f9134159a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:20.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.945+0000 7f9138c00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f91340d4f00 con 0x7f9118004800 2024-05-28T06:24:20.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.945+0000 7f9138c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f91340d4d80 con 0x7f913415cdc0 2024-05-28T06:24:20.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.945+0000 7f9138c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f91340b3a80 con 0x7f91341f4690 2024-05-28T06:24:20.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.945+0000 7f9133400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9118004800 0x7f9134156d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:20.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.945+0000 7f9133400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9118004800 0x7f9134156d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:45322/0 (socket says 172.21.15.96:45322) 2024-05-28T06:24:20.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.945+0000 7f9133400640 1 -- 172.21.15.96:0/539698951 learned_addr learned my addr 172.21.15.96:0/539698951 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:24:20.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.945+0000 7f9133400640 1 -- 172.21.15.96:0/539698951 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f91341f4690 msgr2=0x7f913415c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:20.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.945+0000 7f9133400640 1 --2- 172.21.15.96:0/539698951 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f91341f4690 0x7f913415c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:20.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.945+0000 7f9133400640 1 -- 172.21.15.96:0/539698951 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f913415cdc0 msgr2=0x7f9134159a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:20.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.945+0000 7f9133400640 1 --2- 172.21.15.96:0/539698951 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f913415cdc0 0x7f9134159a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:20.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.945+0000 7f9133400640 1 -- 172.21.15.96:0/539698951 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f913415bc60 con 0x7f9118004800 2024-05-28T06:24:20.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.946+0000 7f9133400640 1 --2- 172.21.15.96:0/539698951 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9118004800 0x7f9134156d60 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7f912400d5a0 tx=0x7f912400da60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:20.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.946+0000 7f912be00640 1 -- 172.21.15.96:0/539698951 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9124005480 con 0x7f9118004800 2024-05-28T06:24:20.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.946+0000 7f912be00640 1 -- 172.21.15.96:0/539698951 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9124017070 con 0x7f9118004800 2024-05-28T06:24:20.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.946+0000 7f9138c00640 1 -- 172.21.15.96:0/539698951 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9134157770 con 0x7f9118004800 2024-05-28T06:24:20.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.946+0000 7f912be00640 1 -- 172.21.15.96:0/539698951 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9124003810 con 0x7f9118004800 2024-05-28T06:24:20.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.946+0000 7f9138c00640 1 -- 172.21.15.96:0/539698951 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9134157d30 con 0x7f9118004800 2024-05-28T06:24:20.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.947+0000 7f9138c00640 1 -- 172.21.15.96:0/539698951 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91340690f0 con 0x7f9118004800 2024-05-28T06:24:20.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.948+0000 7f912be00640 1 -- 172.21.15.96:0/539698951 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 42) ==== 93590+0+0 (secure 0 0 0) 0x7f91240039b0 con 0x7f9118004800 2024-05-28T06:24:20.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.949+0000 7f912be00640 1 --2- 172.21.15.96:0/539698951 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f911006e540 0x7f9110070a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:20.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.949+0000 7f912be00640 1 -- 172.21.15.96:0/539698951 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 4913+0+0 (secure 0 0 0) 0x7f912408a8b0 con 0x7f9118004800 2024-05-28T06:24:20.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.950+0000 7f912aa00640 1 --2- 172.21.15.96:0/539698951 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f911006e540 0x7f9110070a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:20.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.951+0000 7f912aa00640 1 --2- 172.21.15.96:0/539698951 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f911006e540 0x7f9110070a00 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f911c00ba00 tx=0x7f911c02c000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:20.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:20.953+0000 7f912be00640 1 -- 172.21.15.96:0/539698951 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f912405a740 con 0x7f9118004800 2024-05-28T06:24:21.145 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:21.151+0000 7f9138c00640 1 -- 172.21.15.96:0/539698951 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f91340d3430 con 0x7f9118004800 2024-05-28T06:24:21.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:21.950+0000 7f912be00640 1 -- 172.21.15.96:0/539698951 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v127) ==== 138+0+0 (secure 0 0 0) 0x7f912405a040 con 0x7f9118004800 2024-05-28T06:24:21.945 INFO:teuthology.orchestra.run.smithi096.stderr:pool 'cephfs_metadata' created 2024-05-28T06:24:21.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:21.953+0000 7f9138c00640 1 -- 172.21.15.96:0/539698951 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f911006e540 msgr2=0x7f9110070a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:21.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:21.953+0000 7f9138c00640 1 --2- 172.21.15.96:0/539698951 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f911006e540 0x7f9110070a00 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f911c00ba00 tx=0x7f911c02c000 comp rx=0 tx=0).stop 2024-05-28T06:24:21.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:21.953+0000 7f9138c00640 1 -- 172.21.15.96:0/539698951 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9118004800 msgr2=0x7f9134156d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:21.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:21.953+0000 7f9138c00640 1 --2- 172.21.15.96:0/539698951 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9118004800 0x7f9134156d60 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7f912400d5a0 tx=0x7f912400da60 comp rx=0 tx=0).stop 2024-05-28T06:24:21.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:21.953+0000 7f9138c00640 1 -- 172.21.15.96:0/539698951 shutdown_connections 2024-05-28T06:24:21.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:21.953+0000 7f9138c00640 1 --2- 172.21.15.96:0/539698951 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f911006e540 0x7f9110070a00 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:21.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:21.953+0000 7f9138c00640 1 --2- 172.21.15.96:0/539698951 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f913415cdc0 0x7f9134159a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:21.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:21.953+0000 7f9138c00640 1 --2- 172.21.15.96:0/539698951 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f91341f4690 0x7f913415c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:21.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:21.953+0000 7f9138c00640 1 --2- 172.21.15.96:0/539698951 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9118004800 0x7f9134156d60 unknown :-1 s=CLOSED pgs=1998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:21.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:21.954+0000 7f9138c00640 1 -- 172.21.15.96:0/539698951 >> 172.21.15.96:0/539698951 conn(0x7f913409d7d0 msgr2=0x7f91340749f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:21.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:21.954+0000 7f9138c00640 1 -- 172.21.15.96:0/539698951 shutdown_connections 2024-05-28T06:24:21.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:21.954+0000 7f9138c00640 1 -- 172.21.15.96:0/539698951 wait complete. 2024-05-28T06:24:21.976 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-05-28T06:24:22.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:22.183+0000 7fd21d600640 1 Processor -- start 2024-05-28T06:24:22.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:22.183+0000 7fd21d600640 1 -- start start 2024-05-28T06:24:22.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:22.184+0000 7fd21d600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fd21815b6f0 0x7fd21806e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:22.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:22.184+0000 7fd21d600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fd2180725e0 0x7fd2180ae650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:22.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:22.184+0000 7fd21d600640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2180ade90 con 0x7fd2180d0640 2024-05-28T06:24:22.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:22.184+0000 7fd21d600640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fd218071c40 con 0x7fd2180725e0 2024-05-28T06:24:22.180 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:22.184+0000 7fd21d600640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fd218071e70 con 0x7fd21815b6f0 2024-05-28T06:24:22.180 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:22.184+0000 7fd217400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fd2180725e0 0x7fd2180ae650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:22.180 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:22.185+0000 7fd216a00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fd2180d0640 0x7fd2180d0a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:33774/0 (socket says 172.21.15.96:33774) 2024-05-28T06:24:22.180 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:22.185+0000 7fd216a00640 1 -- 172.21.15.96:0/2567748138 learned_addr learned my addr 172.21.15.96:0/2567748138 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:24:22.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:22.185+0000 7fd20e000640 1 --2- 172.21.15.96:0/2567748138 >> v2:172.21.15.123:3300/0 conn(0x7fd21815b6f0 0x7fd21806e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:22.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:22.185+0000 7fd216000640 1 -- 172.21.15.96:0/2567748138 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1890035364 0 0) 0x7fd2180ade90 con 0x7fd2180d0640 2024-05-28T06:24:22.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:22.185+0000 7fd217400640 1 -- 172.21.15.96:0/2567748138 >> v2:172.21.15.123:3300/0 conn(0x7fd21815b6f0 msgr2=0x7fd21806e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:22.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:22.185+0000 7fd217400640 1 --2- 172.21.15.96:0/2567748138 >> v2:172.21.15.123:3300/0 conn(0x7fd21815b6f0 0x7fd21806e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:22.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:22.185+0000 7fd217400640 1 -- 172.21.15.96:0/2567748138 >> v1:172.21.15.96:6789/0 conn(0x7fd2180d0640 legacy=0x7fd2180d0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:22.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:22.185+0000 7fd217400640 1 -- 172.21.15.96:0/2567748138 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd218072130 con 0x7fd2180725e0 2024-05-28T06:24:22.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:22.185+0000 7fd20e000640 1 --2- 172.21.15.96:0/2567748138 >> v2:172.21.15.123:3300/0 conn(0x7fd21815b6f0 0x7fd21806e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:24:22.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:22.186+0000 7fd217400640 1 --2- 172.21.15.96:0/2567748138 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd2180725e0 0x7fd2180ae650 secure :-1 s=READY pgs=2000 cs=0 l=1 rev1=1 crypto rx=0x7fd2180d0220 tx=0x7fd21002a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ee00a7c9222670d server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:22.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:22.186+0000 7fd216000640 1 -- 172.21.15.96:0/2567748138 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd210007740 con 0x7fd2180725e0 2024-05-28T06:24:22.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:22.186+0000 7fd216000640 1 -- 172.21.15.96:0/2567748138 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2100078e0 con 0x7fd2180725e0 2024-05-28T06:24:22.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:22.187+0000 7fd216000640 1 -- 172.21.15.96:0/2567748138 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd210007cb0 con 0x7fd2180725e0 2024-05-28T06:24:22.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:22.187+0000 7fd21d600640 1 -- 172.21.15.96:0/2567748138 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd2180725e0 msgr2=0x7fd2180ae650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:22.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:22.187+0000 7fd21d600640 1 --2- 172.21.15.96:0/2567748138 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd2180725e0 0x7fd2180ae650 secure :-1 s=READY pgs=2000 cs=0 l=1 rev1=1 crypto rx=0x7fd2180d0220 tx=0x7fd21002a680 comp rx=0 tx=0).stop 2024-05-28T06:24:22.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:22.187+0000 7fd21d600640 1 -- 172.21.15.96:0/2567748138 shutdown_connections 2024-05-28T06:24:22.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:22.187+0000 7fd21d600640 1 --2- 172.21.15.96:0/2567748138 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd2180725e0 0x7fd2180ae650 unknown :-1 s=CLOSED pgs=2000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:22.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:22.187+0000 7fd21d600640 1 --2- 172.21.15.96:0/2567748138 >> v2:172.21.15.123:3300/0 conn(0x7fd21815b6f0 0x7fd21806e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:22.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:22.187+0000 7fd21d600640 1 -- 172.21.15.96:0/2567748138 >> 172.21.15.96:0/2567748138 conn(0x7fd21806bd20 msgr2=0x7fd21806c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:22.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:22.187+0000 7fd21d600640 1 -- 172.21.15.96:0/2567748138 shutdown_connections 2024-05-28T06:24:22.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:22.187+0000 7fd21d600640 1 -- 172.21.15.96:0/2567748138 wait complete. 2024-05-28T06:24:22.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:22.189+0000 7fd21d600640 1 Processor -- start 2024-05-28T06:24:22.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:22.189+0000 7fd21d600640 1 -- start start 2024-05-28T06:24:22.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:22.190+0000 7fd21d600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2180725e0 0x7fd2180cf2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:22.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:22.190+0000 7fd21d600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd2180d0640 0x7fd2181f39f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:22.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:22.190+0000 7fd21d600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd21815b6f0 0x7fd21815d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:22.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:22.190+0000 7fd21d600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd2180aeb90 con 0x7fd21815b6f0 2024-05-28T06:24:22.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:22.190+0000 7fd21d600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd21806e890 con 0x7fd2180725e0 2024-05-28T06:24:22.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:22.190+0000 7fd21d600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fd2180aed10 con 0x7fd2180d0640 2024-05-28T06:24:22.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:22.190+0000 7fd217400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd21815b6f0 0x7fd21815d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:22.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:22.190+0000 7fd217400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd21815b6f0 0x7fd21815d220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:45340/0 (socket says 172.21.15.96:45340) 2024-05-28T06:24:22.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:22.190+0000 7fd217400640 1 -- 172.21.15.96:0/2328611152 learned_addr learned my addr 172.21.15.96:0/2328611152 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:24:22.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:22.191+0000 7fd216a00640 1 --2- 172.21.15.96:0/2328611152 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2180725e0 0x7fd2180cf2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:22.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:22.191+0000 7fd20e000640 1 --2- 172.21.15.96:0/2328611152 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd2180d0640 0x7fd2181f39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:22.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:22.191+0000 7fd217400640 1 -- 172.21.15.96:0/2328611152 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd2180d0640 msgr2=0x7fd2181f39f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:22.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:22.191+0000 7fd217400640 1 --2- 172.21.15.96:0/2328611152 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd2180d0640 0x7fd2181f39f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:22.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:22.191+0000 7fd217400640 1 -- 172.21.15.96:0/2328611152 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2180725e0 msgr2=0x7fd2180cf2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:22.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:22.191+0000 7fd217400640 1 --2- 172.21.15.96:0/2328611152 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2180725e0 0x7fd2180cf2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:22.192 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:22.191+0000 7fd217400640 1 -- 172.21.15.96:0/2328611152 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd21815c180 con 0x7fd21815b6f0 2024-05-28T06:24:22.192 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:22.191+0000 7fd216a00640 1 --2- 172.21.15.96:0/2328611152 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2180725e0 0x7fd2180cf2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:24:22.192 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:22.191+0000 7fd217400640 1 --2- 172.21.15.96:0/2328611152 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd21815b6f0 0x7fd21815d220 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x7fd210009e80 tx=0x7fd210004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:22.192 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:22.192+0000 7fd20f400640 1 -- 172.21.15.96:0/2328611152 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd210040070 con 0x7fd21815b6f0 2024-05-28T06:24:22.192 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:22.192+0000 7fd20f400640 1 -- 172.21.15.96:0/2328611152 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd210031c40 con 0x7fd21815b6f0 2024-05-28T06:24:22.192 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:22.192+0000 7fd21d600640 1 -- 172.21.15.96:0/2328611152 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd21815c460 con 0x7fd21815b6f0 2024-05-28T06:24:22.193 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:22.192+0000 7fd20f400640 1 -- 172.21.15.96:0/2328611152 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd210003b90 con 0x7fd21815b6f0 2024-05-28T06:24:22.193 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:22.192+0000 7fd21d600640 1 -- 172.21.15.96:0/2328611152 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd21815ca20 con 0x7fd21815b6f0 2024-05-28T06:24:22.193 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:22.193+0000 7fd21d600640 1 -- 172.21.15.96:0/2328611152 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2180689e0 con 0x7fd21815b6f0 2024-05-28T06:24:22.193 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:22.194+0000 7fd20f400640 1 -- 172.21.15.96:0/2328611152 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 42) ==== 93590+0+0 (secure 0 0 0) 0x7fd210039040 con 0x7fd21815b6f0 2024-05-28T06:24:22.194 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:22.195+0000 7fd20f400640 1 --2- 172.21.15.96:0/2328611152 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd1f406e510 0x7fd1f40709d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:22.195 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:22.195+0000 7fd20f400640 1 -- 172.21.15.96:0/2328611152 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5283+0+0 (secure 0 0 0) 0x7fd2100ac2b0 con 0x7fd21815b6f0 2024-05-28T06:24:22.195 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:22.196+0000 7fd216a00640 1 --2- 172.21.15.96:0/2328611152 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd1f406e510 0x7fd1f40709d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:22.195 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:22.197+0000 7fd216a00640 1 --2- 172.21.15.96:0/2328611152 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd1f406e510 0x7fd1f40709d0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fd208000f60 tx=0x7fd208009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:22.195 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:22.199+0000 7fd20f400640 1 -- 172.21.15.96:0/2328611152 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd21007c2e0 con 0x7fd21815b6f0 2024-05-28T06:24:22.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:22.400+0000 7fd21d600640 1 -- 172.21.15.96:0/2328611152 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fd2180af660 con 0x7fd21815b6f0 2024-05-28T06:24:22.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:22.965+0000 7fd20f400640 1 -- 172.21.15.96:0/2328611152 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v128) ==== 170+0+0 (secure 0 0 0) 0x7fd21007bbe0 con 0x7fd21815b6f0 2024-05-28T06:24:22.960 INFO:teuthology.orchestra.run.smithi096.stderr:pool 'cephfs_data' created 2024-05-28T06:24:22.964 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:22.970+0000 7fd21d600640 1 -- 172.21.15.96:0/2328611152 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd1f406e510 msgr2=0x7fd1f40709d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:22.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:22.970+0000 7fd21d600640 1 --2- 172.21.15.96:0/2328611152 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd1f406e510 0x7fd1f40709d0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fd208000f60 tx=0x7fd208009040 comp rx=0 tx=0).stop 2024-05-28T06:24:22.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:22.970+0000 7fd21d600640 1 -- 172.21.15.96:0/2328611152 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd21815b6f0 msgr2=0x7fd21815d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:22.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:22.970+0000 7fd21d600640 1 --2- 172.21.15.96:0/2328611152 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd21815b6f0 0x7fd21815d220 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x7fd210009e80 tx=0x7fd210004a40 comp rx=0 tx=0).stop 2024-05-28T06:24:22.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:22.971+0000 7fd21d600640 1 -- 172.21.15.96:0/2328611152 shutdown_connections 2024-05-28T06:24:22.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:22.971+0000 7fd21d600640 1 --2- 172.21.15.96:0/2328611152 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd1f406e510 0x7fd1f40709d0 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:22.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:22.971+0000 7fd21d600640 1 --2- 172.21.15.96:0/2328611152 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd21815b6f0 0x7fd21815d220 unknown :-1 s=CLOSED pgs=2001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:22.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:22.971+0000 7fd21d600640 1 --2- 172.21.15.96:0/2328611152 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd2180d0640 0x7fd2181f39f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:22.966 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:22.971+0000 7fd21d600640 1 --2- 172.21.15.96:0/2328611152 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2180725e0 0x7fd2180cf2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:22.966 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:22.971+0000 7fd21d600640 1 -- 172.21.15.96:0/2328611152 >> 172.21.15.96:0/2328611152 conn(0x7fd21806bd20 msgr2=0x7fd21815a9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:22.966 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:22.971+0000 7fd21d600640 1 -- 172.21.15.96:0/2328611152 shutdown_connections 2024-05-28T06:24:22.966 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:22.971+0000 7fd21d600640 1 -- 172.21.15.96:0/2328611152 wait complete. 2024-05-28T06:24:23.003 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-05-28T06:24:23.174 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.179+0000 7fc29c200640 1 Processor -- start 2024-05-28T06:24:23.174 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.179+0000 7fc29c200640 1 -- start start 2024-05-28T06:24:23.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.181+0000 7fc29c200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fc2940d0640 0x7fc29415d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:23.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.181+0000 7fc29c200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc2940b32c0 0x7fc2940b3730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:23.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.181+0000 7fc29c200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2940d0060 con 0x7fc2940d0a20 2024-05-28T06:24:23.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.181+0000 7fc29c200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fc2940adbb0 con 0x7fc2940d0640 2024-05-28T06:24:23.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.181+0000 7fc29c200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fc29415db10 con 0x7fc2940b32c0 2024-05-28T06:24:23.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.181+0000 7fc298e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fc2940d0a20 0x7fc2940cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:33788/0 (socket says 172.21.15.96:33788) 2024-05-28T06:24:23.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.181+0000 7fc298e00640 1 -- 172.21.15.96:0/80640060 learned_addr learned my addr 172.21.15.96:0/80640060 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:24:23.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.181+0000 7fc293e00640 1 -- 172.21.15.96:0/80640060 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2970110305 0 0) 0x7fc2940d0060 con 0x7fc2940d0a20 2024-05-28T06:24:23.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.181+0000 7fc293e00640 1 -- 172.21.15.96:0/80640060 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc278003520 con 0x7fc2940d0a20 2024-05-28T06:24:23.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.181+0000 7fc299800640 1 --2- 172.21.15.96:0/80640060 >> v2:172.21.15.123:3300/0 conn(0x7fc2940d0640 0x7fc29415d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:23.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.181+0000 7fc29a200640 1 --2- 172.21.15.96:0/80640060 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc2940b32c0 0x7fc2940b3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:23.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.182+0000 7fc293e00640 1 -- 172.21.15.96:0/80640060 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3526779531 0 0) 0x7fc278003520 con 0x7fc2940d0a20 2024-05-28T06:24:23.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.182+0000 7fc293e00640 1 -- 172.21.15.96:0/80640060 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc2940b32c0 msgr2=0x7fc2940b3730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:23.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.182+0000 7fc293e00640 1 --2- 172.21.15.96:0/80640060 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc2940b32c0 0x7fc2940b3730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:23.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.182+0000 7fc293e00640 1 -- 172.21.15.96:0/80640060 >> v2:172.21.15.123:3300/0 conn(0x7fc2940d0640 msgr2=0x7fc29415d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:23.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.182+0000 7fc293e00640 1 --2- 172.21.15.96:0/80640060 >> v2:172.21.15.123:3300/0 conn(0x7fc2940d0640 0x7fc29415d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:23.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.182+0000 7fc293e00640 1 -- 172.21.15.96:0/80640060 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc29415cb10 con 0x7fc2940d0a20 2024-05-28T06:24:23.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.182+0000 7fc293e00640 1 -- 172.21.15.96:0/80640060 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7fc280001670 con 0x7fc2940d0a20 2024-05-28T06:24:23.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.182+0000 7fc293e00640 1 -- 172.21.15.96:0/80640060 >> v1:172.21.15.96:6789/0 conn(0x7fc2940d0a20 legacy=0x7fc2940cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:23.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.182+0000 7fc293e00640 1 --2- 172.21.15.96:0/80640060 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc2940b32c0 0x7fc2780039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:23.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.182+0000 7fc293e00640 1 --2- 172.21.15.96:0/80640060 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc278003fe0 0x7fc278004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:23.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.182+0000 7fc293e00640 1 --2- 172.21.15.96:0/80640060 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc278004d40 0x7fc2780051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:23.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.182+0000 7fc293e00640 1 -- 172.21.15.96:0/80640060 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc29415db10 con 0x7fc278003fe0 2024-05-28T06:24:23.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.182+0000 7fc293e00640 1 -- 172.21.15.96:0/80640060 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc27800ba70 con 0x7fc278004d40 2024-05-28T06:24:23.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.182+0000 7fc293e00640 1 -- 172.21.15.96:0/80640060 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fc27800bc40 con 0x7fc2940b32c0 2024-05-28T06:24:23.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.182+0000 7fc298e00640 1 --2- 172.21.15.96:0/80640060 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc278003fe0 0x7fc278004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:23.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.183+0000 7fc299800640 1 --2- 172.21.15.96:0/80640060 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc2940b32c0 0x7fc2780039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:23.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.183+0000 7fc29a200640 1 --2- 172.21.15.96:0/80640060 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc278004d40 0x7fc2780051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:23.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.183+0000 7fc298e00640 1 -- 172.21.15.96:0/80640060 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc2940b32c0 msgr2=0x7fc2780039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:23.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.183+0000 7fc298e00640 1 --2- 172.21.15.96:0/80640060 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc2940b32c0 0x7fc2780039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:23.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.183+0000 7fc298e00640 1 -- 172.21.15.96:0/80640060 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc278004d40 msgr2=0x7fc2780051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:23.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.183+0000 7fc298e00640 1 --2- 172.21.15.96:0/80640060 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc278004d40 0x7fc2780051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:23.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.183+0000 7fc298e00640 1 -- 172.21.15.96:0/80640060 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc278003520 con 0x7fc278003fe0 2024-05-28T06:24:23.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.184+0000 7fc298e00640 1 --2- 172.21.15.96:0/80640060 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc278003fe0 0x7fc278004800 secure :-1 s=READY pgs=2003 cs=0 l=1 rev1=1 crypto rx=0x7fc28000c290 tx=0x7fc278005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=956a9a24bdb3e281 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:23.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.184+0000 7fc293e00640 1 -- 172.21.15.96:0/80640060 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc280007d50 con 0x7fc278003fe0 2024-05-28T06:24:23.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.184+0000 7fc293e00640 1 -- 172.21.15.96:0/80640060 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc280005af0 con 0x7fc278003fe0 2024-05-28T06:24:23.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.184+0000 7fc293e00640 1 -- 172.21.15.96:0/80640060 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc280007070 con 0x7fc278003fe0 2024-05-28T06:24:23.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.184+0000 7fc29c200640 1 -- 172.21.15.96:0/80640060 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc278003fe0 msgr2=0x7fc278004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:23.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.184+0000 7fc29c200640 1 --2- 172.21.15.96:0/80640060 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc278003fe0 0x7fc278004800 secure :-1 s=READY pgs=2003 cs=0 l=1 rev1=1 crypto rx=0x7fc28000c290 tx=0x7fc278005830 comp rx=0 tx=0).stop 2024-05-28T06:24:23.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.185+0000 7fc29a200640 1 -- 172.21.15.96:0/80640060 reap_dead start 2024-05-28T06:24:23.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.185+0000 7fc29c200640 1 -- 172.21.15.96:0/80640060 shutdown_connections 2024-05-28T06:24:23.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.185+0000 7fc29c200640 1 -- 172.21.15.96:0/80640060 >> 172.21.15.96:0/80640060 conn(0x7fc29406bd20 msgr2=0x7fc29406c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:23.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.185+0000 7fc29c200640 1 -- 172.21.15.96:0/80640060 shutdown_connections 2024-05-28T06:24:23.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.185+0000 7fc29c200640 1 -- 172.21.15.96:0/80640060 wait complete. 2024-05-28T06:24:23.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.186+0000 7fc29c200640 1 Processor -- start 2024-05-28T06:24:23.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.187+0000 7fc29c200640 1 -- start start 2024-05-28T06:24:23.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.187+0000 7fc29c200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc278003fe0 0x7fc2940b2420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:23.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.187+0000 7fc29c200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc2940b0940 0x7fc294154780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:23.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.187+0000 7fc29c200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc29414d320 0x7fc29414d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:23.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.188+0000 7fc29c200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc29415cdd0 con 0x7fc278003fe0 2024-05-28T06:24:23.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.188+0000 7fc29c200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc2940ad8d0 con 0x7fc2940b0940 2024-05-28T06:24:23.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.188+0000 7fc29c200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fc29415d0b0 con 0x7fc29414d320 2024-05-28T06:24:23.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.188+0000 7fc299800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc278003fe0 0x7fc2940b2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:23.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.188+0000 7fc299800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc278003fe0 0x7fc2940b2420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:45370/0 (socket says 172.21.15.96:45370) 2024-05-28T06:24:23.185 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.188+0000 7fc299800640 1 -- 172.21.15.96:0/437586253 learned_addr learned my addr 172.21.15.96:0/437586253 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:24:23.185 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.188+0000 7fc298e00640 1 --2- 172.21.15.96:0/437586253 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc2940b0940 0x7fc294154780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:23.185 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.188+0000 7fc299800640 1 -- 172.21.15.96:0/437586253 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc29414d320 msgr2=0x7fc29414d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:23.185 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.188+0000 7fc299800640 1 --2- 172.21.15.96:0/437586253 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc29414d320 0x7fc29414d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:23.185 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.188+0000 7fc299800640 1 -- 172.21.15.96:0/437586253 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc2940b0940 msgr2=0x7fc294154780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:23.185 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.188+0000 7fc299800640 1 --2- 172.21.15.96:0/437586253 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc2940b0940 0x7fc294154780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:23.185 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.188+0000 7fc299800640 1 -- 172.21.15.96:0/437586253 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2940b0e20 con 0x7fc278003fe0 2024-05-28T06:24:23.185 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.188+0000 7fc299800640 1 --2- 172.21.15.96:0/437586253 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc278003fe0 0x7fc2940b2420 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7fc28400f600 tx=0x7fc28400fac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:23.185 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.189+0000 7fc291600640 1 -- 172.21.15.96:0/437586253 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc284019020 con 0x7fc278003fe0 2024-05-28T06:24:23.185 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.189+0000 7fc291600640 1 -- 172.21.15.96:0/437586253 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc284014870 con 0x7fc278003fe0 2024-05-28T06:24:23.185 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.189+0000 7fc29c200640 1 -- 172.21.15.96:0/437586253 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2940b1100 con 0x7fc278003fe0 2024-05-28T06:24:23.185 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.189+0000 7fc291600640 1 -- 172.21.15.96:0/437586253 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc284014c20 con 0x7fc278003fe0 2024-05-28T06:24:23.185 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.189+0000 7fc29c200640 1 -- 172.21.15.96:0/437586253 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2940b16c0 con 0x7fc278003fe0 2024-05-28T06:24:23.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.190+0000 7fc29c200640 1 -- 172.21.15.96:0/437586253 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2941f0140 con 0x7fc278003fe0 2024-05-28T06:24:23.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.191+0000 7fc291600640 1 -- 172.21.15.96:0/437586253 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 42) ==== 93590+0+0 (secure 0 0 0) 0x7fc28401c020 con 0x7fc278003fe0 2024-05-28T06:24:23.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.192+0000 7fc291600640 1 --2- 172.21.15.96:0/437586253 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc27406e510 0x7fc2740709d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:23.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.192+0000 7fc291600640 1 -- 172.21.15.96:0/437586253 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(128..128 src has 1..128) ==== 5665+0+0 (secure 0 0 0) 0x7fc28408b460 con 0x7fc278003fe0 2024-05-28T06:24:23.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.193+0000 7fc298e00640 1 --2- 172.21.15.96:0/437586253 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc27406e510 0x7fc2740709d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:23.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.194+0000 7fc298e00640 1 --2- 172.21.15.96:0/437586253 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc27406e510 0x7fc2740709d0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fc280033000 tx=0x7fc280009030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:23.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.197+0000 7fc291600640 1 -- 172.21.15.96:0/437586253 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc284023020 con 0x7fc278003fe0 2024-05-28T06:24:23.412 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.417+0000 7fc29c200640 1 -- 172.21.15.96:0/437586253 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fc2940689e0 con 0x7fc278003fe0 2024-05-28T06:24:23.964 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.969+0000 7fc291600640 1 -- 172.21.15.96:0/437586253 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 18 and data pool 19 v78) ==== 174+0+0 (secure 0 0 0) 0x7fc28405b2f0 con 0x7fc278003fe0 2024-05-28T06:24:23.964 INFO:teuthology.orchestra.run.smithi096.stderr:new fs with metadata pool 18 and data pool 19 2024-05-28T06:24:23.967 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.973+0000 7fc29c200640 1 -- 172.21.15.96:0/437586253 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc27406e510 msgr2=0x7fc2740709d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:23.968 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.973+0000 7fc29c200640 1 --2- 172.21.15.96:0/437586253 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc27406e510 0x7fc2740709d0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fc280033000 tx=0x7fc280009030 comp rx=0 tx=0).stop 2024-05-28T06:24:23.968 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.973+0000 7fc29c200640 1 -- 172.21.15.96:0/437586253 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc278003fe0 msgr2=0x7fc2940b2420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:23.968 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.973+0000 7fc29c200640 1 --2- 172.21.15.96:0/437586253 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc278003fe0 0x7fc2940b2420 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7fc28400f600 tx=0x7fc28400fac0 comp rx=0 tx=0).stop 2024-05-28T06:24:23.968 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.974+0000 7fc29c200640 1 -- 172.21.15.96:0/437586253 shutdown_connections 2024-05-28T06:24:23.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.974+0000 7fc29c200640 1 --2- 172.21.15.96:0/437586253 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc27406e510 0x7fc2740709d0 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:23.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.974+0000 7fc29c200640 1 --2- 172.21.15.96:0/437586253 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc29414d320 0x7fc29414d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:23.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.974+0000 7fc29c200640 1 --2- 172.21.15.96:0/437586253 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc2940b0940 0x7fc294154780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:23.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.974+0000 7fc29c200640 1 --2- 172.21.15.96:0/437586253 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc278003fe0 0x7fc2940b2420 unknown :-1 s=CLOSED pgs=2004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:23.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.974+0000 7fc29c200640 1 -- 172.21.15.96:0/437586253 >> 172.21.15.96:0/437586253 conn(0x7fc29406bd20 msgr2=0x7fc29406d9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:23.970 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.975+0000 7fc29c200640 1 -- 172.21.15.96:0/437586253 shutdown_connections 2024-05-28T06:24:23.970 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:23.975+0000 7fc29c200640 1 -- 172.21.15.96:0/437586253 wait complete. 2024-05-28T06:24:24.011 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:24:24.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.182+0000 7feb0c800640 1 Processor -- start 2024-05-28T06:24:24.177 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.183+0000 7feb0c800640 1 -- start start 2024-05-28T06:24:24.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.184+0000 7feb0c800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7feb04145ff0 0x7feb040b0690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:24.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.184+0000 7feb0c800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7feb040b5e80 0x7feb040b4f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:24.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.184+0000 7feb0c800640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb040b6380 con 0x7feb040b54b0 2024-05-28T06:24:24.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.184+0000 7feb0c800640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7feb041463d0 con 0x7feb040b5e80 2024-05-28T06:24:24.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.184+0000 7feb0c800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7feb040b4240 con 0x7feb04145ff0 2024-05-28T06:24:24.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.184+0000 7feb0a800640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7feb040b54b0 0x7feb0406b900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:33804/0 (socket says 172.21.15.96:33804) 2024-05-28T06:24:24.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.184+0000 7feb0a800640 1 -- 172.21.15.96:0/908824519 learned_addr learned my addr 172.21.15.96:0/908824519 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:24:24.181 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.185+0000 7feb09400640 1 --2- 172.21.15.96:0/908824519 >> v2:172.21.15.123:3300/0 conn(0x7feb040b5e80 0x7feb040b4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:24.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.185+0000 7feb09e00640 1 --2- 172.21.15.96:0/908824519 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7feb04145ff0 0x7feb040b0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:24.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.185+0000 7feb08a00640 1 -- 172.21.15.96:0/908824519 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3304847727 0 0) 0x7feb040b6380 con 0x7feb040b54b0 2024-05-28T06:24:24.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.185+0000 7feb08a00640 1 -- 172.21.15.96:0/908824519 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feaec003520 con 0x7feb040b54b0 2024-05-28T06:24:24.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.185+0000 7feb08a00640 1 -- 172.21.15.96:0/908824519 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2296459039 0 0) 0x7feaec003520 con 0x7feb040b54b0 2024-05-28T06:24:24.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.185+0000 7feb08a00640 1 -- 172.21.15.96:0/908824519 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7feb04145ff0 msgr2=0x7feb040b0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:24.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.185+0000 7feb08a00640 1 --2- 172.21.15.96:0/908824519 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7feb04145ff0 0x7feb040b0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:24.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.185+0000 7feb08a00640 1 -- 172.21.15.96:0/908824519 >> v2:172.21.15.123:3300/0 conn(0x7feb040b5e80 msgr2=0x7feb040b4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:24.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.185+0000 7feb08a00640 1 --2- 172.21.15.96:0/908824519 >> v2:172.21.15.123:3300/0 conn(0x7feb040b5e80 0x7feb040b4f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:24.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.185+0000 7feb08a00640 1 -- 172.21.15.96:0/908824519 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb040b4490 con 0x7feb040b54b0 2024-05-28T06:24:24.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.185+0000 7feb08a00640 1 -- 172.21.15.96:0/908824519 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7feb000019f0 con 0x7feb040b54b0 2024-05-28T06:24:24.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.185+0000 7feb09400640 1 --2- 172.21.15.96:0/908824519 >> v2:172.21.15.123:3300/0 conn(0x7feb040b5e80 0x7feb040b4f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-28T06:24:24.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.185+0000 7feb08a00640 1 -- 172.21.15.96:0/908824519 >> v1:172.21.15.96:6789/0 conn(0x7feb040b54b0 legacy=0x7feb0406b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:24.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.185+0000 7feb08a00640 1 --2- 172.21.15.96:0/908824519 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feaec0039b0 0x7feaec003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:24.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.185+0000 7feb08a00640 1 --2- 172.21.15.96:0/908824519 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feaec004800 0x7feaec004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:24.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.185+0000 7feb08a00640 1 --2- 172.21.15.96:0/908824519 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7feb04145ff0 0x7feaec0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:24.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.185+0000 7feb08a00640 1 -- 172.21.15.96:0/908824519 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7feb041463d0 con 0x7feaec0039b0 2024-05-28T06:24:24.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.185+0000 7feb08a00640 1 -- 172.21.15.96:0/908824519 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7feb040b4240 con 0x7feaec004800 2024-05-28T06:24:24.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.185+0000 7feb08a00640 1 -- 172.21.15.96:0/908824519 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7feaec00baa0 con 0x7feb04145ff0 2024-05-28T06:24:24.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.185+0000 7feb09e00640 1 --2- 172.21.15.96:0/908824519 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feaec0039b0 0x7feaec003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:24.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.186+0000 7feb09400640 1 --2- 172.21.15.96:0/908824519 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feaec004800 0x7feaec004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:24.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.186+0000 7feb09e00640 1 -- 172.21.15.96:0/908824519 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7feb04145ff0 msgr2=0x7feaec0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:24.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.186+0000 7feb09e00640 1 --2- 172.21.15.96:0/908824519 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7feb04145ff0 0x7feaec0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:24.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.186+0000 7feb09e00640 1 -- 172.21.15.96:0/908824519 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feaec004800 msgr2=0x7feaec004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:24.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.186+0000 7feb09e00640 1 --2- 172.21.15.96:0/908824519 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feaec004800 0x7feaec004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:24.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.186+0000 7feb09e00640 1 -- 172.21.15.96:0/908824519 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feaec003520 con 0x7feaec0039b0 2024-05-28T06:24:24.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.187+0000 7feb09e00640 1 --2- 172.21.15.96:0/908824519 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feaec0039b0 0x7feaec003df0 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7feaf8009a90 tx=0x7feaec005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=538d2980630c1cf2 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:24.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.187+0000 7feb08a00640 1 -- 172.21.15.96:0/908824519 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaf8036070 con 0x7feaec0039b0 2024-05-28T06:24:24.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.187+0000 7feb08a00640 1 -- 172.21.15.96:0/908824519 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feaf8031400 con 0x7feaec0039b0 2024-05-28T06:24:24.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.187+0000 7feb08a00640 1 -- 172.21.15.96:0/908824519 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaf8009690 con 0x7feaec0039b0 2024-05-28T06:24:24.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.188+0000 7feb0c800640 1 -- 172.21.15.96:0/908824519 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feaec0039b0 msgr2=0x7feaec003df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:24.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.188+0000 7feb0c800640 1 --2- 172.21.15.96:0/908824519 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feaec0039b0 0x7feaec003df0 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7feaf8009a90 tx=0x7feaec005830 comp rx=0 tx=0).stop 2024-05-28T06:24:24.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.188+0000 7feb0a800640 1 -- 172.21.15.96:0/908824519 reap_dead start 2024-05-28T06:24:24.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.188+0000 7feb0c800640 1 -- 172.21.15.96:0/908824519 shutdown_connections 2024-05-28T06:24:24.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.188+0000 7feb0c800640 1 -- 172.21.15.96:0/908824519 >> 172.21.15.96:0/908824519 conn(0x7feb040b6760 msgr2=0x7feb040b6b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:24.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.188+0000 7feb0c800640 1 -- 172.21.15.96:0/908824519 shutdown_connections 2024-05-28T06:24:24.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.188+0000 7feb0c800640 1 -- 172.21.15.96:0/908824519 wait complete. 2024-05-28T06:24:24.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.189+0000 7feb0c800640 1 Processor -- start 2024-05-28T06:24:24.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.190+0000 7feb0c800640 1 -- start start 2024-05-28T06:24:24.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.190+0000 7feb0c800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feaec0039b0 0x7feb041f44b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:24.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.190+0000 7feb0c800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7feb0415bf00 0x7feb0413cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:24.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.190+0000 7feb0c800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feb0413d040 0x7feb040b2540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:24.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.191+0000 7feb0c800640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7feb0406c010 con 0x7feb0413d040 2024-05-28T06:24:24.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.191+0000 7feb0c800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7feb040b4ae0 con 0x7feaec0039b0 2024-05-28T06:24:24.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.191+0000 7feb0c800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7feb0406c190 con 0x7feb0415bf00 2024-05-28T06:24:24.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.191+0000 7feb0a800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feb0413d040 0x7feb040b2540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:24.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.191+0000 7feb0a800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feb0413d040 0x7feb040b2540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:45394/0 (socket says 172.21.15.96:45394) 2024-05-28T06:24:24.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.191+0000 7feb0a800640 1 -- 172.21.15.96:0/3596760037 learned_addr learned my addr 172.21.15.96:0/3596760037 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:24:24.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.191+0000 7feb09400640 1 --2- 172.21.15.96:0/3596760037 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7feb0415bf00 0x7feb0413cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:24.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.191+0000 7feb09e00640 1 --2- 172.21.15.96:0/3596760037 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feaec0039b0 0x7feb041f44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:24.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.191+0000 7feb0a800640 1 -- 172.21.15.96:0/3596760037 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7feb0415bf00 msgr2=0x7feb0413cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:24.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.191+0000 7feb0a800640 1 --2- 172.21.15.96:0/3596760037 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7feb0415bf00 0x7feb0413cb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:24.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.191+0000 7feb0a800640 1 -- 172.21.15.96:0/3596760037 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feaec0039b0 msgr2=0x7feb041f44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:24.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.191+0000 7feb0a800640 1 --2- 172.21.15.96:0/3596760037 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feaec0039b0 0x7feb041f44b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:24.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.191+0000 7feb0a800640 1 -- 172.21.15.96:0/3596760037 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb040b3220 con 0x7feb0413d040 2024-05-28T06:24:24.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.192+0000 7feb0a800640 1 --2- 172.21.15.96:0/3596760037 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feb0413d040 0x7feb040b2540 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 crypto rx=0x7feb040b5890 tx=0x7feb0000cb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:24.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.192+0000 7feafe000640 1 -- 172.21.15.96:0/3596760037 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb00004d60 con 0x7feb0413d040 2024-05-28T06:24:24.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.192+0000 7feafe000640 1 -- 172.21.15.96:0/3596760037 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb0000a640 con 0x7feb0413d040 2024-05-28T06:24:24.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.192+0000 7feafe000640 1 -- 172.21.15.96:0/3596760037 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb0000aa10 con 0x7feb0413d040 2024-05-28T06:24:24.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.192+0000 7feb0c800640 1 -- 172.21.15.96:0/3596760037 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb040b3500 con 0x7feb0413d040 2024-05-28T06:24:24.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.192+0000 7feb0c800640 1 -- 172.21.15.96:0/3596760037 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb0413fe90 con 0x7feb0413d040 2024-05-28T06:24:24.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.193+0000 7feb0c800640 1 -- 172.21.15.96:0/3596760037 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb04140090 con 0x7feb0413d040 2024-05-28T06:24:24.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.194+0000 7feafe000640 1 -- 172.21.15.96:0/3596760037 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 42) ==== 93590+0+0 (secure 0 0 0) 0x7feb0000abb0 con 0x7feb0413d040 2024-05-28T06:24:24.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.195+0000 7feafe000640 1 --2- 172.21.15.96:0/3596760037 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7feae806e540 0x7feae8070a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:24.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.195+0000 7feafe000640 1 -- 172.21.15.96:0/3596760037 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 5765+0+0 (secure 0 0 0) 0x7feb0008a3d0 con 0x7feb0413d040 2024-05-28T06:24:24.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.196+0000 7feb09e00640 1 --2- 172.21.15.96:0/3596760037 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7feae806e540 0x7feae8070a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:24.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.197+0000 7feb09e00640 1 --2- 172.21.15.96:0/3596760037 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7feae806e540 0x7feae8070a00 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7feaf8033000 tx=0x7feaf8008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:24.194 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.199+0000 7feafe000640 1 -- 172.21.15.96:0/3596760037 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7feb04140090 con 0x7feb0413d040 2024-05-28T06:24:24.380 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.385+0000 7feb0c800640 1 -- 172.21.15.96:0/3596760037 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7feb040690f0 con 0x7feb0413d040 2024-05-28T06:24:24.382 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.387+0000 7feafe000640 1 -- 172.21.15.96:0/3596760037 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v129) ==== 74+0+16792 (secure 0 0 0) 0x7feb0005a260 con 0x7feb0413d040 2024-05-28T06:24:24.383 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:24:24.383 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":129,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:24:23.420062+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-05-28T06:24:21.166134+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"129","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-05-28T06:24:22.415938+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"129","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":13,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:24:24.386 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.390+0000 7feb0c800640 1 -- 172.21.15.96:0/3596760037 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7feae806e540 msgr2=0x7feae8070a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:24.386 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.390+0000 7feb0c800640 1 --2- 172.21.15.96:0/3596760037 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7feae806e540 0x7feae8070a00 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7feaf8033000 tx=0x7feaf8008030 comp rx=0 tx=0).stop 2024-05-28T06:24:24.386 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.390+0000 7feb0c800640 1 -- 172.21.15.96:0/3596760037 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feb0413d040 msgr2=0x7feb040b2540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:24.387 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.390+0000 7feb0c800640 1 --2- 172.21.15.96:0/3596760037 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feb0413d040 0x7feb040b2540 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 crypto rx=0x7feb040b5890 tx=0x7feb0000cb60 comp rx=0 tx=0).stop 2024-05-28T06:24:24.387 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.390+0000 7feb0c800640 1 -- 172.21.15.96:0/3596760037 shutdown_connections 2024-05-28T06:24:24.387 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.390+0000 7feb0c800640 1 --2- 172.21.15.96:0/3596760037 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7feae806e540 0x7feae8070a00 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:24.387 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.390+0000 7feb0c800640 1 --2- 172.21.15.96:0/3596760037 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feb0413d040 0x7feb040b2540 unknown :-1 s=CLOSED pgs=2007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:24.387 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.390+0000 7feb0c800640 1 --2- 172.21.15.96:0/3596760037 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7feb0415bf00 0x7feb0413cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:24.387 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.390+0000 7feb0c800640 1 --2- 172.21.15.96:0/3596760037 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feaec0039b0 0x7feb041f44b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:24.387 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.390+0000 7feb0c800640 1 -- 172.21.15.96:0/3596760037 >> 172.21.15.96:0/3596760037 conn(0x7feb040b6760 msgr2=0x7feb04143340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:24.387 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.390+0000 7feb0c800640 1 -- 172.21.15.96:0/3596760037 shutdown_connections 2024-05-28T06:24:24.387 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.390+0000 7feb0c800640 1 -- 172.21.15.96:0/3596760037 wait complete. 2024-05-28T06:24:24.403 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:24:24.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.586+0000 7fc7f8200640 1 Processor -- start 2024-05-28T06:24:24.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.586+0000 7fc7f8200640 1 -- start start 2024-05-28T06:24:24.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.588+0000 7fc7f8200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fc7f00b4e10 0x7fc7f00cde30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:24.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.588+0000 7fc7f8200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7f00b51f0 con 0x7fc7f00d0ca0 2024-05-28T06:24:24.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.588+0000 7fc7f8200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7f00aef60 con 0x7fc7f00ce370 2024-05-28T06:24:24.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.588+0000 7fc7f8200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fc7f00ad1d0 con 0x7fc7f00b4e10 2024-05-28T06:24:24.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.588+0000 7fc7f5800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fc7f00b4e10 0x7fc7f00cde30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:24.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.588+0000 7fc7f5800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fc7f00b4e10 0x7fc7f00cde30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:45404/0 (socket says 172.21.15.96:45404) 2024-05-28T06:24:24.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.588+0000 7fc7f5800640 1 -- 172.21.15.96:0/3552304172 learned_addr learned my addr 172.21.15.96:0/3552304172 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:24:24.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.588+0000 7fc7f5800640 1 -- 172.21.15.96:0/3552304172 >> v1:172.21.15.123:6789/0 conn(0x7fc7f00ce370 legacy=0x7fc7f00d0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:24.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.588+0000 7fc7f5800640 1 -- 172.21.15.96:0/3552304172 >> v1:172.21.15.96:6789/0 conn(0x7fc7f00d0ca0 legacy=0x7fc7f0099090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:24.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.589+0000 7fc7f5800640 1 -- 172.21.15.96:0/3552304172 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7f00ad470 con 0x7fc7f00b4e10 2024-05-28T06:24:24.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.590+0000 7fc7f5800640 1 --2- 172.21.15.96:0/3552304172 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc7f00b4e10 0x7fc7f00cde30 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 crypto rx=0x7fc7e400bb10 tx=0x7fc7e402c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac247fc8eff388de server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:24.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.590+0000 7fc7efe00640 1 -- 172.21.15.96:0/3552304172 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7e40325a0 con 0x7fc7f00b4e10 2024-05-28T06:24:24.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.590+0000 7fc7efe00640 1 -- 172.21.15.96:0/3552304172 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7e4038070 con 0x7fc7f00b4e10 2024-05-28T06:24:24.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.590+0000 7fc7efe00640 1 -- 172.21.15.96:0/3552304172 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7e4032960 con 0x7fc7f00b4e10 2024-05-28T06:24:24.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.590+0000 7fc7f8200640 1 -- 172.21.15.96:0/3552304172 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc7f00b4e10 msgr2=0x7fc7f00cde30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:24.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.590+0000 7fc7f8200640 1 --2- 172.21.15.96:0/3552304172 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc7f00b4e10 0x7fc7f00cde30 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 crypto rx=0x7fc7e400bb10 tx=0x7fc7e402c8e0 comp rx=0 tx=0).stop 2024-05-28T06:24:24.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.591+0000 7fc7f8200640 1 -- 172.21.15.96:0/3552304172 shutdown_connections 2024-05-28T06:24:24.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.591+0000 7fc7f8200640 1 --2- 172.21.15.96:0/3552304172 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc7f00b4e10 0x7fc7f00cde30 unknown :-1 s=CLOSED pgs=2009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:24.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.591+0000 7fc7f8200640 1 -- 172.21.15.96:0/3552304172 >> 172.21.15.96:0/3552304172 conn(0x7fc7f006bae0 msgr2=0x7fc7f006bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:24.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.591+0000 7fc7f8200640 1 -- 172.21.15.96:0/3552304172 shutdown_connections 2024-05-28T06:24:24.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.591+0000 7fc7f8200640 1 -- 172.21.15.96:0/3552304172 wait complete. 2024-05-28T06:24:24.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.592+0000 7fc7f8200640 1 Processor -- start 2024-05-28T06:24:24.588 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.592+0000 7fc7f8200640 1 -- start start 2024-05-28T06:24:24.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.593+0000 7fc7f8200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc7f00b4e10 0x7fc7f00c58d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:24.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.593+0000 7fc7f8200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc7f00ce370 0x7fc7f00c3d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:24.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.593+0000 7fc7f8200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc7f00d0ca0 0x7fc7f00b0010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:24.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.593+0000 7fc7f8200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc7f00cea00 con 0x7fc7f00b4e10 2024-05-28T06:24:24.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.593+0000 7fc7f8200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc7f00ce880 con 0x7fc7f00ce370 2024-05-28T06:24:24.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.593+0000 7fc7f8200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fc7f00b3150 con 0x7fc7f00d0ca0 2024-05-28T06:24:24.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.593+0000 7fc7f5800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc7f00b4e10 0x7fc7f00c58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:24.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.593+0000 7fc7f5800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc7f00b4e10 0x7fc7f00c58d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:45412/0 (socket says 172.21.15.96:45412) 2024-05-28T06:24:24.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.593+0000 7fc7f5800640 1 -- 172.21.15.96:0/803111973 learned_addr learned my addr 172.21.15.96:0/803111973 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:24:24.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.594+0000 7fc7f5800640 1 -- 172.21.15.96:0/803111973 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc7f00d0ca0 msgr2=0x7fc7f00b0010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:24.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.594+0000 7fc7f5800640 1 --2- 172.21.15.96:0/803111973 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc7f00d0ca0 0x7fc7f00b0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:24.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.594+0000 7fc7f5800640 1 -- 172.21.15.96:0/803111973 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc7f00ce370 msgr2=0x7fc7f00c3d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:24.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.594+0000 7fc7f5800640 1 --2- 172.21.15.96:0/803111973 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc7f00ce370 0x7fc7f00c3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:24.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.594+0000 7fc7f5800640 1 -- 172.21.15.96:0/803111973 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7f00c4850 con 0x7fc7f00b4e10 2024-05-28T06:24:24.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.594+0000 7fc7f5800640 1 --2- 172.21.15.96:0/803111973 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc7f00b4e10 0x7fc7f00c58d0 secure :-1 s=READY pgs=2010 cs=0 l=1 rev1=1 crypto rx=0x7fc7e400b2c0 tx=0x7fc7e4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:24.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.594+0000 7fc7ed600640 1 -- 172.21.15.96:0/803111973 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7e4043070 con 0x7fc7f00b4e10 2024-05-28T06:24:24.594 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.594+0000 7fc7ed600640 1 -- 172.21.15.96:0/803111973 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7e4033c90 con 0x7fc7f00b4e10 2024-05-28T06:24:24.594 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.594+0000 7fc7f8200640 1 -- 172.21.15.96:0/803111973 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7f00c4b30 con 0x7fc7f00b4e10 2024-05-28T06:24:24.594 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.594+0000 7fc7ed600640 1 -- 172.21.15.96:0/803111973 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7e403c660 con 0x7fc7f00b4e10 2024-05-28T06:24:24.594 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.594+0000 7fc7f8200640 1 -- 172.21.15.96:0/803111973 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7f00c50f0 con 0x7fc7f00b4e10 2024-05-28T06:24:24.594 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.596+0000 7fc7f8200640 1 -- 172.21.15.96:0/803111973 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7f00689e0 con 0x7fc7f00b4e10 2024-05-28T06:24:24.594 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.596+0000 7fc7ed600640 1 -- 172.21.15.96:0/803111973 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 42) ==== 93590+0+0 (secure 0 0 0) 0x7fc7e4046070 con 0x7fc7f00b4e10 2024-05-28T06:24:24.595 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.597+0000 7fc7ed600640 1 --2- 172.21.15.96:0/803111973 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc7d006e510 0x7fc7d00709d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:24.595 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.597+0000 7fc7ed600640 1 -- 172.21.15.96:0/803111973 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 5765+0+0 (secure 0 0 0) 0x7fc7e4030020 con 0x7fc7f00b4e10 2024-05-28T06:24:24.595 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.598+0000 7fc7f4e00640 1 --2- 172.21.15.96:0/803111973 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc7d006e510 0x7fc7d00709d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:24.595 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.599+0000 7fc7f4e00640 1 --2- 172.21.15.96:0/803111973 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc7d006e510 0x7fc7d00709d0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fc7dc001560 tx=0x7fc7dc007990 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:24.596 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.601+0000 7fc7ed600640 1 -- 172.21.15.96:0/803111973 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc7e404d030 con 0x7fc7f00b4e10 2024-05-28T06:24:24.798 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.804+0000 7fc7f8200640 1 -- 172.21.15.96:0/803111973 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc7f006f540 con 0x7fc7f00b4e10 2024-05-28T06:24:24.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.806+0000 7fc7ed600640 1 -- 172.21.15.96:0/803111973 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v129) ==== 74+0+16792 (secure 0 0 0) 0x7fc7e407b5c0 con 0x7fc7f00b4e10 2024-05-28T06:24:24.801 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:24:24.801 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":129,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:24:23.420062+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-05-28T06:24:21.166134+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"129","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-05-28T06:24:22.415938+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"129","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":13,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:24:24.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.808+0000 7fc7f8200640 1 -- 172.21.15.96:0/803111973 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc7d006e510 msgr2=0x7fc7d00709d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:24.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.808+0000 7fc7f8200640 1 --2- 172.21.15.96:0/803111973 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc7d006e510 0x7fc7d00709d0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fc7dc001560 tx=0x7fc7dc007990 comp rx=0 tx=0).stop 2024-05-28T06:24:24.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.808+0000 7fc7f8200640 1 -- 172.21.15.96:0/803111973 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc7f00b4e10 msgr2=0x7fc7f00c58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:24.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.808+0000 7fc7f8200640 1 --2- 172.21.15.96:0/803111973 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc7f00b4e10 0x7fc7f00c58d0 secure :-1 s=READY pgs=2010 cs=0 l=1 rev1=1 crypto rx=0x7fc7e400b2c0 tx=0x7fc7e4004730 comp rx=0 tx=0).stop 2024-05-28T06:24:24.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.808+0000 7fc7f8200640 1 -- 172.21.15.96:0/803111973 shutdown_connections 2024-05-28T06:24:24.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.808+0000 7fc7f8200640 1 --2- 172.21.15.96:0/803111973 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc7d006e510 0x7fc7d00709d0 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:24.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.808+0000 7fc7f8200640 1 --2- 172.21.15.96:0/803111973 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc7f00d0ca0 0x7fc7f00b0010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:24.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.808+0000 7fc7f8200640 1 --2- 172.21.15.96:0/803111973 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc7f00ce370 0x7fc7f00c3d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:24.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.808+0000 7fc7f8200640 1 --2- 172.21.15.96:0/803111973 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc7f00b4e10 0x7fc7f00c58d0 unknown :-1 s=CLOSED pgs=2010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:24.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.809+0000 7fc7f8200640 1 -- 172.21.15.96:0/803111973 >> 172.21.15.96:0/803111973 conn(0x7fc7f006bae0 msgr2=0x7fc7f00ca8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:24.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.809+0000 7fc7f8200640 1 -- 172.21.15.96:0/803111973 shutdown_connections 2024-05-28T06:24:24.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:24.809+0000 7fc7f8200640 1 -- 172.21.15.96:0/803111973 wait complete. 2024-05-28T06:24:24.823 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-05-28T06:24:25.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:25.040+0000 7fd8d7c00640 1 Processor -- start 2024-05-28T06:24:25.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:25.041+0000 7fd8d7c00640 1 -- start start 2024-05-28T06:24:25.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:25.042+0000 7fd8d7c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd8d0159ff0 0x7fd8d015d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:25.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:25.042+0000 7fd8d7c00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fd8d00b5d20 0x7fd8d00b4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:25.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:25.042+0000 7fd8d7c00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8d006c360 con 0x7fd8d006b900 2024-05-28T06:24:25.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:25.042+0000 7fd8d7c00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fd8d00b4650 con 0x7fd8d00b5d20 2024-05-28T06:24:25.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:25.042+0000 7fd8d7c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fd8d00b0080 con 0x7fd8d0159ff0 2024-05-28T06:24:25.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:25.043+0000 7fd8d5c00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fd8d006b900 0x7fd8d00add70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:33812/0 (socket says 172.21.15.96:33812) 2024-05-28T06:24:25.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:25.043+0000 7fd8c7e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fd8d00b5d20 0x7fd8d00b4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:25.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:25.043+0000 7fd8d5c00640 1 -- 172.21.15.96:0/3605327165 learned_addr learned my addr 172.21.15.96:0/3605327165 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:24:25.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:25.043+0000 7fd8d5200640 1 --2- 172.21.15.96:0/3605327165 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd8d0159ff0 0x7fd8d015d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:25.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:25.043+0000 7fd8c7e00640 1 -- 172.21.15.96:0/3605327165 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd8d0159ff0 msgr2=0x7fd8d015d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:25.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:25.043+0000 7fd8c7e00640 1 --2- 172.21.15.96:0/3605327165 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd8d0159ff0 0x7fd8d015d5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:25.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:25.043+0000 7fd8c7e00640 1 -- 172.21.15.96:0/3605327165 >> v1:172.21.15.96:6789/0 conn(0x7fd8d006b900 legacy=0x7fd8d00add70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:25.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:25.043+0000 7fd8c7e00640 1 -- 172.21.15.96:0/3605327165 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8d00b0360 con 0x7fd8d00b5d20 2024-05-28T06:24:25.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:25.043+0000 7fd8cfe00640 1 -- 172.21.15.96:0/3605327165 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2205739212 0 0) 0x7fd8d006c360 con 0x7fd8d006b900 2024-05-28T06:24:25.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:25.044+0000 7fd8c7e00640 1 --2- 172.21.15.96:0/3605327165 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd8d00b5d20 0x7fd8d00b4ae0 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 crypto rx=0x7fd8b80097a0 tx=0x7fd8b802a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f35bbf2c4b5cd420 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:25.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:25.044+0000 7fd8cfe00640 1 -- 172.21.15.96:0/3605327165 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b8037070 con 0x7fd8d00b5d20 2024-05-28T06:24:25.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:25.044+0000 7fd8cfe00640 1 -- 172.21.15.96:0/3605327165 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8b8032440 con 0x7fd8d00b5d20 2024-05-28T06:24:25.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:25.044+0000 7fd8cfe00640 1 -- 172.21.15.96:0/3605327165 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b8032810 con 0x7fd8d00b5d20 2024-05-28T06:24:25.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:25.045+0000 7fd8d7c00640 1 -- 172.21.15.96:0/3605327165 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd8d00b5d20 msgr2=0x7fd8d00b4ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:25.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:25.045+0000 7fd8d7c00640 1 --2- 172.21.15.96:0/3605327165 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd8d00b5d20 0x7fd8d00b4ae0 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 crypto rx=0x7fd8b80097a0 tx=0x7fd8b802a180 comp rx=0 tx=0).stop 2024-05-28T06:24:25.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:25.045+0000 7fd8d7c00640 1 -- 172.21.15.96:0/3605327165 shutdown_connections 2024-05-28T06:24:25.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:25.045+0000 7fd8d7c00640 1 --2- 172.21.15.96:0/3605327165 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd8d00b5d20 0x7fd8d00b4ae0 unknown :-1 s=CLOSED pgs=2012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:25.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:25.045+0000 7fd8d7c00640 1 --2- 172.21.15.96:0/3605327165 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd8d0159ff0 0x7fd8d015d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:25.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:25.045+0000 7fd8d7c00640 1 -- 172.21.15.96:0/3605327165 >> 172.21.15.96:0/3605327165 conn(0x7fd8d00b3240 msgr2=0x7fd8d00d0b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:25.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:25.045+0000 7fd8d7c00640 1 -- 172.21.15.96:0/3605327165 shutdown_connections 2024-05-28T06:24:25.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:25.045+0000 7fd8d7c00640 1 -- 172.21.15.96:0/3605327165 wait complete. 2024-05-28T06:24:25.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:25.047+0000 7fd8d7c00640 1 Processor -- start 2024-05-28T06:24:25.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:25.047+0000 7fd8d7c00640 1 -- start start 2024-05-28T06:24:25.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:25.048+0000 7fd8d7c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd8d006b900 0x7fd8d015f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:25.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:25.048+0000 7fd8d7c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd8d00b5d20 0x7fd8d015e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:25.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:25.048+0000 7fd8d7c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd8d0159ff0 0x7fd8d015ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:25.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:25.048+0000 7fd8d7c00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd8d015a3d0 con 0x7fd8d00b5d20 2024-05-28T06:24:25.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:25.048+0000 7fd8d7c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd8d00b42e0 con 0x7fd8d006b900 2024-05-28T06:24:25.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:25.048+0000 7fd8d7c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fd8d00b5020 con 0x7fd8d0159ff0 2024-05-28T06:24:25.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:25.048+0000 7fd8c7e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd8d00b5d20 0x7fd8d015e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:25.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:25.048+0000 7fd8c7e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd8d00b5d20 0x7fd8d015e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:45430/0 (socket says 172.21.15.96:45430) 2024-05-28T06:24:25.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:25.048+0000 7fd8c7e00640 1 -- 172.21.15.96:0/2278623723 learned_addr learned my addr 172.21.15.96:0/2278623723 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:24:25.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:25.049+0000 7fd8d5200640 1 --2- 172.21.15.96:0/2278623723 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd8d006b900 0x7fd8d015f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:25.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:25.049+0000 7fd8d5c00640 1 --2- 172.21.15.96:0/2278623723 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd8d0159ff0 0x7fd8d015ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:25.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:25.049+0000 7fd8c7e00640 1 -- 172.21.15.96:0/2278623723 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd8d0159ff0 msgr2=0x7fd8d015ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:25.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:25.049+0000 7fd8c7e00640 1 --2- 172.21.15.96:0/2278623723 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd8d0159ff0 0x7fd8d015ebe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:25.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:25.049+0000 7fd8c7e00640 1 -- 172.21.15.96:0/2278623723 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd8d006b900 msgr2=0x7fd8d015f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:25.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:25.049+0000 7fd8c7e00640 1 --2- 172.21.15.96:0/2278623723 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd8d006b900 0x7fd8d015f380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:25.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:25.049+0000 7fd8c7e00640 1 -- 172.21.15.96:0/2278623723 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8d0154c90 con 0x7fd8d00b5d20 2024-05-28T06:24:25.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:25.049+0000 7fd8d5200640 1 --2- 172.21.15.96:0/2278623723 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd8d006b900 0x7fd8d015f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:24:25.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:25.049+0000 7fd8c7e00640 1 --2- 172.21.15.96:0/2278623723 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd8d00b5d20 0x7fd8d015e6a0 secure :-1 s=READY pgs=2013 cs=0 l=1 rev1=1 crypto rx=0x7fd8b8035040 tx=0x7fd8b80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:25.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:25.049+0000 7fd8cd600640 1 -- 172.21.15.96:0/2278623723 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b8037040 con 0x7fd8d00b5d20 2024-05-28T06:24:25.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:25.049+0000 7fd8cd600640 1 -- 172.21.15.96:0/2278623723 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8b8030770 con 0x7fd8d00b5d20 2024-05-28T06:24:25.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:25.049+0000 7fd8d7c00640 1 -- 172.21.15.96:0/2278623723 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8d0154080 con 0x7fd8d00b5d20 2024-05-28T06:24:25.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:25.049+0000 7fd8cd600640 1 -- 172.21.15.96:0/2278623723 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b8030b40 con 0x7fd8d00b5d20 2024-05-28T06:24:25.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:25.050+0000 7fd8d7c00640 1 -- 172.21.15.96:0/2278623723 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8d0154640 con 0x7fd8d00b5d20 2024-05-28T06:24:25.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:25.051+0000 7fd8d7c00640 1 -- 172.21.15.96:0/2278623723 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8d00690f0 con 0x7fd8d00b5d20 2024-05-28T06:24:25.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:25.052+0000 7fd8cd600640 1 -- 172.21.15.96:0/2278623723 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 42) ==== 93590+0+0 (secure 0 0 0) 0x7fd8b8044020 con 0x7fd8d00b5d20 2024-05-28T06:24:25.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:25.052+0000 7fd8cd600640 1 --2- 172.21.15.96:0/2278623723 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd8ac06e510 0x7fd8ac0709d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:25.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:25.053+0000 7fd8cd600640 1 -- 172.21.15.96:0/2278623723 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 5765+0+0 (secure 0 0 0) 0x7fd8b80ab010 con 0x7fd8d00b5d20 2024-05-28T06:24:25.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:25.053+0000 7fd8d5200640 1 --2- 172.21.15.96:0/2278623723 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd8ac06e510 0x7fd8ac0709d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:25.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:25.054+0000 7fd8d5200640 1 --2- 172.21.15.96:0/2278623723 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd8ac06e510 0x7fd8ac0709d0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fd8c0006fd0 tx=0x7fd8c0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:25.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:25.057+0000 7fd8cd600640 1 -- 172.21.15.96:0/2278623723 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd8d0154c90 con 0x7fd8d00b5d20 2024-05-28T06:24:25.290 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:25.296+0000 7fd8d7c00640 1 -- 172.21.15.96:0/2278623723 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fd8d0160d60 con 0x7fd8d00b5d20 2024-05-28T06:24:26.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.017+0000 7fd8cd600640 1 -- 172.21.15.96:0/2278623723 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v81) ==== 118+0+0 (secure 0 0 0) 0x7fd8b8032e20 con 0x7fd8d00b5d20 2024-05-28T06:24:26.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.020+0000 7fd8d7c00640 1 -- 172.21.15.96:0/2278623723 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd8ac06e510 msgr2=0x7fd8ac0709d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:26.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.020+0000 7fd8d7c00640 1 --2- 172.21.15.96:0/2278623723 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd8ac06e510 0x7fd8ac0709d0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fd8c0006fd0 tx=0x7fd8c0008040 comp rx=0 tx=0).stop 2024-05-28T06:24:26.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.020+0000 7fd8d7c00640 1 -- 172.21.15.96:0/2278623723 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd8d00b5d20 msgr2=0x7fd8d015e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:26.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.021+0000 7fd8d7c00640 1 --2- 172.21.15.96:0/2278623723 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd8d00b5d20 0x7fd8d015e6a0 secure :-1 s=READY pgs=2013 cs=0 l=1 rev1=1 crypto rx=0x7fd8b8035040 tx=0x7fd8b80046c0 comp rx=0 tx=0).stop 2024-05-28T06:24:26.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.021+0000 7fd8d7c00640 1 -- 172.21.15.96:0/2278623723 shutdown_connections 2024-05-28T06:24:26.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.021+0000 7fd8d7c00640 1 --2- 172.21.15.96:0/2278623723 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd8ac06e510 0x7fd8ac0709d0 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:26.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.021+0000 7fd8d7c00640 1 --2- 172.21.15.96:0/2278623723 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd8d0159ff0 0x7fd8d015ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:26.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.021+0000 7fd8d7c00640 1 --2- 172.21.15.96:0/2278623723 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd8d00b5d20 0x7fd8d015e6a0 unknown :-1 s=CLOSED pgs=2013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:26.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.021+0000 7fd8d7c00640 1 --2- 172.21.15.96:0/2278623723 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd8d006b900 0x7fd8d015f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:26.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.021+0000 7fd8d7c00640 1 -- 172.21.15.96:0/2278623723 >> 172.21.15.96:0/2278623723 conn(0x7fd8d00b3240 msgr2=0x7fd8d0074990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:26.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.021+0000 7fd8d7c00640 1 -- 172.21.15.96:0/2278623723 shutdown_connections 2024-05-28T06:24:26.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.021+0000 7fd8d7c00640 1 -- 172.21.15.96:0/2278623723 wait complete. 2024-05-28T06:24:26.036 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-05-28T06:24:26.037 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-05-28T06:24:26.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.237+0000 7fc00ee00640 1 Processor -- start 2024-05-28T06:24:26.232 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.238+0000 7fc00ee00640 1 -- start start 2024-05-28T06:24:26.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.239+0000 7fc00ee00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc00815d5d0 0x7fc0080cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:26.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.239+0000 7fc00ee00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc00815cb10 con 0x7fc0080b32c0 2024-05-28T06:24:26.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.239+0000 7fc00ee00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0080ad5f0 con 0x7fc0080d0640 2024-05-28T06:24:26.234 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.239+0000 7fc00ee00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fc0080adbb0 con 0x7fc00815d5d0 2024-05-28T06:24:26.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.239+0000 7fc00ce00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fc0080b32c0 0x7fc0080b3750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:33820/0 (socket says 172.21.15.96:33820) 2024-05-28T06:24:26.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.239+0000 7fc00ce00640 1 -- 172.21.15.96:0/767709506 learned_addr learned my addr 172.21.15.96:0/767709506 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:24:26.235 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.240+0000 7fc007400640 1 --2- 172.21.15.96:0/767709506 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc00815d5d0 0x7fc0080cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:26.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.240+0000 7fc006a00640 1 -- 172.21.15.96:0/767709506 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1336965723 0 0) 0x7fc00815cb10 con 0x7fc0080b32c0 2024-05-28T06:24:26.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.240+0000 7fc006a00640 1 -- 172.21.15.96:0/767709506 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbfe8003520 con 0x7fc0080b32c0 2024-05-28T06:24:26.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.240+0000 7fc006a00640 1 -- 172.21.15.96:0/767709506 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2434687385 0 0) 0x7fbfe8003520 con 0x7fc0080b32c0 2024-05-28T06:24:26.237 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.240+0000 7fc006a00640 1 -- 172.21.15.96:0/767709506 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc00815d5d0 msgr2=0x7fc0080cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:26.238 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.240+0000 7fc006a00640 1 --2- 172.21.15.96:0/767709506 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc00815d5d0 0x7fc0080cf950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:26.238 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.240+0000 7fc006a00640 1 -- 172.21.15.96:0/767709506 >> v1:172.21.15.123:6789/0 conn(0x7fc0080d0640 legacy=0x7fc0080d0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:26.238 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.240+0000 7fc006a00640 1 -- 172.21.15.96:0/767709506 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0080ad8d0 con 0x7fc0080b32c0 2024-05-28T06:24:26.238 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.240+0000 7fc006a00640 1 -- 172.21.15.96:0/767709506 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7fc000001670 con 0x7fc0080b32c0 2024-05-28T06:24:26.238 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.240+0000 7fc006a00640 1 -- 172.21.15.96:0/767709506 >> v1:172.21.15.96:6789/0 conn(0x7fc0080b32c0 legacy=0x7fc0080b3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:26.238 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.241+0000 7fc006a00640 1 --2- 172.21.15.96:0/767709506 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbfe80039e0 0x7fbfe8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:26.238 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.241+0000 7fc006a00640 1 --2- 172.21.15.96:0/767709506 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbfe8004800 0x7fbfe8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:26.238 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.241+0000 7fc006a00640 1 --2- 172.21.15.96:0/767709506 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc00815d5d0 0x7fbfe80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:26.238 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.241+0000 7fc006a00640 1 -- 172.21.15.96:0/767709506 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fbfe800ba70 con 0x7fbfe80039e0 2024-05-28T06:24:26.238 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.241+0000 7fc006a00640 1 -- 172.21.15.96:0/767709506 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fbfe800bbf0 con 0x7fbfe8004800 2024-05-28T06:24:26.238 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.241+0000 7fc006a00640 1 -- 172.21.15.96:0/767709506 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fbfe800bdf0 con 0x7fc00815d5d0 2024-05-28T06:24:26.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.241+0000 7fc007e00640 1 --2- 172.21.15.96:0/767709506 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbfe80039e0 0x7fbfe8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:26.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.241+0000 7fc007400640 1 --2- 172.21.15.96:0/767709506 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbfe8004800 0x7fbfe8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:26.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.241+0000 7fc00ce00640 1 --2- 172.21.15.96:0/767709506 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc00815d5d0 0x7fbfe80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:26.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.241+0000 7fc007e00640 1 -- 172.21.15.96:0/767709506 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc00815d5d0 msgr2=0x7fbfe80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:26.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.241+0000 7fc007e00640 1 --2- 172.21.15.96:0/767709506 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc00815d5d0 0x7fbfe80051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:26.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.241+0000 7fc007e00640 1 -- 172.21.15.96:0/767709506 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbfe8004800 msgr2=0x7fbfe8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:26.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.241+0000 7fc007e00640 1 --2- 172.21.15.96:0/767709506 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbfe8004800 0x7fbfe8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:26.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.241+0000 7fc007e00640 1 -- 172.21.15.96:0/767709506 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfe8003520 con 0x7fbfe80039e0 2024-05-28T06:24:26.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.241+0000 7fc007400640 1 --2- 172.21.15.96:0/767709506 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbfe8004800 0x7fbfe8004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:24:26.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.242+0000 7fc007e00640 1 --2- 172.21.15.96:0/767709506 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbfe80039e0 0x7fbfe8003e00 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 crypto rx=0x7fbff8001720 tx=0x7fbfe8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b401950ad7c2205 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:26.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.243+0000 7fc006a00640 1 -- 172.21.15.96:0/767709506 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff8001370 con 0x7fbfe80039e0 2024-05-28T06:24:26.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.243+0000 7fc006a00640 1 -- 172.21.15.96:0/767709506 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbff80052d0 con 0x7fbfe80039e0 2024-05-28T06:24:26.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.243+0000 7fc006a00640 1 -- 172.21.15.96:0/767709506 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff802ab90 con 0x7fbfe80039e0 2024-05-28T06:24:26.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.243+0000 7fc00ee00640 1 -- 172.21.15.96:0/767709506 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbfe80039e0 msgr2=0x7fbfe8003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:26.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.243+0000 7fc00ee00640 1 --2- 172.21.15.96:0/767709506 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbfe80039e0 0x7fbfe8003e00 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 crypto rx=0x7fbff8001720 tx=0x7fbfe8005830 comp rx=0 tx=0).stop 2024-05-28T06:24:26.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.243+0000 7fc00ce00640 1 -- 172.21.15.96:0/767709506 reap_dead start 2024-05-28T06:24:26.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.243+0000 7fc00ee00640 1 -- 172.21.15.96:0/767709506 shutdown_connections 2024-05-28T06:24:26.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.243+0000 7fc00ee00640 1 -- 172.21.15.96:0/767709506 >> 172.21.15.96:0/767709506 conn(0x7fc00806bd20 msgr2=0x7fc00806c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:26.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.244+0000 7fc00ee00640 1 -- 172.21.15.96:0/767709506 shutdown_connections 2024-05-28T06:24:26.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.244+0000 7fc00ee00640 1 -- 172.21.15.96:0/767709506 wait complete. 2024-05-28T06:24:26.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.245+0000 7fc00ee00640 1 Processor -- start 2024-05-28T06:24:26.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.245+0000 7fc00ee00640 1 -- start start 2024-05-28T06:24:26.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.246+0000 7fc00ee00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbfe80039e0 0x7fc0081555c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:26.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.246+0000 7fc00ee00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc008150550 0x7fc0081509d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:26.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.246+0000 7fc00ee00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc00809bd90 0x7fc00809c240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:26.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.246+0000 7fc00ee00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc0080b3e60 con 0x7fbfe80039e0 2024-05-28T06:24:26.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.246+0000 7fc00ee00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc00815d0b0 con 0x7fc00809bd90 2024-05-28T06:24:26.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.246+0000 7fc00ee00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fc00815da50 con 0x7fc008150550 2024-05-28T06:24:26.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.247+0000 7fc007e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbfe80039e0 0x7fc0081555c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:26.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.247+0000 7fc007e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbfe80039e0 0x7fc0081555c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:45454/0 (socket says 172.21.15.96:45454) 2024-05-28T06:24:26.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.247+0000 7fc007e00640 1 -- 172.21.15.96:0/2313412355 learned_addr learned my addr 172.21.15.96:0/2313412355 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:24:26.243 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.247+0000 7fc00ce00640 1 --2- 172.21.15.96:0/2313412355 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc00809bd90 0x7fc00809c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:26.243 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.247+0000 7fc007400640 1 --2- 172.21.15.96:0/2313412355 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc008150550 0x7fc0081509d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:26.243 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.247+0000 7fc007e00640 1 -- 172.21.15.96:0/2313412355 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc008150550 msgr2=0x7fc0081509d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:26.243 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.247+0000 7fc007e00640 1 --2- 172.21.15.96:0/2313412355 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc008150550 0x7fc0081509d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:26.243 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.247+0000 7fc007e00640 1 -- 172.21.15.96:0/2313412355 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc00809bd90 msgr2=0x7fc00809c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:26.244 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.247+0000 7fc007e00640 1 --2- 172.21.15.96:0/2313412355 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc00809bd90 0x7fc00809c240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:26.244 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.247+0000 7fc007e00640 1 -- 172.21.15.96:0/2313412355 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc008155b00 con 0x7fbfe80039e0 2024-05-28T06:24:26.244 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.247+0000 7fc007e00640 1 --2- 172.21.15.96:0/2313412355 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbfe80039e0 0x7fc0081555c0 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 crypto rx=0x7fbff8008fd0 tx=0x7fbff8001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:26.244 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.248+0000 7fbfffe00640 1 -- 172.21.15.96:0/2313412355 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff8037040 con 0x7fbfe80039e0 2024-05-28T06:24:26.244 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.248+0000 7fbfffe00640 1 -- 172.21.15.96:0/2313412355 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbff8005480 con 0x7fbfe80039e0 2024-05-28T06:24:26.244 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.248+0000 7fbfffe00640 1 -- 172.21.15.96:0/2313412355 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff8007a30 con 0x7fbfe80039e0 2024-05-28T06:24:26.244 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.248+0000 7fc00ee00640 1 -- 172.21.15.96:0/2313412355 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc00809cb00 con 0x7fbfe80039e0 2024-05-28T06:24:26.244 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.248+0000 7fc00ee00640 1 -- 172.21.15.96:0/2313412355 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc00809d0c0 con 0x7fbfe80039e0 2024-05-28T06:24:26.245 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.249+0000 7fc00ee00640 1 -- 172.21.15.96:0/2313412355 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc008150f10 con 0x7fbfe80039e0 2024-05-28T06:24:26.245 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.250+0000 7fbfffe00640 1 -- 172.21.15.96:0/2313412355 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 42) ==== 93590+0+0 (secure 0 0 0) 0x7fbff8044020 con 0x7fbfe80039e0 2024-05-28T06:24:26.245 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.251+0000 7fbfffe00640 1 --2- 172.21.15.96:0/2313412355 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbfe406e540 0x7fbfe4070a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:26.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.251+0000 7fbfffe00640 1 -- 172.21.15.96:0/2313412355 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 5765+0+0 (secure 0 0 0) 0x7fbff80aa500 con 0x7fbfe80039e0 2024-05-28T06:24:26.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.252+0000 7fc007400640 1 --2- 172.21.15.96:0/2313412355 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbfe406e540 0x7fbfe4070a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:26.247 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.253+0000 7fc007400640 1 --2- 172.21.15.96:0/2313412355 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbfe406e540 0x7fbfe4070a00 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fc008152500 tx=0x7fbff000d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:26.249 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.255+0000 7fbfffe00640 1 -- 172.21.15.96:0/2313412355 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbff804b020 con 0x7fbfe80039e0 2024-05-28T06:24:26.491 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:26.496+0000 7fc00ee00640 1 -- 172.21.15.96:0/2313412355 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fc0080689e0 con 0x7fbfe80039e0 2024-05-28T06:24:27.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.020+0000 7fbfffe00640 1 -- 172.21.15.96:0/2313412355 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v82) ==== 122+0+0 (secure 0 0 0) 0x7fbff807a390 con 0x7fbfe80039e0 2024-05-28T06:24:27.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.024+0000 7fc00ee00640 1 -- 172.21.15.96:0/2313412355 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbfe406e540 msgr2=0x7fbfe4070a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:27.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.024+0000 7fc00ee00640 1 --2- 172.21.15.96:0/2313412355 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbfe406e540 0x7fbfe4070a00 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fc008152500 tx=0x7fbff000d040 comp rx=0 tx=0).stop 2024-05-28T06:24:27.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.024+0000 7fc00ee00640 1 -- 172.21.15.96:0/2313412355 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbfe80039e0 msgr2=0x7fc0081555c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:27.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.024+0000 7fc00ee00640 1 --2- 172.21.15.96:0/2313412355 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbfe80039e0 0x7fc0081555c0 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 crypto rx=0x7fbff8008fd0 tx=0x7fbff8001120 comp rx=0 tx=0).stop 2024-05-28T06:24:27.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.024+0000 7fc00ee00640 1 -- 172.21.15.96:0/2313412355 shutdown_connections 2024-05-28T06:24:27.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.024+0000 7fc00ee00640 1 --2- 172.21.15.96:0/2313412355 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbfe406e540 0x7fbfe4070a00 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:27.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.024+0000 7fc00ee00640 1 --2- 172.21.15.96:0/2313412355 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc00809bd90 0x7fc00809c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:27.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.024+0000 7fc00ee00640 1 --2- 172.21.15.96:0/2313412355 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc008150550 0x7fc0081509d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:27.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.024+0000 7fc00ee00640 1 --2- 172.21.15.96:0/2313412355 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbfe80039e0 0x7fc0081555c0 unknown :-1 s=CLOSED pgs=2016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:27.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.024+0000 7fc00ee00640 1 -- 172.21.15.96:0/2313412355 >> 172.21.15.96:0/2313412355 conn(0x7fc00806bd20 msgr2=0x7fc008151680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:27.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.024+0000 7fc00ee00640 1 -- 172.21.15.96:0/2313412355 shutdown_connections 2024-05-28T06:24:27.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.024+0000 7fc00ee00640 1 -- 172.21.15.96:0/2313412355 wait complete. 2024-05-28T06:24:27.042 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-05-28T06:24:27.043 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-28T06:24:27.238 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.244+0000 7efe1b200640 1 Processor -- start 2024-05-28T06:24:27.238 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.244+0000 7efe1b200640 1 -- start start 2024-05-28T06:24:27.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.245+0000 7efe1b200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7efe140b4e10 0x7efe140cde30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:27.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.245+0000 7efe1b200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efe140d0ca0 0x7efe14099090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:27.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.245+0000 7efe1b200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe140b51f0 con 0x7efe140ce370 2024-05-28T06:24:27.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.245+0000 7efe1b200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7efe140aef60 con 0x7efe140b4e10 2024-05-28T06:24:27.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.245+0000 7efe1b200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7efe140ad1d0 con 0x7efe140d0ca0 2024-05-28T06:24:27.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.246+0000 7efe13e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7efe140b4e10 0x7efe140cde30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:27.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.246+0000 7efe0b400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7efe140ce370 0x7efe140d0590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:33824/0 (socket says 172.21.15.96:33824) 2024-05-28T06:24:27.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.246+0000 7efe0b400640 1 -- 172.21.15.96:0/3812960743 learned_addr learned my addr 172.21.15.96:0/3812960743 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:24:27.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.246+0000 7efe19200640 1 --2- 172.21.15.96:0/3812960743 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efe140d0ca0 0x7efe14099090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:27.241 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.246+0000 7efe13e00640 1 -- 172.21.15.96:0/3812960743 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efe140d0ca0 msgr2=0x7efe14099090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:27.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.246+0000 7efe13e00640 1 --2- 172.21.15.96:0/3812960743 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efe140d0ca0 0x7efe14099090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:27.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.246+0000 7efe13e00640 1 -- 172.21.15.96:0/3812960743 >> v1:172.21.15.96:6789/0 conn(0x7efe140ce370 legacy=0x7efe140d0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:27.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.246+0000 7efe13e00640 1 -- 172.21.15.96:0/3812960743 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe140ad470 con 0x7efe140b4e10 2024-05-28T06:24:27.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.246+0000 7efe13400640 1 -- 172.21.15.96:0/3812960743 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2858985602 0 0) 0x7efe140b51f0 con 0x7efe140ce370 2024-05-28T06:24:27.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.247+0000 7efe13e00640 1 --2- 172.21.15.96:0/3812960743 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efe140b4e10 0x7efe140cde30 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 crypto rx=0x7efe040097a0 tx=0x7efe0402a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e3506a75dbd0a26 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:27.243 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.248+0000 7efe13400640 1 -- 172.21.15.96:0/3812960743 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe04037070 con 0x7efe140b4e10 2024-05-28T06:24:27.243 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.248+0000 7efe13400640 1 -- 172.21.15.96:0/3812960743 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe0402ae50 con 0x7efe140b4e10 2024-05-28T06:24:27.243 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.248+0000 7efe13400640 1 -- 172.21.15.96:0/3812960743 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe04032660 con 0x7efe140b4e10 2024-05-28T06:24:27.244 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.248+0000 7efe1b200640 1 -- 172.21.15.96:0/3812960743 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efe140b4e10 msgr2=0x7efe140cde30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:27.244 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.248+0000 7efe1b200640 1 --2- 172.21.15.96:0/3812960743 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efe140b4e10 0x7efe140cde30 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 crypto rx=0x7efe040097a0 tx=0x7efe0402a100 comp rx=0 tx=0).stop 2024-05-28T06:24:27.244 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.248+0000 7efe1b200640 1 -- 172.21.15.96:0/3812960743 shutdown_connections 2024-05-28T06:24:27.244 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.248+0000 7efe1b200640 1 --2- 172.21.15.96:0/3812960743 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efe140d0ca0 0x7efe14099090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:27.244 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.248+0000 7efe1b200640 1 --2- 172.21.15.96:0/3812960743 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efe140b4e10 0x7efe140cde30 unknown :-1 s=CLOSED pgs=2018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:27.244 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.248+0000 7efe1b200640 1 -- 172.21.15.96:0/3812960743 >> 172.21.15.96:0/3812960743 conn(0x7efe1406bae0 msgr2=0x7efe1406bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:27.244 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.248+0000 7efe1b200640 1 -- 172.21.15.96:0/3812960743 shutdown_connections 2024-05-28T06:24:27.245 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.248+0000 7efe1b200640 1 -- 172.21.15.96:0/3812960743 wait complete. 2024-05-28T06:24:27.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.250+0000 7efe1b200640 1 Processor -- start 2024-05-28T06:24:27.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.250+0000 7efe1b200640 1 -- start start 2024-05-28T06:24:27.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.251+0000 7efe1b200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efe140b4e10 0x7efe14099a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:27.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.251+0000 7efe1b200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efe140ce370 0x7efe141f2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:27.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.251+0000 7efe1b200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efe140d0ca0 0x7efe14072b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:27.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.251+0000 7efe1b200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7efe140cea00 con 0x7efe140b4e10 2024-05-28T06:24:27.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.251+0000 7efe1b200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7efe140ce880 con 0x7efe140ce370 2024-05-28T06:24:27.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.251+0000 7efe1b200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7efe140b3150 con 0x7efe140d0ca0 2024-05-28T06:24:27.247 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.251+0000 7efe13e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efe140b4e10 0x7efe14099a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:27.247 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.251+0000 7efe13e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efe140b4e10 0x7efe14099a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:45480/0 (socket says 172.21.15.96:45480) 2024-05-28T06:24:27.247 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.251+0000 7efe13e00640 1 -- 172.21.15.96:0/1426496979 learned_addr learned my addr 172.21.15.96:0/1426496979 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:24:27.247 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.252+0000 7efe13e00640 1 -- 172.21.15.96:0/1426496979 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efe140d0ca0 msgr2=0x7efe14072b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:27.247 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.252+0000 7efe13e00640 1 --2- 172.21.15.96:0/1426496979 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efe140d0ca0 0x7efe14072b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:27.247 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.252+0000 7efe13e00640 1 -- 172.21.15.96:0/1426496979 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efe140ce370 msgr2=0x7efe141f2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:27.247 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.252+0000 7efe13e00640 1 --2- 172.21.15.96:0/1426496979 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efe140ce370 0x7efe141f2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:27.247 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.252+0000 7efe13e00640 1 -- 172.21.15.96:0/1426496979 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe1415bc20 con 0x7efe140b4e10 2024-05-28T06:24:27.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.252+0000 7efe13e00640 1 --2- 172.21.15.96:0/1426496979 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efe140b4e10 0x7efe14099a50 secure :-1 s=READY pgs=2019 cs=0 l=1 rev1=1 crypto rx=0x7efe04000c00 tx=0x7efe040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:27.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.252+0000 7efe10c00640 1 -- 172.21.15.96:0/1426496979 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe04041070 con 0x7efe140b4e10 2024-05-28T06:24:27.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.252+0000 7efe10c00640 1 -- 172.21.15.96:0/1426496979 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe0403c670 con 0x7efe140b4e10 2024-05-28T06:24:27.248 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.252+0000 7efe10c00640 1 -- 172.21.15.96:0/1426496979 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe04037040 con 0x7efe140b4e10 2024-05-28T06:24:27.249 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.252+0000 7efe1b200640 1 -- 172.21.15.96:0/1426496979 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe1415bf00 con 0x7efe140b4e10 2024-05-28T06:24:27.249 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.252+0000 7efe1b200640 1 -- 172.21.15.96:0/1426496979 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe1415c4c0 con 0x7efe140b4e10 2024-05-28T06:24:27.249 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.254+0000 7efe1b200640 1 -- 172.21.15.96:0/1426496979 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe140689e0 con 0x7efe140b4e10 2024-05-28T06:24:27.249 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.254+0000 7efe10c00640 1 -- 172.21.15.96:0/1426496979 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 42) ==== 93590+0+0 (secure 0 0 0) 0x7efe04044070 con 0x7efe140b4e10 2024-05-28T06:24:27.249 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.255+0000 7efe10c00640 1 --2- 172.21.15.96:0/1426496979 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7efdf006e540 0x7efdf0070a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:27.250 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.255+0000 7efe10c00640 1 -- 172.21.15.96:0/1426496979 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 5765+0+0 (secure 0 0 0) 0x7efe040ab410 con 0x7efe140b4e10 2024-05-28T06:24:27.250 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.256+0000 7efe0b400640 1 --2- 172.21.15.96:0/1426496979 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7efdf006e540 0x7efdf0070a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:27.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.257+0000 7efe0b400640 1 --2- 172.21.15.96:0/1426496979 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7efdf006e540 0x7efdf0070a00 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7efdfc007fd0 tx=0x7efdfc007960 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:27.257 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.263+0000 7efe10c00640 1 -- 172.21.15.96:0/1426496979 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7efe0407b3c0 con 0x7efe140b4e10 2024-05-28T06:24:27.500 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.505+0000 7efe1b200640 1 -- 172.21.15.96:0/1426496979 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efe1406f540 con 0x7efe140b4e10 2024-05-28T06:24:27.501 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.506+0000 7efe10c00640 1 -- 172.21.15.96:0/1426496979 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) ==== 94+0+5186 (secure 0 0 0) 0x7efe0407acc0 con 0x7efe140b4e10 2024-05-28T06:24:27.501 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:24:27.501 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":82,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6287,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/227668757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":227668757},{"type":"v1","addr":"172.21.15.123:6835","nonce":227668757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":77},{"gid":6290,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/1637451105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":1637451105},{"type":"v1","addr":"172.21.15.123:6837","nonce":1637451105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":77},{"gid":9843,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2263843873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2263843873},{"type":"v1","addr":"172.21.15.96:6833","nonce":2263843873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-28T06:24:23.419984+0000","modified":"2024-05-28T06:24:26.498371+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9849},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9849":{"gid":9849,"name":"a","rank":0,"incarnation":78,"state":"up:active","state_seq":3,"addr":"172.21.15.96:6835/1720339900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":1720339900},{"type":"v1","addr":"172.21.15.96:6835","nonce":1720339900}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9849,"qdb_cluster":[9849]},"id":9}]} 2024-05-28T06:24:27.501 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 82 2024-05-28T06:24:27.503 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.509+0000 7efe1b200640 1 -- 172.21.15.96:0/1426496979 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7efdf006e540 msgr2=0x7efdf0070a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:27.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.509+0000 7efe1b200640 1 --2- 172.21.15.96:0/1426496979 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7efdf006e540 0x7efdf0070a00 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7efdfc007fd0 tx=0x7efdfc007960 comp rx=0 tx=0).stop 2024-05-28T06:24:27.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.509+0000 7efe1b200640 1 -- 172.21.15.96:0/1426496979 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efe140b4e10 msgr2=0x7efe14099a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:27.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.509+0000 7efe1b200640 1 --2- 172.21.15.96:0/1426496979 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efe140b4e10 0x7efe14099a50 secure :-1 s=READY pgs=2019 cs=0 l=1 rev1=1 crypto rx=0x7efe04000c00 tx=0x7efe040046c0 comp rx=0 tx=0).stop 2024-05-28T06:24:27.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.509+0000 7efe1b200640 1 -- 172.21.15.96:0/1426496979 shutdown_connections 2024-05-28T06:24:27.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.509+0000 7efe1b200640 1 --2- 172.21.15.96:0/1426496979 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7efdf006e540 0x7efdf0070a00 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:27.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.509+0000 7efe1b200640 1 --2- 172.21.15.96:0/1426496979 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efe140d0ca0 0x7efe14072b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:27.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.509+0000 7efe1b200640 1 --2- 172.21.15.96:0/1426496979 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efe140ce370 0x7efe141f2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:27.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.509+0000 7efe1b200640 1 --2- 172.21.15.96:0/1426496979 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efe140b4e10 0x7efe14099a50 unknown :-1 s=CLOSED pgs=2019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:27.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.509+0000 7efe1b200640 1 -- 172.21.15.96:0/1426496979 >> 172.21.15.96:0/1426496979 conn(0x7efe1406bae0 msgr2=0x7efe140ca8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:27.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.509+0000 7efe1b200640 1 -- 172.21.15.96:0/1426496979 shutdown_connections 2024-05-28T06:24:27.504 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.509+0000 7efe1b200640 1 -- 172.21.15.96:0/1426496979 wait complete. 2024-05-28T06:24:27.522 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:24:27.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.730+0000 7fc317c00640 1 Processor -- start 2024-05-28T06:24:27.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.730+0000 7fc317c00640 1 -- start start 2024-05-28T06:24:27.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.731+0000 7fc317c00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fc3100d04a0 0x7fc3100d0930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:27.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.731+0000 7fc317c00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fc3100ad130 0x7fc3100ad540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:27.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.731+0000 7fc317c00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc31006ed10 con 0x7fc31006f860 2024-05-28T06:24:27.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.731+0000 7fc317c00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fc31015cb00 con 0x7fc3100d04a0 2024-05-28T06:24:27.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.731+0000 7fc317c00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fc3100b3cc0 con 0x7fc3100ad130 2024-05-28T06:24:27.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.732+0000 7fc30fe00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fc3100d04a0 0x7fc3100d0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:27.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.732+0000 7fc315200640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fc31006f860 0x7fc3100b3150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:33830/0 (socket says 172.21.15.96:33830) 2024-05-28T06:24:27.727 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.732+0000 7fc315200640 1 -- 172.21.15.96:0/2135850786 learned_addr learned my addr 172.21.15.96:0/2135850786 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:24:27.728 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.732+0000 7fc315c00640 1 --2- 172.21.15.96:0/2135850786 >> v2:172.21.15.123:3300/0 conn(0x7fc3100ad130 0x7fc3100ad540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:27.728 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.732+0000 7fc30fe00640 1 -- 172.21.15.96:0/2135850786 >> v2:172.21.15.123:3300/0 conn(0x7fc3100ad130 msgr2=0x7fc3100ad540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:27.728 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.732+0000 7fc30fe00640 1 --2- 172.21.15.96:0/2135850786 >> v2:172.21.15.123:3300/0 conn(0x7fc3100ad130 0x7fc3100ad540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:27.728 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.732+0000 7fc30fe00640 1 -- 172.21.15.96:0/2135850786 >> v1:172.21.15.96:6789/0 conn(0x7fc31006f860 legacy=0x7fc3100b3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:27.728 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.732+0000 7fc30fe00640 1 -- 172.21.15.96:0/2135850786 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3100b3fa0 con 0x7fc3100d04a0 2024-05-28T06:24:27.728 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.732+0000 7fc315c00640 1 --2- 172.21.15.96:0/2135850786 >> v2:172.21.15.123:3300/0 conn(0x7fc3100ad130 0x7fc3100ad540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:24:27.728 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.733+0000 7fc30fe00640 1 --2- 172.21.15.96:0/2135850786 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc3100d04a0 0x7fc3100d0930 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7fc3100af6e0 tx=0x7fc2f802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=949399fc7a791c6f server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:27.728 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.733+0000 7fc30f400640 1 -- 172.21.15.96:0/2135850786 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f8037070 con 0x7fc3100d04a0 2024-05-28T06:24:27.728 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.733+0000 7fc30f400640 1 -- 172.21.15.96:0/2135850786 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2f8032440 con 0x7fc3100d04a0 2024-05-28T06:24:27.728 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.733+0000 7fc30f400640 1 -- 172.21.15.96:0/2135850786 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f8032810 con 0x7fc3100d04a0 2024-05-28T06:24:27.729 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.734+0000 7fc317c00640 1 -- 172.21.15.96:0/2135850786 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc3100d04a0 msgr2=0x7fc3100d0930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:27.729 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.734+0000 7fc317c00640 1 --2- 172.21.15.96:0/2135850786 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc3100d04a0 0x7fc3100d0930 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7fc3100af6e0 tx=0x7fc2f802a1a0 comp rx=0 tx=0).stop 2024-05-28T06:24:27.729 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.734+0000 7fc317c00640 1 -- 172.21.15.96:0/2135850786 shutdown_connections 2024-05-28T06:24:27.729 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.734+0000 7fc317c00640 1 --2- 172.21.15.96:0/2135850786 >> v2:172.21.15.123:3300/0 conn(0x7fc3100ad130 0x7fc3100ad540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:27.729 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.734+0000 7fc317c00640 1 --2- 172.21.15.96:0/2135850786 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc3100d04a0 0x7fc3100d0930 unknown :-1 s=CLOSED pgs=2021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:27.730 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.734+0000 7fc317c00640 1 -- 172.21.15.96:0/2135850786 >> 172.21.15.96:0/2135850786 conn(0x7fc31006bae0 msgr2=0x7fc31006bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:27.730 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.734+0000 7fc317c00640 1 -- 172.21.15.96:0/2135850786 shutdown_connections 2024-05-28T06:24:27.730 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.734+0000 7fc317c00640 1 -- 172.21.15.96:0/2135850786 wait complete. 2024-05-28T06:24:27.730 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.736+0000 7fc317c00640 1 Processor -- start 2024-05-28T06:24:27.730 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.736+0000 7fc317c00640 1 -- start start 2024-05-28T06:24:27.732 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.737+0000 7fc317c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc31006f860 0x7fc31015cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:27.732 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.737+0000 7fc317c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc3100ad130 0x7fc3101f3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:27.732 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.737+0000 7fc317c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3100d04a0 0x7fc3100b2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:27.732 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.737+0000 7fc317c00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc3100b4840 con 0x7fc31006f860 2024-05-28T06:24:27.732 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.737+0000 7fc317c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc3100b4560 con 0x7fc3100d04a0 2024-05-28T06:24:27.733 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.737+0000 7fc317c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fc3100b3860 con 0x7fc3100ad130 2024-05-28T06:24:27.733 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.737+0000 7fc315200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc31006f860 0x7fc31015cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:27.733 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.737+0000 7fc315200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc31006f860 0x7fc31015cca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:45506/0 (socket says 172.21.15.96:45506) 2024-05-28T06:24:27.733 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.737+0000 7fc315200640 1 -- 172.21.15.96:0/1680219198 learned_addr learned my addr 172.21.15.96:0/1680219198 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:24:27.733 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.738+0000 7fc315200640 1 -- 172.21.15.96:0/1680219198 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc3100ad130 msgr2=0x7fc3101f3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:27.733 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.738+0000 7fc315200640 1 --2- 172.21.15.96:0/1680219198 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc3100ad130 0x7fc3101f3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:27.733 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.738+0000 7fc315c00640 1 --2- 172.21.15.96:0/1680219198 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3100d04a0 0x7fc3100b2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:27.733 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.738+0000 7fc315200640 1 -- 172.21.15.96:0/1680219198 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3100d04a0 msgr2=0x7fc3100b2cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:27.733 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.738+0000 7fc315200640 1 --2- 172.21.15.96:0/1680219198 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3100d04a0 0x7fc3100b2cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:27.733 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.738+0000 7fc315200640 1 -- 172.21.15.96:0/1680219198 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc31015d410 con 0x7fc31006f860 2024-05-28T06:24:27.733 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.738+0000 7fc315200640 1 --2- 172.21.15.96:0/1680219198 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc31006f860 0x7fc31015cca0 secure :-1 s=READY pgs=2022 cs=0 l=1 rev1=1 crypto rx=0x7fc30000c400 tx=0x7fc30000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:27.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.738+0000 7fc30cc00640 1 -- 172.21.15.96:0/1680219198 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc300017070 con 0x7fc31006f860 2024-05-28T06:24:27.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.738+0000 7fc30cc00640 1 -- 172.21.15.96:0/1680219198 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc300005ce0 con 0x7fc31006f860 2024-05-28T06:24:27.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.738+0000 7fc317c00640 1 -- 172.21.15.96:0/1680219198 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc31015d6f0 con 0x7fc31006f860 2024-05-28T06:24:27.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.738+0000 7fc30cc00640 1 -- 172.21.15.96:0/1680219198 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3000072f0 con 0x7fc31006f860 2024-05-28T06:24:27.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.738+0000 7fc317c00640 1 -- 172.21.15.96:0/1680219198 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc31015dcb0 con 0x7fc31006f860 2024-05-28T06:24:27.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.739+0000 7fc317c00640 1 -- 172.21.15.96:0/1680219198 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3100c6e80 con 0x7fc31006f860 2024-05-28T06:24:27.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.741+0000 7fc30cc00640 1 -- 172.21.15.96:0/1680219198 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 42) ==== 93590+0+0 (secure 0 0 0) 0x7fc30001a070 con 0x7fc31006f860 2024-05-28T06:24:27.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.742+0000 7fc30cc00640 1 --2- 172.21.15.96:0/1680219198 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc2ec06e540 0x7fc2ec070a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:27.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.742+0000 7fc30cc00640 1 -- 172.21.15.96:0/1680219198 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 5765+0+0 (secure 0 0 0) 0x7fc300089e00 con 0x7fc31006f860 2024-05-28T06:24:27.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.742+0000 7fc30fe00640 1 --2- 172.21.15.96:0/1680219198 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc2ec06e540 0x7fc2ec070a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:27.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.743+0000 7fc30fe00640 1 --2- 172.21.15.96:0/1680219198 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc2ec06e540 0x7fc2ec070a00 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fc2f8034000 tx=0x7fc2f802c000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:27.740 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.746+0000 7fc30cc00640 1 -- 172.21.15.96:0/1680219198 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc3100c6e80 con 0x7fc31006f860 2024-05-28T06:24:27.943 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.948+0000 7fc317c00640 1 -- 172.21.15.96:0/1680219198 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc3100689e0 con 0x7fc31006f860 2024-05-28T06:24:27.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.951+0000 7fc30cc00640 1 -- 172.21.15.96:0/1680219198 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v130) ==== 74+0+16779 (secure 0 0 0) 0x7fc300090e20 con 0x7fc31006f860 2024-05-28T06:24:27.946 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:24:27.946 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":130,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:24:24.988646+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-05-28T06:24:21.166134+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"129","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-05-28T06:24:22.415938+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"130","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":13,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:24:27.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.953+0000 7fc317c00640 1 -- 172.21.15.96:0/1680219198 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc2ec06e540 msgr2=0x7fc2ec070a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:27.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.954+0000 7fc317c00640 1 --2- 172.21.15.96:0/1680219198 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc2ec06e540 0x7fc2ec070a00 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fc2f8034000 tx=0x7fc2f802c000 comp rx=0 tx=0).stop 2024-05-28T06:24:27.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.954+0000 7fc317c00640 1 -- 172.21.15.96:0/1680219198 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc31006f860 msgr2=0x7fc31015cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:27.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.954+0000 7fc317c00640 1 --2- 172.21.15.96:0/1680219198 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc31006f860 0x7fc31015cca0 secure :-1 s=READY pgs=2022 cs=0 l=1 rev1=1 crypto rx=0x7fc30000c400 tx=0x7fc30000c8c0 comp rx=0 tx=0).stop 2024-05-28T06:24:27.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.954+0000 7fc317c00640 1 -- 172.21.15.96:0/1680219198 shutdown_connections 2024-05-28T06:24:27.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.954+0000 7fc317c00640 1 --2- 172.21.15.96:0/1680219198 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc2ec06e540 0x7fc2ec070a00 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:27.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.954+0000 7fc317c00640 1 --2- 172.21.15.96:0/1680219198 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3100d04a0 0x7fc3100b2cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:27.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.954+0000 7fc317c00640 1 --2- 172.21.15.96:0/1680219198 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc3100ad130 0x7fc3101f3fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:27.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.954+0000 7fc317c00640 1 --2- 172.21.15.96:0/1680219198 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc31006f860 0x7fc31015cca0 unknown :-1 s=CLOSED pgs=2022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:27.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.954+0000 7fc317c00640 1 -- 172.21.15.96:0/1680219198 >> 172.21.15.96:0/1680219198 conn(0x7fc31006bae0 msgr2=0x7fc3100cc200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:27.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.954+0000 7fc317c00640 1 -- 172.21.15.96:0/1680219198 shutdown_connections 2024-05-28T06:24:27.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:27.954+0000 7fc317c00640 1 -- 172.21.15.96:0/1680219198 wait complete. 2024-05-28T06:24:27.969 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-05-28T06:24:27.969 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-28T06:24:28.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.163+0000 7fef5cc00640 1 Processor -- start 2024-05-28T06:24:28.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.163+0000 7fef5cc00640 1 -- start start 2024-05-28T06:24:28.159 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.164+0000 7fef5cc00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fef58156b30 0x7fef580af6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:28.159 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.165+0000 7fef5cc00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fef580ad940 0x7fef580b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:28.159 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.165+0000 7fef5cc00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef58156f10 con 0x7fef5806ee40 2024-05-28T06:24:28.159 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.165+0000 7fef5cc00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fef5806dda0 con 0x7fef580ad940 2024-05-28T06:24:28.159 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.165+0000 7fef5cc00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fef5806e080 con 0x7fef58156b30 2024-05-28T06:24:28.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.165+0000 7fef4fe00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fef5806ee40 0x7fef580ad230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:33844/0 (socket says 172.21.15.96:33844) 2024-05-28T06:24:28.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.165+0000 7fef4fe00640 1 -- 172.21.15.96:0/1832122240 learned_addr learned my addr 172.21.15.96:0/1832122240 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:24:28.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.165+0000 7fef56a00640 1 -- 172.21.15.96:0/1832122240 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2453983311 0 0) 0x7fef58156f10 con 0x7fef5806ee40 2024-05-28T06:24:28.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.165+0000 7fef56a00640 1 -- 172.21.15.96:0/1832122240 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fef3c003520 con 0x7fef5806ee40 2024-05-28T06:24:28.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.165+0000 7fef57e00640 1 --2- 172.21.15.96:0/1832122240 >> v2:172.21.15.123:3300/0 conn(0x7fef580ad940 0x7fef580b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:28.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.166+0000 7fef57400640 1 --2- 172.21.15.96:0/1832122240 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fef58156b30 0x7fef580af6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:28.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.166+0000 7fef56a00640 1 -- 172.21.15.96:0/1832122240 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3279112008 0 0) 0x7fef3c003520 con 0x7fef5806ee40 2024-05-28T06:24:28.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.166+0000 7fef56a00640 1 -- 172.21.15.96:0/1832122240 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fef58156b30 msgr2=0x7fef580af6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:28.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.166+0000 7fef56a00640 1 --2- 172.21.15.96:0/1832122240 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fef58156b30 0x7fef580af6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:28.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.166+0000 7fef56a00640 1 -- 172.21.15.96:0/1832122240 >> v2:172.21.15.123:3300/0 conn(0x7fef580ad940 msgr2=0x7fef580b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:28.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.166+0000 7fef56a00640 1 --2- 172.21.15.96:0/1832122240 >> v2:172.21.15.123:3300/0 conn(0x7fef580ad940 0x7fef580b32c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:28.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.166+0000 7fef56a00640 1 -- 172.21.15.96:0/1832122240 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef5806e360 con 0x7fef5806ee40 2024-05-28T06:24:28.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.166+0000 7fef56a00640 1 -- 172.21.15.96:0/1832122240 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7fef40001670 con 0x7fef5806ee40 2024-05-28T06:24:28.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.166+0000 7fef57e00640 1 --2- 172.21.15.96:0/1832122240 >> v2:172.21.15.123:3300/0 conn(0x7fef580ad940 0x7fef580b32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-28T06:24:28.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.166+0000 7fef56a00640 1 -- 172.21.15.96:0/1832122240 >> v1:172.21.15.96:6789/0 conn(0x7fef5806ee40 legacy=0x7fef580ad230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:28.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.166+0000 7fef56a00640 1 --2- 172.21.15.96:0/1832122240 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef3c0039b0 0x7fef3c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:28.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.166+0000 7fef56a00640 1 --2- 172.21.15.96:0/1832122240 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef3c004800 0x7fef3c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:28.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.166+0000 7fef56a00640 1 --2- 172.21.15.96:0/1832122240 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fef58156b30 0x7fef3c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:28.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.166+0000 7fef56a00640 1 -- 172.21.15.96:0/1832122240 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fef5806dda0 con 0x7fef3c0039b0 2024-05-28T06:24:28.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.166+0000 7fef56a00640 1 -- 172.21.15.96:0/1832122240 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fef5806e080 con 0x7fef3c004800 2024-05-28T06:24:28.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.166+0000 7fef56a00640 1 -- 172.21.15.96:0/1832122240 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fef3c00bb20 con 0x7fef58156b30 2024-05-28T06:24:28.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.166+0000 7fef57400640 1 --2- 172.21.15.96:0/1832122240 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef3c0039b0 0x7fef3c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:28.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.167+0000 7fef4fe00640 1 --2- 172.21.15.96:0/1832122240 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef3c004800 0x7fef3c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:28.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.167+0000 7fef57e00640 1 --2- 172.21.15.96:0/1832122240 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fef58156b30 0x7fef3c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:28.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.167+0000 7fef57400640 1 -- 172.21.15.96:0/1832122240 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fef58156b30 msgr2=0x7fef3c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:28.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.167+0000 7fef57400640 1 --2- 172.21.15.96:0/1832122240 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fef58156b30 0x7fef3c0051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:28.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.167+0000 7fef57400640 1 -- 172.21.15.96:0/1832122240 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef3c004800 msgr2=0x7fef3c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:28.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.167+0000 7fef57400640 1 --2- 172.21.15.96:0/1832122240 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef3c004800 0x7fef3c004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:28.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.167+0000 7fef57400640 1 -- 172.21.15.96:0/1832122240 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef3c003520 con 0x7fef3c0039b0 2024-05-28T06:24:28.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.168+0000 7fef57400640 1 --2- 172.21.15.96:0/1832122240 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef3c0039b0 0x7fef3c003dd0 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 crypto rx=0x7fef4800ba00 tx=0x7fef3c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a5e00ef19a8eb8f server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:28.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.168+0000 7fef56a00640 1 -- 172.21.15.96:0/1832122240 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef48030070 con 0x7fef3c0039b0 2024-05-28T06:24:28.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.168+0000 7fef56a00640 1 -- 172.21.15.96:0/1832122240 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef4802ad80 con 0x7fef3c0039b0 2024-05-28T06:24:28.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.168+0000 7fef56a00640 1 -- 172.21.15.96:0/1832122240 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef4800b600 con 0x7fef3c0039b0 2024-05-28T06:24:28.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.169+0000 7fef5cc00640 1 -- 172.21.15.96:0/1832122240 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef3c0039b0 msgr2=0x7fef3c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:28.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.169+0000 7fef5cc00640 1 --2- 172.21.15.96:0/1832122240 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef3c0039b0 0x7fef3c003dd0 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 crypto rx=0x7fef4800ba00 tx=0x7fef3c005830 comp rx=0 tx=0).stop 2024-05-28T06:24:28.166 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.169+0000 7fef57e00640 1 -- 172.21.15.96:0/1832122240 reap_dead start 2024-05-28T06:24:28.166 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.169+0000 7fef5cc00640 1 -- 172.21.15.96:0/1832122240 shutdown_connections 2024-05-28T06:24:28.166 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.169+0000 7fef5cc00640 1 -- 172.21.15.96:0/1832122240 >> 172.21.15.96:0/1832122240 conn(0x7fef580b1fc0 msgr2=0x7fef580b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:28.166 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.169+0000 7fef5cc00640 1 -- 172.21.15.96:0/1832122240 shutdown_connections 2024-05-28T06:24:28.166 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.169+0000 7fef5cc00640 1 -- 172.21.15.96:0/1832122240 wait complete. 2024-05-28T06:24:28.166 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.172+0000 7fef5cc00640 1 Processor -- start 2024-05-28T06:24:28.166 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.172+0000 7fef5cc00640 1 -- start start 2024-05-28T06:24:28.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.173+0000 7fef5cc00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef3c0039b0 0x7fef581f3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:28.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.173+0000 7fef5cc00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fef580b0aa0 0x7fef5815ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:28.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.173+0000 7fef5cc00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef580b48d0 0x7fef5815f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:28.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.173+0000 7fef5cc00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fef580b2db0 con 0x7fef580b48d0 2024-05-28T06:24:28.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.173+0000 7fef5cc00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fef580b2c30 con 0x7fef3c0039b0 2024-05-28T06:24:28.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.173+0000 7fef5cc00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fef5806f2c0 con 0x7fef580b0aa0 2024-05-28T06:24:28.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.173+0000 7fef57e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef580b48d0 0x7fef5815f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:28.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.173+0000 7fef57e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef580b48d0 0x7fef5815f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:45516/0 (socket says 172.21.15.96:45516) 2024-05-28T06:24:28.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.173+0000 7fef57e00640 1 -- 172.21.15.96:0/4031710463 learned_addr learned my addr 172.21.15.96:0/4031710463 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:24:28.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.174+0000 7fef57400640 1 --2- 172.21.15.96:0/4031710463 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef3c0039b0 0x7fef581f3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:28.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.174+0000 7fef4fe00640 1 --2- 172.21.15.96:0/4031710463 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fef580b0aa0 0x7fef5815ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:28.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.174+0000 7fef57e00640 1 -- 172.21.15.96:0/4031710463 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fef580b0aa0 msgr2=0x7fef5815ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:28.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.174+0000 7fef57e00640 1 --2- 172.21.15.96:0/4031710463 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fef580b0aa0 0x7fef5815ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:28.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.174+0000 7fef57e00640 1 -- 172.21.15.96:0/4031710463 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef3c0039b0 msgr2=0x7fef581f3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:28.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.174+0000 7fef57e00640 1 --2- 172.21.15.96:0/4031710463 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef3c0039b0 0x7fef581f3ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:28.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.174+0000 7fef57e00640 1 -- 172.21.15.96:0/4031710463 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef581f4410 con 0x7fef580b48d0 2024-05-28T06:24:28.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.174+0000 7fef4fe00640 1 --2- 172.21.15.96:0/4031710463 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fef580b0aa0 0x7fef5815ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:24:28.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.174+0000 7fef57e00640 1 --2- 172.21.15.96:0/4031710463 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef580b48d0 0x7fef5815f010 secure :-1 s=READY pgs=2025 cs=0 l=1 rev1=1 crypto rx=0x7fef3c004780 tx=0x7fef50004840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:28.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.174+0000 7fef4f400640 1 -- 172.21.15.96:0/4031710463 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef50019070 con 0x7fef580b48d0 2024-05-28T06:24:28.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.174+0000 7fef4f400640 1 -- 172.21.15.96:0/4031710463 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef50007cd0 con 0x7fef580b48d0 2024-05-28T06:24:28.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.174+0000 7fef4f400640 1 -- 172.21.15.96:0/4031710463 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef50014640 con 0x7fef580b48d0 2024-05-28T06:24:28.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.174+0000 7fef5cc00640 1 -- 172.21.15.96:0/4031710463 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef580af310 con 0x7fef580b48d0 2024-05-28T06:24:28.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.175+0000 7fef5cc00640 1 -- 172.21.15.96:0/4031710463 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef5809dfd0 con 0x7fef580b48d0 2024-05-28T06:24:28.171 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.176+0000 7fef5cc00640 1 -- 172.21.15.96:0/4031710463 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef580689e0 con 0x7fef580b48d0 2024-05-28T06:24:28.171 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.177+0000 7fef4f400640 1 -- 172.21.15.96:0/4031710463 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 42) ==== 93590+0+0 (secure 0 0 0) 0x7fef5001c070 con 0x7fef580b48d0 2024-05-28T06:24:28.172 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.178+0000 7fef4f400640 1 --2- 172.21.15.96:0/4031710463 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fef3406e540 0x7fef34070a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:28.172 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.178+0000 7fef4f400640 1 -- 172.21.15.96:0/4031710463 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 5765+0+0 (secure 0 0 0) 0x7fef5008a420 con 0x7fef580b48d0 2024-05-28T06:24:28.172 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.178+0000 7fef57400640 1 --2- 172.21.15.96:0/4031710463 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fef3406e540 0x7fef34070a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:28.174 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.179+0000 7fef57400640 1 --2- 172.21.15.96:0/4031710463 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fef3406e540 0x7fef34070a00 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7fef3c0057e0 tx=0x7fef48030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:28.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.182+0000 7fef4f400640 1 -- 172.21.15.96:0/4031710463 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fef5005a350 con 0x7fef580b48d0 2024-05-28T06:24:28.373 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.378+0000 7fef5cc00640 1 -- 172.21.15.96:0/4031710463 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fef5809dd80 con 0x7fef3406e540 2024-05-28T06:24:28.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.399+0000 7fef4f400640 1 -- 172.21.15.96:0/4031710463 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252459 (secure 0 0 0) 0x7fef5809dd80 con 0x7fef3406e540 2024-05-28T06:24:28.397 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:24:28.399 INFO:teuthology.orchestra.run.smithi096.stderr:dumped all 2024-05-28T06:24:28.400 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.405+0000 7fef5cc00640 1 -- 172.21.15.96:0/4031710463 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fef3406e540 msgr2=0x7fef34070a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:28.400 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.405+0000 7fef5cc00640 1 --2- 172.21.15.96:0/4031710463 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fef3406e540 0x7fef34070a00 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7fef3c0057e0 tx=0x7fef48030000 comp rx=0 tx=0).stop 2024-05-28T06:24:28.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.405+0000 7fef5cc00640 1 -- 172.21.15.96:0/4031710463 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef580b48d0 msgr2=0x7fef5815f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:28.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.405+0000 7fef5cc00640 1 --2- 172.21.15.96:0/4031710463 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef580b48d0 0x7fef5815f010 secure :-1 s=READY pgs=2025 cs=0 l=1 rev1=1 crypto rx=0x7fef3c004780 tx=0x7fef50004840 comp rx=0 tx=0).stop 2024-05-28T06:24:28.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.405+0000 7fef5cc00640 1 -- 172.21.15.96:0/4031710463 shutdown_connections 2024-05-28T06:24:28.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.405+0000 7fef5cc00640 1 --2- 172.21.15.96:0/4031710463 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fef3406e540 0x7fef34070a00 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:28.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.405+0000 7fef5cc00640 1 --2- 172.21.15.96:0/4031710463 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef580b48d0 0x7fef5815f010 unknown :-1 s=CLOSED pgs=2025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:28.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.405+0000 7fef5cc00640 1 --2- 172.21.15.96:0/4031710463 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fef580b0aa0 0x7fef5815ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:28.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.405+0000 7fef5cc00640 1 --2- 172.21.15.96:0/4031710463 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef3c0039b0 0x7fef581f3ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:28.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.405+0000 7fef5cc00640 1 -- 172.21.15.96:0/4031710463 >> 172.21.15.96:0/4031710463 conn(0x7fef580b1fc0 msgr2=0x7fef580afab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:28.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.405+0000 7fef5cc00640 1 -- 172.21.15.96:0/4031710463 shutdown_connections 2024-05-28T06:24:28.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.406+0000 7fef5cc00640 1 -- 172.21.15.96:0/4031710463 wait complete. 2024-05-28T06:24:28.419 INFO:teuthology.orchestra.run.smithi096.stdout:{"pg_ready":true,"pg_map":{"version":737,"stamp":"2024-05-28T06:24:28.018212+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":1040200,"kb_used_data":13812,"kb_used_omap":12,"kb_used_meta":1026227,"kb_avail":753934520,"statfs":{"total":773094113280,"available":772028948480,"internally_reserved":0,"allocated":14143488,"data_stored":6861448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":1050857008},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001869"},"pg_stats":[{"pgid":"19.3f","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.008757+0000","last_change":"2024-05-28T06:24:24.007430+0000","last_active":"2024-05-28T06:24:24.008757+0000","last_peered":"2024-05-28T06:24:24.008757+0000","last_clean":"2024-05-28T06:24:24.008757+0000","last_became_active":"2024-05-28T06:24:24.006988+0000","last_became_peered":"2024-05-28T06:24:24.006988+0000","last_unstale":"2024-05-28T06:24:24.008757+0000","last_undegraded":"2024-05-28T06:24:24.008757+0000","last_fullsized":"2024-05-28T06:24:24.008757+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T18:04:39.239146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3e","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:23.990554+0000","last_change":"2024-05-28T06:24:23.986742+0000","last_active":"2024-05-28T06:24:23.990554+0000","last_peered":"2024-05-28T06:24:23.990554+0000","last_clean":"2024-05-28T06:24:23.990554+0000","last_became_active":"2024-05-28T06:24:23.986471+0000","last_became_peered":"2024-05-28T06:24:23.986471+0000","last_unstale":"2024-05-28T06:24:23.990554+0000","last_undegraded":"2024-05-28T06:24:23.990554+0000","last_fullsized":"2024-05-28T06:24:23.990554+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:15:02.161297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3d","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.016001+0000","last_change":"2024-05-28T06:24:24.007698+0000","last_active":"2024-05-28T06:24:24.016001+0000","last_peered":"2024-05-28T06:24:24.016001+0000","last_clean":"2024-05-28T06:24:24.016001+0000","last_became_active":"2024-05-28T06:24:24.007416+0000","last_became_peered":"2024-05-28T06:24:24.007416+0000","last_unstale":"2024-05-28T06:24:24.016001+0000","last_undegraded":"2024-05-28T06:24:24.016001+0000","last_fullsized":"2024-05-28T06:24:24.016001+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:44:39.421718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.3c","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.012365+0000","last_change":"2024-05-28T06:24:24.011512+0000","last_active":"2024-05-28T06:24:24.012365+0000","last_peered":"2024-05-28T06:24:24.012365+0000","last_clean":"2024-05-28T06:24:24.012365+0000","last_became_active":"2024-05-28T06:24:24.011234+0000","last_became_peered":"2024-05-28T06:24:24.011234+0000","last_unstale":"2024-05-28T06:24:24.012365+0000","last_undegraded":"2024-05-28T06:24:24.012365+0000","last_fullsized":"2024-05-28T06:24:24.012365+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:08:20.720556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3b","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-05-28T06:24:25.900674+0000","last_change":"2024-05-28T06:24:24.019397+0000","last_active":"2024-05-28T06:24:25.900674+0000","last_peered":"2024-05-28T06:24:25.900674+0000","last_clean":"2024-05-28T06:24:25.900674+0000","last_became_active":"2024-05-28T06:24:24.018334+0000","last_became_peered":"2024-05-28T06:24:24.018334+0000","last_unstale":"2024-05-28T06:24:25.900674+0000","last_undegraded":"2024-05-28T06:24:25.900674+0000","last_fullsized":"2024-05-28T06:24:25.900674+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:22:34.611665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.3a","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-05-28T06:24:25.012292+0000","last_change":"2024-05-28T06:24:24.003959+0000","last_active":"2024-05-28T06:24:25.012292+0000","last_peered":"2024-05-28T06:24:25.012292+0000","last_clean":"2024-05-28T06:24:25.012292+0000","last_became_active":"2024-05-28T06:24:23.997362+0000","last_became_peered":"2024-05-28T06:24:23.997362+0000","last_unstale":"2024-05-28T06:24:25.012292+0000","last_undegraded":"2024-05-28T06:24:25.012292+0000","last_fullsized":"2024-05-28T06:24:25.012292+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:07:49.435286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.39","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-05-28T06:24:25.011686+0000","last_change":"2024-05-28T06:24:24.006117+0000","last_active":"2024-05-28T06:24:25.011686+0000","last_peered":"2024-05-28T06:24:25.011686+0000","last_clean":"2024-05-28T06:24:25.011686+0000","last_became_active":"2024-05-28T06:24:24.005839+0000","last_became_peered":"2024-05-28T06:24:24.005839+0000","last_unstale":"2024-05-28T06:24:25.011686+0000","last_undegraded":"2024-05-28T06:24:25.011686+0000","last_fullsized":"2024-05-28T06:24:25.011686+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:47:32.965627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.38","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.017553+0000","last_change":"2024-05-28T06:24:24.008772+0000","last_active":"2024-05-28T06:24:24.017553+0000","last_peered":"2024-05-28T06:24:24.017553+0000","last_clean":"2024-05-28T06:24:24.017553+0000","last_became_active":"2024-05-28T06:24:24.008334+0000","last_became_peered":"2024-05-28T06:24:24.008334+0000","last_unstale":"2024-05-28T06:24:24.017553+0000","last_undegraded":"2024-05-28T06:24:24.017553+0000","last_fullsized":"2024-05-28T06:24:24.017553+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:41:33.057222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.37","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.021117+0000","last_change":"2024-05-28T06:24:24.018229+0000","last_active":"2024-05-28T06:24:24.021117+0000","last_peered":"2024-05-28T06:24:24.021117+0000","last_clean":"2024-05-28T06:24:24.021117+0000","last_became_active":"2024-05-28T06:24:24.017929+0000","last_became_peered":"2024-05-28T06:24:24.017929+0000","last_unstale":"2024-05-28T06:24:24.021117+0000","last_undegraded":"2024-05-28T06:24:24.021117+0000","last_fullsized":"2024-05-28T06:24:24.021117+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:05:18.151961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.36","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:23.997228+0000","last_change":"2024-05-28T06:24:23.995403+0000","last_active":"2024-05-28T06:24:23.997228+0000","last_peered":"2024-05-28T06:24:23.997228+0000","last_clean":"2024-05-28T06:24:23.997228+0000","last_became_active":"2024-05-28T06:24:23.994945+0000","last_became_peered":"2024-05-28T06:24:23.994945+0000","last_unstale":"2024-05-28T06:24:23.997228+0000","last_undegraded":"2024-05-28T06:24:23.997228+0000","last_fullsized":"2024-05-28T06:24:23.997228+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:06:42.676403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.35","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:23.997024+0000","last_change":"2024-05-28T06:24:23.992865+0000","last_active":"2024-05-28T06:24:23.997024+0000","last_peered":"2024-05-28T06:24:23.997024+0000","last_clean":"2024-05-28T06:24:23.997024+0000","last_became_active":"2024-05-28T06:24:23.991405+0000","last_became_peered":"2024-05-28T06:24:23.991405+0000","last_unstale":"2024-05-28T06:24:23.997024+0000","last_undegraded":"2024-05-28T06:24:23.997024+0000","last_fullsized":"2024-05-28T06:24:23.997024+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:32:12.353432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.34","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.019721+0000","last_change":"2024-05-28T06:24:24.018385+0000","last_active":"2024-05-28T06:24:24.019721+0000","last_peered":"2024-05-28T06:24:24.019721+0000","last_clean":"2024-05-28T06:24:24.019721+0000","last_became_active":"2024-05-28T06:24:24.017931+0000","last_became_peered":"2024-05-28T06:24:24.017931+0000","last_unstale":"2024-05-28T06:24:24.019721+0000","last_undegraded":"2024-05-28T06:24:24.019721+0000","last_fullsized":"2024-05-28T06:24:24.019721+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:59:26.272700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.33","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.008608+0000","last_change":"2024-05-28T06:24:24.007399+0000","last_active":"2024-05-28T06:24:24.008608+0000","last_peered":"2024-05-28T06:24:24.008608+0000","last_clean":"2024-05-28T06:24:24.008608+0000","last_became_active":"2024-05-28T06:24:24.007121+0000","last_became_peered":"2024-05-28T06:24:24.007121+0000","last_unstale":"2024-05-28T06:24:24.008608+0000","last_undegraded":"2024-05-28T06:24:24.008608+0000","last_fullsized":"2024-05-28T06:24:24.008608+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:50:21.530645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.32","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:23.997543+0000","last_change":"2024-05-28T06:24:23.996362+0000","last_active":"2024-05-28T06:24:23.997543+0000","last_peered":"2024-05-28T06:24:23.997543+0000","last_clean":"2024-05-28T06:24:23.997543+0000","last_became_active":"2024-05-28T06:24:23.994791+0000","last_became_peered":"2024-05-28T06:24:23.994791+0000","last_unstale":"2024-05-28T06:24:23.997543+0000","last_undegraded":"2024-05-28T06:24:23.997543+0000","last_fullsized":"2024-05-28T06:24:23.997543+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:29:55.277690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.31","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.021218+0000","last_change":"2024-05-28T06:24:24.020598+0000","last_active":"2024-05-28T06:24:24.021218+0000","last_peered":"2024-05-28T06:24:24.021218+0000","last_clean":"2024-05-28T06:24:24.021218+0000","last_became_active":"2024-05-28T06:24:24.020301+0000","last_became_peered":"2024-05-28T06:24:24.020301+0000","last_unstale":"2024-05-28T06:24:24.021218+0000","last_undegraded":"2024-05-28T06:24:24.021218+0000","last_fullsized":"2024-05-28T06:24:24.021218+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:28:25.639532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.30","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:23.999124+0000","last_change":"2024-05-28T06:24:23.995628+0000","last_active":"2024-05-28T06:24:23.999124+0000","last_peered":"2024-05-28T06:24:23.999124+0000","last_clean":"2024-05-28T06:24:23.999124+0000","last_became_active":"2024-05-28T06:24:23.995367+0000","last_became_peered":"2024-05-28T06:24:23.995367+0000","last_unstale":"2024-05-28T06:24:23.999124+0000","last_undegraded":"2024-05-28T06:24:23.999124+0000","last_fullsized":"2024-05-28T06:24:23.999124+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:31:43.576159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2f","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-05-28T06:24:25.013294+0000","last_change":"2024-05-28T06:24:23.987587+0000","last_active":"2024-05-28T06:24:25.013294+0000","last_peered":"2024-05-28T06:24:25.013294+0000","last_clean":"2024-05-28T06:24:25.013294+0000","last_became_active":"2024-05-28T06:24:23.986777+0000","last_became_peered":"2024-05-28T06:24:23.986777+0000","last_unstale":"2024-05-28T06:24:25.013294+0000","last_undegraded":"2024-05-28T06:24:25.013294+0000","last_fullsized":"2024-05-28T06:24:25.013294+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:06:00.030718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.2e","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:23.999601+0000","last_change":"2024-05-28T06:24:23.998500+0000","last_active":"2024-05-28T06:24:23.999601+0000","last_peered":"2024-05-28T06:24:23.999601+0000","last_clean":"2024-05-28T06:24:23.999601+0000","last_became_active":"2024-05-28T06:24:23.998105+0000","last_became_peered":"2024-05-28T06:24:23.998105+0000","last_unstale":"2024-05-28T06:24:23.999601+0000","last_undegraded":"2024-05-28T06:24:23.999601+0000","last_fullsized":"2024-05-28T06:24:23.999601+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:05:14.127643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2d","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.000368+0000","last_change":"2024-05-28T06:24:23.997899+0000","last_active":"2024-05-28T06:24:24.000368+0000","last_peered":"2024-05-28T06:24:24.000368+0000","last_clean":"2024-05-28T06:24:24.000368+0000","last_became_active":"2024-05-28T06:24:23.997362+0000","last_became_peered":"2024-05-28T06:24:23.997362+0000","last_unstale":"2024-05-28T06:24:24.000368+0000","last_undegraded":"2024-05-28T06:24:24.000368+0000","last_fullsized":"2024-05-28T06:24:24.000368+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:54:07.720252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2c","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:23.998894+0000","last_change":"2024-05-28T06:24:23.996440+0000","last_active":"2024-05-28T06:24:23.998894+0000","last_peered":"2024-05-28T06:24:23.998894+0000","last_clean":"2024-05-28T06:24:23.998894+0000","last_became_active":"2024-05-28T06:24:23.996067+0000","last_became_peered":"2024-05-28T06:24:23.996067+0000","last_unstale":"2024-05-28T06:24:23.998894+0000","last_undegraded":"2024-05-28T06:24:23.998894+0000","last_fullsized":"2024-05-28T06:24:23.998894+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:36:13.769185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2b","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-05-28T06:24:25.900477+0000","last_change":"2024-05-28T06:24:24.018537+0000","last_active":"2024-05-28T06:24:25.900477+0000","last_peered":"2024-05-28T06:24:25.900477+0000","last_clean":"2024-05-28T06:24:25.900477+0000","last_became_active":"2024-05-28T06:24:24.018253+0000","last_became_peered":"2024-05-28T06:24:24.018253+0000","last_unstale":"2024-05-28T06:24:25.900477+0000","last_undegraded":"2024-05-28T06:24:25.900477+0000","last_fullsized":"2024-05-28T06:24:25.900477+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:13:05.023858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2a","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:23.999700+0000","last_change":"2024-05-28T06:24:23.994663+0000","last_active":"2024-05-28T06:24:23.999700+0000","last_peered":"2024-05-28T06:24:23.999700+0000","last_clean":"2024-05-28T06:24:23.999700+0000","last_became_active":"2024-05-28T06:24:23.994410+0000","last_became_peered":"2024-05-28T06:24:23.994410+0000","last_unstale":"2024-05-28T06:24:23.999700+0000","last_undegraded":"2024-05-28T06:24:23.999700+0000","last_fullsized":"2024-05-28T06:24:23.999700+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:03:29.345159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.29","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.018227+0000","last_change":"2024-05-28T06:24:24.015438+0000","last_active":"2024-05-28T06:24:24.018227+0000","last_peered":"2024-05-28T06:24:24.018227+0000","last_clean":"2024-05-28T06:24:24.018227+0000","last_became_active":"2024-05-28T06:24:24.015124+0000","last_became_peered":"2024-05-28T06:24:24.015124+0000","last_unstale":"2024-05-28T06:24:24.018227+0000","last_undegraded":"2024-05-28T06:24:24.018227+0000","last_fullsized":"2024-05-28T06:24:24.018227+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:23:32.514259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.28","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.017967+0000","last_change":"2024-05-28T06:24:24.017147+0000","last_active":"2024-05-28T06:24:24.017967+0000","last_peered":"2024-05-28T06:24:24.017967+0000","last_clean":"2024-05-28T06:24:24.017967+0000","last_became_active":"2024-05-28T06:24:24.016692+0000","last_became_peered":"2024-05-28T06:24:24.016692+0000","last_unstale":"2024-05-28T06:24:24.017967+0000","last_undegraded":"2024-05-28T06:24:24.017967+0000","last_fullsized":"2024-05-28T06:24:24.017967+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:25:56.671088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.27","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.008439+0000","last_change":"2024-05-28T06:24:24.007398+0000","last_active":"2024-05-28T06:24:24.008439+0000","last_peered":"2024-05-28T06:24:24.008439+0000","last_clean":"2024-05-28T06:24:24.008439+0000","last_became_active":"2024-05-28T06:24:24.006983+0000","last_became_peered":"2024-05-28T06:24:24.006983+0000","last_unstale":"2024-05-28T06:24:24.008439+0000","last_undegraded":"2024-05-28T06:24:24.008439+0000","last_fullsized":"2024-05-28T06:24:24.008439+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:27:41.877671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.26","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.000526+0000","last_change":"2024-05-28T06:24:23.999225+0000","last_active":"2024-05-28T06:24:24.000526+0000","last_peered":"2024-05-28T06:24:24.000526+0000","last_clean":"2024-05-28T06:24:24.000526+0000","last_became_active":"2024-05-28T06:24:23.996859+0000","last_became_peered":"2024-05-28T06:24:23.996859+0000","last_unstale":"2024-05-28T06:24:24.000526+0000","last_undegraded":"2024-05-28T06:24:24.000526+0000","last_fullsized":"2024-05-28T06:24:24.000526+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:49:43.070723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.25","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.008804+0000","last_change":"2024-05-28T06:24:24.007463+0000","last_active":"2024-05-28T06:24:24.008804+0000","last_peered":"2024-05-28T06:24:24.008804+0000","last_clean":"2024-05-28T06:24:24.008804+0000","last_became_active":"2024-05-28T06:24:24.007194+0000","last_became_peered":"2024-05-28T06:24:24.007194+0000","last_unstale":"2024-05-28T06:24:24.008804+0000","last_undegraded":"2024-05-28T06:24:24.008804+0000","last_fullsized":"2024-05-28T06:24:24.008804+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:10:46.986223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.24","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.019702+0000","last_change":"2024-05-28T06:24:24.016291+0000","last_active":"2024-05-28T06:24:24.019702+0000","last_peered":"2024-05-28T06:24:24.019702+0000","last_clean":"2024-05-28T06:24:24.019702+0000","last_became_active":"2024-05-28T06:24:24.015990+0000","last_became_peered":"2024-05-28T06:24:24.015990+0000","last_unstale":"2024-05-28T06:24:24.019702+0000","last_undegraded":"2024-05-28T06:24:24.019702+0000","last_fullsized":"2024-05-28T06:24:24.019702+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:23:18.676845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.23","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.020933+0000","last_change":"2024-05-28T06:24:24.019908+0000","last_active":"2024-05-28T06:24:24.020933+0000","last_peered":"2024-05-28T06:24:24.020933+0000","last_clean":"2024-05-28T06:24:24.020933+0000","last_became_active":"2024-05-28T06:24:24.019497+0000","last_became_peered":"2024-05-28T06:24:24.019497+0000","last_unstale":"2024-05-28T06:24:24.020933+0000","last_undegraded":"2024-05-28T06:24:24.020933+0000","last_fullsized":"2024-05-28T06:24:24.020933+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:40:41.852779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.22","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.020666+0000","last_change":"2024-05-28T06:24:24.019852+0000","last_active":"2024-05-28T06:24:24.020666+0000","last_peered":"2024-05-28T06:24:24.020666+0000","last_clean":"2024-05-28T06:24:24.020666+0000","last_became_active":"2024-05-28T06:24:24.019462+0000","last_became_peered":"2024-05-28T06:24:24.019462+0000","last_unstale":"2024-05-28T06:24:24.020666+0000","last_undegraded":"2024-05-28T06:24:24.020666+0000","last_fullsized":"2024-05-28T06:24:24.020666+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:15:21.891198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.21","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-05-28T06:24:25.900369+0000","last_change":"2024-05-28T06:24:24.011632+0000","last_active":"2024-05-28T06:24:25.900369+0000","last_peered":"2024-05-28T06:24:25.900369+0000","last_clean":"2024-05-28T06:24:25.900369+0000","last_became_active":"2024-05-28T06:24:24.011246+0000","last_became_peered":"2024-05-28T06:24:24.011246+0000","last_unstale":"2024-05-28T06:24:25.900369+0000","last_undegraded":"2024-05-28T06:24:25.900369+0000","last_fullsized":"2024-05-28T06:24:25.900369+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:05:09.592919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.20","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-05-28T06:24:25.011477+0000","last_change":"2024-05-28T06:24:23.991803+0000","last_active":"2024-05-28T06:24:25.011477+0000","last_peered":"2024-05-28T06:24:25.011477+0000","last_clean":"2024-05-28T06:24:25.011477+0000","last_became_active":"2024-05-28T06:24:23.991245+0000","last_became_peered":"2024-05-28T06:24:23.991245+0000","last_unstale":"2024-05-28T06:24:25.011477+0000","last_undegraded":"2024-05-28T06:24:25.011477+0000","last_fullsized":"2024-05-28T06:24:25.011477+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:55:45.102561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.018662+0000","last_change":"2024-05-28T06:24:24.018032+0000","last_active":"2024-05-28T06:24:24.018662+0000","last_peered":"2024-05-28T06:24:24.018662+0000","last_clean":"2024-05-28T06:24:24.018662+0000","last_became_active":"2024-05-28T06:24:24.017727+0000","last_became_peered":"2024-05-28T06:24:24.017727+0000","last_unstale":"2024-05-28T06:24:24.018662+0000","last_undegraded":"2024-05-28T06:24:24.018662+0000","last_fullsized":"2024-05-28T06:24:24.018662+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:37:59.260972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1f","version":"129'4","reported_seq":18,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-05-28T06:24:25.901165+0000","last_change":"2024-05-28T06:24:23.006418+0000","last_active":"2024-05-28T06:24:25.901165+0000","last_peered":"2024-05-28T06:24:25.901165+0000","last_clean":"2024-05-28T06:24:25.901165+0000","last_became_active":"2024-05-28T06:24:23.006043+0000","last_became_peered":"2024-05-28T06:24:23.006043+0000","last_unstale":"2024-05-28T06:24:25.901165+0000","last_undegraded":"2024-05-28T06:24:25.901165+0000","last_fullsized":"2024-05-28T06:24:25.901165+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:31:42.560242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.016226+0000","last_change":"2024-05-28T06:24:24.008420+0000","last_active":"2024-05-28T06:24:24.016226+0000","last_peered":"2024-05-28T06:24:24.016226+0000","last_clean":"2024-05-28T06:24:24.016226+0000","last_became_active":"2024-05-28T06:24:24.008078+0000","last_became_peered":"2024-05-28T06:24:24.008078+0000","last_unstale":"2024-05-28T06:24:24.016226+0000","last_undegraded":"2024-05-28T06:24:24.016226+0000","last_fullsized":"2024-05-28T06:24:24.016226+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:33:49.362474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1e","version":"129'1","reported_seq":13,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.017323+0000","last_change":"2024-05-28T06:24:22.995399+0000","last_active":"2024-05-28T06:24:24.017323+0000","last_peered":"2024-05-28T06:24:24.017323+0000","last_clean":"2024-05-28T06:24:24.017323+0000","last_became_active":"2024-05-28T06:24:22.994896+0000","last_became_peered":"2024-05-28T06:24:22.994896+0000","last_unstale":"2024-05-28T06:24:24.017323+0000","last_undegraded":"2024-05-28T06:24:24.017323+0000","last_fullsized":"2024-05-28T06:24:24.017323+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:06:41.660486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-05-28T06:24:25.901145+0000","last_change":"2024-05-28T06:24:24.006543+0000","last_active":"2024-05-28T06:24:25.901145+0000","last_peered":"2024-05-28T06:24:25.901145+0000","last_clean":"2024-05-28T06:24:25.901145+0000","last_became_active":"2024-05-28T06:24:24.006258+0000","last_became_peered":"2024-05-28T06:24:24.006258+0000","last_unstale":"2024-05-28T06:24:25.901145+0000","last_undegraded":"2024-05-28T06:24:25.901145+0000","last_fullsized":"2024-05-28T06:24:25.901145+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:34:58.279352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1d","version":"129'2","reported_seq":14,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.000303+0000","last_change":"2024-05-28T06:24:22.992163+0000","last_active":"2024-05-28T06:24:24.000303+0000","last_peered":"2024-05-28T06:24:24.000303+0000","last_clean":"2024-05-28T06:24:24.000303+0000","last_became_active":"2024-05-28T06:24:22.991407+0000","last_became_peered":"2024-05-28T06:24:22.991407+0000","last_unstale":"2024-05-28T06:24:24.000303+0000","last_undegraded":"2024-05-28T06:24:24.000303+0000","last_fullsized":"2024-05-28T06:24:24.000303+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:34:43.911764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-05-28T06:24:25.900703+0000","last_change":"2024-05-28T06:24:24.013028+0000","last_active":"2024-05-28T06:24:25.900703+0000","last_peered":"2024-05-28T06:24:25.900703+0000","last_clean":"2024-05-28T06:24:25.900703+0000","last_became_active":"2024-05-28T06:24:24.010356+0000","last_became_peered":"2024-05-28T06:24:24.010356+0000","last_unstale":"2024-05-28T06:24:25.900703+0000","last_undegraded":"2024-05-28T06:24:25.900703+0000","last_fullsized":"2024-05-28T06:24:25.900703+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:46:38.879485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1c","version":"129'2","reported_seq":14,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:23.997825+0000","last_change":"2024-05-28T06:24:22.989493+0000","last_active":"2024-05-28T06:24:23.997825+0000","last_peered":"2024-05-28T06:24:23.997825+0000","last_clean":"2024-05-28T06:24:23.997825+0000","last_became_active":"2024-05-28T06:24:22.989103+0000","last_became_peered":"2024-05-28T06:24:22.989103+0000","last_unstale":"2024-05-28T06:24:23.997825+0000","last_undegraded":"2024-05-28T06:24:23.997825+0000","last_fullsized":"2024-05-28T06:24:23.997825+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:28:25.855171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.009066+0000","last_change":"2024-05-28T06:24:24.008297+0000","last_active":"2024-05-28T06:24:24.009066+0000","last_peered":"2024-05-28T06:24:24.009066+0000","last_clean":"2024-05-28T06:24:24.009066+0000","last_became_active":"2024-05-28T06:24:24.007987+0000","last_became_peered":"2024-05-28T06:24:24.007987+0000","last_unstale":"2024-05-28T06:24:24.009066+0000","last_undegraded":"2024-05-28T06:24:24.009066+0000","last_fullsized":"2024-05-28T06:24:24.009066+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:55:45.102561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1b","version":"129'1","reported_seq":13,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:23.998832+0000","last_change":"2024-05-28T06:24:22.987047+0000","last_active":"2024-05-28T06:24:23.998832+0000","last_peered":"2024-05-28T06:24:23.998832+0000","last_clean":"2024-05-28T06:24:23.998832+0000","last_became_active":"2024-05-28T06:24:22.985617+0000","last_became_peered":"2024-05-28T06:24:22.985617+0000","last_unstale":"2024-05-28T06:24:23.998832+0000","last_undegraded":"2024-05-28T06:24:23.998832+0000","last_fullsized":"2024-05-28T06:24:23.998832+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:59:04.673636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-05-28T06:24:25.012080+0000","last_change":"2024-05-28T06:24:23.987152+0000","last_active":"2024-05-28T06:24:25.012080+0000","last_peered":"2024-05-28T06:24:25.012080+0000","last_clean":"2024-05-28T06:24:25.012080+0000","last_became_active":"2024-05-28T06:24:23.985828+0000","last_became_peered":"2024-05-28T06:24:23.985828+0000","last_unstale":"2024-05-28T06:24:25.012080+0000","last_undegraded":"2024-05-28T06:24:25.012080+0000","last_fullsized":"2024-05-28T06:24:25.012080+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:48:49.925554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1a","version":"129'1","reported_seq":13,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:23.996435+0000","last_change":"2024-05-28T06:24:22.990085+0000","last_active":"2024-05-28T06:24:23.996435+0000","last_peered":"2024-05-28T06:24:23.996435+0000","last_clean":"2024-05-28T06:24:23.996435+0000","last_became_active":"2024-05-28T06:24:22.989801+0000","last_became_peered":"2024-05-28T06:24:22.989801+0000","last_unstale":"2024-05-28T06:24:23.996435+0000","last_undegraded":"2024-05-28T06:24:23.996435+0000","last_fullsized":"2024-05-28T06:24:23.996435+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:53:06.353639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.011512+0000","last_change":"2024-05-28T06:24:24.003806+0000","last_active":"2024-05-28T06:24:24.011512+0000","last_peered":"2024-05-28T06:24:24.011512+0000","last_clean":"2024-05-28T06:24:24.011512+0000","last_became_active":"2024-05-28T06:24:24.003446+0000","last_became_peered":"2024-05-28T06:24:24.003446+0000","last_unstale":"2024-05-28T06:24:24.011512+0000","last_undegraded":"2024-05-28T06:24:24.011512+0000","last_fullsized":"2024-05-28T06:24:24.011512+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:12:39.142307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-05-28T06:24:25.011943+0000","last_change":"2024-05-28T06:24:23.007861+0000","last_active":"2024-05-28T06:24:25.011943+0000","last_peered":"2024-05-28T06:24:25.011943+0000","last_clean":"2024-05-28T06:24:25.011943+0000","last_became_active":"2024-05-28T06:24:23.007494+0000","last_became_peered":"2024-05-28T06:24:23.007494+0000","last_unstale":"2024-05-28T06:24:25.011943+0000","last_undegraded":"2024-05-28T06:24:25.011943+0000","last_fullsized":"2024-05-28T06:24:25.011943+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:06:19.878458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:23.994727+0000","last_change":"2024-05-28T06:24:23.993309+0000","last_active":"2024-05-28T06:24:23.994727+0000","last_peered":"2024-05-28T06:24:23.994727+0000","last_clean":"2024-05-28T06:24:23.994727+0000","last_became_active":"2024-05-28T06:24:23.987466+0000","last_became_peered":"2024-05-28T06:24:23.987466+0000","last_unstale":"2024-05-28T06:24:23.994727+0000","last_undegraded":"2024-05-28T06:24:23.994727+0000","last_fullsized":"2024-05-28T06:24:23.994727+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:59:20.653065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:23.975815+0000","last_change":"2024-05-28T06:24:23.000407+0000","last_active":"2024-05-28T06:24:23.975815+0000","last_peered":"2024-05-28T06:24:23.975815+0000","last_clean":"2024-05-28T06:24:23.975815+0000","last_became_active":"2024-05-28T06:24:23.000037+0000","last_became_peered":"2024-05-28T06:24:23.000037+0000","last_unstale":"2024-05-28T06:24:23.975815+0000","last_undegraded":"2024-05-28T06:24:23.975815+0000","last_fullsized":"2024-05-28T06:24:23.975815+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:59:08.824871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:23.996246+0000","last_change":"2024-05-28T06:24:23.995548+0000","last_active":"2024-05-28T06:24:23.996246+0000","last_peered":"2024-05-28T06:24:23.996246+0000","last_clean":"2024-05-28T06:24:23.996246+0000","last_became_active":"2024-05-28T06:24:23.995204+0000","last_became_peered":"2024-05-28T06:24:23.995204+0000","last_unstale":"2024-05-28T06:24:23.996246+0000","last_undegraded":"2024-05-28T06:24:23.996246+0000","last_fullsized":"2024-05-28T06:24:23.996246+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:00:42.665433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.005289+0000","last_change":"2024-05-28T06:24:23.001114+0000","last_active":"2024-05-28T06:24:24.005289+0000","last_peered":"2024-05-28T06:24:24.005289+0000","last_clean":"2024-05-28T06:24:24.005289+0000","last_became_active":"2024-05-28T06:24:22.999720+0000","last_became_peered":"2024-05-28T06:24:22.999720+0000","last_unstale":"2024-05-28T06:24:24.005289+0000","last_undegraded":"2024-05-28T06:24:24.005289+0000","last_fullsized":"2024-05-28T06:24:24.005289+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:29:54.261773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.021370+0000","last_change":"2024-05-28T06:24:24.016902+0000","last_active":"2024-05-28T06:24:24.021370+0000","last_peered":"2024-05-28T06:24:24.021370+0000","last_clean":"2024-05-28T06:24:24.021370+0000","last_became_active":"2024-05-28T06:24:24.016597+0000","last_became_peered":"2024-05-28T06:24:24.016597+0000","last_unstale":"2024-05-28T06:24:24.021370+0000","last_undegraded":"2024-05-28T06:24:24.021370+0000","last_fullsized":"2024-05-28T06:24:24.021370+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:52:44.413787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:23.985715+0000","last_change":"2024-05-28T06:24:23.001188+0000","last_active":"2024-05-28T06:24:23.985715+0000","last_peered":"2024-05-28T06:24:23.985715+0000","last_clean":"2024-05-28T06:24:23.985715+0000","last_became_active":"2024-05-28T06:24:23.000878+0000","last_became_peered":"2024-05-28T06:24:23.000878+0000","last_unstale":"2024-05-28T06:24:23.985715+0000","last_undegraded":"2024-05-28T06:24:23.985715+0000","last_fullsized":"2024-05-28T06:24:23.985715+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:18:06.253798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-05-28T06:24:25.901305+0000","last_change":"2024-05-28T06:24:24.011311+0000","last_active":"2024-05-28T06:24:25.901305+0000","last_peered":"2024-05-28T06:24:25.901305+0000","last_clean":"2024-05-28T06:24:25.901305+0000","last_became_active":"2024-05-28T06:24:24.010110+0000","last_became_peered":"2024-05-28T06:24:24.010110+0000","last_unstale":"2024-05-28T06:24:25.901305+0000","last_undegraded":"2024-05-28T06:24:25.901305+0000","last_fullsized":"2024-05-28T06:24:25.901305+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:46:31.105891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.15","version":"129'3","reported_seq":15,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.017657+0000","last_change":"2024-05-28T06:24:22.999088+0000","last_active":"2024-05-28T06:24:24.017657+0000","last_peered":"2024-05-28T06:24:24.017657+0000","last_clean":"2024-05-28T06:24:24.017657+0000","last_became_active":"2024-05-28T06:24:22.998379+0000","last_became_peered":"2024-05-28T06:24:22.998379+0000","last_unstale":"2024-05-28T06:24:24.017657+0000","last_undegraded":"2024-05-28T06:24:24.017657+0000","last_fullsized":"2024-05-28T06:24:24.017657+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:59:19.637147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-05-28T06:24:25.901246+0000","last_change":"2024-05-28T06:24:24.020278+0000","last_active":"2024-05-28T06:24:25.901246+0000","last_peered":"2024-05-28T06:24:25.901246+0000","last_clean":"2024-05-28T06:24:25.901246+0000","last_became_active":"2024-05-28T06:24:24.019973+0000","last_became_peered":"2024-05-28T06:24:24.019973+0000","last_unstale":"2024-05-28T06:24:25.901246+0000","last_undegraded":"2024-05-28T06:24:25.901246+0000","last_fullsized":"2024-05-28T06:24:25.901246+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:46:47.603731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.14","version":"129'6","reported_seq":18,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.023651+0000","last_change":"2024-05-28T06:24:23.005145+0000","last_active":"2024-05-28T06:24:24.023651+0000","last_peered":"2024-05-28T06:24:24.023651+0000","last_clean":"2024-05-28T06:24:24.023651+0000","last_became_active":"2024-05-28T06:24:23.004831+0000","last_became_peered":"2024-05-28T06:24:23.004831+0000","last_unstale":"2024-05-28T06:24:24.023651+0000","last_undegraded":"2024-05-28T06:24:24.023651+0000","last_fullsized":"2024-05-28T06:24:24.023651+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:09:55.668305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.004123+0000","last_change":"2024-05-28T06:24:24.002061+0000","last_active":"2024-05-28T06:24:24.004123+0000","last_peered":"2024-05-28T06:24:24.004123+0000","last_clean":"2024-05-28T06:24:24.004123+0000","last_became_active":"2024-05-28T06:24:24.001429+0000","last_became_peered":"2024-05-28T06:24:24.001429+0000","last_unstale":"2024-05-28T06:24:24.004123+0000","last_undegraded":"2024-05-28T06:24:24.004123+0000","last_fullsized":"2024-05-28T06:24:24.004123+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:19:21.657411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:23.989711+0000","last_change":"2024-05-28T06:24:22.993624+0000","last_active":"2024-05-28T06:24:23.989711+0000","last_peered":"2024-05-28T06:24:23.989711+0000","last_clean":"2024-05-28T06:24:23.989711+0000","last_became_active":"2024-05-28T06:24:22.993301+0000","last_became_peered":"2024-05-28T06:24:22.993301+0000","last_unstale":"2024-05-28T06:24:23.989711+0000","last_undegraded":"2024-05-28T06:24:23.989711+0000","last_fullsized":"2024-05-28T06:24:23.989711+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:26:44.582201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.006076+0000","last_change":"2024-05-28T06:24:24.005361+0000","last_active":"2024-05-28T06:24:24.006076+0000","last_peered":"2024-05-28T06:24:24.006076+0000","last_clean":"2024-05-28T06:24:24.006076+0000","last_became_active":"2024-05-28T06:24:24.005080+0000","last_became_peered":"2024-05-28T06:24:24.005080+0000","last_unstale":"2024-05-28T06:24:24.006076+0000","last_undegraded":"2024-05-28T06:24:24.006076+0000","last_fullsized":"2024-05-28T06:24:24.006076+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:43:02.389778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:23.976103+0000","last_change":"2024-05-28T06:24:23.005565+0000","last_active":"2024-05-28T06:24:23.976103+0000","last_peered":"2024-05-28T06:24:23.976103+0000","last_clean":"2024-05-28T06:24:23.976103+0000","last_became_active":"2024-05-28T06:24:23.005194+0000","last_became_peered":"2024-05-28T06:24:23.005194+0000","last_unstale":"2024-05-28T06:24:23.976103+0000","last_undegraded":"2024-05-28T06:24:23.976103+0000","last_fullsized":"2024-05-28T06:24:23.976103+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:26:55.784217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.018042+0000","last_change":"2024-05-28T06:24:24.009724+0000","last_active":"2024-05-28T06:24:24.018042+0000","last_peered":"2024-05-28T06:24:24.018042+0000","last_clean":"2024-05-28T06:24:24.018042+0000","last_became_active":"2024-05-28T06:24:24.009163+0000","last_became_peered":"2024-05-28T06:24:24.009163+0000","last_unstale":"2024-05-28T06:24:24.018042+0000","last_undegraded":"2024-05-28T06:24:24.018042+0000","last_fullsized":"2024-05-28T06:24:24.018042+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:24:28.571714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.10","version":"129'8","reported_seq":20,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.017620+0000","last_change":"2024-05-28T06:24:22.996927+0000","last_active":"2024-05-28T06:24:24.017620+0000","last_peered":"2024-05-28T06:24:24.017620+0000","last_clean":"2024-05-28T06:24:24.017620+0000","last_became_active":"2024-05-28T06:24:22.996644+0000","last_became_peered":"2024-05-28T06:24:22.996644+0000","last_unstale":"2024-05-28T06:24:24.017620+0000","last_undegraded":"2024-05-28T06:24:24.017620+0000","last_fullsized":"2024-05-28T06:24:24.017620+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:17:26.759155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.018242+0000","last_change":"2024-05-28T06:24:24.009329+0000","last_active":"2024-05-28T06:24:24.018242+0000","last_peered":"2024-05-28T06:24:24.018242+0000","last_clean":"2024-05-28T06:24:24.018242+0000","last_became_active":"2024-05-28T06:24:24.009033+0000","last_became_peered":"2024-05-28T06:24:24.009033+0000","last_unstale":"2024-05-28T06:24:24.018242+0000","last_undegraded":"2024-05-28T06:24:24.018242+0000","last_fullsized":"2024-05-28T06:24:24.018242+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:48:43.340748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.f","version":"129'1","reported_seq":13,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.017941+0000","last_change":"2024-05-28T06:24:23.006567+0000","last_active":"2024-05-28T06:24:24.017941+0000","last_peered":"2024-05-28T06:24:24.017941+0000","last_clean":"2024-05-28T06:24:24.017941+0000","last_became_active":"2024-05-28T06:24:23.005650+0000","last_became_peered":"2024-05-28T06:24:23.005650+0000","last_unstale":"2024-05-28T06:24:24.017941+0000","last_undegraded":"2024-05-28T06:24:24.017941+0000","last_fullsized":"2024-05-28T06:24:24.017941+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:01:42.283187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.021344+0000","last_change":"2024-05-28T06:24:24.016763+0000","last_active":"2024-05-28T06:24:24.021344+0000","last_peered":"2024-05-28T06:24:24.021344+0000","last_clean":"2024-05-28T06:24:24.021344+0000","last_became_active":"2024-05-28T06:24:24.016255+0000","last_became_peered":"2024-05-28T06:24:24.016255+0000","last_unstale":"2024-05-28T06:24:24.021344+0000","last_undegraded":"2024-05-28T06:24:24.021344+0000","last_fullsized":"2024-05-28T06:24:24.021344+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:11:18.017207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.e","version":"129'2","reported_seq":14,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.015790+0000","last_change":"2024-05-28T06:24:23.002513+0000","last_active":"2024-05-28T06:24:24.015790+0000","last_peered":"2024-05-28T06:24:24.015790+0000","last_clean":"2024-05-28T06:24:24.015790+0000","last_became_active":"2024-05-28T06:24:23.001946+0000","last_became_peered":"2024-05-28T06:24:23.001946+0000","last_unstale":"2024-05-28T06:24:24.015790+0000","last_undegraded":"2024-05-28T06:24:24.015790+0000","last_fullsized":"2024-05-28T06:24:24.015790+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:27:48.979163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:23.999634+0000","last_change":"2024-05-28T06:24:23.998032+0000","last_active":"2024-05-28T06:24:23.999634+0000","last_peered":"2024-05-28T06:24:23.999634+0000","last_clean":"2024-05-28T06:24:23.999634+0000","last_became_active":"2024-05-28T06:24:23.996433+0000","last_became_peered":"2024-05-28T06:24:23.996433+0000","last_unstale":"2024-05-28T06:24:23.999634+0000","last_undegraded":"2024-05-28T06:24:23.999634+0000","last_fullsized":"2024-05-28T06:24:23.999634+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:01:59.233483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.d","version":"129'1","reported_seq":15,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-05-28T06:24:25.012501+0000","last_change":"2024-05-28T06:24:22.992776+0000","last_active":"2024-05-28T06:24:25.012501+0000","last_peered":"2024-05-28T06:24:25.012501+0000","last_clean":"2024-05-28T06:24:25.012501+0000","last_became_active":"2024-05-28T06:24:22.992147+0000","last_became_peered":"2024-05-28T06:24:22.992147+0000","last_unstale":"2024-05-28T06:24:25.012501+0000","last_undegraded":"2024-05-28T06:24:25.012501+0000","last_fullsized":"2024-05-28T06:24:25.012501+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:36:09.562056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:23.992367+0000","last_change":"2024-05-28T06:24:23.989538+0000","last_active":"2024-05-28T06:24:23.992367+0000","last_peered":"2024-05-28T06:24:23.992367+0000","last_clean":"2024-05-28T06:24:23.992367+0000","last_became_active":"2024-05-28T06:24:23.988631+0000","last_became_peered":"2024-05-28T06:24:23.988631+0000","last_unstale":"2024-05-28T06:24:23.992367+0000","last_undegraded":"2024-05-28T06:24:23.992367+0000","last_fullsized":"2024-05-28T06:24:23.992367+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:40:57.763291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.007020+0000","last_change":"2024-05-28T06:24:22.997400+0000","last_active":"2024-05-28T06:24:24.007020+0000","last_peered":"2024-05-28T06:24:24.007020+0000","last_clean":"2024-05-28T06:24:24.007020+0000","last_became_active":"2024-05-28T06:24:22.993636+0000","last_became_peered":"2024-05-28T06:24:22.993636+0000","last_unstale":"2024-05-28T06:24:24.007020+0000","last_undegraded":"2024-05-28T06:24:24.007020+0000","last_fullsized":"2024-05-28T06:24:24.007020+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:43:01.373860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.008871+0000","last_change":"2024-05-28T06:24:24.007538+0000","last_active":"2024-05-28T06:24:24.008871+0000","last_peered":"2024-05-28T06:24:24.008871+0000","last_clean":"2024-05-28T06:24:24.008871+0000","last_became_active":"2024-05-28T06:24:24.007259+0000","last_became_peered":"2024-05-28T06:24:24.007259+0000","last_unstale":"2024-05-28T06:24:24.008871+0000","last_undegraded":"2024-05-28T06:24:24.008871+0000","last_fullsized":"2024-05-28T06:24:24.008871+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:49:25.488174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.b","version":"129'2","reported_seq":14,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.018078+0000","last_change":"2024-05-28T06:24:22.996819+0000","last_active":"2024-05-28T06:24:24.018078+0000","last_peered":"2024-05-28T06:24:24.018078+0000","last_clean":"2024-05-28T06:24:24.018078+0000","last_became_active":"2024-05-28T06:24:22.995489+0000","last_became_peered":"2024-05-28T06:24:22.995489+0000","last_unstale":"2024-05-28T06:24:24.018078+0000","last_undegraded":"2024-05-28T06:24:24.018078+0000","last_fullsized":"2024-05-28T06:24:24.018078+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:24:52.967145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-05-28T06:24:25.012022+0000","last_change":"2024-05-28T06:24:24.015790+0000","last_active":"2024-05-28T06:24:25.012022+0000","last_peered":"2024-05-28T06:24:25.012022+0000","last_clean":"2024-05-28T06:24:25.012022+0000","last_became_active":"2024-05-28T06:24:24.015295+0000","last_became_peered":"2024-05-28T06:24:24.015295+0000","last_unstale":"2024-05-28T06:24:25.012022+0000","last_undegraded":"2024-05-28T06:24:25.012022+0000","last_fullsized":"2024-05-28T06:24:25.012022+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:37:59.260972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:23.985595+0000","last_change":"2024-05-28T06:24:22.995343+0000","last_active":"2024-05-28T06:24:23.985595+0000","last_peered":"2024-05-28T06:24:23.985595+0000","last_clean":"2024-05-28T06:24:23.985595+0000","last_became_active":"2024-05-28T06:24:22.994227+0000","last_became_peered":"2024-05-28T06:24:22.994227+0000","last_unstale":"2024-05-28T06:24:23.985595+0000","last_undegraded":"2024-05-28T06:24:23.985595+0000","last_fullsized":"2024-05-28T06:24:23.985595+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:50:39.661313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:23.991209+0000","last_change":"2024-05-28T06:24:23.987172+0000","last_active":"2024-05-28T06:24:23.991209+0000","last_peered":"2024-05-28T06:24:23.991209+0000","last_clean":"2024-05-28T06:24:23.991209+0000","last_became_active":"2024-05-28T06:24:23.986897+0000","last_became_peered":"2024-05-28T06:24:23.986897+0000","last_unstale":"2024-05-28T06:24:23.991209+0000","last_undegraded":"2024-05-28T06:24:23.991209+0000","last_fullsized":"2024-05-28T06:24:23.991209+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T18:06:22.409116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:23.983196+0000","last_change":"2024-05-28T06:24:22.991355+0000","last_active":"2024-05-28T06:24:23.983196+0000","last_peered":"2024-05-28T06:24:23.983196+0000","last_clean":"2024-05-28T06:24:23.983196+0000","last_became_active":"2024-05-28T06:24:22.991078+0000","last_became_peered":"2024-05-28T06:24:22.991078+0000","last_unstale":"2024-05-28T06:24:23.983196+0000","last_undegraded":"2024-05-28T06:24:23.983196+0000","last_fullsized":"2024-05-28T06:24:23.983196+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:06:19.878458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-05-28T06:24:25.900149+0000","last_change":"2024-05-28T06:24:24.009275+0000","last_active":"2024-05-28T06:24:25.900149+0000","last_peered":"2024-05-28T06:24:25.900149+0000","last_clean":"2024-05-28T06:24:25.900149+0000","last_became_active":"2024-05-28T06:24:24.008955+0000","last_became_peered":"2024-05-28T06:24:24.008955+0000","last_unstale":"2024-05-28T06:24:25.900149+0000","last_undegraded":"2024-05-28T06:24:25.900149+0000","last_fullsized":"2024-05-28T06:24:25.900149+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:32:09.823976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:23.981770+0000","last_change":"2024-05-28T06:24:23.002572+0000","last_active":"2024-05-28T06:24:23.981770+0000","last_peered":"2024-05-28T06:24:23.981770+0000","last_clean":"2024-05-28T06:24:23.981770+0000","last_became_active":"2024-05-28T06:24:23.002292+0000","last_became_peered":"2024-05-28T06:24:23.002292+0000","last_unstale":"2024-05-28T06:24:23.981770+0000","last_undegraded":"2024-05-28T06:24:23.981770+0000","last_fullsized":"2024-05-28T06:24:23.981770+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:50:39.856735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-05-28T06:24:25.013427+0000","last_change":"2024-05-28T06:24:24.017529+0000","last_active":"2024-05-28T06:24:25.013427+0000","last_peered":"2024-05-28T06:24:25.013427+0000","last_clean":"2024-05-28T06:24:25.013427+0000","last_became_active":"2024-05-28T06:24:24.017221+0000","last_became_peered":"2024-05-28T06:24:24.017221+0000","last_unstale":"2024-05-28T06:24:25.013427+0000","last_undegraded":"2024-05-28T06:24:25.013427+0000","last_fullsized":"2024-05-28T06:24:25.013427+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:32:09.823976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.7","version":"129'2","reported_seq":14,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.019314+0000","last_change":"2024-05-28T06:24:23.003960+0000","last_active":"2024-05-28T06:24:24.019314+0000","last_peered":"2024-05-28T06:24:24.019314+0000","last_clean":"2024-05-28T06:24:24.019314+0000","last_became_active":"2024-05-28T06:24:23.003502+0000","last_became_peered":"2024-05-28T06:24:23.003502+0000","last_unstale":"2024-05-28T06:24:24.019314+0000","last_undegraded":"2024-05-28T06:24:24.019314+0000","last_fullsized":"2024-05-28T06:24:24.019314+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:13:19.450560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.020939+0000","last_change":"2024-05-28T06:24:24.020448+0000","last_active":"2024-05-28T06:24:24.020939+0000","last_peered":"2024-05-28T06:24:24.020939+0000","last_clean":"2024-05-28T06:24:24.020939+0000","last_became_active":"2024-05-28T06:24:24.019730+0000","last_became_peered":"2024-05-28T06:24:24.019730+0000","last_unstale":"2024-05-28T06:24:24.020939+0000","last_undegraded":"2024-05-28T06:24:24.020939+0000","last_fullsized":"2024-05-28T06:24:24.020939+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:25:49.241649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.6","version":"130'1","reported_seq":15,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-05-28T06:24:25.988359+0000","last_change":"2024-05-28T06:24:23.003133+0000","last_active":"2024-05-28T06:24:25.988359+0000","last_peered":"2024-05-28T06:24:25.988359+0000","last_clean":"2024-05-28T06:24:25.988359+0000","last_became_active":"2024-05-28T06:24:23.002821+0000","last_became_peered":"2024-05-28T06:24:23.002821+0000","last_unstale":"2024-05-28T06:24:25.988359+0000","last_undegraded":"2024-05-28T06:24:25.988359+0000","last_fullsized":"2024-05-28T06:24:25.988359+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:18:06.253798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.005933+0000","last_change":"2024-05-28T06:24:24.004928+0000","last_active":"2024-05-28T06:24:24.005933+0000","last_peered":"2024-05-28T06:24:24.005933+0000","last_clean":"2024-05-28T06:24:24.005933+0000","last_became_active":"2024-05-28T06:24:24.004599+0000","last_became_peered":"2024-05-28T06:24:24.004599+0000","last_unstale":"2024-05-28T06:24:24.005933+0000","last_undegraded":"2024-05-28T06:24:24.005933+0000","last_fullsized":"2024-05-28T06:24:24.005933+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:01:43.299104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:23.976699+0000","last_change":"2024-05-28T06:24:23.003603+0000","last_active":"2024-05-28T06:24:23.976699+0000","last_peered":"2024-05-28T06:24:23.976699+0000","last_clean":"2024-05-28T06:24:23.976699+0000","last_became_active":"2024-05-28T06:24:23.002468+0000","last_became_peered":"2024-05-28T06:24:23.002468+0000","last_unstale":"2024-05-28T06:24:23.976699+0000","last_undegraded":"2024-05-28T06:24:23.976699+0000","last_fullsized":"2024-05-28T06:24:23.976699+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:48:27.218311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-05-28T06:24:25.012540+0000","last_change":"2024-05-28T06:24:23.999563+0000","last_active":"2024-05-28T06:24:25.012540+0000","last_peered":"2024-05-28T06:24:25.012540+0000","last_clean":"2024-05-28T06:24:25.012540+0000","last_became_active":"2024-05-28T06:24:23.999013+0000","last_became_peered":"2024-05-28T06:24:23.999013+0000","last_unstale":"2024-05-28T06:24:25.012540+0000","last_undegraded":"2024-05-28T06:24:25.012540+0000","last_fullsized":"2024-05-28T06:24:25.012540+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:47:25.671388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:23.984679+0000","last_change":"2024-05-28T06:24:22.986876+0000","last_active":"2024-05-28T06:24:23.984679+0000","last_peered":"2024-05-28T06:24:23.984679+0000","last_clean":"2024-05-28T06:24:23.984679+0000","last_became_active":"2024-05-28T06:24:22.986620+0000","last_became_peered":"2024-05-28T06:24:22.986620+0000","last_unstale":"2024-05-28T06:24:23.984679+0000","last_undegraded":"2024-05-28T06:24:23.984679+0000","last_fullsized":"2024-05-28T06:24:23.984679+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:20:23.711600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-05-28T06:24:25.900260+0000","last_change":"2024-05-28T06:24:24.012293+0000","last_active":"2024-05-28T06:24:25.900260+0000","last_peered":"2024-05-28T06:24:25.900260+0000","last_clean":"2024-05-28T06:24:25.900260+0000","last_became_active":"2024-05-28T06:24:24.011989+0000","last_became_peered":"2024-05-28T06:24:24.011989+0000","last_unstale":"2024-05-28T06:24:25.900260+0000","last_undegraded":"2024-05-28T06:24:25.900260+0000","last_fullsized":"2024-05-28T06:24:25.900260+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:47:31.291023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:23.976319+0000","last_change":"2024-05-28T06:24:23.008363+0000","last_active":"2024-05-28T06:24:23.976319+0000","last_peered":"2024-05-28T06:24:23.976319+0000","last_clean":"2024-05-28T06:24:23.976319+0000","last_became_active":"2024-05-28T06:24:23.007859+0000","last_became_peered":"2024-05-28T06:24:23.007859+0000","last_unstale":"2024-05-28T06:24:23.976319+0000","last_undegraded":"2024-05-28T06:24:23.976319+0000","last_fullsized":"2024-05-28T06:24:23.976319+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:45:32.156309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.017941+0000","last_change":"2024-05-28T06:24:24.013906+0000","last_active":"2024-05-28T06:24:24.017941+0000","last_peered":"2024-05-28T06:24:24.017941+0000","last_clean":"2024-05-28T06:24:24.017941+0000","last_became_active":"2024-05-28T06:24:24.011269+0000","last_became_peered":"2024-05-28T06:24:24.011269+0000","last_unstale":"2024-05-28T06:24:24.017941+0000","last_undegraded":"2024-05-28T06:24:24.017941+0000","last_fullsized":"2024-05-28T06:24:24.017941+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:27:41.877671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-05-28T06:24:25.900546+0000","last_change":"2024-05-28T06:24:23.006728+0000","last_active":"2024-05-28T06:24:25.900546+0000","last_peered":"2024-05-28T06:24:25.900546+0000","last_clean":"2024-05-28T06:24:25.900546+0000","last_became_active":"2024-05-28T06:24:23.005895+0000","last_became_peered":"2024-05-28T06:24:23.005895+0000","last_unstale":"2024-05-28T06:24:25.900546+0000","last_undegraded":"2024-05-28T06:24:25.900546+0000","last_fullsized":"2024-05-28T06:24:25.900546+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:53:06.353639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.013602+0000","last_change":"2024-05-28T06:24:24.013072+0000","last_active":"2024-05-28T06:24:24.013602+0000","last_peered":"2024-05-28T06:24:24.013602+0000","last_clean":"2024-05-28T06:24:24.013602+0000","last_became_active":"2024-05-28T06:24:24.012814+0000","last_became_peered":"2024-05-28T06:24:24.012814+0000","last_unstale":"2024-05-28T06:24:24.013602+0000","last_undegraded":"2024-05-28T06:24:24.013602+0000","last_fullsized":"2024-05-28T06:24:24.013602+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:47:32.965627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:23.983354+0000","last_change":"2024-05-28T06:24:23.001199+0000","last_active":"2024-05-28T06:24:23.983354+0000","last_peered":"2024-05-28T06:24:23.983354+0000","last_clean":"2024-05-28T06:24:23.983354+0000","last_became_active":"2024-05-28T06:24:23.000861+0000","last_became_peered":"2024-05-28T06:24:23.000861+0000","last_unstale":"2024-05-28T06:24:23.983354+0000","last_undegraded":"2024-05-28T06:24:23.983354+0000","last_fullsized":"2024-05-28T06:24:23.983354+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:36:09.562056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.006877+0000","last_change":"2024-05-28T06:24:24.005940+0000","last_active":"2024-05-28T06:24:24.006877+0000","last_peered":"2024-05-28T06:24:24.006877+0000","last_clean":"2024-05-28T06:24:24.006877+0000","last_became_active":"2024-05-28T06:24:24.005485+0000","last_became_peered":"2024-05-28T06:24:24.005485+0000","last_unstale":"2024-05-28T06:24:24.006877+0000","last_undegraded":"2024-05-28T06:24:24.006877+0000","last_fullsized":"2024-05-28T06:24:24.006877+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:24:53.983062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:23.981899+0000","last_change":"2024-05-28T06:24:22.987451+0000","last_active":"2024-05-28T06:24:23.981899+0000","last_peered":"2024-05-28T06:24:23.981899+0000","last_clean":"2024-05-28T06:24:23.981899+0000","last_became_active":"2024-05-28T06:24:22.987181+0000","last_became_peered":"2024-05-28T06:24:22.987181+0000","last_unstale":"2024-05-28T06:24:23.981899+0000","last_undegraded":"2024-05-28T06:24:23.981899+0000","last_fullsized":"2024-05-28T06:24:23.981899+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:45:51.036866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-05-28T06:24:25.900863+0000","last_change":"2024-05-28T06:24:24.007933+0000","last_active":"2024-05-28T06:24:25.900863+0000","last_peered":"2024-05-28T06:24:25.900863+0000","last_clean":"2024-05-28T06:24:25.900863+0000","last_became_active":"2024-05-28T06:24:24.007651+0000","last_became_peered":"2024-05-28T06:24:24.007651+0000","last_unstale":"2024-05-28T06:24:25.900863+0000","last_undegraded":"2024-05-28T06:24:25.900863+0000","last_fullsized":"2024-05-28T06:24:25.900863+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:59:32.549372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"129'2","reported_seq":16,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-05-28T06:24:25.900772+0000","last_change":"2024-05-28T06:24:23.008078+0000","last_active":"2024-05-28T06:24:25.900772+0000","last_peered":"2024-05-28T06:24:25.900772+0000","last_clean":"2024-05-28T06:24:25.900772+0000","last_became_active":"2024-05-28T06:24:23.007331+0000","last_became_peered":"2024-05-28T06:24:23.007331+0000","last_unstale":"2024-05-28T06:24:25.900772+0000","last_undegraded":"2024-05-28T06:24:25.900772+0000","last_fullsized":"2024-05-28T06:24:25.900772+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:50:39.661313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":383,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-05-28T06:24:25.899651+0000","last_change":"2024-05-28T06:04:01.039129+0000","last_active":"2024-05-28T06:24:25.899651+0000","last_peered":"2024-05-28T06:24:25.899651+0000","last_clean":"2024-05-28T06:24:25.899651+0000","last_became_active":"2024-05-28T06:04:01.033906+0000","last_became_peered":"2024-05-28T06:04:01.033906+0000","last_unstale":"2024-05-28T06:24:25.899651+0000","last_undegraded":"2024-05-28T06:24:25.899651+0000","last_fullsized":"2024-05-28T06:24:25.899651+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:03:59.864237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:03:59.864237+0000","last_clean_scrub_stamp":"2024-05-28T06:03:59.864237+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:17:36.440245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575096,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":131860,"kb_used_data":2176,"kb_used_omap":1,"kb_used_meta":129662,"kb_avail":94239980,"statfs":{"total":96636764160,"available":96501739520,"internally_reserved":0,"allocated":2228224,"data_stored":1301310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":132774346},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575098,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":131268,"kb_used_data":1584,"kb_used_omap":1,"kb_used_meta":129662,"kb_avail":94240572,"statfs":{"total":96636764160,"available":96502345728,"internally_reserved":0,"allocated":1622016,"data_stored":710304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":132774342},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575096,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64636,"kb_used_data":1580,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94307204,"statfs":{"total":96636764160,"available":96570576896,"internally_reserved":0,"allocated":1617920,"data_stored":708804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575096,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":149312,"kb_used_data":1588,"kb_used_omap":1,"kb_used_meta":147710,"kb_avail":94222528,"statfs":{"total":96636764160,"available":96483868672,"internally_reserved":0,"allocated":1626112,"data_stored":715118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151255493},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575098,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":118784,"kb_used_data":1580,"kb_used_omap":1,"kb_used_meta":117182,"kb_avail":94253056,"statfs":{"total":96636764160,"available":96515129344,"internally_reserved":0,"allocated":1617920,"data_stored":708852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":119994823},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575097,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":149312,"kb_used_data":1580,"kb_used_omap":1,"kb_used_meta":147710,"kb_avail":94222528,"statfs":{"total":96636764160,"available":96483868672,"internally_reserved":0,"allocated":1617920,"data_stored":709308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151255493},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575097,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":149304,"kb_used_data":1572,"kb_used_omap":1,"kb_used_meta":147710,"kb_avail":94222536,"statfs":{"total":96636764160,"available":96483876864,"internally_reserved":0,"allocated":1609728,"data_stored":708692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151255493},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575096,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":145724,"kb_used_data":2152,"kb_used_omap":1,"kb_used_meta":143550,"kb_avail":94226116,"statfs":{"total":96636764160,"available":96487542784,"internally_reserved":0,"allocated":2203648,"data_stored":1299060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":146995653},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-28T06:24:28.425 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-28T06:24:28.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.659+0000 7f2b5e000640 1 Processor -- start 2024-05-28T06:24:28.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.660+0000 7f2b5e000640 1 -- start start 2024-05-28T06:24:28.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.661+0000 7f2b5e000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f2b5806f860 0x7f2b580b3150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:28.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.661+0000 7f2b5e000640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b5806ed10 con 0x7f2b580b3690 2024-05-28T06:24:28.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.661+0000 7f2b5e000640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b5815cb00 con 0x7f2b580d0bb0 2024-05-28T06:24:28.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.661+0000 7f2b5e000640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f2b580b3cc0 con 0x7f2b5806f860 2024-05-28T06:24:28.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.661+0000 7f2b4ea00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f2b580b3690 0x7f2b580d04a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:33854/0 (socket says 172.21.15.96:33854) 2024-05-28T06:24:28.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.661+0000 7f2b4ea00640 1 -- 172.21.15.96:0/3426231505 learned_addr learned my addr 172.21.15.96:0/3426231505 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:24:28.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.662+0000 7f2b57400640 1 --2- 172.21.15.96:0/3426231505 >> v2:172.21.15.123:3300/0 conn(0x7f2b5806f860 0x7f2b580b3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:28.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.662+0000 7f2b56a00640 1 -- 172.21.15.96:0/3426231505 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1083035148 0 0) 0x7f2b40001110 con 0x7f2b580b3690 2024-05-28T06:24:28.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.662+0000 7f2b56a00640 1 -- 172.21.15.96:0/3426231505 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2b3c003520 con 0x7f2b580b3690 2024-05-28T06:24:28.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.662+0000 7f2b56a00640 1 -- 172.21.15.96:0/3426231505 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2084228767 0 0) 0x7f2b3c003520 con 0x7f2b580b3690 2024-05-28T06:24:28.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.662+0000 7f2b56a00640 1 -- 172.21.15.96:0/3426231505 >> v2:172.21.15.123:3300/0 conn(0x7f2b5806f860 msgr2=0x7f2b580b3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:28.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.662+0000 7f2b56a00640 1 --2- 172.21.15.96:0/3426231505 >> v2:172.21.15.123:3300/0 conn(0x7f2b5806f860 0x7f2b580b3150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:28.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.662+0000 7f2b56a00640 1 -- 172.21.15.96:0/3426231505 >> v1:172.21.15.123:6789/0 conn(0x7f2b580d0bb0 legacy=0x7f2b580ad130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:28.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.662+0000 7f2b56a00640 1 -- 172.21.15.96:0/3426231505 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b580b3fa0 con 0x7f2b580b3690 2024-05-28T06:24:28.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.662+0000 7f2b56a00640 1 -- 172.21.15.96:0/3426231505 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f2b40001840 con 0x7f2b580b3690 2024-05-28T06:24:28.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.662+0000 7f2b56a00640 1 -- 172.21.15.96:0/3426231505 >> v1:172.21.15.96:6789/0 conn(0x7f2b580b3690 legacy=0x7f2b580d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:28.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.662+0000 7f2b56a00640 1 --2- 172.21.15.96:0/3426231505 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2b3c0039b0 0x7f2b3c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:28.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.662+0000 7f2b56a00640 1 --2- 172.21.15.96:0/3426231505 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2b3c004800 0x7f2b3c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:28.659 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.662+0000 7f2b56a00640 1 --2- 172.21.15.96:0/3426231505 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2b3c0051c0 0x7f2b3c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:28.659 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.662+0000 7f2b56a00640 1 -- 172.21.15.96:0/3426231505 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f2b580b3cc0 con 0x7f2b3c0051c0 2024-05-28T06:24:28.659 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.662+0000 7f2b56a00640 1 -- 172.21.15.96:0/3426231505 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f2b3c00c060 con 0x7f2b3c004800 2024-05-28T06:24:28.659 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.662+0000 7f2b56a00640 1 -- 172.21.15.96:0/3426231505 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f2b3c00c260 con 0x7f2b3c0039b0 2024-05-28T06:24:28.659 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.663+0000 7f2b57e00640 1 --2- 172.21.15.96:0/3426231505 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2b3c0051c0 0x7f2b3c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:28.659 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.663+0000 7f2b57400640 1 --2- 172.21.15.96:0/3426231505 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2b3c0039b0 0x7f2b3c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:28.659 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.663+0000 7f2b4ea00640 1 --2- 172.21.15.96:0/3426231505 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2b3c004800 0x7f2b3c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:28.659 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.663+0000 7f2b57e00640 1 -- 172.21.15.96:0/3426231505 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2b3c0039b0 msgr2=0x7f2b3c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:28.659 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.663+0000 7f2b57e00640 1 --2- 172.21.15.96:0/3426231505 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2b3c0039b0 0x7f2b3c003dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:28.659 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.663+0000 7f2b57e00640 1 -- 172.21.15.96:0/3426231505 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2b3c004800 msgr2=0x7f2b3c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:28.659 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.663+0000 7f2b57e00640 1 --2- 172.21.15.96:0/3426231505 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2b3c004800 0x7f2b3c004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:28.659 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.663+0000 7f2b57e00640 1 -- 172.21.15.96:0/3426231505 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b3c003520 con 0x7f2b3c0051c0 2024-05-28T06:24:28.661 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.664+0000 7f2b57e00640 1 -- 172.21.15.96:0/3426231505 reap_dead start 2024-05-28T06:24:28.661 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.664+0000 7f2b57e00640 1 --2- 172.21.15.96:0/3426231505 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2b3c0051c0 0x7f2b3c00b9c0 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 crypto rx=0x7f2b50001e50 tx=0x7f2b3c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70be4b94f7b0d3f6 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:28.661 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.664+0000 7f2b56a00640 1 -- 172.21.15.96:0/3426231505 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b50001a70 con 0x7f2b3c0051c0 2024-05-28T06:24:28.661 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.664+0000 7f2b56a00640 1 -- 172.21.15.96:0/3426231505 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b50003970 con 0x7f2b3c0051c0 2024-05-28T06:24:28.661 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.664+0000 7f2b56a00640 1 -- 172.21.15.96:0/3426231505 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b50003d60 con 0x7f2b3c0051c0 2024-05-28T06:24:28.661 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.665+0000 7f2b5e000640 1 -- 172.21.15.96:0/3426231505 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2b3c0051c0 msgr2=0x7f2b3c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:28.661 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.665+0000 7f2b5e000640 1 --2- 172.21.15.96:0/3426231505 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2b3c0051c0 0x7f2b3c00b9c0 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 crypto rx=0x7f2b50001e50 tx=0x7f2b3c005830 comp rx=0 tx=0).stop 2024-05-28T06:24:28.661 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.665+0000 7f2b5e000640 1 -- 172.21.15.96:0/3426231505 shutdown_connections 2024-05-28T06:24:28.661 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.665+0000 7f2b5e000640 1 --2- 172.21.15.96:0/3426231505 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2b3c0051c0 0x7f2b3c00b9c0 unknown :-1 s=CLOSED pgs=2027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:28.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.665+0000 7f2b5e000640 1 -- 172.21.15.96:0/3426231505 >> 172.21.15.96:0/3426231505 conn(0x7f2b5806bae0 msgr2=0x7f2b5806bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:28.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.665+0000 7f2b5e000640 1 -- 172.21.15.96:0/3426231505 shutdown_connections 2024-05-28T06:24:28.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.665+0000 7f2b5e000640 1 -- 172.21.15.96:0/3426231505 wait complete. 2024-05-28T06:24:28.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.667+0000 7f2b5e000640 1 Processor -- start 2024-05-28T06:24:28.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.667+0000 7f2b5e000640 1 -- start start 2024-05-28T06:24:28.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.668+0000 7f2b5e000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2b3c0051c0 0x7f2b5815ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:28.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.668+0000 7f2b5e000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2b580c3a50 0x7f2b581f4310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:28.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.668+0000 7f2b5e000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2b58152280 0x7f2b580c5f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:28.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.668+0000 7f2b5e000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f2b580b4840 con 0x7f2b3c0051c0 2024-05-28T06:24:28.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.668+0000 7f2b5e000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f2b580b4560 con 0x7f2b580c3a50 2024-05-28T06:24:28.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.668+0000 7f2b5e000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f2b580b3150 con 0x7f2b58152280 2024-05-28T06:24:28.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.668+0000 7f2b57400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2b3c0051c0 0x7f2b5815ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:28.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.668+0000 7f2b57400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2b3c0051c0 0x7f2b5815ecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:45536/0 (socket says 172.21.15.96:45536) 2024-05-28T06:24:28.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.668+0000 7f2b57400640 1 -- 172.21.15.96:0/1967564817 learned_addr learned my addr 172.21.15.96:0/1967564817 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:24:28.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.669+0000 7f2b57e00640 1 --2- 172.21.15.96:0/1967564817 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2b58152280 0x7f2b580c5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:28.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.669+0000 7f2b4ea00640 1 --2- 172.21.15.96:0/1967564817 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2b580c3a50 0x7f2b581f4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:28.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.669+0000 7f2b57400640 1 -- 172.21.15.96:0/1967564817 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2b58152280 msgr2=0x7f2b580c5f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:28.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.669+0000 7f2b57400640 1 --2- 172.21.15.96:0/1967564817 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2b58152280 0x7f2b580c5f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:28.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.669+0000 7f2b57400640 1 -- 172.21.15.96:0/1967564817 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2b580c3a50 msgr2=0x7f2b581f4310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:28.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.669+0000 7f2b57400640 1 --2- 172.21.15.96:0/1967564817 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2b580c3a50 0x7f2b581f4310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:28.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.669+0000 7f2b57400640 1 -- 172.21.15.96:0/1967564817 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b5815f2b0 con 0x7f2b3c0051c0 2024-05-28T06:24:28.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.669+0000 7f2b57400640 1 --2- 172.21.15.96:0/1967564817 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2b3c0051c0 0x7f2b5815ecd0 secure :-1 s=READY pgs=2028 cs=0 l=1 rev1=1 crypto rx=0x7f2b48010860 tx=0x7f2b48010d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:28.666 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.669+0000 7f2b4fe00640 1 -- 172.21.15.96:0/1967564817 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b4800f040 con 0x7f2b3c0051c0 2024-05-28T06:24:28.666 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.669+0000 7f2b4fe00640 1 -- 172.21.15.96:0/1967564817 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b4800eb00 con 0x7f2b3c0051c0 2024-05-28T06:24:28.666 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.669+0000 7f2b4fe00640 1 -- 172.21.15.96:0/1967564817 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b4801c400 con 0x7f2b3c0051c0 2024-05-28T06:24:28.666 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.669+0000 7f2b5e000640 1 -- 172.21.15.96:0/1967564817 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b580cdf60 con 0x7f2b3c0051c0 2024-05-28T06:24:28.666 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.670+0000 7f2b5e000640 1 -- 172.21.15.96:0/1967564817 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b58152700 con 0x7f2b3c0051c0 2024-05-28T06:24:28.666 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.671+0000 7f2b5e000640 1 -- 172.21.15.96:0/1967564817 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b580689e0 con 0x7f2b3c0051c0 2024-05-28T06:24:28.666 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.672+0000 7f2b4fe00640 1 -- 172.21.15.96:0/1967564817 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 42) ==== 93590+0+0 (secure 0 0 0) 0x7f2b4801c5a0 con 0x7f2b3c0051c0 2024-05-28T06:24:28.667 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.672+0000 7f2b4fe00640 1 --2- 172.21.15.96:0/1967564817 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2b3406e540 0x7f2b34070a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:28.667 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.673+0000 7f2b4fe00640 1 -- 172.21.15.96:0/1967564817 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 5765+0+0 (secure 0 0 0) 0x7f2b4808b050 con 0x7f2b3c0051c0 2024-05-28T06:24:28.667 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.673+0000 7f2b4ea00640 1 --2- 172.21.15.96:0/1967564817 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2b3406e540 0x7f2b34070a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:28.668 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.674+0000 7f2b4ea00640 1 --2- 172.21.15.96:0/1967564817 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2b3406e540 0x7f2b34070a00 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f2b400019b0 tx=0x7f2b40009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:28.671 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.677+0000 7f2b4fe00640 1 -- 172.21.15.96:0/1967564817 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2b48018030 con 0x7f2b3c0051c0 2024-05-28T06:24:28.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.875+0000 7f2b5e000640 1 -- 172.21.15.96:0/1967564817 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2b5815f590 con 0x7f2b3406e540 2024-05-28T06:24:28.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.895+0000 7f2b4fe00640 1 -- 172.21.15.96:0/1967564817 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252459 (secure 0 0 0) 0x7f2b5815f590 con 0x7f2b3406e540 2024-05-28T06:24:28.896 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:24:28.899 INFO:teuthology.orchestra.run.smithi096.stderr:dumped all 2024-05-28T06:24:28.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.901+0000 7f2b5e000640 1 -- 172.21.15.96:0/1967564817 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2b3406e540 msgr2=0x7f2b34070a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:28.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.901+0000 7f2b5e000640 1 --2- 172.21.15.96:0/1967564817 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2b3406e540 0x7f2b34070a00 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f2b400019b0 tx=0x7f2b40009040 comp rx=0 tx=0).stop 2024-05-28T06:24:28.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.901+0000 7f2b5e000640 1 -- 172.21.15.96:0/1967564817 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2b3c0051c0 msgr2=0x7f2b5815ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:28.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.901+0000 7f2b5e000640 1 --2- 172.21.15.96:0/1967564817 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2b3c0051c0 0x7f2b5815ecd0 secure :-1 s=READY pgs=2028 cs=0 l=1 rev1=1 crypto rx=0x7f2b48010860 tx=0x7f2b48010d20 comp rx=0 tx=0).stop 2024-05-28T06:24:28.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.901+0000 7f2b5e000640 1 -- 172.21.15.96:0/1967564817 shutdown_connections 2024-05-28T06:24:28.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.901+0000 7f2b5e000640 1 --2- 172.21.15.96:0/1967564817 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2b3406e540 0x7f2b34070a00 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:28.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.901+0000 7f2b5e000640 1 --2- 172.21.15.96:0/1967564817 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2b58152280 0x7f2b580c5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:28.904 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.901+0000 7f2b5e000640 1 --2- 172.21.15.96:0/1967564817 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2b580c3a50 0x7f2b581f4310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:28.904 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.901+0000 7f2b5e000640 1 --2- 172.21.15.96:0/1967564817 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2b3c0051c0 0x7f2b5815ecd0 unknown :-1 s=CLOSED pgs=2028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:28.904 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.901+0000 7f2b5e000640 1 -- 172.21.15.96:0/1967564817 >> 172.21.15.96:0/1967564817 conn(0x7f2b5806bae0 msgr2=0x7f2b58071be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:28.904 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.901+0000 7f2b5e000640 1 -- 172.21.15.96:0/1967564817 shutdown_connections 2024-05-28T06:24:28.904 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:28.901+0000 7f2b5e000640 1 -- 172.21.15.96:0/1967564817 wait complete. 2024-05-28T06:24:28.919 INFO:teuthology.orchestra.run.smithi096.stdout:{"pg_ready":true,"pg_map":{"version":737,"stamp":"2024-05-28T06:24:28.018212+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":1040200,"kb_used_data":13812,"kb_used_omap":12,"kb_used_meta":1026227,"kb_avail":753934520,"statfs":{"total":773094113280,"available":772028948480,"internally_reserved":0,"allocated":14143488,"data_stored":6861448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":1050857008},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001869"},"pg_stats":[{"pgid":"19.3f","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.008757+0000","last_change":"2024-05-28T06:24:24.007430+0000","last_active":"2024-05-28T06:24:24.008757+0000","last_peered":"2024-05-28T06:24:24.008757+0000","last_clean":"2024-05-28T06:24:24.008757+0000","last_became_active":"2024-05-28T06:24:24.006988+0000","last_became_peered":"2024-05-28T06:24:24.006988+0000","last_unstale":"2024-05-28T06:24:24.008757+0000","last_undegraded":"2024-05-28T06:24:24.008757+0000","last_fullsized":"2024-05-28T06:24:24.008757+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T18:04:39.239146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3e","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:23.990554+0000","last_change":"2024-05-28T06:24:23.986742+0000","last_active":"2024-05-28T06:24:23.990554+0000","last_peered":"2024-05-28T06:24:23.990554+0000","last_clean":"2024-05-28T06:24:23.990554+0000","last_became_active":"2024-05-28T06:24:23.986471+0000","last_became_peered":"2024-05-28T06:24:23.986471+0000","last_unstale":"2024-05-28T06:24:23.990554+0000","last_undegraded":"2024-05-28T06:24:23.990554+0000","last_fullsized":"2024-05-28T06:24:23.990554+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:15:02.161297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3d","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.016001+0000","last_change":"2024-05-28T06:24:24.007698+0000","last_active":"2024-05-28T06:24:24.016001+0000","last_peered":"2024-05-28T06:24:24.016001+0000","last_clean":"2024-05-28T06:24:24.016001+0000","last_became_active":"2024-05-28T06:24:24.007416+0000","last_became_peered":"2024-05-28T06:24:24.007416+0000","last_unstale":"2024-05-28T06:24:24.016001+0000","last_undegraded":"2024-05-28T06:24:24.016001+0000","last_fullsized":"2024-05-28T06:24:24.016001+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:44:39.421718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.3c","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.012365+0000","last_change":"2024-05-28T06:24:24.011512+0000","last_active":"2024-05-28T06:24:24.012365+0000","last_peered":"2024-05-28T06:24:24.012365+0000","last_clean":"2024-05-28T06:24:24.012365+0000","last_became_active":"2024-05-28T06:24:24.011234+0000","last_became_peered":"2024-05-28T06:24:24.011234+0000","last_unstale":"2024-05-28T06:24:24.012365+0000","last_undegraded":"2024-05-28T06:24:24.012365+0000","last_fullsized":"2024-05-28T06:24:24.012365+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:08:20.720556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3b","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-05-28T06:24:25.900674+0000","last_change":"2024-05-28T06:24:24.019397+0000","last_active":"2024-05-28T06:24:25.900674+0000","last_peered":"2024-05-28T06:24:25.900674+0000","last_clean":"2024-05-28T06:24:25.900674+0000","last_became_active":"2024-05-28T06:24:24.018334+0000","last_became_peered":"2024-05-28T06:24:24.018334+0000","last_unstale":"2024-05-28T06:24:25.900674+0000","last_undegraded":"2024-05-28T06:24:25.900674+0000","last_fullsized":"2024-05-28T06:24:25.900674+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:22:34.611665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.3a","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-05-28T06:24:25.012292+0000","last_change":"2024-05-28T06:24:24.003959+0000","last_active":"2024-05-28T06:24:25.012292+0000","last_peered":"2024-05-28T06:24:25.012292+0000","last_clean":"2024-05-28T06:24:25.012292+0000","last_became_active":"2024-05-28T06:24:23.997362+0000","last_became_peered":"2024-05-28T06:24:23.997362+0000","last_unstale":"2024-05-28T06:24:25.012292+0000","last_undegraded":"2024-05-28T06:24:25.012292+0000","last_fullsized":"2024-05-28T06:24:25.012292+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:07:49.435286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.39","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-05-28T06:24:25.011686+0000","last_change":"2024-05-28T06:24:24.006117+0000","last_active":"2024-05-28T06:24:25.011686+0000","last_peered":"2024-05-28T06:24:25.011686+0000","last_clean":"2024-05-28T06:24:25.011686+0000","last_became_active":"2024-05-28T06:24:24.005839+0000","last_became_peered":"2024-05-28T06:24:24.005839+0000","last_unstale":"2024-05-28T06:24:25.011686+0000","last_undegraded":"2024-05-28T06:24:25.011686+0000","last_fullsized":"2024-05-28T06:24:25.011686+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:47:32.965627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.38","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.017553+0000","last_change":"2024-05-28T06:24:24.008772+0000","last_active":"2024-05-28T06:24:24.017553+0000","last_peered":"2024-05-28T06:24:24.017553+0000","last_clean":"2024-05-28T06:24:24.017553+0000","last_became_active":"2024-05-28T06:24:24.008334+0000","last_became_peered":"2024-05-28T06:24:24.008334+0000","last_unstale":"2024-05-28T06:24:24.017553+0000","last_undegraded":"2024-05-28T06:24:24.017553+0000","last_fullsized":"2024-05-28T06:24:24.017553+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:41:33.057222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.37","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.021117+0000","last_change":"2024-05-28T06:24:24.018229+0000","last_active":"2024-05-28T06:24:24.021117+0000","last_peered":"2024-05-28T06:24:24.021117+0000","last_clean":"2024-05-28T06:24:24.021117+0000","last_became_active":"2024-05-28T06:24:24.017929+0000","last_became_peered":"2024-05-28T06:24:24.017929+0000","last_unstale":"2024-05-28T06:24:24.021117+0000","last_undegraded":"2024-05-28T06:24:24.021117+0000","last_fullsized":"2024-05-28T06:24:24.021117+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:05:18.151961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.36","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:23.997228+0000","last_change":"2024-05-28T06:24:23.995403+0000","last_active":"2024-05-28T06:24:23.997228+0000","last_peered":"2024-05-28T06:24:23.997228+0000","last_clean":"2024-05-28T06:24:23.997228+0000","last_became_active":"2024-05-28T06:24:23.994945+0000","last_became_peered":"2024-05-28T06:24:23.994945+0000","last_unstale":"2024-05-28T06:24:23.997228+0000","last_undegraded":"2024-05-28T06:24:23.997228+0000","last_fullsized":"2024-05-28T06:24:23.997228+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:06:42.676403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.35","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:23.997024+0000","last_change":"2024-05-28T06:24:23.992865+0000","last_active":"2024-05-28T06:24:23.997024+0000","last_peered":"2024-05-28T06:24:23.997024+0000","last_clean":"2024-05-28T06:24:23.997024+0000","last_became_active":"2024-05-28T06:24:23.991405+0000","last_became_peered":"2024-05-28T06:24:23.991405+0000","last_unstale":"2024-05-28T06:24:23.997024+0000","last_undegraded":"2024-05-28T06:24:23.997024+0000","last_fullsized":"2024-05-28T06:24:23.997024+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:32:12.353432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.34","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.019721+0000","last_change":"2024-05-28T06:24:24.018385+0000","last_active":"2024-05-28T06:24:24.019721+0000","last_peered":"2024-05-28T06:24:24.019721+0000","last_clean":"2024-05-28T06:24:24.019721+0000","last_became_active":"2024-05-28T06:24:24.017931+0000","last_became_peered":"2024-05-28T06:24:24.017931+0000","last_unstale":"2024-05-28T06:24:24.019721+0000","last_undegraded":"2024-05-28T06:24:24.019721+0000","last_fullsized":"2024-05-28T06:24:24.019721+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:59:26.272700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.33","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.008608+0000","last_change":"2024-05-28T06:24:24.007399+0000","last_active":"2024-05-28T06:24:24.008608+0000","last_peered":"2024-05-28T06:24:24.008608+0000","last_clean":"2024-05-28T06:24:24.008608+0000","last_became_active":"2024-05-28T06:24:24.007121+0000","last_became_peered":"2024-05-28T06:24:24.007121+0000","last_unstale":"2024-05-28T06:24:24.008608+0000","last_undegraded":"2024-05-28T06:24:24.008608+0000","last_fullsized":"2024-05-28T06:24:24.008608+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:50:21.530645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.32","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:23.997543+0000","last_change":"2024-05-28T06:24:23.996362+0000","last_active":"2024-05-28T06:24:23.997543+0000","last_peered":"2024-05-28T06:24:23.997543+0000","last_clean":"2024-05-28T06:24:23.997543+0000","last_became_active":"2024-05-28T06:24:23.994791+0000","last_became_peered":"2024-05-28T06:24:23.994791+0000","last_unstale":"2024-05-28T06:24:23.997543+0000","last_undegraded":"2024-05-28T06:24:23.997543+0000","last_fullsized":"2024-05-28T06:24:23.997543+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:29:55.277690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.31","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.021218+0000","last_change":"2024-05-28T06:24:24.020598+0000","last_active":"2024-05-28T06:24:24.021218+0000","last_peered":"2024-05-28T06:24:24.021218+0000","last_clean":"2024-05-28T06:24:24.021218+0000","last_became_active":"2024-05-28T06:24:24.020301+0000","last_became_peered":"2024-05-28T06:24:24.020301+0000","last_unstale":"2024-05-28T06:24:24.021218+0000","last_undegraded":"2024-05-28T06:24:24.021218+0000","last_fullsized":"2024-05-28T06:24:24.021218+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:28:25.639532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.30","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:23.999124+0000","last_change":"2024-05-28T06:24:23.995628+0000","last_active":"2024-05-28T06:24:23.999124+0000","last_peered":"2024-05-28T06:24:23.999124+0000","last_clean":"2024-05-28T06:24:23.999124+0000","last_became_active":"2024-05-28T06:24:23.995367+0000","last_became_peered":"2024-05-28T06:24:23.995367+0000","last_unstale":"2024-05-28T06:24:23.999124+0000","last_undegraded":"2024-05-28T06:24:23.999124+0000","last_fullsized":"2024-05-28T06:24:23.999124+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:31:43.576159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2f","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-05-28T06:24:25.013294+0000","last_change":"2024-05-28T06:24:23.987587+0000","last_active":"2024-05-28T06:24:25.013294+0000","last_peered":"2024-05-28T06:24:25.013294+0000","last_clean":"2024-05-28T06:24:25.013294+0000","last_became_active":"2024-05-28T06:24:23.986777+0000","last_became_peered":"2024-05-28T06:24:23.986777+0000","last_unstale":"2024-05-28T06:24:25.013294+0000","last_undegraded":"2024-05-28T06:24:25.013294+0000","last_fullsized":"2024-05-28T06:24:25.013294+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:06:00.030718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.2e","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:23.999601+0000","last_change":"2024-05-28T06:24:23.998500+0000","last_active":"2024-05-28T06:24:23.999601+0000","last_peered":"2024-05-28T06:24:23.999601+0000","last_clean":"2024-05-28T06:24:23.999601+0000","last_became_active":"2024-05-28T06:24:23.998105+0000","last_became_peered":"2024-05-28T06:24:23.998105+0000","last_unstale":"2024-05-28T06:24:23.999601+0000","last_undegraded":"2024-05-28T06:24:23.999601+0000","last_fullsized":"2024-05-28T06:24:23.999601+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:05:14.127643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2d","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.000368+0000","last_change":"2024-05-28T06:24:23.997899+0000","last_active":"2024-05-28T06:24:24.000368+0000","last_peered":"2024-05-28T06:24:24.000368+0000","last_clean":"2024-05-28T06:24:24.000368+0000","last_became_active":"2024-05-28T06:24:23.997362+0000","last_became_peered":"2024-05-28T06:24:23.997362+0000","last_unstale":"2024-05-28T06:24:24.000368+0000","last_undegraded":"2024-05-28T06:24:24.000368+0000","last_fullsized":"2024-05-28T06:24:24.000368+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:54:07.720252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2c","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:23.998894+0000","last_change":"2024-05-28T06:24:23.996440+0000","last_active":"2024-05-28T06:24:23.998894+0000","last_peered":"2024-05-28T06:24:23.998894+0000","last_clean":"2024-05-28T06:24:23.998894+0000","last_became_active":"2024-05-28T06:24:23.996067+0000","last_became_peered":"2024-05-28T06:24:23.996067+0000","last_unstale":"2024-05-28T06:24:23.998894+0000","last_undegraded":"2024-05-28T06:24:23.998894+0000","last_fullsized":"2024-05-28T06:24:23.998894+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:36:13.769185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2b","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-05-28T06:24:25.900477+0000","last_change":"2024-05-28T06:24:24.018537+0000","last_active":"2024-05-28T06:24:25.900477+0000","last_peered":"2024-05-28T06:24:25.900477+0000","last_clean":"2024-05-28T06:24:25.900477+0000","last_became_active":"2024-05-28T06:24:24.018253+0000","last_became_peered":"2024-05-28T06:24:24.018253+0000","last_unstale":"2024-05-28T06:24:25.900477+0000","last_undegraded":"2024-05-28T06:24:25.900477+0000","last_fullsized":"2024-05-28T06:24:25.900477+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:13:05.023858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2a","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:23.999700+0000","last_change":"2024-05-28T06:24:23.994663+0000","last_active":"2024-05-28T06:24:23.999700+0000","last_peered":"2024-05-28T06:24:23.999700+0000","last_clean":"2024-05-28T06:24:23.999700+0000","last_became_active":"2024-05-28T06:24:23.994410+0000","last_became_peered":"2024-05-28T06:24:23.994410+0000","last_unstale":"2024-05-28T06:24:23.999700+0000","last_undegraded":"2024-05-28T06:24:23.999700+0000","last_fullsized":"2024-05-28T06:24:23.999700+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:03:29.345159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.29","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.018227+0000","last_change":"2024-05-28T06:24:24.015438+0000","last_active":"2024-05-28T06:24:24.018227+0000","last_peered":"2024-05-28T06:24:24.018227+0000","last_clean":"2024-05-28T06:24:24.018227+0000","last_became_active":"2024-05-28T06:24:24.015124+0000","last_became_peered":"2024-05-28T06:24:24.015124+0000","last_unstale":"2024-05-28T06:24:24.018227+0000","last_undegraded":"2024-05-28T06:24:24.018227+0000","last_fullsized":"2024-05-28T06:24:24.018227+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:23:32.514259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.28","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.017967+0000","last_change":"2024-05-28T06:24:24.017147+0000","last_active":"2024-05-28T06:24:24.017967+0000","last_peered":"2024-05-28T06:24:24.017967+0000","last_clean":"2024-05-28T06:24:24.017967+0000","last_became_active":"2024-05-28T06:24:24.016692+0000","last_became_peered":"2024-05-28T06:24:24.016692+0000","last_unstale":"2024-05-28T06:24:24.017967+0000","last_undegraded":"2024-05-28T06:24:24.017967+0000","last_fullsized":"2024-05-28T06:24:24.017967+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:25:56.671088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.27","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.008439+0000","last_change":"2024-05-28T06:24:24.007398+0000","last_active":"2024-05-28T06:24:24.008439+0000","last_peered":"2024-05-28T06:24:24.008439+0000","last_clean":"2024-05-28T06:24:24.008439+0000","last_became_active":"2024-05-28T06:24:24.006983+0000","last_became_peered":"2024-05-28T06:24:24.006983+0000","last_unstale":"2024-05-28T06:24:24.008439+0000","last_undegraded":"2024-05-28T06:24:24.008439+0000","last_fullsized":"2024-05-28T06:24:24.008439+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:27:41.877671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.26","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.000526+0000","last_change":"2024-05-28T06:24:23.999225+0000","last_active":"2024-05-28T06:24:24.000526+0000","last_peered":"2024-05-28T06:24:24.000526+0000","last_clean":"2024-05-28T06:24:24.000526+0000","last_became_active":"2024-05-28T06:24:23.996859+0000","last_became_peered":"2024-05-28T06:24:23.996859+0000","last_unstale":"2024-05-28T06:24:24.000526+0000","last_undegraded":"2024-05-28T06:24:24.000526+0000","last_fullsized":"2024-05-28T06:24:24.000526+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:49:43.070723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.25","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.008804+0000","last_change":"2024-05-28T06:24:24.007463+0000","last_active":"2024-05-28T06:24:24.008804+0000","last_peered":"2024-05-28T06:24:24.008804+0000","last_clean":"2024-05-28T06:24:24.008804+0000","last_became_active":"2024-05-28T06:24:24.007194+0000","last_became_peered":"2024-05-28T06:24:24.007194+0000","last_unstale":"2024-05-28T06:24:24.008804+0000","last_undegraded":"2024-05-28T06:24:24.008804+0000","last_fullsized":"2024-05-28T06:24:24.008804+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:10:46.986223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.24","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.019702+0000","last_change":"2024-05-28T06:24:24.016291+0000","last_active":"2024-05-28T06:24:24.019702+0000","last_peered":"2024-05-28T06:24:24.019702+0000","last_clean":"2024-05-28T06:24:24.019702+0000","last_became_active":"2024-05-28T06:24:24.015990+0000","last_became_peered":"2024-05-28T06:24:24.015990+0000","last_unstale":"2024-05-28T06:24:24.019702+0000","last_undegraded":"2024-05-28T06:24:24.019702+0000","last_fullsized":"2024-05-28T06:24:24.019702+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:23:18.676845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.23","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.020933+0000","last_change":"2024-05-28T06:24:24.019908+0000","last_active":"2024-05-28T06:24:24.020933+0000","last_peered":"2024-05-28T06:24:24.020933+0000","last_clean":"2024-05-28T06:24:24.020933+0000","last_became_active":"2024-05-28T06:24:24.019497+0000","last_became_peered":"2024-05-28T06:24:24.019497+0000","last_unstale":"2024-05-28T06:24:24.020933+0000","last_undegraded":"2024-05-28T06:24:24.020933+0000","last_fullsized":"2024-05-28T06:24:24.020933+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:40:41.852779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.22","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.020666+0000","last_change":"2024-05-28T06:24:24.019852+0000","last_active":"2024-05-28T06:24:24.020666+0000","last_peered":"2024-05-28T06:24:24.020666+0000","last_clean":"2024-05-28T06:24:24.020666+0000","last_became_active":"2024-05-28T06:24:24.019462+0000","last_became_peered":"2024-05-28T06:24:24.019462+0000","last_unstale":"2024-05-28T06:24:24.020666+0000","last_undegraded":"2024-05-28T06:24:24.020666+0000","last_fullsized":"2024-05-28T06:24:24.020666+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:15:21.891198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.21","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-05-28T06:24:25.900369+0000","last_change":"2024-05-28T06:24:24.011632+0000","last_active":"2024-05-28T06:24:25.900369+0000","last_peered":"2024-05-28T06:24:25.900369+0000","last_clean":"2024-05-28T06:24:25.900369+0000","last_became_active":"2024-05-28T06:24:24.011246+0000","last_became_peered":"2024-05-28T06:24:24.011246+0000","last_unstale":"2024-05-28T06:24:25.900369+0000","last_undegraded":"2024-05-28T06:24:25.900369+0000","last_fullsized":"2024-05-28T06:24:25.900369+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:05:09.592919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.20","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-05-28T06:24:25.011477+0000","last_change":"2024-05-28T06:24:23.991803+0000","last_active":"2024-05-28T06:24:25.011477+0000","last_peered":"2024-05-28T06:24:25.011477+0000","last_clean":"2024-05-28T06:24:25.011477+0000","last_became_active":"2024-05-28T06:24:23.991245+0000","last_became_peered":"2024-05-28T06:24:23.991245+0000","last_unstale":"2024-05-28T06:24:25.011477+0000","last_undegraded":"2024-05-28T06:24:25.011477+0000","last_fullsized":"2024-05-28T06:24:25.011477+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:55:45.102561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.018662+0000","last_change":"2024-05-28T06:24:24.018032+0000","last_active":"2024-05-28T06:24:24.018662+0000","last_peered":"2024-05-28T06:24:24.018662+0000","last_clean":"2024-05-28T06:24:24.018662+0000","last_became_active":"2024-05-28T06:24:24.017727+0000","last_became_peered":"2024-05-28T06:24:24.017727+0000","last_unstale":"2024-05-28T06:24:24.018662+0000","last_undegraded":"2024-05-28T06:24:24.018662+0000","last_fullsized":"2024-05-28T06:24:24.018662+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:37:59.260972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1f","version":"129'4","reported_seq":18,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-05-28T06:24:25.901165+0000","last_change":"2024-05-28T06:24:23.006418+0000","last_active":"2024-05-28T06:24:25.901165+0000","last_peered":"2024-05-28T06:24:25.901165+0000","last_clean":"2024-05-28T06:24:25.901165+0000","last_became_active":"2024-05-28T06:24:23.006043+0000","last_became_peered":"2024-05-28T06:24:23.006043+0000","last_unstale":"2024-05-28T06:24:25.901165+0000","last_undegraded":"2024-05-28T06:24:25.901165+0000","last_fullsized":"2024-05-28T06:24:25.901165+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:31:42.560242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.016226+0000","last_change":"2024-05-28T06:24:24.008420+0000","last_active":"2024-05-28T06:24:24.016226+0000","last_peered":"2024-05-28T06:24:24.016226+0000","last_clean":"2024-05-28T06:24:24.016226+0000","last_became_active":"2024-05-28T06:24:24.008078+0000","last_became_peered":"2024-05-28T06:24:24.008078+0000","last_unstale":"2024-05-28T06:24:24.016226+0000","last_undegraded":"2024-05-28T06:24:24.016226+0000","last_fullsized":"2024-05-28T06:24:24.016226+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:33:49.362474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1e","version":"129'1","reported_seq":13,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.017323+0000","last_change":"2024-05-28T06:24:22.995399+0000","last_active":"2024-05-28T06:24:24.017323+0000","last_peered":"2024-05-28T06:24:24.017323+0000","last_clean":"2024-05-28T06:24:24.017323+0000","last_became_active":"2024-05-28T06:24:22.994896+0000","last_became_peered":"2024-05-28T06:24:22.994896+0000","last_unstale":"2024-05-28T06:24:24.017323+0000","last_undegraded":"2024-05-28T06:24:24.017323+0000","last_fullsized":"2024-05-28T06:24:24.017323+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:06:41.660486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-05-28T06:24:25.901145+0000","last_change":"2024-05-28T06:24:24.006543+0000","last_active":"2024-05-28T06:24:25.901145+0000","last_peered":"2024-05-28T06:24:25.901145+0000","last_clean":"2024-05-28T06:24:25.901145+0000","last_became_active":"2024-05-28T06:24:24.006258+0000","last_became_peered":"2024-05-28T06:24:24.006258+0000","last_unstale":"2024-05-28T06:24:25.901145+0000","last_undegraded":"2024-05-28T06:24:25.901145+0000","last_fullsized":"2024-05-28T06:24:25.901145+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:34:58.279352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1d","version":"129'2","reported_seq":14,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.000303+0000","last_change":"2024-05-28T06:24:22.992163+0000","last_active":"2024-05-28T06:24:24.000303+0000","last_peered":"2024-05-28T06:24:24.000303+0000","last_clean":"2024-05-28T06:24:24.000303+0000","last_became_active":"2024-05-28T06:24:22.991407+0000","last_became_peered":"2024-05-28T06:24:22.991407+0000","last_unstale":"2024-05-28T06:24:24.000303+0000","last_undegraded":"2024-05-28T06:24:24.000303+0000","last_fullsized":"2024-05-28T06:24:24.000303+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:34:43.911764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-05-28T06:24:25.900703+0000","last_change":"2024-05-28T06:24:24.013028+0000","last_active":"2024-05-28T06:24:25.900703+0000","last_peered":"2024-05-28T06:24:25.900703+0000","last_clean":"2024-05-28T06:24:25.900703+0000","last_became_active":"2024-05-28T06:24:24.010356+0000","last_became_peered":"2024-05-28T06:24:24.010356+0000","last_unstale":"2024-05-28T06:24:25.900703+0000","last_undegraded":"2024-05-28T06:24:25.900703+0000","last_fullsized":"2024-05-28T06:24:25.900703+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:46:38.879485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1c","version":"129'2","reported_seq":14,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:23.997825+0000","last_change":"2024-05-28T06:24:22.989493+0000","last_active":"2024-05-28T06:24:23.997825+0000","last_peered":"2024-05-28T06:24:23.997825+0000","last_clean":"2024-05-28T06:24:23.997825+0000","last_became_active":"2024-05-28T06:24:22.989103+0000","last_became_peered":"2024-05-28T06:24:22.989103+0000","last_unstale":"2024-05-28T06:24:23.997825+0000","last_undegraded":"2024-05-28T06:24:23.997825+0000","last_fullsized":"2024-05-28T06:24:23.997825+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:28:25.855171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.009066+0000","last_change":"2024-05-28T06:24:24.008297+0000","last_active":"2024-05-28T06:24:24.009066+0000","last_peered":"2024-05-28T06:24:24.009066+0000","last_clean":"2024-05-28T06:24:24.009066+0000","last_became_active":"2024-05-28T06:24:24.007987+0000","last_became_peered":"2024-05-28T06:24:24.007987+0000","last_unstale":"2024-05-28T06:24:24.009066+0000","last_undegraded":"2024-05-28T06:24:24.009066+0000","last_fullsized":"2024-05-28T06:24:24.009066+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:55:45.102561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1b","version":"129'1","reported_seq":13,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:23.998832+0000","last_change":"2024-05-28T06:24:22.987047+0000","last_active":"2024-05-28T06:24:23.998832+0000","last_peered":"2024-05-28T06:24:23.998832+0000","last_clean":"2024-05-28T06:24:23.998832+0000","last_became_active":"2024-05-28T06:24:22.985617+0000","last_became_peered":"2024-05-28T06:24:22.985617+0000","last_unstale":"2024-05-28T06:24:23.998832+0000","last_undegraded":"2024-05-28T06:24:23.998832+0000","last_fullsized":"2024-05-28T06:24:23.998832+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:59:04.673636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-05-28T06:24:25.012080+0000","last_change":"2024-05-28T06:24:23.987152+0000","last_active":"2024-05-28T06:24:25.012080+0000","last_peered":"2024-05-28T06:24:25.012080+0000","last_clean":"2024-05-28T06:24:25.012080+0000","last_became_active":"2024-05-28T06:24:23.985828+0000","last_became_peered":"2024-05-28T06:24:23.985828+0000","last_unstale":"2024-05-28T06:24:25.012080+0000","last_undegraded":"2024-05-28T06:24:25.012080+0000","last_fullsized":"2024-05-28T06:24:25.012080+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:48:49.925554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1a","version":"129'1","reported_seq":13,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:23.996435+0000","last_change":"2024-05-28T06:24:22.990085+0000","last_active":"2024-05-28T06:24:23.996435+0000","last_peered":"2024-05-28T06:24:23.996435+0000","last_clean":"2024-05-28T06:24:23.996435+0000","last_became_active":"2024-05-28T06:24:22.989801+0000","last_became_peered":"2024-05-28T06:24:22.989801+0000","last_unstale":"2024-05-28T06:24:23.996435+0000","last_undegraded":"2024-05-28T06:24:23.996435+0000","last_fullsized":"2024-05-28T06:24:23.996435+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:53:06.353639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.011512+0000","last_change":"2024-05-28T06:24:24.003806+0000","last_active":"2024-05-28T06:24:24.011512+0000","last_peered":"2024-05-28T06:24:24.011512+0000","last_clean":"2024-05-28T06:24:24.011512+0000","last_became_active":"2024-05-28T06:24:24.003446+0000","last_became_peered":"2024-05-28T06:24:24.003446+0000","last_unstale":"2024-05-28T06:24:24.011512+0000","last_undegraded":"2024-05-28T06:24:24.011512+0000","last_fullsized":"2024-05-28T06:24:24.011512+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:12:39.142307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-05-28T06:24:25.011943+0000","last_change":"2024-05-28T06:24:23.007861+0000","last_active":"2024-05-28T06:24:25.011943+0000","last_peered":"2024-05-28T06:24:25.011943+0000","last_clean":"2024-05-28T06:24:25.011943+0000","last_became_active":"2024-05-28T06:24:23.007494+0000","last_became_peered":"2024-05-28T06:24:23.007494+0000","last_unstale":"2024-05-28T06:24:25.011943+0000","last_undegraded":"2024-05-28T06:24:25.011943+0000","last_fullsized":"2024-05-28T06:24:25.011943+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:06:19.878458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:23.994727+0000","last_change":"2024-05-28T06:24:23.993309+0000","last_active":"2024-05-28T06:24:23.994727+0000","last_peered":"2024-05-28T06:24:23.994727+0000","last_clean":"2024-05-28T06:24:23.994727+0000","last_became_active":"2024-05-28T06:24:23.987466+0000","last_became_peered":"2024-05-28T06:24:23.987466+0000","last_unstale":"2024-05-28T06:24:23.994727+0000","last_undegraded":"2024-05-28T06:24:23.994727+0000","last_fullsized":"2024-05-28T06:24:23.994727+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:59:20.653065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:23.975815+0000","last_change":"2024-05-28T06:24:23.000407+0000","last_active":"2024-05-28T06:24:23.975815+0000","last_peered":"2024-05-28T06:24:23.975815+0000","last_clean":"2024-05-28T06:24:23.975815+0000","last_became_active":"2024-05-28T06:24:23.000037+0000","last_became_peered":"2024-05-28T06:24:23.000037+0000","last_unstale":"2024-05-28T06:24:23.975815+0000","last_undegraded":"2024-05-28T06:24:23.975815+0000","last_fullsized":"2024-05-28T06:24:23.975815+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:59:08.824871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:23.996246+0000","last_change":"2024-05-28T06:24:23.995548+0000","last_active":"2024-05-28T06:24:23.996246+0000","last_peered":"2024-05-28T06:24:23.996246+0000","last_clean":"2024-05-28T06:24:23.996246+0000","last_became_active":"2024-05-28T06:24:23.995204+0000","last_became_peered":"2024-05-28T06:24:23.995204+0000","last_unstale":"2024-05-28T06:24:23.996246+0000","last_undegraded":"2024-05-28T06:24:23.996246+0000","last_fullsized":"2024-05-28T06:24:23.996246+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:00:42.665433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.005289+0000","last_change":"2024-05-28T06:24:23.001114+0000","last_active":"2024-05-28T06:24:24.005289+0000","last_peered":"2024-05-28T06:24:24.005289+0000","last_clean":"2024-05-28T06:24:24.005289+0000","last_became_active":"2024-05-28T06:24:22.999720+0000","last_became_peered":"2024-05-28T06:24:22.999720+0000","last_unstale":"2024-05-28T06:24:24.005289+0000","last_undegraded":"2024-05-28T06:24:24.005289+0000","last_fullsized":"2024-05-28T06:24:24.005289+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:29:54.261773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.021370+0000","last_change":"2024-05-28T06:24:24.016902+0000","last_active":"2024-05-28T06:24:24.021370+0000","last_peered":"2024-05-28T06:24:24.021370+0000","last_clean":"2024-05-28T06:24:24.021370+0000","last_became_active":"2024-05-28T06:24:24.016597+0000","last_became_peered":"2024-05-28T06:24:24.016597+0000","last_unstale":"2024-05-28T06:24:24.021370+0000","last_undegraded":"2024-05-28T06:24:24.021370+0000","last_fullsized":"2024-05-28T06:24:24.021370+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:52:44.413787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:23.985715+0000","last_change":"2024-05-28T06:24:23.001188+0000","last_active":"2024-05-28T06:24:23.985715+0000","last_peered":"2024-05-28T06:24:23.985715+0000","last_clean":"2024-05-28T06:24:23.985715+0000","last_became_active":"2024-05-28T06:24:23.000878+0000","last_became_peered":"2024-05-28T06:24:23.000878+0000","last_unstale":"2024-05-28T06:24:23.985715+0000","last_undegraded":"2024-05-28T06:24:23.985715+0000","last_fullsized":"2024-05-28T06:24:23.985715+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:18:06.253798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-05-28T06:24:25.901305+0000","last_change":"2024-05-28T06:24:24.011311+0000","last_active":"2024-05-28T06:24:25.901305+0000","last_peered":"2024-05-28T06:24:25.901305+0000","last_clean":"2024-05-28T06:24:25.901305+0000","last_became_active":"2024-05-28T06:24:24.010110+0000","last_became_peered":"2024-05-28T06:24:24.010110+0000","last_unstale":"2024-05-28T06:24:25.901305+0000","last_undegraded":"2024-05-28T06:24:25.901305+0000","last_fullsized":"2024-05-28T06:24:25.901305+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:46:31.105891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.15","version":"129'3","reported_seq":15,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.017657+0000","last_change":"2024-05-28T06:24:22.999088+0000","last_active":"2024-05-28T06:24:24.017657+0000","last_peered":"2024-05-28T06:24:24.017657+0000","last_clean":"2024-05-28T06:24:24.017657+0000","last_became_active":"2024-05-28T06:24:22.998379+0000","last_became_peered":"2024-05-28T06:24:22.998379+0000","last_unstale":"2024-05-28T06:24:24.017657+0000","last_undegraded":"2024-05-28T06:24:24.017657+0000","last_fullsized":"2024-05-28T06:24:24.017657+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:59:19.637147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-05-28T06:24:25.901246+0000","last_change":"2024-05-28T06:24:24.020278+0000","last_active":"2024-05-28T06:24:25.901246+0000","last_peered":"2024-05-28T06:24:25.901246+0000","last_clean":"2024-05-28T06:24:25.901246+0000","last_became_active":"2024-05-28T06:24:24.019973+0000","last_became_peered":"2024-05-28T06:24:24.019973+0000","last_unstale":"2024-05-28T06:24:25.901246+0000","last_undegraded":"2024-05-28T06:24:25.901246+0000","last_fullsized":"2024-05-28T06:24:25.901246+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:46:47.603731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.14","version":"129'6","reported_seq":18,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.023651+0000","last_change":"2024-05-28T06:24:23.005145+0000","last_active":"2024-05-28T06:24:24.023651+0000","last_peered":"2024-05-28T06:24:24.023651+0000","last_clean":"2024-05-28T06:24:24.023651+0000","last_became_active":"2024-05-28T06:24:23.004831+0000","last_became_peered":"2024-05-28T06:24:23.004831+0000","last_unstale":"2024-05-28T06:24:24.023651+0000","last_undegraded":"2024-05-28T06:24:24.023651+0000","last_fullsized":"2024-05-28T06:24:24.023651+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:09:55.668305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.004123+0000","last_change":"2024-05-28T06:24:24.002061+0000","last_active":"2024-05-28T06:24:24.004123+0000","last_peered":"2024-05-28T06:24:24.004123+0000","last_clean":"2024-05-28T06:24:24.004123+0000","last_became_active":"2024-05-28T06:24:24.001429+0000","last_became_peered":"2024-05-28T06:24:24.001429+0000","last_unstale":"2024-05-28T06:24:24.004123+0000","last_undegraded":"2024-05-28T06:24:24.004123+0000","last_fullsized":"2024-05-28T06:24:24.004123+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:19:21.657411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:23.989711+0000","last_change":"2024-05-28T06:24:22.993624+0000","last_active":"2024-05-28T06:24:23.989711+0000","last_peered":"2024-05-28T06:24:23.989711+0000","last_clean":"2024-05-28T06:24:23.989711+0000","last_became_active":"2024-05-28T06:24:22.993301+0000","last_became_peered":"2024-05-28T06:24:22.993301+0000","last_unstale":"2024-05-28T06:24:23.989711+0000","last_undegraded":"2024-05-28T06:24:23.989711+0000","last_fullsized":"2024-05-28T06:24:23.989711+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:26:44.582201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.006076+0000","last_change":"2024-05-28T06:24:24.005361+0000","last_active":"2024-05-28T06:24:24.006076+0000","last_peered":"2024-05-28T06:24:24.006076+0000","last_clean":"2024-05-28T06:24:24.006076+0000","last_became_active":"2024-05-28T06:24:24.005080+0000","last_became_peered":"2024-05-28T06:24:24.005080+0000","last_unstale":"2024-05-28T06:24:24.006076+0000","last_undegraded":"2024-05-28T06:24:24.006076+0000","last_fullsized":"2024-05-28T06:24:24.006076+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:43:02.389778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:23.976103+0000","last_change":"2024-05-28T06:24:23.005565+0000","last_active":"2024-05-28T06:24:23.976103+0000","last_peered":"2024-05-28T06:24:23.976103+0000","last_clean":"2024-05-28T06:24:23.976103+0000","last_became_active":"2024-05-28T06:24:23.005194+0000","last_became_peered":"2024-05-28T06:24:23.005194+0000","last_unstale":"2024-05-28T06:24:23.976103+0000","last_undegraded":"2024-05-28T06:24:23.976103+0000","last_fullsized":"2024-05-28T06:24:23.976103+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:26:55.784217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.018042+0000","last_change":"2024-05-28T06:24:24.009724+0000","last_active":"2024-05-28T06:24:24.018042+0000","last_peered":"2024-05-28T06:24:24.018042+0000","last_clean":"2024-05-28T06:24:24.018042+0000","last_became_active":"2024-05-28T06:24:24.009163+0000","last_became_peered":"2024-05-28T06:24:24.009163+0000","last_unstale":"2024-05-28T06:24:24.018042+0000","last_undegraded":"2024-05-28T06:24:24.018042+0000","last_fullsized":"2024-05-28T06:24:24.018042+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:24:28.571714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.10","version":"129'8","reported_seq":20,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.017620+0000","last_change":"2024-05-28T06:24:22.996927+0000","last_active":"2024-05-28T06:24:24.017620+0000","last_peered":"2024-05-28T06:24:24.017620+0000","last_clean":"2024-05-28T06:24:24.017620+0000","last_became_active":"2024-05-28T06:24:22.996644+0000","last_became_peered":"2024-05-28T06:24:22.996644+0000","last_unstale":"2024-05-28T06:24:24.017620+0000","last_undegraded":"2024-05-28T06:24:24.017620+0000","last_fullsized":"2024-05-28T06:24:24.017620+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:17:26.759155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.018242+0000","last_change":"2024-05-28T06:24:24.009329+0000","last_active":"2024-05-28T06:24:24.018242+0000","last_peered":"2024-05-28T06:24:24.018242+0000","last_clean":"2024-05-28T06:24:24.018242+0000","last_became_active":"2024-05-28T06:24:24.009033+0000","last_became_peered":"2024-05-28T06:24:24.009033+0000","last_unstale":"2024-05-28T06:24:24.018242+0000","last_undegraded":"2024-05-28T06:24:24.018242+0000","last_fullsized":"2024-05-28T06:24:24.018242+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:48:43.340748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.f","version":"129'1","reported_seq":13,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.017941+0000","last_change":"2024-05-28T06:24:23.006567+0000","last_active":"2024-05-28T06:24:24.017941+0000","last_peered":"2024-05-28T06:24:24.017941+0000","last_clean":"2024-05-28T06:24:24.017941+0000","last_became_active":"2024-05-28T06:24:23.005650+0000","last_became_peered":"2024-05-28T06:24:23.005650+0000","last_unstale":"2024-05-28T06:24:24.017941+0000","last_undegraded":"2024-05-28T06:24:24.017941+0000","last_fullsized":"2024-05-28T06:24:24.017941+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:01:42.283187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.021344+0000","last_change":"2024-05-28T06:24:24.016763+0000","last_active":"2024-05-28T06:24:24.021344+0000","last_peered":"2024-05-28T06:24:24.021344+0000","last_clean":"2024-05-28T06:24:24.021344+0000","last_became_active":"2024-05-28T06:24:24.016255+0000","last_became_peered":"2024-05-28T06:24:24.016255+0000","last_unstale":"2024-05-28T06:24:24.021344+0000","last_undegraded":"2024-05-28T06:24:24.021344+0000","last_fullsized":"2024-05-28T06:24:24.021344+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:11:18.017207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.e","version":"129'2","reported_seq":14,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.015790+0000","last_change":"2024-05-28T06:24:23.002513+0000","last_active":"2024-05-28T06:24:24.015790+0000","last_peered":"2024-05-28T06:24:24.015790+0000","last_clean":"2024-05-28T06:24:24.015790+0000","last_became_active":"2024-05-28T06:24:23.001946+0000","last_became_peered":"2024-05-28T06:24:23.001946+0000","last_unstale":"2024-05-28T06:24:24.015790+0000","last_undegraded":"2024-05-28T06:24:24.015790+0000","last_fullsized":"2024-05-28T06:24:24.015790+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:27:48.979163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:23.999634+0000","last_change":"2024-05-28T06:24:23.998032+0000","last_active":"2024-05-28T06:24:23.999634+0000","last_peered":"2024-05-28T06:24:23.999634+0000","last_clean":"2024-05-28T06:24:23.999634+0000","last_became_active":"2024-05-28T06:24:23.996433+0000","last_became_peered":"2024-05-28T06:24:23.996433+0000","last_unstale":"2024-05-28T06:24:23.999634+0000","last_undegraded":"2024-05-28T06:24:23.999634+0000","last_fullsized":"2024-05-28T06:24:23.999634+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:01:59.233483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.d","version":"129'1","reported_seq":15,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-05-28T06:24:25.012501+0000","last_change":"2024-05-28T06:24:22.992776+0000","last_active":"2024-05-28T06:24:25.012501+0000","last_peered":"2024-05-28T06:24:25.012501+0000","last_clean":"2024-05-28T06:24:25.012501+0000","last_became_active":"2024-05-28T06:24:22.992147+0000","last_became_peered":"2024-05-28T06:24:22.992147+0000","last_unstale":"2024-05-28T06:24:25.012501+0000","last_undegraded":"2024-05-28T06:24:25.012501+0000","last_fullsized":"2024-05-28T06:24:25.012501+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:36:09.562056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:23.992367+0000","last_change":"2024-05-28T06:24:23.989538+0000","last_active":"2024-05-28T06:24:23.992367+0000","last_peered":"2024-05-28T06:24:23.992367+0000","last_clean":"2024-05-28T06:24:23.992367+0000","last_became_active":"2024-05-28T06:24:23.988631+0000","last_became_peered":"2024-05-28T06:24:23.988631+0000","last_unstale":"2024-05-28T06:24:23.992367+0000","last_undegraded":"2024-05-28T06:24:23.992367+0000","last_fullsized":"2024-05-28T06:24:23.992367+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:40:57.763291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.007020+0000","last_change":"2024-05-28T06:24:22.997400+0000","last_active":"2024-05-28T06:24:24.007020+0000","last_peered":"2024-05-28T06:24:24.007020+0000","last_clean":"2024-05-28T06:24:24.007020+0000","last_became_active":"2024-05-28T06:24:22.993636+0000","last_became_peered":"2024-05-28T06:24:22.993636+0000","last_unstale":"2024-05-28T06:24:24.007020+0000","last_undegraded":"2024-05-28T06:24:24.007020+0000","last_fullsized":"2024-05-28T06:24:24.007020+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:43:01.373860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.008871+0000","last_change":"2024-05-28T06:24:24.007538+0000","last_active":"2024-05-28T06:24:24.008871+0000","last_peered":"2024-05-28T06:24:24.008871+0000","last_clean":"2024-05-28T06:24:24.008871+0000","last_became_active":"2024-05-28T06:24:24.007259+0000","last_became_peered":"2024-05-28T06:24:24.007259+0000","last_unstale":"2024-05-28T06:24:24.008871+0000","last_undegraded":"2024-05-28T06:24:24.008871+0000","last_fullsized":"2024-05-28T06:24:24.008871+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:49:25.488174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.b","version":"129'2","reported_seq":14,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.018078+0000","last_change":"2024-05-28T06:24:22.996819+0000","last_active":"2024-05-28T06:24:24.018078+0000","last_peered":"2024-05-28T06:24:24.018078+0000","last_clean":"2024-05-28T06:24:24.018078+0000","last_became_active":"2024-05-28T06:24:22.995489+0000","last_became_peered":"2024-05-28T06:24:22.995489+0000","last_unstale":"2024-05-28T06:24:24.018078+0000","last_undegraded":"2024-05-28T06:24:24.018078+0000","last_fullsized":"2024-05-28T06:24:24.018078+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:24:52.967145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-05-28T06:24:25.012022+0000","last_change":"2024-05-28T06:24:24.015790+0000","last_active":"2024-05-28T06:24:25.012022+0000","last_peered":"2024-05-28T06:24:25.012022+0000","last_clean":"2024-05-28T06:24:25.012022+0000","last_became_active":"2024-05-28T06:24:24.015295+0000","last_became_peered":"2024-05-28T06:24:24.015295+0000","last_unstale":"2024-05-28T06:24:25.012022+0000","last_undegraded":"2024-05-28T06:24:25.012022+0000","last_fullsized":"2024-05-28T06:24:25.012022+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:37:59.260972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:23.985595+0000","last_change":"2024-05-28T06:24:22.995343+0000","last_active":"2024-05-28T06:24:23.985595+0000","last_peered":"2024-05-28T06:24:23.985595+0000","last_clean":"2024-05-28T06:24:23.985595+0000","last_became_active":"2024-05-28T06:24:22.994227+0000","last_became_peered":"2024-05-28T06:24:22.994227+0000","last_unstale":"2024-05-28T06:24:23.985595+0000","last_undegraded":"2024-05-28T06:24:23.985595+0000","last_fullsized":"2024-05-28T06:24:23.985595+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:50:39.661313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:23.991209+0000","last_change":"2024-05-28T06:24:23.987172+0000","last_active":"2024-05-28T06:24:23.991209+0000","last_peered":"2024-05-28T06:24:23.991209+0000","last_clean":"2024-05-28T06:24:23.991209+0000","last_became_active":"2024-05-28T06:24:23.986897+0000","last_became_peered":"2024-05-28T06:24:23.986897+0000","last_unstale":"2024-05-28T06:24:23.991209+0000","last_undegraded":"2024-05-28T06:24:23.991209+0000","last_fullsized":"2024-05-28T06:24:23.991209+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T18:06:22.409116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:23.983196+0000","last_change":"2024-05-28T06:24:22.991355+0000","last_active":"2024-05-28T06:24:23.983196+0000","last_peered":"2024-05-28T06:24:23.983196+0000","last_clean":"2024-05-28T06:24:23.983196+0000","last_became_active":"2024-05-28T06:24:22.991078+0000","last_became_peered":"2024-05-28T06:24:22.991078+0000","last_unstale":"2024-05-28T06:24:23.983196+0000","last_undegraded":"2024-05-28T06:24:23.983196+0000","last_fullsized":"2024-05-28T06:24:23.983196+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:06:19.878458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-05-28T06:24:25.900149+0000","last_change":"2024-05-28T06:24:24.009275+0000","last_active":"2024-05-28T06:24:25.900149+0000","last_peered":"2024-05-28T06:24:25.900149+0000","last_clean":"2024-05-28T06:24:25.900149+0000","last_became_active":"2024-05-28T06:24:24.008955+0000","last_became_peered":"2024-05-28T06:24:24.008955+0000","last_unstale":"2024-05-28T06:24:25.900149+0000","last_undegraded":"2024-05-28T06:24:25.900149+0000","last_fullsized":"2024-05-28T06:24:25.900149+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:32:09.823976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:23.981770+0000","last_change":"2024-05-28T06:24:23.002572+0000","last_active":"2024-05-28T06:24:23.981770+0000","last_peered":"2024-05-28T06:24:23.981770+0000","last_clean":"2024-05-28T06:24:23.981770+0000","last_became_active":"2024-05-28T06:24:23.002292+0000","last_became_peered":"2024-05-28T06:24:23.002292+0000","last_unstale":"2024-05-28T06:24:23.981770+0000","last_undegraded":"2024-05-28T06:24:23.981770+0000","last_fullsized":"2024-05-28T06:24:23.981770+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:50:39.856735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-05-28T06:24:25.013427+0000","last_change":"2024-05-28T06:24:24.017529+0000","last_active":"2024-05-28T06:24:25.013427+0000","last_peered":"2024-05-28T06:24:25.013427+0000","last_clean":"2024-05-28T06:24:25.013427+0000","last_became_active":"2024-05-28T06:24:24.017221+0000","last_became_peered":"2024-05-28T06:24:24.017221+0000","last_unstale":"2024-05-28T06:24:25.013427+0000","last_undegraded":"2024-05-28T06:24:25.013427+0000","last_fullsized":"2024-05-28T06:24:25.013427+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:32:09.823976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.7","version":"129'2","reported_seq":14,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.019314+0000","last_change":"2024-05-28T06:24:23.003960+0000","last_active":"2024-05-28T06:24:24.019314+0000","last_peered":"2024-05-28T06:24:24.019314+0000","last_clean":"2024-05-28T06:24:24.019314+0000","last_became_active":"2024-05-28T06:24:23.003502+0000","last_became_peered":"2024-05-28T06:24:23.003502+0000","last_unstale":"2024-05-28T06:24:24.019314+0000","last_undegraded":"2024-05-28T06:24:24.019314+0000","last_fullsized":"2024-05-28T06:24:24.019314+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:13:19.450560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.020939+0000","last_change":"2024-05-28T06:24:24.020448+0000","last_active":"2024-05-28T06:24:24.020939+0000","last_peered":"2024-05-28T06:24:24.020939+0000","last_clean":"2024-05-28T06:24:24.020939+0000","last_became_active":"2024-05-28T06:24:24.019730+0000","last_became_peered":"2024-05-28T06:24:24.019730+0000","last_unstale":"2024-05-28T06:24:24.020939+0000","last_undegraded":"2024-05-28T06:24:24.020939+0000","last_fullsized":"2024-05-28T06:24:24.020939+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:25:49.241649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.6","version":"130'1","reported_seq":15,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-05-28T06:24:25.988359+0000","last_change":"2024-05-28T06:24:23.003133+0000","last_active":"2024-05-28T06:24:25.988359+0000","last_peered":"2024-05-28T06:24:25.988359+0000","last_clean":"2024-05-28T06:24:25.988359+0000","last_became_active":"2024-05-28T06:24:23.002821+0000","last_became_peered":"2024-05-28T06:24:23.002821+0000","last_unstale":"2024-05-28T06:24:25.988359+0000","last_undegraded":"2024-05-28T06:24:25.988359+0000","last_fullsized":"2024-05-28T06:24:25.988359+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:18:06.253798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.005933+0000","last_change":"2024-05-28T06:24:24.004928+0000","last_active":"2024-05-28T06:24:24.005933+0000","last_peered":"2024-05-28T06:24:24.005933+0000","last_clean":"2024-05-28T06:24:24.005933+0000","last_became_active":"2024-05-28T06:24:24.004599+0000","last_became_peered":"2024-05-28T06:24:24.004599+0000","last_unstale":"2024-05-28T06:24:24.005933+0000","last_undegraded":"2024-05-28T06:24:24.005933+0000","last_fullsized":"2024-05-28T06:24:24.005933+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:01:43.299104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:23.976699+0000","last_change":"2024-05-28T06:24:23.003603+0000","last_active":"2024-05-28T06:24:23.976699+0000","last_peered":"2024-05-28T06:24:23.976699+0000","last_clean":"2024-05-28T06:24:23.976699+0000","last_became_active":"2024-05-28T06:24:23.002468+0000","last_became_peered":"2024-05-28T06:24:23.002468+0000","last_unstale":"2024-05-28T06:24:23.976699+0000","last_undegraded":"2024-05-28T06:24:23.976699+0000","last_fullsized":"2024-05-28T06:24:23.976699+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:48:27.218311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-05-28T06:24:25.012540+0000","last_change":"2024-05-28T06:24:23.999563+0000","last_active":"2024-05-28T06:24:25.012540+0000","last_peered":"2024-05-28T06:24:25.012540+0000","last_clean":"2024-05-28T06:24:25.012540+0000","last_became_active":"2024-05-28T06:24:23.999013+0000","last_became_peered":"2024-05-28T06:24:23.999013+0000","last_unstale":"2024-05-28T06:24:25.012540+0000","last_undegraded":"2024-05-28T06:24:25.012540+0000","last_fullsized":"2024-05-28T06:24:25.012540+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:47:25.671388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:23.984679+0000","last_change":"2024-05-28T06:24:22.986876+0000","last_active":"2024-05-28T06:24:23.984679+0000","last_peered":"2024-05-28T06:24:23.984679+0000","last_clean":"2024-05-28T06:24:23.984679+0000","last_became_active":"2024-05-28T06:24:22.986620+0000","last_became_peered":"2024-05-28T06:24:22.986620+0000","last_unstale":"2024-05-28T06:24:23.984679+0000","last_undegraded":"2024-05-28T06:24:23.984679+0000","last_fullsized":"2024-05-28T06:24:23.984679+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:20:23.711600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-05-28T06:24:25.900260+0000","last_change":"2024-05-28T06:24:24.012293+0000","last_active":"2024-05-28T06:24:25.900260+0000","last_peered":"2024-05-28T06:24:25.900260+0000","last_clean":"2024-05-28T06:24:25.900260+0000","last_became_active":"2024-05-28T06:24:24.011989+0000","last_became_peered":"2024-05-28T06:24:24.011989+0000","last_unstale":"2024-05-28T06:24:25.900260+0000","last_undegraded":"2024-05-28T06:24:25.900260+0000","last_fullsized":"2024-05-28T06:24:25.900260+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:47:31.291023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:23.976319+0000","last_change":"2024-05-28T06:24:23.008363+0000","last_active":"2024-05-28T06:24:23.976319+0000","last_peered":"2024-05-28T06:24:23.976319+0000","last_clean":"2024-05-28T06:24:23.976319+0000","last_became_active":"2024-05-28T06:24:23.007859+0000","last_became_peered":"2024-05-28T06:24:23.007859+0000","last_unstale":"2024-05-28T06:24:23.976319+0000","last_undegraded":"2024-05-28T06:24:23.976319+0000","last_fullsized":"2024-05-28T06:24:23.976319+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:45:32.156309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.017941+0000","last_change":"2024-05-28T06:24:24.013906+0000","last_active":"2024-05-28T06:24:24.017941+0000","last_peered":"2024-05-28T06:24:24.017941+0000","last_clean":"2024-05-28T06:24:24.017941+0000","last_became_active":"2024-05-28T06:24:24.011269+0000","last_became_peered":"2024-05-28T06:24:24.011269+0000","last_unstale":"2024-05-28T06:24:24.017941+0000","last_undegraded":"2024-05-28T06:24:24.017941+0000","last_fullsized":"2024-05-28T06:24:24.017941+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:27:41.877671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-05-28T06:24:25.900546+0000","last_change":"2024-05-28T06:24:23.006728+0000","last_active":"2024-05-28T06:24:25.900546+0000","last_peered":"2024-05-28T06:24:25.900546+0000","last_clean":"2024-05-28T06:24:25.900546+0000","last_became_active":"2024-05-28T06:24:23.005895+0000","last_became_peered":"2024-05-28T06:24:23.005895+0000","last_unstale":"2024-05-28T06:24:25.900546+0000","last_undegraded":"2024-05-28T06:24:25.900546+0000","last_fullsized":"2024-05-28T06:24:25.900546+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:53:06.353639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.013602+0000","last_change":"2024-05-28T06:24:24.013072+0000","last_active":"2024-05-28T06:24:24.013602+0000","last_peered":"2024-05-28T06:24:24.013602+0000","last_clean":"2024-05-28T06:24:24.013602+0000","last_became_active":"2024-05-28T06:24:24.012814+0000","last_became_peered":"2024-05-28T06:24:24.012814+0000","last_unstale":"2024-05-28T06:24:24.013602+0000","last_undegraded":"2024-05-28T06:24:24.013602+0000","last_fullsized":"2024-05-28T06:24:24.013602+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:47:32.965627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:23.983354+0000","last_change":"2024-05-28T06:24:23.001199+0000","last_active":"2024-05-28T06:24:23.983354+0000","last_peered":"2024-05-28T06:24:23.983354+0000","last_clean":"2024-05-28T06:24:23.983354+0000","last_became_active":"2024-05-28T06:24:23.000861+0000","last_became_peered":"2024-05-28T06:24:23.000861+0000","last_unstale":"2024-05-28T06:24:23.983354+0000","last_undegraded":"2024-05-28T06:24:23.983354+0000","last_fullsized":"2024-05-28T06:24:23.983354+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:36:09.562056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":9,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:24.006877+0000","last_change":"2024-05-28T06:24:24.005940+0000","last_active":"2024-05-28T06:24:24.006877+0000","last_peered":"2024-05-28T06:24:24.006877+0000","last_clean":"2024-05-28T06:24:24.006877+0000","last_became_active":"2024-05-28T06:24:24.005485+0000","last_became_peered":"2024-05-28T06:24:24.005485+0000","last_unstale":"2024-05-28T06:24:24.006877+0000","last_undegraded":"2024-05-28T06:24:24.006877+0000","last_fullsized":"2024-05-28T06:24:24.006877+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:24:53.983062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-05-28T06:24:23.981899+0000","last_change":"2024-05-28T06:24:22.987451+0000","last_active":"2024-05-28T06:24:23.981899+0000","last_peered":"2024-05-28T06:24:23.981899+0000","last_clean":"2024-05-28T06:24:23.981899+0000","last_became_active":"2024-05-28T06:24:22.987181+0000","last_became_peered":"2024-05-28T06:24:22.987181+0000","last_unstale":"2024-05-28T06:24:23.981899+0000","last_undegraded":"2024-05-28T06:24:23.981899+0000","last_fullsized":"2024-05-28T06:24:23.981899+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:45:51.036866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-05-28T06:24:25.900863+0000","last_change":"2024-05-28T06:24:24.007933+0000","last_active":"2024-05-28T06:24:25.900863+0000","last_peered":"2024-05-28T06:24:25.900863+0000","last_clean":"2024-05-28T06:24:25.900863+0000","last_became_active":"2024-05-28T06:24:24.007651+0000","last_became_peered":"2024-05-28T06:24:24.007651+0000","last_unstale":"2024-05-28T06:24:25.900863+0000","last_undegraded":"2024-05-28T06:24:25.900863+0000","last_fullsized":"2024-05-28T06:24:25.900863+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:59:32.549372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"129'2","reported_seq":16,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-05-28T06:24:25.900772+0000","last_change":"2024-05-28T06:24:23.008078+0000","last_active":"2024-05-28T06:24:25.900772+0000","last_peered":"2024-05-28T06:24:25.900772+0000","last_clean":"2024-05-28T06:24:25.900772+0000","last_became_active":"2024-05-28T06:24:23.007331+0000","last_became_peered":"2024-05-28T06:24:23.007331+0000","last_unstale":"2024-05-28T06:24:25.900772+0000","last_undegraded":"2024-05-28T06:24:25.900772+0000","last_fullsized":"2024-05-28T06:24:25.900772+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:50:39.661313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":383,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-05-28T06:24:25.899651+0000","last_change":"2024-05-28T06:04:01.039129+0000","last_active":"2024-05-28T06:24:25.899651+0000","last_peered":"2024-05-28T06:24:25.899651+0000","last_clean":"2024-05-28T06:24:25.899651+0000","last_became_active":"2024-05-28T06:04:01.033906+0000","last_became_peered":"2024-05-28T06:04:01.033906+0000","last_unstale":"2024-05-28T06:24:25.899651+0000","last_undegraded":"2024-05-28T06:24:25.899651+0000","last_fullsized":"2024-05-28T06:24:25.899651+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:03:59.864237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:03:59.864237+0000","last_clean_scrub_stamp":"2024-05-28T06:03:59.864237+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:17:36.440245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575096,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":131860,"kb_used_data":2176,"kb_used_omap":1,"kb_used_meta":129662,"kb_avail":94239980,"statfs":{"total":96636764160,"available":96501739520,"internally_reserved":0,"allocated":2228224,"data_stored":1301310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":132774346},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575098,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":131268,"kb_used_data":1584,"kb_used_omap":1,"kb_used_meta":129662,"kb_avail":94240572,"statfs":{"total":96636764160,"available":96502345728,"internally_reserved":0,"allocated":1622016,"data_stored":710304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":132774342},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575096,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64636,"kb_used_data":1580,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94307204,"statfs":{"total":96636764160,"available":96570576896,"internally_reserved":0,"allocated":1617920,"data_stored":708804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575096,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":149312,"kb_used_data":1588,"kb_used_omap":1,"kb_used_meta":147710,"kb_avail":94222528,"statfs":{"total":96636764160,"available":96483868672,"internally_reserved":0,"allocated":1626112,"data_stored":715118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151255493},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575098,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":118784,"kb_used_data":1580,"kb_used_omap":1,"kb_used_meta":117182,"kb_avail":94253056,"statfs":{"total":96636764160,"available":96515129344,"internally_reserved":0,"allocated":1617920,"data_stored":708852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":119994823},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575097,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":149312,"kb_used_data":1580,"kb_used_omap":1,"kb_used_meta":147710,"kb_avail":94222528,"statfs":{"total":96636764160,"available":96483868672,"internally_reserved":0,"allocated":1617920,"data_stored":709308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151255493},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575097,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":149304,"kb_used_data":1572,"kb_used_omap":1,"kb_used_meta":147710,"kb_avail":94222536,"statfs":{"total":96636764160,"available":96483876864,"internally_reserved":0,"allocated":1609728,"data_stored":708692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151255493},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575096,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":145724,"kb_used_data":2152,"kb_used_omap":1,"kb_used_meta":143550,"kb_avail":94226116,"statfs":{"total":96636764160,"available":96487542784,"internally_reserved":0,"allocated":2203648,"data_stored":1299060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":146995653},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-28T06:24:28.923 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-05-28T06:24:28.924 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-05-28T06:24:29.175 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.181+0000 7f1a71600640 1 Processor -- start 2024-05-28T06:24:29.176 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.181+0000 7f1a71600640 1 -- start start 2024-05-28T06:24:29.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.183+0000 7f1a71600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f1a6c15d5d0 0x7f1a6c0cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:29.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.183+0000 7f1a71600640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a6c15cb10 con 0x7f1a6c0b32c0 2024-05-28T06:24:29.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.183+0000 7f1a71600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a6c0ad5f0 con 0x7f1a6c0d0640 2024-05-28T06:24:29.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.183+0000 7f1a71600640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f1a6c0adbb0 con 0x7f1a6c15d5d0 2024-05-28T06:24:29.178 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.183+0000 7f1a6b400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f1a6c0b32c0 0x7f1a6c0b3700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:57188/0 (socket says 172.21.15.96:57188) 2024-05-28T06:24:29.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.183+0000 7f1a6b400640 1 -- 172.21.15.96:0/486674500 learned_addr learned my addr 172.21.15.96:0/486674500 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:24:29.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.183+0000 7f1a63e00640 1 --2- 172.21.15.96:0/486674500 >> v2:172.21.15.96:3300/0 conn(0x7f1a6c15d5d0 0x7f1a6c0cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:29.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.183+0000 7f1a6a000640 1 -- 172.21.15.96:0/486674500 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2553436885 0 0) 0x7f1a6c15cb10 con 0x7f1a6c0b32c0 2024-05-28T06:24:29.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.183+0000 7f1a6a000640 1 -- 172.21.15.96:0/486674500 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1a50003520 con 0x7f1a6c0b32c0 2024-05-28T06:24:29.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.184+0000 7f1a63e00640 1 -- 172.21.15.96:0/486674500 >> v1:172.21.15.123:6789/0 conn(0x7f1a6c0d0640 legacy=0x7f1a6c0d0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:29.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.184+0000 7f1a63e00640 1 -- 172.21.15.96:0/486674500 >> v1:172.21.15.96:6789/0 conn(0x7f1a6c0b32c0 legacy=0x7f1a6c0b3700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:29.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.184+0000 7f1a63e00640 1 -- 172.21.15.96:0/486674500 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a6c0ad8d0 con 0x7f1a6c15d5d0 2024-05-28T06:24:29.182 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.185+0000 7f1a63e00640 1 --2- 172.21.15.96:0/486674500 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a6c15d5d0 0x7f1a6c0cf950 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 crypto rx=0x7f1a540097a0 tx=0x7f1a5402a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a64e8ffac66585b0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:29.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.185+0000 7f1a6a000640 1 -- 172.21.15.96:0/486674500 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a54037070 con 0x7f1a6c15d5d0 2024-05-28T06:24:29.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.185+0000 7f1a6a000640 1 -- 172.21.15.96:0/486674500 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a5402ae50 con 0x7f1a6c15d5d0 2024-05-28T06:24:29.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.185+0000 7f1a6a000640 1 -- 172.21.15.96:0/486674500 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a54032660 con 0x7f1a6c15d5d0 2024-05-28T06:24:29.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.186+0000 7f1a71600640 1 -- 172.21.15.96:0/486674500 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a6c15d5d0 msgr2=0x7f1a6c0cf950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:29.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.186+0000 7f1a71600640 1 --2- 172.21.15.96:0/486674500 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a6c15d5d0 0x7f1a6c0cf950 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 crypto rx=0x7f1a540097a0 tx=0x7f1a5402a140 comp rx=0 tx=0).stop 2024-05-28T06:24:29.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.186+0000 7f1a71600640 1 -- 172.21.15.96:0/486674500 shutdown_connections 2024-05-28T06:24:29.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.186+0000 7f1a71600640 1 --2- 172.21.15.96:0/486674500 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a6c15d5d0 0x7f1a6c0cf950 unknown :-1 s=CLOSED pgs=2030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:29.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.186+0000 7f1a71600640 1 -- 172.21.15.96:0/486674500 >> 172.21.15.96:0/486674500 conn(0x7f1a6c06bd20 msgr2=0x7f1a6c06c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:29.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.186+0000 7f1a71600640 1 -- 172.21.15.96:0/486674500 shutdown_connections 2024-05-28T06:24:29.183 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.186+0000 7f1a71600640 1 -- 172.21.15.96:0/486674500 wait complete. 2024-05-28T06:24:29.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.188+0000 7f1a71600640 1 Processor -- start 2024-05-28T06:24:29.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.188+0000 7f1a71600640 1 -- start start 2024-05-28T06:24:29.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.189+0000 7f1a71600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1a6c0b32c0 0x7f1a6c14b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:29.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.189+0000 7f1a71600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a6c0d0640 0x7f1a6c06f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:29.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.189+0000 7f1a71600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a6c15d5d0 0x7f1a6c06f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:29.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.189+0000 7f1a71600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f1a6c15da50 con 0x7f1a6c0d0640 2024-05-28T06:24:29.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.189+0000 7f1a71600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f1a6c15d0b0 con 0x7f1a6c15d5d0 2024-05-28T06:24:29.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.189+0000 7f1a71600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f1a6c0b3e10 con 0x7f1a6c0b32c0 2024-05-28T06:24:29.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.189+0000 7f1a63e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a6c0d0640 0x7f1a6c06f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:29.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.189+0000 7f1a63e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a6c0d0640 0x7f1a6c06f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:46956/0 (socket says 172.21.15.96:46956) 2024-05-28T06:24:29.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.189+0000 7f1a63e00640 1 -- 172.21.15.96:0/2383257450 learned_addr learned my addr 172.21.15.96:0/2383257450 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:24:29.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.190+0000 7f1a6b400640 1 --2- 172.21.15.96:0/2383257450 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a6c15d5d0 0x7f1a6c06f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:29.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.190+0000 7f1a63e00640 1 -- 172.21.15.96:0/2383257450 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1a6c0b32c0 msgr2=0x7f1a6c14b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:29.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.190+0000 7f1a63e00640 1 --2- 172.21.15.96:0/2383257450 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1a6c0b32c0 0x7f1a6c14b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:29.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.190+0000 7f1a63e00640 1 -- 172.21.15.96:0/2383257450 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a6c15d5d0 msgr2=0x7f1a6c06f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:29.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.190+0000 7f1a63e00640 1 --2- 172.21.15.96:0/2383257450 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a6c15d5d0 0x7f1a6c06f590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:29.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.190+0000 7f1a63e00640 1 -- 172.21.15.96:0/2383257450 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a6c1f7ba0 con 0x7f1a6c0d0640 2024-05-28T06:24:29.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.190+0000 7f1a6b400640 1 --2- 172.21.15.96:0/2383257450 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a6c15d5d0 0x7f1a6c06f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:24:29.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.190+0000 7f1a63e00640 1 --2- 172.21.15.96:0/2383257450 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a6c0d0640 0x7f1a6c06f050 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 crypto rx=0x7f1a54035040 tx=0x7f1a540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:29.187 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.190+0000 7f1a62a00640 1 -- 172.21.15.96:0/2383257450 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a54042020 con 0x7f1a6c0d0640 2024-05-28T06:24:29.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.190+0000 7f1a62a00640 1 -- 172.21.15.96:0/2383257450 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a5403d470 con 0x7f1a6c0d0640 2024-05-28T06:24:29.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.190+0000 7f1a62a00640 1 -- 172.21.15.96:0/2383257450 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a54037040 con 0x7f1a6c0d0640 2024-05-28T06:24:29.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.190+0000 7f1a71600640 1 -- 172.21.15.96:0/2383257450 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a6c1f7e80 con 0x7f1a6c0d0640 2024-05-28T06:24:29.188 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.190+0000 7f1a71600640 1 -- 172.21.15.96:0/2383257450 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a6c1f8440 con 0x7f1a6c0d0640 2024-05-28T06:24:29.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.192+0000 7f1a71600640 1 -- 172.21.15.96:0/2383257450 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a6c0689e0 con 0x7f1a6c0d0640 2024-05-28T06:24:29.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.193+0000 7f1a62a00640 1 -- 172.21.15.96:0/2383257450 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 42) ==== 93590+0+0 (secure 0 0 0) 0x7f1a54045020 con 0x7f1a6c0d0640 2024-05-28T06:24:29.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.194+0000 7f1a62a00640 1 --2- 172.21.15.96:0/2383257450 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1a4806e510 0x7f1a480709d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:29.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.194+0000 7f1a62a00640 1 -- 172.21.15.96:0/2383257450 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 5765+0+0 (secure 0 0 0) 0x7f1a540ab340 con 0x7f1a6c0d0640 2024-05-28T06:24:29.189 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.194+0000 7f1a6aa00640 1 --2- 172.21.15.96:0/2383257450 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1a4806e510 0x7f1a480709d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:29.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.196+0000 7f1a6aa00640 1 --2- 172.21.15.96:0/2383257450 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1a4806e510 0x7f1a480709d0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c003cc0 tx=0x7f1a5c0079b0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:29.192 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.198+0000 7f1a62a00640 1 -- 172.21.15.96:0/2383257450 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1a5404c030 con 0x7f1a6c0d0640 2024-05-28T06:24:29.465 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.471+0000 7f1a71600640 1 -- 172.21.15.96:0/2383257450 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f1a6c071ef0 con 0x7f1a6c0d0640 2024-05-28T06:24:29.470 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.475+0000 7f1a62a00640 1 -- 172.21.15.96:0/2383257450 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v24) ==== 190+0+147 (secure 0 0 0) 0x7f1a5407b1d0 con 0x7f1a6c0d0640 2024-05-28T06:24:29.470 INFO:teuthology.orchestra.run.smithi096.stdout:[client.0] 2024-05-28T06:24:29.470 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQCXc1Vm91hbGBAASTL3HQR8XOs7wVQEHbPl/g== 2024-05-28T06:24:29.470 INFO:teuthology.orchestra.run.smithi096.stdout: caps mds = "allow" 2024-05-28T06:24:29.470 INFO:teuthology.orchestra.run.smithi096.stdout: caps mon = "allow r" 2024-05-28T06:24:29.471 INFO:teuthology.orchestra.run.smithi096.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-05-28T06:24:29.471 INFO:teuthology.orchestra.run.smithi096.stderr:updated caps for client.0 2024-05-28T06:24:29.473 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.479+0000 7f1a71600640 1 -- 172.21.15.96:0/2383257450 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1a4806e510 msgr2=0x7f1a480709d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:29.473 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.479+0000 7f1a71600640 1 --2- 172.21.15.96:0/2383257450 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1a4806e510 0x7f1a480709d0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c003cc0 tx=0x7f1a5c0079b0 comp rx=0 tx=0).stop 2024-05-28T06:24:29.474 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.479+0000 7f1a71600640 1 -- 172.21.15.96:0/2383257450 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a6c0d0640 msgr2=0x7f1a6c06f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:29.474 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.479+0000 7f1a71600640 1 --2- 172.21.15.96:0/2383257450 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a6c0d0640 0x7f1a6c06f050 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 crypto rx=0x7f1a54035040 tx=0x7f1a540046c0 comp rx=0 tx=0).stop 2024-05-28T06:24:29.474 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.479+0000 7f1a71600640 1 -- 172.21.15.96:0/2383257450 shutdown_connections 2024-05-28T06:24:29.474 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.479+0000 7f1a71600640 1 --2- 172.21.15.96:0/2383257450 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1a4806e510 0x7f1a480709d0 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:29.474 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.479+0000 7f1a71600640 1 --2- 172.21.15.96:0/2383257450 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a6c15d5d0 0x7f1a6c06f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:29.474 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.479+0000 7f1a71600640 1 --2- 172.21.15.96:0/2383257450 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a6c0d0640 0x7f1a6c06f050 unknown :-1 s=CLOSED pgs=2031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:29.474 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.479+0000 7f1a71600640 1 --2- 172.21.15.96:0/2383257450 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1a6c0b32c0 0x7f1a6c14b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:29.474 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.479+0000 7f1a71600640 1 -- 172.21.15.96:0/2383257450 >> 172.21.15.96:0/2383257450 conn(0x7f1a6c06bd20 msgr2=0x7f1a6c15e2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:29.474 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.479+0000 7f1a71600640 1 -- 172.21.15.96:0/2383257450 shutdown_connections 2024-05-28T06:24:29.474 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.479+0000 7f1a71600640 1 -- 172.21.15.96:0/2383257450 wait complete. 2024-05-28T06:24:29.492 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-28T06:24:29.689 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.695+0000 7f7ca3200640 1 Processor -- start 2024-05-28T06:24:29.690 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.695+0000 7f7ca3200640 1 -- start start 2024-05-28T06:24:29.691 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.696+0000 7f7ca3200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7ca4156b30 0x7f7ca40af6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:29.691 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.696+0000 7f7ca3200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f7ca406ee40 0x7f7ca40ad230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:29.691 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.696+0000 7f7ca3200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f7ca406f2c0 0x7f7ca40ad770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:29.691 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.696+0000 7f7ca3200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f7ca4156f10 con 0x7f7ca406ee40 2024-05-28T06:24:29.691 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.696+0000 7f7ca3200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f7ca406dda0 con 0x7f7ca406f2c0 2024-05-28T06:24:29.691 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.696+0000 7f7ca3200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f7ca406e080 con 0x7f7ca4156b30 2024-05-28T06:24:29.691 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.697+0000 7f7c99400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f7ca406ee40 0x7f7ca40ad230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:29.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.697+0000 7f7c99400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f7ca406ee40 0x7f7ca40ad230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:46964/0 (socket says 172.21.15.96:46964) 2024-05-28T06:24:29.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.697+0000 7f7c99400640 1 -- 172.21.15.96:0/1741983350 learned_addr learned my addr 172.21.15.96:0/1741983350 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:24:29.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.697+0000 7f7ca1e00640 1 --2- 172.21.15.96:0/1741983350 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7ca4156b30 0x7f7ca40af6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:29.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.697+0000 7f7ca2800640 1 --2- 172.21.15.96:0/1741983350 >> v2:172.21.15.123:3300/0 conn(0x7f7ca406f2c0 0x7f7ca40ad770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:29.693 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.697+0000 7f7c99400640 1 -- 172.21.15.96:0/1741983350 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7ca4156b30 msgr2=0x7f7ca40af6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:29.693 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.697+0000 7f7c99400640 1 --2- 172.21.15.96:0/1741983350 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7ca4156b30 0x7f7ca40af6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:29.693 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.697+0000 7f7c99400640 1 -- 172.21.15.96:0/1741983350 >> v2:172.21.15.123:3300/0 conn(0x7f7ca406f2c0 msgr2=0x7f7ca40ad770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:29.693 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.697+0000 7f7c99400640 1 --2- 172.21.15.96:0/1741983350 >> v2:172.21.15.123:3300/0 conn(0x7f7ca406f2c0 0x7f7ca40ad770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:29.693 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.697+0000 7f7c99400640 1 -- 172.21.15.96:0/1741983350 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ca406e360 con 0x7f7ca406ee40 2024-05-28T06:24:29.693 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.698+0000 7f7c99400640 1 --2- 172.21.15.96:0/1741983350 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7ca406ee40 0x7f7ca40ad230 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 crypto rx=0x7f7ca4071410 tx=0x7f7c8c02a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82b84bca1811e05 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:29.694 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.699+0000 7f7ca1400640 1 -- 172.21.15.96:0/1741983350 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c8c037070 con 0x7f7ca406ee40 2024-05-28T06:24:29.694 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.699+0000 7f7ca1400640 1 -- 172.21.15.96:0/1741983350 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c8c032440 con 0x7f7ca406ee40 2024-05-28T06:24:29.694 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.699+0000 7f7ca1400640 1 -- 172.21.15.96:0/1741983350 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c8c032810 con 0x7f7ca406ee40 2024-05-28T06:24:29.694 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.699+0000 7f7ca3200640 1 -- 172.21.15.96:0/1741983350 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7ca406ee40 msgr2=0x7f7ca40ad230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:29.694 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.699+0000 7f7ca3200640 1 --2- 172.21.15.96:0/1741983350 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7ca406ee40 0x7f7ca40ad230 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 crypto rx=0x7f7ca4071410 tx=0x7f7c8c02a140 comp rx=0 tx=0).stop 2024-05-28T06:24:29.694 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.700+0000 7f7ca3200640 1 -- 172.21.15.96:0/1741983350 shutdown_connections 2024-05-28T06:24:29.695 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.700+0000 7f7ca3200640 1 --2- 172.21.15.96:0/1741983350 >> v2:172.21.15.123:3300/0 conn(0x7f7ca406f2c0 0x7f7ca40ad770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:29.695 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.700+0000 7f7ca3200640 1 --2- 172.21.15.96:0/1741983350 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7ca406ee40 0x7f7ca40ad230 unknown :-1 s=CLOSED pgs=2032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:29.695 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.700+0000 7f7ca3200640 1 --2- 172.21.15.96:0/1741983350 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7ca4156b30 0x7f7ca40af6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:29.695 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.700+0000 7f7ca3200640 1 -- 172.21.15.96:0/1741983350 >> 172.21.15.96:0/1741983350 conn(0x7f7ca40b1fc0 msgr2=0x7f7ca40b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:29.695 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.700+0000 7f7ca3200640 1 -- 172.21.15.96:0/1741983350 shutdown_connections 2024-05-28T06:24:29.695 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.700+0000 7f7ca3200640 1 -- 172.21.15.96:0/1741983350 wait complete. 2024-05-28T06:24:29.695 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.701+0000 7f7ca3200640 1 Processor -- start 2024-05-28T06:24:29.696 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.701+0000 7f7ca3200640 1 -- start start 2024-05-28T06:24:29.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.702+0000 7f7ca3200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7ca406ee40 0x7f7ca4155c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:29.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.702+0000 7f7ca3200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7ca406f2c0 0x7f7ca41ef310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:29.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.702+0000 7f7ca3200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7ca4156b30 0x7f7ca4072d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:29.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.702+0000 7f7ca3200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f7ca40afc20 con 0x7f7ca406f2c0 2024-05-28T06:24:29.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.702+0000 7f7ca3200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f7ca40b2c30 con 0x7f7ca4156b30 2024-05-28T06:24:29.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.702+0000 7f7ca3200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f7ca4141f70 con 0x7f7ca406ee40 2024-05-28T06:24:29.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.703+0000 7f7c99400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7ca406f2c0 0x7f7ca41ef310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:29.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.703+0000 7f7c99400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7ca406f2c0 0x7f7ca41ef310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:46966/0 (socket says 172.21.15.96:46966) 2024-05-28T06:24:29.698 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.703+0000 7f7c99400640 1 -- 172.21.15.96:0/1935959519 learned_addr learned my addr 172.21.15.96:0/1935959519 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:24:29.699 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.703+0000 7f7ca2800640 1 --2- 172.21.15.96:0/1935959519 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7ca4156b30 0x7f7ca4072d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:29.699 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.703+0000 7f7ca1e00640 1 --2- 172.21.15.96:0/1935959519 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7ca406ee40 0x7f7ca4155c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:29.699 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.703+0000 7f7c99400640 1 -- 172.21.15.96:0/1935959519 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7ca406ee40 msgr2=0x7f7ca4155c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:29.699 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.703+0000 7f7c99400640 1 --2- 172.21.15.96:0/1935959519 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7ca406ee40 0x7f7ca4155c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:29.699 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.703+0000 7f7c99400640 1 -- 172.21.15.96:0/1935959519 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7ca4156b30 msgr2=0x7f7ca4072d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:29.699 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.703+0000 7f7c99400640 1 --2- 172.21.15.96:0/1935959519 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7ca4156b30 0x7f7ca4072d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:29.699 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.703+0000 7f7c99400640 1 -- 172.21.15.96:0/1935959519 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ca4154250 con 0x7f7ca406f2c0 2024-05-28T06:24:29.699 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.703+0000 7f7c99400640 1 --2- 172.21.15.96:0/1935959519 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7ca406f2c0 0x7f7ca41ef310 secure :-1 s=READY pgs=2033 cs=0 l=1 rev1=1 crypto rx=0x7f7c8c02a180 tx=0x7f7c8c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:29.700 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.704+0000 7f7c9aa00640 1 -- 172.21.15.96:0/1935959519 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c8c037040 con 0x7f7ca406f2c0 2024-05-28T06:24:29.700 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.704+0000 7f7c9aa00640 1 -- 172.21.15.96:0/1935959519 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c8c042070 con 0x7f7ca406f2c0 2024-05-28T06:24:29.700 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.704+0000 7f7c9aa00640 1 -- 172.21.15.96:0/1935959519 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c8c030bb0 con 0x7f7ca406f2c0 2024-05-28T06:24:29.700 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.704+0000 7f7ca3200640 1 -- 172.21.15.96:0/1935959519 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ca41544f0 con 0x7f7ca406f2c0 2024-05-28T06:24:29.700 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.704+0000 7f7ca3200640 1 -- 172.21.15.96:0/1935959519 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ca41561a0 con 0x7f7ca406f2c0 2024-05-28T06:24:29.700 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.705+0000 7f7ca3200640 1 -- 172.21.15.96:0/1935959519 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ca41420f0 con 0x7f7ca406f2c0 2024-05-28T06:24:29.700 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.706+0000 7f7c9aa00640 1 -- 172.21.15.96:0/1935959519 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 42) ==== 93590+0+0 (secure 0 0 0) 0x7f7c8c047460 con 0x7f7ca406f2c0 2024-05-28T06:24:29.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.707+0000 7f7c9aa00640 1 --2- 172.21.15.96:0/1935959519 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7c8006e510 0x7f7c800709d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:29.701 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.707+0000 7f7c9aa00640 1 -- 172.21.15.96:0/1935959519 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 5765+0+0 (secure 0 0 0) 0x7f7c8c0ab7b0 con 0x7f7ca406f2c0 2024-05-28T06:24:29.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.708+0000 7f7ca1e00640 1 --2- 172.21.15.96:0/1935959519 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7c8006e510 0x7f7c800709d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:29.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.709+0000 7f7ca1e00640 1 --2- 172.21.15.96:0/1935959519 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7c8006e510 0x7f7c800709d0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f7c9400b7c0 tx=0x7f7c94008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:29.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.712+0000 7f7c9aa00640 1 -- 172.21.15.96:0/1935959519 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7c8c045360 con 0x7f7ca406f2c0 2024-05-28T06:24:29.925 INFO:tasks.ceph.mds.a.smithi096.stderr:2024-05-28T06:24:29.930+0000 7f68afe00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-28T06:24:29.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.954+0000 7f7ca3200640 1 -- 172.21.15.96:0/1935959519 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7ca4156370 con 0x7f7ca406f2c0 2024-05-28T06:24:29.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.955+0000 7f7c9aa00640 1 -- 172.21.15.96:0/1935959519 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) ==== 94+0+5186 (secure 0 0 0) 0x7f7c8c07b640 con 0x7f7ca406f2c0 2024-05-28T06:24:29.949 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:24:29.949 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":82,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6287,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/227668757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":227668757},{"type":"v1","addr":"172.21.15.123:6835","nonce":227668757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":77},{"gid":6290,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/1637451105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":1637451105},{"type":"v1","addr":"172.21.15.123:6837","nonce":1637451105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":77},{"gid":9843,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2263843873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2263843873},{"type":"v1","addr":"172.21.15.96:6833","nonce":2263843873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-28T06:24:23.419984+0000","modified":"2024-05-28T06:24:26.498371+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9849},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9849":{"gid":9849,"name":"a","rank":0,"incarnation":78,"state":"up:active","state_seq":3,"addr":"172.21.15.96:6835/1720339900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":1720339900},{"type":"v1","addr":"172.21.15.96:6835","nonce":1720339900}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9849,"qdb_cluster":[9849]},"id":9}]} 2024-05-28T06:24:29.950 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 82 2024-05-28T06:24:29.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.957+0000 7f7ca3200640 1 -- 172.21.15.96:0/1935959519 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7c8006e510 msgr2=0x7f7c800709d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:29.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.957+0000 7f7ca3200640 1 --2- 172.21.15.96:0/1935959519 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7c8006e510 0x7f7c800709d0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f7c9400b7c0 tx=0x7f7c94008040 comp rx=0 tx=0).stop 2024-05-28T06:24:29.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.957+0000 7f7ca3200640 1 -- 172.21.15.96:0/1935959519 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7ca406f2c0 msgr2=0x7f7ca41ef310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:29.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.957+0000 7f7ca3200640 1 --2- 172.21.15.96:0/1935959519 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7ca406f2c0 0x7f7ca41ef310 secure :-1 s=READY pgs=2033 cs=0 l=1 rev1=1 crypto rx=0x7f7c8c02a180 tx=0x7f7c8c0046c0 comp rx=0 tx=0).stop 2024-05-28T06:24:29.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.957+0000 7f7ca3200640 1 -- 172.21.15.96:0/1935959519 shutdown_connections 2024-05-28T06:24:29.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.957+0000 7f7ca3200640 1 --2- 172.21.15.96:0/1935959519 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7c8006e510 0x7f7c800709d0 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:29.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.957+0000 7f7ca3200640 1 --2- 172.21.15.96:0/1935959519 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7ca4156b30 0x7f7ca4072d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:29.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.957+0000 7f7ca3200640 1 --2- 172.21.15.96:0/1935959519 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7ca406f2c0 0x7f7ca41ef310 unknown :-1 s=CLOSED pgs=2033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:29.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.957+0000 7f7ca3200640 1 --2- 172.21.15.96:0/1935959519 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7ca406ee40 0x7f7ca4155c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:29.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.957+0000 7f7ca3200640 1 -- 172.21.15.96:0/1935959519 >> 172.21.15.96:0/1935959519 conn(0x7f7ca40b1fc0 msgr2=0x7f7ca415bac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:29.953 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.957+0000 7f7ca3200640 1 -- 172.21.15.96:0/1935959519 shutdown_connections 2024-05-28T06:24:29.953 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:29.957+0000 7f7ca3200640 1 -- 172.21.15.96:0/1935959519 wait complete. 2024-05-28T06:24:30.374 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-05-28T06:24:23.419984+0000', 'modified': '2024-05-28T06:24:26.498371+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9849}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9849': {'gid': 9849, 'name': 'a', 'rank': 0, 'incarnation': 78, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.96:6835/1720339900', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.96:6834', 'nonce': 1720339900}, {'type': 'v1', 'addr': '172.21.15.96:6835', 'nonce': 1720339900}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9849, 'qdb_cluster': [9849]} 2024-05-28T06:24:30.374 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-05-28T06:24:30.374 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-05-28T06:24:30.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.593+0000 7f1a8fe00640 1 Processor -- start 2024-05-28T06:24:30.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.593+0000 7f1a8fe00640 1 -- start start 2024-05-28T06:24:30.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.594+0000 7f1a8fe00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1a9006ee40 0x7f1a900ad230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:30.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.594+0000 7f1a8fe00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f1a9006f2c0 0x7f1a900ad770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:30.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.594+0000 7f1a8fe00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a90156f10 con 0x7f1a90156b30 2024-05-28T06:24:30.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.594+0000 7f1a8fe00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f1a9006dda0 con 0x7f1a9006f2c0 2024-05-28T06:24:30.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.594+0000 7f1a8fe00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f1a9006e080 con 0x7f1a9006ee40 2024-05-28T06:24:30.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.595+0000 7f1a8ea00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f1a90156b30 0x7f1a900af6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:57192/0 (socket says 172.21.15.96:57192) 2024-05-28T06:24:30.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.595+0000 7f1a8ea00640 1 -- 172.21.15.96:0/595040386 learned_addr learned my addr 172.21.15.96:0/595040386 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:24:30.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.595+0000 7f1a8e000640 1 -- 172.21.15.96:0/595040386 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1916889605 0 0) 0x7f1a90156f10 con 0x7f1a90156b30 2024-05-28T06:24:30.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.595+0000 7f1a86000640 1 --2- 172.21.15.96:0/595040386 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1a9006ee40 0x7f1a900ad230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:30.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.595+0000 7f1a8e000640 1 -- 172.21.15.96:0/595040386 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1a70003520 con 0x7f1a90156b30 2024-05-28T06:24:30.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.595+0000 7f1a8f400640 1 --2- 172.21.15.96:0/595040386 >> v2:172.21.15.123:3300/0 conn(0x7f1a9006f2c0 0x7f1a900ad770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:30.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.595+0000 7f1a8e000640 1 -- 172.21.15.96:0/595040386 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1916581032 0 0) 0x7f1a70003520 con 0x7f1a90156b30 2024-05-28T06:24:30.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.595+0000 7f1a8e000640 1 -- 172.21.15.96:0/595040386 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1a9006ee40 msgr2=0x7f1a900ad230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:30.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.595+0000 7f1a8e000640 1 --2- 172.21.15.96:0/595040386 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1a9006ee40 0x7f1a900ad230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:30.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.595+0000 7f1a8e000640 1 -- 172.21.15.96:0/595040386 >> v2:172.21.15.123:3300/0 conn(0x7f1a9006f2c0 msgr2=0x7f1a900ad770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:30.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.595+0000 7f1a8e000640 1 --2- 172.21.15.96:0/595040386 >> v2:172.21.15.123:3300/0 conn(0x7f1a9006f2c0 0x7f1a900ad770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:30.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.595+0000 7f1a8e000640 1 -- 172.21.15.96:0/595040386 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a9006e360 con 0x7f1a90156b30 2024-05-28T06:24:30.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.596+0000 7f1a8e000640 1 -- 172.21.15.96:0/595040386 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f1a80001670 con 0x7f1a90156b30 2024-05-28T06:24:30.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.596+0000 7f1a8e000640 1 -- 172.21.15.96:0/595040386 >> v1:172.21.15.96:6789/0 conn(0x7f1a90156b30 legacy=0x7f1a900af6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:30.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.596+0000 7f1a8e000640 1 --2- 172.21.15.96:0/595040386 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1a700039b0 0x7f1a70003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:30.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.596+0000 7f1a8e000640 1 --2- 172.21.15.96:0/595040386 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a70004800 0x7f1a70004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:30.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.596+0000 7f1a8e000640 1 --2- 172.21.15.96:0/595040386 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a700051c0 0x7f1a7000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:30.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.596+0000 7f1a8e000640 1 -- 172.21.15.96:0/595040386 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f1a9006dda0 con 0x7f1a70004800 2024-05-28T06:24:30.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.596+0000 7f1a8e000640 1 -- 172.21.15.96:0/595040386 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f1a9006e080 con 0x7f1a700051c0 2024-05-28T06:24:30.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.596+0000 7f1a8e000640 1 -- 172.21.15.96:0/595040386 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f1a7000c090 con 0x7f1a700039b0 2024-05-28T06:24:30.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.596+0000 7f1a86000640 1 --2- 172.21.15.96:0/595040386 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a70004800 0x7f1a70004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:30.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.596+0000 7f1a8ea00640 1 --2- 172.21.15.96:0/595040386 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1a700039b0 0x7f1a70003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:30.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.596+0000 7f1a8f400640 1 --2- 172.21.15.96:0/595040386 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a700051c0 0x7f1a7000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:30.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.596+0000 7f1a86000640 1 -- 172.21.15.96:0/595040386 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1a700039b0 msgr2=0x7f1a70003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:30.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.596+0000 7f1a86000640 1 --2- 172.21.15.96:0/595040386 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1a700039b0 0x7f1a70003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:30.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.596+0000 7f1a86000640 1 -- 172.21.15.96:0/595040386 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a700051c0 msgr2=0x7f1a7000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:30.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.596+0000 7f1a86000640 1 --2- 172.21.15.96:0/595040386 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a700051c0 0x7f1a7000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:30.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.596+0000 7f1a86000640 1 -- 172.21.15.96:0/595040386 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a70003520 con 0x7f1a70004800 2024-05-28T06:24:30.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.596+0000 7f1a8f400640 1 --2- 172.21.15.96:0/595040386 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a700051c0 0x7f1a7000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:24:30.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.596+0000 7f1a8ea00640 1 --2- 172.21.15.96:0/595040386 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1a700039b0 0x7f1a70003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:24:30.595 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.597+0000 7f1a86000640 1 --2- 172.21.15.96:0/595040386 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a70004800 0x7f1a70004c80 secure :-1 s=READY pgs=2035 cs=0 l=1 rev1=1 crypto rx=0x7f1a7800b900 tx=0x7f1a70005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61db9f6fac16e53 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:30.595 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.598+0000 7f1a8e000640 1 -- 172.21.15.96:0/595040386 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a78030070 con 0x7f1a70004800 2024-05-28T06:24:30.595 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.598+0000 7f1a8e000640 1 -- 172.21.15.96:0/595040386 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a7802ace0 con 0x7f1a70004800 2024-05-28T06:24:30.595 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.598+0000 7f1a8e000640 1 -- 172.21.15.96:0/595040386 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a7800b550 con 0x7f1a70004800 2024-05-28T06:24:30.595 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.598+0000 7f1a8fe00640 1 -- 172.21.15.96:0/595040386 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a70004800 msgr2=0x7f1a70004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:30.595 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.598+0000 7f1a8fe00640 1 --2- 172.21.15.96:0/595040386 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a70004800 0x7f1a70004c80 secure :-1 s=READY pgs=2035 cs=0 l=1 rev1=1 crypto rx=0x7f1a7800b900 tx=0x7f1a70005830 comp rx=0 tx=0).stop 2024-05-28T06:24:30.595 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.598+0000 7f1a8f400640 1 -- 172.21.15.96:0/595040386 reap_dead start 2024-05-28T06:24:30.595 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.598+0000 7f1a8fe00640 1 -- 172.21.15.96:0/595040386 shutdown_connections 2024-05-28T06:24:30.595 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.599+0000 7f1a8fe00640 1 -- 172.21.15.96:0/595040386 >> 172.21.15.96:0/595040386 conn(0x7f1a900b1fc0 msgr2=0x7f1a900b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:30.596 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.599+0000 7f1a8fe00640 1 -- 172.21.15.96:0/595040386 shutdown_connections 2024-05-28T06:24:30.596 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.599+0000 7f1a8fe00640 1 -- 172.21.15.96:0/595040386 wait complete. 2024-05-28T06:24:30.596 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.600+0000 7f1a8fe00640 1 Processor -- start 2024-05-28T06:24:30.596 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.601+0000 7f1a8fe00640 1 -- start start 2024-05-28T06:24:30.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.601+0000 7f1a8fe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a70004800 0x7f1a9015c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:30.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.601+0000 7f1a8fe00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1a900b13a0 0x7f1a90156f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:30.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.601+0000 7f1a8fe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a90157450 0x7f1a900b00b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:30.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.601+0000 7f1a8fe00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f1a900b2db0 con 0x7f1a90157450 2024-05-28T06:24:30.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.601+0000 7f1a8fe00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f1a900b2c30 con 0x7f1a70004800 2024-05-28T06:24:30.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.601+0000 7f1a8fe00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f1a900ad230 con 0x7f1a900b13a0 2024-05-28T06:24:30.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.602+0000 7f1a8f400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a90157450 0x7f1a900b00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:30.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.602+0000 7f1a8f400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a90157450 0x7f1a900b00b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:46994/0 (socket says 172.21.15.96:46994) 2024-05-28T06:24:30.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.602+0000 7f1a8f400640 1 -- 172.21.15.96:0/2204464384 learned_addr learned my addr 172.21.15.96:0/2204464384 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:24:30.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.602+0000 7f1a8f400640 1 -- 172.21.15.96:0/2204464384 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1a900b13a0 msgr2=0x7f1a90156f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:30.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.602+0000 7f1a8f400640 1 --2- 172.21.15.96:0/2204464384 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1a900b13a0 0x7f1a90156f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:30.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.602+0000 7f1a8f400640 1 -- 172.21.15.96:0/2204464384 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a70004800 msgr2=0x7f1a9015c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:30.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.602+0000 7f1a8f400640 1 --2- 172.21.15.96:0/2204464384 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a70004800 0x7f1a9015c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:30.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.602+0000 7f1a8f400640 1 -- 172.21.15.96:0/2204464384 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a901f23d0 con 0x7f1a90157450 2024-05-28T06:24:30.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.602+0000 7f1a8f400640 1 --2- 172.21.15.96:0/2204464384 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a90157450 0x7f1a900b00b0 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 crypto rx=0x7f1a90076c10 tx=0x7f1a88009cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:30.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.603+0000 7f1a87e00640 1 -- 172.21.15.96:0/2204464384 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a88019070 con 0x7f1a90157450 2024-05-28T06:24:30.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.603+0000 7f1a87e00640 1 -- 172.21.15.96:0/2204464384 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a88007d50 con 0x7f1a90157450 2024-05-28T06:24:30.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.603+0000 7f1a87e00640 1 -- 172.21.15.96:0/2204464384 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a88014640 con 0x7f1a90157450 2024-05-28T06:24:30.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.603+0000 7f1a8fe00640 1 -- 172.21.15.96:0/2204464384 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a900ccb70 con 0x7f1a90157450 2024-05-28T06:24:30.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.603+0000 7f1a8fe00640 1 -- 172.21.15.96:0/2204464384 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a900cd130 con 0x7f1a90157450 2024-05-28T06:24:30.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.605+0000 7f1a87e00640 1 -- 172.21.15.96:0/2204464384 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 42) ==== 93590+0+0 (secure 0 0 0) 0x7f1a8801c070 con 0x7f1a90157450 2024-05-28T06:24:30.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.606+0000 7f1a87e00640 1 --2- 172.21.15.96:0/2204464384 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1a6c06e510 0x7f1a6c0709d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:30.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.606+0000 7f1a87e00640 1 -- 172.21.15.96:0/2204464384 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 5765+0+0 (secure 0 0 0) 0x7f1a8808a460 con 0x7f1a90157450 2024-05-28T06:24:30.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.607+0000 7f1a98c03740 1 Processor -- start 2024-05-28T06:24:30.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.607+0000 7f1a8ea00640 1 --2- 172.21.15.96:0/2204464384 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1a6c06e510 0x7f1a6c0709d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:30.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.607+0000 7f1a98c03740 1 -- start start 2024-05-28T06:24:30.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.608+0000 7f1a98c03740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f3a0c7e490 0x55f3a0c81c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:30.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.608+0000 7f1a98c03740 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x55f3a0c82150 0x55f3a0c9e090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:30.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.608+0000 7f1a98c03740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f3a0c9e5d0 0x55f3a0c9ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:30.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.608+0000 7f1a98c03740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55f3a0bbbb10 con 0x55f3a0c9e5d0 2024-05-28T06:24:30.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.608+0000 7f1a98c03740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55f3a0b622b0 con 0x55f3a0c7e490 2024-05-28T06:24:30.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.608+0000 7f1a98c03740 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x55f3a0bb7ad0 con 0x55f3a0c82150 2024-05-28T06:24:30.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.608+0000 7f1a8ea00640 1 --2- 172.21.15.96:0/2204464384 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1a6c06e510 0x7f1a6c0709d0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f1a80001eb0 tx=0x7f1a80009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:30.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.608+0000 7f1a8f400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f3a0c9e5d0 0x55f3a0c9ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:30.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.608+0000 7f1a8f400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f3a0c9e5d0 0x55f3a0c9ea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47004/0 (socket says 172.21.15.96:47004) 2024-05-28T06:24:30.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.608+0000 7f1a8f400640 1 -- 172.21.15.96:0/4144882977 learned_addr learned my addr 172.21.15.96:0/4144882977 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:24:30.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.608+0000 7f1a86000640 1 --2- 172.21.15.96:0/4144882977 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x55f3a0c82150 0x55f3a0c9e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:30.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.608+0000 7f1a8ea00640 1 --2- 172.21.15.96:0/4144882977 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f3a0c7e490 0x55f3a0c81c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:30.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.608+0000 7f1a8f400640 1 -- 172.21.15.96:0/4144882977 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x55f3a0c82150 msgr2=0x55f3a0c9e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:30.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.609+0000 7f1a8f400640 1 --2- 172.21.15.96:0/4144882977 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x55f3a0c82150 0x55f3a0c9e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:30.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.609+0000 7f1a8f400640 1 -- 172.21.15.96:0/4144882977 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f3a0c7e490 msgr2=0x55f3a0c81c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:30.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.609+0000 7f1a8f400640 1 --2- 172.21.15.96:0/4144882977 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f3a0c7e490 0x55f3a0c81c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:30.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.609+0000 7f1a8f400640 1 -- 172.21.15.96:0/4144882977 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f3a08ce2d0 con 0x55f3a0c9e5d0 2024-05-28T06:24:30.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.609+0000 7f1a8f400640 1 --2- 172.21.15.96:0/4144882977 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f3a0c9e5d0 0x55f3a0c9ea30 secure :-1 s=READY pgs=2037 cs=0 l=1 rev1=1 crypto rx=0x7f1a88059c30 tx=0x7f1a88009f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c96efd02059087b0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:30.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.609+0000 7f1a7f400640 1 -- 172.21.15.96:0/4144882977 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a880720d0 con 0x55f3a0c9e5d0 2024-05-28T06:24:30.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.609+0000 7f1a7f400640 1 -- 172.21.15.96:0/4144882977 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a88056310 con 0x55f3a0c9e5d0 2024-05-28T06:24:30.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.609+0000 7f1a7f400640 1 -- 172.21.15.96:0/4144882977 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a88062560 con 0x55f3a0c9e5d0 2024-05-28T06:24:30.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.610+0000 7f1a98c03740 1 -- 172.21.15.96:0/4144882977 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f3a0c9e5d0 msgr2=0x55f3a0c9ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:30.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.610+0000 7f1a98c03740 1 --2- 172.21.15.96:0/4144882977 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f3a0c9e5d0 0x55f3a0c9ea30 secure :-1 s=READY pgs=2037 cs=0 l=1 rev1=1 crypto rx=0x7f1a88059c30 tx=0x7f1a88009f70 comp rx=0 tx=0).stop 2024-05-28T06:24:30.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.610+0000 7f1a98c03740 1 -- 172.21.15.96:0/4144882977 shutdown_connections 2024-05-28T06:24:30.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.610+0000 7f1a98c03740 1 --2- 172.21.15.96:0/4144882977 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f3a0c9e5d0 0x55f3a0c9ea30 unknown :-1 s=CLOSED pgs=2037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:30.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.610+0000 7f1a98c03740 1 --2- 172.21.15.96:0/4144882977 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x55f3a0c82150 0x55f3a0c9e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:30.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.610+0000 7f1a98c03740 1 --2- 172.21.15.96:0/4144882977 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f3a0c7e490 0x55f3a0c81c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:30.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.610+0000 7f1a98c03740 1 -- 172.21.15.96:0/4144882977 >> 172.21.15.96:0/4144882977 conn(0x55f3a0c54470 msgr2=0x55f3a0c55470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:30.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.610+0000 7f1a98c03740 1 -- 172.21.15.96:0/4144882977 shutdown_connections 2024-05-28T06:24:30.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.610+0000 7f1a98c03740 1 -- 172.21.15.96:0/4144882977 wait complete. 2024-05-28T06:24:30.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.611+0000 7f1a98c03740 1 -- start start 2024-05-28T06:24:30.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.611+0000 7f1a98c03740 1 Processor -- start 2024-05-28T06:24:30.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.612+0000 7f1a98c03740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f3a0c82150 0x55f3a0c6f690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:30.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.612+0000 7f1a98c03740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f3a0c9e5d0 0x55f3a0cb08e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:30.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.612+0000 7f1a98c03740 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x55f3a0c99d30 0x55f3a0c512b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:30.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.612+0000 7f1a98c03740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55f3a0c522a0 con 0x55f3a0c9e5d0 2024-05-28T06:24:30.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.612+0000 7f1a98c03740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55f3a0b75570 con 0x55f3a0c82150 2024-05-28T06:24:30.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.612+0000 7f1a98c03740 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x55f3a0c52580 con 0x55f3a0c99d30 2024-05-28T06:24:30.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.612+0000 7f1a98c03740 10 client.0 _ll_register_callbacks cb 0x55f3a0b43fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-28T06:24:30.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.612+0000 7f1a86000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f3a0c9e5d0 0x55f3a0cb08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:30.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.612+0000 7f1a86000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f3a0c9e5d0 0x55f3a0cb08e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47014/0 (socket says 172.21.15.96:47014) 2024-05-28T06:24:30.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.612+0000 7f1a86000640 1 -- 172.21.15.96:0/2115561205 learned_addr learned my addr 172.21.15.96:0/2115561205 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:24:30.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.613+0000 7f1a86000640 1 -- 172.21.15.96:0/2115561205 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x55f3a0c99d30 msgr2=0x55f3a0c512b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:30.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.613+0000 7f1a86000640 1 --2- 172.21.15.96:0/2115561205 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x55f3a0c99d30 0x55f3a0c512b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:30.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.613+0000 7f1a86000640 1 -- 172.21.15.96:0/2115561205 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f3a0c82150 msgr2=0x55f3a0c6f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:30.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.613+0000 7f1a8ea00640 1 --2- 172.21.15.96:0/2115561205 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f3a0c82150 0x55f3a0c6f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:30.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.613+0000 7f1a86000640 1 --2- 172.21.15.96:0/2115561205 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f3a0c82150 0x55f3a0c6f690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:30.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.613+0000 7f1a86000640 1 -- 172.21.15.96:0/2115561205 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f3a09986b0 con 0x55f3a0c9e5d0 2024-05-28T06:24:30.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.613+0000 7f1a86000640 1 --2- 172.21.15.96:0/2115561205 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f3a0c9e5d0 0x55f3a0cb08e0 secure :-1 s=READY pgs=2038 cs=0 l=1 rev1=1 crypto rx=0x7f1a7800ba20 tx=0x7f1a78004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6094a1a979b1e129 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:30.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.613+0000 7f1a77e00640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-05-28T06:24:30.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.613+0000 7f1a77e00640 1 -- 172.21.15.96:0/2115561205 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a78042020 con 0x55f3a0c9e5d0 2024-05-28T06:24:30.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.613+0000 7f1a77e00640 1 -- 172.21.15.96:0/2115561205 <== mon.0 v2:172.21.15.96:3300/0 2 ==== osd_map(130..130 src has 1..130) ==== 5765+0+0 (secure 0 0 0) 0x7f1a78003610 con 0x55f3a0c9e5d0 2024-05-28T06:24:30.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.613+0000 7f1a98c03740 1 -- 172.21.15.96:0/2115561205 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1a88056310 con 0x55f3a0c9e5d0 2024-05-28T06:24:30.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.613+0000 7f1a77e00640 1 -- 172.21.15.96:0/2115561205 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) ==== 24+0+0 (secure 0 0 0) 0x7f1a7804c050 con 0x55f3a0c9e5d0 2024-05-28T06:24:30.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.613+0000 7f1a98c03740 10 client.9996 fetch_fsmap learned FSMap version 82 2024-05-28T06:24:30.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.614+0000 7f1a98c03740 1 -- 172.21.15.96:0/2115561205 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=2+}) -- 0x55f3a0c99530 con 0x55f3a0c9e5d0 2024-05-28T06:24:30.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.614+0000 7f1a77e00640 1 -- 172.21.15.96:0/2115561205 <== mon.0 v2:172.21.15.96:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a88056310 con 0x55f3a0c9e5d0 2024-05-28T06:24:30.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.614+0000 7f1a77e00640 1 -- 172.21.15.96:0/2115561205 <== mon.0 v2:172.21.15.96:3300/0 5 ==== fsmap(e 82) ==== 3494+0+0 (secure 0 0 0) 0x7f1a7804f020 con 0x55f3a0c9e5d0 2024-05-28T06:24:30.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.614+0000 7f1a98c03740 10 client.9996 fetch_fsmap finished waiting for FSMap version 82 2024-05-28T06:24:30.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.614+0000 7f1a98c03740 10 client.9996 resolve_mds: resolved name 'a' to daemon mds.a 2024-05-28T06:24:30.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.614+0000 7f1a98c03740 20 client.9996 populate_metadata read hostname 'smithi096' 2024-05-28T06:24:30.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.614+0000 7f1a98c03740 1 --2- 172.21.15.96:0/2115561205 >> [v2:172.21.15.96:6834/1720339900,v1:172.21.15.96:6835/1720339900] conn(0x55f3a0c83160 0x55f3a0c7c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:30.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.614+0000 7f1a98c03740 4 client.9996 mds_command: new command op to 9849 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-28T06:24:30.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.614+0000 7f1a98c03740 1 -- 172.21.15.96:0/2115561205 --> [v2:172.21.15.96:6834/1720339900,v1:172.21.15.96:6835/1720339900] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f3a0bc9530 con 0x55f3a0c83160 2024-05-28T06:24:30.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.614+0000 7f1a8ea00640 1 --2- 172.21.15.96:0/2115561205 >> [v2:172.21.15.96:6834/1720339900,v1:172.21.15.96:6835/1720339900] conn(0x55f3a0c83160 0x55f3a0c7c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:30.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.615+0000 7f1a8ea00640 1 --2- 172.21.15.96:0/2115561205 >> [v2:172.21.15.96:6834/1720339900,v1:172.21.15.96:6835/1720339900] conn(0x55f3a0c83160 0x55f3a0c7c7c0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=76245385bd3f782a server_cookie=6746305f450c4aa3 in_seq=0 out_seq=0 2024-05-28T06:24:30.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.615+0000 7f1a77e00640 10 client.9996 ms_handle_connect on v2:172.21.15.96:6834/1720339900 2024-05-28T06:24:30.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.616+0000 7f1a77e00640 1 -- 172.21.15.96:0/2115561205 <== mds.0 v2:172.21.15.96:6834/1720339900 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f3a0bc9530 con 0x55f3a0c83160 2024-05-28T06:24:30.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.616+0000 7f1a77e00640 10 client.9996 handle_command_reply: tid=0 2024-05-28T06:24:30.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.616+0000 7f1a98c03740 1 client.9996 shutdown 2024-05-28T06:24:30.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.616+0000 7f1a98c03740 1 -- 172.21.15.96:0/2115561205 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f3a0c9e5d0 msgr2=0x55f3a0cb08e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:30.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.616+0000 7f1a98c03740 1 --2- 172.21.15.96:0/2115561205 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f3a0c9e5d0 0x55f3a0cb08e0 secure :-1 s=READY pgs=2038 cs=0 l=1 rev1=1 crypto rx=0x7f1a7800ba20 tx=0x7f1a78004730 comp rx=0 tx=0).stop 2024-05-28T06:24:30.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.616+0000 7f1a98c03740 1 -- 172.21.15.96:0/2115561205 shutdown_connections 2024-05-28T06:24:30.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.616+0000 7f1a98c03740 1 --2- 172.21.15.96:0/2115561205 >> [v2:172.21.15.96:6834/1720339900,v1:172.21.15.96:6835/1720339900] conn(0x55f3a0c83160 0x55f3a0c7c7c0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:30.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.616+0000 7f1a98c03740 1 --2- 172.21.15.96:0/2115561205 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x55f3a0c99d30 0x55f3a0c512b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:30.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.616+0000 7f1a98c03740 1 --2- 172.21.15.96:0/2115561205 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f3a0c9e5d0 0x55f3a0cb08e0 unknown :-1 s=CLOSED pgs=2038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:30.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.616+0000 7f1a98c03740 1 --2- 172.21.15.96:0/2115561205 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f3a0c82150 0x55f3a0c6f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:30.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.616+0000 7f1a77e00640 0 client.9996 ms_handle_reset on v2:172.21.15.96:6834/1720339900 2024-05-28T06:24:30.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.616+0000 7f1a98c03740 1 -- 172.21.15.96:0/2115561205 >> 172.21.15.96:0/2115561205 conn(0x55f3a0c7e490 msgr2=0x55f3a0c9cbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:30.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.617+0000 7f1a98c03740 1 -- 172.21.15.96:0/2115561205 shutdown_connections 2024-05-28T06:24:30.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.617+0000 7f1a98c03740 1 -- 172.21.15.96:0/2115561205 wait complete. 2024-05-28T06:24:30.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.617+0000 7f1a98c03740 20 client.9996 trim_cache size 0 max 16384 2024-05-28T06:24:30.631 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.636+0000 7f1a98c03740 1 Processor -- start 2024-05-28T06:24:30.631 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.636+0000 7f1a98c03740 1 -- start start 2024-05-28T06:24:30.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.637+0000 7f1a98c03740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f3a0c82150 0x55f3a0c511b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:30.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.637+0000 7f1a98c03740 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x55f3a0c99d30 0x55f3a0c9cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:30.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.637+0000 7f1a98c03740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f3a0c9e5d0 0x55f3a0c57d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:30.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.637+0000 7f1a98c03740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f1a901fe280 con 0x55f3a0c82150 2024-05-28T06:24:30.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.637+0000 7f1a98c03740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f1a90159880 con 0x55f3a0c9e5d0 2024-05-28T06:24:30.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.637+0000 7f1a98c03740 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f1a901fdd10 con 0x55f3a0c99d30 2024-05-28T06:24:30.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.637+0000 7f1a8ea00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f3a0c82150 0x55f3a0c511b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:30.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.637+0000 7f1a8ea00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f3a0c82150 0x55f3a0c511b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47024/0 (socket says 172.21.15.96:47024) 2024-05-28T06:24:30.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.637+0000 7f1a8ea00640 1 -- 172.21.15.96:0/1418459715 learned_addr learned my addr 172.21.15.96:0/1418459715 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:24:30.635 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.638+0000 7f1a8ea00640 1 -- 172.21.15.96:0/1418459715 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x55f3a0c99d30 msgr2=0x55f3a0c9cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:30.635 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.638+0000 7f1a8ea00640 1 --2- 172.21.15.96:0/1418459715 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x55f3a0c99d30 0x55f3a0c9cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:30.635 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.638+0000 7f1a8f400640 1 --2- 172.21.15.96:0/1418459715 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f3a0c9e5d0 0x55f3a0c57d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:30.635 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.638+0000 7f1a8ea00640 1 -- 172.21.15.96:0/1418459715 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f3a0c9e5d0 msgr2=0x55f3a0c57d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:30.635 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.638+0000 7f1a8ea00640 1 --2- 172.21.15.96:0/1418459715 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f3a0c9e5d0 0x55f3a0c57d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:30.635 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.638+0000 7f1a8ea00640 1 -- 172.21.15.96:0/1418459715 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f3a0bb0c40 con 0x55f3a0c82150 2024-05-28T06:24:30.635 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.638+0000 7f1a8ea00640 1 --2- 172.21.15.96:0/1418459715 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f3a0c82150 0x55f3a0c511b0 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 crypto rx=0x7f1a80000ff0 tx=0x7f1a8000b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e348fc3e8c30a0d7 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:30.635 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.638+0000 7f1a7ea00640 1 -- 172.21.15.96:0/1418459715 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a80028070 con 0x55f3a0c82150 2024-05-28T06:24:30.635 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.638+0000 7f1a7ea00640 1 -- 172.21.15.96:0/1418459715 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a8000dca0 con 0x55f3a0c82150 2024-05-28T06:24:30.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.638+0000 7f1a7ea00640 1 -- 172.21.15.96:0/1418459715 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a800076d0 con 0x55f3a0c82150 2024-05-28T06:24:30.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.639+0000 7f1a98c03740 1 -- 172.21.15.96:0/1418459715 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f3a0c82150 msgr2=0x55f3a0c511b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:30.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.639+0000 7f1a98c03740 1 --2- 172.21.15.96:0/1418459715 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f3a0c82150 0x55f3a0c511b0 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 crypto rx=0x7f1a80000ff0 tx=0x7f1a8000b000 comp rx=0 tx=0).stop 2024-05-28T06:24:30.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.639+0000 7f1a98c03740 1 -- 172.21.15.96:0/1418459715 shutdown_connections 2024-05-28T06:24:30.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.639+0000 7f1a98c03740 1 --2- 172.21.15.96:0/1418459715 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f3a0c9e5d0 0x55f3a0c57d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:30.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.639+0000 7f1a98c03740 1 --2- 172.21.15.96:0/1418459715 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x55f3a0c99d30 0x55f3a0c9cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:30.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.639+0000 7f1a98c03740 1 --2- 172.21.15.96:0/1418459715 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f3a0c82150 0x55f3a0c511b0 unknown :-1 s=CLOSED pgs=2039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:30.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.639+0000 7f1a98c03740 1 -- 172.21.15.96:0/1418459715 >> 172.21.15.96:0/1418459715 conn(0x55f3a0c7e490 msgr2=0x55f3a0c9f000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:30.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.639+0000 7f1a98c03740 1 -- 172.21.15.96:0/1418459715 shutdown_connections 2024-05-28T06:24:30.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.639+0000 7f1a98c03740 1 -- 172.21.15.96:0/1418459715 wait complete. 2024-05-28T06:24:30.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.640+0000 7f1a98c03740 1 -- start start 2024-05-28T06:24:30.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.640+0000 7f1a98c03740 1 Processor -- start 2024-05-28T06:24:30.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.641+0000 7f1a98c03740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f3a0c82150 0x55f3a0c57d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:30.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.641+0000 7f1a98c03740 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f3a0c99d30 0x55f3a0c582b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:30.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.641+0000 7f1a98c03740 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x55f3a0c9e5d0 0x55f3a0d1a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:30.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.641+0000 7f1a98c03740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55f3a0c82fe0 con 0x55f3a0c82150 2024-05-28T06:24:30.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.641+0000 7f1a98c03740 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55f3a0c83a80 con 0x55f3a0c99d30 2024-05-28T06:24:30.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.641+0000 7f1a98c03740 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x55f3a0c83590 con 0x55f3a0c9e5d0 2024-05-28T06:24:30.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.641+0000 7f1a98c03740 10 client.0 _ll_register_callbacks cb 0x7f1a70004330 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-28T06:24:30.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.641+0000 7f1a8ea00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f3a0c82150 0x55f3a0c57d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:30.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.641+0000 7f1a8ea00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f3a0c82150 0x55f3a0c57d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47034/0 (socket says 172.21.15.96:47034) 2024-05-28T06:24:30.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.641+0000 7f1a8ea00640 1 -- 172.21.15.96:0/895923285 learned_addr learned my addr 172.21.15.96:0/895923285 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:24:30.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.642+0000 7f1a8ea00640 1 -- 172.21.15.96:0/895923285 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x55f3a0c9e5d0 msgr2=0x55f3a0d1a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:30.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.642+0000 7f1a8ea00640 1 --2- 172.21.15.96:0/895923285 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x55f3a0c9e5d0 0x55f3a0d1a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:30.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.642+0000 7f1a8ea00640 1 -- 172.21.15.96:0/895923285 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f3a0c99d30 msgr2=0x55f3a0c582b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:30.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.642+0000 7f1a8ea00640 1 --2- 172.21.15.96:0/895923285 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f3a0c99d30 0x55f3a0c582b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:30.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.642+0000 7f1a8ea00640 1 -- 172.21.15.96:0/895923285 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f3a0c97e10 con 0x55f3a0c82150 2024-05-28T06:24:30.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.642+0000 7f1a8ea00640 1 --2- 172.21.15.96:0/895923285 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f3a0c82150 0x55f3a0c57d70 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 crypto rx=0x7f1a80007780 tx=0x7f1a80000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e686b44d9150a71 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:30.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.642+0000 7f1a77e00640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-05-28T06:24:30.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.642+0000 7f1a77e00640 1 -- 172.21.15.96:0/895923285 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a80032060 con 0x55f3a0c82150 2024-05-28T06:24:30.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.642+0000 7f1a77e00640 1 -- 172.21.15.96:0/895923285 <== mon.0 v2:172.21.15.96:3300/0 2 ==== osd_map(130..130 src has 1..130) ==== 5765+0+0 (secure 0 0 0) 0x7f1a80004bf0 con 0x55f3a0c82150 2024-05-28T06:24:30.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.642+0000 7f1a98c03740 1 -- 172.21.15.96:0/895923285 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1a8000dca0 con 0x55f3a0c82150 2024-05-28T06:24:30.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.642+0000 7f1a77e00640 1 -- 172.21.15.96:0/895923285 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) ==== 24+0+0 (secure 0 0 0) 0x7f1a80028070 con 0x55f3a0c82150 2024-05-28T06:24:30.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.643+0000 7f1a98c03740 10 client.10002 fetch_fsmap learned FSMap version 82 2024-05-28T06:24:30.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.643+0000 7f1a98c03740 1 -- 172.21.15.96:0/895923285 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=2+}) -- 0x55f3a0c97900 con 0x55f3a0c82150 2024-05-28T06:24:30.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.643+0000 7f1a77e00640 1 -- 172.21.15.96:0/895923285 <== mon.0 v2:172.21.15.96:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a8000dca0 con 0x55f3a0c82150 2024-05-28T06:24:30.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.643+0000 7f1a77e00640 1 -- 172.21.15.96:0/895923285 <== mon.0 v2:172.21.15.96:3300/0 5 ==== fsmap(e 82) ==== 3494+0+0 (secure 0 0 0) 0x7f1a80005040 con 0x55f3a0c82150 2024-05-28T06:24:30.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.643+0000 7f1a98c03740 10 client.10002 fetch_fsmap finished waiting for FSMap version 82 2024-05-28T06:24:30.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.643+0000 7f1a98c03740 10 client.10002 resolve_mds: resolved name 'a' to daemon mds.a 2024-05-28T06:24:30.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.643+0000 7f1a98c03740 20 client.10002 populate_metadata read hostname 'smithi096' 2024-05-28T06:24:30.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.643+0000 7f1a98c03740 1 --2- 172.21.15.96:0/895923285 >> [v2:172.21.15.96:6834/1720339900,v1:172.21.15.96:6835/1720339900] conn(0x55f3a0cae9c0 0x55f3a0caeda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:30.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.643+0000 7f1a98c03740 4 client.10002 mds_command: new command op to 9849 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-28T06:24:30.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.643+0000 7f1a98c03740 1 -- 172.21.15.96:0/895923285 --> [v2:172.21.15.96:6834/1720339900,v1:172.21.15.96:6835/1720339900] -- command(tid 0: {"prefix": "status"}) -- 0x55f3a0bc33a0 con 0x55f3a0cae9c0 2024-05-28T06:24:30.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.643+0000 7f1a86000640 1 --2- 172.21.15.96:0/895923285 >> [v2:172.21.15.96:6834/1720339900,v1:172.21.15.96:6835/1720339900] conn(0x55f3a0cae9c0 0x55f3a0caeda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:30.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.644+0000 7f1a86000640 1 --2- 172.21.15.96:0/895923285 >> [v2:172.21.15.96:6834/1720339900,v1:172.21.15.96:6835/1720339900] conn(0x55f3a0cae9c0 0x55f3a0caeda0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=252ed48a364cb754 server_cookie=51d22d84340d0eae in_seq=0 out_seq=0 2024-05-28T06:24:30.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.644+0000 7f1a77e00640 10 client.10002 ms_handle_connect on v2:172.21.15.96:6834/1720339900 2024-05-28T06:24:30.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.644+0000 7f1a77e00640 1 -- 172.21.15.96:0/895923285 <== mds.0 v2:172.21.15.96:6834/1720339900 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55f3a0bc33a0 con 0x55f3a0cae9c0 2024-05-28T06:24:30.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.644+0000 7f1a77e00640 10 client.10002 handle_command_reply: tid=0 2024-05-28T06:24:30.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.644+0000 7f1a98c03740 1 client.10002 shutdown 2024-05-28T06:24:30.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.644+0000 7f1a98c03740 1 -- 172.21.15.96:0/895923285 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f3a0c82150 msgr2=0x55f3a0c57d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:30.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.644+0000 7f1a98c03740 1 --2- 172.21.15.96:0/895923285 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f3a0c82150 0x55f3a0c57d70 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 crypto rx=0x7f1a80007780 tx=0x7f1a80000ed0 comp rx=0 tx=0).stop 2024-05-28T06:24:30.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.644+0000 7f1a98c03740 1 -- 172.21.15.96:0/895923285 shutdown_connections 2024-05-28T06:24:30.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.644+0000 7f1a98c03740 1 --2- 172.21.15.96:0/895923285 >> [v2:172.21.15.96:6834/1720339900,v1:172.21.15.96:6835/1720339900] conn(0x55f3a0cae9c0 0x55f3a0caeda0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:30.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.644+0000 7f1a98c03740 1 --2- 172.21.15.96:0/895923285 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x55f3a0c9e5d0 0x55f3a0d1a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:30.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.644+0000 7f1a98c03740 1 --2- 172.21.15.96:0/895923285 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f3a0c99d30 0x55f3a0c582b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:30.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.644+0000 7f1a77e00640 0 client.10002 ms_handle_reset on v2:172.21.15.96:6834/1720339900 2024-05-28T06:24:30.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.644+0000 7f1a98c03740 1 --2- 172.21.15.96:0/895923285 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f3a0c82150 0x55f3a0c57d70 unknown :-1 s=CLOSED pgs=2040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:30.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.644+0000 7f1a98c03740 1 -- 172.21.15.96:0/895923285 >> 172.21.15.96:0/895923285 conn(0x55f3a0c7e490 msgr2=0x55f3a0c9cbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:30.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.645+0000 7f1a98c03740 1 -- 172.21.15.96:0/895923285 shutdown_connections 2024-05-28T06:24:30.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.645+0000 7f1a98c03740 1 -- 172.21.15.96:0/895923285 wait complete. 2024-05-28T06:24:30.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.645+0000 7f1a98c03740 20 client.10002 trim_cache size 0 max 16384 2024-05-28T06:24:30.647 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:24:30.647 INFO:teuthology.orchestra.run.smithi096.stdout: "cluster_fsid": "e4a831b5-e6d1-47a7-96cb-bf85154ee0d2", 2024-05-28T06:24:30.647 INFO:teuthology.orchestra.run.smithi096.stdout: "whoami": 0, 2024-05-28T06:24:30.648 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 9849, 2024-05-28T06:24:30.648 INFO:teuthology.orchestra.run.smithi096.stdout: "want_state": "up:active", 2024-05-28T06:24:30.648 INFO:teuthology.orchestra.run.smithi096.stdout: "state": "up:active", 2024-05-28T06:24:30.648 INFO:teuthology.orchestra.run.smithi096.stdout: "fs_name": "cephfs", 2024-05-28T06:24:30.648 INFO:teuthology.orchestra.run.smithi096.stdout: "rank_uptime": 6.6726437220000001, 2024-05-28T06:24:30.648 INFO:teuthology.orchestra.run.smithi096.stdout: "mdsmap_epoch": 82, 2024-05-28T06:24:30.648 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap_epoch": 130, 2024-05-28T06:24:30.648 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap_epoch_barrier": 130, 2024-05-28T06:24:30.648 INFO:teuthology.orchestra.run.smithi096.stdout: "uptime": 12.108165759 2024-05-28T06:24:30.648 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:24:30.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.645+0000 7f1a8fe00640 1 -- 172.21.15.96:0/2204464384 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1a6c06e510 msgr2=0x7f1a6c0709d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:30.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.645+0000 7f1a8fe00640 1 --2- 172.21.15.96:0/2204464384 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1a6c06e510 0x7f1a6c0709d0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f1a80001eb0 tx=0x7f1a80009040 comp rx=0 tx=0).stop 2024-05-28T06:24:30.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.645+0000 7f1a8fe00640 1 -- 172.21.15.96:0/2204464384 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a90157450 msgr2=0x7f1a900b00b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:30.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.645+0000 7f1a8fe00640 1 --2- 172.21.15.96:0/2204464384 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a90157450 0x7f1a900b00b0 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 crypto rx=0x7f1a90076c10 tx=0x7f1a88009cd0 comp rx=0 tx=0).stop 2024-05-28T06:24:30.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.645+0000 7f1a8fe00640 1 -- 172.21.15.96:0/2204464384 shutdown_connections 2024-05-28T06:24:30.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.645+0000 7f1a8fe00640 1 --2- 172.21.15.96:0/2204464384 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1a6c06e510 0x7f1a6c0709d0 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:30.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.645+0000 7f1a8fe00640 1 --2- 172.21.15.96:0/2204464384 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a90157450 0x7f1a900b00b0 unknown :-1 s=CLOSED pgs=2036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:30.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.645+0000 7f1a8fe00640 1 --2- 172.21.15.96:0/2204464384 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1a900b13a0 0x7f1a90156f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:30.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.645+0000 7f1a8fe00640 1 --2- 172.21.15.96:0/2204464384 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a70004800 0x7f1a9015c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:30.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.645+0000 7f1a8fe00640 1 -- 172.21.15.96:0/2204464384 >> 172.21.15.96:0/2204464384 conn(0x7f1a900b1fc0 msgr2=0x7f1a90154120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:30.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.646+0000 7f1a8fe00640 1 -- 172.21.15.96:0/2204464384 shutdown_connections 2024-05-28T06:24:30.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.646+0000 7f1a8fe00640 1 -- 172.21.15.96:0/2204464384 wait complete. 2024-05-28T06:24:30.658 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-05-28T06:24:30.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.866+0000 7f569b200640 1 Processor -- start 2024-05-28T06:24:30.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.866+0000 7f569b200640 1 -- start start 2024-05-28T06:24:30.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.867+0000 7f569b200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f56940d0590 0x7f56940d09f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:30.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.867+0000 7f569b200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f5694099090 0x7f56940994e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:30.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.867+0000 7f569b200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56940b51f0 con 0x7f56940b4e10 2024-05-28T06:24:30.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.867+0000 7f569b200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f56940d0f30 con 0x7f5694099090 2024-05-28T06:24:30.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.867+0000 7f569b200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f56940aef60 con 0x7f56940d0590 2024-05-28T06:24:30.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.868+0000 7f5693e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f56940b4e10 0x7f56940cde30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:57208/0 (socket says 172.21.15.96:57208) 2024-05-28T06:24:30.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.868+0000 7f5693e00640 1 -- 172.21.15.96:0/2792238980 learned_addr learned my addr 172.21.15.96:0/2792238980 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:24:30.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.868+0000 7f5699200640 1 --2- 172.21.15.96:0/2792238980 >> v2:172.21.15.96:3300/0 conn(0x7f5694099090 0x7f56940994e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:30.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.868+0000 7f5693400640 1 --2- 172.21.15.96:0/2792238980 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f56940d0590 0x7f56940d09f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:30.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.868+0000 7f5692a00640 1 -- 172.21.15.96:0/2792238980 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 79672563 0 0) 0x7f56940b51f0 con 0x7f56940b4e10 2024-05-28T06:24:30.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.868+0000 7f5692a00640 1 -- 172.21.15.96:0/2792238980 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5678003520 con 0x7f56940b4e10 2024-05-28T06:24:30.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.868+0000 7f5699200640 1 -- 172.21.15.96:0/2792238980 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f56940d0590 msgr2=0x7f56940d09f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:30.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.868+0000 7f5699200640 1 --2- 172.21.15.96:0/2792238980 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f56940d0590 0x7f56940d09f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:30.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.868+0000 7f5699200640 1 -- 172.21.15.96:0/2792238980 >> v1:172.21.15.96:6789/0 conn(0x7f56940b4e10 legacy=0x7f56940cde30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:30.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.868+0000 7f5699200640 1 -- 172.21.15.96:0/2792238980 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56940ad1d0 con 0x7f5694099090 2024-05-28T06:24:30.865 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.869+0000 7f5699200640 1 --2- 172.21.15.96:0/2792238980 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5694099090 0x7f56940994e0 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 crypto rx=0x7f568c009d90 tx=0x7f568c02a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6da3182b2684b622 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:30.865 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.870+0000 7f5692a00640 1 -- 172.21.15.96:0/2792238980 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f568c031600 con 0x7f5694099090 2024-05-28T06:24:30.865 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.870+0000 7f5692a00640 1 -- 172.21.15.96:0/2792238980 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f568c0317a0 con 0x7f5694099090 2024-05-28T06:24:30.865 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.870+0000 7f5692a00640 1 -- 172.21.15.96:0/2792238980 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f568c031b50 con 0x7f5694099090 2024-05-28T06:24:30.865 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.870+0000 7f569b200640 1 -- 172.21.15.96:0/2792238980 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5694099090 msgr2=0x7f56940994e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:30.865 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.870+0000 7f569b200640 1 --2- 172.21.15.96:0/2792238980 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5694099090 0x7f56940994e0 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 crypto rx=0x7f568c009d90 tx=0x7f568c02a5d0 comp rx=0 tx=0).stop 2024-05-28T06:24:30.865 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.870+0000 7f569b200640 1 -- 172.21.15.96:0/2792238980 shutdown_connections 2024-05-28T06:24:30.865 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.870+0000 7f569b200640 1 --2- 172.21.15.96:0/2792238980 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5694099090 0x7f56940994e0 unknown :-1 s=CLOSED pgs=2042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:30.865 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.870+0000 7f569b200640 1 --2- 172.21.15.96:0/2792238980 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f56940d0590 0x7f56940d09f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:30.866 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.870+0000 7f569b200640 1 -- 172.21.15.96:0/2792238980 >> 172.21.15.96:0/2792238980 conn(0x7f569406bae0 msgr2=0x7f569406bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:30.866 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.870+0000 7f569b200640 1 -- 172.21.15.96:0/2792238980 shutdown_connections 2024-05-28T06:24:30.866 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.871+0000 7f569b200640 1 -- 172.21.15.96:0/2792238980 wait complete. 2024-05-28T06:24:30.866 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.872+0000 7f569b200640 1 Processor -- start 2024-05-28T06:24:30.867 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.872+0000 7f569b200640 1 -- start start 2024-05-28T06:24:30.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.873+0000 7f569b200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5694099090 0x7f5694098720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:30.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.873+0000 7f569b200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f56940b4e10 0x7f56941512a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:30.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.873+0000 7f569b200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f56940d0590 0x7f569415ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:30.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.873+0000 7f569b200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f56940ce540 con 0x7f5694099090 2024-05-28T06:24:30.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.873+0000 7f569b200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f56940ad750 con 0x7f56940d0590 2024-05-28T06:24:30.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.873+0000 7f569b200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f56940ce880 con 0x7f56940b4e10 2024-05-28T06:24:30.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.874+0000 7f5693e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5694099090 0x7f5694098720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:30.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.874+0000 7f5693e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5694099090 0x7f5694098720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47046/0 (socket says 172.21.15.96:47046) 2024-05-28T06:24:30.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.874+0000 7f5693e00640 1 -- 172.21.15.96:0/3853339669 learned_addr learned my addr 172.21.15.96:0/3853339669 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:24:30.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.874+0000 7f5693e00640 1 -- 172.21.15.96:0/3853339669 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f56940b4e10 msgr2=0x7f56941512a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:30.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.874+0000 7f5693e00640 1 --2- 172.21.15.96:0/3853339669 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f56940b4e10 0x7f56941512a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:30.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.874+0000 7f5693e00640 1 -- 172.21.15.96:0/3853339669 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f56940d0590 msgr2=0x7f569415ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:30.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.874+0000 7f5693e00640 1 --2- 172.21.15.96:0/3853339669 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f56940d0590 0x7f569415ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:30.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.874+0000 7f5693e00640 1 -- 172.21.15.96:0/3853339669 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5694098c60 con 0x7f5694099090 2024-05-28T06:24:30.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.874+0000 7f5693e00640 1 --2- 172.21.15.96:0/3853339669 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5694099090 0x7f5694098720 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 crypto rx=0x7f568400c460 tx=0x7f568400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:30.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.875+0000 7f568be00640 1 -- 172.21.15.96:0/3853339669 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5684017070 con 0x7f5694099090 2024-05-28T06:24:30.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.875+0000 7f568be00640 1 -- 172.21.15.96:0/3853339669 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5684005ce0 con 0x7f5694099090 2024-05-28T06:24:30.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.875+0000 7f569b200640 1 -- 172.21.15.96:0/3853339669 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5694072560 con 0x7f5694099090 2024-05-28T06:24:30.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.875+0000 7f568be00640 1 -- 172.21.15.96:0/3853339669 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5684007290 con 0x7f5694099090 2024-05-28T06:24:30.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.875+0000 7f569b200640 1 -- 172.21.15.96:0/3853339669 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5694072b20 con 0x7f5694099090 2024-05-28T06:24:30.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.876+0000 7f569b200640 1 -- 172.21.15.96:0/3853339669 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56940689e0 con 0x7f5694099090 2024-05-28T06:24:30.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.877+0000 7f568be00640 1 -- 172.21.15.96:0/3853339669 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 42) ==== 93590+0+0 (secure 0 0 0) 0x7f568401a070 con 0x7f5694099090 2024-05-28T06:24:30.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.878+0000 7f568be00640 1 --2- 172.21.15.96:0/3853339669 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f567406e540 0x7f5674070a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:30.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.878+0000 7f568be00640 1 -- 172.21.15.96:0/3853339669 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 5765+0+0 (secure 0 0 0) 0x7f5684089dd0 con 0x7f5694099090 2024-05-28T06:24:30.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.879+0000 7f5693400640 1 --2- 172.21.15.96:0/3853339669 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f567406e540 0x7f5674070a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:30.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.880+0000 7f5693400640 1 --2- 172.21.15.96:0/3853339669 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f567406e540 0x7f5674070a00 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f569409e480 tx=0x7f5680008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:30.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:30.883+0000 7f568be00640 1 -- 172.21.15.96:0/3853339669 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5694098c60 con 0x7f5694099090 2024-05-28T06:24:31.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:31.128+0000 7f569b200640 1 -- 172.21.15.96:0/3853339669 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f569406c170 con 0x7f5694099090 2024-05-28T06:24:31.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:31.129+0000 7f568be00640 1 -- 172.21.15.96:0/3853339669 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v82) ==== 53+0+73 (secure 0 0 0) 0x7f5684059dc0 con 0x7f5694099090 2024-05-28T06:24:31.124 INFO:teuthology.orchestra.run.smithi096.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-05-28T06:24:31.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:31.131+0000 7f569b200640 1 -- 172.21.15.96:0/3853339669 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f567406e540 msgr2=0x7f5674070a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:31.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:31.131+0000 7f569b200640 1 --2- 172.21.15.96:0/3853339669 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f567406e540 0x7f5674070a00 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f569409e480 tx=0x7f5680008040 comp rx=0 tx=0).stop 2024-05-28T06:24:31.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:31.131+0000 7f569b200640 1 -- 172.21.15.96:0/3853339669 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5694099090 msgr2=0x7f5694098720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:31.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:31.131+0000 7f569b200640 1 --2- 172.21.15.96:0/3853339669 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5694099090 0x7f5694098720 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 crypto rx=0x7f568400c460 tx=0x7f568400c920 comp rx=0 tx=0).stop 2024-05-28T06:24:31.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:31.132+0000 7f569b200640 1 -- 172.21.15.96:0/3853339669 shutdown_connections 2024-05-28T06:24:31.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:31.132+0000 7f569b200640 1 --2- 172.21.15.96:0/3853339669 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f567406e540 0x7f5674070a00 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:31.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:31.132+0000 7f569b200640 1 --2- 172.21.15.96:0/3853339669 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f56940d0590 0x7f569415ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:31.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:31.132+0000 7f569b200640 1 --2- 172.21.15.96:0/3853339669 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f56940b4e10 0x7f56941512a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:31.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:31.132+0000 7f569b200640 1 --2- 172.21.15.96:0/3853339669 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5694099090 0x7f5694098720 unknown :-1 s=CLOSED pgs=2043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:31.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:31.132+0000 7f569b200640 1 -- 172.21.15.96:0/3853339669 >> 172.21.15.96:0/3853339669 conn(0x7f569406bae0 msgr2=0x7f56940ca8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:31.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:31.132+0000 7f569b200640 1 -- 172.21.15.96:0/3853339669 shutdown_connections 2024-05-28T06:24:31.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:31.132+0000 7f569b200640 1 -- 172.21.15.96:0/3853339669 wait complete. 2024-05-28T06:24:31.146 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-05-28T06:24:31.146 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-28T06:24:31.146 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi096.front.sepia.ceph.com 2024-05-28T06:24:31.146 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-05-28T06:24:31.146 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-28T06:24:31.146 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-05-28T06:24:31.147 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-05-28T06:24:31.147 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-28T06:24:31.147 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-28T06:24:31.147 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:24:31.147 DEBUG:teuthology.orchestra.run.smithi096:> ip addr 2024-05-28T06:24:31.197 INFO:teuthology.orchestra.run.smithi096.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-28T06:24:31.197 INFO:teuthology.orchestra.run.smithi096.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-28T06:24:31.197 INFO:teuthology.orchestra.run.smithi096.stdout: inet 127.0.0.1/8 scope host lo 2024-05-28T06:24:31.198 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:24:31.198 INFO:teuthology.orchestra.run.smithi096.stdout: inet6 ::1/128 scope host 2024-05-28T06:24:31.198 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:24:31.198 INFO:teuthology.orchestra.run.smithi096.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-28T06:24:31.198 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether 0c:c4:7a:d8:d9:bc brd ff:ff:ff:ff:ff:ff 2024-05-28T06:24:31.198 INFO:teuthology.orchestra.run.smithi096.stdout: altname enp7s0f0 2024-05-28T06:24:31.198 INFO:teuthology.orchestra.run.smithi096.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-28T06:24:31.198 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether 0c:c4:7a:d8:d9:bd brd ff:ff:ff:ff:ff:ff 2024-05-28T06:24:31.198 INFO:teuthology.orchestra.run.smithi096.stdout: altname enp7s0f1 2024-05-28T06:24:31.198 INFO:teuthology.orchestra.run.smithi096.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-28T06:24:31.198 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether 0c:c4:7a:8f:cc:0c brd ff:ff:ff:ff:ff:ff 2024-05-28T06:24:31.198 INFO:teuthology.orchestra.run.smithi096.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-28T06:24:31.198 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether 0c:c4:7a:8f:cc:0d brd ff:ff:ff:ff:ff:ff 2024-05-28T06:24:31.199 INFO:teuthology.orchestra.run.smithi096.stdout: inet 172.21.15.96/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-05-28T06:24:31.199 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft 41366sec preferred_lft 41366sec 2024-05-28T06:24:31.199 INFO:teuthology.orchestra.run.smithi096.stdout: inet6 fe80::ec4:7aff:fe8f:cc0d/64 scope link 2024-05-28T06:24:31.199 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:24:31.199 INFO:teuthology.orchestra.run.smithi096.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-05-28T06:24:31.199 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether ae:d9:f5:b0:4f:f3 brd ff:ff:ff:ff:ff:ff 2024-05-28T06:24:31.199 INFO:teuthology.orchestra.run.smithi096.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-05-28T06:24:31.199 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:24:31.199 INFO:teuthology.orchestra.run.smithi096.stdout: inet6 fe80::604e:fcff:fe60:7892/64 scope link 2024-05-28T06:24:31.199 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:24:31.199 INFO:teuthology.orchestra.run.smithi096.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-28T06:24:31.199 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether ce:6f:19:93:44:f9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-28T06:24:31.199 INFO:teuthology.orchestra.run.smithi096.stdout: inet6 fe80::cc6f:19ff:fe93:44f9/64 scope link 2024-05-28T06:24:31.199 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:24:31.200 INFO:teuthology.orchestra.run.smithi096.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-28T06:24:31.200 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether ae:d9:f5:b0:4f:f3 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-28T06:24:31.200 INFO:teuthology.orchestra.run.smithi096.stdout: inet6 fe80::acd9:f5ff:feb0:4ff3/64 scope link 2024-05-28T06:24:31.200 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-05-28T06:24:31.201 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:24:31.201 DEBUG:teuthology.orchestra.run.smithi096:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-28T06:24:31.277 INFO:teuthology.orchestra.run.smithi096.stdout:1 2024-05-28T06:24:31.281 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:24:31.281 DEBUG:teuthology.orchestra.run.smithi096:> route 2024-05-28T06:24:31.332 INFO:teuthology.orchestra.run.smithi096.stdout:Kernel IP routing table 2024-05-28T06:24:31.332 INFO:teuthology.orchestra.run.smithi096.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-28T06:24:31.332 INFO:teuthology.orchestra.run.smithi096.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-05-28T06:24:31.332 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-05-28T06:24:31.332 INFO:teuthology.orchestra.run.smithi096.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-28T06:24:31.333 DEBUG:teuthology.orchestra.run.smithi096:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-28T06:24:31.333 DEBUG:teuthology.orchestra.run.smithi096:> set -e 2024-05-28T06:24:31.333 DEBUG:teuthology.orchestra.run.smithi096:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-05-28T06:24:31.333 DEBUG:teuthology.orchestra.run.smithi096:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-05-28T06:24:31.333 DEBUG:teuthology.orchestra.run.smithi096:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-05-28T06:24:31.333 DEBUG:teuthology.orchestra.run.smithi096:> ') 2024-05-28T06:24:31.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:31 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-28T06:24:31.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:31 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-28T06:24:31.544 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:24:31.545 DEBUG:teuthology.orchestra.run.smithi096:> ip netns list 2024-05-28T06:24:31.594 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-05-28T06:24:31.594 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-05-28T06:24:31.595 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-28T06:24:31.595 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:24:31.595 DEBUG:teuthology.orchestra.run.smithi096:> sudo ip link set brx.0 up 2024-05-28T06:24:31.673 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-05-28T06:24:31.673 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-28T06:24:31.673 DEBUG:teuthology.orchestra.run.smithi096:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-05-28T06:24:31.727 INFO:teuthology.orchestra.run.smithi096.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-05-28T06:24:31.727 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-28T06:24:31.727 DEBUG:teuthology.orchestra.run.smithi096:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-05-28T06:24:31.777 DEBUG:teuthology.orchestra.run.smithi096:> sudo modprobe fuse 2024-05-28T06:24:31.864 DEBUG:teuthology.orchestra.run.smithi096:> cat /proc/self/mounts | awk '{print $2}' 2024-05-28T06:24:31.923 INFO:teuthology.orchestra.run.smithi096.stdout:/proc 2024-05-28T06:24:31.923 INFO:teuthology.orchestra.run.smithi096.stdout:/sys 2024-05-28T06:24:31.923 INFO:teuthology.orchestra.run.smithi096.stdout:/dev 2024-05-28T06:24:31.923 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/security 2024-05-28T06:24:31.923 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/shm 2024-05-28T06:24:31.923 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/pts 2024-05-28T06:24:31.923 INFO:teuthology.orchestra.run.smithi096.stdout:/run 2024-05-28T06:24:31.923 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/cgroup 2024-05-28T06:24:31.923 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/pstore 2024-05-28T06:24:31.923 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/bpf 2024-05-28T06:24:31.924 INFO:teuthology.orchestra.run.smithi096.stdout:/ 2024-05-28T06:24:31.924 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/selinux 2024-05-28T06:24:31.924 INFO:teuthology.orchestra.run.smithi096.stdout:/proc/sys/fs/binfmt_misc 2024-05-28T06:24:31.924 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/mqueue 2024-05-28T06:24:31.924 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/debug 2024-05-28T06:24:31.924 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/hugepages 2024-05-28T06:24:31.924 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/tracing 2024-05-28T06:24:31.924 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/fuse/connections 2024-05-28T06:24:31.924 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/config 2024-05-28T06:24:31.924 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-sysctl.service 2024-05-28T06:24:31.924 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-28T06:24:31.925 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-28T06:24:31.925 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/nfs/rpc_pipefs 2024-05-28T06:24:31.925 INFO:teuthology.orchestra.run.smithi096.stdout:/run/user/1000 2024-05-28T06:24:31.925 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph 2024-05-28T06:24:31.925 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-0 2024-05-28T06:24:31.925 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-1 2024-05-28T06:24:31.925 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-2 2024-05-28T06:24:31.925 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-3 2024-05-28T06:24:31.925 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns 2024-05-28T06:24:31.925 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-28T06:24:31.925 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-28T06:24:31.925 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-28T06:24:31.925 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-28T06:24:31.926 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:24:31.926 DEBUG:teuthology.orchestra.run.smithi096:> ls /sys/fs/fuse/connections 2024-05-28T06:24:31.979 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-05-28T06:24:31.980 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-05-28T06:24:31.985 DEBUG:teuthology.orchestra.run.smithi096:> sudo modprobe fuse 2024-05-28T06:24:32.053 DEBUG:teuthology.orchestra.run.smithi096:> cat /proc/self/mounts | awk '{print $2}' 2024-05-28T06:24:32.096 INFO:teuthology.orchestra.run.smithi096.stdout:/proc 2024-05-28T06:24:32.096 INFO:teuthology.orchestra.run.smithi096.stdout:/sys 2024-05-28T06:24:32.096 INFO:teuthology.orchestra.run.smithi096.stdout:/dev 2024-05-28T06:24:32.096 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/security 2024-05-28T06:24:32.096 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/shm 2024-05-28T06:24:32.096 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/pts 2024-05-28T06:24:32.096 INFO:teuthology.orchestra.run.smithi096.stdout:/run 2024-05-28T06:24:32.096 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/cgroup 2024-05-28T06:24:32.096 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/pstore 2024-05-28T06:24:32.096 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/bpf 2024-05-28T06:24:32.096 INFO:teuthology.orchestra.run.smithi096.stdout:/ 2024-05-28T06:24:32.096 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/selinux 2024-05-28T06:24:32.097 INFO:teuthology.orchestra.run.smithi096.stdout:/proc/sys/fs/binfmt_misc 2024-05-28T06:24:32.097 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/mqueue 2024-05-28T06:24:32.097 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/debug 2024-05-28T06:24:32.097 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/hugepages 2024-05-28T06:24:32.097 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/tracing 2024-05-28T06:24:32.097 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/fuse/connections 2024-05-28T06:24:32.097 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/config 2024-05-28T06:24:32.097 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-sysctl.service 2024-05-28T06:24:32.097 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-28T06:24:32.097 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-28T06:24:32.097 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/nfs/rpc_pipefs 2024-05-28T06:24:32.097 INFO:teuthology.orchestra.run.smithi096.stdout:/run/user/1000 2024-05-28T06:24:32.097 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph 2024-05-28T06:24:32.097 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-0 2024-05-28T06:24:32.098 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-1 2024-05-28T06:24:32.098 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-2 2024-05-28T06:24:32.098 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-3 2024-05-28T06:24:32.098 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns 2024-05-28T06:24:32.098 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-28T06:24:32.098 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-28T06:24:32.098 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-28T06:24:32.098 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-28T06:24:32.099 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:24:32.099 DEBUG:teuthology.orchestra.run.smithi096:> ls /sys/fs/fuse/connections 2024-05-28T06:24:32.153 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi096.stderr:2024-05-28T06:24:32.159+0000 7f1079e0c480 -1 init, newargv = 0x558ae0278d20 newargc=15 2024-05-28T06:24:32.154 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi096.stderr:ceph-fuse[83889]: starting ceph client 2024-05-28T06:24:32.163 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi096.stderr:ceph-fuse[83889]: starting fuse 2024-05-28T06:24:32.183 INFO:teuthology.orchestra.run.smithi096.stdout:40 2024-05-28T06:24:32.184 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40] 2024-05-28T06:24:32.184 DEBUG:teuthology.orchestra.run.smithi096:> sudo stdin-killer -- python3 -c ' 2024-05-28T06:24:32.184 DEBUG:teuthology.orchestra.run.smithi096:> import glob 2024-05-28T06:24:32.184 DEBUG:teuthology.orchestra.run.smithi096:> import re 2024-05-28T06:24:32.184 DEBUG:teuthology.orchestra.run.smithi096:> import os 2024-05-28T06:24:32.184 DEBUG:teuthology.orchestra.run.smithi096:> import subprocess 2024-05-28T06:24:32.184 DEBUG:teuthology.orchestra.run.smithi096:> 2024-05-28T06:24:32.184 DEBUG:teuthology.orchestra.run.smithi096:> def _find_admin_socket(client_name): 2024-05-28T06:24:32.185 DEBUG:teuthology.orchestra.run.smithi096:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-05-28T06:24:32.185 DEBUG:teuthology.orchestra.run.smithi096:> files = glob.glob(asok_path) 2024-05-28T06:24:32.185 DEBUG:teuthology.orchestra.run.smithi096:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-05-28T06:24:32.185 DEBUG:teuthology.orchestra.run.smithi096:> 2024-05-28T06:24:32.185 DEBUG:teuthology.orchestra.run.smithi096:> # Given a non-glob path, it better be there 2024-05-28T06:24:32.185 DEBUG:teuthology.orchestra.run.smithi096:> if "*" not in asok_path: 2024-05-28T06:24:32.185 DEBUG:teuthology.orchestra.run.smithi096:> assert(len(files) == 1) 2024-05-28T06:24:32.185 DEBUG:teuthology.orchestra.run.smithi096:> return files[0] 2024-05-28T06:24:32.185 DEBUG:teuthology.orchestra.run.smithi096:> 2024-05-28T06:24:32.185 DEBUG:teuthology.orchestra.run.smithi096:> for f in files: 2024-05-28T06:24:32.185 DEBUG:teuthology.orchestra.run.smithi096:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-05-28T06:24:32.185 DEBUG:teuthology.orchestra.run.smithi096:> if os.path.exists("/proc/{0}".format(pid)): 2024-05-28T06:24:32.185 DEBUG:teuthology.orchestra.run.smithi096:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-05-28T06:24:32.185 DEBUG:teuthology.orchestra.run.smithi096:> contents = proc_f.read() 2024-05-28T06:24:32.185 DEBUG:teuthology.orchestra.run.smithi096:> if mountpoint in contents: 2024-05-28T06:24:32.185 DEBUG:teuthology.orchestra.run.smithi096:> return f 2024-05-28T06:24:32.186 DEBUG:teuthology.orchestra.run.smithi096:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-05-28T06:24:32.186 DEBUG:teuthology.orchestra.run.smithi096:> 2024-05-28T06:24:32.186 DEBUG:teuthology.orchestra.run.smithi096:> print(_find_admin_socket("client.0")) 2024-05-28T06:24:32.186 DEBUG:teuthology.orchestra.run.smithi096:> ' 2024-05-28T06:24:32.376 INFO:teuthology.orchestra.run.smithi096.stdout:/var/run/ceph/ceph-client.0.83889.asok 2024-05-28T06:24:32.381 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:32 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-28T06:24:32.394 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.83889.asok 2024-05-28T06:24:32.394 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:24:32.394 DEBUG:teuthology.orchestra.run.smithi096:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.83889.asok status 2024-05-28T06:24:32.627 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:24:32.627 INFO:teuthology.orchestra.run.smithi096.stdout: "metadata": { 2024-05-28T06:24:32.627 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph_sha1": "9ed025fbf4406fb54980a0647506b15895b6d863", 2024-05-28T06:24:32.627 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph_version": "ceph version 19.0.0-3154-g9ed025fb (9ed025fbf4406fb54980a0647506b15895b6d863) squid (dev)", 2024-05-28T06:24:32.627 INFO:teuthology.orchestra.run.smithi096.stdout: "entity_id": "0", 2024-05-28T06:24:32.627 INFO:teuthology.orchestra.run.smithi096.stdout: "hostname": "smithi096", 2024-05-28T06:24:32.627 INFO:teuthology.orchestra.run.smithi096.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-05-28T06:24:32.627 INFO:teuthology.orchestra.run.smithi096.stdout: "pid": "83889", 2024-05-28T06:24:32.628 INFO:teuthology.orchestra.run.smithi096.stdout: "root": "/" 2024-05-28T06:24:32.628 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:24:32.628 INFO:teuthology.orchestra.run.smithi096.stdout: "dentry_count": 0, 2024-05-28T06:24:32.628 INFO:teuthology.orchestra.run.smithi096.stdout: "dentry_pinned_count": 0, 2024-05-28T06:24:32.628 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 10017, 2024-05-28T06:24:32.628 INFO:teuthology.orchestra.run.smithi096.stdout: "inst": { 2024-05-28T06:24:32.628 INFO:teuthology.orchestra.run.smithi096.stdout: "name": { 2024-05-28T06:24:32.628 INFO:teuthology.orchestra.run.smithi096.stdout: "type": "client", 2024-05-28T06:24:32.628 INFO:teuthology.orchestra.run.smithi096.stdout: "num": 10017 2024-05-28T06:24:32.628 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:24:32.628 INFO:teuthology.orchestra.run.smithi096.stdout: "addr": { 2024-05-28T06:24:32.628 INFO:teuthology.orchestra.run.smithi096.stdout: "type": "v1", 2024-05-28T06:24:32.628 INFO:teuthology.orchestra.run.smithi096.stdout: "addr": "192.168.0.1:0", 2024-05-28T06:24:32.629 INFO:teuthology.orchestra.run.smithi096.stdout: "nonce": 3081820890 2024-05-28T06:24:32.629 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:24:32.629 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:24:32.629 INFO:teuthology.orchestra.run.smithi096.stdout: "addr": { 2024-05-28T06:24:32.629 INFO:teuthology.orchestra.run.smithi096.stdout: "type": "v1", 2024-05-28T06:24:32.629 INFO:teuthology.orchestra.run.smithi096.stdout: "addr": "192.168.0.1:0", 2024-05-28T06:24:32.629 INFO:teuthology.orchestra.run.smithi096.stdout: "nonce": 3081820890 2024-05-28T06:24:32.629 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:24:32.629 INFO:teuthology.orchestra.run.smithi096.stdout: "inst_str": "client.10017 192.168.0.1:0/3081820890", 2024-05-28T06:24:32.629 INFO:teuthology.orchestra.run.smithi096.stdout: "addr_str": "192.168.0.1:0/3081820890", 2024-05-28T06:24:32.629 INFO:teuthology.orchestra.run.smithi096.stdout: "inode_count": 1, 2024-05-28T06:24:32.629 INFO:teuthology.orchestra.run.smithi096.stdout: "mds_epoch": 82, 2024-05-28T06:24:32.629 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_epoch": 130, 2024-05-28T06:24:32.629 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_epoch_barrier": 0, 2024-05-28T06:24:32.630 INFO:teuthology.orchestra.run.smithi096.stdout: "blocklisted": false, 2024-05-28T06:24:32.630 INFO:teuthology.orchestra.run.smithi096.stdout: "fs_name": "cephfs" 2024-05-28T06:24:32.630 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:24:32.641 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:24:32.642 DEBUG:teuthology.orchestra.run.smithi096:> stat --file-system '--printf=%T 2024-05-28T06:24:32.642 DEBUG:teuthology.orchestra.run.smithi096:> ' -- /home/ubuntu/cephtest/mnt.0 2024-05-28T06:24:32.693 INFO:teuthology.orchestra.run.smithi096.stdout:fuseblk 2024-05-28T06:24:32.694 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-05-28T06:24:32.694 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:24:32.694 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-05-28T06:24:32.776 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-05-28T06:24:32.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:32.990+0000 7ffaf0a00640 1 Processor -- start 2024-05-28T06:24:32.984 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:32.990+0000 7ffaf0a00640 1 -- start start 2024-05-28T06:24:32.986 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:32.991+0000 7ffaf0a00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7ffaec15fa00 0x7ffaec06e970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:32.986 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:32.991+0000 7ffaf0a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffaec1600f0 0x7ffaec0d0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:32.986 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:32.991+0000 7ffaf0a00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffaec0d0d90 con 0x7ffaec0d0970 2024-05-28T06:24:32.986 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:32.992+0000 7ffaf0a00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7ffaec06fa00 con 0x7ffaec15fa00 2024-05-28T06:24:32.986 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:32.992+0000 7ffaf0a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ffaec06eeb0 con 0x7ffaec1600f0 2024-05-28T06:24:32.988 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:32.992+0000 7ffaeb200640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7ffaec0d0970 0x7ffaec0b3d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:57210/0 (socket says 172.21.15.96:57210) 2024-05-28T06:24:32.988 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:32.992+0000 7ffaeb200640 1 -- 172.21.15.96:0/2173564040 learned_addr learned my addr 172.21.15.96:0/2173564040 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:24:32.988 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:32.992+0000 7ffaea800640 1 --2- 172.21.15.96:0/2173564040 >> v2:172.21.15.123:3300/0 conn(0x7ffaec15fa00 0x7ffaec06e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:32.988 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:32.992+0000 7ffae1e00640 1 --2- 172.21.15.96:0/2173564040 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffaec1600f0 0x7ffaec0d0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:32.988 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:32.992+0000 7ffae9e00640 1 -- 172.21.15.96:0/2173564040 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2908010245 0 0) 0x7ffaec0d0d90 con 0x7ffaec0d0970 2024-05-28T06:24:32.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:32.993+0000 7ffae9e00640 1 -- 172.21.15.96:0/2173564040 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffad0003520 con 0x7ffaec0d0970 2024-05-28T06:24:32.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:32.993+0000 7ffae9e00640 1 -- 172.21.15.96:0/2173564040 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4275658417 0 0) 0x7ffad0003520 con 0x7ffaec0d0970 2024-05-28T06:24:32.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:32.993+0000 7ffae9e00640 1 -- 172.21.15.96:0/2173564040 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffaec1600f0 msgr2=0x7ffaec0d0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:32.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:32.993+0000 7ffae9e00640 1 --2- 172.21.15.96:0/2173564040 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffaec1600f0 0x7ffaec0d0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:32.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:32.993+0000 7ffae9e00640 1 -- 172.21.15.96:0/2173564040 >> v2:172.21.15.123:3300/0 conn(0x7ffaec15fa00 msgr2=0x7ffaec06e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:32.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:32.993+0000 7ffae9e00640 1 --2- 172.21.15.96:0/2173564040 >> v2:172.21.15.123:3300/0 conn(0x7ffaec15fa00 0x7ffaec06e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:32.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:32.993+0000 7ffae9e00640 1 -- 172.21.15.96:0/2173564040 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffaec06f160 con 0x7ffaec0d0970 2024-05-28T06:24:32.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:32.993+0000 7ffae9e00640 1 -- 172.21.15.96:0/2173564040 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7ffae40019f0 con 0x7ffaec0d0970 2024-05-28T06:24:32.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:32.993+0000 7ffae9e00640 1 -- 172.21.15.96:0/2173564040 >> v1:172.21.15.96:6789/0 conn(0x7ffaec0d0970 legacy=0x7ffaec0b3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:32.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:32.993+0000 7ffae9e00640 1 --2- 172.21.15.96:0/2173564040 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffad00039b0 0x7ffad0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:32.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:32.993+0000 7ffae9e00640 1 --2- 172.21.15.96:0/2173564040 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffaec1600f0 0x7ffad0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:32.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:32.993+0000 7ffae9e00640 1 --2- 172.21.15.96:0/2173564040 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffad0004d40 0x7ffad00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:32.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:32.993+0000 7ffae9e00640 1 -- 172.21.15.96:0/2173564040 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ffaec06fa00 con 0x7ffad00039b0 2024-05-28T06:24:32.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:32.993+0000 7ffae9e00640 1 -- 172.21.15.96:0/2173564040 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ffaec06eeb0 con 0x7ffad0004d40 2024-05-28T06:24:32.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:32.993+0000 7ffae9e00640 1 -- 172.21.15.96:0/2173564040 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ffad000ba40 con 0x7ffaec1600f0 2024-05-28T06:24:32.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:32.993+0000 7ffaea800640 1 --2- 172.21.15.96:0/2173564040 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffad00039b0 0x7ffad0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:32.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:32.994+0000 7ffae1e00640 1 --2- 172.21.15.96:0/2173564040 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffaec1600f0 0x7ffad0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:32.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:32.994+0000 7ffaeb200640 1 --2- 172.21.15.96:0/2173564040 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffad0004d40 0x7ffad00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:32.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:32.994+0000 7ffaea800640 1 -- 172.21.15.96:0/2173564040 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffaec1600f0 msgr2=0x7ffad0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:32.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:32.994+0000 7ffaea800640 1 --2- 172.21.15.96:0/2173564040 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffaec1600f0 0x7ffad0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:32.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:32.994+0000 7ffaea800640 1 -- 172.21.15.96:0/2173564040 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffad0004d40 msgr2=0x7ffad00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:32.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:32.994+0000 7ffaea800640 1 --2- 172.21.15.96:0/2173564040 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffad0004d40 0x7ffad00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:32.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:32.994+0000 7ffaea800640 1 -- 172.21.15.96:0/2173564040 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffad0003520 con 0x7ffad00039b0 2024-05-28T06:24:32.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:32.994+0000 7ffaeb200640 1 --2- 172.21.15.96:0/2173564040 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffad0004d40 0x7ffad00051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:24:32.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:32.994+0000 7ffae1e00640 1 --2- 172.21.15.96:0/2173564040 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffaec1600f0 0x7ffad0004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:24:32.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:32.995+0000 7ffaea800640 1 --2- 172.21.15.96:0/2173564040 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffad00039b0 0x7ffad0003dd0 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 crypto rx=0x7ffadc009ac0 tx=0x7ffad0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7475f37c05b2958c server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:32.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:32.995+0000 7ffae9e00640 1 -- 172.21.15.96:0/2173564040 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffadc036070 con 0x7ffad00039b0 2024-05-28T06:24:32.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:32.995+0000 7ffae9e00640 1 -- 172.21.15.96:0/2173564040 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffadc031400 con 0x7ffad00039b0 2024-05-28T06:24:32.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:32.995+0000 7ffae9e00640 1 -- 172.21.15.96:0/2173564040 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffadc009690 con 0x7ffad00039b0 2024-05-28T06:24:32.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:32.996+0000 7ffaf0a00640 1 -- 172.21.15.96:0/2173564040 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffad00039b0 msgr2=0x7ffad0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:32.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:32.996+0000 7ffaf0a00640 1 --2- 172.21.15.96:0/2173564040 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffad00039b0 0x7ffad0003dd0 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 crypto rx=0x7ffadc009ac0 tx=0x7ffad0005830 comp rx=0 tx=0).stop 2024-05-28T06:24:32.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:32.996+0000 7ffaeb200640 1 -- 172.21.15.96:0/2173564040 reap_dead start 2024-05-28T06:24:32.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:32.996+0000 7ffaf0a00640 1 -- 172.21.15.96:0/2173564040 shutdown_connections 2024-05-28T06:24:32.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:32.996+0000 7ffaf0a00640 1 -- 172.21.15.96:0/2173564040 >> 172.21.15.96:0/2173564040 conn(0x7ffaec06bae0 msgr2=0x7ffaec06bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:32.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:32.996+0000 7ffaf0a00640 1 -- 172.21.15.96:0/2173564040 shutdown_connections 2024-05-28T06:24:32.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:32.996+0000 7ffaf0a00640 1 -- 172.21.15.96:0/2173564040 wait complete. 2024-05-28T06:24:32.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:32.998+0000 7ffaf0a00640 1 Processor -- start 2024-05-28T06:24:32.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:32.998+0000 7ffaf0a00640 1 -- start start 2024-05-28T06:24:32.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:32.999+0000 7ffaf0a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffad00039b0 0x7ffaec152e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:32.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:32.999+0000 7ffaf0a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffaec155640 0x7ffaec156f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:32.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:32.999+0000 7ffaf0a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffaec1574d0 0x7ffaec15c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:32.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:32.999+0000 7ffaf0a00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ffaec071c70 con 0x7ffaec1574d0 2024-05-28T06:24:32.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:32.999+0000 7ffaf0a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ffaec06f720 con 0x7ffaec155640 2024-05-28T06:24:32.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:32.999+0000 7ffaf0a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ffaec15fde0 con 0x7ffad00039b0 2024-05-28T06:24:32.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:32.999+0000 7ffaeb200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffaec1574d0 0x7ffaec15c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:32.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:32.999+0000 7ffaeb200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffaec1574d0 0x7ffaec15c330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47054/0 (socket says 172.21.15.96:47054) 2024-05-28T06:24:32.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:32.999+0000 7ffaeb200640 1 -- 172.21.15.96:0/1573311496 learned_addr learned my addr 172.21.15.96:0/1573311496 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:24:32.995 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.000+0000 7ffaea800640 1 --2- 172.21.15.96:0/1573311496 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffad00039b0 0x7ffaec152e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:32.995 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.000+0000 7ffae1e00640 1 --2- 172.21.15.96:0/1573311496 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffaec155640 0x7ffaec156f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:32.995 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.000+0000 7ffaeb200640 1 -- 172.21.15.96:0/1573311496 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffad00039b0 msgr2=0x7ffaec152e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:32.995 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.000+0000 7ffaeb200640 1 --2- 172.21.15.96:0/1573311496 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffad00039b0 0x7ffaec152e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:32.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.000+0000 7ffaeb200640 1 -- 172.21.15.96:0/1573311496 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffaec155640 msgr2=0x7ffaec156f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:32.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.000+0000 7ffaeb200640 1 --2- 172.21.15.96:0/1573311496 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffaec155640 0x7ffaec156f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:32.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.000+0000 7ffaeb200640 1 -- 172.21.15.96:0/1573311496 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffaec157970 con 0x7ffaec1574d0 2024-05-28T06:24:32.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.000+0000 7ffaeb200640 1 --2- 172.21.15.96:0/1573311496 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffaec1574d0 0x7ffaec15c330 secure :-1 s=READY pgs=2049 cs=0 l=1 rev1=1 crypto rx=0x7ffaec0cdcd0 tx=0x7ffae400b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:32.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.000+0000 7ffae3400640 1 -- 172.21.15.96:0/1573311496 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffae4004d60 con 0x7ffaec1574d0 2024-05-28T06:24:32.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.000+0000 7ffae3400640 1 -- 172.21.15.96:0/1573311496 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffae400bb80 con 0x7ffaec1574d0 2024-05-28T06:24:32.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.000+0000 7ffaf0a00640 1 -- 172.21.15.96:0/1573311496 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffaec154880 con 0x7ffaec1574d0 2024-05-28T06:24:32.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.000+0000 7ffae3400640 1 -- 172.21.15.96:0/1573311496 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffae4007070 con 0x7ffaec1574d0 2024-05-28T06:24:32.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.000+0000 7ffaf0a00640 1 -- 172.21.15.96:0/1573311496 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffaec154e40 con 0x7ffaec1574d0 2024-05-28T06:24:32.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.001+0000 7ffaf0a00640 1 -- 172.21.15.96:0/1573311496 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffaec0689e0 con 0x7ffaec1574d0 2024-05-28T06:24:32.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.003+0000 7ffae3400640 1 -- 172.21.15.96:0/1573311496 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 42) ==== 93590+0+0 (secure 0 0 0) 0x7ffae401b020 con 0x7ffaec1574d0 2024-05-28T06:24:32.998 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.004+0000 7ffae3400640 1 --2- 172.21.15.96:0/1573311496 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ffac806e510 0x7ffac80709d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:32.998 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.004+0000 7ffae3400640 1 -- 172.21.15.96:0/1573311496 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 5765+0+0 (secure 0 0 0) 0x7ffae408ad10 con 0x7ffaec1574d0 2024-05-28T06:24:32.998 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.004+0000 7ffaea800640 1 --2- 172.21.15.96:0/1573311496 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ffac806e510 0x7ffac80709d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:33.000 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.006+0000 7ffaea800640 1 --2- 172.21.15.96:0/1573311496 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ffac806e510 0x7ffac80709d0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7ffadc033000 tx=0x7ffadc008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:33.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.008+0000 7ffae3400640 1 -- 172.21.15.96:0/1573311496 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ffae405ab10 con 0x7ffaec1574d0 2024-05-28T06:24:33.247 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.252+0000 7ffaf0a00640 1 -- 172.21.15.96:0/1573311496 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) -- 0x7ffaec0cda60 con 0x7ffaec1574d0 2024-05-28T06:24:33.481 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.487+0000 7ffae3400640 1 -- 172.21.15.96:0/1573311496 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v83) ==== 136+0+0 (secure 0 0 0) 0x7ffae405a410 con 0x7ffaec1574d0 2024-05-28T06:24:33.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.490+0000 7ffaf0a00640 1 -- 172.21.15.96:0/1573311496 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ffac806e510 msgr2=0x7ffac80709d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:33.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.490+0000 7ffaf0a00640 1 --2- 172.21.15.96:0/1573311496 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ffac806e510 0x7ffac80709d0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7ffadc033000 tx=0x7ffadc008030 comp rx=0 tx=0).stop 2024-05-28T06:24:33.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.490+0000 7ffaf0a00640 1 -- 172.21.15.96:0/1573311496 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffaec1574d0 msgr2=0x7ffaec15c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:33.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.490+0000 7ffaf0a00640 1 --2- 172.21.15.96:0/1573311496 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffaec1574d0 0x7ffaec15c330 secure :-1 s=READY pgs=2049 cs=0 l=1 rev1=1 crypto rx=0x7ffaec0cdcd0 tx=0x7ffae400b4a0 comp rx=0 tx=0).stop 2024-05-28T06:24:33.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.491+0000 7ffaf0a00640 1 -- 172.21.15.96:0/1573311496 shutdown_connections 2024-05-28T06:24:33.485 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.491+0000 7ffaf0a00640 1 --2- 172.21.15.96:0/1573311496 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ffac806e510 0x7ffac80709d0 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:33.486 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.491+0000 7ffaf0a00640 1 --2- 172.21.15.96:0/1573311496 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffaec1574d0 0x7ffaec15c330 unknown :-1 s=CLOSED pgs=2049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:33.486 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.491+0000 7ffaf0a00640 1 --2- 172.21.15.96:0/1573311496 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffaec155640 0x7ffaec156f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:33.486 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.491+0000 7ffaf0a00640 1 --2- 172.21.15.96:0/1573311496 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ffad00039b0 0x7ffaec152e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:33.486 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.491+0000 7ffaf0a00640 1 -- 172.21.15.96:0/1573311496 >> 172.21.15.96:0/1573311496 conn(0x7ffaec06bae0 msgr2=0x7ffaec0afe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:33.486 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.491+0000 7ffaf0a00640 1 -- 172.21.15.96:0/1573311496 shutdown_connections 2024-05-28T06:24:33.486 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.491+0000 7ffaf0a00640 1 -- 172.21.15.96:0/1573311496 wait complete. 2024-05-28T06:24:33.507 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-05-28T06:24:33.703 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.709+0000 7f4072c00640 1 Processor -- start 2024-05-28T06:24:33.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.709+0000 7f4072c00640 1 -- start start 2024-05-28T06:24:33.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.710+0000 7f4072c00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f406c06ee40 0x7f406c0ad230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:33.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.710+0000 7f4072c00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f406c156f10 con 0x7f406c06f2f0 2024-05-28T06:24:33.710 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.710+0000 7f4072c00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f406c06dda0 con 0x7f406c156b30 2024-05-28T06:24:33.710 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.710+0000 7f4072c00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f406c06e080 con 0x7f406c06ee40 2024-05-28T06:24:33.710 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.710+0000 7f4072200640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f406c06f2f0 0x7f406c0b32c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:57216/0 (socket says 172.21.15.96:57216) 2024-05-28T06:24:33.710 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.710+0000 7f4072200640 1 -- 172.21.15.96:0/3096620709 learned_addr learned my addr 172.21.15.96:0/3096620709 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:24:33.710 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.711+0000 7f4070e00640 1 --2- 172.21.15.96:0/3096620709 >> v2:172.21.15.123:3300/0 conn(0x7f406c06ee40 0x7f406c0ad230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:33.710 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.711+0000 7f406be00640 1 -- 172.21.15.96:0/3096620709 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 983029136 0 0) 0x7f406c156f10 con 0x7f406c06f2f0 2024-05-28T06:24:33.711 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.711+0000 7f406be00640 1 -- 172.21.15.96:0/3096620709 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4054003520 con 0x7f406c06f2f0 2024-05-28T06:24:33.711 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.711+0000 7f406be00640 1 -- 172.21.15.96:0/3096620709 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2702804244 0 0) 0x7f4054003520 con 0x7f406c06f2f0 2024-05-28T06:24:33.711 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.711+0000 7f406be00640 1 -- 172.21.15.96:0/3096620709 >> v2:172.21.15.123:3300/0 conn(0x7f406c06ee40 msgr2=0x7f406c0ad230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:33.711 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.711+0000 7f406be00640 1 --2- 172.21.15.96:0/3096620709 >> v2:172.21.15.123:3300/0 conn(0x7f406c06ee40 0x7f406c0ad230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:33.711 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.711+0000 7f406be00640 1 -- 172.21.15.96:0/3096620709 >> v1:172.21.15.123:6789/0 conn(0x7f406c156b30 legacy=0x7f406c0af6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:33.711 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.711+0000 7f406be00640 1 -- 172.21.15.96:0/3096620709 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f406c06e360 con 0x7f406c06f2f0 2024-05-28T06:24:33.712 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.711+0000 7f406be00640 1 -- 172.21.15.96:0/3096620709 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f40640019f0 con 0x7f406c06f2f0 2024-05-28T06:24:33.712 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.711+0000 7f406be00640 1 -- 172.21.15.96:0/3096620709 >> v1:172.21.15.96:6789/0 conn(0x7f406c06f2f0 legacy=0x7f406c0b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:33.712 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.711+0000 7f406be00640 1 --2- 172.21.15.96:0/3096620709 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40540039b0 0x7f4054003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:33.712 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.711+0000 7f406be00640 1 --2- 172.21.15.96:0/3096620709 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4054004800 0x7f4054004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:33.712 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.711+0000 7f406be00640 1 --2- 172.21.15.96:0/3096620709 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f40540051c0 0x7f405400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:33.712 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.711+0000 7f406be00640 1 -- 172.21.15.96:0/3096620709 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f406c06e080 con 0x7f40540039b0 2024-05-28T06:24:33.713 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.711+0000 7f406be00640 1 -- 172.21.15.96:0/3096620709 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f405400c160 con 0x7f4054004800 2024-05-28T06:24:33.713 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.711+0000 7f406be00640 1 -- 172.21.15.96:0/3096620709 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f405400c360 con 0x7f40540051c0 2024-05-28T06:24:33.713 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.712+0000 7f4071800640 1 --2- 172.21.15.96:0/3096620709 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40540039b0 0x7f4054003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:33.713 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.712+0000 7f4070e00640 1 --2- 172.21.15.96:0/3096620709 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4054004800 0x7f4054004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:33.713 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.712+0000 7f4071800640 1 -- 172.21.15.96:0/3096620709 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f40540051c0 msgr2=0x7f405400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:33.713 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.712+0000 7f4071800640 1 --2- 172.21.15.96:0/3096620709 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f40540051c0 0x7f405400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:33.714 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.712+0000 7f4071800640 1 -- 172.21.15.96:0/3096620709 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4054004800 msgr2=0x7f4054004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:33.714 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.712+0000 7f4071800640 1 --2- 172.21.15.96:0/3096620709 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4054004800 0x7f4054004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:33.714 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.712+0000 7f4071800640 1 -- 172.21.15.96:0/3096620709 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4054003520 con 0x7f40540039b0 2024-05-28T06:24:33.714 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.712+0000 7f4072200640 1 -- 172.21.15.96:0/3096620709 reap_dead start 2024-05-28T06:24:33.714 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.713+0000 7f4071800640 1 --2- 172.21.15.96:0/3096620709 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40540039b0 0x7f4054003dd0 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7f4060001750 tx=0x7f4054005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e787e3f9770b66a server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:33.714 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.713+0000 7f406be00640 1 -- 172.21.15.96:0/3096620709 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4060001370 con 0x7f40540039b0 2024-05-28T06:24:33.715 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.713+0000 7f406be00640 1 -- 172.21.15.96:0/3096620709 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40600052d0 con 0x7f40540039b0 2024-05-28T06:24:33.715 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.713+0000 7f406be00640 1 -- 172.21.15.96:0/3096620709 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f406002abe0 con 0x7f40540039b0 2024-05-28T06:24:33.715 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.714+0000 7f4072c00640 1 -- 172.21.15.96:0/3096620709 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40540039b0 msgr2=0x7f4054003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:33.715 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.714+0000 7f4072c00640 1 --2- 172.21.15.96:0/3096620709 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40540039b0 0x7f4054003dd0 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7f4060001750 tx=0x7f4054005830 comp rx=0 tx=0).stop 2024-05-28T06:24:33.715 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.714+0000 7f4072c00640 1 -- 172.21.15.96:0/3096620709 shutdown_connections 2024-05-28T06:24:33.715 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.714+0000 7f4072c00640 1 --2- 172.21.15.96:0/3096620709 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40540039b0 0x7f4054003dd0 unknown :-1 s=CLOSED pgs=2051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:33.715 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.714+0000 7f4072c00640 1 -- 172.21.15.96:0/3096620709 >> 172.21.15.96:0/3096620709 conn(0x7f406c0b1fc0 msgr2=0x7f406c0b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:33.716 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.714+0000 7f4072c00640 1 -- 172.21.15.96:0/3096620709 shutdown_connections 2024-05-28T06:24:33.716 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.714+0000 7f4072c00640 1 -- 172.21.15.96:0/3096620709 wait complete. 2024-05-28T06:24:33.719 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.716+0000 7f4072c00640 1 Processor -- start 2024-05-28T06:24:33.719 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.716+0000 7f4072c00640 1 -- start start 2024-05-28T06:24:33.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.717+0000 7f4072c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40540039b0 0x7f406c157180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:33.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.717+0000 7f4072c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f406c160010 0x7f406c1f7730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:33.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.717+0000 7f4072c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f406c0b1400 0x7f406c1f7c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:33.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.717+0000 7f4072c00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f406c0b2db0 con 0x7f40540039b0 2024-05-28T06:24:33.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.717+0000 7f4072c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f406c0b2c30 con 0x7f406c0b1400 2024-05-28T06:24:33.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.717+0000 7f4072c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f406c0af6e0 con 0x7f406c160010 2024-05-28T06:24:33.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.717+0000 7f4071800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40540039b0 0x7f406c157180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:33.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.717+0000 7f4071800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40540039b0 0x7f406c157180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47068/0 (socket says 172.21.15.96:47068) 2024-05-28T06:24:33.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.717+0000 7f4071800640 1 -- 172.21.15.96:0/2112488282 learned_addr learned my addr 172.21.15.96:0/2112488282 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:24:33.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.718+0000 7f4071800640 1 -- 172.21.15.96:0/2112488282 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f406c160010 msgr2=0x7f406c1f7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:33.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.718+0000 7f4071800640 1 --2- 172.21.15.96:0/2112488282 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f406c160010 0x7f406c1f7730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:33.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.718+0000 7f4071800640 1 -- 172.21.15.96:0/2112488282 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f406c0b1400 msgr2=0x7f406c1f7c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:33.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.718+0000 7f4071800640 1 --2- 172.21.15.96:0/2112488282 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f406c0b1400 0x7f406c1f7c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:33.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.718+0000 7f4071800640 1 -- 172.21.15.96:0/2112488282 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f406c09dca0 con 0x7f40540039b0 2024-05-28T06:24:33.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.718+0000 7f4071800640 1 --2- 172.21.15.96:0/2112488282 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40540039b0 0x7f406c157180 secure :-1 s=READY pgs=2052 cs=0 l=1 rev1=1 crypto rx=0x7f4060001870 tx=0x7f4060001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:33.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.718+0000 7f4069600640 1 -- 172.21.15.96:0/2112488282 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4060037040 con 0x7f40540039b0 2024-05-28T06:24:33.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.718+0000 7f4069600640 1 -- 172.21.15.96:0/2112488282 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4060040070 con 0x7f40540039b0 2024-05-28T06:24:33.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.718+0000 7f4069600640 1 -- 172.21.15.96:0/2112488282 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4060001370 con 0x7f40540039b0 2024-05-28T06:24:33.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.718+0000 7f4072c00640 1 -- 172.21.15.96:0/2112488282 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f406c15e9a0 con 0x7f40540039b0 2024-05-28T06:24:33.722 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.719+0000 7f4072c00640 1 -- 172.21.15.96:0/2112488282 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f406c09d970 con 0x7f40540039b0 2024-05-28T06:24:33.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.720+0000 7f4072c00640 1 -- 172.21.15.96:0/2112488282 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f406c0689e0 con 0x7f40540039b0 2024-05-28T06:24:33.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.721+0000 7f4069600640 1 -- 172.21.15.96:0/2112488282 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 42) ==== 93590+0+0 (secure 0 0 0) 0x7f40600052d0 con 0x7f40540039b0 2024-05-28T06:24:33.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.721+0000 7f4069600640 1 --2- 172.21.15.96:0/2112488282 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f405006e510 0x7f40500709d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:33.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.722+0000 7f4069600640 1 -- 172.21.15.96:0/2112488282 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 5765+0+0 (secure 0 0 0) 0x7f40600aa800 con 0x7f40540039b0 2024-05-28T06:24:33.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.722+0000 7f4070e00640 1 --2- 172.21.15.96:0/2112488282 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f405006e510 0x7f40500709d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:33.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.723+0000 7f4070e00640 1 --2- 172.21.15.96:0/2112488282 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f405006e510 0x7f40500709d0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f406c15f520 tx=0x7f405c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:33.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.726+0000 7f4069600640 1 -- 172.21.15.96:0/2112488282 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f406007a690 con 0x7f40540039b0 2024-05-28T06:24:33.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.948+0000 7f4072c00640 1 -- 172.21.15.96:0/2112488282 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7f406c06f7f0 con 0x7f405006e510 2024-05-28T06:24:33.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.953+0000 7f4069600640 1 -- 172.21.15.96:0/2112488282 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7f406c06f7f0 con 0x7f405006e510 2024-05-28T06:24:33.948 INFO:teuthology.orchestra.run.smithi096.stdout:[ 2024-05-28T06:24:33.948 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:24:33.948 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "cephfs" 2024-05-28T06:24:33.948 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:24:33.948 INFO:teuthology.orchestra.run.smithi096.stdout:] 2024-05-28T06:24:33.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.955+0000 7f4072c00640 1 -- 172.21.15.96:0/2112488282 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f405006e510 msgr2=0x7f40500709d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:33.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.955+0000 7f4072c00640 1 --2- 172.21.15.96:0/2112488282 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f405006e510 0x7f40500709d0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f406c15f520 tx=0x7f405c008040 comp rx=0 tx=0).stop 2024-05-28T06:24:33.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.956+0000 7f4072c00640 1 -- 172.21.15.96:0/2112488282 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40540039b0 msgr2=0x7f406c157180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:33.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.956+0000 7f4072c00640 1 --2- 172.21.15.96:0/2112488282 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40540039b0 0x7f406c157180 secure :-1 s=READY pgs=2052 cs=0 l=1 rev1=1 crypto rx=0x7f4060001870 tx=0x7f4060001120 comp rx=0 tx=0).stop 2024-05-28T06:24:33.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.956+0000 7f4072c00640 1 -- 172.21.15.96:0/2112488282 shutdown_connections 2024-05-28T06:24:33.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.956+0000 7f4072c00640 1 --2- 172.21.15.96:0/2112488282 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f405006e510 0x7f40500709d0 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:33.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.956+0000 7f4072c00640 1 --2- 172.21.15.96:0/2112488282 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f406c0b1400 0x7f406c1f7c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:33.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.956+0000 7f4072c00640 1 --2- 172.21.15.96:0/2112488282 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f406c160010 0x7f406c1f7730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:33.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.956+0000 7f4072c00640 1 --2- 172.21.15.96:0/2112488282 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40540039b0 0x7f406c157180 unknown :-1 s=CLOSED pgs=2052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:33.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.956+0000 7f4072c00640 1 -- 172.21.15.96:0/2112488282 >> 172.21.15.96:0/2112488282 conn(0x7f406c0b1fc0 msgr2=0x7f406c15b830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:33.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.956+0000 7f4072c00640 1 -- 172.21.15.96:0/2112488282 shutdown_connections 2024-05-28T06:24:33.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:33.956+0000 7f4072c00640 1 -- 172.21.15.96:0/2112488282 wait complete. 2024-05-28T06:24:33.968 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-05-28T06:24:34.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.160+0000 7f1138e00640 1 Processor -- start 2024-05-28T06:24:34.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.161+0000 7f1138e00640 1 -- start start 2024-05-28T06:24:34.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.162+0000 7f1138e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f11340b68f0 0x7f11340cff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:34.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.162+0000 7f1138e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f11340b7e50 0x7f11340cedb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:34.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.162+0000 7f1138e00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f113415f160 con 0x7f113415ed80 2024-05-28T06:24:34.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.162+0000 7f1138e00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f1134163540 con 0x7f11340b68f0 2024-05-28T06:24:34.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.162+0000 7f1138e00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f1134162930 con 0x7f11340b7e50 2024-05-28T06:24:34.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.162+0000 7f1131e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f113415ed80 0x7f11340b61e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:57226/0 (socket says 172.21.15.96:57226) 2024-05-28T06:24:34.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.162+0000 7f1131e00640 1 -- 172.21.15.96:0/787741433 learned_addr learned my addr 172.21.15.96:0/787741433 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:24:34.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.162+0000 7f1131400640 1 --2- 172.21.15.96:0/787741433 >> v2:172.21.15.96:3300/0 conn(0x7f11340b68f0 0x7f11340cff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:34.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.163+0000 7f1132800640 1 --2- 172.21.15.96:0/787741433 >> v2:172.21.15.123:3300/0 conn(0x7f11340b7e50 0x7f11340cedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:34.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.163+0000 7f1131400640 1 -- 172.21.15.96:0/787741433 >> v2:172.21.15.123:3300/0 conn(0x7f11340b7e50 msgr2=0x7f11340cedb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:34.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.163+0000 7f1131400640 1 --2- 172.21.15.96:0/787741433 >> v2:172.21.15.123:3300/0 conn(0x7f11340b7e50 0x7f11340cedb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:34.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.163+0000 7f1131400640 1 -- 172.21.15.96:0/787741433 >> v1:172.21.15.96:6789/0 conn(0x7f113415ed80 legacy=0x7f11340b61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:34.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.163+0000 7f1131400640 1 -- 172.21.15.96:0/787741433 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1134162c10 con 0x7f11340b68f0 2024-05-28T06:24:34.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.164+0000 7f1131400640 1 --2- 172.21.15.96:0/787741433 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11340b68f0 0x7f11340cff80 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 crypto rx=0x7f11340b4170 tx=0x7f111c02a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76ab9abd1dcb339a server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:34.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.164+0000 7f1130a00640 1 -- 172.21.15.96:0/787741433 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f111c007740 con 0x7f11340b68f0 2024-05-28T06:24:34.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.164+0000 7f1130a00640 1 -- 172.21.15.96:0/787741433 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f111c0078e0 con 0x7f11340b68f0 2024-05-28T06:24:34.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.164+0000 7f1130a00640 1 -- 172.21.15.96:0/787741433 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f111c007cb0 con 0x7f11340b68f0 2024-05-28T06:24:34.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.165+0000 7f1138e00640 1 -- 172.21.15.96:0/787741433 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11340b68f0 msgr2=0x7f11340cff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:34.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.165+0000 7f1138e00640 1 --2- 172.21.15.96:0/787741433 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11340b68f0 0x7f11340cff80 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 crypto rx=0x7f11340b4170 tx=0x7f111c02a680 comp rx=0 tx=0).stop 2024-05-28T06:24:34.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.165+0000 7f1138e00640 1 -- 172.21.15.96:0/787741433 shutdown_connections 2024-05-28T06:24:34.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.165+0000 7f1138e00640 1 --2- 172.21.15.96:0/787741433 >> v2:172.21.15.123:3300/0 conn(0x7f11340b7e50 0x7f11340cedb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:34.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.165+0000 7f1138e00640 1 --2- 172.21.15.96:0/787741433 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11340b68f0 0x7f11340cff80 unknown :-1 s=CLOSED pgs=2054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:34.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.165+0000 7f1138e00640 1 -- 172.21.15.96:0/787741433 >> 172.21.15.96:0/787741433 conn(0x7f11340d13c0 msgr2=0x7f11340d17d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:34.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.165+0000 7f1138e00640 1 -- 172.21.15.96:0/787741433 shutdown_connections 2024-05-28T06:24:34.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.165+0000 7f1138e00640 1 -- 172.21.15.96:0/787741433 wait complete. 2024-05-28T06:24:34.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.166+0000 7f1138e00640 1 Processor -- start 2024-05-28T06:24:34.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.167+0000 7f1138e00640 1 -- start start 2024-05-28T06:24:34.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.167+0000 7f1138e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f11340b68f0 0x7f1134145550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:34.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.167+0000 7f1138e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11340b7e50 0x7f1134147f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:34.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.167+0000 7f1138e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f113415ed80 0x7f11341484b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:34.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.167+0000 7f1138e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f11340677b0 con 0x7f11340b7e50 2024-05-28T06:24:34.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.167+0000 7f1138e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f11341631d0 con 0x7f11340b68f0 2024-05-28T06:24:34.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.167+0000 7f1138e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f1134067930 con 0x7f113415ed80 2024-05-28T06:24:34.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.168+0000 7f1131400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11340b7e50 0x7f1134147f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:34.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.168+0000 7f1131400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11340b7e50 0x7f1134147f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47092/0 (socket says 172.21.15.96:47092) 2024-05-28T06:24:34.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.168+0000 7f1131400640 1 -- 172.21.15.96:0/111271408 learned_addr learned my addr 172.21.15.96:0/111271408 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:24:34.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.168+0000 7f1131e00640 1 --2- 172.21.15.96:0/111271408 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f11340b68f0 0x7f1134145550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:34.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.168+0000 7f1132800640 1 --2- 172.21.15.96:0/111271408 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f113415ed80 0x7f11341484b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:34.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.168+0000 7f1131400640 1 -- 172.21.15.96:0/111271408 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f113415ed80 msgr2=0x7f11341484b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:34.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.168+0000 7f1131400640 1 --2- 172.21.15.96:0/111271408 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f113415ed80 0x7f11341484b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:34.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.168+0000 7f1131400640 1 -- 172.21.15.96:0/111271408 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f11340b68f0 msgr2=0x7f1134145550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:34.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.168+0000 7f1131400640 1 --2- 172.21.15.96:0/111271408 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f11340b68f0 0x7f1134145550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:34.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.168+0000 7f1131400640 1 -- 172.21.15.96:0/111271408 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1134145f80 con 0x7f11340b7e50 2024-05-28T06:24:34.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.168+0000 7f1131e00640 1 --2- 172.21.15.96:0/111271408 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f11340b68f0 0x7f1134145550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:24:34.165 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.169+0000 7f1131400640 1 --2- 172.21.15.96:0/111271408 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11340b7e50 0x7f1134147f70 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 crypto rx=0x7f111c009e80 tx=0x7f111c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:34.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.169+0000 7f112a000640 1 -- 172.21.15.96:0/111271408 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f111c040070 con 0x7f11340b7e50 2024-05-28T06:24:34.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.169+0000 7f112a000640 1 -- 172.21.15.96:0/111271408 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f111c031c40 con 0x7f11340b7e50 2024-05-28T06:24:34.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.169+0000 7f112a000640 1 -- 172.21.15.96:0/111271408 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f111c03c420 con 0x7f11340b7e50 2024-05-28T06:24:34.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.169+0000 7f1138e00640 1 -- 172.21.15.96:0/111271408 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11341461f0 con 0x7f11340b7e50 2024-05-28T06:24:34.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.169+0000 7f1138e00640 1 -- 172.21.15.96:0/111271408 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f113415ab70 con 0x7f11340b7e50 2024-05-28T06:24:34.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.170+0000 7f1138e00640 1 -- 172.21.15.96:0/111271408 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11340ba290 con 0x7f11340b7e50 2024-05-28T06:24:34.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.171+0000 7f112a000640 1 -- 172.21.15.96:0/111271408 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 42) ==== 93590+0+0 (secure 0 0 0) 0x7f111c039040 con 0x7f11340b7e50 2024-05-28T06:24:34.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.172+0000 7f112a000640 1 --2- 172.21.15.96:0/111271408 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f111006e510 0x7f11100709d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:34.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.172+0000 7f112a000640 1 -- 172.21.15.96:0/111271408 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 5765+0+0 (secure 0 0 0) 0x7f111c0ac0a0 con 0x7f11340b7e50 2024-05-28T06:24:34.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.173+0000 7f1131e00640 1 --2- 172.21.15.96:0/111271408 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f111006e510 0x7f11100709d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:34.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.174+0000 7f1131e00640 1 --2- 172.21.15.96:0/111271408 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f111006e510 0x7f11100709d0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f1124000f60 tx=0x7f1124009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:34.171 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.177+0000 7f112a000640 1 -- 172.21.15.96:0/111271408 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f11340ba290 con 0x7f11340b7e50 2024-05-28T06:24:34.373 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.378+0000 7f1138e00640 1 -- 172.21.15.96:0/111271408 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f11340689e0 con 0x7f11340b7e50 2024-05-28T06:24:34.373 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.379+0000 7f112a000640 1 -- 172.21.15.96:0/111271408 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v28)=0 v28) ==== 122+0+0 (secure 0 0 0) 0x7f111c07bf30 con 0x7f11340b7e50 2024-05-28T06:24:34.377 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.382+0000 7f1138e00640 1 -- 172.21.15.96:0/111271408 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f111006e510 msgr2=0x7f11100709d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:34.377 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.382+0000 7f1138e00640 1 --2- 172.21.15.96:0/111271408 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f111006e510 0x7f11100709d0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f1124000f60 tx=0x7f1124009040 comp rx=0 tx=0).stop 2024-05-28T06:24:34.377 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.382+0000 7f1138e00640 1 -- 172.21.15.96:0/111271408 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11340b7e50 msgr2=0x7f1134147f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:34.377 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.382+0000 7f1138e00640 1 --2- 172.21.15.96:0/111271408 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11340b7e50 0x7f1134147f70 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 crypto rx=0x7f111c009e80 tx=0x7f111c004a40 comp rx=0 tx=0).stop 2024-05-28T06:24:34.377 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.382+0000 7f1138e00640 1 -- 172.21.15.96:0/111271408 shutdown_connections 2024-05-28T06:24:34.377 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.382+0000 7f1138e00640 1 --2- 172.21.15.96:0/111271408 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f111006e510 0x7f11100709d0 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:34.377 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.382+0000 7f1138e00640 1 --2- 172.21.15.96:0/111271408 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f113415ed80 0x7f11341484b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:34.377 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.382+0000 7f1138e00640 1 --2- 172.21.15.96:0/111271408 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11340b7e50 0x7f1134147f70 unknown :-1 s=CLOSED pgs=2055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:34.378 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.382+0000 7f1138e00640 1 --2- 172.21.15.96:0/111271408 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f11340b68f0 0x7f1134145550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:34.378 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.382+0000 7f1138e00640 1 -- 172.21.15.96:0/111271408 >> 172.21.15.96:0/111271408 conn(0x7f11340d13c0 msgr2=0x7f11340c8b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:34.378 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.382+0000 7f1138e00640 1 -- 172.21.15.96:0/111271408 shutdown_connections 2024-05-28T06:24:34.378 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.382+0000 7f1138e00640 1 -- 172.21.15.96:0/111271408 wait complete. 2024-05-28T06:24:34.396 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/snapshot_clone_no_wait false 2024-05-28T06:24:34.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.606+0000 7f135f000640 1 Processor -- start 2024-05-28T06:24:34.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.606+0000 7f135f000640 1 -- start start 2024-05-28T06:24:34.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.607+0000 7f135f000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1358155b30 0x7f1358073940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:34.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.607+0000 7f135f000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f13580af0d0 0x7f13580d0b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:34.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.607+0000 7f135f000640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1358155f10 con 0x7f13580d06d0 2024-05-28T06:24:34.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.607+0000 7f135f000640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f13580d1070 con 0x7f13580af0d0 2024-05-28T06:24:34.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.607+0000 7f135f000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f135815f1a0 con 0x7f1358155b30 2024-05-28T06:24:34.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.608+0000 7f135d000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f13580af0d0 0x7f13580d0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:34.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.608+0000 7f1357e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1358155b30 0x7f1358073940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:34.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.608+0000 7f135d000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f13580af0d0 0x7f13580d0b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.96:47464/0 (socket says 172.21.15.96:47464) 2024-05-28T06:24:34.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.608+0000 7f135d000640 1 -- 172.21.15.96:0/675170530 learned_addr learned my addr 172.21.15.96:0/675170530 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:24:34.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.609+0000 7f135d000640 1 -- 172.21.15.96:0/675170530 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1358155b30 msgr2=0x7f1358073940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:34.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.609+0000 7f135d000640 1 --2- 172.21.15.96:0/675170530 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1358155b30 0x7f1358073940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:34.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.609+0000 7f135d000640 1 -- 172.21.15.96:0/675170530 >> v1:172.21.15.123:6789/0 conn(0x7f13580d06d0 legacy=0x7f13580b43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:34.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.609+0000 7f135d000640 1 -- 172.21.15.96:0/675170530 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f135815f480 con 0x7f13580af0d0 2024-05-28T06:24:34.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.609+0000 7f1357e00640 1 --2- 172.21.15.96:0/675170530 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1358155b30 0x7f1358073940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-28T06:24:34.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.610+0000 7f135d000640 1 --2- 172.21.15.96:0/675170530 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f13580af0d0 0x7f13580d0b30 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f1350009e50 tx=0x7f135002a680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=25017d89f5ac9126 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:34.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.610+0000 7f1357400640 1 -- 172.21.15.96:0/675170530 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1350036070 con 0x7f13580af0d0 2024-05-28T06:24:34.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.610+0000 7f1357400640 1 -- 172.21.15.96:0/675170530 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1350031600 con 0x7f13580af0d0 2024-05-28T06:24:34.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.610+0000 7f1357400640 1 -- 172.21.15.96:0/675170530 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13500319d0 con 0x7f13580af0d0 2024-05-28T06:24:34.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.611+0000 7f135f000640 1 -- 172.21.15.96:0/675170530 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f13580af0d0 msgr2=0x7f13580d0b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:34.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.611+0000 7f135f000640 1 --2- 172.21.15.96:0/675170530 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f13580af0d0 0x7f13580d0b30 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f1350009e50 tx=0x7f135002a680 comp rx=0 tx=0).stop 2024-05-28T06:24:34.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.611+0000 7f135f000640 1 -- 172.21.15.96:0/675170530 shutdown_connections 2024-05-28T06:24:34.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.611+0000 7f135f000640 1 --2- 172.21.15.96:0/675170530 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f13580af0d0 0x7f13580d0b30 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:34.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.611+0000 7f135f000640 1 --2- 172.21.15.96:0/675170530 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1358155b30 0x7f1358073940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:34.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.611+0000 7f135f000640 1 -- 172.21.15.96:0/675170530 >> 172.21.15.96:0/675170530 conn(0x7f13580b22e0 msgr2=0x7f13580b26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:34.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.611+0000 7f135f000640 1 -- 172.21.15.96:0/675170530 shutdown_connections 2024-05-28T06:24:34.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.611+0000 7f135f000640 1 -- 172.21.15.96:0/675170530 wait complete. 2024-05-28T06:24:34.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.613+0000 7f135f000640 1 Processor -- start 2024-05-28T06:24:34.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.613+0000 7f135f000640 1 -- start start 2024-05-28T06:24:34.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.614+0000 7f135f000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13580af0d0 0x7f1358156b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:34.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.614+0000 7f135f000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f13580d06d0 0x7f1358152970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:34.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.614+0000 7f135f000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1358155b30 0x7f13580b2f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:34.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.614+0000 7f135f000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f135815fd20 con 0x7f13580af0d0 2024-05-28T06:24:34.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.614+0000 7f135f000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f135815fa40 con 0x7f1358155b30 2024-05-28T06:24:34.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.614+0000 7f135f000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f1358073e80 con 0x7f13580d06d0 2024-05-28T06:24:34.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.614+0000 7f1357e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13580af0d0 0x7f1358156b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:34.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.614+0000 7f1357e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13580af0d0 0x7f1358156b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47106/0 (socket says 172.21.15.96:47106) 2024-05-28T06:24:34.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.614+0000 7f1357e00640 1 -- 172.21.15.96:0/1123480811 learned_addr learned my addr 172.21.15.96:0/1123480811 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:24:34.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.615+0000 7f135d000640 1 --2- 172.21.15.96:0/1123480811 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1358155b30 0x7f13580b2f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:34.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.615+0000 7f134f400640 1 --2- 172.21.15.96:0/1123480811 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f13580d06d0 0x7f1358152970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:34.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.615+0000 7f1357e00640 1 -- 172.21.15.96:0/1123480811 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f13580d06d0 msgr2=0x7f1358152970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:34.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.615+0000 7f1357e00640 1 --2- 172.21.15.96:0/1123480811 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f13580d06d0 0x7f1358152970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:34.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.615+0000 7f1357e00640 1 -- 172.21.15.96:0/1123480811 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1358155b30 msgr2=0x7f13580b2f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:34.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.615+0000 7f1357e00640 1 --2- 172.21.15.96:0/1123480811 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1358155b30 0x7f13580b2f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:34.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.615+0000 7f1357e00640 1 -- 172.21.15.96:0/1123480811 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13581589b0 con 0x7f13580af0d0 2024-05-28T06:24:34.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.615+0000 7f134f400640 1 --2- 172.21.15.96:0/1123480811 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f13580d06d0 0x7f1358152970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:24:34.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.615+0000 7f135d000640 1 --2- 172.21.15.96:0/1123480811 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1358155b30 0x7f13580b2f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:24:34.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.615+0000 7f1357e00640 1 --2- 172.21.15.96:0/1123480811 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13580af0d0 0x7f1358156b00 secure :-1 s=READY pgs=2056 cs=0 l=1 rev1=1 crypto rx=0x7f134800ebe0 tx=0x7f134800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:34.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.615+0000 7f1354c00640 1 -- 172.21.15.96:0/1123480811 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1348017070 con 0x7f13580af0d0 2024-05-28T06:24:34.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.615+0000 7f1354c00640 1 -- 172.21.15.96:0/1123480811 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f134800cc30 con 0x7f13580af0d0 2024-05-28T06:24:34.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.615+0000 7f1354c00640 1 -- 172.21.15.96:0/1123480811 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1348013470 con 0x7f13580af0d0 2024-05-28T06:24:34.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.615+0000 7f135f000640 1 -- 172.21.15.96:0/1123480811 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1358158c90 con 0x7f13580af0d0 2024-05-28T06:24:34.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.616+0000 7f135f000640 1 -- 172.21.15.96:0/1123480811 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1358158660 con 0x7f13580af0d0 2024-05-28T06:24:34.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.617+0000 7f135f000640 1 -- 172.21.15.96:0/1123480811 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13580689e0 con 0x7f13580af0d0 2024-05-28T06:24:34.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.618+0000 7f1354c00640 1 -- 172.21.15.96:0/1123480811 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 42) ==== 93590+0+0 (secure 0 0 0) 0x7f134801a070 con 0x7f13580af0d0 2024-05-28T06:24:34.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.619+0000 7f1354c00640 1 --2- 172.21.15.96:0/1123480811 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f133406e540 0x7f1334070a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:34.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.619+0000 7f1354c00640 1 -- 172.21.15.96:0/1123480811 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 5765+0+0 (secure 0 0 0) 0x7f134808ade0 con 0x7f13580af0d0 2024-05-28T06:24:34.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.619+0000 7f134f400640 1 --2- 172.21.15.96:0/1123480811 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f133406e540 0x7f1334070a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:34.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.620+0000 7f134f400640 1 --2- 172.21.15.96:0/1123480811 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f133406e540 0x7f1334070a00 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f1358156880 tx=0x7f1340009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:34.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.623+0000 7f1354c00640 1 -- 172.21.15.96:0/1123480811 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f134805ac70 con 0x7f13580af0d0 2024-05-28T06:24:34.818 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.823+0000 7f135f000640 1 -- 172.21.15.96:0/1123480811 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/snapshot_clone_no_wait}] v 0) -- 0x7f1358072b80 con 0x7f13580af0d0 2024-05-28T06:24:34.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.829+0000 7f1354c00640 1 -- 172.21.15.96:0/1123480811 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/snapshot_clone_no_wait}]=0 v29)=0 v29) ==== 136+0+0 (secure 0 0 0) 0x7f134805a570 con 0x7f13580af0d0 2024-05-28T06:24:34.827 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.832+0000 7f135f000640 1 -- 172.21.15.96:0/1123480811 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f133406e540 msgr2=0x7f1334070a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:34.827 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.832+0000 7f135f000640 1 --2- 172.21.15.96:0/1123480811 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f133406e540 0x7f1334070a00 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f1358156880 tx=0x7f1340009040 comp rx=0 tx=0).stop 2024-05-28T06:24:34.827 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.832+0000 7f135f000640 1 -- 172.21.15.96:0/1123480811 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13580af0d0 msgr2=0x7f1358156b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:34.827 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.832+0000 7f135f000640 1 --2- 172.21.15.96:0/1123480811 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13580af0d0 0x7f1358156b00 secure :-1 s=READY pgs=2056 cs=0 l=1 rev1=1 crypto rx=0x7f134800ebe0 tx=0x7f134800c520 comp rx=0 tx=0).stop 2024-05-28T06:24:34.827 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.832+0000 7f135f000640 1 -- 172.21.15.96:0/1123480811 shutdown_connections 2024-05-28T06:24:34.827 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.832+0000 7f135f000640 1 --2- 172.21.15.96:0/1123480811 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f133406e540 0x7f1334070a00 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:34.827 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.832+0000 7f135f000640 1 --2- 172.21.15.96:0/1123480811 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1358155b30 0x7f13580b2f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:34.827 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.832+0000 7f135f000640 1 --2- 172.21.15.96:0/1123480811 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f13580d06d0 0x7f1358152970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:34.827 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.832+0000 7f135f000640 1 --2- 172.21.15.96:0/1123480811 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13580af0d0 0x7f1358156b00 unknown :-1 s=CLOSED pgs=2056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:34.828 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.832+0000 7f135f000640 1 -- 172.21.15.96:0/1123480811 >> 172.21.15.96:0/1123480811 conn(0x7f13580b22e0 msgr2=0x7f135806e570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:34.828 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.832+0000 7f135f000640 1 -- 172.21.15.96:0/1123480811 shutdown_connections 2024-05-28T06:24:34.828 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:34.832+0000 7f135f000640 1 -- 172.21.15.96:0/1123480811 wait complete. 2024-05-28T06:24:34.846 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv1 --mode=777 2024-05-28T06:24:34.867 DEBUG:teuthology.orchestra.run.smithi096:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:24:34.873 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:24:34.878+0000 7fea5c600640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:24:34.873 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:24:34.878+0000 7fea5c600640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:24:34.873 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:24:34.878+0000 7fea5c600640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:24:34.874 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:24:34.879+0000 7fea5c600640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:24:34.874 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:24:34.879+0000 7fea5c600640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:24:34.896 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:24:35.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.043+0000 7f1162a00640 1 Processor -- start 2024-05-28T06:24:35.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.044+0000 7f1162a00640 1 -- start start 2024-05-28T06:24:35.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.045+0000 7f1162a00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f115c0b4a70 0x7f115c15f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:35.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.045+0000 7f1162a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f115c15fbf0 0x7f115c0d0320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:35.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.045+0000 7f1162a00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f115c15eef0 con 0x7f115c0af4a0 2024-05-28T06:24:35.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.045+0000 7f1162a00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f115c15c1c0 con 0x7f115c0b4a70 2024-05-28T06:24:35.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.045+0000 7f1162a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f115c15c4a0 con 0x7f115c15fbf0 2024-05-28T06:24:35.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.045+0000 7f1161600640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f115c0af4a0 0x7f115c15b8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:57234/0 (socket says 172.21.15.96:57234) 2024-05-28T06:24:35.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.045+0000 7f1161600640 1 -- 172.21.15.96:0/1915611733 learned_addr learned my addr 172.21.15.96:0/1915611733 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:24:35.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.046+0000 7f1160c00640 1 --2- 172.21.15.96:0/1915611733 >> v2:172.21.15.123:3300/0 conn(0x7f115c0b4a70 0x7f115c15f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:35.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.046+0000 7f1162000640 1 --2- 172.21.15.96:0/1915611733 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f115c15fbf0 0x7f115c0d0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:35.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.046+0000 7f115be00640 1 -- 172.21.15.96:0/1915611733 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 599750841 0 0) 0x7f115c15eef0 con 0x7f115c0af4a0 2024-05-28T06:24:35.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.046+0000 7f115be00640 1 -- 172.21.15.96:0/1915611733 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1140005730 con 0x7f115c0af4a0 2024-05-28T06:24:35.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.046+0000 7f115be00640 1 -- 172.21.15.96:0/1915611733 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1129638081 0 0) 0x7f1140005730 con 0x7f115c0af4a0 2024-05-28T06:24:35.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.046+0000 7f115be00640 1 -- 172.21.15.96:0/1915611733 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f115c15fbf0 msgr2=0x7f115c0d0320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:35.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.046+0000 7f115be00640 1 --2- 172.21.15.96:0/1915611733 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f115c15fbf0 0x7f115c0d0320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:35.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.046+0000 7f115be00640 1 -- 172.21.15.96:0/1915611733 >> v2:172.21.15.123:3300/0 conn(0x7f115c0b4a70 msgr2=0x7f115c15f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:35.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.046+0000 7f115be00640 1 --2- 172.21.15.96:0/1915611733 >> v2:172.21.15.123:3300/0 conn(0x7f115c0b4a70 0x7f115c15f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:35.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.046+0000 7f115be00640 1 -- 172.21.15.96:0/1915611733 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f115c15c780 con 0x7f115c0af4a0 2024-05-28T06:24:35.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.046+0000 7f115be00640 1 -- 172.21.15.96:0/1915611733 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f114c001690 con 0x7f115c0af4a0 2024-05-28T06:24:35.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.046+0000 7f115be00640 1 -- 172.21.15.96:0/1915611733 >> v1:172.21.15.96:6789/0 conn(0x7f115c0af4a0 legacy=0x7f115c15b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:35.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.046+0000 7f115be00640 1 --2- 172.21.15.96:0/1915611733 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1140005bc0 0x7f1140005fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:35.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.046+0000 7f115be00640 1 --2- 172.21.15.96:0/1915611733 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1140006a10 0x7f1140006e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:35.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.046+0000 7f115be00640 1 --2- 172.21.15.96:0/1915611733 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f11400073d0 0x7f114000dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:35.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.046+0000 7f115be00640 1 -- 172.21.15.96:0/1915611733 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f115c15c1c0 con 0x7f1140005bc0 2024-05-28T06:24:35.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.046+0000 7f115be00640 1 -- 172.21.15.96:0/1915611733 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f115c15c4a0 con 0x7f1140006a10 2024-05-28T06:24:35.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.046+0000 7f115be00640 1 -- 172.21.15.96:0/1915611733 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f114000e140 con 0x7f11400073d0 2024-05-28T06:24:35.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.047+0000 7f1162000640 1 --2- 172.21.15.96:0/1915611733 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f115c15fbf0 0x7f115c0d0320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-28T06:24:35.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.047+0000 7f1161600640 1 --2- 172.21.15.96:0/1915611733 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1140005bc0 0x7f1140005fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:35.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.047+0000 7f1160c00640 1 --2- 172.21.15.96:0/1915611733 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1140006a10 0x7f1140006e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:35.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.047+0000 7f1161600640 1 -- 172.21.15.96:0/1915611733 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f11400073d0 msgr2=0x7f114000dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:35.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.047+0000 7f1161600640 1 --2- 172.21.15.96:0/1915611733 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f11400073d0 0x7f114000dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:35.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.047+0000 7f1161600640 1 -- 172.21.15.96:0/1915611733 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1140006a10 msgr2=0x7f1140006e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:35.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.047+0000 7f1161600640 1 --2- 172.21.15.96:0/1915611733 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1140006a10 0x7f1140006e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:35.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.047+0000 7f1161600640 1 -- 172.21.15.96:0/1915611733 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1140005730 con 0x7f1140005bc0 2024-05-28T06:24:35.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.047+0000 7f1160c00640 1 --2- 172.21.15.96:0/1915611733 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1140006a10 0x7f1140006e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:24:35.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.048+0000 7f1161600640 1 --2- 172.21.15.96:0/1915611733 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1140005bc0 0x7f1140005fe0 secure :-1 s=READY pgs=2058 cs=0 l=1 rev1=1 crypto rx=0x7f114c00c210 tx=0x7f1140007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70f6ff420c064b32 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:35.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.049+0000 7f115be00640 1 -- 172.21.15.96:0/1915611733 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f114c007d60 con 0x7f1140005bc0 2024-05-28T06:24:35.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.049+0000 7f115be00640 1 -- 172.21.15.96:0/1915611733 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f114c005af0 con 0x7f1140005bc0 2024-05-28T06:24:35.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.049+0000 7f115be00640 1 -- 172.21.15.96:0/1915611733 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f114c007070 con 0x7f1140005bc0 2024-05-28T06:24:35.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.049+0000 7f1162a00640 1 -- 172.21.15.96:0/1915611733 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1140005bc0 msgr2=0x7f1140005fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:35.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.049+0000 7f1162a00640 1 --2- 172.21.15.96:0/1915611733 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1140005bc0 0x7f1140005fe0 secure :-1 s=READY pgs=2058 cs=0 l=1 rev1=1 crypto rx=0x7f114c00c210 tx=0x7f1140007a40 comp rx=0 tx=0).stop 2024-05-28T06:24:35.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.049+0000 7f1162000640 1 -- 172.21.15.96:0/1915611733 reap_dead start 2024-05-28T06:24:35.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.049+0000 7f1162a00640 1 -- 172.21.15.96:0/1915611733 shutdown_connections 2024-05-28T06:24:35.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.049+0000 7f1162a00640 1 -- 172.21.15.96:0/1915611733 >> 172.21.15.96:0/1915611733 conn(0x7f115c06bae0 msgr2=0x7f115c06bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:35.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.049+0000 7f1162a00640 1 -- 172.21.15.96:0/1915611733 shutdown_connections 2024-05-28T06:24:35.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.049+0000 7f1162a00640 1 -- 172.21.15.96:0/1915611733 wait complete. 2024-05-28T06:24:35.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.051+0000 7f1162a00640 1 Processor -- start 2024-05-28T06:24:35.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.051+0000 7f1162a00640 1 -- start start 2024-05-28T06:24:35.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.053+0000 7f1162a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1140005bc0 0x7f115c09ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:35.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.053+0000 7f1162a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f115c152020 0x7f115c0b2d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:35.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.053+0000 7f1162a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f115c154800 0x7f115c151490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:35.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.053+0000 7f1162a00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f115c0afa00 con 0x7f115c152020 2024-05-28T06:24:35.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.053+0000 7f1162a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f115c0af880 con 0x7f1140005bc0 2024-05-28T06:24:35.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.053+0000 7f1162a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f115c0d0320 con 0x7f115c154800 2024-05-28T06:24:35.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.053+0000 7f1160c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f115c152020 0x7f115c0b2d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:35.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.053+0000 7f1160c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f115c152020 0x7f115c0b2d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47124/0 (socket says 172.21.15.96:47124) 2024-05-28T06:24:35.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.053+0000 7f1160c00640 1 -- 172.21.15.96:0/339297574 learned_addr learned my addr 172.21.15.96:0/339297574 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:24:35.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.053+0000 7f1161600640 1 --2- 172.21.15.96:0/339297574 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1140005bc0 0x7f115c09ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:35.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.053+0000 7f1160c00640 1 -- 172.21.15.96:0/339297574 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f115c154800 msgr2=0x7f115c151490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:35.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.053+0000 7f1160c00640 1 --2- 172.21.15.96:0/339297574 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f115c154800 0x7f115c151490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:35.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.053+0000 7f1160c00640 1 -- 172.21.15.96:0/339297574 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1140005bc0 msgr2=0x7f115c09ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:35.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.053+0000 7f1160c00640 1 --2- 172.21.15.96:0/339297574 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1140005bc0 0x7f115c09ca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:35.049 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.053+0000 7f1160c00640 1 -- 172.21.15.96:0/339297574 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f115c0b3620 con 0x7f115c152020 2024-05-28T06:24:35.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.054+0000 7f1160c00640 1 --2- 172.21.15.96:0/339297574 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f115c152020 0x7f115c0b2d30 secure :-1 s=READY pgs=2059 cs=0 l=1 rev1=1 crypto rx=0x7f11480109d0 tx=0x7f1148010e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:35.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.054+0000 7f1159600640 1 -- 172.21.15.96:0/339297574 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f114800f040 con 0x7f115c152020 2024-05-28T06:24:35.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.054+0000 7f1159600640 1 -- 172.21.15.96:0/339297574 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f114800eb40 con 0x7f115c152020 2024-05-28T06:24:35.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.054+0000 7f1159600640 1 -- 172.21.15.96:0/339297574 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f114801c400 con 0x7f115c152020 2024-05-28T06:24:35.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.054+0000 7f1162a00640 1 -- 172.21.15.96:0/339297574 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f115c0b3900 con 0x7f115c152020 2024-05-28T06:24:35.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.054+0000 7f1162a00640 1 -- 172.21.15.96:0/339297574 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f115c09cf40 con 0x7f115c152020 2024-05-28T06:24:35.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.055+0000 7f1162a00640 1 -- 172.21.15.96:0/339297574 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f115c0689e0 con 0x7f115c152020 2024-05-28T06:24:35.050 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.056+0000 7f1159600640 1 -- 172.21.15.96:0/339297574 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 42) ==== 93590+0+0 (secure 0 0 0) 0x7f114801c5a0 con 0x7f115c152020 2024-05-28T06:24:35.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.057+0000 7f1159600640 1 --2- 172.21.15.96:0/339297574 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f113c06e4e0 0x7f113c0709a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:35.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.057+0000 7f1159600640 1 -- 172.21.15.96:0/339297574 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 5765+0+0 (secure 0 0 0) 0x7f114808b070 con 0x7f115c152020 2024-05-28T06:24:35.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.058+0000 7f1161600640 1 --2- 172.21.15.96:0/339297574 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f113c06e4e0 0x7f113c0709a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:35.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.060+0000 7f1161600640 1 --2- 172.21.15.96:0/339297574 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f113c06e4e0 0x7f113c0709a0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f114c033000 tx=0x7f114c009030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:35.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.062+0000 7f1159600640 1 -- 172.21.15.96:0/339297574 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1148018030 con 0x7f115c152020 2024-05-28T06:24:35.290 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.295+0000 7f1162a00640 1 -- 172.21.15.96:0/339297574 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv1", "mode": "777", "target": ["mon-mgr", ""]}) -- 0x7f115c0b3e20 con 0x7f113c06e4e0 2024-05-28T06:24:35.304 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:24:35.310+0000 7fea59e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:24:35.304 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:24:35.310+0000 7fea59e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:24:35.304 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:24:35.310+0000 7fea59e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:24:35.305 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:24:35.310+0000 7fea59e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:24:35.305 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:24:35.310+0000 7fea59e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:24:35.384 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.390+0000 7f1159600640 1 -- 172.21.15.96:0/339297574 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f115c0b3e20 con 0x7f113c06e4e0 2024-05-28T06:24:35.387 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.393+0000 7f1162a00640 1 -- 172.21.15.96:0/339297574 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f113c06e4e0 msgr2=0x7f113c0709a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:35.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.393+0000 7f1162a00640 1 --2- 172.21.15.96:0/339297574 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f113c06e4e0 0x7f113c0709a0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f114c033000 tx=0x7f114c009030 comp rx=0 tx=0).stop 2024-05-28T06:24:35.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.393+0000 7f1162a00640 1 -- 172.21.15.96:0/339297574 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f115c152020 msgr2=0x7f115c0b2d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:35.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.393+0000 7f1162a00640 1 --2- 172.21.15.96:0/339297574 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f115c152020 0x7f115c0b2d30 secure :-1 s=READY pgs=2059 cs=0 l=1 rev1=1 crypto rx=0x7f11480109d0 tx=0x7f1148010e90 comp rx=0 tx=0).stop 2024-05-28T06:24:35.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.394+0000 7f1162a00640 1 -- 172.21.15.96:0/339297574 shutdown_connections 2024-05-28T06:24:35.389 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.394+0000 7f1162a00640 1 --2- 172.21.15.96:0/339297574 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f113c06e4e0 0x7f113c0709a0 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:35.389 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.394+0000 7f1162a00640 1 --2- 172.21.15.96:0/339297574 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f115c154800 0x7f115c151490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:35.389 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.394+0000 7f1162a00640 1 --2- 172.21.15.96:0/339297574 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f115c152020 0x7f115c0b2d30 unknown :-1 s=CLOSED pgs=2059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:35.389 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.394+0000 7f1162a00640 1 --2- 172.21.15.96:0/339297574 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1140005bc0 0x7f115c09ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:35.389 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.394+0000 7f1162a00640 1 -- 172.21.15.96:0/339297574 >> 172.21.15.96:0/339297574 conn(0x7f115c06bae0 msgr2=0x7f115c152dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:35.389 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.394+0000 7f1162a00640 1 -- 172.21.15.96:0/339297574 shutdown_connections 2024-05-28T06:24:35.389 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.394+0000 7f1162a00640 1 -- 172.21.15.96:0/339297574 wait complete. 2024-05-28T06:24:35.410 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-05-28T06:24:35.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.618+0000 7efe0fa00640 1 Processor -- start 2024-05-28T06:24:35.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.619+0000 7efe0fa00640 1 -- start start 2024-05-28T06:24:35.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.620+0000 7efe0fa00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7efe0815e940 0x7efe080d2b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:35.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.620+0000 7efe0fa00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efe080b6870 0x7efe080b7880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:35.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.620+0000 7efe0fa00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe080d3060 con 0x7efe080ba070 2024-05-28T06:24:35.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.620+0000 7efe0fa00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7efe0815ed20 con 0x7efe0815e940 2024-05-28T06:24:35.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.620+0000 7efe0fa00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7efe080d32e0 con 0x7efe080b6870 2024-05-28T06:24:35.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.621+0000 7efe0d000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7efe0815e940 0x7efe080d2b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:35.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.621+0000 7efe0d000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7efe0815e940 0x7efe080d2b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.96:47492/0 (socket says 172.21.15.96:47492) 2024-05-28T06:24:35.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.621+0000 7efe0d000640 1 -- 172.21.15.96:0/711728908 learned_addr learned my addr 172.21.15.96:0/711728908 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:24:35.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.621+0000 7efe0da00640 1 --2- 172.21.15.96:0/711728908 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efe080b6870 0x7efe080b7880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:35.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.622+0000 7efe0d000640 1 -- 172.21.15.96:0/711728908 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efe080b6870 msgr2=0x7efe080b7880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:35.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.622+0000 7efe0d000640 1 --2- 172.21.15.96:0/711728908 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efe080b6870 0x7efe080b7880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:35.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.622+0000 7efe0d000640 1 -- 172.21.15.96:0/711728908 >> v1:172.21.15.123:6789/0 conn(0x7efe080ba070 legacy=0x7efe080b6160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:35.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.622+0000 7efe0d000640 1 -- 172.21.15.96:0/711728908 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe080d35c0 con 0x7efe0815e940 2024-05-28T06:24:35.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.623+0000 7efe0d000640 1 --2- 172.21.15.96:0/711728908 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efe0815e940 0x7efe080d2b20 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7efdf8009e50 tx=0x7efdf802a710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a9b3755a62eff6ec server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:35.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.623+0000 7efe07400640 1 -- 172.21.15.96:0/711728908 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdf8036070 con 0x7efe0815e940 2024-05-28T06:24:35.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.623+0000 7efe07400640 1 -- 172.21.15.96:0/711728908 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efdf8031600 con 0x7efe0815e940 2024-05-28T06:24:35.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.623+0000 7efe07400640 1 -- 172.21.15.96:0/711728908 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdf80319d0 con 0x7efe0815e940 2024-05-28T06:24:35.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.624+0000 7efe0fa00640 1 -- 172.21.15.96:0/711728908 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efe0815e940 msgr2=0x7efe080d2b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:35.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.624+0000 7efe0fa00640 1 --2- 172.21.15.96:0/711728908 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efe0815e940 0x7efe080d2b20 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7efdf8009e50 tx=0x7efdf802a710 comp rx=0 tx=0).stop 2024-05-28T06:24:35.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.624+0000 7efe0fa00640 1 -- 172.21.15.96:0/711728908 shutdown_connections 2024-05-28T06:24:35.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.624+0000 7efe0fa00640 1 --2- 172.21.15.96:0/711728908 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efe080b6870 0x7efe080b7880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:35.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.624+0000 7efe0fa00640 1 --2- 172.21.15.96:0/711728908 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efe0815e940 0x7efe080d2b20 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:35.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.624+0000 7efe0fa00640 1 -- 172.21.15.96:0/711728908 >> 172.21.15.96:0/711728908 conn(0x7efe0806d150 msgr2=0x7efe080d1160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:35.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.624+0000 7efe0fa00640 1 -- 172.21.15.96:0/711728908 shutdown_connections 2024-05-28T06:24:35.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.624+0000 7efe0fa00640 1 -- 172.21.15.96:0/711728908 wait complete. 2024-05-28T06:24:35.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.626+0000 7efe0fa00640 1 Processor -- start 2024-05-28T06:24:35.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.627+0000 7efe0fa00640 1 -- start start 2024-05-28T06:24:35.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.628+0000 7efe0fa00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efe080b6870 0x7efe0815ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:35.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.628+0000 7efe0fa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efe080ba070 0x7efe08158910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:35.631 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.628+0000 7efe0fa00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efe0815e940 0x7efe0814f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:35.631 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.628+0000 7efe0fa00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7efe080d3e60 con 0x7efe080ba070 2024-05-28T06:24:35.631 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.628+0000 7efe0fa00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7efe080d3b80 con 0x7efe0815e940 2024-05-28T06:24:35.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.628+0000 7efe0fa00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7efe080d4380 con 0x7efe080b6870 2024-05-28T06:24:35.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.628+0000 7efe07e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efe080ba070 0x7efe08158910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:35.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.628+0000 7efe07e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efe080ba070 0x7efe08158910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47138/0 (socket says 172.21.15.96:47138) 2024-05-28T06:24:35.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.628+0000 7efe07e00640 1 -- 172.21.15.96:0/2524508461 learned_addr learned my addr 172.21.15.96:0/2524508461 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:24:35.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.629+0000 7efe0d000640 1 --2- 172.21.15.96:0/2524508461 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efe080b6870 0x7efe0815ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:35.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.629+0000 7efe07e00640 1 -- 172.21.15.96:0/2524508461 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efe080b6870 msgr2=0x7efe0815ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:35.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.629+0000 7efe0da00640 1 --2- 172.21.15.96:0/2524508461 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efe0815e940 0x7efe0814f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:35.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.629+0000 7efe07e00640 1 --2- 172.21.15.96:0/2524508461 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efe080b6870 0x7efe0815ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:35.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.629+0000 7efe07e00640 1 -- 172.21.15.96:0/2524508461 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efe0815e940 msgr2=0x7efe0814f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:35.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.629+0000 7efe07e00640 1 --2- 172.21.15.96:0/2524508461 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efe0815e940 0x7efe0814f620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:35.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.629+0000 7efe07e00640 1 -- 172.21.15.96:0/2524508461 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe081567e0 con 0x7efe080ba070 2024-05-28T06:24:35.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.629+0000 7efe07e00640 1 --2- 172.21.15.96:0/2524508461 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efe080ba070 0x7efe08158910 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 crypto rx=0x7efdf000c400 tx=0x7efdf000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:35.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.629+0000 7efe04c00640 1 -- 172.21.15.96:0/2524508461 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdf00052d0 con 0x7efe080ba070 2024-05-28T06:24:35.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.629+0000 7efe04c00640 1 -- 172.21.15.96:0/2524508461 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efdf00054b0 con 0x7efe080ba070 2024-05-28T06:24:35.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.629+0000 7efe04c00640 1 -- 172.21.15.96:0/2524508461 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdf0006a40 con 0x7efe080ba070 2024-05-28T06:24:35.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.629+0000 7efe0fa00640 1 -- 172.21.15.96:0/2524508461 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe08156ac0 con 0x7efe080ba070 2024-05-28T06:24:35.635 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.629+0000 7efe0fa00640 1 -- 172.21.15.96:0/2524508461 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe08157080 con 0x7efe080ba070 2024-05-28T06:24:35.635 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.630+0000 7efe0fa00640 1 -- 172.21.15.96:0/2524508461 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe08157560 con 0x7efe080ba070 2024-05-28T06:24:35.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.632+0000 7efe04c00640 1 -- 172.21.15.96:0/2524508461 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 42) ==== 93590+0+0 (secure 0 0 0) 0x7efdf0006be0 con 0x7efe080ba070 2024-05-28T06:24:35.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.632+0000 7efe04c00640 1 --2- 172.21.15.96:0/2524508461 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7efde406e510 0x7efde40709d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:35.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.633+0000 7efe04c00640 1 -- 172.21.15.96:0/2524508461 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 5765+0+0 (secure 0 0 0) 0x7efdf0089fc0 con 0x7efe080ba070 2024-05-28T06:24:35.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.633+0000 7efe0d000640 1 --2- 172.21.15.96:0/2524508461 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7efde406e510 0x7efde40709d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:35.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.634+0000 7efe0d000640 1 --2- 172.21.15.96:0/2524508461 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7efde406e510 0x7efde40709d0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7efdf8033000 tx=0x7efdf8009000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:35.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.637+0000 7efe04c00640 1 -- 172.21.15.96:0/2524508461 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7efe08157560 con 0x7efe080ba070 2024-05-28T06:24:35.859 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.865+0000 7efe0fa00640 1 -- 172.21.15.96:0/2524508461 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) -- 0x7efe080689e0 con 0x7efde406e510 2024-05-28T06:24:35.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.876+0000 7efe04c00640 1 -- 172.21.15.96:0/2524508461 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+59 (secure 0 0 0) 0x7efe080689e0 con 0x7efde406e510 2024-05-28T06:24:35.871 INFO:teuthology.orchestra.run.smithi096.stdout:/volumes/_nogroup/sv1/aeb880aa-33ae-4200-bc32-7e8a9ed82480 2024-05-28T06:24:35.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.878+0000 7efe0fa00640 1 -- 172.21.15.96:0/2524508461 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7efde406e510 msgr2=0x7efde40709d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:35.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.878+0000 7efe0fa00640 1 --2- 172.21.15.96:0/2524508461 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7efde406e510 0x7efde40709d0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7efdf8033000 tx=0x7efdf8009000 comp rx=0 tx=0).stop 2024-05-28T06:24:35.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.878+0000 7efe0fa00640 1 -- 172.21.15.96:0/2524508461 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efe080ba070 msgr2=0x7efe08158910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:35.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.879+0000 7efe0fa00640 1 --2- 172.21.15.96:0/2524508461 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efe080ba070 0x7efe08158910 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 crypto rx=0x7efdf000c400 tx=0x7efdf000c8c0 comp rx=0 tx=0).stop 2024-05-28T06:24:35.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.879+0000 7efe0fa00640 1 -- 172.21.15.96:0/2524508461 shutdown_connections 2024-05-28T06:24:35.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.879+0000 7efe0fa00640 1 --2- 172.21.15.96:0/2524508461 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7efde406e510 0x7efde40709d0 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:35.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.879+0000 7efe0fa00640 1 --2- 172.21.15.96:0/2524508461 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efe0815e940 0x7efe0814f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:35.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.879+0000 7efe0fa00640 1 --2- 172.21.15.96:0/2524508461 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efe080ba070 0x7efe08158910 unknown :-1 s=CLOSED pgs=2060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:35.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.879+0000 7efe0fa00640 1 --2- 172.21.15.96:0/2524508461 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7efe080b6870 0x7efe0815ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:35.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.879+0000 7efe0fa00640 1 -- 172.21.15.96:0/2524508461 >> 172.21.15.96:0/2524508461 conn(0x7efe0806d150 msgr2=0x7efe080b8cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:35.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.879+0000 7efe0fa00640 1 -- 172.21.15.96:0/2524508461 shutdown_connections 2024-05-28T06:24:35.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:35.879+0000 7efe0fa00640 1 -- 172.21.15.96:0/2524508461 wait complete. 2024-05-28T06:24:35.893 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-05-28T06:24:36.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:36.101+0000 7fbee3e00640 1 Processor -- start 2024-05-28T06:24:36.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:36.101+0000 7fbee3e00640 1 -- start start 2024-05-28T06:24:36.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:36.102+0000 7fbee3e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbedc15d5d0 0x7fbedc0cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:36.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:36.103+0000 7fbee3e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fbedc0b32c0 0x7fbedc0b3750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:36.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:36.103+0000 7fbee3e00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbedc15cb10 con 0x7fbedc0d0640 2024-05-28T06:24:36.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:36.103+0000 7fbee3e00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fbedc0adbb0 con 0x7fbedc0b32c0 2024-05-28T06:24:36.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:36.103+0000 7fbee3e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fbedc0ad5f0 con 0x7fbedc15d5d0 2024-05-28T06:24:36.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:36.103+0000 7fbee1e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fbedc0b32c0 0x7fbedc0b3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:36.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:36.103+0000 7fbee1e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fbedc0b32c0 0x7fbedc0b3750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47150/0 (socket says 172.21.15.96:47150) 2024-05-28T06:24:36.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:36.103+0000 7fbee1e00640 1 -- 172.21.15.96:0/3332586232 learned_addr learned my addr 172.21.15.96:0/3332586232 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:24:36.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:36.103+0000 7fbee1400640 1 --1- 172.21.15.96:0/3332586232 >> v1:172.21.15.96:6789/0 conn(0x7fbedc0d0640 0x7fbedc0d0a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:57238/0 (socket says 172.21.15.96:57238) 2024-05-28T06:24:36.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:36.103+0000 7fbee0a00640 1 --2- 172.21.15.96:0/3332586232 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbedc15d5d0 0x7fbedc0cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:36.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:36.103+0000 7fbee1e00640 1 -- 172.21.15.96:0/3332586232 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbedc15d5d0 msgr2=0x7fbedc0cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:24:36.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:36.103+0000 7fbee1e00640 1 --2- 172.21.15.96:0/3332586232 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbedc15d5d0 0x7fbedc0cf950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:36.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:36.103+0000 7fbee1e00640 1 -- 172.21.15.96:0/3332586232 >> v1:172.21.15.96:6789/0 conn(0x7fbedc0d0640 legacy=0x7fbedc0d0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:36.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:36.104+0000 7fbee1e00640 1 -- 172.21.15.96:0/3332586232 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbedc0ad8d0 con 0x7fbedc0b32c0 2024-05-28T06:24:36.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:36.103+0000 7fbedbe00640 1 -- 172.21.15.96:0/3332586232 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1899621777 0 0) 0x7fbedc15cb10 con 0x7fbedc0d0640 2024-05-28T06:24:36.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:36.105+0000 7fbee1e00640 1 --2- 172.21.15.96:0/3332586232 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbedc0b32c0 0x7fbedc0b3750 secure :-1 s=READY pgs=2062 cs=0 l=1 rev1=1 crypto rx=0x7fbed4009e50 tx=0x7fbed402a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61f39c9ff992d2f8 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:36.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:36.105+0000 7fbedbe00640 1 -- 172.21.15.96:0/3332586232 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbed4036070 con 0x7fbedc0b32c0 2024-05-28T06:24:36.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:36.105+0000 7fbedbe00640 1 -- 172.21.15.96:0/3332586232 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbed4031600 con 0x7fbedc0b32c0 2024-05-28T06:24:36.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:36.105+0000 7fbedbe00640 1 -- 172.21.15.96:0/3332586232 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbed40319d0 con 0x7fbedc0b32c0 2024-05-28T06:24:36.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:36.105+0000 7fbee3e00640 1 -- 172.21.15.96:0/3332586232 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbedc0b32c0 msgr2=0x7fbedc0b3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:36.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:36.105+0000 7fbee3e00640 1 --2- 172.21.15.96:0/3332586232 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbedc0b32c0 0x7fbedc0b3750 secure :-1 s=READY pgs=2062 cs=0 l=1 rev1=1 crypto rx=0x7fbed4009e50 tx=0x7fbed402a710 comp rx=0 tx=0).stop 2024-05-28T06:24:36.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:36.105+0000 7fbee3e00640 1 -- 172.21.15.96:0/3332586232 shutdown_connections 2024-05-28T06:24:36.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:36.105+0000 7fbee3e00640 1 --2- 172.21.15.96:0/3332586232 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbedc0b32c0 0x7fbedc0b3750 unknown :-1 s=CLOSED pgs=2062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:36.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:36.105+0000 7fbee3e00640 1 --2- 172.21.15.96:0/3332586232 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbedc15d5d0 0x7fbedc0cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:36.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:36.105+0000 7fbee3e00640 1 -- 172.21.15.96:0/3332586232 >> 172.21.15.96:0/3332586232 conn(0x7fbedc06bd20 msgr2=0x7fbedc06c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:36.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:36.106+0000 7fbee3e00640 1 -- 172.21.15.96:0/3332586232 shutdown_connections 2024-05-28T06:24:36.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:36.106+0000 7fbee3e00640 1 -- 172.21.15.96:0/3332586232 wait complete. 2024-05-28T06:24:36.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:36.107+0000 7fbee3e00640 1 Processor -- start 2024-05-28T06:24:36.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:36.107+0000 7fbee3e00640 1 -- start start 2024-05-28T06:24:36.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:36.108+0000 7fbee3e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbedc0b32c0 0x7fbedc14ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:36.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:36.108+0000 7fbee3e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbedc0d0640 0x7fbedc0b14d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:36.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:36.108+0000 7fbee3e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbedc15d5d0 0x7fbedc14d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:36.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:36.108+0000 7fbee3e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fbedc15da50 con 0x7fbedc0d0640 2024-05-28T06:24:36.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:36.108+0000 7fbee3e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fbedc15d0b0 con 0x7fbedc0b32c0 2024-05-28T06:24:36.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:36.108+0000 7fbee3e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fbedc0ad2a0 con 0x7fbedc15d5d0 2024-05-28T06:24:36.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:36.108+0000 7fbee0a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbedc0d0640 0x7fbedc0b14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:36.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:36.108+0000 7fbee0a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbedc0d0640 0x7fbedc0b14d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:47154/0 (socket says 172.21.15.96:47154) 2024-05-28T06:24:36.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:36.108+0000 7fbee0a00640 1 -- 172.21.15.96:0/657722255 learned_addr learned my addr 172.21.15.96:0/657722255 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:24:36.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:36.109+0000 7fbee1e00640 1 --2- 172.21.15.96:0/657722255 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbedc15d5d0 0x7fbedc14d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:36.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:36.109+0000 7fbee0a00640 1 -- 172.21.15.96:0/657722255 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbedc15d5d0 msgr2=0x7fbedc14d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:36.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:36.109+0000 7fbee0a00640 1 --2- 172.21.15.96:0/657722255 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbedc15d5d0 0x7fbedc14d520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:36.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:36.109+0000 7fbee1400640 1 --2- 172.21.15.96:0/657722255 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbedc0b32c0 0x7fbedc14ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:36.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:36.109+0000 7fbee0a00640 1 -- 172.21.15.96:0/657722255 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbedc0b32c0 msgr2=0x7fbedc14ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:36.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:36.109+0000 7fbee0a00640 1 --2- 172.21.15.96:0/657722255 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbedc0b32c0 0x7fbedc14ecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:36.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:36.109+0000 7fbee0a00640 1 -- 172.21.15.96:0/657722255 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbedc14f360 con 0x7fbedc0d0640 2024-05-28T06:24:36.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:36.109+0000 7fbee0a00640 1 --2- 172.21.15.96:0/657722255 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbedc0d0640 0x7fbedc0b14d0 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 crypto rx=0x7fbec800c7f0 tx=0x7fbec800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:36.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:36.109+0000 7fbed9600640 1 -- 172.21.15.96:0/657722255 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec8015070 con 0x7fbedc0d0640 2024-05-28T06:24:36.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:36.109+0000 7fbed9600640 1 -- 172.21.15.96:0/657722255 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbec8004220 con 0x7fbedc0d0640 2024-05-28T06:24:36.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:36.109+0000 7fbed9600640 1 -- 172.21.15.96:0/657722255 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec8007620 con 0x7fbedc0d0640 2024-05-28T06:24:36.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:36.109+0000 7fbee3e00640 1 -- 172.21.15.96:0/657722255 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbedc14f640 con 0x7fbedc0d0640 2024-05-28T06:24:36.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:36.110+0000 7fbee3e00640 1 -- 172.21.15.96:0/657722255 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbedc14fc00 con 0x7fbedc0d0640 2024-05-28T06:24:36.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:36.111+0000 7fbee3e00640 1 -- 172.21.15.96:0/657722255 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbedc0689e0 con 0x7fbedc0d0640 2024-05-28T06:24:36.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:36.112+0000 7fbed9600640 1 -- 172.21.15.96:0/657722255 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 42) ==== 93590+0+0 (secure 0 0 0) 0x7fbec80079d0 con 0x7fbedc0d0640 2024-05-28T06:24:36.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:36.113+0000 7fbed9600640 1 --2- 172.21.15.96:0/657722255 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbebc06e510 0x7fbebc0709d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:24:36.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:36.113+0000 7fbed9600640 1 -- 172.21.15.96:0/657722255 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 5765+0+0 (secure 0 0 0) 0x7fbec808aee0 con 0x7fbedc0d0640 2024-05-28T06:24:36.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:36.113+0000 7fbee1400640 1 --2- 172.21.15.96:0/657722255 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbebc06e510 0x7fbebc0709d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:24:36.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:36.115+0000 7fbee1400640 1 --2- 172.21.15.96:0/657722255 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbebc06e510 0x7fbebc0709d0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fbed0000f60 tx=0x7fbed0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:24:36.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:36.117+0000 7fbed9600640 1 -- 172.21.15.96:0/657722255 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbec805ad70 con 0x7fbedc0d0640 2024-05-28T06:24:36.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:36.331+0000 7fbee3e00640 1 -- 172.21.15.96:0/657722255 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) -- 0x7fbedc156440 con 0x7fbebc06e510 2024-05-28T06:24:36.337 INFO:teuthology.orchestra.run.smithi096.stdout:/volumes/_nogroup/sv1/aeb880aa-33ae-4200-bc32-7e8a9ed82480 2024-05-28T06:24:36.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:36.343+0000 7fbed9600640 1 -- 172.21.15.96:0/657722255 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+59 (secure 0 0 0) 0x7fbedc156440 con 0x7fbebc06e510 2024-05-28T06:24:36.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:36.345+0000 7fbee3e00640 1 -- 172.21.15.96:0/657722255 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbebc06e510 msgr2=0x7fbebc0709d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:36.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:36.345+0000 7fbee3e00640 1 --2- 172.21.15.96:0/657722255 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbebc06e510 0x7fbebc0709d0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fbed0000f60 tx=0x7fbed0009040 comp rx=0 tx=0).stop 2024-05-28T06:24:36.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:36.345+0000 7fbee3e00640 1 -- 172.21.15.96:0/657722255 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbedc0d0640 msgr2=0x7fbedc0b14d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:24:36.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:36.345+0000 7fbee3e00640 1 --2- 172.21.15.96:0/657722255 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbedc0d0640 0x7fbedc0b14d0 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 crypto rx=0x7fbec800c7f0 tx=0x7fbec800ccb0 comp rx=0 tx=0).stop 2024-05-28T06:24:36.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:36.345+0000 7fbee3e00640 1 -- 172.21.15.96:0/657722255 shutdown_connections 2024-05-28T06:24:36.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:36.345+0000 7fbee3e00640 1 --2- 172.21.15.96:0/657722255 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fbebc06e510 0x7fbebc0709d0 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:36.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:36.345+0000 7fbee3e00640 1 --2- 172.21.15.96:0/657722255 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fbedc15d5d0 0x7fbedc14d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:36.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:36.345+0000 7fbee3e00640 1 --2- 172.21.15.96:0/657722255 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbedc0d0640 0x7fbedc0b14d0 unknown :-1 s=CLOSED pgs=2063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:36.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:36.345+0000 7fbee3e00640 1 --2- 172.21.15.96:0/657722255 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbedc0b32c0 0x7fbedc14ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:24:36.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:36.345+0000 7fbee3e00640 1 -- 172.21.15.96:0/657722255 >> 172.21.15.96:0/657722255 conn(0x7fbedc06bd20 msgr2=0x7fbedc072110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:24:36.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:36.346+0000 7fbee3e00640 1 -- 172.21.15.96:0/657722255 shutdown_connections 2024-05-28T06:24:36.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:24:36.346+0000 7fbee3e00640 1 -- 172.21.15.96:0/657722255 wait complete. 2024-05-28T06:24:36.356 DEBUG:tasks.cephfs.test_volumes:filling subvolume sv1 with 3 files each 1024MB size under directory volumes/_nogroup/sv1/aeb880aa-33ae-4200-bc32-7e8a9ed82480 2024-05-28T06:24:36.357 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:24:36.357 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:24:36.408 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:24:36.408 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/aeb880aa-33ae-4200-bc32-7e8a9ed82480/subvolume_file.0 bs=1M conv=fdatasync count=1024 seek=0) 2024-05-28T06:24:50.158 INFO:teuthology.orchestra.run.smithi096.stderr:1024+0 records in 2024-05-28T06:24:50.158 INFO:teuthology.orchestra.run.smithi096.stderr:1024+0 records out 2024-05-28T06:24:50.159 INFO:teuthology.orchestra.run.smithi096.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 13.6766 s, 78.5 MB/s 2024-05-28T06:24:50.160 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:24:50.160 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:24:50.216 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:24:50.216 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/aeb880aa-33ae-4200-bc32-7e8a9ed82480/subvolume_file.1 bs=1M conv=fdatasync count=1024 seek=0) 2024-05-28T06:25:01.485 INFO:teuthology.orchestra.run.smithi096.stderr:1024+0 records in 2024-05-28T06:25:01.486 INFO:teuthology.orchestra.run.smithi096.stderr:1024+0 records out 2024-05-28T06:25:01.486 INFO:teuthology.orchestra.run.smithi096.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 11.2146 s, 95.7 MB/s 2024-05-28T06:25:01.487 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:25:01.487 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:25:01.541 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:25:01.541 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/aeb880aa-33ae-4200-bc32-7e8a9ed82480/subvolume_file.2 bs=1M conv=fdatasync count=1024 seek=0) 2024-05-28T06:25:05.005 DEBUG:teuthology.orchestra.run.smithi096:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:25:05.016 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:25:14.301 INFO:teuthology.orchestra.run.smithi096.stderr:1024+0 records in 2024-05-28T06:25:14.301 INFO:teuthology.orchestra.run.smithi096.stderr:1024+0 records out 2024-05-28T06:25:14.301 INFO:teuthology.orchestra.run.smithi096.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 12.7065 s, 84.5 MB/s 2024-05-28T06:25:14.303 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot create cephfs sv1 ss1 2024-05-28T06:25:14.511 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:14.516+0000 7ff459000640 1 Processor -- start 2024-05-28T06:25:14.511 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:14.517+0000 7ff459000640 1 -- start start 2024-05-28T06:25:14.512 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:14.518+0000 7ff459000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7ff45406edf0 0x7ff45413fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:14.512 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:14.518+0000 7ff459000640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4540d0020 con 0x7ff45406f250 2024-05-28T06:25:14.512 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:14.518+0000 7ff459000640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff45406dd50 con 0x7ff4540cf2f0 2024-05-28T06:25:14.512 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:14.518+0000 7ff459000640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7ff45406e030 con 0x7ff45406edf0 2024-05-28T06:25:14.513 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:14.518+0000 7ff44be00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7ff45406edf0 0x7ff45413fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:14.513 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:14.519+0000 7ff452800640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7ff45406f250 0x7ff4540ad130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:56490/0 (socket says 172.21.15.96:56490) 2024-05-28T06:25:14.513 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:14.519+0000 7ff452800640 1 -- 172.21.15.96:0/2989406774 learned_addr learned my addr 172.21.15.96:0/2989406774 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:25:14.514 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:14.519+0000 7ff451400640 1 -- 172.21.15.96:0/2989406774 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2217840900 0 0) 0x7ff4540d0020 con 0x7ff45406f250 2024-05-28T06:25:14.514 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:14.519+0000 7ff451400640 1 -- 172.21.15.96:0/2989406774 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff438003520 con 0x7ff45406f250 2024-05-28T06:25:14.514 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:14.519+0000 7ff44be00640 1 -- 172.21.15.96:0/2989406774 >> v1:172.21.15.123:6789/0 conn(0x7ff4540cf2f0 legacy=0x7ff4540cf910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:25:14.514 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:14.519+0000 7ff44be00640 1 -- 172.21.15.96:0/2989406774 >> v1:172.21.15.96:6789/0 conn(0x7ff45406f250 legacy=0x7ff4540ad130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:14.514 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:14.519+0000 7ff44be00640 1 -- 172.21.15.96:0/2989406774 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff45406e310 con 0x7ff45406edf0 2024-05-28T06:25:14.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:14.520+0000 7ff44be00640 1 --2- 172.21.15.96:0/2989406774 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff45406edf0 0x7ff45413fdf0 secure :-1 s=READY pgs=2065 cs=0 l=1 rev1=1 crypto rx=0x7ff43c0097a0 tx=0x7ff43c02a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce95eaa26e6aa09 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:25:14.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:14.520+0000 7ff451400640 1 -- 172.21.15.96:0/2989406774 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff43c037070 con 0x7ff45406edf0 2024-05-28T06:25:14.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:14.520+0000 7ff451400640 1 -- 172.21.15.96:0/2989406774 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff43c02ae50 con 0x7ff45406edf0 2024-05-28T06:25:14.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:14.520+0000 7ff451400640 1 -- 172.21.15.96:0/2989406774 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff43c032660 con 0x7ff45406edf0 2024-05-28T06:25:14.516 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:14.521+0000 7ff459000640 1 -- 172.21.15.96:0/2989406774 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff45406edf0 msgr2=0x7ff45413fdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:14.516 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:14.521+0000 7ff459000640 1 --2- 172.21.15.96:0/2989406774 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff45406edf0 0x7ff45413fdf0 secure :-1 s=READY pgs=2065 cs=0 l=1 rev1=1 crypto rx=0x7ff43c0097a0 tx=0x7ff43c02a140 comp rx=0 tx=0).stop 2024-05-28T06:25:14.516 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:14.521+0000 7ff459000640 1 -- 172.21.15.96:0/2989406774 shutdown_connections 2024-05-28T06:25:14.516 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:14.521+0000 7ff459000640 1 --2- 172.21.15.96:0/2989406774 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff45406edf0 0x7ff45413fdf0 unknown :-1 s=CLOSED pgs=2065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:14.516 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:14.521+0000 7ff459000640 1 -- 172.21.15.96:0/2989406774 >> 172.21.15.96:0/2989406774 conn(0x7ff4540b1e50 msgr2=0x7ff4540b2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:25:14.516 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:14.521+0000 7ff459000640 1 -- 172.21.15.96:0/2989406774 shutdown_connections 2024-05-28T06:25:14.516 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:14.521+0000 7ff459000640 1 -- 172.21.15.96:0/2989406774 wait complete. 2024-05-28T06:25:14.517 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:14.523+0000 7ff459000640 1 Processor -- start 2024-05-28T06:25:14.517 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:14.523+0000 7ff459000640 1 -- start start 2024-05-28T06:25:14.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:14.524+0000 7ff459000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff45406edf0 0x7ff45415ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:14.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:14.524+0000 7ff459000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff45406f250 0x7ff45415e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:14.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:14.524+0000 7ff459000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff4540cf2f0 0x7ff4541f7c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:14.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:14.524+0000 7ff459000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff454146020 con 0x7ff45406edf0 2024-05-28T06:25:14.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:14.524+0000 7ff459000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff45406e8d0 con 0x7ff45406f250 2024-05-28T06:25:14.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:14.524+0000 7ff459000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ff4541461a0 con 0x7ff4540cf2f0 2024-05-28T06:25:14.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:14.524+0000 7ff451e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff45406edf0 0x7ff45415ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:14.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:14.524+0000 7ff451e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff45406edf0 0x7ff45415ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:53990/0 (socket says 172.21.15.96:53990) 2024-05-28T06:25:14.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:14.524+0000 7ff451e00640 1 -- 172.21.15.96:0/2280220415 learned_addr learned my addr 172.21.15.96:0/2280220415 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:25:14.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:14.525+0000 7ff44be00640 1 --2- 172.21.15.96:0/2280220415 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff45406f250 0x7ff45415e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:14.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:14.525+0000 7ff452800640 1 --2- 172.21.15.96:0/2280220415 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff4540cf2f0 0x7ff4541f7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:14.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:14.525+0000 7ff451e00640 1 -- 172.21.15.96:0/2280220415 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff4540cf2f0 msgr2=0x7ff4541f7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:14.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:14.525+0000 7ff451e00640 1 --2- 172.21.15.96:0/2280220415 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff4540cf2f0 0x7ff4541f7c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:14.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:14.525+0000 7ff451e00640 1 -- 172.21.15.96:0/2280220415 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff45406f250 msgr2=0x7ff45415e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:14.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:14.525+0000 7ff451e00640 1 --2- 172.21.15.96:0/2280220415 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff45406f250 0x7ff45415e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:14.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:14.525+0000 7ff451e00640 1 -- 172.21.15.96:0/2280220415 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff454156a70 con 0x7ff45406edf0 2024-05-28T06:25:14.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:14.525+0000 7ff451e00640 1 --2- 172.21.15.96:0/2280220415 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff45406edf0 0x7ff45415ed80 secure :-1 s=READY pgs=2066 cs=0 l=1 rev1=1 crypto rx=0x7ff44400c430 tx=0x7ff44400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:25:14.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:14.525+0000 7ff44a000640 1 -- 172.21.15.96:0/2280220415 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff444011790 con 0x7ff45406edf0 2024-05-28T06:25:14.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:14.525+0000 7ff44a000640 1 -- 172.21.15.96:0/2280220415 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff444017070 con 0x7ff45406edf0 2024-05-28T06:25:14.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:14.525+0000 7ff44a000640 1 -- 172.21.15.96:0/2280220415 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff444011bb0 con 0x7ff45406edf0 2024-05-28T06:25:14.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:14.525+0000 7ff459000640 1 -- 172.21.15.96:0/2280220415 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff454156d50 con 0x7ff45406edf0 2024-05-28T06:25:14.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:14.526+0000 7ff459000640 1 -- 172.21.15.96:0/2280220415 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff454157310 con 0x7ff45406edf0 2024-05-28T06:25:14.523 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:14.527+0000 7ff459000640 1 -- 172.21.15.96:0/2280220415 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4540689e0 con 0x7ff45406edf0 2024-05-28T06:25:14.523 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:14.528+0000 7ff44a000640 1 -- 172.21.15.96:0/2280220415 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 43) ==== 93692+0+0 (secure 0 0 0) 0x7ff44401c460 con 0x7ff45406edf0 2024-05-28T06:25:14.523 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:14.528+0000 7ff44a000640 1 --2- 172.21.15.96:0/2280220415 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff43006e630 0x7ff430070af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:14.523 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:14.529+0000 7ff44a000640 1 -- 172.21.15.96:0/2280220415 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 5823+0+0 (secure 0 0 0) 0x7ff44408aa20 con 0x7ff45406edf0 2024-05-28T06:25:14.523 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:14.529+0000 7ff44be00640 1 --2- 172.21.15.96:0/2280220415 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff43006e630 0x7ff430070af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:14.524 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:14.530+0000 7ff44be00640 1 --2- 172.21.15.96:0/2280220415 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff43006e630 0x7ff430070af0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7ff43c034000 tx=0x7ff43c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:25:14.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:14.533+0000 7ff44a000640 1 -- 172.21.15.96:0/2280220415 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff44405a930 con 0x7ff45406edf0 2024-05-28T06:25:14.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:14.771+0000 7ff459000640 1 -- 172.21.15.96:0/2280220415 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""]}) -- 0x7ff454073f30 con 0x7ff43006e630 2024-05-28T06:25:14.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:14.808+0000 7ff44a000640 1 -- 172.21.15.96:0/2280220415 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7ff454073f30 con 0x7ff43006e630 2024-05-28T06:25:14.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:14.811+0000 7ff459000640 1 -- 172.21.15.96:0/2280220415 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff43006e630 msgr2=0x7ff430070af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:14.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:14.811+0000 7ff459000640 1 --2- 172.21.15.96:0/2280220415 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff43006e630 0x7ff430070af0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7ff43c034000 tx=0x7ff43c02e000 comp rx=0 tx=0).stop 2024-05-28T06:25:14.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:14.811+0000 7ff459000640 1 -- 172.21.15.96:0/2280220415 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff45406edf0 msgr2=0x7ff45415ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:14.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:14.811+0000 7ff459000640 1 --2- 172.21.15.96:0/2280220415 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff45406edf0 0x7ff45415ed80 secure :-1 s=READY pgs=2066 cs=0 l=1 rev1=1 crypto rx=0x7ff44400c430 tx=0x7ff44400c8f0 comp rx=0 tx=0).stop 2024-05-28T06:25:14.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:14.812+0000 7ff459000640 1 -- 172.21.15.96:0/2280220415 shutdown_connections 2024-05-28T06:25:14.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:14.812+0000 7ff459000640 1 --2- 172.21.15.96:0/2280220415 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff43006e630 0x7ff430070af0 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:14.807 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:14.812+0000 7ff459000640 1 --2- 172.21.15.96:0/2280220415 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff4540cf2f0 0x7ff4541f7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:14.807 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:14.812+0000 7ff459000640 1 --2- 172.21.15.96:0/2280220415 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff45406f250 0x7ff45415e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:14.807 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:14.812+0000 7ff459000640 1 --2- 172.21.15.96:0/2280220415 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff45406edf0 0x7ff45415ed80 unknown :-1 s=CLOSED pgs=2066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:14.807 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:14.812+0000 7ff459000640 1 -- 172.21.15.96:0/2280220415 >> 172.21.15.96:0/2280220415 conn(0x7ff4540b1e50 msgr2=0x7ff4540ce260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:25:14.807 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:14.812+0000 7ff459000640 1 -- 172.21.15.96:0/2280220415 shutdown_connections 2024-05-28T06:25:14.807 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:14.812+0000 7ff459000640 1 -- 172.21.15.96:0/2280220415 wait complete. 2024-05-28T06:25:14.826 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-05-28T06:25:15.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:15.036+0000 7fa95aa00640 1 Processor -- start 2024-05-28T06:25:15.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:15.036+0000 7fa95aa00640 1 -- start start 2024-05-28T06:25:15.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:15.038+0000 7fa95aa00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fa9540cb1e0 0x7fa9540cc0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:15.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:15.038+0000 7fa95aa00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa9540cc620 0x7fa95406b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:15.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:15.038+0000 7fa95aa00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9541449d0 con 0x7fa9541445f0 2024-05-28T06:25:15.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:15.038+0000 7fa95aa00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fa9540cca90 con 0x7fa9540cb1e0 2024-05-28T06:25:15.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:15.038+0000 7fa95aa00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fa9540d0490 con 0x7fa9540cc620 2024-05-28T06:25:15.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:15.039+0000 7fa953e00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7fa9541445f0 0x7fa95415bbb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.96:43268/0 (socket says 172.21.15.96:43268) 2024-05-28T06:25:15.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:15.039+0000 7fa953e00640 1 -- 172.21.15.96:0/199424106 learned_addr learned my addr 172.21.15.96:0/199424106 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:25:15.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:15.039+0000 7fa958a00640 1 --2- 172.21.15.96:0/199424106 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa9540cc620 0x7fa95406b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:15.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:15.039+0000 7fa953400640 1 --2- 172.21.15.96:0/199424106 >> v2:172.21.15.123:3300/0 conn(0x7fa9540cb1e0 0x7fa9540cc0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:15.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:15.040+0000 7fa958a00640 1 -- 172.21.15.96:0/199424106 >> v2:172.21.15.123:3300/0 conn(0x7fa9540cb1e0 msgr2=0x7fa9540cc0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:25:15.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:15.040+0000 7fa958a00640 1 --2- 172.21.15.96:0/199424106 >> v2:172.21.15.123:3300/0 conn(0x7fa9540cb1e0 0x7fa9540cc0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:15.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:15.040+0000 7fa958a00640 1 -- 172.21.15.96:0/199424106 >> v1:172.21.15.123:6789/0 conn(0x7fa9541445f0 legacy=0x7fa95415bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:15.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:15.040+0000 7fa958a00640 1 -- 172.21.15.96:0/199424106 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9540cf880 con 0x7fa9540cc620 2024-05-28T06:25:15.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:15.041+0000 7fa958a00640 1 --2- 172.21.15.96:0/199424106 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa9540cc620 0x7fa95406b900 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa94c0097a0 tx=0x7fa94c02a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b58053ffa216cdb8 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:25:15.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:15.042+0000 7fa952a00640 1 -- 172.21.15.96:0/199424106 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa94c037070 con 0x7fa9540cc620 2024-05-28T06:25:15.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:15.042+0000 7fa952a00640 1 -- 172.21.15.96:0/199424106 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa94c032440 con 0x7fa9540cc620 2024-05-28T06:25:15.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:15.042+0000 7fa952a00640 1 -- 172.21.15.96:0/199424106 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa94c032810 con 0x7fa9540cc620 2024-05-28T06:25:15.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:15.043+0000 7fa95aa00640 1 -- 172.21.15.96:0/199424106 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa9540cc620 msgr2=0x7fa95406b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:15.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:15.043+0000 7fa95aa00640 1 --2- 172.21.15.96:0/199424106 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa9540cc620 0x7fa95406b900 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa94c0097a0 tx=0x7fa94c02a180 comp rx=0 tx=0).stop 2024-05-28T06:25:15.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:15.043+0000 7fa95aa00640 1 -- 172.21.15.96:0/199424106 shutdown_connections 2024-05-28T06:25:15.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:15.043+0000 7fa95aa00640 1 --2- 172.21.15.96:0/199424106 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa9540cc620 0x7fa95406b900 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:15.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:15.043+0000 7fa95aa00640 1 --2- 172.21.15.96:0/199424106 >> v2:172.21.15.123:3300/0 conn(0x7fa9540cb1e0 0x7fa9540cc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:15.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:15.043+0000 7fa95aa00640 1 -- 172.21.15.96:0/199424106 >> 172.21.15.96:0/199424106 conn(0x7fa9540cd530 msgr2=0x7fa9540cd940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:25:15.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:15.043+0000 7fa95aa00640 1 -- 172.21.15.96:0/199424106 shutdown_connections 2024-05-28T06:25:15.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:15.043+0000 7fa95aa00640 1 -- 172.21.15.96:0/199424106 wait complete. 2024-05-28T06:25:15.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:15.045+0000 7fa95aa00640 1 Processor -- start 2024-05-28T06:25:15.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:15.045+0000 7fa95aa00640 1 -- start start 2024-05-28T06:25:15.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:15.046+0000 7fa95aa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa9540cb1e0 0x7fa954159ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:15.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:15.046+0000 7fa95aa00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa9540cc620 0x7fa954159850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:15.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:15.046+0000 7fa95aa00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa9541445f0 0x7fa95413ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:15.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:15.046+0000 7fa95aa00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa9540d0120 con 0x7fa9540cb1e0 2024-05-28T06:25:15.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:15.046+0000 7fa95aa00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa9540cfe40 con 0x7fa9540cc620 2024-05-28T06:25:15.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:15.046+0000 7fa95aa00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fa9540676b0 con 0x7fa9541445f0 2024-05-28T06:25:15.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:15.046+0000 7fa953e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa9540cb1e0 0x7fa954159ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:15.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:15.047+0000 7fa953e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa9540cb1e0 0x7fa954159ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:53992/0 (socket says 172.21.15.96:53992) 2024-05-28T06:25:15.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:15.047+0000 7fa953e00640 1 -- 172.21.15.96:0/2864645537 learned_addr learned my addr 172.21.15.96:0/2864645537 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:25:15.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:15.047+0000 7fa958a00640 1 --2- 172.21.15.96:0/2864645537 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa9541445f0 0x7fa95413ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:15.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:15.047+0000 7fa953400640 1 --2- 172.21.15.96:0/2864645537 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa9540cc620 0x7fa954159850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:15.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:15.047+0000 7fa953e00640 1 -- 172.21.15.96:0/2864645537 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa9541445f0 msgr2=0x7fa95413ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:15.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:15.047+0000 7fa953e00640 1 --2- 172.21.15.96:0/2864645537 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa9541445f0 0x7fa95413ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:15.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:15.047+0000 7fa953e00640 1 -- 172.21.15.96:0/2864645537 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa9540cc620 msgr2=0x7fa954159850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:15.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:15.047+0000 7fa953e00640 1 --2- 172.21.15.96:0/2864645537 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa9540cc620 0x7fa954159850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:15.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:15.047+0000 7fa953e00640 1 -- 172.21.15.96:0/2864645537 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa95413f380 con 0x7fa9540cb1e0 2024-05-28T06:25:15.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:15.047+0000 7fa953e00640 1 --2- 172.21.15.96:0/2864645537 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa9540cb1e0 0x7fa954159ec0 secure :-1 s=READY pgs=2067 cs=0 l=1 rev1=1 crypto rx=0x7fa94400c3f0 tx=0x7fa94400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:25:15.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:15.048+0000 7fa94be00640 1 -- 172.21.15.96:0/2864645537 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa944017070 con 0x7fa9540cb1e0 2024-05-28T06:25:15.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:15.048+0000 7fa94be00640 1 -- 172.21.15.96:0/2864645537 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa944005ce0 con 0x7fa9540cb1e0 2024-05-28T06:25:15.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:15.048+0000 7fa94be00640 1 -- 172.21.15.96:0/2864645537 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9440072f0 con 0x7fa9540cb1e0 2024-05-28T06:25:15.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:15.048+0000 7fa95aa00640 1 -- 172.21.15.96:0/2864645537 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa95415c860 con 0x7fa9540cb1e0 2024-05-28T06:25:15.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:15.048+0000 7fa95aa00640 1 -- 172.21.15.96:0/2864645537 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9540b53d0 con 0x7fa9540cb1e0 2024-05-28T06:25:15.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:15.049+0000 7fa95aa00640 1 -- 172.21.15.96:0/2864645537 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9540689e0 con 0x7fa9540cb1e0 2024-05-28T06:25:15.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:15.050+0000 7fa94be00640 1 -- 172.21.15.96:0/2864645537 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 43) ==== 93692+0+0 (secure 0 0 0) 0x7fa94401a070 con 0x7fa9540cb1e0 2024-05-28T06:25:15.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:15.051+0000 7fa94be00640 1 --2- 172.21.15.96:0/2864645537 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa93006e630 0x7fa930070af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:15.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:15.051+0000 7fa94be00640 1 -- 172.21.15.96:0/2864645537 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 5823+0+0 (secure 0 0 0) 0x7fa944089ee0 con 0x7fa9540cb1e0 2024-05-28T06:25:15.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:15.052+0000 7fa953400640 1 --2- 172.21.15.96:0/2864645537 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa93006e630 0x7fa930070af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:15.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:15.053+0000 7fa953400640 1 --2- 172.21.15.96:0/2864645537 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa93006e630 0x7fa930070af0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fa95415a910 tx=0x7fa93c00b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:25:15.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:15.056+0000 7fa94be00640 1 -- 172.21.15.96:0/2864645537 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa944059d70 con 0x7fa9540cb1e0 2024-05-28T06:25:15.281 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:15.286+0000 7fa95aa00640 1 -- 172.21.15.96:0/2864645537 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) -- 0x7fa954070b70 con 0x7fa93006e630 2024-05-28T06:25:15.293 INFO:teuthology.orchestra.run.smithi096.stdout:/volumes/_nogroup/sv1/aeb880aa-33ae-4200-bc32-7e8a9ed82480 2024-05-28T06:25:15.293 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:15.299+0000 7fa94be00640 1 -- 172.21.15.96:0/2864645537 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+59 (secure 0 0 0) 0x7fa954070b70 con 0x7fa93006e630 2024-05-28T06:25:15.296 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:15.301+0000 7fa95aa00640 1 -- 172.21.15.96:0/2864645537 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa93006e630 msgr2=0x7fa930070af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:15.296 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:15.301+0000 7fa95aa00640 1 --2- 172.21.15.96:0/2864645537 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa93006e630 0x7fa930070af0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fa95415a910 tx=0x7fa93c00b040 comp rx=0 tx=0).stop 2024-05-28T06:25:15.296 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:15.301+0000 7fa95aa00640 1 -- 172.21.15.96:0/2864645537 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa9540cb1e0 msgr2=0x7fa954159ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:15.296 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:15.301+0000 7fa95aa00640 1 --2- 172.21.15.96:0/2864645537 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa9540cb1e0 0x7fa954159ec0 secure :-1 s=READY pgs=2067 cs=0 l=1 rev1=1 crypto rx=0x7fa94400c3f0 tx=0x7fa94400c8b0 comp rx=0 tx=0).stop 2024-05-28T06:25:15.296 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:15.301+0000 7fa95aa00640 1 -- 172.21.15.96:0/2864645537 shutdown_connections 2024-05-28T06:25:15.296 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:15.301+0000 7fa95aa00640 1 --2- 172.21.15.96:0/2864645537 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa93006e630 0x7fa930070af0 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:15.296 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:15.301+0000 7fa95aa00640 1 --2- 172.21.15.96:0/2864645537 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa9541445f0 0x7fa95413ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:15.296 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:15.301+0000 7fa95aa00640 1 --2- 172.21.15.96:0/2864645537 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa9540cc620 0x7fa954159850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:15.296 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:15.301+0000 7fa95aa00640 1 --2- 172.21.15.96:0/2864645537 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa9540cb1e0 0x7fa954159ec0 unknown :-1 s=CLOSED pgs=2067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:15.296 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:15.301+0000 7fa95aa00640 1 -- 172.21.15.96:0/2864645537 >> 172.21.15.96:0/2864645537 conn(0x7fa9540cd530 msgr2=0x7fa954143870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:25:15.297 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:15.302+0000 7fa95aa00640 1 -- 172.21.15.96:0/2864645537 shutdown_connections 2024-05-28T06:25:15.297 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:15.302+0000 7fa95aa00640 1 -- 172.21.15.96:0/2864645537 wait complete. 2024-05-28T06:25:15.315 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:25:15.316 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/aeb880aa-33ae-4200-bc32-7e8a9ed82480 2024-05-28T06:25:15.316 DEBUG:teuthology.orchestra.run.smithi096:> ) 2024-05-28T06:25:15.371 INFO:teuthology.orchestra.run.smithi096.stdout:# file: volumes/_nogroup/sv1/aeb880aa-33ae-4200-bc32-7e8a9ed82480 2024-05-28T06:25:15.371 INFO:teuthology.orchestra.run.smithi096.stdout:ceph.dir.rbytes="3138256896" 2024-05-28T06:25:15.371 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:25:15.372 INFO:tasks.cephfs.test_volumes:r_size = 3138256896 exp_size = 3221225472 2024-05-28T06:25:17.373 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:25:17.373 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/aeb880aa-33ae-4200-bc32-7e8a9ed82480 2024-05-28T06:25:17.373 DEBUG:teuthology.orchestra.run.smithi096:> ) 2024-05-28T06:25:17.437 INFO:teuthology.orchestra.run.smithi096.stdout:# file: volumes/_nogroup/sv1/aeb880aa-33ae-4200-bc32-7e8a9ed82480 2024-05-28T06:25:17.437 INFO:teuthology.orchestra.run.smithi096.stdout:ceph.dir.rbytes="3138256896" 2024-05-28T06:25:17.437 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:25:17.438 INFO:tasks.cephfs.test_volumes:r_size = 3138256896 exp_size = 3221225472 2024-05-28T06:25:19.438 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:25:19.439 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/aeb880aa-33ae-4200-bc32-7e8a9ed82480 2024-05-28T06:25:19.439 DEBUG:teuthology.orchestra.run.smithi096:> ) 2024-05-28T06:25:19.505 INFO:teuthology.orchestra.run.smithi096.stdout:# file: volumes/_nogroup/sv1/aeb880aa-33ae-4200-bc32-7e8a9ed82480 2024-05-28T06:25:19.505 INFO:teuthology.orchestra.run.smithi096.stdout:ceph.dir.rbytes="3221225472" 2024-05-28T06:25:19.505 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:25:19.506 INFO:tasks.cephfs.test_volumes:r_size = 3221225472 exp_size = 3221225472 2024-05-28T06:25:19.506 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone10991 2024-05-28T06:25:19.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:19.726+0000 7f6713e00640 1 Processor -- start 2024-05-28T06:25:19.720 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:19.726+0000 7f6713e00640 1 -- start start 2024-05-28T06:25:19.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:19.727+0000 7f6713e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f671415f240 0x7f671415bc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:19.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:19.727+0000 7f6713e00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67140aef60 con 0x7f671415b180 2024-05-28T06:25:19.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:19.727+0000 7f6713e00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67140af200 con 0x7f671415edd0 2024-05-28T06:25:19.721 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:19.727+0000 7f6713e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f671415fc30 con 0x7f671415f240 2024-05-28T06:25:19.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:19.728+0000 7f6712a00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f671415b180 0x7f67140b4f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:37122/0 (socket says 172.21.15.96:37122) 2024-05-28T06:25:19.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:19.728+0000 7f6712a00640 1 -- 172.21.15.96:0/4224342394 learned_addr learned my addr 172.21.15.96:0/4224342394 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:25:19.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:19.728+0000 7f6713400640 1 --2- 172.21.15.96:0/4224342394 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f671415f240 0x7f671415bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:19.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:19.728+0000 7f6712000640 1 -- 172.21.15.96:0/4224342394 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3116851840 0 0) 0x7f67140aef60 con 0x7f671415b180 2024-05-28T06:25:19.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:19.728+0000 7f6712000640 1 -- 172.21.15.96:0/4224342394 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f66f8005730 con 0x7f671415b180 2024-05-28T06:25:19.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:19.729+0000 7f6712000640 1 -- 172.21.15.96:0/4224342394 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2212279315 0 0) 0x7f66f8005730 con 0x7f671415b180 2024-05-28T06:25:19.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:19.729+0000 7f6712000640 1 -- 172.21.15.96:0/4224342394 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f671415f240 msgr2=0x7f671415bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:25:19.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:19.729+0000 7f6712000640 1 --2- 172.21.15.96:0/4224342394 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f671415f240 0x7f671415bc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:19.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:19.729+0000 7f6712000640 1 -- 172.21.15.96:0/4224342394 >> v1:172.21.15.123:6789/0 conn(0x7f671415edd0 legacy=0x7f671415b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:19.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:19.729+0000 7f6712000640 1 -- 172.21.15.96:0/4224342394 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f671415f6d0 con 0x7f671415b180 2024-05-28T06:25:19.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:19.729+0000 7f6712000640 1 -- 172.21.15.96:0/4224342394 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f6704001670 con 0x7f671415b180 2024-05-28T06:25:19.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:19.729+0000 7f6712000640 1 -- 172.21.15.96:0/4224342394 >> v1:172.21.15.96:6789/0 conn(0x7f671415b180 legacy=0x7f67140b4f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:19.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:19.729+0000 7f6712000640 1 --2- 172.21.15.96:0/4224342394 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f66f8005bf0 0x7f66f8006010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:19.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:19.729+0000 7f6712000640 1 --2- 172.21.15.96:0/4224342394 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f671415f240 0x7f66f8006a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:19.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:19.729+0000 7f6712000640 1 --2- 172.21.15.96:0/4224342394 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66f8006f50 0x7f66f80073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:19.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:19.729+0000 7f6712000640 1 -- 172.21.15.96:0/4224342394 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f66f800dc80 con 0x7f66f8005bf0 2024-05-28T06:25:19.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:19.729+0000 7f6712000640 1 -- 172.21.15.96:0/4224342394 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f66f800de30 con 0x7f66f8006f50 2024-05-28T06:25:19.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:19.729+0000 7f6712000640 1 -- 172.21.15.96:0/4224342394 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f66f800e030 con 0x7f671415f240 2024-05-28T06:25:19.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:19.729+0000 7f6712a00640 1 --2- 172.21.15.96:0/4224342394 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f66f8005bf0 0x7f66f8006010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:19.725 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:19.729+0000 7f6712a00640 1 -- 172.21.15.96:0/4224342394 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f671415f240 msgr2=0x7f66f8006a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:25:19.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:19.729+0000 7f6712a00640 1 --2- 172.21.15.96:0/4224342394 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f671415f240 0x7f66f8006a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:19.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:19.730+0000 7f6712a00640 1 -- 172.21.15.96:0/4224342394 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66f8006f50 msgr2=0x7f66f80073d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:25:19.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:19.730+0000 7f6712a00640 1 --2- 172.21.15.96:0/4224342394 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66f8006f50 0x7f66f80073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:19.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:19.730+0000 7f6712a00640 1 -- 172.21.15.96:0/4224342394 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66f8005730 con 0x7f66f8005bf0 2024-05-28T06:25:19.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:19.731+0000 7f6712a00640 1 --2- 172.21.15.96:0/4224342394 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f66f8005bf0 0x7f66f8006010 secure :-1 s=READY pgs=2069 cs=0 l=1 rev1=1 crypto rx=0x7f670400c160 tx=0x7f66f8007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b414c31a2dd9796c server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:25:19.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:19.731+0000 7f6712000640 1 -- 172.21.15.96:0/4224342394 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6704007c60 con 0x7f66f8005bf0 2024-05-28T06:25:19.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:19.731+0000 7f6712000640 1 -- 172.21.15.96:0/4224342394 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6704005ad0 con 0x7f66f8005bf0 2024-05-28T06:25:19.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:19.731+0000 7f6712000640 1 -- 172.21.15.96:0/4224342394 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6704007070 con 0x7f66f8005bf0 2024-05-28T06:25:19.727 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:19.731+0000 7f6713e00640 1 -- 172.21.15.96:0/4224342394 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f66f8005bf0 msgr2=0x7f66f8006010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:19.727 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:19.731+0000 7f6713e00640 1 --2- 172.21.15.96:0/4224342394 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f66f8005bf0 0x7f66f8006010 secure :-1 s=READY pgs=2069 cs=0 l=1 rev1=1 crypto rx=0x7f670400c160 tx=0x7f66f8007a40 comp rx=0 tx=0).stop 2024-05-28T06:25:19.727 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:19.731+0000 7f6713400640 1 -- 172.21.15.96:0/4224342394 reap_dead start 2024-05-28T06:25:19.727 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:19.731+0000 7f6713e00640 1 -- 172.21.15.96:0/4224342394 shutdown_connections 2024-05-28T06:25:19.727 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:19.731+0000 7f6713e00640 1 -- 172.21.15.96:0/4224342394 >> 172.21.15.96:0/4224342394 conn(0x7f67140b1db0 msgr2=0x7f67140b21c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:25:19.727 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:19.732+0000 7f6713e00640 1 -- 172.21.15.96:0/4224342394 shutdown_connections 2024-05-28T06:25:19.727 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:19.732+0000 7f6713e00640 1 -- 172.21.15.96:0/4224342394 wait complete. 2024-05-28T06:25:19.729 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:19.733+0000 7f6713e00640 1 Processor -- start 2024-05-28T06:25:19.729 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:19.733+0000 7f6713e00640 1 -- start start 2024-05-28T06:25:19.729 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:19.734+0000 7f6713e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66f8005bf0 0x7f6714151390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:19.730 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:19.734+0000 7f6713e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f67141f3950 0x7f6714151a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:19.730 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:19.734+0000 7f6713e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6714151fa0 0x7f671414f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:19.730 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:19.734+0000 7f6713e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f67140b5610 con 0x7f6714151fa0 2024-05-28T06:25:19.730 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:19.734+0000 7f6713e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f67140b4a40 con 0x7f66f8005bf0 2024-05-28T06:25:19.730 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:19.734+0000 7f6713e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f67140b4110 con 0x7f67141f3950 2024-05-28T06:25:19.730 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:19.735+0000 7f6713400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6714151fa0 0x7f671414f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:19.730 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:19.735+0000 7f6713400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6714151fa0 0x7f671414f670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:45676/0 (socket says 172.21.15.96:45676) 2024-05-28T06:25:19.730 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:19.735+0000 7f6713400640 1 -- 172.21.15.96:0/3416266044 learned_addr learned my addr 172.21.15.96:0/3416266044 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:25:19.730 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:19.735+0000 7f6712a00640 1 --2- 172.21.15.96:0/3416266044 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66f8005bf0 0x7f6714151390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:19.730 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:19.735+0000 7f670a000640 1 --2- 172.21.15.96:0/3416266044 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f67141f3950 0x7f6714151a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:19.730 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:19.735+0000 7f6713400640 1 -- 172.21.15.96:0/3416266044 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f67141f3950 msgr2=0x7f6714151a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:19.730 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:19.735+0000 7f6713400640 1 --2- 172.21.15.96:0/3416266044 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f67141f3950 0x7f6714151a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:19.730 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:19.735+0000 7f6713400640 1 -- 172.21.15.96:0/3416266044 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66f8005bf0 msgr2=0x7f6714151390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:19.731 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:19.735+0000 7f6713400640 1 --2- 172.21.15.96:0/3416266044 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66f8005bf0 0x7f6714151390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:19.731 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:19.735+0000 7f6713400640 1 -- 172.21.15.96:0/3416266044 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67141502f0 con 0x7f6714151fa0 2024-05-28T06:25:19.731 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:19.735+0000 7f6712a00640 1 --2- 172.21.15.96:0/3416266044 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66f8005bf0 0x7f6714151390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:25:19.731 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:19.735+0000 7f670a000640 1 --2- 172.21.15.96:0/3416266044 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f67141f3950 0x7f6714151a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:25:19.731 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:19.735+0000 7f6713400640 1 --2- 172.21.15.96:0/3416266044 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6714151fa0 0x7f671414f670 secure :-1 s=READY pgs=2070 cs=0 l=1 rev1=1 crypto rx=0x7f671406ffb0 tx=0x7f670c00b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:25:19.732 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:19.736+0000 7f670b400640 1 -- 172.21.15.96:0/3416266044 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f670c018020 con 0x7f6714151fa0 2024-05-28T06:25:19.732 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:19.736+0000 7f670b400640 1 -- 172.21.15.96:0/3416266044 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f670c00ba80 con 0x7f6714151fa0 2024-05-28T06:25:19.732 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:19.736+0000 7f6713e00640 1 -- 172.21.15.96:0/3416266044 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67141505d0 con 0x7f6714151fa0 2024-05-28T06:25:19.732 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:19.736+0000 7f670b400640 1 -- 172.21.15.96:0/3416266044 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f670c014400 con 0x7f6714151fa0 2024-05-28T06:25:19.733 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:19.736+0000 7f6713e00640 1 -- 172.21.15.96:0/3416266044 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6714150b90 con 0x7f6714151fa0 2024-05-28T06:25:19.733 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:19.737+0000 7f6713e00640 1 -- 172.21.15.96:0/3416266044 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67140689e0 con 0x7f6714151fa0 2024-05-28T06:25:19.733 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:19.738+0000 7f670b400640 1 -- 172.21.15.96:0/3416266044 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 43) ==== 93692+0+0 (secure 0 0 0) 0x7f670c01b020 con 0x7f6714151fa0 2024-05-28T06:25:19.733 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:19.739+0000 7f670b400640 1 --2- 172.21.15.96:0/3416266044 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f66f006e630 0x7f66f0070af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:19.733 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:19.739+0000 7f670b400640 1 -- 172.21.15.96:0/3416266044 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 5823+0+0 (secure 0 0 0) 0x7f670c08b180 con 0x7f6714151fa0 2024-05-28T06:25:19.734 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:19.740+0000 7f6712a00640 1 --2- 172.21.15.96:0/3416266044 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f66f006e630 0x7f66f0070af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:19.735 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:19.741+0000 7f6712a00640 1 --2- 172.21.15.96:0/3416266044 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f66f006e630 0x7f66f0070af0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f66f80079f0 tx=0x7f670402d000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:25:19.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:19.744+0000 7f670b400640 1 -- 172.21.15.96:0/3416266044 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f670c022020 con 0x7f6714151fa0 2024-05-28T06:25:19.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:19.983+0000 7f6713e00640 1 -- 172.21.15.96:0/3416266044 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone10991", "target": ["mon-mgr", ""]}) -- 0x7f67140b17f0 con 0x7f66f006e630 2024-05-28T06:25:20.133 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.139+0000 7f670b400640 1 -- 172.21.15.96:0/3416266044 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f67140b17f0 con 0x7f66f006e630 2024-05-28T06:25:20.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.142+0000 7f6713e00640 1 -- 172.21.15.96:0/3416266044 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f66f006e630 msgr2=0x7f66f0070af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:20.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.142+0000 7f6713e00640 1 --2- 172.21.15.96:0/3416266044 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f66f006e630 0x7f66f0070af0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f66f80079f0 tx=0x7f670402d000 comp rx=0 tx=0).stop 2024-05-28T06:25:20.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.142+0000 7f6713e00640 1 -- 172.21.15.96:0/3416266044 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6714151fa0 msgr2=0x7f671414f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:20.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.142+0000 7f6713e00640 1 --2- 172.21.15.96:0/3416266044 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6714151fa0 0x7f671414f670 secure :-1 s=READY pgs=2070 cs=0 l=1 rev1=1 crypto rx=0x7f671406ffb0 tx=0x7f670c00b480 comp rx=0 tx=0).stop 2024-05-28T06:25:20.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.142+0000 7f6713e00640 1 -- 172.21.15.96:0/3416266044 shutdown_connections 2024-05-28T06:25:20.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.142+0000 7f6713e00640 1 --2- 172.21.15.96:0/3416266044 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f66f006e630 0x7f66f0070af0 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:20.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.143+0000 7f6713e00640 1 --2- 172.21.15.96:0/3416266044 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6714151fa0 0x7f671414f670 unknown :-1 s=CLOSED pgs=2070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:20.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.143+0000 7f6713e00640 1 --2- 172.21.15.96:0/3416266044 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f67141f3950 0x7f6714151a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:20.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.143+0000 7f6713e00640 1 --2- 172.21.15.96:0/3416266044 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66f8005bf0 0x7f6714151390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:20.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.143+0000 7f6713e00640 1 -- 172.21.15.96:0/3416266044 >> 172.21.15.96:0/3416266044 conn(0x7f67140b1db0 msgr2=0x7f67140b0a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:25:20.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.143+0000 7f6713e00640 1 -- 172.21.15.96:0/3416266044 shutdown_connections 2024-05-28T06:25:20.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.143+0000 7f6713e00640 1 -- 172.21.15.96:0/3416266044 wait complete. 2024-05-28T06:25:20.140 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:25:20.146+0000 7fea61600640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:25:20.140 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:25:20.146+0000 7fea61600640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:25:20.140 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:25:20.146+0000 7fea61600640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:25:20.140 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:25:20.146+0000 7fea61600640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:25:20.140 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:25:20.146+0000 7fea61600640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:25:20.160 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone10992 2024-05-28T06:25:20.169 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:25:20.175+0000 7fea5f800640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:25:20.169 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:25:20.175+0000 7fea5f800640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:25:20.169 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:25:20.175+0000 7fea5f800640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:25:20.169 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:25:20.175+0000 7fea5f800640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:25:20.169 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:25:20.175+0000 7fea5f800640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:25:20.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.366+0000 7f8d4aa00640 1 Processor -- start 2024-05-28T06:25:20.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.366+0000 7f8d4aa00640 1 -- start start 2024-05-28T06:25:20.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.368+0000 7f8d4aa00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8d44162c20 0x7f8d44072640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:20.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.368+0000 7f8d4aa00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f8d4406e450 0x7f8d440d06d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:20.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.368+0000 7f8d4aa00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d44072db0 con 0x7f8d440d0c10 2024-05-28T06:25:20.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.368+0000 7f8d4aa00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f8d44072b80 con 0x7f8d4406e450 2024-05-28T06:25:20.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.368+0000 7f8d4aa00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f8d44163000 con 0x7f8d44162c20 2024-05-28T06:25:20.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.368+0000 7f8d48a00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f8d440d0c10 0x7f8d440b31c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:37126/0 (socket says 172.21.15.96:37126) 2024-05-28T06:25:20.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.368+0000 7f8d48a00640 1 -- 172.21.15.96:0/560060910 learned_addr learned my addr 172.21.15.96:0/560060910 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:25:20.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.369+0000 7f8d43400640 1 --2- 172.21.15.96:0/560060910 >> v2:172.21.15.123:3300/0 conn(0x7f8d4406e450 0x7f8d440d06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:20.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.369+0000 7f8d43e00640 1 --2- 172.21.15.96:0/560060910 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8d44162c20 0x7f8d44072640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:20.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.369+0000 7f8d42a00640 1 -- 172.21.15.96:0/560060910 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 526878024 0 0) 0x7f8d44072db0 con 0x7f8d440d0c10 2024-05-28T06:25:20.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.369+0000 7f8d42a00640 1 -- 172.21.15.96:0/560060910 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8d28003520 con 0x7f8d440d0c10 2024-05-28T06:25:20.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.369+0000 7f8d42a00640 1 -- 172.21.15.96:0/560060910 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4207990318 0 0) 0x7f8d28003520 con 0x7f8d440d0c10 2024-05-28T06:25:20.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.369+0000 7f8d42a00640 1 -- 172.21.15.96:0/560060910 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8d44162c20 msgr2=0x7f8d44072640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:25:20.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.369+0000 7f8d42a00640 1 --2- 172.21.15.96:0/560060910 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8d44162c20 0x7f8d44072640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:20.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.369+0000 7f8d42a00640 1 -- 172.21.15.96:0/560060910 >> v2:172.21.15.123:3300/0 conn(0x7f8d4406e450 msgr2=0x7f8d440d06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:25:20.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.369+0000 7f8d42a00640 1 --2- 172.21.15.96:0/560060910 >> v2:172.21.15.123:3300/0 conn(0x7f8d4406e450 0x7f8d440d06d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:20.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.369+0000 7f8d42a00640 1 -- 172.21.15.96:0/560060910 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d44073090 con 0x7f8d440d0c10 2024-05-28T06:25:20.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.369+0000 7f8d42a00640 1 -- 172.21.15.96:0/560060910 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f8d3c0019f0 con 0x7f8d440d0c10 2024-05-28T06:25:20.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.369+0000 7f8d43400640 1 --2- 172.21.15.96:0/560060910 >> v2:172.21.15.123:3300/0 conn(0x7f8d4406e450 0x7f8d440d06d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-28T06:25:20.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.369+0000 7f8d42a00640 1 -- 172.21.15.96:0/560060910 >> v1:172.21.15.96:6789/0 conn(0x7f8d440d0c10 legacy=0x7f8d440b31c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:20.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.369+0000 7f8d42a00640 1 --2- 172.21.15.96:0/560060910 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d280039b0 0x7f8d28003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:20.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.369+0000 7f8d42a00640 1 --2- 172.21.15.96:0/560060910 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8d28004800 0x7f8d28004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:20.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.369+0000 7f8d42a00640 1 --2- 172.21.15.96:0/560060910 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8d44162c20 0x7f8d280051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:20.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.369+0000 7f8d42a00640 1 -- 172.21.15.96:0/560060910 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f8d44163000 con 0x7f8d280039b0 2024-05-28T06:25:20.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.369+0000 7f8d42a00640 1 -- 172.21.15.96:0/560060910 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f8d2800b9f0 con 0x7f8d28004800 2024-05-28T06:25:20.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.369+0000 7f8d42a00640 1 -- 172.21.15.96:0/560060910 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f8d2800bbf0 con 0x7f8d44162c20 2024-05-28T06:25:20.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.370+0000 7f8d43e00640 1 --2- 172.21.15.96:0/560060910 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d280039b0 0x7f8d28003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:20.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.370+0000 7f8d48a00640 1 --2- 172.21.15.96:0/560060910 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8d44162c20 0x7f8d280051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:20.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.370+0000 7f8d43400640 1 --2- 172.21.15.96:0/560060910 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8d28004800 0x7f8d28004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:20.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.370+0000 7f8d43e00640 1 -- 172.21.15.96:0/560060910 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8d44162c20 msgr2=0x7f8d280051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:25:20.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.370+0000 7f8d43e00640 1 --2- 172.21.15.96:0/560060910 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8d44162c20 0x7f8d280051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:20.367 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.370+0000 7f8d43e00640 1 -- 172.21.15.96:0/560060910 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8d28004800 msgr2=0x7f8d28004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:25:20.367 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.370+0000 7f8d43e00640 1 --2- 172.21.15.96:0/560060910 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8d28004800 0x7f8d28004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:20.367 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.370+0000 7f8d43e00640 1 -- 172.21.15.96:0/560060910 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d28003520 con 0x7f8d280039b0 2024-05-28T06:25:20.368 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.372+0000 7f8d43e00640 1 --2- 172.21.15.96:0/560060910 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d280039b0 0x7f8d28003dd0 secure :-1 s=READY pgs=2072 cs=0 l=1 rev1=1 crypto rx=0x7f8d34009a90 tx=0x7f8d28005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63594cc8ea1c799b server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:25:20.368 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.372+0000 7f8d42a00640 1 -- 172.21.15.96:0/560060910 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d34036070 con 0x7f8d280039b0 2024-05-28T06:25:20.368 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.372+0000 7f8d42a00640 1 -- 172.21.15.96:0/560060910 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d34031400 con 0x7f8d280039b0 2024-05-28T06:25:20.368 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.372+0000 7f8d42a00640 1 -- 172.21.15.96:0/560060910 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d34009690 con 0x7f8d280039b0 2024-05-28T06:25:20.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.372+0000 7f8d4aa00640 1 -- 172.21.15.96:0/560060910 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d280039b0 msgr2=0x7f8d28003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:20.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.372+0000 7f8d4aa00640 1 --2- 172.21.15.96:0/560060910 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d280039b0 0x7f8d28003dd0 secure :-1 s=READY pgs=2072 cs=0 l=1 rev1=1 crypto rx=0x7f8d34009a90 tx=0x7f8d28005830 comp rx=0 tx=0).stop 2024-05-28T06:25:20.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.372+0000 7f8d48a00640 1 -- 172.21.15.96:0/560060910 reap_dead start 2024-05-28T06:25:20.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.373+0000 7f8d4aa00640 1 -- 172.21.15.96:0/560060910 shutdown_connections 2024-05-28T06:25:20.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.373+0000 7f8d4aa00640 1 -- 172.21.15.96:0/560060910 >> 172.21.15.96:0/560060910 conn(0x7f8d4406bd20 msgr2=0x7f8d4406c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:25:20.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.373+0000 7f8d4aa00640 1 -- 172.21.15.96:0/560060910 shutdown_connections 2024-05-28T06:25:20.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.373+0000 7f8d4aa00640 1 -- 172.21.15.96:0/560060910 wait complete. 2024-05-28T06:25:20.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.374+0000 7f8d4aa00640 1 Processor -- start 2024-05-28T06:25:20.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.375+0000 7f8d4aa00640 1 -- start start 2024-05-28T06:25:20.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.375+0000 7f8d4aa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d280039b0 0x7f8d4414c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:20.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.375+0000 7f8d4aa00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8d44155840 0x7f8d44156630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:20.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.376+0000 7f8d4aa00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8d44163a20 0x7f8d440ae1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:20.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.376+0000 7f8d4aa00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f8d440d1040 con 0x7f8d280039b0 2024-05-28T06:25:20.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.376+0000 7f8d4aa00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f8d440739c0 con 0x7f8d44155840 2024-05-28T06:25:20.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.376+0000 7f8d4aa00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f8d44073650 con 0x7f8d44163a20 2024-05-28T06:25:20.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.376+0000 7f8d43e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d280039b0 0x7f8d4414c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:20.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.376+0000 7f8d43e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d280039b0 0x7f8d4414c6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:45698/0 (socket says 172.21.15.96:45698) 2024-05-28T06:25:20.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.376+0000 7f8d43e00640 1 -- 172.21.15.96:0/1275150418 learned_addr learned my addr 172.21.15.96:0/1275150418 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:25:20.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.376+0000 7f8d43e00640 1 -- 172.21.15.96:0/1275150418 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8d44163a20 msgr2=0x7f8d440ae1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:20.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.376+0000 7f8d43e00640 1 --2- 172.21.15.96:0/1275150418 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8d44163a20 0x7f8d440ae1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:20.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.376+0000 7f8d43e00640 1 -- 172.21.15.96:0/1275150418 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8d44155840 msgr2=0x7f8d44156630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:20.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.376+0000 7f8d43e00640 1 --2- 172.21.15.96:0/1275150418 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8d44155840 0x7f8d44156630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:20.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.376+0000 7f8d43e00640 1 -- 172.21.15.96:0/1275150418 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d44164e30 con 0x7f8d280039b0 2024-05-28T06:25:20.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.377+0000 7f8d43e00640 1 --2- 172.21.15.96:0/1275150418 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d280039b0 0x7f8d4414c6a0 secure :-1 s=READY pgs=2073 cs=0 l=1 rev1=1 crypto rx=0x7f8d34009a90 tx=0x7f8d34004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:25:20.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.377+0000 7f8d3be00640 1 -- 172.21.15.96:0/1275150418 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d34040020 con 0x7f8d280039b0 2024-05-28T06:25:20.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.377+0000 7f8d3be00640 1 -- 172.21.15.96:0/1275150418 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d34003b40 con 0x7f8d280039b0 2024-05-28T06:25:20.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.377+0000 7f8d3be00640 1 -- 172.21.15.96:0/1275150418 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d34036040 con 0x7f8d280039b0 2024-05-28T06:25:20.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.377+0000 7f8d4aa00640 1 -- 172.21.15.96:0/1275150418 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d44164870 con 0x7f8d280039b0 2024-05-28T06:25:20.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.377+0000 7f8d4aa00640 1 -- 172.21.15.96:0/1275150418 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d44163220 con 0x7f8d280039b0 2024-05-28T06:25:20.373 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.378+0000 7f8d4aa00640 1 -- 172.21.15.96:0/1275150418 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d440689e0 con 0x7f8d280039b0 2024-05-28T06:25:20.373 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.379+0000 7f8d3be00640 1 -- 172.21.15.96:0/1275150418 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 43) ==== 93692+0+0 (secure 0 0 0) 0x7f8d34043020 con 0x7f8d280039b0 2024-05-28T06:25:20.374 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.380+0000 7f8d3be00640 1 --2- 172.21.15.96:0/1275150418 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8d2406e630 0x7f8d24070af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:20.374 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.380+0000 7f8d3be00640 1 -- 172.21.15.96:0/1275150418 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 5823+0+0 (secure 0 0 0) 0x7f8d340aa740 con 0x7f8d280039b0 2024-05-28T06:25:20.374 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.381+0000 7f8d43400640 1 --2- 172.21.15.96:0/1275150418 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8d2406e630 0x7f8d24070af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:20.376 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.382+0000 7f8d43400640 1 --2- 172.21.15.96:0/1275150418 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8d2406e630 0x7f8d24070af0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f8d30006e80 tx=0x7f8d30006e10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:25:20.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.385+0000 7f8d3be00640 1 -- 172.21.15.96:0/1275150418 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8d340044e0 con 0x7f8d280039b0 2024-05-28T06:25:20.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:20.621+0000 7f8d4aa00640 1 -- 172.21.15.96:0/1275150418 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone10992", "target": ["mon-mgr", ""]}) -- 0x7f8d440cfd80 con 0x7f8d2406e630 2024-05-28T06:25:21.140 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:25:21.146+0000 7fe9fae00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:25:21.140 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:25:21.146+0000 7fe9fae00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:25:21.140 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:25:21.146+0000 7fe9fae00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:25:21.140 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:25:21.146+0000 7fe9fae00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:25:21.140 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:25:21.146+0000 7fe9fae00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:25:21.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:21.192+0000 7f8d3be00640 1 -- 172.21.15.96:0/1275150418 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mgrmap(e 44) ==== 93794+0+0 (secure 0 0 0) 0x7f8d34076540 con 0x7f8d280039b0 2024-05-28T06:25:21.430 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:21.436+0000 7f8d3be00640 1 -- 172.21.15.96:0/1275150418 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f8d440cfd80 con 0x7f8d2406e630 2024-05-28T06:25:21.433 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:21.439+0000 7f8d4aa00640 1 -- 172.21.15.96:0/1275150418 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8d2406e630 msgr2=0x7f8d24070af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:21.433 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:21.439+0000 7f8d4aa00640 1 --2- 172.21.15.96:0/1275150418 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8d2406e630 0x7f8d24070af0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f8d30006e80 tx=0x7f8d30006e10 comp rx=0 tx=0).stop 2024-05-28T06:25:21.433 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:21.439+0000 7f8d4aa00640 1 -- 172.21.15.96:0/1275150418 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d280039b0 msgr2=0x7f8d4414c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:21.433 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:21.439+0000 7f8d4aa00640 1 --2- 172.21.15.96:0/1275150418 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d280039b0 0x7f8d4414c6a0 secure :-1 s=READY pgs=2073 cs=0 l=1 rev1=1 crypto rx=0x7f8d34009a90 tx=0x7f8d34004730 comp rx=0 tx=0).stop 2024-05-28T06:25:21.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:21.439+0000 7f8d4aa00640 1 -- 172.21.15.96:0/1275150418 shutdown_connections 2024-05-28T06:25:21.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:21.439+0000 7f8d4aa00640 1 --2- 172.21.15.96:0/1275150418 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8d2406e630 0x7f8d24070af0 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:21.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:21.439+0000 7f8d4aa00640 1 --2- 172.21.15.96:0/1275150418 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8d44163a20 0x7f8d440ae1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:21.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:21.439+0000 7f8d4aa00640 1 --2- 172.21.15.96:0/1275150418 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8d44155840 0x7f8d44156630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:21.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:21.439+0000 7f8d4aa00640 1 --2- 172.21.15.96:0/1275150418 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d280039b0 0x7f8d4414c6a0 unknown :-1 s=CLOSED pgs=2073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:21.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:21.439+0000 7f8d4aa00640 1 -- 172.21.15.96:0/1275150418 >> 172.21.15.96:0/1275150418 conn(0x7f8d4406bd20 msgr2=0x7f8d44161890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:25:21.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:21.439+0000 7f8d4aa00640 1 -- 172.21.15.96:0/1275150418 shutdown_connections 2024-05-28T06:25:21.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:21.439+0000 7f8d4aa00640 1 -- 172.21.15.96:0/1275150418 wait complete. 2024-05-28T06:25:21.452 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:25:21.458+0000 7fea60200640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:25:21.452 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:25:21.458+0000 7fea60200640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:25:21.452 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:25:21.458+0000 7fea60200640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:25:21.452 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:25:21.458+0000 7fea60200640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:25:21.452 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:25:21.458+0000 7fea60200640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:25:21.464 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone10993 2024-05-28T06:25:21.760 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:21.765+0000 7f542e800640 1 Processor -- start 2024-05-28T06:25:21.760 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:21.765+0000 7f542e800640 1 -- start start 2024-05-28T06:25:21.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:21.767+0000 7f542e800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f542809aa30 0x7f54280ce030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:21.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:21.767+0000 7f542e800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f542809b770 0x7f5428160de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:21.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:21.767+0000 7f542e800640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f542815ded0 con 0x7f542809ae10 2024-05-28T06:25:21.763 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:21.767+0000 7f542e800640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f5428161320 con 0x7f542809b770 2024-05-28T06:25:21.763 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:21.767+0000 7f542e800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f542809be70 con 0x7f542809aa30 2024-05-28T06:25:21.763 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:21.768+0000 7f5426a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f542809b770 0x7f5428160de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:21.764 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:21.768+0000 7f5426a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f542809b770 0x7f5428160de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:45712/0 (socket says 172.21.15.96:45712) 2024-05-28T06:25:21.764 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:21.768+0000 7f5426a00640 1 -- 172.21.15.96:0/1028934297 learned_addr learned my addr 172.21.15.96:0/1028934297 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:25:21.764 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:21.768+0000 7f5427400640 1 --2- 172.21.15.96:0/1028934297 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f542809aa30 0x7f54280ce030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:21.764 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:21.769+0000 7f5426a00640 1 -- 172.21.15.96:0/1028934297 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f542809aa30 msgr2=0x7f54280ce030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:25:21.764 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:21.769+0000 7f5426a00640 1 --2- 172.21.15.96:0/1028934297 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f542809aa30 0x7f54280ce030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:21.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:21.769+0000 7f5426a00640 1 -- 172.21.15.96:0/1028934297 >> v1:172.21.15.123:6789/0 conn(0x7f542809ae10 legacy=0x7f542815e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:21.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:21.769+0000 7f5426a00640 1 -- 172.21.15.96:0/1028934297 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f542809bc00 con 0x7f542809b770 2024-05-28T06:25:21.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:21.772+0000 7f5426a00640 1 --2- 172.21.15.96:0/1028934297 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f542809b770 0x7f5428160de0 secure :-1 s=READY pgs=2074 cs=0 l=1 rev1=1 crypto rx=0x7f5418009870 tx=0x7f541802a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a218573b4fd9ae server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:25:21.767 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:21.773+0000 7f5426000640 1 -- 172.21.15.96:0/1028934297 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5418037070 con 0x7f542809b770 2024-05-28T06:25:21.767 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:21.773+0000 7f5426000640 1 -- 172.21.15.96:0/1028934297 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5418032440 con 0x7f542809b770 2024-05-28T06:25:21.767 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:21.773+0000 7f5426000640 1 -- 172.21.15.96:0/1028934297 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5418032810 con 0x7f542809b770 2024-05-28T06:25:21.768 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:21.774+0000 7f542e800640 1 -- 172.21.15.96:0/1028934297 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f542809b770 msgr2=0x7f5428160de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:21.768 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:21.774+0000 7f542e800640 1 --2- 172.21.15.96:0/1028934297 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f542809b770 0x7f5428160de0 secure :-1 s=READY pgs=2074 cs=0 l=1 rev1=1 crypto rx=0x7f5418009870 tx=0x7f541802a250 comp rx=0 tx=0).stop 2024-05-28T06:25:21.770 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:21.775+0000 7f542e800640 1 -- 172.21.15.96:0/1028934297 shutdown_connections 2024-05-28T06:25:21.770 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:21.775+0000 7f542e800640 1 --2- 172.21.15.96:0/1028934297 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f542809b770 0x7f5428160de0 unknown :-1 s=CLOSED pgs=2074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:21.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:21.775+0000 7f542e800640 1 --2- 172.21.15.96:0/1028934297 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f542809aa30 0x7f54280ce030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:21.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:21.775+0000 7f542e800640 1 -- 172.21.15.96:0/1028934297 >> 172.21.15.96:0/1028934297 conn(0x7f54280b3f20 msgr2=0x7f54280b4330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:25:21.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:21.775+0000 7f542e800640 1 -- 172.21.15.96:0/1028934297 shutdown_connections 2024-05-28T06:25:21.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:21.776+0000 7f542e800640 1 -- 172.21.15.96:0/1028934297 wait complete. 2024-05-28T06:25:21.773 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:21.779+0000 7f542e800640 1 Processor -- start 2024-05-28T06:25:21.774 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:21.779+0000 7f542e800640 1 -- start start 2024-05-28T06:25:21.774 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:21.780+0000 7f542e800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f542809aa30 0x7f542809fe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:21.775 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:21.780+0000 7f542e800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f542809ae10 0x7f5428158b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:21.775 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:21.780+0000 7f542e800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f542809b770 0x7f54280c83d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:21.775 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:21.780+0000 7f542e800640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f54280cd550 con 0x7f542809b770 2024-05-28T06:25:21.775 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:21.780+0000 7f542e800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f54280ccfd0 con 0x7f542809ae10 2024-05-28T06:25:21.775 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:21.780+0000 7f542e800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f54280cd830 con 0x7f542809aa30 2024-05-28T06:25:21.777 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:21.781+0000 7f5427e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f542809b770 0x7f54280c83d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:21.777 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:21.781+0000 7f5427e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f542809b770 0x7f54280c83d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:45728/0 (socket says 172.21.15.96:45728) 2024-05-28T06:25:21.777 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:21.781+0000 7f5427e00640 1 -- 172.21.15.96:0/529568808 learned_addr learned my addr 172.21.15.96:0/529568808 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:25:21.777 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:21.781+0000 7f5427400640 1 --2- 172.21.15.96:0/529568808 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f542809aa30 0x7f542809fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:21.777 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:21.782+0000 7f5426a00640 1 --2- 172.21.15.96:0/529568808 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f542809ae10 0x7f5428158b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:21.777 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:21.782+0000 7f5427e00640 1 -- 172.21.15.96:0/529568808 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f542809aa30 msgr2=0x7f542809fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:21.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:21.782+0000 7f5427e00640 1 --2- 172.21.15.96:0/529568808 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f542809aa30 0x7f542809fe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:21.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:21.782+0000 7f5427e00640 1 -- 172.21.15.96:0/529568808 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f542809ae10 msgr2=0x7f5428158b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:21.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:21.782+0000 7f5427e00640 1 --2- 172.21.15.96:0/529568808 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f542809ae10 0x7f5428158b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:21.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:21.782+0000 7f5427e00640 1 -- 172.21.15.96:0/529568808 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54280c9270 con 0x7f542809b770 2024-05-28T06:25:21.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:21.782+0000 7f5426a00640 1 --2- 172.21.15.96:0/529568808 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f542809ae10 0x7f5428158b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:25:21.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:21.782+0000 7f5427e00640 1 --2- 172.21.15.96:0/529568808 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f542809b770 0x7f54280c83d0 secure :-1 s=READY pgs=2075 cs=0 l=1 rev1=1 crypto rx=0x7f542000c710 tx=0x7f542000cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:25:21.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:21.783+0000 7f541f400640 1 -- 172.21.15.96:0/529568808 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5420016020 con 0x7f542809b770 2024-05-28T06:25:21.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:21.783+0000 7f541f400640 1 -- 172.21.15.96:0/529568808 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5420006d50 con 0x7f542809b770 2024-05-28T06:25:21.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:21.783+0000 7f541f400640 1 -- 172.21.15.96:0/529568808 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5420003b90 con 0x7f542809b770 2024-05-28T06:25:21.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:21.783+0000 7f542e800640 1 -- 172.21.15.96:0/529568808 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54280c8cb0 con 0x7f542809b770 2024-05-28T06:25:21.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:21.783+0000 7f542e800640 1 -- 172.21.15.96:0/529568808 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54280b6100 con 0x7f542809b770 2024-05-28T06:25:21.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:21.786+0000 7f542e800640 1 -- 172.21.15.96:0/529568808 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53fc005290 con 0x7f542809b770 2024-05-28T06:25:21.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:21.789+0000 7f541f400640 1 -- 172.21.15.96:0/529568808 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 44) ==== 93794+0+0 (secure 0 0 0) 0x7f542000a850 con 0x7f542809b770 2024-05-28T06:25:21.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:21.790+0000 7f541f400640 1 --2- 172.21.15.96:0/529568808 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f540406e7a0 0x7f5404070c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:21.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:21.790+0000 7f541f400640 1 -- 172.21.15.96:0/529568808 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 5823+0+0 (secure 0 0 0) 0x7f542008a710 con 0x7f542809b770 2024-05-28T06:25:21.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:21.797+0000 7f5427400640 1 --2- 172.21.15.96:0/529568808 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f540406e7a0 0x7f5404070c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:21.793 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:21.799+0000 7f541f400640 1 -- 172.21.15.96:0/529568808 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5420020020 con 0x7f542809b770 2024-05-28T06:25:21.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:21.805+0000 7f5427400640 1 --2- 172.21.15.96:0/529568808 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f540406e7a0 0x7f5404070c60 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f5410006fd0 tx=0x7f5410008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:25:22.186 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:22.192+0000 7f542e800640 1 -- 172.21.15.96:0/529568808 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone10993", "target": ["mon-mgr", ""]}) -- 0x7f53fc000c80 con 0x7f540406e7a0 2024-05-28T06:25:22.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:22.668+0000 7f541f400640 1 -- 172.21.15.96:0/529568808 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f53fc000c80 con 0x7f540406e7a0 2024-05-28T06:25:22.666 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:22.672+0000 7f542e800640 1 -- 172.21.15.96:0/529568808 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f540406e7a0 msgr2=0x7f5404070c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:22.666 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:22.672+0000 7f542e800640 1 --2- 172.21.15.96:0/529568808 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f540406e7a0 0x7f5404070c60 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f5410006fd0 tx=0x7f5410008040 comp rx=0 tx=0).stop 2024-05-28T06:25:22.666 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:22.672+0000 7f542e800640 1 -- 172.21.15.96:0/529568808 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f542809b770 msgr2=0x7f54280c83d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:22.666 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:22.672+0000 7f542e800640 1 --2- 172.21.15.96:0/529568808 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f542809b770 0x7f54280c83d0 secure :-1 s=READY pgs=2075 cs=0 l=1 rev1=1 crypto rx=0x7f542000c710 tx=0x7f542000cbd0 comp rx=0 tx=0).stop 2024-05-28T06:25:22.667 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:22.672+0000 7f542e800640 1 -- 172.21.15.96:0/529568808 shutdown_connections 2024-05-28T06:25:22.667 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:22.672+0000 7f542e800640 1 --2- 172.21.15.96:0/529568808 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f540406e7a0 0x7f5404070c60 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:22.667 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:22.672+0000 7f542e800640 1 --2- 172.21.15.96:0/529568808 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f542809b770 0x7f54280c83d0 unknown :-1 s=CLOSED pgs=2075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:22.667 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:22.672+0000 7f542e800640 1 --2- 172.21.15.96:0/529568808 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f542809ae10 0x7f5428158b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:22.667 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:22.672+0000 7f542e800640 1 --2- 172.21.15.96:0/529568808 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f542809aa30 0x7f542809fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:22.667 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:22.672+0000 7f542e800640 1 -- 172.21.15.96:0/529568808 >> 172.21.15.96:0/529568808 conn(0x7f54280b3f20 msgr2=0x7f542806fe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:25:22.667 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:22.673+0000 7f542e800640 1 -- 172.21.15.96:0/529568808 shutdown_connections 2024-05-28T06:25:22.667 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:22.673+0000 7f542e800640 1 -- 172.21.15.96:0/529568808 wait complete. 2024-05-28T06:25:22.692 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone10994 2024-05-28T06:25:22.726 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:25:22.732+0000 7fea60200640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:25:22.726 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:25:22.732+0000 7fea60200640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:25:22.726 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:25:22.732+0000 7fea60200640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:25:22.727 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:25:22.732+0000 7fea60200640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:25:22.727 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:25:22.732+0000 7fea60200640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:25:22.968 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:22.973+0000 7ff465c00640 1 Processor -- start 2024-05-28T06:25:22.968 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:22.974+0000 7ff465c00640 1 -- start start 2024-05-28T06:25:22.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:22.975+0000 7ff465c00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7ff46006e5d0 0x7ff46006ea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:22.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:22.975+0000 7ff465c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff4600d02b0 0x7ff4600b4300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:22.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:22.975+0000 7ff465c00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4600b3260 con 0x7ff4601447b0 2024-05-28T06:25:22.970 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:22.975+0000 7ff465c00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7ff4600b3540 con 0x7ff46006e5d0 2024-05-28T06:25:22.970 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:22.975+0000 7ff465c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ff4600b3820 con 0x7ff4600d02b0 2024-05-28T06:25:22.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:22.975+0000 7ff45e000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7ff46006e5d0 0x7ff46006ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:22.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:22.975+0000 7ff45ea00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7ff4601447b0 0x7ff4600ad6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:37134/0 (socket says 172.21.15.96:37134) 2024-05-28T06:25:22.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:22.975+0000 7ff45ea00640 1 -- 172.21.15.96:0/3300411163 learned_addr learned my addr 172.21.15.96:0/3300411163 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:25:22.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:22.976+0000 7ff45e000640 1 -- 172.21.15.96:0/3300411163 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff4600d02b0 msgr2=0x7ff4600b4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:25:22.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:22.976+0000 7ff45e000640 1 --2- 172.21.15.96:0/3300411163 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff4600d02b0 0x7ff4600b4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:22.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:22.976+0000 7ff45e000640 1 -- 172.21.15.96:0/3300411163 >> v1:172.21.15.96:6789/0 conn(0x7ff4601447b0 legacy=0x7ff4600ad6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:22.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:22.976+0000 7ff45e000640 1 -- 172.21.15.96:0/3300411163 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4600b3b00 con 0x7ff46006e5d0 2024-05-28T06:25:22.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:22.976+0000 7ff45d600640 1 -- 172.21.15.96:0/3300411163 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4041708718 0 0) 0x7ff4600b3260 con 0x7ff4601447b0 2024-05-28T06:25:22.972 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:22.977+0000 7ff45e000640 1 --2- 172.21.15.96:0/3300411163 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff46006e5d0 0x7ff46006ea60 secure :-1 s=READY pgs=2078 cs=0 l=1 rev1=1 crypto rx=0x7ff44c0097a0 tx=0x7ff44c02a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6633a22c76eb5771 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:25:22.973 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:22.977+0000 7ff45d600640 1 -- 172.21.15.96:0/3300411163 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff44c037070 con 0x7ff46006e5d0 2024-05-28T06:25:22.973 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:22.977+0000 7ff45d600640 1 -- 172.21.15.96:0/3300411163 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff44c032440 con 0x7ff46006e5d0 2024-05-28T06:25:22.973 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:22.977+0000 7ff45d600640 1 -- 172.21.15.96:0/3300411163 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff44c032810 con 0x7ff46006e5d0 2024-05-28T06:25:22.973 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:22.978+0000 7ff465c00640 1 -- 172.21.15.96:0/3300411163 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff46006e5d0 msgr2=0x7ff46006ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:22.973 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:22.978+0000 7ff465c00640 1 --2- 172.21.15.96:0/3300411163 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff46006e5d0 0x7ff46006ea60 secure :-1 s=READY pgs=2078 cs=0 l=1 rev1=1 crypto rx=0x7ff44c0097a0 tx=0x7ff44c02a180 comp rx=0 tx=0).stop 2024-05-28T06:25:22.973 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:22.978+0000 7ff465c00640 1 -- 172.21.15.96:0/3300411163 shutdown_connections 2024-05-28T06:25:22.973 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:22.978+0000 7ff465c00640 1 --2- 172.21.15.96:0/3300411163 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff4600d02b0 0x7ff4600b4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:22.973 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:22.978+0000 7ff465c00640 1 --2- 172.21.15.96:0/3300411163 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff46006e5d0 0x7ff46006ea60 unknown :-1 s=CLOSED pgs=2078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:22.973 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:22.978+0000 7ff465c00640 1 -- 172.21.15.96:0/3300411163 >> 172.21.15.96:0/3300411163 conn(0x7ff46006bd20 msgr2=0x7ff46006c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:25:22.973 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:22.979+0000 7ff465c00640 1 -- 172.21.15.96:0/3300411163 shutdown_connections 2024-05-28T06:25:22.974 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:22.979+0000 7ff465c00640 1 -- 172.21.15.96:0/3300411163 wait complete. 2024-05-28T06:25:22.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:22.981+0000 7ff465c00640 1 Processor -- start 2024-05-28T06:25:23.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:22.981+0000 7ff465c00640 1 -- start start 2024-05-28T06:25:23.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:22.982+0000 7ff465c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff46006e5d0 0x7ff46015a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:23.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:22.982+0000 7ff465c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff4600d02b0 0x7ff4601435f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:23.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:22.982+0000 7ff465c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4601447b0 0x7ff4601472d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:23.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:22.982+0000 7ff465c00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff4600d0700 con 0x7ff4600d02b0 2024-05-28T06:25:23.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:22.982+0000 7ff465c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff4600ade00 con 0x7ff4601447b0 2024-05-28T06:25:23.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:22.982+0000 7ff465c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ff4600d0880 con 0x7ff46006e5d0 2024-05-28T06:25:23.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:22.983+0000 7ff45e000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff4600d02b0 0x7ff4601435f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:23.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:22.983+0000 7ff45e000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff4600d02b0 0x7ff4601435f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:45752/0 (socket says 172.21.15.96:45752) 2024-05-28T06:25:23.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:22.983+0000 7ff45e000640 1 -- 172.21.15.96:0/3995649946 learned_addr learned my addr 172.21.15.96:0/3995649946 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:25:23.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:22.984+0000 7ff45f400640 1 --2- 172.21.15.96:0/3995649946 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4601447b0 0x7ff4601472d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:23.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:22.984+0000 7ff45ea00640 1 --2- 172.21.15.96:0/3995649946 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff46006e5d0 0x7ff46015a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:23.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:22.984+0000 7ff45e000640 1 -- 172.21.15.96:0/3995649946 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff46006e5d0 msgr2=0x7ff46015a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:23.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:22.984+0000 7ff45e000640 1 --2- 172.21.15.96:0/3995649946 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff46006e5d0 0x7ff46015a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:23.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:22.984+0000 7ff45e000640 1 -- 172.21.15.96:0/3995649946 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4601447b0 msgr2=0x7ff4601472d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:23.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:22.984+0000 7ff45e000640 1 --2- 172.21.15.96:0/3995649946 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4601447b0 0x7ff4601472d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:23.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:22.985+0000 7ff45e000640 1 -- 172.21.15.96:0/3995649946 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff46015ac00 con 0x7ff4600d02b0 2024-05-28T06:25:23.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:22.985+0000 7ff45ea00640 1 --2- 172.21.15.96:0/3995649946 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff46006e5d0 0x7ff46015a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-28T06:25:23.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:22.985+0000 7ff45e000640 1 --2- 172.21.15.96:0/3995649946 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff4600d02b0 0x7ff4601435f0 secure :-1 s=READY pgs=2079 cs=0 l=1 rev1=1 crypto rx=0x7ff44c0097a0 tx=0x7ff44c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:25:23.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:22.986+0000 7ff456a00640 1 -- 172.21.15.96:0/3995649946 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff44c041070 con 0x7ff4600d02b0 2024-05-28T06:25:23.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:22.986+0000 7ff456a00640 1 -- 172.21.15.96:0/3995649946 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff44c03c870 con 0x7ff4600d02b0 2024-05-28T06:25:23.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:22.986+0000 7ff456a00640 1 -- 172.21.15.96:0/3995649946 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff44c037040 con 0x7ff4600d02b0 2024-05-28T06:25:23.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:22.986+0000 7ff465c00640 1 -- 172.21.15.96:0/3995649946 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4600cebc0 con 0x7ff4600d02b0 2024-05-28T06:25:23.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:22.986+0000 7ff465c00640 1 -- 172.21.15.96:0/3995649946 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4600cf180 con 0x7ff4600d02b0 2024-05-28T06:25:23.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:22.988+0000 7ff465c00640 1 -- 172.21.15.96:0/3995649946 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4600689e0 con 0x7ff4600d02b0 2024-05-28T06:25:23.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:22.991+0000 7ff456a00640 1 -- 172.21.15.96:0/3995649946 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 44) ==== 93794+0+0 (secure 0 0 0) 0x7ff44c044070 con 0x7ff4600d02b0 2024-05-28T06:25:23.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:22.993+0000 7ff456a00640 1 --2- 172.21.15.96:0/3995649946 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff44006e7a0 0x7ff440070c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:23.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:22.993+0000 7ff456a00640 1 -- 172.21.15.96:0/3995649946 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 5823+0+0 (secure 0 0 0) 0x7ff44c0ab510 con 0x7ff4600d02b0 2024-05-28T06:25:23.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:23.001+0000 7ff456a00640 1 -- 172.21.15.96:0/3995649946 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff44c07aff0 con 0x7ff4600d02b0 2024-05-28T06:25:23.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:23.010+0000 7ff45ea00640 1 --2- 172.21.15.96:0/3995649946 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff44006e7a0 0x7ff440070c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:23.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:23.018+0000 7ff45ea00640 1 --2- 172.21.15.96:0/3995649946 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff44006e7a0 0x7ff440070c60 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7ff450000f60 tx=0x7ff450009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:25:23.236 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:23.241+0000 7ff456a00640 1 -- 172.21.15.96:0/3995649946 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mgrmap(e 45) ==== 93896+0+0 (secure 0 0 0) 0x7ff44c03a070 con 0x7ff4600d02b0 2024-05-28T06:25:23.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:23.411+0000 7ff465c00640 1 -- 172.21.15.96:0/3995649946 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone10994", "target": ["mon-mgr", ""]}) -- 0x7ff4600ae720 con 0x7ff44006e7a0 2024-05-28T06:25:24.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.116+0000 7ff456a00640 1 -- 172.21.15.96:0/3995649946 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7ff4600ae720 con 0x7ff44006e7a0 2024-05-28T06:25:24.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.119+0000 7ff465c00640 1 -- 172.21.15.96:0/3995649946 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff44006e7a0 msgr2=0x7ff440070c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:24.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.119+0000 7ff465c00640 1 --2- 172.21.15.96:0/3995649946 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff44006e7a0 0x7ff440070c60 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7ff450000f60 tx=0x7ff450009040 comp rx=0 tx=0).stop 2024-05-28T06:25:24.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.119+0000 7ff465c00640 1 -- 172.21.15.96:0/3995649946 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff4600d02b0 msgr2=0x7ff4601435f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:24.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.119+0000 7ff465c00640 1 --2- 172.21.15.96:0/3995649946 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff4600d02b0 0x7ff4601435f0 secure :-1 s=READY pgs=2079 cs=0 l=1 rev1=1 crypto rx=0x7ff44c0097a0 tx=0x7ff44c0046c0 comp rx=0 tx=0).stop 2024-05-28T06:25:24.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.119+0000 7ff465c00640 1 -- 172.21.15.96:0/3995649946 shutdown_connections 2024-05-28T06:25:24.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.119+0000 7ff465c00640 1 --2- 172.21.15.96:0/3995649946 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff44006e7a0 0x7ff440070c60 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:24.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.119+0000 7ff465c00640 1 --2- 172.21.15.96:0/3995649946 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4601447b0 0x7ff4601472d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:24.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.119+0000 7ff465c00640 1 --2- 172.21.15.96:0/3995649946 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff4600d02b0 0x7ff4601435f0 unknown :-1 s=CLOSED pgs=2079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:24.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.119+0000 7ff465c00640 1 --2- 172.21.15.96:0/3995649946 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff46006e5d0 0x7ff46015a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:24.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.119+0000 7ff465c00640 1 -- 172.21.15.96:0/3995649946 >> 172.21.15.96:0/3995649946 conn(0x7ff46006bd20 msgr2=0x7ff4600b10c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:25:24.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.119+0000 7ff465c00640 1 -- 172.21.15.96:0/3995649946 shutdown_connections 2024-05-28T06:25:24.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.119+0000 7ff465c00640 1 -- 172.21.15.96:0/3995649946 wait complete. 2024-05-28T06:25:24.149 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone10995 2024-05-28T06:25:24.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.435+0000 7f5a77400640 1 Processor -- start 2024-05-28T06:25:24.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.435+0000 7f5a77400640 1 -- start start 2024-05-28T06:25:24.430 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.437+0000 7f5a77400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5a700d17b0 0x7f5a700cf2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:24.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.437+0000 7f5a77400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f5a700b5540 0x7f5a700cf800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:24.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.437+0000 7f5a77400640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a700d2d60 con 0x7f5a70162170 2024-05-28T06:25:24.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.437+0000 7f5a77400640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f5a700d3040 con 0x7f5a700b5540 2024-05-28T06:25:24.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.437+0000 7f5a77400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f5a700d3320 con 0x7f5a700d17b0 2024-05-28T06:25:24.431 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.437+0000 7f5a74a00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f5a70162170 0x7f5a700d10a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:37138/0 (socket says 172.21.15.96:37138) 2024-05-28T06:25:24.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.437+0000 7f5a74a00640 1 -- 172.21.15.96:0/3006320272 learned_addr learned my addr 172.21.15.96:0/3006320272 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:25:24.433 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.438+0000 7f5a75400640 1 --2- 172.21.15.96:0/3006320272 >> v2:172.21.15.123:3300/0 conn(0x7f5a700b5540 0x7f5a700cf800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:24.433 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.439+0000 7f5a6fe00640 1 --2- 172.21.15.96:0/3006320272 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5a700d17b0 0x7f5a700cf2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:24.433 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.439+0000 7f5a6f400640 1 -- 172.21.15.96:0/3006320272 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1520456203 0 0) 0x7f5a700d2d60 con 0x7f5a70162170 2024-05-28T06:25:24.433 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.439+0000 7f5a6f400640 1 -- 172.21.15.96:0/3006320272 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5a58003520 con 0x7f5a70162170 2024-05-28T06:25:24.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.439+0000 7f5a6f400640 1 -- 172.21.15.96:0/3006320272 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2101340030 0 0) 0x7f5a58003520 con 0x7f5a70162170 2024-05-28T06:25:24.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.439+0000 7f5a6f400640 1 -- 172.21.15.96:0/3006320272 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5a700d17b0 msgr2=0x7f5a700cf2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:25:24.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.439+0000 7f5a6f400640 1 --2- 172.21.15.96:0/3006320272 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5a700d17b0 0x7f5a700cf2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:24.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.439+0000 7f5a6f400640 1 -- 172.21.15.96:0/3006320272 >> v2:172.21.15.123:3300/0 conn(0x7f5a700b5540 msgr2=0x7f5a700cf800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:25:24.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.439+0000 7f5a6f400640 1 --2- 172.21.15.96:0/3006320272 >> v2:172.21.15.123:3300/0 conn(0x7f5a700b5540 0x7f5a700cf800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:24.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.439+0000 7f5a6f400640 1 -- 172.21.15.96:0/3006320272 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a700d3600 con 0x7f5a70162170 2024-05-28T06:25:24.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.439+0000 7f5a6f400640 1 -- 172.21.15.96:0/3006320272 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f5a640019f0 con 0x7f5a70162170 2024-05-28T06:25:24.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.440+0000 7f5a6f400640 1 -- 172.21.15.96:0/3006320272 >> v1:172.21.15.96:6789/0 conn(0x7f5a70162170 legacy=0x7f5a700d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:24.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.440+0000 7f5a6f400640 1 --2- 172.21.15.96:0/3006320272 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5a580039b0 0x7f5a58003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:24.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.440+0000 7f5a6f400640 1 --2- 172.21.15.96:0/3006320272 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5a58004800 0x7f5a58004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:24.436 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.440+0000 7f5a6f400640 1 --2- 172.21.15.96:0/3006320272 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5a700d17b0 0x7f5a580051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:24.436 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.440+0000 7f5a6f400640 1 -- 172.21.15.96:0/3006320272 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f5a700d3040 con 0x7f5a580039b0 2024-05-28T06:25:24.436 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.440+0000 7f5a6f400640 1 -- 172.21.15.96:0/3006320272 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5a700d3320 con 0x7f5a58004800 2024-05-28T06:25:24.436 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.440+0000 7f5a6f400640 1 -- 172.21.15.96:0/3006320272 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f5a5800bac0 con 0x7f5a700d17b0 2024-05-28T06:25:24.436 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.441+0000 7f5a74a00640 1 --2- 172.21.15.96:0/3006320272 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5a580039b0 0x7f5a58003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:24.436 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.441+0000 7f5a74a00640 1 -- 172.21.15.96:0/3006320272 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5a700d17b0 msgr2=0x7f5a580051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:25:24.436 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.441+0000 7f5a74a00640 1 --2- 172.21.15.96:0/3006320272 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5a700d17b0 0x7f5a580051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:24.436 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.441+0000 7f5a74a00640 1 -- 172.21.15.96:0/3006320272 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5a58004800 msgr2=0x7f5a58004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:25:24.436 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.441+0000 7f5a74a00640 1 --2- 172.21.15.96:0/3006320272 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5a58004800 0x7f5a58004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:24.436 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.441+0000 7f5a74a00640 1 -- 172.21.15.96:0/3006320272 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a58003520 con 0x7f5a580039b0 2024-05-28T06:25:24.437 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.443+0000 7f5a74a00640 1 --2- 172.21.15.96:0/3006320272 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5a580039b0 0x7f5a58003dd0 secure :-1 s=READY pgs=2081 cs=0 l=1 rev1=1 crypto rx=0x7f5a6400a780 tx=0x7f5a58005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dda0f409ed6b1cb7 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:25:24.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.444+0000 7f5a6f400640 1 -- 172.21.15.96:0/3006320272 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a6400a3d0 con 0x7f5a580039b0 2024-05-28T06:25:24.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.444+0000 7f5a6f400640 1 -- 172.21.15.96:0/3006320272 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a6402f070 con 0x7f5a580039b0 2024-05-28T06:25:24.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.444+0000 7f5a6f400640 1 -- 172.21.15.96:0/3006320272 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a6402bc00 con 0x7f5a580039b0 2024-05-28T06:25:24.439 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.445+0000 7f5a77400640 1 -- 172.21.15.96:0/3006320272 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5a580039b0 msgr2=0x7f5a58003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:24.439 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.445+0000 7f5a77400640 1 --2- 172.21.15.96:0/3006320272 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5a580039b0 0x7f5a58003dd0 secure :-1 s=READY pgs=2081 cs=0 l=1 rev1=1 crypto rx=0x7f5a6400a780 tx=0x7f5a58005830 comp rx=0 tx=0).stop 2024-05-28T06:25:24.439 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.445+0000 7f5a75400640 1 -- 172.21.15.96:0/3006320272 reap_dead start 2024-05-28T06:25:24.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.446+0000 7f5a77400640 1 -- 172.21.15.96:0/3006320272 shutdown_connections 2024-05-28T06:25:24.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.446+0000 7f5a77400640 1 -- 172.21.15.96:0/3006320272 >> 172.21.15.96:0/3006320272 conn(0x7f5a7008bfc0 msgr2=0x7f5a700d46b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:25:24.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.447+0000 7f5a77400640 1 -- 172.21.15.96:0/3006320272 shutdown_connections 2024-05-28T06:25:24.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.447+0000 7f5a77400640 1 -- 172.21.15.96:0/3006320272 wait complete. 2024-05-28T06:25:24.444 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.450+0000 7f5a77400640 1 Processor -- start 2024-05-28T06:25:24.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.451+0000 7f5a77400640 1 -- start start 2024-05-28T06:25:24.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.452+0000 7f5a77400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5a580039b0 0x7f5a700ce0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:24.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.452+0000 7f5a77400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5a700cf280 0x7f5a700cf6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:24.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.452+0000 7f5a77400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5a700b9950 0x7f5a700b9dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:24.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.452+0000 7f5a77400640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f5a700b37a0 con 0x7f5a580039b0 2024-05-28T06:25:24.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.452+0000 7f5a77400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5a700d29e0 con 0x7f5a700cf280 2024-05-28T06:25:24.447 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.452+0000 7f5a77400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f5a700b3920 con 0x7f5a700b9950 2024-05-28T06:25:24.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.453+0000 7f5a74a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5a580039b0 0x7f5a700ce0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:24.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.453+0000 7f5a74a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5a580039b0 0x7f5a700ce0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:45762/0 (socket says 172.21.15.96:45762) 2024-05-28T06:25:24.448 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.453+0000 7f5a74a00640 1 -- 172.21.15.96:0/4065267557 learned_addr learned my addr 172.21.15.96:0/4065267557 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:25:24.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.454+0000 7f5a74a00640 1 -- 172.21.15.96:0/4065267557 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5a700b9950 msgr2=0x7f5a700b9dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:24.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.454+0000 7f5a74a00640 1 --2- 172.21.15.96:0/4065267557 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5a700b9950 0x7f5a700b9dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:24.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.454+0000 7f5a74a00640 1 -- 172.21.15.96:0/4065267557 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5a700cf280 msgr2=0x7f5a700cf6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:24.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.454+0000 7f5a74a00640 1 --2- 172.21.15.96:0/4065267557 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5a700cf280 0x7f5a700cf6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:24.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.454+0000 7f5a74a00640 1 -- 172.21.15.96:0/4065267557 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a7015bf00 con 0x7f5a580039b0 2024-05-28T06:25:24.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.455+0000 7f5a74a00640 1 --2- 172.21.15.96:0/4065267557 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5a580039b0 0x7f5a700ce0b0 secure :-1 s=READY pgs=2082 cs=0 l=1 rev1=1 crypto rx=0x7f5a6402b430 tx=0x7f5a7008e570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:25:24.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.455+0000 7f5a6cc00640 1 -- 172.21.15.96:0/4065267557 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a64044070 con 0x7f5a580039b0 2024-05-28T06:25:24.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.455+0000 7f5a77400640 1 -- 172.21.15.96:0/4065267557 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a701fc180 con 0x7f5a580039b0 2024-05-28T06:25:24.451 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.456+0000 7f5a77400640 1 -- 172.21.15.96:0/4065267557 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a701fc740 con 0x7f5a580039b0 2024-05-28T06:25:24.452 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.458+0000 7f5a6cc00640 1 -- 172.21.15.96:0/4065267557 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a6400a3d0 con 0x7f5a580039b0 2024-05-28T06:25:24.452 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.458+0000 7f5a6cc00640 1 -- 172.21.15.96:0/4065267557 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a6404e060 con 0x7f5a580039b0 2024-05-28T06:25:24.454 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.460+0000 7f5a6cc00640 1 -- 172.21.15.96:0/4065267557 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 45) ==== 93896+0+0 (secure 0 0 0) 0x7f5a64047070 con 0x7f5a580039b0 2024-05-28T06:25:24.454 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.461+0000 7f5a77400640 1 -- 172.21.15.96:0/4065267557 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a4c005290 con 0x7f5a580039b0 2024-05-28T06:25:24.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.461+0000 7f5a6cc00640 1 --2- 172.21.15.96:0/4065267557 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5a5006e870 0x7f5a50070d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:24.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.462+0000 7f5a6cc00640 1 -- 172.21.15.96:0/4065267557 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 5823+0+0 (secure 0 0 0) 0x7f5a640ab090 con 0x7f5a580039b0 2024-05-28T06:25:24.458 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.464+0000 7f5a6fe00640 1 --2- 172.21.15.96:0/4065267557 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5a5006e870 0x7f5a50070d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:24.461 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.467+0000 7f5a6fe00640 1 --2- 172.21.15.96:0/4065267557 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5a5006e870 0x7f5a50070d30 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f5a5c006f00 tx=0x7f5a5c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:25:24.468 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.473+0000 7f5a6cc00640 1 -- 172.21.15.96:0/4065267557 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5a7015bf00 con 0x7f5a580039b0 2024-05-28T06:25:24.821 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:24.827+0000 7f5a77400640 1 -- 172.21.15.96:0/4065267557 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone10995", "target": ["mon-mgr", ""]}) -- 0x7f5a4c000c80 con 0x7f5a5006e870 2024-05-28T06:25:25.374 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:25.377+0000 7f5a6cc00640 1 -- 172.21.15.96:0/4065267557 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f5a4c000c80 con 0x7f5a5006e870 2024-05-28T06:25:25.377 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:25.382+0000 7f5a6cc00640 1 -- 172.21.15.96:0/4065267557 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mgrmap(e 46) ==== 93947+0+0 (secure 0 0 0) 0x7f5a64077100 con 0x7f5a580039b0 2024-05-28T06:25:25.377 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:25.383+0000 7f5a77400640 1 -- 172.21.15.96:0/4065267557 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5a5006e870 msgr2=0x7f5a50070d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:25.378 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:25.383+0000 7f5a77400640 1 --2- 172.21.15.96:0/4065267557 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5a5006e870 0x7f5a50070d30 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f5a5c006f00 tx=0x7f5a5c008040 comp rx=0 tx=0).stop 2024-05-28T06:25:25.378 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:25.383+0000 7f5a77400640 1 -- 172.21.15.96:0/4065267557 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5a580039b0 msgr2=0x7f5a700ce0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:25.378 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:25.383+0000 7f5a77400640 1 --2- 172.21.15.96:0/4065267557 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5a580039b0 0x7f5a700ce0b0 secure :-1 s=READY pgs=2082 cs=0 l=1 rev1=1 crypto rx=0x7f5a6402b430 tx=0x7f5a7008e570 comp rx=0 tx=0).stop 2024-05-28T06:25:25.378 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:25.383+0000 7f5a77400640 1 -- 172.21.15.96:0/4065267557 shutdown_connections 2024-05-28T06:25:25.378 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:25.383+0000 7f5a77400640 1 --2- 172.21.15.96:0/4065267557 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5a5006e870 0x7f5a50070d30 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:25.378 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:25.383+0000 7f5a77400640 1 --2- 172.21.15.96:0/4065267557 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5a700b9950 0x7f5a700b9dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:25.378 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:25.383+0000 7f5a77400640 1 --2- 172.21.15.96:0/4065267557 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5a700cf280 0x7f5a700cf6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:25.378 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:25.383+0000 7f5a77400640 1 --2- 172.21.15.96:0/4065267557 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5a580039b0 0x7f5a700ce0b0 unknown :-1 s=CLOSED pgs=2082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:25.378 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:25.383+0000 7f5a77400640 1 -- 172.21.15.96:0/4065267557 >> 172.21.15.96:0/4065267557 conn(0x7f5a7008bfc0 msgr2=0x7f5a700d10a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:25:25.378 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:25.383+0000 7f5a77400640 1 -- 172.21.15.96:0/4065267557 shutdown_connections 2024-05-28T06:25:25.378 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:25.383+0000 7f5a77400640 1 -- 172.21.15.96:0/4065267557 wait complete. 2024-05-28T06:25:25.400 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone10996 2024-05-28T06:25:25.671 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:25.677+0000 7fa06be00640 1 Processor -- start 2024-05-28T06:25:25.671 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:25.677+0000 7fa06be00640 1 -- start start 2024-05-28T06:25:25.673 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:25.678+0000 7fa06be00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fa06415adb0 0x7fa0640d06d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:25.673 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:25.679+0000 7fa06be00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa06406e610 0x7fa064072f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:25.673 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:25.679+0000 7fa06be00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa06406ec90 con 0x7fa064072aa0 2024-05-28T06:25:25.673 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:25.679+0000 7fa06be00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fa064073450 con 0x7fa06415adb0 2024-05-28T06:25:25.673 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:25.679+0000 7fa06be00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fa06406ef70 con 0x7fa06406e610 2024-05-28T06:25:25.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:25.679+0000 7fa069400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fa06415adb0 0x7fa0640d06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:25.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:25.679+0000 7fa069400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fa06415adb0 0x7fa0640d06d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:45778/0 (socket says 172.21.15.96:45778) 2024-05-28T06:25:25.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:25.679+0000 7fa069400640 1 -- 172.21.15.96:0/357955853 learned_addr learned my addr 172.21.15.96:0/357955853 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:25:25.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:25.680+0000 7fa069e00640 1 --2- 172.21.15.96:0/357955853 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa06406e610 0x7fa064072f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:25.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:25.680+0000 7fa069400640 1 -- 172.21.15.96:0/357955853 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa06406e610 msgr2=0x7fa064072f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:25:25.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:25.680+0000 7fa069400640 1 --2- 172.21.15.96:0/357955853 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa06406e610 0x7fa064072f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:25.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:25.680+0000 7fa069400640 1 -- 172.21.15.96:0/357955853 >> v1:172.21.15.123:6789/0 conn(0x7fa064072aa0 legacy=0x7fa06406df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:25:25.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:25.680+0000 7fa069400640 1 -- 172.21.15.96:0/357955853 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0640b32d0 con 0x7fa06415adb0 2024-05-28T06:25:25.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:25.682+0000 7fa069400640 1 --2- 172.21.15.96:0/357955853 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa06415adb0 0x7fa0640d06d0 secure :-1 s=READY pgs=2083 cs=0 l=1 rev1=1 crypto rx=0x7fa058009870 tx=0x7fa05802a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=effac35f3079dec1 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:25:25.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:25.683+0000 7fa063e00640 1 -- 172.21.15.96:0/357955853 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa058037070 con 0x7fa06415adb0 2024-05-28T06:25:25.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:25.683+0000 7fa063e00640 1 -- 172.21.15.96:0/357955853 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa058032440 con 0x7fa06415adb0 2024-05-28T06:25:25.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:25.683+0000 7fa063e00640 1 -- 172.21.15.96:0/357955853 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa058032810 con 0x7fa06415adb0 2024-05-28T06:25:25.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:25.684+0000 7fa06be00640 1 -- 172.21.15.96:0/357955853 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa06415adb0 msgr2=0x7fa0640d06d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:25.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:25.684+0000 7fa06be00640 1 --2- 172.21.15.96:0/357955853 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa06415adb0 0x7fa0640d06d0 secure :-1 s=READY pgs=2083 cs=0 l=1 rev1=1 crypto rx=0x7fa058009870 tx=0x7fa05802a210 comp rx=0 tx=0).stop 2024-05-28T06:25:25.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:25.685+0000 7fa06be00640 1 -- 172.21.15.96:0/357955853 shutdown_connections 2024-05-28T06:25:25.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:25.685+0000 7fa06be00640 1 --2- 172.21.15.96:0/357955853 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa06406e610 0x7fa064072f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:25.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:25.685+0000 7fa06be00640 1 --2- 172.21.15.96:0/357955853 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa06415adb0 0x7fa0640d06d0 unknown :-1 s=CLOSED pgs=2083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:25.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:25.685+0000 7fa06be00640 1 -- 172.21.15.96:0/357955853 >> 172.21.15.96:0/357955853 conn(0x7fa06406bd20 msgr2=0x7fa06406c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:25:25.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:25.685+0000 7fa06be00640 1 -- 172.21.15.96:0/357955853 shutdown_connections 2024-05-28T06:25:25.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:25.685+0000 7fa06be00640 1 -- 172.21.15.96:0/357955853 wait complete. 2024-05-28T06:25:25.683 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:25.689+0000 7fa06be00640 1 Processor -- start 2024-05-28T06:25:25.684 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:25.689+0000 7fa06be00640 1 -- start start 2024-05-28T06:25:25.684 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:25.690+0000 7fa06be00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa06406e610 0x7fa0641618b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:25.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:25.690+0000 7fa06be00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa064072aa0 0x7fa06414bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:25.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:25.690+0000 7fa06be00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa06415adb0 0x7fa0641f8e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:25.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:25.690+0000 7fa06be00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa0640d0c10 con 0x7fa06406e610 2024-05-28T06:25:25.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:25.690+0000 7fa06be00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa06406f250 con 0x7fa064072aa0 2024-05-28T06:25:25.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:25.690+0000 7fa06be00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fa0640d0d90 con 0x7fa06415adb0 2024-05-28T06:25:25.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:25.691+0000 7fa069400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa06406e610 0x7fa0641618b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:25.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:25.691+0000 7fa069400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa06406e610 0x7fa0641618b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:45794/0 (socket says 172.21.15.96:45794) 2024-05-28T06:25:25.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:25.691+0000 7fa069400640 1 -- 172.21.15.96:0/127106210 learned_addr learned my addr 172.21.15.96:0/127106210 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:25:25.687 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:25.691+0000 7fa069e00640 1 --2- 172.21.15.96:0/127106210 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa06415adb0 0x7fa0641f8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:25.687 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:25.691+0000 7fa068a00640 1 --2- 172.21.15.96:0/127106210 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa064072aa0 0x7fa06414bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:25.687 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:25.691+0000 7fa069400640 1 -- 172.21.15.96:0/127106210 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa06415adb0 msgr2=0x7fa0641f8e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:25.687 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:25.691+0000 7fa069400640 1 --2- 172.21.15.96:0/127106210 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa06415adb0 0x7fa0641f8e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:25.687 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:25.691+0000 7fa069400640 1 -- 172.21.15.96:0/127106210 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa064072aa0 msgr2=0x7fa06414bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:25.687 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:25.691+0000 7fa069400640 1 --2- 172.21.15.96:0/127106210 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa064072aa0 0x7fa06414bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:25.687 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:25.691+0000 7fa069400640 1 -- 172.21.15.96:0/127106210 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa064161f40 con 0x7fa06406e610 2024-05-28T06:25:25.688 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:25.692+0000 7fa069400640 1 --2- 172.21.15.96:0/127106210 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa06406e610 0x7fa0641618b0 secure :-1 s=READY pgs=2084 cs=0 l=1 rev1=1 crypto rx=0x7fa058000c00 tx=0x7fa0580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:25:25.688 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:25.692+0000 7fa061600640 1 -- 172.21.15.96:0/127106210 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa058041020 con 0x7fa06406e610 2024-05-28T06:25:25.688 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:25.692+0000 7fa06be00640 1 -- 172.21.15.96:0/127106210 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa064162220 con 0x7fa06406e610 2024-05-28T06:25:25.688 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:25.692+0000 7fa06be00640 1 -- 172.21.15.96:0/127106210 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0641627e0 con 0x7fa06406e610 2024-05-28T06:25:25.688 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:25.692+0000 7fa061600640 1 -- 172.21.15.96:0/127106210 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa058030780 con 0x7fa06406e610 2024-05-28T06:25:25.688 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:25.692+0000 7fa061600640 1 -- 172.21.15.96:0/127106210 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa058037040 con 0x7fa06406e610 2024-05-28T06:25:25.689 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:25.693+0000 7fa06be00640 1 -- 172.21.15.96:0/127106210 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0640b39d0 con 0x7fa06406e610 2024-05-28T06:25:25.689 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:25.695+0000 7fa061600640 1 -- 172.21.15.96:0/127106210 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 46) ==== 93947+0+0 (secure 0 0 0) 0x7fa058044020 con 0x7fa06406e610 2024-05-28T06:25:25.690 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:25.696+0000 7fa061600640 1 --2- 172.21.15.96:0/127106210 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa04406e900 0x7fa044070dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:25.690 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:25.696+0000 7fa061600640 1 -- 172.21.15.96:0/127106210 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 5823+0+0 (secure 0 0 0) 0x7fa0580ab3b0 con 0x7fa06406e610 2024-05-28T06:25:25.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:25.703+0000 7fa061600640 1 -- 172.21.15.96:0/127106210 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa0640b39d0 con 0x7fa06406e610 2024-05-28T06:25:25.697 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:25.703+0000 7fa068a00640 1 --2- 172.21.15.96:0/127106210 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa04406e900 0x7fa044070dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:25.702 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:25.708+0000 7fa068a00640 1 --2- 172.21.15.96:0/127106210 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa04406e900 0x7fa044070dc0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fa0640b4ce0 tx=0x7fa050009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:25:26.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:26.113+0000 7fa06be00640 1 -- 172.21.15.96:0/127106210 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone10996", "target": ["mon-mgr", ""]}) -- 0x7fa06414b310 con 0x7fa04406e900 2024-05-28T06:25:27.061 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:27.067+0000 7fa061600640 1 -- 172.21.15.96:0/127106210 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fa050007db0 con 0x7fa04406e900 2024-05-28T06:25:27.065 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:27.071+0000 7fa06be00640 1 -- 172.21.15.96:0/127106210 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa04406e900 msgr2=0x7fa044070dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:27.065 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:27.071+0000 7fa06be00640 1 --2- 172.21.15.96:0/127106210 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa04406e900 0x7fa044070dc0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fa0640b4ce0 tx=0x7fa050009040 comp rx=0 tx=0).stop 2024-05-28T06:25:27.065 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:27.071+0000 7fa06be00640 1 -- 172.21.15.96:0/127106210 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa06406e610 msgr2=0x7fa0641618b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:27.065 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:27.071+0000 7fa06be00640 1 --2- 172.21.15.96:0/127106210 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa06406e610 0x7fa0641618b0 secure :-1 s=READY pgs=2084 cs=0 l=1 rev1=1 crypto rx=0x7fa058000c00 tx=0x7fa0580046c0 comp rx=0 tx=0).stop 2024-05-28T06:25:27.065 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:27.071+0000 7fa06be00640 1 -- 172.21.15.96:0/127106210 shutdown_connections 2024-05-28T06:25:27.065 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:27.071+0000 7fa06be00640 1 --2- 172.21.15.96:0/127106210 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa04406e900 0x7fa044070dc0 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:27.065 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:27.071+0000 7fa06be00640 1 --2- 172.21.15.96:0/127106210 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa06415adb0 0x7fa0641f8e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:27.065 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:27.071+0000 7fa06be00640 1 --2- 172.21.15.96:0/127106210 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa064072aa0 0x7fa06414bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:27.065 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:27.071+0000 7fa06be00640 1 --2- 172.21.15.96:0/127106210 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa06406e610 0x7fa0641618b0 unknown :-1 s=CLOSED pgs=2084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:27.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:27.071+0000 7fa06be00640 1 -- 172.21.15.96:0/127106210 >> 172.21.15.96:0/127106210 conn(0x7fa06406bd20 msgr2=0x7fa064073660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:25:27.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:27.071+0000 7fa06be00640 1 -- 172.21.15.96:0/127106210 shutdown_connections 2024-05-28T06:25:27.066 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:27.071+0000 7fa06be00640 1 -- 172.21.15.96:0/127106210 wait complete. 2024-05-28T06:25:27.101 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone10997 2024-05-28T06:25:27.377 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:27.382+0000 7f09b2c00640 1 Processor -- start 2024-05-28T06:25:27.377 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:27.382+0000 7f09b2c00640 1 -- start start 2024-05-28T06:25:27.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:27.385+0000 7f09b2c00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f09ac1525d0 0x7f09ac073910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:27.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:27.385+0000 7f09b2c00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f09ac0d06d0 0x7f09ac0d0b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:27.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:27.385+0000 7f09b2c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f09ac0b43d0 0x7f09ac0b4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:27.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:27.385+0000 7f09b2c00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f09ac15f050 con 0x7f09ac0d06d0 2024-05-28T06:25:27.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:27.385+0000 7f09b2c00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f09ac0d1050 con 0x7f09ac1525d0 2024-05-28T06:25:27.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:27.385+0000 7f09b2c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f09ac1529b0 con 0x7f09ac0b43d0 2024-05-28T06:25:27.381 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:27.386+0000 7f09ab400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f09ac0d06d0 0x7f09ac0d0b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:27.381 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:27.386+0000 7f09ab400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f09ac0d06d0 0x7f09ac0d0b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:45800/0 (socket says 172.21.15.96:45800) 2024-05-28T06:25:27.381 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:27.386+0000 7f09ab400640 1 -- 172.21.15.96:0/3622322070 learned_addr learned my addr 172.21.15.96:0/3622322070 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:25:27.381 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:27.387+0000 7f09b0c00640 1 --2- 172.21.15.96:0/3622322070 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f09ac0b43d0 0x7f09ac0b4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:27.381 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:27.387+0000 7f09abe00640 1 --2- 172.21.15.96:0/3622322070 >> v2:172.21.15.123:3300/0 conn(0x7f09ac1525d0 0x7f09ac073910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:27.382 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:27.387+0000 7f09ab400640 1 -- 172.21.15.96:0/3622322070 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f09ac0b43d0 msgr2=0x7f09ac0b4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:25:27.382 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:27.387+0000 7f09ab400640 1 --2- 172.21.15.96:0/3622322070 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f09ac0b43d0 0x7f09ac0b4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:27.382 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:27.387+0000 7f09ab400640 1 -- 172.21.15.96:0/3622322070 >> v2:172.21.15.123:3300/0 conn(0x7f09ac1525d0 msgr2=0x7f09ac073910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:25:27.382 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:27.387+0000 7f09ab400640 1 --2- 172.21.15.96:0/3622322070 >> v2:172.21.15.123:3300/0 conn(0x7f09ac1525d0 0x7f09ac073910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:27.382 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:27.387+0000 7f09ab400640 1 -- 172.21.15.96:0/3622322070 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09ac15f2a0 con 0x7f09ac0d06d0 2024-05-28T06:25:27.383 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:27.389+0000 7f09ab400640 1 --2- 172.21.15.96:0/3622322070 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f09ac0d06d0 0x7f09ac0d0b10 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 crypto rx=0x7f09ac072930 tx=0x7f099802a1d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5ea02c5b51294f7 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:25:27.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:27.390+0000 7f09aaa00640 1 -- 172.21.15.96:0/3622322070 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0998037070 con 0x7f09ac0d06d0 2024-05-28T06:25:27.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:27.390+0000 7f09aaa00640 1 -- 172.21.15.96:0/3622322070 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0998032440 con 0x7f09ac0d06d0 2024-05-28T06:25:27.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:27.390+0000 7f09aaa00640 1 -- 172.21.15.96:0/3622322070 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0998032810 con 0x7f09ac0d06d0 2024-05-28T06:25:27.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:27.391+0000 7f09b2c00640 1 -- 172.21.15.96:0/3622322070 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f09ac0d06d0 msgr2=0x7f09ac0d0b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:27.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:27.391+0000 7f09b2c00640 1 --2- 172.21.15.96:0/3622322070 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f09ac0d06d0 0x7f09ac0d0b10 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 crypto rx=0x7f09ac072930 tx=0x7f099802a1d0 comp rx=0 tx=0).stop 2024-05-28T06:25:27.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:27.392+0000 7f09b2c00640 1 -- 172.21.15.96:0/3622322070 shutdown_connections 2024-05-28T06:25:27.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:27.392+0000 7f09b2c00640 1 --2- 172.21.15.96:0/3622322070 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f09ac0b43d0 0x7f09ac0b4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:27.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:27.392+0000 7f09b2c00640 1 --2- 172.21.15.96:0/3622322070 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f09ac0d06d0 0x7f09ac0d0b10 unknown :-1 s=CLOSED pgs=2085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:27.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:27.392+0000 7f09b2c00640 1 --2- 172.21.15.96:0/3622322070 >> v2:172.21.15.123:3300/0 conn(0x7f09ac1525d0 0x7f09ac073910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:27.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:27.392+0000 7f09b2c00640 1 -- 172.21.15.96:0/3622322070 >> 172.21.15.96:0/3622322070 conn(0x7f09ac06bd20 msgr2=0x7f09ac06c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:25:27.388 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:27.394+0000 7f09b2c00640 1 -- 172.21.15.96:0/3622322070 shutdown_connections 2024-05-28T06:25:27.389 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:27.394+0000 7f09b2c00640 1 -- 172.21.15.96:0/3622322070 wait complete. 2024-05-28T06:25:27.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:27.400+0000 7f09b2c00640 1 Processor -- start 2024-05-28T06:25:27.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:27.402+0000 7f09b2c00640 1 -- start start 2024-05-28T06:25:27.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:27.404+0000 7f09b2c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f09ac0b43d0 0x7f09ac145310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:27.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:27.404+0000 7f09b2c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f09ac0d06d0 0x7f09ac073860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:27.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:27.404+0000 7f09b2c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09ac1525d0 0x7f09ac1f8780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:27.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:27.404+0000 7f09b2c00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f09ac15fb40 con 0x7f09ac0b43d0 2024-05-28T06:25:27.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:27.404+0000 7f09b2c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f09ac15f860 con 0x7f09ac1525d0 2024-05-28T06:25:27.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:27.404+0000 7f09b2c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f09ac15fe20 con 0x7f09ac0d06d0 2024-05-28T06:25:27.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:27.405+0000 7f09ab400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f09ac0d06d0 0x7f09ac073860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:27.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:27.405+0000 7f09ab400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f09ac0d06d0 0x7f09ac073860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.96:39900/0 (socket says 172.21.15.96:39900) 2024-05-28T06:25:27.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:27.405+0000 7f09ab400640 1 -- 172.21.15.96:0/3761719469 learned_addr learned my addr 172.21.15.96:0/3761719469 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:25:27.399 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:27.406+0000 7f09b0c00640 1 --2- 172.21.15.96:0/3761719469 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09ac1525d0 0x7f09ac1f8780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:27.401 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:27.407+0000 7f09b0c00640 1 -- 172.21.15.96:0/3761719469 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f09ac0d06d0 msgr2=0x7f09ac073860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:27.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:27.407+0000 7f09b0c00640 1 --2- 172.21.15.96:0/3761719469 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f09ac0d06d0 0x7f09ac073860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:27.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:27.407+0000 7f09b0c00640 1 -- 172.21.15.96:0/3761719469 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f09ac0b43d0 msgr2=0x7f09ac145310 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-28T06:25:27.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:27.409+0000 7f09abe00640 1 --2- 172.21.15.96:0/3761719469 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f09ac0b43d0 0x7f09ac145310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:27.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:27.409+0000 7f09b0c00640 1 --2- 172.21.15.96:0/3761719469 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f09ac0b43d0 0x7f09ac145310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:27.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:27.409+0000 7f09b0c00640 1 -- 172.21.15.96:0/3761719469 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09ac15cd30 con 0x7f09ac1525d0 2024-05-28T06:25:27.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:27.409+0000 7f09ab400640 1 --2- 172.21.15.96:0/3761719469 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f09ac0d06d0 0x7f09ac073860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-28T06:25:27.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:27.409+0000 7f09abe00640 1 --2- 172.21.15.96:0/3761719469 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f09ac0b43d0 0x7f09ac145310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:25:27.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:27.410+0000 7f09b0c00640 1 --2- 172.21.15.96:0/3761719469 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09ac1525d0 0x7f09ac1f8780 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f099c006b10 tx=0x7f099c00c950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:25:27.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:27.411+0000 7f09a3e00640 1 -- 172.21.15.96:0/3761719469 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f099c015070 con 0x7f09ac1525d0 2024-05-28T06:25:27.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:27.411+0000 7f09a3e00640 1 -- 172.21.15.96:0/3761719469 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f099c004220 con 0x7f09ac1525d0 2024-05-28T06:25:27.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:27.411+0000 7f09a3e00640 1 -- 172.21.15.96:0/3761719469 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f099c003610 con 0x7f09ac1525d0 2024-05-28T06:25:27.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:27.411+0000 7f09b2c00640 1 -- 172.21.15.96:0/3761719469 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09ac145eb0 con 0x7f09ac1525d0 2024-05-28T06:25:27.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:27.412+0000 7f09b2c00640 1 -- 172.21.15.96:0/3761719469 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09ac15cfd0 con 0x7f09ac1525d0 2024-05-28T06:25:27.416 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:27.418+0000 7f09b2c00640 1 -- 172.21.15.96:0/3761719469 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0984005290 con 0x7f09ac1525d0 2024-05-28T06:25:27.417 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:27.422+0000 7f09a3e00640 1 -- 172.21.15.96:0/3761719469 <== mon.1 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 46) ==== 93947+0+0 (secure 0 0 0) 0x7f099c0037b0 con 0x7f09ac1525d0 2024-05-28T06:25:27.418 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:27.424+0000 7f09a3e00640 1 --2- 172.21.15.96:0/3761719469 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f098c06e9d0 0x7f098c070e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:27.419 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:27.425+0000 7f09a3e00640 1 -- 172.21.15.96:0/3761719469 <== mon.1 v2:172.21.15.123:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 5823+0+0 (secure 0 0 0) 0x7f099c08a850 con 0x7f09ac1525d0 2024-05-28T06:25:27.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:27.427+0000 7f09abe00640 1 --2- 172.21.15.96:0/3761719469 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f098c06e9d0 0x7f098c070e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:27.439 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:27.443+0000 7f09a3e00640 1 -- 172.21.15.96:0/3761719469 <== mon.1 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f099c05a110 con 0x7f09ac1525d0 2024-05-28T06:25:27.445 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:27.451+0000 7f09abe00640 1 --2- 172.21.15.96:0/3761719469 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f098c06e9d0 0x7f098c070e90 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f09a4009f50 tx=0x7f09a400b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:25:27.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:27.870+0000 7f09b2c00640 1 -- 172.21.15.96:0/3761719469 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone10997", "target": ["mon-mgr", ""]}) -- 0x7f0984000c80 con 0x7f098c06e9d0 2024-05-28T06:25:28.727 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:28.733+0000 7f09a3e00640 1 -- 172.21.15.96:0/3761719469 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f0984000c80 con 0x7f098c06e9d0 2024-05-28T06:25:28.729 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:28.735+0000 7f09b2c00640 1 -- 172.21.15.96:0/3761719469 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f098c06e9d0 msgr2=0x7f098c070e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:28.729 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:28.735+0000 7f09b2c00640 1 --2- 172.21.15.96:0/3761719469 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f098c06e9d0 0x7f098c070e90 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f09a4009f50 tx=0x7f09a400b040 comp rx=0 tx=0).stop 2024-05-28T06:25:28.729 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:28.735+0000 7f09b2c00640 1 -- 172.21.15.96:0/3761719469 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09ac1525d0 msgr2=0x7f09ac1f8780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:28.730 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:28.735+0000 7f09b2c00640 1 --2- 172.21.15.96:0/3761719469 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09ac1525d0 0x7f09ac1f8780 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f099c006b10 tx=0x7f099c00c950 comp rx=0 tx=0).stop 2024-05-28T06:25:28.730 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:28.735+0000 7f09b2c00640 1 -- 172.21.15.96:0/3761719469 shutdown_connections 2024-05-28T06:25:28.730 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:28.735+0000 7f09b2c00640 1 --2- 172.21.15.96:0/3761719469 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f098c06e9d0 0x7f098c070e90 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:28.730 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:28.735+0000 7f09b2c00640 1 --2- 172.21.15.96:0/3761719469 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09ac1525d0 0x7f09ac1f8780 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:28.730 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:28.735+0000 7f09b2c00640 1 --2- 172.21.15.96:0/3761719469 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f09ac0d06d0 0x7f09ac073860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:28.730 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:28.735+0000 7f09b2c00640 1 --2- 172.21.15.96:0/3761719469 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f09ac0b43d0 0x7f09ac145310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:28.730 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:28.735+0000 7f09b2c00640 1 -- 172.21.15.96:0/3761719469 >> 172.21.15.96:0/3761719469 conn(0x7f09ac06bd20 msgr2=0x7f09ac0b3c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:25:28.730 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:28.735+0000 7f09b2c00640 1 -- 172.21.15.96:0/3761719469 shutdown_connections 2024-05-28T06:25:28.730 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:28.736+0000 7f09b2c00640 1 -- 172.21.15.96:0/3761719469 wait complete. 2024-05-28T06:25:33.762 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone10991 2024-05-28T06:25:34.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:34.073+0000 7fa4fd000640 1 Processor -- start 2024-05-28T06:25:34.067 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:34.073+0000 7fa4fd000640 1 -- start start 2024-05-28T06:25:34.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:34.074+0000 7fa4fd000640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fa4f80b68f0 0x7fa4f80cff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:34.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:34.074+0000 7fa4fd000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fa4f80b7e50 0x7fa4f80cedb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:34.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:34.074+0000 7fa4fd000640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4f815f160 con 0x7fa4f815ed80 2024-05-28T06:25:34.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:34.074+0000 7fa4fd000640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fa4f8163540 con 0x7fa4f80b7e50 2024-05-28T06:25:34.068 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:34.074+0000 7fa4fd000640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fa4f8162930 con 0x7fa4f80b68f0 2024-05-28T06:25:34.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:34.074+0000 7fa4f6800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fa4f80b7e50 0x7fa4f80cedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:34.071 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:34.075+0000 7fa4f6800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fa4f80b7e50 0x7fa4f80cedb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:50022/0 (socket says 172.21.15.96:50022) 2024-05-28T06:25:34.071 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:34.075+0000 7fa4f6800640 1 -- 172.21.15.96:0/99709693 learned_addr learned my addr 172.21.15.96:0/99709693 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:25:34.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:34.075+0000 7fa4f5400640 1 --2- 172.21.15.96:0/99709693 >> v2:172.21.15.123:3300/0 conn(0x7fa4f80b68f0 0x7fa4f80cff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:34.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:34.075+0000 7fa4f6800640 1 -- 172.21.15.96:0/99709693 >> v2:172.21.15.123:3300/0 conn(0x7fa4f80b68f0 msgr2=0x7fa4f80cff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:25:34.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:34.075+0000 7fa4f6800640 1 --2- 172.21.15.96:0/99709693 >> v2:172.21.15.123:3300/0 conn(0x7fa4f80b68f0 0x7fa4f80cff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:34.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:34.075+0000 7fa4f6800640 1 -- 172.21.15.96:0/99709693 >> v1:172.21.15.96:6789/0 conn(0x7fa4f815ed80 legacy=0x7fa4f80b61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:25:34.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:34.075+0000 7fa4f6800640 1 -- 172.21.15.96:0/99709693 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4f8162c10 con 0x7fa4f80b7e50 2024-05-28T06:25:34.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:34.081+0000 7fa4f6800640 1 --2- 172.21.15.96:0/99709693 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa4f80b7e50 0x7fa4f80cedb0 secure :-1 s=READY pgs=2086 cs=0 l=1 rev1=1 crypto rx=0x7fa4f808ce10 tx=0x7fa4f002a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8906a5ef82d7ddd4 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:25:34.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:34.082+0000 7fa4f4a00640 1 -- 172.21.15.96:0/99709693 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4f0036070 con 0x7fa4f80b7e50 2024-05-28T06:25:34.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:34.082+0000 7fa4f4a00640 1 -- 172.21.15.96:0/99709693 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4f0031600 con 0x7fa4f80b7e50 2024-05-28T06:25:34.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:34.082+0000 7fa4f4a00640 1 -- 172.21.15.96:0/99709693 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4f00319d0 con 0x7fa4f80b7e50 2024-05-28T06:25:34.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:34.086+0000 7fa4fd000640 1 -- 172.21.15.96:0/99709693 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa4f80b7e50 msgr2=0x7fa4f80cedb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:34.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:34.086+0000 7fa4fd000640 1 --2- 172.21.15.96:0/99709693 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa4f80b7e50 0x7fa4f80cedb0 secure :-1 s=READY pgs=2086 cs=0 l=1 rev1=1 crypto rx=0x7fa4f808ce10 tx=0x7fa4f002a6f0 comp rx=0 tx=0).stop 2024-05-28T06:25:34.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:34.090+0000 7fa4fd000640 1 -- 172.21.15.96:0/99709693 shutdown_connections 2024-05-28T06:25:34.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:34.090+0000 7fa4fd000640 1 --2- 172.21.15.96:0/99709693 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa4f80b7e50 0x7fa4f80cedb0 unknown :-1 s=CLOSED pgs=2086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:34.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:34.090+0000 7fa4fd000640 1 --2- 172.21.15.96:0/99709693 >> v2:172.21.15.123:3300/0 conn(0x7fa4f80b68f0 0x7fa4f80cff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:34.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:34.090+0000 7fa4fd000640 1 -- 172.21.15.96:0/99709693 >> 172.21.15.96:0/99709693 conn(0x7fa4f80d13c0 msgr2=0x7fa4f80d17d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:25:34.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:34.091+0000 7fa4fd000640 1 -- 172.21.15.96:0/99709693 shutdown_connections 2024-05-28T06:25:34.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:34.091+0000 7fa4fd000640 1 -- 172.21.15.96:0/99709693 wait complete. 2024-05-28T06:25:34.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:34.096+0000 7fa4fd000640 1 Processor -- start 2024-05-28T06:25:34.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:34.096+0000 7fa4fd000640 1 -- start start 2024-05-28T06:25:34.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:34.098+0000 7fa4fd000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa4f80b68f0 0x7fa4f814fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:34.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:34.098+0000 7fa4fd000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa4f80b7e50 0x7fa4f81510a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:34.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:34.098+0000 7fa4fd000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa4f815ed80 0x7fa4f8151b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:34.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:34.099+0000 7fa4fd000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa4f80677b0 con 0x7fa4f80b7e50 2024-05-28T06:25:34.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:34.099+0000 7fa4fd000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa4f81631d0 con 0x7fa4f80b68f0 2024-05-28T06:25:34.093 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:34.099+0000 7fa4fd000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fa4f8067930 con 0x7fa4f815ed80 2024-05-28T06:25:34.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:34.099+0000 7fa4f5400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa4f80b7e50 0x7fa4f81510a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:34.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:34.099+0000 7fa4f5400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa4f80b7e50 0x7fa4f81510a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:50032/0 (socket says 172.21.15.96:50032) 2024-05-28T06:25:34.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:34.099+0000 7fa4f5400640 1 -- 172.21.15.96:0/4138643896 learned_addr learned my addr 172.21.15.96:0/4138643896 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:25:34.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:34.100+0000 7fa4f5400640 1 -- 172.21.15.96:0/4138643896 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa4f815ed80 msgr2=0x7fa4f8151b00 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-28T06:25:34.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:34.100+0000 7fa4f5400640 1 --2- 172.21.15.96:0/4138643896 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa4f815ed80 0x7fa4f8151b00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:34.095 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:34.100+0000 7fa4f5400640 1 -- 172.21.15.96:0/4138643896 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa4f80b68f0 msgr2=0x7fa4f814fcb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-28T06:25:34.096 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:34.102+0000 7fa4f5e00640 1 --2- 172.21.15.96:0/4138643896 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa4f80b68f0 0x7fa4f814fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:34.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:34.103+0000 7fa4f5400640 1 --2- 172.21.15.96:0/4138643896 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa4f80b68f0 0x7fa4f814fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:34.098 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:34.103+0000 7fa4f5400640 1 -- 172.21.15.96:0/4138643896 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4f815fc40 con 0x7fa4f80b7e50 2024-05-28T06:25:34.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:34.106+0000 7fa4f5e00640 1 --2- 172.21.15.96:0/4138643896 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa4f80b68f0 0x7fa4f814fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:25:34.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:34.110+0000 7fa4f5400640 1 --2- 172.21.15.96:0/4138643896 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa4f80b7e50 0x7fa4f81510a0 secure :-1 s=READY pgs=2087 cs=0 l=1 rev1=1 crypto rx=0x7fa4e400d920 tx=0x7fa4e400dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:25:34.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:34.110+0000 7fa4ee000640 1 -- 172.21.15.96:0/4138643896 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4e400b780 con 0x7fa4f80b7e50 2024-05-28T06:25:34.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:34.111+0000 7fa4ee000640 1 -- 172.21.15.96:0/4138643896 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4e4019070 con 0x7fa4f80b7e50 2024-05-28T06:25:34.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:34.111+0000 7fa4ee000640 1 -- 172.21.15.96:0/4138643896 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4e400bb40 con 0x7fa4f80b7e50 2024-05-28T06:25:34.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:34.111+0000 7fa4fd000640 1 -- 172.21.15.96:0/4138643896 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4f815ff20 con 0x7fa4f80b7e50 2024-05-28T06:25:34.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:34.112+0000 7fa4fd000640 1 -- 172.21.15.96:0/4138643896 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4f81604e0 con 0x7fa4f80b7e50 2024-05-28T06:25:34.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:34.114+0000 7fa4fd000640 1 -- 172.21.15.96:0/4138643896 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4f80689e0 con 0x7fa4f80b7e50 2024-05-28T06:25:34.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:34.116+0000 7fa4ee000640 1 -- 172.21.15.96:0/4138643896 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 46) ==== 93947+0+0 (secure 0 0 0) 0x7fa4e400bce0 con 0x7fa4f80b7e50 2024-05-28T06:25:34.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:34.118+0000 7fa4ee000640 1 --2- 172.21.15.96:0/4138643896 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa4d806e950 0x7fa4d8070e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:34.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:34.118+0000 7fa4ee000640 1 -- 172.21.15.96:0/4138643896 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 5823+0+0 (secure 0 0 0) 0x7fa4e408af00 con 0x7fa4f80b7e50 2024-05-28T06:25:34.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:34.122+0000 7fa4f5e00640 1 --2- 172.21.15.96:0/4138643896 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa4d806e950 0x7fa4d8070e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:34.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:34.124+0000 7fa4ee000640 1 -- 172.21.15.96:0/4138643896 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa4e405af10 con 0x7fa4f80b7e50 2024-05-28T06:25:34.134 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:34.139+0000 7fa4f5e00640 1 --2- 172.21.15.96:0/4138643896 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa4d806e950 0x7fa4d8070e10 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fa4e800a810 tx=0x7fa4e8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:25:34.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:34.571+0000 7fa4fd000640 1 -- 172.21.15.96:0/4138643896 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone10991", "target": ["mon-mgr", ""]}) -- 0x7fa4f815a900 con 0x7fa4d806e950 2024-05-28T06:25:34.644 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:25:34.649+0000 7fea59e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:25:34.644 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:25:34.649+0000 7fea59e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:25:34.644 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:25:34.649+0000 7fea59e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:25:34.644 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:25:34.649+0000 7fea59e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:25:34.644 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:25:34.649+0000 7fea59e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:25:35.143 DEBUG:teuthology.orchestra.run.smithi096:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:25:35.149 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:25:35.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:35.729+0000 7fa4ee000640 1 -- 172.21.15.96:0/4138643896 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fa4f815a900 con 0x7fa4d806e950 2024-05-28T06:25:35.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:35.732+0000 7fa4fd000640 1 -- 172.21.15.96:0/4138643896 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa4d806e950 msgr2=0x7fa4d8070e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:35.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:35.732+0000 7fa4fd000640 1 --2- 172.21.15.96:0/4138643896 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa4d806e950 0x7fa4d8070e10 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fa4e800a810 tx=0x7fa4e8008040 comp rx=0 tx=0).stop 2024-05-28T06:25:35.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:35.732+0000 7fa4fd000640 1 -- 172.21.15.96:0/4138643896 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa4f80b7e50 msgr2=0x7fa4f81510a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:35.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:35.732+0000 7fa4fd000640 1 --2- 172.21.15.96:0/4138643896 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa4f80b7e50 0x7fa4f81510a0 secure :-1 s=READY pgs=2087 cs=0 l=1 rev1=1 crypto rx=0x7fa4e400d920 tx=0x7fa4e400dde0 comp rx=0 tx=0).stop 2024-05-28T06:25:35.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:35.732+0000 7fa4fd000640 1 -- 172.21.15.96:0/4138643896 shutdown_connections 2024-05-28T06:25:35.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:35.732+0000 7fa4fd000640 1 --2- 172.21.15.96:0/4138643896 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa4d806e950 0x7fa4d8070e10 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:35.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:35.732+0000 7fa4fd000640 1 --2- 172.21.15.96:0/4138643896 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa4f815ed80 0x7fa4f8151b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:35.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:35.732+0000 7fa4fd000640 1 --2- 172.21.15.96:0/4138643896 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa4f80b7e50 0x7fa4f81510a0 unknown :-1 s=CLOSED pgs=2087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:35.726 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:35.732+0000 7fa4fd000640 1 --2- 172.21.15.96:0/4138643896 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa4f80b68f0 0x7fa4f814fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:35.727 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:35.732+0000 7fa4fd000640 1 -- 172.21.15.96:0/4138643896 >> 172.21.15.96:0/4138643896 conn(0x7fa4f80d13c0 msgr2=0x7fa4f8158940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:25:35.727 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:35.732+0000 7fa4fd000640 1 -- 172.21.15.96:0/4138643896 shutdown_connections 2024-05-28T06:25:35.727 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:35.732+0000 7fa4fd000640 1 -- 172.21.15.96:0/4138643896 wait complete. 2024-05-28T06:25:35.748 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone10992 2024-05-28T06:25:36.022 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:36.027+0000 7f3192a00640 1 Processor -- start 2024-05-28T06:25:36.022 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:36.027+0000 7f3192a00640 1 -- start start 2024-05-28T06:25:36.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:36.029+0000 7f3192a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f318c15abe0 0x7f318c06f380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:36.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:36.029+0000 7f3192a00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f318c0725c0 0x7f318c072a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:36.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:36.029+0000 7f3192a00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f318c06e2e0 con 0x7f318c06f8c0 2024-05-28T06:25:36.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:36.029+0000 7f3192a00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f318c06e5c0 con 0x7f318c0725c0 2024-05-28T06:25:36.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:36.029+0000 7f3192a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f318c06e8a0 con 0x7f318c15abe0 2024-05-28T06:25:36.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:36.029+0000 7f318b400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f318c06f8c0 0x7f318c0b45d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:60964/0 (socket says 172.21.15.96:60964) 2024-05-28T06:25:36.023 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:36.029+0000 7f318b400640 1 -- 172.21.15.96:0/3510939173 learned_addr learned my addr 172.21.15.96:0/3510939173 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:25:36.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:36.030+0000 7f3190a00640 1 --2- 172.21.15.96:0/3510939173 >> v2:172.21.15.123:3300/0 conn(0x7f318c0725c0 0x7f318c072a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:36.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:36.030+0000 7f318aa00640 1 -- 172.21.15.96:0/3510939173 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1618800671 0 0) 0x7f318c06e2e0 con 0x7f318c06f8c0 2024-05-28T06:25:36.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:36.030+0000 7f318aa00640 1 -- 172.21.15.96:0/3510939173 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3170003520 con 0x7f318c06f8c0 2024-05-28T06:25:36.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:36.030+0000 7f318aa00640 1 -- 172.21.15.96:0/3510939173 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 949458295 0 0) 0x7f3170003520 con 0x7f318c06f8c0 2024-05-28T06:25:36.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:36.030+0000 7f318aa00640 1 -- 172.21.15.96:0/3510939173 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f318c15abe0 msgr2=0x7f318c06f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:25:36.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:36.030+0000 7f318aa00640 1 --2- 172.21.15.96:0/3510939173 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f318c15abe0 0x7f318c06f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:36.025 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:36.030+0000 7f318aa00640 1 -- 172.21.15.96:0/3510939173 >> v2:172.21.15.123:3300/0 conn(0x7f318c0725c0 msgr2=0x7f318c072a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:25:36.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:36.030+0000 7f318aa00640 1 --2- 172.21.15.96:0/3510939173 >> v2:172.21.15.123:3300/0 conn(0x7f318c0725c0 0x7f318c072a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:36.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:36.030+0000 7f318aa00640 1 -- 172.21.15.96:0/3510939173 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f318c06eb80 con 0x7f318c06f8c0 2024-05-28T06:25:36.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:36.030+0000 7f318aa00640 1 -- 172.21.15.96:0/3510939173 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f3178001670 con 0x7f318c06f8c0 2024-05-28T06:25:36.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:36.030+0000 7f318aa00640 1 -- 172.21.15.96:0/3510939173 >> v1:172.21.15.96:6789/0 conn(0x7f318c06f8c0 legacy=0x7f318c0b45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:36.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:36.030+0000 7f318aa00640 1 --2- 172.21.15.96:0/3510939173 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f31700039b0 0x7f3170003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:36.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:36.030+0000 7f318aa00640 1 --2- 172.21.15.96:0/3510939173 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3170004800 0x7f3170004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:36.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:36.030+0000 7f318aa00640 1 --2- 172.21.15.96:0/3510939173 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f318c15abe0 0x7f31700051e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:36.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:36.030+0000 7f318aa00640 1 -- 172.21.15.96:0/3510939173 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f318c06e5c0 con 0x7f3170004800 2024-05-28T06:25:36.026 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:36.030+0000 7f318aa00640 1 -- 172.21.15.96:0/3510939173 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f318c06e8a0 con 0x7f31700039b0 2024-05-28T06:25:36.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:36.030+0000 7f318aa00640 1 -- 172.21.15.96:0/3510939173 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f317000dd60 con 0x7f318c15abe0 2024-05-28T06:25:36.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:36.030+0000 7f318b400640 1 --2- 172.21.15.96:0/3510939173 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3170004800 0x7f3170004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:36.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:36.031+0000 7f318b400640 1 -- 172.21.15.96:0/3510939173 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f318c15abe0 msgr2=0x7f31700051e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:25:36.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:36.031+0000 7f318b400640 1 --2- 172.21.15.96:0/3510939173 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f318c15abe0 0x7f31700051e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:36.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:36.031+0000 7f318b400640 1 -- 172.21.15.96:0/3510939173 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f31700039b0 msgr2=0x7f3170003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:25:36.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:36.031+0000 7f318b400640 1 --2- 172.21.15.96:0/3510939173 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f31700039b0 0x7f3170003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:36.027 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:36.031+0000 7f318b400640 1 -- 172.21.15.96:0/3510939173 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3170003520 con 0x7f3170004800 2024-05-28T06:25:36.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:36.032+0000 7f318b400640 1 --2- 172.21.15.96:0/3510939173 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3170004800 0x7f3170004c80 secure :-1 s=READY pgs=2089 cs=0 l=1 rev1=1 crypto rx=0x7f317800c240 tx=0x7f3170005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dacb02e0a12b68f5 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:25:36.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:36.032+0000 7f318aa00640 1 -- 172.21.15.96:0/3510939173 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f317802e070 con 0x7f3170004800 2024-05-28T06:25:36.028 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:36.032+0000 7f318aa00640 1 -- 172.21.15.96:0/3510939173 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3178005af0 con 0x7f3170004800 2024-05-28T06:25:36.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:36.032+0000 7f318aa00640 1 -- 172.21.15.96:0/3510939173 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3178007d30 con 0x7f3170004800 2024-05-28T06:25:36.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:36.033+0000 7f3192a00640 1 -- 172.21.15.96:0/3510939173 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3170004800 msgr2=0x7f3170004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:36.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:36.033+0000 7f3192a00640 1 --2- 172.21.15.96:0/3510939173 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3170004800 0x7f3170004c80 secure :-1 s=READY pgs=2089 cs=0 l=1 rev1=1 crypto rx=0x7f317800c240 tx=0x7f3170005830 comp rx=0 tx=0).stop 2024-05-28T06:25:36.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:36.033+0000 7f3190a00640 1 -- 172.21.15.96:0/3510939173 reap_dead start 2024-05-28T06:25:36.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:36.033+0000 7f3192a00640 1 -- 172.21.15.96:0/3510939173 shutdown_connections 2024-05-28T06:25:36.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:36.033+0000 7f3192a00640 1 -- 172.21.15.96:0/3510939173 >> 172.21.15.96:0/3510939173 conn(0x7f318c06bd20 msgr2=0x7f318c06c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:25:36.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:36.034+0000 7f3192a00640 1 -- 172.21.15.96:0/3510939173 shutdown_connections 2024-05-28T06:25:36.029 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:36.034+0000 7f3192a00640 1 -- 172.21.15.96:0/3510939173 wait complete. 2024-05-28T06:25:36.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:36.037+0000 7f3192a00640 1 Processor -- start 2024-05-28T06:25:36.031 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:36.037+0000 7f3192a00640 1 -- start start 2024-05-28T06:25:36.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:36.038+0000 7f3192a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3170004800 0x7f318c1597f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:36.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:36.038+0000 7f3192a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f318c1f9970 0x7f318c09d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:36.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:36.038+0000 7f3192a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f318c1f9df0 0x7f318c0b3080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:36.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:36.038+0000 7f3192a00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f318c0b4ce0 con 0x7f318c1f9df0 2024-05-28T06:25:36.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:36.038+0000 7f3192a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f318c15afc0 con 0x7f3170004800 2024-05-28T06:25:36.032 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:36.038+0000 7f3192a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f318c06df60 con 0x7f318c1f9970 2024-05-28T06:25:36.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:36.039+0000 7f3190a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f318c1f9df0 0x7f318c0b3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:36.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:36.039+0000 7f3190a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f318c1f9df0 0x7f318c0b3080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:50048/0 (socket says 172.21.15.96:50048) 2024-05-28T06:25:36.033 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:36.039+0000 7f3190a00640 1 -- 172.21.15.96:0/3624412686 learned_addr learned my addr 172.21.15.96:0/3624412686 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:25:36.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:36.039+0000 7f3190a00640 1 -- 172.21.15.96:0/3624412686 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f318c1f9970 msgr2=0x7f318c09d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:36.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:36.039+0000 7f318b400640 1 --2- 172.21.15.96:0/3624412686 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f318c1f9970 0x7f318c09d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:36.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:36.039+0000 7f3190a00640 1 --2- 172.21.15.96:0/3624412686 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f318c1f9970 0x7f318c09d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:36.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:36.039+0000 7f3190a00640 1 -- 172.21.15.96:0/3624412686 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3170004800 msgr2=0x7f318c1597f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:36.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:36.039+0000 7f3190a00640 1 --2- 172.21.15.96:0/3624412686 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3170004800 0x7f318c1597f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:36.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:36.039+0000 7f3190a00640 1 -- 172.21.15.96:0/3624412686 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f318c159d30 con 0x7f318c1f9df0 2024-05-28T06:25:36.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:36.039+0000 7f318b400640 1 --2- 172.21.15.96:0/3624412686 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f318c1f9970 0x7f318c09d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:25:36.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:36.040+0000 7f3190a00640 1 --2- 172.21.15.96:0/3624412686 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f318c1f9df0 0x7f318c0b3080 secure :-1 s=READY pgs=2090 cs=0 l=1 rev1=1 crypto rx=0x7f3170004780 tx=0x7f318400d490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:25:36.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:36.040+0000 7f3183e00640 1 -- 172.21.15.96:0/3624412686 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f318400a0d0 con 0x7f318c1f9df0 2024-05-28T06:25:36.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:36.040+0000 7f3183e00640 1 -- 172.21.15.96:0/3624412686 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f318400a270 con 0x7f318c1f9df0 2024-05-28T06:25:36.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:36.040+0000 7f3192a00640 1 -- 172.21.15.96:0/3624412686 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f318c09aa40 con 0x7f318c1f9df0 2024-05-28T06:25:36.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:36.040+0000 7f3183e00640 1 -- 172.21.15.96:0/3624412686 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f318400dde0 con 0x7f318c1f9df0 2024-05-28T06:25:36.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:36.041+0000 7f3192a00640 1 -- 172.21.15.96:0/3624412686 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f318c09b000 con 0x7f318c1f9df0 2024-05-28T06:25:36.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:36.042+0000 7f3192a00640 1 -- 172.21.15.96:0/3624412686 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f318c0689e0 con 0x7f318c1f9df0 2024-05-28T06:25:36.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:36.044+0000 7f3183e00640 1 -- 172.21.15.96:0/3624412686 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 46) ==== 93947+0+0 (secure 0 0 0) 0x7f318401a020 con 0x7f318c1f9df0 2024-05-28T06:25:36.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:36.045+0000 7f3183e00640 1 --2- 172.21.15.96:0/3624412686 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f316c06e980 0x7f316c070e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:36.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:36.045+0000 7f3183e00640 1 -- 172.21.15.96:0/3624412686 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 5823+0+0 (secure 0 0 0) 0x7f318408a560 con 0x7f318c1f9df0 2024-05-28T06:25:36.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:36.050+0000 7f3183e00640 1 -- 172.21.15.96:0/3624412686 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3184021020 con 0x7f318c1f9df0 2024-05-28T06:25:36.085 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:36.091+0000 7f318be00640 1 --2- 172.21.15.96:0/3624412686 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f316c06e980 0x7f316c070e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:36.125 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:36.129+0000 7f318be00640 1 --2- 172.21.15.96:0/3624412686 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f316c06e980 0x7f316c070e40 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f318c144a60 tx=0x7f317c006c90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:25:36.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:36.455+0000 7f3192a00640 1 -- 172.21.15.96:0/3624412686 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone10992", "target": ["mon-mgr", ""]}) -- 0x7f318c06dd40 con 0x7f316c06e980 2024-05-28T06:25:36.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:36.954+0000 7f3183e00640 1 -- 172.21.15.96:0/3624412686 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mgrmap(e 47) ==== 93998+0+0 (secure 0 0 0) 0x7f3184091d10 con 0x7f318c1f9df0 2024-05-28T06:25:37.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:37.646+0000 7f3183e00640 1 -- 172.21.15.96:0/3624412686 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f318c06dd40 con 0x7f316c06e980 2024-05-28T06:25:37.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:37.653+0000 7f3192a00640 1 -- 172.21.15.96:0/3624412686 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f316c06e980 msgr2=0x7f316c070e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:37.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:37.653+0000 7f3192a00640 1 --2- 172.21.15.96:0/3624412686 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f316c06e980 0x7f316c070e40 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f318c144a60 tx=0x7f317c006c90 comp rx=0 tx=0).stop 2024-05-28T06:25:37.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:37.653+0000 7f3192a00640 1 -- 172.21.15.96:0/3624412686 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f318c1f9df0 msgr2=0x7f318c0b3080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:37.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:37.653+0000 7f3192a00640 1 --2- 172.21.15.96:0/3624412686 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f318c1f9df0 0x7f318c0b3080 secure :-1 s=READY pgs=2090 cs=0 l=1 rev1=1 crypto rx=0x7f3170004780 tx=0x7f318400d490 comp rx=0 tx=0).stop 2024-05-28T06:25:37.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:37.655+0000 7f3192a00640 1 -- 172.21.15.96:0/3624412686 shutdown_connections 2024-05-28T06:25:37.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:37.655+0000 7f3192a00640 1 --2- 172.21.15.96:0/3624412686 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f316c06e980 0x7f316c070e40 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:37.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:37.655+0000 7f3192a00640 1 --2- 172.21.15.96:0/3624412686 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f318c1f9df0 0x7f318c0b3080 unknown :-1 s=CLOSED pgs=2090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:37.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:37.655+0000 7f3192a00640 1 --2- 172.21.15.96:0/3624412686 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f318c1f9970 0x7f318c09d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:37.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:37.655+0000 7f3192a00640 1 --2- 172.21.15.96:0/3624412686 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3170004800 0x7f318c1597f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:37.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:37.655+0000 7f3192a00640 1 -- 172.21.15.96:0/3624412686 >> 172.21.15.96:0/3624412686 conn(0x7f318c06bd20 msgr2=0x7f318c0b1440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:25:37.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:37.656+0000 7f3192a00640 1 -- 172.21.15.96:0/3624412686 shutdown_connections 2024-05-28T06:25:37.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:37.656+0000 7f3192a00640 1 -- 172.21.15.96:0/3624412686 wait complete. 2024-05-28T06:25:37.679 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone10993 2024-05-28T06:25:37.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:37.950+0000 7f06e1e00640 1 Processor -- start 2024-05-28T06:25:37.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:37.950+0000 7f06e1e00640 1 -- start start 2024-05-28T06:25:37.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:37.951+0000 7f06e1e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f06dc0b49e0 0x7f06dc0d1bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:37.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:37.951+0000 7f06e1e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f06dc0ced60 0x7f06dc0cf1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:37.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:37.952+0000 7f06e1e00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06dc0b2290 con 0x7f06dc161ed0 2024-05-28T06:25:37.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:37.952+0000 7f06e1e00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f06dc0d0690 con 0x7f06dc0b49e0 2024-05-28T06:25:37.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:37.952+0000 7f06e1e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f06dc0b1700 con 0x7f06dc0ced60 2024-05-28T06:25:37.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:37.952+0000 7f06dbe00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f06dc0b49e0 0x7f06dc0d1bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:37.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:37.952+0000 7f06dbe00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f06dc0b49e0 0x7f06dc0d1bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.96:35864/0 (socket says 172.21.15.96:35864) 2024-05-28T06:25:37.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:37.952+0000 7f06dbe00640 1 -- 172.21.15.96:0/4112870601 learned_addr learned my addr 172.21.15.96:0/4112870601 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:25:37.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:37.953+0000 7f06e1400640 1 --2- 172.21.15.96:0/4112870601 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f06dc0ced60 0x7f06dc0cf1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:37.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:37.953+0000 7f06dbe00640 1 -- 172.21.15.96:0/4112870601 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f06dc0ced60 msgr2=0x7f06dc0cf1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:25:37.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:37.953+0000 7f06dbe00640 1 --2- 172.21.15.96:0/4112870601 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f06dc0ced60 0x7f06dc0cf1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:37.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:37.953+0000 7f06dbe00640 1 -- 172.21.15.96:0/4112870601 >> v1:172.21.15.123:6789/0 conn(0x7f06dc161ed0 legacy=0x7f06dc06ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:37.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:37.953+0000 7f06dbe00640 1 -- 172.21.15.96:0/4112870601 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06dc0d03f0 con 0x7f06dc0b49e0 2024-05-28T06:25:37.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:37.956+0000 7f06dbe00640 1 --2- 172.21.15.96:0/4112870601 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f06dc0b49e0 0x7f06dc0d1bb0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f06c8009870 tx=0x7f06c802a1c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c07e5a2c4cadeaa8 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:25:37.964 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:37.968+0000 7f06db400640 1 -- 172.21.15.96:0/4112870601 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06c8037070 con 0x7f06dc0b49e0 2024-05-28T06:25:37.964 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:37.968+0000 7f06db400640 1 -- 172.21.15.96:0/4112870601 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06c8032440 con 0x7f06dc0b49e0 2024-05-28T06:25:37.964 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:37.968+0000 7f06db400640 1 -- 172.21.15.96:0/4112870601 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06c8032810 con 0x7f06dc0b49e0 2024-05-28T06:25:37.966 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:37.970+0000 7f06e1e00640 1 -- 172.21.15.96:0/4112870601 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f06dc0b49e0 msgr2=0x7f06dc0d1bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:37.966 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:37.970+0000 7f06e1e00640 1 --2- 172.21.15.96:0/4112870601 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f06dc0b49e0 0x7f06dc0d1bb0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f06c8009870 tx=0x7f06c802a1c0 comp rx=0 tx=0).stop 2024-05-28T06:25:37.966 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:37.971+0000 7f06e1e00640 1 -- 172.21.15.96:0/4112870601 shutdown_connections 2024-05-28T06:25:37.966 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:37.971+0000 7f06e1e00640 1 --2- 172.21.15.96:0/4112870601 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f06dc0ced60 0x7f06dc0cf1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:37.966 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:37.971+0000 7f06e1e00640 1 --2- 172.21.15.96:0/4112870601 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f06dc0b49e0 0x7f06dc0d1bb0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:37.966 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:37.971+0000 7f06e1e00640 1 -- 172.21.15.96:0/4112870601 >> 172.21.15.96:0/4112870601 conn(0x7f06dc0b3ec0 msgr2=0x7f06dc0b42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:25:37.966 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:37.971+0000 7f06e1e00640 1 -- 172.21.15.96:0/4112870601 shutdown_connections 2024-05-28T06:25:37.966 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:37.971+0000 7f06e1e00640 1 -- 172.21.15.96:0/4112870601 wait complete. 2024-05-28T06:25:37.967 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:37.973+0000 7f06e1e00640 1 Processor -- start 2024-05-28T06:25:37.968 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:37.974+0000 7f06e1e00640 1 -- start start 2024-05-28T06:25:37.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:37.975+0000 7f06e1e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f06dc0b49e0 0x7f06dc15aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:37.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:37.975+0000 7f06e1e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f06dc0ced60 0x7f06dc1fc6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:37.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:37.975+0000 7f06e1e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f06dc161ed0 0x7f06dc1fcc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:37.969 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:37.975+0000 7f06e1e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f06dc0b3a60 con 0x7f06dc0ced60 2024-05-28T06:25:37.970 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:37.975+0000 7f06e1e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f06dc0b4e00 con 0x7f06dc161ed0 2024-05-28T06:25:37.970 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:37.975+0000 7f06e1e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f06dc0b2750 con 0x7f06dc0b49e0 2024-05-28T06:25:37.970 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:37.976+0000 7f06dbe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f06dc0ced60 0x7f06dc1fc6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:37.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:37.976+0000 7f06dbe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f06dc0ced60 0x7f06dc1fc6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:50062/0 (socket says 172.21.15.96:50062) 2024-05-28T06:25:37.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:37.976+0000 7f06dbe00640 1 -- 172.21.15.96:0/747901957 learned_addr learned my addr 172.21.15.96:0/747901957 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:25:37.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:37.976+0000 7f06dbe00640 1 -- 172.21.15.96:0/747901957 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f06dc0b49e0 msgr2=0x7f06dc15aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:37.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:37.976+0000 7f06dbe00640 1 --2- 172.21.15.96:0/747901957 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f06dc0b49e0 0x7f06dc15aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:37.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:37.976+0000 7f06dbe00640 1 -- 172.21.15.96:0/747901957 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f06dc161ed0 msgr2=0x7f06dc1fcc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:37.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:37.976+0000 7f06dbe00640 1 --2- 172.21.15.96:0/747901957 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f06dc161ed0 0x7f06dc1fcc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:37.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:37.976+0000 7f06dbe00640 1 -- 172.21.15.96:0/747901957 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06dc15ba30 con 0x7f06dc0ced60 2024-05-28T06:25:37.971 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:37.977+0000 7f06dbe00640 1 --2- 172.21.15.96:0/747901957 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f06dc0ced60 0x7f06dc1fc6d0 secure :-1 s=READY pgs=2091 cs=0 l=1 rev1=1 crypto rx=0x7f06c8000c00 tx=0x7f06c80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:25:37.972 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:37.977+0000 7f06d8c00640 1 -- 172.21.15.96:0/747901957 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06c8041020 con 0x7f06dc0ced60 2024-05-28T06:25:37.972 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:37.978+0000 7f06d8c00640 1 -- 172.21.15.96:0/747901957 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06c80306c0 con 0x7f06dc0ced60 2024-05-28T06:25:37.972 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:37.978+0000 7f06d8c00640 1 -- 172.21.15.96:0/747901957 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06c8037040 con 0x7f06dc0ced60 2024-05-28T06:25:37.972 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:37.978+0000 7f06e1e00640 1 -- 172.21.15.96:0/747901957 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06dc15bd10 con 0x7f06dc0ced60 2024-05-28T06:25:37.972 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:37.978+0000 7f06e1e00640 1 -- 172.21.15.96:0/747901957 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06dc15c2d0 con 0x7f06dc0ced60 2024-05-28T06:25:37.973 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:37.980+0000 7f06e1e00640 1 -- 172.21.15.96:0/747901957 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06dc0689e0 con 0x7f06dc0ced60 2024-05-28T06:25:37.974 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:37.980+0000 7f06d8c00640 1 -- 172.21.15.96:0/747901957 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 47) ==== 93998+0+0 (secure 0 0 0) 0x7f06c8044020 con 0x7f06dc0ced60 2024-05-28T06:25:37.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:37.981+0000 7f06d8c00640 1 --2- 172.21.15.96:0/747901957 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f06bc06e990 0x7f06bc070e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:37.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:37.982+0000 7f06d8c00640 1 -- 172.21.15.96:0/747901957 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 5823+0+0 (secure 0 0 0) 0x7f06c80ab530 con 0x7f06dc0ced60 2024-05-28T06:25:37.977 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:37.983+0000 7f06e0a00640 1 --2- 172.21.15.96:0/747901957 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f06bc06e990 0x7f06bc070e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:37.979 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:37.985+0000 7f06e0a00640 1 --2- 172.21.15.96:0/747901957 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f06bc06e990 0x7f06bc070e50 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f06d0000f60 tx=0x7f06d00079d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:25:37.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:37.988+0000 7f06d8c00640 1 -- 172.21.15.96:0/747901957 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f06c804b030 con 0x7f06dc0ced60 2024-05-28T06:25:38.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:38.342+0000 7f06e1e00640 1 -- 172.21.15.96:0/747901957 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone10993", "target": ["mon-mgr", ""]}) -- 0x7f06dc0cf1d0 con 0x7f06bc06e990 2024-05-28T06:25:38.434 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:25:38.440+0000 7fea59e00640 -1 mgr.server reply reply (22) Invalid argument cannot cancel -- clone finished (check clone status) 2024-05-28T06:25:38.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:38.440+0000 7f06d8c00640 1 -- 172.21.15.96:0/747901957 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: -22 cannot cancel -- clone finished (check clone status)) ==== 60+0+0 (secure 0 0 0) 0x7f06dc0cf1d0 con 0x7f06bc06e990 2024-05-28T06:25:38.435 INFO:teuthology.orchestra.run.smithi096.stderr:Error EINVAL: cannot cancel -- clone finished (check clone status) 2024-05-28T06:25:38.450 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-05-28T06:25:38.451 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone10994 2024-05-28T06:25:38.741 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:38.747+0000 7fba1b600640 1 Processor -- start 2024-05-28T06:25:38.742 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:38.747+0000 7fba1b600640 1 -- start start 2024-05-28T06:25:38.742 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:38.749+0000 7fba1b600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fba14162130 0x7fba140d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:38.743 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:38.749+0000 7fba1b600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fba140d02e0 0x7fba140d0770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:38.743 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:38.749+0000 7fba1b600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fba140b13d0 con 0x7fba140d15e0 2024-05-28T06:25:38.743 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:38.749+0000 7fba1b600640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fba140d1a40 con 0x7fba140d02e0 2024-05-28T06:25:38.743 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:38.749+0000 7fba1b600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fba140d33b0 con 0x7fba14162130 2024-05-28T06:25:38.744 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:38.749+0000 7fba13e00640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7fba140d15e0 0x7fba140b1e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.96:60060/0 (socket says 172.21.15.96:60060) 2024-05-28T06:25:38.745 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:38.749+0000 7fba18c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fba14162130 0x7fba140d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:38.745 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:38.749+0000 7fba13e00640 1 -- 172.21.15.96:0/126272755 learned_addr learned my addr 172.21.15.96:0/126272755 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:25:38.745 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:38.749+0000 7fba19600640 1 --2- 172.21.15.96:0/126272755 >> v2:172.21.15.123:3300/0 conn(0x7fba140d02e0 0x7fba140d0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:38.745 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:38.750+0000 7fba13400640 1 -- 172.21.15.96:0/126272755 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1053203307 0 0) 0x7fba140b13d0 con 0x7fba140d15e0 2024-05-28T06:25:38.745 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:38.750+0000 7fba13400640 1 -- 172.21.15.96:0/126272755 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb9fc003520 con 0x7fba140d15e0 2024-05-28T06:25:38.745 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:38.750+0000 7fba19600640 1 -- 172.21.15.96:0/126272755 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fba14162130 msgr2=0x7fba140d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:25:38.745 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:38.750+0000 7fba19600640 1 --2- 172.21.15.96:0/126272755 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fba14162130 0x7fba140d10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:38.746 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:38.750+0000 7fba19600640 1 -- 172.21.15.96:0/126272755 >> v1:172.21.15.123:6789/0 conn(0x7fba140d15e0 legacy=0x7fba140b1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:38.746 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:38.750+0000 7fba19600640 1 -- 172.21.15.96:0/126272755 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba140b1640 con 0x7fba140d02e0 2024-05-28T06:25:38.748 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:38.752+0000 7fba19600640 1 --2- 172.21.15.96:0/126272755 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fba140d02e0 0x7fba140d0770 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fba0c009d90 tx=0x7fba0c02a660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eb927fb30fb54684 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:25:38.748 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:38.752+0000 7fba13400640 1 -- 172.21.15.96:0/126272755 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba0c007740 con 0x7fba140d02e0 2024-05-28T06:25:38.748 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:38.752+0000 7fba13400640 1 -- 172.21.15.96:0/126272755 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba0c0078e0 con 0x7fba140d02e0 2024-05-28T06:25:38.748 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:38.752+0000 7fba13400640 1 -- 172.21.15.96:0/126272755 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba0c007cb0 con 0x7fba140d02e0 2024-05-28T06:25:38.748 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:38.753+0000 7fba1b600640 1 -- 172.21.15.96:0/126272755 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fba140d02e0 msgr2=0x7fba140d0770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:38.748 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:38.753+0000 7fba1b600640 1 --2- 172.21.15.96:0/126272755 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fba140d02e0 0x7fba140d0770 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fba0c009d90 tx=0x7fba0c02a660 comp rx=0 tx=0).stop 2024-05-28T06:25:38.748 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:38.753+0000 7fba1b600640 1 -- 172.21.15.96:0/126272755 shutdown_connections 2024-05-28T06:25:38.748 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:38.753+0000 7fba1b600640 1 --2- 172.21.15.96:0/126272755 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fba140d02e0 0x7fba140d0770 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:38.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:38.753+0000 7fba1b600640 1 --2- 172.21.15.96:0/126272755 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fba14162130 0x7fba140d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:38.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:38.753+0000 7fba1b600640 1 -- 172.21.15.96:0/126272755 >> 172.21.15.96:0/126272755 conn(0x7fba1408bfc0 msgr2=0x7fba140d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:25:38.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:38.753+0000 7fba1b600640 1 -- 172.21.15.96:0/126272755 shutdown_connections 2024-05-28T06:25:38.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:38.753+0000 7fba1b600640 1 -- 172.21.15.96:0/126272755 wait complete. 2024-05-28T06:25:38.749 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:38.756+0000 7fba1b600640 1 Processor -- start 2024-05-28T06:25:38.750 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:38.756+0000 7fba1b600640 1 -- start start 2024-05-28T06:25:38.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:38.757+0000 7fba1b600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fba140d02e0 0x7fba140a1d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:38.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:38.757+0000 7fba1b600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fba140d15e0 0x7fba141f6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:38.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:38.757+0000 7fba1b600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fba14162130 0x7fba1406d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:38.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:38.757+0000 7fba1b600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fba140d3170 con 0x7fba14162130 2024-05-28T06:25:38.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:38.757+0000 7fba1b600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fba140d2ff0 con 0x7fba140d15e0 2024-05-28T06:25:38.752 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:38.757+0000 7fba1b600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fba14162510 con 0x7fba140d02e0 2024-05-28T06:25:38.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:38.757+0000 7fba19600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fba14162130 0x7fba1406d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:38.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:38.757+0000 7fba19600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fba14162130 0x7fba1406d6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:50064/0 (socket says 172.21.15.96:50064) 2024-05-28T06:25:38.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:38.757+0000 7fba19600640 1 -- 172.21.15.96:0/2084399545 learned_addr learned my addr 172.21.15.96:0/2084399545 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:25:38.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:38.757+0000 7fba18c00640 1 --2- 172.21.15.96:0/2084399545 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fba140d02e0 0x7fba140a1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:38.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:38.758+0000 7fba19600640 1 -- 172.21.15.96:0/2084399545 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fba140d02e0 msgr2=0x7fba140a1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:38.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:38.758+0000 7fba19600640 1 --2- 172.21.15.96:0/2084399545 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fba140d02e0 0x7fba140a1d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:38.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:38.758+0000 7fba13e00640 1 --2- 172.21.15.96:0/2084399545 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fba140d15e0 0x7fba141f6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:38.753 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:38.758+0000 7fba19600640 1 -- 172.21.15.96:0/2084399545 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fba140d15e0 msgr2=0x7fba141f6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:38.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:38.758+0000 7fba19600640 1 --2- 172.21.15.96:0/2084399545 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fba140d15e0 0x7fba141f6240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:38.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:38.758+0000 7fba19600640 1 -- 172.21.15.96:0/2084399545 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba141f6c10 con 0x7fba14162130 2024-05-28T06:25:38.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:38.758+0000 7fba13e00640 1 --2- 172.21.15.96:0/2084399545 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fba140d15e0 0x7fba141f6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:25:38.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:38.758+0000 7fba19600640 1 --2- 172.21.15.96:0/2084399545 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fba14162130 0x7fba1406d6a0 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 crypto rx=0x7fba0c009d90 tx=0x7fba0c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:25:38.755 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:38.759+0000 7fba10c00640 1 -- 172.21.15.96:0/2084399545 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba0c040070 con 0x7fba14162130 2024-05-28T06:25:38.755 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:38.759+0000 7fba10c00640 1 -- 172.21.15.96:0/2084399545 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba0c031bc0 con 0x7fba14162130 2024-05-28T06:25:38.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:38.759+0000 7fba10c00640 1 -- 172.21.15.96:0/2084399545 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba0c009450 con 0x7fba14162130 2024-05-28T06:25:38.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:38.759+0000 7fba1b600640 1 -- 172.21.15.96:0/2084399545 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba141f6ef0 con 0x7fba14162130 2024-05-28T06:25:38.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:38.759+0000 7fba1b600640 1 -- 172.21.15.96:0/2084399545 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba141f74b0 con 0x7fba14162130 2024-05-28T06:25:38.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:38.760+0000 7fba1b600640 1 -- 172.21.15.96:0/2084399545 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba140690f0 con 0x7fba14162130 2024-05-28T06:25:38.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:38.761+0000 7fba10c00640 1 -- 172.21.15.96:0/2084399545 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 47) ==== 93998+0+0 (secure 0 0 0) 0x7fba0c039040 con 0x7fba14162130 2024-05-28T06:25:38.757 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:38.763+0000 7fba10c00640 1 --2- 172.21.15.96:0/2084399545 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb9f406ea10 0x7fb9f4070ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:38.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:38.763+0000 7fba10c00640 1 -- 172.21.15.96:0/2084399545 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 5823+0+0 (secure 0 0 0) 0x7fba0c0abf30 con 0x7fba14162130 2024-05-28T06:25:38.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:38.767+0000 7fba18c00640 1 --2- 172.21.15.96:0/2084399545 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb9f406ea10 0x7fb9f4070ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:38.762 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:38.769+0000 7fba10c00640 1 -- 172.21.15.96:0/2084399545 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fba0c04b340 con 0x7fba14162130 2024-05-28T06:25:38.763 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:38.769+0000 7fba18c00640 1 --2- 172.21.15.96:0/2084399545 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fb9f406ea10 0x7fb9f4070ed0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fba08009a10 tx=0x7fba08008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:25:39.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:39.159+0000 7fba1b600640 1 -- 172.21.15.96:0/2084399545 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone10994", "target": ["mon-mgr", ""]}) -- 0x7fba140b2f40 con 0x7fb9f406ea10 2024-05-28T06:25:39.680 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:25:39.687+0000 7fea59e00640 -1 mgr.server reply reply (22) Invalid argument cannot cancel -- clone finished (check clone status) 2024-05-28T06:25:39.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:39.688+0000 7fba10c00640 1 -- 172.21.15.96:0/2084399545 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: -22 cannot cancel -- clone finished (check clone status)) ==== 60+0+0 (secure 0 0 0) 0x7fba140b2f40 con 0x7fb9f406ea10 2024-05-28T06:25:39.681 INFO:teuthology.orchestra.run.smithi096.stderr:Error EINVAL: cannot cancel -- clone finished (check clone status) 2024-05-28T06:25:39.699 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-05-28T06:25:39.700 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone10995 2024-05-28T06:25:39.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:39.944+0000 7fce68200640 1 Processor -- start 2024-05-28T06:25:39.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:39.944+0000 7fce68200640 1 -- start start 2024-05-28T06:25:39.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:39.945+0000 7fce68200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fce60161a90 0x7fce6006e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:39.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:39.945+0000 7fce68200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fce60161e70 0x7fce600b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:39.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:39.945+0000 7fce68200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce600b3d40 con 0x7fce6006e9b0 2024-05-28T06:25:39.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:39.945+0000 7fce68200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fce600b4020 con 0x7fce60161a90 2024-05-28T06:25:39.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:39.945+0000 7fce68200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fce600b4300 con 0x7fce60161e70 2024-05-28T06:25:39.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:39.946+0000 7fce64e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fce60161e70 0x7fce600b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:39.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:39.946+0000 7fce64e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fce60161e70 0x7fce600b32c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.96:59724/0 (socket says 172.21.15.96:59724) 2024-05-28T06:25:39.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:39.946+0000 7fce64e00640 1 -- 172.21.15.96:0/1263144414 learned_addr learned my addr 172.21.15.96:0/1263144414 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:25:39.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:39.946+0000 7fce65800640 1 --2- 172.21.15.96:0/1263144414 >> v2:172.21.15.123:3300/0 conn(0x7fce60161a90 0x7fce6006e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:39.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:39.946+0000 7fce64e00640 1 -- 172.21.15.96:0/1263144414 >> v2:172.21.15.123:3300/0 conn(0x7fce60161a90 msgr2=0x7fce6006e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:25:39.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:39.946+0000 7fce64e00640 1 --2- 172.21.15.96:0/1263144414 >> v2:172.21.15.123:3300/0 conn(0x7fce60161a90 0x7fce6006e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:39.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:39.946+0000 7fce64e00640 1 -- 172.21.15.96:0/1263144414 >> v1:172.21.15.123:6789/0 conn(0x7fce6006e9b0 legacy=0x7fce600ad2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:39.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:39.946+0000 7fce5fe00640 1 -- 172.21.15.96:0/1263144414 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 732155842 0 0) 0x7fce600b3d40 con 0x7fce6006e9b0 2024-05-28T06:25:39.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:39.946+0000 7fce64e00640 1 -- 172.21.15.96:0/1263144414 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce6006dc70 con 0x7fce60161e70 2024-05-28T06:25:39.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:39.946+0000 7fce65800640 1 --2- 172.21.15.96:0/1263144414 >> v2:172.21.15.123:3300/0 conn(0x7fce60161a90 0x7fce6006e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-28T06:25:39.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:39.947+0000 7fce64e00640 1 --2- 172.21.15.96:0/1263144414 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fce60161e70 0x7fce600b32c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fce4c009870 tx=0x7fce4c02a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=19e55cabdab929ef server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:25:39.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:39.948+0000 7fce5fe00640 1 -- 172.21.15.96:0/1263144414 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce4c037070 con 0x7fce60161e70 2024-05-28T06:25:39.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:39.948+0000 7fce5fe00640 1 -- 172.21.15.96:0/1263144414 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce4c032440 con 0x7fce60161e70 2024-05-28T06:25:39.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:39.948+0000 7fce5fe00640 1 -- 172.21.15.96:0/1263144414 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce4c032810 con 0x7fce60161e70 2024-05-28T06:25:39.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:39.948+0000 7fce68200640 1 -- 172.21.15.96:0/1263144414 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fce60161e70 msgr2=0x7fce600b32c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:39.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:39.948+0000 7fce68200640 1 --2- 172.21.15.96:0/1263144414 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fce60161e70 0x7fce600b32c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fce4c009870 tx=0x7fce4c02a270 comp rx=0 tx=0).stop 2024-05-28T06:25:39.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:39.948+0000 7fce68200640 1 -- 172.21.15.96:0/1263144414 shutdown_connections 2024-05-28T06:25:39.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:39.948+0000 7fce68200640 1 --2- 172.21.15.96:0/1263144414 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fce60161e70 0x7fce600b32c0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:39.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:39.948+0000 7fce68200640 1 --2- 172.21.15.96:0/1263144414 >> v2:172.21.15.123:3300/0 conn(0x7fce60161a90 0x7fce6006e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:39.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:39.948+0000 7fce68200640 1 -- 172.21.15.96:0/1263144414 >> 172.21.15.96:0/1263144414 conn(0x7fce6006bd20 msgr2=0x7fce6006c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:25:39.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:39.948+0000 7fce68200640 1 -- 172.21.15.96:0/1263144414 shutdown_connections 2024-05-28T06:25:39.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:39.948+0000 7fce68200640 1 -- 172.21.15.96:0/1263144414 wait complete. 2024-05-28T06:25:39.943 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:39.950+0000 7fce68200640 1 Processor -- start 2024-05-28T06:25:39.943 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:39.950+0000 7fce68200640 1 -- start start 2024-05-28T06:25:39.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:39.951+0000 7fce68200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fce6006e9b0 0x7fce60150dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:39.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:39.951+0000 7fce68200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fce60161a90 0x7fce60156910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:39.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:39.951+0000 7fce68200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce60161e70 0x7fce601600d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:39.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:39.951+0000 7fce68200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fce600ad9b0 con 0x7fce6006e9b0 2024-05-28T06:25:39.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:39.951+0000 7fce68200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fce600b5020 con 0x7fce60161e70 2024-05-28T06:25:39.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:39.951+0000 7fce68200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fce600adb30 con 0x7fce60161a90 2024-05-28T06:25:39.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:39.951+0000 7fce65800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fce6006e9b0 0x7fce60150dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:39.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:39.951+0000 7fce65800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fce6006e9b0 0x7fce60150dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:56248/0 (socket says 172.21.15.96:56248) 2024-05-28T06:25:39.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:39.951+0000 7fce65800640 1 -- 172.21.15.96:0/1401255581 learned_addr learned my addr 172.21.15.96:0/1401255581 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:25:39.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:39.951+0000 7fce66200640 1 --2- 172.21.15.96:0/1401255581 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce60161e70 0x7fce601600d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:39.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:39.951+0000 7fce64e00640 1 --2- 172.21.15.96:0/1401255581 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fce60161a90 0x7fce60156910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:39.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:39.952+0000 7fce65800640 1 -- 172.21.15.96:0/1401255581 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fce60161a90 msgr2=0x7fce60156910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:39.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:39.952+0000 7fce65800640 1 --2- 172.21.15.96:0/1401255581 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fce60161a90 0x7fce60156910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:39.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:39.952+0000 7fce65800640 1 -- 172.21.15.96:0/1401255581 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce60161e70 msgr2=0x7fce601600d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:39.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:39.952+0000 7fce65800640 1 --2- 172.21.15.96:0/1401255581 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce60161e70 0x7fce601600d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:39.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:39.952+0000 7fce65800640 1 -- 172.21.15.96:0/1401255581 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce60158960 con 0x7fce6006e9b0 2024-05-28T06:25:39.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:39.952+0000 7fce66200640 1 --2- 172.21.15.96:0/1401255581 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce60161e70 0x7fce601600d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-28T06:25:39.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:39.952+0000 7fce65800640 1 --2- 172.21.15.96:0/1401255581 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fce6006e9b0 0x7fce60150dd0 secure :-1 s=READY pgs=2093 cs=0 l=1 rev1=1 crypto rx=0x7fce5400e960 tx=0x7fce5400ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:25:39.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:39.952+0000 7fce5d600640 1 -- 172.21.15.96:0/1401255581 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce54017070 con 0x7fce6006e9b0 2024-05-28T06:25:39.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:39.952+0000 7fce5d600640 1 -- 172.21.15.96:0/1401255581 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce5400ca00 con 0x7fce6006e9b0 2024-05-28T06:25:39.948 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:39.952+0000 7fce68200640 1 -- 172.21.15.96:0/1401255581 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce60158c40 con 0x7fce6006e9b0 2024-05-28T06:25:39.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:39.952+0000 7fce5d600640 1 -- 172.21.15.96:0/1401255581 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce5400cdd0 con 0x7fce6006e9b0 2024-05-28T06:25:39.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:39.952+0000 7fce68200640 1 -- 172.21.15.96:0/1401255581 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce60159200 con 0x7fce6006e9b0 2024-05-28T06:25:39.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:39.954+0000 7fce68200640 1 -- 172.21.15.96:0/1401255581 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce600689e0 con 0x7fce6006e9b0 2024-05-28T06:25:39.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:39.955+0000 7fce5d600640 1 -- 172.21.15.96:0/1401255581 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 47) ==== 93998+0+0 (secure 0 0 0) 0x7fce5401a070 con 0x7fce6006e9b0 2024-05-28T06:25:39.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:39.956+0000 7fce5d600640 1 --2- 172.21.15.96:0/1401255581 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fce4006e990 0x7fce40070e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:39.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:39.956+0000 7fce5d600640 1 -- 172.21.15.96:0/1401255581 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 5823+0+0 (secure 0 0 0) 0x7fce5408afa0 con 0x7fce6006e9b0 2024-05-28T06:25:39.954 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:39.961+0000 7fce5d600640 1 -- 172.21.15.96:0/1401255581 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fce5405ada0 con 0x7fce6006e9b0 2024-05-28T06:25:39.963 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:39.970+0000 7fce64e00640 1 --2- 172.21.15.96:0/1401255581 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fce4006e990 0x7fce40070e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:39.965 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:39.972+0000 7fce64e00640 1 --2- 172.21.15.96:0/1401255581 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fce4006e990 0x7fce40070e50 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7fce4c034000 tx=0x7fce4c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:25:40.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:40.366+0000 7fce68200640 1 -- 172.21.15.96:0/1401255581 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone10995", "target": ["mon-mgr", ""]}) -- 0x7fce60157f80 con 0x7fce4006e990 2024-05-28T06:25:41.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.117+0000 7fce5d600640 1 -- 172.21.15.96:0/1401255581 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fce60157f80 con 0x7fce4006e990 2024-05-28T06:25:41.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.122+0000 7fce68200640 1 -- 172.21.15.96:0/1401255581 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fce4006e990 msgr2=0x7fce40070e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:41.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.122+0000 7fce68200640 1 --2- 172.21.15.96:0/1401255581 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fce4006e990 0x7fce40070e50 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7fce4c034000 tx=0x7fce4c02e000 comp rx=0 tx=0).stop 2024-05-28T06:25:41.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.122+0000 7fce68200640 1 -- 172.21.15.96:0/1401255581 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fce6006e9b0 msgr2=0x7fce60150dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:41.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.122+0000 7fce68200640 1 --2- 172.21.15.96:0/1401255581 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fce6006e9b0 0x7fce60150dd0 secure :-1 s=READY pgs=2093 cs=0 l=1 rev1=1 crypto rx=0x7fce5400e960 tx=0x7fce5400ee20 comp rx=0 tx=0).stop 2024-05-28T06:25:41.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.123+0000 7fce68200640 1 -- 172.21.15.96:0/1401255581 shutdown_connections 2024-05-28T06:25:41.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.123+0000 7fce68200640 1 --2- 172.21.15.96:0/1401255581 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fce4006e990 0x7fce40070e50 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:41.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.123+0000 7fce68200640 1 --2- 172.21.15.96:0/1401255581 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce60161e70 0x7fce601600d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:41.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.123+0000 7fce68200640 1 --2- 172.21.15.96:0/1401255581 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fce60161a90 0x7fce60156910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:41.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.123+0000 7fce68200640 1 --2- 172.21.15.96:0/1401255581 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fce6006e9b0 0x7fce60150dd0 unknown :-1 s=CLOSED pgs=2093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:41.116 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.123+0000 7fce68200640 1 -- 172.21.15.96:0/1401255581 >> 172.21.15.96:0/1401255581 conn(0x7fce6006bd20 msgr2=0x7fce600b3050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:25:41.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.123+0000 7fce68200640 1 -- 172.21.15.96:0/1401255581 shutdown_connections 2024-05-28T06:25:41.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.123+0000 7fce68200640 1 -- 172.21.15.96:0/1401255581 wait complete. 2024-05-28T06:25:41.142 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone10996 2024-05-28T06:25:41.420 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.427+0000 7f7c4f200640 1 Processor -- start 2024-05-28T06:25:41.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.428+0000 7f7c4f200640 1 -- start start 2024-05-28T06:25:41.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.430+0000 7f7c4f200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f7c480d4680 0x7f7c480d2d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:41.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.430+0000 7f7c4f200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7c480d3280 0x7f7c4815e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:41.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.430+0000 7f7c4f200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c480d0150 con 0x7f7c480d4a60 2024-05-28T06:25:41.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.430+0000 7f7c4f200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f7c480d28b0 con 0x7f7c480d4680 2024-05-28T06:25:41.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.430+0000 7f7c4f200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f7c480d0430 con 0x7f7c480d3280 2024-05-28T06:25:41.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.430+0000 7f7c4d400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f7c480d4a60 0x7f7c480b46a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:60330/0 (socket says 172.21.15.96:60330) 2024-05-28T06:25:41.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.430+0000 7f7c4d400640 1 -- 172.21.15.96:0/84317276 learned_addr learned my addr 172.21.15.96:0/84317276 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:25:41.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.431+0000 7f7c4de00640 1 --2- 172.21.15.96:0/84317276 >> v2:172.21.15.123:3300/0 conn(0x7f7c480d4680 0x7f7c480d2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:41.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.431+0000 7f7c4ca00640 1 -- 172.21.15.96:0/84317276 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 762637724 0 0) 0x7f7c480d0150 con 0x7f7c480d4a60 2024-05-28T06:25:41.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.431+0000 7f7c4ca00640 1 -- 172.21.15.96:0/84317276 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7c30003520 con 0x7f7c480d4a60 2024-05-28T06:25:41.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.431+0000 7f7c4ca00640 1 -- 172.21.15.96:0/84317276 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3687270344 0 0) 0x7f7c30003520 con 0x7f7c480d4a60 2024-05-28T06:25:41.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.431+0000 7f7c4ca00640 1 -- 172.21.15.96:0/84317276 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7c480d3280 msgr2=0x7f7c4815e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:25:41.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.431+0000 7f7c4ca00640 1 --2- 172.21.15.96:0/84317276 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7c480d3280 0x7f7c4815e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:41.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.431+0000 7f7c4ca00640 1 -- 172.21.15.96:0/84317276 >> v2:172.21.15.123:3300/0 conn(0x7f7c480d4680 msgr2=0x7f7c480d2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:25:41.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.431+0000 7f7c4ca00640 1 --2- 172.21.15.96:0/84317276 >> v2:172.21.15.123:3300/0 conn(0x7f7c480d4680 0x7f7c480d2d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:41.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.431+0000 7f7c4ca00640 1 -- 172.21.15.96:0/84317276 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c480d2260 con 0x7f7c480d4a60 2024-05-28T06:25:41.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.431+0000 7f7c4ca00640 1 -- 172.21.15.96:0/84317276 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f7c380017f0 con 0x7f7c480d4a60 2024-05-28T06:25:41.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.431+0000 7f7c4ca00640 1 -- 172.21.15.96:0/84317276 >> v1:172.21.15.96:6789/0 conn(0x7f7c480d4a60 legacy=0x7f7c480b46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:41.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.431+0000 7f7c4ca00640 1 --2- 172.21.15.96:0/84317276 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7c480d3280 0x7f7c300039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:41.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.431+0000 7f7c4ca00640 1 --2- 172.21.15.96:0/84317276 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c30003fe0 0x7f7c30004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:41.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.431+0000 7f7c4ca00640 1 --2- 172.21.15.96:0/84317276 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c30004d40 0x7f7c300051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:41.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.431+0000 7f7c4ca00640 1 -- 172.21.15.96:0/84317276 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f7c480d28b0 con 0x7f7c30003fe0 2024-05-28T06:25:41.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.431+0000 7f7c4ca00640 1 -- 172.21.15.96:0/84317276 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f7c480d0430 con 0x7f7c30004d40 2024-05-28T06:25:41.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.431+0000 7f7c4ca00640 1 -- 172.21.15.96:0/84317276 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f7c3000bbd0 con 0x7f7c480d3280 2024-05-28T06:25:41.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.432+0000 7f7c4de00640 1 --2- 172.21.15.96:0/84317276 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7c480d3280 0x7f7c300039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:41.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.432+0000 7f7c4d400640 1 --2- 172.21.15.96:0/84317276 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c30003fe0 0x7f7c30004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:41.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.433+0000 7f7c4d400640 1 -- 172.21.15.96:0/84317276 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7c480d3280 msgr2=0x7f7c300039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:25:41.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.433+0000 7f7c4d400640 1 --2- 172.21.15.96:0/84317276 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7c480d3280 0x7f7c300039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:41.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.433+0000 7f7c4d400640 1 -- 172.21.15.96:0/84317276 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c30004d40 msgr2=0x7f7c300051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:25:41.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.433+0000 7f7c4d400640 1 --2- 172.21.15.96:0/84317276 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c30004d40 0x7f7c300051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:41.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.434+0000 7f7c4d400640 1 -- 172.21.15.96:0/84317276 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c480d2260 con 0x7f7c30003fe0 2024-05-28T06:25:41.433 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.439+0000 7f7c4d400640 1 --2- 172.21.15.96:0/84317276 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c30003fe0 0x7f7c30004800 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 crypto rx=0x7f7c3800c260 tx=0x7f7c30005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=817d3d9904bb6998 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:25:41.433 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.440+0000 7f7c4ca00640 1 -- 172.21.15.96:0/84317276 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c38007d60 con 0x7f7c30003fe0 2024-05-28T06:25:41.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.440+0000 7f7c4ca00640 1 -- 172.21.15.96:0/84317276 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c38007270 con 0x7f7c30003fe0 2024-05-28T06:25:41.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.440+0000 7f7c4ca00640 1 -- 172.21.15.96:0/84317276 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c380057a0 con 0x7f7c30003fe0 2024-05-28T06:25:41.434 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.441+0000 7f7c4f200640 1 -- 172.21.15.96:0/84317276 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c30003fe0 msgr2=0x7f7c30004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:41.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.441+0000 7f7c4f200640 1 --2- 172.21.15.96:0/84317276 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c30003fe0 0x7f7c30004800 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 crypto rx=0x7f7c3800c260 tx=0x7f7c30005830 comp rx=0 tx=0).stop 2024-05-28T06:25:41.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.441+0000 7f7c4e800640 1 -- 172.21.15.96:0/84317276 reap_dead start 2024-05-28T06:25:41.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.441+0000 7f7c4f200640 1 -- 172.21.15.96:0/84317276 shutdown_connections 2024-05-28T06:25:41.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.441+0000 7f7c4f200640 1 -- 172.21.15.96:0/84317276 >> 172.21.15.96:0/84317276 conn(0x7f7c4809d810 msgr2=0x7f7c4808bfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:25:41.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.442+0000 7f7c4f200640 1 -- 172.21.15.96:0/84317276 shutdown_connections 2024-05-28T06:25:41.435 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.442+0000 7f7c4f200640 1 -- 172.21.15.96:0/84317276 wait complete. 2024-05-28T06:25:41.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.445+0000 7f7c4f200640 1 Processor -- start 2024-05-28T06:25:41.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.445+0000 7f7c4f200640 1 -- start start 2024-05-28T06:25:41.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.447+0000 7f7c4f200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c481538f0 0x7f7c480ccf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:41.440 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.447+0000 7f7c4f200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c48153cd0 0x7f7c480cd4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:41.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.447+0000 7f7c4f200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7c48154150 0x7f7c48154600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:41.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.447+0000 7f7c4f200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f7c480b4f30 con 0x7f7c481538f0 2024-05-28T06:25:41.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.447+0000 7f7c4f200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f7c480b4db0 con 0x7f7c48153cd0 2024-05-28T06:25:41.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.447+0000 7f7c4f200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f7c4815e2e0 con 0x7f7c48154150 2024-05-28T06:25:41.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.447+0000 7f7c4de00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c481538f0 0x7f7c480ccf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:41.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.447+0000 7f7c4de00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c481538f0 0x7f7c480ccf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:56274/0 (socket says 172.21.15.96:56274) 2024-05-28T06:25:41.441 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.447+0000 7f7c4de00640 1 -- 172.21.15.96:0/3598577765 learned_addr learned my addr 172.21.15.96:0/3598577765 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:25:41.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.448+0000 7f7c4de00640 1 -- 172.21.15.96:0/3598577765 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7c48154150 msgr2=0x7f7c48154600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:41.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.448+0000 7f7c4de00640 1 --2- 172.21.15.96:0/3598577765 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7c48154150 0x7f7c48154600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:41.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.448+0000 7f7c4de00640 1 -- 172.21.15.96:0/3598577765 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c48153cd0 msgr2=0x7f7c480cd4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:41.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.448+0000 7f7c4de00640 1 --2- 172.21.15.96:0/3598577765 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c48153cd0 0x7f7c480cd4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:41.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.448+0000 7f7c4de00640 1 -- 172.21.15.96:0/3598577765 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c481f8950 con 0x7f7c481538f0 2024-05-28T06:25:41.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.449+0000 7f7c4de00640 1 --2- 172.21.15.96:0/3598577765 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c481538f0 0x7f7c480ccf70 secure :-1 s=READY pgs=2096 cs=0 l=1 rev1=1 crypto rx=0x7f7c3c00b820 tx=0x7f7c3c00bce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:25:41.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.449+0000 7f7c42000640 1 -- 172.21.15.96:0/3598577765 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c3c018070 con 0x7f7c481538f0 2024-05-28T06:25:41.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.449+0000 7f7c42000640 1 -- 172.21.15.96:0/3598577765 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c3c00be10 con 0x7f7c481538f0 2024-05-28T06:25:41.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.449+0000 7f7c42000640 1 -- 172.21.15.96:0/3598577765 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c3c010c60 con 0x7f7c481538f0 2024-05-28T06:25:41.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.449+0000 7f7c4f200640 1 -- 172.21.15.96:0/3598577765 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c480cdb20 con 0x7f7c481538f0 2024-05-28T06:25:41.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.450+0000 7f7c4f200640 1 -- 172.21.15.96:0/3598577765 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c4815f330 con 0x7f7c481538f0 2024-05-28T06:25:41.444 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.451+0000 7f7c4f200640 1 -- 172.21.15.96:0/3598577765 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c480689e0 con 0x7f7c481538f0 2024-05-28T06:25:41.446 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.454+0000 7f7c42000640 1 -- 172.21.15.96:0/3598577765 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 47) ==== 93998+0+0 (secure 0 0 0) 0x7f7c3c01b070 con 0x7f7c481538f0 2024-05-28T06:25:41.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.455+0000 7f7c42000640 1 --2- 172.21.15.96:0/3598577765 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7c2c06e9c0 0x7f7c2c070e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:41.449 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.456+0000 7f7c42000640 1 -- 172.21.15.96:0/3598577765 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 5823+0+0 (secure 0 0 0) 0x7f7c3c08b0d0 con 0x7f7c481538f0 2024-05-28T06:25:41.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.462+0000 7f7c42000640 1 -- 172.21.15.96:0/3598577765 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7c3c05af60 con 0x7f7c481538f0 2024-05-28T06:25:41.463 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.471+0000 7f7c4d400640 1 --2- 172.21.15.96:0/3598577765 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7c2c06e9c0 0x7f7c2c070e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:41.470 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.478+0000 7f7c4d400640 1 --2- 172.21.15.96:0/3598577765 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7c2c06e9c0 0x7f7c2c070e80 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f7c3800c260 tx=0x7f7c38007000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:25:41.822 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:41.829+0000 7f7c4f200640 1 -- 172.21.15.96:0/3598577765 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone10996", "target": ["mon-mgr", ""]}) -- 0x7f7c480b4090 con 0x7f7c2c06e9c0 2024-05-28T06:25:42.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:42.333+0000 7f7c42000640 1 -- 172.21.15.96:0/3598577765 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f7c480b4090 con 0x7f7c2c06e9c0 2024-05-28T06:25:42.328 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:42.335+0000 7f7c4f200640 1 -- 172.21.15.96:0/3598577765 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7c2c06e9c0 msgr2=0x7f7c2c070e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:42.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:42.335+0000 7f7c4f200640 1 --2- 172.21.15.96:0/3598577765 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7c2c06e9c0 0x7f7c2c070e80 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f7c3800c260 tx=0x7f7c38007000 comp rx=0 tx=0).stop 2024-05-28T06:25:42.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:42.335+0000 7f7c4f200640 1 -- 172.21.15.96:0/3598577765 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c481538f0 msgr2=0x7f7c480ccf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:42.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:42.335+0000 7f7c4f200640 1 --2- 172.21.15.96:0/3598577765 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c481538f0 0x7f7c480ccf70 secure :-1 s=READY pgs=2096 cs=0 l=1 rev1=1 crypto rx=0x7f7c3c00b820 tx=0x7f7c3c00bce0 comp rx=0 tx=0).stop 2024-05-28T06:25:42.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:42.335+0000 7f7c4f200640 1 -- 172.21.15.96:0/3598577765 shutdown_connections 2024-05-28T06:25:42.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:42.335+0000 7f7c4f200640 1 --2- 172.21.15.96:0/3598577765 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7c2c06e9c0 0x7f7c2c070e80 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:42.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:42.335+0000 7f7c4f200640 1 --2- 172.21.15.96:0/3598577765 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7c48154150 0x7f7c48154600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:42.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:42.335+0000 7f7c4f200640 1 --2- 172.21.15.96:0/3598577765 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c48153cd0 0x7f7c480cd4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:42.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:42.335+0000 7f7c4f200640 1 --2- 172.21.15.96:0/3598577765 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c481538f0 0x7f7c480ccf70 unknown :-1 s=CLOSED pgs=2096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:42.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:42.335+0000 7f7c4f200640 1 -- 172.21.15.96:0/3598577765 >> 172.21.15.96:0/3598577765 conn(0x7f7c4809d810 msgr2=0x7f7c480d0670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:25:42.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:42.336+0000 7f7c4f200640 1 -- 172.21.15.96:0/3598577765 shutdown_connections 2024-05-28T06:25:42.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:42.336+0000 7f7c4f200640 1 -- 172.21.15.96:0/3598577765 wait complete. 2024-05-28T06:25:42.354 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone10997 2024-05-28T06:25:42.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:42.586+0000 7f2db8a00640 1 Processor -- start 2024-05-28T06:25:42.580 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:42.586+0000 7f2db8a00640 1 -- start start 2024-05-28T06:25:42.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:42.588+0000 7f2db8a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f2db40d0d50 0x7f2db41633b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:42.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:42.588+0000 7f2db8a00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2db41640e0 con 0x7f2db40b4120 2024-05-28T06:25:42.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:42.588+0000 7f2db8a00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2db40b3360 con 0x7f2db4162ac0 2024-05-28T06:25:42.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:42.588+0000 7f2db8a00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f2db40b3c90 con 0x7f2db40d0d50 2024-05-28T06:25:42.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:42.589+0000 7f2db1400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f2db40d0d50 0x7f2db41633b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:42.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:42.589+0000 7f2db1400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f2db40d0d50 0x7f2db41633b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:56290/0 (socket says 172.21.15.96:56290) 2024-05-28T06:25:42.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:42.589+0000 7f2db1400640 1 -- 172.21.15.96:0/2970318822 learned_addr learned my addr 172.21.15.96:0/2970318822 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:25:42.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:42.589+0000 7f2db2800640 1 --1- 172.21.15.96:0/2970318822 >> v1:172.21.15.96:6789/0 conn(0x7f2db40b4120 0x7f2db41638f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:60342/0 (socket says 172.21.15.96:60342) 2024-05-28T06:25:42.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:42.589+0000 7f2db1400640 1 -- 172.21.15.96:0/2970318822 >> v1:172.21.15.123:6789/0 conn(0x7f2db4162ac0 legacy=0x7f2db40d0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:42.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:42.589+0000 7f2db1400640 1 -- 172.21.15.96:0/2970318822 >> v1:172.21.15.96:6789/0 conn(0x7f2db40b4120 legacy=0x7f2db41638f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:42.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:42.589+0000 7f2db1400640 1 -- 172.21.15.96:0/2970318822 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2db40b3640 con 0x7f2db40d0d50 2024-05-28T06:25:42.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:42.589+0000 7f2db0a00640 1 -- 172.21.15.96:0/2970318822 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2534294233 0 0) 0x7f2db41640e0 con 0x7f2db40b4120 2024-05-28T06:25:42.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:42.590+0000 7f2db1400640 1 --2- 172.21.15.96:0/2970318822 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2db40d0d50 0x7f2db41633b0 secure :-1 s=READY pgs=2098 cs=0 l=1 rev1=1 crypto rx=0x7f2da0009870 tx=0x7f2da002a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa41f0879528b081 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:25:42.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:42.591+0000 7f2db0a00640 1 -- 172.21.15.96:0/2970318822 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2da0037070 con 0x7f2db40d0d50 2024-05-28T06:25:42.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:42.591+0000 7f2db0a00640 1 -- 172.21.15.96:0/2970318822 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2da002ae50 con 0x7f2db40d0d50 2024-05-28T06:25:42.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:42.591+0000 7f2db0a00640 1 -- 172.21.15.96:0/2970318822 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2da0032860 con 0x7f2db40d0d50 2024-05-28T06:25:42.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:42.592+0000 7f2db8a00640 1 -- 172.21.15.96:0/2970318822 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2db40d0d50 msgr2=0x7f2db41633b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:42.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:42.592+0000 7f2db8a00640 1 --2- 172.21.15.96:0/2970318822 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2db40d0d50 0x7f2db41633b0 secure :-1 s=READY pgs=2098 cs=0 l=1 rev1=1 crypto rx=0x7f2da0009870 tx=0x7f2da002a300 comp rx=0 tx=0).stop 2024-05-28T06:25:42.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:42.593+0000 7f2db8a00640 1 -- 172.21.15.96:0/2970318822 shutdown_connections 2024-05-28T06:25:42.586 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:42.593+0000 7f2db8a00640 1 --2- 172.21.15.96:0/2970318822 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2db40d0d50 0x7f2db41633b0 unknown :-1 s=CLOSED pgs=2098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:42.586 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:42.593+0000 7f2db8a00640 1 -- 172.21.15.96:0/2970318822 >> 172.21.15.96:0/2970318822 conn(0x7f2db406bd20 msgr2=0x7f2db406c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:25:42.586 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:42.593+0000 7f2db8a00640 1 -- 172.21.15.96:0/2970318822 shutdown_connections 2024-05-28T06:25:42.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:42.593+0000 7f2db8a00640 1 -- 172.21.15.96:0/2970318822 wait complete. 2024-05-28T06:25:42.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:42.594+0000 7f2db8a00640 1 Processor -- start 2024-05-28T06:25:42.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:42.595+0000 7f2db8a00640 1 -- start start 2024-05-28T06:25:42.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:42.596+0000 7f2db8a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2db40b4120 0x7f2db414adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:42.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:42.596+0000 7f2db8a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2db40d0d50 0x7f2db40b11b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:42.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:42.596+0000 7f2db8a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2db4162ac0 0x7f2db40b16f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:42.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:42.596+0000 7f2db8a00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f2db4163020 con 0x7f2db40b4120 2024-05-28T06:25:42.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:42.596+0000 7f2db8a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f2db4162ea0 con 0x7f2db4162ac0 2024-05-28T06:25:42.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:42.596+0000 7f2db8a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f2db416d8d0 con 0x7f2db40d0d50 2024-05-28T06:25:42.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:42.596+0000 7f2db1e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2db40b4120 0x7f2db414adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:42.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:42.596+0000 7f2db1e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2db40b4120 0x7f2db414adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:56294/0 (socket says 172.21.15.96:56294) 2024-05-28T06:25:42.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:42.596+0000 7f2db1e00640 1 -- 172.21.15.96:0/4167851393 learned_addr learned my addr 172.21.15.96:0/4167851393 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:25:42.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:42.597+0000 7f2db1e00640 1 -- 172.21.15.96:0/4167851393 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2db40d0d50 msgr2=0x7f2db40b11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:42.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:42.597+0000 7f2db1e00640 1 --2- 172.21.15.96:0/4167851393 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2db40d0d50 0x7f2db40b11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:42.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:42.596+0000 7f2db2800640 1 --2- 172.21.15.96:0/4167851393 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2db4162ac0 0x7f2db40b16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:42.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:42.597+0000 7f2db1e00640 1 -- 172.21.15.96:0/4167851393 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2db4162ac0 msgr2=0x7f2db40b16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:42.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:42.597+0000 7f2db1e00640 1 --2- 172.21.15.96:0/4167851393 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2db4162ac0 0x7f2db40b16f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:42.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:42.597+0000 7f2db1e00640 1 -- 172.21.15.96:0/4167851393 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2db414d390 con 0x7f2db40b4120 2024-05-28T06:25:42.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:42.597+0000 7f2db2800640 1 --2- 172.21.15.96:0/4167851393 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2db4162ac0 0x7f2db40b16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:25:42.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:42.597+0000 7f2db1e00640 1 --2- 172.21.15.96:0/4167851393 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2db40b4120 0x7f2db414adb0 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 crypto rx=0x7f2da400c400 tx=0x7f2da400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:25:42.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:42.597+0000 7f2daa000640 1 -- 172.21.15.96:0/4167851393 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2da4017070 con 0x7f2db40b4120 2024-05-28T06:25:42.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:42.597+0000 7f2daa000640 1 -- 172.21.15.96:0/4167851393 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2da40052d0 con 0x7f2db40b4120 2024-05-28T06:25:42.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:42.597+0000 7f2db8a00640 1 -- 172.21.15.96:0/4167851393 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2db415fa90 con 0x7f2db40b4120 2024-05-28T06:25:42.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:42.598+0000 7f2daa000640 1 -- 172.21.15.96:0/4167851393 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2da4003830 con 0x7f2db40b4120 2024-05-28T06:25:42.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:42.598+0000 7f2db8a00640 1 -- 172.21.15.96:0/4167851393 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2db4160050 con 0x7f2db40b4120 2024-05-28T06:25:42.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:42.599+0000 7f2db8a00640 1 -- 172.21.15.96:0/4167851393 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2db40689e0 con 0x7f2db40b4120 2024-05-28T06:25:42.594 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:42.601+0000 7f2daa000640 1 -- 172.21.15.96:0/4167851393 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 47) ==== 93998+0+0 (secure 0 0 0) 0x7f2da401a070 con 0x7f2db40b4120 2024-05-28T06:25:42.595 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:42.603+0000 7f2daa000640 1 --2- 172.21.15.96:0/4167851393 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2d9406e990 0x7f2d94070e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:42.596 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:42.603+0000 7f2daa000640 1 -- 172.21.15.96:0/4167851393 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 5823+0+0 (secure 0 0 0) 0x7f2da408a030 con 0x7f2db40b4120 2024-05-28T06:25:42.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:42.607+0000 7f2db1400640 1 --2- 172.21.15.96:0/4167851393 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2d9406e990 0x7f2d94070e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:42.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:42.608+0000 7f2db1400640 1 --2- 172.21.15.96:0/4167851393 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2d9406e990 0x7f2d94070e50 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f2da0034000 tx=0x7f2da002c000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:25:42.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:42.612+0000 7f2daa000640 1 -- 172.21.15.96:0/4167851393 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2da4059ed0 con 0x7f2db40b4120 2024-05-28T06:25:42.960 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:42.967+0000 7f2db8a00640 1 -- 172.21.15.96:0/4167851393 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone10997", "target": ["mon-mgr", ""]}) -- 0x7f2db41655e0 con 0x7f2d9406e990 2024-05-28T06:25:43.184 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.191+0000 7f2daa000640 1 -- 172.21.15.96:0/4167851393 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f2db41655e0 con 0x7f2d9406e990 2024-05-28T06:25:43.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.194+0000 7f2db8a00640 1 -- 172.21.15.96:0/4167851393 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2d9406e990 msgr2=0x7f2d94070e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:43.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.194+0000 7f2db8a00640 1 --2- 172.21.15.96:0/4167851393 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2d9406e990 0x7f2d94070e50 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f2da0034000 tx=0x7f2da002c000 comp rx=0 tx=0).stop 2024-05-28T06:25:43.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.194+0000 7f2db8a00640 1 -- 172.21.15.96:0/4167851393 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2db40b4120 msgr2=0x7f2db414adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:43.190 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.194+0000 7f2db8a00640 1 --2- 172.21.15.96:0/4167851393 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2db40b4120 0x7f2db414adb0 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 crypto rx=0x7f2da400c400 tx=0x7f2da400c8c0 comp rx=0 tx=0).stop 2024-05-28T06:25:43.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.194+0000 7f2db8a00640 1 -- 172.21.15.96:0/4167851393 shutdown_connections 2024-05-28T06:25:43.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.194+0000 7f2db8a00640 1 --2- 172.21.15.96:0/4167851393 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2d9406e990 0x7f2d94070e50 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:43.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.194+0000 7f2db8a00640 1 --2- 172.21.15.96:0/4167851393 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2db4162ac0 0x7f2db40b16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:43.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.194+0000 7f2db8a00640 1 --2- 172.21.15.96:0/4167851393 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2db40d0d50 0x7f2db40b11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:43.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.194+0000 7f2db8a00640 1 --2- 172.21.15.96:0/4167851393 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2db40b4120 0x7f2db414adb0 unknown :-1 s=CLOSED pgs=2099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:43.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.194+0000 7f2db8a00640 1 -- 172.21.15.96:0/4167851393 >> 172.21.15.96:0/4167851393 conn(0x7f2db406bd20 msgr2=0x7f2db40b05e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:25:43.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.194+0000 7f2db8a00640 1 -- 172.21.15.96:0/4167851393 shutdown_connections 2024-05-28T06:25:43.191 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.194+0000 7f2db8a00640 1 -- 172.21.15.96:0/4167851393 wait complete. 2024-05-28T06:25:43.209 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-05-28T06:25:43.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.409+0000 7f0c95200640 1 Processor -- start 2024-05-28T06:25:43.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.409+0000 7f0c95200640 1 -- start start 2024-05-28T06:25:43.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.410+0000 7f0c95200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f0c90145320 0x7f0c90071cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:43.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.410+0000 7f0c95200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f0c90072200 0x7f0c900b3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:43.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.410+0000 7f0c95200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0c900b41e0 0x7f0c90160240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:43.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.410+0000 7f0c95200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f0c900cf0c0 con 0x7f0c90145320 2024-05-28T06:25:43.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.411+0000 7f0c95200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f0c900b4a20 con 0x7f0c90072200 2024-05-28T06:25:43.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.411+0000 7f0c95200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f0c900b4d00 con 0x7f0c900b41e0 2024-05-28T06:25:43.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.411+0000 7f0c8de00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f0c90145320 0x7f0c90071cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:43.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.411+0000 7f0c8de00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f0c90145320 0x7f0c90071cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:56300/0 (socket says 172.21.15.96:56300) 2024-05-28T06:25:43.404 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.411+0000 7f0c8de00640 1 -- 172.21.15.96:0/2685181240 learned_addr learned my addr 172.21.15.96:0/2685181240 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:25:43.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.411+0000 7f0c87e00640 1 --2- 172.21.15.96:0/2685181240 >> v2:172.21.15.123:3300/0 conn(0x7f0c90072200 0x7f0c900b3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:43.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.411+0000 7f0c8e800640 1 --2- 172.21.15.96:0/2685181240 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0c900b41e0 0x7f0c90160240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:43.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.411+0000 7f0c8de00640 1 -- 172.21.15.96:0/2685181240 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0c900b41e0 msgr2=0x7f0c90160240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:25:43.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.411+0000 7f0c8de00640 1 --2- 172.21.15.96:0/2685181240 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0c900b41e0 0x7f0c90160240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:43.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.412+0000 7f0c8de00640 1 -- 172.21.15.96:0/2685181240 >> v2:172.21.15.123:3300/0 conn(0x7f0c90072200 msgr2=0x7f0c900b3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:25:43.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.412+0000 7f0c8de00640 1 --2- 172.21.15.96:0/2685181240 >> v2:172.21.15.123:3300/0 conn(0x7f0c90072200 0x7f0c900b3ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:43.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.412+0000 7f0c8de00640 1 -- 172.21.15.96:0/2685181240 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c900b4fe0 con 0x7f0c90145320 2024-05-28T06:25:43.405 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.412+0000 7f0c87e00640 1 --2- 172.21.15.96:0/2685181240 >> v2:172.21.15.123:3300/0 conn(0x7f0c90072200 0x7f0c900b3ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:25:43.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.413+0000 7f0c8de00640 1 --2- 172.21.15.96:0/2685181240 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c90145320 0x7f0c90071cc0 secure :-1 s=READY pgs=2100 cs=0 l=1 rev1=1 crypto rx=0x7f0c9006db90 tx=0x7f0c8002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92ff0b8880a29148 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:25:43.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.413+0000 7f0c8d400640 1 -- 172.21.15.96:0/2685181240 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c80037070 con 0x7f0c90145320 2024-05-28T06:25:43.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.413+0000 7f0c8d400640 1 -- 172.21.15.96:0/2685181240 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c80032440 con 0x7f0c90145320 2024-05-28T06:25:43.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.413+0000 7f0c8d400640 1 -- 172.21.15.96:0/2685181240 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c80032810 con 0x7f0c90145320 2024-05-28T06:25:43.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.414+0000 7f0c95200640 1 -- 172.21.15.96:0/2685181240 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c90145320 msgr2=0x7f0c90071cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:43.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.414+0000 7f0c95200640 1 --2- 172.21.15.96:0/2685181240 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c90145320 0x7f0c90071cc0 secure :-1 s=READY pgs=2100 cs=0 l=1 rev1=1 crypto rx=0x7f0c9006db90 tx=0x7f0c8002a1e0 comp rx=0 tx=0).stop 2024-05-28T06:25:43.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.414+0000 7f0c95200640 1 -- 172.21.15.96:0/2685181240 shutdown_connections 2024-05-28T06:25:43.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.414+0000 7f0c95200640 1 --2- 172.21.15.96:0/2685181240 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0c900b41e0 0x7f0c90160240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:43.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.414+0000 7f0c95200640 1 --2- 172.21.15.96:0/2685181240 >> v2:172.21.15.123:3300/0 conn(0x7f0c90072200 0x7f0c900b3ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:43.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.414+0000 7f0c95200640 1 --2- 172.21.15.96:0/2685181240 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c90145320 0x7f0c90071cc0 unknown :-1 s=CLOSED pgs=2100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:43.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.414+0000 7f0c95200640 1 -- 172.21.15.96:0/2685181240 >> 172.21.15.96:0/2685181240 conn(0x7f0c90067ce0 msgr2=0x7f0c9006b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:25:43.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.414+0000 7f0c95200640 1 -- 172.21.15.96:0/2685181240 shutdown_connections 2024-05-28T06:25:43.408 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.414+0000 7f0c95200640 1 -- 172.21.15.96:0/2685181240 wait complete. 2024-05-28T06:25:43.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.416+0000 7f0c95200640 1 Processor -- start 2024-05-28T06:25:43.409 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.417+0000 7f0c95200640 1 -- start start 2024-05-28T06:25:43.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.418+0000 7f0c95200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0c90072200 0x7f0c90143bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:43.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.418+0000 7f0c95200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c900b41e0 0x7f0c90134bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:43.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.418+0000 7f0c95200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0c90145320 0x7f0c901e68e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:43.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.418+0000 7f0c95200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f0c900b4670 con 0x7f0c900b41e0 2024-05-28T06:25:43.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.418+0000 7f0c95200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f0c90145700 con 0x7f0c90072200 2024-05-28T06:25:43.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.418+0000 7f0c95200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f0c901350f0 con 0x7f0c90145320 2024-05-28T06:25:43.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.418+0000 7f0c87e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c900b41e0 0x7f0c90134bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:43.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.418+0000 7f0c87e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c900b41e0 0x7f0c90134bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:56308/0 (socket says 172.21.15.96:56308) 2024-05-28T06:25:43.411 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.418+0000 7f0c87e00640 1 -- 172.21.15.96:0/3582387159 learned_addr learned my addr 172.21.15.96:0/3582387159 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:25:43.412 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.418+0000 7f0c87e00640 1 -- 172.21.15.96:0/3582387159 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0c90145320 msgr2=0x7f0c901e68e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:43.412 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.418+0000 7f0c87e00640 1 --2- 172.21.15.96:0/3582387159 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0c90145320 0x7f0c901e68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:43.412 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.418+0000 7f0c87e00640 1 -- 172.21.15.96:0/3582387159 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0c90072200 msgr2=0x7f0c90143bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:43.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.418+0000 7f0c8de00640 1 --2- 172.21.15.96:0/3582387159 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0c90072200 0x7f0c90143bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:43.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.418+0000 7f0c87e00640 1 --2- 172.21.15.96:0/3582387159 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0c90072200 0x7f0c90143bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:43.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.419+0000 7f0c87e00640 1 -- 172.21.15.96:0/3582387159 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c90142e00 con 0x7f0c900b41e0 2024-05-28T06:25:43.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.419+0000 7f0c87e00640 1 --2- 172.21.15.96:0/3582387159 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c900b41e0 0x7f0c90134bb0 secure :-1 s=READY pgs=2101 cs=0 l=1 rev1=1 crypto rx=0x7f0c7800c7f0 tx=0x7f0c7800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:25:43.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.419+0000 7f0c86000640 1 -- 172.21.15.96:0/3582387159 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c78015070 con 0x7f0c900b41e0 2024-05-28T06:25:43.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.419+0000 7f0c86000640 1 -- 172.21.15.96:0/3582387159 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c78007820 con 0x7f0c900b41e0 2024-05-28T06:25:43.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.419+0000 7f0c95200640 1 -- 172.21.15.96:0/3582387159 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c90142040 con 0x7f0c900b41e0 2024-05-28T06:25:43.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.419+0000 7f0c86000640 1 -- 172.21.15.96:0/3582387159 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c78007c10 con 0x7f0c900b41e0 2024-05-28T06:25:43.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.419+0000 7f0c95200640 1 -- 172.21.15.96:0/3582387159 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c90142600 con 0x7f0c900b41e0 2024-05-28T06:25:43.414 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.421+0000 7f0c95200640 1 -- 172.21.15.96:0/3582387159 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c900689e0 con 0x7f0c900b41e0 2024-05-28T06:25:43.414 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.422+0000 7f0c86000640 1 -- 172.21.15.96:0/3582387159 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 47) ==== 93998+0+0 (secure 0 0 0) 0x7f0c78018070 con 0x7f0c900b41e0 2024-05-28T06:25:43.415 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.423+0000 7f0c86000640 1 --2- 172.21.15.96:0/3582387159 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0c6c06e990 0x7f0c6c070e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:43.416 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.423+0000 7f0c86000640 1 -- 172.21.15.96:0/3582387159 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 5823+0+0 (secure 0 0 0) 0x7f0c7808ae30 con 0x7f0c900b41e0 2024-05-28T06:25:43.416 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.423+0000 7f0c8de00640 1 --2- 172.21.15.96:0/3582387159 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0c6c06e990 0x7f0c6c070e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:43.417 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.425+0000 7f0c8de00640 1 --2- 172.21.15.96:0/3582387159 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0c6c06e990 0x7f0c6c070e50 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f0c80034000 tx=0x7f0c8002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:25:43.420 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.427+0000 7f0c86000640 1 -- 172.21.15.96:0/3582387159 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0c7805ac40 con 0x7f0c900b41e0 2024-05-28T06:25:43.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.715+0000 7f0c95200640 1 -- 172.21.15.96:0/3582387159 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f0c90144130 con 0x7f0c900b41e0 2024-05-28T06:25:43.709 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.716+0000 7f0c86000640 1 -- 172.21.15.96:0/3582387159 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+2423 (secure 0 0 0) 0x7f0c7805a540 con 0x7f0c900b41e0 2024-05-28T06:25:43.709 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:25:43.709 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:25:43.709 INFO:teuthology.orchestra.run.smithi096.stdout: "fsid": "e4a831b5-e6d1-47a7-96cb-bf85154ee0d2", 2024-05-28T06:25:43.709 INFO:teuthology.orchestra.run.smithi096.stdout: "health": { 2024-05-28T06:25:43.710 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "HEALTH_OK", 2024-05-28T06:25:43.710 INFO:teuthology.orchestra.run.smithi096.stdout: "checks": {}, 2024-05-28T06:25:43.710 INFO:teuthology.orchestra.run.smithi096.stdout: "mutes": [] 2024-05-28T06:25:43.710 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:25:43.710 INFO:teuthology.orchestra.run.smithi096.stdout: "election_epoch": 8, 2024-05-28T06:25:43.710 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum": [ 2024-05-28T06:25:43.710 INFO:teuthology.orchestra.run.smithi096.stdout: 0, 2024-05-28T06:25:43.710 INFO:teuthology.orchestra.run.smithi096.stdout: 1, 2024-05-28T06:25:43.710 INFO:teuthology.orchestra.run.smithi096.stdout: 2 2024-05-28T06:25:43.710 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:25:43.711 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_names": [ 2024-05-28T06:25:43.711 INFO:teuthology.orchestra.run.smithi096.stdout: "a", 2024-05-28T06:25:43.711 INFO:teuthology.orchestra.run.smithi096.stdout: "b", 2024-05-28T06:25:43.711 INFO:teuthology.orchestra.run.smithi096.stdout: "c" 2024-05-28T06:25:43.711 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:25:43.711 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_age": 1317, 2024-05-28T06:25:43.711 INFO:teuthology.orchestra.run.smithi096.stdout: "monmap": { 2024-05-28T06:25:43.711 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 1, 2024-05-28T06:25:43.711 INFO:teuthology.orchestra.run.smithi096.stdout: "min_mon_release_name": "squid", 2024-05-28T06:25:43.711 INFO:teuthology.orchestra.run.smithi096.stdout: "num_mons": 3 2024-05-28T06:25:43.712 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:25:43.712 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap": { 2024-05-28T06:25:43.712 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 132, 2024-05-28T06:25:43.713 INFO:teuthology.orchestra.run.smithi096.stdout: "num_osds": 8, 2024-05-28T06:25:43.713 INFO:teuthology.orchestra.run.smithi096.stdout: "num_up_osds": 8, 2024-05-28T06:25:43.713 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_up_since": 1716876238, 2024-05-28T06:25:43.713 INFO:teuthology.orchestra.run.smithi096.stdout: "num_in_osds": 8, 2024-05-28T06:25:43.713 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_in_since": 1716876229, 2024-05-28T06:25:43.713 INFO:teuthology.orchestra.run.smithi096.stdout: "num_remapped_pgs": 0 2024-05-28T06:25:43.713 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:25:43.713 INFO:teuthology.orchestra.run.smithi096.stdout: "pgmap": { 2024-05-28T06:25:43.713 INFO:teuthology.orchestra.run.smithi096.stdout: "pgs_by_state": [ 2024-05-28T06:25:43.713 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:25:43.714 INFO:teuthology.orchestra.run.smithi096.stdout: "state_name": "active+clean", 2024-05-28T06:25:43.714 INFO:teuthology.orchestra.run.smithi096.stdout: "count": 97 2024-05-28T06:25:43.714 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:25:43.714 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:25:43.714 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pgs": 97, 2024-05-28T06:25:43.714 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pools": 3, 2024-05-28T06:25:43.714 INFO:teuthology.orchestra.run.smithi096.stdout: "num_objects": 4196, 2024-05-28T06:25:43.714 INFO:teuthology.orchestra.run.smithi096.stdout: "data_bytes": 17425095022, 2024-05-28T06:25:43.714 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_used": 35347124224, 2024-05-28T06:25:43.714 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_avail": 737746989056, 2024-05-28T06:25:43.715 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_total": 773094113280, 2024-05-28T06:25:43.715 INFO:teuthology.orchestra.run.smithi096.stdout: "read_bytes_sec": 795730656, 2024-05-28T06:25:43.715 INFO:teuthology.orchestra.run.smithi096.stdout: "write_bytes_sec": 793703448, 2024-05-28T06:25:43.715 INFO:teuthology.orchestra.run.smithi096.stdout: "read_op_per_sec": 192, 2024-05-28T06:25:43.715 INFO:teuthology.orchestra.run.smithi096.stdout: "write_op_per_sec": 211 2024-05-28T06:25:43.715 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:25:43.716 INFO:teuthology.orchestra.run.smithi096.stdout: "fsmap": { 2024-05-28T06:25:43.716 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 83, 2024-05-28T06:25:43.716 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 9, 2024-05-28T06:25:43.716 INFO:teuthology.orchestra.run.smithi096.stdout: "up": 1, 2024-05-28T06:25:43.717 INFO:teuthology.orchestra.run.smithi096.stdout: "in": 1, 2024-05-28T06:25:43.717 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 1, 2024-05-28T06:25:43.717 INFO:teuthology.orchestra.run.smithi096.stdout: "by_rank": [ 2024-05-28T06:25:43.717 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:25:43.717 INFO:teuthology.orchestra.run.smithi096.stdout: "filesystem_id": 9, 2024-05-28T06:25:43.718 INFO:teuthology.orchestra.run.smithi096.stdout: "rank": 0, 2024-05-28T06:25:43.718 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "a", 2024-05-28T06:25:43.718 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "up:active", 2024-05-28T06:25:43.718 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 9849 2024-05-28T06:25:43.718 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:25:43.718 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:25:43.718 INFO:teuthology.orchestra.run.smithi096.stdout: "up:standby": 3 2024-05-28T06:25:43.718 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:25:43.718 INFO:teuthology.orchestra.run.smithi096.stdout: "mgrmap": { 2024-05-28T06:25:43.719 INFO:teuthology.orchestra.run.smithi096.stdout: "available": true, 2024-05-28T06:25:43.719 INFO:teuthology.orchestra.run.smithi096.stdout: "num_standbys": 1, 2024-05-28T06:25:43.719 INFO:teuthology.orchestra.run.smithi096.stdout: "modules": [ 2024-05-28T06:25:43.719 INFO:teuthology.orchestra.run.smithi096.stdout: "iostat", 2024-05-28T06:25:43.719 INFO:teuthology.orchestra.run.smithi096.stdout: "nfs", 2024-05-28T06:25:43.719 INFO:teuthology.orchestra.run.smithi096.stdout: "restful" 2024-05-28T06:25:43.719 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:25:43.719 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:25:43.720 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:25:43.720 INFO:teuthology.orchestra.run.smithi096.stdout: "servicemap": { 2024-05-28T06:25:43.720 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 25, 2024-05-28T06:25:43.720 INFO:teuthology.orchestra.run.smithi096.stdout: "modified": "2024-05-28T06:25:30.049023+0000", 2024-05-28T06:25:43.720 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:25:43.720 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:25:43.720 INFO:teuthology.orchestra.run.smithi096.stdout: "progress_events": { 2024-05-28T06:25:43.720 INFO:teuthology.orchestra.run.smithi096.stdout: "f2372170-bf4f-4ba1-a7df-9e0c76ce0de6": { 2024-05-28T06:25:43.721 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "2 ongoing+pending clones; Avg progress is 11.198% (13s)\n [===.........................] (remaining: 109s)", 2024-05-28T06:25:43.721 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": 0.11197999864816666, 2024-05-28T06:25:43.721 INFO:teuthology.orchestra.run.smithi096.stdout: "add_to_ceph_s": true 2024-05-28T06:25:43.721 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:25:43.721 INFO:teuthology.orchestra.run.smithi096.stdout: "f7bbcf64-a8ee-4f95-9006-d32bd5fb36cf": { 2024-05-28T06:25:43.721 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "2 ongoing clones; Avg progress is 11.198% (20s)\n [===.........................] (remaining: 2m)", 2024-05-28T06:25:43.721 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": 0.11197999864816666, 2024-05-28T06:25:43.721 INFO:teuthology.orchestra.run.smithi096.stdout: "add_to_ceph_s": true 2024-05-28T06:25:43.721 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:25:43.722 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:25:43.722 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:25:43.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.719+0000 7f0c95200640 1 -- 172.21.15.96:0/3582387159 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0c6c06e990 msgr2=0x7f0c6c070e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:43.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.719+0000 7f0c95200640 1 --2- 172.21.15.96:0/3582387159 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0c6c06e990 0x7f0c6c070e50 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f0c80034000 tx=0x7f0c8002e000 comp rx=0 tx=0).stop 2024-05-28T06:25:43.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.719+0000 7f0c95200640 1 -- 172.21.15.96:0/3582387159 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c900b41e0 msgr2=0x7f0c90134bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:43.723 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.719+0000 7f0c95200640 1 --2- 172.21.15.96:0/3582387159 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c900b41e0 0x7f0c90134bb0 secure :-1 s=READY pgs=2101 cs=0 l=1 rev1=1 crypto rx=0x7f0c7800c7f0 tx=0x7f0c7800ccb0 comp rx=0 tx=0).stop 2024-05-28T06:25:43.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.719+0000 7f0c95200640 1 -- 172.21.15.96:0/3582387159 shutdown_connections 2024-05-28T06:25:43.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.719+0000 7f0c95200640 1 --2- 172.21.15.96:0/3582387159 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0c6c06e990 0x7f0c6c070e50 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:43.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.719+0000 7f0c95200640 1 --2- 172.21.15.96:0/3582387159 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0c90145320 0x7f0c901e68e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:43.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.719+0000 7f0c95200640 1 --2- 172.21.15.96:0/3582387159 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c900b41e0 0x7f0c90134bb0 unknown :-1 s=CLOSED pgs=2101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:43.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.719+0000 7f0c95200640 1 --2- 172.21.15.96:0/3582387159 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0c90072200 0x7f0c90143bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:43.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.719+0000 7f0c95200640 1 -- 172.21.15.96:0/3582387159 >> 172.21.15.96:0/3582387159 conn(0x7f0c90067ce0 msgr2=0x7f0c9015cd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:25:43.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.719+0000 7f0c95200640 1 -- 172.21.15.96:0/3582387159 shutdown_connections 2024-05-28T06:25:43.724 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:43.720+0000 7f0c95200640 1 -- 172.21.15.96:0/3582387159 wait complete. 2024-05-28T06:25:45.738 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-05-28T06:25:45.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:45.954+0000 7fccbbe00640 1 Processor -- start 2024-05-28T06:25:45.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:45.954+0000 7fccbbe00640 1 -- start start 2024-05-28T06:25:45.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:45.956+0000 7fccbbe00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fccb40b4410 0x7fccb406f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:45.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:45.956+0000 7fccbbe00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fccb4163160 0x7fccb4164960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:45.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:45.956+0000 7fccbbe00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fccb41638c0 con 0x7fccb4162d80 2024-05-28T06:25:45.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:45.956+0000 7fccbbe00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fccb4163ba0 con 0x7fccb40b4410 2024-05-28T06:25:45.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:45.956+0000 7fccbbe00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fccb4163e80 con 0x7fccb4163160 2024-05-28T06:25:45.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:45.956+0000 7fccb8a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fccb40b4410 0x7fccb406f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:45.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:45.956+0000 7fccb8a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fccb40b4410 0x7fccb406f180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:56310/0 (socket says 172.21.15.96:56310) 2024-05-28T06:25:45.949 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:45.956+0000 7fccb8a00640 1 -- 172.21.15.96:0/2105435504 learned_addr learned my addr 172.21.15.96:0/2105435504 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:25:45.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:45.957+0000 7fccb8a00640 1 -- 172.21.15.96:0/2105435504 >> v2:172.21.15.123:3300/0 conn(0x7fccb4163160 msgr2=0x7fccb4164960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:25:45.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:45.957+0000 7fccb8a00640 1 --2- 172.21.15.96:0/2105435504 >> v2:172.21.15.123:3300/0 conn(0x7fccb4163160 0x7fccb4164960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:45.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:45.957+0000 7fccb8a00640 1 -- 172.21.15.96:0/2105435504 >> v1:172.21.15.123:6789/0 conn(0x7fccb4162d80 legacy=0x7fccb40b3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:25:45.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:45.957+0000 7fccb8a00640 1 -- 172.21.15.96:0/2105435504 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccb4164160 con 0x7fccb40b4410 2024-05-28T06:25:45.950 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:45.958+0000 7fccb8a00640 1 --2- 172.21.15.96:0/2105435504 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fccb40b4410 0x7fccb406f180 secure :-1 s=READY pgs=2102 cs=0 l=1 rev1=1 crypto rx=0x7fccb40b3aa0 tx=0x7fcca002a2c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a897d01a6b627149 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:25:45.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:45.958+0000 7fccb3e00640 1 -- 172.21.15.96:0/2105435504 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcca0037070 con 0x7fccb40b4410 2024-05-28T06:25:45.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:45.958+0000 7fccb3e00640 1 -- 172.21.15.96:0/2105435504 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcca0032440 con 0x7fccb40b4410 2024-05-28T06:25:45.951 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:45.958+0000 7fccb3e00640 1 -- 172.21.15.96:0/2105435504 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcca0032810 con 0x7fccb40b4410 2024-05-28T06:25:45.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:45.959+0000 7fccbbe00640 1 -- 172.21.15.96:0/2105435504 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fccb40b4410 msgr2=0x7fccb406f180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:45.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:45.959+0000 7fccbbe00640 1 --2- 172.21.15.96:0/2105435504 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fccb40b4410 0x7fccb406f180 secure :-1 s=READY pgs=2102 cs=0 l=1 rev1=1 crypto rx=0x7fccb40b3aa0 tx=0x7fcca002a2c0 comp rx=0 tx=0).stop 2024-05-28T06:25:45.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:45.959+0000 7fccbbe00640 1 -- 172.21.15.96:0/2105435504 shutdown_connections 2024-05-28T06:25:45.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:45.959+0000 7fccbbe00640 1 --2- 172.21.15.96:0/2105435504 >> v2:172.21.15.123:3300/0 conn(0x7fccb4163160 0x7fccb4164960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:45.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:45.959+0000 7fccbbe00640 1 --2- 172.21.15.96:0/2105435504 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fccb40b4410 0x7fccb406f180 unknown :-1 s=CLOSED pgs=2102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:45.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:45.959+0000 7fccbbe00640 1 -- 172.21.15.96:0/2105435504 >> 172.21.15.96:0/2105435504 conn(0x7fccb4067ce0 msgr2=0x7fccb406b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:25:45.952 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:45.959+0000 7fccbbe00640 1 -- 172.21.15.96:0/2105435504 shutdown_connections 2024-05-28T06:25:45.953 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:45.959+0000 7fccbbe00640 1 -- 172.21.15.96:0/2105435504 wait complete. 2024-05-28T06:25:45.953 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:45.961+0000 7fccbbe00640 1 Processor -- start 2024-05-28T06:25:45.953 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:45.961+0000 7fccbbe00640 1 -- start start 2024-05-28T06:25:45.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:45.962+0000 7fccbbe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fccb40b4410 0x7fccb415cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:45.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:45.962+0000 7fccbbe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fccb4162d80 0x7fccb414b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:45.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:45.962+0000 7fccbbe00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fccb4163160 0x7fccb414b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:45.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:45.962+0000 7fccbbe00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fccb4164ea0 con 0x7fccb40b4410 2024-05-28T06:25:45.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:45.962+0000 7fccbbe00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fccb4163590 con 0x7fccb4162d80 2024-05-28T06:25:45.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:45.962+0000 7fccbbe00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fccb4165020 con 0x7fccb4163160 2024-05-28T06:25:45.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:45.962+0000 7fccb9400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fccb40b4410 0x7fccb415cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:45.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:45.962+0000 7fccb9400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fccb40b4410 0x7fccb415cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:56318/0 (socket says 172.21.15.96:56318) 2024-05-28T06:25:45.955 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:45.962+0000 7fccb9400640 1 -- 172.21.15.96:0/1776291620 learned_addr learned my addr 172.21.15.96:0/1776291620 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:25:45.956 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:45.962+0000 7fccb9400640 1 -- 172.21.15.96:0/1776291620 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fccb4163160 msgr2=0x7fccb414b640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:45.956 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:45.962+0000 7fccb9400640 1 --2- 172.21.15.96:0/1776291620 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fccb4163160 0x7fccb414b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:45.956 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:45.962+0000 7fccb9400640 1 -- 172.21.15.96:0/1776291620 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fccb4162d80 msgr2=0x7fccb414b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:45.956 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:45.962+0000 7fccb9400640 1 --2- 172.21.15.96:0/1776291620 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fccb4162d80 0x7fccb414b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:45.956 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:45.962+0000 7fccb9400640 1 -- 172.21.15.96:0/1776291620 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccb40cd0d0 con 0x7fccb40b4410 2024-05-28T06:25:45.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:45.963+0000 7fccb9400640 1 --2- 172.21.15.96:0/1776291620 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fccb40b4410 0x7fccb415cd60 secure :-1 s=READY pgs=2103 cs=0 l=1 rev1=1 crypto rx=0x7fcca800c460 tx=0x7fcca800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:25:45.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:45.963+0000 7fccb1600640 1 -- 172.21.15.96:0/1776291620 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcca8016020 con 0x7fccb40b4410 2024-05-28T06:25:45.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:45.963+0000 7fccb1600640 1 -- 172.21.15.96:0/1776291620 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcca8012870 con 0x7fccb40b4410 2024-05-28T06:25:45.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:45.963+0000 7fccb1600640 1 -- 172.21.15.96:0/1776291620 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcca8012c60 con 0x7fccb40b4410 2024-05-28T06:25:45.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:45.963+0000 7fccbbe00640 1 -- 172.21.15.96:0/1776291620 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fccb40cd3b0 con 0x7fccb40b4410 2024-05-28T06:25:45.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:45.963+0000 7fccbbe00640 1 -- 172.21.15.96:0/1776291620 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fccb415d2a0 con 0x7fccb40b4410 2024-05-28T06:25:45.957 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:45.964+0000 7fccbbe00640 1 -- 172.21.15.96:0/1776291620 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fccb40689e0 con 0x7fccb40b4410 2024-05-28T06:25:45.958 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:45.965+0000 7fccb1600640 1 -- 172.21.15.96:0/1776291620 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 47) ==== 93998+0+0 (secure 0 0 0) 0x7fcca8019020 con 0x7fccb40b4410 2024-05-28T06:25:45.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:45.966+0000 7fccb1600640 1 --2- 172.21.15.96:0/1776291620 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fcc9406e990 0x7fcc94070e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:45.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:45.966+0000 7fccb1600640 1 -- 172.21.15.96:0/1776291620 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 5823+0+0 (secure 0 0 0) 0x7fcca808a990 con 0x7fccb40b4410 2024-05-28T06:25:45.959 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:45.967+0000 7fccb8a00640 1 --2- 172.21.15.96:0/1776291620 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fcc9406e990 0x7fcc94070e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:45.961 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:45.968+0000 7fccb8a00640 1 --2- 172.21.15.96:0/1776291620 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fcc9406e990 0x7fcc94070e50 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fcca0034000 tx=0x7fcca002c000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:25:45.963 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:45.971+0000 7fccb1600640 1 -- 172.21.15.96:0/1776291620 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fcca8020030 con 0x7fccb40b4410 2024-05-28T06:25:46.244 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.251+0000 7fccbbe00640 1 -- 172.21.15.96:0/1776291620 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fccb406cca0 con 0x7fccb40b4410 2024-05-28T06:25:46.244 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.252+0000 7fccb1600640 1 -- 172.21.15.96:0/1776291620 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+2381 (secure 0 0 0) 0x7fcca805a820 con 0x7fccb40b4410 2024-05-28T06:25:46.245 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:25:46.245 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:25:46.245 INFO:teuthology.orchestra.run.smithi096.stdout: "fsid": "e4a831b5-e6d1-47a7-96cb-bf85154ee0d2", 2024-05-28T06:25:46.245 INFO:teuthology.orchestra.run.smithi096.stdout: "health": { 2024-05-28T06:25:46.245 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "HEALTH_OK", 2024-05-28T06:25:46.245 INFO:teuthology.orchestra.run.smithi096.stdout: "checks": {}, 2024-05-28T06:25:46.245 INFO:teuthology.orchestra.run.smithi096.stdout: "mutes": [] 2024-05-28T06:25:46.245 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:25:46.245 INFO:teuthology.orchestra.run.smithi096.stdout: "election_epoch": 8, 2024-05-28T06:25:46.245 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum": [ 2024-05-28T06:25:46.245 INFO:teuthology.orchestra.run.smithi096.stdout: 0, 2024-05-28T06:25:46.246 INFO:teuthology.orchestra.run.smithi096.stdout: 1, 2024-05-28T06:25:46.246 INFO:teuthology.orchestra.run.smithi096.stdout: 2 2024-05-28T06:25:46.246 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:25:46.246 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_names": [ 2024-05-28T06:25:46.246 INFO:teuthology.orchestra.run.smithi096.stdout: "a", 2024-05-28T06:25:46.246 INFO:teuthology.orchestra.run.smithi096.stdout: "b", 2024-05-28T06:25:46.246 INFO:teuthology.orchestra.run.smithi096.stdout: "c" 2024-05-28T06:25:46.246 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:25:46.246 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_age": 1319, 2024-05-28T06:25:46.246 INFO:teuthology.orchestra.run.smithi096.stdout: "monmap": { 2024-05-28T06:25:46.246 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 1, 2024-05-28T06:25:46.246 INFO:teuthology.orchestra.run.smithi096.stdout: "min_mon_release_name": "squid", 2024-05-28T06:25:46.247 INFO:teuthology.orchestra.run.smithi096.stdout: "num_mons": 3 2024-05-28T06:25:46.247 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:25:46.247 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap": { 2024-05-28T06:25:46.247 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 132, 2024-05-28T06:25:46.247 INFO:teuthology.orchestra.run.smithi096.stdout: "num_osds": 8, 2024-05-28T06:25:46.247 INFO:teuthology.orchestra.run.smithi096.stdout: "num_up_osds": 8, 2024-05-28T06:25:46.247 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_up_since": 1716876238, 2024-05-28T06:25:46.247 INFO:teuthology.orchestra.run.smithi096.stdout: "num_in_osds": 8, 2024-05-28T06:25:46.247 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_in_since": 1716876229, 2024-05-28T06:25:46.247 INFO:teuthology.orchestra.run.smithi096.stdout: "num_remapped_pgs": 0 2024-05-28T06:25:46.247 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:25:46.247 INFO:teuthology.orchestra.run.smithi096.stdout: "pgmap": { 2024-05-28T06:25:46.247 INFO:teuthology.orchestra.run.smithi096.stdout: "pgs_by_state": [ 2024-05-28T06:25:46.247 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:25:46.248 INFO:teuthology.orchestra.run.smithi096.stdout: "state_name": "active+clean", 2024-05-28T06:25:46.248 INFO:teuthology.orchestra.run.smithi096.stdout: "count": 97 2024-05-28T06:25:46.248 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:25:46.248 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:25:46.248 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pgs": 97, 2024-05-28T06:25:46.248 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pools": 3, 2024-05-28T06:25:46.248 INFO:teuthology.orchestra.run.smithi096.stdout: "num_objects": 4618, 2024-05-28T06:25:46.248 INFO:teuthology.orchestra.run.smithi096.stdout: "data_bytes": 19216285008, 2024-05-28T06:25:46.248 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_used": 39541395456, 2024-05-28T06:25:46.248 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_avail": 733552717824, 2024-05-28T06:25:46.248 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_total": 773094113280, 2024-05-28T06:25:46.248 INFO:teuthology.orchestra.run.smithi096.stdout: "read_bytes_sec": 777387543, 2024-05-28T06:25:46.248 INFO:teuthology.orchestra.run.smithi096.stdout: "write_bytes_sec": 783045954, 2024-05-28T06:25:46.249 INFO:teuthology.orchestra.run.smithi096.stdout: "read_op_per_sec": 189, 2024-05-28T06:25:46.249 INFO:teuthology.orchestra.run.smithi096.stdout: "write_op_per_sec": 208 2024-05-28T06:25:46.249 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:25:46.249 INFO:teuthology.orchestra.run.smithi096.stdout: "fsmap": { 2024-05-28T06:25:46.249 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 83, 2024-05-28T06:25:46.249 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 9, 2024-05-28T06:25:46.249 INFO:teuthology.orchestra.run.smithi096.stdout: "up": 1, 2024-05-28T06:25:46.249 INFO:teuthology.orchestra.run.smithi096.stdout: "in": 1, 2024-05-28T06:25:46.249 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 1, 2024-05-28T06:25:46.249 INFO:teuthology.orchestra.run.smithi096.stdout: "by_rank": [ 2024-05-28T06:25:46.249 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:25:46.249 INFO:teuthology.orchestra.run.smithi096.stdout: "filesystem_id": 9, 2024-05-28T06:25:46.249 INFO:teuthology.orchestra.run.smithi096.stdout: "rank": 0, 2024-05-28T06:25:46.249 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "a", 2024-05-28T06:25:46.249 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "up:active", 2024-05-28T06:25:46.250 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 9849 2024-05-28T06:25:46.250 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:25:46.250 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:25:46.250 INFO:teuthology.orchestra.run.smithi096.stdout: "up:standby": 3 2024-05-28T06:25:46.250 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:25:46.250 INFO:teuthology.orchestra.run.smithi096.stdout: "mgrmap": { 2024-05-28T06:25:46.250 INFO:teuthology.orchestra.run.smithi096.stdout: "available": true, 2024-05-28T06:25:46.250 INFO:teuthology.orchestra.run.smithi096.stdout: "num_standbys": 1, 2024-05-28T06:25:46.250 INFO:teuthology.orchestra.run.smithi096.stdout: "modules": [ 2024-05-28T06:25:46.250 INFO:teuthology.orchestra.run.smithi096.stdout: "iostat", 2024-05-28T06:25:46.250 INFO:teuthology.orchestra.run.smithi096.stdout: "nfs", 2024-05-28T06:25:46.251 INFO:teuthology.orchestra.run.smithi096.stdout: "restful" 2024-05-28T06:25:46.251 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:25:46.251 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:25:46.251 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:25:46.251 INFO:teuthology.orchestra.run.smithi096.stdout: "servicemap": { 2024-05-28T06:25:46.251 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 25, 2024-05-28T06:25:46.251 INFO:teuthology.orchestra.run.smithi096.stdout: "modified": "2024-05-28T06:25:30.049023+0000", 2024-05-28T06:25:46.251 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:25:46.251 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:25:46.251 INFO:teuthology.orchestra.run.smithi096.stdout: "progress_events": { 2024-05-28T06:25:46.251 INFO:teuthology.orchestra.run.smithi096.stdout: "f2372170-bf4f-4ba1-a7df-9e0c76ce0de6": { 2024-05-28T06:25:46.251 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "No pending clones left, waiting 28.0 seconds before exiting (17s)\n [============================] ", 2024-05-28T06:25:46.251 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": 1, 2024-05-28T06:25:46.251 INFO:teuthology.orchestra.run.smithi096.stdout: "add_to_ceph_s": true 2024-05-28T06:25:46.252 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:25:46.252 INFO:teuthology.orchestra.run.smithi096.stdout: "f7bbcf64-a8ee-4f95-9006-d32bd5fb36cf": { 2024-05-28T06:25:46.252 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "All cloning finished, waiting 28.0 seconds before exiting (24s)\n [============================] ", 2024-05-28T06:25:46.252 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": 1, 2024-05-28T06:25:46.252 INFO:teuthology.orchestra.run.smithi096.stdout: "add_to_ceph_s": true 2024-05-28T06:25:46.252 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:25:46.252 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:25:46.252 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:25:46.253 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.254+0000 7fccbbe00640 1 -- 172.21.15.96:0/1776291620 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fcc9406e990 msgr2=0x7fcc94070e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:46.253 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.254+0000 7fccbbe00640 1 --2- 172.21.15.96:0/1776291620 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fcc9406e990 0x7fcc94070e50 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fcca0034000 tx=0x7fcca002c000 comp rx=0 tx=0).stop 2024-05-28T06:25:46.253 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.254+0000 7fccbbe00640 1 -- 172.21.15.96:0/1776291620 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fccb40b4410 msgr2=0x7fccb415cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:46.253 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.254+0000 7fccbbe00640 1 --2- 172.21.15.96:0/1776291620 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fccb40b4410 0x7fccb415cd60 secure :-1 s=READY pgs=2103 cs=0 l=1 rev1=1 crypto rx=0x7fcca800c460 tx=0x7fcca800c920 comp rx=0 tx=0).stop 2024-05-28T06:25:46.253 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.254+0000 7fccbbe00640 1 -- 172.21.15.96:0/1776291620 shutdown_connections 2024-05-28T06:25:46.253 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.254+0000 7fccbbe00640 1 --2- 172.21.15.96:0/1776291620 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fcc9406e990 0x7fcc94070e50 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:46.253 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.254+0000 7fccbbe00640 1 --2- 172.21.15.96:0/1776291620 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fccb4163160 0x7fccb414b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:46.253 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.254+0000 7fccbbe00640 1 --2- 172.21.15.96:0/1776291620 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fccb4162d80 0x7fccb414b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:46.253 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.254+0000 7fccbbe00640 1 --2- 172.21.15.96:0/1776291620 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fccb40b4410 0x7fccb415cd60 unknown :-1 s=CLOSED pgs=2103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:46.253 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.254+0000 7fccbbe00640 1 -- 172.21.15.96:0/1776291620 >> 172.21.15.96:0/1776291620 conn(0x7fccb4067ce0 msgr2=0x7fccb40b2060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:25:46.254 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.255+0000 7fccbbe00640 1 -- 172.21.15.96:0/1776291620 shutdown_connections 2024-05-28T06:25:46.254 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.255+0000 7fccbbe00640 1 -- 172.21.15.96:0/1776291620 wait complete. 2024-05-28T06:25:46.266 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:25:46.266 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls -lh volumes/_nogroup/subvol_clone10991/aeb880aa-33ae-4200-bc32-7e8a9ed82480 2024-05-28T06:25:46.267 DEBUG:teuthology.orchestra.run.smithi096:> ) 2024-05-28T06:25:46.321 INFO:teuthology.orchestra.run.smithi096.stderr:ls: cannot access 'volumes/_nogroup/subvol_clone10991/aeb880aa-33ae-4200-bc32-7e8a9ed82480': No such file or directory 2024-05-28T06:25:46.322 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-05-28T06:25:46.323 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-05-28T06:25:46.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.533+0000 7fd7da400640 1 Processor -- start 2024-05-28T06:25:46.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.534+0000 7fd7da400640 1 -- start start 2024-05-28T06:25:46.528 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.535+0000 7fd7da400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd7d40d0930 0x7fd7d40ad840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:46.528 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.535+0000 7fd7da400640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7d406ed10 con 0x7fd7d406f860 2024-05-28T06:25:46.528 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.535+0000 7fd7da400640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7d415cb00 con 0x7fd7d40d04a0 2024-05-28T06:25:46.528 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.535+0000 7fd7da400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fd7d40b3cc0 con 0x7fd7d40d0930 2024-05-28T06:25:46.528 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.535+0000 7fd7d3400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fd7d406f860 0x7fd7d40b3150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:60354/0 (socket says 172.21.15.96:60354) 2024-05-28T06:25:46.528 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.535+0000 7fd7d3400640 1 -- 172.21.15.96:0/3974872805 learned_addr learned my addr 172.21.15.96:0/3974872805 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:25:46.530 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.536+0000 7fd7d2a00640 1 -- 172.21.15.96:0/3974872805 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3834694975 0 0) 0x7fd7c4001110 con 0x7fd7d406f860 2024-05-28T06:25:46.530 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.536+0000 7fd7d3e00640 1 --2- 172.21.15.96:0/3974872805 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd7d40d0930 0x7fd7d40ad840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:46.530 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.536+0000 7fd7d2a00640 1 -- 172.21.15.96:0/3974872805 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd7b8003520 con 0x7fd7d406f860 2024-05-28T06:25:46.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.536+0000 7fd7d2a00640 1 -- 172.21.15.96:0/3974872805 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1190630423 0 0) 0x7fd7b8003520 con 0x7fd7d406f860 2024-05-28T06:25:46.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.536+0000 7fd7d2a00640 1 -- 172.21.15.96:0/3974872805 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd7d40d0930 msgr2=0x7fd7d40ad840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:25:46.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.536+0000 7fd7d2a00640 1 --2- 172.21.15.96:0/3974872805 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd7d40d0930 0x7fd7d40ad840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:46.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.536+0000 7fd7d2a00640 1 -- 172.21.15.96:0/3974872805 >> v1:172.21.15.123:6789/0 conn(0x7fd7d40d04a0 legacy=0x7fd7d40ad130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:46.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.536+0000 7fd7d2a00640 1 -- 172.21.15.96:0/3974872805 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7d40b3fa0 con 0x7fd7d406f860 2024-05-28T06:25:46.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.536+0000 7fd7d2a00640 1 -- 172.21.15.96:0/3974872805 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7fd7c4001840 con 0x7fd7d406f860 2024-05-28T06:25:46.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.536+0000 7fd7d3e00640 1 --2- 172.21.15.96:0/3974872805 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd7d40d0930 0x7fd7d40ad840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-28T06:25:46.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.536+0000 7fd7d2a00640 1 -- 172.21.15.96:0/3974872805 >> v1:172.21.15.96:6789/0 conn(0x7fd7d406f860 legacy=0x7fd7d40b3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:46.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.536+0000 7fd7d2a00640 1 --2- 172.21.15.96:0/3974872805 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd7b80039e0 0x7fd7b8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:46.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.536+0000 7fd7d2a00640 1 --2- 172.21.15.96:0/3974872805 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd7d40d0930 0x7fd7b8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:46.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.536+0000 7fd7d2a00640 1 --2- 172.21.15.96:0/3974872805 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7b8004d40 0x7fd7b80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:46.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.536+0000 7fd7d2a00640 1 -- 172.21.15.96:0/3974872805 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd7d40b3cc0 con 0x7fd7b80039e0 2024-05-28T06:25:46.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.536+0000 7fd7d2a00640 1 -- 172.21.15.96:0/3974872805 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd7b800bad0 con 0x7fd7b8004d40 2024-05-28T06:25:46.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.536+0000 7fd7d2a00640 1 -- 172.21.15.96:0/3974872805 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fd7b800bcd0 con 0x7fd7d40d0930 2024-05-28T06:25:46.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.537+0000 7fd7d3400640 1 --2- 172.21.15.96:0/3974872805 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd7b80039e0 0x7fd7b8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:46.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.537+0000 7fd7d3e00640 1 --2- 172.21.15.96:0/3974872805 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7b8004d40 0x7fd7b80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:46.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.537+0000 7fd7d3400640 1 -- 172.21.15.96:0/3974872805 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd7d40d0930 msgr2=0x7fd7b8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:25:46.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.537+0000 7fd7cbe00640 1 --2- 172.21.15.96:0/3974872805 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd7d40d0930 0x7fd7b8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:46.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.537+0000 7fd7d3400640 1 --2- 172.21.15.96:0/3974872805 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd7d40d0930 0x7fd7b8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:46.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.537+0000 7fd7d3400640 1 -- 172.21.15.96:0/3974872805 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7b8004d40 msgr2=0x7fd7b80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:25:46.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.537+0000 7fd7d3400640 1 --2- 172.21.15.96:0/3974872805 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7b8004d40 0x7fd7b80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:46.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.537+0000 7fd7d3400640 1 -- 172.21.15.96:0/3974872805 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7b8003520 con 0x7fd7b80039e0 2024-05-28T06:25:46.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.537+0000 7fd7d3e00640 1 --2- 172.21.15.96:0/3974872805 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7b8004d40 0x7fd7b80051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:25:46.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.537+0000 7fd7cbe00640 1 --2- 172.21.15.96:0/3974872805 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd7d40d0930 0x7fd7b8004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:25:46.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.538+0000 7fd7d3400640 1 --2- 172.21.15.96:0/3974872805 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd7b80039e0 0x7fd7b8003e00 secure :-1 s=READY pgs=2105 cs=0 l=1 rev1=1 crypto rx=0x7fd7c400c490 tx=0x7fd7b8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=401736dee7cb0027 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:25:46.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.538+0000 7fd7d2a00640 1 -- 172.21.15.96:0/3974872805 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7c400c0e0 con 0x7fd7b80039e0 2024-05-28T06:25:46.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.538+0000 7fd7d2a00640 1 -- 172.21.15.96:0/3974872805 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7c4005af0 con 0x7fd7b80039e0 2024-05-28T06:25:46.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.538+0000 7fd7d2a00640 1 -- 172.21.15.96:0/3974872805 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7c4007070 con 0x7fd7b80039e0 2024-05-28T06:25:46.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.539+0000 7fd7da400640 1 -- 172.21.15.96:0/3974872805 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd7b80039e0 msgr2=0x7fd7b8003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:46.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.539+0000 7fd7da400640 1 --2- 172.21.15.96:0/3974872805 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd7b80039e0 0x7fd7b8003e00 secure :-1 s=READY pgs=2105 cs=0 l=1 rev1=1 crypto rx=0x7fd7c400c490 tx=0x7fd7b8005830 comp rx=0 tx=0).stop 2024-05-28T06:25:46.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.539+0000 7fd7d3e00640 1 -- 172.21.15.96:0/3974872805 reap_dead start 2024-05-28T06:25:46.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.539+0000 7fd7da400640 1 -- 172.21.15.96:0/3974872805 shutdown_connections 2024-05-28T06:25:46.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.539+0000 7fd7da400640 1 -- 172.21.15.96:0/3974872805 >> 172.21.15.96:0/3974872805 conn(0x7fd7d406bae0 msgr2=0x7fd7d406bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:25:46.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.539+0000 7fd7da400640 1 -- 172.21.15.96:0/3974872805 shutdown_connections 2024-05-28T06:25:46.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.539+0000 7fd7da400640 1 -- 172.21.15.96:0/3974872805 wait complete. 2024-05-28T06:25:46.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.541+0000 7fd7da400640 1 Processor -- start 2024-05-28T06:25:46.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.541+0000 7fd7da400640 1 -- start start 2024-05-28T06:25:46.536 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.542+0000 7fd7da400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7b80039e0 0x7fd7d40c65b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:46.536 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.542+0000 7fd7da400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd7d415d6d0 0x7fd7d415ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:46.536 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.542+0000 7fd7da400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd7d40c8a00 0x7fd7d40c8e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:46.536 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.542+0000 7fd7da400640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd7d40b4840 con 0x7fd7d40c8a00 2024-05-28T06:25:46.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.542+0000 7fd7da400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd7d40b4560 con 0x7fd7b80039e0 2024-05-28T06:25:46.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.542+0000 7fd7da400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fd7d40d0d80 con 0x7fd7d415d6d0 2024-05-28T06:25:46.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.542+0000 7fd7d3e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd7d40c8a00 0x7fd7d40c8e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:46.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.542+0000 7fd7d3e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd7d40c8a00 0x7fd7d40c8e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:56342/0 (socket says 172.21.15.96:56342) 2024-05-28T06:25:46.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.542+0000 7fd7d3e00640 1 -- 172.21.15.96:0/942823959 learned_addr learned my addr 172.21.15.96:0/942823959 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:25:46.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.543+0000 7fd7cbe00640 1 --2- 172.21.15.96:0/942823959 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd7d415d6d0 0x7fd7d415ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:46.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.543+0000 7fd7d3400640 1 --2- 172.21.15.96:0/942823959 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7b80039e0 0x7fd7d40c65b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:46.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.543+0000 7fd7d3e00640 1 -- 172.21.15.96:0/942823959 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd7d415d6d0 msgr2=0x7fd7d415ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:46.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.543+0000 7fd7d3e00640 1 --2- 172.21.15.96:0/942823959 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd7d415d6d0 0x7fd7d415ebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:46.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.543+0000 7fd7d3e00640 1 -- 172.21.15.96:0/942823959 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7b80039e0 msgr2=0x7fd7d40c65b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:46.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.543+0000 7fd7d3e00640 1 --2- 172.21.15.96:0/942823959 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7b80039e0 0x7fd7d40c65b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:46.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.543+0000 7fd7d3e00640 1 -- 172.21.15.96:0/942823959 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7d415dc20 con 0x7fd7d40c8a00 2024-05-28T06:25:46.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.543+0000 7fd7d3e00640 1 --2- 172.21.15.96:0/942823959 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd7d40c8a00 0x7fd7d40c8e80 secure :-1 s=READY pgs=2106 cs=0 l=1 rev1=1 crypto rx=0x7fd7cc0042d0 tx=0x7fd7cc0046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:25:46.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.543+0000 7fd7cb400640 1 -- 172.21.15.96:0/942823959 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7cc019070 con 0x7fd7d40c8a00 2024-05-28T06:25:46.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.543+0000 7fd7cb400640 1 -- 172.21.15.96:0/942823959 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7cc007d20 con 0x7fd7d40c8a00 2024-05-28T06:25:46.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.543+0000 7fd7cb400640 1 -- 172.21.15.96:0/942823959 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7cc014620 con 0x7fd7d40c8a00 2024-05-28T06:25:46.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.543+0000 7fd7da400640 1 -- 172.21.15.96:0/942823959 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7d415df00 con 0x7fd7d40c8a00 2024-05-28T06:25:46.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.543+0000 7fd7da400640 1 -- 172.21.15.96:0/942823959 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7d415e4c0 con 0x7fd7d40c8a00 2024-05-28T06:25:46.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.544+0000 7fd7da400640 1 -- 172.21.15.96:0/942823959 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7d40689e0 con 0x7fd7d40c8a00 2024-05-28T06:25:46.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.545+0000 7fd7cb400640 1 -- 172.21.15.96:0/942823959 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 47) ==== 93998+0+0 (secure 0 0 0) 0x7fd7cc01c070 con 0x7fd7d40c8a00 2024-05-28T06:25:46.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.546+0000 7fd7cb400640 1 --2- 172.21.15.96:0/942823959 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd7b006e990 0x7fd7b0070e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:46.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.547+0000 7fd7cb400640 1 -- 172.21.15.96:0/942823959 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 5823+0+0 (secure 0 0 0) 0x7fd7cc08a610 con 0x7fd7d40c8a00 2024-05-28T06:25:46.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.547+0000 7fd7d3400640 1 --2- 172.21.15.96:0/942823959 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd7b006e990 0x7fd7b0070e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:46.541 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.548+0000 7fd7d3400640 1 --2- 172.21.15.96:0/942823959 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd7b006e990 0x7fd7b0070e50 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fd7b8004780 tx=0x7fd7c402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:25:46.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.551+0000 7fd7cb400640 1 -- 172.21.15.96:0/942823959 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd7cc05a540 con 0x7fd7d40c8a00 2024-05-28T06:25:46.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.784+0000 7fd7da400640 1 -- 172.21.15.96:0/942823959 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7fd7d415e830 con 0x7fd7b006e990 2024-05-28T06:25:46.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.787+0000 7fd7cb400640 1 -- 172.21.15.96:0/942823959 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7fd7d415e830 con 0x7fd7b006e990 2024-05-28T06:25:46.780 INFO:teuthology.orchestra.run.smithi096.stdout:[ 2024-05-28T06:25:46.780 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:25:46.780 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "cephfs" 2024-05-28T06:25:46.780 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:25:46.781 INFO:teuthology.orchestra.run.smithi096.stdout:] 2024-05-28T06:25:46.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.790+0000 7fd7da400640 1 -- 172.21.15.96:0/942823959 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd7b006e990 msgr2=0x7fd7b0070e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:46.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.790+0000 7fd7da400640 1 --2- 172.21.15.96:0/942823959 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd7b006e990 0x7fd7b0070e50 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fd7b8004780 tx=0x7fd7c402e000 comp rx=0 tx=0).stop 2024-05-28T06:25:46.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.790+0000 7fd7da400640 1 -- 172.21.15.96:0/942823959 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd7d40c8a00 msgr2=0x7fd7d40c8e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:46.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.790+0000 7fd7da400640 1 --2- 172.21.15.96:0/942823959 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd7d40c8a00 0x7fd7d40c8e80 secure :-1 s=READY pgs=2106 cs=0 l=1 rev1=1 crypto rx=0x7fd7cc0042d0 tx=0x7fd7cc0046d0 comp rx=0 tx=0).stop 2024-05-28T06:25:46.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.790+0000 7fd7da400640 1 -- 172.21.15.96:0/942823959 shutdown_connections 2024-05-28T06:25:46.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.790+0000 7fd7da400640 1 --2- 172.21.15.96:0/942823959 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd7b006e990 0x7fd7b0070e50 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:46.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.790+0000 7fd7da400640 1 --2- 172.21.15.96:0/942823959 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd7d40c8a00 0x7fd7d40c8e80 unknown :-1 s=CLOSED pgs=2106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:46.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.790+0000 7fd7da400640 1 --2- 172.21.15.96:0/942823959 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd7d415d6d0 0x7fd7d415ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:46.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.790+0000 7fd7da400640 1 --2- 172.21.15.96:0/942823959 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7b80039e0 0x7fd7d40c65b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:46.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.790+0000 7fd7da400640 1 -- 172.21.15.96:0/942823959 >> 172.21.15.96:0/942823959 conn(0x7fd7d406bae0 msgr2=0x7fd7d40cb590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:25:46.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.790+0000 7fd7da400640 1 -- 172.21.15.96:0/942823959 shutdown_connections 2024-05-28T06:25:46.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.790+0000 7fd7da400640 1 -- 172.21.15.96:0/942823959 wait complete. 2024-05-28T06:25:46.802 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-05-28T06:25:46.988 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.995+0000 7fe91be00640 1 Processor -- start 2024-05-28T06:25:46.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.996+0000 7fe91be00640 1 -- start start 2024-05-28T06:25:46.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.997+0000 7fe91be00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fe91c0ca870 0x7fe91c0ccc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:46.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.997+0000 7fe91be00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fe91c0af720 0x7fe91c0cd140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:46.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.997+0000 7fe91be00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe91c0b4be0 con 0x7fe91c15ac80 2024-05-28T06:25:46.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.997+0000 7fe91be00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fe91c0cf8a0 con 0x7fe91c0ca870 2024-05-28T06:25:46.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.997+0000 7fe91be00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fe91c0cde90 con 0x7fe91c0af720 2024-05-28T06:25:46.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.997+0000 7fe912000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fe91c0ca870 0x7fe91c0ccc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:46.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.997+0000 7fe912000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fe91c0ca870 0x7fe91c0ccc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:56358/0 (socket says 172.21.15.96:56358) 2024-05-28T06:25:46.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.997+0000 7fe912000640 1 -- 172.21.15.96:0/310332286 learned_addr learned my addr 172.21.15.96:0/310332286 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:25:46.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.998+0000 7fe912000640 1 -- 172.21.15.96:0/310332286 >> v2:172.21.15.123:3300/0 conn(0x7fe91c0af720 msgr2=0x7fe91c0cd140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:25:46.991 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.998+0000 7fe912000640 1 --2- 172.21.15.96:0/310332286 >> v2:172.21.15.123:3300/0 conn(0x7fe91c0af720 0x7fe91c0cd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:46.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.998+0000 7fe912000640 1 -- 172.21.15.96:0/310332286 >> v1:172.21.15.123:6789/0 conn(0x7fe91c15ac80 legacy=0x7fe91c15f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:25:46.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.998+0000 7fe912000640 1 -- 172.21.15.96:0/310332286 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe91c0ce170 con 0x7fe91c0ca870 2024-05-28T06:25:46.992 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.999+0000 7fe912000640 1 --2- 172.21.15.96:0/310332286 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe91c0ca870 0x7fe91c0ccc00 secure :-1 s=READY pgs=2107 cs=0 l=1 rev1=1 crypto rx=0x7fe91c0b4430 tx=0x7fe90402a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dc268c535cfca2e server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:25:46.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.999+0000 7fe91a000640 1 -- 172.21.15.96:0/310332286 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe904037070 con 0x7fe91c0ca870 2024-05-28T06:25:46.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.999+0000 7fe91a000640 1 -- 172.21.15.96:0/310332286 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe904032440 con 0x7fe91c0ca870 2024-05-28T06:25:46.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:46.999+0000 7fe91a000640 1 -- 172.21.15.96:0/310332286 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe904032810 con 0x7fe91c0ca870 2024-05-28T06:25:46.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:47.000+0000 7fe91be00640 1 -- 172.21.15.96:0/310332286 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe91c0ca870 msgr2=0x7fe91c0ccc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:46.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:47.000+0000 7fe91be00640 1 --2- 172.21.15.96:0/310332286 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe91c0ca870 0x7fe91c0ccc00 secure :-1 s=READY pgs=2107 cs=0 l=1 rev1=1 crypto rx=0x7fe91c0b4430 tx=0x7fe90402a230 comp rx=0 tx=0).stop 2024-05-28T06:25:46.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:47.000+0000 7fe91be00640 1 -- 172.21.15.96:0/310332286 shutdown_connections 2024-05-28T06:25:46.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:47.000+0000 7fe91be00640 1 --2- 172.21.15.96:0/310332286 >> v2:172.21.15.123:3300/0 conn(0x7fe91c0af720 0x7fe91c0cd140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:46.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:47.000+0000 7fe91be00640 1 --2- 172.21.15.96:0/310332286 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe91c0ca870 0x7fe91c0ccc00 unknown :-1 s=CLOSED pgs=2107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:46.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:47.000+0000 7fe91be00640 1 -- 172.21.15.96:0/310332286 >> 172.21.15.96:0/310332286 conn(0x7fe91c06bae0 msgr2=0x7fe91c06bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:25:46.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:47.000+0000 7fe91be00640 1 -- 172.21.15.96:0/310332286 shutdown_connections 2024-05-28T06:25:46.993 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:47.000+0000 7fe91be00640 1 -- 172.21.15.96:0/310332286 wait complete. 2024-05-28T06:25:46.994 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:47.002+0000 7fe91be00640 1 Processor -- start 2024-05-28T06:25:46.995 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:47.002+0000 7fe91be00640 1 -- start start 2024-05-28T06:25:46.995 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:47.003+0000 7fe91be00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe91c0af720 0x7fe91c152240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:46.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:47.003+0000 7fe91be00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe91c0ca870 0x7fe91c0c52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:46.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:47.003+0000 7fe91be00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe91c15ac80 0x7fe91c0c5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:46.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:47.003+0000 7fe91be00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe91c0d0460 con 0x7fe91c0af720 2024-05-28T06:25:46.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:47.003+0000 7fe91be00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe91c0cfe20 con 0x7fe91c15ac80 2024-05-28T06:25:46.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:47.003+0000 7fe91be00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fe91c0d05e0 con 0x7fe91c0ca870 2024-05-28T06:25:46.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:47.003+0000 7fe91aa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe91c0af720 0x7fe91c152240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:46.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:47.003+0000 7fe91aa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe91c0af720 0x7fe91c152240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:56374/0 (socket says 172.21.15.96:56374) 2024-05-28T06:25:46.996 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:47.003+0000 7fe91aa00640 1 -- 172.21.15.96:0/2517977386 learned_addr learned my addr 172.21.15.96:0/2517977386 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:25:46.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:47.003+0000 7fe91aa00640 1 -- 172.21.15.96:0/2517977386 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe91c0ca870 msgr2=0x7fe91c0c52d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:46.998 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:47.003+0000 7fe91aa00640 1 --2- 172.21.15.96:0/2517977386 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe91c0ca870 0x7fe91c0c52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:46.998 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:47.003+0000 7fe91aa00640 1 -- 172.21.15.96:0/2517977386 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe91c15ac80 msgr2=0x7fe91c0c5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:46.998 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:47.003+0000 7fe91aa00640 1 --2- 172.21.15.96:0/2517977386 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe91c15ac80 0x7fe91c0c5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:46.998 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:47.003+0000 7fe91aa00640 1 -- 172.21.15.96:0/2517977386 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe91c0c5e40 con 0x7fe91c0af720 2024-05-28T06:25:46.998 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:47.004+0000 7fe91aa00640 1 --2- 172.21.15.96:0/2517977386 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe91c0af720 0x7fe91c152240 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 crypto rx=0x7fe90c00b3e0 tx=0x7fe90c00b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:25:46.998 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:47.004+0000 7fe913400640 1 -- 172.21.15.96:0/2517977386 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe90c015020 con 0x7fe91c0af720 2024-05-28T06:25:46.998 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:47.004+0000 7fe913400640 1 -- 172.21.15.96:0/2517977386 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe90c011850 con 0x7fe91c0af720 2024-05-28T06:25:46.998 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:47.004+0000 7fe913400640 1 -- 172.21.15.96:0/2517977386 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe90c011c40 con 0x7fe91c0af720 2024-05-28T06:25:46.998 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:47.004+0000 7fe91be00640 1 -- 172.21.15.96:0/2517977386 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe91c0c6120 con 0x7fe91c0af720 2024-05-28T06:25:46.999 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:47.004+0000 7fe91be00640 1 -- 172.21.15.96:0/2517977386 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe91c0c66e0 con 0x7fe91c0af720 2024-05-28T06:25:46.999 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:47.005+0000 7fe91be00640 1 -- 172.21.15.96:0/2517977386 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe91c0689e0 con 0x7fe91c0af720 2024-05-28T06:25:46.999 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:47.006+0000 7fe913400640 1 -- 172.21.15.96:0/2517977386 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 47) ==== 93998+0+0 (secure 0 0 0) 0x7fe90c018020 con 0x7fe91c0af720 2024-05-28T06:25:47.000 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:47.007+0000 7fe913400640 1 --2- 172.21.15.96:0/2517977386 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe8f806e990 0x7fe8f8070e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:47.000 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:47.007+0000 7fe913400640 1 -- 172.21.15.96:0/2517977386 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 5823+0+0 (secure 0 0 0) 0x7fe90c08ac50 con 0x7fe91c0af720 2024-05-28T06:25:47.000 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:47.008+0000 7fe912000640 1 --2- 172.21.15.96:0/2517977386 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe8f806e990 0x7fe8f8070e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:47.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:47.009+0000 7fe912000640 1 --2- 172.21.15.96:0/2517977386 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe8f806e990 0x7fe8f8070e50 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fe904034000 tx=0x7fe90402c000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:25:47.004 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:47.012+0000 7fe913400640 1 -- 172.21.15.96:0/2517977386 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe90c01f020 con 0x7fe91c0af720 2024-05-28T06:25:47.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:47.286+0000 7fe91be00640 1 -- 172.21.15.96:0/2517977386 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fe91c06ea50 con 0x7fe91c0af720 2024-05-28T06:25:47.279 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:47.287+0000 7fe913400640 1 -- 172.21.15.96:0/2517977386 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+2381 (secure 0 0 0) 0x7fe90c0172c0 con 0x7fe91c0af720 2024-05-28T06:25:47.280 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:25:47.280 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:25:47.280 INFO:teuthology.orchestra.run.smithi096.stdout: "fsid": "e4a831b5-e6d1-47a7-96cb-bf85154ee0d2", 2024-05-28T06:25:47.280 INFO:teuthology.orchestra.run.smithi096.stdout: "health": { 2024-05-28T06:25:47.280 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "HEALTH_OK", 2024-05-28T06:25:47.280 INFO:teuthology.orchestra.run.smithi096.stdout: "checks": {}, 2024-05-28T06:25:47.280 INFO:teuthology.orchestra.run.smithi096.stdout: "mutes": [] 2024-05-28T06:25:47.280 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:25:47.280 INFO:teuthology.orchestra.run.smithi096.stdout: "election_epoch": 8, 2024-05-28T06:25:47.281 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum": [ 2024-05-28T06:25:47.281 INFO:teuthology.orchestra.run.smithi096.stdout: 0, 2024-05-28T06:25:47.281 INFO:teuthology.orchestra.run.smithi096.stdout: 1, 2024-05-28T06:25:47.281 INFO:teuthology.orchestra.run.smithi096.stdout: 2 2024-05-28T06:25:47.281 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:25:47.281 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_names": [ 2024-05-28T06:25:47.281 INFO:teuthology.orchestra.run.smithi096.stdout: "a", 2024-05-28T06:25:47.281 INFO:teuthology.orchestra.run.smithi096.stdout: "b", 2024-05-28T06:25:47.281 INFO:teuthology.orchestra.run.smithi096.stdout: "c" 2024-05-28T06:25:47.281 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:25:47.281 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_age": 1320, 2024-05-28T06:25:47.281 INFO:teuthology.orchestra.run.smithi096.stdout: "monmap": { 2024-05-28T06:25:47.281 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 1, 2024-05-28T06:25:47.282 INFO:teuthology.orchestra.run.smithi096.stdout: "min_mon_release_name": "squid", 2024-05-28T06:25:47.282 INFO:teuthology.orchestra.run.smithi096.stdout: "num_mons": 3 2024-05-28T06:25:47.282 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:25:47.282 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap": { 2024-05-28T06:25:47.282 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 132, 2024-05-28T06:25:47.282 INFO:teuthology.orchestra.run.smithi096.stdout: "num_osds": 8, 2024-05-28T06:25:47.282 INFO:teuthology.orchestra.run.smithi096.stdout: "num_up_osds": 8, 2024-05-28T06:25:47.282 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_up_since": 1716876238, 2024-05-28T06:25:47.282 INFO:teuthology.orchestra.run.smithi096.stdout: "num_in_osds": 8, 2024-05-28T06:25:47.282 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_in_since": 1716876229, 2024-05-28T06:25:47.282 INFO:teuthology.orchestra.run.smithi096.stdout: "num_remapped_pgs": 0 2024-05-28T06:25:47.282 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:25:47.282 INFO:teuthology.orchestra.run.smithi096.stdout: "pgmap": { 2024-05-28T06:25:47.282 INFO:teuthology.orchestra.run.smithi096.stdout: "pgs_by_state": [ 2024-05-28T06:25:47.283 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:25:47.283 INFO:teuthology.orchestra.run.smithi096.stdout: "state_name": "active+clean", 2024-05-28T06:25:47.283 INFO:teuthology.orchestra.run.smithi096.stdout: "count": 97 2024-05-28T06:25:47.283 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:25:47.283 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:25:47.283 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pgs": 97, 2024-05-28T06:25:47.283 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pools": 3, 2024-05-28T06:25:47.283 INFO:teuthology.orchestra.run.smithi096.stdout: "num_objects": 4618, 2024-05-28T06:25:47.283 INFO:teuthology.orchestra.run.smithi096.stdout: "data_bytes": 19216285008, 2024-05-28T06:25:47.283 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_used": 39541395456, 2024-05-28T06:25:47.283 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_avail": 733552717824, 2024-05-28T06:25:47.283 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_total": 773094113280, 2024-05-28T06:25:47.283 INFO:teuthology.orchestra.run.smithi096.stdout: "read_bytes_sec": 777387543, 2024-05-28T06:25:47.284 INFO:teuthology.orchestra.run.smithi096.stdout: "write_bytes_sec": 783045954, 2024-05-28T06:25:47.284 INFO:teuthology.orchestra.run.smithi096.stdout: "read_op_per_sec": 189, 2024-05-28T06:25:47.284 INFO:teuthology.orchestra.run.smithi096.stdout: "write_op_per_sec": 208 2024-05-28T06:25:47.284 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:25:47.284 INFO:teuthology.orchestra.run.smithi096.stdout: "fsmap": { 2024-05-28T06:25:47.284 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 83, 2024-05-28T06:25:47.284 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 9, 2024-05-28T06:25:47.284 INFO:teuthology.orchestra.run.smithi096.stdout: "up": 1, 2024-05-28T06:25:47.284 INFO:teuthology.orchestra.run.smithi096.stdout: "in": 1, 2024-05-28T06:25:47.284 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 1, 2024-05-28T06:25:47.284 INFO:teuthology.orchestra.run.smithi096.stdout: "by_rank": [ 2024-05-28T06:25:47.284 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:25:47.284 INFO:teuthology.orchestra.run.smithi096.stdout: "filesystem_id": 9, 2024-05-28T06:25:47.285 INFO:teuthology.orchestra.run.smithi096.stdout: "rank": 0, 2024-05-28T06:25:47.285 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "a", 2024-05-28T06:25:47.285 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "up:active", 2024-05-28T06:25:47.285 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 9849 2024-05-28T06:25:47.285 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:25:47.285 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:25:47.285 INFO:teuthology.orchestra.run.smithi096.stdout: "up:standby": 3 2024-05-28T06:25:47.285 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:25:47.285 INFO:teuthology.orchestra.run.smithi096.stdout: "mgrmap": { 2024-05-28T06:25:47.285 INFO:teuthology.orchestra.run.smithi096.stdout: "available": true, 2024-05-28T06:25:47.285 INFO:teuthology.orchestra.run.smithi096.stdout: "num_standbys": 1, 2024-05-28T06:25:47.285 INFO:teuthology.orchestra.run.smithi096.stdout: "modules": [ 2024-05-28T06:25:47.285 INFO:teuthology.orchestra.run.smithi096.stdout: "iostat", 2024-05-28T06:25:47.285 INFO:teuthology.orchestra.run.smithi096.stdout: "nfs", 2024-05-28T06:25:47.286 INFO:teuthology.orchestra.run.smithi096.stdout: "restful" 2024-05-28T06:25:47.286 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:25:47.286 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:25:47.286 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:25:47.286 INFO:teuthology.orchestra.run.smithi096.stdout: "servicemap": { 2024-05-28T06:25:47.286 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 25, 2024-05-28T06:25:47.286 INFO:teuthology.orchestra.run.smithi096.stdout: "modified": "2024-05-28T06:25:30.049023+0000", 2024-05-28T06:25:47.286 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:25:47.286 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:25:47.286 INFO:teuthology.orchestra.run.smithi096.stdout: "progress_events": { 2024-05-28T06:25:47.286 INFO:teuthology.orchestra.run.smithi096.stdout: "f2372170-bf4f-4ba1-a7df-9e0c76ce0de6": { 2024-05-28T06:25:47.286 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "No pending clones left, waiting 28.0 seconds before exiting (17s)\n [============================] ", 2024-05-28T06:25:47.286 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": 1, 2024-05-28T06:25:47.287 INFO:teuthology.orchestra.run.smithi096.stdout: "add_to_ceph_s": true 2024-05-28T06:25:47.287 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:25:47.287 INFO:teuthology.orchestra.run.smithi096.stdout: "f7bbcf64-a8ee-4f95-9006-d32bd5fb36cf": { 2024-05-28T06:25:47.287 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "All cloning finished, waiting 28.0 seconds before exiting (24s)\n [============================] ", 2024-05-28T06:25:47.287 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": 1, 2024-05-28T06:25:47.287 INFO:teuthology.orchestra.run.smithi096.stdout: "add_to_ceph_s": true 2024-05-28T06:25:47.287 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:25:47.287 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:25:47.287 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:25:47.288 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:47.289+0000 7fe91be00640 1 -- 172.21.15.96:0/2517977386 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe8f806e990 msgr2=0x7fe8f8070e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:47.288 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:47.290+0000 7fe91be00640 1 --2- 172.21.15.96:0/2517977386 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe8f806e990 0x7fe8f8070e50 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fe904034000 tx=0x7fe90402c000 comp rx=0 tx=0).stop 2024-05-28T06:25:47.288 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:47.290+0000 7fe91be00640 1 -- 172.21.15.96:0/2517977386 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe91c0af720 msgr2=0x7fe91c152240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:47.288 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:47.290+0000 7fe91be00640 1 --2- 172.21.15.96:0/2517977386 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe91c0af720 0x7fe91c152240 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 crypto rx=0x7fe90c00b3e0 tx=0x7fe90c00b8a0 comp rx=0 tx=0).stop 2024-05-28T06:25:47.288 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:47.290+0000 7fe91be00640 1 -- 172.21.15.96:0/2517977386 shutdown_connections 2024-05-28T06:25:47.288 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:47.290+0000 7fe91be00640 1 --2- 172.21.15.96:0/2517977386 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe8f806e990 0x7fe8f8070e50 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:47.288 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:47.290+0000 7fe91be00640 1 --2- 172.21.15.96:0/2517977386 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe91c15ac80 0x7fe91c0c5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:47.289 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:47.290+0000 7fe91be00640 1 --2- 172.21.15.96:0/2517977386 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe91c0ca870 0x7fe91c0c52d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:47.289 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:47.290+0000 7fe91be00640 1 --2- 172.21.15.96:0/2517977386 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe91c0af720 0x7fe91c152240 unknown :-1 s=CLOSED pgs=2108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:47.289 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:47.290+0000 7fe91be00640 1 -- 172.21.15.96:0/2517977386 >> 172.21.15.96:0/2517977386 conn(0x7fe91c06bae0 msgr2=0x7fe91c153ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:25:47.289 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:47.290+0000 7fe91be00640 1 -- 172.21.15.96:0/2517977386 shutdown_connections 2024-05-28T06:25:47.289 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:47.290+0000 7fe91be00640 1 -- 172.21.15.96:0/2517977386 wait complete. 2024-05-28T06:25:52.303 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-05-28T06:25:52.509 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:52.515+0000 7f0a36e00640 1 Processor -- start 2024-05-28T06:25:52.510 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:52.516+0000 7f0a36e00640 1 -- start start 2024-05-28T06:25:52.511 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:52.517+0000 7f0a36e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f0a3006faa0 0x7f0a300d0970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:52.511 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:52.517+0000 7f0a36e00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a3006e3d0 con 0x7f0a30155a00 2024-05-28T06:25:52.511 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:52.517+0000 7f0a36e00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a3015c0b0 con 0x7f0a3015baa0 2024-05-28T06:25:52.511 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:52.517+0000 7f0a36e00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f0a300ccf20 con 0x7f0a3006faa0 2024-05-28T06:25:52.512 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:52.518+0000 7f0a2fe00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f0a30155a00 0x7f0a3006dcc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:34790/0 (socket says 172.21.15.96:34790) 2024-05-28T06:25:52.512 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:52.518+0000 7f0a2fe00640 1 -- 172.21.15.96:0/3598548930 learned_addr learned my addr 172.21.15.96:0/3598548930 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:25:52.512 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:52.518+0000 7f0a34e00640 1 --2- 172.21.15.96:0/3598548930 >> v2:172.21.15.96:3300/0 conn(0x7f0a3006faa0 0x7f0a300d0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:52.512 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:52.518+0000 7f0a34e00640 1 -- 172.21.15.96:0/3598548930 >> v1:172.21.15.123:6789/0 conn(0x7f0a3015baa0 legacy=0x7f0a3006f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:25:52.512 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:52.518+0000 7f0a2f400640 1 -- 172.21.15.96:0/3598548930 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4033817259 0 0) 0x7f0a3006e3d0 con 0x7f0a30155a00 2024-05-28T06:25:52.512 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:52.518+0000 7f0a34e00640 1 -- 172.21.15.96:0/3598548930 >> v1:172.21.15.96:6789/0 conn(0x7f0a30155a00 legacy=0x7f0a3006dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:52.512 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:52.518+0000 7f0a34e00640 1 -- 172.21.15.96:0/3598548930 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a3015c390 con 0x7f0a3006faa0 2024-05-28T06:25:52.513 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:52.519+0000 7f0a34e00640 1 --2- 172.21.15.96:0/3598548930 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a3006faa0 0x7f0a300d0970 secure :-1 s=READY pgs=2110 cs=0 l=1 rev1=1 crypto rx=0x7f0a28009d90 tx=0x7f0a2802a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a89795a84d6c088 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:25:52.513 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:52.519+0000 7f0a2f400640 1 -- 172.21.15.96:0/3598548930 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a28007740 con 0x7f0a3006faa0 2024-05-28T06:25:52.513 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:52.519+0000 7f0a2f400640 1 -- 172.21.15.96:0/3598548930 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a280078e0 con 0x7f0a3006faa0 2024-05-28T06:25:52.513 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:52.520+0000 7f0a2f400640 1 -- 172.21.15.96:0/3598548930 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a28007cb0 con 0x7f0a3006faa0 2024-05-28T06:25:52.514 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:52.520+0000 7f0a36e00640 1 -- 172.21.15.96:0/3598548930 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a3006faa0 msgr2=0x7f0a300d0970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:52.514 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:52.520+0000 7f0a36e00640 1 --2- 172.21.15.96:0/3598548930 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a3006faa0 0x7f0a300d0970 secure :-1 s=READY pgs=2110 cs=0 l=1 rev1=1 crypto rx=0x7f0a28009d90 tx=0x7f0a2802a660 comp rx=0 tx=0).stop 2024-05-28T06:25:52.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:52.520+0000 7f0a36e00640 1 -- 172.21.15.96:0/3598548930 shutdown_connections 2024-05-28T06:25:52.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:52.520+0000 7f0a36e00640 1 --2- 172.21.15.96:0/3598548930 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a3006faa0 0x7f0a300d0970 unknown :-1 s=CLOSED pgs=2110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:52.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:52.520+0000 7f0a36e00640 1 -- 172.21.15.96:0/3598548930 >> 172.21.15.96:0/3598548930 conn(0x7f0a30067ce0 msgr2=0x7f0a3006b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:25:52.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:52.520+0000 7f0a36e00640 1 -- 172.21.15.96:0/3598548930 shutdown_connections 2024-05-28T06:25:52.515 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:52.520+0000 7f0a36e00640 1 -- 172.21.15.96:0/3598548930 wait complete. 2024-05-28T06:25:52.516 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:52.522+0000 7f0a36e00640 1 Processor -- start 2024-05-28T06:25:52.516 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:52.522+0000 7f0a36e00640 1 -- start start 2024-05-28T06:25:52.518 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:52.523+0000 7f0a36e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a3006faa0 0x7f0a300aec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:52.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:52.523+0000 7f0a36e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0a30155a00 0x7f0a301f3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:52.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:52.523+0000 7f0a36e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0a3015baa0 0x7f0a301e52f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:52.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:52.523+0000 7f0a36e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f0a3015c950 con 0x7f0a3006faa0 2024-05-28T06:25:52.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:52.523+0000 7f0a36e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f0a3015f090 con 0x7f0a3015baa0 2024-05-28T06:25:52.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:52.523+0000 7f0a36e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f0a30155de0 con 0x7f0a30155a00 2024-05-28T06:25:52.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:52.523+0000 7f0a2fe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a3006faa0 0x7f0a300aec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:52.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:52.523+0000 7f0a2fe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a3006faa0 0x7f0a300aec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:55926/0 (socket says 172.21.15.96:55926) 2024-05-28T06:25:52.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:52.523+0000 7f0a2fe00640 1 -- 172.21.15.96:0/1283243719 learned_addr learned my addr 172.21.15.96:0/1283243719 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:25:52.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:52.524+0000 7f0a34e00640 1 --2- 172.21.15.96:0/1283243719 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0a3015baa0 0x7f0a301e52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:52.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:52.524+0000 7f0a27400640 1 --2- 172.21.15.96:0/1283243719 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0a30155a00 0x7f0a301f3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:52.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:52.524+0000 7f0a2fe00640 1 -- 172.21.15.96:0/1283243719 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0a30155a00 msgr2=0x7f0a301f3060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:52.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:52.524+0000 7f0a2fe00640 1 --2- 172.21.15.96:0/1283243719 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0a30155a00 0x7f0a301f3060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:52.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:52.524+0000 7f0a2fe00640 1 -- 172.21.15.96:0/1283243719 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0a3015baa0 msgr2=0x7f0a301e52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:52.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:52.524+0000 7f0a2fe00640 1 --2- 172.21.15.96:0/1283243719 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0a3015baa0 0x7f0a301e52f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:52.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:52.524+0000 7f0a2fe00640 1 -- 172.21.15.96:0/1283243719 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a300af150 con 0x7f0a3006faa0 2024-05-28T06:25:52.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:52.524+0000 7f0a34e00640 1 --2- 172.21.15.96:0/1283243719 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0a3015baa0 0x7f0a301e52f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:25:52.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:52.524+0000 7f0a2fe00640 1 --2- 172.21.15.96:0/1283243719 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a3006faa0 0x7f0a300aec10 secure :-1 s=READY pgs=2111 cs=0 l=1 rev1=1 crypto rx=0x7f0a2000c4a0 tx=0x7f0a2000c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:25:52.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:52.524+0000 7f0a2cc00640 1 -- 172.21.15.96:0/1283243719 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a20018070 con 0x7f0a3006faa0 2024-05-28T06:25:52.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:52.524+0000 7f0a2cc00640 1 -- 172.21.15.96:0/1283243719 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a200074e0 con 0x7f0a3006faa0 2024-05-28T06:25:52.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:52.524+0000 7f0a36e00640 1 -- 172.21.15.96:0/1283243719 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a301e5d40 con 0x7f0a3006faa0 2024-05-28T06:25:52.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:52.524+0000 7f0a2cc00640 1 -- 172.21.15.96:0/1283243719 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a20004a20 con 0x7f0a3006faa0 2024-05-28T06:25:52.520 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:52.524+0000 7f0a36e00640 1 -- 172.21.15.96:0/1283243719 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a301e6300 con 0x7f0a3006faa0 2024-05-28T06:25:52.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:52.526+0000 7f0a36e00640 1 -- 172.21.15.96:0/1283243719 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a300d0450 con 0x7f0a3006faa0 2024-05-28T06:25:52.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:52.527+0000 7f0a2cc00640 1 -- 172.21.15.96:0/1283243719 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 47) ==== 93998+0+0 (secure 0 0 0) 0x7f0a2001b070 con 0x7f0a3006faa0 2024-05-28T06:25:52.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:52.528+0000 7f0a2cc00640 1 --2- 172.21.15.96:0/1283243719 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0a0c06e990 0x7f0a0c070e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:52.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:52.528+0000 7f0a2cc00640 1 -- 172.21.15.96:0/1283243719 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 5823+0+0 (secure 0 0 0) 0x7f0a20089f10 con 0x7f0a3006faa0 2024-05-28T06:25:52.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:52.528+0000 7f0a27400640 1 --2- 172.21.15.96:0/1283243719 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0a0c06e990 0x7f0a0c070e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:52.523 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:52.530+0000 7f0a27400640 1 --2- 172.21.15.96:0/1283243719 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0a0c06e990 0x7f0a0c070e50 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f0a3015b0d0 tx=0x7f0a18009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:25:52.526 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:52.532+0000 7f0a2cc00640 1 -- 172.21.15.96:0/1283243719 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0a20059e40 con 0x7f0a3006faa0 2024-05-28T06:25:52.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:52.810+0000 7f0a36e00640 1 -- 172.21.15.96:0/1283243719 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f0a3006f170 con 0x7f0a3006faa0 2024-05-28T06:25:52.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:52.810+0000 7f0a2cc00640 1 -- 172.21.15.96:0/1283243719 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+2379 (secure 0 0 0) 0x7f0a20059740 con 0x7f0a3006faa0 2024-05-28T06:25:52.805 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:25:52.805 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:25:52.805 INFO:teuthology.orchestra.run.smithi096.stdout: "fsid": "e4a831b5-e6d1-47a7-96cb-bf85154ee0d2", 2024-05-28T06:25:52.805 INFO:teuthology.orchestra.run.smithi096.stdout: "health": { 2024-05-28T06:25:52.805 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "HEALTH_OK", 2024-05-28T06:25:52.805 INFO:teuthology.orchestra.run.smithi096.stdout: "checks": {}, 2024-05-28T06:25:52.805 INFO:teuthology.orchestra.run.smithi096.stdout: "mutes": [] 2024-05-28T06:25:52.805 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:25:52.805 INFO:teuthology.orchestra.run.smithi096.stdout: "election_epoch": 8, 2024-05-28T06:25:52.805 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum": [ 2024-05-28T06:25:52.806 INFO:teuthology.orchestra.run.smithi096.stdout: 0, 2024-05-28T06:25:52.806 INFO:teuthology.orchestra.run.smithi096.stdout: 1, 2024-05-28T06:25:52.806 INFO:teuthology.orchestra.run.smithi096.stdout: 2 2024-05-28T06:25:52.806 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:25:52.806 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_names": [ 2024-05-28T06:25:52.806 INFO:teuthology.orchestra.run.smithi096.stdout: "a", 2024-05-28T06:25:52.806 INFO:teuthology.orchestra.run.smithi096.stdout: "b", 2024-05-28T06:25:52.806 INFO:teuthology.orchestra.run.smithi096.stdout: "c" 2024-05-28T06:25:52.806 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:25:52.806 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_age": 1326, 2024-05-28T06:25:52.806 INFO:teuthology.orchestra.run.smithi096.stdout: "monmap": { 2024-05-28T06:25:52.806 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 1, 2024-05-28T06:25:52.806 INFO:teuthology.orchestra.run.smithi096.stdout: "min_mon_release_name": "squid", 2024-05-28T06:25:52.807 INFO:teuthology.orchestra.run.smithi096.stdout: "num_mons": 3 2024-05-28T06:25:52.807 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:25:52.807 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap": { 2024-05-28T06:25:52.807 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 132, 2024-05-28T06:25:52.807 INFO:teuthology.orchestra.run.smithi096.stdout: "num_osds": 8, 2024-05-28T06:25:52.807 INFO:teuthology.orchestra.run.smithi096.stdout: "num_up_osds": 8, 2024-05-28T06:25:52.807 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_up_since": 1716876238, 2024-05-28T06:25:52.807 INFO:teuthology.orchestra.run.smithi096.stdout: "num_in_osds": 8, 2024-05-28T06:25:52.807 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_in_since": 1716876229, 2024-05-28T06:25:52.807 INFO:teuthology.orchestra.run.smithi096.stdout: "num_remapped_pgs": 0 2024-05-28T06:25:52.807 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:25:52.807 INFO:teuthology.orchestra.run.smithi096.stdout: "pgmap": { 2024-05-28T06:25:52.807 INFO:teuthology.orchestra.run.smithi096.stdout: "pgs_by_state": [ 2024-05-28T06:25:52.808 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:25:52.808 INFO:teuthology.orchestra.run.smithi096.stdout: "state_name": "active+clean", 2024-05-28T06:25:52.808 INFO:teuthology.orchestra.run.smithi096.stdout: "count": 97 2024-05-28T06:25:52.808 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:25:52.808 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:25:52.808 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pgs": 97, 2024-05-28T06:25:52.808 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pools": 3, 2024-05-28T06:25:52.808 INFO:teuthology.orchestra.run.smithi096.stdout: "num_objects": 4663, 2024-05-28T06:25:52.808 INFO:teuthology.orchestra.run.smithi096.stdout: "data_bytes": 19413464020, 2024-05-28T06:25:52.808 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_used": 39973691392, 2024-05-28T06:25:52.808 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_avail": 733120421888, 2024-05-28T06:25:52.808 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_total": 773094113280, 2024-05-28T06:25:52.808 INFO:teuthology.orchestra.run.smithi096.stdout: "read_bytes_sec": 311301822, 2024-05-28T06:25:52.809 INFO:teuthology.orchestra.run.smithi096.stdout: "write_bytes_sec": 317649032, 2024-05-28T06:25:52.809 INFO:teuthology.orchestra.run.smithi096.stdout: "read_op_per_sec": 77, 2024-05-28T06:25:52.809 INFO:teuthology.orchestra.run.smithi096.stdout: "write_op_per_sec": 93 2024-05-28T06:25:52.809 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:25:52.809 INFO:teuthology.orchestra.run.smithi096.stdout: "fsmap": { 2024-05-28T06:25:52.809 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 83, 2024-05-28T06:25:52.809 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 9, 2024-05-28T06:25:52.809 INFO:teuthology.orchestra.run.smithi096.stdout: "up": 1, 2024-05-28T06:25:52.809 INFO:teuthology.orchestra.run.smithi096.stdout: "in": 1, 2024-05-28T06:25:52.809 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 1, 2024-05-28T06:25:52.809 INFO:teuthology.orchestra.run.smithi096.stdout: "by_rank": [ 2024-05-28T06:25:52.809 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:25:52.809 INFO:teuthology.orchestra.run.smithi096.stdout: "filesystem_id": 9, 2024-05-28T06:25:52.809 INFO:teuthology.orchestra.run.smithi096.stdout: "rank": 0, 2024-05-28T06:25:52.810 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "a", 2024-05-28T06:25:52.810 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "up:active", 2024-05-28T06:25:52.810 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 9849 2024-05-28T06:25:52.810 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:25:52.810 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:25:52.810 INFO:teuthology.orchestra.run.smithi096.stdout: "up:standby": 3 2024-05-28T06:25:52.810 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:25:52.810 INFO:teuthology.orchestra.run.smithi096.stdout: "mgrmap": { 2024-05-28T06:25:52.810 INFO:teuthology.orchestra.run.smithi096.stdout: "available": true, 2024-05-28T06:25:52.810 INFO:teuthology.orchestra.run.smithi096.stdout: "num_standbys": 1, 2024-05-28T06:25:52.811 INFO:teuthology.orchestra.run.smithi096.stdout: "modules": [ 2024-05-28T06:25:52.811 INFO:teuthology.orchestra.run.smithi096.stdout: "iostat", 2024-05-28T06:25:52.811 INFO:teuthology.orchestra.run.smithi096.stdout: "nfs", 2024-05-28T06:25:52.811 INFO:teuthology.orchestra.run.smithi096.stdout: "restful" 2024-05-28T06:25:52.811 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:25:52.811 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:25:52.812 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:25:52.812 INFO:teuthology.orchestra.run.smithi096.stdout: "servicemap": { 2024-05-28T06:25:52.812 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 25, 2024-05-28T06:25:52.812 INFO:teuthology.orchestra.run.smithi096.stdout: "modified": "2024-05-28T06:25:30.049023+0000", 2024-05-28T06:25:52.812 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:25:52.812 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:25:52.812 INFO:teuthology.orchestra.run.smithi096.stdout: "progress_events": { 2024-05-28T06:25:52.812 INFO:teuthology.orchestra.run.smithi096.stdout: "f2372170-bf4f-4ba1-a7df-9e0c76ce0de6": { 2024-05-28T06:25:52.812 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "No pending clones left, waiting 22.0 seconds before exiting (23s)\n [============================] ", 2024-05-28T06:25:52.812 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": 1, 2024-05-28T06:25:52.812 INFO:teuthology.orchestra.run.smithi096.stdout: "add_to_ceph_s": true 2024-05-28T06:25:52.812 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:25:52.812 INFO:teuthology.orchestra.run.smithi096.stdout: "f7bbcf64-a8ee-4f95-9006-d32bd5fb36cf": { 2024-05-28T06:25:52.812 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "All cloning finished, waiting 22.0 seconds before exiting (30s)\n [============================] ", 2024-05-28T06:25:52.813 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": 1, 2024-05-28T06:25:52.813 INFO:teuthology.orchestra.run.smithi096.stdout: "add_to_ceph_s": true 2024-05-28T06:25:52.813 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:25:52.813 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:25:52.813 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:25:52.814 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:52.813+0000 7f0a36e00640 1 -- 172.21.15.96:0/1283243719 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0a0c06e990 msgr2=0x7f0a0c070e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:52.814 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:52.813+0000 7f0a36e00640 1 --2- 172.21.15.96:0/1283243719 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0a0c06e990 0x7f0a0c070e50 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f0a3015b0d0 tx=0x7f0a18009040 comp rx=0 tx=0).stop 2024-05-28T06:25:52.814 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:52.813+0000 7f0a36e00640 1 -- 172.21.15.96:0/1283243719 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a3006faa0 msgr2=0x7f0a300aec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:52.814 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:52.813+0000 7f0a36e00640 1 --2- 172.21.15.96:0/1283243719 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a3006faa0 0x7f0a300aec10 secure :-1 s=READY pgs=2111 cs=0 l=1 rev1=1 crypto rx=0x7f0a2000c4a0 tx=0x7f0a2000c960 comp rx=0 tx=0).stop 2024-05-28T06:25:52.814 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:52.813+0000 7f0a36e00640 1 -- 172.21.15.96:0/1283243719 shutdown_connections 2024-05-28T06:25:52.814 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:52.813+0000 7f0a36e00640 1 --2- 172.21.15.96:0/1283243719 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f0a0c06e990 0x7f0a0c070e50 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:52.814 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:52.813+0000 7f0a36e00640 1 --2- 172.21.15.96:0/1283243719 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0a3015baa0 0x7f0a301e52f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:52.814 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:52.813+0000 7f0a36e00640 1 --2- 172.21.15.96:0/1283243719 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f0a30155a00 0x7f0a301f3060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:52.814 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:52.813+0000 7f0a36e00640 1 --2- 172.21.15.96:0/1283243719 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a3006faa0 0x7f0a300aec10 unknown :-1 s=CLOSED pgs=2111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:52.814 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:52.813+0000 7f0a36e00640 1 -- 172.21.15.96:0/1283243719 >> 172.21.15.96:0/1283243719 conn(0x7f0a30067ce0 msgr2=0x7f0a300cef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:25:52.814 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:52.813+0000 7f0a36e00640 1 -- 172.21.15.96:0/1283243719 shutdown_connections 2024-05-28T06:25:52.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:52.813+0000 7f0a36e00640 1 -- 172.21.15.96:0/1283243719 wait complete. 2024-05-28T06:25:57.829 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-05-28T06:25:58.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:58.040+0000 7f5edfe00640 1 Processor -- start 2024-05-28T06:25:58.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:58.040+0000 7f5edfe00640 1 -- start start 2024-05-28T06:25:58.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:58.042+0000 7f5edfe00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f5ee015e690 0x7f5ee00d4530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:58.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:58.042+0000 7f5edfe00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5ee015ea70 0x7f5ee00d2a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:58.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:58.042+0000 7f5edfe00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ee00d3060 con 0x7f5ee00d1ca0 2024-05-28T06:25:58.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:58.042+0000 7f5edfe00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f5ee00d3340 con 0x7f5ee015e690 2024-05-28T06:25:58.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:58.042+0000 7f5edfe00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f5ee00d3620 con 0x7f5ee015ea70 2024-05-28T06:25:58.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:58.042+0000 7f5edea00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f5ee015e690 0x7f5ee00d4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:58.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:58.042+0000 7f5edea00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f5ee015e690 0x7f5ee00d4530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:55940/0 (socket says 172.21.15.96:55940) 2024-05-28T06:25:58.036 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:58.042+0000 7f5edea00640 1 -- 172.21.15.96:0/2830283935 learned_addr learned my addr 172.21.15.96:0/2830283935 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:25:58.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:58.043+0000 7f5edf400640 1 --2- 172.21.15.96:0/2830283935 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5ee015ea70 0x7f5ee00d2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:58.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:58.043+0000 7f5edea00640 1 -- 172.21.15.96:0/2830283935 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5ee015ea70 msgr2=0x7f5ee00d2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:25:58.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:58.043+0000 7f5edea00640 1 --2- 172.21.15.96:0/2830283935 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5ee015ea70 0x7f5ee00d2a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:58.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:58.043+0000 7f5edea00640 1 -- 172.21.15.96:0/2830283935 >> v1:172.21.15.123:6789/0 conn(0x7f5ee00d1ca0 legacy=0x7f5ee00d2330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:25:58.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:58.043+0000 7f5edea00640 1 -- 172.21.15.96:0/2830283935 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ee00cf770 con 0x7f5ee015e690 2024-05-28T06:25:58.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:58.043+0000 7f5edf400640 1 --2- 172.21.15.96:0/2830283935 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5ee015ea70 0x7f5ee00d2a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:25:58.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:58.044+0000 7f5edea00640 1 --2- 172.21.15.96:0/2830283935 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5ee015e690 0x7f5ee00d4530 secure :-1 s=READY pgs=2112 cs=0 l=1 rev1=1 crypto rx=0x7f5ed00098a0 tx=0x7f5ed002a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6108d7b992f3f3e6 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:25:58.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:58.044+0000 7f5ede000640 1 -- 172.21.15.96:0/2830283935 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ed0037070 con 0x7f5ee015e690 2024-05-28T06:25:58.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:58.044+0000 7f5ede000640 1 -- 172.21.15.96:0/2830283935 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ed0032440 con 0x7f5ee015e690 2024-05-28T06:25:58.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:58.044+0000 7f5ede000640 1 -- 172.21.15.96:0/2830283935 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ed0032810 con 0x7f5ee015e690 2024-05-28T06:25:58.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:58.045+0000 7f5edfe00640 1 -- 172.21.15.96:0/2830283935 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5ee015e690 msgr2=0x7f5ee00d4530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:58.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:58.045+0000 7f5edfe00640 1 --2- 172.21.15.96:0/2830283935 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5ee015e690 0x7f5ee00d4530 secure :-1 s=READY pgs=2112 cs=0 l=1 rev1=1 crypto rx=0x7f5ed00098a0 tx=0x7f5ed002a250 comp rx=0 tx=0).stop 2024-05-28T06:25:58.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:58.045+0000 7f5edfe00640 1 -- 172.21.15.96:0/2830283935 shutdown_connections 2024-05-28T06:25:58.039 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:58.045+0000 7f5edfe00640 1 --2- 172.21.15.96:0/2830283935 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5ee015ea70 0x7f5ee00d2a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:58.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:58.045+0000 7f5edfe00640 1 --2- 172.21.15.96:0/2830283935 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5ee015e690 0x7f5ee00d4530 unknown :-1 s=CLOSED pgs=2112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:58.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:58.045+0000 7f5edfe00640 1 -- 172.21.15.96:0/2830283935 >> 172.21.15.96:0/2830283935 conn(0x7f5ee009d910 msgr2=0x7f5ee008bfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:25:58.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:58.045+0000 7f5edfe00640 1 -- 172.21.15.96:0/2830283935 shutdown_connections 2024-05-28T06:25:58.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:58.045+0000 7f5edfe00640 1 -- 172.21.15.96:0/2830283935 wait complete. 2024-05-28T06:25:58.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:58.047+0000 7f5edfe00640 1 Processor -- start 2024-05-28T06:25:58.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:58.047+0000 7f5edfe00640 1 -- start start 2024-05-28T06:25:58.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:58.048+0000 7f5edfe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5ee00d1ca0 0x7f5ee015fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:58.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:58.049+0000 7f5edfe00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5ee015e690 0x7f5ee01f77a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:58.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:58.049+0000 7f5edfe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5ee015ea70 0x7f5ee00cdb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:58.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:58.049+0000 7f5edfe00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f5ee00d4bf0 con 0x7f5ee015ea70 2024-05-28T06:25:58.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:58.049+0000 7f5edfe00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5ee00d4a70 con 0x7f5ee00d1ca0 2024-05-28T06:25:58.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:58.049+0000 7f5edfe00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f5ee00d4d70 con 0x7f5ee015e690 2024-05-28T06:25:58.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:58.049+0000 7f5edf400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5ee015ea70 0x7f5ee00cdb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:58.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:58.049+0000 7f5edf400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5ee015ea70 0x7f5ee00cdb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:55950/0 (socket says 172.21.15.96:55950) 2024-05-28T06:25:58.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:58.049+0000 7f5edf400640 1 -- 172.21.15.96:0/590604504 learned_addr learned my addr 172.21.15.96:0/590604504 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:25:58.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:58.049+0000 7f5edf400640 1 -- 172.21.15.96:0/590604504 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5ee015e690 msgr2=0x7f5ee01f77a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:58.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:58.049+0000 7f5edf400640 1 --2- 172.21.15.96:0/590604504 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5ee015e690 0x7f5ee01f77a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:58.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:58.049+0000 7f5edf400640 1 -- 172.21.15.96:0/590604504 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5ee00d1ca0 msgr2=0x7f5ee015fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:58.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:58.049+0000 7f5edf400640 1 --2- 172.21.15.96:0/590604504 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5ee00d1ca0 0x7f5ee015fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:58.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:58.049+0000 7f5edf400640 1 -- 172.21.15.96:0/590604504 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ee0160dc0 con 0x7f5ee015ea70 2024-05-28T06:25:58.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:58.050+0000 7f5edf400640 1 --2- 172.21.15.96:0/590604504 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5ee015ea70 0x7f5ee00cdb90 secure :-1 s=READY pgs=2113 cs=0 l=1 rev1=1 crypto rx=0x7f5ed800ce80 tx=0x7f5ed800a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:25:58.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:58.050+0000 7f5ed7e00640 1 -- 172.21.15.96:0/590604504 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ed801a070 con 0x7f5ee015ea70 2024-05-28T06:25:58.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:58.050+0000 7f5ed7e00640 1 -- 172.21.15.96:0/590604504 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ed800d880 con 0x7f5ee015ea70 2024-05-28T06:25:58.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:58.050+0000 7f5edfe00640 1 -- 172.21.15.96:0/590604504 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ee0161060 con 0x7f5ee015ea70 2024-05-28T06:25:58.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:58.050+0000 7f5ed7e00640 1 -- 172.21.15.96:0/590604504 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ed800dc70 con 0x7f5ee015ea70 2024-05-28T06:25:58.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:58.050+0000 7f5edfe00640 1 -- 172.21.15.96:0/590604504 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ee0161620 con 0x7f5ee015ea70 2024-05-28T06:25:58.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:58.051+0000 7f5edfe00640 1 -- 172.21.15.96:0/590604504 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ee00689e0 con 0x7f5ee015ea70 2024-05-28T06:25:58.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:58.052+0000 7f5ed7e00640 1 -- 172.21.15.96:0/590604504 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 48) ==== 93947+0+0 (secure 0 0 0) 0x7f5ed801d070 con 0x7f5ee015ea70 2024-05-28T06:25:58.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:58.053+0000 7f5ed7e00640 1 --2- 172.21.15.96:0/590604504 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5ebc06e900 0x7f5ebc070dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:25:58.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:58.053+0000 7f5ed7e00640 1 -- 172.21.15.96:0/590604504 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 5823+0+0 (secure 0 0 0) 0x7f5ed808b360 con 0x7f5ee015ea70 2024-05-28T06:25:58.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:58.054+0000 7f5edea00640 1 --2- 172.21.15.96:0/590604504 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5ebc06e900 0x7f5ebc070dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:25:58.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:58.055+0000 7f5edea00640 1 --2- 172.21.15.96:0/590604504 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5ebc06e900 0x7f5ebc070dc0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f5ed0034000 tx=0x7f5ed002c000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:25:58.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:58.058+0000 7f5ed7e00640 1 -- 172.21.15.96:0/590604504 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5ed805aba0 con 0x7f5ee015ea70 2024-05-28T06:25:58.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:58.327+0000 7f5edfe00640 1 -- 172.21.15.96:0/590604504 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f5ee006b900 con 0x7f5ee015ea70 2024-05-28T06:25:58.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:58.327+0000 7f5ed7e00640 1 -- 172.21.15.96:0/590604504 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+2375 (secure 0 0 0) 0x7f5ed805eea0 con 0x7f5ee015ea70 2024-05-28T06:25:58.322 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:25:58.322 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:25:58.322 INFO:teuthology.orchestra.run.smithi096.stdout: "fsid": "e4a831b5-e6d1-47a7-96cb-bf85154ee0d2", 2024-05-28T06:25:58.322 INFO:teuthology.orchestra.run.smithi096.stdout: "health": { 2024-05-28T06:25:58.322 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "HEALTH_OK", 2024-05-28T06:25:58.322 INFO:teuthology.orchestra.run.smithi096.stdout: "checks": {}, 2024-05-28T06:25:58.322 INFO:teuthology.orchestra.run.smithi096.stdout: "mutes": [] 2024-05-28T06:25:58.323 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:25:58.323 INFO:teuthology.orchestra.run.smithi096.stdout: "election_epoch": 8, 2024-05-28T06:25:58.323 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum": [ 2024-05-28T06:25:58.323 INFO:teuthology.orchestra.run.smithi096.stdout: 0, 2024-05-28T06:25:58.323 INFO:teuthology.orchestra.run.smithi096.stdout: 1, 2024-05-28T06:25:58.323 INFO:teuthology.orchestra.run.smithi096.stdout: 2 2024-05-28T06:25:58.323 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:25:58.323 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_names": [ 2024-05-28T06:25:58.323 INFO:teuthology.orchestra.run.smithi096.stdout: "a", 2024-05-28T06:25:58.324 INFO:teuthology.orchestra.run.smithi096.stdout: "b", 2024-05-28T06:25:58.324 INFO:teuthology.orchestra.run.smithi096.stdout: "c" 2024-05-28T06:25:58.324 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:25:58.324 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_age": 1331, 2024-05-28T06:25:58.324 INFO:teuthology.orchestra.run.smithi096.stdout: "monmap": { 2024-05-28T06:25:58.324 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 1, 2024-05-28T06:25:58.324 INFO:teuthology.orchestra.run.smithi096.stdout: "min_mon_release_name": "squid", 2024-05-28T06:25:58.324 INFO:teuthology.orchestra.run.smithi096.stdout: "num_mons": 3 2024-05-28T06:25:58.324 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:25:58.324 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap": { 2024-05-28T06:25:58.325 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 132, 2024-05-28T06:25:58.325 INFO:teuthology.orchestra.run.smithi096.stdout: "num_osds": 8, 2024-05-28T06:25:58.325 INFO:teuthology.orchestra.run.smithi096.stdout: "num_up_osds": 8, 2024-05-28T06:25:58.325 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_up_since": 1716876238, 2024-05-28T06:25:58.325 INFO:teuthology.orchestra.run.smithi096.stdout: "num_in_osds": 8, 2024-05-28T06:25:58.325 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_in_since": 1716876229, 2024-05-28T06:25:58.325 INFO:teuthology.orchestra.run.smithi096.stdout: "num_remapped_pgs": 0 2024-05-28T06:25:58.325 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:25:58.325 INFO:teuthology.orchestra.run.smithi096.stdout: "pgmap": { 2024-05-28T06:25:58.325 INFO:teuthology.orchestra.run.smithi096.stdout: "pgs_by_state": [ 2024-05-28T06:25:58.326 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:25:58.326 INFO:teuthology.orchestra.run.smithi096.stdout: "state_name": "active+clean", 2024-05-28T06:25:58.326 INFO:teuthology.orchestra.run.smithi096.stdout: "count": 97 2024-05-28T06:25:58.326 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:25:58.327 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:25:58.327 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pgs": 97, 2024-05-28T06:25:58.327 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pools": 3, 2024-05-28T06:25:58.327 INFO:teuthology.orchestra.run.smithi096.stdout: "num_objects": 4663, 2024-05-28T06:25:58.327 INFO:teuthology.orchestra.run.smithi096.stdout: "data_bytes": 19413472918, 2024-05-28T06:25:58.327 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_used": 39973707776, 2024-05-28T06:25:58.327 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_avail": 733120405504, 2024-05-28T06:25:58.327 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_total": 773094113280, 2024-05-28T06:25:58.327 INFO:teuthology.orchestra.run.smithi096.stdout: "read_bytes_sec": 17119602, 2024-05-28T06:25:58.328 INFO:teuthology.orchestra.run.smithi096.stdout: "write_bytes_sec": 16425975, 2024-05-28T06:25:58.328 INFO:teuthology.orchestra.run.smithi096.stdout: "read_op_per_sec": 4, 2024-05-28T06:25:58.328 INFO:teuthology.orchestra.run.smithi096.stdout: "write_op_per_sec": 6 2024-05-28T06:25:58.328 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:25:58.328 INFO:teuthology.orchestra.run.smithi096.stdout: "fsmap": { 2024-05-28T06:25:58.328 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 83, 2024-05-28T06:25:58.328 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 9, 2024-05-28T06:25:58.328 INFO:teuthology.orchestra.run.smithi096.stdout: "up": 1, 2024-05-28T06:25:58.328 INFO:teuthology.orchestra.run.smithi096.stdout: "in": 1, 2024-05-28T06:25:58.328 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 1, 2024-05-28T06:25:58.328 INFO:teuthology.orchestra.run.smithi096.stdout: "by_rank": [ 2024-05-28T06:25:58.328 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:25:58.328 INFO:teuthology.orchestra.run.smithi096.stdout: "filesystem_id": 9, 2024-05-28T06:25:58.329 INFO:teuthology.orchestra.run.smithi096.stdout: "rank": 0, 2024-05-28T06:25:58.329 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "a", 2024-05-28T06:25:58.329 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "up:active", 2024-05-28T06:25:58.329 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 9849 2024-05-28T06:25:58.329 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:25:58.329 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:25:58.329 INFO:teuthology.orchestra.run.smithi096.stdout: "up:standby": 3 2024-05-28T06:25:58.330 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:25:58.330 INFO:teuthology.orchestra.run.smithi096.stdout: "mgrmap": { 2024-05-28T06:25:58.330 INFO:teuthology.orchestra.run.smithi096.stdout: "available": true, 2024-05-28T06:25:58.330 INFO:teuthology.orchestra.run.smithi096.stdout: "num_standbys": 1, 2024-05-28T06:25:58.330 INFO:teuthology.orchestra.run.smithi096.stdout: "modules": [ 2024-05-28T06:25:58.330 INFO:teuthology.orchestra.run.smithi096.stdout: "iostat", 2024-05-28T06:25:58.330 INFO:teuthology.orchestra.run.smithi096.stdout: "nfs", 2024-05-28T06:25:58.330 INFO:teuthology.orchestra.run.smithi096.stdout: "restful" 2024-05-28T06:25:58.330 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:25:58.330 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:25:58.331 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:25:58.331 INFO:teuthology.orchestra.run.smithi096.stdout: "servicemap": { 2024-05-28T06:25:58.331 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 25, 2024-05-28T06:25:58.331 INFO:teuthology.orchestra.run.smithi096.stdout: "modified": "2024-05-28T06:25:30.049023+0000", 2024-05-28T06:25:58.331 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:25:58.331 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:25:58.331 INFO:teuthology.orchestra.run.smithi096.stdout: "progress_events": { 2024-05-28T06:25:58.331 INFO:teuthology.orchestra.run.smithi096.stdout: "f2372170-bf4f-4ba1-a7df-9e0c76ce0de6": { 2024-05-28T06:25:58.331 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "No pending clones left, waiting 16.0 seconds before exiting (29s)\n [============================] ", 2024-05-28T06:25:58.331 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": 1, 2024-05-28T06:25:58.331 INFO:teuthology.orchestra.run.smithi096.stdout: "add_to_ceph_s": true 2024-05-28T06:25:58.331 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:25:58.331 INFO:teuthology.orchestra.run.smithi096.stdout: "f7bbcf64-a8ee-4f95-9006-d32bd5fb36cf": { 2024-05-28T06:25:58.331 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "All cloning finished, waiting 16.0 seconds before exiting (36s)\n [============================] ", 2024-05-28T06:25:58.332 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": 1, 2024-05-28T06:25:58.332 INFO:teuthology.orchestra.run.smithi096.stdout: "add_to_ceph_s": true 2024-05-28T06:25:58.332 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:25:58.332 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:25:58.332 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:25:58.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:58.330+0000 7f5edfe00640 1 -- 172.21.15.96:0/590604504 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5ebc06e900 msgr2=0x7f5ebc070dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:58.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:58.330+0000 7f5edfe00640 1 --2- 172.21.15.96:0/590604504 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5ebc06e900 0x7f5ebc070dc0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f5ed0034000 tx=0x7f5ed002c000 comp rx=0 tx=0).stop 2024-05-28T06:25:58.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:58.330+0000 7f5edfe00640 1 -- 172.21.15.96:0/590604504 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5ee015ea70 msgr2=0x7f5ee00cdb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:25:58.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:58.330+0000 7f5edfe00640 1 --2- 172.21.15.96:0/590604504 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5ee015ea70 0x7f5ee00cdb90 secure :-1 s=READY pgs=2113 cs=0 l=1 rev1=1 crypto rx=0x7f5ed800ce80 tx=0x7f5ed800a710 comp rx=0 tx=0).stop 2024-05-28T06:25:58.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:58.330+0000 7f5edfe00640 1 -- 172.21.15.96:0/590604504 shutdown_connections 2024-05-28T06:25:58.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:58.330+0000 7f5edfe00640 1 --2- 172.21.15.96:0/590604504 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5ebc06e900 0x7f5ebc070dc0 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:58.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:58.330+0000 7f5edfe00640 1 --2- 172.21.15.96:0/590604504 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5ee015ea70 0x7f5ee00cdb90 unknown :-1 s=CLOSED pgs=2113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:58.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:58.330+0000 7f5edfe00640 1 --2- 172.21.15.96:0/590604504 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5ee015e690 0x7f5ee01f77a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:58.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:58.330+0000 7f5edfe00640 1 --2- 172.21.15.96:0/590604504 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5ee00d1ca0 0x7f5ee015fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:25:58.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:58.330+0000 7f5edfe00640 1 -- 172.21.15.96:0/590604504 >> 172.21.15.96:0/590604504 conn(0x7f5ee009d910 msgr2=0x7f5ee00b4430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:25:58.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:58.331+0000 7f5edfe00640 1 -- 172.21.15.96:0/590604504 shutdown_connections 2024-05-28T06:25:58.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:25:58.331+0000 7f5edfe00640 1 -- 172.21.15.96:0/590604504 wait complete. 2024-05-28T06:26:03.344 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-05-28T06:26:03.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.554+0000 7f1a89200640 1 Processor -- start 2024-05-28T06:26:03.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.554+0000 7f1a89200640 1 -- start start 2024-05-28T06:26:03.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.556+0000 7f1a89200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f1a84155a00 0x7f1a8415baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:03.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.556+0000 7f1a89200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1a840d0970 0x7f1a8406dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:03.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.556+0000 7f1a89200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a8415c0b0 con 0x7f1a8406f390 2024-05-28T06:26:03.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.556+0000 7f1a89200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f1a840ccf20 con 0x7f1a84155a00 2024-05-28T06:26:03.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.556+0000 7f1a89200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f1a840d0490 con 0x7f1a840d0970 2024-05-28T06:26:03.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.556+0000 7f1a82800640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f1a8406f390 0x7f1a8415f5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:34284/0 (socket says 172.21.15.96:34284) 2024-05-28T06:26:03.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.556+0000 7f1a82800640 1 -- 172.21.15.96:0/3051025565 learned_addr learned my addr 172.21.15.96:0/3051025565 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:26:03.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.556+0000 7f1a81400640 1 --2- 172.21.15.96:0/3051025565 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1a840d0970 0x7f1a8406dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:03.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.556+0000 7f1a81e00640 1 --2- 172.21.15.96:0/3051025565 >> v2:172.21.15.123:3300/0 conn(0x7f1a84155a00 0x7f1a8415baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:03.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.557+0000 7f1a80a00640 1 -- 172.21.15.96:0/3051025565 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2122833931 0 0) 0x7f1a8415c0b0 con 0x7f1a8406f390 2024-05-28T06:26:03.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.557+0000 7f1a80a00640 1 -- 172.21.15.96:0/3051025565 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1a68003520 con 0x7f1a8406f390 2024-05-28T06:26:03.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.557+0000 7f1a80a00640 1 -- 172.21.15.96:0/3051025565 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 848694035 0 0) 0x7f1a68003520 con 0x7f1a8406f390 2024-05-28T06:26:03.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.557+0000 7f1a80a00640 1 -- 172.21.15.96:0/3051025565 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1a840d0970 msgr2=0x7f1a8406dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:03.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.557+0000 7f1a80a00640 1 --2- 172.21.15.96:0/3051025565 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1a840d0970 0x7f1a8406dcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:03.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.557+0000 7f1a80a00640 1 -- 172.21.15.96:0/3051025565 >> v2:172.21.15.123:3300/0 conn(0x7f1a84155a00 msgr2=0x7f1a8415baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:03.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.557+0000 7f1a80a00640 1 --2- 172.21.15.96:0/3051025565 >> v2:172.21.15.123:3300/0 conn(0x7f1a84155a00 0x7f1a8415baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:03.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.557+0000 7f1a80a00640 1 -- 172.21.15.96:0/3051025565 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a8415c390 con 0x7f1a8406f390 2024-05-28T06:26:03.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.557+0000 7f1a80a00640 1 -- 172.21.15.96:0/3051025565 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f1a7c0019f0 con 0x7f1a8406f390 2024-05-28T06:26:03.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.557+0000 7f1a80a00640 1 -- 172.21.15.96:0/3051025565 >> v1:172.21.15.96:6789/0 conn(0x7f1a8406f390 legacy=0x7f1a8415f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:03.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.557+0000 7f1a80a00640 1 --2- 172.21.15.96:0/3051025565 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a680039b0 0x7f1a68003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:03.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.557+0000 7f1a80a00640 1 --2- 172.21.15.96:0/3051025565 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1a840d0970 0x7f1a68004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:03.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.557+0000 7f1a80a00640 1 --2- 172.21.15.96:0/3051025565 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a68004d40 0x7f1a680051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:03.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.557+0000 7f1a80a00640 1 -- 172.21.15.96:0/3051025565 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f1a6800fdd0 con 0x7f1a680039b0 2024-05-28T06:26:03.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.557+0000 7f1a80a00640 1 -- 172.21.15.96:0/3051025565 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f1a6800ff50 con 0x7f1a68004d40 2024-05-28T06:26:03.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.557+0000 7f1a80a00640 1 -- 172.21.15.96:0/3051025565 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f1a680100d0 con 0x7f1a840d0970 2024-05-28T06:26:03.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.557+0000 7f1a81e00640 1 --2- 172.21.15.96:0/3051025565 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a680039b0 0x7f1a68003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:03.553 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.558+0000 7f1a81e00640 1 -- 172.21.15.96:0/3051025565 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1a840d0970 msgr2=0x7f1a68004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:03.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.558+0000 7f1a81400640 1 --2- 172.21.15.96:0/3051025565 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1a840d0970 0x7f1a68004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:03.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.558+0000 7f1a81e00640 1 --2- 172.21.15.96:0/3051025565 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1a840d0970 0x7f1a68004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:03.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.558+0000 7f1a81e00640 1 -- 172.21.15.96:0/3051025565 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a68004d40 msgr2=0x7f1a680051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:03.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.558+0000 7f1a81e00640 1 --2- 172.21.15.96:0/3051025565 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a68004d40 0x7f1a680051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:03.554 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.558+0000 7f1a81e00640 1 -- 172.21.15.96:0/3051025565 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a68003520 con 0x7f1a680039b0 2024-05-28T06:26:03.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.559+0000 7f1a81e00640 1 --2- 172.21.15.96:0/3051025565 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a680039b0 0x7f1a68003dd0 secure :-1 s=READY pgs=2115 cs=0 l=1 rev1=1 crypto rx=0x7f1a74009ac0 tx=0x7f1a68005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8356cbeae172438c server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:03.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.559+0000 7f1a80a00640 1 -- 172.21.15.96:0/3051025565 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a74036070 con 0x7f1a680039b0 2024-05-28T06:26:03.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.559+0000 7f1a80a00640 1 -- 172.21.15.96:0/3051025565 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a74031400 con 0x7f1a680039b0 2024-05-28T06:26:03.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.559+0000 7f1a80a00640 1 -- 172.21.15.96:0/3051025565 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a74009690 con 0x7f1a680039b0 2024-05-28T06:26:03.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.560+0000 7f1a89200640 1 -- 172.21.15.96:0/3051025565 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a680039b0 msgr2=0x7f1a68003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:03.555 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.560+0000 7f1a89200640 1 --2- 172.21.15.96:0/3051025565 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a680039b0 0x7f1a68003dd0 secure :-1 s=READY pgs=2115 cs=0 l=1 rev1=1 crypto rx=0x7f1a74009ac0 tx=0x7f1a68005830 comp rx=0 tx=0).stop 2024-05-28T06:26:03.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.560+0000 7f1a82800640 1 -- 172.21.15.96:0/3051025565 reap_dead start 2024-05-28T06:26:03.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.560+0000 7f1a89200640 1 -- 172.21.15.96:0/3051025565 shutdown_connections 2024-05-28T06:26:03.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.560+0000 7f1a89200640 1 -- 172.21.15.96:0/3051025565 >> 172.21.15.96:0/3051025565 conn(0x7f1a84067ce0 msgr2=0x7f1a8406b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:26:03.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.560+0000 7f1a89200640 1 -- 172.21.15.96:0/3051025565 shutdown_connections 2024-05-28T06:26:03.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.560+0000 7f1a89200640 1 -- 172.21.15.96:0/3051025565 wait complete. 2024-05-28T06:26:03.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.561+0000 7f1a89200640 1 Processor -- start 2024-05-28T06:26:03.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.562+0000 7f1a89200640 1 -- start start 2024-05-28T06:26:03.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.562+0000 7f1a89200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a680039b0 0x7f1a840b0260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:03.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.562+0000 7f1a89200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1a84159f80 0x7f1a841ef510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:03.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.563+0000 7f1a89200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a841525d0 0x7f1a841efa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:03.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.563+0000 7f1a89200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f1a8415f090 con 0x7f1a841525d0 2024-05-28T06:26:03.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.563+0000 7f1a89200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f1a8415c950 con 0x7f1a680039b0 2024-05-28T06:26:03.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.563+0000 7f1a89200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f1a84155de0 con 0x7f1a84159f80 2024-05-28T06:26:03.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.563+0000 7f1a82800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a841525d0 0x7f1a841efa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:03.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.563+0000 7f1a82800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a841525d0 0x7f1a841efa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:40742/0 (socket says 172.21.15.96:40742) 2024-05-28T06:26:03.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.563+0000 7f1a82800640 1 -- 172.21.15.96:0/1670465338 learned_addr learned my addr 172.21.15.96:0/1670465338 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:26:03.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.563+0000 7f1a81e00640 1 --2- 172.21.15.96:0/1670465338 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a680039b0 0x7f1a840b0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:03.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.563+0000 7f1a82800640 1 -- 172.21.15.96:0/1670465338 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1a84159f80 msgr2=0x7f1a841ef510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:03.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.563+0000 7f1a81400640 1 --2- 172.21.15.96:0/1670465338 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1a84159f80 0x7f1a841ef510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:03.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.563+0000 7f1a82800640 1 --2- 172.21.15.96:0/1670465338 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1a84159f80 0x7f1a841ef510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:03.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.563+0000 7f1a82800640 1 -- 172.21.15.96:0/1670465338 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a680039b0 msgr2=0x7f1a840b0260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:03.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.563+0000 7f1a82800640 1 --2- 172.21.15.96:0/1670465338 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a680039b0 0x7f1a840b0260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:03.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.563+0000 7f1a82800640 1 -- 172.21.15.96:0/1670465338 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a8415a410 con 0x7f1a841525d0 2024-05-28T06:26:03.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.564+0000 7f1a82800640 1 --2- 172.21.15.96:0/1670465338 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a841525d0 0x7f1a841efa50 secure :-1 s=READY pgs=2116 cs=0 l=1 rev1=1 crypto rx=0x7f1a7c001370 tx=0x7f1a7c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:03.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.564+0000 7f1a7a000640 1 -- 172.21.15.96:0/1670465338 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a7c00a5b0 con 0x7f1a841525d0 2024-05-28T06:26:03.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.564+0000 7f1a7a000640 1 -- 172.21.15.96:0/1670465338 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a7c00a750 con 0x7f1a841525d0 2024-05-28T06:26:03.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.564+0000 7f1a7a000640 1 -- 172.21.15.96:0/1670465338 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a7c00ab20 con 0x7f1a841525d0 2024-05-28T06:26:03.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.564+0000 7f1a89200640 1 -- 172.21.15.96:0/1670465338 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a8415a6f0 con 0x7f1a841525d0 2024-05-28T06:26:03.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.564+0000 7f1a89200640 1 -- 172.21.15.96:0/1670465338 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a8415acb0 con 0x7f1a841525d0 2024-05-28T06:26:03.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.565+0000 7f1a89200640 1 -- 172.21.15.96:0/1670465338 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a840689e0 con 0x7f1a841525d0 2024-05-28T06:26:03.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.566+0000 7f1a7a000640 1 -- 172.21.15.96:0/1670465338 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 48) ==== 93947+0+0 (secure 0 0 0) 0x7f1a7c011070 con 0x7f1a841525d0 2024-05-28T06:26:03.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.567+0000 7f1a7a000640 1 --2- 172.21.15.96:0/1670465338 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1a6406e900 0x7f1a64070dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:03.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.567+0000 7f1a7a000640 1 -- 172.21.15.96:0/1670465338 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 5823+0+0 (secure 0 0 0) 0x7f1a7c08a640 con 0x7f1a841525d0 2024-05-28T06:26:03.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.568+0000 7f1a81e00640 1 --2- 172.21.15.96:0/1670465338 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1a6406e900 0x7f1a64070dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:03.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.569+0000 7f1a81e00640 1 --2- 172.21.15.96:0/1670465338 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1a6406e900 0x7f1a64070dc0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f1a74033000 tx=0x7f1a74008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:03.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.572+0000 7f1a7a000640 1 -- 172.21.15.96:0/1670465338 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1a7c01a060 con 0x7f1a841525d0 2024-05-28T06:26:03.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.841+0000 7f1a89200640 1 -- 172.21.15.96:0/1670465338 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f1a8406f170 con 0x7f1a841525d0 2024-05-28T06:26:03.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.842+0000 7f1a7a000640 1 -- 172.21.15.96:0/1670465338 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+2335 (secure 0 0 0) 0x7f1a7c00f260 con 0x7f1a841525d0 2024-05-28T06:26:03.835 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:26:03.836 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:26:03.836 INFO:teuthology.orchestra.run.smithi096.stdout: "fsid": "e4a831b5-e6d1-47a7-96cb-bf85154ee0d2", 2024-05-28T06:26:03.836 INFO:teuthology.orchestra.run.smithi096.stdout: "health": { 2024-05-28T06:26:03.836 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "HEALTH_OK", 2024-05-28T06:26:03.836 INFO:teuthology.orchestra.run.smithi096.stdout: "checks": {}, 2024-05-28T06:26:03.836 INFO:teuthology.orchestra.run.smithi096.stdout: "mutes": [] 2024-05-28T06:26:03.836 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:26:03.836 INFO:teuthology.orchestra.run.smithi096.stdout: "election_epoch": 8, 2024-05-28T06:26:03.836 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum": [ 2024-05-28T06:26:03.836 INFO:teuthology.orchestra.run.smithi096.stdout: 0, 2024-05-28T06:26:03.836 INFO:teuthology.orchestra.run.smithi096.stdout: 1, 2024-05-28T06:26:03.836 INFO:teuthology.orchestra.run.smithi096.stdout: 2 2024-05-28T06:26:03.837 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:26:03.837 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_names": [ 2024-05-28T06:26:03.837 INFO:teuthology.orchestra.run.smithi096.stdout: "a", 2024-05-28T06:26:03.837 INFO:teuthology.orchestra.run.smithi096.stdout: "b", 2024-05-28T06:26:03.837 INFO:teuthology.orchestra.run.smithi096.stdout: "c" 2024-05-28T06:26:03.837 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:26:03.837 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_age": 1337, 2024-05-28T06:26:03.837 INFO:teuthology.orchestra.run.smithi096.stdout: "monmap": { 2024-05-28T06:26:03.837 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 1, 2024-05-28T06:26:03.837 INFO:teuthology.orchestra.run.smithi096.stdout: "min_mon_release_name": "squid", 2024-05-28T06:26:03.837 INFO:teuthology.orchestra.run.smithi096.stdout: "num_mons": 3 2024-05-28T06:26:03.837 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:26:03.838 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap": { 2024-05-28T06:26:03.838 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 132, 2024-05-28T06:26:03.838 INFO:teuthology.orchestra.run.smithi096.stdout: "num_osds": 8, 2024-05-28T06:26:03.838 INFO:teuthology.orchestra.run.smithi096.stdout: "num_up_osds": 8, 2024-05-28T06:26:03.838 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_up_since": 1716876238, 2024-05-28T06:26:03.838 INFO:teuthology.orchestra.run.smithi096.stdout: "num_in_osds": 8, 2024-05-28T06:26:03.838 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_in_since": 1716876229, 2024-05-28T06:26:03.838 INFO:teuthology.orchestra.run.smithi096.stdout: "num_remapped_pgs": 0 2024-05-28T06:26:03.838 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:26:03.838 INFO:teuthology.orchestra.run.smithi096.stdout: "pgmap": { 2024-05-28T06:26:03.838 INFO:teuthology.orchestra.run.smithi096.stdout: "pgs_by_state": [ 2024-05-28T06:26:03.839 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:26:03.839 INFO:teuthology.orchestra.run.smithi096.stdout: "state_name": "active+clean", 2024-05-28T06:26:03.839 INFO:teuthology.orchestra.run.smithi096.stdout: "count": 97 2024-05-28T06:26:03.839 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:26:03.839 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:26:03.839 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pgs": 97, 2024-05-28T06:26:03.839 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pools": 3, 2024-05-28T06:26:03.839 INFO:teuthology.orchestra.run.smithi096.stdout: "num_objects": 4663, 2024-05-28T06:26:03.839 INFO:teuthology.orchestra.run.smithi096.stdout: "data_bytes": 19413472918, 2024-05-28T06:26:03.839 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_used": 39973707776, 2024-05-28T06:26:03.839 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_avail": 733120405504, 2024-05-28T06:26:03.839 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_total": 773094113280, 2024-05-28T06:26:03.839 INFO:teuthology.orchestra.run.smithi096.stdout: "write_bytes_sec": 4947, 2024-05-28T06:26:03.839 INFO:teuthology.orchestra.run.smithi096.stdout: "read_op_per_sec": 0, 2024-05-28T06:26:03.840 INFO:teuthology.orchestra.run.smithi096.stdout: "write_op_per_sec": 1 2024-05-28T06:26:03.840 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:26:03.840 INFO:teuthology.orchestra.run.smithi096.stdout: "fsmap": { 2024-05-28T06:26:03.840 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 83, 2024-05-28T06:26:03.840 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 9, 2024-05-28T06:26:03.840 INFO:teuthology.orchestra.run.smithi096.stdout: "up": 1, 2024-05-28T06:26:03.840 INFO:teuthology.orchestra.run.smithi096.stdout: "in": 1, 2024-05-28T06:26:03.840 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 1, 2024-05-28T06:26:03.840 INFO:teuthology.orchestra.run.smithi096.stdout: "by_rank": [ 2024-05-28T06:26:03.840 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:26:03.840 INFO:teuthology.orchestra.run.smithi096.stdout: "filesystem_id": 9, 2024-05-28T06:26:03.840 INFO:teuthology.orchestra.run.smithi096.stdout: "rank": 0, 2024-05-28T06:26:03.840 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "a", 2024-05-28T06:26:03.841 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "up:active", 2024-05-28T06:26:03.841 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 9849 2024-05-28T06:26:03.841 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:26:03.841 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:26:03.841 INFO:teuthology.orchestra.run.smithi096.stdout: "up:standby": 3 2024-05-28T06:26:03.841 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:26:03.841 INFO:teuthology.orchestra.run.smithi096.stdout: "mgrmap": { 2024-05-28T06:26:03.841 INFO:teuthology.orchestra.run.smithi096.stdout: "available": true, 2024-05-28T06:26:03.841 INFO:teuthology.orchestra.run.smithi096.stdout: "num_standbys": 1, 2024-05-28T06:26:03.841 INFO:teuthology.orchestra.run.smithi096.stdout: "modules": [ 2024-05-28T06:26:03.841 INFO:teuthology.orchestra.run.smithi096.stdout: "iostat", 2024-05-28T06:26:03.841 INFO:teuthology.orchestra.run.smithi096.stdout: "nfs", 2024-05-28T06:26:03.841 INFO:teuthology.orchestra.run.smithi096.stdout: "restful" 2024-05-28T06:26:03.841 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:26:03.842 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:26:03.842 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:26:03.842 INFO:teuthology.orchestra.run.smithi096.stdout: "servicemap": { 2024-05-28T06:26:03.842 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 25, 2024-05-28T06:26:03.842 INFO:teuthology.orchestra.run.smithi096.stdout: "modified": "2024-05-28T06:25:30.049023+0000", 2024-05-28T06:26:03.842 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:26:03.842 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:26:03.842 INFO:teuthology.orchestra.run.smithi096.stdout: "progress_events": { 2024-05-28T06:26:03.842 INFO:teuthology.orchestra.run.smithi096.stdout: "f2372170-bf4f-4ba1-a7df-9e0c76ce0de6": { 2024-05-28T06:26:03.842 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "No pending clones left, waiting 12.0 seconds before exiting (34s)\n [============================] ", 2024-05-28T06:26:03.842 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": 1, 2024-05-28T06:26:03.842 INFO:teuthology.orchestra.run.smithi096.stdout: "add_to_ceph_s": true 2024-05-28T06:26:03.842 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:26:03.842 INFO:teuthology.orchestra.run.smithi096.stdout: "f7bbcf64-a8ee-4f95-9006-d32bd5fb36cf": { 2024-05-28T06:26:03.843 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "All cloning finished, waiting 12.0 seconds before exiting (40s)\n [============================] ", 2024-05-28T06:26:03.843 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": 1, 2024-05-28T06:26:03.843 INFO:teuthology.orchestra.run.smithi096.stdout: "add_to_ceph_s": true 2024-05-28T06:26:03.843 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:26:03.843 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:26:03.843 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:26:03.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.844+0000 7f1a89200640 1 -- 172.21.15.96:0/1670465338 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1a6406e900 msgr2=0x7f1a64070dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:03.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.844+0000 7f1a89200640 1 --2- 172.21.15.96:0/1670465338 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1a6406e900 0x7f1a64070dc0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f1a74033000 tx=0x7f1a74008030 comp rx=0 tx=0).stop 2024-05-28T06:26:03.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.844+0000 7f1a89200640 1 -- 172.21.15.96:0/1670465338 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a841525d0 msgr2=0x7f1a841efa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:03.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.844+0000 7f1a89200640 1 --2- 172.21.15.96:0/1670465338 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a841525d0 0x7f1a841efa50 secure :-1 s=READY pgs=2116 cs=0 l=1 rev1=1 crypto rx=0x7f1a7c001370 tx=0x7f1a7c00ccb0 comp rx=0 tx=0).stop 2024-05-28T06:26:03.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.844+0000 7f1a89200640 1 -- 172.21.15.96:0/1670465338 shutdown_connections 2024-05-28T06:26:03.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.844+0000 7f1a89200640 1 --2- 172.21.15.96:0/1670465338 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f1a6406e900 0x7f1a64070dc0 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:03.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.844+0000 7f1a89200640 1 --2- 172.21.15.96:0/1670465338 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a841525d0 0x7f1a841efa50 unknown :-1 s=CLOSED pgs=2116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:03.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.844+0000 7f1a89200640 1 --2- 172.21.15.96:0/1670465338 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f1a84159f80 0x7f1a841ef510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:03.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.844+0000 7f1a89200640 1 --2- 172.21.15.96:0/1670465338 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a680039b0 0x7f1a840b0260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:03.846 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.844+0000 7f1a89200640 1 -- 172.21.15.96:0/1670465338 >> 172.21.15.96:0/1670465338 conn(0x7f1a84067ce0 msgr2=0x7f1a840b3140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:26:03.847 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.845+0000 7f1a89200640 1 -- 172.21.15.96:0/1670465338 shutdown_connections 2024-05-28T06:26:03.847 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:03.845+0000 7f1a89200640 1 -- 172.21.15.96:0/1670465338 wait complete. 2024-05-28T06:26:05.342 DEBUG:teuthology.orchestra.run.smithi096:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:26:05.350 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:26:08.858 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-05-28T06:26:09.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:09.075+0000 7f95bce00640 1 Processor -- start 2024-05-28T06:26:09.069 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:09.075+0000 7f95bce00640 1 -- start start 2024-05-28T06:26:09.071 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:09.077+0000 7f95bce00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f95b80723d0 0x7f95b80b3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:09.071 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:09.077+0000 7f95bce00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f95b80b41e0 0x7f95b80cf880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:09.071 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:09.077+0000 7f95bce00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95b80cf0c0 con 0x7f95b8145320 2024-05-28T06:26:09.071 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:09.077+0000 7f95bce00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f95b80b4a20 con 0x7f95b80723d0 2024-05-28T06:26:09.071 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:09.077+0000 7f95bce00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f95b80b4d00 con 0x7f95b80b41e0 2024-05-28T06:26:09.071 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:09.077+0000 7f95ad400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f95b80723d0 0x7f95b80b3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:09.071 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:09.077+0000 7f95ad400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f95b80723d0 0x7f95b80b3ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35564/0 (socket says 172.21.15.96:35564) 2024-05-28T06:26:09.071 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:09.077+0000 7f95ad400640 1 -- 172.21.15.96:0/1513324053 learned_addr learned my addr 172.21.15.96:0/1513324053 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:26:09.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:09.078+0000 7f95ad400640 1 -- 172.21.15.96:0/1513324053 >> v2:172.21.15.123:3300/0 conn(0x7f95b80b41e0 msgr2=0x7f95b80cf880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:09.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:09.078+0000 7f95ad400640 1 --2- 172.21.15.96:0/1513324053 >> v2:172.21.15.123:3300/0 conn(0x7f95b80b41e0 0x7f95b80cf880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:09.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:09.078+0000 7f95ad400640 1 -- 172.21.15.96:0/1513324053 >> v1:172.21.15.123:6789/0 conn(0x7f95b8145320 legacy=0x7f95b8071cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:09.072 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:09.078+0000 7f95ad400640 1 -- 172.21.15.96:0/1513324053 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95b80b4fe0 con 0x7f95b80723d0 2024-05-28T06:26:09.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:09.079+0000 7f95ad400640 1 --2- 172.21.15.96:0/1513324053 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f95b80723d0 0x7f95b80b3ca0 secure :-1 s=READY pgs=2117 cs=0 l=1 rev1=1 crypto rx=0x7f95b806db90 tx=0x7f95a002a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27124959a6d2abbf server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:09.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:09.079+0000 7f95b5400640 1 -- 172.21.15.96:0/1513324053 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95a0037070 con 0x7f95b80723d0 2024-05-28T06:26:09.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:09.079+0000 7f95b5400640 1 -- 172.21.15.96:0/1513324053 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95a0032440 con 0x7f95b80723d0 2024-05-28T06:26:09.073 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:09.079+0000 7f95b5400640 1 -- 172.21.15.96:0/1513324053 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95a0032810 con 0x7f95b80723d0 2024-05-28T06:26:09.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:09.080+0000 7f95bce00640 1 -- 172.21.15.96:0/1513324053 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f95b80723d0 msgr2=0x7f95b80b3ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:09.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:09.080+0000 7f95bce00640 1 --2- 172.21.15.96:0/1513324053 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f95b80723d0 0x7f95b80b3ca0 secure :-1 s=READY pgs=2117 cs=0 l=1 rev1=1 crypto rx=0x7f95b806db90 tx=0x7f95a002a210 comp rx=0 tx=0).stop 2024-05-28T06:26:09.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:09.080+0000 7f95bce00640 1 -- 172.21.15.96:0/1513324053 shutdown_connections 2024-05-28T06:26:09.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:09.080+0000 7f95bce00640 1 --2- 172.21.15.96:0/1513324053 >> v2:172.21.15.123:3300/0 conn(0x7f95b80b41e0 0x7f95b80cf880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:09.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:09.080+0000 7f95bce00640 1 --2- 172.21.15.96:0/1513324053 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f95b80723d0 0x7f95b80b3ca0 unknown :-1 s=CLOSED pgs=2117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:09.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:09.080+0000 7f95bce00640 1 -- 172.21.15.96:0/1513324053 >> 172.21.15.96:0/1513324053 conn(0x7f95b8067ce0 msgr2=0x7f95b806b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:26:09.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:09.080+0000 7f95bce00640 1 -- 172.21.15.96:0/1513324053 shutdown_connections 2024-05-28T06:26:09.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:09.080+0000 7f95bce00640 1 -- 172.21.15.96:0/1513324053 wait complete. 2024-05-28T06:26:09.075 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:09.082+0000 7f95bce00640 1 Processor -- start 2024-05-28T06:26:09.075 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:09.082+0000 7f95bce00640 1 -- start start 2024-05-28T06:26:09.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:09.083+0000 7f95bce00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f95b80723d0 0x7f95b80ae950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:09.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:09.083+0000 7f95bce00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f95b80b41e0 0x7f95b80b0b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:09.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:09.083+0000 7f95bce00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f95b8145320 0x7f95b813e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:09.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:09.083+0000 7f95bce00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f95b80b4650 con 0x7f95b80b41e0 2024-05-28T06:26:09.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:09.083+0000 7f95bce00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f95b8145700 con 0x7f95b8145320 2024-05-28T06:26:09.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:09.083+0000 7f95bce00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f95b80b47d0 con 0x7f95b80723d0 2024-05-28T06:26:09.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:09.083+0000 7f95ad400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f95b80b41e0 0x7f95b80b0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:09.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:09.083+0000 7f95ad400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f95b80b41e0 0x7f95b80b0b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35572/0 (socket says 172.21.15.96:35572) 2024-05-28T06:26:09.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:09.083+0000 7f95ad400640 1 -- 172.21.15.96:0/683927830 learned_addr learned my addr 172.21.15.96:0/683927830 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:26:09.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:09.083+0000 7f95ad400640 1 -- 172.21.15.96:0/683927830 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f95b80723d0 msgr2=0x7f95b80ae950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:09.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:09.083+0000 7f95ad400640 1 --2- 172.21.15.96:0/683927830 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f95b80723d0 0x7f95b80ae950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:09.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:09.083+0000 7f95ad400640 1 -- 172.21.15.96:0/683927830 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f95b8145320 msgr2=0x7f95b813e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:09.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:09.083+0000 7f95ad400640 1 --2- 172.21.15.96:0/683927830 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f95b8145320 0x7f95b813e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:09.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:09.083+0000 7f95ad400640 1 -- 172.21.15.96:0/683927830 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95b80b1600 con 0x7f95b80b41e0 2024-05-28T06:26:09.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:09.084+0000 7f95ad400640 1 --2- 172.21.15.96:0/683927830 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f95b80b41e0 0x7f95b80b0b60 secure :-1 s=READY pgs=2118 cs=0 l=1 rev1=1 crypto rx=0x7f95a002a250 tx=0x7f95a00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:09.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:09.084+0000 7f95aea00640 1 -- 172.21.15.96:0/683927830 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95a0041070 con 0x7f95b80b41e0 2024-05-28T06:26:09.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:09.084+0000 7f95aea00640 1 -- 172.21.15.96:0/683927830 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95a003c890 con 0x7f95b80b41e0 2024-05-28T06:26:09.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:09.084+0000 7f95aea00640 1 -- 172.21.15.96:0/683927830 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95a0037040 con 0x7f95b80b41e0 2024-05-28T06:26:09.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:09.084+0000 7f95bce00640 1 -- 172.21.15.96:0/683927830 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95b80b18e0 con 0x7f95b80b41e0 2024-05-28T06:26:09.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:09.084+0000 7f95bce00640 1 -- 172.21.15.96:0/683927830 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95b815d810 con 0x7f95b80b41e0 2024-05-28T06:26:09.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:09.085+0000 7f95bce00640 1 -- 172.21.15.96:0/683927830 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95b80689e0 con 0x7f95b80b41e0 2024-05-28T06:26:09.080 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:09.086+0000 7f95aea00640 1 -- 172.21.15.96:0/683927830 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 48) ==== 93947+0+0 (secure 0 0 0) 0x7f95a0044070 con 0x7f95b80b41e0 2024-05-28T06:26:09.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:09.088+0000 7f95aea00640 1 --2- 172.21.15.96:0/683927830 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f959406e900 0x7f9594070dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:09.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:09.088+0000 7f95aea00640 1 -- 172.21.15.96:0/683927830 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 5823+0+0 (secure 0 0 0) 0x7f95a00aba10 con 0x7f95b80b41e0 2024-05-28T06:26:09.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:09.088+0000 7f95b5e00640 1 --2- 172.21.15.96:0/683927830 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f959406e900 0x7f9594070dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:09.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:09.090+0000 7f95b5e00640 1 --2- 172.21.15.96:0/683927830 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f959406e900 0x7f9594070dc0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f95b81f7d00 tx=0x7f95a8006ba0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:09.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:09.093+0000 7f95aea00640 1 -- 172.21.15.96:0/683927830 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f95a007b9c0 con 0x7f95b80b41e0 2024-05-28T06:26:09.349 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:09.355+0000 7f95bce00640 1 -- 172.21.15.96:0/683927830 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f95b80b34e0 con 0x7f95b80b41e0 2024-05-28T06:26:09.350 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:09.357+0000 7f95aea00640 1 -- 172.21.15.96:0/683927830 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+2239 (secure 0 0 0) 0x7f95a007b2c0 con 0x7f95b80b41e0 2024-05-28T06:26:09.350 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:26:09.350 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:26:09.351 INFO:teuthology.orchestra.run.smithi096.stdout: "fsid": "e4a831b5-e6d1-47a7-96cb-bf85154ee0d2", 2024-05-28T06:26:09.351 INFO:teuthology.orchestra.run.smithi096.stdout: "health": { 2024-05-28T06:26:09.351 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "HEALTH_OK", 2024-05-28T06:26:09.351 INFO:teuthology.orchestra.run.smithi096.stdout: "checks": {}, 2024-05-28T06:26:09.351 INFO:teuthology.orchestra.run.smithi096.stdout: "mutes": [] 2024-05-28T06:26:09.351 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:26:09.351 INFO:teuthology.orchestra.run.smithi096.stdout: "election_epoch": 8, 2024-05-28T06:26:09.351 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum": [ 2024-05-28T06:26:09.351 INFO:teuthology.orchestra.run.smithi096.stdout: 0, 2024-05-28T06:26:09.351 INFO:teuthology.orchestra.run.smithi096.stdout: 1, 2024-05-28T06:26:09.351 INFO:teuthology.orchestra.run.smithi096.stdout: 2 2024-05-28T06:26:09.351 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:26:09.351 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_names": [ 2024-05-28T06:26:09.352 INFO:teuthology.orchestra.run.smithi096.stdout: "a", 2024-05-28T06:26:09.352 INFO:teuthology.orchestra.run.smithi096.stdout: "b", 2024-05-28T06:26:09.352 INFO:teuthology.orchestra.run.smithi096.stdout: "c" 2024-05-28T06:26:09.352 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:26:09.352 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_age": 1342, 2024-05-28T06:26:09.352 INFO:teuthology.orchestra.run.smithi096.stdout: "monmap": { 2024-05-28T06:26:09.352 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 1, 2024-05-28T06:26:09.352 INFO:teuthology.orchestra.run.smithi096.stdout: "min_mon_release_name": "squid", 2024-05-28T06:26:09.352 INFO:teuthology.orchestra.run.smithi096.stdout: "num_mons": 3 2024-05-28T06:26:09.352 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:26:09.352 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap": { 2024-05-28T06:26:09.352 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 132, 2024-05-28T06:26:09.352 INFO:teuthology.orchestra.run.smithi096.stdout: "num_osds": 8, 2024-05-28T06:26:09.352 INFO:teuthology.orchestra.run.smithi096.stdout: "num_up_osds": 8, 2024-05-28T06:26:09.353 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_up_since": 1716876238, 2024-05-28T06:26:09.353 INFO:teuthology.orchestra.run.smithi096.stdout: "num_in_osds": 8, 2024-05-28T06:26:09.353 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_in_since": 1716876229, 2024-05-28T06:26:09.353 INFO:teuthology.orchestra.run.smithi096.stdout: "num_remapped_pgs": 0 2024-05-28T06:26:09.353 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:26:09.353 INFO:teuthology.orchestra.run.smithi096.stdout: "pgmap": { 2024-05-28T06:26:09.353 INFO:teuthology.orchestra.run.smithi096.stdout: "pgs_by_state": [ 2024-05-28T06:26:09.353 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:26:09.353 INFO:teuthology.orchestra.run.smithi096.stdout: "state_name": "active+clean", 2024-05-28T06:26:09.353 INFO:teuthology.orchestra.run.smithi096.stdout: "count": 97 2024-05-28T06:26:09.353 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:26:09.353 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:26:09.353 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pgs": 97, 2024-05-28T06:26:09.354 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pools": 3, 2024-05-28T06:26:09.354 INFO:teuthology.orchestra.run.smithi096.stdout: "num_objects": 4663, 2024-05-28T06:26:09.354 INFO:teuthology.orchestra.run.smithi096.stdout: "data_bytes": 19413472918, 2024-05-28T06:26:09.354 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_used": 39973707776, 2024-05-28T06:26:09.354 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_avail": 733120405504, 2024-05-28T06:26:09.354 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_total": 773094113280 2024-05-28T06:26:09.354 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:26:09.354 INFO:teuthology.orchestra.run.smithi096.stdout: "fsmap": { 2024-05-28T06:26:09.354 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 83, 2024-05-28T06:26:09.354 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 9, 2024-05-28T06:26:09.354 INFO:teuthology.orchestra.run.smithi096.stdout: "up": 1, 2024-05-28T06:26:09.354 INFO:teuthology.orchestra.run.smithi096.stdout: "in": 1, 2024-05-28T06:26:09.354 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 1, 2024-05-28T06:26:09.355 INFO:teuthology.orchestra.run.smithi096.stdout: "by_rank": [ 2024-05-28T06:26:09.355 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:26:09.355 INFO:teuthology.orchestra.run.smithi096.stdout: "filesystem_id": 9, 2024-05-28T06:26:09.355 INFO:teuthology.orchestra.run.smithi096.stdout: "rank": 0, 2024-05-28T06:26:09.355 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "a", 2024-05-28T06:26:09.355 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "up:active", 2024-05-28T06:26:09.355 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 9849 2024-05-28T06:26:09.355 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:26:09.355 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:26:09.355 INFO:teuthology.orchestra.run.smithi096.stdout: "up:standby": 3 2024-05-28T06:26:09.355 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:26:09.355 INFO:teuthology.orchestra.run.smithi096.stdout: "mgrmap": { 2024-05-28T06:26:09.355 INFO:teuthology.orchestra.run.smithi096.stdout: "available": true, 2024-05-28T06:26:09.355 INFO:teuthology.orchestra.run.smithi096.stdout: "num_standbys": 1, 2024-05-28T06:26:09.356 INFO:teuthology.orchestra.run.smithi096.stdout: "modules": [ 2024-05-28T06:26:09.356 INFO:teuthology.orchestra.run.smithi096.stdout: "iostat", 2024-05-28T06:26:09.356 INFO:teuthology.orchestra.run.smithi096.stdout: "nfs", 2024-05-28T06:26:09.356 INFO:teuthology.orchestra.run.smithi096.stdout: "restful" 2024-05-28T06:26:09.356 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:26:09.356 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:26:09.356 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:26:09.356 INFO:teuthology.orchestra.run.smithi096.stdout: "servicemap": { 2024-05-28T06:26:09.356 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 25, 2024-05-28T06:26:09.356 INFO:teuthology.orchestra.run.smithi096.stdout: "modified": "2024-05-28T06:25:30.049023+0000", 2024-05-28T06:26:09.356 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:26:09.357 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:26:09.357 INFO:teuthology.orchestra.run.smithi096.stdout: "progress_events": { 2024-05-28T06:26:09.357 INFO:teuthology.orchestra.run.smithi096.stdout: "f2372170-bf4f-4ba1-a7df-9e0c76ce0de6": { 2024-05-28T06:26:09.357 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "No pending clones left, waiting 6.0 seconds before exiting (40s)\n [============================] ", 2024-05-28T06:26:09.357 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": 1, 2024-05-28T06:26:09.357 INFO:teuthology.orchestra.run.smithi096.stdout: "add_to_ceph_s": true 2024-05-28T06:26:09.357 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:26:09.357 INFO:teuthology.orchestra.run.smithi096.stdout: "f7bbcf64-a8ee-4f95-9006-d32bd5fb36cf": { 2024-05-28T06:26:09.357 INFO:teuthology.orchestra.run.smithi096.stdout: "message": "All cloning finished, waiting 6.0 seconds before exiting (46s)\n [============================] ", 2024-05-28T06:26:09.357 INFO:teuthology.orchestra.run.smithi096.stdout: "progress": 1, 2024-05-28T06:26:09.357 INFO:teuthology.orchestra.run.smithi096.stdout: "add_to_ceph_s": true 2024-05-28T06:26:09.357 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:26:09.357 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:26:09.357 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:26:09.358 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:09.359+0000 7f95bce00640 1 -- 172.21.15.96:0/683927830 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f959406e900 msgr2=0x7f9594070dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:09.358 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:09.359+0000 7f95bce00640 1 --2- 172.21.15.96:0/683927830 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f959406e900 0x7f9594070dc0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f95b81f7d00 tx=0x7f95a8006ba0 comp rx=0 tx=0).stop 2024-05-28T06:26:09.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:09.359+0000 7f95bce00640 1 -- 172.21.15.96:0/683927830 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f95b80b41e0 msgr2=0x7f95b80b0b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:09.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:09.359+0000 7f95bce00640 1 --2- 172.21.15.96:0/683927830 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f95b80b41e0 0x7f95b80b0b60 secure :-1 s=READY pgs=2118 cs=0 l=1 rev1=1 crypto rx=0x7f95a002a250 tx=0x7f95a00046c0 comp rx=0 tx=0).stop 2024-05-28T06:26:09.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:09.359+0000 7f95bce00640 1 -- 172.21.15.96:0/683927830 shutdown_connections 2024-05-28T06:26:09.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:09.359+0000 7f95bce00640 1 --2- 172.21.15.96:0/683927830 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f959406e900 0x7f9594070dc0 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:09.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:09.359+0000 7f95bce00640 1 --2- 172.21.15.96:0/683927830 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f95b8145320 0x7f95b813e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:09.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:09.359+0000 7f95bce00640 1 --2- 172.21.15.96:0/683927830 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f95b80b41e0 0x7f95b80b0b60 unknown :-1 s=CLOSED pgs=2118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:09.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:09.359+0000 7f95bce00640 1 --2- 172.21.15.96:0/683927830 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f95b80723d0 0x7f95b80ae950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:09.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:09.359+0000 7f95bce00640 1 -- 172.21.15.96:0/683927830 >> 172.21.15.96:0/683927830 conn(0x7f95b8067ce0 msgr2=0x7f95b8140b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:26:09.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:09.359+0000 7f95bce00640 1 -- 172.21.15.96:0/683927830 shutdown_connections 2024-05-28T06:26:09.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:09.359+0000 7f95bce00640 1 -- 172.21.15.96:0/683927830 wait complete. 2024-05-28T06:26:14.374 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-05-28T06:26:14.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:14.587+0000 7f374a200640 1 Processor -- start 2024-05-28T06:26:14.581 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:14.587+0000 7f374a200640 1 -- start start 2024-05-28T06:26:14.582 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:14.589+0000 7f374a200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f3744145320 0x7f3744071cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:14.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:14.589+0000 7f374a200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3744072200 0x7f37440b3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:14.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:14.589+0000 7f374a200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f37440b41e0 0x7f3744160240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:14.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:14.589+0000 7f374a200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f3744072660 con 0x7f37440b41e0 2024-05-28T06:26:14.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:14.589+0000 7f374a200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f37440cf0c0 con 0x7f3744145320 2024-05-28T06:26:14.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:14.589+0000 7f374a200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f37440b4a20 con 0x7f3744072200 2024-05-28T06:26:14.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:14.589+0000 7f3743e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f37440b41e0 0x7f3744160240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:14.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:14.589+0000 7f3743e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f37440b41e0 0x7f3744160240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35576/0 (socket says 172.21.15.96:35576) 2024-05-28T06:26:14.583 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:14.589+0000 7f3743e00640 1 -- 172.21.15.96:0/3964482109 learned_addr learned my addr 172.21.15.96:0/3964482109 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:26:14.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:14.589+0000 7f3743e00640 1 -- 172.21.15.96:0/3964482109 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3744072200 msgr2=0x7f37440b3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:14.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:14.589+0000 7f3743e00640 1 --2- 172.21.15.96:0/3964482109 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3744072200 0x7f37440b3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:14.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:14.589+0000 7f3743e00640 1 -- 172.21.15.96:0/3964482109 >> v2:172.21.15.123:3300/0 conn(0x7f3744145320 msgr2=0x7f3744071cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:14.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:14.589+0000 7f3743e00640 1 --2- 172.21.15.96:0/3964482109 >> v2:172.21.15.123:3300/0 conn(0x7f3744145320 0x7f3744071cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:14.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:14.589+0000 7f3743e00640 1 -- 172.21.15.96:0/3964482109 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37440b4d00 con 0x7f37440b41e0 2024-05-28T06:26:14.584 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:14.590+0000 7f3743e00640 1 --2- 172.21.15.96:0/3964482109 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f37440b41e0 0x7f3744160240 secure :-1 s=READY pgs=2119 cs=0 l=1 rev1=1 crypto rx=0x7f37440d1350 tx=0x7f373c02a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17ef61d6ea43a75a server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:14.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:14.591+0000 7f3742a00640 1 -- 172.21.15.96:0/3964482109 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f373c036070 con 0x7f37440b41e0 2024-05-28T06:26:14.585 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:14.591+0000 7f3742a00640 1 -- 172.21.15.96:0/3964482109 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f373c031600 con 0x7f37440b41e0 2024-05-28T06:26:14.586 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:14.591+0000 7f3742a00640 1 -- 172.21.15.96:0/3964482109 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f373c0319d0 con 0x7f37440b41e0 2024-05-28T06:26:14.586 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:14.591+0000 7f374a200640 1 -- 172.21.15.96:0/3964482109 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f37440b41e0 msgr2=0x7f3744160240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:14.586 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:14.591+0000 7f374a200640 1 --2- 172.21.15.96:0/3964482109 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f37440b41e0 0x7f3744160240 secure :-1 s=READY pgs=2119 cs=0 l=1 rev1=1 crypto rx=0x7f37440d1350 tx=0x7f373c02a6a0 comp rx=0 tx=0).stop 2024-05-28T06:26:14.586 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:14.592+0000 7f374a200640 1 -- 172.21.15.96:0/3964482109 shutdown_connections 2024-05-28T06:26:14.586 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:14.592+0000 7f374a200640 1 --2- 172.21.15.96:0/3964482109 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f37440b41e0 0x7f3744160240 unknown :-1 s=CLOSED pgs=2119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:14.586 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:14.592+0000 7f374a200640 1 --2- 172.21.15.96:0/3964482109 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3744072200 0x7f37440b3ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:14.586 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:14.592+0000 7f374a200640 1 --2- 172.21.15.96:0/3964482109 >> v2:172.21.15.123:3300/0 conn(0x7f3744145320 0x7f3744071cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:14.586 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:14.592+0000 7f374a200640 1 -- 172.21.15.96:0/3964482109 >> 172.21.15.96:0/3964482109 conn(0x7f3744067ce0 msgr2=0x7f374406b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:26:14.586 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:14.592+0000 7f374a200640 1 -- 172.21.15.96:0/3964482109 shutdown_connections 2024-05-28T06:26:14.586 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:14.592+0000 7f374a200640 1 -- 172.21.15.96:0/3964482109 wait complete. 2024-05-28T06:26:14.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:14.593+0000 7f374a200640 1 Processor -- start 2024-05-28T06:26:14.587 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:14.593+0000 7f374a200640 1 -- start start 2024-05-28T06:26:14.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:14.594+0000 7f374a200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3744072200 0x7f37441f4f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:14.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:14.594+0000 7f374a200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f37440b41e0 0x7f374415f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:14.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:14.594+0000 7f374a200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3744145320 0x7f37441f7a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:14.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:14.594+0000 7f374a200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f3744145700 con 0x7f3744145320 2024-05-28T06:26:14.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:14.594+0000 7f374a200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f37440cf360 con 0x7f3744072200 2024-05-28T06:26:14.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:14.594+0000 7f374a200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f37440b4670 con 0x7f37440b41e0 2024-05-28T06:26:14.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:14.595+0000 7f3743e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3744145320 0x7f37441f7a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:14.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:14.595+0000 7f3743e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3744145320 0x7f37441f7a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35590/0 (socket says 172.21.15.96:35590) 2024-05-28T06:26:14.589 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:14.595+0000 7f3743e00640 1 -- 172.21.15.96:0/611965308 learned_addr learned my addr 172.21.15.96:0/611965308 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:26:14.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:14.595+0000 7f3743400640 1 --2- 172.21.15.96:0/611965308 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3744072200 0x7f37441f4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:14.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:14.595+0000 7f373aa00640 1 --2- 172.21.15.96:0/611965308 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f37440b41e0 0x7f374415f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:14.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:14.595+0000 7f3743e00640 1 -- 172.21.15.96:0/611965308 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f37440b41e0 msgr2=0x7f374415f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:14.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:14.595+0000 7f3743e00640 1 --2- 172.21.15.96:0/611965308 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f37440b41e0 0x7f374415f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:14.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:14.595+0000 7f3743e00640 1 -- 172.21.15.96:0/611965308 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3744072200 msgr2=0x7f37441f4f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:14.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:14.595+0000 7f3743e00640 1 --2- 172.21.15.96:0/611965308 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3744072200 0x7f37441f4f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:14.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:14.595+0000 7f3743e00640 1 -- 172.21.15.96:0/611965308 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37441f8510 con 0x7f3744145320 2024-05-28T06:26:14.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:14.595+0000 7f3743400640 1 --2- 172.21.15.96:0/611965308 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3744072200 0x7f37441f4f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:26:14.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:14.595+0000 7f373aa00640 1 --2- 172.21.15.96:0/611965308 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f37440b41e0 0x7f374415f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:26:14.590 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:14.595+0000 7f3743e00640 1 --2- 172.21.15.96:0/611965308 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3744145320 0x7f37441f7a50 secure :-1 s=READY pgs=2120 cs=0 l=1 rev1=1 crypto rx=0x7f373c02ac90 tx=0x7f373c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:14.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:14.596+0000 7f373be00640 1 -- 172.21.15.96:0/611965308 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f373c041020 con 0x7f3744145320 2024-05-28T06:26:14.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:14.596+0000 7f373be00640 1 -- 172.21.15.96:0/611965308 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f373c007780 con 0x7f3744145320 2024-05-28T06:26:14.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:14.596+0000 7f373be00640 1 -- 172.21.15.96:0/611965308 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f373c036040 con 0x7f3744145320 2024-05-28T06:26:14.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:14.596+0000 7f374a200640 1 -- 172.21.15.96:0/611965308 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37441f87f0 con 0x7f3744145320 2024-05-28T06:26:14.591 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:14.596+0000 7f374a200640 1 -- 172.21.15.96:0/611965308 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37441f4460 con 0x7f3744145320 2024-05-28T06:26:14.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:14.597+0000 7f374a200640 1 -- 172.21.15.96:0/611965308 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37440689e0 con 0x7f3744145320 2024-05-28T06:26:14.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:14.598+0000 7f373be00640 1 -- 172.21.15.96:0/611965308 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 48) ==== 93947+0+0 (secure 0 0 0) 0x7f373c044020 con 0x7f3744145320 2024-05-28T06:26:14.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:14.599+0000 7f373be00640 1 --2- 172.21.15.96:0/611965308 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f372006e900 0x7f3720070dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:14.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:14.599+0000 7f373be00640 1 -- 172.21.15.96:0/611965308 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 5823+0+0 (secure 0 0 0) 0x7f373c0aba30 con 0x7f3744145320 2024-05-28T06:26:14.593 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:14.600+0000 7f3743400640 1 --2- 172.21.15.96:0/611965308 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f372006e900 0x7f3720070dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:14.594 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:14.601+0000 7f3743400640 1 --2- 172.21.15.96:0/611965308 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f372006e900 0x7f3720070dc0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f373400a870 tx=0x7f3734008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:14.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:14.604+0000 7f373be00640 1 -- 172.21.15.96:0/611965308 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f373c04b030 con 0x7f3744145320 2024-05-28T06:26:14.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:14.875+0000 7f374a200640 1 -- 172.21.15.96:0/611965308 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f37440b34e0 con 0x7f3744145320 2024-05-28T06:26:14.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:14.875+0000 7f373be00640 1 -- 172.21.15.96:0/611965308 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1733 (secure 0 0 0) 0x7f373c07b8c0 con 0x7f3744145320 2024-05-28T06:26:14.870 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:26:14.871 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:26:14.871 INFO:teuthology.orchestra.run.smithi096.stdout: "fsid": "e4a831b5-e6d1-47a7-96cb-bf85154ee0d2", 2024-05-28T06:26:14.871 INFO:teuthology.orchestra.run.smithi096.stdout: "health": { 2024-05-28T06:26:14.871 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "HEALTH_OK", 2024-05-28T06:26:14.871 INFO:teuthology.orchestra.run.smithi096.stdout: "checks": {}, 2024-05-28T06:26:14.871 INFO:teuthology.orchestra.run.smithi096.stdout: "mutes": [] 2024-05-28T06:26:14.871 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:26:14.871 INFO:teuthology.orchestra.run.smithi096.stdout: "election_epoch": 8, 2024-05-28T06:26:14.871 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum": [ 2024-05-28T06:26:14.871 INFO:teuthology.orchestra.run.smithi096.stdout: 0, 2024-05-28T06:26:14.871 INFO:teuthology.orchestra.run.smithi096.stdout: 1, 2024-05-28T06:26:14.872 INFO:teuthology.orchestra.run.smithi096.stdout: 2 2024-05-28T06:26:14.872 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:26:14.872 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_names": [ 2024-05-28T06:26:14.872 INFO:teuthology.orchestra.run.smithi096.stdout: "a", 2024-05-28T06:26:14.872 INFO:teuthology.orchestra.run.smithi096.stdout: "b", 2024-05-28T06:26:14.872 INFO:teuthology.orchestra.run.smithi096.stdout: "c" 2024-05-28T06:26:14.872 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:26:14.872 INFO:teuthology.orchestra.run.smithi096.stdout: "quorum_age": 1348, 2024-05-28T06:26:14.873 INFO:teuthology.orchestra.run.smithi096.stdout: "monmap": { 2024-05-28T06:26:14.873 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 1, 2024-05-28T06:26:14.873 INFO:teuthology.orchestra.run.smithi096.stdout: "min_mon_release_name": "squid", 2024-05-28T06:26:14.873 INFO:teuthology.orchestra.run.smithi096.stdout: "num_mons": 3 2024-05-28T06:26:14.873 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:26:14.873 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap": { 2024-05-28T06:26:14.873 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 132, 2024-05-28T06:26:14.873 INFO:teuthology.orchestra.run.smithi096.stdout: "num_osds": 8, 2024-05-28T06:26:14.873 INFO:teuthology.orchestra.run.smithi096.stdout: "num_up_osds": 8, 2024-05-28T06:26:14.873 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_up_since": 1716876238, 2024-05-28T06:26:14.873 INFO:teuthology.orchestra.run.smithi096.stdout: "num_in_osds": 8, 2024-05-28T06:26:14.874 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_in_since": 1716876229, 2024-05-28T06:26:14.874 INFO:teuthology.orchestra.run.smithi096.stdout: "num_remapped_pgs": 0 2024-05-28T06:26:14.874 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:26:14.874 INFO:teuthology.orchestra.run.smithi096.stdout: "pgmap": { 2024-05-28T06:26:14.874 INFO:teuthology.orchestra.run.smithi096.stdout: "pgs_by_state": [ 2024-05-28T06:26:14.874 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:26:14.874 INFO:teuthology.orchestra.run.smithi096.stdout: "state_name": "active+clean", 2024-05-28T06:26:14.874 INFO:teuthology.orchestra.run.smithi096.stdout: "count": 97 2024-05-28T06:26:14.874 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:26:14.874 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:26:14.875 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pgs": 97, 2024-05-28T06:26:14.875 INFO:teuthology.orchestra.run.smithi096.stdout: "num_pools": 3, 2024-05-28T06:26:14.875 INFO:teuthology.orchestra.run.smithi096.stdout: "num_objects": 4663, 2024-05-28T06:26:14.875 INFO:teuthology.orchestra.run.smithi096.stdout: "data_bytes": 19413472918, 2024-05-28T06:26:14.875 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_used": 39973707776, 2024-05-28T06:26:14.875 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_avail": 733120405504, 2024-05-28T06:26:14.875 INFO:teuthology.orchestra.run.smithi096.stdout: "bytes_total": 773094113280 2024-05-28T06:26:14.875 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:26:14.875 INFO:teuthology.orchestra.run.smithi096.stdout: "fsmap": { 2024-05-28T06:26:14.875 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 83, 2024-05-28T06:26:14.875 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 9, 2024-05-28T06:26:14.875 INFO:teuthology.orchestra.run.smithi096.stdout: "up": 1, 2024-05-28T06:26:14.876 INFO:teuthology.orchestra.run.smithi096.stdout: "in": 1, 2024-05-28T06:26:14.876 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 1, 2024-05-28T06:26:14.876 INFO:teuthology.orchestra.run.smithi096.stdout: "by_rank": [ 2024-05-28T06:26:14.876 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:26:14.876 INFO:teuthology.orchestra.run.smithi096.stdout: "filesystem_id": 9, 2024-05-28T06:26:14.876 INFO:teuthology.orchestra.run.smithi096.stdout: "rank": 0, 2024-05-28T06:26:14.876 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "a", 2024-05-28T06:26:14.876 INFO:teuthology.orchestra.run.smithi096.stdout: "status": "up:active", 2024-05-28T06:26:14.876 INFO:teuthology.orchestra.run.smithi096.stdout: "gid": 9849 2024-05-28T06:26:14.876 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:26:14.876 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:26:14.877 INFO:teuthology.orchestra.run.smithi096.stdout: "up:standby": 3 2024-05-28T06:26:14.877 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:26:14.877 INFO:teuthology.orchestra.run.smithi096.stdout: "mgrmap": { 2024-05-28T06:26:14.877 INFO:teuthology.orchestra.run.smithi096.stdout: "available": true, 2024-05-28T06:26:14.877 INFO:teuthology.orchestra.run.smithi096.stdout: "num_standbys": 1, 2024-05-28T06:26:14.877 INFO:teuthology.orchestra.run.smithi096.stdout: "modules": [ 2024-05-28T06:26:14.877 INFO:teuthology.orchestra.run.smithi096.stdout: "iostat", 2024-05-28T06:26:14.877 INFO:teuthology.orchestra.run.smithi096.stdout: "nfs", 2024-05-28T06:26:14.877 INFO:teuthology.orchestra.run.smithi096.stdout: "restful" 2024-05-28T06:26:14.877 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2024-05-28T06:26:14.878 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:26:14.878 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:26:14.878 INFO:teuthology.orchestra.run.smithi096.stdout: "servicemap": { 2024-05-28T06:26:14.878 INFO:teuthology.orchestra.run.smithi096.stdout: "epoch": 25, 2024-05-28T06:26:14.878 INFO:teuthology.orchestra.run.smithi096.stdout: "modified": "2024-05-28T06:25:30.049023+0000", 2024-05-28T06:26:14.878 INFO:teuthology.orchestra.run.smithi096.stdout: "services": {} 2024-05-28T06:26:14.878 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:26:14.878 INFO:teuthology.orchestra.run.smithi096.stdout: "progress_events": {} 2024-05-28T06:26:14.878 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:26:14.879 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:14.878+0000 7f374a200640 1 -- 172.21.15.96:0/611965308 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f372006e900 msgr2=0x7f3720070dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:14.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:14.878+0000 7f374a200640 1 --2- 172.21.15.96:0/611965308 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f372006e900 0x7f3720070dc0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f373400a870 tx=0x7f3734008040 comp rx=0 tx=0).stop 2024-05-28T06:26:14.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:14.878+0000 7f374a200640 1 -- 172.21.15.96:0/611965308 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3744145320 msgr2=0x7f37441f7a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:14.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:14.878+0000 7f374a200640 1 --2- 172.21.15.96:0/611965308 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3744145320 0x7f37441f7a50 secure :-1 s=READY pgs=2120 cs=0 l=1 rev1=1 crypto rx=0x7f373c02ac90 tx=0x7f373c004a40 comp rx=0 tx=0).stop 2024-05-28T06:26:14.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:14.878+0000 7f374a200640 1 -- 172.21.15.96:0/611965308 shutdown_connections 2024-05-28T06:26:14.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:14.878+0000 7f374a200640 1 --2- 172.21.15.96:0/611965308 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f372006e900 0x7f3720070dc0 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:14.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:14.878+0000 7f374a200640 1 --2- 172.21.15.96:0/611965308 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3744145320 0x7f37441f7a50 unknown :-1 s=CLOSED pgs=2120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:14.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:14.878+0000 7f374a200640 1 --2- 172.21.15.96:0/611965308 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f37440b41e0 0x7f374415f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:14.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:14.878+0000 7f374a200640 1 --2- 172.21.15.96:0/611965308 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3744072200 0x7f37441f4f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:14.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:14.878+0000 7f374a200640 1 -- 172.21.15.96:0/611965308 >> 172.21.15.96:0/611965308 conn(0x7f3744067ce0 msgr2=0x7f37441408d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:26:14.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:14.879+0000 7f374a200640 1 -- 172.21.15.96:0/611965308 shutdown_connections 2024-05-28T06:26:14.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:14.879+0000 7f374a200640 1 -- 172.21.15.96:0/611965308 wait complete. 2024-05-28T06:26:14.893 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume ls cephfs --format json 2024-05-28T06:26:15.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.103+0000 7f154e200640 1 Processor -- start 2024-05-28T06:26:15.097 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.103+0000 7f154e200640 1 -- start start 2024-05-28T06:26:15.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.105+0000 7f154e200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f15480ad2a0 0x7f15480ad730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:15.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.105+0000 7f154e200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15480b3d40 con 0x7f1548161e70 2024-05-28T06:26:15.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.105+0000 7f154e200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15480b4020 con 0x7f1548161a90 2024-05-28T06:26:15.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.105+0000 7f154e200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f15480b4300 con 0x7f15480ad2a0 2024-05-28T06:26:15.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.105+0000 7f153fe00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f1548161e70 0x7f15480b32c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:53612/0 (socket says 172.21.15.96:53612) 2024-05-28T06:26:15.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.105+0000 7f153fe00640 1 -- 172.21.15.96:0/1018257109 learned_addr learned my addr 172.21.15.96:0/1018257109 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:26:15.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.106+0000 7f1546a00640 1 -- 172.21.15.96:0/1018257109 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1696928807 0 0) 0x7f15480b3d40 con 0x7f1548161e70 2024-05-28T06:26:15.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.106+0000 7f1547e00640 1 --2- 172.21.15.96:0/1018257109 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f15480ad2a0 0x7f15480ad730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:15.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.106+0000 7f1546a00640 1 -- 172.21.15.96:0/1018257109 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f152c003520 con 0x7f1548161e70 2024-05-28T06:26:15.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.106+0000 7f1546a00640 1 -- 172.21.15.96:0/1018257109 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1542341004 0 0) 0x7f152c003520 con 0x7f1548161e70 2024-05-28T06:26:15.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.106+0000 7f1546a00640 1 -- 172.21.15.96:0/1018257109 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f15480ad2a0 msgr2=0x7f15480ad730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:15.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.106+0000 7f1546a00640 1 --2- 172.21.15.96:0/1018257109 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f15480ad2a0 0x7f15480ad730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:15.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.106+0000 7f1546a00640 1 -- 172.21.15.96:0/1018257109 >> v1:172.21.15.123:6789/0 conn(0x7f1548161a90 legacy=0x7f154806e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:15.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.106+0000 7f1546a00640 1 -- 172.21.15.96:0/1018257109 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f154806dc70 con 0x7f1548161e70 2024-05-28T06:26:15.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.106+0000 7f1546a00640 1 -- 172.21.15.96:0/1018257109 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f1530001670 con 0x7f1548161e70 2024-05-28T06:26:15.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.106+0000 7f1546a00640 1 -- 172.21.15.96:0/1018257109 >> v1:172.21.15.96:6789/0 conn(0x7f1548161e70 legacy=0x7f15480b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:15.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.106+0000 7f1546a00640 1 --2- 172.21.15.96:0/1018257109 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f15480ad2a0 0x7f152c0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:15.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.106+0000 7f1546a00640 1 --2- 172.21.15.96:0/1018257109 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f152c003fb0 0x7f152c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:15.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.106+0000 7f1546a00640 1 --2- 172.21.15.96:0/1018257109 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f152c004d40 0x7f152c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:15.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.106+0000 7f1546a00640 1 -- 172.21.15.96:0/1018257109 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f15480b4300 con 0x7f152c003fb0 2024-05-28T06:26:15.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.106+0000 7f1546a00640 1 -- 172.21.15.96:0/1018257109 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f152c00bb00 con 0x7f152c004d40 2024-05-28T06:26:15.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.106+0000 7f1546a00640 1 -- 172.21.15.96:0/1018257109 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f152c00bd00 con 0x7f15480ad2a0 2024-05-28T06:26:15.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.107+0000 7f153fe00640 1 --2- 172.21.15.96:0/1018257109 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f152c003fb0 0x7f152c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:15.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.107+0000 7f1547400640 1 --2- 172.21.15.96:0/1018257109 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f15480ad2a0 0x7f152c0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:15.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.107+0000 7f1547e00640 1 --2- 172.21.15.96:0/1018257109 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f152c004d40 0x7f152c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:15.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.107+0000 7f153fe00640 1 -- 172.21.15.96:0/1018257109 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f15480ad2a0 msgr2=0x7f152c0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:15.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.107+0000 7f153fe00640 1 --2- 172.21.15.96:0/1018257109 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f15480ad2a0 0x7f152c0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:15.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.107+0000 7f153fe00640 1 -- 172.21.15.96:0/1018257109 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f152c004d40 msgr2=0x7f152c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:15.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.107+0000 7f153fe00640 1 --2- 172.21.15.96:0/1018257109 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f152c004d40 0x7f152c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:15.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.107+0000 7f153fe00640 1 -- 172.21.15.96:0/1018257109 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f152c003520 con 0x7f152c003fb0 2024-05-28T06:26:15.103 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.107+0000 7f1547e00640 1 --2- 172.21.15.96:0/1018257109 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f152c004d40 0x7f152c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:26:15.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.108+0000 7f153fe00640 1 --2- 172.21.15.96:0/1018257109 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f152c003fb0 0x7f152c004800 secure :-1 s=READY pgs=2122 cs=0 l=1 rev1=1 crypto rx=0x7f1530007fd0 tx=0x7f152c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=134822137c4c36d server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:15.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.108+0000 7f1546a00640 1 -- 172.21.15.96:0/1018257109 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1530007c20 con 0x7f152c003fb0 2024-05-28T06:26:15.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.109+0000 7f1546a00640 1 -- 172.21.15.96:0/1018257109 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1530005af0 con 0x7f152c003fb0 2024-05-28T06:26:15.104 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.109+0000 7f1546a00640 1 -- 172.21.15.96:0/1018257109 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1530007070 con 0x7f152c003fb0 2024-05-28T06:26:15.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.109+0000 7f154e200640 1 -- 172.21.15.96:0/1018257109 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f152c003fb0 msgr2=0x7f152c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:15.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.109+0000 7f154e200640 1 --2- 172.21.15.96:0/1018257109 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f152c003fb0 0x7f152c004800 secure :-1 s=READY pgs=2122 cs=0 l=1 rev1=1 crypto rx=0x7f1530007fd0 tx=0x7f152c005830 comp rx=0 tx=0).stop 2024-05-28T06:26:15.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.109+0000 7f1547e00640 1 -- 172.21.15.96:0/1018257109 reap_dead start 2024-05-28T06:26:15.105 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.109+0000 7f154e200640 1 -- 172.21.15.96:0/1018257109 shutdown_connections 2024-05-28T06:26:15.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.109+0000 7f154e200640 1 -- 172.21.15.96:0/1018257109 >> 172.21.15.96:0/1018257109 conn(0x7f154806bd20 msgr2=0x7f154806c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:26:15.106 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.109+0000 7f154e200640 1 -- 172.21.15.96:0/1018257109 shutdown_connections 2024-05-28T06:26:15.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.109+0000 7f154e200640 1 -- 172.21.15.96:0/1018257109 wait complete. 2024-05-28T06:26:15.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.111+0000 7f154e200640 1 Processor -- start 2024-05-28T06:26:15.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.111+0000 7f154e200640 1 -- start start 2024-05-28T06:26:15.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.112+0000 7f154e200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f152c003fb0 0x7f1548142e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:15.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.112+0000 7f154e200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f154814f2d0 0x7f15481f7380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:15.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.112+0000 7f154e200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1548152100 0x7f15480b0b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:15.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.112+0000 7f154e200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f154806eb80 con 0x7f154814f2d0 2024-05-28T06:26:15.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.112+0000 7f154e200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f15480b5020 con 0x7f1548152100 2024-05-28T06:26:15.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.112+0000 7f154e200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f15480b39d0 con 0x7f152c003fb0 2024-05-28T06:26:15.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.112+0000 7f153fe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f154814f2d0 0x7f15481f7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:15.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.113+0000 7f153fe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f154814f2d0 0x7f15481f7380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35608/0 (socket says 172.21.15.96:35608) 2024-05-28T06:26:15.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.113+0000 7f153fe00640 1 -- 172.21.15.96:0/360181867 learned_addr learned my addr 172.21.15.96:0/360181867 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:26:15.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.113+0000 7f1547400640 1 --2- 172.21.15.96:0/360181867 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f152c003fb0 0x7f1548142e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:15.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.113+0000 7f1547e00640 1 --2- 172.21.15.96:0/360181867 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1548152100 0x7f15480b0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:15.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.113+0000 7f153fe00640 1 -- 172.21.15.96:0/360181867 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f152c003fb0 msgr2=0x7f1548142e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:15.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.113+0000 7f153fe00640 1 --2- 172.21.15.96:0/360181867 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f152c003fb0 0x7f1548142e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:15.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.113+0000 7f153fe00640 1 -- 172.21.15.96:0/360181867 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1548152100 msgr2=0x7f15480b0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:15.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.113+0000 7f153fe00640 1 --2- 172.21.15.96:0/360181867 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1548152100 0x7f15480b0b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:15.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.113+0000 7f153fe00640 1 -- 172.21.15.96:0/360181867 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f154814f7d0 con 0x7f154814f2d0 2024-05-28T06:26:15.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.113+0000 7f1547e00640 1 --2- 172.21.15.96:0/360181867 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1548152100 0x7f15480b0b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:26:15.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.113+0000 7f153fe00640 1 --2- 172.21.15.96:0/360181867 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f154814f2d0 0x7f15481f7380 secure :-1 s=READY pgs=2123 cs=0 l=1 rev1=1 crypto rx=0x7f153000c0e0 tx=0x7f15480b41f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:15.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.114+0000 7f153f400640 1 -- 172.21.15.96:0/360181867 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1530001630 con 0x7f154814f2d0 2024-05-28T06:26:15.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.114+0000 7f153f400640 1 -- 172.21.15.96:0/360181867 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1530042070 con 0x7f154814f2d0 2024-05-28T06:26:15.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.114+0000 7f153f400640 1 -- 172.21.15.96:0/360181867 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1530007290 con 0x7f154814f2d0 2024-05-28T06:26:15.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.114+0000 7f154e200640 1 -- 172.21.15.96:0/360181867 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15481541a0 con 0x7f154814f2d0 2024-05-28T06:26:15.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.114+0000 7f154e200640 1 -- 172.21.15.96:0/360181867 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1548154760 con 0x7f154814f2d0 2024-05-28T06:26:15.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.115+0000 7f154e200640 1 -- 172.21.15.96:0/360181867 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15480689e0 con 0x7f154814f2d0 2024-05-28T06:26:15.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.116+0000 7f153f400640 1 -- 172.21.15.96:0/360181867 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 48) ==== 93947+0+0 (secure 0 0 0) 0x7f1530007430 con 0x7f154814f2d0 2024-05-28T06:26:15.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.117+0000 7f153f400640 1 --2- 172.21.15.96:0/360181867 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f152406e980 0x7f1524070e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:15.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.117+0000 7f153f400640 1 -- 172.21.15.96:0/360181867 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 5823+0+0 (secure 0 0 0) 0x7f15300aad60 con 0x7f154814f2d0 2024-05-28T06:26:15.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.118+0000 7f1547400640 1 --2- 172.21.15.96:0/360181867 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f152406e980 0x7f1524070e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:15.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.119+0000 7f1547400640 1 --2- 172.21.15.96:0/360181867 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f152406e980 0x7f1524070e40 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f1538000c00 tx=0x7f1538009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:15.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.121+0000 7f153f400640 1 -- 172.21.15.96:0/360181867 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f153007ac70 con 0x7f154814f2d0 2024-05-28T06:26:15.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.352+0000 7f154e200640 1 -- 172.21.15.96:0/360181867 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume ls", "vol_name": "cephfs", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f15480b0340 con 0x7f152406e980 2024-05-28T06:26:15.367 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.373+0000 7f153f400640 1 -- 172.21.15.96:0/360181867 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+381 (secure 0 0 0) 0x7f15480b0340 con 0x7f152406e980 2024-05-28T06:26:15.367 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:26:15.367 INFO:teuthology.orchestra.run.smithi096.stdout:[ 2024-05-28T06:26:15.367 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:26:15.367 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "subvol_clone10996" 2024-05-28T06:26:15.367 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:26:15.367 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:26:15.367 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "sv1" 2024-05-28T06:26:15.368 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:26:15.368 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:26:15.368 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "subvol_clone10997" 2024-05-28T06:26:15.368 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:26:15.368 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:26:15.368 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "subvol_clone10991" 2024-05-28T06:26:15.368 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:26:15.368 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:26:15.368 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "subvol_clone10993" 2024-05-28T06:26:15.368 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:26:15.368 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:26:15.368 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "subvol_clone10992" 2024-05-28T06:26:15.368 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:26:15.369 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:26:15.369 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "subvol_clone10995" 2024-05-28T06:26:15.369 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-05-28T06:26:15.369 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:26:15.369 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "subvol_clone10994" 2024-05-28T06:26:15.369 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:26:15.369 INFO:teuthology.orchestra.run.smithi096.stdout:] 2024-05-28T06:26:15.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.376+0000 7f154e200640 1 -- 172.21.15.96:0/360181867 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f152406e980 msgr2=0x7f1524070e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:15.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.376+0000 7f154e200640 1 --2- 172.21.15.96:0/360181867 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f152406e980 0x7f1524070e40 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f1538000c00 tx=0x7f1538009040 comp rx=0 tx=0).stop 2024-05-28T06:26:15.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.376+0000 7f154e200640 1 -- 172.21.15.96:0/360181867 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f154814f2d0 msgr2=0x7f15481f7380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:15.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.376+0000 7f154e200640 1 --2- 172.21.15.96:0/360181867 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f154814f2d0 0x7f15481f7380 secure :-1 s=READY pgs=2123 cs=0 l=1 rev1=1 crypto rx=0x7f153000c0e0 tx=0x7f15480b41f0 comp rx=0 tx=0).stop 2024-05-28T06:26:15.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.376+0000 7f154e200640 1 -- 172.21.15.96:0/360181867 shutdown_connections 2024-05-28T06:26:15.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.376+0000 7f154e200640 1 --2- 172.21.15.96:0/360181867 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f152406e980 0x7f1524070e40 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:15.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.376+0000 7f154e200640 1 --2- 172.21.15.96:0/360181867 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1548152100 0x7f15480b0b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:15.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.376+0000 7f154e200640 1 --2- 172.21.15.96:0/360181867 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f154814f2d0 0x7f15481f7380 unknown :-1 s=CLOSED pgs=2123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:15.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.376+0000 7f154e200640 1 --2- 172.21.15.96:0/360181867 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f152c003fb0 0x7f1548142e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:15.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.376+0000 7f154e200640 1 -- 172.21.15.96:0/360181867 >> 172.21.15.96:0/360181867 conn(0x7f154806bd20 msgr2=0x7f15480ad710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:26:15.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.376+0000 7f154e200640 1 -- 172.21.15.96:0/360181867 shutdown_connections 2024-05-28T06:26:15.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.376+0000 7f154e200640 1 -- 172.21.15.96:0/360181867 wait complete. 2024-05-28T06:26:15.389 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone10996 2024-05-28T06:26:15.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.598+0000 7f5dc1a00640 1 Processor -- start 2024-05-28T06:26:15.592 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.598+0000 7f5dc1a00640 1 -- start start 2024-05-28T06:26:15.594 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.600+0000 7f5dc1a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f5dbc0d0430 0x7f5dbc0d0860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:15.594 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.600+0000 7f5dc1a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5dbc15f0c0 0x7f5dbc15f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:15.594 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.600+0000 7f5dc1a00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5dbc15bb30 con 0x7f5dbc0ce560 2024-05-28T06:26:15.594 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.600+0000 7f5dc1a00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f5dbc0b5290 con 0x7f5dbc0d0430 2024-05-28T06:26:15.594 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.600+0000 7f5dc1a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f5dbc0d0da0 con 0x7f5dbc15f0c0 2024-05-28T06:26:15.595 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.600+0000 7f5dbb400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f5dbc0d0430 0x7f5dbc0d0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:15.595 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.600+0000 7f5dbb400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f5dbc0d0430 0x7f5dbc0d0860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35622/0 (socket says 172.21.15.96:35622) 2024-05-28T06:26:15.595 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.600+0000 7f5dbb400640 1 -- 172.21.15.96:0/3507276847 learned_addr learned my addr 172.21.15.96:0/3507276847 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:26:15.595 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.601+0000 7f5dc1000640 1 --2- 172.21.15.96:0/3507276847 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5dbc15f0c0 0x7f5dbc15f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:15.595 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.601+0000 7f5dbb400640 1 -- 172.21.15.96:0/3507276847 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5dbc15f0c0 msgr2=0x7f5dbc15f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:15.596 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.601+0000 7f5dbb400640 1 --2- 172.21.15.96:0/3507276847 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5dbc15f0c0 0x7f5dbc15f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:15.596 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.601+0000 7f5dbb400640 1 -- 172.21.15.96:0/3507276847 >> v1:172.21.15.123:6789/0 conn(0x7f5dbc0ce560 legacy=0x7f5dbc0b4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:15.596 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.601+0000 7f5dbb400640 1 -- 172.21.15.96:0/3507276847 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dbc0b3ba0 con 0x7f5dbc0d0430 2024-05-28T06:26:15.596 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.602+0000 7f5dbb400640 1 --2- 172.21.15.96:0/3507276847 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5dbc0d0430 0x7f5dbc0d0860 secure :-1 s=READY pgs=2124 cs=0 l=1 rev1=1 crypto rx=0x7f5da800bb10 tx=0x7f5da802c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd32599f746e5c75 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:15.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.602+0000 7f5dbaa00640 1 -- 172.21.15.96:0/3507276847 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5da8036020 con 0x7f5dbc0d0430 2024-05-28T06:26:15.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.602+0000 7f5dbaa00640 1 -- 172.21.15.96:0/3507276847 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5da8032820 con 0x7f5dbc0d0430 2024-05-28T06:26:15.597 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.602+0000 7f5dbaa00640 1 -- 172.21.15.96:0/3507276847 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5da8032bf0 con 0x7f5dbc0d0430 2024-05-28T06:26:15.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.603+0000 7f5dc1a00640 1 -- 172.21.15.96:0/3507276847 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5dbc0d0430 msgr2=0x7f5dbc0d0860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:15.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.603+0000 7f5dc1a00640 1 --2- 172.21.15.96:0/3507276847 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5dbc0d0430 0x7f5dbc0d0860 secure :-1 s=READY pgs=2124 cs=0 l=1 rev1=1 crypto rx=0x7f5da800bb10 tx=0x7f5da802c920 comp rx=0 tx=0).stop 2024-05-28T06:26:15.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.603+0000 7f5dc1a00640 1 -- 172.21.15.96:0/3507276847 shutdown_connections 2024-05-28T06:26:15.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.603+0000 7f5dc1a00640 1 --2- 172.21.15.96:0/3507276847 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5dbc15f0c0 0x7f5dbc15f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:15.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.603+0000 7f5dc1a00640 1 --2- 172.21.15.96:0/3507276847 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5dbc0d0430 0x7f5dbc0d0860 unknown :-1 s=CLOSED pgs=2124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:15.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.603+0000 7f5dc1a00640 1 -- 172.21.15.96:0/3507276847 >> 172.21.15.96:0/3507276847 conn(0x7f5dbc06bae0 msgr2=0x7f5dbc06bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:26:15.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.603+0000 7f5dc1a00640 1 -- 172.21.15.96:0/3507276847 shutdown_connections 2024-05-28T06:26:15.598 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.603+0000 7f5dc1a00640 1 -- 172.21.15.96:0/3507276847 wait complete. 2024-05-28T06:26:15.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.605+0000 7f5dc1a00640 1 Processor -- start 2024-05-28T06:26:15.599 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.605+0000 7f5dc1a00640 1 -- start start 2024-05-28T06:26:15.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.606+0000 7f5dc1a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5dbc0ce560 0x7f5dbc1f49a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:15.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.606+0000 7f5dc1a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5dbc0d0430 0x7f5dbc1586e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:15.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.606+0000 7f5dc1a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5dbc15f0c0 0x7f5dbc15de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:15.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.606+0000 7f5dc1a00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f5dbc15c390 con 0x7f5dbc0d0430 2024-05-28T06:26:15.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.606+0000 7f5dc1a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5dbc15c0b0 con 0x7f5dbc0ce560 2024-05-28T06:26:15.601 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.606+0000 7f5dc1a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f5dbc15c670 con 0x7f5dbc15f0c0 2024-05-28T06:26:15.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.607+0000 7f5dbb400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5dbc0d0430 0x7f5dbc1586e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:15.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.607+0000 7f5dbb400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5dbc0d0430 0x7f5dbc1586e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35624/0 (socket says 172.21.15.96:35624) 2024-05-28T06:26:15.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.607+0000 7f5dbb400640 1 -- 172.21.15.96:0/4256729146 learned_addr learned my addr 172.21.15.96:0/4256729146 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:26:15.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.607+0000 7f5dc1000640 1 --2- 172.21.15.96:0/4256729146 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5dbc15f0c0 0x7f5dbc15de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:15.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.607+0000 7f5dbbe00640 1 --2- 172.21.15.96:0/4256729146 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5dbc0ce560 0x7f5dbc1f49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:15.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.607+0000 7f5dbb400640 1 -- 172.21.15.96:0/4256729146 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5dbc15f0c0 msgr2=0x7f5dbc15de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:15.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.607+0000 7f5dbb400640 1 --2- 172.21.15.96:0/4256729146 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5dbc15f0c0 0x7f5dbc15de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:15.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.607+0000 7f5dbb400640 1 -- 172.21.15.96:0/4256729146 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5dbc0ce560 msgr2=0x7f5dbc1f49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:15.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.607+0000 7f5dbb400640 1 --2- 172.21.15.96:0/4256729146 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5dbc0ce560 0x7f5dbc1f49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:15.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.607+0000 7f5dbb400640 1 -- 172.21.15.96:0/4256729146 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dbc1f4ee0 con 0x7f5dbc0d0430 2024-05-28T06:26:15.603 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.607+0000 7f5dbb400640 1 --2- 172.21.15.96:0/4256729146 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5dbc0d0430 0x7f5dbc1586e0 secure :-1 s=READY pgs=2125 cs=0 l=1 rev1=1 crypto rx=0x7f5da802c960 tx=0x7f5da8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:15.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.608+0000 7f5db3e00640 1 -- 172.21.15.96:0/4256729146 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5da8042070 con 0x7f5dbc0d0430 2024-05-28T06:26:15.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.608+0000 7f5db3e00640 1 -- 172.21.15.96:0/4256729146 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5da8031cb0 con 0x7f5dbc0d0430 2024-05-28T06:26:15.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.608+0000 7f5dc1a00640 1 -- 172.21.15.96:0/4256729146 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5dbc0b0d20 con 0x7f5dbc0d0430 2024-05-28T06:26:15.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.608+0000 7f5db3e00640 1 -- 172.21.15.96:0/4256729146 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5da8036040 con 0x7f5dbc0d0430 2024-05-28T06:26:15.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.608+0000 7f5dc1a00640 1 -- 172.21.15.96:0/4256729146 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5dbc0b12e0 con 0x7f5dbc0d0430 2024-05-28T06:26:15.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.609+0000 7f5dc1a00640 1 -- 172.21.15.96:0/4256729146 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5dbc0689e0 con 0x7f5dbc0d0430 2024-05-28T06:26:15.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.610+0000 7f5db3e00640 1 -- 172.21.15.96:0/4256729146 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 48) ==== 93947+0+0 (secure 0 0 0) 0x7f5da8045070 con 0x7f5dbc0d0430 2024-05-28T06:26:15.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.611+0000 7f5db3e00640 1 --2- 172.21.15.96:0/4256729146 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5d9c06e900 0x7f5d9c070dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:15.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.611+0000 7f5db3e00640 1 -- 172.21.15.96:0/4256729146 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 5823+0+0 (secure 0 0 0) 0x7f5da80ab770 con 0x7f5dbc0d0430 2024-05-28T06:26:15.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.612+0000 7f5dbbe00640 1 --2- 172.21.15.96:0/4256729146 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5d9c06e900 0x7f5d9c070dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:15.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.613+0000 7f5dbbe00640 1 --2- 172.21.15.96:0/4256729146 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5d9c06e900 0x7f5d9c070dc0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f5dac007fd0 tx=0x7f5dac009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:15.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.615+0000 7f5db3e00640 1 -- 172.21.15.96:0/4256729146 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5da802f070 con 0x7f5dbc0d0430 2024-05-28T06:26:15.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.840+0000 7f5dc1a00640 1 -- 172.21.15.96:0/4256729146 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone10996", "target": ["mon-mgr", ""]}) -- 0x7f5dbc0ad6d0 con 0x7f5d9c06e900 2024-05-28T06:26:15.880 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.886+0000 7f5db3e00640 1 -- 172.21.15.96:0/4256729146 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f5dbc0ad6d0 con 0x7f5d9c06e900 2024-05-28T06:26:15.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.889+0000 7f5dc1a00640 1 -- 172.21.15.96:0/4256729146 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5d9c06e900 msgr2=0x7f5d9c070dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:15.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.889+0000 7f5dc1a00640 1 --2- 172.21.15.96:0/4256729146 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5d9c06e900 0x7f5d9c070dc0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f5dac007fd0 tx=0x7f5dac009040 comp rx=0 tx=0).stop 2024-05-28T06:26:15.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.889+0000 7f5dc1a00640 1 -- 172.21.15.96:0/4256729146 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5dbc0d0430 msgr2=0x7f5dbc1586e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:15.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.889+0000 7f5dc1a00640 1 --2- 172.21.15.96:0/4256729146 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5dbc0d0430 0x7f5dbc1586e0 secure :-1 s=READY pgs=2125 cs=0 l=1 rev1=1 crypto rx=0x7f5da802c960 tx=0x7f5da8004730 comp rx=0 tx=0).stop 2024-05-28T06:26:15.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.889+0000 7f5dc1a00640 1 -- 172.21.15.96:0/4256729146 shutdown_connections 2024-05-28T06:26:15.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.889+0000 7f5dc1a00640 1 --2- 172.21.15.96:0/4256729146 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5d9c06e900 0x7f5d9c070dc0 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:15.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.889+0000 7f5dc1a00640 1 --2- 172.21.15.96:0/4256729146 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5dbc15f0c0 0x7f5dbc15de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:15.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.889+0000 7f5dc1a00640 1 --2- 172.21.15.96:0/4256729146 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5dbc0d0430 0x7f5dbc1586e0 unknown :-1 s=CLOSED pgs=2125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:15.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.889+0000 7f5dc1a00640 1 --2- 172.21.15.96:0/4256729146 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5dbc0ce560 0x7f5dbc1f49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:15.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.889+0000 7f5dc1a00640 1 -- 172.21.15.96:0/4256729146 >> 172.21.15.96:0/4256729146 conn(0x7f5dbc06bae0 msgr2=0x7f5dbc06bec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:26:15.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.889+0000 7f5dc1a00640 1 -- 172.21.15.96:0/4256729146 shutdown_connections 2024-05-28T06:26:15.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:15.889+0000 7f5dc1a00640 1 -- 172.21.15.96:0/4256729146 wait complete. 2024-05-28T06:26:15.893 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:26:15.897+0000 7fea5da00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:26:15.894 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:26:15.897+0000 7fea5da00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:26:15.894 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:26:15.897+0000 7fea5da00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:26:15.894 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:26:15.897+0000 7fea5da00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:26:15.894 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:26:15.897+0000 7fea5da00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:26:15.905 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot ls cephfs sv1 --format json 2024-05-28T06:26:15.919 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:26:15.925+0000 7fea5bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:26:15.919 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:26:15.925+0000 7fea5bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:26:15.919 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:26:15.925+0000 7fea5bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:26:15.919 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:26:15.925+0000 7fea5bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:26:15.920 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:26:15.925+0000 7fea5bc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-28T06:26:16.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.113+0000 7f6f76a00640 1 Processor -- start 2024-05-28T06:26:16.107 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.113+0000 7f6f76a00640 1 -- start start 2024-05-28T06:26:16.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.115+0000 7f6f76a00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f6f7015fa00 0x7f6f7006e970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:16.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.115+0000 7f6f76a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f6f701600f0 0x7f6f700d0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:16.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.115+0000 7f6f76a00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f7006fa00 con 0x7f6f700d0970 2024-05-28T06:26:16.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.115+0000 7f6f76a00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f6f700d0dc0 con 0x7f6f701600f0 2024-05-28T06:26:16.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.115+0000 7f6f76a00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f6f7006eeb0 con 0x7f6f7015fa00 2024-05-28T06:26:16.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.115+0000 7f6f74c00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f6f701600f0 0x7f6f700d0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:16.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.116+0000 7f6f74c00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f6f701600f0 0x7f6f700d0430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35630/0 (socket says 172.21.15.96:35630) 2024-05-28T06:26:16.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.116+0000 7f6f74c00640 1 -- 172.21.15.96:0/3583303249 learned_addr learned my addr 172.21.15.96:0/3583303249 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:26:16.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.116+0000 7f6f75600640 1 --2- 172.21.15.96:0/3583303249 >> v2:172.21.15.123:3300/0 conn(0x7f6f7015fa00 0x7f6f7006e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:16.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.116+0000 7f6f74c00640 1 -- 172.21.15.96:0/3583303249 >> v2:172.21.15.123:3300/0 conn(0x7f6f7015fa00 msgr2=0x7f6f7006e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:16.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.116+0000 7f6f74c00640 1 --2- 172.21.15.96:0/3583303249 >> v2:172.21.15.123:3300/0 conn(0x7f6f7015fa00 0x7f6f7006e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:16.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.116+0000 7f6f74c00640 1 -- 172.21.15.96:0/3583303249 >> v1:172.21.15.123:6789/0 conn(0x7f6f700d0970 legacy=0x7f6f700b3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:16.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.116+0000 7f6f74c00640 1 -- 172.21.15.96:0/3583303249 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f7006f160 con 0x7f6f701600f0 2024-05-28T06:26:16.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.117+0000 7f6f74c00640 1 --2- 172.21.15.96:0/3583303249 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6f701600f0 0x7f6f700d0430 secure :-1 s=READY pgs=2126 cs=0 l=1 rev1=1 crypto rx=0x7f6f7006d980 tx=0x7f6f5c02a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2c36a110bc716a7 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:16.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.117+0000 7f6f6fe00640 1 -- 172.21.15.96:0/3583303249 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f5c037070 con 0x7f6f701600f0 2024-05-28T06:26:16.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.117+0000 7f6f6fe00640 1 -- 172.21.15.96:0/3583303249 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f5c032440 con 0x7f6f701600f0 2024-05-28T06:26:16.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.117+0000 7f6f6fe00640 1 -- 172.21.15.96:0/3583303249 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f5c032810 con 0x7f6f701600f0 2024-05-28T06:26:16.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.118+0000 7f6f76a00640 1 -- 172.21.15.96:0/3583303249 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6f701600f0 msgr2=0x7f6f700d0430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:16.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.118+0000 7f6f76a00640 1 --2- 172.21.15.96:0/3583303249 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6f701600f0 0x7f6f700d0430 secure :-1 s=READY pgs=2126 cs=0 l=1 rev1=1 crypto rx=0x7f6f7006d980 tx=0x7f6f5c02a180 comp rx=0 tx=0).stop 2024-05-28T06:26:16.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.118+0000 7f6f76a00640 1 -- 172.21.15.96:0/3583303249 shutdown_connections 2024-05-28T06:26:16.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.118+0000 7f6f76a00640 1 --2- 172.21.15.96:0/3583303249 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6f701600f0 0x7f6f700d0430 unknown :-1 s=CLOSED pgs=2126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:16.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.118+0000 7f6f76a00640 1 --2- 172.21.15.96:0/3583303249 >> v2:172.21.15.123:3300/0 conn(0x7f6f7015fa00 0x7f6f7006e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:16.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.118+0000 7f6f76a00640 1 -- 172.21.15.96:0/3583303249 >> 172.21.15.96:0/3583303249 conn(0x7f6f7006bae0 msgr2=0x7f6f7006bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:26:16.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.118+0000 7f6f76a00640 1 -- 172.21.15.96:0/3583303249 shutdown_connections 2024-05-28T06:26:16.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.118+0000 7f6f76a00640 1 -- 172.21.15.96:0/3583303249 wait complete. 2024-05-28T06:26:16.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.120+0000 7f6f76a00640 1 Processor -- start 2024-05-28T06:26:16.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.120+0000 7f6f76a00640 1 -- start start 2024-05-28T06:26:16.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.121+0000 7f6f76a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6f700d0970 0x7f6f700cddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:16.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.121+0000 7f6f76a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6f7015fa00 0x7f6f700adab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:16.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.121+0000 7f6f76a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f701600f0 0x7f6f701e9ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:16.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.121+0000 7f6f76a00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f6f70071c70 con 0x7f6f700d0970 2024-05-28T06:26:16.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.121+0000 7f6f76a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f6f7006f720 con 0x7f6f701600f0 2024-05-28T06:26:16.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.121+0000 7f6f76a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f6f7015fde0 con 0x7f6f7015fa00 2024-05-28T06:26:16.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.121+0000 7f6f75600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6f700d0970 0x7f6f700cddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:16.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.121+0000 7f6f75600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6f700d0970 0x7f6f700cddc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35638/0 (socket says 172.21.15.96:35638) 2024-05-28T06:26:16.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.121+0000 7f6f75600640 1 -- 172.21.15.96:0/1966859903 learned_addr learned my addr 172.21.15.96:0/1966859903 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:26:16.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.122+0000 7f6f76000640 1 --2- 172.21.15.96:0/1966859903 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f701600f0 0x7f6f701e9ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:16.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.122+0000 7f6f75600640 1 -- 172.21.15.96:0/1966859903 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6f7015fa00 msgr2=0x7f6f700adab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:16.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.122+0000 7f6f74c00640 1 --2- 172.21.15.96:0/1966859903 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6f7015fa00 0x7f6f700adab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:16.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.122+0000 7f6f75600640 1 --2- 172.21.15.96:0/1966859903 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6f7015fa00 0x7f6f700adab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:16.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.122+0000 7f6f75600640 1 -- 172.21.15.96:0/1966859903 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f701600f0 msgr2=0x7f6f701e9ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:16.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.122+0000 7f6f75600640 1 --2- 172.21.15.96:0/1966859903 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f701600f0 0x7f6f701e9ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:16.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.122+0000 7f6f75600640 1 -- 172.21.15.96:0/1966859903 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f70148050 con 0x7f6f700d0970 2024-05-28T06:26:16.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.122+0000 7f6f76000640 1 --2- 172.21.15.96:0/1966859903 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f701600f0 0x7f6f701e9ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:26:16.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.122+0000 7f6f75600640 1 --2- 172.21.15.96:0/1966859903 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6f700d0970 0x7f6f700cddc0 secure :-1 s=READY pgs=2127 cs=0 l=1 rev1=1 crypto rx=0x7f6f6400d5f0 tx=0x7f6f6400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:16.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.123+0000 7f6f6d600640 1 -- 172.21.15.96:0/1966859903 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f640125f0 con 0x7f6f700d0970 2024-05-28T06:26:16.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.123+0000 7f6f6d600640 1 -- 172.21.15.96:0/1966859903 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f64019070 con 0x7f6f700d0970 2024-05-28T06:26:16.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.123+0000 7f6f6d600640 1 -- 172.21.15.96:0/1966859903 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f640129d0 con 0x7f6f700d0970 2024-05-28T06:26:16.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.123+0000 7f6f76a00640 1 -- 172.21.15.96:0/1966859903 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f70148330 con 0x7f6f700d0970 2024-05-28T06:26:16.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.123+0000 7f6f76a00640 1 -- 172.21.15.96:0/1966859903 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f701488f0 con 0x7f6f700d0970 2024-05-28T06:26:16.119 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.124+0000 7f6f76a00640 1 -- 172.21.15.96:0/1966859903 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f700689e0 con 0x7f6f700d0970 2024-05-28T06:26:16.119 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.125+0000 7f6f6d600640 1 -- 172.21.15.96:0/1966859903 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 48) ==== 93947+0+0 (secure 0 0 0) 0x7f6f64012b70 con 0x7f6f700d0970 2024-05-28T06:26:16.120 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.126+0000 7f6f6d600640 1 --2- 172.21.15.96:0/1966859903 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6f5006e930 0x7f6f50070df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:16.120 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.126+0000 7f6f6d600640 1 -- 172.21.15.96:0/1966859903 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 5823+0+0 (secure 0 0 0) 0x7f6f6408b000 con 0x7f6f700d0970 2024-05-28T06:26:16.120 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.127+0000 7f6f74c00640 1 --2- 172.21.15.96:0/1966859903 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6f5006e930 0x7f6f50070df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:16.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.128+0000 7f6f74c00640 1 --2- 172.21.15.96:0/1966859903 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6f5006e930 0x7f6f50070df0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f6f5c034000 tx=0x7f6f5c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:16.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.130+0000 7f6f6d600640 1 -- 172.21.15.96:0/1966859903 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6f6405af30 con 0x7f6f700d0970 2024-05-28T06:26:16.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.364+0000 7f6f76a00640 1 -- 172.21.15.96:0/1966859903 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot ls", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f6f700b0360 con 0x7f6f5006e930 2024-05-28T06:26:16.390 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.396+0000 7f6f6d600640 1 -- 172.21.15.96:0/1966859903 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f6f700b0360 con 0x7f6f5006e930 2024-05-28T06:26:16.390 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:26:16.390 INFO:teuthology.orchestra.run.smithi096.stdout:[ 2024-05-28T06:26:16.390 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-05-28T06:26:16.391 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "ss1" 2024-05-28T06:26:16.391 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-05-28T06:26:16.391 INFO:teuthology.orchestra.run.smithi096.stdout:] 2024-05-28T06:26:16.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.398+0000 7f6f76a00640 1 -- 172.21.15.96:0/1966859903 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6f5006e930 msgr2=0x7f6f50070df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:16.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.398+0000 7f6f76a00640 1 --2- 172.21.15.96:0/1966859903 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6f5006e930 0x7f6f50070df0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f6f5c034000 tx=0x7f6f5c02e000 comp rx=0 tx=0).stop 2024-05-28T06:26:16.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.398+0000 7f6f76a00640 1 -- 172.21.15.96:0/1966859903 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6f700d0970 msgr2=0x7f6f700cddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:16.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.398+0000 7f6f76a00640 1 --2- 172.21.15.96:0/1966859903 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6f700d0970 0x7f6f700cddc0 secure :-1 s=READY pgs=2127 cs=0 l=1 rev1=1 crypto rx=0x7f6f6400d5f0 tx=0x7f6f6400dab0 comp rx=0 tx=0).stop 2024-05-28T06:26:16.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.399+0000 7f6f76a00640 1 -- 172.21.15.96:0/1966859903 shutdown_connections 2024-05-28T06:26:16.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.399+0000 7f6f76a00640 1 --2- 172.21.15.96:0/1966859903 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f6f5006e930 0x7f6f50070df0 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:16.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.399+0000 7f6f76a00640 1 --2- 172.21.15.96:0/1966859903 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f701600f0 0x7f6f701e9ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:16.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.399+0000 7f6f76a00640 1 --2- 172.21.15.96:0/1966859903 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6f7015fa00 0x7f6f700adab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:16.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.399+0000 7f6f76a00640 1 --2- 172.21.15.96:0/1966859903 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6f700d0970 0x7f6f700cddc0 unknown :-1 s=CLOSED pgs=2127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:16.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.399+0000 7f6f76a00640 1 -- 172.21.15.96:0/1966859903 >> 172.21.15.96:0/1966859903 conn(0x7f6f7006bae0 msgr2=0x7f6f700b0880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:26:16.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.399+0000 7f6f76a00640 1 -- 172.21.15.96:0/1966859903 shutdown_connections 2024-05-28T06:26:16.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.399+0000 7f6f76a00640 1 -- 172.21.15.96:0/1966859903 wait complete. 2024-05-28T06:26:16.410 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-05-28T06:26:16.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.619+0000 7fa88ae00640 1 Processor -- start 2024-05-28T06:26:16.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.620+0000 7fa88ae00640 1 -- start start 2024-05-28T06:26:16.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.621+0000 7fa88ae00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fa88415b6f0 0x7fa88406e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:16.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.621+0000 7fa88ae00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa8840725e0 0x7fa8840ae650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:16.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.621+0000 7fa88ae00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8840ade90 con 0x7fa8840d0640 2024-05-28T06:26:16.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.621+0000 7fa88ae00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fa884071c40 con 0x7fa88415b6f0 2024-05-28T06:26:16.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.621+0000 7fa88ae00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fa884071e70 con 0x7fa8840725e0 2024-05-28T06:26:16.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.621+0000 7fa87b400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fa88415b6f0 0x7fa88406e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:16.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.622+0000 7fa87b400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fa88415b6f0 0x7fa88406e350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35644/0 (socket says 172.21.15.96:35644) 2024-05-28T06:26:16.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.622+0000 7fa87b400640 1 -- 172.21.15.96:0/2939409023 learned_addr learned my addr 172.21.15.96:0/2939409023 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:26:16.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.622+0000 7fa888e00640 1 --2- 172.21.15.96:0/2939409023 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa8840725e0 0x7fa8840ae650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:16.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.622+0000 7fa87b400640 1 -- 172.21.15.96:0/2939409023 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa8840725e0 msgr2=0x7fa8840ae650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:16.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.622+0000 7fa87b400640 1 --2- 172.21.15.96:0/2939409023 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa8840725e0 0x7fa8840ae650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:16.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.622+0000 7fa87b400640 1 -- 172.21.15.96:0/2939409023 >> v1:172.21.15.123:6789/0 conn(0x7fa8840d0640 legacy=0x7fa8840d0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:16.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.622+0000 7fa87b400640 1 -- 172.21.15.96:0/2939409023 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa884072130 con 0x7fa88415b6f0 2024-05-28T06:26:16.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.623+0000 7fa87b400640 1 --2- 172.21.15.96:0/2939409023 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa88415b6f0 0x7fa88406e350 secure :-1 s=READY pgs=2128 cs=0 l=1 rev1=1 crypto rx=0x7fa86c009870 tx=0x7fa86c02a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcfb6faacad59778 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:16.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.624+0000 7fa883400640 1 -- 172.21.15.96:0/2939409023 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa86c037070 con 0x7fa88415b6f0 2024-05-28T06:26:16.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.624+0000 7fa883400640 1 -- 172.21.15.96:0/2939409023 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa86c032440 con 0x7fa88415b6f0 2024-05-28T06:26:16.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.624+0000 7fa883400640 1 -- 172.21.15.96:0/2939409023 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa86c032810 con 0x7fa88415b6f0 2024-05-28T06:26:16.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.624+0000 7fa88ae00640 1 -- 172.21.15.96:0/2939409023 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa88415b6f0 msgr2=0x7fa88406e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:16.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.624+0000 7fa88ae00640 1 --2- 172.21.15.96:0/2939409023 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa88415b6f0 0x7fa88406e350 secure :-1 s=READY pgs=2128 cs=0 l=1 rev1=1 crypto rx=0x7fa86c009870 tx=0x7fa86c02a210 comp rx=0 tx=0).stop 2024-05-28T06:26:16.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.624+0000 7fa88ae00640 1 -- 172.21.15.96:0/2939409023 shutdown_connections 2024-05-28T06:26:16.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.624+0000 7fa88ae00640 1 --2- 172.21.15.96:0/2939409023 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa8840725e0 0x7fa8840ae650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:16.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.624+0000 7fa88ae00640 1 --2- 172.21.15.96:0/2939409023 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa88415b6f0 0x7fa88406e350 unknown :-1 s=CLOSED pgs=2128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:16.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.624+0000 7fa88ae00640 1 -- 172.21.15.96:0/2939409023 >> 172.21.15.96:0/2939409023 conn(0x7fa88406bd20 msgr2=0x7fa88406c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:26:16.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.624+0000 7fa88ae00640 1 -- 172.21.15.96:0/2939409023 shutdown_connections 2024-05-28T06:26:16.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.624+0000 7fa88ae00640 1 -- 172.21.15.96:0/2939409023 wait complete. 2024-05-28T06:26:16.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.626+0000 7fa88ae00640 1 Processor -- start 2024-05-28T06:26:16.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.626+0000 7fa88ae00640 1 -- start start 2024-05-28T06:26:16.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.627+0000 7fa88ae00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa8840725e0 0x7fa88414ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:16.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.627+0000 7fa88ae00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa8840d0640 0x7fa8840cc740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:16.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.627+0000 7fa88ae00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa88415b6f0 0x7fa8840b0e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:16.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.627+0000 7fa88ae00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa8840aeb90 con 0x7fa8840725e0 2024-05-28T06:26:16.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.627+0000 7fa88ae00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa88406e890 con 0x7fa88415b6f0 2024-05-28T06:26:16.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.627+0000 7fa88ae00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fa8840aed10 con 0x7fa8840d0640 2024-05-28T06:26:16.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.627+0000 7fa883e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa8840725e0 0x7fa88414ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:16.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.627+0000 7fa883e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa8840725e0 0x7fa88414ba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35658/0 (socket says 172.21.15.96:35658) 2024-05-28T06:26:16.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.627+0000 7fa883e00640 1 -- 172.21.15.96:0/3382811367 learned_addr learned my addr 172.21.15.96:0/3382811367 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:26:16.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.628+0000 7fa883e00640 1 -- 172.21.15.96:0/3382811367 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa8840d0640 msgr2=0x7fa8840cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:16.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.628+0000 7fa883e00640 1 --2- 172.21.15.96:0/3382811367 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa8840d0640 0x7fa8840cc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:16.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.628+0000 7fa883e00640 1 -- 172.21.15.96:0/3382811367 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa88415b6f0 msgr2=0x7fa8840b0e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:16.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.628+0000 7fa883e00640 1 --2- 172.21.15.96:0/3382811367 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa88415b6f0 0x7fa8840b0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:16.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.628+0000 7fa883e00640 1 -- 172.21.15.96:0/3382811367 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8840cd120 con 0x7fa8840725e0 2024-05-28T06:26:16.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.628+0000 7fa883e00640 1 --2- 172.21.15.96:0/3382811367 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa8840725e0 0x7fa88414ba40 secure :-1 s=READY pgs=2129 cs=0 l=1 rev1=1 crypto rx=0x7fa87400c7f0 tx=0x7fa87400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:16.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.628+0000 7fa880c00640 1 -- 172.21.15.96:0/3382811367 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa874016020 con 0x7fa8840725e0 2024-05-28T06:26:16.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.628+0000 7fa880c00640 1 -- 172.21.15.96:0/3382811367 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa87400aa50 con 0x7fa8840725e0 2024-05-28T06:26:16.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.628+0000 7fa88ae00640 1 -- 172.21.15.96:0/3382811367 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8840cd400 con 0x7fa8840725e0 2024-05-28T06:26:16.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.628+0000 7fa880c00640 1 -- 172.21.15.96:0/3382811367 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa87400ae40 con 0x7fa8840725e0 2024-05-28T06:26:16.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.629+0000 7fa88ae00640 1 -- 172.21.15.96:0/3382811367 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa884137f60 con 0x7fa8840725e0 2024-05-28T06:26:16.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.630+0000 7fa88ae00640 1 -- 172.21.15.96:0/3382811367 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8840689e0 con 0x7fa8840725e0 2024-05-28T06:26:16.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.631+0000 7fa880c00640 1 -- 172.21.15.96:0/3382811367 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 48) ==== 93947+0+0 (secure 0 0 0) 0x7fa874019020 con 0x7fa8840725e0 2024-05-28T06:26:16.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.632+0000 7fa880c00640 1 --2- 172.21.15.96:0/3382811367 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa86006e900 0x7fa860070dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:16.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.632+0000 7fa880c00640 1 -- 172.21.15.96:0/3382811367 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 5823+0+0 (secure 0 0 0) 0x7fa87408a950 con 0x7fa8840725e0 2024-05-28T06:26:16.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.633+0000 7fa87b400640 1 --2- 172.21.15.96:0/3382811367 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa86006e900 0x7fa860070dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:16.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.634+0000 7fa87b400640 1 --2- 172.21.15.96:0/3382811367 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa86006e900 0x7fa860070dc0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7fa86c034000 tx=0x7fa86c02c000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:16.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.637+0000 7fa880c00640 1 -- 172.21.15.96:0/3382811367 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa874020070 con 0x7fa8840725e0 2024-05-28T06:26:16.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.877+0000 7fa88ae00640 1 -- 172.21.15.96:0/3382811367 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa88414bf80 con 0x7fa86006e900 2024-05-28T06:26:16.884 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:26:16.891+0000 7fea59e00640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-05-28T06:26:16.885 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:16.891+0000 7fa880c00640 1 -- 172.21.15.96:0/3382811367 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) ==== 41+0+0 (secure 0 0 0) 0x7fa88414bf80 con 0x7fa86006e900 2024-05-28T06:26:16.885 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:26:16.885 INFO:teuthology.orchestra.run.smithi096.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-05-28T06:26:16.895 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-05-28T06:26:18.897 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-05-28T06:26:19.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:19.123+0000 7f02b3800640 1 Processor -- start 2024-05-28T06:26:19.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:19.123+0000 7f02b3800640 1 -- start start 2024-05-28T06:26:19.120 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:19.124+0000 7f02b3800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f02ac162ac0 0x7f02ac0d0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:19.120 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:19.124+0000 7f02b3800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f02ac163ac0 0x7f02ac0d0b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:19.120 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:19.124+0000 7f02b3800640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02ac1640e0 con 0x7f02ac0b4120 2024-05-28T06:26:19.120 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:19.124+0000 7f02b3800640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f02ac0b3c90 con 0x7f02ac162ac0 2024-05-28T06:26:19.120 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:19.124+0000 7f02b3800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f02ac0b3360 con 0x7f02ac163ac0 2024-05-28T06:26:19.120 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:19.125+0000 7f02a3e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f02ac0b4120 0x7f02ac1633b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:44674/0 (socket says 172.21.15.96:44674) 2024-05-28T06:26:19.120 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:19.125+0000 7f02a3e00640 1 -- 172.21.15.96:0/2143005595 learned_addr learned my addr 172.21.15.96:0/2143005595 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:26:19.120 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:19.125+0000 7f02b0e00640 1 --2- 172.21.15.96:0/2143005595 >> v2:172.21.15.123:3300/0 conn(0x7f02ac162ac0 0x7f02ac0d0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:19.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:19.125+0000 7f02b1800640 1 --2- 172.21.15.96:0/2143005595 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f02ac163ac0 0x7f02ac0d0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:19.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:19.125+0000 7f02abe00640 1 -- 172.21.15.96:0/2143005595 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3531328233 0 0) 0x7f02ac1640e0 con 0x7f02ac0b4120 2024-05-28T06:26:19.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:19.125+0000 7f02abe00640 1 -- 172.21.15.96:0/2143005595 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0290003520 con 0x7f02ac0b4120 2024-05-28T06:26:19.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:19.126+0000 7f02abe00640 1 -- 172.21.15.96:0/2143005595 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3499113272 0 0) 0x7f0290003520 con 0x7f02ac0b4120 2024-05-28T06:26:19.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:19.126+0000 7f02abe00640 1 -- 172.21.15.96:0/2143005595 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f02ac163ac0 msgr2=0x7f02ac0d0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:19.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:19.126+0000 7f02abe00640 1 --2- 172.21.15.96:0/2143005595 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f02ac163ac0 0x7f02ac0d0b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:19.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:19.126+0000 7f02abe00640 1 -- 172.21.15.96:0/2143005595 >> v2:172.21.15.123:3300/0 conn(0x7f02ac162ac0 msgr2=0x7f02ac0d0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:19.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:19.126+0000 7f02abe00640 1 --2- 172.21.15.96:0/2143005595 >> v2:172.21.15.123:3300/0 conn(0x7f02ac162ac0 0x7f02ac0d0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:19.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:19.126+0000 7f02abe00640 1 -- 172.21.15.96:0/2143005595 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02ac0b3640 con 0x7f02ac0b4120 2024-05-28T06:26:19.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:19.126+0000 7f02abe00640 1 -- 172.21.15.96:0/2143005595 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f0294001670 con 0x7f02ac0b4120 2024-05-28T06:26:19.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:19.126+0000 7f02abe00640 1 -- 172.21.15.96:0/2143005595 >> v1:172.21.15.96:6789/0 conn(0x7f02ac0b4120 legacy=0x7f02ac1633b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:19.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:19.126+0000 7f02abe00640 1 --2- 172.21.15.96:0/2143005595 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f02900039b0 0x7f0290003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:19.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:19.126+0000 7f02abe00640 1 --2- 172.21.15.96:0/2143005595 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f02900043a0 0x7f0290004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:19.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:19.126+0000 7f02abe00640 1 --2- 172.21.15.96:0/2143005595 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0290004ef0 0x7f029000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:19.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:19.126+0000 7f02abe00640 1 -- 172.21.15.96:0/2143005595 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f02ac0b3c90 con 0x7f0290004ef0 2024-05-28T06:26:19.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:19.126+0000 7f02abe00640 1 -- 172.21.15.96:0/2143005595 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f02ac0b3360 con 0x7f02900043a0 2024-05-28T06:26:19.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:19.126+0000 7f02abe00640 1 -- 172.21.15.96:0/2143005595 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f029000bf00 con 0x7f02900039b0 2024-05-28T06:26:19.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:19.126+0000 7f02b1800640 1 --2- 172.21.15.96:0/2143005595 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0290004ef0 0x7f029000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:19.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:19.127+0000 7f02a3e00640 1 --2- 172.21.15.96:0/2143005595 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f02900043a0 0x7f0290004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:19.122 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:19.127+0000 7f02b0e00640 1 --2- 172.21.15.96:0/2143005595 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f02900039b0 0x7f0290003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:19.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:19.127+0000 7f02b1800640 1 -- 172.21.15.96:0/2143005595 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f02900039b0 msgr2=0x7f0290003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:19.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:19.127+0000 7f02b1800640 1 --2- 172.21.15.96:0/2143005595 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f02900039b0 0x7f0290003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:19.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:19.127+0000 7f02b1800640 1 -- 172.21.15.96:0/2143005595 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f02900043a0 msgr2=0x7f0290004880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:19.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:19.127+0000 7f02b1800640 1 --2- 172.21.15.96:0/2143005595 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f02900043a0 0x7f0290004880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:19.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:19.127+0000 7f02b1800640 1 -- 172.21.15.96:0/2143005595 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0290003520 con 0x7f0290004ef0 2024-05-28T06:26:19.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:19.128+0000 7f02b1800640 1 --2- 172.21.15.96:0/2143005595 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0290004ef0 0x7f029000b9c0 secure :-1 s=READY pgs=2131 cs=0 l=1 rev1=1 crypto rx=0x7f02ac163ea0 tx=0x7f0290005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a41e06bace22dbf server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:19.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:19.128+0000 7f02abe00640 1 -- 172.21.15.96:0/2143005595 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02a4036070 con 0x7f0290004ef0 2024-05-28T06:26:19.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:19.128+0000 7f02abe00640 1 -- 172.21.15.96:0/2143005595 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02a4031600 con 0x7f0290004ef0 2024-05-28T06:26:19.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:19.128+0000 7f02abe00640 1 -- 172.21.15.96:0/2143005595 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02a4009bd0 con 0x7f0290004ef0 2024-05-28T06:26:19.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:19.129+0000 7f02b3800640 1 -- 172.21.15.96:0/2143005595 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0290004ef0 msgr2=0x7f029000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:19.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:19.129+0000 7f02b3800640 1 --2- 172.21.15.96:0/2143005595 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0290004ef0 0x7f029000b9c0 secure :-1 s=READY pgs=2131 cs=0 l=1 rev1=1 crypto rx=0x7f02ac163ea0 tx=0x7f0290005830 comp rx=0 tx=0).stop 2024-05-28T06:26:19.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:19.129+0000 7f02b1800640 1 -- 172.21.15.96:0/2143005595 reap_dead start 2024-05-28T06:26:19.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:19.129+0000 7f02b3800640 1 -- 172.21.15.96:0/2143005595 shutdown_connections 2024-05-28T06:26:19.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:19.129+0000 7f02b3800640 1 -- 172.21.15.96:0/2143005595 >> 172.21.15.96:0/2143005595 conn(0x7f02ac06bd20 msgr2=0x7f02ac06c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:26:19.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:19.129+0000 7f02b3800640 1 -- 172.21.15.96:0/2143005595 shutdown_connections 2024-05-28T06:26:19.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:19.129+0000 7f02b3800640 1 -- 172.21.15.96:0/2143005595 wait complete. 2024-05-28T06:26:19.125 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:19.131+0000 7f02b3800640 1 Processor -- start 2024-05-28T06:26:19.125 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:19.131+0000 7f02b3800640 1 -- start start 2024-05-28T06:26:19.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:19.132+0000 7f02b3800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0290004ef0 0x7f02ac15c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:19.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:19.132+0000 7f02b3800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f02ac06dbd0 0x7f02ac06e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:19.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:19.132+0000 7f02b3800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f02ac160ae0 0x7f02ac160f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:19.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:19.132+0000 7f02b3800640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f02ac163020 con 0x7f0290004ef0 2024-05-28T06:26:19.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:19.132+0000 7f02b3800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f02ac162ea0 con 0x7f02ac06dbd0 2024-05-28T06:26:19.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:19.132+0000 7f02b3800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f02ac0d0b80 con 0x7f02ac160ae0 2024-05-28T06:26:19.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:19.132+0000 7f02b0e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0290004ef0 0x7f02ac15c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:19.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:19.132+0000 7f02b0e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0290004ef0 0x7f02ac15c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:50776/0 (socket says 172.21.15.96:50776) 2024-05-28T06:26:19.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:19.132+0000 7f02b0e00640 1 -- 172.21.15.96:0/1067856393 learned_addr learned my addr 172.21.15.96:0/1067856393 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:26:19.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:19.133+0000 7f02b0e00640 1 -- 172.21.15.96:0/1067856393 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f02ac160ae0 msgr2=0x7f02ac160f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:19.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:19.133+0000 7f02b0e00640 1 --2- 172.21.15.96:0/1067856393 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f02ac160ae0 0x7f02ac160f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:19.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:19.133+0000 7f02b0e00640 1 -- 172.21.15.96:0/1067856393 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f02ac06dbd0 msgr2=0x7f02ac06e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:19.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:19.133+0000 7f02b0e00640 1 --2- 172.21.15.96:0/1067856393 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f02ac06dbd0 0x7f02ac06e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:19.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:19.133+0000 7f02b0e00640 1 -- 172.21.15.96:0/1067856393 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02ac161860 con 0x7f0290004ef0 2024-05-28T06:26:19.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:19.133+0000 7f02b0e00640 1 --2- 172.21.15.96:0/1067856393 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0290004ef0 0x7f02ac15c920 secure :-1 s=READY pgs=2132 cs=0 l=1 rev1=1 crypto rx=0x7f029c0116d0 tx=0x7f029c011b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:19.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:19.133+0000 7f02a9600640 1 -- 172.21.15.96:0/1067856393 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f029c00f940 con 0x7f0290004ef0 2024-05-28T06:26:19.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:19.133+0000 7f02a9600640 1 -- 172.21.15.96:0/1067856393 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f029c00fae0 con 0x7f0290004ef0 2024-05-28T06:26:19.128 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:19.133+0000 7f02a9600640 1 -- 172.21.15.96:0/1067856393 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f029c00fe90 con 0x7f0290004ef0 2024-05-28T06:26:19.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:19.134+0000 7f02b3800640 1 -- 172.21.15.96:0/1067856393 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02ac1f8860 con 0x7f0290004ef0 2024-05-28T06:26:19.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:19.134+0000 7f02b3800640 1 -- 172.21.15.96:0/1067856393 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02ac1f8e20 con 0x7f0290004ef0 2024-05-28T06:26:19.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:19.135+0000 7f02b3800640 1 -- 172.21.15.96:0/1067856393 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02ac06e590 con 0x7f0290004ef0 2024-05-28T06:26:19.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:19.136+0000 7f02a9600640 1 -- 172.21.15.96:0/1067856393 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 49) ==== 94049+0+0 (secure 0 0 0) 0x7f029c015460 con 0x7f0290004ef0 2024-05-28T06:26:19.131 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:19.137+0000 7f02a9600640 1 --2- 172.21.15.96:0/1067856393 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f028806ebc0 0x7f0288071080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:19.131 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:19.137+0000 7f02a9600640 1 -- 172.21.15.96:0/1067856393 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 5823+0+0 (secure 0 0 0) 0x7f029c08b6e0 con 0x7f0290004ef0 2024-05-28T06:26:19.131 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:19.138+0000 7f02a3e00640 1 --2- 172.21.15.96:0/1067856393 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f028806ebc0 0x7f0288071080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:19.132 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:19.139+0000 7f02a3e00640 1 --2- 172.21.15.96:0/1067856393 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f028806ebc0 0x7f0288071080 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f0294001ce0 tx=0x7f0294001c30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:19.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:19.142+0000 7f02a9600640 1 -- 172.21.15.96:0/1067856393 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f029c019030 con 0x7f0290004ef0 2024-05-28T06:26:19.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:19.375+0000 7f02b3800640 1 -- 172.21.15.96:0/1067856393 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f02ac1658e0 con 0x7f028806ebc0 2024-05-28T06:26:19.386 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:19.390+0000 7f02a9600640 1 -- 172.21.15.96:0/1067856393 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) ==== 41+0+0 (secure 0 0 0) 0x7f02ac1658e0 con 0x7f028806ebc0 2024-05-28T06:26:19.386 INFO:teuthology.orchestra.run.smithi096.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-05-28T06:26:19.386 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:26:19.389+0000 7fea59e00640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-05-28T06:26:19.387 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:26:19.395 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-05-28T06:26:21.396 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-05-28T06:26:21.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:21.620+0000 7f2e6c400640 1 Processor -- start 2024-05-28T06:26:21.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:21.620+0000 7f2e6c400640 1 -- start start 2024-05-28T06:26:21.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:21.622+0000 7f2e6c400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2e6415abe0 0x7f2e640d0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:21.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:21.622+0000 7f2e6c400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f2e6406f380 0x7f2e640b45d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:21.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:21.622+0000 7f2e6c400640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e6406e2e0 con 0x7f2e640b4b10 2024-05-28T06:26:21.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:21.622+0000 7f2e6c400640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f2e6406e5c0 con 0x7f2e6406f380 2024-05-28T06:26:21.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:21.622+0000 7f2e6c400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f2e6406e8a0 con 0x7f2e6415abe0 2024-05-28T06:26:21.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:21.622+0000 7f2e69000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f2e6406f380 0x7f2e640b45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:21.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:21.622+0000 7f2e69000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f2e6406f380 0x7f2e640b45d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:50790/0 (socket says 172.21.15.96:50790) 2024-05-28T06:26:21.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:21.622+0000 7f2e69000640 1 -- 172.21.15.96:0/2056085351 learned_addr learned my addr 172.21.15.96:0/2056085351 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:26:21.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:21.623+0000 7f2e69a00640 1 --2- 172.21.15.96:0/2056085351 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2e6415abe0 0x7f2e640d0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:21.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:21.623+0000 7f2e69000640 1 -- 172.21.15.96:0/2056085351 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2e6415abe0 msgr2=0x7f2e640d0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:21.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:21.623+0000 7f2e69000640 1 --2- 172.21.15.96:0/2056085351 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2e6415abe0 0x7f2e640d0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:21.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:21.623+0000 7f2e69000640 1 -- 172.21.15.96:0/2056085351 >> v1:172.21.15.123:6789/0 conn(0x7f2e640b4b10 legacy=0x7f2e640725c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:21.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:21.623+0000 7f2e69000640 1 -- 172.21.15.96:0/2056085351 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e6406eb80 con 0x7f2e6406f380 2024-05-28T06:26:21.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:21.623+0000 7f2e69a00640 1 --2- 172.21.15.96:0/2056085351 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2e6415abe0 0x7f2e640d0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:26:21.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:21.624+0000 7f2e69000640 1 --2- 172.21.15.96:0/2056085351 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2e6406f380 0x7f2e640b45d0 secure :-1 s=READY pgs=2133 cs=0 l=1 rev1=1 crypto rx=0x7f2e50009870 tx=0x7f2e5002a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77d0af85ec2611e1 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:21.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:21.624+0000 7f2e63e00640 1 -- 172.21.15.96:0/2056085351 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e50037070 con 0x7f2e6406f380 2024-05-28T06:26:21.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:21.624+0000 7f2e63e00640 1 -- 172.21.15.96:0/2056085351 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e50032440 con 0x7f2e6406f380 2024-05-28T06:26:21.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:21.624+0000 7f2e63e00640 1 -- 172.21.15.96:0/2056085351 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e50032810 con 0x7f2e6406f380 2024-05-28T06:26:21.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:21.625+0000 7f2e6c400640 1 -- 172.21.15.96:0/2056085351 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2e6406f380 msgr2=0x7f2e640b45d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:21.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:21.625+0000 7f2e6c400640 1 --2- 172.21.15.96:0/2056085351 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2e6406f380 0x7f2e640b45d0 secure :-1 s=READY pgs=2133 cs=0 l=1 rev1=1 crypto rx=0x7f2e50009870 tx=0x7f2e5002a210 comp rx=0 tx=0).stop 2024-05-28T06:26:21.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:21.625+0000 7f2e6c400640 1 -- 172.21.15.96:0/2056085351 shutdown_connections 2024-05-28T06:26:21.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:21.625+0000 7f2e6c400640 1 --2- 172.21.15.96:0/2056085351 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2e6406f380 0x7f2e640b45d0 unknown :-1 s=CLOSED pgs=2133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:21.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:21.625+0000 7f2e6c400640 1 --2- 172.21.15.96:0/2056085351 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2e6415abe0 0x7f2e640d0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:21.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:21.625+0000 7f2e6c400640 1 -- 172.21.15.96:0/2056085351 >> 172.21.15.96:0/2056085351 conn(0x7f2e6406bd20 msgr2=0x7f2e6406c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:26:21.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:21.625+0000 7f2e6c400640 1 -- 172.21.15.96:0/2056085351 shutdown_connections 2024-05-28T06:26:21.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:21.625+0000 7f2e6c400640 1 -- 172.21.15.96:0/2056085351 wait complete. 2024-05-28T06:26:21.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:21.627+0000 7f2e6c400640 1 Processor -- start 2024-05-28T06:26:21.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:21.627+0000 7f2e6c400640 1 -- start start 2024-05-28T06:26:21.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:21.628+0000 7f2e6c400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2e6406f380 0x7f2e640b0f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:21.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:21.628+0000 7f2e6c400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e640b4b10 0x7f2e640b14d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:21.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:21.628+0000 7f2e6c400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2e6415abe0 0x7f2e641f4d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:21.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:21.628+0000 7f2e6c400640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f2e6406df60 con 0x7f2e6415abe0 2024-05-28T06:26:21.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:21.628+0000 7f2e6c400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f2e6415afc0 con 0x7f2e640b4b10 2024-05-28T06:26:21.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:21.628+0000 7f2e6c400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f2e640d0b80 con 0x7f2e6406f380 2024-05-28T06:26:21.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:21.628+0000 7f2e6a400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2e6415abe0 0x7f2e641f4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:21.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:21.628+0000 7f2e6a400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2e6415abe0 0x7f2e641f4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:50806/0 (socket says 172.21.15.96:50806) 2024-05-28T06:26:21.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:21.628+0000 7f2e6a400640 1 -- 172.21.15.96:0/1650906304 learned_addr learned my addr 172.21.15.96:0/1650906304 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:26:21.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:21.629+0000 7f2e69a00640 1 --2- 172.21.15.96:0/1650906304 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2e6406f380 0x7f2e640b0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:21.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:21.629+0000 7f2e69000640 1 --2- 172.21.15.96:0/1650906304 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e640b4b10 0x7f2e640b14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:21.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:21.629+0000 7f2e6a400640 1 -- 172.21.15.96:0/1650906304 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2e6406f380 msgr2=0x7f2e640b0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:21.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:21.629+0000 7f2e6a400640 1 --2- 172.21.15.96:0/1650906304 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2e6406f380 0x7f2e640b0f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:21.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:21.629+0000 7f2e6a400640 1 -- 172.21.15.96:0/1650906304 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e640b4b10 msgr2=0x7f2e640b14d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:21.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:21.629+0000 7f2e6a400640 1 --2- 172.21.15.96:0/1650906304 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e640b4b10 0x7f2e640b14d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:21.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:21.629+0000 7f2e6a400640 1 -- 172.21.15.96:0/1650906304 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e6409a670 con 0x7f2e6415abe0 2024-05-28T06:26:21.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:21.629+0000 7f2e69000640 1 --2- 172.21.15.96:0/1650906304 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e640b4b10 0x7f2e640b14d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:26:21.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:21.629+0000 7f2e6a400640 1 --2- 172.21.15.96:0/1650906304 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2e6415abe0 0x7f2e641f4d00 secure :-1 s=READY pgs=2134 cs=0 l=1 rev1=1 crypto rx=0x7f2e5c00c3e0 tx=0x7f2e5c00c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:21.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:21.629+0000 7f2e61600640 1 -- 172.21.15.96:0/1650906304 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e5c011790 con 0x7f2e6415abe0 2024-05-28T06:26:21.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:21.629+0000 7f2e61600640 1 -- 172.21.15.96:0/1650906304 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e5c017070 con 0x7f2e6415abe0 2024-05-28T06:26:21.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:21.629+0000 7f2e61600640 1 -- 172.21.15.96:0/1650906304 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e5c011bb0 con 0x7f2e6415abe0 2024-05-28T06:26:21.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:21.629+0000 7f2e6c400640 1 -- 172.21.15.96:0/1650906304 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e640aeb70 con 0x7f2e6415abe0 2024-05-28T06:26:21.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:21.630+0000 7f2e6c400640 1 -- 172.21.15.96:0/1650906304 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e640af130 con 0x7f2e6415abe0 2024-05-28T06:26:21.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:21.631+0000 7f2e6c400640 1 -- 172.21.15.96:0/1650906304 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e640689e0 con 0x7f2e6415abe0 2024-05-28T06:26:21.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:21.632+0000 7f2e61600640 1 -- 172.21.15.96:0/1650906304 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 49) ==== 94049+0+0 (secure 0 0 0) 0x7f2e5c01c460 con 0x7f2e6415abe0 2024-05-28T06:26:21.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:21.633+0000 7f2e61600640 1 --2- 172.21.15.96:0/1650906304 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2e4406ea20 0x7f2e44070ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:21.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:21.633+0000 7f2e61600640 1 -- 172.21.15.96:0/1650906304 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 5823+0+0 (secure 0 0 0) 0x7f2e5c08aa20 con 0x7f2e6415abe0 2024-05-28T06:26:21.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:21.633+0000 7f2e69a00640 1 --2- 172.21.15.96:0/1650906304 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2e4406ea20 0x7f2e44070ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:21.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:21.635+0000 7f2e69a00640 1 --2- 172.21.15.96:0/1650906304 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2e4406ea20 0x7f2e44070ee0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f2e64143fb0 tx=0x7f2e54008450 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:21.631 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:21.637+0000 7f2e61600640 1 -- 172.21.15.96:0/1650906304 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2e5c05a950 con 0x7f2e6415abe0 2024-05-28T06:26:21.865 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:21.871+0000 7f2e6c400640 1 -- 172.21.15.96:0/1650906304 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2e6415b1f0 con 0x7f2e4406ea20 2024-05-28T06:26:21.878 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:26:21.885+0000 7fea59e00640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-05-28T06:26:21.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:21.886+0000 7f2e61600640 1 -- 172.21.15.96:0/1650906304 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) ==== 41+0+0 (secure 0 0 0) 0x7f2e54008b10 con 0x7f2e4406ea20 2024-05-28T06:26:21.879 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:26:21.879 INFO:teuthology.orchestra.run.smithi096.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-05-28T06:26:21.895 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-05-28T06:26:23.898 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-05-28T06:26:24.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:24.128+0000 7f7e8e600640 1 Processor -- start 2024-05-28T06:26:24.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:24.128+0000 7f7e8e600640 1 -- start start 2024-05-28T06:26:24.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:24.130+0000 7f7e8e600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7e8806f5e0 0x7f7e880cdd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:24.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:24.130+0000 7f7e8e600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f7e8815afd0 0x7f7e880cc230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:24.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:24.130+0000 7f7e8e600640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e8815b520 con 0x7f7e880cc770 2024-05-28T06:26:24.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:24.130+0000 7f7e8e600640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f7e8806f9c0 con 0x7f7e8815afd0 2024-05-28T06:26:24.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:24.130+0000 7f7e8e600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f7e8815b800 con 0x7f7e8806f5e0 2024-05-28T06:26:24.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:24.130+0000 7f7e7fe00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f7e8815afd0 0x7f7e880cc230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:24.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:24.130+0000 7f7e8dc00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f7e880cc770 0x7f7e880ccb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:44678/0 (socket says 172.21.15.96:44678) 2024-05-28T06:26:24.123 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:24.130+0000 7f7e8dc00640 1 -- 172.21.15.96:0/4165759550 learned_addr learned my addr 172.21.15.96:0/4165759550 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:26:24.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:24.131+0000 7f7e8d200640 1 --2- 172.21.15.96:0/4165759550 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7e8806f5e0 0x7f7e880cdd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:24.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:24.131+0000 7f7e87e00640 1 -- 172.21.15.96:0/4165759550 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3393777097 0 0) 0x7f7e8815b520 con 0x7f7e880cc770 2024-05-28T06:26:24.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:24.131+0000 7f7e87e00640 1 -- 172.21.15.96:0/4165759550 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7e6c003520 con 0x7f7e880cc770 2024-05-28T06:26:24.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:24.131+0000 7f7e7fe00640 1 -- 172.21.15.96:0/4165759550 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7e8806f5e0 msgr2=0x7f7e880cdd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:24.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:24.131+0000 7f7e7fe00640 1 --2- 172.21.15.96:0/4165759550 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7e8806f5e0 0x7f7e880cdd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:24.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:24.131+0000 7f7e7fe00640 1 -- 172.21.15.96:0/4165759550 >> v1:172.21.15.96:6789/0 conn(0x7f7e880cc770 legacy=0x7f7e880ccb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:24.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:24.131+0000 7f7e7fe00640 1 -- 172.21.15.96:0/4165759550 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e8815bae0 con 0x7f7e8815afd0 2024-05-28T06:26:24.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:24.132+0000 7f7e7fe00640 1 --2- 172.21.15.96:0/4165759550 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7e8815afd0 0x7f7e880cc230 secure :-1 s=READY pgs=2136 cs=0 l=1 rev1=1 crypto rx=0x7f7e700097a0 tx=0x7f7e7002a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7093810405aad7e server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:24.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:24.132+0000 7f7e87e00640 1 -- 172.21.15.96:0/4165759550 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e70037070 con 0x7f7e8815afd0 2024-05-28T06:26:24.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:24.132+0000 7f7e87e00640 1 -- 172.21.15.96:0/4165759550 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e70032440 con 0x7f7e8815afd0 2024-05-28T06:26:24.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:24.132+0000 7f7e87e00640 1 -- 172.21.15.96:0/4165759550 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e70032810 con 0x7f7e8815afd0 2024-05-28T06:26:24.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:24.133+0000 7f7e8e600640 1 -- 172.21.15.96:0/4165759550 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7e8815afd0 msgr2=0x7f7e880cc230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:24.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:24.133+0000 7f7e8e600640 1 --2- 172.21.15.96:0/4165759550 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7e8815afd0 0x7f7e880cc230 secure :-1 s=READY pgs=2136 cs=0 l=1 rev1=1 crypto rx=0x7f7e700097a0 tx=0x7f7e7002a180 comp rx=0 tx=0).stop 2024-05-28T06:26:24.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:24.133+0000 7f7e8e600640 1 -- 172.21.15.96:0/4165759550 shutdown_connections 2024-05-28T06:26:24.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:24.133+0000 7f7e8e600640 1 --2- 172.21.15.96:0/4165759550 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7e8815afd0 0x7f7e880cc230 unknown :-1 s=CLOSED pgs=2136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:24.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:24.133+0000 7f7e8e600640 1 --2- 172.21.15.96:0/4165759550 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7e8806f5e0 0x7f7e880cdd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:24.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:24.133+0000 7f7e8e600640 1 -- 172.21.15.96:0/4165759550 >> 172.21.15.96:0/4165759550 conn(0x7f7e8806b900 msgr2=0x7f7e8806bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:26:24.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:24.133+0000 7f7e8e600640 1 -- 172.21.15.96:0/4165759550 shutdown_connections 2024-05-28T06:26:24.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:24.133+0000 7f7e8e600640 1 -- 172.21.15.96:0/4165759550 wait complete. 2024-05-28T06:26:24.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:24.135+0000 7f7e8e600640 1 Processor -- start 2024-05-28T06:26:24.127 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:24.135+0000 7f7e8e600640 1 -- start start 2024-05-28T06:26:24.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:24.136+0000 7f7e8e600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7e8806f5e0 0x7f7e8815f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:24.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:24.136+0000 7f7e8e600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7e8815afd0 0x7f7e88159840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:24.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:24.136+0000 7f7e8e600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7e8814d640 0x7f7e8815df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:24.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:24.136+0000 7f7e8e600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f7e880cd420 con 0x7f7e8815afd0 2024-05-28T06:26:24.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:24.136+0000 7f7e8e600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f7e880cd2a0 con 0x7f7e8806f5e0 2024-05-28T06:26:24.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:24.136+0000 7f7e8e600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f7e880d0180 con 0x7f7e8814d640 2024-05-28T06:26:24.129 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:24.136+0000 7f7e7fe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7e8815afd0 0x7f7e88159840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:24.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:24.136+0000 7f7e7fe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7e8815afd0 0x7f7e88159840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:50814/0 (socket says 172.21.15.96:50814) 2024-05-28T06:26:24.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:24.136+0000 7f7e7fe00640 1 -- 172.21.15.96:0/4123048468 learned_addr learned my addr 172.21.15.96:0/4123048468 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:26:24.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:24.136+0000 7f7e7fe00640 1 -- 172.21.15.96:0/4123048468 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7e8814d640 msgr2=0x7f7e8815df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:24.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:24.136+0000 7f7e7fe00640 1 --2- 172.21.15.96:0/4123048468 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7e8814d640 0x7f7e8815df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:24.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:24.136+0000 7f7e7fe00640 1 -- 172.21.15.96:0/4123048468 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7e8806f5e0 msgr2=0x7f7e8815f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:24.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:24.136+0000 7f7e7fe00640 1 --2- 172.21.15.96:0/4123048468 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7e8806f5e0 0x7f7e8815f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:24.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:24.136+0000 7f7e7fe00640 1 -- 172.21.15.96:0/4123048468 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e88152aa0 con 0x7f7e8815afd0 2024-05-28T06:26:24.130 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:24.137+0000 7f7e7fe00640 1 --2- 172.21.15.96:0/4123048468 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7e8815afd0 0x7f7e88159840 secure :-1 s=READY pgs=2137 cs=0 l=1 rev1=1 crypto rx=0x7f7e700098c0 tx=0x7f7e700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:24.131 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:24.137+0000 7f7e85600640 1 -- 172.21.15.96:0/4123048468 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e70037040 con 0x7f7e8815afd0 2024-05-28T06:26:24.131 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:24.137+0000 7f7e85600640 1 -- 172.21.15.96:0/4123048468 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e70030500 con 0x7f7e8815afd0 2024-05-28T06:26:24.132 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:24.137+0000 7f7e85600640 1 -- 172.21.15.96:0/4123048468 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e700308d0 con 0x7f7e8815afd0 2024-05-28T06:26:24.132 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:24.137+0000 7f7e8e600640 1 -- 172.21.15.96:0/4123048468 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e88152d80 con 0x7f7e8815afd0 2024-05-28T06:26:24.132 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:24.137+0000 7f7e8e600640 1 -- 172.21.15.96:0/4123048468 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e88159040 con 0x7f7e8815afd0 2024-05-28T06:26:24.132 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:24.139+0000 7f7e8e600640 1 -- 172.21.15.96:0/4123048468 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e880689e0 con 0x7f7e8815afd0 2024-05-28T06:26:24.132 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:24.140+0000 7f7e85600640 1 -- 172.21.15.96:0/4123048468 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 49) ==== 94049+0+0 (secure 0 0 0) 0x7f7e70044020 con 0x7f7e8815afd0 2024-05-28T06:26:24.133 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:24.140+0000 7f7e85600640 1 --2- 172.21.15.96:0/4123048468 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7e6406ea20 0x7f7e64070ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:24.133 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:24.141+0000 7f7e85600640 1 -- 172.21.15.96:0/4123048468 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 5823+0+0 (secure 0 0 0) 0x7f7e700aafa0 con 0x7f7e8815afd0 2024-05-28T06:26:24.133 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:24.141+0000 7f7e8d200640 1 --2- 172.21.15.96:0/4123048468 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7e6406ea20 0x7f7e64070ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:24.135 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:24.142+0000 7f7e8d200640 1 --2- 172.21.15.96:0/4123048468 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7e6406ea20 0x7f7e64070ee0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f7e780098f0 tx=0x7f7e78008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:24.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:24.145+0000 7f7e85600640 1 -- 172.21.15.96:0/4123048468 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7e7007ae30 con 0x7f7e8815afd0 2024-05-28T06:26:24.378 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:24.385+0000 7f7e8e600640 1 -- 172.21.15.96:0/4123048468 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f7e880ce8b0 con 0x7f7e6406ea20 2024-05-28T06:26:24.391 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:26:24.398+0000 7fea59e00640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-05-28T06:26:24.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:24.399+0000 7f7e85600640 1 -- 172.21.15.96:0/4123048468 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) ==== 41+0+0 (secure 0 0 0) 0x7f7e880ce8b0 con 0x7f7e6406ea20 2024-05-28T06:26:24.392 INFO:teuthology.orchestra.run.smithi096.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-05-28T06:26:24.392 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:26:24.406 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-05-28T06:26:26.408 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-05-28T06:26:26.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:26.639+0000 7f7d16400640 1 Processor -- start 2024-05-28T06:26:26.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:26.639+0000 7f7d16400640 1 -- start start 2024-05-28T06:26:26.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:26.641+0000 7f7d16400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7d10161a90 0x7f7d1006e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:26.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:26.641+0000 7f7d16400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f7d10161e70 0x7f7d100b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:26.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:26.641+0000 7f7d16400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f7d1006e9b0 0x7f7d100ad2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:26.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:26.641+0000 7f7d16400640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f7d100b3d40 con 0x7f7d1006e9b0 2024-05-28T06:26:26.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:26.641+0000 7f7d16400640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f7d100b4020 con 0x7f7d10161e70 2024-05-28T06:26:26.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:26.641+0000 7f7d16400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f7d100b4300 con 0x7f7d10161a90 2024-05-28T06:26:26.635 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:26.641+0000 7f7d0fe00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f7d1006e9b0 0x7f7d100ad2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:26.635 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:26.641+0000 7f7d0fe00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f7d1006e9b0 0x7f7d100ad2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:50818/0 (socket says 172.21.15.96:50818) 2024-05-28T06:26:26.635 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:26.641+0000 7f7d0fe00640 1 -- 172.21.15.96:0/2756741332 learned_addr learned my addr 172.21.15.96:0/2756741332 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:26:26.635 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:26.642+0000 7f7d07e00640 1 --2- 172.21.15.96:0/2756741332 >> v2:172.21.15.123:3300/0 conn(0x7f7d10161e70 0x7f7d100b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:26.635 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:26.642+0000 7f7d0f400640 1 --2- 172.21.15.96:0/2756741332 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7d10161a90 0x7f7d1006e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:26.635 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:26.642+0000 7f7d0fe00640 1 -- 172.21.15.96:0/2756741332 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7d10161a90 msgr2=0x7f7d1006e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:26.635 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:26.642+0000 7f7d0fe00640 1 --2- 172.21.15.96:0/2756741332 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7d10161a90 0x7f7d1006e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:26.635 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:26.642+0000 7f7d0fe00640 1 -- 172.21.15.96:0/2756741332 >> v2:172.21.15.123:3300/0 conn(0x7f7d10161e70 msgr2=0x7f7d100b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:26.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:26.642+0000 7f7d0fe00640 1 --2- 172.21.15.96:0/2756741332 >> v2:172.21.15.123:3300/0 conn(0x7f7d10161e70 0x7f7d100b32c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:26.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:26.642+0000 7f7d0fe00640 1 -- 172.21.15.96:0/2756741332 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d1006dc70 con 0x7f7d1006e9b0 2024-05-28T06:26:26.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:26.643+0000 7f7d0fe00640 1 --2- 172.21.15.96:0/2756741332 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7d1006e9b0 0x7f7d100ad2a0 secure :-1 s=READY pgs=2138 cs=0 l=1 rev1=1 crypto rx=0x7f7d100d11a0 tx=0x7f7d0802a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3f98f0eda5b1f33 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:26.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:26.644+0000 7f7d0ea00640 1 -- 172.21.15.96:0/2756741332 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d08036070 con 0x7f7d1006e9b0 2024-05-28T06:26:26.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:26.644+0000 7f7d0ea00640 1 -- 172.21.15.96:0/2756741332 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d08031600 con 0x7f7d1006e9b0 2024-05-28T06:26:26.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:26.644+0000 7f7d0ea00640 1 -- 172.21.15.96:0/2756741332 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d080319d0 con 0x7f7d1006e9b0 2024-05-28T06:26:26.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:26.644+0000 7f7d16400640 1 -- 172.21.15.96:0/2756741332 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7d1006e9b0 msgr2=0x7f7d100ad2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:26.638 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:26.644+0000 7f7d16400640 1 --2- 172.21.15.96:0/2756741332 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7d1006e9b0 0x7f7d100ad2a0 secure :-1 s=READY pgs=2138 cs=0 l=1 rev1=1 crypto rx=0x7f7d100d11a0 tx=0x7f7d0802a6a0 comp rx=0 tx=0).stop 2024-05-28T06:26:26.638 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:26.645+0000 7f7d16400640 1 -- 172.21.15.96:0/2756741332 shutdown_connections 2024-05-28T06:26:26.638 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:26.645+0000 7f7d16400640 1 --2- 172.21.15.96:0/2756741332 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7d1006e9b0 0x7f7d100ad2a0 unknown :-1 s=CLOSED pgs=2138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:26.638 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:26.645+0000 7f7d16400640 1 --2- 172.21.15.96:0/2756741332 >> v2:172.21.15.123:3300/0 conn(0x7f7d10161e70 0x7f7d100b32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:26.638 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:26.645+0000 7f7d16400640 1 --2- 172.21.15.96:0/2756741332 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7d10161a90 0x7f7d1006e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:26.638 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:26.645+0000 7f7d16400640 1 -- 172.21.15.96:0/2756741332 >> 172.21.15.96:0/2756741332 conn(0x7f7d1006bd20 msgr2=0x7f7d1006c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:26:26.638 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:26.645+0000 7f7d16400640 1 -- 172.21.15.96:0/2756741332 shutdown_connections 2024-05-28T06:26:26.638 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:26.645+0000 7f7d16400640 1 -- 172.21.15.96:0/2756741332 wait complete. 2024-05-28T06:26:26.639 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:26.646+0000 7f7d16400640 1 Processor -- start 2024-05-28T06:26:26.639 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:26.647+0000 7f7d16400640 1 -- start start 2024-05-28T06:26:26.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:26.647+0000 7f7d16400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d1006e9b0 0x7f7d100998f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:26.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:26.648+0000 7f7d16400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7d10161a90 0x7f7d101586a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:26.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:26.648+0000 7f7d16400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7d10161e70 0x7f7d10160690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:26.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:26.648+0000 7f7d16400640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f7d100b3800 con 0x7f7d10161e70 2024-05-28T06:26:26.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:26.648+0000 7f7d16400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f7d100b5020 con 0x7f7d1006e9b0 2024-05-28T06:26:26.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:26.648+0000 7f7d16400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f7d100b3980 con 0x7f7d10161a90 2024-05-28T06:26:26.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:26.648+0000 7f7d0fe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7d10161e70 0x7f7d10160690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:26.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:26.648+0000 7f7d0fe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7d10161e70 0x7f7d10160690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:50832/0 (socket says 172.21.15.96:50832) 2024-05-28T06:26:26.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:26.648+0000 7f7d0fe00640 1 -- 172.21.15.96:0/4000793007 learned_addr learned my addr 172.21.15.96:0/4000793007 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:26:26.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:26.648+0000 7f7d07e00640 1 --2- 172.21.15.96:0/4000793007 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7d10161a90 0x7f7d101586a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:26.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:26.648+0000 7f7d0f400640 1 --2- 172.21.15.96:0/4000793007 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d1006e9b0 0x7f7d100998f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:26.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:26.648+0000 7f7d0fe00640 1 -- 172.21.15.96:0/4000793007 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7d10161a90 msgr2=0x7f7d101586a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:26.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:26.648+0000 7f7d0fe00640 1 --2- 172.21.15.96:0/4000793007 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7d10161a90 0x7f7d101586a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:26.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:26.648+0000 7f7d0fe00640 1 -- 172.21.15.96:0/4000793007 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d1006e9b0 msgr2=0x7f7d100998f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:26.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:26.648+0000 7f7d0fe00640 1 --2- 172.21.15.96:0/4000793007 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d1006e9b0 0x7f7d100998f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:26.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:26.648+0000 7f7d0fe00640 1 -- 172.21.15.96:0/4000793007 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d10159170 con 0x7f7d10161e70 2024-05-28T06:26:26.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:26.648+0000 7f7d0f400640 1 --2- 172.21.15.96:0/4000793007 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d1006e9b0 0x7f7d100998f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:26:26.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:26.649+0000 7f7d0fe00640 1 --2- 172.21.15.96:0/4000793007 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7d10161e70 0x7f7d10160690 secure :-1 s=READY pgs=2139 cs=0 l=1 rev1=1 crypto rx=0x7f7d08009f40 tx=0x7f7d08004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:26.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:26.649+0000 7f7d07400640 1 -- 172.21.15.96:0/4000793007 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d08036050 con 0x7f7d10161e70 2024-05-28T06:26:26.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:26.649+0000 7f7d07400640 1 -- 172.21.15.96:0/4000793007 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d08042070 con 0x7f7d10161e70 2024-05-28T06:26:26.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:26.649+0000 7f7d07400640 1 -- 172.21.15.96:0/4000793007 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d08009510 con 0x7f7d10161e70 2024-05-28T06:26:26.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:26.649+0000 7f7d16400640 1 -- 172.21.15.96:0/4000793007 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d10142e40 con 0x7f7d10161e70 2024-05-28T06:26:26.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:26.649+0000 7f7d16400640 1 -- 172.21.15.96:0/4000793007 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d10143400 con 0x7f7d10161e70 2024-05-28T06:26:26.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:26.651+0000 7f7d16400640 1 -- 172.21.15.96:0/4000793007 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d100689e0 con 0x7f7d10161e70 2024-05-28T06:26:26.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:26.652+0000 7f7d07400640 1 -- 172.21.15.96:0/4000793007 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 49) ==== 94049+0+0 (secure 0 0 0) 0x7f7d08007bf0 con 0x7f7d10161e70 2024-05-28T06:26:26.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:26.652+0000 7f7d07400640 1 --2- 172.21.15.96:0/4000793007 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7cec06ea20 0x7f7cec070ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:26.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:26.653+0000 7f7d07400640 1 -- 172.21.15.96:0/4000793007 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 5823+0+0 (secure 0 0 0) 0x7f7d080ab9e0 con 0x7f7d10161e70 2024-05-28T06:26:26.645 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:26.653+0000 7f7d0f400640 1 --2- 172.21.15.96:0/4000793007 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7cec06ea20 0x7f7cec070ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:26.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:26.654+0000 7f7d0f400640 1 --2- 172.21.15.96:0/4000793007 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7cec06ea20 0x7f7cec070ee0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f7d000099a0 tx=0x7f7d00008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:26.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:26.657+0000 7f7d07400640 1 -- 172.21.15.96:0/4000793007 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7d08045360 con 0x7f7d10161e70 2024-05-28T06:26:26.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:26.897+0000 7f7d16400640 1 -- 172.21.15.96:0/4000793007 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f7d100b1400 con 0x7f7cec06ea20 2024-05-28T06:26:26.903 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:26:26.909+0000 7fea59e00640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-05-28T06:26:26.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:26.911+0000 7f7d07400640 1 -- 172.21.15.96:0/4000793007 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) ==== 41+0+0 (secure 0 0 0) 0x7f7d100b1400 con 0x7f7cec06ea20 2024-05-28T06:26:26.904 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:26:26.904 INFO:teuthology.orchestra.run.smithi096.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-05-28T06:26:26.918 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-05-28T06:26:28.920 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-05-28T06:26:29.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:29.144+0000 7f8faa600640 1 Processor -- start 2024-05-28T06:26:29.137 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:29.144+0000 7f8faa600640 1 -- start start 2024-05-28T06:26:29.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:29.146+0000 7f8faa600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f8fa4153510 0x7f8fa40b4d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:29.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:29.146+0000 7f8faa600640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8fa415d900 con 0x7f8fa40ccb90 2024-05-28T06:26:29.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:29.146+0000 7f8faa600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8fa40b3f90 con 0x7f8fa415d270 2024-05-28T06:26:29.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:29.146+0000 7f8faa600640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f8fa41538f0 con 0x7f8fa4153510 2024-05-28T06:26:29.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:29.146+0000 7f8fa9200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f8fa4153510 0x7f8fa40b4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:29.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:29.146+0000 7f8fa3e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f8fa40ccb90 0x7f8fa415cb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:35514/0 (socket says 172.21.15.96:35514) 2024-05-28T06:26:29.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:29.146+0000 7f8fa3e00640 1 -- 172.21.15.96:0/2614908624 learned_addr learned my addr 172.21.15.96:0/2614908624 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:26:29.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:29.147+0000 7f8fa9200640 1 -- 172.21.15.96:0/2614908624 >> v1:172.21.15.123:6789/0 conn(0x7f8fa415d270 legacy=0x7f8fa406e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:29.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:29.147+0000 7f8fa9200640 1 -- 172.21.15.96:0/2614908624 >> v1:172.21.15.96:6789/0 conn(0x7f8fa40ccb90 legacy=0x7f8fa415cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:29.139 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:29.147+0000 7f8fa9200640 1 -- 172.21.15.96:0/2614908624 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fa40b4270 con 0x7f8fa4153510 2024-05-28T06:26:29.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:29.147+0000 7f8fa3400640 1 -- 172.21.15.96:0/2614908624 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 805680041 0 0) 0x7f8fa415d900 con 0x7f8fa40ccb90 2024-05-28T06:26:29.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:29.148+0000 7f8fa9200640 1 --2- 172.21.15.96:0/2614908624 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8fa4153510 0x7f8fa40b4d50 secure :-1 s=READY pgs=2141 cs=0 l=1 rev1=1 crypto rx=0x7f8f980097a0 tx=0x7f8f9802a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=baec5bc8fb4d3007 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:29.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:29.148+0000 7f8fa3400640 1 -- 172.21.15.96:0/2614908624 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f98037070 con 0x7f8fa4153510 2024-05-28T06:26:29.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:29.148+0000 7f8fa3400640 1 -- 172.21.15.96:0/2614908624 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f9802ae50 con 0x7f8fa4153510 2024-05-28T06:26:29.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:29.148+0000 7f8fa3400640 1 -- 172.21.15.96:0/2614908624 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f98032660 con 0x7f8fa4153510 2024-05-28T06:26:29.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:29.149+0000 7f8faa600640 1 -- 172.21.15.96:0/2614908624 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8fa4153510 msgr2=0x7f8fa40b4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:29.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:29.149+0000 7f8faa600640 1 --2- 172.21.15.96:0/2614908624 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8fa4153510 0x7f8fa40b4d50 secure :-1 s=READY pgs=2141 cs=0 l=1 rev1=1 crypto rx=0x7f8f980097a0 tx=0x7f8f9802a140 comp rx=0 tx=0).stop 2024-05-28T06:26:29.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:29.149+0000 7f8faa600640 1 -- 172.21.15.96:0/2614908624 shutdown_connections 2024-05-28T06:26:29.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:29.149+0000 7f8faa600640 1 --2- 172.21.15.96:0/2614908624 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8fa4153510 0x7f8fa40b4d50 unknown :-1 s=CLOSED pgs=2141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:29.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:29.149+0000 7f8faa600640 1 -- 172.21.15.96:0/2614908624 >> 172.21.15.96:0/2614908624 conn(0x7f8fa406b900 msgr2=0x7f8fa406bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:26:29.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:29.149+0000 7f8faa600640 1 -- 172.21.15.96:0/2614908624 shutdown_connections 2024-05-28T06:26:29.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:29.149+0000 7f8faa600640 1 -- 172.21.15.96:0/2614908624 wait complete. 2024-05-28T06:26:29.143 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:29.151+0000 7f8faa600640 1 Processor -- start 2024-05-28T06:26:29.143 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:29.151+0000 7f8faa600640 1 -- start start 2024-05-28T06:26:29.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:29.152+0000 7f8faa600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8fa40ccb90 0x7f8fa4072970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:29.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:29.152+0000 7f8faa600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8fa415d270 0x7f8fa40aef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:29.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:29.152+0000 7f8faa600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8fa40ad130 0x7f8fa41484d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:29.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:29.152+0000 7f8faa600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f8fa415d6a0 con 0x7f8fa40ad130 2024-05-28T06:26:29.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:29.152+0000 7f8faa600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f8fa40b48c0 con 0x7f8fa40ccb90 2024-05-28T06:26:29.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:29.152+0000 7f8faa600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f8fa40ccff0 con 0x7f8fa415d270 2024-05-28T06:26:29.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:29.152+0000 7f8fa9c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8fa40ad130 0x7f8fa41484d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:29.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:29.153+0000 7f8fa9c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8fa40ad130 0x7f8fa41484d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:43998/0 (socket says 172.21.15.96:43998) 2024-05-28T06:26:29.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:29.153+0000 7f8fa9c00640 1 -- 172.21.15.96:0/3800222470 learned_addr learned my addr 172.21.15.96:0/3800222470 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:26:29.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:29.153+0000 7f8fa3e00640 1 --2- 172.21.15.96:0/3800222470 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8fa415d270 0x7f8fa40aef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:29.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:29.153+0000 7f8fa9200640 1 --2- 172.21.15.96:0/3800222470 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8fa40ccb90 0x7f8fa4072970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:29.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:29.153+0000 7f8fa9c00640 1 -- 172.21.15.96:0/3800222470 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8fa415d270 msgr2=0x7f8fa40aef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:29.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:29.153+0000 7f8fa9c00640 1 --2- 172.21.15.96:0/3800222470 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8fa415d270 0x7f8fa40aef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:29.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:29.153+0000 7f8fa9c00640 1 -- 172.21.15.96:0/3800222470 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8fa40ccb90 msgr2=0x7f8fa4072970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:29.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:29.153+0000 7f8fa9c00640 1 --2- 172.21.15.96:0/3800222470 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8fa40ccb90 0x7f8fa4072970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:29.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:29.153+0000 7f8fa9c00640 1 -- 172.21.15.96:0/3800222470 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fa40ad5e0 con 0x7f8fa40ad130 2024-05-28T06:26:29.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:29.153+0000 7f8fa9c00640 1 --2- 172.21.15.96:0/3800222470 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8fa40ad130 0x7f8fa41484d0 secure :-1 s=READY pgs=2142 cs=0 l=1 rev1=1 crypto rx=0x7f8f9c00b970 tx=0x7f8f9c00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:29.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:29.153+0000 7f8fa0c00640 1 -- 172.21.15.96:0/3800222470 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f9c019070 con 0x7f8fa40ad130 2024-05-28T06:26:29.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:29.154+0000 7f8fa0c00640 1 -- 172.21.15.96:0/3800222470 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f9c00cb40 con 0x7f8fa40ad130 2024-05-28T06:26:29.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:29.154+0000 7f8faa600640 1 -- 172.21.15.96:0/3800222470 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fa4148a10 con 0x7f8fa40ad130 2024-05-28T06:26:29.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:29.154+0000 7f8fa0c00640 1 -- 172.21.15.96:0/3800222470 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f9c014440 con 0x7f8fa40ad130 2024-05-28T06:26:29.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:29.154+0000 7f8faa600640 1 -- 172.21.15.96:0/3800222470 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fa4071e90 con 0x7f8fa40ad130 2024-05-28T06:26:29.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:29.155+0000 7f8faa600640 1 -- 172.21.15.96:0/3800222470 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8fa40689e0 con 0x7f8fa40ad130 2024-05-28T06:26:29.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:29.156+0000 7f8fa0c00640 1 -- 172.21.15.96:0/3800222470 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 49) ==== 94049+0+0 (secure 0 0 0) 0x7f8f9c01c070 con 0x7f8fa40ad130 2024-05-28T06:26:29.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:29.157+0000 7f8fa0c00640 1 --2- 172.21.15.96:0/3800222470 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8f8406ea20 0x7f8f84070ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:29.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:29.157+0000 7f8fa0c00640 1 -- 172.21.15.96:0/3800222470 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 5823+0+0 (secure 0 0 0) 0x7f8f9c08a960 con 0x7f8fa40ad130 2024-05-28T06:26:29.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:29.158+0000 7f8fa9200640 1 --2- 172.21.15.96:0/3800222470 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8f8406ea20 0x7f8f84070ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:29.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:29.159+0000 7f8fa9200640 1 --2- 172.21.15.96:0/3800222470 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f8f8406ea20 0x7f8f84070ee0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f8f98034000 tx=0x7f8f9802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:29.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:29.161+0000 7f8fa0c00640 1 -- 172.21.15.96:0/3800222470 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8f9c05a7f0 con 0x7f8fa40ad130 2024-05-28T06:26:29.384 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:29.391+0000 7f8faa600640 1 -- 172.21.15.96:0/3800222470 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f8fa40b3c40 con 0x7f8f8406ea20 2024-05-28T06:26:29.398 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:26:29.405+0000 7fea59e00640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-05-28T06:26:29.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:29.406+0000 7f8fa0c00640 1 -- 172.21.15.96:0/3800222470 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) ==== 41+0+0 (secure 0 0 0) 0x7f8fa40b3c40 con 0x7f8f8406ea20 2024-05-28T06:26:29.399 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:26:29.399 INFO:teuthology.orchestra.run.smithi096.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-05-28T06:26:29.413 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-05-28T06:26:31.415 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-05-28T06:26:31.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:31.642+0000 7feeddc00640 1 Processor -- start 2024-05-28T06:26:31.635 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:31.642+0000 7feeddc00640 1 -- start start 2024-05-28T06:26:31.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:31.644+0000 7feeddc00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7feed80d0640 0x7feed815d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:31.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:31.644+0000 7feeddc00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7feed80b32c0 0x7feed80b3750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:31.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:31.644+0000 7feeddc00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feed80d0060 con 0x7feed80d0a20 2024-05-28T06:26:31.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:31.644+0000 7feeddc00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7feed80adbb0 con 0x7feed80b32c0 2024-05-28T06:26:31.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:31.644+0000 7feeddc00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7feed815db10 con 0x7feed80d0640 2024-05-28T06:26:31.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:31.645+0000 7feed6a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7feed80d0640 0x7feed815d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:31.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:31.645+0000 7feed7400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7feed80b32c0 0x7feed80b3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:31.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:31.645+0000 7feed6000640 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7feed80d0a20 0x7feed80cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.96:55872/0 (socket says 172.21.15.96:55872) 2024-05-28T06:26:31.638 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:31.645+0000 7feed7400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7feed80b32c0 0x7feed80b3750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.96:45552/0 (socket says 172.21.15.96:45552) 2024-05-28T06:26:31.638 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:31.645+0000 7feed6000640 1 -- 172.21.15.96:0/1974836828 learned_addr learned my addr 172.21.15.96:0/1974836828 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:26:31.638 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:31.646+0000 7feed7400640 1 -- 172.21.15.96:0/1974836828 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7feed80d0640 msgr2=0x7feed815d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:31.638 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:31.646+0000 7feed7400640 1 --2- 172.21.15.96:0/1974836828 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7feed80d0640 0x7feed815d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:31.639 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:31.646+0000 7feed7400640 1 -- 172.21.15.96:0/1974836828 >> v1:172.21.15.123:6789/0 conn(0x7feed80d0a20 legacy=0x7feed80cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:31.639 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:31.646+0000 7feed7400640 1 -- 172.21.15.96:0/1974836828 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feed815cb10 con 0x7feed80b32c0 2024-05-28T06:26:31.639 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:31.646+0000 7feed6a00640 1 --2- 172.21.15.96:0/1974836828 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7feed80d0640 0x7feed815d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-28T06:26:31.639 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:31.647+0000 7feed7400640 1 --2- 172.21.15.96:0/1974836828 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feed80b32c0 0x7feed80b3750 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7feed0009d90 tx=0x7feed002a660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e2eac6d2bb9eeb3e server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:31.640 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:31.648+0000 7feed5600640 1 -- 172.21.15.96:0/1974836828 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feed0007740 con 0x7feed80b32c0 2024-05-28T06:26:31.640 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:31.648+0000 7feed5600640 1 -- 172.21.15.96:0/1974836828 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feed00078e0 con 0x7feed80b32c0 2024-05-28T06:26:31.640 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:31.648+0000 7feed5600640 1 -- 172.21.15.96:0/1974836828 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feed0007cb0 con 0x7feed80b32c0 2024-05-28T06:26:31.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:31.648+0000 7feeddc00640 1 -- 172.21.15.96:0/1974836828 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feed80b32c0 msgr2=0x7feed80b3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:31.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:31.648+0000 7feeddc00640 1 --2- 172.21.15.96:0/1974836828 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feed80b32c0 0x7feed80b3750 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7feed0009d90 tx=0x7feed002a660 comp rx=0 tx=0).stop 2024-05-28T06:26:31.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:31.648+0000 7feeddc00640 1 -- 172.21.15.96:0/1974836828 shutdown_connections 2024-05-28T06:26:31.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:31.648+0000 7feeddc00640 1 --2- 172.21.15.96:0/1974836828 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feed80b32c0 0x7feed80b3750 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:31.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:31.648+0000 7feeddc00640 1 --2- 172.21.15.96:0/1974836828 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7feed80d0640 0x7feed815d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:31.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:31.648+0000 7feeddc00640 1 -- 172.21.15.96:0/1974836828 >> 172.21.15.96:0/1974836828 conn(0x7feed806bd20 msgr2=0x7feed806c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:26:31.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:31.649+0000 7feeddc00640 1 -- 172.21.15.96:0/1974836828 shutdown_connections 2024-05-28T06:26:31.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:31.649+0000 7feeddc00640 1 -- 172.21.15.96:0/1974836828 wait complete. 2024-05-28T06:26:31.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:31.651+0000 7feeddc00640 1 Processor -- start 2024-05-28T06:26:31.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:31.651+0000 7feeddc00640 1 -- start start 2024-05-28T06:26:31.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:31.652+0000 7feeddc00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feed80b32c0 0x7feed809ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:31.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:31.652+0000 7feeddc00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7feed80d0640 0x7feed81f7cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:31.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:31.652+0000 7feeddc00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feed80d0a20 0x7feed81f81f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:31.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:31.652+0000 7feeddc00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7feed815cdd0 con 0x7feed80d0a20 2024-05-28T06:26:31.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:31.652+0000 7feeddc00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7feed80ad8d0 con 0x7feed80b32c0 2024-05-28T06:26:31.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:31.652+0000 7feeddc00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7feed815d0b0 con 0x7feed80d0640 2024-05-28T06:26:31.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:31.653+0000 7feed7400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feed80d0a20 0x7feed81f81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:31.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:31.653+0000 7feed7400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feed80d0a20 0x7feed81f81f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44008/0 (socket says 172.21.15.96:44008) 2024-05-28T06:26:31.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:31.653+0000 7feed7400640 1 -- 172.21.15.96:0/268613498 learned_addr learned my addr 172.21.15.96:0/268613498 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:26:31.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:31.653+0000 7feed6000640 1 --2- 172.21.15.96:0/268613498 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7feed80d0640 0x7feed81f7cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:31.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:31.653+0000 7feed6a00640 1 --2- 172.21.15.96:0/268613498 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feed80b32c0 0x7feed809ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:31.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:31.653+0000 7feed7400640 1 -- 172.21.15.96:0/268613498 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7feed80d0640 msgr2=0x7feed81f7cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:31.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:31.653+0000 7feed7400640 1 --2- 172.21.15.96:0/268613498 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7feed80d0640 0x7feed81f7cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:31.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:31.653+0000 7feed7400640 1 -- 172.21.15.96:0/268613498 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feed80b32c0 msgr2=0x7feed809ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:31.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:31.653+0000 7feed7400640 1 --2- 172.21.15.96:0/268613498 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feed80b32c0 0x7feed809ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:31.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:31.653+0000 7feed7400640 1 -- 172.21.15.96:0/268613498 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feed809b5c0 con 0x7feed80d0a20 2024-05-28T06:26:31.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:31.653+0000 7feed6a00640 1 --2- 172.21.15.96:0/268613498 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feed80b32c0 0x7feed809ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:26:31.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:31.653+0000 7feed7400640 1 --2- 172.21.15.96:0/268613498 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feed80d0a20 0x7feed81f81f0 secure :-1 s=READY pgs=2143 cs=0 l=1 rev1=1 crypto rx=0x7feed0009eb0 tx=0x7feed0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:31.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:31.654+0000 7feecea00640 1 -- 172.21.15.96:0/268613498 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feed0040070 con 0x7feed80d0a20 2024-05-28T06:26:31.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:31.654+0000 7feecea00640 1 -- 172.21.15.96:0/268613498 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feed0031bc0 con 0x7feed80d0a20 2024-05-28T06:26:31.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:31.654+0000 7feeddc00640 1 -- 172.21.15.96:0/268613498 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feed809d6e0 con 0x7feed80d0a20 2024-05-28T06:26:31.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:31.654+0000 7feecea00640 1 -- 172.21.15.96:0/268613498 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feed0009450 con 0x7feed80d0a20 2024-05-28T06:26:31.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:31.654+0000 7feeddc00640 1 -- 172.21.15.96:0/268613498 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feed809dca0 con 0x7feed80d0a20 2024-05-28T06:26:31.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:31.655+0000 7feeddc00640 1 -- 172.21.15.96:0/268613498 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feed80689e0 con 0x7feed80d0a20 2024-05-28T06:26:31.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:31.656+0000 7feecea00640 1 -- 172.21.15.96:0/268613498 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 49) ==== 94049+0+0 (secure 0 0 0) 0x7feed0039040 con 0x7feed80d0a20 2024-05-28T06:26:31.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:31.657+0000 7feecea00640 1 --2- 172.21.15.96:0/268613498 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7feeb806ea20 0x7feeb8070ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:31.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:31.657+0000 7feecea00640 1 -- 172.21.15.96:0/268613498 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 5823+0+0 (secure 0 0 0) 0x7feed00ac5c0 con 0x7feed80d0a20 2024-05-28T06:26:31.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:31.658+0000 7feed6a00640 1 --2- 172.21.15.96:0/268613498 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7feeb806ea20 0x7feeb8070ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:31.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:31.659+0000 7feed6a00640 1 --2- 172.21.15.96:0/268613498 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7feeb806ea20 0x7feeb8070ee0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7feec8009ac0 tx=0x7feec8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:31.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:31.662+0000 7feecea00640 1 -- 172.21.15.96:0/268613498 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7feed007c450 con 0x7feed80d0a20 2024-05-28T06:26:31.898 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:31.905+0000 7feeddc00640 1 -- 172.21.15.96:0/268613498 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7feed8071ef0 con 0x7feeb806ea20 2024-05-28T06:26:31.913 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:26:31.920+0000 7fea59e00640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-05-28T06:26:31.914 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:31.921+0000 7feecea00640 1 -- 172.21.15.96:0/268613498 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) ==== 41+0+0 (secure 0 0 0) 0x7feed8071ef0 con 0x7feeb806ea20 2024-05-28T06:26:31.914 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:26:31.915 INFO:teuthology.orchestra.run.smithi096.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-05-28T06:26:31.934 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-05-28T06:26:33.936 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-05-28T06:26:34.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:34.165+0000 7f920e000640 1 Processor -- start 2024-05-28T06:26:34.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:34.165+0000 7f920e000640 1 -- start start 2024-05-28T06:26:34.159 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:34.166+0000 7f920e000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f920815d5d0 0x7f92080cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:34.159 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:34.166+0000 7f920e000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f92080b32c0 0x7f92080b3750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:34.159 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:34.166+0000 7f920e000640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f920815cb10 con 0x7f92080d0640 2024-05-28T06:26:34.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:34.166+0000 7f920e000640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f92080adbb0 con 0x7f920815d5d0 2024-05-28T06:26:34.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:34.166+0000 7f920e000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f92080ad5f0 con 0x7f92080b32c0 2024-05-28T06:26:34.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:34.166+0000 7f9206a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f920815d5d0 0x7f92080cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:34.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:34.167+0000 7f9206a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f920815d5d0 0x7f92080cf950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44012/0 (socket says 172.21.15.96:44012) 2024-05-28T06:26:34.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:34.167+0000 7f9206a00640 1 -- 172.21.15.96:0/3856172264 learned_addr learned my addr 172.21.15.96:0/3856172264 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:26:34.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:34.167+0000 7f9207400640 1 --1- 172.21.15.96:0/3856172264 >> v1:172.21.15.96:6789/0 conn(0x7f92080d0640 0x7f92080d0a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:35524/0 (socket says 172.21.15.96:35524) 2024-05-28T06:26:34.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:34.167+0000 7f9207e00640 1 --2- 172.21.15.96:0/3856172264 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f92080b32c0 0x7f92080b3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:34.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:34.167+0000 7f9206a00640 1 -- 172.21.15.96:0/3856172264 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f92080b32c0 msgr2=0x7f92080b3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:34.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:34.167+0000 7f9206a00640 1 --2- 172.21.15.96:0/3856172264 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f92080b32c0 0x7f92080b3750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:34.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:34.167+0000 7f9206a00640 1 -- 172.21.15.96:0/3856172264 >> v1:172.21.15.96:6789/0 conn(0x7f92080d0640 legacy=0x7f92080d0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:34.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:34.167+0000 7f9206a00640 1 -- 172.21.15.96:0/3856172264 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92080ad8d0 con 0x7f920815d5d0 2024-05-28T06:26:34.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:34.168+0000 7f9206a00640 1 --2- 172.21.15.96:0/3856172264 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f920815d5d0 0x7f92080cf950 secure :-1 s=READY pgs=2145 cs=0 l=1 rev1=1 crypto rx=0x7f91f0009870 tx=0x7f91f002a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=524b3883c7ab3d2b server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:34.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:34.168+0000 7f9206000640 1 -- 172.21.15.96:0/3856172264 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91f0037070 con 0x7f920815d5d0 2024-05-28T06:26:34.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:34.168+0000 7f9206000640 1 -- 172.21.15.96:0/3856172264 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91f0032440 con 0x7f920815d5d0 2024-05-28T06:26:34.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:34.168+0000 7f9206000640 1 -- 172.21.15.96:0/3856172264 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91f0032810 con 0x7f920815d5d0 2024-05-28T06:26:34.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:34.169+0000 7f920e000640 1 -- 172.21.15.96:0/3856172264 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f920815d5d0 msgr2=0x7f92080cf950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:34.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:34.169+0000 7f920e000640 1 --2- 172.21.15.96:0/3856172264 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f920815d5d0 0x7f92080cf950 secure :-1 s=READY pgs=2145 cs=0 l=1 rev1=1 crypto rx=0x7f91f0009870 tx=0x7f91f002a250 comp rx=0 tx=0).stop 2024-05-28T06:26:34.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:34.169+0000 7f920e000640 1 -- 172.21.15.96:0/3856172264 shutdown_connections 2024-05-28T06:26:34.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:34.169+0000 7f920e000640 1 --2- 172.21.15.96:0/3856172264 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f92080b32c0 0x7f92080b3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:34.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:34.169+0000 7f920e000640 1 --2- 172.21.15.96:0/3856172264 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f920815d5d0 0x7f92080cf950 unknown :-1 s=CLOSED pgs=2145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:34.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:34.169+0000 7f920e000640 1 -- 172.21.15.96:0/3856172264 >> 172.21.15.96:0/3856172264 conn(0x7f920806bd20 msgr2=0x7f920806c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:26:34.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:34.169+0000 7f920e000640 1 -- 172.21.15.96:0/3856172264 shutdown_connections 2024-05-28T06:26:34.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:34.169+0000 7f920e000640 1 -- 172.21.15.96:0/3856172264 wait complete. 2024-05-28T06:26:34.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:34.171+0000 7f920e000640 1 Processor -- start 2024-05-28T06:26:34.164 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:34.171+0000 7f920e000640 1 -- start start 2024-05-28T06:26:34.166 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:34.172+0000 7f920e000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f92080b32c0 0x7f92080998f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:34.166 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:34.172+0000 7f920e000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92080d0640 0x7f92080b2760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:34.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:34.172+0000 7f920e000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f920815d5d0 0x7f920814d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:34.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:34.172+0000 7f920e000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f920815da50 con 0x7f920815d5d0 2024-05-28T06:26:34.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:34.172+0000 7f920e000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f920815d0b0 con 0x7f92080d0640 2024-05-28T06:26:34.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:34.172+0000 7f920e000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f92080ad2a0 con 0x7f92080b32c0 2024-05-28T06:26:34.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:34.172+0000 7f9207e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f920815d5d0 0x7f920814d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:34.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:34.172+0000 7f9207e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f920815d5d0 0x7f920814d260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44026/0 (socket says 172.21.15.96:44026) 2024-05-28T06:26:34.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:34.172+0000 7f9207e00640 1 -- 172.21.15.96:0/4154152838 learned_addr learned my addr 172.21.15.96:0/4154152838 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:26:34.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:34.173+0000 7f9207e00640 1 -- 172.21.15.96:0/4154152838 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f92080b32c0 msgr2=0x7f92080998f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:34.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:34.173+0000 7f9207400640 1 --2- 172.21.15.96:0/4154152838 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f92080b32c0 0x7f92080998f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:34.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:34.173+0000 7f9206a00640 1 --2- 172.21.15.96:0/4154152838 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92080d0640 0x7f92080b2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:34.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:34.173+0000 7f9207e00640 1 --2- 172.21.15.96:0/4154152838 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f92080b32c0 0x7f92080998f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:34.167 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:34.173+0000 7f9207e00640 1 -- 172.21.15.96:0/4154152838 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92080d0640 msgr2=0x7f92080b2760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:34.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:34.173+0000 7f9207e00640 1 --2- 172.21.15.96:0/4154152838 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92080d0640 0x7f92080b2760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:34.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:34.173+0000 7f9207e00640 1 -- 172.21.15.96:0/4154152838 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f920814bee0 con 0x7f920815d5d0 2024-05-28T06:26:34.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:34.173+0000 7f9207e00640 1 --2- 172.21.15.96:0/4154152838 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f920815d5d0 0x7f920814d260 secure :-1 s=READY pgs=2146 cs=0 l=1 rev1=1 crypto rx=0x7f9200007e40 tx=0x7f920000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:34.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:34.173+0000 7f91ff400640 1 -- 172.21.15.96:0/4154152838 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9200014620 con 0x7f920815d5d0 2024-05-28T06:26:34.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:34.173+0000 7f91ff400640 1 -- 172.21.15.96:0/4154152838 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9200019070 con 0x7f920815d5d0 2024-05-28T06:26:34.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:34.173+0000 7f91ff400640 1 -- 172.21.15.96:0/4154152838 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9200014a20 con 0x7f920815d5d0 2024-05-28T06:26:34.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:34.173+0000 7f920e000640 1 -- 172.21.15.96:0/4154152838 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f920814c1c0 con 0x7f920815d5d0 2024-05-28T06:26:34.168 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:34.174+0000 7f920e000640 1 -- 172.21.15.96:0/4154152838 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f920814c780 con 0x7f920815d5d0 2024-05-28T06:26:34.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:34.175+0000 7f920e000640 1 -- 172.21.15.96:0/4154152838 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92080689e0 con 0x7f920815d5d0 2024-05-28T06:26:34.169 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:34.176+0000 7f91ff400640 1 -- 172.21.15.96:0/4154152838 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 49) ==== 94049+0+0 (secure 0 0 0) 0x7f9200014bc0 con 0x7f920815d5d0 2024-05-28T06:26:34.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:34.177+0000 7f91ff400640 1 --2- 172.21.15.96:0/4154152838 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f91e406ea20 0x7f91e4070ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:34.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:34.177+0000 7f91ff400640 1 -- 172.21.15.96:0/4154152838 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 5823+0+0 (secure 0 0 0) 0x7f920008b600 con 0x7f920815d5d0 2024-05-28T06:26:34.170 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:34.178+0000 7f9207400640 1 --2- 172.21.15.96:0/4154152838 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f91e406ea20 0x7f91e4070ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:34.171 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:34.179+0000 7f9207400640 1 --2- 172.21.15.96:0/4154152838 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f91e406ea20 0x7f91e4070ee0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f91f8000f60 tx=0x7f91f8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:34.174 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:34.182+0000 7f91ff400640 1 -- 172.21.15.96:0/4154152838 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f920005b490 con 0x7f920815d5d0 2024-05-28T06:26:34.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:34.414+0000 7f920e000640 1 -- 172.21.15.96:0/4154152838 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f92080b4d40 con 0x7f91e406ea20 2024-05-28T06:26:34.419 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:26:34.426+0000 7fea59e00640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-05-28T06:26:34.419 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:34.427+0000 7f91ff400640 1 -- 172.21.15.96:0/4154152838 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) ==== 41+0+0 (secure 0 0 0) 0x7f92080b4d40 con 0x7f91e406ea20 2024-05-28T06:26:34.420 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:26:34.420 INFO:teuthology.orchestra.run.smithi096.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-05-28T06:26:34.438 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-05-28T06:26:35.470 DEBUG:teuthology.orchestra.run.smithi096:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:26:35.479 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:26:36.440 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-05-28T06:26:36.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:36.654+0000 7f06f8a00640 1 Processor -- start 2024-05-28T06:26:36.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:36.654+0000 7f06f8a00640 1 -- start start 2024-05-28T06:26:36.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:36.656+0000 7f06f8a00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f06f40ce1f0 0x7f06f40b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:36.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:36.656+0000 7f06f8a00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f06f40b3d60 0x7f06f40b4210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:36.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:36.656+0000 7f06f8a00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06f40b3580 con 0x7f06f40ce5d0 2024-05-28T06:26:36.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:36.656+0000 7f06f8a00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f06f415f490 con 0x7f06f40ce1f0 2024-05-28T06:26:36.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:36.656+0000 7f06f8a00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f06f415f770 con 0x7f06f40b3d60 2024-05-28T06:26:36.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:36.656+0000 7f06f2800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f06f40ce1f0 0x7f06f40b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:36.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:36.656+0000 7f06e9e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f06f40ce5d0 0x7f06f415ec40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:35534/0 (socket says 172.21.15.96:35534) 2024-05-28T06:26:36.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:36.656+0000 7f06e9e00640 1 -- 172.21.15.96:0/4288377661 learned_addr learned my addr 172.21.15.96:0/4288377661 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:26:36.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:36.657+0000 7f06f3200640 1 --2- 172.21.15.96:0/4288377661 >> v2:172.21.15.123:3300/0 conn(0x7f06f40b3d60 0x7f06f40b4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:36.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:36.657+0000 7f06f2800640 1 -- 172.21.15.96:0/4288377661 >> v2:172.21.15.123:3300/0 conn(0x7f06f40b3d60 msgr2=0x7f06f40b4210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:36.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:36.657+0000 7f06f2800640 1 --2- 172.21.15.96:0/4288377661 >> v2:172.21.15.123:3300/0 conn(0x7f06f40b3d60 0x7f06f40b4210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:36.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:36.657+0000 7f06f2800640 1 -- 172.21.15.96:0/4288377661 >> v1:172.21.15.96:6789/0 conn(0x7f06f40ce5d0 legacy=0x7f06f415ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:36.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:36.657+0000 7f06f2800640 1 -- 172.21.15.96:0/4288377661 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06f415fa50 con 0x7f06f40ce1f0 2024-05-28T06:26:36.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:36.658+0000 7f06f2800640 1 --2- 172.21.15.96:0/4288377661 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f06f40ce1f0 0x7f06f40b4dd0 secure :-1 s=READY pgs=2148 cs=0 l=1 rev1=1 crypto rx=0x7f06f40cf860 tx=0x7f06e402a550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33a3373ff27470c4 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:36.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:36.658+0000 7f06f1e00640 1 -- 172.21.15.96:0/4288377661 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06e4007740 con 0x7f06f40ce1f0 2024-05-28T06:26:36.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:36.658+0000 7f06f1e00640 1 -- 172.21.15.96:0/4288377661 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06e40078e0 con 0x7f06f40ce1f0 2024-05-28T06:26:36.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:36.658+0000 7f06f1e00640 1 -- 172.21.15.96:0/4288377661 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06e4007cb0 con 0x7f06f40ce1f0 2024-05-28T06:26:36.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:36.659+0000 7f06f8a00640 1 -- 172.21.15.96:0/4288377661 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f06f40ce1f0 msgr2=0x7f06f40b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:36.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:36.659+0000 7f06f8a00640 1 --2- 172.21.15.96:0/4288377661 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f06f40ce1f0 0x7f06f40b4dd0 secure :-1 s=READY pgs=2148 cs=0 l=1 rev1=1 crypto rx=0x7f06f40cf860 tx=0x7f06e402a550 comp rx=0 tx=0).stop 2024-05-28T06:26:36.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:36.659+0000 7f06f8a00640 1 -- 172.21.15.96:0/4288377661 shutdown_connections 2024-05-28T06:26:36.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:36.659+0000 7f06f8a00640 1 --2- 172.21.15.96:0/4288377661 >> v2:172.21.15.123:3300/0 conn(0x7f06f40b3d60 0x7f06f40b4210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:36.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:36.659+0000 7f06f8a00640 1 --2- 172.21.15.96:0/4288377661 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f06f40ce1f0 0x7f06f40b4dd0 unknown :-1 s=CLOSED pgs=2148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:36.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:36.659+0000 7f06f8a00640 1 -- 172.21.15.96:0/4288377661 >> 172.21.15.96:0/4288377661 conn(0x7f06f406b900 msgr2=0x7f06f406bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:26:36.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:36.659+0000 7f06f8a00640 1 -- 172.21.15.96:0/4288377661 shutdown_connections 2024-05-28T06:26:36.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:36.659+0000 7f06f8a00640 1 -- 172.21.15.96:0/4288377661 wait complete. 2024-05-28T06:26:36.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:36.661+0000 7f06f8a00640 1 Processor -- start 2024-05-28T06:26:36.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:36.661+0000 7f06f8a00640 1 -- start start 2024-05-28T06:26:36.657 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:36.662+0000 7f06f8a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f06f40b3d60 0x7f06f41627f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:36.657 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:36.662+0000 7f06f8a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f06f40ce1f0 0x7f06f4071a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:36.657 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:36.662+0000 7f06f8a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f06f40ce5d0 0x7f06f40b2580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:36.657 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:36.662+0000 7f06f8a00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f06f40ccd10 con 0x7f06f40b3d60 2024-05-28T06:26:36.657 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:36.662+0000 7f06f8a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f06f40ccb90 con 0x7f06f40ce5d0 2024-05-28T06:26:36.657 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:36.662+0000 7f06f8a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f06f40cce90 con 0x7f06f40ce1f0 2024-05-28T06:26:36.657 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:36.662+0000 7f06f2800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f06f40b3d60 0x7f06f41627f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:36.657 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:36.662+0000 7f06f2800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f06f40b3d60 0x7f06f41627f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44050/0 (socket says 172.21.15.96:44050) 2024-05-28T06:26:36.657 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:36.662+0000 7f06f2800640 1 -- 172.21.15.96:0/2076771329 learned_addr learned my addr 172.21.15.96:0/2076771329 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:26:36.657 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:36.662+0000 7f06f2800640 1 -- 172.21.15.96:0/2076771329 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f06f40ce1f0 msgr2=0x7f06f4071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:36.657 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:36.662+0000 7f06f2800640 1 --2- 172.21.15.96:0/2076771329 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f06f40ce1f0 0x7f06f4071a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:36.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:36.662+0000 7f06f2800640 1 -- 172.21.15.96:0/2076771329 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f06f40ce5d0 msgr2=0x7f06f40b2580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:36.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:36.662+0000 7f06f2800640 1 --2- 172.21.15.96:0/2076771329 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f06f40ce5d0 0x7f06f40b2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:36.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:36.662+0000 7f06f2800640 1 -- 172.21.15.96:0/2076771329 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06f40b2ac0 con 0x7f06f40b3d60 2024-05-28T06:26:36.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:36.663+0000 7f06f2800640 1 --2- 172.21.15.96:0/2076771329 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f06f40b3d60 0x7f06f41627f0 secure :-1 s=READY pgs=2149 cs=0 l=1 rev1=1 crypto rx=0x7f06e4009e80 tx=0x7f06e4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:36.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:36.663+0000 7f06eb400640 1 -- 172.21.15.96:0/2076771329 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06e4040070 con 0x7f06f40b3d60 2024-05-28T06:26:36.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:36.663+0000 7f06eb400640 1 -- 172.21.15.96:0/2076771329 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06e4031c40 con 0x7f06f40b3d60 2024-05-28T06:26:36.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:36.663+0000 7f06eb400640 1 -- 172.21.15.96:0/2076771329 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06e403c420 con 0x7f06f40b3d60 2024-05-28T06:26:36.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:36.663+0000 7f06f8a00640 1 -- 172.21.15.96:0/2076771329 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06f40721f0 con 0x7f06f40b3d60 2024-05-28T06:26:36.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:36.663+0000 7f06f8a00640 1 -- 172.21.15.96:0/2076771329 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06f40727b0 con 0x7f06f40b3d60 2024-05-28T06:26:36.659 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:36.664+0000 7f06f8a00640 1 -- 172.21.15.96:0/2076771329 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06f40689e0 con 0x7f06f40b3d60 2024-05-28T06:26:36.659 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:36.665+0000 7f06eb400640 1 -- 172.21.15.96:0/2076771329 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 49) ==== 94049+0+0 (secure 0 0 0) 0x7f06e4039040 con 0x7f06f40b3d60 2024-05-28T06:26:36.659 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:36.666+0000 7f06eb400640 1 --2- 172.21.15.96:0/2076771329 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f06d006ea20 0x7f06d0070ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:36.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:36.666+0000 7f06eb400640 1 -- 172.21.15.96:0/2076771329 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 5823+0+0 (secure 0 0 0) 0x7f06e40ac330 con 0x7f06f40b3d60 2024-05-28T06:26:36.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:36.667+0000 7f06e9e00640 1 --2- 172.21.15.96:0/2076771329 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f06d006ea20 0x7f06d0070ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:36.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:36.668+0000 7f06e9e00640 1 --2- 172.21.15.96:0/2076771329 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f06d006ea20 0x7f06d0070ee0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f06dc000f60 tx=0x7f06dc007a80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:36.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:36.671+0000 7f06eb400640 1 -- 172.21.15.96:0/2076771329 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f06e407c1c0 con 0x7f06f40b3d60 2024-05-28T06:26:36.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:36.909+0000 7f06f8a00640 1 -- 172.21.15.96:0/2076771329 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f06f40b4420 con 0x7f06d006ea20 2024-05-28T06:26:36.916 INFO:tasks.ceph.mgr.x.smithi123.stderr:2024-05-28T06:26:36.923+0000 7fea59e00640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-05-28T06:26:36.917 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:36.923+0000 7f06eb400640 1 -- 172.21.15.96:0/2076771329 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) ==== 41+0+0 (secure 0 0 0) 0x7f06f40b4420 con 0x7f06d006ea20 2024-05-28T06:26:36.917 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:26:36.917 INFO:teuthology.orchestra.run.smithi096.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-05-28T06:26:36.930 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-05-28T06:26:36.933 INFO:tasks.cephfs_test_runner:Test that both the progress bars, that are printed for all 7 clone ... ERROR 2024-05-28T06:26:36.934 INFO:tasks.cephfs_test_runner: 2024-05-28T06:26:36.934 INFO:tasks.cephfs_test_runner:====================================================================== 2024-05-28T06:26:36.934 INFO:tasks.cephfs_test_runner:ERROR: test_when_7_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgressReporter) 2024-05-28T06:26:36.934 INFO:tasks.cephfs_test_runner:Test that both the progress bars, that are printed for all 7 clone 2024-05-28T06:26:36.934 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-05-28T06:26:36.934 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-05-28T06:26:36.934 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_9ed025fbf4406fb54980a0647506b15895b6d863/qa/tasks/cephfs/test_volumes.py", line 7795, in tearDown 2024-05-28T06:26:36.934 INFO:tasks.cephfs_test_runner: self.loop_cmd_if_eagain('fs subvolume snapshot rm --force ' 2024-05-28T06:26:36.934 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_9ed025fbf4406fb54980a0647506b15895b6d863/qa/tasks/cephfs/test_volumes.py", line 7749, in loop_cmd_if_eagain 2024-05-28T06:26:36.934 INFO:tasks.cephfs_test_runner: raise RuntimeError(f'This cmd failed 10 times: {cmdargs}') 2024-05-28T06:26:36.934 INFO:tasks.cephfs_test_runner:RuntimeError: This cmd failed 10 times: fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-05-28T06:26:36.935 INFO:tasks.cephfs_test_runner: 2024-05-28T06:26:36.935 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-05-28T06:26:36.935 INFO:tasks.cephfs_test_runner:Ran 8 tests in 1306.252s 2024-05-28T06:26:36.935 INFO:tasks.cephfs_test_runner: 2024-05-28T06:26:36.935 INFO:tasks.cephfs_test_runner:FAILED (errors=1) 2024-05-28T06:26:36.935 INFO:tasks.cephfs_test_runner: 2024-05-28T06:26:36.935 INFO:tasks.cephfs_test_runner:====================================================================== 2024-05-28T06:26:36.935 INFO:tasks.cephfs_test_runner:ERROR: test_when_7_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgressReporter) 2024-05-28T06:26:36.935 INFO:tasks.cephfs_test_runner:Test that both the progress bars, that are printed for all 7 clone 2024-05-28T06:26:36.935 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-05-28T06:26:36.935 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-05-28T06:26:36.935 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_9ed025fbf4406fb54980a0647506b15895b6d863/qa/tasks/cephfs/test_volumes.py", line 7795, in tearDown 2024-05-28T06:26:36.936 INFO:tasks.cephfs_test_runner: self.loop_cmd_if_eagain('fs subvolume snapshot rm --force ' 2024-05-28T06:26:36.936 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_9ed025fbf4406fb54980a0647506b15895b6d863/qa/tasks/cephfs/test_volumes.py", line 7749, in loop_cmd_if_eagain 2024-05-28T06:26:36.936 INFO:tasks.cephfs_test_runner: raise RuntimeError(f'This cmd failed 10 times: {cmdargs}') 2024-05-28T06:26:36.936 INFO:tasks.cephfs_test_runner:RuntimeError: This cmd failed 10 times: fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-05-28T06:26:36.936 INFO:tasks.cephfs_test_runner: 2024-05-28T06:26:36.936 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_9ed025fbf4406fb54980a0647506b15895b6d863/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_when_7_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgressReporter) 2024-05-28T06:26:37.132 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=58bbfcc312174deeb0e2710034db0d5a Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_9ed025fbf4406fb54980a0647506b15895b6d863/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_when_7_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgressReporter) 2024-05-28T06:26:37.134 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2024-05-28T06:26:37.145 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-05-28T06:26:37.153 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-05-28T06:26:37.154 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:26:37.154 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:26:37.216 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:26:37.216 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:26:37.268 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:26:37.470 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.476+0000 7f2ad3000640 1 Processor -- start 2024-05-28T06:26:37.470 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.476+0000 7f2ad3000640 1 -- start start 2024-05-28T06:26:37.471 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.478+0000 7f2ad3000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f2acc0ad940 0x7f2acc0b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:37.471 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.478+0000 7f2ad3000640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2acc156f10 con 0x7f2acc06ee40 2024-05-28T06:26:37.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.478+0000 7f2ad3000640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2acc06dda0 con 0x7f2acc156b30 2024-05-28T06:26:37.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.478+0000 7f2ad3000640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f2acc06e080 con 0x7f2acc0ad940 2024-05-28T06:26:37.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.478+0000 7f2ad1200640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f2acc06ee40 0x7f2acc0ad230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:35536/0 (socket says 172.21.15.96:35536) 2024-05-28T06:26:37.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.478+0000 7f2ad1200640 1 -- 172.21.15.96:0/3545154187 learned_addr learned my addr 172.21.15.96:0/3545154187 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:26:37.472 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.478+0000 7f2ad2600640 1 --2- 172.21.15.96:0/3545154187 >> v2:172.21.15.96:3300/0 conn(0x7f2acc0ad940 0x7f2acc0b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:37.473 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.478+0000 7f2acbe00640 1 -- 172.21.15.96:0/3545154187 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2638062626 0 0) 0x7f2acc156f10 con 0x7f2acc06ee40 2024-05-28T06:26:37.473 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.478+0000 7f2acbe00640 1 -- 172.21.15.96:0/3545154187 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2ab4003520 con 0x7f2acc06ee40 2024-05-28T06:26:37.473 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.478+0000 7f2ad2600640 1 -- 172.21.15.96:0/3545154187 >> v1:172.21.15.123:6789/0 conn(0x7f2acc156b30 legacy=0x7f2acc0af6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:37.473 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.478+0000 7f2ad2600640 1 -- 172.21.15.96:0/3545154187 >> v1:172.21.15.96:6789/0 conn(0x7f2acc06ee40 legacy=0x7f2acc0ad230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:37.473 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.479+0000 7f2ad2600640 1 -- 172.21.15.96:0/3545154187 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2acc06e360 con 0x7f2acc0ad940 2024-05-28T06:26:37.475 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.480+0000 7f2ad2600640 1 --2- 172.21.15.96:0/3545154187 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2acc0ad940 0x7f2acc0b32c0 secure :-1 s=READY pgs=2151 cs=0 l=1 rev1=1 crypto rx=0x7f2ac4009d90 tx=0x7f2ac402a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76889a2daa63dd21 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:37.475 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.480+0000 7f2acbe00640 1 -- 172.21.15.96:0/3545154187 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ac4031600 con 0x7f2acc0ad940 2024-05-28T06:26:37.475 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.480+0000 7f2acbe00640 1 -- 172.21.15.96:0/3545154187 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ac40317a0 con 0x7f2acc0ad940 2024-05-28T06:26:37.475 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.480+0000 7f2acbe00640 1 -- 172.21.15.96:0/3545154187 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ac4031b50 con 0x7f2acc0ad940 2024-05-28T06:26:37.475 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.481+0000 7f2ad3000640 1 -- 172.21.15.96:0/3545154187 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2acc0ad940 msgr2=0x7f2acc0b32c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:37.475 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.481+0000 7f2ad3000640 1 --2- 172.21.15.96:0/3545154187 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2acc0ad940 0x7f2acc0b32c0 secure :-1 s=READY pgs=2151 cs=0 l=1 rev1=1 crypto rx=0x7f2ac4009d90 tx=0x7f2ac402a610 comp rx=0 tx=0).stop 2024-05-28T06:26:37.475 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.481+0000 7f2ad3000640 1 -- 172.21.15.96:0/3545154187 shutdown_connections 2024-05-28T06:26:37.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.481+0000 7f2ad3000640 1 --2- 172.21.15.96:0/3545154187 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2acc0ad940 0x7f2acc0b32c0 unknown :-1 s=CLOSED pgs=2151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:37.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.481+0000 7f2ad3000640 1 -- 172.21.15.96:0/3545154187 >> 172.21.15.96:0/3545154187 conn(0x7f2acc0b1fc0 msgr2=0x7f2acc0b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:26:37.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.481+0000 7f2ad3000640 1 -- 172.21.15.96:0/3545154187 shutdown_connections 2024-05-28T06:26:37.476 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.481+0000 7f2ad3000640 1 -- 172.21.15.96:0/3545154187 wait complete. 2024-05-28T06:26:37.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.483+0000 7f2ad3000640 1 Processor -- start 2024-05-28T06:26:37.477 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.483+0000 7f2ad3000640 1 -- start start 2024-05-28T06:26:37.478 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.484+0000 7f2ad3000640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2acc06ee40 0x7f2acc14d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:37.478 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.484+0000 7f2ad3000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2acc0ad940 0x7f2acc0b0450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:37.478 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.484+0000 7f2ad3000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2acc156b30 0x7f2acc0b4100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:37.478 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.484+0000 7f2ad3000640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f2acc0b2db0 con 0x7f2acc0ad940 2024-05-28T06:26:37.479 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.484+0000 7f2ad3000640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f2acc0b2c30 con 0x7f2acc06ee40 2024-05-28T06:26:37.479 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.484+0000 7f2ad3000640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f2acc06f2f0 con 0x7f2acc156b30 2024-05-28T06:26:37.479 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.485+0000 7f2ad1200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2acc0ad940 0x7f2acc0b0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:37.479 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.485+0000 7f2ad1200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2acc0ad940 0x7f2acc0b0450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44072/0 (socket says 172.21.15.96:44072) 2024-05-28T06:26:37.479 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.485+0000 7f2ad1200640 1 -- 172.21.15.96:0/3605204356 learned_addr learned my addr 172.21.15.96:0/3605204356 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:26:37.480 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.485+0000 7f2ad1c00640 1 --2- 172.21.15.96:0/3605204356 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2acc06ee40 0x7f2acc14d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:37.480 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.485+0000 7f2ad2600640 1 --2- 172.21.15.96:0/3605204356 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2acc156b30 0x7f2acc0b4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:37.481 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.485+0000 7f2ad1200640 1 -- 172.21.15.96:0/3605204356 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2acc156b30 msgr2=0x7f2acc0b4100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:37.481 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.485+0000 7f2ad1200640 1 --2- 172.21.15.96:0/3605204356 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2acc156b30 0x7f2acc0b4100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:37.481 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.485+0000 7f2ad1200640 1 -- 172.21.15.96:0/3605204356 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2acc06ee40 msgr2=0x7f2acc14d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:37.481 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.485+0000 7f2ad1200640 1 --2- 172.21.15.96:0/3605204356 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2acc06ee40 0x7f2acc14d620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:37.481 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.485+0000 7f2ad1200640 1 -- 172.21.15.96:0/3605204356 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2acc14fa20 con 0x7f2acc0ad940 2024-05-28T06:26:37.481 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.485+0000 7f2ad1200640 1 --2- 172.21.15.96:0/3605204356 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2acc0ad940 0x7f2acc0b0450 secure :-1 s=READY pgs=2152 cs=0 l=1 rev1=1 crypto rx=0x7f2ab800c430 tx=0x7f2ab800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:37.481 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.486+0000 7f2ac9600640 1 -- 172.21.15.96:0/3605204356 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ab8017070 con 0x7f2acc0ad940 2024-05-28T06:26:37.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.486+0000 7f2ac9600640 1 -- 172.21.15.96:0/3605204356 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ab8005ce0 con 0x7f2acc0ad940 2024-05-28T06:26:37.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.486+0000 7f2ac9600640 1 -- 172.21.15.96:0/3605204356 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ab80072f0 con 0x7f2acc0ad940 2024-05-28T06:26:37.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.486+0000 7f2ad3000640 1 -- 172.21.15.96:0/3605204356 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2acc14fd00 con 0x7f2acc0ad940 2024-05-28T06:26:37.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.486+0000 7f2ad3000640 1 -- 172.21.15.96:0/3605204356 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2acc1502c0 con 0x7f2acc0ad940 2024-05-28T06:26:37.482 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.487+0000 7f2ad3000640 1 -- 172.21.15.96:0/3605204356 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2acc0689e0 con 0x7f2acc0ad940 2024-05-28T06:26:37.483 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.488+0000 7f2ac9600640 1 -- 172.21.15.96:0/3605204356 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 49) ==== 94049+0+0 (secure 0 0 0) 0x7f2ab801a070 con 0x7f2acc0ad940 2024-05-28T06:26:37.483 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.489+0000 7f2ac9600640 1 --2- 172.21.15.96:0/3605204356 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2aac06ea20 0x7f2aac070ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:37.483 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.489+0000 7f2ac9600640 1 -- 172.21.15.96:0/3605204356 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 5823+0+0 (secure 0 0 0) 0x7f2ab808a1d0 con 0x7f2acc0ad940 2024-05-28T06:26:37.483 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.490+0000 7f2ad1c00640 1 --2- 172.21.15.96:0/3605204356 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2aac06ea20 0x7f2aac070ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:37.484 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.491+0000 7f2ad1c00640 1 --2- 172.21.15.96:0/3605204356 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2aac06ea20 0x7f2aac070ee0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f2ac000a750 tx=0x7f2ac0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:37.486 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.493+0000 7f2ac9600640 1 -- 172.21.15.96:0/3605204356 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2ab805a060 con 0x7f2acc0ad940 2024-05-28T06:26:37.673 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.679+0000 7f2ad3000640 1 -- 172.21.15.96:0/3605204356 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2acc15fb50 con 0x7f2acc0ad940 2024-05-28T06:26:37.675 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.682+0000 7f2ac9600640 1 -- 172.21.15.96:0/3605204356 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v132) ==== 74+0+16877 (secure 0 0 0) 0x7f2ab8059960 con 0x7f2acc0ad940 2024-05-28T06:26:37.676 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:26:37.676 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":132,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:24:54.967903+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-05-28T06:24:21.166134+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"129","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-05-28T06:24:22.415938+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"130","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"19.32","mappings":[{"from":2,"to":0}]},{"pgid":"19.3a","mappings":[{"from":2,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":13,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:26:37.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.684+0000 7f2ad3000640 1 -- 172.21.15.96:0/3605204356 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2aac06ea20 msgr2=0x7f2aac070ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:37.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.684+0000 7f2ad3000640 1 --2- 172.21.15.96:0/3605204356 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2aac06ea20 0x7f2aac070ee0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f2ac000a750 tx=0x7f2ac0008040 comp rx=0 tx=0).stop 2024-05-28T06:26:37.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.684+0000 7f2ad3000640 1 -- 172.21.15.96:0/3605204356 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2acc0ad940 msgr2=0x7f2acc0b0450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:37.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.684+0000 7f2ad3000640 1 --2- 172.21.15.96:0/3605204356 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2acc0ad940 0x7f2acc0b0450 secure :-1 s=READY pgs=2152 cs=0 l=1 rev1=1 crypto rx=0x7f2ab800c430 tx=0x7f2ab800c8f0 comp rx=0 tx=0).stop 2024-05-28T06:26:37.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.684+0000 7f2ad3000640 1 -- 172.21.15.96:0/3605204356 shutdown_connections 2024-05-28T06:26:37.678 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.684+0000 7f2ad3000640 1 --2- 172.21.15.96:0/3605204356 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2aac06ea20 0x7f2aac070ee0 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:37.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.684+0000 7f2ad3000640 1 --2- 172.21.15.96:0/3605204356 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2acc156b30 0x7f2acc0b4100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:37.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.684+0000 7f2ad3000640 1 --2- 172.21.15.96:0/3605204356 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2acc0ad940 0x7f2acc0b0450 unknown :-1 s=CLOSED pgs=2152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:37.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.684+0000 7f2ad3000640 1 --2- 172.21.15.96:0/3605204356 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2acc06ee40 0x7f2acc14d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:37.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.684+0000 7f2ad3000640 1 -- 172.21.15.96:0/3605204356 >> 172.21.15.96:0/3605204356 conn(0x7f2acc0b1fc0 msgr2=0x7f2acc0ae1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:26:37.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.684+0000 7f2ad3000640 1 -- 172.21.15.96:0/3605204356 shutdown_connections 2024-05-28T06:26:37.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.685+0000 7f2ad3000640 1 -- 172.21.15.96:0/3605204356 wait complete. 2024-05-28T06:26:37.696 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-28T06:03:59.517102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-28T06:24:21.166134+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '129', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 19, 'pool_name': 'cephfs_data', 'create_time': '2024-05-28T06:24:22.415938+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '130', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-28T06:26:37.697 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-28T06:26:37.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.905+0000 7ff3dfc00640 1 Processor -- start 2024-05-28T06:26:37.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.905+0000 7ff3dfc00640 1 -- start start 2024-05-28T06:26:37.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.907+0000 7ff3dfc00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff3d80c9e00 0x7ff3d8141f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:37.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.907+0000 7ff3dfc00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7ff3d806cde0 0x7ff3d80ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:37.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.907+0000 7ff3dfc00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7ff3d806bc70 0x7ff3d806c100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:37.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.907+0000 7ff3dfc00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7ff3d8163270 con 0x7ff3d806cde0 2024-05-28T06:26:37.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.907+0000 7ff3dfc00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7ff3d80cd360 con 0x7ff3d806bc70 2024-05-28T06:26:37.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.907+0000 7ff3dfc00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ff3d80ca720 con 0x7ff3d80c9e00 2024-05-28T06:26:37.901 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.907+0000 7ff3d7e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7ff3d806cde0 0x7ff3d80ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:37.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.907+0000 7ff3d7e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7ff3d806cde0 0x7ff3d80ca1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44084/0 (socket says 172.21.15.96:44084) 2024-05-28T06:26:37.902 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.907+0000 7ff3d7e00640 1 -- 172.21.15.96:0/2792442711 learned_addr learned my addr 172.21.15.96:0/2792442711 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:26:37.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.908+0000 7ff3ddc00640 1 --2- 172.21.15.96:0/2792442711 >> v2:172.21.15.123:3300/0 conn(0x7ff3d806bc70 0x7ff3d806c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:37.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.908+0000 7ff3dd200640 1 --2- 172.21.15.96:0/2792442711 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff3d80c9e00 0x7ff3d8141f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:37.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.908+0000 7ff3d7e00640 1 -- 172.21.15.96:0/2792442711 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff3d80c9e00 msgr2=0x7ff3d8141f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:37.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.908+0000 7ff3d7e00640 1 --2- 172.21.15.96:0/2792442711 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff3d80c9e00 0x7ff3d8141f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:37.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.908+0000 7ff3d7e00640 1 -- 172.21.15.96:0/2792442711 >> v2:172.21.15.123:3300/0 conn(0x7ff3d806bc70 msgr2=0x7ff3d806c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:37.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.908+0000 7ff3d7e00640 1 --2- 172.21.15.96:0/2792442711 >> v2:172.21.15.123:3300/0 conn(0x7ff3d806bc70 0x7ff3d806c100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:37.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.908+0000 7ff3d7e00640 1 -- 172.21.15.96:0/2792442711 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3d81626f0 con 0x7ff3d806cde0 2024-05-28T06:26:37.904 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.908+0000 7ff3ddc00640 1 --2- 172.21.15.96:0/2792442711 >> v2:172.21.15.123:3300/0 conn(0x7ff3d806bc70 0x7ff3d806c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:26:37.904 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.909+0000 7ff3d7e00640 1 --2- 172.21.15.96:0/2792442711 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff3d806cde0 0x7ff3d80ca1e0 secure :-1 s=READY pgs=2153 cs=0 l=1 rev1=1 crypto rx=0x7ff3d80cb8b0 tx=0x7ff3c002a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25a1715d7dae476d server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:37.904 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.909+0000 7ff3d7400640 1 -- 172.21.15.96:0/2792442711 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3c0036070 con 0x7ff3d806cde0 2024-05-28T06:26:37.904 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.909+0000 7ff3d7400640 1 -- 172.21.15.96:0/2792442711 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3c0031600 con 0x7ff3d806cde0 2024-05-28T06:26:37.904 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.909+0000 7ff3d7400640 1 -- 172.21.15.96:0/2792442711 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3c00319d0 con 0x7ff3d806cde0 2024-05-28T06:26:37.905 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.910+0000 7ff3dfc00640 1 -- 172.21.15.96:0/2792442711 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff3d806cde0 msgr2=0x7ff3d80ca1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:37.905 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.910+0000 7ff3dfc00640 1 --2- 172.21.15.96:0/2792442711 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff3d806cde0 0x7ff3d80ca1e0 secure :-1 s=READY pgs=2153 cs=0 l=1 rev1=1 crypto rx=0x7ff3d80cb8b0 tx=0x7ff3c002a610 comp rx=0 tx=0).stop 2024-05-28T06:26:37.905 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.910+0000 7ff3dfc00640 1 -- 172.21.15.96:0/2792442711 shutdown_connections 2024-05-28T06:26:37.905 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.910+0000 7ff3dfc00640 1 --2- 172.21.15.96:0/2792442711 >> v2:172.21.15.123:3300/0 conn(0x7ff3d806bc70 0x7ff3d806c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:37.906 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.910+0000 7ff3dfc00640 1 --2- 172.21.15.96:0/2792442711 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff3d806cde0 0x7ff3d80ca1e0 unknown :-1 s=CLOSED pgs=2153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:37.906 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.910+0000 7ff3dfc00640 1 --2- 172.21.15.96:0/2792442711 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff3d80c9e00 0x7ff3d8141f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:37.906 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.910+0000 7ff3dfc00640 1 -- 172.21.15.96:0/2792442711 >> 172.21.15.96:0/2792442711 conn(0x7ff3d80cd530 msgr2=0x7ff3d80cd940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:26:37.906 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.910+0000 7ff3dfc00640 1 -- 172.21.15.96:0/2792442711 shutdown_connections 2024-05-28T06:26:37.906 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.910+0000 7ff3dfc00640 1 -- 172.21.15.96:0/2792442711 wait complete. 2024-05-28T06:26:37.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.912+0000 7ff3dfc00640 1 Processor -- start 2024-05-28T06:26:37.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.912+0000 7ff3dfc00640 1 -- start start 2024-05-28T06:26:37.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.913+0000 7ff3dfc00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff3d806bc70 0x7ff3d806e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:37.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.913+0000 7ff3dfc00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff3d806cde0 0x7ff3d806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:37.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.913+0000 7ff3dfc00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3d80c9e00 0x7ff3d8143db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:37.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.913+0000 7ff3dfc00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff3d8162f90 con 0x7ff3d806cde0 2024-05-28T06:26:37.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.913+0000 7ff3dfc00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff3d8162cb0 con 0x7ff3d80c9e00 2024-05-28T06:26:37.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.913+0000 7ff3dfc00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ff3d806d240 con 0x7ff3d806bc70 2024-05-28T06:26:37.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.914+0000 7ff3d7e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff3d806cde0 0x7ff3d806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:37.909 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.914+0000 7ff3d7e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff3d806cde0 0x7ff3d806e9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44100/0 (socket says 172.21.15.96:44100) 2024-05-28T06:26:37.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.914+0000 7ff3d7e00640 1 -- 172.21.15.96:0/1766724328 learned_addr learned my addr 172.21.15.96:0/1766724328 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:26:37.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.914+0000 7ff3ddc00640 1 --2- 172.21.15.96:0/1766724328 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3d80c9e00 0x7ff3d8143db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:37.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.914+0000 7ff3dd200640 1 --2- 172.21.15.96:0/1766724328 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff3d806bc70 0x7ff3d806e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:37.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.914+0000 7ff3d7e00640 1 -- 172.21.15.96:0/1766724328 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff3d806bc70 msgr2=0x7ff3d806e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:37.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.914+0000 7ff3d7e00640 1 --2- 172.21.15.96:0/1766724328 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff3d806bc70 0x7ff3d806e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:37.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.914+0000 7ff3d7e00640 1 -- 172.21.15.96:0/1766724328 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3d80c9e00 msgr2=0x7ff3d8143db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:37.910 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.914+0000 7ff3d7e00640 1 --2- 172.21.15.96:0/1766724328 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3d80c9e00 0x7ff3d8143db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:37.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.914+0000 7ff3d7e00640 1 -- 172.21.15.96:0/1766724328 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3d81f49f0 con 0x7ff3d806cde0 2024-05-28T06:26:37.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.914+0000 7ff3dd200640 1 --2- 172.21.15.96:0/1766724328 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff3d806bc70 0x7ff3d806e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:26:37.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.914+0000 7ff3d7e00640 1 --2- 172.21.15.96:0/1766724328 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff3d806cde0 0x7ff3d806e9a0 secure :-1 s=READY pgs=2154 cs=0 l=1 rev1=1 crypto rx=0x7ff3c0034040 tx=0x7ff3c0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:37.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.915+0000 7ff3d4c00640 1 -- 172.21.15.96:0/1766724328 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3c0041020 con 0x7ff3d806cde0 2024-05-28T06:26:37.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.915+0000 7ff3d4c00640 1 -- 172.21.15.96:0/1766724328 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3c0007820 con 0x7ff3d806cde0 2024-05-28T06:26:37.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.915+0000 7ff3dfc00640 1 -- 172.21.15.96:0/1766724328 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3d8151a80 con 0x7ff3d806cde0 2024-05-28T06:26:37.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.915+0000 7ff3d4c00640 1 -- 172.21.15.96:0/1766724328 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3c0036040 con 0x7ff3d806cde0 2024-05-28T06:26:37.911 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.915+0000 7ff3dfc00640 1 -- 172.21.15.96:0/1766724328 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3d8152040 con 0x7ff3d806cde0 2024-05-28T06:26:37.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.916+0000 7ff3dfc00640 1 -- 172.21.15.96:0/1766724328 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3d80689e0 con 0x7ff3d806cde0 2024-05-28T06:26:37.912 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.917+0000 7ff3d4c00640 1 -- 172.21.15.96:0/1766724328 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 49) ==== 94049+0+0 (secure 0 0 0) 0x7ff3c0044020 con 0x7ff3d806cde0 2024-05-28T06:26:37.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.918+0000 7ff3d4c00640 1 --2- 172.21.15.96:0/1766724328 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff3b406ea20 0x7ff3b4070ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:37.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.918+0000 7ff3d4c00640 1 -- 172.21.15.96:0/1766724328 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 5823+0+0 (secure 0 0 0) 0x7ff3c00abb50 con 0x7ff3d806cde0 2024-05-28T06:26:37.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.919+0000 7ff3dd200640 1 --2- 172.21.15.96:0/1766724328 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff3b406ea20 0x7ff3b4070ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:37.913 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.920+0000 7ff3dd200640 1 --2- 172.21.15.96:0/1766724328 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff3b406ea20 0x7ff3b4070ee0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7ff3c800b820 tx=0x7ff3c8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:37.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:37.922+0000 7ff3d4c00640 1 -- 172.21.15.96:0/1766724328 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff3c004b030 con 0x7ff3d806cde0 2024-05-28T06:26:38.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.120+0000 7ff3dfc00640 1 -- 172.21.15.96:0/1766724328 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff3d806cbc0 con 0x7ff3d806cde0 2024-05-28T06:26:38.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.121+0000 7ff3d4c00640 1 -- 172.21.15.96:0/1766724328 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v132) ==== 93+0+10 (secure 0 0 0) 0x7ff3d81f49f0 con 0x7ff3d806cde0 2024-05-28T06:26:38.114 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 1 2024-05-28T06:26:38.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.123+0000 7ff3dfc00640 1 -- 172.21.15.96:0/1766724328 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff3b406ea20 msgr2=0x7ff3b4070ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:38.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.123+0000 7ff3dfc00640 1 --2- 172.21.15.96:0/1766724328 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff3b406ea20 0x7ff3b4070ee0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7ff3c800b820 tx=0x7ff3c8008040 comp rx=0 tx=0).stop 2024-05-28T06:26:38.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.123+0000 7ff3dfc00640 1 -- 172.21.15.96:0/1766724328 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff3d806cde0 msgr2=0x7ff3d806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:38.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.123+0000 7ff3dfc00640 1 --2- 172.21.15.96:0/1766724328 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff3d806cde0 0x7ff3d806e9a0 secure :-1 s=READY pgs=2154 cs=0 l=1 rev1=1 crypto rx=0x7ff3c0034040 tx=0x7ff3c0004a40 comp rx=0 tx=0).stop 2024-05-28T06:26:38.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.123+0000 7ff3dfc00640 1 -- 172.21.15.96:0/1766724328 shutdown_connections 2024-05-28T06:26:38.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.123+0000 7ff3dfc00640 1 --2- 172.21.15.96:0/1766724328 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff3b406ea20 0x7ff3b4070ee0 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:38.117 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.123+0000 7ff3dfc00640 1 --2- 172.21.15.96:0/1766724328 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3d80c9e00 0x7ff3d8143db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:38.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.123+0000 7ff3dfc00640 1 --2- 172.21.15.96:0/1766724328 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff3d806cde0 0x7ff3d806e9a0 unknown :-1 s=CLOSED pgs=2154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:38.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.123+0000 7ff3dfc00640 1 --2- 172.21.15.96:0/1766724328 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff3d806bc70 0x7ff3d806e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:38.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.123+0000 7ff3dfc00640 1 -- 172.21.15.96:0/1766724328 >> 172.21.15.96:0/1766724328 conn(0x7ff3d80cd530 msgr2=0x7ff3d80702d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:26:38.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.124+0000 7ff3dfc00640 1 -- 172.21.15.96:0/1766724328 shutdown_connections 2024-05-28T06:26:38.118 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.124+0000 7ff3dfc00640 1 -- 172.21.15.96:0/1766724328 wait complete. 2024-05-28T06:26:38.136 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-28T06:26:38.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.337+0000 7fd2bfe00640 1 Processor -- start 2024-05-28T06:26:38.331 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.337+0000 7fd2bfe00640 1 -- start start 2024-05-28T06:26:38.331 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.338+0000 7fd2bfe00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fd2c00caa40 0x7fd2c00ce960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:38.331 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.338+0000 7fd2bfe00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd2c00cae20 0x7fd2c0151db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:38.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.338+0000 7fd2bfe00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2c0152780 con 0x7fd2c01522f0 2024-05-28T06:26:38.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.338+0000 7fd2bfe00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fd2c00d0b30 con 0x7fd2c00caa40 2024-05-28T06:26:38.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.338+0000 7fd2bfe00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fd2c00b4160 con 0x7fd2c00cae20 2024-05-28T06:26:38.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.338+0000 7fd2bea00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fd2c00caa40 0x7fd2c00ce960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:38.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.338+0000 7fd2bea00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fd2c00caa40 0x7fd2c00ce960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44104/0 (socket says 172.21.15.96:44104) 2024-05-28T06:26:38.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.338+0000 7fd2bea00640 1 -- 172.21.15.96:0/3760557211 learned_addr learned my addr 172.21.15.96:0/3760557211 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:26:38.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.339+0000 7fd2be000640 1 --2- 172.21.15.96:0/3760557211 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd2c00cae20 0x7fd2c0151db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:38.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.339+0000 7fd2bea00640 1 -- 172.21.15.96:0/3760557211 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd2c00cae20 msgr2=0x7fd2c0151db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:38.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.339+0000 7fd2bea00640 1 --2- 172.21.15.96:0/3760557211 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd2c00cae20 0x7fd2c0151db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:38.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.339+0000 7fd2bea00640 1 -- 172.21.15.96:0/3760557211 >> v1:172.21.15.123:6789/0 conn(0x7fd2c01522f0 legacy=0x7fd2c00cf710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:38.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.339+0000 7fd2bea00640 1 -- 172.21.15.96:0/3760557211 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2c00b3e80 con 0x7fd2c00caa40 2024-05-28T06:26:38.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.340+0000 7fd2bea00640 1 --2- 172.21.15.96:0/3760557211 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd2c00caa40 0x7fd2c00ce960 secure :-1 s=READY pgs=2155 cs=0 l=1 rev1=1 crypto rx=0x7fd2c00b42e0 tx=0x7fd2b002a160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad47fb666c76de10 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:38.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.340+0000 7fd2bd600640 1 -- 172.21.15.96:0/3760557211 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2b0037070 con 0x7fd2c00caa40 2024-05-28T06:26:38.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.340+0000 7fd2bd600640 1 -- 172.21.15.96:0/3760557211 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2b0032440 con 0x7fd2c00caa40 2024-05-28T06:26:38.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.340+0000 7fd2bd600640 1 -- 172.21.15.96:0/3760557211 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2b0032810 con 0x7fd2c00caa40 2024-05-28T06:26:38.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.341+0000 7fd2bfe00640 1 -- 172.21.15.96:0/3760557211 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd2c00caa40 msgr2=0x7fd2c00ce960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:38.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.341+0000 7fd2bfe00640 1 --2- 172.21.15.96:0/3760557211 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd2c00caa40 0x7fd2c00ce960 secure :-1 s=READY pgs=2155 cs=0 l=1 rev1=1 crypto rx=0x7fd2c00b42e0 tx=0x7fd2b002a160 comp rx=0 tx=0).stop 2024-05-28T06:26:38.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.341+0000 7fd2bfe00640 1 -- 172.21.15.96:0/3760557211 shutdown_connections 2024-05-28T06:26:38.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.341+0000 7fd2bfe00640 1 --2- 172.21.15.96:0/3760557211 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd2c00cae20 0x7fd2c0151db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:38.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.341+0000 7fd2bfe00640 1 --2- 172.21.15.96:0/3760557211 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd2c00caa40 0x7fd2c00ce960 unknown :-1 s=CLOSED pgs=2155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:38.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.341+0000 7fd2bfe00640 1 -- 172.21.15.96:0/3760557211 >> 172.21.15.96:0/3760557211 conn(0x7fd2c006b900 msgr2=0x7fd2c006bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:26:38.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.341+0000 7fd2bfe00640 1 -- 172.21.15.96:0/3760557211 shutdown_connections 2024-05-28T06:26:38.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.341+0000 7fd2bfe00640 1 -- 172.21.15.96:0/3760557211 wait complete. 2024-05-28T06:26:38.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.343+0000 7fd2bfe00640 1 Processor -- start 2024-05-28T06:26:38.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.343+0000 7fd2bfe00640 1 -- start start 2024-05-28T06:26:38.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.344+0000 7fd2bfe00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd2c00caa40 0x7fd2c01f3e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:38.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.344+0000 7fd2bfe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2c00cae20 0x7fd2c0071e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:38.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.344+0000 7fd2bfe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd2c01522f0 0x7fd2c00c94d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:38.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.344+0000 7fd2bfe00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd2c00ce440 con 0x7fd2c01522f0 2024-05-28T06:26:38.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.344+0000 7fd2bfe00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd2c00ce160 con 0x7fd2c00cae20 2024-05-28T06:26:38.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.344+0000 7fd2bfe00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fd2c00ceea0 con 0x7fd2c00caa40 2024-05-28T06:26:38.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.344+0000 7fd2bf400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd2c01522f0 0x7fd2c00c94d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:38.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.344+0000 7fd2bf400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd2c01522f0 0x7fd2c00c94d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44116/0 (socket says 172.21.15.96:44116) 2024-05-28T06:26:38.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.344+0000 7fd2bf400640 1 -- 172.21.15.96:0/1682971318 learned_addr learned my addr 172.21.15.96:0/1682971318 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:26:38.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.344+0000 7fd2bea00640 1 --2- 172.21.15.96:0/1682971318 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd2c00caa40 0x7fd2c01f3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:38.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.344+0000 7fd2be000640 1 --2- 172.21.15.96:0/1682971318 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2c00cae20 0x7fd2c0071e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:38.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.345+0000 7fd2bf400640 1 -- 172.21.15.96:0/1682971318 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd2c00caa40 msgr2=0x7fd2c01f3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:38.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.345+0000 7fd2bf400640 1 --2- 172.21.15.96:0/1682971318 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd2c00caa40 0x7fd2c01f3e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:38.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.345+0000 7fd2bf400640 1 -- 172.21.15.96:0/1682971318 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2c00cae20 msgr2=0x7fd2c0071e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:38.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.345+0000 7fd2bf400640 1 --2- 172.21.15.96:0/1682971318 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2c00cae20 0x7fd2c0071e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:38.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.345+0000 7fd2bf400640 1 -- 172.21.15.96:0/1682971318 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2c0151050 con 0x7fd2c01522f0 2024-05-28T06:26:38.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.345+0000 7fd2bf400640 1 --2- 172.21.15.96:0/1682971318 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd2c01522f0 0x7fd2c00c94d0 secure :-1 s=READY pgs=2156 cs=0 l=1 rev1=1 crypto rx=0x7fd2b800c780 tx=0x7fd2b800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:38.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.345+0000 7fd2b6a00640 1 -- 172.21.15.96:0/1682971318 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2b8004b60 con 0x7fd2c01522f0 2024-05-28T06:26:38.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.345+0000 7fd2b6a00640 1 -- 172.21.15.96:0/1682971318 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2b8004d00 con 0x7fd2c01522f0 2024-05-28T06:26:38.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.345+0000 7fd2bfe00640 1 -- 172.21.15.96:0/1682971318 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2c0151330 con 0x7fd2c01522f0 2024-05-28T06:26:38.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.345+0000 7fd2b6a00640 1 -- 172.21.15.96:0/1682971318 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2b800aa90 con 0x7fd2c01522f0 2024-05-28T06:26:38.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.345+0000 7fd2bfe00640 1 -- 172.21.15.96:0/1682971318 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2c01518f0 con 0x7fd2c01522f0 2024-05-28T06:26:38.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.346+0000 7fd2bfe00640 1 -- 172.21.15.96:0/1682971318 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2c00689e0 con 0x7fd2c01522f0 2024-05-28T06:26:38.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.347+0000 7fd2b6a00640 1 -- 172.21.15.96:0/1682971318 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 49) ==== 94049+0+0 (secure 0 0 0) 0x7fd2b800ac30 con 0x7fd2c01522f0 2024-05-28T06:26:38.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.349+0000 7fd2b6a00640 1 --2- 172.21.15.96:0/1682971318 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd2a006ea20 0x7fd2a0070ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:38.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.349+0000 7fd2b6a00640 1 -- 172.21.15.96:0/1682971318 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 5823+0+0 (secure 0 0 0) 0x7fd2b808af00 con 0x7fd2c01522f0 2024-05-28T06:26:38.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.349+0000 7fd2bea00640 1 --2- 172.21.15.96:0/1682971318 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd2a006ea20 0x7fd2a0070ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:38.344 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.351+0000 7fd2bea00640 1 --2- 172.21.15.96:0/1682971318 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd2a006ea20 0x7fd2a0070ee0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7fd2b0034000 tx=0x7fd2b002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:38.347 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.353+0000 7fd2b6a00640 1 -- 172.21.15.96:0/1682971318 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd2b8020070 con 0x7fd2c01522f0 2024-05-28T06:26:38.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.557+0000 7fd2bfe00640 1 -- 172.21.15.96:0/1682971318 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fd2c00cd920 con 0x7fd2c01522f0 2024-05-28T06:26:38.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.559+0000 7fd2b6a00640 1 -- 172.21.15.96:0/1682971318 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v132) ==== 104+0+11 (secure 0 0 0) 0x7fd2b805ae30 con 0x7fd2c01522f0 2024-05-28T06:26:38.552 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 32 2024-05-28T06:26:38.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.562+0000 7fd2bfe00640 1 -- 172.21.15.96:0/1682971318 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd2a006ea20 msgr2=0x7fd2a0070ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:38.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.562+0000 7fd2bfe00640 1 --2- 172.21.15.96:0/1682971318 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd2a006ea20 0x7fd2a0070ee0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7fd2b0034000 tx=0x7fd2b002e000 comp rx=0 tx=0).stop 2024-05-28T06:26:38.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.562+0000 7fd2bfe00640 1 -- 172.21.15.96:0/1682971318 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd2c01522f0 msgr2=0x7fd2c00c94d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:38.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.562+0000 7fd2bfe00640 1 --2- 172.21.15.96:0/1682971318 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd2c01522f0 0x7fd2c00c94d0 secure :-1 s=READY pgs=2156 cs=0 l=1 rev1=1 crypto rx=0x7fd2b800c780 tx=0x7fd2b800cc40 comp rx=0 tx=0).stop 2024-05-28T06:26:38.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.562+0000 7fd2bfe00640 1 -- 172.21.15.96:0/1682971318 shutdown_connections 2024-05-28T06:26:38.556 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.562+0000 7fd2bfe00640 1 --2- 172.21.15.96:0/1682971318 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fd2a006ea20 0x7fd2a0070ee0 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:38.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.562+0000 7fd2bfe00640 1 --2- 172.21.15.96:0/1682971318 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd2c01522f0 0x7fd2c00c94d0 unknown :-1 s=CLOSED pgs=2156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:38.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.562+0000 7fd2bfe00640 1 --2- 172.21.15.96:0/1682971318 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2c00cae20 0x7fd2c0071e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:38.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.562+0000 7fd2bfe00640 1 --2- 172.21.15.96:0/1682971318 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd2c00caa40 0x7fd2c01f3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:38.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.562+0000 7fd2bfe00640 1 -- 172.21.15.96:0/1682971318 >> 172.21.15.96:0/1682971318 conn(0x7fd2c006b900 msgr2=0x7fd2c00b16a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:26:38.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.563+0000 7fd2bfe00640 1 -- 172.21.15.96:0/1682971318 shutdown_connections 2024-05-28T06:26:38.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.563+0000 7fd2bfe00640 1 -- 172.21.15.96:0/1682971318 wait complete. 2024-05-28T06:26:38.575 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-28T06:26:38.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.782+0000 7ff823a00640 1 Processor -- start 2024-05-28T06:26:38.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.783+0000 7ff823a00640 1 -- start start 2024-05-28T06:26:38.777 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.784+0000 7ff823a00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7ff81c161e90 0x7ff81c06ff90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:38.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.784+0000 7ff823a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff81c0b14f0 0x7ff81c0ced30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:38.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.784+0000 7ff823a00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff81c09aa30 con 0x7ff81c0cf270 2024-05-28T06:26:38.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.784+0000 7ff823a00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7ff81c0cf6e0 con 0x7ff81c161e90 2024-05-28T06:26:38.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.784+0000 7ff823a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ff81c162270 con 0x7ff81c0b14f0 2024-05-28T06:26:38.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.784+0000 7ff821a00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7ff81c0cf270 0x7ff81c162490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:35550/0 (socket says 172.21.15.96:35550) 2024-05-28T06:26:38.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.784+0000 7ff821a00640 1 -- 172.21.15.96:0/3734784674 learned_addr learned my addr 172.21.15.96:0/3734784674 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:26:38.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.785+0000 7ff81be00640 1 --2- 172.21.15.96:0/3734784674 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff81c0b14f0 0x7ff81c0ced30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:38.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.785+0000 7ff821000640 1 --2- 172.21.15.96:0/3734784674 >> v2:172.21.15.123:3300/0 conn(0x7ff81c161e90 0x7ff81c06ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:38.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.785+0000 7ff81b400640 1 -- 172.21.15.96:0/3734784674 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3835309617 0 0) 0x7ff81c09aa30 con 0x7ff81c0cf270 2024-05-28T06:26:38.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.785+0000 7ff81b400640 1 -- 172.21.15.96:0/3734784674 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff800003520 con 0x7ff81c0cf270 2024-05-28T06:26:38.781 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.785+0000 7ff81b400640 1 -- 172.21.15.96:0/3734784674 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2733752031 0 0) 0x7ff800003520 con 0x7ff81c0cf270 2024-05-28T06:26:38.781 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.785+0000 7ff81b400640 1 -- 172.21.15.96:0/3734784674 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff81c0b14f0 msgr2=0x7ff81c0ced30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:38.781 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.785+0000 7ff81b400640 1 --2- 172.21.15.96:0/3734784674 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff81c0b14f0 0x7ff81c0ced30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:38.781 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.785+0000 7ff81b400640 1 -- 172.21.15.96:0/3734784674 >> v2:172.21.15.123:3300/0 conn(0x7ff81c161e90 msgr2=0x7ff81c06ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:38.781 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.785+0000 7ff81b400640 1 --2- 172.21.15.96:0/3734784674 >> v2:172.21.15.123:3300/0 conn(0x7ff81c161e90 0x7ff81c06ff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:38.781 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.785+0000 7ff81b400640 1 -- 172.21.15.96:0/3734784674 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff81c09ad10 con 0x7ff81c0cf270 2024-05-28T06:26:38.781 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.785+0000 7ff81b400640 1 -- 172.21.15.96:0/3734784674 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7ff814001670 con 0x7ff81c0cf270 2024-05-28T06:26:38.781 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.785+0000 7ff81b400640 1 -- 172.21.15.96:0/3734784674 >> v1:172.21.15.96:6789/0 conn(0x7ff81c0cf270 legacy=0x7ff81c162490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:38.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.785+0000 7ff81b400640 1 --2- 172.21.15.96:0/3734784674 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff81c0b14f0 0x7ff8000039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:38.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.785+0000 7ff81b400640 1 --2- 172.21.15.96:0/3734784674 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff800003f30 0x7ff800004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:38.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.785+0000 7ff81b400640 1 --2- 172.21.15.96:0/3734784674 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff800004d40 0x7ff8000051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:38.782 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.786+0000 7ff81b400640 1 -- 172.21.15.96:0/3734784674 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff81c0cf6e0 con 0x7ff800004d40 2024-05-28T06:26:38.786 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.786+0000 7ff81b400640 1 -- 172.21.15.96:0/3734784674 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff81c162270 con 0x7ff800003f30 2024-05-28T06:26:38.786 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.786+0000 7ff81b400640 1 -- 172.21.15.96:0/3734784674 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ff80000b9c0 con 0x7ff81c0b14f0 2024-05-28T06:26:38.786 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.786+0000 7ff821a00640 1 --2- 172.21.15.96:0/3734784674 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff800004d40 0x7ff8000051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:38.786 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.786+0000 7ff821000640 1 --2- 172.21.15.96:0/3734784674 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff81c0b14f0 0x7ff8000039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:38.787 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.786+0000 7ff81be00640 1 --2- 172.21.15.96:0/3734784674 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff800003f30 0x7ff800004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:38.787 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.786+0000 7ff821a00640 1 -- 172.21.15.96:0/3734784674 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff81c0b14f0 msgr2=0x7ff8000039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:38.787 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.786+0000 7ff821a00640 1 --2- 172.21.15.96:0/3734784674 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff81c0b14f0 0x7ff8000039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:38.787 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.786+0000 7ff821a00640 1 -- 172.21.15.96:0/3734784674 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff800003f30 msgr2=0x7ff800004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:38.787 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.786+0000 7ff821a00640 1 --2- 172.21.15.96:0/3734784674 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff800003f30 0x7ff800004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:38.787 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.786+0000 7ff821a00640 1 -- 172.21.15.96:0/3734784674 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff800003520 con 0x7ff800004d40 2024-05-28T06:26:38.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.787+0000 7ff821a00640 1 --2- 172.21.15.96:0/3734784674 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff800004d40 0x7ff8000051c0 secure :-1 s=READY pgs=2158 cs=0 l=1 rev1=1 crypto rx=0x7ff814001050 tx=0x7ff800005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ff4e289ccb7e563 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:38.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.788+0000 7ff81b400640 1 -- 172.21.15.96:0/3734784674 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff814007d70 con 0x7ff800004d40 2024-05-28T06:26:38.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.788+0000 7ff81b400640 1 -- 172.21.15.96:0/3734784674 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff814005af0 con 0x7ff800004d40 2024-05-28T06:26:38.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.788+0000 7ff81b400640 1 -- 172.21.15.96:0/3734784674 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff814007070 con 0x7ff800004d40 2024-05-28T06:26:38.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.788+0000 7ff823a00640 1 -- 172.21.15.96:0/3734784674 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff800004d40 msgr2=0x7ff8000051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:38.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.788+0000 7ff823a00640 1 --2- 172.21.15.96:0/3734784674 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff800004d40 0x7ff8000051c0 secure :-1 s=READY pgs=2158 cs=0 l=1 rev1=1 crypto rx=0x7ff814001050 tx=0x7ff800005830 comp rx=0 tx=0).stop 2024-05-28T06:26:38.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.788+0000 7ff821a00640 1 -- 172.21.15.96:0/3734784674 reap_dead start 2024-05-28T06:26:38.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.788+0000 7ff823a00640 1 -- 172.21.15.96:0/3734784674 shutdown_connections 2024-05-28T06:26:38.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.788+0000 7ff823a00640 1 -- 172.21.15.96:0/3734784674 >> 172.21.15.96:0/3734784674 conn(0x7ff81c0b4040 msgr2=0x7ff81c0b4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:26:38.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.788+0000 7ff823a00640 1 -- 172.21.15.96:0/3734784674 shutdown_connections 2024-05-28T06:26:38.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.788+0000 7ff823a00640 1 -- 172.21.15.96:0/3734784674 wait complete. 2024-05-28T06:26:38.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.790+0000 7ff823a00640 1 Processor -- start 2024-05-28T06:26:38.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.790+0000 7ff823a00640 1 -- start start 2024-05-28T06:26:38.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.791+0000 7ff823a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff800004d40 0x7ff81c153750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:38.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.791+0000 7ff823a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff81c1f4980 0x7ff81c15e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:38.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.791+0000 7ff823a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff81c15eef0 0x7ff81c0c76b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:38.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.791+0000 7ff823a00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff81c09b5b0 con 0x7ff800004d40 2024-05-28T06:26:38.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.791+0000 7ff823a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff81c09b2d0 con 0x7ff81c1f4980 2024-05-28T06:26:38.793 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.791+0000 7ff823a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ff81c09b890 con 0x7ff81c15eef0 2024-05-28T06:26:38.793 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.791+0000 7ff821000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff800004d40 0x7ff81c153750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:38.793 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.792+0000 7ff821000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff800004d40 0x7ff81c153750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:44128/0 (socket says 172.21.15.96:44128) 2024-05-28T06:26:38.793 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.792+0000 7ff821000640 1 -- 172.21.15.96:0/3468580357 learned_addr learned my addr 172.21.15.96:0/3468580357 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:26:38.793 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.792+0000 7ff821000640 1 -- 172.21.15.96:0/3468580357 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff81c15eef0 msgr2=0x7ff81c0c76b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:38.793 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.792+0000 7ff821000640 1 --2- 172.21.15.96:0/3468580357 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff81c15eef0 0x7ff81c0c76b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:38.793 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.792+0000 7ff821000640 1 -- 172.21.15.96:0/3468580357 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff81c1f4980 msgr2=0x7ff81c15e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:38.793 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.792+0000 7ff821000640 1 --2- 172.21.15.96:0/3468580357 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff81c1f4980 0x7ff81c15e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:38.794 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.792+0000 7ff821000640 1 -- 172.21.15.96:0/3468580357 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff81c0d0770 con 0x7ff800004d40 2024-05-28T06:26:38.794 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.792+0000 7ff821000640 1 --2- 172.21.15.96:0/3468580357 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff800004d40 0x7ff81c153750 secure :-1 s=READY pgs=2159 cs=0 l=1 rev1=1 crypto rx=0x7ff80c007f30 tx=0x7ff80c011660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:38.794 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.792+0000 7ff818c00640 1 -- 172.21.15.96:0/3468580357 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff80c01a020 con 0x7ff800004d40 2024-05-28T06:26:38.794 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.792+0000 7ff818c00640 1 -- 172.21.15.96:0/3468580357 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff80c016470 con 0x7ff800004d40 2024-05-28T06:26:38.794 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.792+0000 7ff818c00640 1 -- 172.21.15.96:0/3468580357 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff80c016840 con 0x7ff800004d40 2024-05-28T06:26:38.794 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.792+0000 7ff823a00640 1 -- 172.21.15.96:0/3468580357 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff81c0d0a50 con 0x7ff800004d40 2024-05-28T06:26:38.794 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.793+0000 7ff823a00640 1 -- 172.21.15.96:0/3468580357 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff81c0d1010 con 0x7ff800004d40 2024-05-28T06:26:38.794 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.794+0000 7ff823a00640 1 -- 172.21.15.96:0/3468580357 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff81c0690f0 con 0x7ff800004d40 2024-05-28T06:26:38.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.795+0000 7ff818c00640 1 -- 172.21.15.96:0/3468580357 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 49) ==== 94049+0+0 (secure 0 0 0) 0x7ff80c01d020 con 0x7ff800004d40 2024-05-28T06:26:38.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.796+0000 7ff818c00640 1 --2- 172.21.15.96:0/3468580357 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff7fc06ea50 0x7ff7fc070f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:38.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.796+0000 7ff818c00640 1 -- 172.21.15.96:0/3468580357 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 5823+0+0 (secure 0 0 0) 0x7ff80c08bbe0 con 0x7ff800004d40 2024-05-28T06:26:38.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.796+0000 7ff81be00640 1 --2- 172.21.15.96:0/3468580357 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff7fc06ea50 0x7ff7fc070f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:38.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.798+0000 7ff81be00640 1 --2- 172.21.15.96:0/3468580357 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff7fc06ea50 0x7ff7fc070f10 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7ff81c15f690 tx=0x7ff808008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:38.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:38.800+0000 7ff818c00640 1 -- 172.21.15.96:0/3468580357 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff80c024020 con 0x7ff800004d40 2024-05-28T06:26:38.997 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.004+0000 7ff823a00640 1 -- 172.21.15.96:0/3468580357 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7ff81c075320 con 0x7ff800004d40 2024-05-28T06:26:38.998 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.005+0000 7ff818c00640 1 -- 172.21.15.96:0/3468580357 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v132) ==== 100+0+11 (secure 0 0 0) 0x7ff80c05ba70 con 0x7ff800004d40 2024-05-28T06:26:38.998 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 64 2024-05-28T06:26:39.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.007+0000 7ff823a00640 1 -- 172.21.15.96:0/3468580357 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff7fc06ea50 msgr2=0x7ff7fc070f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:39.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.007+0000 7ff823a00640 1 --2- 172.21.15.96:0/3468580357 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff7fc06ea50 0x7ff7fc070f10 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7ff81c15f690 tx=0x7ff808008040 comp rx=0 tx=0).stop 2024-05-28T06:26:39.001 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.007+0000 7ff823a00640 1 -- 172.21.15.96:0/3468580357 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff800004d40 msgr2=0x7ff81c153750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:39.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.007+0000 7ff823a00640 1 --2- 172.21.15.96:0/3468580357 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff800004d40 0x7ff81c153750 secure :-1 s=READY pgs=2159 cs=0 l=1 rev1=1 crypto rx=0x7ff80c007f30 tx=0x7ff80c011660 comp rx=0 tx=0).stop 2024-05-28T06:26:39.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.007+0000 7ff823a00640 1 -- 172.21.15.96:0/3468580357 shutdown_connections 2024-05-28T06:26:39.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.007+0000 7ff823a00640 1 --2- 172.21.15.96:0/3468580357 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff7fc06ea50 0x7ff7fc070f10 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:39.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.007+0000 7ff823a00640 1 --2- 172.21.15.96:0/3468580357 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff81c15eef0 0x7ff81c0c76b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:39.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.007+0000 7ff823a00640 1 --2- 172.21.15.96:0/3468580357 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff81c1f4980 0x7ff81c15e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:39.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.007+0000 7ff823a00640 1 --2- 172.21.15.96:0/3468580357 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff800004d40 0x7ff81c153750 unknown :-1 s=CLOSED pgs=2159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:39.002 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.008+0000 7ff823a00640 1 -- 172.21.15.96:0/3468580357 >> 172.21.15.96:0/3468580357 conn(0x7ff81c0b4040 msgr2=0x7ff81c0b00d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:26:39.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.008+0000 7ff823a00640 1 -- 172.21.15.96:0/3468580357 shutdown_connections 2024-05-28T06:26:39.003 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.008+0000 7ff823a00640 1 -- 172.21.15.96:0/3468580357 wait complete. 2024-05-28T06:26:39.021 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-28T06:26:39.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.217+0000 7f329b200640 1 Processor -- start 2024-05-28T06:26:39.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.217+0000 7f329b200640 1 -- start start 2024-05-28T06:26:39.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.218+0000 7f329b200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f329c06dc20 0x7f329c0b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:39.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.218+0000 7f329b200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f329c0afd60 con 0x7f329c073310 2024-05-28T06:26:39.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.218+0000 7f329b200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f329c0af7a0 con 0x7f329c15b3a0 2024-05-28T06:26:39.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.218+0000 7f329b200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f329c06f980 con 0x7f329c06dc20 2024-05-28T06:26:39.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.218+0000 7f3291400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f329c073310 0x7f329c0ae550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:44810/0 (socket says 172.21.15.96:44810) 2024-05-28T06:26:39.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.218+0000 7f3291400640 1 -- 172.21.15.96:0/1550763699 learned_addr learned my addr 172.21.15.96:0/1550763699 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:26:39.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.219+0000 7f329a800640 1 --2- 172.21.15.96:0/1550763699 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f329c06dc20 0x7f329c0b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:39.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.219+0000 7f3299400640 1 -- 172.21.15.96:0/1550763699 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3918204935 0 0) 0x7f329c0afd60 con 0x7f329c073310 2024-05-28T06:26:39.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.219+0000 7f3299400640 1 -- 172.21.15.96:0/1550763699 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3280003520 con 0x7f329c073310 2024-05-28T06:26:39.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.219+0000 7f3299400640 1 -- 172.21.15.96:0/1550763699 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3561856249 0 0) 0x7f3280003520 con 0x7f329c073310 2024-05-28T06:26:39.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.219+0000 7f3299400640 1 -- 172.21.15.96:0/1550763699 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f329c06dc20 msgr2=0x7f329c0b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:39.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.219+0000 7f3299400640 1 --2- 172.21.15.96:0/1550763699 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f329c06dc20 0x7f329c0b32c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:39.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.219+0000 7f3299400640 1 -- 172.21.15.96:0/1550763699 >> v1:172.21.15.123:6789/0 conn(0x7f329c15b3a0 legacy=0x7f329c072c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:39.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.219+0000 7f3299400640 1 -- 172.21.15.96:0/1550763699 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f329c0afa80 con 0x7f329c073310 2024-05-28T06:26:39.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.219+0000 7f3299400640 1 -- 172.21.15.96:0/1550763699 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f3284001670 con 0x7f329c073310 2024-05-28T06:26:39.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.219+0000 7f3299400640 1 -- 172.21.15.96:0/1550763699 >> v1:172.21.15.96:6789/0 conn(0x7f329c073310 legacy=0x7f329c0ae550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:39.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.219+0000 7f3299400640 1 --2- 172.21.15.96:0/1550763699 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32800039e0 0x7f3280003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:39.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.220+0000 7f3299400640 1 --2- 172.21.15.96:0/1550763699 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f329c06dc20 0x7f3280004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:39.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.220+0000 7f3299400640 1 --2- 172.21.15.96:0/1550763699 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3280004d40 0x7f32800051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:39.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.220+0000 7f3299400640 1 -- 172.21.15.96:0/1550763699 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f329c06f980 con 0x7f3280004d40 2024-05-28T06:26:39.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.220+0000 7f3299400640 1 -- 172.21.15.96:0/1550763699 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f328000bc40 con 0x7f32800039e0 2024-05-28T06:26:39.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.220+0000 7f3299400640 1 -- 172.21.15.96:0/1550763699 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f328000be40 con 0x7f329c06dc20 2024-05-28T06:26:39.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.220+0000 7f329a800640 1 --2- 172.21.15.96:0/1550763699 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3280004d40 0x7f32800051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:39.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.220+0000 7f3291400640 1 --2- 172.21.15.96:0/1550763699 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f329c06dc20 0x7f3280004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:39.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.220+0000 7f3299e00640 1 --2- 172.21.15.96:0/1550763699 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32800039e0 0x7f3280003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:39.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.220+0000 7f329a800640 1 -- 172.21.15.96:0/1550763699 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f329c06dc20 msgr2=0x7f3280004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:39.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.220+0000 7f329a800640 1 --2- 172.21.15.96:0/1550763699 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f329c06dc20 0x7f3280004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:39.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.220+0000 7f329a800640 1 -- 172.21.15.96:0/1550763699 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32800039e0 msgr2=0x7f3280003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:39.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.220+0000 7f329a800640 1 --2- 172.21.15.96:0/1550763699 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32800039e0 0x7f3280003e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:39.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.220+0000 7f329a800640 1 -- 172.21.15.96:0/1550763699 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3280003520 con 0x7f3280004d40 2024-05-28T06:26:39.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.220+0000 7f3291400640 1 --2- 172.21.15.96:0/1550763699 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f329c06dc20 0x7f3280004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:26:39.218 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.221+0000 7f329a800640 1 --2- 172.21.15.96:0/1550763699 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3280004d40 0x7f32800051c0 secure :-1 s=READY pgs=2161 cs=0 l=1 rev1=1 crypto rx=0x7f3294007360 tx=0x7f3280005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c06904d6dbd87f5b server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:39.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.221+0000 7f3299400640 1 -- 172.21.15.96:0/1550763699 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3294009be0 con 0x7f3280004d40 2024-05-28T06:26:39.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.222+0000 7f3299400640 1 -- 172.21.15.96:0/1550763699 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3294036600 con 0x7f3280004d40 2024-05-28T06:26:39.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.222+0000 7f3299400640 1 -- 172.21.15.96:0/1550763699 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32940369d0 con 0x7f3280004d40 2024-05-28T06:26:39.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.222+0000 7f329b200640 1 -- 172.21.15.96:0/1550763699 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3280004d40 msgr2=0x7f32800051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:39.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.222+0000 7f329b200640 1 --2- 172.21.15.96:0/1550763699 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3280004d40 0x7f32800051c0 secure :-1 s=READY pgs=2161 cs=0 l=1 rev1=1 crypto rx=0x7f3294007360 tx=0x7f3280005830 comp rx=0 tx=0).stop 2024-05-28T06:26:39.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.222+0000 7f329a800640 1 -- 172.21.15.96:0/1550763699 reap_dead start 2024-05-28T06:26:39.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.222+0000 7f329b200640 1 -- 172.21.15.96:0/1550763699 shutdown_connections 2024-05-28T06:26:39.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.222+0000 7f329b200640 1 -- 172.21.15.96:0/1550763699 >> 172.21.15.96:0/1550763699 conn(0x7f329c0b1fc0 msgr2=0x7f329c0b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:26:39.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.222+0000 7f329b200640 1 -- 172.21.15.96:0/1550763699 shutdown_connections 2024-05-28T06:26:39.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.222+0000 7f329b200640 1 -- 172.21.15.96:0/1550763699 wait complete. 2024-05-28T06:26:39.221 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.224+0000 7f329b200640 1 Processor -- start 2024-05-28T06:26:39.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.224+0000 7f329b200640 1 -- start start 2024-05-28T06:26:39.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.225+0000 7f329b200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3280004d40 0x7f329c14d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:39.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.225+0000 7f329b200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f329c14a080 0x7f329c149460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:39.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.225+0000 7f329b200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f329c14a5b0 0x7f329c1499a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:39.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.225+0000 7f329b200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f329c0b32c0 con 0x7f329c14a5b0 2024-05-28T06:26:39.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.225+0000 7f329b200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f329c0ae0c0 con 0x7f329c14a080 2024-05-28T06:26:39.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.225+0000 7f329b200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f329c0b3440 con 0x7f3280004d40 2024-05-28T06:26:39.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.225+0000 7f329a800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f329c14a5b0 0x7f329c1499a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:39.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.225+0000 7f329a800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f329c14a5b0 0x7f329c1499a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:34968/0 (socket says 172.21.15.96:34968) 2024-05-28T06:26:39.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.225+0000 7f329a800640 1 -- 172.21.15.96:0/4075117786 learned_addr learned my addr 172.21.15.96:0/4075117786 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:26:39.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.226+0000 7f3291400640 1 --2- 172.21.15.96:0/4075117786 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f329c14a080 0x7f329c149460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:39.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.226+0000 7f3299e00640 1 --2- 172.21.15.96:0/4075117786 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3280004d40 0x7f329c14d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:39.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.226+0000 7f329a800640 1 -- 172.21.15.96:0/4075117786 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3280004d40 msgr2=0x7f329c14d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:39.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.226+0000 7f329a800640 1 --2- 172.21.15.96:0/4075117786 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3280004d40 0x7f329c14d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:39.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.226+0000 7f329a800640 1 -- 172.21.15.96:0/4075117786 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f329c14a080 msgr2=0x7f329c149460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:39.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.226+0000 7f329a800640 1 --2- 172.21.15.96:0/4075117786 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f329c14a080 0x7f329c149460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:39.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.226+0000 7f329a800640 1 -- 172.21.15.96:0/4075117786 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f329c155300 con 0x7f329c14a5b0 2024-05-28T06:26:39.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.226+0000 7f329a800640 1 --2- 172.21.15.96:0/4075117786 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f329c14a5b0 0x7f329c1499a0 secure :-1 s=READY pgs=2162 cs=0 l=1 rev1=1 crypto rx=0x7f329402b070 tx=0x7f3294004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:39.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.226+0000 7f3292a00640 1 -- 172.21.15.96:0/4075117786 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3294045070 con 0x7f329c14a5b0 2024-05-28T06:26:39.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.226+0000 7f3292a00640 1 -- 172.21.15.96:0/4075117786 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3294003990 con 0x7f329c14a5b0 2024-05-28T06:26:39.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.226+0000 7f329b200640 1 -- 172.21.15.96:0/4075117786 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f329c1f4ee0 con 0x7f329c14a5b0 2024-05-28T06:26:39.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.226+0000 7f3292a00640 1 -- 172.21.15.96:0/4075117786 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3294009be0 con 0x7f329c14a5b0 2024-05-28T06:26:39.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.227+0000 7f329b200640 1 -- 172.21.15.96:0/4075117786 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f329c1f54a0 con 0x7f329c14a5b0 2024-05-28T06:26:39.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.228+0000 7f329b200640 1 -- 172.21.15.96:0/4075117786 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f329c0689e0 con 0x7f329c14a5b0 2024-05-28T06:26:39.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.229+0000 7f3292a00640 1 -- 172.21.15.96:0/4075117786 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 49) ==== 94049+0+0 (secure 0 0 0) 0x7f329402e830 con 0x7f329c14a5b0 2024-05-28T06:26:39.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.229+0000 7f3292a00640 1 --2- 172.21.15.96:0/4075117786 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f327806ea20 0x7f3278070ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:39.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.230+0000 7f3292a00640 1 -- 172.21.15.96:0/4075117786 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 5823+0+0 (secure 0 0 0) 0x7f32940ab1f0 con 0x7f329c14a5b0 2024-05-28T06:26:39.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.230+0000 7f3299e00640 1 --2- 172.21.15.96:0/4075117786 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f327806ea20 0x7f3278070ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:39.226 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.231+0000 7f3299e00640 1 --2- 172.21.15.96:0/4075117786 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f327806ea20 0x7f3278070ee0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f328c000c00 tx=0x7f328c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:39.227 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.234+0000 7f3292a00640 1 -- 172.21.15.96:0/4075117786 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f329403d070 con 0x7f329c14a5b0 2024-05-28T06:26:39.438 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.445+0000 7f329b200640 1 -- 172.21.15.96:0/4075117786 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f329c0b2e90 con 0x7f329c14a5b0 2024-05-28T06:26:39.439 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.446+0000 7f3292a00640 1 -- 172.21.15.96:0/4075117786 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v132) ==== 80+0+0 (secure 0 0 0) 0x7f3294032030 con 0x7f329c14a5b0 2024-05-28T06:26:39.439 INFO:teuthology.orchestra.run.smithi096.stderr:listed 0 entries 2024-05-28T06:26:39.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.448+0000 7f329b200640 1 -- 172.21.15.96:0/4075117786 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f327806ea20 msgr2=0x7f3278070ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:39.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.448+0000 7f329b200640 1 --2- 172.21.15.96:0/4075117786 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f327806ea20 0x7f3278070ee0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f328c000c00 tx=0x7f328c009040 comp rx=0 tx=0).stop 2024-05-28T06:26:39.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.448+0000 7f329b200640 1 -- 172.21.15.96:0/4075117786 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f329c14a5b0 msgr2=0x7f329c1499a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:39.442 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.448+0000 7f329b200640 1 --2- 172.21.15.96:0/4075117786 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f329c14a5b0 0x7f329c1499a0 secure :-1 s=READY pgs=2162 cs=0 l=1 rev1=1 crypto rx=0x7f329402b070 tx=0x7f3294004ab0 comp rx=0 tx=0).stop 2024-05-28T06:26:39.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.448+0000 7f329b200640 1 -- 172.21.15.96:0/4075117786 shutdown_connections 2024-05-28T06:26:39.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.448+0000 7f329b200640 1 --2- 172.21.15.96:0/4075117786 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f327806ea20 0x7f3278070ee0 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:39.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.448+0000 7f329b200640 1 --2- 172.21.15.96:0/4075117786 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f329c14a5b0 0x7f329c1499a0 unknown :-1 s=CLOSED pgs=2162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:39.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.448+0000 7f329b200640 1 --2- 172.21.15.96:0/4075117786 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f329c14a080 0x7f329c149460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:39.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.448+0000 7f329b200640 1 --2- 172.21.15.96:0/4075117786 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f3280004d40 0x7f329c14d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:39.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.448+0000 7f329b200640 1 -- 172.21.15.96:0/4075117786 >> 172.21.15.96:0/4075117786 conn(0x7f329c0b1fc0 msgr2=0x7f329c06ea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:26:39.443 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.449+0000 7f329b200640 1 -- 172.21.15.96:0/4075117786 shutdown_connections 2024-05-28T06:26:39.444 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.449+0000 7f329b200640 1 -- 172.21.15.96:0/4075117786 wait complete. 2024-05-28T06:26:39.461 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:26:39.461 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:26:39.509 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:26:39.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.760+0000 7fc2e0a00640 1 Processor -- start 2024-05-28T06:26:39.754 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.760+0000 7fc2e0a00640 1 -- start start 2024-05-28T06:26:39.755 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.762+0000 7fc2e0a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc2dc0ad940 0x7fc2dc0b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:39.755 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.762+0000 7fc2e0a00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2dc156f10 con 0x7fc2dc06ee40 2024-05-28T06:26:39.755 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.762+0000 7fc2e0a00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2dc06dda0 con 0x7fc2dc156b30 2024-05-28T06:26:39.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.762+0000 7fc2e0a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fc2dc06e080 con 0x7fc2dc0ad940 2024-05-28T06:26:39.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.762+0000 7fc2d3e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fc2dc06ee40 0x7fc2dc0ad230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:44814/0 (socket says 172.21.15.96:44814) 2024-05-28T06:26:39.756 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.762+0000 7fc2d3e00640 1 -- 172.21.15.96:0/2233514779 learned_addr learned my addr 172.21.15.96:0/2233514779 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:26:39.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.762+0000 7fc2db200640 1 --2- 172.21.15.96:0/2233514779 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc2dc0ad940 0x7fc2dc0b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:39.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.763+0000 7fc2d9e00640 1 -- 172.21.15.96:0/2233514779 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 942997095 0 0) 0x7fc2dc156f10 con 0x7fc2dc06ee40 2024-05-28T06:26:39.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.763+0000 7fc2d9e00640 1 -- 172.21.15.96:0/2233514779 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc2bc003520 con 0x7fc2dc06ee40 2024-05-28T06:26:39.758 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.763+0000 7fc2d9e00640 1 -- 172.21.15.96:0/2233514779 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1933918480 0 0) 0x7fc2bc003520 con 0x7fc2dc06ee40 2024-05-28T06:26:39.759 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.763+0000 7fc2d9e00640 1 -- 172.21.15.96:0/2233514779 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc2dc0ad940 msgr2=0x7fc2dc0b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:39.759 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.763+0000 7fc2d9e00640 1 --2- 172.21.15.96:0/2233514779 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc2dc0ad940 0x7fc2dc0b32c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:39.759 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.763+0000 7fc2d9e00640 1 -- 172.21.15.96:0/2233514779 >> v1:172.21.15.123:6789/0 conn(0x7fc2dc156b30 legacy=0x7fc2dc0af6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:39.759 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.763+0000 7fc2d9e00640 1 -- 172.21.15.96:0/2233514779 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2dc06e360 con 0x7fc2dc06ee40 2024-05-28T06:26:39.759 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.763+0000 7fc2d9e00640 1 -- 172.21.15.96:0/2233514779 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7fc2c4001670 con 0x7fc2dc06ee40 2024-05-28T06:26:39.759 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.763+0000 7fc2d9e00640 1 -- 172.21.15.96:0/2233514779 >> v1:172.21.15.96:6789/0 conn(0x7fc2dc06ee40 legacy=0x7fc2dc0ad230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:39.759 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.763+0000 7fc2d9e00640 1 --2- 172.21.15.96:0/2233514779 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc2bc0039e0 0x7fc2bc003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:39.759 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.763+0000 7fc2d9e00640 1 --2- 172.21.15.96:0/2233514779 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc2dc0ad940 0x7fc2bc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:39.760 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.763+0000 7fc2d9e00640 1 --2- 172.21.15.96:0/2233514779 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc2bc004d40 0x7fc2bc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:39.760 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.763+0000 7fc2d9e00640 1 -- 172.21.15.96:0/2233514779 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc2dc06e080 con 0x7fc2bc004d40 2024-05-28T06:26:39.760 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.763+0000 7fc2d9e00640 1 -- 172.21.15.96:0/2233514779 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc2bc00bb80 con 0x7fc2bc0039e0 2024-05-28T06:26:39.760 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.763+0000 7fc2d9e00640 1 -- 172.21.15.96:0/2233514779 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fc2bc00bd80 con 0x7fc2dc0ad940 2024-05-28T06:26:39.760 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.764+0000 7fc2db200640 1 --2- 172.21.15.96:0/2233514779 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc2bc004d40 0x7fc2bc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:39.760 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.764+0000 7fc2da800640 1 --2- 172.21.15.96:0/2233514779 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc2bc0039e0 0x7fc2bc003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:39.760 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.764+0000 7fc2d3e00640 1 --2- 172.21.15.96:0/2233514779 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc2dc0ad940 0x7fc2bc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:39.760 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.764+0000 7fc2db200640 1 -- 172.21.15.96:0/2233514779 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc2dc0ad940 msgr2=0x7fc2bc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:39.761 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.764+0000 7fc2db200640 1 --2- 172.21.15.96:0/2233514779 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc2dc0ad940 0x7fc2bc004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:39.761 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.764+0000 7fc2db200640 1 -- 172.21.15.96:0/2233514779 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc2bc0039e0 msgr2=0x7fc2bc003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:39.761 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.764+0000 7fc2db200640 1 --2- 172.21.15.96:0/2233514779 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc2bc0039e0 0x7fc2bc003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:39.761 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.764+0000 7fc2db200640 1 -- 172.21.15.96:0/2233514779 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2bc003520 con 0x7fc2bc004d40 2024-05-28T06:26:39.764 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.765+0000 7fc2db200640 1 --2- 172.21.15.96:0/2233514779 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc2bc004d40 0x7fc2bc0051c0 secure :-1 s=READY pgs=2164 cs=0 l=1 rev1=1 crypto rx=0x7fc2d4009f90 tx=0x7fc2bc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eefe0ccf1601681e server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:39.764 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.765+0000 7fc2d9e00640 1 -- 172.21.15.96:0/2233514779 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2d4009c10 con 0x7fc2bc004d40 2024-05-28T06:26:39.764 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.765+0000 7fc2d9e00640 1 -- 172.21.15.96:0/2233514779 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2d4036070 con 0x7fc2bc004d40 2024-05-28T06:26:39.764 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.765+0000 7fc2d9e00640 1 -- 172.21.15.96:0/2233514779 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2d4031810 con 0x7fc2bc004d40 2024-05-28T06:26:39.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.766+0000 7fc2e0a00640 1 -- 172.21.15.96:0/2233514779 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc2bc004d40 msgr2=0x7fc2bc0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:39.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.766+0000 7fc2e0a00640 1 --2- 172.21.15.96:0/2233514779 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc2bc004d40 0x7fc2bc0051c0 secure :-1 s=READY pgs=2164 cs=0 l=1 rev1=1 crypto rx=0x7fc2d4009f90 tx=0x7fc2bc005830 comp rx=0 tx=0).stop 2024-05-28T06:26:39.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.766+0000 7fc2db200640 1 -- 172.21.15.96:0/2233514779 reap_dead start 2024-05-28T06:26:39.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.766+0000 7fc2e0a00640 1 -- 172.21.15.96:0/2233514779 shutdown_connections 2024-05-28T06:26:39.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.766+0000 7fc2e0a00640 1 -- 172.21.15.96:0/2233514779 >> 172.21.15.96:0/2233514779 conn(0x7fc2dc0b1fc0 msgr2=0x7fc2dc0b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:26:39.765 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.766+0000 7fc2e0a00640 1 -- 172.21.15.96:0/2233514779 shutdown_connections 2024-05-28T06:26:39.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.766+0000 7fc2e0a00640 1 -- 172.21.15.96:0/2233514779 wait complete. 2024-05-28T06:26:39.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.768+0000 7fc2e0a00640 1 Processor -- start 2024-05-28T06:26:39.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.768+0000 7fc2e0a00640 1 -- start start 2024-05-28T06:26:39.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.769+0000 7fc2e0a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc2bc004d40 0x7fc2dc1f7470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:39.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.769+0000 7fc2e0a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc2dc15af20 0x7fc2dc149e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:39.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.769+0000 7fc2e0a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc2dc1493e0 0x7fc2dc14a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:39.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.769+0000 7fc2e0a00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc2dc0b2db0 con 0x7fc2bc004d40 2024-05-28T06:26:39.766 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.769+0000 7fc2e0a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc2dc0b2c30 con 0x7fc2dc15af20 2024-05-28T06:26:39.767 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.769+0000 7fc2e0a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fc2dc06f2f0 con 0x7fc2dc1493e0 2024-05-28T06:26:39.767 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.769+0000 7fc2da800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc2bc004d40 0x7fc2dc1f7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:39.767 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.769+0000 7fc2da800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc2bc004d40 0x7fc2dc1f7470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:34984/0 (socket says 172.21.15.96:34984) 2024-05-28T06:26:39.767 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.769+0000 7fc2da800640 1 -- 172.21.15.96:0/3559690828 learned_addr learned my addr 172.21.15.96:0/3559690828 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:26:39.767 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.770+0000 7fc2d3e00640 1 --2- 172.21.15.96:0/3559690828 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc2dc15af20 0x7fc2dc149e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:39.767 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.770+0000 7fc2da800640 1 -- 172.21.15.96:0/3559690828 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc2dc1493e0 msgr2=0x7fc2dc14a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:39.768 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.770+0000 7fc2db200640 1 --2- 172.21.15.96:0/3559690828 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc2dc1493e0 0x7fc2dc14a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:39.768 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.770+0000 7fc2da800640 1 --2- 172.21.15.96:0/3559690828 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc2dc1493e0 0x7fc2dc14a3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:39.768 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.770+0000 7fc2da800640 1 -- 172.21.15.96:0/3559690828 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc2dc15af20 msgr2=0x7fc2dc149e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:39.768 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.770+0000 7fc2da800640 1 --2- 172.21.15.96:0/3559690828 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc2dc15af20 0x7fc2dc149e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:39.768 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.770+0000 7fc2da800640 1 -- 172.21.15.96:0/3559690828 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2dc149c60 con 0x7fc2bc004d40 2024-05-28T06:26:39.768 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.770+0000 7fc2da800640 1 --2- 172.21.15.96:0/3559690828 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc2bc004d40 0x7fc2dc1f7470 secure :-1 s=READY pgs=2165 cs=0 l=1 rev1=1 crypto rx=0x7fc2cc00e550 tx=0x7fc2cc00ea10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:39.768 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.770+0000 7fc2d2a00640 1 -- 172.21.15.96:0/3559690828 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2cc0052e0 con 0x7fc2bc004d40 2024-05-28T06:26:39.768 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.770+0000 7fc2d2a00640 1 -- 172.21.15.96:0/3559690828 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2cc018070 con 0x7fc2bc004d40 2024-05-28T06:26:39.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.770+0000 7fc2d2a00640 1 -- 172.21.15.96:0/3559690828 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2cc003810 con 0x7fc2bc004d40 2024-05-28T06:26:39.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.770+0000 7fc2e0a00640 1 -- 172.21.15.96:0/3559690828 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2dc15b940 con 0x7fc2bc004d40 2024-05-28T06:26:39.769 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.771+0000 7fc2e0a00640 1 -- 172.21.15.96:0/3559690828 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2dc14ae10 con 0x7fc2bc004d40 2024-05-28T06:26:39.770 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.772+0000 7fc2e0a00640 1 -- 172.21.15.96:0/3559690828 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2dc0689e0 con 0x7fc2bc004d40 2024-05-28T06:26:39.770 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.773+0000 7fc2d2a00640 1 -- 172.21.15.96:0/3559690828 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 49) ==== 94049+0+0 (secure 0 0 0) 0x7fc2cc0039b0 con 0x7fc2bc004d40 2024-05-28T06:26:39.770 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.774+0000 7fc2d2a00640 1 --2- 172.21.15.96:0/3559690828 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc2b806ea20 0x7fc2b8070ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:39.770 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.774+0000 7fc2d2a00640 1 -- 172.21.15.96:0/3559690828 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 5823+0+0 (secure 0 0 0) 0x7fc2cc08a8f0 con 0x7fc2bc004d40 2024-05-28T06:26:39.770 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.775+0000 7fc2d3e00640 1 --2- 172.21.15.96:0/3559690828 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc2b806ea20 0x7fc2b8070ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:39.770 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.776+0000 7fc2d3e00640 1 --2- 172.21.15.96:0/3559690828 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc2b806ea20 0x7fc2b8070ee0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7fc2c4001bf0 tx=0x7fc2c4000f70 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:39.771 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.778+0000 7fc2d2a00640 1 -- 172.21.15.96:0/3559690828 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc2dc149c60 con 0x7fc2bc004d40 2024-05-28T06:26:39.975 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.981+0000 7fc2e0a00640 1 -- 172.21.15.96:0/3559690828 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc2dc06f7f0 con 0x7fc2bc004d40 2024-05-28T06:26:39.978 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.985+0000 7fc2d2a00640 1 -- 172.21.15.96:0/3559690828 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v132) ==== 74+0+16877 (secure 0 0 0) 0x7fc2cc05a820 con 0x7fc2bc004d40 2024-05-28T06:26:39.979 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:26:39.979 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":132,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:24:54.967903+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-05-28T06:24:21.166134+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"129","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-05-28T06:24:22.415938+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"130","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"19.32","mappings":[{"from":2,"to":0}]},{"pgid":"19.3a","mappings":[{"from":2,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":13,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:26:39.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.987+0000 7fc2e0a00640 1 -- 172.21.15.96:0/3559690828 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc2b806ea20 msgr2=0x7fc2b8070ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:39.981 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.987+0000 7fc2e0a00640 1 --2- 172.21.15.96:0/3559690828 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc2b806ea20 0x7fc2b8070ee0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7fc2c4001bf0 tx=0x7fc2c4000f70 comp rx=0 tx=0).stop 2024-05-28T06:26:39.982 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.987+0000 7fc2e0a00640 1 -- 172.21.15.96:0/3559690828 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc2bc004d40 msgr2=0x7fc2dc1f7470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:39.982 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.987+0000 7fc2e0a00640 1 --2- 172.21.15.96:0/3559690828 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc2bc004d40 0x7fc2dc1f7470 secure :-1 s=READY pgs=2165 cs=0 l=1 rev1=1 crypto rx=0x7fc2cc00e550 tx=0x7fc2cc00ea10 comp rx=0 tx=0).stop 2024-05-28T06:26:39.982 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.987+0000 7fc2e0a00640 1 -- 172.21.15.96:0/3559690828 shutdown_connections 2024-05-28T06:26:39.982 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.987+0000 7fc2e0a00640 1 --2- 172.21.15.96:0/3559690828 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc2b806ea20 0x7fc2b8070ee0 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:39.982 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.987+0000 7fc2e0a00640 1 --2- 172.21.15.96:0/3559690828 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc2dc1493e0 0x7fc2dc14a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:39.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.987+0000 7fc2e0a00640 1 --2- 172.21.15.96:0/3559690828 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc2dc15af20 0x7fc2dc149e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:39.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.988+0000 7fc2e0a00640 1 --2- 172.21.15.96:0/3559690828 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc2bc004d40 0x7fc2dc1f7470 unknown :-1 s=CLOSED pgs=2165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:39.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.988+0000 7fc2e0a00640 1 -- 172.21.15.96:0/3559690828 >> 172.21.15.96:0/3559690828 conn(0x7fc2dc0b1fc0 msgr2=0x7fc2dc1558b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:26:39.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.988+0000 7fc2e0a00640 1 -- 172.21.15.96:0/3559690828 shutdown_connections 2024-05-28T06:26:39.983 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:39.988+0000 7fc2e0a00640 1 -- 172.21.15.96:0/3559690828 wait complete. 2024-05-28T06:26:40.001 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-28T06:03:59.517102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-28T06:24:21.166134+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '129', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 19, 'pool_name': 'cephfs_data', 'create_time': '2024-05-28T06:24:22.415938+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '130', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-28T06:26:40.002 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-28T06:26:40.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.209+0000 7f70d6800640 1 Processor -- start 2024-05-28T06:26:40.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.209+0000 7f70d6800640 1 -- start start 2024-05-28T06:26:40.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.211+0000 7f70d6800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f70d0161a90 0x7f70d006e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:40.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.211+0000 7f70d6800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f70d006e9b0 0x7f70d00ad2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:40.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.211+0000 7f70d6800640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70d00b3d40 con 0x7f70d0161e70 2024-05-28T06:26:40.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.211+0000 7f70d6800640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f70d00b4020 con 0x7f70d006e9b0 2024-05-28T06:26:40.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.211+0000 7f70d6800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f70d00b4300 con 0x7f70d0161a90 2024-05-28T06:26:40.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.211+0000 7f70cfe00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f70d006e9b0 0x7f70d00ad2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:40.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.211+0000 7f70cfe00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f70d006e9b0 0x7f70d00ad2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:34986/0 (socket says 172.21.15.96:34986) 2024-05-28T06:26:40.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.211+0000 7f70cfe00640 1 -- 172.21.15.96:0/3741951441 learned_addr learned my addr 172.21.15.96:0/3741951441 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:26:40.206 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.212+0000 7f70cf400640 1 --2- 172.21.15.96:0/3741951441 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f70d0161a90 0x7f70d006e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:40.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.212+0000 7f70cfe00640 1 -- 172.21.15.96:0/3741951441 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f70d0161a90 msgr2=0x7f70d006e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:40.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.212+0000 7f70cfe00640 1 --2- 172.21.15.96:0/3741951441 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f70d0161a90 0x7f70d006e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:40.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.212+0000 7f70cfe00640 1 -- 172.21.15.96:0/3741951441 >> v1:172.21.15.123:6789/0 conn(0x7f70d0161e70 legacy=0x7f70d00b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:40.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.212+0000 7f70cfe00640 1 -- 172.21.15.96:0/3741951441 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70d006dc70 con 0x7f70d006e9b0 2024-05-28T06:26:40.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.213+0000 7f70cfe00640 1 --2- 172.21.15.96:0/3741951441 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f70d006e9b0 0x7f70d00ad2a0 secure :-1 s=READY pgs=2166 cs=0 l=1 rev1=1 crypto rx=0x7f70c8009e50 tx=0x7f70c802a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=846026137749d2d server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:40.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.213+0000 7f70cea00640 1 -- 172.21.15.96:0/3741951441 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70c8036070 con 0x7f70d006e9b0 2024-05-28T06:26:40.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.213+0000 7f70cea00640 1 -- 172.21.15.96:0/3741951441 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70c8031600 con 0x7f70d006e9b0 2024-05-28T06:26:40.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.213+0000 7f70cea00640 1 -- 172.21.15.96:0/3741951441 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70c80319d0 con 0x7f70d006e9b0 2024-05-28T06:26:40.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.214+0000 7f70d6800640 1 -- 172.21.15.96:0/3741951441 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f70d006e9b0 msgr2=0x7f70d00ad2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:40.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.214+0000 7f70d6800640 1 --2- 172.21.15.96:0/3741951441 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f70d006e9b0 0x7f70d00ad2a0 secure :-1 s=READY pgs=2166 cs=0 l=1 rev1=1 crypto rx=0x7f70c8009e50 tx=0x7f70c802a6c0 comp rx=0 tx=0).stop 2024-05-28T06:26:40.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.214+0000 7f70d6800640 1 -- 172.21.15.96:0/3741951441 shutdown_connections 2024-05-28T06:26:40.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.214+0000 7f70d6800640 1 --2- 172.21.15.96:0/3741951441 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f70d006e9b0 0x7f70d00ad2a0 unknown :-1 s=CLOSED pgs=2166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:40.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.214+0000 7f70d6800640 1 --2- 172.21.15.96:0/3741951441 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f70d0161a90 0x7f70d006e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:40.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.214+0000 7f70d6800640 1 -- 172.21.15.96:0/3741951441 >> 172.21.15.96:0/3741951441 conn(0x7f70d006bd20 msgr2=0x7f70d006c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:26:40.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.214+0000 7f70d6800640 1 -- 172.21.15.96:0/3741951441 shutdown_connections 2024-05-28T06:26:40.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.214+0000 7f70d6800640 1 -- 172.21.15.96:0/3741951441 wait complete. 2024-05-28T06:26:40.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.216+0000 7f70d6800640 1 Processor -- start 2024-05-28T06:26:40.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.216+0000 7f70d6800640 1 -- start start 2024-05-28T06:26:40.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.217+0000 7f70d6800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f70d006e9b0 0x7f70d0152530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:40.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.217+0000 7f70d6800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f70d0161a90 0x7f70d01552f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:40.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.217+0000 7f70d6800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f70d0161e70 0x7f70d0155830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:40.211 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.217+0000 7f70d6800640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f70d00b39d0 con 0x7f70d0161a90 2024-05-28T06:26:40.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.217+0000 7f70d6800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f70d00b5020 con 0x7f70d0161e70 2024-05-28T06:26:40.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.217+0000 7f70d6800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f70d00ad7e0 con 0x7f70d006e9b0 2024-05-28T06:26:40.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.217+0000 7f70c6a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f70d0161a90 0x7f70d01552f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:40.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.217+0000 7f70c6a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f70d0161a90 0x7f70d01552f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:34992/0 (socket says 172.21.15.96:34992) 2024-05-28T06:26:40.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.217+0000 7f70c6a00640 1 -- 172.21.15.96:0/269055466 learned_addr learned my addr 172.21.15.96:0/269055466 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:26:40.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.218+0000 7f70cf400640 1 --2- 172.21.15.96:0/269055466 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f70d006e9b0 0x7f70d0152530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:40.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.218+0000 7f70c6a00640 1 -- 172.21.15.96:0/269055466 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f70d006e9b0 msgr2=0x7f70d0152530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:40.212 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.218+0000 7f70c6a00640 1 --2- 172.21.15.96:0/269055466 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f70d006e9b0 0x7f70d0152530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:40.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.218+0000 7f70c6a00640 1 -- 172.21.15.96:0/269055466 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f70d0161e70 msgr2=0x7f70d0155830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:40.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.218+0000 7f70c6a00640 1 --2- 172.21.15.96:0/269055466 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f70d0161e70 0x7f70d0155830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:40.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.218+0000 7f70c6a00640 1 -- 172.21.15.96:0/269055466 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70d015b860 con 0x7f70d0161a90 2024-05-28T06:26:40.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.218+0000 7f70c6a00640 1 --2- 172.21.15.96:0/269055466 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f70d0161a90 0x7f70d01552f0 secure :-1 s=READY pgs=2167 cs=0 l=1 rev1=1 crypto rx=0x7f70b800b3c0 tx=0x7f70b800b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:40.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.218+0000 7f70c7e00640 1 -- 172.21.15.96:0/269055466 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70b8011830 con 0x7f70d0161a90 2024-05-28T06:26:40.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.218+0000 7f70c7e00640 1 -- 172.21.15.96:0/269055466 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70b8016070 con 0x7f70d0161a90 2024-05-28T06:26:40.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.218+0000 7f70c7e00640 1 -- 172.21.15.96:0/269055466 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70b8011c10 con 0x7f70d0161a90 2024-05-28T06:26:40.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.218+0000 7f70d6800640 1 -- 172.21.15.96:0/269055466 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70d015f080 con 0x7f70d0161a90 2024-05-28T06:26:40.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.218+0000 7f70d6800640 1 -- 172.21.15.96:0/269055466 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70d0156040 con 0x7f70d0161a90 2024-05-28T06:26:40.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.220+0000 7f70d6800640 1 -- 172.21.15.96:0/269055466 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70d00aed60 con 0x7f70d0161a90 2024-05-28T06:26:40.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.221+0000 7f70c7e00640 1 -- 172.21.15.96:0/269055466 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 49) ==== 94049+0+0 (secure 0 0 0) 0x7f70b801b460 con 0x7f70d0161a90 2024-05-28T06:26:40.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.222+0000 7f70c7e00640 1 --2- 172.21.15.96:0/269055466 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f70ac06ea20 0x7f70ac070ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:40.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.222+0000 7f70c7e00640 1 -- 172.21.15.96:0/269055466 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 5823+0+0 (secure 0 0 0) 0x7f70b808a6a0 con 0x7f70d0161a90 2024-05-28T06:26:40.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.222+0000 7f70cf400640 1 --2- 172.21.15.96:0/269055466 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f70ac06ea20 0x7f70ac070ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:40.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.224+0000 7f70cf400640 1 --2- 172.21.15.96:0/269055466 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f70ac06ea20 0x7f70ac070ee0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f70c0006fd0 tx=0x7f70c0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:40.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.226+0000 7f70c7e00640 1 -- 172.21.15.96:0/269055466 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f70d00aed60 con 0x7f70d0161a90 2024-05-28T06:26:40.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.429+0000 7f70d6800640 1 -- 172.21.15.96:0/269055466 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f70d0072ca0 con 0x7f70d0161a90 2024-05-28T06:26:40.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.430+0000 7f70c7e00640 1 -- 172.21.15.96:0/269055466 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v132) ==== 93+0+10 (secure 0 0 0) 0x7f70b8020020 con 0x7f70d0161a90 2024-05-28T06:26:40.424 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 1 2024-05-28T06:26:40.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.433+0000 7f70d6800640 1 -- 172.21.15.96:0/269055466 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f70ac06ea20 msgr2=0x7f70ac070ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:40.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.433+0000 7f70d6800640 1 --2- 172.21.15.96:0/269055466 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f70ac06ea20 0x7f70ac070ee0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f70c0006fd0 tx=0x7f70c0008040 comp rx=0 tx=0).stop 2024-05-28T06:26:40.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.433+0000 7f70d6800640 1 -- 172.21.15.96:0/269055466 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f70d0161a90 msgr2=0x7f70d01552f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:40.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.433+0000 7f70d6800640 1 --2- 172.21.15.96:0/269055466 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f70d0161a90 0x7f70d01552f0 secure :-1 s=READY pgs=2167 cs=0 l=1 rev1=1 crypto rx=0x7f70b800b3c0 tx=0x7f70b800b880 comp rx=0 tx=0).stop 2024-05-28T06:26:40.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.433+0000 7f70d6800640 1 -- 172.21.15.96:0/269055466 shutdown_connections 2024-05-28T06:26:40.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.433+0000 7f70d6800640 1 --2- 172.21.15.96:0/269055466 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f70ac06ea20 0x7f70ac070ee0 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:40.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.433+0000 7f70d6800640 1 --2- 172.21.15.96:0/269055466 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f70d0161e70 0x7f70d0155830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:40.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.433+0000 7f70d6800640 1 --2- 172.21.15.96:0/269055466 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f70d0161a90 0x7f70d01552f0 unknown :-1 s=CLOSED pgs=2167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:40.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.433+0000 7f70d6800640 1 --2- 172.21.15.96:0/269055466 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f70d006e9b0 0x7f70d0152530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:40.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.433+0000 7f70d6800640 1 -- 172.21.15.96:0/269055466 >> 172.21.15.96:0/269055466 conn(0x7f70d006bd20 msgr2=0x7f70d00afe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:26:40.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.433+0000 7f70d6800640 1 -- 172.21.15.96:0/269055466 shutdown_connections 2024-05-28T06:26:40.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.433+0000 7f70d6800640 1 -- 172.21.15.96:0/269055466 wait complete. 2024-05-28T06:26:40.445 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-28T06:26:40.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.656+0000 7f5e0d200640 1 Processor -- start 2024-05-28T06:26:40.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.656+0000 7f5e0d200640 1 -- start start 2024-05-28T06:26:40.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.658+0000 7f5e0d200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f5e08153510 0x7f5e080b4d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:40.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.658+0000 7f5e0d200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f5e0815d270 0x7f5e080ccff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:40.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.658+0000 7f5e0d200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e0815d900 con 0x7f5e080ccb90 2024-05-28T06:26:40.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.658+0000 7f5e0d200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f5e081538f0 con 0x7f5e0815d270 2024-05-28T06:26:40.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.658+0000 7f5e0d200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f5e080cd530 con 0x7f5e08153510 2024-05-28T06:26:40.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.658+0000 7f5e07e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f5e0815d270 0x7f5e080ccff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:40.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.658+0000 7f5dfea00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f5e080ccb90 0x7f5e0815cb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:44818/0 (socket says 172.21.15.96:44818) 2024-05-28T06:26:40.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.658+0000 7f5dfea00640 1 -- 172.21.15.96:0/537547890 learned_addr learned my addr 172.21.15.96:0/537547890 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:26:40.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.659+0000 7f5e07e00640 1 -- 172.21.15.96:0/537547890 >> v2:172.21.15.123:3300/0 conn(0x7f5e08153510 msgr2=0x7f5e080b4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:40.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.659+0000 7f5e07400640 1 --2- 172.21.15.96:0/537547890 >> v2:172.21.15.123:3300/0 conn(0x7f5e08153510 0x7f5e080b4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:40.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.659+0000 7f5e07e00640 1 --2- 172.21.15.96:0/537547890 >> v2:172.21.15.123:3300/0 conn(0x7f5e08153510 0x7f5e080b4d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:40.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.659+0000 7f5e07e00640 1 -- 172.21.15.96:0/537547890 >> v1:172.21.15.96:6789/0 conn(0x7f5e080ccb90 legacy=0x7f5e0815cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:40.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.659+0000 7f5e07e00640 1 -- 172.21.15.96:0/537547890 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e080b3f90 con 0x7f5e0815d270 2024-05-28T06:26:40.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.659+0000 7f5e06a00640 1 -- 172.21.15.96:0/537547890 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2733020817 0 0) 0x7f5e0815d900 con 0x7f5e080ccb90 2024-05-28T06:26:40.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.659+0000 7f5e07400640 1 --2- 172.21.15.96:0/537547890 >> v2:172.21.15.123:3300/0 conn(0x7f5e08153510 0x7f5e080b4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:26:40.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.660+0000 7f5e07e00640 1 --2- 172.21.15.96:0/537547890 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5e0815d270 0x7f5e080ccff0 secure :-1 s=READY pgs=2169 cs=0 l=1 rev1=1 crypto rx=0x7f5e080b3970 tx=0x7f5e0002a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdc4dad136e396f8 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:40.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.660+0000 7f5e06a00640 1 -- 172.21.15.96:0/537547890 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e00007740 con 0x7f5e0815d270 2024-05-28T06:26:40.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.660+0000 7f5e06a00640 1 -- 172.21.15.96:0/537547890 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e000078e0 con 0x7f5e0815d270 2024-05-28T06:26:40.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.660+0000 7f5e06a00640 1 -- 172.21.15.96:0/537547890 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e00007cb0 con 0x7f5e0815d270 2024-05-28T06:26:40.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.661+0000 7f5e0d200640 1 -- 172.21.15.96:0/537547890 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5e0815d270 msgr2=0x7f5e080ccff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:40.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.661+0000 7f5e0d200640 1 --2- 172.21.15.96:0/537547890 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5e0815d270 0x7f5e080ccff0 secure :-1 s=READY pgs=2169 cs=0 l=1 rev1=1 crypto rx=0x7f5e080b3970 tx=0x7f5e0002a5f0 comp rx=0 tx=0).stop 2024-05-28T06:26:40.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.661+0000 7f5e0d200640 1 -- 172.21.15.96:0/537547890 shutdown_connections 2024-05-28T06:26:40.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.661+0000 7f5e0d200640 1 --2- 172.21.15.96:0/537547890 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5e0815d270 0x7f5e080ccff0 unknown :-1 s=CLOSED pgs=2169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:40.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.661+0000 7f5e0d200640 1 --2- 172.21.15.96:0/537547890 >> v2:172.21.15.123:3300/0 conn(0x7f5e08153510 0x7f5e080b4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:40.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.661+0000 7f5e0d200640 1 -- 172.21.15.96:0/537547890 >> 172.21.15.96:0/537547890 conn(0x7f5e0806b900 msgr2=0x7f5e0806bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:26:40.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.661+0000 7f5e0d200640 1 -- 172.21.15.96:0/537547890 shutdown_connections 2024-05-28T06:26:40.657 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.661+0000 7f5e0d200640 1 -- 172.21.15.96:0/537547890 wait complete. 2024-05-28T06:26:40.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.663+0000 7f5e0d200640 1 Processor -- start 2024-05-28T06:26:40.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.663+0000 7f5e0d200640 1 -- start start 2024-05-28T06:26:40.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.664+0000 7f5e0d200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e080ccb90 0x7f5e08148360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:40.659 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.664+0000 7f5e0d200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5e08153510 0x7f5e0815b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:40.659 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.664+0000 7f5e0d200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5e0815d270 0x7f5e0815b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:40.659 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.664+0000 7f5e0d200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f5e0815d6a0 con 0x7f5e0815d270 2024-05-28T06:26:40.659 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.664+0000 7f5e0d200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5e080b48c0 con 0x7f5e080ccb90 2024-05-28T06:26:40.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.664+0000 7f5e0d200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f5e080b4550 con 0x7f5e08153510 2024-05-28T06:26:40.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.664+0000 7f5e07e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5e0815d270 0x7f5e0815b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:40.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.664+0000 7f5e07e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5e0815d270 0x7f5e0815b970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35004/0 (socket says 172.21.15.96:35004) 2024-05-28T06:26:40.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.664+0000 7f5e07e00640 1 -- 172.21.15.96:0/3952838082 learned_addr learned my addr 172.21.15.96:0/3952838082 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:26:40.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.664+0000 7f5e07e00640 1 -- 172.21.15.96:0/3952838082 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5e08153510 msgr2=0x7f5e0815b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:40.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.664+0000 7f5e07e00640 1 --2- 172.21.15.96:0/3952838082 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5e08153510 0x7f5e0815b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:40.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.664+0000 7f5e07e00640 1 -- 172.21.15.96:0/3952838082 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e080ccb90 msgr2=0x7f5e08148360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:40.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.664+0000 7f5e07400640 1 --2- 172.21.15.96:0/3952838082 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e080ccb90 0x7f5e08148360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:40.661 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.664+0000 7f5e07e00640 1 --2- 172.21.15.96:0/3952838082 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e080ccb90 0x7f5e08148360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:40.661 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.664+0000 7f5e07e00640 1 -- 172.21.15.96:0/3952838082 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e080cf750 con 0x7f5e0815d270 2024-05-28T06:26:40.661 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.665+0000 7f5e07e00640 1 --2- 172.21.15.96:0/3952838082 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5e0815d270 0x7f5e0815b970 secure :-1 s=READY pgs=2170 cs=0 l=1 rev1=1 crypto rx=0x7f5e00009e80 tx=0x7f5e00004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:40.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.665+0000 7f5dffe00640 1 -- 172.21.15.96:0/3952838082 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e00040070 con 0x7f5e0815d270 2024-05-28T06:26:40.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.665+0000 7f5dffe00640 1 -- 172.21.15.96:0/3952838082 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e00031c40 con 0x7f5e0815d270 2024-05-28T06:26:40.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.665+0000 7f5dffe00640 1 -- 172.21.15.96:0/3952838082 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e00003990 con 0x7f5e0815d270 2024-05-28T06:26:40.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.665+0000 7f5e0d200640 1 -- 172.21.15.96:0/3952838082 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e08150e00 con 0x7f5e0815d270 2024-05-28T06:26:40.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.665+0000 7f5e0d200640 1 -- 172.21.15.96:0/3952838082 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e080cfcf0 con 0x7f5e0815d270 2024-05-28T06:26:40.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.667+0000 7f5e0d200640 1 -- 172.21.15.96:0/3952838082 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e080689e0 con 0x7f5e0815d270 2024-05-28T06:26:40.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.668+0000 7f5dffe00640 1 -- 172.21.15.96:0/3952838082 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 49) ==== 94049+0+0 (secure 0 0 0) 0x7f5e00039040 con 0x7f5e0815d270 2024-05-28T06:26:40.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.668+0000 7f5dffe00640 1 --2- 172.21.15.96:0/3952838082 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5de406ea20 0x7f5de4070ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:40.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.669+0000 7f5dffe00640 1 -- 172.21.15.96:0/3952838082 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 5823+0+0 (secure 0 0 0) 0x7f5e000abfa0 con 0x7f5e0815d270 2024-05-28T06:26:40.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.669+0000 7f5e07400640 1 --2- 172.21.15.96:0/3952838082 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5de406ea20 0x7f5de4070ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:40.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.670+0000 7f5e07400640 1 --2- 172.21.15.96:0/3952838082 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5de406ea20 0x7f5de4070ee0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f5df8008850 tx=0x7f5df8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:40.666 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.673+0000 7f5dffe00640 1 -- 172.21.15.96:0/3952838082 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5e0007bed0 con 0x7f5e0815d270 2024-05-28T06:26:40.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.875+0000 7f5e0d200640 1 -- 172.21.15.96:0/3952838082 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f5e080b3c40 con 0x7f5e0815d270 2024-05-28T06:26:40.986 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.876+0000 7f5dffe00640 1 -- 172.21.15.96:0/3952838082 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v132) ==== 104+0+11 (secure 0 0 0) 0x7f5e0007b7d0 con 0x7f5e0815d270 2024-05-28T06:26:40.987 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 32 2024-05-28T06:26:40.988 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.878+0000 7f5e0d200640 1 -- 172.21.15.96:0/3952838082 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5de406ea20 msgr2=0x7f5de4070ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:40.988 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.878+0000 7f5e0d200640 1 --2- 172.21.15.96:0/3952838082 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5de406ea20 0x7f5de4070ee0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f5df8008850 tx=0x7f5df8008040 comp rx=0 tx=0).stop 2024-05-28T06:26:40.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.878+0000 7f5e0d200640 1 -- 172.21.15.96:0/3952838082 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5e0815d270 msgr2=0x7f5e0815b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:40.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.878+0000 7f5e0d200640 1 --2- 172.21.15.96:0/3952838082 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5e0815d270 0x7f5e0815b970 secure :-1 s=READY pgs=2170 cs=0 l=1 rev1=1 crypto rx=0x7f5e00009e80 tx=0x7f5e00004a40 comp rx=0 tx=0).stop 2024-05-28T06:26:40.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.879+0000 7f5e0d200640 1 -- 172.21.15.96:0/3952838082 shutdown_connections 2024-05-28T06:26:40.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.879+0000 7f5e0d200640 1 --2- 172.21.15.96:0/3952838082 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f5de406ea20 0x7f5de4070ee0 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:40.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.879+0000 7f5e0d200640 1 --2- 172.21.15.96:0/3952838082 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5e0815d270 0x7f5e0815b970 unknown :-1 s=CLOSED pgs=2170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:40.989 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.879+0000 7f5e0d200640 1 --2- 172.21.15.96:0/3952838082 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5e08153510 0x7f5e0815b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:40.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.879+0000 7f5e0d200640 1 --2- 172.21.15.96:0/3952838082 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e080ccb90 0x7f5e08148360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:40.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.879+0000 7f5e0d200640 1 -- 172.21.15.96:0/3952838082 >> 172.21.15.96:0/3952838082 conn(0x7f5e0806b900 msgr2=0x7f5e0815fee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:26:40.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.879+0000 7f5e0d200640 1 -- 172.21.15.96:0/3952838082 shutdown_connections 2024-05-28T06:26:40.990 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:40.879+0000 7f5e0d200640 1 -- 172.21.15.96:0/3952838082 wait complete. 2024-05-28T06:26:40.990 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-28T06:26:41.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.209+0000 7f10e3400640 1 Processor -- start 2024-05-28T06:26:41.203 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.209+0000 7f10e3400640 1 -- start start 2024-05-28T06:26:41.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.211+0000 7f10e3400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f10dc0725e0 0x7f10dc166050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:41.204 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.211+0000 7f10e3400640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10dc0ade90 con 0x7f10dc0d0640 2024-05-28T06:26:41.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.211+0000 7f10e3400640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10dc071e70 con 0x7f10dc15b6f0 2024-05-28T06:26:41.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.211+0000 7f10e3400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f10dc071c40 con 0x7f10dc0725e0 2024-05-28T06:26:41.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.211+0000 7f10e0a00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f10dc0d0640 0x7f10dc0d0a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:44828/0 (socket says 172.21.15.96:44828) 2024-05-28T06:26:41.205 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.211+0000 7f10e0a00640 1 -- 172.21.15.96:0/1174439040 learned_addr learned my addr 172.21.15.96:0/1174439040 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:26:41.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.211+0000 7f10e1400640 1 --2- 172.21.15.96:0/1174439040 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f10dc0725e0 0x7f10dc166050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:41.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.212+0000 7f10db400640 1 -- 172.21.15.96:0/1174439040 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3124934412 0 0) 0x7f10dc0ade90 con 0x7f10dc0d0640 2024-05-28T06:26:41.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.212+0000 7f10db400640 1 -- 172.21.15.96:0/1174439040 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f10c4003520 con 0x7f10dc0d0640 2024-05-28T06:26:41.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.212+0000 7f10db400640 1 -- 172.21.15.96:0/1174439040 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2191540092 0 0) 0x7f10c4003520 con 0x7f10dc0d0640 2024-05-28T06:26:41.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.212+0000 7f10db400640 1 -- 172.21.15.96:0/1174439040 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f10dc0725e0 msgr2=0x7f10dc166050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:41.207 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.212+0000 7f10db400640 1 --2- 172.21.15.96:0/1174439040 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f10dc0725e0 0x7f10dc166050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:41.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.212+0000 7f10db400640 1 -- 172.21.15.96:0/1174439040 >> v1:172.21.15.123:6789/0 conn(0x7f10dc15b6f0 legacy=0x7f10dc06e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:41.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.212+0000 7f10db400640 1 -- 172.21.15.96:0/1174439040 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10dc072130 con 0x7f10dc0d0640 2024-05-28T06:26:41.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.212+0000 7f10db400640 1 -- 172.21.15.96:0/1174439040 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f10d0001670 con 0x7f10dc0d0640 2024-05-28T06:26:41.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.212+0000 7f10db400640 1 -- 172.21.15.96:0/1174439040 >> v1:172.21.15.96:6789/0 conn(0x7f10dc0d0640 legacy=0x7f10dc0d0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:41.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.212+0000 7f10db400640 1 --2- 172.21.15.96:0/1174439040 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f10c40039e0 0x7f10c4003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:41.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.212+0000 7f10db400640 1 --2- 172.21.15.96:0/1174439040 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f10dc0725e0 0x7f10c4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:41.208 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.212+0000 7f10db400640 1 --2- 172.21.15.96:0/1174439040 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f10c4004d40 0x7f10c40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:41.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.212+0000 7f10db400640 1 -- 172.21.15.96:0/1174439040 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f10c400ba70 con 0x7f10c4004d40 2024-05-28T06:26:41.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.212+0000 7f10db400640 1 -- 172.21.15.96:0/1174439040 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f10c400bbf0 con 0x7f10c40039e0 2024-05-28T06:26:41.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.212+0000 7f10db400640 1 -- 172.21.15.96:0/1174439040 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f10c400e000 con 0x7f10dc0725e0 2024-05-28T06:26:41.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.212+0000 7f10e1400640 1 --2- 172.21.15.96:0/1174439040 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f10c4004d40 0x7f10c40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:41.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.213+0000 7f10e0a00640 1 --2- 172.21.15.96:0/1174439040 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f10c40039e0 0x7f10c4003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:41.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.213+0000 7f10dbe00640 1 --2- 172.21.15.96:0/1174439040 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f10dc0725e0 0x7f10c4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:41.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.213+0000 7f10e1400640 1 -- 172.21.15.96:0/1174439040 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f10dc0725e0 msgr2=0x7f10c4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:41.209 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.213+0000 7f10e1400640 1 --2- 172.21.15.96:0/1174439040 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f10dc0725e0 0x7f10c4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:41.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.213+0000 7f10e1400640 1 -- 172.21.15.96:0/1174439040 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f10c40039e0 msgr2=0x7f10c4003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:41.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.213+0000 7f10e1400640 1 --2- 172.21.15.96:0/1174439040 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f10c40039e0 0x7f10c4003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:41.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.213+0000 7f10e1400640 1 -- 172.21.15.96:0/1174439040 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10c4003520 con 0x7f10c4004d40 2024-05-28T06:26:41.210 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.213+0000 7f10dbe00640 1 --2- 172.21.15.96:0/1174439040 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f10dc0725e0 0x7f10c4004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:26:41.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.214+0000 7f10e1400640 1 --2- 172.21.15.96:0/1174439040 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f10c4004d40 0x7f10c40051c0 secure :-1 s=READY pgs=2172 cs=0 l=1 rev1=1 crypto rx=0x7f10d4009f90 tx=0x7f10c4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cc90567cfe17fe5 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:41.213 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.214+0000 7f10db400640 1 -- 172.21.15.96:0/1174439040 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10d4009c10 con 0x7f10c4004d40 2024-05-28T06:26:41.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.214+0000 7f10db400640 1 -- 172.21.15.96:0/1174439040 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10d4036070 con 0x7f10c4004d40 2024-05-28T06:26:41.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.214+0000 7f10db400640 1 -- 172.21.15.96:0/1174439040 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10d4031850 con 0x7f10c4004d40 2024-05-28T06:26:41.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.215+0000 7f10e3400640 1 -- 172.21.15.96:0/1174439040 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f10c4004d40 msgr2=0x7f10c40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:41.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.215+0000 7f10e3400640 1 --2- 172.21.15.96:0/1174439040 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f10c4004d40 0x7f10c40051c0 secure :-1 s=READY pgs=2172 cs=0 l=1 rev1=1 crypto rx=0x7f10d4009f90 tx=0x7f10c4005830 comp rx=0 tx=0).stop 2024-05-28T06:26:41.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.215+0000 7f10e1400640 1 -- 172.21.15.96:0/1174439040 reap_dead start 2024-05-28T06:26:41.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.215+0000 7f10e3400640 1 -- 172.21.15.96:0/1174439040 shutdown_connections 2024-05-28T06:26:41.214 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.215+0000 7f10e3400640 1 -- 172.21.15.96:0/1174439040 >> 172.21.15.96:0/1174439040 conn(0x7f10dc06bd20 msgr2=0x7f10dc06c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:26:41.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.215+0000 7f10e3400640 1 -- 172.21.15.96:0/1174439040 shutdown_connections 2024-05-28T06:26:41.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.215+0000 7f10e3400640 1 -- 172.21.15.96:0/1174439040 wait complete. 2024-05-28T06:26:41.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.217+0000 7f10e3400640 1 Processor -- start 2024-05-28T06:26:41.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.217+0000 7f10e3400640 1 -- start start 2024-05-28T06:26:41.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.218+0000 7f10e3400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f10c4004d40 0x7f10dc1f3b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:41.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.218+0000 7f10e3400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f10dc151a80 0x7f10dc150dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:41.215 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.218+0000 7f10e3400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f10dc151300 0x7f10dc1f7780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:41.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.218+0000 7f10e3400640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f10dc0af7e0 con 0x7f10dc151a80 2024-05-28T06:26:41.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.218+0000 7f10e3400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f10dc0af660 con 0x7f10dc151300 2024-05-28T06:26:41.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.218+0000 7f10e3400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f10dc0af960 con 0x7f10c4004d40 2024-05-28T06:26:41.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.218+0000 7f10dbe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f10dc151a80 0x7f10dc150dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:41.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.218+0000 7f10dbe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f10dc151a80 0x7f10dc150dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35024/0 (socket says 172.21.15.96:35024) 2024-05-28T06:26:41.216 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.218+0000 7f10dbe00640 1 -- 172.21.15.96:0/1372566338 learned_addr learned my addr 172.21.15.96:0/1372566338 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:26:41.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.218+0000 7f10e0a00640 1 --2- 172.21.15.96:0/1372566338 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f10c4004d40 0x7f10dc1f3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:41.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.219+0000 7f10dbe00640 1 -- 172.21.15.96:0/1372566338 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f10c4004d40 msgr2=0x7f10dc1f3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:41.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.219+0000 7f10dbe00640 1 --2- 172.21.15.96:0/1372566338 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f10c4004d40 0x7f10dc1f3b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:41.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.219+0000 7f10dbe00640 1 -- 172.21.15.96:0/1372566338 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f10dc151300 msgr2=0x7f10dc1f7780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:41.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.219+0000 7f10dbe00640 1 --2- 172.21.15.96:0/1372566338 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f10dc151300 0x7f10dc1f7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:41.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.219+0000 7f10dbe00640 1 -- 172.21.15.96:0/1372566338 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10dc1f40b0 con 0x7f10dc151a80 2024-05-28T06:26:41.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.219+0000 7f10dbe00640 1 --2- 172.21.15.96:0/1372566338 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f10dc151a80 0x7f10dc150dc0 secure :-1 s=READY pgs=2173 cs=0 l=1 rev1=1 crypto rx=0x7f10c800d800 tx=0x7f10c800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:41.217 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.219+0000 7f10d8c00640 1 -- 172.21.15.96:0/1372566338 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10c8005070 con 0x7f10dc151a80 2024-05-28T06:26:41.218 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.219+0000 7f10d8c00640 1 -- 172.21.15.96:0/1372566338 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10c8005210 con 0x7f10dc151a80 2024-05-28T06:26:41.218 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.219+0000 7f10d8c00640 1 -- 172.21.15.96:0/1372566338 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10c8007030 con 0x7f10dc151a80 2024-05-28T06:26:41.218 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.219+0000 7f10e3400640 1 -- 172.21.15.96:0/1372566338 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10dc15de60 con 0x7f10dc151a80 2024-05-28T06:26:41.218 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.219+0000 7f10e3400640 1 -- 172.21.15.96:0/1372566338 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10dc15e420 con 0x7f10dc151a80 2024-05-28T06:26:41.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.221+0000 7f10e3400640 1 -- 172.21.15.96:0/1372566338 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10dc151720 con 0x7f10dc151a80 2024-05-28T06:26:41.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.222+0000 7f10d8c00640 1 -- 172.21.15.96:0/1372566338 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 49) ==== 94049+0+0 (secure 0 0 0) 0x7f10c801a020 con 0x7f10dc151a80 2024-05-28T06:26:41.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.222+0000 7f10d8c00640 1 --2- 172.21.15.96:0/1372566338 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f10bc06ea20 0x7f10bc070ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:41.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.223+0000 7f10d8c00640 1 -- 172.21.15.96:0/1372566338 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 5823+0+0 (secure 0 0 0) 0x7f10c808a4b0 con 0x7f10dc151a80 2024-05-28T06:26:41.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.223+0000 7f10e0a00640 1 --2- 172.21.15.96:0/1372566338 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f10bc06ea20 0x7f10bc070ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:41.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.224+0000 7f10e0a00640 1 --2- 172.21.15.96:0/1372566338 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f10bc06ea20 0x7f10bc070ee0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f10d0001cd0 tx=0x7f10d0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:41.220 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.227+0000 7f10d8c00640 1 -- 172.21.15.96:0/1372566338 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f10c805a340 con 0x7f10dc151a80 2024-05-28T06:26:41.402 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.409+0000 7f10e3400640 1 -- 172.21.15.96:0/1372566338 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f10dc0689e0 con 0x7f10dc151a80 2024-05-28T06:26:41.403 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.409+0000 7f10d8c00640 1 -- 172.21.15.96:0/1372566338 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v132) ==== 100+0+11 (secure 0 0 0) 0x7f10c8059c40 con 0x7f10dc151a80 2024-05-28T06:26:41.403 INFO:teuthology.orchestra.run.smithi096.stdout:pg_num: 64 2024-05-28T06:26:41.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.412+0000 7f10e3400640 1 -- 172.21.15.96:0/1372566338 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f10bc06ea20 msgr2=0x7f10bc070ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:41.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.412+0000 7f10e3400640 1 --2- 172.21.15.96:0/1372566338 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f10bc06ea20 0x7f10bc070ee0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f10d0001cd0 tx=0x7f10d0009040 comp rx=0 tx=0).stop 2024-05-28T06:26:41.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.412+0000 7f10e3400640 1 -- 172.21.15.96:0/1372566338 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f10dc151a80 msgr2=0x7f10dc150dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:41.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.412+0000 7f10e3400640 1 --2- 172.21.15.96:0/1372566338 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f10dc151a80 0x7f10dc150dc0 secure :-1 s=READY pgs=2173 cs=0 l=1 rev1=1 crypto rx=0x7f10c800d800 tx=0x7f10c800dcc0 comp rx=0 tx=0).stop 2024-05-28T06:26:41.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.412+0000 7f10e3400640 1 -- 172.21.15.96:0/1372566338 shutdown_connections 2024-05-28T06:26:41.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.412+0000 7f10e3400640 1 --2- 172.21.15.96:0/1372566338 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f10bc06ea20 0x7f10bc070ee0 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:41.406 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.412+0000 7f10e3400640 1 --2- 172.21.15.96:0/1372566338 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f10dc151300 0x7f10dc1f7780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:41.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.412+0000 7f10e3400640 1 --2- 172.21.15.96:0/1372566338 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f10dc151a80 0x7f10dc150dc0 unknown :-1 s=CLOSED pgs=2173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:41.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.412+0000 7f10e3400640 1 --2- 172.21.15.96:0/1372566338 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f10c4004d40 0x7f10dc1f3b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:41.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.412+0000 7f10e3400640 1 -- 172.21.15.96:0/1372566338 >> 172.21.15.96:0/1372566338 conn(0x7f10dc06bd20 msgr2=0x7f10dc072c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:26:41.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.412+0000 7f10e3400640 1 -- 172.21.15.96:0/1372566338 shutdown_connections 2024-05-28T06:26:41.407 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.412+0000 7f10e3400640 1 -- 172.21.15.96:0/1372566338 wait complete. 2024-05-28T06:26:41.424 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-28T06:26:41.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.630+0000 7f5621400640 1 Processor -- start 2024-05-28T06:26:41.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.630+0000 7f5621400640 1 -- start start 2024-05-28T06:26:41.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.632+0000 7f5621400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f561c15e870 0x7f561c0d0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:41.625 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.632+0000 7f5621400640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f561c15ec50 con 0x7f561c0d14d0 2024-05-28T06:26:41.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.632+0000 7f5621400640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f561c0cff80 con 0x7f561c0cfa40 2024-05-28T06:26:41.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.632+0000 7f5621400640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f561c0d2150 con 0x7f561c15e870 2024-05-28T06:26:41.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.632+0000 7f561b400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f561c0d14d0 0x7f561c0cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:44840/0 (socket says 172.21.15.96:44840) 2024-05-28T06:26:41.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.632+0000 7f561b400640 1 -- 172.21.15.96:0/2977328778 learned_addr learned my addr 172.21.15.96:0/2977328778 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:26:41.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.633+0000 7f561be00640 1 --2- 172.21.15.96:0/2977328778 >> v2:172.21.15.123:3300/0 conn(0x7f561c15e870 0x7f561c0d0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:41.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.633+0000 7f561aa00640 1 -- 172.21.15.96:0/2977328778 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4100820473 0 0) 0x7f561c15ec50 con 0x7f561c0d14d0 2024-05-28T06:26:41.631 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.633+0000 7f561aa00640 1 -- 172.21.15.96:0/2977328778 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f55fc003520 con 0x7f561c0d14d0 2024-05-28T06:26:41.631 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.633+0000 7f561aa00640 1 -- 172.21.15.96:0/2977328778 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2812259356 0 0) 0x7f55fc003520 con 0x7f561c0d14d0 2024-05-28T06:26:41.631 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.633+0000 7f561aa00640 1 -- 172.21.15.96:0/2977328778 >> v2:172.21.15.123:3300/0 conn(0x7f561c15e870 msgr2=0x7f561c0d0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:41.631 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.633+0000 7f561aa00640 1 --2- 172.21.15.96:0/2977328778 >> v2:172.21.15.123:3300/0 conn(0x7f561c15e870 0x7f561c0d0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:41.631 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.633+0000 7f561aa00640 1 -- 172.21.15.96:0/2977328778 >> v1:172.21.15.123:6789/0 conn(0x7f561c0cfa40 legacy=0x7f561c0d25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:41.631 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.633+0000 7f561aa00640 1 -- 172.21.15.96:0/2977328778 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f561c0d0260 con 0x7f561c0d14d0 2024-05-28T06:26:41.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.633+0000 7f561aa00640 1 -- 172.21.15.96:0/2977328778 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f5604001b70 con 0x7f561c0d14d0 2024-05-28T06:26:41.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.633+0000 7f561aa00640 1 -- 172.21.15.96:0/2977328778 >> v1:172.21.15.96:6789/0 conn(0x7f561c0d14d0 legacy=0x7f561c0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:41.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.633+0000 7f561aa00640 1 --2- 172.21.15.96:0/2977328778 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f55fc003a00 0x7f55fc003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:41.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.633+0000 7f561aa00640 1 --2- 172.21.15.96:0/2977328778 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f55fc004800 0x7f55fc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:41.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.633+0000 7f561aa00640 1 --2- 172.21.15.96:0/2977328778 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f55fc0051c0 0x7f55fc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:41.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.633+0000 7f561aa00640 1 -- 172.21.15.96:0/2977328778 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f561c0d2150 con 0x7f55fc003a00 2024-05-28T06:26:41.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.633+0000 7f561aa00640 1 -- 172.21.15.96:0/2977328778 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f55fc00c0c0 con 0x7f55fc0051c0 2024-05-28T06:26:41.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.633+0000 7f561aa00640 1 -- 172.21.15.96:0/2977328778 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f55fc00c2c0 con 0x7f55fc004800 2024-05-28T06:26:41.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.634+0000 7f561be00640 1 --2- 172.21.15.96:0/2977328778 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f55fc003a00 0x7f55fc003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:41.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.634+0000 7f5620a00640 1 --2- 172.21.15.96:0/2977328778 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f55fc0051c0 0x7f55fc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:41.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.634+0000 7f561be00640 1 -- 172.21.15.96:0/2977328778 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f55fc004800 msgr2=0x7f55fc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:41.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.634+0000 7f561b400640 1 --2- 172.21.15.96:0/2977328778 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f55fc004800 0x7f55fc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:41.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.634+0000 7f561be00640 1 --2- 172.21.15.96:0/2977328778 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f55fc004800 0x7f55fc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:41.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.634+0000 7f561be00640 1 -- 172.21.15.96:0/2977328778 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f55fc0051c0 msgr2=0x7f55fc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:41.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.634+0000 7f561be00640 1 --2- 172.21.15.96:0/2977328778 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f55fc0051c0 0x7f55fc00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:41.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.634+0000 7f561be00640 1 -- 172.21.15.96:0/2977328778 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55fc003520 con 0x7f55fc003a00 2024-05-28T06:26:41.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.634+0000 7f5620a00640 1 --2- 172.21.15.96:0/2977328778 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f55fc0051c0 0x7f55fc00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:26:41.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.634+0000 7f5620a00640 1 -- 172.21.15.96:0/2977328778 reap_dead start 2024-05-28T06:26:41.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.634+0000 7f561b400640 1 --2- 172.21.15.96:0/2977328778 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f55fc004800 0x7f55fc004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:26:41.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.635+0000 7f561be00640 1 --2- 172.21.15.96:0/2977328778 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f55fc003a00 0x7f55fc003e20 secure :-1 s=READY pgs=2175 cs=0 l=1 rev1=1 crypto rx=0x7f560c009ac0 tx=0x7f55fc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35c870621d735ccb server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:41.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.635+0000 7f561aa00640 1 -- 172.21.15.96:0/2977328778 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f560c036070 con 0x7f55fc003a00 2024-05-28T06:26:41.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.635+0000 7f561aa00640 1 -- 172.21.15.96:0/2977328778 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f560c031400 con 0x7f55fc003a00 2024-05-28T06:26:41.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.635+0000 7f561aa00640 1 -- 172.21.15.96:0/2977328778 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f560c009690 con 0x7f55fc003a00 2024-05-28T06:26:41.635 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.636+0000 7f5621400640 1 -- 172.21.15.96:0/2977328778 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f55fc003a00 msgr2=0x7f55fc003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:41.635 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.636+0000 7f5621400640 1 --2- 172.21.15.96:0/2977328778 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f55fc003a00 0x7f55fc003e20 secure :-1 s=READY pgs=2175 cs=0 l=1 rev1=1 crypto rx=0x7f560c009ac0 tx=0x7f55fc005830 comp rx=0 tx=0).stop 2024-05-28T06:26:41.635 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.636+0000 7f5621400640 1 -- 172.21.15.96:0/2977328778 shutdown_connections 2024-05-28T06:26:41.635 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.636+0000 7f5621400640 1 --2- 172.21.15.96:0/2977328778 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f55fc003a00 0x7f55fc003e20 unknown :-1 s=CLOSED pgs=2175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:41.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.636+0000 7f5621400640 1 -- 172.21.15.96:0/2977328778 >> 172.21.15.96:0/2977328778 conn(0x7f561c08bfc0 msgr2=0x7f561c0d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:26:41.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.636+0000 7f5621400640 1 -- 172.21.15.96:0/2977328778 shutdown_connections 2024-05-28T06:26:41.636 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.636+0000 7f5621400640 1 -- 172.21.15.96:0/2977328778 wait complete. 2024-05-28T06:26:41.638 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.638+0000 7f5621400640 1 Processor -- start 2024-05-28T06:26:41.638 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.638+0000 7f5621400640 1 -- start start 2024-05-28T06:26:41.639 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.639+0000 7f5621400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f55fc003a00 0x7f561c159ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:41.639 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.639+0000 7f5621400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f561c0cb650 0x7f561c06d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:41.639 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.639+0000 7f5621400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f561c0cbad0 0x7f561c06d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:41.639 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.639+0000 7f5621400640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f561c0d4fb0 con 0x7f55fc003a00 2024-05-28T06:26:41.639 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.639+0000 7f5621400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f561c0d0820 con 0x7f561c0cbad0 2024-05-28T06:26:41.639 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.639+0000 7f5621400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f561c0d0f90 con 0x7f561c0cb650 2024-05-28T06:26:41.639 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.639+0000 7f561be00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f55fc003a00 0x7f561c159ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:41.640 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.639+0000 7f561be00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f55fc003a00 0x7f561c159ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35046/0 (socket says 172.21.15.96:35046) 2024-05-28T06:26:41.640 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.639+0000 7f561be00640 1 -- 172.21.15.96:0/1200042366 learned_addr learned my addr 172.21.15.96:0/1200042366 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:26:41.640 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.640+0000 7f561be00640 1 -- 172.21.15.96:0/1200042366 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f561c0cb650 msgr2=0x7f561c06d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:41.640 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.640+0000 7f561be00640 1 --2- 172.21.15.96:0/1200042366 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f561c0cb650 0x7f561c06d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:41.640 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.640+0000 7f561be00640 1 -- 172.21.15.96:0/1200042366 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f561c0cbad0 msgr2=0x7f561c06d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:41.640 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.640+0000 7f561be00640 1 --2- 172.21.15.96:0/1200042366 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f561c0cbad0 0x7f561c06d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:41.640 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.640+0000 7f561be00640 1 -- 172.21.15.96:0/1200042366 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f561c15a020 con 0x7f55fc003a00 2024-05-28T06:26:41.640 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.640+0000 7f561be00640 1 --2- 172.21.15.96:0/1200042366 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f55fc003a00 0x7f561c159ae0 secure :-1 s=READY pgs=2176 cs=0 l=1 rev1=1 crypto rx=0x7f560c009be0 tx=0x7f560c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:41.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.640+0000 7f5613e00640 1 -- 172.21.15.96:0/1200042366 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f560c03f020 con 0x7f55fc003a00 2024-05-28T06:26:41.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.640+0000 7f5613e00640 1 -- 172.21.15.96:0/1200042366 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f560c037040 con 0x7f55fc003a00 2024-05-28T06:26:41.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.640+0000 7f5613e00640 1 -- 172.21.15.96:0/1200042366 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f560c036070 con 0x7f55fc003a00 2024-05-28T06:26:41.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.640+0000 7f5621400640 1 -- 172.21.15.96:0/1200042366 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f561c0c9df0 con 0x7f55fc003a00 2024-05-28T06:26:41.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.641+0000 7f5621400640 1 -- 172.21.15.96:0/1200042366 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f561c15bbf0 con 0x7f55fc003a00 2024-05-28T06:26:41.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.642+0000 7f5621400640 1 -- 172.21.15.96:0/1200042366 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f561c0689e0 con 0x7f55fc003a00 2024-05-28T06:26:41.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.643+0000 7f5613e00640 1 -- 172.21.15.96:0/1200042366 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 49) ==== 94049+0+0 (secure 0 0 0) 0x7f560c042020 con 0x7f55fc003a00 2024-05-28T06:26:41.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.644+0000 7f5613e00640 1 --2- 172.21.15.96:0/1200042366 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f55f806ea50 0x7f55f8070f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:41.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.644+0000 7f5613e00640 1 -- 172.21.15.96:0/1200042366 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 5823+0+0 (secure 0 0 0) 0x7f560c0aac90 con 0x7f55fc003a00 2024-05-28T06:26:41.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.645+0000 7f561b400640 1 --2- 172.21.15.96:0/1200042366 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f55f806ea50 0x7f55f8070f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:41.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.646+0000 7f561b400640 1 --2- 172.21.15.96:0/1200042366 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f55f806ea50 0x7f55f8070f10 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f5604001ce0 tx=0x7f5604009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:41.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.649+0000 7f5613e00640 1 -- 172.21.15.96:0/1200042366 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f560c049030 con 0x7f55fc003a00 2024-05-28T06:26:41.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.843+0000 7f5621400640 1 -- 172.21.15.96:0/1200042366 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f561c0ba500 con 0x7f55fc003a00 2024-05-28T06:26:41.837 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.844+0000 7f5613e00640 1 -- 172.21.15.96:0/1200042366 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v132) ==== 80+0+0 (secure 0 0 0) 0x7f561c15a020 con 0x7f55fc003a00 2024-05-28T06:26:41.838 INFO:teuthology.orchestra.run.smithi096.stderr:listed 0 entries 2024-05-28T06:26:41.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.846+0000 7f5621400640 1 -- 172.21.15.96:0/1200042366 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f55f806ea50 msgr2=0x7f55f8070f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:41.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.846+0000 7f5621400640 1 --2- 172.21.15.96:0/1200042366 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f55f806ea50 0x7f55f8070f10 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f5604001ce0 tx=0x7f5604009040 comp rx=0 tx=0).stop 2024-05-28T06:26:41.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.846+0000 7f5621400640 1 -- 172.21.15.96:0/1200042366 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f55fc003a00 msgr2=0x7f561c159ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:41.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.846+0000 7f5621400640 1 --2- 172.21.15.96:0/1200042366 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f55fc003a00 0x7f561c159ae0 secure :-1 s=READY pgs=2176 cs=0 l=1 rev1=1 crypto rx=0x7f560c009be0 tx=0x7f560c004730 comp rx=0 tx=0).stop 2024-05-28T06:26:41.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.847+0000 7f5621400640 1 -- 172.21.15.96:0/1200042366 shutdown_connections 2024-05-28T06:26:41.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.847+0000 7f5621400640 1 --2- 172.21.15.96:0/1200042366 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f55f806ea50 0x7f55f8070f10 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:41.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.847+0000 7f5621400640 1 --2- 172.21.15.96:0/1200042366 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f561c0cbad0 0x7f561c06d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:41.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.847+0000 7f5621400640 1 --2- 172.21.15.96:0/1200042366 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f561c0cb650 0x7f561c06d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:41.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.847+0000 7f5621400640 1 --2- 172.21.15.96:0/1200042366 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f55fc003a00 0x7f561c159ae0 unknown :-1 s=CLOSED pgs=2176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:41.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.847+0000 7f5621400640 1 -- 172.21.15.96:0/1200042366 >> 172.21.15.96:0/1200042366 conn(0x7f561c08bfc0 msgr2=0x7f561c0b10f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:26:41.841 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.847+0000 7f5621400640 1 -- 172.21.15.96:0/1200042366 shutdown_connections 2024-05-28T06:26:41.842 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:41.847+0000 7f5621400640 1 -- 172.21.15.96:0/1200042366 wait complete. 2024-05-28T06:26:41.859 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi096.front.sepia.ceph.com... 2024-05-28T06:26:41.860 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:26:41.860 DEBUG:teuthology.orchestra.run.smithi096:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-05-28T06:26:41.927 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi096.stderr:ceph-fuse[83889]: fuse finished with error 0 and tester_r 0 2024-05-28T06:26:42.487 INFO:teuthology.orchestra.run:waiting for 300 2024-05-28T06:26:42.488 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi096.front.sepia.ceph.com 2024-05-28T06:26:42.488 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:26:42.488 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-28T06:26:42.535 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-05-28T06:26:42.535 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:26:42.582 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052731,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248996k,nr_inodes=812249,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-28T06:26:42.583 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-05-28T06:26:42.583 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:26:42.639 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052731,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248996k,nr_inodes=812249,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-05-28T06:26:42.639 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-05-28T06:26:42.639 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-28T06:26:42.689 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052731,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248996k,nr_inodes=812249,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-05-28T06:26:42.689 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:26:42.689 DEBUG:teuthology.orchestra.run.smithi123:> ip netns list 2024-05-28T06:26:42.787 INFO:teuthology.orchestra.run.smithi123.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-05-28T06:26:42.787 INFO:teuthology.orchestra.run.smithi123.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-05-28T06:26:42.788 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:26:42.789 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2024-05-28T06:26:42.877 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:26:42.877 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2024-05-28T06:26:42.948 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:26:42.948 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip link delete ceph-brx 2024-05-28T06:26:43.098 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:26:43.098 DEBUG:teuthology.orchestra.run.smithi096:> ip netns list 2024-05-28T06:26:43.157 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-05-28T06:26:43.158 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-05-28T06:26:43.162 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:26:43.162 DEBUG:teuthology.orchestra.run.smithi096:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-28T06:26:43.252 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:26:43.252 DEBUG:teuthology.orchestra.run.smithi096:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-28T06:26:43.323 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-28T06:26:43.323 DEBUG:teuthology.orchestra.run.smithi096:> sudo ip link delete ceph-brx 2024-05-28T06:26:43.451 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-05-28T06:26:43.462 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-05-28T06:26:43.463 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-28T06:26:43.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.647+0000 7ff187200640 1 Processor -- start 2024-05-28T06:26:43.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.647+0000 7ff187200640 1 -- start start 2024-05-28T06:26:43.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.648+0000 7ff187200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff188158b80 0x7ff18806dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:43.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.648+0000 7ff187200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff18806e6e0 con 0x7ff1880722d0 2024-05-28T06:26:43.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.648+0000 7ff187200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff18806e9c0 con 0x7ff18806e160 2024-05-28T06:26:43.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.648+0000 7ff187200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ff18806eca0 con 0x7ff188158b80 2024-05-28T06:26:43.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.649+0000 7ff186800640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7ff1880722d0 0x7ff1880b32c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:44846/0 (socket says 172.21.15.96:44846) 2024-05-28T06:26:43.646 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.649+0000 7ff186800640 1 -- 172.21.15.96:0/3962815118 learned_addr learned my addr 172.21.15.96:0/3962815118 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:26:43.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.649+0000 7ff185e00640 1 --2- 172.21.15.96:0/3962815118 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff188158b80 0x7ff18806dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:43.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.649+0000 7ff184a00640 1 -- 172.21.15.96:0/3962815118 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3064439879 0 0) 0x7ff18806e6e0 con 0x7ff1880722d0 2024-05-28T06:26:43.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.649+0000 7ff184a00640 1 -- 172.21.15.96:0/3962815118 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff16c003520 con 0x7ff1880722d0 2024-05-28T06:26:43.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.649+0000 7ff184a00640 1 -- 172.21.15.96:0/3962815118 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2898619410 0 0) 0x7ff16c003520 con 0x7ff1880722d0 2024-05-28T06:26:43.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.649+0000 7ff184a00640 1 -- 172.21.15.96:0/3962815118 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff188158b80 msgr2=0x7ff18806dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:43.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.649+0000 7ff184a00640 1 --2- 172.21.15.96:0/3962815118 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff188158b80 0x7ff18806dc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:43.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.649+0000 7ff184a00640 1 -- 172.21.15.96:0/3962815118 >> v1:172.21.15.123:6789/0 conn(0x7ff18806e160 legacy=0x7ff188071bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:43.647 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.649+0000 7ff184a00640 1 -- 172.21.15.96:0/3962815118 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff18806ef80 con 0x7ff1880722d0 2024-05-28T06:26:43.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.649+0000 7ff184a00640 1 -- 172.21.15.96:0/3962815118 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7ff1800019f0 con 0x7ff1880722d0 2024-05-28T06:26:43.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.649+0000 7ff184a00640 1 -- 172.21.15.96:0/3962815118 >> v1:172.21.15.96:6789/0 conn(0x7ff1880722d0 legacy=0x7ff1880b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:43.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.650+0000 7ff184a00640 1 --2- 172.21.15.96:0/3962815118 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff16c0039e0 0x7ff16c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:43.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.650+0000 7ff184a00640 1 --2- 172.21.15.96:0/3962815118 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff188158b80 0x7ff16c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:43.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.650+0000 7ff184a00640 1 --2- 172.21.15.96:0/3962815118 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff16c004d40 0x7ff16c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:43.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.650+0000 7ff184a00640 1 -- 172.21.15.96:0/3962815118 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff18806eca0 con 0x7ff16c004d40 2024-05-28T06:26:43.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.650+0000 7ff184a00640 1 -- 172.21.15.96:0/3962815118 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff16c00baa0 con 0x7ff16c0039e0 2024-05-28T06:26:43.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.650+0000 7ff184a00640 1 -- 172.21.15.96:0/3962815118 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ff16c00bca0 con 0x7ff188158b80 2024-05-28T06:26:43.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.650+0000 7ff186800640 1 --2- 172.21.15.96:0/3962815118 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff16c004d40 0x7ff16c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:43.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.650+0000 7ff185400640 1 --2- 172.21.15.96:0/3962815118 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff188158b80 0x7ff16c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:43.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.650+0000 7ff185e00640 1 --2- 172.21.15.96:0/3962815118 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff16c0039e0 0x7ff16c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:43.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.650+0000 7ff186800640 1 -- 172.21.15.96:0/3962815118 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff188158b80 msgr2=0x7ff16c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:43.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.650+0000 7ff186800640 1 --2- 172.21.15.96:0/3962815118 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff188158b80 0x7ff16c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:43.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.650+0000 7ff186800640 1 -- 172.21.15.96:0/3962815118 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff16c0039e0 msgr2=0x7ff16c003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:43.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.650+0000 7ff186800640 1 --2- 172.21.15.96:0/3962815118 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff16c0039e0 0x7ff16c003e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:43.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.650+0000 7ff186800640 1 -- 172.21.15.96:0/3962815118 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff16c003520 con 0x7ff16c004d40 2024-05-28T06:26:43.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.651+0000 7ff186800640 1 --2- 172.21.15.96:0/3962815118 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff16c004d40 0x7ff16c0051c0 secure :-1 s=READY pgs=2178 cs=0 l=1 rev1=1 crypto rx=0x7ff188162150 tx=0x7ff16c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b55ec9a56e98c192 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:43.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.651+0000 7ff184a00640 1 -- 172.21.15.96:0/3962815118 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff18000a3b0 con 0x7ff16c004d40 2024-05-28T06:26:43.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.652+0000 7ff184a00640 1 -- 172.21.15.96:0/3962815118 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff18002f070 con 0x7ff16c004d40 2024-05-28T06:26:43.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.652+0000 7ff184a00640 1 -- 172.21.15.96:0/3962815118 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff18002bba0 con 0x7ff16c004d40 2024-05-28T06:26:43.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.652+0000 7ff187200640 1 -- 172.21.15.96:0/3962815118 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff16c004d40 msgr2=0x7ff16c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:43.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.652+0000 7ff187200640 1 --2- 172.21.15.96:0/3962815118 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff16c004d40 0x7ff16c0051c0 secure :-1 s=READY pgs=2178 cs=0 l=1 rev1=1 crypto rx=0x7ff188162150 tx=0x7ff16c005830 comp rx=0 tx=0).stop 2024-05-28T06:26:43.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.652+0000 7ff186800640 1 -- 172.21.15.96:0/3962815118 reap_dead start 2024-05-28T06:26:43.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.652+0000 7ff187200640 1 -- 172.21.15.96:0/3962815118 shutdown_connections 2024-05-28T06:26:43.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.652+0000 7ff187200640 1 -- 172.21.15.96:0/3962815118 >> 172.21.15.96:0/3962815118 conn(0x7ff1880b1fc0 msgr2=0x7ff1880b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:26:43.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.652+0000 7ff187200640 1 -- 172.21.15.96:0/3962815118 shutdown_connections 2024-05-28T06:26:43.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.652+0000 7ff187200640 1 -- 172.21.15.96:0/3962815118 wait complete. 2024-05-28T06:26:43.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.654+0000 7ff187200640 1 Processor -- start 2024-05-28T06:26:43.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.654+0000 7ff187200640 1 -- start start 2024-05-28T06:26:43.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.655+0000 7ff187200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff16c004d40 0x7ff18815f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:43.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.655+0000 7ff187200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff1881427f0 0x7ff1881f41f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:43.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.655+0000 7ff187200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff1881f4730 0x7ff188151890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:43.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.655+0000 7ff187200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff18806f780 con 0x7ff16c004d40 2024-05-28T06:26:43.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.655+0000 7ff187200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff1880b2d30 con 0x7ff1881427f0 2024-05-28T06:26:43.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.655+0000 7ff187200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7ff1881f4bb0 con 0x7ff1881f4730 2024-05-28T06:26:43.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.655+0000 7ff185e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff16c004d40 0x7ff18815f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:43.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.655+0000 7ff185e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff16c004d40 0x7ff18815f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35064/0 (socket says 172.21.15.96:35064) 2024-05-28T06:26:43.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.655+0000 7ff185e00640 1 -- 172.21.15.96:0/416048497 learned_addr learned my addr 172.21.15.96:0/416048497 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:26:43.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.656+0000 7ff185e00640 1 -- 172.21.15.96:0/416048497 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff1881f4730 msgr2=0x7ff188151890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:43.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.656+0000 7ff185e00640 1 --2- 172.21.15.96:0/416048497 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff1881f4730 0x7ff188151890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:43.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.656+0000 7ff185e00640 1 -- 172.21.15.96:0/416048497 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff1881427f0 msgr2=0x7ff1881f41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:43.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.656+0000 7ff185e00640 1 --2- 172.21.15.96:0/416048497 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff1881427f0 0x7ff1881f41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:43.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.656+0000 7ff185e00640 1 -- 172.21.15.96:0/416048497 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff188160b20 con 0x7ff16c004d40 2024-05-28T06:26:43.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.656+0000 7ff185e00640 1 --2- 172.21.15.96:0/416048497 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff16c004d40 0x7ff18815f750 secure :-1 s=READY pgs=2179 cs=0 l=1 rev1=1 crypto rx=0x7ff178010860 tx=0x7ff178010d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:43.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.656+0000 7ff17e000640 1 -- 172.21.15.96:0/416048497 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff17800f040 con 0x7ff16c004d40 2024-05-28T06:26:43.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.656+0000 7ff17e000640 1 -- 172.21.15.96:0/416048497 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff17800ea80 con 0x7ff16c004d40 2024-05-28T06:26:43.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.656+0000 7ff17e000640 1 -- 172.21.15.96:0/416048497 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff17800ee50 con 0x7ff16c004d40 2024-05-28T06:26:43.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.656+0000 7ff187200640 1 -- 172.21.15.96:0/416048497 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1881f4dd0 con 0x7ff16c004d40 2024-05-28T06:26:43.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.656+0000 7ff187200640 1 -- 172.21.15.96:0/416048497 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff18815ec70 con 0x7ff16c004d40 2024-05-28T06:26:43.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.657+0000 7ff187200640 1 -- 172.21.15.96:0/416048497 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1880689e0 con 0x7ff16c004d40 2024-05-28T06:26:43.657 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.658+0000 7ff17e000640 1 -- 172.21.15.96:0/416048497 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 49) ==== 94049+0+0 (secure 0 0 0) 0x7ff17801c400 con 0x7ff16c004d40 2024-05-28T06:26:43.657 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.660+0000 7ff17e000640 1 --2- 172.21.15.96:0/416048497 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff16406ea50 0x7ff164070f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:43.657 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.660+0000 7ff17e000640 1 -- 172.21.15.96:0/416048497 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 5823+0+0 (secure 0 0 0) 0x7ff17808b130 con 0x7ff16c004d40 2024-05-28T06:26:43.657 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.661+0000 7ff185400640 1 --2- 172.21.15.96:0/416048497 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff16406ea50 0x7ff164070f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:43.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.662+0000 7ff185400640 1 --2- 172.21.15.96:0/416048497 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff16406ea50 0x7ff164070f10 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7ff170007fd0 tx=0x7ff170009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:43.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.664+0000 7ff17e000640 1 -- 172.21.15.96:0/416048497 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff178018030 con 0x7ff16c004d40 2024-05-28T06:26:43.855 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.861+0000 7ff187200640 1 -- 172.21.15.96:0/416048497 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff188160ff0 con 0x7ff16406ea50 2024-05-28T06:26:43.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.883+0000 7ff17e000640 1 -- 172.21.15.96:0/416048497 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+254734 (secure 0 0 0) 0x7ff188160ff0 con 0x7ff16406ea50 2024-05-28T06:26:43.880 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:26:43.881 INFO:teuthology.orchestra.run.smithi096.stderr:dumped all 2024-05-28T06:26:43.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.889+0000 7ff187200640 1 -- 172.21.15.96:0/416048497 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff16406ea50 msgr2=0x7ff164070f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:43.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.889+0000 7ff187200640 1 --2- 172.21.15.96:0/416048497 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff16406ea50 0x7ff164070f10 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7ff170007fd0 tx=0x7ff170009040 comp rx=0 tx=0).stop 2024-05-28T06:26:43.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.889+0000 7ff187200640 1 -- 172.21.15.96:0/416048497 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff16c004d40 msgr2=0x7ff18815f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:43.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.889+0000 7ff187200640 1 --2- 172.21.15.96:0/416048497 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff16c004d40 0x7ff18815f750 secure :-1 s=READY pgs=2179 cs=0 l=1 rev1=1 crypto rx=0x7ff178010860 tx=0x7ff178010d20 comp rx=0 tx=0).stop 2024-05-28T06:26:43.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.889+0000 7ff187200640 1 -- 172.21.15.96:0/416048497 shutdown_connections 2024-05-28T06:26:43.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.889+0000 7ff187200640 1 --2- 172.21.15.96:0/416048497 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7ff16406ea50 0x7ff164070f10 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:43.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.889+0000 7ff187200640 1 --2- 172.21.15.96:0/416048497 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff1881f4730 0x7ff188151890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:43.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.889+0000 7ff187200640 1 --2- 172.21.15.96:0/416048497 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff1881427f0 0x7ff1881f41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:43.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.889+0000 7ff187200640 1 --2- 172.21.15.96:0/416048497 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff16c004d40 0x7ff18815f750 unknown :-1 s=CLOSED pgs=2179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:43.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.889+0000 7ff187200640 1 -- 172.21.15.96:0/416048497 >> 172.21.15.96:0/416048497 conn(0x7ff1880b1fc0 msgr2=0x7ff1880aebc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:26:43.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.889+0000 7ff187200640 1 -- 172.21.15.96:0/416048497 shutdown_connections 2024-05-28T06:26:43.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:43.889+0000 7ff187200640 1 -- 172.21.15.96:0/416048497 wait complete. 2024-05-28T06:26:43.904 INFO:teuthology.orchestra.run.smithi096.stdout:{"pg_ready":true,"pg_map":{"version":806,"stamp":"2024-05-28T06:26:42.080648+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":18398487910,"num_objects":4420,"num_object_clones":1,"num_object_copies":8840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4420,"num_whiteouts":1,"num_read":4092,"num_read_kb":15909056,"num_write":5706,"num_write_kb":18960626,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":41943040,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5619,"ondisk_log_size":5619,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":37058584,"kb_used_data":35947544,"kb_used_omap":12,"kb_used_meta":1110963,"kb_avail":717916136,"statfs":{"total":773094113280,"available":735146123264,"internally_reserved":0,"allocated":36810285056,"data_stored":36802793164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":1137626672},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005007"},"pg_stats":[{"pgid":"19.3f","version":"132'72","reported_seq":159,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.319580+0000","last_change":"2024-05-28T06:24:24.007430+0000","last_active":"2024-05-28T06:26:18.319580+0000","last_peered":"2024-05-28T06:26:18.319580+0000","last_clean":"2024-05-28T06:26:18.319580+0000","last_became_active":"2024-05-28T06:24:24.006988+0000","last_became_peered":"2024-05-28T06:24:24.006988+0000","last_unstale":"2024-05-28T06:26:18.319580+0000","last_undegraded":"2024-05-28T06:26:18.319580+0000","last_fullsized":"2024-05-28T06:26:18.319580+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:45:33.172226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260046848,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":70,"num_read_kb":286720,"num_write":72,"num_write_kb":274432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3e","version":"132'84","reported_seq":169,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.291618+0000","last_change":"2024-05-28T06:24:23.986742+0000","last_active":"2024-05-28T06:26:18.291618+0000","last_peered":"2024-05-28T06:26:18.291618+0000","last_clean":"2024-05-28T06:26:18.291618+0000","last_became_active":"2024-05-28T06:24:23.986471+0000","last_became_peered":"2024-05-28T06:24:23.986471+0000","last_unstale":"2024-05-28T06:26:18.291618+0000","last_undegraded":"2024-05-28T06:26:18.291618+0000","last_fullsized":"2024-05-28T06:26:18.291618+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:02:16.956326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327155712,"num_objects":78,"num_object_clones":0,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78,"num_whiteouts":0,"num_read":68,"num_read_kb":278528,"num_write":84,"num_write_kb":331776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3d","version":"132'93","reported_seq":180,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.325236+0000","last_change":"2024-05-28T06:24:24.007698+0000","last_active":"2024-05-28T06:26:18.325236+0000","last_peered":"2024-05-28T06:26:18.325236+0000","last_clean":"2024-05-28T06:26:18.325236+0000","last_became_active":"2024-05-28T06:24:24.007416+0000","last_became_peered":"2024-05-28T06:24:24.007416+0000","last_unstale":"2024-05-28T06:26:18.325236+0000","last_undegraded":"2024-05-28T06:26:18.325236+0000","last_fullsized":"2024-05-28T06:26:18.325236+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:24:31.676324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306184192,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":70,"num_read_kb":253960,"num_write":93,"num_write_kb":323588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.3c","version":"132'71","reported_seq":133,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.305549+0000","last_change":"2024-05-28T06:24:24.011512+0000","last_active":"2024-05-28T06:26:18.305549+0000","last_peered":"2024-05-28T06:26:18.305549+0000","last_clean":"2024-05-28T06:26:18.305549+0000","last_became_active":"2024-05-28T06:24:24.011234+0000","last_became_peered":"2024-05-28T06:24:24.011234+0000","last_unstale":"2024-05-28T06:26:18.305549+0000","last_undegraded":"2024-05-28T06:26:18.305549+0000","last_fullsized":"2024-05-28T06:26:18.305549+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:59:05.689554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281018368,"num_objects":67,"num_object_clones":0,"num_object_copies":134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":0,"num_read":45,"num_read_kb":176130,"num_write":71,"num_write_kb":278529,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3b","version":"132'65","reported_seq":119,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.288952+0000","last_change":"2024-05-28T06:24:24.019397+0000","last_active":"2024-05-28T06:26:18.288952+0000","last_peered":"2024-05-28T06:26:18.288952+0000","last_clean":"2024-05-28T06:26:18.288952+0000","last_became_active":"2024-05-28T06:24:24.018334+0000","last_became_peered":"2024-05-28T06:24:24.018334+0000","last_unstale":"2024-05-28T06:26:18.288952+0000","last_undegraded":"2024-05-28T06:26:18.288952+0000","last_fullsized":"2024-05-28T06:26:18.288952+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:22:34.611665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":230686720,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":38,"num_read_kb":155648,"num_write":65,"num_write_kb":245760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.3a","version":"132'97","reported_seq":216,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.278772+0000","last_change":"2024-05-28T06:24:55.231438+0000","last_active":"2024-05-28T06:26:18.278772+0000","last_peered":"2024-05-28T06:26:18.278772+0000","last_clean":"2024-05-28T06:26:18.278772+0000","last_became_active":"2024-05-28T06:24:54.990013+0000","last_became_peered":"2024-05-28T06:24:54.990013+0000","last_unstale":"2024-05-28T06:26:18.278772+0000","last_undegraded":"2024-05-28T06:26:18.278772+0000","last_fullsized":"2024-05-28T06:26:18.278772+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:50:40.872653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":360710405,"num_objects":87,"num_object_clones":0,"num_object_copies":174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87,"num_whiteouts":0,"num_read":84,"num_read_kb":339969,"num_write":97,"num_write_kb":368642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":25165824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.39","version":"132'65","reported_seq":126,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.259648+0000","last_change":"2024-05-28T06:24:24.006117+0000","last_active":"2024-05-28T06:26:18.259648+0000","last_peered":"2024-05-28T06:26:18.259648+0000","last_clean":"2024-05-28T06:26:18.259648+0000","last_became_active":"2024-05-28T06:24:24.005839+0000","last_became_peered":"2024-05-28T06:24:24.005839+0000","last_unstale":"2024-05-28T06:26:18.259648+0000","last_undegraded":"2024-05-28T06:26:18.259648+0000","last_fullsized":"2024-05-28T06:26:18.259648+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:47:32.965627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222298112,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":45,"num_read_kb":184320,"num_write":65,"num_write_kb":241664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.38","version":"132'79","reported_seq":142,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.263031+0000","last_change":"2024-05-28T06:24:24.008772+0000","last_active":"2024-05-28T06:26:18.263031+0000","last_peered":"2024-05-28T06:26:18.263031+0000","last_clean":"2024-05-28T06:26:18.263031+0000","last_became_active":"2024-05-28T06:24:24.008334+0000","last_became_peered":"2024-05-28T06:24:24.008334+0000","last_unstale":"2024-05-28T06:26:18.263031+0000","last_undegraded":"2024-05-28T06:26:18.263031+0000","last_fullsized":"2024-05-28T06:26:18.263031+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:10:50.477572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":297795584,"num_objects":71,"num_object_clones":0,"num_object_copies":142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":47,"num_read_kb":192512,"num_write":79,"num_write_kb":307200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.37","version":"132'80","reported_seq":149,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.239272+0000","last_change":"2024-05-28T06:24:24.018229+0000","last_active":"2024-05-28T06:26:18.239272+0000","last_peered":"2024-05-28T06:26:18.239272+0000","last_clean":"2024-05-28T06:26:18.239272+0000","last_became_active":"2024-05-28T06:24:24.017929+0000","last_became_peered":"2024-05-28T06:24:24.017929+0000","last_unstale":"2024-05-28T06:26:18.239272+0000","last_undegraded":"2024-05-28T06:26:18.239272+0000","last_fullsized":"2024-05-28T06:26:18.239272+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:58:16.573705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":297795716,"num_objects":72,"num_object_clones":0,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72,"num_whiteouts":0,"num_read":53,"num_read_kb":204803,"num_write":80,"num_write_kb":303106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.36","version":"132'63","reported_seq":137,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:25:42.637629+0000","last_change":"2024-05-28T06:24:23.995403+0000","last_active":"2024-05-28T06:25:42.637629+0000","last_peered":"2024-05-28T06:25:42.637629+0000","last_clean":"2024-05-28T06:25:42.637629+0000","last_became_active":"2024-05-28T06:24:23.994945+0000","last_became_peered":"2024-05-28T06:24:23.994945+0000","last_unstale":"2024-05-28T06:25:42.637629+0000","last_undegraded":"2024-05-28T06:25:42.637629+0000","last_fullsized":"2024-05-28T06:25:42.637629+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:11:29.809986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":255852544,"num_objects":61,"num_object_clones":0,"num_object_copies":122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":0,"num_read":58,"num_read_kb":237568,"num_write":63,"num_write_kb":253952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.35","version":"132'83","reported_seq":194,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.260079+0000","last_change":"2024-05-28T06:24:23.992865+0000","last_active":"2024-05-28T06:26:18.260079+0000","last_peered":"2024-05-28T06:26:18.260079+0000","last_clean":"2024-05-28T06:26:18.260079+0000","last_became_active":"2024-05-28T06:24:23.991405+0000","last_became_peered":"2024-05-28T06:24:23.991405+0000","last_unstale":"2024-05-28T06:26:18.260079+0000","last_undegraded":"2024-05-28T06:26:18.260079+0000","last_fullsized":"2024-05-28T06:26:18.260079+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:06:47.846653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":322961408,"num_objects":77,"num_object_clones":0,"num_object_copies":154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77,"num_whiteouts":0,"num_read":95,"num_read_kb":389120,"num_write":83,"num_write_kb":327680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.34","version":"132'77","reported_seq":185,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.283958+0000","last_change":"2024-05-28T06:24:24.018385+0000","last_active":"2024-05-28T06:26:18.283958+0000","last_peered":"2024-05-28T06:26:18.283958+0000","last_clean":"2024-05-28T06:26:18.283958+0000","last_became_active":"2024-05-28T06:24:24.017931+0000","last_became_peered":"2024-05-28T06:24:24.017931+0000","last_unstale":"2024-05-28T06:26:18.283958+0000","last_undegraded":"2024-05-28T06:26:18.283958+0000","last_fullsized":"2024-05-28T06:26:18.283958+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:11:21.508556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":289406976,"num_objects":69,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":0,"num_read":92,"num_read_kb":360452,"num_write":77,"num_write_kb":294913,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.33","version":"132'75","reported_seq":171,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.229026+0000","last_change":"2024-05-28T06:24:24.007399+0000","last_active":"2024-05-28T06:26:18.229026+0000","last_peered":"2024-05-28T06:26:18.229026+0000","last_clean":"2024-05-28T06:26:18.229026+0000","last_became_active":"2024-05-28T06:24:24.007121+0000","last_became_peered":"2024-05-28T06:24:24.007121+0000","last_unstale":"2024-05-28T06:26:18.229026+0000","last_undegraded":"2024-05-28T06:26:18.229026+0000","last_fullsized":"2024-05-28T06:26:18.229026+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:29:33.118245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":289406976,"num_objects":69,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":0,"num_read":80,"num_read_kb":327680,"num_write":75,"num_write_kb":294912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.32","version":"132'77","reported_seq":170,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.301921+0000","last_change":"2024-05-28T06:24:55.363255+0000","last_active":"2024-05-28T06:26:18.301921+0000","last_peered":"2024-05-28T06:26:18.301921+0000","last_clean":"2024-05-28T06:26:18.301921+0000","last_became_active":"2024-05-28T06:24:54.998097+0000","last_became_peered":"2024-05-28T06:24:54.998097+0000","last_unstale":"2024-05-28T06:26:18.301921+0000","last_undegraded":"2024-05-28T06:26:18.301921+0000","last_fullsized":"2024-05-28T06:26:18.301921+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:41:33.057222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":289406976,"num_objects":69,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":0,"num_read":62,"num_read_kb":253952,"num_write":77,"num_write_kb":299008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":16777216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.31","version":"132'70","reported_seq":116,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.292733+0000","last_change":"2024-05-28T06:24:24.020598+0000","last_active":"2024-05-28T06:26:18.292733+0000","last_peered":"2024-05-28T06:26:18.292733+0000","last_clean":"2024-05-28T06:26:18.292733+0000","last_became_active":"2024-05-28T06:24:24.020301+0000","last_became_peered":"2024-05-28T06:24:24.020301+0000","last_unstale":"2024-05-28T06:26:18.292733+0000","last_undegraded":"2024-05-28T06:26:18.292733+0000","last_fullsized":"2024-05-28T06:26:18.292733+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:18:18.796231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251658240,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":30,"num_read_kb":122880,"num_write":70,"num_write_kb":266240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.30","version":"132'81","reported_seq":148,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.299313+0000","last_change":"2024-05-28T06:24:23.995628+0000","last_active":"2024-05-28T06:26:18.299313+0000","last_peered":"2024-05-28T06:26:18.299313+0000","last_clean":"2024-05-28T06:26:18.299313+0000","last_became_active":"2024-05-28T06:24:23.995367+0000","last_became_peered":"2024-05-28T06:24:23.995367+0000","last_unstale":"2024-05-28T06:26:18.299313+0000","last_undegraded":"2024-05-28T06:26:18.299313+0000","last_fullsized":"2024-05-28T06:26:18.299313+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:07:49.435286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306184192,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":51,"num_read_kb":208896,"num_write":81,"num_write_kb":315392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2f","version":"132'65","reported_seq":123,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.267214+0000","last_change":"2024-05-28T06:24:23.987587+0000","last_active":"2024-05-28T06:26:18.267214+0000","last_peered":"2024-05-28T06:26:18.267214+0000","last_clean":"2024-05-28T06:26:18.267214+0000","last_became_active":"2024-05-28T06:24:23.986777+0000","last_became_peered":"2024-05-28T06:24:23.986777+0000","last_unstale":"2024-05-28T06:26:18.267214+0000","last_undegraded":"2024-05-28T06:26:18.267214+0000","last_fullsized":"2024-05-28T06:26:18.267214+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:06:00.030718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":230686720,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":42,"num_read_kb":172032,"num_write":65,"num_write_kb":245760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.2e","version":"132'82","reported_seq":185,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.245601+0000","last_change":"2024-05-28T06:24:23.998500+0000","last_active":"2024-05-28T06:26:18.245601+0000","last_peered":"2024-05-28T06:26:18.245601+0000","last_clean":"2024-05-28T06:26:18.245601+0000","last_became_active":"2024-05-28T06:24:23.998105+0000","last_became_peered":"2024-05-28T06:24:23.998105+0000","last_unstale":"2024-05-28T06:26:18.245601+0000","last_undegraded":"2024-05-28T06:26:18.245601+0000","last_fullsized":"2024-05-28T06:26:18.245601+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:47:31.291023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":310378496,"num_objects":74,"num_object_clones":0,"num_object_copies":148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74,"num_whiteouts":0,"num_read":87,"num_read_kb":356352,"num_write":82,"num_write_kb":315393,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2d","version":"132'83","reported_seq":167,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.323316+0000","last_change":"2024-05-28T06:24:23.997899+0000","last_active":"2024-05-28T06:26:18.323316+0000","last_peered":"2024-05-28T06:26:18.323316+0000","last_clean":"2024-05-28T06:26:18.323316+0000","last_became_active":"2024-05-28T06:24:23.997362+0000","last_became_peered":"2024-05-28T06:24:23.997362+0000","last_unstale":"2024-05-28T06:26:18.323316+0000","last_undegraded":"2024-05-28T06:26:18.323316+0000","last_fullsized":"2024-05-28T06:26:18.323316+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:48:26.250684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":310378496,"num_objects":74,"num_object_clones":0,"num_object_copies":148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74,"num_whiteouts":0,"num_read":67,"num_read_kb":274432,"num_write":82,"num_write_kb":319488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2c","version":"132'70","reported_seq":144,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.318303+0000","last_change":"2024-05-28T06:24:23.996440+0000","last_active":"2024-05-28T06:26:18.318303+0000","last_peered":"2024-05-28T06:26:18.318303+0000","last_clean":"2024-05-28T06:26:18.318303+0000","last_became_active":"2024-05-28T06:24:23.996067+0000","last_became_peered":"2024-05-28T06:24:23.996067+0000","last_unstale":"2024-05-28T06:26:18.318303+0000","last_undegraded":"2024-05-28T06:26:18.318303+0000","last_fullsized":"2024-05-28T06:26:18.318303+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:06:00.030718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":272629760,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":57,"num_read_kb":233472,"num_write":69,"num_write_kb":274432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2b","version":"132'82","reported_seq":190,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.301761+0000","last_change":"2024-05-28T06:24:24.018537+0000","last_active":"2024-05-28T06:26:18.301761+0000","last_peered":"2024-05-28T06:26:18.301761+0000","last_clean":"2024-05-28T06:26:18.301761+0000","last_became_active":"2024-05-28T06:24:24.018253+0000","last_became_peered":"2024-05-28T06:24:24.018253+0000","last_unstale":"2024-05-28T06:26:18.301761+0000","last_undegraded":"2024-05-28T06:26:18.301761+0000","last_fullsized":"2024-05-28T06:26:18.301761+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:13:05.023858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":293601280,"num_objects":70,"num_object_clones":0,"num_object_copies":140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70,"num_whiteouts":0,"num_read":91,"num_read_kb":364546,"num_write":82,"num_write_kb":303106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2a","version":"132'75","reported_seq":147,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.271529+0000","last_change":"2024-05-28T06:24:23.994663+0000","last_active":"2024-05-28T06:26:18.271529+0000","last_peered":"2024-05-28T06:26:18.271529+0000","last_clean":"2024-05-28T06:26:18.271529+0000","last_became_active":"2024-05-28T06:24:23.994410+0000","last_became_peered":"2024-05-28T06:24:23.994410+0000","last_unstale":"2024-05-28T06:26:18.271529+0000","last_undegraded":"2024-05-28T06:26:18.271529+0000","last_fullsized":"2024-05-28T06:26:18.271529+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:05:58.537398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":285212672,"num_objects":68,"num_object_clones":0,"num_object_copies":136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68,"num_whiteouts":0,"num_read":55,"num_read_kb":217090,"num_write":74,"num_write_kb":286721,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.29","version":"132'81","reported_seq":149,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.292813+0000","last_change":"2024-05-28T06:24:24.015438+0000","last_active":"2024-05-28T06:26:18.292813+0000","last_peered":"2024-05-28T06:26:18.292813+0000","last_clean":"2024-05-28T06:26:18.292813+0000","last_became_active":"2024-05-28T06:24:24.015124+0000","last_became_peered":"2024-05-28T06:24:24.015124+0000","last_unstale":"2024-05-28T06:26:18.292813+0000","last_undegraded":"2024-05-28T06:26:18.292813+0000","last_fullsized":"2024-05-28T06:26:18.292813+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:40:05.545358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281018368,"num_objects":67,"num_object_clones":0,"num_object_copies":134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":0,"num_read":51,"num_read_kb":208896,"num_write":81,"num_write_kb":303104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.28","version":"132'75","reported_seq":143,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.270972+0000","last_change":"2024-05-28T06:24:24.017147+0000","last_active":"2024-05-28T06:26:18.270972+0000","last_peered":"2024-05-28T06:26:18.270972+0000","last_clean":"2024-05-28T06:26:18.270972+0000","last_became_active":"2024-05-28T06:24:24.016692+0000","last_became_peered":"2024-05-28T06:24:24.016692+0000","last_unstale":"2024-05-28T06:26:18.270972+0000","last_undegraded":"2024-05-28T06:26:18.270972+0000","last_fullsized":"2024-05-28T06:26:18.270972+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:10:49.969260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281018368,"num_objects":67,"num_object_clones":0,"num_object_copies":134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":0,"num_read":51,"num_read_kb":208896,"num_write":75,"num_write_kb":286721,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.27","version":"132'81","reported_seq":161,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.270839+0000","last_change":"2024-05-28T06:24:24.007398+0000","last_active":"2024-05-28T06:26:18.270839+0000","last_peered":"2024-05-28T06:26:18.270839+0000","last_clean":"2024-05-28T06:26:18.270839+0000","last_became_active":"2024-05-28T06:24:24.006983+0000","last_became_peered":"2024-05-28T06:24:24.006983+0000","last_unstale":"2024-05-28T06:26:18.270839+0000","last_undegraded":"2024-05-28T06:26:18.270839+0000","last_fullsized":"2024-05-28T06:26:18.270839+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:02:59.158681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306184192,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":63,"num_read_kb":249858,"num_write":81,"num_write_kb":307202,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.26","version":"132'83","reported_seq":162,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.291454+0000","last_change":"2024-05-28T06:24:23.999225+0000","last_active":"2024-05-28T06:26:18.291454+0000","last_peered":"2024-05-28T06:26:18.291454+0000","last_clean":"2024-05-28T06:26:18.291454+0000","last_became_active":"2024-05-28T06:24:23.996859+0000","last_became_peered":"2024-05-28T06:24:23.996859+0000","last_unstale":"2024-05-28T06:26:18.291454+0000","last_undegraded":"2024-05-28T06:26:18.291454+0000","last_fullsized":"2024-05-28T06:26:18.291454+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:05:09.592919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":310378496,"num_objects":74,"num_object_clones":0,"num_object_copies":148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74,"num_whiteouts":0,"num_read":62,"num_read_kb":253952,"num_write":82,"num_write_kb":319488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.25","version":"132'73","reported_seq":156,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.305983+0000","last_change":"2024-05-28T06:24:24.007463+0000","last_active":"2024-05-28T06:26:18.305983+0000","last_peered":"2024-05-28T06:26:18.305983+0000","last_clean":"2024-05-28T06:26:18.305983+0000","last_became_active":"2024-05-28T06:24:24.007194+0000","last_became_peered":"2024-05-28T06:24:24.007194+0000","last_unstale":"2024-05-28T06:26:18.305983+0000","last_undegraded":"2024-05-28T06:26:18.305983+0000","last_fullsized":"2024-05-28T06:26:18.305983+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:27:47.069088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":272629760,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":66,"num_read_kb":258051,"num_write":73,"num_write_kb":278529,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.24","version":"132'77","reported_seq":128,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.306695+0000","last_change":"2024-05-28T06:24:24.016291+0000","last_active":"2024-05-28T06:26:18.306695+0000","last_peered":"2024-05-28T06:26:18.306695+0000","last_clean":"2024-05-28T06:26:18.306695+0000","last_became_active":"2024-05-28T06:24:24.015990+0000","last_became_peered":"2024-05-28T06:24:24.015990+0000","last_unstale":"2024-05-28T06:26:18.306695+0000","last_undegraded":"2024-05-28T06:26:18.306695+0000","last_fullsized":"2024-05-28T06:26:18.306695+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:02:16.956326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281018368,"num_objects":67,"num_object_clones":0,"num_object_copies":134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":0,"num_read":34,"num_read_kb":131074,"num_write":77,"num_write_kb":290817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.23","version":"132'81","reported_seq":169,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.308583+0000","last_change":"2024-05-28T06:24:24.019908+0000","last_active":"2024-05-28T06:26:18.308583+0000","last_peered":"2024-05-28T06:26:18.308583+0000","last_clean":"2024-05-28T06:26:18.308583+0000","last_became_active":"2024-05-28T06:24:24.019497+0000","last_became_peered":"2024-05-28T06:24:24.019497+0000","last_unstale":"2024-05-28T06:26:18.308583+0000","last_undegraded":"2024-05-28T06:26:18.308583+0000","last_fullsized":"2024-05-28T06:26:18.308583+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:16:02.402760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":297795584,"num_objects":71,"num_object_clones":0,"num_object_copies":142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":71,"num_read_kb":274436,"num_write":81,"num_write_kb":307201,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.22","version":"132'85","reported_seq":164,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.308511+0000","last_change":"2024-05-28T06:24:24.019852+0000","last_active":"2024-05-28T06:26:18.308511+0000","last_peered":"2024-05-28T06:26:18.308511+0000","last_clean":"2024-05-28T06:26:18.308511+0000","last_became_active":"2024-05-28T06:24:24.019462+0000","last_became_peered":"2024-05-28T06:24:24.019462+0000","last_unstale":"2024-05-28T06:26:18.308511+0000","last_undegraded":"2024-05-28T06:26:18.308511+0000","last_fullsized":"2024-05-28T06:26:18.308511+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:52:19.161622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":314572800,"num_objects":75,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75,"num_whiteouts":0,"num_read":62,"num_read_kb":253952,"num_write":85,"num_write_kb":327680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.21","version":"132'66","reported_seq":172,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:25:43.030729+0000","last_change":"2024-05-28T06:24:24.011632+0000","last_active":"2024-05-28T06:25:43.030729+0000","last_peered":"2024-05-28T06:25:43.030729+0000","last_clean":"2024-05-28T06:25:43.030729+0000","last_became_active":"2024-05-28T06:24:24.011246+0000","last_became_peered":"2024-05-28T06:24:24.011246+0000","last_unstale":"2024-05-28T06:25:43.030729+0000","last_undegraded":"2024-05-28T06:25:43.030729+0000","last_fullsized":"2024-05-28T06:25:43.030729+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:05:09.592919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":276824064,"num_objects":66,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":89,"num_read_kb":364544,"num_write":66,"num_write_kb":270336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.20","version":"132'75","reported_seq":163,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.316590+0000","last_change":"2024-05-28T06:24:23.991803+0000","last_active":"2024-05-28T06:26:18.316590+0000","last_peered":"2024-05-28T06:26:18.316590+0000","last_clean":"2024-05-28T06:26:18.316590+0000","last_became_active":"2024-05-28T06:24:23.991245+0000","last_became_peered":"2024-05-28T06:24:23.991245+0000","last_unstale":"2024-05-28T06:26:18.316590+0000","last_undegraded":"2024-05-28T06:26:18.316590+0000","last_fullsized":"2024-05-28T06:26:18.316590+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:55:45.102561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":272629760,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":71,"num_read_kb":282626,"num_write":75,"num_write_kb":282625,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1e","version":"132'68","reported_seq":127,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.309071+0000","last_change":"2024-05-28T06:24:24.018032+0000","last_active":"2024-05-28T06:26:18.309071+0000","last_peered":"2024-05-28T06:26:18.309071+0000","last_clean":"2024-05-28T06:26:18.309071+0000","last_became_active":"2024-05-28T06:24:24.017727+0000","last_became_peered":"2024-05-28T06:24:24.017727+0000","last_unstale":"2024-05-28T06:26:18.309071+0000","last_undegraded":"2024-05-28T06:26:18.309071+0000","last_fullsized":"2024-05-28T06:26:18.309071+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:13:05.023858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260046848,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":42,"num_read_kb":172032,"num_write":68,"num_write_kb":266240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1f","version":"129'4","reported_seq":23,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:55.134481+0000","last_change":"2024-05-28T06:24:23.006418+0000","last_active":"2024-05-28T06:24:55.134481+0000","last_peered":"2024-05-28T06:24:55.134481+0000","last_clean":"2024-05-28T06:24:55.134481+0000","last_became_active":"2024-05-28T06:24:23.006043+0000","last_became_peered":"2024-05-28T06:24:23.006043+0000","last_unstale":"2024-05-28T06:24:55.134481+0000","last_undegraded":"2024-05-28T06:24:55.134481+0000","last_fullsized":"2024-05-28T06:24:55.134481+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:31:42.560242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.1f","version":"132'82","reported_seq":167,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.285593+0000","last_change":"2024-05-28T06:24:24.008420+0000","last_active":"2024-05-28T06:26:18.285593+0000","last_peered":"2024-05-28T06:26:18.285593+0000","last_clean":"2024-05-28T06:26:18.285593+0000","last_became_active":"2024-05-28T06:24:24.008078+0000","last_became_peered":"2024-05-28T06:24:24.008078+0000","last_unstale":"2024-05-28T06:26:18.285593+0000","last_undegraded":"2024-05-28T06:26:18.285593+0000","last_fullsized":"2024-05-28T06:26:18.285593+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:08:27.007107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":293601280,"num_objects":70,"num_object_clones":0,"num_object_copies":140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70,"num_whiteouts":0,"num_read":68,"num_read_kb":262148,"num_write":82,"num_write_kb":303106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1e","version":"129'1","reported_seq":20,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:55.454913+0000","last_change":"2024-05-28T06:24:22.995399+0000","last_active":"2024-05-28T06:24:55.454913+0000","last_peered":"2024-05-28T06:24:55.454913+0000","last_clean":"2024-05-28T06:24:55.454913+0000","last_became_active":"2024-05-28T06:24:22.994896+0000","last_became_peered":"2024-05-28T06:24:22.994896+0000","last_unstale":"2024-05-28T06:24:55.454913+0000","last_undegraded":"2024-05-28T06:24:55.454913+0000","last_fullsized":"2024-05-28T06:24:55.454913+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:06:41.660486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1c","version":"132'70","reported_seq":150,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.265732+0000","last_change":"2024-05-28T06:24:24.006543+0000","last_active":"2024-05-28T06:26:18.265732+0000","last_peered":"2024-05-28T06:26:18.265732+0000","last_clean":"2024-05-28T06:26:18.265732+0000","last_became_active":"2024-05-28T06:24:24.006258+0000","last_became_peered":"2024-05-28T06:24:24.006258+0000","last_unstale":"2024-05-28T06:26:18.265732+0000","last_undegraded":"2024-05-28T06:26:18.265732+0000","last_fullsized":"2024-05-28T06:26:18.265732+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:34:58.279352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":255852805,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":64,"num_read_kb":262144,"num_write":70,"num_write_kb":266241,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1d","version":"132'3","reported_seq":21,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:54.985140+0000","last_change":"2024-05-28T06:24:22.992163+0000","last_active":"2024-05-28T06:24:54.985140+0000","last_peered":"2024-05-28T06:24:54.985140+0000","last_clean":"2024-05-28T06:24:54.985140+0000","last_became_active":"2024-05-28T06:24:22.991407+0000","last_became_peered":"2024-05-28T06:24:22.991407+0000","last_unstale":"2024-05-28T06:24:54.985140+0000","last_undegraded":"2024-05-28T06:24:54.985140+0000","last_fullsized":"2024-05-28T06:24:54.985140+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:34:43.911764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1d","version":"132'89","reported_seq":192,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.314145+0000","last_change":"2024-05-28T06:24:24.013028+0000","last_active":"2024-05-28T06:26:18.314145+0000","last_peered":"2024-05-28T06:26:18.314145+0000","last_clean":"2024-05-28T06:26:18.314145+0000","last_became_active":"2024-05-28T06:24:24.010356+0000","last_became_peered":"2024-05-28T06:24:24.010356+0000","last_unstale":"2024-05-28T06:26:18.314145+0000","last_undegraded":"2024-05-28T06:26:18.314145+0000","last_fullsized":"2024-05-28T06:26:18.314145+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:46:38.879485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":297795584,"num_objects":71,"num_object_clones":0,"num_object_copies":142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":87,"num_read_kb":356352,"num_write":89,"num_write_kb":327680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1c","version":"129'2","reported_seq":20,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:54.994094+0000","last_change":"2024-05-28T06:24:22.989493+0000","last_active":"2024-05-28T06:24:54.994094+0000","last_peered":"2024-05-28T06:24:54.994094+0000","last_clean":"2024-05-28T06:24:54.994094+0000","last_became_active":"2024-05-28T06:24:22.989103+0000","last_became_peered":"2024-05-28T06:24:22.989103+0000","last_unstale":"2024-05-28T06:24:54.994094+0000","last_undegraded":"2024-05-28T06:24:54.994094+0000","last_fullsized":"2024-05-28T06:24:54.994094+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:28:25.855171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1a","version":"132'76","reported_seq":145,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.241565+0000","last_change":"2024-05-28T06:24:24.008297+0000","last_active":"2024-05-28T06:26:18.241565+0000","last_peered":"2024-05-28T06:26:18.241565+0000","last_clean":"2024-05-28T06:26:18.241565+0000","last_became_active":"2024-05-28T06:24:24.007987+0000","last_became_peered":"2024-05-28T06:24:24.007987+0000","last_unstale":"2024-05-28T06:26:18.241565+0000","last_undegraded":"2024-05-28T06:26:18.241565+0000","last_fullsized":"2024-05-28T06:26:18.241565+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:48:49.925554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301989888,"num_objects":72,"num_object_clones":0,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72,"num_whiteouts":0,"num_read":53,"num_read_kb":217088,"num_write":76,"num_write_kb":303104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1b","version":"129'1","reported_seq":19,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:54.995380+0000","last_change":"2024-05-28T06:24:22.987047+0000","last_active":"2024-05-28T06:24:54.995380+0000","last_peered":"2024-05-28T06:24:54.995380+0000","last_clean":"2024-05-28T06:24:54.995380+0000","last_became_active":"2024-05-28T06:24:22.985617+0000","last_became_peered":"2024-05-28T06:24:22.985617+0000","last_unstale":"2024-05-28T06:24:54.995380+0000","last_undegraded":"2024-05-28T06:24:54.995380+0000","last_fullsized":"2024-05-28T06:24:54.995380+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:59:04.673636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1b","version":"132'77","reported_seq":182,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.245202+0000","last_change":"2024-05-28T06:24:23.987152+0000","last_active":"2024-05-28T06:26:18.245202+0000","last_peered":"2024-05-28T06:26:18.245202+0000","last_clean":"2024-05-28T06:26:18.245202+0000","last_became_active":"2024-05-28T06:24:23.985828+0000","last_became_peered":"2024-05-28T06:24:23.985828+0000","last_unstale":"2024-05-28T06:26:18.245202+0000","last_undegraded":"2024-05-28T06:26:18.245202+0000","last_fullsized":"2024-05-28T06:26:18.245202+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:48:49.925554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306184192,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":89,"num_read_kb":364544,"num_write":77,"num_write_kb":307200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1a","version":"129'1","reported_seq":19,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:55.341429+0000","last_change":"2024-05-28T06:24:22.990085+0000","last_active":"2024-05-28T06:24:55.341429+0000","last_peered":"2024-05-28T06:24:55.341429+0000","last_clean":"2024-05-28T06:24:55.341429+0000","last_became_active":"2024-05-28T06:24:22.989801+0000","last_became_peered":"2024-05-28T06:24:22.989801+0000","last_unstale":"2024-05-28T06:24:55.341429+0000","last_undegraded":"2024-05-28T06:24:55.341429+0000","last_fullsized":"2024-05-28T06:24:55.341429+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:53:06.353639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.18","version":"132'72","reported_seq":155,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.299919+0000","last_change":"2024-05-28T06:24:24.003806+0000","last_active":"2024-05-28T06:26:18.299919+0000","last_peered":"2024-05-28T06:26:18.299919+0000","last_clean":"2024-05-28T06:26:18.299919+0000","last_became_active":"2024-05-28T06:24:24.003446+0000","last_became_peered":"2024-05-28T06:24:24.003446+0000","last_unstale":"2024-05-28T06:26:18.299919+0000","last_undegraded":"2024-05-28T06:26:18.299919+0000","last_fullsized":"2024-05-28T06:26:18.299919+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:05:52.238604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260046848,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":67,"num_read_kb":274432,"num_write":72,"num_write_kb":274432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":18,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:54.984156+0000","last_change":"2024-05-28T06:24:23.007861+0000","last_active":"2024-05-28T06:24:54.984156+0000","last_peered":"2024-05-28T06:24:54.984156+0000","last_clean":"2024-05-28T06:24:54.984156+0000","last_became_active":"2024-05-28T06:24:23.007494+0000","last_became_peered":"2024-05-28T06:24:23.007494+0000","last_unstale":"2024-05-28T06:24:54.984156+0000","last_undegraded":"2024-05-28T06:24:54.984156+0000","last_fullsized":"2024-05-28T06:24:54.984156+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:06:19.878458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.19","version":"132'72","reported_seq":150,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.195341+0000","last_change":"2024-05-28T06:24:23.993309+0000","last_active":"2024-05-28T06:26:18.195341+0000","last_peered":"2024-05-28T06:26:18.195341+0000","last_clean":"2024-05-28T06:26:18.195341+0000","last_became_active":"2024-05-28T06:24:23.987466+0000","last_became_peered":"2024-05-28T06:24:23.987466+0000","last_unstale":"2024-05-28T06:26:18.195341+0000","last_undegraded":"2024-05-28T06:26:18.195341+0000","last_fullsized":"2024-05-28T06:26:18.195341+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:07:16.445323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":276824064,"num_objects":66,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":62,"num_read_kb":245762,"num_write":72,"num_write_kb":278529,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":18,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:54.994422+0000","last_change":"2024-05-28T06:24:23.000407+0000","last_active":"2024-05-28T06:24:54.994422+0000","last_peered":"2024-05-28T06:24:54.994422+0000","last_clean":"2024-05-28T06:24:54.994422+0000","last_became_active":"2024-05-28T06:24:23.000037+0000","last_became_peered":"2024-05-28T06:24:23.000037+0000","last_unstale":"2024-05-28T06:24:54.994422+0000","last_undegraded":"2024-05-28T06:24:54.994422+0000","last_fullsized":"2024-05-28T06:24:54.994422+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:59:08.824871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"132'71","reported_seq":120,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.295941+0000","last_change":"2024-05-28T06:24:23.995548+0000","last_active":"2024-05-28T06:26:18.295941+0000","last_peered":"2024-05-28T06:26:18.295941+0000","last_clean":"2024-05-28T06:26:18.295941+0000","last_became_active":"2024-05-28T06:24:23.995204+0000","last_became_peered":"2024-05-28T06:24:23.995204+0000","last_unstale":"2024-05-28T06:26:18.295941+0000","last_undegraded":"2024-05-28T06:26:18.295941+0000","last_fullsized":"2024-05-28T06:26:18.295941+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:25:36.261157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260046848,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":33,"num_read_kb":135168,"num_write":70,"num_write_kb":270336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.17","version":"132'1","reported_seq":19,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:55.454932+0000","last_change":"2024-05-28T06:24:23.001114+0000","last_active":"2024-05-28T06:24:55.454932+0000","last_peered":"2024-05-28T06:24:55.454932+0000","last_clean":"2024-05-28T06:24:55.454932+0000","last_became_active":"2024-05-28T06:24:22.999720+0000","last_became_peered":"2024-05-28T06:24:22.999720+0000","last_unstale":"2024-05-28T06:24:55.454932+0000","last_undegraded":"2024-05-28T06:24:55.454932+0000","last_fullsized":"2024-05-28T06:24:55.454932+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:29:54.261773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.17","version":"132'63","reported_seq":117,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.234194+0000","last_change":"2024-05-28T06:24:24.016902+0000","last_active":"2024-05-28T06:26:18.234194+0000","last_peered":"2024-05-28T06:26:18.234194+0000","last_clean":"2024-05-28T06:26:18.234194+0000","last_became_active":"2024-05-28T06:24:24.016597+0000","last_became_peered":"2024-05-28T06:24:24.016597+0000","last_unstale":"2024-05-28T06:26:18.234194+0000","last_undegraded":"2024-05-28T06:26:18.234194+0000","last_fullsized":"2024-05-28T06:26:18.234194+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:08:42.734914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247463936,"num_objects":59,"num_object_clones":0,"num_object_copies":118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":0,"num_read":38,"num_read_kb":155648,"num_write":63,"num_write_kb":249856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":18,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:55.342025+0000","last_change":"2024-05-28T06:24:23.001188+0000","last_active":"2024-05-28T06:24:55.342025+0000","last_peered":"2024-05-28T06:24:55.342025+0000","last_clean":"2024-05-28T06:24:55.342025+0000","last_became_active":"2024-05-28T06:24:23.000878+0000","last_became_peered":"2024-05-28T06:24:23.000878+0000","last_unstale":"2024-05-28T06:24:55.342025+0000","last_undegraded":"2024-05-28T06:24:55.342025+0000","last_fullsized":"2024-05-28T06:24:55.342025+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:18:06.253798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.14","version":"132'75","reported_seq":166,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.313193+0000","last_change":"2024-05-28T06:24:24.011311+0000","last_active":"2024-05-28T06:26:18.313193+0000","last_peered":"2024-05-28T06:26:18.313193+0000","last_clean":"2024-05-28T06:26:18.313193+0000","last_became_active":"2024-05-28T06:24:24.010110+0000","last_became_peered":"2024-05-28T06:24:24.010110+0000","last_unstale":"2024-05-28T06:26:18.313193+0000","last_undegraded":"2024-05-28T06:26:18.313193+0000","last_fullsized":"2024-05-28T06:26:18.313193+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:46:31.105891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251658240,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":75,"num_read_kb":307200,"num_write":74,"num_write_kb":274432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.15","version":"132'24","reported_seq":42,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.184203+0000","last_change":"2024-05-28T06:24:22.999088+0000","last_active":"2024-05-28T06:26:18.184203+0000","last_peered":"2024-05-28T06:26:18.184203+0000","last_clean":"2024-05-28T06:26:18.184203+0000","last_became_active":"2024-05-28T06:24:22.998379+0000","last_became_peered":"2024-05-28T06:24:22.998379+0000","last_unstale":"2024-05-28T06:26:18.184203+0000","last_undegraded":"2024-05-28T06:26:18.184203+0000","last_fullsized":"2024-05-28T06:26:18.184203+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:59:19.637147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.15","version":"132'68","reported_seq":144,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.268436+0000","last_change":"2024-05-28T06:24:24.020278+0000","last_active":"2024-05-28T06:26:18.268436+0000","last_peered":"2024-05-28T06:26:18.268436+0000","last_clean":"2024-05-28T06:26:18.268436+0000","last_became_active":"2024-05-28T06:24:24.019973+0000","last_became_peered":"2024-05-28T06:24:24.019973+0000","last_unstale":"2024-05-28T06:26:18.268436+0000","last_undegraded":"2024-05-28T06:26:18.268436+0000","last_fullsized":"2024-05-28T06:26:18.268436+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:46:47.603731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":230686720,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":60,"num_read_kb":229380,"num_write":67,"num_write_kb":241666,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.14","version":"132'416","reported_seq":434,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:23.552552+0000","last_change":"2024-05-28T06:24:23.005145+0000","last_active":"2024-05-28T06:26:23.552552+0000","last_peered":"2024-05-28T06:26:23.552552+0000","last_clean":"2024-05-28T06:26:23.552552+0000","last_became_active":"2024-05-28T06:24:23.004831+0000","last_became_peered":"2024-05-28T06:24:23.004831+0000","last_unstale":"2024-05-28T06:26:23.552552+0000","last_undegraded":"2024-05-28T06:26:23.552552+0000","last_fullsized":"2024-05-28T06:26:23.552552+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":416,"log_dups_size":0,"ondisk_log_size":416,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:09:55.668305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1673237,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":415,"num_write_kb":1791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.13","version":"132'75","reported_seq":141,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:25:42.876820+0000","last_change":"2024-05-28T06:24:24.002061+0000","last_active":"2024-05-28T06:25:42.876820+0000","last_peered":"2024-05-28T06:25:42.876820+0000","last_clean":"2024-05-28T06:25:42.876820+0000","last_became_active":"2024-05-28T06:24:24.001429+0000","last_became_peered":"2024-05-28T06:24:24.001429+0000","last_unstale":"2024-05-28T06:25:42.876820+0000","last_undegraded":"2024-05-28T06:25:42.876820+0000","last_fullsized":"2024-05-28T06:25:42.876820+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:39:41.026878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":297795584,"num_objects":71,"num_object_clones":0,"num_object_copies":142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":50,"num_read_kb":196610,"num_write":75,"num_write_kb":294913,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":18,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:54.992958+0000","last_change":"2024-05-28T06:24:22.993624+0000","last_active":"2024-05-28T06:24:54.992958+0000","last_peered":"2024-05-28T06:24:54.992958+0000","last_clean":"2024-05-28T06:24:54.992958+0000","last_became_active":"2024-05-28T06:24:22.993301+0000","last_became_peered":"2024-05-28T06:24:22.993301+0000","last_unstale":"2024-05-28T06:24:54.992958+0000","last_undegraded":"2024-05-28T06:24:54.992958+0000","last_fullsized":"2024-05-28T06:24:54.992958+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:26:44.582201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.10","version":"132'69","reported_seq":142,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.196154+0000","last_change":"2024-05-28T06:24:24.005361+0000","last_active":"2024-05-28T06:26:18.196154+0000","last_peered":"2024-05-28T06:26:18.196154+0000","last_clean":"2024-05-28T06:26:18.196154+0000","last_became_active":"2024-05-28T06:24:24.005080+0000","last_became_peered":"2024-05-28T06:24:24.005080+0000","last_unstale":"2024-05-28T06:26:18.196154+0000","last_undegraded":"2024-05-28T06:26:18.196154+0000","last_fullsized":"2024-05-28T06:26:18.196154+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:09:03.314508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":272629760,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":57,"num_read_kb":233472,"num_write":69,"num_write_kb":274432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":18,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:54.994900+0000","last_change":"2024-05-28T06:24:23.005565+0000","last_active":"2024-05-28T06:24:54.994900+0000","last_peered":"2024-05-28T06:24:54.994900+0000","last_clean":"2024-05-28T06:24:54.994900+0000","last_became_active":"2024-05-28T06:24:23.005194+0000","last_became_peered":"2024-05-28T06:24:23.005194+0000","last_unstale":"2024-05-28T06:24:54.994900+0000","last_undegraded":"2024-05-28T06:24:54.994900+0000","last_fullsized":"2024-05-28T06:24:54.994900+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:26:55.784217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.11","version":"132'84","reported_seq":162,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.224702+0000","last_change":"2024-05-28T06:24:24.009724+0000","last_active":"2024-05-28T06:26:18.224702+0000","last_peered":"2024-05-28T06:26:18.224702+0000","last_clean":"2024-05-28T06:26:18.224702+0000","last_became_active":"2024-05-28T06:24:24.009163+0000","last_became_peered":"2024-05-28T06:24:24.009163+0000","last_unstale":"2024-05-28T06:26:18.224702+0000","last_undegraded":"2024-05-28T06:26:18.224702+0000","last_fullsized":"2024-05-28T06:26:18.224702+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:36:46.619746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":318767104,"num_objects":76,"num_object_clones":0,"num_object_copies":152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76,"num_whiteouts":0,"num_read":62,"num_read_kb":253952,"num_write":84,"num_write_kb":323585,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.10","version":"129'8","reported_seq":26,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:55.070167+0000","last_change":"2024-05-28T06:24:22.996927+0000","last_active":"2024-05-28T06:24:55.070167+0000","last_peered":"2024-05-28T06:24:55.070167+0000","last_clean":"2024-05-28T06:24:55.070167+0000","last_became_active":"2024-05-28T06:24:22.996644+0000","last_became_peered":"2024-05-28T06:24:22.996644+0000","last_unstale":"2024-05-28T06:24:55.070167+0000","last_undegraded":"2024-05-28T06:24:55.070167+0000","last_fullsized":"2024-05-28T06:24:55.070167+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:17:26.759155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.e","version":"132'58","reported_seq":98,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.315898+0000","last_change":"2024-05-28T06:24:24.009329+0000","last_active":"2024-05-28T06:26:18.315898+0000","last_peered":"2024-05-28T06:26:18.315898+0000","last_clean":"2024-05-28T06:26:18.315898+0000","last_became_active":"2024-05-28T06:24:24.009033+0000","last_became_peered":"2024-05-28T06:24:24.009033+0000","last_unstale":"2024-05-28T06:26:18.315898+0000","last_undegraded":"2024-05-28T06:26:18.315898+0000","last_fullsized":"2024-05-28T06:26:18.315898+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:23:27.401090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218103808,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":23,"num_read_kb":94208,"num_write":58,"num_write_kb":225280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.f","version":"129'1","reported_seq":20,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:55.456823+0000","last_change":"2024-05-28T06:24:23.006567+0000","last_active":"2024-05-28T06:24:55.456823+0000","last_peered":"2024-05-28T06:24:55.456823+0000","last_clean":"2024-05-28T06:24:55.456823+0000","last_became_active":"2024-05-28T06:24:23.005650+0000","last_became_peered":"2024-05-28T06:24:23.005650+0000","last_unstale":"2024-05-28T06:24:55.456823+0000","last_undegraded":"2024-05-28T06:24:55.456823+0000","last_fullsized":"2024-05-28T06:24:55.456823+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:01:42.283187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.f","version":"132'83","reported_seq":174,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.285828+0000","last_change":"2024-05-28T06:24:24.016763+0000","last_active":"2024-05-28T06:26:18.285828+0000","last_peered":"2024-05-28T06:26:18.285828+0000","last_clean":"2024-05-28T06:26:18.285828+0000","last_became_active":"2024-05-28T06:24:24.016255+0000","last_became_peered":"2024-05-28T06:24:24.016255+0000","last_unstale":"2024-05-28T06:26:18.285828+0000","last_undegraded":"2024-05-28T06:26:18.285828+0000","last_fullsized":"2024-05-28T06:26:18.285828+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:06:19.643800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306184192,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":74,"num_read_kb":303104,"num_write":83,"num_write_kb":315393,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.e","version":"129'2","reported_seq":21,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:54.985300+0000","last_change":"2024-05-28T06:24:23.002513+0000","last_active":"2024-05-28T06:24:54.985300+0000","last_peered":"2024-05-28T06:24:54.985300+0000","last_clean":"2024-05-28T06:24:54.985300+0000","last_became_active":"2024-05-28T06:24:23.001946+0000","last_became_peered":"2024-05-28T06:24:23.001946+0000","last_unstale":"2024-05-28T06:24:54.985300+0000","last_undegraded":"2024-05-28T06:24:54.985300+0000","last_fullsized":"2024-05-28T06:24:54.985300+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:27:48.979163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"132'66","reported_seq":143,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.288093+0000","last_change":"2024-05-28T06:24:23.998032+0000","last_active":"2024-05-28T06:26:18.288093+0000","last_peered":"2024-05-28T06:26:18.288093+0000","last_clean":"2024-05-28T06:26:18.288093+0000","last_became_active":"2024-05-28T06:24:23.996433+0000","last_became_peered":"2024-05-28T06:24:23.996433+0000","last_unstale":"2024-05-28T06:26:18.288093+0000","last_undegraded":"2024-05-28T06:26:18.288093+0000","last_fullsized":"2024-05-28T06:26:18.288093+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:47:25.671388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":230686900,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":60,"num_read_kb":221190,"num_write":66,"num_write_kb":233476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.d","version":"129'1","reported_seq":20,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:54.984968+0000","last_change":"2024-05-28T06:24:22.992776+0000","last_active":"2024-05-28T06:24:54.984968+0000","last_peered":"2024-05-28T06:24:54.984968+0000","last_clean":"2024-05-28T06:24:54.984968+0000","last_became_active":"2024-05-28T06:24:22.992147+0000","last_became_peered":"2024-05-28T06:24:22.992147+0000","last_unstale":"2024-05-28T06:24:54.984968+0000","last_undegraded":"2024-05-28T06:24:54.984968+0000","last_fullsized":"2024-05-28T06:24:54.984968+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:36:09.562056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.d","version":"132'79","reported_seq":168,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.322183+0000","last_change":"2024-05-28T06:24:23.989538+0000","last_active":"2024-05-28T06:26:18.322183+0000","last_peered":"2024-05-28T06:26:18.322183+0000","last_clean":"2024-05-28T06:26:18.322183+0000","last_became_active":"2024-05-28T06:24:23.988631+0000","last_became_peered":"2024-05-28T06:24:23.988631+0000","last_unstale":"2024-05-28T06:26:18.322183+0000","last_undegraded":"2024-05-28T06:26:18.322183+0000","last_fullsized":"2024-05-28T06:26:18.322183+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:08:42.734914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":272629760,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":72,"num_read_kb":294912,"num_write":79,"num_write_kb":294912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":19,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:55.454861+0000","last_change":"2024-05-28T06:24:22.997400+0000","last_active":"2024-05-28T06:24:55.454861+0000","last_peered":"2024-05-28T06:24:55.454861+0000","last_clean":"2024-05-28T06:24:55.454861+0000","last_became_active":"2024-05-28T06:24:22.993636+0000","last_became_peered":"2024-05-28T06:24:22.993636+0000","last_unstale":"2024-05-28T06:24:55.454861+0000","last_undegraded":"2024-05-28T06:24:55.454861+0000","last_fullsized":"2024-05-28T06:24:55.454861+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:43:01.373860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.a","version":"132'79","reported_seq":170,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.293566+0000","last_change":"2024-05-28T06:24:24.007538+0000","last_active":"2024-05-28T06:26:18.293566+0000","last_peered":"2024-05-28T06:26:18.293566+0000","last_clean":"2024-05-28T06:26:18.293566+0000","last_became_active":"2024-05-28T06:24:24.007259+0000","last_became_peered":"2024-05-28T06:24:24.007259+0000","last_unstale":"2024-05-28T06:26:18.293566+0000","last_undegraded":"2024-05-28T06:26:18.293566+0000","last_fullsized":"2024-05-28T06:26:18.293566+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:59:09.840788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306184192,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":74,"num_read_kb":303104,"num_write":79,"num_write_kb":311296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.b","version":"132'3","reported_seq":22,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:25:22.100610+0000","last_change":"2024-05-28T06:24:22.996819+0000","last_active":"2024-05-28T06:25:22.100610+0000","last_peered":"2024-05-28T06:25:22.100610+0000","last_clean":"2024-05-28T06:25:22.100610+0000","last_became_active":"2024-05-28T06:24:22.995489+0000","last_became_peered":"2024-05-28T06:24:22.995489+0000","last_unstale":"2024-05-28T06:25:22.100610+0000","last_undegraded":"2024-05-28T06:25:22.100610+0000","last_fullsized":"2024-05-28T06:25:22.100610+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:24:52.967145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.b","version":"132'90","reported_seq":171,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.295653+0000","last_change":"2024-05-28T06:24:24.015790+0000","last_active":"2024-05-28T06:26:18.295653+0000","last_peered":"2024-05-28T06:26:18.295653+0000","last_clean":"2024-05-28T06:26:18.295653+0000","last_became_active":"2024-05-28T06:24:24.015295+0000","last_became_peered":"2024-05-28T06:24:24.015295+0000","last_unstale":"2024-05-28T06:26:18.295653+0000","last_undegraded":"2024-05-28T06:26:18.295653+0000","last_fullsized":"2024-05-28T06:26:18.295653+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:37:59.260972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":314572932,"num_objects":76,"num_object_clones":0,"num_object_copies":152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76,"num_whiteouts":0,"num_read":64,"num_read_kb":237574,"num_write":90,"num_write_kb":323588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":19,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:55.341579+0000","last_change":"2024-05-28T06:24:22.995343+0000","last_active":"2024-05-28T06:24:55.341579+0000","last_peered":"2024-05-28T06:24:55.341579+0000","last_clean":"2024-05-28T06:24:55.341579+0000","last_became_active":"2024-05-28T06:24:22.994227+0000","last_became_peered":"2024-05-28T06:24:22.994227+0000","last_unstale":"2024-05-28T06:24:55.341579+0000","last_undegraded":"2024-05-28T06:24:55.341579+0000","last_fullsized":"2024-05-28T06:24:55.341579+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:50:39.661313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.8","version":"132'88","reported_seq":146,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.296033+0000","last_change":"2024-05-28T06:24:23.987172+0000","last_active":"2024-05-28T06:26:18.296033+0000","last_peered":"2024-05-28T06:26:18.296033+0000","last_clean":"2024-05-28T06:26:18.296033+0000","last_became_active":"2024-05-28T06:24:23.986897+0000","last_became_peered":"2024-05-28T06:24:23.986897+0000","last_unstale":"2024-05-28T06:26:18.296033+0000","last_undegraded":"2024-05-28T06:26:18.296033+0000","last_fullsized":"2024-05-28T06:26:18.296033+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:11:18.017207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":343932928,"num_objects":82,"num_object_clones":0,"num_object_copies":164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82,"num_whiteouts":0,"num_read":41,"num_read_kb":167936,"num_write":88,"num_write_kb":348160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":19,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:55.341333+0000","last_change":"2024-05-28T06:24:22.991355+0000","last_active":"2024-05-28T06:24:55.341333+0000","last_peered":"2024-05-28T06:24:55.341333+0000","last_clean":"2024-05-28T06:24:55.341333+0000","last_became_active":"2024-05-28T06:24:22.991078+0000","last_became_peered":"2024-05-28T06:24:22.991078+0000","last_unstale":"2024-05-28T06:24:55.341333+0000","last_undegraded":"2024-05-28T06:24:55.341333+0000","last_fullsized":"2024-05-28T06:24:55.341333+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:06:19.878458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"132'68","reported_seq":141,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.282953+0000","last_change":"2024-05-28T06:24:24.009275+0000","last_active":"2024-05-28T06:26:18.282953+0000","last_peered":"2024-05-28T06:26:18.282953+0000","last_clean":"2024-05-28T06:26:18.282953+0000","last_became_active":"2024-05-28T06:24:24.008955+0000","last_became_peered":"2024-05-28T06:24:24.008955+0000","last_unstale":"2024-05-28T06:26:18.282953+0000","last_undegraded":"2024-05-28T06:26:18.282953+0000","last_fullsized":"2024-05-28T06:26:18.282953+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:32:09.823976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":276824064,"num_objects":66,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":56,"num_read_kb":229376,"num_write":68,"num_write_kb":274432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":19,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:54.984300+0000","last_change":"2024-05-28T06:24:23.002572+0000","last_active":"2024-05-28T06:24:54.984300+0000","last_peered":"2024-05-28T06:24:54.984300+0000","last_clean":"2024-05-28T06:24:54.984300+0000","last_became_active":"2024-05-28T06:24:23.002292+0000","last_became_peered":"2024-05-28T06:24:23.002292+0000","last_unstale":"2024-05-28T06:24:54.984300+0000","last_undegraded":"2024-05-28T06:24:54.984300+0000","last_fullsized":"2024-05-28T06:24:54.984300+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:50:39.856735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.6","version":"132'87","reported_seq":139,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.318611+0000","last_change":"2024-05-28T06:24:24.017529+0000","last_active":"2024-05-28T06:26:18.318611+0000","last_peered":"2024-05-28T06:26:18.318611+0000","last_clean":"2024-05-28T06:26:18.318611+0000","last_became_active":"2024-05-28T06:24:24.017221+0000","last_became_peered":"2024-05-28T06:24:24.017221+0000","last_unstale":"2024-05-28T06:26:18.318611+0000","last_undegraded":"2024-05-28T06:26:18.318611+0000","last_fullsized":"2024-05-28T06:26:18.318611+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:32:09.823976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306184192,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":35,"num_read_kb":143360,"num_write":87,"num_write_kb":327680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.7","version":"129'2","reported_seq":21,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:54.994004+0000","last_change":"2024-05-28T06:24:23.003960+0000","last_active":"2024-05-28T06:24:54.994004+0000","last_peered":"2024-05-28T06:24:54.994004+0000","last_clean":"2024-05-28T06:24:54.994004+0000","last_became_active":"2024-05-28T06:24:23.003502+0000","last_became_peered":"2024-05-28T06:24:23.003502+0000","last_unstale":"2024-05-28T06:24:54.994004+0000","last_undegraded":"2024-05-28T06:24:54.994004+0000","last_fullsized":"2024-05-28T06:24:54.994004+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:13:19.450560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.7","version":"132'84","reported_seq":171,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.313436+0000","last_change":"2024-05-28T06:24:24.020448+0000","last_active":"2024-05-28T06:26:18.313436+0000","last_peered":"2024-05-28T06:26:18.313436+0000","last_clean":"2024-05-28T06:26:18.313436+0000","last_became_active":"2024-05-28T06:24:24.019730+0000","last_became_peered":"2024-05-28T06:24:24.019730+0000","last_unstale":"2024-05-28T06:26:18.313436+0000","last_undegraded":"2024-05-28T06:26:18.313436+0000","last_fullsized":"2024-05-28T06:26:18.313436+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:26:56.800134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":310378496,"num_objects":74,"num_object_clones":0,"num_object_copies":148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74,"num_whiteouts":0,"num_read":70,"num_read_kb":286720,"num_write":84,"num_write_kb":323584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.6","version":"132'41","reported_seq":60,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:21.025193+0000","last_change":"2024-05-28T06:24:23.003133+0000","last_active":"2024-05-28T06:26:21.025193+0000","last_peered":"2024-05-28T06:26:21.025193+0000","last_clean":"2024-05-28T06:26:21.025193+0000","last_became_active":"2024-05-28T06:24:23.002821+0000","last_became_peered":"2024-05-28T06:24:23.002821+0000","last_unstale":"2024-05-28T06:26:21.025193+0000","last_undegraded":"2024-05-28T06:26:21.025193+0000","last_fullsized":"2024-05-28T06:26:21.025193+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:18:06.253798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":95,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.4","version":"132'80","reported_seq":184,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.322862+0000","last_change":"2024-05-28T06:24:24.004928+0000","last_active":"2024-05-28T06:26:18.322862+0000","last_peered":"2024-05-28T06:26:18.322862+0000","last_clean":"2024-05-28T06:26:18.322862+0000","last_became_active":"2024-05-28T06:24:24.004599+0000","last_became_peered":"2024-05-28T06:24:24.004599+0000","last_unstale":"2024-05-28T06:26:18.322862+0000","last_undegraded":"2024-05-28T06:26:18.322862+0000","last_fullsized":"2024-05-28T06:26:18.322862+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:09:56.044033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":276824064,"num_objects":66,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":87,"num_read_kb":356352,"num_write":80,"num_write_kb":294913,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":19,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:54.996483+0000","last_change":"2024-05-28T06:24:23.003603+0000","last_active":"2024-05-28T06:24:54.996483+0000","last_peered":"2024-05-28T06:24:54.996483+0000","last_clean":"2024-05-28T06:24:54.996483+0000","last_became_active":"2024-05-28T06:24:23.002468+0000","last_became_peered":"2024-05-28T06:24:23.002468+0000","last_unstale":"2024-05-28T06:24:54.996483+0000","last_undegraded":"2024-05-28T06:24:54.996483+0000","last_fullsized":"2024-05-28T06:24:54.996483+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:48:27.218311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.5","version":"132'86","reported_seq":149,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.318905+0000","last_change":"2024-05-28T06:24:23.999563+0000","last_active":"2024-05-28T06:26:18.318905+0000","last_peered":"2024-05-28T06:26:18.318905+0000","last_clean":"2024-05-28T06:26:18.318905+0000","last_became_active":"2024-05-28T06:24:23.999013+0000","last_became_peered":"2024-05-28T06:24:23.999013+0000","last_unstale":"2024-05-28T06:26:18.318905+0000","last_undegraded":"2024-05-28T06:26:18.318905+0000","last_fullsized":"2024-05-28T06:26:18.318905+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:47:25.671388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":285213316,"num_objects":72,"num_object_clones":1,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72,"num_whiteouts":1,"num_read":47,"num_read_kb":192512,"num_write":84,"num_write_kb":303107,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":19,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:54.985023+0000","last_change":"2024-05-28T06:24:22.986876+0000","last_active":"2024-05-28T06:24:54.985023+0000","last_peered":"2024-05-28T06:24:54.985023+0000","last_clean":"2024-05-28T06:24:54.985023+0000","last_became_active":"2024-05-28T06:24:22.986620+0000","last_became_peered":"2024-05-28T06:24:22.986620+0000","last_unstale":"2024-05-28T06:24:54.985023+0000","last_undegraded":"2024-05-28T06:24:54.985023+0000","last_fullsized":"2024-05-28T06:24:54.985023+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:20:23.711600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2","version":"132'90","reported_seq":159,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.297667+0000","last_change":"2024-05-28T06:24:24.012293+0000","last_active":"2024-05-28T06:26:18.297667+0000","last_peered":"2024-05-28T06:26:18.297667+0000","last_clean":"2024-05-28T06:26:18.297667+0000","last_became_active":"2024-05-28T06:24:24.011989+0000","last_became_peered":"2024-05-28T06:24:24.011989+0000","last_unstale":"2024-05-28T06:26:18.297667+0000","last_undegraded":"2024-05-28T06:26:18.297667+0000","last_fullsized":"2024-05-28T06:26:18.297667+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:47:31.291023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":352321536,"num_objects":84,"num_object_clones":0,"num_object_copies":168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84,"num_whiteouts":0,"num_read":52,"num_read_kb":212992,"num_write":90,"num_write_kb":356352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":19,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:54.995987+0000","last_change":"2024-05-28T06:24:23.008363+0000","last_active":"2024-05-28T06:24:54.995987+0000","last_peered":"2024-05-28T06:24:54.995987+0000","last_clean":"2024-05-28T06:24:54.995987+0000","last_became_active":"2024-05-28T06:24:23.007859+0000","last_became_peered":"2024-05-28T06:24:23.007859+0000","last_unstale":"2024-05-28T06:24:54.995987+0000","last_undegraded":"2024-05-28T06:24:54.995987+0000","last_fullsized":"2024-05-28T06:24:54.995987+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:45:32.156309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"132'98","reported_seq":198,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.323247+0000","last_change":"2024-05-28T06:24:24.013906+0000","last_active":"2024-05-28T06:26:18.323247+0000","last_peered":"2024-05-28T06:26:18.323247+0000","last_clean":"2024-05-28T06:26:18.323247+0000","last_became_active":"2024-05-28T06:24:24.011269+0000","last_became_peered":"2024-05-28T06:24:24.011269+0000","last_unstale":"2024-05-28T06:26:18.323247+0000","last_undegraded":"2024-05-28T06:26:18.323247+0000","last_fullsized":"2024-05-28T06:26:18.323247+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:09:03.314508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369098752,"num_objects":88,"num_object_clones":0,"num_object_copies":176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88,"num_whiteouts":0,"num_read":83,"num_read_kb":327683,"num_write":98,"num_write_kb":376833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":19,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:55.133578+0000","last_change":"2024-05-28T06:24:23.006728+0000","last_active":"2024-05-28T06:24:55.133578+0000","last_peered":"2024-05-28T06:24:55.133578+0000","last_clean":"2024-05-28T06:24:55.133578+0000","last_became_active":"2024-05-28T06:24:23.005895+0000","last_became_peered":"2024-05-28T06:24:23.005895+0000","last_unstale":"2024-05-28T06:24:55.133578+0000","last_undegraded":"2024-05-28T06:24:55.133578+0000","last_fullsized":"2024-05-28T06:24:55.133578+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:53:06.353639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"132'87","reported_seq":175,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.310121+0000","last_change":"2024-05-28T06:24:24.013072+0000","last_active":"2024-05-28T06:26:18.310121+0000","last_peered":"2024-05-28T06:26:18.310121+0000","last_clean":"2024-05-28T06:26:18.310121+0000","last_became_active":"2024-05-28T06:24:24.012814+0000","last_became_peered":"2024-05-28T06:24:24.012814+0000","last_unstale":"2024-05-28T06:26:18.310121+0000","last_undegraded":"2024-05-28T06:26:18.310121+0000","last_fullsized":"2024-05-28T06:26:18.310121+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:32:09.823976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":339738624,"num_objects":81,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81,"num_whiteouts":0,"num_read":71,"num_read_kb":290816,"num_write":87,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1","version":"132'31","reported_seq":80,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.171426+0000","last_change":"2024-05-28T06:24:23.001199+0000","last_active":"2024-05-28T06:26:18.171426+0000","last_peered":"2024-05-28T06:26:18.171426+0000","last_clean":"2024-05-28T06:26:18.171426+0000","last_became_active":"2024-05-28T06:24:23.000861+0000","last_became_peered":"2024-05-28T06:24:23.000861+0000","last_unstale":"2024-05-28T06:26:18.171426+0000","last_undegraded":"2024-05-28T06:26:18.171426+0000","last_fullsized":"2024-05-28T06:26:18.171426+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:36:09.562056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4553,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":30,"num_read_kb":30,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1","version":"132'93","reported_seq":186,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.322559+0000","last_change":"2024-05-28T06:24:24.005940+0000","last_active":"2024-05-28T06:26:18.322559+0000","last_peered":"2024-05-28T06:26:18.322559+0000","last_clean":"2024-05-28T06:26:18.322559+0000","last_became_active":"2024-05-28T06:24:24.005485+0000","last_became_peered":"2024-05-28T06:24:24.005485+0000","last_unstale":"2024-05-28T06:26:18.322559+0000","last_undegraded":"2024-05-28T06:26:18.322559+0000","last_fullsized":"2024-05-28T06:26:18.322559+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:52:44.413787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":314572800,"num_objects":75,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75,"num_whiteouts":0,"num_read":76,"num_read_kb":311296,"num_write":93,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":19,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:54.984220+0000","last_change":"2024-05-28T06:24:22.987451+0000","last_active":"2024-05-28T06:24:54.984220+0000","last_peered":"2024-05-28T06:24:54.984220+0000","last_clean":"2024-05-28T06:24:54.984220+0000","last_became_active":"2024-05-28T06:24:22.987181+0000","last_became_peered":"2024-05-28T06:24:22.987181+0000","last_unstale":"2024-05-28T06:24:54.984220+0000","last_undegraded":"2024-05-28T06:24:54.984220+0000","last_fullsized":"2024-05-28T06:24:54.984220+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:45:51.036866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.12","version":"132'65","reported_seq":128,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.240342+0000","last_change":"2024-05-28T06:24:24.007933+0000","last_active":"2024-05-28T06:26:18.240342+0000","last_peered":"2024-05-28T06:26:18.240342+0000","last_clean":"2024-05-28T06:26:18.240342+0000","last_became_active":"2024-05-28T06:24:24.007651+0000","last_became_peered":"2024-05-28T06:24:24.007651+0000","last_unstale":"2024-05-28T06:26:18.240342+0000","last_undegraded":"2024-05-28T06:26:18.240342+0000","last_fullsized":"2024-05-28T06:26:18.240342+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:59:32.549372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":255852544,"num_objects":61,"num_object_clones":0,"num_object_copies":122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":0,"num_read":47,"num_read_kb":192512,"num_write":65,"num_write_kb":258048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"129'2","reported_seq":20,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:55.133764+0000","last_change":"2024-05-28T06:24:23.008078+0000","last_active":"2024-05-28T06:24:55.133764+0000","last_peered":"2024-05-28T06:24:55.133764+0000","last_clean":"2024-05-28T06:24:55.133764+0000","last_became_active":"2024-05-28T06:24:23.007331+0000","last_became_peered":"2024-05-28T06:24:23.007331+0000","last_unstale":"2024-05-28T06:24:55.133764+0000","last_undegraded":"2024-05-28T06:24:55.133764+0000","last_fullsized":"2024-05-28T06:24:55.133764+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:50:39.661313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":388,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:55.134179+0000","last_change":"2024-05-28T06:04:01.039129+0000","last_active":"2024-05-28T06:24:55.134179+0000","last_peered":"2024-05-28T06:24:55.134179+0000","last_clean":"2024-05-28T06:24:55.134179+0000","last_became_active":"2024-05-28T06:04:01.033906+0000","last_became_peered":"2024-05-28T06:04:01.033906+0000","last_unstale":"2024-05-28T06:24:55.134179+0000","last_undegraded":"2024-05-28T06:24:55.134179+0000","last_fullsized":"2024-05-28T06:24:55.134179+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:03:59.864237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:03:59.864237+0000","last_clean_scrub_stamp":"2024-05-28T06:03:59.864237+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:17:36.440245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":18396218954,"num_objects":4395,"num_object_clones":1,"num_object_copies":8790,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4395,"num_whiteouts":1,"num_read":3946,"num_read_kb":15908926,"num_write":4949,"num_write_kb":18956331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":41943040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36792500224,"data_stored":36792437908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4958,"ondisk_log_size":4958,"up":128,"acting":128,"num_store_stats":8},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":1678588,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":30,"num_read_kb":30,"num_write":598,"num_write_kb":1879,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3424256,"data_stored":3357176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":544,"ondisk_log_size":544,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575123,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3952528,"kb_used_data":3813128,"kb_used_omap":1,"kb_used_meta":139390,"kb_avail":90419312,"statfs":{"total":96636764160,"available":92589375488,"internally_reserved":0,"allocated":3904643072,"data_stored":3903694392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":142735818},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575125,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5139788,"kb_used_data":5000388,"kb_used_omap":1,"kb_used_meta":139390,"kb_avail":89232052,"statfs":{"total":96636764160,"available":91373621248,"internally_reserved":0,"allocated":5120397312,"data_stored":5119456102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":142735814},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575123,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3815132,"kb_used_data":3675732,"kb_used_omap":1,"kb_used_meta":139390,"kb_avail":90556708,"statfs":{"total":96636764160,"available":92730068992,"internally_reserved":0,"allocated":3763949568,"data_stored":3763018100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":142735813},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575123,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5547692,"kb_used_data":5412452,"kb_used_omap":1,"kb_used_meta":135230,"kb_avail":88824148,"statfs":{"total":96636764160,"available":90955927552,"internally_reserved":0,"allocated":5542350848,"data_stored":5541404120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":138475973},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575125,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4527844,"kb_used_data":4388444,"kb_used_omap":1,"kb_used_meta":139390,"kb_avail":89843996,"statfs":{"total":96636764160,"available":92000251904,"internally_reserved":0,"allocated":4493766656,"data_stored":4492831645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":142735815},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575124,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5961384,"kb_used_data":5826144,"kb_used_omap":1,"kb_used_meta":135230,"kb_avail":88410456,"statfs":{"total":96636764160,"available":90532306944,"internally_reserved":0,"allocated":5965971456,"data_stored":5965028896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":138475973},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575124,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4155024,"kb_used_data":4019784,"kb_used_omap":1,"kb_used_meta":135230,"kb_avail":90216816,"statfs":{"total":96636764160,"available":92382019584,"internally_reserved":0,"allocated":4116258816,"data_stored":4115339392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":138475973},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575123,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3959192,"kb_used_data":3811472,"kb_used_omap":1,"kb_used_meta":147710,"kb_avail":90412648,"statfs":{"total":96636764160,"available":92582551552,"internally_reserved":0,"allocated":3902947328,"data_stored":3902020517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151255493},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1691648,"data_stored":1677790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1695744,"data_stored":1673875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3900706816,"data_stored":3900702981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4114612224,"data_stored":4114612224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":5964316672,"data_stored":5964301112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4492103680,"data_stored":4492099764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":5540700160,"data_stored":5540676882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3762294784,"data_stored":3762290820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":5117059072,"data_stored":5117051144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3900706816,"data_stored":3900702981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-28T06:26:43.909 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-28T06:26:44.145 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.151+0000 7fa0d7200640 1 Processor -- start 2024-05-28T06:26:44.145 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.151+0000 7fa0d7200640 1 -- start start 2024-05-28T06:26:44.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.153+0000 7fa0d7200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fa0d015ae80 0x7fa0d00b49c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:44.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.153+0000 7fa0d7200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0d00b31f0 con 0x7fa0d00b3fb0 2024-05-28T06:26:44.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.153+0000 7fa0d7200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0d00b34d0 con 0x7fa0d006e2d0 2024-05-28T06:26:44.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.153+0000 7fa0d7200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fa0d00b4f00 con 0x7fa0d015ae80 2024-05-28T06:26:44.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.153+0000 7fa0c7400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fa0d00b3fb0 0x7fa0d006dbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:44848/0 (socket says 172.21.15.96:44848) 2024-05-28T06:26:44.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.153+0000 7fa0c7400640 1 -- 172.21.15.96:0/3761553947 learned_addr learned my addr 172.21.15.96:0/3761553947 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:26:44.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.154+0000 7fa0cfe00640 1 --2- 172.21.15.96:0/3761553947 >> v2:172.21.15.123:3300/0 conn(0x7fa0d015ae80 0x7fa0d00b49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:44.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.154+0000 7fa0cf400640 1 -- 172.21.15.96:0/3761553947 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2123546193 0 0) 0x7fa0d00b31f0 con 0x7fa0d00b3fb0 2024-05-28T06:26:44.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.154+0000 7fa0cf400640 1 -- 172.21.15.96:0/3761553947 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa0b4003520 con 0x7fa0d00b3fb0 2024-05-28T06:26:44.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.154+0000 7fa0cf400640 1 -- 172.21.15.96:0/3761553947 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 234547704 0 0) 0x7fa0b4003520 con 0x7fa0d00b3fb0 2024-05-28T06:26:44.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.154+0000 7fa0cf400640 1 -- 172.21.15.96:0/3761553947 >> v2:172.21.15.123:3300/0 conn(0x7fa0d015ae80 msgr2=0x7fa0d00b49c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:44.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.154+0000 7fa0cf400640 1 --2- 172.21.15.96:0/3761553947 >> v2:172.21.15.123:3300/0 conn(0x7fa0d015ae80 0x7fa0d00b49c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:44.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.154+0000 7fa0cf400640 1 -- 172.21.15.96:0/3761553947 >> v1:172.21.15.123:6789/0 conn(0x7fa0d006e2d0 legacy=0x7fa0d015e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:44.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.154+0000 7fa0cf400640 1 -- 172.21.15.96:0/3761553947 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0d00b37b0 con 0x7fa0d00b3fb0 2024-05-28T06:26:44.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.154+0000 7fa0cf400640 1 -- 172.21.15.96:0/3761553947 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7fa0b8001670 con 0x7fa0d00b3fb0 2024-05-28T06:26:44.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.154+0000 7fa0cfe00640 1 --2- 172.21.15.96:0/3761553947 >> v2:172.21.15.123:3300/0 conn(0x7fa0d015ae80 0x7fa0d00b49c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-28T06:26:44.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.154+0000 7fa0cf400640 1 -- 172.21.15.96:0/3761553947 >> v1:172.21.15.96:6789/0 conn(0x7fa0d00b3fb0 legacy=0x7fa0d006dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:44.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.154+0000 7fa0cf400640 1 --2- 172.21.15.96:0/3761553947 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa0b4003a00 0x7fa0b4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:44.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.154+0000 7fa0cf400640 1 --2- 172.21.15.96:0/3761553947 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa0b4004800 0x7fa0b4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:44.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.154+0000 7fa0cf400640 1 --2- 172.21.15.96:0/3761553947 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa0b4005210 0x7fa0b400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:44.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.154+0000 7fa0cf400640 1 -- 172.21.15.96:0/3761553947 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa0b400c130 con 0x7fa0b4005210 2024-05-28T06:26:44.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.154+0000 7fa0cf400640 1 -- 172.21.15.96:0/3761553947 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa0b400c2e0 con 0x7fa0b4003a00 2024-05-28T06:26:44.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.154+0000 7fa0cf400640 1 -- 172.21.15.96:0/3761553947 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fa0b400c4e0 con 0x7fa0b4004800 2024-05-28T06:26:44.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.155+0000 7fa0d5200640 1 --2- 172.21.15.96:0/3761553947 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa0b4005210 0x7fa0b400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:44.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.155+0000 7fa0c7400640 1 --2- 172.21.15.96:0/3761553947 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa0b4004800 0x7fa0b4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:44.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.155+0000 7fa0cfe00640 1 --2- 172.21.15.96:0/3761553947 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa0b4003a00 0x7fa0b4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:44.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.155+0000 7fa0d5200640 1 -- 172.21.15.96:0/3761553947 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa0b4004800 msgr2=0x7fa0b4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:44.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.155+0000 7fa0d5200640 1 --2- 172.21.15.96:0/3761553947 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa0b4004800 0x7fa0b4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:44.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.155+0000 7fa0d5200640 1 -- 172.21.15.96:0/3761553947 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa0b4003a00 msgr2=0x7fa0b4003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:44.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.155+0000 7fa0d5200640 1 --2- 172.21.15.96:0/3761553947 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa0b4003a00 0x7fa0b4003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:44.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.155+0000 7fa0d5200640 1 -- 172.21.15.96:0/3761553947 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0b4003520 con 0x7fa0b4005210 2024-05-28T06:26:44.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.156+0000 7fa0d5200640 1 -- 172.21.15.96:0/3761553947 reap_dead start 2024-05-28T06:26:44.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.156+0000 7fa0d5200640 1 --2- 172.21.15.96:0/3761553947 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa0b4005210 0x7fa0b400b9c0 secure :-1 s=READY pgs=2181 cs=0 l=1 rev1=1 crypto rx=0x7fa0c8001e50 tx=0x7fa0b4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc6026c5e7a5f9c6 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:44.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.156+0000 7fa0cf400640 1 -- 172.21.15.96:0/3761553947 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0c8001a70 con 0x7fa0b4005210 2024-05-28T06:26:44.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.156+0000 7fa0cf400640 1 -- 172.21.15.96:0/3761553947 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0c8003970 con 0x7fa0b4005210 2024-05-28T06:26:44.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.156+0000 7fa0cf400640 1 -- 172.21.15.96:0/3761553947 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0c8003d60 con 0x7fa0b4005210 2024-05-28T06:26:44.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.157+0000 7fa0d7200640 1 -- 172.21.15.96:0/3761553947 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa0b4005210 msgr2=0x7fa0b400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:44.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.157+0000 7fa0d7200640 1 --2- 172.21.15.96:0/3761553947 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa0b4005210 0x7fa0b400b9c0 secure :-1 s=READY pgs=2181 cs=0 l=1 rev1=1 crypto rx=0x7fa0c8001e50 tx=0x7fa0b4005830 comp rx=0 tx=0).stop 2024-05-28T06:26:44.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.157+0000 7fa0d7200640 1 -- 172.21.15.96:0/3761553947 shutdown_connections 2024-05-28T06:26:44.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.157+0000 7fa0d7200640 1 --2- 172.21.15.96:0/3761553947 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa0b4005210 0x7fa0b400b9c0 unknown :-1 s=CLOSED pgs=2181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:44.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.157+0000 7fa0d7200640 1 -- 172.21.15.96:0/3761553947 >> 172.21.15.96:0/3761553947 conn(0x7fa0d006bae0 msgr2=0x7fa0d006bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:26:44.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.157+0000 7fa0d7200640 1 -- 172.21.15.96:0/3761553947 shutdown_connections 2024-05-28T06:26:44.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.157+0000 7fa0d7200640 1 -- 172.21.15.96:0/3761553947 wait complete. 2024-05-28T06:26:44.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.159+0000 7fa0d7200640 1 Processor -- start 2024-05-28T06:26:44.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.159+0000 7fa0d7200640 1 -- start start 2024-05-28T06:26:44.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.160+0000 7fa0d7200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa0b4005210 0x7fa0d00c5760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:44.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.160+0000 7fa0d7200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa0d00cd070 0x7fa0d00cd4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:44.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.160+0000 7fa0d7200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa0d00b1c90 0x7fa0d00b2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:44.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.160+0000 7fa0d7200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa0d00b5130 con 0x7fa0d00cd070 2024-05-28T06:26:44.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.160+0000 7fa0d7200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa0d015b260 con 0x7fa0d00b1c90 2024-05-28T06:26:44.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.160+0000 7fa0d7200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fa0d006e760 con 0x7fa0b4005210 2024-05-28T06:26:44.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.160+0000 7fa0c7400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa0d00cd070 0x7fa0d00cd4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:44.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.160+0000 7fa0c7400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa0d00cd070 0x7fa0d00cd4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35078/0 (socket says 172.21.15.96:35078) 2024-05-28T06:26:44.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.160+0000 7fa0c7400640 1 -- 172.21.15.96:0/3742750704 learned_addr learned my addr 172.21.15.96:0/3742750704 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:26:44.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.161+0000 7fa0d5200640 1 --2- 172.21.15.96:0/3742750704 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa0d00b1c90 0x7fa0d00b2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:44.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.161+0000 7fa0cfe00640 1 --2- 172.21.15.96:0/3742750704 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa0b4005210 0x7fa0d00c5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:44.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.161+0000 7fa0c7400640 1 -- 172.21.15.96:0/3742750704 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa0b4005210 msgr2=0x7fa0d00c5760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:44.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.161+0000 7fa0c7400640 1 --2- 172.21.15.96:0/3742750704 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa0b4005210 0x7fa0d00c5760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:44.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.161+0000 7fa0c7400640 1 -- 172.21.15.96:0/3742750704 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa0d00b1c90 msgr2=0x7fa0d00b2110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:44.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.161+0000 7fa0c7400640 1 --2- 172.21.15.96:0/3742750704 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa0d00b1c90 0x7fa0d00b2110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:44.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.161+0000 7fa0c7400640 1 -- 172.21.15.96:0/3742750704 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0d01f46d0 con 0x7fa0d00cd070 2024-05-28T06:26:44.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.161+0000 7fa0d5200640 1 --2- 172.21.15.96:0/3742750704 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa0d00b1c90 0x7fa0d00b2110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:26:44.159 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.161+0000 7fa0c7400640 1 --2- 172.21.15.96:0/3742750704 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa0d00cd070 0x7fa0d00cd4d0 secure :-1 s=READY pgs=2182 cs=0 l=1 rev1=1 crypto rx=0x7fa0b800e730 tx=0x7fa0b800ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:44.159 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.161+0000 7fa0ccc00640 1 -- 172.21.15.96:0/3742750704 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0b800ee40 con 0x7fa0d00cd070 2024-05-28T06:26:44.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.161+0000 7fa0ccc00640 1 -- 172.21.15.96:0/3742750704 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0b8018070 con 0x7fa0d00cd070 2024-05-28T06:26:44.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.161+0000 7fa0d7200640 1 -- 172.21.15.96:0/3742750704 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0d01f49b0 con 0x7fa0d00cd070 2024-05-28T06:26:44.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.162+0000 7fa0ccc00640 1 -- 172.21.15.96:0/3742750704 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0b800cab0 con 0x7fa0d00cd070 2024-05-28T06:26:44.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.162+0000 7fa0d7200640 1 -- 172.21.15.96:0/3742750704 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0d01f4f70 con 0x7fa0d00cd070 2024-05-28T06:26:44.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.163+0000 7fa0d7200640 1 -- 172.21.15.96:0/3742750704 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0d00689e0 con 0x7fa0d00cd070 2024-05-28T06:26:44.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.164+0000 7fa0ccc00640 1 -- 172.21.15.96:0/3742750704 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 49) ==== 94049+0+0 (secure 0 0 0) 0x7fa0b800cc50 con 0x7fa0d00cd070 2024-05-28T06:26:44.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.165+0000 7fa0ccc00640 1 --2- 172.21.15.96:0/3742750704 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa0ac06ea50 0x7fa0ac070f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:44.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.165+0000 7fa0ccc00640 1 -- 172.21.15.96:0/3742750704 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 5823+0+0 (secure 0 0 0) 0x7fa0b808a890 con 0x7fa0d00cd070 2024-05-28T06:26:44.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.165+0000 7fa0cfe00640 1 --2- 172.21.15.96:0/3742750704 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa0ac06ea50 0x7fa0ac070f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:44.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.167+0000 7fa0cfe00640 1 --2- 172.21.15.96:0/3742750704 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa0ac06ea50 0x7fa0ac070f10 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7fa0c0006fd0 tx=0x7fa0c000f040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:44.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.169+0000 7fa0ccc00640 1 -- 172.21.15.96:0/3742750704 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa0b8096080 con 0x7fa0d00cd070 2024-05-28T06:26:44.356 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.363+0000 7fa0d7200640 1 -- 172.21.15.96:0/3742750704 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa0d006ed10 con 0x7fa0ac06ea50 2024-05-28T06:26:44.377 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.383+0000 7fa0ccc00640 1 -- 172.21.15.96:0/3742750704 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+254734 (secure 0 0 0) 0x7fa0c005b320 con 0x7fa0ac06ea50 2024-05-28T06:26:44.381 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:26:44.382 INFO:teuthology.orchestra.run.smithi096.stderr:dumped all 2024-05-28T06:26:44.384 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.389+0000 7fa0d7200640 1 -- 172.21.15.96:0/3742750704 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa0ac06ea50 msgr2=0x7fa0ac070f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:44.384 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.389+0000 7fa0d7200640 1 --2- 172.21.15.96:0/3742750704 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa0ac06ea50 0x7fa0ac070f10 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7fa0c0006fd0 tx=0x7fa0c000f040 comp rx=0 tx=0).stop 2024-05-28T06:26:44.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.389+0000 7fa0d7200640 1 -- 172.21.15.96:0/3742750704 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa0d00cd070 msgr2=0x7fa0d00cd4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:44.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.389+0000 7fa0d7200640 1 --2- 172.21.15.96:0/3742750704 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa0d00cd070 0x7fa0d00cd4d0 secure :-1 s=READY pgs=2182 cs=0 l=1 rev1=1 crypto rx=0x7fa0b800e730 tx=0x7fa0b800ebf0 comp rx=0 tx=0).stop 2024-05-28T06:26:44.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.390+0000 7fa0d7200640 1 -- 172.21.15.96:0/3742750704 shutdown_connections 2024-05-28T06:26:44.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.390+0000 7fa0d7200640 1 --2- 172.21.15.96:0/3742750704 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa0ac06ea50 0x7fa0ac070f10 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:44.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.390+0000 7fa0d7200640 1 --2- 172.21.15.96:0/3742750704 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa0d00b1c90 0x7fa0d00b2110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:44.385 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.390+0000 7fa0d7200640 1 --2- 172.21.15.96:0/3742750704 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa0d00cd070 0x7fa0d00cd4d0 unknown :-1 s=CLOSED pgs=2182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:44.386 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.390+0000 7fa0d7200640 1 --2- 172.21.15.96:0/3742750704 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa0b4005210 0x7fa0d00c5760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:44.386 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.390+0000 7fa0d7200640 1 -- 172.21.15.96:0/3742750704 >> 172.21.15.96:0/3742750704 conn(0x7fa0d006bae0 msgr2=0x7fa0d00731a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:26:44.386 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.390+0000 7fa0d7200640 1 -- 172.21.15.96:0/3742750704 shutdown_connections 2024-05-28T06:26:44.386 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.390+0000 7fa0d7200640 1 -- 172.21.15.96:0/3742750704 wait complete. 2024-05-28T06:26:44.403 INFO:teuthology.orchestra.run.smithi096.stdout:{"pg_ready":true,"pg_map":{"version":807,"stamp":"2024-05-28T06:26:44.081370+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":18398487910,"num_objects":4420,"num_object_clones":1,"num_object_copies":8840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4420,"num_whiteouts":1,"num_read":4092,"num_read_kb":15909056,"num_write":5706,"num_write_kb":18960626,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":41943040,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5619,"ondisk_log_size":5619,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":37058584,"kb_used_data":35947544,"kb_used_omap":12,"kb_used_meta":1110963,"kb_avail":717916136,"statfs":{"total":773094113280,"available":735146123264,"internally_reserved":0,"allocated":36810285056,"data_stored":36802793164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":1137626672},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004902"},"pg_stats":[{"pgid":"19.3f","version":"132'72","reported_seq":159,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.319580+0000","last_change":"2024-05-28T06:24:24.007430+0000","last_active":"2024-05-28T06:26:18.319580+0000","last_peered":"2024-05-28T06:26:18.319580+0000","last_clean":"2024-05-28T06:26:18.319580+0000","last_became_active":"2024-05-28T06:24:24.006988+0000","last_became_peered":"2024-05-28T06:24:24.006988+0000","last_unstale":"2024-05-28T06:26:18.319580+0000","last_undegraded":"2024-05-28T06:26:18.319580+0000","last_fullsized":"2024-05-28T06:26:18.319580+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:45:33.172226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260046848,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":70,"num_read_kb":286720,"num_write":72,"num_write_kb":274432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3e","version":"132'84","reported_seq":169,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.291618+0000","last_change":"2024-05-28T06:24:23.986742+0000","last_active":"2024-05-28T06:26:18.291618+0000","last_peered":"2024-05-28T06:26:18.291618+0000","last_clean":"2024-05-28T06:26:18.291618+0000","last_became_active":"2024-05-28T06:24:23.986471+0000","last_became_peered":"2024-05-28T06:24:23.986471+0000","last_unstale":"2024-05-28T06:26:18.291618+0000","last_undegraded":"2024-05-28T06:26:18.291618+0000","last_fullsized":"2024-05-28T06:26:18.291618+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:02:16.956326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327155712,"num_objects":78,"num_object_clones":0,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78,"num_whiteouts":0,"num_read":68,"num_read_kb":278528,"num_write":84,"num_write_kb":331776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3d","version":"132'93","reported_seq":180,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.325236+0000","last_change":"2024-05-28T06:24:24.007698+0000","last_active":"2024-05-28T06:26:18.325236+0000","last_peered":"2024-05-28T06:26:18.325236+0000","last_clean":"2024-05-28T06:26:18.325236+0000","last_became_active":"2024-05-28T06:24:24.007416+0000","last_became_peered":"2024-05-28T06:24:24.007416+0000","last_unstale":"2024-05-28T06:26:18.325236+0000","last_undegraded":"2024-05-28T06:26:18.325236+0000","last_fullsized":"2024-05-28T06:26:18.325236+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:24:31.676324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306184192,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":70,"num_read_kb":253960,"num_write":93,"num_write_kb":323588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.3c","version":"132'71","reported_seq":133,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.305549+0000","last_change":"2024-05-28T06:24:24.011512+0000","last_active":"2024-05-28T06:26:18.305549+0000","last_peered":"2024-05-28T06:26:18.305549+0000","last_clean":"2024-05-28T06:26:18.305549+0000","last_became_active":"2024-05-28T06:24:24.011234+0000","last_became_peered":"2024-05-28T06:24:24.011234+0000","last_unstale":"2024-05-28T06:26:18.305549+0000","last_undegraded":"2024-05-28T06:26:18.305549+0000","last_fullsized":"2024-05-28T06:26:18.305549+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:59:05.689554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281018368,"num_objects":67,"num_object_clones":0,"num_object_copies":134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":0,"num_read":45,"num_read_kb":176130,"num_write":71,"num_write_kb":278529,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3b","version":"132'65","reported_seq":119,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.288952+0000","last_change":"2024-05-28T06:24:24.019397+0000","last_active":"2024-05-28T06:26:18.288952+0000","last_peered":"2024-05-28T06:26:18.288952+0000","last_clean":"2024-05-28T06:26:18.288952+0000","last_became_active":"2024-05-28T06:24:24.018334+0000","last_became_peered":"2024-05-28T06:24:24.018334+0000","last_unstale":"2024-05-28T06:26:18.288952+0000","last_undegraded":"2024-05-28T06:26:18.288952+0000","last_fullsized":"2024-05-28T06:26:18.288952+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:22:34.611665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":230686720,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":38,"num_read_kb":155648,"num_write":65,"num_write_kb":245760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.3a","version":"132'97","reported_seq":216,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.278772+0000","last_change":"2024-05-28T06:24:55.231438+0000","last_active":"2024-05-28T06:26:18.278772+0000","last_peered":"2024-05-28T06:26:18.278772+0000","last_clean":"2024-05-28T06:26:18.278772+0000","last_became_active":"2024-05-28T06:24:54.990013+0000","last_became_peered":"2024-05-28T06:24:54.990013+0000","last_unstale":"2024-05-28T06:26:18.278772+0000","last_undegraded":"2024-05-28T06:26:18.278772+0000","last_fullsized":"2024-05-28T06:26:18.278772+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:50:40.872653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":360710405,"num_objects":87,"num_object_clones":0,"num_object_copies":174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87,"num_whiteouts":0,"num_read":84,"num_read_kb":339969,"num_write":97,"num_write_kb":368642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":25165824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.39","version":"132'65","reported_seq":126,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.259648+0000","last_change":"2024-05-28T06:24:24.006117+0000","last_active":"2024-05-28T06:26:18.259648+0000","last_peered":"2024-05-28T06:26:18.259648+0000","last_clean":"2024-05-28T06:26:18.259648+0000","last_became_active":"2024-05-28T06:24:24.005839+0000","last_became_peered":"2024-05-28T06:24:24.005839+0000","last_unstale":"2024-05-28T06:26:18.259648+0000","last_undegraded":"2024-05-28T06:26:18.259648+0000","last_fullsized":"2024-05-28T06:26:18.259648+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:47:32.965627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222298112,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":45,"num_read_kb":184320,"num_write":65,"num_write_kb":241664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.38","version":"132'79","reported_seq":142,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.263031+0000","last_change":"2024-05-28T06:24:24.008772+0000","last_active":"2024-05-28T06:26:18.263031+0000","last_peered":"2024-05-28T06:26:18.263031+0000","last_clean":"2024-05-28T06:26:18.263031+0000","last_became_active":"2024-05-28T06:24:24.008334+0000","last_became_peered":"2024-05-28T06:24:24.008334+0000","last_unstale":"2024-05-28T06:26:18.263031+0000","last_undegraded":"2024-05-28T06:26:18.263031+0000","last_fullsized":"2024-05-28T06:26:18.263031+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:10:50.477572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":297795584,"num_objects":71,"num_object_clones":0,"num_object_copies":142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":47,"num_read_kb":192512,"num_write":79,"num_write_kb":307200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.37","version":"132'80","reported_seq":149,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.239272+0000","last_change":"2024-05-28T06:24:24.018229+0000","last_active":"2024-05-28T06:26:18.239272+0000","last_peered":"2024-05-28T06:26:18.239272+0000","last_clean":"2024-05-28T06:26:18.239272+0000","last_became_active":"2024-05-28T06:24:24.017929+0000","last_became_peered":"2024-05-28T06:24:24.017929+0000","last_unstale":"2024-05-28T06:26:18.239272+0000","last_undegraded":"2024-05-28T06:26:18.239272+0000","last_fullsized":"2024-05-28T06:26:18.239272+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:58:16.573705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":297795716,"num_objects":72,"num_object_clones":0,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72,"num_whiteouts":0,"num_read":53,"num_read_kb":204803,"num_write":80,"num_write_kb":303106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.36","version":"132'63","reported_seq":137,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:25:42.637629+0000","last_change":"2024-05-28T06:24:23.995403+0000","last_active":"2024-05-28T06:25:42.637629+0000","last_peered":"2024-05-28T06:25:42.637629+0000","last_clean":"2024-05-28T06:25:42.637629+0000","last_became_active":"2024-05-28T06:24:23.994945+0000","last_became_peered":"2024-05-28T06:24:23.994945+0000","last_unstale":"2024-05-28T06:25:42.637629+0000","last_undegraded":"2024-05-28T06:25:42.637629+0000","last_fullsized":"2024-05-28T06:25:42.637629+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:11:29.809986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":255852544,"num_objects":61,"num_object_clones":0,"num_object_copies":122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":0,"num_read":58,"num_read_kb":237568,"num_write":63,"num_write_kb":253952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.35","version":"132'83","reported_seq":194,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.260079+0000","last_change":"2024-05-28T06:24:23.992865+0000","last_active":"2024-05-28T06:26:18.260079+0000","last_peered":"2024-05-28T06:26:18.260079+0000","last_clean":"2024-05-28T06:26:18.260079+0000","last_became_active":"2024-05-28T06:24:23.991405+0000","last_became_peered":"2024-05-28T06:24:23.991405+0000","last_unstale":"2024-05-28T06:26:18.260079+0000","last_undegraded":"2024-05-28T06:26:18.260079+0000","last_fullsized":"2024-05-28T06:26:18.260079+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:06:47.846653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":322961408,"num_objects":77,"num_object_clones":0,"num_object_copies":154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77,"num_whiteouts":0,"num_read":95,"num_read_kb":389120,"num_write":83,"num_write_kb":327680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.34","version":"132'77","reported_seq":185,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.283958+0000","last_change":"2024-05-28T06:24:24.018385+0000","last_active":"2024-05-28T06:26:18.283958+0000","last_peered":"2024-05-28T06:26:18.283958+0000","last_clean":"2024-05-28T06:26:18.283958+0000","last_became_active":"2024-05-28T06:24:24.017931+0000","last_became_peered":"2024-05-28T06:24:24.017931+0000","last_unstale":"2024-05-28T06:26:18.283958+0000","last_undegraded":"2024-05-28T06:26:18.283958+0000","last_fullsized":"2024-05-28T06:26:18.283958+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:11:21.508556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":289406976,"num_objects":69,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":0,"num_read":92,"num_read_kb":360452,"num_write":77,"num_write_kb":294913,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.33","version":"132'75","reported_seq":171,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.229026+0000","last_change":"2024-05-28T06:24:24.007399+0000","last_active":"2024-05-28T06:26:18.229026+0000","last_peered":"2024-05-28T06:26:18.229026+0000","last_clean":"2024-05-28T06:26:18.229026+0000","last_became_active":"2024-05-28T06:24:24.007121+0000","last_became_peered":"2024-05-28T06:24:24.007121+0000","last_unstale":"2024-05-28T06:26:18.229026+0000","last_undegraded":"2024-05-28T06:26:18.229026+0000","last_fullsized":"2024-05-28T06:26:18.229026+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:29:33.118245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":289406976,"num_objects":69,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":0,"num_read":80,"num_read_kb":327680,"num_write":75,"num_write_kb":294912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.32","version":"132'77","reported_seq":170,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.301921+0000","last_change":"2024-05-28T06:24:55.363255+0000","last_active":"2024-05-28T06:26:18.301921+0000","last_peered":"2024-05-28T06:26:18.301921+0000","last_clean":"2024-05-28T06:26:18.301921+0000","last_became_active":"2024-05-28T06:24:54.998097+0000","last_became_peered":"2024-05-28T06:24:54.998097+0000","last_unstale":"2024-05-28T06:26:18.301921+0000","last_undegraded":"2024-05-28T06:26:18.301921+0000","last_fullsized":"2024-05-28T06:26:18.301921+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:41:33.057222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":289406976,"num_objects":69,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":0,"num_read":62,"num_read_kb":253952,"num_write":77,"num_write_kb":299008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":16777216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.31","version":"132'70","reported_seq":116,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.292733+0000","last_change":"2024-05-28T06:24:24.020598+0000","last_active":"2024-05-28T06:26:18.292733+0000","last_peered":"2024-05-28T06:26:18.292733+0000","last_clean":"2024-05-28T06:26:18.292733+0000","last_became_active":"2024-05-28T06:24:24.020301+0000","last_became_peered":"2024-05-28T06:24:24.020301+0000","last_unstale":"2024-05-28T06:26:18.292733+0000","last_undegraded":"2024-05-28T06:26:18.292733+0000","last_fullsized":"2024-05-28T06:26:18.292733+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:18:18.796231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251658240,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":30,"num_read_kb":122880,"num_write":70,"num_write_kb":266240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.30","version":"132'81","reported_seq":148,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.299313+0000","last_change":"2024-05-28T06:24:23.995628+0000","last_active":"2024-05-28T06:26:18.299313+0000","last_peered":"2024-05-28T06:26:18.299313+0000","last_clean":"2024-05-28T06:26:18.299313+0000","last_became_active":"2024-05-28T06:24:23.995367+0000","last_became_peered":"2024-05-28T06:24:23.995367+0000","last_unstale":"2024-05-28T06:26:18.299313+0000","last_undegraded":"2024-05-28T06:26:18.299313+0000","last_fullsized":"2024-05-28T06:26:18.299313+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:07:49.435286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306184192,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":51,"num_read_kb":208896,"num_write":81,"num_write_kb":315392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2f","version":"132'65","reported_seq":123,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.267214+0000","last_change":"2024-05-28T06:24:23.987587+0000","last_active":"2024-05-28T06:26:18.267214+0000","last_peered":"2024-05-28T06:26:18.267214+0000","last_clean":"2024-05-28T06:26:18.267214+0000","last_became_active":"2024-05-28T06:24:23.986777+0000","last_became_peered":"2024-05-28T06:24:23.986777+0000","last_unstale":"2024-05-28T06:26:18.267214+0000","last_undegraded":"2024-05-28T06:26:18.267214+0000","last_fullsized":"2024-05-28T06:26:18.267214+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:06:00.030718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":230686720,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":42,"num_read_kb":172032,"num_write":65,"num_write_kb":245760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.2e","version":"132'82","reported_seq":185,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.245601+0000","last_change":"2024-05-28T06:24:23.998500+0000","last_active":"2024-05-28T06:26:18.245601+0000","last_peered":"2024-05-28T06:26:18.245601+0000","last_clean":"2024-05-28T06:26:18.245601+0000","last_became_active":"2024-05-28T06:24:23.998105+0000","last_became_peered":"2024-05-28T06:24:23.998105+0000","last_unstale":"2024-05-28T06:26:18.245601+0000","last_undegraded":"2024-05-28T06:26:18.245601+0000","last_fullsized":"2024-05-28T06:26:18.245601+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:47:31.291023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":310378496,"num_objects":74,"num_object_clones":0,"num_object_copies":148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74,"num_whiteouts":0,"num_read":87,"num_read_kb":356352,"num_write":82,"num_write_kb":315393,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2d","version":"132'83","reported_seq":167,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.323316+0000","last_change":"2024-05-28T06:24:23.997899+0000","last_active":"2024-05-28T06:26:18.323316+0000","last_peered":"2024-05-28T06:26:18.323316+0000","last_clean":"2024-05-28T06:26:18.323316+0000","last_became_active":"2024-05-28T06:24:23.997362+0000","last_became_peered":"2024-05-28T06:24:23.997362+0000","last_unstale":"2024-05-28T06:26:18.323316+0000","last_undegraded":"2024-05-28T06:26:18.323316+0000","last_fullsized":"2024-05-28T06:26:18.323316+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:48:26.250684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":310378496,"num_objects":74,"num_object_clones":0,"num_object_copies":148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74,"num_whiteouts":0,"num_read":67,"num_read_kb":274432,"num_write":82,"num_write_kb":319488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2c","version":"132'70","reported_seq":144,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.318303+0000","last_change":"2024-05-28T06:24:23.996440+0000","last_active":"2024-05-28T06:26:18.318303+0000","last_peered":"2024-05-28T06:26:18.318303+0000","last_clean":"2024-05-28T06:26:18.318303+0000","last_became_active":"2024-05-28T06:24:23.996067+0000","last_became_peered":"2024-05-28T06:24:23.996067+0000","last_unstale":"2024-05-28T06:26:18.318303+0000","last_undegraded":"2024-05-28T06:26:18.318303+0000","last_fullsized":"2024-05-28T06:26:18.318303+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:06:00.030718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":272629760,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":57,"num_read_kb":233472,"num_write":69,"num_write_kb":274432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2b","version":"132'82","reported_seq":190,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.301761+0000","last_change":"2024-05-28T06:24:24.018537+0000","last_active":"2024-05-28T06:26:18.301761+0000","last_peered":"2024-05-28T06:26:18.301761+0000","last_clean":"2024-05-28T06:26:18.301761+0000","last_became_active":"2024-05-28T06:24:24.018253+0000","last_became_peered":"2024-05-28T06:24:24.018253+0000","last_unstale":"2024-05-28T06:26:18.301761+0000","last_undegraded":"2024-05-28T06:26:18.301761+0000","last_fullsized":"2024-05-28T06:26:18.301761+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:13:05.023858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":293601280,"num_objects":70,"num_object_clones":0,"num_object_copies":140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70,"num_whiteouts":0,"num_read":91,"num_read_kb":364546,"num_write":82,"num_write_kb":303106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2a","version":"132'75","reported_seq":147,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.271529+0000","last_change":"2024-05-28T06:24:23.994663+0000","last_active":"2024-05-28T06:26:18.271529+0000","last_peered":"2024-05-28T06:26:18.271529+0000","last_clean":"2024-05-28T06:26:18.271529+0000","last_became_active":"2024-05-28T06:24:23.994410+0000","last_became_peered":"2024-05-28T06:24:23.994410+0000","last_unstale":"2024-05-28T06:26:18.271529+0000","last_undegraded":"2024-05-28T06:26:18.271529+0000","last_fullsized":"2024-05-28T06:26:18.271529+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:05:58.537398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":285212672,"num_objects":68,"num_object_clones":0,"num_object_copies":136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68,"num_whiteouts":0,"num_read":55,"num_read_kb":217090,"num_write":74,"num_write_kb":286721,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.29","version":"132'81","reported_seq":149,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.292813+0000","last_change":"2024-05-28T06:24:24.015438+0000","last_active":"2024-05-28T06:26:18.292813+0000","last_peered":"2024-05-28T06:26:18.292813+0000","last_clean":"2024-05-28T06:26:18.292813+0000","last_became_active":"2024-05-28T06:24:24.015124+0000","last_became_peered":"2024-05-28T06:24:24.015124+0000","last_unstale":"2024-05-28T06:26:18.292813+0000","last_undegraded":"2024-05-28T06:26:18.292813+0000","last_fullsized":"2024-05-28T06:26:18.292813+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:40:05.545358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281018368,"num_objects":67,"num_object_clones":0,"num_object_copies":134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":0,"num_read":51,"num_read_kb":208896,"num_write":81,"num_write_kb":303104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.28","version":"132'75","reported_seq":143,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.270972+0000","last_change":"2024-05-28T06:24:24.017147+0000","last_active":"2024-05-28T06:26:18.270972+0000","last_peered":"2024-05-28T06:26:18.270972+0000","last_clean":"2024-05-28T06:26:18.270972+0000","last_became_active":"2024-05-28T06:24:24.016692+0000","last_became_peered":"2024-05-28T06:24:24.016692+0000","last_unstale":"2024-05-28T06:26:18.270972+0000","last_undegraded":"2024-05-28T06:26:18.270972+0000","last_fullsized":"2024-05-28T06:26:18.270972+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:10:49.969260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281018368,"num_objects":67,"num_object_clones":0,"num_object_copies":134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":0,"num_read":51,"num_read_kb":208896,"num_write":75,"num_write_kb":286721,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.27","version":"132'81","reported_seq":161,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.270839+0000","last_change":"2024-05-28T06:24:24.007398+0000","last_active":"2024-05-28T06:26:18.270839+0000","last_peered":"2024-05-28T06:26:18.270839+0000","last_clean":"2024-05-28T06:26:18.270839+0000","last_became_active":"2024-05-28T06:24:24.006983+0000","last_became_peered":"2024-05-28T06:24:24.006983+0000","last_unstale":"2024-05-28T06:26:18.270839+0000","last_undegraded":"2024-05-28T06:26:18.270839+0000","last_fullsized":"2024-05-28T06:26:18.270839+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:02:59.158681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306184192,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":63,"num_read_kb":249858,"num_write":81,"num_write_kb":307202,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.26","version":"132'83","reported_seq":162,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.291454+0000","last_change":"2024-05-28T06:24:23.999225+0000","last_active":"2024-05-28T06:26:18.291454+0000","last_peered":"2024-05-28T06:26:18.291454+0000","last_clean":"2024-05-28T06:26:18.291454+0000","last_became_active":"2024-05-28T06:24:23.996859+0000","last_became_peered":"2024-05-28T06:24:23.996859+0000","last_unstale":"2024-05-28T06:26:18.291454+0000","last_undegraded":"2024-05-28T06:26:18.291454+0000","last_fullsized":"2024-05-28T06:26:18.291454+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:05:09.592919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":310378496,"num_objects":74,"num_object_clones":0,"num_object_copies":148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74,"num_whiteouts":0,"num_read":62,"num_read_kb":253952,"num_write":82,"num_write_kb":319488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.25","version":"132'73","reported_seq":156,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.305983+0000","last_change":"2024-05-28T06:24:24.007463+0000","last_active":"2024-05-28T06:26:18.305983+0000","last_peered":"2024-05-28T06:26:18.305983+0000","last_clean":"2024-05-28T06:26:18.305983+0000","last_became_active":"2024-05-28T06:24:24.007194+0000","last_became_peered":"2024-05-28T06:24:24.007194+0000","last_unstale":"2024-05-28T06:26:18.305983+0000","last_undegraded":"2024-05-28T06:26:18.305983+0000","last_fullsized":"2024-05-28T06:26:18.305983+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:27:47.069088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":272629760,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":66,"num_read_kb":258051,"num_write":73,"num_write_kb":278529,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.24","version":"132'77","reported_seq":128,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.306695+0000","last_change":"2024-05-28T06:24:24.016291+0000","last_active":"2024-05-28T06:26:18.306695+0000","last_peered":"2024-05-28T06:26:18.306695+0000","last_clean":"2024-05-28T06:26:18.306695+0000","last_became_active":"2024-05-28T06:24:24.015990+0000","last_became_peered":"2024-05-28T06:24:24.015990+0000","last_unstale":"2024-05-28T06:26:18.306695+0000","last_undegraded":"2024-05-28T06:26:18.306695+0000","last_fullsized":"2024-05-28T06:26:18.306695+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:02:16.956326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281018368,"num_objects":67,"num_object_clones":0,"num_object_copies":134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":0,"num_read":34,"num_read_kb":131074,"num_write":77,"num_write_kb":290817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.23","version":"132'81","reported_seq":169,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.308583+0000","last_change":"2024-05-28T06:24:24.019908+0000","last_active":"2024-05-28T06:26:18.308583+0000","last_peered":"2024-05-28T06:26:18.308583+0000","last_clean":"2024-05-28T06:26:18.308583+0000","last_became_active":"2024-05-28T06:24:24.019497+0000","last_became_peered":"2024-05-28T06:24:24.019497+0000","last_unstale":"2024-05-28T06:26:18.308583+0000","last_undegraded":"2024-05-28T06:26:18.308583+0000","last_fullsized":"2024-05-28T06:26:18.308583+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:16:02.402760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":297795584,"num_objects":71,"num_object_clones":0,"num_object_copies":142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":71,"num_read_kb":274436,"num_write":81,"num_write_kb":307201,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.22","version":"132'85","reported_seq":164,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.308511+0000","last_change":"2024-05-28T06:24:24.019852+0000","last_active":"2024-05-28T06:26:18.308511+0000","last_peered":"2024-05-28T06:26:18.308511+0000","last_clean":"2024-05-28T06:26:18.308511+0000","last_became_active":"2024-05-28T06:24:24.019462+0000","last_became_peered":"2024-05-28T06:24:24.019462+0000","last_unstale":"2024-05-28T06:26:18.308511+0000","last_undegraded":"2024-05-28T06:26:18.308511+0000","last_fullsized":"2024-05-28T06:26:18.308511+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:52:19.161622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":314572800,"num_objects":75,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75,"num_whiteouts":0,"num_read":62,"num_read_kb":253952,"num_write":85,"num_write_kb":327680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.21","version":"132'66","reported_seq":172,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:25:43.030729+0000","last_change":"2024-05-28T06:24:24.011632+0000","last_active":"2024-05-28T06:25:43.030729+0000","last_peered":"2024-05-28T06:25:43.030729+0000","last_clean":"2024-05-28T06:25:43.030729+0000","last_became_active":"2024-05-28T06:24:24.011246+0000","last_became_peered":"2024-05-28T06:24:24.011246+0000","last_unstale":"2024-05-28T06:25:43.030729+0000","last_undegraded":"2024-05-28T06:25:43.030729+0000","last_fullsized":"2024-05-28T06:25:43.030729+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:05:09.592919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":276824064,"num_objects":66,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":89,"num_read_kb":364544,"num_write":66,"num_write_kb":270336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.20","version":"132'75","reported_seq":163,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.316590+0000","last_change":"2024-05-28T06:24:23.991803+0000","last_active":"2024-05-28T06:26:18.316590+0000","last_peered":"2024-05-28T06:26:18.316590+0000","last_clean":"2024-05-28T06:26:18.316590+0000","last_became_active":"2024-05-28T06:24:23.991245+0000","last_became_peered":"2024-05-28T06:24:23.991245+0000","last_unstale":"2024-05-28T06:26:18.316590+0000","last_undegraded":"2024-05-28T06:26:18.316590+0000","last_fullsized":"2024-05-28T06:26:18.316590+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:55:45.102561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":272629760,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":71,"num_read_kb":282626,"num_write":75,"num_write_kb":282625,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1e","version":"132'68","reported_seq":127,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.309071+0000","last_change":"2024-05-28T06:24:24.018032+0000","last_active":"2024-05-28T06:26:18.309071+0000","last_peered":"2024-05-28T06:26:18.309071+0000","last_clean":"2024-05-28T06:26:18.309071+0000","last_became_active":"2024-05-28T06:24:24.017727+0000","last_became_peered":"2024-05-28T06:24:24.017727+0000","last_unstale":"2024-05-28T06:26:18.309071+0000","last_undegraded":"2024-05-28T06:26:18.309071+0000","last_fullsized":"2024-05-28T06:26:18.309071+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:13:05.023858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260046848,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":42,"num_read_kb":172032,"num_write":68,"num_write_kb":266240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1f","version":"129'4","reported_seq":23,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:55.134481+0000","last_change":"2024-05-28T06:24:23.006418+0000","last_active":"2024-05-28T06:24:55.134481+0000","last_peered":"2024-05-28T06:24:55.134481+0000","last_clean":"2024-05-28T06:24:55.134481+0000","last_became_active":"2024-05-28T06:24:23.006043+0000","last_became_peered":"2024-05-28T06:24:23.006043+0000","last_unstale":"2024-05-28T06:24:55.134481+0000","last_undegraded":"2024-05-28T06:24:55.134481+0000","last_fullsized":"2024-05-28T06:24:55.134481+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:31:42.560242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.1f","version":"132'82","reported_seq":167,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.285593+0000","last_change":"2024-05-28T06:24:24.008420+0000","last_active":"2024-05-28T06:26:18.285593+0000","last_peered":"2024-05-28T06:26:18.285593+0000","last_clean":"2024-05-28T06:26:18.285593+0000","last_became_active":"2024-05-28T06:24:24.008078+0000","last_became_peered":"2024-05-28T06:24:24.008078+0000","last_unstale":"2024-05-28T06:26:18.285593+0000","last_undegraded":"2024-05-28T06:26:18.285593+0000","last_fullsized":"2024-05-28T06:26:18.285593+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:08:27.007107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":293601280,"num_objects":70,"num_object_clones":0,"num_object_copies":140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70,"num_whiteouts":0,"num_read":68,"num_read_kb":262148,"num_write":82,"num_write_kb":303106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1e","version":"129'1","reported_seq":20,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:55.454913+0000","last_change":"2024-05-28T06:24:22.995399+0000","last_active":"2024-05-28T06:24:55.454913+0000","last_peered":"2024-05-28T06:24:55.454913+0000","last_clean":"2024-05-28T06:24:55.454913+0000","last_became_active":"2024-05-28T06:24:22.994896+0000","last_became_peered":"2024-05-28T06:24:22.994896+0000","last_unstale":"2024-05-28T06:24:55.454913+0000","last_undegraded":"2024-05-28T06:24:55.454913+0000","last_fullsized":"2024-05-28T06:24:55.454913+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:06:41.660486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1c","version":"132'70","reported_seq":150,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.265732+0000","last_change":"2024-05-28T06:24:24.006543+0000","last_active":"2024-05-28T06:26:18.265732+0000","last_peered":"2024-05-28T06:26:18.265732+0000","last_clean":"2024-05-28T06:26:18.265732+0000","last_became_active":"2024-05-28T06:24:24.006258+0000","last_became_peered":"2024-05-28T06:24:24.006258+0000","last_unstale":"2024-05-28T06:26:18.265732+0000","last_undegraded":"2024-05-28T06:26:18.265732+0000","last_fullsized":"2024-05-28T06:26:18.265732+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:34:58.279352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":255852805,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":64,"num_read_kb":262144,"num_write":70,"num_write_kb":266241,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1d","version":"132'3","reported_seq":21,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:54.985140+0000","last_change":"2024-05-28T06:24:22.992163+0000","last_active":"2024-05-28T06:24:54.985140+0000","last_peered":"2024-05-28T06:24:54.985140+0000","last_clean":"2024-05-28T06:24:54.985140+0000","last_became_active":"2024-05-28T06:24:22.991407+0000","last_became_peered":"2024-05-28T06:24:22.991407+0000","last_unstale":"2024-05-28T06:24:54.985140+0000","last_undegraded":"2024-05-28T06:24:54.985140+0000","last_fullsized":"2024-05-28T06:24:54.985140+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:34:43.911764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1d","version":"132'89","reported_seq":192,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.314145+0000","last_change":"2024-05-28T06:24:24.013028+0000","last_active":"2024-05-28T06:26:18.314145+0000","last_peered":"2024-05-28T06:26:18.314145+0000","last_clean":"2024-05-28T06:26:18.314145+0000","last_became_active":"2024-05-28T06:24:24.010356+0000","last_became_peered":"2024-05-28T06:24:24.010356+0000","last_unstale":"2024-05-28T06:26:18.314145+0000","last_undegraded":"2024-05-28T06:26:18.314145+0000","last_fullsized":"2024-05-28T06:26:18.314145+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:46:38.879485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":297795584,"num_objects":71,"num_object_clones":0,"num_object_copies":142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":87,"num_read_kb":356352,"num_write":89,"num_write_kb":327680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1c","version":"129'2","reported_seq":20,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:54.994094+0000","last_change":"2024-05-28T06:24:22.989493+0000","last_active":"2024-05-28T06:24:54.994094+0000","last_peered":"2024-05-28T06:24:54.994094+0000","last_clean":"2024-05-28T06:24:54.994094+0000","last_became_active":"2024-05-28T06:24:22.989103+0000","last_became_peered":"2024-05-28T06:24:22.989103+0000","last_unstale":"2024-05-28T06:24:54.994094+0000","last_undegraded":"2024-05-28T06:24:54.994094+0000","last_fullsized":"2024-05-28T06:24:54.994094+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:28:25.855171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1a","version":"132'76","reported_seq":145,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.241565+0000","last_change":"2024-05-28T06:24:24.008297+0000","last_active":"2024-05-28T06:26:18.241565+0000","last_peered":"2024-05-28T06:26:18.241565+0000","last_clean":"2024-05-28T06:26:18.241565+0000","last_became_active":"2024-05-28T06:24:24.007987+0000","last_became_peered":"2024-05-28T06:24:24.007987+0000","last_unstale":"2024-05-28T06:26:18.241565+0000","last_undegraded":"2024-05-28T06:26:18.241565+0000","last_fullsized":"2024-05-28T06:26:18.241565+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:48:49.925554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301989888,"num_objects":72,"num_object_clones":0,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72,"num_whiteouts":0,"num_read":53,"num_read_kb":217088,"num_write":76,"num_write_kb":303104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1b","version":"129'1","reported_seq":19,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:54.995380+0000","last_change":"2024-05-28T06:24:22.987047+0000","last_active":"2024-05-28T06:24:54.995380+0000","last_peered":"2024-05-28T06:24:54.995380+0000","last_clean":"2024-05-28T06:24:54.995380+0000","last_became_active":"2024-05-28T06:24:22.985617+0000","last_became_peered":"2024-05-28T06:24:22.985617+0000","last_unstale":"2024-05-28T06:24:54.995380+0000","last_undegraded":"2024-05-28T06:24:54.995380+0000","last_fullsized":"2024-05-28T06:24:54.995380+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:59:04.673636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1b","version":"132'77","reported_seq":182,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.245202+0000","last_change":"2024-05-28T06:24:23.987152+0000","last_active":"2024-05-28T06:26:18.245202+0000","last_peered":"2024-05-28T06:26:18.245202+0000","last_clean":"2024-05-28T06:26:18.245202+0000","last_became_active":"2024-05-28T06:24:23.985828+0000","last_became_peered":"2024-05-28T06:24:23.985828+0000","last_unstale":"2024-05-28T06:26:18.245202+0000","last_undegraded":"2024-05-28T06:26:18.245202+0000","last_fullsized":"2024-05-28T06:26:18.245202+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:48:49.925554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306184192,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":89,"num_read_kb":364544,"num_write":77,"num_write_kb":307200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1a","version":"129'1","reported_seq":19,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:55.341429+0000","last_change":"2024-05-28T06:24:22.990085+0000","last_active":"2024-05-28T06:24:55.341429+0000","last_peered":"2024-05-28T06:24:55.341429+0000","last_clean":"2024-05-28T06:24:55.341429+0000","last_became_active":"2024-05-28T06:24:22.989801+0000","last_became_peered":"2024-05-28T06:24:22.989801+0000","last_unstale":"2024-05-28T06:24:55.341429+0000","last_undegraded":"2024-05-28T06:24:55.341429+0000","last_fullsized":"2024-05-28T06:24:55.341429+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:53:06.353639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.18","version":"132'72","reported_seq":155,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.299919+0000","last_change":"2024-05-28T06:24:24.003806+0000","last_active":"2024-05-28T06:26:18.299919+0000","last_peered":"2024-05-28T06:26:18.299919+0000","last_clean":"2024-05-28T06:26:18.299919+0000","last_became_active":"2024-05-28T06:24:24.003446+0000","last_became_peered":"2024-05-28T06:24:24.003446+0000","last_unstale":"2024-05-28T06:26:18.299919+0000","last_undegraded":"2024-05-28T06:26:18.299919+0000","last_fullsized":"2024-05-28T06:26:18.299919+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:05:52.238604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260046848,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":67,"num_read_kb":274432,"num_write":72,"num_write_kb":274432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":18,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:54.984156+0000","last_change":"2024-05-28T06:24:23.007861+0000","last_active":"2024-05-28T06:24:54.984156+0000","last_peered":"2024-05-28T06:24:54.984156+0000","last_clean":"2024-05-28T06:24:54.984156+0000","last_became_active":"2024-05-28T06:24:23.007494+0000","last_became_peered":"2024-05-28T06:24:23.007494+0000","last_unstale":"2024-05-28T06:24:54.984156+0000","last_undegraded":"2024-05-28T06:24:54.984156+0000","last_fullsized":"2024-05-28T06:24:54.984156+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:06:19.878458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.19","version":"132'72","reported_seq":150,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.195341+0000","last_change":"2024-05-28T06:24:23.993309+0000","last_active":"2024-05-28T06:26:18.195341+0000","last_peered":"2024-05-28T06:26:18.195341+0000","last_clean":"2024-05-28T06:26:18.195341+0000","last_became_active":"2024-05-28T06:24:23.987466+0000","last_became_peered":"2024-05-28T06:24:23.987466+0000","last_unstale":"2024-05-28T06:26:18.195341+0000","last_undegraded":"2024-05-28T06:26:18.195341+0000","last_fullsized":"2024-05-28T06:26:18.195341+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:07:16.445323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":276824064,"num_objects":66,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":62,"num_read_kb":245762,"num_write":72,"num_write_kb":278529,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":18,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:54.994422+0000","last_change":"2024-05-28T06:24:23.000407+0000","last_active":"2024-05-28T06:24:54.994422+0000","last_peered":"2024-05-28T06:24:54.994422+0000","last_clean":"2024-05-28T06:24:54.994422+0000","last_became_active":"2024-05-28T06:24:23.000037+0000","last_became_peered":"2024-05-28T06:24:23.000037+0000","last_unstale":"2024-05-28T06:24:54.994422+0000","last_undegraded":"2024-05-28T06:24:54.994422+0000","last_fullsized":"2024-05-28T06:24:54.994422+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:59:08.824871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"132'71","reported_seq":120,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.295941+0000","last_change":"2024-05-28T06:24:23.995548+0000","last_active":"2024-05-28T06:26:18.295941+0000","last_peered":"2024-05-28T06:26:18.295941+0000","last_clean":"2024-05-28T06:26:18.295941+0000","last_became_active":"2024-05-28T06:24:23.995204+0000","last_became_peered":"2024-05-28T06:24:23.995204+0000","last_unstale":"2024-05-28T06:26:18.295941+0000","last_undegraded":"2024-05-28T06:26:18.295941+0000","last_fullsized":"2024-05-28T06:26:18.295941+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:25:36.261157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260046848,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":33,"num_read_kb":135168,"num_write":70,"num_write_kb":270336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.17","version":"132'1","reported_seq":19,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:55.454932+0000","last_change":"2024-05-28T06:24:23.001114+0000","last_active":"2024-05-28T06:24:55.454932+0000","last_peered":"2024-05-28T06:24:55.454932+0000","last_clean":"2024-05-28T06:24:55.454932+0000","last_became_active":"2024-05-28T06:24:22.999720+0000","last_became_peered":"2024-05-28T06:24:22.999720+0000","last_unstale":"2024-05-28T06:24:55.454932+0000","last_undegraded":"2024-05-28T06:24:55.454932+0000","last_fullsized":"2024-05-28T06:24:55.454932+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:29:54.261773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.17","version":"132'63","reported_seq":117,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.234194+0000","last_change":"2024-05-28T06:24:24.016902+0000","last_active":"2024-05-28T06:26:18.234194+0000","last_peered":"2024-05-28T06:26:18.234194+0000","last_clean":"2024-05-28T06:26:18.234194+0000","last_became_active":"2024-05-28T06:24:24.016597+0000","last_became_peered":"2024-05-28T06:24:24.016597+0000","last_unstale":"2024-05-28T06:26:18.234194+0000","last_undegraded":"2024-05-28T06:26:18.234194+0000","last_fullsized":"2024-05-28T06:26:18.234194+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:08:42.734914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247463936,"num_objects":59,"num_object_clones":0,"num_object_copies":118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":0,"num_read":38,"num_read_kb":155648,"num_write":63,"num_write_kb":249856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":18,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:55.342025+0000","last_change":"2024-05-28T06:24:23.001188+0000","last_active":"2024-05-28T06:24:55.342025+0000","last_peered":"2024-05-28T06:24:55.342025+0000","last_clean":"2024-05-28T06:24:55.342025+0000","last_became_active":"2024-05-28T06:24:23.000878+0000","last_became_peered":"2024-05-28T06:24:23.000878+0000","last_unstale":"2024-05-28T06:24:55.342025+0000","last_undegraded":"2024-05-28T06:24:55.342025+0000","last_fullsized":"2024-05-28T06:24:55.342025+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:18:06.253798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.14","version":"132'75","reported_seq":166,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.313193+0000","last_change":"2024-05-28T06:24:24.011311+0000","last_active":"2024-05-28T06:26:18.313193+0000","last_peered":"2024-05-28T06:26:18.313193+0000","last_clean":"2024-05-28T06:26:18.313193+0000","last_became_active":"2024-05-28T06:24:24.010110+0000","last_became_peered":"2024-05-28T06:24:24.010110+0000","last_unstale":"2024-05-28T06:26:18.313193+0000","last_undegraded":"2024-05-28T06:26:18.313193+0000","last_fullsized":"2024-05-28T06:26:18.313193+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:46:31.105891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251658240,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":75,"num_read_kb":307200,"num_write":74,"num_write_kb":274432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.15","version":"132'24","reported_seq":42,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.184203+0000","last_change":"2024-05-28T06:24:22.999088+0000","last_active":"2024-05-28T06:26:18.184203+0000","last_peered":"2024-05-28T06:26:18.184203+0000","last_clean":"2024-05-28T06:26:18.184203+0000","last_became_active":"2024-05-28T06:24:22.998379+0000","last_became_peered":"2024-05-28T06:24:22.998379+0000","last_unstale":"2024-05-28T06:26:18.184203+0000","last_undegraded":"2024-05-28T06:26:18.184203+0000","last_fullsized":"2024-05-28T06:26:18.184203+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:59:19.637147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.15","version":"132'68","reported_seq":144,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.268436+0000","last_change":"2024-05-28T06:24:24.020278+0000","last_active":"2024-05-28T06:26:18.268436+0000","last_peered":"2024-05-28T06:26:18.268436+0000","last_clean":"2024-05-28T06:26:18.268436+0000","last_became_active":"2024-05-28T06:24:24.019973+0000","last_became_peered":"2024-05-28T06:24:24.019973+0000","last_unstale":"2024-05-28T06:26:18.268436+0000","last_undegraded":"2024-05-28T06:26:18.268436+0000","last_fullsized":"2024-05-28T06:26:18.268436+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:46:47.603731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":230686720,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":60,"num_read_kb":229380,"num_write":67,"num_write_kb":241666,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.14","version":"132'416","reported_seq":434,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:23.552552+0000","last_change":"2024-05-28T06:24:23.005145+0000","last_active":"2024-05-28T06:26:23.552552+0000","last_peered":"2024-05-28T06:26:23.552552+0000","last_clean":"2024-05-28T06:26:23.552552+0000","last_became_active":"2024-05-28T06:24:23.004831+0000","last_became_peered":"2024-05-28T06:24:23.004831+0000","last_unstale":"2024-05-28T06:26:23.552552+0000","last_undegraded":"2024-05-28T06:26:23.552552+0000","last_fullsized":"2024-05-28T06:26:23.552552+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":416,"log_dups_size":0,"ondisk_log_size":416,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:09:55.668305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1673237,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":415,"num_write_kb":1791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.13","version":"132'75","reported_seq":141,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:25:42.876820+0000","last_change":"2024-05-28T06:24:24.002061+0000","last_active":"2024-05-28T06:25:42.876820+0000","last_peered":"2024-05-28T06:25:42.876820+0000","last_clean":"2024-05-28T06:25:42.876820+0000","last_became_active":"2024-05-28T06:24:24.001429+0000","last_became_peered":"2024-05-28T06:24:24.001429+0000","last_unstale":"2024-05-28T06:25:42.876820+0000","last_undegraded":"2024-05-28T06:25:42.876820+0000","last_fullsized":"2024-05-28T06:25:42.876820+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:39:41.026878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":297795584,"num_objects":71,"num_object_clones":0,"num_object_copies":142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":50,"num_read_kb":196610,"num_write":75,"num_write_kb":294913,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":18,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:54.992958+0000","last_change":"2024-05-28T06:24:22.993624+0000","last_active":"2024-05-28T06:24:54.992958+0000","last_peered":"2024-05-28T06:24:54.992958+0000","last_clean":"2024-05-28T06:24:54.992958+0000","last_became_active":"2024-05-28T06:24:22.993301+0000","last_became_peered":"2024-05-28T06:24:22.993301+0000","last_unstale":"2024-05-28T06:24:54.992958+0000","last_undegraded":"2024-05-28T06:24:54.992958+0000","last_fullsized":"2024-05-28T06:24:54.992958+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:26:44.582201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.10","version":"132'69","reported_seq":142,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.196154+0000","last_change":"2024-05-28T06:24:24.005361+0000","last_active":"2024-05-28T06:26:18.196154+0000","last_peered":"2024-05-28T06:26:18.196154+0000","last_clean":"2024-05-28T06:26:18.196154+0000","last_became_active":"2024-05-28T06:24:24.005080+0000","last_became_peered":"2024-05-28T06:24:24.005080+0000","last_unstale":"2024-05-28T06:26:18.196154+0000","last_undegraded":"2024-05-28T06:26:18.196154+0000","last_fullsized":"2024-05-28T06:26:18.196154+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:09:03.314508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":272629760,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":57,"num_read_kb":233472,"num_write":69,"num_write_kb":274432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":18,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:54.994900+0000","last_change":"2024-05-28T06:24:23.005565+0000","last_active":"2024-05-28T06:24:54.994900+0000","last_peered":"2024-05-28T06:24:54.994900+0000","last_clean":"2024-05-28T06:24:54.994900+0000","last_became_active":"2024-05-28T06:24:23.005194+0000","last_became_peered":"2024-05-28T06:24:23.005194+0000","last_unstale":"2024-05-28T06:24:54.994900+0000","last_undegraded":"2024-05-28T06:24:54.994900+0000","last_fullsized":"2024-05-28T06:24:54.994900+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:26:55.784217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.11","version":"132'84","reported_seq":162,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.224702+0000","last_change":"2024-05-28T06:24:24.009724+0000","last_active":"2024-05-28T06:26:18.224702+0000","last_peered":"2024-05-28T06:26:18.224702+0000","last_clean":"2024-05-28T06:26:18.224702+0000","last_became_active":"2024-05-28T06:24:24.009163+0000","last_became_peered":"2024-05-28T06:24:24.009163+0000","last_unstale":"2024-05-28T06:26:18.224702+0000","last_undegraded":"2024-05-28T06:26:18.224702+0000","last_fullsized":"2024-05-28T06:26:18.224702+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:36:46.619746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":318767104,"num_objects":76,"num_object_clones":0,"num_object_copies":152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76,"num_whiteouts":0,"num_read":62,"num_read_kb":253952,"num_write":84,"num_write_kb":323585,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.10","version":"129'8","reported_seq":26,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:55.070167+0000","last_change":"2024-05-28T06:24:22.996927+0000","last_active":"2024-05-28T06:24:55.070167+0000","last_peered":"2024-05-28T06:24:55.070167+0000","last_clean":"2024-05-28T06:24:55.070167+0000","last_became_active":"2024-05-28T06:24:22.996644+0000","last_became_peered":"2024-05-28T06:24:22.996644+0000","last_unstale":"2024-05-28T06:24:55.070167+0000","last_undegraded":"2024-05-28T06:24:55.070167+0000","last_fullsized":"2024-05-28T06:24:55.070167+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:17:26.759155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.e","version":"132'58","reported_seq":98,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.315898+0000","last_change":"2024-05-28T06:24:24.009329+0000","last_active":"2024-05-28T06:26:18.315898+0000","last_peered":"2024-05-28T06:26:18.315898+0000","last_clean":"2024-05-28T06:26:18.315898+0000","last_became_active":"2024-05-28T06:24:24.009033+0000","last_became_peered":"2024-05-28T06:24:24.009033+0000","last_unstale":"2024-05-28T06:26:18.315898+0000","last_undegraded":"2024-05-28T06:26:18.315898+0000","last_fullsized":"2024-05-28T06:26:18.315898+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:23:27.401090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218103808,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":23,"num_read_kb":94208,"num_write":58,"num_write_kb":225280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.f","version":"129'1","reported_seq":20,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:55.456823+0000","last_change":"2024-05-28T06:24:23.006567+0000","last_active":"2024-05-28T06:24:55.456823+0000","last_peered":"2024-05-28T06:24:55.456823+0000","last_clean":"2024-05-28T06:24:55.456823+0000","last_became_active":"2024-05-28T06:24:23.005650+0000","last_became_peered":"2024-05-28T06:24:23.005650+0000","last_unstale":"2024-05-28T06:24:55.456823+0000","last_undegraded":"2024-05-28T06:24:55.456823+0000","last_fullsized":"2024-05-28T06:24:55.456823+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:01:42.283187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.f","version":"132'83","reported_seq":174,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.285828+0000","last_change":"2024-05-28T06:24:24.016763+0000","last_active":"2024-05-28T06:26:18.285828+0000","last_peered":"2024-05-28T06:26:18.285828+0000","last_clean":"2024-05-28T06:26:18.285828+0000","last_became_active":"2024-05-28T06:24:24.016255+0000","last_became_peered":"2024-05-28T06:24:24.016255+0000","last_unstale":"2024-05-28T06:26:18.285828+0000","last_undegraded":"2024-05-28T06:26:18.285828+0000","last_fullsized":"2024-05-28T06:26:18.285828+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:06:19.643800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306184192,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":74,"num_read_kb":303104,"num_write":83,"num_write_kb":315393,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.e","version":"129'2","reported_seq":21,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:54.985300+0000","last_change":"2024-05-28T06:24:23.002513+0000","last_active":"2024-05-28T06:24:54.985300+0000","last_peered":"2024-05-28T06:24:54.985300+0000","last_clean":"2024-05-28T06:24:54.985300+0000","last_became_active":"2024-05-28T06:24:23.001946+0000","last_became_peered":"2024-05-28T06:24:23.001946+0000","last_unstale":"2024-05-28T06:24:54.985300+0000","last_undegraded":"2024-05-28T06:24:54.985300+0000","last_fullsized":"2024-05-28T06:24:54.985300+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:27:48.979163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"132'66","reported_seq":143,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.288093+0000","last_change":"2024-05-28T06:24:23.998032+0000","last_active":"2024-05-28T06:26:18.288093+0000","last_peered":"2024-05-28T06:26:18.288093+0000","last_clean":"2024-05-28T06:26:18.288093+0000","last_became_active":"2024-05-28T06:24:23.996433+0000","last_became_peered":"2024-05-28T06:24:23.996433+0000","last_unstale":"2024-05-28T06:26:18.288093+0000","last_undegraded":"2024-05-28T06:26:18.288093+0000","last_fullsized":"2024-05-28T06:26:18.288093+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:47:25.671388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":230686900,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":60,"num_read_kb":221190,"num_write":66,"num_write_kb":233476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.d","version":"129'1","reported_seq":20,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:54.984968+0000","last_change":"2024-05-28T06:24:22.992776+0000","last_active":"2024-05-28T06:24:54.984968+0000","last_peered":"2024-05-28T06:24:54.984968+0000","last_clean":"2024-05-28T06:24:54.984968+0000","last_became_active":"2024-05-28T06:24:22.992147+0000","last_became_peered":"2024-05-28T06:24:22.992147+0000","last_unstale":"2024-05-28T06:24:54.984968+0000","last_undegraded":"2024-05-28T06:24:54.984968+0000","last_fullsized":"2024-05-28T06:24:54.984968+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:36:09.562056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.d","version":"132'79","reported_seq":168,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.322183+0000","last_change":"2024-05-28T06:24:23.989538+0000","last_active":"2024-05-28T06:26:18.322183+0000","last_peered":"2024-05-28T06:26:18.322183+0000","last_clean":"2024-05-28T06:26:18.322183+0000","last_became_active":"2024-05-28T06:24:23.988631+0000","last_became_peered":"2024-05-28T06:24:23.988631+0000","last_unstale":"2024-05-28T06:26:18.322183+0000","last_undegraded":"2024-05-28T06:26:18.322183+0000","last_fullsized":"2024-05-28T06:26:18.322183+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:08:42.734914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":272629760,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":72,"num_read_kb":294912,"num_write":79,"num_write_kb":294912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":19,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:55.454861+0000","last_change":"2024-05-28T06:24:22.997400+0000","last_active":"2024-05-28T06:24:55.454861+0000","last_peered":"2024-05-28T06:24:55.454861+0000","last_clean":"2024-05-28T06:24:55.454861+0000","last_became_active":"2024-05-28T06:24:22.993636+0000","last_became_peered":"2024-05-28T06:24:22.993636+0000","last_unstale":"2024-05-28T06:24:55.454861+0000","last_undegraded":"2024-05-28T06:24:55.454861+0000","last_fullsized":"2024-05-28T06:24:55.454861+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:43:01.373860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.a","version":"132'79","reported_seq":170,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.293566+0000","last_change":"2024-05-28T06:24:24.007538+0000","last_active":"2024-05-28T06:26:18.293566+0000","last_peered":"2024-05-28T06:26:18.293566+0000","last_clean":"2024-05-28T06:26:18.293566+0000","last_became_active":"2024-05-28T06:24:24.007259+0000","last_became_peered":"2024-05-28T06:24:24.007259+0000","last_unstale":"2024-05-28T06:26:18.293566+0000","last_undegraded":"2024-05-28T06:26:18.293566+0000","last_fullsized":"2024-05-28T06:26:18.293566+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:59:09.840788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306184192,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":74,"num_read_kb":303104,"num_write":79,"num_write_kb":311296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.b","version":"132'3","reported_seq":22,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:25:22.100610+0000","last_change":"2024-05-28T06:24:22.996819+0000","last_active":"2024-05-28T06:25:22.100610+0000","last_peered":"2024-05-28T06:25:22.100610+0000","last_clean":"2024-05-28T06:25:22.100610+0000","last_became_active":"2024-05-28T06:24:22.995489+0000","last_became_peered":"2024-05-28T06:24:22.995489+0000","last_unstale":"2024-05-28T06:25:22.100610+0000","last_undegraded":"2024-05-28T06:25:22.100610+0000","last_fullsized":"2024-05-28T06:25:22.100610+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:24:52.967145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.b","version":"132'90","reported_seq":171,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.295653+0000","last_change":"2024-05-28T06:24:24.015790+0000","last_active":"2024-05-28T06:26:18.295653+0000","last_peered":"2024-05-28T06:26:18.295653+0000","last_clean":"2024-05-28T06:26:18.295653+0000","last_became_active":"2024-05-28T06:24:24.015295+0000","last_became_peered":"2024-05-28T06:24:24.015295+0000","last_unstale":"2024-05-28T06:26:18.295653+0000","last_undegraded":"2024-05-28T06:26:18.295653+0000","last_fullsized":"2024-05-28T06:26:18.295653+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:37:59.260972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":314572932,"num_objects":76,"num_object_clones":0,"num_object_copies":152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76,"num_whiteouts":0,"num_read":64,"num_read_kb":237574,"num_write":90,"num_write_kb":323588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":19,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:55.341579+0000","last_change":"2024-05-28T06:24:22.995343+0000","last_active":"2024-05-28T06:24:55.341579+0000","last_peered":"2024-05-28T06:24:55.341579+0000","last_clean":"2024-05-28T06:24:55.341579+0000","last_became_active":"2024-05-28T06:24:22.994227+0000","last_became_peered":"2024-05-28T06:24:22.994227+0000","last_unstale":"2024-05-28T06:24:55.341579+0000","last_undegraded":"2024-05-28T06:24:55.341579+0000","last_fullsized":"2024-05-28T06:24:55.341579+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:50:39.661313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.8","version":"132'88","reported_seq":146,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.296033+0000","last_change":"2024-05-28T06:24:23.987172+0000","last_active":"2024-05-28T06:26:18.296033+0000","last_peered":"2024-05-28T06:26:18.296033+0000","last_clean":"2024-05-28T06:26:18.296033+0000","last_became_active":"2024-05-28T06:24:23.986897+0000","last_became_peered":"2024-05-28T06:24:23.986897+0000","last_unstale":"2024-05-28T06:26:18.296033+0000","last_undegraded":"2024-05-28T06:26:18.296033+0000","last_fullsized":"2024-05-28T06:26:18.296033+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:11:18.017207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":343932928,"num_objects":82,"num_object_clones":0,"num_object_copies":164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82,"num_whiteouts":0,"num_read":41,"num_read_kb":167936,"num_write":88,"num_write_kb":348160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":19,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:55.341333+0000","last_change":"2024-05-28T06:24:22.991355+0000","last_active":"2024-05-28T06:24:55.341333+0000","last_peered":"2024-05-28T06:24:55.341333+0000","last_clean":"2024-05-28T06:24:55.341333+0000","last_became_active":"2024-05-28T06:24:22.991078+0000","last_became_peered":"2024-05-28T06:24:22.991078+0000","last_unstale":"2024-05-28T06:24:55.341333+0000","last_undegraded":"2024-05-28T06:24:55.341333+0000","last_fullsized":"2024-05-28T06:24:55.341333+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:06:19.878458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"132'68","reported_seq":141,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.282953+0000","last_change":"2024-05-28T06:24:24.009275+0000","last_active":"2024-05-28T06:26:18.282953+0000","last_peered":"2024-05-28T06:26:18.282953+0000","last_clean":"2024-05-28T06:26:18.282953+0000","last_became_active":"2024-05-28T06:24:24.008955+0000","last_became_peered":"2024-05-28T06:24:24.008955+0000","last_unstale":"2024-05-28T06:26:18.282953+0000","last_undegraded":"2024-05-28T06:26:18.282953+0000","last_fullsized":"2024-05-28T06:26:18.282953+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:32:09.823976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":276824064,"num_objects":66,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":56,"num_read_kb":229376,"num_write":68,"num_write_kb":274432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":19,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:54.984300+0000","last_change":"2024-05-28T06:24:23.002572+0000","last_active":"2024-05-28T06:24:54.984300+0000","last_peered":"2024-05-28T06:24:54.984300+0000","last_clean":"2024-05-28T06:24:54.984300+0000","last_became_active":"2024-05-28T06:24:23.002292+0000","last_became_peered":"2024-05-28T06:24:23.002292+0000","last_unstale":"2024-05-28T06:24:54.984300+0000","last_undegraded":"2024-05-28T06:24:54.984300+0000","last_fullsized":"2024-05-28T06:24:54.984300+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:50:39.856735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.6","version":"132'87","reported_seq":139,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.318611+0000","last_change":"2024-05-28T06:24:24.017529+0000","last_active":"2024-05-28T06:26:18.318611+0000","last_peered":"2024-05-28T06:26:18.318611+0000","last_clean":"2024-05-28T06:26:18.318611+0000","last_became_active":"2024-05-28T06:24:24.017221+0000","last_became_peered":"2024-05-28T06:24:24.017221+0000","last_unstale":"2024-05-28T06:26:18.318611+0000","last_undegraded":"2024-05-28T06:26:18.318611+0000","last_fullsized":"2024-05-28T06:26:18.318611+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:32:09.823976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306184192,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":35,"num_read_kb":143360,"num_write":87,"num_write_kb":327680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.7","version":"129'2","reported_seq":21,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:54.994004+0000","last_change":"2024-05-28T06:24:23.003960+0000","last_active":"2024-05-28T06:24:54.994004+0000","last_peered":"2024-05-28T06:24:54.994004+0000","last_clean":"2024-05-28T06:24:54.994004+0000","last_became_active":"2024-05-28T06:24:23.003502+0000","last_became_peered":"2024-05-28T06:24:23.003502+0000","last_unstale":"2024-05-28T06:24:54.994004+0000","last_undegraded":"2024-05-28T06:24:54.994004+0000","last_fullsized":"2024-05-28T06:24:54.994004+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:13:19.450560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.7","version":"132'84","reported_seq":171,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.313436+0000","last_change":"2024-05-28T06:24:24.020448+0000","last_active":"2024-05-28T06:26:18.313436+0000","last_peered":"2024-05-28T06:26:18.313436+0000","last_clean":"2024-05-28T06:26:18.313436+0000","last_became_active":"2024-05-28T06:24:24.019730+0000","last_became_peered":"2024-05-28T06:24:24.019730+0000","last_unstale":"2024-05-28T06:26:18.313436+0000","last_undegraded":"2024-05-28T06:26:18.313436+0000","last_fullsized":"2024-05-28T06:26:18.313436+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:26:56.800134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":310378496,"num_objects":74,"num_object_clones":0,"num_object_copies":148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74,"num_whiteouts":0,"num_read":70,"num_read_kb":286720,"num_write":84,"num_write_kb":323584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.6","version":"132'41","reported_seq":60,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:21.025193+0000","last_change":"2024-05-28T06:24:23.003133+0000","last_active":"2024-05-28T06:26:21.025193+0000","last_peered":"2024-05-28T06:26:21.025193+0000","last_clean":"2024-05-28T06:26:21.025193+0000","last_became_active":"2024-05-28T06:24:23.002821+0000","last_became_peered":"2024-05-28T06:24:23.002821+0000","last_unstale":"2024-05-28T06:26:21.025193+0000","last_undegraded":"2024-05-28T06:26:21.025193+0000","last_fullsized":"2024-05-28T06:26:21.025193+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:18:06.253798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":95,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.4","version":"132'80","reported_seq":184,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.322862+0000","last_change":"2024-05-28T06:24:24.004928+0000","last_active":"2024-05-28T06:26:18.322862+0000","last_peered":"2024-05-28T06:26:18.322862+0000","last_clean":"2024-05-28T06:26:18.322862+0000","last_became_active":"2024-05-28T06:24:24.004599+0000","last_became_peered":"2024-05-28T06:24:24.004599+0000","last_unstale":"2024-05-28T06:26:18.322862+0000","last_undegraded":"2024-05-28T06:26:18.322862+0000","last_fullsized":"2024-05-28T06:26:18.322862+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:09:56.044033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":276824064,"num_objects":66,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":87,"num_read_kb":356352,"num_write":80,"num_write_kb":294913,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":19,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:54.996483+0000","last_change":"2024-05-28T06:24:23.003603+0000","last_active":"2024-05-28T06:24:54.996483+0000","last_peered":"2024-05-28T06:24:54.996483+0000","last_clean":"2024-05-28T06:24:54.996483+0000","last_became_active":"2024-05-28T06:24:23.002468+0000","last_became_peered":"2024-05-28T06:24:23.002468+0000","last_unstale":"2024-05-28T06:24:54.996483+0000","last_undegraded":"2024-05-28T06:24:54.996483+0000","last_fullsized":"2024-05-28T06:24:54.996483+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:48:27.218311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.5","version":"132'86","reported_seq":149,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.318905+0000","last_change":"2024-05-28T06:24:23.999563+0000","last_active":"2024-05-28T06:26:18.318905+0000","last_peered":"2024-05-28T06:26:18.318905+0000","last_clean":"2024-05-28T06:26:18.318905+0000","last_became_active":"2024-05-28T06:24:23.999013+0000","last_became_peered":"2024-05-28T06:24:23.999013+0000","last_unstale":"2024-05-28T06:26:18.318905+0000","last_undegraded":"2024-05-28T06:26:18.318905+0000","last_fullsized":"2024-05-28T06:26:18.318905+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:47:25.671388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":285213316,"num_objects":72,"num_object_clones":1,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72,"num_whiteouts":1,"num_read":47,"num_read_kb":192512,"num_write":84,"num_write_kb":303107,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":19,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:54.985023+0000","last_change":"2024-05-28T06:24:22.986876+0000","last_active":"2024-05-28T06:24:54.985023+0000","last_peered":"2024-05-28T06:24:54.985023+0000","last_clean":"2024-05-28T06:24:54.985023+0000","last_became_active":"2024-05-28T06:24:22.986620+0000","last_became_peered":"2024-05-28T06:24:22.986620+0000","last_unstale":"2024-05-28T06:24:54.985023+0000","last_undegraded":"2024-05-28T06:24:54.985023+0000","last_fullsized":"2024-05-28T06:24:54.985023+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:20:23.711600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2","version":"132'90","reported_seq":159,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.297667+0000","last_change":"2024-05-28T06:24:24.012293+0000","last_active":"2024-05-28T06:26:18.297667+0000","last_peered":"2024-05-28T06:26:18.297667+0000","last_clean":"2024-05-28T06:26:18.297667+0000","last_became_active":"2024-05-28T06:24:24.011989+0000","last_became_peered":"2024-05-28T06:24:24.011989+0000","last_unstale":"2024-05-28T06:26:18.297667+0000","last_undegraded":"2024-05-28T06:26:18.297667+0000","last_fullsized":"2024-05-28T06:26:18.297667+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:47:31.291023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":352321536,"num_objects":84,"num_object_clones":0,"num_object_copies":168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84,"num_whiteouts":0,"num_read":52,"num_read_kb":212992,"num_write":90,"num_write_kb":356352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":19,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:54.995987+0000","last_change":"2024-05-28T06:24:23.008363+0000","last_active":"2024-05-28T06:24:54.995987+0000","last_peered":"2024-05-28T06:24:54.995987+0000","last_clean":"2024-05-28T06:24:54.995987+0000","last_became_active":"2024-05-28T06:24:23.007859+0000","last_became_peered":"2024-05-28T06:24:23.007859+0000","last_unstale":"2024-05-28T06:24:54.995987+0000","last_undegraded":"2024-05-28T06:24:54.995987+0000","last_fullsized":"2024-05-28T06:24:54.995987+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:45:32.156309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"132'98","reported_seq":198,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.323247+0000","last_change":"2024-05-28T06:24:24.013906+0000","last_active":"2024-05-28T06:26:18.323247+0000","last_peered":"2024-05-28T06:26:18.323247+0000","last_clean":"2024-05-28T06:26:18.323247+0000","last_became_active":"2024-05-28T06:24:24.011269+0000","last_became_peered":"2024-05-28T06:24:24.011269+0000","last_unstale":"2024-05-28T06:26:18.323247+0000","last_undegraded":"2024-05-28T06:26:18.323247+0000","last_fullsized":"2024-05-28T06:26:18.323247+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:09:03.314508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369098752,"num_objects":88,"num_object_clones":0,"num_object_copies":176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88,"num_whiteouts":0,"num_read":83,"num_read_kb":327683,"num_write":98,"num_write_kb":376833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":19,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:55.133578+0000","last_change":"2024-05-28T06:24:23.006728+0000","last_active":"2024-05-28T06:24:55.133578+0000","last_peered":"2024-05-28T06:24:55.133578+0000","last_clean":"2024-05-28T06:24:55.133578+0000","last_became_active":"2024-05-28T06:24:23.005895+0000","last_became_peered":"2024-05-28T06:24:23.005895+0000","last_unstale":"2024-05-28T06:24:55.133578+0000","last_undegraded":"2024-05-28T06:24:55.133578+0000","last_fullsized":"2024-05-28T06:24:55.133578+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:53:06.353639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"132'87","reported_seq":175,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.310121+0000","last_change":"2024-05-28T06:24:24.013072+0000","last_active":"2024-05-28T06:26:18.310121+0000","last_peered":"2024-05-28T06:26:18.310121+0000","last_clean":"2024-05-28T06:26:18.310121+0000","last_became_active":"2024-05-28T06:24:24.012814+0000","last_became_peered":"2024-05-28T06:24:24.012814+0000","last_unstale":"2024-05-28T06:26:18.310121+0000","last_undegraded":"2024-05-28T06:26:18.310121+0000","last_fullsized":"2024-05-28T06:26:18.310121+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:32:09.823976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":339738624,"num_objects":81,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81,"num_whiteouts":0,"num_read":71,"num_read_kb":290816,"num_write":87,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1","version":"132'31","reported_seq":80,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.171426+0000","last_change":"2024-05-28T06:24:23.001199+0000","last_active":"2024-05-28T06:26:18.171426+0000","last_peered":"2024-05-28T06:26:18.171426+0000","last_clean":"2024-05-28T06:26:18.171426+0000","last_became_active":"2024-05-28T06:24:23.000861+0000","last_became_peered":"2024-05-28T06:24:23.000861+0000","last_unstale":"2024-05-28T06:26:18.171426+0000","last_undegraded":"2024-05-28T06:26:18.171426+0000","last_fullsized":"2024-05-28T06:26:18.171426+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:36:09.562056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4553,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":30,"num_read_kb":30,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1","version":"132'93","reported_seq":186,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.322559+0000","last_change":"2024-05-28T06:24:24.005940+0000","last_active":"2024-05-28T06:26:18.322559+0000","last_peered":"2024-05-28T06:26:18.322559+0000","last_clean":"2024-05-28T06:26:18.322559+0000","last_became_active":"2024-05-28T06:24:24.005485+0000","last_became_peered":"2024-05-28T06:24:24.005485+0000","last_unstale":"2024-05-28T06:26:18.322559+0000","last_undegraded":"2024-05-28T06:26:18.322559+0000","last_fullsized":"2024-05-28T06:26:18.322559+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:52:44.413787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":314572800,"num_objects":75,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75,"num_whiteouts":0,"num_read":76,"num_read_kb":311296,"num_write":93,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":19,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:54.984220+0000","last_change":"2024-05-28T06:24:22.987451+0000","last_active":"2024-05-28T06:24:54.984220+0000","last_peered":"2024-05-28T06:24:54.984220+0000","last_clean":"2024-05-28T06:24:54.984220+0000","last_became_active":"2024-05-28T06:24:22.987181+0000","last_became_peered":"2024-05-28T06:24:22.987181+0000","last_unstale":"2024-05-28T06:24:54.984220+0000","last_undegraded":"2024-05-28T06:24:54.984220+0000","last_fullsized":"2024-05-28T06:24:54.984220+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:45:51.036866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.12","version":"132'65","reported_seq":128,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.240342+0000","last_change":"2024-05-28T06:24:24.007933+0000","last_active":"2024-05-28T06:26:18.240342+0000","last_peered":"2024-05-28T06:26:18.240342+0000","last_clean":"2024-05-28T06:26:18.240342+0000","last_became_active":"2024-05-28T06:24:24.007651+0000","last_became_peered":"2024-05-28T06:24:24.007651+0000","last_unstale":"2024-05-28T06:26:18.240342+0000","last_undegraded":"2024-05-28T06:26:18.240342+0000","last_fullsized":"2024-05-28T06:26:18.240342+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:59:32.549372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":255852544,"num_objects":61,"num_object_clones":0,"num_object_copies":122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":0,"num_read":47,"num_read_kb":192512,"num_write":65,"num_write_kb":258048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"129'2","reported_seq":20,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:55.133764+0000","last_change":"2024-05-28T06:24:23.008078+0000","last_active":"2024-05-28T06:24:55.133764+0000","last_peered":"2024-05-28T06:24:55.133764+0000","last_clean":"2024-05-28T06:24:55.133764+0000","last_became_active":"2024-05-28T06:24:23.007331+0000","last_became_peered":"2024-05-28T06:24:23.007331+0000","last_unstale":"2024-05-28T06:24:55.133764+0000","last_undegraded":"2024-05-28T06:24:55.133764+0000","last_fullsized":"2024-05-28T06:24:55.133764+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:50:39.661313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":388,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:55.134179+0000","last_change":"2024-05-28T06:04:01.039129+0000","last_active":"2024-05-28T06:24:55.134179+0000","last_peered":"2024-05-28T06:24:55.134179+0000","last_clean":"2024-05-28T06:24:55.134179+0000","last_became_active":"2024-05-28T06:04:01.033906+0000","last_became_peered":"2024-05-28T06:04:01.033906+0000","last_unstale":"2024-05-28T06:24:55.134179+0000","last_undegraded":"2024-05-28T06:24:55.134179+0000","last_fullsized":"2024-05-28T06:24:55.134179+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:03:59.864237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:03:59.864237+0000","last_clean_scrub_stamp":"2024-05-28T06:03:59.864237+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:17:36.440245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":18396218954,"num_objects":4395,"num_object_clones":1,"num_object_copies":8790,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4395,"num_whiteouts":1,"num_read":3946,"num_read_kb":15908926,"num_write":4949,"num_write_kb":18956331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":41943040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36792500224,"data_stored":36792437908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4958,"ondisk_log_size":4958,"up":128,"acting":128,"num_store_stats":8},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":1678588,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":30,"num_read_kb":30,"num_write":598,"num_write_kb":1879,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3424256,"data_stored":3357176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":544,"ondisk_log_size":544,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575123,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3952528,"kb_used_data":3813128,"kb_used_omap":1,"kb_used_meta":139390,"kb_avail":90419312,"statfs":{"total":96636764160,"available":92589375488,"internally_reserved":0,"allocated":3904643072,"data_stored":3903694392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":142735818},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575125,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5139788,"kb_used_data":5000388,"kb_used_omap":1,"kb_used_meta":139390,"kb_avail":89232052,"statfs":{"total":96636764160,"available":91373621248,"internally_reserved":0,"allocated":5120397312,"data_stored":5119456102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":142735814},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575123,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3815132,"kb_used_data":3675732,"kb_used_omap":1,"kb_used_meta":139390,"kb_avail":90556708,"statfs":{"total":96636764160,"available":92730068992,"internally_reserved":0,"allocated":3763949568,"data_stored":3763018100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":142735813},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575123,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5547692,"kb_used_data":5412452,"kb_used_omap":1,"kb_used_meta":135230,"kb_avail":88824148,"statfs":{"total":96636764160,"available":90955927552,"internally_reserved":0,"allocated":5542350848,"data_stored":5541404120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":138475973},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575125,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4527844,"kb_used_data":4388444,"kb_used_omap":1,"kb_used_meta":139390,"kb_avail":89843996,"statfs":{"total":96636764160,"available":92000251904,"internally_reserved":0,"allocated":4493766656,"data_stored":4492831645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":142735815},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575124,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5961384,"kb_used_data":5826144,"kb_used_omap":1,"kb_used_meta":135230,"kb_avail":88410456,"statfs":{"total":96636764160,"available":90532306944,"internally_reserved":0,"allocated":5965971456,"data_stored":5965028896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":138475973},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575124,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4155024,"kb_used_data":4019784,"kb_used_omap":1,"kb_used_meta":135230,"kb_avail":90216816,"statfs":{"total":96636764160,"available":92382019584,"internally_reserved":0,"allocated":4116258816,"data_stored":4115339392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":138475973},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575123,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3959192,"kb_used_data":3811472,"kb_used_omap":1,"kb_used_meta":147710,"kb_avail":90412648,"statfs":{"total":96636764160,"available":92582551552,"internally_reserved":0,"allocated":3902947328,"data_stored":3902020517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151255493},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1691648,"data_stored":1677790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1695744,"data_stored":1673875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3900706816,"data_stored":3900702981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4114612224,"data_stored":4114612224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":5964316672,"data_stored":5964301112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4492103680,"data_stored":4492099764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":5540700160,"data_stored":5540676882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3762294784,"data_stored":3762290820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":5117059072,"data_stored":5117051144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3900706816,"data_stored":3900702981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-28T06:26:44.407 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-05-28T06:26:44.407 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-28T06:26:44.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.655+0000 7f72e1e00640 1 Processor -- start 2024-05-28T06:26:44.649 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.655+0000 7f72e1e00640 1 -- start start 2024-05-28T06:26:44.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.656+0000 7f72e1e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f72dc15e870 0x7f72dc0d0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:44.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.657+0000 7f72e1e00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72dc15ec50 con 0x7f72dc0d14d0 2024-05-28T06:26:44.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.657+0000 7f72e1e00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72dc0cff80 con 0x7f72dc0cfa40 2024-05-28T06:26:44.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.657+0000 7f72e1e00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f72dc0d2150 con 0x7f72dc15e870 2024-05-28T06:26:44.652 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.657+0000 7f72d3e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f72dc0d14d0 0x7f72dc0cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:44858/0 (socket says 172.21.15.96:44858) 2024-05-28T06:26:44.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.657+0000 7f72d3e00640 1 -- 172.21.15.96:0/3322230738 learned_addr learned my addr 172.21.15.96:0/3322230738 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:26:44.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.657+0000 7f72e0a00640 1 --2- 172.21.15.96:0/3322230738 >> v2:172.21.15.123:3300/0 conn(0x7f72dc15e870 0x7f72dc0d0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:44.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.657+0000 7f72dbe00640 1 -- 172.21.15.96:0/3322230738 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1847157352 0 0) 0x7f72dc15ec50 con 0x7f72dc0d14d0 2024-05-28T06:26:44.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.657+0000 7f72dbe00640 1 -- 172.21.15.96:0/3322230738 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f72c0003520 con 0x7f72dc0d14d0 2024-05-28T06:26:44.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.658+0000 7f72dbe00640 1 -- 172.21.15.96:0/3322230738 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4068803545 0 0) 0x7f72c0003520 con 0x7f72dc0d14d0 2024-05-28T06:26:44.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.658+0000 7f72dbe00640 1 -- 172.21.15.96:0/3322230738 >> v2:172.21.15.123:3300/0 conn(0x7f72dc15e870 msgr2=0x7f72dc0d0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:44.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.658+0000 7f72dbe00640 1 --2- 172.21.15.96:0/3322230738 >> v2:172.21.15.123:3300/0 conn(0x7f72dc15e870 0x7f72dc0d0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:44.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.658+0000 7f72dbe00640 1 -- 172.21.15.96:0/3322230738 >> v1:172.21.15.123:6789/0 conn(0x7f72dc0cfa40 legacy=0x7f72dc0d25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:44.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.658+0000 7f72dbe00640 1 -- 172.21.15.96:0/3322230738 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72dc0d0260 con 0x7f72dc0d14d0 2024-05-28T06:26:44.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.658+0000 7f72dbe00640 1 -- 172.21.15.96:0/3322230738 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f72c40017f0 con 0x7f72dc0d14d0 2024-05-28T06:26:44.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.658+0000 7f72dbe00640 1 -- 172.21.15.96:0/3322230738 >> v1:172.21.15.96:6789/0 conn(0x7f72dc0d14d0 legacy=0x7f72dc0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:44.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.658+0000 7f72dbe00640 1 --2- 172.21.15.96:0/3322230738 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f72c00039b0 0x7f72c0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:44.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.658+0000 7f72dbe00640 1 --2- 172.21.15.96:0/3322230738 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f72c0004800 0x7f72c0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:44.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.658+0000 7f72dbe00640 1 --2- 172.21.15.96:0/3322230738 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f72c00051c0 0x7f72c000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:44.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.658+0000 7f72dbe00640 1 -- 172.21.15.96:0/3322230738 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f72dc0d2150 con 0x7f72c0004800 2024-05-28T06:26:44.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.658+0000 7f72dbe00640 1 -- 172.21.15.96:0/3322230738 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f72c000c060 con 0x7f72c00039b0 2024-05-28T06:26:44.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.658+0000 7f72dbe00640 1 -- 172.21.15.96:0/3322230738 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f72c000c260 con 0x7f72c00051c0 2024-05-28T06:26:44.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.658+0000 7f72d3e00640 1 --2- 172.21.15.96:0/3322230738 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f72c0004800 0x7f72c0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:44.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.659+0000 7f72e0a00640 1 --2- 172.21.15.96:0/3322230738 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f72c00039b0 0x7f72c0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:44.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.659+0000 7f72e1400640 1 --2- 172.21.15.96:0/3322230738 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f72c00051c0 0x7f72c000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:44.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.659+0000 7f72d3e00640 1 -- 172.21.15.96:0/3322230738 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f72c00051c0 msgr2=0x7f72c000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:44.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.659+0000 7f72d3e00640 1 --2- 172.21.15.96:0/3322230738 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f72c00051c0 0x7f72c000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:44.655 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.659+0000 7f72d3e00640 1 -- 172.21.15.96:0/3322230738 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f72c00039b0 msgr2=0x7f72c0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:44.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.659+0000 7f72d3e00640 1 --2- 172.21.15.96:0/3322230738 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f72c00039b0 0x7f72c0003dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:44.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.659+0000 7f72d3e00640 1 -- 172.21.15.96:0/3322230738 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72c0003520 con 0x7f72c0004800 2024-05-28T06:26:44.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.659+0000 7f72e1400640 1 --2- 172.21.15.96:0/3322230738 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f72c00051c0 0x7f72c000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:26:44.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.659+0000 7f72e1400640 1 -- 172.21.15.96:0/3322230738 reap_dead start 2024-05-28T06:26:44.659 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.660+0000 7f72d3e00640 1 --2- 172.21.15.96:0/3322230738 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f72c0004800 0x7f72c0004c80 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 crypto rx=0x7f72c400c210 tx=0x7f72c0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f33083041f2a82e server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:44.659 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.660+0000 7f72dbe00640 1 -- 172.21.15.96:0/3322230738 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72c4007d20 con 0x7f72c0004800 2024-05-28T06:26:44.659 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.660+0000 7f72dbe00640 1 -- 172.21.15.96:0/3322230738 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72c4007270 con 0x7f72c0004800 2024-05-28T06:26:44.659 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.660+0000 7f72dbe00640 1 -- 172.21.15.96:0/3322230738 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72c40057a0 con 0x7f72c0004800 2024-05-28T06:26:44.659 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.660+0000 7f72e1e00640 1 -- 172.21.15.96:0/3322230738 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f72c0004800 msgr2=0x7f72c0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:44.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.660+0000 7f72e1e00640 1 --2- 172.21.15.96:0/3322230738 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f72c0004800 0x7f72c0004c80 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 crypto rx=0x7f72c400c210 tx=0x7f72c0005830 comp rx=0 tx=0).stop 2024-05-28T06:26:44.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.661+0000 7f72e1e00640 1 -- 172.21.15.96:0/3322230738 shutdown_connections 2024-05-28T06:26:44.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.661+0000 7f72e1e00640 1 --2- 172.21.15.96:0/3322230738 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f72c0004800 0x7f72c0004c80 unknown :-1 s=CLOSED pgs=2184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:44.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.661+0000 7f72e1e00640 1 -- 172.21.15.96:0/3322230738 >> 172.21.15.96:0/3322230738 conn(0x7f72dc08bfc0 msgr2=0x7f72dc0d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:26:44.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.661+0000 7f72e1e00640 1 -- 172.21.15.96:0/3322230738 shutdown_connections 2024-05-28T06:26:44.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.661+0000 7f72e1e00640 1 -- 172.21.15.96:0/3322230738 wait complete. 2024-05-28T06:26:44.660 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.662+0000 7f72e1e00640 1 Processor -- start 2024-05-28T06:26:44.661 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.663+0000 7f72e1e00640 1 -- start start 2024-05-28T06:26:44.661 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.664+0000 7f72e1e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f72c0004800 0x7f72dc159a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:44.661 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.664+0000 7f72e1e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f72dc0b9740 0x7f72dc0ca750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:44.661 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.664+0000 7f72e1e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f72dc0cc4e0 0x7f72dc0cac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:44.661 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.664+0000 7f72e1e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f72dc0d4fb0 con 0x7f72dc0b9740 2024-05-28T06:26:44.661 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.664+0000 7f72e1e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f72dc0d0820 con 0x7f72c0004800 2024-05-28T06:26:44.661 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.664+0000 7f72e1e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f72dc0d0f90 con 0x7f72dc0cc4e0 2024-05-28T06:26:44.661 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.664+0000 7f72d3e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f72dc0b9740 0x7f72dc0ca750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:44.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.664+0000 7f72d3e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f72dc0b9740 0x7f72dc0ca750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35096/0 (socket says 172.21.15.96:35096) 2024-05-28T06:26:44.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.664+0000 7f72d3e00640 1 -- 172.21.15.96:0/4039576683 learned_addr learned my addr 172.21.15.96:0/4039576683 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:26:44.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.664+0000 7f72e1400640 1 --2- 172.21.15.96:0/4039576683 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f72dc0cc4e0 0x7f72dc0cac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:44.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.664+0000 7f72e0a00640 1 --2- 172.21.15.96:0/4039576683 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f72c0004800 0x7f72dc159a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:44.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.664+0000 7f72d3e00640 1 -- 172.21.15.96:0/4039576683 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f72dc0cc4e0 msgr2=0x7f72dc0cac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:44.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.664+0000 7f72d3e00640 1 --2- 172.21.15.96:0/4039576683 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f72dc0cc4e0 0x7f72dc0cac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:44.662 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.664+0000 7f72d3e00640 1 -- 172.21.15.96:0/4039576683 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f72c0004800 msgr2=0x7f72dc159a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:44.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.664+0000 7f72d3e00640 1 --2- 172.21.15.96:0/4039576683 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f72c0004800 0x7f72dc159a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:44.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.664+0000 7f72d3e00640 1 -- 172.21.15.96:0/4039576683 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72dc0cc9c0 con 0x7f72dc0b9740 2024-05-28T06:26:44.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.665+0000 7f72d3e00640 1 --2- 172.21.15.96:0/4039576683 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f72dc0b9740 0x7f72dc0ca750 secure :-1 s=READY pgs=2185 cs=0 l=1 rev1=1 crypto rx=0x7f72c4000ef0 tx=0x7f72c4001bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:44.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.665+0000 7f72d9600640 1 -- 172.21.15.96:0/4039576683 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72c4042070 con 0x7f72dc0b9740 2024-05-28T06:26:44.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.665+0000 7f72d9600640 1 -- 172.21.15.96:0/4039576683 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72c4005bb0 con 0x7f72dc0b9740 2024-05-28T06:26:44.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.665+0000 7f72e1e00640 1 -- 172.21.15.96:0/4039576683 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72dc0ccca0 con 0x7f72dc0b9740 2024-05-28T06:26:44.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.665+0000 7f72d9600640 1 -- 172.21.15.96:0/4039576683 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72c403c430 con 0x7f72dc0b9740 2024-05-28T06:26:44.663 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.665+0000 7f72e1e00640 1 -- 172.21.15.96:0/4039576683 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72dc0cd260 con 0x7f72dc0b9740 2024-05-28T06:26:44.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.666+0000 7f72e1e00640 1 -- 172.21.15.96:0/4039576683 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72dc0689e0 con 0x7f72dc0b9740 2024-05-28T06:26:44.664 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.667+0000 7f72d9600640 1 -- 172.21.15.96:0/4039576683 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 49) ==== 94049+0+0 (secure 0 0 0) 0x7f72c4045070 con 0x7f72dc0b9740 2024-05-28T06:26:44.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.669+0000 7f72d9600640 1 --2- 172.21.15.96:0/4039576683 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f72b806ea20 0x7f72b8070ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:44.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.669+0000 7f72d9600640 1 -- 172.21.15.96:0/4039576683 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 5823+0+0 (secure 0 0 0) 0x7f72c40ab3a0 con 0x7f72dc0b9740 2024-05-28T06:26:44.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.669+0000 7f72e0a00640 1 --2- 172.21.15.96:0/4039576683 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f72b806ea20 0x7f72b8070ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:44.665 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.670+0000 7f72e0a00640 1 --2- 172.21.15.96:0/4039576683 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f72b806ea20 0x7f72b8070ee0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f72cc009990 tx=0x7f72cc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:44.666 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.673+0000 7f72d9600640 1 -- 172.21.15.96:0/4039576683 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f72c404c030 con 0x7f72dc0b9740 2024-05-28T06:26:44.865 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.871+0000 7f72e1e00640 1 -- 172.21.15.96:0/4039576683 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f72dc159f90 con 0x7f72b806ea20 2024-05-28T06:26:44.883 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.890+0000 7f72d9600640 1 -- 172.21.15.96:0/4039576683 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+254734 (secure 0 0 0) 0x7f72dc159f90 con 0x7f72b806ea20 2024-05-28T06:26:44.887 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:26:44.888 INFO:teuthology.orchestra.run.smithi096.stderr:dumped all 2024-05-28T06:26:44.889 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.894+0000 7f72e1e00640 1 -- 172.21.15.96:0/4039576683 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f72b806ea20 msgr2=0x7f72b8070ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:44.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.894+0000 7f72e1e00640 1 --2- 172.21.15.96:0/4039576683 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f72b806ea20 0x7f72b8070ee0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f72cc009990 tx=0x7f72cc008040 comp rx=0 tx=0).stop 2024-05-28T06:26:44.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.894+0000 7f72e1e00640 1 -- 172.21.15.96:0/4039576683 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f72dc0b9740 msgr2=0x7f72dc0ca750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:44.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.894+0000 7f72e1e00640 1 --2- 172.21.15.96:0/4039576683 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f72dc0b9740 0x7f72dc0ca750 secure :-1 s=READY pgs=2185 cs=0 l=1 rev1=1 crypto rx=0x7f72c4000ef0 tx=0x7f72c4001bc0 comp rx=0 tx=0).stop 2024-05-28T06:26:44.890 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.895+0000 7f72e1e00640 1 -- 172.21.15.96:0/4039576683 shutdown_connections 2024-05-28T06:26:44.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.895+0000 7f72e1e00640 1 --2- 172.21.15.96:0/4039576683 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f72b806ea20 0x7f72b8070ee0 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:44.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.895+0000 7f72e1e00640 1 --2- 172.21.15.96:0/4039576683 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f72dc0cc4e0 0x7f72dc0cac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:44.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.895+0000 7f72e1e00640 1 --2- 172.21.15.96:0/4039576683 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f72dc0b9740 0x7f72dc0ca750 unknown :-1 s=CLOSED pgs=2185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:44.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.895+0000 7f72e1e00640 1 --2- 172.21.15.96:0/4039576683 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f72c0004800 0x7f72dc159a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:44.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.895+0000 7f72e1e00640 1 -- 172.21.15.96:0/4039576683 >> 172.21.15.96:0/4039576683 conn(0x7f72dc08bfc0 msgr2=0x7f72dc0b4230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:26:44.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.895+0000 7f72e1e00640 1 -- 172.21.15.96:0/4039576683 shutdown_connections 2024-05-28T06:26:44.891 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:44.895+0000 7f72e1e00640 1 -- 172.21.15.96:0/4039576683 wait complete. 2024-05-28T06:26:44.908 INFO:teuthology.orchestra.run.smithi096.stdout:{"pg_ready":true,"pg_map":{"version":807,"stamp":"2024-05-28T06:26:44.081370+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":18398487910,"num_objects":4420,"num_object_clones":1,"num_object_copies":8840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4420,"num_whiteouts":1,"num_read":4092,"num_read_kb":15909056,"num_write":5706,"num_write_kb":18960626,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":41943040,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5619,"ondisk_log_size":5619,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":37058584,"kb_used_data":35947544,"kb_used_omap":12,"kb_used_meta":1110963,"kb_avail":717916136,"statfs":{"total":773094113280,"available":735146123264,"internally_reserved":0,"allocated":36810285056,"data_stored":36802793164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":1137626672},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004902"},"pg_stats":[{"pgid":"19.3f","version":"132'72","reported_seq":159,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.319580+0000","last_change":"2024-05-28T06:24:24.007430+0000","last_active":"2024-05-28T06:26:18.319580+0000","last_peered":"2024-05-28T06:26:18.319580+0000","last_clean":"2024-05-28T06:26:18.319580+0000","last_became_active":"2024-05-28T06:24:24.006988+0000","last_became_peered":"2024-05-28T06:24:24.006988+0000","last_unstale":"2024-05-28T06:26:18.319580+0000","last_undegraded":"2024-05-28T06:26:18.319580+0000","last_fullsized":"2024-05-28T06:26:18.319580+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:45:33.172226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260046848,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":70,"num_read_kb":286720,"num_write":72,"num_write_kb":274432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3e","version":"132'84","reported_seq":169,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.291618+0000","last_change":"2024-05-28T06:24:23.986742+0000","last_active":"2024-05-28T06:26:18.291618+0000","last_peered":"2024-05-28T06:26:18.291618+0000","last_clean":"2024-05-28T06:26:18.291618+0000","last_became_active":"2024-05-28T06:24:23.986471+0000","last_became_peered":"2024-05-28T06:24:23.986471+0000","last_unstale":"2024-05-28T06:26:18.291618+0000","last_undegraded":"2024-05-28T06:26:18.291618+0000","last_fullsized":"2024-05-28T06:26:18.291618+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:02:16.956326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327155712,"num_objects":78,"num_object_clones":0,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78,"num_whiteouts":0,"num_read":68,"num_read_kb":278528,"num_write":84,"num_write_kb":331776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3d","version":"132'93","reported_seq":180,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.325236+0000","last_change":"2024-05-28T06:24:24.007698+0000","last_active":"2024-05-28T06:26:18.325236+0000","last_peered":"2024-05-28T06:26:18.325236+0000","last_clean":"2024-05-28T06:26:18.325236+0000","last_became_active":"2024-05-28T06:24:24.007416+0000","last_became_peered":"2024-05-28T06:24:24.007416+0000","last_unstale":"2024-05-28T06:26:18.325236+0000","last_undegraded":"2024-05-28T06:26:18.325236+0000","last_fullsized":"2024-05-28T06:26:18.325236+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:24:31.676324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306184192,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":70,"num_read_kb":253960,"num_write":93,"num_write_kb":323588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.3c","version":"132'71","reported_seq":133,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.305549+0000","last_change":"2024-05-28T06:24:24.011512+0000","last_active":"2024-05-28T06:26:18.305549+0000","last_peered":"2024-05-28T06:26:18.305549+0000","last_clean":"2024-05-28T06:26:18.305549+0000","last_became_active":"2024-05-28T06:24:24.011234+0000","last_became_peered":"2024-05-28T06:24:24.011234+0000","last_unstale":"2024-05-28T06:26:18.305549+0000","last_undegraded":"2024-05-28T06:26:18.305549+0000","last_fullsized":"2024-05-28T06:26:18.305549+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:59:05.689554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281018368,"num_objects":67,"num_object_clones":0,"num_object_copies":134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":0,"num_read":45,"num_read_kb":176130,"num_write":71,"num_write_kb":278529,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3b","version":"132'65","reported_seq":119,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.288952+0000","last_change":"2024-05-28T06:24:24.019397+0000","last_active":"2024-05-28T06:26:18.288952+0000","last_peered":"2024-05-28T06:26:18.288952+0000","last_clean":"2024-05-28T06:26:18.288952+0000","last_became_active":"2024-05-28T06:24:24.018334+0000","last_became_peered":"2024-05-28T06:24:24.018334+0000","last_unstale":"2024-05-28T06:26:18.288952+0000","last_undegraded":"2024-05-28T06:26:18.288952+0000","last_fullsized":"2024-05-28T06:26:18.288952+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:22:34.611665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":230686720,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":38,"num_read_kb":155648,"num_write":65,"num_write_kb":245760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.3a","version":"132'97","reported_seq":216,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.278772+0000","last_change":"2024-05-28T06:24:55.231438+0000","last_active":"2024-05-28T06:26:18.278772+0000","last_peered":"2024-05-28T06:26:18.278772+0000","last_clean":"2024-05-28T06:26:18.278772+0000","last_became_active":"2024-05-28T06:24:54.990013+0000","last_became_peered":"2024-05-28T06:24:54.990013+0000","last_unstale":"2024-05-28T06:26:18.278772+0000","last_undegraded":"2024-05-28T06:26:18.278772+0000","last_fullsized":"2024-05-28T06:26:18.278772+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:50:40.872653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":360710405,"num_objects":87,"num_object_clones":0,"num_object_copies":174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87,"num_whiteouts":0,"num_read":84,"num_read_kb":339969,"num_write":97,"num_write_kb":368642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":25165824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.39","version":"132'65","reported_seq":126,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.259648+0000","last_change":"2024-05-28T06:24:24.006117+0000","last_active":"2024-05-28T06:26:18.259648+0000","last_peered":"2024-05-28T06:26:18.259648+0000","last_clean":"2024-05-28T06:26:18.259648+0000","last_became_active":"2024-05-28T06:24:24.005839+0000","last_became_peered":"2024-05-28T06:24:24.005839+0000","last_unstale":"2024-05-28T06:26:18.259648+0000","last_undegraded":"2024-05-28T06:26:18.259648+0000","last_fullsized":"2024-05-28T06:26:18.259648+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:47:32.965627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222298112,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":45,"num_read_kb":184320,"num_write":65,"num_write_kb":241664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.38","version":"132'79","reported_seq":142,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.263031+0000","last_change":"2024-05-28T06:24:24.008772+0000","last_active":"2024-05-28T06:26:18.263031+0000","last_peered":"2024-05-28T06:26:18.263031+0000","last_clean":"2024-05-28T06:26:18.263031+0000","last_became_active":"2024-05-28T06:24:24.008334+0000","last_became_peered":"2024-05-28T06:24:24.008334+0000","last_unstale":"2024-05-28T06:26:18.263031+0000","last_undegraded":"2024-05-28T06:26:18.263031+0000","last_fullsized":"2024-05-28T06:26:18.263031+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:10:50.477572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":297795584,"num_objects":71,"num_object_clones":0,"num_object_copies":142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":47,"num_read_kb":192512,"num_write":79,"num_write_kb":307200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.37","version":"132'80","reported_seq":149,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.239272+0000","last_change":"2024-05-28T06:24:24.018229+0000","last_active":"2024-05-28T06:26:18.239272+0000","last_peered":"2024-05-28T06:26:18.239272+0000","last_clean":"2024-05-28T06:26:18.239272+0000","last_became_active":"2024-05-28T06:24:24.017929+0000","last_became_peered":"2024-05-28T06:24:24.017929+0000","last_unstale":"2024-05-28T06:26:18.239272+0000","last_undegraded":"2024-05-28T06:26:18.239272+0000","last_fullsized":"2024-05-28T06:26:18.239272+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:58:16.573705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":297795716,"num_objects":72,"num_object_clones":0,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72,"num_whiteouts":0,"num_read":53,"num_read_kb":204803,"num_write":80,"num_write_kb":303106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.36","version":"132'63","reported_seq":137,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:25:42.637629+0000","last_change":"2024-05-28T06:24:23.995403+0000","last_active":"2024-05-28T06:25:42.637629+0000","last_peered":"2024-05-28T06:25:42.637629+0000","last_clean":"2024-05-28T06:25:42.637629+0000","last_became_active":"2024-05-28T06:24:23.994945+0000","last_became_peered":"2024-05-28T06:24:23.994945+0000","last_unstale":"2024-05-28T06:25:42.637629+0000","last_undegraded":"2024-05-28T06:25:42.637629+0000","last_fullsized":"2024-05-28T06:25:42.637629+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:11:29.809986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":255852544,"num_objects":61,"num_object_clones":0,"num_object_copies":122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":0,"num_read":58,"num_read_kb":237568,"num_write":63,"num_write_kb":253952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.35","version":"132'83","reported_seq":194,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.260079+0000","last_change":"2024-05-28T06:24:23.992865+0000","last_active":"2024-05-28T06:26:18.260079+0000","last_peered":"2024-05-28T06:26:18.260079+0000","last_clean":"2024-05-28T06:26:18.260079+0000","last_became_active":"2024-05-28T06:24:23.991405+0000","last_became_peered":"2024-05-28T06:24:23.991405+0000","last_unstale":"2024-05-28T06:26:18.260079+0000","last_undegraded":"2024-05-28T06:26:18.260079+0000","last_fullsized":"2024-05-28T06:26:18.260079+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:06:47.846653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":322961408,"num_objects":77,"num_object_clones":0,"num_object_copies":154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77,"num_whiteouts":0,"num_read":95,"num_read_kb":389120,"num_write":83,"num_write_kb":327680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.34","version":"132'77","reported_seq":185,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.283958+0000","last_change":"2024-05-28T06:24:24.018385+0000","last_active":"2024-05-28T06:26:18.283958+0000","last_peered":"2024-05-28T06:26:18.283958+0000","last_clean":"2024-05-28T06:26:18.283958+0000","last_became_active":"2024-05-28T06:24:24.017931+0000","last_became_peered":"2024-05-28T06:24:24.017931+0000","last_unstale":"2024-05-28T06:26:18.283958+0000","last_undegraded":"2024-05-28T06:26:18.283958+0000","last_fullsized":"2024-05-28T06:26:18.283958+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:11:21.508556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":289406976,"num_objects":69,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":0,"num_read":92,"num_read_kb":360452,"num_write":77,"num_write_kb":294913,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.33","version":"132'75","reported_seq":171,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.229026+0000","last_change":"2024-05-28T06:24:24.007399+0000","last_active":"2024-05-28T06:26:18.229026+0000","last_peered":"2024-05-28T06:26:18.229026+0000","last_clean":"2024-05-28T06:26:18.229026+0000","last_became_active":"2024-05-28T06:24:24.007121+0000","last_became_peered":"2024-05-28T06:24:24.007121+0000","last_unstale":"2024-05-28T06:26:18.229026+0000","last_undegraded":"2024-05-28T06:26:18.229026+0000","last_fullsized":"2024-05-28T06:26:18.229026+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:29:33.118245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":289406976,"num_objects":69,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":0,"num_read":80,"num_read_kb":327680,"num_write":75,"num_write_kb":294912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.32","version":"132'77","reported_seq":170,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.301921+0000","last_change":"2024-05-28T06:24:55.363255+0000","last_active":"2024-05-28T06:26:18.301921+0000","last_peered":"2024-05-28T06:26:18.301921+0000","last_clean":"2024-05-28T06:26:18.301921+0000","last_became_active":"2024-05-28T06:24:54.998097+0000","last_became_peered":"2024-05-28T06:24:54.998097+0000","last_unstale":"2024-05-28T06:26:18.301921+0000","last_undegraded":"2024-05-28T06:26:18.301921+0000","last_fullsized":"2024-05-28T06:26:18.301921+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:41:33.057222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":289406976,"num_objects":69,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":0,"num_read":62,"num_read_kb":253952,"num_write":77,"num_write_kb":299008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":16777216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.31","version":"132'70","reported_seq":116,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.292733+0000","last_change":"2024-05-28T06:24:24.020598+0000","last_active":"2024-05-28T06:26:18.292733+0000","last_peered":"2024-05-28T06:26:18.292733+0000","last_clean":"2024-05-28T06:26:18.292733+0000","last_became_active":"2024-05-28T06:24:24.020301+0000","last_became_peered":"2024-05-28T06:24:24.020301+0000","last_unstale":"2024-05-28T06:26:18.292733+0000","last_undegraded":"2024-05-28T06:26:18.292733+0000","last_fullsized":"2024-05-28T06:26:18.292733+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:18:18.796231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251658240,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":30,"num_read_kb":122880,"num_write":70,"num_write_kb":266240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.30","version":"132'81","reported_seq":148,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.299313+0000","last_change":"2024-05-28T06:24:23.995628+0000","last_active":"2024-05-28T06:26:18.299313+0000","last_peered":"2024-05-28T06:26:18.299313+0000","last_clean":"2024-05-28T06:26:18.299313+0000","last_became_active":"2024-05-28T06:24:23.995367+0000","last_became_peered":"2024-05-28T06:24:23.995367+0000","last_unstale":"2024-05-28T06:26:18.299313+0000","last_undegraded":"2024-05-28T06:26:18.299313+0000","last_fullsized":"2024-05-28T06:26:18.299313+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:07:49.435286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306184192,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":51,"num_read_kb":208896,"num_write":81,"num_write_kb":315392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2f","version":"132'65","reported_seq":123,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.267214+0000","last_change":"2024-05-28T06:24:23.987587+0000","last_active":"2024-05-28T06:26:18.267214+0000","last_peered":"2024-05-28T06:26:18.267214+0000","last_clean":"2024-05-28T06:26:18.267214+0000","last_became_active":"2024-05-28T06:24:23.986777+0000","last_became_peered":"2024-05-28T06:24:23.986777+0000","last_unstale":"2024-05-28T06:26:18.267214+0000","last_undegraded":"2024-05-28T06:26:18.267214+0000","last_fullsized":"2024-05-28T06:26:18.267214+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:06:00.030718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":230686720,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":42,"num_read_kb":172032,"num_write":65,"num_write_kb":245760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.2e","version":"132'82","reported_seq":185,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.245601+0000","last_change":"2024-05-28T06:24:23.998500+0000","last_active":"2024-05-28T06:26:18.245601+0000","last_peered":"2024-05-28T06:26:18.245601+0000","last_clean":"2024-05-28T06:26:18.245601+0000","last_became_active":"2024-05-28T06:24:23.998105+0000","last_became_peered":"2024-05-28T06:24:23.998105+0000","last_unstale":"2024-05-28T06:26:18.245601+0000","last_undegraded":"2024-05-28T06:26:18.245601+0000","last_fullsized":"2024-05-28T06:26:18.245601+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:47:31.291023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":310378496,"num_objects":74,"num_object_clones":0,"num_object_copies":148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74,"num_whiteouts":0,"num_read":87,"num_read_kb":356352,"num_write":82,"num_write_kb":315393,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2d","version":"132'83","reported_seq":167,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.323316+0000","last_change":"2024-05-28T06:24:23.997899+0000","last_active":"2024-05-28T06:26:18.323316+0000","last_peered":"2024-05-28T06:26:18.323316+0000","last_clean":"2024-05-28T06:26:18.323316+0000","last_became_active":"2024-05-28T06:24:23.997362+0000","last_became_peered":"2024-05-28T06:24:23.997362+0000","last_unstale":"2024-05-28T06:26:18.323316+0000","last_undegraded":"2024-05-28T06:26:18.323316+0000","last_fullsized":"2024-05-28T06:26:18.323316+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:48:26.250684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":310378496,"num_objects":74,"num_object_clones":0,"num_object_copies":148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74,"num_whiteouts":0,"num_read":67,"num_read_kb":274432,"num_write":82,"num_write_kb":319488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2c","version":"132'70","reported_seq":144,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.318303+0000","last_change":"2024-05-28T06:24:23.996440+0000","last_active":"2024-05-28T06:26:18.318303+0000","last_peered":"2024-05-28T06:26:18.318303+0000","last_clean":"2024-05-28T06:26:18.318303+0000","last_became_active":"2024-05-28T06:24:23.996067+0000","last_became_peered":"2024-05-28T06:24:23.996067+0000","last_unstale":"2024-05-28T06:26:18.318303+0000","last_undegraded":"2024-05-28T06:26:18.318303+0000","last_fullsized":"2024-05-28T06:26:18.318303+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:06:00.030718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":272629760,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":57,"num_read_kb":233472,"num_write":69,"num_write_kb":274432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2b","version":"132'82","reported_seq":190,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.301761+0000","last_change":"2024-05-28T06:24:24.018537+0000","last_active":"2024-05-28T06:26:18.301761+0000","last_peered":"2024-05-28T06:26:18.301761+0000","last_clean":"2024-05-28T06:26:18.301761+0000","last_became_active":"2024-05-28T06:24:24.018253+0000","last_became_peered":"2024-05-28T06:24:24.018253+0000","last_unstale":"2024-05-28T06:26:18.301761+0000","last_undegraded":"2024-05-28T06:26:18.301761+0000","last_fullsized":"2024-05-28T06:26:18.301761+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:13:05.023858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":293601280,"num_objects":70,"num_object_clones":0,"num_object_copies":140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70,"num_whiteouts":0,"num_read":91,"num_read_kb":364546,"num_write":82,"num_write_kb":303106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2a","version":"132'75","reported_seq":147,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.271529+0000","last_change":"2024-05-28T06:24:23.994663+0000","last_active":"2024-05-28T06:26:18.271529+0000","last_peered":"2024-05-28T06:26:18.271529+0000","last_clean":"2024-05-28T06:26:18.271529+0000","last_became_active":"2024-05-28T06:24:23.994410+0000","last_became_peered":"2024-05-28T06:24:23.994410+0000","last_unstale":"2024-05-28T06:26:18.271529+0000","last_undegraded":"2024-05-28T06:26:18.271529+0000","last_fullsized":"2024-05-28T06:26:18.271529+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:05:58.537398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":285212672,"num_objects":68,"num_object_clones":0,"num_object_copies":136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68,"num_whiteouts":0,"num_read":55,"num_read_kb":217090,"num_write":74,"num_write_kb":286721,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.29","version":"132'81","reported_seq":149,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.292813+0000","last_change":"2024-05-28T06:24:24.015438+0000","last_active":"2024-05-28T06:26:18.292813+0000","last_peered":"2024-05-28T06:26:18.292813+0000","last_clean":"2024-05-28T06:26:18.292813+0000","last_became_active":"2024-05-28T06:24:24.015124+0000","last_became_peered":"2024-05-28T06:24:24.015124+0000","last_unstale":"2024-05-28T06:26:18.292813+0000","last_undegraded":"2024-05-28T06:26:18.292813+0000","last_fullsized":"2024-05-28T06:26:18.292813+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:40:05.545358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281018368,"num_objects":67,"num_object_clones":0,"num_object_copies":134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":0,"num_read":51,"num_read_kb":208896,"num_write":81,"num_write_kb":303104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.28","version":"132'75","reported_seq":143,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.270972+0000","last_change":"2024-05-28T06:24:24.017147+0000","last_active":"2024-05-28T06:26:18.270972+0000","last_peered":"2024-05-28T06:26:18.270972+0000","last_clean":"2024-05-28T06:26:18.270972+0000","last_became_active":"2024-05-28T06:24:24.016692+0000","last_became_peered":"2024-05-28T06:24:24.016692+0000","last_unstale":"2024-05-28T06:26:18.270972+0000","last_undegraded":"2024-05-28T06:26:18.270972+0000","last_fullsized":"2024-05-28T06:26:18.270972+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:10:49.969260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281018368,"num_objects":67,"num_object_clones":0,"num_object_copies":134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":0,"num_read":51,"num_read_kb":208896,"num_write":75,"num_write_kb":286721,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.27","version":"132'81","reported_seq":161,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.270839+0000","last_change":"2024-05-28T06:24:24.007398+0000","last_active":"2024-05-28T06:26:18.270839+0000","last_peered":"2024-05-28T06:26:18.270839+0000","last_clean":"2024-05-28T06:26:18.270839+0000","last_became_active":"2024-05-28T06:24:24.006983+0000","last_became_peered":"2024-05-28T06:24:24.006983+0000","last_unstale":"2024-05-28T06:26:18.270839+0000","last_undegraded":"2024-05-28T06:26:18.270839+0000","last_fullsized":"2024-05-28T06:26:18.270839+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:02:59.158681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306184192,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":63,"num_read_kb":249858,"num_write":81,"num_write_kb":307202,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.26","version":"132'83","reported_seq":162,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.291454+0000","last_change":"2024-05-28T06:24:23.999225+0000","last_active":"2024-05-28T06:26:18.291454+0000","last_peered":"2024-05-28T06:26:18.291454+0000","last_clean":"2024-05-28T06:26:18.291454+0000","last_became_active":"2024-05-28T06:24:23.996859+0000","last_became_peered":"2024-05-28T06:24:23.996859+0000","last_unstale":"2024-05-28T06:26:18.291454+0000","last_undegraded":"2024-05-28T06:26:18.291454+0000","last_fullsized":"2024-05-28T06:26:18.291454+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:05:09.592919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":310378496,"num_objects":74,"num_object_clones":0,"num_object_copies":148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74,"num_whiteouts":0,"num_read":62,"num_read_kb":253952,"num_write":82,"num_write_kb":319488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.25","version":"132'73","reported_seq":156,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.305983+0000","last_change":"2024-05-28T06:24:24.007463+0000","last_active":"2024-05-28T06:26:18.305983+0000","last_peered":"2024-05-28T06:26:18.305983+0000","last_clean":"2024-05-28T06:26:18.305983+0000","last_became_active":"2024-05-28T06:24:24.007194+0000","last_became_peered":"2024-05-28T06:24:24.007194+0000","last_unstale":"2024-05-28T06:26:18.305983+0000","last_undegraded":"2024-05-28T06:26:18.305983+0000","last_fullsized":"2024-05-28T06:26:18.305983+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:27:47.069088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":272629760,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":66,"num_read_kb":258051,"num_write":73,"num_write_kb":278529,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.24","version":"132'77","reported_seq":128,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.306695+0000","last_change":"2024-05-28T06:24:24.016291+0000","last_active":"2024-05-28T06:26:18.306695+0000","last_peered":"2024-05-28T06:26:18.306695+0000","last_clean":"2024-05-28T06:26:18.306695+0000","last_became_active":"2024-05-28T06:24:24.015990+0000","last_became_peered":"2024-05-28T06:24:24.015990+0000","last_unstale":"2024-05-28T06:26:18.306695+0000","last_undegraded":"2024-05-28T06:26:18.306695+0000","last_fullsized":"2024-05-28T06:26:18.306695+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:02:16.956326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281018368,"num_objects":67,"num_object_clones":0,"num_object_copies":134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":0,"num_read":34,"num_read_kb":131074,"num_write":77,"num_write_kb":290817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.23","version":"132'81","reported_seq":169,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.308583+0000","last_change":"2024-05-28T06:24:24.019908+0000","last_active":"2024-05-28T06:26:18.308583+0000","last_peered":"2024-05-28T06:26:18.308583+0000","last_clean":"2024-05-28T06:26:18.308583+0000","last_became_active":"2024-05-28T06:24:24.019497+0000","last_became_peered":"2024-05-28T06:24:24.019497+0000","last_unstale":"2024-05-28T06:26:18.308583+0000","last_undegraded":"2024-05-28T06:26:18.308583+0000","last_fullsized":"2024-05-28T06:26:18.308583+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:16:02.402760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":297795584,"num_objects":71,"num_object_clones":0,"num_object_copies":142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":71,"num_read_kb":274436,"num_write":81,"num_write_kb":307201,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.22","version":"132'85","reported_seq":164,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.308511+0000","last_change":"2024-05-28T06:24:24.019852+0000","last_active":"2024-05-28T06:26:18.308511+0000","last_peered":"2024-05-28T06:26:18.308511+0000","last_clean":"2024-05-28T06:26:18.308511+0000","last_became_active":"2024-05-28T06:24:24.019462+0000","last_became_peered":"2024-05-28T06:24:24.019462+0000","last_unstale":"2024-05-28T06:26:18.308511+0000","last_undegraded":"2024-05-28T06:26:18.308511+0000","last_fullsized":"2024-05-28T06:26:18.308511+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:52:19.161622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":314572800,"num_objects":75,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75,"num_whiteouts":0,"num_read":62,"num_read_kb":253952,"num_write":85,"num_write_kb":327680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.21","version":"132'66","reported_seq":172,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:25:43.030729+0000","last_change":"2024-05-28T06:24:24.011632+0000","last_active":"2024-05-28T06:25:43.030729+0000","last_peered":"2024-05-28T06:25:43.030729+0000","last_clean":"2024-05-28T06:25:43.030729+0000","last_became_active":"2024-05-28T06:24:24.011246+0000","last_became_peered":"2024-05-28T06:24:24.011246+0000","last_unstale":"2024-05-28T06:25:43.030729+0000","last_undegraded":"2024-05-28T06:25:43.030729+0000","last_fullsized":"2024-05-28T06:25:43.030729+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:05:09.592919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":276824064,"num_objects":66,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":89,"num_read_kb":364544,"num_write":66,"num_write_kb":270336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.20","version":"132'75","reported_seq":163,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.316590+0000","last_change":"2024-05-28T06:24:23.991803+0000","last_active":"2024-05-28T06:26:18.316590+0000","last_peered":"2024-05-28T06:26:18.316590+0000","last_clean":"2024-05-28T06:26:18.316590+0000","last_became_active":"2024-05-28T06:24:23.991245+0000","last_became_peered":"2024-05-28T06:24:23.991245+0000","last_unstale":"2024-05-28T06:26:18.316590+0000","last_undegraded":"2024-05-28T06:26:18.316590+0000","last_fullsized":"2024-05-28T06:26:18.316590+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:55:45.102561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":272629760,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":71,"num_read_kb":282626,"num_write":75,"num_write_kb":282625,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1e","version":"132'68","reported_seq":127,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.309071+0000","last_change":"2024-05-28T06:24:24.018032+0000","last_active":"2024-05-28T06:26:18.309071+0000","last_peered":"2024-05-28T06:26:18.309071+0000","last_clean":"2024-05-28T06:26:18.309071+0000","last_became_active":"2024-05-28T06:24:24.017727+0000","last_became_peered":"2024-05-28T06:24:24.017727+0000","last_unstale":"2024-05-28T06:26:18.309071+0000","last_undegraded":"2024-05-28T06:26:18.309071+0000","last_fullsized":"2024-05-28T06:26:18.309071+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:13:05.023858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260046848,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":42,"num_read_kb":172032,"num_write":68,"num_write_kb":266240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1f","version":"129'4","reported_seq":23,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:55.134481+0000","last_change":"2024-05-28T06:24:23.006418+0000","last_active":"2024-05-28T06:24:55.134481+0000","last_peered":"2024-05-28T06:24:55.134481+0000","last_clean":"2024-05-28T06:24:55.134481+0000","last_became_active":"2024-05-28T06:24:23.006043+0000","last_became_peered":"2024-05-28T06:24:23.006043+0000","last_unstale":"2024-05-28T06:24:55.134481+0000","last_undegraded":"2024-05-28T06:24:55.134481+0000","last_fullsized":"2024-05-28T06:24:55.134481+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:31:42.560242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.1f","version":"132'82","reported_seq":167,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.285593+0000","last_change":"2024-05-28T06:24:24.008420+0000","last_active":"2024-05-28T06:26:18.285593+0000","last_peered":"2024-05-28T06:26:18.285593+0000","last_clean":"2024-05-28T06:26:18.285593+0000","last_became_active":"2024-05-28T06:24:24.008078+0000","last_became_peered":"2024-05-28T06:24:24.008078+0000","last_unstale":"2024-05-28T06:26:18.285593+0000","last_undegraded":"2024-05-28T06:26:18.285593+0000","last_fullsized":"2024-05-28T06:26:18.285593+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:08:27.007107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":293601280,"num_objects":70,"num_object_clones":0,"num_object_copies":140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70,"num_whiteouts":0,"num_read":68,"num_read_kb":262148,"num_write":82,"num_write_kb":303106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1e","version":"129'1","reported_seq":20,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:55.454913+0000","last_change":"2024-05-28T06:24:22.995399+0000","last_active":"2024-05-28T06:24:55.454913+0000","last_peered":"2024-05-28T06:24:55.454913+0000","last_clean":"2024-05-28T06:24:55.454913+0000","last_became_active":"2024-05-28T06:24:22.994896+0000","last_became_peered":"2024-05-28T06:24:22.994896+0000","last_unstale":"2024-05-28T06:24:55.454913+0000","last_undegraded":"2024-05-28T06:24:55.454913+0000","last_fullsized":"2024-05-28T06:24:55.454913+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:06:41.660486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1c","version":"132'70","reported_seq":150,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.265732+0000","last_change":"2024-05-28T06:24:24.006543+0000","last_active":"2024-05-28T06:26:18.265732+0000","last_peered":"2024-05-28T06:26:18.265732+0000","last_clean":"2024-05-28T06:26:18.265732+0000","last_became_active":"2024-05-28T06:24:24.006258+0000","last_became_peered":"2024-05-28T06:24:24.006258+0000","last_unstale":"2024-05-28T06:26:18.265732+0000","last_undegraded":"2024-05-28T06:26:18.265732+0000","last_fullsized":"2024-05-28T06:26:18.265732+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:34:58.279352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":255852805,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":64,"num_read_kb":262144,"num_write":70,"num_write_kb":266241,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1d","version":"132'3","reported_seq":21,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:54.985140+0000","last_change":"2024-05-28T06:24:22.992163+0000","last_active":"2024-05-28T06:24:54.985140+0000","last_peered":"2024-05-28T06:24:54.985140+0000","last_clean":"2024-05-28T06:24:54.985140+0000","last_became_active":"2024-05-28T06:24:22.991407+0000","last_became_peered":"2024-05-28T06:24:22.991407+0000","last_unstale":"2024-05-28T06:24:54.985140+0000","last_undegraded":"2024-05-28T06:24:54.985140+0000","last_fullsized":"2024-05-28T06:24:54.985140+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:34:43.911764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1d","version":"132'89","reported_seq":192,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.314145+0000","last_change":"2024-05-28T06:24:24.013028+0000","last_active":"2024-05-28T06:26:18.314145+0000","last_peered":"2024-05-28T06:26:18.314145+0000","last_clean":"2024-05-28T06:26:18.314145+0000","last_became_active":"2024-05-28T06:24:24.010356+0000","last_became_peered":"2024-05-28T06:24:24.010356+0000","last_unstale":"2024-05-28T06:26:18.314145+0000","last_undegraded":"2024-05-28T06:26:18.314145+0000","last_fullsized":"2024-05-28T06:26:18.314145+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:46:38.879485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":297795584,"num_objects":71,"num_object_clones":0,"num_object_copies":142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":87,"num_read_kb":356352,"num_write":89,"num_write_kb":327680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1c","version":"129'2","reported_seq":20,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:54.994094+0000","last_change":"2024-05-28T06:24:22.989493+0000","last_active":"2024-05-28T06:24:54.994094+0000","last_peered":"2024-05-28T06:24:54.994094+0000","last_clean":"2024-05-28T06:24:54.994094+0000","last_became_active":"2024-05-28T06:24:22.989103+0000","last_became_peered":"2024-05-28T06:24:22.989103+0000","last_unstale":"2024-05-28T06:24:54.994094+0000","last_undegraded":"2024-05-28T06:24:54.994094+0000","last_fullsized":"2024-05-28T06:24:54.994094+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:28:25.855171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1a","version":"132'76","reported_seq":145,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.241565+0000","last_change":"2024-05-28T06:24:24.008297+0000","last_active":"2024-05-28T06:26:18.241565+0000","last_peered":"2024-05-28T06:26:18.241565+0000","last_clean":"2024-05-28T06:26:18.241565+0000","last_became_active":"2024-05-28T06:24:24.007987+0000","last_became_peered":"2024-05-28T06:24:24.007987+0000","last_unstale":"2024-05-28T06:26:18.241565+0000","last_undegraded":"2024-05-28T06:26:18.241565+0000","last_fullsized":"2024-05-28T06:26:18.241565+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:48:49.925554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301989888,"num_objects":72,"num_object_clones":0,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72,"num_whiteouts":0,"num_read":53,"num_read_kb":217088,"num_write":76,"num_write_kb":303104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1b","version":"129'1","reported_seq":19,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:54.995380+0000","last_change":"2024-05-28T06:24:22.987047+0000","last_active":"2024-05-28T06:24:54.995380+0000","last_peered":"2024-05-28T06:24:54.995380+0000","last_clean":"2024-05-28T06:24:54.995380+0000","last_became_active":"2024-05-28T06:24:22.985617+0000","last_became_peered":"2024-05-28T06:24:22.985617+0000","last_unstale":"2024-05-28T06:24:54.995380+0000","last_undegraded":"2024-05-28T06:24:54.995380+0000","last_fullsized":"2024-05-28T06:24:54.995380+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:59:04.673636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1b","version":"132'77","reported_seq":182,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.245202+0000","last_change":"2024-05-28T06:24:23.987152+0000","last_active":"2024-05-28T06:26:18.245202+0000","last_peered":"2024-05-28T06:26:18.245202+0000","last_clean":"2024-05-28T06:26:18.245202+0000","last_became_active":"2024-05-28T06:24:23.985828+0000","last_became_peered":"2024-05-28T06:24:23.985828+0000","last_unstale":"2024-05-28T06:26:18.245202+0000","last_undegraded":"2024-05-28T06:26:18.245202+0000","last_fullsized":"2024-05-28T06:26:18.245202+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:48:49.925554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306184192,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":89,"num_read_kb":364544,"num_write":77,"num_write_kb":307200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1a","version":"129'1","reported_seq":19,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:55.341429+0000","last_change":"2024-05-28T06:24:22.990085+0000","last_active":"2024-05-28T06:24:55.341429+0000","last_peered":"2024-05-28T06:24:55.341429+0000","last_clean":"2024-05-28T06:24:55.341429+0000","last_became_active":"2024-05-28T06:24:22.989801+0000","last_became_peered":"2024-05-28T06:24:22.989801+0000","last_unstale":"2024-05-28T06:24:55.341429+0000","last_undegraded":"2024-05-28T06:24:55.341429+0000","last_fullsized":"2024-05-28T06:24:55.341429+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:53:06.353639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.18","version":"132'72","reported_seq":155,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.299919+0000","last_change":"2024-05-28T06:24:24.003806+0000","last_active":"2024-05-28T06:26:18.299919+0000","last_peered":"2024-05-28T06:26:18.299919+0000","last_clean":"2024-05-28T06:26:18.299919+0000","last_became_active":"2024-05-28T06:24:24.003446+0000","last_became_peered":"2024-05-28T06:24:24.003446+0000","last_unstale":"2024-05-28T06:26:18.299919+0000","last_undegraded":"2024-05-28T06:26:18.299919+0000","last_fullsized":"2024-05-28T06:26:18.299919+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:05:52.238604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260046848,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":67,"num_read_kb":274432,"num_write":72,"num_write_kb":274432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":18,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:54.984156+0000","last_change":"2024-05-28T06:24:23.007861+0000","last_active":"2024-05-28T06:24:54.984156+0000","last_peered":"2024-05-28T06:24:54.984156+0000","last_clean":"2024-05-28T06:24:54.984156+0000","last_became_active":"2024-05-28T06:24:23.007494+0000","last_became_peered":"2024-05-28T06:24:23.007494+0000","last_unstale":"2024-05-28T06:24:54.984156+0000","last_undegraded":"2024-05-28T06:24:54.984156+0000","last_fullsized":"2024-05-28T06:24:54.984156+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:06:19.878458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.19","version":"132'72","reported_seq":150,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.195341+0000","last_change":"2024-05-28T06:24:23.993309+0000","last_active":"2024-05-28T06:26:18.195341+0000","last_peered":"2024-05-28T06:26:18.195341+0000","last_clean":"2024-05-28T06:26:18.195341+0000","last_became_active":"2024-05-28T06:24:23.987466+0000","last_became_peered":"2024-05-28T06:24:23.987466+0000","last_unstale":"2024-05-28T06:26:18.195341+0000","last_undegraded":"2024-05-28T06:26:18.195341+0000","last_fullsized":"2024-05-28T06:26:18.195341+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:07:16.445323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":276824064,"num_objects":66,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":62,"num_read_kb":245762,"num_write":72,"num_write_kb":278529,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":18,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:54.994422+0000","last_change":"2024-05-28T06:24:23.000407+0000","last_active":"2024-05-28T06:24:54.994422+0000","last_peered":"2024-05-28T06:24:54.994422+0000","last_clean":"2024-05-28T06:24:54.994422+0000","last_became_active":"2024-05-28T06:24:23.000037+0000","last_became_peered":"2024-05-28T06:24:23.000037+0000","last_unstale":"2024-05-28T06:24:54.994422+0000","last_undegraded":"2024-05-28T06:24:54.994422+0000","last_fullsized":"2024-05-28T06:24:54.994422+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:59:08.824871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"132'71","reported_seq":120,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.295941+0000","last_change":"2024-05-28T06:24:23.995548+0000","last_active":"2024-05-28T06:26:18.295941+0000","last_peered":"2024-05-28T06:26:18.295941+0000","last_clean":"2024-05-28T06:26:18.295941+0000","last_became_active":"2024-05-28T06:24:23.995204+0000","last_became_peered":"2024-05-28T06:24:23.995204+0000","last_unstale":"2024-05-28T06:26:18.295941+0000","last_undegraded":"2024-05-28T06:26:18.295941+0000","last_fullsized":"2024-05-28T06:26:18.295941+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:25:36.261157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260046848,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":33,"num_read_kb":135168,"num_write":70,"num_write_kb":270336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.17","version":"132'1","reported_seq":19,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:55.454932+0000","last_change":"2024-05-28T06:24:23.001114+0000","last_active":"2024-05-28T06:24:55.454932+0000","last_peered":"2024-05-28T06:24:55.454932+0000","last_clean":"2024-05-28T06:24:55.454932+0000","last_became_active":"2024-05-28T06:24:22.999720+0000","last_became_peered":"2024-05-28T06:24:22.999720+0000","last_unstale":"2024-05-28T06:24:55.454932+0000","last_undegraded":"2024-05-28T06:24:55.454932+0000","last_fullsized":"2024-05-28T06:24:55.454932+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:29:54.261773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.17","version":"132'63","reported_seq":117,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.234194+0000","last_change":"2024-05-28T06:24:24.016902+0000","last_active":"2024-05-28T06:26:18.234194+0000","last_peered":"2024-05-28T06:26:18.234194+0000","last_clean":"2024-05-28T06:26:18.234194+0000","last_became_active":"2024-05-28T06:24:24.016597+0000","last_became_peered":"2024-05-28T06:24:24.016597+0000","last_unstale":"2024-05-28T06:26:18.234194+0000","last_undegraded":"2024-05-28T06:26:18.234194+0000","last_fullsized":"2024-05-28T06:26:18.234194+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:08:42.734914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247463936,"num_objects":59,"num_object_clones":0,"num_object_copies":118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":0,"num_read":38,"num_read_kb":155648,"num_write":63,"num_write_kb":249856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":18,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:55.342025+0000","last_change":"2024-05-28T06:24:23.001188+0000","last_active":"2024-05-28T06:24:55.342025+0000","last_peered":"2024-05-28T06:24:55.342025+0000","last_clean":"2024-05-28T06:24:55.342025+0000","last_became_active":"2024-05-28T06:24:23.000878+0000","last_became_peered":"2024-05-28T06:24:23.000878+0000","last_unstale":"2024-05-28T06:24:55.342025+0000","last_undegraded":"2024-05-28T06:24:55.342025+0000","last_fullsized":"2024-05-28T06:24:55.342025+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:18:06.253798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.14","version":"132'75","reported_seq":166,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.313193+0000","last_change":"2024-05-28T06:24:24.011311+0000","last_active":"2024-05-28T06:26:18.313193+0000","last_peered":"2024-05-28T06:26:18.313193+0000","last_clean":"2024-05-28T06:26:18.313193+0000","last_became_active":"2024-05-28T06:24:24.010110+0000","last_became_peered":"2024-05-28T06:24:24.010110+0000","last_unstale":"2024-05-28T06:26:18.313193+0000","last_undegraded":"2024-05-28T06:26:18.313193+0000","last_fullsized":"2024-05-28T06:26:18.313193+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:46:31.105891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251658240,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":75,"num_read_kb":307200,"num_write":74,"num_write_kb":274432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.15","version":"132'24","reported_seq":42,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.184203+0000","last_change":"2024-05-28T06:24:22.999088+0000","last_active":"2024-05-28T06:26:18.184203+0000","last_peered":"2024-05-28T06:26:18.184203+0000","last_clean":"2024-05-28T06:26:18.184203+0000","last_became_active":"2024-05-28T06:24:22.998379+0000","last_became_peered":"2024-05-28T06:24:22.998379+0000","last_unstale":"2024-05-28T06:26:18.184203+0000","last_undegraded":"2024-05-28T06:26:18.184203+0000","last_fullsized":"2024-05-28T06:26:18.184203+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:59:19.637147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.15","version":"132'68","reported_seq":144,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.268436+0000","last_change":"2024-05-28T06:24:24.020278+0000","last_active":"2024-05-28T06:26:18.268436+0000","last_peered":"2024-05-28T06:26:18.268436+0000","last_clean":"2024-05-28T06:26:18.268436+0000","last_became_active":"2024-05-28T06:24:24.019973+0000","last_became_peered":"2024-05-28T06:24:24.019973+0000","last_unstale":"2024-05-28T06:26:18.268436+0000","last_undegraded":"2024-05-28T06:26:18.268436+0000","last_fullsized":"2024-05-28T06:26:18.268436+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:46:47.603731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":230686720,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":60,"num_read_kb":229380,"num_write":67,"num_write_kb":241666,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.14","version":"132'416","reported_seq":434,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:23.552552+0000","last_change":"2024-05-28T06:24:23.005145+0000","last_active":"2024-05-28T06:26:23.552552+0000","last_peered":"2024-05-28T06:26:23.552552+0000","last_clean":"2024-05-28T06:26:23.552552+0000","last_became_active":"2024-05-28T06:24:23.004831+0000","last_became_peered":"2024-05-28T06:24:23.004831+0000","last_unstale":"2024-05-28T06:26:23.552552+0000","last_undegraded":"2024-05-28T06:26:23.552552+0000","last_fullsized":"2024-05-28T06:26:23.552552+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":416,"log_dups_size":0,"ondisk_log_size":416,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:09:55.668305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1673237,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":415,"num_write_kb":1791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.13","version":"132'75","reported_seq":141,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:25:42.876820+0000","last_change":"2024-05-28T06:24:24.002061+0000","last_active":"2024-05-28T06:25:42.876820+0000","last_peered":"2024-05-28T06:25:42.876820+0000","last_clean":"2024-05-28T06:25:42.876820+0000","last_became_active":"2024-05-28T06:24:24.001429+0000","last_became_peered":"2024-05-28T06:24:24.001429+0000","last_unstale":"2024-05-28T06:25:42.876820+0000","last_undegraded":"2024-05-28T06:25:42.876820+0000","last_fullsized":"2024-05-28T06:25:42.876820+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:39:41.026878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":297795584,"num_objects":71,"num_object_clones":0,"num_object_copies":142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":50,"num_read_kb":196610,"num_write":75,"num_write_kb":294913,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":18,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:54.992958+0000","last_change":"2024-05-28T06:24:22.993624+0000","last_active":"2024-05-28T06:24:54.992958+0000","last_peered":"2024-05-28T06:24:54.992958+0000","last_clean":"2024-05-28T06:24:54.992958+0000","last_became_active":"2024-05-28T06:24:22.993301+0000","last_became_peered":"2024-05-28T06:24:22.993301+0000","last_unstale":"2024-05-28T06:24:54.992958+0000","last_undegraded":"2024-05-28T06:24:54.992958+0000","last_fullsized":"2024-05-28T06:24:54.992958+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:26:44.582201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.10","version":"132'69","reported_seq":142,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.196154+0000","last_change":"2024-05-28T06:24:24.005361+0000","last_active":"2024-05-28T06:26:18.196154+0000","last_peered":"2024-05-28T06:26:18.196154+0000","last_clean":"2024-05-28T06:26:18.196154+0000","last_became_active":"2024-05-28T06:24:24.005080+0000","last_became_peered":"2024-05-28T06:24:24.005080+0000","last_unstale":"2024-05-28T06:26:18.196154+0000","last_undegraded":"2024-05-28T06:26:18.196154+0000","last_fullsized":"2024-05-28T06:26:18.196154+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:09:03.314508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":272629760,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":57,"num_read_kb":233472,"num_write":69,"num_write_kb":274432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":18,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:54.994900+0000","last_change":"2024-05-28T06:24:23.005565+0000","last_active":"2024-05-28T06:24:54.994900+0000","last_peered":"2024-05-28T06:24:54.994900+0000","last_clean":"2024-05-28T06:24:54.994900+0000","last_became_active":"2024-05-28T06:24:23.005194+0000","last_became_peered":"2024-05-28T06:24:23.005194+0000","last_unstale":"2024-05-28T06:24:54.994900+0000","last_undegraded":"2024-05-28T06:24:54.994900+0000","last_fullsized":"2024-05-28T06:24:54.994900+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:26:55.784217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.11","version":"132'84","reported_seq":162,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.224702+0000","last_change":"2024-05-28T06:24:24.009724+0000","last_active":"2024-05-28T06:26:18.224702+0000","last_peered":"2024-05-28T06:26:18.224702+0000","last_clean":"2024-05-28T06:26:18.224702+0000","last_became_active":"2024-05-28T06:24:24.009163+0000","last_became_peered":"2024-05-28T06:24:24.009163+0000","last_unstale":"2024-05-28T06:26:18.224702+0000","last_undegraded":"2024-05-28T06:26:18.224702+0000","last_fullsized":"2024-05-28T06:26:18.224702+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:36:46.619746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":318767104,"num_objects":76,"num_object_clones":0,"num_object_copies":152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76,"num_whiteouts":0,"num_read":62,"num_read_kb":253952,"num_write":84,"num_write_kb":323585,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.10","version":"129'8","reported_seq":26,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:55.070167+0000","last_change":"2024-05-28T06:24:22.996927+0000","last_active":"2024-05-28T06:24:55.070167+0000","last_peered":"2024-05-28T06:24:55.070167+0000","last_clean":"2024-05-28T06:24:55.070167+0000","last_became_active":"2024-05-28T06:24:22.996644+0000","last_became_peered":"2024-05-28T06:24:22.996644+0000","last_unstale":"2024-05-28T06:24:55.070167+0000","last_undegraded":"2024-05-28T06:24:55.070167+0000","last_fullsized":"2024-05-28T06:24:55.070167+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:17:26.759155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.e","version":"132'58","reported_seq":98,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.315898+0000","last_change":"2024-05-28T06:24:24.009329+0000","last_active":"2024-05-28T06:26:18.315898+0000","last_peered":"2024-05-28T06:26:18.315898+0000","last_clean":"2024-05-28T06:26:18.315898+0000","last_became_active":"2024-05-28T06:24:24.009033+0000","last_became_peered":"2024-05-28T06:24:24.009033+0000","last_unstale":"2024-05-28T06:26:18.315898+0000","last_undegraded":"2024-05-28T06:26:18.315898+0000","last_fullsized":"2024-05-28T06:26:18.315898+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:23:27.401090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218103808,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":23,"num_read_kb":94208,"num_write":58,"num_write_kb":225280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.f","version":"129'1","reported_seq":20,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:55.456823+0000","last_change":"2024-05-28T06:24:23.006567+0000","last_active":"2024-05-28T06:24:55.456823+0000","last_peered":"2024-05-28T06:24:55.456823+0000","last_clean":"2024-05-28T06:24:55.456823+0000","last_became_active":"2024-05-28T06:24:23.005650+0000","last_became_peered":"2024-05-28T06:24:23.005650+0000","last_unstale":"2024-05-28T06:24:55.456823+0000","last_undegraded":"2024-05-28T06:24:55.456823+0000","last_fullsized":"2024-05-28T06:24:55.456823+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:01:42.283187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.f","version":"132'83","reported_seq":174,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.285828+0000","last_change":"2024-05-28T06:24:24.016763+0000","last_active":"2024-05-28T06:26:18.285828+0000","last_peered":"2024-05-28T06:26:18.285828+0000","last_clean":"2024-05-28T06:26:18.285828+0000","last_became_active":"2024-05-28T06:24:24.016255+0000","last_became_peered":"2024-05-28T06:24:24.016255+0000","last_unstale":"2024-05-28T06:26:18.285828+0000","last_undegraded":"2024-05-28T06:26:18.285828+0000","last_fullsized":"2024-05-28T06:26:18.285828+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:06:19.643800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306184192,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":74,"num_read_kb":303104,"num_write":83,"num_write_kb":315393,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.e","version":"129'2","reported_seq":21,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:54.985300+0000","last_change":"2024-05-28T06:24:23.002513+0000","last_active":"2024-05-28T06:24:54.985300+0000","last_peered":"2024-05-28T06:24:54.985300+0000","last_clean":"2024-05-28T06:24:54.985300+0000","last_became_active":"2024-05-28T06:24:23.001946+0000","last_became_peered":"2024-05-28T06:24:23.001946+0000","last_unstale":"2024-05-28T06:24:54.985300+0000","last_undegraded":"2024-05-28T06:24:54.985300+0000","last_fullsized":"2024-05-28T06:24:54.985300+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:27:48.979163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"132'66","reported_seq":143,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.288093+0000","last_change":"2024-05-28T06:24:23.998032+0000","last_active":"2024-05-28T06:26:18.288093+0000","last_peered":"2024-05-28T06:26:18.288093+0000","last_clean":"2024-05-28T06:26:18.288093+0000","last_became_active":"2024-05-28T06:24:23.996433+0000","last_became_peered":"2024-05-28T06:24:23.996433+0000","last_unstale":"2024-05-28T06:26:18.288093+0000","last_undegraded":"2024-05-28T06:26:18.288093+0000","last_fullsized":"2024-05-28T06:26:18.288093+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:47:25.671388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":230686900,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":60,"num_read_kb":221190,"num_write":66,"num_write_kb":233476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.d","version":"129'1","reported_seq":20,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:54.984968+0000","last_change":"2024-05-28T06:24:22.992776+0000","last_active":"2024-05-28T06:24:54.984968+0000","last_peered":"2024-05-28T06:24:54.984968+0000","last_clean":"2024-05-28T06:24:54.984968+0000","last_became_active":"2024-05-28T06:24:22.992147+0000","last_became_peered":"2024-05-28T06:24:22.992147+0000","last_unstale":"2024-05-28T06:24:54.984968+0000","last_undegraded":"2024-05-28T06:24:54.984968+0000","last_fullsized":"2024-05-28T06:24:54.984968+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:36:09.562056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.d","version":"132'79","reported_seq":168,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.322183+0000","last_change":"2024-05-28T06:24:23.989538+0000","last_active":"2024-05-28T06:26:18.322183+0000","last_peered":"2024-05-28T06:26:18.322183+0000","last_clean":"2024-05-28T06:26:18.322183+0000","last_became_active":"2024-05-28T06:24:23.988631+0000","last_became_peered":"2024-05-28T06:24:23.988631+0000","last_unstale":"2024-05-28T06:26:18.322183+0000","last_undegraded":"2024-05-28T06:26:18.322183+0000","last_fullsized":"2024-05-28T06:26:18.322183+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:08:42.734914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":272629760,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":72,"num_read_kb":294912,"num_write":79,"num_write_kb":294912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":19,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:55.454861+0000","last_change":"2024-05-28T06:24:22.997400+0000","last_active":"2024-05-28T06:24:55.454861+0000","last_peered":"2024-05-28T06:24:55.454861+0000","last_clean":"2024-05-28T06:24:55.454861+0000","last_became_active":"2024-05-28T06:24:22.993636+0000","last_became_peered":"2024-05-28T06:24:22.993636+0000","last_unstale":"2024-05-28T06:24:55.454861+0000","last_undegraded":"2024-05-28T06:24:55.454861+0000","last_fullsized":"2024-05-28T06:24:55.454861+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:43:01.373860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.a","version":"132'79","reported_seq":170,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.293566+0000","last_change":"2024-05-28T06:24:24.007538+0000","last_active":"2024-05-28T06:26:18.293566+0000","last_peered":"2024-05-28T06:26:18.293566+0000","last_clean":"2024-05-28T06:26:18.293566+0000","last_became_active":"2024-05-28T06:24:24.007259+0000","last_became_peered":"2024-05-28T06:24:24.007259+0000","last_unstale":"2024-05-28T06:26:18.293566+0000","last_undegraded":"2024-05-28T06:26:18.293566+0000","last_fullsized":"2024-05-28T06:26:18.293566+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:59:09.840788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306184192,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":74,"num_read_kb":303104,"num_write":79,"num_write_kb":311296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.b","version":"132'3","reported_seq":22,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:25:22.100610+0000","last_change":"2024-05-28T06:24:22.996819+0000","last_active":"2024-05-28T06:25:22.100610+0000","last_peered":"2024-05-28T06:25:22.100610+0000","last_clean":"2024-05-28T06:25:22.100610+0000","last_became_active":"2024-05-28T06:24:22.995489+0000","last_became_peered":"2024-05-28T06:24:22.995489+0000","last_unstale":"2024-05-28T06:25:22.100610+0000","last_undegraded":"2024-05-28T06:25:22.100610+0000","last_fullsized":"2024-05-28T06:25:22.100610+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:24:52.967145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.b","version":"132'90","reported_seq":171,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.295653+0000","last_change":"2024-05-28T06:24:24.015790+0000","last_active":"2024-05-28T06:26:18.295653+0000","last_peered":"2024-05-28T06:26:18.295653+0000","last_clean":"2024-05-28T06:26:18.295653+0000","last_became_active":"2024-05-28T06:24:24.015295+0000","last_became_peered":"2024-05-28T06:24:24.015295+0000","last_unstale":"2024-05-28T06:26:18.295653+0000","last_undegraded":"2024-05-28T06:26:18.295653+0000","last_fullsized":"2024-05-28T06:26:18.295653+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:37:59.260972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":314572932,"num_objects":76,"num_object_clones":0,"num_object_copies":152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76,"num_whiteouts":0,"num_read":64,"num_read_kb":237574,"num_write":90,"num_write_kb":323588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":19,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:55.341579+0000","last_change":"2024-05-28T06:24:22.995343+0000","last_active":"2024-05-28T06:24:55.341579+0000","last_peered":"2024-05-28T06:24:55.341579+0000","last_clean":"2024-05-28T06:24:55.341579+0000","last_became_active":"2024-05-28T06:24:22.994227+0000","last_became_peered":"2024-05-28T06:24:22.994227+0000","last_unstale":"2024-05-28T06:24:55.341579+0000","last_undegraded":"2024-05-28T06:24:55.341579+0000","last_fullsized":"2024-05-28T06:24:55.341579+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:50:39.661313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.8","version":"132'88","reported_seq":146,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.296033+0000","last_change":"2024-05-28T06:24:23.987172+0000","last_active":"2024-05-28T06:26:18.296033+0000","last_peered":"2024-05-28T06:26:18.296033+0000","last_clean":"2024-05-28T06:26:18.296033+0000","last_became_active":"2024-05-28T06:24:23.986897+0000","last_became_peered":"2024-05-28T06:24:23.986897+0000","last_unstale":"2024-05-28T06:26:18.296033+0000","last_undegraded":"2024-05-28T06:26:18.296033+0000","last_fullsized":"2024-05-28T06:26:18.296033+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:11:18.017207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":343932928,"num_objects":82,"num_object_clones":0,"num_object_copies":164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82,"num_whiteouts":0,"num_read":41,"num_read_kb":167936,"num_write":88,"num_write_kb":348160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":19,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:55.341333+0000","last_change":"2024-05-28T06:24:22.991355+0000","last_active":"2024-05-28T06:24:55.341333+0000","last_peered":"2024-05-28T06:24:55.341333+0000","last_clean":"2024-05-28T06:24:55.341333+0000","last_became_active":"2024-05-28T06:24:22.991078+0000","last_became_peered":"2024-05-28T06:24:22.991078+0000","last_unstale":"2024-05-28T06:24:55.341333+0000","last_undegraded":"2024-05-28T06:24:55.341333+0000","last_fullsized":"2024-05-28T06:24:55.341333+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:06:19.878458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"132'68","reported_seq":141,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.282953+0000","last_change":"2024-05-28T06:24:24.009275+0000","last_active":"2024-05-28T06:26:18.282953+0000","last_peered":"2024-05-28T06:26:18.282953+0000","last_clean":"2024-05-28T06:26:18.282953+0000","last_became_active":"2024-05-28T06:24:24.008955+0000","last_became_peered":"2024-05-28T06:24:24.008955+0000","last_unstale":"2024-05-28T06:26:18.282953+0000","last_undegraded":"2024-05-28T06:26:18.282953+0000","last_fullsized":"2024-05-28T06:26:18.282953+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:32:09.823976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":276824064,"num_objects":66,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":56,"num_read_kb":229376,"num_write":68,"num_write_kb":274432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":19,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:54.984300+0000","last_change":"2024-05-28T06:24:23.002572+0000","last_active":"2024-05-28T06:24:54.984300+0000","last_peered":"2024-05-28T06:24:54.984300+0000","last_clean":"2024-05-28T06:24:54.984300+0000","last_became_active":"2024-05-28T06:24:23.002292+0000","last_became_peered":"2024-05-28T06:24:23.002292+0000","last_unstale":"2024-05-28T06:24:54.984300+0000","last_undegraded":"2024-05-28T06:24:54.984300+0000","last_fullsized":"2024-05-28T06:24:54.984300+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:50:39.856735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.6","version":"132'87","reported_seq":139,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.318611+0000","last_change":"2024-05-28T06:24:24.017529+0000","last_active":"2024-05-28T06:26:18.318611+0000","last_peered":"2024-05-28T06:26:18.318611+0000","last_clean":"2024-05-28T06:26:18.318611+0000","last_became_active":"2024-05-28T06:24:24.017221+0000","last_became_peered":"2024-05-28T06:24:24.017221+0000","last_unstale":"2024-05-28T06:26:18.318611+0000","last_undegraded":"2024-05-28T06:26:18.318611+0000","last_fullsized":"2024-05-28T06:26:18.318611+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:32:09.823976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306184192,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":35,"num_read_kb":143360,"num_write":87,"num_write_kb":327680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.7","version":"129'2","reported_seq":21,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:54.994004+0000","last_change":"2024-05-28T06:24:23.003960+0000","last_active":"2024-05-28T06:24:54.994004+0000","last_peered":"2024-05-28T06:24:54.994004+0000","last_clean":"2024-05-28T06:24:54.994004+0000","last_became_active":"2024-05-28T06:24:23.003502+0000","last_became_peered":"2024-05-28T06:24:23.003502+0000","last_unstale":"2024-05-28T06:24:54.994004+0000","last_undegraded":"2024-05-28T06:24:54.994004+0000","last_fullsized":"2024-05-28T06:24:54.994004+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:13:19.450560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.7","version":"132'84","reported_seq":171,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.313436+0000","last_change":"2024-05-28T06:24:24.020448+0000","last_active":"2024-05-28T06:26:18.313436+0000","last_peered":"2024-05-28T06:26:18.313436+0000","last_clean":"2024-05-28T06:26:18.313436+0000","last_became_active":"2024-05-28T06:24:24.019730+0000","last_became_peered":"2024-05-28T06:24:24.019730+0000","last_unstale":"2024-05-28T06:26:18.313436+0000","last_undegraded":"2024-05-28T06:26:18.313436+0000","last_fullsized":"2024-05-28T06:26:18.313436+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:26:56.800134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":310378496,"num_objects":74,"num_object_clones":0,"num_object_copies":148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74,"num_whiteouts":0,"num_read":70,"num_read_kb":286720,"num_write":84,"num_write_kb":323584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.6","version":"132'41","reported_seq":60,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:21.025193+0000","last_change":"2024-05-28T06:24:23.003133+0000","last_active":"2024-05-28T06:26:21.025193+0000","last_peered":"2024-05-28T06:26:21.025193+0000","last_clean":"2024-05-28T06:26:21.025193+0000","last_became_active":"2024-05-28T06:24:23.002821+0000","last_became_peered":"2024-05-28T06:24:23.002821+0000","last_unstale":"2024-05-28T06:26:21.025193+0000","last_undegraded":"2024-05-28T06:26:21.025193+0000","last_fullsized":"2024-05-28T06:26:21.025193+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:18:06.253798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":95,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.4","version":"132'80","reported_seq":184,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.322862+0000","last_change":"2024-05-28T06:24:24.004928+0000","last_active":"2024-05-28T06:26:18.322862+0000","last_peered":"2024-05-28T06:26:18.322862+0000","last_clean":"2024-05-28T06:26:18.322862+0000","last_became_active":"2024-05-28T06:24:24.004599+0000","last_became_peered":"2024-05-28T06:24:24.004599+0000","last_unstale":"2024-05-28T06:26:18.322862+0000","last_undegraded":"2024-05-28T06:26:18.322862+0000","last_fullsized":"2024-05-28T06:26:18.322862+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:09:56.044033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":276824064,"num_objects":66,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":87,"num_read_kb":356352,"num_write":80,"num_write_kb":294913,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":19,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:54.996483+0000","last_change":"2024-05-28T06:24:23.003603+0000","last_active":"2024-05-28T06:24:54.996483+0000","last_peered":"2024-05-28T06:24:54.996483+0000","last_clean":"2024-05-28T06:24:54.996483+0000","last_became_active":"2024-05-28T06:24:23.002468+0000","last_became_peered":"2024-05-28T06:24:23.002468+0000","last_unstale":"2024-05-28T06:24:54.996483+0000","last_undegraded":"2024-05-28T06:24:54.996483+0000","last_fullsized":"2024-05-28T06:24:54.996483+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:48:27.218311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.5","version":"132'86","reported_seq":149,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.318905+0000","last_change":"2024-05-28T06:24:23.999563+0000","last_active":"2024-05-28T06:26:18.318905+0000","last_peered":"2024-05-28T06:26:18.318905+0000","last_clean":"2024-05-28T06:26:18.318905+0000","last_became_active":"2024-05-28T06:24:23.999013+0000","last_became_peered":"2024-05-28T06:24:23.999013+0000","last_unstale":"2024-05-28T06:26:18.318905+0000","last_undegraded":"2024-05-28T06:26:18.318905+0000","last_fullsized":"2024-05-28T06:26:18.318905+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:47:25.671388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":285213316,"num_objects":72,"num_object_clones":1,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72,"num_whiteouts":1,"num_read":47,"num_read_kb":192512,"num_write":84,"num_write_kb":303107,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":19,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:54.985023+0000","last_change":"2024-05-28T06:24:22.986876+0000","last_active":"2024-05-28T06:24:54.985023+0000","last_peered":"2024-05-28T06:24:54.985023+0000","last_clean":"2024-05-28T06:24:54.985023+0000","last_became_active":"2024-05-28T06:24:22.986620+0000","last_became_peered":"2024-05-28T06:24:22.986620+0000","last_unstale":"2024-05-28T06:24:54.985023+0000","last_undegraded":"2024-05-28T06:24:54.985023+0000","last_fullsized":"2024-05-28T06:24:54.985023+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:20:23.711600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2","version":"132'90","reported_seq":159,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.297667+0000","last_change":"2024-05-28T06:24:24.012293+0000","last_active":"2024-05-28T06:26:18.297667+0000","last_peered":"2024-05-28T06:26:18.297667+0000","last_clean":"2024-05-28T06:26:18.297667+0000","last_became_active":"2024-05-28T06:24:24.011989+0000","last_became_peered":"2024-05-28T06:24:24.011989+0000","last_unstale":"2024-05-28T06:26:18.297667+0000","last_undegraded":"2024-05-28T06:26:18.297667+0000","last_fullsized":"2024-05-28T06:26:18.297667+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:47:31.291023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":352321536,"num_objects":84,"num_object_clones":0,"num_object_copies":168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84,"num_whiteouts":0,"num_read":52,"num_read_kb":212992,"num_write":90,"num_write_kb":356352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":19,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:54.995987+0000","last_change":"2024-05-28T06:24:23.008363+0000","last_active":"2024-05-28T06:24:54.995987+0000","last_peered":"2024-05-28T06:24:54.995987+0000","last_clean":"2024-05-28T06:24:54.995987+0000","last_became_active":"2024-05-28T06:24:23.007859+0000","last_became_peered":"2024-05-28T06:24:23.007859+0000","last_unstale":"2024-05-28T06:24:54.995987+0000","last_undegraded":"2024-05-28T06:24:54.995987+0000","last_fullsized":"2024-05-28T06:24:54.995987+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:45:32.156309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"132'98","reported_seq":198,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.323247+0000","last_change":"2024-05-28T06:24:24.013906+0000","last_active":"2024-05-28T06:26:18.323247+0000","last_peered":"2024-05-28T06:26:18.323247+0000","last_clean":"2024-05-28T06:26:18.323247+0000","last_became_active":"2024-05-28T06:24:24.011269+0000","last_became_peered":"2024-05-28T06:24:24.011269+0000","last_unstale":"2024-05-28T06:26:18.323247+0000","last_undegraded":"2024-05-28T06:26:18.323247+0000","last_fullsized":"2024-05-28T06:26:18.323247+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:09:03.314508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369098752,"num_objects":88,"num_object_clones":0,"num_object_copies":176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88,"num_whiteouts":0,"num_read":83,"num_read_kb":327683,"num_write":98,"num_write_kb":376833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":19,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:55.133578+0000","last_change":"2024-05-28T06:24:23.006728+0000","last_active":"2024-05-28T06:24:55.133578+0000","last_peered":"2024-05-28T06:24:55.133578+0000","last_clean":"2024-05-28T06:24:55.133578+0000","last_became_active":"2024-05-28T06:24:23.005895+0000","last_became_peered":"2024-05-28T06:24:23.005895+0000","last_unstale":"2024-05-28T06:24:55.133578+0000","last_undegraded":"2024-05-28T06:24:55.133578+0000","last_fullsized":"2024-05-28T06:24:55.133578+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:53:06.353639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"132'87","reported_seq":175,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.310121+0000","last_change":"2024-05-28T06:24:24.013072+0000","last_active":"2024-05-28T06:26:18.310121+0000","last_peered":"2024-05-28T06:26:18.310121+0000","last_clean":"2024-05-28T06:26:18.310121+0000","last_became_active":"2024-05-28T06:24:24.012814+0000","last_became_peered":"2024-05-28T06:24:24.012814+0000","last_unstale":"2024-05-28T06:26:18.310121+0000","last_undegraded":"2024-05-28T06:26:18.310121+0000","last_fullsized":"2024-05-28T06:26:18.310121+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:32:09.823976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":339738624,"num_objects":81,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81,"num_whiteouts":0,"num_read":71,"num_read_kb":290816,"num_write":87,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1","version":"132'31","reported_seq":80,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.171426+0000","last_change":"2024-05-28T06:24:23.001199+0000","last_active":"2024-05-28T06:26:18.171426+0000","last_peered":"2024-05-28T06:26:18.171426+0000","last_clean":"2024-05-28T06:26:18.171426+0000","last_became_active":"2024-05-28T06:24:23.000861+0000","last_became_peered":"2024-05-28T06:24:23.000861+0000","last_unstale":"2024-05-28T06:26:18.171426+0000","last_undegraded":"2024-05-28T06:26:18.171426+0000","last_fullsized":"2024-05-28T06:26:18.171426+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:36:09.562056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4553,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":30,"num_read_kb":30,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1","version":"132'93","reported_seq":186,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.322559+0000","last_change":"2024-05-28T06:24:24.005940+0000","last_active":"2024-05-28T06:26:18.322559+0000","last_peered":"2024-05-28T06:26:18.322559+0000","last_clean":"2024-05-28T06:26:18.322559+0000","last_became_active":"2024-05-28T06:24:24.005485+0000","last_became_peered":"2024-05-28T06:24:24.005485+0000","last_unstale":"2024-05-28T06:26:18.322559+0000","last_undegraded":"2024-05-28T06:26:18.322559+0000","last_fullsized":"2024-05-28T06:26:18.322559+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:52:44.413787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":314572800,"num_objects":75,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75,"num_whiteouts":0,"num_read":76,"num_read_kb":311296,"num_write":93,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":19,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:54.984220+0000","last_change":"2024-05-28T06:24:22.987451+0000","last_active":"2024-05-28T06:24:54.984220+0000","last_peered":"2024-05-28T06:24:54.984220+0000","last_clean":"2024-05-28T06:24:54.984220+0000","last_became_active":"2024-05-28T06:24:22.987181+0000","last_became_peered":"2024-05-28T06:24:22.987181+0000","last_unstale":"2024-05-28T06:24:54.984220+0000","last_undegraded":"2024-05-28T06:24:54.984220+0000","last_fullsized":"2024-05-28T06:24:54.984220+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:45:51.036866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.12","version":"132'65","reported_seq":128,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.240342+0000","last_change":"2024-05-28T06:24:24.007933+0000","last_active":"2024-05-28T06:26:18.240342+0000","last_peered":"2024-05-28T06:26:18.240342+0000","last_clean":"2024-05-28T06:26:18.240342+0000","last_became_active":"2024-05-28T06:24:24.007651+0000","last_became_peered":"2024-05-28T06:24:24.007651+0000","last_unstale":"2024-05-28T06:26:18.240342+0000","last_undegraded":"2024-05-28T06:26:18.240342+0000","last_fullsized":"2024-05-28T06:26:18.240342+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:59:32.549372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":255852544,"num_objects":61,"num_object_clones":0,"num_object_copies":122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":0,"num_read":47,"num_read_kb":192512,"num_write":65,"num_write_kb":258048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"129'2","reported_seq":20,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:55.133764+0000","last_change":"2024-05-28T06:24:23.008078+0000","last_active":"2024-05-28T06:24:55.133764+0000","last_peered":"2024-05-28T06:24:55.133764+0000","last_clean":"2024-05-28T06:24:55.133764+0000","last_became_active":"2024-05-28T06:24:23.007331+0000","last_became_peered":"2024-05-28T06:24:23.007331+0000","last_unstale":"2024-05-28T06:24:55.133764+0000","last_undegraded":"2024-05-28T06:24:55.133764+0000","last_fullsized":"2024-05-28T06:24:55.133764+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:50:39.661313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":388,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:55.134179+0000","last_change":"2024-05-28T06:04:01.039129+0000","last_active":"2024-05-28T06:24:55.134179+0000","last_peered":"2024-05-28T06:24:55.134179+0000","last_clean":"2024-05-28T06:24:55.134179+0000","last_became_active":"2024-05-28T06:04:01.033906+0000","last_became_peered":"2024-05-28T06:04:01.033906+0000","last_unstale":"2024-05-28T06:24:55.134179+0000","last_undegraded":"2024-05-28T06:24:55.134179+0000","last_fullsized":"2024-05-28T06:24:55.134179+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:03:59.864237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:03:59.864237+0000","last_clean_scrub_stamp":"2024-05-28T06:03:59.864237+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:17:36.440245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":18396218954,"num_objects":4395,"num_object_clones":1,"num_object_copies":8790,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4395,"num_whiteouts":1,"num_read":3946,"num_read_kb":15908926,"num_write":4949,"num_write_kb":18956331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":41943040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36792500224,"data_stored":36792437908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4958,"ondisk_log_size":4958,"up":128,"acting":128,"num_store_stats":8},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":1678588,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":30,"num_read_kb":30,"num_write":598,"num_write_kb":1879,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3424256,"data_stored":3357176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":544,"ondisk_log_size":544,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575123,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3952528,"kb_used_data":3813128,"kb_used_omap":1,"kb_used_meta":139390,"kb_avail":90419312,"statfs":{"total":96636764160,"available":92589375488,"internally_reserved":0,"allocated":3904643072,"data_stored":3903694392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":142735818},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575125,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5139788,"kb_used_data":5000388,"kb_used_omap":1,"kb_used_meta":139390,"kb_avail":89232052,"statfs":{"total":96636764160,"available":91373621248,"internally_reserved":0,"allocated":5120397312,"data_stored":5119456102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":142735814},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575123,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3815132,"kb_used_data":3675732,"kb_used_omap":1,"kb_used_meta":139390,"kb_avail":90556708,"statfs":{"total":96636764160,"available":92730068992,"internally_reserved":0,"allocated":3763949568,"data_stored":3763018100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":142735813},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575123,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5547692,"kb_used_data":5412452,"kb_used_omap":1,"kb_used_meta":135230,"kb_avail":88824148,"statfs":{"total":96636764160,"available":90955927552,"internally_reserved":0,"allocated":5542350848,"data_stored":5541404120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":138475973},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575125,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4527844,"kb_used_data":4388444,"kb_used_omap":1,"kb_used_meta":139390,"kb_avail":89843996,"statfs":{"total":96636764160,"available":92000251904,"internally_reserved":0,"allocated":4493766656,"data_stored":4492831645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":142735815},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575124,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5961384,"kb_used_data":5826144,"kb_used_omap":1,"kb_used_meta":135230,"kb_avail":88410456,"statfs":{"total":96636764160,"available":90532306944,"internally_reserved":0,"allocated":5965971456,"data_stored":5965028896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":138475973},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575124,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4155024,"kb_used_data":4019784,"kb_used_omap":1,"kb_used_meta":135230,"kb_avail":90216816,"statfs":{"total":96636764160,"available":92382019584,"internally_reserved":0,"allocated":4116258816,"data_stored":4115339392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":138475973},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575123,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3959192,"kb_used_data":3811472,"kb_used_omap":1,"kb_used_meta":147710,"kb_avail":90412648,"statfs":{"total":96636764160,"available":92582551552,"internally_reserved":0,"allocated":3902947328,"data_stored":3902020517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151255493},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1691648,"data_stored":1677790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1695744,"data_stored":1673875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3900706816,"data_stored":3900702981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4114612224,"data_stored":4114612224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":5964316672,"data_stored":5964301112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4492103680,"data_stored":4492099764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":5540700160,"data_stored":5540676882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3762294784,"data_stored":3762290820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":5117059072,"data_stored":5117051144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3900706816,"data_stored":3900702981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-28T06:26:44.913 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-28T06:26:45.145 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:45.151+0000 7fc677200640 1 Processor -- start 2024-05-28T06:26:45.145 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:45.151+0000 7fc677200640 1 -- start start 2024-05-28T06:26:45.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:45.153+0000 7fc677200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc67815b3a0 0x7fc6780b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:45.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:45.153+0000 7fc677200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fc67806dc20 0x7fc67815f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:45.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:45.153+0000 7fc677200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6780afd60 con 0x7fc678072c00 2024-05-28T06:26:45.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:45.153+0000 7fc677200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fc67806f980 con 0x7fc67806dc20 2024-05-28T06:26:45.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:45.153+0000 7fc677200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fc6780ae0c0 con 0x7fc67815b3a0 2024-05-28T06:26:45.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:45.153+0000 7fc675400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fc67806dc20 0x7fc67815f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:45.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:45.153+0000 7fc675400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fc67806dc20 0x7fc67815f9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35110/0 (socket says 172.21.15.96:35110) 2024-05-28T06:26:45.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:45.153+0000 7fc675400640 1 -- 172.21.15.96:0/243535636 learned_addr learned my addr 172.21.15.96:0/243535636 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:26:45.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:45.153+0000 7fc675e00640 1 --2- 172.21.15.96:0/243535636 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc67815b3a0 0x7fc6780b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:45.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:45.153+0000 7fc675400640 1 -- 172.21.15.96:0/243535636 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc67815b3a0 msgr2=0x7fc6780b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:45.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:45.153+0000 7fc675400640 1 --2- 172.21.15.96:0/243535636 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc67815b3a0 0x7fc6780b32c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:45.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:45.153+0000 7fc675400640 1 -- 172.21.15.96:0/243535636 >> v1:172.21.15.123:6789/0 conn(0x7fc678072c00 legacy=0x7fc6780ae550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:45.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:45.154+0000 7fc675400640 1 -- 172.21.15.96:0/243535636 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6780af7a0 con 0x7fc67806dc20 2024-05-28T06:26:45.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:45.154+0000 7fc675400640 1 --2- 172.21.15.96:0/243535636 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc67806dc20 0x7fc67815f9a0 secure :-1 s=READY pgs=2186 cs=0 l=1 rev1=1 crypto rx=0x7fc660009e50 tx=0x7fc66002a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47c7a5dfe99bf6d server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:45.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:45.155+0000 7fc674a00640 1 -- 172.21.15.96:0/243535636 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc660036070 con 0x7fc67806dc20 2024-05-28T06:26:45.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:45.155+0000 7fc674a00640 1 -- 172.21.15.96:0/243535636 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc660031600 con 0x7fc67806dc20 2024-05-28T06:26:45.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:45.155+0000 7fc674a00640 1 -- 172.21.15.96:0/243535636 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6600319d0 con 0x7fc67806dc20 2024-05-28T06:26:45.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:45.155+0000 7fc677200640 1 -- 172.21.15.96:0/243535636 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc67806dc20 msgr2=0x7fc67815f9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:45.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:45.155+0000 7fc677200640 1 --2- 172.21.15.96:0/243535636 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc67806dc20 0x7fc67815f9a0 secure :-1 s=READY pgs=2186 cs=0 l=1 rev1=1 crypto rx=0x7fc660009e50 tx=0x7fc66002a6c0 comp rx=0 tx=0).stop 2024-05-28T06:26:45.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:45.156+0000 7fc677200640 1 -- 172.21.15.96:0/243535636 shutdown_connections 2024-05-28T06:26:45.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:45.156+0000 7fc677200640 1 --2- 172.21.15.96:0/243535636 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc67806dc20 0x7fc67815f9a0 unknown :-1 s=CLOSED pgs=2186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:45.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:45.156+0000 7fc677200640 1 --2- 172.21.15.96:0/243535636 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc67815b3a0 0x7fc6780b32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:45.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:45.156+0000 7fc677200640 1 -- 172.21.15.96:0/243535636 >> 172.21.15.96:0/243535636 conn(0x7fc6780b1fc0 msgr2=0x7fc6780b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:26:45.151 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:45.156+0000 7fc677200640 1 -- 172.21.15.96:0/243535636 shutdown_connections 2024-05-28T06:26:45.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:45.156+0000 7fc677200640 1 -- 172.21.15.96:0/243535636 wait complete. 2024-05-28T06:26:45.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:45.157+0000 7fc677200640 1 Processor -- start 2024-05-28T06:26:45.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:45.157+0000 7fc677200640 1 -- start start 2024-05-28T06:26:45.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:45.158+0000 7fc677200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc67806dc20 0x7fc67815dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:45.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:45.158+0000 7fc677200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc678072c00 0x7fc678148b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:45.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:45.158+0000 7fc677200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc67815b3a0 0x7fc6781490a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:45.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:45.158+0000 7fc677200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc67815b780 con 0x7fc67806dc20 2024-05-28T06:26:45.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:45.158+0000 7fc677200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc67815fee0 con 0x7fc678072c00 2024-05-28T06:26:45.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:45.158+0000 7fc677200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fc678073050 con 0x7fc67815b3a0 2024-05-28T06:26:45.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:45.159+0000 7fc675e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc67806dc20 0x7fc67815dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:45.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:45.159+0000 7fc675e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc67806dc20 0x7fc67815dbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35112/0 (socket says 172.21.15.96:35112) 2024-05-28T06:26:45.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:45.159+0000 7fc675e00640 1 -- 172.21.15.96:0/2594655219 learned_addr learned my addr 172.21.15.96:0/2594655219 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:26:45.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:45.159+0000 7fc675400640 1 --2- 172.21.15.96:0/2594655219 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc678072c00 0x7fc678148b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:45.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:45.159+0000 7fc675e00640 1 -- 172.21.15.96:0/2594655219 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc67815b3a0 msgr2=0x7fc6781490a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:45.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:45.159+0000 7fc676800640 1 --2- 172.21.15.96:0/2594655219 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc67815b3a0 0x7fc6781490a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:45.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:45.159+0000 7fc675e00640 1 --2- 172.21.15.96:0/2594655219 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc67815b3a0 0x7fc6781490a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:45.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:45.159+0000 7fc675e00640 1 -- 172.21.15.96:0/2594655219 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc678072c00 msgr2=0x7fc678148b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:45.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:45.159+0000 7fc675e00640 1 --2- 172.21.15.96:0/2594655219 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc678072c00 0x7fc678148b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:45.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:45.159+0000 7fc675e00640 1 -- 172.21.15.96:0/2594655219 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc678165480 con 0x7fc67806dc20 2024-05-28T06:26:45.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:45.159+0000 7fc675400640 1 --2- 172.21.15.96:0/2594655219 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc678072c00 0x7fc678148b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:26:45.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:45.159+0000 7fc676800640 1 --2- 172.21.15.96:0/2594655219 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc67815b3a0 0x7fc6781490a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:26:45.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:45.159+0000 7fc675e00640 1 --2- 172.21.15.96:0/2594655219 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc67806dc20 0x7fc67815dbc0 secure :-1 s=READY pgs=2187 cs=0 l=1 rev1=1 crypto rx=0x7fc66800d5f0 tx=0x7fc66800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:45.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:45.160+0000 7fc66e000640 1 -- 172.21.15.96:0/2594655219 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6680125f0 con 0x7fc67806dc20 2024-05-28T06:26:45.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:45.160+0000 7fc66e000640 1 -- 172.21.15.96:0/2594655219 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc668019070 con 0x7fc67806dc20 2024-05-28T06:26:45.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:45.160+0000 7fc66e000640 1 -- 172.21.15.96:0/2594655219 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6680129d0 con 0x7fc67806dc20 2024-05-28T06:26:45.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:45.160+0000 7fc677200640 1 -- 172.21.15.96:0/2594655219 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc678165760 con 0x7fc67806dc20 2024-05-28T06:26:45.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:45.160+0000 7fc677200640 1 -- 172.21.15.96:0/2594655219 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc678165d20 con 0x7fc67806dc20 2024-05-28T06:26:45.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:45.161+0000 7fc677200640 1 -- 172.21.15.96:0/2594655219 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6780689e0 con 0x7fc67806dc20 2024-05-28T06:26:45.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:45.162+0000 7fc66e000640 1 -- 172.21.15.96:0/2594655219 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 49) ==== 94049+0+0 (secure 0 0 0) 0x7fc668012b70 con 0x7fc67806dc20 2024-05-28T06:26:45.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:45.163+0000 7fc66e000640 1 --2- 172.21.15.96:0/2594655219 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc65406ea20 0x7fc654070ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:45.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:45.163+0000 7fc66e000640 1 -- 172.21.15.96:0/2594655219 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 5823+0+0 (secure 0 0 0) 0x7fc66808b110 con 0x7fc67806dc20 2024-05-28T06:26:45.159 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:45.163+0000 7fc675400640 1 --2- 172.21.15.96:0/2594655219 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc65406ea20 0x7fc654070ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:45.159 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:45.164+0000 7fc675400640 1 --2- 172.21.15.96:0/2594655219 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc65406ea20 0x7fc654070ee0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fc660033000 tx=0x7fc660009000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:45.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:45.167+0000 7fc66e000640 1 -- 172.21.15.96:0/2594655219 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc66805afa0 con 0x7fc67806dc20 2024-05-28T06:26:45.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:45.366+0000 7fc677200640 1 -- 172.21.15.96:0/2594655219 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc6780cfd60 con 0x7fc67806dc20 2024-05-28T06:26:45.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:45.369+0000 7fc66e000640 1 -- 172.21.15.96:0/2594655219 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v132) ==== 74+0+16877 (secure 0 0 0) 0x7fc66805a8a0 con 0x7fc67806dc20 2024-05-28T06:26:45.363 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:26:45.363 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":132,"fsid":"e4a831b5-e6d1-47a7-96cb-bf85154ee0d2","created":"2024-05-28T06:03:46.454885+0000","modified":"2024-05-28T06:24:54.967903+0000","last_up_change":"2024-05-28T06:03:58.831135+0000","last_in_change":"2024-05-28T06:03:49.786426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-28T06:03:59.517102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-05-28T06:24:21.166134+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"129","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-05-28T06:24:22.415938+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"130","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6cb568f-6a4b-4921-942f-3595eac140a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6809","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6811","nonce":1243655428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6813","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6815","nonce":1243655428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6825","nonce":1243655428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6817","nonce":1243655428},{"type":"v1","addr":"172.21.15.96:6819","nonce":1243655428}]},"public_addr":"172.21.15.96:6811/1243655428","cluster_addr":"172.21.15.96:6815/1243655428","heartbeat_back_addr":"172.21.15.96:6825/1243655428","heartbeat_front_addr":"172.21.15.96:6819/1243655428","state":["exists","up"]},{"osd":1,"uuid":"50ee4c07-7f99-4217-8245-964866eecd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6801","nonce":4059252009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6803","nonce":4059252009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6807","nonce":4059252009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":4059252009},{"type":"v1","addr":"172.21.15.96:6805","nonce":4059252009}]},"public_addr":"172.21.15.96:6801/4059252009","cluster_addr":"172.21.15.96:6803/4059252009","heartbeat_back_addr":"172.21.15.96:6807/4059252009","heartbeat_front_addr":"172.21.15.96:6805/4059252009","state":["exists","up"]},{"osd":2,"uuid":"98441e44-a96f-4969-a671-947eb707a6f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6823","nonce":3070158265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6827","nonce":3070158265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6831","nonce":3070158265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":3070158265},{"type":"v1","addr":"172.21.15.96:6829","nonce":3070158265}]},"public_addr":"172.21.15.96:6823/3070158265","cluster_addr":"172.21.15.96:6827/3070158265","heartbeat_back_addr":"172.21.15.96:6831/3070158265","heartbeat_front_addr":"172.21.15.96:6829/3070158265","state":["exists","up"]},{"osd":3,"uuid":"bf43d98d-903b-4ba0-b8a7-040a9bfbe05d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6810","nonce":2384313931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6814","nonce":2384313931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6821","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6824","nonce":2384313931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2384313931},{"type":"v1","addr":"172.21.15.96:6818","nonce":2384313931}]},"public_addr":"172.21.15.96:6810/2384313931","cluster_addr":"172.21.15.96:6814/2384313931","heartbeat_back_addr":"172.21.15.96:6824/2384313931","heartbeat_front_addr":"172.21.15.96:6818/2384313931","state":["exists","up"]},{"osd":4,"uuid":"275f4739-a589-49e9-9c6d-671ab5716338","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6801","nonce":3445772216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6803","nonce":3445772216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6807","nonce":3445772216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3445772216},{"type":"v1","addr":"172.21.15.123:6805","nonce":3445772216}]},"public_addr":"172.21.15.123:6801/3445772216","cluster_addr":"172.21.15.123:6803/3445772216","heartbeat_back_addr":"172.21.15.123:6807/3445772216","heartbeat_front_addr":"172.21.15.123:6805/3445772216","state":["exists","up"]},{"osd":5,"uuid":"be79dc9d-ad70-484b-8341-0e0bd36de687","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6817","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6819","nonce":2041107525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6821","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6823","nonce":2041107525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6831","nonce":2041107525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6825","nonce":2041107525},{"type":"v1","addr":"172.21.15.123:6827","nonce":2041107525}]},"public_addr":"172.21.15.123:6819/2041107525","cluster_addr":"172.21.15.123:6823/2041107525","heartbeat_back_addr":"172.21.15.123:6831/2041107525","heartbeat_front_addr":"172.21.15.123:6827/2041107525","state":["exists","up"]},{"osd":6,"uuid":"6397e1cd-f914-4f53-87b7-c95a3dcb16f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6809","nonce":1217470164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6811","nonce":1217470164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6815","nonce":1217470164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":1217470164},{"type":"v1","addr":"172.21.15.123:6813","nonce":1217470164}]},"public_addr":"172.21.15.123:6809/1217470164","cluster_addr":"172.21.15.123:6811/1217470164","heartbeat_back_addr":"172.21.15.123:6815/1217470164","heartbeat_front_addr":"172.21.15.123:6813/1217470164","state":["exists","up"]},{"osd":7,"uuid":"be2a4947-16f1-4d46-b732-d511a2c19fae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6818","nonce":2085300276}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6822","nonce":2085300276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6829","nonce":2085300276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2085300276},{"type":"v1","addr":"172.21.15.123:6826","nonce":2085300276}]},"public_addr":"172.21.15.123:6818/2085300276","cluster_addr":"172.21.15.123:6822/2085300276","heartbeat_back_addr":"172.21.15.123:6829/2085300276","heartbeat_front_addr":"172.21.15.123:6826/2085300276","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.152861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.340079+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.048575+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.134858+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.149471+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.185166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:54.959518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-28T06:03:55.210765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"19.32","mappings":[{"from":2,"to":0}]},{"pgid":"19.3a","mappings":[{"from":2,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":13,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-28T06:26:45.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:45.371+0000 7fc677200640 1 -- 172.21.15.96:0/2594655219 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc65406ea20 msgr2=0x7fc654070ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:45.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:45.371+0000 7fc677200640 1 --2- 172.21.15.96:0/2594655219 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc65406ea20 0x7fc654070ee0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fc660033000 tx=0x7fc660009000 comp rx=0 tx=0).stop 2024-05-28T06:26:45.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:45.371+0000 7fc677200640 1 -- 172.21.15.96:0/2594655219 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc67806dc20 msgr2=0x7fc67815dbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:45.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:45.371+0000 7fc677200640 1 --2- 172.21.15.96:0/2594655219 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc67806dc20 0x7fc67815dbc0 secure :-1 s=READY pgs=2187 cs=0 l=1 rev1=1 crypto rx=0x7fc66800d5f0 tx=0x7fc66800dab0 comp rx=0 tx=0).stop 2024-05-28T06:26:45.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:45.372+0000 7fc677200640 1 -- 172.21.15.96:0/2594655219 shutdown_connections 2024-05-28T06:26:45.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:45.372+0000 7fc677200640 1 --2- 172.21.15.96:0/2594655219 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc65406ea20 0x7fc654070ee0 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:45.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:45.372+0000 7fc677200640 1 --2- 172.21.15.96:0/2594655219 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc67815b3a0 0x7fc6781490a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:45.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:45.372+0000 7fc677200640 1 --2- 172.21.15.96:0/2594655219 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc678072c00 0x7fc678148b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:45.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:45.372+0000 7fc677200640 1 --2- 172.21.15.96:0/2594655219 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc67806dc20 0x7fc67815dbc0 unknown :-1 s=CLOSED pgs=2187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:45.367 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:45.372+0000 7fc677200640 1 -- 172.21.15.96:0/2594655219 >> 172.21.15.96:0/2594655219 conn(0x7fc6780b1fc0 msgr2=0x7fc6780b3a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:26:45.367 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:45.372+0000 7fc677200640 1 -- 172.21.15.96:0/2594655219 shutdown_connections 2024-05-28T06:26:45.367 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:45.372+0000 7fc677200640 1 -- 172.21.15.96:0/2594655219 wait complete. 2024-05-28T06:26:46.385 INFO:tasks.ceph:Scrubbing osd.0 2024-05-28T06:26:46.386 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-05-28T06:26:46.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.610+0000 7f2e71c00640 1 Processor -- start 2024-05-28T06:26:46.604 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.610+0000 7f2e71c00640 1 -- start start 2024-05-28T06:26:46.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.612+0000 7f2e71c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2e6c15b720 0x7f2e6c15bb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:46.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.612+0000 7f2e71c00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f2e6c15c0c0 0x7f2e6c0b3300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:46.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.612+0000 7f2e71c00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e6c0aeda0 con 0x7f2e6c15b340 2024-05-28T06:26:46.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.612+0000 7f2e71c00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f2e6c15c510 con 0x7f2e6c15c0c0 2024-05-28T06:26:46.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.612+0000 7f2e71c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f2e6c0af9b0 con 0x7f2e6c15b720 2024-05-28T06:26:46.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.612+0000 7f2e6aa00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f2e6c15b340 0x7f2e6c06f550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:44868/0 (socket says 172.21.15.96:44868) 2024-05-28T06:26:46.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.612+0000 7f2e6aa00640 1 -- 172.21.15.96:0/3420217777 learned_addr learned my addr 172.21.15.96:0/3420217777 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:26:46.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.612+0000 7f2e6b400640 1 --2- 172.21.15.96:0/3420217777 >> v2:172.21.15.96:3300/0 conn(0x7f2e6c15c0c0 0x7f2e6c0b3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:46.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.613+0000 7f2e62000640 1 --2- 172.21.15.96:0/3420217777 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2e6c15b720 0x7f2e6c15bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:46.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.613+0000 7f2e6a000640 1 -- 172.21.15.96:0/3420217777 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 893525241 0 0) 0x7f2e6c0aeda0 con 0x7f2e6c15b340 2024-05-28T06:26:46.607 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.613+0000 7f2e6a000640 1 -- 172.21.15.96:0/3420217777 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2e50005730 con 0x7f2e6c15b340 2024-05-28T06:26:46.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.613+0000 7f2e6b400640 1 -- 172.21.15.96:0/3420217777 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2e6c15b720 msgr2=0x7f2e6c15bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:46.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.613+0000 7f2e6b400640 1 --2- 172.21.15.96:0/3420217777 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2e6c15b720 0x7f2e6c15bb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:46.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.613+0000 7f2e6b400640 1 -- 172.21.15.96:0/3420217777 >> v1:172.21.15.96:6789/0 conn(0x7f2e6c15b340 legacy=0x7f2e6c06f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:46.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.613+0000 7f2e6b400640 1 -- 172.21.15.96:0/3420217777 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e6c0af080 con 0x7f2e6c15c0c0 2024-05-28T06:26:46.608 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.614+0000 7f2e6b400640 1 --2- 172.21.15.96:0/3420217777 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2e6c15c0c0 0x7f2e6c0b3300 secure :-1 s=READY pgs=2189 cs=0 l=1 rev1=1 crypto rx=0x7f2e64009d90 tx=0x7f2e6402a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f285fac82a12a0b7 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:46.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.614+0000 7f2e6a000640 1 -- 172.21.15.96:0/3420217777 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e64007740 con 0x7f2e6c15c0c0 2024-05-28T06:26:46.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.614+0000 7f2e6a000640 1 -- 172.21.15.96:0/3420217777 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e640078e0 con 0x7f2e6c15c0c0 2024-05-28T06:26:46.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.614+0000 7f2e6a000640 1 -- 172.21.15.96:0/3420217777 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e64007cb0 con 0x7f2e6c15c0c0 2024-05-28T06:26:46.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.615+0000 7f2e71c00640 1 -- 172.21.15.96:0/3420217777 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2e6c15c0c0 msgr2=0x7f2e6c0b3300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:46.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.615+0000 7f2e71c00640 1 --2- 172.21.15.96:0/3420217777 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2e6c15c0c0 0x7f2e6c0b3300 secure :-1 s=READY pgs=2189 cs=0 l=1 rev1=1 crypto rx=0x7f2e64009d90 tx=0x7f2e6402a660 comp rx=0 tx=0).stop 2024-05-28T06:26:46.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.615+0000 7f2e71c00640 1 -- 172.21.15.96:0/3420217777 shutdown_connections 2024-05-28T06:26:46.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.615+0000 7f2e71c00640 1 --2- 172.21.15.96:0/3420217777 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2e6c15c0c0 0x7f2e6c0b3300 unknown :-1 s=CLOSED pgs=2189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:46.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.615+0000 7f2e71c00640 1 --2- 172.21.15.96:0/3420217777 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2e6c15b720 0x7f2e6c15bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:46.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.615+0000 7f2e71c00640 1 -- 172.21.15.96:0/3420217777 >> 172.21.15.96:0/3420217777 conn(0x7f2e6c0b1e50 msgr2=0x7f2e6c0b2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:26:46.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.615+0000 7f2e71c00640 1 -- 172.21.15.96:0/3420217777 shutdown_connections 2024-05-28T06:26:46.610 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.615+0000 7f2e71c00640 1 -- 172.21.15.96:0/3420217777 wait complete. 2024-05-28T06:26:46.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.617+0000 7f2e71c00640 1 Processor -- start 2024-05-28T06:26:46.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.617+0000 7f2e71c00640 1 -- start start 2024-05-28T06:26:46.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.618+0000 7f2e71c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2e6c15b340 0x7f2e6c1fc010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:46.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.618+0000 7f2e71c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e6c15b720 0x7f2e6c158aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:46.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.618+0000 7f2e71c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2e6c15c0c0 0x7f2e6c158fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:46.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.618+0000 7f2e71c00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f2e6c0aea30 con 0x7f2e6c15b340 2024-05-28T06:26:46.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.618+0000 7f2e71c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f2e6c0af640 con 0x7f2e6c15b720 2024-05-28T06:26:46.613 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.618+0000 7f2e71c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f2e6c15c8c0 con 0x7f2e6c15c0c0 2024-05-28T06:26:46.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.618+0000 7f2e6aa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2e6c15b340 0x7f2e6c1fc010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:46.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.618+0000 7f2e6aa00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2e6c15b340 0x7f2e6c1fc010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35124/0 (socket says 172.21.15.96:35124) 2024-05-28T06:26:46.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.618+0000 7f2e6aa00640 1 -- 172.21.15.96:0/740546888 learned_addr learned my addr 172.21.15.96:0/740546888 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:26:46.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.619+0000 7f2e62000640 1 --2- 172.21.15.96:0/740546888 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e6c15b720 0x7f2e6c158aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:46.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.619+0000 7f2e6b400640 1 --2- 172.21.15.96:0/740546888 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2e6c15c0c0 0x7f2e6c158fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:46.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.619+0000 7f2e6aa00640 1 -- 172.21.15.96:0/740546888 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2e6c15c0c0 msgr2=0x7f2e6c158fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:46.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.619+0000 7f2e6aa00640 1 --2- 172.21.15.96:0/740546888 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f2e6c15c0c0 0x7f2e6c158fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:46.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.619+0000 7f2e6aa00640 1 -- 172.21.15.96:0/740546888 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e6c15b720 msgr2=0x7f2e6c158aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:46.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.619+0000 7f2e6aa00640 1 --2- 172.21.15.96:0/740546888 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e6c15b720 0x7f2e6c158aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:46.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.619+0000 7f2e6aa00640 1 -- 172.21.15.96:0/740546888 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e6c156040 con 0x7f2e6c15b340 2024-05-28T06:26:46.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.619+0000 7f2e6aa00640 1 --2- 172.21.15.96:0/740546888 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2e6c15b340 0x7f2e6c1fc010 secure :-1 s=READY pgs=2190 cs=0 l=1 rev1=1 crypto rx=0x7f2e5c00c430 tx=0x7f2e5c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:46.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.619+0000 7f2e63400640 1 -- 172.21.15.96:0/740546888 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e5c017070 con 0x7f2e6c15b340 2024-05-28T06:26:46.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.619+0000 7f2e63400640 1 -- 172.21.15.96:0/740546888 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e5c005ce0 con 0x7f2e6c15b340 2024-05-28T06:26:46.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.619+0000 7f2e63400640 1 -- 172.21.15.96:0/740546888 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e5c0072f0 con 0x7f2e6c15b340 2024-05-28T06:26:46.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.619+0000 7f2e71c00640 1 -- 172.21.15.96:0/740546888 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e6c156320 con 0x7f2e6c15b340 2024-05-28T06:26:46.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.620+0000 7f2e71c00640 1 -- 172.21.15.96:0/740546888 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e6c1568e0 con 0x7f2e6c15b340 2024-05-28T06:26:46.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.621+0000 7f2e71c00640 1 -- 172.21.15.96:0/740546888 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2e6c0689e0 con 0x7f2e6c15b340 2024-05-28T06:26:46.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.622+0000 7f2e63400640 1 -- 172.21.15.96:0/740546888 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 49) ==== 94049+0+0 (secure 0 0 0) 0x7f2e5c01a070 con 0x7f2e6c15b340 2024-05-28T06:26:46.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.623+0000 7f2e63400640 1 --2- 172.21.15.96:0/740546888 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2e4806ea20 0x7f2e48070ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:46.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.623+0000 7f2e63400640 1 -- 172.21.15.96:0/740546888 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 5823+0+0 (secure 0 0 0) 0x7f2e5c08a150 con 0x7f2e6c15b340 2024-05-28T06:26:46.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.623+0000 7f2e63400640 1 --2- 172.21.15.96:0/740546888 >> [v2:172.21.15.96:6809/1243655428,v1:172.21.15.96:6811/1243655428] conn(0x7f2e48076d40 0x7f2e480791c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:46.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.623+0000 7f2e63400640 1 -- 172.21.15.96:0/740546888 --> [v2:172.21.15.96:6809/1243655428,v1:172.21.15.96:6811/1243655428] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2e5c015e70 con 0x7f2e48076d40 2024-05-28T06:26:46.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.623+0000 7f2e63400640 1 -- 172.21.15.96:0/740546888 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_get_version_reply(handle=1 version=132) ==== 24+0+0 (secure 0 0 0) 0x7f2e5c096050 con 0x7f2e6c15b340 2024-05-28T06:26:46.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.623+0000 7f2e6b400640 1 --2- 172.21.15.96:0/740546888 >> [v2:172.21.15.96:6809/1243655428,v1:172.21.15.96:6811/1243655428] conn(0x7f2e48076d40 0x7f2e480791c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:46.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.624+0000 7f2e62000640 1 --2- 172.21.15.96:0/740546888 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2e4806ea20 0x7f2e48070ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:46.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.624+0000 7f2e6b400640 1 --2- 172.21.15.96:0/740546888 >> [v2:172.21.15.96:6809/1243655428,v1:172.21.15.96:6811/1243655428] conn(0x7f2e48076d40 0x7f2e480791c0 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:46.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.625+0000 7f2e62000640 1 --2- 172.21.15.96:0/740546888 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2e4806ea20 0x7f2e48070ee0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f2e6c15fc60 tx=0x7f2e54008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:46.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.625+0000 7f2e63400640 1 -- 172.21.15.96:0/740546888 <== osd.0 v2:172.21.15.96:6809/1243655428 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2e5c015e70 con 0x7f2e48076d40 2024-05-28T06:26:46.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.640+0000 7f2e71c00640 1 -- 172.21.15.96:0/740546888 --> [v2:172.21.15.96:6809/1243655428,v1:172.21.15.96:6811/1243655428] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f2e6c159f70 con 0x7f2e48076d40 2024-05-28T06:26:46.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.640+0000 7f2e63400640 1 -- 172.21.15.96:0/740546888 <== osd.0 v2:172.21.15.96:6809/1243655428 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f2e6c159f70 con 0x7f2e48076d40 2024-05-28T06:26:46.634 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:26:46.634 INFO:teuthology.orchestra.run.smithi096.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-28T06:26:46.639 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:26:46.640 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.641+0000 7f2e71c00640 1 -- 172.21.15.96:0/740546888 >> [v2:172.21.15.96:6809/1243655428,v1:172.21.15.96:6811/1243655428] conn(0x7f2e48076d40 msgr2=0x7f2e480791c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:46.640 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.641+0000 7f2e71c00640 1 --2- 172.21.15.96:0/740546888 >> [v2:172.21.15.96:6809/1243655428,v1:172.21.15.96:6811/1243655428] conn(0x7f2e48076d40 0x7f2e480791c0 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:46.640 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.641+0000 7f2e71c00640 1 -- 172.21.15.96:0/740546888 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2e4806ea20 msgr2=0x7f2e48070ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:46.640 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.641+0000 7f2e71c00640 1 --2- 172.21.15.96:0/740546888 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f2e4806ea20 0x7f2e48070ee0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f2e6c15fc60 tx=0x7f2e54008040 comp rx=0 tx=0).stop 2024-05-28T06:26:46.640 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.641+0000 7f2e71c00640 1 -- 172.21.15.96:0/740546888 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2e6c15b340 msgr2=0x7f2e6c1fc010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:46.640 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.641+0000 7f2e71c00640 1 --2- 172.21.15.96:0/740546888 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2e6c15b340 0x7f2e6c1fc010 secure :-1 s=READY pgs=2190 cs=0 l=1 rev1=1 crypto rx=0x7f2e5c00c430 tx=0x7f2e5c00c8f0 comp rx=0 tx=0).stop 2024-05-28T06:26:46.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.641+0000 7f2e6b400640 1 -- 172.21.15.96:0/740546888 reap_dead start 2024-05-28T06:26:46.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.641+0000 7f2e71c00640 1 -- 172.21.15.96:0/740546888 shutdown_connections 2024-05-28T06:26:46.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.641+0000 7f2e71c00640 1 -- 172.21.15.96:0/740546888 >> 172.21.15.96:0/740546888 conn(0x7f2e6c0b1e50 msgr2=0x7f2e6c0b36c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:26:46.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.641+0000 7f2e71c00640 1 -- 172.21.15.96:0/740546888 shutdown_connections 2024-05-28T06:26:46.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.641+0000 7f2e71c00640 1 -- 172.21.15.96:0/740546888 wait complete. 2024-05-28T06:26:46.654 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-05-28T06:26:46.860 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.867+0000 7fc928200640 1 Processor -- start 2024-05-28T06:26:46.860 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.867+0000 7fc928200640 1 -- start start 2024-05-28T06:26:46.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.868+0000 7fc928200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fc9200d0590 0x7fc9200d0a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:46.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.868+0000 7fc928200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc920099090 0x7fc9200994e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:46.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.868+0000 7fc928200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9200b51f0 con 0x7fc9200b4e10 2024-05-28T06:26:46.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.868+0000 7fc928200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fc9200aef60 con 0x7fc9200d0590 2024-05-28T06:26:46.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.868+0000 7fc928200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fc9200ad1d0 con 0x7fc920099090 2024-05-28T06:26:46.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.868+0000 7fc925800640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fc9200b4e10 0x7fc9200cde30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:44884/0 (socket says 172.21.15.96:44884) 2024-05-28T06:26:46.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.868+0000 7fc925800640 1 -- 172.21.15.96:0/3025788972 learned_addr learned my addr 172.21.15.96:0/3025788972 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:26:46.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.869+0000 7fc924e00640 1 --2- 172.21.15.96:0/3025788972 >> v2:172.21.15.123:3300/0 conn(0x7fc9200d0590 0x7fc9200d0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:46.865 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.869+0000 7fc926200640 1 --2- 172.21.15.96:0/3025788972 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc920099090 0x7fc9200994e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:46.865 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.869+0000 7fc91fe00640 1 -- 172.21.15.96:0/3025788972 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3628392379 0 0) 0x7fc9200b51f0 con 0x7fc9200b4e10 2024-05-28T06:26:46.865 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.869+0000 7fc91fe00640 1 -- 172.21.15.96:0/3025788972 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc908003520 con 0x7fc9200b4e10 2024-05-28T06:26:46.865 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.869+0000 7fc91fe00640 1 -- 172.21.15.96:0/3025788972 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2005513876 0 0) 0x7fc908003520 con 0x7fc9200b4e10 2024-05-28T06:26:46.865 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.869+0000 7fc91fe00640 1 -- 172.21.15.96:0/3025788972 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc920099090 msgr2=0x7fc9200994e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:46.865 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.869+0000 7fc91fe00640 1 --2- 172.21.15.96:0/3025788972 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc920099090 0x7fc9200994e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:46.865 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.869+0000 7fc91fe00640 1 -- 172.21.15.96:0/3025788972 >> v2:172.21.15.123:3300/0 conn(0x7fc9200d0590 msgr2=0x7fc9200d0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:46.865 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.869+0000 7fc91fe00640 1 --2- 172.21.15.96:0/3025788972 >> v2:172.21.15.123:3300/0 conn(0x7fc9200d0590 0x7fc9200d0a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:46.866 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.869+0000 7fc91fe00640 1 -- 172.21.15.96:0/3025788972 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9200ad470 con 0x7fc9200b4e10 2024-05-28T06:26:46.866 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.869+0000 7fc91fe00640 1 -- 172.21.15.96:0/3025788972 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7fc914001670 con 0x7fc9200b4e10 2024-05-28T06:26:46.866 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.869+0000 7fc91fe00640 1 -- 172.21.15.96:0/3025788972 >> v1:172.21.15.96:6789/0 conn(0x7fc9200b4e10 legacy=0x7fc9200cde30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:46.866 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.869+0000 7fc91fe00640 1 --2- 172.21.15.96:0/3025788972 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc9080039b0 0x7fc908003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:46.866 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.869+0000 7fc91fe00640 1 --2- 172.21.15.96:0/3025788972 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc908004800 0x7fc908004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:46.866 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.870+0000 7fc91fe00640 1 --2- 172.21.15.96:0/3025788972 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc920099090 0x7fc9080051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:46.866 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.870+0000 7fc91fe00640 1 -- 172.21.15.96:0/3025788972 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc9200aef60 con 0x7fc9080039b0 2024-05-28T06:26:46.867 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.870+0000 7fc91fe00640 1 -- 172.21.15.96:0/3025788972 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc9200ad1d0 con 0x7fc908004800 2024-05-28T06:26:46.867 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.870+0000 7fc91fe00640 1 -- 172.21.15.96:0/3025788972 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fc90800dbd0 con 0x7fc920099090 2024-05-28T06:26:46.867 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.870+0000 7fc925800640 1 --2- 172.21.15.96:0/3025788972 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc9080039b0 0x7fc908003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:46.867 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.870+0000 7fc924e00640 1 --2- 172.21.15.96:0/3025788972 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc908004800 0x7fc908004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:46.867 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.870+0000 7fc926200640 1 --2- 172.21.15.96:0/3025788972 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc920099090 0x7fc9080051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:46.867 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.870+0000 7fc925800640 1 -- 172.21.15.96:0/3025788972 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc920099090 msgr2=0x7fc9080051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:46.867 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.870+0000 7fc925800640 1 --2- 172.21.15.96:0/3025788972 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc920099090 0x7fc9080051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:46.867 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.870+0000 7fc925800640 1 -- 172.21.15.96:0/3025788972 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc908004800 msgr2=0x7fc908004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:46.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.870+0000 7fc925800640 1 --2- 172.21.15.96:0/3025788972 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc908004800 0x7fc908004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:46.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.870+0000 7fc925800640 1 -- 172.21.15.96:0/3025788972 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc908003520 con 0x7fc9080039b0 2024-05-28T06:26:46.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.870+0000 7fc926200640 1 --2- 172.21.15.96:0/3025788972 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc920099090 0x7fc9080051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:26:46.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.871+0000 7fc925800640 1 --2- 172.21.15.96:0/3025788972 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc9080039b0 0x7fc908003dd0 secure :-1 s=READY pgs=2192 cs=0 l=1 rev1=1 crypto rx=0x7fc91400c210 tx=0x7fc908005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f28c58706dd6ecf3 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:46.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.872+0000 7fc91fe00640 1 -- 172.21.15.96:0/3025788972 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc914007d50 con 0x7fc9080039b0 2024-05-28T06:26:46.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.872+0000 7fc91fe00640 1 -- 172.21.15.96:0/3025788972 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc914005ad0 con 0x7fc9080039b0 2024-05-28T06:26:46.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.872+0000 7fc91fe00640 1 -- 172.21.15.96:0/3025788972 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc914007070 con 0x7fc9080039b0 2024-05-28T06:26:46.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.872+0000 7fc928200640 1 -- 172.21.15.96:0/3025788972 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc9080039b0 msgr2=0x7fc908003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:46.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.872+0000 7fc928200640 1 --2- 172.21.15.96:0/3025788972 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc9080039b0 0x7fc908003dd0 secure :-1 s=READY pgs=2192 cs=0 l=1 rev1=1 crypto rx=0x7fc91400c210 tx=0x7fc908005830 comp rx=0 tx=0).stop 2024-05-28T06:26:46.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.872+0000 7fc926200640 1 -- 172.21.15.96:0/3025788972 reap_dead start 2024-05-28T06:26:46.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.872+0000 7fc928200640 1 -- 172.21.15.96:0/3025788972 shutdown_connections 2024-05-28T06:26:46.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.872+0000 7fc928200640 1 -- 172.21.15.96:0/3025788972 >> 172.21.15.96:0/3025788972 conn(0x7fc92006bae0 msgr2=0x7fc92006bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:26:46.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.872+0000 7fc928200640 1 -- 172.21.15.96:0/3025788972 shutdown_connections 2024-05-28T06:26:46.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.873+0000 7fc928200640 1 -- 172.21.15.96:0/3025788972 wait complete. 2024-05-28T06:26:46.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.874+0000 7fc928200640 1 Processor -- start 2024-05-28T06:26:46.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.874+0000 7fc928200640 1 -- start start 2024-05-28T06:26:46.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.875+0000 7fc928200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc9080039b0 0x7fc9201f75b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:46.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.875+0000 7fc928200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc9200afdc0 0x7fc9200b0220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:46.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.875+0000 7fc928200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc9200cd0a0 0x7fc9200cd520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:46.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.875+0000 7fc928200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc9200ce880 con 0x7fc9080039b0 2024-05-28T06:26:46.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.875+0000 7fc928200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc9200ce540 con 0x7fc9200cd0a0 2024-05-28T06:26:46.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.875+0000 7fc928200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fc9200cea00 con 0x7fc9200afdc0 2024-05-28T06:26:46.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.876+0000 7fc925800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc9080039b0 0x7fc9201f75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:46.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.876+0000 7fc925800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc9080039b0 0x7fc9201f75b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35142/0 (socket says 172.21.15.96:35142) 2024-05-28T06:26:46.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.876+0000 7fc925800640 1 -- 172.21.15.96:0/3850440584 learned_addr learned my addr 172.21.15.96:0/3850440584 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:26:46.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.876+0000 7fc926200640 1 --2- 172.21.15.96:0/3850440584 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc9200cd0a0 0x7fc9200cd520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:46.873 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.876+0000 7fc924e00640 1 --2- 172.21.15.96:0/3850440584 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc9200afdc0 0x7fc9200b0220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:46.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.876+0000 7fc925800640 1 -- 172.21.15.96:0/3850440584 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc9200afdc0 msgr2=0x7fc9200b0220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:46.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.876+0000 7fc925800640 1 --2- 172.21.15.96:0/3850440584 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc9200afdc0 0x7fc9200b0220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:46.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.876+0000 7fc925800640 1 -- 172.21.15.96:0/3850440584 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc9200cd0a0 msgr2=0x7fc9200cd520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:46.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.876+0000 7fc925800640 1 --2- 172.21.15.96:0/3850440584 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc9200cd0a0 0x7fc9200cd520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:46.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.876+0000 7fc925800640 1 -- 172.21.15.96:0/3850440584 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9201f7af0 con 0x7fc9080039b0 2024-05-28T06:26:46.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.876+0000 7fc924e00640 1 --2- 172.21.15.96:0/3850440584 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc9200afdc0 0x7fc9200b0220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:26:46.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.876+0000 7fc925800640 1 --2- 172.21.15.96:0/3850440584 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc9080039b0 0x7fc9201f75b0 secure :-1 s=READY pgs=2193 cs=0 l=1 rev1=1 crypto rx=0x7fc91400c300 tx=0x7fc9200af0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:46.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.877+0000 7fc91d600640 1 -- 172.21.15.96:0/3850440584 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc914042020 con 0x7fc9080039b0 2024-05-28T06:26:46.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.877+0000 7fc91d600640 1 -- 172.21.15.96:0/3850440584 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc914007280 con 0x7fc9080039b0 2024-05-28T06:26:46.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.877+0000 7fc91d600640 1 -- 172.21.15.96:0/3850440584 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc914007d50 con 0x7fc9080039b0 2024-05-28T06:26:46.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.877+0000 7fc928200640 1 -- 172.21.15.96:0/3850440584 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc92015bbe0 con 0x7fc9080039b0 2024-05-28T06:26:46.875 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.877+0000 7fc928200640 1 -- 172.21.15.96:0/3850440584 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc92015c1a0 con 0x7fc9080039b0 2024-05-28T06:26:46.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.878+0000 7fc928200640 1 -- 172.21.15.96:0/3850440584 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9200cda60 con 0x7fc9080039b0 2024-05-28T06:26:46.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.879+0000 7fc91d600640 1 -- 172.21.15.96:0/3850440584 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 49) ==== 94049+0+0 (secure 0 0 0) 0x7fc914045020 con 0x7fc9080039b0 2024-05-28T06:26:46.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.880+0000 7fc91d600640 1 --2- 172.21.15.96:0/3850440584 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc90006ea20 0x7fc900070ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:46.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.880+0000 7fc91d600640 1 -- 172.21.15.96:0/3850440584 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 5823+0+0 (secure 0 0 0) 0x7fc9140ab470 con 0x7fc9080039b0 2024-05-28T06:26:46.876 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.881+0000 7fc924e00640 1 --2- 172.21.15.96:0/3850440584 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc90006ea20 0x7fc900070ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:46.877 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.882+0000 7fc924e00640 1 --2- 172.21.15.96:0/3850440584 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc90006ea20 0x7fc900070ee0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fc9200ca7c0 tx=0x7fc90c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:46.878 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:46.885+0000 7fc91d600640 1 -- 172.21.15.96:0/3850440584 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc91404c030 con 0x7fc9080039b0 2024-05-28T06:26:47.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.088+0000 7fc928200640 1 -- 172.21.15.96:0/3850440584 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7fc9200689e0 con 0x7fc90006ea20 2024-05-28T06:26:47.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.089+0000 7fc91d600640 1 -- 172.21.15.96:0/3850440584 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fc9200689e0 con 0x7fc90006ea20 2024-05-28T06:26:47.083 INFO:teuthology.orchestra.run.smithi096.stderr:instructed osd(s) 0 to deep-scrub 2024-05-28T06:26:47.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.092+0000 7fc928200640 1 -- 172.21.15.96:0/3850440584 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc90006ea20 msgr2=0x7fc900070ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:47.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.092+0000 7fc928200640 1 --2- 172.21.15.96:0/3850440584 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc90006ea20 0x7fc900070ee0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fc9200ca7c0 tx=0x7fc90c008040 comp rx=0 tx=0).stop 2024-05-28T06:26:47.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.092+0000 7fc928200640 1 -- 172.21.15.96:0/3850440584 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc9080039b0 msgr2=0x7fc9201f75b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:47.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.092+0000 7fc928200640 1 --2- 172.21.15.96:0/3850440584 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc9080039b0 0x7fc9201f75b0 secure :-1 s=READY pgs=2193 cs=0 l=1 rev1=1 crypto rx=0x7fc91400c300 tx=0x7fc9200af0e0 comp rx=0 tx=0).stop 2024-05-28T06:26:47.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.092+0000 7fc928200640 1 -- 172.21.15.96:0/3850440584 shutdown_connections 2024-05-28T06:26:47.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.092+0000 7fc928200640 1 --2- 172.21.15.96:0/3850440584 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc90006ea20 0x7fc900070ee0 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:47.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.092+0000 7fc928200640 1 --2- 172.21.15.96:0/3850440584 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc9200cd0a0 0x7fc9200cd520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:47.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.092+0000 7fc928200640 1 --2- 172.21.15.96:0/3850440584 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc9200afdc0 0x7fc9200b0220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:47.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.092+0000 7fc928200640 1 --2- 172.21.15.96:0/3850440584 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc9080039b0 0x7fc9201f75b0 unknown :-1 s=CLOSED pgs=2193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:47.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.092+0000 7fc928200640 1 -- 172.21.15.96:0/3850440584 >> 172.21.15.96:0/3850440584 conn(0x7fc92006bae0 msgr2=0x7fc92006e880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:26:47.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.092+0000 7fc928200640 1 -- 172.21.15.96:0/3850440584 shutdown_connections 2024-05-28T06:26:47.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.093+0000 7fc928200640 1 -- 172.21.15.96:0/3850440584 wait complete. 2024-05-28T06:26:47.106 INFO:tasks.ceph:Scrubbing osd.1 2024-05-28T06:26:47.107 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-05-28T06:26:47.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.317+0000 7f524e600640 1 Processor -- start 2024-05-28T06:26:47.311 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.317+0000 7f524e600640 1 -- start start 2024-05-28T06:26:47.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.319+0000 7f524e600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f524815c2c0 0x7f5248166460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:47.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.319+0000 7f524e600640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52480aeda0 con 0x7f524815b340 2024-05-28T06:26:47.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.319+0000 7f524e600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52480af080 con 0x7f524815b720 2024-05-28T06:26:47.312 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.319+0000 7f524e600640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f52480af9b0 con 0x7f524815c2c0 2024-05-28T06:26:47.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.319+0000 7f5247e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f524815c2c0 0x7f5248166460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:47.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.319+0000 7f5247400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f524815b340 0x7f524806f550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:44886/0 (socket says 172.21.15.96:44886) 2024-05-28T06:26:47.313 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.319+0000 7f5247400640 1 -- 172.21.15.96:0/2197531350 learned_addr learned my addr 172.21.15.96:0/2197531350 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:26:47.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.319+0000 7f5246000640 1 -- 172.21.15.96:0/2197531350 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 286317899 0 0) 0x7f52480aeda0 con 0x7f524815b340 2024-05-28T06:26:47.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.320+0000 7f5246000640 1 -- 172.21.15.96:0/2197531350 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f522c005730 con 0x7f524815b340 2024-05-28T06:26:47.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.320+0000 7f5247e00640 1 -- 172.21.15.96:0/2197531350 >> v1:172.21.15.123:6789/0 conn(0x7f524815b720 legacy=0x7f524815bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:47.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.320+0000 7f5247e00640 1 -- 172.21.15.96:0/2197531350 >> v1:172.21.15.96:6789/0 conn(0x7f524815b340 legacy=0x7f524806f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:47.314 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.320+0000 7f5247e00640 1 -- 172.21.15.96:0/2197531350 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52480af360 con 0x7f524815c2c0 2024-05-28T06:26:47.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.321+0000 7f5247e00640 1 --2- 172.21.15.96:0/2197531350 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f524815c2c0 0x7f5248166460 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 crypto rx=0x7f5240009d90 tx=0x7f524002a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3eac75f4de373b68 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:47.315 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.321+0000 7f5246000640 1 -- 172.21.15.96:0/2197531350 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5240007740 con 0x7f524815c2c0 2024-05-28T06:26:47.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.321+0000 7f5246000640 1 -- 172.21.15.96:0/2197531350 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52400078e0 con 0x7f524815c2c0 2024-05-28T06:26:47.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.321+0000 7f5246000640 1 -- 172.21.15.96:0/2197531350 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5240007cb0 con 0x7f524815c2c0 2024-05-28T06:26:47.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.322+0000 7f524e600640 1 -- 172.21.15.96:0/2197531350 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f524815c2c0 msgr2=0x7f5248166460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:47.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.322+0000 7f524e600640 1 --2- 172.21.15.96:0/2197531350 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f524815c2c0 0x7f5248166460 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 crypto rx=0x7f5240009d90 tx=0x7f524002a6a0 comp rx=0 tx=0).stop 2024-05-28T06:26:47.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.322+0000 7f524e600640 1 -- 172.21.15.96:0/2197531350 shutdown_connections 2024-05-28T06:26:47.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.322+0000 7f524e600640 1 --2- 172.21.15.96:0/2197531350 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f524815c2c0 0x7f5248166460 unknown :-1 s=CLOSED pgs=2195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:47.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.322+0000 7f524e600640 1 -- 172.21.15.96:0/2197531350 >> 172.21.15.96:0/2197531350 conn(0x7f52480b1e50 msgr2=0x7f52480b2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:26:47.316 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.322+0000 7f524e600640 1 -- 172.21.15.96:0/2197531350 shutdown_connections 2024-05-28T06:26:47.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.322+0000 7f524e600640 1 -- 172.21.15.96:0/2197531350 wait complete. 2024-05-28T06:26:47.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.324+0000 7f524e600640 1 Processor -- start 2024-05-28T06:26:47.317 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.324+0000 7f524e600640 1 -- start start 2024-05-28T06:26:47.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.325+0000 7f524e600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f524815b340 0x7f52481f4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:47.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.325+0000 7f524e600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f524815b720 0x7f5248156380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:47.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.325+0000 7f524e600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f524815c2c0 0x7f52480cebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:47.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.325+0000 7f524e600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f524815c8c0 con 0x7f524815b340 2024-05-28T06:26:47.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.325+0000 7f524e600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f52480aea30 con 0x7f524815c2c0 2024-05-28T06:26:47.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.325+0000 7f524e600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f52480b4310 con 0x7f524815b720 2024-05-28T06:26:47.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.325+0000 7f5247400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f524815b340 0x7f52481f4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:47.320 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.325+0000 7f5247400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f524815b340 0x7f52481f4fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35156/0 (socket says 172.21.15.96:35156) 2024-05-28T06:26:47.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.325+0000 7f5247400640 1 -- 172.21.15.96:0/3137901130 learned_addr learned my addr 172.21.15.96:0/3137901130 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:26:47.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.325+0000 7f5247400640 1 -- 172.21.15.96:0/3137901130 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f524815b720 msgr2=0x7f5248156380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:47.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.325+0000 7f5247400640 1 --2- 172.21.15.96:0/3137901130 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f524815b720 0x7f5248156380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:47.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.325+0000 7f5247400640 1 -- 172.21.15.96:0/3137901130 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f524815c2c0 msgr2=0x7f52480cebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:47.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.325+0000 7f5247e00640 1 --2- 172.21.15.96:0/3137901130 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f524815c2c0 0x7f52480cebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:47.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.326+0000 7f5247400640 1 --2- 172.21.15.96:0/3137901130 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f524815c2c0 0x7f52480cebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:47.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.326+0000 7f5247400640 1 -- 172.21.15.96:0/3137901130 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52480ccdb0 con 0x7f524815b340 2024-05-28T06:26:47.321 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.326+0000 7f5247400640 1 --2- 172.21.15.96:0/3137901130 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f524815b340 0x7f52481f4fb0 secure :-1 s=READY pgs=2196 cs=0 l=1 rev1=1 crypto rx=0x7f523800c430 tx=0x7f523800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:47.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.326+0000 7f523f400640 1 -- 172.21.15.96:0/3137901130 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5238017070 con 0x7f524815b340 2024-05-28T06:26:47.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.326+0000 7f523f400640 1 -- 172.21.15.96:0/3137901130 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5238005ce0 con 0x7f524815b340 2024-05-28T06:26:47.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.326+0000 7f523f400640 1 -- 172.21.15.96:0/3137901130 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52380072f0 con 0x7f524815b340 2024-05-28T06:26:47.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.326+0000 7f524e600640 1 -- 172.21.15.96:0/3137901130 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52481568c0 con 0x7f524815b340 2024-05-28T06:26:47.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.326+0000 7f524e600640 1 -- 172.21.15.96:0/3137901130 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52480cd330 con 0x7f524815b340 2024-05-28T06:26:47.323 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.327+0000 7f524e600640 1 -- 172.21.15.96:0/3137901130 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f52480689e0 con 0x7f524815b340 2024-05-28T06:26:47.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.329+0000 7f523f400640 1 -- 172.21.15.96:0/3137901130 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 49) ==== 94049+0+0 (secure 0 0 0) 0x7f523801a070 con 0x7f524815b340 2024-05-28T06:26:47.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.330+0000 7f523f400640 1 --2- 172.21.15.96:0/3137901130 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f522406ea50 0x7f5224070f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:47.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.330+0000 7f523f400640 1 -- 172.21.15.96:0/3137901130 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 5823+0+0 (secure 0 0 0) 0x7f523808a8d0 con 0x7f524815b340 2024-05-28T06:26:47.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.330+0000 7f523f400640 1 --2- 172.21.15.96:0/3137901130 >> [v2:172.21.15.96:6800/4059252009,v1:172.21.15.96:6801/4059252009] conn(0x7f5224076d70 0x7f52240791f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:47.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.330+0000 7f523f400640 1 -- 172.21.15.96:0/3137901130 --> [v2:172.21.15.96:6800/4059252009,v1:172.21.15.96:6801/4059252009] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f52240798c0 con 0x7f5224076d70 2024-05-28T06:26:47.324 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.330+0000 7f523f400640 1 -- 172.21.15.96:0/3137901130 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_get_version_reply(handle=1 version=132) ==== 24+0+0 (secure 0 0 0) 0x7f523808bd80 con 0x7f524815b340 2024-05-28T06:26:47.325 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.330+0000 7f5247e00640 1 --2- 172.21.15.96:0/3137901130 >> [v2:172.21.15.96:6800/4059252009,v1:172.21.15.96:6801/4059252009] conn(0x7f5224076d70 0x7f52240791f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:47.325 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.330+0000 7f5246a00640 1 --2- 172.21.15.96:0/3137901130 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f522406ea50 0x7f5224070f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:47.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.331+0000 7f5247e00640 1 --2- 172.21.15.96:0/3137901130 >> [v2:172.21.15.96:6800/4059252009,v1:172.21.15.96:6801/4059252009] conn(0x7f5224076d70 0x7f52240791f0 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:47.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.332+0000 7f5246a00640 1 --2- 172.21.15.96:0/3137901130 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f522406ea50 0x7f5224070f10 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f524815e550 tx=0x7f52300079b0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:47.326 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.332+0000 7f523f400640 1 -- 172.21.15.96:0/3137901130 <== osd.1 v2:172.21.15.96:6800/4059252009 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f52240798c0 con 0x7f5224076d70 2024-05-28T06:26:47.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.346+0000 7f524e600640 1 -- 172.21.15.96:0/3137901130 --> [v2:172.21.15.96:6800/4059252009,v1:172.21.15.96:6801/4059252009] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f52480b4e60 con 0x7f5224076d70 2024-05-28T06:26:47.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.347+0000 7f523f400640 1 -- 172.21.15.96:0/3137901130 <== osd.1 v2:172.21.15.96:6800/4059252009 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f52480b4e60 con 0x7f5224076d70 2024-05-28T06:26:47.340 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:26:47.341 INFO:teuthology.orchestra.run.smithi096.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-28T06:26:47.341 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:26:47.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.347+0000 7f524e600640 1 -- 172.21.15.96:0/3137901130 >> [v2:172.21.15.96:6800/4059252009,v1:172.21.15.96:6801/4059252009] conn(0x7f5224076d70 msgr2=0x7f52240791f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:47.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.347+0000 7f524e600640 1 --2- 172.21.15.96:0/3137901130 >> [v2:172.21.15.96:6800/4059252009,v1:172.21.15.96:6801/4059252009] conn(0x7f5224076d70 0x7f52240791f0 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:47.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.347+0000 7f524e600640 1 -- 172.21.15.96:0/3137901130 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f522406ea50 msgr2=0x7f5224070f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:47.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.347+0000 7f524e600640 1 --2- 172.21.15.96:0/3137901130 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f522406ea50 0x7f5224070f10 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f524815e550 tx=0x7f52300079b0 comp rx=0 tx=0).stop 2024-05-28T06:26:47.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.347+0000 7f524e600640 1 -- 172.21.15.96:0/3137901130 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f524815b340 msgr2=0x7f52481f4fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:47.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.347+0000 7f524e600640 1 --2- 172.21.15.96:0/3137901130 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f524815b340 0x7f52481f4fb0 secure :-1 s=READY pgs=2196 cs=0 l=1 rev1=1 crypto rx=0x7f523800c430 tx=0x7f523800c8f0 comp rx=0 tx=0).stop 2024-05-28T06:26:47.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.347+0000 7f5247e00640 1 -- 172.21.15.96:0/3137901130 reap_dead start 2024-05-28T06:26:47.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.348+0000 7f524e600640 1 -- 172.21.15.96:0/3137901130 shutdown_connections 2024-05-28T06:26:47.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.348+0000 7f524e600640 1 -- 172.21.15.96:0/3137901130 >> 172.21.15.96:0/3137901130 conn(0x7f52480b1e50 msgr2=0x7f52480725a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:26:47.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.348+0000 7f524e600640 1 -- 172.21.15.96:0/3137901130 shutdown_connections 2024-05-28T06:26:47.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.348+0000 7f524e600640 1 -- 172.21.15.96:0/3137901130 wait complete. 2024-05-28T06:26:47.358 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-05-28T06:26:47.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.563+0000 7f7d56e00640 1 Processor -- start 2024-05-28T06:26:47.557 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.563+0000 7f7d56e00640 1 -- start start 2024-05-28T06:26:47.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.565+0000 7f7d56e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f7d500d04a0 0x7f7d500d0930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:47.558 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.565+0000 7f7d56e00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d5006ed10 con 0x7f7d500ad130 2024-05-28T06:26:47.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.565+0000 7f7d56e00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d5015cb00 con 0x7f7d5006f860 2024-05-28T06:26:47.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.565+0000 7f7d56e00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f7d500b3cc0 con 0x7f7d500d04a0 2024-05-28T06:26:47.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.565+0000 7f7d47400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f7d500d04a0 0x7f7d500d0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:47.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.565+0000 7f7d54e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f7d500ad130 0x7f7d500ad540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:44896/0 (socket says 172.21.15.96:44896) 2024-05-28T06:26:47.559 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.565+0000 7f7d54e00640 1 -- 172.21.15.96:0/2703939840 learned_addr learned my addr 172.21.15.96:0/2703939840 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:26:47.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.566+0000 7f7d47400640 1 -- 172.21.15.96:0/2703939840 >> v1:172.21.15.123:6789/0 conn(0x7f7d5006f860 legacy=0x7f7d500b3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:47.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.566+0000 7f7d47400640 1 -- 172.21.15.96:0/2703939840 >> v1:172.21.15.96:6789/0 conn(0x7f7d500ad130 legacy=0x7f7d500ad540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:47.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.566+0000 7f7d47400640 1 -- 172.21.15.96:0/2703939840 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d500b3fa0 con 0x7f7d500d04a0 2024-05-28T06:26:47.560 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.567+0000 7f7d47400640 1 --2- 172.21.15.96:0/2703939840 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7d500d04a0 0x7f7d500d0930 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 crypto rx=0x7f7d380097a0 tx=0x7f7d3802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=890a4627dee117ad server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:47.561 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.567+0000 7f7d4f400640 1 -- 172.21.15.96:0/2703939840 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d38037070 con 0x7f7d500d04a0 2024-05-28T06:26:47.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.567+0000 7f7d4f400640 1 -- 172.21.15.96:0/2703939840 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d38032440 con 0x7f7d500d04a0 2024-05-28T06:26:47.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.567+0000 7f7d4f400640 1 -- 172.21.15.96:0/2703939840 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d38032810 con 0x7f7d500d04a0 2024-05-28T06:26:47.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.568+0000 7f7d56e00640 1 -- 172.21.15.96:0/2703939840 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7d500d04a0 msgr2=0x7f7d500d0930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:47.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.568+0000 7f7d56e00640 1 --2- 172.21.15.96:0/2703939840 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7d500d04a0 0x7f7d500d0930 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 crypto rx=0x7f7d380097a0 tx=0x7f7d3802a1e0 comp rx=0 tx=0).stop 2024-05-28T06:26:47.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.568+0000 7f7d56e00640 1 -- 172.21.15.96:0/2703939840 shutdown_connections 2024-05-28T06:26:47.562 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.568+0000 7f7d56e00640 1 --2- 172.21.15.96:0/2703939840 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7d500d04a0 0x7f7d500d0930 unknown :-1 s=CLOSED pgs=2198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:47.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.568+0000 7f7d56e00640 1 -- 172.21.15.96:0/2703939840 >> 172.21.15.96:0/2703939840 conn(0x7f7d5006bae0 msgr2=0x7f7d5006bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:26:47.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.568+0000 7f7d56e00640 1 -- 172.21.15.96:0/2703939840 shutdown_connections 2024-05-28T06:26:47.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.568+0000 7f7d56e00640 1 -- 172.21.15.96:0/2703939840 wait complete. 2024-05-28T06:26:47.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.570+0000 7f7d56e00640 1 Processor -- start 2024-05-28T06:26:47.563 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.570+0000 7f7d56e00640 1 -- start start 2024-05-28T06:26:47.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.571+0000 7f7d56e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7d5006f860 0x7f7d500c5a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:47.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.571+0000 7f7d56e00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d500ad130 0x7f7d500c3a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:47.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.571+0000 7f7d56e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7d500d04a0 0x7f7d50073050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:47.565 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.571+0000 7f7d56e00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f7d500b4840 con 0x7f7d500d04a0 2024-05-28T06:26:47.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.571+0000 7f7d56e00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f7d500b4560 con 0x7f7d500ad130 2024-05-28T06:26:47.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.571+0000 7f7d56e00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f7d500b3860 con 0x7f7d5006f860 2024-05-28T06:26:47.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.571+0000 7f7d54e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7d500d04a0 0x7f7d50073050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:47.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.571+0000 7f7d54e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7d500d04a0 0x7f7d50073050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35168/0 (socket says 172.21.15.96:35168) 2024-05-28T06:26:47.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.571+0000 7f7d54e00640 1 -- 172.21.15.96:0/3720237602 learned_addr learned my addr 172.21.15.96:0/3720237602 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:26:47.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.571+0000 7f7d47400640 1 --2- 172.21.15.96:0/3720237602 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d500ad130 0x7f7d500c3a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:47.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.571+0000 7f7d4fe00640 1 --2- 172.21.15.96:0/3720237602 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7d5006f860 0x7f7d500c5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:47.566 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.572+0000 7f7d54e00640 1 -- 172.21.15.96:0/3720237602 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7d5006f860 msgr2=0x7f7d500c5a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:47.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.572+0000 7f7d54e00640 1 --2- 172.21.15.96:0/3720237602 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7d5006f860 0x7f7d500c5a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:47.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.572+0000 7f7d54e00640 1 -- 172.21.15.96:0/3720237602 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d500ad130 msgr2=0x7f7d500c3a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:47.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.572+0000 7f7d54e00640 1 --2- 172.21.15.96:0/3720237602 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d500ad130 0x7f7d500c3a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:47.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.572+0000 7f7d54e00640 1 -- 172.21.15.96:0/3720237602 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d500c4510 con 0x7f7d500d04a0 2024-05-28T06:26:47.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.572+0000 7f7d47400640 1 --2- 172.21.15.96:0/3720237602 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d500ad130 0x7f7d500c3a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:26:47.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.572+0000 7f7d4fe00640 1 --2- 172.21.15.96:0/3720237602 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7d5006f860 0x7f7d500c5a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:26:47.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.572+0000 7f7d54e00640 1 --2- 172.21.15.96:0/3720237602 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7d500d04a0 0x7f7d50073050 secure :-1 s=READY pgs=2199 cs=0 l=1 rev1=1 crypto rx=0x7f7d4800c450 tx=0x7f7d4800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:47.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.572+0000 7f7d4cc00640 1 -- 172.21.15.96:0/3720237602 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d48017070 con 0x7f7d500d04a0 2024-05-28T06:26:47.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.572+0000 7f7d4cc00640 1 -- 172.21.15.96:0/3720237602 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d48005ce0 con 0x7f7d500d04a0 2024-05-28T06:26:47.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.572+0000 7f7d4cc00640 1 -- 172.21.15.96:0/3720237602 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d48007270 con 0x7f7d500d04a0 2024-05-28T06:26:47.568 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.572+0000 7f7d56e00640 1 -- 172.21.15.96:0/3720237602 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d500c47f0 con 0x7f7d500d04a0 2024-05-28T06:26:47.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.572+0000 7f7d56e00640 1 -- 172.21.15.96:0/3720237602 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d500c4db0 con 0x7f7d500d04a0 2024-05-28T06:26:47.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.573+0000 7f7d56e00640 1 -- 172.21.15.96:0/3720237602 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d500690f0 con 0x7f7d500d04a0 2024-05-28T06:26:47.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.574+0000 7f7d4cc00640 1 -- 172.21.15.96:0/3720237602 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 49) ==== 94049+0+0 (secure 0 0 0) 0x7f7d4801a070 con 0x7f7d500d04a0 2024-05-28T06:26:47.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.576+0000 7f7d4cc00640 1 --2- 172.21.15.96:0/3720237602 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7d2c06ea50 0x7f7d2c070f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:47.569 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.576+0000 7f7d4cc00640 1 -- 172.21.15.96:0/3720237602 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 5823+0+0 (secure 0 0 0) 0x7f7d48089ec0 con 0x7f7d500d04a0 2024-05-28T06:26:47.570 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.577+0000 7f7d4fe00640 1 --2- 172.21.15.96:0/3720237602 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7d2c06ea50 0x7f7d2c070f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:47.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.578+0000 7f7d4fe00640 1 --2- 172.21.15.96:0/3720237602 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7d2c06ea50 0x7f7d2c070f10 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f7d40004520 tx=0x7f7d40009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:47.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.581+0000 7f7d4cc00640 1 -- 172.21.15.96:0/3720237602 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7d500690f0 con 0x7f7d500d04a0 2024-05-28T06:26:47.774 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.780+0000 7f7d56e00640 1 -- 172.21.15.96:0/3720237602 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7f7d500689e0 con 0x7f7d2c06ea50 2024-05-28T06:26:47.774 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.781+0000 7f7d4cc00640 1 -- 172.21.15.96:0/3720237602 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f7d500689e0 con 0x7f7d2c06ea50 2024-05-28T06:26:47.775 INFO:teuthology.orchestra.run.smithi096.stderr:instructed osd(s) 1 to deep-scrub 2024-05-28T06:26:47.778 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.784+0000 7f7d56e00640 1 -- 172.21.15.96:0/3720237602 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7d2c06ea50 msgr2=0x7f7d2c070f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:47.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.784+0000 7f7d56e00640 1 --2- 172.21.15.96:0/3720237602 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7d2c06ea50 0x7f7d2c070f10 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f7d40004520 tx=0x7f7d40009040 comp rx=0 tx=0).stop 2024-05-28T06:26:47.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.784+0000 7f7d56e00640 1 -- 172.21.15.96:0/3720237602 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7d500d04a0 msgr2=0x7f7d50073050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:47.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.784+0000 7f7d56e00640 1 --2- 172.21.15.96:0/3720237602 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7d500d04a0 0x7f7d50073050 secure :-1 s=READY pgs=2199 cs=0 l=1 rev1=1 crypto rx=0x7f7d4800c450 tx=0x7f7d4800c910 comp rx=0 tx=0).stop 2024-05-28T06:26:47.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.785+0000 7f7d56e00640 1 -- 172.21.15.96:0/3720237602 shutdown_connections 2024-05-28T06:26:47.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.785+0000 7f7d56e00640 1 --2- 172.21.15.96:0/3720237602 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7d2c06ea50 0x7f7d2c070f10 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:47.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.785+0000 7f7d56e00640 1 --2- 172.21.15.96:0/3720237602 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7d500d04a0 0x7f7d50073050 unknown :-1 s=CLOSED pgs=2199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:47.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.785+0000 7f7d56e00640 1 --2- 172.21.15.96:0/3720237602 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d500ad130 0x7f7d500c3a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:47.779 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.785+0000 7f7d56e00640 1 --2- 172.21.15.96:0/3720237602 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7d5006f860 0x7f7d500c5a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:47.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.785+0000 7f7d56e00640 1 -- 172.21.15.96:0/3720237602 >> 172.21.15.96:0/3720237602 conn(0x7f7d5006bae0 msgr2=0x7f7d500c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:26:47.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.785+0000 7f7d56e00640 1 -- 172.21.15.96:0/3720237602 shutdown_connections 2024-05-28T06:26:47.780 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:47.785+0000 7f7d56e00640 1 -- 172.21.15.96:0/3720237602 wait complete. 2024-05-28T06:26:47.799 INFO:tasks.ceph:Scrubbing osd.2 2024-05-28T06:26:47.799 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-05-28T06:26:48.005 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.012+0000 7f0036400640 1 Processor -- start 2024-05-28T06:26:48.006 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.012+0000 7f0036400640 1 -- start start 2024-05-28T06:26:48.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.013+0000 7f0036400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f00300d0860 0x7f003015f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:48.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.013+0000 7f0036400640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f003015bb30 con 0x7f00300ce560 2024-05-28T06:26:48.007 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.014+0000 7f0036400640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00300b3ba0 con 0x7f00300d0430 2024-05-28T06:26:48.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.014+0000 7f0036400640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f00300b5290 con 0x7f00300d0860 2024-05-28T06:26:48.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.014+0000 7f0035000640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f00300ce560 0x7f00300b4b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:44906/0 (socket says 172.21.15.96:44906) 2024-05-28T06:26:48.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.014+0000 7f0035000640 1 -- 172.21.15.96:0/4226538071 learned_addr learned my addr 172.21.15.96:0/4226538071 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:26:48.008 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.014+0000 7f0035a00640 1 --2- 172.21.15.96:0/4226538071 >> v2:172.21.15.96:3300/0 conn(0x7f00300d0860 0x7f003015f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:48.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.014+0000 7f002f400640 1 -- 172.21.15.96:0/4226538071 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3839773232 0 0) 0x7f003015bb30 con 0x7f00300ce560 2024-05-28T06:26:48.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.014+0000 7f002f400640 1 -- 172.21.15.96:0/4226538071 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0010005730 con 0x7f00300ce560 2024-05-28T06:26:48.009 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.014+0000 7f0035a00640 1 -- 172.21.15.96:0/4226538071 >> v1:172.21.15.123:6789/0 conn(0x7f00300d0430 legacy=0x7f003015f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:48.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.014+0000 7f0035a00640 1 -- 172.21.15.96:0/4226538071 >> v1:172.21.15.96:6789/0 conn(0x7f00300ce560 legacy=0x7f00300b4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:48.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.014+0000 7f0035a00640 1 -- 172.21.15.96:0/4226538071 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f003015bdd0 con 0x7f00300d0860 2024-05-28T06:26:48.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.016+0000 7f0035a00640 1 --2- 172.21.15.96:0/4226538071 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00300d0860 0x7f003015f7d0 secure :-1 s=READY pgs=2201 cs=0 l=1 rev1=1 crypto rx=0x7f00280097a0 tx=0x7f002802a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a1d55698f33a09 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:48.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.016+0000 7f002f400640 1 -- 172.21.15.96:0/4226538071 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0028037070 con 0x7f00300d0860 2024-05-28T06:26:48.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.016+0000 7f002f400640 1 -- 172.21.15.96:0/4226538071 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0028032440 con 0x7f00300d0860 2024-05-28T06:26:48.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.016+0000 7f002f400640 1 -- 172.21.15.96:0/4226538071 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0028032810 con 0x7f00300d0860 2024-05-28T06:26:48.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.016+0000 7f0036400640 1 -- 172.21.15.96:0/4226538071 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00300d0860 msgr2=0x7f003015f7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:48.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.016+0000 7f0036400640 1 --2- 172.21.15.96:0/4226538071 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00300d0860 0x7f003015f7d0 secure :-1 s=READY pgs=2201 cs=0 l=1 rev1=1 crypto rx=0x7f00280097a0 tx=0x7f002802a230 comp rx=0 tx=0).stop 2024-05-28T06:26:48.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.017+0000 7f0036400640 1 -- 172.21.15.96:0/4226538071 shutdown_connections 2024-05-28T06:26:48.011 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.017+0000 7f0036400640 1 --2- 172.21.15.96:0/4226538071 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00300d0860 0x7f003015f7d0 unknown :-1 s=CLOSED pgs=2201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:48.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.017+0000 7f0036400640 1 -- 172.21.15.96:0/4226538071 >> 172.21.15.96:0/4226538071 conn(0x7f003006bae0 msgr2=0x7f003006bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:26:48.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.017+0000 7f0036400640 1 -- 172.21.15.96:0/4226538071 shutdown_connections 2024-05-28T06:26:48.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.017+0000 7f0036400640 1 -- 172.21.15.96:0/4226538071 wait complete. 2024-05-28T06:26:48.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.019+0000 7f0036400640 1 Processor -- start 2024-05-28T06:26:48.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.019+0000 7f0036400640 1 -- start start 2024-05-28T06:26:48.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.020+0000 7f0036400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00300ce560 0x7f00301f35d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:48.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.020+0000 7f0036400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f00300d0430 0x7f00300c9540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:48.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.020+0000 7f0036400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f00300d0860 0x7f00300c9a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:48.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.020+0000 7f0036400640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f003015c670 con 0x7f00300ce560 2024-05-28T06:26:48.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.020+0000 7f0036400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f003015c390 con 0x7f00300d0430 2024-05-28T06:26:48.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.020+0000 7f0036400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f00300d0cd0 con 0x7f00300d0860 2024-05-28T06:26:48.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.020+0000 7f0035000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00300ce560 0x7f00301f35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:48.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.020+0000 7f0035000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00300ce560 0x7f00301f35d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35190/0 (socket says 172.21.15.96:35190) 2024-05-28T06:26:48.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.020+0000 7f0035000640 1 -- 172.21.15.96:0/1231473195 learned_addr learned my addr 172.21.15.96:0/1231473195 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:26:48.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.020+0000 7f0035000640 1 -- 172.21.15.96:0/1231473195 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f00300d0860 msgr2=0x7f00300c9a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:48.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.020+0000 7f0035000640 1 --2- 172.21.15.96:0/1231473195 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f00300d0860 0x7f00300c9a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:48.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.020+0000 7f0035000640 1 -- 172.21.15.96:0/1231473195 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f00300d0430 msgr2=0x7f00300c9540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:48.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.020+0000 7f0035000640 1 --2- 172.21.15.96:0/1231473195 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f00300d0430 0x7f00300c9540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:48.015 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.021+0000 7f0035000640 1 -- 172.21.15.96:0/1231473195 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00301f3b10 con 0x7f00300ce560 2024-05-28T06:26:48.016 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.021+0000 7f0035000640 1 --2- 172.21.15.96:0/1231473195 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00300ce560 0x7f00301f35d0 secure :-1 s=READY pgs=2202 cs=0 l=1 rev1=1 crypto rx=0x7f002000cbc0 tx=0x7f002000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:48.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.021+0000 7f002cc00640 1 -- 172.21.15.96:0/1231473195 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0020017070 con 0x7f00300ce560 2024-05-28T06:26:48.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.021+0000 7f002cc00640 1 -- 172.21.15.96:0/1231473195 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f002000ad40 con 0x7f00300ce560 2024-05-28T06:26:48.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.021+0000 7f0036400640 1 -- 172.21.15.96:0/1231473195 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0030073000 con 0x7f00300ce560 2024-05-28T06:26:48.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.021+0000 7f002cc00640 1 -- 172.21.15.96:0/1231473195 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0020012620 con 0x7f00300ce560 2024-05-28T06:26:48.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.021+0000 7f0036400640 1 -- 172.21.15.96:0/1231473195 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00300735c0 con 0x7f00300ce560 2024-05-28T06:26:48.017 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.022+0000 7f0036400640 1 -- 172.21.15.96:0/1231473195 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f00300689e0 con 0x7f00300ce560 2024-05-28T06:26:48.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.024+0000 7f002cc00640 1 -- 172.21.15.96:0/1231473195 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 49) ==== 94049+0+0 (secure 0 0 0) 0x7f002001a070 con 0x7f00300ce560 2024-05-28T06:26:48.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.025+0000 7f002cc00640 1 --2- 172.21.15.96:0/1231473195 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f000c06ea20 0x7f000c070ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:48.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.025+0000 7f002cc00640 1 -- 172.21.15.96:0/1231473195 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 5823+0+0 (secure 0 0 0) 0x7f002008b1e0 con 0x7f00300ce560 2024-05-28T06:26:48.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.025+0000 7f002cc00640 1 --2- 172.21.15.96:0/1231473195 >> [v2:172.21.15.96:6820/3070158265,v1:172.21.15.96:6823/3070158265] conn(0x7f000c076d40 0x7f000c0791c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:48.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.025+0000 7f002cc00640 1 -- 172.21.15.96:0/1231473195 --> [v2:172.21.15.96:6820/3070158265,v1:172.21.15.96:6823/3070158265] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0020016ec0 con 0x7f000c076d40 2024-05-28T06:26:48.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.025+0000 7f002cc00640 1 -- 172.21.15.96:0/1231473195 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_get_version_reply(handle=1 version=132) ==== 24+0+0 (secure 0 0 0) 0x7f002008c730 con 0x7f00300ce560 2024-05-28T06:26:48.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.025+0000 7f0035a00640 1 --2- 172.21.15.96:0/1231473195 >> [v2:172.21.15.96:6820/3070158265,v1:172.21.15.96:6823/3070158265] conn(0x7f000c076d40 0x7f000c0791c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:48.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.026+0000 7f002fe00640 1 --2- 172.21.15.96:0/1231473195 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f000c06ea20 0x7f000c070ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:48.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.026+0000 7f0035a00640 1 --2- 172.21.15.96:0/1231473195 >> [v2:172.21.15.96:6820/3070158265,v1:172.21.15.96:6823/3070158265] conn(0x7f000c076d40 0x7f000c0791c0 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:48.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.027+0000 7f002fe00640 1 --2- 172.21.15.96:0/1231473195 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f000c06ea20 0x7f000c070ee0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f0018000f60 tx=0x7f00180079d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:48.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.027+0000 7f002cc00640 1 -- 172.21.15.96:0/1231473195 <== osd.2 v2:172.21.15.96:6820/3070158265 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0020016ec0 con 0x7f000c076d40 2024-05-28T06:26:48.034 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.041+0000 7f0036400640 1 -- 172.21.15.96:0/1231473195 --> [v2:172.21.15.96:6820/3070158265,v1:172.21.15.96:6823/3070158265] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f0030150e90 con 0x7f000c076d40 2024-05-28T06:26:48.035 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.042+0000 7f002cc00640 1 -- 172.21.15.96:0/1231473195 <== osd.2 v2:172.21.15.96:6820/3070158265 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f0030150e90 con 0x7f000c076d40 2024-05-28T06:26:48.035 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:26:48.036 INFO:teuthology.orchestra.run.smithi096.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-28T06:26:48.036 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:26:48.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.042+0000 7f0036400640 1 -- 172.21.15.96:0/1231473195 >> [v2:172.21.15.96:6820/3070158265,v1:172.21.15.96:6823/3070158265] conn(0x7f000c076d40 msgr2=0x7f000c0791c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:48.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.042+0000 7f0036400640 1 --2- 172.21.15.96:0/1231473195 >> [v2:172.21.15.96:6820/3070158265,v1:172.21.15.96:6823/3070158265] conn(0x7f000c076d40 0x7f000c0791c0 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:48.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.042+0000 7f0036400640 1 -- 172.21.15.96:0/1231473195 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f000c06ea20 msgr2=0x7f000c070ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:48.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.042+0000 7f0036400640 1 --2- 172.21.15.96:0/1231473195 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f000c06ea20 0x7f000c070ee0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f0018000f60 tx=0x7f00180079d0 comp rx=0 tx=0).stop 2024-05-28T06:26:48.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.042+0000 7f0036400640 1 -- 172.21.15.96:0/1231473195 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00300ce560 msgr2=0x7f00301f35d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:48.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.042+0000 7f0036400640 1 --2- 172.21.15.96:0/1231473195 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00300ce560 0x7f00301f35d0 secure :-1 s=READY pgs=2202 cs=0 l=1 rev1=1 crypto rx=0x7f002000cbc0 tx=0x7f002000a430 comp rx=0 tx=0).stop 2024-05-28T06:26:48.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.042+0000 7f0035a00640 1 -- 172.21.15.96:0/1231473195 reap_dead start 2024-05-28T06:26:48.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.042+0000 7f0036400640 1 -- 172.21.15.96:0/1231473195 shutdown_connections 2024-05-28T06:26:48.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.042+0000 7f0036400640 1 -- 172.21.15.96:0/1231473195 >> 172.21.15.96:0/1231473195 conn(0x7f003006bae0 msgr2=0x7f00300ccc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:26:48.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.043+0000 7f0036400640 1 -- 172.21.15.96:0/1231473195 shutdown_connections 2024-05-28T06:26:48.038 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.043+0000 7f0036400640 1 -- 172.21.15.96:0/1231473195 wait complete. 2024-05-28T06:26:48.052 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-05-28T06:26:48.238 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.244+0000 7fe41f200640 1 Processor -- start 2024-05-28T06:26:48.238 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.245+0000 7fe41f200640 1 -- start start 2024-05-28T06:26:48.239 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.246+0000 7fe41f200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fe4200cfa40 0x7fe420169690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:48.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.246+0000 7fe41f200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe42015ec50 con 0x7fe42015e870 2024-05-28T06:26:48.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.246+0000 7fe41f200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4200cff80 con 0x7fe4200d16a0 2024-05-28T06:26:48.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.246+0000 7fe41f200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fe4200d2150 con 0x7fe4200cfa40 2024-05-28T06:26:48.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.246+0000 7fe41de00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fe42015e870 0x7fe4200d0f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:44914/0 (socket says 172.21.15.96:44914) 2024-05-28T06:26:48.240 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.246+0000 7fe41de00640 1 -- 172.21.15.96:0/3688491811 learned_addr learned my addr 172.21.15.96:0/3688491811 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:26:48.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.247+0000 7fe41e800640 1 --2- 172.21.15.96:0/3688491811 >> v2:172.21.15.123:3300/0 conn(0x7fe4200cfa40 0x7fe420169690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:48.242 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.247+0000 7fe41ca00640 1 -- 172.21.15.96:0/3688491811 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 412687061 0 0) 0x7fe42015ec50 con 0x7fe42015e870 2024-05-28T06:26:48.243 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.247+0000 7fe41ca00640 1 -- 172.21.15.96:0/3688491811 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe400003520 con 0x7fe42015e870 2024-05-28T06:26:48.243 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.247+0000 7fe41ca00640 1 -- 172.21.15.96:0/3688491811 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2182514220 0 0) 0x7fe400003520 con 0x7fe42015e870 2024-05-28T06:26:48.243 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.247+0000 7fe41ca00640 1 -- 172.21.15.96:0/3688491811 >> v2:172.21.15.123:3300/0 conn(0x7fe4200cfa40 msgr2=0x7fe420169690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:48.243 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.247+0000 7fe41ca00640 1 --2- 172.21.15.96:0/3688491811 >> v2:172.21.15.123:3300/0 conn(0x7fe4200cfa40 0x7fe420169690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:48.243 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.247+0000 7fe41ca00640 1 -- 172.21.15.96:0/3688491811 >> v1:172.21.15.123:6789/0 conn(0x7fe4200d16a0 legacy=0x7fe4200cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:48.243 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.247+0000 7fe41ca00640 1 -- 172.21.15.96:0/3688491811 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4200d0260 con 0x7fe42015e870 2024-05-28T06:26:48.244 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.247+0000 7fe41ca00640 1 -- 172.21.15.96:0/3688491811 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7fe410001670 con 0x7fe42015e870 2024-05-28T06:26:48.244 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.247+0000 7fe41ca00640 1 -- 172.21.15.96:0/3688491811 >> v1:172.21.15.96:6789/0 conn(0x7fe42015e870 legacy=0x7fe4200d0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:48.244 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.247+0000 7fe41ca00640 1 --2- 172.21.15.96:0/3688491811 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe400003a00 0x7fe400003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:48.244 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.247+0000 7fe41ca00640 1 --2- 172.21.15.96:0/3688491811 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe400004800 0x7fe400004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:48.244 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.247+0000 7fe41ca00640 1 --2- 172.21.15.96:0/3688491811 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe4000051c0 0x7fe40000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:48.244 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.247+0000 7fe41ca00640 1 -- 172.21.15.96:0/3688491811 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe4200d2150 con 0x7fe400004800 2024-05-28T06:26:48.244 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.247+0000 7fe41ca00640 1 -- 172.21.15.96:0/3688491811 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe40000c0c0 con 0x7fe400003a00 2024-05-28T06:26:48.245 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.247+0000 7fe41ca00640 1 -- 172.21.15.96:0/3688491811 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fe40000c2c0 con 0x7fe4000051c0 2024-05-28T06:26:48.245 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.248+0000 7fe41d400640 1 --2- 172.21.15.96:0/3688491811 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe400004800 0x7fe400004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:48.245 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.248+0000 7fe41de00640 1 --2- 172.21.15.96:0/3688491811 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe400003a00 0x7fe400003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:48.245 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.248+0000 7fe41d400640 1 -- 172.21.15.96:0/3688491811 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe4000051c0 msgr2=0x7fe40000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:48.245 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.248+0000 7fe41e800640 1 --2- 172.21.15.96:0/3688491811 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe4000051c0 0x7fe40000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:48.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.248+0000 7fe41d400640 1 --2- 172.21.15.96:0/3688491811 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe4000051c0 0x7fe40000b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:48.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.248+0000 7fe41d400640 1 -- 172.21.15.96:0/3688491811 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe400003a00 msgr2=0x7fe400003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:48.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.248+0000 7fe41d400640 1 --2- 172.21.15.96:0/3688491811 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe400003a00 0x7fe400003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:48.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.248+0000 7fe41d400640 1 -- 172.21.15.96:0/3688491811 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe400003520 con 0x7fe400004800 2024-05-28T06:26:48.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.248+0000 7fe41de00640 1 --2- 172.21.15.96:0/3688491811 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe400003a00 0x7fe400003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:26:48.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.248+0000 7fe41e800640 1 -- 172.21.15.96:0/3688491811 reap_dead start 2024-05-28T06:26:48.246 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.249+0000 7fe41d400640 1 --2- 172.21.15.96:0/3688491811 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe400004800 0x7fe400004c80 secure :-1 s=READY pgs=2204 cs=0 l=1 rev1=1 crypto rx=0x7fe408001e70 tx=0x7fe400005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c36334ffbecf6943 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:48.249 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.249+0000 7fe41ca00640 1 -- 172.21.15.96:0/3688491811 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe408001ac0 con 0x7fe400004800 2024-05-28T06:26:48.249 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.250+0000 7fe41ca00640 1 -- 172.21.15.96:0/3688491811 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe40802ae40 con 0x7fe400004800 2024-05-28T06:26:48.249 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.250+0000 7fe41ca00640 1 -- 172.21.15.96:0/3688491811 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe408003c00 con 0x7fe400004800 2024-05-28T06:26:48.249 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.250+0000 7fe41f200640 1 -- 172.21.15.96:0/3688491811 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe400004800 msgr2=0x7fe400004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:48.249 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.250+0000 7fe41f200640 1 --2- 172.21.15.96:0/3688491811 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe400004800 0x7fe400004c80 secure :-1 s=READY pgs=2204 cs=0 l=1 rev1=1 crypto rx=0x7fe408001e70 tx=0x7fe400005830 comp rx=0 tx=0).stop 2024-05-28T06:26:48.250 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.250+0000 7fe41f200640 1 -- 172.21.15.96:0/3688491811 shutdown_connections 2024-05-28T06:26:48.250 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.250+0000 7fe41f200640 1 --2- 172.21.15.96:0/3688491811 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe400004800 0x7fe400004c80 unknown :-1 s=CLOSED pgs=2204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:48.250 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.250+0000 7fe41f200640 1 -- 172.21.15.96:0/3688491811 >> 172.21.15.96:0/3688491811 conn(0x7fe42008bfc0 msgr2=0x7fe4200d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:26:48.250 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.250+0000 7fe41f200640 1 -- 172.21.15.96:0/3688491811 shutdown_connections 2024-05-28T06:26:48.250 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.250+0000 7fe41f200640 1 -- 172.21.15.96:0/3688491811 wait complete. 2024-05-28T06:26:48.250 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.252+0000 7fe41f200640 1 Processor -- start 2024-05-28T06:26:48.250 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.252+0000 7fe41f200640 1 -- start start 2024-05-28T06:26:48.250 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.253+0000 7fe41f200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe400004800 0x7fe42014fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:48.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.253+0000 7fe41f200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe42015a440 0x7fe42015a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:48.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.253+0000 7fe41f200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe42015c6c0 0x7fe42015cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:48.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.253+0000 7fe41f200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe4200d4fb0 con 0x7fe400004800 2024-05-28T06:26:48.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.253+0000 7fe41f200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe4200d0820 con 0x7fe42015c6c0 2024-05-28T06:26:48.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.253+0000 7fe41f200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fe4200cf330 con 0x7fe42015a440 2024-05-28T06:26:48.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.253+0000 7fe41de00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe400004800 0x7fe42014fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:48.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.253+0000 7fe41de00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe400004800 0x7fe42014fa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35200/0 (socket says 172.21.15.96:35200) 2024-05-28T06:26:48.251 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.253+0000 7fe41de00640 1 -- 172.21.15.96:0/2522673635 learned_addr learned my addr 172.21.15.96:0/2522673635 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:26:48.252 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.254+0000 7fe41de00640 1 -- 172.21.15.96:0/2522673635 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe42015a440 msgr2=0x7fe42015a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:48.252 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.254+0000 7fe41de00640 1 --2- 172.21.15.96:0/2522673635 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe42015a440 0x7fe42015a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:48.252 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.254+0000 7fe41de00640 1 -- 172.21.15.96:0/2522673635 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe42015c6c0 msgr2=0x7fe42015cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:48.252 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.254+0000 7fe41e800640 1 --2- 172.21.15.96:0/2522673635 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe42015c6c0 0x7fe42015cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:48.252 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.254+0000 7fe41de00640 1 --2- 172.21.15.96:0/2522673635 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe42015c6c0 0x7fe42015cb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:48.252 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.254+0000 7fe41de00640 1 -- 172.21.15.96:0/2522673635 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe42015d410 con 0x7fe400004800 2024-05-28T06:26:48.253 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.254+0000 7fe41de00640 1 --2- 172.21.15.96:0/2522673635 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe400004800 0x7fe42014fa60 secure :-1 s=READY pgs=2205 cs=0 l=1 rev1=1 crypto rx=0x7fe41000d770 tx=0x7fe41000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:48.253 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.254+0000 7fe416000640 1 -- 172.21.15.96:0/2522673635 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe410019070 con 0x7fe400004800 2024-05-28T06:26:48.253 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.255+0000 7fe416000640 1 -- 172.21.15.96:0/2522673635 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4100072d0 con 0x7fe400004800 2024-05-28T06:26:48.253 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.255+0000 7fe41f200640 1 -- 172.21.15.96:0/2522673635 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe42015d6f0 con 0x7fe400004800 2024-05-28T06:26:48.253 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.255+0000 7fe416000640 1 -- 172.21.15.96:0/2522673635 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4100047e0 con 0x7fe400004800 2024-05-28T06:26:48.253 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.255+0000 7fe41f200640 1 -- 172.21.15.96:0/2522673635 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe42015dcb0 con 0x7fe400004800 2024-05-28T06:26:48.254 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.256+0000 7fe41f200640 1 -- 172.21.15.96:0/2522673635 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe42015deb0 con 0x7fe400004800 2024-05-28T06:26:48.254 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.257+0000 7fe416000640 1 -- 172.21.15.96:0/2522673635 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 49) ==== 94049+0+0 (secure 0 0 0) 0x7fe41001c070 con 0x7fe400004800 2024-05-28T06:26:48.254 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.258+0000 7fe416000640 1 --2- 172.21.15.96:0/2522673635 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe3fc06ea20 0x7fe3fc070ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:48.255 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.258+0000 7fe416000640 1 -- 172.21.15.96:0/2522673635 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 5823+0+0 (secure 0 0 0) 0x7fe41008a7c0 con 0x7fe400004800 2024-05-28T06:26:48.255 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.258+0000 7fe41d400640 1 --2- 172.21.15.96:0/2522673635 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe3fc06ea20 0x7fe3fc070ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:48.255 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.260+0000 7fe41d400640 1 --2- 172.21.15.96:0/2522673635 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe3fc06ea20 0x7fe3fc070ee0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7fe408032000 tx=0x7fe40802c000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:48.255 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.262+0000 7fe416000640 1 -- 172.21.15.96:0/2522673635 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe41005a6f0 con 0x7fe400004800 2024-05-28T06:26:48.454 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.461+0000 7fe41f200640 1 -- 172.21.15.96:0/2522673635 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7fe4200689e0 con 0x7fe3fc06ea20 2024-05-28T06:26:48.456 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.462+0000 7fe416000640 1 -- 172.21.15.96:0/2522673635 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fe4200689e0 con 0x7fe3fc06ea20 2024-05-28T06:26:48.456 INFO:teuthology.orchestra.run.smithi096.stderr:instructed osd(s) 2 to deep-scrub 2024-05-28T06:26:48.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.465+0000 7fe41f200640 1 -- 172.21.15.96:0/2522673635 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe3fc06ea20 msgr2=0x7fe3fc070ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:48.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.465+0000 7fe41f200640 1 --2- 172.21.15.96:0/2522673635 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe3fc06ea20 0x7fe3fc070ee0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7fe408032000 tx=0x7fe40802c000 comp rx=0 tx=0).stop 2024-05-28T06:26:48.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.465+0000 7fe41f200640 1 -- 172.21.15.96:0/2522673635 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe400004800 msgr2=0x7fe42014fa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:48.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.465+0000 7fe41f200640 1 --2- 172.21.15.96:0/2522673635 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe400004800 0x7fe42014fa60 secure :-1 s=READY pgs=2205 cs=0 l=1 rev1=1 crypto rx=0x7fe41000d770 tx=0x7fe41000dc30 comp rx=0 tx=0).stop 2024-05-28T06:26:48.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.465+0000 7fe41f200640 1 -- 172.21.15.96:0/2522673635 shutdown_connections 2024-05-28T06:26:48.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.465+0000 7fe41f200640 1 --2- 172.21.15.96:0/2522673635 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fe3fc06ea20 0x7fe3fc070ee0 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:48.459 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.465+0000 7fe41f200640 1 --2- 172.21.15.96:0/2522673635 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe42015c6c0 0x7fe42015cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:48.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.465+0000 7fe41f200640 1 --2- 172.21.15.96:0/2522673635 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fe42015a440 0x7fe42015a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:48.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.465+0000 7fe41f200640 1 --2- 172.21.15.96:0/2522673635 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe400004800 0x7fe42014fa60 unknown :-1 s=CLOSED pgs=2205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:48.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.465+0000 7fe41f200640 1 -- 172.21.15.96:0/2522673635 >> 172.21.15.96:0/2522673635 conn(0x7fe42008bfc0 msgr2=0x7fe4200b10f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:26:48.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.465+0000 7fe41f200640 1 -- 172.21.15.96:0/2522673635 shutdown_connections 2024-05-28T06:26:48.460 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.465+0000 7fe41f200640 1 -- 172.21.15.96:0/2522673635 wait complete. 2024-05-28T06:26:48.477 INFO:tasks.ceph:Scrubbing osd.3 2024-05-28T06:26:48.478 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-05-28T06:26:48.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.680+0000 7fce7cc00640 1 Processor -- start 2024-05-28T06:26:48.674 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.681+0000 7fce7cc00640 1 -- start start 2024-05-28T06:26:48.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.682+0000 7fce7cc00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fce780b4fa0 0x7fce780cc710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:48.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.682+0000 7fce7cc00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fce7806f220 0x7fce7815e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:48.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.682+0000 7fce7cc00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce780ca840 con 0x7fce7815ec90 2024-05-28T06:26:48.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.682+0000 7fce7cc00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fce7815b040 con 0x7fce7806f220 2024-05-28T06:26:48.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.682+0000 7fce7cc00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fce780cd080 con 0x7fce780b4fa0 2024-05-28T06:26:48.676 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.683+0000 7fce77e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fce7815ec90 0x7fce7806daf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:44918/0 (socket says 172.21.15.96:44918) 2024-05-28T06:26:48.677 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.683+0000 7fce77e00640 1 -- 172.21.15.96:0/760984474 learned_addr learned my addr 172.21.15.96:0/760984474 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:26:48.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.683+0000 7fce77400640 1 --2- 172.21.15.96:0/760984474 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fce780b4fa0 0x7fce780cc710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:48.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.683+0000 7fce6ea00640 1 --2- 172.21.15.96:0/760984474 >> v2:172.21.15.123:3300/0 conn(0x7fce7806f220 0x7fce7815e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:48.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.683+0000 7fce76a00640 1 -- 172.21.15.96:0/760984474 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3599552837 0 0) 0x7fce780ca840 con 0x7fce7815ec90 2024-05-28T06:26:48.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.683+0000 7fce76a00640 1 -- 172.21.15.96:0/760984474 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fce5c005730 con 0x7fce7815ec90 2024-05-28T06:26:48.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.684+0000 7fce76a00640 1 -- 172.21.15.96:0/760984474 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3894569459 0 0) 0x7fce5c005730 con 0x7fce7815ec90 2024-05-28T06:26:48.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.684+0000 7fce76a00640 1 -- 172.21.15.96:0/760984474 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fce780b4fa0 msgr2=0x7fce780cc710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:48.679 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.684+0000 7fce76a00640 1 --2- 172.21.15.96:0/760984474 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fce780b4fa0 0x7fce780cc710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:48.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.684+0000 7fce76a00640 1 -- 172.21.15.96:0/760984474 >> v2:172.21.15.123:3300/0 conn(0x7fce7806f220 msgr2=0x7fce7815e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:48.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.684+0000 7fce76a00640 1 --2- 172.21.15.96:0/760984474 >> v2:172.21.15.123:3300/0 conn(0x7fce7806f220 0x7fce7815e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:48.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.684+0000 7fce76a00640 1 -- 172.21.15.96:0/760984474 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce780cd360 con 0x7fce7815ec90 2024-05-28T06:26:48.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.684+0000 7fce76a00640 1 -- 172.21.15.96:0/760984474 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7fce700019f0 con 0x7fce7815ec90 2024-05-28T06:26:48.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.684+0000 7fce76a00640 1 -- 172.21.15.96:0/760984474 >> v1:172.21.15.96:6789/0 conn(0x7fce7815ec90 legacy=0x7fce7806daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:48.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.684+0000 7fce76a00640 1 --2- 172.21.15.96:0/760984474 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce5c005bc0 0x7fce5c005fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:48.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.684+0000 7fce76a00640 1 --2- 172.21.15.96:0/760984474 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fce5c006a10 0x7fce5c006e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:48.680 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.684+0000 7fce76a00640 1 --2- 172.21.15.96:0/760984474 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fce780b4fa0 0x7fce5c0073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:48.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.684+0000 7fce76a00640 1 -- 172.21.15.96:0/760984474 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fce7815b040 con 0x7fce5c006a10 2024-05-28T06:26:48.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.684+0000 7fce76a00640 1 -- 172.21.15.96:0/760984474 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fce780cd080 con 0x7fce5c005bc0 2024-05-28T06:26:48.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.684+0000 7fce76a00640 1 -- 172.21.15.96:0/760984474 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fce5c00dc20 con 0x7fce780b4fa0 2024-05-28T06:26:48.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.684+0000 7fce6ea00640 1 --2- 172.21.15.96:0/760984474 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fce5c006a10 0x7fce5c006e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:48.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.685+0000 7fce6ea00640 1 -- 172.21.15.96:0/760984474 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fce780b4fa0 msgr2=0x7fce5c0073d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:48.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.685+0000 7fce6ea00640 1 --2- 172.21.15.96:0/760984474 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fce780b4fa0 0x7fce5c0073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:48.681 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.685+0000 7fce6ea00640 1 -- 172.21.15.96:0/760984474 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce5c005bc0 msgr2=0x7fce5c005fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:48.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.685+0000 7fce6ea00640 1 --2- 172.21.15.96:0/760984474 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce5c005bc0 0x7fce5c005fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:48.682 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.685+0000 7fce6ea00640 1 -- 172.21.15.96:0/760984474 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce5c005730 con 0x7fce5c006a10 2024-05-28T06:26:48.684 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.686+0000 7fce6ea00640 1 --2- 172.21.15.96:0/760984474 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fce5c006a10 0x7fce5c006e90 secure :-1 s=READY pgs=2207 cs=0 l=1 rev1=1 crypto rx=0x7fce6000a9a0 tx=0x7fce5c007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb644d5bde98342f server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:48.684 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.686+0000 7fce76a00640 1 -- 172.21.15.96:0/760984474 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce60035020 con 0x7fce5c006a10 2024-05-28T06:26:48.684 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.686+0000 7fce76a00640 1 -- 172.21.15.96:0/760984474 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce60031680 con 0x7fce5c006a10 2024-05-28T06:26:48.684 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.686+0000 7fce76a00640 1 -- 172.21.15.96:0/760984474 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce6000b040 con 0x7fce5c006a10 2024-05-28T06:26:48.684 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.686+0000 7fce7cc00640 1 -- 172.21.15.96:0/760984474 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fce5c006a10 msgr2=0x7fce5c006e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:48.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.686+0000 7fce7cc00640 1 --2- 172.21.15.96:0/760984474 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fce5c006a10 0x7fce5c006e90 secure :-1 s=READY pgs=2207 cs=0 l=1 rev1=1 crypto rx=0x7fce6000a9a0 tx=0x7fce5c007a40 comp rx=0 tx=0).stop 2024-05-28T06:26:48.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.687+0000 7fce77e00640 1 -- 172.21.15.96:0/760984474 reap_dead start 2024-05-28T06:26:48.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.687+0000 7fce7cc00640 1 -- 172.21.15.96:0/760984474 shutdown_connections 2024-05-28T06:26:48.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.687+0000 7fce7cc00640 1 -- 172.21.15.96:0/760984474 >> 172.21.15.96:0/760984474 conn(0x7fce7806bae0 msgr2=0x7fce7806bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:26:48.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.687+0000 7fce7cc00640 1 -- 172.21.15.96:0/760984474 shutdown_connections 2024-05-28T06:26:48.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.687+0000 7fce7cc00640 1 -- 172.21.15.96:0/760984474 wait complete. 2024-05-28T06:26:48.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.689+0000 7fce7cc00640 1 Processor -- start 2024-05-28T06:26:48.685 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.689+0000 7fce7cc00640 1 -- start start 2024-05-28T06:26:48.686 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.690+0000 7fce7cc00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fce5c006a10 0x7fce7809ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:48.686 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.690+0000 7fce7cc00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce781556d0 0x7fce781f8b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:48.686 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.690+0000 7fce7cc00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fce7809c220 0x7fce780b1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:48.686 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.690+0000 7fce7cc00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fce780b31c0 con 0x7fce7809c220 2024-05-28T06:26:48.686 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.690+0000 7fce7cc00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fce780cc010 con 0x7fce781556d0 2024-05-28T06:26:48.686 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.690+0000 7fce7cc00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fce780d0430 con 0x7fce5c006a10 2024-05-28T06:26:48.686 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.690+0000 7fce77e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fce7809c220 0x7fce780b1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:48.686 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.690+0000 7fce77e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fce7809c220 0x7fce780b1ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:35218/0 (socket says 172.21.15.96:35218) 2024-05-28T06:26:48.687 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.690+0000 7fce77e00640 1 -- 172.21.15.96:0/1946871240 learned_addr learned my addr 172.21.15.96:0/1946871240 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:26:48.687 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.690+0000 7fce77400640 1 --2- 172.21.15.96:0/1946871240 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fce5c006a10 0x7fce7809ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:48.687 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.690+0000 7fce77e00640 1 -- 172.21.15.96:0/1946871240 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fce5c006a10 msgr2=0x7fce7809ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:48.687 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.690+0000 7fce77e00640 1 --2- 172.21.15.96:0/1946871240 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fce5c006a10 0x7fce7809ba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:48.687 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.690+0000 7fce77e00640 1 -- 172.21.15.96:0/1946871240 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce781556d0 msgr2=0x7fce781f8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:48.687 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.690+0000 7fce77e00640 1 --2- 172.21.15.96:0/1946871240 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce781556d0 0x7fce781f8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:48.687 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.690+0000 7fce77e00640 1 -- 172.21.15.96:0/1946871240 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce781f80e0 con 0x7fce7809c220 2024-05-28T06:26:48.688 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.691+0000 7fce77e00640 1 --2- 172.21.15.96:0/1946871240 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fce7809c220 0x7fce780b1ea0 secure :-1 s=READY pgs=2208 cs=0 l=1 rev1=1 crypto rx=0x7fce7815e690 tx=0x7fce7000ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:48.689 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.691+0000 7fce6fe00640 1 -- 172.21.15.96:0/1946871240 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce7000a610 con 0x7fce7809c220 2024-05-28T06:26:48.689 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.691+0000 7fce6fe00640 1 -- 172.21.15.96:0/1946871240 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce7000a7b0 con 0x7fce7809c220 2024-05-28T06:26:48.689 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.691+0000 7fce7cc00640 1 -- 172.21.15.96:0/1946871240 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce7809ca00 con 0x7fce7809c220 2024-05-28T06:26:48.690 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.691+0000 7fce6fe00640 1 -- 172.21.15.96:0/1946871240 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce7000aba0 con 0x7fce7809c220 2024-05-28T06:26:48.690 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.691+0000 7fce7cc00640 1 -- 172.21.15.96:0/1946871240 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce7809c6a0 con 0x7fce7809c220 2024-05-28T06:26:48.690 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.692+0000 7fce7cc00640 1 -- 172.21.15.96:0/1946871240 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fce781f8760 con 0x7fce7809c220 2024-05-28T06:26:48.690 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.693+0000 7fce6fe00640 1 -- 172.21.15.96:0/1946871240 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 49) ==== 94049+0+0 (secure 0 0 0) 0x7fce70011070 con 0x7fce7809c220 2024-05-28T06:26:48.690 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.694+0000 7fce6fe00640 1 --2- 172.21.15.96:0/1946871240 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fce5406ea50 0x7fce54070f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:48.690 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.694+0000 7fce6fe00640 1 -- 172.21.15.96:0/1946871240 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 5823+0+0 (secure 0 0 0) 0x7fce7008a6d0 con 0x7fce7809c220 2024-05-28T06:26:48.690 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.695+0000 7fce6fe00640 1 --2- 172.21.15.96:0/1946871240 >> [v2:172.21.15.96:6808/2384313931,v1:172.21.15.96:6810/2384313931] conn(0x7fce54076d70 0x7fce540791f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:48.691 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.695+0000 7fce6fe00640 1 -- 172.21.15.96:0/1946871240 --> [v2:172.21.15.96:6808/2384313931,v1:172.21.15.96:6810/2384313931] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fce540798c0 con 0x7fce54076d70 2024-05-28T06:26:48.691 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.695+0000 7fce6fe00640 1 -- 172.21.15.96:0/1946871240 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_get_version_reply(handle=1 version=132) ==== 24+0+0 (secure 0 0 0) 0x7fce70096050 con 0x7fce7809c220 2024-05-28T06:26:48.691 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.695+0000 7fce77400640 1 --2- 172.21.15.96:0/1946871240 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fce5406ea50 0x7fce54070f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:48.691 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.695+0000 7fce6ea00640 1 --2- 172.21.15.96:0/1946871240 >> [v2:172.21.15.96:6808/2384313931,v1:172.21.15.96:6810/2384313931] conn(0x7fce54076d70 0x7fce540791f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:48.691 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.695+0000 7fce6ea00640 1 --2- 172.21.15.96:0/1946871240 >> [v2:172.21.15.96:6808/2384313931,v1:172.21.15.96:6810/2384313931] conn(0x7fce54076d70 0x7fce540791f0 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:48.691 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.696+0000 7fce77400640 1 --2- 172.21.15.96:0/1946871240 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fce5406ea50 0x7fce54070f10 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fce68004770 tx=0x7fce68008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:48.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.696+0000 7fce6fe00640 1 -- 172.21.15.96:0/1946871240 <== osd.3 v2:172.21.15.96:6808/2384313931 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fce540798c0 con 0x7fce54076d70 2024-05-28T06:26:48.704 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.711+0000 7fce7cc00640 1 -- 172.21.15.96:0/1946871240 --> [v2:172.21.15.96:6808/2384313931,v1:172.21.15.96:6810/2384313931] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fce780b3d30 con 0x7fce54076d70 2024-05-28T06:26:48.705 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.711+0000 7fce6fe00640 1 -- 172.21.15.96:0/1946871240 <== osd.3 v2:172.21.15.96:6808/2384313931 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fce780b3d30 con 0x7fce54076d70 2024-05-28T06:26:48.705 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:26:48.705 INFO:teuthology.orchestra.run.smithi096.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-28T06:26:48.705 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:26:48.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.712+0000 7fce7cc00640 1 -- 172.21.15.96:0/1946871240 >> [v2:172.21.15.96:6808/2384313931,v1:172.21.15.96:6810/2384313931] conn(0x7fce54076d70 msgr2=0x7fce540791f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:48.706 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.712+0000 7fce7cc00640 1 --2- 172.21.15.96:0/1946871240 >> [v2:172.21.15.96:6808/2384313931,v1:172.21.15.96:6810/2384313931] conn(0x7fce54076d70 0x7fce540791f0 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:48.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.712+0000 7fce7cc00640 1 -- 172.21.15.96:0/1946871240 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fce5406ea50 msgr2=0x7fce54070f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:48.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.712+0000 7fce7cc00640 1 --2- 172.21.15.96:0/1946871240 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fce5406ea50 0x7fce54070f10 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fce68004770 tx=0x7fce68008040 comp rx=0 tx=0).stop 2024-05-28T06:26:48.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.712+0000 7fce7cc00640 1 -- 172.21.15.96:0/1946871240 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fce7809c220 msgr2=0x7fce780b1ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:48.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.712+0000 7fce7cc00640 1 --2- 172.21.15.96:0/1946871240 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fce7809c220 0x7fce780b1ea0 secure :-1 s=READY pgs=2208 cs=0 l=1 rev1=1 crypto rx=0x7fce7815e690 tx=0x7fce7000ccd0 comp rx=0 tx=0).stop 2024-05-28T06:26:48.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.712+0000 7fce77e00640 1 -- 172.21.15.96:0/1946871240 reap_dead start 2024-05-28T06:26:48.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.712+0000 7fce7cc00640 1 -- 172.21.15.96:0/1946871240 shutdown_connections 2024-05-28T06:26:48.707 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.712+0000 7fce7cc00640 1 -- 172.21.15.96:0/1946871240 >> 172.21.15.96:0/1946871240 conn(0x7fce7806bae0 msgr2=0x7fce78156930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:26:48.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.712+0000 7fce7cc00640 1 -- 172.21.15.96:0/1946871240 shutdown_connections 2024-05-28T06:26:48.708 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.712+0000 7fce7cc00640 1 -- 172.21.15.96:0/1946871240 wait complete. 2024-05-28T06:26:48.724 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-05-28T06:26:48.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.935+0000 7f31afe00640 1 Processor -- start 2024-05-28T06:26:48.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.935+0000 7f31afe00640 1 -- start start 2024-05-28T06:26:48.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.937+0000 7f31afe00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f31b0156b30 0x7f31b00af6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:48.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.937+0000 7f31afe00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f31b00ad940 0x7f31b00b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:48.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.937+0000 7f31afe00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31b0156f10 con 0x7f31b006ee40 2024-05-28T06:26:48.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.937+0000 7f31afe00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f31b006dda0 con 0x7f31b0156b30 2024-05-28T06:26:48.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.937+0000 7f31afe00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f31b006e080 con 0x7f31b00ad940 2024-05-28T06:26:48.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.937+0000 7f31a7e00640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f31b006ee40 0x7f31b00ad230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:43374/0 (socket says 172.21.15.96:43374) 2024-05-28T06:26:48.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.937+0000 7f31a7e00640 1 -- 172.21.15.96:0/1828019855 learned_addr learned my addr 172.21.15.96:0/1828019855 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:26:48.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.937+0000 7f31aea00640 1 --2- 172.21.15.96:0/1828019855 >> v2:172.21.15.96:3300/0 conn(0x7f31b0156b30 0x7f31b00af6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:48.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.937+0000 7f31af400640 1 --2- 172.21.15.96:0/1828019855 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f31b00ad940 0x7f31b00b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:48.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.938+0000 7f31aea00640 1 -- 172.21.15.96:0/1828019855 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f31b00ad940 msgr2=0x7f31b00b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:48.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.938+0000 7f31aea00640 1 --2- 172.21.15.96:0/1828019855 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f31b00ad940 0x7f31b00b32c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:48.933 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.938+0000 7f31aea00640 1 -- 172.21.15.96:0/1828019855 >> v1:172.21.15.96:6789/0 conn(0x7f31b006ee40 legacy=0x7f31b00ad230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:48.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.938+0000 7f31aea00640 1 -- 172.21.15.96:0/1828019855 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31b006e360 con 0x7f31b0156b30 2024-05-28T06:26:48.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.939+0000 7f31aea00640 1 --2- 172.21.15.96:0/1828019855 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f31b0156b30 0x7f31b00af6e0 secure :-1 s=READY pgs=2210 cs=0 l=1 rev1=1 crypto rx=0x7f31a00097a0 tx=0x7f31a002a0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed6d22ab7a4d8dc0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:48.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.939+0000 7f31ae000640 1 -- 172.21.15.96:0/1828019855 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31a0037070 con 0x7f31b0156b30 2024-05-28T06:26:48.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.939+0000 7f31ae000640 1 -- 172.21.15.96:0/1828019855 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31a002ae40 con 0x7f31b0156b30 2024-05-28T06:26:48.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.939+0000 7f31ae000640 1 -- 172.21.15.96:0/1828019855 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31a0032660 con 0x7f31b0156b30 2024-05-28T06:26:48.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.940+0000 7f31afe00640 1 -- 172.21.15.96:0/1828019855 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f31b0156b30 msgr2=0x7f31b00af6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:48.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.940+0000 7f31afe00640 1 --2- 172.21.15.96:0/1828019855 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f31b0156b30 0x7f31b00af6e0 secure :-1 s=READY pgs=2210 cs=0 l=1 rev1=1 crypto rx=0x7f31a00097a0 tx=0x7f31a002a0f0 comp rx=0 tx=0).stop 2024-05-28T06:26:48.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.940+0000 7f31afe00640 1 -- 172.21.15.96:0/1828019855 shutdown_connections 2024-05-28T06:26:48.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.940+0000 7f31afe00640 1 --2- 172.21.15.96:0/1828019855 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f31b00ad940 0x7f31b00b32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:48.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.940+0000 7f31afe00640 1 --2- 172.21.15.96:0/1828019855 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f31b0156b30 0x7f31b00af6e0 unknown :-1 s=CLOSED pgs=2210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:48.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.940+0000 7f31afe00640 1 -- 172.21.15.96:0/1828019855 >> 172.21.15.96:0/1828019855 conn(0x7f31b00b1fc0 msgr2=0x7f31b00b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:26:48.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.940+0000 7f31afe00640 1 -- 172.21.15.96:0/1828019855 shutdown_connections 2024-05-28T06:26:48.935 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.940+0000 7f31afe00640 1 -- 172.21.15.96:0/1828019855 wait complete. 2024-05-28T06:26:48.938 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.942+0000 7f31afe00640 1 Processor -- start 2024-05-28T06:26:48.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.942+0000 7f31afe00640 1 -- start start 2024-05-28T06:26:48.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.943+0000 7f31afe00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f31b006ee40 0x7f31b0142430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:48.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.943+0000 7f31afe00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f31b00ad940 0x7f31b0072c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:48.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.943+0000 7f31afe00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f31b0156b30 0x7f31b0073180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:48.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.943+0000 7f31afe00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f31b00afc20 con 0x7f31b00ad940 2024-05-28T06:26:48.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.943+0000 7f31afe00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f31b00b2c30 con 0x7f31b006ee40 2024-05-28T06:26:48.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.943+0000 7f31afe00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f31b0157e90 con 0x7f31b0156b30 2024-05-28T06:26:48.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.943+0000 7f31a7e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f31b00ad940 0x7f31b0072c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:48.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.943+0000 7f31a7e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f31b00ad940 0x7f31b0072c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:45866/0 (socket says 172.21.15.96:45866) 2024-05-28T06:26:48.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.943+0000 7f31a7e00640 1 -- 172.21.15.96:0/2061884306 learned_addr learned my addr 172.21.15.96:0/2061884306 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:26:48.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.943+0000 7f31af400640 1 --2- 172.21.15.96:0/2061884306 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f31b0156b30 0x7f31b0073180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:48.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.943+0000 7f31aea00640 1 --2- 172.21.15.96:0/2061884306 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f31b006ee40 0x7f31b0142430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:48.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.943+0000 7f31a7e00640 1 -- 172.21.15.96:0/2061884306 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f31b0156b30 msgr2=0x7f31b0073180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:48.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.943+0000 7f31a7e00640 1 --2- 172.21.15.96:0/2061884306 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f31b0156b30 0x7f31b0073180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:48.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.943+0000 7f31a7e00640 1 -- 172.21.15.96:0/2061884306 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f31b006ee40 msgr2=0x7f31b0142430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:48.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.943+0000 7f31a7e00640 1 --2- 172.21.15.96:0/2061884306 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f31b006ee40 0x7f31b0142430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:48.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.943+0000 7f31a7e00640 1 -- 172.21.15.96:0/2061884306 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31b0142970 con 0x7f31b00ad940 2024-05-28T06:26:48.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.944+0000 7f31a7e00640 1 --2- 172.21.15.96:0/2061884306 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f31b00ad940 0x7f31b0072c40 secure :-1 s=READY pgs=2211 cs=0 l=1 rev1=1 crypto rx=0x7f319800c3c0 tx=0x7f319800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:48.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.944+0000 7f31a6a00640 1 -- 172.21.15.96:0/2061884306 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3198017070 con 0x7f31b00ad940 2024-05-28T06:26:48.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.944+0000 7f31a6a00640 1 -- 172.21.15.96:0/2061884306 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3198005ce0 con 0x7f31b00ad940 2024-05-28T06:26:48.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.944+0000 7f31afe00640 1 -- 172.21.15.96:0/2061884306 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31b01f47b0 con 0x7f31b00ad940 2024-05-28T06:26:48.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.944+0000 7f31a6a00640 1 -- 172.21.15.96:0/2061884306 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31980072f0 con 0x7f31b00ad940 2024-05-28T06:26:48.943 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.944+0000 7f31afe00640 1 -- 172.21.15.96:0/2061884306 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31b0157560 con 0x7f31b00ad940 2024-05-28T06:26:48.943 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.945+0000 7f31afe00640 1 -- 172.21.15.96:0/2061884306 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31b00689e0 con 0x7f31b00ad940 2024-05-28T06:26:48.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.946+0000 7f31a6a00640 1 -- 172.21.15.96:0/2061884306 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 49) ==== 94049+0+0 (secure 0 0 0) 0x7f319801a070 con 0x7f31b00ad940 2024-05-28T06:26:48.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.947+0000 7f31a6a00640 1 --2- 172.21.15.96:0/2061884306 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f318c06ea20 0x7f318c070ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:48.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.948+0000 7f31a6a00640 1 -- 172.21.15.96:0/2061884306 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 5823+0+0 (secure 0 0 0) 0x7f319808a050 con 0x7f31b00ad940 2024-05-28T06:26:48.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.948+0000 7f31aea00640 1 --2- 172.21.15.96:0/2061884306 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f318c06ea20 0x7f318c070ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:48.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.949+0000 7f31aea00640 1 --2- 172.21.15.96:0/2061884306 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f318c06ea20 0x7f318c070ee0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f31a0034000 tx=0x7f31a002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:48.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:48.951+0000 7f31a6a00640 1 -- 172.21.15.96:0/2061884306 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3198059ee0 con 0x7f31b00ad940 2024-05-28T06:26:49.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.151+0000 7f31afe00640 1 -- 172.21.15.96:0/2061884306 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7f31b006f7f0 con 0x7f318c06ea20 2024-05-28T06:26:49.145 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.152+0000 7f31a6a00640 1 -- 172.21.15.96:0/2061884306 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f31b006f7f0 con 0x7f318c06ea20 2024-05-28T06:26:49.145 INFO:teuthology.orchestra.run.smithi096.stderr:instructed osd(s) 3 to deep-scrub 2024-05-28T06:26:49.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.155+0000 7f31afe00640 1 -- 172.21.15.96:0/2061884306 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f318c06ea20 msgr2=0x7f318c070ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:49.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.155+0000 7f31afe00640 1 --2- 172.21.15.96:0/2061884306 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f318c06ea20 0x7f318c070ee0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f31a0034000 tx=0x7f31a002e000 comp rx=0 tx=0).stop 2024-05-28T06:26:49.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.155+0000 7f31afe00640 1 -- 172.21.15.96:0/2061884306 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f31b00ad940 msgr2=0x7f31b0072c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:49.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.155+0000 7f31afe00640 1 --2- 172.21.15.96:0/2061884306 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f31b00ad940 0x7f31b0072c40 secure :-1 s=READY pgs=2211 cs=0 l=1 rev1=1 crypto rx=0x7f319800c3c0 tx=0x7f319800c880 comp rx=0 tx=0).stop 2024-05-28T06:26:49.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.155+0000 7f31afe00640 1 -- 172.21.15.96:0/2061884306 shutdown_connections 2024-05-28T06:26:49.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.155+0000 7f31afe00640 1 --2- 172.21.15.96:0/2061884306 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f318c06ea20 0x7f318c070ee0 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:49.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.155+0000 7f31afe00640 1 --2- 172.21.15.96:0/2061884306 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f31b0156b30 0x7f31b0073180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:49.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.155+0000 7f31afe00640 1 --2- 172.21.15.96:0/2061884306 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f31b00ad940 0x7f31b0072c40 unknown :-1 s=CLOSED pgs=2211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:49.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.155+0000 7f31afe00640 1 --2- 172.21.15.96:0/2061884306 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f31b006ee40 0x7f31b0142430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:49.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.155+0000 7f31afe00640 1 -- 172.21.15.96:0/2061884306 >> 172.21.15.96:0/2061884306 conn(0x7f31b00b1fc0 msgr2=0x7f31b014c3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:26:49.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.155+0000 7f31afe00640 1 -- 172.21.15.96:0/2061884306 shutdown_connections 2024-05-28T06:26:49.150 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.155+0000 7f31afe00640 1 -- 172.21.15.96:0/2061884306 wait complete. 2024-05-28T06:26:49.166 INFO:tasks.ceph:Scrubbing osd.4 2024-05-28T06:26:49.167 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-05-28T06:26:49.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.370+0000 7feb7de00640 1 Processor -- start 2024-05-28T06:26:49.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.370+0000 7feb7de00640 1 -- start start 2024-05-28T06:26:49.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.371+0000 7feb7de00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7feb7815b340 0x7feb7806f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:49.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.371+0000 7feb7de00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7feb7815c290 0x7feb780b3300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:49.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.371+0000 7feb7de00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb7806fa90 con 0x7feb7815b720 2024-05-28T06:26:49.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.371+0000 7feb7de00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7feb780af9b0 con 0x7feb7815c290 2024-05-28T06:26:49.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.371+0000 7feb7de00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7feb780aeda0 con 0x7feb7815b340 2024-05-28T06:26:49.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.372+0000 7feb77400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7feb7815c290 0x7feb780b3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:49.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.372+0000 7feb77400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7feb7815c290 0x7feb780b3300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:45880/0 (socket says 172.21.15.96:45880) 2024-05-28T06:26:49.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.372+0000 7feb77400640 1 -- 172.21.15.96:0/1985484801 learned_addr learned my addr 172.21.15.96:0/1985484801 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:26:49.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.372+0000 7feb76a00640 1 --2- 172.21.15.96:0/1985484801 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7feb7815b340 0x7feb7806f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:49.367 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.372+0000 7feb77400640 1 -- 172.21.15.96:0/1985484801 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7feb7815b340 msgr2=0x7feb7806f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:49.367 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.372+0000 7feb77400640 1 --2- 172.21.15.96:0/1985484801 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7feb7815b340 0x7feb7806f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:49.367 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.372+0000 7feb77400640 1 -- 172.21.15.96:0/1985484801 >> v1:172.21.15.123:6789/0 conn(0x7feb7815b720 legacy=0x7feb7815bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:49.367 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.372+0000 7feb77400640 1 -- 172.21.15.96:0/1985484801 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb780af080 con 0x7feb7815c290 2024-05-28T06:26:49.368 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.373+0000 7feb77400640 1 --2- 172.21.15.96:0/1985484801 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feb7815c290 0x7feb780b3300 secure :-1 s=READY pgs=2212 cs=0 l=1 rev1=1 crypto rx=0x7feb70009e50 tx=0x7feb7002a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5b834a52a325ddf server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:49.368 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.374+0000 7feb75600640 1 -- 172.21.15.96:0/1985484801 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb70036070 con 0x7feb7815c290 2024-05-28T06:26:49.368 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.374+0000 7feb75600640 1 -- 172.21.15.96:0/1985484801 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb70031600 con 0x7feb7815c290 2024-05-28T06:26:49.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.374+0000 7feb75600640 1 -- 172.21.15.96:0/1985484801 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb700319d0 con 0x7feb7815c290 2024-05-28T06:26:49.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.374+0000 7feb7de00640 1 -- 172.21.15.96:0/1985484801 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feb7815c290 msgr2=0x7feb780b3300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:49.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.374+0000 7feb7de00640 1 --2- 172.21.15.96:0/1985484801 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feb7815c290 0x7feb780b3300 secure :-1 s=READY pgs=2212 cs=0 l=1 rev1=1 crypto rx=0x7feb70009e50 tx=0x7feb7002a6c0 comp rx=0 tx=0).stop 2024-05-28T06:26:49.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.374+0000 7feb7de00640 1 -- 172.21.15.96:0/1985484801 shutdown_connections 2024-05-28T06:26:49.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.374+0000 7feb7de00640 1 --2- 172.21.15.96:0/1985484801 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feb7815c290 0x7feb780b3300 unknown :-1 s=CLOSED pgs=2212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:49.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.374+0000 7feb7de00640 1 --2- 172.21.15.96:0/1985484801 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7feb7815b340 0x7feb7806f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:49.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.374+0000 7feb7de00640 1 -- 172.21.15.96:0/1985484801 >> 172.21.15.96:0/1985484801 conn(0x7feb780b1e50 msgr2=0x7feb780b2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:26:49.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.374+0000 7feb7de00640 1 -- 172.21.15.96:0/1985484801 shutdown_connections 2024-05-28T06:26:49.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.374+0000 7feb7de00640 1 -- 172.21.15.96:0/1985484801 wait complete. 2024-05-28T06:26:49.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.376+0000 7feb7de00640 1 Processor -- start 2024-05-28T06:26:49.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.376+0000 7feb7de00640 1 -- start start 2024-05-28T06:26:49.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.377+0000 7feb7de00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7feb7815b340 0x7feb780b16b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:49.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.377+0000 7feb7de00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feb7815b720 0x7feb7814e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:49.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.377+0000 7feb7de00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb7815c290 0x7feb78159a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:49.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.377+0000 7feb7de00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7feb780aea30 con 0x7feb7815b720 2024-05-28T06:26:49.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.377+0000 7feb7de00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7feb780af640 con 0x7feb7815c290 2024-05-28T06:26:49.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.377+0000 7feb7de00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7feb7815c8c0 con 0x7feb7815b340 2024-05-28T06:26:49.373 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.378+0000 7feb76000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feb7815b720 0x7feb7814e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:49.373 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.378+0000 7feb76000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feb7815b720 0x7feb7814e8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:45892/0 (socket says 172.21.15.96:45892) 2024-05-28T06:26:49.373 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.378+0000 7feb76000640 1 -- 172.21.15.96:0/1038757552 learned_addr learned my addr 172.21.15.96:0/1038757552 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:26:49.373 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.378+0000 7feb77400640 1 --2- 172.21.15.96:0/1038757552 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb7815c290 0x7feb78159a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:49.373 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.378+0000 7feb76a00640 1 --2- 172.21.15.96:0/1038757552 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7feb7815b340 0x7feb780b16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:49.373 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.378+0000 7feb76000640 1 -- 172.21.15.96:0/1038757552 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7feb7815b340 msgr2=0x7feb780b16b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:49.373 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.378+0000 7feb76000640 1 --2- 172.21.15.96:0/1038757552 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7feb7815b340 0x7feb780b16b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:49.373 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.378+0000 7feb76000640 1 -- 172.21.15.96:0/1038757552 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb7815c290 msgr2=0x7feb78159a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:49.374 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.378+0000 7feb76000640 1 --2- 172.21.15.96:0/1038757552 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb7815c290 0x7feb78159a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:49.374 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.378+0000 7feb76000640 1 -- 172.21.15.96:0/1038757552 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb7815d8d0 con 0x7feb7815b720 2024-05-28T06:26:49.374 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.378+0000 7feb76a00640 1 --2- 172.21.15.96:0/1038757552 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7feb7815b340 0x7feb780b16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:26:49.374 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.378+0000 7feb76000640 1 --2- 172.21.15.96:0/1038757552 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feb7815b720 0x7feb7814e8e0 secure :-1 s=READY pgs=2213 cs=0 l=1 rev1=1 crypto rx=0x7feb6400c430 tx=0x7feb6400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:49.375 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.379+0000 7feb6f400640 1 -- 172.21.15.96:0/1038757552 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb64011790 con 0x7feb7815b720 2024-05-28T06:26:49.375 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.379+0000 7feb6f400640 1 -- 172.21.15.96:0/1038757552 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb64017070 con 0x7feb7815b720 2024-05-28T06:26:49.375 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.379+0000 7feb7de00640 1 -- 172.21.15.96:0/1038757552 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb7815db90 con 0x7feb7815b720 2024-05-28T06:26:49.375 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.379+0000 7feb6f400640 1 -- 172.21.15.96:0/1038757552 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb64011bb0 con 0x7feb7815b720 2024-05-28T06:26:49.375 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.379+0000 7feb7de00640 1 -- 172.21.15.96:0/1038757552 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb7814ee20 con 0x7feb7815b720 2024-05-28T06:26:49.376 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.380+0000 7feb7de00640 1 -- 172.21.15.96:0/1038757552 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7feb780b1bf0 con 0x7feb7815b720 2024-05-28T06:26:49.376 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.381+0000 7feb6f400640 1 -- 172.21.15.96:0/1038757552 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 49) ==== 94049+0+0 (secure 0 0 0) 0x7feb6401c460 con 0x7feb7815b720 2024-05-28T06:26:49.377 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.382+0000 7feb6f400640 1 --2- 172.21.15.96:0/1038757552 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7feb5806ea20 0x7feb58070ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:49.377 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.382+0000 7feb6f400640 1 -- 172.21.15.96:0/1038757552 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 5823+0+0 (secure 0 0 0) 0x7feb6408abe0 con 0x7feb7815b720 2024-05-28T06:26:49.377 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.382+0000 7feb6f400640 1 --2- 172.21.15.96:0/1038757552 >> [v2:172.21.15.123:6800/3445772216,v1:172.21.15.123:6801/3445772216] conn(0x7feb58076d40 0x7feb580791c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:49.377 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.382+0000 7feb6f400640 1 -- 172.21.15.96:0/1038757552 --> [v2:172.21.15.123:6800/3445772216,v1:172.21.15.123:6801/3445772216] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7feb64021e50 con 0x7feb58076d40 2024-05-28T06:26:49.377 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.382+0000 7feb6f400640 1 -- 172.21.15.96:0/1038757552 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_get_version_reply(handle=1 version=132) ==== 24+0+0 (secure 0 0 0) 0x7feb6408c250 con 0x7feb7815b720 2024-05-28T06:26:49.377 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.383+0000 7feb76a00640 1 --2- 172.21.15.96:0/1038757552 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7feb5806ea20 0x7feb58070ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:49.377 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.383+0000 7feb77400640 1 --2- 172.21.15.96:0/1038757552 >> [v2:172.21.15.123:6800/3445772216,v1:172.21.15.123:6801/3445772216] conn(0x7feb58076d40 0x7feb580791c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:49.378 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.384+0000 7feb76a00640 1 --2- 172.21.15.96:0/1038757552 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7feb5806ea20 0x7feb58070ee0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7feb68006fd0 tx=0x7feb68008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:49.378 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.384+0000 7feb77400640 1 --2- 172.21.15.96:0/1038757552 >> [v2:172.21.15.123:6800/3445772216,v1:172.21.15.123:6801/3445772216] conn(0x7feb58076d40 0x7feb580791c0 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:49.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.386+0000 7feb6f400640 1 -- 172.21.15.96:0/1038757552 <== osd.4 v2:172.21.15.123:6800/3445772216 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7feb64021e50 con 0x7feb58076d40 2024-05-28T06:26:49.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.401+0000 7feb7de00640 1 -- 172.21.15.96:0/1038757552 --> [v2:172.21.15.123:6800/3445772216,v1:172.21.15.123:6801/3445772216] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7feb78155fe0 con 0x7feb58076d40 2024-05-28T06:26:49.395 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.402+0000 7feb6f400640 1 -- 172.21.15.96:0/1038757552 <== osd.4 v2:172.21.15.123:6800/3445772216 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7feb78155fe0 con 0x7feb58076d40 2024-05-28T06:26:49.395 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:26:49.396 INFO:teuthology.orchestra.run.smithi096.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-28T06:26:49.396 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:26:49.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.402+0000 7feb7de00640 1 -- 172.21.15.96:0/1038757552 >> [v2:172.21.15.123:6800/3445772216,v1:172.21.15.123:6801/3445772216] conn(0x7feb58076d40 msgr2=0x7feb580791c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:49.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.402+0000 7feb7de00640 1 --2- 172.21.15.96:0/1038757552 >> [v2:172.21.15.123:6800/3445772216,v1:172.21.15.123:6801/3445772216] conn(0x7feb58076d40 0x7feb580791c0 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:49.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.402+0000 7feb7de00640 1 -- 172.21.15.96:0/1038757552 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7feb5806ea20 msgr2=0x7feb58070ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:49.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.402+0000 7feb7de00640 1 --2- 172.21.15.96:0/1038757552 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7feb5806ea20 0x7feb58070ee0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7feb68006fd0 tx=0x7feb68008040 comp rx=0 tx=0).stop 2024-05-28T06:26:49.397 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.402+0000 7feb7de00640 1 -- 172.21.15.96:0/1038757552 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feb7815b720 msgr2=0x7feb7814e8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:49.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.402+0000 7feb7de00640 1 --2- 172.21.15.96:0/1038757552 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feb7815b720 0x7feb7814e8e0 secure :-1 s=READY pgs=2213 cs=0 l=1 rev1=1 crypto rx=0x7feb6400c430 tx=0x7feb6400c8f0 comp rx=0 tx=0).stop 2024-05-28T06:26:49.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.402+0000 7feb77400640 1 -- 172.21.15.96:0/1038757552 reap_dead start 2024-05-28T06:26:49.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.402+0000 7feb7de00640 1 -- 172.21.15.96:0/1038757552 shutdown_connections 2024-05-28T06:26:49.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.402+0000 7feb7de00640 1 -- 172.21.15.96:0/1038757552 >> 172.21.15.96:0/1038757552 conn(0x7feb780b1e50 msgr2=0x7feb780b3640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:26:49.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.403+0000 7feb7de00640 1 -- 172.21.15.96:0/1038757552 shutdown_connections 2024-05-28T06:26:49.398 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.403+0000 7feb7de00640 1 -- 172.21.15.96:0/1038757552 wait complete. 2024-05-28T06:26:49.413 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-05-28T06:26:49.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.620+0000 7f43e9800640 1 Processor -- start 2024-05-28T06:26:49.614 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.620+0000 7f43e9800640 1 -- start start 2024-05-28T06:26:49.615 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.622+0000 7f43e9800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f43e40ad940 0x7f43e40b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:49.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.622+0000 7f43e9800640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43e4156f10 con 0x7f43e406ee40 2024-05-28T06:26:49.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.622+0000 7f43e9800640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43e406dda0 con 0x7f43e4156b30 2024-05-28T06:26:49.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.622+0000 7f43e9800640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f43e406e080 con 0x7f43e40ad940 2024-05-28T06:26:49.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.622+0000 7f43db400640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f43e406ee40 0x7f43e40ad230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:43390/0 (socket says 172.21.15.96:43390) 2024-05-28T06:26:49.616 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.622+0000 7f43db400640 1 -- 172.21.15.96:0/3490618555 learned_addr learned my addr 172.21.15.96:0/3490618555 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:26:49.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.623+0000 7f43e8e00640 1 --2- 172.21.15.96:0/3490618555 >> v2:172.21.15.123:3300/0 conn(0x7f43e40ad940 0x7f43e40b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:49.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.623+0000 7f43e3400640 1 -- 172.21.15.96:0/3490618555 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 272429001 0 0) 0x7f43e4156f10 con 0x7f43e406ee40 2024-05-28T06:26:49.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.623+0000 7f43e3400640 1 -- 172.21.15.96:0/3490618555 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f43c8003520 con 0x7f43e406ee40 2024-05-28T06:26:49.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.623+0000 7f43e3400640 1 -- 172.21.15.96:0/3490618555 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 485995069 0 0) 0x7f43c8003520 con 0x7f43e406ee40 2024-05-28T06:26:49.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.623+0000 7f43e3400640 1 -- 172.21.15.96:0/3490618555 >> v2:172.21.15.123:3300/0 conn(0x7f43e40ad940 msgr2=0x7f43e40b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:49.618 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.623+0000 7f43e3400640 1 --2- 172.21.15.96:0/3490618555 >> v2:172.21.15.123:3300/0 conn(0x7f43e40ad940 0x7f43e40b32c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:49.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.623+0000 7f43e3400640 1 -- 172.21.15.96:0/3490618555 >> v1:172.21.15.123:6789/0 conn(0x7f43e4156b30 legacy=0x7f43e40af6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:49.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.623+0000 7f43e3400640 1 -- 172.21.15.96:0/3490618555 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43e406e360 con 0x7f43e406ee40 2024-05-28T06:26:49.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.623+0000 7f43e3400640 1 -- 172.21.15.96:0/3490618555 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f43cc001670 con 0x7f43e406ee40 2024-05-28T06:26:49.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.623+0000 7f43e3400640 1 -- 172.21.15.96:0/3490618555 >> v1:172.21.15.96:6789/0 conn(0x7f43e406ee40 legacy=0x7f43e40ad230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:49.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.623+0000 7f43e3400640 1 --2- 172.21.15.96:0/3490618555 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f43c80039b0 0x7f43c8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:49.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.623+0000 7f43e3400640 1 --2- 172.21.15.96:0/3490618555 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f43c8004800 0x7f43c8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:49.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.623+0000 7f43e3400640 1 --2- 172.21.15.96:0/3490618555 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f43c80051c0 0x7f43c800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:49.619 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.623+0000 7f43e3400640 1 -- 172.21.15.96:0/3490618555 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f43e406e080 con 0x7f43c8004800 2024-05-28T06:26:49.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.623+0000 7f43e3400640 1 -- 172.21.15.96:0/3490618555 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f43c800c160 con 0x7f43c80051c0 2024-05-28T06:26:49.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.623+0000 7f43e3400640 1 -- 172.21.15.96:0/3490618555 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f43c800c360 con 0x7f43c80039b0 2024-05-28T06:26:49.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.624+0000 7f43db400640 1 --2- 172.21.15.96:0/3490618555 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f43c8004800 0x7f43c8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:49.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.624+0000 7f43e3e00640 1 --2- 172.21.15.96:0/3490618555 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f43c80039b0 0x7f43c8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:49.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.624+0000 7f43e8e00640 1 --2- 172.21.15.96:0/3490618555 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f43c80051c0 0x7f43c800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:49.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.624+0000 7f43db400640 1 -- 172.21.15.96:0/3490618555 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f43c80039b0 msgr2=0x7f43c8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:49.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.624+0000 7f43db400640 1 --2- 172.21.15.96:0/3490618555 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f43c80039b0 0x7f43c8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:49.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.624+0000 7f43db400640 1 -- 172.21.15.96:0/3490618555 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f43c80051c0 msgr2=0x7f43c800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:49.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.624+0000 7f43db400640 1 --2- 172.21.15.96:0/3490618555 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f43c80051c0 0x7f43c800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:49.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.624+0000 7f43db400640 1 -- 172.21.15.96:0/3490618555 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43c8003520 con 0x7f43c8004800 2024-05-28T06:26:49.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.624+0000 7f43e8e00640 1 --2- 172.21.15.96:0/3490618555 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f43c80051c0 0x7f43c800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-28T06:26:49.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.624+0000 7f43e8e00640 1 -- 172.21.15.96:0/3490618555 reap_dead start 2024-05-28T06:26:49.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.625+0000 7f43db400640 1 --2- 172.21.15.96:0/3490618555 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f43c8004800 0x7f43c8004c80 secure :-1 s=READY pgs=2215 cs=0 l=1 rev1=1 crypto rx=0x7f43cc00c240 tx=0x7f43c8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7d1f2b8b1e7f049 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:49.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.626+0000 7f43e3400640 1 -- 172.21.15.96:0/3490618555 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43cc007cf0 con 0x7f43c8004800 2024-05-28T06:26:49.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.626+0000 7f43e3400640 1 -- 172.21.15.96:0/3490618555 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43cc005af0 con 0x7f43c8004800 2024-05-28T06:26:49.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.626+0000 7f43e3400640 1 -- 172.21.15.96:0/3490618555 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43cc007070 con 0x7f43c8004800 2024-05-28T06:26:49.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.626+0000 7f43e9800640 1 -- 172.21.15.96:0/3490618555 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f43c8004800 msgr2=0x7f43c8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:49.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.626+0000 7f43e9800640 1 --2- 172.21.15.96:0/3490618555 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f43c8004800 0x7f43c8004c80 secure :-1 s=READY pgs=2215 cs=0 l=1 rev1=1 crypto rx=0x7f43cc00c240 tx=0x7f43c8005830 comp rx=0 tx=0).stop 2024-05-28T06:26:49.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.627+0000 7f43e9800640 1 -- 172.21.15.96:0/3490618555 shutdown_connections 2024-05-28T06:26:49.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.627+0000 7f43e9800640 1 --2- 172.21.15.96:0/3490618555 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f43c8004800 0x7f43c8004c80 unknown :-1 s=CLOSED pgs=2215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:49.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.627+0000 7f43e9800640 1 -- 172.21.15.96:0/3490618555 >> 172.21.15.96:0/3490618555 conn(0x7f43e40b1fc0 msgr2=0x7f43e40b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:26:49.623 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.627+0000 7f43e9800640 1 -- 172.21.15.96:0/3490618555 shutdown_connections 2024-05-28T06:26:49.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.627+0000 7f43e9800640 1 -- 172.21.15.96:0/3490618555 wait complete. 2024-05-28T06:26:49.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.628+0000 7f43e9800640 1 Processor -- start 2024-05-28T06:26:49.624 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.629+0000 7f43e9800640 1 -- start start 2024-05-28T06:26:49.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.629+0000 7f43e9800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f43c8004800 0x7f43e4157f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:49.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.629+0000 7f43e9800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f43e4155cf0 0x7f43e4158ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:49.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.629+0000 7f43e9800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f43e415a740 0x7f43e41f7470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:49.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.629+0000 7f43e9800640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f43e40b2db0 con 0x7f43e4155cf0 2024-05-28T06:26:49.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.629+0000 7f43e9800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f43e40b2c30 con 0x7f43e415a740 2024-05-28T06:26:49.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.629+0000 7f43e9800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f43e406f2f0 con 0x7f43c8004800 2024-05-28T06:26:49.626 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.630+0000 7f43db400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f43e4155cf0 0x7f43e4158ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:49.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.630+0000 7f43db400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f43e4155cf0 0x7f43e4158ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:45906/0 (socket says 172.21.15.96:45906) 2024-05-28T06:26:49.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.630+0000 7f43db400640 1 -- 172.21.15.96:0/2740058390 learned_addr learned my addr 172.21.15.96:0/2740058390 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:26:49.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.630+0000 7f43e3e00640 1 --2- 172.21.15.96:0/2740058390 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f43c8004800 0x7f43e4157f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:49.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.630+0000 7f43e8e00640 1 --2- 172.21.15.96:0/2740058390 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f43e415a740 0x7f43e41f7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:49.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.630+0000 7f43db400640 1 -- 172.21.15.96:0/2740058390 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f43c8004800 msgr2=0x7f43e4157f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:49.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.630+0000 7f43db400640 1 --2- 172.21.15.96:0/2740058390 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f43c8004800 0x7f43e4157f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:49.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.630+0000 7f43db400640 1 -- 172.21.15.96:0/2740058390 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f43e415a740 msgr2=0x7f43e41f7470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:49.627 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.630+0000 7f43db400640 1 --2- 172.21.15.96:0/2740058390 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f43e415a740 0x7f43e41f7470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:49.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.630+0000 7f43db400640 1 -- 172.21.15.96:0/2740058390 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43e415b240 con 0x7f43e4155cf0 2024-05-28T06:26:49.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.631+0000 7f43db400640 1 --2- 172.21.15.96:0/2740058390 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f43e4155cf0 0x7f43e4158ae0 secure :-1 s=READY pgs=2216 cs=0 l=1 rev1=1 crypto rx=0x7f43cc001970 tx=0x7f43e415ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:49.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.631+0000 7f43e0c00640 1 -- 172.21.15.96:0/2740058390 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43cc037040 con 0x7f43e4155cf0 2024-05-28T06:26:49.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.631+0000 7f43e0c00640 1 -- 172.21.15.96:0/2740058390 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43cc0047e0 con 0x7f43e4155cf0 2024-05-28T06:26:49.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.631+0000 7f43e0c00640 1 -- 172.21.15.96:0/2740058390 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43cc001670 con 0x7f43e4155cf0 2024-05-28T06:26:49.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.631+0000 7f43e9800640 1 -- 172.21.15.96:0/2740058390 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43e415b4e0 con 0x7f43e4155cf0 2024-05-28T06:26:49.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.631+0000 7f43e9800640 1 -- 172.21.15.96:0/2740058390 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43e414d0b0 con 0x7f43e4155cf0 2024-05-28T06:26:49.628 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.632+0000 7f43e9800640 1 -- 172.21.15.96:0/2740058390 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43e4159590 con 0x7f43e4155cf0 2024-05-28T06:26:49.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.633+0000 7f43e0c00640 1 -- 172.21.15.96:0/2740058390 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 49) ==== 94049+0+0 (secure 0 0 0) 0x7f43cc004b90 con 0x7f43e4155cf0 2024-05-28T06:26:49.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.634+0000 7f43e0c00640 1 --2- 172.21.15.96:0/2740058390 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f43c006ea20 0x7f43c0070ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:49.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.634+0000 7f43e0c00640 1 -- 172.21.15.96:0/2740058390 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 5823+0+0 (secure 0 0 0) 0x7f43cc0aae50 con 0x7f43e4155cf0 2024-05-28T06:26:49.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.635+0000 7f43e3e00640 1 --2- 172.21.15.96:0/2740058390 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f43c006ea20 0x7f43c0070ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:49.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.636+0000 7f43e3e00640 1 --2- 172.21.15.96:0/2740058390 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f43c006ea20 0x7f43c0070ee0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f43d4007fd0 tx=0x7f43d4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:49.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.639+0000 7f43e0c00640 1 -- 172.21.15.96:0/2740058390 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f43cc07ad60 con 0x7f43e4155cf0 2024-05-28T06:26:49.839 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.846+0000 7f43e9800640 1 -- 172.21.15.96:0/2740058390 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7f43e406f7f0 con 0x7f43c006ea20 2024-05-28T06:26:49.840 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.847+0000 7f43e0c00640 1 -- 172.21.15.96:0/2740058390 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f43e406f7f0 con 0x7f43c006ea20 2024-05-28T06:26:49.841 INFO:teuthology.orchestra.run.smithi096.stderr:instructed osd(s) 4 to deep-scrub 2024-05-28T06:26:49.843 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.849+0000 7f43e9800640 1 -- 172.21.15.96:0/2740058390 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f43c006ea20 msgr2=0x7f43c0070ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:49.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.849+0000 7f43e9800640 1 --2- 172.21.15.96:0/2740058390 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f43c006ea20 0x7f43c0070ee0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f43d4007fd0 tx=0x7f43d4009040 comp rx=0 tx=0).stop 2024-05-28T06:26:49.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.850+0000 7f43e9800640 1 -- 172.21.15.96:0/2740058390 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f43e4155cf0 msgr2=0x7f43e4158ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:49.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.850+0000 7f43e9800640 1 --2- 172.21.15.96:0/2740058390 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f43e4155cf0 0x7f43e4158ae0 secure :-1 s=READY pgs=2216 cs=0 l=1 rev1=1 crypto rx=0x7f43cc001970 tx=0x7f43e415ca90 comp rx=0 tx=0).stop 2024-05-28T06:26:49.844 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.850+0000 7f43e9800640 1 -- 172.21.15.96:0/2740058390 shutdown_connections 2024-05-28T06:26:49.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.850+0000 7f43e9800640 1 --2- 172.21.15.96:0/2740058390 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f43c006ea20 0x7f43c0070ee0 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:49.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.850+0000 7f43e9800640 1 --2- 172.21.15.96:0/2740058390 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f43e415a740 0x7f43e41f7470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:49.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.850+0000 7f43e9800640 1 --2- 172.21.15.96:0/2740058390 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f43e4155cf0 0x7f43e4158ae0 unknown :-1 s=CLOSED pgs=2216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:49.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.850+0000 7f43e9800640 1 --2- 172.21.15.96:0/2740058390 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f43c8004800 0x7f43e4157f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:49.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.850+0000 7f43e9800640 1 -- 172.21.15.96:0/2740058390 >> 172.21.15.96:0/2740058390 conn(0x7f43e40b1fc0 msgr2=0x7f43e415b760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:26:49.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.850+0000 7f43e9800640 1 -- 172.21.15.96:0/2740058390 shutdown_connections 2024-05-28T06:26:49.845 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:49.850+0000 7f43e9800640 1 -- 172.21.15.96:0/2740058390 wait complete. 2024-05-28T06:26:49.863 INFO:tasks.ceph:Scrubbing osd.5 2024-05-28T06:26:49.863 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-05-28T06:26:50.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.084+0000 7f99ae400640 1 Processor -- start 2024-05-28T06:26:50.078 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.084+0000 7f99ae400640 1 -- start start 2024-05-28T06:26:50.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.086+0000 7f99ae400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f99a80ce560 0x7f99a80b4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:50.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.086+0000 7f99ae400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f99a80d0430 0x7f99a80d0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:50.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.086+0000 7f99ae400640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f99a80d0d70 con 0x7f99a815f0c0 2024-05-28T06:26:50.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.086+0000 7f99ae400640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f99a815bb30 con 0x7f99a80ce560 2024-05-28T06:26:50.079 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.086+0000 7f99ae400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f99a80b3ba0 con 0x7f99a80d0430 2024-05-28T06:26:50.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.086+0000 7f99ad000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f99a80ce560 0x7f99a80b4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:50.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.086+0000 7f99ad000640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f99a80ce560 0x7f99a80b4b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:45916/0 (socket says 172.21.15.96:45916) 2024-05-28T06:26:50.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.086+0000 7f99ad000640 1 -- 172.21.15.96:0/3839895131 learned_addr learned my addr 172.21.15.96:0/3839895131 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:26:50.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.086+0000 7f99ad000640 1 -- 172.21.15.96:0/3839895131 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f99a80d0430 msgr2=0x7f99a80d0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:50.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.086+0000 7f99a7e00640 1 --2- 172.21.15.96:0/3839895131 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f99a80d0430 0x7f99a80d0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:50.081 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.086+0000 7f99ad000640 1 --2- 172.21.15.96:0/3839895131 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f99a80d0430 0x7f99a80d0830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:50.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.087+0000 7f99ad000640 1 -- 172.21.15.96:0/3839895131 >> v1:172.21.15.123:6789/0 conn(0x7f99a815f0c0 legacy=0x7f99a815f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:50.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.087+0000 7f99ad000640 1 -- 172.21.15.96:0/3839895131 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99a815bdd0 con 0x7f99a80ce560 2024-05-28T06:26:50.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.087+0000 7f99a7e00640 1 --2- 172.21.15.96:0/3839895131 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f99a80d0430 0x7f99a80d0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:26:50.082 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.088+0000 7f99ad000640 1 --2- 172.21.15.96:0/3839895131 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f99a80ce560 0x7f99a80b4b80 secure :-1 s=READY pgs=2217 cs=0 l=1 rev1=1 crypto rx=0x7f999c009870 tx=0x7f999c02a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c51c61da8061461 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:50.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.088+0000 7f99a7400640 1 -- 172.21.15.96:0/3839895131 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f999c037070 con 0x7f99a80ce560 2024-05-28T06:26:50.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.088+0000 7f99a7400640 1 -- 172.21.15.96:0/3839895131 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f999c032440 con 0x7f99a80ce560 2024-05-28T06:26:50.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.088+0000 7f99a7400640 1 -- 172.21.15.96:0/3839895131 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f999c032810 con 0x7f99a80ce560 2024-05-28T06:26:50.083 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.088+0000 7f99ae400640 1 -- 172.21.15.96:0/3839895131 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f99a80ce560 msgr2=0x7f99a80b4b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:50.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.088+0000 7f99ae400640 1 --2- 172.21.15.96:0/3839895131 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f99a80ce560 0x7f99a80b4b80 secure :-1 s=READY pgs=2217 cs=0 l=1 rev1=1 crypto rx=0x7f999c009870 tx=0x7f999c02a210 comp rx=0 tx=0).stop 2024-05-28T06:26:50.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.089+0000 7f99ae400640 1 -- 172.21.15.96:0/3839895131 shutdown_connections 2024-05-28T06:26:50.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.089+0000 7f99ae400640 1 --2- 172.21.15.96:0/3839895131 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f99a80d0430 0x7f99a80d0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:50.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.089+0000 7f99ae400640 1 --2- 172.21.15.96:0/3839895131 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f99a80ce560 0x7f99a80b4b80 unknown :-1 s=CLOSED pgs=2217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:50.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.089+0000 7f99ae400640 1 -- 172.21.15.96:0/3839895131 >> 172.21.15.96:0/3839895131 conn(0x7f99a806bae0 msgr2=0x7f99a806bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:26:50.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.089+0000 7f99ae400640 1 -- 172.21.15.96:0/3839895131 shutdown_connections 2024-05-28T06:26:50.084 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.089+0000 7f99ae400640 1 -- 172.21.15.96:0/3839895131 wait complete. 2024-05-28T06:26:50.086 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.091+0000 7f99ae400640 1 Processor -- start 2024-05-28T06:26:50.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.091+0000 7f99ae400640 1 -- start start 2024-05-28T06:26:50.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.092+0000 7f99ae400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99a80ce560 0x7f99a8156fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:50.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.092+0000 7f99ae400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f99a80d0430 0x7f99a80aed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:50.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.092+0000 7f99ae400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f99a815f0c0 0x7f99a8072bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:50.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.092+0000 7f99ae400640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f99a815c670 con 0x7f99a80d0430 2024-05-28T06:26:50.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.092+0000 7f99ae400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f99a815c390 con 0x7f99a80ce560 2024-05-28T06:26:50.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.092+0000 7f99ae400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f99a806f630 con 0x7f99a815f0c0 2024-05-28T06:26:50.087 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.092+0000 7f99a7e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f99a80d0430 0x7f99a80aed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:50.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.092+0000 7f99a7e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f99a80d0430 0x7f99a80aed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:45922/0 (socket says 172.21.15.96:45922) 2024-05-28T06:26:50.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.092+0000 7f99a7e00640 1 -- 172.21.15.96:0/796789097 learned_addr learned my addr 172.21.15.96:0/796789097 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:26:50.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.092+0000 7f99ad000640 1 --2- 172.21.15.96:0/796789097 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99a80ce560 0x7f99a8156fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:50.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.092+0000 7f99a7e00640 1 -- 172.21.15.96:0/796789097 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f99a815f0c0 msgr2=0x7f99a8072bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:50.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.092+0000 7f99a7e00640 1 --2- 172.21.15.96:0/796789097 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f99a815f0c0 0x7f99a8072bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:50.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.092+0000 7f99a7e00640 1 -- 172.21.15.96:0/796789097 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99a80ce560 msgr2=0x7f99a8156fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:50.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.092+0000 7f99a7e00640 1 --2- 172.21.15.96:0/796789097 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99a80ce560 0x7f99a8156fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:50.088 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.092+0000 7f99a7e00640 1 -- 172.21.15.96:0/796789097 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99a80afdd0 con 0x7f99a80d0430 2024-05-28T06:26:50.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.093+0000 7f99a7e00640 1 --2- 172.21.15.96:0/796789097 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f99a80d0430 0x7f99a80aed70 secure :-1 s=READY pgs=2218 cs=0 l=1 rev1=1 crypto rx=0x7f999400c7f0 tx=0x7f999400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:50.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.093+0000 7f99a4c00640 1 -- 172.21.15.96:0/796789097 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9994015070 con 0x7f99a80d0430 2024-05-28T06:26:50.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.093+0000 7f99a4c00640 1 -- 172.21.15.96:0/796789097 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99940079d0 con 0x7f99a80d0430 2024-05-28T06:26:50.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.093+0000 7f99ae400640 1 -- 172.21.15.96:0/796789097 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99a80b00b0 con 0x7f99a80d0430 2024-05-28T06:26:50.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.093+0000 7f99a4c00640 1 -- 172.21.15.96:0/796789097 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9994007da0 con 0x7f99a80d0430 2024-05-28T06:26:50.089 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.093+0000 7f99ae400640 1 -- 172.21.15.96:0/796789097 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99a80b0670 con 0x7f99a80d0430 2024-05-28T06:26:50.090 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.094+0000 7f99ae400640 1 -- 172.21.15.96:0/796789097 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f99a80689e0 con 0x7f99a80d0430 2024-05-28T06:26:50.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.095+0000 7f99a4c00640 1 -- 172.21.15.96:0/796789097 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 49) ==== 94049+0+0 (secure 0 0 0) 0x7f9994018070 con 0x7f99a80d0430 2024-05-28T06:26:50.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.096+0000 7f99a4c00640 1 --2- 172.21.15.96:0/796789097 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f998806ea50 0x7f9988070f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:50.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.096+0000 7f99a4c00640 1 -- 172.21.15.96:0/796789097 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 5823+0+0 (secure 0 0 0) 0x7f999408ac30 con 0x7f99a80d0430 2024-05-28T06:26:50.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.096+0000 7f99a4c00640 1 --2- 172.21.15.96:0/796789097 >> [v2:172.21.15.123:6817/2041107525,v1:172.21.15.123:6819/2041107525] conn(0x7f9988076dd0 0x7f9988079250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:50.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.096+0000 7f99a4c00640 1 -- 172.21.15.96:0/796789097 --> [v2:172.21.15.123:6817/2041107525,v1:172.21.15.123:6819/2041107525] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9988079920 con 0x7f9988076dd0 2024-05-28T06:26:50.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.096+0000 7f99a4c00640 1 -- 172.21.15.96:0/796789097 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_get_version_reply(handle=1 version=132) ==== 24+0+0 (secure 0 0 0) 0x7f9994097050 con 0x7f99a80d0430 2024-05-28T06:26:50.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.097+0000 7f99ad000640 1 --2- 172.21.15.96:0/796789097 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f998806ea50 0x7f9988070f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:50.091 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.097+0000 7f99ada00640 1 --2- 172.21.15.96:0/796789097 >> [v2:172.21.15.123:6817/2041107525,v1:172.21.15.123:6819/2041107525] conn(0x7f9988076dd0 0x7f9988079250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:50.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.098+0000 7f99ad000640 1 --2- 172.21.15.96:0/796789097 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f998806ea50 0x7f9988070f10 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f999c034000 tx=0x7f999c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:50.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.098+0000 7f99ada00640 1 --2- 172.21.15.96:0/796789097 >> [v2:172.21.15.123:6817/2041107525,v1:172.21.15.123:6819/2041107525] conn(0x7f9988076dd0 0x7f9988079250 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:50.094 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.100+0000 7f99a4c00640 1 -- 172.21.15.96:0/796789097 <== osd.5 v2:172.21.15.123:6817/2041107525 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9988079920 con 0x7f9988076dd0 2024-05-28T06:26:50.108 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.115+0000 7f99ae400640 1 -- 172.21.15.96:0/796789097 --> [v2:172.21.15.123:6817/2041107525,v1:172.21.15.123:6819/2041107525] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f99a8073c90 con 0x7f9988076dd0 2024-05-28T06:26:50.109 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.116+0000 7f99a4c00640 1 -- 172.21.15.96:0/796789097 <== osd.5 v2:172.21.15.123:6817/2041107525 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f99a8073c90 con 0x7f9988076dd0 2024-05-28T06:26:50.109 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:26:50.110 INFO:teuthology.orchestra.run.smithi096.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-28T06:26:50.110 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:26:50.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.116+0000 7f99ae400640 1 -- 172.21.15.96:0/796789097 >> [v2:172.21.15.123:6817/2041107525,v1:172.21.15.123:6819/2041107525] conn(0x7f9988076dd0 msgr2=0x7f9988079250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:50.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.116+0000 7f99ae400640 1 --2- 172.21.15.96:0/796789097 >> [v2:172.21.15.123:6817/2041107525,v1:172.21.15.123:6819/2041107525] conn(0x7f9988076dd0 0x7f9988079250 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:50.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.116+0000 7f99ae400640 1 -- 172.21.15.96:0/796789097 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f998806ea50 msgr2=0x7f9988070f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:50.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.116+0000 7f99ae400640 1 --2- 172.21.15.96:0/796789097 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f998806ea50 0x7f9988070f10 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f999c034000 tx=0x7f999c02e000 comp rx=0 tx=0).stop 2024-05-28T06:26:50.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.116+0000 7f99ae400640 1 -- 172.21.15.96:0/796789097 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f99a80d0430 msgr2=0x7f99a80aed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:50.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.116+0000 7f99ae400640 1 --2- 172.21.15.96:0/796789097 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f99a80d0430 0x7f99a80aed70 secure :-1 s=READY pgs=2218 cs=0 l=1 rev1=1 crypto rx=0x7f999400c7f0 tx=0x7f999400cbf0 comp rx=0 tx=0).stop 2024-05-28T06:26:50.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.117+0000 7f99ada00640 1 -- 172.21.15.96:0/796789097 reap_dead start 2024-05-28T06:26:50.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.117+0000 7f99ae400640 1 -- 172.21.15.96:0/796789097 shutdown_connections 2024-05-28T06:26:50.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.117+0000 7f99ae400640 1 -- 172.21.15.96:0/796789097 >> 172.21.15.96:0/796789097 conn(0x7f99a806bae0 msgr2=0x7f99a806bec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:26:50.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.117+0000 7f99ae400640 1 -- 172.21.15.96:0/796789097 shutdown_connections 2024-05-28T06:26:50.112 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.117+0000 7f99ae400640 1 -- 172.21.15.96:0/796789097 wait complete. 2024-05-28T06:26:50.128 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-05-28T06:26:50.328 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.335+0000 7f11d8800640 1 Processor -- start 2024-05-28T06:26:50.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.335+0000 7f11d8800640 1 -- start start 2024-05-28T06:26:50.329 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.336+0000 7f11d8800640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f11d006f860 0x7f11d00b3150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:50.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.336+0000 7f11d8800640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f11d00b3690 0x7f11d00d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:50.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.336+0000 7f11d8800640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11d006ed10 con 0x7f11d00d09e0 2024-05-28T06:26:50.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.336+0000 7f11d8800640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f11d00d0df0 con 0x7f11d006f860 2024-05-28T06:26:50.330 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.336+0000 7f11d8800640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f11d015cb00 con 0x7f11d00b3690 2024-05-28T06:26:50.331 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.337+0000 7f11d5e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f11d006f860 0x7f11d00b3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:50.331 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.337+0000 7f11d5e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f11d006f860 0x7f11d00b3150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:45932/0 (socket says 172.21.15.96:45932) 2024-05-28T06:26:50.331 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.337+0000 7f11d5e00640 1 -- 172.21.15.96:0/1755198708 learned_addr learned my addr 172.21.15.96:0/1755198708 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:26:50.331 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.337+0000 7f11d5400640 1 --2- 172.21.15.96:0/1755198708 >> v2:172.21.15.123:3300/0 conn(0x7f11d00b3690 0x7f11d00d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:50.331 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.337+0000 7f11d5e00640 1 -- 172.21.15.96:0/1755198708 >> v2:172.21.15.123:3300/0 conn(0x7f11d00b3690 msgr2=0x7f11d00d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:50.331 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.337+0000 7f11d5e00640 1 --2- 172.21.15.96:0/1755198708 >> v2:172.21.15.123:3300/0 conn(0x7f11d00b3690 0x7f11d00d04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:50.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.337+0000 7f11d5e00640 1 -- 172.21.15.96:0/1755198708 >> v1:172.21.15.123:6789/0 conn(0x7f11d00d09e0 legacy=0x7f11d0160210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:50.332 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.337+0000 7f11d5e00640 1 -- 172.21.15.96:0/1755198708 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11d00b3cc0 con 0x7f11d006f860 2024-05-28T06:26:50.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.338+0000 7f11d5e00640 1 --2- 172.21.15.96:0/1755198708 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11d006f860 0x7f11d00b3150 secure :-1 s=READY pgs=2219 cs=0 l=1 rev1=1 crypto rx=0x7f11d0070120 tx=0x7f11c402a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c64d153ac428b00 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:50.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.339+0000 7f11d4a00640 1 -- 172.21.15.96:0/1755198708 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11c4037070 con 0x7f11d006f860 2024-05-28T06:26:50.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.339+0000 7f11d4a00640 1 -- 172.21.15.96:0/1755198708 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11c4032440 con 0x7f11d006f860 2024-05-28T06:26:50.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.339+0000 7f11d4a00640 1 -- 172.21.15.96:0/1755198708 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11c4032810 con 0x7f11d006f860 2024-05-28T06:26:50.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.339+0000 7f11d8800640 1 -- 172.21.15.96:0/1755198708 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11d006f860 msgr2=0x7f11d00b3150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:50.333 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.339+0000 7f11d8800640 1 --2- 172.21.15.96:0/1755198708 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11d006f860 0x7f11d00b3150 secure :-1 s=READY pgs=2219 cs=0 l=1 rev1=1 crypto rx=0x7f11d0070120 tx=0x7f11c402a230 comp rx=0 tx=0).stop 2024-05-28T06:26:50.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.339+0000 7f11d8800640 1 -- 172.21.15.96:0/1755198708 shutdown_connections 2024-05-28T06:26:50.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.339+0000 7f11d8800640 1 --2- 172.21.15.96:0/1755198708 >> v2:172.21.15.123:3300/0 conn(0x7f11d00b3690 0x7f11d00d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:50.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.339+0000 7f11d8800640 1 --2- 172.21.15.96:0/1755198708 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11d006f860 0x7f11d00b3150 unknown :-1 s=CLOSED pgs=2219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:50.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.339+0000 7f11d8800640 1 -- 172.21.15.96:0/1755198708 >> 172.21.15.96:0/1755198708 conn(0x7f11d006bae0 msgr2=0x7f11d006bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:26:50.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.339+0000 7f11d8800640 1 -- 172.21.15.96:0/1755198708 shutdown_connections 2024-05-28T06:26:50.334 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.340+0000 7f11d8800640 1 -- 172.21.15.96:0/1755198708 wait complete. 2024-05-28T06:26:50.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.341+0000 7f11d8800640 1 Processor -- start 2024-05-28T06:26:50.335 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.341+0000 7f11d8800640 1 -- start start 2024-05-28T06:26:50.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.342+0000 7f11d8800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11d006f860 0x7f11d015f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:50.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.342+0000 7f11d8800640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f11d00b3690 0x7f11d00ced50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:50.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.342+0000 7f11d8800640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f11d00d09e0 0x7f11d0092020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:50.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.342+0000 7f11d8800640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f11d00b4560 con 0x7f11d006f860 2024-05-28T06:26:50.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.342+0000 7f11d8800640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f11d00b4280 con 0x7f11d00d09e0 2024-05-28T06:26:50.336 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.342+0000 7f11d8800640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f11d00b4840 con 0x7f11d00b3690 2024-05-28T06:26:50.337 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.343+0000 7f11d5e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11d006f860 0x7f11d015f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:50.337 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.343+0000 7f11d5e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11d006f860 0x7f11d015f570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:45948/0 (socket says 172.21.15.96:45948) 2024-05-28T06:26:50.337 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.343+0000 7f11d5e00640 1 -- 172.21.15.96:0/2067999919 learned_addr learned my addr 172.21.15.96:0/2067999919 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:26:50.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.343+0000 7f11d5400640 1 --2- 172.21.15.96:0/2067999919 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f11d00b3690 0x7f11d00ced50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:50.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.343+0000 7f11d5e00640 1 -- 172.21.15.96:0/2067999919 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f11d00b3690 msgr2=0x7f11d00ced50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:50.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.343+0000 7f11d5e00640 1 --2- 172.21.15.96:0/2067999919 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f11d00b3690 0x7f11d00ced50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:50.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.343+0000 7f11d5e00640 1 -- 172.21.15.96:0/2067999919 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f11d00d09e0 msgr2=0x7f11d0092020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:50.338 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.343+0000 7f11d5e00640 1 --2- 172.21.15.96:0/2067999919 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f11d00d09e0 0x7f11d0092020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:50.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.343+0000 7f11d5e00640 1 -- 172.21.15.96:0/2067999919 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11d00cf7a0 con 0x7f11d006f860 2024-05-28T06:26:50.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.343+0000 7f11d5e00640 1 --2- 172.21.15.96:0/2067999919 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11d006f860 0x7f11d015f570 secure :-1 s=READY pgs=2220 cs=0 l=1 rev1=1 crypto rx=0x7f11c4000c00 tx=0x7f11c40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:50.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.344+0000 7f11ca000640 1 -- 172.21.15.96:0/2067999919 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11c4041020 con 0x7f11d006f860 2024-05-28T06:26:50.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.344+0000 7f11ca000640 1 -- 172.21.15.96:0/2067999919 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11c40306c0 con 0x7f11d006f860 2024-05-28T06:26:50.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.344+0000 7f11ca000640 1 -- 172.21.15.96:0/2067999919 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11c4037040 con 0x7f11d006f860 2024-05-28T06:26:50.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.344+0000 7f11d8800640 1 -- 172.21.15.96:0/2067999919 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11d015cdb0 con 0x7f11d006f860 2024-05-28T06:26:50.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.344+0000 7f11d8800640 1 -- 172.21.15.96:0/2067999919 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11d015d370 con 0x7f11d006f860 2024-05-28T06:26:50.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.345+0000 7f11d8800640 1 -- 172.21.15.96:0/2067999919 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11d00689e0 con 0x7f11d006f860 2024-05-28T06:26:50.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.346+0000 7f11ca000640 1 -- 172.21.15.96:0/2067999919 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 49) ==== 94049+0+0 (secure 0 0 0) 0x7f11c4044020 con 0x7f11d006f860 2024-05-28T06:26:50.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.347+0000 7f11ca000640 1 --2- 172.21.15.96:0/2067999919 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f11b406ea20 0x7f11b4070ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:50.340 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.347+0000 7f11ca000640 1 -- 172.21.15.96:0/2067999919 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 5823+0+0 (secure 0 0 0) 0x7f11c40ab520 con 0x7f11d006f860 2024-05-28T06:26:50.341 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.347+0000 7f11d5400640 1 --2- 172.21.15.96:0/2067999919 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f11b406ea20 0x7f11b4070ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:50.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.349+0000 7f11d5400640 1 --2- 172.21.15.96:0/2067999919 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f11b406ea20 0x7f11b4070ee0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f11d0099e90 tx=0x7f11c0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:50.345 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.351+0000 7f11ca000640 1 -- 172.21.15.96:0/2067999919 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f11c404b030 con 0x7f11d006f860 2024-05-28T06:26:50.545 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.551+0000 7f11d8800640 1 -- 172.21.15.96:0/2067999919 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7f11d006e2f0 con 0x7f11b406ea20 2024-05-28T06:26:50.546 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.552+0000 7f11ca000640 1 -- 172.21.15.96:0/2067999919 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f11d006e2f0 con 0x7f11b406ea20 2024-05-28T06:26:50.546 INFO:teuthology.orchestra.run.smithi096.stderr:instructed osd(s) 5 to deep-scrub 2024-05-28T06:26:50.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.556+0000 7f11d8800640 1 -- 172.21.15.96:0/2067999919 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f11b406ea20 msgr2=0x7f11b4070ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:50.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.556+0000 7f11d8800640 1 --2- 172.21.15.96:0/2067999919 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f11b406ea20 0x7f11b4070ee0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f11d0099e90 tx=0x7f11c0008040 comp rx=0 tx=0).stop 2024-05-28T06:26:50.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.556+0000 7f11d8800640 1 -- 172.21.15.96:0/2067999919 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11d006f860 msgr2=0x7f11d015f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:50.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.556+0000 7f11d8800640 1 --2- 172.21.15.96:0/2067999919 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11d006f860 0x7f11d015f570 secure :-1 s=READY pgs=2220 cs=0 l=1 rev1=1 crypto rx=0x7f11c4000c00 tx=0x7f11c40046c0 comp rx=0 tx=0).stop 2024-05-28T06:26:50.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.556+0000 7f11d8800640 1 -- 172.21.15.96:0/2067999919 shutdown_connections 2024-05-28T06:26:50.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.556+0000 7f11d8800640 1 --2- 172.21.15.96:0/2067999919 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f11b406ea20 0x7f11b4070ee0 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:50.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.556+0000 7f11d8800640 1 --2- 172.21.15.96:0/2067999919 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f11d00d09e0 0x7f11d0092020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:50.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.556+0000 7f11d8800640 1 --2- 172.21.15.96:0/2067999919 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f11d00b3690 0x7f11d00ced50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:50.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.556+0000 7f11d8800640 1 --2- 172.21.15.96:0/2067999919 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11d006f860 0x7f11d015f570 unknown :-1 s=CLOSED pgs=2220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:50.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.556+0000 7f11d8800640 1 -- 172.21.15.96:0/2067999919 >> 172.21.15.96:0/2067999919 conn(0x7f11d006bae0 msgr2=0x7f11d00ad130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:26:50.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.556+0000 7f11d8800640 1 -- 172.21.15.96:0/2067999919 shutdown_connections 2024-05-28T06:26:50.552 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.556+0000 7f11d8800640 1 -- 172.21.15.96:0/2067999919 wait complete. 2024-05-28T06:26:50.569 INFO:tasks.ceph:Scrubbing osd.6 2024-05-28T06:26:50.569 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-05-28T06:26:50.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.791+0000 7fc6a2200640 1 Processor -- start 2024-05-28T06:26:50.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.791+0000 7fc6a2200640 1 -- start start 2024-05-28T06:26:50.787 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.792+0000 7fc6a2200640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fc69c15e5a0 0x7fc69c0d2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:50.787 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.792+0000 7fc6a2200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc69c0d3500 con 0x7fc69c0d1420 2024-05-28T06:26:50.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.792+0000 7fc6a2200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc69c0d2200 con 0x7fc69c0cfb40 2024-05-28T06:26:50.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.792+0000 7fc6a2200640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fc69c0d2b30 con 0x7fc69c15e5a0 2024-05-28T06:26:50.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.793+0000 7fc6a0e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fc69c15e5a0 0x7fc69c0d2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:50.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.793+0000 7fc6a1800640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fc69c0d1420 0x7fc69c06c740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:43396/0 (socket says 172.21.15.96:43396) 2024-05-28T06:26:50.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.793+0000 7fc6a1800640 1 -- 172.21.15.96:0/4163578957 learned_addr learned my addr 172.21.15.96:0/4163578957 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:26:50.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.793+0000 7fc6a0e00640 1 -- 172.21.15.96:0/4163578957 >> v1:172.21.15.123:6789/0 conn(0x7fc69c0cfb40 legacy=0x7fc69c0d0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:50.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.793+0000 7fc6a0e00640 1 -- 172.21.15.96:0/4163578957 >> v1:172.21.15.96:6789/0 conn(0x7fc69c0d1420 legacy=0x7fc69c06c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:50.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.793+0000 7fc69b400640 1 -- 172.21.15.96:0/4163578957 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 861201617 0 0) 0x7fc69c0d3500 con 0x7fc69c0d1420 2024-05-28T06:26:50.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.793+0000 7fc6a0e00640 1 -- 172.21.15.96:0/4163578957 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc69c0d24e0 con 0x7fc69c15e5a0 2024-05-28T06:26:50.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.795+0000 7fc6a0e00640 1 --2- 172.21.15.96:0/4163578957 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc69c15e5a0 0x7fc69c0d2fc0 secure :-1 s=READY pgs=2222 cs=0 l=1 rev1=1 crypto rx=0x7fc6900097a0 tx=0x7fc69002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3124f5621e7bc53 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:50.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.795+0000 7fc69b400640 1 -- 172.21.15.96:0/4163578957 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc690037070 con 0x7fc69c15e5a0 2024-05-28T06:26:50.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.795+0000 7fc69b400640 1 -- 172.21.15.96:0/4163578957 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc690032440 con 0x7fc69c15e5a0 2024-05-28T06:26:50.797 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.795+0000 7fc69b400640 1 -- 172.21.15.96:0/4163578957 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc690032810 con 0x7fc69c15e5a0 2024-05-28T06:26:50.797 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.796+0000 7fc6a2200640 1 -- 172.21.15.96:0/4163578957 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc69c15e5a0 msgr2=0x7fc69c0d2fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:50.797 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.796+0000 7fc6a2200640 1 --2- 172.21.15.96:0/4163578957 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc69c15e5a0 0x7fc69c0d2fc0 secure :-1 s=READY pgs=2222 cs=0 l=1 rev1=1 crypto rx=0x7fc6900097a0 tx=0x7fc69002a1e0 comp rx=0 tx=0).stop 2024-05-28T06:26:50.798 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.796+0000 7fc6a2200640 1 -- 172.21.15.96:0/4163578957 shutdown_connections 2024-05-28T06:26:50.798 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.796+0000 7fc6a2200640 1 --2- 172.21.15.96:0/4163578957 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc69c15e5a0 0x7fc69c0d2fc0 unknown :-1 s=CLOSED pgs=2222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:50.798 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.796+0000 7fc6a2200640 1 -- 172.21.15.96:0/4163578957 >> 172.21.15.96:0/4163578957 conn(0x7fc69c08bfc0 msgr2=0x7fc69c0d44b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:26:50.798 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.796+0000 7fc6a2200640 1 -- 172.21.15.96:0/4163578957 shutdown_connections 2024-05-28T06:26:50.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.796+0000 7fc6a2200640 1 -- 172.21.15.96:0/4163578957 wait complete. 2024-05-28T06:26:50.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.798+0000 7fc6a2200640 1 Processor -- start 2024-05-28T06:26:50.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.798+0000 7fc6a2200640 1 -- start start 2024-05-28T06:26:50.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.799+0000 7fc6a2200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc69c0cfb40 0x7fc69c0b3f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:50.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.799+0000 7fc6a2200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc69c0d1420 0x7fc69c14e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:50.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.799+0000 7fc6a2200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc69c15e5a0 0x7fc69c0c95d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:50.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.799+0000 7fc6a2200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc69c15eb00 con 0x7fc69c0d1420 2024-05-28T06:26:50.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.799+0000 7fc6a2200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc69c15e980 con 0x7fc69c15e5a0 2024-05-28T06:26:50.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.799+0000 7fc6a2200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fc69c0d5220 con 0x7fc69c0cfb40 2024-05-28T06:26:50.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.799+0000 7fc69be00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc69c0d1420 0x7fc69c14e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:50.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.800+0000 7fc69be00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc69c0d1420 0x7fc69c14e8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:45978/0 (socket says 172.21.15.96:45978) 2024-05-28T06:26:50.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.800+0000 7fc69be00640 1 -- 172.21.15.96:0/46448845 learned_addr learned my addr 172.21.15.96:0/46448845 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:26:50.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.800+0000 7fc6a0e00640 1 --2- 172.21.15.96:0/46448845 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc69c0cfb40 0x7fc69c0b3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:50.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.800+0000 7fc6a1800640 1 --2- 172.21.15.96:0/46448845 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc69c15e5a0 0x7fc69c0c95d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:50.802 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.800+0000 7fc69be00640 1 -- 172.21.15.96:0/46448845 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc69c0cfb40 msgr2=0x7fc69c0b3f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:50.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.800+0000 7fc69be00640 1 --2- 172.21.15.96:0/46448845 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc69c0cfb40 0x7fc69c0b3f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:50.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.800+0000 7fc69be00640 1 -- 172.21.15.96:0/46448845 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc69c15e5a0 msgr2=0x7fc69c0c95d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:50.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.800+0000 7fc69be00640 1 --2- 172.21.15.96:0/46448845 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc69c15e5a0 0x7fc69c0c95d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:50.803 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.800+0000 7fc69be00640 1 -- 172.21.15.96:0/46448845 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc69c14ee30 con 0x7fc69c0d1420 2024-05-28T06:26:50.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.800+0000 7fc6a1800640 1 --2- 172.21.15.96:0/46448845 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc69c15e5a0 0x7fc69c0c95d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:26:50.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.801+0000 7fc69be00640 1 --2- 172.21.15.96:0/46448845 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc69c0d1420 0x7fc69c14e8f0 secure :-1 s=READY pgs=2223 cs=0 l=1 rev1=1 crypto rx=0x7fc68800c440 tx=0x7fc68800c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:50.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.801+0000 7fc698c00640 1 -- 172.21.15.96:0/46448845 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc688017070 con 0x7fc69c0d1420 2024-05-28T06:26:50.804 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.801+0000 7fc698c00640 1 -- 172.21.15.96:0/46448845 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6880035f0 con 0x7fc69c0d1420 2024-05-28T06:26:50.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.801+0000 7fc6a2200640 1 -- 172.21.15.96:0/46448845 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc69c15b110 con 0x7fc69c0d1420 2024-05-28T06:26:50.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.801+0000 7fc698c00640 1 -- 172.21.15.96:0/46448845 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6880039e0 con 0x7fc69c0d1420 2024-05-28T06:26:50.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.801+0000 7fc6a2200640 1 -- 172.21.15.96:0/46448845 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc69c0cba00 con 0x7fc69c0d1420 2024-05-28T06:26:50.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.802+0000 7fc6a2200640 1 -- 172.21.15.96:0/46448845 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc69c0689e0 con 0x7fc69c0d1420 2024-05-28T06:26:50.808 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.804+0000 7fc698c00640 1 -- 172.21.15.96:0/46448845 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 49) ==== 94049+0+0 (secure 0 0 0) 0x7fc68801a070 con 0x7fc69c0d1420 2024-05-28T06:26:50.808 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.805+0000 7fc698c00640 1 --2- 172.21.15.96:0/46448845 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc67c06ea20 0x7fc67c070ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:50.808 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.805+0000 7fc698c00640 1 -- 172.21.15.96:0/46448845 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 5823+0+0 (secure 0 0 0) 0x7fc68808a080 con 0x7fc69c0d1420 2024-05-28T06:26:50.809 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.805+0000 7fc698c00640 1 --2- 172.21.15.96:0/46448845 >> [v2:172.21.15.123:6808/1217470164,v1:172.21.15.123:6809/1217470164] conn(0x7fc67c076d40 0x7fc67c0791c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:50.809 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.805+0000 7fc698c00640 1 -- 172.21.15.96:0/46448845 --> [v2:172.21.15.123:6808/1217470164,v1:172.21.15.123:6809/1217470164] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc67c079890 con 0x7fc67c076d40 2024-05-28T06:26:50.809 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.805+0000 7fc698c00640 1 -- 172.21.15.96:0/46448845 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_get_version_reply(handle=1 version=132) ==== 24+0+0 (secure 0 0 0) 0x7fc688096050 con 0x7fc69c0d1420 2024-05-28T06:26:50.809 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.805+0000 7fc6a0e00640 1 --2- 172.21.15.96:0/46448845 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc67c06ea20 0x7fc67c070ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:50.810 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.806+0000 7fc6a1800640 1 --2- 172.21.15.96:0/46448845 >> [v2:172.21.15.123:6808/1217470164,v1:172.21.15.123:6809/1217470164] conn(0x7fc67c076d40 0x7fc67c0791c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:50.810 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.807+0000 7fc6a0e00640 1 --2- 172.21.15.96:0/46448845 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc67c06ea20 0x7fc67c070ee0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7fc690034000 tx=0x7fc69002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:50.810 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.807+0000 7fc6a1800640 1 --2- 172.21.15.96:0/46448845 >> [v2:172.21.15.123:6808/1217470164,v1:172.21.15.123:6809/1217470164] conn(0x7fc67c076d40 0x7fc67c0791c0 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:50.811 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.809+0000 7fc698c00640 1 -- 172.21.15.96:0/46448845 <== osd.6 v2:172.21.15.123:6808/1217470164 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc67c079890 con 0x7fc67c076d40 2024-05-28T06:26:50.819 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.826+0000 7fc6a2200640 1 -- 172.21.15.96:0/46448845 --> [v2:172.21.15.123:6808/1217470164,v1:172.21.15.123:6809/1217470164] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fc69c0ce410 con 0x7fc67c076d40 2024-05-28T06:26:50.821 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.827+0000 7fc698c00640 1 -- 172.21.15.96:0/46448845 <== osd.6 v2:172.21.15.123:6808/1217470164 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fc69c0ce410 con 0x7fc67c076d40 2024-05-28T06:26:50.822 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:26:50.822 INFO:teuthology.orchestra.run.smithi096.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-28T06:26:50.822 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:26:50.823 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.828+0000 7fc6a2200640 1 -- 172.21.15.96:0/46448845 >> [v2:172.21.15.123:6808/1217470164,v1:172.21.15.123:6809/1217470164] conn(0x7fc67c076d40 msgr2=0x7fc67c0791c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:50.823 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.828+0000 7fc6a2200640 1 --2- 172.21.15.96:0/46448845 >> [v2:172.21.15.123:6808/1217470164,v1:172.21.15.123:6809/1217470164] conn(0x7fc67c076d40 0x7fc67c0791c0 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:50.823 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.828+0000 7fc6a2200640 1 -- 172.21.15.96:0/46448845 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc67c06ea20 msgr2=0x7fc67c070ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:50.823 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.828+0000 7fc6a2200640 1 --2- 172.21.15.96:0/46448845 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fc67c06ea20 0x7fc67c070ee0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7fc690034000 tx=0x7fc69002e000 comp rx=0 tx=0).stop 2024-05-28T06:26:50.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.828+0000 7fc6a2200640 1 -- 172.21.15.96:0/46448845 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc69c0d1420 msgr2=0x7fc69c14e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:50.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.828+0000 7fc6a2200640 1 --2- 172.21.15.96:0/46448845 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc69c0d1420 0x7fc69c14e8f0 secure :-1 s=READY pgs=2223 cs=0 l=1 rev1=1 crypto rx=0x7fc68800c440 tx=0x7fc68800c900 comp rx=0 tx=0).stop 2024-05-28T06:26:50.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.828+0000 7fc6a1800640 1 -- 172.21.15.96:0/46448845 reap_dead start 2024-05-28T06:26:50.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.828+0000 7fc6a2200640 1 -- 172.21.15.96:0/46448845 shutdown_connections 2024-05-28T06:26:50.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.828+0000 7fc6a2200640 1 -- 172.21.15.96:0/46448845 >> 172.21.15.96:0/46448845 conn(0x7fc69c08bfc0 msgr2=0x7fc69c161530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:26:50.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.828+0000 7fc6a2200640 1 -- 172.21.15.96:0/46448845 shutdown_connections 2024-05-28T06:26:50.824 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:50.828+0000 7fc6a2200640 1 -- 172.21.15.96:0/46448845 wait complete. 2024-05-28T06:26:50.841 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-05-28T06:26:51.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.046+0000 7fa289c00640 1 Processor -- start 2024-05-28T06:26:51.040 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.046+0000 7fa289c00640 1 -- start start 2024-05-28T06:26:51.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.048+0000 7fa289c00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fa28415ea40 0x7fa2840cfab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:51.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.048+0000 7fa289c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa28415ee20 0x7fa28415fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:51.041 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.048+0000 7fa289c00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2840d37b0 con 0x7fa2840d5190 2024-05-28T06:26:51.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.048+0000 7fa289c00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fa2840cecf0 con 0x7fa28415ea40 2024-05-28T06:26:51.042 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.048+0000 7fa289c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fa2840cefd0 con 0x7fa28415ee20 2024-05-28T06:26:51.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.048+0000 7fa283e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fa28415ea40 0x7fa2840cfab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:51.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.048+0000 7fa283e00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fa28415ea40 0x7fa2840cfab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.96:45316/0 (socket says 172.21.15.96:45316) 2024-05-28T06:26:51.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.048+0000 7fa283e00640 1 -- 172.21.15.96:0/3153283326 learned_addr learned my addr 172.21.15.96:0/3153283326 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:26:51.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.048+0000 7fa289200640 1 --2- 172.21.15.96:0/3153283326 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa28415ee20 0x7fa28415fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:51.043 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.049+0000 7fa283e00640 1 -- 172.21.15.96:0/3153283326 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa28415ee20 msgr2=0x7fa28415fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:51.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.049+0000 7fa283e00640 1 --2- 172.21.15.96:0/3153283326 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa28415ee20 0x7fa28415fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:51.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.049+0000 7fa283e00640 1 -- 172.21.15.96:0/3153283326 >> v1:172.21.15.123:6789/0 conn(0x7fa2840d5190 legacy=0x7fa2840d30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:51.044 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.049+0000 7fa283e00640 1 -- 172.21.15.96:0/3153283326 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2840cf2b0 con 0x7fa28415ea40 2024-05-28T06:26:51.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.050+0000 7fa283e00640 1 --2- 172.21.15.96:0/3153283326 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa28415ea40 0x7fa2840cfab0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fa274009870 tx=0x7fa27402a250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7240c7122c997ea4 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:51.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.051+0000 7fa282a00640 1 -- 172.21.15.96:0/3153283326 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa274037070 con 0x7fa28415ea40 2024-05-28T06:26:51.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.051+0000 7fa282a00640 1 -- 172.21.15.96:0/3153283326 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa274032440 con 0x7fa28415ea40 2024-05-28T06:26:51.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.051+0000 7fa282a00640 1 -- 172.21.15.96:0/3153283326 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa274032810 con 0x7fa28415ea40 2024-05-28T06:26:51.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.052+0000 7fa289c00640 1 -- 172.21.15.96:0/3153283326 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa28415ea40 msgr2=0x7fa2840cfab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:51.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.052+0000 7fa289c00640 1 --2- 172.21.15.96:0/3153283326 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa28415ea40 0x7fa2840cfab0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fa274009870 tx=0x7fa27402a250 comp rx=0 tx=0).stop 2024-05-28T06:26:51.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.052+0000 7fa289c00640 1 -- 172.21.15.96:0/3153283326 shutdown_connections 2024-05-28T06:26:51.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.052+0000 7fa289c00640 1 --2- 172.21.15.96:0/3153283326 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa28415ee20 0x7fa28415fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:51.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.052+0000 7fa289c00640 1 --2- 172.21.15.96:0/3153283326 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa28415ea40 0x7fa2840cfab0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:51.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.052+0000 7fa289c00640 1 -- 172.21.15.96:0/3153283326 >> 172.21.15.96:0/3153283326 conn(0x7fa28408bfc0 msgr2=0x7fa2840d4500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:26:51.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.052+0000 7fa289c00640 1 -- 172.21.15.96:0/3153283326 shutdown_connections 2024-05-28T06:26:51.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.052+0000 7fa289c00640 1 -- 172.21.15.96:0/3153283326 wait complete. 2024-05-28T06:26:51.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.054+0000 7fa289c00640 1 Processor -- start 2024-05-28T06:26:51.048 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.054+0000 7fa289c00640 1 -- start start 2024-05-28T06:26:51.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.055+0000 7fa289c00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa2840d5190 0x7fa2840b81f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:51.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.055+0000 7fa289c00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa28415ea40 0x7fa284155330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:51.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.055+0000 7fa289c00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa28415ee20 0x7fa2841ea220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:51.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.055+0000 7fa289c00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa28415f2d0 con 0x7fa2840d5190 2024-05-28T06:26:51.051 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.055+0000 7fa289c00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa2840cfff0 con 0x7fa28415ea40 2024-05-28T06:26:51.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.055+0000 7fa289c00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fa28415f450 con 0x7fa28415ee20 2024-05-28T06:26:51.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.055+0000 7fa283e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa2840d5190 0x7fa2840b81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:51.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.055+0000 7fa283e00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa2840d5190 0x7fa2840b81f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:45990/0 (socket says 172.21.15.96:45990) 2024-05-28T06:26:51.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.055+0000 7fa283e00640 1 -- 172.21.15.96:0/387354873 learned_addr learned my addr 172.21.15.96:0/387354873 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:26:51.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.055+0000 7fa283e00640 1 -- 172.21.15.96:0/387354873 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa28415ee20 msgr2=0x7fa2841ea220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:51.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.055+0000 7fa283e00640 1 --2- 172.21.15.96:0/387354873 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa28415ee20 0x7fa2841ea220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:51.052 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.055+0000 7fa283e00640 1 -- 172.21.15.96:0/387354873 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa28415ea40 msgr2=0x7fa284155330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:51.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.055+0000 7fa283400640 1 --2- 172.21.15.96:0/387354873 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa28415ea40 0x7fa284155330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:51.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.056+0000 7fa283e00640 1 --2- 172.21.15.96:0/387354873 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa28415ea40 0x7fa284155330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:51.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.056+0000 7fa283e00640 1 -- 172.21.15.96:0/387354873 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2840b7150 con 0x7fa2840d5190 2024-05-28T06:26:51.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.056+0000 7fa283400640 1 --2- 172.21.15.96:0/387354873 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa28415ea40 0x7fa284155330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:26:51.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.056+0000 7fa283e00640 1 --2- 172.21.15.96:0/387354873 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa2840d5190 0x7fa2840b81f0 secure :-1 s=READY pgs=2224 cs=0 l=1 rev1=1 crypto rx=0x7fa274000c00 tx=0x7fa2740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:51.053 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.056+0000 7fa27be00640 1 -- 172.21.15.96:0/387354873 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa274037040 con 0x7fa2840d5190 2024-05-28T06:26:51.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.056+0000 7fa27be00640 1 -- 172.21.15.96:0/387354873 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa274042070 con 0x7fa2840d5190 2024-05-28T06:26:51.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.056+0000 7fa27be00640 1 -- 172.21.15.96:0/387354873 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa27403b870 con 0x7fa2840d5190 2024-05-28T06:26:51.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.056+0000 7fa289c00640 1 -- 172.21.15.96:0/387354873 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2840b7430 con 0x7fa2840d5190 2024-05-28T06:26:51.054 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.057+0000 7fa289c00640 1 -- 172.21.15.96:0/387354873 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2840b79f0 con 0x7fa2840d5190 2024-05-28T06:26:51.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.058+0000 7fa289c00640 1 -- 172.21.15.96:0/387354873 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa284068a90 con 0x7fa2840d5190 2024-05-28T06:26:51.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.059+0000 7fa27be00640 1 -- 172.21.15.96:0/387354873 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 49) ==== 94049+0+0 (secure 0 0 0) 0x7fa27403ba10 con 0x7fa2840d5190 2024-05-28T06:26:51.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.060+0000 7fa27be00640 1 --2- 172.21.15.96:0/387354873 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa26406ea20 0x7fa264070ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:51.055 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.060+0000 7fa27be00640 1 -- 172.21.15.96:0/387354873 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 5823+0+0 (secure 0 0 0) 0x7fa2740ab4e0 con 0x7fa2840d5190 2024-05-28T06:26:51.056 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.061+0000 7fa283400640 1 --2- 172.21.15.96:0/387354873 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa26406ea20 0x7fa264070ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:51.057 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.064+0000 7fa283400640 1 --2- 172.21.15.96:0/387354873 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa26406ea20 0x7fa264070ee0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7fa2840cfd90 tx=0x7fa270009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:51.060 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.067+0000 7fa27be00640 1 -- 172.21.15.96:0/387354873 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa284068a90 con 0x7fa2840d5190 2024-05-28T06:26:51.287 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.293+0000 7fa289c00640 1 -- 172.21.15.96:0/387354873 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) -- 0x7fa2840690f0 con 0x7fa26406ea20 2024-05-28T06:26:51.288 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.295+0000 7fa27be00640 1 -- 172.21.15.96:0/387354873 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fa2840690f0 con 0x7fa26406ea20 2024-05-28T06:26:51.288 INFO:teuthology.orchestra.run.smithi096.stderr:instructed osd(s) 6 to deep-scrub 2024-05-28T06:26:51.291 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.297+0000 7fa289c00640 1 -- 172.21.15.96:0/387354873 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa26406ea20 msgr2=0x7fa264070ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:51.291 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.297+0000 7fa289c00640 1 --2- 172.21.15.96:0/387354873 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa26406ea20 0x7fa264070ee0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7fa2840cfd90 tx=0x7fa270009040 comp rx=0 tx=0).stop 2024-05-28T06:26:51.291 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.297+0000 7fa289c00640 1 -- 172.21.15.96:0/387354873 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa2840d5190 msgr2=0x7fa2840b81f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:51.292 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.297+0000 7fa289c00640 1 --2- 172.21.15.96:0/387354873 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa2840d5190 0x7fa2840b81f0 secure :-1 s=READY pgs=2224 cs=0 l=1 rev1=1 crypto rx=0x7fa274000c00 tx=0x7fa2740046c0 comp rx=0 tx=0).stop 2024-05-28T06:26:51.292 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.297+0000 7fa289c00640 1 -- 172.21.15.96:0/387354873 shutdown_connections 2024-05-28T06:26:51.292 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.297+0000 7fa289c00640 1 --2- 172.21.15.96:0/387354873 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa26406ea20 0x7fa264070ee0 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:51.292 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.297+0000 7fa289c00640 1 --2- 172.21.15.96:0/387354873 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa28415ee20 0x7fa2841ea220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:51.292 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.297+0000 7fa289c00640 1 --2- 172.21.15.96:0/387354873 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa28415ea40 0x7fa284155330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:51.292 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.297+0000 7fa289c00640 1 --2- 172.21.15.96:0/387354873 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa2840d5190 0x7fa2840b81f0 unknown :-1 s=CLOSED pgs=2224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:51.292 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.297+0000 7fa289c00640 1 -- 172.21.15.96:0/387354873 >> 172.21.15.96:0/387354873 conn(0x7fa28408bfc0 msgr2=0x7fa284160530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:26:51.292 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.297+0000 7fa289c00640 1 -- 172.21.15.96:0/387354873 shutdown_connections 2024-05-28T06:26:51.293 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.298+0000 7fa289c00640 1 -- 172.21.15.96:0/387354873 wait complete. 2024-05-28T06:26:51.310 INFO:tasks.ceph:Scrubbing osd.7 2024-05-28T06:26:51.310 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-05-28T06:26:51.517 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.523+0000 7fef0a400640 1 Processor -- start 2024-05-28T06:26:51.517 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.524+0000 7fef0a400640 1 -- start start 2024-05-28T06:26:51.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.525+0000 7fef0a400640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fef040cfb40 0x7fef040d0d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:51.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.525+0000 7fef0a400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fef040cffd0 0x7fef040d1250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:51.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.525+0000 7fef0a400640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef040d2200 con 0x7fef0415e5a0 2024-05-28T06:26:51.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.525+0000 7fef0a400640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fef040d2b30 con 0x7fef040cfb40 2024-05-28T06:26:51.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.525+0000 7fef0a400640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fef040d24e0 con 0x7fef040cffd0 2024-05-28T06:26:51.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.525+0000 7fef09000640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fef0415e5a0 0x7fef040d2fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:43410/0 (socket says 172.21.15.96:43410) 2024-05-28T06:26:51.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.525+0000 7fef09000640 1 -- 172.21.15.96:0/1615667348 learned_addr learned my addr 172.21.15.96:0/1615667348 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:26:51.519 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.525+0000 7fef03e00640 1 --2- 172.21.15.96:0/1615667348 >> v2:172.21.15.96:3300/0 conn(0x7fef040cfb40 0x7fef040d0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:51.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.526+0000 7fef09a00640 1 --2- 172.21.15.96:0/1615667348 >> v2:172.21.15.123:3300/0 conn(0x7fef040cffd0 0x7fef040d1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:51.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.526+0000 7fef03400640 1 -- 172.21.15.96:0/1615667348 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 855171917 0 0) 0x7fef040d2200 con 0x7fef0415e5a0 2024-05-28T06:26:51.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.526+0000 7fef03400640 1 -- 172.21.15.96:0/1615667348 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feee8003520 con 0x7fef0415e5a0 2024-05-28T06:26:51.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.526+0000 7fef03e00640 1 -- 172.21.15.96:0/1615667348 >> v2:172.21.15.123:3300/0 conn(0x7fef040cffd0 msgr2=0x7fef040d1250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:51.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.526+0000 7fef03e00640 1 --2- 172.21.15.96:0/1615667348 >> v2:172.21.15.123:3300/0 conn(0x7fef040cffd0 0x7fef040d1250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:51.521 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.526+0000 7fef03e00640 1 -- 172.21.15.96:0/1615667348 >> v1:172.21.15.96:6789/0 conn(0x7fef0415e5a0 legacy=0x7fef040d2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:51.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.526+0000 7fef03e00640 1 -- 172.21.15.96:0/1615667348 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef040d27c0 con 0x7fef040cfb40 2024-05-28T06:26:51.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.527+0000 7fef03e00640 1 --2- 172.21.15.96:0/1615667348 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef040cfb40 0x7fef040d0d10 secure :-1 s=READY pgs=2226 cs=0 l=1 rev1=1 crypto rx=0x7fef0408e740 tx=0x7feef402a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb047cf3db40ce26 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:51.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.527+0000 7fef03400640 1 -- 172.21.15.96:0/1615667348 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feef4007740 con 0x7fef040cfb40 2024-05-28T06:26:51.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.527+0000 7fef03400640 1 -- 172.21.15.96:0/1615667348 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feef40078e0 con 0x7fef040cfb40 2024-05-28T06:26:51.522 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.527+0000 7fef03400640 1 -- 172.21.15.96:0/1615667348 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feef4007c90 con 0x7fef040cfb40 2024-05-28T06:26:51.523 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.528+0000 7fef0a400640 1 -- 172.21.15.96:0/1615667348 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef040cfb40 msgr2=0x7fef040d0d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:51.523 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.528+0000 7fef0a400640 1 --2- 172.21.15.96:0/1615667348 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef040cfb40 0x7fef040d0d10 secure :-1 s=READY pgs=2226 cs=0 l=1 rev1=1 crypto rx=0x7fef0408e740 tx=0x7feef402a680 comp rx=0 tx=0).stop 2024-05-28T06:26:51.523 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.528+0000 7fef0a400640 1 -- 172.21.15.96:0/1615667348 shutdown_connections 2024-05-28T06:26:51.524 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.528+0000 7fef0a400640 1 --2- 172.21.15.96:0/1615667348 >> v2:172.21.15.123:3300/0 conn(0x7fef040cffd0 0x7fef040d1250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:51.524 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.528+0000 7fef0a400640 1 --2- 172.21.15.96:0/1615667348 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef040cfb40 0x7fef040d0d10 unknown :-1 s=CLOSED pgs=2226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:51.524 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.528+0000 7fef0a400640 1 -- 172.21.15.96:0/1615667348 >> 172.21.15.96:0/1615667348 conn(0x7fef0408bfc0 msgr2=0x7fef040d44b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:26:51.524 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.528+0000 7fef0a400640 1 -- 172.21.15.96:0/1615667348 shutdown_connections 2024-05-28T06:26:51.524 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.528+0000 7fef0a400640 1 -- 172.21.15.96:0/1615667348 wait complete. 2024-05-28T06:26:51.524 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.530+0000 7fef0a400640 1 Processor -- start 2024-05-28T06:26:51.525 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.530+0000 7fef0a400640 1 -- start start 2024-05-28T06:26:51.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.531+0000 7fef0a400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef040cfb40 0x7fef0414b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:51.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.531+0000 7fef0a400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef040cffd0 0x7fef040cd810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:51.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.531+0000 7fef0a400640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fef0415e5a0 0x7fef040b9780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:51.527 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.531+0000 7fef0a400640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fef040d53a0 con 0x7fef040cfb40 2024-05-28T06:26:51.528 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.531+0000 7fef0a400640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fef040d5220 con 0x7fef040cffd0 2024-05-28T06:26:51.528 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.531+0000 7fef0a400640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fef0406c740 con 0x7fef0415e5a0 2024-05-28T06:26:51.528 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.531+0000 7fef09000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef040cfb40 0x7fef0414b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:51.528 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.531+0000 7fef09000640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef040cfb40 0x7fef0414b240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:46014/0 (socket says 172.21.15.96:46014) 2024-05-28T06:26:51.528 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.531+0000 7fef09000640 1 -- 172.21.15.96:0/3150488140 learned_addr learned my addr 172.21.15.96:0/3150488140 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:26:51.528 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.532+0000 7fef03e00640 1 --2- 172.21.15.96:0/3150488140 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef040cffd0 0x7fef040cd810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:51.528 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.532+0000 7fef09000640 1 -- 172.21.15.96:0/3150488140 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fef0415e5a0 msgr2=0x7fef040b9780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:51.528 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.532+0000 7fef09000640 1 --2- 172.21.15.96:0/3150488140 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fef0415e5a0 0x7fef040b9780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:51.529 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.532+0000 7fef09000640 1 -- 172.21.15.96:0/3150488140 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef040cffd0 msgr2=0x7fef040cd810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:51.529 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.532+0000 7fef09000640 1 --2- 172.21.15.96:0/3150488140 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef040cffd0 0x7fef040cd810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:51.529 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.532+0000 7fef09000640 1 -- 172.21.15.96:0/3150488140 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef040cc770 con 0x7fef040cfb40 2024-05-28T06:26:51.529 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.532+0000 7fef09000640 1 --2- 172.21.15.96:0/3150488140 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef040cfb40 0x7fef0414b240 secure :-1 s=READY pgs=2227 cs=0 l=1 rev1=1 crypto rx=0x7feeec00c3c0 tx=0x7feeec00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:51.529 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.532+0000 7fef00c00640 1 -- 172.21.15.96:0/3150488140 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeec017070 con 0x7fef040cfb40 2024-05-28T06:26:51.529 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.532+0000 7fef00c00640 1 -- 172.21.15.96:0/3150488140 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feeec005ce0 con 0x7fef040cfb40 2024-05-28T06:26:51.529 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.532+0000 7fef00c00640 1 -- 172.21.15.96:0/3150488140 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeec0072f0 con 0x7fef040cfb40 2024-05-28T06:26:51.529 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.532+0000 7fef0a400640 1 -- 172.21.15.96:0/3150488140 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef040cca50 con 0x7fef040cfb40 2024-05-28T06:26:51.530 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.532+0000 7fef0a400640 1 -- 172.21.15.96:0/3150488140 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef040cd010 con 0x7fef040cfb40 2024-05-28T06:26:51.530 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.533+0000 7fef0a400640 1 -- 172.21.15.96:0/3150488140 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fef040689e0 con 0x7fef040cfb40 2024-05-28T06:26:51.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.535+0000 7fef00c00640 1 -- 172.21.15.96:0/3150488140 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 49) ==== 94049+0+0 (secure 0 0 0) 0x7feeec01a070 con 0x7fef040cfb40 2024-05-28T06:26:51.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.536+0000 7fef00c00640 1 --2- 172.21.15.96:0/3150488140 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7feee006ea20 0x7feee0070ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:51.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.536+0000 7fef00c00640 1 -- 172.21.15.96:0/3150488140 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 5823+0+0 (secure 0 0 0) 0x7feeec08a040 con 0x7fef040cfb40 2024-05-28T06:26:51.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.536+0000 7fef00c00640 1 --2- 172.21.15.96:0/3150488140 >> [v2:172.21.15.123:6816/2085300276,v1:172.21.15.123:6818/2085300276] conn(0x7feee0076d40 0x7feee00791c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:51.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.536+0000 7fef00c00640 1 -- 172.21.15.96:0/3150488140 --> [v2:172.21.15.123:6816/2085300276,v1:172.21.15.123:6818/2085300276] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7feeec01fe50 con 0x7feee0076d40 2024-05-28T06:26:51.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.536+0000 7fef00c00640 1 -- 172.21.15.96:0/3150488140 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_get_version_reply(handle=1 version=132) ==== 24+0+0 (secure 0 0 0) 0x7feeec096050 con 0x7fef040cfb40 2024-05-28T06:26:51.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.536+0000 7fef03e00640 1 --2- 172.21.15.96:0/3150488140 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7feee006ea20 0x7feee0070ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:51.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.537+0000 7fef09a00640 1 --2- 172.21.15.96:0/3150488140 >> [v2:172.21.15.123:6816/2085300276,v1:172.21.15.123:6818/2085300276] conn(0x7feee0076d40 0x7feee00791c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:51.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.538+0000 7fef03e00640 1 --2- 172.21.15.96:0/3150488140 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7feee006ea20 0x7feee0070ee0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7feef4034000 tx=0x7feef402f000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:51.532 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.538+0000 7fef09a00640 1 --2- 172.21.15.96:0/3150488140 >> [v2:172.21.15.123:6816/2085300276,v1:172.21.15.123:6818/2085300276] conn(0x7feee0076d40 0x7feee00791c0 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:51.533 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.540+0000 7fef00c00640 1 -- 172.21.15.96:0/3150488140 <== osd.7 v2:172.21.15.123:6816/2085300276 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7feeec01fe50 con 0x7feee0076d40 2024-05-28T06:26:51.547 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.554+0000 7fef0a400640 1 -- 172.21.15.96:0/3150488140 --> [v2:172.21.15.123:6816/2085300276,v1:172.21.15.123:6818/2085300276] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fef040c7720 con 0x7feee0076d40 2024-05-28T06:26:51.548 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.555+0000 7fef00c00640 1 -- 172.21.15.96:0/3150488140 <== osd.7 v2:172.21.15.123:6816/2085300276 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fef040c7720 con 0x7feee0076d40 2024-05-28T06:26:51.549 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-05-28T06:26:51.549 INFO:teuthology.orchestra.run.smithi096.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-28T06:26:51.549 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-05-28T06:26:51.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.556+0000 7fef0a400640 1 -- 172.21.15.96:0/3150488140 >> [v2:172.21.15.123:6816/2085300276,v1:172.21.15.123:6818/2085300276] conn(0x7feee0076d40 msgr2=0x7feee00791c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:51.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.556+0000 7fef0a400640 1 --2- 172.21.15.96:0/3150488140 >> [v2:172.21.15.123:6816/2085300276,v1:172.21.15.123:6818/2085300276] conn(0x7feee0076d40 0x7feee00791c0 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:51.550 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.556+0000 7fef0a400640 1 -- 172.21.15.96:0/3150488140 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7feee006ea20 msgr2=0x7feee0070ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:51.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.556+0000 7fef0a400640 1 --2- 172.21.15.96:0/3150488140 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7feee006ea20 0x7feee0070ee0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7feef4034000 tx=0x7feef402f000 comp rx=0 tx=0).stop 2024-05-28T06:26:51.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.556+0000 7fef0a400640 1 -- 172.21.15.96:0/3150488140 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef040cfb40 msgr2=0x7fef0414b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:51.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.556+0000 7fef0a400640 1 --2- 172.21.15.96:0/3150488140 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef040cfb40 0x7fef0414b240 secure :-1 s=READY pgs=2227 cs=0 l=1 rev1=1 crypto rx=0x7feeec00c3c0 tx=0x7feeec00c880 comp rx=0 tx=0).stop 2024-05-28T06:26:51.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.556+0000 7fef09a00640 1 -- 172.21.15.96:0/3150488140 reap_dead start 2024-05-28T06:26:51.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.556+0000 7fef0a400640 1 -- 172.21.15.96:0/3150488140 shutdown_connections 2024-05-28T06:26:51.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.556+0000 7fef0a400640 1 -- 172.21.15.96:0/3150488140 >> 172.21.15.96:0/3150488140 conn(0x7fef0408bfc0 msgr2=0x7fef040b1cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:26:51.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.556+0000 7fef0a400640 1 -- 172.21.15.96:0/3150488140 shutdown_connections 2024-05-28T06:26:51.551 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.556+0000 7fef0a400640 1 -- 172.21.15.96:0/3150488140 wait complete. 2024-05-28T06:26:51.568 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-05-28T06:26:51.783 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.790+0000 7f7a37200640 1 Processor -- start 2024-05-28T06:26:51.784 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.790+0000 7f7a37200640 1 -- start start 2024-05-28T06:26:51.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.791+0000 7f7a37200640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f7a3815e110 0x7f7a380b4de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:51.785 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.791+0000 7f7a37200640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a3815e4f0 con 0x7f7a3806ebc0 2024-05-28T06:26:51.786 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.792+0000 7f7a37200640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a380b5320 con 0x7f7a380b32c0 2024-05-28T06:26:51.786 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.792+0000 7f7a37200640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f7a380b3d40 con 0x7f7a3815e110 2024-05-28T06:26:51.786 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.792+0000 7f7a36800640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f7a3806ebc0 0x7f7a380ad5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:43424/0 (socket says 172.21.15.96:43424) 2024-05-28T06:26:51.786 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.792+0000 7f7a36800640 1 -- 172.21.15.96:0/1037949824 learned_addr learned my addr 172.21.15.96:0/1037949824 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:26:51.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.792+0000 7f7a35e00640 1 --2- 172.21.15.96:0/1037949824 >> v2:172.21.15.123:3300/0 conn(0x7f7a3815e110 0x7f7a380b4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:51.788 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.792+0000 7f7a34a00640 1 -- 172.21.15.96:0/1037949824 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 344389906 0 0) 0x7f7a3815e4f0 con 0x7f7a3806ebc0 2024-05-28T06:26:51.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.793+0000 7f7a34a00640 1 -- 172.21.15.96:0/1037949824 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7a1c005730 con 0x7f7a3806ebc0 2024-05-28T06:26:51.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.793+0000 7f7a34a00640 1 -- 172.21.15.96:0/1037949824 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 431991126 0 0) 0x7f7a1c005730 con 0x7f7a3806ebc0 2024-05-28T06:26:51.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.793+0000 7f7a34a00640 1 -- 172.21.15.96:0/1037949824 >> v2:172.21.15.123:3300/0 conn(0x7f7a3815e110 msgr2=0x7f7a380b4de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:51.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.793+0000 7f7a34a00640 1 --2- 172.21.15.96:0/1037949824 >> v2:172.21.15.123:3300/0 conn(0x7f7a3815e110 0x7f7a380b4de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:51.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.793+0000 7f7a34a00640 1 -- 172.21.15.96:0/1037949824 >> v1:172.21.15.123:6789/0 conn(0x7f7a380b32c0 legacy=0x7f7a3806e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:51.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.793+0000 7f7a34a00640 1 -- 172.21.15.96:0/1037949824 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a380b4020 con 0x7f7a3806ebc0 2024-05-28T06:26:51.789 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.793+0000 7f7a34a00640 1 -- 172.21.15.96:0/1037949824 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f7a300019f0 con 0x7f7a3806ebc0 2024-05-28T06:26:51.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.793+0000 7f7a34a00640 1 -- 172.21.15.96:0/1037949824 >> v1:172.21.15.96:6789/0 conn(0x7f7a3806ebc0 legacy=0x7f7a380ad5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:51.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.793+0000 7f7a34a00640 1 --2- 172.21.15.96:0/1037949824 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7a1c005c10 0x7f7a1c006030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:51.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.793+0000 7f7a34a00640 1 --2- 172.21.15.96:0/1037949824 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a1c006a10 0x7f7a1c006e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:51.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.793+0000 7f7a34a00640 1 --2- 172.21.15.96:0/1037949824 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7a1c0073d0 0x7f7a1c00dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:51.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.793+0000 7f7a34a00640 1 -- 172.21.15.96:0/1037949824 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f7a380b3d40 con 0x7f7a1c005c10 2024-05-28T06:26:51.790 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.793+0000 7f7a34a00640 1 -- 172.21.15.96:0/1037949824 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f7a1c00e2d0 con 0x7f7a1c006a10 2024-05-28T06:26:51.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.793+0000 7f7a34a00640 1 -- 172.21.15.96:0/1037949824 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f7a1c00e4d0 con 0x7f7a1c0073d0 2024-05-28T06:26:51.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.793+0000 7f7a35e00640 1 --2- 172.21.15.96:0/1037949824 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7a1c005c10 0x7f7a1c006030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:51.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.794+0000 7f7a35400640 1 --2- 172.21.15.96:0/1037949824 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a1c006a10 0x7f7a1c006e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:51.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.794+0000 7f7a35e00640 1 -- 172.21.15.96:0/1037949824 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7a1c0073d0 msgr2=0x7f7a1c00dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:51.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.794+0000 7f7a35e00640 1 --2- 172.21.15.96:0/1037949824 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7a1c0073d0 0x7f7a1c00dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:51.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.794+0000 7f7a35e00640 1 -- 172.21.15.96:0/1037949824 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a1c006a10 msgr2=0x7f7a1c006e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:51.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.794+0000 7f7a35e00640 1 --2- 172.21.15.96:0/1037949824 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a1c006a10 0x7f7a1c006e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:51.791 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.794+0000 7f7a35e00640 1 -- 172.21.15.96:0/1037949824 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a1c005730 con 0x7f7a1c005c10 2024-05-28T06:26:51.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.794+0000 7f7a36800640 1 -- 172.21.15.96:0/1037949824 reap_dead start 2024-05-28T06:26:51.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.795+0000 7f7a35e00640 1 --2- 172.21.15.96:0/1037949824 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7a1c005c10 0x7f7a1c006030 secure :-1 s=READY pgs=2229 cs=0 l=1 rev1=1 crypto rx=0x7f7a28009ac0 tx=0x7f7a1c007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a37c964202f0119 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:51.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.795+0000 7f7a34a00640 1 -- 172.21.15.96:0/1037949824 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a28036070 con 0x7f7a1c005c10 2024-05-28T06:26:51.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.795+0000 7f7a34a00640 1 -- 172.21.15.96:0/1037949824 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a28031400 con 0x7f7a1c005c10 2024-05-28T06:26:51.792 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.795+0000 7f7a34a00640 1 -- 172.21.15.96:0/1037949824 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a28009690 con 0x7f7a1c005c10 2024-05-28T06:26:51.794 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.796+0000 7f7a37200640 1 -- 172.21.15.96:0/1037949824 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7a1c005c10 msgr2=0x7f7a1c006030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:51.794 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.796+0000 7f7a37200640 1 --2- 172.21.15.96:0/1037949824 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7a1c005c10 0x7f7a1c006030 secure :-1 s=READY pgs=2229 cs=0 l=1 rev1=1 crypto rx=0x7f7a28009ac0 tx=0x7f7a1c007a40 comp rx=0 tx=0).stop 2024-05-28T06:26:51.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.796+0000 7f7a37200640 1 -- 172.21.15.96:0/1037949824 shutdown_connections 2024-05-28T06:26:51.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.796+0000 7f7a37200640 1 --2- 172.21.15.96:0/1037949824 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7a1c005c10 0x7f7a1c006030 unknown :-1 s=CLOSED pgs=2229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:51.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.796+0000 7f7a37200640 1 -- 172.21.15.96:0/1037949824 >> 172.21.15.96:0/1037949824 conn(0x7f7a380b1fc0 msgr2=0x7f7a380b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:26:51.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.796+0000 7f7a37200640 1 -- 172.21.15.96:0/1037949824 shutdown_connections 2024-05-28T06:26:51.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.796+0000 7f7a37200640 1 -- 172.21.15.96:0/1037949824 wait complete. 2024-05-28T06:26:51.795 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.798+0000 7f7a37200640 1 Processor -- start 2024-05-28T06:26:51.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.798+0000 7f7a37200640 1 -- start start 2024-05-28T06:26:51.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.799+0000 7f7a37200640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7a1c005c10 0x7f7a381515c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:51.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.799+0000 7f7a37200640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7a3814e8c0 0x7f7a3809d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:51.796 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.799+0000 7f7a37200640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a3809a3b0 0x7f7a3809a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:51.797 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.799+0000 7f7a37200640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f7a380b48c0 con 0x7f7a3814e8c0 2024-05-28T06:26:51.797 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.799+0000 7f7a37200640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f7a380b45e0 con 0x7f7a3809a3b0 2024-05-28T06:26:51.797 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.799+0000 7f7a37200640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f7a3806f4d0 con 0x7f7a1c005c10 2024-05-28T06:26:51.797 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.799+0000 7f7a35400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7a3814e8c0 0x7f7a3809d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:51.798 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.799+0000 7f7a35400640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7a3814e8c0 0x7f7a3809d2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:46032/0 (socket says 172.21.15.96:46032) 2024-05-28T06:26:51.798 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.799+0000 7f7a35400640 1 -- 172.21.15.96:0/1448665385 learned_addr learned my addr 172.21.15.96:0/1448665385 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:26:51.798 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.799+0000 7f7a35e00640 1 --2- 172.21.15.96:0/1448665385 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7a1c005c10 0x7f7a381515c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:51.798 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.799+0000 7f7a36800640 1 --2- 172.21.15.96:0/1448665385 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a3809a3b0 0x7f7a3809a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:51.798 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.800+0000 7f7a35400640 1 -- 172.21.15.96:0/1448665385 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7a1c005c10 msgr2=0x7f7a381515c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:51.798 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.800+0000 7f7a35400640 1 --2- 172.21.15.96:0/1448665385 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7a1c005c10 0x7f7a381515c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:51.798 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.800+0000 7f7a35400640 1 -- 172.21.15.96:0/1448665385 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a3809a3b0 msgr2=0x7f7a3809a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:51.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.800+0000 7f7a35400640 1 --2- 172.21.15.96:0/1448665385 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a3809a3b0 0x7f7a3809a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:51.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.800+0000 7f7a35400640 1 -- 172.21.15.96:0/1448665385 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a38150520 con 0x7f7a3814e8c0 2024-05-28T06:26:51.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.800+0000 7f7a36800640 1 --2- 172.21.15.96:0/1448665385 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a3809a3b0 0x7f7a3809a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:26:51.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.800+0000 7f7a35400640 1 --2- 172.21.15.96:0/1448665385 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7a3814e8c0 0x7f7a3809d2e0 secure :-1 s=READY pgs=2230 cs=0 l=1 rev1=1 crypto rx=0x7f7a2400e580 tx=0x7f7a2400ea40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:51.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.800+0000 7f7a2e000640 1 -- 172.21.15.96:0/1448665385 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a240052e0 con 0x7f7a3814e8c0 2024-05-28T06:26:51.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.800+0000 7f7a2e000640 1 -- 172.21.15.96:0/1448665385 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a24018070 con 0x7f7a3814e8c0 2024-05-28T06:26:51.799 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.800+0000 7f7a2e000640 1 -- 172.21.15.96:0/1448665385 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a24003810 con 0x7f7a3814e8c0 2024-05-28T06:26:51.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.800+0000 7f7a37200640 1 -- 172.21.15.96:0/1448665385 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a38150800 con 0x7f7a3814e8c0 2024-05-28T06:26:51.800 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.800+0000 7f7a37200640 1 -- 172.21.15.96:0/1448665385 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a38150dc0 con 0x7f7a3814e8c0 2024-05-28T06:26:51.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.802+0000 7f7a37200640 1 -- 172.21.15.96:0/1448665385 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a38151b00 con 0x7f7a3814e8c0 2024-05-28T06:26:51.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.803+0000 7f7a2e000640 1 -- 172.21.15.96:0/1448665385 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 49) ==== 94049+0+0 (secure 0 0 0) 0x7f7a240039b0 con 0x7f7a3814e8c0 2024-05-28T06:26:51.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.804+0000 7f7a2e000640 1 --2- 172.21.15.96:0/1448665385 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7a1806ea50 0x7f7a18070f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:51.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.804+0000 7f7a2e000640 1 -- 172.21.15.96:0/1448665385 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 5823+0+0 (secure 0 0 0) 0x7f7a2408aa20 con 0x7f7a3814e8c0 2024-05-28T06:26:51.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.805+0000 7f7a35e00640 1 --2- 172.21.15.96:0/1448665385 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7a1806ea50 0x7f7a18070f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:51.801 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.807+0000 7f7a35e00640 1 --2- 172.21.15.96:0/1448665385 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7a1806ea50 0x7f7a18070f10 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f7a28033000 tx=0x7f7a28008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:51.805 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:51.812+0000 7f7a2e000640 1 -- 172.21.15.96:0/1448665385 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7a2405a8b0 con 0x7f7a3814e8c0 2024-05-28T06:26:52.070 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.076+0000 7f7a37200640 1 -- 172.21.15.96:0/1448665385 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) -- 0x7f7a380689e0 con 0x7f7a1806ea50 2024-05-28T06:26:52.071 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.078+0000 7f7a2e000640 1 -- 172.21.15.96:0/1448665385 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f7a380689e0 con 0x7f7a1806ea50 2024-05-28T06:26:52.071 INFO:teuthology.orchestra.run.smithi096.stderr:instructed osd(s) 7 to deep-scrub 2024-05-28T06:26:52.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.080+0000 7f7a37200640 1 -- 172.21.15.96:0/1448665385 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7a1806ea50 msgr2=0x7f7a18070f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:52.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.080+0000 7f7a37200640 1 --2- 172.21.15.96:0/1448665385 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7a1806ea50 0x7f7a18070f10 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f7a28033000 tx=0x7f7a28008030 comp rx=0 tx=0).stop 2024-05-28T06:26:52.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.080+0000 7f7a37200640 1 -- 172.21.15.96:0/1448665385 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7a3814e8c0 msgr2=0x7f7a3809d2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:52.074 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.080+0000 7f7a37200640 1 --2- 172.21.15.96:0/1448665385 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7a3814e8c0 0x7f7a3809d2e0 secure :-1 s=READY pgs=2230 cs=0 l=1 rev1=1 crypto rx=0x7f7a2400e580 tx=0x7f7a2400ea40 comp rx=0 tx=0).stop 2024-05-28T06:26:52.075 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.081+0000 7f7a37200640 1 -- 172.21.15.96:0/1448665385 shutdown_connections 2024-05-28T06:26:52.075 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.081+0000 7f7a37200640 1 --2- 172.21.15.96:0/1448665385 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f7a1806ea50 0x7f7a18070f10 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:52.075 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.081+0000 7f7a37200640 1 --2- 172.21.15.96:0/1448665385 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a3809a3b0 0x7f7a3809a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:52.075 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.081+0000 7f7a37200640 1 --2- 172.21.15.96:0/1448665385 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7a3814e8c0 0x7f7a3809d2e0 unknown :-1 s=CLOSED pgs=2230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:52.075 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.081+0000 7f7a37200640 1 --2- 172.21.15.96:0/1448665385 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7a1c005c10 0x7f7a381515c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:52.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.081+0000 7f7a37200640 1 -- 172.21.15.96:0/1448665385 >> 172.21.15.96:0/1448665385 conn(0x7f7a380b1fc0 msgr2=0x7f7a3815ecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:26:52.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.081+0000 7f7a37200640 1 -- 172.21.15.96:0/1448665385 shutdown_connections 2024-05-28T06:26:52.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.081+0000 7f7a37200640 1 -- 172.21.15.96:0/1448665385 wait complete. 2024-05-28T06:26:52.097 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-28T06:26:52.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.363+0000 7fa19d600640 1 Processor -- start 2024-05-28T06:26:52.357 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.364+0000 7fa19d600640 1 -- start start 2024-05-28T06:26:52.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.365+0000 7fa19d600640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fa198160380 0x7fa1980b5e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:52.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.365+0000 7fa19d600640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fa1980b5430 0x7fa1980b63a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:52.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.365+0000 7fa19d600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1980d3300 con 0x7fa19815fae0 2024-05-28T06:26:52.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.365+0000 7fa19d600640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fa1980ced60 con 0x7fa1980b5430 2024-05-28T06:26:52.359 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.365+0000 7fa19d600640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7fa1980d35e0 con 0x7fa198160380 2024-05-28T06:26:52.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.366+0000 7fa197400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fa198160380 0x7fa1980b5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:52.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.366+0000 7fa19cc00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fa1980b5430 0x7fa1980b63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:52.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.366+0000 7fa197400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fa198160380 0x7fa1980b5e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.96:45400/0 (socket says 172.21.15.96:45400) 2024-05-28T06:26:52.360 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.366+0000 7fa197400640 1 -- 172.21.15.96:0/1047541780 learned_addr learned my addr 172.21.15.96:0/1047541780 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:26:52.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.366+0000 7fa19cc00640 1 -- 172.21.15.96:0/1047541780 >> v2:172.21.15.123:3300/0 conn(0x7fa198160380 msgr2=0x7fa1980b5e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:26:52.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.366+0000 7fa19cc00640 1 --2- 172.21.15.96:0/1047541780 >> v2:172.21.15.123:3300/0 conn(0x7fa198160380 0x7fa1980b5e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:52.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.366+0000 7fa19cc00640 1 -- 172.21.15.96:0/1047541780 >> v1:172.21.15.123:6789/0 conn(0x7fa19815fae0 legacy=0x7fa1980d4e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:52.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.366+0000 7fa19cc00640 1 -- 172.21.15.96:0/1047541780 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1980d38c0 con 0x7fa1980b5430 2024-05-28T06:26:52.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.366+0000 7fa197400640 1 --2- 172.21.15.96:0/1047541780 >> v2:172.21.15.123:3300/0 conn(0x7fa198160380 0x7fa1980b5e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-28T06:26:52.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.368+0000 7fa19cc00640 1 --2- 172.21.15.96:0/1047541780 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa1980b5430 0x7fa1980b63a0 secure :-1 s=READY pgs=2231 cs=0 l=1 rev1=1 crypto rx=0x7fa1980ceee0 tx=0x7fa19002a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d07cca471f9833de server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:52.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.368+0000 7fa196a00640 1 -- 172.21.15.96:0/1047541780 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa190007740 con 0x7fa1980b5430 2024-05-28T06:26:52.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.368+0000 7fa196a00640 1 -- 172.21.15.96:0/1047541780 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1900078e0 con 0x7fa1980b5430 2024-05-28T06:26:52.362 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.368+0000 7fa196a00640 1 -- 172.21.15.96:0/1047541780 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa190007cb0 con 0x7fa1980b5430 2024-05-28T06:26:52.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.369+0000 7fa19d600640 1 -- 172.21.15.96:0/1047541780 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa1980b5430 msgr2=0x7fa1980b63a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:52.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.369+0000 7fa19d600640 1 --2- 172.21.15.96:0/1047541780 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa1980b5430 0x7fa1980b63a0 secure :-1 s=READY pgs=2231 cs=0 l=1 rev1=1 crypto rx=0x7fa1980ceee0 tx=0x7fa19002a660 comp rx=0 tx=0).stop 2024-05-28T06:26:52.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.369+0000 7fa19d600640 1 -- 172.21.15.96:0/1047541780 shutdown_connections 2024-05-28T06:26:52.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.369+0000 7fa19d600640 1 --2- 172.21.15.96:0/1047541780 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa1980b5430 0x7fa1980b63a0 unknown :-1 s=CLOSED pgs=2231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:52.363 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.369+0000 7fa19d600640 1 --2- 172.21.15.96:0/1047541780 >> v2:172.21.15.123:3300/0 conn(0x7fa198160380 0x7fa1980b5e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:52.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.369+0000 7fa19d600640 1 -- 172.21.15.96:0/1047541780 >> 172.21.15.96:0/1047541780 conn(0x7fa19808bfc0 msgr2=0x7fa1980d4500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:26:52.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.369+0000 7fa19d600640 1 -- 172.21.15.96:0/1047541780 shutdown_connections 2024-05-28T06:26:52.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.369+0000 7fa19d600640 1 -- 172.21.15.96:0/1047541780 wait complete. 2024-05-28T06:26:52.364 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.371+0000 7fa19d600640 1 Processor -- start 2024-05-28T06:26:52.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.371+0000 7fa19d600640 1 -- start start 2024-05-28T06:26:52.365 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.372+0000 7fa19d600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa1980b5430 0x7fa198158b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:52.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.372+0000 7fa19d600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa19815fae0 0x7fa1980c8000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:52.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.372+0000 7fa19d600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa198160380 0x7fa1980c8540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:52.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.372+0000 7fa19d600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa1980cf370 con 0x7fa198160380 2024-05-28T06:26:52.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.372+0000 7fa19d600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa1980cf1f0 con 0x7fa19815fae0 2024-05-28T06:26:52.366 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.372+0000 7fa19d600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7fa1980cf4f0 con 0x7fa1980b5430 2024-05-28T06:26:52.368 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.373+0000 7fa197e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa1980b5430 0x7fa198158b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:52.368 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.373+0000 7fa197400640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa19815fae0 0x7fa1980c8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:52.368 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.373+0000 7fa19cc00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa198160380 0x7fa1980c8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:52.368 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.373+0000 7fa19cc00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa198160380 0x7fa1980c8540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:46058/0 (socket says 172.21.15.96:46058) 2024-05-28T06:26:52.368 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.373+0000 7fa19cc00640 1 -- 172.21.15.96:0/3031456652 learned_addr learned my addr 172.21.15.96:0/3031456652 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:26:52.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.373+0000 7fa197e00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa1980b5430 0x7fa198158b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.96:40214/0 (socket says 172.21.15.96:40214) 2024-05-28T06:26:52.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.373+0000 7fa197400640 1 -- 172.21.15.96:0/3031456652 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa1980b5430 msgr2=0x7fa198158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:52.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.373+0000 7fa197400640 1 --2- 172.21.15.96:0/3031456652 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa1980b5430 0x7fa198158b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:52.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.373+0000 7fa197400640 1 -- 172.21.15.96:0/3031456652 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa198160380 msgr2=0x7fa1980c8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:52.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.373+0000 7fa197400640 1 --2- 172.21.15.96:0/3031456652 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa198160380 0x7fa1980c8540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:52.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.373+0000 7fa197400640 1 -- 172.21.15.96:0/3031456652 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa198159400 con 0x7fa19815fae0 2024-05-28T06:26:52.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.374+0000 7fa197400640 1 --2- 172.21.15.96:0/3031456652 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa19815fae0 0x7fa1980c8000 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fa18400e900 tx=0x7fa18400edc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:52.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.374+0000 7fa18fe00640 1 -- 172.21.15.96:0/3031456652 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa18400ca00 con 0x7fa19815fae0 2024-05-28T06:26:52.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.374+0000 7fa18fe00640 1 -- 172.21.15.96:0/3031456652 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa18400cba0 con 0x7fa19815fae0 2024-05-28T06:26:52.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.374+0000 7fa19d600640 1 -- 172.21.15.96:0/3031456652 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1981596e0 con 0x7fa19815fae0 2024-05-28T06:26:52.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.374+0000 7fa18fe00640 1 -- 172.21.15.96:0/3031456652 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa184005990 con 0x7fa19815fae0 2024-05-28T06:26:52.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.374+0000 7fa19d600640 1 -- 172.21.15.96:0/3031456652 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa198159ca0 con 0x7fa19815fae0 2024-05-28T06:26:52.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.375+0000 7fa19d600640 1 -- 172.21.15.96:0/3031456652 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1980689e0 con 0x7fa19815fae0 2024-05-28T06:26:52.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.377+0000 7fa18fe00640 1 -- 172.21.15.96:0/3031456652 <== mon.1 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 49) ==== 94049+0+0 (secure 0 0 0) 0x7fa184005b30 con 0x7fa19815fae0 2024-05-28T06:26:52.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.378+0000 7fa18fe00640 1 --2- 172.21.15.96:0/3031456652 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa17806ea20 0x7fa178070ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:26:52.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.378+0000 7fa18fe00640 1 -- 172.21.15.96:0/3031456652 <== mon.1 v2:172.21.15.123:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 5823+0+0 (secure 0 0 0) 0x7fa18408afd0 con 0x7fa19815fae0 2024-05-28T06:26:52.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.378+0000 7fa197e00640 1 --2- 172.21.15.96:0/3031456652 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa17806ea20 0x7fa178070ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:26:52.373 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.379+0000 7fa197e00640 1 --2- 172.21.15.96:0/3031456652 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa17806ea20 0x7fa178070ee0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7fa188001780 tx=0x7fa188009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:26:52.376 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.383+0000 7fa18fe00640 1 -- 172.21.15.96:0/3031456652 <== mon.1 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa184017020 con 0x7fa19815fae0 2024-05-28T06:26:52.631 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.638+0000 7fa19d600640 1 -- 172.21.15.96:0/3031456652 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa1980b6c90 con 0x7fa17806ea20 2024-05-28T06:26:52.650 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.657+0000 7fa18fe00640 1 -- 172.21.15.96:0/3031456652 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252855 (secure 0 0 0) 0x7fa1980b6c90 con 0x7fa17806ea20 2024-05-28T06:26:52.653 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:26:52.655 INFO:teuthology.orchestra.run.smithi096.stderr:dumped all 2024-05-28T06:26:52.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.662+0000 7fa19d600640 1 -- 172.21.15.96:0/3031456652 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa17806ea20 msgr2=0x7fa178070ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:52.656 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.662+0000 7fa19d600640 1 --2- 172.21.15.96:0/3031456652 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa17806ea20 0x7fa178070ee0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7fa188001780 tx=0x7fa188009040 comp rx=0 tx=0).stop 2024-05-28T06:26:52.657 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.662+0000 7fa19d600640 1 -- 172.21.15.96:0/3031456652 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa19815fae0 msgr2=0x7fa1980c8000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:26:52.657 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.662+0000 7fa19d600640 1 --2- 172.21.15.96:0/3031456652 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa19815fae0 0x7fa1980c8000 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fa18400e900 tx=0x7fa18400edc0 comp rx=0 tx=0).stop 2024-05-28T06:26:52.657 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.662+0000 7fa19d600640 1 -- 172.21.15.96:0/3031456652 shutdown_connections 2024-05-28T06:26:52.657 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.662+0000 7fa19d600640 1 --2- 172.21.15.96:0/3031456652 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7fa17806ea20 0x7fa178070ee0 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:52.657 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.662+0000 7fa19d600640 1 --2- 172.21.15.96:0/3031456652 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa198160380 0x7fa1980c8540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:52.657 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.662+0000 7fa19d600640 1 --2- 172.21.15.96:0/3031456652 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa19815fae0 0x7fa1980c8000 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:52.657 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.662+0000 7fa19d600640 1 --2- 172.21.15.96:0/3031456652 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fa1980b5430 0x7fa198158b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:26:52.657 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.662+0000 7fa19d600640 1 -- 172.21.15.96:0/3031456652 >> 172.21.15.96:0/3031456652 conn(0x7fa19808bfc0 msgr2=0x7fa19815d950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:26:52.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.663+0000 7fa19d600640 1 -- 172.21.15.96:0/3031456652 shutdown_connections 2024-05-28T06:26:52.658 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:26:52.663+0000 7fa19d600640 1 -- 172.21.15.96:0/3031456652 wait complete. 2024-05-28T06:26:52.680 INFO:teuthology.orchestra.run.smithi096.stdout:{"pg_ready":true,"pg_map":{"version":811,"stamp":"2024-05-28T06:26:52.084318+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":18398488159,"num_objects":4420,"num_object_clones":1,"num_object_copies":8840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4420,"num_whiteouts":1,"num_read":4092,"num_read_kb":15909056,"num_write":5711,"num_write_kb":18960629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":41943040,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6127,"ondisk_log_size":6127,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":37076632,"kb_used_data":35947544,"kb_used_omap":12,"kb_used_meta":1129011,"kb_avail":717898088,"statfs":{"total":773094113280,"available":735127642112,"internally_reserved":0,"allocated":36810285056,"data_stored":36802793662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":1156107824},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":249,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004521"},"pg_stats":[{"pgid":"19.3f","version":"132'81","reported_seq":169,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:42.005903+0000","last_change":"2024-05-28T06:24:24.007430+0000","last_active":"2024-05-28T06:26:42.005903+0000","last_peered":"2024-05-28T06:26:42.005903+0000","last_clean":"2024-05-28T06:26:42.005903+0000","last_became_active":"2024-05-28T06:24:24.006988+0000","last_became_peered":"2024-05-28T06:24:24.006988+0000","last_unstale":"2024-05-28T06:26:42.005903+0000","last_undegraded":"2024-05-28T06:26:42.005903+0000","last_fullsized":"2024-05-28T06:26:42.005903+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260046848,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":70,"num_read_kb":286720,"num_write":72,"num_write_kb":274432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3e","version":"132'98","reported_seq":184,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:42.010512+0000","last_change":"2024-05-28T06:24:23.986742+0000","last_active":"2024-05-28T06:26:42.010512+0000","last_peered":"2024-05-28T06:26:42.010512+0000","last_clean":"2024-05-28T06:26:42.010512+0000","last_became_active":"2024-05-28T06:24:23.986471+0000","last_became_peered":"2024-05-28T06:24:23.986471+0000","last_unstale":"2024-05-28T06:26:42.010512+0000","last_undegraded":"2024-05-28T06:26:42.010512+0000","last_fullsized":"2024-05-28T06:26:42.010512+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327155712,"num_objects":78,"num_object_clones":0,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78,"num_whiteouts":0,"num_read":68,"num_read_kb":278528,"num_write":84,"num_write_kb":331776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3d","version":"132'101","reported_seq":188,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:42.004491+0000","last_change":"2024-05-28T06:24:24.007698+0000","last_active":"2024-05-28T06:26:42.004491+0000","last_peered":"2024-05-28T06:26:42.004491+0000","last_clean":"2024-05-28T06:26:42.004491+0000","last_became_active":"2024-05-28T06:24:24.007416+0000","last_became_peered":"2024-05-28T06:24:24.007416+0000","last_unstale":"2024-05-28T06:26:42.004491+0000","last_undegraded":"2024-05-28T06:26:42.004491+0000","last_fullsized":"2024-05-28T06:26:42.004491+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:24:31.676324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306184192,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":70,"num_read_kb":253960,"num_write":93,"num_write_kb":323588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.3c","version":"132'78","reported_seq":141,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:42.016262+0000","last_change":"2024-05-28T06:24:24.011512+0000","last_active":"2024-05-28T06:26:42.016262+0000","last_peered":"2024-05-28T06:26:42.016262+0000","last_clean":"2024-05-28T06:26:42.016262+0000","last_became_active":"2024-05-28T06:24:24.011234+0000","last_became_peered":"2024-05-28T06:24:24.011234+0000","last_unstale":"2024-05-28T06:26:42.016262+0000","last_undegraded":"2024-05-28T06:26:42.016262+0000","last_fullsized":"2024-05-28T06:26:42.016262+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281018368,"num_objects":67,"num_object_clones":0,"num_object_copies":134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":0,"num_read":45,"num_read_kb":176130,"num_write":71,"num_write_kb":278529,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3b","version":"132'69","reported_seq":123,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:41.986798+0000","last_change":"2024-05-28T06:24:24.019397+0000","last_active":"2024-05-28T06:26:41.986798+0000","last_peered":"2024-05-28T06:26:41.986798+0000","last_clean":"2024-05-28T06:26:41.986798+0000","last_became_active":"2024-05-28T06:24:24.018334+0000","last_became_peered":"2024-05-28T06:24:24.018334+0000","last_unstale":"2024-05-28T06:26:41.986798+0000","last_undegraded":"2024-05-28T06:26:41.986798+0000","last_fullsized":"2024-05-28T06:26:41.986798+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:22:34.611665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":230686720,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":38,"num_read_kb":155648,"num_write":65,"num_write_kb":245760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.3a","version":"132'106","reported_seq":226,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:42.003885+0000","last_change":"2024-05-28T06:24:55.231438+0000","last_active":"2024-05-28T06:26:42.003885+0000","last_peered":"2024-05-28T06:26:42.003885+0000","last_clean":"2024-05-28T06:26:42.003885+0000","last_became_active":"2024-05-28T06:24:54.990013+0000","last_became_peered":"2024-05-28T06:24:54.990013+0000","last_unstale":"2024-05-28T06:26:42.003885+0000","last_undegraded":"2024-05-28T06:26:42.003885+0000","last_fullsized":"2024-05-28T06:26:42.003885+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":360710405,"num_objects":87,"num_object_clones":0,"num_object_copies":174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87,"num_whiteouts":0,"num_read":84,"num_read_kb":339969,"num_write":97,"num_write_kb":368642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":25165824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.39","version":"132'79","reported_seq":141,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:42.006082+0000","last_change":"2024-05-28T06:24:24.006117+0000","last_active":"2024-05-28T06:26:42.006082+0000","last_peered":"2024-05-28T06:26:42.006082+0000","last_clean":"2024-05-28T06:26:42.006082+0000","last_became_active":"2024-05-28T06:24:24.005839+0000","last_became_peered":"2024-05-28T06:24:24.005839+0000","last_unstale":"2024-05-28T06:26:42.006082+0000","last_undegraded":"2024-05-28T06:26:42.006082+0000","last_fullsized":"2024-05-28T06:26:42.006082+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222298112,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":45,"num_read_kb":184320,"num_write":65,"num_write_kb":241664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.38","version":"132'86","reported_seq":149,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:42.008386+0000","last_change":"2024-05-28T06:24:24.008772+0000","last_active":"2024-05-28T06:26:42.008386+0000","last_peered":"2024-05-28T06:26:42.008386+0000","last_clean":"2024-05-28T06:26:42.008386+0000","last_became_active":"2024-05-28T06:24:24.008334+0000","last_became_peered":"2024-05-28T06:24:24.008334+0000","last_unstale":"2024-05-28T06:26:42.008386+0000","last_undegraded":"2024-05-28T06:26:42.008386+0000","last_fullsized":"2024-05-28T06:26:42.008386+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:10:50.477572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":297795584,"num_objects":71,"num_object_clones":0,"num_object_copies":142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":47,"num_read_kb":192512,"num_write":79,"num_write_kb":307200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.37","version":"132'88","reported_seq":158,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:42.003299+0000","last_change":"2024-05-28T06:24:24.018229+0000","last_active":"2024-05-28T06:26:42.003299+0000","last_peered":"2024-05-28T06:26:42.003299+0000","last_clean":"2024-05-28T06:26:42.003299+0000","last_became_active":"2024-05-28T06:24:24.017929+0000","last_became_peered":"2024-05-28T06:24:24.017929+0000","last_unstale":"2024-05-28T06:26:42.003299+0000","last_undegraded":"2024-05-28T06:26:42.003299+0000","last_fullsized":"2024-05-28T06:26:42.003299+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:58:16.573705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":297795716,"num_objects":72,"num_object_clones":0,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72,"num_whiteouts":0,"num_read":53,"num_read_kb":204803,"num_write":80,"num_write_kb":303106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.36","version":"132'69","reported_seq":145,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:47.783544+0000","last_change":"2024-05-28T06:24:23.995403+0000","last_active":"2024-05-28T06:26:47.783544+0000","last_peered":"2024-05-28T06:26:47.783544+0000","last_clean":"2024-05-28T06:26:47.783544+0000","last_became_active":"2024-05-28T06:24:23.994945+0000","last_became_peered":"2024-05-28T06:24:23.994945+0000","last_unstale":"2024-05-28T06:26:47.783544+0000","last_undegraded":"2024-05-28T06:26:47.783544+0000","last_fullsized":"2024-05-28T06:26:47.783544+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":255852544,"num_objects":61,"num_object_clones":0,"num_object_copies":122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":0,"num_read":58,"num_read_kb":237568,"num_write":63,"num_write_kb":253952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.35","version":"132'92","reported_seq":205,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:47.091575+0000","last_change":"2024-05-28T06:24:23.992865+0000","last_active":"2024-05-28T06:26:47.091575+0000","last_peered":"2024-05-28T06:26:47.091575+0000","last_clean":"2024-05-28T06:26:47.091575+0000","last_became_active":"2024-05-28T06:24:23.991405+0000","last_became_peered":"2024-05-28T06:24:23.991405+0000","last_unstale":"2024-05-28T06:26:47.091575+0000","last_undegraded":"2024-05-28T06:26:47.091575+0000","last_fullsized":"2024-05-28T06:26:47.091575+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":322961408,"num_objects":77,"num_object_clones":0,"num_object_copies":154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77,"num_whiteouts":0,"num_read":95,"num_read_kb":389120,"num_write":83,"num_write_kb":327680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.34","version":"132'80","reported_seq":189,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:42.002503+0000","last_change":"2024-05-28T06:24:24.018385+0000","last_active":"2024-05-28T06:26:42.002503+0000","last_peered":"2024-05-28T06:26:42.002503+0000","last_clean":"2024-05-28T06:26:42.002503+0000","last_became_active":"2024-05-28T06:24:24.017931+0000","last_became_peered":"2024-05-28T06:24:24.017931+0000","last_unstale":"2024-05-28T06:26:42.002503+0000","last_undegraded":"2024-05-28T06:26:42.002503+0000","last_fullsized":"2024-05-28T06:26:42.002503+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:11:21.508556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":289406976,"num_objects":69,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":0,"num_read":92,"num_read_kb":360452,"num_write":77,"num_write_kb":294913,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.33","version":"132'80","reported_seq":178,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:48.464372+0000","last_change":"2024-05-28T06:24:24.007399+0000","last_active":"2024-05-28T06:26:48.464372+0000","last_peered":"2024-05-28T06:26:48.464372+0000","last_clean":"2024-05-28T06:26:48.464372+0000","last_became_active":"2024-05-28T06:24:24.007121+0000","last_became_peered":"2024-05-28T06:24:24.007121+0000","last_unstale":"2024-05-28T06:26:48.464372+0000","last_undegraded":"2024-05-28T06:26:48.464372+0000","last_fullsized":"2024-05-28T06:26:48.464372+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":289406976,"num_objects":69,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":0,"num_read":80,"num_read_kb":327680,"num_write":75,"num_write_kb":294912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.32","version":"132'85","reported_seq":180,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:44.923771+0000","last_change":"2024-05-28T06:24:55.363255+0000","last_active":"2024-05-28T06:26:44.923771+0000","last_peered":"2024-05-28T06:26:44.923771+0000","last_clean":"2024-05-28T06:26:44.923771+0000","last_became_active":"2024-05-28T06:24:54.998097+0000","last_became_peered":"2024-05-28T06:24:54.998097+0000","last_unstale":"2024-05-28T06:26:44.923771+0000","last_undegraded":"2024-05-28T06:26:44.923771+0000","last_fullsized":"2024-05-28T06:26:44.923771+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":289406976,"num_objects":69,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":0,"num_read":62,"num_read_kb":253952,"num_write":77,"num_write_kb":299008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":16777216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.31","version":"132'79","reported_seq":127,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:44.969066+0000","last_change":"2024-05-28T06:24:24.020598+0000","last_active":"2024-05-28T06:26:44.969066+0000","last_peered":"2024-05-28T06:26:44.969066+0000","last_clean":"2024-05-28T06:26:44.969066+0000","last_became_active":"2024-05-28T06:24:24.020301+0000","last_became_peered":"2024-05-28T06:24:24.020301+0000","last_unstale":"2024-05-28T06:26:44.969066+0000","last_undegraded":"2024-05-28T06:26:44.969066+0000","last_fullsized":"2024-05-28T06:26:44.969066+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251658240,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":30,"num_read_kb":122880,"num_write":70,"num_write_kb":266240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.30","version":"132'88","reported_seq":157,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:44.978161+0000","last_change":"2024-05-28T06:24:23.995628+0000","last_active":"2024-05-28T06:26:44.978161+0000","last_peered":"2024-05-28T06:26:44.978161+0000","last_clean":"2024-05-28T06:26:44.978161+0000","last_became_active":"2024-05-28T06:24:23.995367+0000","last_became_peered":"2024-05-28T06:24:23.995367+0000","last_unstale":"2024-05-28T06:26:44.978161+0000","last_undegraded":"2024-05-28T06:26:44.978161+0000","last_fullsized":"2024-05-28T06:26:44.978161+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306184192,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":51,"num_read_kb":208896,"num_write":81,"num_write_kb":315392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2f","version":"132'71","reported_seq":131,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:45.165361+0000","last_change":"2024-05-28T06:24:23.987587+0000","last_active":"2024-05-28T06:26:45.165361+0000","last_peered":"2024-05-28T06:26:45.165361+0000","last_clean":"2024-05-28T06:26:45.165361+0000","last_became_active":"2024-05-28T06:24:23.986777+0000","last_became_peered":"2024-05-28T06:24:23.986777+0000","last_unstale":"2024-05-28T06:26:45.165361+0000","last_undegraded":"2024-05-28T06:26:45.165361+0000","last_fullsized":"2024-05-28T06:26:45.165361+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":230686720,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":42,"num_read_kb":172032,"num_write":65,"num_write_kb":245760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.2e","version":"132'93","reported_seq":198,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:44.978223+0000","last_change":"2024-05-28T06:24:23.998500+0000","last_active":"2024-05-28T06:26:44.978223+0000","last_peered":"2024-05-28T06:26:44.978223+0000","last_clean":"2024-05-28T06:26:44.978223+0000","last_became_active":"2024-05-28T06:24:23.998105+0000","last_became_peered":"2024-05-28T06:24:23.998105+0000","last_unstale":"2024-05-28T06:26:44.978223+0000","last_undegraded":"2024-05-28T06:26:44.978223+0000","last_fullsized":"2024-05-28T06:26:44.978223+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":310378496,"num_objects":74,"num_object_clones":0,"num_object_copies":148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74,"num_whiteouts":0,"num_read":87,"num_read_kb":356352,"num_write":82,"num_write_kb":315393,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2d","version":"132'90","reported_seq":175,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:42.007075+0000","last_change":"2024-05-28T06:24:23.997899+0000","last_active":"2024-05-28T06:26:42.007075+0000","last_peered":"2024-05-28T06:26:42.007075+0000","last_clean":"2024-05-28T06:26:42.007075+0000","last_became_active":"2024-05-28T06:24:23.997362+0000","last_became_peered":"2024-05-28T06:24:23.997362+0000","last_unstale":"2024-05-28T06:26:42.007075+0000","last_undegraded":"2024-05-28T06:26:42.007075+0000","last_fullsized":"2024-05-28T06:26:42.007075+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":310378496,"num_objects":74,"num_object_clones":0,"num_object_copies":148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74,"num_whiteouts":0,"num_read":67,"num_read_kb":274432,"num_write":82,"num_write_kb":319488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2c","version":"132'81","reported_seq":156,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:42.016166+0000","last_change":"2024-05-28T06:24:23.996440+0000","last_active":"2024-05-28T06:26:42.016166+0000","last_peered":"2024-05-28T06:26:42.016166+0000","last_clean":"2024-05-28T06:26:42.016166+0000","last_became_active":"2024-05-28T06:24:23.996067+0000","last_became_peered":"2024-05-28T06:24:23.996067+0000","last_unstale":"2024-05-28T06:26:42.016166+0000","last_undegraded":"2024-05-28T06:26:42.016166+0000","last_fullsized":"2024-05-28T06:26:42.016166+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":272629760,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":57,"num_read_kb":233472,"num_write":69,"num_write_kb":274432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2b","version":"132'92","reported_seq":200,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:41.998919+0000","last_change":"2024-05-28T06:24:24.018537+0000","last_active":"2024-05-28T06:26:41.998919+0000","last_peered":"2024-05-28T06:26:41.998919+0000","last_clean":"2024-05-28T06:26:41.998919+0000","last_became_active":"2024-05-28T06:24:24.018253+0000","last_became_peered":"2024-05-28T06:24:24.018253+0000","last_unstale":"2024-05-28T06:26:41.998919+0000","last_undegraded":"2024-05-28T06:26:41.998919+0000","last_fullsized":"2024-05-28T06:26:41.998919+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:13:05.023858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":293601280,"num_objects":70,"num_object_clones":0,"num_object_copies":140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70,"num_whiteouts":0,"num_read":91,"num_read_kb":364546,"num_write":82,"num_write_kb":303106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2a","version":"132'81","reported_seq":154,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:42.001486+0000","last_change":"2024-05-28T06:24:23.994663+0000","last_active":"2024-05-28T06:26:42.001486+0000","last_peered":"2024-05-28T06:26:42.001486+0000","last_clean":"2024-05-28T06:26:42.001486+0000","last_became_active":"2024-05-28T06:24:23.994410+0000","last_became_peered":"2024-05-28T06:24:23.994410+0000","last_unstale":"2024-05-28T06:26:42.001486+0000","last_undegraded":"2024-05-28T06:26:42.001486+0000","last_fullsized":"2024-05-28T06:26:42.001486+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":285212672,"num_objects":68,"num_object_clones":0,"num_object_copies":136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68,"num_whiteouts":0,"num_read":55,"num_read_kb":217090,"num_write":74,"num_write_kb":286721,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.29","version":"132'89","reported_seq":157,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:42.007074+0000","last_change":"2024-05-28T06:24:24.015438+0000","last_active":"2024-05-28T06:26:42.007074+0000","last_peered":"2024-05-28T06:26:42.007074+0000","last_clean":"2024-05-28T06:26:42.007074+0000","last_became_active":"2024-05-28T06:24:24.015124+0000","last_became_peered":"2024-05-28T06:24:24.015124+0000","last_unstale":"2024-05-28T06:26:42.007074+0000","last_undegraded":"2024-05-28T06:26:42.007074+0000","last_fullsized":"2024-05-28T06:26:42.007074+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:40:05.545358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281018368,"num_objects":67,"num_object_clones":0,"num_object_copies":134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":0,"num_read":51,"num_read_kb":208896,"num_write":81,"num_write_kb":303104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.28","version":"132'83","reported_seq":152,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:42.003075+0000","last_change":"2024-05-28T06:24:24.017147+0000","last_active":"2024-05-28T06:26:42.003075+0000","last_peered":"2024-05-28T06:26:42.003075+0000","last_clean":"2024-05-28T06:26:42.003075+0000","last_became_active":"2024-05-28T06:24:24.016692+0000","last_became_peered":"2024-05-28T06:24:24.016692+0000","last_unstale":"2024-05-28T06:26:42.003075+0000","last_undegraded":"2024-05-28T06:26:42.003075+0000","last_fullsized":"2024-05-28T06:26:42.003075+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281018368,"num_objects":67,"num_object_clones":0,"num_object_copies":134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":0,"num_read":51,"num_read_kb":208896,"num_write":75,"num_write_kb":286721,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.27","version":"132'91","reported_seq":172,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:42.012491+0000","last_change":"2024-05-28T06:24:24.007398+0000","last_active":"2024-05-28T06:26:42.012491+0000","last_peered":"2024-05-28T06:26:42.012491+0000","last_clean":"2024-05-28T06:26:42.012491+0000","last_became_active":"2024-05-28T06:24:24.006983+0000","last_became_peered":"2024-05-28T06:24:24.006983+0000","last_unstale":"2024-05-28T06:26:42.012491+0000","last_undegraded":"2024-05-28T06:26:42.012491+0000","last_fullsized":"2024-05-28T06:26:42.012491+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306184192,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":63,"num_read_kb":249858,"num_write":81,"num_write_kb":307202,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.26","version":"132'91","reported_seq":171,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:42.018038+0000","last_change":"2024-05-28T06:24:23.999225+0000","last_active":"2024-05-28T06:26:42.018038+0000","last_peered":"2024-05-28T06:26:42.018038+0000","last_clean":"2024-05-28T06:26:42.018038+0000","last_became_active":"2024-05-28T06:24:23.996859+0000","last_became_peered":"2024-05-28T06:24:23.996859+0000","last_unstale":"2024-05-28T06:26:42.018038+0000","last_undegraded":"2024-05-28T06:26:42.018038+0000","last_fullsized":"2024-05-28T06:26:42.018038+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":310378496,"num_objects":74,"num_object_clones":0,"num_object_copies":148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74,"num_whiteouts":0,"num_read":62,"num_read_kb":253952,"num_write":82,"num_write_kb":319488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.25","version":"132'78","reported_seq":162,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:42.008351+0000","last_change":"2024-05-28T06:24:24.007463+0000","last_active":"2024-05-28T06:26:42.008351+0000","last_peered":"2024-05-28T06:26:42.008351+0000","last_clean":"2024-05-28T06:26:42.008351+0000","last_became_active":"2024-05-28T06:24:24.007194+0000","last_became_peered":"2024-05-28T06:24:24.007194+0000","last_unstale":"2024-05-28T06:26:42.008351+0000","last_undegraded":"2024-05-28T06:26:42.008351+0000","last_fullsized":"2024-05-28T06:26:42.008351+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":272629760,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":66,"num_read_kb":258051,"num_write":73,"num_write_kb":278529,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.24","version":"132'82","reported_seq":133,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:42.005084+0000","last_change":"2024-05-28T06:24:24.016291+0000","last_active":"2024-05-28T06:26:42.005084+0000","last_peered":"2024-05-28T06:26:42.005084+0000","last_clean":"2024-05-28T06:26:42.005084+0000","last_became_active":"2024-05-28T06:24:24.015990+0000","last_became_peered":"2024-05-28T06:24:24.015990+0000","last_unstale":"2024-05-28T06:26:42.005084+0000","last_undegraded":"2024-05-28T06:26:42.005084+0000","last_fullsized":"2024-05-28T06:26:42.005084+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:02:16.956326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281018368,"num_objects":67,"num_object_clones":0,"num_object_copies":134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":0,"num_read":34,"num_read_kb":131074,"num_write":77,"num_write_kb":290817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.23","version":"132'88","reported_seq":177,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:42.018110+0000","last_change":"2024-05-28T06:24:24.019908+0000","last_active":"2024-05-28T06:26:42.018110+0000","last_peered":"2024-05-28T06:26:42.018110+0000","last_clean":"2024-05-28T06:26:42.018110+0000","last_became_active":"2024-05-28T06:24:24.019497+0000","last_became_peered":"2024-05-28T06:24:24.019497+0000","last_unstale":"2024-05-28T06:26:42.018110+0000","last_undegraded":"2024-05-28T06:26:42.018110+0000","last_fullsized":"2024-05-28T06:26:42.018110+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":297795584,"num_objects":71,"num_object_clones":0,"num_object_copies":142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":71,"num_read_kb":274436,"num_write":81,"num_write_kb":307201,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.22","version":"132'96","reported_seq":176,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:42.018867+0000","last_change":"2024-05-28T06:24:24.019852+0000","last_active":"2024-05-28T06:26:42.018867+0000","last_peered":"2024-05-28T06:26:42.018867+0000","last_clean":"2024-05-28T06:26:42.018867+0000","last_became_active":"2024-05-28T06:24:24.019462+0000","last_became_peered":"2024-05-28T06:24:24.019462+0000","last_unstale":"2024-05-28T06:26:42.018867+0000","last_undegraded":"2024-05-28T06:26:42.018867+0000","last_fullsized":"2024-05-28T06:26:42.018867+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":314572800,"num_objects":75,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75,"num_whiteouts":0,"num_read":62,"num_read_kb":253952,"num_write":85,"num_write_kb":327680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.21","version":"132'77","reported_seq":183,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:42.020605+0000","last_change":"2024-05-28T06:24:24.011632+0000","last_active":"2024-05-28T06:26:42.020605+0000","last_peered":"2024-05-28T06:26:42.020605+0000","last_clean":"2024-05-28T06:26:42.020605+0000","last_became_active":"2024-05-28T06:24:24.011246+0000","last_became_peered":"2024-05-28T06:24:24.011246+0000","last_unstale":"2024-05-28T06:26:42.020605+0000","last_undegraded":"2024-05-28T06:26:42.020605+0000","last_fullsized":"2024-05-28T06:26:42.020605+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:05:09.592919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":276824064,"num_objects":66,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":89,"num_read_kb":364544,"num_write":66,"num_write_kb":270336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.20","version":"132'85","reported_seq":174,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:42.007574+0000","last_change":"2024-05-28T06:24:23.991803+0000","last_active":"2024-05-28T06:26:42.007574+0000","last_peered":"2024-05-28T06:26:42.007574+0000","last_clean":"2024-05-28T06:26:42.007574+0000","last_became_active":"2024-05-28T06:24:23.991245+0000","last_became_peered":"2024-05-28T06:24:23.991245+0000","last_unstale":"2024-05-28T06:26:42.007574+0000","last_undegraded":"2024-05-28T06:26:42.007574+0000","last_fullsized":"2024-05-28T06:26:42.007574+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":272629760,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":71,"num_read_kb":282626,"num_write":75,"num_write_kb":282625,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1e","version":"132'72","reported_seq":132,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:42.002253+0000","last_change":"2024-05-28T06:24:24.018032+0000","last_active":"2024-05-28T06:26:42.002253+0000","last_peered":"2024-05-28T06:26:42.002253+0000","last_clean":"2024-05-28T06:26:42.002253+0000","last_became_active":"2024-05-28T06:24:24.017727+0000","last_became_peered":"2024-05-28T06:24:24.017727+0000","last_unstale":"2024-05-28T06:26:42.002253+0000","last_undegraded":"2024-05-28T06:26:42.002253+0000","last_fullsized":"2024-05-28T06:26:42.002253+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260046848,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":42,"num_read_kb":172032,"num_write":68,"num_write_kb":266240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1f","version":"129'4","reported_seq":23,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:55.134481+0000","last_change":"2024-05-28T06:24:23.006418+0000","last_active":"2024-05-28T06:24:55.134481+0000","last_peered":"2024-05-28T06:24:55.134481+0000","last_clean":"2024-05-28T06:24:55.134481+0000","last_became_active":"2024-05-28T06:24:23.006043+0000","last_became_peered":"2024-05-28T06:24:23.006043+0000","last_unstale":"2024-05-28T06:24:55.134481+0000","last_undegraded":"2024-05-28T06:24:55.134481+0000","last_fullsized":"2024-05-28T06:24:55.134481+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:31:42.560242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.1f","version":"132'91","reported_seq":176,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:42.019722+0000","last_change":"2024-05-28T06:24:24.008420+0000","last_active":"2024-05-28T06:26:42.019722+0000","last_peered":"2024-05-28T06:26:42.019722+0000","last_clean":"2024-05-28T06:26:42.019722+0000","last_became_active":"2024-05-28T06:24:24.008078+0000","last_became_peered":"2024-05-28T06:24:24.008078+0000","last_unstale":"2024-05-28T06:26:42.019722+0000","last_undegraded":"2024-05-28T06:26:42.019722+0000","last_fullsized":"2024-05-28T06:26:42.019722+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:08:27.007107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":293601280,"num_objects":70,"num_object_clones":0,"num_object_copies":140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70,"num_whiteouts":0,"num_read":68,"num_read_kb":262148,"num_write":82,"num_write_kb":303106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1e","version":"129'1","reported_seq":20,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:55.454913+0000","last_change":"2024-05-28T06:24:22.995399+0000","last_active":"2024-05-28T06:24:55.454913+0000","last_peered":"2024-05-28T06:24:55.454913+0000","last_clean":"2024-05-28T06:24:55.454913+0000","last_became_active":"2024-05-28T06:24:22.994896+0000","last_became_peered":"2024-05-28T06:24:22.994896+0000","last_unstale":"2024-05-28T06:24:55.454913+0000","last_undegraded":"2024-05-28T06:24:55.454913+0000","last_fullsized":"2024-05-28T06:24:55.454913+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:06:41.660486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1c","version":"132'77","reported_seq":157,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:42.006491+0000","last_change":"2024-05-28T06:24:24.006543+0000","last_active":"2024-05-28T06:26:42.006491+0000","last_peered":"2024-05-28T06:26:42.006491+0000","last_clean":"2024-05-28T06:26:42.006491+0000","last_became_active":"2024-05-28T06:24:24.006258+0000","last_became_peered":"2024-05-28T06:24:24.006258+0000","last_unstale":"2024-05-28T06:26:42.006491+0000","last_undegraded":"2024-05-28T06:26:42.006491+0000","last_fullsized":"2024-05-28T06:26:42.006491+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:34:58.279352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":255852805,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":64,"num_read_kb":262144,"num_write":70,"num_write_kb":266241,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1d","version":"132'4","reported_seq":31,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:49.524027+0000","last_change":"2024-05-28T06:26:49.524081+0000","last_active":"2024-05-28T06:26:49.524027+0000","last_peered":"2024-05-28T06:26:49.524027+0000","last_clean":"2024-05-28T06:26:49.524027+0000","last_became_active":"2024-05-28T06:24:22.991407+0000","last_became_peered":"2024-05-28T06:24:22.991407+0000","last_unstale":"2024-05-28T06:26:49.524027+0000","last_undegraded":"2024-05-28T06:26:49.524027+0000","last_fullsized":"2024-05-28T06:26:49.524027+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'4","last_scrub_stamp":"2024-05-28T06:26:49.523934+0000","last_deep_scrub":"132'4","last_deep_scrub_stamp":"2024-05-28T06:26:49.523934+0000","last_clean_scrub_stamp":"2024-05-28T06:26:49.523934+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:56:48.794586+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1d","version":"132'100","reported_seq":203,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:42.011634+0000","last_change":"2024-05-28T06:24:24.013028+0000","last_active":"2024-05-28T06:26:42.011634+0000","last_peered":"2024-05-28T06:26:42.011634+0000","last_clean":"2024-05-28T06:26:42.011634+0000","last_became_active":"2024-05-28T06:24:24.010356+0000","last_became_peered":"2024-05-28T06:24:24.010356+0000","last_unstale":"2024-05-28T06:26:42.011634+0000","last_undegraded":"2024-05-28T06:26:42.011634+0000","last_fullsized":"2024-05-28T06:26:42.011634+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:46:38.879485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":297795584,"num_objects":71,"num_object_clones":0,"num_object_copies":142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":87,"num_read_kb":356352,"num_write":89,"num_write_kb":327680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1c","version":"132'3","reported_seq":30,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:48.983517+0000","last_change":"2024-05-28T06:26:48.983579+0000","last_active":"2024-05-28T06:26:48.983517+0000","last_peered":"2024-05-28T06:26:48.983517+0000","last_clean":"2024-05-28T06:26:48.983517+0000","last_became_active":"2024-05-28T06:24:22.989103+0000","last_became_peered":"2024-05-28T06:24:22.989103+0000","last_unstale":"2024-05-28T06:26:48.983517+0000","last_undegraded":"2024-05-28T06:26:48.983517+0000","last_fullsized":"2024-05-28T06:26:48.983517+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'3","last_scrub_stamp":"2024-05-28T06:26:48.983322+0000","last_deep_scrub":"132'3","last_deep_scrub_stamp":"2024-05-28T06:26:48.983322+0000","last_clean_scrub_stamp":"2024-05-28T06:26:48.983322+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:37:10.958945+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1a","version":"132'83","reported_seq":153,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:42.006150+0000","last_change":"2024-05-28T06:24:24.008297+0000","last_active":"2024-05-28T06:26:42.006150+0000","last_peered":"2024-05-28T06:26:42.006150+0000","last_clean":"2024-05-28T06:26:42.006150+0000","last_became_active":"2024-05-28T06:24:24.007987+0000","last_became_peered":"2024-05-28T06:24:24.007987+0000","last_unstale":"2024-05-28T06:26:42.006150+0000","last_undegraded":"2024-05-28T06:26:42.006150+0000","last_fullsized":"2024-05-28T06:26:42.006150+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301989888,"num_objects":72,"num_object_clones":0,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72,"num_whiteouts":0,"num_read":53,"num_read_kb":217088,"num_write":76,"num_write_kb":303104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1b","version":"129'1","reported_seq":28,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:48.008602+0000","last_change":"2024-05-28T06:26:48.008624+0000","last_active":"2024-05-28T06:26:48.008602+0000","last_peered":"2024-05-28T06:26:48.008602+0000","last_clean":"2024-05-28T06:26:48.008602+0000","last_became_active":"2024-05-28T06:24:22.985617+0000","last_became_peered":"2024-05-28T06:24:22.985617+0000","last_unstale":"2024-05-28T06:26:48.008602+0000","last_undegraded":"2024-05-28T06:26:48.008602+0000","last_fullsized":"2024-05-28T06:26:48.008602+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'1","last_scrub_stamp":"2024-05-28T06:26:48.008531+0000","last_deep_scrub":"129'1","last_deep_scrub_stamp":"2024-05-28T06:26:48.008531+0000","last_clean_scrub_stamp":"2024-05-28T06:26:48.008531+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:48:17.109256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1b","version":"132'89","reported_seq":195,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:42.004645+0000","last_change":"2024-05-28T06:24:23.987152+0000","last_active":"2024-05-28T06:26:42.004645+0000","last_peered":"2024-05-28T06:26:42.004645+0000","last_clean":"2024-05-28T06:26:42.004645+0000","last_became_active":"2024-05-28T06:24:23.985828+0000","last_became_peered":"2024-05-28T06:24:23.985828+0000","last_unstale":"2024-05-28T06:26:42.004645+0000","last_undegraded":"2024-05-28T06:26:42.004645+0000","last_fullsized":"2024-05-28T06:26:42.004645+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306184192,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":89,"num_read_kb":364544,"num_write":77,"num_write_kb":307200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1a","version":"129'1","reported_seq":28,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:49.195535+0000","last_change":"2024-05-28T06:26:49.195556+0000","last_active":"2024-05-28T06:26:49.195535+0000","last_peered":"2024-05-28T06:26:49.195535+0000","last_clean":"2024-05-28T06:26:49.195535+0000","last_became_active":"2024-05-28T06:24:22.989801+0000","last_became_peered":"2024-05-28T06:24:22.989801+0000","last_unstale":"2024-05-28T06:26:49.195535+0000","last_undegraded":"2024-05-28T06:26:49.195535+0000","last_fullsized":"2024-05-28T06:26:49.195535+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'1","last_scrub_stamp":"2024-05-28T06:26:49.195459+0000","last_deep_scrub":"129'1","last_deep_scrub_stamp":"2024-05-28T06:26:49.195459+0000","last_clean_scrub_stamp":"2024-05-28T06:26:49.195459+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:31:06.539914+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.18","version":"132'79","reported_seq":163,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:42.008595+0000","last_change":"2024-05-28T06:24:24.003806+0000","last_active":"2024-05-28T06:26:42.008595+0000","last_peered":"2024-05-28T06:26:42.008595+0000","last_clean":"2024-05-28T06:26:42.008595+0000","last_became_active":"2024-05-28T06:24:24.003446+0000","last_became_peered":"2024-05-28T06:24:24.003446+0000","last_unstale":"2024-05-28T06:26:42.008595+0000","last_undegraded":"2024-05-28T06:26:42.008595+0000","last_fullsized":"2024-05-28T06:26:42.008595+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:05:52.238604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260046848,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":67,"num_read_kb":274432,"num_write":72,"num_write_kb":274432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":20,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:49.848927+0000","last_change":"2024-05-28T06:24:23.007861+0000","last_active":"2024-05-28T06:26:49.848927+0000","last_peered":"2024-05-28T06:26:49.848927+0000","last_clean":"2024-05-28T06:26:49.848927+0000","last_became_active":"2024-05-28T06:24:23.007494+0000","last_became_peered":"2024-05-28T06:24:23.007494+0000","last_unstale":"2024-05-28T06:26:49.848927+0000","last_undegraded":"2024-05-28T06:26:49.848927+0000","last_fullsized":"2024-05-28T06:26:49.848927+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.19","version":"132'82","reported_seq":162,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:47.783689+0000","last_change":"2024-05-28T06:24:23.993309+0000","last_active":"2024-05-28T06:26:47.783689+0000","last_peered":"2024-05-28T06:26:47.783689+0000","last_clean":"2024-05-28T06:26:47.783689+0000","last_became_active":"2024-05-28T06:24:23.987466+0000","last_became_peered":"2024-05-28T06:24:23.987466+0000","last_unstale":"2024-05-28T06:26:47.783689+0000","last_undegraded":"2024-05-28T06:26:47.783689+0000","last_fullsized":"2024-05-28T06:26:47.783689+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":276824064,"num_objects":66,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":62,"num_read_kb":245762,"num_write":72,"num_write_kb":278529,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":20,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:47.783233+0000","last_change":"2024-05-28T06:24:23.000407+0000","last_active":"2024-05-28T06:26:47.783233+0000","last_peered":"2024-05-28T06:26:47.783233+0000","last_clean":"2024-05-28T06:26:47.783233+0000","last_became_active":"2024-05-28T06:24:23.000037+0000","last_became_peered":"2024-05-28T06:24:23.000037+0000","last_unstale":"2024-05-28T06:26:47.783233+0000","last_undegraded":"2024-05-28T06:26:47.783233+0000","last_fullsized":"2024-05-28T06:26:47.783233+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"132'82","reported_seq":133,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:48.464215+0000","last_change":"2024-05-28T06:24:23.995548+0000","last_active":"2024-05-28T06:26:48.464215+0000","last_peered":"2024-05-28T06:26:48.464215+0000","last_clean":"2024-05-28T06:26:48.464215+0000","last_became_active":"2024-05-28T06:24:23.995204+0000","last_became_peered":"2024-05-28T06:24:23.995204+0000","last_unstale":"2024-05-28T06:26:48.464215+0000","last_undegraded":"2024-05-28T06:26:48.464215+0000","last_fullsized":"2024-05-28T06:26:48.464215+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260046848,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":33,"num_read_kb":135168,"num_write":70,"num_write_kb":270336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.17","version":"132'1","reported_seq":20,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:25:22.116453+0000","last_change":"2024-05-28T06:24:23.001114+0000","last_active":"2024-05-28T06:25:22.116453+0000","last_peered":"2024-05-28T06:25:22.116453+0000","last_clean":"2024-05-28T06:25:22.116453+0000","last_became_active":"2024-05-28T06:24:22.999720+0000","last_became_peered":"2024-05-28T06:24:22.999720+0000","last_unstale":"2024-05-28T06:25:22.116453+0000","last_undegraded":"2024-05-28T06:25:22.116453+0000","last_fullsized":"2024-05-28T06:25:22.116453+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:29:54.261773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.17","version":"132'70","reported_seq":125,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:41.998543+0000","last_change":"2024-05-28T06:24:24.016902+0000","last_active":"2024-05-28T06:26:41.998543+0000","last_peered":"2024-05-28T06:26:41.998543+0000","last_clean":"2024-05-28T06:26:41.998543+0000","last_became_active":"2024-05-28T06:24:24.016597+0000","last_became_peered":"2024-05-28T06:24:24.016597+0000","last_unstale":"2024-05-28T06:26:41.998543+0000","last_undegraded":"2024-05-28T06:26:41.998543+0000","last_fullsized":"2024-05-28T06:26:41.998543+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:08:42.734914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247463936,"num_objects":59,"num_object_clones":0,"num_object_copies":118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":0,"num_read":38,"num_read_kb":155648,"num_write":63,"num_write_kb":249856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":20,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:49.153826+0000","last_change":"2024-05-28T06:24:23.001188+0000","last_active":"2024-05-28T06:26:49.153826+0000","last_peered":"2024-05-28T06:26:49.153826+0000","last_clean":"2024-05-28T06:26:49.153826+0000","last_became_active":"2024-05-28T06:24:23.000878+0000","last_became_peered":"2024-05-28T06:24:23.000878+0000","last_unstale":"2024-05-28T06:26:49.153826+0000","last_undegraded":"2024-05-28T06:26:49.153826+0000","last_fullsized":"2024-05-28T06:26:49.153826+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.14","version":"132'79","reported_seq":171,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:42.006703+0000","last_change":"2024-05-28T06:24:24.011311+0000","last_active":"2024-05-28T06:26:42.006703+0000","last_peered":"2024-05-28T06:26:42.006703+0000","last_clean":"2024-05-28T06:26:42.006703+0000","last_became_active":"2024-05-28T06:24:24.010110+0000","last_became_peered":"2024-05-28T06:24:24.010110+0000","last_unstale":"2024-05-28T06:26:42.006703+0000","last_undegraded":"2024-05-28T06:26:42.006703+0000","last_fullsized":"2024-05-28T06:26:42.006703+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:46:31.105891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251658240,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":75,"num_read_kb":307200,"num_write":74,"num_write_kb":274432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.15","version":"132'24","reported_seq":43,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:18.326773+0000","last_change":"2024-05-28T06:24:22.999088+0000","last_active":"2024-05-28T06:26:18.326773+0000","last_peered":"2024-05-28T06:26:18.326773+0000","last_clean":"2024-05-28T06:26:18.326773+0000","last_became_active":"2024-05-28T06:24:22.998379+0000","last_became_peered":"2024-05-28T06:24:22.998379+0000","last_unstale":"2024-05-28T06:26:18.326773+0000","last_undegraded":"2024-05-28T06:26:18.326773+0000","last_fullsized":"2024-05-28T06:26:18.326773+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:59:19.637147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.15","version":"132'76","reported_seq":153,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:42.007863+0000","last_change":"2024-05-28T06:24:24.020278+0000","last_active":"2024-05-28T06:26:42.007863+0000","last_peered":"2024-05-28T06:26:42.007863+0000","last_clean":"2024-05-28T06:26:42.007863+0000","last_became_active":"2024-05-28T06:24:24.019973+0000","last_became_peered":"2024-05-28T06:24:24.019973+0000","last_unstale":"2024-05-28T06:26:42.007863+0000","last_undegraded":"2024-05-28T06:26:42.007863+0000","last_fullsized":"2024-05-28T06:26:42.007863+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:46:47.603731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":230686720,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":60,"num_read_kb":229380,"num_write":67,"num_write_kb":241666,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.14","version":"132'419","reported_seq":438,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:42.022522+0000","last_change":"2024-05-28T06:24:23.005145+0000","last_active":"2024-05-28T06:26:42.022522+0000","last_peered":"2024-05-28T06:26:42.022522+0000","last_clean":"2024-05-28T06:26:42.022522+0000","last_became_active":"2024-05-28T06:24:23.004831+0000","last_became_peered":"2024-05-28T06:24:23.004831+0000","last_unstale":"2024-05-28T06:26:42.022522+0000","last_undegraded":"2024-05-28T06:26:42.022522+0000","last_fullsized":"2024-05-28T06:26:42.022522+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":419,"log_dups_size":0,"ondisk_log_size":419,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:09:55.668305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1673486,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":418,"num_write_kb":1794,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.13","version":"132'81","reported_seq":149,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:44.969135+0000","last_change":"2024-05-28T06:24:24.002061+0000","last_active":"2024-05-28T06:26:44.969135+0000","last_peered":"2024-05-28T06:26:44.969135+0000","last_clean":"2024-05-28T06:26:44.969135+0000","last_became_active":"2024-05-28T06:24:24.001429+0000","last_became_peered":"2024-05-28T06:24:24.001429+0000","last_unstale":"2024-05-28T06:26:44.969135+0000","last_undegraded":"2024-05-28T06:26:44.969135+0000","last_fullsized":"2024-05-28T06:26:44.969135+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":297795584,"num_objects":71,"num_object_clones":0,"num_object_copies":142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":50,"num_read_kb":196610,"num_write":75,"num_write_kb":294913,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":28,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:48.685424+0000","last_change":"2024-05-28T06:26:48.685445+0000","last_active":"2024-05-28T06:26:48.685424+0000","last_peered":"2024-05-28T06:26:48.685424+0000","last_clean":"2024-05-28T06:26:48.685424+0000","last_became_active":"2024-05-28T06:24:22.993301+0000","last_became_peered":"2024-05-28T06:24:22.993301+0000","last_unstale":"2024-05-28T06:26:48.685424+0000","last_undegraded":"2024-05-28T06:26:48.685424+0000","last_fullsized":"2024-05-28T06:26:48.685424+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:26:48.685333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:26:48.685333+0000","last_clean_scrub_stamp":"2024-05-28T06:26:48.685333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:38:36.311248+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.10","version":"132'73","reported_seq":148,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:44.923686+0000","last_change":"2024-05-28T06:24:24.005361+0000","last_active":"2024-05-28T06:26:44.923686+0000","last_peered":"2024-05-28T06:26:44.923686+0000","last_clean":"2024-05-28T06:26:44.923686+0000","last_became_active":"2024-05-28T06:24:24.005080+0000","last_became_peered":"2024-05-28T06:24:24.005080+0000","last_unstale":"2024-05-28T06:26:44.923686+0000","last_undegraded":"2024-05-28T06:26:44.923686+0000","last_fullsized":"2024-05-28T06:26:44.923686+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":272629760,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":57,"num_read_kb":233472,"num_write":69,"num_write_kb":274432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":20,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:44.923748+0000","last_change":"2024-05-28T06:24:23.005565+0000","last_active":"2024-05-28T06:26:44.923748+0000","last_peered":"2024-05-28T06:26:44.923748+0000","last_clean":"2024-05-28T06:26:44.923748+0000","last_became_active":"2024-05-28T06:24:23.005194+0000","last_became_peered":"2024-05-28T06:24:23.005194+0000","last_unstale":"2024-05-28T06:26:44.923748+0000","last_undegraded":"2024-05-28T06:26:44.923748+0000","last_fullsized":"2024-05-28T06:26:44.923748+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.11","version":"132'89","reported_seq":168,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:42.002961+0000","last_change":"2024-05-28T06:24:24.009724+0000","last_active":"2024-05-28T06:26:42.002961+0000","last_peered":"2024-05-28T06:26:42.002961+0000","last_clean":"2024-05-28T06:26:42.002961+0000","last_became_active":"2024-05-28T06:24:24.009163+0000","last_became_peered":"2024-05-28T06:24:24.009163+0000","last_unstale":"2024-05-28T06:26:42.002961+0000","last_undegraded":"2024-05-28T06:26:42.002961+0000","last_fullsized":"2024-05-28T06:26:42.002961+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:36:46.619746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":318767104,"num_objects":76,"num_object_clones":0,"num_object_copies":152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76,"num_whiteouts":0,"num_read":62,"num_read_kb":253952,"num_write":84,"num_write_kb":323585,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.10","version":"129'8","reported_seq":27,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:55.070328+0000","last_change":"2024-05-28T06:24:22.996927+0000","last_active":"2024-05-28T06:24:55.070328+0000","last_peered":"2024-05-28T06:24:55.070328+0000","last_clean":"2024-05-28T06:24:55.070328+0000","last_became_active":"2024-05-28T06:24:22.996644+0000","last_became_peered":"2024-05-28T06:24:22.996644+0000","last_unstale":"2024-05-28T06:24:55.070328+0000","last_undegraded":"2024-05-28T06:24:55.070328+0000","last_fullsized":"2024-05-28T06:24:55.070328+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:17:26.759155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.e","version":"132'62","reported_seq":102,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:42.014137+0000","last_change":"2024-05-28T06:24:24.009329+0000","last_active":"2024-05-28T06:26:42.014137+0000","last_peered":"2024-05-28T06:26:42.014137+0000","last_clean":"2024-05-28T06:26:42.014137+0000","last_became_active":"2024-05-28T06:24:24.009033+0000","last_became_peered":"2024-05-28T06:24:24.009033+0000","last_unstale":"2024-05-28T06:26:42.014137+0000","last_undegraded":"2024-05-28T06:26:42.014137+0000","last_fullsized":"2024-05-28T06:26:42.014137+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:23:27.401090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218103808,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":23,"num_read_kb":94208,"num_write":58,"num_write_kb":225280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.f","version":"129'1","reported_seq":20,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:55.456823+0000","last_change":"2024-05-28T06:24:23.006567+0000","last_active":"2024-05-28T06:24:55.456823+0000","last_peered":"2024-05-28T06:24:55.456823+0000","last_clean":"2024-05-28T06:24:55.456823+0000","last_became_active":"2024-05-28T06:24:23.005650+0000","last_became_peered":"2024-05-28T06:24:23.005650+0000","last_unstale":"2024-05-28T06:24:55.456823+0000","last_undegraded":"2024-05-28T06:24:55.456823+0000","last_fullsized":"2024-05-28T06:24:55.456823+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:01:42.283187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.f","version":"132'89","reported_seq":180,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:41.999688+0000","last_change":"2024-05-28T06:24:24.016763+0000","last_active":"2024-05-28T06:26:41.999688+0000","last_peered":"2024-05-28T06:26:41.999688+0000","last_clean":"2024-05-28T06:26:41.999688+0000","last_became_active":"2024-05-28T06:24:24.016255+0000","last_became_peered":"2024-05-28T06:24:24.016255+0000","last_unstale":"2024-05-28T06:26:41.999688+0000","last_undegraded":"2024-05-28T06:26:41.999688+0000","last_fullsized":"2024-05-28T06:26:41.999688+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:06:19.643800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306184192,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":74,"num_read_kb":303104,"num_write":83,"num_write_kb":315393,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.e","version":"129'2","reported_seq":22,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:39.903129+0000","last_change":"2024-05-28T06:24:23.002513+0000","last_active":"2024-05-28T06:26:39.903129+0000","last_peered":"2024-05-28T06:26:39.903129+0000","last_clean":"2024-05-28T06:26:39.903129+0000","last_became_active":"2024-05-28T06:24:23.001946+0000","last_became_peered":"2024-05-28T06:24:23.001946+0000","last_unstale":"2024-05-28T06:26:39.903129+0000","last_undegraded":"2024-05-28T06:26:39.903129+0000","last_fullsized":"2024-05-28T06:26:39.903129+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"132'76","reported_seq":154,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:42.018578+0000","last_change":"2024-05-28T06:24:23.998032+0000","last_active":"2024-05-28T06:26:42.018578+0000","last_peered":"2024-05-28T06:26:42.018578+0000","last_clean":"2024-05-28T06:26:42.018578+0000","last_became_active":"2024-05-28T06:24:23.996433+0000","last_became_peered":"2024-05-28T06:24:23.996433+0000","last_unstale":"2024-05-28T06:26:42.018578+0000","last_undegraded":"2024-05-28T06:26:42.018578+0000","last_fullsized":"2024-05-28T06:26:42.018578+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":230686900,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":60,"num_read_kb":221190,"num_write":66,"num_write_kb":233476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.d","version":"132'2","reported_seq":30,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:50.053151+0000","last_change":"2024-05-28T06:26:50.053220+0000","last_active":"2024-05-28T06:26:50.053151+0000","last_peered":"2024-05-28T06:26:50.053151+0000","last_clean":"2024-05-28T06:26:50.053151+0000","last_became_active":"2024-05-28T06:24:22.992147+0000","last_became_peered":"2024-05-28T06:24:22.992147+0000","last_unstale":"2024-05-28T06:26:50.053151+0000","last_undegraded":"2024-05-28T06:26:50.053151+0000","last_fullsized":"2024-05-28T06:26:50.053151+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'2","last_scrub_stamp":"2024-05-28T06:26:50.052936+0000","last_deep_scrub":"132'2","last_deep_scrub_stamp":"2024-05-28T06:26:50.052936+0000","last_clean_scrub_stamp":"2024-05-28T06:26:50.052936+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:55:50.028331+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.d","version":"132'84","reported_seq":174,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:42.014496+0000","last_change":"2024-05-28T06:24:23.989538+0000","last_active":"2024-05-28T06:26:42.014496+0000","last_peered":"2024-05-28T06:26:42.014496+0000","last_clean":"2024-05-28T06:26:42.014496+0000","last_became_active":"2024-05-28T06:24:23.988631+0000","last_became_peered":"2024-05-28T06:24:23.988631+0000","last_unstale":"2024-05-28T06:26:42.014496+0000","last_undegraded":"2024-05-28T06:26:42.014496+0000","last_fullsized":"2024-05-28T06:26:42.014496+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":272629760,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":72,"num_read_kb":294912,"num_write":79,"num_write_kb":294912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":19,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:55.454861+0000","last_change":"2024-05-28T06:24:22.997400+0000","last_active":"2024-05-28T06:24:55.454861+0000","last_peered":"2024-05-28T06:24:55.454861+0000","last_clean":"2024-05-28T06:24:55.454861+0000","last_became_active":"2024-05-28T06:24:22.993636+0000","last_became_peered":"2024-05-28T06:24:22.993636+0000","last_unstale":"2024-05-28T06:24:55.454861+0000","last_undegraded":"2024-05-28T06:24:55.454861+0000","last_fullsized":"2024-05-28T06:24:55.454861+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:43:01.373860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.a","version":"132'83","reported_seq":175,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:42.018802+0000","last_change":"2024-05-28T06:24:24.007538+0000","last_active":"2024-05-28T06:26:42.018802+0000","last_peered":"2024-05-28T06:26:42.018802+0000","last_clean":"2024-05-28T06:26:42.018802+0000","last_became_active":"2024-05-28T06:24:24.007259+0000","last_became_peered":"2024-05-28T06:24:24.007259+0000","last_unstale":"2024-05-28T06:26:42.018802+0000","last_undegraded":"2024-05-28T06:26:42.018802+0000","last_fullsized":"2024-05-28T06:26:42.018802+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306184192,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":74,"num_read_kb":303104,"num_write":79,"num_write_kb":311296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.b","version":"132'3","reported_seq":22,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:25:22.100610+0000","last_change":"2024-05-28T06:24:22.996819+0000","last_active":"2024-05-28T06:25:22.100610+0000","last_peered":"2024-05-28T06:25:22.100610+0000","last_clean":"2024-05-28T06:25:22.100610+0000","last_became_active":"2024-05-28T06:24:22.995489+0000","last_became_peered":"2024-05-28T06:24:22.995489+0000","last_unstale":"2024-05-28T06:25:22.100610+0000","last_undegraded":"2024-05-28T06:25:22.100610+0000","last_fullsized":"2024-05-28T06:25:22.100610+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:24:52.967145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.b","version":"132'98","reported_seq":180,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:42.001384+0000","last_change":"2024-05-28T06:24:24.015790+0000","last_active":"2024-05-28T06:26:42.001384+0000","last_peered":"2024-05-28T06:26:42.001384+0000","last_clean":"2024-05-28T06:26:42.001384+0000","last_became_active":"2024-05-28T06:24:24.015295+0000","last_became_peered":"2024-05-28T06:24:24.015295+0000","last_unstale":"2024-05-28T06:26:42.001384+0000","last_undegraded":"2024-05-28T06:26:42.001384+0000","last_fullsized":"2024-05-28T06:26:42.001384+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":314572932,"num_objects":76,"num_object_clones":0,"num_object_copies":152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76,"num_whiteouts":0,"num_read":64,"num_read_kb":237574,"num_write":90,"num_write_kb":323588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":20,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:34.976737+0000","last_change":"2024-05-28T06:24:22.995343+0000","last_active":"2024-05-28T06:26:34.976737+0000","last_peered":"2024-05-28T06:26:34.976737+0000","last_clean":"2024-05-28T06:26:34.976737+0000","last_became_active":"2024-05-28T06:24:22.994227+0000","last_became_peered":"2024-05-28T06:24:22.994227+0000","last_unstale":"2024-05-28T06:26:34.976737+0000","last_undegraded":"2024-05-28T06:26:34.976737+0000","last_fullsized":"2024-05-28T06:26:34.976737+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.8","version":"132'99","reported_seq":158,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:42.011781+0000","last_change":"2024-05-28T06:24:23.987172+0000","last_active":"2024-05-28T06:26:42.011781+0000","last_peered":"2024-05-28T06:26:42.011781+0000","last_clean":"2024-05-28T06:26:42.011781+0000","last_became_active":"2024-05-28T06:24:23.986897+0000","last_became_peered":"2024-05-28T06:24:23.986897+0000","last_unstale":"2024-05-28T06:26:42.011781+0000","last_undegraded":"2024-05-28T06:26:42.011781+0000","last_fullsized":"2024-05-28T06:26:42.011781+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":343932928,"num_objects":82,"num_object_clones":0,"num_object_copies":164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82,"num_whiteouts":0,"num_read":41,"num_read_kb":167936,"num_write":88,"num_write_kb":348160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":20,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:34.976681+0000","last_change":"2024-05-28T06:24:22.991355+0000","last_active":"2024-05-28T06:26:34.976681+0000","last_peered":"2024-05-28T06:26:34.976681+0000","last_clean":"2024-05-28T06:26:34.976681+0000","last_became_active":"2024-05-28T06:24:22.991078+0000","last_became_peered":"2024-05-28T06:24:22.991078+0000","last_unstale":"2024-05-28T06:26:34.976681+0000","last_undegraded":"2024-05-28T06:26:34.976681+0000","last_fullsized":"2024-05-28T06:26:34.976681+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"132'79","reported_seq":152,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:42.020384+0000","last_change":"2024-05-28T06:24:24.009275+0000","last_active":"2024-05-28T06:26:42.020384+0000","last_peered":"2024-05-28T06:26:42.020384+0000","last_clean":"2024-05-28T06:26:42.020384+0000","last_became_active":"2024-05-28T06:24:24.008955+0000","last_became_peered":"2024-05-28T06:24:24.008955+0000","last_unstale":"2024-05-28T06:26:42.020384+0000","last_undegraded":"2024-05-28T06:26:42.020384+0000","last_fullsized":"2024-05-28T06:26:42.020384+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:32:09.823976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":276824064,"num_objects":66,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":56,"num_read_kb":229376,"num_write":68,"num_write_kb":274432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":20,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:34.902324+0000","last_change":"2024-05-28T06:24:23.002572+0000","last_active":"2024-05-28T06:26:34.902324+0000","last_peered":"2024-05-28T06:26:34.902324+0000","last_clean":"2024-05-28T06:26:34.902324+0000","last_became_active":"2024-05-28T06:24:23.002292+0000","last_became_peered":"2024-05-28T06:24:23.002292+0000","last_unstale":"2024-05-28T06:26:34.902324+0000","last_undegraded":"2024-05-28T06:26:34.902324+0000","last_fullsized":"2024-05-28T06:26:34.902324+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.6","version":"132'96","reported_seq":149,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:42.002879+0000","last_change":"2024-05-28T06:24:24.017529+0000","last_active":"2024-05-28T06:26:42.002879+0000","last_peered":"2024-05-28T06:26:42.002879+0000","last_clean":"2024-05-28T06:26:42.002879+0000","last_became_active":"2024-05-28T06:24:24.017221+0000","last_became_peered":"2024-05-28T06:24:24.017221+0000","last_unstale":"2024-05-28T06:26:42.002879+0000","last_undegraded":"2024-05-28T06:26:42.002879+0000","last_fullsized":"2024-05-28T06:26:42.002879+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306184192,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":35,"num_read_kb":143360,"num_write":87,"num_write_kb":327680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.7","version":"132'3","reported_seq":31,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:49.651675+0000","last_change":"2024-05-28T06:26:49.651701+0000","last_active":"2024-05-28T06:26:49.651675+0000","last_peered":"2024-05-28T06:26:49.651675+0000","last_clean":"2024-05-28T06:26:49.651675+0000","last_became_active":"2024-05-28T06:24:23.003502+0000","last_became_peered":"2024-05-28T06:24:23.003502+0000","last_unstale":"2024-05-28T06:26:49.651675+0000","last_undegraded":"2024-05-28T06:26:49.651675+0000","last_fullsized":"2024-05-28T06:26:49.651675+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'3","last_scrub_stamp":"2024-05-28T06:26:49.651607+0000","last_deep_scrub":"132'3","last_deep_scrub_stamp":"2024-05-28T06:26:49.651607+0000","last_clean_scrub_stamp":"2024-05-28T06:26:49.651607+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:53:07.376780+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.7","version":"132'92","reported_seq":180,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:42.017385+0000","last_change":"2024-05-28T06:24:24.020448+0000","last_active":"2024-05-28T06:26:42.017385+0000","last_peered":"2024-05-28T06:26:42.017385+0000","last_clean":"2024-05-28T06:26:42.017385+0000","last_became_active":"2024-05-28T06:24:24.019730+0000","last_became_peered":"2024-05-28T06:24:24.019730+0000","last_unstale":"2024-05-28T06:26:42.017385+0000","last_undegraded":"2024-05-28T06:26:42.017385+0000","last_fullsized":"2024-05-28T06:26:42.017385+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":310378496,"num_objects":74,"num_object_clones":0,"num_object_copies":148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74,"num_whiteouts":0,"num_read":70,"num_read_kb":286720,"num_write":84,"num_write_kb":323584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.6","version":"132'42","reported_seq":61,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:36.228053+0000","last_change":"2024-05-28T06:24:23.003133+0000","last_active":"2024-05-28T06:26:36.228053+0000","last_peered":"2024-05-28T06:26:36.228053+0000","last_clean":"2024-05-28T06:26:36.228053+0000","last_became_active":"2024-05-28T06:24:23.002821+0000","last_became_peered":"2024-05-28T06:24:23.002821+0000","last_unstale":"2024-05-28T06:26:36.228053+0000","last_undegraded":"2024-05-28T06:26:36.228053+0000","last_fullsized":"2024-05-28T06:26:36.228053+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:18:06.253798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":97,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.4","version":"132'87","reported_seq":192,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:42.013636+0000","last_change":"2024-05-28T06:24:24.004928+0000","last_active":"2024-05-28T06:26:42.013636+0000","last_peered":"2024-05-28T06:26:42.013636+0000","last_clean":"2024-05-28T06:26:42.013636+0000","last_became_active":"2024-05-28T06:24:24.004599+0000","last_became_peered":"2024-05-28T06:24:24.004599+0000","last_unstale":"2024-05-28T06:26:42.013636+0000","last_undegraded":"2024-05-28T06:26:42.013636+0000","last_fullsized":"2024-05-28T06:26:42.013636+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":276824064,"num_objects":66,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":87,"num_read_kb":356352,"num_write":80,"num_write_kb":294913,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":20,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:29.921423+0000","last_change":"2024-05-28T06:24:23.003603+0000","last_active":"2024-05-28T06:26:29.921423+0000","last_peered":"2024-05-28T06:26:29.921423+0000","last_clean":"2024-05-28T06:26:29.921423+0000","last_became_active":"2024-05-28T06:24:23.002468+0000","last_became_peered":"2024-05-28T06:24:23.002468+0000","last_unstale":"2024-05-28T06:26:29.921423+0000","last_undegraded":"2024-05-28T06:26:29.921423+0000","last_fullsized":"2024-05-28T06:26:29.921423+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.5","version":"132'93","reported_seq":157,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:42.017887+0000","last_change":"2024-05-28T06:24:23.999563+0000","last_active":"2024-05-28T06:26:42.017887+0000","last_peered":"2024-05-28T06:26:42.017887+0000","last_clean":"2024-05-28T06:26:42.017887+0000","last_became_active":"2024-05-28T06:24:23.999013+0000","last_became_peered":"2024-05-28T06:24:23.999013+0000","last_unstale":"2024-05-28T06:26:42.017887+0000","last_undegraded":"2024-05-28T06:26:42.017887+0000","last_fullsized":"2024-05-28T06:26:42.017887+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":285213316,"num_objects":72,"num_object_clones":1,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72,"num_whiteouts":1,"num_read":47,"num_read_kb":192512,"num_write":84,"num_write_kb":303107,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":28,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:47.544478+0000","last_change":"2024-05-28T06:26:47.544535+0000","last_active":"2024-05-28T06:26:47.544478+0000","last_peered":"2024-05-28T06:26:47.544478+0000","last_clean":"2024-05-28T06:26:47.544478+0000","last_became_active":"2024-05-28T06:24:22.986620+0000","last_became_peered":"2024-05-28T06:24:22.986620+0000","last_unstale":"2024-05-28T06:26:47.544478+0000","last_undegraded":"2024-05-28T06:26:47.544478+0000","last_fullsized":"2024-05-28T06:26:47.544478+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:26:47.544229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:26:47.544229+0000","last_clean_scrub_stamp":"2024-05-28T06:26:47.544229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:00:30.600799+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2","version":"132'102","reported_seq":171,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:42.006069+0000","last_change":"2024-05-28T06:24:24.012293+0000","last_active":"2024-05-28T06:26:42.006069+0000","last_peered":"2024-05-28T06:26:42.006069+0000","last_clean":"2024-05-28T06:26:42.006069+0000","last_became_active":"2024-05-28T06:24:24.011989+0000","last_became_peered":"2024-05-28T06:24:24.011989+0000","last_unstale":"2024-05-28T06:26:42.006069+0000","last_undegraded":"2024-05-28T06:26:42.006069+0000","last_fullsized":"2024-05-28T06:26:42.006069+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:47:31.291023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":352321536,"num_objects":84,"num_object_clones":0,"num_object_copies":168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84,"num_whiteouts":0,"num_read":52,"num_read_kb":212992,"num_write":90,"num_write_kb":356352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":20,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:29.921379+0000","last_change":"2024-05-28T06:24:23.008363+0000","last_active":"2024-05-28T06:26:29.921379+0000","last_peered":"2024-05-28T06:26:29.921379+0000","last_clean":"2024-05-28T06:26:29.921379+0000","last_became_active":"2024-05-28T06:24:23.007859+0000","last_became_peered":"2024-05-28T06:24:23.007859+0000","last_unstale":"2024-05-28T06:26:29.921379+0000","last_undegraded":"2024-05-28T06:26:29.921379+0000","last_fullsized":"2024-05-28T06:26:29.921379+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"132'106","reported_seq":206,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:42.014707+0000","last_change":"2024-05-28T06:24:24.013906+0000","last_active":"2024-05-28T06:26:42.014707+0000","last_peered":"2024-05-28T06:26:42.014707+0000","last_clean":"2024-05-28T06:26:42.014707+0000","last_became_active":"2024-05-28T06:24:24.011269+0000","last_became_peered":"2024-05-28T06:24:24.011269+0000","last_unstale":"2024-05-28T06:26:42.014707+0000","last_undegraded":"2024-05-28T06:26:42.014707+0000","last_fullsized":"2024-05-28T06:26:42.014707+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:09:03.314508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369098752,"num_objects":88,"num_object_clones":0,"num_object_copies":176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88,"num_whiteouts":0,"num_read":83,"num_read_kb":327683,"num_write":98,"num_write_kb":376833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":19,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:55.133578+0000","last_change":"2024-05-28T06:24:23.006728+0000","last_active":"2024-05-28T06:24:55.133578+0000","last_peered":"2024-05-28T06:24:55.133578+0000","last_clean":"2024-05-28T06:24:55.133578+0000","last_became_active":"2024-05-28T06:24:23.005895+0000","last_became_peered":"2024-05-28T06:24:23.005895+0000","last_unstale":"2024-05-28T06:24:55.133578+0000","last_undegraded":"2024-05-28T06:24:55.133578+0000","last_fullsized":"2024-05-28T06:24:55.133578+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:53:06.353639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"132'92","reported_seq":181,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:42.006649+0000","last_change":"2024-05-28T06:24:24.013072+0000","last_active":"2024-05-28T06:26:42.006649+0000","last_peered":"2024-05-28T06:26:42.006649+0000","last_clean":"2024-05-28T06:26:42.006649+0000","last_became_active":"2024-05-28T06:24:24.012814+0000","last_became_peered":"2024-05-28T06:24:24.012814+0000","last_unstale":"2024-05-28T06:26:42.006649+0000","last_undegraded":"2024-05-28T06:26:42.006649+0000","last_fullsized":"2024-05-28T06:26:42.006649+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":339738624,"num_objects":81,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81,"num_whiteouts":0,"num_read":71,"num_read_kb":290816,"num_write":87,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1","version":"132'31","reported_seq":81,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:29.975849+0000","last_change":"2024-05-28T06:24:23.001199+0000","last_active":"2024-05-28T06:26:29.975849+0000","last_peered":"2024-05-28T06:26:29.975849+0000","last_clean":"2024-05-28T06:26:29.975849+0000","last_became_active":"2024-05-28T06:24:23.000861+0000","last_became_peered":"2024-05-28T06:24:23.000861+0000","last_unstale":"2024-05-28T06:26:29.975849+0000","last_undegraded":"2024-05-28T06:26:29.975849+0000","last_fullsized":"2024-05-28T06:26:29.975849+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4553,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":30,"num_read_kb":30,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1","version":"132'100","reported_seq":194,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:42.012104+0000","last_change":"2024-05-28T06:24:24.005940+0000","last_active":"2024-05-28T06:26:42.012104+0000","last_peered":"2024-05-28T06:26:42.012104+0000","last_clean":"2024-05-28T06:26:42.012104+0000","last_became_active":"2024-05-28T06:24:24.005485+0000","last_became_peered":"2024-05-28T06:24:24.005485+0000","last_unstale":"2024-05-28T06:26:42.012104+0000","last_undegraded":"2024-05-28T06:26:42.012104+0000","last_fullsized":"2024-05-28T06:26:42.012104+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":314572800,"num_objects":75,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75,"num_whiteouts":0,"num_read":76,"num_read_kb":311296,"num_write":93,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":28,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:48.512890+0000","last_change":"2024-05-28T06:26:48.512938+0000","last_active":"2024-05-28T06:26:48.512890+0000","last_peered":"2024-05-28T06:26:48.512890+0000","last_clean":"2024-05-28T06:26:48.512890+0000","last_became_active":"2024-05-28T06:24:22.987181+0000","last_became_peered":"2024-05-28T06:24:22.987181+0000","last_unstale":"2024-05-28T06:26:48.512890+0000","last_undegraded":"2024-05-28T06:26:48.512890+0000","last_fullsized":"2024-05-28T06:26:48.512890+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:26:48.512762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:26:48.512762+0000","last_clean_scrub_stamp":"2024-05-28T06:26:48.512762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:47:34.449612+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.12","version":"132'70","reported_seq":134,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:42.014728+0000","last_change":"2024-05-28T06:24:24.007933+0000","last_active":"2024-05-28T06:26:42.014728+0000","last_peered":"2024-05-28T06:26:42.014728+0000","last_clean":"2024-05-28T06:26:42.014728+0000","last_became_active":"2024-05-28T06:24:24.007651+0000","last_became_peered":"2024-05-28T06:24:24.007651+0000","last_unstale":"2024-05-28T06:26:42.014728+0000","last_undegraded":"2024-05-28T06:26:42.014728+0000","last_fullsized":"2024-05-28T06:26:42.014728+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:22.952058+0000","last_clean_scrub_stamp":"2024-05-28T06:24:22.952058+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:59:32.549372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":255852544,"num_objects":61,"num_object_clones":0,"num_object_copies":122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":0,"num_read":47,"num_read_kb":192512,"num_write":65,"num_write_kb":258048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"129'2","reported_seq":21,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:55.133918+0000","last_change":"2024-05-28T06:24:23.008078+0000","last_active":"2024-05-28T06:24:55.133918+0000","last_peered":"2024-05-28T06:24:55.133918+0000","last_clean":"2024-05-28T06:24:55.133918+0000","last_became_active":"2024-05-28T06:24:23.007331+0000","last_became_peered":"2024-05-28T06:24:23.007331+0000","last_unstale":"2024-05-28T06:24:55.133918+0000","last_undegraded":"2024-05-28T06:24:55.133918+0000","last_fullsized":"2024-05-28T06:24:55.133918+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:24:21.936141+0000","last_clean_scrub_stamp":"2024-05-28T06:24:21.936141+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:50:39.661313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":388,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:24:55.134179+0000","last_change":"2024-05-28T06:04:01.039129+0000","last_active":"2024-05-28T06:24:55.134179+0000","last_peered":"2024-05-28T06:24:55.134179+0000","last_clean":"2024-05-28T06:24:55.134179+0000","last_became_active":"2024-05-28T06:04:01.033906+0000","last_became_peered":"2024-05-28T06:04:01.033906+0000","last_unstale":"2024-05-28T06:24:55.134179+0000","last_undegraded":"2024-05-28T06:24:55.134179+0000","last_fullsized":"2024-05-28T06:24:55.134179+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:03:59.864237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:03:59.864237+0000","last_clean_scrub_stamp":"2024-05-28T06:03:59.864237+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:17:36.440245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":18396218954,"num_objects":4395,"num_object_clones":1,"num_object_copies":8790,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4395,"num_whiteouts":1,"num_read":3946,"num_read_kb":15908926,"num_write":4949,"num_write_kb":18956331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":41943040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36792500224,"data_stored":36792437908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5458,"ondisk_log_size":5458,"up":128,"acting":128,"num_store_stats":8},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":1678837,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":30,"num_read_kb":30,"num_write":603,"num_write_kb":1882,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3424256,"data_stored":3357674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":552,"ondisk_log_size":552,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575125,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3952528,"kb_used_data":3813128,"kb_used_omap":1,"kb_used_meta":139390,"kb_avail":90419312,"statfs":{"total":96636764160,"available":92589375488,"internally_reserved":0,"allocated":3904643072,"data_stored":3903694641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":142735818},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575127,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5139788,"kb_used_data":5000388,"kb_used_omap":1,"kb_used_meta":139390,"kb_avail":89232052,"statfs":{"total":96636764160,"available":91373621248,"internally_reserved":0,"allocated":5120397312,"data_stored":5119456351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":142735814},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575125,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3815132,"kb_used_data":3675732,"kb_used_omap":1,"kb_used_meta":139390,"kb_avail":90556708,"statfs":{"total":96636764160,"available":92730068992,"internally_reserved":0,"allocated":3763949568,"data_stored":3763018100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":142735813},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575125,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5547692,"kb_used_data":5412452,"kb_used_omap":1,"kb_used_meta":135230,"kb_avail":88824148,"statfs":{"total":96636764160,"available":90955927552,"internally_reserved":0,"allocated":5542350848,"data_stored":5541404120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":138475973},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575127,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4527844,"kb_used_data":4388444,"kb_used_omap":1,"kb_used_meta":139390,"kb_avail":89843996,"statfs":{"total":96636764160,"available":92000251904,"internally_reserved":0,"allocated":4493766656,"data_stored":4492831645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":142735815},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575126,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5979432,"kb_used_data":5826144,"kb_used_omap":1,"kb_used_meta":153278,"kb_avail":88392408,"statfs":{"total":96636764160,"available":90513825792,"internally_reserved":0,"allocated":5965971456,"data_stored":5965028896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":156957125},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575126,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4155024,"kb_used_data":4019784,"kb_used_omap":1,"kb_used_meta":135230,"kb_avail":90216816,"statfs":{"total":96636764160,"available":92382019584,"internally_reserved":0,"allocated":4116258816,"data_stored":4115339392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":138475973},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575125,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3959192,"kb_used_data":3811472,"kb_used_omap":1,"kb_used_meta":147710,"kb_avail":90412648,"statfs":{"total":96636764160,"available":92582551552,"internally_reserved":0,"allocated":3902947328,"data_stored":3902020517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151255493},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1691648,"data_stored":1678039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1695744,"data_stored":1674124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3900706816,"data_stored":3900702981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4114612224,"data_stored":4114612224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":5964316672,"data_stored":5964301112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4492103680,"data_stored":4492099764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":5540700160,"data_stored":5540676882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3762294784,"data_stored":3762290820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":5117059072,"data_stored":5117051144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3900706816,"data_stored":3900702981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-28T06:26:52.687 INFO:tasks.ceph:pgid 19.3f last_scrub_stamp 2024-05-28T06:24:22.952058+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=22, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.687 INFO:tasks.ceph:pgid 19.3e last_scrub_stamp 2024-05-28T06:24:22.952058+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=22, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.687 INFO:tasks.ceph:pgid 19.3d last_scrub_stamp 2024-05-28T06:24:22.952058+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=22, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.687 INFO:tasks.ceph:pgid 19.3c last_scrub_stamp 2024-05-28T06:24:22.952058+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=22, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.688 INFO:tasks.ceph:pgid 19.3b last_scrub_stamp 2024-05-28T06:24:22.952058+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=22, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.688 INFO:tasks.ceph:pgid 19.3a last_scrub_stamp 2024-05-28T06:24:22.952058+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=22, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.688 INFO:tasks.ceph:pgid 19.39 last_scrub_stamp 2024-05-28T06:24:22.952058+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=22, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.688 INFO:tasks.ceph:pgid 19.38 last_scrub_stamp 2024-05-28T06:24:22.952058+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=22, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.688 INFO:tasks.ceph:pgid 19.37 last_scrub_stamp 2024-05-28T06:24:22.952058+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=22, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.689 INFO:tasks.ceph:pgid 19.36 last_scrub_stamp 2024-05-28T06:24:22.952058+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=22, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.689 INFO:tasks.ceph:pgid 19.35 last_scrub_stamp 2024-05-28T06:24:22.952058+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=22, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.689 INFO:tasks.ceph:pgid 19.34 last_scrub_stamp 2024-05-28T06:24:22.952058+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=22, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.689 INFO:tasks.ceph:pgid 19.33 last_scrub_stamp 2024-05-28T06:24:22.952058+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=22, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.690 INFO:tasks.ceph:pgid 19.32 last_scrub_stamp 2024-05-28T06:24:22.952058+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=22, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.690 INFO:tasks.ceph:pgid 19.31 last_scrub_stamp 2024-05-28T06:24:22.952058+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=22, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.690 INFO:tasks.ceph:pgid 19.30 last_scrub_stamp 2024-05-28T06:24:22.952058+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=22, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.690 INFO:tasks.ceph:pgid 19.2f last_scrub_stamp 2024-05-28T06:24:22.952058+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=22, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.690 INFO:tasks.ceph:pgid 19.2e last_scrub_stamp 2024-05-28T06:24:22.952058+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=22, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.690 INFO:tasks.ceph:pgid 19.2d last_scrub_stamp 2024-05-28T06:24:22.952058+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=22, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.691 INFO:tasks.ceph:pgid 19.2c last_scrub_stamp 2024-05-28T06:24:22.952058+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=22, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.691 INFO:tasks.ceph:pgid 19.2b last_scrub_stamp 2024-05-28T06:24:22.952058+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=22, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.691 INFO:tasks.ceph:pgid 19.2a last_scrub_stamp 2024-05-28T06:24:22.952058+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=22, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.691 INFO:tasks.ceph:pgid 19.29 last_scrub_stamp 2024-05-28T06:24:22.952058+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=22, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.691 INFO:tasks.ceph:pgid 19.28 last_scrub_stamp 2024-05-28T06:24:22.952058+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=22, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.691 INFO:tasks.ceph:pgid 19.27 last_scrub_stamp 2024-05-28T06:24:22.952058+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=22, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.691 INFO:tasks.ceph:pgid 19.26 last_scrub_stamp 2024-05-28T06:24:22.952058+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=22, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.692 INFO:tasks.ceph:pgid 19.25 last_scrub_stamp 2024-05-28T06:24:22.952058+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=22, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.692 INFO:tasks.ceph:pgid 19.24 last_scrub_stamp 2024-05-28T06:24:22.952058+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=22, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.692 INFO:tasks.ceph:pgid 19.23 last_scrub_stamp 2024-05-28T06:24:22.952058+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=22, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.692 INFO:tasks.ceph:pgid 19.22 last_scrub_stamp 2024-05-28T06:24:22.952058+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=22, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.692 INFO:tasks.ceph:pgid 19.21 last_scrub_stamp 2024-05-28T06:24:22.952058+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=22, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.692 INFO:tasks.ceph:pgid 19.20 last_scrub_stamp 2024-05-28T06:24:22.952058+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=22, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.693 INFO:tasks.ceph:pgid 19.1e last_scrub_stamp 2024-05-28T06:24:22.952058+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=22, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.693 INFO:tasks.ceph:pgid 18.1f last_scrub_stamp 2024-05-28T06:24:21.936141+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=21, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.693 INFO:tasks.ceph:pgid 19.1f last_scrub_stamp 2024-05-28T06:24:22.952058+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=22, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.693 INFO:tasks.ceph:pgid 18.1e last_scrub_stamp 2024-05-28T06:24:21.936141+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=21, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.693 INFO:tasks.ceph:pgid 19.1c last_scrub_stamp 2024-05-28T06:24:22.952058+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=22, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.693 INFO:tasks.ceph:pgid 19.1d last_scrub_stamp 2024-05-28T06:24:22.952058+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=22, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.694 INFO:tasks.ceph:pgid 19.1a last_scrub_stamp 2024-05-28T06:24:22.952058+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=22, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.694 INFO:tasks.ceph:pgid 19.1b last_scrub_stamp 2024-05-28T06:24:22.952058+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=22, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.694 INFO:tasks.ceph:pgid 19.18 last_scrub_stamp 2024-05-28T06:24:22.952058+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=22, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.694 INFO:tasks.ceph:pgid 18.19 last_scrub_stamp 2024-05-28T06:24:21.936141+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=21, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.695 INFO:tasks.ceph:pgid 19.19 last_scrub_stamp 2024-05-28T06:24:22.952058+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=22, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.695 INFO:tasks.ceph:pgid 18.18 last_scrub_stamp 2024-05-28T06:24:21.936141+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=21, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.695 INFO:tasks.ceph:pgid 19.16 last_scrub_stamp 2024-05-28T06:24:22.952058+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=22, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.695 INFO:tasks.ceph:pgid 18.17 last_scrub_stamp 2024-05-28T06:24:21.936141+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=21, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.695 INFO:tasks.ceph:pgid 19.17 last_scrub_stamp 2024-05-28T06:24:22.952058+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=22, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.695 INFO:tasks.ceph:pgid 18.16 last_scrub_stamp 2024-05-28T06:24:21.936141+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=21, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.696 INFO:tasks.ceph:pgid 19.14 last_scrub_stamp 2024-05-28T06:24:22.952058+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=22, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.696 INFO:tasks.ceph:pgid 18.15 last_scrub_stamp 2024-05-28T06:24:21.936141+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=21, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.696 INFO:tasks.ceph:pgid 19.15 last_scrub_stamp 2024-05-28T06:24:22.952058+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=22, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.696 INFO:tasks.ceph:pgid 18.14 last_scrub_stamp 2024-05-28T06:24:21.936141+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=21, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.696 INFO:tasks.ceph:pgid 19.13 last_scrub_stamp 2024-05-28T06:24:22.952058+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=22, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.696 INFO:tasks.ceph:pgid 19.10 last_scrub_stamp 2024-05-28T06:24:22.952058+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=22, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.697 INFO:tasks.ceph:pgid 18.11 last_scrub_stamp 2024-05-28T06:24:21.936141+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=21, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.697 INFO:tasks.ceph:pgid 19.11 last_scrub_stamp 2024-05-28T06:24:22.952058+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=22, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.697 INFO:tasks.ceph:pgid 18.10 last_scrub_stamp 2024-05-28T06:24:21.936141+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=21, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.697 INFO:tasks.ceph:pgid 19.e last_scrub_stamp 2024-05-28T06:24:22.952058+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=22, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.697 INFO:tasks.ceph:pgid 18.f last_scrub_stamp 2024-05-28T06:24:21.936141+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=21, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.697 INFO:tasks.ceph:pgid 19.f last_scrub_stamp 2024-05-28T06:24:22.952058+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=22, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.698 INFO:tasks.ceph:pgid 18.e last_scrub_stamp 2024-05-28T06:24:21.936141+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=21, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.698 INFO:tasks.ceph:pgid 19.c last_scrub_stamp 2024-05-28T06:24:22.952058+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=22, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.698 INFO:tasks.ceph:pgid 19.d last_scrub_stamp 2024-05-28T06:24:22.952058+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=22, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.698 INFO:tasks.ceph:pgid 18.c last_scrub_stamp 2024-05-28T06:24:21.936141+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=21, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.698 INFO:tasks.ceph:pgid 19.a last_scrub_stamp 2024-05-28T06:24:22.952058+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=22, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.698 INFO:tasks.ceph:pgid 18.b last_scrub_stamp 2024-05-28T06:24:21.936141+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=21, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.699 INFO:tasks.ceph:pgid 19.b last_scrub_stamp 2024-05-28T06:24:22.952058+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=22, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.699 INFO:tasks.ceph:pgid 18.a last_scrub_stamp 2024-05-28T06:24:21.936141+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=21, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.699 INFO:tasks.ceph:pgid 19.8 last_scrub_stamp 2024-05-28T06:24:22.952058+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=22, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.699 INFO:tasks.ceph:pgid 18.9 last_scrub_stamp 2024-05-28T06:24:21.936141+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=21, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.699 INFO:tasks.ceph:pgid 19.9 last_scrub_stamp 2024-05-28T06:24:22.952058+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=22, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.699 INFO:tasks.ceph:pgid 18.8 last_scrub_stamp 2024-05-28T06:24:21.936141+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=21, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.700 INFO:tasks.ceph:pgid 19.6 last_scrub_stamp 2024-05-28T06:24:22.952058+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=22, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.700 INFO:tasks.ceph:pgid 19.7 last_scrub_stamp 2024-05-28T06:24:22.952058+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=22, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.700 INFO:tasks.ceph:pgid 18.6 last_scrub_stamp 2024-05-28T06:24:21.936141+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=21, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.700 INFO:tasks.ceph:pgid 19.4 last_scrub_stamp 2024-05-28T06:24:22.952058+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=22, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.700 INFO:tasks.ceph:pgid 18.5 last_scrub_stamp 2024-05-28T06:24:21.936141+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=21, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.700 INFO:tasks.ceph:pgid 19.5 last_scrub_stamp 2024-05-28T06:24:22.952058+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=22, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.701 INFO:tasks.ceph:pgid 19.2 last_scrub_stamp 2024-05-28T06:24:22.952058+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=22, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.701 INFO:tasks.ceph:pgid 18.3 last_scrub_stamp 2024-05-28T06:24:21.936141+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=21, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.701 INFO:tasks.ceph:pgid 19.3 last_scrub_stamp 2024-05-28T06:24:22.952058+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=22, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.701 INFO:tasks.ceph:pgid 18.2 last_scrub_stamp 2024-05-28T06:24:21.936141+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=21, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.701 INFO:tasks.ceph:pgid 19.0 last_scrub_stamp 2024-05-28T06:24:22.952058+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=22, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.701 INFO:tasks.ceph:pgid 18.1 last_scrub_stamp 2024-05-28T06:24:21.936141+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=21, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.702 INFO:tasks.ceph:pgid 19.1 last_scrub_stamp 2024-05-28T06:24:22.952058+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=22, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.702 INFO:tasks.ceph:pgid 19.12 last_scrub_stamp 2024-05-28T06:24:22.952058+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=22, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.702 INFO:tasks.ceph:pgid 18.13 last_scrub_stamp 2024-05-28T06:24:21.936141+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=24, tm_sec=21, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.702 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-05-28T06:03:59.864237+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=3, tm_sec=59, tm_wday=1, tm_yday=149, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=28, tm_hour=6, tm_min=26, tm_sec=45, tm_wday=1, tm_yday=149, tm_isdst=0) 2024-05-28T06:26:52.702 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-05-28T06:27:05.602 DEBUG:teuthology.orchestra.run.smithi096:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:27:05.611 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:27:12.703 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-28T06:27:12.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:12.924+0000 7f438c600640 1 Processor -- start 2024-05-28T06:27:12.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:12.924+0000 7f438c600640 1 -- start start 2024-05-28T06:27:12.917 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:12.925+0000 7f438c600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f438406f860 0x7f43840b3150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:27:12.917 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:12.925+0000 7f438c600640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f438406ed10 con 0x7f43840d0bb0 2024-05-28T06:27:12.917 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:12.925+0000 7f438c600640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f438415cb00 con 0x7f43840b3690 2024-05-28T06:27:12.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:12.925+0000 7f438c600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f43840b3cc0 con 0x7f438406f860 2024-05-28T06:27:12.919 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:12.925+0000 7f438a600640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f43840d0bb0 0x7f43840ad130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:33356/0 (socket says 172.21.15.96:33356) 2024-05-28T06:27:12.920 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:12.925+0000 7f438a600640 1 -- 172.21.15.96:0/468925853 learned_addr learned my addr 172.21.15.96:0/468925853 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-05-28T06:27:12.920 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:12.926+0000 7f4383e00640 1 -- 172.21.15.96:0/468925853 <== mon.0 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 402555260 0 0) 0x7f437c001490 con 0x7f43840d0bb0 2024-05-28T06:27:12.920 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:12.926+0000 7f4389c00640 1 --2- 172.21.15.96:0/468925853 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f438406f860 0x7f43840b3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:27:12.920 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:12.926+0000 7f4383e00640 1 -- 172.21.15.96:0/468925853 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f436c003520 con 0x7f43840d0bb0 2024-05-28T06:27:12.920 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:12.926+0000 7f4383e00640 1 -- 172.21.15.96:0/468925853 <== mon.0 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 61779637 0 0) 0x7f436c003520 con 0x7f43840d0bb0 2024-05-28T06:27:12.920 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:12.926+0000 7f4383e00640 1 -- 172.21.15.96:0/468925853 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f438406f860 msgr2=0x7f43840b3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:27:12.920 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:12.926+0000 7f4383e00640 1 --2- 172.21.15.96:0/468925853 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f438406f860 0x7f43840b3150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:27:12.921 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:12.926+0000 7f4383e00640 1 -- 172.21.15.96:0/468925853 >> v1:172.21.15.123:6789/0 conn(0x7f43840b3690 legacy=0x7f43840d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:27:12.921 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:12.926+0000 7f4383e00640 1 -- 172.21.15.96:0/468925853 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43840b3fa0 con 0x7f43840d0bb0 2024-05-28T06:27:12.921 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:12.926+0000 7f4383e00640 1 -- 172.21.15.96:0/468925853 <== mon.0 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2419889326 0 0) 0x7f437c001bc0 con 0x7f43840d0bb0 2024-05-28T06:27:12.921 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:12.926+0000 7f4383e00640 1 -- 172.21.15.96:0/468925853 >> v1:172.21.15.96:6789/0 conn(0x7f43840d0bb0 legacy=0x7f43840ad130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:27:12.921 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:12.926+0000 7f4383e00640 1 --2- 172.21.15.96:0/468925853 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f438406f860 0x7f436c0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:27:12.921 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:12.926+0000 7f4383e00640 1 --2- 172.21.15.96:0/468925853 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f436c004060 0x7f436c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:27:12.921 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:12.926+0000 7f4383e00640 1 --2- 172.21.15.96:0/468925853 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f436c004d40 0x7f436c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:27:12.921 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:12.926+0000 7f4383e00640 1 -- 172.21.15.96:0/468925853 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f43840b3cc0 con 0x7f436c004d40 2024-05-28T06:27:12.922 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:12.926+0000 7f4383e00640 1 -- 172.21.15.96:0/468925853 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f436c00bc00 con 0x7f436c004060 2024-05-28T06:27:12.922 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:12.926+0000 7f4383e00640 1 -- 172.21.15.96:0/468925853 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f436c00be00 con 0x7f438406f860 2024-05-28T06:27:12.922 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:12.927+0000 7f438a600640 1 --2- 172.21.15.96:0/468925853 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f436c004d40 0x7f436c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:27:12.922 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:12.927+0000 7f4389200640 1 --2- 172.21.15.96:0/468925853 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f436c004060 0x7f436c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:27:12.922 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:12.927+0000 7f4389c00640 1 --2- 172.21.15.96:0/468925853 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f438406f860 0x7f436c0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:27:12.922 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:12.927+0000 7f438a600640 1 -- 172.21.15.96:0/468925853 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f438406f860 msgr2=0x7f436c0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:27:12.922 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:12.927+0000 7f438a600640 1 --2- 172.21.15.96:0/468925853 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f438406f860 0x7f436c0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:27:12.922 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:12.927+0000 7f438a600640 1 -- 172.21.15.96:0/468925853 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f436c004060 msgr2=0x7f436c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:27:12.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:12.927+0000 7f438a600640 1 --2- 172.21.15.96:0/468925853 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f436c004060 0x7f436c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:27:12.923 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:12.927+0000 7f438a600640 1 -- 172.21.15.96:0/468925853 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f436c003520 con 0x7f436c004d40 2024-05-28T06:27:12.924 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:12.928+0000 7f438a600640 1 --2- 172.21.15.96:0/468925853 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f436c004d40 0x7f436c0051c0 secure :-1 s=READY pgs=2233 cs=0 l=1 rev1=1 crypto rx=0x7f43840d1360 tx=0x7f436c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a55acb5d4261fcf server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:27:12.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:12.929+0000 7f4383e00640 1 -- 172.21.15.96:0/468925853 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f437c00a580 con 0x7f436c004d40 2024-05-28T06:27:12.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:12.929+0000 7f4383e00640 1 -- 172.21.15.96:0/468925853 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f437c02f070 con 0x7f436c004d40 2024-05-28T06:27:12.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:12.929+0000 7f4383e00640 1 -- 172.21.15.96:0/468925853 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f437c02bdb0 con 0x7f436c004d40 2024-05-28T06:27:12.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:12.929+0000 7f438c600640 1 -- 172.21.15.96:0/468925853 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f436c004d40 msgr2=0x7f436c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:27:12.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:12.929+0000 7f438c600640 1 --2- 172.21.15.96:0/468925853 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f436c004d40 0x7f436c0051c0 secure :-1 s=READY pgs=2233 cs=0 l=1 rev1=1 crypto rx=0x7f43840d1360 tx=0x7f436c005830 comp rx=0 tx=0).stop 2024-05-28T06:27:12.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:12.929+0000 7f438a600640 1 -- 172.21.15.96:0/468925853 reap_dead start 2024-05-28T06:27:12.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:12.929+0000 7f438c600640 1 -- 172.21.15.96:0/468925853 shutdown_connections 2024-05-28T06:27:12.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:12.929+0000 7f438c600640 1 -- 172.21.15.96:0/468925853 >> 172.21.15.96:0/468925853 conn(0x7f438406bae0 msgr2=0x7f438406bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:27:12.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:12.929+0000 7f438c600640 1 -- 172.21.15.96:0/468925853 shutdown_connections 2024-05-28T06:27:12.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:12.929+0000 7f438c600640 1 -- 172.21.15.96:0/468925853 wait complete. 2024-05-28T06:27:12.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:12.931+0000 7f438c600640 1 Processor -- start 2024-05-28T06:27:12.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:12.931+0000 7f438c600640 1 -- start start 2024-05-28T06:27:12.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:12.932+0000 7f438c600640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f436c004d40 0x7f4384153140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:27:12.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:12.932+0000 7f438c600640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f43840ce0a0 0x7f43841f41c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:27:12.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:12.932+0000 7f438c600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f43840ce4e0 0x7f43841f4700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:27:12.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:12.932+0000 7f438c600640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f43840b4840 con 0x7f43840ce4e0 2024-05-28T06:27:12.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:12.932+0000 7f438c600640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f43840b4560 con 0x7f436c004d40 2024-05-28T06:27:12.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:12.932+0000 7f438c600640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f43840b3150 con 0x7f43840ce0a0 2024-05-28T06:27:12.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:12.932+0000 7f438a600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f43840ce4e0 0x7f43841f4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:27:12.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:12.933+0000 7f438a600640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f43840ce4e0 0x7f43841f4700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:52182/0 (socket says 172.21.15.96:52182) 2024-05-28T06:27:12.927 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:12.933+0000 7f438a600640 1 -- 172.21.15.96:0/3716081185 learned_addr learned my addr 172.21.15.96:0/3716081185 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:27:12.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:12.933+0000 7f4389c00640 1 --2- 172.21.15.96:0/3716081185 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f436c004d40 0x7f4384153140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:27:12.928 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:12.933+0000 7f438a600640 1 -- 172.21.15.96:0/3716081185 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f43840ce0a0 msgr2=0x7f43841f41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:27:12.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:12.933+0000 7f438a600640 1 --2- 172.21.15.96:0/3716081185 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f43840ce0a0 0x7f43841f41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:27:12.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:12.933+0000 7f438a600640 1 -- 172.21.15.96:0/3716081185 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f436c004d40 msgr2=0x7f4384153140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:27:12.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:12.933+0000 7f438a600640 1 --2- 172.21.15.96:0/3716081185 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f436c004d40 0x7f4384153140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:27:12.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:12.933+0000 7f438a600640 1 -- 172.21.15.96:0/3716081185 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43840cd340 con 0x7f43840ce4e0 2024-05-28T06:27:12.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:12.933+0000 7f438a600640 1 --2- 172.21.15.96:0/3716081185 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f43840ce4e0 0x7f43841f4700 secure :-1 s=READY pgs=2234 cs=0 l=1 rev1=1 crypto rx=0x7f437c00aa20 tx=0x7f43840b4170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:27:12.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:12.934+0000 7f4381600640 1 -- 172.21.15.96:0/3716081185 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f437c044070 con 0x7f43840ce4e0 2024-05-28T06:27:12.929 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:12.934+0000 7f4381600640 1 -- 172.21.15.96:0/3716081185 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f437c001c30 con 0x7f43840ce4e0 2024-05-28T06:27:12.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:12.934+0000 7f4381600640 1 -- 172.21.15.96:0/3716081185 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f437c00a580 con 0x7f43840ce4e0 2024-05-28T06:27:12.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:12.934+0000 7f438c600640 1 -- 172.21.15.96:0/3716081185 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43840c4aa0 con 0x7f43840ce4e0 2024-05-28T06:27:12.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:12.934+0000 7f438c600640 1 -- 172.21.15.96:0/3716081185 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43840cd5c0 con 0x7f43840ce4e0 2024-05-28T06:27:12.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:12.935+0000 7f438c600640 1 -- 172.21.15.96:0/3716081185 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43840689e0 con 0x7f43840ce4e0 2024-05-28T06:27:12.930 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:12.936+0000 7f4381600640 1 -- 172.21.15.96:0/3716081185 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 51) ==== 93743+0+0 (secure 0 0 0) 0x7f437c047070 con 0x7f43840ce4e0 2024-05-28T06:27:12.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:12.937+0000 7f4381600640 1 --2- 172.21.15.96:0/3716081185 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f436406e6f0 0x7f4364070bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:27:12.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:12.937+0000 7f4381600640 1 -- 172.21.15.96:0/3716081185 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 5823+0+0 (secure 0 0 0) 0x7f437c0aad20 con 0x7f43840ce4e0 2024-05-28T06:27:12.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:12.938+0000 7f4389c00640 1 --2- 172.21.15.96:0/3716081185 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f436406e6f0 0x7f4364070bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:27:12.931 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:12.939+0000 7f4389c00640 1 --2- 172.21.15.96:0/3716081185 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f436406e6f0 0x7f4364070bb0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f4378006fd0 tx=0x7f4378008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:27:12.934 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:12.942+0000 7f4381600640 1 -- 172.21.15.96:0/3716081185 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f43840cd340 con 0x7f43840ce4e0 2024-05-28T06:27:13.131 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.139+0000 7f438c600640 1 -- 172.21.15.96:0/3716081185 --> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f43840af7d0 con 0x7f436406e6f0 2024-05-28T06:27:13.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.162+0000 7f4381600640 1 -- 172.21.15.96:0/3716081185 <== mgr.4105 v2:172.21.15.123:6832/259466469 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+255821 (secure 0 0 0) 0x7f43840af7d0 con 0x7f436406e6f0 2024-05-28T06:27:13.156 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:27:13.160 INFO:teuthology.orchestra.run.smithi096.stderr:dumped all 2024-05-28T06:27:13.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.168+0000 7f438c600640 1 -- 172.21.15.96:0/3716081185 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f436406e6f0 msgr2=0x7f4364070bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:27:13.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.168+0000 7f438c600640 1 --2- 172.21.15.96:0/3716081185 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f436406e6f0 0x7f4364070bb0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f4378006fd0 tx=0x7f4378008040 comp rx=0 tx=0).stop 2024-05-28T06:27:13.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.168+0000 7f438c600640 1 -- 172.21.15.96:0/3716081185 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f43840ce4e0 msgr2=0x7f43841f4700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:27:13.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.168+0000 7f438c600640 1 --2- 172.21.15.96:0/3716081185 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f43840ce4e0 0x7f43841f4700 secure :-1 s=READY pgs=2234 cs=0 l=1 rev1=1 crypto rx=0x7f437c00aa20 tx=0x7f43840b4170 comp rx=0 tx=0).stop 2024-05-28T06:27:13.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.169+0000 7f438c600640 1 -- 172.21.15.96:0/3716081185 shutdown_connections 2024-05-28T06:27:13.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.169+0000 7f438c600640 1 --2- 172.21.15.96:0/3716081185 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f436406e6f0 0x7f4364070bb0 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:27:13.162 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.169+0000 7f438c600640 1 --2- 172.21.15.96:0/3716081185 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f43840ce4e0 0x7f43841f4700 unknown :-1 s=CLOSED pgs=2234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:27:13.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.169+0000 7f438c600640 1 --2- 172.21.15.96:0/3716081185 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f43840ce0a0 0x7f43841f41c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:27:13.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.169+0000 7f438c600640 1 --2- 172.21.15.96:0/3716081185 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f436c004d40 0x7f4384153140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:27:13.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.169+0000 7f438c600640 1 -- 172.21.15.96:0/3716081185 >> 172.21.15.96:0/3716081185 conn(0x7f438406bae0 msgr2=0x7f43840b0010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:27:13.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.169+0000 7f438c600640 1 -- 172.21.15.96:0/3716081185 shutdown_connections 2024-05-28T06:27:13.163 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.169+0000 7f438c600640 1 -- 172.21.15.96:0/3716081185 wait complete. 2024-05-28T06:27:13.179 INFO:teuthology.orchestra.run.smithi096.stdout:{"pg_ready":true,"pg_map":{"version":821,"stamp":"2024-05-28T06:27:12.092024+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":18398489717,"num_objects":4420,"num_object_clones":1,"num_object_copies":8840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4420,"num_whiteouts":1,"num_read":4092,"num_read_kb":15909056,"num_write":5727,"num_write_kb":18960641,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":41943040,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5930,"num_omap_keys":26,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":13554,"ondisk_log_size":13554,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":37097432,"kb_used_data":35947544,"kb_used_omap":12,"kb_used_meta":1149811,"kb_avail":717877288,"statfs":{"total":773094113280,"available":735106342912,"internally_reserved":0,"allocated":36810285056,"data_stored":36802796778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":1177407024},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004767"},"pg_stats":[{"pgid":"19.3f","version":"132'201","reported_seq":300,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:54.807783+0000","last_change":"2024-05-28T06:26:54.807855+0000","last_active":"2024-05-28T06:26:54.807783+0000","last_peered":"2024-05-28T06:26:54.807783+0000","last_clean":"2024-05-28T06:26:54.807783+0000","last_became_active":"2024-05-28T06:24:24.006988+0000","last_became_peered":"2024-05-28T06:24:24.006988+0000","last_unstale":"2024-05-28T06:26:54.807783+0000","last_undegraded":"2024-05-28T06:26:54.807783+0000","last_fullsized":"2024-05-28T06:26:54.807783+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'201","last_scrub_stamp":"2024-05-28T06:26:54.807653+0000","last_deep_scrub":"132'201","last_deep_scrub_stamp":"2024-05-28T06:26:54.807653+0000","last_clean_scrub_stamp":"2024-05-28T06:26:54.807653+0000","objects_scrubbed":62,"log_size":201,"log_dups_size":0,"ondisk_log_size":201,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:34:41.679571+0000","scrub_duration":1225,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260046848,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":70,"num_read_kb":286720,"num_write":72,"num_write_kb":274432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3e","version":"132'225","reported_seq":323,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:55.285755+0000","last_change":"2024-05-28T06:26:55.285799+0000","last_active":"2024-05-28T06:26:55.285755+0000","last_peered":"2024-05-28T06:26:55.285755+0000","last_clean":"2024-05-28T06:26:55.285755+0000","last_became_active":"2024-05-28T06:24:23.986471+0000","last_became_peered":"2024-05-28T06:24:23.986471+0000","last_unstale":"2024-05-28T06:26:55.285755+0000","last_undegraded":"2024-05-28T06:26:55.285755+0000","last_fullsized":"2024-05-28T06:26:55.285755+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'225","last_scrub_stamp":"2024-05-28T06:26:55.285669+0000","last_deep_scrub":"132'225","last_deep_scrub_stamp":"2024-05-28T06:26:55.285669+0000","last_clean_scrub_stamp":"2024-05-28T06:26:55.285669+0000","objects_scrubbed":78,"log_size":225,"log_dups_size":0,"ondisk_log_size":225,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:05:09.304282+0000","scrub_duration":1151,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327155712,"num_objects":78,"num_object_clones":0,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78,"num_whiteouts":0,"num_read":68,"num_read_kb":278528,"num_write":84,"num_write_kb":331776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3d","version":"132'217","reported_seq":316,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:55.994739+0000","last_change":"2024-05-28T06:26:55.994883+0000","last_active":"2024-05-28T06:26:55.994739+0000","last_peered":"2024-05-28T06:26:55.994739+0000","last_clean":"2024-05-28T06:26:55.994739+0000","last_became_active":"2024-05-28T06:24:24.007416+0000","last_became_peered":"2024-05-28T06:24:24.007416+0000","last_unstale":"2024-05-28T06:26:55.994739+0000","last_undegraded":"2024-05-28T06:26:55.994739+0000","last_fullsized":"2024-05-28T06:26:55.994739+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'217","last_scrub_stamp":"2024-05-28T06:26:55.994642+0000","last_deep_scrub":"132'217","last_deep_scrub_stamp":"2024-05-28T06:26:55.994642+0000","last_clean_scrub_stamp":"2024-05-28T06:26:55.994642+0000","objects_scrubbed":73,"log_size":217,"log_dups_size":0,"ondisk_log_size":217,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:55:13.661762+0000","scrub_duration":1217,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306184192,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":70,"num_read_kb":253960,"num_write":93,"num_write_kb":323588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.3c","version":"132'202","reported_seq":275,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:57.370059+0000","last_change":"2024-05-28T06:26:57.370171+0000","last_active":"2024-05-28T06:26:57.370059+0000","last_peered":"2024-05-28T06:26:57.370059+0000","last_clean":"2024-05-28T06:26:57.370059+0000","last_became_active":"2024-05-28T06:24:24.011234+0000","last_became_peered":"2024-05-28T06:24:24.011234+0000","last_unstale":"2024-05-28T06:26:57.370059+0000","last_undegraded":"2024-05-28T06:26:57.370059+0000","last_fullsized":"2024-05-28T06:26:57.370059+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'202","last_scrub_stamp":"2024-05-28T06:26:57.369962+0000","last_deep_scrub":"132'202","last_deep_scrub_stamp":"2024-05-28T06:26:57.369962+0000","last_clean_scrub_stamp":"2024-05-28T06:26:57.369962+0000","objects_scrubbed":67,"log_size":202,"log_dups_size":0,"ondisk_log_size":202,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:02:06.967276+0000","scrub_duration":793,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281018368,"num_objects":67,"num_object_clones":0,"num_object_copies":134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":0,"num_read":45,"num_read_kb":176130,"num_write":71,"num_write_kb":278529,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3b","version":"132'168","reported_seq":234,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:27:05.865125+0000","last_change":"2024-05-28T06:27:05.865230+0000","last_active":"2024-05-28T06:27:05.865125+0000","last_peered":"2024-05-28T06:27:05.865125+0000","last_clean":"2024-05-28T06:27:05.865125+0000","last_became_active":"2024-05-28T06:24:24.018334+0000","last_became_peered":"2024-05-28T06:24:24.018334+0000","last_unstale":"2024-05-28T06:27:05.865125+0000","last_undegraded":"2024-05-28T06:27:05.865125+0000","last_fullsized":"2024-05-28T06:27:05.865125+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'168","last_scrub_stamp":"2024-05-28T06:27:05.864952+0000","last_deep_scrub":"132'168","last_deep_scrub_stamp":"2024-05-28T06:27:05.864952+0000","last_clean_scrub_stamp":"2024-05-28T06:27:05.864952+0000","objects_scrubbed":55,"log_size":168,"log_dups_size":0,"ondisk_log_size":168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:36:54.606051+0000","scrub_duration":638,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":230686720,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":38,"num_read_kb":155648,"num_write":65,"num_write_kb":245760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.3a","version":"132'238","reported_seq":370,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:27:00.248132+0000","last_change":"2024-05-28T06:27:00.248196+0000","last_active":"2024-05-28T06:27:00.248132+0000","last_peered":"2024-05-28T06:27:00.248132+0000","last_clean":"2024-05-28T06:27:00.248132+0000","last_became_active":"2024-05-28T06:24:54.990013+0000","last_became_peered":"2024-05-28T06:24:54.990013+0000","last_unstale":"2024-05-28T06:27:00.248132+0000","last_undegraded":"2024-05-28T06:27:00.248132+0000","last_fullsized":"2024-05-28T06:27:00.248132+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'238","last_scrub_stamp":"2024-05-28T06:27:00.248043+0000","last_deep_scrub":"132'238","last_deep_scrub_stamp":"2024-05-28T06:27:00.248043+0000","last_clean_scrub_stamp":"2024-05-28T06:27:00.248043+0000","objects_scrubbed":87,"log_size":238,"log_dups_size":0,"ondisk_log_size":238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:56:59.518694+0000","scrub_duration":1123,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":360710405,"num_objects":87,"num_object_clones":0,"num_object_copies":174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87,"num_whiteouts":0,"num_read":84,"num_read_kb":339969,"num_write":97,"num_write_kb":368642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":25165824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.39","version":"132'186","reported_seq":259,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:56.635777+0000","last_change":"2024-05-28T06:26:56.635814+0000","last_active":"2024-05-28T06:26:56.635777+0000","last_peered":"2024-05-28T06:26:56.635777+0000","last_clean":"2024-05-28T06:26:56.635777+0000","last_became_active":"2024-05-28T06:24:24.005839+0000","last_became_peered":"2024-05-28T06:24:24.005839+0000","last_unstale":"2024-05-28T06:26:56.635777+0000","last_undegraded":"2024-05-28T06:26:56.635777+0000","last_fullsized":"2024-05-28T06:26:56.635777+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'186","last_scrub_stamp":"2024-05-28T06:26:56.635697+0000","last_deep_scrub":"132'186","last_deep_scrub_stamp":"2024-05-28T06:26:56.635697+0000","last_clean_scrub_stamp":"2024-05-28T06:26:56.635697+0000","objects_scrubbed":53,"log_size":186,"log_dups_size":0,"ondisk_log_size":186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:58:56.639700+0000","scrub_duration":573,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222298112,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":45,"num_read_kb":184320,"num_write":65,"num_write_kb":241664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.38","version":"132'189","reported_seq":264,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:57.627182+0000","last_change":"2024-05-28T06:26:57.627300+0000","last_active":"2024-05-28T06:26:57.627182+0000","last_peered":"2024-05-28T06:26:57.627182+0000","last_clean":"2024-05-28T06:26:57.627182+0000","last_became_active":"2024-05-28T06:24:24.008334+0000","last_became_peered":"2024-05-28T06:24:24.008334+0000","last_unstale":"2024-05-28T06:26:57.627182+0000","last_undegraded":"2024-05-28T06:26:57.627182+0000","last_fullsized":"2024-05-28T06:26:57.627182+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'189","last_scrub_stamp":"2024-05-28T06:26:57.627043+0000","last_deep_scrub":"132'189","last_deep_scrub_stamp":"2024-05-28T06:26:57.627043+0000","last_clean_scrub_stamp":"2024-05-28T06:26:57.627043+0000","objects_scrubbed":71,"log_size":189,"log_dups_size":0,"ondisk_log_size":189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:56:49.013193+0000","scrub_duration":804,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":297795584,"num_objects":71,"num_object_clones":0,"num_object_copies":142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":47,"num_read_kb":192512,"num_write":79,"num_write_kb":307200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.37","version":"132'212","reported_seq":293,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:27:01.726327+0000","last_change":"2024-05-28T06:27:01.726434+0000","last_active":"2024-05-28T06:27:01.726327+0000","last_peered":"2024-05-28T06:27:01.726327+0000","last_clean":"2024-05-28T06:27:01.726327+0000","last_became_active":"2024-05-28T06:24:24.017929+0000","last_became_peered":"2024-05-28T06:24:24.017929+0000","last_unstale":"2024-05-28T06:27:01.726327+0000","last_undegraded":"2024-05-28T06:27:01.726327+0000","last_fullsized":"2024-05-28T06:27:01.726327+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'212","last_scrub_stamp":"2024-05-28T06:27:01.726227+0000","last_deep_scrub":"132'212","last_deep_scrub_stamp":"2024-05-28T06:27:01.726227+0000","last_clean_scrub_stamp":"2024-05-28T06:27:01.726227+0000","objects_scrubbed":72,"log_size":212,"log_dups_size":0,"ondisk_log_size":212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:26:41.497065+0000","scrub_duration":810,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":297795716,"num_objects":72,"num_object_clones":0,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72,"num_whiteouts":0,"num_read":53,"num_read_kb":204803,"num_write":80,"num_write_kb":303106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.36","version":"132'176","reported_seq":262,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:58.897343+0000","last_change":"2024-05-28T06:26:58.897425+0000","last_active":"2024-05-28T06:26:58.897343+0000","last_peered":"2024-05-28T06:26:58.897343+0000","last_clean":"2024-05-28T06:26:58.897343+0000","last_became_active":"2024-05-28T06:24:23.994945+0000","last_became_peered":"2024-05-28T06:24:23.994945+0000","last_unstale":"2024-05-28T06:26:58.897343+0000","last_undegraded":"2024-05-28T06:26:58.897343+0000","last_fullsized":"2024-05-28T06:26:58.897343+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'176","last_scrub_stamp":"2024-05-28T06:26:58.897252+0000","last_deep_scrub":"132'176","last_deep_scrub_stamp":"2024-05-28T06:26:58.897252+0000","last_clean_scrub_stamp":"2024-05-28T06:26:58.897252+0000","objects_scrubbed":61,"log_size":176,"log_dups_size":0,"ondisk_log_size":176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:29:17.233708+0000","scrub_duration":780,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":255852544,"num_objects":61,"num_object_clones":0,"num_object_copies":122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":0,"num_read":58,"num_read_kb":237568,"num_write":63,"num_write_kb":253952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.35","version":"132'206","reported_seq":330,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:53.893600+0000","last_change":"2024-05-28T06:26:53.637376+0000","last_active":"2024-05-28T06:26:53.893600+0000","last_peered":"2024-05-28T06:26:53.893600+0000","last_clean":"2024-05-28T06:26:53.893600+0000","last_became_active":"2024-05-28T06:24:23.991405+0000","last_became_peered":"2024-05-28T06:24:23.991405+0000","last_unstale":"2024-05-28T06:26:53.893600+0000","last_undegraded":"2024-05-28T06:26:53.893600+0000","last_fullsized":"2024-05-28T06:26:53.893600+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'191","last_scrub_stamp":"2024-05-28T06:26:53.637203+0000","last_deep_scrub":"132'191","last_deep_scrub_stamp":"2024-05-28T06:26:53.637203+0000","last_clean_scrub_stamp":"2024-05-28T06:26:53.637203+0000","objects_scrubbed":77,"log_size":206,"log_dups_size":0,"ondisk_log_size":206,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:00:23.044354+0000","scrub_duration":1117,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":322961408,"num_objects":77,"num_object_clones":0,"num_object_copies":154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77,"num_whiteouts":0,"num_read":95,"num_read_kb":389120,"num_write":83,"num_write_kb":327680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.34","version":"132'197","reported_seq":317,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:27:02.646638+0000","last_change":"2024-05-28T06:27:02.646732+0000","last_active":"2024-05-28T06:27:02.646638+0000","last_peered":"2024-05-28T06:27:02.646638+0000","last_clean":"2024-05-28T06:27:02.646638+0000","last_became_active":"2024-05-28T06:24:24.017931+0000","last_became_peered":"2024-05-28T06:24:24.017931+0000","last_unstale":"2024-05-28T06:27:02.646638+0000","last_undegraded":"2024-05-28T06:27:02.646638+0000","last_fullsized":"2024-05-28T06:27:02.646638+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'197","last_scrub_stamp":"2024-05-28T06:27:02.646563+0000","last_deep_scrub":"132'197","last_deep_scrub_stamp":"2024-05-28T06:27:02.646563+0000","last_clean_scrub_stamp":"2024-05-28T06:27:02.646563+0000","objects_scrubbed":69,"log_size":197,"log_dups_size":0,"ondisk_log_size":197,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:51:11.088291+0000","scrub_duration":772,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":289406976,"num_objects":69,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":0,"num_read":92,"num_read_kb":360452,"num_write":77,"num_write_kb":294913,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.33","version":"132'195","reported_seq":304,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:53.924192+0000","last_change":"2024-05-28T06:26:53.708978+0000","last_active":"2024-05-28T06:26:53.924192+0000","last_peered":"2024-05-28T06:26:53.924192+0000","last_clean":"2024-05-28T06:26:53.924192+0000","last_became_active":"2024-05-28T06:24:24.007121+0000","last_became_peered":"2024-05-28T06:24:24.007121+0000","last_unstale":"2024-05-28T06:26:53.924192+0000","last_undegraded":"2024-05-28T06:26:53.924192+0000","last_fullsized":"2024-05-28T06:26:53.924192+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'177","last_scrub_stamp":"2024-05-28T06:26:53.708819+0000","last_deep_scrub":"132'177","last_deep_scrub_stamp":"2024-05-28T06:26:53.708819+0000","last_clean_scrub_stamp":"2024-05-28T06:26:53.708819+0000","objects_scrubbed":69,"log_size":195,"log_dups_size":0,"ondisk_log_size":195,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:10:20.192047+0000","scrub_duration":1145,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":289406976,"num_objects":69,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":0,"num_read":80,"num_read_kb":327680,"num_write":75,"num_write_kb":294912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.32","version":"132'195","reported_seq":300,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:27:04.004698+0000","last_change":"2024-05-28T06:27:04.004841+0000","last_active":"2024-05-28T06:27:04.004698+0000","last_peered":"2024-05-28T06:27:04.004698+0000","last_clean":"2024-05-28T06:27:04.004698+0000","last_became_active":"2024-05-28T06:24:54.998097+0000","last_became_peered":"2024-05-28T06:24:54.998097+0000","last_unstale":"2024-05-28T06:27:04.004698+0000","last_undegraded":"2024-05-28T06:27:04.004698+0000","last_fullsized":"2024-05-28T06:27:04.004698+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'195","last_scrub_stamp":"2024-05-28T06:27:04.004571+0000","last_deep_scrub":"132'195","last_deep_scrub_stamp":"2024-05-28T06:27:04.004571+0000","last_clean_scrub_stamp":"2024-05-28T06:27:04.004571+0000","objects_scrubbed":69,"log_size":195,"log_dups_size":0,"ondisk_log_size":195,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:44:47.369165+0000","scrub_duration":918,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":289406976,"num_objects":69,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":0,"num_read":62,"num_read_kb":253952,"num_write":77,"num_write_kb":299008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":16777216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.31","version":"132'182","reported_seq":240,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:27:01.385002+0000","last_change":"2024-05-28T06:27:01.385086+0000","last_active":"2024-05-28T06:27:01.385002+0000","last_peered":"2024-05-28T06:27:01.385002+0000","last_clean":"2024-05-28T06:27:01.385002+0000","last_became_active":"2024-05-28T06:24:24.020301+0000","last_became_peered":"2024-05-28T06:24:24.020301+0000","last_unstale":"2024-05-28T06:27:01.385002+0000","last_undegraded":"2024-05-28T06:27:01.385002+0000","last_fullsized":"2024-05-28T06:27:01.385002+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'182","last_scrub_stamp":"2024-05-28T06:27:01.384889+0000","last_deep_scrub":"132'182","last_deep_scrub_stamp":"2024-05-28T06:27:01.384889+0000","last_clean_scrub_stamp":"2024-05-28T06:27:01.384889+0000","objects_scrubbed":60,"log_size":182,"log_dups_size":0,"ondisk_log_size":182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:30:28.427911+0000","scrub_duration":728,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251658240,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":30,"num_read_kb":122880,"num_write":70,"num_write_kb":266240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.30","version":"132'210","reported_seq":289,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:55.778720+0000","last_change":"2024-05-28T06:26:55.778848+0000","last_active":"2024-05-28T06:26:55.778720+0000","last_peered":"2024-05-28T06:26:55.778720+0000","last_clean":"2024-05-28T06:26:55.778720+0000","last_became_active":"2024-05-28T06:24:23.995367+0000","last_became_peered":"2024-05-28T06:24:23.995367+0000","last_unstale":"2024-05-28T06:26:55.778720+0000","last_undegraded":"2024-05-28T06:26:55.778720+0000","last_fullsized":"2024-05-28T06:26:55.778720+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'210","last_scrub_stamp":"2024-05-28T06:26:55.778609+0000","last_deep_scrub":"132'210","last_deep_scrub_stamp":"2024-05-28T06:26:55.778609+0000","last_clean_scrub_stamp":"2024-05-28T06:26:55.778609+0000","objects_scrubbed":73,"log_size":210,"log_dups_size":0,"ondisk_log_size":210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:58:55.782613+0000","scrub_duration":1447,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306184192,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":51,"num_read_kb":208896,"num_write":81,"num_write_kb":315392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2f","version":"132'171","reported_seq":242,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:54.457906+0000","last_change":"2024-05-28T06:26:54.458025+0000","last_active":"2024-05-28T06:26:54.457906+0000","last_peered":"2024-05-28T06:26:54.457906+0000","last_clean":"2024-05-28T06:26:54.457906+0000","last_became_active":"2024-05-28T06:24:23.986777+0000","last_became_peered":"2024-05-28T06:24:23.986777+0000","last_unstale":"2024-05-28T06:26:54.457906+0000","last_undegraded":"2024-05-28T06:26:54.457906+0000","last_fullsized":"2024-05-28T06:26:54.457906+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'171","last_scrub_stamp":"2024-05-28T06:26:54.457733+0000","last_deep_scrub":"132'171","last_deep_scrub_stamp":"2024-05-28T06:26:54.457733+0000","last_clean_scrub_stamp":"2024-05-28T06:26:54.457733+0000","objects_scrubbed":55,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:51:40.637616+0000","scrub_duration":1393,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":230686720,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":42,"num_read_kb":172032,"num_write":65,"num_write_kb":245760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.2e","version":"132'211","reported_seq":326,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:57.218750+0000","last_change":"2024-05-28T06:26:57.218904+0000","last_active":"2024-05-28T06:26:57.218750+0000","last_peered":"2024-05-28T06:26:57.218750+0000","last_clean":"2024-05-28T06:26:57.218750+0000","last_became_active":"2024-05-28T06:24:23.998105+0000","last_became_peered":"2024-05-28T06:24:23.998105+0000","last_unstale":"2024-05-28T06:26:57.218750+0000","last_undegraded":"2024-05-28T06:26:57.218750+0000","last_fullsized":"2024-05-28T06:26:57.218750+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'211","last_scrub_stamp":"2024-05-28T06:26:57.218602+0000","last_deep_scrub":"132'211","last_deep_scrub_stamp":"2024-05-28T06:26:57.218602+0000","last_clean_scrub_stamp":"2024-05-28T06:26:57.218602+0000","objects_scrubbed":74,"log_size":211,"log_dups_size":0,"ondisk_log_size":211,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:00:40.275171+0000","scrub_duration":902,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":310378496,"num_objects":74,"num_object_clones":0,"num_object_copies":148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74,"num_whiteouts":0,"num_read":87,"num_read_kb":356352,"num_write":82,"num_write_kb":315393,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2d","version":"132'207","reported_seq":303,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:55.631015+0000","last_change":"2024-05-28T06:26:55.631154+0000","last_active":"2024-05-28T06:26:55.631015+0000","last_peered":"2024-05-28T06:26:55.631015+0000","last_clean":"2024-05-28T06:26:55.631015+0000","last_became_active":"2024-05-28T06:24:23.997362+0000","last_became_peered":"2024-05-28T06:24:23.997362+0000","last_unstale":"2024-05-28T06:26:55.631015+0000","last_undegraded":"2024-05-28T06:26:55.631015+0000","last_fullsized":"2024-05-28T06:26:55.631015+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'207","last_scrub_stamp":"2024-05-28T06:26:55.630904+0000","last_deep_scrub":"132'207","last_deep_scrub_stamp":"2024-05-28T06:26:55.630904+0000","last_clean_scrub_stamp":"2024-05-28T06:26:55.630904+0000","objects_scrubbed":74,"log_size":207,"log_dups_size":0,"ondisk_log_size":207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:24:18.346715+0000","scrub_duration":1171,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":310378496,"num_objects":74,"num_object_clones":0,"num_object_copies":148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74,"num_whiteouts":0,"num_read":67,"num_read_kb":274432,"num_write":82,"num_write_kb":319488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2c","version":"132'196","reported_seq":281,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:56.385070+0000","last_change":"2024-05-28T06:26:56.385149+0000","last_active":"2024-05-28T06:26:56.385070+0000","last_peered":"2024-05-28T06:26:56.385070+0000","last_clean":"2024-05-28T06:26:56.385070+0000","last_became_active":"2024-05-28T06:24:23.996067+0000","last_became_peered":"2024-05-28T06:24:23.996067+0000","last_unstale":"2024-05-28T06:26:56.385070+0000","last_undegraded":"2024-05-28T06:26:56.385070+0000","last_fullsized":"2024-05-28T06:26:56.385070+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'196","last_scrub_stamp":"2024-05-28T06:26:56.384977+0000","last_deep_scrub":"132'196","last_deep_scrub_stamp":"2024-05-28T06:26:56.384977+0000","last_clean_scrub_stamp":"2024-05-28T06:26:56.384977+0000","objects_scrubbed":65,"log_size":196,"log_dups_size":0,"ondisk_log_size":196,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:19:24.654707+0000","scrub_duration":1090,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":272629760,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":57,"num_read_kb":233472,"num_write":69,"num_write_kb":274432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2b","version":"132'201","reported_seq":320,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:27:05.117460+0000","last_change":"2024-05-28T06:27:05.117567+0000","last_active":"2024-05-28T06:27:05.117460+0000","last_peered":"2024-05-28T06:27:05.117460+0000","last_clean":"2024-05-28T06:27:05.117460+0000","last_became_active":"2024-05-28T06:24:24.018253+0000","last_became_peered":"2024-05-28T06:24:24.018253+0000","last_unstale":"2024-05-28T06:27:05.117460+0000","last_undegraded":"2024-05-28T06:27:05.117460+0000","last_fullsized":"2024-05-28T06:27:05.117460+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'201","last_scrub_stamp":"2024-05-28T06:27:05.117305+0000","last_deep_scrub":"132'201","last_deep_scrub_stamp":"2024-05-28T06:27:05.117305+0000","last_clean_scrub_stamp":"2024-05-28T06:27:05.117305+0000","objects_scrubbed":70,"log_size":201,"log_dups_size":0,"ondisk_log_size":201,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:43:35.910714+0000","scrub_duration":886,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":293601280,"num_objects":70,"num_object_clones":0,"num_object_copies":140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70,"num_whiteouts":0,"num_read":91,"num_read_kb":364546,"num_write":82,"num_write_kb":303106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2a","version":"132'202","reported_seq":286,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:54.896433+0000","last_change":"2024-05-28T06:26:54.896525+0000","last_active":"2024-05-28T06:26:54.896433+0000","last_peered":"2024-05-28T06:26:54.896433+0000","last_clean":"2024-05-28T06:26:54.896433+0000","last_became_active":"2024-05-28T06:24:23.994410+0000","last_became_peered":"2024-05-28T06:24:23.994410+0000","last_unstale":"2024-05-28T06:26:54.896433+0000","last_undegraded":"2024-05-28T06:26:54.896433+0000","last_fullsized":"2024-05-28T06:26:54.896433+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'202","last_scrub_stamp":"2024-05-28T06:26:54.896309+0000","last_deep_scrub":"132'202","last_deep_scrub_stamp":"2024-05-28T06:26:54.896309+0000","last_clean_scrub_stamp":"2024-05-28T06:26:54.896309+0000","objects_scrubbed":68,"log_size":202,"log_dups_size":0,"ondisk_log_size":202,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:00:04.074298+0000","scrub_duration":1423,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":285212672,"num_objects":68,"num_object_clones":0,"num_object_copies":136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68,"num_whiteouts":0,"num_read":55,"num_read_kb":217090,"num_write":74,"num_write_kb":286721,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.29","version":"132'207","reported_seq":287,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:27:00.797622+0000","last_change":"2024-05-28T06:27:00.797699+0000","last_active":"2024-05-28T06:27:00.797622+0000","last_peered":"2024-05-28T06:27:00.797622+0000","last_clean":"2024-05-28T06:27:00.797622+0000","last_became_active":"2024-05-28T06:24:24.015124+0000","last_became_peered":"2024-05-28T06:24:24.015124+0000","last_unstale":"2024-05-28T06:27:00.797622+0000","last_undegraded":"2024-05-28T06:27:00.797622+0000","last_fullsized":"2024-05-28T06:27:00.797622+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'207","last_scrub_stamp":"2024-05-28T06:27:00.797547+0000","last_deep_scrub":"132'207","last_deep_scrub_stamp":"2024-05-28T06:27:00.797547+0000","last_clean_scrub_stamp":"2024-05-28T06:27:00.797547+0000","objects_scrubbed":67,"log_size":207,"log_dups_size":0,"ondisk_log_size":207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:16:03.328904+0000","scrub_duration":888,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281018368,"num_objects":67,"num_object_clones":0,"num_object_copies":134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":0,"num_read":51,"num_read_kb":208896,"num_write":81,"num_write_kb":303104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.28","version":"132'197","reported_seq":276,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:56.536911+0000","last_change":"2024-05-28T06:26:56.537016+0000","last_active":"2024-05-28T06:26:56.536911+0000","last_peered":"2024-05-28T06:26:56.536911+0000","last_clean":"2024-05-28T06:26:56.536911+0000","last_became_active":"2024-05-28T06:24:24.016692+0000","last_became_peered":"2024-05-28T06:24:24.016692+0000","last_unstale":"2024-05-28T06:26:56.536911+0000","last_undegraded":"2024-05-28T06:26:56.536911+0000","last_fullsized":"2024-05-28T06:26:56.536911+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'197","last_scrub_stamp":"2024-05-28T06:26:56.536790+0000","last_deep_scrub":"132'197","last_deep_scrub_stamp":"2024-05-28T06:26:56.536790+0000","last_clean_scrub_stamp":"2024-05-28T06:26:56.536790+0000","objects_scrubbed":67,"log_size":197,"log_dups_size":0,"ondisk_log_size":197,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:38:12.152919+0000","scrub_duration":1080,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281018368,"num_objects":67,"num_object_clones":0,"num_object_copies":134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":0,"num_read":51,"num_read_kb":208896,"num_write":75,"num_write_kb":286721,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.27","version":"132'216","reported_seq":307,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:27:02.973334+0000","last_change":"2024-05-28T06:27:02.973518+0000","last_active":"2024-05-28T06:27:02.973334+0000","last_peered":"2024-05-28T06:27:02.973334+0000","last_clean":"2024-05-28T06:27:02.973334+0000","last_became_active":"2024-05-28T06:24:24.006983+0000","last_became_peered":"2024-05-28T06:24:24.006983+0000","last_unstale":"2024-05-28T06:27:02.973334+0000","last_undegraded":"2024-05-28T06:27:02.973334+0000","last_fullsized":"2024-05-28T06:27:02.973334+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'216","last_scrub_stamp":"2024-05-28T06:27:02.973118+0000","last_deep_scrub":"132'216","last_deep_scrub_stamp":"2024-05-28T06:27:02.973118+0000","last_clean_scrub_stamp":"2024-05-28T06:27:02.973118+0000","objects_scrubbed":73,"log_size":216,"log_dups_size":0,"ondisk_log_size":216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:55:20.640238+0000","scrub_duration":864,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306184192,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":63,"num_read_kb":249858,"num_write":81,"num_write_kb":307202,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.26","version":"132'217","reported_seq":307,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:59.159248+0000","last_change":"2024-05-28T06:26:59.159402+0000","last_active":"2024-05-28T06:26:59.159248+0000","last_peered":"2024-05-28T06:26:59.159248+0000","last_clean":"2024-05-28T06:26:59.159248+0000","last_became_active":"2024-05-28T06:24:23.996859+0000","last_became_peered":"2024-05-28T06:24:23.996859+0000","last_unstale":"2024-05-28T06:26:59.159248+0000","last_undegraded":"2024-05-28T06:26:59.159248+0000","last_fullsized":"2024-05-28T06:26:59.159248+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'217","last_scrub_stamp":"2024-05-28T06:26:59.159151+0000","last_deep_scrub":"132'217","last_deep_scrub_stamp":"2024-05-28T06:26:59.159151+0000","last_clean_scrub_stamp":"2024-05-28T06:26:59.159151+0000","objects_scrubbed":74,"log_size":217,"log_dups_size":0,"ondisk_log_size":217,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:56:58.429802+0000","scrub_duration":870,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":310378496,"num_objects":74,"num_object_clones":0,"num_object_copies":148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74,"num_whiteouts":0,"num_read":62,"num_read_kb":253952,"num_write":82,"num_write_kb":319488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.25","version":"132'182","reported_seq":277,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:55.620636+0000","last_change":"2024-05-28T06:26:55.620757+0000","last_active":"2024-05-28T06:26:55.620636+0000","last_peered":"2024-05-28T06:26:55.620636+0000","last_clean":"2024-05-28T06:26:55.620636+0000","last_became_active":"2024-05-28T06:24:24.007194+0000","last_became_peered":"2024-05-28T06:24:24.007194+0000","last_unstale":"2024-05-28T06:26:55.620636+0000","last_undegraded":"2024-05-28T06:26:55.620636+0000","last_fullsized":"2024-05-28T06:26:55.620636+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'182","last_scrub_stamp":"2024-05-28T06:26:55.620476+0000","last_deep_scrub":"132'182","last_deep_scrub_stamp":"2024-05-28T06:26:55.620476+0000","last_clean_scrub_stamp":"2024-05-28T06:26:55.620476+0000","objects_scrubbed":65,"log_size":182,"log_dups_size":0,"ondisk_log_size":182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:07:42.261337+0000","scrub_duration":1053,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":272629760,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":66,"num_read_kb":258051,"num_write":73,"num_write_kb":278529,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.24","version":"132'195","reported_seq":257,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:59.389040+0000","last_change":"2024-05-28T06:26:59.389172+0000","last_active":"2024-05-28T06:26:59.389040+0000","last_peered":"2024-05-28T06:26:59.389040+0000","last_clean":"2024-05-28T06:26:59.389040+0000","last_became_active":"2024-05-28T06:24:24.015990+0000","last_became_peered":"2024-05-28T06:24:24.015990+0000","last_unstale":"2024-05-28T06:26:59.389040+0000","last_undegraded":"2024-05-28T06:26:59.389040+0000","last_fullsized":"2024-05-28T06:26:59.389040+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'195","last_scrub_stamp":"2024-05-28T06:26:59.388909+0000","last_deep_scrub":"132'195","last_deep_scrub_stamp":"2024-05-28T06:26:59.388909+0000","last_clean_scrub_stamp":"2024-05-28T06:26:59.388909+0000","objects_scrubbed":67,"log_size":195,"log_dups_size":0,"ondisk_log_size":195,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:40:46.434026+0000","scrub_duration":811,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281018368,"num_objects":67,"num_object_clones":0,"num_object_copies":134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":0,"num_read":34,"num_read_kb":131074,"num_write":77,"num_write_kb":290817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.23","version":"132'211","reported_seq":310,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:59.479127+0000","last_change":"2024-05-28T06:26:59.479231+0000","last_active":"2024-05-28T06:26:59.479127+0000","last_peered":"2024-05-28T06:26:59.479127+0000","last_clean":"2024-05-28T06:26:59.479127+0000","last_became_active":"2024-05-28T06:24:24.019497+0000","last_became_peered":"2024-05-28T06:24:24.019497+0000","last_unstale":"2024-05-28T06:26:59.479127+0000","last_undegraded":"2024-05-28T06:26:59.479127+0000","last_fullsized":"2024-05-28T06:26:59.479127+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'211","last_scrub_stamp":"2024-05-28T06:26:59.479038+0000","last_deep_scrub":"132'211","last_deep_scrub_stamp":"2024-05-28T06:26:59.479038+0000","last_clean_scrub_stamp":"2024-05-28T06:26:59.479038+0000","objects_scrubbed":71,"log_size":211,"log_dups_size":0,"ondisk_log_size":211,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:53:17.399632+0000","scrub_duration":861,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":297795584,"num_objects":71,"num_object_clones":0,"num_object_copies":142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":71,"num_read_kb":274436,"num_write":81,"num_write_kb":307201,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.22","version":"132'213","reported_seq":304,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:58.498264+0000","last_change":"2024-05-28T06:26:58.498309+0000","last_active":"2024-05-28T06:26:58.498264+0000","last_peered":"2024-05-28T06:26:58.498264+0000","last_clean":"2024-05-28T06:26:58.498264+0000","last_became_active":"2024-05-28T06:24:24.019462+0000","last_became_peered":"2024-05-28T06:24:24.019462+0000","last_unstale":"2024-05-28T06:26:58.498264+0000","last_undegraded":"2024-05-28T06:26:58.498264+0000","last_fullsized":"2024-05-28T06:26:58.498264+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'213","last_scrub_stamp":"2024-05-28T06:26:58.498202+0000","last_deep_scrub":"132'213","last_deep_scrub_stamp":"2024-05-28T06:26:58.498202+0000","last_clean_scrub_stamp":"2024-05-28T06:26:58.498202+0000","objects_scrubbed":75,"log_size":213,"log_dups_size":0,"ondisk_log_size":213,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:49:23.149875+0000","scrub_duration":925,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":314572800,"num_objects":75,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75,"num_whiteouts":0,"num_read":62,"num_read_kb":253952,"num_write":85,"num_write_kb":327680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.21","version":"132'194","reported_seq":312,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:27:01.969956+0000","last_change":"2024-05-28T06:27:01.970026+0000","last_active":"2024-05-28T06:27:01.969956+0000","last_peered":"2024-05-28T06:27:01.969956+0000","last_clean":"2024-05-28T06:27:01.969956+0000","last_became_active":"2024-05-28T06:24:24.011246+0000","last_became_peered":"2024-05-28T06:24:24.011246+0000","last_unstale":"2024-05-28T06:27:01.969956+0000","last_undegraded":"2024-05-28T06:27:01.969956+0000","last_fullsized":"2024-05-28T06:27:01.969956+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'194","last_scrub_stamp":"2024-05-28T06:27:01.969887+0000","last_deep_scrub":"132'194","last_deep_scrub_stamp":"2024-05-28T06:27:01.969887+0000","last_clean_scrub_stamp":"2024-05-28T06:27:01.969887+0000","objects_scrubbed":66,"log_size":194,"log_dups_size":0,"ondisk_log_size":194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:36:09.104787+0000","scrub_duration":747,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":276824064,"num_objects":66,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":89,"num_read_kb":364544,"num_write":66,"num_write_kb":270336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.20","version":"132'202","reported_seq":302,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:55.342597+0000","last_change":"2024-05-28T06:26:55.342722+0000","last_active":"2024-05-28T06:26:55.342597+0000","last_peered":"2024-05-28T06:26:55.342597+0000","last_clean":"2024-05-28T06:26:55.342597+0000","last_became_active":"2024-05-28T06:24:23.991245+0000","last_became_peered":"2024-05-28T06:24:23.991245+0000","last_unstale":"2024-05-28T06:26:55.342597+0000","last_undegraded":"2024-05-28T06:26:55.342597+0000","last_fullsized":"2024-05-28T06:26:55.342597+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'202","last_scrub_stamp":"2024-05-28T06:26:55.342365+0000","last_deep_scrub":"132'202","last_deep_scrub_stamp":"2024-05-28T06:26:55.342365+0000","last_clean_scrub_stamp":"2024-05-28T06:26:55.342365+0000","objects_scrubbed":65,"log_size":202,"log_dups_size":0,"ondisk_log_size":202,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:11:10.407902+0000","scrub_duration":1263,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":272629760,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":71,"num_read_kb":282626,"num_write":75,"num_write_kb":282625,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1e","version":"132'182","reported_seq":252,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:27:01.987222+0000","last_change":"2024-05-28T06:27:01.987309+0000","last_active":"2024-05-28T06:27:01.987222+0000","last_peered":"2024-05-28T06:27:01.987222+0000","last_clean":"2024-05-28T06:27:01.987222+0000","last_became_active":"2024-05-28T06:24:24.017727+0000","last_became_peered":"2024-05-28T06:24:24.017727+0000","last_unstale":"2024-05-28T06:27:01.987222+0000","last_undegraded":"2024-05-28T06:27:01.987222+0000","last_fullsized":"2024-05-28T06:27:01.987222+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'182","last_scrub_stamp":"2024-05-28T06:27:01.987099+0000","last_deep_scrub":"132'182","last_deep_scrub_stamp":"2024-05-28T06:27:01.987099+0000","last_clean_scrub_stamp":"2024-05-28T06:27:01.987099+0000","objects_scrubbed":62,"log_size":182,"log_dups_size":0,"ondisk_log_size":182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:55:58.916707+0000","scrub_duration":704,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260046848,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":42,"num_read_kb":172032,"num_write":68,"num_write_kb":266240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1f","version":"132'6","reported_seq":34,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:54.140074+0000","last_change":"2024-05-28T06:26:54.140162+0000","last_active":"2024-05-28T06:26:54.140074+0000","last_peered":"2024-05-28T06:26:54.140074+0000","last_clean":"2024-05-28T06:26:54.140074+0000","last_became_active":"2024-05-28T06:24:23.006043+0000","last_became_peered":"2024-05-28T06:24:23.006043+0000","last_unstale":"2024-05-28T06:26:54.140074+0000","last_undegraded":"2024-05-28T06:26:54.140074+0000","last_fullsized":"2024-05-28T06:26:54.140074+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'6","last_scrub_stamp":"2024-05-28T06:26:54.139941+0000","last_deep_scrub":"132'6","last_deep_scrub_stamp":"2024-05-28T06:26:54.139941+0000","last_clean_scrub_stamp":"2024-05-28T06:26:54.139941+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:08:27.242219+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.1f","version":"132'209","reported_seq":305,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:56.821337+0000","last_change":"2024-05-28T06:26:56.821506+0000","last_active":"2024-05-28T06:26:56.821337+0000","last_peered":"2024-05-28T06:26:56.821337+0000","last_clean":"2024-05-28T06:26:56.821337+0000","last_became_active":"2024-05-28T06:24:24.008078+0000","last_became_peered":"2024-05-28T06:24:24.008078+0000","last_unstale":"2024-05-28T06:26:56.821337+0000","last_undegraded":"2024-05-28T06:26:56.821337+0000","last_fullsized":"2024-05-28T06:26:56.821337+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'209","last_scrub_stamp":"2024-05-28T06:26:56.821055+0000","last_deep_scrub":"132'209","last_deep_scrub_stamp":"2024-05-28T06:26:56.821055+0000","last_clean_scrub_stamp":"2024-05-28T06:26:56.821055+0000","objects_scrubbed":70,"log_size":209,"log_dups_size":0,"ondisk_log_size":209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:58:13.168447+0000","scrub_duration":993,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":293601280,"num_objects":70,"num_object_clones":0,"num_object_copies":140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70,"num_whiteouts":0,"num_read":68,"num_read_kb":262148,"num_write":82,"num_write_kb":303106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1e","version":"129'1","reported_seq":29,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:51.471860+0000","last_change":"2024-05-28T06:26:51.472017+0000","last_active":"2024-05-28T06:26:51.471860+0000","last_peered":"2024-05-28T06:26:51.471860+0000","last_clean":"2024-05-28T06:26:51.471860+0000","last_became_active":"2024-05-28T06:24:22.994896+0000","last_became_peered":"2024-05-28T06:24:22.994896+0000","last_unstale":"2024-05-28T06:26:51.471860+0000","last_undegraded":"2024-05-28T06:26:51.471860+0000","last_fullsized":"2024-05-28T06:26:51.471860+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'1","last_scrub_stamp":"2024-05-28T06:26:51.471616+0000","last_deep_scrub":"129'1","last_deep_scrub_stamp":"2024-05-28T06:26:51.471616+0000","last_clean_scrub_stamp":"2024-05-28T06:26:51.471616+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:37:26.798910+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1c","version":"132'185","reported_seq":277,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:57.864820+0000","last_change":"2024-05-28T06:26:57.864913+0000","last_active":"2024-05-28T06:26:57.864820+0000","last_peered":"2024-05-28T06:26:57.864820+0000","last_clean":"2024-05-28T06:26:57.864820+0000","last_became_active":"2024-05-28T06:24:24.006258+0000","last_became_peered":"2024-05-28T06:24:24.006258+0000","last_unstale":"2024-05-28T06:26:57.864820+0000","last_undegraded":"2024-05-28T06:26:57.864820+0000","last_fullsized":"2024-05-28T06:26:57.864820+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'185","last_scrub_stamp":"2024-05-28T06:26:57.864687+0000","last_deep_scrub":"132'185","last_deep_scrub_stamp":"2024-05-28T06:26:57.864687+0000","last_clean_scrub_stamp":"2024-05-28T06:26:57.864687+0000","objects_scrubbed":62,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:56:57.135338+0000","scrub_duration":721,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":255852805,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":64,"num_read_kb":262144,"num_write":70,"num_write_kb":266241,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1d","version":"132'4","reported_seq":32,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:49.524081+0000","last_change":"2024-05-28T06:26:49.524081+0000","last_active":"2024-05-28T06:26:49.524081+0000","last_peered":"2024-05-28T06:26:49.524081+0000","last_clean":"2024-05-28T06:26:49.524081+0000","last_became_active":"2024-05-28T06:24:22.991407+0000","last_became_peered":"2024-05-28T06:24:22.991407+0000","last_unstale":"2024-05-28T06:26:49.524081+0000","last_undegraded":"2024-05-28T06:26:49.524081+0000","last_fullsized":"2024-05-28T06:26:49.524081+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'4","last_scrub_stamp":"2024-05-28T06:26:49.523934+0000","last_deep_scrub":"132'4","last_deep_scrub_stamp":"2024-05-28T06:26:49.523934+0000","last_clean_scrub_stamp":"2024-05-28T06:26:49.523934+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:56:48.794586+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1d","version":"132'211","reported_seq":326,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:27:04.163974+0000","last_change":"2024-05-28T06:27:04.164085+0000","last_active":"2024-05-28T06:27:04.163974+0000","last_peered":"2024-05-28T06:27:04.163974+0000","last_clean":"2024-05-28T06:27:04.163974+0000","last_became_active":"2024-05-28T06:24:24.010356+0000","last_became_peered":"2024-05-28T06:24:24.010356+0000","last_unstale":"2024-05-28T06:27:04.163974+0000","last_undegraded":"2024-05-28T06:27:04.163974+0000","last_fullsized":"2024-05-28T06:27:04.163974+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'211","last_scrub_stamp":"2024-05-28T06:27:04.163798+0000","last_deep_scrub":"132'211","last_deep_scrub_stamp":"2024-05-28T06:27:04.163798+0000","last_clean_scrub_stamp":"2024-05-28T06:27:04.163798+0000","objects_scrubbed":71,"log_size":211,"log_dups_size":0,"ondisk_log_size":211,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:48:08.897021+0000","scrub_duration":907,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":297795584,"num_objects":71,"num_object_clones":0,"num_object_copies":142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":87,"num_read_kb":356352,"num_write":89,"num_write_kb":327680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1c","version":"132'3","reported_seq":31,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:48.983579+0000","last_change":"2024-05-28T06:26:48.983579+0000","last_active":"2024-05-28T06:26:48.983579+0000","last_peered":"2024-05-28T06:26:48.983579+0000","last_clean":"2024-05-28T06:26:48.983579+0000","last_became_active":"2024-05-28T06:24:22.989103+0000","last_became_peered":"2024-05-28T06:24:22.989103+0000","last_unstale":"2024-05-28T06:26:48.983579+0000","last_undegraded":"2024-05-28T06:26:48.983579+0000","last_fullsized":"2024-05-28T06:26:48.983579+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'3","last_scrub_stamp":"2024-05-28T06:26:48.983322+0000","last_deep_scrub":"132'3","last_deep_scrub_stamp":"2024-05-28T06:26:48.983322+0000","last_clean_scrub_stamp":"2024-05-28T06:26:48.983322+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:37:10.958945+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1a","version":"132'198","reported_seq":279,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:27:00.238296+0000","last_change":"2024-05-28T06:27:00.238392+0000","last_active":"2024-05-28T06:27:00.238296+0000","last_peered":"2024-05-28T06:27:00.238296+0000","last_clean":"2024-05-28T06:27:00.238296+0000","last_became_active":"2024-05-28T06:24:24.007987+0000","last_became_peered":"2024-05-28T06:24:24.007987+0000","last_unstale":"2024-05-28T06:27:00.238296+0000","last_undegraded":"2024-05-28T06:27:00.238296+0000","last_fullsized":"2024-05-28T06:27:00.238296+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'198","last_scrub_stamp":"2024-05-28T06:27:00.238183+0000","last_deep_scrub":"132'198","last_deep_scrub_stamp":"2024-05-28T06:27:00.238183+0000","last_clean_scrub_stamp":"2024-05-28T06:27:00.238183+0000","objects_scrubbed":72,"log_size":198,"log_dups_size":0,"ondisk_log_size":198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:18:34.323288+0000","scrub_duration":985,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301989888,"num_objects":72,"num_object_clones":0,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72,"num_whiteouts":0,"num_read":53,"num_read_kb":217088,"num_write":76,"num_write_kb":303104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1b","version":"129'1","reported_seq":29,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:48.008624+0000","last_change":"2024-05-28T06:26:48.008624+0000","last_active":"2024-05-28T06:26:48.008624+0000","last_peered":"2024-05-28T06:26:48.008624+0000","last_clean":"2024-05-28T06:26:48.008624+0000","last_became_active":"2024-05-28T06:24:22.985617+0000","last_became_peered":"2024-05-28T06:24:22.985617+0000","last_unstale":"2024-05-28T06:26:48.008624+0000","last_undegraded":"2024-05-28T06:26:48.008624+0000","last_fullsized":"2024-05-28T06:26:48.008624+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'1","last_scrub_stamp":"2024-05-28T06:26:48.008531+0000","last_deep_scrub":"129'1","last_deep_scrub_stamp":"2024-05-28T06:26:48.008531+0000","last_clean_scrub_stamp":"2024-05-28T06:26:48.008531+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:48:17.109256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1b","version":"132'195","reported_seq":313,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:53.897483+0000","last_change":"2024-05-28T06:26:53.212999+0000","last_active":"2024-05-28T06:26:53.897483+0000","last_peered":"2024-05-28T06:26:53.897483+0000","last_clean":"2024-05-28T06:26:53.897483+0000","last_became_active":"2024-05-28T06:24:23.985828+0000","last_became_peered":"2024-05-28T06:24:23.985828+0000","last_unstale":"2024-05-28T06:26:53.897483+0000","last_undegraded":"2024-05-28T06:26:53.897483+0000","last_fullsized":"2024-05-28T06:26:53.897483+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'173","last_scrub_stamp":"2024-05-28T06:26:53.212587+0000","last_deep_scrub":"132'173","last_deep_scrub_stamp":"2024-05-28T06:26:53.212587+0000","last_clean_scrub_stamp":"2024-05-28T06:26:53.212587+0000","objects_scrubbed":73,"log_size":195,"log_dups_size":0,"ondisk_log_size":195,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:51:01.654315+0000","scrub_duration":1161,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306184192,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":89,"num_read_kb":364544,"num_write":77,"num_write_kb":307200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1a","version":"129'1","reported_seq":29,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:49.195556+0000","last_change":"2024-05-28T06:26:49.195556+0000","last_active":"2024-05-28T06:26:49.195556+0000","last_peered":"2024-05-28T06:26:49.195556+0000","last_clean":"2024-05-28T06:26:49.195556+0000","last_became_active":"2024-05-28T06:24:22.989801+0000","last_became_peered":"2024-05-28T06:24:22.989801+0000","last_unstale":"2024-05-28T06:26:49.195556+0000","last_undegraded":"2024-05-28T06:26:49.195556+0000","last_fullsized":"2024-05-28T06:26:49.195556+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'1","last_scrub_stamp":"2024-05-28T06:26:49.195459+0000","last_deep_scrub":"129'1","last_deep_scrub_stamp":"2024-05-28T06:26:49.195459+0000","last_clean_scrub_stamp":"2024-05-28T06:26:49.195459+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:31:06.539914+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.18","version":"132'197","reported_seq":293,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:55.262396+0000","last_change":"2024-05-28T06:26:55.262428+0000","last_active":"2024-05-28T06:26:55.262396+0000","last_peered":"2024-05-28T06:26:55.262396+0000","last_clean":"2024-05-28T06:26:55.262396+0000","last_became_active":"2024-05-28T06:24:24.003446+0000","last_became_peered":"2024-05-28T06:24:24.003446+0000","last_unstale":"2024-05-28T06:26:55.262396+0000","last_undegraded":"2024-05-28T06:26:55.262396+0000","last_fullsized":"2024-05-28T06:26:55.262396+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'197","last_scrub_stamp":"2024-05-28T06:26:55.262310+0000","last_deep_scrub":"132'197","last_deep_scrub_stamp":"2024-05-28T06:26:55.262310+0000","last_clean_scrub_stamp":"2024-05-28T06:26:55.262310+0000","objects_scrubbed":62,"log_size":197,"log_dups_size":0,"ondisk_log_size":197,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:19:05.216601+0000","scrub_duration":1508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260046848,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":67,"num_read_kb":274432,"num_write":72,"num_write_kb":274432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":28,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:51.025865+0000","last_change":"2024-05-28T06:26:51.025917+0000","last_active":"2024-05-28T06:26:51.025865+0000","last_peered":"2024-05-28T06:26:51.025865+0000","last_clean":"2024-05-28T06:26:51.025865+0000","last_became_active":"2024-05-28T06:24:23.007494+0000","last_became_peered":"2024-05-28T06:24:23.007494+0000","last_unstale":"2024-05-28T06:26:51.025865+0000","last_undegraded":"2024-05-28T06:26:51.025865+0000","last_fullsized":"2024-05-28T06:26:51.025865+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:26:51.025669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:26:51.025669+0000","last_clean_scrub_stamp":"2024-05-28T06:26:51.025669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:31:08.370125+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.19","version":"132'195","reported_seq":285,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:57.893870+0000","last_change":"2024-05-28T06:26:57.893945+0000","last_active":"2024-05-28T06:26:57.893870+0000","last_peered":"2024-05-28T06:26:57.893870+0000","last_clean":"2024-05-28T06:26:57.893870+0000","last_became_active":"2024-05-28T06:24:23.987466+0000","last_became_peered":"2024-05-28T06:24:23.987466+0000","last_unstale":"2024-05-28T06:26:57.893870+0000","last_undegraded":"2024-05-28T06:26:57.893870+0000","last_fullsized":"2024-05-28T06:26:57.893870+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'195","last_scrub_stamp":"2024-05-28T06:26:57.893791+0000","last_deep_scrub":"132'195","last_deep_scrub_stamp":"2024-05-28T06:26:57.893791+0000","last_clean_scrub_stamp":"2024-05-28T06:26:57.893791+0000","objects_scrubbed":66,"log_size":195,"log_dups_size":0,"ondisk_log_size":195,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:54:06.632912+0000","scrub_duration":751,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":276824064,"num_objects":66,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":62,"num_read_kb":245762,"num_write":72,"num_write_kb":278529,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":28,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:50.006843+0000","last_change":"2024-05-28T06:26:50.006881+0000","last_active":"2024-05-28T06:26:50.006843+0000","last_peered":"2024-05-28T06:26:50.006843+0000","last_clean":"2024-05-28T06:26:50.006843+0000","last_became_active":"2024-05-28T06:24:23.000037+0000","last_became_peered":"2024-05-28T06:24:23.000037+0000","last_unstale":"2024-05-28T06:26:50.006843+0000","last_undegraded":"2024-05-28T06:26:50.006843+0000","last_fullsized":"2024-05-28T06:26:50.006843+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:26:50.006725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:26:50.006725+0000","last_clean_scrub_stamp":"2024-05-28T06:26:50.006725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:34:39.408098+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"132'202","reported_seq":263,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:53.917353+0000","last_change":"2024-05-28T06:26:51.179357+0000","last_active":"2024-05-28T06:26:53.917353+0000","last_peered":"2024-05-28T06:26:53.917353+0000","last_clean":"2024-05-28T06:26:53.917353+0000","last_became_active":"2024-05-28T06:24:23.995204+0000","last_became_peered":"2024-05-28T06:24:23.995204+0000","last_unstale":"2024-05-28T06:26:53.917353+0000","last_undegraded":"2024-05-28T06:26:53.917353+0000","last_fullsized":"2024-05-28T06:26:53.917353+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'144","last_scrub_stamp":"2024-05-28T06:26:51.179074+0000","last_deep_scrub":"132'144","last_deep_scrub_stamp":"2024-05-28T06:26:51.179074+0000","last_clean_scrub_stamp":"2024-05-28T06:26:51.179074+0000","objects_scrubbed":62,"log_size":202,"log_dups_size":0,"ondisk_log_size":202,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:07:42.354658+0000","scrub_duration":558,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260046848,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":33,"num_read_kb":135168,"num_write":70,"num_write_kb":270336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.17","version":"132'1","reported_seq":29,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:55.566012+0000","last_change":"2024-05-28T06:26:55.566080+0000","last_active":"2024-05-28T06:26:55.566012+0000","last_peered":"2024-05-28T06:26:55.566012+0000","last_clean":"2024-05-28T06:26:55.566012+0000","last_became_active":"2024-05-28T06:24:22.999720+0000","last_became_peered":"2024-05-28T06:24:22.999720+0000","last_unstale":"2024-05-28T06:26:55.566012+0000","last_undegraded":"2024-05-28T06:26:55.566012+0000","last_fullsized":"2024-05-28T06:26:55.566012+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'1","last_scrub_stamp":"2024-05-28T06:26:55.565815+0000","last_deep_scrub":"132'1","last_deep_scrub_stamp":"2024-05-28T06:26:55.565815+0000","last_clean_scrub_stamp":"2024-05-28T06:26:55.565815+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:06:01.958916+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.17","version":"132'181","reported_seq":247,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:27:01.205809+0000","last_change":"2024-05-28T06:27:01.205889+0000","last_active":"2024-05-28T06:27:01.205809+0000","last_peered":"2024-05-28T06:27:01.205809+0000","last_clean":"2024-05-28T06:27:01.205809+0000","last_became_active":"2024-05-28T06:24:24.016597+0000","last_became_peered":"2024-05-28T06:24:24.016597+0000","last_unstale":"2024-05-28T06:27:01.205809+0000","last_undegraded":"2024-05-28T06:27:01.205809+0000","last_fullsized":"2024-05-28T06:27:01.205809+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'181","last_scrub_stamp":"2024-05-28T06:27:01.205696+0000","last_deep_scrub":"132'181","last_deep_scrub_stamp":"2024-05-28T06:27:01.205696+0000","last_clean_scrub_stamp":"2024-05-28T06:27:01.205696+0000","objects_scrubbed":59,"log_size":181,"log_dups_size":0,"ondisk_log_size":181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:10:35.937904+0000","scrub_duration":722,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247463936,"num_objects":59,"num_object_clones":0,"num_object_copies":118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":0,"num_read":38,"num_read_kb":155648,"num_write":63,"num_write_kb":249856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":28,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:52.327894+0000","last_change":"2024-05-28T06:26:52.327932+0000","last_active":"2024-05-28T06:26:52.327894+0000","last_peered":"2024-05-28T06:26:52.327894+0000","last_clean":"2024-05-28T06:26:52.327894+0000","last_became_active":"2024-05-28T06:24:23.000878+0000","last_became_peered":"2024-05-28T06:24:23.000878+0000","last_unstale":"2024-05-28T06:26:52.327894+0000","last_undegraded":"2024-05-28T06:26:52.327894+0000","last_fullsized":"2024-05-28T06:26:52.327894+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:26:52.327803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:26:52.327803+0000","last_clean_scrub_stamp":"2024-05-28T06:26:52.327803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:32:07.505104+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.14","version":"132'180","reported_seq":283,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:27:01.004781+0000","last_change":"2024-05-28T06:27:01.004870+0000","last_active":"2024-05-28T06:27:01.004781+0000","last_peered":"2024-05-28T06:27:01.004781+0000","last_clean":"2024-05-28T06:27:01.004781+0000","last_became_active":"2024-05-28T06:24:24.010110+0000","last_became_peered":"2024-05-28T06:24:24.010110+0000","last_unstale":"2024-05-28T06:27:01.004781+0000","last_undegraded":"2024-05-28T06:27:01.004781+0000","last_fullsized":"2024-05-28T06:27:01.004781+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'180","last_scrub_stamp":"2024-05-28T06:27:01.004649+0000","last_deep_scrub":"132'180","last_deep_scrub_stamp":"2024-05-28T06:27:01.004649+0000","last_clean_scrub_stamp":"2024-05-28T06:27:01.004649+0000","objects_scrubbed":60,"log_size":180,"log_dups_size":0,"ondisk_log_size":180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:55:57.934256+0000","scrub_duration":760,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251658240,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":75,"num_read_kb":307200,"num_write":74,"num_write_kb":274432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.15","version":"132'26","reported_seq":54,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:54.526570+0000","last_change":"2024-05-28T06:26:54.526621+0000","last_active":"2024-05-28T06:26:54.526570+0000","last_peered":"2024-05-28T06:26:54.526570+0000","last_clean":"2024-05-28T06:26:54.526570+0000","last_became_active":"2024-05-28T06:24:22.998379+0000","last_became_peered":"2024-05-28T06:24:22.998379+0000","last_unstale":"2024-05-28T06:26:54.526570+0000","last_undegraded":"2024-05-28T06:26:54.526570+0000","last_fullsized":"2024-05-28T06:26:54.526570+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'26","last_scrub_stamp":"2024-05-28T06:26:54.526470+0000","last_deep_scrub":"132'26","last_deep_scrub_stamp":"2024-05-28T06:26:54.526470+0000","last_clean_scrub_stamp":"2024-05-28T06:26:54.526470+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T07:30:21.569491+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.15","version":"132'175","reported_seq":264,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:27:06.949570+0000","last_change":"2024-05-28T06:27:06.949638+0000","last_active":"2024-05-28T06:27:06.949570+0000","last_peered":"2024-05-28T06:27:06.949570+0000","last_clean":"2024-05-28T06:27:06.949570+0000","last_became_active":"2024-05-28T06:24:24.019973+0000","last_became_peered":"2024-05-28T06:24:24.019973+0000","last_unstale":"2024-05-28T06:27:06.949570+0000","last_undegraded":"2024-05-28T06:27:06.949570+0000","last_fullsized":"2024-05-28T06:27:06.949570+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'175","last_scrub_stamp":"2024-05-28T06:27:06.949435+0000","last_deep_scrub":"132'175","last_deep_scrub_stamp":"2024-05-28T06:27:06.949435+0000","last_clean_scrub_stamp":"2024-05-28T06:27:06.949435+0000","objects_scrubbed":55,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:32:11.871805+0000","scrub_duration":734,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":230686720,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":60,"num_read_kb":229380,"num_write":67,"num_write_kb":241666,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.14","version":"132'433","reported_seq":461,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:53.901552+0000","last_change":"2024-05-28T06:26:52.811967+0000","last_active":"2024-05-28T06:26:53.901552+0000","last_peered":"2024-05-28T06:26:53.901552+0000","last_clean":"2024-05-28T06:26:53.901552+0000","last_became_active":"2024-05-28T06:24:23.004831+0000","last_became_peered":"2024-05-28T06:24:23.004831+0000","last_unstale":"2024-05-28T06:26:53.901552+0000","last_undegraded":"2024-05-28T06:26:53.901552+0000","last_fullsized":"2024-05-28T06:26:53.901552+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'425","last_scrub_stamp":"2024-05-28T06:26:52.811731+0000","last_deep_scrub":"132'425","last_deep_scrub_stamp":"2024-05-28T06:26:52.811731+0000","last_clean_scrub_stamp":"2024-05-28T06:26:52.811731+0000","objects_scrubbed":4,"log_size":433,"log_dups_size":0,"ondisk_log_size":433,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:08:40.790399+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1675044,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":430,"num_write_kb":1806,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.13","version":"132'202","reported_seq":281,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:53.924399+0000","last_change":"2024-05-28T06:26:52.559785+0000","last_active":"2024-05-28T06:26:53.924399+0000","last_peered":"2024-05-28T06:26:53.924399+0000","last_clean":"2024-05-28T06:26:53.924399+0000","last_became_active":"2024-05-28T06:24:24.001429+0000","last_became_peered":"2024-05-28T06:24:24.001429+0000","last_unstale":"2024-05-28T06:26:53.924399+0000","last_undegraded":"2024-05-28T06:26:53.924399+0000","last_fullsized":"2024-05-28T06:26:53.924399+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'171","last_scrub_stamp":"2024-05-28T06:26:52.559657+0000","last_deep_scrub":"132'171","last_deep_scrub_stamp":"2024-05-28T06:26:52.559657+0000","last_clean_scrub_stamp":"2024-05-28T06:26:52.559657+0000","objects_scrubbed":71,"log_size":202,"log_dups_size":0,"ondisk_log_size":202,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:58:14.710159+0000","scrub_duration":966,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":297795584,"num_objects":71,"num_object_clones":0,"num_object_copies":142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":50,"num_read_kb":196610,"num_write":75,"num_write_kb":294913,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":28,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:48.685424+0000","last_change":"2024-05-28T06:26:48.685445+0000","last_active":"2024-05-28T06:26:48.685424+0000","last_peered":"2024-05-28T06:26:48.685424+0000","last_clean":"2024-05-28T06:26:48.685424+0000","last_became_active":"2024-05-28T06:24:22.993301+0000","last_became_peered":"2024-05-28T06:24:22.993301+0000","last_unstale":"2024-05-28T06:26:48.685424+0000","last_undegraded":"2024-05-28T06:26:48.685424+0000","last_fullsized":"2024-05-28T06:26:48.685424+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:26:48.685333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:26:48.685333+0000","last_clean_scrub_stamp":"2024-05-28T06:26:48.685333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:38:36.311248+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.10","version":"132'185","reported_seq":270,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:27:00.930506+0000","last_change":"2024-05-28T06:27:00.930662+0000","last_active":"2024-05-28T06:27:00.930506+0000","last_peered":"2024-05-28T06:27:00.930506+0000","last_clean":"2024-05-28T06:27:00.930506+0000","last_became_active":"2024-05-28T06:24:24.005080+0000","last_became_peered":"2024-05-28T06:24:24.005080+0000","last_unstale":"2024-05-28T06:27:00.930506+0000","last_undegraded":"2024-05-28T06:27:00.930506+0000","last_fullsized":"2024-05-28T06:27:00.930506+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'185","last_scrub_stamp":"2024-05-28T06:27:00.930387+0000","last_deep_scrub":"132'185","last_deep_scrub_stamp":"2024-05-28T06:27:00.930387+0000","last_clean_scrub_stamp":"2024-05-28T06:27:00.930387+0000","objects_scrubbed":65,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:08:48.909055+0000","scrub_duration":854,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":272629760,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":57,"num_read_kb":233472,"num_write":69,"num_write_kb":274432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":28,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:52.041985+0000","last_change":"2024-05-28T06:26:52.042062+0000","last_active":"2024-05-28T06:26:52.041985+0000","last_peered":"2024-05-28T06:26:52.041985+0000","last_clean":"2024-05-28T06:26:52.041985+0000","last_became_active":"2024-05-28T06:24:23.005194+0000","last_became_peered":"2024-05-28T06:24:23.005194+0000","last_unstale":"2024-05-28T06:26:52.041985+0000","last_undegraded":"2024-05-28T06:26:52.041985+0000","last_fullsized":"2024-05-28T06:26:52.041985+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:26:52.041754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:26:52.041754+0000","last_clean_scrub_stamp":"2024-05-28T06:26:52.041754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:09:16.936348+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.11","version":"132'224","reported_seq":315,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:59.851271+0000","last_change":"2024-05-28T06:26:59.851311+0000","last_active":"2024-05-28T06:26:59.851271+0000","last_peered":"2024-05-28T06:26:59.851271+0000","last_clean":"2024-05-28T06:26:59.851271+0000","last_became_active":"2024-05-28T06:24:24.009163+0000","last_became_peered":"2024-05-28T06:24:24.009163+0000","last_unstale":"2024-05-28T06:26:59.851271+0000","last_undegraded":"2024-05-28T06:26:59.851271+0000","last_fullsized":"2024-05-28T06:26:59.851271+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'224","last_scrub_stamp":"2024-05-28T06:26:59.851176+0000","last_deep_scrub":"132'224","last_deep_scrub_stamp":"2024-05-28T06:26:59.851176+0000","last_clean_scrub_stamp":"2024-05-28T06:26:59.851176+0000","objects_scrubbed":76,"log_size":224,"log_dups_size":0,"ondisk_log_size":224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:55:30.369635+0000","scrub_duration":989,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":318767104,"num_objects":76,"num_object_clones":0,"num_object_copies":152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76,"num_whiteouts":0,"num_read":62,"num_read_kb":253952,"num_write":84,"num_write_kb":323585,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.10","version":"132'12","reported_seq":40,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:51.810775+0000","last_change":"2024-05-28T06:26:51.810847+0000","last_active":"2024-05-28T06:26:51.810775+0000","last_peered":"2024-05-28T06:26:51.810775+0000","last_clean":"2024-05-28T06:26:51.810775+0000","last_became_active":"2024-05-28T06:24:22.996644+0000","last_became_peered":"2024-05-28T06:24:22.996644+0000","last_unstale":"2024-05-28T06:26:51.810775+0000","last_undegraded":"2024-05-28T06:26:51.810775+0000","last_fullsized":"2024-05-28T06:26:51.810775+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'12","last_scrub_stamp":"2024-05-28T06:26:51.810586+0000","last_deep_scrub":"132'12","last_deep_scrub_stamp":"2024-05-28T06:26:51.810586+0000","last_clean_scrub_stamp":"2024-05-28T06:26:51.810586+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:42:21.868679+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.e","version":"132'156","reported_seq":207,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:58.477765+0000","last_change":"2024-05-28T06:26:58.477824+0000","last_active":"2024-05-28T06:26:58.477765+0000","last_peered":"2024-05-28T06:26:58.477765+0000","last_clean":"2024-05-28T06:26:58.477765+0000","last_became_active":"2024-05-28T06:24:24.009033+0000","last_became_peered":"2024-05-28T06:24:24.009033+0000","last_unstale":"2024-05-28T06:26:58.477765+0000","last_undegraded":"2024-05-28T06:26:58.477765+0000","last_fullsized":"2024-05-28T06:26:58.477765+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'156","last_scrub_stamp":"2024-05-28T06:26:58.477621+0000","last_deep_scrub":"132'156","last_deep_scrub_stamp":"2024-05-28T06:26:58.477621+0000","last_clean_scrub_stamp":"2024-05-28T06:26:58.477621+0000","objects_scrubbed":52,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:39:14.758386+0000","scrub_duration":657,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218103808,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":23,"num_read_kb":94208,"num_write":58,"num_write_kb":225280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.f","version":"132'2","reported_seq":30,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:56.577273+0000","last_change":"2024-05-28T06:26:56.577314+0000","last_active":"2024-05-28T06:26:56.577273+0000","last_peered":"2024-05-28T06:26:56.577273+0000","last_clean":"2024-05-28T06:26:56.577273+0000","last_became_active":"2024-05-28T06:24:23.005650+0000","last_became_peered":"2024-05-28T06:24:23.005650+0000","last_unstale":"2024-05-28T06:26:56.577273+0000","last_undegraded":"2024-05-28T06:26:56.577273+0000","last_fullsized":"2024-05-28T06:26:56.577273+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'2","last_scrub_stamp":"2024-05-28T06:26:56.577132+0000","last_deep_scrub":"132'2","last_deep_scrub_stamp":"2024-05-28T06:26:56.577132+0000","last_clean_scrub_stamp":"2024-05-28T06:26:56.577132+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:13:23.594334+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.f","version":"132'205","reported_seq":308,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:27:00.514150+0000","last_change":"2024-05-28T06:27:00.514246+0000","last_active":"2024-05-28T06:27:00.514150+0000","last_peered":"2024-05-28T06:27:00.514150+0000","last_clean":"2024-05-28T06:27:00.514150+0000","last_became_active":"2024-05-28T06:24:24.016255+0000","last_became_peered":"2024-05-28T06:24:24.016255+0000","last_unstale":"2024-05-28T06:27:00.514150+0000","last_undegraded":"2024-05-28T06:27:00.514150+0000","last_fullsized":"2024-05-28T06:27:00.514150+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'205","last_scrub_stamp":"2024-05-28T06:27:00.514048+0000","last_deep_scrub":"132'205","last_deep_scrub_stamp":"2024-05-28T06:27:00.514048+0000","last_clean_scrub_stamp":"2024-05-28T06:27:00.514048+0000","objects_scrubbed":73,"log_size":205,"log_dups_size":0,"ondisk_log_size":205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:57:40.309471+0000","scrub_duration":983,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306184192,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":74,"num_read_kb":303104,"num_write":83,"num_write_kb":315393,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.e","version":"132'3","reported_seq":31,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:50.499896+0000","last_change":"2024-05-28T06:26:50.499956+0000","last_active":"2024-05-28T06:26:50.499896+0000","last_peered":"2024-05-28T06:26:50.499896+0000","last_clean":"2024-05-28T06:26:50.499896+0000","last_became_active":"2024-05-28T06:24:23.001946+0000","last_became_peered":"2024-05-28T06:24:23.001946+0000","last_unstale":"2024-05-28T06:26:50.499896+0000","last_undegraded":"2024-05-28T06:26:50.499896+0000","last_fullsized":"2024-05-28T06:26:50.499896+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'3","last_scrub_stamp":"2024-05-28T06:26:50.499713+0000","last_deep_scrub":"132'3","last_deep_scrub_stamp":"2024-05-28T06:26:50.499713+0000","last_clean_scrub_stamp":"2024-05-28T06:26:50.499713+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:18:24.584818+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"132'189","reported_seq":277,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:57.968750+0000","last_change":"2024-05-28T06:26:57.968823+0000","last_active":"2024-05-28T06:26:57.968750+0000","last_peered":"2024-05-28T06:26:57.968750+0000","last_clean":"2024-05-28T06:26:57.968750+0000","last_became_active":"2024-05-28T06:24:23.996433+0000","last_became_peered":"2024-05-28T06:24:23.996433+0000","last_unstale":"2024-05-28T06:26:57.968750+0000","last_undegraded":"2024-05-28T06:26:57.968750+0000","last_fullsized":"2024-05-28T06:26:57.968750+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'189","last_scrub_stamp":"2024-05-28T06:26:57.968656+0000","last_deep_scrub":"132'189","last_deep_scrub_stamp":"2024-05-28T06:26:57.968656+0000","last_clean_scrub_stamp":"2024-05-28T06:26:57.968656+0000","objects_scrubbed":56,"log_size":189,"log_dups_size":0,"ondisk_log_size":189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:47:43.905506+0000","scrub_duration":652,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":230686900,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":60,"num_read_kb":221190,"num_write":66,"num_write_kb":233476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.d","version":"132'2","reported_seq":30,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:50.053151+0000","last_change":"2024-05-28T06:26:50.053220+0000","last_active":"2024-05-28T06:26:50.053151+0000","last_peered":"2024-05-28T06:26:50.053151+0000","last_clean":"2024-05-28T06:26:50.053151+0000","last_became_active":"2024-05-28T06:24:22.992147+0000","last_became_peered":"2024-05-28T06:24:22.992147+0000","last_unstale":"2024-05-28T06:26:50.053151+0000","last_undegraded":"2024-05-28T06:26:50.053151+0000","last_fullsized":"2024-05-28T06:26:50.053151+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'2","last_scrub_stamp":"2024-05-28T06:26:50.052936+0000","last_deep_scrub":"132'2","last_deep_scrub_stamp":"2024-05-28T06:26:50.052936+0000","last_clean_scrub_stamp":"2024-05-28T06:26:50.052936+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T06:55:50.028331+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.d","version":"132'194","reported_seq":294,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:57.011599+0000","last_change":"2024-05-28T06:26:57.011665+0000","last_active":"2024-05-28T06:26:57.011599+0000","last_peered":"2024-05-28T06:26:57.011599+0000","last_clean":"2024-05-28T06:26:57.011599+0000","last_became_active":"2024-05-28T06:24:23.988631+0000","last_became_peered":"2024-05-28T06:24:23.988631+0000","last_unstale":"2024-05-28T06:26:57.011599+0000","last_undegraded":"2024-05-28T06:26:57.011599+0000","last_fullsized":"2024-05-28T06:26:57.011599+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'194","last_scrub_stamp":"2024-05-28T06:26:57.011477+0000","last_deep_scrub":"132'194","last_deep_scrub_stamp":"2024-05-28T06:26:57.011477+0000","last_clean_scrub_stamp":"2024-05-28T06:26:57.011477+0000","objects_scrubbed":65,"log_size":194,"log_dups_size":0,"ondisk_log_size":194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:33:40.082670+0000","scrub_duration":843,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":272629760,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":72,"num_read_kb":294912,"num_write":79,"num_write_kb":294912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":28,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:52.424375+0000","last_change":"2024-05-28T06:26:52.424403+0000","last_active":"2024-05-28T06:26:52.424375+0000","last_peered":"2024-05-28T06:26:52.424375+0000","last_clean":"2024-05-28T06:26:52.424375+0000","last_became_active":"2024-05-28T06:24:22.993636+0000","last_became_peered":"2024-05-28T06:24:22.993636+0000","last_unstale":"2024-05-28T06:26:52.424375+0000","last_undegraded":"2024-05-28T06:26:52.424375+0000","last_fullsized":"2024-05-28T06:26:52.424375+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:26:52.424265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:26:52.424265+0000","last_clean_scrub_stamp":"2024-05-28T06:26:52.424265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:49:00.578097+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.a","version":"132'210","reported_seq":312,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:56.672164+0000","last_change":"2024-05-28T06:26:56.672298+0000","last_active":"2024-05-28T06:26:56.672164+0000","last_peered":"2024-05-28T06:26:56.672164+0000","last_clean":"2024-05-28T06:26:56.672164+0000","last_became_active":"2024-05-28T06:24:24.007259+0000","last_became_peered":"2024-05-28T06:24:24.007259+0000","last_unstale":"2024-05-28T06:26:56.672164+0000","last_undegraded":"2024-05-28T06:26:56.672164+0000","last_fullsized":"2024-05-28T06:26:56.672164+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'210","last_scrub_stamp":"2024-05-28T06:26:56.672070+0000","last_deep_scrub":"132'210","last_deep_scrub_stamp":"2024-05-28T06:26:56.672070+0000","last_clean_scrub_stamp":"2024-05-28T06:26:56.672070+0000","objects_scrubbed":73,"log_size":210,"log_dups_size":0,"ondisk_log_size":210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:25:08.331676+0000","scrub_duration":1118,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306184192,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":74,"num_read_kb":303104,"num_write":79,"num_write_kb":311296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.b","version":"132'4","reported_seq":32,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:53.477027+0000","last_change":"2024-05-28T06:26:53.477250+0000","last_active":"2024-05-28T06:26:53.477027+0000","last_peered":"2024-05-28T06:26:53.477027+0000","last_clean":"2024-05-28T06:26:53.477027+0000","last_became_active":"2024-05-28T06:24:22.995489+0000","last_became_peered":"2024-05-28T06:24:22.995489+0000","last_unstale":"2024-05-28T06:26:53.477027+0000","last_undegraded":"2024-05-28T06:26:53.477027+0000","last_fullsized":"2024-05-28T06:26:53.477027+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'4","last_scrub_stamp":"2024-05-28T06:26:53.476750+0000","last_deep_scrub":"132'4","last_deep_scrub_stamp":"2024-05-28T06:26:53.476750+0000","last_clean_scrub_stamp":"2024-05-28T06:26:53.476750+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:22:55.252208+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.b","version":"132'229","reported_seq":322,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:57.917173+0000","last_change":"2024-05-28T06:26:57.917245+0000","last_active":"2024-05-28T06:26:57.917173+0000","last_peered":"2024-05-28T06:26:57.917173+0000","last_clean":"2024-05-28T06:26:57.917173+0000","last_became_active":"2024-05-28T06:24:24.015295+0000","last_became_peered":"2024-05-28T06:24:24.015295+0000","last_unstale":"2024-05-28T06:26:57.917173+0000","last_undegraded":"2024-05-28T06:26:57.917173+0000","last_fullsized":"2024-05-28T06:26:57.917173+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'229","last_scrub_stamp":"2024-05-28T06:26:57.917063+0000","last_deep_scrub":"132'229","last_deep_scrub_stamp":"2024-05-28T06:26:57.917063+0000","last_clean_scrub_stamp":"2024-05-28T06:26:57.917063+0000","objects_scrubbed":76,"log_size":229,"log_dups_size":0,"ondisk_log_size":229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:06:43.497620+0000","scrub_duration":884,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":314572932,"num_objects":76,"num_object_clones":0,"num_object_copies":152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76,"num_whiteouts":0,"num_read":64,"num_read_kb":237574,"num_write":90,"num_write_kb":323588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":28,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:51.266422+0000","last_change":"2024-05-28T06:26:51.266473+0000","last_active":"2024-05-28T06:26:51.266422+0000","last_peered":"2024-05-28T06:26:51.266422+0000","last_clean":"2024-05-28T06:26:51.266422+0000","last_became_active":"2024-05-28T06:24:22.994227+0000","last_became_peered":"2024-05-28T06:24:22.994227+0000","last_unstale":"2024-05-28T06:26:51.266422+0000","last_undegraded":"2024-05-28T06:26:51.266422+0000","last_fullsized":"2024-05-28T06:26:51.266422+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:26:51.266199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:26:51.266199+0000","last_clean_scrub_stamp":"2024-05-28T06:26:51.266199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T17:06:11.600957+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.8","version":"132'227","reported_seq":297,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:56.507582+0000","last_change":"2024-05-28T06:26:56.507639+0000","last_active":"2024-05-28T06:26:56.507582+0000","last_peered":"2024-05-28T06:26:56.507582+0000","last_clean":"2024-05-28T06:26:56.507582+0000","last_became_active":"2024-05-28T06:24:23.986897+0000","last_became_peered":"2024-05-28T06:24:23.986897+0000","last_unstale":"2024-05-28T06:26:56.507582+0000","last_undegraded":"2024-05-28T06:26:56.507582+0000","last_fullsized":"2024-05-28T06:26:56.507582+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'227","last_scrub_stamp":"2024-05-28T06:26:56.507468+0000","last_deep_scrub":"132'227","last_deep_scrub_stamp":"2024-05-28T06:26:56.507468+0000","last_clean_scrub_stamp":"2024-05-28T06:26:56.507468+0000","objects_scrubbed":82,"log_size":227,"log_dups_size":0,"ondisk_log_size":227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:05:23.174789+0000","scrub_duration":1387,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":343932928,"num_objects":82,"num_object_clones":0,"num_object_copies":164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82,"num_whiteouts":0,"num_read":41,"num_read_kb":167936,"num_write":88,"num_write_kb":348160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":28,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:50.223761+0000","last_change":"2024-05-28T06:26:50.223790+0000","last_active":"2024-05-28T06:26:50.223761+0000","last_peered":"2024-05-28T06:26:50.223761+0000","last_clean":"2024-05-28T06:26:50.223761+0000","last_became_active":"2024-05-28T06:24:22.991078+0000","last_became_peered":"2024-05-28T06:24:22.991078+0000","last_unstale":"2024-05-28T06:26:50.223761+0000","last_undegraded":"2024-05-28T06:26:50.223761+0000","last_fullsized":"2024-05-28T06:26:50.223761+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:26:50.223647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:26:50.223647+0000","last_clean_scrub_stamp":"2024-05-28T06:26:50.223647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T08:50:58.665376+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"132'193","reported_seq":277,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:27:00.112713+0000","last_change":"2024-05-28T06:27:00.112831+0000","last_active":"2024-05-28T06:27:00.112713+0000","last_peered":"2024-05-28T06:27:00.112713+0000","last_clean":"2024-05-28T06:27:00.112713+0000","last_became_active":"2024-05-28T06:24:24.008955+0000","last_became_peered":"2024-05-28T06:24:24.008955+0000","last_unstale":"2024-05-28T06:27:00.112713+0000","last_undegraded":"2024-05-28T06:27:00.112713+0000","last_fullsized":"2024-05-28T06:27:00.112713+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'193","last_scrub_stamp":"2024-05-28T06:27:00.112549+0000","last_deep_scrub":"132'193","last_deep_scrub_stamp":"2024-05-28T06:27:00.112549+0000","last_clean_scrub_stamp":"2024-05-28T06:27:00.112549+0000","objects_scrubbed":66,"log_size":193,"log_dups_size":0,"ondisk_log_size":193,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:31:29.544285+0000","scrub_duration":888,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":276824064,"num_objects":66,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":56,"num_read_kb":229376,"num_write":68,"num_write_kb":274432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":28,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:51.507384+0000","last_change":"2024-05-28T06:26:51.507420+0000","last_active":"2024-05-28T06:26:51.507384+0000","last_peered":"2024-05-28T06:26:51.507384+0000","last_clean":"2024-05-28T06:26:51.507384+0000","last_became_active":"2024-05-28T06:24:23.002292+0000","last_became_peered":"2024-05-28T06:24:23.002292+0000","last_unstale":"2024-05-28T06:26:51.507384+0000","last_undegraded":"2024-05-28T06:26:51.507384+0000","last_fullsized":"2024-05-28T06:26:51.507384+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:26:51.507251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:26:51.507251+0000","last_clean_scrub_stamp":"2024-05-28T06:26:51.507251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:31:20.938988+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.6","version":"132'219","reported_seq":282,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:58.995533+0000","last_change":"2024-05-28T06:26:58.995701+0000","last_active":"2024-05-28T06:26:58.995533+0000","last_peered":"2024-05-28T06:26:58.995533+0000","last_clean":"2024-05-28T06:26:58.995533+0000","last_became_active":"2024-05-28T06:24:24.017221+0000","last_became_peered":"2024-05-28T06:24:24.017221+0000","last_unstale":"2024-05-28T06:26:58.995533+0000","last_undegraded":"2024-05-28T06:26:58.995533+0000","last_fullsized":"2024-05-28T06:26:58.995533+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'219","last_scrub_stamp":"2024-05-28T06:26:58.995379+0000","last_deep_scrub":"132'219","last_deep_scrub_stamp":"2024-05-28T06:26:58.995379+0000","last_clean_scrub_stamp":"2024-05-28T06:26:58.995379+0000","objects_scrubbed":73,"log_size":219,"log_dups_size":0,"ondisk_log_size":219,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:16:59.036876+0000","scrub_duration":916,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306184192,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":35,"num_read_kb":143360,"num_write":87,"num_write_kb":327680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.7","version":"132'3","reported_seq":31,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:49.651675+0000","last_change":"2024-05-28T06:26:49.651701+0000","last_active":"2024-05-28T06:26:49.651675+0000","last_peered":"2024-05-28T06:26:49.651675+0000","last_clean":"2024-05-28T06:26:49.651675+0000","last_became_active":"2024-05-28T06:24:23.003502+0000","last_became_peered":"2024-05-28T06:24:23.003502+0000","last_unstale":"2024-05-28T06:26:49.651675+0000","last_undegraded":"2024-05-28T06:26:49.651675+0000","last_fullsized":"2024-05-28T06:26:49.651675+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'3","last_scrub_stamp":"2024-05-28T06:26:49.651607+0000","last_deep_scrub":"132'3","last_deep_scrub_stamp":"2024-05-28T06:26:49.651607+0000","last_clean_scrub_stamp":"2024-05-28T06:26:49.651607+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T10:53:07.376780+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.7","version":"132'209","reported_seq":308,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:27:00.658454+0000","last_change":"2024-05-28T06:27:00.658624+0000","last_active":"2024-05-28T06:27:00.658454+0000","last_peered":"2024-05-28T06:27:00.658454+0000","last_clean":"2024-05-28T06:27:00.658454+0000","last_became_active":"2024-05-28T06:24:24.019730+0000","last_became_peered":"2024-05-28T06:24:24.019730+0000","last_unstale":"2024-05-28T06:27:00.658454+0000","last_undegraded":"2024-05-28T06:27:00.658454+0000","last_fullsized":"2024-05-28T06:27:00.658454+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'209","last_scrub_stamp":"2024-05-28T06:27:00.658318+0000","last_deep_scrub":"132'209","last_deep_scrub_stamp":"2024-05-28T06:27:00.658318+0000","last_clean_scrub_stamp":"2024-05-28T06:27:00.658318+0000","objects_scrubbed":74,"log_size":209,"log_dups_size":0,"ondisk_log_size":209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:37:22.633941+0000","scrub_duration":1008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":310378496,"num_objects":74,"num_object_clones":0,"num_object_copies":148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74,"num_whiteouts":0,"num_read":70,"num_read_kb":286720,"num_write":84,"num_write_kb":323584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.6","version":"132'44","reported_seq":72,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:53.187800+0000","last_change":"2024-05-28T06:26:53.187800+0000","last_active":"2024-05-28T06:26:53.187800+0000","last_peered":"2024-05-28T06:26:53.187800+0000","last_clean":"2024-05-28T06:26:53.187800+0000","last_became_active":"2024-05-28T06:24:23.002821+0000","last_became_peered":"2024-05-28T06:24:23.002821+0000","last_unstale":"2024-05-28T06:26:53.187800+0000","last_undegraded":"2024-05-28T06:26:53.187800+0000","last_fullsized":"2024-05-28T06:26:53.187800+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'43","last_scrub_stamp":"2024-05-28T06:26:53.187532+0000","last_deep_scrub":"132'43","last_deep_scrub_stamp":"2024-05-28T06:26:53.187532+0000","last_clean_scrub_stamp":"2024-05-28T06:26:53.187532+0000","objects_scrubbed":1,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:59:51.350170+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":930,"num_omap_keys":16,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.4","version":"132'203","reported_seq":319,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:59.942665+0000","last_change":"2024-05-28T06:26:59.942728+0000","last_active":"2024-05-28T06:26:59.942665+0000","last_peered":"2024-05-28T06:26:59.942665+0000","last_clean":"2024-05-28T06:26:59.942665+0000","last_became_active":"2024-05-28T06:24:24.004599+0000","last_became_peered":"2024-05-28T06:24:24.004599+0000","last_unstale":"2024-05-28T06:26:59.942665+0000","last_undegraded":"2024-05-28T06:26:59.942665+0000","last_fullsized":"2024-05-28T06:26:59.942665+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'203","last_scrub_stamp":"2024-05-28T06:26:59.942570+0000","last_deep_scrub":"132'203","last_deep_scrub_stamp":"2024-05-28T06:26:59.942570+0000","last_clean_scrub_stamp":"2024-05-28T06:26:59.942570+0000","objects_scrubbed":66,"log_size":203,"log_dups_size":0,"ondisk_log_size":203,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T18:12:18.259804+0000","scrub_duration":857,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":276824064,"num_objects":66,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":87,"num_read_kb":356352,"num_write":80,"num_write_kb":294913,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":28,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:51.008350+0000","last_change":"2024-05-28T06:26:51.008412+0000","last_active":"2024-05-28T06:26:51.008350+0000","last_peered":"2024-05-28T06:26:51.008350+0000","last_clean":"2024-05-28T06:26:51.008350+0000","last_became_active":"2024-05-28T06:24:23.002468+0000","last_became_peered":"2024-05-28T06:24:23.002468+0000","last_unstale":"2024-05-28T06:26:51.008350+0000","last_undegraded":"2024-05-28T06:26:51.008350+0000","last_fullsized":"2024-05-28T06:26:51.008350+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:26:51.008230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:26:51.008230+0000","last_clean_scrub_stamp":"2024-05-28T06:26:51.008230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:56:35.776423+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.5","version":"132'216","reported_seq":291,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:56.265345+0000","last_change":"2024-05-28T06:26:56.265477+0000","last_active":"2024-05-28T06:26:56.265345+0000","last_peered":"2024-05-28T06:26:56.265345+0000","last_clean":"2024-05-28T06:26:56.265345+0000","last_became_active":"2024-05-28T06:24:23.999013+0000","last_became_peered":"2024-05-28T06:24:23.999013+0000","last_unstale":"2024-05-28T06:26:56.265345+0000","last_undegraded":"2024-05-28T06:26:56.265345+0000","last_fullsized":"2024-05-28T06:26:56.265345+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'216","last_scrub_stamp":"2024-05-28T06:26:56.265208+0000","last_deep_scrub":"132'216","last_deep_scrub_stamp":"2024-05-28T06:26:56.265208+0000","last_clean_scrub_stamp":"2024-05-28T06:26:56.265208+0000","objects_scrubbed":72,"log_size":216,"log_dups_size":0,"ondisk_log_size":216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:59:54.427846+0000","scrub_duration":1193,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":285213316,"num_objects":72,"num_object_clones":1,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72,"num_whiteouts":1,"num_read":47,"num_read_kb":192512,"num_write":84,"num_write_kb":303107,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":28,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:47.544478+0000","last_change":"2024-05-28T06:26:47.544535+0000","last_active":"2024-05-28T06:26:47.544478+0000","last_peered":"2024-05-28T06:26:47.544478+0000","last_clean":"2024-05-28T06:26:47.544478+0000","last_became_active":"2024-05-28T06:24:22.986620+0000","last_became_peered":"2024-05-28T06:24:22.986620+0000","last_unstale":"2024-05-28T06:26:47.544478+0000","last_undegraded":"2024-05-28T06:26:47.544478+0000","last_fullsized":"2024-05-28T06:26:47.544478+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:26:47.544229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:26:47.544229+0000","last_clean_scrub_stamp":"2024-05-28T06:26:47.544229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:00:30.600799+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2","version":"132'225","reported_seq":306,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:27:03.223202+0000","last_change":"2024-05-28T06:27:03.223292+0000","last_active":"2024-05-28T06:27:03.223202+0000","last_peered":"2024-05-28T06:27:03.223202+0000","last_clean":"2024-05-28T06:27:03.223202+0000","last_became_active":"2024-05-28T06:24:24.011989+0000","last_became_peered":"2024-05-28T06:24:24.011989+0000","last_unstale":"2024-05-28T06:27:03.223202+0000","last_undegraded":"2024-05-28T06:27:03.223202+0000","last_fullsized":"2024-05-28T06:27:03.223202+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'225","last_scrub_stamp":"2024-05-28T06:27:03.222971+0000","last_deep_scrub":"132'225","last_deep_scrub_stamp":"2024-05-28T06:27:03.222971+0000","last_clean_scrub_stamp":"2024-05-28T06:27:03.222971+0000","objects_scrubbed":84,"log_size":225,"log_dups_size":0,"ondisk_log_size":225,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T14:08:47.621437+0000","scrub_duration":992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":352321536,"num_objects":84,"num_object_clones":0,"num_object_copies":168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84,"num_whiteouts":0,"num_read":52,"num_read_kb":212992,"num_write":90,"num_write_kb":356352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":28,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:53.085901+0000","last_change":"2024-05-28T06:26:53.085927+0000","last_active":"2024-05-28T06:26:53.085901+0000","last_peered":"2024-05-28T06:26:53.085901+0000","last_clean":"2024-05-28T06:26:53.085901+0000","last_became_active":"2024-05-28T06:24:23.007859+0000","last_became_peered":"2024-05-28T06:24:23.007859+0000","last_unstale":"2024-05-28T06:26:53.085901+0000","last_undegraded":"2024-05-28T06:26:53.085901+0000","last_fullsized":"2024-05-28T06:26:53.085901+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:26:53.085748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:26:53.085748+0000","last_clean_scrub_stamp":"2024-05-28T06:26:53.085748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:40:40.130865+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"132'245","reported_seq":357,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:58.636848+0000","last_change":"2024-05-28T06:26:58.636983+0000","last_active":"2024-05-28T06:26:58.636848+0000","last_peered":"2024-05-28T06:26:58.636848+0000","last_clean":"2024-05-28T06:26:58.636848+0000","last_became_active":"2024-05-28T06:24:24.011269+0000","last_became_peered":"2024-05-28T06:24:24.011269+0000","last_unstale":"2024-05-28T06:26:58.636848+0000","last_undegraded":"2024-05-28T06:26:58.636848+0000","last_fullsized":"2024-05-28T06:26:58.636848+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'245","last_scrub_stamp":"2024-05-28T06:26:58.636673+0000","last_deep_scrub":"132'245","last_deep_scrub_stamp":"2024-05-28T06:26:58.636673+0000","last_clean_scrub_stamp":"2024-05-28T06:26:58.636673+0000","objects_scrubbed":88,"log_size":245,"log_dups_size":0,"ondisk_log_size":245,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:33:54.340678+0000","scrub_duration":1091,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369098752,"num_objects":88,"num_object_clones":0,"num_object_copies":176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88,"num_whiteouts":0,"num_read":83,"num_read_kb":327683,"num_write":98,"num_write_kb":376833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":28,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:55.151070+0000","last_change":"2024-05-28T06:26:55.151127+0000","last_active":"2024-05-28T06:26:55.151070+0000","last_peered":"2024-05-28T06:26:55.151070+0000","last_clean":"2024-05-28T06:26:55.151070+0000","last_became_active":"2024-05-28T06:24:23.005895+0000","last_became_peered":"2024-05-28T06:24:23.005895+0000","last_unstale":"2024-05-28T06:26:55.151070+0000","last_undegraded":"2024-05-28T06:26:55.151070+0000","last_fullsized":"2024-05-28T06:26:55.151070+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:26:55.150969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:26:55.150969+0000","last_clean_scrub_stamp":"2024-05-28T06:26:55.150969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T13:00:38.207539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"132'210","reported_seq":310,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:27:01.274455+0000","last_change":"2024-05-28T06:27:01.274516+0000","last_active":"2024-05-28T06:27:01.274455+0000","last_peered":"2024-05-28T06:27:01.274455+0000","last_clean":"2024-05-28T06:27:01.274455+0000","last_became_active":"2024-05-28T06:24:24.012814+0000","last_became_peered":"2024-05-28T06:24:24.012814+0000","last_unstale":"2024-05-28T06:27:01.274455+0000","last_undegraded":"2024-05-28T06:27:01.274455+0000","last_fullsized":"2024-05-28T06:27:01.274455+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'210","last_scrub_stamp":"2024-05-28T06:27:01.274334+0000","last_deep_scrub":"132'210","last_deep_scrub_stamp":"2024-05-28T06:27:01.274334+0000","last_clean_scrub_stamp":"2024-05-28T06:27:01.274334+0000","objects_scrubbed":81,"log_size":210,"log_dups_size":0,"ondisk_log_size":210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T09:31:30.706070+0000","scrub_duration":1037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":339738624,"num_objects":81,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81,"num_whiteouts":0,"num_read":71,"num_read_kb":290816,"num_write":87,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1","version":"132'32","reported_seq":90,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:53.322484+0000","last_change":"2024-05-28T06:26:53.322508+0000","last_active":"2024-05-28T06:26:53.322484+0000","last_peered":"2024-05-28T06:26:53.322484+0000","last_clean":"2024-05-28T06:26:53.322484+0000","last_became_active":"2024-05-28T06:24:23.000861+0000","last_became_peered":"2024-05-28T06:24:23.000861+0000","last_unstale":"2024-05-28T06:26:53.322484+0000","last_undegraded":"2024-05-28T06:26:53.322484+0000","last_fullsized":"2024-05-28T06:26:53.322484+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'32","last_scrub_stamp":"2024-05-28T06:26:53.322374+0000","last_deep_scrub":"132'32","last_deep_scrub_stamp":"2024-05-28T06:26:53.322374+0000","last_clean_scrub_stamp":"2024-05-28T06:26:53.322374+0000","objects_scrubbed":1,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:14:33.167798+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4553,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":30,"num_read_kb":30,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1","version":"132'211","reported_seq":316,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:27:01.968237+0000","last_change":"2024-05-28T06:27:01.968336+0000","last_active":"2024-05-28T06:27:01.968237+0000","last_peered":"2024-05-28T06:27:01.968237+0000","last_clean":"2024-05-28T06:27:01.968237+0000","last_became_active":"2024-05-28T06:24:24.005485+0000","last_became_peered":"2024-05-28T06:24:24.005485+0000","last_unstale":"2024-05-28T06:27:01.968237+0000","last_undegraded":"2024-05-28T06:27:01.968237+0000","last_fullsized":"2024-05-28T06:27:01.968237+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'211","last_scrub_stamp":"2024-05-28T06:27:01.968111+0000","last_deep_scrub":"132'211","last_deep_scrub_stamp":"2024-05-28T06:27:01.968111+0000","last_clean_scrub_stamp":"2024-05-28T06:27:01.968111+0000","objects_scrubbed":75,"log_size":211,"log_dups_size":0,"ondisk_log_size":211,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T11:19:11.922402+0000","scrub_duration":866,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":314572800,"num_objects":75,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75,"num_whiteouts":0,"num_read":76,"num_read_kb":311296,"num_write":93,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":28,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:48.512890+0000","last_change":"2024-05-28T06:26:48.512938+0000","last_active":"2024-05-28T06:26:48.512890+0000","last_peered":"2024-05-28T06:26:48.512890+0000","last_clean":"2024-05-28T06:26:48.512890+0000","last_became_active":"2024-05-28T06:24:22.987181+0000","last_became_peered":"2024-05-28T06:24:22.987181+0000","last_unstale":"2024-05-28T06:26:48.512890+0000","last_undegraded":"2024-05-28T06:26:48.512890+0000","last_fullsized":"2024-05-28T06:26:48.512890+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-28T06:26:48.512762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-28T06:26:48.512762+0000","last_clean_scrub_stamp":"2024-05-28T06:26:48.512762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:47:34.449612+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.12","version":"132'170","reported_seq":245,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:58.946706+0000","last_change":"2024-05-28T06:26:58.946786+0000","last_active":"2024-05-28T06:26:58.946706+0000","last_peered":"2024-05-28T06:26:58.946706+0000","last_clean":"2024-05-28T06:26:58.946706+0000","last_became_active":"2024-05-28T06:24:24.007651+0000","last_became_peered":"2024-05-28T06:24:24.007651+0000","last_unstale":"2024-05-28T06:26:58.946706+0000","last_undegraded":"2024-05-28T06:26:58.946706+0000","last_fullsized":"2024-05-28T06:26:58.946706+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'170","last_scrub_stamp":"2024-05-28T06:26:58.946628+0000","last_deep_scrub":"132'170","last_deep_scrub_stamp":"2024-05-28T06:26:58.946628+0000","last_clean_scrub_stamp":"2024-05-28T06:26:58.946628+0000","objects_scrubbed":61,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T16:18:33.031733+0000","scrub_duration":761,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":255852544,"num_objects":61,"num_object_clones":0,"num_object_copies":122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":0,"num_read":47,"num_read_kb":192512,"num_write":65,"num_write_kb":258048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"132'4","reported_seq":32,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:56.155740+0000","last_change":"2024-05-28T06:26:56.155772+0000","last_active":"2024-05-28T06:26:56.155740+0000","last_peered":"2024-05-28T06:26:56.155740+0000","last_clean":"2024-05-28T06:26:56.155740+0000","last_became_active":"2024-05-28T06:24:23.007331+0000","last_became_peered":"2024-05-28T06:24:23.007331+0000","last_unstale":"2024-05-28T06:26:56.155740+0000","last_undegraded":"2024-05-28T06:26:56.155740+0000","last_fullsized":"2024-05-28T06:26:56.155740+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'4","last_scrub_stamp":"2024-05-28T06:26:56.155622+0000","last_deep_scrub":"132'4","last_deep_scrub_stamp":"2024-05-28T06:26:56.155622+0000","last_clean_scrub_stamp":"2024-05-28T06:26:56.155622+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T15:47:42.092473+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"132'119","reported_seq":399,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-05-28T06:26:52.309561+0000","last_change":"2024-05-28T06:26:52.309824+0000","last_active":"2024-05-28T06:26:52.309561+0000","last_peered":"2024-05-28T06:26:52.309561+0000","last_clean":"2024-05-28T06:26:52.309561+0000","last_became_active":"2024-05-28T06:04:01.033906+0000","last_became_peered":"2024-05-28T06:04:01.033906+0000","last_unstale":"2024-05-28T06:26:52.309561+0000","last_undegraded":"2024-05-28T06:26:52.309561+0000","last_fullsized":"2024-05-28T06:26:52.309561+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'119","last_scrub_stamp":"2024-05-28T06:26:52.309456+0000","last_deep_scrub":"132'119","last_deep_scrub_stamp":"2024-05-28T06:26:52.309456+0000","last_clean_scrub_stamp":"2024-05-28T06:26:52.309456+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-29T12:14:32.154881+0000","scrub_duration":176,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":18396218954,"num_objects":4395,"num_object_clones":1,"num_object_copies":8790,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4395,"num_whiteouts":1,"num_read":3946,"num_read_kb":15908926,"num_write":4949,"num_write_kb":18956331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":41943040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36792500224,"data_stored":36792437908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12853,"ondisk_log_size":12853,"up":128,"acting":128,"num_store_stats":8},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":1680395,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":30,"num_read_kb":30,"num_write":619,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5930,"num_omap_keys":26,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3424256,"data_stored":3360790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":582,"ondisk_log_size":582,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575129,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3956688,"kb_used_data":3813128,"kb_used_omap":1,"kb_used_meta":143550,"kb_avail":90415152,"statfs":{"total":96636764160,"available":92585115648,"internally_reserved":0,"allocated":3904643072,"data_stored":3903696199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":146995658},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575131,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5143948,"kb_used_data":5000388,"kb_used_omap":1,"kb_used_meta":143550,"kb_avail":89227892,"statfs":{"total":96636764160,"available":91369361408,"internally_reserved":0,"allocated":5120397312,"data_stored":5119457909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":146995654},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575129,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3819292,"kb_used_data":3675732,"kb_used_omap":1,"kb_used_meta":143550,"kb_avail":90552548,"statfs":{"total":96636764160,"available":92725809152,"internally_reserved":0,"allocated":3763949568,"data_stored":3763018100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":146995653},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575129,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5551852,"kb_used_data":5412452,"kb_used_omap":1,"kb_used_meta":139390,"kb_avail":88819988,"statfs":{"total":96636764160,"available":90951667712,"internally_reserved":0,"allocated":5542350848,"data_stored":5541404120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":142735813},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575131,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4532004,"kb_used_data":4388444,"kb_used_omap":1,"kb_used_meta":143550,"kb_avail":89839836,"statfs":{"total":96636764160,"available":91995992064,"internally_reserved":0,"allocated":4493766656,"data_stored":4492831645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":146995655},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575130,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5987752,"kb_used_data":5826144,"kb_used_omap":1,"kb_used_meta":161598,"kb_avail":88384088,"statfs":{"total":96636764160,"available":90505306112,"internally_reserved":0,"allocated":5965971456,"data_stored":5965028896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":165476805},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575130,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4159184,"kb_used_data":4019784,"kb_used_omap":1,"kb_used_meta":139390,"kb_avail":90212656,"statfs":{"total":96636764160,"available":92377759744,"internally_reserved":0,"allocated":4116258816,"data_stored":4115339392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":142735813},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575129,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3946712,"kb_used_data":3811472,"kb_used_omap":1,"kb_used_meta":135230,"kb_avail":90425128,"statfs":{"total":96636764160,"available":92595331072,"internally_reserved":0,"allocated":3902947328,"data_stored":3902020517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":138475973},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1691648,"data_stored":1679597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1695744,"data_stored":1675682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3900706816,"data_stored":3900702981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4114612224,"data_stored":4114612224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":5964316672,"data_stored":5964301112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4492103680,"data_stored":4492099764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":5540700160,"data_stored":5540676882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3762294784,"data_stored":3762290820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":5117059072,"data_stored":5117051144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3900706816,"data_stored":3900702981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-28T06:27:13.186 DEBUG:teuthology.orchestra.run.smithi096:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-05-28T06:27:13.415 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.422+0000 7f1831a00640 1 Processor -- start 2024-05-28T06:27:13.415 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.423+0000 7f1831a00640 1 -- start start 2024-05-28T06:27:13.416 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.424+0000 7f1831a00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f182c0cfcc0 0x7f182c0ccc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:27:13.416 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.424+0000 7f1831a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f182c0d0440 0x7f182c0d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:27:13.416 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.424+0000 7f1831a00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f182c0cd2a0 con 0x7f182c145e60 2024-05-28T06:27:13.416 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.424+0000 7f1831a00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f182c0d0df0 con 0x7f182c0cfcc0 2024-05-28T06:27:13.417 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.424+0000 7f1831a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f182c0cdeb0 con 0x7f182c0d0440 2024-05-28T06:27:13.417 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.425+0000 7f1831000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f182c0d0440 0x7f182c0d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:27:13.417 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.425+0000 7f182b400640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f182c0cfcc0 0x7f182c0ccc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:27:13.417 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.425+0000 7f1831000640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f182c0d0440 0x7f182c0d08b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.96:48564/0 (socket says 172.21.15.96:48564) 2024-05-28T06:27:13.418 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.425+0000 7f1831000640 1 -- 172.21.15.96:0/1212848278 learned_addr learned my addr 172.21.15.96:0/1212848278 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:27:13.418 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.425+0000 7f1831000640 1 -- 172.21.15.96:0/1212848278 >> v2:172.21.15.123:3300/0 conn(0x7f182c0cfcc0 msgr2=0x7f182c0ccc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:27:13.418 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.425+0000 7f1831000640 1 --2- 172.21.15.96:0/1212848278 >> v2:172.21.15.123:3300/0 conn(0x7f182c0cfcc0 0x7f182c0ccc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:27:13.418 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.425+0000 7f1831000640 1 -- 172.21.15.96:0/1212848278 >> v1:172.21.15.123:6789/0 conn(0x7f182c145e60 legacy=0x7f182c0ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:27:13.418 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.426+0000 7f1831000640 1 -- 172.21.15.96:0/1212848278 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f182c0cd580 con 0x7f182c0d0440 2024-05-28T06:27:13.419 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.427+0000 7f1831000640 1 --2- 172.21.15.96:0/1212848278 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f182c0d0440 0x7f182c0d08b0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f1824009870 tx=0x7f182402a1c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fae661b96a9dff9e server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:27:13.420 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.428+0000 7f182aa00640 1 -- 172.21.15.96:0/1212848278 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1824037070 con 0x7f182c0d0440 2024-05-28T06:27:13.420 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.428+0000 7f182aa00640 1 -- 172.21.15.96:0/1212848278 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1824032440 con 0x7f182c0d0440 2024-05-28T06:27:13.420 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.428+0000 7f182aa00640 1 -- 172.21.15.96:0/1212848278 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1824032810 con 0x7f182c0d0440 2024-05-28T06:27:13.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.428+0000 7f1831a00640 1 -- 172.21.15.96:0/1212848278 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f182c0d0440 msgr2=0x7f182c0d08b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:27:13.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.428+0000 7f1831a00640 1 --2- 172.21.15.96:0/1212848278 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f182c0d0440 0x7f182c0d08b0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f1824009870 tx=0x7f182402a1c0 comp rx=0 tx=0).stop 2024-05-28T06:27:13.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.428+0000 7f1831a00640 1 -- 172.21.15.96:0/1212848278 shutdown_connections 2024-05-28T06:27:13.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.428+0000 7f1831a00640 1 --2- 172.21.15.96:0/1212848278 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f182c0d0440 0x7f182c0d08b0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:27:13.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.428+0000 7f1831a00640 1 --2- 172.21.15.96:0/1212848278 >> v2:172.21.15.123:3300/0 conn(0x7f182c0cfcc0 0x7f182c0ccc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:27:13.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.428+0000 7f1831a00640 1 -- 172.21.15.96:0/1212848278 >> 172.21.15.96:0/1212848278 conn(0x7f182c0b2ed0 msgr2=0x7f182c0b32e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:27:13.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.429+0000 7f1831a00640 1 -- 172.21.15.96:0/1212848278 shutdown_connections 2024-05-28T06:27:13.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.429+0000 7f1831a00640 1 -- 172.21.15.96:0/1212848278 wait complete. 2024-05-28T06:27:13.422 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.430+0000 7f1831a00640 1 Processor -- start 2024-05-28T06:27:13.423 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.430+0000 7f1831a00640 1 -- start start 2024-05-28T06:27:13.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.431+0000 7f1831a00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f182c0cfcc0 0x7f182c141b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:27:13.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.431+0000 7f1831a00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f182c0d0440 0x7f182c09bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:27:13.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.431+0000 7f1831a00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f182c145e60 0x7f182c0ca550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:27:13.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.431+0000 7f1831a00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f182c0cea50 con 0x7f182c0cfcc0 2024-05-28T06:27:13.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.431+0000 7f1831a00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f182c0cdb40 con 0x7f182c145e60 2024-05-28T06:27:13.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.431+0000 7f1831a00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f182c0cf000 con 0x7f182c0d0440 2024-05-28T06:27:13.424 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.432+0000 7f182be00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f182c0cfcc0 0x7f182c141b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:27:13.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.432+0000 7f182be00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f182c0cfcc0 0x7f182c141b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:52196/0 (socket says 172.21.15.96:52196) 2024-05-28T06:27:13.425 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.432+0000 7f182be00640 1 -- 172.21.15.96:0/2825719143 learned_addr learned my addr 172.21.15.96:0/2825719143 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:27:13.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.432+0000 7f182b400640 1 --2- 172.21.15.96:0/2825719143 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f182c0d0440 0x7f182c09bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:27:13.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.432+0000 7f1831000640 1 --2- 172.21.15.96:0/2825719143 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f182c145e60 0x7f182c0ca550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:27:13.426 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.432+0000 7f182be00640 1 -- 172.21.15.96:0/2825719143 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f182c0d0440 msgr2=0x7f182c09bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:27:13.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.432+0000 7f182be00640 1 --2- 172.21.15.96:0/2825719143 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f182c0d0440 0x7f182c09bef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:27:13.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.432+0000 7f182be00640 1 -- 172.21.15.96:0/2825719143 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f182c145e60 msgr2=0x7f182c0ca550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:27:13.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.432+0000 7f182be00640 1 --2- 172.21.15.96:0/2825719143 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f182c145e60 0x7f182c0ca550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:27:13.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.432+0000 7f182be00640 1 -- 172.21.15.96:0/2825719143 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f182c09c7f0 con 0x7f182c0cfcc0 2024-05-28T06:27:13.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.432+0000 7f1831000640 1 --2- 172.21.15.96:0/2825719143 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f182c145e60 0x7f182c0ca550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-28T06:27:13.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.432+0000 7f182be00640 1 --2- 172.21.15.96:0/2825719143 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f182c0cfcc0 0x7f182c141b90 secure :-1 s=READY pgs=2235 cs=0 l=1 rev1=1 crypto rx=0x7f181c00c730 tx=0x7f181c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:27:13.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.433+0000 7f1823e00640 1 -- 172.21.15.96:0/2825719143 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f181c016020 con 0x7f182c0cfcc0 2024-05-28T06:27:13.427 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.433+0000 7f1823e00640 1 -- 172.21.15.96:0/2825719143 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f181c006d50 con 0x7f182c0cfcc0 2024-05-28T06:27:13.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.433+0000 7f1823e00640 1 -- 172.21.15.96:0/2825719143 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f181c003bb0 con 0x7f182c0cfcc0 2024-05-28T06:27:13.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.433+0000 7f1831a00640 1 -- 172.21.15.96:0/2825719143 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f182c1f4590 con 0x7f182c0cfcc0 2024-05-28T06:27:13.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.433+0000 7f1831a00640 1 -- 172.21.15.96:0/2825719143 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f182c1f4b50 con 0x7f182c0cfcc0 2024-05-28T06:27:13.428 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.434+0000 7f1831a00640 1 -- 172.21.15.96:0/2825719143 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f182c0690f0 con 0x7f182c0cfcc0 2024-05-28T06:27:13.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.435+0000 7f1823e00640 1 -- 172.21.15.96:0/2825719143 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 51) ==== 93743+0+0 (secure 0 0 0) 0x7f181c00a850 con 0x7f182c0cfcc0 2024-05-28T06:27:13.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.436+0000 7f1823e00640 1 --2- 172.21.15.96:0/2825719143 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f180806e6c0 0x7f1808070b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:27:13.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.436+0000 7f1823e00640 1 -- 172.21.15.96:0/2825719143 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 5823+0+0 (secure 0 0 0) 0x7f181c08a7c0 con 0x7f182c0cfcc0 2024-05-28T06:27:13.429 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.437+0000 7f182b400640 1 --2- 172.21.15.96:0/2825719143 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f180806e6c0 0x7f1808070b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:27:13.430 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.438+0000 7f182b400640 1 --2- 172.21.15.96:0/2825719143 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f180806e6c0 0x7f1808070b80 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f182c144070 tx=0x7f1814008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:27:13.433 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.441+0000 7f1823e00640 1 -- 172.21.15.96:0/2825719143 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f181c020020 con 0x7f182c0cfcc0 2024-05-28T06:27:13.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.639+0000 7f1831a00640 1 -- 172.21.15.96:0/2825719143 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7f182c0b0bc0 con 0x7f182c0cfcc0 2024-05-28T06:27:13.638 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.645+0000 7f1823e00640 1 -- 172.21.15.96:0/2825719143 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v30)=0 v30) ==== 123+0+0 (secure 0 0 0) 0x7f181c019310 con 0x7f182c0cfcc0 2024-05-28T06:27:13.640 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.646+0000 7f1823e00640 1 -- 172.21.15.96:0/2825719143 <== mon.0 v2:172.21.15.96:3300/0 8 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f181c00ce40 con 0x7f182c0cfcc0 2024-05-28T06:27:13.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.649+0000 7f1831a00640 1 -- 172.21.15.96:0/2825719143 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f180806e6c0 msgr2=0x7f1808070b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:27:13.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.649+0000 7f1831a00640 1 --2- 172.21.15.96:0/2825719143 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f180806e6c0 0x7f1808070b80 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f182c144070 tx=0x7f1814008040 comp rx=0 tx=0).stop 2024-05-28T06:27:13.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.649+0000 7f1831a00640 1 -- 172.21.15.96:0/2825719143 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f182c0cfcc0 msgr2=0x7f182c141b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:27:13.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.649+0000 7f1831a00640 1 --2- 172.21.15.96:0/2825719143 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f182c0cfcc0 0x7f182c141b90 secure :-1 s=READY pgs=2235 cs=0 l=1 rev1=1 crypto rx=0x7f181c00c730 tx=0x7f181c00cbf0 comp rx=0 tx=0).stop 2024-05-28T06:27:13.642 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.649+0000 7f1831a00640 1 -- 172.21.15.96:0/2825719143 shutdown_connections 2024-05-28T06:27:13.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.649+0000 7f1831a00640 1 --2- 172.21.15.96:0/2825719143 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f180806e6c0 0x7f1808070b80 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:27:13.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.649+0000 7f1831a00640 1 --2- 172.21.15.96:0/2825719143 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f182c145e60 0x7f182c0ca550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:27:13.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.649+0000 7f1831a00640 1 --2- 172.21.15.96:0/2825719143 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f182c0d0440 0x7f182c09bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:27:13.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.649+0000 7f1831a00640 1 --2- 172.21.15.96:0/2825719143 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f182c0cfcc0 0x7f182c141b90 unknown :-1 s=CLOSED pgs=2235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:27:13.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.649+0000 7f1831a00640 1 -- 172.21.15.96:0/2825719143 >> 172.21.15.96:0/2825719143 conn(0x7f182c0b2ed0 msgr2=0x7f182c0b01c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:27:13.643 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.649+0000 7f1831a00640 1 -- 172.21.15.96:0/2825719143 shutdown_connections 2024-05-28T06:27:13.644 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.649+0000 7f1831a00640 1 -- 172.21.15.96:0/2825719143 wait complete. 2024-05-28T06:27:13.661 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-05-28T06:27:13.661 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-28T06:27:13.857 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.865+0000 7f058cc00640 1 Processor -- start 2024-05-28T06:27:13.858 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.865+0000 7f058cc00640 1 -- start start 2024-05-28T06:27:13.860 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.866+0000 7f058cc00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f05880b4e10 0x7f05880cde30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:27:13.860 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.866+0000 7f058cc00640 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f05880ce370 0x7f05880d0590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:27:13.860 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.866+0000 7f058cc00640 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05880b51f0 con 0x7f05880d0ad0 2024-05-28T06:27:13.860 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.866+0000 7f058cc00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f05880d0f00 con 0x7f05880b4e10 2024-05-28T06:27:13.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.866+0000 7f058cc00640 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 -- 0x7f05880aef60 con 0x7f05880ce370 2024-05-28T06:27:13.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.867+0000 7f0585e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f05880b4e10 0x7f05880cde30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:27:13.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.867+0000 7f0585e00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f05880b4e10 0x7f05880cde30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:52212/0 (socket says 172.21.15.96:52212) 2024-05-28T06:27:13.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.867+0000 7f0585e00640 1 -- 172.21.15.96:0/513014580 learned_addr learned my addr 172.21.15.96:0/513014580 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:27:13.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.867+0000 7f057fe00640 1 --2- 172.21.15.96:0/513014580 >> v2:172.21.15.123:3300/0 conn(0x7f05880ce370 0x7f05880d0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:27:13.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.867+0000 7f0585e00640 1 -- 172.21.15.96:0/513014580 >> v2:172.21.15.123:3300/0 conn(0x7f05880ce370 msgr2=0x7f05880d0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:27:13.861 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.867+0000 7f0585e00640 1 --2- 172.21.15.96:0/513014580 >> v2:172.21.15.123:3300/0 conn(0x7f05880ce370 0x7f05880d0590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:27:13.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.867+0000 7f0585e00640 1 -- 172.21.15.96:0/513014580 >> v1:172.21.15.123:6789/0 conn(0x7f05880d0ad0 legacy=0x7f0588099090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-28T06:27:13.862 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.867+0000 7f0585e00640 1 -- 172.21.15.96:0/513014580 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05880ad1d0 con 0x7f05880b4e10 2024-05-28T06:27:13.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.869+0000 7f0585e00640 1 --2- 172.21.15.96:0/513014580 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f05880b4e10 0x7f05880cde30 secure :-1 s=READY pgs=2236 cs=0 l=1 rev1=1 crypto rx=0x7f05880ca860 tx=0x7f057802c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=785f31d40205a6ff server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:27:13.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.869+0000 7f0585400640 1 -- 172.21.15.96:0/513014580 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0578032710 con 0x7f05880b4e10 2024-05-28T06:27:13.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.869+0000 7f0585400640 1 -- 172.21.15.96:0/513014580 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f05780328b0 con 0x7f05880b4e10 2024-05-28T06:27:13.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.869+0000 7f0585400640 1 -- 172.21.15.96:0/513014580 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0578032cc0 con 0x7f05880b4e10 2024-05-28T06:27:13.863 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.870+0000 7f058cc00640 1 -- 172.21.15.96:0/513014580 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f05880b4e10 msgr2=0x7f05880cde30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:27:13.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.870+0000 7f058cc00640 1 --2- 172.21.15.96:0/513014580 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f05880b4e10 0x7f05880cde30 secure :-1 s=READY pgs=2236 cs=0 l=1 rev1=1 crypto rx=0x7f05880ca860 tx=0x7f057802c850 comp rx=0 tx=0).stop 2024-05-28T06:27:13.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.870+0000 7f058cc00640 1 -- 172.21.15.96:0/513014580 shutdown_connections 2024-05-28T06:27:13.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.870+0000 7f058cc00640 1 --2- 172.21.15.96:0/513014580 >> v2:172.21.15.123:3300/0 conn(0x7f05880ce370 0x7f05880d0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:27:13.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.870+0000 7f058cc00640 1 --2- 172.21.15.96:0/513014580 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f05880b4e10 0x7f05880cde30 unknown :-1 s=CLOSED pgs=2236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:27:13.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.870+0000 7f058cc00640 1 -- 172.21.15.96:0/513014580 >> 172.21.15.96:0/513014580 conn(0x7f058806bae0 msgr2=0x7f058806bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:27:13.864 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.870+0000 7f058cc00640 1 -- 172.21.15.96:0/513014580 shutdown_connections 2024-05-28T06:27:13.865 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.870+0000 7f058cc00640 1 -- 172.21.15.96:0/513014580 wait complete. 2024-05-28T06:27:13.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.872+0000 7f058cc00640 1 Processor -- start 2024-05-28T06:27:13.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.872+0000 7f058cc00640 1 -- start start 2024-05-28T06:27:13.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.873+0000 7f058cc00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f05880b4e10 0x7f058815e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:27:13.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.873+0000 7f058cc00640 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f05880ce370 0x7f05881f43c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:27:13.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.873+0000 7f058cc00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f05880d0ad0 0x7f05881f7f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:27:13.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.873+0000 7f058cc00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f05880ce880 con 0x7f05880d0ad0 2024-05-28T06:27:13.868 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.873+0000 7f058cc00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f05880ad750 con 0x7f05880b4e10 2024-05-28T06:27:13.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.873+0000 7f058cc00640 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 -- 0x7f05880cea00 con 0x7f05880ce370 2024-05-28T06:27:13.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.874+0000 7f0586800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f05880d0ad0 0x7f05881f7f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:27:13.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.874+0000 7f0586800640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f05880d0ad0 0x7f05881f7f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:52224/0 (socket says 172.21.15.96:52224) 2024-05-28T06:27:13.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.874+0000 7f0586800640 1 -- 172.21.15.96:0/3077720176 learned_addr learned my addr 172.21.15.96:0/3077720176 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-05-28T06:27:13.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.874+0000 7f0585e00640 1 --2- 172.21.15.96:0/3077720176 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f05880b4e10 0x7f058815e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:27:13.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.874+0000 7f0586800640 1 -- 172.21.15.96:0/3077720176 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f05880ce370 msgr2=0x7f05881f43c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:27:13.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.874+0000 7f057fe00640 1 --2- 172.21.15.96:0/3077720176 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f05880ce370 0x7f05881f43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:27:13.869 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.874+0000 7f0586800640 1 --2- 172.21.15.96:0/3077720176 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f05880ce370 0x7f05881f43c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:27:13.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.874+0000 7f0586800640 1 -- 172.21.15.96:0/3077720176 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f05880b4e10 msgr2=0x7f058815e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:27:13.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.874+0000 7f0586800640 1 --2- 172.21.15.96:0/3077720176 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f05880b4e10 0x7f058815e2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:27:13.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.874+0000 7f0586800640 1 -- 172.21.15.96:0/3077720176 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05880ca240 con 0x7f05880d0ad0 2024-05-28T06:27:13.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.874+0000 7f0586800640 1 --2- 172.21.15.96:0/3077720176 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f05880d0ad0 0x7f05881f7f70 secure :-1 s=READY pgs=2237 cs=0 l=1 rev1=1 crypto rx=0x7f058000b970 tx=0x7f058000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:27:13.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.875+0000 7f057e000640 1 -- 172.21.15.96:0/3077720176 <== mon.0 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0580019070 con 0x7f05880d0ad0 2024-05-28T06:27:13.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.875+0000 7f057e000640 1 -- 172.21.15.96:0/3077720176 <== mon.0 v2:172.21.15.96:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f058000cb40 con 0x7f05880d0ad0 2024-05-28T06:27:13.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.875+0000 7f058cc00640 1 -- 172.21.15.96:0/3077720176 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05880ca500 con 0x7f05880d0ad0 2024-05-28T06:27:13.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.875+0000 7f057e000640 1 -- 172.21.15.96:0/3077720176 <== mon.0 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0580014440 con 0x7f05880d0ad0 2024-05-28T06:27:13.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.875+0000 7f058cc00640 1 -- 172.21.15.96:0/3077720176 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f058815dac0 con 0x7f05880d0ad0 2024-05-28T06:27:13.871 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.876+0000 7f058cc00640 1 -- 172.21.15.96:0/3077720176 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05880689e0 con 0x7f05880d0ad0 2024-05-28T06:27:13.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.877+0000 7f057e000640 1 -- 172.21.15.96:0/3077720176 <== mon.0 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 51) ==== 93743+0+0 (secure 0 0 0) 0x7f058001c070 con 0x7f05880d0ad0 2024-05-28T06:27:13.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.878+0000 7f057e000640 1 --2- 172.21.15.96:0/3077720176 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f056406e6c0 0x7f0564070b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-28T06:27:13.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.878+0000 7f057e000640 1 -- 172.21.15.96:0/3077720176 <== mon.0 v2:172.21.15.96:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 5823+0+0 (secure 0 0 0) 0x7f058008a900 con 0x7f05880d0ad0 2024-05-28T06:27:13.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.878+0000 7f0585e00640 1 --2- 172.21.15.96:0/3077720176 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f056406e6c0 0x7f0564070b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-28T06:27:13.872 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.880+0000 7f0585e00640 1 --2- 172.21.15.96:0/3077720176 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f056406e6c0 0x7f0564070b80 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f0578037000 tx=0x7f0578032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-28T06:27:13.874 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:13.882+0000 7f057e000640 1 -- 172.21.15.96:0/3077720176 <== mon.0 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f058005a700 con 0x7f05880d0ad0 2024-05-28T06:27:14.120 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:14.128+0000 7f058cc00640 1 -- 172.21.15.96:0/3077720176 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f058806f540 con 0x7f05880d0ad0 2024-05-28T06:27:14.121 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:14.129+0000 7f057e000640 1 -- 172.21.15.96:0/3077720176 <== mon.0 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) ==== 94+0+5186 (secure 0 0 0) 0x7f058005a000 con 0x7f05880d0ad0 2024-05-28T06:27:14.121 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-05-28T06:27:14.121 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":83,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6287,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/227668757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":227668757},{"type":"v1","addr":"172.21.15.123:6835","nonce":227668757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":77},{"gid":6290,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/1637451105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":1637451105},{"type":"v1","addr":"172.21.15.123:6837","nonce":1637451105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":77},{"gid":9843,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2263843873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2263843873},{"type":"v1","addr":"172.21.15.96:6833","nonce":2263843873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-28T06:24:23.419984+0000","modified":"2024-05-28T06:24:26.498371+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9849},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9849":{"gid":9849,"name":"a","rank":0,"incarnation":78,"state":"up:active","state_seq":3,"addr":"172.21.15.96:6835/1720339900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":1720339900},{"type":"v1","addr":"172.21.15.96:6835","nonce":1720339900}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9849,"qdb_cluster":[9849]},"id":9}]} 2024-05-28T06:27:14.122 INFO:teuthology.orchestra.run.smithi096.stderr:dumped fsmap epoch 83 2024-05-28T06:27:14.124 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:14.132+0000 7f058cc00640 1 -- 172.21.15.96:0/3077720176 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f056406e6c0 msgr2=0x7f0564070b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:27:14.125 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:14.132+0000 7f058cc00640 1 --2- 172.21.15.96:0/3077720176 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f056406e6c0 0x7f0564070b80 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f0578037000 tx=0x7f0578032000 comp rx=0 tx=0).stop 2024-05-28T06:27:14.125 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:14.132+0000 7f058cc00640 1 -- 172.21.15.96:0/3077720176 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f05880d0ad0 msgr2=0x7f05881f7f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-28T06:27:14.125 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:14.132+0000 7f058cc00640 1 --2- 172.21.15.96:0/3077720176 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f05880d0ad0 0x7f05881f7f70 secure :-1 s=READY pgs=2237 cs=0 l=1 rev1=1 crypto rx=0x7f058000b970 tx=0x7f058000be30 comp rx=0 tx=0).stop 2024-05-28T06:27:14.125 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:14.132+0000 7f058cc00640 1 -- 172.21.15.96:0/3077720176 shutdown_connections 2024-05-28T06:27:14.125 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:14.132+0000 7f058cc00640 1 --2- 172.21.15.96:0/3077720176 >> [v2:172.21.15.123:6832/259466469,v1:172.21.15.123:6833/259466469] conn(0x7f056406e6c0 0x7f0564070b80 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:27:14.125 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:14.132+0000 7f058cc00640 1 --2- 172.21.15.96:0/3077720176 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f05880d0ad0 0x7f05881f7f70 unknown :-1 s=CLOSED pgs=2237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:27:14.125 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:14.132+0000 7f058cc00640 1 --2- 172.21.15.96:0/3077720176 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f05880ce370 0x7f05881f43c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:27:14.125 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:14.132+0000 7f058cc00640 1 --2- 172.21.15.96:0/3077720176 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f05880b4e10 0x7f058815e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-28T06:27:14.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:14.132+0000 7f058cc00640 1 -- 172.21.15.96:0/3077720176 >> 172.21.15.96:0/3077720176 conn(0x7f058806bae0 msgr2=0x7f058815b110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-28T06:27:14.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:14.132+0000 7f058cc00640 1 -- 172.21.15.96:0/3077720176 shutdown_connections 2024-05-28T06:27:14.126 INFO:teuthology.orchestra.run.smithi096.stderr:2024-05-28T06:27:14.132+0000 7f058cc00640 1 -- 172.21.15.96:0/3077720176 wait complete. 2024-05-28T06:27:14.144 DEBUG:tasks.cephfs.filesystem:already dead... 2024-05-28T06:27:14.144 INFO:teuthology.misc:Shutting down mds daemons... 2024-05-28T06:27:14.144 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-05-28T06:27:14.145 INFO:teuthology.orchestra.run:waiting for 300 2024-05-28T06:27:14.171 INFO:tasks.ceph.mds.a:Stopped 2024-05-28T06:27:14.172 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-05-28T06:27:14.172 INFO:teuthology.orchestra.run:waiting for 300 2024-05-28T06:27:14.235 INFO:tasks.ceph.mds.c:Stopped 2024-05-28T06:27:14.235 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-05-28T06:27:14.235 INFO:teuthology.orchestra.run:waiting for 300 2024-05-28T06:27:14.259 INFO:tasks.ceph.mds.b:Stopped 2024-05-28T06:27:14.259 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-05-28T06:27:14.259 INFO:teuthology.orchestra.run:waiting for 300 2024-05-28T06:27:14.331 INFO:tasks.ceph.mds.d:Stopped 2024-05-28T06:27:14.332 INFO:teuthology.misc:Shutting down osd daemons... 2024-05-28T06:27:14.332 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-05-28T06:27:14.332 INFO:teuthology.orchestra.run:waiting for 300 2024-05-28T06:27:14.403 INFO:tasks.ceph.osd.0:Stopped 2024-05-28T06:27:14.403 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-05-28T06:27:14.404 INFO:teuthology.orchestra.run:waiting for 300 2024-05-28T06:27:14.499 INFO:tasks.ceph.osd.1:Stopped 2024-05-28T06:27:14.499 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-05-28T06:27:14.500 INFO:teuthology.orchestra.run:waiting for 300 2024-05-28T06:27:14.533 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~0s 2024-05-28T06:27:20.543 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-05-28T06:27:20.544 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d has been restored 2024-05-28T06:27:20.601 INFO:tasks.ceph.osd.2:Stopped 2024-05-28T06:27:20.602 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-05-28T06:27:20.602 INFO:teuthology.orchestra.run:waiting for 300 2024-05-28T06:27:26.453 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2024-05-28T06:27:26.453 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2024-05-28T06:27:26.704 INFO:tasks.ceph.osd.3:Stopped 2024-05-28T06:27:26.705 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-05-28T06:27:26.705 INFO:teuthology.orchestra.run:waiting for 300 2024-05-28T06:27:32.265 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~0s 2024-05-28T06:27:32.265 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2024-05-28T06:27:32.806 INFO:tasks.ceph.osd.4:Stopped 2024-05-28T06:27:32.807 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-05-28T06:27:32.807 INFO:teuthology.orchestra.run:waiting for 300 2024-05-28T06:27:32.888 INFO:tasks.ceph.osd.5:Stopped 2024-05-28T06:27:32.889 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-05-28T06:27:32.889 INFO:teuthology.orchestra.run:waiting for 300 2024-05-28T06:27:32.989 INFO:tasks.ceph.osd.6:Stopped 2024-05-28T06:27:32.989 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-05-28T06:27:32.989 INFO:teuthology.orchestra.run:waiting for 300 2024-05-28T06:27:35.741 DEBUG:teuthology.orchestra.run.smithi096:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:27:35.749 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:27:37.773 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~0s 2024-05-28T06:27:37.773 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 has been restored 2024-05-28T06:27:39.091 INFO:tasks.ceph.osd.7:Stopped 2024-05-28T06:27:39.092 INFO:teuthology.misc:Shutting down mgr daemons... 2024-05-28T06:27:39.093 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-05-28T06:27:39.093 INFO:teuthology.orchestra.run:waiting for 300 2024-05-28T06:27:43.178 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~0s 2024-05-28T06:27:43.178 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 has been restored 2024-05-28T06:27:45.195 INFO:tasks.ceph.mgr.y:Stopped 2024-05-28T06:27:45.196 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-05-28T06:27:45.196 INFO:teuthology.orchestra.run:waiting for 300 2024-05-28T06:27:48.481 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~0s 2024-05-28T06:27:48.482 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y has been restored 2024-05-28T06:27:51.298 INFO:tasks.ceph.mgr.x:Stopped 2024-05-28T06:27:51.298 INFO:teuthology.misc:Shutting down mon daemons... 2024-05-28T06:27:51.299 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-05-28T06:27:51.299 INFO:teuthology.orchestra.run:waiting for 300 2024-05-28T06:27:51.358 INFO:tasks.ceph.mon.a:Stopped 2024-05-28T06:27:51.359 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-05-28T06:27:51.359 INFO:teuthology.orchestra.run:waiting for 300 2024-05-28T06:27:51.391 INFO:tasks.ceph.mon.b:Stopped 2024-05-28T06:27:51.392 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-05-28T06:27:51.392 INFO:teuthology.orchestra.run:waiting for 300 2024-05-28T06:27:51.477 INFO:tasks.ceph.mon.c:Stopped 2024-05-28T06:27:51.477 INFO:tasks.ceph:Checking cluster log for badness... 2024-05-28T06:27:51.477 DEBUG:teuthology.orchestra.run.smithi096:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'OSD full dropping all updates' | egrep -v 'OSD near full' | egrep -v 'pausewr flag' | egrep -v 'failsafe engaged, dropping updates' | egrep -v 'failsafe disengaged, no longer dropping' | egrep -v 'is full \(reached quota' | egrep -v POOL_FULL | egrep -v POOL_BACKFILLFULL | head -n 1 2024-05-28T06:27:51.589 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi096.front.sepia.ceph.com 2024-05-28T06:27:51.590 DEBUG:teuthology.orchestra.run.smithi096:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-05-28T06:27:51.812 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi096.front.sepia.ceph.com 2024-05-28T06:27:51.812 DEBUG:teuthology.orchestra.run.smithi096:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-05-28T06:27:51.987 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi096.front.sepia.ceph.com 2024-05-28T06:27:51.987 DEBUG:teuthology.orchestra.run.smithi096:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-05-28T06:27:52.149 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi096.front.sepia.ceph.com 2024-05-28T06:27:52.149 DEBUG:teuthology.orchestra.run.smithi096:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-05-28T06:27:52.317 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi123.front.sepia.ceph.com 2024-05-28T06:27:52.318 DEBUG:teuthology.orchestra.run.smithi123:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-05-28T06:27:52.587 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi123.front.sepia.ceph.com 2024-05-28T06:27:52.587 DEBUG:teuthology.orchestra.run.smithi123:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-05-28T06:27:52.741 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi123.front.sepia.ceph.com 2024-05-28T06:27:52.742 DEBUG:teuthology.orchestra.run.smithi123:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-05-28T06:27:52.889 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi123.front.sepia.ceph.com 2024-05-28T06:27:52.889 DEBUG:teuthology.orchestra.run.smithi123:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-05-28T06:27:53.045 INFO:tasks.ceph:Archiving mon data... 2024-05-28T06:27:53.046 DEBUG:teuthology.misc:Transferring archived files from smithi096:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/rishabh-2024-05-28_05:40:25-fs:volumes-rishabh-clone-stats-testing-default-smithi/7729374/data/mon.a.tgz 2024-05-28T06:27:53.047 DEBUG:teuthology.orchestra.run.smithi096:> mktemp 2024-05-28T06:27:53.103 INFO:teuthology.orchestra.run.smithi096.stdout:/tmp/tmp.RwGbUwntLM 2024-05-28T06:27:53.104 DEBUG:teuthology.orchestra.run.smithi096:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.RwGbUwntLM 2024-05-28T06:27:53.483 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x has been restored 2024-05-28T06:27:53.491 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod 0666 /tmp/tmp.RwGbUwntLM 2024-05-28T06:27:53.633 DEBUG:teuthology.orchestra.remote:smithi096:/tmp/tmp.RwGbUwntLM is 2MB 2024-05-28T06:27:53.762 DEBUG:teuthology.orchestra.run.smithi096:> rm -fr /tmp/tmp.RwGbUwntLM 2024-05-28T06:27:53.821 DEBUG:teuthology.misc:Transferring archived files from smithi123:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/rishabh-2024-05-28_05:40:25-fs:volumes-rishabh-clone-stats-testing-default-smithi/7729374/data/mon.b.tgz 2024-05-28T06:27:53.822 DEBUG:teuthology.orchestra.run.smithi123:> mktemp 2024-05-28T06:27:53.878 INFO:teuthology.orchestra.run.smithi123.stdout:/tmp/tmp.gJjYDyE5AX 2024-05-28T06:27:53.878 DEBUG:teuthology.orchestra.run.smithi123:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.gJjYDyE5AX 2024-05-28T06:27:54.256 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 0666 /tmp/tmp.gJjYDyE5AX 2024-05-28T06:27:54.393 DEBUG:teuthology.orchestra.remote:smithi123:/tmp/tmp.gJjYDyE5AX is 2MB 2024-05-28T06:27:54.521 DEBUG:teuthology.orchestra.run.smithi123:> rm -fr /tmp/tmp.gJjYDyE5AX 2024-05-28T06:27:54.578 DEBUG:teuthology.misc:Transferring archived files from smithi123:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/rishabh-2024-05-28_05:40:25-fs:volumes-rishabh-clone-stats-testing-default-smithi/7729374/data/mon.c.tgz 2024-05-28T06:27:54.579 DEBUG:teuthology.orchestra.run.smithi123:> mktemp 2024-05-28T06:27:54.628 INFO:teuthology.orchestra.run.smithi123.stdout:/tmp/tmp.0jE1nzvxgw 2024-05-28T06:27:54.628 DEBUG:teuthology.orchestra.run.smithi123:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.0jE1nzvxgw 2024-05-28T06:27:54.980 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 0666 /tmp/tmp.0jE1nzvxgw 2024-05-28T06:27:55.112 DEBUG:teuthology.orchestra.remote:smithi123:/tmp/tmp.0jE1nzvxgw is 2MB 2024-05-28T06:27:55.234 DEBUG:teuthology.orchestra.run.smithi123:> rm -fr /tmp/tmp.0jE1nzvxgw 2024-05-28T06:27:55.289 INFO:tasks.ceph:Cleaning ceph cluster... 2024-05-28T06:27:55.290 DEBUG:teuthology.orchestra.run.smithi096:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-05-28T06:27:55.297 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-05-28T06:27:55.435 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-05-28T06:27:55.491 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-05-28T06:27:55.491 INFO:tasks.ceph:Archiving crash dumps... 2024-05-28T06:27:55.494 DEBUG:teuthology.misc:Transferring archived files from smithi096:/var/lib/ceph/crash to /home/teuthworker/archive/rishabh-2024-05-28_05:40:25-fs:volumes-rishabh-clone-stats-testing-default-smithi/7729374/remote/smithi096/crash 2024-05-28T06:27:55.495 DEBUG:teuthology.orchestra.run.smithi096:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-05-28T06:27:55.580 DEBUG:teuthology.misc:Transferring archived files from smithi123:/var/lib/ceph/crash to /home/teuthworker/archive/rishabh-2024-05-28_05:40:25-fs:volumes-rishabh-clone-stats-testing-default-smithi/7729374/remote/smithi123/crash 2024-05-28T06:27:55.581 DEBUG:teuthology.orchestra.run.smithi123:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-05-28T06:27:55.667 INFO:tasks.ceph:Shutting down logrotate 2024-05-28T06:27:55.668 DEBUG:teuthology.orchestra.run.smithi096:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:27:55.676 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-28T06:27:55.803 DEBUG:teuthology.orchestra.run.smithi096:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-05-28T06:27:55.890 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-05-28T06:27:55.980 INFO:tasks.ceph:Compressing logs... 2024-05-28T06:27:55.980 DEBUG:teuthology.orchestra.run.smithi096:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-28T06:27:55.988 DEBUG:teuthology.orchestra.run.smithi123:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-28T06:27:56.063 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53017.log 2024-05-28T06:27:56.064 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66369.log 2024-05-28T06:27:56.065 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.53017.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53017.log.gz 2024-05-28T06:27:56.066 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42593.log 2024-05-28T06:27:56.066 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.66369.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48163.log 2024-05-28T06:27:56.067 INFO:teuthology.orchestra.run.smithi096.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.66369.log.gz 2024-05-28T06:27:56.067 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.42593.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63546.log 2024-05-28T06:27:56.067 INFO:teuthology.orchestra.run.smithi096.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.42593.log.gz 2024-05-28T06:27:56.068 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.48163.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62208.log 2024-05-28T06:27:56.069 INFO:teuthology.orchestra.run.smithi096.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.48163.log.gz 2024-05-28T06:27:56.069 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.63546.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52566.log 2024-05-28T06:27:56.070 INFO:teuthology.orchestra.run.smithi096.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.63546.log.gz 2024-05-28T06:27:56.070 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.62208.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69173.log 2024-05-28T06:27:56.071 INFO:teuthology.orchestra.run.smithi096.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.62208.log.gz 2024-05-28T06:27:56.071 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36829.log 2024-05-28T06:27:56.072 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-05-28T06:27:56.073 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.52566.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52566.log.gz 2024-05-28T06:27:56.073 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58018.log 2024-05-28T06:27:56.073 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.69173.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44361.log 2024-05-28T06:27:56.073 INFO:teuthology.orchestra.run.smithi096.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.69173.log.gz 2024-05-28T06:27:56.074 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.58018.log: gzip 87.5% -5 -- replaced with /var/log/ceph/ceph-client.admin.58018.log.gz --verbose 2024-05-28T06:27:56.075 INFO:teuthology.orchestra.run.smithi096.stderr: -- /var/log/ceph/ceph-client.admin.63847.log 2024-05-28T06:27:56.075 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37120.log 2024-05-28T06:27:56.076 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.36829.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.36829.log.gz 2024-05-28T06:27:56.076 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-05-28T06:27:56.076 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.37120.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-05-28T06:27:56.076 INFO:teuthology.orchestra.run.smithi123.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.37120.log.gz 2024-05-28T06:27:56.078 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-05-28T06:27:56.078 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-05-28T06:27:56.079 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.44361.log: gzip -5 --verbose -- 88.5% /var/log/ceph/ceph-client.admin.54437.log -- replaced with /var/log/ceph/ceph-client.admin.44361.log.gz 2024-05-28T06:27:56.080 INFO:teuthology.orchestra.run.smithi096.stderr: 2024-05-28T06:27:56.080 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.63847.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52878.log 2024-05-28T06:27:56.080 INFO:teuthology.orchestra.run.smithi096.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.63847.log.gz 2024-05-28T06:27:56.080 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.54437.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.54437.log.gz 2024-05-28T06:27:56.080 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80397.log 2024-05-28T06:27:56.080 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.52878.log: gzip 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52878.log.gz -5 2024-05-28T06:27:56.080 INFO:teuthology.orchestra.run.smithi096.stderr: --verbose -- /var/log/ceph/ceph-client.admin.59701.log 2024-05-28T06:27:56.081 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.80397.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44026.log 2024-05-28T06:27:56.081 INFO:teuthology.orchestra.run.smithi096.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.80397.log.gz 2024-05-28T06:27:56.083 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.43096.log 2024-05-28T06:27:56.083 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-05-28T06:27:56.083 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.1.43096.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37080.log 2024-05-28T06:27:56.083 INFO:teuthology.orchestra.run.smithi123.stderr: 92.0% -- replaced with /var/log/ceph/ceph.log.gz 2024-05-28T06:27:56.083 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-osd.4.log: 90.6% -- replaced with /var/log/ceph/ceph-client.1.43096.log.gz 2024-05-28T06:27:56.084 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.59701.log: gzip -5 --verbose 89.3% -- -- replaced with /var/log/ceph/ceph-client.admin.59701.log.gz 2024-05-28T06:27:56.084 INFO:teuthology.orchestra.run.smithi096.stderr: /var/log/ceph/ceph-client.admin.57627.log 2024-05-28T06:27:56.085 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.44026.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62981.log 2024-05-28T06:27:56.085 INFO:teuthology.orchestra.run.smithi096.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.44026.log.gz 2024-05-28T06:27:56.085 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.57627.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70082.log 2024-05-28T06:27:56.085 INFO:teuthology.orchestra.run.smithi096.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.57627.log.gz 2024-05-28T06:27:56.085 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.62981.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74332.log 2024-05-28T06:27:56.085 INFO:teuthology.orchestra.run.smithi096.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.62981.log.gz 2024-05-28T06:27:56.087 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-05-28T06:27:56.088 INFO:teuthology.orchestra.run.smithi123.stderr: 93.7% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-05-28T06:27:56.088 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.37080.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-05-28T06:27:56.088 INFO:teuthology.orchestra.run.smithi123.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.37080.log.gz 2024-05-28T06:27:56.088 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-05-28T06:27:56.088 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36960.log 2024-05-28T06:27:56.089 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.70082.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.70082.log.gz 2024-05-28T06:27:56.090 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58255.log 2024-05-28T06:27:56.090 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.74332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66594.log 2024-05-28T06:27:56.090 INFO:teuthology.orchestra.run.smithi096.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.74332.log.gz 2024-05-28T06:27:56.090 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.58255.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65232.log 2024-05-28T06:27:56.090 INFO:teuthology.orchestra.run.smithi096.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.58255.log.gz 2024-05-28T06:27:56.090 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.66594.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.66594.log.gz 2024-05-28T06:27:56.090 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83545.log 2024-05-28T06:27:56.091 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.65232.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52296.log 2024-05-28T06:27:56.091 INFO:teuthology.orchestra.run.smithi096.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.65232.log.gz 2024-05-28T06:27:56.098 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.83545.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58129.log 2024-05-28T06:27:56.099 INFO:teuthology.orchestra.run.smithi096.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.83545.log.gz 2024-05-28T06:27:56.099 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.52296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62599.log 2024-05-28T06:27:56.099 INFO:teuthology.orchestra.run.smithi096.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.52296.log.gz 2024-05-28T06:27:56.099 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.58129.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69131.log 2024-05-28T06:27:56.099 INFO:teuthology.orchestra.run.smithi096.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.58129.log.gz 2024-05-28T06:27:56.099 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.62599.log: gzip 87.7% -- replaced with /var/log/ceph/ceph-client.admin.62599.log.gz 2024-05-28T06:27:56.100 INFO:teuthology.orchestra.run.smithi096.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.77857.log 2024-05-28T06:27:56.101 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/ceph-client.admin.36960.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37040.log 2024-05-28T06:27:56.101 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.37040.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.36960.log.gzgzip 2024-05-28T06:27:56.101 INFO:teuthology.orchestra.run.smithi123.stderr: -5 --verbose -- /var/log/ceph/ceph-client.3.43552.log 2024-05-28T06:27:56.101 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37000.log 2024-05-28T06:27:56.101 INFO:teuthology.orchestra.run.smithi123.stderr: 88.1%/var/log/ceph/ceph-client.3.43552.log: -- replaced with /var/log/ceph/ceph-client.admin.37040.log.gz/var/log/ceph/ceph-client.admin.37000.log: 2024-05-28T06:27:56.103 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36871.log 2024-05-28T06:27:56.106 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.69131.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.69131.log.gz 2024-05-28T06:27:56.106 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83611.log 2024-05-28T06:27:56.107 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.77857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44319.log 2024-05-28T06:27:56.107 INFO:teuthology.orchestra.run.smithi096.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.77857.log.gz 2024-05-28T06:27:56.107 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.83611.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66285.log 2024-05-28T06:27:56.107 INFO:teuthology.orchestra.run.smithi096.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.83611.log.gz 2024-05-28T06:27:56.107 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.44319.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44319.log.gz 2024-05-28T06:27:56.107 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57850.log 2024-05-28T06:27:56.107 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.66285.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85819.log 2024-05-28T06:27:56.108 INFO:teuthology.orchestra.run.smithi096.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.66285.log.gz 2024-05-28T06:27:56.108 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.57850.log: gzip 88.5% -5 -- replaced with /var/log/ceph/ceph-client.admin.57850.log.gz --verbose 2024-05-28T06:27:56.108 INFO:teuthology.orchestra.run.smithi096.stderr: -- /var/log/ceph/ceph-client.admin.84990.log 2024-05-28T06:27:56.108 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.85819.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.85819.log.gz 2024-05-28T06:27:56.108 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76707.log 2024-05-28T06:27:56.108 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.84990.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46664.log 2024-05-28T06:27:56.108 INFO:teuthology.orchestra.run.smithi096.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.84990.log.gz 2024-05-28T06:27:56.108 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.76707.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64713.log 2024-05-28T06:27:56.109 INFO:teuthology.orchestra.run.smithi096.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.76707.log.gz 2024-05-28T06:27:56.109 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.46664.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46664.log.gz 2024-05-28T06:27:56.109 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48719.log 2024-05-28T06:27:56.109 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.64713.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66636.log 2024-05-28T06:27:56.109 INFO:teuthology.orchestra.run.smithi096.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.64713.log.gz 2024-05-28T06:27:56.109 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.48719.log: gzip -5 --verbose 87.3% -- -- replaced with /var/log/ceph/ceph-client.admin.48719.log.gz 2024-05-28T06:27:56.109 INFO:teuthology.orchestra.run.smithi096.stderr: /var/log/ceph/ceph-client.admin.86083.log 2024-05-28T06:27:56.114 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.66636.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.66636.log.gz 2024-05-28T06:27:56.114 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77103.log 2024-05-28T06:27:56.115 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.86083.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60862.log 2024-05-28T06:27:56.115 INFO:teuthology.orchestra.run.smithi096.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.86083.log.gz 2024-05-28T06:27:56.115 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.77103.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66015.log 2024-05-28T06:27:56.115 INFO:teuthology.orchestra.run.smithi096.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.77103.log.gz 2024-05-28T06:27:56.115 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.60862.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.60862.log.gz 2024-05-28T06:27:56.115 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44823.log 2024-05-28T06:27:56.115 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.66015.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54271.log 2024-05-28T06:27:56.115 INFO:teuthology.orchestra.run.smithi096.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.66015.log.gz 2024-05-28T06:27:56.116 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.44823.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44823.log.gz 2024-05-28T06:27:56.116 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79357.log 2024-05-28T06:27:56.116 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.54271.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.54271.log.gz 2024-05-28T06:27:56.116 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61686.log 2024-05-28T06:27:56.116 INFO:teuthology.orchestra.run.smithi123.stderr: 87.6% 91.6% -- replaced with /var/log/ceph/ceph-client.admin.37000.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36920.log 2024-05-28T06:27:56.117 INFO:teuthology.orchestra.run.smithi123.stderr: 2024-05-28T06:27:56.117 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.36871.log: -- replaced with /var/log/ceph/ceph-client.3.43552.log.gz 2024-05-28T06:27:56.117 INFO:teuthology.orchestra.run.smithi123.stderr: 88.7%gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-05-28T06:27:56.117 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.36920.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.36920.log.gz 2024-05-28T06:27:56.117 INFO:teuthology.orchestra.run.smithi123.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36871.log.gz 2024-05-28T06:27:56.117 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-05-28T06:27:56.119 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.79357.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68698.log 2024-05-28T06:27:56.119 INFO:teuthology.orchestra.run.smithi096.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.79357.log.gz 2024-05-28T06:27:56.119 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.61686.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.61686.log.gz 2024-05-28T06:27:56.119 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82515.log 2024-05-28T06:27:56.119 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.68698.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75517.log 2024-05-28T06:27:56.120 INFO:teuthology.orchestra.run.smithi096.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.68698.log.gz 2024-05-28T06:27:56.120 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.82515.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57399.log 2024-05-28T06:27:56.120 INFO:teuthology.orchestra.run.smithi096.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.82515.log.gz 2024-05-28T06:27:56.120 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.75517.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57753.log 2024-05-28T06:27:56.121 INFO:teuthology.orchestra.run.smithi096.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.75517.log.gz 2024-05-28T06:27:56.121 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.57399.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.57399.log.gz 2024-05-28T06:27:56.121 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76564.log 2024-05-28T06:27:56.121 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.57753.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48205.log 2024-05-28T06:27:56.122 INFO:teuthology.orchestra.run.smithi096.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.57753.log.gz 2024-05-28T06:27:56.122 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.76564.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62641.log 2024-05-28T06:27:56.122 INFO:teuthology.orchestra.run.smithi096.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.76564.log.gz 2024-05-28T06:27:56.123 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.48205.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.48205.log.gz 2024-05-28T06:27:56.123 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.60297.log 2024-05-28T06:27:56.124 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.62641.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42765.log 2024-05-28T06:27:56.125 INFO:teuthology.orchestra.run.smithi096.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.62641.log.gz 2024-05-28T06:27:56.125 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.0.60297.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69662.log 2024-05-28T06:27:56.126 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.42765.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.42765.log.gz 2024-05-28T06:27:56.127 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.30984.log 2024-05-28T06:27:56.127 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.69662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55754.log 2024-05-28T06:27:56.128 INFO:teuthology.orchestra.run.smithi096.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.69662.log.gz 2024-05-28T06:27:56.128 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph.tmp-client.admin.30984.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.30984.log.gz 2024-05-28T06:27:56.129 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80532.log 2024-05-28T06:27:56.130 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.55754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81155.log 2024-05-28T06:27:56.130 INFO:teuthology.orchestra.run.smithi096.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55754.log.gz 2024-05-28T06:27:56.131 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.80532.log: gzip 88.9% -- replaced with /var/log/ceph/ceph-client.admin.80532.log.gz 2024-05-28T06:27:56.131 INFO:teuthology.orchestra.run.smithi096.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.53424.log 2024-05-28T06:27:56.132 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.81155.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71382.log 2024-05-28T06:27:56.133 INFO:teuthology.orchestra.run.smithi096.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.81155.log.gz 2024-05-28T06:27:56.133 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.53424.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78127.log 2024-05-28T06:27:56.134 INFO:teuthology.orchestra.run.smithi096.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.53424.log.gz 2024-05-28T06:27:56.134 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.71382.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57087.log 2024-05-28T06:27:56.135 INFO:teuthology.orchestra.run.smithi096.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.71382.log.gz 2024-05-28T06:27:56.136 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.78127.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50835.log 2024-05-28T06:27:56.136 INFO:teuthology.orchestra.run.smithi096.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.78127.log.gz 2024-05-28T06:27:56.136 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.57087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75265.log 2024-05-28T06:27:56.137 INFO:teuthology.orchestra.run.smithi096.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.57087.log.gz 2024-05-28T06:27:56.137 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.50835.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50835.log.gz 2024-05-28T06:27:56.138 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82095.log 2024-05-28T06:27:56.139 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.75265.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84458.log 2024-05-28T06:27:56.139 INFO:teuthology.orchestra.run.smithi096.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.75265.log.gz 2024-05-28T06:27:56.140 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.82095.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49097.log 2024-05-28T06:27:56.140 INFO:teuthology.orchestra.run.smithi096.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.82095.log.gz 2024-05-28T06:27:56.141 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.84458.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82221.log 2024-05-28T06:27:56.141 INFO:teuthology.orchestra.run.smithi096.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.84458.log.gz 2024-05-28T06:27:56.142 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.49097.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53185.log 2024-05-28T06:27:56.142 INFO:teuthology.orchestra.run.smithi096.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49097.log.gz 2024-05-28T06:27:56.143 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.82221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77647.log 2024-05-28T06:27:56.143 INFO:teuthology.orchestra.run.smithi096.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.82221.log.gz 2024-05-28T06:27:56.144 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.53185.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63588.log 2024-05-28T06:27:56.145 INFO:teuthology.orchestra.run.smithi096.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.53185.log.gz 2024-05-28T06:27:56.145 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.77647.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79399.log 2024-05-28T06:27:56.145 INFO:teuthology.orchestra.run.smithi096.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.77647.log.gz 2024-05-28T06:27:56.146 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.63588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52920.log 2024-05-28T06:27:56.146 INFO:teuthology.orchestra.run.smithi096.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.63588.log.gz 2024-05-28T06:27:56.147 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.79399.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70962.log 2024-05-28T06:27:56.147 INFO:teuthology.orchestra.run.smithi096.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.79399.log.gz 2024-05-28T06:27:56.148 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.52920.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73875.log 2024-05-28T06:27:56.149 INFO:teuthology.orchestra.run.smithi096.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.52920.log.gz 2024-05-28T06:27:56.149 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.70962.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43101.log 2024-05-28T06:27:56.149 INFO:teuthology.orchestra.run.smithi096.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.70962.log.gz 2024-05-28T06:27:56.150 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.73875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54012.log 2024-05-28T06:27:56.153 INFO:teuthology.orchestra.run.smithi096.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.73875.log.gz 2024-05-28T06:27:56.153 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.43101.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73595.log 2024-05-28T06:27:56.153 INFO:teuthology.orchestra.run.smithi096.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.43101.log.gz 2024-05-28T06:27:56.154 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.54012.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61644.log 2024-05-28T06:27:56.154 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.73595.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.54012.log.gz 2024-05-28T06:27:56.156 INFO:teuthology.orchestra.run.smithi096.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.73595.log.gz 2024-05-28T06:27:56.156 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68908.log 2024-05-28T06:27:56.156 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.61644.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52380.log 2024-05-28T06:27:56.156 INFO:teuthology.orchestra.run.smithi096.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.61644.log.gz 2024-05-28T06:27:56.156 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.68908.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84028.log 2024-05-28T06:27:56.157 INFO:teuthology.orchestra.run.smithi096.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.68908.log.gz 2024-05-28T06:27:56.157 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.52380.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87103.log 2024-05-28T06:27:56.158 INFO:teuthology.orchestra.run.smithi096.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.52380.log.gz 2024-05-28T06:27:56.158 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.84028.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67423.log 2024-05-28T06:27:56.158 INFO:teuthology.orchestra.run.smithi096.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.84028.log.gz 2024-05-28T06:27:56.159 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.87103.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47963.log 2024-05-28T06:27:56.160 INFO:teuthology.orchestra.run.smithi096.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.87103.log.gz 2024-05-28T06:27:56.160 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.67423.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74290.log 2024-05-28T06:27:56.161 INFO:teuthology.orchestra.run.smithi096.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.67423.log.gz 2024-05-28T06:27:56.161 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.47963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49223.log 2024-05-28T06:27:56.161 INFO:teuthology.orchestra.run.smithi096.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.47963.log.gz 2024-05-28T06:27:56.162 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.74290.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84593.log 2024-05-28T06:27:56.162 INFO:teuthology.orchestra.run.smithi096.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.74290.log.gz 2024-05-28T06:27:56.163 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.49223.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49013.log 2024-05-28T06:27:56.163 INFO:teuthology.orchestra.run.smithi096.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49223.log.gz 2024-05-28T06:27:56.164 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.84593.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82137.log 2024-05-28T06:27:56.164 INFO:teuthology.orchestra.run.smithi096.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.84593.log.gz 2024-05-28T06:27:56.165 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.49013.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86893.log 2024-05-28T06:27:56.165 INFO:teuthology.orchestra.run.smithi096.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.49013.log.gz 2024-05-28T06:27:56.166 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.82137.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65805.log 2024-05-28T06:27:56.166 INFO:teuthology.orchestra.run.smithi096.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.82137.log.gz 2024-05-28T06:27:56.167 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.86893.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.86893.log.gz 2024-05-28T06:27:56.167 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-05-28T06:27:56.168 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.65805.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64479.log 2024-05-28T06:27:56.168 INFO:teuthology.orchestra.run.smithi096.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.65805.log.gz 2024-05-28T06:27:56.169 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43702.log 2024-05-28T06:27:56.170 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.64479.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.64479.log.gz 2024-05-28T06:27:56.170 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46328.log 2024-05-28T06:27:56.171 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.43702.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49475.log 2024-05-28T06:27:56.172 INFO:teuthology.orchestra.run.smithi096.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.43702.log.gz 2024-05-28T06:27:56.172 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.46328.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46328.log.gz 2024-05-28T06:27:56.172 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51444.log 2024-05-28T06:27:56.173 INFO:teuthology.orchestra.run.smithi096.stderr: 86.5% -- replaced with /var/log/ceph/ceph.log.gz 2024-05-28T06:27:56.173 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.49475.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78337.log 2024-05-28T06:27:56.174 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.51444.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49475.log.gz 2024-05-28T06:27:56.175 INFO:teuthology.orchestra.run.smithi096.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.51444.log.gz 2024-05-28T06:27:56.175 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.71726.log 2024-05-28T06:27:56.175 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.78337.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85467.log 2024-05-28T06:27:56.177 INFO:teuthology.orchestra.run.smithi096.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.78337.log.gz 2024-05-28T06:27:56.177 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.0.71726.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57934.log 2024-05-28T06:27:56.178 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.85467.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66720.log 2024-05-28T06:27:56.178 INFO:teuthology.orchestra.run.smithi096.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.85467.log.gz 2024-05-28T06:27:56.179 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.57934.log: gzip 88.4% -5 -- replaced with /var/log/ceph/ceph-client.admin.57934.log.gz 2024-05-28T06:27:56.180 INFO:teuthology.orchestra.run.smithi096.stderr: --verbose -- /var/log/ceph/ceph-client.admin.51528.log 2024-05-28T06:27:56.181 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.66720.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69536.log 2024-05-28T06:27:56.181 INFO:teuthology.orchestra.run.smithi096.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.66720.log.gz 2024-05-28T06:27:56.182 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.51528.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.51528.log.gz 2024-05-28T06:27:56.182 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58549.log 2024-05-28T06:27:56.183 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.69536.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85611.log 2024-05-28T06:27:56.183 INFO:teuthology.orchestra.run.smithi096.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.69536.log.gz 2024-05-28T06:27:56.185 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.58549.log: gzip -5 --verbose -- 88.5% /var/log/ceph/ceph-client.admin.72514.log 2024-05-28T06:27:56.185 INFO:teuthology.orchestra.run.smithi096.stderr: -- replaced with /var/log/ceph/ceph-client.admin.58549.log.gz 2024-05-28T06:27:56.186 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.85611.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58778.log 2024-05-28T06:27:56.186 INFO:teuthology.orchestra.run.smithi096.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.85611.log.gz 2024-05-28T06:27:56.186 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.72514.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.72514.log.gzgzip 2024-05-28T06:27:56.187 INFO:teuthology.orchestra.run.smithi096.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.76210.log 2024-05-28T06:27:56.188 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.58778.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81197.log 2024-05-28T06:27:56.188 INFO:teuthology.orchestra.run.smithi096.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.58778.log.gz 2024-05-28T06:27:56.189 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.76210.log: gzip 88.3% -- replaced with /var/log/ceph/ceph-client.admin.76210.log.gz 2024-05-28T06:27:56.189 INFO:teuthology.orchestra.run.smithi096.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.80092.log 2024-05-28T06:27:56.190 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.81197.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50289.log 2024-05-28T06:27:56.190 INFO:teuthology.orchestra.run.smithi096.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.81197.log.gz 2024-05-28T06:27:56.191 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.80092.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.80092.log.gz 2024-05-28T06:27:56.192 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53143.log 2024-05-28T06:27:56.192 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.50289.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68950.log 2024-05-28T06:27:56.193 INFO:teuthology.orchestra.run.smithi096.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50289.log.gz 2024-05-28T06:27:56.194 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.53143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62897.log 2024-05-28T06:27:56.194 INFO:teuthology.orchestra.run.smithi096.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53143.log.gz 2024-05-28T06:27:56.195 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.68950.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58507.log 2024-05-28T06:27:56.195 INFO:teuthology.orchestra.run.smithi096.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.68950.log.gz 2024-05-28T06:27:56.196 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.62897.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76429.log 2024-05-28T06:27:56.197 INFO:teuthology.orchestra.run.smithi096.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.62897.log.gz 2024-05-28T06:27:56.197 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.58507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48515.log 2024-05-28T06:27:56.198 INFO:teuthology.orchestra.run.smithi096.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.58507.log.gz 2024-05-28T06:27:56.198 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.76429.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65130.log 2024-05-28T06:27:56.199 INFO:teuthology.orchestra.run.smithi096.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.76429.log.gz 2024-05-28T06:27:56.200 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.48515.log: gzip -5 --verbose 88.1% -- -- replaced with /var/log/ceph/ceph-client.admin.48515.log.gz 2024-05-28T06:27:56.200 INFO:teuthology.orchestra.run.smithi096.stderr: /var/log/ceph/ceph-client.admin.73553.log 2024-05-28T06:27:56.201 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.65130.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75139.log 2024-05-28T06:27:56.201 INFO:teuthology.orchestra.run.smithi096.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.65130.log.gz 2024-05-28T06:27:56.202 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.73553.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78763.log 2024-05-28T06:27:56.202 INFO:teuthology.orchestra.run.smithi096.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.73553.log.gz 2024-05-28T06:27:56.203 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.75139.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84803.log 2024-05-28T06:27:56.204 INFO:teuthology.orchestra.run.smithi096.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.75139.log.gz 2024-05-28T06:27:56.205 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.78763.log: 88.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.78763.log.gz 2024-05-28T06:27:56.205 INFO:teuthology.orchestra.run.smithi096.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.53298.log 2024-05-28T06:27:56.206 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.84803.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70124.log 2024-05-28T06:27:56.206 INFO:teuthology.orchestra.run.smithi096.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.84803.log.gz 2024-05-28T06:27:56.207 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.53298.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74103.log 2024-05-28T06:27:56.207 INFO:teuthology.orchestra.run.smithi096.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.53298.log.gz 2024-05-28T06:27:56.208 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.70124.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-05-28T06:27:56.209 INFO:teuthology.orchestra.run.smithi096.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.70124.log.gz 2024-05-28T06:27:56.210 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.74103.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70609.log 2024-05-28T06:27:56.210 INFO:teuthology.orchestra.run.smithi096.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.74103.log.gz 2024-05-28T06:27:56.211 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66989.log 2024-05-28T06:27:56.212 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.70609.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.70609.log.gz 2024-05-28T06:27:56.212 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73707.log 2024-05-28T06:27:56.213 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.66989.log: 85.3% -- replaced with /var/log/ceph/ceph-client.admin.66989.log.gzgzip 2024-05-28T06:27:56.214 INFO:teuthology.orchestra.run.smithi096.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.68194.log 2024-05-28T06:27:56.216 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.73707.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63336.log 2024-05-28T06:27:56.216 INFO:teuthology.orchestra.run.smithi096.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.73707.log.gz 2024-05-28T06:27:56.216 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.68194.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86809.log 2024-05-28T06:27:56.217 INFO:teuthology.orchestra.run.smithi096.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.68194.log.gz 2024-05-28T06:27:56.217 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.63336.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68050.log 2024-05-28T06:27:56.218 INFO:teuthology.orchestra.run.smithi096.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.63336.log.gz 2024-05-28T06:27:56.218 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.86809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76387.log 2024-05-28T06:27:56.219 INFO:teuthology.orchestra.run.smithi096.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.86809.log.gz 2024-05-28T06:27:56.220 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.68050.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.68050.log.gz 2024-05-28T06:27:56.220 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52128.log 2024-05-28T06:27:56.221 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.76387.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48971.log 2024-05-28T06:27:56.222 INFO:teuthology.orchestra.run.smithi096.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.76387.log.gz 2024-05-28T06:27:56.223 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.52128.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43358.log 2024-05-28T06:27:56.223 INFO:teuthology.orchestra.run.smithi096.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52128.log.gz 2024-05-28T06:27:56.224 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.48971.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86424.log 2024-05-28T06:27:56.224 INFO:teuthology.orchestra.run.smithi096.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.48971.log.gz 2024-05-28T06:27:56.225 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.43358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47753.log 2024-05-28T06:27:56.225 INFO:teuthology.orchestra.run.smithi096.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.43358.log.gz 2024-05-28T06:27:56.227 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.86424.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80490.log 2024-05-28T06:27:56.227 INFO:teuthology.orchestra.run.smithi096.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.86424.log.gz 2024-05-28T06:27:56.228 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.47753.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56503.log 2024-05-28T06:27:56.228 INFO:teuthology.orchestra.run.smithi096.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.47753.log.gz 2024-05-28T06:27:56.229 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.80490.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.80490.log.gz 2024-05-28T06:27:56.229 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49139.log 2024-05-28T06:27:56.230 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.56503.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60604.log 2024-05-28T06:27:56.230 INFO:teuthology.orchestra.run.smithi096.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.56503.log.gz 2024-05-28T06:27:56.231 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.49139.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86550.log 2024-05-28T06:27:56.232 INFO:teuthology.orchestra.run.smithi096.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.49139.log.gz 2024-05-28T06:27:56.233 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.60604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81504.log 2024-05-28T06:27:56.233 INFO:teuthology.orchestra.run.smithi096.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.60604.log.gz 2024-05-28T06:27:56.234 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.86550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60478.log 2024-05-28T06:27:56.234 INFO:teuthology.orchestra.run.smithi096.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.86550.log.gz 2024-05-28T06:27:56.235 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.81504.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67235.log 2024-05-28T06:27:56.236 INFO:teuthology.orchestra.run.smithi096.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.81504.log.gz 2024-05-28T06:27:56.237 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.60478.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43975.log 2024-05-28T06:27:56.237 INFO:teuthology.orchestra.run.smithi096.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.60478.log.gz 2024-05-28T06:27:56.237 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.67235.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85299.log 2024-05-28T06:27:56.238 INFO:teuthology.orchestra.run.smithi096.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.67235.log.gz 2024-05-28T06:27:56.239 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.43975.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42807.log 2024-05-28T06:27:56.239 INFO:teuthology.orchestra.run.smithi096.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43975.log.gz 2024-05-28T06:27:56.240 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.85299.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66865.log 2024-05-28T06:27:56.240 INFO:teuthology.orchestra.run.smithi096.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.85299.log.gz 2024-05-28T06:27:56.241 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.42807.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42509.log 2024-05-28T06:27:56.242 INFO:teuthology.orchestra.run.smithi096.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42807.log.gz 2024-05-28T06:27:56.243 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.66865.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59041.log 2024-05-28T06:27:56.243 INFO:teuthology.orchestra.run.smithi096.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.66865.log.gz 2024-05-28T06:27:56.244 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.42509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58381.log 2024-05-28T06:27:56.244 INFO:teuthology.orchestra.run.smithi096.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42509.log.gz 2024-05-28T06:27:56.245 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.59041.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80176.log 2024-05-28T06:27:56.246 INFO:teuthology.orchestra.run.smithi096.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.59041.log.gz 2024-05-28T06:27:56.246 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.58381.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48247.log 2024-05-28T06:27:56.247 INFO:teuthology.orchestra.run.smithi096.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.58381.log.gz 2024-05-28T06:27:56.248 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.80176.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85942.log 2024-05-28T06:27:56.248 INFO:teuthology.orchestra.run.smithi096.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.80176.log.gz 2024-05-28T06:27:56.249 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.48247.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80008.log 2024-05-28T06:27:56.249 INFO:teuthology.orchestra.run.smithi096.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.48247.log.gz 2024-05-28T06:27:56.250 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.85942.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74882.log 2024-05-28T06:27:56.251 INFO:teuthology.orchestra.run.smithi096.stderr: 85.3% -- replaced with /var/log/ceph/ceph-client.admin.85942.log.gz 2024-05-28T06:27:56.252 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.80008.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62166.log 2024-05-28T06:27:56.252 INFO:teuthology.orchestra.run.smithi096.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.80008.log.gz 2024-05-28T06:27:56.253 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.74882.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53466.log 2024-05-28T06:27:56.253 INFO:teuthology.orchestra.run.smithi096.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.74882.log.gz 2024-05-28T06:27:56.254 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.62166.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55337.log 2024-05-28T06:27:56.255 INFO:teuthology.orchestra.run.smithi096.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.62166.log.gz 2024-05-28T06:27:56.255 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.53466.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70321.log 2024-05-28T06:27:56.256 INFO:teuthology.orchestra.run.smithi096.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.53466.log.gz 2024-05-28T06:27:56.256 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.55337.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75391.log 2024-05-28T06:27:56.257 INFO:teuthology.orchestra.run.smithi096.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.55337.log.gz 2024-05-28T06:27:56.258 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.70321.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42635.log 2024-05-28T06:27:56.258 INFO:teuthology.orchestra.run.smithi096.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.70321.log.gz 2024-05-28T06:27:56.259 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.75391.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42551.log 2024-05-28T06:27:56.259 INFO:teuthology.orchestra.run.smithi096.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.75391.log.gz 2024-05-28T06:27:56.260 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.42635.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65358.log 2024-05-28T06:27:56.261 INFO:teuthology.orchestra.run.smithi096.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42635.log.gz 2024-05-28T06:27:56.262 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.42551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62557.log 2024-05-28T06:27:56.262 INFO:teuthology.orchestra.run.smithi096.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42551.log.gz 2024-05-28T06:27:56.263 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.65358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61914.log 2024-05-28T06:27:56.263 INFO:teuthology.orchestra.run.smithi096.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.65358.log.gz 2024-05-28T06:27:56.264 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.62557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53886.log 2024-05-28T06:27:56.264 INFO:teuthology.orchestra.run.smithi096.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.62557.log.gz 2024-05-28T06:27:56.265 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.61914.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64353.log 2024-05-28T06:27:56.266 INFO:teuthology.orchestra.run.smithi096.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.61914.log.gz 2024-05-28T06:27:56.267 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.53886.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65316.log 2024-05-28T06:27:56.267 INFO:teuthology.orchestra.run.smithi096.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.53886.log.gz 2024-05-28T06:27:56.268 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.64353.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65274.log 2024-05-28T06:27:56.268 INFO:teuthology.orchestra.run.smithi096.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.64353.log.gz 2024-05-28T06:27:56.269 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.65316.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74717.log 2024-05-28T06:27:56.269 INFO:teuthology.orchestra.run.smithi096.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.65316.log.gz 2024-05-28T06:27:56.271 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.65274.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43703.log 2024-05-28T06:27:56.271 INFO:teuthology.orchestra.run.smithi096.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.65274.log.gz 2024-05-28T06:27:56.272 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.74717.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87145.log 2024-05-28T06:27:56.272 INFO:teuthology.orchestra.run.smithi096.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.74717.log.gz 2024-05-28T06:27:56.273 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.43703.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60520.log 2024-05-28T06:27:56.273 INFO:teuthology.orchestra.run.smithi096.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.43703.log.gz 2024-05-28T06:27:56.274 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.87145.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-05-28T06:27:56.274 INFO:teuthology.orchestra.run.smithi096.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.87145.log.gz 2024-05-28T06:27:56.275 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.60520.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54054.log 2024-05-28T06:27:56.276 INFO:teuthology.orchestra.run.smithi096.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.60520.log.gz 2024-05-28T06:27:56.277 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49931.log 2024-05-28T06:27:56.278 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.54054.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61560.log 2024-05-28T06:27:56.279 INFO:teuthology.orchestra.run.smithi096.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54054.log.gz 2024-05-28T06:27:56.280 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.49931.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48473.log 2024-05-28T06:27:56.280 INFO:teuthology.orchestra.run.smithi096.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49931.log.gz 2024-05-28T06:27:56.281 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.61560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78432.log 2024-05-28T06:27:56.281 INFO:teuthology.orchestra.run.smithi096.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.61560.log.gz 2024-05-28T06:27:56.281 INFO:teuthology.orchestra.run.smithi096.stderr: 89.8% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-05-28T06:27:56.282 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.48473.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78979.log 2024-05-28T06:27:56.282 INFO:teuthology.orchestra.run.smithi096.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.48473.log.gz 2024-05-28T06:27:56.283 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.78432.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69830.log 2024-05-28T06:27:56.284 INFO:teuthology.orchestra.run.smithi096.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.78432.log.gz 2024-05-28T06:27:56.285 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.78979.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74248.log 2024-05-28T06:27:56.286 INFO:teuthology.orchestra.run.smithi096.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.78979.log.gz 2024-05-28T06:27:56.287 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.69830.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.69830.log.gz 2024-05-28T06:27:56.287 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53676.log 2024-05-28T06:27:56.287 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.74248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50183.log 2024-05-28T06:27:56.288 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.53676.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.74248.log.gz 2024-05-28T06:27:56.288 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81969.log 2024-05-28T06:27:56.288 INFO:teuthology.orchestra.run.smithi096.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.53676.log.gz 2024-05-28T06:27:56.289 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.50183.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50183.log.gz 2024-05-28T06:27:56.290 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54394.log 2024-05-28T06:27:56.290 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.81969.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86767.log 2024-05-28T06:27:56.291 INFO:teuthology.orchestra.run.smithi096.stderr: 88.8%/var/log/ceph/ceph-client.admin.54394.log: -- replaced with /var/log/ceph/ceph-client.admin.81969.log.gz 2024-05-28T06:27:56.292 INFO:teuthology.orchestra.run.smithi096.stderr: 88.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.54394.log.gz 2024-05-28T06:27:56.292 INFO:teuthology.orchestra.run.smithi096.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.47305.log 2024-05-28T06:27:56.293 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.86767.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.86767.log.gz 2024-05-28T06:27:56.293 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72934.log 2024-05-28T06:27:56.294 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.47305.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68236.log 2024-05-28T06:27:56.294 INFO:teuthology.orchestra.run.smithi096.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47305.log.gz 2024-05-28T06:27:56.295 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.72934.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50415.log 2024-05-28T06:27:56.295 INFO:teuthology.orchestra.run.smithi096.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.72934.log.gz 2024-05-28T06:27:56.296 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.68236.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.68236.log.gz 2024-05-28T06:27:56.296 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73959.log 2024-05-28T06:27:56.297 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.50415.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74925.log 2024-05-28T06:27:56.298 INFO:teuthology.orchestra.run.smithi096.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50415.log.gz 2024-05-28T06:27:56.299 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.73959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63107.log 2024-05-28T06:27:56.299 INFO:teuthology.orchestra.run.smithi096.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.73959.log.gz 2024-05-28T06:27:56.300 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.74925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82855.log 2024-05-28T06:27:56.300 INFO:teuthology.orchestra.run.smithi096.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.74925.log.gz 2024-05-28T06:27:56.301 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.63107.log: gzip -5 --verbose 89.2% -- -- replaced with /var/log/ceph/ceph-client.admin.63107.log.gz 2024-05-28T06:27:56.301 INFO:teuthology.orchestra.run.smithi096.stderr: /var/log/ceph/ceph-client.admin.44529.log 2024-05-28T06:27:56.302 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.82855.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80574.log 2024-05-28T06:27:56.303 INFO:teuthology.orchestra.run.smithi096.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.82855.log.gz 2024-05-28T06:27:56.304 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.44529.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80134.log 2024-05-28T06:27:56.304 INFO:teuthology.orchestra.run.smithi096.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44529.log.gz 2024-05-28T06:27:56.305 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.80574.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.80574.log.gz 2024-05-28T06:27:56.305 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85737.log 2024-05-28T06:27:56.306 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.80134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85901.log 2024-05-28T06:27:56.306 INFO:teuthology.orchestra.run.smithi096.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.80134.log.gz 2024-05-28T06:27:56.307 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.85737.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.85737.log.gz 2024-05-28T06:27:56.307 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77941.log 2024-05-28T06:27:56.309 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.85901.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75583.log 2024-05-28T06:27:56.309 INFO:teuthology.orchestra.run.smithi096.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.85901.log.gz 2024-05-28T06:27:56.310 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.77941.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85156.log 2024-05-28T06:27:56.310 INFO:teuthology.orchestra.run.smithi096.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.77941.log.gz 2024-05-28T06:27:56.310 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.75583.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67570.log 2024-05-28T06:27:56.311 INFO:teuthology.orchestra.run.smithi096.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.75583.log.gz 2024-05-28T06:27:56.312 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.85156.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50793.log 2024-05-28T06:27:56.312 INFO:teuthology.orchestra.run.smithi096.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.85156.log.gz 2024-05-28T06:27:56.313 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.67570.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77326.log 2024-05-28T06:27:56.313 INFO:teuthology.orchestra.run.smithi096.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.67570.log.gz 2024-05-28T06:27:56.314 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.50793.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55103.log 2024-05-28T06:27:56.314 INFO:teuthology.orchestra.run.smithi096.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50793.log.gz 2024-05-28T06:27:56.315 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.77326.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42891.log 2024-05-28T06:27:56.316 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.55103.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.77326.log.gz 2024-05-28T06:27:56.317 INFO:teuthology.orchestra.run.smithi096.stderr: 88.4%gzip -- replaced with /var/log/ceph/ceph-client.admin.55103.log.gz -5 2024-05-28T06:27:56.317 INFO:teuthology.orchestra.run.smithi096.stderr: --verbose -- /var/log/ceph/ceph-client.admin.77605.log 2024-05-28T06:27:56.318 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.42891.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42891.log.gz 2024-05-28T06:27:56.318 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82473.log 2024-05-28T06:27:56.319 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.77605.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48047.log 2024-05-28T06:27:56.319 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.82473.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.77605.log.gz 2024-05-28T06:27:56.320 INFO:teuthology.orchestra.run.smithi096.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.82473.log.gz 2024-05-28T06:27:56.320 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85569.log 2024-05-28T06:27:56.321 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.48047.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48047.log.gz 2024-05-28T06:27:56.321 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49889.log 2024-05-28T06:27:56.322 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.85569.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75349.log 2024-05-28T06:27:56.323 INFO:teuthology.orchestra.run.smithi096.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.85569.log.gz 2024-05-28T06:27:56.324 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.49889.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79315.log 2024-05-28T06:27:56.324 INFO:teuthology.orchestra.run.smithi096.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.49889.log.gz 2024-05-28T06:27:56.325 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.75349.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.75349.log.gz 2024-05-28T06:27:56.325 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62683.log 2024-05-28T06:27:56.326 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.79315.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82431.log 2024-05-28T06:27:56.326 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.62683.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.79315.log.gz 2024-05-28T06:27:56.327 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43697.log 2024-05-28T06:27:56.327 INFO:teuthology.orchestra.run.smithi096.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.62683.log.gz 2024-05-28T06:27:56.328 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.82431.log: gzip -5 88.2% --verbose -- replaced with /var/log/ceph/ceph-client.admin.82431.log.gz -- 2024-05-28T06:27:56.328 INFO:teuthology.orchestra.run.smithi096.stderr: /var/log/ceph/ceph-client.admin.61267.log 2024-05-28T06:27:56.329 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.43697.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42679.log 2024-05-28T06:27:56.330 INFO:teuthology.orchestra.run.smithi096.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.43697.log.gz 2024-05-28T06:27:56.331 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.61267.log: gzip 87.7% -- replaced with /var/log/ceph/ceph-client.admin.61267.log.gz -5 2024-05-28T06:27:56.331 INFO:teuthology.orchestra.run.smithi096.stderr: --verbose -- /var/log/ceph/ceph-client.admin.69746.log 2024-05-28T06:27:56.332 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.42679.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.42679.log.gz 2024-05-28T06:27:56.332 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48887.log 2024-05-28T06:27:56.333 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.69746.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71004.log 2024-05-28T06:27:56.333 INFO:teuthology.orchestra.run.smithi096.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.69746.log.gz 2024-05-28T06:27:56.334 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.48887.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54790.log 2024-05-28T06:27:56.334 INFO:teuthology.orchestra.run.smithi096.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48887.log.gz 2024-05-28T06:27:56.335 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.71004.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.71004.log.gz 2024-05-28T06:27:56.336 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75223.log 2024-05-28T06:27:56.337 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.54790.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49181.log 2024-05-28T06:27:56.337 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.75223.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.54790.log.gz 2024-05-28T06:27:56.338 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83503.log 2024-05-28T06:27:56.338 INFO:teuthology.orchestra.run.smithi096.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.75223.log.gz 2024-05-28T06:27:56.339 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.49181.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.75861.log 2024-05-28T06:27:56.339 INFO:teuthology.orchestra.run.smithi096.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.49181.log.gz 2024-05-28T06:27:56.340 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.83503.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42975.log 2024-05-28T06:27:56.340 INFO:teuthology.orchestra.run.smithi096.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.83503.log.gz 2024-05-28T06:27:56.341 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.0.75861.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49433.log 2024-05-28T06:27:56.342 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.42975.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59329.log 2024-05-28T06:27:56.343 INFO:teuthology.orchestra.run.smithi096.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.42975.log.gz 2024-05-28T06:27:56.344 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.49433.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50373.log 2024-05-28T06:27:56.344 INFO:teuthology.orchestra.run.smithi096.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.49433.log.gz 2024-05-28T06:27:56.345 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.59329.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46874.log 2024-05-28T06:27:56.345 INFO:teuthology.orchestra.run.smithi096.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.59329.log.gz 2024-05-28T06:27:56.346 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.50373.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85114.log 2024-05-28T06:27:56.347 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.46874.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.50373.log.gz 2024-05-28T06:27:56.348 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71046.log 2024-05-28T06:27:56.348 INFO:teuthology.orchestra.run.smithi096.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46874.log.gz 2024-05-28T06:27:56.349 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.85114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64437.log 2024-05-28T06:27:56.349 INFO:teuthology.orchestra.run.smithi096.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.85114.log.gz 2024-05-28T06:27:56.350 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.71046.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58904.log 2024-05-28T06:27:56.350 INFO:teuthology.orchestra.run.smithi096.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.71046.log.gz 2024-05-28T06:27:56.351 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.64437.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.64437.log.gz 2024-05-28T06:27:56.351 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-05-28T06:27:56.352 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.58904.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50331.log 2024-05-28T06:27:56.354 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-osd.3.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.58904.log.gz 2024-05-28T06:27:56.354 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72211.log 2024-05-28T06:27:56.355 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.50331.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76606.log 2024-05-28T06:27:56.355 INFO:teuthology.orchestra.run.smithi096.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50331.log.gz 2024-05-28T06:27:56.356 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.72211.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.72211.log.gz 2024-05-28T06:27:56.357 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52794.log 2024-05-28T06:27:56.357 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.76606.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.76606.log.gz 2024-05-28T06:27:56.358 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47837.log 2024-05-28T06:27:56.359 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.52794.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71172.log 2024-05-28T06:27:56.359 INFO:teuthology.orchestra.run.smithi096.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.52794.log.gz 2024-05-28T06:27:56.361 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.47837.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69089.log 2024-05-28T06:27:56.361 INFO:teuthology.orchestra.run.smithi096.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.47837.log.gz 2024-05-28T06:27:56.362 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.71172.log: gzip -5 87.8% --verbose -- replaced with /var/log/ceph/ceph-client.admin.71172.log.gz -- 2024-05-28T06:27:56.362 INFO:teuthology.orchestra.run.smithi096.stderr: /var/log/ceph/ceph-client.admin.50985.log 2024-05-28T06:27:56.363 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.69089.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81281.log 2024-05-28T06:27:56.363 INFO:teuthology.orchestra.run.smithi096.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.69089.log.gz 2024-05-28T06:27:56.365 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.50985.log: gzip -5 --verbose -- 88.2% /var/log/ceph/ceph-client.admin.66510.log 2024-05-28T06:27:56.365 INFO:teuthology.orchestra.run.smithi096.stderr: -- replaced with /var/log/ceph/ceph-client.admin.50985.log.gz 2024-05-28T06:27:56.366 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.81281.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53928.log 2024-05-28T06:27:56.366 INFO:teuthology.orchestra.run.smithi096.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.81281.log.gz 2024-05-28T06:27:56.367 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.66510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82305.log 2024-05-28T06:27:56.367 INFO:teuthology.orchestra.run.smithi096.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.66510.log.gz 2024-05-28T06:27:56.371 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.53928.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69578.log 2024-05-28T06:27:56.371 INFO:teuthology.orchestra.run.smithi096.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.53928.log.gz 2024-05-28T06:27:56.371 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.82305.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.82305.log.gz 2024-05-28T06:27:56.372 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.69578.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.69578.log.gz 2024-05-28T06:27:56.373 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70920.log 2024-05-28T06:27:56.374 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49307.log 2024-05-28T06:27:56.375 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.70920.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57189.log 2024-05-28T06:27:56.375 INFO:teuthology.orchestra.run.smithi096.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.70920.log.gz 2024-05-28T06:27:56.376 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.49307.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73120.log 2024-05-28T06:27:56.377 INFO:teuthology.orchestra.run.smithi096.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.49307.log.gz 2024-05-28T06:27:56.378 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.57189.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.57189.log.gz 2024-05-28T06:27:56.379 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70878.log 2024-05-28T06:27:56.379 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.73120.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.73120.log.gz 2024-05-28T06:27:56.379 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84070.log 2024-05-28T06:27:56.381 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.70878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68362.log 2024-05-28T06:27:56.381 INFO:teuthology.orchestra.run.smithi096.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.70878.log.gz 2024-05-28T06:27:56.382 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.84070.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.84070.log.gz 2024-05-28T06:27:56.382 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70166.log 2024-05-28T06:27:56.383 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.68362.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43357.log 2024-05-28T06:27:56.384 INFO:teuthology.orchestra.run.smithi096.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.68362.log.gz 2024-05-28T06:27:56.385 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.70166.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52027.log 2024-05-28T06:27:56.385 INFO:teuthology.orchestra.run.smithi096.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.70166.log.gz 2024-05-28T06:27:56.386 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.43357.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.43357.log.gz 2024-05-28T06:27:56.387 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60955.log 2024-05-28T06:27:56.387 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.52027.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.52027.log.gz 2024-05-28T06:27:56.388 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60646.log 2024-05-28T06:27:56.389 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.60955.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62813.log 2024-05-28T06:27:56.389 INFO:teuthology.orchestra.run.smithi096.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.60955.log.gz 2024-05-28T06:27:56.391 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.60646.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.60646.log.gz 2024-05-28T06:27:56.392 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.62813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57669.log 2024-05-28T06:27:56.392 INFO:teuthology.orchestra.run.smithi096.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.62813.log.gz 2024-05-28T06:27:56.393 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77899.log 2024-05-28T06:27:56.394 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.57669.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73917.log 2024-05-28T06:27:56.395 INFO:teuthology.orchestra.run.smithi096.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.57669.log.gz 2024-05-28T06:27:56.395 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.77899.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.77899.log.gz 2024-05-28T06:27:56.396 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82263.log 2024-05-28T06:27:56.397 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.73917.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58862.log 2024-05-28T06:27:56.397 INFO:teuthology.orchestra.run.smithi096.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.73917.log.gz 2024-05-28T06:27:56.398 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.82263.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81644.log 2024-05-28T06:27:56.399 INFO:teuthology.orchestra.run.smithi096.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.82263.log.gz 2024-05-28T06:27:56.400 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.58862.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.58862.log.gz 2024-05-28T06:27:56.401 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74195.log 2024-05-28T06:27:56.401 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.81644.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.81644.log.gz 2024-05-28T06:27:56.402 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56734.log 2024-05-28T06:27:56.403 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.74195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76893.log 2024-05-28T06:27:56.403 INFO:teuthology.orchestra.run.smithi096.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.74195.log.gz 2024-05-28T06:27:56.404 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.56734.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.56734.log.gz 2024-05-28T06:27:56.405 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53382.log 2024-05-28T06:27:56.406 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.76893.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.76893.log.gz 2024-05-28T06:27:56.406 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78295.log 2024-05-28T06:27:56.407 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.53382.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48929.log 2024-05-28T06:27:56.407 INFO:teuthology.orchestra.run.smithi096.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.53382.log.gz 2024-05-28T06:27:56.409 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.78295.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.78295.log.gz 2024-05-28T06:27:56.410 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5/var/log/ceph/ceph-client.admin.48929.log: --verbose -- /var/log/ceph/ceph-client.admin.44655.log 2024-05-28T06:27:56.410 INFO:teuthology.orchestra.run.smithi096.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.48929.log.gz 2024-05-28T06:27:56.411 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57315.log 2024-05-28T06:27:56.412 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.44655.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50877.log 2024-05-28T06:27:56.413 INFO:teuthology.orchestra.run.smithi096.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44655.log.gz 2024-05-28T06:27:56.414 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.57315.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.57315.log.gz 2024-05-28T06:27:56.414 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66327.log 2024-05-28T06:27:56.415 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.50877.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.50877.log.gz 2024-05-28T06:27:56.416 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52212.log 2024-05-28T06:27:56.417 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.66327.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71448.log 2024-05-28T06:27:56.417 INFO:teuthology.orchestra.run.smithi096.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.66327.log.gz 2024-05-28T06:27:56.418 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.52212.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.52212.log.gz 2024-05-28T06:27:56.422 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.71448.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.71448.log.gz 2024-05-28T06:27:56.423 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72033.log 2024-05-28T06:27:56.423 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54565.log 2024-05-28T06:27:56.423 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.72033.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.72033.log.gz 2024-05-28T06:27:56.423 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80987.log 2024-05-28T06:27:56.424 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.54565.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61998.log 2024-05-28T06:27:56.424 INFO:teuthology.orchestra.run.smithi096.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.54565.log.gz 2024-05-28T06:27:56.426 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.80987.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.80987.log.gz 2024-05-28T06:27:56.426 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.61998.log: -- /var/log/ceph/ceph-client.admin.42148.log 2024-05-28T06:27:56.427 INFO:teuthology.orchestra.run.smithi096.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.61998.log.gz 2024-05-28T06:27:56.427 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73469.log 2024-05-28T06:27:56.429 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.42148.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52254.log 2024-05-28T06:27:56.429 INFO:teuthology.orchestra.run.smithi096.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42148.log.gz 2024-05-28T06:27:56.430 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.73469.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.73469.log.gz 2024-05-28T06:27:56.431 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53760.log 2024-05-28T06:27:56.431 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.52254.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.52254.log.gz 2024-05-28T06:27:56.432 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54521.log 2024-05-28T06:27:56.433 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.53760.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69620.log 2024-05-28T06:27:56.433 INFO:teuthology.orchestra.run.smithi096.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53760.log.gz 2024-05-28T06:27:56.434 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.54521.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84207.log 2024-05-28T06:27:56.435 INFO:teuthology.orchestra.run.smithi096.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.54521.log.gz 2024-05-28T06:27:56.436 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.69620.log: -- /var/log/ceph/ceph-client.admin.73036.log 2024-05-28T06:27:56.437 INFO:teuthology.orchestra.run.smithi096.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.69620.log.gz/var/log/ceph/ceph-client.admin.84207.log: 2024-05-28T06:27:56.438 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 88.5% -- replaced with /var/log/ceph/ceph-client.admin.84207.log.gz 2024-05-28T06:27:56.438 INFO:teuthology.orchestra.run.smithi096.stderr: --verbose -- /var/log/ceph/ceph-client.admin.50015.log 2024-05-28T06:27:56.439 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.73036.log: 89.0%gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-05-28T06:27:56.439 INFO:teuthology.orchestra.run.smithi096.stderr: -- replaced with /var/log/ceph/ceph-client.admin.73036.log.gz 2024-05-28T06:27:56.440 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.50015.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58736.log 2024-05-28T06:27:56.441 INFO:teuthology.orchestra.run.smithi096.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50015.log.gz 2024-05-28T06:27:56.442 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-mds.c.log: /var/log/ceph/ceph-client.admin.58736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59953.log 2024-05-28T06:27:56.443 INFO:teuthology.orchestra.run.smithi096.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.58736.log.gz 2024-05-28T06:27:56.445 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.59953.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.59953.log.gz 2024-05-28T06:27:56.446 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72472.log 2024-05-28T06:27:56.448 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61830.log 2024-05-28T06:27:56.449 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.72472.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.72472.log.gz 2024-05-28T06:27:56.450 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77019.log 2024-05-28T06:27:56.450 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.61830.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.61830.log.gz 2024-05-28T06:27:56.451 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43059.log 2024-05-28T06:27:56.452 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.77019.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.77019.log.gz 2024-05-28T06:27:56.453 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55880.log 2024-05-28T06:27:56.454 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.43059.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.43059.log.gz 2024-05-28T06:27:56.455 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61602.log 2024-05-28T06:27:56.456 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.55880.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.55880.log.gz 2024-05-28T06:27:56.458 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.61602.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.61602.log.gz 2024-05-28T06:27:56.458 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.51263.log 2024-05-28T06:27:56.460 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46580.log 2024-05-28T06:27:56.463 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.0.51263.log: /var/log/ceph/ceph-client.admin.46580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58946.log 2024-05-28T06:27:56.463 INFO:teuthology.orchestra.run.smithi096.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.46580.log.gz 2024-05-28T06:27:56.464 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87440.log 2024-05-28T06:27:56.466 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.58946.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58297.log 2024-05-28T06:27:56.467 INFO:teuthology.orchestra.run.smithi096.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.58946.log.gz 2024-05-28T06:27:56.469 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.58297.log: /var/log/ceph/ceph-client.admin.87440.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.87440.log.gz 2024-05-28T06:27:56.470 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71130.log 2024-05-28T06:27:56.472 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.71130.log: gzip -5 --verbose -- 88.6% -- replaced with /var/log/ceph/ceph-client.admin.58297.log.gz 2024-05-28T06:27:56.472 INFO:teuthology.orchestra.run.smithi096.stderr: /var/log/ceph/ceph-client.admin.47447.log 2024-05-28T06:27:56.474 INFO:teuthology.orchestra.run.smithi096.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.71130.log.gz 2024-05-28T06:27:56.475 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77061.log 2024-05-28T06:27:56.479 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.47447.log: /var/log/ceph/ceph-client.admin.77061.log: 89.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68866.log 2024-05-28T06:27:56.480 INFO:teuthology.orchestra.run.smithi096.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47447.log.gz 2024-05-28T06:27:56.481 INFO:teuthology.orchestra.run.smithi096.stderr: -- replaced with /var/log/ceph/ceph-client.admin.77061.log.gz 2024-05-28T06:27:56.482 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44781.log 2024-05-28T06:27:56.487 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.68866.log: /var/log/ceph/ceph-client.admin.44781.log: 88.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73330.log 2024-05-28T06:27:56.487 INFO:teuthology.orchestra.run.smithi096.stderr: -- replaced with /var/log/ceph/ceph-client.admin.68866.log.gz 2024-05-28T06:27:56.488 INFO:teuthology.orchestra.run.smithi096.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44781.log.gz 2024-05-28T06:27:56.490 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64521.log 2024-05-28T06:27:56.494 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.73330.log: /var/log/ceph/ceph-client.admin.64521.log: 88.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78253.log 2024-05-28T06:27:56.495 INFO:teuthology.orchestra.run.smithi096.stderr: -- replaced with /var/log/ceph/ceph-client.admin.73330.log.gz 2024-05-28T06:27:56.496 INFO:teuthology.orchestra.run.smithi096.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.64521.log.gz 2024-05-28T06:27:56.497 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74542.log 2024-05-28T06:27:56.502 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.78253.log: /var/log/ceph/ceph-client.admin.74542.log: 87.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64013.log 2024-05-28T06:27:56.502 INFO:teuthology.orchestra.run.smithi096.stderr: -- replaced with /var/log/ceph/ceph-client.admin.78253.log.gz 2024-05-28T06:27:56.503 INFO:teuthology.orchestra.run.smithi096.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.74542.log.gz 2024-05-28T06:27:56.504 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53634.log 2024-05-28T06:27:56.509 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.64013.log: 87.2%gzip/var/log/ceph/ceph-client.admin.53634.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.77521.log 2024-05-28T06:27:56.510 INFO:teuthology.orchestra.run.smithi096.stderr: -- replaced with /var/log/ceph/ceph-client.admin.64013.log.gz 2024-05-28T06:27:56.511 INFO:teuthology.orchestra.run.smithi096.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53634.log.gz 2024-05-28T06:27:56.511 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68572.log 2024-05-28T06:27:56.516 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.77521.log: /var/log/ceph/ceph-client.admin.68572.log: 89.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47347.log 2024-05-28T06:27:56.516 INFO:teuthology.orchestra.run.smithi096.stderr: -- replaced with /var/log/ceph/ceph-client.admin.77521.log.gz 2024-05-28T06:27:56.517 INFO:teuthology.orchestra.run.smithi096.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.68572.log.gz 2024-05-28T06:27:56.518 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44039.log 2024-05-28T06:27:56.523 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.47347.log: /var/log/ceph/ceph-client.admin.44039.log: 88.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86198.log 2024-05-28T06:27:56.524 INFO:teuthology.orchestra.run.smithi096.stderr: -- replaced with /var/log/ceph/ceph-client.admin.47347.log.gz 2024-05-28T06:27:56.524 INFO:teuthology.orchestra.run.smithi096.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44039.log.gz 2024-05-28T06:27:56.527 INFO:teuthology.orchestra.run.smithi096.stderr:gzip/var/log/ceph/ceph-client.admin.86198.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.78853.log 2024-05-28T06:27:56.531 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.78853.log: 89.4%gzip -5 --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.86198.log.gz /var/log/ceph/ceph-client.admin.59415.log 2024-05-28T06:27:56.531 INFO:teuthology.orchestra.run.smithi096.stderr: 2024-05-28T06:27:56.532 INFO:teuthology.orchestra.run.smithi096.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.78853.log.gz 2024-05-28T06:27:56.533 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70226.log 2024-05-28T06:27:56.537 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.70226.log: /var/log/ceph/ceph-client.admin.59415.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43694.log 2024-05-28T06:27:56.539 INFO:teuthology.orchestra.run.smithi096.stderr: 88.5% 88.2% -- replaced with /var/log/ceph/ceph-client.admin.59415.log.gz 2024-05-28T06:27:56.540 INFO:teuthology.orchestra.run.smithi096.stderr: -- replaced with /var/log/ceph/ceph-client.admin.70226.log.gz 2024-05-28T06:27:56.541 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65847.log 2024-05-28T06:27:56.545 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.65847.log: /var/log/ceph/ceph-client.admin.43694.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.43694.log.gz 2024-05-28T06:27:56.546 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86466.log 2024-05-28T06:27:56.546 INFO:teuthology.orchestra.run.smithi096.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.65847.log.gz 2024-05-28T06:27:56.547 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43690.log 2024-05-28T06:27:56.551 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.43690.log: /var/log/ceph/ceph-client.admin.86466.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-05-28T06:27:56.553 INFO:teuthology.orchestra.run.smithi096.stderr: 87.9% 88.7% -- replaced with /var/log/ceph/ceph-client.admin.86466.log.gz 2024-05-28T06:27:56.554 INFO:teuthology.orchestra.run.smithi096.stderr: -- replaced with /var/log/ceph/ceph-client.admin.43690.log.gz 2024-05-28T06:27:56.555 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-05-28T06:27:56.556 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66678.log 2024-05-28T06:27:56.558 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80843.log 2024-05-28T06:27:56.564 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-osd.1.log: /var/log/ceph/ceph-client.admin.80843.log: /var/log/ceph/ceph-client.admin.66678.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.66678.log.gz 2024-05-28T06:27:56.565 INFO:teuthology.orchestra.run.smithi096.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.80843.log.gz 2024-05-28T06:27:56.565 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62515.log 2024-05-28T06:27:56.566 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84888.log 2024-05-28T06:27:56.567 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.62515.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.62515.log.gz 2024-05-28T06:27:56.571 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74001.log 2024-05-28T06:27:56.572 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.84888.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51486.log 2024-05-28T06:27:56.573 INFO:teuthology.orchestra.run.smithi096.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.84888.log.gz 2024-05-28T06:27:56.574 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.74001.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63378.log 2024-05-28T06:27:56.574 INFO:teuthology.orchestra.run.smithi096.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.74001.log.gz 2024-05-28T06:27:56.577 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.51486.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62250.log 2024-05-28T06:27:56.579 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.63378.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51486.log.gz 2024-05-28T06:27:56.579 INFO:teuthology.orchestra.run.smithi096.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.63378.log.gz 2024-05-28T06:27:56.584 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.62250.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.62250.log.gz 2024-05-28T06:27:56.585 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58820.log 2024-05-28T06:27:56.586 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76000.log 2024-05-28T06:27:56.590 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65518.log 2024-05-28T06:27:56.592 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.58820.log: /var/log/ceph/ceph-client.admin.76000.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.58820.log.gz 2024-05-28T06:27:56.592 INFO:teuthology.orchestra.run.smithi096.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.76000.log.gz 2024-05-28T06:27:56.592 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87314.log 2024-05-28T06:27:56.594 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.65518.log: 94.6% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-05-28T06:27:56.597 INFO:teuthology.orchestra.run.smithi096.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.65518.log.gz 2024-05-28T06:27:56.599 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.87314.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.87314.log.gz 2024-05-28T06:27:56.602 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65636.log 2024-05-28T06:27:56.603 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46958.log 2024-05-28T06:27:56.608 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.65636.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.65636.log.gz 2024-05-28T06:27:56.609 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73511.log 2024-05-28T06:27:56.609 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.46958.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.46958.log.gz 2024-05-28T06:27:56.612 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80355.log 2024-05-28T06:27:56.614 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.73511.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.73511.log.gz 2024-05-28T06:27:56.615 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75475.log 2024-05-28T06:27:56.617 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.80355.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.80355.log.gz 2024-05-28T06:27:56.621 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58171.log 2024-05-28T06:27:56.623 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.75475.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.75475.log.gz 2024-05-28T06:27:56.625 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.58171.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.58171.log.gz 2024-05-28T06:27:56.627 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-05-28T06:27:56.630 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82813.log 2024-05-28T06:27:56.632 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85383.log 2024-05-28T06:27:56.634 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.82813.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.82813.log.gz 2024-05-28T06:27:56.636 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63293.log 2024-05-28T06:27:56.638 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.85383.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.85383.log.gz 2024-05-28T06:27:56.638 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52752.log 2024-05-28T06:27:56.640 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.63293.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.63293.log.gz 2024-05-28T06:27:56.642 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52482.log 2024-05-28T06:27:56.644 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.52752.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52752.log.gz 2024-05-28T06:27:56.644 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55019.log 2024-05-28T06:27:56.650 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.52482.log: /var/log/ceph/ceph-client.admin.55019.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52482.log.gz 2024-05-28T06:27:56.650 INFO:teuthology.orchestra.run.smithi096.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.55019.log.gz 2024-05-28T06:27:56.651 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67840.log 2024-05-28T06:27:56.654 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78043.log 2024-05-28T06:27:56.656 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.67840.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.67840.log.gz 2024-05-28T06:27:56.658 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55229.log 2024-05-28T06:27:56.660 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.78043.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.78043.log.gz 2024-05-28T06:27:56.660 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46134.log 2024-05-28T06:27:56.662 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.55229.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.55229.log.gz 2024-05-28T06:27:56.664 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-mds.d.log: /var/log/ceph/ceph-mon.b.log: 91.9% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-05-28T06:27:56.664 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64395.log 2024-05-28T06:27:56.666 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.46134.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46134.log.gz 2024-05-28T06:27:56.666 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50457.log 2024-05-28T06:27:56.667 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.64395.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.64395.log.gz 2024-05-28T06:27:56.669 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84298.log 2024-05-28T06:27:56.671 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.50457.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.50457.log.gz 2024-05-28T06:27:56.672 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.84298.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47263.log 2024-05-28T06:27:56.673 INFO:teuthology.orchestra.run.smithi096.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.84298.log.gz 2024-05-28T06:27:56.677 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68740.log 2024-05-28T06:27:56.679 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.47263.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.47263.log.gz 2024-05-28T06:27:56.681 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.68740.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.68740.log.gz 2024-05-28T06:27:56.681 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57585.log 2024-05-28T06:27:56.682 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83251.log 2024-05-28T06:27:56.685 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.57585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67966.log 2024-05-28T06:27:56.686 INFO:teuthology.orchestra.run.smithi096.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.57585.log.gz 2024-05-28T06:27:56.687 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.83251.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.83251.log.gz 2024-05-28T06:27:56.690 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63420.log 2024-05-28T06:27:56.692 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.67966.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.67966.log.gz 2024-05-28T06:27:56.695 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.63420.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.63420.log.gz 2024-05-28T06:27:56.696 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59510.log 2024-05-28T06:27:56.700 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48557.log 2024-05-28T06:27:56.701 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.59510.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.59510.log.gz 2024-05-28T06:27:56.704 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73162.log 2024-05-28T06:27:56.706 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.48557.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.48557.log.gz 2024-05-28T06:27:56.708 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81588.log 2024-05-28T06:27:56.709 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.73162.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.73162.log.gz 2024-05-28T06:27:56.713 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57976.log 2024-05-28T06:27:56.715 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.81588.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.81588.log.gz 2024-05-28T06:27:56.719 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.57976.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.57976.log.gz 2024-05-28T06:27:56.720 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53059.log 2024-05-28T06:27:56.725 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70790.log 2024-05-28T06:27:56.727 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.53059.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53059.log.gz 2024-05-28T06:27:56.727 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68278.log 2024-05-28T06:27:56.729 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.70790.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.70790.log.gz 2024-05-28T06:27:56.729 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69494.log 2024-05-28T06:27:56.730 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.68278.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.68278.log.gz 2024-05-28T06:27:56.733 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63065.log 2024-05-28T06:27:56.735 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.69494.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.69494.log.gz 2024-05-28T06:27:56.738 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45566.log 2024-05-28T06:27:56.740 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.63065.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.63065.log.gz 2024-05-28T06:27:56.743 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.45566.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44697.log 2024-05-28T06:27:56.744 INFO:teuthology.orchestra.run.smithi096.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.45566.log.gz 2024-05-28T06:27:56.746 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66906.log 2024-05-28T06:27:56.748 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.44697.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.44697.log.gz 2024-05-28T06:27:56.750 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64563.log 2024-05-28T06:27:56.755 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.66906.log: /var/log/ceph/ceph-client.admin.64563.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.66906.log.gz 89.0% -- replaced with /var/log/ceph/ceph-client.admin.64563.log.gz 2024-05-28T06:27:56.755 INFO:teuthology.orchestra.run.smithi096.stderr: 2024-05-28T06:27:56.756 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61081.log 2024-05-28T06:27:56.762 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87679.log 2024-05-28T06:27:56.763 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.61081.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.61081.log.gz 2024-05-28T06:27:56.764 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58213.log 2024-05-28T06:27:56.766 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.87679.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.87679.log.gz 2024-05-28T06:27:56.766 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53101.log 2024-05-28T06:27:56.768 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.58213.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.58213.log.gz 2024-05-28T06:27:56.770 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58609.log 2024-05-28T06:27:56.772 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.53101.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.53101.log.gz 2024-05-28T06:27:56.772 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53508.log 2024-05-28T06:27:56.773 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.58609.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.58609.log.gz 2024-05-28T06:27:56.778 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.53508.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.53508.log.gz 2024-05-28T06:27:56.778 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85425.log 2024-05-28T06:27:56.782 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71214.log 2024-05-28T06:27:56.783 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.85425.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.85425.log.gz 2024-05-28T06:27:56.786 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50751.log 2024-05-28T06:27:56.788 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.71214.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.71214.log.gz 2024-05-28T06:27:56.788 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79105.log 2024-05-28T06:27:56.790 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.50751.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47126.log 2024-05-28T06:27:56.792 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.79105.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.50751.log.gz 2024-05-28T06:27:56.795 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.47126.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85240.log 2024-05-28T06:27:56.795 INFO:teuthology.orchestra.run.smithi096.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.79105.log.gz 2024-05-28T06:27:56.796 INFO:teuthology.orchestra.run.smithi096.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47126.log.gz 2024-05-28T06:27:56.800 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76126.log 2024-05-28T06:27:56.801 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.85240.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52710.log 2024-05-28T06:27:56.801 INFO:teuthology.orchestra.run.smithi096.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.85240.log.gz 2024-05-28T06:27:56.803 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.76126.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.76126.log.gz 2024-05-28T06:27:56.803 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46244.log 2024-05-28T06:27:56.804 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.52710.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52710.log.gz 2024-05-28T06:27:56.806 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57711.log 2024-05-28T06:27:56.807 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.46244.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.46244.log.gz 2024-05-28T06:27:56.809 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62376.log 2024-05-28T06:27:56.811 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.57711.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.57711.log.gz 2024-05-28T06:27:56.812 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76977.log 2024-05-28T06:27:56.813 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.62376.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.62376.log.gz 2024-05-28T06:27:56.818 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.76977.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46538.log 2024-05-28T06:27:56.818 INFO:teuthology.orchestra.run.smithi096.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.76977.log.gz 2024-05-28T06:27:56.820 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78169.log 2024-05-28T06:27:56.821 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.46538.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46538.log.gz 2024-05-28T06:27:56.825 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.78169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62040.log 2024-05-28T06:27:56.826 INFO:teuthology.orchestra.run.smithi096.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.78169.log.gz 2024-05-28T06:27:56.830 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46832.log 2024-05-28T06:27:56.832 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.62040.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.62040.log.gz 2024-05-28T06:27:56.832 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68320.log 2024-05-28T06:27:56.834 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.46832.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.46832.log.gz 2024-05-28T06:27:56.837 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.68320.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55271.log 2024-05-28T06:27:56.838 INFO:teuthology.orchestra.run.smithi096.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.68320.log.gz 2024-05-28T06:27:56.841 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75433.log 2024-05-28T06:27:56.843 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.55271.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.55271.log.gz 2024-05-28T06:27:56.846 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67756.log 2024-05-28T06:27:56.847 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.75433.log: gzip 88.9% -- replaced with /var/log/ceph/ceph-client.admin.75433.log.gz 2024-05-28T06:27:56.847 INFO:teuthology.orchestra.run.smithi096.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.50099.log 2024-05-28T06:27:56.852 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.67756.log: 88.9%/var/log/ceph/ceph-client.admin.50099.log: -- replaced with /var/log/ceph/ceph-client.admin.67756.log.gz 2024-05-28T06:27:56.853 INFO:teuthology.orchestra.run.smithi096.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50099.log.gz 2024-05-28T06:27:56.854 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57129.log 2024-05-28T06:27:56.860 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70695.log 2024-05-28T06:27:56.861 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.57129.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.57129.log.gz 2024-05-28T06:27:56.863 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74637.log 2024-05-28T06:27:56.866 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.70695.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.70695.log.gz 2024-05-28T06:27:56.866 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43352.log 2024-05-28T06:27:56.868 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44403.log 2024-05-28T06:27:56.871 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.43352.log: /var/log/ceph/ceph-client.admin.74637.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.74637.log.gz 2024-05-28T06:27:56.871 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61039.log 2024-05-28T06:27:56.873 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.44403.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44403.log.gz 2024-05-28T06:27:56.874 INFO:teuthology.orchestra.run.smithi096.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.43352.log.gz 2024-05-28T06:27:56.874 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55187.log 2024-05-28T06:27:56.876 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.61039.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.61039.log.gz 2024-05-28T06:27:56.879 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.55187.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.55187.log.gz 2024-05-28T06:27:56.879 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68446.log 2024-05-28T06:27:56.881 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48635.log 2024-05-28T06:27:56.882 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.68446.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.68446.log.gz 2024-05-28T06:27:56.883 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51570.log 2024-05-28T06:27:56.886 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.48635.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59552.log 2024-05-28T06:27:56.887 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.51570.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48635.log.gz 2024-05-28T06:27:56.887 INFO:teuthology.orchestra.run.smithi096.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.51570.log.gz 2024-05-28T06:27:56.893 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.59552.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59121.log 2024-05-28T06:27:56.894 INFO:teuthology.orchestra.run.smithi096.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.59552.log.gz 2024-05-28T06:27:56.897 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48389.log 2024-05-28T06:27:56.898 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.59121.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86851.log 2024-05-28T06:27:56.899 INFO:teuthology.orchestra.run.smithi096.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.59121.log.gz 2024-05-28T06:27:56.900 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.48389.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.48389.log.gz 2024-05-28T06:27:56.902 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70040.log 2024-05-28T06:27:56.904 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.86851.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.86851.log.gz 2024-05-28T06:27:56.907 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85983.log 2024-05-28T06:27:56.910 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.70040.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.70040.log.gz 2024-05-28T06:27:56.911 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57045.log 2024-05-28T06:27:56.913 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.85983.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.85983.log.gz 2024-05-28T06:27:56.914 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78679.log 2024-05-28T06:27:56.915 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.57045.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.57045.log.gz 2024-05-28T06:27:56.921 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.78679.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-05-28T06:27:56.921 INFO:teuthology.orchestra.run.smithi096.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.78679.log.gz 2024-05-28T06:27:56.925 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46496.log 2024-05-28T06:27:56.926 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78512.log 2024-05-28T06:27:56.929 INFO:teuthology.orchestra.run.smithi096.stderr: 93.2%/var/log/ceph/ceph-client.admin.46496.log: -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-05-28T06:27:56.929 INFO:teuthology.orchestra.run.smithi096.stderr: 87.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77368.log 2024-05-28T06:27:56.929 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.78512.log: -- replaced with /var/log/ceph/ceph-client.admin.46496.log.gz 2024-05-28T06:27:56.931 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67337.log 2024-05-28T06:27:56.932 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.77368.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.77368.log.gz 2024-05-28T06:27:56.933 INFO:teuthology.orchestra.run.smithi096.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.78512.log.gz 2024-05-28T06:27:56.934 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78809.log 2024-05-28T06:27:56.936 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.67337.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77983.log 2024-05-28T06:27:56.937 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.78809.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.78809.log.gz 2024-05-28T06:27:56.938 INFO:teuthology.orchestra.run.smithi096.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.67337.log.gz 2024-05-28T06:27:56.941 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.77983.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.77983.log.gz 2024-05-28T06:27:56.941 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62292.log 2024-05-28T06:27:56.944 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86156.log 2024-05-28T06:27:56.945 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.62292.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.62292.log.gz 2024-05-28T06:27:56.947 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80945.log 2024-05-28T06:27:56.948 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.86156.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.86156.log.gz 2024-05-28T06:27:56.953 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68992.log 2024-05-28T06:27:56.955 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.80945.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.80945.log.gz 2024-05-28T06:27:56.956 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79549.log 2024-05-28T06:27:56.958 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.68992.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.68992.log.gz 2024-05-28T06:27:56.964 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.79549.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.79549.log.gz 2024-05-28T06:27:56.964 INFO:teuthology.orchestra.run.smithi123.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-05-28T06:27:56.965 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52422.log 2024-05-28T06:27:56.968 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64269.log 2024-05-28T06:27:56.970 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.52422.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52422.log.gz 2024-05-28T06:27:56.972 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71949.log 2024-05-28T06:27:56.974 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.64269.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.64269.log.gz 2024-05-28T06:27:56.975 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74840.log 2024-05-28T06:27:56.979 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.71949.log: 88.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47921.log 2024-05-28T06:27:56.981 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.74840.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.71949.log.gz 2024-05-28T06:27:56.983 INFO:teuthology.orchestra.run.smithi096.stderr: -- replaced with /var/log/ceph/ceph-client.admin.74840.log.gz 2024-05-28T06:27:56.985 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47389.log 2024-05-28T06:27:56.986 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.47921.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.47921.log.gz 2024-05-28T06:27:56.988 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58694.log 2024-05-28T06:27:56.990 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.47389.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.47389.log.gz 2024-05-28T06:27:56.991 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60562.log 2024-05-28T06:27:56.991 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.58694.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.58694.log.gz 2024-05-28T06:27:56.994 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.60562.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44031.log 2024-05-28T06:27:56.996 INFO:teuthology.orchestra.run.smithi096.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.60562.log.gz 2024-05-28T06:27:56.997 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54744.log 2024-05-28T06:27:56.998 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.44031.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44031.log.gz 2024-05-28T06:27:56.998 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68782.log 2024-05-28T06:27:57.002 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.54744.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.54744.log.gz 2024-05-28T06:27:57.003 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.68782.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.68782.log.gz 2024-05-28T06:27:57.004 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76851.log 2024-05-28T06:27:57.007 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47042.log 2024-05-28T06:27:57.008 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.76851.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.76851.log.gz 2024-05-28T06:27:57.015 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.47042.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.47042.log.gz 2024-05-28T06:27:57.015 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69410.log 2024-05-28T06:27:57.018 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59459.log 2024-05-28T06:27:57.020 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.69410.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.69410.log.gz 2024-05-28T06:27:57.022 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86240.log 2024-05-28T06:27:57.023 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.59459.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.59459.log.gz 2024-05-28T06:27:57.027 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50583.log 2024-05-28T06:27:57.028 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.86240.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.86240.log.gz 2024-05-28T06:27:57.034 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.50583.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70482.log 2024-05-28T06:27:57.034 INFO:teuthology.orchestra.run.smithi096.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.50583.log.gz 2024-05-28T06:27:57.037 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47221.log 2024-05-28T06:27:57.039 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.70482.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.70482.log.gz 2024-05-28T06:27:57.041 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.47221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71340.log 2024-05-28T06:27:57.042 INFO:teuthology.orchestra.run.smithi096.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.47221.log.gz 2024-05-28T06:27:57.044 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52650.log 2024-05-28T06:27:57.046 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.71340.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.71340.log.gz 2024-05-28T06:27:57.047 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84948.log 2024-05-28T06:27:57.048 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.52650.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.52650.log.gz 2024-05-28T06:27:57.050 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.84948.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.55615.log 2024-05-28T06:27:57.051 INFO:teuthology.orchestra.run.smithi096.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.84948.log.gz 2024-05-28T06:27:57.052 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83205.log 2024-05-28T06:27:57.054 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.0.55615.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59743.log 2024-05-28T06:27:57.058 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.59743.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.59743.log.gz 2024-05-28T06:27:57.059 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.83205.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.83205.log.gz 2024-05-28T06:27:57.060 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82347.log 2024-05-28T06:27:57.064 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59203.log 2024-05-28T06:27:57.066 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.82347.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.82347.log.gz 2024-05-28T06:27:57.066 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71865.log 2024-05-28T06:27:57.067 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.59203.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.59203.log.gz 2024-05-28T06:27:57.070 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77773.log 2024-05-28T06:27:57.071 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.71865.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.71865.log.gz 2024-05-28T06:27:57.072 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43700.log 2024-05-28T06:27:57.073 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.77773.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.77773.log.gz 2024-05-28T06:27:57.076 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42933.log 2024-05-28T06:27:57.077 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.43700.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.43700.log.gz 2024-05-28T06:27:57.078 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72598.log 2024-05-28T06:27:57.084 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.42933.log: /var/log/ceph/ceph-client.admin.72598.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48331.log 2024-05-28T06:27:57.084 INFO:teuthology.orchestra.run.smithi096.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.42933.log.gz 2024-05-28T06:27:57.084 INFO:teuthology.orchestra.run.smithi096.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.72598.log.gz 2024-05-28T06:27:57.086 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87230.log 2024-05-28T06:27:57.087 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.48331.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48331.log.gz 2024-05-28T06:27:57.088 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69914.log 2024-05-28T06:27:57.089 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.87230.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.87230.log.gz 2024-05-28T06:27:57.091 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.69914.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49847.log 2024-05-28T06:27:57.092 INFO:teuthology.orchestra.run.smithi096.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.69914.log.gz 2024-05-28T06:27:57.094 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54660.log 2024-05-28T06:27:57.095 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.49847.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49847.log.gz 2024-05-28T06:27:57.098 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85527.log 2024-05-28T06:27:57.099 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.54660.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.54660.log.gz 2024-05-28T06:27:57.103 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.85527.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.85527.log.gz 2024-05-28T06:27:57.105 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78895.log 2024-05-28T06:27:57.109 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71907.log 2024-05-28T06:27:57.110 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.78895.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.78895.log.gz 2024-05-28T06:27:57.112 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62855.log 2024-05-28T06:27:57.113 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.71907.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.71907.log.gz 2024-05-28T06:27:57.116 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43365.log 2024-05-28T06:27:57.117 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.62855.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.62855.log.gz 2024-05-28T06:27:57.121 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.43365.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.43365.log.gz 2024-05-28T06:27:57.122 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77410.log 2024-05-28T06:27:57.123 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66947.log 2024-05-28T06:27:57.124 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.77410.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.77410.log.gz 2024-05-28T06:27:57.128 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86508.log 2024-05-28T06:27:57.129 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.66947.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.66947.log.gz 2024-05-28T06:27:57.133 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.86508.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.86508.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58652.log 2024-05-28T06:27:57.133 INFO:teuthology.orchestra.run.smithi096.stderr: 2024-05-28T06:27:57.136 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74967.log 2024-05-28T06:27:57.138 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.58652.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.58652.log.gz 2024-05-28T06:27:57.140 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82011.log 2024-05-28T06:27:57.143 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.74967.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.74967.log.gz 2024-05-28T06:27:57.146 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.82011.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.82011.log.gz 2024-05-28T06:27:57.146 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60997.log 2024-05-28T06:27:57.148 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49601.log 2024-05-28T06:27:57.150 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.60997.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.60997.log.gz 2024-05-28T06:27:57.152 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84500.log 2024-05-28T06:27:57.154 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.49601.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.49601.log.gz 2024-05-28T06:27:57.158 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.84500.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.84500.log.gz 2024-05-28T06:27:57.159 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72430.log 2024-05-28T06:27:57.163 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66411.log 2024-05-28T06:27:57.165 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.72430.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.72430.log.gz 2024-05-28T06:27:57.167 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85653.log 2024-05-28T06:27:57.169 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.66411.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.66411.log.gz 2024-05-28T06:27:57.171 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72346.log 2024-05-28T06:27:57.173 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.85653.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.85653.log.gz 2024-05-28T06:27:57.175 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67152.log 2024-05-28T06:27:57.177 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.72346.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.72346.log.gz 2024-05-28T06:27:57.179 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68656.log 2024-05-28T06:27:57.181 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.67152.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.67152.log.gz 2024-05-28T06:27:57.185 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.68656.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.68656.log.gz 2024-05-28T06:27:57.186 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83070.log 2024-05-28T06:27:57.189 INFO:teuthology.orchestra.run.smithi123.stderr: 93.3% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-05-28T06:27:57.191 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59911.log 2024-05-28T06:27:57.193 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.83070.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.83070.log.gz 2024-05-28T06:27:57.195 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57357.log 2024-05-28T06:27:57.197 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.59911.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.59911.log.gz 2024-05-28T06:27:57.199 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87566.log 2024-05-28T06:27:57.201 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.57357.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42849.log 2024-05-28T06:27:57.204 INFO:teuthology.orchestra.run.smithi096.stderr: 89.0%/var/log/ceph/ceph-client.admin.87566.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67924.log 2024-05-28T06:27:57.205 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.42849.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.42849.log.gz 2024-05-28T06:27:57.206 INFO:teuthology.orchestra.run.smithi096.stderr: -- replaced with /var/log/ceph/ceph-client.admin.57357.log.gz 2024-05-28T06:27:57.207 INFO:teuthology.orchestra.run.smithi096.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.87566.log.gz 2024-05-28T06:27:57.208 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.67924.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72994.log 2024-05-28T06:27:57.210 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.72994.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55922.log 2024-05-28T06:27:57.211 INFO:teuthology.orchestra.run.smithi096.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.67924.log.gz 2024-05-28T06:27:57.213 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.55922.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.72994.log.gz 2024-05-28T06:27:57.214 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59785.log 2024-05-28T06:27:57.215 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70739.log 2024-05-28T06:27:57.218 INFO:teuthology.orchestra.run.smithi096.stderr: 88.4%/var/log/ceph/ceph-client.admin.59785.log: -- replaced with /var/log/ceph/ceph-client.admin.55922.log.gz 2024-05-28T06:27:57.219 INFO:teuthology.orchestra.run.smithi096.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.59785.log.gz 2024-05-28T06:27:57.220 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.70739.log: 88.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52836.log 2024-05-28T06:27:57.222 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84154.log 2024-05-28T06:27:57.222 INFO:teuthology.orchestra.run.smithi096.stderr: -- replaced with /var/log/ceph/ceph-client.admin.70739.log.gz 2024-05-28T06:27:57.225 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.52836.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52836.log.gz 2024-05-28T06:27:57.225 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82732.log 2024-05-28T06:27:57.226 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.84154.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.84154.log.gz 2024-05-28T06:27:57.228 INFO:teuthology.orchestra.run.smithi096.stderr:gzip/var/log/ceph/ceph-client.admin.82732.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.77284.log 2024-05-28T06:27:57.232 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.77284.log: 88.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.82732.log.gz -5 2024-05-28T06:27:57.232 INFO:teuthology.orchestra.run.smithi096.stderr: --verbose -- /var/log/ceph/ceph-client.admin.46370.log 2024-05-28T06:27:57.233 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.45410.log 2024-05-28T06:27:57.239 INFO:teuthology.orchestra.run.smithi096.stderr: 88.5%/var/log/ceph/ceph-client.admin.46370.log: -- replaced with /var/log/ceph/ceph-client.admin.77284.log.gz 2024-05-28T06:27:57.239 INFO:teuthology.orchestra.run.smithi096.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.46370.log.gz 2024-05-28T06:27:57.239 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.0.45410.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85031.log 2024-05-28T06:27:57.242 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.85031.log: 91.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65889.log 2024-05-28T06:27:57.242 INFO:teuthology.orchestra.run.smithi096.stderr: -- replaced with /var/log/ceph/ceph-client.0.45410.log.gz 2024-05-28T06:27:57.243 INFO:teuthology.orchestra.run.smithi096.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.85031.log.gz 2024-05-28T06:27:57.244 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67654.log 2024-05-28T06:27:57.246 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62334.log 2024-05-28T06:27:57.251 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.65889.log: /var/log/ceph/ceph-client.admin.67654.log: 88.6% 88.9% -- replaced with /var/log/ceph/ceph-client.admin.65889.log.gz 2024-05-28T06:27:57.251 INFO:teuthology.orchestra.run.smithi096.stderr: -- replaced with /var/log/ceph/ceph-client.admin.67654.log.gz 2024-05-28T06:27:57.251 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43699.log 2024-05-28T06:27:57.252 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.62334.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.62334.log.gz 2024-05-28T06:27:57.254 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.43699.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76168.log 2024-05-28T06:27:57.255 INFO:teuthology.orchestra.run.smithi096.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43699.log.gz 2024-05-28T06:27:57.257 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85198.log 2024-05-28T06:27:57.261 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.76168.log: /var/log/ceph/ceph-client.admin.85198.log: 88.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69872.log 2024-05-28T06:27:57.262 INFO:teuthology.orchestra.run.smithi096.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.76168.log.gz 2024-05-28T06:27:57.262 INFO:teuthology.orchestra.run.smithi096.stderr: -- replaced with /var/log/ceph/ceph-client.admin.85198.log.gz 2024-05-28T06:27:57.264 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43345.log 2024-05-28T06:27:57.268 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.69872.log: /var/log/ceph/ceph-client.admin.43345.log: 88.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79966.log 2024-05-28T06:27:57.269 INFO:teuthology.orchestra.run.smithi096.stderr: -- replaced with /var/log/ceph/ceph-client.admin.69872.log.gz 2024-05-28T06:27:57.270 INFO:teuthology.orchestra.run.smithi096.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43345.log.gz 2024-05-28T06:27:57.271 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73288.log 2024-05-28T06:27:57.275 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.79966.log: /var/log/ceph/ceph-client.admin.73288.log: 88.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81029.log 2024-05-28T06:27:57.276 INFO:teuthology.orchestra.run.smithi096.stderr: -- replaced with /var/log/ceph/ceph-client.admin.79966.log.gz 2024-05-28T06:27:57.278 INFO:teuthology.orchestra.run.smithi096.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.73288.log.gz 2024-05-28T06:27:57.278 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68488.log 2024-05-28T06:27:57.288 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.81029.log: /var/log/ceph/ceph-client.admin.68488.log: 88.4%gzip -- replaced with /var/log/ceph/ceph-client.admin.81029.log.gz -5 --verbose 2024-05-28T06:27:57.288 INFO:teuthology.orchestra.run.smithi096.stderr: -- /var/log/ceph/ceph-client.admin.64055.log 2024-05-28T06:27:57.288 INFO:teuthology.orchestra.run.smithi096.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.68488.log.gz 2024-05-28T06:27:57.288 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61770.log 2024-05-28T06:27:57.289 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.61770.log: /var/log/ceph/ceph-client.admin.64055.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64647.log 2024-05-28T06:27:57.291 INFO:teuthology.orchestra.run.smithi096.stderr: 87.4% 88.6% -- replaced with /var/log/ceph/ceph-client.admin.64055.log.gz 2024-05-28T06:27:57.291 INFO:teuthology.orchestra.run.smithi096.stderr: -- replaced with /var/log/ceph/ceph-client.admin.61770.log.gz 2024-05-28T06:27:57.293 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81365.log 2024-05-28T06:27:57.296 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.81365.log: /var/log/ceph/ceph-client.admin.64647.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60436.log 2024-05-28T06:27:57.298 INFO:teuthology.orchestra.run.smithi096.stderr: 90.9% 88.1% -- replaced with /var/log/ceph/ceph-client.admin.64647.log.gz 2024-05-28T06:27:57.299 INFO:teuthology.orchestra.run.smithi096.stderr: -- replaced with /var/log/ceph/ceph-client.admin.81365.log.gz 2024-05-28T06:27:57.301 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.60436.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.60436.log.gz 2024-05-28T06:27:57.302 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55145.log 2024-05-28T06:27:57.304 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.55145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81071.log 2024-05-28T06:27:57.305 INFO:teuthology.orchestra.run.smithi096.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.55145.log.gz 2024-05-28T06:27:57.310 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.79827.log 2024-05-28T06:27:57.311 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.81071.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.81071.log.gz 2024-05-28T06:27:57.313 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.0.79827.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65560.log 2024-05-28T06:27:57.315 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73246.log 2024-05-28T06:27:57.320 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.65560.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.65560.log.gz/var/log/ceph/ceph-client.admin.73246.log: 2024-05-28T06:27:57.321 INFO:teuthology.orchestra.run.smithi096.stderr: 88.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74500.log 2024-05-28T06:27:57.322 INFO:teuthology.orchestra.run.smithi096.stderr: -- replaced with /var/log/ceph/ceph-client.admin.73246.log.gz 2024-05-28T06:27:57.325 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59640.log 2024-05-28T06:27:57.327 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.74500.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.74500.log.gz 2024-05-28T06:27:57.329 INFO:teuthology.orchestra.run.smithi096.stderr:gzip/var/log/ceph/ceph-client.admin.59640.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.80801.log 2024-05-28T06:27:57.333 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63149.log 2024-05-28T06:27:57.334 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.80801.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.59640.log.gz 2024-05-28T06:27:57.334 INFO:teuthology.orchestra.run.smithi096.stderr: 96.2% -- replaced with /var/log/ceph/ceph-client.0.60297.log.gz 2024-05-28T06:27:57.337 INFO:teuthology.orchestra.run.smithi096.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.80801.log.gz 2024-05-28T06:27:57.340 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44235.log/var/log/ceph/ceph-client.admin.63149.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.63149.log.gz 2024-05-28T06:27:57.342 INFO:teuthology.orchestra.run.smithi096.stderr: 2024-05-28T06:27:57.347 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86977.log 2024-05-28T06:27:57.349 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.44235.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44235.log.gz 2024-05-28T06:27:57.350 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52069.log 2024-05-28T06:27:57.351 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.86977.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.86977.log.gz 2024-05-28T06:27:57.357 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66243.log 2024-05-28T06:27:57.357 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.52069.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52069.log.gz 2024-05-28T06:27:57.359 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47711.log 2024-05-28T06:27:57.360 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.66243.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.66243.log.gz 2024-05-28T06:27:57.362 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61392.log 2024-05-28T06:27:57.364 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.47711.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.47711.log.gz 2024-05-28T06:27:57.365 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67193.log 2024-05-28T06:27:57.366 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.61392.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.61392.log.gz 2024-05-28T06:27:57.369 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56961.log 2024-05-28T06:27:57.370 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.67193.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.67193.log.gz 2024-05-28T06:27:57.372 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.56961.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47615.log 2024-05-28T06:27:57.373 INFO:teuthology.orchestra.run.smithi096.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.56961.log.gz 2024-05-28T06:27:57.375 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66761.log 2024-05-28T06:27:57.376 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.47615.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.47615.log.gz 2024-05-28T06:27:57.378 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.66761.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87398.log 2024-05-28T06:27:57.379 INFO:teuthology.orchestra.run.smithi096.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.66761.log.gz 2024-05-28T06:27:57.383 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63023.log 2024-05-28T06:27:57.386 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.87398.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.87398.log.gz 2024-05-28T06:27:57.386 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64311.log 2024-05-28T06:27:57.388 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.63023.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.63023.log.gz 2024-05-28T06:27:57.391 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43367.log 2024-05-28T06:27:57.391 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.64311.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.64311.log.gz 2024-05-28T06:27:57.392 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46286.log 2024-05-28T06:27:57.393 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.43367.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.43367.log.gz 2024-05-28T06:27:57.397 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81546.log 2024-05-28T06:27:57.397 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.46286.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46286.log.gz 2024-05-28T06:27:57.403 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.81546.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43017.log 2024-05-28T06:27:57.404 INFO:teuthology.orchestra.run.smithi096.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.81546.log.gz 2024-05-28T06:27:57.406 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73833.log 2024-05-28T06:27:57.408 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.43017.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.43017.log.gz 2024-05-28T06:27:57.408 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71298.log 2024-05-28T06:27:57.410 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.73833.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.73833.log.gz 2024-05-28T06:27:57.414 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84719.log 2024-05-28T06:27:57.414 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.71298.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.71298.log.gz 2024-05-28T06:27:57.415 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.84719.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.84719.log.gz 2024-05-28T06:27:57.419 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72724.log 2024-05-28T06:27:57.422 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46748.log 2024-05-28T06:27:57.424 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.72724.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.72724.log.gz 2024-05-28T06:27:57.427 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.46748.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72892.log 2024-05-28T06:27:57.428 INFO:teuthology.orchestra.run.smithi096.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.46748.log.gz 2024-05-28T06:27:57.432 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78085.log 2024-05-28T06:27:57.433 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.72892.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.72892.log.gz 2024-05-28T06:27:57.435 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43366.log 2024-05-28T06:27:57.436 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.78085.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.78085.log.gz 2024-05-28T06:27:57.438 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71256.log 2024-05-28T06:27:57.439 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.43366.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43366.log.gz 2024-05-28T06:27:57.442 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.71256.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.71256.log.gz 2024-05-28T06:27:57.443 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43695.log 2024-05-28T06:27:57.446 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82053.log 2024-05-28T06:27:57.447 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.43695.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.43695.log.gz 2024-05-28T06:27:57.452 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.82053.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.82053.log.gz 2024-05-28T06:27:57.454 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80616.log 2024-05-28T06:27:57.458 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44613.log 2024-05-28T06:27:57.460 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.80616.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.80616.log.gz 2024-05-28T06:27:57.460 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64227.log 2024-05-28T06:27:57.461 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.44613.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.44613.log.gz 2024-05-28T06:27:57.464 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74145.log 2024-05-28T06:27:57.466 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.64227.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.64227.log.gz 2024-05-28T06:27:57.467 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.74145.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.74145.log.gz 2024-05-28T06:27:57.470 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61728.log 2024-05-28T06:27:57.472 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69452.log 2024-05-28T06:27:57.478 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.61728.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.61728.log.gz 2024-05-28T06:27:57.478 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.69452.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81462.log 2024-05-28T06:27:57.479 INFO:teuthology.orchestra.run.smithi096.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.69452.log.gz 2024-05-28T06:27:57.480 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70567.log 2024-05-28T06:27:57.482 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.81462.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.81462.log.gz 2024-05-28T06:27:57.483 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65931.log 2024-05-28T06:27:57.485 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.70567.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.70567.log.gz 2024-05-28T06:27:57.485 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54917.log 2024-05-28T06:27:57.486 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.65931.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.65931.log.gz 2024-05-28T06:27:57.488 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65720.log 2024-05-28T06:27:57.489 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.54917.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.54917.log.gz 2024-05-28T06:27:57.493 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.65720.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.65720.log.gz 2024-05-28T06:27:57.494 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78593.log 2024-05-28T06:27:57.496 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81759.log 2024-05-28T06:27:57.498 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.78593.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.78593.log.gz 2024-05-28T06:27:57.499 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67528.log 2024-05-28T06:27:57.501 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.81759.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.81759.log.gz 2024-05-28T06:27:57.504 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84256.log 2024-05-28T06:27:57.506 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.67528.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.67528.log.gz 2024-05-28T06:27:57.509 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68824.log 2024-05-28T06:27:57.510 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.84256.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.84256.log.gz 2024-05-28T06:27:57.512 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81843.log 2024-05-28T06:27:57.513 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.68824.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.68824.log.gz 2024-05-28T06:27:57.517 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.81843.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.81843.log.gz 2024-05-28T06:27:57.517 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65973.log 2024-05-28T06:27:57.521 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72808.log 2024-05-28T06:27:57.522 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.65973.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.65973.log.gz 2024-05-28T06:27:57.523 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49973.log 2024-05-28T06:27:57.525 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.72808.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.72808.log.gz 2024-05-28T06:27:57.525 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44445.log 2024-05-28T06:27:57.527 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.49973.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49973.log.gz 2024-05-28T06:27:57.528 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82557.log 2024-05-28T06:27:57.529 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.44445.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44445.log.gz 2024-05-28T06:27:57.532 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82389.log 2024-05-28T06:27:57.535 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.82557.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.82557.log.gz 2024-05-28T06:27:57.535 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76084.log 2024-05-28T06:27:57.537 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.82389.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.82389.log.gz 2024-05-28T06:27:57.539 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57003.log 2024-05-28T06:27:57.541 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.76084.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.76084.log.gz 2024-05-28T06:27:57.541 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44967.log 2024-05-28T06:27:57.545 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.57003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60019.log 2024-05-28T06:27:57.547 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.44967.log: 87.4% 89.5% -- replaced with /var/log/ceph/ceph-client.admin.44967.log.gz 2024-05-28T06:27:57.547 INFO:teuthology.orchestra.run.smithi096.stderr: -- replaced with /var/log/ceph/ceph-client.admin.57003.log.gz 2024-05-28T06:27:57.548 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58423.log 2024-05-28T06:27:57.549 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.60019.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.60019.log.gz 2024-05-28T06:27:57.553 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.58423.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46790.log 2024-05-28T06:27:57.557 INFO:teuthology.orchestra.run.smithi096.stderr: 88.6%/var/log/ceph/ceph-client.admin.46790.log: -- replaced with /var/log/ceph/ceph-client.admin.58423.log.gz 2024-05-28T06:27:57.559 INFO:teuthology.orchestra.run.smithi096.stderr: 87.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79147.log 2024-05-28T06:27:57.559 INFO:teuthology.orchestra.run.smithi096.stderr: -- replaced with /var/log/ceph/ceph-client.admin.46790.log.gz 2024-05-28T06:27:57.560 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87639.log 2024-05-28T06:27:57.566 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.87639.log: /var/log/ceph/ceph-client.admin.79147.log: 88.3% 88.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87019.log 2024-05-28T06:27:57.566 INFO:teuthology.orchestra.run.smithi096.stderr: -- replaced with /var/log/ceph/ceph-client.admin.87639.log.gz 2024-05-28T06:27:57.566 INFO:teuthology.orchestra.run.smithi096.stderr: -- replaced with /var/log/ceph/ceph-client.admin.79147.log.gz 2024-05-28T06:27:57.569 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47795.log 2024-05-28T06:27:57.572 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.87019.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.87019.log.gz 2024-05-28T06:27:57.573 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54874.log 2024-05-28T06:27:57.575 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.47795.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.47795.log.gz 2024-05-28T06:27:57.577 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66117.log 2024-05-28T06:27:57.578 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.54874.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.54874.log.gz 2024-05-28T06:27:57.580 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54479.log 2024-05-28T06:27:57.581 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.66117.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.66117.log.gz 2024-05-28T06:27:57.586 INFO:teuthology.orchestra.run.smithi096.stderr:gzip/var/log/ceph/ceph-client.admin.54479.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54479.log.gz 2024-05-28T06:27:57.586 INFO:teuthology.orchestra.run.smithi096.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.42466.log 2024-05-28T06:27:57.588 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50141.log 2024-05-28T06:27:57.589 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.42466.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.42466.log.gz 2024-05-28T06:27:57.591 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53802.log 2024-05-28T06:27:57.593 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.50141.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83461.log 2024-05-28T06:27:57.595 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.53802.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.50141.log.gz 2024-05-28T06:27:57.595 INFO:teuthology.orchestra.run.smithi096.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.53802.log.gz 2024-05-28T06:27:57.597 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61455.log 2024-05-28T06:27:57.599 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.83461.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.83461.log.gz 2024-05-28T06:27:57.603 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.61455.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.61455.log.gz 2024-05-28T06:27:57.603 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78635.log 2024-05-28T06:27:57.605 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72850.log 2024-05-28T06:27:57.607 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.78635.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66159.log 2024-05-28T06:27:57.609 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.72850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51402.log 2024-05-28T06:27:57.609 INFO:teuthology.orchestra.run.smithi096.stderr: 88.7% 87.6% -- replaced with /var/log/ceph/ceph-client.admin.78635.log.gz -- replaced with /var/log/ceph/ceph-client.admin.72850.log.gz 2024-05-28T06:27:57.609 INFO:teuthology.orchestra.run.smithi096.stderr: 2024-05-28T06:27:57.611 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.66159.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66552.log 2024-05-28T06:27:57.613 INFO:teuthology.orchestra.run.smithi096.stderr: 87.5%/var/log/ceph/ceph-client.admin.51402.log: -- replaced with /var/log/ceph/ceph-client.admin.66159.log.gz 2024-05-28T06:27:57.616 INFO:teuthology.orchestra.run.smithi096.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.51402.log.gz 2024-05-28T06:27:57.616 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65678.log 2024-05-28T06:27:57.619 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.66552.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.66552.log.gz 2024-05-28T06:27:57.619 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72556.log 2024-05-28T06:27:57.623 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.65678.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.65678.log.gz 2024-05-28T06:27:57.624 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74458.log 2024-05-28T06:27:57.625 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.72556.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.72556.log.gz 2024-05-28T06:27:57.627 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42424.log 2024-05-28T06:27:57.630 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.74458.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50227.log 2024-05-28T06:27:57.631 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.42424.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.74458.log.gz 2024-05-28T06:27:57.633 INFO:teuthology.orchestra.run.smithi096.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.42424.log.gz 2024-05-28T06:27:57.634 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52170.log 2024-05-28T06:27:57.636 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.50227.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.50227.log.gz 2024-05-28T06:27:57.640 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.52170.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52170.log.gz 2024-05-28T06:27:57.642 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68530.log 2024-05-28T06:27:57.645 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57501.log 2024-05-28T06:27:57.646 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.68530.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.68530.log.gz 2024-05-28T06:27:57.648 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80700.log 2024-05-28T06:27:57.649 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.57501.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.57501.log.gz 2024-05-28T06:27:57.652 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63233.log 2024-05-28T06:27:57.654 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.80700.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.80700.log.gz 2024-05-28T06:27:57.656 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44030.log 2024-05-28T06:27:57.658 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.63233.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.63233.log.gz 2024-05-28T06:27:57.661 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.44030.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49055.log 2024-05-28T06:27:57.662 INFO:teuthology.orchestra.run.smithi096.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44030.log.gz 2024-05-28T06:27:57.664 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87061.log 2024-05-28T06:27:57.665 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.49055.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49055.log.gz 2024-05-28T06:27:57.667 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67110.log 2024-05-28T06:27:57.669 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.87061.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.87061.log.gz 2024-05-28T06:27:57.673 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.67110.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83335.log 2024-05-28T06:27:57.674 INFO:teuthology.orchestra.run.smithi096.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.67110.log.gz 2024-05-28T06:27:57.676 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48431.log 2024-05-28T06:27:57.677 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.83335.log: 88.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67295.log 2024-05-28T06:27:57.679 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.48431.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44277.log 2024-05-28T06:27:57.680 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.67295.log: -- replaced with /var/log/ceph/ceph-client.admin.83335.log.gz 2024-05-28T06:27:57.682 INFO:teuthology.orchestra.run.smithi096.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48431.log.gz 2024-05-28T06:27:57.682 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78937.log 2024-05-28T06:27:57.684 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.44277.log: 88.3% 88.0% -- replaced with /var/log/ceph/ceph-client.admin.67295.log.gz 2024-05-28T06:27:57.685 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49681.log 2024-05-28T06:27:57.686 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.78937.log: -- replaced with /var/log/ceph/ceph-client.admin.44277.log.gz 2024-05-28T06:27:57.687 INFO:teuthology.orchestra.run.smithi096.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.78937.log.gz 2024-05-28T06:27:57.691 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.49681.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49681.log.gz 2024-05-28T06:27:57.693 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61351.log 2024-05-28T06:27:57.697 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85341.log 2024-05-28T06:27:57.698 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.61351.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64185.log 2024-05-28T06:27:57.699 INFO:teuthology.orchestra.run.smithi096.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.61351.log.gz 2024-05-28T06:27:57.699 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.85341.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.85341.log.gz 2024-05-28T06:27:57.701 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68008.log 2024-05-28T06:27:57.702 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.64185.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.64185.log.gz 2024-05-28T06:27:57.703 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68110.log 2024-05-28T06:27:57.704 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.68008.log: 96.3% -- replaced with /var/log/ceph/ceph-client.0.71726.log.gz 2024-05-28T06:27:57.705 INFO:teuthology.orchestra.run.smithi096.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.68008.log.gz 2024-05-28T06:27:57.706 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54832.log 2024-05-28T06:27:57.708 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.68110.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.68110.log.gz 2024-05-28T06:27:57.710 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55796.log 2024-05-28T06:27:57.711 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.54832.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.54832.log.gz 2024-05-28T06:27:57.713 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.55796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57543.log 2024-05-28T06:27:57.714 INFO:teuthology.orchestra.run.smithi096.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.55796.log.gz 2024-05-28T06:27:57.716 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62418.log 2024-05-28T06:27:57.717 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.57543.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.57543.log.gz 2024-05-28T06:27:57.720 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.62418.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.62418.log.gz 2024-05-28T06:27:57.720 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63462.log 2024-05-28T06:27:57.724 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79063.log 2024-05-28T06:27:57.726 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.63462.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.63462.log.gz 2024-05-28T06:27:57.729 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68152.log 2024-05-28T06:27:57.731 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.79063.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.79063.log.gz 2024-05-28T06:27:57.733 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70401.log 2024-05-28T06:27:57.735 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.68152.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.68152.log.gz 2024-05-28T06:27:57.735 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52608.log 2024-05-28T06:27:57.737 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.70401.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.70401.log.gz 2024-05-28T06:27:57.741 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.52608.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.52608.log.gz 2024-05-28T06:27:57.744 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75181.log 2024-05-28T06:27:57.745 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85778.log 2024-05-28T06:27:57.747 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.75181.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.75181.log.gz 2024-05-28T06:27:57.749 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82652.log 2024-05-28T06:27:57.751 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.85778.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.85778.log.gz 2024-05-28T06:27:57.751 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62082.log 2024-05-28T06:27:57.753 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.82652.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.82652.log.gz 2024-05-28T06:27:57.755 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46081.log 2024-05-28T06:27:57.757 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.62082.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.62082.log.gz 2024-05-28T06:27:57.757 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76749.log 2024-05-28T06:27:57.761 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.46081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52338.log 2024-05-28T06:27:57.762 INFO:teuthology.orchestra.run.smithi096.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.46081.log.gz 2024-05-28T06:27:57.762 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.76749.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.76749.log.gz 2024-05-28T06:27:57.764 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83293.log 2024-05-28T06:27:57.766 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.52338.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52338.log.gz 2024-05-28T06:27:57.770 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.83293.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48005.log 2024-05-28T06:27:57.772 INFO:teuthology.orchestra.run.smithi096.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.83293.log.gz 2024-05-28T06:27:57.773 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84845.log 2024-05-28T06:27:57.775 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.48005.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.48005.log.gz 2024-05-28T06:27:57.775 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79441.log 2024-05-28T06:27:57.777 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.84845.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.84845.log.gz 2024-05-28T06:27:57.779 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44739.log 2024-05-28T06:27:57.781 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.79441.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.79441.log.gz 2024-05-28T06:27:57.783 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63630.log 2024-05-28T06:27:57.785 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.44739.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.44739.log.gz 2024-05-28T06:27:57.785 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67612.log 2024-05-28T06:27:57.787 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.63630.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.63630.log.gz 2024-05-28T06:27:57.789 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59869.log 2024-05-28T06:27:57.790 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.67612.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.67612.log.gz 2024-05-28T06:27:57.793 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.59869.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.59869.log.gz 2024-05-28T06:27:57.794 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47489.log 2024-05-28T06:27:57.799 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66201.log 2024-05-28T06:27:57.800 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.47489.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50919.log 2024-05-28T06:27:57.801 INFO:teuthology.orchestra.run.smithi096.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.47489.log.gz 2024-05-28T06:27:57.805 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.66201.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.66201.log.gz 2024-05-28T06:27:57.806 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56874.log 2024-05-28T06:27:57.806 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.50919.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48289.log 2024-05-28T06:27:57.807 INFO:teuthology.orchestra.run.smithi096.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.50919.log.gz 2024-05-28T06:27:57.808 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.56874.log: 88.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54609.log 2024-05-28T06:27:57.810 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.48289.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.56874.log.gz 2024-05-28T06:27:57.811 INFO:teuthology.orchestra.run.smithi096.stderr: -- replaced with /var/log/ceph/ceph-client.admin.48289.log.gz 2024-05-28T06:27:57.813 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.54609.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83163.log 2024-05-28T06:27:57.814 INFO:teuthology.orchestra.run.smithi096.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.54609.log.gz 2024-05-28T06:27:57.814 INFO:teuthology.orchestra.run.smithi096.stderr: 96.3% -- replaced with /var/log/ceph/ceph-client.0.75861.log.gz 2024-05-28T06:27:57.815 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79231.log 2024-05-28T06:27:57.817 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.83163.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.83163.log.gz 2024-05-28T06:27:57.818 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44571.log 2024-05-28T06:27:57.820 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.79231.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.79231.log.gz 2024-05-28T06:27:57.823 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.44571.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73749.log 2024-05-28T06:27:57.823 INFO:teuthology.orchestra.run.smithi096.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.44571.log.gz 2024-05-28T06:27:57.824 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80742.log 2024-05-28T06:27:57.825 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.73749.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47000.log 2024-05-28T06:27:57.826 INFO:teuthology.orchestra.run.smithi096.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.73749.log.gz 2024-05-28T06:27:57.827 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.80742.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.80742.log.gz 2024-05-28T06:27:57.831 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79483.log 2024-05-28T06:27:57.833 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.47000.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47000.log.gz 2024-05-28T06:27:57.835 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49265.log 2024-05-28T06:27:57.837 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.79483.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.79483.log.gz 2024-05-28T06:27:57.839 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76042.log 2024-05-28T06:27:57.841 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.49265.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49265.log.gz 2024-05-28T06:27:57.842 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43143.log 2024-05-28T06:27:57.844 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.76042.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.76042.log.gz 2024-05-28T06:27:57.850 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.43143.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43143.log.gz 2024-05-28T06:27:57.851 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58339.log 2024-05-28T06:27:57.851 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77689.log 2024-05-28T06:27:57.853 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.58339.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.58339.log.gz 2024-05-28T06:27:57.853 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61956.log 2024-05-28T06:27:57.855 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.77689.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.77689.log.gz 2024-05-28T06:27:57.857 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57459.log 2024-05-28T06:27:57.859 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.61956.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.61956.log.gz 2024-05-28T06:27:57.861 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.57459.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.57459.log.gz 2024-05-28T06:27:57.862 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78211.log 2024-05-28T06:27:57.867 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67798.log 2024-05-28T06:27:57.869 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.78211.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64097.log 2024-05-28T06:27:57.869 INFO:teuthology.orchestra.run.smithi096.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.78211.log.gz 2024-05-28T06:27:57.873 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.67798.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58465.log 2024-05-28T06:27:57.873 INFO:teuthology.orchestra.run.smithi096.stderr: 88.8%/var/log/ceph/ceph-client.admin.64097.log: -- replaced with /var/log/ceph/ceph-client.admin.67798.log.gz 2024-05-28T06:27:57.874 INFO:teuthology.orchestra.run.smithi096.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.64097.log.gz 2024-05-28T06:27:57.876 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82940.log 2024-05-28T06:27:57.879 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.58465.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61518.log 87.9% -- replaced with /var/log/ceph/ceph-client.admin.58465.log.gz 2024-05-28T06:27:57.879 INFO:teuthology.orchestra.run.smithi096.stderr: 2024-05-28T06:27:57.879 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.82940.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.82940.log.gz 2024-05-28T06:27:57.880 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64141.log 2024-05-28T06:27:57.884 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.61518.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.61518.log.gz 2024-05-28T06:27:57.885 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.64141.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59287.log 2024-05-28T06:27:57.889 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72682.log 2024-05-28T06:27:57.889 INFO:teuthology.orchestra.run.smithi096.stderr: 87.4%/var/log/ceph/ceph-client.admin.59287.log: -- replaced with /var/log/ceph/ceph-client.admin.64141.log.gz 2024-05-28T06:27:57.892 INFO:teuthology.orchestra.run.smithi096.stderr: 88.7%/var/log/ceph/ceph-client.admin.72682.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.59287.log.gz 2024-05-28T06:27:57.894 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72388.log 2024-05-28T06:27:57.895 INFO:teuthology.orchestra.run.smithi096.stderr: -- replaced with /var/log/ceph/ceph-client.admin.72682.log.gz 2024-05-28T06:27:57.895 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52524.log 2024-05-28T06:27:57.896 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.72388.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.72388.log.gz 2024-05-28T06:27:57.898 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83419.log 2024-05-28T06:27:57.900 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.52524.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.52524.log.gz 2024-05-28T06:27:57.903 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66806.log 2024-05-28T06:27:57.905 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.83419.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.83419.log.gz 2024-05-28T06:27:57.905 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68404.log 2024-05-28T06:27:57.907 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.66806.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.66806.log.gz 2024-05-28T06:27:57.909 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72640.log 2024-05-28T06:27:57.911 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.68404.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.68404.log.gz 2024-05-28T06:27:57.911 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65190.log 2024-05-28T06:27:57.913 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.72640.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.72640.log.gz 2024-05-28T06:27:57.915 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.65190.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.65190.log.gz 2024-05-28T06:27:57.916 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79021.log 2024-05-28T06:27:57.920 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50057.log 2024-05-28T06:27:57.921 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.79021.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.79021.log.gz 2024-05-28T06:27:57.922 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64605.log 2024-05-28T06:27:57.924 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.50057.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.50057.log.gz 2024-05-28T06:27:57.926 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66057.log 2024-05-28T06:27:57.928 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.64605.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.64605.log.gz 2024-05-28T06:27:57.928 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59245.log 2024-05-28T06:27:57.930 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.66057.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.66057.log.gz 2024-05-28T06:27:57.934 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.59245.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.59245.log.gz 2024-05-28T06:27:57.936 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48761.log 2024-05-28T06:27:57.937 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84112.log 2024-05-28T06:27:57.938 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.48761.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.48761.log.gz 2024-05-28T06:27:57.942 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53340.log 2024-05-28T06:27:57.944 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.84112.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.84112.log.gz 2024-05-28T06:27:57.944 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75055.log 2024-05-28T06:27:57.946 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.53340.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.53340.log.gz 2024-05-28T06:27:57.948 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85860.log 2024-05-28T06:27:57.950 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.75055.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.75055.log.gz 2024-05-28T06:27:57.950 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70832.log 2024-05-28T06:27:57.954 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.85860.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67882.log 2024-05-28T06:27:57.956 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.70832.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.70832.log.gz 2024-05-28T06:27:57.956 INFO:teuthology.orchestra.run.smithi096.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.85860.log.gz 2024-05-28T06:27:57.959 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.67882.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61123.log 2024-05-28T06:27:57.960 INFO:teuthology.orchestra.run.smithi096.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.67882.log.gz 2024-05-28T06:27:57.964 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49391.log 2024-05-28T06:27:57.966 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.61123.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.61123.log.gz 2024-05-28T06:27:57.966 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74061.log 2024-05-28T06:27:57.967 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.49391.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.49391.log.gz 2024-05-28T06:27:57.971 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.74061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44037.log 2024-05-28T06:27:57.972 INFO:teuthology.orchestra.run.smithi096.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.74061.log.gz 2024-05-28T06:27:57.975 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56545.log 2024-05-28T06:27:57.976 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.44037.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44037.log.gz 2024-05-28T06:27:57.981 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53550.log 2024-05-28T06:27:57.982 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.56545.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.56545.log.gz 2024-05-28T06:27:57.984 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.53550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81885.log 2024-05-28T06:27:57.984 INFO:teuthology.orchestra.run.smithi096.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.53550.log.gz 2024-05-28T06:27:57.986 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75307.log 2024-05-28T06:27:57.987 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.81885.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.81885.log.gz 2024-05-28T06:27:57.991 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.75307.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.75307.log.gz 2024-05-28T06:27:57.991 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82179.log 2024-05-28T06:27:57.993 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77242.log 2024-05-28T06:27:57.995 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.82179.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80903.log 2024-05-28T06:27:57.996 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.77242.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.77242.log.gz 2024-05-28T06:27:57.998 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61309.log 2024-05-28T06:27:57.999 INFO:teuthology.orchestra.run.smithi096.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.82179.log.gz 2024-05-28T06:27:57.999 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.80903.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.80903.log.gz 2024-05-28T06:27:58.002 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71088.log 2024-05-28T06:27:58.004 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.61309.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.61309.log.gz 2024-05-28T06:27:58.008 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.71088.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69215.log 2024-05-28T06:27:58.009 INFO:teuthology.orchestra.run.smithi096.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.71088.log.gz 2024-05-28T06:27:58.013 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77145.log 2024-05-28T06:27:58.015 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.69215.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.69215.log.gz 2024-05-28T06:27:58.016 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81323.log 2024-05-28T06:27:58.018 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.77145.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.77145.log.gz 2024-05-28T06:27:58.020 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.81323.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77731.log 2024-05-28T06:27:58.021 INFO:teuthology.orchestra.run.smithi096.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.81323.log.gz 2024-05-28T06:27:58.022 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54352.log 2024-05-28T06:27:58.023 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.77731.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.77731.log.gz 2024-05-28T06:27:58.024 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86282.log 2024-05-28T06:27:58.026 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.54352.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.54352.log.gz 2024-05-28T06:27:58.030 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.86282.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.86282.log.gz 2024-05-28T06:27:58.031 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46916.log 2024-05-28T06:27:58.035 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.64991.log 2024-05-28T06:27:58.037 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.46916.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.46916.log.gz 2024-05-28T06:27:58.037 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85072.log 2024-05-28T06:27:58.041 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.0.64991.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76809.log 2024-05-28T06:27:58.043 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.85072.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.85072.log.gz 2024-05-28T06:27:58.046 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49762.log 2024-05-28T06:27:58.048 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.76809.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.76809.log.gz 2024-05-28T06:27:58.050 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79273.log 2024-05-28T06:27:58.052 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.49762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42343.log 2024-05-28T06:27:58.052 INFO:teuthology.orchestra.run.smithi096.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49762.log.gz 2024-05-28T06:27:58.059 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.42343.log: /var/log/ceph/ceph-client.admin.79273.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.42343.log.gz 2024-05-28T06:27:58.059 INFO:teuthology.orchestra.run.smithi096.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.79273.log.gz 2024-05-28T06:27:58.059 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67381.log 2024-05-28T06:27:58.062 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54959.log 2024-05-28T06:27:58.063 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.67381.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.67381.log.gz 2024-05-28T06:27:58.066 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81927.log 2024-05-28T06:27:58.068 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.54959.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.54959.log.gz 2024-05-28T06:27:58.070 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47879.log 2024-05-28T06:27:58.071 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.81927.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.81927.log.gz 2024-05-28T06:27:58.074 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76522.log 2024-05-28T06:27:58.076 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.47879.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.47879.log.gz 2024-05-28T06:27:58.078 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76665.log 2024-05-28T06:27:58.080 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.76522.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.76522.log.gz 2024-05-28T06:27:58.082 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47573.log 2024-05-28T06:27:58.084 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.76665.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.76665.log.gz 2024-05-28T06:27:58.087 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.47573.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.47573.log.gz 2024-05-28T06:27:58.089 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-05-28T06:27:58.092 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63191.log 2024-05-28T06:27:58.097 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43360.log 2024-05-28T06:27:58.098 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.63191.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.63191.log.gz 2024-05-28T06:27:58.101 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74416.log 2024-05-28T06:27:58.102 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.43360.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.43360.log.gz 2024-05-28T06:27:58.105 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75097.log 2024-05-28T06:27:58.107 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.74416.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.74416.log.gz 2024-05-28T06:27:58.109 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65763.log 2024-05-28T06:27:58.110 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.75097.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.75097.log.gz 2024-05-28T06:27:58.113 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.65763.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59594.log 2024-05-28T06:27:58.116 INFO:teuthology.orchestra.run.smithi096.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.65763.log.gz 2024-05-28T06:27:58.117 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42247.log 2024-05-28T06:27:58.118 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.59594.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.59594.log.gz 2024-05-28T06:27:58.121 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63504.log 2024-05-28T06:27:58.123 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.42247.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.42247.log.gz 2024-05-28T06:27:58.124 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.63504.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.63504.log.gz 2024-05-28T06:27:58.125 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86725.log 2024-05-28T06:27:58.129 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84761.log 2024-05-28T06:27:58.131 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.86725.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.86725.log.gz 2024-05-28T06:27:58.132 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46706.log 2024-05-28T06:27:58.134 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.84761.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.84761.log.gz 2024-05-28T06:27:58.134 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63767.log 2024-05-28T06:27:58.135 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.46706.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.46706.log.gz 2024-05-28T06:27:58.136 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86935.log 2024-05-28T06:27:58.142 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.63767.log: /var/log/ceph/ceph-client.admin.86935.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.86935.log.gz 2024-05-28T06:27:58.142 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50709.log 2024-05-28T06:27:58.142 INFO:teuthology.orchestra.run.smithi096.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.63767.log.gz 2024-05-28T06:27:58.144 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57231.log 2024-05-28T06:27:58.146 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.50709.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50709.log.gz 2024-05-28T06:27:58.148 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86324.log 2024-05-28T06:27:58.149 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.57231.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.57231.log.gz 2024-05-28T06:27:58.151 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42723.log 2024-05-28T06:27:58.152 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77815.log 2024-05-28T06:27:58.154 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.86324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57273.log 2024-05-28T06:27:58.154 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.77815.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.86324.log.gz 2024-05-28T06:27:58.156 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.57273.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.57273.log.gz 2024-05-28T06:27:58.157 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.42723.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42723.log.gz 2024-05-28T06:27:58.158 INFO:teuthology.orchestra.run.smithi096.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.77815.log.gz 2024-05-28T06:27:58.161 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69998.log 2024-05-28T06:27:58.162 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50625.log 2024-05-28T06:27:58.164 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.69998.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.69998.log.gz 2024-05-28T06:27:58.166 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43185.log 2024-05-28T06:27:58.168 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.50625.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.50625.log.gz 2024-05-28T06:27:58.170 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59827.log 2024-05-28T06:27:58.171 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.43185.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.43185.log.gz 2024-05-28T06:27:58.175 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.59827.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.59827.log.gz 2024-05-28T06:27:58.175 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53844.log 2024-05-28T06:27:58.177 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56587.log 2024-05-28T06:27:58.179 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.53844.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.53844.log.gz 2024-05-28T06:27:58.181 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87356.log 2024-05-28T06:27:58.183 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.56587.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.56587.log.gz 2024-05-28T06:27:58.186 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74798.log 2024-05-28T06:27:58.187 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.87356.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.87356.log.gz 2024-05-28T06:27:58.190 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.74798.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48803.log 2024-05-28T06:27:58.191 INFO:teuthology.orchestra.run.smithi096.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.74798.log.gz 2024-05-28T06:27:58.193 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81239.log 2024-05-28T06:27:58.195 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.48803.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.48803.log.gz 2024-05-28T06:27:58.197 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76935.log 2024-05-28T06:27:58.198 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.81239.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.81239.log.gz 2024-05-28T06:27:58.203 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.76935.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.76935.log.gz 2024-05-28T06:27:58.209 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69368.log 2024-05-28T06:27:58.209 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61872.log 2024-05-28T06:27:58.211 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.69368.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.69368.log.gz 2024-05-28T06:27:58.212 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75011.log 2024-05-28T06:27:58.213 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.61872.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.61872.log.gz 2024-05-28T06:27:58.216 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84635.log 2024-05-28T06:27:58.218 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.75011.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.75011.log.gz 2024-05-28T06:27:58.220 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69704.log 2024-05-28T06:27:58.222 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.84635.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.84635.log.gz 2024-05-28T06:27:58.224 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69956.log 2024-05-28T06:27:58.226 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.69704.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.69704.log.gz 2024-05-28T06:27:58.228 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61165.log 2024-05-28T06:27:58.230 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.69956.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.69956.log.gz 2024-05-28T06:27:58.232 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67030.log 2024-05-28T06:27:58.234 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.61165.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.61165.log.gz 2024-05-28T06:27:58.236 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53718.log 2024-05-28T06:27:58.237 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.67030.log: 85.4% -- replaced with /var/log/ceph/ceph-client.admin.67030.log.gz 2024-05-28T06:27:58.242 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.53718.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.53718.log.gz 2024-05-28T06:27:58.243 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59371.log 2024-05-28T06:27:58.248 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44038.log 2024-05-28T06:27:58.250 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.59371.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.59371.log.gz 2024-05-28T06:27:58.251 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54096.log 2024-05-28T06:27:58.255 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.44038.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56368.log 2024-05-28T06:27:58.256 INFO:teuthology.orchestra.run.smithi096.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44038.log.gz 2024-05-28T06:27:58.257 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.54096.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.54096.log.gz 2024-05-28T06:27:58.260 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87482.log 2024-05-28T06:27:58.262 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.56368.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.56368.log.gz 2024-05-28T06:27:58.265 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46622.log 2024-05-28T06:27:58.267 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.87482.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.87482.log.gz 2024-05-28T06:27:58.270 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.46622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67696.log 2024-05-28T06:27:58.271 INFO:teuthology.orchestra.run.smithi096.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.46622.log.gz 2024-05-28T06:27:58.273 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50499.log 2024-05-28T06:27:58.275 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.67696.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.67696.log.gz 2024-05-28T06:27:58.277 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87188.log 2024-05-28T06:27:58.278 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.50499.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.50499.log.gz 2024-05-28T06:27:58.282 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.87188.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53970.log 2024-05-28T06:27:58.284 INFO:teuthology.orchestra.run.smithi096.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.87188.log.gz 2024-05-28T06:27:58.286 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87721.log 2024-05-28T06:27:58.288 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.53970.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.53970.log.gz 2024-05-28T06:27:58.290 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48845.log 2024-05-28T06:27:58.291 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.87721.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.87721.log.gz 2024-05-28T06:27:58.294 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.2.45976.log 2024-05-28T06:27:58.296 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.48845.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.48845.log.gz 2024-05-28T06:27:58.298 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70524.log 2024-05-28T06:27:58.300 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.2.45976.log: 91.8% -- replaced with /var/log/ceph/ceph-client.2.45976.log.gz 2024-05-28T06:27:58.302 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63971.log 2024-05-28T06:27:58.304 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.70524.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.70524.log.gz 2024-05-28T06:27:58.308 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.63971.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.63971.log.gz 2024-05-28T06:27:58.310 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54702.log 2024-05-28T06:27:58.311 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49517.log 2024-05-28T06:27:58.312 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.54702.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.54702.log.gz 2024-05-28T06:27:58.313 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79189.log 2024-05-28T06:27:58.315 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.49517.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.49517.log.gz 2024-05-28T06:27:58.316 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73078.log 2024-05-28T06:27:58.319 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.79189.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86382.log 2024-05-28T06:27:58.321 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.73078.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.73078.log.gz 2024-05-28T06:27:58.321 INFO:teuthology.orchestra.run.smithi096.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.79189.log.gz 2024-05-28T06:27:58.323 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72766.log 2024-05-28T06:27:58.325 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.86382.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.86382.log.gz 2024-05-28T06:27:58.325 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50667.log 2024-05-28T06:27:58.329 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.50667.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.50667.log.gz 2024-05-28T06:27:58.331 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.72766.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63929.log 2024-05-28T06:27:58.331 INFO:teuthology.orchestra.run.smithi096.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.72766.log.gz 2024-05-28T06:27:58.334 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80050.log 2024-05-28T06:27:58.335 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.63929.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.63929.log.gz 2024-05-28T06:27:58.339 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67474.log 2024-05-28T06:27:58.339 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.80050.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.80050.log.gz 2024-05-28T06:27:58.339 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49804.log 2024-05-28T06:27:58.341 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.67474.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.67474.log.gz 2024-05-28T06:27:58.342 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83121.log 2024-05-28T06:27:58.344 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87272.log 2024-05-28T06:27:58.346 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.49804.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.49804.log.gz 2024-05-28T06:27:58.347 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.87272.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.87272.log.gz 2024-05-28T06:27:58.348 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.83121.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.83121.log.gz 2024-05-28T06:27:58.349 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46454.log 2024-05-28T06:27:58.352 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73427.log 2024-05-28T06:27:58.355 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.46454.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56410.log 2024-05-28T06:27:58.357 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.73427.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.73427.log.gz 2024-05-28T06:27:58.357 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.56410.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.46454.log.gz 2024-05-28T06:27:58.358 INFO:teuthology.orchestra.run.smithi096.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.56410.log.gz 2024-05-28T06:27:58.360 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54191.log 2024-05-28T06:27:58.362 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77563.log 2024-05-28T06:27:58.365 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.54191.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81801.log 2024-05-28T06:27:58.366 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.77563.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60820.log 2024-05-28T06:27:58.367 INFO:teuthology.orchestra.run.smithi096.stderr: 88.5% 88.8% -- replaced with /var/log/ceph/ceph-client.admin.54191.log.gz -- replaced with /var/log/ceph/ceph-client.admin.77563.log.gz 2024-05-28T06:27:58.367 INFO:teuthology.orchestra.run.smithi096.stderr: 2024-05-28T06:27:58.369 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.60820.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.60820.log.gz 2024-05-28T06:27:58.371 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.81801.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47084.log 2024-05-28T06:27:58.372 INFO:teuthology.orchestra.run.smithi096.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.81801.log.gz 2024-05-28T06:27:58.374 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47531.log 2024-05-28T06:27:58.376 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.47084.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47084.log.gz 2024-05-28T06:27:58.376 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83377.log 2024-05-28T06:27:58.378 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.47531.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.47531.log.gz 2024-05-28T06:27:58.379 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63672.log 2024-05-28T06:27:58.381 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.83377.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.83377.log.gz 2024-05-28T06:27:58.382 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66453.log 2024-05-28T06:27:58.384 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.63672.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.63672.log.gz 2024-05-28T06:27:58.388 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.66453.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55061.log 2024-05-28T06:27:58.389 INFO:teuthology.orchestra.run.smithi096.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.66453.log.gz 2024-05-28T06:27:58.392 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.83889.log 2024-05-28T06:27:58.394 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.55061.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.55061.log.gz 2024-05-28T06:27:58.400 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.0.83889.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44865.log 2024-05-28T06:27:58.402 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37011.log 2024-05-28T06:27:58.408 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.44865.log: /var/log/ceph/ceph-client.admin.37011.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.44865.log.gz 2024-05-28T06:27:58.409 INFO:teuthology.orchestra.run.smithi096.stderr: 92.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44035.log 2024-05-28T06:27:58.409 INFO:teuthology.orchestra.run.smithi096.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37011.log.gz 2024-05-28T06:27:58.412 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50541.log 2024-05-28T06:27:58.414 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.44035.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.44035.log.gz 2024-05-28T06:27:58.417 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61225.log 2024-05-28T06:27:58.418 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.50541.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50541.log.gz 2024-05-28T06:27:58.422 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55838.log 2024-05-28T06:27:58.422 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.61225.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.61225.log.gz 2024-05-28T06:27:58.424 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37278.log 2024-05-28T06:27:58.426 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.55838.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.55838.log.gz 2024-05-28T06:27:58.430 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.37278.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.37278.log.gz 2024-05-28T06:27:58.430 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86024.log 2024-05-28T06:27:58.432 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71991.log 2024-05-28T06:27:58.434 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.86024.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.86024.log.gz 2024-05-28T06:27:58.436 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73204.log 2024-05-28T06:27:58.437 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.71991.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.71991.log.gz 2024-05-28T06:27:58.440 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53592.log 2024-05-28T06:27:58.441 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.73204.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.73204.log.gz 2024-05-28T06:27:58.445 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.53592.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.53592.log.gz 2024-05-28T06:27:58.447 INFO:teuthology.orchestra.run.smithi096.stderr: -- /var/log/ceph/ceph-client.admin.57892.log 2024-05-28T06:27:58.451 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49559.log 2024-05-28T06:27:58.453 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.57892.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.57892.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82982.log 2024-05-28T06:27:58.453 INFO:teuthology.orchestra.run.smithi096.stderr: 2024-05-28T06:27:58.456 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.49559.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80658.log 2024-05-28T06:27:58.457 INFO:teuthology.orchestra.run.smithi096.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49559.log.gz 2024-05-28T06:27:58.460 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.82982.log: 88.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83026.log 2024-05-28T06:27:58.460 INFO:teuthology.orchestra.run.smithi096.stderr: -- replaced with /var/log/ceph/ceph-client.admin.82982.log.gz 2024-05-28T06:27:58.462 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.80658.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.80658.log.gz 2024-05-28T06:27:58.463 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84677.log 2024-05-28T06:27:58.467 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.83026.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.83026.log.gz 2024-05-28T06:27:58.467 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48677.log 2024-05-28T06:27:58.467 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.84677.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.84677.log.gz 2024-05-28T06:27:58.470 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74374.log 2024-05-28T06:27:58.472 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.48677.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48677.log.gz 2024-05-28T06:27:58.476 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.74374.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.74374.log.gz 2024-05-28T06:27:58.476 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49349.log 2024-05-28T06:27:58.479 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68614.log 2024-05-28T06:27:58.481 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.49349.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.49349.log.gz 2024-05-28T06:27:58.482 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82898.log 2024-05-28T06:27:58.483 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.68614.log: 88.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62124.log 2024-05-28T06:27:58.486 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.82898.log: -- replaced with /var/log/ceph/ceph-client.admin.68614.log.gz 2024-05-28T06:27:58.488 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.62124.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.82898.log.gz 2024-05-28T06:27:58.490 INFO:teuthology.orchestra.run.smithi096.stderr: 87.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69257.log 2024-05-28T06:27:58.491 INFO:teuthology.orchestra.run.smithi096.stderr: -- replaced with /var/log/ceph/ceph-client.admin.62124.log.gz 2024-05-28T06:27:58.494 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62939.log 2024-05-28T06:27:58.494 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.69257.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.69257.log.gz 2024-05-28T06:27:58.495 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73791.log 2024-05-28T06:27:58.496 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.62939.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.62939.log.gz 2024-05-28T06:27:58.499 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81113.log 2024-05-28T06:27:58.500 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.73791.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.73791.log.gz 2024-05-28T06:27:58.503 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69788.log 2024-05-28T06:27:58.505 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.81113.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.81113.log.gz 2024-05-28T06:27:58.508 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87524.log 2024-05-28T06:27:58.509 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.69788.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.69788.log.gz 2024-05-28T06:27:58.511 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78721.log 2024-05-28T06:27:58.513 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.87524.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.87524.log.gz 2024-05-28T06:27:58.520 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.78721.log: 88.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85695.log 2024-05-28T06:27:58.520 INFO:teuthology.orchestra.run.smithi096.stderr: -- replaced with /var/log/ceph/ceph-client.admin.78721.log.gz 2024-05-28T06:27:58.522 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44487.log 2024-05-28T06:27:58.525 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.85695.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.85695.log.gz 2024-05-28T06:27:58.528 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.44487.log: gzip -5 --verbose 87.5% -- replaced with /var/log/ceph/ceph-client.admin.44487.log.gz 2024-05-28T06:27:58.528 INFO:teuthology.orchestra.run.smithi096.stderr: -- /var/log/ceph/ceph-client.admin.48089.log 2024-05-28T06:27:58.531 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56646.log 2024-05-28T06:27:58.532 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.48089.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.48089.log.gz 2024-05-28T06:27:58.534 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72253.log 2024-05-28T06:27:58.536 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.56646.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.56646.log.gz 2024-05-28T06:27:58.538 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37379.log 2024-05-28T06:27:58.540 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.72253.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.72253.log.gz 2024-05-28T06:27:58.543 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70651.log 2024-05-28T06:27:58.544 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.37379.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.37379.log.gz 2024-05-28T06:27:58.546 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46412.log 2024-05-28T06:27:58.548 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.70651.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.70651.log.gz 2024-05-28T06:27:58.552 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.46412.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46412.log.gz 2024-05-28T06:27:58.976 INFO:teuthology.orchestra.run.smithi096.stderr: 96.2% -- replaced with /var/log/ceph/ceph-client.0.79827.log.gz 2024-05-28T06:27:59.249 INFO:teuthology.orchestra.run.smithi096.stderr: 90.1% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-05-28T06:27:59.674 INFO:teuthology.orchestra.run.smithi123.stderr: 93.5% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-05-28T06:27:59.718 INFO:teuthology.orchestra.run.smithi096.stderr: 96.3% -- replaced with /var/log/ceph/ceph-client.0.64991.log.gz 2024-05-28T06:27:59.892 INFO:teuthology.orchestra.run.smithi096.stderr: 96.3% -- replaced with /var/log/ceph/ceph-client.0.83889.log.gz 2024-05-28T06:28:00.673 INFO:teuthology.orchestra.run.smithi096.stderr: 96.3% -- replaced with /var/log/ceph/ceph-client.0.51263.log.gz 2024-05-28T06:28:01.297 INFO:teuthology.orchestra.run.smithi096.stderr: 96.3% -- replaced with /var/log/ceph/ceph-client.0.55615.log.gz 2024-05-28T06:28:01.630 INFO:teuthology.orchestra.run.smithi123.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-05-28T06:28:01.634 INFO:teuthology.orchestra.run.smithi123.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-05-28T06:28:01.693 INFO:teuthology.orchestra.run.smithi123.stderr: 92.7% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-05-28T06:28:02.037 INFO:teuthology.orchestra.run.smithi123.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-05-28T06:28:02.060 INFO:teuthology.orchestra.run.smithi096.stderr: 93.4% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-05-28T06:28:02.401 INFO:teuthology.orchestra.run.smithi123.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-05-28T06:28:02.406 INFO:teuthology.orchestra.run.smithi123.stderr: 2024-05-28T06:28:02.406 INFO:teuthology.orchestra.run.smithi123.stderr:real 0m6.365s 2024-05-28T06:28:02.407 INFO:teuthology.orchestra.run.smithi123.stderr:user 0m32.089s 2024-05-28T06:28:02.407 INFO:teuthology.orchestra.run.smithi123.stderr:sys 0m2.574s 2024-05-28T06:28:02.764 INFO:teuthology.orchestra.run.smithi096.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-05-28T06:28:02.804 INFO:teuthology.orchestra.run.smithi096.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-05-28T06:28:02.828 INFO:teuthology.orchestra.run.smithi096.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-05-28T06:28:03.036 INFO:teuthology.orchestra.run.smithi096.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-05-28T06:28:03.040 INFO:teuthology.orchestra.run.smithi096.stderr: 2024-05-28T06:28:03.040 INFO:teuthology.orchestra.run.smithi096.stderr:real 0m7.007s 2024-05-28T06:28:03.041 INFO:teuthology.orchestra.run.smithi096.stderr:user 0m40.222s 2024-05-28T06:28:03.041 INFO:teuthology.orchestra.run.smithi096.stderr:sys 0m5.348s 2024-05-28T06:28:03.042 INFO:tasks.ceph:Archiving logs... 2024-05-28T06:28:03.042 DEBUG:teuthology.misc:Transferring archived files from smithi096:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-05-28_05:40:25-fs:volumes-rishabh-clone-stats-testing-default-smithi/7729374/remote/smithi096/log 2024-05-28T06:28:03.043 DEBUG:teuthology.orchestra.run.smithi096:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-28T06:28:07.667 DEBUG:teuthology.misc:Transferring archived files from smithi123:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-05-28_05:40:25-fs:volumes-rishabh-clone-stats-testing-default-smithi/7729374/remote/smithi123/log 2024-05-28T06:28:07.668 DEBUG:teuthology.orchestra.run.smithi123:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-28T06:28:11.524 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-05-28T06:28:11.534 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-05-28T06:28:11.535 DEBUG:teuthology.orchestra.run.smithi096:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-28T06:28:11.543 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-28T06:28:11.688 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-05-28T06:28:11.698 INFO:teuthology.task.clock:Checking final clock skew... 2024-05-28T06:28:11.698 DEBUG:teuthology.orchestra.run.smithi096:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-28T06:28:11.706 DEBUG:teuthology.orchestra.run.smithi123:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-28T06:28:11.751 INFO:teuthology.orchestra.run.smithi096.stderr:bash: line 1: ntpq: command not found 2024-05-28T06:28:11.761 INFO:teuthology.orchestra.run.smithi123.stderr:bash: line 1: ntpq: command not found 2024-05-28T06:28:11.762 INFO:teuthology.orchestra.run.smithi096.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-28T06:28:11.762 INFO:teuthology.orchestra.run.smithi096.stdout:=============================================================================== 2024-05-28T06:28:11.763 INFO:teuthology.orchestra.run.smithi096.stdout:^* hv01.front.sepia.ceph.com 3 7 377 104 -330us[ -292us] +/- 52ms 2024-05-28T06:28:11.763 INFO:teuthology.orchestra.run.smithi096.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 37 -695us[ -695us] +/- 68ms 2024-05-28T06:28:11.763 INFO:teuthology.orchestra.run.smithi096.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 35 +1704us[+1704us] +/- 81ms 2024-05-28T06:28:11.763 INFO:teuthology.orchestra.run.smithi096.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-05-28T06:28:11.770 INFO:teuthology.orchestra.run.smithi123.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-28T06:28:11.771 INFO:teuthology.orchestra.run.smithi123.stdout:=============================================================================== 2024-05-28T06:28:11.771 INFO:teuthology.orchestra.run.smithi123.stdout:^* hv01.front.sepia.ceph.com 3 6 377 35 -397us[ -378us] +/- 53ms 2024-05-28T06:28:11.771 INFO:teuthology.orchestra.run.smithi123.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 38 -698us[ -679us] +/- 68ms 2024-05-28T06:28:11.771 INFO:teuthology.orchestra.run.smithi123.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 32 +1695us[+1695us] +/- 81ms 2024-05-28T06:28:11.771 INFO:teuthology.orchestra.run.smithi123.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-05-28T06:28:11.772 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-05-28T06:28:11.782 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-05-28T06:28:11.783 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-05-28T06:28:11.793 DEBUG:teuthology.orchestra.run.smithi096:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-28T06:28:12.016 DEBUG:teuthology.orchestra.run.smithi123:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-28T06:28:12.215 DEBUG:teuthology.orchestra.run.smithi096:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-28T06:28:12.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-28T06:28:12.314 DEBUG:teuthology.orchestra.run.smithi123:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-28T06:28:12.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-28T06:28:12.408 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-05-28T06:28:12.418 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-05-28T06:28:12.448 INFO:teuthology.task.internal:Duration was 2044.250598 seconds 2024-05-28T06:28:12.449 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-05-28T06:28:12.457 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-05-28T06:28:12.458 DEBUG:teuthology.orchestra.run.smithi096:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-28T06:28:12.466 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-28T06:28:12.610 INFO:teuthology.orchestra.run.smithi096.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-28T06:28:12.615 INFO:teuthology.orchestra.run.smithi123.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-28T06:28:13.077 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-05-28T06:28:13.077 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi096.front.sepia.ceph.com 2024-05-28T06:28:13.077 DEBUG:teuthology.orchestra.run.smithi096:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-28T06:28:13.143 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi123.front.sepia.ceph.com 2024-05-28T06:28:13.143 DEBUG:teuthology.orchestra.run.smithi123:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-28T06:28:13.220 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-05-28T06:28:13.221 DEBUG:teuthology.orchestra.run.smithi096:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-28T06:28:13.226 DEBUG:teuthology.orchestra.run.smithi123:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-28T06:28:13.307 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-05-28T06:28:13.307 DEBUG:teuthology.orchestra.run.smithi096:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-28T06:28:13.312 DEBUG:teuthology.orchestra.run.smithi123:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-28T06:28:13.524 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-05-28T06:28:13.535 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-05-28T06:28:13.535 DEBUG:teuthology.orchestra.run.smithi096:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-28T06:28:13.618 DEBUG:teuthology.orchestra.run.smithi123:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-28T06:28:13.699 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-05-28T06:28:13.709 DEBUG:teuthology.orchestra.run.smithi096:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-28T06:28:13.717 DEBUG:teuthology.orchestra.run.smithi123:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-28T06:28:13.809 INFO:teuthology.orchestra.run.smithi096.stdout:kernel.core_pattern = core 2024-05-28T06:28:13.811 INFO:teuthology.orchestra.run.smithi123.stdout:kernel.core_pattern = core 2024-05-28T06:28:13.850 DEBUG:teuthology.orchestra.run.smithi096:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-28T06:28:13.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-28T06:28:13.901 DEBUG:teuthology.orchestra.run.smithi123:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-28T06:28:13.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-28T06:28:13.955 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-05-28T06:28:13.966 INFO:teuthology.task.internal:Transferring archived files... 2024-05-28T06:28:13.967 DEBUG:teuthology.misc:Transferring archived files from smithi096:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-05-28_05:40:25-fs:volumes-rishabh-clone-stats-testing-default-smithi/7729374/remote/smithi096 2024-05-28T06:28:13.967 DEBUG:teuthology.orchestra.run.smithi096:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-28T06:28:14.069 DEBUG:teuthology.misc:Transferring archived files from smithi123:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-05-28_05:40:25-fs:volumes-rishabh-clone-stats-testing-default-smithi/7729374/remote/smithi123 2024-05-28T06:28:14.069 DEBUG:teuthology.orchestra.run.smithi123:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-28T06:28:14.166 INFO:teuthology.task.internal:Removing archive directory... 2024-05-28T06:28:14.166 DEBUG:teuthology.orchestra.run.smithi096:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-28T06:28:14.175 DEBUG:teuthology.orchestra.run.smithi123:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-28T06:28:14.228 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-05-28T06:28:14.240 INFO:teuthology.task.internal:Not uploading archives. 2024-05-28T06:28:14.240 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-05-28T06:28:14.252 INFO:teuthology.task.internal:Tidying up after the test... 2024-05-28T06:28:14.253 DEBUG:teuthology.orchestra.run.smithi096:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-28T06:28:14.260 DEBUG:teuthology.orchestra.run.smithi123:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-28T06:28:14.311 INFO:teuthology.orchestra.run.smithi096.stdout: 393712 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 28 06:28 /home/ubuntu/cephtest 2024-05-28T06:28:14.315 INFO:teuthology.orchestra.run.smithi123.stdout: 393249 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 28 06:28 /home/ubuntu/cephtest 2024-05-28T06:28:14.318 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-05-28T06:28:14.331 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-05-28T06:28:14.365 INFO:teuthology.run:Summary data: description: fs:volumes/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/volumes/{overrides test/clone-progress}} duration: 2044.250598192215 failure_reason: 'Test failure: test_when_7_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgressReporter)' flavor: default mon.a-kernel-sha1: b9459a26b0888785d05c47deb35abc9d0354ab93 mon.b-kernel-sha1: b9459a26b0888785d05c47deb35abc9d0354ab93 owner: scheduled_rishabh@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=58bbfcc312174deeb0e2710034db0d5a status: fail success: false 2024-05-28T06:28:14.366 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-28T06:28:14.457 INFO:teuthology.run:FAIL